Compare commits

...
1 Commits
Author SHA1 Message Date
ketle-manandClaude Sonnet 5 9d952c4b6e Add Unsloth backend to AI TOOL tab / sidebar A tab (v0.3.99)
Unsloth's OpenAI-compatible API always requires an API key, unlike the
existing Ollama/LM Studio/Lemonade backends, so it's relayed through a
new server-side proxy (py/routes/unsloth_routes.py) that attaches the
key from a local .env file (UNSLOTH_API_KEY, loaded via python-dotenv
in prestartup_script.py) instead of ever sending it to the browser.

Also fixes a client-controlled baseUrl SSRF that could have exfiltrated
the API key, and folds Unsloth's separate reasoning_content field back
into <think> tags so low Max tokens no longer produces silently empty
replies from reasoning models.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-18 17:51:13 +09:00
13 changed files with 302 additions and 21 deletions
+11
View File
@@ -0,0 +1,11 @@
# ComfyUI-Workflow-Studio - Unsloth backend (AI TOOL tab / sidebar A tab)
#
# Unsloth Desktop requires an API key even for local access
# (Settings -> API in the Unsloth app, shown only once when issued).
#
# 1. Copy this file to ".env" (same directory as this example file)
# 2. Paste your key after the "=" below (do NOT add a leading #, or the
# line becomes a comment and the key will not be loaded)
# 3. Restart ComfyUI
UNSLOTH_API_KEY=
+28
View File
@@ -2,6 +2,34 @@
---
## v0.3.99
### 機能追加: AI TOOLタブ + サイドパネルAタブにUnslothバックエンド追加(APIキーは.env管理、SSRF脆弱性修正、reasoning_content対応)
ユーザーから「作業計画のUnslothをAI TOOLタブ、サイドパネルAタブに導入したい。APIキーは.envファイルで設定したい」との依頼を受けて着手。v0.3.98時点で「今回は見送り」として`project_next_tasks`メモリに保留していたUnslothバックエンド追加を実装した一連のセッション。
**1. Unslothバックエンド追加**: Ollama/LM Studio/Lemonadeに続く4つ目のバックエンドとしてUnsloth(OpenAI互換API、既定`http://localhost:8888`)を追加。既存3バックエンドと異なりローカルアクセスでも常にAPIキー認証が必須(`Authorization: Bearer sk-unsloth-...`)なため、フロントエンドから直接fetchする既存方式ではなく、新設したサーバー側プロキシ(`py/routes/unsloth_routes.py`, `POST /api/wfm/unsloth/proxy`)経由にした。CivitAI連携(`civitai_service.py`)のAPIキー管理パターン(環境変数優先)を踏襲しつつ、今回はユーザーの要望通りAPIキーを`.env`ファイル(`UNSLOTH_API_KEY`)で管理する方式にした。`python-dotenv`を新規依存として追加し、ノードインポート前に実行される唯一のタイミングである`prestartup_script.py`(新規作成)で`.env`を読み込み`os.environ`へ反映する。フロントエンドはSPA(`ai-tab.js`)・サイドパネル(`node_sets_menu.js`)の両方で`fetchModels`/`callLLM`/`callVLM`/`callChat`各関数に`backend === "unsloth"`分岐を追加(既存の重複実装パターンを踏襲)。
**2. セキュリティレビュー対応(SSRF修正)**: 実装直後の自動セキュリティレビューで、プロキシがクライアント指定の`baseUrl`をそのまま信頼しており、悪意のあるページが`baseUrl`に任意の外部URLを指定すれば`UNSLOTH_API_KEY`を外部へ流出させられる脆弱性(SSRF経由の資格情報窃取)が指摘された。`baseUrl`のホスト名を`localhost`/`127.0.0.1`/`::1`のみに限定するアローリスト検証を追加して修正(ポート番号は自由のままとし、Unslothのカスタムポート起動に対応)。
**3. `.env.example`のコメントアウトの罠**: 動作確認時、ユーザーが`.env.example`の`# UNSLOTH_API_KEY=sk-unsloth-...`という例示行をそのまま`.env`にコピーし、先頭の`#`を消し忘れてコメントアウトされたまま値を入力してしまい「UNSLOTH_API_KEY is not set」エラーが発生。テンプレート自体を「空の実行可能な行」(`UNSLOTH_API_KEY=`)+手順コメントの形に変更し、同じ罠が再発しないようにした。
**4. Unsloth Desktop側「モデル自動切り替え」設定の仕様判明**: 動作確認中、Unsloth側の「モデル自動切り替え (OpenAI API)」設定がOFFだと、Unsloth Studio UI上で事前にモデルを手動ロードしていない限りAPIリクエストがHTTP 404で失敗することが判明(公式ドキュメント調査で確認:未知のモデル名は安全に現在ロード中のモデルへフォールバックするが、通常は明示的にONにしてAPI経由のモデル切り替えを許可する必要がある仕様)。本プラグイン側の不具合ではなくUnsloth Desktop自体の仕様のため、ヘルプ(`ai-6`)とREADMEに「Unsloth Desktop側でこの設定をONにする必要がある」旨を追記した。
**5. `reasoning_content`未対応バグ修正**: 「モデル自動切り替え」をONにした後も、会話が数ターン進むとチャットの応答が空のバブルになる不具合が発生。Unsloth Studioのサーバーログ(`~/.unsloth/studio/logs/server/`)と、ブラウザ操作MCP(kapture)の`evaluate`経由でプロキシへ生のAPIリクエストを直接送って実レスポンスを検証した結果、Unslothのreasoningモデル(`gemma-4-12B-it-QAT-GGUF`)はOpenAI互換APIで思考過程を`message.content`ではなく別フィールド`message.reasoning_content`に返す仕様であることが判明した。Ollama/LM Studio/Lemonadeはインラインの`<think>...</think>`タグ方式のため既存のThinking mode処理で問題にならなかったが、Unslothは別フィールド方式のため、Max tokens不足(初期値512)で思考中にトークンを使い切ると(`finish_reason: "length"`)、`content`が常に空文字列のまま返り、チャットに中身のないバブルだけが残っていた。`reasoning_content`を`<think>...</think>`として`content`に合成してから既存のThinking mode ON/OFF処理(`stripThinkingTags`)に流すよう修正し、SPA・サイドパネル両方に適用。Max tokensが小さすぎると根本的に回答本文が生成されない制約自体は残るため、ヘルプに「Unslothのreasoningモデルを使う場合はMax tokensを2048以上に」という推奨値も追記した。
**ヘルプタブ更新**: `project_v0336_conventions`の慣習(index.html + i18n.js 3言語 + app.jsのhelpIdMap)に従い、AI TOOL TabヘルプのSettings pane説明(`ai-6`)にUnsloth関連3項目(APIキーの`.env`設定・モデル自動切り替え要件・`reasoning_content`/Max tokens注意)を追記。`sidepanel-16`・`ai-2`にもバックエンド一覧へUnslothを追加。README.mdのBackend support・Settings pane等の該当箇所も同様に更新。
**検証**: 全JS/PythonファイルをそれぞれNode.jsの構文チェック(`node --check`)・`ast.parse()`で確認。開発元gitリポジトリと実行時`custom_nodes`フォルダは別実体のため([[project_dev_deploy_sync]]参照)、変更の都度`diff`で意図した差分のみであることを確認してから両フォルダへ同期し、`cmp`でファイル同一性を検証。実行時フォルダの`__pycache__`もクリア。最終的にユーザー自身が実機(Unsloth Desktop + ComfyUI)で一通り動作確認済み。
**How to apply**:
1. 外部APIキーが常に必須なバックエンドを追加する場合、既存バックエンドと同じ「フロントエンドから直接fetch」方式は使えない(APIキーがブラウザに露出する)。サーバー側プロキシ方式が必須になる。
2. サーバー側プロキシでクライアント指定のURLをそのまま信頼してAuthorizationヘッダーを付与すると、SSRF経由の資格情報漏洩につながる。ホスト名のアローリスト検証は必須。
3. `.env`テンプレートに「コメントアウトされた例」を書くと、ユーザーが`#`を消し忘れてハマる典型パターンになる。テンプレートは「空の実行可能な行」+説明コメントの形にする。
4. サードパーティAPIサーバーの独自拡張(`reasoning_content`等)は、標準的なOpenAI Chat Completions仕様と食い違うことがある。空応答やトークン切れなど不可解な挙動が出たら、実際の生レスポンスをブラウザから直接fetchして中身を見るのが最も早い切り分け方法(kaptureの`evaluate`ツールでプロキシへ直接リクエストを送るのが有効だった)。
---
## v0.3.98
### 機能追加・バグ修正: AI TOOL翻訳の信頼性向上 + GenerateUI Modelタブ再構成(ハイライト色・折りたたみ・並び替え)+ Thinking mode/Max tokens設定
+9 -7
View File
@@ -22,7 +22,7 @@ A comprehensive workflow, asset management, and generation UI plugin for [ComfyU
- Built-in AI tools (translation and more)
![Workflow Studio](https://img.shields.io/badge/ComfyUI-Custom_Node-blue)
![Version](https://img.shields.io/badge/version-0.3.98-green)
![Version](https://img.shields.io/badge/version-0.3.99-green)
## Screenshots
@@ -375,9 +375,9 @@ An experimental batch generator: runs the workflow currently loaded in GenerateU
<summary><h3>AI TOOL Tab (v0.3.14)</h3></summary>
- **4-pane layout** — Translation | Chat | TOOLS | Settings; all panes always visible simultaneously; no sub-tab switching required
- **Translation pane** — translate text between Japanese, English, Chinese, or a custom Free language using Ollama, LM Studio, or Lemonade; language selectors with ⇄ swap button (swaps both language selectors and text content); selections saved automatically
- **Translation pane** — translate text between Japanese, English, Chinese, or a custom Free language using Ollama, LM Studio, Lemonade, or Unsloth; language selectors with ⇄ swap button (swaps both language selectors and text content); selections saved automatically
- **Reliability fix** (v0.3.98) — switched from a raw single-string prompt to a system-role chat request (Ollama `/api/chat`, LM Studio/Lemonade `/v1/chat/completions` with a `system` message) so small local models follow the "translation only" instruction more reliably; output is cleaned of common LLM preambles/quotes, and a warning toast appears if the model appears to have echoed the source text unchanged instead of translating it
- **Chat pane** (v0.3.40) — multi-turn conversation with the LLM; full conversation history sent each turn for context; Enter to send, Shift+Enter for a newline; Clear button resets history; Ollama uses `/api/chat`, LM Studio and Lemonade use `/v1/chat/completions`
- **Chat pane** (v0.3.40) — multi-turn conversation with the LLM; full conversation history sent each turn for context; Enter to send, Shift+Enter for a newline; Clear button resets history; Ollama uses `/api/chat`, LM Studio, Lemonade, and Unsloth use `/v1/chat/completions`
- **Chat pane — image generation via Tool Calling** (v0.3.73) — if the selected model supports tool calling and the user asks for an image, a `generate_image` tool is invoked and the request runs through the same generation pipeline as the GenerateUI Generate button, with the result shown inline in the chat; an "Allow image generation" checkbox in the input row toggles this on/off (default on); the workflow used is the one currently loaded in GenerateUI, or a dedicated saved workflow configured in the Settings pane; models verified to work well for tool calling and vision: **Gemma 4:e4b**, **Qwen 3.5:9b**, **Ministral 3:3b** (Ollama)
- **TOOLS pane (VLM)** — drop an image into the 110px drop zone, select a task (Describe Image / Create Prompt / Create Tags), and click Run to analyze with a vision model; result shown in the output area with a Copy button
- **TOOLS pane — shared Chat attachment** (v0.3.74) — an image dropped in the TOOLS drop zone is also usable as the Chat pane's attachment: it's sent to the LLM as vision input, and used as the base image when `generate_image` runs image-to-image; a ✕ button clears it (from either the TOOLS preview or the Chat attachment indicator); it stays attached across turns until explicitly cleared
@@ -385,12 +385,13 @@ An experimental batch generator: runs the workflow currently loaded in GenerateU
- **Chat pane — image-to-image (I2I)** (v0.3.74) — when an image is attached and `generate_image` is invoked, the attachment is uploaded and swapped into the target workflow's LoadImage node instead of running text-to-image; the target workflow is the one currently loaded in GenerateUI, or a dedicated I2I workflow configured in the Settings pane
- **Chat pane — SVG generation, no ComfyUI workflow involved** (v0.3.77) — the LLM can also produce SVG graphics directly as text; if a reply contains SVG code (wrapped in a markdown code fence or written directly as a bare `<svg>...</svg>`), it's rendered inline as a preview and a "Save to Gallery" button writes it as a `.svg` file to the Gallery tab's output folder; unlike the Tool Calling image generation flow above, this is plain text pattern matching, so it works with any model regardless of tool-calling support
- **Chat pane — Skills** (v0.3.79) — select a `.md` skill file from a dropdown above the message list to prepend its instructions to the conversation as a system prompt (not shown in the visible chat log); a pencil button opens an in-pane manager (list + inline editor, same pattern as the Prompt tab's wildcard file manager) to create, edit, or delete skill files stored in `user/default/Workflow-Studio/ai_skills/`; a file's `---`-delimited frontmatter (`name`/`description`) is shown in the dropdown; two starter skills are bundled — **SVG Icon Generator** (explicitly decomposes the subject into parts before drawing, which markedly improves recognizability from small local models) and **Skill Creator** (interviews you conversationally and drafts a new skill as a ```skill code block; a "Save as new skill" button appears under such replies and opens the manager pre-filled with the draft)
- **Settings pane** — choose backend (Ollama / LM Studio / Lemonade), set the API URL, test connection, select a model (with refresh button), and configure Free language names for translation source and destination
- **Settings pane** — choose backend (Ollama / LM Studio / Lemonade / Unsloth), set the API URL, test connection, select a model (with refresh button), and configure Free language names for translation source and destination
- **Unsloth backend** (v0.3.99) — Unsloth Desktop's OpenAI-compatible API always requires an API key, even for local access; unlike the other backends it's proxied server-side (`py/routes/unsloth_routes.py`, `POST /api/wfm/unsloth/proxy`) so the key never reaches the browser: set `UNSLOTH_API_KEY` in a `.env` file in the plugin folder (copy `.env.example`, loaded via `python-dotenv` in `prestartup_script.py`) and restart ComfyUI; also requires **"Auto model switching (OpenAI API)"** to be turned on in the Unsloth Desktop app's own Settings screen — with it off, requests fail with HTTP 404 unless a model was already manually loaded in the Unsloth Studio UI
- **Settings pane — Chat Image Generation** (v0.3.73) — "Use dedicated workflow" checkbox lets Chat-triggered image generation use a saved workflow of your choice instead of the one currently loaded in GenerateUI, without disturbing what's shown there (same pattern as Image Edit's Inpaint dedicated workflow)
- **Settings pane — Chat I2I Generation** (v0.3.74) — same "Use dedicated workflow" pattern as above, but for image-to-image generation when an image is attached; independent of the text-to-image dedicated workflow setting
- **Settings pane — Generation** (v0.3.98) — **Thinking mode** toggle (off by default; sends Ollama's `think` flag, and regardless of backend, any `<think>...</think>` block is stripped from the output when off so reasoning models don't clutter Translation/Chat/TOOLS results) and **Max tokens** (Ollama: `options.num_predict`; LM Studio/Lemonade: `max_tokens`; `0` = unlimited)
- **Settings pane — Generation** (v0.3.98) — **Thinking mode** toggle (off by default; sends Ollama's `think` flag, and regardless of backend, any `<think>...</think>` block is stripped from the output when off so reasoning models don't clutter Translation/Chat/TOOLS results) and **Max tokens** (Ollama: `options.num_predict`; LM Studio/Lemonade/Unsloth: `max_tokens`; `0` = unlimited)
- **Settings shared** — settings saved to `localStorage` under `wfm_ai_settings`; shared with the Library panel's AI TOOL tab so configuration is consistent across both interfaces
- **Backend support** — Ollama (`/api/generate` for text, `/api/chat` for conversations, `/api/tags` for model list); LM Studio and Lemonade via the OpenAI-compatible API (`/v1/chat/completions`, `/v1/models`); VLM images sent as base64 (`images:[]` for Ollama, `image_url` content block for LM Studio/Lemonade)
- **Backend support** — Ollama (`/api/generate` for text, `/api/chat` for conversations, `/api/tags` for model list); LM Studio and Lemonade via the OpenAI-compatible API directly from the browser (`/v1/chat/completions`, `/v1/models`); Unsloth via the same OpenAI-compatible API but relayed through a server-side proxy that attaches the `Authorization: Bearer` header; VLM images sent as base64 (`images:[]` for Ollama, `image_url` content block for LM Studio/Lemonade/Unsloth)
- **URL security** — backend URL validated via `new URL()` to enforce `http://` or `https://` scheme
- **Lemonade's other endpoints (not integrated)** — Lemonade Server also exposes OpenAI-compatible image generation (`/v1/images/generations`), text-to-speech (`/v1/audio/speech`), and transcription (`/v1/audio/transcriptions`) endpoints; these are not wired up — image generation would duplicate the existing ComfyUI-workflow-based `generate_image` tool call, and audio/TTS has no corresponding UI yet, so both are future considerations rather than a natural extension of the current backend switch
@@ -407,7 +408,7 @@ An experimental batch generator: runs the workflow currently loaded in GenerateU
- **M — Models tab** — browse installed models (All / ★ Favorites / Groups / By Type sub-tabs); LoRA groups show an **All N LoRAs** item — drag to canvas to place a `Lora Loader (LoraManager)` node with all LoRAs pre-loaded
- **P — Prompts tab** — browse prompt presets with All / ★ Favorites / Categories sub-tabs; **Groups sub-tab** (row 2) — view presets by group (shared with the Batch tab's `wfm_prompt_preset_groups`)
- **I — Information tab** — drop a ComfyUI-generated PNG/WebP or workflow JSON in the side panel to view its metadata; detects LoRAs from `LoraLoader`, `LoraLoaderModelOnly`, and `Lora Loader (LoraManager)` nodes (API format supported); supports `UnetLoaderGGUF` and `QuadrupleCLIPLoader` node types; preview area fixed at 110px
- **A — AI TOOL tab** — Translation, Chat, TOOLS, and Settings sub-tabs powered by Ollama, LM Studio, or Lemonade; Chat supports multi-turn conversations (full history sent each turn); TOOLS includes VLM image analysis and wildcard generation; settings (backend, URL, model) shared with the SPA AI TOOL tab via `localStorage`
- **A — AI TOOL tab** — Translation, Chat, TOOLS, and Settings sub-tabs powered by Ollama, LM Studio, Lemonade, or Unsloth; Chat supports multi-turn conversations (full history sent each turn); TOOLS includes VLM image analysis and wildcard generation; settings (backend, URL, model) shared with the SPA AI TOOL tab via `localStorage`
- **model sub-tab** — Checkpoint, VAE, Diffusion Model, and Text Encoder; drag items to canvas to place the corresponding loader node (Checkpoint → `CheckpointLoaderSimple`, VAE → `VAELoader`, Diffusion Model → `UNETLoader`, Text Encoder → `CLIPLoader`); double-click also places at canvas center
- **lora sub-tab** — detects LoRAs from `LoraLoader`, `LoraLoaderModelOnly`, and `Lora Loader (LoraManager)` nodes (API format `inputs.loras.__value__` supported); shows `strength_model / strength_clip` values; drag individual LoRA to place `LoraLoader`; **Multiple LORA** section (appears for 1+ LoRAs) drags all LoRAs into a single `Lora Loader (LoraManager)` node with LoRA syntax pre-filled
- **Prompts sub-tab** — POS / NEG badge list; drag a prompt to place `CLIPTextEncode` with text pre-filled; click any entry to view full text + Copy button
@@ -504,6 +505,7 @@ Click the **camera icon** (next to the W button) in ComfyUI's top bar to capture
- **[Ollama](https://ollama.com/)** — for AI chat assistant, translation, and VLM features
- **[LM Studio](https://lmstudio.ai/)** — alternative backend for translation and VLM (OpenAI-compatible API)
- **[Lemonade Server](https://lemonade-server.ai/)** — alternative backend for translation, chat, and VLM (OpenAI-compatible API)
- **[Unsloth](https://unsloth.ai/)** — alternative backend for translation, chat, and VLM (OpenAI-compatible API); always requires an API key, set via `UNSLOTH_API_KEY` in a `.env` file — `pip install -r requirements.txt` for `python-dotenv`
- **[Eagle](https://eagle.cool/)** — for auto-saving generated images with metadata
- **[comfyui-mask-editor-one](https://github.com/ketle-man/comfyui-mask-editor-one) (v0.1.9+)** — enables BiRefNet background removal, SAM3 text-prompt segmentation, and ABR stamp-brush library in the Image Edit Mask tool; `birefnet.safetensors` must be placed in `ComfyUI/models/background_removal/` for BiRefNet
+24
View File
@@ -0,0 +1,24 @@
"""Runs before ComfyUI imports any custom node — the only place early enough
to populate os.environ for code that reads env vars at import/request time.
Loads a .env file (if present) from this plugin's root directory so the
Unsloth backend's API key (UNSLOTH_API_KEY) can be set without editing
settings.json or exposing it to the frontend.
"""
from pathlib import Path
PLUGIN_DIR = Path(__file__).resolve().parent
ENV_FILE = PLUGIN_DIR / ".env"
if ENV_FILE.exists():
try:
from dotenv import load_dotenv
load_dotenv(ENV_FILE)
except ImportError:
print(
"[WARNING] Workflow Studio: .env file found but python-dotenv is not installed "
"(pip install -r requirements.txt). UNSLOTH_API_KEY will not be loaded from .env."
)
except Exception as e:
print(f"[WARNING] Workflow Studio: failed to load .env file: {e}")
+99
View File
@@ -0,0 +1,99 @@
"""Unsloth API proxy routes.
Unlike Ollama/LM Studio/Lemonade, Unsloth Desktop requires an API key even
for local access (Authorization: Bearer sk-unsloth-...). The key lives in
UNSLOTH_API_KEY (loaded from a .env file by prestartup_script.py) and is
never sent to the frontend — the frontend calls this proxy with the target
path/method/payload, and the server attaches the Authorization header before
relaying to Unsloth's OpenAI-compatible API.
"""
import json
import logging
import os
import urllib.request
from urllib.error import HTTPError, URLError
from urllib.parse import urlparse
from aiohttp import web
logger = logging.getLogger(__name__)
UNSLOTH_DEFAULT_URL = "http://localhost:8888"
_ALLOWED_PATHS = {"/v1/models", "/v1/chat/completions"}
# baseUrl is client-supplied (so a custom Unsloth port works), but the
# Authorization header carries a real secret — restrict the host to loopback
# so this proxy can't be used to exfiltrate UNSLOTH_API_KEY to an arbitrary
# server (SSRF). Port is unrestricted.
_ALLOWED_HOSTS = {"localhost", "127.0.0.1", "::1"}
def _get_api_key():
"""Return the Unsloth API key from the environment (.env), or None."""
return os.environ.get("UNSLOTH_API_KEY", "").strip() or None
def _is_allowed_base_url(base_url):
"""Only relay (and attach the API key) to a loopback host."""
try:
parsed = urlparse(base_url)
return parsed.scheme in ("http", "https") and parsed.hostname in _ALLOWED_HOSTS
except Exception:
return False
def setup_routes(app: web.Application):
"""Register Unsloth API routes."""
app.router.add_post("/api/wfm/unsloth/proxy", handle_proxy)
async def handle_proxy(request: web.Request) -> web.Response:
"""POST /api/wfm/unsloth/proxy - Relay a request to Unsloth's OpenAI-compatible API.
Body: { baseUrl, path: "/v1/models" | "/v1/chat/completions", method: "GET" | "POST", payload }
"""
import asyncio
try:
body = await request.json()
base_url = (body.get("baseUrl") or UNSLOTH_DEFAULT_URL).rstrip("/")
path = body.get("path") or "/v1/models"
method = (body.get("method") or "GET").upper()
payload = body.get("payload")
if path not in _ALLOWED_PATHS or method not in ("GET", "POST"):
return web.json_response({"message": "Unsupported proxy target"}, status=400)
if not _is_allowed_base_url(base_url):
return web.json_response({
"message": "Unsloth backend URL must point to localhost/127.0.0.1/::1",
}, status=400)
api_key = _get_api_key()
if not api_key:
return web.json_response({
"message": "UNSLOTH_API_KEY is not set. Copy .env.example to .env in the "
"plugin folder, fill in the key, and restart ComfyUI.",
}, status=401)
def _fetch():
data = json.dumps(payload).encode("utf-8") if payload is not None else None
headers = {"Authorization": f"Bearer {api_key}"}
if data is not None:
headers["Content-Type"] = "application/json"
req = urllib.request.Request(f"{base_url}{path}", data=data, headers=headers, method=method)
with urllib.request.urlopen(req, timeout=120) as resp:
return json.loads(resp.read().decode("utf-8"))
data = await asyncio.to_thread(_fetch)
return web.json_response(data)
except HTTPError as e:
detail = e.read().decode("utf-8", errors="replace") if e.fp else str(e)
logger.warning("Unsloth proxy HTTP error: %s %s", e.code, detail)
return web.json_response({"message": f"Unsloth API error: HTTP {e.code}"}, status=e.code)
except URLError as e:
logger.warning("Unsloth proxy connection error: %s", e)
return web.json_response({"message": f"Could not reach Unsloth: {e.reason}"}, status=502)
except Exception as e:
logger.error("Unsloth proxy error: %s", e)
return web.json_response({"message": str(e)}, status=500)
+2 -1
View File
@@ -80,11 +80,12 @@ class WorkflowStudio:
app.router.add_get("/wfm", serve_index_page)
# API routes
from .routes import workflow_routes, settings_routes, ollama_routes, eagle_routes, nodes_routes, prompts_routes, models_routes, gallery_routes, wildcard_routes, tagger_routes, gmic_routes, skill_routes, lab_routes
from .routes import workflow_routes, settings_routes, ollama_routes, unsloth_routes, eagle_routes, nodes_routes, prompts_routes, models_routes, gallery_routes, wildcard_routes, tagger_routes, gmic_routes, skill_routes, lab_routes
workflow_routes.setup_routes(app)
settings_routes.setup_routes(app)
ollama_routes.setup_routes(app)
unsloth_routes.setup_routes(app)
eagle_routes.setup_routes(app)
nodes_routes.setup_routes(app)
prompts_routes.setup_routes(app)
+1 -1
View File
@@ -1,7 +1,7 @@
[project]
name = "comfyui-workflow-studio"
description = "A comprehensive workflow, asset management, and generation UI plugin for ComfyUI. Manage workflows, models, images, and prompts; batch generation and Image Feeder in the GenerateUI tab; library side panel with canvas drag & drop and AI tools."
version = "0.3.98"
version = "0.3.99"
license = "MIT"
requires-python = ">= 3.10"
dependencies = ["jinja2"]
+3
View File
@@ -19,3 +19,6 @@ piexif>=1.1.3
# Omit if you do not use DeepDanbooru models.
# TensorFlow is large (~600MB) and conflicts with some CUDA setups.
# tensorflow>=2.10
# -- Required to load UNSLOTH_API_KEY from a local .env file (Unsloth backend, AI TOOL tab) --
python-dotenv>=1.0,<2.0
+56
View File
@@ -86,6 +86,18 @@ async function callVLM(url, backend, model, prompt, base64Image, mimeType, setti
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const text = (await res.json()).response || "";
return settings.thinkingMode ? text : stripThinkingTags(text);
} else if (backend === "unsloth") {
const body = _applyGenOptions({
model,
messages: [{ role: "user", content: [
{ type: "text", text: prompt },
{ type: "image_url", image_url: { url: `data:${mimeType};base64,${base64Image}` } },
]}],
stream: false,
}, backend, settings);
const data = await unslothProxy(url, "/v1/chat/completions", "POST", body);
const text = _unslothContent(data.choices?.[0]?.message);
return settings.thinkingMode ? text : stripThinkingTags(text);
} else {
const res = await fetch(`${url}/v1/chat/completions`, {
method: "POST",
@@ -105,6 +117,35 @@ async function callVLM(url, backend, model, prompt, base64Image, mimeType, setti
}
}
// ============================================
// Unsloth proxy (server relays the request, attaching the API key from .env
// so it never reaches the frontend — see py/routes/unsloth_routes.py)
// ============================================
async function unslothProxy(baseUrl, path, method, payload) {
const res = await fetch("/api/wfm/unsloth/proxy", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ baseUrl, path, method, payload }),
});
const data = await res.json().catch(() => ({}));
if (!res.ok) throw new Error(data.message || `HTTP ${res.status}`);
return data;
}
// Unlike Ollama/LM Studio/Lemonade (which inline reasoning as <think> tags
// inside content), Unsloth's OpenAI-compatible API returns it in a separate
// `reasoning_content` field. Fold it back into a <think> block so the
// existing Thinking-mode show/strip logic (_applyGenOptions/stripThinkingTags)
// still applies uniformly. Without this, a low max_tokens can make the model
// spend its whole budget reasoning and return an empty `content` with no
// visible explanation.
function _unslothContent(message) {
const reasoning = message?.reasoning_content;
const content = message?.content || "";
return reasoning ? `<think>${reasoning}</think>${content}` : content;
}
// ============================================
// URL validation
// ============================================
@@ -128,6 +169,9 @@ async function fetchModels(url, backend) {
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const data = await res.json();
return (data.models || []).map((m) => m.name);
} else if (backend === "unsloth") {
const data = await unslothProxy(url, "/v1/models", "GET");
return (data.data || []).map((m) => m.id);
} else {
// LM Studio (OpenAI-compatible)
const res = await fetch(`${url}/v1/models`);
@@ -153,6 +197,11 @@ async function callLLM(url, backend, model, prompt, settings = {}) {
const data = await res.json();
const text = data.response || "";
return settings.thinkingMode ? text : stripThinkingTags(text);
} else if (backend === "unsloth") {
const body = _applyGenOptions({ model, messages: [{ role: "user", content: prompt }], stream: false }, backend, settings);
const data = await unslothProxy(url, "/v1/chat/completions", "POST", body);
const text = _unslothContent(data.choices?.[0]?.message);
return settings.thinkingMode ? text : stripThinkingTags(text);
} else {
// LM Studio (OpenAI-compatible)
const res = await fetch(`${url}/v1/chat/completions`, {
@@ -543,6 +592,13 @@ async function callChat(url, backend, model, messages, tools, settings = {}) {
const message = (await res.json()).message || {};
const content = message.content || "";
return { content: settings.thinkingMode ? content : stripThinkingTags(content), toolCalls: message.tool_calls || null };
} else if (backend === "unsloth") {
const body = _applyGenOptions({ model, messages: formattedMessages, stream: false }, backend, settings);
if (tools) { body.tools = tools; body.tool_choice = "auto"; }
const data = await unslothProxy(url, "/v1/chat/completions", "POST", body);
const message = data.choices?.[0]?.message || {};
const content = _unslothContent(message);
return { content: settings.thinkingMode ? content : stripThinkingTags(content), toolCalls: message.tool_calls || null };
} else {
const body = _applyGenOptions({ model, messages: formattedMessages, stream: false }, backend, settings);
if (tools) { body.tools = tools; body.tool_choice = "auto"; }
+9 -9
View File
@@ -939,14 +939,14 @@ const LANGUAGES = {
helpSidepanel13: "I tab (Information): drop a ComfyUI PNG/WebP/JSON to inspect model, LoRA, and prompt metadata in the side panel — sub-tabs: model / lora / Prompts; supports UnetLoaderGGUF, QuadrupleCLIPLoader, and Lora Loader (LoraManager) node types",
helpSidepanel14: "Information tab — drag items to canvas: model sub-tab places the loader node (Checkpoint → CheckpointLoaderSimple, VAE → VAELoader, Diffusion Model → UNETLoader, Text Encoder → CLIPLoader); double-click also places at canvas center",
helpSidepanel15: "Information tab — LoRA sub-tab: detects LoRAs from LoraLoader, LoraLoaderModelOnly, and Lora Loader (LoraManager) nodes (API format supported); drag individual LoRA to place LoraLoader; \"Multiple LORA\" section places all LoRAs in a single Lora Loader (LoraManager) node with LoRA syntax pre-filled; Prompts sub-tab: drag a prompt to place CLIPTextEncode node with text pre-filled; click prompt to view full text and copy",
helpSidepanel16: "A tab (AI Tools): Translation, Chat, TOOLS, and Settings sub-tabs powered by Ollama, LM Studio, or Lemonade; Chat supports multi-turn conversations; TOOLS includes VLM image analysis and wildcard generation; settings shared with the SPA AI TOOL tab via localStorage",
helpSidepanel16: "A tab (AI Tools): Translation, Chat, TOOLS, and Settings sub-tabs powered by Ollama, LM Studio, Lemonade, or Unsloth; Chat supports multi-turn conversations; TOOLS includes VLM image analysis and wildcard generation; settings shared with the SPA AI TOOL tab via localStorage",
helpSidepanel17: "Send to Canvas: clicking 'Send to Canvas' (Workflow tab) or 'Copy & Send Canvas' (Gallery tab) sends the workflow directly to the ComfyUI canvas — both UI and API formats are supported; if Workflow Studio was not opened from the ComfyUI toolbar (window.opener unavailable), UI-format workflows fall back to title drag (title turns blue with a green ●)",
helpAiTitle: "AI Tab (A)",
helpAi1: "4-pane layout — Translation | Chat | TOOLS | Settings — all panes are always visible simultaneously",
helpAi2: "Translation pane: translate between Japanese, English, Chinese, or Free language; ⇄ swap button swaps both language selectors and text content; selections are saved automatically",
helpAi2: "Translation pane: translate between Japanese, English, Chinese, or Free language using Ollama, LM Studio, Lemonade, or Unsloth; ⇄ swap button swaps both language selectors and text content; selections are saved automatically",
helpAi3: "Chat pane: multi-turn conversation with the LLM; Enter to send a message, Shift+Enter for a newline; Clear button resets the conversation history; if the model supports tool calling and asks for an image (\"generate a cat picture\", etc.) it will run generation on the workflow currently loaded in GenerateUI and show the result inline; the \"Allow image generation\" checkbox in the pane header toggles this on/off; if an image is attached (dropped in the TOOLS pane) it is sent to the LLM as vision input and, when generate_image is called, used as the base for image-to-image generation instead of text-to-image; models verified to work well for tool calling and vision: Gemma 4:e4b, Qwen 3.5:9b, Ministral 3:3b (Ollama)",
helpAi4: "TOOLS pane — VLM: drop an image (PNG/JPG/WebP) and select a task (Describe Image, Create Prompt, Create Tags), click Run; or select \"Create wildcards\", enter a category name and count to generate plain-text entries one per line; the dropped image is also shared with the Chat pane as its attachment (shown there and used for image-to-image generation) until cleared with the ✕ button",
helpAi6: "Settings pane: choose backend (Ollama / LM Studio / Lemonade), set API URL, test connection, select model; Generation section — Thinking mode toggle (off by default; Ollama receives a think flag, and regardless of backend any <think>...</think> block is stripped from the output when off, so reasoning models don't clutter Translation/Chat/TOOLS results) and Max tokens (Ollama: num_predict, LM Studio/Lemonade: max_tokens; 0 = unlimited); configure Free language names; Chat Image Generation section lets you enable a dedicated workflow for Chat-triggered text-to-image generation instead of the one currently loaded in GenerateUI; Chat I2I Generation section is the same but for image-to-image generation when an image is attached; settings are shared with the Library panel's A tab via localStorage",
helpAi6: "Settings pane: choose backend (Ollama / LM Studio / Lemonade / Unsloth), set API URL, test connection, select model; Unsloth always requires an API key even for local access — set UNSLOTH_API_KEY in a .env file in the plugin folder (copy .env.example) and restart ComfyUI, the key is read server-side and never sent to the browser; Unsloth also requires \"Auto model switching (OpenAI API)\" to be turned ON in the Unsloth Desktop app itself (its own Settings screen) — without it, requests fail with HTTP 404 unless a model was already manually loaded in the Unsloth Studio UI; Unsloth reasoning models return their thinking in a separate reasoning_content field rather than inline <think> tags, so a low Max tokens can make the model spend its whole budget reasoning and return an empty reply — increase Max tokens (2048+) if replies come back blank; Generation section — Thinking mode toggle (off by default; Ollama receives a think flag, and regardless of backend any <think>...</think> block is stripped from the output when off, so reasoning models don't clutter Translation/Chat/TOOLS results) and Max tokens (Ollama: num_predict, LM Studio/Lemonade/Unsloth: max_tokens; 0 = unlimited); configure Free language names; Chat Image Generation section lets you enable a dedicated workflow for Chat-triggered text-to-image generation instead of the one currently loaded in GenerateUI; Chat I2I Generation section is the same but for image-to-image generation when an image is attached; settings are shared with the Library panel's A tab via localStorage",
helpAi7: "Chat pane — SVG: the LLM can also produce SVG graphics directly as text, with no ComfyUI workflow involved; ask it to draw something in SVG and, if the reply contains SVG code, it is rendered inline as a preview with a \"Save to Gallery\" button that writes it as a .svg file to the Gallery tab's output folder; this does not use tool calling, so it works with any model regardless of tool-calling support",
helpAi8: "Chat pane — Skills: pick a skill from the dropdown above the message list to prepend its instructions to the conversation as a system prompt (not shown in the visible chat log); use the pencil button to open the skill manager and create, edit, or delete .md skill files, stored in user/default/Workflow-Studio/ai_skills/; a file's first line block between --- markers can define name/description shown in the dropdown; two starter skills are included by default: \"SVG Icon Generator\" (decomposes the subject into parts before drawing, for more recognizable SVG output) and \"Skill Creator\" (interviews you conversationally and drafts a new skill file, ready to save); when a reply contains a ```skill code block, a \"Save as new skill\" button appears below it that opens the manager with the draft pre-filled",
helpAi9: "Tips for writing your own skills: give a clear frontmatter name/description so it reads well in the dropdown and tooltip; state the exact output format you want and, if there is a deliverable to extract later (like SVG or another skill file), wrap it in a distinctive fenced code block; write concrete, imperative instructions rather than vague ones - smaller local models follow explicit rules and a short worked example far better than general guidance; state hard constraints (forbidden content, security, length limits) explicitly rather than assuming they are implied; when in doubt, select the \"Skill Creator\" skill in the dropdown and describe what you want - it will interview you and draft the file for you",
@@ -2191,14 +2191,14 @@ const LANGUAGES = {
helpSidepanel13: "I タブ(Information):ComfyUI の PNG / WebP / JSON をドロップするとサイドパネル内でモデル・LoRA・プロンプトのメタデータを確認可能 — サブタブ: model / lora / Prompts;UnetLoaderGGUF・QuadrupleCLIPLoader・Lora Loader (LoraManager) ノードタイプに対応",
helpSidepanel14: "Information タブ — キャンバスへドラッグ:model サブタブのアイテムをドラッグすると対応ローダーノードを配置(Checkpoint → CheckpointLoaderSimple、VAE → VAELoader、Diffusion Model → UNETLoader、Text Encoder → CLIPLoader);ダブルクリックでもキャンバス中央に配置可能",
helpSidepanel15: "Information タブ — lora サブタブ:LoraLoader・LoraLoaderModelOnly・Lora Loader (LoraManager) ノード(APIフォーマット含む)から LoRA を検出;個別 LoRA をドラッグで LoraLoader を配置;「Multiple LORA」セクションで全 LoRA を Lora Loader (LoraManager) ノードに一括配置(LoRA構文自動入力);Prompts サブタブ:プロンプトをドラッグするとテキスト入力済みの CLIPTextEncode ノードを配置;クリックで全文表示・コピー可能",
helpSidepanel16: "A タブ(AI ツール):翻訳・Chat・TOOLS・設定のサブタブ;Chat は LLM とのマルチターン会話;TOOLS は VLM 画像解析とワイルドカード生成に対応;設定は SPA の AI タブと localStorage で共有される",
helpSidepanel16: "A タブ(AI ツール):翻訳・Chat・TOOLS・設定のサブタブ。Ollama、LM Studio、Lemonade、Unsloth に対応;Chat は LLM とのマルチターン会話;TOOLS は VLM 画像解析とワイルドカード生成に対応;設定は SPA の AI タブと localStorage で共有される",
helpSidepanel17: "Send to Canvas:WorkflowタブのSend to CanvasまたはGalleryタブのCopy & Send Canvasをクリックするとワークフローをキャンバスへ直接送信(UI形式・API形式どちらも対応);ComfyUIのツールバーから開いていない場合はUI形式のみタイトルドラッグのフォールバック(タイトルが青くなり緑の●が表示される — ドラッグで読み込み)",
helpAiTitle: "AI タブ(A)",
helpAi1: "4ペインレイアウト — Translation | Chat | TOOLS | Settings — 全ペインが常時表示される",
helpAi2: "翻訳ペイン:Ollama、LM Studio、または Lemonade を使って日本語・英語・中国語・カスタム言語(Free)間でテキストを翻訳;⇄ ボタンで言語とテキストを同時入替;選択は自動保存",
helpAi2: "翻訳ペイン:Ollama、LM Studio、Lemonade、または Unsloth を使って日本語・英語・中国語・カスタム言語(Free)間でテキストを翻訳;⇄ ボタンで言語とテキストを同時入替;選択は自動保存",
helpAi3: "Chat ペイン:LLM とのマルチターン会話;Enter で送信、Shift+Enter で改行;Clear ボタンで会話履歴をリセット;モデルがtool callingに対応している場合、「猫の画像を作って」のように頼むとGenerateUIに読み込み中のワークフローで生成を実行し、結果をチャット内に表示する;ペインヘッダーの「画像生成を許可する」チェックボックスでこの機能のON/OFFを切り替えられる;画像が添付されている場合(TOOLSペインにドロップした画像)はLLMにもビジョン入力として送信され、generate_image呼び出し時はT2Iの代わりにI2I(image-to-image)の元画像として使われる;tool callingとビジョンの動作を確認済みのモデル: Gemma 4:e4b、Qwen 3.5:9b、Ministral 3:3b(Ollama)",
helpAi4: "TOOLS ペイン — VLM:画像(PNG/JPG/WebP)をドロップしてタスク(画像解説 / プロンプト作成 / タグ作成)を選んで Run;または「Create wildcards」を選んでカテゴリ名と件数を入力すると1行1エントリのワイルドカードを生成;ドロップした画像はChatペインの添付としても共有され(Chat側にも表示、I2I生成に使用)、✕ボタンでクリアするまで保持される",
helpAi6: "Settings ペイン:バックエンド(Ollama / LM Studio / Lemonade)の選択、APIのURL設定、接続テスト、モデル選択;Generationセクション — Thinking modeトグル(デフォルトOFF;OllamaにはthinkフラグとしてOFF/ONを送信、バックエンドを問わずOFF時は出力中の<think>...</think>ブロックをクライアント側で除去するため、推論モデルの思考過程がTranslation/Chat/TOOLSの結果に混入しない)とMax tokens(Ollama: num_predict、LM Studio/Lemonade: max_tokens;0で無制限);Free言語名の設定;Chat Image Generationセクションでは、ChatからのT2I画像生成にGenerateUIの現在の読み込みワークフローの代わりに専用ワークフローを使うよう設定できる;Chat I2I Generationセクションは同様の設定を画像添付時のI2I生成向けに提供する;設定は localStorage に保存され、Library パネルの A タブと共有される",
helpAi6: "Settings ペイン:バックエンド(Ollama / LM Studio / Lemonade / Unsloth)の選択、APIのURL設定、接続テスト、モデル選択;Unslothはローカルアクセスでも常にAPIキー認証が必須——プラグインフォルダの .env.example を .env にコピーして UNSLOTH_API_KEY を設定し ComfyUI を再起動する(キーはサーバー側でのみ読み込まれ、ブラウザには一切送信されない);Unsloth Desktopアプリ側の設定画面で「モデル自動切り替え (OpenAI API)」もONにする必要がある——OFFのままだと、Unsloth Studio上で事前にモデルを手動ロードしていない限り HTTP 404 で接続に失敗する;Unslothのreasoningモデルは思考内容をインラインの<think>タグではなく別のreasoning_contentフィールドで返すため、Max tokensが少ないとモデルが思考だけでトークンを使い切り、空の返信になることがある——返信が空になる場合はMax tokensを増やす(2048以上を推奨);Generationセクション — Thinking modeトグル(デフォルトOFF;OllamaにはthinkフラグとしてOFF/ONを送信、バックエンドを問わずOFF時は出力中の<think>...</think>ブロックをクライアント側で除去するため、推論モデルの思考過程がTranslation/Chat/TOOLSの結果に混入しない)とMax tokens(Ollama: num_predict、LM Studio/Lemonade/Unsloth: max_tokens;0で無制限);Free言語名の設定;Chat Image Generationセクションでは、ChatからのT2I画像生成にGenerateUIの現在の読み込みワークフローの代わりに専用ワークフローを使うよう設定できる;Chat I2I Generationセクションは同様の設定を画像添付時のI2I生成向けに提供する;設定は localStorage に保存され、Library パネルの A タブと共有される",
helpAi7: "Chat ペイン — SVG:LLMはComfyUIワークフローを一切使わず、SVGグラフィックをテキストとして直接生成することもできる;「SVGで〜を描いて」のように頼み、応答にSVGコードが含まれていれば自動でインラインプレビュー表示され、「Galleryに保存」ボタンでGalleryタブの出力フォルダに.svgファイルとして保存できる;tool calling(function calling)を使用しないため、tool calling非対応のモデルでも動作する",
helpAi8: "Chat ペイン — スキル:メッセージ一覧上部のドロップダウンでスキルを選ぶと、その指示内容が system prompt として会話の先頭に付加される(表示上のチャット履歴には含まれない);鉛筆ボタンでスキル管理パネルを開き、.mdスキルファイルを作成・編集・削除できる(保存先: user/default/Workflow-Studio/ai_skills/);ファイル先頭の---で囲まれた部分にname/descriptionを書くとドロップダウンの表示に反映される;初期状態で2つのスキルを同梱: 「SVG Icon Generator」(描く前に対象を部位分解することで認識しやすいSVGを生成)と「Skill Creator」(対話形式で質問しながら新しいスキルファイルを下書きする);応答に```skillコードブロックが含まれる場合、メッセージ下部に「スキルとして保存」ボタンが表示され、内容が入った状態で管理パネルが開く",
helpAi9: "スキル作成のヒント:ドロップダウンやツールチップで分かりやすいよう、frontmatterのname/descriptionは明確に書く;期待する出力形式を具体的に指示し、SVGや別のスキルファイルのように後で抽出したい成果物がある場合は専用のコードフェンスで囲むよう指示する;曖昧な指示より具体的で命令形の指示の方がローカルの小型モデルには効きやすく、短い実例を1つ添えるとさらに効果的;禁止事項(不適切な内容、セキュリティ、文字数制限など)は暗黙に期待せず明示的に書く;迷ったらドロップダウンで「Skill Creator」スキルを選び、作りたいスキルの概要を伝えれば対話形式で下書きしてくれる",
@@ -3443,14 +3443,14 @@ const LANGUAGES = {
helpSidepanel13: "I 标签(Information):拖放 ComfyUI PNG/WebP/JSON 即可在侧面板内查看模型、LoRA 和提示词元数据 — 子标签:model / lora / Prompts;支持 UnetLoaderGGUF、QuadrupleCLIPLoader 和 Lora Loader (LoraManager) 节点类型",
helpSidepanel14: "Information 标签 — 拖拽到画布:model 子标签拖拽可放置对应加载节点(Checkpoint → CheckpointLoaderSimple、VAE → VAELoader、Diffusion Model → UNETLoader、Text Encoder → CLIPLoader);双击也可放置到画布中央",
helpSidepanel15: "Information 标签 — lora 子标签:可检测 LoraLoader、LoraLoaderModelOnly 及 Lora Loader (LoraManager) 节点(含API格式)的 LoRA;拖拽单个 LoRA 放置 LoraLoader;「Multiple LORA」区域可将全部 LoRA 一次性放入 Lora Loader (LoraManager) 节点并自动填入LoRA语法;Prompts 子标签:拖拽提示词放置预填文本的 CLIPTextEncode 节点;点击可查看全文并复制",
helpSidepanel16: "A 标签(AI 工具):翻译、Chat、TOOLS 和设置子标签;Chat 支持与 LLM 的多轮对话;TOOLS 包含 VLM 图像分析和通配符生成;设置通过 localStorage 与 SPA 的 AI 标签共享",
helpSidepanel16: "A 标签(AI 工具):翻译、Chat、TOOLS 和设置子标签,支持 Ollama、LM Studio、Lemonade 或 Unsloth;Chat 支持与 LLM 的多轮对话;TOOLS 包含 VLM 图像分析和通配符生成;设置通过 localStorage 与 SPA 的 AI 标签共享",
helpSidepanel17: "Send to Canvas:在Workflow标签点击'Send to Canvas'或Gallery标签点击'Copy & Send Canvas',工作流直接发送到ComfyUI画布(支持UI和API所有格式);若Workflow Studio不是从ComfyUI工具栏打开,UI格式工作流将回退到标题拖拽(标题变蓝并显示绿色●——拖拽到画布加载)",
helpAiTitle: "AI 标签页(A)",
helpAi1: "4 面板布局 — Translation | Chat | TOOLS | Settings — 所有面板始终可见",
helpAi2: "翻译面板:使用 Ollama、LM Studio 或 Lemonade 在日语、英语、中文或自定义语言(Free)之间翻译;⇄ 按钮同时互换语言和文本内容;选择自动保存",
helpAi2: "翻译面板:使用 Ollama、LM Studio、Lemonade 或 Unsloth 在日语、英语、中文或自定义语言(Free)之间翻译;⇄ 按钮同时互换语言和文本内容;选择自动保存",
helpAi3: "Chat 面板:与 LLM 进行多轮对话;Enter 发送消息,Shift+Enter 换行;Clear 按钮重置对话历史;如果模型支持 tool calling,当你要求生成图像(例如\"生成一张猫的图片\")时,会使用 GenerateUI 中当前加载的工作流执行生成,并在对话中显示结果;面板标题栏的\"允许生成图像\"复选框可开关此功能;如果附加了图像(在 TOOLS 面板中拖放的图像),该图像会作为视觉输入发送给 LLM,调用 generate_image 时会以该图像为基础执行图生图(I2I),而不是文生图;已验证可正常支持 tool calling 与视觉功能的模型:Gemma 4:e4b、Qwen 3.5:9b、Ministral 3:3b(Ollama)",
helpAi4: "TOOLS 面板 — VLM:拖放图像(PNG/JPG/WebP)并选择任务(图像描述 / 提示词生成 / 标签生成),点击 Run;或选择「Create wildcards」,输入类别名称和数量生成纯文本通配符条目(每行一条);拖放的图像也会作为 Chat 面板的附件共享(在该处显示并用于图生图生成),直到点击 ✕ 按钮清除",
helpAi6: "Settings 面板:选择后端(Ollama / LM Studio / Lemonade)、设置 API URL、测试连接、选择模型;Generation 部分 — Thinking mode 开关(默认关闭;Ollama会收到think标志,无论使用哪个后端,关闭时都会从输出中去除<think>...</think>思考块,避免推理模型的思考过程混入Translation/Chat/TOOLS的结果)以及 Max tokens(Ollama: num_predict,LM Studio/Lemonade: max_tokens;0为不限);配置 Free 语言名称;Chat 图像生成部分可让 Chat 触发的文生图使用专用工作流,而不是 GenerateUI 中当前加载的工作流;Chat I2I 生成部分提供相同设置,用于附加图像时的图生图生成;设置保存至 localStorage,并与 Library 面板的 A 标签共享",
helpAi6: "Settings 面板:选择后端(Ollama / LM Studio / Lemonade / Unsloth)、设置 API URL、测试连接、选择模型;Unsloth 即使在本地访问时也始终需要 API 密钥认证——将插件文件夹中的 .env.example 复制为 .env 并设置 UNSLOTH_API_KEY,然后重启 ComfyUI(密钥仅在服务器端读取,绝不会发送到浏览器);还需要在 Unsloth Desktop 应用自身的设置界面中开启「Auto model switching (OpenAI API)」——若保持关闭,除非已在 Unsloth Studio 中手动加载过模型,否则请求会以 HTTP 404 失败;Unsloth 的推理模型会将思考内容放在单独的 reasoning_content 字段中,而不是内联的 <think> 标签里,因此 Max tokens 太小会导致模型把全部配额都用在思考上,返回空白回复——如果回复为空,请调高 Max tokens(建议 2048 以上);Generation 部分 — Thinking mode 开关(默认关闭;Ollama会收到think标志,无论使用哪个后端,关闭时都会从输出中去除<think>...</think>思考块,避免推理模型的思考过程混入Translation/Chat/TOOLS的结果)以及 Max tokens(Ollama: num_predict,LM Studio/Lemonade/Unsloth: max_tokens;0为不限);配置 Free 语言名称;Chat 图像生成部分可让 Chat 触发的文生图使用专用工作流,而不是 GenerateUI 中当前加载的工作流;Chat I2I 生成部分提供相同设置,用于附加图像时的图生图生成;设置保存至 localStorage,并与 Library 面板的 A 标签共享",
helpAi7: "Chat 面板 — SVG:LLM 还可以直接以文本形式生成 SVG 图形,完全不涉及 ComfyUI 工作流;只需要求它用 SVG 绘制内容,如果回复中包含 SVG 代码,会自动渲染为内嵌预览,并提供\"保存到图库\"按钮,将其保存为 .svg 文件到 Gallery 标签页的输出文件夹;该功能不使用 tool calling(函数调用),因此不支持 tool calling 的模型也可以使用",
helpAi8: "Chat 面板 — 技能:在消息列表上方的下拉菜单中选择一个技能,其内容会作为 system prompt 添加到对话开头(不会显示在可见的聊天记录中);点击铅笔按钮打开技能管理面板,可创建、编辑、删除 .md 技能文件(保存路径:user/default/Workflow-Studio/ai_skills/);文件开头 --- 之间的 name/description 字段会显示在下拉菜单中;默认内置两个技能:「SVG Icon Generator」(在绘制前先将对象拆解为多个部件,以生成更易识别的 SVG)和「Skill Creator」(通过对话向你提问并起草新的技能文件);当回复中包含```skill代码块时,消息下方会出现「保存为技能」按钮,点击后会打开管理面板并预填内容",
helpAi9: "编写技能的小贴士:为 frontmatter 的 name/description 写清楚的内容,便于在下拉菜单和提示中阅读;明确说明期望的输出格式,如果有需要后续提取的成果物(如 SVG 或另一个技能文件),指示它用专门的代码围栏包裹;具体、祈使句式的指令比笼统的说明对本地小模型更有效,附上一个简短示例效果更佳;明确写出硬性约束(禁止内容、安全性、长度限制等),不要指望模型会自行推断;如果不确定怎么写,可以在下拉菜单中选择「Skill Creator」技能并描述你想要的技能,它会通过提问帮你起草",
+1
View File
@@ -102,6 +102,7 @@ export const AI_BACKEND_DEFAULT_URLS = {
ollama: "http://localhost:11434",
lmstudio: "http://localhost:1234",
lemonade: "http://localhost:13305",
unsloth: "http://localhost:8888",
};
export function getAiBackendDefaultUrl(backend) {
+6 -3
View File
@@ -1756,7 +1756,7 @@
<div class="wfm-help-card">
<h3 id="wfm-help-sidepanel-card-ai-title">AI TOOL tab</h3>
<ul>
<li id="wfm-help-sidepanel-16">AI TOOL tab: Translation, Chat, TOOLS, and Settings sub-tabs powered by Ollama, LM Studio, or Lemonade; Chat supports multi-turn conversations; TOOLS includes VLM image analysis and wildcard generation; settings shared with the SPA AI TOOL tab via localStorage</li>
<li id="wfm-help-sidepanel-16">AI TOOL tab: Translation, Chat, TOOLS, and Settings sub-tabs powered by Ollama, LM Studio, Lemonade, or Unsloth; Chat supports multi-turn conversations; TOOLS includes VLM image analysis and wildcard generation; settings shared with the SPA AI TOOL tab via localStorage</li>
</ul>
</div>
<div class="wfm-help-card">
@@ -1779,7 +1779,7 @@
</div>
<div class="wfm-help-card">
<h3 id="wfm-help-ai-card-translate-title">Translation pane</h3>
<p id="wfm-help-ai-2">Translation pane: translate text between Japanese, English, Chinese, or a custom Free language using Ollama, LM Studio, or Lemonade; ⇄ swap button swaps both language selectors and text content; selections are saved automatically</p>
<p id="wfm-help-ai-2">Translation pane: translate text between Japanese, English, Chinese, or a custom Free language using Ollama, LM Studio, Lemonade, or Unsloth; ⇄ swap button swaps both language selectors and text content; selections are saved automatically</p>
</div>
<div class="wfm-help-card">
<h3 id="wfm-help-ai-card-chat-title">Chat pane</h3>
@@ -1803,7 +1803,7 @@
</div>
<div class="wfm-help-card">
<h3 id="wfm-help-ai-card-settings-title">Settings pane</h3>
<p id="wfm-help-ai-6">Settings pane: choose backend (Ollama / LM Studio / Lemonade), set API URL, test connection, select model; Generation section — Thinking mode toggle (off by default; Ollama receives a `think` flag, and regardless of backend any &lt;think&gt;...&lt;/think&gt; block is stripped from the output when off, so reasoning models don't clutter Translation/Chat/TOOLS results) and Max tokens (Ollama: num_predict, LM Studio/Lemonade: max_tokens; 0 = unlimited); configure Free language names; Chat Image Generation section lets you enable a dedicated workflow for Chat-triggered text-to-image generation instead of the one currently loaded in GenerateUI; Chat I2I Generation section is the same but for image-to-image generation when an image is attached; settings are shared with the Library panel's A tab via localStorage</p>
<p id="wfm-help-ai-6">Settings pane: choose backend (Ollama / LM Studio / Lemonade / Unsloth), set API URL, test connection, select model; Unsloth always requires an API key even for local access — set UNSLOTH_API_KEY in a .env file in the plugin folder (copy .env.example) and restart ComfyUI, the key is read server-side and never sent to the browser; Unsloth also requires "Auto model switching (OpenAI API)" to be turned ON in the Unsloth Desktop app itself (its own Settings screen) — without it, requests fail with HTTP 404 unless a model was already manually loaded in the Unsloth Studio UI; Unsloth reasoning models return their thinking in a separate reasoning_content field rather than inline &lt;think&gt; tags, so a low Max tokens can make the model spend its whole budget reasoning and return an empty reply — increase Max tokens (2048+) if replies come back blank; Generation section — Thinking mode toggle (off by default; Ollama receives a `think` flag, and regardless of backend any &lt;think&gt;...&lt;/think&gt; block is stripped from the output when off, so reasoning models don't clutter Translation/Chat/TOOLS results) and Max tokens (Ollama: num_predict, LM Studio/Lemonade/Unsloth: max_tokens; 0 = unlimited); configure Free language names; Chat Image Generation section lets you enable a dedicated workflow for Chat-triggered text-to-image generation instead of the one currently loaded in GenerateUI; Chat I2I Generation section is the same but for image-to-image generation when an image is attached; settings are shared with the Library panel's A tab via localStorage</p>
</div>
</div>
@@ -2811,6 +2811,9 @@
<label class="wfm-ai-radio-label">
<input type="radio" name="wfm-ai-backend" value="lemonade"> Lemonade
</label>
<label class="wfm-ai-radio-label">
<input type="radio" name="wfm-ai-backend" value="unsloth"> Unsloth
</label>
</div>
</div>
<div class="wfm-ai-settings-section">
+53
View File
@@ -3622,11 +3622,38 @@ const AI_BACKEND_DEFAULT_URLS = {
ollama: "http://localhost:11434",
lmstudio: "http://localhost:1234",
lemonade: "http://localhost:13305",
unsloth: "http://localhost:8888",
};
function getAiBackendDefaultUrl(backend) {
return AI_BACKEND_DEFAULT_URLS[backend] || AI_BACKEND_DEFAULT_URLS.ollama;
}
// Server relays the request, attaching the Unsloth API key from .env so it
// never reaches the frontend — see py/routes/unsloth_routes.py.
async function aiUnslothProxy(baseUrl, path, method, payload) {
const r = await fetch("/api/wfm/unsloth/proxy", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ baseUrl, path, method, payload }),
});
const data = await r.json().catch(() => ({}));
if (!r.ok) throw new Error(data.message || `HTTP ${r.status}`);
return data;
}
// Unlike Ollama/LM Studio/Lemonade (which inline reasoning as <think> tags
// inside content), Unsloth's OpenAI-compatible API returns it in a separate
// `reasoning_content` field. Fold it back into a <think> block so the
// existing Thinking-mode show/strip logic (_aiApplyGenOptions/aiStripThinkingTags)
// still applies uniformly. Without this, a low max_tokens can make the model
// spend its whole budget reasoning and return an empty `content` with no
// visible explanation.
function _aiUnslothContent(message) {
const reasoning = message?.reasoning_content;
const content = message?.content || "";
return reasoning ? `<think>${reasoning}</think>${content}` : content;
}
function isValidAiUrl(url) {
try {
const u = new URL(url);
@@ -3676,6 +3703,11 @@ async function aiCallLLM(url, backend, model, prompt, settings = {}) {
if (!r.ok) throw new Error(`HTTP ${r.status}`);
const text = (await r.json()).response || "";
return settings.thinkingMode ? text : aiStripThinkingTags(text);
} else if (backend === "unsloth") {
const body = _aiApplyGenOptions({ model, messages: [{ role: "user", content: prompt }], stream: false }, backend, settings);
const d = await aiUnslothProxy(url, "/v1/chat/completions", "POST", body);
const text = _aiUnslothContent(d.choices?.[0]?.message);
return settings.thinkingMode ? text : aiStripThinkingTags(text);
} else {
const r = await fetch(`${url}/v1/chat/completions`, {
method: "POST", headers: { "Content-Type": "application/json" },
@@ -3709,6 +3741,18 @@ async function aiCallVLM(url, backend, model, prompt, base64Image, mimeType, set
if (!r.ok) throw new Error(`HTTP ${r.status}`);
const text = (await r.json()).response || "";
return settings.thinkingMode ? text : aiStripThinkingTags(text);
} else if (backend === "unsloth") {
const body = _aiApplyGenOptions({
model,
messages: [{ role: "user", content: [
{ type: "text", text: prompt },
{ type: "image_url", image_url: { url: `data:${mimeType};base64,${base64Image}` } },
]}],
stream: false,
}, backend, settings);
const d = await aiUnslothProxy(url, "/v1/chat/completions", "POST", body);
const text = _aiUnslothContent(d.choices?.[0]?.message);
return settings.thinkingMode ? text : aiStripThinkingTags(text);
} else {
const r = await fetch(`${url}/v1/chat/completions`, {
method: "POST", headers: { "Content-Type": "application/json" },
@@ -3736,6 +3780,11 @@ async function aiCallChat(url, backend, model, messages, settings = {}) {
if (!r.ok) throw new Error(`HTTP ${r.status}`);
const text = (await r.json()).message?.content || "";
return settings.thinkingMode ? text : aiStripThinkingTags(text);
} else if (backend === "unsloth") {
const body = _aiApplyGenOptions({ model, messages, stream: false }, backend, settings);
const d = await aiUnslothProxy(url, "/v1/chat/completions", "POST", body);
const text = _aiUnslothContent(d.choices?.[0]?.message);
return settings.thinkingMode ? text : aiStripThinkingTags(text);
} else {
const r = await fetch(`${url}/v1/chat/completions`, {
method: "POST", headers: { "Content-Type": "application/json" },
@@ -3773,6 +3822,9 @@ async function aiFetchModels(url, backend) {
const r = await fetch(`${url}/api/tags`);
if (!r.ok) throw new Error(`HTTP ${r.status}`);
return ((await r.json()).models || []).map(m => m.name);
} else if (backend === "unsloth") {
const d = await aiUnslothProxy(url, "/v1/models", "GET");
return (d.data || []).map(m => m.id);
} else {
const r = await fetch(`${url}/v1/models`);
if (!r.ok) throw new Error(`HTTP ${r.status}`);
@@ -3924,6 +3976,7 @@ const renderAiTab = (container) => {
<label class="wfm-nlp-ai-radio"><input type="radio" name="wfm-nlp-ai-backend" value="ollama"> Ollama</label>
<label class="wfm-nlp-ai-radio"><input type="radio" name="wfm-nlp-ai-backend" value="lmstudio"> LM Studio</label>
<label class="wfm-nlp-ai-radio"><input type="radio" name="wfm-nlp-ai-backend" value="lemonade"> Lemonade</label>
<label class="wfm-nlp-ai-radio"><input type="radio" name="wfm-nlp-ai-backend" value="unsloth"> Unsloth</label>
</div>
</div>
<div class="wfm-nlp-ai-sec">