Initial commit of ComfyUI-SimpleChat

This commit is contained in:
Nakano Kenji
2025-12-09 23:39:00 +08:00
commit 364a6fc705
44 changed files with 2429 additions and 0 deletions
+98
View File
@@ -0,0 +1,98 @@
# ComfyUI-SimpleChat
Simple, no-nonsense LLM chat nodes for ComfyUI. Connect to OpenAI, Claude, Gemini, and local LLMs with ease.
[English](README.md) | [中文](README_CN.md)
## Features
- **All-in-One Configuration**: Single, powerful config node for all providers.
- **Dynamic Model Fetching**: Click one button to fetch the *actual* list of models available to your API key. No hardcoded lists!
- **Universal Compatibility**: Works with official APIs and any OpenAI-compatible proxy/local endpoint.
- **Multimodal Support**: Chat with images using vision-capable models.
- **NoASS Roleplay Mode (Exp)**: Experimental Hardcore roleplay node with Assistant Prefill support.
- **Gemini Native Features**: Dedicated nodes for Gemini's image generation and editing capabilities.
## Installation
1. Clone this repository into your `ComfyUI/custom_nodes` folder:
```bash
git clone https://github.com/yourname/ComfyUI-SimpleChat.git
```
2. Restart ComfyUI.
## Usage Guide
### 1. API Configuration (The Core)
Everything starts with the **API Config** node.
1. **Add Node**: Right-click -> `SimpleChat` -> `API Config`.
2. **Select Provider**: Choose `openai`, `claude`, or `gemini`.
3. **Enter API Key**: Paste your secret key.
* *OpenAI*: `sk-...`
* *Claude*: `sk-ant-...`
* *Gemini*: Your Google AI Studio key.
4. **Base URL (Optional)**:
* Leave empty for official APIs.
* Enter a custom URL if you use a proxy, a local LLM (like LM Studio/Ollama), or a third-party service.
* *Example*: `https://api.deepseek.com/v1` or `http://localhost:1234/v1`.
5. **Select Model**:
* **Click the "Refresh Models" button**.
* The node will connect to the API and fetch the *real-time list* of models you can use.
* Select your desired model from the dropdown.
### 2. Basic Chat
Use this for standard text generation or Q&A.
* **Node**: `Chat`
* **Connection**: Connect `config` from **API Config** to `config` on **Chat**.
* **Input**: Enter your prompt in `text`.
* **Output**: Returns the AI's response as a string.
### 3. Chat with Images
Use this to have the AI analyze or describe an image.
* **Node**: `Chat with Image`
* **Connection**:
* Connect `config` from **API Config**.
* Connect an image (from `Load Image` node) to the `image` input.
* **Input**: Enter a prompt (e.g., "Describe this image").
### 4. NoASS Chat (Experimental)
The "No Assistant" (NoASS) node is an experimental feature designed for **hardcore roleplay and creative writing**. It forces the AI to break character restrictions by pre-filling its response start.
* **Node**: `Chat NoASS (Exp)`
* **Inputs**:
* **Scenario Instructions**: (User Message) Define the world, characters, and rules. This sets the stage.
* **User Action**: Your current turn or dialogue.
* **Prefill Start** (Optional): Force the first few words of the AI's response. The AI *must* continue from here. Great for steering the tone.
* **History**: Connect output `history` to the next node's input `history` to maintain conversation context.
### 5. Gemini Image Generation
Generate images using Google's Gemini models.
* **Node**: `Gemini Image Gen`
* **Requirement**: Must use `gemini` provider in **API Config**.
* **Output**: Generates a standard ComfyUI IMAGE.
### 6. Gemini Image Editing
Modify existing images using text instructions.
* **Node**: `Gemini Image Edit`
* **Input**: Source image + Mask image + Prompt.
## FAQ
**Q: I don't see my model in the list.**
A: Make sure you entered your API Key and Base URL correctly, then click **Refresh Models**. If the API supports listing models, they will appear. If not, you can try typing the model name directly (if the UI allows) or checking your provider's documentation.
**Q: Can I use local models?**
A: Yes! Select `openai` as the provider, set the **Base URL** to your local server address (e.g., `http://127.0.0.1:1234/v1`), and click Refresh.
**Q: My dropdown is empty.**
A: Check your internet connection and API Key. If the API Key is invalid, the list will not populate.
## License
MIT
BIN
View File
Binary file not shown.
+47
View File
@@ -0,0 +1,47 @@
"""
ComfyUI-SimpleChat: Simple LLM chat nodes for ComfyUI.
Supports OpenAI, Claude, and Gemini APIs with custom base URL support.
Features NoASS mode for roleplay optimization and Gemini image generation.
"""
from .nodes import (
SimpleChatConfig,
SimpleChatText,
SimpleChatImage,
SimpleChatNoASS,
GeminiImageGen,
GeminiImageEdit,
)
# Register API routes
from .api import setup_routes
setup_routes()
# Node class mappings for ComfyUI
NODE_CLASS_MAPPINGS = {
"SimpleChatConfig": SimpleChatConfig,
"SimpleChatText": SimpleChatText,
"SimpleChatImage": SimpleChatImage,
"SimpleChatNoASS": SimpleChatNoASS,
"GeminiImageGen": GeminiImageGen,
"GeminiImageEdit": GeminiImageEdit,
}
# Display names for the UI
NODE_DISPLAY_NAME_MAPPINGS = {
"SimpleChatConfig": "API Config",
"SimpleChatText": "Chat",
"SimpleChatImage": "Chat with Image",
"SimpleChatNoASS": "Chat NoASS",
"GeminiImageGen": "Gemini Image Gen",
"GeminiImageEdit": "Gemini Image Edit",
}
# Expose web directory for JS extensions
WEB_DIRECTORY = "./web"
__all__ = [
"NODE_CLASS_MAPPINGS",
"NODE_DISPLAY_NAME_MAPPINGS",
"WEB_DIRECTORY",
]
Binary file not shown.
+6
View File
@@ -0,0 +1,6 @@
"""
API routes for ComfyUI-SimpleChat.
"""
from .routes import setup_routes
__all__ = ["setup_routes"]
Binary file not shown.
Binary file not shown.
+201
View File
@@ -0,0 +1,201 @@
"""
API routes for fetching model lists from various providers.
"""
import json
import aiohttp
from aiohttp import web
try:
from server import PromptServer
HAS_SERVER = True
except ImportError:
HAS_SERVER = False
# Predefined model lists for each provider
PREDEFINED_MODELS = {
"openai": [
"gpt-4o",
"gpt-4o-mini",
"gpt-4-turbo",
"gpt-4",
"gpt-3.5-turbo",
"o1-preview",
"o1-mini",
],
"claude": [
"claude-sonnet-4-20250514",
"claude-3-5-sonnet-20241022",
"claude-3-5-haiku-20241022",
"claude-3-opus-20240229",
"claude-3-sonnet-20240229",
"claude-3-haiku-20240307",
],
"gemini": [
"gemini-2.5-flash-preview-05-20",
"gemini-2.0-flash",
"gemini-2.0-flash-exp",
"gemini-1.5-pro",
"gemini-1.5-flash",
"gemini-2.5-flash-image",
"gemini-3-pro-image-preview",
],
}
async def fetch_openai_models(api_key: str, base_url: str) -> list[str]:
"""Fetch model list from OpenAI-compatible API."""
url = f"{base_url.rstrip('/')}/models"
headers = {"Authorization": f"Bearer {api_key}"}
try:
async with aiohttp.ClientSession() as session:
async with session.get(url, headers=headers, timeout=10) as resp:
if resp.status == 200:
data = await resp.json()
models = [m["id"] for m in data.get("data", [])]
# Return all models, sorted alphabetically
return sorted(models) if models else PREDEFINED_MODELS["openai"]
except Exception as e:
print(f"[SimpleChat] Failed to fetch OpenAI models: {e}")
return PREDEFINED_MODELS["openai"]
async def fetch_claude_models(api_key: str, base_url: str) -> list[str]:
"""
Fetch models for Claude provider.
Tries to hit the /models endpoint (supported by some proxies and newer Anthropic API).
Falls back to predefined list if fails.
"""
# Anthropic API format for models list is /v1/models
# Note: Official Anthropic API recently added models endpoint support
url = f"{base_url.rstrip('/')}/models"
headers = {
"x-api-key": api_key,
"anthropic-version": "2023-06-01"
}
try:
async with aiohttp.ClientSession() as session:
async with session.get(url, headers=headers, timeout=10) as resp:
if resp.status == 200:
data = await resp.json()
# Anthropic format: {"data": [{"id": "...", ...}]} similar to OpenAI
# Some proxies might return OpenAI format
models = []
# Handle both standard formats just in case
items = data.get("data", []) if "data" in data else data.get("models", [])
if isinstance(items, list):
for m in items:
if isinstance(m, dict):
models.append(m.get("id", m.get("name", "")))
elif isinstance(m, str):
models.append(m)
# Filter out empty strings
models = [m for m in models if m]
if models:
return sorted(models)
except Exception as e:
# Silently fail for Claude since it's common for this endpoint to not exist/fail
pass
return PREDEFINED_MODELS["claude"]
async def fetch_gemini_models(api_key: str, base_url: str) -> list[str]:
"""Fetch model list from Gemini API."""
url = f"{base_url.rstrip('/')}/models?key={api_key}"
try:
async with aiohttp.ClientSession() as session:
async with session.get(url, timeout=10) as resp:
if resp.status == 200:
data = await resp.json()
models = []
for m in data.get("models", []):
name = m.get("name", "")
# Extract model name from full path (models/gemini-xxx)
if name.startswith("models/"):
name = name[7:]
if name:
models.append(name)
# Return all models
return sorted(models) if models else PREDEFINED_MODELS["gemini"]
except Exception as e:
print(f"[SimpleChat] Failed to fetch Gemini models: {e}")
return PREDEFINED_MODELS["gemini"]
def get_all_predefined_models_list() -> list[str]:
"""Get a flat list of all predefined models."""
all_models = []
for provider_models in PREDEFINED_MODELS.values():
all_models.extend(provider_models)
return all_models
def setup_routes():
"""Setup API routes for SimpleChat."""
if not HAS_SERVER:
return
@PromptServer.instance.routes.get("/simplechat/models/{provider}")
async def get_models(request):
"""
Get available models for a provider.
Query params:
- api_key: API key for the provider
- base_url: (optional) Custom base URL
"""
provider = request.match_info.get("provider", "").lower()
api_key = request.query.get("api_key", "")
base_url = request.query.get("base_url", "")
# Return predefined models if no API key provided
if not api_key:
models = PREDEFINED_MODELS.get(provider, [])
return web.json_response(models)
# Default base URLs
default_urls = {
"openai": "https://api.openai.com/v1",
"claude": "https://api.anthropic.com/v1",
"gemini": "https://generativelanguage.googleapis.com/v1beta",
}
if not base_url:
base_url = default_urls.get(provider, "")
# Fetch models based on provider
if provider == "openai":
models = await fetch_openai_models(api_key, base_url)
elif provider == "claude":
models = await fetch_claude_models(api_key, base_url)
elif provider == "gemini":
models = await fetch_gemini_models(api_key, base_url)
else:
models = []
return web.json_response(models)
@PromptServer.instance.routes.get("/simplechat/models")
async def get_all_models_list(request):
"""
Get all predefined models as a flat list (for COMBO remote input).
Returns a simple array of model names.
"""
return web.json_response(get_all_predefined_models_list())
@PromptServer.instance.routes.get("/simplechat/models_dict")
async def get_all_predefined_models(request):
"""Get all predefined models organized by provider."""
return web.json_response(PREDEFINED_MODELS)
print("[SimpleChat] API routes registered")
+51
View File
@@ -0,0 +1,51 @@
"""
SimpleChat core module.
"""
from .providers import (
BaseProvider,
ChatResponse,
ChatConfig,
OpenAIProvider,
ClaudeProvider,
GeminiProvider,
PROVIDERS,
get_provider,
)
from .image_utils import (
tensor_to_pil,
pil_to_tensor,
tensor_to_base64,
base64_to_tensor,
create_data_uri,
)
from .noass import (
format_noass_prompt,
build_noass_messages,
extract_noass_response,
build_full_history,
get_stop_sequences,
)
__all__ = [
# Providers
"BaseProvider",
"ChatResponse",
"ChatConfig",
"OpenAIProvider",
"ClaudeProvider",
"GeminiProvider",
"PROVIDERS",
"get_provider",
# Image utils
"tensor_to_pil",
"pil_to_tensor",
"tensor_to_base64",
"base64_to_tensor",
"create_data_uri",
# NoASS
"format_noass_prompt",
"build_noass_messages",
"extract_noass_response",
"build_full_history",
"get_stop_sequences",
]
Binary file not shown.
Binary file not shown.
Binary file not shown.
+122
View File
@@ -0,0 +1,122 @@
"""
Image conversion utilities for ComfyUI tensors.
"""
import base64
from io import BytesIO
import torch
import numpy as np
from PIL import Image
def tensor_to_pil(tensor: torch.Tensor) -> Image.Image:
"""
Convert ComfyUI tensor to PIL Image.
Args:
tensor: Shape (B, H, W, C) or (H, W, C), values 0-1
Returns:
PIL Image (first image if batched)
"""
if tensor.dim() == 4:
tensor = tensor[0] # Take first image from batch
# Convert to numpy and scale to 0-255
arr = (tensor.cpu().numpy() * 255).astype(np.uint8)
# Handle different channel counts
if arr.shape[-1] == 1:
arr = arr.squeeze(-1)
return Image.fromarray(arr, mode='L')
elif arr.shape[-1] == 3:
return Image.fromarray(arr, mode='RGB')
elif arr.shape[-1] == 4:
return Image.fromarray(arr, mode='RGBA')
else:
raise ValueError(f"Unsupported channel count: {arr.shape[-1]}")
def pil_to_tensor(image: Image.Image) -> torch.Tensor:
"""
Convert PIL Image to ComfyUI tensor.
Args:
image: PIL Image
Returns:
Tensor shape (1, H, W, C), values 0-1
"""
# Convert to RGB if needed
if image.mode == 'RGBA':
pass # Keep alpha
elif image.mode != 'RGB':
image = image.convert('RGB')
# Convert to tensor
arr = np.array(image).astype(np.float32) / 255.0
tensor = torch.from_numpy(arr)
# Add batch dimension
if tensor.dim() == 2: # Grayscale
tensor = tensor.unsqueeze(-1)
tensor = tensor.unsqueeze(0)
return tensor
def tensor_to_base64(tensor: torch.Tensor, format: str = "PNG") -> str:
"""
Convert ComfyUI tensor to base64 string.
Args:
tensor: ComfyUI image tensor
format: Image format (PNG, JPEG, WEBP)
Returns:
Base64 encoded string
"""
pil_image = tensor_to_pil(tensor)
# Convert RGBA to RGB for JPEG
if format.upper() == "JPEG" and pil_image.mode == "RGBA":
pil_image = pil_image.convert("RGB")
buffer = BytesIO()
pil_image.save(buffer, format=format)
return base64.b64encode(buffer.getvalue()).decode('utf-8')
def base64_to_tensor(b64_string: str) -> torch.Tensor:
"""
Convert base64 string to ComfyUI tensor.
Args:
b64_string: Base64 encoded image
Returns:
ComfyUI image tensor (1, H, W, C)
"""
image_data = base64.b64decode(b64_string)
image = Image.open(BytesIO(image_data))
return pil_to_tensor(image)
def create_data_uri(tensor: torch.Tensor, mime_type: str = "image/png") -> str:
"""
Create a data URI from tensor for API requests.
Args:
tensor: ComfyUI image tensor
mime_type: MIME type (image/png, image/jpeg, image/webp)
Returns:
Data URI string
"""
format_map = {
"image/png": "PNG",
"image/jpeg": "JPEG",
"image/webp": "WEBP",
}
format = format_map.get(mime_type, "PNG")
b64 = tensor_to_base64(tensor, format)
return f"data:{mime_type};base64,{b64}"
+174
View File
@@ -0,0 +1,174 @@
"""
NoASS (No Assistant) format processing for roleplay optimization.
NoASS transforms traditional chat into a unified narrative where all dialogue
is combined into a single assistant message, with character prefixes marking speakers.
"""
def format_noass_prompt(
system: str,
history: str | None,
user_input: str,
prefill_start: str = "",
user_name: str = "User",
char_name: str = "Assistant",
) -> tuple[str, str]:
"""
Format conversation in NoASS style.
Args:
system: System prompt / world setting
history: Previous conversation history (NoASS format)
user_input: Current user input
prefill_start: Optional starting text for assistant response
user_name: User's character name
char_name: AI character name
Returns:
Tuple of (system_prompt, prefilled_content)
- system_prompt: Instructions for the model
- prefilled_content: The conversation to continue from
"""
user_prefix = f"**{user_name}:**"
char_prefix = f"**{char_name}:**"
# Build the conversation history structure
content_parts = []
# 1. Add existing history if any
if history:
content_parts.append(history)
# 2. Add current user input
content_parts.append(f"{user_prefix} {user_input}")
# 3. Add character prefix to prompt response
if prefill_start.strip():
# If we have a forced start, include it
content_parts.append(f"{char_prefix} {prefill_start}")
else:
# Otherwise just the prefix
content_parts.append(f"{char_prefix}")
prefilled = "\n\n".join(content_parts)
return system, prefilled
def build_noass_messages(
system: str,
history: str | None,
user_input: str,
prefill_start: str = "",
user_name: str = "User",
char_name: str = "Assistant",
) -> list[dict[str, str]]:
"""
Build message list for NoASS format using proper Assistant Prefill where possible.
Structure:
[
{"role": "user", "content": system}, # System/Scenario as first user message
{"role": "assistant", "content": [History] + [User Input] + [Prefill]} # The entire story so far
]
The API is expected to continue directly from the end of the assistant message.
"""
messages = []
# 1. System/Scenario (as User message to ensure it's read as instructions)
# Using 'system' role is also fine but 'user' often works better for "Instruction" in NoASS context
# However, standard practice is still system role for actual instructions.
# Let's stick to 'user' for the "Scenario Description" as requested by the user's mental model.
if system:
messages.append({"role": "user", "content": system})
else:
# Fallback if system is empty
messages.append({"role": "user", "content": "Narrative Roleplay."})
# 2. Build the giant Assistant Prefill block
user_prefix = f"**{user_name}:**"
char_prefix = f"**{char_name}:**"
story_blocks = []
if history:
story_blocks.append(history)
# Append current user action
story_blocks.append(f"{user_prefix} {user_input}")
# Append the start of the assistant's turn
if prefill_start:
story_blocks.append(f"{char_prefix} {prefill_start}")
else:
# Just the prefix to cue the AI
story_blocks.append(f"{char_prefix}")
full_prefill = "\n\n".join(story_blocks)
messages.append({
"role": "assistant",
"content": full_prefill
})
return messages
def extract_noass_response(
response_text: str,
user_name: str = "User",
char_name: str = "Assistant",
) -> str:
"""
Extract the character's response from NoASS output.
Since we are using prefill, the API might return JUST the continuation,
or (depending on provider behavior) might repeat some text.
Standard OpenAI/Claude behavior for prefill is to return just the NEW tokens.
"""
user_prefix = f"**{user_name}:**"
# If response contains user prefix (hallucinated next turn), truncate there
if user_prefix in response_text:
response_text = response_text.split(user_prefix)[0]
return response_text.strip()
def build_full_history(
history: str | None,
user_input: str,
prefill_start: str,
response: str,
user_name: str = "User",
char_name: str = "Assistant",
) -> str:
"""
Build complete conversation history after a response.
Reconstructs the full text including the prefill and the new response.
"""
user_prefix = f"**{user_name}:**"
char_prefix = f"**{char_name}:**"
# Construct the assistant's full turn
if prefill_start:
full_response = f"{char_prefix} {prefill_start} {response}"
else:
full_response = f"{char_prefix} {response}"
# Append to history
if history:
return f"{history}\n\n{user_prefix} {user_input}\n\n{full_response}"
else:
return f"{user_prefix} {user_input}\n\n{full_response}"
def get_stop_sequences(user_name: str = "User") -> list[str]:
"""
Get stop sequences for NoASS mode.
The model should stop when it tries to write for the user.
"""
user_prefix = f"**{user_name}:**"
return [user_prefix, f"\n{user_prefix}", f"\n\n{user_prefix}"]
+34
View File
@@ -0,0 +1,34 @@
"""
Provider implementations for SimpleChat.
"""
from .base import BaseProvider, ChatResponse, ChatConfig
from .openai import OpenAIProvider
from .claude import ClaudeProvider
from .gemini import GeminiProvider
PROVIDERS = {
"openai": OpenAIProvider,
"claude": ClaudeProvider,
"gemini": GeminiProvider,
}
def get_provider(config: ChatConfig) -> BaseProvider:
"""Get provider instance from config."""
provider_cls = PROVIDERS.get(config.provider)
if not provider_cls:
raise ValueError(f"Unknown provider: {config.provider}")
return provider_cls(api_key=config.api_key, base_url=config.base_url or None)
__all__ = [
"BaseProvider",
"ChatResponse",
"ChatConfig",
"OpenAIProvider",
"ClaudeProvider",
"GeminiProvider",
"PROVIDERS",
"get_provider",
]
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+72
View File
@@ -0,0 +1,72 @@
"""
Base provider class for LLM API integrations.
"""
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import Any
import torch
@dataclass
class ChatResponse:
"""Unified response from any LLM provider."""
text: str
image: torch.Tensor | None = None
raw_response: dict | None = None
@dataclass
class ChatConfig:
"""Configuration for API connection."""
provider: str
api_key: str
base_url: str
model: str
def to_dict(self) -> dict:
return {
"provider": self.provider,
"api_key": self.api_key,
"base_url": self.base_url,
"model": self.model,
}
class BaseProvider(ABC):
"""Abstract base class for LLM providers."""
def __init__(self, api_key: str, base_url: str | None = None):
self.api_key = api_key
self.base_url = base_url or self.default_base_url
@property
@abstractmethod
def default_base_url(self) -> str:
"""Default API endpoint for this provider."""
pass
@abstractmethod
async def chat(
self,
messages: list[dict[str, Any]],
model: str,
temperature: float = 1.0,
max_tokens: int = 2048,
images: list[torch.Tensor] | None = None,
**kwargs,
) -> ChatResponse:
"""Send a chat request to the API."""
pass
@abstractmethod
async def generate_image(
self,
prompt: str,
model: str,
reference_image: torch.Tensor | None = None,
aspect_ratio: str = "1:1",
size: str = "1K",
**kwargs,
) -> ChatResponse:
"""Generate or edit an image. Only supported by some providers."""
pass
+121
View File
@@ -0,0 +1,121 @@
"""
Claude/Anthropic provider implementation.
"""
import aiohttp
from typing import Any
import torch
from .base import BaseProvider, ChatResponse
from ..image_utils import tensor_to_base64, create_data_uri
class ClaudeProvider(BaseProvider):
"""Anthropic Claude API provider."""
@property
def default_base_url(self) -> str:
return "https://api.anthropic.com/v1"
def _build_messages(
self,
messages: list[dict[str, Any]],
images: list[torch.Tensor] | None = None,
) -> tuple[str | None, list[dict[str, Any]]]:
"""
Build Claude-format messages with optional images.
Returns:
Tuple of (system_prompt, messages)
"""
system = None
result = []
for i, msg in enumerate(messages):
role = msg["role"]
content = msg["content"]
# Extract system message
if role == "system":
system = content
continue
# Handle user message with images
if role == "user" and images and i == len(messages) - 1:
content_parts = []
for img in images:
b64 = tensor_to_base64(img)
content_parts.append({
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": b64,
}
})
content_parts.append({"type": "text", "text": content})
result.append({"role": "user", "content": content_parts})
else:
result.append({"role": role, "content": content})
return system, result
async def chat(
self,
messages: list[dict[str, Any]],
model: str,
temperature: float = 1.0,
max_tokens: int = 2048,
images: list[torch.Tensor] | None = None,
**kwargs,
) -> ChatResponse:
"""Send chat request to Claude API."""
url = f"{self.base_url}/messages"
headers = {
"x-api-key": self.api_key,
"anthropic-version": "2023-06-01",
"Content-Type": "application/json",
}
system, claude_messages = self._build_messages(messages, images)
payload = {
"model": model,
"messages": claude_messages,
"temperature": temperature,
"max_tokens": max_tokens,
}
if system:
payload["system"] = system
async with aiohttp.ClientSession() as session:
async with session.post(url, headers=headers, json=payload) as resp:
if resp.status != 200:
error_text = await resp.text()
raise RuntimeError(f"Claude API error {resp.status}: {error_text}")
data = await resp.json()
# Extract text from response
text = ""
for block in data.get("content", []):
if block.get("type") == "text":
text += block.get("text", "")
return ChatResponse(text=text, raw_response=data)
async def generate_image(
self,
prompt: str,
model: str,
reference_image: torch.Tensor | None = None,
aspect_ratio: str = "1:1",
size: str = "1K",
**kwargs,
) -> ChatResponse:
"""Claude doesn't support image generation."""
raise NotImplementedError(
"Claude does not support image generation. "
"Use Gemini provider for image generation."
)
+199
View File
@@ -0,0 +1,199 @@
"""
Google Gemini provider implementation.
Supports chat and image generation (Nano Banana / Nano Banana Pro).
"""
import aiohttp
from typing import Any
import torch
from .base import BaseProvider, ChatResponse
from ..image_utils import tensor_to_base64, base64_to_tensor
class GeminiProvider(BaseProvider):
"""Google Gemini API provider with image generation support."""
@property
def default_base_url(self) -> str:
return "https://generativelanguage.googleapis.com/v1beta"
def _build_contents(
self,
messages: list[dict[str, Any]],
images: list[torch.Tensor] | None = None,
) -> tuple[str | None, list[dict[str, Any]]]:
"""
Build Gemini-format contents.
Returns:
Tuple of (system_instruction, contents)
"""
system = None
contents = []
for i, msg in enumerate(messages):
role = msg["role"]
content = msg["content"]
# Extract system message
if role == "system":
system = content
continue
# Map roles
gemini_role = "user" if role == "user" else "model"
# Build parts
parts = []
# Add images to last user message
if role == "user" and images and i == len(messages) - 1:
for img in images:
b64 = tensor_to_base64(img)
parts.append({
"inline_data": {
"mime_type": "image/png",
"data": b64,
}
})
parts.append({"text": content})
contents.append({"role": gemini_role, "parts": parts})
return system, contents
async def chat(
self,
messages: list[dict[str, Any]],
model: str,
temperature: float = 1.0,
max_tokens: int = 2048,
images: list[torch.Tensor] | None = None,
enable_image_generation: bool = False,
**kwargs,
) -> ChatResponse:
"""Send chat request to Gemini API."""
url = f"{self.base_url}/models/{model}:generateContent"
# Add API key as query parameter
if "?" in url:
url += f"&key={self.api_key}"
else:
url += f"?key={self.api_key}"
headers = {"Content-Type": "application/json"}
system, contents = self._build_contents(messages, images)
payload = {
"contents": contents,
"generationConfig": {
"temperature": temperature,
"maxOutputTokens": max_tokens,
}
}
if system:
payload["systemInstruction"] = {"parts": [{"text": system}]}
# Enable image generation if requested
if enable_image_generation:
payload["generationConfig"]["responseModalities"] = ["TEXT", "IMAGE"]
async with aiohttp.ClientSession() as session:
async with session.post(url, headers=headers, json=payload) as resp:
if resp.status != 200:
error_text = await resp.text()
raise RuntimeError(f"Gemini API error {resp.status}: {error_text}")
data = await resp.json()
# Extract text and images from response
text = ""
result_image = None
candidates = data.get("candidates", [])
if candidates:
parts = candidates[0].get("content", {}).get("parts", [])
for part in parts:
if "text" in part:
text += part["text"]
elif "inlineData" in part:
inline = part["inlineData"]
if inline.get("mimeType", "").startswith("image/"):
result_image = base64_to_tensor(inline["data"])
return ChatResponse(text=text, image=result_image, raw_response=data)
async def generate_image(
self,
prompt: str,
model: str,
reference_image: torch.Tensor | None = None,
aspect_ratio: str = "1:1",
size: str = "1K",
**kwargs,
) -> ChatResponse:
"""Generate or edit image using Gemini."""
url = f"{self.base_url}/models/{model}:generateContent?key={self.api_key}"
headers = {"Content-Type": "application/json"}
# Build parts
parts = []
# Add reference image if provided
if reference_image is not None:
b64 = tensor_to_base64(reference_image)
parts.append({
"inline_data": {
"mime_type": "image/png",
"data": b64,
}
})
parts.append({"text": prompt})
payload = {
"contents": [{"parts": parts}],
"generationConfig": {
"responseModalities": ["TEXT", "IMAGE"],
}
}
# Add image config if supported
if aspect_ratio or size:
payload["generationConfig"]["imageConfig"] = {}
if aspect_ratio:
payload["generationConfig"]["imageConfig"]["aspectRatio"] = aspect_ratio
if size:
payload["generationConfig"]["imageConfig"]["imageSize"] = size
async with aiohttp.ClientSession() as session:
async with session.post(url, headers=headers, json=payload) as resp:
if resp.status != 200:
error_text = await resp.text()
raise RuntimeError(f"Gemini API error {resp.status}: {error_text}")
data = await resp.json()
# Extract text and image
text = ""
result_image = None
candidates = data.get("candidates", [])
if candidates:
parts = candidates[0].get("content", {}).get("parts", [])
for part in parts:
if "text" in part:
text += part["text"]
elif "inlineData" in part:
inline = part["inlineData"]
if inline.get("mimeType", "").startswith("image/"):
result_image = base64_to_tensor(inline["data"])
if result_image is None:
raise RuntimeError("Gemini did not return an image. Try a different prompt or model.")
return ChatResponse(text=text, image=result_image, raw_response=data)
+94
View File
@@ -0,0 +1,94 @@
"""
OpenAI provider implementation.
Also works with OpenAI-compatible APIs (e.g., local LLMs, other providers).
"""
import aiohttp
from typing import Any
import torch
from .base import BaseProvider, ChatResponse
from ..image_utils import tensor_to_base64, base64_to_tensor, create_data_uri
class OpenAIProvider(BaseProvider):
"""OpenAI API provider (and compatible APIs)."""
@property
def default_base_url(self) -> str:
return "https://api.openai.com/v1"
def _build_messages(
self,
messages: list[dict[str, Any]],
images: list[torch.Tensor] | None = None,
) -> list[dict[str, Any]]:
"""Build OpenAI-format messages with optional images."""
if not images:
return messages
# Find the last user message and add images to it
result = []
for i, msg in enumerate(messages):
if msg["role"] == "user" and i == len(messages) - 1:
# Convert content to multimodal format
content = [{"type": "text", "text": msg["content"]}]
for img in images:
content.append({
"type": "image_url",
"image_url": {"url": create_data_uri(img)}
})
result.append({"role": "user", "content": content})
else:
result.append(msg)
return result
async def chat(
self,
messages: list[dict[str, Any]],
model: str,
temperature: float = 1.0,
max_tokens: int = 2048,
images: list[torch.Tensor] | None = None,
**kwargs,
) -> ChatResponse:
"""Send chat request to OpenAI API."""
url = f"{self.base_url}/chat/completions"
headers = {
"Authorization": f"Bearer {self.api_key}",
"Content-Type": "application/json",
}
payload = {
"model": model,
"messages": self._build_messages(messages, images),
"temperature": temperature,
"max_tokens": max_tokens,
}
async with aiohttp.ClientSession() as session:
async with session.post(url, headers=headers, json=payload) as resp:
if resp.status != 200:
error_text = await resp.text()
raise RuntimeError(f"OpenAI API error {resp.status}: {error_text}")
data = await resp.json()
text = data["choices"][0]["message"]["content"]
return ChatResponse(text=text, raw_response=data)
async def generate_image(
self,
prompt: str,
model: str,
reference_image: torch.Tensor | None = None,
aspect_ratio: str = "1:1",
size: str = "1K",
**kwargs,
) -> ChatResponse:
"""OpenAI doesn't support native image generation in chat. Use DALL-E nodes instead."""
raise NotImplementedError(
"OpenAI image generation is not supported in SimpleChat. "
"Use dedicated DALL-E nodes or Gemini provider for image generation."
)
+312
View File
@@ -0,0 +1,312 @@
# ComfyUI-SimpleChat Architecture Design
## Design Philosophy
**Do one thing, do best** - 每个节点只做一件事,做到极致。
## 节点设计原则
1. **单一职责**: 每个节点只负责一个功能
2. **清晰命名**: 节点名称直接表达功能
3. **最少配置**: 合理默认值,减少必填项
4. **美观布局**: 输入输出顺序符合直觉
---
## 节点清单
### 配置节点
| 节点 | 功能 | 设计理念 |
|------|------|----------|
| `API Config` | 配置 API 连接 | 一次配置,到处使用 |
### 对话节点
| 节点 | 功能 | 设计理念 |
|------|------|----------|
| `Chat` | 基础对话 | 最简单的文本对话 |
| `Chat with Image` | 发送图片给 LLM | 视觉理解 |
| `Chat NoASS` | NoASS 格式对话 | 角色扮演优化 |
### 图片生成节点 (Gemini Only)
| 节点 | 功能 | 设计理念 |
|------|------|----------|
| `Gemini Image Gen` | 文生图 | 纯文本生成图片 |
| `Gemini Image Edit` | 图片编辑 | 基于参考图修改 |
---
## 节点详细设计
### 1. API Config
**功能**: 配置 API 连接参数
```
┌─────────────────────────────────────┐
│ 🔧 API Config │
├─────────────────────────────────────┤
│ provider [openai ▼] │
│ api_key [••••••••••] │
│ base_url [ ] │
│ model [gpt-4o ▼] │
├─────────────────────────────────────┤
│ CONFIG ○───│
└─────────────────────────────────────┘
```
**输入:**
- `provider`: 下拉 ["openai", "claude", "gemini"]
- `api_key`: 密码输入
- `base_url`: 文本 (可选,留空使用默认)
- `model`: 可编辑下拉
**输出:**
- `CONFIG`: 配置对象
---
### 2. Chat
**功能**: 基础文本对话
```
┌─────────────────────────────────────┐
│ 💬 Chat │
├─────────────────────────────────────┤
│ ───○ config │
├─────────────────────────────────────┤
│ prompt [ ] │
│ [ ] │
│ system [ ] │
│ temperature [1.0 ] │
│ max_tokens [2048 ] │
├─────────────────────────────────────┤
│ text ○───│
└─────────────────────────────────────┘
```
**输入:**
- `config`: CONFIG (必需)
- `prompt`: 多行文本
- `system`: 系统提示 (可选)
- `temperature`: 0-2, 默认 1.0
- `max_tokens`: 默认 2048
**输出:**
- `text`: STRING
---
### 3. Chat with Image
**功能**: 发送图片给 LLM 分析
```
┌─────────────────────────────────────┐
│ 🖼️ Chat with Image │
├─────────────────────────────────────┤
│ ───○ config │
│ ───○ image │
├─────────────────────────────────────┤
│ prompt [Describe this image ] │
│ [ ] │
│ system [ ] │
│ temperature [1.0 ] │
│ max_tokens [2048 ] │
├─────────────────────────────────────┤
│ text ○───│
└─────────────────────────────────────┘
```
**输入:**
- `config`: CONFIG
- `image`: IMAGE
- `prompt`: 多行文本
- `system`: 系统提示 (可选)
- `temperature`: 0-2
- `max_tokens`: 默认 2048
**输出:**
- `text`: STRING
---
### 4. Chat NoASS
**功能**: NoASS 格式的角色扮演对话
```
┌─────────────────────────────────────┐
│ 🎭 Chat NoASS │
├─────────────────────────────────────┤
│ ───○ config │
│ ───○ history (optional) │
│ ───○ image (optional) │
├─────────────────────────────────────┤
│ system [You are... ] │
│ [ ] │
│ user_input [ ] │
│ user_name [User ] │
│ char_name [Assistant] │
│ temperature [1.0 ] │
│ max_tokens [2048 ] │
├─────────────────────────────────────┤
│ text ○───│
│ history ○───│
└─────────────────────────────────────┘
```
**输入:**
- `config`: CONFIG
- `history`: STRING (可选,上一轮输出)
- `image`: IMAGE (可选)
- `system`: 系统提示/世界观设定
- `user_input`: 用户本轮输入
- `user_name`: 用户角色名
- `char_name`: AI 角色名
- `temperature`: 0-2
- `max_tokens`: 默认 2048
**输出:**
- `text`: STRING (本轮 AI 回复)
- `history`: STRING (完整对话历史,可连接到下一个节点的 history 输入)
---
### 5. Gemini Image Gen
**功能**: Gemini 文生图
```
┌─────────────────────────────────────┐
│ 🎨 Gemini Image Gen │
├─────────────────────────────────────┤
│ ───○ config (Gemini only) │
├─────────────────────────────────────┤
│ prompt [A cat wearing a hat ] │
│ [ ] │
│ aspect [1:1 ▼] │
│ size [1K ▼] │
├─────────────────────────────────────┤
│ IMAGE ○───│
│ text ○───│
└─────────────────────────────────────┘
```
**输入:**
- `config`: CONFIG (必须是 Gemini provider)
- `prompt`: 多行文本
- `aspect`: 下拉 ["1:1", "16:9", "9:16", "4:3", "3:4"]
- `size`: 下拉 ["1K", "2K", "4K"]
**输出:**
- `IMAGE`: IMAGE
- `text`: STRING (Gemini 的文字描述)
---
### 6. Gemini Image Edit
**功能**: Gemini 图片编辑
```
┌─────────────────────────────────────┐
│ ✏️ Gemini Image Edit │
├─────────────────────────────────────┤
│ ───○ config (Gemini only) │
│ ───○ image │
├─────────────────────────────────────┤
│ prompt [Add a red hat ] │
│ [ ] │
│ size [1K ▼] │
├─────────────────────────────────────┤
│ IMAGE ○───│
│ text ○───│
└─────────────────────────────────────┘
```
**输入:**
- `config`: CONFIG (必须是 Gemini provider)
- `image`: IMAGE (要编辑的图片)
- `prompt`: 编辑指令
- `size`: 下拉 ["1K", "2K", "4K"]
**输出:**
- `IMAGE`: IMAGE
- `text`: STRING
---
## 数据流示例
### 基础对话
```
[API Config] ──CONFIG──> [Chat] ──text──> [Display Text]
```
### 图片分析
```
[Load Image] ──IMAGE──┐
├──> [Chat with Image] ──text──> [Display Text]
[API Config] ──CONFIG─┘
```
### 多轮 NoASS 对话
```
[API Config] ──CONFIG──> [Chat NoASS] ──history──> [Chat NoASS] ──history──> ...
│ │
└──text──> └──text──>
```
### Gemini 图片生成
```
[API Config] ──CONFIG──> [Gemini Image Gen] ──IMAGE──> [Preview Image]
│ │
│ ──text──> [Display Text]
```
---
## 文件结构
```
custom_nodes/ComfyUI-SimpleChat/
├── __init__.py # 节点注册
├── nodes/
│ ├── __init__.py
│ ├── config.py # API Config
│ ├── chat.py # Chat
│ ├── chat_image.py # Chat with Image
│ ├── chat_noass.py # Chat NoASS
│ ├── gemini_gen.py # Gemini Image Gen
│ └── gemini_edit.py # Gemini Image Edit
├── core/
│ ├── __init__.py
│ ├── providers/
│ │ ├── __init__.py
│ │ ├── base.py # Provider 抽象基类
│ │ ├── openai.py # OpenAI (+ 兼容 API)
│ │ ├── claude.py # Claude/Anthropic
│ │ └── gemini.py # Google Gemini
│ ├── noass.py # NoASS 格式处理
│ └── image_utils.py # 图片转换
├── docs/ # 文档
├── requirements.txt
└── README.md
```
---
## 实现优先级
1. **Phase 1**: API Config + Chat (最小可用)
2. **Phase 2**: Chat with Image
3. **Phase 3**: Chat NoASS
4. **Phase 4**: Gemini Image Gen/Edit
+153
View File
@@ -0,0 +1,153 @@
# ComfyUI API Nodes Patterns Research
## 目录结构
ComfyUI 官方 API 节点位于 `/comfy_api_nodes/`:
```
comfy_api_nodes/
├── apis/ # Pydantic request/response models
├── util/ # 共享工具
│ ├── client.py # HTTP 客户端 (sync_op, poll_op)
│ ├── _helpers.py # 认证, base URL
│ ├── conversions.py # 图片/音频/视频转换
│ └── ...
├── nodes_openai.py # OpenAI 节点
├── nodes_gemini.py # Gemini 节点
└── ...
```
## 节点定义模式
### 新版 API 节点模式 (IO.ComfyNode)
```python
class MyApiNode(IO.ComfyNode):
@classmethod
def define_schema(cls):
return IO.Schema(
node_id="MyApiNode",
display_name="My API Node",
category="api node/category",
inputs=[
IO.String.Input("prompt", multiline=True),
IO.Image.Input("image", optional=True),
],
outputs=[
IO.Image.Output("IMAGE"),
IO.String.Output("TEXT"),
],
hidden=[
IO.Hidden.auth_token_comfy_org,
IO.Hidden.api_key_comfy_org,
],
is_api_node=True,
)
@classmethod
async def execute(cls, prompt: str, image=None):
# Implementation
return IO.NodeOutput(result_image, result_text)
```
### 传统节点模式 (ComfyNodeABC)
```python
class MyNode(ComfyNodeABC):
@classmethod
def INPUT_TYPES(s) -> InputTypeDict:
return {
"required": {
"prompt": ("STRING", {"multiline": True}),
},
"optional": {
"image": ("IMAGE",),
}
}
RETURN_TYPES = ("IMAGE", "STRING")
RETURN_NAMES = ("image", "text")
FUNCTION = "execute"
CATEGORY = "my_category"
def execute(self, prompt, image=None):
return (result_image, result_text)
```
## 图片处理
### Tensor 转 Base64
```python
from comfy_api_nodes.util import tensor_to_base64_string
base64_str = tensor_to_base64_string(image_tensor)
data_uri = f"data:image/png;base64,{base64_str}"
```
### Base64 转 Tensor
```python
import base64
from io import BytesIO
from comfy_api_nodes.util import bytesio_to_image_tensor
image_data = base64.b64decode(base64_string)
tensor = bytesio_to_image_tensor(BytesIO(image_data))
```
### Tensor 格式
ComfyUI 图片 tensor: `(B, H, W, C)` - Batch, Height, Width, Channels (RGBA)
## HTTP 请求
### 同步请求
```python
from comfy_api_nodes.util import sync_op
response = await sync_op(
cls,
ApiEndpoint(path="/api/endpoint", method="POST"),
response_model=ResponseModel,
data=RequestModel(...),
timeout=3600.0,
)
```
### 轮询请求
```python
from comfy_api_nodes.util import poll_op
response = await poll_op(
cls,
ApiEndpoint(path="/api/status/{id}", method="GET"),
response_model=ResponseModel,
status_extractor=lambda r: r.status,
completed_statuses=["completed"],
failed_statuses=["failed"],
)
```
## 认证
官方节点使用 ComfyUI 代理认证。我们的自定义节点将支持:
1. 直接 API Key 输入
2. 自定义 base_url
## 重要发现
1. **无 Claude/Anthropic 集成** - 官方没有 Claude 节点
2. **代理模式** - 官方节点通过 ComfyUI 代理访问 API
3. **异步执行** - 所有 API 调用使用 async/await
4. **Pydantic 验证** - 请求/响应使用 Pydantic 模型
## 对我们的启示
由于我们要支持自定义 base_url 和直接 API 访问,我们将:
1. 使用传统节点模式(更简单,更灵活)
2. 直接使用 aiohttp/httpx 发请求
3. 手动处理认证
4. 复用 ComfyUI 的图片转换工具
+127
View File
@@ -0,0 +1,127 @@
# Gemini Image Generation Research Notes
## Available Models
| Model | Use Case | Features |
|-------|----------|----------|
| `gemini-2.5-flash-image` | Fast, general image generation | Lower cost, faster |
| `gemini-3-pro-image-preview` | Professional asset production | Higher quality, 1K/2K/4K output |
## API Endpoint
```
POST https://generativelanguage.googleapis.com/v1beta/models/{model-id}:generateContent
```
## Request Format
### 关键:启用图像生成
必须在 `generationConfig` 中指定 `responseModalities`:
```json
{
"contents": [{
"parts": [
{"text": "Generate an image of a cat"}
]
}],
"generationConfig": {
"responseModalities": ["TEXT", "IMAGE"]
}
}
```
### 带图片输入的请求(图片编辑/参考)
```json
{
"contents": [{
"parts": [
{"text": "Edit this image to add a hat"},
{
"inline_data": {
"mime_type": "image/png",
"data": "base64_encoded_image_data"
}
}
]
}],
"generationConfig": {
"responseModalities": ["TEXT", "IMAGE"]
}
}
```
### Image Configuration
```json
{
"generationConfig": {
"responseModalities": ["TEXT", "IMAGE"],
"imageConfig": {
"aspectRatio": "16:9",
"imageSize": "2K"
}
}
}
```
支持的 aspectRatio: "1:1", "16:9", "9:16", "5:4", "4:5", "3:4", "4:3"
支持的 imageSize: "1K", "2K", "4K"
## Response Format
```json
{
"candidates": [{
"content": {
"parts": [
{"text": "Here is your generated image:"},
{
"inlineData": {
"mimeType": "image/png",
"data": "base64_encoded_image_data"
}
}
]
}
}]
}
```
## Input Modalities
1. **Text-to-Image**: 纯文本提示词
2. **Text-and-Image-to-Image**: 文本 + 参考图片
3. **Multi-Reference Images**: 最多 14 张参考图片
## Authentication
使用 API Key:
```
x-goog-api-key: YOUR_API_KEY
```
或 Bearer Token:
```
Authorization: Bearer YOUR_TOKEN
```
## Special Features
- **Thought Signatures**: 多轮对话保持推理上下文
- **SynthID**: 自动添加 AI 生成图片水印
- **Grounding with Google Search**: 可使用搜索验证事实
## Implementation Notes for ComfyUI-SimpleChat
1. 需要支持自定义 base_url(用户可能使用代理)
2. 需要处理 base64 图片编解码
3. 需要支持图片输入作为参考
4. 返回的图片需要转换为 ComfyUI tensor 格式
## Reference
- https://ai.google.dev/gemini-api/docs/image-generation
- https://ai.google.dev/gemini-api/docs/imagen
+69
View File
@@ -0,0 +1,69 @@
# NoASS Research Notes
## What is NoASS?
NoASS (No Assistant) 是一种用于角色扮演和创意写作的提示词方法论。它将基于聊天的助手交互转换为统一的叙事格式。
## Core Concept
传统聊天是 human/assistant 轮流对话,NoASS 将整个 roleplay 合并成一个连续的故事,用户通过编辑来贡献角色动作。
## Key Motivation
1. **消除角色偏见**: 传统聊天模型对 human 角色有内在偏见,往往给予不合理的优势
2. **解决统一响应长度问题**: 标准聊天训练鼓励一致的回答长度,而动态叙事需要变化的响应规模
3. **消除闭合模式**: 传统响应以隐式呼吁用户结束(反问句),破坏沉浸感
4. **平等对待角色**: 所有角色被平等对待,因为没有角色正式"属于"任何一方
## Technical Implementation
### Context Transformation
**传统 ChatML 格式:**
```
<|im_start|>user
[Instructions]
<|im_end|>
<|im_start|>assistant
[Greeting]
<|im_end|>
<|im_start|>user
[User action]
<|im_end|>
```
**NoASS 格式:**
```
<|im_start|>user
[Instructions for story with all characters]
<|im_start|>assistant
[Greeting]
{{user}}: [user's action]
{{char}}: [character's action]
...
```
关键机制:所有 roleplay 变成单个预填充的 assistant 消息,用户输入被"拼接"到叙事中,使用角色前缀标记。
## Formatting Standards
- 一段 = 一个角色前缀 (`**char name:**`)
- 前缀下:*动作*、"对话"、'思想',顺序任意
- 最小叙事描述;主要动作使用前缀
- 用户响应省略自己的前缀(扩展自动插入)
## Stop Token Mechanism
使用用户角色的前缀作为停止令牌,防止模型为玩家角色编写对话。这是防止模型为用户生成动作的关键。
## Implementation for ComfyUI-SimpleChat
在我们的实现中,NoASS 功能将:
1. 允许用户选择是否启用 NoASS 模式
2. 自动处理消息格式转换
3. 支持自定义角色前缀
4. 使用用户前缀作为停止字符串
## Reference
- https://rentry.org/noass_ext
+18
View File
@@ -0,0 +1,18 @@
"""
SimpleChat nodes module.
"""
from .config import SimpleChatConfig
from .chat import SimpleChatText
from .chat_image import SimpleChatImage
from .chat_noass import SimpleChatNoASS
from .gemini_gen import GeminiImageGen
from .gemini_edit import GeminiImageEdit
__all__ = [
"SimpleChatConfig",
"SimpleChatText",
"SimpleChatImage",
"SimpleChatNoASS",
"GeminiImageGen",
"GeminiImageEdit",
]
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+55
View File
@@ -0,0 +1,55 @@
"""
Chat node - Basic text conversation.
"""
from ..core import get_provider, ChatConfig
class SimpleChatText:
"""Basic text conversation with LLM."""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"config": ("SIMPLECHAT_CONFIG",),
"prompt": ("STRING", {"multiline": True, "default": ""}),
},
"optional": {
"system": ("STRING", {"multiline": True, "default": ""}),
"temperature": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 2.0, "step": 0.1}),
"max_tokens": ("INT", {"default": 2048, "min": 1, "max": 128000}),
}
}
RETURN_TYPES = ("STRING",)
RETURN_NAMES = ("text",)
FUNCTION = "chat"
CATEGORY = "SimpleChat"
DESCRIPTION = "Send a text prompt to LLM and get a response."
async def chat(
self,
config: ChatConfig,
prompt: str,
system: str = "",
temperature: float = 1.0,
max_tokens: int = 2048,
):
# Build messages
messages = []
if system.strip():
messages.append({"role": "system", "content": system})
messages.append({"role": "user", "content": prompt})
# Get provider and send request
provider = get_provider(config)
# Directly await the async provider method
response = await provider.chat(
messages=messages,
model=config.model,
temperature=temperature,
max_tokens=max_tokens,
)
return (response.text,)
+59
View File
@@ -0,0 +1,59 @@
"""
Chat with Image node - Send image to LLM for analysis.
"""
import torch
from ..core import get_provider, ChatConfig
class SimpleChatImage:
"""Send image to LLM for visual understanding."""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"config": ("SIMPLECHAT_CONFIG",),
"image": ("IMAGE",),
"prompt": ("STRING", {"multiline": True, "default": "Describe this image."}),
},
"optional": {
"system": ("STRING", {"multiline": True, "default": ""}),
"temperature": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 2.0, "step": 0.1}),
"max_tokens": ("INT", {"default": 2048, "min": 1, "max": 128000}),
}
}
RETURN_TYPES = ("STRING",)
RETURN_NAMES = ("text",)
FUNCTION = "chat"
CATEGORY = "SimpleChat"
DESCRIPTION = "Send an image to LLM for visual analysis and get a text response."
async def chat(
self,
config: ChatConfig,
image: torch.Tensor,
prompt: str,
system: str = "",
temperature: float = 1.0,
max_tokens: int = 2048,
):
# Build messages
messages = []
if system.strip():
messages.append({"role": "system", "content": system})
messages.append({"role": "user", "content": prompt})
# Get provider and send request
provider = get_provider(config)
# Directly await the async provider method
response = await provider.chat(
messages=messages,
model=config.model,
temperature=temperature,
max_tokens=max_tokens,
images=[image],
)
return (response.text,)
+132
View File
@@ -0,0 +1,132 @@
"""
Chat NoASS node - Roleplay-optimized conversation with NoASS format.
"""
import torch
from ..core import (
get_provider,
ChatConfig,
format_noass_prompt,
build_noass_messages,
extract_noass_response,
build_full_history,
get_stop_sequences,
)
class SimpleChatNoASS:
"""NoASS format conversation for roleplay optimization."""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"config": ("SIMPLECHAT_CONFIG",),
"scenario_instructions": ("STRING", {
"multiline": True,
"default": "",
"placeholder": "Define world setting, characters, and instructions here (The Scenario)."
}),
"user_action": ("STRING", {
"multiline": True,
"default": "",
"placeholder": "Your current action or dialogue."
}),
},
"optional": {
"prefill_start": ("STRING", {
"multiline": True,
"default": "",
"placeholder": "Force the start of the assistant's response (Prefill)."
}),
"history": ("STRING", {"multiline": True, "default": "", "forceInput": True}),
"image": ("IMAGE",),
"user_name": ("STRING", {"default": "User"}),
"char_name": ("STRING", {"default": "Assistant"}),
"temperature": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 2.0, "step": 0.1}),
"max_tokens": ("INT", {"default": 2048, "min": 1, "max": 128000}),
}
}
RETURN_TYPES = ("STRING", "STRING")
RETURN_NAMES = ("text", "history")
FUNCTION = "chat"
CATEGORY = "SimpleChat"
DESCRIPTION = "NoASS (Experimental) - Hardcore Roleplay with Assistant Prefill. Uses single-turn context with giant assistant prefill to force formatting."
async def chat(
self,
config: ChatConfig,
scenario_instructions: str,
user_action: str,
prefill_start: str = "",
history: str = "",
image: torch.Tensor = None,
user_name: str = "User",
char_name: str = "Assistant",
temperature: float = 1.0,
max_tokens: int = 2048,
):
# Format in NoASS style
system_prompt, prefilled = format_noass_prompt(
system=scenario_instructions,
history=history if history.strip() else None,
user_input=user_action,
prefill_start=prefill_start,
user_name=user_name,
char_name=char_name,
)
# Build messages using the new prefill logic
messages = build_noass_messages(
system=scenario_instructions,
history=history if history.strip() else None,
user_input=user_action,
prefill_start=prefill_start,
user_name=user_name,
char_name=char_name,
)
# Get provider
provider = get_provider(config)
# Prepare images
images = [image] if image is not None else None
# Get stop sequences
stop_sequences = get_stop_sequences(user_name)
# Directly await the async provider method
response = await provider.chat(
messages=messages,
model=config.model,
temperature=temperature,
max_tokens=max_tokens,
images=images,
stop=stop_sequences, # Some providers support this
)
# Extract response and build history
response_text = extract_noass_response(
response.text,
user_name=user_name,
char_name=char_name,
)
# Reconstruct the full text for the history output
new_history = build_full_history(
history=history if history.strip() else None,
user_input=user_action,
prefill_start=prefill_start,
response=response_text,
user_name=user_name,
char_name=char_name,
)
# If we had a prefill, the returned 'text' should probably include it
# so the user sees the full response in the text output node
if prefill_start.strip():
final_text_output = f"{prefill_start} {response_text}"
else:
final_text_output = response_text
return (final_text_output, new_history)
+79
View File
@@ -0,0 +1,79 @@
"""
API Config node - Configure API connection with dynamic model selection.
"""
from ..core.providers import ChatConfig
class SimpleChatConfig:
"""
Configure API connection for SimpleChat nodes.
Supports dynamic model fetching via the "Refresh Models" button.
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"provider": (["openai", "claude", "gemini"], {
"default": "openai",
"tooltip": "Select the LLM provider",
}),
"api_key": ("STRING", {
"default": "",
"tooltip": "Your API key for the selected provider",
}),
# Defined as STRING to bypass ComfyUI's strict list validation.
# Frontend JS will convert this to a COMBO widget for dropdown functionality.
"model": ("STRING", {
"default": "",
"tooltip": "Select a model. Use the Refresh button to fetch available models.",
}),
},
"optional": {
"base_url": ("STRING", {
"default": "",
"placeholder": "Leave empty for default URL",
"tooltip": "Custom API base URL (for proxies or self-hosted deployments)",
}),
}
}
RETURN_TYPES = ("SIMPLECHAT_CONFIG",)
RETURN_NAMES = ("config",)
FUNCTION = "create_config"
CATEGORY = "SimpleChat"
DESCRIPTION = """Configure API connection.
Usage:
1. Select Provider
2. Enter API Key
3. Click 'Refresh Models' button to fetch available models
4. Select Model
"""
def create_config(
self,
provider: str,
api_key: str,
model: str,
base_url: str = "",
):
# Default URLs map
DEFAULT_URLS = {
"openai": "https://api.openai.com/v1",
"claude": "https://api.anthropic.com/v1",
"gemini": "https://generativelanguage.googleapis.com/v1beta",
}
# Use default URL if not provided
if not base_url.strip():
base_url = DEFAULT_URLS.get(provider, "")
config = ChatConfig(
provider=provider,
api_key=api_key,
base_url=base_url,
model=model,
)
return (config,)
+58
View File
@@ -0,0 +1,58 @@
"""
Gemini Image Edit node - Edit images using Gemini.
"""
import torch
from ..core import get_provider, ChatConfig
class GeminiImageEdit:
"""Edit images using Gemini (Nano Banana / Nano Banana Pro)."""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"config": ("SIMPLECHAT_CONFIG",),
"image": ("IMAGE",),
"prompt": ("STRING", {"multiline": True, "default": ""}),
},
"optional": {
"size": (["1K", "2K", "4K"], {"default": "1K"}),
}
}
RETURN_TYPES = ("IMAGE", "STRING")
RETURN_NAMES = ("IMAGE", "text")
FUNCTION = "edit"
CATEGORY = "SimpleChat/Gemini"
DESCRIPTION = "Edit or modify an image using Gemini image models with a text prompt."
async def edit(
self,
config: ChatConfig,
image: torch.Tensor,
prompt: str,
size: str = "1K",
):
# Validate provider
if config.provider != "gemini":
raise ValueError("Gemini Image Edit requires Gemini provider. Please use a Gemini config.")
# Get provider
provider = get_provider(config)
# Directly await the async provider method
response = await provider.generate_image(
prompt=prompt,
model=config.model,
reference_image=image,
aspect_ratio=None, # Preserve original aspect ratio for edits
size=size,
)
# Ensure we have an image
if response.image is None:
# Return original image if edit failed
return (image, response.text or "No image generated")
return (response.image, response.text)
+59
View File
@@ -0,0 +1,59 @@
"""
Gemini Image Gen node - Text-to-image generation with Gemini.
"""
import torch
from ..core import get_provider, ChatConfig
class GeminiImageGen:
"""Generate images from text using Gemini (Nano Banana / Nano Banana Pro)."""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"config": ("SIMPLECHAT_CONFIG",),
"prompt": ("STRING", {"multiline": True, "default": ""}),
},
"optional": {
"aspect_ratio": (["1:1", "16:9", "9:16", "4:3", "3:4", "5:4", "4:5"], {"default": "1:1"}),
"size": (["1K", "2K", "4K"], {"default": "1K"}),
}
}
RETURN_TYPES = ("IMAGE", "STRING")
RETURN_NAMES = ("IMAGE", "text")
FUNCTION = "generate"
CATEGORY = "SimpleChat/Gemini"
DESCRIPTION = "Generate image from text using Gemini image models (gemini-2.5-flash-image or gemini-3-pro-image-preview)."
async def generate(
self,
config: ChatConfig,
prompt: str,
aspect_ratio: str = "1:1",
size: str = "1K",
):
# Validate provider
if config.provider != "gemini":
raise ValueError("Gemini Image Gen requires Gemini provider. Please use a Gemini config.")
# Get provider
provider = get_provider(config)
# Directly await the async provider method
response = await provider.generate_image(
prompt=prompt,
model=config.model,
reference_image=None,
aspect_ratio=aspect_ratio,
size=size,
)
# Ensure we have an image
if response.image is None:
# Return empty image if generation failed
empty = torch.zeros((1, 512, 512, 3))
return (empty, response.text or "No image generated")
return (response.image, response.text)
+1
View File
@@ -0,0 +1 @@
aiohttp>=3.8.0
+88
View File
@@ -0,0 +1,88 @@
import { app } from "../../scripts/app.js";
import { api } from "../../scripts/api.js";
app.registerExtension({
name: "ComfyUI.SimpleChat.Config.V3", // Use a new name to avoid browser caching issues
async nodeCreated(node, app) {
if (node.comfyClass === "SimpleChatConfig") {
// --- WIDGET REPLACEMENT AND REORDERING ---
// Find the original string widget for 'model'
const modelWidgetIndex = node.widgets.findIndex((w) => w.name === "model");
if (modelWidgetIndex === -1) return;
const originalModelWidget = node.widgets[modelWidgetIndex];
const originalModelValue = originalModelWidget.value;
// Step 1: Create new widgets using the addWidget API. This appends them temporarily.
const newModelCombo = node.addWidget("combo", "model", originalModelValue, () => {}, {
values: [originalModelValue || ""]
});
const refreshButton = node.addWidget("button", "Refresh Models", "refresh", () => {});
// Step 2: Remove the newly created widgets from the end of the array.
// We now have fully constructed widget objects, detached from the node's widget list.
node.widgets.pop(); // removes refreshButton
node.widgets.pop(); // removes newModelCombo
// Step 3: Remove the original string widget from its position.
node.widgets.splice(modelWidgetIndex, 1);
// Step 4: Insert the new, constructed widgets back into the array at the correct positions.
node.widgets.splice(modelWidgetIndex, 0, newModelCombo); // Insert combo where the old one was.
node.widgets.splice(modelWidgetIndex + 1, 0, refreshButton); // Insert button right after it.
// --- END WIDGET MANIPULATION ---
// Now, find all widgets again to be safe and attach logic
const providerWidget = node.widgets.find((w) => w.name === "provider");
const apiKeyWidget = node.widgets.find((w) => w.name === "api_key");
const baseUrlWidget = node.widgets.find((w) => w.name === "base_url");
// Attach the callback logic to the refresh button
refreshButton.callback = () => {
const provider = providerWidget.value;
const apiKey = apiKeyWidget.value;
const baseUrl = baseUrlWidget ? baseUrlWidget.value : "";
if (!apiKey) {
alert("Please enter an API Key first.");
return;
}
refreshButton.label = "Loading...";
refreshButton.disabled = true;
node.setDirtyCanvas(true, true);
const url = `/simplechat/models/${provider}?api_key=${encodeURIComponent(apiKey)}&base_url=${encodeURIComponent(baseUrl)}`;
api.fetchApi(url)
.then(response => response.json())
.then(models => {
if (Array.isArray(models) && models.length > 0) {
newModelCombo.options.values = models;
if (!newModelCombo.value || !models.includes(newModelCombo.value)) {
newModelCombo.value = models[0];
}
} else {
newModelCombo.options.values = [];
newModelCombo.value = "";
alert("No models found.");
}
})
.catch(err => {
console.error("Error fetching models:", err);
alert("Error fetching models: " + err.message);
})
.finally(() => {
refreshButton.label = "Refresh Models";
refreshButton.disabled = false;
node.setDirtyCanvas(true, true);
});
};
// Trigger a redraw to ensure the node resizes correctly after manipulation
node.computeSize();
node.setDirtyCanvas(true, true);
}
}
});