Initial commit of ComfyUI-SimpleChat
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
# ComfyUI-SimpleChat
|
||||
|
||||
Simple, no-nonsense LLM chat nodes for ComfyUI. Connect to OpenAI, Claude, Gemini, and local LLMs with ease.
|
||||
|
||||
[English](README.md) | [中文](README_CN.md)
|
||||
|
||||
## Features
|
||||
|
||||
- **All-in-One Configuration**: Single, powerful config node for all providers.
|
||||
- **Dynamic Model Fetching**: Click one button to fetch the *actual* list of models available to your API key. No hardcoded lists!
|
||||
- **Universal Compatibility**: Works with official APIs and any OpenAI-compatible proxy/local endpoint.
|
||||
- **Multimodal Support**: Chat with images using vision-capable models.
|
||||
- **NoASS Roleplay Mode (Exp)**: Experimental Hardcore roleplay node with Assistant Prefill support.
|
||||
- **Gemini Native Features**: Dedicated nodes for Gemini's image generation and editing capabilities.
|
||||
|
||||
## Installation
|
||||
|
||||
1. Clone this repository into your `ComfyUI/custom_nodes` folder:
|
||||
```bash
|
||||
git clone https://github.com/yourname/ComfyUI-SimpleChat.git
|
||||
```
|
||||
2. Restart ComfyUI.
|
||||
|
||||
## Usage Guide
|
||||
|
||||
### 1. API Configuration (The Core)
|
||||
|
||||
Everything starts with the **API Config** node.
|
||||
|
||||
1. **Add Node**: Right-click -> `SimpleChat` -> `API Config`.
|
||||
2. **Select Provider**: Choose `openai`, `claude`, or `gemini`.
|
||||
3. **Enter API Key**: Paste your secret key.
|
||||
* *OpenAI*: `sk-...`
|
||||
* *Claude*: `sk-ant-...`
|
||||
* *Gemini*: Your Google AI Studio key.
|
||||
4. **Base URL (Optional)**:
|
||||
* Leave empty for official APIs.
|
||||
* Enter a custom URL if you use a proxy, a local LLM (like LM Studio/Ollama), or a third-party service.
|
||||
* *Example*: `https://api.deepseek.com/v1` or `http://localhost:1234/v1`.
|
||||
5. **Select Model**:
|
||||
* **Click the "Refresh Models" button**.
|
||||
* The node will connect to the API and fetch the *real-time list* of models you can use.
|
||||
* Select your desired model from the dropdown.
|
||||
|
||||
### 2. Basic Chat
|
||||
Use this for standard text generation or Q&A.
|
||||
|
||||
* **Node**: `Chat`
|
||||
* **Connection**: Connect `config` from **API Config** to `config` on **Chat**.
|
||||
* **Input**: Enter your prompt in `text`.
|
||||
* **Output**: Returns the AI's response as a string.
|
||||
|
||||
### 3. Chat with Images
|
||||
Use this to have the AI analyze or describe an image.
|
||||
|
||||
* **Node**: `Chat with Image`
|
||||
* **Connection**:
|
||||
* Connect `config` from **API Config**.
|
||||
* Connect an image (from `Load Image` node) to the `image` input.
|
||||
* **Input**: Enter a prompt (e.g., "Describe this image").
|
||||
|
||||
### 4. NoASS Chat (Experimental)
|
||||
The "No Assistant" (NoASS) node is an experimental feature designed for **hardcore roleplay and creative writing**. It forces the AI to break character restrictions by pre-filling its response start.
|
||||
|
||||
* **Node**: `Chat NoASS (Exp)`
|
||||
* **Inputs**:
|
||||
* **Scenario Instructions**: (User Message) Define the world, characters, and rules. This sets the stage.
|
||||
* **User Action**: Your current turn or dialogue.
|
||||
* **Prefill Start** (Optional): Force the first few words of the AI's response. The AI *must* continue from here. Great for steering the tone.
|
||||
* **History**: Connect output `history` to the next node's input `history` to maintain conversation context.
|
||||
|
||||
### 5. Gemini Image Generation
|
||||
Generate images using Google's Gemini models.
|
||||
|
||||
* **Node**: `Gemini Image Gen`
|
||||
* **Requirement**: Must use `gemini` provider in **API Config**.
|
||||
* **Output**: Generates a standard ComfyUI IMAGE.
|
||||
|
||||
### 6. Gemini Image Editing
|
||||
Modify existing images using text instructions.
|
||||
|
||||
* **Node**: `Gemini Image Edit`
|
||||
* **Input**: Source image + Mask image + Prompt.
|
||||
|
||||
## FAQ
|
||||
|
||||
**Q: I don't see my model in the list.**
|
||||
A: Make sure you entered your API Key and Base URL correctly, then click **Refresh Models**. If the API supports listing models, they will appear. If not, you can try typing the model name directly (if the UI allows) or checking your provider's documentation.
|
||||
|
||||
**Q: Can I use local models?**
|
||||
A: Yes! Select `openai` as the provider, set the **Base URL** to your local server address (e.g., `http://127.0.0.1:1234/v1`), and click Refresh.
|
||||
|
||||
**Q: My dropdown is empty.**
|
||||
A: Check your internet connection and API Key. If the API Key is invalid, the list will not populate.
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
Binary file not shown.
+47
@@ -0,0 +1,47 @@
|
||||
"""
|
||||
ComfyUI-SimpleChat: Simple LLM chat nodes for ComfyUI.
|
||||
|
||||
Supports OpenAI, Claude, and Gemini APIs with custom base URL support.
|
||||
Features NoASS mode for roleplay optimization and Gemini image generation.
|
||||
"""
|
||||
from .nodes import (
|
||||
SimpleChatConfig,
|
||||
SimpleChatText,
|
||||
SimpleChatImage,
|
||||
SimpleChatNoASS,
|
||||
GeminiImageGen,
|
||||
GeminiImageEdit,
|
||||
)
|
||||
|
||||
# Register API routes
|
||||
from .api import setup_routes
|
||||
setup_routes()
|
||||
|
||||
# Node class mappings for ComfyUI
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"SimpleChatConfig": SimpleChatConfig,
|
||||
"SimpleChatText": SimpleChatText,
|
||||
"SimpleChatImage": SimpleChatImage,
|
||||
"SimpleChatNoASS": SimpleChatNoASS,
|
||||
"GeminiImageGen": GeminiImageGen,
|
||||
"GeminiImageEdit": GeminiImageEdit,
|
||||
}
|
||||
|
||||
# Display names for the UI
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"SimpleChatConfig": "API Config",
|
||||
"SimpleChatText": "Chat",
|
||||
"SimpleChatImage": "Chat with Image",
|
||||
"SimpleChatNoASS": "Chat NoASS",
|
||||
"GeminiImageGen": "Gemini Image Gen",
|
||||
"GeminiImageEdit": "Gemini Image Edit",
|
||||
}
|
||||
|
||||
# Expose web directory for JS extensions
|
||||
WEB_DIRECTORY = "./web"
|
||||
|
||||
__all__ = [
|
||||
"NODE_CLASS_MAPPINGS",
|
||||
"NODE_DISPLAY_NAME_MAPPINGS",
|
||||
"WEB_DIRECTORY",
|
||||
]
|
||||
Binary file not shown.
@@ -0,0 +1,6 @@
|
||||
"""
|
||||
API routes for ComfyUI-SimpleChat.
|
||||
"""
|
||||
from .routes import setup_routes
|
||||
|
||||
__all__ = ["setup_routes"]
|
||||
Binary file not shown.
Binary file not shown.
+201
@@ -0,0 +1,201 @@
|
||||
"""
|
||||
API routes for fetching model lists from various providers.
|
||||
"""
|
||||
import json
|
||||
import aiohttp
|
||||
from aiohttp import web
|
||||
|
||||
try:
|
||||
from server import PromptServer
|
||||
HAS_SERVER = True
|
||||
except ImportError:
|
||||
HAS_SERVER = False
|
||||
|
||||
|
||||
# Predefined model lists for each provider
|
||||
PREDEFINED_MODELS = {
|
||||
"openai": [
|
||||
"gpt-4o",
|
||||
"gpt-4o-mini",
|
||||
"gpt-4-turbo",
|
||||
"gpt-4",
|
||||
"gpt-3.5-turbo",
|
||||
"o1-preview",
|
||||
"o1-mini",
|
||||
],
|
||||
"claude": [
|
||||
"claude-sonnet-4-20250514",
|
||||
"claude-3-5-sonnet-20241022",
|
||||
"claude-3-5-haiku-20241022",
|
||||
"claude-3-opus-20240229",
|
||||
"claude-3-sonnet-20240229",
|
||||
"claude-3-haiku-20240307",
|
||||
],
|
||||
"gemini": [
|
||||
"gemini-2.5-flash-preview-05-20",
|
||||
"gemini-2.0-flash",
|
||||
"gemini-2.0-flash-exp",
|
||||
"gemini-1.5-pro",
|
||||
"gemini-1.5-flash",
|
||||
"gemini-2.5-flash-image",
|
||||
"gemini-3-pro-image-preview",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
async def fetch_openai_models(api_key: str, base_url: str) -> list[str]:
|
||||
"""Fetch model list from OpenAI-compatible API."""
|
||||
url = f"{base_url.rstrip('/')}/models"
|
||||
headers = {"Authorization": f"Bearer {api_key}"}
|
||||
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(url, headers=headers, timeout=10) as resp:
|
||||
if resp.status == 200:
|
||||
data = await resp.json()
|
||||
models = [m["id"] for m in data.get("data", [])]
|
||||
# Return all models, sorted alphabetically
|
||||
return sorted(models) if models else PREDEFINED_MODELS["openai"]
|
||||
except Exception as e:
|
||||
print(f"[SimpleChat] Failed to fetch OpenAI models: {e}")
|
||||
|
||||
return PREDEFINED_MODELS["openai"]
|
||||
|
||||
|
||||
async def fetch_claude_models(api_key: str, base_url: str) -> list[str]:
|
||||
"""
|
||||
Fetch models for Claude provider.
|
||||
Tries to hit the /models endpoint (supported by some proxies and newer Anthropic API).
|
||||
Falls back to predefined list if fails.
|
||||
"""
|
||||
# Anthropic API format for models list is /v1/models
|
||||
# Note: Official Anthropic API recently added models endpoint support
|
||||
url = f"{base_url.rstrip('/')}/models"
|
||||
headers = {
|
||||
"x-api-key": api_key,
|
||||
"anthropic-version": "2023-06-01"
|
||||
}
|
||||
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(url, headers=headers, timeout=10) as resp:
|
||||
if resp.status == 200:
|
||||
data = await resp.json()
|
||||
# Anthropic format: {"data": [{"id": "...", ...}]} similar to OpenAI
|
||||
# Some proxies might return OpenAI format
|
||||
models = []
|
||||
|
||||
# Handle both standard formats just in case
|
||||
items = data.get("data", []) if "data" in data else data.get("models", [])
|
||||
|
||||
if isinstance(items, list):
|
||||
for m in items:
|
||||
if isinstance(m, dict):
|
||||
models.append(m.get("id", m.get("name", "")))
|
||||
elif isinstance(m, str):
|
||||
models.append(m)
|
||||
|
||||
# Filter out empty strings
|
||||
models = [m for m in models if m]
|
||||
|
||||
if models:
|
||||
return sorted(models)
|
||||
except Exception as e:
|
||||
# Silently fail for Claude since it's common for this endpoint to not exist/fail
|
||||
pass
|
||||
|
||||
return PREDEFINED_MODELS["claude"]
|
||||
|
||||
|
||||
async def fetch_gemini_models(api_key: str, base_url: str) -> list[str]:
|
||||
"""Fetch model list from Gemini API."""
|
||||
url = f"{base_url.rstrip('/')}/models?key={api_key}"
|
||||
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(url, timeout=10) as resp:
|
||||
if resp.status == 200:
|
||||
data = await resp.json()
|
||||
models = []
|
||||
for m in data.get("models", []):
|
||||
name = m.get("name", "")
|
||||
# Extract model name from full path (models/gemini-xxx)
|
||||
if name.startswith("models/"):
|
||||
name = name[7:]
|
||||
if name:
|
||||
models.append(name)
|
||||
# Return all models
|
||||
return sorted(models) if models else PREDEFINED_MODELS["gemini"]
|
||||
except Exception as e:
|
||||
print(f"[SimpleChat] Failed to fetch Gemini models: {e}")
|
||||
|
||||
return PREDEFINED_MODELS["gemini"]
|
||||
|
||||
|
||||
def get_all_predefined_models_list() -> list[str]:
|
||||
"""Get a flat list of all predefined models."""
|
||||
all_models = []
|
||||
for provider_models in PREDEFINED_MODELS.values():
|
||||
all_models.extend(provider_models)
|
||||
return all_models
|
||||
|
||||
|
||||
def setup_routes():
|
||||
"""Setup API routes for SimpleChat."""
|
||||
if not HAS_SERVER:
|
||||
return
|
||||
|
||||
@PromptServer.instance.routes.get("/simplechat/models/{provider}")
|
||||
async def get_models(request):
|
||||
"""
|
||||
Get available models for a provider.
|
||||
|
||||
Query params:
|
||||
- api_key: API key for the provider
|
||||
- base_url: (optional) Custom base URL
|
||||
"""
|
||||
provider = request.match_info.get("provider", "").lower()
|
||||
api_key = request.query.get("api_key", "")
|
||||
base_url = request.query.get("base_url", "")
|
||||
|
||||
# Return predefined models if no API key provided
|
||||
if not api_key:
|
||||
models = PREDEFINED_MODELS.get(provider, [])
|
||||
return web.json_response(models)
|
||||
|
||||
# Default base URLs
|
||||
default_urls = {
|
||||
"openai": "https://api.openai.com/v1",
|
||||
"claude": "https://api.anthropic.com/v1",
|
||||
"gemini": "https://generativelanguage.googleapis.com/v1beta",
|
||||
}
|
||||
|
||||
if not base_url:
|
||||
base_url = default_urls.get(provider, "")
|
||||
|
||||
# Fetch models based on provider
|
||||
if provider == "openai":
|
||||
models = await fetch_openai_models(api_key, base_url)
|
||||
elif provider == "claude":
|
||||
models = await fetch_claude_models(api_key, base_url)
|
||||
elif provider == "gemini":
|
||||
models = await fetch_gemini_models(api_key, base_url)
|
||||
else:
|
||||
models = []
|
||||
|
||||
return web.json_response(models)
|
||||
|
||||
@PromptServer.instance.routes.get("/simplechat/models")
|
||||
async def get_all_models_list(request):
|
||||
"""
|
||||
Get all predefined models as a flat list (for COMBO remote input).
|
||||
Returns a simple array of model names.
|
||||
"""
|
||||
return web.json_response(get_all_predefined_models_list())
|
||||
|
||||
@PromptServer.instance.routes.get("/simplechat/models_dict")
|
||||
async def get_all_predefined_models(request):
|
||||
"""Get all predefined models organized by provider."""
|
||||
return web.json_response(PREDEFINED_MODELS)
|
||||
|
||||
print("[SimpleChat] API routes registered")
|
||||
@@ -0,0 +1,51 @@
|
||||
"""
|
||||
SimpleChat core module.
|
||||
"""
|
||||
from .providers import (
|
||||
BaseProvider,
|
||||
ChatResponse,
|
||||
ChatConfig,
|
||||
OpenAIProvider,
|
||||
ClaudeProvider,
|
||||
GeminiProvider,
|
||||
PROVIDERS,
|
||||
get_provider,
|
||||
)
|
||||
from .image_utils import (
|
||||
tensor_to_pil,
|
||||
pil_to_tensor,
|
||||
tensor_to_base64,
|
||||
base64_to_tensor,
|
||||
create_data_uri,
|
||||
)
|
||||
from .noass import (
|
||||
format_noass_prompt,
|
||||
build_noass_messages,
|
||||
extract_noass_response,
|
||||
build_full_history,
|
||||
get_stop_sequences,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
# Providers
|
||||
"BaseProvider",
|
||||
"ChatResponse",
|
||||
"ChatConfig",
|
||||
"OpenAIProvider",
|
||||
"ClaudeProvider",
|
||||
"GeminiProvider",
|
||||
"PROVIDERS",
|
||||
"get_provider",
|
||||
# Image utils
|
||||
"tensor_to_pil",
|
||||
"pil_to_tensor",
|
||||
"tensor_to_base64",
|
||||
"base64_to_tensor",
|
||||
"create_data_uri",
|
||||
# NoASS
|
||||
"format_noass_prompt",
|
||||
"build_noass_messages",
|
||||
"extract_noass_response",
|
||||
"build_full_history",
|
||||
"get_stop_sequences",
|
||||
]
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,122 @@
|
||||
"""
|
||||
Image conversion utilities for ComfyUI tensors.
|
||||
"""
|
||||
import base64
|
||||
from io import BytesIO
|
||||
import torch
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
|
||||
def tensor_to_pil(tensor: torch.Tensor) -> Image.Image:
|
||||
"""
|
||||
Convert ComfyUI tensor to PIL Image.
|
||||
|
||||
Args:
|
||||
tensor: Shape (B, H, W, C) or (H, W, C), values 0-1
|
||||
|
||||
Returns:
|
||||
PIL Image (first image if batched)
|
||||
"""
|
||||
if tensor.dim() == 4:
|
||||
tensor = tensor[0] # Take first image from batch
|
||||
|
||||
# Convert to numpy and scale to 0-255
|
||||
arr = (tensor.cpu().numpy() * 255).astype(np.uint8)
|
||||
|
||||
# Handle different channel counts
|
||||
if arr.shape[-1] == 1:
|
||||
arr = arr.squeeze(-1)
|
||||
return Image.fromarray(arr, mode='L')
|
||||
elif arr.shape[-1] == 3:
|
||||
return Image.fromarray(arr, mode='RGB')
|
||||
elif arr.shape[-1] == 4:
|
||||
return Image.fromarray(arr, mode='RGBA')
|
||||
else:
|
||||
raise ValueError(f"Unsupported channel count: {arr.shape[-1]}")
|
||||
|
||||
|
||||
def pil_to_tensor(image: Image.Image) -> torch.Tensor:
|
||||
"""
|
||||
Convert PIL Image to ComfyUI tensor.
|
||||
|
||||
Args:
|
||||
image: PIL Image
|
||||
|
||||
Returns:
|
||||
Tensor shape (1, H, W, C), values 0-1
|
||||
"""
|
||||
# Convert to RGB if needed
|
||||
if image.mode == 'RGBA':
|
||||
pass # Keep alpha
|
||||
elif image.mode != 'RGB':
|
||||
image = image.convert('RGB')
|
||||
|
||||
# Convert to tensor
|
||||
arr = np.array(image).astype(np.float32) / 255.0
|
||||
tensor = torch.from_numpy(arr)
|
||||
|
||||
# Add batch dimension
|
||||
if tensor.dim() == 2: # Grayscale
|
||||
tensor = tensor.unsqueeze(-1)
|
||||
tensor = tensor.unsqueeze(0)
|
||||
|
||||
return tensor
|
||||
|
||||
|
||||
def tensor_to_base64(tensor: torch.Tensor, format: str = "PNG") -> str:
|
||||
"""
|
||||
Convert ComfyUI tensor to base64 string.
|
||||
|
||||
Args:
|
||||
tensor: ComfyUI image tensor
|
||||
format: Image format (PNG, JPEG, WEBP)
|
||||
|
||||
Returns:
|
||||
Base64 encoded string
|
||||
"""
|
||||
pil_image = tensor_to_pil(tensor)
|
||||
|
||||
# Convert RGBA to RGB for JPEG
|
||||
if format.upper() == "JPEG" and pil_image.mode == "RGBA":
|
||||
pil_image = pil_image.convert("RGB")
|
||||
|
||||
buffer = BytesIO()
|
||||
pil_image.save(buffer, format=format)
|
||||
return base64.b64encode(buffer.getvalue()).decode('utf-8')
|
||||
|
||||
|
||||
def base64_to_tensor(b64_string: str) -> torch.Tensor:
|
||||
"""
|
||||
Convert base64 string to ComfyUI tensor.
|
||||
|
||||
Args:
|
||||
b64_string: Base64 encoded image
|
||||
|
||||
Returns:
|
||||
ComfyUI image tensor (1, H, W, C)
|
||||
"""
|
||||
image_data = base64.b64decode(b64_string)
|
||||
image = Image.open(BytesIO(image_data))
|
||||
return pil_to_tensor(image)
|
||||
|
||||
|
||||
def create_data_uri(tensor: torch.Tensor, mime_type: str = "image/png") -> str:
|
||||
"""
|
||||
Create a data URI from tensor for API requests.
|
||||
|
||||
Args:
|
||||
tensor: ComfyUI image tensor
|
||||
mime_type: MIME type (image/png, image/jpeg, image/webp)
|
||||
|
||||
Returns:
|
||||
Data URI string
|
||||
"""
|
||||
format_map = {
|
||||
"image/png": "PNG",
|
||||
"image/jpeg": "JPEG",
|
||||
"image/webp": "WEBP",
|
||||
}
|
||||
format = format_map.get(mime_type, "PNG")
|
||||
b64 = tensor_to_base64(tensor, format)
|
||||
return f"data:{mime_type};base64,{b64}"
|
||||
+174
@@ -0,0 +1,174 @@
|
||||
"""
|
||||
NoASS (No Assistant) format processing for roleplay optimization.
|
||||
|
||||
NoASS transforms traditional chat into a unified narrative where all dialogue
|
||||
is combined into a single assistant message, with character prefixes marking speakers.
|
||||
"""
|
||||
|
||||
|
||||
def format_noass_prompt(
|
||||
system: str,
|
||||
history: str | None,
|
||||
user_input: str,
|
||||
prefill_start: str = "",
|
||||
user_name: str = "User",
|
||||
char_name: str = "Assistant",
|
||||
) -> tuple[str, str]:
|
||||
"""
|
||||
Format conversation in NoASS style.
|
||||
|
||||
Args:
|
||||
system: System prompt / world setting
|
||||
history: Previous conversation history (NoASS format)
|
||||
user_input: Current user input
|
||||
prefill_start: Optional starting text for assistant response
|
||||
user_name: User's character name
|
||||
char_name: AI character name
|
||||
|
||||
Returns:
|
||||
Tuple of (system_prompt, prefilled_content)
|
||||
- system_prompt: Instructions for the model
|
||||
- prefilled_content: The conversation to continue from
|
||||
"""
|
||||
user_prefix = f"**{user_name}:**"
|
||||
char_prefix = f"**{char_name}:**"
|
||||
|
||||
# Build the conversation history structure
|
||||
content_parts = []
|
||||
|
||||
# 1. Add existing history if any
|
||||
if history:
|
||||
content_parts.append(history)
|
||||
|
||||
# 2. Add current user input
|
||||
content_parts.append(f"{user_prefix} {user_input}")
|
||||
|
||||
# 3. Add character prefix to prompt response
|
||||
if prefill_start.strip():
|
||||
# If we have a forced start, include it
|
||||
content_parts.append(f"{char_prefix} {prefill_start}")
|
||||
else:
|
||||
# Otherwise just the prefix
|
||||
content_parts.append(f"{char_prefix}")
|
||||
|
||||
prefilled = "\n\n".join(content_parts)
|
||||
|
||||
return system, prefilled
|
||||
|
||||
|
||||
def build_noass_messages(
|
||||
system: str,
|
||||
history: str | None,
|
||||
user_input: str,
|
||||
prefill_start: str = "",
|
||||
user_name: str = "User",
|
||||
char_name: str = "Assistant",
|
||||
) -> list[dict[str, str]]:
|
||||
"""
|
||||
Build message list for NoASS format using proper Assistant Prefill where possible.
|
||||
|
||||
Structure:
|
||||
[
|
||||
{"role": "user", "content": system}, # System/Scenario as first user message
|
||||
{"role": "assistant", "content": [History] + [User Input] + [Prefill]} # The entire story so far
|
||||
]
|
||||
|
||||
The API is expected to continue directly from the end of the assistant message.
|
||||
"""
|
||||
messages = []
|
||||
|
||||
# 1. System/Scenario (as User message to ensure it's read as instructions)
|
||||
# Using 'system' role is also fine but 'user' often works better for "Instruction" in NoASS context
|
||||
# However, standard practice is still system role for actual instructions.
|
||||
# Let's stick to 'user' for the "Scenario Description" as requested by the user's mental model.
|
||||
if system:
|
||||
messages.append({"role": "user", "content": system})
|
||||
else:
|
||||
# Fallback if system is empty
|
||||
messages.append({"role": "user", "content": "Narrative Roleplay."})
|
||||
|
||||
# 2. Build the giant Assistant Prefill block
|
||||
user_prefix = f"**{user_name}:**"
|
||||
char_prefix = f"**{char_name}:**"
|
||||
|
||||
story_blocks = []
|
||||
|
||||
if history:
|
||||
story_blocks.append(history)
|
||||
|
||||
# Append current user action
|
||||
story_blocks.append(f"{user_prefix} {user_input}")
|
||||
|
||||
# Append the start of the assistant's turn
|
||||
if prefill_start:
|
||||
story_blocks.append(f"{char_prefix} {prefill_start}")
|
||||
else:
|
||||
# Just the prefix to cue the AI
|
||||
story_blocks.append(f"{char_prefix}")
|
||||
|
||||
full_prefill = "\n\n".join(story_blocks)
|
||||
|
||||
messages.append({
|
||||
"role": "assistant",
|
||||
"content": full_prefill
|
||||
})
|
||||
|
||||
return messages
|
||||
|
||||
|
||||
def extract_noass_response(
|
||||
response_text: str,
|
||||
user_name: str = "User",
|
||||
char_name: str = "Assistant",
|
||||
) -> str:
|
||||
"""
|
||||
Extract the character's response from NoASS output.
|
||||
|
||||
Since we are using prefill, the API might return JUST the continuation,
|
||||
or (depending on provider behavior) might repeat some text.
|
||||
Standard OpenAI/Claude behavior for prefill is to return just the NEW tokens.
|
||||
"""
|
||||
user_prefix = f"**{user_name}:**"
|
||||
|
||||
# If response contains user prefix (hallucinated next turn), truncate there
|
||||
if user_prefix in response_text:
|
||||
response_text = response_text.split(user_prefix)[0]
|
||||
|
||||
return response_text.strip()
|
||||
|
||||
|
||||
def build_full_history(
|
||||
history: str | None,
|
||||
user_input: str,
|
||||
prefill_start: str,
|
||||
response: str,
|
||||
user_name: str = "User",
|
||||
char_name: str = "Assistant",
|
||||
) -> str:
|
||||
"""
|
||||
Build complete conversation history after a response.
|
||||
Reconstructs the full text including the prefill and the new response.
|
||||
"""
|
||||
user_prefix = f"**{user_name}:**"
|
||||
char_prefix = f"**{char_name}:**"
|
||||
|
||||
# Construct the assistant's full turn
|
||||
if prefill_start:
|
||||
full_response = f"{char_prefix} {prefill_start} {response}"
|
||||
else:
|
||||
full_response = f"{char_prefix} {response}"
|
||||
|
||||
# Append to history
|
||||
if history:
|
||||
return f"{history}\n\n{user_prefix} {user_input}\n\n{full_response}"
|
||||
else:
|
||||
return f"{user_prefix} {user_input}\n\n{full_response}"
|
||||
|
||||
|
||||
def get_stop_sequences(user_name: str = "User") -> list[str]:
|
||||
"""
|
||||
Get stop sequences for NoASS mode.
|
||||
The model should stop when it tries to write for the user.
|
||||
"""
|
||||
user_prefix = f"**{user_name}:**"
|
||||
return [user_prefix, f"\n{user_prefix}", f"\n\n{user_prefix}"]
|
||||
@@ -0,0 +1,34 @@
|
||||
"""
|
||||
Provider implementations for SimpleChat.
|
||||
"""
|
||||
from .base import BaseProvider, ChatResponse, ChatConfig
|
||||
from .openai import OpenAIProvider
|
||||
from .claude import ClaudeProvider
|
||||
from .gemini import GeminiProvider
|
||||
|
||||
|
||||
PROVIDERS = {
|
||||
"openai": OpenAIProvider,
|
||||
"claude": ClaudeProvider,
|
||||
"gemini": GeminiProvider,
|
||||
}
|
||||
|
||||
|
||||
def get_provider(config: ChatConfig) -> BaseProvider:
|
||||
"""Get provider instance from config."""
|
||||
provider_cls = PROVIDERS.get(config.provider)
|
||||
if not provider_cls:
|
||||
raise ValueError(f"Unknown provider: {config.provider}")
|
||||
return provider_cls(api_key=config.api_key, base_url=config.base_url or None)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BaseProvider",
|
||||
"ChatResponse",
|
||||
"ChatConfig",
|
||||
"OpenAIProvider",
|
||||
"ClaudeProvider",
|
||||
"GeminiProvider",
|
||||
"PROVIDERS",
|
||||
"get_provider",
|
||||
]
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,72 @@
|
||||
"""
|
||||
Base provider class for LLM API integrations.
|
||||
"""
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
import torch
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatResponse:
|
||||
"""Unified response from any LLM provider."""
|
||||
text: str
|
||||
image: torch.Tensor | None = None
|
||||
raw_response: dict | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatConfig:
|
||||
"""Configuration for API connection."""
|
||||
provider: str
|
||||
api_key: str
|
||||
base_url: str
|
||||
model: str
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"provider": self.provider,
|
||||
"api_key": self.api_key,
|
||||
"base_url": self.base_url,
|
||||
"model": self.model,
|
||||
}
|
||||
|
||||
|
||||
class BaseProvider(ABC):
|
||||
"""Abstract base class for LLM providers."""
|
||||
|
||||
def __init__(self, api_key: str, base_url: str | None = None):
|
||||
self.api_key = api_key
|
||||
self.base_url = base_url or self.default_base_url
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def default_base_url(self) -> str:
|
||||
"""Default API endpoint for this provider."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def chat(
|
||||
self,
|
||||
messages: list[dict[str, Any]],
|
||||
model: str,
|
||||
temperature: float = 1.0,
|
||||
max_tokens: int = 2048,
|
||||
images: list[torch.Tensor] | None = None,
|
||||
**kwargs,
|
||||
) -> ChatResponse:
|
||||
"""Send a chat request to the API."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def generate_image(
|
||||
self,
|
||||
prompt: str,
|
||||
model: str,
|
||||
reference_image: torch.Tensor | None = None,
|
||||
aspect_ratio: str = "1:1",
|
||||
size: str = "1K",
|
||||
**kwargs,
|
||||
) -> ChatResponse:
|
||||
"""Generate or edit an image. Only supported by some providers."""
|
||||
pass
|
||||
@@ -0,0 +1,121 @@
|
||||
"""
|
||||
Claude/Anthropic provider implementation.
|
||||
"""
|
||||
import aiohttp
|
||||
from typing import Any
|
||||
import torch
|
||||
|
||||
from .base import BaseProvider, ChatResponse
|
||||
from ..image_utils import tensor_to_base64, create_data_uri
|
||||
|
||||
|
||||
class ClaudeProvider(BaseProvider):
|
||||
"""Anthropic Claude API provider."""
|
||||
|
||||
@property
|
||||
def default_base_url(self) -> str:
|
||||
return "https://api.anthropic.com/v1"
|
||||
|
||||
def _build_messages(
|
||||
self,
|
||||
messages: list[dict[str, Any]],
|
||||
images: list[torch.Tensor] | None = None,
|
||||
) -> tuple[str | None, list[dict[str, Any]]]:
|
||||
"""
|
||||
Build Claude-format messages with optional images.
|
||||
|
||||
Returns:
|
||||
Tuple of (system_prompt, messages)
|
||||
"""
|
||||
system = None
|
||||
result = []
|
||||
|
||||
for i, msg in enumerate(messages):
|
||||
role = msg["role"]
|
||||
content = msg["content"]
|
||||
|
||||
# Extract system message
|
||||
if role == "system":
|
||||
system = content
|
||||
continue
|
||||
|
||||
# Handle user message with images
|
||||
if role == "user" and images and i == len(messages) - 1:
|
||||
content_parts = []
|
||||
for img in images:
|
||||
b64 = tensor_to_base64(img)
|
||||
content_parts.append({
|
||||
"type": "image",
|
||||
"source": {
|
||||
"type": "base64",
|
||||
"media_type": "image/png",
|
||||
"data": b64,
|
||||
}
|
||||
})
|
||||
content_parts.append({"type": "text", "text": content})
|
||||
result.append({"role": "user", "content": content_parts})
|
||||
else:
|
||||
result.append({"role": role, "content": content})
|
||||
|
||||
return system, result
|
||||
|
||||
async def chat(
|
||||
self,
|
||||
messages: list[dict[str, Any]],
|
||||
model: str,
|
||||
temperature: float = 1.0,
|
||||
max_tokens: int = 2048,
|
||||
images: list[torch.Tensor] | None = None,
|
||||
**kwargs,
|
||||
) -> ChatResponse:
|
||||
"""Send chat request to Claude API."""
|
||||
|
||||
url = f"{self.base_url}/messages"
|
||||
headers = {
|
||||
"x-api-key": self.api_key,
|
||||
"anthropic-version": "2023-06-01",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
system, claude_messages = self._build_messages(messages, images)
|
||||
|
||||
payload = {
|
||||
"model": model,
|
||||
"messages": claude_messages,
|
||||
"temperature": temperature,
|
||||
"max_tokens": max_tokens,
|
||||
}
|
||||
|
||||
if system:
|
||||
payload["system"] = system
|
||||
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(url, headers=headers, json=payload) as resp:
|
||||
if resp.status != 200:
|
||||
error_text = await resp.text()
|
||||
raise RuntimeError(f"Claude API error {resp.status}: {error_text}")
|
||||
|
||||
data = await resp.json()
|
||||
|
||||
# Extract text from response
|
||||
text = ""
|
||||
for block in data.get("content", []):
|
||||
if block.get("type") == "text":
|
||||
text += block.get("text", "")
|
||||
|
||||
return ChatResponse(text=text, raw_response=data)
|
||||
|
||||
async def generate_image(
|
||||
self,
|
||||
prompt: str,
|
||||
model: str,
|
||||
reference_image: torch.Tensor | None = None,
|
||||
aspect_ratio: str = "1:1",
|
||||
size: str = "1K",
|
||||
**kwargs,
|
||||
) -> ChatResponse:
|
||||
"""Claude doesn't support image generation."""
|
||||
raise NotImplementedError(
|
||||
"Claude does not support image generation. "
|
||||
"Use Gemini provider for image generation."
|
||||
)
|
||||
@@ -0,0 +1,199 @@
|
||||
"""
|
||||
Google Gemini provider implementation.
|
||||
Supports chat and image generation (Nano Banana / Nano Banana Pro).
|
||||
"""
|
||||
import aiohttp
|
||||
from typing import Any
|
||||
import torch
|
||||
|
||||
from .base import BaseProvider, ChatResponse
|
||||
from ..image_utils import tensor_to_base64, base64_to_tensor
|
||||
|
||||
|
||||
class GeminiProvider(BaseProvider):
|
||||
"""Google Gemini API provider with image generation support."""
|
||||
|
||||
@property
|
||||
def default_base_url(self) -> str:
|
||||
return "https://generativelanguage.googleapis.com/v1beta"
|
||||
|
||||
def _build_contents(
|
||||
self,
|
||||
messages: list[dict[str, Any]],
|
||||
images: list[torch.Tensor] | None = None,
|
||||
) -> tuple[str | None, list[dict[str, Any]]]:
|
||||
"""
|
||||
Build Gemini-format contents.
|
||||
|
||||
Returns:
|
||||
Tuple of (system_instruction, contents)
|
||||
"""
|
||||
system = None
|
||||
contents = []
|
||||
|
||||
for i, msg in enumerate(messages):
|
||||
role = msg["role"]
|
||||
content = msg["content"]
|
||||
|
||||
# Extract system message
|
||||
if role == "system":
|
||||
system = content
|
||||
continue
|
||||
|
||||
# Map roles
|
||||
gemini_role = "user" if role == "user" else "model"
|
||||
|
||||
# Build parts
|
||||
parts = []
|
||||
|
||||
# Add images to last user message
|
||||
if role == "user" and images and i == len(messages) - 1:
|
||||
for img in images:
|
||||
b64 = tensor_to_base64(img)
|
||||
parts.append({
|
||||
"inline_data": {
|
||||
"mime_type": "image/png",
|
||||
"data": b64,
|
||||
}
|
||||
})
|
||||
|
||||
parts.append({"text": content})
|
||||
contents.append({"role": gemini_role, "parts": parts})
|
||||
|
||||
return system, contents
|
||||
|
||||
async def chat(
|
||||
self,
|
||||
messages: list[dict[str, Any]],
|
||||
model: str,
|
||||
temperature: float = 1.0,
|
||||
max_tokens: int = 2048,
|
||||
images: list[torch.Tensor] | None = None,
|
||||
enable_image_generation: bool = False,
|
||||
**kwargs,
|
||||
) -> ChatResponse:
|
||||
"""Send chat request to Gemini API."""
|
||||
|
||||
url = f"{self.base_url}/models/{model}:generateContent"
|
||||
|
||||
# Add API key as query parameter
|
||||
if "?" in url:
|
||||
url += f"&key={self.api_key}"
|
||||
else:
|
||||
url += f"?key={self.api_key}"
|
||||
|
||||
headers = {"Content-Type": "application/json"}
|
||||
|
||||
system, contents = self._build_contents(messages, images)
|
||||
|
||||
payload = {
|
||||
"contents": contents,
|
||||
"generationConfig": {
|
||||
"temperature": temperature,
|
||||
"maxOutputTokens": max_tokens,
|
||||
}
|
||||
}
|
||||
|
||||
if system:
|
||||
payload["systemInstruction"] = {"parts": [{"text": system}]}
|
||||
|
||||
# Enable image generation if requested
|
||||
if enable_image_generation:
|
||||
payload["generationConfig"]["responseModalities"] = ["TEXT", "IMAGE"]
|
||||
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(url, headers=headers, json=payload) as resp:
|
||||
if resp.status != 200:
|
||||
error_text = await resp.text()
|
||||
raise RuntimeError(f"Gemini API error {resp.status}: {error_text}")
|
||||
|
||||
data = await resp.json()
|
||||
|
||||
# Extract text and images from response
|
||||
text = ""
|
||||
result_image = None
|
||||
|
||||
candidates = data.get("candidates", [])
|
||||
if candidates:
|
||||
parts = candidates[0].get("content", {}).get("parts", [])
|
||||
for part in parts:
|
||||
if "text" in part:
|
||||
text += part["text"]
|
||||
elif "inlineData" in part:
|
||||
inline = part["inlineData"]
|
||||
if inline.get("mimeType", "").startswith("image/"):
|
||||
result_image = base64_to_tensor(inline["data"])
|
||||
|
||||
return ChatResponse(text=text, image=result_image, raw_response=data)
|
||||
|
||||
async def generate_image(
|
||||
self,
|
||||
prompt: str,
|
||||
model: str,
|
||||
reference_image: torch.Tensor | None = None,
|
||||
aspect_ratio: str = "1:1",
|
||||
size: str = "1K",
|
||||
**kwargs,
|
||||
) -> ChatResponse:
|
||||
"""Generate or edit image using Gemini."""
|
||||
|
||||
url = f"{self.base_url}/models/{model}:generateContent?key={self.api_key}"
|
||||
headers = {"Content-Type": "application/json"}
|
||||
|
||||
# Build parts
|
||||
parts = []
|
||||
|
||||
# Add reference image if provided
|
||||
if reference_image is not None:
|
||||
b64 = tensor_to_base64(reference_image)
|
||||
parts.append({
|
||||
"inline_data": {
|
||||
"mime_type": "image/png",
|
||||
"data": b64,
|
||||
}
|
||||
})
|
||||
|
||||
parts.append({"text": prompt})
|
||||
|
||||
payload = {
|
||||
"contents": [{"parts": parts}],
|
||||
"generationConfig": {
|
||||
"responseModalities": ["TEXT", "IMAGE"],
|
||||
}
|
||||
}
|
||||
|
||||
# Add image config if supported
|
||||
if aspect_ratio or size:
|
||||
payload["generationConfig"]["imageConfig"] = {}
|
||||
if aspect_ratio:
|
||||
payload["generationConfig"]["imageConfig"]["aspectRatio"] = aspect_ratio
|
||||
if size:
|
||||
payload["generationConfig"]["imageConfig"]["imageSize"] = size
|
||||
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(url, headers=headers, json=payload) as resp:
|
||||
if resp.status != 200:
|
||||
error_text = await resp.text()
|
||||
raise RuntimeError(f"Gemini API error {resp.status}: {error_text}")
|
||||
|
||||
data = await resp.json()
|
||||
|
||||
# Extract text and image
|
||||
text = ""
|
||||
result_image = None
|
||||
|
||||
candidates = data.get("candidates", [])
|
||||
if candidates:
|
||||
parts = candidates[0].get("content", {}).get("parts", [])
|
||||
for part in parts:
|
||||
if "text" in part:
|
||||
text += part["text"]
|
||||
elif "inlineData" in part:
|
||||
inline = part["inlineData"]
|
||||
if inline.get("mimeType", "").startswith("image/"):
|
||||
result_image = base64_to_tensor(inline["data"])
|
||||
|
||||
if result_image is None:
|
||||
raise RuntimeError("Gemini did not return an image. Try a different prompt or model.")
|
||||
|
||||
return ChatResponse(text=text, image=result_image, raw_response=data)
|
||||
@@ -0,0 +1,94 @@
|
||||
"""
|
||||
OpenAI provider implementation.
|
||||
Also works with OpenAI-compatible APIs (e.g., local LLMs, other providers).
|
||||
"""
|
||||
import aiohttp
|
||||
from typing import Any
|
||||
import torch
|
||||
|
||||
from .base import BaseProvider, ChatResponse
|
||||
from ..image_utils import tensor_to_base64, base64_to_tensor, create_data_uri
|
||||
|
||||
|
||||
class OpenAIProvider(BaseProvider):
|
||||
"""OpenAI API provider (and compatible APIs)."""
|
||||
|
||||
@property
|
||||
def default_base_url(self) -> str:
|
||||
return "https://api.openai.com/v1"
|
||||
|
||||
def _build_messages(
|
||||
self,
|
||||
messages: list[dict[str, Any]],
|
||||
images: list[torch.Tensor] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Build OpenAI-format messages with optional images."""
|
||||
if not images:
|
||||
return messages
|
||||
|
||||
# Find the last user message and add images to it
|
||||
result = []
|
||||
for i, msg in enumerate(messages):
|
||||
if msg["role"] == "user" and i == len(messages) - 1:
|
||||
# Convert content to multimodal format
|
||||
content = [{"type": "text", "text": msg["content"]}]
|
||||
for img in images:
|
||||
content.append({
|
||||
"type": "image_url",
|
||||
"image_url": {"url": create_data_uri(img)}
|
||||
})
|
||||
result.append({"role": "user", "content": content})
|
||||
else:
|
||||
result.append(msg)
|
||||
|
||||
return result
|
||||
|
||||
async def chat(
|
||||
self,
|
||||
messages: list[dict[str, Any]],
|
||||
model: str,
|
||||
temperature: float = 1.0,
|
||||
max_tokens: int = 2048,
|
||||
images: list[torch.Tensor] | None = None,
|
||||
**kwargs,
|
||||
) -> ChatResponse:
|
||||
"""Send chat request to OpenAI API."""
|
||||
|
||||
url = f"{self.base_url}/chat/completions"
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.api_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
payload = {
|
||||
"model": model,
|
||||
"messages": self._build_messages(messages, images),
|
||||
"temperature": temperature,
|
||||
"max_tokens": max_tokens,
|
||||
}
|
||||
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(url, headers=headers, json=payload) as resp:
|
||||
if resp.status != 200:
|
||||
error_text = await resp.text()
|
||||
raise RuntimeError(f"OpenAI API error {resp.status}: {error_text}")
|
||||
|
||||
data = await resp.json()
|
||||
|
||||
text = data["choices"][0]["message"]["content"]
|
||||
return ChatResponse(text=text, raw_response=data)
|
||||
|
||||
async def generate_image(
|
||||
self,
|
||||
prompt: str,
|
||||
model: str,
|
||||
reference_image: torch.Tensor | None = None,
|
||||
aspect_ratio: str = "1:1",
|
||||
size: str = "1K",
|
||||
**kwargs,
|
||||
) -> ChatResponse:
|
||||
"""OpenAI doesn't support native image generation in chat. Use DALL-E nodes instead."""
|
||||
raise NotImplementedError(
|
||||
"OpenAI image generation is not supported in SimpleChat. "
|
||||
"Use dedicated DALL-E nodes or Gemini provider for image generation."
|
||||
)
|
||||
@@ -0,0 +1,312 @@
|
||||
# ComfyUI-SimpleChat Architecture Design
|
||||
|
||||
## Design Philosophy
|
||||
|
||||
**Do one thing, do best** - 每个节点只做一件事,做到极致。
|
||||
|
||||
## 节点设计原则
|
||||
|
||||
1. **单一职责**: 每个节点只负责一个功能
|
||||
2. **清晰命名**: 节点名称直接表达功能
|
||||
3. **最少配置**: 合理默认值,减少必填项
|
||||
4. **美观布局**: 输入输出顺序符合直觉
|
||||
|
||||
---
|
||||
|
||||
## 节点清单
|
||||
|
||||
### 配置节点
|
||||
|
||||
| 节点 | 功能 | 设计理念 |
|
||||
|------|------|----------|
|
||||
| `API Config` | 配置 API 连接 | 一次配置,到处使用 |
|
||||
|
||||
### 对话节点
|
||||
|
||||
| 节点 | 功能 | 设计理念 |
|
||||
|------|------|----------|
|
||||
| `Chat` | 基础对话 | 最简单的文本对话 |
|
||||
| `Chat with Image` | 发送图片给 LLM | 视觉理解 |
|
||||
| `Chat NoASS` | NoASS 格式对话 | 角色扮演优化 |
|
||||
|
||||
### 图片生成节点 (Gemini Only)
|
||||
|
||||
| 节点 | 功能 | 设计理念 |
|
||||
|------|------|----------|
|
||||
| `Gemini Image Gen` | 文生图 | 纯文本生成图片 |
|
||||
| `Gemini Image Edit` | 图片编辑 | 基于参考图修改 |
|
||||
|
||||
---
|
||||
|
||||
## 节点详细设计
|
||||
|
||||
### 1. API Config
|
||||
|
||||
**功能**: 配置 API 连接参数
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ 🔧 API Config │
|
||||
├─────────────────────────────────────┤
|
||||
│ provider [openai ▼] │
|
||||
│ api_key [••••••••••] │
|
||||
│ base_url [ ] │
|
||||
│ model [gpt-4o ▼] │
|
||||
├─────────────────────────────────────┤
|
||||
│ CONFIG ○───│
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**输入:**
|
||||
- `provider`: 下拉 ["openai", "claude", "gemini"]
|
||||
- `api_key`: 密码输入
|
||||
- `base_url`: 文本 (可选,留空使用默认)
|
||||
- `model`: 可编辑下拉
|
||||
|
||||
**输出:**
|
||||
- `CONFIG`: 配置对象
|
||||
|
||||
---
|
||||
|
||||
### 2. Chat
|
||||
|
||||
**功能**: 基础文本对话
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ 💬 Chat │
|
||||
├─────────────────────────────────────┤
|
||||
│ ───○ config │
|
||||
├─────────────────────────────────────┤
|
||||
│ prompt [ ] │
|
||||
│ [ ] │
|
||||
│ system [ ] │
|
||||
│ temperature [1.0 ] │
|
||||
│ max_tokens [2048 ] │
|
||||
├─────────────────────────────────────┤
|
||||
│ text ○───│
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**输入:**
|
||||
- `config`: CONFIG (必需)
|
||||
- `prompt`: 多行文本
|
||||
- `system`: 系统提示 (可选)
|
||||
- `temperature`: 0-2, 默认 1.0
|
||||
- `max_tokens`: 默认 2048
|
||||
|
||||
**输出:**
|
||||
- `text`: STRING
|
||||
|
||||
---
|
||||
|
||||
### 3. Chat with Image
|
||||
|
||||
**功能**: 发送图片给 LLM 分析
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ 🖼️ Chat with Image │
|
||||
├─────────────────────────────────────┤
|
||||
│ ───○ config │
|
||||
│ ───○ image │
|
||||
├─────────────────────────────────────┤
|
||||
│ prompt [Describe this image ] │
|
||||
│ [ ] │
|
||||
│ system [ ] │
|
||||
│ temperature [1.0 ] │
|
||||
│ max_tokens [2048 ] │
|
||||
├─────────────────────────────────────┤
|
||||
│ text ○───│
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**输入:**
|
||||
- `config`: CONFIG
|
||||
- `image`: IMAGE
|
||||
- `prompt`: 多行文本
|
||||
- `system`: 系统提示 (可选)
|
||||
- `temperature`: 0-2
|
||||
- `max_tokens`: 默认 2048
|
||||
|
||||
**输出:**
|
||||
- `text`: STRING
|
||||
|
||||
---
|
||||
|
||||
### 4. Chat NoASS
|
||||
|
||||
**功能**: NoASS 格式的角色扮演对话
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ 🎭 Chat NoASS │
|
||||
├─────────────────────────────────────┤
|
||||
│ ───○ config │
|
||||
│ ───○ history (optional) │
|
||||
│ ───○ image (optional) │
|
||||
├─────────────────────────────────────┤
|
||||
│ system [You are... ] │
|
||||
│ [ ] │
|
||||
│ user_input [ ] │
|
||||
│ user_name [User ] │
|
||||
│ char_name [Assistant] │
|
||||
│ temperature [1.0 ] │
|
||||
│ max_tokens [2048 ] │
|
||||
├─────────────────────────────────────┤
|
||||
│ text ○───│
|
||||
│ history ○───│
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**输入:**
|
||||
- `config`: CONFIG
|
||||
- `history`: STRING (可选,上一轮输出)
|
||||
- `image`: IMAGE (可选)
|
||||
- `system`: 系统提示/世界观设定
|
||||
- `user_input`: 用户本轮输入
|
||||
- `user_name`: 用户角色名
|
||||
- `char_name`: AI 角色名
|
||||
- `temperature`: 0-2
|
||||
- `max_tokens`: 默认 2048
|
||||
|
||||
**输出:**
|
||||
- `text`: STRING (本轮 AI 回复)
|
||||
- `history`: STRING (完整对话历史,可连接到下一个节点的 history 输入)
|
||||
|
||||
---
|
||||
|
||||
### 5. Gemini Image Gen
|
||||
|
||||
**功能**: Gemini 文生图
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ 🎨 Gemini Image Gen │
|
||||
├─────────────────────────────────────┤
|
||||
│ ───○ config (Gemini only) │
|
||||
├─────────────────────────────────────┤
|
||||
│ prompt [A cat wearing a hat ] │
|
||||
│ [ ] │
|
||||
│ aspect [1:1 ▼] │
|
||||
│ size [1K ▼] │
|
||||
├─────────────────────────────────────┤
|
||||
│ IMAGE ○───│
|
||||
│ text ○───│
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**输入:**
|
||||
- `config`: CONFIG (必须是 Gemini provider)
|
||||
- `prompt`: 多行文本
|
||||
- `aspect`: 下拉 ["1:1", "16:9", "9:16", "4:3", "3:4"]
|
||||
- `size`: 下拉 ["1K", "2K", "4K"]
|
||||
|
||||
**输出:**
|
||||
- `IMAGE`: IMAGE
|
||||
- `text`: STRING (Gemini 的文字描述)
|
||||
|
||||
---
|
||||
|
||||
### 6. Gemini Image Edit
|
||||
|
||||
**功能**: Gemini 图片编辑
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ ✏️ Gemini Image Edit │
|
||||
├─────────────────────────────────────┤
|
||||
│ ───○ config (Gemini only) │
|
||||
│ ───○ image │
|
||||
├─────────────────────────────────────┤
|
||||
│ prompt [Add a red hat ] │
|
||||
│ [ ] │
|
||||
│ size [1K ▼] │
|
||||
├─────────────────────────────────────┤
|
||||
│ IMAGE ○───│
|
||||
│ text ○───│
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**输入:**
|
||||
- `config`: CONFIG (必须是 Gemini provider)
|
||||
- `image`: IMAGE (要编辑的图片)
|
||||
- `prompt`: 编辑指令
|
||||
- `size`: 下拉 ["1K", "2K", "4K"]
|
||||
|
||||
**输出:**
|
||||
- `IMAGE`: IMAGE
|
||||
- `text`: STRING
|
||||
|
||||
---
|
||||
|
||||
## 数据流示例
|
||||
|
||||
### 基础对话
|
||||
|
||||
```
|
||||
[API Config] ──CONFIG──> [Chat] ──text──> [Display Text]
|
||||
```
|
||||
|
||||
### 图片分析
|
||||
|
||||
```
|
||||
[Load Image] ──IMAGE──┐
|
||||
├──> [Chat with Image] ──text──> [Display Text]
|
||||
[API Config] ──CONFIG─┘
|
||||
```
|
||||
|
||||
### 多轮 NoASS 对话
|
||||
|
||||
```
|
||||
[API Config] ──CONFIG──> [Chat NoASS] ──history──> [Chat NoASS] ──history──> ...
|
||||
│ │
|
||||
└──text──> └──text──>
|
||||
```
|
||||
|
||||
### Gemini 图片生成
|
||||
|
||||
```
|
||||
[API Config] ──CONFIG──> [Gemini Image Gen] ──IMAGE──> [Preview Image]
|
||||
│ │
|
||||
│ ──text──> [Display Text]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 文件结构
|
||||
|
||||
```
|
||||
custom_nodes/ComfyUI-SimpleChat/
|
||||
├── __init__.py # 节点注册
|
||||
├── nodes/
|
||||
│ ├── __init__.py
|
||||
│ ├── config.py # API Config
|
||||
│ ├── chat.py # Chat
|
||||
│ ├── chat_image.py # Chat with Image
|
||||
│ ├── chat_noass.py # Chat NoASS
|
||||
│ ├── gemini_gen.py # Gemini Image Gen
|
||||
│ └── gemini_edit.py # Gemini Image Edit
|
||||
├── core/
|
||||
│ ├── __init__.py
|
||||
│ ├── providers/
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── base.py # Provider 抽象基类
|
||||
│ │ ├── openai.py # OpenAI (+ 兼容 API)
|
||||
│ │ ├── claude.py # Claude/Anthropic
|
||||
│ │ └── gemini.py # Google Gemini
|
||||
│ ├── noass.py # NoASS 格式处理
|
||||
│ └── image_utils.py # 图片转换
|
||||
├── docs/ # 文档
|
||||
├── requirements.txt
|
||||
└── README.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 实现优先级
|
||||
|
||||
1. **Phase 1**: API Config + Chat (最小可用)
|
||||
2. **Phase 2**: Chat with Image
|
||||
3. **Phase 3**: Chat NoASS
|
||||
4. **Phase 4**: Gemini Image Gen/Edit
|
||||
@@ -0,0 +1,153 @@
|
||||
# ComfyUI API Nodes Patterns Research
|
||||
|
||||
## 目录结构
|
||||
|
||||
ComfyUI 官方 API 节点位于 `/comfy_api_nodes/`:
|
||||
|
||||
```
|
||||
comfy_api_nodes/
|
||||
├── apis/ # Pydantic request/response models
|
||||
├── util/ # 共享工具
|
||||
│ ├── client.py # HTTP 客户端 (sync_op, poll_op)
|
||||
│ ├── _helpers.py # 认证, base URL
|
||||
│ ├── conversions.py # 图片/音频/视频转换
|
||||
│ └── ...
|
||||
├── nodes_openai.py # OpenAI 节点
|
||||
├── nodes_gemini.py # Gemini 节点
|
||||
└── ...
|
||||
```
|
||||
|
||||
## 节点定义模式
|
||||
|
||||
### 新版 API 节点模式 (IO.ComfyNode)
|
||||
|
||||
```python
|
||||
class MyApiNode(IO.ComfyNode):
|
||||
@classmethod
|
||||
def define_schema(cls):
|
||||
return IO.Schema(
|
||||
node_id="MyApiNode",
|
||||
display_name="My API Node",
|
||||
category="api node/category",
|
||||
inputs=[
|
||||
IO.String.Input("prompt", multiline=True),
|
||||
IO.Image.Input("image", optional=True),
|
||||
],
|
||||
outputs=[
|
||||
IO.Image.Output("IMAGE"),
|
||||
IO.String.Output("TEXT"),
|
||||
],
|
||||
hidden=[
|
||||
IO.Hidden.auth_token_comfy_org,
|
||||
IO.Hidden.api_key_comfy_org,
|
||||
],
|
||||
is_api_node=True,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
async def execute(cls, prompt: str, image=None):
|
||||
# Implementation
|
||||
return IO.NodeOutput(result_image, result_text)
|
||||
```
|
||||
|
||||
### 传统节点模式 (ComfyNodeABC)
|
||||
|
||||
```python
|
||||
class MyNode(ComfyNodeABC):
|
||||
@classmethod
|
||||
def INPUT_TYPES(s) -> InputTypeDict:
|
||||
return {
|
||||
"required": {
|
||||
"prompt": ("STRING", {"multiline": True}),
|
||||
},
|
||||
"optional": {
|
||||
"image": ("IMAGE",),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "STRING")
|
||||
RETURN_NAMES = ("image", "text")
|
||||
FUNCTION = "execute"
|
||||
CATEGORY = "my_category"
|
||||
|
||||
def execute(self, prompt, image=None):
|
||||
return (result_image, result_text)
|
||||
```
|
||||
|
||||
## 图片处理
|
||||
|
||||
### Tensor 转 Base64
|
||||
|
||||
```python
|
||||
from comfy_api_nodes.util import tensor_to_base64_string
|
||||
|
||||
base64_str = tensor_to_base64_string(image_tensor)
|
||||
data_uri = f"data:image/png;base64,{base64_str}"
|
||||
```
|
||||
|
||||
### Base64 转 Tensor
|
||||
|
||||
```python
|
||||
import base64
|
||||
from io import BytesIO
|
||||
from comfy_api_nodes.util import bytesio_to_image_tensor
|
||||
|
||||
image_data = base64.b64decode(base64_string)
|
||||
tensor = bytesio_to_image_tensor(BytesIO(image_data))
|
||||
```
|
||||
|
||||
### Tensor 格式
|
||||
|
||||
ComfyUI 图片 tensor: `(B, H, W, C)` - Batch, Height, Width, Channels (RGBA)
|
||||
|
||||
## HTTP 请求
|
||||
|
||||
### 同步请求
|
||||
|
||||
```python
|
||||
from comfy_api_nodes.util import sync_op
|
||||
|
||||
response = await sync_op(
|
||||
cls,
|
||||
ApiEndpoint(path="/api/endpoint", method="POST"),
|
||||
response_model=ResponseModel,
|
||||
data=RequestModel(...),
|
||||
timeout=3600.0,
|
||||
)
|
||||
```
|
||||
|
||||
### 轮询请求
|
||||
|
||||
```python
|
||||
from comfy_api_nodes.util import poll_op
|
||||
|
||||
response = await poll_op(
|
||||
cls,
|
||||
ApiEndpoint(path="/api/status/{id}", method="GET"),
|
||||
response_model=ResponseModel,
|
||||
status_extractor=lambda r: r.status,
|
||||
completed_statuses=["completed"],
|
||||
failed_statuses=["failed"],
|
||||
)
|
||||
```
|
||||
|
||||
## 认证
|
||||
|
||||
官方节点使用 ComfyUI 代理认证。我们的自定义节点将支持:
|
||||
1. 直接 API Key 输入
|
||||
2. 自定义 base_url
|
||||
|
||||
## 重要发现
|
||||
|
||||
1. **无 Claude/Anthropic 集成** - 官方没有 Claude 节点
|
||||
2. **代理模式** - 官方节点通过 ComfyUI 代理访问 API
|
||||
3. **异步执行** - 所有 API 调用使用 async/await
|
||||
4. **Pydantic 验证** - 请求/响应使用 Pydantic 模型
|
||||
|
||||
## 对我们的启示
|
||||
|
||||
由于我们要支持自定义 base_url 和直接 API 访问,我们将:
|
||||
1. 使用传统节点模式(更简单,更灵活)
|
||||
2. 直接使用 aiohttp/httpx 发请求
|
||||
3. 手动处理认证
|
||||
4. 复用 ComfyUI 的图片转换工具
|
||||
@@ -0,0 +1,127 @@
|
||||
# Gemini Image Generation Research Notes
|
||||
|
||||
## Available Models
|
||||
|
||||
| Model | Use Case | Features |
|
||||
|-------|----------|----------|
|
||||
| `gemini-2.5-flash-image` | Fast, general image generation | Lower cost, faster |
|
||||
| `gemini-3-pro-image-preview` | Professional asset production | Higher quality, 1K/2K/4K output |
|
||||
|
||||
## API Endpoint
|
||||
|
||||
```
|
||||
POST https://generativelanguage.googleapis.com/v1beta/models/{model-id}:generateContent
|
||||
```
|
||||
|
||||
## Request Format
|
||||
|
||||
### 关键:启用图像生成
|
||||
|
||||
必须在 `generationConfig` 中指定 `responseModalities`:
|
||||
|
||||
```json
|
||||
{
|
||||
"contents": [{
|
||||
"parts": [
|
||||
{"text": "Generate an image of a cat"}
|
||||
]
|
||||
}],
|
||||
"generationConfig": {
|
||||
"responseModalities": ["TEXT", "IMAGE"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 带图片输入的请求(图片编辑/参考)
|
||||
|
||||
```json
|
||||
{
|
||||
"contents": [{
|
||||
"parts": [
|
||||
{"text": "Edit this image to add a hat"},
|
||||
{
|
||||
"inline_data": {
|
||||
"mime_type": "image/png",
|
||||
"data": "base64_encoded_image_data"
|
||||
}
|
||||
}
|
||||
]
|
||||
}],
|
||||
"generationConfig": {
|
||||
"responseModalities": ["TEXT", "IMAGE"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Image Configuration
|
||||
|
||||
```json
|
||||
{
|
||||
"generationConfig": {
|
||||
"responseModalities": ["TEXT", "IMAGE"],
|
||||
"imageConfig": {
|
||||
"aspectRatio": "16:9",
|
||||
"imageSize": "2K"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
支持的 aspectRatio: "1:1", "16:9", "9:16", "5:4", "4:5", "3:4", "4:3"
|
||||
支持的 imageSize: "1K", "2K", "4K"
|
||||
|
||||
## Response Format
|
||||
|
||||
```json
|
||||
{
|
||||
"candidates": [{
|
||||
"content": {
|
||||
"parts": [
|
||||
{"text": "Here is your generated image:"},
|
||||
{
|
||||
"inlineData": {
|
||||
"mimeType": "image/png",
|
||||
"data": "base64_encoded_image_data"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
## Input Modalities
|
||||
|
||||
1. **Text-to-Image**: 纯文本提示词
|
||||
2. **Text-and-Image-to-Image**: 文本 + 参考图片
|
||||
3. **Multi-Reference Images**: 最多 14 张参考图片
|
||||
|
||||
## Authentication
|
||||
|
||||
使用 API Key:
|
||||
```
|
||||
x-goog-api-key: YOUR_API_KEY
|
||||
```
|
||||
|
||||
或 Bearer Token:
|
||||
```
|
||||
Authorization: Bearer YOUR_TOKEN
|
||||
```
|
||||
|
||||
## Special Features
|
||||
|
||||
- **Thought Signatures**: 多轮对话保持推理上下文
|
||||
- **SynthID**: 自动添加 AI 生成图片水印
|
||||
- **Grounding with Google Search**: 可使用搜索验证事实
|
||||
|
||||
## Implementation Notes for ComfyUI-SimpleChat
|
||||
|
||||
1. 需要支持自定义 base_url(用户可能使用代理)
|
||||
2. 需要处理 base64 图片编解码
|
||||
3. 需要支持图片输入作为参考
|
||||
4. 返回的图片需要转换为 ComfyUI tensor 格式
|
||||
|
||||
## Reference
|
||||
|
||||
- https://ai.google.dev/gemini-api/docs/image-generation
|
||||
- https://ai.google.dev/gemini-api/docs/imagen
|
||||
@@ -0,0 +1,69 @@
|
||||
# NoASS Research Notes
|
||||
|
||||
## What is NoASS?
|
||||
|
||||
NoASS (No Assistant) 是一种用于角色扮演和创意写作的提示词方法论。它将基于聊天的助手交互转换为统一的叙事格式。
|
||||
|
||||
## Core Concept
|
||||
|
||||
传统聊天是 human/assistant 轮流对话,NoASS 将整个 roleplay 合并成一个连续的故事,用户通过编辑来贡献角色动作。
|
||||
|
||||
## Key Motivation
|
||||
|
||||
1. **消除角色偏见**: 传统聊天模型对 human 角色有内在偏见,往往给予不合理的优势
|
||||
2. **解决统一响应长度问题**: 标准聊天训练鼓励一致的回答长度,而动态叙事需要变化的响应规模
|
||||
3. **消除闭合模式**: 传统响应以隐式呼吁用户结束(反问句),破坏沉浸感
|
||||
4. **平等对待角色**: 所有角色被平等对待,因为没有角色正式"属于"任何一方
|
||||
|
||||
## Technical Implementation
|
||||
|
||||
### Context Transformation
|
||||
|
||||
**传统 ChatML 格式:**
|
||||
```
|
||||
<|im_start|>user
|
||||
[Instructions]
|
||||
<|im_end|>
|
||||
<|im_start|>assistant
|
||||
[Greeting]
|
||||
<|im_end|>
|
||||
<|im_start|>user
|
||||
[User action]
|
||||
<|im_end|>
|
||||
```
|
||||
|
||||
**NoASS 格式:**
|
||||
```
|
||||
<|im_start|>user
|
||||
[Instructions for story with all characters]
|
||||
<|im_start|>assistant
|
||||
[Greeting]
|
||||
{{user}}: [user's action]
|
||||
{{char}}: [character's action]
|
||||
...
|
||||
```
|
||||
|
||||
关键机制:所有 roleplay 变成单个预填充的 assistant 消息,用户输入被"拼接"到叙事中,使用角色前缀标记。
|
||||
|
||||
## Formatting Standards
|
||||
|
||||
- 一段 = 一个角色前缀 (`**char name:**`)
|
||||
- 前缀下:*动作*、"对话"、'思想',顺序任意
|
||||
- 最小叙事描述;主要动作使用前缀
|
||||
- 用户响应省略自己的前缀(扩展自动插入)
|
||||
|
||||
## Stop Token Mechanism
|
||||
|
||||
使用用户角色的前缀作为停止令牌,防止模型为玩家角色编写对话。这是防止模型为用户生成动作的关键。
|
||||
|
||||
## Implementation for ComfyUI-SimpleChat
|
||||
|
||||
在我们的实现中,NoASS 功能将:
|
||||
1. 允许用户选择是否启用 NoASS 模式
|
||||
2. 自动处理消息格式转换
|
||||
3. 支持自定义角色前缀
|
||||
4. 使用用户前缀作为停止字符串
|
||||
|
||||
## Reference
|
||||
|
||||
- https://rentry.org/noass_ext
|
||||
@@ -0,0 +1,18 @@
|
||||
"""
|
||||
SimpleChat nodes module.
|
||||
"""
|
||||
from .config import SimpleChatConfig
|
||||
from .chat import SimpleChatText
|
||||
from .chat_image import SimpleChatImage
|
||||
from .chat_noass import SimpleChatNoASS
|
||||
from .gemini_gen import GeminiImageGen
|
||||
from .gemini_edit import GeminiImageEdit
|
||||
|
||||
__all__ = [
|
||||
"SimpleChatConfig",
|
||||
"SimpleChatText",
|
||||
"SimpleChatImage",
|
||||
"SimpleChatNoASS",
|
||||
"GeminiImageGen",
|
||||
"GeminiImageEdit",
|
||||
]
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,55 @@
|
||||
"""
|
||||
Chat node - Basic text conversation.
|
||||
"""
|
||||
from ..core import get_provider, ChatConfig
|
||||
|
||||
|
||||
class SimpleChatText:
|
||||
"""Basic text conversation with LLM."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"config": ("SIMPLECHAT_CONFIG",),
|
||||
"prompt": ("STRING", {"multiline": True, "default": ""}),
|
||||
},
|
||||
"optional": {
|
||||
"system": ("STRING", {"multiline": True, "default": ""}),
|
||||
"temperature": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 2.0, "step": 0.1}),
|
||||
"max_tokens": ("INT", {"default": 2048, "min": 1, "max": 128000}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("STRING",)
|
||||
RETURN_NAMES = ("text",)
|
||||
FUNCTION = "chat"
|
||||
CATEGORY = "SimpleChat"
|
||||
DESCRIPTION = "Send a text prompt to LLM and get a response."
|
||||
|
||||
async def chat(
|
||||
self,
|
||||
config: ChatConfig,
|
||||
prompt: str,
|
||||
system: str = "",
|
||||
temperature: float = 1.0,
|
||||
max_tokens: int = 2048,
|
||||
):
|
||||
# Build messages
|
||||
messages = []
|
||||
if system.strip():
|
||||
messages.append({"role": "system", "content": system})
|
||||
messages.append({"role": "user", "content": prompt})
|
||||
|
||||
# Get provider and send request
|
||||
provider = get_provider(config)
|
||||
|
||||
# Directly await the async provider method
|
||||
response = await provider.chat(
|
||||
messages=messages,
|
||||
model=config.model,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
)
|
||||
|
||||
return (response.text,)
|
||||
@@ -0,0 +1,59 @@
|
||||
"""
|
||||
Chat with Image node - Send image to LLM for analysis.
|
||||
"""
|
||||
import torch
|
||||
from ..core import get_provider, ChatConfig
|
||||
|
||||
|
||||
class SimpleChatImage:
|
||||
"""Send image to LLM for visual understanding."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"config": ("SIMPLECHAT_CONFIG",),
|
||||
"image": ("IMAGE",),
|
||||
"prompt": ("STRING", {"multiline": True, "default": "Describe this image."}),
|
||||
},
|
||||
"optional": {
|
||||
"system": ("STRING", {"multiline": True, "default": ""}),
|
||||
"temperature": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 2.0, "step": 0.1}),
|
||||
"max_tokens": ("INT", {"default": 2048, "min": 1, "max": 128000}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("STRING",)
|
||||
RETURN_NAMES = ("text",)
|
||||
FUNCTION = "chat"
|
||||
CATEGORY = "SimpleChat"
|
||||
DESCRIPTION = "Send an image to LLM for visual analysis and get a text response."
|
||||
|
||||
async def chat(
|
||||
self,
|
||||
config: ChatConfig,
|
||||
image: torch.Tensor,
|
||||
prompt: str,
|
||||
system: str = "",
|
||||
temperature: float = 1.0,
|
||||
max_tokens: int = 2048,
|
||||
):
|
||||
# Build messages
|
||||
messages = []
|
||||
if system.strip():
|
||||
messages.append({"role": "system", "content": system})
|
||||
messages.append({"role": "user", "content": prompt})
|
||||
|
||||
# Get provider and send request
|
||||
provider = get_provider(config)
|
||||
|
||||
# Directly await the async provider method
|
||||
response = await provider.chat(
|
||||
messages=messages,
|
||||
model=config.model,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
images=[image],
|
||||
)
|
||||
|
||||
return (response.text,)
|
||||
@@ -0,0 +1,132 @@
|
||||
"""
|
||||
Chat NoASS node - Roleplay-optimized conversation with NoASS format.
|
||||
"""
|
||||
import torch
|
||||
from ..core import (
|
||||
get_provider,
|
||||
ChatConfig,
|
||||
format_noass_prompt,
|
||||
build_noass_messages,
|
||||
extract_noass_response,
|
||||
build_full_history,
|
||||
get_stop_sequences,
|
||||
)
|
||||
|
||||
|
||||
class SimpleChatNoASS:
|
||||
"""NoASS format conversation for roleplay optimization."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"config": ("SIMPLECHAT_CONFIG",),
|
||||
"scenario_instructions": ("STRING", {
|
||||
"multiline": True,
|
||||
"default": "",
|
||||
"placeholder": "Define world setting, characters, and instructions here (The Scenario)."
|
||||
}),
|
||||
"user_action": ("STRING", {
|
||||
"multiline": True,
|
||||
"default": "",
|
||||
"placeholder": "Your current action or dialogue."
|
||||
}),
|
||||
},
|
||||
"optional": {
|
||||
"prefill_start": ("STRING", {
|
||||
"multiline": True,
|
||||
"default": "",
|
||||
"placeholder": "Force the start of the assistant's response (Prefill)."
|
||||
}),
|
||||
"history": ("STRING", {"multiline": True, "default": "", "forceInput": True}),
|
||||
"image": ("IMAGE",),
|
||||
"user_name": ("STRING", {"default": "User"}),
|
||||
"char_name": ("STRING", {"default": "Assistant"}),
|
||||
"temperature": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 2.0, "step": 0.1}),
|
||||
"max_tokens": ("INT", {"default": 2048, "min": 1, "max": 128000}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("STRING", "STRING")
|
||||
RETURN_NAMES = ("text", "history")
|
||||
FUNCTION = "chat"
|
||||
CATEGORY = "SimpleChat"
|
||||
DESCRIPTION = "NoASS (Experimental) - Hardcore Roleplay with Assistant Prefill. Uses single-turn context with giant assistant prefill to force formatting."
|
||||
|
||||
async def chat(
|
||||
self,
|
||||
config: ChatConfig,
|
||||
scenario_instructions: str,
|
||||
user_action: str,
|
||||
prefill_start: str = "",
|
||||
history: str = "",
|
||||
image: torch.Tensor = None,
|
||||
user_name: str = "User",
|
||||
char_name: str = "Assistant",
|
||||
temperature: float = 1.0,
|
||||
max_tokens: int = 2048,
|
||||
):
|
||||
# Format in NoASS style
|
||||
system_prompt, prefilled = format_noass_prompt(
|
||||
system=scenario_instructions,
|
||||
history=history if history.strip() else None,
|
||||
user_input=user_action,
|
||||
prefill_start=prefill_start,
|
||||
user_name=user_name,
|
||||
char_name=char_name,
|
||||
)
|
||||
|
||||
# Build messages using the new prefill logic
|
||||
messages = build_noass_messages(
|
||||
system=scenario_instructions,
|
||||
history=history if history.strip() else None,
|
||||
user_input=user_action,
|
||||
prefill_start=prefill_start,
|
||||
user_name=user_name,
|
||||
char_name=char_name,
|
||||
)
|
||||
|
||||
# Get provider
|
||||
provider = get_provider(config)
|
||||
|
||||
# Prepare images
|
||||
images = [image] if image is not None else None
|
||||
|
||||
# Get stop sequences
|
||||
stop_sequences = get_stop_sequences(user_name)
|
||||
|
||||
# Directly await the async provider method
|
||||
response = await provider.chat(
|
||||
messages=messages,
|
||||
model=config.model,
|
||||
temperature=temperature,
|
||||
max_tokens=max_tokens,
|
||||
images=images,
|
||||
stop=stop_sequences, # Some providers support this
|
||||
)
|
||||
|
||||
# Extract response and build history
|
||||
response_text = extract_noass_response(
|
||||
response.text,
|
||||
user_name=user_name,
|
||||
char_name=char_name,
|
||||
)
|
||||
|
||||
# Reconstruct the full text for the history output
|
||||
new_history = build_full_history(
|
||||
history=history if history.strip() else None,
|
||||
user_input=user_action,
|
||||
prefill_start=prefill_start,
|
||||
response=response_text,
|
||||
user_name=user_name,
|
||||
char_name=char_name,
|
||||
)
|
||||
|
||||
# If we had a prefill, the returned 'text' should probably include it
|
||||
# so the user sees the full response in the text output node
|
||||
if prefill_start.strip():
|
||||
final_text_output = f"{prefill_start} {response_text}"
|
||||
else:
|
||||
final_text_output = response_text
|
||||
|
||||
return (final_text_output, new_history)
|
||||
@@ -0,0 +1,79 @@
|
||||
"""
|
||||
API Config node - Configure API connection with dynamic model selection.
|
||||
"""
|
||||
from ..core.providers import ChatConfig
|
||||
|
||||
|
||||
class SimpleChatConfig:
|
||||
"""
|
||||
Configure API connection for SimpleChat nodes.
|
||||
Supports dynamic model fetching via the "Refresh Models" button.
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"provider": (["openai", "claude", "gemini"], {
|
||||
"default": "openai",
|
||||
"tooltip": "Select the LLM provider",
|
||||
}),
|
||||
"api_key": ("STRING", {
|
||||
"default": "",
|
||||
"tooltip": "Your API key for the selected provider",
|
||||
}),
|
||||
# Defined as STRING to bypass ComfyUI's strict list validation.
|
||||
# Frontend JS will convert this to a COMBO widget for dropdown functionality.
|
||||
"model": ("STRING", {
|
||||
"default": "",
|
||||
"tooltip": "Select a model. Use the Refresh button to fetch available models.",
|
||||
}),
|
||||
},
|
||||
"optional": {
|
||||
"base_url": ("STRING", {
|
||||
"default": "",
|
||||
"placeholder": "Leave empty for default URL",
|
||||
"tooltip": "Custom API base URL (for proxies or self-hosted deployments)",
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("SIMPLECHAT_CONFIG",)
|
||||
RETURN_NAMES = ("config",)
|
||||
FUNCTION = "create_config"
|
||||
CATEGORY = "SimpleChat"
|
||||
DESCRIPTION = """Configure API connection.
|
||||
|
||||
Usage:
|
||||
1. Select Provider
|
||||
2. Enter API Key
|
||||
3. Click 'Refresh Models' button to fetch available models
|
||||
4. Select Model
|
||||
"""
|
||||
|
||||
def create_config(
|
||||
self,
|
||||
provider: str,
|
||||
api_key: str,
|
||||
model: str,
|
||||
base_url: str = "",
|
||||
):
|
||||
# Default URLs map
|
||||
DEFAULT_URLS = {
|
||||
"openai": "https://api.openai.com/v1",
|
||||
"claude": "https://api.anthropic.com/v1",
|
||||
"gemini": "https://generativelanguage.googleapis.com/v1beta",
|
||||
}
|
||||
|
||||
# Use default URL if not provided
|
||||
if not base_url.strip():
|
||||
base_url = DEFAULT_URLS.get(provider, "")
|
||||
|
||||
config = ChatConfig(
|
||||
provider=provider,
|
||||
api_key=api_key,
|
||||
base_url=base_url,
|
||||
model=model,
|
||||
)
|
||||
|
||||
return (config,)
|
||||
@@ -0,0 +1,58 @@
|
||||
"""
|
||||
Gemini Image Edit node - Edit images using Gemini.
|
||||
"""
|
||||
import torch
|
||||
from ..core import get_provider, ChatConfig
|
||||
|
||||
|
||||
class GeminiImageEdit:
|
||||
"""Edit images using Gemini (Nano Banana / Nano Banana Pro)."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"config": ("SIMPLECHAT_CONFIG",),
|
||||
"image": ("IMAGE",),
|
||||
"prompt": ("STRING", {"multiline": True, "default": ""}),
|
||||
},
|
||||
"optional": {
|
||||
"size": (["1K", "2K", "4K"], {"default": "1K"}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "STRING")
|
||||
RETURN_NAMES = ("IMAGE", "text")
|
||||
FUNCTION = "edit"
|
||||
CATEGORY = "SimpleChat/Gemini"
|
||||
DESCRIPTION = "Edit or modify an image using Gemini image models with a text prompt."
|
||||
|
||||
async def edit(
|
||||
self,
|
||||
config: ChatConfig,
|
||||
image: torch.Tensor,
|
||||
prompt: str,
|
||||
size: str = "1K",
|
||||
):
|
||||
# Validate provider
|
||||
if config.provider != "gemini":
|
||||
raise ValueError("Gemini Image Edit requires Gemini provider. Please use a Gemini config.")
|
||||
|
||||
# Get provider
|
||||
provider = get_provider(config)
|
||||
|
||||
# Directly await the async provider method
|
||||
response = await provider.generate_image(
|
||||
prompt=prompt,
|
||||
model=config.model,
|
||||
reference_image=image,
|
||||
aspect_ratio=None, # Preserve original aspect ratio for edits
|
||||
size=size,
|
||||
)
|
||||
|
||||
# Ensure we have an image
|
||||
if response.image is None:
|
||||
# Return original image if edit failed
|
||||
return (image, response.text or "No image generated")
|
||||
|
||||
return (response.image, response.text)
|
||||
@@ -0,0 +1,59 @@
|
||||
"""
|
||||
Gemini Image Gen node - Text-to-image generation with Gemini.
|
||||
"""
|
||||
import torch
|
||||
from ..core import get_provider, ChatConfig
|
||||
|
||||
|
||||
class GeminiImageGen:
|
||||
"""Generate images from text using Gemini (Nano Banana / Nano Banana Pro)."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"config": ("SIMPLECHAT_CONFIG",),
|
||||
"prompt": ("STRING", {"multiline": True, "default": ""}),
|
||||
},
|
||||
"optional": {
|
||||
"aspect_ratio": (["1:1", "16:9", "9:16", "4:3", "3:4", "5:4", "4:5"], {"default": "1:1"}),
|
||||
"size": (["1K", "2K", "4K"], {"default": "1K"}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "STRING")
|
||||
RETURN_NAMES = ("IMAGE", "text")
|
||||
FUNCTION = "generate"
|
||||
CATEGORY = "SimpleChat/Gemini"
|
||||
DESCRIPTION = "Generate image from text using Gemini image models (gemini-2.5-flash-image or gemini-3-pro-image-preview)."
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
config: ChatConfig,
|
||||
prompt: str,
|
||||
aspect_ratio: str = "1:1",
|
||||
size: str = "1K",
|
||||
):
|
||||
# Validate provider
|
||||
if config.provider != "gemini":
|
||||
raise ValueError("Gemini Image Gen requires Gemini provider. Please use a Gemini config.")
|
||||
|
||||
# Get provider
|
||||
provider = get_provider(config)
|
||||
|
||||
# Directly await the async provider method
|
||||
response = await provider.generate_image(
|
||||
prompt=prompt,
|
||||
model=config.model,
|
||||
reference_image=None,
|
||||
aspect_ratio=aspect_ratio,
|
||||
size=size,
|
||||
)
|
||||
|
||||
# Ensure we have an image
|
||||
if response.image is None:
|
||||
# Return empty image if generation failed
|
||||
empty = torch.zeros((1, 512, 512, 3))
|
||||
return (empty, response.text or "No image generated")
|
||||
|
||||
return (response.image, response.text)
|
||||
@@ -0,0 +1 @@
|
||||
aiohttp>=3.8.0
|
||||
@@ -0,0 +1,88 @@
|
||||
import { app } from "../../scripts/app.js";
|
||||
import { api } from "../../scripts/api.js";
|
||||
|
||||
app.registerExtension({
|
||||
name: "ComfyUI.SimpleChat.Config.V3", // Use a new name to avoid browser caching issues
|
||||
async nodeCreated(node, app) {
|
||||
if (node.comfyClass === "SimpleChatConfig") {
|
||||
// --- WIDGET REPLACEMENT AND REORDERING ---
|
||||
|
||||
// Find the original string widget for 'model'
|
||||
const modelWidgetIndex = node.widgets.findIndex((w) => w.name === "model");
|
||||
if (modelWidgetIndex === -1) return;
|
||||
const originalModelWidget = node.widgets[modelWidgetIndex];
|
||||
const originalModelValue = originalModelWidget.value;
|
||||
|
||||
// Step 1: Create new widgets using the addWidget API. This appends them temporarily.
|
||||
const newModelCombo = node.addWidget("combo", "model", originalModelValue, () => {}, {
|
||||
values: [originalModelValue || ""]
|
||||
});
|
||||
const refreshButton = node.addWidget("button", "Refresh Models", "refresh", () => {});
|
||||
|
||||
// Step 2: Remove the newly created widgets from the end of the array.
|
||||
// We now have fully constructed widget objects, detached from the node's widget list.
|
||||
node.widgets.pop(); // removes refreshButton
|
||||
node.widgets.pop(); // removes newModelCombo
|
||||
|
||||
// Step 3: Remove the original string widget from its position.
|
||||
node.widgets.splice(modelWidgetIndex, 1);
|
||||
|
||||
// Step 4: Insert the new, constructed widgets back into the array at the correct positions.
|
||||
node.widgets.splice(modelWidgetIndex, 0, newModelCombo); // Insert combo where the old one was.
|
||||
node.widgets.splice(modelWidgetIndex + 1, 0, refreshButton); // Insert button right after it.
|
||||
|
||||
// --- END WIDGET MANIPULATION ---
|
||||
|
||||
// Now, find all widgets again to be safe and attach logic
|
||||
const providerWidget = node.widgets.find((w) => w.name === "provider");
|
||||
const apiKeyWidget = node.widgets.find((w) => w.name === "api_key");
|
||||
const baseUrlWidget = node.widgets.find((w) => w.name === "base_url");
|
||||
|
||||
// Attach the callback logic to the refresh button
|
||||
refreshButton.callback = () => {
|
||||
const provider = providerWidget.value;
|
||||
const apiKey = apiKeyWidget.value;
|
||||
const baseUrl = baseUrlWidget ? baseUrlWidget.value : "";
|
||||
|
||||
if (!apiKey) {
|
||||
alert("Please enter an API Key first.");
|
||||
return;
|
||||
}
|
||||
|
||||
refreshButton.label = "Loading...";
|
||||
refreshButton.disabled = true;
|
||||
node.setDirtyCanvas(true, true);
|
||||
|
||||
const url = `/simplechat/models/${provider}?api_key=${encodeURIComponent(apiKey)}&base_url=${encodeURIComponent(baseUrl)}`;
|
||||
|
||||
api.fetchApi(url)
|
||||
.then(response => response.json())
|
||||
.then(models => {
|
||||
if (Array.isArray(models) && models.length > 0) {
|
||||
newModelCombo.options.values = models;
|
||||
if (!newModelCombo.value || !models.includes(newModelCombo.value)) {
|
||||
newModelCombo.value = models[0];
|
||||
}
|
||||
} else {
|
||||
newModelCombo.options.values = [];
|
||||
newModelCombo.value = "";
|
||||
alert("No models found.");
|
||||
}
|
||||
})
|
||||
.catch(err => {
|
||||
console.error("Error fetching models:", err);
|
||||
alert("Error fetching models: " + err.message);
|
||||
})
|
||||
.finally(() => {
|
||||
refreshButton.label = "Refresh Models";
|
||||
refreshButton.disabled = false;
|
||||
node.setDirtyCanvas(true, true);
|
||||
});
|
||||
};
|
||||
|
||||
// Trigger a redraw to ensure the node resizes correctly after manipulation
|
||||
node.computeSize();
|
||||
node.setDirtyCanvas(true, true);
|
||||
}
|
||||
}
|
||||
});
|
||||
Reference in New Issue
Block a user