121 lines
3.6 KiB
Markdown
121 lines
3.6 KiB
Markdown
# ComfyUI-ImageToPrompt
|
|
|
|
A custom ComfyUI node that sends images to OpenAI-compatible vision APIs (like GPT-4 Vision, LLaVA, etc.) and returns the LLM's text response. Perfect for image analysis, captioning, and visual question answering workflows.
|
|
|
|
## Features
|
|
|
|
- 📷 **Upload images** via ComfyUI's image input nodes
|
|
- 🤖 **Send to any OpenAI-compatible API** (GPT-4 Vision, local LLaVA, llama.cpp, etc.)
|
|
- 💬 **Optional system and user prompts** for controlling the analysis
|
|
- ⚙️ **Configurable parameters**: temperature, max tokens, model selection
|
|
- 📝 **Text output** that can be used in downstream nodes or displayed in the UI
|
|
- 🔍 **Debug logging** to monitor API calls and responses
|
|
|
|
## Installation
|
|
|
|
### Prerequisites
|
|
|
|
- [ComfyUI](https://github.com/comfyanonymous/ComfyUI) installed and working
|
|
|
|
### Install via ComfyUI Manager (Recommended)
|
|
|
|
1. Open ComfyUI Manager
|
|
2. Search for "ComfyUI-ImageToPrompt"
|
|
3. Click Install
|
|
4. Restart ComfyUI
|
|
|
|
### Manual Installation
|
|
|
|
1. Navigate to your ComfyUI custom nodes directory:
|
|
```bash
|
|
cd ComfyUI/custom_nodes/
|
|
```
|
|
|
|
2. Clone this repository:
|
|
```bash
|
|
git clone https://github.com/yourusername/ComfyUI-ImageToPrompt.git
|
|
```
|
|
|
|
3. Install dependencies:
|
|
```bash
|
|
cd ComfyUI-ImageToPrompt
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
4. Restart ComfyUI
|
|
|
|
## Usage
|
|
|
|
1. **Add the node**: In ComfyUI, add the "Image to LLM Prompt" node (found under the "LLM" category)
|
|
|
|
2. **Connect an image**: Connect an image output (from LoadImage, etc.) to the node's image input
|
|
|
|
3. **Configure the API**:
|
|
- `base_url`: Your OpenAI-compatible API endpoint (e.g., `http://localhost:8080/v1`)
|
|
- `api_key`: Your API key (use `sk-test` for local servers that don't require authentication)
|
|
- `model`: Model name (e.g., `gpt-4o`, `llava`, etc.)
|
|
|
|
4. **Add prompts (optional)**:
|
|
- `system_prompt`: Instructions for the AI (e.g., "You are an expert art critic")
|
|
- `user_prompt`: Specific question about the image (e.g., "Describe this image in detail")
|
|
|
|
5. **Adjust parameters (optional)**:
|
|
- `temperature`: Controls randomness (0.0 = deterministic, 2.0 = very random)
|
|
- `max_tokens`: Maximum response length
|
|
|
|
6. **Execute the workflow**: The LLM's response will appear in the node's output and can be passed to other nodes
|
|
|
|
## Example Workflow
|
|
|
|
```
|
|
LoadImage → ImageToLLMPrompt → (use text output in other nodes)
|
|
```
|
|
|
|
The node outputs a STRING that can be connected to:
|
|
- Text processing nodes
|
|
- Prompt nodes for image generation
|
|
- File writing nodes
|
|
- Any node that accepts STRING input
|
|
|
|
## API Compatibility
|
|
|
|
This node works with any OpenAI-compatible vision API endpoint, including:
|
|
|
|
- **OpenAI GPT-4 Vision** (`https://api.openai.com/v1`)
|
|
- **Local LLaVA servers** (via llama.cpp, ollama, etc.)
|
|
- **Azure OpenAI** with vision models
|
|
- **Other compatible backends**
|
|
|
|
## Configuration Examples
|
|
|
|
### OpenAI GPT-4 Vision
|
|
```
|
|
base_url: https://api.openai.com/v1
|
|
api_key: sk-your-actual-api-key
|
|
model: gpt-4o
|
|
```
|
|
|
|
### Local LLaVA (llama.cpp server)
|
|
```
|
|
base_url: http://localhost:8080/v1
|
|
api_key: sk-test
|
|
model: llava
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
- **"LLM API call failed" errors**: Check that your base_url is correct and the server is running
|
|
- **Empty responses**: Verify your model supports vision/image inputs
|
|
- **Timeout errors**: Increase the timeout in the code or use a faster model
|
|
- **Node not appearing**: Restart ComfyUI and check the console for import errors
|
|
|
|
## Dependencies
|
|
|
|
- `openai` - OpenAI Python client library
|
|
- `pillow` - Python Imaging Library for image processing
|
|
- `numpy` - Array processing (typically already installed with ComfyUI)
|
|
|
|
## Contributing
|
|
|
|
Contributions are welcome! Please feel free to submit a Pull Request.
|