Eric Hiss
00c692782d
Major Release - Complete Advanced Node Redesign ✨ New Features: - Advanced node: 4 operation modes (expand/refine/modify_style/add_details) - Advanced node: Clear detail levels (concise/moderate/detailed/exhaustive) - Advanced node: Optional image input with Qwen3-VL integration - Video nodes: Emphasis syntax preservation (keyword:1.5) - Video nodes: Alternation syntax support {option1|option2} 🔧 Changes: - Replaced confusing tier system with intuitive detail levels - Added operation modes for modifying existing prompts - Removed manual mode selection (auto-detected from image presence) - Added Qwen3-VL backend for image analysis - Backward compatible tier name mapping in expansion_engine 📚 Documentation: - README.md completely rewritten - docs/ADVANCED_NODE_REDESIGN.md - Complete redesign guide - docs/BUGFIX_ADVANCED_NODE.md - Syntax preservation details - Organized old docs into docs/archive/ 🐛 Fixes: - Emphasis syntax now properly preserved through LLM processing - Alternation syntax for random element selection - Image-to-video mode now functional with Qwen3-VL ⚡ Breaking Changes: None (fully backward compatible) See RELEASE_NOTES_v1.8.0.md for complete details.
212 lines
5.9 KiB
Markdown
212 lines
5.9 KiB
Markdown
# Vision Backend Guide
|
|
|
|
## Overview
|
|
|
|
The Text-to-Image Prompt Enhancer node now supports **three vision captioning backends** for analyzing reference images:
|
|
|
|
1. **Auto** (default) - Inherits from main LLM if it supports vision
|
|
2. **Qwen3-VL** - Local on-device vision model (no server needed)
|
|
3. **LM Studio / Ollama** - Use separate vision-capable LLM server
|
|
|
|
## Backend Options
|
|
|
|
### 1. Auto (Recommended for most users)
|
|
|
|
**What it does:**
|
|
- If your main LLM model supports vision (e.g., LLaVA, GPT-4 Vision), it will use that
|
|
- If your main LLM is text-only, vision captioning is disabled (falls back to heuristic analysis)
|
|
|
|
**When to use:**
|
|
- You're already running a vision-capable model in LM Studio/Ollama
|
|
- You want the simplest setup with no extra configuration
|
|
|
|
**What you need:**
|
|
- Nothing! Just select "auto" in the vision_backend dropdown
|
|
|
|
---
|
|
|
|
### 2. Qwen3-VL (Local, No Server Required)
|
|
|
|
**What it does:**
|
|
- Downloads and runs `Qwen/Qwen3-VL-4B-Instruct` directly on your GPU
|
|
- No LM Studio or Ollama server needed
|
|
- Fully integrated - just select and go
|
|
|
|
**When to use:**
|
|
- You want local vision captioning without running a separate LLM server
|
|
- You have ~8-16GB VRAM available for the vision model
|
|
- You want the best quality vision captions (Qwen3-VL is excellent at image description)
|
|
|
|
**What you need:**
|
|
1. Install the optional dependencies (if not already installed):
|
|
```bash
|
|
pip install transformers>=4.41 torch torchvision Pillow huggingface-hub
|
|
```
|
|
|
|
2. **(Optional)** For quantization support (4-bit/8-bit to save VRAM):
|
|
```bash
|
|
pip install bitsandbytes
|
|
```
|
|
|
|
3. Select `qwen3_vl` from the vision_backend dropdown
|
|
|
|
**First run behavior:**
|
|
- The model will auto-download to `ComfyUI/models/VLM/Qwen3-VL-4B-Instruct/`
|
|
- Download is ~9GB, happens once
|
|
- Subsequent runs load from disk (much faster)
|
|
- **Reuses models** from Granddyser nodes if already downloaded!
|
|
|
|
**Performance:**
|
|
- First inference: ~5-15 seconds (model loading)
|
|
- Subsequent inferences: ~2-5 seconds per image
|
|
- Memory: ~6-8GB VRAM (full precision), ~3-4GB (8-bit), ~2-3GB (4-bit)
|
|
|
|
---
|
|
|
|
### 3. LM Studio / Ollama (Separate Vision Model)
|
|
|
|
**What it does:**
|
|
- Uses the same LM Studio/Ollama endpoint as your main LLM
|
|
- Reuses the model name from your main LLM configuration
|
|
- Useful if you're running a vision model in LM Studio/Ollama
|
|
|
|
**When to use:**
|
|
- You're already running a vision-capable model server
|
|
- You want to explicitly specify LM Studio or Ollama for vision (even if main LLM is different)
|
|
|
|
**What you need:**
|
|
1. Run a vision-capable model in LM Studio or Ollama:
|
|
- **LM Studio**: Load models like `llava`, `bakllava`, `moondream`, etc.
|
|
- **Ollama**: `ollama run llava`, `ollama run bakllava`, etc.
|
|
|
|
2. Select `lm_studio` or `ollama` from the vision_backend dropdown
|
|
|
|
3. Make sure your **model_name** field matches your vision model
|
|
|
|
---
|
|
|
|
### 4. Disable
|
|
|
|
**What it does:**
|
|
- Turns off all vision captioning
|
|
- Reference images still analyzed using color/lighting heuristics
|
|
- Useful for testing or when you don't need detailed captions
|
|
|
|
**When to use:**
|
|
- Debugging or performance testing
|
|
- You only care about basic image statistics (colors, brightness, etc.)
|
|
|
|
---
|
|
|
|
## Usage Examples
|
|
|
|
### Example 1: Using Qwen3-VL (Simplest for quality captions)
|
|
|
|
```
|
|
vision_backend: qwen3_vl
|
|
```
|
|
|
|
That's it! No other fields needed. The node will:
|
|
- Auto-download Qwen3-VL-4B-Instruct if not present
|
|
- Load it into VRAM
|
|
- Generate detailed captions for your reference images
|
|
|
|
---
|
|
|
|
### Example 2: Using LM Studio with LLaVA
|
|
|
|
Main LLM settings:
|
|
```
|
|
llm_backend: lm_studio
|
|
model_name: llava-v1.6-mistral-7b
|
|
api_endpoint: http://localhost:1234/v1
|
|
```
|
|
|
|
Vision settings:
|
|
```
|
|
vision_backend: auto (will detect LLaVA supports vision and use it automatically)
|
|
```
|
|
|
|
Or explicitly:
|
|
```
|
|
vision_backend: lm_studio
|
|
```
|
|
|
|
---
|
|
|
|
### Example 3: Mixed setup (Ollama for main LLM, Qwen3-VL for vision)
|
|
|
|
Main LLM settings:
|
|
```
|
|
llm_backend: ollama
|
|
model_name: llama3.1
|
|
api_endpoint: http://localhost:11434
|
|
```
|
|
|
|
Vision settings:
|
|
```
|
|
vision_backend: qwen3_vl (uses local Qwen3-VL instead of Ollama)
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### "Vision backend init failed"
|
|
- Make sure your LM Studio/Ollama server is running
|
|
- Verify the endpoint URL is correct
|
|
- Check that the model name matches what's loaded in the server
|
|
|
|
### "Model does not advertise image support"
|
|
- Your selected model is text-only
|
|
- Switch to `qwen3_vl` or load a vision-capable model
|
|
|
|
### Qwen3-VL fails to load
|
|
- Check you have the required dependencies installed
|
|
- Verify you have enough VRAM (~6-8GB minimum)
|
|
- Check console for detailed error messages
|
|
|
|
### Slow performance with Qwen3-VL
|
|
- First run always slower (model loading)
|
|
- Consider using quantization (install `bitsandbytes`)
|
|
- Model stays loaded in memory between runs (faster subsequent uses)
|
|
|
|
---
|
|
|
|
## Advanced: Qwen3-VL Quantization
|
|
|
|
If you need to save VRAM, you can force quantization by modifying `qwen3_vl_backend.py` defaults, but currently the node uses auto-precision (lets Transformers decide based on available VRAM).
|
|
|
|
For 4-bit/8-bit quantization, ensure `bitsandbytes` is installed:
|
|
```bash
|
|
pip install bitsandbytes
|
|
```
|
|
|
|
---
|
|
|
|
## Model Download Locations
|
|
|
|
- **Qwen3-VL**: `ComfyUI/models/VLM/Qwen3-VL-4B-Instruct/`
|
|
- Downloads happen automatically on first use
|
|
- **Compatible with Granddyser nodes**: Uses the same location, so models are shared!
|
|
- You can pre-download using:
|
|
```python
|
|
from huggingface_hub import snapshot_download
|
|
snapshot_download("Qwen/Qwen3-VL-4B-Instruct", local_dir="ComfyUI/models/VLM/Qwen3-VL-4B-Instruct")
|
|
```
|
|
|
|
---
|
|
|
|
## Recommendations
|
|
|
|
- **For quality captions**: Use `qwen3_vl` - it's specifically trained for visual understanding
|
|
- **For speed**: Use `auto` with a fast vision model in LM Studio/Ollama
|
|
- **For low VRAM**: Use `auto` with Ollama (runs vision model on CPU if needed)
|
|
- **For no dependencies**: Stick with `disable` and use heuristic analysis
|
|
|
|
---
|
|
|
|
## Credits
|
|
|
|
Qwen3-VL integration inspired by [Granddyser's qwen3-vl-comfy-ui nodes](https://github.com/Granddyser/qwen3-vl-comfy-ui).
|