slahiri
f9bbb1e081
- models.md: Supported models by tier (Quick/Standard/Detailed/Extreme) - captioning.md: Complete VLM guide including: - Captioning approaches (Booru tags, e621, natural language, hybrid) - Model-specific recommendations for SD1.5/SDXL/Flux/Z-Image - Detailed model specs for 14+ captioning models - Vision model landscape (DINOv2, InternVL, SigLIP, etc.) - Workflow patterns and ComfyUI nodes reference Generated with Claude Code
60 lines
2.4 KiB
Markdown
60 lines
2.4 KiB
Markdown
# Supported Models (MVP)
|
|
|
|
## Quick
|
|
Fast captioning, lower quality. Good for batch processing.
|
|
|
|
| Model | Type | Notes |
|
|
|-------|------|-------|
|
|
| `MiaoshouAI/Florence-2-large-PromptGen-v2.0` | Local | Fast, lightweight |
|
|
| `Salesforce/blip-image-captioning-large` | Local | Classic captioner |
|
|
|
|
## Standard
|
|
Balanced quality and speed. Good for most use cases.
|
|
|
|
| Model | Type | Notes |
|
|
|-------|------|-------|
|
|
| `mradermacher/llama-joycaption-beta-one-hf-llava-GGUF` | Local | Q4_K_M quantized |
|
|
| `Salesforce/blip2-opt-2.7b` | Local | Good quality |
|
|
| `Qwen/Qwen2.5-VL-3B-Instruct` | Local | Qwen vision-language |
|
|
| `Qwen/Qwen2.5-VL-7B-Instruct` | Local | 4-bit quantized |
|
|
|
|
## Detailed
|
|
High quality, slower. Best for single image analysis.
|
|
|
|
| Model | Type | Notes |
|
|
|-------|------|-------|
|
|
| `fancyfeast/llama-joycaption-beta-one-hf-llava` | Local | Full precision |
|
|
| `Ertugrul/Qwen2.5-VL-7B-Captioner-Relaxed` | Local | Fine-tuned for captions |
|
|
| `Salesforce/blip2-opt-6.7b` | Local | Larger BLIP2 |
|
|
|
|
## Extreme (API)
|
|
Best quality, requires API keys. Cloud-based.
|
|
|
|
| Provider | URL | Notes |
|
|
|----------|-----|-------|
|
|
| Gemini | https://ai.google.dev/ | Google AI |
|
|
| OpenAI | https://platform.openai.com/ | GPT-4o Vision |
|
|
| Claude | https://console.anthropic.com/ | Anthropic |
|
|
| Grok | https://console.x.ai/ | xAI |
|
|
|
|
---
|
|
|
|
# Final Captioning Tiers (Detailed Reference)
|
|
|
|
| Tier | Model | VRAM / Cost | Speed | Best For |
|
|
|------|-------|-------------|-------|----------|
|
|
| **Quick** | Florence-2-large-PromptGen v2.0 | ~1.5 GB | ⚡⚡⚡ | Bulk tagging, initial passes |
|
|
| | BLIP-image-captioning-large | ~1 GB | ⚡⚡⚡ | Simple baseline |
|
|
| **Standard** | JoyCaption Beta One Q4_K_M | ~5-7 GB | ⚡⚡ | Balanced quality/speed |
|
|
| | BLIP-2-opt-2.7b | ~7 GB | ⚡⚡ | Conditional captioning |
|
|
| | Qwen2.5-VL-3B-Instruct | ~6-8 GB | ⚡⚡ | Multilingual |
|
|
| | Qwen2.5-VL-7B (4-bit) | ~6-8 GB | ⚡ | Better Qwen, quantized |
|
|
| **Detailed** | JoyCaption Beta One FP16 | ~16 GB | 🐢 | Best for diffusion training |
|
|
| | Qwen2.5-VL-7B-Captioner-Relaxed | ~16 GB | 🐢 | T2I optimized |
|
|
| | Qwen2.5-VL-7B (8-bit) | ~10 GB | 🐢 | Comfortable fit |
|
|
| | BLIP-2-opt-6.7b | ~14 GB | 🐢 | Larger BLIP-2 |
|
|
| **Extreme (API)** | Gemini 2.5 Pro | ~$1.25/M in | 🐌 | Best vision overall |
|
|
| | GPT-4o | ~$5/M in | 🐌 | Creative, detailed |
|
|
| | Claude Sonnet 4 | ~$3/M in | 🐌 | Follows format precisely |
|
|
| | Grok 4 | ~$0.20/M in | 🐌 | Cheapest API option |
|