first draft

This commit is contained in:
Andrea Pozzetti
2025-10-22 19:15:53 +00:00
parent d9411fb4cc
commit 3884d3d5cf
11 changed files with 1339 additions and 1 deletions
View File
+353 -1
View File
@@ -1,2 +1,354 @@
# ComfyUI-Grounding
ComfyUI node to run grounding models
ComfyUI custom nodes for open-vocabulary object detection using state-of-the-art grounding models.
## Features
- **GroundingDINO**: Open-vocabulary object detection with natural language prompts
- **YOLO-World**: Real-time open-vocabulary detection
- **Batch Processing**: Process multiple images at once for improved efficiency
- **Referring Expression Comprehension (REC)**: Use full text as label when no separators present
- **Flexible Architecture**: Support for both local and HuggingFace implementations
- **Utility Nodes**: Bounding box visualization and mask conversion
## Supported Models
### GroundingDINO (Original)
- GroundingDINO-T (SwinT backbone, 694MB)
- GroundingDINO-B (SwinB backbone, 938MB)
### MM-GroundingDINO (Multimodal)
**Tiny Models (Swin-T backbone):**
- MM-GroundingDINO-Tiny (O365+GoldG, 50.4 mAP)
- MM-GroundingDINO-Tiny (O365+GoldG+GRIT, 50.5 mAP)
- MM-GroundingDINO-Tiny (O365+GoldG+V3Det, 50.6 mAP)
- MM-GroundingDINO-Tiny (O365+GoldG+GRIT+V3Det, 50.4 mAP)
**Base Models (Swin-B backbone):**
- MM-GroundingDINO-Base (O365+GoldG+V3Det, 52.5 mAP)
- MM-GroundingDINO-Base (All datasets, 59.5 mAP) ⭐ Recommended
**Large Models (Swin-L backbone):**
- MM-GroundingDINO-Large (O365v2+OIv6+GoldG, 53.0 mAP)
- MM-GroundingDINO-Large (All datasets, 60.3 mAP) ⭐ Best accuracy
### YOLO-World
- YOLOv8s-World (small)
- YOLOv8m-World (medium)
- YOLOv8l-World (large)
- YOLOv8x-World (extra large)
## Installation
1. Navigate to your ComfyUI custom nodes directory:
```bash
cd ComfyUI/custom_nodes/
```
2. Clone or copy this repository:
```bash
git clone <repository-url> ComfyUI-Grounding
```
3. Install dependencies:
```bash
cd ComfyUI-Grounding
pip install -r requirements.txt
```
## Nodes
### 1. GroundingDINOModelLoader
Loads a GroundingDINO or MM-GroundingDINO model for object detection.
**Inputs:**
- `model_name`: Select from available GroundingDINO/MM-GroundingDINO models
- `keep_in_memory`: Cache model in memory for faster reuse (default: True)
**Outputs:**
- `model`: Loaded GroundingDINO model
**Features:**
- ✅ Model caching - loads instantly on second use
- ✅ Local model storage in `ComfyUI-Grounding/Models/`
- ✅ Supports both original and MM-GroundingDINO variants
### 2. GroundingDINODetector
Detects objects in images using text prompts.
**Inputs:**
- `model`: GroundingDINO model (from loader)
- `image`: Input image (supports batches)
- `prompt`: Text description of objects to detect (e.g., "person . car . dog .")
- `box_threshold`: Confidence threshold for bounding boxes (0.0-1.0)
- `text_threshold`: Confidence threshold for text matching (0.0-1.0)
- `single_box_mode`: (Optional) Return only highest-scoring detection for REC
- `output_format`: (Optional) "list_only" (default, SAM2-compatible) or "dict_with_data"
**Outputs:**
- `bboxes`: Detected bounding boxes (format depends on output_format setting)
- `list_only`: List of numpy arrays `[[boxes_img0], [boxes_img1], ...]` - SAM2-compatible
- `dict_with_data`: Dict `{"boxes": [...], "labels": [...], "scores": [...]}`
- `annotated_image`: Image with drawn bounding boxes (batched)
- `labels`: String with detected object labels
**Features:**
- ✅ Batch processing - process multiple images at once
- ✅ Automatic REC mode - when prompt has no separators (. or ,), uses full text as label
- ✅ Single box mode - returns only highest-scoring detection for referring expressions
- ✅ Flexible output format - choose between SAM2-compatible list or full dict with metadata
### 3. YOLOWorldModelLoader
Loads a YOLO-World model for real-time detection.
**Inputs:**
- `model_name`: Select from available YOLO-World models
- `keep_in_memory`: Cache model in memory for faster reuse (default: True)
**Outputs:**
- `model`: Loaded YOLO-World model
**Features:**
- ✅ Model caching - loads instantly on second use
- ✅ Local model storage in `ComfyUI-Grounding/Models/`
- ✅ Auto-downloads models on first use
### 4. YOLOWorldDetector
Detects objects using YOLO-World with custom classes.
**Inputs:**
- `model`: YOLO-World model (from loader)
- `image`: Input image (supports batches)
- `classes`: Comma-separated list of classes (e.g., "person, car, dog")
- `confidence_threshold`: Minimum confidence for detections (0.0-1.0)
- `single_box_mode`: (Optional) Return only highest-scoring detection for REC
- `output_format`: (Optional) "list_only" (default, SAM2-compatible) or "dict_with_data"
**Outputs:**
- `bboxes`: Detected bounding boxes (format depends on output_format setting)
- `list_only`: List of numpy arrays `[[boxes_img0], [boxes_img1], ...]` - SAM2-compatible
- `dict_with_data`: Dict `{"boxes": [...], "labels": [...], "scores": [...]}`
- `annotated_image`: Image with drawn bounding boxes (batched)
- `labels`: String with detected object labels
**Features:**
- ✅ Batch processing - process multiple images at once
- ✅ Automatic REC mode - when classes has no comma, uses full text as label
- ✅ Single box mode - returns only highest-scoring detection for referring expressions
- ✅ Flexible output format - choose between SAM2-compatible list or full dict with metadata
### 5. BboxVisualizer
Draws bounding boxes on images.
**Inputs:**
- `image`: Input image
- `bboxes`: Bounding box data
- `line_width`: Thickness of bounding box lines (optional)
**Outputs:**
- `annotated_image`: Image with drawn bounding boxes
### 5. UnifiedDetector ⭐ NEW
Universal detector that works with BOTH GroundingDINO and YOLO-World models.
**Inputs:**
- `model`: Any model (GroundingDINO or YOLO-World)
- `image`: Input image
- `prompt`: Text prompt (supports both formats)
- For GroundingDINO: "cat . dog . car ."
- For YOLO-World: "cat, dog, car"
- `threshold`: Confidence threshold (0.0-1.0)
- `text_threshold`: Text matching threshold (optional, GroundingDINO only)
**Outputs:**
- `bboxes`: Detected bounding boxes with labels and scores
- `annotated_image`: Image with drawn bounding boxes
- `labels`: String with detected object labels
**Features:**
- ✅ Auto-detects model type
- ✅ Single node for all detection models
- ✅ Simplified workflow
### 6. BboxToMask
Converts bounding boxes to binary masks.
**Inputs:**
- `image`: Input image (for dimensions)
- `bboxes`: Bounding box data
- `combine_masks`: Whether to combine all boxes into one mask (optional)
**Outputs:**
- `masks`: Binary mask(s) for detected regions
## Key Features
### 🚀 Model Caching
Models are cached in memory by default for instant reuse:
- First load: Downloads and initializes model
- Second load: Instant loading from cache
- Toggle `keep_in_memory=False` to clear cache
### 📁 Local Model Storage
All models are downloaded to the plugin's local Models directory:
- Location: `ComfyUI/custom_nodes/ComfyUI-Grounding/Models/`
- Auto-created on first use with a `models_go_here.txt` placeholder
- GroundingDINO/MM-GroundingDINO: Stored in HuggingFace transformers cache format
- YOLO-World: Stored as `.pt` files (yolov8s-world.pt, etc.)
### 🎯 UnifiedDetector
One detector for all models - automatically detects whether you're using GroundingDINO or YOLO-World and routes to the appropriate inference method.
### 📦 Batch Processing
All detector nodes support batch processing:
- Process multiple images in a single forward pass
- Input images as a batch tensor (B, H, W, C)
- Returns batched bounding boxes with lists of detections for each image
- Annotated images are returned as a single batch tensor
### 🎯 Referring Expression Comprehension (REC)
Automatic REC mode for natural language queries:
- **GroundingDINO**: If prompt has no periods (`.`) or commas (`,`), uses full text as label
- Example: `"main object in the center of the picture"` → all detections labeled with full text
- With separators: `"person . car . dog ."` → uses standard noun phrase extraction
- **YOLO-World**: If classes has no comma (`,`), uses full text as label
- Example: `"red car in the parking lot"` → all detections labeled with full text
- With separators: `"person, car, dog"` → uses standard class mapping
- **Single Box Mode**: Enable `single_box_mode` to return only the highest-scoring detection (perfect for REC tasks)
### 📋 Output Formats
Detector nodes support two output formats for bounding boxes:
**1. `list_only` (Default - SAM2 Compatible)**
- Returns: List of numpy arrays, one per image in batch
- Format: `[[bbox1, bbox2], [bbox3], ...]` where each bbox is `[x1, y1, x2, y2]`
- Use when: Connecting to SAM2 segmentation nodes
- Example:
```python
# For 2 images with 2 and 1 detections respectively
[
array([[10, 20, 100, 200], [50, 60, 150, 250]]), # Image 0: 2 boxes
array([[30, 40, 120, 180]]) # Image 1: 1 box
]
```
**2. `dict_with_data` (Full Metadata)**
- Returns: Dictionary with boxes, labels, and scores
- Format: `{"boxes": [...], "labels": [...], "scores": [...]}`
- Use when: Need label and confidence information for downstream processing
- NOT compatible with SAM2 (dict iteration gives keys, not values)
- Example:
```python
{
"boxes": [array([...]), array([...])], # Boxes per image
"labels": [["cat", "dog"], ["bird"]], # Labels per image
"scores": [array([0.9, 0.8]), array([0.7])] # Scores per image
}
```
**Recommendation:** Use `list_only` (default) unless you specifically need label/score metadata for your workflow.
## Usage Examples
### Example 1: Basic Object Detection with GroundingDINO
1. Add `GroundingDINOModelLoader` node
2. Select model (e.g., "GroundingDINO_SwinT_OGC (694MB)")
3. Add `GroundingDINODetector` node
4. Connect model output to detector
5. Load an image and connect to detector
6. Set prompt: "person . car . bicycle ."
7. Adjust thresholds as needed
8. View annotated image output
### Example 2: YOLO-World Detection
1. Add `YOLOWorldModelLoader` node
2. Select model (e.g., "yolov8s-world")
3. Add `YOLOWorldDetector` node
4. Connect model and image
5. Set classes: "cat, dog, bird, horse"
6. Adjust confidence threshold
7. View results
### Example 3: Using UnifiedDetector (Recommended)
1. Add any model loader (GroundingDINO or YOLO-World)
2. Load your model
3. Add `UnifiedDetector` node
4. Connect model and image
5. Set prompt (works with both formats!)
6. The node automatically detects model type and uses the right inference
### Example 4: MM-GroundingDINO for High Accuracy
1. Add `GroundingDINOModelLoader` node
2. Select "MM-GroundingDINO-Base (All datasets, 59.5 mAP)"
3. Enable `keep_in_memory=True` (default)
4. Add `GroundingDINODetector` or `UnifiedDetector`
5. Set prompt: "specific object . another object ."
6. Enjoy state-of-the-art detection accuracy!
### Example 5: Convert Detections to Masks
1. Use any detector node (GroundingDINO or YOLO-World)
2. Add `BboxToMask` node
3. Connect image and bboxes outputs
4. Choose whether to combine masks
5. Use masks for segmentation, inpainting, etc.
## Model Downloads
Models will be automatically downloaded on first use:
- **All models**: Stored in `ComfyUI/custom_nodes/ComfyUI-Grounding/Models/`
- **GroundingDINO/MM-GroundingDINO**: HuggingFace transformers cache format
- **YOLO-World**: Direct `.pt` files (e.g., yolov8s-world.pt)
## Tips
- **GroundingDINO prompts**: End each object with a period (e.g., "cat . dog . car .")
- **YOLO-World classes**: Use comma-separated values (e.g., "cat, dog, car")
- **Threshold tuning**: Start with 0.3 and adjust based on results
- Lower threshold = more detections (including false positives)
- Higher threshold = fewer, more confident detections
- **Performance**: YOLO-World is faster but GroundingDINO may be more accurate
## Integration with SAM2
These nodes work seamlessly with SAM2 for instance segmentation:
1. Use GroundingDINO or YOLO-World to detect objects
2. Convert bboxes to masks using `BboxToMask`
3. Use masks with SAM2 for precise segmentation
## Troubleshooting
### ImportError: transformers not found
```bash
pip install transformers timm
```
### ImportError: ultralytics not found
```bash
pip install ultralytics
```
### Models not downloading
Check your internet connection and ensure you have write permissions to the ComfyUI models directory.
### CUDA out of memory
Try using a smaller model variant (e.g., GroundingDINO-T instead of -B, or yolov8s instead of yolov8x).
## License
MIT License - See LICENSE file for details
## Credits
- GroundingDINO: [IDEA-Research](https://github.com/IDEA-Research/GroundingDINO)
- YOLO-World: [Ultralytics](https://github.com/ultralytics/ultralytics)
- Transformers: [HuggingFace](https://huggingface.co/transformers)
## Support
For issues and feature requests, please visit the GitHub repository.
+36
View File
@@ -0,0 +1,36 @@
"""
ComfyUI-Grounding: Object detection nodes for ComfyUI
Supports GroundingDINO, MM-GroundingDINO, and YOLO-World models
"""
from .nodes import (
GroundingDINOModelLoader,
GroundingDINODetector,
YOLOWorldModelLoader,
YOLOWorldDetector,
UnifiedDetector,
BboxVisualizer,
BboxToMask
)
NODE_CLASS_MAPPINGS = {
"GroundingDINOModelLoader": GroundingDINOModelLoader,
"GroundingDINODetector": GroundingDINODetector,
"YOLOWorldModelLoader": YOLOWorldModelLoader,
"YOLOWorldDetector": YOLOWorldDetector,
"UnifiedDetector": UnifiedDetector,
"BboxVisualizer": BboxVisualizer,
"BboxToMask": BboxToMask,
}
NODE_DISPLAY_NAME_MAPPINGS = {
"GroundingDINOModelLoader": "Grounding DINO Model Loader",
"GroundingDINODetector": "Grounding DINO Detector",
"YOLOWorldModelLoader": "YOLO-World Model Loader",
"YOLOWorldDetector": "YOLO-World Detector",
"UnifiedDetector": "Unified Detector (Auto)",
"BboxVisualizer": "Bounding Box Visualizer",
"BboxToMask": "Bounding Box to Mask",
}
__all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS']
+935
View File
@@ -0,0 +1,935 @@
"""
Main node implementations for ComfyUI-Grounding
"""
import torch
import numpy as np
from PIL import Image
import folder_paths
import comfy.model_management as mm
import os
from typing import Tuple, List, Optional
import torchvision.transforms as T
# Global model cache for keeping models in memory
MODEL_CACHE = {}
# Model registry for GroundingDINO variants
GROUNDING_DINO_MODELS = {
# Original GroundingDINO models
"GroundingDINO_SwinT_OGC (694MB)": {
"config": "GroundingDINO_SwinT_OGC.cfg.py",
"checkpoint": "groundingdino_swint_ogc.pth",
"url": "https://huggingface.co/ShilongLiu/GroundingDINO/resolve/main/groundingdino_swint_ogc.pth",
"type": "local",
},
"GroundingDINO_SwinB (938MB)": {
"config": "GroundingDINO_SwinB.cfg.py",
"checkpoint": "groundingdino_swinb_cogcoor.pth",
"url": "https://huggingface.co/ShilongLiu/GroundingDINO/resolve/main/groundingdino_swinb_cogcoor.pth",
"type": "local",
},
# MM-GroundingDINO Tiny models (Swin-T backbone) - 0.2B parameters
"MM-GroundingDINO-Tiny (O365+GoldG, 50.4 mAP)": {
"hf_id": "openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg",
"type": "transformers",
},
"MM-GroundingDINO-Tiny (O365+GoldG+GRIT, 50.5 mAP)": {
"hf_id": "openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_grit",
"type": "transformers",
},
"MM-GroundingDINO-Tiny (O365+GoldG+V3Det, 50.6 mAP)": {
"hf_id": "openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_v3det",
"type": "transformers",
},
"MM-GroundingDINO-Tiny (O365+GoldG+GRIT+V3Det, 50.4 mAP)": {
"hf_id": "openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_grit_v3det",
"type": "transformers",
},
# MM-GroundingDINO Base models (Swin-B backbone) - 0.2B parameters
"MM-GroundingDINO-Base (O365+GoldG+V3Det, 52.5 mAP)": {
"hf_id": "openmmlab-community/mm_grounding_dino_base_o365v1_goldg_v3det",
"type": "transformers",
},
"MM-GroundingDINO-Base (All datasets, 59.5 mAP)": {
"hf_id": "openmmlab-community/mm_grounding_dino_base_all",
"type": "transformers",
},
# MM-GroundingDINO Large models (Swin-L backbone) - 0.3B parameters
"MM-GroundingDINO-Large (O365v2+OIv6+GoldG, 53.0 mAP)": {
"hf_id": "openmmlab-community/mm_grounding_dino_large_o365v2_oiv6_goldg",
"type": "transformers",
},
"MM-GroundingDINO-Large (All datasets, 60.3 mAP)": {
"hf_id": "openmmlab-community/mm_grounding_dino_large_all",
"type": "transformers",
},
}
YOLO_WORLD_MODELS = {
"yolov8s-world": {
"url": "https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8s-world.pt",
},
"yolov8m-world": {
"url": "https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8m-world.pt",
},
"yolov8l-world": {
"url": "https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8l-world.pt",
},
"yolov8x-world": {
"url": "https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x-world.pt",
},
}
class GroundingDINOModelLoader:
"""
Loads GroundingDINO models for open-vocabulary object detection
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"model_name": (list(GROUNDING_DINO_MODELS.keys()), {"default": "GroundingDINO_SwinT_OGC (694MB)"}),
"keep_in_memory": ("BOOLEAN", {"default": True}),
}
}
RETURN_TYPES = ("GROUNDING_DINO_MODEL",)
RETURN_NAMES = ("model",)
FUNCTION = "load_model"
CATEGORY = "grounding"
def load_model(self, model_name: str, keep_in_memory: bool = True):
"""Load GroundingDINO model with caching support"""
# Create cache key
cache_key = f"grounding_dino_{model_name}"
# Check cache if keep_in_memory is True
if keep_in_memory and cache_key in MODEL_CACHE:
print(f"Loading {model_name} from cache")
return (MODEL_CACHE[cache_key],)
# Clear from cache if keep_in_memory is False
if not keep_in_memory and cache_key in MODEL_CACHE:
print(f"Removing {model_name} from cache")
del MODEL_CACHE[cache_key]
# Get model config
model_config = GROUNDING_DINO_MODELS[model_name]
model_type = model_config.get("type", "local")
# Load model based on type
if model_type == "local":
model = self._load_local(model_name, model_config)
else: # transformers
model = self._load_with_transformers(model_name, model_config)
# Cache if requested
if keep_in_memory:
MODEL_CACHE[cache_key] = model
print(f"Cached {model_name} in memory")
return (model,)
def _load_local(self, model_name: str, model_config: dict):
"""Load GroundingDINO using local implementation"""
try:
# Try using the local implementation from comfyui-sam2
from custom_nodes.comfyui_sam2.node import load_groundingdino_model
model = load_groundingdino_model(model_name)
return model
except ImportError:
raise RuntimeError(
f"Local GroundingDINO implementation not found. "
f"Please install ComfyUI-segment-anything-2 to use {model_name}, "
f"or use MM-GroundingDINO models instead."
)
def _load_with_transformers(self, model_name: str, model_config: dict):
"""Load MM-GroundingDINO using HuggingFace transformers"""
try:
from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection
device = mm.get_torch_device()
# Save models to local Models folder in plugin directory
script_dir = os.path.dirname(os.path.abspath(__file__))
models_dir = os.path.join(script_dir, "Models")
os.makedirs(models_dir, exist_ok=True)
# Get HuggingFace model ID
hf_id = model_config.get("hf_id")
if not hf_id:
raise ValueError(f"No HuggingFace model ID found for {model_name}")
print(f"📥 Loading {model_name} from HuggingFace ({hf_id})...")
print("Downloading model files (progress bars will appear below):")
# Load processor and model (automatically shows tqdm progress bars)
processor = AutoProcessor.from_pretrained(hf_id, cache_dir=models_dir)
model = AutoModelForZeroShotObjectDetection.from_pretrained(hf_id, cache_dir=models_dir)
model.to(device)
model.eval()
print(f"✅ Successfully loaded {model_name}")
return {"model": model, "processor": processor, "type": "transformers"}
except Exception as e:
raise RuntimeError(f"Failed to load GroundingDINO model: {e}")
class GroundingDINODetector:
"""
Performs object detection using GroundingDINO with text prompts
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"model": ("GROUNDING_DINO_MODEL",),
"image": ("IMAGE",),
"prompt": ("STRING", {
"default": "person . car . dog .",
"multiline": True,
}),
"box_threshold": ("FLOAT", {
"default": 0.3,
"min": 0.0,
"max": 1.0,
"step": 0.01,
}),
"text_threshold": ("FLOAT", {
"default": 0.25,
"min": 0.0,
"max": 1.0,
"step": 0.01,
}),
},
"optional": {
"single_box_mode": ("BOOLEAN", {
"default": False,
"tooltip": "Return only the highest-scoring detection (REC mode for referring expressions)"
}),
"output_format": (["list_only", "dict_with_data"], {
"default": "list_only",
"tooltip": "list_only: SAM2-compatible list of lists | dict_with_data: dict with boxes/labels/scores"
}),
}
}
RETURN_TYPES = ("BBOX", "IMAGE", "STRING")
RETURN_NAMES = ("bboxes", "annotated_image", "labels")
FUNCTION = "detect"
CATEGORY = "grounding"
def detect(self, model, image, prompt: str, box_threshold: float, text_threshold: float, single_box_mode: bool = False, output_format: str = "list_only"):
"""Run detection on image with text prompt"""
# Process all images in batch
batch_size = image.shape[0]
all_boxes = []
all_labels = []
all_scores = []
annotated_images = []
for i in range(batch_size):
# Convert ComfyUI image format (B, H, W, C) in [0, 1] to PIL
image_np = (image[i].cpu().numpy() * 255).astype(np.uint8)
pil_image = Image.fromarray(image_np)
# Check model type and run detection
if isinstance(model, dict) and model.get("type") == "transformers":
bboxes, annotated_tensor, _ = self._detect_transformers(model, pil_image, prompt, box_threshold, text_threshold, single_box_mode)
else:
bboxes, annotated_tensor, _ = self._detect_local(model, pil_image, prompt, box_threshold, single_box_mode)
# Accumulate results
all_boxes.append(bboxes["boxes"])
all_labels.append(bboxes["labels"])
all_scores.append(bboxes["scores"])
annotated_images.append(annotated_tensor)
# Combine annotated images
annotated_batch = torch.cat(annotated_images, dim=0)
# Format batched output based on output_format
if output_format == "list_only":
# SAM2-compatible format: list of lists (boxes only)
batched_bboxes = all_boxes
else: # dict_with_data
# Dict format with all information
batched_bboxes = {
"boxes": all_boxes,
"labels": all_labels,
"scores": all_scores,
}
# Format labels string (from first image)
if len(all_labels) > 0 and len(all_labels[0]) > 0:
labels_str = ", ".join([f"{label} ({score:.2f})" for label, score in zip(all_labels[0], all_scores[0])])
else:
labels_str = ""
return (batched_bboxes, annotated_batch, labels_str)
def _detect_transformers(self, model_dict, pil_image, prompt, box_threshold, text_threshold, single_box_mode=False):
"""Detection using transformers implementation"""
model = model_dict["model"]
processor = model_dict["processor"]
device = mm.get_torch_device()
# Prepare inputs
inputs = processor(images=pil_image, text=prompt, return_tensors="pt").to(device)
# Run inference
with torch.no_grad():
outputs = model(**inputs)
# Post-process (use 'threshold' instead of 'box_threshold' for GroundingDINO)
results = processor.post_process_grounded_object_detection(
outputs,
inputs.input_ids,
threshold=box_threshold,
text_threshold=text_threshold,
target_sizes=[pil_image.size[::-1]]
)[0]
# Extract boxes and labels
boxes = results["boxes"].cpu().numpy()
labels = results["labels"]
scores = results["scores"].cpu().numpy()
# Override labels if prompt has no separators
# Check for periods and commas (common separators in GroundingDINO prompts)
if '.' not in prompt and ',' not in prompt:
# Use full prompt as label for all detections
clean_prompt = prompt.strip()
labels = [clean_prompt] * len(labels)
# Single box mode: return only highest-scoring detection (REC mode)
if single_box_mode and len(boxes) > 0:
top_idx = scores.argmax()
boxes = boxes[top_idx:top_idx+1]
labels = [labels[top_idx]]
scores = scores[top_idx:top_idx+1]
# Format output
bboxes = {
"boxes": boxes,
"labels": labels,
"scores": scores,
}
# Create annotated image
annotated = self._draw_boxes(pil_image, boxes, labels, scores)
annotated_tensor = torch.from_numpy(np.array(annotated).astype(np.float32) / 255.0).unsqueeze(0)
# Format labels string
labels_str = ", ".join([f"{label} ({score:.2f})" for label, score in zip(labels, scores)])
return (bboxes, annotated_tensor, labels_str)
def _detect_local(self, model, pil_image, prompt, box_threshold, single_box_mode=False):
"""Detection using local GroundingDINO implementation"""
# Use the implementation from comfyui-sam2
from custom_nodes.comfyui_sam2.node import groundingdino_predict
boxes = groundingdino_predict(model, pil_image, prompt, box_threshold)
# Convert to numpy
boxes_np = boxes.cpu().numpy()
# Create labels (simplified - just use prompt)
labels = [prompt.strip().rstrip('.')] * len(boxes_np)
scores = np.ones(len(boxes_np)) * box_threshold # Approximate
# Single box mode: return only first detection (local impl doesn't provide scores)
if single_box_mode and len(boxes_np) > 0:
boxes_np = boxes_np[0:1]
labels = [labels[0]]
scores = scores[0:1]
bboxes = {
"boxes": boxes_np,
"labels": labels,
"scores": scores,
}
# Create annotated image
annotated = self._draw_boxes(pil_image, boxes_np, labels, scores)
annotated_tensor = torch.from_numpy(np.array(annotated).astype(np.float32) / 255.0).unsqueeze(0)
labels_str = ", ".join(labels)
return (bboxes, annotated_tensor, labels_str)
def _draw_boxes(self, image, boxes, labels, scores):
"""Draw bounding boxes on image"""
from PIL import ImageDraw, ImageFont
draw = ImageDraw.Draw(image)
for box, label, score in zip(boxes, labels, scores):
# box format: [x1, y1, x2, y2]
x1, y1, x2, y2 = box
# Draw rectangle
draw.rectangle([x1, y1, x2, y2], outline="red", width=3)
# Draw label
text = f"{label}: {score:.2f}"
draw.text((x1, y1 - 10), text, fill="red")
return image
class YOLOWorldModelLoader:
"""
Loads YOLO-World models for open-vocabulary object detection
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"model_name": (list(YOLO_WORLD_MODELS.keys()), {"default": "yolov8s-world"}),
"keep_in_memory": ("BOOLEAN", {"default": True}),
}
}
RETURN_TYPES = ("YOLO_WORLD_MODEL",)
RETURN_NAMES = ("model",)
FUNCTION = "load_model"
CATEGORY = "grounding"
def load_model(self, model_name: str, keep_in_memory: bool = True):
"""Load YOLO-World model with caching support"""
# Create cache key
cache_key = f"yolo_world_{model_name}"
# Check cache if keep_in_memory is True
if keep_in_memory and cache_key in MODEL_CACHE:
print(f"Loading {model_name} from cache")
return (MODEL_CACHE[cache_key],)
# Clear from cache if keep_in_memory is False
if not keep_in_memory and cache_key in MODEL_CACHE:
print(f"Removing {model_name} from cache")
del MODEL_CACHE[cache_key]
try:
from ultralytics import YOLOWorld
# Save models to local Models folder in plugin directory
script_dir = os.path.dirname(os.path.abspath(__file__))
models_dir = os.path.join(script_dir, "Models")
os.makedirs(models_dir, exist_ok=True)
model_filename = f"{model_name}.pt"
model_path = os.path.join(models_dir, model_filename)
# Download if needed
if not os.path.exists(model_path):
print(f"📥 Downloading {model_name} to {models_dir}...")
print("Progress bar:")
from torch.hub import download_url_to_file
download_url_to_file(
YOLO_WORLD_MODELS[model_name]["url"],
model_path,
progress=True # Explicitly enable progress bar
)
print(f"✅ Downloaded {model_name} successfully")
# Load model
print(f"Loading {model_name} from {model_path}")
model = YOLOWorld(model_path)
# Cache if requested
if keep_in_memory:
MODEL_CACHE[cache_key] = model
print(f"Cached {model_name} in memory")
return (model,)
except ImportError:
raise RuntimeError("ultralytics not installed. Run: pip install ultralytics")
except Exception as e:
raise RuntimeError(f"Failed to load YOLO-World model: {e}")
class YOLOWorldDetector:
"""
Performs object detection using YOLO-World with text prompts
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"model": ("YOLO_WORLD_MODEL",),
"image": ("IMAGE",),
"classes": ("STRING", {
"default": "person, car, dog",
"multiline": True,
}),
"confidence_threshold": ("FLOAT", {
"default": 0.3,
"min": 0.0,
"max": 1.0,
"step": 0.01,
}),
},
"optional": {
"single_box_mode": ("BOOLEAN", {
"default": False,
"tooltip": "Return only the highest-scoring detection (REC mode for referring expressions)"
}),
"output_format": (["list_only", "dict_with_data"], {
"default": "list_only",
"tooltip": "list_only: SAM2-compatible list of lists | dict_with_data: dict with boxes/labels/scores"
}),
}
}
RETURN_TYPES = ("BBOX", "IMAGE", "STRING")
RETURN_NAMES = ("bboxes", "annotated_image", "labels")
FUNCTION = "detect"
CATEGORY = "grounding"
def detect(self, model, image, classes: str, confidence_threshold: float, single_box_mode: bool = False, output_format: str = "list_only"):
"""Run detection with YOLO-World"""
# Process all images in batch
batch_size = image.shape[0]
all_boxes = []
all_labels = []
all_scores = []
annotated_images = []
# Parse classes - check if we should use full text as label
has_separator = ',' in classes
if has_separator:
class_list = [c.strip() for c in classes.split(",")]
else:
# No comma separator - treat as single class/referring expression
class_list = [classes.strip()]
# Set classes for the model
model.set_classes(class_list)
for i in range(batch_size):
# Convert ComfyUI image format
image_np = (image[i].cpu().numpy() * 255).astype(np.uint8)
# Run inference
results = model(image_np, conf=confidence_threshold)
# Extract detections
result = results[0]
boxes = result.boxes.xyxy.cpu().numpy() if result.boxes is not None else np.array([])
scores = result.boxes.conf.cpu().numpy() if result.boxes is not None else np.array([])
class_ids = result.boxes.cls.cpu().numpy().astype(int) if result.boxes is not None else np.array([])
# Map class IDs to names
labels = [class_list[cid] for cid in class_ids]
# Override labels if no separator (referring expression mode)
if not has_separator and len(labels) > 0:
# Use full prompt as label for all detections
labels = [classes.strip()] * len(labels)
# Single box mode: return only highest-scoring detection (REC mode)
if single_box_mode and len(boxes) > 0:
top_idx = scores.argmax()
boxes = boxes[top_idx:top_idx+1]
labels = [labels[top_idx]]
scores = scores[top_idx:top_idx+1]
# Get annotated image
annotated_np = result.plot()
annotated_tensor = torch.from_numpy(annotated_np.astype(np.float32) / 255.0).unsqueeze(0)
# Accumulate results
all_boxes.append(boxes)
all_labels.append(labels)
all_scores.append(scores)
annotated_images.append(annotated_tensor)
# Combine annotated images
annotated_batch = torch.cat(annotated_images, dim=0)
# Format batched output based on output_format
if output_format == "list_only":
# SAM2-compatible format: list of lists (boxes only)
batched_bboxes = all_boxes
else: # dict_with_data
# Dict format with all information
batched_bboxes = {
"boxes": all_boxes,
"labels": all_labels,
"scores": all_scores,
}
# Format labels string (from first image)
if len(all_labels) > 0 and len(all_labels[0]) > 0:
labels_str = ", ".join([f"{label} ({score:.2f})" for label, score in zip(all_labels[0], all_scores[0])])
else:
labels_str = ""
return (batched_bboxes, annotated_batch, labels_str)
class BboxVisualizer:
"""
Visualizes bounding boxes on images
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"image": ("IMAGE",),
"bboxes": ("BBOX",),
},
"optional": {
"line_width": ("INT", {"default": 3, "min": 1, "max": 10}),
}
}
RETURN_TYPES = ("IMAGE",)
RETURN_NAMES = ("annotated_image",)
FUNCTION = "visualize"
CATEGORY = "grounding"
def visualize(self, image, bboxes, line_width=3):
"""Draw bounding boxes on image"""
from PIL import ImageDraw
# Convert to PIL
image_np = (image[0].cpu().numpy() * 255).astype(np.uint8)
pil_image = Image.fromarray(image_np)
draw = ImageDraw.Draw(pil_image)
boxes = bboxes["boxes"]
labels = bboxes.get("labels", [])
scores = bboxes.get("scores", [])
for i, box in enumerate(boxes):
x1, y1, x2, y2 = box
draw.rectangle([x1, y1, x2, y2], outline="red", width=line_width)
if i < len(labels):
label = labels[i]
score = scores[i] if i < len(scores) else 0.0
text = f"{label}: {score:.2f}"
draw.text((x1, y1 - 10), text, fill="red")
# Convert back to tensor
annotated_np = np.array(pil_image).astype(np.float32) / 255.0
annotated_tensor = torch.from_numpy(annotated_np).unsqueeze(0)
return (annotated_tensor,)
class BboxToMask:
"""
Converts bounding boxes to masks
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"image": ("IMAGE",),
"bboxes": ("BBOX",),
},
"optional": {
"combine_masks": ("BOOLEAN", {"default": False}),
}
}
RETURN_TYPES = ("MASK",)
RETURN_NAMES = ("masks",)
FUNCTION = "bbox_to_mask"
CATEGORY = "grounding"
def bbox_to_mask(self, image, bboxes, combine_masks=False):
"""Convert bounding boxes to binary masks"""
# Get image dimensions
batch_size, height, width, channels = image.shape
boxes = bboxes["boxes"]
if combine_masks:
# Single combined mask
mask = np.zeros((height, width), dtype=np.float32)
for box in boxes:
x1, y1, x2, y2 = box.astype(int)
x1, y1 = max(0, x1), max(0, y1)
x2, y2 = min(width, x2), min(height, y2)
mask[y1:y2, x1:x2] = 1.0
mask_tensor = torch.from_numpy(mask).unsqueeze(0)
else:
# Individual masks for each box
masks = []
for box in boxes:
mask = np.zeros((height, width), dtype=np.float32)
x1, y1, x2, y2 = box.astype(int)
x1, y1 = max(0, x1), max(0, y1)
x2, y2 = min(width, x2), min(height, y2)
mask[y1:y2, x1:x2] = 1.0
masks.append(mask)
if len(masks) == 0:
# No detections - return empty mask
mask_tensor = torch.zeros((1, height, width))
else:
mask_tensor = torch.from_numpy(np.stack(masks))
return (mask_tensor,)
class UnifiedDetector:
"""
Universal detector that works with both GroundingDINO and YOLO-World models
Automatically detects model type and uses appropriate inference method
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"model": ("GROUNDING_DINO_MODEL,YOLO_WORLD_MODEL",),
"image": ("IMAGE",),
"prompt": ("STRING", {
"default": "person . car . dog .",
"multiline": True,
"tooltip": "For GroundingDINO: period-separated (e.g., 'cat . dog .'). For YOLO-World: comma-separated (e.g., 'cat, dog')"
}),
"threshold": ("FLOAT", {
"default": 0.3,
"min": 0.0,
"max": 1.0,
"step": 0.01,
}),
},
"optional": {
"text_threshold": ("FLOAT", {
"default": 0.25,
"min": 0.0,
"max": 1.0,
"step": 0.01,
"tooltip": "Only used for GroundingDINO models"
}),
"single_box_mode": ("BOOLEAN", {
"default": False,
"tooltip": "Return only the highest-scoring detection (REC mode for referring expressions)"
}),
}
}
RETURN_TYPES = ("BBOX", "IMAGE", "STRING")
RETURN_NAMES = ("bboxes", "annotated_image", "labels")
FUNCTION = "detect"
CATEGORY = "grounding"
def detect(self, model, image, prompt: str, threshold: float, text_threshold: float = 0.25, single_box_mode: bool = False):
"""Universal detection that auto-detects model type"""
# Detect model type
model_type = self._detect_model_type(model)
if model_type == "yolo_world":
return self._detect_yolo_world(model, image, prompt, threshold, single_box_mode)
else: # grounding_dino or transformers
return self._detect_grounding_dino(model, image, prompt, threshold, text_threshold, single_box_mode)
def _detect_model_type(self, model):
"""Detect whether this is a YOLO-World or GroundingDINO model"""
# Check for YOLO-World model
try:
from ultralytics import YOLOWorld
if isinstance(model, YOLOWorld):
return "yolo_world"
except ImportError:
pass
# Check for transformers-based model
if isinstance(model, dict) and model.get("type") == "transformers":
return "grounding_dino_transformers"
# Default to GroundingDINO (local)
return "grounding_dino_local"
def _detect_grounding_dino(self, model, image, prompt, box_threshold, text_threshold, single_box_mode=False):
"""Detection using GroundingDINO"""
# Convert ComfyUI image format
image_np = (image[0].cpu().numpy() * 255).astype(np.uint8)
pil_image = Image.fromarray(image_np)
# Check model type
if isinstance(model, dict) and model.get("type") == "transformers":
return self._detect_transformers(model, pil_image, prompt, box_threshold, text_threshold, single_box_mode)
else:
return self._detect_local(model, pil_image, prompt, box_threshold, single_box_mode)
def _detect_transformers(self, model_dict, pil_image, prompt, box_threshold, text_threshold, single_box_mode=False):
"""Detection using transformers implementation"""
model = model_dict["model"]
processor = model_dict["processor"]
device = mm.get_torch_device()
# Prepare inputs
inputs = processor(images=pil_image, text=prompt, return_tensors="pt").to(device)
# Run inference
with torch.no_grad():
outputs = model(**inputs)
# Post-process (use 'threshold' instead of 'box_threshold' for GroundingDINO)
results = processor.post_process_grounded_object_detection(
outputs,
inputs.input_ids,
threshold=box_threshold,
text_threshold=text_threshold,
target_sizes=[pil_image.size[::-1]]
)[0]
# Extract boxes and labels
boxes = results["boxes"].cpu().numpy()
labels = results["labels"]
scores = results["scores"].cpu().numpy()
# Single box mode: return only highest-scoring detection (REC mode)
if single_box_mode and len(boxes) > 0:
top_idx = scores.argmax()
boxes = boxes[top_idx:top_idx+1]
labels = [labels[top_idx]]
scores = scores[top_idx:top_idx+1]
# Format output
bboxes = {
"boxes": boxes,
"labels": labels,
"scores": scores,
}
# Create annotated image
annotated = self._draw_boxes(pil_image, boxes, labels, scores)
annotated_tensor = torch.from_numpy(np.array(annotated).astype(np.float32) / 255.0).unsqueeze(0)
# Format labels string
labels_str = ", ".join([f"{label} ({score:.2f})" for label, score in zip(labels, scores)])
return (bboxes, annotated_tensor, labels_str)
def _detect_local(self, model, pil_image, prompt, box_threshold, single_box_mode=False):
"""Detection using local GroundingDINO implementation"""
from custom_nodes.comfyui_sam2.node import groundingdino_predict
boxes = groundingdino_predict(model, pil_image, prompt, box_threshold)
# Convert to numpy
boxes_np = boxes.cpu().numpy()
# Create labels
labels = [prompt.strip().rstrip('.')] * len(boxes_np)
scores = np.ones(len(boxes_np)) * box_threshold
# Single box mode: return only first detection (local impl doesn't provide scores)
if single_box_mode and len(boxes_np) > 0:
boxes_np = boxes_np[0:1]
labels = [labels[0]]
scores = scores[0:1]
bboxes = {
"boxes": boxes_np,
"labels": labels,
"scores": scores,
}
# Create annotated image
annotated = self._draw_boxes(pil_image, boxes_np, labels, scores)
annotated_tensor = torch.from_numpy(np.array(annotated).astype(np.float32) / 255.0).unsqueeze(0)
labels_str = ", ".join(labels)
return (bboxes, annotated_tensor, labels_str)
def _detect_yolo_world(self, model, image, classes, confidence_threshold, single_box_mode=False):
"""Detection using YOLO-World"""
# Convert ComfyUI image format
image_np = (image[0].cpu().numpy() * 255).astype(np.uint8)
# Parse classes (support both comma and period separation)
if ',' in classes:
class_list = [c.strip() for c in classes.split(",")]
else:
class_list = [c.strip() for c in classes.split(".") if c.strip()]
# Set classes for the model
model.set_classes(class_list)
# Run inference
results = model(image_np, conf=confidence_threshold)
# Extract detections
result = results[0]
boxes = result.boxes.xyxy.cpu().numpy() if result.boxes is not None else np.array([])
scores = result.boxes.conf.cpu().numpy() if result.boxes is not None else np.array([])
class_ids = result.boxes.cls.cpu().numpy().astype(int) if result.boxes is not None else np.array([])
# Map class IDs to names
labels = [class_list[cid] for cid in class_ids]
# Single box mode: return only highest-scoring detection (REC mode)
if single_box_mode and len(boxes) > 0:
top_idx = scores.argmax()
boxes = boxes[top_idx:top_idx+1]
labels = [labels[top_idx]]
scores = scores[top_idx:top_idx+1]
# Format output
bboxes = {
"boxes": boxes,
"labels": labels,
"scores": scores,
}
# Get annotated image
annotated_np = result.plot()
annotated_tensor = torch.from_numpy(annotated_np.astype(np.float32) / 255.0).unsqueeze(0)
# Format labels string
labels_str = ", ".join([f"{label} ({score:.2f})" for label, score in zip(labels, scores)])
return (bboxes, annotated_tensor, labels_str)
def _draw_boxes(self, image, boxes, labels, scores):
"""Draw bounding boxes on image"""
from PIL import ImageDraw
# Make a copy to avoid modifying original
image = image.copy()
draw = ImageDraw.Draw(image)
for box, label, score in zip(boxes, labels, scores):
# box format: [x1, y1, x2, y2]
x1, y1, x2, y2 = box
# Draw rectangle
draw.rectangle([x1, y1, x2, y2], outline="red", width=3)
# Draw label
text = f"{label}: {score:.2f}"
draw.text((x1, y1 - 10), text, fill="red")
return image
+15
View File
@@ -0,0 +1,15 @@
# Core dependencies for ComfyUI-Grounding
# For GroundingDINO (HuggingFace transformers implementation)
transformers>=4.30.0
timm>=0.9.0
# For YOLO-World
ultralytics>=8.2.0
# Image processing
opencv-python>=4.8.0
pillow>=10.0.0
# Optional: supervision for advanced visualization
# supervision>=0.16.0