first draft
This commit is contained in:
@@ -1,2 +1,354 @@
|
||||
# ComfyUI-Grounding
|
||||
ComfyUI node to run grounding models
|
||||
|
||||
ComfyUI custom nodes for open-vocabulary object detection using state-of-the-art grounding models.
|
||||
|
||||
## Features
|
||||
|
||||
- **GroundingDINO**: Open-vocabulary object detection with natural language prompts
|
||||
- **YOLO-World**: Real-time open-vocabulary detection
|
||||
- **Batch Processing**: Process multiple images at once for improved efficiency
|
||||
- **Referring Expression Comprehension (REC)**: Use full text as label when no separators present
|
||||
- **Flexible Architecture**: Support for both local and HuggingFace implementations
|
||||
- **Utility Nodes**: Bounding box visualization and mask conversion
|
||||
|
||||
## Supported Models
|
||||
|
||||
### GroundingDINO (Original)
|
||||
- GroundingDINO-T (SwinT backbone, 694MB)
|
||||
- GroundingDINO-B (SwinB backbone, 938MB)
|
||||
|
||||
### MM-GroundingDINO (Multimodal)
|
||||
**Tiny Models (Swin-T backbone):**
|
||||
- MM-GroundingDINO-Tiny (O365+GoldG, 50.4 mAP)
|
||||
- MM-GroundingDINO-Tiny (O365+GoldG+GRIT, 50.5 mAP)
|
||||
- MM-GroundingDINO-Tiny (O365+GoldG+V3Det, 50.6 mAP)
|
||||
- MM-GroundingDINO-Tiny (O365+GoldG+GRIT+V3Det, 50.4 mAP)
|
||||
|
||||
**Base Models (Swin-B backbone):**
|
||||
- MM-GroundingDINO-Base (O365+GoldG+V3Det, 52.5 mAP)
|
||||
- MM-GroundingDINO-Base (All datasets, 59.5 mAP) ⭐ Recommended
|
||||
|
||||
**Large Models (Swin-L backbone):**
|
||||
- MM-GroundingDINO-Large (O365v2+OIv6+GoldG, 53.0 mAP)
|
||||
- MM-GroundingDINO-Large (All datasets, 60.3 mAP) ⭐ Best accuracy
|
||||
|
||||
### YOLO-World
|
||||
- YOLOv8s-World (small)
|
||||
- YOLOv8m-World (medium)
|
||||
- YOLOv8l-World (large)
|
||||
- YOLOv8x-World (extra large)
|
||||
|
||||
## Installation
|
||||
|
||||
1. Navigate to your ComfyUI custom nodes directory:
|
||||
```bash
|
||||
cd ComfyUI/custom_nodes/
|
||||
```
|
||||
|
||||
2. Clone or copy this repository:
|
||||
```bash
|
||||
git clone <repository-url> ComfyUI-Grounding
|
||||
```
|
||||
|
||||
3. Install dependencies:
|
||||
```bash
|
||||
cd ComfyUI-Grounding
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Nodes
|
||||
|
||||
### 1. GroundingDINOModelLoader
|
||||
Loads a GroundingDINO or MM-GroundingDINO model for object detection.
|
||||
|
||||
**Inputs:**
|
||||
- `model_name`: Select from available GroundingDINO/MM-GroundingDINO models
|
||||
- `keep_in_memory`: Cache model in memory for faster reuse (default: True)
|
||||
|
||||
**Outputs:**
|
||||
- `model`: Loaded GroundingDINO model
|
||||
|
||||
**Features:**
|
||||
- ✅ Model caching - loads instantly on second use
|
||||
- ✅ Local model storage in `ComfyUI-Grounding/Models/`
|
||||
- ✅ Supports both original and MM-GroundingDINO variants
|
||||
|
||||
### 2. GroundingDINODetector
|
||||
Detects objects in images using text prompts.
|
||||
|
||||
**Inputs:**
|
||||
- `model`: GroundingDINO model (from loader)
|
||||
- `image`: Input image (supports batches)
|
||||
- `prompt`: Text description of objects to detect (e.g., "person . car . dog .")
|
||||
- `box_threshold`: Confidence threshold for bounding boxes (0.0-1.0)
|
||||
- `text_threshold`: Confidence threshold for text matching (0.0-1.0)
|
||||
- `single_box_mode`: (Optional) Return only highest-scoring detection for REC
|
||||
- `output_format`: (Optional) "list_only" (default, SAM2-compatible) or "dict_with_data"
|
||||
|
||||
**Outputs:**
|
||||
- `bboxes`: Detected bounding boxes (format depends on output_format setting)
|
||||
- `list_only`: List of numpy arrays `[[boxes_img0], [boxes_img1], ...]` - SAM2-compatible
|
||||
- `dict_with_data`: Dict `{"boxes": [...], "labels": [...], "scores": [...]}`
|
||||
- `annotated_image`: Image with drawn bounding boxes (batched)
|
||||
- `labels`: String with detected object labels
|
||||
|
||||
**Features:**
|
||||
- ✅ Batch processing - process multiple images at once
|
||||
- ✅ Automatic REC mode - when prompt has no separators (. or ,), uses full text as label
|
||||
- ✅ Single box mode - returns only highest-scoring detection for referring expressions
|
||||
- ✅ Flexible output format - choose between SAM2-compatible list or full dict with metadata
|
||||
|
||||
### 3. YOLOWorldModelLoader
|
||||
Loads a YOLO-World model for real-time detection.
|
||||
|
||||
**Inputs:**
|
||||
- `model_name`: Select from available YOLO-World models
|
||||
- `keep_in_memory`: Cache model in memory for faster reuse (default: True)
|
||||
|
||||
**Outputs:**
|
||||
- `model`: Loaded YOLO-World model
|
||||
|
||||
**Features:**
|
||||
- ✅ Model caching - loads instantly on second use
|
||||
- ✅ Local model storage in `ComfyUI-Grounding/Models/`
|
||||
- ✅ Auto-downloads models on first use
|
||||
|
||||
### 4. YOLOWorldDetector
|
||||
Detects objects using YOLO-World with custom classes.
|
||||
|
||||
**Inputs:**
|
||||
- `model`: YOLO-World model (from loader)
|
||||
- `image`: Input image (supports batches)
|
||||
- `classes`: Comma-separated list of classes (e.g., "person, car, dog")
|
||||
- `confidence_threshold`: Minimum confidence for detections (0.0-1.0)
|
||||
- `single_box_mode`: (Optional) Return only highest-scoring detection for REC
|
||||
- `output_format`: (Optional) "list_only" (default, SAM2-compatible) or "dict_with_data"
|
||||
|
||||
**Outputs:**
|
||||
- `bboxes`: Detected bounding boxes (format depends on output_format setting)
|
||||
- `list_only`: List of numpy arrays `[[boxes_img0], [boxes_img1], ...]` - SAM2-compatible
|
||||
- `dict_with_data`: Dict `{"boxes": [...], "labels": [...], "scores": [...]}`
|
||||
- `annotated_image`: Image with drawn bounding boxes (batched)
|
||||
- `labels`: String with detected object labels
|
||||
|
||||
**Features:**
|
||||
- ✅ Batch processing - process multiple images at once
|
||||
- ✅ Automatic REC mode - when classes has no comma, uses full text as label
|
||||
- ✅ Single box mode - returns only highest-scoring detection for referring expressions
|
||||
- ✅ Flexible output format - choose between SAM2-compatible list or full dict with metadata
|
||||
|
||||
### 5. BboxVisualizer
|
||||
Draws bounding boxes on images.
|
||||
|
||||
**Inputs:**
|
||||
- `image`: Input image
|
||||
- `bboxes`: Bounding box data
|
||||
- `line_width`: Thickness of bounding box lines (optional)
|
||||
|
||||
**Outputs:**
|
||||
- `annotated_image`: Image with drawn bounding boxes
|
||||
|
||||
### 5. UnifiedDetector ⭐ NEW
|
||||
Universal detector that works with BOTH GroundingDINO and YOLO-World models.
|
||||
|
||||
**Inputs:**
|
||||
- `model`: Any model (GroundingDINO or YOLO-World)
|
||||
- `image`: Input image
|
||||
- `prompt`: Text prompt (supports both formats)
|
||||
- For GroundingDINO: "cat . dog . car ."
|
||||
- For YOLO-World: "cat, dog, car"
|
||||
- `threshold`: Confidence threshold (0.0-1.0)
|
||||
- `text_threshold`: Text matching threshold (optional, GroundingDINO only)
|
||||
|
||||
**Outputs:**
|
||||
- `bboxes`: Detected bounding boxes with labels and scores
|
||||
- `annotated_image`: Image with drawn bounding boxes
|
||||
- `labels`: String with detected object labels
|
||||
|
||||
**Features:**
|
||||
- ✅ Auto-detects model type
|
||||
- ✅ Single node for all detection models
|
||||
- ✅ Simplified workflow
|
||||
|
||||
### 6. BboxToMask
|
||||
Converts bounding boxes to binary masks.
|
||||
|
||||
**Inputs:**
|
||||
- `image`: Input image (for dimensions)
|
||||
- `bboxes`: Bounding box data
|
||||
- `combine_masks`: Whether to combine all boxes into one mask (optional)
|
||||
|
||||
**Outputs:**
|
||||
- `masks`: Binary mask(s) for detected regions
|
||||
|
||||
## Key Features
|
||||
|
||||
### 🚀 Model Caching
|
||||
Models are cached in memory by default for instant reuse:
|
||||
- First load: Downloads and initializes model
|
||||
- Second load: Instant loading from cache
|
||||
- Toggle `keep_in_memory=False` to clear cache
|
||||
|
||||
### 📁 Local Model Storage
|
||||
All models are downloaded to the plugin's local Models directory:
|
||||
- Location: `ComfyUI/custom_nodes/ComfyUI-Grounding/Models/`
|
||||
- Auto-created on first use with a `models_go_here.txt` placeholder
|
||||
- GroundingDINO/MM-GroundingDINO: Stored in HuggingFace transformers cache format
|
||||
- YOLO-World: Stored as `.pt` files (yolov8s-world.pt, etc.)
|
||||
|
||||
### 🎯 UnifiedDetector
|
||||
One detector for all models - automatically detects whether you're using GroundingDINO or YOLO-World and routes to the appropriate inference method.
|
||||
|
||||
### 📦 Batch Processing
|
||||
All detector nodes support batch processing:
|
||||
- Process multiple images in a single forward pass
|
||||
- Input images as a batch tensor (B, H, W, C)
|
||||
- Returns batched bounding boxes with lists of detections for each image
|
||||
- Annotated images are returned as a single batch tensor
|
||||
|
||||
### 🎯 Referring Expression Comprehension (REC)
|
||||
Automatic REC mode for natural language queries:
|
||||
- **GroundingDINO**: If prompt has no periods (`.`) or commas (`,`), uses full text as label
|
||||
- Example: `"main object in the center of the picture"` → all detections labeled with full text
|
||||
- With separators: `"person . car . dog ."` → uses standard noun phrase extraction
|
||||
- **YOLO-World**: If classes has no comma (`,`), uses full text as label
|
||||
- Example: `"red car in the parking lot"` → all detections labeled with full text
|
||||
- With separators: `"person, car, dog"` → uses standard class mapping
|
||||
- **Single Box Mode**: Enable `single_box_mode` to return only the highest-scoring detection (perfect for REC tasks)
|
||||
|
||||
### 📋 Output Formats
|
||||
Detector nodes support two output formats for bounding boxes:
|
||||
|
||||
**1. `list_only` (Default - SAM2 Compatible)**
|
||||
- Returns: List of numpy arrays, one per image in batch
|
||||
- Format: `[[bbox1, bbox2], [bbox3], ...]` where each bbox is `[x1, y1, x2, y2]`
|
||||
- Use when: Connecting to SAM2 segmentation nodes
|
||||
- Example:
|
||||
```python
|
||||
# For 2 images with 2 and 1 detections respectively
|
||||
[
|
||||
array([[10, 20, 100, 200], [50, 60, 150, 250]]), # Image 0: 2 boxes
|
||||
array([[30, 40, 120, 180]]) # Image 1: 1 box
|
||||
]
|
||||
```
|
||||
|
||||
**2. `dict_with_data` (Full Metadata)**
|
||||
- Returns: Dictionary with boxes, labels, and scores
|
||||
- Format: `{"boxes": [...], "labels": [...], "scores": [...]}`
|
||||
- Use when: Need label and confidence information for downstream processing
|
||||
- NOT compatible with SAM2 (dict iteration gives keys, not values)
|
||||
- Example:
|
||||
```python
|
||||
{
|
||||
"boxes": [array([...]), array([...])], # Boxes per image
|
||||
"labels": [["cat", "dog"], ["bird"]], # Labels per image
|
||||
"scores": [array([0.9, 0.8]), array([0.7])] # Scores per image
|
||||
}
|
||||
```
|
||||
|
||||
**Recommendation:** Use `list_only` (default) unless you specifically need label/score metadata for your workflow.
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Example 1: Basic Object Detection with GroundingDINO
|
||||
|
||||
1. Add `GroundingDINOModelLoader` node
|
||||
2. Select model (e.g., "GroundingDINO_SwinT_OGC (694MB)")
|
||||
3. Add `GroundingDINODetector` node
|
||||
4. Connect model output to detector
|
||||
5. Load an image and connect to detector
|
||||
6. Set prompt: "person . car . bicycle ."
|
||||
7. Adjust thresholds as needed
|
||||
8. View annotated image output
|
||||
|
||||
### Example 2: YOLO-World Detection
|
||||
|
||||
1. Add `YOLOWorldModelLoader` node
|
||||
2. Select model (e.g., "yolov8s-world")
|
||||
3. Add `YOLOWorldDetector` node
|
||||
4. Connect model and image
|
||||
5. Set classes: "cat, dog, bird, horse"
|
||||
6. Adjust confidence threshold
|
||||
7. View results
|
||||
|
||||
### Example 3: Using UnifiedDetector (Recommended)
|
||||
|
||||
1. Add any model loader (GroundingDINO or YOLO-World)
|
||||
2. Load your model
|
||||
3. Add `UnifiedDetector` node
|
||||
4. Connect model and image
|
||||
5. Set prompt (works with both formats!)
|
||||
6. The node automatically detects model type and uses the right inference
|
||||
|
||||
### Example 4: MM-GroundingDINO for High Accuracy
|
||||
|
||||
1. Add `GroundingDINOModelLoader` node
|
||||
2. Select "MM-GroundingDINO-Base (All datasets, 59.5 mAP)"
|
||||
3. Enable `keep_in_memory=True` (default)
|
||||
4. Add `GroundingDINODetector` or `UnifiedDetector`
|
||||
5. Set prompt: "specific object . another object ."
|
||||
6. Enjoy state-of-the-art detection accuracy!
|
||||
|
||||
### Example 5: Convert Detections to Masks
|
||||
|
||||
1. Use any detector node (GroundingDINO or YOLO-World)
|
||||
2. Add `BboxToMask` node
|
||||
3. Connect image and bboxes outputs
|
||||
4. Choose whether to combine masks
|
||||
5. Use masks for segmentation, inpainting, etc.
|
||||
|
||||
## Model Downloads
|
||||
|
||||
Models will be automatically downloaded on first use:
|
||||
- **All models**: Stored in `ComfyUI/custom_nodes/ComfyUI-Grounding/Models/`
|
||||
- **GroundingDINO/MM-GroundingDINO**: HuggingFace transformers cache format
|
||||
- **YOLO-World**: Direct `.pt` files (e.g., yolov8s-world.pt)
|
||||
|
||||
## Tips
|
||||
|
||||
- **GroundingDINO prompts**: End each object with a period (e.g., "cat . dog . car .")
|
||||
- **YOLO-World classes**: Use comma-separated values (e.g., "cat, dog, car")
|
||||
- **Threshold tuning**: Start with 0.3 and adjust based on results
|
||||
- Lower threshold = more detections (including false positives)
|
||||
- Higher threshold = fewer, more confident detections
|
||||
- **Performance**: YOLO-World is faster but GroundingDINO may be more accurate
|
||||
|
||||
## Integration with SAM2
|
||||
|
||||
These nodes work seamlessly with SAM2 for instance segmentation:
|
||||
|
||||
1. Use GroundingDINO or YOLO-World to detect objects
|
||||
2. Convert bboxes to masks using `BboxToMask`
|
||||
3. Use masks with SAM2 for precise segmentation
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### ImportError: transformers not found
|
||||
```bash
|
||||
pip install transformers timm
|
||||
```
|
||||
|
||||
### ImportError: ultralytics not found
|
||||
```bash
|
||||
pip install ultralytics
|
||||
```
|
||||
|
||||
### Models not downloading
|
||||
Check your internet connection and ensure you have write permissions to the ComfyUI models directory.
|
||||
|
||||
### CUDA out of memory
|
||||
Try using a smaller model variant (e.g., GroundingDINO-T instead of -B, or yolov8s instead of yolov8x).
|
||||
|
||||
## License
|
||||
|
||||
MIT License - See LICENSE file for details
|
||||
|
||||
## Credits
|
||||
|
||||
- GroundingDINO: [IDEA-Research](https://github.com/IDEA-Research/GroundingDINO)
|
||||
- YOLO-World: [Ultralytics](https://github.com/ultralytics/ultralytics)
|
||||
- Transformers: [HuggingFace](https://huggingface.co/transformers)
|
||||
|
||||
## Support
|
||||
|
||||
For issues and feature requests, please visit the GitHub repository.
|
||||
|
||||
+36
@@ -0,0 +1,36 @@
|
||||
"""
|
||||
ComfyUI-Grounding: Object detection nodes for ComfyUI
|
||||
Supports GroundingDINO, MM-GroundingDINO, and YOLO-World models
|
||||
"""
|
||||
|
||||
from .nodes import (
|
||||
GroundingDINOModelLoader,
|
||||
GroundingDINODetector,
|
||||
YOLOWorldModelLoader,
|
||||
YOLOWorldDetector,
|
||||
UnifiedDetector,
|
||||
BboxVisualizer,
|
||||
BboxToMask
|
||||
)
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"GroundingDINOModelLoader": GroundingDINOModelLoader,
|
||||
"GroundingDINODetector": GroundingDINODetector,
|
||||
"YOLOWorldModelLoader": YOLOWorldModelLoader,
|
||||
"YOLOWorldDetector": YOLOWorldDetector,
|
||||
"UnifiedDetector": UnifiedDetector,
|
||||
"BboxVisualizer": BboxVisualizer,
|
||||
"BboxToMask": BboxToMask,
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"GroundingDINOModelLoader": "Grounding DINO Model Loader",
|
||||
"GroundingDINODetector": "Grounding DINO Detector",
|
||||
"YOLOWorldModelLoader": "YOLO-World Model Loader",
|
||||
"YOLOWorldDetector": "YOLO-World Detector",
|
||||
"UnifiedDetector": "Unified Detector (Auto)",
|
||||
"BboxVisualizer": "Bounding Box Visualizer",
|
||||
"BboxToMask": "Bounding Box to Mask",
|
||||
}
|
||||
|
||||
__all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS']
|
||||
@@ -0,0 +1,935 @@
|
||||
"""
|
||||
Main node implementations for ComfyUI-Grounding
|
||||
"""
|
||||
|
||||
import torch
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
import folder_paths
|
||||
import comfy.model_management as mm
|
||||
import os
|
||||
from typing import Tuple, List, Optional
|
||||
import torchvision.transforms as T
|
||||
|
||||
# Global model cache for keeping models in memory
|
||||
MODEL_CACHE = {}
|
||||
|
||||
# Model registry for GroundingDINO variants
|
||||
GROUNDING_DINO_MODELS = {
|
||||
# Original GroundingDINO models
|
||||
"GroundingDINO_SwinT_OGC (694MB)": {
|
||||
"config": "GroundingDINO_SwinT_OGC.cfg.py",
|
||||
"checkpoint": "groundingdino_swint_ogc.pth",
|
||||
"url": "https://huggingface.co/ShilongLiu/GroundingDINO/resolve/main/groundingdino_swint_ogc.pth",
|
||||
"type": "local",
|
||||
},
|
||||
"GroundingDINO_SwinB (938MB)": {
|
||||
"config": "GroundingDINO_SwinB.cfg.py",
|
||||
"checkpoint": "groundingdino_swinb_cogcoor.pth",
|
||||
"url": "https://huggingface.co/ShilongLiu/GroundingDINO/resolve/main/groundingdino_swinb_cogcoor.pth",
|
||||
"type": "local",
|
||||
},
|
||||
# MM-GroundingDINO Tiny models (Swin-T backbone) - 0.2B parameters
|
||||
"MM-GroundingDINO-Tiny (O365+GoldG, 50.4 mAP)": {
|
||||
"hf_id": "openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg",
|
||||
"type": "transformers",
|
||||
},
|
||||
"MM-GroundingDINO-Tiny (O365+GoldG+GRIT, 50.5 mAP)": {
|
||||
"hf_id": "openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_grit",
|
||||
"type": "transformers",
|
||||
},
|
||||
"MM-GroundingDINO-Tiny (O365+GoldG+V3Det, 50.6 mAP)": {
|
||||
"hf_id": "openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_v3det",
|
||||
"type": "transformers",
|
||||
},
|
||||
"MM-GroundingDINO-Tiny (O365+GoldG+GRIT+V3Det, 50.4 mAP)": {
|
||||
"hf_id": "openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_grit_v3det",
|
||||
"type": "transformers",
|
||||
},
|
||||
# MM-GroundingDINO Base models (Swin-B backbone) - 0.2B parameters
|
||||
"MM-GroundingDINO-Base (O365+GoldG+V3Det, 52.5 mAP)": {
|
||||
"hf_id": "openmmlab-community/mm_grounding_dino_base_o365v1_goldg_v3det",
|
||||
"type": "transformers",
|
||||
},
|
||||
"MM-GroundingDINO-Base (All datasets, 59.5 mAP)": {
|
||||
"hf_id": "openmmlab-community/mm_grounding_dino_base_all",
|
||||
"type": "transformers",
|
||||
},
|
||||
# MM-GroundingDINO Large models (Swin-L backbone) - 0.3B parameters
|
||||
"MM-GroundingDINO-Large (O365v2+OIv6+GoldG, 53.0 mAP)": {
|
||||
"hf_id": "openmmlab-community/mm_grounding_dino_large_o365v2_oiv6_goldg",
|
||||
"type": "transformers",
|
||||
},
|
||||
"MM-GroundingDINO-Large (All datasets, 60.3 mAP)": {
|
||||
"hf_id": "openmmlab-community/mm_grounding_dino_large_all",
|
||||
"type": "transformers",
|
||||
},
|
||||
}
|
||||
|
||||
YOLO_WORLD_MODELS = {
|
||||
"yolov8s-world": {
|
||||
"url": "https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8s-world.pt",
|
||||
},
|
||||
"yolov8m-world": {
|
||||
"url": "https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8m-world.pt",
|
||||
},
|
||||
"yolov8l-world": {
|
||||
"url": "https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8l-world.pt",
|
||||
},
|
||||
"yolov8x-world": {
|
||||
"url": "https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x-world.pt",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class GroundingDINOModelLoader:
|
||||
"""
|
||||
Loads GroundingDINO models for open-vocabulary object detection
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"model_name": (list(GROUNDING_DINO_MODELS.keys()), {"default": "GroundingDINO_SwinT_OGC (694MB)"}),
|
||||
"keep_in_memory": ("BOOLEAN", {"default": True}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("GROUNDING_DINO_MODEL",)
|
||||
RETURN_NAMES = ("model",)
|
||||
FUNCTION = "load_model"
|
||||
CATEGORY = "grounding"
|
||||
|
||||
def load_model(self, model_name: str, keep_in_memory: bool = True):
|
||||
"""Load GroundingDINO model with caching support"""
|
||||
|
||||
# Create cache key
|
||||
cache_key = f"grounding_dino_{model_name}"
|
||||
|
||||
# Check cache if keep_in_memory is True
|
||||
if keep_in_memory and cache_key in MODEL_CACHE:
|
||||
print(f"Loading {model_name} from cache")
|
||||
return (MODEL_CACHE[cache_key],)
|
||||
|
||||
# Clear from cache if keep_in_memory is False
|
||||
if not keep_in_memory and cache_key in MODEL_CACHE:
|
||||
print(f"Removing {model_name} from cache")
|
||||
del MODEL_CACHE[cache_key]
|
||||
|
||||
# Get model config
|
||||
model_config = GROUNDING_DINO_MODELS[model_name]
|
||||
model_type = model_config.get("type", "local")
|
||||
|
||||
# Load model based on type
|
||||
if model_type == "local":
|
||||
model = self._load_local(model_name, model_config)
|
||||
else: # transformers
|
||||
model = self._load_with_transformers(model_name, model_config)
|
||||
|
||||
# Cache if requested
|
||||
if keep_in_memory:
|
||||
MODEL_CACHE[cache_key] = model
|
||||
print(f"Cached {model_name} in memory")
|
||||
|
||||
return (model,)
|
||||
|
||||
def _load_local(self, model_name: str, model_config: dict):
|
||||
"""Load GroundingDINO using local implementation"""
|
||||
try:
|
||||
# Try using the local implementation from comfyui-sam2
|
||||
from custom_nodes.comfyui_sam2.node import load_groundingdino_model
|
||||
model = load_groundingdino_model(model_name)
|
||||
return model
|
||||
except ImportError:
|
||||
raise RuntimeError(
|
||||
f"Local GroundingDINO implementation not found. "
|
||||
f"Please install ComfyUI-segment-anything-2 to use {model_name}, "
|
||||
f"or use MM-GroundingDINO models instead."
|
||||
)
|
||||
|
||||
def _load_with_transformers(self, model_name: str, model_config: dict):
|
||||
"""Load MM-GroundingDINO using HuggingFace transformers"""
|
||||
try:
|
||||
from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection
|
||||
|
||||
device = mm.get_torch_device()
|
||||
|
||||
# Save models to local Models folder in plugin directory
|
||||
script_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
models_dir = os.path.join(script_dir, "Models")
|
||||
os.makedirs(models_dir, exist_ok=True)
|
||||
|
||||
# Get HuggingFace model ID
|
||||
hf_id = model_config.get("hf_id")
|
||||
|
||||
if not hf_id:
|
||||
raise ValueError(f"No HuggingFace model ID found for {model_name}")
|
||||
|
||||
print(f"📥 Loading {model_name} from HuggingFace ({hf_id})...")
|
||||
print("Downloading model files (progress bars will appear below):")
|
||||
|
||||
# Load processor and model (automatically shows tqdm progress bars)
|
||||
processor = AutoProcessor.from_pretrained(hf_id, cache_dir=models_dir)
|
||||
model = AutoModelForZeroShotObjectDetection.from_pretrained(hf_id, cache_dir=models_dir)
|
||||
|
||||
model.to(device)
|
||||
model.eval()
|
||||
|
||||
print(f"✅ Successfully loaded {model_name}")
|
||||
|
||||
return {"model": model, "processor": processor, "type": "transformers"}
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"Failed to load GroundingDINO model: {e}")
|
||||
|
||||
|
||||
class GroundingDINODetector:
|
||||
"""
|
||||
Performs object detection using GroundingDINO with text prompts
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"model": ("GROUNDING_DINO_MODEL",),
|
||||
"image": ("IMAGE",),
|
||||
"prompt": ("STRING", {
|
||||
"default": "person . car . dog .",
|
||||
"multiline": True,
|
||||
}),
|
||||
"box_threshold": ("FLOAT", {
|
||||
"default": 0.3,
|
||||
"min": 0.0,
|
||||
"max": 1.0,
|
||||
"step": 0.01,
|
||||
}),
|
||||
"text_threshold": ("FLOAT", {
|
||||
"default": 0.25,
|
||||
"min": 0.0,
|
||||
"max": 1.0,
|
||||
"step": 0.01,
|
||||
}),
|
||||
},
|
||||
"optional": {
|
||||
"single_box_mode": ("BOOLEAN", {
|
||||
"default": False,
|
||||
"tooltip": "Return only the highest-scoring detection (REC mode for referring expressions)"
|
||||
}),
|
||||
"output_format": (["list_only", "dict_with_data"], {
|
||||
"default": "list_only",
|
||||
"tooltip": "list_only: SAM2-compatible list of lists | dict_with_data: dict with boxes/labels/scores"
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("BBOX", "IMAGE", "STRING")
|
||||
RETURN_NAMES = ("bboxes", "annotated_image", "labels")
|
||||
FUNCTION = "detect"
|
||||
CATEGORY = "grounding"
|
||||
|
||||
def detect(self, model, image, prompt: str, box_threshold: float, text_threshold: float, single_box_mode: bool = False, output_format: str = "list_only"):
|
||||
"""Run detection on image with text prompt"""
|
||||
|
||||
# Process all images in batch
|
||||
batch_size = image.shape[0]
|
||||
all_boxes = []
|
||||
all_labels = []
|
||||
all_scores = []
|
||||
annotated_images = []
|
||||
|
||||
for i in range(batch_size):
|
||||
# Convert ComfyUI image format (B, H, W, C) in [0, 1] to PIL
|
||||
image_np = (image[i].cpu().numpy() * 255).astype(np.uint8)
|
||||
pil_image = Image.fromarray(image_np)
|
||||
|
||||
# Check model type and run detection
|
||||
if isinstance(model, dict) and model.get("type") == "transformers":
|
||||
bboxes, annotated_tensor, _ = self._detect_transformers(model, pil_image, prompt, box_threshold, text_threshold, single_box_mode)
|
||||
else:
|
||||
bboxes, annotated_tensor, _ = self._detect_local(model, pil_image, prompt, box_threshold, single_box_mode)
|
||||
|
||||
# Accumulate results
|
||||
all_boxes.append(bboxes["boxes"])
|
||||
all_labels.append(bboxes["labels"])
|
||||
all_scores.append(bboxes["scores"])
|
||||
annotated_images.append(annotated_tensor)
|
||||
|
||||
# Combine annotated images
|
||||
annotated_batch = torch.cat(annotated_images, dim=0)
|
||||
|
||||
# Format batched output based on output_format
|
||||
if output_format == "list_only":
|
||||
# SAM2-compatible format: list of lists (boxes only)
|
||||
batched_bboxes = all_boxes
|
||||
else: # dict_with_data
|
||||
# Dict format with all information
|
||||
batched_bboxes = {
|
||||
"boxes": all_boxes,
|
||||
"labels": all_labels,
|
||||
"scores": all_scores,
|
||||
}
|
||||
|
||||
# Format labels string (from first image)
|
||||
if len(all_labels) > 0 and len(all_labels[0]) > 0:
|
||||
labels_str = ", ".join([f"{label} ({score:.2f})" for label, score in zip(all_labels[0], all_scores[0])])
|
||||
else:
|
||||
labels_str = ""
|
||||
|
||||
return (batched_bboxes, annotated_batch, labels_str)
|
||||
|
||||
def _detect_transformers(self, model_dict, pil_image, prompt, box_threshold, text_threshold, single_box_mode=False):
|
||||
"""Detection using transformers implementation"""
|
||||
model = model_dict["model"]
|
||||
processor = model_dict["processor"]
|
||||
device = mm.get_torch_device()
|
||||
|
||||
# Prepare inputs
|
||||
inputs = processor(images=pil_image, text=prompt, return_tensors="pt").to(device)
|
||||
|
||||
# Run inference
|
||||
with torch.no_grad():
|
||||
outputs = model(**inputs)
|
||||
|
||||
# Post-process (use 'threshold' instead of 'box_threshold' for GroundingDINO)
|
||||
results = processor.post_process_grounded_object_detection(
|
||||
outputs,
|
||||
inputs.input_ids,
|
||||
threshold=box_threshold,
|
||||
text_threshold=text_threshold,
|
||||
target_sizes=[pil_image.size[::-1]]
|
||||
)[0]
|
||||
|
||||
# Extract boxes and labels
|
||||
boxes = results["boxes"].cpu().numpy()
|
||||
labels = results["labels"]
|
||||
scores = results["scores"].cpu().numpy()
|
||||
|
||||
# Override labels if prompt has no separators
|
||||
# Check for periods and commas (common separators in GroundingDINO prompts)
|
||||
if '.' not in prompt and ',' not in prompt:
|
||||
# Use full prompt as label for all detections
|
||||
clean_prompt = prompt.strip()
|
||||
labels = [clean_prompt] * len(labels)
|
||||
|
||||
# Single box mode: return only highest-scoring detection (REC mode)
|
||||
if single_box_mode and len(boxes) > 0:
|
||||
top_idx = scores.argmax()
|
||||
boxes = boxes[top_idx:top_idx+1]
|
||||
labels = [labels[top_idx]]
|
||||
scores = scores[top_idx:top_idx+1]
|
||||
|
||||
# Format output
|
||||
bboxes = {
|
||||
"boxes": boxes,
|
||||
"labels": labels,
|
||||
"scores": scores,
|
||||
}
|
||||
|
||||
# Create annotated image
|
||||
annotated = self._draw_boxes(pil_image, boxes, labels, scores)
|
||||
annotated_tensor = torch.from_numpy(np.array(annotated).astype(np.float32) / 255.0).unsqueeze(0)
|
||||
|
||||
# Format labels string
|
||||
labels_str = ", ".join([f"{label} ({score:.2f})" for label, score in zip(labels, scores)])
|
||||
|
||||
return (bboxes, annotated_tensor, labels_str)
|
||||
|
||||
def _detect_local(self, model, pil_image, prompt, box_threshold, single_box_mode=False):
|
||||
"""Detection using local GroundingDINO implementation"""
|
||||
# Use the implementation from comfyui-sam2
|
||||
from custom_nodes.comfyui_sam2.node import groundingdino_predict
|
||||
|
||||
boxes = groundingdino_predict(model, pil_image, prompt, box_threshold)
|
||||
|
||||
# Convert to numpy
|
||||
boxes_np = boxes.cpu().numpy()
|
||||
|
||||
# Create labels (simplified - just use prompt)
|
||||
labels = [prompt.strip().rstrip('.')] * len(boxes_np)
|
||||
scores = np.ones(len(boxes_np)) * box_threshold # Approximate
|
||||
|
||||
# Single box mode: return only first detection (local impl doesn't provide scores)
|
||||
if single_box_mode and len(boxes_np) > 0:
|
||||
boxes_np = boxes_np[0:1]
|
||||
labels = [labels[0]]
|
||||
scores = scores[0:1]
|
||||
|
||||
bboxes = {
|
||||
"boxes": boxes_np,
|
||||
"labels": labels,
|
||||
"scores": scores,
|
||||
}
|
||||
|
||||
# Create annotated image
|
||||
annotated = self._draw_boxes(pil_image, boxes_np, labels, scores)
|
||||
annotated_tensor = torch.from_numpy(np.array(annotated).astype(np.float32) / 255.0).unsqueeze(0)
|
||||
|
||||
labels_str = ", ".join(labels)
|
||||
|
||||
return (bboxes, annotated_tensor, labels_str)
|
||||
|
||||
def _draw_boxes(self, image, boxes, labels, scores):
|
||||
"""Draw bounding boxes on image"""
|
||||
from PIL import ImageDraw, ImageFont
|
||||
|
||||
draw = ImageDraw.Draw(image)
|
||||
|
||||
for box, label, score in zip(boxes, labels, scores):
|
||||
# box format: [x1, y1, x2, y2]
|
||||
x1, y1, x2, y2 = box
|
||||
|
||||
# Draw rectangle
|
||||
draw.rectangle([x1, y1, x2, y2], outline="red", width=3)
|
||||
|
||||
# Draw label
|
||||
text = f"{label}: {score:.2f}"
|
||||
draw.text((x1, y1 - 10), text, fill="red")
|
||||
|
||||
return image
|
||||
|
||||
|
||||
class YOLOWorldModelLoader:
|
||||
"""
|
||||
Loads YOLO-World models for open-vocabulary object detection
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"model_name": (list(YOLO_WORLD_MODELS.keys()), {"default": "yolov8s-world"}),
|
||||
"keep_in_memory": ("BOOLEAN", {"default": True}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("YOLO_WORLD_MODEL",)
|
||||
RETURN_NAMES = ("model",)
|
||||
FUNCTION = "load_model"
|
||||
CATEGORY = "grounding"
|
||||
|
||||
def load_model(self, model_name: str, keep_in_memory: bool = True):
|
||||
"""Load YOLO-World model with caching support"""
|
||||
|
||||
# Create cache key
|
||||
cache_key = f"yolo_world_{model_name}"
|
||||
|
||||
# Check cache if keep_in_memory is True
|
||||
if keep_in_memory and cache_key in MODEL_CACHE:
|
||||
print(f"Loading {model_name} from cache")
|
||||
return (MODEL_CACHE[cache_key],)
|
||||
|
||||
# Clear from cache if keep_in_memory is False
|
||||
if not keep_in_memory and cache_key in MODEL_CACHE:
|
||||
print(f"Removing {model_name} from cache")
|
||||
del MODEL_CACHE[cache_key]
|
||||
|
||||
try:
|
||||
from ultralytics import YOLOWorld
|
||||
|
||||
# Save models to local Models folder in plugin directory
|
||||
script_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
models_dir = os.path.join(script_dir, "Models")
|
||||
os.makedirs(models_dir, exist_ok=True)
|
||||
|
||||
model_filename = f"{model_name}.pt"
|
||||
model_path = os.path.join(models_dir, model_filename)
|
||||
|
||||
# Download if needed
|
||||
if not os.path.exists(model_path):
|
||||
print(f"📥 Downloading {model_name} to {models_dir}...")
|
||||
print("Progress bar:")
|
||||
from torch.hub import download_url_to_file
|
||||
download_url_to_file(
|
||||
YOLO_WORLD_MODELS[model_name]["url"],
|
||||
model_path,
|
||||
progress=True # Explicitly enable progress bar
|
||||
)
|
||||
print(f"✅ Downloaded {model_name} successfully")
|
||||
|
||||
# Load model
|
||||
print(f"Loading {model_name} from {model_path}")
|
||||
model = YOLOWorld(model_path)
|
||||
|
||||
# Cache if requested
|
||||
if keep_in_memory:
|
||||
MODEL_CACHE[cache_key] = model
|
||||
print(f"Cached {model_name} in memory")
|
||||
|
||||
return (model,)
|
||||
except ImportError:
|
||||
raise RuntimeError("ultralytics not installed. Run: pip install ultralytics")
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"Failed to load YOLO-World model: {e}")
|
||||
|
||||
|
||||
class YOLOWorldDetector:
|
||||
"""
|
||||
Performs object detection using YOLO-World with text prompts
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"model": ("YOLO_WORLD_MODEL",),
|
||||
"image": ("IMAGE",),
|
||||
"classes": ("STRING", {
|
||||
"default": "person, car, dog",
|
||||
"multiline": True,
|
||||
}),
|
||||
"confidence_threshold": ("FLOAT", {
|
||||
"default": 0.3,
|
||||
"min": 0.0,
|
||||
"max": 1.0,
|
||||
"step": 0.01,
|
||||
}),
|
||||
},
|
||||
"optional": {
|
||||
"single_box_mode": ("BOOLEAN", {
|
||||
"default": False,
|
||||
"tooltip": "Return only the highest-scoring detection (REC mode for referring expressions)"
|
||||
}),
|
||||
"output_format": (["list_only", "dict_with_data"], {
|
||||
"default": "list_only",
|
||||
"tooltip": "list_only: SAM2-compatible list of lists | dict_with_data: dict with boxes/labels/scores"
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("BBOX", "IMAGE", "STRING")
|
||||
RETURN_NAMES = ("bboxes", "annotated_image", "labels")
|
||||
FUNCTION = "detect"
|
||||
CATEGORY = "grounding"
|
||||
|
||||
def detect(self, model, image, classes: str, confidence_threshold: float, single_box_mode: bool = False, output_format: str = "list_only"):
|
||||
"""Run detection with YOLO-World"""
|
||||
|
||||
# Process all images in batch
|
||||
batch_size = image.shape[0]
|
||||
all_boxes = []
|
||||
all_labels = []
|
||||
all_scores = []
|
||||
annotated_images = []
|
||||
|
||||
# Parse classes - check if we should use full text as label
|
||||
has_separator = ',' in classes
|
||||
if has_separator:
|
||||
class_list = [c.strip() for c in classes.split(",")]
|
||||
else:
|
||||
# No comma separator - treat as single class/referring expression
|
||||
class_list = [classes.strip()]
|
||||
|
||||
# Set classes for the model
|
||||
model.set_classes(class_list)
|
||||
|
||||
for i in range(batch_size):
|
||||
# Convert ComfyUI image format
|
||||
image_np = (image[i].cpu().numpy() * 255).astype(np.uint8)
|
||||
|
||||
# Run inference
|
||||
results = model(image_np, conf=confidence_threshold)
|
||||
|
||||
# Extract detections
|
||||
result = results[0]
|
||||
boxes = result.boxes.xyxy.cpu().numpy() if result.boxes is not None else np.array([])
|
||||
scores = result.boxes.conf.cpu().numpy() if result.boxes is not None else np.array([])
|
||||
class_ids = result.boxes.cls.cpu().numpy().astype(int) if result.boxes is not None else np.array([])
|
||||
|
||||
# Map class IDs to names
|
||||
labels = [class_list[cid] for cid in class_ids]
|
||||
|
||||
# Override labels if no separator (referring expression mode)
|
||||
if not has_separator and len(labels) > 0:
|
||||
# Use full prompt as label for all detections
|
||||
labels = [classes.strip()] * len(labels)
|
||||
|
||||
# Single box mode: return only highest-scoring detection (REC mode)
|
||||
if single_box_mode and len(boxes) > 0:
|
||||
top_idx = scores.argmax()
|
||||
boxes = boxes[top_idx:top_idx+1]
|
||||
labels = [labels[top_idx]]
|
||||
scores = scores[top_idx:top_idx+1]
|
||||
|
||||
# Get annotated image
|
||||
annotated_np = result.plot()
|
||||
annotated_tensor = torch.from_numpy(annotated_np.astype(np.float32) / 255.0).unsqueeze(0)
|
||||
|
||||
# Accumulate results
|
||||
all_boxes.append(boxes)
|
||||
all_labels.append(labels)
|
||||
all_scores.append(scores)
|
||||
annotated_images.append(annotated_tensor)
|
||||
|
||||
# Combine annotated images
|
||||
annotated_batch = torch.cat(annotated_images, dim=0)
|
||||
|
||||
# Format batched output based on output_format
|
||||
if output_format == "list_only":
|
||||
# SAM2-compatible format: list of lists (boxes only)
|
||||
batched_bboxes = all_boxes
|
||||
else: # dict_with_data
|
||||
# Dict format with all information
|
||||
batched_bboxes = {
|
||||
"boxes": all_boxes,
|
||||
"labels": all_labels,
|
||||
"scores": all_scores,
|
||||
}
|
||||
|
||||
# Format labels string (from first image)
|
||||
if len(all_labels) > 0 and len(all_labels[0]) > 0:
|
||||
labels_str = ", ".join([f"{label} ({score:.2f})" for label, score in zip(all_labels[0], all_scores[0])])
|
||||
else:
|
||||
labels_str = ""
|
||||
|
||||
return (batched_bboxes, annotated_batch, labels_str)
|
||||
|
||||
|
||||
class BboxVisualizer:
|
||||
"""
|
||||
Visualizes bounding boxes on images
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"image": ("IMAGE",),
|
||||
"bboxes": ("BBOX",),
|
||||
},
|
||||
"optional": {
|
||||
"line_width": ("INT", {"default": 3, "min": 1, "max": 10}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("IMAGE",)
|
||||
RETURN_NAMES = ("annotated_image",)
|
||||
FUNCTION = "visualize"
|
||||
CATEGORY = "grounding"
|
||||
|
||||
def visualize(self, image, bboxes, line_width=3):
|
||||
"""Draw bounding boxes on image"""
|
||||
from PIL import ImageDraw
|
||||
|
||||
# Convert to PIL
|
||||
image_np = (image[0].cpu().numpy() * 255).astype(np.uint8)
|
||||
pil_image = Image.fromarray(image_np)
|
||||
|
||||
draw = ImageDraw.Draw(pil_image)
|
||||
|
||||
boxes = bboxes["boxes"]
|
||||
labels = bboxes.get("labels", [])
|
||||
scores = bboxes.get("scores", [])
|
||||
|
||||
for i, box in enumerate(boxes):
|
||||
x1, y1, x2, y2 = box
|
||||
draw.rectangle([x1, y1, x2, y2], outline="red", width=line_width)
|
||||
|
||||
if i < len(labels):
|
||||
label = labels[i]
|
||||
score = scores[i] if i < len(scores) else 0.0
|
||||
text = f"{label}: {score:.2f}"
|
||||
draw.text((x1, y1 - 10), text, fill="red")
|
||||
|
||||
# Convert back to tensor
|
||||
annotated_np = np.array(pil_image).astype(np.float32) / 255.0
|
||||
annotated_tensor = torch.from_numpy(annotated_np).unsqueeze(0)
|
||||
|
||||
return (annotated_tensor,)
|
||||
|
||||
|
||||
class BboxToMask:
|
||||
"""
|
||||
Converts bounding boxes to masks
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"image": ("IMAGE",),
|
||||
"bboxes": ("BBOX",),
|
||||
},
|
||||
"optional": {
|
||||
"combine_masks": ("BOOLEAN", {"default": False}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("MASK",)
|
||||
RETURN_NAMES = ("masks",)
|
||||
FUNCTION = "bbox_to_mask"
|
||||
CATEGORY = "grounding"
|
||||
|
||||
def bbox_to_mask(self, image, bboxes, combine_masks=False):
|
||||
"""Convert bounding boxes to binary masks"""
|
||||
|
||||
# Get image dimensions
|
||||
batch_size, height, width, channels = image.shape
|
||||
boxes = bboxes["boxes"]
|
||||
|
||||
if combine_masks:
|
||||
# Single combined mask
|
||||
mask = np.zeros((height, width), dtype=np.float32)
|
||||
for box in boxes:
|
||||
x1, y1, x2, y2 = box.astype(int)
|
||||
x1, y1 = max(0, x1), max(0, y1)
|
||||
x2, y2 = min(width, x2), min(height, y2)
|
||||
mask[y1:y2, x1:x2] = 1.0
|
||||
|
||||
mask_tensor = torch.from_numpy(mask).unsqueeze(0)
|
||||
else:
|
||||
# Individual masks for each box
|
||||
masks = []
|
||||
for box in boxes:
|
||||
mask = np.zeros((height, width), dtype=np.float32)
|
||||
x1, y1, x2, y2 = box.astype(int)
|
||||
x1, y1 = max(0, x1), max(0, y1)
|
||||
x2, y2 = min(width, x2), min(height, y2)
|
||||
mask[y1:y2, x1:x2] = 1.0
|
||||
masks.append(mask)
|
||||
|
||||
if len(masks) == 0:
|
||||
# No detections - return empty mask
|
||||
mask_tensor = torch.zeros((1, height, width))
|
||||
else:
|
||||
mask_tensor = torch.from_numpy(np.stack(masks))
|
||||
|
||||
return (mask_tensor,)
|
||||
|
||||
|
||||
class UnifiedDetector:
|
||||
"""
|
||||
Universal detector that works with both GroundingDINO and YOLO-World models
|
||||
Automatically detects model type and uses appropriate inference method
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"model": ("GROUNDING_DINO_MODEL,YOLO_WORLD_MODEL",),
|
||||
"image": ("IMAGE",),
|
||||
"prompt": ("STRING", {
|
||||
"default": "person . car . dog .",
|
||||
"multiline": True,
|
||||
"tooltip": "For GroundingDINO: period-separated (e.g., 'cat . dog .'). For YOLO-World: comma-separated (e.g., 'cat, dog')"
|
||||
}),
|
||||
"threshold": ("FLOAT", {
|
||||
"default": 0.3,
|
||||
"min": 0.0,
|
||||
"max": 1.0,
|
||||
"step": 0.01,
|
||||
}),
|
||||
},
|
||||
"optional": {
|
||||
"text_threshold": ("FLOAT", {
|
||||
"default": 0.25,
|
||||
"min": 0.0,
|
||||
"max": 1.0,
|
||||
"step": 0.01,
|
||||
"tooltip": "Only used for GroundingDINO models"
|
||||
}),
|
||||
"single_box_mode": ("BOOLEAN", {
|
||||
"default": False,
|
||||
"tooltip": "Return only the highest-scoring detection (REC mode for referring expressions)"
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("BBOX", "IMAGE", "STRING")
|
||||
RETURN_NAMES = ("bboxes", "annotated_image", "labels")
|
||||
FUNCTION = "detect"
|
||||
CATEGORY = "grounding"
|
||||
|
||||
def detect(self, model, image, prompt: str, threshold: float, text_threshold: float = 0.25, single_box_mode: bool = False):
|
||||
"""Universal detection that auto-detects model type"""
|
||||
|
||||
# Detect model type
|
||||
model_type = self._detect_model_type(model)
|
||||
|
||||
if model_type == "yolo_world":
|
||||
return self._detect_yolo_world(model, image, prompt, threshold, single_box_mode)
|
||||
else: # grounding_dino or transformers
|
||||
return self._detect_grounding_dino(model, image, prompt, threshold, text_threshold, single_box_mode)
|
||||
|
||||
def _detect_model_type(self, model):
|
||||
"""Detect whether this is a YOLO-World or GroundingDINO model"""
|
||||
# Check for YOLO-World model
|
||||
try:
|
||||
from ultralytics import YOLOWorld
|
||||
if isinstance(model, YOLOWorld):
|
||||
return "yolo_world"
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
# Check for transformers-based model
|
||||
if isinstance(model, dict) and model.get("type") == "transformers":
|
||||
return "grounding_dino_transformers"
|
||||
|
||||
# Default to GroundingDINO (local)
|
||||
return "grounding_dino_local"
|
||||
|
||||
def _detect_grounding_dino(self, model, image, prompt, box_threshold, text_threshold, single_box_mode=False):
|
||||
"""Detection using GroundingDINO"""
|
||||
# Convert ComfyUI image format
|
||||
image_np = (image[0].cpu().numpy() * 255).astype(np.uint8)
|
||||
pil_image = Image.fromarray(image_np)
|
||||
|
||||
# Check model type
|
||||
if isinstance(model, dict) and model.get("type") == "transformers":
|
||||
return self._detect_transformers(model, pil_image, prompt, box_threshold, text_threshold, single_box_mode)
|
||||
else:
|
||||
return self._detect_local(model, pil_image, prompt, box_threshold, single_box_mode)
|
||||
|
||||
def _detect_transformers(self, model_dict, pil_image, prompt, box_threshold, text_threshold, single_box_mode=False):
|
||||
"""Detection using transformers implementation"""
|
||||
model = model_dict["model"]
|
||||
processor = model_dict["processor"]
|
||||
device = mm.get_torch_device()
|
||||
|
||||
# Prepare inputs
|
||||
inputs = processor(images=pil_image, text=prompt, return_tensors="pt").to(device)
|
||||
|
||||
# Run inference
|
||||
with torch.no_grad():
|
||||
outputs = model(**inputs)
|
||||
|
||||
# Post-process (use 'threshold' instead of 'box_threshold' for GroundingDINO)
|
||||
results = processor.post_process_grounded_object_detection(
|
||||
outputs,
|
||||
inputs.input_ids,
|
||||
threshold=box_threshold,
|
||||
text_threshold=text_threshold,
|
||||
target_sizes=[pil_image.size[::-1]]
|
||||
)[0]
|
||||
|
||||
# Extract boxes and labels
|
||||
boxes = results["boxes"].cpu().numpy()
|
||||
labels = results["labels"]
|
||||
scores = results["scores"].cpu().numpy()
|
||||
|
||||
# Single box mode: return only highest-scoring detection (REC mode)
|
||||
if single_box_mode and len(boxes) > 0:
|
||||
top_idx = scores.argmax()
|
||||
boxes = boxes[top_idx:top_idx+1]
|
||||
labels = [labels[top_idx]]
|
||||
scores = scores[top_idx:top_idx+1]
|
||||
|
||||
# Format output
|
||||
bboxes = {
|
||||
"boxes": boxes,
|
||||
"labels": labels,
|
||||
"scores": scores,
|
||||
}
|
||||
|
||||
# Create annotated image
|
||||
annotated = self._draw_boxes(pil_image, boxes, labels, scores)
|
||||
annotated_tensor = torch.from_numpy(np.array(annotated).astype(np.float32) / 255.0).unsqueeze(0)
|
||||
|
||||
# Format labels string
|
||||
labels_str = ", ".join([f"{label} ({score:.2f})" for label, score in zip(labels, scores)])
|
||||
|
||||
return (bboxes, annotated_tensor, labels_str)
|
||||
|
||||
def _detect_local(self, model, pil_image, prompt, box_threshold, single_box_mode=False):
|
||||
"""Detection using local GroundingDINO implementation"""
|
||||
from custom_nodes.comfyui_sam2.node import groundingdino_predict
|
||||
|
||||
boxes = groundingdino_predict(model, pil_image, prompt, box_threshold)
|
||||
|
||||
# Convert to numpy
|
||||
boxes_np = boxes.cpu().numpy()
|
||||
|
||||
# Create labels
|
||||
labels = [prompt.strip().rstrip('.')] * len(boxes_np)
|
||||
scores = np.ones(len(boxes_np)) * box_threshold
|
||||
|
||||
# Single box mode: return only first detection (local impl doesn't provide scores)
|
||||
if single_box_mode and len(boxes_np) > 0:
|
||||
boxes_np = boxes_np[0:1]
|
||||
labels = [labels[0]]
|
||||
scores = scores[0:1]
|
||||
|
||||
bboxes = {
|
||||
"boxes": boxes_np,
|
||||
"labels": labels,
|
||||
"scores": scores,
|
||||
}
|
||||
|
||||
# Create annotated image
|
||||
annotated = self._draw_boxes(pil_image, boxes_np, labels, scores)
|
||||
annotated_tensor = torch.from_numpy(np.array(annotated).astype(np.float32) / 255.0).unsqueeze(0)
|
||||
|
||||
labels_str = ", ".join(labels)
|
||||
|
||||
return (bboxes, annotated_tensor, labels_str)
|
||||
|
||||
def _detect_yolo_world(self, model, image, classes, confidence_threshold, single_box_mode=False):
|
||||
"""Detection using YOLO-World"""
|
||||
# Convert ComfyUI image format
|
||||
image_np = (image[0].cpu().numpy() * 255).astype(np.uint8)
|
||||
|
||||
# Parse classes (support both comma and period separation)
|
||||
if ',' in classes:
|
||||
class_list = [c.strip() for c in classes.split(",")]
|
||||
else:
|
||||
class_list = [c.strip() for c in classes.split(".") if c.strip()]
|
||||
|
||||
# Set classes for the model
|
||||
model.set_classes(class_list)
|
||||
|
||||
# Run inference
|
||||
results = model(image_np, conf=confidence_threshold)
|
||||
|
||||
# Extract detections
|
||||
result = results[0]
|
||||
boxes = result.boxes.xyxy.cpu().numpy() if result.boxes is not None else np.array([])
|
||||
scores = result.boxes.conf.cpu().numpy() if result.boxes is not None else np.array([])
|
||||
class_ids = result.boxes.cls.cpu().numpy().astype(int) if result.boxes is not None else np.array([])
|
||||
|
||||
# Map class IDs to names
|
||||
labels = [class_list[cid] for cid in class_ids]
|
||||
|
||||
# Single box mode: return only highest-scoring detection (REC mode)
|
||||
if single_box_mode and len(boxes) > 0:
|
||||
top_idx = scores.argmax()
|
||||
boxes = boxes[top_idx:top_idx+1]
|
||||
labels = [labels[top_idx]]
|
||||
scores = scores[top_idx:top_idx+1]
|
||||
|
||||
# Format output
|
||||
bboxes = {
|
||||
"boxes": boxes,
|
||||
"labels": labels,
|
||||
"scores": scores,
|
||||
}
|
||||
|
||||
# Get annotated image
|
||||
annotated_np = result.plot()
|
||||
annotated_tensor = torch.from_numpy(annotated_np.astype(np.float32) / 255.0).unsqueeze(0)
|
||||
|
||||
# Format labels string
|
||||
labels_str = ", ".join([f"{label} ({score:.2f})" for label, score in zip(labels, scores)])
|
||||
|
||||
return (bboxes, annotated_tensor, labels_str)
|
||||
|
||||
def _draw_boxes(self, image, boxes, labels, scores):
|
||||
"""Draw bounding boxes on image"""
|
||||
from PIL import ImageDraw
|
||||
|
||||
# Make a copy to avoid modifying original
|
||||
image = image.copy()
|
||||
draw = ImageDraw.Draw(image)
|
||||
|
||||
for box, label, score in zip(boxes, labels, scores):
|
||||
# box format: [x1, y1, x2, y2]
|
||||
x1, y1, x2, y2 = box
|
||||
|
||||
# Draw rectangle
|
||||
draw.rectangle([x1, y1, x2, y2], outline="red", width=3)
|
||||
|
||||
# Draw label
|
||||
text = f"{label}: {score:.2f}"
|
||||
draw.text((x1, y1 - 10), text, fill="red")
|
||||
|
||||
return image
|
||||
@@ -0,0 +1,15 @@
|
||||
# Core dependencies for ComfyUI-Grounding
|
||||
|
||||
# For GroundingDINO (HuggingFace transformers implementation)
|
||||
transformers>=4.30.0
|
||||
timm>=0.9.0
|
||||
|
||||
# For YOLO-World
|
||||
ultralytics>=8.2.0
|
||||
|
||||
# Image processing
|
||||
opencv-python>=4.8.0
|
||||
pillow>=10.0.0
|
||||
|
||||
# Optional: supervision for advanced visualization
|
||||
# supervision>=0.16.0
|
||||
Reference in New Issue
Block a user