diff --git a/Models/.locks/models--openmmlab-community--mm_grounding_dino_tiny_o365v1_goldg_grit/50c31678d60a62869549da406ee8731f5f0a1a80.lock b/Models/.locks/models--openmmlab-community--mm_grounding_dino_tiny_o365v1_goldg_grit/50c31678d60a62869549da406ee8731f5f0a1a80.lock deleted file mode 100644 index e69de29..0000000 diff --git a/Models/.locks/models--openmmlab-community--mm_grounding_dino_tiny_o365v1_goldg_grit/63919bced01f1298630f49f8ef2ae89e132052cc5dd3d5bbf639c296a216a5ac.lock b/Models/.locks/models--openmmlab-community--mm_grounding_dino_tiny_o365v1_goldg_grit/63919bced01f1298630f49f8ef2ae89e132052cc5dd3d5bbf639c296a216a5ac.lock deleted file mode 100644 index e69de29..0000000 diff --git a/Models/.locks/models--openmmlab-community--mm_grounding_dino_tiny_o365v1_goldg_grit/82269b347fa24c2124f53e018a9f620f3d6d275b.lock b/Models/.locks/models--openmmlab-community--mm_grounding_dino_tiny_o365v1_goldg_grit/82269b347fa24c2124f53e018a9f620f3d6d275b.lock deleted file mode 100644 index e69de29..0000000 diff --git a/Models/.locks/models--openmmlab-community--mm_grounding_dino_tiny_o365v1_goldg_grit/a8b3208c2884c4efb86e49300fdd3dc877220cdf.lock b/Models/.locks/models--openmmlab-community--mm_grounding_dino_tiny_o365v1_goldg_grit/a8b3208c2884c4efb86e49300fdd3dc877220cdf.lock deleted file mode 100644 index e69de29..0000000 diff --git a/Models/.locks/models--openmmlab-community--mm_grounding_dino_tiny_o365v1_goldg_grit/e87fd0839a1d28c473aa6f328a640bcc8b779859.lock b/Models/.locks/models--openmmlab-community--mm_grounding_dino_tiny_o365v1_goldg_grit/e87fd0839a1d28c473aa6f328a640bcc8b779859.lock deleted file mode 100644 index e69de29..0000000 diff --git a/Models/.locks/models--openmmlab-community--mm_grounding_dino_tiny_o365v1_goldg_grit/fb140275c155a9c7c5a3b3e0e77a9e839594a938.lock b/Models/.locks/models--openmmlab-community--mm_grounding_dino_tiny_o365v1_goldg_grit/fb140275c155a9c7c5a3b3e0e77a9e839594a938.lock deleted file mode 100644 index e69de29..0000000 diff --git a/README.md b/README.md index fb214a1..4a55611 100644 --- a/README.md +++ b/README.md @@ -1,354 +1,85 @@ # ComfyUI-Grounding -ComfyUI custom nodes for open-vocabulary object detection using state-of-the-art grounding models. +**Grounding for dummies, simplest workflow** -## Features +![Simple Workflow](docs/simple.png) -- **GroundingDINO**: Open-vocabulary object detection with natural language prompts -- **YOLO-World**: Real-time open-vocabulary detection -- **Batch Processing**: Process multiple images at once for improved efficiency -- **Referring Expression Comprehension (REC)**: Use full text as label when no separators present -- **Flexible Architecture**: Support for both local and HuggingFace implementations -- **Utility Nodes**: Bounding box visualization and mask conversion +🎯 **3 Nodes Total** - Loader → Detector → Visualizer +🚀 **5 Model Families** - GroundingDINO, MM-GroundingDINO, OWLv2, Florence-2, YOLO-World +💾 **Smart Caching** - Instant reload +📦 **Batch Processing** - Multiple images at once +🎭 **Built-in Masks** - No separate BboxToMask node needed -## Supported Models +## Quick Start -### GroundingDINO (Original) -- GroundingDINO-T (SwinT backbone, 694MB) -- GroundingDINO-B (SwinB backbone, 938MB) - -### MM-GroundingDINO (Multimodal) -**Tiny Models (Swin-T backbone):** -- MM-GroundingDINO-Tiny (O365+GoldG, 50.4 mAP) -- MM-GroundingDINO-Tiny (O365+GoldG+GRIT, 50.5 mAP) -- MM-GroundingDINO-Tiny (O365+GoldG+V3Det, 50.6 mAP) -- MM-GroundingDINO-Tiny (O365+GoldG+GRIT+V3Det, 50.4 mAP) - -**Base Models (Swin-B backbone):** -- MM-GroundingDINO-Base (O365+GoldG+V3Det, 52.5 mAP) -- MM-GroundingDINO-Base (All datasets, 59.5 mAP) ⭐ Recommended - -**Large Models (Swin-L backbone):** -- MM-GroundingDINO-Large (O365v2+OIv6+GoldG, 53.0 mAP) -- MM-GroundingDINO-Large (All datasets, 60.3 mAP) ⭐ Best accuracy - -### YOLO-World -- YOLOv8s-World (small) -- YOLOv8m-World (medium) -- YOLOv8l-World (large) -- YOLOv8x-World (extra large) - -## Installation - -1. Navigate to your ComfyUI custom nodes directory: ```bash cd ComfyUI/custom_nodes/ -``` - -2. Clone or copy this repository: -```bash git clone ComfyUI-Grounding -``` - -3. Install dependencies: -```bash cd ComfyUI-Grounding pip install -r requirements.txt ``` -## Nodes +## The Nodes -### 1. GroundingDINOModelLoader -Loads a GroundingDINO or MM-GroundingDINO model for object detection. +### 1. Grounding Model Loader +Load any of 10+ models from a single dropdown. -**Inputs:** -- `model_name`: Select from available GroundingDINO/MM-GroundingDINO models -- `keep_in_memory`: Cache model in memory for faster reuse (default: True) + 1. GroundingDINO: SwinT OGC (694MB) - IDEA-Research/grounding-dino-tiny + 2. GroundingDINO: SwinB (938MB) - IDEA-Research/grounding-dino-base + 3. MM-GroundingDINO: Tiny O365+GoldG (50.4 mAP) - openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg + 4. MM-GroundingDINO: Tiny O365+GoldG+GRIT (50.5 mAP) - openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_grit + 5. MM-GroundingDINO: Tiny O365+GoldG+V3Det (50.6 mAP) - openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_v3det + 6. MM-GroundingDINO: Base O365+GoldG+V3Det (52.5 mAP) - openmmlab-community/mm_grounding_dino_base_o365v1_goldg_v3det + 7. MM-GroundingDINO: Base All Datasets (59.5 mAP) - openmmlab-community/mm_grounding_dino_base_all + 8. MM-GroundingDINO: Large O365v2+OIv6+GoldG (53.0 mAP) - openmmlab-community/mm_grounding_dino_large_o365v2_oiv6_goldg + 9. MM-GroundingDINO: Large All Datasets (60.3 mAP) - openmmlab-community/mm_grounding_dino_large_all + 10. OWLv2: Base Patch16 - google/owlv2-base-patch16 + 11. OWLv2: Large Patch14 - google/owlv2-large-patch14 + 12. OWLv2: Base Patch16 Ensemble - google/owlv2-base-patch16-ensemble + 13. OWLv2: Large Patch14 Ensemble - google/owlv2-large-patch14-ensemble ⭐ + 14. Florence-2: Base (0.23B params) - microsoft/Florence-2-base ⚡ Best Balance + 15. Florence-2: Large (0.77B params) - microsoft/Florence-2-large ⭐ + 16. YOLO-World: v8s (Small) - yolov8s-worldv2.pt ⚡ Fastest + 17. YOLO-World: v8m (Medium) - yolov8m-worldv2.pt + 18. YOLO-World: v8l (Large) - yolov8l-worldv2.pt + 19. YOLO-World: v8x (Extra Large) - yolov8x-worldv2.pt -**Outputs:** -- `model`: Loaded GroundingDINO model - -**Features:** -- ✅ Model caching - loads instantly on second use -- ✅ Local model storage in `ComfyUI-Grounding/Models/` -- ✅ Supports both original and MM-GroundingDINO variants - -### 2. GroundingDINODetector -Detects objects in images using text prompts. - -**Inputs:** -- `model`: GroundingDINO model (from loader) -- `image`: Input image (supports batches) -- `prompt`: Text description of objects to detect (e.g., "person . car . dog .") -- `box_threshold`: Confidence threshold for bounding boxes (0.0-1.0) -- `text_threshold`: Confidence threshold for text matching (0.0-1.0) -- `single_box_mode`: (Optional) Return only highest-scoring detection for REC -- `output_format`: (Optional) "list_only" (default, SAM2-compatible) or "dict_with_data" - -**Outputs:** -- `bboxes`: Detected bounding boxes (format depends on output_format setting) - - `list_only`: List of numpy arrays `[[boxes_img0], [boxes_img1], ...]` - SAM2-compatible - - `dict_with_data`: Dict `{"boxes": [...], "labels": [...], "scores": [...]}` -- `annotated_image`: Image with drawn bounding boxes (batched) -- `labels`: String with detected object labels - -**Features:** -- ✅ Batch processing - process multiple images at once -- ✅ Automatic REC mode - when prompt has no separators (. or ,), uses full text as label -- ✅ Single box mode - returns only highest-scoring detection for referring expressions -- ✅ Flexible output format - choose between SAM2-compatible list or full dict with metadata - -### 3. YOLOWorldModelLoader -Loads a YOLO-World model for real-time detection. - -**Inputs:** -- `model_name`: Select from available YOLO-World models -- `keep_in_memory`: Cache model in memory for faster reuse (default: True) - -**Outputs:** -- `model`: Loaded YOLO-World model - -**Features:** -- ✅ Model caching - loads instantly on second use -- ✅ Local model storage in `ComfyUI-Grounding/Models/` -- ✅ Auto-downloads models on first use - -### 4. YOLOWorldDetector -Detects objects using YOLO-World with custom classes. - -**Inputs:** -- `model`: YOLO-World model (from loader) -- `image`: Input image (supports batches) -- `classes`: Comma-separated list of classes (e.g., "person, car, dog") -- `confidence_threshold`: Minimum confidence for detections (0.0-1.0) -- `single_box_mode`: (Optional) Return only highest-scoring detection for REC -- `output_format`: (Optional) "list_only" (default, SAM2-compatible) or "dict_with_data" - -**Outputs:** -- `bboxes`: Detected bounding boxes (format depends on output_format setting) - - `list_only`: List of numpy arrays `[[boxes_img0], [boxes_img1], ...]` - SAM2-compatible - - `dict_with_data`: Dict `{"boxes": [...], "labels": [...], "scores": [...]}` -- `annotated_image`: Image with drawn bounding boxes (batched) -- `labels`: String with detected object labels - -**Features:** -- ✅ Batch processing - process multiple images at once -- ✅ Automatic REC mode - when classes has no comma, uses full text as label -- ✅ Single box mode - returns only highest-scoring detection for referring expressions -- ✅ Flexible output format - choose between SAM2-compatible list or full dict with metadata - -### 5. BboxVisualizer -Draws bounding boxes on images. - -**Inputs:** -- `image`: Input image -- `bboxes`: Bounding box data -- `line_width`: Thickness of bounding box lines (optional) - -**Outputs:** -- `annotated_image`: Image with drawn bounding boxes - -### 5. UnifiedDetector ⭐ NEW -Universal detector that works with BOTH GroundingDINO and YOLO-World models. - -**Inputs:** -- `model`: Any model (GroundingDINO or YOLO-World) -- `image`: Input image -- `prompt`: Text prompt (supports both formats) - - For GroundingDINO: "cat . dog . car ." - - For YOLO-World: "cat, dog, car" -- `threshold`: Confidence threshold (0.0-1.0) -- `text_threshold`: Text matching threshold (optional, GroundingDINO only) - -**Outputs:** -- `bboxes`: Detected bounding boxes with labels and scores -- `annotated_image`: Image with drawn bounding boxes -- `labels`: String with detected object labels - -**Features:** -- ✅ Auto-detects model type -- ✅ Single node for all detection models -- ✅ Simplified workflow - -### 6. BboxToMask -Converts bounding boxes to binary masks. - -**Inputs:** -- `image`: Input image (for dimensions) -- `bboxes`: Bounding box data -- `combine_masks`: Whether to combine all boxes into one mask (optional) - -**Outputs:** -- `masks`: Binary mask(s) for detected regions +### 2. Grounding Detector +Universal detector for all models. ## Key Features -### 🚀 Model Caching -Models are cached in memory by default for instant reuse: -- First load: Downloads and initializes model -- Second load: Instant loading from cache -- Toggle `keep_in_memory=False` to clear cache +- Overrides standard text label splitting. It splits only at ".", otherwise label is what you write +- Enable `single_box_mode` for single detection +- First load: Downloads model +- Second load: Instant from cache +- Models stored in `ComfyUI-Grounding/models/` +- Process multiple images in one pass +- All nodes support batches -### 📁 Local Model Storage -All models are downloaded to the plugin's local Models directory: -- Location: `ComfyUI/custom_nodes/ComfyUI-Grounding/Models/` -- Auto-created on first use with a `models_go_here.txt` placeholder -- GroundingDINO/MM-GroundingDINO: Stored in HuggingFace transformers cache format -- YOLO-World: Stored as `.pt` files (yolov8s-world.pt, etc.) +## Florence-2 Attention Modes -### 🎯 UnifiedDetector -One detector for all models - automatically detects whether you're using GroundingDINO or YOLO-World and routes to the appropriate inference method. +- `eager` - Most compatible (default) +- `sdpa` - Faster on PyTorch 2.0+ +- `flash_attention_2` - Fastest on A100/H100 -### 📦 Batch Processing -All detector nodes support batch processing: -- Process multiple images in a single forward pass -- Input images as a batch tensor (B, H, W, C) -- Returns batched bounding boxes with lists of detections for each image -- Annotated images are returned as a single batch tensor +### 2. Bounding Box Visualizer -### 🎯 Referring Expression Comprehension (REC) -Automatic REC mode for natural language queries: -- **GroundingDINO**: If prompt has no periods (`.`) or commas (`,`), uses full text as label - - Example: `"main object in the center of the picture"` → all detections labeled with full text - - With separators: `"person . car . dog ."` → uses standard noun phrase extraction -- **YOLO-World**: If classes has no comma (`,`), uses full text as label - - Example: `"red car in the parking lot"` → all detections labeled with full text - - With separators: `"person, car, dog"` → uses standard class mapping -- **Single Box Mode**: Enable `single_box_mode` to return only the highest-scoring detection (perfect for REC tasks) - -### 📋 Output Formats -Detector nodes support two output formats for bounding boxes: - -**1. `list_only` (Default - SAM2 Compatible)** -- Returns: List of numpy arrays, one per image in batch -- Format: `[[bbox1, bbox2], [bbox3], ...]` where each bbox is `[x1, y1, x2, y2]` -- Use when: Connecting to SAM2 segmentation nodes -- Example: - ```python - # For 2 images with 2 and 1 detections respectively - [ - array([[10, 20, 100, 200], [50, 60, 150, 250]]), # Image 0: 2 boxes - array([[30, 40, 120, 180]]) # Image 1: 1 box - ] - ``` - -**2. `dict_with_data` (Full Metadata)** -- Returns: Dictionary with boxes, labels, and scores -- Format: `{"boxes": [...], "labels": [...], "scores": [...]}` -- Use when: Need label and confidence information for downstream processing -- NOT compatible with SAM2 (dict iteration gives keys, not values) -- Example: - ```python - { - "boxes": [array([...]), array([...])], # Boxes per image - "labels": [["cat", "dog"], ["bird"]], # Labels per image - "scores": [array([0.9, 0.8]), array([0.7])] # Scores per image - } - ``` - -**Recommendation:** Use `list_only` (default) unless you specifically need label/score metadata for your workflow. - -## Usage Examples - -### Example 1: Basic Object Detection with GroundingDINO - -1. Add `GroundingDINOModelLoader` node -2. Select model (e.g., "GroundingDINO_SwinT_OGC (694MB)") -3. Add `GroundingDINODetector` node -4. Connect model output to detector -5. Load an image and connect to detector -6. Set prompt: "person . car . bicycle ." -7. Adjust thresholds as needed -8. View annotated image output - -### Example 2: YOLO-World Detection - -1. Add `YOLOWorldModelLoader` node -2. Select model (e.g., "yolov8s-world") -3. Add `YOLOWorldDetector` node -4. Connect model and image -5. Set classes: "cat, dog, bird, horse" -6. Adjust confidence threshold -7. View results - -### Example 3: Using UnifiedDetector (Recommended) - -1. Add any model loader (GroundingDINO or YOLO-World) -2. Load your model -3. Add `UnifiedDetector` node -4. Connect model and image -5. Set prompt (works with both formats!) -6. The node automatically detects model type and uses the right inference - -### Example 4: MM-GroundingDINO for High Accuracy - -1. Add `GroundingDINOModelLoader` node -2. Select "MM-GroundingDINO-Base (All datasets, 59.5 mAP)" -3. Enable `keep_in_memory=True` (default) -4. Add `GroundingDINODetector` or `UnifiedDetector` -5. Set prompt: "specific object . another object ." -6. Enjoy state-of-the-art detection accuracy! - -### Example 5: Convert Detections to Masks - -1. Use any detector node (GroundingDINO or YOLO-World) -2. Add `BboxToMask` node -3. Connect image and bboxes outputs -4. Choose whether to combine masks -5. Use masks for segmentation, inpainting, etc. - -## Model Downloads - -Models will be automatically downloaded on first use: -- **All models**: Stored in `ComfyUI/custom_nodes/ComfyUI-Grounding/Models/` -- **GroundingDINO/MM-GroundingDINO**: HuggingFace transformers cache format -- **YOLO-World**: Direct `.pt` files (e.g., yolov8s-world.pt) +Re-draw bboxes on images with custom line width. Optional since detector already returns annotated images. ## Tips -- **GroundingDINO prompts**: End each object with a period (e.g., "cat . dog . car .") -- **YOLO-World classes**: Use comma-separated values (e.g., "cat, dog, car") -- **Threshold tuning**: Start with 0.3 and adjust based on results - - Lower threshold = more detections (including false positives) - - Higher threshold = fewer, more confident detections -- **Performance**: YOLO-World is faster but GroundingDINO may be more accurate - -## Integration with SAM2 - -These nodes work seamlessly with SAM2 for instance segmentation: - -1. Use GroundingDINO or YOLO-World to detect objects -2. Convert bboxes to masks using `BboxToMask` -3. Use masks with SAM2 for precise segmentation - -## Troubleshooting - -### ImportError: transformers not found -```bash -pip install transformers timm -``` - -### ImportError: ultralytics not found -```bash -pip install ultralytics -``` - -### Models not downloading -Check your internet connection and ensure you have write permissions to the ComfyUI models directory. - -### CUDA out of memory -Try using a smaller model variant (e.g., GroundingDINO-T instead of -B, or yolov8s instead of yolov8x). - -## License - -MIT License - See LICENSE file for details +**Prompt Format:** +- Use periods for multiple labels → `"cat . dog ."` +- Use commas to keep a single label → `"cat, dog"` ## Credits -- GroundingDINO: [IDEA-Research](https://github.com/IDEA-Research/GroundingDINO) -- YOLO-World: [Ultralytics](https://github.com/ultralytics/ultralytics) -- Transformers: [HuggingFace](https://huggingface.co/transformers) +- [GroundingDINO](https://github.com/IDEA-Research/GroundingDINO) - IDEA-Research +- [OWLv2](https://huggingface.co/google/owlv2-base-patch16) - Google Research +- [Florence-2](https://huggingface.co/microsoft/Florence-2-base) - Microsoft Research +- [YOLO-World](https://github.com/ultralytics/ultralytics) - Ultralytics -## Support +## License -For issues and feature requests, please visit the GitHub repository. +MIT License diff --git a/__init__.py b/__init__.py index d0dc830..aff8ac1 100644 --- a/__init__.py +++ b/__init__.py @@ -1,36 +1,24 @@ """ -ComfyUI-Grounding: Object detection nodes for ComfyUI -Supports GroundingDINO, MM-GroundingDINO, and YOLO-World models +ComfyUI-Grounding: Simplified object detection nodes for ComfyUI +Supports GroundingDINO, MM-GroundingDINO, OWLv2, Florence-2, and YOLO-World """ from .nodes import ( - GroundingDINOModelLoader, - GroundingDINODetector, - YOLOWorldModelLoader, - YOLOWorldDetector, - UnifiedDetector, + GroundingModelLoader, + GroundingDetector, BboxVisualizer, - BboxToMask ) NODE_CLASS_MAPPINGS = { - "GroundingDINOModelLoader": GroundingDINOModelLoader, - "GroundingDINODetector": GroundingDINODetector, - "YOLOWorldModelLoader": YOLOWorldModelLoader, - "YOLOWorldDetector": YOLOWorldDetector, - "UnifiedDetector": UnifiedDetector, + "GroundingModelLoader": GroundingModelLoader, + "GroundingDetector": GroundingDetector, "BboxVisualizer": BboxVisualizer, - "BboxToMask": BboxToMask, } NODE_DISPLAY_NAME_MAPPINGS = { - "GroundingDINOModelLoader": "Grounding DINO Model Loader", - "GroundingDINODetector": "Grounding DINO Detector", - "YOLOWorldModelLoader": "YOLO-World Model Loader", - "YOLOWorldDetector": "YOLO-World Detector", - "UnifiedDetector": "Unified Detector (Auto)", + "GroundingModelLoader": "Grounding Model Loader", + "GroundingDetector": "Grounding Detector", "BboxVisualizer": "Bounding Box Visualizer", - "BboxToMask": "Bounding Box to Mask", } __all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS'] diff --git a/docs/simple.png b/docs/simple.png new file mode 100644 index 0000000..e2b3857 Binary files /dev/null and b/docs/simple.png differ diff --git a/models/.gitignore b/models/.gitignore new file mode 100644 index 0000000..4611e4d --- /dev/null +++ b/models/.gitignore @@ -0,0 +1,8 @@ +# Ignore all files in this directory (model downloads) +* + +# But keep this .gitignore file +!.gitignore + +# And keep the placeholder file +!models_go_here.txt diff --git a/Models/models_go_here.txt b/models/models_go_here.txt similarity index 100% rename from Models/models_go_here.txt rename to models/models_go_here.txt diff --git a/nodes.py b/nodes.py index e746bb3..68c54d0 100644 --- a/nodes.py +++ b/nodes.py @@ -1,5 +1,6 @@ """ -Main node implementations for ComfyUI-Grounding +ComfyUI-Grounding: Unified object detection nodes for ComfyUI +Simplified architecture with just 3 nodes for all grounding models """ import torch @@ -14,469 +15,284 @@ import torchvision.transforms as T # Global model cache for keeping models in memory MODEL_CACHE = {} -# Model registry for GroundingDINO variants -GROUNDING_DINO_MODELS = { - # Original GroundingDINO models - "GroundingDINO_SwinT_OGC (694MB)": { - "config": "GroundingDINO_SwinT_OGC.cfg.py", - "checkpoint": "groundingdino_swint_ogc.pth", - "url": "https://huggingface.co/ShilongLiu/GroundingDINO/resolve/main/groundingdino_swint_ogc.pth", - "type": "local", +# Unified model registry with prefixed names for easy identification +MODEL_REGISTRY = { + # GroundingDINO models + "GroundingDINO: SwinT OGC (694MB)": { + "type": "grounding_dino", + "hf_id": "IDEA-Research/grounding-dino-tiny", + "framework": "transformers", }, - "GroundingDINO_SwinB (938MB)": { - "config": "GroundingDINO_SwinB.cfg.py", - "checkpoint": "groundingdino_swinb_cogcoor.pth", - "url": "https://huggingface.co/ShilongLiu/GroundingDINO/resolve/main/groundingdino_swinb_cogcoor.pth", - "type": "local", + "GroundingDINO: SwinB (938MB)": { + "type": "grounding_dino", + "hf_id": "IDEA-Research/grounding-dino-base", + "framework": "transformers", }, - # MM-GroundingDINO Tiny models (Swin-T backbone) - 0.2B parameters - "MM-GroundingDINO-Tiny (O365+GoldG, 50.4 mAP)": { + # MM-GroundingDINO Tiny models + "MM-GroundingDINO: Tiny O365+GoldG (50.4 mAP)": { + "type": "grounding_dino", "hf_id": "openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg", - "type": "transformers", + "framework": "transformers", }, - "MM-GroundingDINO-Tiny (O365+GoldG+GRIT, 50.5 mAP)": { + "MM-GroundingDINO: Tiny O365+GoldG+GRIT (50.5 mAP)": { + "type": "grounding_dino", "hf_id": "openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_grit", - "type": "transformers", + "framework": "transformers", }, - "MM-GroundingDINO-Tiny (O365+GoldG+V3Det, 50.6 mAP)": { + "MM-GroundingDINO: Tiny O365+GoldG+V3Det (50.6 mAP)": { + "type": "grounding_dino", "hf_id": "openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_v3det", - "type": "transformers", + "framework": "transformers", }, - "MM-GroundingDINO-Tiny (O365+GoldG+GRIT+V3Det, 50.4 mAP)": { - "hf_id": "openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_grit_v3det", - "type": "transformers", - }, - # MM-GroundingDINO Base models (Swin-B backbone) - 0.2B parameters - "MM-GroundingDINO-Base (O365+GoldG+V3Det, 52.5 mAP)": { + # MM-GroundingDINO Base models + "MM-GroundingDINO: Base O365+GoldG+V3Det (52.5 mAP)": { + "type": "grounding_dino", "hf_id": "openmmlab-community/mm_grounding_dino_base_o365v1_goldg_v3det", - "type": "transformers", + "framework": "transformers", }, - "MM-GroundingDINO-Base (All datasets, 59.5 mAP)": { + "MM-GroundingDINO: Base All Datasets (59.5 mAP)": { + "type": "grounding_dino", "hf_id": "openmmlab-community/mm_grounding_dino_base_all", - "type": "transformers", + "framework": "transformers", }, - # MM-GroundingDINO Large models (Swin-L backbone) - 0.3B parameters - "MM-GroundingDINO-Large (O365v2+OIv6+GoldG, 53.0 mAP)": { + # MM-GroundingDINO Large models + "MM-GroundingDINO: Large O365v2+OIv6+GoldG (53.0 mAP)": { + "type": "grounding_dino", "hf_id": "openmmlab-community/mm_grounding_dino_large_o365v2_oiv6_goldg", - "type": "transformers", + "framework": "transformers", }, - "MM-GroundingDINO-Large (All datasets, 60.3 mAP)": { + "MM-GroundingDINO: Large All Datasets (60.3 mAP)": { + "type": "grounding_dino", "hf_id": "openmmlab-community/mm_grounding_dino_large_all", - "type": "transformers", + "framework": "transformers", }, -} - -YOLO_WORLD_MODELS = { - "yolov8s-world": { - "url": "https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8s-world.pt", + # OWLv2 models + "OWLv2: Base Patch16": { + "type": "owlv2", + "hf_id": "google/owlv2-base-patch16", + "framework": "transformers", }, - "yolov8m-world": { - "url": "https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8m-world.pt", + "OWLv2: Large Patch14": { + "type": "owlv2", + "hf_id": "google/owlv2-large-patch14", + "framework": "transformers", }, - "yolov8l-world": { - "url": "https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8l-world.pt", + "OWLv2: Base Patch16 Ensemble": { + "type": "owlv2", + "hf_id": "google/owlv2-base-patch16-ensemble", + "framework": "transformers", }, - "yolov8x-world": { - "url": "https://github.com/ultralytics/assets/releases/download/v8.2.0/yolov8x-world.pt", + "OWLv2: Large Patch14 Ensemble": { + "type": "owlv2", + "hf_id": "google/owlv2-large-patch14-ensemble", + "framework": "transformers", + }, + # Florence-2 models + "Florence-2: Base (0.23B params)": { + "type": "florence2", + "hf_id": "microsoft/Florence-2-base", + "framework": "transformers", + }, + "Florence-2: Large (0.77B params)": { + "type": "florence2", + "hf_id": "microsoft/Florence-2-large", + "framework": "transformers", + }, + # YOLO-World models + "YOLO-World: v8s (Small)": { + "type": "yolo_world", + "url": "https://github.com/ultralytics/assets/releases/download/v8.3.0/yolov8s-worldv2.pt", + "framework": "ultralytics", + }, + "YOLO-World: v8m (Medium)": { + "type": "yolo_world", + "url": "https://github.com/ultralytics/assets/releases/download/v8.3.0/yolov8m-worldv2.pt", + "framework": "ultralytics", + }, + "YOLO-World: v8l (Large)": { + "type": "yolo_world", + "url": "https://github.com/ultralytics/assets/releases/download/v8.3.0/yolov8l-worldv2.pt", + "framework": "ultralytics", + }, + "YOLO-World: v8x (Extra Large)": { + "type": "yolo_world", + "url": "https://github.com/ultralytics/assets/releases/download/v8.3.0/yolov8x-worldv2.pt", + "framework": "ultralytics", }, } -class GroundingDINOModelLoader: +class GroundingModelLoader: """ - Loads GroundingDINO models for open-vocabulary object detection + Unified model loader for all grounding models + Supports: GroundingDINO, MM-GroundingDINO, OWLv2, Florence-2, YOLO-World """ @classmethod def INPUT_TYPES(cls): return { "required": { - "model_name": (list(GROUNDING_DINO_MODELS.keys()), {"default": "GroundingDINO_SwinT_OGC (694MB)"}), + "model": (list(MODEL_REGISTRY.keys()), { + "default": "Florence-2: Base (0.23B params)", + }), "keep_in_memory": ("BOOLEAN", {"default": True}), + }, + "optional": { + "florence2_attn": (["eager", "sdpa", "flash_attention_2"], { + "default": "eager", + "tooltip": "⚠️ FLORENCE-2 ONLY! Ignored for all other models. eager=most compatible, sdpa=PyTorch 2.0+, flash_attention_2=A100/H100" + }), } } - RETURN_TYPES = ("GROUNDING_DINO_MODEL",) + RETURN_TYPES = ("GROUNDING_MODEL",) RETURN_NAMES = ("model",) FUNCTION = "load_model" CATEGORY = "grounding" - def load_model(self, model_name: str, keep_in_memory: bool = True): - """Load GroundingDINO model with caching support""" - - # Create cache key - cache_key = f"grounding_dino_{model_name}" - - # Check cache if keep_in_memory is True - if keep_in_memory and cache_key in MODEL_CACHE: - print(f"Loading {model_name} from cache") - return (MODEL_CACHE[cache_key],) - - # Clear from cache if keep_in_memory is False - if not keep_in_memory and cache_key in MODEL_CACHE: - print(f"Removing {model_name} from cache") - del MODEL_CACHE[cache_key] + def load_model(self, model: str, keep_in_memory: bool = True, florence2_attn: str = "eager"): + """Load any grounding model with unified interface""" # Get model config - model_config = GROUNDING_DINO_MODELS[model_name] - model_type = model_config.get("type", "local") + if model not in MODEL_REGISTRY: + raise ValueError(f"Unknown model: {model}") - # Load model based on type - if model_type == "local": - model = self._load_local(model_name, model_config) - else: # transformers - model = self._load_with_transformers(model_name, model_config) + config = MODEL_REGISTRY[model] + model_type = config["type"] + + # Create cache key + cache_key = f"{model_type}_{model}_{florence2_attn if model_type == 'florence2' else 'default'}" + + # Check cache + if keep_in_memory and cache_key in MODEL_CACHE: + print(f"✅ Loading {model} from cache") + return (MODEL_CACHE[cache_key],) + + # Clear cache if requested + if not keep_in_memory and cache_key in MODEL_CACHE: + print(f"🗑️ Removing {model} from cache") + del MODEL_CACHE[cache_key] + + # Load model based on framework + if config["framework"] == "transformers": + loaded_model = self._load_transformers_model(model, config, florence2_attn) + elif config["framework"] == "ultralytics": + loaded_model = self._load_yolo_model(model, config) + else: + raise ValueError(f"Unknown framework: {config['framework']}") # Cache if requested if keep_in_memory: - MODEL_CACHE[cache_key] = model - print(f"Cached {model_name} in memory") + MODEL_CACHE[cache_key] = loaded_model + print(f"💾 Cached {model} in memory") - return (model,) + return (loaded_model,) - def _load_local(self, model_name: str, model_config: dict): - """Load GroundingDINO using local implementation""" - try: - # Try using the local implementation from comfyui-sam2 - from custom_nodes.comfyui_sam2.node import load_groundingdino_model - model = load_groundingdino_model(model_name) - return model - except ImportError: - raise RuntimeError( - f"Local GroundingDINO implementation not found. " - f"Please install ComfyUI-segment-anything-2 to use {model_name}, " - f"or use MM-GroundingDINO models instead." - ) + def _load_transformers_model(self, model_name: str, config: dict, florence2_attn: str): + """Load transformers-based models (GroundingDINO, OWLv2, Florence-2)""" + model_type = config["type"] + hf_id = config["hf_id"] + device = mm.get_torch_device() - def _load_with_transformers(self, model_name: str, model_config: dict): - """Load MM-GroundingDINO using HuggingFace transformers""" - try: + # Setup models directory + script_dir = os.path.dirname(os.path.abspath(__file__)) + models_dir = os.path.join(script_dir, "models") + os.makedirs(models_dir, exist_ok=True) + + print(f"📥 Loading {model_name} from HuggingFace ({hf_id})...") + + if model_type in ["grounding_dino"]: from transformers import AutoProcessor, AutoModelForZeroShotObjectDetection - - device = mm.get_torch_device() - - # Save models to local Models folder in plugin directory - script_dir = os.path.dirname(os.path.abspath(__file__)) - models_dir = os.path.join(script_dir, "Models") - os.makedirs(models_dir, exist_ok=True) - - # Get HuggingFace model ID - hf_id = model_config.get("hf_id") - - if not hf_id: - raise ValueError(f"No HuggingFace model ID found for {model_name}") - - print(f"📥 Loading {model_name} from HuggingFace ({hf_id})...") - print("Downloading model files (progress bars will appear below):") - - # Load processor and model (automatically shows tqdm progress bars) processor = AutoProcessor.from_pretrained(hf_id, cache_dir=models_dir) model = AutoModelForZeroShotObjectDetection.from_pretrained(hf_id, cache_dir=models_dir) - model.to(device) - model.eval() + elif model_type == "owlv2": + from transformers import Owlv2Processor, Owlv2ForObjectDetection + processor = Owlv2Processor.from_pretrained(hf_id, cache_dir=models_dir) + model = Owlv2ForObjectDetection.from_pretrained(hf_id, cache_dir=models_dir) - print(f"✅ Successfully loaded {model_name}") + elif model_type == "florence2": + from transformers import AutoProcessor, AutoModelForCausalLM + print(f"Using Florence-2 attention implementation: {florence2_attn}") + processor = AutoProcessor.from_pretrained(hf_id, cache_dir=models_dir, trust_remote_code=True) + model = AutoModelForCausalLM.from_pretrained( + hf_id, + cache_dir=models_dir, + trust_remote_code=True, + attn_implementation=florence2_attn + ) + else: + raise ValueError(f"Unknown transformers model type: {model_type}") - return {"model": model, "processor": processor, "type": "transformers"} - except Exception as e: - raise RuntimeError(f"Failed to load GroundingDINO model: {e}") + model.to(device) + model.eval() + + print(f"✅ Successfully loaded {model_name}") + + return { + "model": model, + "processor": processor, + "type": model_type, + "framework": "transformers" + } + + def _load_yolo_model(self, model_name: str, config: dict): + """Load YOLO-World models""" + try: + from ultralytics import YOLOWorld + except ImportError: + raise RuntimeError("ultralytics not installed. Run: pip install ultralytics") + + # Setup models directory + script_dir = os.path.dirname(os.path.abspath(__file__)) + models_dir = os.path.join(script_dir, "models") + os.makedirs(models_dir, exist_ok=True) + + # Extract filename from URL + url = config["url"] + filename = url.split("/")[-1] + model_path = os.path.join(models_dir, filename) + + # Download if needed + if not os.path.exists(model_path): + print(f"📥 Downloading {model_name}...") + from torch.hub import download_url_to_file + download_url_to_file(url, model_path, progress=True) + print(f"✅ Downloaded {model_name}") + + # Load model + print(f"📂 Loading {model_name} from {model_path}") + model = YOLOWorld(model_path) + + # Move to device + device = mm.get_torch_device() + model.to(device) + + print(f"✅ Successfully loaded {model_name}") + + return { + "model": model, + "type": "yolo_world", + "framework": "ultralytics" + } -class GroundingDINODetector: +class GroundingDetector: """ - Performs object detection using GroundingDINO with text prompts + Unified detector for all grounding models + Auto-detects model type and uses appropriate detection method """ @classmethod def INPUT_TYPES(cls): return { "required": { - "model": ("GROUNDING_DINO_MODEL",), + "model": ("GROUNDING_MODEL",), "image": ("IMAGE",), "prompt": ("STRING", { "default": "person . car . dog .", "multiline": True, - }), - "box_threshold": ("FLOAT", { - "default": 0.3, - "min": 0.0, - "max": 1.0, - "step": 0.01, - }), - "text_threshold": ("FLOAT", { - "default": 0.25, - "min": 0.0, - "max": 1.0, - "step": 0.01, - }), - }, - "optional": { - "single_box_mode": ("BOOLEAN", { - "default": False, - "tooltip": "Return only the highest-scoring detection (REC mode for referring expressions)" - }), - "output_format": (["list_only", "dict_with_data"], { - "default": "list_only", - "tooltip": "list_only: SAM2-compatible list of lists | dict_with_data: dict with boxes/labels/scores" - }), - } - } - - RETURN_TYPES = ("BBOX", "IMAGE", "STRING") - RETURN_NAMES = ("bboxes", "annotated_image", "labels") - FUNCTION = "detect" - CATEGORY = "grounding" - - def detect(self, model, image, prompt: str, box_threshold: float, text_threshold: float, single_box_mode: bool = False, output_format: str = "list_only"): - """Run detection on image with text prompt""" - - # Process all images in batch - batch_size = image.shape[0] - all_boxes = [] - all_labels = [] - all_scores = [] - annotated_images = [] - - for i in range(batch_size): - # Convert ComfyUI image format (B, H, W, C) in [0, 1] to PIL - image_np = (image[i].cpu().numpy() * 255).astype(np.uint8) - pil_image = Image.fromarray(image_np) - - # Check model type and run detection - if isinstance(model, dict) and model.get("type") == "transformers": - bboxes, annotated_tensor, _ = self._detect_transformers(model, pil_image, prompt, box_threshold, text_threshold, single_box_mode) - else: - bboxes, annotated_tensor, _ = self._detect_local(model, pil_image, prompt, box_threshold, single_box_mode) - - # Accumulate results - all_boxes.append(bboxes["boxes"]) - all_labels.append(bboxes["labels"]) - all_scores.append(bboxes["scores"]) - annotated_images.append(annotated_tensor) - - # Combine annotated images - annotated_batch = torch.cat(annotated_images, dim=0) - - # Format batched output based on output_format - if output_format == "list_only": - # SAM2-compatible format: list of lists (boxes only) - batched_bboxes = all_boxes - else: # dict_with_data - # Dict format with all information - batched_bboxes = { - "boxes": all_boxes, - "labels": all_labels, - "scores": all_scores, - } - - # Format labels string (from first image) - if len(all_labels) > 0 and len(all_labels[0]) > 0: - labels_str = ", ".join([f"{label} ({score:.2f})" for label, score in zip(all_labels[0], all_scores[0])]) - else: - labels_str = "" - - return (batched_bboxes, annotated_batch, labels_str) - - def _detect_transformers(self, model_dict, pil_image, prompt, box_threshold, text_threshold, single_box_mode=False): - """Detection using transformers implementation""" - model = model_dict["model"] - processor = model_dict["processor"] - device = mm.get_torch_device() - - # Prepare inputs - inputs = processor(images=pil_image, text=prompt, return_tensors="pt").to(device) - - # Run inference - with torch.no_grad(): - outputs = model(**inputs) - - # Post-process (use 'threshold' instead of 'box_threshold' for GroundingDINO) - results = processor.post_process_grounded_object_detection( - outputs, - inputs.input_ids, - threshold=box_threshold, - text_threshold=text_threshold, - target_sizes=[pil_image.size[::-1]] - )[0] - - # Extract boxes and labels - boxes = results["boxes"].cpu().numpy() - labels = results["labels"] - scores = results["scores"].cpu().numpy() - - # Override labels if prompt has no separators - # Check for periods and commas (common separators in GroundingDINO prompts) - if '.' not in prompt and ',' not in prompt: - # Use full prompt as label for all detections - clean_prompt = prompt.strip() - labels = [clean_prompt] * len(labels) - - # Single box mode: return only highest-scoring detection (REC mode) - if single_box_mode and len(boxes) > 0: - top_idx = scores.argmax() - boxes = boxes[top_idx:top_idx+1] - labels = [labels[top_idx]] - scores = scores[top_idx:top_idx+1] - - # Format output - bboxes = { - "boxes": boxes, - "labels": labels, - "scores": scores, - } - - # Create annotated image - annotated = self._draw_boxes(pil_image, boxes, labels, scores) - annotated_tensor = torch.from_numpy(np.array(annotated).astype(np.float32) / 255.0).unsqueeze(0) - - # Format labels string - labels_str = ", ".join([f"{label} ({score:.2f})" for label, score in zip(labels, scores)]) - - return (bboxes, annotated_tensor, labels_str) - - def _detect_local(self, model, pil_image, prompt, box_threshold, single_box_mode=False): - """Detection using local GroundingDINO implementation""" - # Use the implementation from comfyui-sam2 - from custom_nodes.comfyui_sam2.node import groundingdino_predict - - boxes = groundingdino_predict(model, pil_image, prompt, box_threshold) - - # Convert to numpy - boxes_np = boxes.cpu().numpy() - - # Create labels (simplified - just use prompt) - labels = [prompt.strip().rstrip('.')] * len(boxes_np) - scores = np.ones(len(boxes_np)) * box_threshold # Approximate - - # Single box mode: return only first detection (local impl doesn't provide scores) - if single_box_mode and len(boxes_np) > 0: - boxes_np = boxes_np[0:1] - labels = [labels[0]] - scores = scores[0:1] - - bboxes = { - "boxes": boxes_np, - "labels": labels, - "scores": scores, - } - - # Create annotated image - annotated = self._draw_boxes(pil_image, boxes_np, labels, scores) - annotated_tensor = torch.from_numpy(np.array(annotated).astype(np.float32) / 255.0).unsqueeze(0) - - labels_str = ", ".join(labels) - - return (bboxes, annotated_tensor, labels_str) - - def _draw_boxes(self, image, boxes, labels, scores): - """Draw bounding boxes on image""" - from PIL import ImageDraw, ImageFont - - draw = ImageDraw.Draw(image) - - for box, label, score in zip(boxes, labels, scores): - # box format: [x1, y1, x2, y2] - x1, y1, x2, y2 = box - - # Draw rectangle - draw.rectangle([x1, y1, x2, y2], outline="red", width=3) - - # Draw label - text = f"{label}: {score:.2f}" - draw.text((x1, y1 - 10), text, fill="red") - - return image - - -class YOLOWorldModelLoader: - """ - Loads YOLO-World models for open-vocabulary object detection - """ - - @classmethod - def INPUT_TYPES(cls): - return { - "required": { - "model_name": (list(YOLO_WORLD_MODELS.keys()), {"default": "yolov8s-world"}), - "keep_in_memory": ("BOOLEAN", {"default": True}), - } - } - - RETURN_TYPES = ("YOLO_WORLD_MODEL",) - RETURN_NAMES = ("model",) - FUNCTION = "load_model" - CATEGORY = "grounding" - - def load_model(self, model_name: str, keep_in_memory: bool = True): - """Load YOLO-World model with caching support""" - - # Create cache key - cache_key = f"yolo_world_{model_name}" - - # Check cache if keep_in_memory is True - if keep_in_memory and cache_key in MODEL_CACHE: - print(f"Loading {model_name} from cache") - return (MODEL_CACHE[cache_key],) - - # Clear from cache if keep_in_memory is False - if not keep_in_memory and cache_key in MODEL_CACHE: - print(f"Removing {model_name} from cache") - del MODEL_CACHE[cache_key] - - try: - from ultralytics import YOLOWorld - - # Save models to local Models folder in plugin directory - script_dir = os.path.dirname(os.path.abspath(__file__)) - models_dir = os.path.join(script_dir, "Models") - os.makedirs(models_dir, exist_ok=True) - - model_filename = f"{model_name}.pt" - model_path = os.path.join(models_dir, model_filename) - - # Download if needed - if not os.path.exists(model_path): - print(f"📥 Downloading {model_name} to {models_dir}...") - print("Progress bar:") - from torch.hub import download_url_to_file - download_url_to_file( - YOLO_WORLD_MODELS[model_name]["url"], - model_path, - progress=True # Explicitly enable progress bar - ) - print(f"✅ Downloaded {model_name} successfully") - - # Load model - print(f"Loading {model_name} from {model_path}") - model = YOLOWorld(model_path) - - # Cache if requested - if keep_in_memory: - MODEL_CACHE[cache_key] = model - print(f"Cached {model_name} in memory") - - return (model,) - except ImportError: - raise RuntimeError("ultralytics not installed. Run: pip install ultralytics") - except Exception as e: - raise RuntimeError(f"Failed to load YOLO-World model: {e}") - - -class YOLOWorldDetector: - """ - Performs object detection using YOLO-World with text prompts - """ - - @classmethod - def INPUT_TYPES(cls): - return { - "required": { - "model": ("YOLO_WORLD_MODEL",), - "image": ("IMAGE",), - "classes": ("STRING", { - "default": "person, car, dog", - "multiline": True, + "tooltip": "Use periods (.) for GroundingDINO/OWLv2/Florence-2, commas (,) for YOLO-World" }), "confidence_threshold": ("FLOAT", { "default": 0.3, @@ -486,45 +302,294 @@ class YOLOWorldDetector: }), }, "optional": { + "text_threshold": ("FLOAT", { + "default": 0.25, + "min": 0.0, + "max": 1.0, + "step": 0.01, + "tooltip": "⚠️ Only used for GroundingDINO models" + }), "single_box_mode": ("BOOLEAN", { "default": False, - "tooltip": "Return only the highest-scoring detection (REC mode for referring expressions)" + "tooltip": "Return only highest-scoring detection (for referring expressions)" }), - "output_format": (["list_only", "dict_with_data"], { + "bbox_output_format": (["list_only", "dict_with_data"], { "default": "list_only", - "tooltip": "list_only: SAM2-compatible list of lists | dict_with_data: dict with boxes/labels/scores" + "tooltip": "list_only: SAM2-compatible | dict_with_data: includes labels/scores" + }), + "output_masks": ("BOOLEAN", { + "default": False, + "tooltip": "Convert bounding boxes to binary masks" }), } } - RETURN_TYPES = ("BBOX", "IMAGE", "STRING") - RETURN_NAMES = ("bboxes", "annotated_image", "labels") + RETURN_TYPES = ("BBOX", "IMAGE", "STRING", "MASK") + RETURN_NAMES = ("bboxes", "annotated_image", "labels", "masks") FUNCTION = "detect" CATEGORY = "grounding" - def detect(self, model, image, classes: str, confidence_threshold: float, single_box_mode: bool = False, output_format: str = "list_only"): - """Run detection with YOLO-World""" + def detect(self, model, image, prompt: str, confidence_threshold: float, + text_threshold: float = 0.25, single_box_mode: bool = False, + bbox_output_format: str = "list_only", output_masks: bool = False): + """Universal detection that works with any model""" + + model_type = model["type"] + + # Route to appropriate detection method + if model_type == "grounding_dino": + return self._detect_grounding_dino(model, image, prompt, confidence_threshold, + text_threshold, single_box_mode, bbox_output_format, output_masks) + elif model_type == "owlv2": + return self._detect_owlv2(model, image, prompt, confidence_threshold, + single_box_mode, bbox_output_format, output_masks) + elif model_type == "florence2": + return self._detect_florence2(model, image, prompt, confidence_threshold, + single_box_mode, bbox_output_format, output_masks) + elif model_type == "yolo_world": + return self._detect_yolo_world(model, image, prompt, confidence_threshold, + single_box_mode, bbox_output_format, output_masks) + else: + raise ValueError(f"Unknown model type: {model_type}") + + def _detect_grounding_dino(self, model_dict, image, prompt, box_threshold, text_threshold, + single_box_mode, bbox_output_format, output_masks): + """Detection using GroundingDINO/MM-GroundingDINO""" + model = model_dict["model"] + processor = model_dict["processor"] + device = mm.get_torch_device() - # Process all images in batch batch_size = image.shape[0] all_boxes = [] all_labels = [] all_scores = [] annotated_images = [] - # Parse classes - check if we should use full text as label + for i in range(batch_size): + # Convert to PIL + image_np = (image[i].cpu().numpy() * 255).astype(np.uint8) + pil_image = Image.fromarray(image_np) + + # Prepare inputs + inputs = processor(images=pil_image, text=prompt, return_tensors="pt").to(device) + + # Run inference + with torch.no_grad(): + outputs = model(**inputs) + + # Post-process + results = processor.post_process_grounded_object_detection( + outputs, + inputs.input_ids, + threshold=box_threshold, + text_threshold=text_threshold, + target_sizes=[pil_image.size[::-1]] + )[0] + + # Extract results + boxes = results["boxes"].cpu().numpy() + labels = results["labels"] + scores = results["scores"].cpu().numpy() + + # Override labels if no separators (REC mode) + if '.' not in prompt and ',' not in prompt: + labels = [prompt.strip()] * len(labels) + + # Single box mode + if single_box_mode and len(boxes) > 0: + top_idx = scores.argmax() + boxes = boxes[top_idx:top_idx+1] + labels = [labels[top_idx]] + scores = scores[top_idx:top_idx+1] + + # Draw boxes + annotated = self._draw_boxes(pil_image, boxes, labels, scores) + annotated_tensor = torch.from_numpy(np.array(annotated).astype(np.float32) / 255.0).unsqueeze(0) + + all_boxes.append(boxes) + all_labels.append(labels) + all_scores.append(scores) + annotated_images.append(annotated_tensor) + + return self._format_output(all_boxes, all_labels, all_scores, annotated_images, + image.shape, bbox_output_format, output_masks) + + def _detect_owlv2(self, model_dict, image, prompt, box_threshold, + single_box_mode, bbox_output_format, output_masks): + """Detection using OWLv2""" + model = model_dict["model"] + processor = model_dict["processor"] + device = mm.get_torch_device() + + batch_size = image.shape[0] + all_boxes = [] + all_labels = [] + all_scores = [] + annotated_images = [] + + for i in range(batch_size): + # Convert to PIL + image_np = (image[i].cpu().numpy() * 255).astype(np.uint8) + pil_image = Image.fromarray(image_np) + + # Parse text queries + has_separator = '.' in prompt + if has_separator: + text_queries = [[c.strip() for c in prompt.split(".") if c.strip()]] + else: + text_queries = [[prompt.strip()]] + + # Prepare inputs + inputs = processor(images=pil_image, text=text_queries, return_tensors="pt").to(device) + + # Run inference + with torch.no_grad(): + outputs = model(**inputs) + + # Post-process + target_sizes = torch.tensor([pil_image.size[::-1]]).to(device) + results = processor.post_process_object_detection( + outputs=outputs, + threshold=box_threshold, + target_sizes=target_sizes + )[0] + + # Extract results + boxes = results["boxes"].cpu().numpy() + labels_indices = results["labels"].cpu().numpy() + scores = results["scores"].cpu().numpy() + + # Map labels + labels = [text_queries[0][idx] for idx in labels_indices] + + # Override labels if no separator + if not has_separator and len(labels) > 0: + labels = [prompt.strip()] * len(labels) + + # Single box mode + if single_box_mode and len(boxes) > 0: + top_idx = scores.argmax() + boxes = boxes[top_idx:top_idx+1] + labels = [labels[top_idx]] + scores = scores[top_idx:top_idx+1] + + # Draw boxes + annotated = self._draw_boxes(pil_image, boxes, labels, scores) + annotated_tensor = torch.from_numpy(np.array(annotated).astype(np.float32) / 255.0).unsqueeze(0) + + all_boxes.append(boxes) + all_labels.append(labels) + all_scores.append(scores) + annotated_images.append(annotated_tensor) + + return self._format_output(all_boxes, all_labels, all_scores, annotated_images, + image.shape, bbox_output_format, output_masks) + + def _detect_florence2(self, model_dict, image, prompt, box_threshold, + single_box_mode, bbox_output_format, output_masks): + """Detection using Florence-2""" + model = model_dict["model"] + processor = model_dict["processor"] + device = mm.get_torch_device() + + batch_size = image.shape[0] + all_boxes = [] + all_labels = [] + all_scores = [] + annotated_images = [] + + for i in range(batch_size): + # Convert to PIL + image_np = (image[i].cpu().numpy() * 255).astype(np.uint8) + pil_image = Image.fromarray(image_np) + + # Parse text queries + has_separator = '.' in prompt + if has_separator: + text_queries = [c.strip() for c in prompt.split(".") if c.strip()] + caption = " ".join(text_queries) + else: + caption = prompt.strip() + text_queries = [caption] + + # Prepare task and inputs + task_prompt = "" + inputs = processor(text=task_prompt + caption, images=pil_image, return_tensors="pt").to(device) + + # Run inference + with torch.no_grad(): + generated_ids = model.generate( + input_ids=inputs["input_ids"], + pixel_values=inputs["pixel_values"], + max_new_tokens=1024, + num_beams=3, + use_cache=False, # Prevent past_key_values issues + ) + + # Decode and parse + generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0] + parsed_answer = processor.post_process_generation(generated_text, task=task_prompt, image_size=pil_image.size) + + # Extract bboxes (already in pixel coordinates!) + result = parsed_answer.get(task_prompt, {}) + boxes = np.array(result.get("bboxes", [])) + labels_from_model = result.get("labels", []) + + # Assign labels + if len(labels_from_model) == len(boxes) and all(labels_from_model): + labels = labels_from_model + else: + labels = text_queries * (len(boxes) // max(len(text_queries), 1) + 1) + labels = labels[:len(boxes)] + + # Create dummy scores + scores = np.ones(len(boxes)) * 0.9 + + # Override labels if no separator + if not has_separator and len(labels) > 0: + labels = [prompt.strip()] * len(labels) + + # Single box mode + if single_box_mode and len(boxes) > 0: + boxes = boxes[0:1] + labels = [labels[0]] + scores = scores[0:1] + + # Draw boxes + annotated = self._draw_boxes(pil_image, boxes, labels, scores) + annotated_tensor = torch.from_numpy(np.array(annotated).astype(np.float32) / 255.0).unsqueeze(0) + + all_boxes.append(boxes) + all_labels.append(labels) + all_scores.append(scores) + annotated_images.append(annotated_tensor) + + return self._format_output(all_boxes, all_labels, all_scores, annotated_images, + image.shape, bbox_output_format, output_masks) + + def _detect_yolo_world(self, model_dict, image, classes, confidence_threshold, + single_box_mode, bbox_output_format, output_masks): + """Detection using YOLO-World""" + model = model_dict["model"] + + batch_size = image.shape[0] + all_boxes = [] + all_labels = [] + all_scores = [] + annotated_images = [] + + # Parse classes has_separator = ',' in classes if has_separator: class_list = [c.strip() for c in classes.split(",")] else: - # No comma separator - treat as single class/referring expression class_list = [classes.strip()] - # Set classes for the model + # Set classes for model model.set_classes(class_list) for i in range(batch_size): - # Convert ComfyUI image format + # Convert to numpy image_np = (image[i].cpu().numpy() * 255).astype(np.uint8) # Run inference @@ -539,37 +604,56 @@ class YOLOWorldDetector: # Map class IDs to names labels = [class_list[cid] for cid in class_ids] - # Override labels if no separator (referring expression mode) + # Override labels if no separator if not has_separator and len(labels) > 0: - # Use full prompt as label for all detections labels = [classes.strip()] * len(labels) - # Single box mode: return only highest-scoring detection (REC mode) + # Single box mode if single_box_mode and len(boxes) > 0: top_idx = scores.argmax() boxes = boxes[top_idx:top_idx+1] labels = [labels[top_idx]] scores = scores[top_idx:top_idx+1] - # Get annotated image - annotated_np = result.plot() - annotated_tensor = torch.from_numpy(annotated_np.astype(np.float32) / 255.0).unsqueeze(0) + # Draw boxes + pil_image = Image.fromarray(image_np) + annotated = self._draw_boxes(pil_image, boxes, labels, scores) + annotated_tensor = torch.from_numpy(np.array(annotated).astype(np.float32) / 255.0).unsqueeze(0) - # Accumulate results all_boxes.append(boxes) all_labels.append(labels) all_scores.append(scores) annotated_images.append(annotated_tensor) + return self._format_output(all_boxes, all_labels, all_scores, annotated_images, + image.shape, bbox_output_format, output_masks) + + def _draw_boxes(self, image, boxes, labels, scores): + """Draw bounding boxes on image""" + from PIL import ImageDraw + + image = image.copy() + draw = ImageDraw.Draw(image) + + for box, label, score in zip(boxes, labels, scores): + x1, y1, x2, y2 = box + draw.rectangle([x1, y1, x2, y2], outline="red", width=5) + text = f"{label}: {score:.2f}" + draw.text((x1, max(y1 - 15, 0)), text, fill="red") + + return image + + def _format_output(self, all_boxes, all_labels, all_scores, annotated_images, + image_shape, bbox_output_format, output_masks): + """Format detection output and optionally create masks""" + # Combine annotated images annotated_batch = torch.cat(annotated_images, dim=0) - # Format batched output based on output_format - if output_format == "list_only": - # SAM2-compatible format: list of lists (boxes only) + # Format bboxes based on bbox_output_format + if bbox_output_format == "list_only": batched_bboxes = all_boxes else: # dict_with_data - # Dict format with all information batched_bboxes = { "boxes": all_boxes, "labels": all_labels, @@ -582,7 +666,67 @@ class YOLOWorldDetector: else: labels_str = "" - return (batched_bboxes, annotated_batch, labels_str) + # Create masks if requested + if output_masks: + masks = self._boxes_to_masks(all_boxes, image_shape) + else: + # Return empty mask + batch_size, height, width, _ = image_shape + masks = torch.zeros((1, height, width)) + + return (batched_bboxes, annotated_batch, labels_str, masks) + + def _boxes_to_masks(self, all_boxes, image_shape): + """Convert bounding boxes to binary masks + + Creates one mask per bounding box across all images in batch. + Returns tensor of shape (N, H, W) where N is total number of boxes. + """ + batch_size, height, width, _ = image_shape + all_masks = [] + + print(f"[Mask Creation] Image shape: {image_shape}") + print(f"[Mask Creation] Number of images in batch: {len(all_boxes)}") + + for img_idx, boxes in enumerate(all_boxes): + print(f"[Mask Creation] Image {img_idx}: {len(boxes)} boxes") + + # Create individual masks for each box in this image + for box_idx, box in enumerate(boxes): + mask = np.zeros((height, width), dtype=np.float32) + x1, y1, x2, y2 = box.astype(int) + + print(f" Box {box_idx}: [{x1}, {y1}, {x2}, {y2}]") + + # Clamp to image boundaries + x1, y1 = max(0, x1), max(0, y1) + x2, y2 = min(width, x2), min(height, y2) + + print(f" Clamped: [{x1}, {y1}, {x2}, {y2}]") + + # Check if box is valid + if x2 > x1 and y2 > y1: + # Fill the box region with 1.0 + mask[y1:y2, x1:x2] = 1.0 + filled_pixels = (y2 - y1) * (x2 - x1) + print(f" Filled {filled_pixels} pixels") + else: + print(f" WARNING: Invalid box dimensions!") + + all_masks.append(mask) + + # If no boxes detected, return single empty mask + if len(all_masks) == 0: + print("[Mask Creation] No boxes detected, returning empty mask") + return torch.zeros((1, height, width)) + + # Stack all masks + masks_tensor = torch.from_numpy(np.stack(all_masks)) + print(f"[Mask Creation] Final masks shape: {masks_tensor.shape}") + print(f"[Mask Creation] Masks min/max values: {masks_tensor.min()}/{masks_tensor.max()}") + print(f"[Mask Creation] Non-zero pixels: {(masks_tensor > 0).sum().item()}") + + return masks_tensor class BboxVisualizer: @@ -608,328 +752,56 @@ class BboxVisualizer: CATEGORY = "grounding" def visualize(self, image, bboxes, line_width=3): - """Draw bounding boxes on image""" + """Draw bounding boxes on image (supports batches)""" from PIL import ImageDraw - # Convert to PIL - image_np = (image[0].cpu().numpy() * 255).astype(np.uint8) - pil_image = Image.fromarray(image_np) + batch_size = image.shape[0] + annotated_images = [] - draw = ImageDraw.Draw(pil_image) + for batch_idx in range(batch_size): + # Convert to PIL + image_np = (image[batch_idx].cpu().numpy() * 255).astype(np.uint8) + pil_image = Image.fromarray(image_np) - boxes = bboxes["boxes"] - labels = bboxes.get("labels", []) - scores = bboxes.get("scores", []) + draw = ImageDraw.Draw(pil_image) - for i, box in enumerate(boxes): - x1, y1, x2, y2 = box - draw.rectangle([x1, y1, x2, y2], outline="red", width=line_width) - - if i < len(labels): - label = labels[i] - score = scores[i] if i < len(scores) else 0.0 - text = f"{label}: {score:.2f}" - draw.text((x1, y1 - 10), text, fill="red") - - # Convert back to tensor - annotated_np = np.array(pil_image).astype(np.float32) / 255.0 - annotated_tensor = torch.from_numpy(annotated_np).unsqueeze(0) - - return (annotated_tensor,) - - -class BboxToMask: - """ - Converts bounding boxes to masks - """ - - @classmethod - def INPUT_TYPES(cls): - return { - "required": { - "image": ("IMAGE",), - "bboxes": ("BBOX",), - }, - "optional": { - "combine_masks": ("BOOLEAN", {"default": False}), - } - } - - RETURN_TYPES = ("MASK",) - RETURN_NAMES = ("masks",) - FUNCTION = "bbox_to_mask" - CATEGORY = "grounding" - - def bbox_to_mask(self, image, bboxes, combine_masks=False): - """Convert bounding boxes to binary masks""" - - # Get image dimensions - batch_size, height, width, channels = image.shape - boxes = bboxes["boxes"] - - if combine_masks: - # Single combined mask - mask = np.zeros((height, width), dtype=np.float32) - for box in boxes: - x1, y1, x2, y2 = box.astype(int) - x1, y1 = max(0, x1), max(0, y1) - x2, y2 = min(width, x2), min(height, y2) - mask[y1:y2, x1:x2] = 1.0 - - mask_tensor = torch.from_numpy(mask).unsqueeze(0) - else: - # Individual masks for each box - masks = [] - for box in boxes: - mask = np.zeros((height, width), dtype=np.float32) - x1, y1, x2, y2 = box.astype(int) - x1, y1 = max(0, x1), max(0, y1) - x2, y2 = min(width, x2), min(height, y2) - mask[y1:y2, x1:x2] = 1.0 - masks.append(mask) - - if len(masks) == 0: - # No detections - return empty mask - mask_tensor = torch.zeros((1, height, width)) + # Handle both output formats + if isinstance(bboxes, dict): + # dict_with_data format + boxes = bboxes["boxes"][batch_idx] if batch_idx < len(bboxes["boxes"]) else [] + labels = bboxes.get("labels", [[]])[batch_idx] if "labels" in bboxes and batch_idx < len(bboxes["labels"]) else [] + scores = bboxes.get("scores", [[]])[batch_idx] if "scores" in bboxes and batch_idx < len(bboxes["scores"]) else [] + elif isinstance(bboxes, list): + # list_only format + boxes = bboxes[batch_idx] if batch_idx < len(bboxes) else [] + labels = [] + scores = [] else: - mask_tensor = torch.from_numpy(np.stack(masks)) + # Fallback + boxes = bboxes + labels = [] + scores = [] - return (mask_tensor,) + # Draw boxes + for i, box in enumerate(boxes): + x1, y1, x2, y2 = box + draw.rectangle([x1, y1, x2, y2], outline="red", width=line_width) + if i < len(labels) and i < len(scores): + label = labels[i] + score = scores[i] + text = f"{label}: {score:.2f}" + draw.text((x1, max(y1 - 10, 0)), text, fill="red") -class UnifiedDetector: - """ - Universal detector that works with both GroundingDINO and YOLO-World models - Automatically detects model type and uses appropriate inference method - """ + # Convert back to tensor + annotated_np = np.array(pil_image).astype(np.float32) / 255.0 + annotated_tensor = torch.from_numpy(annotated_np) + annotated_images.append(annotated_tensor) - @classmethod - def INPUT_TYPES(cls): - return { - "required": { - "model": ("GROUNDING_DINO_MODEL,YOLO_WORLD_MODEL",), - "image": ("IMAGE",), - "prompt": ("STRING", { - "default": "person . car . dog .", - "multiline": True, - "tooltip": "For GroundingDINO: period-separated (e.g., 'cat . dog .'). For YOLO-World: comma-separated (e.g., 'cat, dog')" - }), - "threshold": ("FLOAT", { - "default": 0.3, - "min": 0.0, - "max": 1.0, - "step": 0.01, - }), - }, - "optional": { - "text_threshold": ("FLOAT", { - "default": 0.25, - "min": 0.0, - "max": 1.0, - "step": 0.01, - "tooltip": "Only used for GroundingDINO models" - }), - "single_box_mode": ("BOOLEAN", { - "default": False, - "tooltip": "Return only the highest-scoring detection (REC mode for referring expressions)" - }), - } - } - - RETURN_TYPES = ("BBOX", "IMAGE", "STRING") - RETURN_NAMES = ("bboxes", "annotated_image", "labels") - FUNCTION = "detect" - CATEGORY = "grounding" - - def detect(self, model, image, prompt: str, threshold: float, text_threshold: float = 0.25, single_box_mode: bool = False): - """Universal detection that auto-detects model type""" - - # Detect model type - model_type = self._detect_model_type(model) - - if model_type == "yolo_world": - return self._detect_yolo_world(model, image, prompt, threshold, single_box_mode) - else: # grounding_dino or transformers - return self._detect_grounding_dino(model, image, prompt, threshold, text_threshold, single_box_mode) - - def _detect_model_type(self, model): - """Detect whether this is a YOLO-World or GroundingDINO model""" - # Check for YOLO-World model - try: - from ultralytics import YOLOWorld - if isinstance(model, YOLOWorld): - return "yolo_world" - except ImportError: - pass - - # Check for transformers-based model - if isinstance(model, dict) and model.get("type") == "transformers": - return "grounding_dino_transformers" - - # Default to GroundingDINO (local) - return "grounding_dino_local" - - def _detect_grounding_dino(self, model, image, prompt, box_threshold, text_threshold, single_box_mode=False): - """Detection using GroundingDINO""" - # Convert ComfyUI image format - image_np = (image[0].cpu().numpy() * 255).astype(np.uint8) - pil_image = Image.fromarray(image_np) - - # Check model type - if isinstance(model, dict) and model.get("type") == "transformers": - return self._detect_transformers(model, pil_image, prompt, box_threshold, text_threshold, single_box_mode) + # Stack all annotated images into batch + if len(annotated_images) > 1: + result = torch.stack(annotated_images) else: - return self._detect_local(model, pil_image, prompt, box_threshold, single_box_mode) + result = annotated_images[0].unsqueeze(0) - def _detect_transformers(self, model_dict, pil_image, prompt, box_threshold, text_threshold, single_box_mode=False): - """Detection using transformers implementation""" - model = model_dict["model"] - processor = model_dict["processor"] - device = mm.get_torch_device() - - # Prepare inputs - inputs = processor(images=pil_image, text=prompt, return_tensors="pt").to(device) - - # Run inference - with torch.no_grad(): - outputs = model(**inputs) - - # Post-process (use 'threshold' instead of 'box_threshold' for GroundingDINO) - results = processor.post_process_grounded_object_detection( - outputs, - inputs.input_ids, - threshold=box_threshold, - text_threshold=text_threshold, - target_sizes=[pil_image.size[::-1]] - )[0] - - # Extract boxes and labels - boxes = results["boxes"].cpu().numpy() - labels = results["labels"] - scores = results["scores"].cpu().numpy() - - # Single box mode: return only highest-scoring detection (REC mode) - if single_box_mode and len(boxes) > 0: - top_idx = scores.argmax() - boxes = boxes[top_idx:top_idx+1] - labels = [labels[top_idx]] - scores = scores[top_idx:top_idx+1] - - # Format output - bboxes = { - "boxes": boxes, - "labels": labels, - "scores": scores, - } - - # Create annotated image - annotated = self._draw_boxes(pil_image, boxes, labels, scores) - annotated_tensor = torch.from_numpy(np.array(annotated).astype(np.float32) / 255.0).unsqueeze(0) - - # Format labels string - labels_str = ", ".join([f"{label} ({score:.2f})" for label, score in zip(labels, scores)]) - - return (bboxes, annotated_tensor, labels_str) - - def _detect_local(self, model, pil_image, prompt, box_threshold, single_box_mode=False): - """Detection using local GroundingDINO implementation""" - from custom_nodes.comfyui_sam2.node import groundingdino_predict - - boxes = groundingdino_predict(model, pil_image, prompt, box_threshold) - - # Convert to numpy - boxes_np = boxes.cpu().numpy() - - # Create labels - labels = [prompt.strip().rstrip('.')] * len(boxes_np) - scores = np.ones(len(boxes_np)) * box_threshold - - # Single box mode: return only first detection (local impl doesn't provide scores) - if single_box_mode and len(boxes_np) > 0: - boxes_np = boxes_np[0:1] - labels = [labels[0]] - scores = scores[0:1] - - bboxes = { - "boxes": boxes_np, - "labels": labels, - "scores": scores, - } - - # Create annotated image - annotated = self._draw_boxes(pil_image, boxes_np, labels, scores) - annotated_tensor = torch.from_numpy(np.array(annotated).astype(np.float32) / 255.0).unsqueeze(0) - - labels_str = ", ".join(labels) - - return (bboxes, annotated_tensor, labels_str) - - def _detect_yolo_world(self, model, image, classes, confidence_threshold, single_box_mode=False): - """Detection using YOLO-World""" - # Convert ComfyUI image format - image_np = (image[0].cpu().numpy() * 255).astype(np.uint8) - - # Parse classes (support both comma and period separation) - if ',' in classes: - class_list = [c.strip() for c in classes.split(",")] - else: - class_list = [c.strip() for c in classes.split(".") if c.strip()] - - # Set classes for the model - model.set_classes(class_list) - - # Run inference - results = model(image_np, conf=confidence_threshold) - - # Extract detections - result = results[0] - boxes = result.boxes.xyxy.cpu().numpy() if result.boxes is not None else np.array([]) - scores = result.boxes.conf.cpu().numpy() if result.boxes is not None else np.array([]) - class_ids = result.boxes.cls.cpu().numpy().astype(int) if result.boxes is not None else np.array([]) - - # Map class IDs to names - labels = [class_list[cid] for cid in class_ids] - - # Single box mode: return only highest-scoring detection (REC mode) - if single_box_mode and len(boxes) > 0: - top_idx = scores.argmax() - boxes = boxes[top_idx:top_idx+1] - labels = [labels[top_idx]] - scores = scores[top_idx:top_idx+1] - - # Format output - bboxes = { - "boxes": boxes, - "labels": labels, - "scores": scores, - } - - # Get annotated image - annotated_np = result.plot() - annotated_tensor = torch.from_numpy(annotated_np.astype(np.float32) / 255.0).unsqueeze(0) - - # Format labels string - labels_str = ", ".join([f"{label} ({score:.2f})" for label, score in zip(labels, scores)]) - - return (bboxes, annotated_tensor, labels_str) - - def _draw_boxes(self, image, boxes, labels, scores): - """Draw bounding boxes on image""" - from PIL import ImageDraw - - # Make a copy to avoid modifying original - image = image.copy() - draw = ImageDraw.Draw(image) - - for box, label, score in zip(boxes, labels, scores): - # box format: [x1, y1, x2, y2] - x1, y1, x2, y2 = box - - # Draw rectangle - draw.rectangle([x1, y1, x2, y2], outline="red", width=3) - - # Draw label - text = f"{label}: {score:.2f}" - draw.text((x1, y1 - 10), text, fill="red") - - return image + return (result,) diff --git a/requirements.txt b/requirements.txt index 6092e2b..d64ce2f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,15 +1,14 @@ # Core dependencies for ComfyUI-Grounding -# For GroundingDINO (HuggingFace transformers implementation) +# For GroundingDINO, MM-GroundingDINO, and OWLv2 (HuggingFace transformers) transformers>=4.30.0 timm>=0.9.0 # For YOLO-World ultralytics>=8.2.0 -# Image processing -opencv-python>=4.8.0 -pillow>=10.0.0 +# Image processing (usually already installed with ComfyUI) +# pillow>=10.0.0 # Optional: supervision for advanced visualization # supervision>=0.16.0 diff --git a/workflows/Example.json b/workflows/Example.json new file mode 100644 index 0000000..2aea663 --- /dev/null +++ b/workflows/Example.json @@ -0,0 +1,353 @@ +{ + "id": "b29971c7-6074-4707-895e-9f2360e11e2d", + "revision": 0, + "last_node_id": 30, + "last_link_id": 40, + "nodes": [ + { + "id": 29, + "type": "LoadImage", + "pos": [ + 691.3480657963805, + 244.71705551823334 + ], + "size": [ + 274.080078125, + 314 + ], + "flags": {}, + "order": 0, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "IMAGE", + "type": "IMAGE", + "links": [ + 38, + 40 + ] + }, + { + "name": "MASK", + "type": "MASK", + "links": null + } + ], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.65", + "Node name for S&R": "LoadImage" + }, + "widgets_values": [ + "example.png", + "image" + ] + }, + { + "id": 28, + "type": "MaskPreview", + "pos": [ + 1252.4602357504743, + 329.44736978995775 + ], + "size": [ + 140, + 246.00000000000003 + ], + "flags": {}, + "order": 5, + "mode": 0, + "inputs": [ + { + "name": "mask", + "type": "MASK", + "link": 37 + } + ], + "outputs": [], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.65", + "Node name for S&R": "MaskPreview" + }, + "widgets_values": [] + }, + { + "id": 22, + "type": "PreviewImage", + "pos": [ + 1937.3573937178255, + 228.10370816997101 + ], + "size": [ + 140, + 246.00000000000003 + ], + "flags": {}, + "order": 6, + "mode": 0, + "inputs": [ + { + "name": "images", + "type": "IMAGE", + "link": 32 + } + ], + "outputs": [], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.65", + "Node name for S&R": "PreviewImage" + }, + "widgets_values": [] + }, + { + "id": 30, + "type": "PreviewImage", + "pos": [ + 1664.5457193908678, + -114.32244507189236 + ], + "size": [ + 140, + 246 + ], + "flags": {}, + "order": 4, + "mode": 0, + "inputs": [ + { + "name": "images", + "type": "IMAGE", + "link": 39 + } + ], + "outputs": [], + "properties": { + "cnr_id": "comfy-core", + "ver": "0.3.65", + "Node name for S&R": "PreviewImage" + }, + "widgets_values": [] + }, + { + "id": 21, + "type": "BboxVisualizer", + "pos": [ + 1608.8053963516986, + 258.9104840919287 + ], + "size": [ + 210, + 78 + ], + "flags": {}, + "order": 3, + "mode": 0, + "inputs": [ + { + "name": "image", + "type": "IMAGE", + "link": 40 + }, + { + "name": "bboxes", + "type": "BBOX", + "link": 31 + } + ], + "outputs": [ + { + "name": "annotated_image", + "type": "IMAGE", + "links": [ + 32 + ] + } + ], + "properties": { + "aux_id": "PozzettiAndrea/ComfyUI-Grounding", + "ver": "c9c72642cd7affb18d5aa54f21ee7d3acf7412eb", + "Node name for S&R": "BboxVisualizer" + }, + "widgets_values": [ + 3 + ] + }, + { + "id": 18, + "type": "GroundingModelLoader", + "pos": [ + 761.1567094157901, + 37.11377863503034 + ], + "size": [ + 272.064453125, + 106 + ], + "flags": {}, + "order": 1, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "model", + "type": "GROUNDING_MODEL", + "links": [ + 27 + ] + } + ], + "properties": { + "aux_id": "PozzettiAndrea/ComfyUI-Grounding", + "ver": "c9c72642cd7affb18d5aa54f21ee7d3acf7412eb", + "Node name for S&R": "GroundingModelLoader" + }, + "widgets_values": [ + "GroundingDINO: SwinB (938MB)", + true, + "eager" + ] + }, + { + "id": 19, + "type": "GroundingDetector", + "pos": [ + 1147.1057310247927, + -143.3627616665786 + ], + "size": [ + 400, + 268 + ], + "flags": {}, + "order": 2, + "mode": 0, + "inputs": [ + { + "name": "model", + "type": "GROUNDING_MODEL", + "link": 27 + }, + { + "name": "image", + "type": "IMAGE", + "link": 38 + } + ], + "outputs": [ + { + "name": "bboxes", + "type": "BBOX", + "links": [ + 31 + ] + }, + { + "name": "annotated_image", + "type": "IMAGE", + "links": [ + 39 + ] + }, + { + "name": "labels", + "type": "STRING", + "links": null + }, + { + "name": "masks", + "type": "MASK", + "links": [ + 37 + ] + } + ], + "properties": { + "aux_id": "PozzettiAndrea/ComfyUI-Grounding", + "ver": "c9c72642cd7affb18d5aa54f21ee7d3acf7412eb", + "Node name for S&R": "GroundingDetector" + }, + "widgets_values": [ + "character", + 0.1, + 0.01, + true, + "dict_with_data", + true + ] + } + ], + "links": [ + [ + 27, + 18, + 0, + 19, + 0, + "GROUNDING_MODEL" + ], + [ + 31, + 19, + 0, + 21, + 1, + "BBOX" + ], + [ + 32, + 21, + 0, + 22, + 0, + "IMAGE" + ], + [ + 37, + 19, + 3, + 28, + 0, + "MASK" + ], + [ + 38, + 29, + 0, + 19, + 1, + "IMAGE" + ], + [ + 39, + 19, + 1, + 30, + 0, + "IMAGE" + ], + [ + 40, + 29, + 0, + 21, + 0, + "IMAGE" + ] + ], + "groups": [], + "config": {}, + "extra": { + "ds": { + "scale": 0.8390550503631276, + "offset": [ + -372.4858306734141, + 284.24946587737446 + ] + }, + "frontendVersion": "1.28.7" + }, + "version": 0.4 +} \ No newline at end of file