nodes/lum3on-comfyui_EdgeTAM

Fork 0

Files

T

Benjamin Bertram f35280a1b8 Add files via upload

2025-06-16 12:36:46 +02:00

11 KiB

Raw Permalink Blame History

EdgeTAM ComfyUI Usage Guide

This guide provides detailed instructions for using EdgeTAM nodes in ComfyUI workflows.

Quick Start

1. Installation

First, install EdgeTAM and its dependencies:

# Option 1: Use the installation script
python install_edgetam.py

# Option 2: Manual installation
git clone https://github.com/facebookresearch/EdgeTAM.git
cd EdgeTAM
pip install -e .

2. Test Installation

Verify everything is working:

python test_edgetam.py

Node Reference

EdgeTAM Video Tracker

Tracks objects across video frames using point or box prompts.

Required Inputs:

video_frames (IMAGE): Batch of video frames

Optional Inputs:

prompt_points (STRING): JSON array of [x, y] coordinates
prompt_labels (STRING): JSON array of labels (1=include, 0=exclude)
click_x (INT): X coordinate for single click point
click_y (INT): Y coordinate for single click point
click_label (COMBO): Label for single click point (include/exclude)
use_click_mode (BOOLEAN): Use single click coordinates instead of JSON
model_path (STRING): Custom model path
device (COMBO): Computation device (auto/cuda/cpu/mps)
max_frames (INT): Maximum frames to process
overlay_masks (BOOLEAN): Show colored overlays

Outputs:

tracked_frames (IMAGE): Original video frames
masks (MASK): Segmentation masks for each frame
overlay_frames (IMAGE): Frames with colored mask overlays

Click Mode Example:

use_click_mode: True
click_x: 320
click_y: 240
click_label: include

JSON Mode Example:

{
  "prompt_points": "[[320, 240], [400, 300]]",
  "prompt_labels": "[1, 1]",
  "use_click_mode": false
}

EdgeTAM Image Segmentor

Performs single-frame segmentation with SAM-like capabilities.

Required Inputs:

image (IMAGE): Input image

Optional Inputs:

prompt_points (STRING): JSON array of [x, y] coordinates
prompt_labels (STRING): JSON array of labels (1=include, 0=exclude)
click_x (INT): X coordinate for single click point
click_y (INT): Y coordinate for single click point
click_label (COMBO): Label for single click point (include/exclude)
use_click_mode (BOOLEAN): Use single click coordinates instead of JSON
model_path (STRING): Custom model path
device (COMBO): Computation device (auto/cuda/cpu/mps)
multimask_output (BOOLEAN): Return multiple mask predictions

Outputs:

mask (MASK): Segmentation mask
overlay_image (IMAGE): Image with colored mask overlay

Click Mode Example:

use_click_mode: True
click_x: 150
click_y: 200
click_label: include

JSON Mode Example:

{
  "prompt_points": "[[150, 200], [300, 250]]",
  "prompt_labels": "[1, 0]",
  "use_click_mode": false
}

Workflow Examples

Basic Video Tracking

Load Video: Use LoadVideo node to load your video file
Add Tracker: Connect video to EdgeTAMVideoTracker
Set Points: Add click coordinates in prompt_points
Set Labels: Add corresponding labels in prompt_labels
Preview/Save: Connect outputs to preview or save nodes

Multi-Object Tracking

For tracking multiple objects, use multiple tracker nodes:

{
  "object1_points": "[[100, 100]]",
  "object1_labels": "[1]",
  "object2_points": "[[300, 200]]", 
  "object2_labels": "[1]"
}

Image Segmentation Pipeline

Load Image: Use LoadImage node
Segment: Connect to EdgeTAMImageSegmentor
Refine: Use multiple points for better segmentation
Post-process: Apply mask operations as needed

Tips and Best Practices

Point Selection

Include points (label=1): Click inside the object you want to track
Exclude points (label=0): Click on areas you want to exclude
Multiple points: Use several points for complex objects
Refinement: Add exclude points to remove unwanted regions

Performance Optimization

Device selection: Use "cuda" for GPU acceleration
Frame limits: Set max_frames for long videos
Batch processing: Process videos in chunks for memory efficiency

Common Issues

Model Download Fails

# Manual download
wget https://github.com/facebookresearch/EdgeTAM/releases/download/v1.0/edgetam.pt
# Place in: models/edgetam.pt

Memory Issues

Reduce video resolution
Process fewer frames at once
Use CPU if GPU memory is limited

Poor Tracking Quality

Add more include points on the object
Add exclude points on background
Ensure first frame has clear object visibility

Advanced Usage

Custom Model Paths

You can use custom trained models:

{
  "model_path": "/path/to/custom/edgetam_model.pt"
}

Coordinate Systems

Points are in image pixel coordinates:

Origin (0,0) is top-left corner
X increases rightward
Y increases downward

JSON Format Examples

Single point:

{
  "prompt_points": "[[320, 240]]",
  "prompt_labels": "[1]"
}

Multiple points:

{
  "prompt_points": "[[100, 100], [200, 150], [300, 200]]",
  "prompt_labels": "[1, 1, 0]"
}

Complex object:

{
  "prompt_points": "[[150, 100], [200, 120], [180, 80], [250, 150]]",
  "prompt_labels": "[1, 1, 1, 0]"
}

Integration with Other Nodes

Video Processing Pipeline

LoadVideo → EdgeTAMVideoTracker → VideoProcessor → SaveVideo

Mask-based Workflows

EdgeTAMImageSegmentor → MaskProcessor → ImageComposite

Batch Processing

ImageBatch → EdgeTAMImageSegmentor → MaskBatch → BatchProcessor

🎯 Visual Point Editor

EdgeTAM now includes an advanced visual point editor that makes point selection intuitive and precise:

✨ Features:

🖼️ Interactive Canvas: Visual image display with real-time point placement
🎨 Color-coded Points: Green for include points, red for exclude points
🔢 Point Numbering: Clear numbering system for point order
📝 Live Coordinate Display: Real-time coordinate feedback
🗑️ Individual Point Deletion: Remove specific points with one click
🧹 Bulk Operations: Clear all points or auto-generate labels
📏 Automatic Scaling: Handles different image resolutions automatically

🚀 How to Use:

Add EdgeTAM node to your workflow
Connect image/video source to the node
Scroll down to find the EdgeTAM Point Editor below the text inputs
Click "Load Image" to display your image in the editor
Select mode: Choose "Include" or "Exclude" point type
Click on the image to add points where you want them
Review points in the point list below the image
Delete individual points using the × button if needed
Points automatically sync to the JSON text fields above

🎮 Editor Controls:

Include/Exclude Buttons: Switch between point types
Clear All: Remove all points and start over
Load Image: Refresh the image display
Point List: Shows all points with coordinates and types
Status Bar: Real-time feedback and coordinate display

💡 Pro Tips:

Automatic Label Sync: The editor automatically updates both points and labels
Precise Coordinates: Hover over the image to see exact pixel coordinates
Multiple Objects: Use include points for objects, exclude points for background
Fine-tuning: Add exclude points to remove unwanted areas from segmentation
Visual Feedback: Green points = include, red points = exclude

🔧 Troubleshooting:

No image displayed: Connect an image source and ensure it's properly loaded
Points not appearing: Check that the image has loaded successfully
Coordinates seem wrong: The editor automatically handles scaling - coordinates are in original image space
Editor not visible: Scroll down below the text input fields to find the point editor

🎯 Professional Points Editor Node

EdgeTAM now includes a professional-grade Points Editor node based on KJNodes architecture but optimized for EdgeTAM workflows:

✨ Key Features:

🖼️ Interactive Canvas: Visual point placement with real-time feedback
⌨️ Keyboard Shortcuts: Professional hotkey system for efficient editing
🎨 Color-Coded Points: Green for positive (include), red for negative (exclude)
�️ Intuitive Controls: Shift+Click to add, Right-Click to delete
� Precise Coordinates: Pixel-perfect point placement
🔄 Format Conversion: Automatic conversion to EdgeTAM format

🚀 Available Nodes:

🎯 EdgeTAM Points Editor

Inputs: Canvas dimensions, background image (optional)
Outputs: Positive coordinates, negative coordinates, preview image
Features: KJNodes-style interactive point editor

� EdgeTAM Points Converter

Inputs: Positive and negative coordinates from editor
Outputs: Points JSON and Labels JSON for EdgeTAM nodes
Use: Convert editor output to EdgeTAM tracker/segmentor format

⌨️ Controls:

Shift + Left Click: Add positive (green) point
Shift + Right Click: Add negative (red) point
Right Click on Point: Delete point
Drag Image: Load background image to canvas
Copy/Paste: Load image via clipboard

🔧 Example Workflows:

Basic Point Selection:

LoadImage → EdgeTAM Points Editor → EdgeTAM Points Converter → EdgeTAM Video Tracker

Advanced Multi-Point Tracking:

LoadImage → EdgeTAM Points Editor → EdgeTAM Points Converter → EdgeTAM Video Tracker
                ↓                           ↓
            Preview Image              Points + Labels

Professional Workflow:

LoadImage → EdgeTAM Points Editor (Interactive Editing)
                ↓
        EdgeTAM Points Converter (Format Conversion)
                ↓
        EdgeTAM Video Tracker (Object Tracking)
                ↓
        Preview Output (Results)

Troubleshooting

Installation Issues

Python version: Ensure Python ≥ 3.10
PyTorch version: Ensure PyTorch ≥ 2.3.1
CUDA compatibility: Match CUDA versions
Dependencies: Install all requirements

Runtime Issues

Import errors: Run python test_edgetam.py
Model loading: Check model path and permissions
Memory errors: Reduce batch size or use CPU
Performance: Enable GPU acceleration

Getting Help

Check the test script output
Review ComfyUI console logs
Verify input formats (JSON syntax)
Test with simple examples first

Performance Benchmarks

Typical performance on different hardware:

RTX 4090: ~30 FPS (1080p video)
RTX 3080: ~20 FPS (1080p video)
Apple M2 Max: ~15 FPS (1080p video)
CPU only: ~2-5 FPS (1080p video)

Results may vary based on video complexity and tracking difficulty.

11 KiB Raw Permalink Blame History Unescape Escape