ComfyUI_SemanticSAM

This is a ComfyUI node based-on Semantic-SAM official implementation. We provide a workflow node for one-click segment. Compared with SAM, Semantic-SAM has better fine-grained capabilities and more candidate masks. Users can take this node as the pre-node for inpainting to obtain the mask region.

This node have been valided on Ubuntu-20.04 & CUDA-11.8. Due to detectron2, this node is currently not supported on Windows (unless the compatibility of Detectron2 on Windows is addressed)

Get Start

1. Prepare repo

cd ComfyUI/custom
git clone https://github.com/eastoc/ComfyUI_SemanticSAM
cd ComfyUI_SemanticSAM
git clone https://github.com/facebookresearch/detectron2
git clone https://github.com/facebookresearch/Mask2Former

Because detectron2 has not been updated, if the CUDA=11.8, you can

git clone https://github.com/johnnynunez/detectron2

or you can try other 3rd detectron2 implementation.

2. Download models

download Semantic-SAM model to "ComfyUI_SemanticSAM/ckpt"

Name	Training Dataset	Backbone	1-IoU@Multi-Granularity	1-IoU@COCO(Max|Oracle)	download
Semantic-SAM | config	SA-1B	SwinT	88.1	54.5|73.8	model
Semantic-SAM | config	SA-1B	SwinL	89.0	55.1|74.1	model

3. Install environment

Install PyTorch & torchvision through the official method. We have been valid that from pytorch 1.13.0 to 2.3.0.

Install Semantic-SAM dependencies

pip install -r requirements.txt

Install detectron2

cd detectron2
pip install -e .
cd ..

Install Mask2Former

cd Mask2Former/mask2former/modeling/pixel_decoder/ops
sh make.sh

Workflow

one-click segment workflow is in "./workflow/workflow.json"