53 lines
2.0 KiB
Markdown
53 lines
2.0 KiB
Markdown
## ComfyUI-BrushNet
|
|
|
|
Custom nodes for ComfyUI allow to inpaint using Brushnet: ["BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion"](https://arxiv.org/abs/2403.06976).
|
|
|
|
My contribution is limited to the ComfyUI adaptation, and all credit goes to the authors of the paper.
|
|
|
|
|
|
## Installation
|
|
|
|
Clone the repo into the custom_nodes directory and install the requirements:
|
|
|
|
```
|
|
git clone https://github.com/nullquant/ComfyUI-BrushNet.git
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
Checkpoints of BrushNet can be downloaded from [here](https://drive.google.com/drive/folders/1fqmS1CEOvXCxNWFrsSYd_jHYXxrydh1n?usp=drive_link).
|
|
|
|
The checkpoint in `segmentation_mask_brushnet_ckpt` provides checkpoints trained on BrushData, which has segmentation prior (mask are with the same shape of objects). The `random_mask_brushnet_ckpt` provides a more general ckpt for random mask shape.
|
|
|
|
Both `segmentation_mask_brushnet_ckpt` and `random_mask_brushnet_ckpt` or only one of them should be placed inside `models/inpaint/brushnet` folder.
|
|
|
|
Base model should be SD1.5 type (realisticVisionV60B1_v51VAE, for example).
|
|
|
|
This is complete rework of previous version. There is no more custom `diffusers` library.
|
|
There is no loadings from github. I also changed the workflow so now it is possible to use LoRa models.
|
|
To users who already have previoous version: please, uninstall `diffusers` library before installation:
|
|
|
|
```
|
|
pip uninstall diffusers
|
|
```
|
|
|
|
Then install as usual.
|
|
|
|
## Usage
|
|
|
|
Below is an example for the intended workflow. The [json file](example/BrushNet_basic.json) for the example can be found inside the 'example' directory.
|
|
|
|
|
|
|
|
This is example with LoRA: [json file](example/BrushNet_with_LoRA.json).
|
|
|
|
|
|
|
|
|
|
## Credits
|
|
|
|
The code is based on
|
|
|
|
- [BrushNet](https://github.com/TencentARC/BrushNet)
|
|
- [PowerPaint](https://github.com/zhuang2002/PowerPaint)
|
|
- [diffusers](https://github.com/huggingface/diffusers)
|