diff --git a/CN.md b/CN.md new file mode 100644 index 0000000..88263a2 --- /dev/null +++ b/CN.md @@ -0,0 +1,39 @@ +## ControlNet Canny Edge + +Let's take the pestered cake and try to inpaint it again. Now I would like to use a sleeping cat for it: + +![sleeping cat](example/sleeping_cat.png?raw=true) + +I use Canny Edge node from [comfyui_controlnet_aux](https://github.com/Fannovel16/comfyui_controlnet_aux). Don't forget to resize canny edge mask to 512 pixels: + +![sleeping cat inpaint](example/sleeping_cat_inpaint1.png?raw=true) + +Let's look at the result: + +![sleeping cat inpaint](example/sleeping_cat_inpaint2.png?raw=true) + +The first problem I see here is some kind of object behind the cat. Such objects appear since the inpainting mask strictly aligns with the removed object, the cake in our case. To remove such artifact we should expand our mask a little: + +![sleeping cat inpaint](example/sleeping_cat_inpaint3.png?raw=true) + +Now. what's up with cat back and tail? Let's see the inpainting mask and canny edge mask side to side: + +![masks](example/sleeping_cat_inpaint4.png?raw=true) + +The inpainting works (mostly) only in masked (white) area, so we cut off cat's back. **The ControlNet mask should be inside the inpaint mask.** + +To address the issue I resized the mask to 256 pixels: + +![sleeping cat inpaint](example/sleeping_cat_inpaint5.png?raw=true) + +This is better but still have a room for improvement. The problem with edge mask downsampling is that edge lines tend to be broken and after some size we will got a mess: + +![sleeping cat inpaint](example/sleeping_cat_inpaint6.png?raw=true) + +Look at the edge mask, at this resolution it is so broken: + +![masks](example/sleeping_cat_mask.png?raw=true) + + + + diff --git a/README.md b/README.md index a3182ea..fa3737b 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,8 @@ My contribution is limited to the ComfyUI adaptation, and all credit goes to the ## Updates +May 9, 2024. Added [ControlNet canny edge](CN.md) examples. + May 6, 2024. PowerPaint v2 model is implemented. After update your workflow probably will not work. Don't panic! Check `end_at` parameter of BrushNode, if it equals 1, change it to some big number. Read about parameters in Usage section below. May 2, 2024. BrushNet SDXL is live. It needs positive and negative conditioning though, so workflow changes a little, see example. @@ -132,6 +134,8 @@ Sometimes inference and VAE broke image, so you need to blend inpaint image with [workflow](example/BrushNet_with_CN.json) +[ControlNet canny edge](CN.md) +
diff --git a/brushnet_nodes.py b/brushnet_nodes.py index 9b5dc82..d66826a 100644 --- a/brushnet_nodes.py +++ b/brushnet_nodes.py @@ -10,7 +10,7 @@ from sys import platform comfy_parent_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '../../')) sys.path.append(comfy_parent_dir) import comfy -#import nodes +import nodes #import latent_preview from accelerate import init_empty_weights, load_checkpoint_and_dispatch @@ -159,7 +159,7 @@ class PowerPaintCLIPLoader: pp_text_encoder.load_state_dict(torch.load(pp_CLIP_file), strict=False) - print('PowerPaint CLIP file: ', base_CLIP_file) + print('PowerPaint CLIP file: ', pp_CLIP_file) pp_clip.tokenizer.clip_l.tokenizer = pp_tokenizer pp_clip.patcher.model.clip_l.transformer = pp_text_encoder @@ -682,11 +682,7 @@ def brushnet_inference(x, timesteps, transformer_options): all_sigmas = mp['all_sigmas'] sigma = transformer_options['sigmas'][0].item() total_steps = all_sigmas.shape[0] - step = ((all_sigmas - sigma).abs() < 1e-3).nonzero(as_tuple=True)[0] - if len(step) == 0: - step = 0 - else: - step = (all_sigmas == sigma).nonzero(as_tuple=True)[0].item() + step = torch.argmin((all_sigmas - sigma).abs()).item() added_cond_kwargs = {} diff --git a/example/sleepeng_cat_ce.png b/example/sleepeng_cat_ce.png new file mode 100644 index 0000000..773520a Binary files /dev/null and b/example/sleepeng_cat_ce.png differ diff --git a/example/sleepeng_cat_ce.png~ b/example/sleepeng_cat_ce.png~ new file mode 100644 index 0000000..855b021 Binary files /dev/null and b/example/sleepeng_cat_ce.png~ differ diff --git a/example/sleeping_cat.png b/example/sleeping_cat.png new file mode 100644 index 0000000..555253c Binary files /dev/null and b/example/sleeping_cat.png differ diff --git a/example/sleeping_cat_inpaint1.png b/example/sleeping_cat_inpaint1.png new file mode 100644 index 0000000..f7774b9 Binary files /dev/null and b/example/sleeping_cat_inpaint1.png differ diff --git a/example/sleeping_cat_inpaint2.png b/example/sleeping_cat_inpaint2.png new file mode 100644 index 0000000..1e686c8 Binary files /dev/null and b/example/sleeping_cat_inpaint2.png differ diff --git a/example/sleeping_cat_inpaint3.png b/example/sleeping_cat_inpaint3.png new file mode 100644 index 0000000..5c8d697 Binary files /dev/null and b/example/sleeping_cat_inpaint3.png differ diff --git a/example/sleeping_cat_inpaint4.png b/example/sleeping_cat_inpaint4.png new file mode 100644 index 0000000..c2b8d47 Binary files /dev/null and b/example/sleeping_cat_inpaint4.png differ diff --git a/example/sleeping_cat_inpaint5.png b/example/sleeping_cat_inpaint5.png new file mode 100644 index 0000000..5c52f17 Binary files /dev/null and b/example/sleeping_cat_inpaint5.png differ diff --git a/example/sleeping_cat_inpaint6.png b/example/sleeping_cat_inpaint6.png new file mode 100644 index 0000000..89782ed Binary files /dev/null and b/example/sleeping_cat_inpaint6.png differ diff --git a/example/sleeping_cat_mask.png b/example/sleeping_cat_mask.png new file mode 100644 index 0000000..f1ba4d4 Binary files /dev/null and b/example/sleeping_cat_mask.png differ