ControlNet canny edge example

CN.md

@@ -0,0 +1,39 @@
+## ControlNet Canny Edge
+
+Let's take the pestered cake and try to inpaint it again. Now I would like to use a sleeping cat for it:
+
+![sleeping cat](example/sleeping_cat.png?raw=true)
+
+I use Canny Edge node from [comfyui_controlnet_aux](https://github.com/Fannovel16/comfyui_controlnet_aux). Don't forget to resize canny edge mask to 512 pixels:
+
+![sleeping cat inpaint](example/sleeping_cat_inpaint1.png?raw=true)
+
+Let's look at the result:
+
+![sleeping cat inpaint](example/sleeping_cat_inpaint2.png?raw=true)
+
+The first problem I see here is some kind of object behind the cat. Such objects appear since the inpainting mask strictly aligns with the removed object, the cake in our case. To remove such artifact we should expand our mask a little:
+
+![sleeping cat inpaint](example/sleeping_cat_inpaint3.png?raw=true)
+
+Now. what's up with cat back and tail? Let's see the inpainting mask and canny edge mask side to side:
+
+![masks](example/sleeping_cat_inpaint4.png?raw=true)
+
+The inpainting works (mostly) only in masked (white) area, so we cut off cat's back. **The ControlNet mask should be inside the inpaint mask.**
+
+To address the issue I resized the mask to 256 pixels:
+
+![sleeping cat inpaint](example/sleeping_cat_inpaint5.png?raw=true)
+
+This is better but still have a room for improvement. The problem with edge mask downsampling is that edge lines tend to be broken and after some size we will got a mess:
+
+![sleeping cat inpaint](example/sleeping_cat_inpaint6.png?raw=true)
+
+Look at the edge mask, at this resolution it is so broken:
+
+![masks](example/sleeping_cat_mask.png?raw=true)
+
+
+
+

README.md

@@ -6,6 +6,8 @@ My contribution is limited to the ComfyUI adaptation, and all credit goes to the
 
 ## Updates
 
+May 9, 2024. Added [ControlNet canny edge](CN.md) examples.
+
 May 6, 2024. PowerPaint v2 model is implemented. After update your workflow probably will not work. Don't panic! Check `end_at` parameter of BrushNode, if it equals 1, change it to some big number. Read about parameters in Usage section below.
 
 May 2, 2024. BrushNet SDXL is live. It needs positive and negative conditioning though, so workflow changes a little, see example.
@@ -132,6 +134,8 @@ Sometimes inference and VAE broke image, so you need to blend inpaint image with
 
 [workflow](example/BrushNet_with_CN.json)
 
+[ControlNet canny edge](CN.md)
+
 </details>
 
 <details>

brushnet_nodes.py

@@ -10,7 +10,7 @@ from sys import platform
 comfy_parent_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '../../'))
 sys.path.append(comfy_parent_dir)
 import comfy
-#import nodes
+import nodes
 #import latent_preview
 
 from accelerate import init_empty_weights, load_checkpoint_and_dispatch
@@ -159,7 +159,7 @@ class PowerPaintCLIPLoader:
 
         pp_text_encoder.load_state_dict(torch.load(pp_CLIP_file), strict=False)
 
-        print('PowerPaint CLIP file: ', base_CLIP_file)
+        print('PowerPaint CLIP file: ', pp_CLIP_file)
 
         pp_clip.tokenizer.clip_l.tokenizer = pp_tokenizer
         pp_clip.patcher.model.clip_l.transformer = pp_text_encoder
@@ -682,11 +682,7 @@ def brushnet_inference(x, timesteps, transformer_options):
         all_sigmas = mp['all_sigmas']
         sigma = transformer_options['sigmas'][0].item()
         total_steps = all_sigmas.shape[0]
-        step = ((all_sigmas - sigma).abs() < 1e-3).nonzero(as_tuple=True)[0]
-        if len(step) == 0:
-            step = 0
-        else:
-            step = (all_sigmas == sigma).nonzero(as_tuple=True)[0].item()
+        step = torch.argmin((all_sigmas - sigma).abs()).item()
 
         added_cond_kwargs = {}