nodes/nullquant-ComfyUI-BrushNet
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

readme

Browse Source
This commit is contained in:
nullquant
2024-05-19 21:20:06 +03:00
parent 582115862d
commit 4788f93153
3 changed files with 50 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
RAUNET.md
+5 -3
View File
@@ -10,13 +10,15 @@ The authors of HiDiffusion invent simple, yet efficient trick to alleviate this
[workflow](example/RAUNet_basic.json)
The left picture is created using ZavyChromaXL checkpoint on 2048x2048 canvas. The right one using RAUNet.
The left picture is created using ZavyChromaXL checkpoint on 2048x2048 canvas. The right one uses RAUNet.
In my experience the node is helpful but quite sensitive to its parameters. And there is no universal solution - you should adjust them for every new image you generate. It also lowers model's imagination, you usually get only what you described in prompt. Look at the example: in first you have a forest in the background, but RAUNet deleted all except fox which is described in the prompt.
In my experience the node is helpful but quite sensitive to its parameters. And there is no universal solution - you should adjust them for every new image you generate. It also lowers model's imagination, you usually get only what you described in the prompt. Look at the example: in first you have a forest in the background, but RAUNet deleted all except fox which is described in the prompt.
From the [paper](https://arxiv.org/abs/2311.17528): Diffusion models denoise from structures to details. RAU-Net introduces additional downsampling and upsampling operations, leading to a certain degree of information loss. In the early stages of denoising, RAU-Net can generate reasonable structures with minimal impact from information loss. However, in the later stages of denoising when generating fine details, the information loss in RAU-Net results in the loss of image details and a degradation in quality.
### Parameters
There are two independent parts in this node: DU (Downsample/Upsample) and XA (CrossAttension). The four parameters are the start and end steps for applying these parts.
There are two independent parts in this node: DU (Downsample/Upsample) and XA (CrossAttention). The four parameters are the start and end steps for applying these parts.
The Downsample/Upsample part lowers models degrees of freedom. If you apply it a lot (for more steps) the resulting images will have a lot of symmetries.
README.md
+42 -35
View File
@@ -31,7 +31,6 @@ Apr 11, 2024. Initial commit.
- [x] BrushNet SDXL
- [x] PowerPaint v2
- [x] Image batch
- [ ] Compatibility with `jank HiDiffusion` and similar nodes
## Installation
@@ -67,40 +66,6 @@ Below is an example for the intended workflow. The [workflow](example/BrushNet_b
![example workflow](example/BrushNet_basic.png?raw=true)
### Parameters
#### Brushnet Loader
- `dtype`, defaults to `torch.float16`. The torch.dtype of BrushNet. If you have old GPU or NVIDIA 16 series card try to switch to `torch.float32`.
#### Brushnet
- `scale`, defaults to 1.0: The "strength" of BrushNet. The outputs of the BrushNet are multiplied by `scale` before they are added to the residual in the original unet.
- `start_at`, defaults to 0: step at which the BrushNet starts applying.
- `end_at`, defaults to 10000: step at which the BrushNet stops applying.
[Here](PARAMS.md) are examples of use these two last parameters.
#### PowerPaint
- `CLIP`: PowerPaint CLIP that should be passed from PowerPaintCLIPLoader node.
- `fitting`: PowerPaint fitting degree.
- `function`: PowerPaint function, see its [page](https://github.com/open-mmlab/PowerPaint) for details.
When using certain network functions, the authors of PowerPaint recommend adding phrases to the prompt:
- object removal: `empty scene blur`
- context aware: `empty scene`
- outpainting: `empty scene`
Many of ComfyUI users use custom text generation nodes, CLIP nodes and a lot of other conditioning. I don't want to break all of these nodes, so I didn't add prompt updating and instead rely on users. Also my own experiments show that these additions to prompt are not strictly necessary.
The latent image can be from BrushNet node or not, but it should be the same size as original image (divided by 8 in latent space).
The both conditioning `positive` and `negative` in BrushNet and PowerPaint nodes are used for calculation inside, but then simply copied to output.
Be advised, not all workflows and nodes will work with BrushNet due to its structure. Also put model changes before BrushNet nodes, not after. If you need model to work with image after BrushNet inference use base one (see Upscale example below).
<details>
<summary>SDXL</summary>
@@ -217,6 +182,48 @@ You should try to add object description to negative prompt and describe empty s
</details>
### Parameters
#### Brushnet Loader
- `dtype`, defaults to `torch.float16`. The torch.dtype of BrushNet. If you have old GPU or NVIDIA 16 series card try to switch to `torch.float32`.
#### Brushnet
- `scale`, defaults to 1.0: The "strength" of BrushNet. The outputs of the BrushNet are multiplied by `scale` before they are added to the residual in the original unet.
- `start_at`, defaults to 0: step at which the BrushNet starts applying.
- `end_at`, defaults to 10000: step at which the BrushNet stops applying.
[Here](PARAMS.md) are examples of use these two last parameters.
#### PowerPaint
- `CLIP`: PowerPaint CLIP that should be passed from PowerPaintCLIPLoader node.
- `fitting`: PowerPaint fitting degree.
- `function`: PowerPaint function, see its [page](https://github.com/open-mmlab/PowerPaint) for details.
When using certain network functions, the authors of PowerPaint recommend adding phrases to the prompt:
- object removal: `empty scene blur`
- context aware: `empty scene`
- outpainting: `empty scene`
Many of ComfyUI users use custom text generation nodes, CLIP nodes and a lot of other conditioning. I don't want to break all of these nodes, so I didn't add prompt updating and instead rely on users. Also my own experiments show that these additions to prompt are not strictly necessary.
The latent image can be from BrushNet node or not, but it should be the same size as original image (divided by 8 in latent space).
The both conditioning `positive` and `negative` in BrushNet and PowerPaint nodes are used for calculation inside, but then simply copied to output.
Be advised, not all workflows and nodes will work with BrushNet due to its structure. Also put model changes before BrushNet nodes, not after. If you need model to work with image after BrushNet inference use base one (see Upscale example below).
#### RAUNet
- `du_start`, defaults to 0: step at which the Downsample/Upsample resize starts applying.
- `du_end`, defaults to 4: step at which the Downsample/Upsample resize stops applying.
- `xa_start`, defaults to 4: step at which the CrossAttention resize starts applying.
- `xa_end`, defaults to 10: step at which the CrossAttention resize stops applying.
For an examples and explanation, please look [here](RAUNET.md).
## Limitations
raunet_nodes.py
+3 -3
View File
@@ -13,9 +13,9 @@ class RAUNet:
{
"model": ("MODEL",),
"du_start": ("INT", {"default": 0, "min": 0, "max": 10000}),
"du_end": ("INT", {"default": 8, "min": 0, "max": 10000}),
"xa_start": ("INT", {"default": 8, "min": 0, "max": 10000}),
"xa_end": ("INT", {"default": 20, "min": 0, "max": 10000}),
"du_end": ("INT", {"default": 4, "min": 0, "max": 10000}),
"xa_start": ("INT", {"default": 4, "min": 0, "max": 10000}),
"xa_end": ("INT", {"default": 10, "min": 0, "max": 10000}),
},
}