62 lines
4.1 KiB
Markdown
62 lines
4.1 KiB
Markdown
# ComfyUI-layerdiffuse
|
|
ComfyUI implementation of https://github.com/layerdiffusion/LayerDiffuse.
|
|
|
|
## Installation
|
|
Download the repository and unpack into the custom_nodes folder in the ComfyUI installation directory.
|
|
|
|
Or clone via GIT, starting from ComfyUI installation directory:
|
|
```bash
|
|
cd custom_nodes
|
|
git clone git@github.com:huchenlei/ComfyUI-layerdiffuse.git
|
|
```
|
|
|
|
Run `pip install -r requirements.txt` to install python dependencies. You might experience version conflict on diffusers if you have other extensions
|
|
that depends on other versions of diffusers. In this case, it is recommended to setup separate Python venvs.
|
|
|
|
## Workflows
|
|
### [Generate foreground](https://github.com/huchenlei/ComfyUI-layerdiffuse/blob/main/examples/layer_diffusion_fg_example_rgba.json)
|
|
|
|
|
|
### [Generate foreground (RGB + alpha)](https://github.com/huchenlei/ComfyUI-layerdiffuse/blob/main/examples/layer_diffusion_fg_example.json)
|
|
If you want more control of getting RGB image and alpha channel mask separately, you can use this workflow.
|
|
|
|
|
|
### [Blending (FG/BG)](https://github.com/huchenlei/ComfyUI-layerdiffuse/blob/main/examples/layer_diffusion_cond_example.json)
|
|
Blending given FG
|
|
|
|
|
|
Blending given BG
|
|
|
|
|
|
### [Extract FG from Blended + BG](https://github.com/huchenlei/ComfyUI-layerdiffuse/blob/main/examples/layer_diffusion_diff_fg.json)
|
|
|
|
|
|
### [Extract BG from Blended + FG](https://github.com/huchenlei/ComfyUI-layerdiffuse/blob/main/examples/layer_diffusion_diff_bg.json)
|
|
[Forge impl's sanity check](https://github.com/layerdiffuse/sd-forge-layerdiffuse#sanity-check) sets `Stop at` to 0.5 to get better quality BG.
|
|
This workflow might be inferior comparing to other object removal workflows.
|
|
|
|
|
|
### [Extract BG from Blended + FG (Stop at 0.5)](https://github.com/huchenlei/ComfyUI-layerdiffuse/blob/main/examples/layer_diffusion_diff_bg_stop_at.json)
|
|
In [SD Forge impl](https://github.com/layerdiffuse/sd-forge-layerdiffuse), there is a `stop at` param that determines when
|
|
layer diffuse should stop in the denosing process. In the background, what this param does is unapply the LoRA and c_concat cond after a certain step
|
|
threshold. This is hard/risky to implement directly in ComfyUI as it requires manually load a model that has every changes except the layer diffusion
|
|
change applied. A workaround in ComfyUI is to have another img2img pass on the layer diffuse result to simulate the effect of `stop at` param.
|
|
|
|
|
|
|
|
### [Generate FG from BG combined](https://github.com/huchenlei/ComfyUI-layerdiffuse/blob/main/examples/layer_diffusion_cond_fg_all.json)
|
|
Combines previous workflows to generate blended and FG given BG. We found that there are some color variations in the extracted FG. Need to confirm
|
|
with layer diffusion authors on whether this is expected.
|
|
|
|
|
|
## Note
|
|
- Currently only SDXL is supported. See https://github.com/layerdiffuse/sd-forge-layerdiffuse#model-notes for more details.
|
|
- To decode RGBA result, the generation dimension must be multiple of 64. Otherwise, you will get decode error: 
|
|
|
|
|
|
## TODO
|
|
- [x] Foreground conditioning
|
|
- [x] Background conditioning
|
|
- [x] Blended + foreground => background
|
|
- [x] Blended + background => foreground
|