update readme and pictures
This commit is contained in:
scraed
Binary file not shown.
|
Before Width: | Height: | Size: 70 KiB After Width: | Height: | Size: 119 KiB |
@@ -1,26 +1,31 @@
|
||||
# LanPaint (Thinking mode Inpaint)
|
||||
|
||||
Unlock precise inpainting without additional training. LanPaint lets the model "think" through multiple iterations before denoising, aiming for seamless and accurate results. This is the official implementation of "Lanpaint: Training-Free Diffusion Inpainting with Exact and Fast Conditional Inference".
|
||||
Unlock precise inpainting without additional training. LanPaint lets the model "think" through multiple iterations before denoising, aiming for seamless and accurate results.
|
||||
|
||||
|
||||
We encourage you to try it out and share your feedback through issues or discussions, as your input will help us enhance the algorithm's performance and stability.
|
||||
|
||||
## Features
|
||||
|
||||
- 🎨 **Zero-Training Inpainting** - Works immediately with ANY SD model (with/without ControlNet), and Flux model! even custom models you've trained yourself
|
||||
- 🛠️ **Simple Integration** - Same workflow as standard ComfyUI KSampler
|
||||
- 🎯 **True Blank-Slate Generation** - No need to set default denoise at 0.7 (preserving 30% original pixels in masks) used in conventional methods: 100% **new content creation**, No "painting over" existing content.
|
||||
- 🌈 **Not only inpaint**: You can even use it as a simple way to generate consistent characters.
|
||||
- **The Most Adaptive Method** - Works immediately with almost ANY model (with/without ControlNet)! SD 1.5, XL, 3.5, Flux, HiDream, even custom models you've trained yourself.
|
||||
- **Zero Training** - No need to train anything, it just works on your existing model.
|
||||
- **Simple Integration** - Same workflow as standard ComfyUI KSampler
|
||||
- **Mask independent** - Inpaint, outpaint, etc. Works for mask of any shape, size, and position.
|
||||
- **No Workarounds** - Based on the theory of diffusion, generate seamless and consistent results without smoothing on masks or latents. No need to set default denoise at 0.7 (preserving 30% original pixels in masks) used in conventional methods: 100% **new content creation**, No "painting over" existing content.
|
||||
- **Not only inpaint**: You can even use it as a simple way to generate consistent characters.
|
||||
|
||||
## How It Works
|
||||
This is the official implementation of "Lanpaint: Training-Free Diffusion Inpainting with Exact and Fast Conditional Inference". LanPaint uses Langevin Dynamics as "thinking" steps, which digs deeper into the diffusion process and allows the model to generate more consistent results.
|
||||
|
||||
LanPaint introduces **two-way alignment** between masked and unmasked areas. It continuously evaluates:
|
||||
LanPaint introduces "BIG score" that creates a **two-way alignment** between masked and unmasked areas. It continuously evaluates:
|
||||
- *"Does the new content make sense with the existing elements?"*
|
||||
- *"Do the existing elements support the new creation?"*
|
||||
|
||||
Based on this evaluation, LanPaint iteratively updates the noise in both the masked and unmasked regions.
|
||||
|
||||
LanPaint also implements an accurate, robust, and fast Langevin dynamics solver.
|
||||
|
||||
## Updates
|
||||
- 2025/05/28
|
||||
- Major update on the Langevin solver. It is now much faster and more stable.
|
||||
- Fix performance issue on Flux and SD 3.5
|
||||
- 2025/04/16
|
||||
- Added Primary HiDream support
|
||||
- 2025/03/22
|
||||
@@ -29,60 +34,63 @@ Based on this evaluation, LanPaint iteratively updates the noise in both the mas
|
||||
- 2025/03/10
|
||||
- LanPaint has received a major update! All examples now use the LanPaint K Sampler, offering a simplified interface with enhanced performance and stability.
|
||||
|
||||
## Example Results
|
||||
## Examples
|
||||
All examples use a random seed 0 to generate batch of 4 images for fair comparison. (Warning: Generating 4 images may exceed your GPU memory; adjust batch size as needed.)
|
||||
|
||||
|
||||
|
||||
|
||||
### Example SD 3.5: InPaint(LanPaint K Sampler, 5 steps of thinking)
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_9)
|
||||
|
||||
You need to follow the ComfyUI version of [SD 3.5 workflow](https://comfyui-wiki.com/en/tutorial/advanced/stable-diffusion-3-5-comfyui-workflow) to download and install the model.
|
||||
|
||||
### Example HiDream: InPaint(LanPaint K Sampler, 5 steps of thinking)
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_8)
|
||||
|
||||
You need to install [ComfyUI GGUF](https://github.com/city96/ComfyUI-GGUF) in order to load the models. Make sure you have the latest (nightly at 2025/04/16) comfyui installed. The following models are needed for Hidream:
|
||||
- [clip_g_hidream.safetensors](https://huggingface.co/Comfy-Org/HiDream-I1_ComfyUI/blob/main/split_files/text_encoders/clip_g_hidream.safetensors)
|
||||
- [clip_l_hidream.safetensors](https://huggingface.co/Comfy-Org/HiDream-I1_ComfyUI/blob/main/split_files/text_encoders/clip_l_hidream.safetensors)
|
||||
- [T5 GGUF](https://huggingface.co/city96/t5-v1_1-xxl-encoder-gguf/tree/main)
|
||||
- [Llama 3.1](https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/tree/main)
|
||||
- [Flux VAE](https://huggingface.co/StableDiffusionVN/Flux/blob/main/Vae/flux_vae.safetensors)
|
||||
You need to follow the ComfyUI version of [HiDream workflow](https://docs.comfy.org/tutorials/image/hidream/hidream-i1) to download and install the model.
|
||||
|
||||
### Example Flux: InPaint(LanPaint K Sampler, 5 steps of thinking)
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_7)
|
||||
[Model Used in This Example](https://huggingface.co/Comfy-Org/flux1-dev/blob/main/flux1-dev-fp8.safetensors)
|
||||
(Note: Prompt First mode is disabled on Flux. As it does not use CFG guidance.)
|
||||
|
||||
### Example 1: Basket to Basket Ball (LanPaint K Sampler, 2 steps of thinking).
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_1)
|
||||
[Model Used in This Example](https://civitai.com/models/1188071?modelVersionId=1408658)
|
||||
### Example 2: White Shirt to Blue Shirt (LanPaint K Sampler, 5 steps of thinking)
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_2)
|
||||
[Model Used in This Example](https://civitai.com/models/1188071?modelVersionId=1408658)
|
||||
### Example 3: Smile to Sad (LanPaint K Sampler, 5 steps of thinking)
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_3)
|
||||
[Model Used in This Example](https://civitai.com/models/133005/juggernaut-xl)
|
||||
### Example 4: Damage Restoration (LanPaint K Sampler, 5 steps of thinking)
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_4)
|
||||
[Model Used in This Example](https://civitai.com/models/133005/juggernaut-xl)
|
||||
### Example 5: Huge Damage Restoration (LanPaint K Sampler, 20 steps of thinking)
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_5)
|
||||
[Model Used in This Example](https://civitai.com/models/133005/juggernaut-xl)
|
||||
### Example 6: Character Consistency (Side View Generation) (LanPaint K Sampler, 5 steps of thinking)
|
||||
### Example SDXL 0: Character Consistency (Side View Generation) (LanPaint K Sampler, 5 steps of thinking)
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_6)
|
||||
[Model Used in This Example](https://civitai.com/models/1188071?modelVersionId=1408658)
|
||||
|
||||
### Example SDXL 1: Basket to Basket Ball (LanPaint K Sampler, 2 steps of thinking).
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_1)
|
||||
[Model Used in This Example](https://civitai.com/models/1188071?modelVersionId=1408658)
|
||||
### Example SDXL 2: White Shirt to Blue Shirt (LanPaint K Sampler, 5 steps of thinking)
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_2)
|
||||
[Model Used in This Example](https://civitai.com/models/1188071?modelVersionId=1408658)
|
||||
### Example SDXL 3: Smile to Sad (LanPaint K Sampler, 5 steps of thinking)
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_3)
|
||||
[Model Used in This Example](https://civitai.com/models/133005/juggernaut-xl)
|
||||
### Example SDXL 4: Damage Restoration (LanPaint K Sampler, 5 steps of thinking)
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_4)
|
||||
[Model Used in This Example](https://civitai.com/models/133005/juggernaut-xl)
|
||||
### Example SDXL 5: Huge Damage Restoration (LanPaint K Sampler, 20 steps of thinking)
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_5)
|
||||
[Model Used in This Example](https://civitai.com/models/133005/juggernaut-xl)
|
||||
|
||||
|
||||
(Tricks 1: You can emphasize the character by copy it's image multiple times with Photoshop. Here I have made one extra copy.)
|
||||
|
||||
(Tricks 2: Use prompts like multiple views, multiple angles, clone, turnaround.)
|
||||
|
||||
(Tricks 3: Remeber LanPaint can in-paint: Mask non-consistent regions and try again!)
|
||||
|
||||
### Example 7: Flux Model InPaint(LanPaint K Sampler, 5 steps of thinking)
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_7)
|
||||
[Model Used in This Example](https://huggingface.co/Comfy-Org/flux1-dev/blob/main/flux1-dev-fp8.safetensors)
|
||||
(Note: Use CFG scale 1.0 for Flux as it don't use CFG. LanPaint_cfg_BIG is also disabled on Flux)
|
||||
|
||||
|
||||
|
||||
|
||||
## **How to Use These Examples:**
|
||||
@@ -127,15 +135,16 @@ Same as default ComfyUI KSampler - simply replace with LanPaint KSampler nodes.
|
||||
|
||||
## Basic Sampler
|
||||
|
||||
|
||||
- LanPaint KSampler: The most basic and easy to use sampler for inpainting.
|
||||
- LanPaint KSampler (Advanced): Full control of all parameters.
|
||||
|
||||
### LanPaint KSampler
|
||||
Simplified interface with recommended defaults:
|
||||
|
||||
- Steps: 50+ recommended
|
||||
- LanPaint NumSteps: The turns of thinking before denoising. Recommend 5 for most of tasks.
|
||||
- LanPaint EndSigma: The noise level below which thinking is disabled. Recommend 0.6 for realistic style (tested on Juggernaut-xl), 3.0 for anime style (tested on Animagine XL 4.0)
|
||||
|
||||
The default settings are tested on Animagine XL 4.0 and Juggernaut-xl. Other model might need some paramter tuning. Please raise issue or share your own setting if it doesn't work on your model.
|
||||
|
||||
- Steps: 20 - 50. More steps will give more "thinking" and better results.
|
||||
- LanPaint NumSteps: The turns of thinking before denoising. Recommend 5 for most of tasks ( which means 5 times slower than sampling without thinking). Use 10 for more challenging tasks.
|
||||
- LanPaint Prompt mode: Image First mode and Prompt First mode. Image First mode focuses on the image, while Prompt First mode focuses more on the prompt. Use Prompt First mode for tasks like character consistency. (Technically, it Prompt First mode change CFG scale to negative value in the BIG score to emphasis prompt, which will costs image quality.)
|
||||
|
||||
### LanPaint KSampler (Advanced)
|
||||
Full parameter control:
|
||||
@@ -143,12 +152,13 @@ Full parameter control:
|
||||
|
||||
| Parameter | Range | Description |
|
||||
|-----------|-------|-------------|
|
||||
| `Steps` | 0-100 | Total steps of diffusion sampling. Higher means better inpainting. Recommend 50. |
|
||||
| `LanPaint_NumSteps` | 0-20 | Reasoning iterations per denoising step ("thinking depth"). Easy task: 1-2. Hard task: 5-10 |
|
||||
| `LanPaint_Lambda` | 0.1-50 | Content alignment strength (higher = stricter). Recommend 8.0 |
|
||||
| `LanPaint_StepSize` | 0.1-1.0 | The StepSize of each thinking step. Recommend 0.5. |
|
||||
| `LanPaint_EndSigma` | 0.0-20.0 | The noise level below which thinking is disabled. recommend 0.3 - 3. High value is faster, but may damage quality. Low value gives more thinking but might make the output blurry. |
|
||||
| `LanPaint_cfg_BIG` | -20-20 | CFG scale used when aligning masked and unmasked region (positive value tends to ignores promts, negative value enhances prompts.). Recommend 8 for seamless inpaint (i.e limbs, faces) when prompt is not important. -0.5 when prompt is important, like character consistency (i.e multiple view) |
|
||||
| `Steps` | 0-100 | Total steps of diffusion sampling. Higher means better inpainting. Recommend 20-50. |
|
||||
| `LanPaint_NumSteps` | 0-20 | Reasoning iterations per denoising step ("thinking depth"). Easy task: 2-5. Hard task: 5-10 |
|
||||
| `LanPaint_Lambda` | 0.1-50 | Content alignment strength (higher = stricter). Recommend 4.0 - 10.0 |
|
||||
| `LanPaint_StepSize` | 0.1-1.0 | The StepSize of each thinking step. Recommend 0.1-0.5. |
|
||||
| `LanPaint_Beta` | 0.1-2.0 | The StepSize ratio between masked / unmasked region. Small value can compensate high lambda values. Recommend 1.0 |
|
||||
| `LanPaint_Friction` | 0.0-100.0 | The friction of Langevin dynamics. Higher means more slow but stable, lower means fast but unstable. Recommend 10.0 - 20.0|
|
||||
| `LanPaint_PromptMode` | Image First / Prompt First | Image First mode focuses on the image, while Prompt First mode focuses more on the prompt. |
|
||||
|
||||
For detailed descriptions of each parameter, simply hover your mouse over the corresponding input field to view tooltips with additional information.
|
||||
|
||||
@@ -157,33 +167,31 @@ For detailed descriptions of each parameter, simply hover your mouse over the co
|
||||
## LanPaint KSampler (Advanced) Tuning Guide
|
||||
For challenging inpainting tasks:
|
||||
|
||||
1️⃣ **Primary Adjustments**:
|
||||
- Decrease **LanPaint_endsigma** increase **LanPaint_NumSteps** (thinking iterations) if the inpainted area is not seamless.
|
||||
|
||||
1️⃣ **Boost Quality**
|
||||
Increase **LanPaint_NumSteps** (thinking iterations) or **LanPaint_Lambda** if the inpainted result does not meet your expectations.
|
||||
|
||||
2️⃣ **Secondary Tweaks**:
|
||||
- Boost **LanPaint_Lambda** (bidirectional guidance scale) will force the masked/unmasked region to align more closely.
|
||||
- If the output is blurry, increase **LanPaint_endsigma** to turn off thinking at the end of denoising. OR decrease **LanPaint_StepSize** to decrease thinking step size.
|
||||
- If prompt is not that important, try increase **LanPaint_cfg_BIG**(cfg scale used for unmasked region, default -0.5 ) to 8 for better inpainting.
|
||||
2️⃣ **Boost Speed**
|
||||
If you want better results but still need fewer steps, consider:
|
||||
- **Increasing LanPaint_StepSize** to speed up the thinking process.
|
||||
- **Decreasing LanPaint_Friction** to make the Langevin dynamics converges more faster.
|
||||
|
||||
3️⃣ **Balance Speed vs Stability**:
|
||||
- Reduce **LanPaint_Friction** to prioritize faster results with fewer "thinking" steps (*may risk instability*).
|
||||
- Increase **LanPaint_Tamed** (noise normalization onto a sphere) or **LanPaint_Alpha** (constraint the friction of underdamped Langevin dynamics) to suppress artifacts like blurry/wired texture.
|
||||
3️⃣ **Fix Unstability**:
|
||||
If you find the results have wired texture, try
|
||||
- Reduce **LanPaint_Friction** to make the Langevin dynamics more stable.
|
||||
- Reduce **LanPaint_StepSize** to use smaller step size.
|
||||
- Reduce **LanPaint_Beta** if you are using a high lambda value.
|
||||
|
||||
⚠️ **Notes**:
|
||||
- Optimal parameters vary depending on the **model** and the **size of the inpainting area**.
|
||||
- For effective tuning, **fix the seed** and adjust parameters incrementally while observing the results. This helps isolate the impact of each setting. Better to do it with a batche of images to avoid overfitting on a single image.
|
||||
|
||||
## ToDo
|
||||
- Fix compatibility issue with Flux Guidance that causing performance degradation on Flux models.
|
||||
- Fix SD 3.5 compatibility problems
|
||||
- Try Implement Detailer
|
||||
|
||||
## Contribute
|
||||
|
||||
- 2025/03/06: Bug Fix for str not callable error and unpack error. Big thanks to [jamesWalker55](https://github.com/jamesWalker55) and [EricBCoding](https://github.com/EricBCoding).
|
||||
|
||||
Help us improve LanPaint! 🚀 **Report bugs**, share **example cases**, or contribute your **personal parameter settings** to benefit the community.
|
||||
Help us improve LanPaint! **Report bugs**, share **example cases**, or contribute your **personal parameter settings** to benefit the community.
|
||||
|
||||
## Citation
|
||||
|
||||
|
||||
Reference in New Issue
Block a user