Compare commits

..
104 Commits
Author SHA1 Message Date
scraed 1820d88594 fix link 2025-09-17 15:59:35 +08:00
scraed c265bcc989 Add Wan2.2 inpainting example and bump version
Added a new example for Wan2.2 inpainting with image and workflow links in the README. Updated project version to 1.3.2 in pyproject.toml.
2025-09-17 15:55:28 +08:00
scraed b8973c570b Support Wan 2.2 Txt 2 Img 2025-09-17 15:46:11 +08:00
scraed 1b50a90424 qwen edit resize fix 2025-09-07 13:07:17 +08:00
charrywhite dab41249da Update README.md 2025-08-30 11:15:11 +08:00
charrywhite b2b2b21bf1 Update README.md 2025-08-30 11:14:53 +08:00
scraed 099d4137a9 Clarify speed improvement suggestions
Reworded instructions for boosting speed in README.
2025-08-26 12:10:11 +08:00
scraed 2f7967a584 update version 2025-08-25 11:37:54 +08:00
scraed e28e115297 add example workflow jsons 2025-08-25 11:37:31 +08:00
scraed f2cc70fed3 Update inpainted example image
Replaces the InPainted_Drag_Me_to_ComfyUI.png file in Example_14 with a new version. This may reflect updated visual content or corrections to the example image.
2025-08-25 11:11:09 +08:00
scraed b8e140f7aa Comment out unused model_sigmas initialization
The initialization of model_sigmas in KSamplerX0Inpaint was commented out to remove warning. Also remove group nodes from example workflow
2025-08-22 10:34:51 +08:00
scraed e6f61e806c Bump version to 1.3.0 in pyproject.toml
Updated the project version from 1.2.0 to 1.3.0 to reflect new changes or improvements in the LanPaint package.
2025-08-22 00:26:46 +08:00
scraed ba506f2654 Add new Qwen edit example and update README
Added Example_14 images and LanPaintQwen_03.jpg to showcase the masked Qwen edit workflow. Updated README with new example references and instructions for using the ComfyUI Qwen Image Edit workflow.
2025-08-22 00:17:40 +08:00
scraed 986dee6bd9 Merge branch 'master' of https://github.com/scraed/LanPaint 2025-08-21 18:12:18 +08:00
scraed e514553120 fix performance issue 2025-08-21 18:12:11 +08:00
scraed 40d7f0854e Update README.md 2025-08-18 23:07:02 +08:00
charrywhite a5e3af84f9 Update README.md 2025-08-18 18:18:21 +08:00
charrywhite 598ca776eb Update README.md 2025-08-18 18:10:50 +08:00
charrywhite b36584b2a7 Update README.md 2025-08-18 18:07:49 +08:00
charrywhite 082c549b24 Update README.md 2025-08-18 18:05:21 +08:00
charrywhite 453b6c090e Update README.md 2025-08-18 17:45:27 +08:00
charrywhite 07229ecbcd Update README.md
reorganize readme
2025-08-18 17:36:02 +08:00
charrywhite 76cf0ff5fe Update README.md 2025-08-18 17:25:21 +08:00
scraed e1f84469f1 Merge branch 'master' of https://github.com/scraed/LanPaint 2025-08-18 17:20:03 +08:00
charrywhite 1fa77b0d1e Update README.md 2025-08-18 17:19:20 +08:00
scraed cd62e71467 Update LanPaintQwen_01.jpg example image
Replaces the existing LanPaintQwen_01.jpg in the examples directory with a new version.
2025-08-18 17:19:02 +08:00
charrywhite deaf2cec2a Update README.md 2025-08-18 17:18:01 +08:00
scraed 587a941607 Update README.md 2025-08-18 15:44:21 +08:00
scraed 7bdbab75e6 Update LanPaintQwen_01.jpg 2025-08-18 15:35:01 +08:00
scraed b77c25677f Fix swapped links for Qwen workflows in README
Corrected the example links for Qwen Inpaint and Outpaint workflows to point to their respective directories.
2025-08-18 15:30:29 +08:00
scraed bf5b607238 update readme 2025-08-18 15:29:21 +08:00
scraed ad77fb7836 Merge branch 'master' of https://github.com/scraed/LanPaint 2025-08-18 15:27:19 +08:00
scraed 3f8eb2552f Update README and add new Qwen inpainting examples
Enhanced README with details and links for Qwen Image workflows and added new example images for Qwen inpainting and outpainting in Example_12 and Example_13 directories. Also included a sample result image for Qwen inpainting.
2025-08-18 15:27:12 +08:00
scraed 238e49e31f Update README.md 2025-08-15 15:34:36 +08:00
scraed 7aeb6e535f Add Qwen image support and new inpainting examples
Updated the README to announce Qwen image support and provide usage instructions. Added new example images and workflow files for Qwen inpainting in the examples directory.
2025-08-08 14:09:53 +08:00
scraed eda0f19944 Generalize dimension handling in LanPaint and fix mask reshape
Refactored LanPaint to dynamically handle tensors with varying numbers of dimensions by introducing add_none_dims and remove_none_dims utility methods. Updated all relevant tensor broadcasting to use these methods, improving flexibility. Also fixed reshape_mask in nodes.py to use the last two dimensions for resizing, ensuring correct mask shape.
2025-08-08 13:42:24 +08:00
scraed e20c8f20ce update version 2025-06-21 14:13:09 +08:00
scraed 89b9010b35 Fix indentation error in sample method
Corrected an indentation issue in the sample method of LanPaint_SamplerCustomAdvanced to ensure proper execution of the end_at_step check.
2025-06-21 14:09:11 +08:00
scraed ee0c65656e simplify custom sampler 2025-06-21 13:18:25 +08:00
scraed 48b1dd4be6 Merge branch 'master' into pr/33 2025-06-21 13:10:05 +08:00
scraed 9d304cd5a3 update readme 2025-06-21 13:03:30 +08:00
scraed 61c19ac31d update algorithm with better outpaint 2025-06-21 12:56:36 +08:00
Bagier 1ca6e53090 adjust info names 2025-06-20 08:31:51 +07:00
Bagier 520932cee0 add invalid value check 2025-06-19 20:49:20 +07:00
Bagier 37612bb399 Logic fix 2025-06-19 20:12:50 +07:00
Bagier 0e6cb45081 re-add lanpaint info 2025-06-19 19:25:59 +07:00
Bagier 0a5e36d55e add advanced custom sampler 2025-06-19 19:15:58 +07:00
Bagier b319f772cb fix parameter update error 2025-06-19 19:00:41 +07:00
Bagier 340376ad18 Add custom sampler variation 2025-06-19 16:14:26 +07:00
scraed dbfc1585fc Merge branch 'master' of https://github.com/scraed/LanPaint 2025-06-16 20:03:31 +08:00
scraed c3ae2c644d update coefficients 2025-06-16 20:03:25 +08:00
scraed c7017373c9 Update pyproject.toml 2025-06-08 17:59:04 +08:00
scraed 850f707eb6 Remove redundant comment 2025-06-08 17:58:40 +08:00
scraed 62870f060a Update README.md 2025-06-06 13:05:50 +08:00
scraed a91cefacf0 Update README.md 2025-06-06 13:05:16 +08:00
scraed 4265f71a85 Update README.md 2025-06-05 18:15:21 +08:00
scraed 4189ba80c2 add error message 2025-06-05 17:10:07 +08:00
scraed 23ad6e47fd add new masked blend node 2025-06-05 16:21:46 +08:00
scraed 4d3d5d17f0 Update README.md 2025-06-05 01:22:09 +08:00
scraed b86f3b7112 update version 2025-06-05 01:14:57 +08:00
scraed 96b7f3eecd fix sigma batch error 2025-06-05 01:11:04 +08:00
scraed e472574783 update readme 2025-06-04 21:33:58 +08:00
scraed 0b799be8ec support more sampler 2025-06-04 21:28:51 +08:00
scraed 82da0b4142 Merge branch 'master' of https://github.com/scraed/LanPaint 2025-06-04 20:04:58 +08:00
scraed c66155e783 add early stop 2025-06-04 20:04:53 +08:00
scraed d802f2f2b0 Update README.md fix typo 2025-06-04 14:34:10 +08:00
scraed 6153a450c9 Update README.md 2025-06-04 14:33:07 +08:00
scraed 493658d23a Update README.md 2025-06-04 14:28:11 +08:00
scraed 7de1054b39 Update README.md 2025-06-04 09:06:39 +08:00
scraed 574d9906ec Update README.md 2025-06-03 20:54:12 +08:00
scraed 61d4ecef75 Update README.md 2025-06-03 18:41:24 +08:00
scraed 5eb2d8ef48 Update README.md 2025-06-03 17:19:52 +08:00
scraed b668bcc42c Update README.md 2025-06-03 13:11:47 +08:00
scraed 05386f98d3 replace outdated flux picture 2025-06-03 11:49:40 +08:00
scraed ad5704d25f correct tricks for consistency 2025-06-03 11:06:35 +08:00
scraed 2ab72851b9 update summary img 2025-06-03 10:29:54 +08:00
scraed 14f7907c70 update version 2025-06-03 10:19:41 +08:00
scraed 5adce4ecdd update readme 2025-05-28 18:11:30 +08:00
scraed 2fd945e69b update readme and pictures 2025-05-28 18:04:52 +08:00
scraed 1bd6932cae change to ve notation and update examples 2025-05-28 17:22:15 +08:00
scraed 59b31303c8 update examples 2025-05-27 10:44:15 +08:00
scraed f906bd9cc9 reduce parameters 2025-05-22 15:01:36 +08:00
scraed 31e4909438 switch to sampler alg 2025-05-22 10:14:58 +08:00
scraed 6775e6ac37 1-a schedule and shared A 2025-05-17 19:06:43 +08:00
scraed 49b35bcb28 fix Zcoef asymp bug 2025-05-14 15:55:18 +08:00
scraed 13bd3182bf switch to separate file 2025-05-13 19:14:46 +08:00
scraed bb3e0078d4 switch to new new alg 2025-05-13 18:33:08 +08:00
scraed 5fc4cf2092 remove y time truncate 2025-05-12 22:34:44 +08:00
scraed 3f8e2b833c change y update alg 2025-05-12 22:27:56 +08:00
scraed 0f546f09b6 add truncate time step 2025-05-12 16:38:35 +08:00
scraed 99ac3ee06d Create utils.py 2025-05-12 11:40:05 +08:00
scraed 505fbfb7fc fix tamed bug 2025-05-12 09:49:12 +08:00
scraed aa541edaea change default lambda schedule to const 2025-05-03 23:16:07 +08:00
scraed 5382fbba21 remove redundant time step schedule 2025-05-03 22:31:59 +08:00
scraed 82a6798e29 change parameter range 2025-05-03 22:28:25 +08:00
scraed 3b7c79f431 add lamb schedule 2025-05-03 01:01:02 +08:00
scraed ae31ac7e70 complete migration to new alg formula 2025-05-02 22:31:37 +08:00
scraed 2a67dd353f update epxm1Dx 2025-05-02 21:14:54 +08:00
scraed 2d0f458695 update eps 2025-05-02 13:50:19 +08:00
scraed 6f2deeda51 separate ld 2025-05-02 00:38:58 +08:00
scraed 76cd0a5c1d sort tamed 2025-05-01 21:08:45 +08:00
scraed f90636d4c0 create hidream img 2025-04-23 22:05:02 +08:00
scraed dc4ef3aed4 Update README.md 2025-04-22 08:55:51 +08:00
scraed eca581ee79 Update README.md 2025-04-17 15:38:32 +08:00
58 changed files with 6390 additions and 421 deletions
BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 70 KiB

After

Width:  |  Height:  |  Size: 119 KiB

+205 -126
View File
@@ -1,105 +1,57 @@
# LanPaint (Thinking mode Inpaint)
<div align="center">
Unlock precise inpainting without additional training. LanPaint lets the model "think" through multiple iterations before denoising, aiming for seamless and accurate results.
# LanPaint: Universal Inpainting Sampler with "Think Mode"
[![arXiv](https://img.shields.io/badge/Arxiv-2502.03491-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2502.03491)
[![Python Benchmark](https://img.shields.io/badge/🐍-Python_Benchmark-3776AB?logo=python)](https://github.com/scraed/LanPaintBench)
[![ComfyUI Extension](https://img.shields.io/badge/ComfyUI-Extension-7B5DFF)](https://github.com/comfyanonymous/ComfyUI)
[![Hugging Face](https://img.shields.io/badge/Hugging%20Face-yellow?logo=huggingface&logoColor=white)](https://huggingface.co/charrywhite/LanPaint)
[![Blog](https://img.shields.io/badge/📝-Blog-9cf)](https://scraed.github.io/scraedBlog/)
[![GitHub stars](https://img.shields.io/github/stars/scraed/LanPaint)](https://github.com/scraed/LanPaint/stargazers)
</div>
We encourage you to try it out and share your feedback through issues or discussions, as your input will help us enhance the algorithm's performance and stability.
Universally applicable inpainting ability for every model. LanPaint sampler lets the model "think" through multiple iterations before denoising, enabling you to invest more computation time for superior inpainting quality.
This is the official implementation of ["Lanpaint: Training-Free Diffusion Inpainting with Exact and Fast Conditional Inference"](https://arxiv.org/abs/2502.03491). The repository is for ComfyUI extension. Local Python benchmark code is published here: [LanPaintBench](https://github.com/scraed/LanPaintBench).
![Qwen Result 2](https://github.com/scraed/LanPaint/blob/master/examples/LanPaintQwen_03.jpg)
Check [Mased Qwen Edit Workflow](https://github.com/scraed/LanPaint/tree/master/examples/Example_14). You need to follow the ComfyUI version of [Qwen Image Edit workflow](https://docs.comfy.org/tutorials/image/qwen/qwen-image-edit) to download and install the model.
![Qwen Result 1](https://github.com/scraed/LanPaint/blob/master/examples/LanPaintQwen_01.jpg)
Also check [Qwen Inpaint Workflow](https://github.com/scraed/LanPaint/tree/master/examples/Example_13) and [Qwen Outpaint Workflow](https://github.com/scraed/LanPaint/tree/master/examples/Example_12). You need to follow the ComfyUI version of [Qwen Image workflow](https://docs.comfy.org/tutorials/image/qwen/qwen-image) to download and install the model.
## Table of Contents
- [Features](#features)
- [Quickstart](#quickstart)
- [How to Use Examples](#how-to-use-examples)
- [Examples](#examples)
- [Wan 2.2 T2I](#example-wan22-inpaintlanpaint-k-sampler-5-steps-of-thinking)
- [Qwen Image](#example-qwen-image-inpaintlanpaint-k-sampler-5-steps-of-thinking)
- [HiDream](#example-hidream-inpaint-lanpaint-k-sampler-5-steps-of-thinking)
- [SD 3.5](#example-sd-35-inpaintlanpaint-k-sampler-5-steps-of-thinking)
- [Flux](#example-flux-inpaintlanpaint-k-sampler-5-steps-of-thinking)
- [SDXL Examples](#example-sdxl-0-character-consistency-side-view-generation-lanpaint-k-sampler-5-steps-of-thinking)
- [Usage](#usage)
- [Basic Sampler](#basic-sampler)
- [Advanced Sampler](#lanpaint-ksampler-advanced)
- [Tuning Guide](#lanpaint-ksampler-advanced-tuning-guide)
- [Community Showcase](#community-showcase-)
- [Updates](#updates)
- [ToDo](#todo)
- [Citation](#citation)
## Features
- 🎨 **Zero-Training Inpainting** - Works immediately with ANY SD model (with/without ControlNet), and Flux model! even custom models you've trained yourself
- 🛠️ **Simple Integration** - Same workflow as standard ComfyUI KSampler
- 🎯 **True Blank-Slate Generation** - No need to set default denoise at 0.7 (preserving 30% original pixels in masks) used in conventional methods: 100% **new content creation**, No "painting over" existing content.
- 🌈 **Not only inpaint**: You can even use it as a simple way to generate consistent characters.
- **Universal Compatibility** – Works instantly with almost any model (**SD 1.5, XL, 3.5, Flux, HiDream, Qwen-Image or custom LoRAs**) and ControlNet.
![Inpainting Result 13](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_13.jpg)
- **No Training Needed** – Works out of the box with your existing model.
- **Easy to Use** – Same workflow as standard ComfyUI KSampler.
- **Flexible Masking** – Supports any mask shape, size, or position for inpainting/outpainting.
- **No Workarounds** – Generates 100% new content (no blending or smoothing) without relying on partial denoising.
- **Beyond Inpainting** – You can even use it as a simple way to generate consistent characters.
## How It Works
LanPaint introduces **two-way alignment** between masked and unmasked areas. It continuously evaluates:
- *"Does the new content make sense with the existing elements?"*
- *"Do the existing elements support the new creation?"*
Based on this evaluation, LanPaint iteratively updates the noise in both the masked and unmasked regions.
## Updates
- 2025/04/16
- Added Primary HiDream support
- 2025/03/22
- Added Primary Flux support
- Added Tease Mode
- 2025/03/10
- LanPaint has received a major update! All examples now use the LanPaint K Sampler, offering a simplified interface with enhanced performance and stability.
## Example Results
All examples use a random seed 0 to generate batch of 4 images for fair comparison. (Warning: Generating 4 images may exceed your GPU memory; adjust batch size as needed.)
### Example HiDream: InPaint(LanPaint K Sampler, 5 steps of thinking)
![Inpainting Result 8](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_11.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_8)
You need to install [ComfyUI GGUF](https://github.com/city96/ComfyUI-GGUF) in order to load the models. Make sure you have the latest (nightly at 2025/04/16) comfyui installed. The following models are needed for Hidream:
- [clip_g_hidream.safetensors](https://huggingface.co/Comfy-Org/HiDream-I1_ComfyUI/blob/main/split_files/text_encoders/clip_g_hidream.safetensors)
- [clip_l_hidream.safetensors](https://huggingface.co/Comfy-Org/HiDream-I1_ComfyUI/blob/main/split_files/text_encoders/clip_l_hidream.safetensors)
- [T5 GGUF](https://huggingface.co/city96/t5-v1_1-xxl-encoder-gguf/tree/main)
- [Llama 3.1](https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/tree/main)
- [Flux VAE](https://huggingface.co/StableDiffusionVN/Flux/blob/main/Vae/flux_vae.safetensors)
### Example 1: Basket to Basket Ball (LanPaint K Sampler, 2 steps of thinking).
![Inpainting Result 1](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_04.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_1)
[Model Used in This Example](https://civitai.com/models/1188071?modelVersionId=1408658)
### Example 2: White Shirt to Blue Shirt (LanPaint K Sampler, 5 steps of thinking)
![Inpainting Result 2](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_05.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_2)
[Model Used in This Example](https://civitai.com/models/1188071?modelVersionId=1408658)
### Example 3: Smile to Sad (LanPaint K Sampler, 5 steps of thinking)
![Inpainting Result 3](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_06.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_3)
[Model Used in This Example](https://civitai.com/models/133005/juggernaut-xl)
### Example 4: Damage Restoration (LanPaint K Sampler, 5 steps of thinking)
![Inpainting Result 4](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_07.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_4)
[Model Used in This Example](https://civitai.com/models/133005/juggernaut-xl)
### Example 5: Huge Damage Restoration (LanPaint K Sampler, 20 steps of thinking)
![Inpainting Result 5](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_08.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_5)
[Model Used in This Example](https://civitai.com/models/133005/juggernaut-xl)
### Example 6: Character Consistency (Side View Generation) (LanPaint K Sampler, 5 steps of thinking)
![Inpainting Result 6](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_09.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_6)
[Model Used in This Example](https://civitai.com/models/1188071?modelVersionId=1408658)
(Tricks 1: You can emphasize the character by copy it's image multiple times with Photoshop. Here I have made one extra copy.)
(Tricks 2: Use prompts like multiple views, multiple angles, clone, turnaround.)
(Tricks 3: Remeber LanPaint can in-paint: Mask non-consistent regions and try again!)
### Example 7: Flux Model InPaint(LanPaint K Sampler, 5 steps of thinking)
![Inpainting Result 7](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_10.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_7)
[Model Used in This Example](https://huggingface.co/Comfy-Org/flux1-dev/blob/main/flux1-dev-fp8.safetensors)
(Note: Use CFG scale 1.0 for Flux as it don't use CFG. LanPaint_cfg_BIG is also disabled on Flux)
## **How to Use These Examples:**
1. Navigate to the **example** folder (i.e example_1) by clicking **View Workflow & Masks**, download all pictures.
2. Drag **InPainted_Drag_Me_to_ComfyUI.png** into ComfyUI to load the workflow.
3. Download the required model from Civitai by clicking **Model Used in This Example**.
4. Load the model into the **"Load Checkpoint"** node.
5. Upload **Original_No_Mask.png** to the **"Load image"** node in the **"Original Image"** group (far left).
6. Upload **Masked_Load_Me_in_Loader.png** to the **"Load image"** node in the **"Mask image for inpainting"** group (second from left).
7. Queue the task, you will get inpainted results from three methods:
- **[VAE Encode for Inpainting](https://comfyanonymous.github.io/ComfyUI_examples/inpaint/)** (middle),
- **[Set Latent Noise Mask](https://comfyui-wiki.com/en/tutorial/basic/how-to-inpaint-an-image-in-comfyui)** (second from right),
- **LanPaint** (far right).
Compare and explore the results from each method!
![WorkFlow](https://github.com/scraed/LanPaint/blob/master/Example.JPG)
**Warning**: LanPaint has degraded performance on distillation models, such as Flux.dev, due to a similar [issue with LORA training](https://medium.com/@zhiwangshi28/why-flux-lora-so-hard-to-train-and-how-to-overcome-it-a0c70bc59eaf). Please use low flux guidance (1.0-2.0) to mitigate this [issue](https://github.com/scraed/LanPaint/issues/30).
## Quickstart
@@ -116,6 +68,99 @@ Compare and explore the results from each method!
Once installed, you'll find the LanPaint nodes under the "sampling" category in ComfyUI. Use them just like the default KSampler for high-quality inpainting!
## **How to Use Examples:**
1. Navigate to the **example** folder (i.e example_1), download all pictures.
2. Drag **InPainted_Drag_Me_to_ComfyUI.png** into ComfyUI to load the workflow.
3. Download the required model (i.e clicking **Model Used in This Example**).
4. Load the model in ComfyUI.
5. Upload **Masked_Load_Me_in_Loader.png** to the **"Load image"** node in the **"Mask image for inpainting"** group (second from left), or the **Prepare Image** node.
7. Queue the task, you will get inpainted results from LanPaint. Some example also gives you inpainted results from the following methods for comparison:
- **[VAE Encode for Inpainting](https://comfyanonymous.github.io/ComfyUI_examples/inpaint/)**
- **[Set Latent Noise Mask](https://comfyui-wiki.com/en/tutorial/basic/how-to-inpaint-an-image-in-comfyui)**
## Examples
### Example Wan2.2: InPaint(LanPaint K Sampler, 5 steps of thinking)
We are excited to announce that LanPaint now supports Wan2.2 text to image generation with Wan2.2 T2V model.
![Inpainting Result 45](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_45.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_15)
You need to follow the ComfyUI version of [Wan2.2 T2V workflow](https://docs.comfy.org/tutorials/video/wan/wan2_2) to download and install the T2V model.
### Example Qwen Image: InPaint(LanPaint K Sampler, 5 steps of thinking)
![Inpainting Result 14](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_14.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_11)
You need to follow the ComfyUI version of [Qwen Image workflow](https://docs.comfy.org/tutorials/image/qwen/qwen-image) to download and install the model.
The following examples utilize a random seed of 0 to generate a batch of 4 images for variance demonstration and fair comparison. (Note: Generating 4 images may exceed your GPU memory; please adjust the batch size as necessary.)
### Example HiDream: InPaint (LanPaint K Sampler, 5 steps of thinking)
![Inpainting Result 8](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_11.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_8)
You need to follow the ComfyUI version of [HiDream workflow](https://docs.comfy.org/tutorials/image/hidream/hidream-i1) to download and install the model.
### Example HiDream: OutPaint(LanPaint K Sampler, 5 steps of thinking)
![Inpainting Result 8](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_13(1).jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_10)
You need to follow the ComfyUI version of [HiDream workflow](https://docs.comfy.org/tutorials/image/hidream/hidream-i1) to download and install the model. Thanks [Amazon90](https://github.com/Amazon90) for providing this example.
### Example SD 3.5: InPaint(LanPaint K Sampler, 5 steps of thinking)
![Inpainting Result 8](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_12.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_9)
You need to follow the ComfyUI version of [SD 3.5 workflow](https://comfyui-wiki.com/en/tutorial/advanced/stable-diffusion-3-5-comfyui-workflow) to download and install the model.
### Example Flux: InPaint(LanPaint K Sampler, 5 steps of thinking)
![Inpainting Result 7](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_10.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_7)
[Model Used in This Example](https://huggingface.co/Comfy-Org/flux1-dev/blob/main/flux1-dev-fp8.safetensors)
(Note: Prompt First mode is disabled on Flux. As it does not use CFG guidance.)
### Example SDXL 0: Character Consistency (Side View Generation) (LanPaint K Sampler, 5 steps of thinking)
![Inpainting Result 6](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_09.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_6)
[Model Used in This Example](https://civitai.com/models/1188071?modelVersionId=1408658)
(Tricks 1: You can emphasize the character by copy it's image multiple times with Photoshop. Here I have made one extra copy.)
(Tricks 2: Use prompts like multiple views, multiple angles, clone, turnaround. Use LanPaint's Prompt first mode (does not support Flux))
(Tricks 3: Remeber LanPaint can in-paint: Mask non-consistent regions and try again!)
### Example SDXL 1: Basket to Basket Ball (LanPaint K Sampler, 2 steps of thinking).
![Inpainting Result 1](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_04.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_1)
[Model Used in This Example](https://civitai.com/models/1188071?modelVersionId=1408658)
### Example SDXL 2: White Shirt to Blue Shirt (LanPaint K Sampler, 5 steps of thinking)
![Inpainting Result 2](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_05.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_2)
[Model Used in This Example](https://civitai.com/models/1188071?modelVersionId=1408658)
### Example SDXL 3: Smile to Sad (LanPaint K Sampler, 5 steps of thinking)
![Inpainting Result 3](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_06.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_3)
[Model Used in This Example](https://civitai.com/models/133005/juggernaut-xl)
### Example SDXL 4: Damage Restoration (LanPaint K Sampler, 5 steps of thinking)
![Inpainting Result 4](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_07.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_4)
[Model Used in This Example](https://civitai.com/models/133005/juggernaut-xl)
### Example SDXL 5: Huge Damage Restoration (LanPaint K Sampler, 20 steps of thinking)
![Inpainting Result 5](https://github.com/scraed/LanPaint/blob/master/examples/InpaintChara_08.jpg)
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_5)
[Model Used in This Example](https://civitai.com/models/133005/juggernaut-xl)
Check more for use cases like inpaint on [fine tuned models](https://github.com/scraed/LanPaint/issues/12#issuecomment-2938662021) and [face swapping](https://github.com/scraed/LanPaint/issues/12#issuecomment-2938723501), thanks to [Amazon90](https://github.com/Amazon90).
## Usage
**Workflow Setup**
@@ -127,15 +172,16 @@ Same as default ComfyUI KSampler - simply replace with LanPaint KSampler nodes.
## Basic Sampler
![Samplers](https://github.com/scraed/LanPaint/blob/master/Nodes.JPG)
- LanPaint KSampler: The most basic and easy to use sampler for inpainting.
- LanPaint KSampler (Advanced): Full control of all parameters.
### LanPaint KSampler
Simplified interface with recommended defaults:
- Steps: 50+ recommended
- LanPaint NumSteps: The turns of thinking before denoising. Recommend 5 for most of tasks.
- LanPaint EndSigma: The noise level below which thinking is disabled. Recommend 0.6 for realistic style (tested on Juggernaut-xl), 3.0 for anime style (tested on Animagine XL 4.0)
The default settings are tested on Animagine XL 4.0 and Juggernaut-xl. Other model might need some paramter tuning. Please raise issue or share your own setting if it doesn't work on your model.
- Steps: 20 - 50. More steps will give more "thinking" and better results.
- LanPaint NumSteps: The turns of thinking before denoising. Recommend 5 for most of tasks ( which means 5 times slower than sampling without thinking). Use 10 for more challenging tasks.
- LanPaint Prompt mode: Image First mode and Prompt First mode. Image First mode focuses on the image, inpaint based on image context (maybe ignore prompt), while Prompt First mode focuses more on the prompt. Use Prompt First mode for tasks like character consistency. (Technically, it Prompt First mode change CFG scale to negative value in the BIG score to emphasis prompt, which will costs image quality.)
### LanPaint KSampler (Advanced)
Full parameter control:
@@ -143,46 +189,79 @@ Full parameter control:
| Parameter | Range | Description |
|-----------|-------|-------------|
| `Steps` | 0-100 | Total steps of diffusion sampling. Higher means better inpainting. Recommend 50. |
| `LanPaint_NumSteps` | 0-20 | Reasoning iterations per denoising step ("thinking depth"). Easy task: 1-2. Hard task: 5-10 |
| `LanPaint_Lambda` | 0.1-50 | Content alignment strength (higher = stricter). Recommend 8.0 |
| `LanPaint_StepSize` | 0.1-1.0 | The StepSize of each thinking step. Recommend 0.5. |
| `LanPaint_EndSigma` | 0.0-20.0 | The noise level below which thinking is disabled. recommend 0.3 - 3. High value is faster, but may damage quality. Low value gives more thinking but might make the output blurry. |
| `LanPaint_cfg_BIG` | -20-20 | CFG scale used when aligning masked and unmasked region (positive value tends to ignores promts, negative value enhances prompts.). Recommend 8 for seamless inpaint (i.e limbs, faces) when prompt is not important. -0.5 when prompt is important, like character consistency (i.e multiple view) |
| `Steps` | 0-100 | Total steps of diffusion sampling. Higher means better inpainting. Recommend 20-50. |
| `LanPaint_NumSteps` | 0-20 | Reasoning iterations per denoising step ("thinking depth"). Easy task: 2-5. Hard task: 5-10 |
| `LanPaint_Lambda` | 0.1-50 | Content alignment strength (higher = stricter). Recommend 4.0 - 10.0 |
| `LanPaint_StepSize` | 0.1-1.0 | The StepSize of each thinking step. Recommend 0.1-0.5. |
| `LanPaint_Beta` | 0.1-2.0 | The StepSize ratio between masked / unmasked region. Small value can compensate high lambda values. Recommend 1.0 |
| `LanPaint_Friction` | 0.0-100.0 | The friction of Langevin dynamics. Higher means more slow but stable, lower means fast but unstable. Recommend 10.0 - 20.0|
| `LanPaint_EarlyStop` | 0-10 | Stop LanPaint iteration before the final sampling step. Helps to remove artifacts in some cases. Recommend 1-5|
| `LanPaint_PromptMode` | Image First / Prompt First | Image First mode focuses on the image context, maybe ignore prompt. Prompt First mode focuses more on the prompt. |
For detailed descriptions of each parameter, simply hover your mouse over the corresponding input field to view tooltips with additional information.
### LanPaint Mask Blend
This node blends the original image with the inpainted image based on the mask. It is useful if you want the unmasked region to match the original image pixel perfectly.
## LanPaint KSampler (Advanced) Tuning Guide
For challenging inpainting tasks:
1️⃣ **Primary Adjustments**:
- Decrease **LanPaint_endsigma** increase **LanPaint_NumSteps** (thinking iterations) if the inpainted area is not seamless.
1️⃣ **Boost Quality**
Increase **total number of sampling steps** (very important!), **LanPaint_NumSteps** (thinking iterations) or **LanPaint_Lambda** if the inpainted result does not meet your expectations.
2️⃣ **Secondary Tweaks**:
- Boost **LanPaint_Lambda** (bidirectional guidance scale) will force the masked/unmasked region to align more closely.
- If the output is blurry, increase **LanPaint_endsigma** to turn off thinking at the end of denoising. OR decrease **LanPaint_StepSize** to decrease thinking step size.
- If prompt is not that important, try increase **LanPaint_cfg_BIG**(cfg scale used for unmasked region, default -0.5 ) to 8 for better inpainting.
2️⃣ **Boost Speed**
Decrease **LanPaint_NumSteps** to accelerate generation! If you want better results but still need fewer steps, consider:
- **Increasing LanPaint_StepSize** to speed up the thinking process.
- **Decreasing LanPaint_Friction** to make the Langevin dynamics converges more faster.
3️⃣ **Balance Speed vs Stability**:
- Reduce **LanPaint_Friction** to prioritize faster results with fewer "thinking" steps (*may risk instability*).
- Increase **LanPaint_Tamed** (noise normalization onto a sphere) or **LanPaint_Alpha** (constraint the friction of underdamped Langevin dynamics) to suppress artifacts like blurry/wired texture.
3️⃣ **Fix Unstability**:
If you find the results have wired texture, try
- Reduce **LanPaint_Friction** to make the Langevin dynamics more stable.
- Reduce **LanPaint_StepSize** to use smaller step size.
- Reduce **LanPaint_Beta** if you are using a high lambda value.
⚠️ **Notes**:
- Optimal parameters vary depending on the **model** and the **size of the inpainting area**.
- For effective tuning, **fix the seed** and adjust parameters incrementally while observing the results. This helps isolate the impact of each setting. Better to do it with a batche of images to avoid overfitting on a single image.
## Community Showcase [](#community-showcase-)
Discover how the community is using LanPaint! Here are some user-created tutorials:
- [Ai绘画进阶148-三大王炸!庆祝高允贞出道6周年!T8即将直播?当AI绘画学会深度思考?!万能修复神器LanPaint,万物皆可修!-T8 Comfyui教程](https://www.youtube.com/watch?v=Z4DSTv3UPJo)
- [Ai绘画进阶151-真相了!T8竟是个AI?!LanPaint进阶(二),人物一致性,多视角实验性测试,新参数讲解,工作流分享-T8 Comfyui教程](https://www.youtube.com/watch?v=landiRhvF3k)
- [重绘和三视图角色一致性解决新方案!LanPaint节点尝试](https://www.youtube.com/watch?v=X0WbXdm6FA0)
- [ComfyUI: HiDream with Perturbation Upscale, LanPaint Inpainting (Workflow Tutorial)](https://www.youtube.com/watch?v=2-mGe4QVIIw&t=2785s)
- [ComfyUI必备LanPaint插件超详细使用教程](https://plugin.aix.ink/archives/lanpaint)
Submit a PR to add your tutorial/video here, or open an [Issue](https://github.com/scraed/LanPaint/issues) with details!
## Updates
- 2025/08/08
- Add Qwen image support
- 2025/06/21
- Update the algorithm with enhanced stability and outpaint performance.
- Add outpaint example
- Supports Sampler Custom (Thanks to [MINENEMA](https://github.com/MINENEMA))
- 2025/06/04
- Add more sampler support.
- Add early stopping to advanced sampler.
- 2025/05/28
- Major update on the Langevin solver. It is now much faster and more stable.
- Greatly simplified the parameters for advanced sampler.
- Fix performance issue on Flux and SD 3.5
- 2025/04/16
- Added Primary HiDream support
- 2025/03/22
- Added Primary Flux support
- Added Tease Mode
- 2025/03/10
- LanPaint has received a major update! All examples now use the LanPaint K Sampler, offering a simplified interface with enhanced performance and stability.
- 2025/03/06:
- Bug Fix for str not callable error and unpack error. Big thanks to [jamesWalker55](https://github.com/jamesWalker55) and [EricBCoding](https://github.com/EricBCoding).
## ToDo
- SD 3.5 also have problems
- Try Implement Detailer
## Contribute
- 2025/03/06: Bug Fix for str not callable error and unpack error. Big thanks to [jamesWalker55](https://github.com/jamesWalker55) and [EricBCoding](https://github.com/EricBCoding).
Help us improve LanPaint! 🚀 **Report bugs**, share **example cases**, or contribute your **personal parameter settings** to benefit the community.
- ~~Provide inference code on without GUI.~~ Check our local Python benchmark code [LanPaintBench](https://github.com/scraed/LanPaintBench).
## Citation
Binary file not shown.

After

Width:  |  Height:  |  Size: 674 KiB

File diff suppressed because it is too large Load Diff
Binary file not shown.

After

Width:  |  Height:  |  Size: 921 KiB

+806
View File
@@ -0,0 +1,806 @@
{
"id": "11cce4ab-536b-4f42-a95c-0be437d04ace",
"revision": 0,
"last_node_id": 128,
"last_link_id": 338,
"nodes": [
{
"id": 74,
"type": "LanPaint_KSampler",
"pos": [
276.219970703125,
179.55892944335938
],
"size": [
388.97625732421875,
572
],
"flags": {},
"order": 10,
"mode": 0,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 325
},
{
"name": "positive",
"type": "CONDITIONING",
"link": 323
},
{
"name": "negative",
"type": "CONDITIONING",
"link": 324
},
{
"name": "latent_image",
"type": "LATENT",
"link": 332
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"slot_index": 0,
"links": [
334
]
}
],
"properties": {
"cnr_id": "LanPaint",
"ver": "56bd6c04e89124cd06682b304245d6ddf8b20522",
"Node name for S&R": "LanPaint_KSampler"
},
"widgets_values": [
0,
"fixed",
20,
4,
"euler",
"simple",
1,
5,
"Image First",
"LanPaint KSampler. For more info, visit https://github.com/scraed/LanPaint. If you find it useful, please give a star ⭐️!"
]
},
{
"id": 113,
"type": "SaveImage",
"pos": [
807.1268310546875,
868.395263671875
],
"size": [
311.2532653808594,
484.7096252441406
],
"flags": {},
"order": 13,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 338
}
],
"outputs": [],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23"
},
"widgets_values": [
"ComfyUI"
]
},
{
"id": 117,
"type": "CLIPLoader",
"pos": [
-824.4296875,
177.9814910888672
],
"size": [
330,
110
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "CLIP",
"type": "CLIP",
"slot_index": 0,
"links": [
318,
319
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.48",
"Node name for S&R": "CLIPLoader",
"models": [
{
"name": "qwen_2.5_vl_7b_fp8_scaled.safetensors",
"url": "https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/text_encoders/qwen_2.5_vl_7b_fp8_scaled.safetensors",
"directory": "text_encoders"
}
],
"enableTabs": false,
"tabWidth": 65,
"tabXOffset": 10,
"hasSecondTab": false,
"secondTabText": "Send Back",
"secondTabOffset": 80,
"secondTabWidth": 65,
"widget_ue_connectable": {}
},
"widgets_values": [
"qwen_2.5_vl_7b_fp8_scaled.safetensors",
"qwen_image",
"default"
]
},
{
"id": 118,
"type": "VAELoader",
"pos": [
-824.4296875,
327.9817199707031
],
"size": [
330,
60
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "VAE",
"type": "VAE",
"slot_index": 0,
"links": [
329,
335
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.48",
"Node name for S&R": "VAELoader",
"models": [
{
"name": "qwen_image_vae.safetensors",
"url": "https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/vae/qwen_image_vae.safetensors",
"directory": "vae"
}
],
"enableTabs": false,
"tabWidth": 65,
"tabXOffset": 10,
"hasSecondTab": false,
"secondTabText": "Send Back",
"secondTabOffset": 80,
"secondTabWidth": 65,
"widget_ue_connectable": {}
},
"widgets_values": [
"qwen_image_vae.safetensors"
]
},
{
"id": 121,
"type": "CLIPTextEncode",
"pos": [
-454.4298095703125,
247.9816436767578
],
"size": [
425.27801513671875,
180.6060791015625
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 319
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"slot_index": 0,
"links": [
324
]
}
],
"title": "CLIP Text Encode (Negative Prompt)",
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.48",
"Node name for S&R": "CLIPTextEncode",
"enableTabs": false,
"tabWidth": 65,
"tabXOffset": 10,
"hasSecondTab": false,
"secondTabText": "Send Back",
"secondTabOffset": 80,
"secondTabWidth": 65,
"widget_ue_connectable": {}
},
"widgets_values": [
" low quality, bad anatomy, extra digits, missing digits, extra limbs, missing limbs"
],
"color": "#322",
"bgcolor": "#533"
},
{
"id": 122,
"type": "ModelSamplingAuraFlow",
"pos": [
-34.14249038696289,
-43.64523696899414
],
"size": [
300,
58
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 320
}
],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
325
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.48",
"Node name for S&R": "ModelSamplingAuraFlow",
"enableTabs": false,
"tabWidth": 65,
"tabXOffset": 10,
"hasSecondTab": false,
"secondTabText": "Send Back",
"secondTabOffset": 80,
"secondTabWidth": 65,
"widget_ue_connectable": {}
},
"widgets_values": [
3.5
]
},
{
"id": 119,
"type": "UNETLoader",
"pos": [
-824.4296875,
37.98154830932617
],
"size": [
330,
90
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"slot_index": 0,
"links": [
320
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.48",
"Node name for S&R": "UNETLoader",
"models": [
{
"name": "qwen_image_fp8_e4m3fn.safetensors",
"url": "https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/diffusion_models/qwen_image_fp8_e4m3fn.safetensors",
"directory": "diffusion_models"
}
],
"enableTabs": false,
"tabWidth": 65,
"tabXOffset": 10,
"hasSecondTab": false,
"secondTabText": "Send Back",
"secondTabOffset": 80,
"secondTabWidth": 65,
"widget_ue_connectable": {}
},
"widgets_values": [
"qwen_image_fp8_e4m3fn.safetensors",
"default"
]
},
{
"id": 124,
"type": "VAEEncode",
"pos": [
-530.8583984375,
708.7066650390625
],
"size": [
210,
46
],
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "pixels",
"type": "IMAGE",
"link": 326
},
{
"name": "vae",
"type": "VAE",
"link": 329
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"slot_index": 0,
"links": [
327
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23",
"Node name for S&R": "VAEEncode"
},
"widgets_values": []
},
{
"id": 125,
"type": "SetLatentNoiseMask",
"pos": [
-234.4196014404297,
705.1629638671875
],
"size": [
264.5999755859375,
46
],
"flags": {},
"order": 9,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 327
},
{
"name": "mask",
"type": "MASK",
"link": 328
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"slot_index": 0,
"links": [
332
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23",
"Node name for S&R": "SetLatentNoiseMask"
},
"widgets_values": []
},
{
"id": 127,
"type": "LanPaint_MaskBlend",
"pos": [
405.3840637207031,
938.8120727539062
],
"size": [
210,
98
],
"flags": {},
"order": 12,
"mode": 0,
"inputs": [
{
"name": "image1",
"type": "IMAGE",
"link": 336
},
{
"name": "image2",
"type": "IMAGE",
"link": 333
},
{
"name": "mask",
"type": "MASK",
"link": 337
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
338
]
}
],
"properties": {
"cnr_id": "LanPaint",
"ver": "4d3d5d17f0105b673df92da5b084cce567c9c712",
"Node name for S&R": "LanPaint_MaskBlend"
},
"widgets_values": [
9
]
},
{
"id": 126,
"type": "VAEDecode",
"pos": [
115.07575225830078,
878.4630737304688
],
"size": [
210,
46
],
"flags": {},
"order": 11,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 334
},
{
"name": "vae",
"type": "VAE",
"link": 335
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"slot_index": 0,
"links": [
333
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23",
"Node name for S&R": "VAEDecode"
},
"widgets_values": []
},
{
"id": 123,
"type": "LoadImage",
"pos": [
-543.5358276367188,
851.759765625
],
"size": [
262.12347412109375,
487.22296142578125
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"slot_index": 0,
"links": [
326,
336
]
},
{
"name": "MASK",
"type": "MASK",
"slot_index": 1,
"links": [
328,
337
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23",
"Node name for S&R": "LoadImage"
},
"widgets_values": [
"Masked_Load_Me_in_Loader (7).png",
"image"
]
},
{
"id": 120,
"type": "CLIPTextEncode",
"pos": [
-454.86480712890625,
41.89194869995117
],
"size": [
422.84503173828125,
164.31304931640625
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 318
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"slot_index": 0,
"links": [
323
]
}
],
"title": "CLIP Text Encode (Positive Prompt)",
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.48",
"Node name for S&R": "CLIPTextEncode",
"enableTabs": false,
"tabWidth": 65,
"tabXOffset": 10,
"hasSecondTab": false,
"secondTabText": "Send Back",
"secondTabOffset": 80,
"secondTabWidth": 65,
"widget_ue_connectable": {}
},
"widgets_values": [
"Cyberpunk-style Einstein portrait: He wears a sleek black coat with glowing cyan circuit patterns, silver-rimmed cybernetic glasses (lenses display faint data streams), and his hair has subtle neon blue highlights. His expression is calm, with a faint smile. Behind him: a dark, rain-washed cybercity backdrop—towering skyscrapers with flickering holographic ads, wet pavement reflecting neon pink/magenta lights. In front of him: giant, glowing white 3D text of \"LanPaint\", with electric blue energy pulses swirling around the equation. Cinematic lighting, hyper-detailed textures, rain droplets visible in the air."
],
"color": "#232",
"bgcolor": "#353"
},
{
"id": 128,
"type": "MarkdownNote",
"pos": [
715.929931640625,
371.1071472167969
],
"size": [
300,
190
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [],
"title": "KSampler settings",
"properties": {},
"widgets_values": [
"Decrease **LanPaint_NumSteps** for faster generation. \n"
],
"color": "#432",
"bgcolor": "#653"
}
],
"links": [
[
318,
117,
0,
120,
0,
"CLIP"
],
[
319,
117,
0,
121,
0,
"CLIP"
],
[
320,
119,
0,
122,
0,
"MODEL"
],
[
323,
120,
0,
74,
1,
"CONDITIONING"
],
[
324,
121,
0,
74,
2,
"CONDITIONING"
],
[
325,
122,
0,
74,
0,
"MODEL"
],
[
326,
123,
0,
124,
0,
"IMAGE"
],
[
327,
124,
0,
125,
0,
"LATENT"
],
[
328,
123,
1,
125,
1,
"MASK"
],
[
329,
118,
0,
124,
1,
"VAE"
],
[
332,
125,
0,
74,
3,
"LATENT"
],
[
333,
126,
0,
127,
1,
"IMAGE"
],
[
334,
74,
0,
126,
0,
"LATENT"
],
[
335,
118,
0,
126,
1,
"VAE"
],
[
336,
123,
0,
127,
0,
"IMAGE"
],
[
337,
123,
1,
127,
2,
"MASK"
],
[
338,
127,
0,
113,
0,
"IMAGE"
]
],
"groups": [],
"config": {},
"extra": {
"ds": {
"scale": 0.7162766973052638,
"offset": [
1084.0595529886727,
5.084234529384386
]
},
"frontendVersion": "1.25.10",
"node_versions": {
"comfy-core": "0.3.18",
"LanPaint": "0f509469ed2cd60c6032f739e282aad5dfc06166"
},
"groupNodes": {}
},
"version": 0.4
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 810 KiB

+909
View File
@@ -0,0 +1,909 @@
{
"id": "11cce4ab-536b-4f42-a95c-0be437d04ace",
"revision": 0,
"last_node_id": 136,
"last_link_id": 351,
"nodes": [
{
"id": 74,
"type": "LanPaint_KSampler",
"pos": [
276.219970703125,
179.55892944335938
],
"size": [
388.97625732421875,
572
],
"flags": {},
"order": 12,
"mode": 0,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 336
},
{
"name": "positive",
"type": "CONDITIONING",
"link": 334
},
{
"name": "negative",
"type": "CONDITIONING",
"link": 335
},
{
"name": "latent_image",
"type": "LATENT",
"link": 345
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"slot_index": 0,
"links": [
347
]
}
],
"properties": {
"cnr_id": "LanPaint",
"ver": "56bd6c04e89124cd06682b304245d6ddf8b20522",
"Node name for S&R": "LanPaint_KSampler"
},
"widgets_values": [
0,
"fixed",
20,
4,
"euler",
"simple",
1,
5,
"Image First",
"LanPaint KSampler. For more info, visit https://github.com/scraed/LanPaint. If you find it useful, please give a star ⭐️!"
]
},
{
"id": 113,
"type": "SaveImage",
"pos": [
807.1268310546875,
868.395263671875
],
"size": [
311.2532653808594,
484.7096252441406
],
"flags": {},
"order": 15,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 351
}
],
"outputs": [],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23"
},
"widgets_values": [
"ComfyUI"
]
},
{
"id": 123,
"type": "CLIPLoader",
"pos": [
-730.1405029296875,
183.70140075683594
],
"size": [
330,
110
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "CLIP",
"type": "CLIP",
"slot_index": 0,
"links": [
329,
330
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.48",
"Node name for S&R": "CLIPLoader",
"models": [
{
"name": "qwen_2.5_vl_7b_fp8_scaled.safetensors",
"url": "https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/text_encoders/qwen_2.5_vl_7b_fp8_scaled.safetensors",
"directory": "text_encoders"
}
],
"enableTabs": false,
"tabWidth": 65,
"tabXOffset": 10,
"hasSecondTab": false,
"secondTabText": "Send Back",
"secondTabOffset": 80,
"secondTabWidth": 65,
"widget_ue_connectable": {}
},
"widgets_values": [
"qwen_2.5_vl_7b_fp8_scaled.safetensors",
"qwen_image",
"default"
]
},
{
"id": 124,
"type": "VAELoader",
"pos": [
-730.1405029296875,
333.70147705078125
],
"size": [
330,
60
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "VAE",
"type": "VAE",
"slot_index": 0,
"links": [
342,
348
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.48",
"Node name for S&R": "VAELoader",
"models": [
{
"name": "qwen_image_vae.safetensors",
"url": "https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/vae/qwen_image_vae.safetensors",
"directory": "vae"
}
],
"enableTabs": false,
"tabWidth": 65,
"tabXOffset": 10,
"hasSecondTab": false,
"secondTabText": "Send Back",
"secondTabOffset": 80,
"secondTabWidth": 65,
"widget_ue_connectable": {}
},
"widgets_values": [
"qwen_image_vae.safetensors"
]
},
{
"id": 125,
"type": "UNETLoader",
"pos": [
-730.1405029296875,
43.7014274597168
],
"size": [
330,
90
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"slot_index": 0,
"links": [
331
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.48",
"Node name for S&R": "UNETLoader",
"models": [
{
"name": "qwen_image_fp8_e4m3fn.safetensors",
"url": "https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/diffusion_models/qwen_image_fp8_e4m3fn.safetensors",
"directory": "diffusion_models"
}
],
"enableTabs": false,
"tabWidth": 65,
"tabXOffset": 10,
"hasSecondTab": false,
"secondTabText": "Send Back",
"secondTabOffset": 80,
"secondTabWidth": 65,
"widget_ue_connectable": {}
},
"widgets_values": [
"qwen_image_fp8_e4m3fn.safetensors",
"default"
]
},
{
"id": 127,
"type": "CLIPTextEncode",
"pos": [
-360.1405029296875,
253.70147705078125
],
"size": [
425.27801513671875,
180.6060791015625
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 330
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"slot_index": 0,
"links": [
335
]
}
],
"title": "CLIP Text Encode (Negative Prompt)",
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.48",
"Node name for S&R": "CLIPTextEncode",
"enableTabs": false,
"tabWidth": 65,
"tabXOffset": 10,
"hasSecondTab": false,
"secondTabText": "Send Back",
"secondTabOffset": 80,
"secondTabWidth": 65,
"widget_ue_connectable": {}
},
"widgets_values": [
" low quality, bad anatomy, extra digits, missing digits, extra limbs, missing limbs"
],
"color": "#322",
"bgcolor": "#533"
},
{
"id": 128,
"type": "ModelSamplingAuraFlow",
"pos": [
60.14683151245117,
-37.92536544799805
],
"size": [
300,
58
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 331
}
],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"links": [
336
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.48",
"Node name for S&R": "ModelSamplingAuraFlow",
"enableTabs": false,
"tabWidth": 65,
"tabXOffset": 10,
"hasSecondTab": false,
"secondTabText": "Send Back",
"secondTabOffset": 80,
"secondTabWidth": 65,
"widget_ue_connectable": {}
},
"widgets_values": [
3.5
]
},
{
"id": 126,
"type": "CLIPTextEncode",
"pos": [
-360.57550048828125,
47.6118278503418
],
"size": [
422.84503173828125,
164.31304931640625
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 329
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"slot_index": 0,
"links": [
334
]
}
],
"title": "CLIP Text Encode (Positive Prompt)",
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.48",
"Node name for S&R": "CLIPTextEncode",
"enableTabs": false,
"tabWidth": 65,
"tabXOffset": 10,
"hasSecondTab": false,
"secondTabText": "Send Back",
"secondTabOffset": 80,
"secondTabWidth": 65,
"widget_ue_connectable": {}
},
"widgets_values": [
"Cyberpunk-inspired portrait of a beautiful young woman with ethereal features: She has long, flowing silver hair with glowing purple neon streaks, wearing a form-fitting black leather jacket adorned with holographic circuit designs in electric blue, and augmented reality earrings that project faint digital fractals. Her eyes are piercing emerald green with subtle cybernetic enhancements showing data overlays, and her expression is mysterious yet alluring, with a subtle smirk. Behind her: a foggy, neon-lit futuristic alleyway—crumbling brick walls covered in vibrant graffiti and flickering LED signs, puddles on the ground reflecting turquoise and violet lights from overhead drones. Dramatic volumetric lighting, ultra-realistic details, mist particles in the air."
],
"color": "#232",
"bgcolor": "#353"
},
{
"id": 130,
"type": "ImagePadForOutpaint",
"pos": [
-437.0110168457031,
895.0103759765625
],
"size": [
210,
174
],
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "image",
"type": "IMAGE",
"link": 337
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
338,
349
]
},
{
"name": "MASK",
"type": "MASK",
"links": [
339
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.39",
"Node name for S&R": "ImagePadForOutpaint",
"widget_ue_connectable": {}
},
"widgets_values": [
200,
200,
200,
200,
20
]
},
{
"id": 131,
"type": "VAEEncode",
"pos": [
-455.7289123535156,
723.4658203125
],
"size": [
210,
46
],
"flags": {},
"order": 9,
"mode": 0,
"inputs": [
{
"name": "pixels",
"type": "IMAGE",
"link": 338
},
{
"name": "vae",
"type": "VAE",
"link": 342
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"slot_index": 0,
"links": [
340
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23",
"Node name for S&R": "VAEEncode"
},
"widgets_values": []
},
{
"id": 132,
"type": "ThresholdMask",
"pos": [
-67.46508026123047,
903.83642578125
],
"size": [
270,
58
],
"flags": {},
"order": 10,
"mode": 0,
"inputs": [
{
"name": "mask",
"type": "MASK",
"link": 339
}
],
"outputs": [
{
"name": "MASK",
"type": "MASK",
"links": [
341,
350
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.38",
"Node name for S&R": "ThresholdMask"
},
"widgets_values": [
0.010000000000000002
]
},
{
"id": 133,
"type": "SetLatentNoiseMask",
"pos": [
-152.71241760253906,
712.6437377929688
],
"size": [
264.5999755859375,
46
],
"flags": {},
"order": 11,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 340
},
{
"name": "mask",
"type": "MASK",
"link": 341
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"slot_index": 0,
"links": [
345
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23",
"Node name for S&R": "SetLatentNoiseMask"
},
"widgets_values": []
},
{
"id": 129,
"type": "LoadImage",
"pos": [
-789.0392456054688,
827.515380859375
],
"size": [
295,
399
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
337
]
},
{
"name": "MASK",
"type": "MASK",
"links": []
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.27",
"Node name for S&R": "LoadImage",
"widget_ue_connectable": {}
},
"widgets_values": [
"ComfyUI_07699_.png",
"image"
]
},
{
"id": 134,
"type": "VAEDecode",
"pos": [
256.8516540527344,
900.2234497070312
],
"size": [
210,
46
],
"flags": {},
"order": 13,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 347
},
{
"name": "vae",
"type": "VAE",
"link": 348
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"slot_index": 0,
"links": [
346
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23",
"Node name for S&R": "VAEDecode"
},
"widgets_values": []
},
{
"id": 135,
"type": "LanPaint_MaskBlend",
"pos": [
547.1599731445312,
960.5724487304688
],
"size": [
210,
98
],
"flags": {},
"order": 14,
"mode": 0,
"inputs": [
{
"name": "image1",
"type": "IMAGE",
"link": 349
},
{
"name": "image2",
"type": "IMAGE",
"link": 346
},
{
"name": "mask",
"type": "MASK",
"link": 350
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
351
]
}
],
"properties": {
"cnr_id": "LanPaint",
"ver": "4d3d5d17f0105b673df92da5b084cce567c9c712",
"Node name for S&R": "LanPaint_MaskBlend"
},
"widgets_values": [
9
]
},
{
"id": 136,
"type": "MarkdownNote",
"pos": [
728.9194946289062,
332.5254821777344
],
"size": [
300,
190
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [],
"outputs": [],
"title": "KSampler settings",
"properties": {},
"widgets_values": [
"Decrease **LanPaint_NumSteps** for faster generation. \n"
],
"color": "#432",
"bgcolor": "#653"
}
],
"links": [
[
329,
123,
0,
126,
0,
"CLIP"
],
[
330,
123,
0,
127,
0,
"CLIP"
],
[
331,
125,
0,
128,
0,
"MODEL"
],
[
334,
126,
0,
74,
1,
"CONDITIONING"
],
[
335,
127,
0,
74,
2,
"CONDITIONING"
],
[
336,
128,
0,
74,
0,
"MODEL"
],
[
337,
129,
0,
130,
0,
"IMAGE"
],
[
338,
130,
0,
131,
0,
"IMAGE"
],
[
339,
130,
1,
132,
0,
"MASK"
],
[
340,
131,
0,
133,
0,
"LATENT"
],
[
341,
132,
0,
133,
1,
"MASK"
],
[
342,
124,
0,
131,
1,
"VAE"
],
[
345,
133,
0,
74,
3,
"LATENT"
],
[
346,
134,
0,
135,
1,
"IMAGE"
],
[
347,
74,
0,
134,
0,
"LATENT"
],
[
348,
124,
0,
134,
1,
"VAE"
],
[
349,
130,
0,
135,
0,
"IMAGE"
],
[
350,
132,
0,
135,
2,
"MASK"
],
[
351,
135,
0,
113,
0,
"IMAGE"
]
],
"groups": [],
"config": {},
"extra": {
"ds": {
"scale": 0.6087272163919795,
"offset": [
1617.0504979084035,
167.0583828525509
]
},
"frontendVersion": "1.25.10",
"node_versions": {
"comfy-core": "0.3.18",
"LanPaint": "0f509469ed2cd60c6032f739e282aad5dfc06166"
},
"groupNodes": {}
},
"version": 0.4
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

+735
View File
@@ -0,0 +1,735 @@
{
"id": "26fb90cb-eb4a-422e-97d0-8b84dd6c3302",
"revision": 0,
"last_node_id": 75,
"last_link_id": 192,
"nodes": [
{
"id": 6,
"type": "CLIPTextEncode",
"pos": [
333.06903076171875,
249.68698120117188
],
"size": [
422.84503173828125,
164.31304931640625
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 81
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"slot_index": 0,
"links": [
184
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23",
"Node name for S&R": "CLIPTextEncode"
},
"widgets_values": [
"1girl, blue shirt, masterpiece, high score, great score, absurdres"
]
},
{
"id": 7,
"type": "CLIPTextEncode",
"pos": [
335.06903076171875,
462.68701171875
],
"size": [
425.27801513671875,
180.6060791015625
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [
{
"name": "clip",
"type": "CLIP",
"link": 82
}
],
"outputs": [
{
"name": "CONDITIONING",
"type": "CONDITIONING",
"slot_index": 0,
"links": [
185
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23",
"Node name for S&R": "CLIPTextEncode"
},
"widgets_values": [
"lowres, bad anatomy, bad hands, text, error, missing finger, extra digits, fewer digits, cropped, worst quality, low quality, low score, bad score, average score, signature, watermark, username, blurry, nude, NSFW"
]
},
{
"id": 29,
"type": "CheckpointLoaderSimple",
"pos": [
-62.627220153808594,
407.01416015625
],
"size": [
315,
98
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "MODEL",
"type": "MODEL",
"slot_index": 0,
"links": [
183
]
},
{
"name": "CLIP",
"type": "CLIP",
"slot_index": 1,
"links": [
81,
82
]
},
{
"name": "VAE",
"type": "VAE",
"slot_index": 2,
"links": [
84,
157
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23",
"Node name for S&R": "CheckpointLoaderSimple"
},
"widgets_values": [
"animagineXL40_v4Opt.safetensors"
]
},
{
"id": 48,
"type": "SaveImage",
"pos": [
1091.036376953125,
1158.526611328125
],
"size": [
311.2532653808594,
484.7096252441406
],
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 103
}
],
"outputs": [],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23"
},
"widgets_values": [
"ComfyUI"
]
},
{
"id": 8,
"type": "VAEDecode",
"pos": [
1211.46484375,
1065.318359375
],
"size": [
210,
46
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 187
},
{
"name": "vae",
"type": "VAE",
"link": 84
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"slot_index": 0,
"links": [
103,
189
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23",
"Node name for S&R": "VAEDecode"
},
"widgets_values": []
},
{
"id": 20,
"type": "LoadImage",
"pos": [
45.70227813720703,
1147.2928466796875
],
"size": [
262.12347412109375,
487.22296142578125
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"slot_index": 0,
"links": [
156,
190
]
},
{
"name": "MASK",
"type": "MASK",
"slot_index": 1,
"links": [
155,
191
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23",
"Node name for S&R": "LoadImage"
},
"widgets_values": [
"clipspace/clipspace-mask-2620525.399999976.png [input]",
"image"
]
},
{
"id": 75,
"type": "SaveImage",
"pos": [
1845.7059326171875,
1044.06201171875
],
"size": [
311.2532653808594,
484.7096252441406
],
"flags": {},
"order": 10,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 188
}
],
"outputs": [],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23"
},
"widgets_values": [
"ComfyUI"
]
},
{
"id": 73,
"type": "LanPaint_KSampler",
"pos": [
996.4982299804688,
299.599365234375
],
"size": [
413.6495666503906,
572
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{
"name": "model",
"type": "MODEL",
"link": 183
},
{
"name": "positive",
"type": "CONDITIONING",
"link": 184
},
{
"name": "negative",
"type": "CONDITIONING",
"link": 185
},
{
"name": "latent_image",
"type": "LATENT",
"link": 186
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"slot_index": 0,
"links": [
187
]
}
],
"properties": {
"cnr_id": "LanPaint",
"ver": "56bd6c04e89124cd06682b304245d6ddf8b20522",
"Node name for S&R": "LanPaint_KSampler"
},
"widgets_values": [
0,
"fixed",
30,
5,
"euler",
"karras",
1,
5,
"Image First",
"LanPaint KSampler. For more info, visit https://github.com/scraed/LanPaint. If you find it useful, please give a star ⭐️!"
]
},
{
"id": 74,
"type": "LanPaint_MaskBlend",
"pos": [
1512.2730712890625,
1176.3270263671875
],
"size": [
210,
98
],
"flags": {},
"order": 9,
"mode": 0,
"inputs": [
{
"name": "image1",
"type": "IMAGE",
"link": 190
},
{
"name": "image2",
"type": "IMAGE",
"link": 189
},
{
"name": "mask",
"type": "MASK",
"link": 191
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
188
]
}
],
"properties": {
"cnr_id": "LanPaint",
"ver": "4d3d5d17f0105b673df92da5b084cce567c9c712",
"Node name for S&R": "LanPaint_MaskBlend"
},
"widgets_values": [
9
]
},
{
"id": 66,
"type": "SetLatentNoiseMask",
"pos": [
480.30780029296875,
821.3880004882812
],
"size": [
264.5999755859375,
46
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 192
},
{
"name": "mask",
"type": "MASK",
"link": 155
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"slot_index": 0,
"links": [
186
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23",
"Node name for S&R": "SetLatentNoiseMask"
},
"widgets_values": []
},
{
"id": 65,
"type": "VAEEncode",
"pos": [
206.43878173828125,
818.9985961914062
],
"size": [
210,
46
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [
{
"name": "pixels",
"type": "IMAGE",
"link": 156
},
{
"name": "vae",
"type": "VAE",
"link": 157
}
],
"outputs": [
{
"name": "LATENT",
"type": "LATENT",
"slot_index": 0,
"links": [
192
]
}
],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.23",
"Node name for S&R": "VAEEncode"
},
"widgets_values": []
}
],
"links": [
[
81,
29,
1,
6,
0,
"CLIP"
],
[
82,
29,
1,
7,
0,
"CLIP"
],
[
84,
29,
2,
8,
1,
"VAE"
],
[
103,
8,
0,
48,
0,
"IMAGE"
],
[
155,
20,
1,
66,
1,
"MASK"
],
[
156,
20,
0,
65,
0,
"IMAGE"
],
[
157,
29,
2,
65,
1,
"VAE"
],
[
183,
29,
0,
73,
0,
"MODEL"
],
[
184,
6,
0,
73,
1,
"CONDITIONING"
],
[
185,
7,
0,
73,
2,
"CONDITIONING"
],
[
186,
66,
0,
73,
3,
"LATENT"
],
[
187,
73,
0,
8,
0,
"LATENT"
],
[
188,
74,
0,
75,
0,
"IMAGE"
],
[
189,
8,
0,
74,
1,
"IMAGE"
],
[
190,
20,
0,
74,
0,
"IMAGE"
],
[
191,
20,
1,
74,
2,
"MASK"
],
[
192,
65,
0,
66,
0,
"LATENT"
]
],
"groups": [
{
"id": 1,
"title": "Mask image for inpainting.",
"bounding": [
36.04227828979492,
989.7313232421875,
278.89093017578125,
669.3414916992188
],
"color": "#3f789e",
"font_size": 24,
"flags": {}
},
{
"id": 2,
"title": "Convert Latents for LanPaint",
"bounding": [
286.0640563964844,
714.343505859375,
489.16796875,
197.81044006347656
],
"color": "#3f789e",
"font_size": 24,
"flags": {}
},
{
"id": 3,
"title": "Load Model and Set Prompts",
"bounding": [
-78.9311294555664,
176.08712768554688,
1006.1721801757812,
514.258544921875
],
"color": "#3f789e",
"font_size": 24,
"flags": {}
},
{
"id": 4,
"title": "Inpaint with the LanPaint KSampler",
"bounding": [
960.8922729492188,
179.17588806152344,
474.8909606933594,
630.4742431640625
],
"color": "#3f789e",
"font_size": 24,
"flags": {}
},
{
"id": 5,
"title": "LanPaint OutPut",
"bounding": [
1085.6029052734375,
994.0775756835938,
345.4561767578125,
669.4969482421875
],
"color": "#3f789e",
"font_size": 24,
"flags": {}
},
{
"id": 11,
"title": "LanPaint",
"bounding": [
-262.59381103515625,
140.46656799316406,
1737.328857421875,
797.4443359375
],
"color": "#3f789e",
"font_size": 24,
"flags": {}
},
{
"id": 13,
"title": "Paste back the original to preserve it exactly, if you want",
"bounding": [
1491.586181640625,
984.3547973632812,
749.66455078125,
637.360595703125
],
"color": "#3f789e",
"font_size": 24,
"flags": {}
}
],
"config": {},
"extra": {
"ds": {
"scale": 0.6209213230591556,
"offset": [
850.0803535605011,
86.6432096141053
]
},
"frontendVersion": "1.25.10",
"node_versions": {
"comfy-core": "0.3.18",
"LanPaint": "0f509469ed2cd60c6032f739e282aad5dfc06166"
}
},
"version": 0.4
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 683 KiB

File diff suppressed because it is too large Load Diff
Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.5 MiB

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.9 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.7 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 830 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.9 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.2 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.1 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.2 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.5 MiB

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.2 MiB

After

Width:  |  Height:  |  Size: 1.2 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.0 MiB

After

Width:  |  Height:  |  Size: 1012 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.1 MiB

After

Width:  |  Height:  |  Size: 1.0 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.1 MiB

After

Width:  |  Height:  |  Size: 1.1 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.2 MiB

After

Width:  |  Height:  |  Size: 1.2 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.3 MiB

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.4 MiB

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.4 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 696 KiB

After

Width:  |  Height:  |  Size: 702 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 654 KiB

After

Width:  |  Height:  |  Size: 644 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 351 KiB

After

Width:  |  Height:  |  Size: 357 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 356 KiB

After

Width:  |  Height:  |  Size: 349 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 392 KiB

After

Width:  |  Height:  |  Size: 379 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.1 MiB

After

Width:  |  Height:  |  Size: 397 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 539 KiB

After

Width:  |  Height:  |  Size: 549 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 601 KiB

After

Width:  |  Height:  |  Size: 606 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 467 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 407 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 782 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 260 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 301 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 292 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 399 KiB

+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "LanPaint"
version = "0.2.2"
version = "1.3.2"
description = "Achieve seamless inpainting results without needing a specialized inpainting model."
authors = [
{name = "LanPaint", email = "czhengac@connect.ust.hk"}
+173
View File
@@ -0,0 +1,173 @@
import torch
from .utils import *
from functools import partial
class LanPaint():
def __init__(self, Model, NSteps, Friction, Lambda, Beta, StepSize, IS_FLUX = False, IS_FLOW = False):
self.n_steps = NSteps
self.chara_lamb = Lambda
self.IS_FLUX = IS_FLUX
self.IS_FLOW = IS_FLOW
self.step_size = StepSize
self.inner_model = Model
self.friction = Friction
self.chara_beta = Beta
self.img_dim_size = None
def add_none_dims(self, array):
# Create a tuple with ':' for the first dimension and 'None' repeated num_nones times
index = (slice(None),) + (None,) * (self.img_dim_size-1)
return array[index]
def remove_none_dims(self, array):
# Create a tuple with ':' for the first dimension and 'None' repeated num_nones times
index = (slice(None),) + (0,) * (self.img_dim_size-1)
return array[index]
def __call__(self, x, latent_image, noise, sigma, latent_mask, current_times, model_options, seed, n_steps=None):
self.img_dim_size = len(x.shape)
self.latent_image = latent_image
self.noise = noise
if n_steps is None:
n_steps = self.n_steps
return self.LanPaint(x, sigma, latent_mask, current_times, n_steps, model_options, seed, self.IS_FLUX, self.IS_FLOW)
def LanPaint(self, x, sigma, latent_mask, current_times, n_steps, model_options, seed, IS_FLUX, IS_FLOW):
VE_Sigma, abt, Flow_t = current_times
step_size = self.step_size * (1 - abt)
step_size = self.add_none_dims(step_size)
# self.inner_model.inner_model.scale_latent_inpaint returns variance exploding x_t values
# This is the replace step
x = x * (1 - latent_mask) + self.inner_model.inner_model.scale_latent_inpaint(x=x, sigma=sigma, noise=self.noise, latent_image=self.latent_image)* latent_mask
if IS_FLUX or IS_FLOW:
x_t = x * ( self.add_none_dims(abt)**0.5 + (1-self.add_none_dims(abt))**0.5 )
else:
x_t = x / ( 1+self.add_none_dims(VE_Sigma)**2 )**0.5 # switch to variance perserving x_t values
############ LanPaint Iterations Start ###############
# after noise_scaling, noise = latent_image + noise * sigma, which is x_t in the variance exploding diffusion model notation for the known region.
args = None
for i in range(n_steps):
score_func = partial( self.score_model, y = self.latent_image, mask = latent_mask, abt = self.add_none_dims(abt), sigma = self.add_none_dims(VE_Sigma), tflow = self.add_none_dims(Flow_t), model_options = model_options, seed = seed )
x_t, args = self.langevin_dynamics(x_t, score_func , latent_mask, step_size , current_times, sigma_x = self.add_none_dims(self.sigma_x(abt)), sigma_y = self.add_none_dims(self.sigma_y(abt)), args = args)
if IS_FLUX or IS_FLOW:
x = x_t / ( self.add_none_dims(abt)**0.5 + (1-self.add_none_dims(abt))**0.5 )
else:
x = x_t * ( 1+self.add_none_dims(VE_Sigma)**2 )**0.5 # switch to variance perserving x_t values
############ LanPaint Iterations End ###############
# out is x_0
out, _ = self.inner_model(x, sigma, model_options=model_options, seed=seed)
out = out * (1-latent_mask) + self.latent_image * latent_mask
return out
def score_model(self, x_t, y, mask, abt, sigma, tflow, model_options, seed):
lamb = self.chara_lamb
if self.IS_FLUX or self.IS_FLOW:
# compute t for flow model, with a small epsilon compensating for numerical error.
x = x_t / ( abt**0.5 + (1-abt)**0.5 ) # switch to Gaussian flow matching
x_0, x_0_BIG = self.inner_model(x, self.remove_none_dims(tflow), model_options=model_options, seed=seed)
else:
x = x_t * ( 1+sigma**2 )**0.5 # switch to variance exploding
x_0, x_0_BIG = self.inner_model(x, self.remove_none_dims(sigma), model_options=model_options, seed=seed)
score_x = -(x_t - x_0)
score_y = - (1 + lamb) * ( x_t - y ) + lamb * (x_t - x_0_BIG)
return score_x * (1 - mask) + score_y * mask
def sigma_x(self, abt):
# the time scale for the x_t update
return abt**0
def sigma_y(self, abt):
beta = self.chara_beta * abt ** 0
return beta
def langevin_dynamics(self, x_t, score, mask, step_size, current_times, sigma_x=1, sigma_y=0, args=None):
# prepare the step size and time parameters
with torch.autocast(device_type=x_t.device.type, dtype=torch.float32):
step_sizes = self.prepare_step_size(current_times, step_size, sigma_x, sigma_y)
sigma, abt, dtx, dty, Gamma_x, Gamma_y, A_x, A_y, D_x, D_y = step_sizes
# print('mask',mask.device)
if torch.mean(dtx) <= 0.:
return x_t, args
# -------------------------------------------------------------------------
# Compute the Langevin dynamics update in variance perserving notation
# -------------------------------------------------------------------------
#x0 = self.x0_evalutation(x_t, score, sigma, args)
#C = abt**0.5 * x0 / (1-abt)
A = A_x * (1-mask) + A_y * mask
D = D_x * (1-mask) + D_y * mask
dt = dtx * (1-mask) + dty * mask
Gamma = Gamma_x * (1-mask) + Gamma_y * mask
def Coef_C(x_t):
x0 = self.x0_evalutation(x_t, score, sigma, args)
C = (abt**0.5 * x0 - x_t )/ (1-abt) + A * x_t
return C
def advance_time(x_t, v, dt, Gamma, A, C, D):
dtype = x_t.dtype
with torch.autocast(device_type=x_t.device.type, dtype=torch.float32):
osc = StochasticHarmonicOscillator(Gamma, A, C, D )
x_t, v = osc.dynamics(x_t, v, dt )
x_t = x_t.to(dtype)
v = v.to(dtype)
return x_t, v
if args is None:
#v = torch.zeros_like(x_t)
v = None
C = Coef_C(x_t)
#print(torch.squeeze(dtx), torch.squeeze(dty))
x_t, v = advance_time(x_t, v, dt, Gamma, A, C, D)
else:
v, C = args
x_t, v = advance_time(x_t, v, dt/2, Gamma, A, C, D)
C_new = Coef_C(x_t)
v = v + Gamma**0.5 * ( C_new - C) *dt
x_t, v = advance_time(x_t, v, dt/2, Gamma, A, C, D)
C = C_new
return x_t, (v, C)
def prepare_step_size(self, current_times, step_size, sigma_x, sigma_y):
# -------------------------------------------------------------------------
# Unpack current times parameters (sigma and abt)
sigma, abt, flow_t = current_times
sigma = self.add_none_dims(sigma)
abt = self.add_none_dims(abt)
# Compute time step (dtx, dty) for x and y branches.
dtx = 2 * step_size * sigma_x
dty = 2 * step_size * sigma_y
# -------------------------------------------------------------------------
# Define friction parameter Gamma_hat for each branch.
# Using dtx**0 provides a tensor of the proper device/dtype.
Gamma_hat_x = self.friction **2 * self.step_size * sigma_x / 0.1 * sigma**0
Gamma_hat_y = self.friction **2 * self.step_size * sigma_y / 0.1 * sigma**0
#print("Gamma_hat_x", torch.mean(Gamma_hat_x).item(), "Gamma_hat_y", torch.mean(Gamma_hat_y).item())
# adjust dt to match denoise-addnoise steps sizes
Gamma_hat_x /= 2.
Gamma_hat_y /= 2.
A_t_x = (1) / ( 1 - abt ) * dtx / 2
A_t_y = (1+self.chara_lamb) / ( 1 - abt ) * dty / 2
A_x = A_t_x / (dtx/2)
A_y = A_t_y / (dty/2)
Gamma_x = Gamma_hat_x / (dtx/2)
Gamma_y = Gamma_hat_y / (dty/2)
#D_x = (2 * (1 + sigma**2) )**0.5
#D_y = (2 * (1 + sigma**2) )**0.5
D_x = (2 * abt**0 )**0.5
D_y = (2 * abt**0 )**0.5
return sigma, abt, dtx/2, dty/2, Gamma_x, Gamma_y, A_x, A_y, D_x, D_y
def x0_evalutation(self, x_t, score, sigma, args):
x0 = x_t + score(x_t)
return x0
+294 -294
View File
@@ -9,15 +9,15 @@ from functools import partial
from comfy.utils import repeat_to_batch_size
from comfy.samplers import *
from comfy.model_base import ModelType
# Monkey patch comfy.samplers module by importing with absolute package path
#exec(inspect.getsource(comfy.samplers).replace("from .", "from comfy."))
from .utils import *
from .lanpaint import LanPaint
def reshape_mask(input_mask, output_shape):
dims = len(output_shape) - 2
scale_mode = "nearest-exact"
mask = torch.nn.functional.interpolate(input_mask, size=output_shape[2:], mode=scale_mode)
mask = torch.nn.functional.interpolate(input_mask, size=output_shape[-2:], mode=scale_mode)
if mask.shape[1] < output_shape[1]:
mask = mask.repeat((1, output_shape[1]) + (1,) * dims)[:,:output_shape[1]]
mask = repeat_to_batch_size(mask, output_shape[0])
@@ -75,8 +75,8 @@ class KSamplerX0Inpaint:
def __init__(self, model, sigmas):
self.inner_model = model
self.sigmas = sigmas
self.model_sigmas = torch.cat( (torch.tensor([0.], device = sigmas.device) , torch.tensor( self.inner_model.model_patcher.get_model_object("model_sampling").sigmas, device = sigmas.device) ) )
self.model_sigmas = torch.tensor( self.model_sigmas, dtype = self.sigmas.dtype )
#self.model_sigmas = torch.cat( (torch.tensor([0.], device = sigmas.device) , torch.tensor( self.inner_model.model_patcher.get_model_object("model_sampling").sigmas, device = sigmas.device) ) )
#self.model_sigmas = torch.tensor( self.model_sigmas, dtype = self.sigmas.dtype )
def __call__(self, x, sigma, denoise_mask, model_options={}, seed=None,**kwargs):
### For 1.5 and XL model
# x is x_t in the notation of variance exploding diffusion model, x_t = x_0 + sigma * noise
@@ -89,10 +89,16 @@ class KSamplerX0Inpaint:
# unify the notations into variance exploding diffusion model
if IS_FLUX or IS_FLOW:
LanPaint_Sigma = sigma / ( torch.maximum( 1 - sigma , sigma*0 + 5e-2 ))
self.LanPaint_Sigmas = self.sigmas / ( torch.maximum( 1 - self.sigmas , self.sigmas*0 + 5e-2 ))
Flow_t = sigma
abt = (1 - Flow_t)**2 / ((1 - Flow_t)**2 + Flow_t**2 )
VE_Sigma = Flow_t / (1 - Flow_t)
#print("t", torch.mean( sigma ).item(), "VE_Sigma", torch.mean( VE_Sigma ).item())
else:
LanPaint_Sigma = sigma
VE_Sigma = sigma
abt = 1/( 1+VE_Sigma**2 )
Flow_t = (1-abt)**0.5 / ( (1-abt)**0.5 + abt**0.5 )
if denoise_mask is not None:
if "denoise_mask_function" in model_options:
@@ -101,63 +107,15 @@ class KSamplerX0Inpaint:
denoise_mask = (denoise_mask > 0.5).float()
latent_mask = 1 - denoise_mask
current_times = (VE_Sigma, abt, Flow_t)
abt = 1/( 1+LanPaint_Sigma**2 )
current_step = torch.argmin( torch.abs( self.sigmas - torch.mean(sigma) ) )
total_steps = len(self.sigmas)-1
print("sigma", LanPaint_Sigma, "abt", abt)
if self.step_time_schedule == "dual_shrink":
step_size = self.step_size * (1 - abt) ** 0.5 * abt ** 0.5
elif self.step_time_schedule == "follow_sampler":
time_ind = torch.argmin(torch.abs(self.LanPaint_Sigmas - LanPaint_Sigma))
times = torch.log( 1+ self.LanPaint_Sigmas**2)
time_intervals = times[1:] - times[:-1]
time_intervals = time_intervals / time_intervals[0]
step_size = time_intervals[time_ind] * self.step_size
if total_steps - current_step <= self.LanPaint_early_stop:
out = self.PaintMethod(x, self.latent_image, self.noise, sigma, latent_mask, current_times, model_options, seed, n_steps=0)
else:
step_size = self.step_size * (1 - abt) ** 0.5
#step_size = self.step_size * (1 - abt) ** b * abt ** a / ( ((a/(a+b))**a*(b/(a+b))**b) )
abt_end = 1/( 1+self.end_sigma**2 )
step_size = self.step_size * (1 - torch.minimum(abt/abt_end, abt**0) ) ** 0.5
step_size = step_size[:, None, None, None]
current_times = (LanPaint_Sigma, abt)
# self.inner_model.inner_model.scale_latent_inpaint returns variance exploding x_t values
x = x * (1 - latent_mask) + self.inner_model.inner_model.scale_latent_inpaint(x=x, sigma=sigma, noise=self.noise, latent_image=self.latent_image)* latent_mask
if IS_FLUX or IS_FLOW:
x_t = x * ( 1 + LanPaint_Sigma[:, None,None,None])
else:
x_t = x #/ ( 1+sigma**2 )**0.5 # switch to variance perserving x_t values
# after noise_scaling, noise = latent_image + noise * sigma, which is x_t in the variance exploding diffusion model notation for the known region.
args = None
for i in range(self.n_steps):
if torch.mean(LanPaint_Sigma) > self.start_sigma or torch.mean(LanPaint_Sigma) < self.end_sigma:
break
score_func = partial( self.score_model, y = self.latent_image, mask = latent_mask, abt = abt[:, None,None,None], sigma = LanPaint_Sigma[:, None,None,None], model_options = model_options, seed = seed )
if self.step_size_schedule == "linear":
step_size_i = step_size * (1 - i/(self.n_steps) )
else:
step_size_i = step_size
x_t, args = self.langevin_dynamics(x_t, score_func , latent_mask, step_size_i , current_times, sigma_x = self.sigma_x(abt)[:, None,None,None], sigma_y = self.sigma_y(abt)[:, None,None,None], args = args)
if IS_FLUX or IS_FLOW:
x = x_t / ( 1 + LanPaint_Sigma[:, None,None,None] )
else:
x = x_t #/ ( 1+sigma**2 )**0.5 # switch to variance perserving x_t values
# out is x_0
out, _ = self.inner_model(x, sigma, model_options=model_options, seed=seed)
out = out * denoise_mask + self.latent_image * latent_mask
out = self.PaintMethod(x, self.latent_image, self.noise, sigma, latent_mask, current_times, model_options, seed)
else:
out, _ = self.inner_model(x, sigma, model_options=model_options, seed=seed)
@@ -173,181 +131,7 @@ class KSamplerX0Inpaint:
callback({"i": current_step, "denoised": out, "x": x})
return out
def mid_times(self, current_times, step_size):
sigma, abt = current_times
tt = torch.log(1+sigma**2)
tt_mid = torch.max( tt - step_size, tt*0 )
sigma_mid = (torch.exp(tt_mid) - 1) ** 0.5
sigma_mid_prev = sigma_mid
# find the closest sigma to sigma_mid from self.sigmas
#sigma_mid = self.model_sigmas[torch.argmin(torch.abs(self.model_sigmas - sigma_mid))]
abt_mid = 1/(1+sigma_mid**2)
return sigma_mid, abt_mid
def score_model(self, x_t, y, mask, abt, sigma, model_options, seed):
# the score function for the Langevin dynamics
lamb = self.chara_lamb
beta = self.chara_beta * (1-abt)**0.5
IS_FLUX = self.inner_model.inner_model.model_type == ModelType.FLUX
IS_FLOW = self.inner_model.inner_model.model_type == ModelType.FLOW
if IS_FLUX or IS_FLOW:
x_0, x_0_BIG = self.inner_model(x_t / ( 1 + sigma ), sigma[:, 0,0,0] / ( 1 + sigma[:, 0,0,0] ), model_options=model_options, seed=seed)
else:
x_0, x_0_BIG = self.inner_model(x_t, sigma[:, 0,0,0], model_options=model_options, seed=seed)
e_t = x_t / ((1 - abt) ** 0.5 * (1 + sigma**2) ** 0.5 )- (abt ** 0.5 / (1 - abt) ** 0.5) * x_0
e_t_BIG = x_t / ((1 - abt) ** 0.5 * (1 + sigma**2) ** 0.5 )- (abt ** 0.5 / (1 - abt) ** 0.5) * x_0_BIG
score_x = -e_t
score_y = - (1 + lamb) * ( x_t/ ((1 + sigma**2) ** 0.5 *(1 - abt)**0.5) - abt**0.5 /(1 - abt)**0.5 * y ) + lamb * e_t_BIG
return score_x * (1 - mask) + score_y * mask
def sigma_x(self, abt):
# the time scale for the x_t update
return abt**0
def sigma_y(self, abt):
# the time scale for the y_t update
if self.beta_scale == "shrink":
beta = self.chara_beta * (1-abt)**0.5
elif self.beta_scale == "dual_shrink":
beta = self.chara_beta * (1-abt)**0.5 * abt ** 0.5
elif self.beta_scale == "back_shrink":
beta = self.chara_beta * abt ** 0.5
else:
beta = self.chara_beta * abt ** 0
return beta
def langevin_dynamics(self, x_t, score, mask, step_size, current_times, sigma_x=1, sigma_y=0, args=None):
# -------------------------------------------------------------------------
# Unpack current times parameters (sigma and abt)
sigma, abt = current_times
sigma = sigma[:, None,None,None]
abt = abt[:, None,None,None]
# Compute time step (dtx, dty) for x and y branches.
dtx = 2 * step_size * sigma_x
dty = 2 * step_size * sigma_y
#ref_dt = 0.1 * (1 - abt) ** b * abt ** a / ( ((a/(a+b))**a*(b/(a+b))**b) )
abt_end = 1/( 1+self.end_sigma**2 )
ref_dt = 0.1 * (1 - torch.minimum(abt/abt_end, abt**0) ) ** 0.5
# -------------------------------------------------------------------------
# Define friction parameter Gamma_hat for each branch.
# Using dtx**0 provides a tensor of the proper device/dtype.
Gamma_hat_x = self.friction * dtx / (1e-4+ 2 * sigma_x * ref_dt)
Gamma_hat_y = self.friction * dty / (1e-4+ 2 * sigma_y * ref_dt)
# Get mid time parameters (sigma_mid and abt_mid) for each branch.
sigma_mid_x, abt_mid_x = self.mid_times(current_times, torch.squeeze(dtx))
sigma_mid_y, abt_mid_y = self.mid_times(current_times, torch.squeeze(dty))
sigma_mid_x = sigma_mid_x[:, None,None,None]
sigma_mid_y = sigma_mid_y[:, None,None,None]
abt_mid_x = abt_mid_x[:, None,None,None]
abt_mid_y = abt_mid_y[:, None,None,None]
if torch.mean(sigma_mid_x) >= torch.mean(sigma) or torch.mean(sigma_mid_y) >= torch.mean(sigma):
return x_t, args
# -------------------------------------------------------------------------
# A: Update epsilon (score estimate and noise initialization)
# -------------------------------------------------------------------------
# Compute the score-based epsilon (scaled as sqrt(1-abt))
score_model = score(x_t)
eps_model = -score_model
# Initialize epsilon and Z if not provided in args.
if args is None:
eps = eps_model
Z = torch.randn_like(x_t)
else:
eps, Z = args
# -------------------------------------------------------------------------
# B: Update epsilon mean dynamics and compute the mid-point in z-space.
# -------------------------------------------------------------------------
# Compute the weighted combination term for epsilon mean update:
# term = (2/Γ_hat)*(1-exp(-0.5*Γ_hat))
term_x = 2.0 / (Gamma_hat_x + 1e-4) * (1 - torch.exp(-0.5 * Gamma_hat_x))
term_y = 2.0 / (Gamma_hat_y + 1e-4) * (1 - torch.exp(-0.5 * Gamma_hat_y))
eps_bar_x = term_x * eps + (1 - term_x) * eps_model
eps_bar_y = term_y * eps + (1 - term_y) * eps_model
# Combine branches according to mask.
eps_bar = eps_bar_x * (1 - mask) + eps_bar_y * mask
# Form the denoised epsilon using self.alpha (assumed to be 1/Ψ)
eps_denoise = self.alpha * eps_bar + (1 - self.alpha) * eps_model
# tamed
eps_model_x = eps_denoise* (1 - mask)
eps_model_x = eps_model_x* (torch.sum(1 - mask, dim = (1,2,3), keepdim = True)/torch.sum(eps_model_x**2, dim = (1,2,3), keepdim = True)) **0.5 ** torch.minimum(self.tamed*(dtx),sigma**0)#/( 1 + self.tamed*(sigma - sigma_mid_x) * (torch.sum(eps_model_x**2)/torch.sum((1 - mask)))**0.5 )
eps_model_y = eps_denoise* mask
eps_model_y = eps_model_y* (torch.sum(mask, dim = (1,2,3), keepdim = True)/torch.sum(eps_model_y**2, dim = (1,2,3), keepdim = True)) **0.5 ** torch.minimum(self.tamed*(dty),sigma**0)#/( 1 + self.tamed*(sigma - sigma_mid_y) * (torch.sum(eps_model_y**2)/torch.sum(mask))**0.5 )
eps_denoise = eps_model_x * (1 - mask) + eps_model_y * mask
# Update the mean epsilon for the next step:
eps_x = eps * torch.exp(-0.5 * Gamma_hat_x) + eps_model * (1 - torch.exp(-0.5 * Gamma_hat_x))
eps_y = eps * torch.exp(-0.5 * Gamma_hat_y) + eps_model * (1 - torch.exp(-0.5 * Gamma_hat_y))
eps = eps_x * (1 - mask) + eps_y * mask
# Transform x to z using z = x * sqrt(1+sigma^2). Here we have already set x to z to avoid floating point stability issue.
z_t = x_t #* (1 + sigma**2) ** 0.5
# Compute the mid-point update in z-space for each branch:
z_mid_x = z_t + eps_denoise * (sigma_mid_x - sigma)
z_mid_y = z_t + eps_denoise * (sigma_mid_y - sigma)
z_mid = z_mid_x * (1 - mask) + z_mid_y * mask
# -------------------------------------------------------------------------
# C: Update noise terms and finalize the x update.
# -------------------------------------------------------------------------
# Generate auxiliary noise terms.
Z_q = torch.randn_like(x_t)
Z_q_avg = torch.randn_like(x_t)
Z_z = torch.randn_like(x_t)
# Update Z for each branch:
Z_x = torch.exp(-0.5 * Gamma_hat_x) * Z + (1 - torch.exp(-Gamma_hat_x)) ** 0.5 * Z_q
Z_y = torch.exp(-0.5 * Gamma_hat_y) * Z + (1 - torch.exp(-Gamma_hat_y)) ** 0.5 * Z_q
Z_next = Z_x * (1 - mask) + Z_y * mask
# Compute the combined noise update following the scheme:
Z_comb_x = (
(1 - torch.exp(-Gamma_hat_x / 2)) / torch.sqrt(Gamma_hat_x + 1e-4) *
(Z + torch.sqrt(torch.tanh(Gamma_hat_x / 4)) * Z_q)
+ torch.sqrt(1 - (4 / (Gamma_hat_x + 1e-4)) * torch.tanh(Gamma_hat_x / 4)) * Z_q_avg
)
Z_comb_y = (
(1 - torch.exp(-Gamma_hat_y / 2)) / torch.sqrt(Gamma_hat_y + 1e-4) *
(Z + torch.sqrt(torch.tanh(Gamma_hat_y / 4)) * Z_q)
+ torch.sqrt(1 - (4 / (Gamma_hat_y + 1e-4)) * torch.tanh(Gamma_hat_y / 4)) * Z_q_avg
)
Z_comb = Z_comb_x * (1 - mask) + Z_comb_y * mask
# Combine with an additional noise term using self.alpha.
Z_comb = self.alpha ** 0.5 * Z_comb + (1 - self.alpha) ** 0.5 * Z_z
# Compute the change in sigma (dsigma = sqrt(sigma^2 - sigma_mid^2)).
dsigma_x = sigma * torch.sqrt(1 - (sigma_mid_x / sigma) ** 2)
dsigma_y = sigma * torch.sqrt(1 - (sigma_mid_y / sigma) ** 2)
dsigma = dsigma_x * (1 - mask) + dsigma_y * mask
# Final z update.
z_final = z_mid + Z_comb * dsigma
# Transform back to x-space: x = z / sqrt(1+sigma^2)
x_t = z_final #/ (1 + sigma**2) ** 0.5
return x_t, (eps, Z_next)
# Custom sampler class extending ComfyUI's KSAMPLER for LanPaint
class KSAMPLER(comfy.samplers.KSAMPLER):
def sample(self, model_wrap, sigmas, extra_args, callback, noise, latent_image=None, denoise_mask=None, disable_pbar=False):
@@ -363,27 +147,23 @@ class KSAMPLER(comfy.samplers.KSAMPLER):
model_k.noise = noise
IS_FLUX = model_wrap.inner_model.model_type == ModelType.FLUX
IS_FLOW = model_wrap.inner_model.model_type == ModelType.FLOW
# unify the notations into variance exploding diffusion model
if IS_FLUX:
model_wrap.cfg_BIG = 1.0
else:
model_wrap.cfg_BIG = model_wrap.model_patcher.LanPaint_cfg_BIG
model_k.step_size = model_wrap.model_patcher.LanPaint_StepSize
model_k.chara_lamb = model_wrap.model_patcher.LanPaint_Lambda
model_k.chara_beta = model_wrap.model_patcher.LanPaint_Beta
model_k.n_steps = model_wrap.model_patcher.LanPaint_NumSteps
model_k.friction = model_wrap.model_patcher.LanPaint_Friction
model_k.alpha = model_wrap.model_patcher.LanPaint_Alpha
model_k.tamed = model_wrap.model_patcher.LanPaint_Tamed
model_k.beta_scale = model_wrap.model_patcher.LanPaint_BetaScale
model_k.step_size_schedule = model_wrap.model_patcher.LanPaint_StepSizeSchedule
model_k.step_time_schedule = model_wrap.model_patcher.LanPaint_StepTimeSchedule
model_k.start_sigma = model_wrap.model_patcher.LanPaint_StartSigma
model_k.end_sigma = model_wrap.model_patcher.LanPaint_EndSigma
noise = model_wrap.inner_model.model_sampling.noise_scaling(sigmas[0], noise, latent_image, self.max_denoise(model_wrap, sigmas))
model_k.PaintMethod = LanPaint(model_k.inner_model,
model_wrap.model_patcher.LanPaint_NumSteps,
model_wrap.model_patcher.LanPaint_Friction,
model_wrap.model_patcher.LanPaint_Lambda,
model_wrap.model_patcher.LanPaint_Beta,
model_wrap.model_patcher.LanPaint_StepSize,
IS_FLUX = IS_FLUX,
IS_FLOW = IS_FLOW)
model_k.LanPaint_early_stop = model_wrap.model_patcher.LanPaint_EarlyStop
#if not inpainting, after noise_scaling, noise = noise * sigma, which is the noise added to the clean latent image in the variance exploding diffusion model notation.
#if inpainting, after noise_scaling, noise = latent_image + noise * sigma, which is x_t in the variance exploding diffusion model notation for the known region.
k_callback = None
@@ -440,7 +220,12 @@ class LanPaint_UpSale_LatentNoiseMask:
s["noise_mask"] = mask
return (s,)
KSAMPLER_NAMES = ["euler", "dpmpp_2m", "uni_pc"]
#KSAMPLER_NAMES = ["euler", "dpmpp_2m", "uni_pc"]
KSAMPLER_NAMES = ["euler","euler_ancestral", "heun", "heunpp2","dpm_2", "dpm_2_ancestral",
"dpm_fast", "dpmpp_sde", "dpmpp_sde_gpu",
"dpmpp_2m", "dpmpp_2m_sde", "dpmpp_2m_sde_gpu", "dpmpp_3m_sde", "dpmpp_3m_sde_gpu", "ddpm",
"deis", "res_multistep", "res_multistep_ancestral",
"gradient_estimation", "er_sde", "seeds_2", "seeds_3"]
class LanPaint_KSampler():
@classmethod
@@ -449,7 +234,7 @@ class LanPaint_KSampler():
"required": {
"model": ("MODEL", {"tooltip": "The model used for denoising the input latent."}),
"seed": ("INT", {"default": 0, "min": 0, "max": 0xffffffffffffffff, "tooltip": "The random seed used for creating the noise."}),
"steps": ("INT", {"default": 50, "min": 1, "max": 10000, "tooltip": "The number of steps used in the denoising process."}),
"steps": ("INT", {"default": 30, "min": 1, "max": 10000, "tooltip": "The number of steps used in the denoising process."}),
"cfg": ("FLOAT", {"default": 5.0, "min": 0.0, "max": 100.0, "step":0.1, "round": 0.01, "tooltip": "The Classifier-Free Guidance scale balances creativity and adherence to the prompt. Higher values result in images more closely matching the prompt however too high values will negatively impact quality."}),
"sampler_name": (KSAMPLER_NAMES, {"tooltip": "Recommended: euler."}),
"scheduler": (comfy.samplers.KSampler.SCHEDULERS, {"default": "karras", "tooltip": "The scheduler controls how noise is gradually removed to form the image."}),
@@ -457,9 +242,9 @@ class LanPaint_KSampler():
"negative": ("CONDITIONING", {"tooltip": "The conditioning describing the attributes you want to exclude from the image."}),
"latent_image": ("LATENT", {"tooltip": "The latent image to denoise."}),
"denoise": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.01, "tooltip": "The amount of denoising applied, lower values will maintain the structure of the initial image allowing for image to image sampling."}),
"LanPaint_NumSteps": ("INT", {"default": 5, "min": 0, "max": 20, "tooltip": "The number of steps for the Langevin dynamics, representing the turns of thinking per step."}),
"LanPaint_EndSigma": ("FLOAT", {"default": 3.0, "min": 0.0, "max": 20.0, "step": 0.01, "tooltip": "The noise level at which the thinking process stops. Higher value give less thinking but helps to deal with blurring when turns of thinking is too high."}),
"LanPaint_Info": ("STRING", {"default": "LanPaint KSampler. Recommend steps 50, LanPaint NumSteps 1-20 depending on the difficulty of task. LanPaint_EndSigma = 3.0 for anime style, 0.6 for realistic style. For more information, visit https://github.com/scraed/LanPaint", "multiline": True}),
"LanPaint_NumSteps": ("INT", {"default": 5, "min": 0, "max": 100, "tooltip": "The number of steps for the Langevin dynamics, representing the turns of thinking per step."}),
"LanPaint_PromptMode": (["Image First", "Prompt First"], {"tooltip": "Image First: emphasis image quality, Prompt First: emphasis prompt following"}),
"LanPaint_Info": ("STRING", {"default": "LanPaint KSampler. For more info, visit https://github.com/scraed/LanPaint. If you find it useful, please give a star ⭐️!", "multiline": True}),
}
}
@@ -470,20 +255,18 @@ class LanPaint_KSampler():
CATEGORY = "sampling"
DESCRIPTION = "Uses the provided model, positive and negative conditioning to denoise the latent image."
def sample(self, model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=1.0, LanPaint_StepSize=0.05, LanPaint_NumSteps=5, LanPaint_EndSigma = 3., LanPaint_Info=""):
model.LanPaint_StepSize = 0.5
model.LanPaint_Lambda = 8.0
model.LanPaint_Beta = 1.2
def sample(self, model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=1.0, LanPaint_NumSteps=5, LanPaint_PromptMode = "Image First", LanPaint_Info=""):
model.LanPaint_StepSize = 0.15
model.LanPaint_Lambda = 16.0
model.LanPaint_Beta = 1.
model.LanPaint_NumSteps = LanPaint_NumSteps
model.LanPaint_Friction = 5.
model.LanPaint_Alpha = 0.9
model.LanPaint_Tamed = 1.
model.LanPaint_BetaScale = "shrink"
model.LanPaint_StepSizeSchedule = "linear"
model.LanPaint_StepTimeSchedule = "shrink"
model.LanPaint_StartSigma = 20.
model.LanPaint_EndSigma = LanPaint_EndSigma
model.LanPaint_cfg_BIG = -0.5
model.LanPaint_Friction = 15.
model.LanPaint_EarlyStop = 1
if LanPaint_PromptMode == "Image First":
model.LanPaint_cfg_BIG = cfg
else:
model.LanPaint_cfg_BIG = 0*cfg - 0.5
with override_sample_function():
return nodes.common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
class LanPaint_KSamplerAdvanced:
@@ -493,7 +276,7 @@ class LanPaint_KSamplerAdvanced:
{"model": ("MODEL",),
"add_noise": (["enable", "disable"], ),
"noise_seed": ("INT", {"default": 0, "min": 0, "max": 0xffffffffffffffff}),
"steps": ("INT", {"default": 50, "min": 1, "max": 10000}),
"steps": ("INT", {"default": 30, "min": 1, "max": 10000}),
"cfg": ("FLOAT", {"default": 5.0, "min": 0.0, "max": 100.0, "step":0.1, "round": 0.01}),
"sampler_name": (KSAMPLER_NAMES, ),
"scheduler": (comfy.samplers.KSampler.SCHEDULERS, ),
@@ -503,20 +286,14 @@ class LanPaint_KSamplerAdvanced:
"start_at_step": ("INT", {"default": 0, "min": 0, "max": 10000}),
"end_at_step": ("INT", {"default": 10000, "min": 0, "max": 10000}),
"return_with_leftover_noise": (["disable", "enable"], ),
"LanPaint_NumSteps": ("INT", {"default": 5, "min": 0, "max": 20, "tooltip": "The number of steps for the Langevin dynamics, representing the turns of thinking per step."}),
"LanPaint_Lambda": ("FLOAT", {"default": 8., "min": 0.1, "max": 50.0, "step": 0.1, "round": 0.1, "tooltip": "The lambda parameter for the bidirectional guidance. Higher values align with known regions more closely, but may result in instability."}),
"LanPaint_StepSize": ("FLOAT", {"default": 0.5, "min": 0.0001, "max": 1., "step": 0.01, "round": 0.001, "tooltip": "The step size for the Langevin dynamics. Higher values result in faster convergence but may be unstable."}),
"LanPaint_Beta": ("FLOAT", {"default": 1.2, "min": 0.0001, "max": 5, "step": 0.1, "round": 0.1, "tooltip": "The beta parameter for the bidirectional guidance. Scale the step size for the known region independently for the Langevin dynamics. Higher values result in faster convergence but may be unstable."}),
"LanPaint_Friction": ("FLOAT", {"default": 5., "min": 1., "max": 50.0, "step": 0.1, "round": 0.1, "tooltip": "The friction parameter for the underdamped Langevin dynamics, higher values result in faster convergence but may be unstable."}),
"LanPaint_Alpha": ("FLOAT", {"default": 0.9, "min": 0.0001, "max": 1., "step": 0.1, "round": 0.1, "tooltip": "The (rescaled) alpha parameter for the HFHR langevin dynamics, mixes Langevin dynamics and underdamped Langevin dynamics with a friction term. 0 corresponds to Langevin dynamics, 1 corresponds to underdamped Langevin dynamics."}),
"LanPaint_Tamed": ("FLOAT", {"default": 1., "min": 0.000, "max": 20., "step": 0.1, "round": 0.1, "tooltip": "The tame strength for the noise, normalize and projects the noise onto unit sphere to enhance stability."}),
"LanPaint_BetaScale": (["shrink", "fixed", "dual_shrink", "back_shrink"], {"default": "shrink", "tooltip": "The beta scale, determines how the beta parameter changes over time. Shrink: beta = beta * (1 - alpha bar) ** 0.5; Fixed: beta = beta; Dual_shrink: beta = beta * (1 - alpha bar) ** 0.5 * alpha bar ** 0.5; Back_shrink: beta = beta * alpha bar ** 0.5; Alpha bar: the alpha cumprod."}),
"LanPaint_StepSizeSchedule": (["const", "linear"], {"default": "linear", "tooltip": "The step size schedule for the Langevin dynamics, const: constant step size, linear: linearly decreasing step size."}),
"LanPaint_StepTimeSchedule": (["shrink", "dual_shrink", "follow_sampler"], {"default": "shrink", "tooltip": "The step size schedule for the first step of Langevin dynamics during diffusion sampling, shrink: step size = step size * (1 - alpha bar) ** 0.5; Dual_shrink: step size = step size * (1 - alpha bar) ** 0.5 * alpha bar ** 0.5; Follow_sampler: scale with the sampler step size."}),
"LanPaint_StartSigma": ("FLOAT", {"default": 20., "min": 0.0001, "max": 20.0, "step": 0.1, "round": 0.1, "tooltip": "Start 'thinking' with Langevin dynamics at this sigma value."}),
"LanPaint_EndSigma": ("FLOAT", {"default": 3., "min": 0.000, "max": 20.0, "step": 0.1, "round": 0.1, "tooltip": "Stop 'thinking' with Langevin dynamics at this sigma value."}),
"LanPaint_cfg_BIG": ("FLOAT", {"default": -0.5, "min": -20, "max": 20.0, "step": 0.1, "round": 0.1, "tooltip": "The CFG scale used in the bidirectional guidance (for the known region only). Higher value results in more closely matching the known region."}),
"LanPaint_Info": ("STRING", {"default": "LanPaint KSampler Advanced. For difficult tasks, first try increasing steps, LanPaint_NumSteps, and LanPaint_cfg_BIG. Then try increase LanPaint_Lambda or LanPaint_StepSize. Decrease LanPaint_Friction if you want to obtain good results with fewer turns of thinking (LanPaint_NumSteps) at the risk of irregular behavior. Increase LanPaint_Tamed or LanPaint_Alpha can suppress irregular behavior. For more information, visit https://github.com/scraed/LanPaint", "multiline": True}),
"LanPaint_NumSteps": ("INT", {"default": 5, "min": 0, "max": 100, "tooltip": "The number of steps for the Langevin dynamics, representing the turns of thinking per step."}),
"LanPaint_Lambda": ("FLOAT", {"default": 16., "min": 0.1, "max": 50.0, "step": 0.1, "round": 0.1, "tooltip": "The bidirectional guidance scale. Higher values align with known regions more closely, but may result in instability."}),
"LanPaint_StepSize": ("FLOAT", {"default": 0.15, "min": 0.0001, "max": 1., "step": 0.01, "round": 0.001, "tooltip": "The step size for the Langevin dynamics. Higher values result in faster convergence but may be unstable."}),
"LanPaint_Beta": ("FLOAT", {"default": 1., "min": 0.0001, "max": 5, "step": 0.1, "round": 0.1, "tooltip": "The step size ratio between masked / unmasked regions. Lower value can compensate high values of LanPaint_Lambda."}),
"LanPaint_Friction": ("FLOAT", {"default": 15, "min": 0., "max": 50.0, "step": 0.1, "round": 0.1, "tooltip": "The friction parameter for fast langevin, lower values result in faster convergence but may be unstable."}),
"LanPaint_PromptMode": (["Image First", "Prompt First"], {"tooltip": "Image First: emphasis image quality, Prompt First: emphasis prompt following"}),
"LanPaint_EarlyStop": ("INT", {"default": 1, "min": 0, "max": 10000, "tooltip": "The number of steps to stop the LanPaint early, useful for preventing the image from irregular patterns."}),
"LanPaint_Info": ("STRING", {"default": "LanPaint KSampler Adv. For more info, visit https://github.com/scraed/LanPaint. If you find it useful, please give a star ⭐️!", "multiline": True}),
},
}
@@ -525,7 +302,7 @@ class LanPaint_KSamplerAdvanced:
CATEGORY = "sampling"
def sample(self, model, add_noise, noise_seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, start_at_step, end_at_step, return_with_leftover_noise, denoise=1.0, LanPaint_StepSize=0.05, LanPaint_Lambda=5, LanPaint_Beta=1, LanPaint_NumSteps=5, LanPaint_Friction=5, LanPaint_Alpha=1, LanPaint_Tamed=0., LanPaint_BetaScale="fixed", LanPaint_StepSizeSchedule = "const", LanPaint_StepTimeSchedule = "shrink", LanPaint_StartSigma=20, LanPaint_EndSigma=0, LanPaint_cfg_BIG = 5., LanPaint_Info=""):
def sample(self, model, add_noise, noise_seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, start_at_step, end_at_step, return_with_leftover_noise, denoise=1.0, LanPaint_StepSize=0.05, LanPaint_Lambda=5, LanPaint_Beta=1, LanPaint_NumSteps=5, LanPaint_Friction=5, LanPaint_PromptMode = "Image First", LanPaint_EarlyStop = 1, LanPaint_Info=""):
force_full_denoise = True
if return_with_leftover_noise == "enable":
force_full_denoise = False
@@ -537,23 +314,243 @@ class LanPaint_KSamplerAdvanced:
model.LanPaint_Beta = LanPaint_Beta
model.LanPaint_NumSteps = LanPaint_NumSteps
model.LanPaint_Friction = LanPaint_Friction
model.LanPaint_Alpha = LanPaint_Alpha
model.LanPaint_Tamed = LanPaint_Tamed
model.LanPaint_BetaScale = LanPaint_BetaScale
model.LanPaint_StepSizeSchedule = LanPaint_StepSizeSchedule
model.LanPaint_StepTimeSchedule = LanPaint_StepTimeSchedule
model.LanPaint_StartSigma = LanPaint_StartSigma
model.LanPaint_EndSigma = LanPaint_EndSigma
model.LanPaint_cfg_BIG = LanPaint_cfg_BIG
model.LanPaint_EarlyStop = LanPaint_EarlyStop
if LanPaint_PromptMode == "Image First":
model.LanPaint_cfg_BIG = cfg
else:
model.LanPaint_cfg_BIG = 0*cfg - 0.5
with override_sample_function():
return nodes.common_ksampler(model, noise_seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise, disable_noise=disable_noise, start_step=start_at_step, last_step=end_at_step, force_full_denoise=force_full_denoise)
class MaskBlend:
def __init__(self):
pass
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"image1": ("IMAGE", {"tooltip": "Image before inpaint"}),
"image2": ("IMAGE", {"tooltip": "Image after inpaint"}),
"mask": ("MASK",),
"blend_overlap": ("INT", {"default": 1, "min": 1, "max": 51, "step": 2, "tooltip": "The number of pixels to blend between the two images."})
},
}
RETURN_TYPES = ("IMAGE",)
FUNCTION = "blend_images"
CATEGORY = "image/postprocessing"
def blend_images(self, image1: torch.Tensor, image2: torch.Tensor, mask: torch.Tensor, blend_overlap: int):
# smooth the binary 01 mask, keep 1 still 1, but smooth the transition from 1 to 0
# for each mask pixel, find out the nearest 1 pixel, and set the mask value to the distance between the two pixels
# check the size of mask and image1, image2, if not the same, assert error
if image1.shape[1] != image2.shape[1] or image1.shape[2] != image2.shape[2]:
raise ValueError("Make sure your image size is a multiple of 8. Otherwise the mask will not be aligned with the output image.")
mask = mask.float()
mask = torch.nn.functional.max_pool2d(mask, kernel_size=blend_overlap, stride=1, padding=blend_overlap//2)
# apply Gaussian blur with kernel size blend_overlap
kernel = self.gaussian_kernel(blend_overlap)
kernel = kernel.to(image1.device)
kernel = kernel[None, None, ...]
mask = torch.nn.functional.conv2d(mask[:,None,:,:], kernel, padding=blend_overlap//2)[:,0,:,:]
blended_image = image1 * (1 - mask[...,None]) + image2 * mask[...,None]
return (blended_image,)
def gaussian_kernel(self,kernel_size):
"""
Creates a 2D Gaussian kernel with the given size and standard deviation (sigma).
"""
sigma = (kernel_size - 1)/4
# Create a grid of (x, y) coordinates
x = torch.arange(kernel_size).float() - kernel_size // 2
y = torch.arange(kernel_size).float() - kernel_size // 2
x_grid, y_grid = torch.meshgrid(x, y, indexing='ij')
# Compute the Gaussian function
kernel = torch.exp(-(x_grid ** 2 + y_grid ** 2) / (2 * sigma ** 2))
kernel = kernel / kernel.sum() # Normalize the kernel
return kernel
class Noise_EmptyNoise:
def generate_noise(self, latent):
return torch.zeros_like(latent["samples"])
class Noise_RandomNoise:
def __init__(self, seed):
self.seed = seed
def generate_noise(self, latent):
torch.manual_seed(self.seed)
return torch.randn_like(latent["samples"])
# Custom sampler implementation mimmicking base comfy nodes_custom_sampler.py
class LanPaint_SamplerCustom:
@classmethod
def INPUT_TYPES(s):
return {"required":
{"model": ("MODEL",),
"add_noise": ("BOOLEAN", {"default": True}),
"noise_seed": ("INT", {"default": 0, "min": 0, "max": 0xffffffffffffffff, "control_after_generate": True}),
"cfg": ("FLOAT", {"default": 8.0, "min": 0.0, "max": 100.0, "step": 0.1, "round": 0.01}),
"positive": ("CONDITIONING",),
"negative": ("CONDITIONING",),
"sampler": ("SAMPLER",),
"sigmas": ("SIGMAS",),
"latent_image": ("LATENT",),
"LanPaint_NumSteps": ("INT", {"default": 5, "min": 0, "max": 100, "tooltip": "Number of steps for Langevin dynamics, representing turns of thinking per step."}),
"LanPaint_PromptMode": (["Image First", "Prompt First"], {"tooltip": "Image First: prioritizes image quality; Prompt First: prioritizes prompt adherence."}),
"LanPaint_Info": ("STRING", {"default": "LanPaint Custom Sampler. For more info, visit https://github.com/scraed/LanPaint. If you find it useful, please give a star ⭐️!", "multiline": True}),
}
}
RETURN_TYPES = ("LATENT", "LATENT")
RETURN_NAMES = ("output", "denoised_output")
FUNCTION = "sample"
CATEGORY = "sampling/custom_sampling"
def sample(self, model, sampler, sigmas, add_noise, noise_seed, cfg, positive, negative, latent_image, LanPaint_NumSteps, LanPaint_PromptMode, LanPaint_Info=""):
model.LanPaint_StepSize = 0.15
model.LanPaint_Lambda = 16.0
model.LanPaint_Beta = 1.
model.LanPaint_NumSteps = LanPaint_NumSteps
model.LanPaint_Friction = 15.
model.LanPaint_EarlyStop = 1
if LanPaint_PromptMode == "Image First":
model.LanPaint_cfg_BIG = cfg
else:
model.LanPaint_cfg_BIG = 0 * cfg - 0.5
with override_sample_function():
latent = latent_image.copy()
latent_image = latent["samples"]
latent_image = comfy.sample.fix_empty_latent_channels(model, latent_image)
latent["samples"] = latent_image
if not add_noise:
noise = Noise_EmptyNoise().generate_noise(latent)
else:
noise = Noise_RandomNoise(noise_seed).generate_noise(latent)
noise_mask = None
if "noise_mask" in latent:
noise_mask = latent["noise_mask"]
x0_output = {}
callback = latent_preview.prepare_callback(model, sigmas.shape[-1] - 1, x0_output)
disable_pbar = not comfy.utils.PROGRESS_BAR_ENABLED
samples = comfy.sample.sample_custom(model, noise, cfg, sampler, sigmas, positive, negative, latent_image,noise_mask=noise_mask, callback=callback, disable_pbar=disable_pbar, seed=noise_seed)
out = latent.copy()
out["samples"] = samples
if "x0" in x0_output:
out_denoised = latent.copy()
out_denoised["samples"] = model.model.process_latent_out(x0_output["x0"].cpu())
else:
out_denoised = out
return (out, out_denoised)
class LanPaint_SamplerCustomAdvanced:
@classmethod
def INPUT_TYPES(s):
return {"required":
{"noise": ("NOISE",),
"guider": ("GUIDER",),
"sampler": ("SAMPLER",),
"sigmas": ("SIGMAS",),
"latent_image": ("LATENT",),
"start_at_step": ("INT", {"default": 0, "min": 0, "max": 10000}),
"end_at_step": ("INT", {"default": 10000, "min": 0, "max": 10000}),
"return_with_leftover_noise": (["disable", "enable"], ),
"LanPaint_NumSteps": ("INT", {"default": 5, "min": 0, "max": 100, "tooltip": "Number of steps for Langevin dynamics, representing turns of thinking per step."}),
"LanPaint_Lambda": ("FLOAT", {"default": 16.0, "min": 0.1, "max": 50.0, "step": 0.1, "tooltip": "Bidirectional guidance scale. Higher values align with known regions but may cause instability."}),
"LanPaint_StepSize": ("FLOAT", {"default": 0.15, "min": 0.0001, "max": 1.0, "step": 0.01, "tooltip": "Step size for Langevin dynamics. Higher values speed convergence but may be unstable."}),
"LanPaint_Beta": ("FLOAT", {"default": 1.0, "min": 0.0001, "max": 5.0, "step": 0.1, "tooltip": "Step size ratio between masked/unmasked regions. Lower values balance high Lambda."}),
"LanPaint_Friction": ("FLOAT", {"default": 15.0, "min": 0.0, "max": 50.0, "step": 0.1, "tooltip": "Friction parameter for fast Langevin. Lower values speed convergence but may be unstable."}),
"LanPaint_PromptMode": (["Image First", "Prompt First"], {"tooltip": "Image First: prioritizes image quality; Prompt First: prioritizes prompt adherence."}),
"LanPaint_EarlyStop": ("INT", {"default": 1, "min": 0, "max": 10000, "tooltip": "Steps to stop LanPaint early, preventing irregular patterns."}),
"LanPaint_Info": ("STRING", {"default": "LanPaint Custom Sampler Adv. For more info, visit https://github.com/scraed/LanPaint. If you find it useful, please give a star ⭐️!", "multiline": True}),
}
}
RETURN_TYPES = ("LATENT", "LATENT")
RETURN_NAMES = ("output", "denoised_output")
FUNCTION = "sample"
CATEGORY = "sampling/custom_sampling"
def sample(self, noise, guider, sampler, sigmas, latent_image, start_at_step, end_at_step, return_with_leftover_noise, LanPaint_NumSteps, LanPaint_Lambda, LanPaint_StepSize, LanPaint_Beta, LanPaint_Friction, LanPaint_PromptMode, LanPaint_EarlyStop, LanPaint_Info=""):
force_full_denoise = True
if end_at_step <= start_at_step:
raise ValueError('end_at_step must be larger than start_at_step')
if return_with_leftover_noise == "enable":
force_full_denoise = False
model = guider.model_patcher
model.LanPaint_StepSize = LanPaint_StepSize
model.LanPaint_Lambda = LanPaint_Lambda
model.LanPaint_Beta = LanPaint_Beta
model.LanPaint_NumSteps = LanPaint_NumSteps
model.LanPaint_Friction = LanPaint_Friction
model.LanPaint_EarlyStop = LanPaint_EarlyStop
if LanPaint_PromptMode == "Image First":
model.LanPaint_cfg_BIG = guider.cfg
else:
model.LanPaint_cfg_BIG = 0 * guider.cfg - 0.5
with override_sample_function():
latent = latent_image.copy()
latent_image_samples = latent["samples"]
latent_image_samples = comfy.sample.fix_empty_latent_channels(model, latent_image_samples)
latent["samples"] = latent_image_samples
noise_mask = None
if "noise_mask" in latent:
noise_mask = latent["noise_mask"]
# From base comfy samplers.py
if end_at_step is not None and end_at_step < (len(sigmas) - 1):
sigmas = sigmas[:end_at_step + 1]
if force_full_denoise:
sigmas[-1] = 0
if start_at_step is not None:
if start_at_step < (len(sigmas) - 1):
sigmas = sigmas[start_at_step:]
else:
if latent_image is not None:
return latent_image
else:
return torch.zeros_like(noise)
x0_output = {}
callback = latent_preview.prepare_callback(model, sigmas.shape[-1] - 1, x0_output)
disable_pbar = not comfy.utils.PROGRESS_BAR_ENABLED
samples = guider.sample( noise.generate_noise(latent), latent_image_samples, sampler, sigmas, denoise_mask=noise_mask, callback=callback,disable_pbar=disable_pbar, seed=noise.seed
)
samples = samples.to(comfy.model_management.intermediate_device())
out = latent.copy()
out["samples"] = samples
if "x0" in x0_output:
out_denoised = latent.copy()
out_denoised["samples"] = model.model.process_latent_out(x0_output["x0"].cpu())
else:
out_denoised = out
return (out, out_denoised)
# A dictionary that contains all nodes you want to export with their names
# NOTE: names should be globally unique
NODE_CLASS_MAPPINGS = {
"LanPaint_KSampler": LanPaint_KSampler,
"LanPaint_KSamplerAdvanced": LanPaint_KSamplerAdvanced,
"LanPaint_SamplerCustom" : LanPaint_SamplerCustom,
"LanPaint_SamplerCustomAdvanced" : LanPaint_SamplerCustomAdvanced,
"LanPaint_MaskBlend": MaskBlend,
# "LanPaint_UpSale_LatentNoiseMask": LanPaint_UpSale_LatentNoiseMask,
}
@@ -561,5 +558,8 @@ NODE_CLASS_MAPPINGS = {
NODE_DISPLAY_NAME_MAPPINGS = {
"LanPaint_KSampler": "LanPaint KSampler",
"LanPaint_KSamplerAdvanced": "LanPaint KSampler (Advanced)",
"LanPaint_SamplerCustom" : "LanPaint Sampler Custom",
"LanPaint_SamplerCustomAdvanced" : "LanPaint Sampler Custom (Advanced)",
"LanPaint_MaskBlend": "LanPaint Mask Blend",
# "LanPaint_UpSale_LatentNoiseMask": "LanPaint UpSale Latent Noise Mask"
}
+301
View File
@@ -0,0 +1,301 @@
import torch
def epxm1_x(x):
# Compute the (exp(x) - 1) / x term with a small value to avoid division by zero.
result = torch.special.expm1(x) / x
# replace NaN or inf values with 0
result = torch.where(torch.isfinite(result), result, torch.zeros_like(result))
mask = torch.abs(x) < 1e-2
result = torch.where(mask, 1 + x/2. + x**2 / 6., result)
return result
def epxm1mx_x2(x):
# Compute the (exp(x) - 1 - x) / x**2 term with a small value to avoid division by zero.
result = (torch.special.expm1(x) - x) / x**2
# replace NaN or inf values with 0
result = torch.where(torch.isfinite(result), result, torch.zeros_like(result))
mask = torch.abs(x**2) < 1e-2
result = torch.where(mask, 1/2. + x/6 + x**2 / 24 + x**3 / 120, result)
return result
def expm1mxmhx2_x3(x):
# Compute the (exp(x) - 1 - x - x**2 / 2) / x**3 term with a small value to avoid division by zero.
result = (torch.special.expm1(x) - x - x**2 / 2) / x**3
# replace NaN or inf values with 0
result = torch.where(torch.isfinite(result), result, torch.zeros_like(result))
mask = torch.abs(x**3) < 1e-2
result = torch.where(mask, 1/6 + x/24 + x**2 / 120 + x**3 / 720 + x**4 / 5040, result)
return result
def exp_1mcosh_GD(gamma_t, delta):
"""
Compute e^(-Γt) * (1 - cosh(Γt√Δ))/ ( (Γt)**2 Δ )
Parameters:
gamma_t: Γ*t term (could be a scalar or tensor)
delta: Δ term (could be a scalar or tensor)
Returns:
Result of the computation with numerical stability handling
"""
# Main computation
is_positive = delta > 0
sqrt_abs_delta = torch.sqrt(torch.abs(delta))
gamma_t_sqrt_delta = gamma_t * sqrt_abs_delta
numerator_pos = torch.exp(-gamma_t) - (torch.exp(gamma_t * (sqrt_abs_delta - 1)) + torch.exp(gamma_t * (-sqrt_abs_delta - 1))) / 2
numerator_neg = torch.exp(-gamma_t) * ( 1 - torch.cos(gamma_t * sqrt_abs_delta ) )
numerator = torch.where(is_positive, numerator_pos, numerator_neg)
result = numerator / (delta * gamma_t**2 )
# Handle NaN/inf cases
result = torch.where(torch.isfinite(result), result, torch.zeros_like(result))
# Handle numerical instability for small delta
mask = torch.abs(gamma_t_sqrt_delta**2) < 5e-2
taylor = ( -0.5 - gamma_t**2 / 24 * delta - gamma_t**4 / 720 * delta**2 ) * torch.exp(-gamma_t)
result = torch.where(mask, taylor, result)
return result
def exp_sinh_GsqrtD(gamma_t, delta):
"""
Compute e^(-Γt) * sinh(Γt√Δ) / (Γt√Δ)
Parameters:
gamma_t: Γ*t term (could be a scalar or tensor)
delta: Δ term (could be a scalar or tensor)
Returns:
Result of the computation with numerical stability handling
"""
# Main computation
is_positive = delta > 0
sqrt_abs_delta = torch.sqrt(torch.abs(delta))
gamma_t_sqrt_delta = gamma_t * sqrt_abs_delta
numerator_pos = (torch.exp(gamma_t * (sqrt_abs_delta - 1)) - torch.exp(gamma_t * (-sqrt_abs_delta - 1))) / 2
denominator_pos = gamma_t_sqrt_delta
result_pos = numerator_pos / gamma_t_sqrt_delta
result_pos = torch.where(torch.isfinite(result_pos), result_pos, torch.zeros_like(result_pos))
# Taylor expansion for small gamma_t_sqrt_delta
mask = torch.abs(gamma_t_sqrt_delta) < 1e-2
taylor = ( 1 + gamma_t**2 / 6 * delta + gamma_t**4 / 120 * delta**2 ) * torch.exp(-gamma_t)
result_pos = torch.where(mask, taylor, result_pos)
# Handle negative delta
result_neg = torch.exp(-gamma_t) * torch.special.sinc(gamma_t_sqrt_delta/torch.pi)
result = torch.where(is_positive, result_pos, result_neg)
return result
def exp_cosh(gamma_t, delta):
"""
Compute e^(-Γt) * cosh(Γt√Δ)
Parameters:
gamma_t: Γ*t term (could be a scalar or tensor)
delta: Δ term (could be a scalar or tensor)
Returns:
Result of the computation with numerical stability handling
"""
exp_1mcosh_GD_result = exp_1mcosh_GD(gamma_t, delta) # e^(-Γt) * (1 - cosh(Γt√Δ))/ ( (Γt)**2 Δ )
result = torch.exp(-gamma_t) - gamma_t**2 * delta * exp_1mcosh_GD_result
return result
def exp_sinh_sqrtD(gamma_t, delta):
"""
Compute e^(-Γt) * sinh(Γt√Δ) / √Δ
Parameters:
gamma_t: Γ*t term (could be a scalar or tensor)
delta: Δ term (could be a scalar or tensor)
Returns:
Result of the computation with numerical stability handling
"""
exp_sinh_GsqrtD_result = exp_sinh_GsqrtD(gamma_t, delta) # e^(-Γt) * sinh(Γt√Δ) / (Γt√Δ)
result = gamma_t * exp_sinh_GsqrtD_result
return result
def zeta1(gamma_t, delta):
# Compute hyperbolic terms and exponential
half_gamma_t = gamma_t / 2
exp_cosh_term = exp_cosh(half_gamma_t, delta)
exp_sinh_term = exp_sinh_sqrtD(half_gamma_t, delta)
# Main computation
numerator = 1 - (exp_cosh_term + exp_sinh_term)
denominator = gamma_t * (1 - delta) / 4
result = 1 - numerator / denominator
# Handle numerical instability
result = torch.where(torch.isfinite(result), result, torch.zeros_like(result))
# Taylor expansion for small x (similar to your epxm1Dx approach)
mask = torch.abs(denominator) < 5e-3
term1 = epxm1_x(-gamma_t)
term2 = epxm1mx_x2(-gamma_t)
term3 = expm1mxmhx2_x3(-gamma_t)
taylor = term1 + (1/2.+ term1-3*term2)*denominator + (-1/6. + term1/2 - 4 * term2 + 10 * term3) * denominator**2
result = torch.where(mask, taylor, result)
return result
def exp_cosh_minus_terms(gamma_t, delta):
"""
Compute E^(-tΓ) * (Cosh[tΓ] - 1 - (Cosh[tΓ√Δ] - 1)/Δ) / (tΓ(1 - Δ))
Parameters:
gamma_t: Γ*t term (could be a scalar or tensor)
delta: Δ term (could be a scalar or tensor)
Returns:
Result of the computation with numerical stability handling
"""
exp_term = torch.exp(-gamma_t)
# Compute individual terms
exp_cosh_term = exp_cosh(gamma_t, gamma_t**0) - exp_term # E^(-tΓ) (Cosh[tΓ] - 1) term
exp_cosh_delta_term = - gamma_t**2 * exp_1mcosh_GD(gamma_t, delta) # E^(-tΓ) (Cosh[tΓ√Δ] - 1)/Δ term
#exp_1mcosh_GD e^(-Γt) * (1 - cosh(Γt√Δ))/ ( (Γt)**2 Δ )
# Main computation
numerator = exp_cosh_term - exp_cosh_delta_term
denominator = gamma_t * (1 - delta)
result = numerator / denominator
# Handle numerical instability
result = torch.where(torch.isfinite(result), result, torch.zeros_like(result))
# Taylor expansion for small gamma_t and delta near 1
mask = (torch.abs(denominator) < 1e-1)
exp_1mcosh_GD_term = exp_1mcosh_GD(gamma_t, delta**0)
taylor = (
gamma_t*exp_1mcosh_GD_term + 0.5 * gamma_t * exp_sinh_GsqrtD(gamma_t, delta**0)
- denominator / 4 * ( 0.5 * exp_cosh(gamma_t, delta**0) - 4 * exp_1mcosh_GD_term - 5 /2 * exp_sinh_GsqrtD(gamma_t, delta**0) )
)
result = torch.where(mask, taylor, result)
return result
def zeta2(gamma_t, delta):
half_gamma_t = gamma_t / 2
return exp_sinh_GsqrtD(half_gamma_t, delta)
def sig11(gamma_t, delta):
return 1 - torch.exp(-gamma_t) + gamma_t**2 * exp_1mcosh_GD(gamma_t, delta) + exp_sinh_sqrtD(gamma_t, delta)
def Zcoefs(gamma_t, delta):
Zeta1 = zeta1(gamma_t, delta)
Zeta2 = zeta2(gamma_t, delta)
sq_total = 1 - Zeta1 + gamma_t * (delta - 1) * (Zeta1 - 1)**2 / 8
amplitude = torch.sqrt(sq_total)
Zcoef1 = ( gamma_t**0.5 * Zeta2 / 2 **0.5 ) / amplitude
Zcoef2 = Zcoef1 * gamma_t *( - 2 * exp_1mcosh_GD(gamma_t, delta) / sig11(gamma_t, delta) ) ** 0.5
#cterm = exp_cosh_minus_terms(gamma_t, delta)
#sterm = exp_sinh_sqrtD(gamma_t, delta**0) + exp_sinh_sqrtD(gamma_t, delta)
#Zcoef3 = 2 * torch.sqrt( cterm / ( gamma_t * (1 - delta) * cterm + sterm ) )
Zcoef3 = torch.sqrt( torch.maximum(1 - Zcoef1**2 - Zcoef2**2, sq_total.new_zeros(sq_total.shape)) )
return Zcoef1 * amplitude, Zcoef2 * amplitude, Zcoef3 * amplitude, amplitude
def Zcoefs_asymp(gamma_t, delta):
A_t = (gamma_t * (1 - delta) )/4
return epxm1_x(- 2 * A_t)
class StochasticHarmonicOscillator:
"""
Simulates a stochastic harmonic oscillator governed by the equations:
dy(t) = q(t) dt
dq(t) = -Γ A y(t) dt + Γ C dt + Γ D dw(t) - Γ q(t) dt
Also define v(t) = q(t) / √Γ, which is numerically more stable.
Where:
y(t) - Position variable
q(t) - Velocity variable
Γ - Damping coefficient
A - Harmonic potential strength
C - Constant force term
D - Noise amplitude
dw(t) - Wiener process (Brownian motion)
"""
def __init__(self, Gamma, A, C, D):
self.Gamma = Gamma
self.A = A
self.C = C
self.D = D
self.Delta = 1 - 4 * A / Gamma
def sig11(self, gamma_t, delta):
return 1 - torch.exp(-gamma_t) + gamma_t**2 * exp_1mcosh_GD(gamma_t, delta) + exp_sinh_sqrtD(gamma_t, delta)
def sig22(self, gamma_t, delta):
return 1- zeta1(2*gamma_t, delta) + 2 * gamma_t * exp_1mcosh_GD(gamma_t, delta)
def dynamics(self, y0, v0, t):
"""
Calculates the position and velocity variables at time t.
Parameters:
y0 (float): Initial position
v0 (float): Initial velocity v(0) = q(0) / √Γ
t (float): Time at which to evaluate the dynamics
Returns:
tuple: (y(t), v(t))
"""
dummyzero = y0.new_zeros(1) # convert scalar to tensor with same device and dtype as y0
Delta = self.Delta + dummyzero
Gamma_hat = self.Gamma * t + dummyzero
A = self.A + dummyzero
C = self.C + dummyzero
D = self.D + dummyzero
Gamma = self.Gamma + dummyzero
zeta_1 = zeta1( Gamma_hat, Delta)
zeta_2 = zeta2( Gamma_hat, Delta)
EE = 1 - Gamma_hat * zeta_2
if v0 is None:
v0 = torch.randn_like(y0) * D / 2 ** 0.5
#v0 = (C - A * y0)/Gamma**0.5
# Calculate mean position and velocity
term1 = (1 - zeta_1) * (C * t - A * t * y0) + zeta_2 * (Gamma ** 0.5) * v0 * t
y_mean = term1 + y0
v_mean = (1 - EE)*(C - A * y0) / (Gamma ** 0.5) + (EE - A * t * (1 - zeta_1)) * v0
cov_yy = D**2 * t * self.sig22(Gamma_hat, Delta)
cov_vv = D**2 * self.sig11(Gamma_hat, Delta) / 2
cov_yv = (zeta2(Gamma_hat, Delta) * Gamma_hat * D ) **2 / 2 / (Gamma ** 0.5)
# sample new position and velocity with multivariate normal distribution
batch_shape = y0.shape
cov_matrix = torch.zeros(*batch_shape, 2, 2, device=y0.device, dtype=y0.dtype)
cov_matrix[..., 0, 0] = cov_yy
cov_matrix[..., 0, 1] = cov_yv
cov_matrix[..., 1, 0] = cov_yv # symmetric
cov_matrix[..., 1, 1] = cov_vv
# Compute the Cholesky decomposition to get scale_tril
#scale_tril = torch.linalg.cholesky(cov_matrix)
scale_tril = torch.zeros(*batch_shape, 2, 2, device=y0.device, dtype=y0.dtype)
tol = 1e-8
cov_yy = torch.clamp( cov_yy, min = tol )
sd_yy = torch.sqrt( cov_yy )
inv_sd_yy = 1/(sd_yy)
scale_tril[..., 0, 0] = sd_yy
scale_tril[..., 0, 1] = 0.
scale_tril[..., 1, 0] = cov_yv * inv_sd_yy
scale_tril[..., 1, 1] = torch.clamp( cov_vv - cov_yv**2 / cov_yy, min = tol ) ** 0.5
# check if it matches torch.linalg.
#assert torch.allclose(torch.linalg.cholesky(cov_matrix), scale_tril, atol = 1e-4, rtol = 1e-4 )
# Sample correlated noise from multivariate normal
mean = torch.zeros(*batch_shape, 2, device=y0.device, dtype=y0.dtype)
mean[..., 0] = y_mean
mean[..., 1] = v_mean
new_yv = torch.distributions.MultivariateNormal(
loc=mean,
scale_tril=scale_tril
).sample()
return new_yv[...,0], new_yv[...,1]