Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
111f9fdffb | ||
|
|
9479106c1d | ||
|
|
db4bbbe9f2 | ||
|
|
1a3e15964c | ||
|
|
4cc30f8a1d | ||
|
|
d1e609192f | ||
|
|
bf46831196 | ||
|
|
c9d9a79b18 | ||
|
|
175156af86 | ||
|
|
3281946b32 | ||
|
|
83f557ff4c | ||
|
|
5e7fe4d5e4 | ||
|
|
69f2aded4d | ||
|
|
5f1b6d8989 | ||
|
|
7b0d144db9 | ||
|
|
27ebb6e7af | ||
|
|
f148e4b631 | ||
|
|
cc9ec8873a | ||
|
|
565087e8f8 | ||
|
|
ebdc5dcc02 | ||
|
|
9336523576 | ||
|
|
21520b38fb | ||
|
|
868bbaeefe | ||
|
|
3581d6c217 | ||
|
|
d4da280a92 | ||
|
|
d0bcd459cc | ||
|
|
469012d183 | ||
|
|
3f4a5cdb5a | ||
|
|
e0ce52b1be |
@@ -1,18 +1,44 @@
|
||||
<div align="center">
|
||||
|
||||
# LanPaint: Universal Inpainting Sampler with "Think Mode"
|
||||
[](https://arxiv.org/abs/2502.03491)
|
||||
[](https://openreview.net/pdf?id=JPC8JyOUSW)
|
||||
[](https://github.com/scraed/LanPaintBench)
|
||||
[](https://github.com/comfyanonymous/ComfyUI)
|
||||
[](https://huggingface.co/charrywhite/LanPaint)
|
||||
[](https://scraed.github.io/scraedBlog/)
|
||||
[](https://github.com/scraed/LanPaint/stargazers)
|
||||
[](https://discord.gg/aCGZutBV)
|
||||
</div>
|
||||
|
||||
|
||||
Universally applicable inpainting ability for every model. LanPaint sampler lets the model "think" through multiple iterations before denoising, enabling you to invest more computation time for superior inpainting quality.
|
||||
|
||||
This is the official implementation of ["Lanpaint: Training-Free Diffusion Inpainting with Exact and Fast Conditional Inference"](https://arxiv.org/abs/2502.03491), accepted by TMLR. The repository is for ComfyUI extension. Local Python benchmark code is published here: [LanPaintBench](https://github.com/scraed/LanPaintBench).
|
||||
This is the official implementation of ["LanPaint: Training-Free Diffusion Inpainting with Asymptotically Exact and Fast Conditional Sampling"](https://arxiv.org/abs/2502.03491), accepted by TMLR. The repository is for ComfyUI extension. Local Python benchmark code is published here: [LanPaintBench](https://github.com/scraed/LanPaintBench).
|
||||
|
||||
## Citation
|
||||
|
||||
```
|
||||
@article{
|
||||
zheng2025lanpaint,
|
||||
title={LanPaint: Training-Free Diffusion Inpainting with Asymptotically Exact and Fast Conditional Sampling},
|
||||
author={Candi Zheng and Yuan Lan and Yang Wang},
|
||||
journal={Transactions on Machine Learning Research},
|
||||
issn={2835-8856},
|
||||
year={2025},
|
||||
url={https://openreview.net/forum?id=JPC8JyOUSW},
|
||||
note={}
|
||||
}
|
||||
```
|
||||
**🎉 NEW 2026: Join our discord!**
|
||||
|
||||
[Join our Discord](https://discord.gg/aCGZutBV) to share experiences, discuss features, and explore future development.
|
||||
|
||||
**🎬 NEW: LanPaint now supports inpainting and outpainting based on Z-Image!**
|
||||
|
||||
| Original | Masked | Inpainted |
|
||||
|:--------:|:------:|:---------:|
|
||||
|  |  |  |
|
||||
|
||||
|
||||
**🎬 NEW: LanPaint now supports video inpainting and outpainting based on Wan 2.2!**
|
||||
|
||||
@@ -40,10 +66,12 @@ Check our latest [Wan 2.2 Video Examples](#video-examples-beta), [Wan 2.2 Image
|
||||
- [Wan 2.2 Video Outpainting](#wan-22-video-outpainting)
|
||||
- [Resource Consumption](#resource-consumption)
|
||||
- [Image Examples](#image-examples)
|
||||
- [Flux.2.Dev](#example-flux2dev-inpaintlanpaint-k-sampler-5-steps-of-thinking)
|
||||
- [Z-image](#example-z-image-inpaintlanpaint-k-sampler-5-steps-of-thinking)
|
||||
- [Hunyuan T2I](#example-hunyuan-t2i-inpaintlanpaint-k-sampler-5-steps-of-thinking)
|
||||
- [Wan 2.2 T2I](#example-wan22-inpaintlanpaint-k-sampler-5-steps-of-thinking)
|
||||
- [Wan 2.2 T2I with reference](#example-wan22-partial-inpaintlanpaint-k-sampler-5-steps-of-thinking)
|
||||
- [Qwen Image Edit 2509](#example-qwen-edit-2509-inpaint)
|
||||
- [Qwen Image Edit 2511 2509](#example-qwen-edit-2509-inpaint)
|
||||
- [Qwen Image Edit 2508](#example-qwen-edit-2508-inpaint)
|
||||
- [Qwen Image](#example-qwen-image-inpaintlanpaint-k-sampler-5-steps-of-thinking)
|
||||
- [HiDream](#example-hidream-inpaint-lanpaint-k-sampler-5-steps-of-thinking)
|
||||
@@ -62,7 +90,7 @@ Check our latest [Wan 2.2 Video Examples](#video-examples-beta), [Wan 2.2 Image
|
||||
|
||||
## Features
|
||||
|
||||
- **Universal Compatibility** – Works instantly with almost any model (**SD 1.5, XL, 3.5, Flux, HiDream, Qwen-Image, Wan2.2 or custom LoRAs**) and ControlNet.
|
||||
- **Universal Compatibility** – Works instantly with almost any model (**Z-image, Hunyuan, Wan 2.2, Qwen Image/Edit, HiDream, SD 3.5, Flux-series, SDXL, SD 1.5 or custom LoRAs**) and ControlNet.
|
||||

|
||||
- **No Training Needed** – Works out of the box with your existing model.
|
||||
- **Easy to Use** – Same workflow as standard ComfyUI KSampler.
|
||||
@@ -197,7 +225,7 @@ You need to follow the ComfyUI version of [Wan2.2 T2V workflow](https://docs.com
|
||||
**Model Used**: `wan2.2_t2v_low_noise_14B_fp8_scaled.safetensors` and `wan2.2_t2v_high_noise_14B_fp8_scaled.safetensors`.<br>
|
||||
**Processing Steps**: 20 sampling steps x 2 (LanPaint steps of thinking).</sub>
|
||||
|
||||
**Note:** To further reduce VRAM requirements, we recommend loading CLIP on CPU.
|
||||
**Note:** Vram is required by the model, not LanPaint. To further reduce VRAM requirements, we recommend generating less frames and loading CLIP on CPU.
|
||||
|
||||
## Image Examples
|
||||
|
||||
@@ -218,6 +246,33 @@ We are excited to announce that LanPaint now supports Wan2.2 text to image gener
|
||||
|
||||
You need to follow the ComfyUI version of [Wan2.2 T2V workflow](https://docs.comfy.org/tutorials/video/wan/wan2_2) to download and install the T2V model.
|
||||
|
||||
### Example Z-image: InPaint(LanPaint K Sampler, 5 steps of thinking)
|
||||
LanPaint also supports inpainting with the Z-image text-to-image model.
|
||||
|
||||
<details open>
|
||||
<summary>View Original / Masked / Inpainted Comparison</summary>
|
||||
|
||||
| Original | Masked | Inpainted |
|
||||
|:--------:|:------:|:---------:|
|
||||
|  |  |  |
|
||||
|
||||
</details>
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_21)
|
||||
|
||||
<details open>
|
||||
<summary>View Z-image Outpainting (Original / Masked / Outpainted)</summary>
|
||||
|
||||
| Original | Masked | Outpainted |
|
||||
|:--------:|:------:|:----------:|
|
||||
|  |  |  |
|
||||
|
||||
</details>
|
||||
|
||||
[View Outpaint Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_22)
|
||||
|
||||
You can download the Z-image model for ComfyUI from [Z-image](https://docs.comfy.org/zh-CN/tutorials/image/z-image/z-image-turbo).
|
||||
|
||||
### Example Wan2.2: Partial InPaint(LanPaint K Sampler, 5 steps of thinking)
|
||||
Sometimes we don't want to inpaint completely new content, but rather let the inpainted image reference the original image. One option to achieve this is to inpaint with an edit model like Qwen Image Edit. Another option is to perform a partial inpaint: allowing the diffusion process to start at some middle steps rather than from 0.
|
||||
|
||||
@@ -227,8 +282,9 @@ Sometimes we don't want to inpaint completely new content, but rather let the in
|
||||
|
||||
You need to follow the ComfyUI version of [Wan2.2 T2V workflow](https://docs.comfy.org/tutorials/video/wan/wan2_2) to download and install the T2V model.
|
||||
|
||||
|
||||
### Example Qwen Edit 2509: InPaint
|
||||
Check our latest updated [Mased Qwen Edit Workflow](https://github.com/scraed/LanPaint/tree/master/examples/Example_14) for Qwen Image Edit 2509. Download the model at [Qwen Image Edit 2509 Comfy](https://huggingface.co/Comfy-Org/Qwen-Image-Edit_ComfyUI/tree/main/split_files/diffusion_models).
|
||||
Check our latest updated [Mased Qwen Edit Workflow](https://github.com/scraed/LanPaint/tree/master/examples/Example_14) for Qwen Image Edit 2509. Download the model at [Qwen Image Edit 2509 Comfy](https://huggingface.co/Comfy-Org/Qwen-Image-Edit_ComfyUI/tree/main/split_files/diffusion_models). This workflow also supports Qwen Image Edit 2511.
|
||||
|
||||

|
||||
|
||||
@@ -269,6 +325,23 @@ You need to follow the ComfyUI version of [HiDream workflow](https://docs.comfy.
|
||||
|
||||
You need to follow the ComfyUI version of [SD 3.5 workflow](https://comfyui-wiki.com/en/tutorial/advanced/stable-diffusion-3-5-comfyui-workflow) to download and install the model.
|
||||
|
||||
### Example Flux.2.Dev: InPaint(LanPaint K Sampler, 5 steps of thinking)
|
||||
|
||||
<details open>
|
||||
<summary>View Original / Masked / Inpainted Comparison</summary>
|
||||
|
||||
| Original | Masked | Inpainted |
|
||||
|:--------:|:------:|:---------:|
|
||||
|  |  |  |
|
||||
|
||||
</details>
|
||||
|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_23)
|
||||
|
||||
[Model Used in This Example](https://huggingface.co/Comfy-Org/flux2-dev)
|
||||
|
||||
(Note: Prompt First mode is disabled on Flux.2.Dev. As it does not use CFG guidance.)
|
||||
|
||||
### Example Flux: InPaint(LanPaint K Sampler, 5 steps of thinking)
|
||||

|
||||
[View Workflow & Masks](https://github.com/scraed/LanPaint/tree/master/examples/Example_7)
|
||||
@@ -416,17 +489,19 @@ Submit a PR to add your tutorial/video here, or open an [Issue](https://github.c
|
||||
- Try Implement Detailer
|
||||
- ~~Provide inference code on without GUI.~~ Check our local Python benchmark code [LanPaintBench](https://github.com/scraed/LanPaintBench).
|
||||
|
||||
|
||||
## Citation
|
||||
|
||||
```
|
||||
@misc{zheng2025lanpainttrainingfreediffusioninpainting,
|
||||
title={Lanpaint: Training-Free Diffusion Inpainting with Exact and Fast Conditional Inference},
|
||||
author={Candi Zheng and Yuan Lan and Yang Wang},
|
||||
year={2025},
|
||||
eprint={2502.03491},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={eess.IV},
|
||||
url={https://arxiv.org/abs/2502.03491},
|
||||
@article{
|
||||
zheng2025lanpaint,
|
||||
title={LanPaint: Training-Free Diffusion Inpainting with Asymptotically Exact and Fast Conditional Sampling},
|
||||
author={Candi Zheng and Yuan Lan and Yang Wang},
|
||||
journal={Transactions on Machine Learning Research},
|
||||
issn={2835-8856},
|
||||
year={2025},
|
||||
url={https://openreview.net/forum?id=JPC8JyOUSW},
|
||||
note={}
|
||||
}
|
||||
```
|
||||
|
||||
@@ -434,4 +509,3 @@ Submit a PR to add your tutorial/video here, or open an [Issue](https://github.c
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
After Width: | Height: | Size: 1.6 MiB |
|
After Width: | Height: | Size: 1.1 MiB |
|
After Width: | Height: | Size: 1.1 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.9 MiB |
|
After Width: | Height: | Size: 1.2 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.6 MiB |
|
After Width: | Height: | Size: 1.8 MiB |
|
After Width: | Height: | Size: 1.6 MiB |
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "LanPaint"
|
||||
version = "1.4.3"
|
||||
version = "1.4.9"
|
||||
description = "Achieve seamless inpainting results without needing a specialized inpainting model."
|
||||
authors = [
|
||||
{name = "LanPaint", email = "czhengac@connect.ust.hk"}
|
||||
|
||||
@@ -64,7 +64,7 @@ class LanPaint():
|
||||
############ LanPaint Iterations End ###############
|
||||
# out is x_0
|
||||
out, _ = self.inner_model(x, sigma, model_options=model_options, seed=seed)
|
||||
#out = out * (1-latent_mask) + self.latent_image * latent_mask
|
||||
out = out * (1-latent_mask) + self.latent_image * latent_mask
|
||||
return out
|
||||
|
||||
def score_model(self, x_t, y, mask, abt, sigma, tflow, model_options, seed):
|
||||
|
||||
@@ -12,8 +12,28 @@ from comfy.model_base import ModelType
|
||||
from .utils import *
|
||||
from .lanpaint import LanPaint
|
||||
from comfy.model_base import WAN22
|
||||
import comfyui_version
|
||||
import comfy.nested_tensor
|
||||
|
||||
def reshape_mask(input_mask, output_shape,video_inpainting=False):
|
||||
|
||||
import comfy.nested_tensor
|
||||
|
||||
# 修改这里的判断条件,不能只用 hasattr("unbind")
|
||||
if isinstance(input_mask, comfy.nested_tensor.NestedTensor):
|
||||
masks = input_mask.unbind()
|
||||
|
||||
# 如果 output_shape 也是嵌套的(通常 noise.shape 在 NestedTensor 下返回 tuple of shapes)
|
||||
if isinstance(output_shape, (list, tuple)) and len(output_shape) > 0 and not isinstance(output_shape[0], int):
|
||||
reshaped_parts = []
|
||||
for i in range(len(masks)):
|
||||
# 递归处理每一个子部分,并传入对应的子 shape
|
||||
reshaped_parts.append(reshape_mask(masks[i], output_shape[i], video_inpainting))
|
||||
return comfy.nested_tensor.NestedTensor(tuple(reshaped_parts))
|
||||
else:
|
||||
# 如果 output_shape 是单一形状(降级处理)
|
||||
return comfy.nested_tensor.NestedTensor(tuple(reshape_mask(m, output_shape, video_inpainting) for m in masks))
|
||||
|
||||
dims = len(output_shape) - 2
|
||||
print('output shape',output_shape)
|
||||
scale_mode = "nearest-exact"
|
||||
@@ -22,43 +42,81 @@ def reshape_mask(input_mask, output_shape,video_inpainting=False):
|
||||
print('input_mask.ndim:', input_mask.ndim, 'output_shape len:', len(output_shape))
|
||||
|
||||
# Handle video case with temporal dimension
|
||||
if video_inpainting: # Video case: (batch, channels, frames, height, width)
|
||||
target_frames = output_shape[2]
|
||||
target_height, target_width = output_shape[-2:]
|
||||
# if video_inpainting: # Video case: (batch, channels, frames, height, width)
|
||||
# target_frames = output_shape[2]
|
||||
# target_height, target_width = output_shape[-2:]
|
||||
|
||||
print('Video case - input_mask initial shape:', input_mask.shape)
|
||||
# print('Video case - input_mask initial shape:', input_mask.shape)
|
||||
|
||||
# First reshape input_mask to have proper dimensions for video processing
|
||||
# Assume input is (frames, channels, height, width) -> (1, channels, frames, height, width)
|
||||
input_mask = input_mask.permute(1, 0, 2, 3).unsqueeze(0)
|
||||
print('Video case - input_mask after reshaping:', input_mask.shape)
|
||||
# Ensure we have the correct 5D shape: (batch, channels, frames, height, width)
|
||||
batch_size, channels, frames, height, width = input_mask.shape
|
||||
print('Video case - dimensions: batch_size={}, channels={}, frames={}, height={}, width={}'.format(batch_size, channels, frames, height, width))
|
||||
print('Video case - target size:', (target_frames, target_height, target_width))
|
||||
# # First reshape input_mask to have proper dimensions for video processing
|
||||
# # Assume input is (frames, channels, height, width) -> (1, channels, frames, height, width)
|
||||
# ## if comfy version < 0.6.0
|
||||
# if comfyui_version.__version__ < "0.6.0":
|
||||
# input_mask = input_mask.permute(1, 0, 2, 3).unsqueeze(0)
|
||||
# print('Video case - input_mask after reshaping:', input_mask.shape)
|
||||
# # Ensure we have the correct 5D shape: (batch, channels, frames, height, width)
|
||||
# batch_size, channels, frames, height, width = input_mask.shape
|
||||
# print('Video case - dimensions: batch_size={}, channels={}, frames={}, height={}, width={}'.format(batch_size, channels, frames, height, width))
|
||||
# print('Video case - target size:', (target_frames, target_height, target_width))
|
||||
|
||||
# 3D nearest-exact interpolation: (batch, channels, frames, height, width) -> (batch, channels, target_frames, target_height, target_width)
|
||||
temp_mask = torch.nn.functional.interpolate(
|
||||
input_mask,
|
||||
size=(target_frames, target_height, target_width),
|
||||
mode=scale_mode,
|
||||
)
|
||||
# # 3D nearest-exact interpolation: (batch, channels, frames, height, width) -> (batch, channels, target_frames, target_height, target_width)
|
||||
# temp_mask = torch.nn.functional.interpolate(
|
||||
# input_mask,
|
||||
# size=(target_frames, target_height, target_width),
|
||||
# mode=scale_mode,
|
||||
# )
|
||||
|
||||
# temp_mask is already 5D: (batch, channels, target_frames, target_height, target_width)
|
||||
mask = temp_mask
|
||||
print('after mask',mask.shape)
|
||||
# Handle channel dimension expansion if needed
|
||||
if mask.shape[1] < output_shape[1]:
|
||||
mask = mask.repeat(1, output_shape[1], 1, 1, 1)[:, :output_shape[1]]
|
||||
# Handle batch dimension
|
||||
mask = repeat_to_batch_size(mask, output_shape[0])
|
||||
# # temp_mask is already 5D: (batch, channels, target_frames, target_height, target_width)
|
||||
# mask = temp_mask
|
||||
# print('after mask',mask.shape)
|
||||
# # Handle channel dimension expansion if needed
|
||||
# if mask.shape[1] < output_shape[1]:
|
||||
# mask = mask.repeat(1, output_shape[1], 1, 1, 1)[:, :output_shape[1]]
|
||||
# # Handle batch dimension
|
||||
# mask = repeat_to_batch_size(mask, output_shape[0])
|
||||
if video_inpainting:
|
||||
# 如果是 3D Token 序列 (LTXV 压平后的情况)
|
||||
if input_mask.ndim == 3 and len(output_shape) == 3:
|
||||
mask = torch.nn.functional.interpolate(
|
||||
input_mask,
|
||||
size=output_shape[2],
|
||||
mode=scale_mode
|
||||
)
|
||||
return mask
|
||||
|
||||
# 只有在确认为 5D 视频张量时才执行原有逻辑
|
||||
if input_mask.ndim == 5:
|
||||
target_frames = output_shape[2]
|
||||
target_height, target_width = output_shape[-2:]
|
||||
|
||||
# (这里保留你原有的 permute 和 unsqueeze 逻辑,但要确保它是针对非 5D 输入的补救)
|
||||
if input_mask.ndim < 5:
|
||||
# 假设输入是 (F, C, H, W) -> (1, C, F, H, W)
|
||||
if hasattr(comfyui_version, "__version__") and comfyui_version.__version__ < "0.6.0":
|
||||
input_mask = input_mask.permute(1, 0, 2, 3).unsqueeze(0)
|
||||
|
||||
# 现在可以安全地解包 5D 形状了
|
||||
batch_size, channels, frames, height, width = input_mask.shape
|
||||
mask = torch.nn.functional.interpolate(
|
||||
input_mask,
|
||||
size=(target_frames, target_height, target_width),
|
||||
mode=scale_mode,
|
||||
)
|
||||
|
||||
if mask.shape[1] < output_shape[1]:
|
||||
mask = mask.repeat(1, output_shape[1], 1, 1, 1)[:, :output_shape[1]]
|
||||
mask = repeat_to_batch_size(mask, output_shape[0])
|
||||
return mask
|
||||
else: # Original 2D image case
|
||||
mask = torch.nn.functional.interpolate(input_mask, size=output_shape[-2:], mode=scale_mode)
|
||||
if comfyui_version.__version__ < "0.6.0":
|
||||
mask = torch.nn.functional.interpolate(input_mask, size=output_shape[-2:], mode=scale_mode)
|
||||
else:
|
||||
mask = torch.nn.functional.interpolate(input_mask, size=output_shape[2:], mode=scale_mode)
|
||||
if mask.shape[1] < output_shape[1]:
|
||||
mask = mask.repeat((1, output_shape[1]) + (1,) * dims)[:,:output_shape[1]]
|
||||
mask = repeat_to_batch_size(mask, output_shape[0])
|
||||
|
||||
print('resize mask',mask.shape,type(mask),torch.max(mask),torch.min(mask))
|
||||
|
||||
return mask
|
||||
def prepare_mask(noise_mask, shape, device,video_inpainting=False):
|
||||
return reshape_mask(noise_mask, shape,video_inpainting).to(device)
|
||||
@@ -88,9 +146,9 @@ class CFGGuider_LanPaint:
|
||||
if isinstance(self.inner_model, WAN22):
|
||||
print("WAN22 detected")
|
||||
self.inner_model.extra_conds = super(WAN22, self.inner_model).extra_conds
|
||||
|
||||
if denoise_mask is not None:
|
||||
video_inpainting = self.model_options.get("video_inpainting", False)
|
||||
print('denoise_mask',denoise_mask.shape,type(denoise_mask))
|
||||
denoise_mask = prepare_mask(denoise_mask, noise.shape, device, video_inpainting)
|
||||
|
||||
noise = noise.to(device)
|
||||
@@ -138,8 +196,6 @@ class KSamplerX0Inpaint:
|
||||
abt = (1 - Flow_t)**2 / ((1 - Flow_t)**2 + Flow_t**2 )
|
||||
VE_Sigma = Flow_t / (1 - Flow_t)
|
||||
#print("t", torch.mean( sigma ).item(), "VE_Sigma", torch.mean( VE_Sigma ).item())
|
||||
|
||||
|
||||
else:
|
||||
VE_Sigma = sigma
|
||||
abt = 1/( 1+VE_Sigma**2 )
|
||||
@@ -149,6 +205,31 @@ class KSamplerX0Inpaint:
|
||||
if "denoise_mask_function" in model_options:
|
||||
denoise_mask = model_options["denoise_mask_function"](sigma, denoise_mask, extra_options={"model": self.inner_model, "sigmas": self.sigmas})
|
||||
|
||||
if isinstance(denoise_mask, comfy.nested_tensor.NestedTensor):
|
||||
masks = denoise_mask.unbind()
|
||||
xs = x.unbind()
|
||||
latent_imgs = self.latent_image.unbind()
|
||||
noises = self.noise.unbind()
|
||||
|
||||
outs = []
|
||||
# 针对 LTXV,通常 i=0 是视频,i=1 是音频
|
||||
for i in range(len(xs)):
|
||||
m = (masks[i] > 0.5).float()
|
||||
lm = 1 - m
|
||||
# 这里的 PaintMethod 通常只支持普通 Tensor,所以我们分块处理
|
||||
# 注意:如果音频部分不需要 Inpaint,可以增加判断
|
||||
current_times = (VE_Sigma, abt, Flow_t)
|
||||
|
||||
# 只有视频部分 (i=0) 应用 LanPaint 逻辑,音频部分通常直接 pass 或原样返回
|
||||
if i == 0:
|
||||
out_part = self.PaintMethod(xs[i], latent_imgs[i], noises[i], sigma, lm, current_times, model_options, seed)
|
||||
else:
|
||||
# 音频部分如果没有对应的 Inpaint 逻辑,通常直接调用 inner_model
|
||||
out_part, _ = self.inner_model(xs[i], sigma, model_options=model_options, seed=seed)
|
||||
outs.append(out_part)
|
||||
|
||||
return comfy.nested_tensor.NestedTensor(tuple(outs))
|
||||
|
||||
denoise_mask = (denoise_mask > 0.5).float()
|
||||
|
||||
latent_mask = 1 - denoise_mask
|
||||
@@ -183,6 +264,7 @@ class KSAMPLER(comfy.samplers.KSAMPLER):
|
||||
#noise here is a randn noise from comfy.sample.prepare_noise
|
||||
#latent_image is the latent image as input of the KSampler node. For inpainting, it is the masked latent image. Otherwise it is zero tensor.
|
||||
extra_args["denoise_mask"] = denoise_mask
|
||||
print("LanPaint KSampler start sampler_function",denoise_mask.shape if denoise_mask is not None else None)
|
||||
model_k = KSamplerX0Inpaint(model_wrap, sigmas)
|
||||
model_k.latent_image = latent_image
|
||||
if self.inpaint_options.get("random", False): #TODO: Should this be the default?
|
||||
@@ -409,8 +491,16 @@ class MaskBlend:
|
||||
# for each mask pixel, find out the nearest 1 pixel, and set the mask value to the distance between the two pixels
|
||||
# check the size of mask and image1, image2, if not the same, assert error
|
||||
if image1.shape[1] != image2.shape[1] or image1.shape[2] != image2.shape[2]:
|
||||
raise ValueError("Make sure your image size is a multiple of 8. Otherwise the mask will not be aligned with the output image.")
|
||||
|
||||
raise ValueError(
|
||||
"Image size mismatch: Image1 and Image2 must have the same dimensions.\n"
|
||||
"Additionally, ensure both images have width and height that are multiples of 8.\n"
|
||||
"This is required because VAE decode always generates images with dimensions that are multiples of 8.\n"
|
||||
"If your input images are not multiples of 8, a size mismatch will occur during the decoding process.\n"
|
||||
"Please resize your images using an image resize node to ensure compatibility.\n"
|
||||
"Current sizes - Image1: {}x{}, Image2: {}x{}".format(
|
||||
image1.shape[2], image1.shape[1], image2.shape[2], image2.shape[1]
|
||||
)
|
||||
)
|
||||
mask = mask.float()
|
||||
mask = torch.nn.functional.max_pool2d(mask, kernel_size=blend_overlap, stride=1, padding=blend_overlap//2)
|
||||
# apply Gaussian blur with kernel size blend_overlap
|
||||
@@ -439,6 +529,77 @@ class MaskBlend:
|
||||
|
||||
return kernel
|
||||
|
||||
class MaskBlendAlpha:
|
||||
"""
|
||||
Create an RGBA image by writing the mask into the PNG alpha channel.
|
||||
|
||||
Requirement:
|
||||
- inpaint region: alpha = 0 (transparent)
|
||||
- other region: alpha = 1 (opaque)
|
||||
|
||||
This node writes the mask into the PNG alpha channel.
|
||||
Current default behavior matches the previous `invert_mask=True` behavior:
|
||||
alpha = mask.
|
||||
"""
|
||||
def __init__(self):
|
||||
pass
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(s):
|
||||
return {
|
||||
"required": {
|
||||
"image": ("IMAGE", {"tooltip": "VAE-decoded image (RGB)."}),
|
||||
"mask": ("MASK", {"tooltip": "Mask used as alpha channel (alpha = mask)."}),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("IMAGE",)
|
||||
FUNCTION = "to_rgba"
|
||||
CATEGORY = "image/postprocessing"
|
||||
|
||||
def to_rgba(self, image: torch.Tensor, mask: torch.Tensor):
|
||||
"""
|
||||
image: [B,H,W,3] float in [0,1]
|
||||
mask: [B,H,W] (or [H,W]) float in [0,1] used as alpha
|
||||
returns RGBA image: [B,H,W,4] float in [0,1]
|
||||
"""
|
||||
if image.ndim != 4 or image.shape[-1] != 3:
|
||||
raise ValueError(f"Expected IMAGE tensor [B,H,W,3], got {tuple(image.shape)}")
|
||||
|
||||
# Normalize mask shape to [B,H,W]
|
||||
if mask.ndim == 2:
|
||||
mask = mask.unsqueeze(0)
|
||||
elif mask.ndim == 3:
|
||||
pass
|
||||
else:
|
||||
# Some pipelines may carry mask as [B,1,H,W]
|
||||
if mask.ndim == 4 and mask.shape[1] == 1:
|
||||
mask = mask[:, 0, :, :]
|
||||
else:
|
||||
raise ValueError(f"Expected MASK tensor [B,H,W] or [H,W], got {tuple(mask.shape)}")
|
||||
|
||||
b, h, w, _ = image.shape
|
||||
|
||||
# Batch align
|
||||
if mask.shape[0] != b:
|
||||
if mask.shape[0] == 1:
|
||||
mask = mask.repeat(b, 1, 1)
|
||||
else:
|
||||
raise ValueError(f"Batch mismatch: image batch={b}, mask batch={mask.shape[0]}")
|
||||
|
||||
# Spatial align (resize mask to image resolution if needed)
|
||||
if mask.shape[1] != h or mask.shape[2] != w:
|
||||
mask_4d = mask.unsqueeze(1) # [B,1,H,W]
|
||||
mask_4d = torch.nn.functional.interpolate(mask_4d, size=(h, w), mode="nearest")
|
||||
mask = mask_4d[:, 0, :, :]
|
||||
|
||||
mask = mask.float().clamp(0.0, 1.0)
|
||||
|
||||
# Default behavior (matches previous invert_mask=True path):
|
||||
# alpha = mask
|
||||
rgba = torch.cat([image, mask.unsqueeze(-1)], dim=-1)
|
||||
return (rgba,)
|
||||
|
||||
class Noise_EmptyNoise:
|
||||
def generate_noise(self, latent):
|
||||
return torch.zeros_like(latent["samples"])
|
||||
@@ -467,6 +628,7 @@ class LanPaint_SamplerCustom:
|
||||
"LanPaint_NumSteps": ("INT", {"default": 5, "min": 0, "max": 100, "tooltip": "Number of steps for Langevin dynamics, representing turns of thinking per step."}),
|
||||
"LanPaint_PromptMode": (["Image First", "Prompt First"], {"tooltip": "Image First: prioritizes image quality; Prompt First: prioritizes prompt adherence."}),
|
||||
"LanPaint_Info": ("STRING", {"default": "LanPaint Custom Sampler. For more info, visit https://github.com/scraed/LanPaint. If you find it useful, please give a star ⭐️!", "multiline": True}),
|
||||
"Inpainting_mode": (["🖼️ Image Inpainting", "🎬 Video Inpainting"], {"default": "🖼️ Image Inpainting", "tooltip": "Choose Image mode for photos or Video mode for video frames with temporal consistency"}),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -475,7 +637,7 @@ class LanPaint_SamplerCustom:
|
||||
FUNCTION = "sample"
|
||||
CATEGORY = "sampling/custom_sampling"
|
||||
|
||||
def sample(self, model, sampler, sigmas, add_noise, noise_seed, cfg, positive, negative, latent_image, LanPaint_NumSteps, LanPaint_PromptMode, LanPaint_Info=""):
|
||||
def sample(self, model, sampler, sigmas, add_noise, noise_seed, cfg, positive, negative, latent_image, LanPaint_NumSteps, LanPaint_PromptMode, LanPaint_Info="",Inpainting_mode="🖼️ Image Inpainting"):
|
||||
model.LanPaint_StepSize = 0.2
|
||||
model.LanPaint_Lambda = 16.0
|
||||
model.LanPaint_Beta = 1.
|
||||
@@ -486,6 +648,10 @@ class LanPaint_SamplerCustom:
|
||||
model.LanPaint_cfg_BIG = cfg
|
||||
else:
|
||||
model.LanPaint_cfg_BIG = 0 * cfg - 0.5
|
||||
video_inpainting = (Inpainting_mode == "🎬 Video Inpainting")
|
||||
if not hasattr(model, 'model_options') or model.model_options is None:
|
||||
model.model_options = {}
|
||||
model.model_options["video_inpainting"] = video_inpainting
|
||||
with override_sample_function():
|
||||
latent = latent_image.copy()
|
||||
latent_image = latent["samples"]
|
||||
@@ -533,6 +699,7 @@ class LanPaint_SamplerCustomAdvanced:
|
||||
"LanPaint_PromptMode": (["Image First", "Prompt First"], {"tooltip": "Image First: prioritizes image quality; Prompt First: prioritizes prompt adherence."}),
|
||||
"LanPaint_EarlyStop": ("INT", {"default": 1, "min": 0, "max": 10000, "tooltip": "Steps to stop LanPaint early, preventing irregular patterns."}),
|
||||
"LanPaint_Info": ("STRING", {"default": "LanPaint Custom Sampler Adv. For more info, visit https://github.com/scraed/LanPaint. If you find it useful, please give a star ⭐️!", "multiline": True}),
|
||||
"Inpainting_mode": (["🖼️ Image Inpainting", "🎬 Video Inpainting"], {"default": "🖼️ Image Inpainting", "tooltip": "Choose Image mode for photos or Video mode for video frames with temporal consistency"}),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -543,7 +710,7 @@ class LanPaint_SamplerCustomAdvanced:
|
||||
|
||||
CATEGORY = "sampling/custom_sampling"
|
||||
|
||||
def sample(self, noise, guider, sampler, sigmas, latent_image, LanPaint_NumSteps, LanPaint_Lambda, LanPaint_StepSize, LanPaint_Beta, LanPaint_Friction, LanPaint_PromptMode, LanPaint_EarlyStop, LanPaint_Info=""):
|
||||
def sample(self, noise, guider, sampler, sigmas, latent_image, LanPaint_NumSteps, LanPaint_Lambda, LanPaint_StepSize, LanPaint_Beta, LanPaint_Friction, LanPaint_PromptMode, LanPaint_EarlyStop, LanPaint_Info="",Inpainting_mode="🖼️ Image Inpainting"):
|
||||
model = guider.model_patcher
|
||||
model.LanPaint_StepSize = LanPaint_StepSize
|
||||
model.LanPaint_Lambda = LanPaint_Lambda
|
||||
@@ -555,16 +722,25 @@ class LanPaint_SamplerCustomAdvanced:
|
||||
model.LanPaint_cfg_BIG = guider.cfg
|
||||
else:
|
||||
model.LanPaint_cfg_BIG = 0 * guider.cfg - 0.5
|
||||
|
||||
video_inpainting = (Inpainting_mode == "🎬 Video Inpainting")
|
||||
if not hasattr(model, 'model_options') or model.model_options is None:
|
||||
model.model_options = {}
|
||||
model.model_options["video_inpainting"] = video_inpainting
|
||||
with override_sample_function():
|
||||
latent = latent_image
|
||||
latent_image = latent["samples"]
|
||||
print('before fix_empty_latent_channels latent_image shape',latent_image.shape)
|
||||
latent = latent.copy()
|
||||
latent_image = comfy.sample.fix_empty_latent_channels(guider.model_patcher, latent_image)
|
||||
latent["samples"] = latent_image
|
||||
|
||||
print('latent_image shape',latent_image.shape)
|
||||
print('outside noise_mask',latent["noise_mask"].shape if "noise_mask" in latent else 'no noise_mask')
|
||||
print('latent keys',latent.keys())
|
||||
noise_mask = None
|
||||
if "noise_mask" in latent:
|
||||
noise_mask = latent["noise_mask"]
|
||||
print('inside noise_mask shape',noise_mask.shape)
|
||||
|
||||
x0_output = {}
|
||||
callback = latent_preview.prepare_callback(guider.model_patcher, sigmas.shape[-1] - 1, x0_output)
|
||||
@@ -580,6 +756,7 @@ class LanPaint_SamplerCustomAdvanced:
|
||||
out_denoised["samples"] = guider.model_patcher.model.process_latent_out(x0_output["x0"].cpu())
|
||||
else:
|
||||
out_denoised = out
|
||||
# print('output',out.keys(),out["samples"].shape,out['noise_mask'].shape)
|
||||
return (out, out_denoised)
|
||||
|
||||
|
||||
@@ -591,6 +768,7 @@ NODE_CLASS_MAPPINGS = {
|
||||
"LanPaint_SamplerCustom" : LanPaint_SamplerCustom,
|
||||
"LanPaint_SamplerCustomAdvanced" : LanPaint_SamplerCustomAdvanced,
|
||||
"LanPaint_MaskBlend": MaskBlend,
|
||||
"LanPaint_MaskBlendAlpha": MaskBlendAlpha,
|
||||
# "LanPaint_UpSale_LatentNoiseMask": LanPaint_UpSale_LatentNoiseMask,
|
||||
}
|
||||
|
||||
@@ -601,5 +779,6 @@ NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"LanPaint_SamplerCustom" : "LanPaint Sampler Custom",
|
||||
"LanPaint_SamplerCustomAdvanced" : "LanPaint Sampler Custom (Advanced)",
|
||||
"LanPaint_MaskBlend": "LanPaint Mask Blend",
|
||||
"LanPaint_MaskBlendAlpha": "MaskBlend (alpha)",
|
||||
# "LanPaint_UpSale_LatentNoiseMask": "LanPaint UpSale Latent Noise Mask"
|
||||
}
|
||||
|
||||