Mask context padding + downscale-only vision cap

- mask_padding: grow the mask crop box by a fraction of image size per side
  for surrounding context (0 = tight crop, default)
- vision resize is now downscale-only (a small mask crop keeps native size
  instead of being magnified to the megapixel target)
- README: document mask_padding and the megapixel cap

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Ethanfel
2026-06-25 03:42:17 +02:00
co-authored by Claude Opus 4.8
parent 1317e81381
commit baeb8496e5
2 changed files with 33 additions and 9 deletions
+6 -2
View File
@@ -31,7 +31,10 @@ image+mask pairs** (connect `image1`, a fresh `image2`/`mask2` pair appears, and
Each reference image has an optional companion mask. When connected, the image is
**cropped to the mask's bounding box** before the vision encoder, so the VLM only sees the
masked region. This is *reference-image* masking — **not inpainting**: Krea2 has no
masked region. Use **`mask_padding`** to keep surrounding context: it grows the crop box by
that fraction of the image size on each side (`0` = tight crop, `0.1` ≈ 10% margin, high
values ≈ the whole image). The mask is used only to compute the crop — it is not itself
sent to the VLM. This is *reference-image* masking — **not inpainting**: Krea2 has no
concat/inpaint pathway to regenerate a masked region of the output. (To spatially restrict
where the *prompt* applies in the output, use ComfyUI's standard `ConditioningSetMask`
downstream of this node — that works generically at the sampler level.)
@@ -44,7 +47,8 @@ downstream of this node — that works generically at the sampler level.)
| `prompt` | STRING | Your text prompt. |
| `image1…N` | IMAGE | Optional reference images; slots grow as you connect them. |
| `mask1…N` | MASK | Optional per-image mask; crops `imageN` to the masked bounding box. |
| `vision_megapixels` | FLOAT | Area each image is resized to before the vision encoder (default `1.0`). |
| `mask_padding` | FLOAT | Context kept around the mask, as a fraction of image size per side (`0` = tight, default). |
| `vision_megapixels` | FLOAT | Max size before the vision encoder; references are downscaled to this cap, never upscaled (default `1.0`). |
Output is a standard `CONDITIONING` for the Krea2 sampler. With no images connected it
works as a plain Krea2 text encoder.
+27 -7
View File
@@ -50,8 +50,15 @@ class TextEncodeKrea2:
"mask1": ("MASK",),
"vision_megapixels": ("FLOAT", {
"default": 1.0, "min": 0.1, "max": 8.0, "step": 0.1,
"tooltip": "Target size (in megapixels) each reference image is area-resized to "
"before the Qwen3-VL vision encoder. Higher = more visual detail, more VRAM.",
"tooltip": "Maximum size (in megapixels) for each reference before the Qwen3-VL "
"vision encoder. References larger than this are downscaled; smaller "
"ones (e.g. a tight mask crop) are kept at native size, never upscaled.",
}),
"mask_padding": ("FLOAT", {
"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.02,
"tooltip": "Context kept around the mask before cropping, as a fraction of the "
"image size added on EACH side. 0 = tight crop to the mask; 0.1 = ~10% "
"margin of surroundings. Only applies when a mask is connected.",
}),
},
}
@@ -74,8 +81,9 @@ class TextEncodeKrea2:
return out
@staticmethod
def _crop_to_mask(image, mask):
"""Crop image (B,H,W,C) to the bounding box of mask. No-op if mask empty/None."""
def _crop_to_mask(image, mask, padding=0.0):
"""Crop image (B,H,W,C) to the mask bounding box, expanded by `padding` (a
fraction of the image size) on each side. No-op if mask empty/None."""
if mask is None:
return image
@@ -97,9 +105,18 @@ class TextEncodeKrea2:
cols = torch.where(torch.any(presence, dim=0))[0]
y0, y1 = int(rows[0]), int(rows[-1])
x0, x1 = int(cols[0]), int(cols[-1])
if padding > 0.0: # grow the box outward for surrounding context, clamped to the image
pad_x = round(padding * w)
pad_y = round(padding * h)
x0 = max(0, x0 - pad_x)
x1 = min(w - 1, x1 + pad_x)
y0 = max(0, y0 - pad_y)
y1 = min(h - 1, y1 + pad_y)
return image[:, y0:y1 + 1, x0:x1 + 1, :]
def encode(self, clip, prompt, vision_megapixels=1.0, **kwargs):
def encode(self, clip, prompt, vision_megapixels=1.0, mask_padding=0.0, **kwargs):
images = self._collect_indexed(kwargs, "image")
masks = self._collect_indexed(kwargs, "mask")
ordered = sorted(images.keys())
@@ -109,10 +126,13 @@ class TextEncodeKrea2:
total = int(vision_megapixels * 1024 * 1024)
for slot, n in enumerate(ordered):
image = self._crop_to_mask(images[n], masks.get(n))
image = self._crop_to_mask(images[n], masks.get(n), padding=mask_padding)
samples = image.movedim(-1, 1)
scale_by = math.sqrt(total / (samples.shape[3] * samples.shape[2]))
# vision_megapixels is an upper CAP, not a fixed target: only downscale
# oversized references, never upscale. Otherwise a small mask crop gets
# magnified to fill the VLM frame and the subject reads as huge/zoomed.
scale_by = min(1.0, math.sqrt(total / (samples.shape[3] * samples.shape[2])))
width = round(samples.shape[3] * scale_by)
height = round(samples.shape[2] * scale_by)