356 lines
19 KiB
Markdown
356 lines
19 KiB
Markdown
# BFS Sampler (crop · mask · loop)
|
||
|
||
Runs a guide-driven LTX IC-LoRA and adds three things around it, each optional.
|
||
Connect only what you need — the node degrades to whatever is wired.
|
||
|
||
| what you connect | what you get |
|
||
|---|---|
|
||
| `guide_video` | the LoRA over the whole clip, one pass |
|
||
| `+ identity_image` | for a LoRA that takes a reference (head swap, identity transfer) |
|
||
| `+ subject_mask` | only the masked region is denoised |
|
||
| `+ crop_mode` | the edit runs inside a box around the subject, then is pasted back |
|
||
| `+ temporal_tile_size` | long clips sampled in chunks with overlap |
|
||
| `+ auto_config` | crop, mask and feather amounts measured off the subject itself |
|
||
|
||
`identity_image` is optional as of 1.38.0. A head-swap LoRA wants one; an
|
||
instruction-edit, restyle or sharpening LoRA works from the guide alone, and
|
||
with the slot empty it is simply not packed — the guide becomes the only
|
||
reference. Everything else in the node is indifferent to which kind it is.
|
||
|
||
The LoRA is never asked to understand a mask or a crop. It keeps doing its one
|
||
job: the guide rides on `source_id 1`, the identity on `source_id 2`. Everything
|
||
else here is plumbing around it.
|
||
|
||
## `auto_config`: let the mask set the amounts
|
||
|
||
Every amount in this node is in pixels — `mask_grow`, `mask_blur`,
|
||
`uncrop_feather`, and the latent dilation — and a pixel only means something
|
||
relative to how big the head is in the frame. The same `mask_grow: 8` is
|
||
generous slack on a head 90 px wide and invisible on one 900 px wide. That
|
||
mismatch is what leaves a visible seam when the crop goes back into the frame:
|
||
the amounts were tuned on one shot and the next shot has the subject at a
|
||
different distance.
|
||
|
||
Turn `auto_config` on and the node measures the subject in `subject_mask` — its
|
||
width, its height, how far it travels, how much its size changes, how fast it
|
||
moves between frames — and sets all of it from those numbers:
|
||
|
||
| knob | derived as | why |
|
||
|---|---|---|
|
||
| `crop_mode` | `off` when the head is ≥45% of the frame width; `zoomed` when its size changes >35%; `tracked` when it travels more than a quarter of the box; else `combined` | a crop buys resolution the head does not already have, and nothing else |
|
||
| `crop_scale` | 1.8, clamped so the box still fits in the frame | head plus neck and shoulders — a face-tight crop is a framing the LoRA never saw |
|
||
| `mask_grow` | 6% of the head's **width**, plus the capped headroom slack — **per side**: up ≫ sideways ≫ down | hair overflows upward and sideways; growing down only eats the neck and collar, which stay the guide's |
|
||
| `mask_blur` | 3% of the head's width | paste-back softness (the denoise mask is binarised anyway) |
|
||
| `uncrop_feather` | half the margin between mask and crop border, floored | **the seam fix**: a ramp wider than that margin fades the head itself |
|
||
| `latent_mask_dilate` | left at 0 | the pixel grow already carries the slack, and the pixel→latent reduction is a MAX, so any cell the mask touches is editable already — adding cells here stacks a whole 32 px on every side |
|
||
| `latent_mask_dilate_frames` | 1 when the head moves more than 15% of its width between frames | a fast head needs slack along time too |
|
||
|
||
### The mask is the old head, not the new one
|
||
|
||
This is the failure the mode is really for. `subject_mask` marks the head that
|
||
is being *replaced*. The reference head can be wider, or have more hair, or a
|
||
taller cut — and wherever it lands outside the mask, the swap is cut off at the
|
||
mask edge. Growing the mask by a fixed 6% does not fix that; it just moves the
|
||
clip.
|
||
|
||
Two things are done about it. The reference's **proportions** are measured —
|
||
`identity_image`'s aspect against the mask's own — and used to decide *where*
|
||
the slack goes: a relatively wider reference gets it at the sides, a taller one
|
||
above. The reference's **size** cannot be measured at all: a cropped head
|
||
carries no scale, and nothing in the graph says whether that crop is a 200 px
|
||
head or a 900 px one. That part is `identity_headroom`, default 1.15 — "the new
|
||
head may be up to 15% bigger than the old one". Raise it for big hair or a
|
||
visibly larger head, drop it to 1.0 when the two heads match.
|
||
|
||
Growth is never symmetric: down is held at the base 6% no matter what, because
|
||
the neck and collar below the jaw have to stay the guide's own pixels.
|
||
|
||
Every slack is a fraction of the head's **width** and capped there (25% sideways,
|
||
30% up). Width is the stable dimension of a head mask — height swings with how
|
||
much neck the mask happened to take — and scaling the upward slack by height,
|
||
as 1.37.0 did, produced holes far larger than the head, with the new head
|
||
generated floating inside the regenerated area. Fixed in 1.37.1.
|
||
|
||
It needs `subject_mask`; without one there is nothing to measure and the widget
|
||
values are kept. The widgets it overrides stay visible but are ignored, and the
|
||
`debug` output prints exactly what it chose:
|
||
|
||
```
|
||
auto: head 120x160px in 1920x1080 (6% of the width), seen in 73/73 frames;
|
||
reference aspect 1.00 vs mask 0.75 (tilt 1.33), headroom 1.15 -> crop tracked @ 1.8
|
||
(the subject travels 1296px, more than a quarter of the box); grow 19px sideways,
|
||
32px up, 7px down, blur 4px, feather 14px, latent dilate 1 cell(s), 0 frame(s)
|
||
```
|
||
|
||
Read that line before trusting it. It is a starting point measured from your own
|
||
shot, not a guarantee — a mask that leaks onto the shoulders makes the "head"
|
||
wider and every amount grows with it.
|
||
|
||
## The mask is native inpainting, not a LoRA feature
|
||
|
||
The mask becomes `latent["noise_mask"]`, which the guider consumes as
|
||
`denoise_mask` — ComfyUI's own inpainting path. Pixels outside the mask keep the
|
||
guide's own content, so **the original face stays visible to the model**. This is
|
||
deliberately different from a masked-hole recipe, where the region is painted out
|
||
before the model sees it.
|
||
|
||
When a mask is connected the node seeds the sampling latent with the
|
||
**VAE-encoded guide**, not with the empty latent you wired in — inpainting keeps
|
||
the initial latent outside the mask, and an empty one leaves grey where the video
|
||
should be. The connected latent still sets the size (and carries the audio
|
||
stream); its video content is replaced.
|
||
|
||
Masks are reduced to the latent grid with **max**, not with ComfyUI's trilinear
|
||
resize. Trilinear blurs a mask across frames and lets the original content bleed
|
||
through the edit. Max keeps a latent cell that any masked pixel touches fully
|
||
editable.
|
||
|
||
### Does the mask need to be semi-transparent so the LoRA can see the expression?
|
||
|
||
No — and it would cost you the swap. The expression does not travel under the
|
||
mask: the whole source clip, original face included, is in the conditioning as an
|
||
aligned reference on `source_id 1`, frame by frame. The model reads the
|
||
performance from there whatever the mask does. That is what the LoRA was trained
|
||
on: the new head follows the old head's acting.
|
||
|
||
Partial mask values do something else. In ComfyUI's inpaint path they blend the
|
||
original latent back in, so you keep the original geometry in pixels **and the
|
||
original identity with it** — the result drifts toward an average of both faces.
|
||
That is what `mask_strength` below 1.0 buys, and it is a deliberate choice, not a
|
||
way to get expressions. Soft edges are for hiding the seam; the middle of the
|
||
mask should stay at 1.0.
|
||
|
||
### Hard for the sampler, soft for the paste
|
||
|
||
`mask_blur` exists for compositing, where a soft edge hides the seam. In the
|
||
**denoise** mask a soft edge means *partial* denoising, which blends the original
|
||
latent — and the original identity — back in right at the edge of the head, so
|
||
the swap is weakest exactly where it is most visible. `mask_hard_for_inpaint`
|
||
(on by default) binarises the mask before it becomes the denoise mask, leaving
|
||
the soft version for the paste-back alone.
|
||
|
||
`latent_mask_dilate` grows the mask by whole **latent cells** after the
|
||
reduction. One cell is 32 px and one latent frame covers 8 video frames, so this
|
||
is far coarser than `mask_grow` in pixels — and it is what guarantees the head
|
||
sits inside editable blocks rather than clipping at a cell boundary. One cell
|
||
took a test mask from 11% to 25% of the grid; two took it to 44%. Watch
|
||
`latent_mask` and stop at the first value that covers the head.
|
||
|
||
**Watch the `latent_mask` output.** It has one frame per *latent* frame, not per
|
||
video frame, and that is the real resolution of the edit. A tight outline around
|
||
a face can nearly vanish at that scale — and then nothing changes, no matter how
|
||
good the LoRA is. If the mask looks thin there, raise `mask_grow`.
|
||
|
||
## Applying a LoRA to one region and putting it back
|
||
|
||
The shape of the thing: cut the region out as its own small clip, run the LoRA
|
||
on it, composite it back. What makes the result honest is that the composite
|
||
goes into the **original frames**, so every pixel outside the mask is the
|
||
source's own — it never went through the VAE at all.
|
||
|
||
```
|
||
subject_mask connected
|
||
inpaint_with_mask true # outside the mask the latent IS the guide, not noise
|
||
mask_hard_for_inpaint true # a soft DENOISE mask means partial denoising at the edge
|
||
mask_strength 1.0 # below this the original is blended back INSIDE the mask
|
||
paste_back true
|
||
paste_confine_to_mask true # only the mask reaches the frame
|
||
latent_mask_dilate 0 # one cell is 32 px of extra regenerated area, per side
|
||
mask_grow the least that fits the new content
|
||
mask_blur 3-5% of the subject's width — the paste's soft edge
|
||
```
|
||
|
||
Since 1.39.0 this works with `crop_mode: off` too. Before that, with no crop the
|
||
node returned the decoded frame whole: nothing *changed* outside the mask, but
|
||
everything had been through an encode/decode round trip and came back softer.
|
||
Now the mask composites against the source either way, and the crop goes back to
|
||
being purely a resolution decision.
|
||
|
||
**Two passes on the region.** Set `paste_back: false` and the images and latent
|
||
stay in the crop's own space, so you can upscale and refine there — where the
|
||
region actually has pixels — and composite at the end with **BFS Paste Back**,
|
||
feeding it the `crop_bboxes` output. That is the whole point of `crop_bboxes`
|
||
being a socket.
|
||
|
||
## Why crop, and which mode
|
||
|
||
At 512×288 a person filling a fifth of the frame leaves a face about 25 px tall.
|
||
No LoRA recovers identity from 25 px — it is missing pixels, not missing
|
||
training. Cropping the head region and sampling it full-frame gives that same
|
||
face 200–300 px, and the result is feathered back into untouched frames.
|
||
|
||
| `crop_mode` | behaviour |
|
||
|---|---|
|
||
| `off` | the whole frame is sampled. Use it first, to confirm the swap works at all. |
|
||
| `combined` | one static box around the subject's whole travel, held for the clip. No dependency. |
|
||
| `tracked` | constant-size box that stays still until the subject would leave it, then moves as little as possible. |
|
||
| `zoomed` | the box also follows the subject's size, planned over the whole clip, every crop resampled to one output size. |
|
||
|
||
`tracked` and `zoomed` use a planner vendored from
|
||
[drozbay/MaskVidExperiments](https://github.com/drozbay/MaskVidExperiments)
|
||
(GPL-3.0, the same licence this pack carries) — nothing extra to install. Those
|
||
boxes hold still through mask noise and occlusion, and **a jittering crop reads
|
||
to a video model as camera motion**, which is what a naive per-frame crop around
|
||
a mask produces.
|
||
|
||
`crop_scale` is the box as a multiple of the subject. At 1.5 the subject occupies
|
||
two thirds and the rest is margin. Keep neck and shoulders inside: a face-tight
|
||
crop is a framing the LoRA never saw in training, and it will show.
|
||
|
||
## Sizing, which is the one fiddly part
|
||
|
||
**Since 1.41.1 the shape is handled for you.** The crop is resized into the
|
||
connected latent, so the box has to share its *aspect* — the sampler now asks
|
||
the planner for exactly the latent's aspect ratio and says so in `debug`:
|
||
|
||
```
|
||
crop shaped to the latent's 1.500 aspect
|
||
```
|
||
|
||
Without it a mask around a face gives a portrait box, sampling it in a landscape
|
||
latent stretches it, and the paste-back squeezes the result back — the face comes
|
||
out deformed, and under `crop_mode: zoomed` by a different amount every frame.
|
||
|
||
The latent's **size** is deliberately left to you, because that is the sampling
|
||
resolution and it is the entire reason to crop: a face 200 px tall in the source
|
||
sampled at 768 px wide is a face with pixels to work with, and the paste-back
|
||
brings it home at the box's size. Choose the size for the quality you want; the
|
||
shape is no longer yours to get right.
|
||
|
||
**BFS Crop Size** is still there for when you want the number, or the per-frame
|
||
boxes, upstream of the sampler — building the latent at the right size skips the
|
||
rebuild, and the boxes feed BFS Paste Back directly:
|
||
|
||
```
|
||
subject_mask ──┬─→ BFS Crop Size ──→ width / height ──→ EmptyLTXVLatentVideo ──┐
|
||
│ │
|
||
└───────────────────────────────────────────→ BFS Sampler ←─────┘
|
||
```
|
||
|
||
Keep its three crop widgets identical to the sampler's — the planner is
|
||
deterministic, so equal inputs give the same box, and a mismatch means the
|
||
sampler silently resizes the crop into a latent that does not fit it.
|
||
|
||
That resize is worth naming, because it does not look like a sizing problem when
|
||
it happens. A mask around a face in a 1920x1080 frame gives a portrait box —
|
||
512x672, say. Sampling it into a 768x512 landscape latent stretches the crop
|
||
horizontally, and the paste-back then squeezes the result back into the portrait
|
||
box. The face comes out visibly deformed, and with `crop_mode: zoomed` the
|
||
amount of deformation changes from frame to frame.
|
||
|
||
### Doing it by hand
|
||
|
||
|
||
|
||
The connected `latent` sets the sampled size. When you crop, the box is smaller
|
||
than the frame, so the latent should match the box — the `debug` output prints
|
||
the size to use:
|
||
|
||
```
|
||
sample size 416x768 (connect EmptyLTXVLatentVideo at this size to avoid a resize)
|
||
```
|
||
|
||
If they disagree the node resamples the crop on the way back, which costs exactly
|
||
the sharpness the crop was there to buy. Run once, read `debug`, set
|
||
`EmptyLTXVLatentVideo`, run again.
|
||
|
||
**The latent must come from the graph.** LTX‑2.5 latents are AV — video and audio
|
||
nested together — and the node cannot fabricate that structure. Without one it
|
||
builds a plain video latent, warns, and the guide may not reach the model at all.
|
||
|
||
## Chunking
|
||
|
||
`temporal_tile_size` at 0 samples the clip in one pass. Set it to the length the
|
||
LoRA trained at (73 for the LTX head-swap recipe) for longer clips, and the node
|
||
slices the guide, the mask and the overlap together per chunk.
|
||
|
||
Chunked sampling with an AV latent is refused rather than silently corrupted —
|
||
slicing a nested latent in time is not a thing the node does yet.
|
||
|
||
## Decoding
|
||
|
||
`decode` chooses how the sampled latent becomes frames:
|
||
|
||
- **full** — one shot. Right for a single pass at the size the sampler ran at.
|
||
- **tiled** — for a latent too large to decode whole, e.g. after a 2x upscaler.
|
||
A full decode there pins VRAM at 99% and shuttles tensors until it looks like
|
||
a hang.
|
||
- **none** — skip it and return the latent alone.
|
||
|
||
**For anything beyond a single pass, prefer `none` and decode outside.** Your own
|
||
VAE Decode (Tiled) is tunable, visible in the graph, and reused by the rest of
|
||
the workflow. The only reason decoding lives in this node at all is the
|
||
paste-back, which needs pixels — so with `none` the compositing is skipped too,
|
||
and **BFS Head Swap Paste Back** does it after your decode.
|
||
|
||
## Second pass at higher resolution
|
||
|
||
With cropping on, `latent` is the **crop's** latent, not the frame's — which is
|
||
what you want to refine, because that is where the face has pixels. To chain a
|
||
second pass:
|
||
|
||
```
|
||
Head Swap Sampler (paste_back: off)
|
||
├─ latent → LTXV upscaler / second sampler → VAE Decode ─┐
|
||
└─ crop_bboxes ───────────────────────────────────────────────┤
|
||
guide_video (original frames) ────────────────────────────────────┤
|
||
▼
|
||
BFS Head Swap Paste Back → frames
|
||
```
|
||
|
||
With `paste_back: off` the node stops compositing, so `images` and `latent` stay
|
||
in the crop's own space. Refine there, then bring the result home with **BFS Head
|
||
Swap Paste Back**, which takes the crops, the original frames and `crop_bboxes`.
|
||
It does the same edge-aware feather and optional mask confinement the sampler
|
||
would have done.
|
||
|
||
Refining before compositing is the right order: upscaling the crop puts the
|
||
pixels where the identity is, and the frame around it never needed a second pass.
|
||
|
||
## Reading the outputs
|
||
|
||
| output | what it answers |
|
||
|---|---|
|
||
| `images` | the result, with the crop pasted back |
|
||
| `mask_over_source` | is the mask where I think it is? Red over the original frames, green box for the crop |
|
||
| `cropped_guide` | exactly what the model was fed. If this is wrong, nothing downstream can be right |
|
||
| `crop_mask` | the mask after grow/blur, in the crop's pixel space |
|
||
| `latent_mask` | the mask as the sampler sees it — where a thin mask disappears |
|
||
| `latent` | the sampled latent — the CROP's, when cropping is on |
|
||
| `crop_bboxes` | the boxes, for Head Swap Paste Back after a second pass |
|
||
| `debug` | crop mode and box, mask ops, tiling, the size to build the latent at |
|
||
|
||
## If a second run behaves differently from the first
|
||
|
||
The `guider` is an object another node builds, and ComfyUI keeps it in the
|
||
execution cache between runs. This node has to write the reference specs into
|
||
its options — the guider captured its own dict by reference when it was built,
|
||
so patching the model alone never reaches the forward — and those specs hold
|
||
the encoded reference **latents**.
|
||
|
||
Since 1.37.0 the write is undone in a `finally` when sampling ends, including
|
||
when it ends by raising. Before that the specs stayed on the cached guider:
|
||
the latents could not be freed while that guider lived, and a later run that
|
||
did not overwrite the same keys — a graph with no references, an aborted run,
|
||
another sampler sharing the same guider — sampled with the *previous* clip's
|
||
specs, whose shapes no longer fit. If you saw runs that worked once and then
|
||
started failing or ignoring the reference until you restarted ComfyUI, that
|
||
was it.
|
||
|
||
Chunked clips also used to keep every sampled chunk on the sampling device
|
||
until the whole clip was done; they now move to the offload device as they come
|
||
out, so VRAM no longer grows with clip length while the model is still loaded.
|
||
|
||
## Suggested order when something looks wrong
|
||
|
||
1. `crop_mode: off`, no mask, `temporal_tile_size: 0`. This is the plain swap; if
|
||
it fails, the problem is the recipe or the LoRA, not this node.
|
||
2. Add the mask. Check `latent_mask` before judging the result.
|
||
3. Add the crop. Check `mask_over_source` for the box, and set the latent to the
|
||
size `debug` reports.
|
||
4. Only then raise the frame count and turn on chunking.
|
||
|
||
Sampling settings follow the LoRA's recipe: CFG 3–5 with a real negative, and no
|
||
LightX2V (CFG 1.0 means no amplification, and the reference stops being
|
||
amplified).
|