The sampler no longer reaches into a sibling pack at runtime: the crop path
planner and its helpers now live here as bfs_crop_planner.py and
bfs_subject_crop.py, copied from drozbay/MaskVidExperiments (d98cc89, v0.2.0).
Both packs are GPL-3.0, which is what makes the copy legal; attribution is in
NOTICE and in each file's header. Only the ComfyUI node classes were dropped
-- they stay in the original pack, where they belong.
tracked/zoomed therefore work out of the box now, and the per-frame paste-back
is ours rather than a call into their uncrop node.
Also adds the node's help page (docs/BFSHeadSwapMaskedSampler.md, the '?' on
the node): what each crop mode does, why the mask is native inpainting rather
than a LoRA feature, why latent_mask is the output to check first, how to size
the latent to the crop, and the order to switch things on when a run looks
wrong. scipy is now a declared dependency (the planner uses ndimage).
The sampled size comes from the connected latent, and the zoomed crop mode
rescales crops by design, so the result is not guaranteed to match the box
it came from -- pasting then fails on a shape mismatch. Resize first, and
report the size the latent should be to avoid the resample entirely.
Debugging this pipeline blind is the slow way. Five outputs now say what the
node actually did:
mask_over_source the mask painted on the ORIGINAL frames with the crop box
outlined -- is the mask where you think it is?
cropped_guide exactly what the model was fed
crop_mask the mask after grow/blur, in the crop's pixel space
latent_mask the mask as the SAMPLER sees it: max-reduced to the latent
grid and upsampled back. One frame per latent frame, which
is the real resolution of the edit and where a thin mask
disappears entirely
debug crop mode and box, mask ops, tiling
CFGGuider captures model_patcher.model_options BY REFERENCE when it is
constructed (comfy/samplers.py:937), so swapping guider.model_patcher does
not propagate -- the guider keeps sampling with the dict it captured, and
the reference specs LTXMultipleControls wrote on the clone never reach the
forward. That is why the guide had no effect at all despite the node
reporting two active slots. The keys are now mutated into the guider's own
transformer_options in place, the way the Edit Anything looping sampler
already does it.
LTX-2.5 latents are AV, and the video VAE takes the video stream alone --
what LTXVSeparateAVLatent does in the stock graphs.
LTX-2.5 latents are AV: video and audio nested together, which is why the
working graphs carry an LTXVSeparateAVLatent. Building a plain
[1,128,t,h,w] tensor hands the model a structure it does not expect, and
the guide stops reaching it -- the run returns the reference face and none
of the source clip.
The node now takes an optional latent (connect EmptyLTXVLatentVideo, sized
to the crop when cropping), slices it per chunk when it is video-only, and
refuses chunked sampling on a nested AV latent rather than corrupting it.
The old fabricated latent stays as a last resort, with a warning.
Two bugs from the first cut, both caught on the first real run.
_sample_chunk samples through guider.model_patcher, so passing the patched
clone as the model argument did nothing: the reference specs sat in
transformer_options that the forward never read, and guide + identity were
silently inert -- sampling ran to completion and ignored both. The pack
already had _set_guider_conds for exactly this; use it.
vae.decode takes the sample tensor, not a latent dict.
The LoRA keeps doing only its job -- guide on source_id 1, identity on
source_id 2. The mask never reaches it: it becomes latent["noise_mask"],
which the guider consumes as denoise_mask, so only the masked region is
denoised and the original face stays visible to the model.
Masks are reduced to the latent grid with MAX rather than ComfyUI's
trilinear resize, which blurs a mask across frames and lets the original
content bleed through the edit.
Crop exists for the case a LoRA cannot fix: a face too small to carry
identity. tracked/zoomed borrow drozbay/MaskVidExperiments' planner when
that pack is installed -- its boxes hold still through mask noise and
occlusion, and a jittering crop reads as camera motion to a video model.
Without it, one static box around the subject's whole travel.
Helpers are unit-tested; the sampling path still needs a GPU run.
ComfyUI's pixel coords are int64 -- latent corners multiplied by integer VAE scale factors --
while centring a reference on the target's height and taking a clip midpoint are half-pixel
operations. Adding a float shift into a Long tensor in place raises "result type Float can't
be cast to the desired output type Long", which is where the sidecar layout landed next.
_apply_tass_layout now promotes both coordinate tensors to float32 for the shift and rounds
back to the caller's dtype on the way out. Rounding matters: the caller casts the reference
coords to the target grid's dtype when concatenating, and a cast truncates where rounding is
what a half-pixel centre wants.
The earlier tests missed this because they built float tensors -- test_tass_layout now runs
every layout on int64 input and checks both the dtype and that the result stays within half
a pixel of the float placement.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HGYHCVjHe6acihEpCRyn1K
1.29.1 stopped the crash but only got the placement right on 2.5. The temporal span the
trainer writes -- bounds = (t_min, t_max) of the target clip -- is collapsed to one number
per token by the model, and the two LTX generations do it differently:
use_middle_indices_grid=True (2.5, and any checkpoint whose metadata declares it)
-> (start + end) / 2, i.e. the middle of the clip
use_middle_indices_grid=False (2.3 checkpoints that predate the flag)
-> the start bound alone, i.e. frame zero
So the same bounds mean "centre of the clip" on 2.5 and "pinned to frame 0" on 2.3 -- one
LoRA, two placements, and the 2.3 one is exactly the single-instant conditioning sidecar
exists to avoid. The layout now takes the model's own flag (read off the transformer in
_process_input) and writes the midpoint into both bounds when the model reads the start,
which lands the reference in the same place under either rule. Legacy [B, 3, N] coordinates
get the midpoint for the same reason.
Verified by collapsing the coordinates the way generate_freqs does and measuring the
reference against the target in the same frame: 2.5 is bit-identical to the trainer, and
2.3 matches to within half a patch (the coordinate convention's own offset, which applies
to the target's tokens equally).
tests/test_tass_layout.py covers both shapes, both collapse rules, the batch-1 reference vs
batch-2 target that crashed, the margin, and that patch spans are translated rather than
squeezed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HGYHCVjHe6acihEpCRyn1K
Two wrong assumptions in 1.29.0, both hit by the first real run:
shape -- LTX-2.5 hands out patch bounds, [B, 3, N, 2], not the one-corner-per-token
[B, 3, N] the node was written against. The other layouts survived it by
accident (their math is elementwise), but each was reducing over tokens only,
so a patch's start and end bound were shifted by their own separate minima --
which squeezes the span instead of translating it.
batch -- layouts run BEFORE the reference tokens are expanded to the CFG batch, so the
target arrives [2, ...] while the reference is still [1, ...]. overlap/st_drc
broadcast out of it; sidecar assigns in place and raised
"The expanded size of the tensor (1) must match the existing size (2)".
_tass_axis_stats now reduces over every dimension except the T/H/W one -- tokens, the
bounds axis when present, and the batch -- which fixes both at once and keeps the older
[B, 3, N] shape working. Verified against ltx_core.conditioning.reference_layout: all four
layouts now agree with the trainer bit-for-bit on the 2.5 shape, in the exact batch-1
reference vs batch-2 target configuration that crashed.
Also passes the spec's sidecar_margin_pixels through instead of always using the default.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HGYHCVjHe6acihEpCRyn1K
The trainer has had four reference layouts for a while — overlap, st_drc, strata
and sidecar — but the nodes only ever exposed three. Training a LoRA with the
reference in sidecar therefore had no matching inference path, which is how you
end up running a checkpoint in a layout it was never trained in.
sidecar places the reference beside the target on the W axis, vertically centred
on it, and stretches its T span to cover the target's whole clip, so an object
reference stays available for every frame instead of sitting at one instant. It
is the layout the replace_refedit recipe validated on 2.3 for exactly this role.
Mirrors ltx_core.conditioning.reference_layout, adapted to ComfyUI's [B, 3, N]
coordinate tensor instead of the trainer's [B, 3, N, 2] patch bounds, and takes
sidecar_margin_pixels like the trainer does.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two formatting rules the VLM breaks consistently, and that no amount of prompt
work fixed — five rounds of rewriting plus temperature 0 still produced the same
violations:
no reference -> the marker sentence must not exist, and a swap is a plain
Replace. The model emits Add + marker + Remove instead, which
is the reference form applied where it does not belong.
reference -> the marker may only follow a sentence starting with Add. When
it lands on a Replace, that pair appears in zero training
samples.
Both are decidable in code: the caller already knows whether it attached a
reference image, so the node takes that boolean rather than asking the model to
infer it. Add+Remove is recomposed into Replace, and Replace+marker into
Add+Remove, preserving the object description the VLM did get right.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The stock SaveLatent calls .contiguous() straight on samples["samples"], which
fails on H3 because the latent is a NestedTensor holding the video and audio
components. This unbinds them first and writes each as its own tensor, so an H3
latent can be taken out of the graph and decoded or inspected offline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The node read the device off the decoder parameters, but ComfyUI keeps the VAE on
CPU until something loads it — vae.decode() calls load_models_gpu first, and the
direct path skipped that. The result was a full ViT decode on CPU: minutes of
work, fans up, no output.
Now it calls comfy.model_management.load_models_gpu([vae.patcher]) and uses
vae.device / vae.vae_dtype like the native path, returning to the intermediate
device. Measured on a 768x1280 decode: 2.4s on cuda:0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The H3 latent arrives as a NestedTensor holding the video and audio components,
so `z.dim()` raised AttributeError as soon as the node was fed a real generated
latent. The native nodes unbind and keep the first component (nodes.py:335); this
does the same, and re-adds the batch dimension that unbind drops.
Verified both paths produce identical output (MSE 0.00e+00) on the same latent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Decoding a DiT-generated H3 latent through the stock VAEDecode produces a visible
grid. The reference path does not use vae.decode() at all: both the checkpoint
card and the multimodalart/MiniMax-H3-images Space call the decoder directly.
This node is that recipe, with the four details its author flags as silently
corrupting if changed:
1. vae.decoder(vae.post_quant_conv(z)) directly, never vae.decode(), whose
temporal chunker cannot form a chunk from the few latent slices an image
generation produces;
2. decoded[:, :, -1] -- the last output slice, not index 0;
3. latent denormalisation, z * latents_std + latents_mean;
4. ImageNet pixel denormalisation, without which the frame comes out dark and
over-contrasted like a tone-mapping bug.
Reconstruction check against a real image: 22.43 dB, block ratio 1.30 at the
32px patch boundary -- matching the raw decoder path measured separately, which
is what confirms the node reproduces the reference rather than approximating it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ports the sharpest-frame selection from scanpe_experiments/h3_image.py, which
takes it from IF-Edit (arXiv:2511.19435): decoded frames carry different amounts
of residual motion blur, so score by Laplacian variance and keep the crispest
instead of a fixed index.
Pairs with running MiniMax-H3 as an image generator: the VAE's minimum clip is 5
pixel frames (2 latents), decode yields frames of uneven sharpness, and the best
one is not always at the same index.
The node documents a limitation found while testing it, because it is easy to
misuse: Laplacian variance is a BLUR heuristic, not a quality metric -- artifacts
are high-frequency too. On a real H3 decode it picked the frame with the WORST
PSNR of its block (19.72 dB against 23.19 for its neighbour) because that frame
was the most artifacted. It is sound when the candidates are the same content
differing only in blur, which is the IF-Edit setting; it is meaningless for
choosing among frames from different moments.
Also relevant and not obvious: the "last frame of each decoded block is the valid
one" rule (decoded[:, :, -1] in the upstream example) held for the first block
here (34.49 dB against 26.80 for its neighbour) but inverted on the second.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The published single-frame decoder (iamkaikai/MiniMax-H3-Single-Frame-VAE-500K)
cannot be loaded by the stock VAELoader: detection in comfy/sd.py requires an
encoder key, and the file is decoder-only in diffusers naming, so it falls back
to Stable Diffusion geometry (4 latent channels, 8x upscale) and decodes blocks.
Two nodes:
- MiniMaxH3SingleFrameVAELoader: combines the official H3 VAE with the author's
decoder-only checkpoint, translating the naming in memory. Validated against
the author's own diffusers path (load_decoder.py) on the same latent: 72.92 dB,
i.e. fp16 noise.
- MiniMaxH3VAELoaderTiled: loads an already complete H3 VAE and exposes tile_size,
which the stock loader does not.
Two key conversions are not derivable from names, and getting either wrong
produces wrong output with no load error. Both were settled by correlating
against the official VAE:
- to_qkv is interleaved per head (heads, 3, dim_head), not stacked [q;k;v]
(see qkv.view(B, S, -1, 3*dim_head) in comfy/ldm/minimax/vae.py); corr 0.998.
- ff.w1 halves are swapped relative to diffusers, since ComfyUI reads gate first;
correlation: straight -0.013, swapped +2.00.
tile_size defaults to 512 rather than the constructor's 256. Measured on a
1056x640 image: 512 gives 26.00 dB with a seam ratio of 0.91 (indistinguishable),
against 22.17/1.49 at 256 and 21.35/2.42 at 1024. The author trained 475k of
500k images at <=512px, so 512-wide tiles keep each piece inside the regime that
has training mass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The learned slot tag was applied at fixed magnitude regardless of the LoRA
strength set on the loader. Tag and adapter are trained together at strength
1.0, so raising the LoRA alone leaves them out of proportion — the adapter
shouts while the signal that tells the references apart stays where it was.
This matters in practice: an undertrained adapter often needs 1.2-1.5 to take
effect at all, and at that point the tag is quietly under-weighted.
slot_strength exposes the same knob for the tag, defaulting to 1.0 so existing
graphs are unchanged.
Three nodes for checkpoints trained with several references on distinct slots:
- LTX Reference Images collects up to six references into one ordered list. A
plain IMAGE batch cannot carry references of different sizes, and subject
references routinely differ — a portrait crop next to a landscape product
shot — so the list keeps each at its own size and puts the slot order in one
visible place.
- LTX Multi Reference Slots attaches them as separate reference blocks, each
with its own source_id and, when the LoRA carries one, its trained slot
embedding. The embedding is read straight from the LoRA file: ComfyUI's LoRA
loader applies only LoRA weights and drops everything else, so without this
the references are untagged and the checkpoint is sampled without a signal it
was trained to use. Pointing it at a LoRA that has no slot embedding fails
with an explanation rather than silently ignoring the setting.
- LTX Reference Tags builds the "<Image 1> is the woman." prompt prefix.
Nothing in the model ties a tag to a slot — that correspondence exists only
because the training captions used it consistently, so sampling with a
different convention simply does not bind.
The identity node gains one hook: a spec may now carry a slot_vector, applied
to the raw patchified tokens before patchify_proj, which is exactly where the
trainer adds it. Specs without the key are untouched, so existing workflows
behave identically.
Builds a ShotPlan-format multishot prompt (global preamble + "Shot N: <modifier>
<size> <move> <action>" blocks) plus the ordered, gray-padded reference batch that
the multishot strata LoRA expects.
No new conditioning node is needed: LTX Identity Transfer already assigns
seg_value = (source_id + i) * phase_scale and strata_slot = i over the image batch,
so source_id=2 with layout="strata" over 4 refs reproduces the trained convention
(source_id 2/3/4/5 on slots 0/1/2/3). Batch order is shot order -- the LoRA was
trained with shuffle_reference_slots=false. Unused slots get a flat RGB-128 frame,
matching the dataset's blank.jpg placeholder.
The dropdown vocabularies are extracted from the 12,470 "Shot N:" headers of
Pensioner/shotplan train_meta_16fps.json (the span between "Shot N:" and the start
of the action sentence), covering 98% of size terms and 97% of move terms.
Modifiers (over-the-shoulder, low/high angle, POV) are a separate axis that
combines with a size rather than replacing it, as in the source captions.
Bump version to 1.21.1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
LTX Identity Overlap installed its four patches with types.MethodType(fn, ltxv) and captured the
originals as bound methods. Both put a strong reference to the model inside the model's own
__dict__, so every patched model sat in a reference cycle that only a full collection could break.
ComfyUI holds a weakref to detect exactly this: it reported "memory leak with model LTXAV", forced
a full gc, and repeated executions of the same workflow got progressively slower (#9).
The originals now come from the class (plain functions, called with self explicitly) and the
patches are installed as weakref-resolving callables with the same signature. Behaviour is
unchanged; the model is released by reference counting alone.
Verified against a stand-in LTXAV with cycle collection DISABLED, so only refcounting can free the
object: before this change the model was still alive after `del` (cycle present), after it the
weakref is dead, with the patched methods returning the same values in both cases.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two inputs doing the same thing is more confusing than a one-time
reconnect for anyone still on a pre-v1.14.0 workflow -- back to a single
required reference_image input. The actual RoPE crash fix (fused
rotation-matrix pe format) from the previous commit is unaffected and
stays in place.
Recent ComfyUI (comfy/ldm/lightricks/model.py freqs_cis_matrix) changed
_prepare_positional_embeddings to return a single fused 2x2 rotation
matrix per (batch, token, head, freq) plus a split_mode bool, instead of
separate cos/sin tensors. _rotate_ref_block still assumed the old
(cos, sin, [flag]) tuple, so `sin` silently became the split_mode bool
and crashed with "'bool' object has no attribute 'dtype'" on the very
first sampling step after updating ComfyUI (reported by an external
user; reproduced with the same traceback on 0.28.0).
_rotate_ref_block now detects which format it received and, for the new
fused format, composes the phase rotation via matmul instead of the old
cos*a-sin*b trick -- 2D rotation matrices commute, so R(phase) @ R(existing)
gives the same result regardless of multiplication order. Verified against
the real (not mocked) freqs_cis_matrix from the currently installed
ComfyUI: output stays a valid rotation matrix (determinant 1), the
untouched token range is unaffected, and the legacy cos/sin path still
works unchanged for anyone on an older ComfyUI.
v1.14.0 renamed reference_face -> reference_image and dropped the
identity_projector/id_strength/arcface_mode ArcFace path, but old saved
workflows still send those exact input/kwarg names -- ComfyUI passes
whatever a saved prompt's "inputs" dict contains straight through as
kwargs, so loading one crashed with "unexpected keyword argument
'reference_face'" instead of just using the renamed input.
reference_image is now optional with reference_face kept as a deprecated
alias (resolved to reference_image when the new name isn't connected),
and **_legacy_kwargs absorbs the removed ArcFace-projector params so old
workflows load and run instead of hard-crashing.
- guide/mask default to source_id=0 (no phase, matches v13's source_phase=false)
- identity default source_id=2 (matches v13's ref source_phase=true)
- mask_video is now expected as plain white-on-black binary (colored masks
leaked hue into generated clothing during training, confirmed via v9-v12)
- add auto_mask_guide (on by default): paints guide_video black wherever
mask_video is white before encoding, matching how v13 was trained -- this
closes the reconstruction shortcut that let the model skip identity_image
entirely and just copy guide's own person into the replacement region
Each slot (guide/mask/identity/identity_mask) now has its own
downscale_factor input: encodes that reference at target_size/N
instead of full resolution (cheaper VRAM for conditions trained at
reduced res, e.g. earlier scail2v2 recipes that downscaled guide/mask).
Ported the missing piece from ltx_core's VideoConditionByReferenceLatent
(positions[:,1:3,...] *= downscale_factor) into the shared process_input
patch in ltx_identity_overlap.py -- without it, an encoded-at-half-res
reference's raw pixel coordinates only spanned half the target's actual
extent instead of being re-stretched to overlap it, silently breaking
the position alignment the training recipe assumed.
Bump version to 1.19.0.
LTX Multiple Controls now supports a 4th slot (identity_mask_image) for
scail2v2-style color-pointer masks, defaulting to inherit the identity
slot's own source_id/phase_scale (matches training convention). Also
updated guide/mask/identity default source_ids to match the scail2v2
3-phase recipe (1/2/3 instead of 0/0/2).
New LTX Color Mask node recolors plain white/binary SAM2-style masks
into the trained convention: 'silhouette' style (recolor, keep shape)
for the guide mask slot, or 'flat_marker' style (recolor + collapse to
a small dot at centroid, no shape) for the identity mask slot -- a
shaped identity mask re-injects a competing pose signal, the same bug
flatten_ref_masks.py fixes on the training-data side.
Bump version to 1.18.0.
Chaining multiple LTXIdentityOverlapConditioning nodes silently drops all
but the last one's reference -- each apply() overwrites transformer_options
["_id_ref_specs"] instead of merging with what the upstream node set. This
node builds every active slot's ref_specs in a single call so guide, mask,
and identity references all combine correctly. Each slot (all optional) has
its own source_id/phase_scale/layout/ref_resize_mode, defaulted (not
hardcoded) to match the trained recipe.
STRATA_SLOT_WIDTH=0.5 let adjacent numbered slots overlap: a reference
image is encoded at fps=1.0 (process_videos.py convention), so a single
1-latent-frame reference spans a full 1.0-second range on the T axis
after get_pixel_coords, not a point. Confirmed numerically: slot 0 at
width=0.5 occupied [target+0.5, target+1.5] and slot 1 occupied
[target+1.0, target+2.0] -- they collided in [target+1.0, target+1.5].
Bumped to 1.5s, which safely clears the 1.0s content span with margin.
Every checkpoint trained before this fix (phantom_stacked_r128/_p2)
learned under the collided geometry, so continuing training from one
now means adapting to the corrected spacing, not a free continuation.
BREAKING: reference_face renamed to reference_image (user confirmed:
not face-specific, references are arbitrary subjects -- toys, animals,
objects; accepted the input-name break since it's cosmetic and doesn't
silently change generation behavior like the earlier v1.13.0 mistake
did). Existing workflows will show this input disconnected on load and
need reconnecting once.
Also drops the identity_projector/id_strength/arcface_mode ArcFace path
entirely per explicit confirmation it isn't used -- kept the shared
_arcface_embed/_load_projector/_append_ctx_tokens/_find_ltxv helpers
intact since ltx_identity_multiangle.py still imports them.
v1.13.0 broke existing workflows: renamed the required reference_face
input to reference_image, and deleted the identity_projector/id_strength
/arcface_mode ArcFace inputs entirely. This repo has real users on the
current node -- reverting both: reference_face keeps its name, ArcFace
is back exactly as it was (still only used for the primary/1st reference).
Multiple stacked references (layout='strata') are now driven by feeding
a BATCH of images into the existing reference_face input (e.g. via an
Image Batch node) instead of a second fixed reference_face_2/source_id_2
pair of inputs -- a single image (the old, default case) behaves exactly
as before; N images each become their own reference block, source_id
auto-incrementing per index (1st keeps source_id as-is, 2nd gets +1, ...).
ref_preview/crop_overlay outputs now return one image per batch entry
instead of only the first, so all stacked refs are visible.
Also fixes a real bug in the reverted 1.13.0 approach the user hit while
testing (second reference silently not applying) -- the model patch now
tracks a list of (offset, length, source_phase) blocks end to end
(process_input concat + prepare_pe rotation), the same proven pattern
already used by ltx_identity_multiangle.py, instead of a single ref
latent/seg_value pair that only the last-installed spec could occupy.
Reference-image conditioning is no longer face-specific: renamed
reference_face(_2) -> reference_image(_2) and removed the ArcFace
identity_projector/id_strength/arcface_mode inputs and code path (kept
the shared _arcface_embed/_append_ctx_tokens/_load_projector helpers --
still used by ltx_identity_multiangle.py).
Added 'strata' to the layout dropdown and an optional reference_image_2
input (source_id_2, default 3) so this node can drive checkpoints
trained on ltx_trainer's TASS strata layout: multiple SEPARATE
reference images sharing the overlap H/W range but placed in their own
numbered temporal slot (target_max_t + slot*STRATA_SLOT_WIDTH), instead
of the old single-reference-only overlap/st_drc path.
Generalizes the model patch from one ref block to a list of (offset,
length, source_phase) blocks -- process_input now shifts/concats N refs
in order, prepare_pe rotates each block's own RoPE phase independently
(pattern lifted from ltx_identity_multiangle.py's proven multi-block
timestep/grid_mask handling), and a new _forward capture point stashes
frame_rate so process_input can convert STRATA_SLOT_WIDTH (seconds, to
match the trainer) into this step's raw pixel/frame units.
set_model_sampler_cfg_function() detects the old (cond, uncond, cond_scale)
calling convention by counting ALL signature parameters, not just
positional ones -- the previous 3-param version (args + two defaulted
closure vars) tripped that check and got called with three raw tensors
instead of the args dict, crashing with "too many indices for tensor of
dimension 3". Fixed by taking a single `args` param and closing over
noref_to/ref_scale normally.
Optional reference_guidance_scale input (default 1.0 = off). When >1,
adds a third forward pass per denoising step with the reference tokens
dropped, isolating the reference's own contribution the same way CFG
isolates the text prompt: denoised += (scale-1)*(with_ref - without_ref).
Implemented via model.set_model_sampler_cfg_function(), reusing the
positive conditioning already passed to KSampler for the extra pass.
Mirrors the same reference-CFG mechanism added to ltx-trainer's
validation_runner.py so training-time validation and this node's
inference now share the same guidance formula.
New optional input `layout` (default "overlap", the only behavior every
checkpoint so far was trained with -- old workflows keep working
unchanged). "st_drc" shifts the whole reference coordinate block past
the target's own extent on every axis (non-overlapping RoPE region),
mirroring ltx_trainer's training_strategies/tass.py apply_tass_layout
st_drc branch -- kept in sync manually since this node can't import the
trainer package, adapted to ComfyUI's own [B,3,N] single-corner
coordinate shape instead of the trainer's [B,3,N,2] patch-bounds shape
(the min/max-per-axis shift math is equivalent either way, verified
against a synthetic test matching apply_tass_layout's own logic).
Only usable with a checkpoint specifically trained with layout='st_drc'
-- using it against an 'overlap'-trained checkpoint will not work,
the model never learned that coordinate convention.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New IMAGE outputs (appended at the end of RETURN_TYPES, existing wiring
unaffected):
- ref_preview: exactly what gets encoded as the reference, after
whatever resize/crop/letterbox ref_resize_mode applied -- lets you
sanity-check the actual input to the model.
- crop_overlay: the ORIGINAL reference with a green rectangle around the
region that survives match_target's crop (everything outside it gets
discarded). For match_target_letterbox/native_resolution, which never
crop, the box just outlines the full frame.
New optional input crop_anchor (center/top/bottom/left/right, default
"center"): controls which part of the reference match_target's crop
keeps, instead of always the exact center -- e.g. set "top" if your
sheet's face closeup sits in the top row so it survives a crop against
a differently-shaped output. Added under "optional", not "required", so
existing saved workflows keep working with zero changes (falls back to
the previous always-center-crop behavior). Verified byte-for-byte
equivalent to the old center-crop path on a smooth test image when left
at the default; the non-default anchors reuse the same crop math with a
different offset.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
match_target resizes the reference to the output video's pixel size via
a center-crop (comfy.utils.common_upscale crop="center") then resize --
for aspect-mismatched refs (e.g. a landscape character sheet used with a
portrait output) this silently discards whatever isn't in the middle of
the reference, which can crop out the face closeup panel entirely.
Add match_target_letterbox: same target pixel size, but fits the WHOLE
reference inside it preserving its own aspect ratio (no crop), padding
the leftover space with white. Trades some wasted canvas space for
never losing reference content -- recommended over match_target when
ref and output aspect ratios differ, and over native_resolution when
native_resolution's landscape-composition bias causes an off-center/
cropped-looking result on the OUTPUT video itself.
Reported by a LoRA user: portrait output + landscape character sheet +
native_resolution produced an off-center-left result (model composing
as landscape then getting cropped to the portrait canvas); switching to
match_target "fixed" centering but lost face detail from the crop.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
alisson-anjos