LTX packs all spatial/temporal dims into a single token [1, 1, C*T*H*W]
during sampling. Per-step latent steering requires hooking into the model
internals, not the k-diffusion callback. Removed: compress_latent(),
_load_latent_value_function(), latent VF training, motion floor callback,
latent_value_guidance toggle in Studio.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously latent_vf_strength was 0 when no latent VF existed, making
motion floor a no-op. Now embed_guidance_strength is always passed so
motion floor applies at the correct strength regardless of latent VF state.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LTX samples in [B, seq, C] token format during sampling, not [B, C, T, H, W].
Use sequence variance as temporal variance proxy — works for both 3D and 5D.
Replace fixed 0.5 sigma threshold with 1-sigma scale so motion floor and
latent VF activate across LTX's distilled high-sigma schedule naturally.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
OUTPUT_NODE = True forces ComfyUI to run the node even with no downstream
connection on the LATENT output. Motion floor now activates as soon as
embed_guidance is on and a refinement key is set, independent of liked_dir.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When temporal variance of the latent drops below 5% of overall variance,
applies a normalised deviation gradient that pushes frames apart from each
other. Fires only at sigma < 0.5 (same window as other latent guidance),
uses embed_guidance_strength as scale. Active whenever embed_guidance is on
and liked_dir is available — no user input needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Trains a second OnlineValueFunction on rated video latents (compressed via
channel mean-pool). At each denoising step, computes ∂reward/∂latent through
the lightweight MLP and applies it as an in-place drift to the noisy latent —
gradients never touch the 22B transformer. Controlled by embed_guidance_strength
alongside the existing conditioning guidance. Studio toggle: "Latent guidance".
Cleared on session reset alongside the conditioning VF.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Before each denoising run, when the value function is ready, gradient-ascend
the text conditioning until predicted reward plateaus (self-terminating: step
size = 0.5% of conditioning norm, stops when delta < 1e-3, caps at 50 steps).
No user-facing params — fully automated. Per-step wrapper still runs on top.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Model weights loaded via torch.load inside ComfyUI's inference_mode become
inference tensors and can't participate in backward. Wrapping the load in
inference_mode(False) ensures parameters are normal autograd tensors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
MLP lives on CPU, conditioning on GPU — device mismatch silently fell back
to fixed direction. Now moves conditioning to MLP device for the forward/
backward, then returns gradient to original device.
Also surface value function fallback errors instead of swallowing them.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
value_function.py: OnlineValueFunction MLP that trains incrementally on
(conditioning, reward) pairs. Uses replay buffer of 100 samples, 20
gradient steps per update. save/load to .pt file via refinement_state_path.
Studio popup: new "Value guidance" toggle. When on, trains the value function
after every rated generation (good and bad). Falls back to no-op if conditioning
deserialization fails.
chain sampler: _load_value_function reads the trained MLP if ready (5+
samples). _build_embed_guidance_wrapper uses value function gradient
(∂reward/∂conditioning) when available, fixed liked direction as fallback.
Log reports which mode is active and sample count.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extends the Refiner's one-shot conditioning steering to apply at every
denoising step. Reads the liked conditioning direction from the Refiner's
state file via refinement_key_input. At each step, nudges c_crossattn toward
the learned liked direction with sigma-scaling (no effect at sigma=1, full
effect below sigma=0.5 where detail refinement happens).
Requires 3+ liked generations to have a meaningful direction. No-ops silently
if direction is unavailable.
New inputs: embed_guidance (BOOLEAN), embed_guidance_strength (FLOAT 0.005-0.1).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Below 0.25: audio degrades to noise (guide frame in latent confuses
cross-modal attention without enough video coherence), character appearance
drifts while background over-anchors to previous scene.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Empirical testing: 0.25 is the sweet spot — locks character position and
action pixel-perfect while allowing background to drift freely toward the
new scene's prompt. Above 0.35 causes character-space conflicts when scene
composition shifts significantly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_prepend_soft_continuation creates the noise mask with ones_like(video_tensor)
giving it 128 channels instead of 1. Using torch.full_like(masks[0][:,:,:1])
copies the exact shape signature from the existing mask so the cat always works
regardless of channel count.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
guide_mask was [B,1,1,1,1] but noise_mask is [B,1,F,H,W] — cat on dim=2
requires matching H,W. Now uses masks[0].shape[3:] for spatial dims.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previous approach (post-block hidden state injection) was fundamentally
broken for LTXAV: modifying video tokens after a block contaminates audio
through joint audio-video attention, causing audio noise and latent bleed.
New approach: append the middle frame of the previous scene as a guide
token to the END of the current chunk latent, with keyframe_idxs pointing
to F//2 of the current scene in pixel space, plus guide_attention_entries
at the specified strength. The model's existing _attention_with_guide_mask
mechanism attenuates attention weights inside the attention computation —
audio tokens never see modified video tokens.
After sampling, _crop_video_tail removes the guide frame.
New inputs: mid_scene_guide (BOOLEAN), mid_scene_guide_strength (FLOAT 0.1-0.8).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Old: average all tokens in middle third → one scalar direction → spread to
surrounding zones. Loses all spatial structure, can't preserve positioning.
New: take the exact spatial token map of the single middle frame (F//2),
inject it position-for-position into every other frame. Token at (h,w) in
the anchor frame maps to (h,w) in all other frames. Character layout and
positioning stays spatially consistent across the scene. Anchor frame
itself is left unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Re-encoding the next scene's conditioning with the previous scene's
last frame is useless for fade-to-black workflows — the last frame
is black, so it's conditioning on nothing. Removing the whole path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Middle-frame tokens are the anchor source — blending them with their
own mean was pointless self-contamination and the main cause of
collapse at higher strengths. Now only the surrounding frames
(before mid_start and after mid_end) receive the injection; the
anchor zone is left untouched each step.
Max strength capped at 0.5 in the UI; tooltip warns against going high.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sampler vision re-encoding: prefix scene text with
"Continuing from this scene: {text}" so the model treats the
previous frame as temporal context, not an unrelated subject.
Studio per-scene reference conditioning: re-enable reference_image
with prefix "The character shown performs the following: {encode_text}"
so the model knows the image is a character reference, not something
to describe. Only applies when source_image is connected.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previous: capture middle-frame tokens once at first sigma < 0.95, reuse
for all remaining steps — injected noisy step-6 states into cleaner
step-7/8 passes, causing latent noise flooding.
Now: stateless per-step operation. At every step below sigma=0.95,
compute the mean of the current step's middle-frame tokens and blend
it into all token positions at the given strength. Anchor is always
from the current denoising state so no high-sigma artefacts persist
beyond the step they were produced in.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_v2_transition_scene_conditionings was encoding per-scene texts without
the reference image — only the main single-scene conditioning path passed
it through. Now passes reference_image=source_image to all per-scene
encodings so every scene conditioning includes the character reference
via Gemma3 vision.
Removes the redundant reference_conditioning option from the chain sampler
(which required a wasteful VAE decode mid-sampling to get the pixel image
that Studio already has).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
reference_conditioning (safe):
Re-encodes every scene's conditioning with the original reference image
as Gemma3 visual context. Reference frame is VAE-decoded once before the
loop. Gives the model character identity in the text conditioning for
every scene, not just the i2v latent anchor. Requires CLIP connected.
self_consistency (experimental):
Registers forward hooks on IDENTITY_BLOCKS [14,20,21,30,33] that capture
hidden states from the middle third of the video (by token index) at the
first sigma < 0.95 crossing, then softly inject them back into those same
tokens for remaining steps in gate (0.35, 0.85). Intended to reduce
within-scene character drift across long clips.
New input: self_consistency_strength (default 0.05).
Hooks are registered per-scene and cleaned up after each sample call,
restoring any pre-existing model_function_wrapper.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LTX-2 is not designed for creative i2i — distilled schedule gives no
reliable middle ground between 1-step (unchanged) and full regeneration
(no character identity). Removing the feature entirely rather than shipping
something that doesn't work.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Single-frame i2i against LTX's distilled schedule is unworkable: low
strength = 1 step (barely changes), high strength = full regeneration with
no character identity.
New approach: pin frame 0 (character reference) at mask=strength, generate
frame 1 freely with the new scene prompt, return frame 1 as the anchor.
The model has proper i2v context and produces character-consistent results.
i2i_strength now means: 0.0 = reference tightly pinned, 1.0 = ignored.
Default changed from 0.75 to 0.3. _i2i_sigmas removed (full schedule now).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two fixes:
- Truncate sigmas to start at strength level (_i2i_sigmas). Full schedule
from σ=1.0 lets the distilled model reconstruct the input unchanged even
at moderate strength values. Now denoises only from σ≈strength downward.
- Restore _build_continuation_chunk: overlap frames are pinned normally at
0..video_overlap-1, i2i anchor placed at offset=video_overlap. Scene starts
as smooth continuation then cuts to the new anchor. Latent blend remains
no-op because overlap frames match previous output.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Last frame of previous scene is black when fade-to-black is used — all
generated anchors came out black. For hard cuts the correct source is always
the original character reference (latent_template frame 0).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When i2i is active the two sides of the seam are different content —
blending them in latent space produces morphing artifacts, not a transition.
Force overlap=0 at the blend step for i2i scenes. Pixel-space transitions
(fade-to-black etc.) still apply independently via boundary_entries.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
i2i_scene_cut now builds a fresh chunk from the template regardless of
frame_overlap. The overlap=0 gate was silently blocking the feature for
anyone not already on hard-cut settings, producing black reference frames
and no anchor generation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When frame_overlap=0 and i2i_scene_cut=True, generates a new starting frame
for each continuation scene by denoising the previous scene's last frame with
the new scene prompt (i2i strength-controlled). The result is pinned as frame 0
of the next chunk (mask=0, i2v style), replacing soft continuation.
reference_frames output collects all generated anchor images as an IMAGE batch.
New inputs: i2i_scene_cut (BOOLEAN), i2i_strength (FLOAT 0.0–1.0).
New output: reference_frames (IMAGE, slot 2 — shifts status/scene_count/etc up by 1).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remnants of the old append-guide-to-end approach (abandoned in e8c3b32).
The current prepend approach needs neither: the guide lands at temporal
position 0 naturally via the causal patchifier, no keyframe_idxs required.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Option 1 (samplers.py): Add optional clip input to FunPackLTXAVSceneChainSampler.
After each scene is sampled, decode its last frame and re-encode the next scene's
conditioning with that frame as vision context. This gives identical scene texts
different conditioning based on temporal position, so the model knows what state
it came from.
Option 2 (conditioning.py): When building scene conditionings, detect duplicate
scene texts and prepend "Returning to an earlier scene: " before encoding the
second+ occurrence. Stores original text in funpack_scene_text for logging and
the actual encode text in funpack_encode_text so the sampler uses the same prefix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When frame_overlap=0, use the last 4 frames of the previous scene as soft
continuation guides (mask=0.4) instead of a single fully-pinned frame (mask=0.0).
This lets the model use previous-scene context for identity continuity while
retaining freedom to change orientation from the text prompt. The frames land
at positions 1-4 after the i2v guide at position 0 (when carry_i2v_guides is
enabled). crop count is returned dynamically so the existing crop/blend logic
requires no changes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Chain sampler accepts optional refinement_key_input (wire from the same
key as the V2 Refiner). On each run it calls update_refinement_sampler_context
to write carry_i2v_guides, frame_overlap, and scene count into the shared
state file. The V2 Refiner reads last_sampler_context from state when it
loads, making it available for the gen_context snapshot with no latent
manipulation and no extra Refiner inputs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Chain sampler now embeds funpack_sampler_context into the output latent
(carry_i2v_guides, frame_overlap, transitions_enabled). V2 Refiner reads
these from its existing latent input — no new Refiner inputs needed.
Removed the three forceInput connectors added in the previous commit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add confirmed-bad-combo warnings to both parameter tooltips, the sampler
doc, and the changelog. Warn users not to combine frame_overlap=0 with
carry_i2v_guides=True in normal use.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
mask=0.5 caused the continuation frame to be partially denoised, distorting
it and triggering the same yellowing artifacts as the temporal position conflict.
mask=0.0 keeps the frame intact so the model uses it as a clean reference.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When no overlap frames are pinned, the next scene has zero context from
the previous one. Prepend the last latent frame from the previous output
with mask=0.5 so the model can use it as a soft continuation reference
without hard-pinning it. The frame is cropped from the head after sampling
and never appears in the output. Works alongside carry_i2v_guides.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Appending the guide (even offset to chunk-end) caused the model to generate
content converging toward the reference at the end of each scene. Scene 3's
overlap started from that converged tail and compounded into a full visual reset.
Prepending the guide places it at temporal position 0 naturally (causal patchifier
assigns it [0,1]). Overlap frames follow at [1,9], [9,17]... — no temporal slot
conflict, no forward-pulling toward the reference. The guide acts as a clean
i2v anchor at the start. Crop now removes from the head after sampling.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Guide tokens were anchored at pixel position 0 (causal_fix default), which is the same
temporal slot occupied by the overlap frames at the start of continuation chunks. The model
saw two different frame contents claiming position 0 and produced color artifacts. Offset
the guide to chunk_end + 1 latent step so it acts as a forward style anchor instead.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Removed per-scene slicing (_decode_video/_decode_tensor) and the aggressive
del-before-decode GC block. Now: whole sampling loop runs to completion →
full latent decoded in one pass → transition pixel effects applied after.
LATENT output is unchanged. Per-scene decode was causing seam cross-contamination
anyway and contradicted the post-process-only contract for transitions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
