Extends the Refiner's one-shot conditioning steering to apply at every
denoising step. Reads the liked conditioning direction from the Refiner's
state file via refinement_key_input. At each step, nudges c_crossattn toward
the learned liked direction with sigma-scaling (no effect at sigma=1, full
effect below sigma=0.5 where detail refinement happens).
Requires 3+ liked generations to have a meaningful direction. No-ops silently
if direction is unavailable.
New inputs: embed_guidance (BOOLEAN), embed_guidance_strength (FLOAT 0.005-0.1).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Below 0.25: audio degrades to noise (guide frame in latent confuses
cross-modal attention without enough video coherence), character appearance
drifts while background over-anchors to previous scene.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Empirical testing: 0.25 is the sweet spot — locks character position and
action pixel-perfect while allowing background to drift freely toward the
new scene's prompt. Above 0.35 causes character-space conflicts when scene
composition shifts significantly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_prepend_soft_continuation creates the noise mask with ones_like(video_tensor)
giving it 128 channels instead of 1. Using torch.full_like(masks[0][:,:,:1])
copies the exact shape signature from the existing mask so the cat always works
regardless of channel count.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
guide_mask was [B,1,1,1,1] but noise_mask is [B,1,F,H,W] — cat on dim=2
requires matching H,W. Now uses masks[0].shape[3:] for spatial dims.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previous approach (post-block hidden state injection) was fundamentally
broken for LTXAV: modifying video tokens after a block contaminates audio
through joint audio-video attention, causing audio noise and latent bleed.
New approach: append the middle frame of the previous scene as a guide
token to the END of the current chunk latent, with keyframe_idxs pointing
to F//2 of the current scene in pixel space, plus guide_attention_entries
at the specified strength. The model's existing _attention_with_guide_mask
mechanism attenuates attention weights inside the attention computation —
audio tokens never see modified video tokens.
After sampling, _crop_video_tail removes the guide frame.
New inputs: mid_scene_guide (BOOLEAN), mid_scene_guide_strength (FLOAT 0.1-0.8).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Reads settings.refiner.vision_conditioning (default true) and gates
source_image passed as reference_image to Gemma3 encoding. Toggle lives
in the popup UI next to Reference injection, not on the node.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Old: average all tokens in middle third → one scalar direction → spread to
surrounding zones. Loses all spatial structure, can't preserve positioning.
New: take the exact spatial token map of the single middle frame (F//2),
inject it position-for-position into every other frame. Token at (h,w) in
the anchor frame maps to (h,w) in all other frames. Character layout and
positioning stays spatially consistent across the scene. Anchor frame
itself is left unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Re-encoding the next scene's conditioning with the previous scene's
last frame is useless for fade-to-black workflows — the last frame
is black, so it's conditioning on nothing. Removing the whole path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Middle-frame tokens are the anchor source — blending them with their
own mean was pointless self-contamination and the main cause of
collapse at higher strengths. Now only the surrounding frames
(before mid_start and after mid_end) receive the injection; the
anchor zone is left untouched each step.
Max strength capped at 0.5 in the UI; tooltip warns against going high.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sampler vision re-encoding: prefix scene text with
"Continuing from this scene: {text}" so the model treats the
previous frame as temporal context, not an unrelated subject.
Studio per-scene reference conditioning: re-enable reference_image
with prefix "The character shown performs the following: {encode_text}"
so the model knows the image is a character reference, not something
to describe. Only applies when source_image is connected.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Vision encoding rewrites the full conditioning embedding — with a 512-token
limit, a poor generic description from the template displaces the actual
scene prompt content. Reverting until the template can be properly designed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previous: capture middle-frame tokens once at first sigma < 0.95, reuse
for all remaining steps — injected noisy step-6 states into cleaner
step-7/8 passes, causing latent noise flooding.
Now: stateless per-step operation. At every step below sigma=0.95,
compute the mean of the current step's middle-frame tokens and blend
it into all token positions at the given strength. Anchor is always
from the current denoising state so no high-sigma artefacts persist
beyond the step they were produced in.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_v2_transition_scene_conditionings was encoding per-scene texts without
the reference image — only the main single-scene conditioning path passed
it through. Now passes reference_image=source_image to all per-scene
encodings so every scene conditioning includes the character reference
via Gemma3 vision.
Removes the redundant reference_conditioning option from the chain sampler
(which required a wasteful VAE decode mid-sampling to get the pixel image
that Studio already has).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
reference_conditioning (safe):
Re-encodes every scene's conditioning with the original reference image
as Gemma3 visual context. Reference frame is VAE-decoded once before the
loop. Gives the model character identity in the text conditioning for
every scene, not just the i2v latent anchor. Requires CLIP connected.
self_consistency (experimental):
Registers forward hooks on IDENTITY_BLOCKS [14,20,21,30,33] that capture
hidden states from the middle third of the video (by token index) at the
first sigma < 0.95 crossing, then softly inject them back into those same
tokens for remaining steps in gate (0.35, 0.85). Intended to reduce
within-scene character drift across long clips.
New input: self_consistency_strength (default 0.05).
Hooks are registered per-scene and cleaned up after each sample call,
restoring any pre-existing model_function_wrapper.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LTX-2 is not designed for creative i2i — distilled schedule gives no
reliable middle ground between 1-step (unchanged) and full regeneration
(no character identity). Removing the feature entirely rather than shipping
something that doesn't work.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Single-frame i2i against LTX's distilled schedule is unworkable: low
strength = 1 step (barely changes), high strength = full regeneration with
no character identity.
New approach: pin frame 0 (character reference) at mask=strength, generate
frame 1 freely with the new scene prompt, return frame 1 as the anchor.
The model has proper i2v context and produces character-consistent results.
i2i_strength now means: 0.0 = reference tightly pinned, 1.0 = ignored.
Default changed from 0.75 to 0.3. _i2i_sigmas removed (full schedule now).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two fixes:
- Truncate sigmas to start at strength level (_i2i_sigmas). Full schedule
from σ=1.0 lets the distilled model reconstruct the input unchanged even
at moderate strength values. Now denoises only from σ≈strength downward.
- Restore _build_continuation_chunk: overlap frames are pinned normally at
0..video_overlap-1, i2i anchor placed at offset=video_overlap. Scene starts
as smooth continuation then cuts to the new anchor. Latent blend remains
no-op because overlap frames match previous output.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Last frame of previous scene is black when fade-to-black is used — all
generated anchors came out black. For hard cuts the correct source is always
the original character reference (latent_template frame 0).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When i2i is active the two sides of the seam are different content —
blending them in latent space produces morphing artifacts, not a transition.
Force overlap=0 at the blend step for i2i scenes. Pixel-space transitions
(fade-to-black etc.) still apply independently via boundary_entries.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
i2i_scene_cut now builds a fresh chunk from the template regardless of
frame_overlap. The overlap=0 gate was silently blocking the feature for
anyone not already on hard-cut settings, producing black reference frames
and no anchor generation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When frame_overlap=0 and i2i_scene_cut=True, generates a new starting frame
for each continuation scene by denoising the previous scene's last frame with
the new scene prompt (i2i strength-controlled). The result is pinned as frame 0
of the next chunk (mask=0, i2v style), replacing soft continuation.
reference_frames output collects all generated anchor images as an IMAGE batch.
New inputs: i2i_scene_cut (BOOLEAN), i2i_strength (FLOAT 0.0–1.0).
New output: reference_frames (IMAGE, slot 2 — shifts status/scene_count/etc up by 1).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remnants of the old append-guide-to-end approach (abandoned in e8c3b32).
The current prepend approach needs neither: the guide lands at temporal
position 0 naturally via the causal patchifier, no keyframe_idxs required.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If the character description ends with . ! ? or , use a space separator
instead of injecting a redundant comma. Only bare text gets the ", " join.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Option 1 (samplers.py): Add optional clip input to FunPackLTXAVSceneChainSampler.
After each scene is sampled, decode its last frame and re-encode the next scene's
conditioning with that frame as vision context. This gives identical scene texts
different conditioning based on temporal position, so the model knows what state
it came from.
Option 2 (conditioning.py): When building scene conditionings, detect duplicate
scene texts and prepend "Returning to an earlier scene: " before encoding the
second+ occurrence. Stores original text in funpack_scene_text for logging and
the actual encode text in funpack_encode_text so the sampler uses the same prefix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When frame_overlap=0, use the last 4 frames of the previous scene as soft
continuation guides (mask=0.4) instead of a single fully-pinned frame (mask=0.0).
This lets the model use previous-scene context for identity continuity while
retaining freedom to change orientation from the text prompt. The frames land
at positions 1-4 after the i2v guide at position 0 (when carry_i2v_guides is
enabled). crop count is returned dynamically so the existing crop/blend logic
requires no changes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LTXAVTEModel.load_sd passes the full checkpoint sd to Gemma3_12B.load_state_dict,
which has vision_model and multi_modal_projector as real attributes with matching
key names. Vision weights are already present after DualCLIPLoader runs.
Drop _vision_loaded flag check; detect vision capability by attribute existence.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Checkpoints converted for ComfyUI use vision_model.* and
multi_modal_projector.* as top-level keys directly, not the HuggingFace
model.vision_tower.* nesting. Accept both formats.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When no vision keys are found, log the top-level key prefixes actually
present in the checkpoint so the user can see whether the vision tower
was stripped. Also rename all [FunPackGemmaLoader] log prefixes to
[FunPackGemmaVision].
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Companion to DualCLIPLoader: takes the already-built CLIP object (which has
the correct LTXAVTEModel structure) and loads the SigLIP vision tower +
multi-modal projector from the Gemma3-12B-IT checkpoint, setting
_vision_loaded=True. Workflow: DualCLIPLoader → FunPackGemmaVision → Studio.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
FunPackGemmaLoader: drop-in replacement for Load CLIP Model that additionally
loads the SigLIP vision tower and multi-modal projector from Google's
Gemma3-12B-IT checkpoint into the already-initialized Gemma3_12B module,
enabling reference-image vision prompting without a separate CLIP Vision model.
Vision prompting: _v2_encode_prompt accepts reference_image; when the clip
has _vision_loaded=True, tokenizes with image=reference_image and
skip_template=False so Gemma3 sees the reference frame before generating
conditioning. Cache key includes image fingerprint to avoid stale hits.
source_image from Studio/refine_v2 flows through to _v2_conditioning_source.
K/V injection revamp (Technique 5): identity blocks in i2v mode now use
in-context conditioning instead of lerp injection. Pre-hook prepends reference
hidden-state tokens to vx and extends v_pe with neutral-rotation entries
(cos=1, sin=0) so video tokens self-attend to reference tokens in attn1.
Post-hook crops the reference prefix from the block output. Strength raised
from 0.02 → 1.0 (sigma-gated: active at 0.35–0.80). _extend_v_pe helper
handles 2-D/3-D/4-D PE tensors. Anchor blocks keep the existing lerp path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
update_refinement_sampler_context was loading the main V2 state file
(which contains base64-encoded conditioning tensors) during sampling.
This re-hydrated large conditioning objects in Python's heap mid-sample,
triggering Python's cyclic GC which detected internal reference cycles
in LTXAVTEModel_ and fired the "potential memory leak" warning.
Fix: write sampler context to a tiny separate file (sampler_ctx prefix,
~60 bytes). refine_v2 reads from that file instead of state["last_sampler_context"].
The main state file is never touched during sampling.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Chain sampler accepts optional refinement_key_input (wire from the same
key as the V2 Refiner). On each run it calls update_refinement_sampler_context
to write carry_i2v_guides, frame_overlap, and scene count into the shared
state file. The V2 Refiner reads last_sampler_context from state when it
loads, making it available for the gen_context snapshot with no latent
manipulation and no extra Refiner inputs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Chain sampler now embeds funpack_sampler_context into the output latent
(carry_i2v_guides, frame_overlap, transitions_enabled). V2 Refiner reads
these from its existing latent input — no new Refiner inputs needed.
Removed the three forceInput connectors added in the previous commit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Stores a gen_context snapshot in last_run each iteration: active LoRAs
(name/type/weight), carry_i2v_guides, frame_overlap, transitions_enabled,
and source image fingerprint. On the next run, diffs previous vs. current
context and emits targeted insights in training_info — e.g. "carry_i2v_guides
is now OFF — appearance drift is likely from missing i2v guides, not the
prompt itself" when rating is Wrong appearance. Three new optional forceInput
connectors on the V2 Refiner node: carry_i2v_guides, frame_overlap,
transitions_enabled — wire from your sampler.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirrors _v2_transition_scene_texts: the text before the first
transition is a character anchor prepended to every scene, not a scene
itself. Show it separately above the rail. scenes[] now matches what
the Scene Chain sampler actually generates.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extract _v2_split_prompt_by_transitions to module-level
split_prompt_by_transitions. parse_timeline_segments now calls it
directly with placement=silent — same code path as generation.
Eliminates all reimplemented splitting logic.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Empty string is falsy so the stacking guard was skipping the phantom
collapse when the first segment was empty. Remove the truthy check so
both empty and transition-only segments are collapsed correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Shortcut expansions that start with a built-in transition phrase create
a phantom segment with no real content, causing N+1 scenes. Apply the
same stacking collapse as _v2_split_prompt_by_transitions: merge the
phantom into the following segment and drop the spurious transition.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The pipeline is shortcuts → transitions → split. The route now runs
apply_prompt_shortcuts first, then parse_timeline_segments on the
expanded text, matching actual generation behavior.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
widgetByName returns nothing when the input is connected. Traverse the
graph link to the source node and read its first widget value instead.
Also fix custom_map always being empty when refinement_key is set.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Read-only horizontal timeline showing detected scenes as boxes with
transition connectors between them. Backed by a new
POST /funpack/parse_timeline route and parse_timeline_segments() in
conditioning.py that reuses the existing transition phrase patterns
and custom transition map.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add confirmed-bad-combo warnings to both parameter tooltips, the sampler
doc, and the changelog. Warn users not to combine frame_overlap=0 with
carry_i2v_guides=True in normal use.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
