Restores the old "Loved it" learning behavior (reward 0.75, intent delta
+0.03, preferred context 0.65, axis conditioning 0.35, LoRA 0.44/0.24)
as a distinct rating. Sits between Perfect and Loved it on the ladder:
prompt adherence is rewarded but not at full strength.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously a weaker "Perfect". Now means: output looks great, prompt
adherence is irrelevant. Skips missing_intent_phrases learning, skips
preferred context lock-in, zeroes intent phrase reinforcement. Keeps
conditioning direction delta and raises value function reward to 0.85.
Quality/style LoRAs rewarded at Perfect level (0.55x).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two server routes (GET /funpack/value_function/export, POST /funpack/value_function/import)
serve and receive the .pt binary. Studio Session tab gains Export and Import buttons
below the value guidance toggle, keyed to the active refinement key.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
torch.load under inference_mode creates inference tensors; MLP params then
can't participate in backward. Same fix as the earlier value function loader
in samplers.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
reset_session now deletes the value function .pt file alongside the
enhancement files. Previously it was orphaned across resets.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Before returning modified_positive, ascend each scene's conditioning tensor
toward higher predicted reward via the value function. Covers both the
standalone Refiner V2 node and FunPack Studio (which delegates to refine_v2).
No-op until value function reaches MIN_SAMPLES; handles transition-split
multi-scene conditioning correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Before each denoising run, when the value function is ready, gradient-ascend
the text conditioning until predicted reward plateaus (self-terminating: step
size = 0.5% of conditioning norm, stops when delta < 1e-3, caps at 50 steps).
No user-facing params — fully automated. Per-step wrapper still runs on top.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Model weights loaded via torch.load inside ComfyUI's inference_mode become
inference tensors and can't participate in backward. Wrapping the load in
inference_mode(False) ensures parameters are normal autograd tensors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
.clone() on an inference tensor inherits the inference flag even inside
inference_mode(False). torch.empty() inside that context creates a true
normal tensor; copy_() fills it from the inference tensor without flag
inheritance.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
MLP lives on CPU, conditioning on GPU — device mismatch silently fell back
to fixed direction. Now moves conditioning to MLP device for the forward/
backward, then returns gradient to original device.
Also surface value function fallback errors instead of swallowing them.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Splitting now only fires on user-defined custom triggers (from templates)
plus the generic 'scene N' numeric pattern. The 40+ predefined English
phrases were causing unwanted splits in user-authored prompts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nn.Linear weights are inference tensors if instantiated under ComfyUI's
inference_mode context. Moving the entire create+train block inside
inference_mode(False) ensures model weights are normal autograd tensors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Buffer tensors were created inside inference_mode — detach().cpu() doesn't
escape the inference flag. Wrapping the whole method ensures the clone and
all subsequent tensor ops produce normal autograd-tracked tensors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
torch.enable_grad() cannot override torch.inference_mode() in older PyTorch.
Add inference_mode(False) as outer wrapper in both train_on and gradient.
Also .clone() tensors from buffer/input before grad tracking to avoid
inference_mode tensor view restrictions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ComfyUI execution context may have torch.no_grad() active, which kills
gradient tracking needed for training and gradient computation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
value_function.py: OnlineValueFunction MLP that trains incrementally on
(conditioning, reward) pairs. Uses replay buffer of 100 samples, 20
gradient steps per update. save/load to .pt file via refinement_state_path.
Studio popup: new "Value guidance" toggle. When on, trains the value function
after every rated generation (good and bad). Falls back to no-op if conditioning
deserialization fails.
chain sampler: _load_value_function reads the trained MLP if ready (5+
samples). _build_embed_guidance_wrapper uses value function gradient
(∂reward/∂conditioning) when available, fixed liked direction as fallback.
Log reports which mode is active and sample count.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extends the Refiner's one-shot conditioning steering to apply at every
denoising step. Reads the liked conditioning direction from the Refiner's
state file via refinement_key_input. At each step, nudges c_crossattn toward
the learned liked direction with sigma-scaling (no effect at sigma=1, full
effect below sigma=0.5 where detail refinement happens).
Requires 3+ liked generations to have a meaningful direction. No-ops silently
if direction is unavailable.
New inputs: embed_guidance (BOOLEAN), embed_guidance_strength (FLOAT 0.005-0.1).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Below 0.25: audio degrades to noise (guide frame in latent confuses
cross-modal attention without enough video coherence), character appearance
drifts while background over-anchors to previous scene.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Empirical testing: 0.25 is the sweet spot — locks character position and
action pixel-perfect while allowing background to drift freely toward the
new scene's prompt. Above 0.35 causes character-space conflicts when scene
composition shifts significantly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_prepend_soft_continuation creates the noise mask with ones_like(video_tensor)
giving it 128 channels instead of 1. Using torch.full_like(masks[0][:,:,:1])
copies the exact shape signature from the existing mask so the cat always works
regardless of channel count.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
guide_mask was [B,1,1,1,1] but noise_mask is [B,1,F,H,W] — cat on dim=2
requires matching H,W. Now uses masks[0].shape[3:] for spatial dims.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previous approach (post-block hidden state injection) was fundamentally
broken for LTXAV: modifying video tokens after a block contaminates audio
through joint audio-video attention, causing audio noise and latent bleed.
New approach: append the middle frame of the previous scene as a guide
token to the END of the current chunk latent, with keyframe_idxs pointing
to F//2 of the current scene in pixel space, plus guide_attention_entries
at the specified strength. The model's existing _attention_with_guide_mask
mechanism attenuates attention weights inside the attention computation —
audio tokens never see modified video tokens.
After sampling, _crop_video_tail removes the guide frame.
New inputs: mid_scene_guide (BOOLEAN), mid_scene_guide_strength (FLOAT 0.1-0.8).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Reads settings.refiner.vision_conditioning (default true) and gates
source_image passed as reference_image to Gemma3 encoding. Toggle lives
in the popup UI next to Reference injection, not on the node.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Old: average all tokens in middle third → one scalar direction → spread to
surrounding zones. Loses all spatial structure, can't preserve positioning.
New: take the exact spatial token map of the single middle frame (F//2),
inject it position-for-position into every other frame. Token at (h,w) in
the anchor frame maps to (h,w) in all other frames. Character layout and
positioning stays spatially consistent across the scene. Anchor frame
itself is left unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Re-encoding the next scene's conditioning with the previous scene's
last frame is useless for fade-to-black workflows — the last frame
is black, so it's conditioning on nothing. Removing the whole path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Middle-frame tokens are the anchor source — blending them with their
own mean was pointless self-contamination and the main cause of
collapse at higher strengths. Now only the surrounding frames
(before mid_start and after mid_end) receive the injection; the
anchor zone is left untouched each step.
Max strength capped at 0.5 in the UI; tooltip warns against going high.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sampler vision re-encoding: prefix scene text with
"Continuing from this scene: {text}" so the model treats the
previous frame as temporal context, not an unrelated subject.
Studio per-scene reference conditioning: re-enable reference_image
with prefix "The character shown performs the following: {encode_text}"
so the model knows the image is a character reference, not something
to describe. Only applies when source_image is connected.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Vision encoding rewrites the full conditioning embedding — with a 512-token
limit, a poor generic description from the template displaces the actual
scene prompt content. Reverting until the template can be properly designed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previous: capture middle-frame tokens once at first sigma < 0.95, reuse
for all remaining steps — injected noisy step-6 states into cleaner
step-7/8 passes, causing latent noise flooding.
Now: stateless per-step operation. At every step below sigma=0.95,
compute the mean of the current step's middle-frame tokens and blend
it into all token positions at the given strength. Anchor is always
from the current denoising state so no high-sigma artefacts persist
beyond the step they were produced in.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_v2_transition_scene_conditionings was encoding per-scene texts without
the reference image — only the main single-scene conditioning path passed
it through. Now passes reference_image=source_image to all per-scene
encodings so every scene conditioning includes the character reference
via Gemma3 vision.
Removes the redundant reference_conditioning option from the chain sampler
(which required a wasteful VAE decode mid-sampling to get the pixel image
that Studio already has).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
reference_conditioning (safe):
Re-encodes every scene's conditioning with the original reference image
as Gemma3 visual context. Reference frame is VAE-decoded once before the
loop. Gives the model character identity in the text conditioning for
every scene, not just the i2v latent anchor. Requires CLIP connected.
self_consistency (experimental):
Registers forward hooks on IDENTITY_BLOCKS [14,20,21,30,33] that capture
hidden states from the middle third of the video (by token index) at the
first sigma < 0.95 crossing, then softly inject them back into those same
tokens for remaining steps in gate (0.35, 0.85). Intended to reduce
within-scene character drift across long clips.
New input: self_consistency_strength (default 0.05).
Hooks are registered per-scene and cleaned up after each sample call,
restoring any pre-existing model_function_wrapper.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LTX-2 is not designed for creative i2i — distilled schedule gives no
reliable middle ground between 1-step (unchanged) and full regeneration
(no character identity). Removing the feature entirely rather than shipping
something that doesn't work.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Single-frame i2i against LTX's distilled schedule is unworkable: low
strength = 1 step (barely changes), high strength = full regeneration with
no character identity.
New approach: pin frame 0 (character reference) at mask=strength, generate
frame 1 freely with the new scene prompt, return frame 1 as the anchor.
The model has proper i2v context and produces character-consistent results.
i2i_strength now means: 0.0 = reference tightly pinned, 1.0 = ignored.
Default changed from 0.75 to 0.3. _i2i_sigmas removed (full schedule now).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two fixes:
- Truncate sigmas to start at strength level (_i2i_sigmas). Full schedule
from σ=1.0 lets the distilled model reconstruct the input unchanged even
at moderate strength values. Now denoises only from σ≈strength downward.
- Restore _build_continuation_chunk: overlap frames are pinned normally at
0..video_overlap-1, i2i anchor placed at offset=video_overlap. Scene starts
as smooth continuation then cuts to the new anchor. Latent blend remains
no-op because overlap frames match previous output.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Last frame of previous scene is black when fade-to-black is used — all
generated anchors came out black. For hard cuts the correct source is always
the original character reference (latent_template frame 0).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When i2i is active the two sides of the seam are different content —
blending them in latent space produces morphing artifacts, not a transition.
Force overlap=0 at the blend step for i2i scenes. Pixel-space transitions
(fade-to-black etc.) still apply independently via boundary_entries.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
i2i_scene_cut now builds a fresh chunk from the template regardless of
frame_overlap. The overlap=0 gate was silently blocking the feature for
anyone not already on hard-cut settings, producing black reference frames
and no anchor generation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When frame_overlap=0 and i2i_scene_cut=True, generates a new starting frame
for each continuation scene by denoising the previous scene's last frame with
the new scene prompt (i2i strength-controlled). The result is pinned as frame 0
of the next chunk (mask=0, i2v style), replacing soft continuation.
reference_frames output collects all generated anchor images as an IMAGE batch.
New inputs: i2i_scene_cut (BOOLEAN), i2i_strength (FLOAT 0.0–1.0).
New output: reference_frames (IMAGE, slot 2 — shifts status/scene_count/etc up by 1).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remnants of the old append-guide-to-end approach (abandoned in e8c3b32).
The current prepend approach needs neither: the guide lands at temporal
position 0 naturally via the causal patchifier, no keyframe_idxs required.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If the character description ends with . ! ? or , use a space separator
instead of injecting a redundant comma. Only bare text gets the ", " join.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Option 1 (samplers.py): Add optional clip input to FunPackLTXAVSceneChainSampler.
After each scene is sampled, decode its last frame and re-encode the next scene's
conditioning with that frame as vision context. This gives identical scene texts
different conditioning based on temporal position, so the model knows what state
it came from.
Option 2 (conditioning.py): When building scene conditionings, detect duplicate
scene texts and prepend "Returning to an earlier scene: " before encoding the
second+ occurrence. Stores original text in funpack_scene_text for logging and
the actual encode text in funpack_encode_text so the sampler uses the same prefix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When frame_overlap=0, use the last 4 frames of the previous scene as soft
continuation guides (mask=0.4) instead of a single fully-pinned frame (mask=0.0).
This lets the model use previous-scene context for identity continuity while
retaining freedom to change orientation from the text prompt. The frames land
at positions 1-4 after the i2v guide at position 0 (when carry_i2v_guides is
enabled). crop count is returned dynamically so the existing crop/blend logic
requires no changes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LTXAVTEModel.load_sd passes the full checkpoint sd to Gemma3_12B.load_state_dict,
which has vision_model and multi_modal_projector as real attributes with matching
key names. Vision weights are already present after DualCLIPLoader runs.
Drop _vision_loaded flag check; detect vision capability by attribute existence.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
