MixCache/Chorus-family step-caching adapted to LTX2.3's distilled schedule.
On the near-pure-noise plateau (sigma >= threshold, default 0.975) the
transformer output barely changes, so compute the base-model forward once
and reuse it for the remaining plateau steps, skipping ~3-4 of 8 full
48-block forwards. Below threshold every step computes for real and the
cache drops, so structure formation is never approximated. Deterministic
given seed -> safe in Batch Training, no rating-diversity impact.
Installed innermost as a per-scene model_function_wrapper (existing
tag/strip leak hygiene) so guidance wrappers still post-process each step's
cached-or-fresh prediction. Cache keyed by (input.shape, cond_or_uncond).
Editor: plateau_cache + plateau_cache_threshold in Engine -> Experimental,
auto-wired via the sampler_inputs bridge (no backend/builder change).
Off by default, UNVALIDATED LIVE. (Chorus cross-variant reuse rejected --
would blur batch-rating diversity; FreeSpec dropped -- wrong topology.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
model=object() lacked model_options (needed by the per-scene wrapper
snapshot/restore added for the scene-wrapper-leak fix), and FakeVAE lacked
decode() (needed since the sampler always decodes IMAGES when the caller
doesn't declare that output as disconnected) — together these silently
killed every test in the file that calls sample(). Added FakeModel and
FakeVAE.decode, then fixed the resulting return-tuple arity (sample() now
returns 6 values, these tests unpacked 4) and three genuinely stale
assertions: the per-scene report line gained a timing segment, carry_i2v_guides
now prepends its guide frame instead of appending it, and _append_i2v_guides
no longer routes through keyframe_idxs/guide_attention_entries.
Verified in isolation: all 14 tests in this file now pass. Running the full
tests/ suite still shows the same file failing — that's a separate,
pre-existing cross-file sys.modules stubbing collision (confirmed via
git stash on a clean dev checkout) affecting this file plus
test_mask_safe_attention.py and test_v2_category_alignment.py; not fixed
here since it isn't a one-line root cause.
carry_overlap_through_anchor lets a scene keep the previous scene's latent
tail for background/environment continuity even when it switches to its own
i2v anchor, instead of the current hard cut. The anchor's own leading frame
stays untouched; only the frames after it carry the old tail.
Also fixes a real gap found along the way: the mixed-anchor branch never
called _apply_configured_guides, so Best-FaceID identity_transfer could
never resolve an identity_pin on an anchor-swap scene — exactly where it's
most useful. Added a standalone lookup so it resolves there too.
Replaces the source-phase-RoPE-only approximation with a faithful native port of
ComfyUI-BFSNodes' LTX Identity Transfer (read upstream source directly): the reference
face is now injected as separate, non-rendered overlap tokens (clean timestep, RoPE
source-phase tag, trimmed before output) instead of blending into Continuity's identity-pin
keyframe, plus an optional ArcFace embedding -> IdentityProjector -> context-token channel
(identity_projector, source_id, phase_scale, id_strength, arcface_mode, debug_log). Stays
on Chain Sampler per the Studio/Sampler boundary law's guide-ops carve-out. New
identity_transfer.py module; insightface added as a soft pip dependency.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reverts the earlier ComfyUI-BFSNodes graph-splice approach (builder.py/timeline.py/
server.py scaffolding) in favor of porting the missing piece natively: FunPack's own
guide-attention path (LTXVAddGuide.add_keyframe_index + guide_attention_entries)
already places a reference image on the frame-0 grid exactly like BFSNodes' overlap
technique, and Continuity's existing "Identity pin" already applies one reference
face to every scene. The only real gap was the LoRA's trained-time source_phase RoPE
tag, added as a tag+strip patch on FunPackLTXAVSceneChainSampler scoped to the exact
guide-entry token range (never co-active mid_scene/carry/prior-scene guides sharing
the same keyframe_idxs tensor). New "Best-FaceID compatibility" + "Source-phase id"
knobs live under Engine Settings -> Experimental, alongside every other unvalidated
technique. LoRA loading itself reuses the existing generic Models-menu lora role.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
alg_blur_guides no longer borrows the sampler's anchor knobs: the Scene Chain
Sampler gains alg_guide_blur_strength (default 2.0) and
alg_guide_blur_sigma_threshold (default 0.975), threaded per-chunk via
extra_options like the tail count. The sampler now precomputes one latent per
(anchor blurred?, tail blurred?) combination — the frame sets are disjoint, so
the both-blurred variant composes the two passes — and each region follows its
own sigma window per step. Anchor ALG (alg_enabled/alg_strength/
alg_sigma_threshold) is untouched and back to anchor-only semantics.
Editor: 'Guide blur strength' and 'Guide blur sigma threshold' rows appear
under Engine ▸ Experimental when the blur toggle is on (sampler_inputs bridge,
names match the new node widgets; inputs appended at the block end to keep the
builder's positional widget mapping aligned).
Verified live: object_info shows both widgets appended last; Editor rows
appear on toggle and persist to project.sampler_inputs. 11 ALG unit tests
pass (new: composed independent-strength blur); same 20 pre-existing suite
failures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Editor ▸ Settings ▸ Editor gains a 'Shortcut revolver' toggle (+ 'Random' sub-option).
When on, each shortcut with several replacements draws from a per-shortcut queue —
sequential (first…last, then restart) or shuffled once per cycle — and nothing repeats
until the whole set has fired. Off (default) keeps the seeded random pick unchanged.
- State is a sidecar (shortcut_revolver.json) next to the shortcut DB: settings +
per-shortcut remaining queue + replacement-set fingerprint (editing a shortcut's
replacements restarts its cycle; changing either setting resets all cycles).
- Generation commits the draw (refine_v2 positive prompt); every preview path
(parse_timeline, split mapping, fragments) peeks the same stored queue without
saving, so previews show exactly what the next Generate will use.
- Batch Training variants each commit a draw: N variants sweep N consecutive
chambers instead of seed-rolled repeats.
- FunPackStudio.IS_CHANGED returns nan while the revolver is on so fixed-seed
re-runs aren't served from ComfyUI's node cache (the next chamber must fire).
Verified live on local ComfyUI: settings round-trip, seed-independent stable peeks,
sequential cycle REP_ONE→TWO→THREE→ONE across processes, peek-after-commit shows the
next chamber. 9 new unit tests; full suite has the same 20 pre-existing failures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'Awful' / 'Wrong appearance' banked on intrusion semantics; now any rating
whose profile reward marks the run a bad OUTCOME (<= -0.25: Missing
quality, Missing details+quality, Missing action+quality, Missing
details+action+quality) also feeds the negative bank - degradation is
visible in the latent, so steering off that attractor is meaningful.
Near-miss ratings with positive reward (Missing details +0.35, Missing
action +0.05) stay excluded: those latents are mostly-wanted content and
banking them would repel future runs from material the user asked for.
Decision moves into negative_memory.is_negative_profile (intrusion keys OR
reward threshold, robust to future profiles); promoted entries now carry
the rating key as provenance for severity-weighted steering later.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Store what went WRONG, not just what was asked for. Ratings 'Awful' and
'Wrong appearance' promote that run's final video latent (fp16, batch
squeezed) + mean-pooled conditioning into a per-key negative bank
(<key>.negative_latents.pt, ring buffer of 8). The Chain Sampler writes a
pending candidate at the end of every keyed run; the rating that scores
that run promotes or discards it - same pairing as the x0 snapshot, and
the stem glob in delete_refinement_key covers both new sidecars.
Sampling (dynashift toggle + strength/threshold, off by default):
per late step each current video frame is pooled and compared against
every banked negative frame (alignment-free in time, so chain position /
guide tails don't matter; other resolutions are skipped). Frames matching
above threshold get the matched negative frame's aligned component
subtracted (projection removal, coef clamped >= 0), scaled by
above-threshold margin - steering self-releases once the unwanted feature
is gone. Negatives are weighted by prompt similarity so bad gens from
unrelated prompts steer less. No uncond pass, no extra model forward;
audio byte-identical (video span slice + cat). Wrapper rides the tagged
per-scene lifecycle (leak-strip covered) and threads through batch
training.
Editor: DynaShift knobs land in Engine Settings > Guidance same-change
(sampler_inputs pass-through, no backend change), verified rendering live.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two audit findings on the experimental feature stack:
1. Leak: an interrupt or OOM inside _sample_chunk skipped the per-feature
unwind blocks, leaking embed_guidance/score_slider/output_guidance/temporal
wrappers on the shared ModelPatcher - every later run then chained new
wrappers onto the leaked ones and double-steered (same in-process
progressive-corruption mode as the old block-hook leak).
- one base-wrapper snapshot per scene, restored in the same finally that
removes the v2a hooks; the four conditional unwind blocks are gone
- every per-scene wrapper is tagged with its predecessor; sample() strips
leaked tagged chains at run start (mirrors strip_funpack_block_hooks)
2. Inert feature: output_guidance applied the RAW value-fn gradient, whose
per-element magnitude is O(1/pool_window) - at video-latent sizes the
correction was ~1e-6 relative, i.e. numerically a no-op. It also fed the
full packed AV tensor to a value function trained on the video stream
alone (shifted pooling buckets + audio contamination).
- gradient now computed on the packed latent's video span only (matches
the training domain of _save_output_value_snapshot)
- delta norm-calibrated: strength = fraction of video-stream norm per
fully-ramped step (0.02 = 2%), same relative convention as the slider
- audio region byte-identical by construction (cat, not post-hoc mask)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Block steering: run-averaged block activity is architecture-determined
(varies in the 4th decimal run-to-run), so no scoring scheme - pole EMAs
or reward correlation - could extract a rating signal from it. The idea
needs within-run per-step measurement to have a chance; that is a
research project, not a feature. Removed: module, sampler wiring, the
two trailing widgets (positional reference mapping unaffected), the
Refiner pairing hook, editor knobs, tests. Stale block_snapshot/
block_profile sidecars are inert and covered by the existing
delete_refinement_key sweep.
FunPackNVFP4ModelLoader: the FP4 GEMM path was real (3.35x at 16k
tokens) but end-to-end the user's runs are dominated by everything the
loader cannot touch (attention, prep, encode - the sampler node is
~23s of a ~60s run) while dynamic input quantization added a fixed
per-layer-call toll. Net observed: overhead. Users are better served
by dedicated NVFP4 loaders. Removed: module, registration, tests.
Kept (independently useful): the chain sampler's phase timing report +
quantized-layer census, the nested-AV x0 snapshot fix, and the sm_120
masked-attention work.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Justified by live data: sampler phase timing shows sampling at 89% of
the chain node, all 792 quantized layers on the fast path, and the probe
puts dynamic input quantization at ~0.30 ms/layer-call vs ~0.11 ms with
a provided scale (amax dominates) - a top-line per-call cost.
New input_scales widget on the loader:
- "auto-calibrate", first load without a sidecar: forward-pre-hooks
record each quantized layer's running input amax (one generation is
enough; sidecar under refinements/nvfp4_scales/<checkpoint>.json,
per-machine and rebuilt in one run whenever absent).
- every later load: injects input_scale = amax * 1.25 / (448*6) into
the state dict next to each quantized weight - core's loader
registers it natively and skips the per-call amax from then on.
Margin leaves clip headroom (NVFP4 clips above the scale).
Tests: suffix-matched injection across checkpoint prefixes, value math,
core Linear registering input_scale and forwarding, sidecar read.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live report: "flat activity across runs" with a working profile. Root
cause is real, not a threshold tuning issue: the fingerprint is each
block's MEAN residual activity over ~1200 steps, which is practically a
property of the architecture - run-to-run variation sits in the 4th
decimal, so raw reward-covariance is numerically dead. Lowering the
epsilon would have "fixed" it by steering on noise.
Now: per-block Pearson CORRELATION between reward and activity (scale-
free - tiny variations count when they consistently track ratings),
fingerprints stored mean-normalized so run-global scale (step counts,
schedulers) can't masquerade as block signal, and a confidence ramp
(min(1,(runs-2)/6)) so a 3-run profile steers gently instead of not at
all, reaching full strength ~8 runs. MIN_RATED_RUNS 2->3 (correlation
over 2 runs is degenerate). Status lines now carry the evidence:
"ready (N runs, peak |corr| X, confidence Y)". Older profiles are
discarded on first pairing (fingerprint semantics changed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'Loaded fine' and 'runs fast' are different claims. Every NVFP4 load now
prints a diagnostics block that settles which one you have:
- comfy_kitchen backend census (the big one: torch built for CUDA < 13
disables ck's CUDA backend, so quantized matmuls silently fall through
to the pure-PyTorch eager backend - correct output, ZERO speedup, with
an explicit warning line when detected)
- torch/cuda build + device capability
- loaded-model layer census: nvfp4 layers vs layers flagged
full-precision emulation (pre-Blackwell GPUs)
- a timed 4096x4096 GEMM probe (incl. dynamic input quantization):
bf16 vs nvfp4 wall time with a FAST PATH ACTIVE / NO SPEEDUP verdict
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three live-run findings from the rental:
1. Block steering said "profile not ready (needs 2+ rated runs)" with 16
paired runs. The v1 liked/disliked-EMA attribution had no contrast when
ratings cluster on one pole (pole-vs-all-runs-mean cancels to zero) and
dropped rewards in (0, 0.3) entirely. Attribution is now the per-block
COVARIANCE between reward and activity over a rolling 32-run history:
every admissible reward contributes with its actual value, contrast
comes from reward variance, and the sampler log now states the real
blocker ("all rated alike - rate both good AND bad runs") instead of a
misleading run count. v1 profiles are discarded on first pairing.
2. output_guidance stayed "not ready (needs 10+)" forever on LTX-AV: the
end-of-run x0 snapshot was gated on output samples being a plain
torch.Tensor, but AV chain output is a NESTED tensor (video+audio) - so
the output value function never received a single sample. The snapshot
now extracts the video stream from nested outputs.
3. FunPackNVFP4ModelLoader hit a size-mismatch wall (9-vs-6 ada params,
connector dims): it loaded the state dict without the safetensors
METADATA that carries the LTX-AV architecture config, so detection
built the wrong model variant. Metadata now flows through, mirroring
comfy.sd.load_diffusion_model.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RLHF at the model level: which REGIONS of the transformer were doing the
work when a generation came out good or bad, and steer accordingly.
1. Measure: tagged forward hooks record each block's relative video-residual
delta per model call (near-zero cost; covered by the existing leak-strip).
2. Attribute: the end-of-run fingerprint is staged (block_snapshot.pt, same
stage-then-pair convention as x0_snapshot) and paired with the rating
reward on the next Refiner cycle into liked/disliked/all-runs EMA
profiles; when output_guidance's value function is trained, an innermost
model-function wrapper additionally credits blocks per step by
(value delta) x (centered activity) - converges in a handful of runs.
3. Steer: vx' = vx_in + g_b*(vx_out - vx_in) on the video stream only,
audio untouched; scores zero-mean/unit-max so emphasis is redistributed
rather than inflated, gains hard-capped at +/-10%. Exact no-op until the
profile has 2+ paired runs; blocks at gain 1.0 get no hook at all.
Chain Sampler toggle + strength (append-only widgets), exposed same-change
in the editor's Chain Sampler Guidance section via the sampler_inputs
bridge. New sidecars ride the defensive <key>.* sweep in
delete_refinement_key, so atomic key deletion covers them.
Tests: toy AV-block model driving the real hooks (recorder ranking, steer
video-only scaling + native path for unit gains, credit wrapper, snapshot/
profile/scores/gains roundtrip, mild-positive band, depth-mismatch reset,
leak-strip tagging). Full suite failure set identical to the pre-existing
baseline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FunPackNVFP4ModelLoader quantizes an existing bf16/fp16 checkpoint to NVFP4
at load time as a pure state-dict transform: selected 2D Linears are
re-serialized in ComfyUI core's quantized-checkpoint format (packed fp4
weights + fp8-e4m3 block scales + fp32 tensor scale + per-layer comfy_quant
metadata) and handed to the stock loading path. Core's MixedPrecisionOps
then does everything: real FP4 tensor-core GEMMs on sm_100+ (sm_120 incl.),
dynamic activation scales (no calibration pass), per-layer LoRA fallback,
and dequant emulation on older GPUs. No custom forward, no monkeypatching.
Zone-aware precision map for the joint AV transformer: default scope
quantizes only video-branch Linears (attn1/attn2/ff) of the middle blocks;
the audio branch, the audio<->video cross-modal bridges, the first/last N
blocks, and all non-matmul weights stay full precision. Scopes widen to
+cross-modal, all blocks, and all 2D layers for experimentation.
Tests drive the REAL comfy.quant_ops/comfy.ops via comfy_kitchen's eager
backend on CPU: scope selection, serialized-format contract, prefix
handling, refusal of pre-quantized checkpoints, and a load+forward
roundtrip through MixedPrecisionOps on both the NVFP4 and emulation paths.
The Movie Editor Models modal picks the node up automatically as a
Unet/Diffusion Model candidate (object_info-driven).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On compute capability > (9, 0) GPUs (Blackwell sm_120, e.g. RTX PRO 6000),
xformers' memory_efficient_attention has no kernel that accepts a tensor
attn_bias — fa3F/cutlassF require cap <= (9,0), fa2F/fa3F reject a tensor
bias. So unmasked attention works (i2v ANCHOR scenes generate) but MASKED
attention crashes: scenes generated from GUIDES hit the LTX guide path
(comfy/ldm/lightricks/model.py _attention_with_guide_mask) which passes a
tensor mask → "No operator found for memory_efficient_attention_forward".
This is the ComfyUI-wide issue normally worked around with
--disable-xformers --use-pytorch-cross-attention; when that flag isn't in
effect on an instance, the guide path still reaches xformers.
Fix: _funpack_install_mask_safe_attention(model), called at the top of the
chain sampler's sample(). When the active backend is xformers AND the device
capability is > (9,0), install ComfyUI's per-call optimized_attention_override
(the wrap_attn hook) so MASKED attention falls back to attention_pytorch
(SDPA handles masks on every GPU) while UNMASKED attention stays on the fast
xformers path. Threaded via transformer_options so it reaches every scene's
forward. No global monkeypatch; no recursion (wrap_attn stamps
_inside_attn_wrapper before calling the override). No-op on supported GPUs,
non-xformers backends, or when another override is already installed;
idempotent.
Tests: tests/test_mask_safe_attention.py — 9 cases covering override routing
(masked→pytorch via kwarg or positional, unmasked→original) and install
gating (xformers+Blackwell installs; supported cap / non-xformers / no-CUDA
skip; existing override respected; idempotent). Run with the ComfyUI venv.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Generalizes the i2v-anchor blur (_alg_blur_video_frame0 -> _alg_blur_frames)
to optionally also blur newly-appended guide-attention frames each scene
(mid_scene_guide / carry_i2v_guides-as-guide / configured guides / JoyAI
memory), reusing the existing guide_tail bookkeeping rather than a mask-value
heuristic (frame_overlap continuation also uses mask=0 and must stay
untouched). New Scene Chain Sampler toggle "Blur i2v guides and JoyAI memory"
mutates the shared KSAMPLER's extra_options per-chunk since guide_tail varies
per scene; always resets so nothing leaks across chunks/samplers. Surfaced in
Engine Settings via the existing Experimental section, no backend/builder
changes needed beyond the new INPUT_TYPES field.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adaptive Low-Pass Guidance (arXiv:2506.08456): blurs the i2v anchor frame
while sigma is in the near-pure-noise region, then swaps back to the sharp
anchor, to stop the model shortcutting to a near-static video that just
matches the reference image. Opt-in, off by default, no-op without an i2v
anchor. Swaps model.latent_image per step (the same attribute ComfyUI's
inpaint compositing already re-pastes every step) instead of monkeypatching
comfy internals.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Deleting a refinement key was non-atomic. A key's state spans <key>.json plus
sidecars (<key>.value_fn.pt, <key>.sampler_ctx.json, attn_maps/blessed maps +
K/V banks + creativity latent + velocity store), and there was no delete route —
users removed the visible <key>.json by hand, orphaning the rest. The orphans are
still applied: _v2_ascend_conditioning loads <key>.value_fn.pt purely by name
(not gated on the json), and blessed K/V inject at a 0.04 floor even at reward 0.
So a "cleared" key kept steering future generations, surviving both a ComfyUI
restart and the manual delete — the persistent half of the long-run quality drift
(the per-session half was the hook leak in the previous commit).
- templates.py: delete_refinement_key() sweeps the canonical json + every <key>.*
sidecar + clear_refinement_data (attn_maps banks, velocity disk+memory).
absolute_store_info()/clear_absolute_store() for the keyless Absolute taste store
(__funpack_absolute_global__): it learns from every rated gen across all prompts,
is invisible in the key list, and was otherwise only wiped by Session Reset.
Routes: POST refinement_keys/delete, GET/POST refinement_keys/absolute|clear_absolute.
- Refinement Key Loader node: "Delete Key" + "Clear Global Taste" buttons.
- Movie Editor: api deleteRefinementKey/absoluteStoreInfo/clearAbsoluteStore +
FunPack menu "Delete refinement key…" / "Clear global taste store…".
- tests: atomic-delete sweep + empty-key guard (real store, unique key, cleanup).
Pre-existing 16 failures unchanged (stale mocks); +2 new passing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend the JoyAI cross-shot memory bank from video-only to paired audio-video,
the part that distinguishes JoyAI-Echo from a generic video-consistency trick.
- _JoyAIMemoryBank now stores (video, audio) pairs under the same pin/roll policy;
frames()/audio() stay 1:1. Audio half is None until joyai_audio_memory is on, so
the phase-1 video path is byte-identical.
- _append_joyai_audio_memory pins each banked prior-shot audio latent into the
chunk's audio stream as a fully-protected (mask=0) tail, cropped back off after
sampling (_crop_audio_tail). The audio analogue of the video guide bank; rides
native audio self-attention since LTX has no audio guide-attention API.
- _harvest_joyai_audio harvests the paired audio frame, position-matched to video.
- v2a_grad_scale scales the model's trained video->audio cross-attention via
reversible forward hooks on each AV block's video_to_audio_attn submodule
(JoyAI uses 2.0). 1.0 installs no hook -> native behavior, zero overhead.
- joyai_audio_memory + v2a_grad_scale wired through sample(), batch training,
refinement context, run_mechanisms, INPUT_TYPES (append-only), and editor knobs.
Deliberately breaks the audio pass-through invariant, so both are opt-in and
off/native by default. 3 new bank tests; zero regressions (same 12 pre-existing
stale-mock failures before/after).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Diagnostic conclusion from the JoyAI-Echo engine: conditioning is fed to the sampler
unchanged (gemma+connector, which Studio already produces), so the ONLY missing sampler
feature is generate_BidirectionalMemory's cross-shot paired AV memory bank. This lands the
VIDEO half: a managed set of clean prior-shot frames injected into each scene so identity
carries across the whole chain — JoyAI's story-level consistency.
Generalizes the single-anchor mid_scene_guide to N frames, reusing the same LTX
guide-attention primitive (_append_guide_latent). _JoyAIMemoryBank holds the policy
(num_fix opening shots pinned as a global anchor + rolling most-recent window capped at
max_size, matching configs/inference.yaml: max_size 7, num_fix_frames 3, center select).
Harvest one clean frame per finished scene; inject the bank before each continuation scene;
supersedes mid_scene_guide when on. Honors the 0.25 audio-safe strength floor.
Off by default, so the audio protect/pass-through invariant is unchanged. Paired audio
memory + v2a_grad_scale coupling is the next phase (needs the ltx_core v2a term).
Inputs: joyai_memory, joyai_memory_size, joyai_fix_frames, joyai_frame_select,
joyai_memory_strength. Threaded through batch-training; logged in run_mechanisms + the
refinement sampler-context. New test_joyai_memory.py covers the bank policy (6 tests).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The score_slider synthesized a symmetric +/-liked pair, so its axis was
"liked vs anti-liked mirror" — meaningless as an away-from-bad signal. When
3+ disliked/awful gens have populated the existing bad_dir slot in the same
taste store, build the slider's minus pole from that real disliked direction
instead. The (eps_+ - eps_-) axis becomes good-vs-bad, actively steering the
noise prediction away from the conditioning region that produced rated-bad
gens. Falls back to the symmetric mirror when no bad_dir exists.
Reuses existing plumbing: bad_dir is already accumulated identically to
liked_dir (running-averaged unit-norm delta from session mean). Generalized
the direction loader to _load_taste_direction(slot); load + pass bad_dir at
the call site; log/run-mechanism note which pole is active.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-scene prompt preview mirrors what Studio encodes, and the postfix is
appended to every scene at generation — so it belongs there. Surface it once
after the scenes as a 'postfix' badge, symmetric to the 'anchor' badge shown
once above them.
- bridge.expand_prompt_fragment: expands a standalone fragment (anchor/postfix)
the same way generation does, so the previewed text matches exactly.
- server preview: result["postfix"] = expanded effective_postfix(p).
- inspector: postfix badge line; _promptPreviewKey includes postfix so editing
it refreshes the preview live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A shortcut in the postfix is expanded at the generation hit (same split_scenes
path as the anchor) and its refinement key folds into every scene. The raw
postfix is preserved on the project for round-trip; expansion is generation-time.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause the user pushed on: the editor flattened all scenes into ONE prompt
string and Studio re-derived boundaries by parsing text, so a carry scene with no
transition needed an injected 'scene N' delimiter. The editor already KNOWS every
boundary (each generative unit is a scene) — it never needed to smuggle markers
into the prompt.
- timeline.build_combined_prompt: no longer injects anything (verbatim anchor +
scene texts). New build_generation_scene_segments(project) lists the boundaries
the editor already has (each unit's raw text, prefixed with its REAL transition
so effects survive; carry = plain text).
- conditioning.split_scenes_from_segments: canonical split from an explicit scene
list (reuses split_scenes per item for expansion/keys/effect, folds the anchor),
same output shape as split_scenes. refine_v2 gains scene_segments and uses it
when present (count-preserving: a text-less i2v scene keeps its slot, no anchor
shift); falls back to text-splitting only for raw node-pack use.
- server/builder: editor sends scene_segments -> studio_settings.refiner.scenes.
Net: the generation prompt Studio encodes is now exactly what the user typed —
no 'scene N' anywhere, in the prompt or the conditioning. Tests updated to assert
the verbatim prompt + structural segments; new tests for split_scenes_from_segments
(boundaries, empty-scene preservation, anchor/key fold). Fallback path unchanged
(stash-verified zero regression). NOTE: montage generation needs GPU verification.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Multi-scene bug: the Movie Editor injects a generic 'scene N' separator before
any scene that lacks a leading transition trigger (build_combined_prompt), purely
to force a boundary. split_scenes was splitting on those markers with placement
'start', which KEEPS the marker word in the new scene's text -> 'scene 1',
'scene 2', ... got encoded into every scene's conditioning. With a text-less
leading i2v scene this also produced a label-only first scene (renders as the
anchor with no prompt) and shifted each scene's content down by one.
Fix in the one canonical splitter: a generic 'scene <N>' label (not a real
custom trigger in the transition DB) now cuts SILENTLY -> boundary only, dropped
from both expanded (generation) and raw (editing) text. Real user/DB triggers
keep their configured placement, so 'cut'/'qcut' etc. are unaffected.
Regression test asserts the markers split but never appear in any scene's
expanded/raw text, and that a real trigger still leads its scene.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Training-free concept sliders (arxiv 2511.00103) in noise-prediction space,
sourced from the learned taste (user's choice). New score_slider + strength(eta)
inputs on the Scene Chain sampler, additive/non-disruptive, reusing the same
learned liked direction + embed_guidance_source + refinement_key_input.
Mechanism: a per-scene model_function_wrapper synthesizes a +/- concept pair from
the base conditioning and the liked direction (no concept prompts), runs 3 forward
passes on quality-phase steps and combines eps_mod = eps_base + eta*(eps_+ - eps_-).
High-sigma steps stay single-pass = the paper's k-step base-only warmup, keeping the
2x cost off early steps. Video-only: +/- conds are audio-protected and the eps blend
is masked to the packed video stream. Composes with embed_guidance (its nudge cancels
in the +/- difference). Installed/torn down per scene in reverse order; threaded
through batch-training re-dispatch. Fully guarded -> falls back to base prediction.
Tests: tests/test_score_slider.py (5) — combine, warmup, eta=0 no-op, eta linearity.
Pre-existing 16 suite failures unchanged (stale-mock/splitter set); zero new.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Inversion-free adaptation of 'When to Lock Attention' (arxiv 2603.09657).
Track x0 prediction instability per step (relative step-change of the predicted
clean latent, video-masked = a cheap proxy for the paper's variance-of-x0
hallucination metric over a sliding window). High instability => the model is
still drifting => raise the shared kvlock_scale so BachVid holds the blessed
identity K/V harder; as x0 settles the multiplier relaxes toward base strength.
No DDIM inversion (LTXAV inverts poorly) — locks against the blessed K/V bank.
No-op unless a BachVid bank installed the shared scale list, so the two compose
cleanly. Wired into all three FunPack sampler loops after each model() call;
fully guarded, cannot break sampling. Tests cover capture/inject/scale + the
schedule's stable-vs-jumpy behavior and the no-bank no-op path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Training-free character carry across scenes/sessions (arxiv 2510.21696),
built on the existing LTXAV attn-hook infra. Capture raw K/V at IDENTITY_BLOCKS
during the mid-sigma window into a per-refinement-key temp bank; promote to a
blessed bank on liked ratings (EMA-merge, alongside bless_attn_weights). On
later runs with a blessed bank, lerp the running gen's K/V toward the blessed
identity at matched token positions (packed-/RoPE-safe analogue of the paper's
concat-attend). Wholesale inject at identity blocks (no spatial mask yet).
Exposes a shared kvlock_scale multiplier in transformer_options for the Phase 2
KV-Lock variance scheduler to drive injection strength per step (1.0 = off now).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route the conditioning variant from per-config outcome history, via the one
charter-safe surface: the INTERNAL value-function ascend+search (not user knobs,
so no WYSIWYG / hidden-settings violation). Sampler-owned visual ops stay on the
sampler (boundary law kept).
Directly serves the original complaint (Perfect not locking, Awful reproducing):
- _v2_path_conditioning_plan: matching-arm Beta posteriors -> liked-rate mean ->
"lock" (>=0.6: hold the liked conditioning, skip the stochastic search),
"explore" (<=0.4: widen the search to flee a disliked region), else "normal"
- value_function.search gains eps_scale to widen/tighten the search radius
- threaded conditioning_plan through ascend -> apply_scene_refinement_keys ->
finalize; computed once per run from path_outcomes + prompt concepts; logged
Only active once the per-key VF is ready, since it rides the existing ascend path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Upgrade the seed explore/exploit gate from a flat reuse chance to a proper
bandit over the path-outcome arms:
- _v2_path_arm_beta: Beta posterior over an arm's liked-rate (liked>=0.3 success,
<0.0 failure, neutral ignored; Laplace prior)
- _v2_path_explore_decision: Thompson-sample theta per concept-matching arm vs a
fresh Beta(1,1) explore arm; good arms exploit, dislike-dominated arms route to
a fresh seed ("do something else")
- wired into _v2_choose_successful_seed (falls back to the flat gate with no data)
- shared _v2_path_matching_arms helper; _V2_KEY_REWARD + liked/disliked consts
5 tests incl. a statistical one: a strongly-liked arm rarely explores, a hated
arm almost always does.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make path_outcomes ACT via the one Studio->sampler lever wired today: the seed
(FunPackStudio.run -> _v2_choose_successful_seed -> seed output -> sampler). No
new data paths invented.
The existing seed chooser only exploited liked seeds with zero avoidance — the
headline gap. Now:
- recorder tags each seed sample avoid = reward<0 and not skip_value_function
(Wrong-* excluded: the seed was fine, the words/identity were off)
- _v2_path_disliked_seeds: seeds that gave a low-quality result for matching concepts
- _v2_choose_successful_seed redraws a disliked fallback, drops liked-then-later-
disliked seeds from reuse, and reports how many it avoided
Broader config levers (guidance/lora) need new wiring and are deferred pending
graph-topology confirmation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Audited every learning sink in the V2 rating flow. Rating-learning and the
Absolute store already honored skip_learning, but two feedback sinks ran gated
only by feedback_prompt — so a 'Just ignore it' run with typed feedback still
wrote advisor_feedback_history + intent_expansion_memory.
Add _v2_rating_is_discard (forget rating or auto-discarded refusal) and gate
both feedback sinks on it, without touching the active continue/fresh_prompt
modes that legitimately keep feedback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reframe ratings from local steering into global routing: every rating —
not just successful ones — is recorded as a labeled outcome on the per-scene
config "arm" that produced it, so a later planner can route away from arms
that keep generating disliked results.
- _v2_path_signature: arm = family + concepts + guidance config, seed EXCLUDED
(seed is the random draw within an arm); md5 arm_id collapses identical configs
- _v2_update_path_outcomes: learns from EVERY rating, decaying counts so recent
behavior dominates; records seed-level samples for later seed routing
- hooked at the shared per-scene learning hub + single-run path; stored in
per-key state under "path_outcomes"
Pure instrumentation — no generation behavior change yet (Phase 2 = Studio planner).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The fake routes object installed into sys.modules as `server` only defined
get/post/delete; server.py also registers put (models) and patch (media), so
importing the real server.py during collection (e.g. test_server.py) raised
AttributeError and aborted the whole run. Add put/patch to the mock.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The canonical generation/preview/key splitter (split_scenes) only treated a
shortcut as a scene cut when the shortcut's TRIGGER word was a key in the
transition DB. But split markers are commonly keyed by the EXPANSION phrase: a
"cut" shortcut expands to "Scene cut.", and the split marker is "scene cut.".
So the lookup `custom_map.get(_norm(raw))` missed, the shortcut was treated as
plain content, and NO cut was made — split_scenes returned one scene for
"A cut B cut C" while the verbatim editor split returned three. That disagreement
collapsed the timeline's middle scenes (and would have generated one scene).
Fix: when the trigger word isn't in the DB, also match the shortcut's EXPANSION
(`custom_map.get(_norm(exp))`). The generic "scene <N>" label is still NOT matched
on expansions, so an incidental "scene 2" inside a content shortcut never cuts.
Verified against the user's on-disk cut/qcut defs: split_scenes now agrees with
the verbatim split (2 scenes for cut x2 / qcut x2 / mixed). Added regression tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deletes the FunPack Scene Builder node and the Studio Scene Builder feature
— the per-key scene database (scene_db / universal_memory / saved scenes),
the Studio "Scene" tab, scene-phrase authority over classification, and the
scene-database wildcard cleanup. It was redundant: helpful on paper but
effectively unused, undrivable from the Cutting Room, and a risk to shortcuts.
Removed:
- node registration (__init__.py, funpack.py) + web/funpack_scene_builder.js
- templates.py: scene-DB storage layer (load/save/normalize/empty_scene_db,
SCENE_DB_VERSION/SCENE_STATE_FIELD/SCENE_CATEGORY_SOURCES, scene_store_path),
all scene_* helpers, resolve_scene_database_wildcards, the FunPackSceneBuilder
class, and the /funpack/scenes* routes
- conditioning.py: _v2_scene_builder_db / _v2_scene_memory_item /
_v2_apply_scene_builder_authority / _v2_scene_builder_category_for_entry /
_v2_sync_scene_builder_memory / _v2_resolve_scene_builder_wildcards (+ wildcard
helpers + _v2_scene_key), the scene_db threading through classify/learning,
the scene_category_locked phrase fields + their training branch, the
state["scene_builder"] preservation, and the Studio scene_builder prompt mode
- web/funpack_studio.js: Scene tab, SB_MODES, scene_builder settings, renderScene,
fetch/saveScene, the Session Scene-Builder section, and the Refiner banner/notice
- tests/test_scene_builder.py (whole file) + scene-builder tests in
test_v2_category_alignment.py; the SceneBuilder example workflow + its doc;
README link; Scene Builder mentions in docs/FunPackStudio.md
Kept intact: shortcuts (apply_prompt_shortcuts / shortcut DB), transitions and
the canonical split_scenes splitter, and FunPackRefinementKeyLoader. Studio now
always uses the connected positive_prompt. No new test failures (the 16/97
remaining in tests/ are pre-existing env/sampler failures); movie_editor 140 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reworks the "Wrong appearance" rating from a (now-removed) prompt-repair
signal into a conditioning-level consistency anchor (option b). "Wrong
appearance" now means "character consistency wasn't preserved" and steers
the next generation back toward the established character.
Capture (_v2_update_appearance_anchor, wired into both the single-run
learning path and the shared per-scene suite _v2_learn_scene_into_state):
- good rating (like/nailed_it/loved_it/loved) → store the rated run's
conditioning as the per-key "blessed appearance" (overwrite = last good)
- wrong_appearance → store the rejected run's conditioning as the "drift"
global_state is already per-refinement-key, so both live with that key.
Apply (_v2_apply_conditioning_memory_impl, after the intent-family delta):
when the user rated the previous gen wrong_appearance, gently pull this gen
toward the blessed appearance (lerp 0.05) and repel from the drift (0.04),
reusing _v2_apply_conditioning_payload / _v2_repel_conditioning_payload. No
ramping; the existing 7.5% delta cap bounds the net effect. No-op when
nothing is stored or shapes mismatch.
Drops the dead wrong_categories field (zero consumers after Part A) and
keeps skip_value_function (the anchor is a conditioning nudge, not a reward).
Tests: 7 new unit tests (storage + overwrite + skip + pull/repel + idle +
non-trigger); updated test_wrong_appearance_rating_is_available for the new
semantics. Zero new failures vs HEAD (the 10/31 remaining are pre-existing
env failures from missing server/comfy modules).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Removes the now-unreferenced repair-only machinery left behind after
Stage 2 (1/3) neutralized the application:
- _v2_prompt_repair_text_allowed
- _v2_emphasized_prompt
- _v2_repair_prompt_for_missing_axes and the repair-candidate helpers
- perfect_repairs / rejected_repairs storage + perfect-repair eviction
note in the intent-family learn path
Shared helpers still used by intent-alignment (e.g. _v2_repair_intent_roots,
_v2_repair_candidate_matches_intent, _v2_prompt_contains_text) are kept, as
is the repair_feedback / active_repair_axes "what's missing" signal the
advisor reads.
Deletes the 11 category-alignment tests that exercised the removed methods
directly. No new test failures (the 10/31 remaining are pre-existing env
failures from missing server/comfy modules).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Neutralize the repair application in refine(): the intent-aligned prompt is
encoded as-is — perfect-repair phrase injection, missing-axis repair, and the
emphasis pass are gone. The advisor remains the only prompt-rewriting path, and
repair_feedback/active_repair_axes is KEPT (it's the 'what's missing' signal the
advisor still reads). Drop the prompt_repair param from refine(), INPUT_TYPES,
FunPackStudio, and both UIs (web + engine_settings).
The dead repair-only methods (_v2_repair_prompt_for_missing_axes, perfect-repair
phrases, etc.) come out in the next commit; shared helpers like
_v2_repair_intent_roots (used by intent alignment) stay.
Deleted the now-obsolete test_repair_candidates_saved_as_json_lists (tested the
removed injection). No new test failures vs dev.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dropping the generic scene-N pattern broke the Movie Editor: build_combined_prompt
(for_generation) INJECTS 'scene N' markers into the generation prompt to force a
split for any unit whose text lacks a leading transition trigger. Without that
recognition the second unit never split -> 'scene 2 exists only on the timeline,
not generated'.
split_scenes now matches the generic scene-N label again, but ONLY in its
literal-text scanner (never over shortcut expansions). So an editor-injected /
user-typed 'scene 2' cuts, while a 'scene 2' buried inside a shortcut expansion
does not — fixing both the user's phantom split AND the broken montage.
For the reported prompt the piece-walker now returns 2 scenes (was 4), matching
the old generation splitter's 2 -> the guard should pass and keys attribute while
the chain/guides keep working.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The verbatim/offset-projection splitter mangled real prompts: long shortcut
expansions desynced the raw<->expanded offset map (empty anchor, qcut+cut
lumped), the hard-coded generic 'scene N' pattern split on incidental text
('scene 2' inside an expansion), and re-scanning expansions re-triggered on
words like the 'cut' inside 'Scene cut.' — 4 phantom chunks instead of 2.
split_scenes now walks _expand_with_map's ordered pieces ONCE: a boundary is
created only by a real transition trigger (a shortcut whose trigger word is in
the transition DB, or a literal typed trigger); expansion CONTENT is never
re-scanned; the generic scene-N pattern is dropped. Each keyed shortcut adds
its key to the current scene; the leading run is the anchor (keys fold into
all). Returns raw + expanded text + keys + effect per scene.
For the reported prompt this yields exactly 2 scenes: [keyA,keyB] and [cs],
with 'scene 2'/'Scene cut.' no longer splitting. Tests updated to use real
triggers + a regression test for the expansion-trap. 21 + 140 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
resolve_scene_refinement_keys now derives purely from split_scenes — drops
the scene_count parameter AND both divergence fallbacks (the band-aids).
Generation reads per-scene keys from the canonical raw split and GUARDS that
its text-split count (prompt_to_encode) matches; on mismatch it screams and
falls back to default-key attribution rather than land a key on the wrong
scene. bridge.scene_refinement_keys + the /preview callers drop scene_count.
Reset (_v2_reset_prompt_keys) now wipes refinement_key_pool_for(prompt), so
the pool is one canonical answer shared with the reset-confirmation modal.
That removes the last user of prompt_scene_shortcut_keys, so its entire
internal split is deleted (and its 3 tests, covered by split_scenes tests).
Tests: 20 refinement-key + 140 movie_editor pass. Generation path needs an
in-app smoke test (no model here) — the guard makes a desync loud, not silent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One rich scene model {anchor, scenes:[{index,raw,keys,effect}]} that every
consumer will derive from — editing (raw), preview (raw->expanded),
attribution (keys), reset (pool = union of keys), generation. Built on the
existing shortcut-aware verbatim splitter + refinement_keys_in_text, so it
can't disagree with itself. Nothing repointed yet; consumers still on their
own splitters until the staged repointing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Attribution was re-splitting the raw prompt independently and comparing
scene counts, then guessing on mismatch (all-keys union -> cross-training,
or empty -> keys not detected, incl. single-scene-with-keys). Both the
80MB cross-training and Darki's "custom uploaded keys not detected" trace
to that.
resolve_scene_refinement_keys now reads each scene directly: split with
the shortcut-aware split_timeline_verbatim (same scenes generation uses)
and collect, via new refinement_keys_in_text(), the keys whose shortcuts
fired in each scene (anchor keys apply to all). No scene-count compare, no
union/empty fallback except a genuine multi-scene misalignment. Single
scene with a custom key is detected; each scene gets exactly its keys.
Reverted the original-text scan added to prompt_scene_shortcut_keys (it
desynced that splitter from parse_timeline/generation and caused the
detection regression); it's now used only for session-reset all_keys.
Training law: a scene owned by custom key(s) trains ONLY them; the project
default key is left untouched and learns solely from scenes with no custom
key. Tests added for all three behaviors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wrong-* ratings (appearance/details/action/combos) are prompt-REPAIR
signals — "good gen, wrong words/identity" — not quality rewards. They
were feeding their 0.0/low reward into both value-function training and
the keyless Absolute taste store, stamping visually-good conditioning as
a low-value valley. The VF ascent (now always the default key, applied to
every scene) then pushed conditioning away from good regions into drift /
wrong-character / distorted gens; Absolute stored good gens as global
bad-taste.
Wrong-* now carry skip_value_function: still drive prompt repair + the
relative per-key direction/category memory, but train no reward asset.
Quality ratings (Missing-*/Perfect/Nailed/Awful) unchanged. Gated all 3
relative VF call sites + _v2_learn_absolute. Preventative only — keys
poisoned before this fix still need reset/retrain.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per-scene key attribution diverged from the generation split whenever the
advisor rewrote prompt_to_encode away from the raw prompt, so
resolve_scene_refinement_keys hit its all-keys UNION fallback and assigned
every key to every scene. That union got persisted into the run and read
back at rating time, training every key on every clip (value_fn bloat).
- prompt_scene_shortcut_keys now scans both expanded and original text
(mirrors split_timeline_verbatim), so identical structure -> matching
scene count -> precise attribution far more often.
- Divergence fallback is now default-key only (empty sets), not the
all-keys union; a diverged run under-trains instead of cross-training.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A replacement is a prose phrase and may contain commas (e.g. "The video
rapidly cuts, showing the next view."). Both the Movie Editor's splitLines
and the backend _shortcut_replacements string fallback split on commas, so
one phrase became several variants: the editor showed it across lines,
expansion picked one half at random (never matching the comma-form transition
trigger), and a comma edit reverted on save because it re-split every time.
Replacements are now one-per-line only (commas preserved); triggers still
accept comma-separation per their UI. Existing torn replacements self-heal on
the next edit + save.
- mediabrowser.js: separate splitTriggers (\n or ,) and splitReplacements (\n).
- templates.py: _shortcut_replacements string fallback splits on \n only.
- tests: comma phrase stays one replacement (list + string + normalize paths).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
