Block steering: run-averaged block activity is architecture-determined
(varies in the 4th decimal run-to-run), so no scoring scheme - pole EMAs
or reward correlation - could extract a rating signal from it. The idea
needs within-run per-step measurement to have a chance; that is a
research project, not a feature. Removed: module, sampler wiring, the
two trailing widgets (positional reference mapping unaffected), the
Refiner pairing hook, editor knobs, tests. Stale block_snapshot/
block_profile sidecars are inert and covered by the existing
delete_refinement_key sweep.
FunPackNVFP4ModelLoader: the FP4 GEMM path was real (3.35x at 16k
tokens) but end-to-end the user's runs are dominated by everything the
loader cannot touch (attention, prep, encode - the sampler node is
~23s of a ~60s run) while dynamic input quantization added a fixed
per-layer-call toll. Net observed: overhead. Users are better served
by dedicated NVFP4 loaders. Removed: module, registration, tests.
Kept (independently useful): the chain sampler's phase timing report +
quantized-layer census, the nested-AV x0 snapshot fix, and the sm_120
masked-attention work.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Justified by live data: sampler phase timing shows sampling at 89% of
the chain node, all 792 quantized layers on the fast path, and the probe
puts dynamic input quantization at ~0.30 ms/layer-call vs ~0.11 ms with
a provided scale (amax dominates) - a top-line per-call cost.
New input_scales widget on the loader:
- "auto-calibrate", first load without a sidecar: forward-pre-hooks
record each quantized layer's running input amax (one generation is
enough; sidecar under refinements/nvfp4_scales/<checkpoint>.json,
per-machine and rebuilt in one run whenever absent).
- every later load: injects input_scale = amax * 1.25 / (448*6) into
the state dict next to each quantized weight - core's loader
registers it natively and skips the per-call amax from then on.
Margin leaves clip headroom (NVFP4 clips above the scale).
Tests: suffix-matched injection across checkpoint prefixes, value math,
core Linear registering input_scale and forwarding, sidecar read.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live report: "flat activity across runs" with a working profile. Root
cause is real, not a threshold tuning issue: the fingerprint is each
block's MEAN residual activity over ~1200 steps, which is practically a
property of the architecture - run-to-run variation sits in the 4th
decimal, so raw reward-covariance is numerically dead. Lowering the
epsilon would have "fixed" it by steering on noise.
Now: per-block Pearson CORRELATION between reward and activity (scale-
free - tiny variations count when they consistently track ratings),
fingerprints stored mean-normalized so run-global scale (step counts,
schedulers) can't masquerade as block signal, and a confidence ramp
(min(1,(runs-2)/6)) so a 3-run profile steers gently instead of not at
all, reaching full strength ~8 runs. MIN_RATED_RUNS 2->3 (correlation
over 2 runs is degenerate). Status lines now carry the evidence:
"ready (N runs, peak |corr| X, confidence Y)". Older profiles are
discarded on first pairing (fingerprint semantics changed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'Loaded fine' and 'runs fast' are different claims. Every NVFP4 load now
prints a diagnostics block that settles which one you have:
- comfy_kitchen backend census (the big one: torch built for CUDA < 13
disables ck's CUDA backend, so quantized matmuls silently fall through
to the pure-PyTorch eager backend - correct output, ZERO speedup, with
an explicit warning line when detected)
- torch/cuda build + device capability
- loaded-model layer census: nvfp4 layers vs layers flagged
full-precision emulation (pre-Blackwell GPUs)
- a timed 4096x4096 GEMM probe (incl. dynamic input quantization):
bf16 vs nvfp4 wall time with a FAST PATH ACTIVE / NO SPEEDUP verdict
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three live-run findings from the rental:
1. Block steering said "profile not ready (needs 2+ rated runs)" with 16
paired runs. The v1 liked/disliked-EMA attribution had no contrast when
ratings cluster on one pole (pole-vs-all-runs-mean cancels to zero) and
dropped rewards in (0, 0.3) entirely. Attribution is now the per-block
COVARIANCE between reward and activity over a rolling 32-run history:
every admissible reward contributes with its actual value, contrast
comes from reward variance, and the sampler log now states the real
blocker ("all rated alike - rate both good AND bad runs") instead of a
misleading run count. v1 profiles are discarded on first pairing.
2. output_guidance stayed "not ready (needs 10+)" forever on LTX-AV: the
end-of-run x0 snapshot was gated on output samples being a plain
torch.Tensor, but AV chain output is a NESTED tensor (video+audio) - so
the output value function never received a single sample. The snapshot
now extracts the video stream from nested outputs.
3. FunPackNVFP4ModelLoader hit a size-mismatch wall (9-vs-6 ada params,
connector dims): it loaded the state dict without the safetensors
METADATA that carries the LTX-AV architecture config, so detection
built the wrong model variant. Metadata now flows through, mirroring
comfy.sd.load_diffusion_model.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RLHF at the model level: which REGIONS of the transformer were doing the
work when a generation came out good or bad, and steer accordingly.
1. Measure: tagged forward hooks record each block's relative video-residual
delta per model call (near-zero cost; covered by the existing leak-strip).
2. Attribute: the end-of-run fingerprint is staged (block_snapshot.pt, same
stage-then-pair convention as x0_snapshot) and paired with the rating
reward on the next Refiner cycle into liked/disliked/all-runs EMA
profiles; when output_guidance's value function is trained, an innermost
model-function wrapper additionally credits blocks per step by
(value delta) x (centered activity) - converges in a handful of runs.
3. Steer: vx' = vx_in + g_b*(vx_out - vx_in) on the video stream only,
audio untouched; scores zero-mean/unit-max so emphasis is redistributed
rather than inflated, gains hard-capped at +/-10%. Exact no-op until the
profile has 2+ paired runs; blocks at gain 1.0 get no hook at all.
Chain Sampler toggle + strength (append-only widgets), exposed same-change
in the editor's Chain Sampler Guidance section via the sampler_inputs
bridge. New sidecars ride the defensive <key>.* sweep in
delete_refinement_key, so atomic key deletion covers them.
Tests: toy AV-block model driving the real hooks (recorder ranking, steer
video-only scaling + native path for unit gains, credit wrapper, snapshot/
profile/scores/gains roundtrip, mild-positive band, depth-mismatch reset,
leak-strip tagging). Full suite failure set identical to the pre-existing
baseline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FunPackNVFP4ModelLoader quantizes an existing bf16/fp16 checkpoint to NVFP4
at load time as a pure state-dict transform: selected 2D Linears are
re-serialized in ComfyUI core's quantized-checkpoint format (packed fp4
weights + fp8-e4m3 block scales + fp32 tensor scale + per-layer comfy_quant
metadata) and handed to the stock loading path. Core's MixedPrecisionOps
then does everything: real FP4 tensor-core GEMMs on sm_100+ (sm_120 incl.),
dynamic activation scales (no calibration pass), per-layer LoRA fallback,
and dequant emulation on older GPUs. No custom forward, no monkeypatching.
Zone-aware precision map for the joint AV transformer: default scope
quantizes only video-branch Linears (attn1/attn2/ff) of the middle blocks;
the audio branch, the audio<->video cross-modal bridges, the first/last N
blocks, and all non-matmul weights stay full precision. Scopes widen to
+cross-modal, all blocks, and all 2D layers for experimentation.
Tests drive the REAL comfy.quant_ops/comfy.ops via comfy_kitchen's eager
backend on CPU: scope selection, serialized-format contract, prefix
handling, refusal of pre-quantized checkpoints, and a load+forward
roundtrip through MixedPrecisionOps on both the NVFP4 and emulation paths.
The Movie Editor Models modal picks the node up automatically as a
Unet/Diffusion Model candidate (object_info-driven).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On compute capability > (9, 0) GPUs (Blackwell sm_120, e.g. RTX PRO 6000),
xformers' memory_efficient_attention has no kernel that accepts a tensor
attn_bias — fa3F/cutlassF require cap <= (9,0), fa2F/fa3F reject a tensor
bias. So unmasked attention works (i2v ANCHOR scenes generate) but MASKED
attention crashes: scenes generated from GUIDES hit the LTX guide path
(comfy/ldm/lightricks/model.py _attention_with_guide_mask) which passes a
tensor mask → "No operator found for memory_efficient_attention_forward".
This is the ComfyUI-wide issue normally worked around with
--disable-xformers --use-pytorch-cross-attention; when that flag isn't in
effect on an instance, the guide path still reaches xformers.
Fix: _funpack_install_mask_safe_attention(model), called at the top of the
chain sampler's sample(). When the active backend is xformers AND the device
capability is > (9,0), install ComfyUI's per-call optimized_attention_override
(the wrap_attn hook) so MASKED attention falls back to attention_pytorch
(SDPA handles masks on every GPU) while UNMASKED attention stays on the fast
xformers path. Threaded via transformer_options so it reaches every scene's
forward. No global monkeypatch; no recursion (wrap_attn stamps
_inside_attn_wrapper before calling the override). No-op on supported GPUs,
non-xformers backends, or when another override is already installed;
idempotent.
Tests: tests/test_mask_safe_attention.py — 9 cases covering override routing
(masked→pytorch via kwarg or positional, unmasked→original) and install
gating (xformers+Blackwell installs; supported cap / non-xformers / no-CUDA
skip; existing override respected; idempotent). Run with the ComfyUI venv.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Generalizes the i2v-anchor blur (_alg_blur_video_frame0 -> _alg_blur_frames)
to optionally also blur newly-appended guide-attention frames each scene
(mid_scene_guide / carry_i2v_guides-as-guide / configured guides / JoyAI
memory), reusing the existing guide_tail bookkeeping rather than a mask-value
heuristic (frame_overlap continuation also uses mask=0 and must stay
untouched). New Scene Chain Sampler toggle "Blur i2v guides and JoyAI memory"
mutates the shared KSAMPLER's extra_options per-chunk since guide_tail varies
per scene; always resets so nothing leaks across chunks/samplers. Surfaced in
Engine Settings via the existing Experimental section, no backend/builder
changes needed beyond the new INPUT_TYPES field.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adaptive Low-Pass Guidance (arXiv:2506.08456): blurs the i2v anchor frame
while sigma is in the near-pure-noise region, then swaps back to the sharp
anchor, to stop the model shortcutting to a near-static video that just
matches the reference image. Opt-in, off by default, no-op without an i2v
anchor. Swaps model.latent_image per step (the same attribute ComfyUI's
inpaint compositing already re-pastes every step) instead of monkeypatching
comfy internals.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Deleting a refinement key was non-atomic. A key's state spans <key>.json plus
sidecars (<key>.value_fn.pt, <key>.sampler_ctx.json, attn_maps/blessed maps +
K/V banks + creativity latent + velocity store), and there was no delete route —
users removed the visible <key>.json by hand, orphaning the rest. The orphans are
still applied: _v2_ascend_conditioning loads <key>.value_fn.pt purely by name
(not gated on the json), and blessed K/V inject at a 0.04 floor even at reward 0.
So a "cleared" key kept steering future generations, surviving both a ComfyUI
restart and the manual delete — the persistent half of the long-run quality drift
(the per-session half was the hook leak in the previous commit).
- templates.py: delete_refinement_key() sweeps the canonical json + every <key>.*
sidecar + clear_refinement_data (attn_maps banks, velocity disk+memory).
absolute_store_info()/clear_absolute_store() for the keyless Absolute taste store
(__funpack_absolute_global__): it learns from every rated gen across all prompts,
is invisible in the key list, and was otherwise only wiped by Session Reset.
Routes: POST refinement_keys/delete, GET/POST refinement_keys/absolute|clear_absolute.
- Refinement Key Loader node: "Delete Key" + "Clear Global Taste" buttons.
- Movie Editor: api deleteRefinementKey/absoluteStoreInfo/clearAbsoluteStore +
FunPack menu "Delete refinement key…" / "Clear global taste store…".
- tests: atomic-delete sweep + empty-key guard (real store, unique key, cleanup).
Pre-existing 16 failures unchanged (stale mocks); +2 new passing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend the JoyAI cross-shot memory bank from video-only to paired audio-video,
the part that distinguishes JoyAI-Echo from a generic video-consistency trick.
- _JoyAIMemoryBank now stores (video, audio) pairs under the same pin/roll policy;
frames()/audio() stay 1:1. Audio half is None until joyai_audio_memory is on, so
the phase-1 video path is byte-identical.
- _append_joyai_audio_memory pins each banked prior-shot audio latent into the
chunk's audio stream as a fully-protected (mask=0) tail, cropped back off after
sampling (_crop_audio_tail). The audio analogue of the video guide bank; rides
native audio self-attention since LTX has no audio guide-attention API.
- _harvest_joyai_audio harvests the paired audio frame, position-matched to video.
- v2a_grad_scale scales the model's trained video->audio cross-attention via
reversible forward hooks on each AV block's video_to_audio_attn submodule
(JoyAI uses 2.0). 1.0 installs no hook -> native behavior, zero overhead.
- joyai_audio_memory + v2a_grad_scale wired through sample(), batch training,
refinement context, run_mechanisms, INPUT_TYPES (append-only), and editor knobs.
Deliberately breaks the audio pass-through invariant, so both are opt-in and
off/native by default. 3 new bank tests; zero regressions (same 12 pre-existing
stale-mock failures before/after).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Diagnostic conclusion from the JoyAI-Echo engine: conditioning is fed to the sampler
unchanged (gemma+connector, which Studio already produces), so the ONLY missing sampler
feature is generate_BidirectionalMemory's cross-shot paired AV memory bank. This lands the
VIDEO half: a managed set of clean prior-shot frames injected into each scene so identity
carries across the whole chain — JoyAI's story-level consistency.
Generalizes the single-anchor mid_scene_guide to N frames, reusing the same LTX
guide-attention primitive (_append_guide_latent). _JoyAIMemoryBank holds the policy
(num_fix opening shots pinned as a global anchor + rolling most-recent window capped at
max_size, matching configs/inference.yaml: max_size 7, num_fix_frames 3, center select).
Harvest one clean frame per finished scene; inject the bank before each continuation scene;
supersedes mid_scene_guide when on. Honors the 0.25 audio-safe strength floor.
Off by default, so the audio protect/pass-through invariant is unchanged. Paired audio
memory + v2a_grad_scale coupling is the next phase (needs the ltx_core v2a term).
Inputs: joyai_memory, joyai_memory_size, joyai_fix_frames, joyai_frame_select,
joyai_memory_strength. Threaded through batch-training; logged in run_mechanisms + the
refinement sampler-context. New test_joyai_memory.py covers the bank policy (6 tests).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The score_slider synthesized a symmetric +/-liked pair, so its axis was
"liked vs anti-liked mirror" — meaningless as an away-from-bad signal. When
3+ disliked/awful gens have populated the existing bad_dir slot in the same
taste store, build the slider's minus pole from that real disliked direction
instead. The (eps_+ - eps_-) axis becomes good-vs-bad, actively steering the
noise prediction away from the conditioning region that produced rated-bad
gens. Falls back to the symmetric mirror when no bad_dir exists.
Reuses existing plumbing: bad_dir is already accumulated identically to
liked_dir (running-averaged unit-norm delta from session mean). Generalized
the direction loader to _load_taste_direction(slot); load + pass bad_dir at
the call site; log/run-mechanism note which pole is active.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-scene prompt preview mirrors what Studio encodes, and the postfix is
appended to every scene at generation — so it belongs there. Surface it once
after the scenes as a 'postfix' badge, symmetric to the 'anchor' badge shown
once above them.
- bridge.expand_prompt_fragment: expands a standalone fragment (anchor/postfix)
the same way generation does, so the previewed text matches exactly.
- server preview: result["postfix"] = expanded effective_postfix(p).
- inspector: postfix badge line; _promptPreviewKey includes postfix so editing
it refreshes the preview live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A shortcut in the postfix is expanded at the generation hit (same split_scenes
path as the anchor) and its refinement key folds into every scene. The raw
postfix is preserved on the project for round-trip; expansion is generation-time.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause the user pushed on: the editor flattened all scenes into ONE prompt
string and Studio re-derived boundaries by parsing text, so a carry scene with no
transition needed an injected 'scene N' delimiter. The editor already KNOWS every
boundary (each generative unit is a scene) — it never needed to smuggle markers
into the prompt.
- timeline.build_combined_prompt: no longer injects anything (verbatim anchor +
scene texts). New build_generation_scene_segments(project) lists the boundaries
the editor already has (each unit's raw text, prefixed with its REAL transition
so effects survive; carry = plain text).
- conditioning.split_scenes_from_segments: canonical split from an explicit scene
list (reuses split_scenes per item for expansion/keys/effect, folds the anchor),
same output shape as split_scenes. refine_v2 gains scene_segments and uses it
when present (count-preserving: a text-less i2v scene keeps its slot, no anchor
shift); falls back to text-splitting only for raw node-pack use.
- server/builder: editor sends scene_segments -> studio_settings.refiner.scenes.
Net: the generation prompt Studio encodes is now exactly what the user typed —
no 'scene N' anywhere, in the prompt or the conditioning. Tests updated to assert
the verbatim prompt + structural segments; new tests for split_scenes_from_segments
(boundaries, empty-scene preservation, anchor/key fold). Fallback path unchanged
(stash-verified zero regression). NOTE: montage generation needs GPU verification.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Multi-scene bug: the Movie Editor injects a generic 'scene N' separator before
any scene that lacks a leading transition trigger (build_combined_prompt), purely
to force a boundary. split_scenes was splitting on those markers with placement
'start', which KEEPS the marker word in the new scene's text -> 'scene 1',
'scene 2', ... got encoded into every scene's conditioning. With a text-less
leading i2v scene this also produced a label-only first scene (renders as the
anchor with no prompt) and shifted each scene's content down by one.
Fix in the one canonical splitter: a generic 'scene <N>' label (not a real
custom trigger in the transition DB) now cuts SILENTLY -> boundary only, dropped
from both expanded (generation) and raw (editing) text. Real user/DB triggers
keep their configured placement, so 'cut'/'qcut' etc. are unaffected.
Regression test asserts the markers split but never appear in any scene's
expanded/raw text, and that a real trigger still leads its scene.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Training-free concept sliders (arxiv 2511.00103) in noise-prediction space,
sourced from the learned taste (user's choice). New score_slider + strength(eta)
inputs on the Scene Chain sampler, additive/non-disruptive, reusing the same
learned liked direction + embed_guidance_source + refinement_key_input.
Mechanism: a per-scene model_function_wrapper synthesizes a +/- concept pair from
the base conditioning and the liked direction (no concept prompts), runs 3 forward
passes on quality-phase steps and combines eps_mod = eps_base + eta*(eps_+ - eps_-).
High-sigma steps stay single-pass = the paper's k-step base-only warmup, keeping the
2x cost off early steps. Video-only: +/- conds are audio-protected and the eps blend
is masked to the packed video stream. Composes with embed_guidance (its nudge cancels
in the +/- difference). Installed/torn down per scene in reverse order; threaded
through batch-training re-dispatch. Fully guarded -> falls back to base prediction.
Tests: tests/test_score_slider.py (5) — combine, warmup, eta=0 no-op, eta linearity.
Pre-existing 16 suite failures unchanged (stale-mock/splitter set); zero new.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Inversion-free adaptation of 'When to Lock Attention' (arxiv 2603.09657).
Track x0 prediction instability per step (relative step-change of the predicted
clean latent, video-masked = a cheap proxy for the paper's variance-of-x0
hallucination metric over a sliding window). High instability => the model is
still drifting => raise the shared kvlock_scale so BachVid holds the blessed
identity K/V harder; as x0 settles the multiplier relaxes toward base strength.
No DDIM inversion (LTXAV inverts poorly) — locks against the blessed K/V bank.
No-op unless a BachVid bank installed the shared scale list, so the two compose
cleanly. Wired into all three FunPack sampler loops after each model() call;
fully guarded, cannot break sampling. Tests cover capture/inject/scale + the
schedule's stable-vs-jumpy behavior and the no-bank no-op path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Training-free character carry across scenes/sessions (arxiv 2510.21696),
built on the existing LTXAV attn-hook infra. Capture raw K/V at IDENTITY_BLOCKS
during the mid-sigma window into a per-refinement-key temp bank; promote to a
blessed bank on liked ratings (EMA-merge, alongside bless_attn_weights). On
later runs with a blessed bank, lerp the running gen's K/V toward the blessed
identity at matched token positions (packed-/RoPE-safe analogue of the paper's
concat-attend). Wholesale inject at identity blocks (no spatial mask yet).
Exposes a shared kvlock_scale multiplier in transformer_options for the Phase 2
KV-Lock variance scheduler to drive injection strength per step (1.0 = off now).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route the conditioning variant from per-config outcome history, via the one
charter-safe surface: the INTERNAL value-function ascend+search (not user knobs,
so no WYSIWYG / hidden-settings violation). Sampler-owned visual ops stay on the
sampler (boundary law kept).
Directly serves the original complaint (Perfect not locking, Awful reproducing):
- _v2_path_conditioning_plan: matching-arm Beta posteriors -> liked-rate mean ->
"lock" (>=0.6: hold the liked conditioning, skip the stochastic search),
"explore" (<=0.4: widen the search to flee a disliked region), else "normal"
- value_function.search gains eps_scale to widen/tighten the search radius
- threaded conditioning_plan through ascend -> apply_scene_refinement_keys ->
finalize; computed once per run from path_outcomes + prompt concepts; logged
Only active once the per-key VF is ready, since it rides the existing ascend path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Upgrade the seed explore/exploit gate from a flat reuse chance to a proper
bandit over the path-outcome arms:
- _v2_path_arm_beta: Beta posterior over an arm's liked-rate (liked>=0.3 success,
<0.0 failure, neutral ignored; Laplace prior)
- _v2_path_explore_decision: Thompson-sample theta per concept-matching arm vs a
fresh Beta(1,1) explore arm; good arms exploit, dislike-dominated arms route to
a fresh seed ("do something else")
- wired into _v2_choose_successful_seed (falls back to the flat gate with no data)
- shared _v2_path_matching_arms helper; _V2_KEY_REWARD + liked/disliked consts
5 tests incl. a statistical one: a strongly-liked arm rarely explores, a hated
arm almost always does.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make path_outcomes ACT via the one Studio->sampler lever wired today: the seed
(FunPackStudio.run -> _v2_choose_successful_seed -> seed output -> sampler). No
new data paths invented.
The existing seed chooser only exploited liked seeds with zero avoidance — the
headline gap. Now:
- recorder tags each seed sample avoid = reward<0 and not skip_value_function
(Wrong-* excluded: the seed was fine, the words/identity were off)
- _v2_path_disliked_seeds: seeds that gave a low-quality result for matching concepts
- _v2_choose_successful_seed redraws a disliked fallback, drops liked-then-later-
disliked seeds from reuse, and reports how many it avoided
Broader config levers (guidance/lora) need new wiring and are deferred pending
graph-topology confirmation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Audited every learning sink in the V2 rating flow. Rating-learning and the
Absolute store already honored skip_learning, but two feedback sinks ran gated
only by feedback_prompt — so a 'Just ignore it' run with typed feedback still
wrote advisor_feedback_history + intent_expansion_memory.
Add _v2_rating_is_discard (forget rating or auto-discarded refusal) and gate
both feedback sinks on it, without touching the active continue/fresh_prompt
modes that legitimately keep feedback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reframe ratings from local steering into global routing: every rating —
not just successful ones — is recorded as a labeled outcome on the per-scene
config "arm" that produced it, so a later planner can route away from arms
that keep generating disliked results.
- _v2_path_signature: arm = family + concepts + guidance config, seed EXCLUDED
(seed is the random draw within an arm); md5 arm_id collapses identical configs
- _v2_update_path_outcomes: learns from EVERY rating, decaying counts so recent
behavior dominates; records seed-level samples for later seed routing
- hooked at the shared per-scene learning hub + single-run path; stored in
per-key state under "path_outcomes"
Pure instrumentation — no generation behavior change yet (Phase 2 = Studio planner).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The fake routes object installed into sys.modules as `server` only defined
get/post/delete; server.py also registers put (models) and patch (media), so
importing the real server.py during collection (e.g. test_server.py) raised
AttributeError and aborted the whole run. Add put/patch to the mock.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The canonical generation/preview/key splitter (split_scenes) only treated a
shortcut as a scene cut when the shortcut's TRIGGER word was a key in the
transition DB. But split markers are commonly keyed by the EXPANSION phrase: a
"cut" shortcut expands to "Scene cut.", and the split marker is "scene cut.".
So the lookup `custom_map.get(_norm(raw))` missed, the shortcut was treated as
plain content, and NO cut was made — split_scenes returned one scene for
"A cut B cut C" while the verbatim editor split returned three. That disagreement
collapsed the timeline's middle scenes (and would have generated one scene).
Fix: when the trigger word isn't in the DB, also match the shortcut's EXPANSION
(`custom_map.get(_norm(exp))`). The generic "scene <N>" label is still NOT matched
on expansions, so an incidental "scene 2" inside a content shortcut never cuts.
Verified against the user's on-disk cut/qcut defs: split_scenes now agrees with
the verbatim split (2 scenes for cut x2 / qcut x2 / mixed). Added regression tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deletes the FunPack Scene Builder node and the Studio Scene Builder feature
— the per-key scene database (scene_db / universal_memory / saved scenes),
the Studio "Scene" tab, scene-phrase authority over classification, and the
scene-database wildcard cleanup. It was redundant: helpful on paper but
effectively unused, undrivable from the Cutting Room, and a risk to shortcuts.
Removed:
- node registration (__init__.py, funpack.py) + web/funpack_scene_builder.js
- templates.py: scene-DB storage layer (load/save/normalize/empty_scene_db,
SCENE_DB_VERSION/SCENE_STATE_FIELD/SCENE_CATEGORY_SOURCES, scene_store_path),
all scene_* helpers, resolve_scene_database_wildcards, the FunPackSceneBuilder
class, and the /funpack/scenes* routes
- conditioning.py: _v2_scene_builder_db / _v2_scene_memory_item /
_v2_apply_scene_builder_authority / _v2_scene_builder_category_for_entry /
_v2_sync_scene_builder_memory / _v2_resolve_scene_builder_wildcards (+ wildcard
helpers + _v2_scene_key), the scene_db threading through classify/learning,
the scene_category_locked phrase fields + their training branch, the
state["scene_builder"] preservation, and the Studio scene_builder prompt mode
- web/funpack_studio.js: Scene tab, SB_MODES, scene_builder settings, renderScene,
fetch/saveScene, the Session Scene-Builder section, and the Refiner banner/notice
- tests/test_scene_builder.py (whole file) + scene-builder tests in
test_v2_category_alignment.py; the SceneBuilder example workflow + its doc;
README link; Scene Builder mentions in docs/FunPackStudio.md
Kept intact: shortcuts (apply_prompt_shortcuts / shortcut DB), transitions and
the canonical split_scenes splitter, and FunPackRefinementKeyLoader. Studio now
always uses the connected positive_prompt. No new test failures (the 16/97
remaining in tests/ are pre-existing env/sampler failures); movie_editor 140 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reworks the "Wrong appearance" rating from a (now-removed) prompt-repair
signal into a conditioning-level consistency anchor (option b). "Wrong
appearance" now means "character consistency wasn't preserved" and steers
the next generation back toward the established character.
Capture (_v2_update_appearance_anchor, wired into both the single-run
learning path and the shared per-scene suite _v2_learn_scene_into_state):
- good rating (like/nailed_it/loved_it/loved) → store the rated run's
conditioning as the per-key "blessed appearance" (overwrite = last good)
- wrong_appearance → store the rejected run's conditioning as the "drift"
global_state is already per-refinement-key, so both live with that key.
Apply (_v2_apply_conditioning_memory_impl, after the intent-family delta):
when the user rated the previous gen wrong_appearance, gently pull this gen
toward the blessed appearance (lerp 0.05) and repel from the drift (0.04),
reusing _v2_apply_conditioning_payload / _v2_repel_conditioning_payload. No
ramping; the existing 7.5% delta cap bounds the net effect. No-op when
nothing is stored or shapes mismatch.
Drops the dead wrong_categories field (zero consumers after Part A) and
keeps skip_value_function (the anchor is a conditioning nudge, not a reward).
Tests: 7 new unit tests (storage + overwrite + skip + pull/repel + idle +
non-trigger); updated test_wrong_appearance_rating_is_available for the new
semantics. Zero new failures vs HEAD (the 10/31 remaining are pre-existing
env failures from missing server/comfy modules).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Removes the now-unreferenced repair-only machinery left behind after
Stage 2 (1/3) neutralized the application:
- _v2_prompt_repair_text_allowed
- _v2_emphasized_prompt
- _v2_repair_prompt_for_missing_axes and the repair-candidate helpers
- perfect_repairs / rejected_repairs storage + perfect-repair eviction
note in the intent-family learn path
Shared helpers still used by intent-alignment (e.g. _v2_repair_intent_roots,
_v2_repair_candidate_matches_intent, _v2_prompt_contains_text) are kept, as
is the repair_feedback / active_repair_axes "what's missing" signal the
advisor reads.
Deletes the 11 category-alignment tests that exercised the removed methods
directly. No new test failures (the 10/31 remaining are pre-existing env
failures from missing server/comfy modules).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Neutralize the repair application in refine(): the intent-aligned prompt is
encoded as-is — perfect-repair phrase injection, missing-axis repair, and the
emphasis pass are gone. The advisor remains the only prompt-rewriting path, and
repair_feedback/active_repair_axes is KEPT (it's the 'what's missing' signal the
advisor still reads). Drop the prompt_repair param from refine(), INPUT_TYPES,
FunPackStudio, and both UIs (web + engine_settings).
The dead repair-only methods (_v2_repair_prompt_for_missing_axes, perfect-repair
phrases, etc.) come out in the next commit; shared helpers like
_v2_repair_intent_roots (used by intent alignment) stay.
Deleted the now-obsolete test_repair_candidates_saved_as_json_lists (tested the
removed injection). No new test failures vs dev.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Dropping the generic scene-N pattern broke the Movie Editor: build_combined_prompt
(for_generation) INJECTS 'scene N' markers into the generation prompt to force a
split for any unit whose text lacks a leading transition trigger. Without that
recognition the second unit never split -> 'scene 2 exists only on the timeline,
not generated'.
split_scenes now matches the generic scene-N label again, but ONLY in its
literal-text scanner (never over shortcut expansions). So an editor-injected /
user-typed 'scene 2' cuts, while a 'scene 2' buried inside a shortcut expansion
does not — fixing both the user's phantom split AND the broken montage.
For the reported prompt the piece-walker now returns 2 scenes (was 4), matching
the old generation splitter's 2 -> the guard should pass and keys attribute while
the chain/guides keep working.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The verbatim/offset-projection splitter mangled real prompts: long shortcut
expansions desynced the raw<->expanded offset map (empty anchor, qcut+cut
lumped), the hard-coded generic 'scene N' pattern split on incidental text
('scene 2' inside an expansion), and re-scanning expansions re-triggered on
words like the 'cut' inside 'Scene cut.' — 4 phantom chunks instead of 2.
split_scenes now walks _expand_with_map's ordered pieces ONCE: a boundary is
created only by a real transition trigger (a shortcut whose trigger word is in
the transition DB, or a literal typed trigger); expansion CONTENT is never
re-scanned; the generic scene-N pattern is dropped. Each keyed shortcut adds
its key to the current scene; the leading run is the anchor (keys fold into
all). Returns raw + expanded text + keys + effect per scene.
For the reported prompt this yields exactly 2 scenes: [keyA,keyB] and [cs],
with 'scene 2'/'Scene cut.' no longer splitting. Tests updated to use real
triggers + a regression test for the expansion-trap. 21 + 140 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
resolve_scene_refinement_keys now derives purely from split_scenes — drops
the scene_count parameter AND both divergence fallbacks (the band-aids).
Generation reads per-scene keys from the canonical raw split and GUARDS that
its text-split count (prompt_to_encode) matches; on mismatch it screams and
falls back to default-key attribution rather than land a key on the wrong
scene. bridge.scene_refinement_keys + the /preview callers drop scene_count.
Reset (_v2_reset_prompt_keys) now wipes refinement_key_pool_for(prompt), so
the pool is one canonical answer shared with the reset-confirmation modal.
That removes the last user of prompt_scene_shortcut_keys, so its entire
internal split is deleted (and its 3 tests, covered by split_scenes tests).
Tests: 20 refinement-key + 140 movie_editor pass. Generation path needs an
in-app smoke test (no model here) — the guard makes a desync loud, not silent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One rich scene model {anchor, scenes:[{index,raw,keys,effect}]} that every
consumer will derive from — editing (raw), preview (raw->expanded),
attribution (keys), reset (pool = union of keys), generation. Built on the
existing shortcut-aware verbatim splitter + refinement_keys_in_text, so it
can't disagree with itself. Nothing repointed yet; consumers still on their
own splitters until the staged repointing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Attribution was re-splitting the raw prompt independently and comparing
scene counts, then guessing on mismatch (all-keys union -> cross-training,
or empty -> keys not detected, incl. single-scene-with-keys). Both the
80MB cross-training and Darki's "custom uploaded keys not detected" trace
to that.
resolve_scene_refinement_keys now reads each scene directly: split with
the shortcut-aware split_timeline_verbatim (same scenes generation uses)
and collect, via new refinement_keys_in_text(), the keys whose shortcuts
fired in each scene (anchor keys apply to all). No scene-count compare, no
union/empty fallback except a genuine multi-scene misalignment. Single
scene with a custom key is detected; each scene gets exactly its keys.
Reverted the original-text scan added to prompt_scene_shortcut_keys (it
desynced that splitter from parse_timeline/generation and caused the
detection regression); it's now used only for session-reset all_keys.
Training law: a scene owned by custom key(s) trains ONLY them; the project
default key is left untouched and learns solely from scenes with no custom
key. Tests added for all three behaviors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wrong-* ratings (appearance/details/action/combos) are prompt-REPAIR
signals — "good gen, wrong words/identity" — not quality rewards. They
were feeding their 0.0/low reward into both value-function training and
the keyless Absolute taste store, stamping visually-good conditioning as
a low-value valley. The VF ascent (now always the default key, applied to
every scene) then pushed conditioning away from good regions into drift /
wrong-character / distorted gens; Absolute stored good gens as global
bad-taste.
Wrong-* now carry skip_value_function: still drive prompt repair + the
relative per-key direction/category memory, but train no reward asset.
Quality ratings (Missing-*/Perfect/Nailed/Awful) unchanged. Gated all 3
relative VF call sites + _v2_learn_absolute. Preventative only — keys
poisoned before this fix still need reset/retrain.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per-scene key attribution diverged from the generation split whenever the
advisor rewrote prompt_to_encode away from the raw prompt, so
resolve_scene_refinement_keys hit its all-keys UNION fallback and assigned
every key to every scene. That union got persisted into the run and read
back at rating time, training every key on every clip (value_fn bloat).
- prompt_scene_shortcut_keys now scans both expanded and original text
(mirrors split_timeline_verbatim), so identical structure -> matching
scene count -> precise attribution far more often.
- Divergence fallback is now default-key only (empty sets), not the
all-keys union; a diverged run under-trains instead of cross-training.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A replacement is a prose phrase and may contain commas (e.g. "The video
rapidly cuts, showing the next view."). Both the Movie Editor's splitLines
and the backend _shortcut_replacements string fallback split on commas, so
one phrase became several variants: the editor showed it across lines,
expansion picked one half at random (never matching the comma-form transition
trigger), and a comma edit reverted on save because it re-split every time.
Replacements are now one-per-line only (commas preserved); triggers still
accept comma-separation per their UI. Existing torn replacements self-heal on
the next edit + save.
- mediabrowser.js: separate splitTriggers (\n or ,) and splitReplacements (\n).
- templates.py: _shortcut_replacements string fallback splits on \n only.
- tests: comma phrase stays one replacement (list + string + normalize paths).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
split_timeline_verbatim expanded shortcuts before detecting transitions, so
a cut marker that is ALSO a shortcut (e.g. qcut/cut, common now that cut
markers carry refinement keys) had its trigger word replaced before
detection -> the split was lost or misplaced ("scene 2 removed, scene 1
shows scene 2's prompt"). It now ALSO scans the original verbatim text for
triggers and merges with the expanded scan; trigger-shortcuts are caught at
their true offset, shortcut-driven splits and plain triggers are unchanged,
and the dedup collapses duplicates. Lossless round-trip preserved.
- conditioning.py: original-text trigger scan in split_timeline_verbatim.
- tests: user's qcut/cut case (2 scenes, actions 10|20,30, lossless) +
shortcut-driven split still works.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The shortcut-internal-cut guard disabled the supported 'shortcut whose
expansion resolves to a transition' splitting (see project prompt model),
so scenes stopped splitting at all. Reverts split_timeline_verbatim to the
prior behavior; the real drop/shift bug needs a targeted fix with the
user's actual prompt + shortcut definition.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
split_timeline_verbatim detected scene cuts in EXPANDED text and projected
them back onto the original. A cut living inside a shortcut's replacement
got projected onto the shortcut's edge; colliding/early boundaries were then
deduped away, dropping a scene and shifting the rest ("scene 2 removed,
scene 1 shows scene 2's prompt") until reload. This surfaced more once
shortcuts could carry refinement keys (more key-bound shortcuts in prompts).
The editor treats shortcuts as opaque (they only expand for preview/
generation), so the verbatim splitter now ignores any transition match that
starts inside a shortcut expansion. Literal user-typed markers still split;
generation still expands and splits unchanged. Result round-trips losslessly.
- conditioning.py: _starts_in_shortcut guard in split_timeline_verbatim.
- tests: shortcut-internal cut is ignored + lossless; literal cuts still split.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per-scene multi-key learning trains every refinement key whose shortcut
fired in the prompt, so a reset must clear them all — not just the
project/default key, which left stale per-key state behind.
- conditioning.py: _v2_reset_prompt_keys() — on reset_session, union the
non-default keys from the raw positive + intent prompts and reset/save
each (primary key excluded; main flow already handles it). Reported in
the state status string.
- store.js: "Reset Studio session" now confirms before arming, listing
every key it will wipe (default + non-default keys in the prompt, from
the preview's scene_refinement_keys). Disarming does not re-prompt.
- tests: union/primary-exclusion + no-op cases.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Surface, before generation, which refinement key(s) each scene will steer
with — exactly matching the generation-time resolution.
- conditioning.py: extract resolve_scene_refinement_keys() as the single
source of truth; FunPackVideoRefinerV2._v2_scene_refinement_keys now
delegates to it (no behavior change).
- bridge.scene_refinement_keys(): preview payload per scene — explicit
non-default keys, or the project default key when none fired.
- server /preview: attach scene_refinement_keys (both gen and non-gen
branches), resolved against the project's refinement_key.
- inspector.js + styles.css: per-scene key chip in the timeline preview
(avg when >1; greyed "(default)" fallback).
- tests: resolver/Studio parity + bridge preview payload.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Shortcuts carry an optional refinement_key (templates model + expansion);
editor gains a "Use non-default refinement key" checkbox + field.
- Project-level Refinement key field (Studio engine card) wired through
server -> builder into FunPackRefinementKeyLoader (settable in any mode).
- Per-scene key attribution (prompt_scene_shortcut_keys): keys whose shortcuts
fired in a scene; anchor keys count for every scene; union fallback on
rewrite/count mismatch.
- Per-scene conditioning merge: steer with each participating key's state and
average (one key = substitute, none = default), replacing the default-key
relative steering for keyed scenes.
- Multi-key rating: full per-scene learning suite + value fn trained into each
participating key's state.
- Tests for shortcut key surfacing, expansion tagging, per-scene attribution,
fallback, and keyloader injection.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Use sampler boundary pixel positions for preview/export offsets instead of subtracting frame_overlap on the client, which misaligned i2v-guided multi-scene runs.
Introduces a new temporal_style option that cycles motion→hold ramps
across each scene's denoise trajectory via a segmented frame_rate wrapper.
Wired through Studio, engine settings, chain sampler conditioning tags,
and build_enhancements for single-scene workflows.
Each mixed scene is its own queue request (max_scenes=1), not continuous
chain overlap. Scene 2+ uses its anchor via graph Img2Video and borrows
hidden guides from the previous scene. Sampler applies guide stacks on
the first chunk for solo mixed runs.
Mixed i2v anchors no longer blend the previous scene tail into the
starting chunk or the stitched output. Dragging an image onto a
continuation scene auto-sets mixed so montage stays one queue request.
Preview in-points skip overlap before mixed scenes.
