'Loaded fine' and 'runs fast' are different claims. Every NVFP4 load now
prints a diagnostics block that settles which one you have:
- comfy_kitchen backend census (the big one: torch built for CUDA < 13
disables ck's CUDA backend, so quantized matmuls silently fall through
to the pure-PyTorch eager backend - correct output, ZERO speedup, with
an explicit warning line when detected)
- torch/cuda build + device capability
- loaded-model layer census: nvfp4 layers vs layers flagged
full-precision emulation (pre-Blackwell GPUs)
- a timed 4096x4096 GEMM probe (incl. dynamic input quantization):
bf16 vs nvfp4 wall time with a FAST PATH ACTIVE / NO SPEEDUP verdict
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three live-run findings from the rental:
1. Block steering said "profile not ready (needs 2+ rated runs)" with 16
paired runs. The v1 liked/disliked-EMA attribution had no contrast when
ratings cluster on one pole (pole-vs-all-runs-mean cancels to zero) and
dropped rewards in (0, 0.3) entirely. Attribution is now the per-block
COVARIANCE between reward and activity over a rolling 32-run history:
every admissible reward contributes with its actual value, contrast
comes from reward variance, and the sampler log now states the real
blocker ("all rated alike - rate both good AND bad runs") instead of a
misleading run count. v1 profiles are discarded on first pairing.
2. output_guidance stayed "not ready (needs 10+)" forever on LTX-AV: the
end-of-run x0 snapshot was gated on output samples being a plain
torch.Tensor, but AV chain output is a NESTED tensor (video+audio) - so
the output value function never received a single sample. The snapshot
now extracts the video stream from nested outputs.
3. FunPackNVFP4ModelLoader hit a size-mismatch wall (9-vs-6 ada params,
connector dims): it loaded the state dict without the safetensors
METADATA that carries the LTX-AV architecture config, so detection
built the wrong model variant. Metadata now flows through, mirroring
comfy.sd.load_diffusion_model.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'No, I'll use my own pipeline' in the missing-nodes prompt silently
PERSISTED disable_core on the project. From then on Engine settings…
greyed out with no reason and Generate died with ComfyUI's raw
"Prompt has no outputs" - both looked like breakage with no visible cause.
- The setup prompt button now confirms first, spelling out what it
disables, what Generate will need, and where to re-enable it.
- Engine settings… never greys over pipeline state anymore: the modal
itself already explains why knobs are unavailable and links to Models.
- Builder: disable_core with NO output node in the graph is now a
BLOCKING error with the fix in the message (wire 🌐 Global video
output, or Models → Enable built-in pipeline) instead of letting
ComfyUI reject the queue.
- GEN_ERROR_MAP: prompt_no_outputs mapped to the same actionable text
for custom pipelines that pass the builder but lack an output node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'What it looks like' section right under the intro: a clean full-workspace
hero shot plus Composer and Engine Settings side by side, generated from
the mocked tour demo (no ComfyUI/GPU needed) by a new
tools/showcase/capture_clean.mjs (npm run capture:clean) - same scaffolding
as capture.mjs but with the tour overlay stripped, so shots are undimmed
and card-free. Images committed under docs/images/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All three buttons are now same-shape shields.io for-the-badge badges
(Ko-fi's rounded SVG pill and Patreon's PNG clashed with the flat
Vast.ai badge), and an IMPORTANT callout right under them sets
expectations: one-person project, provided as is, bug reports welcome.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RLHF at the model level: which REGIONS of the transformer were doing the
work when a generation came out good or bad, and steer accordingly.
1. Measure: tagged forward hooks record each block's relative video-residual
delta per model call (near-zero cost; covered by the existing leak-strip).
2. Attribute: the end-of-run fingerprint is staged (block_snapshot.pt, same
stage-then-pair convention as x0_snapshot) and paired with the rating
reward on the next Refiner cycle into liked/disliked/all-runs EMA
profiles; when output_guidance's value function is trained, an innermost
model-function wrapper additionally credits blocks per step by
(value delta) x (centered activity) - converges in a handful of runs.
3. Steer: vx' = vx_in + g_b*(vx_out - vx_in) on the video stream only,
audio untouched; scores zero-mean/unit-max so emphasis is redistributed
rather than inflated, gains hard-capped at +/-10%. Exact no-op until the
profile has 2+ paired runs; blocks at gain 1.0 get no hook at all.
Chain Sampler toggle + strength (append-only widgets), exposed same-change
in the editor's Chain Sampler Guidance section via the sampler_inputs
bridge. New sidecars ride the defensive <key>.* sweep in
delete_refinement_key, so atomic key deletion covers them.
Tests: toy AV-block model driving the real hooks (recorder ranking, steer
video-only scaling + native path for unit gains, credit wrapper, snapshot/
profile/scores/gains roundtrip, mild-positive band, depth-mismatch reset,
leak-strip tagging). Full suite failure set identical to the pre-existing
baseline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FunPackNVFP4ModelLoader quantizes an existing bf16/fp16 checkpoint to NVFP4
at load time as a pure state-dict transform: selected 2D Linears are
re-serialized in ComfyUI core's quantized-checkpoint format (packed fp4
weights + fp8-e4m3 block scales + fp32 tensor scale + per-layer comfy_quant
metadata) and handed to the stock loading path. Core's MixedPrecisionOps
then does everything: real FP4 tensor-core GEMMs on sm_100+ (sm_120 incl.),
dynamic activation scales (no calibration pass), per-layer LoRA fallback,
and dequant emulation on older GPUs. No custom forward, no monkeypatching.
Zone-aware precision map for the joint AV transformer: default scope
quantizes only video-branch Linears (attn1/attn2/ff) of the middle blocks;
the audio branch, the audio<->video cross-modal bridges, the first/last N
blocks, and all non-matmul weights stay full precision. Scopes widen to
+cross-modal, all blocks, and all 2D layers for experimentation.
Tests drive the REAL comfy.quant_ops/comfy.ops via comfy_kitchen's eager
backend on CPU: scope selection, serialized-format contract, prefix
handling, refusal of pre-quantized checkpoints, and a load+forward
roundtrip through MixedPrecisionOps on both the NVFP4 and emulation paths.
The Movie Editor Models modal picks the node up automatically as a
Unet/Diffusion Model candidate (object_info-driven).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Composer search: filter keystrokes rebuilt the whole window and dropped
focus after every character; now only the list refreshes
- Global prompt: stale funpack-global-prompt-updated events (preview refresh
pinning the LAST completed apply) rewound the Compose textarea while newer
text was still debouncing/mid-parse; overlapping applies could also land
out of order. Added pending-apply guard + apply sequence token
- Composer crashed the editor when opened with no project (welcome screen)
- /result now streams from disk via FileResponse (Range/seek support) instead
of buffering whole videos through a loopback /view fetch that crawled or
aborted while ComfyUI was saving the next run; added '..' traversal guard
- Player keeps a URL-keyed video pool: one failed load left the clip dead
until page reload; now bounded cache-busted retries + recovery on rebuild
- Render validation deleted (and autosaved away!) sceneRenders on ANY HEAD
failure; now only a definitive 404 drops a render
- Temp browser / media bin / filmstrip / duration probes held stalled media
connections open, exhausting Chrome's 6-per-origin pool so /view tabs and
even generate calls hung until reboot; thumbnails snapshot to canvas and
every probe releases its src when done
- Inspector guide frame/apply/strength inputs lacked data-k typing protection
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The free video-only unsharp-on-x0 sharpening (temporal high-pass between
consecutive denoised predictions, no extra model eval) only existed on
Hybrid Euler 2S. Distilled Flow — the recommended sampler for this
pipeline (deterministic, clean audio) — had no equivalent, so the
"quality phase" of the actual production sampler had no sharpening path
at all.
Ports the same _apply_quality_sharpness call into
sample_funpack_distilled_flow, gated to the existing final-correction
(Heun) steps. Threaded through FunPackDistilledFlowSampler's
INPUT_TYPES/get_sampler, Studio's _build_one_sampler, and the Editor's
sampler_panel.js Distilled Flow section — verified live end-to-end
(UI -> studio_settings JSON -> backend build call).
Replace absolute reward regression with Bradley-Terry pairwise ranking
(DPO-style) so the value function learns relative preference between
sampled pairs rather than fitting noisy absolute reward floats.
Add a 3-member bootstrap ensemble; gradient() and ascend() scale by
inter-member agreement so guidance backs off in regions the ensemble
hasn't seen data for, instead of confidently extrapolating (the
static-character collapse this was meant to guard against).
Loosen the "Wrong but loved" exclusion in conditioning.py so a strong
loved_modifier signal is no longer filtered out by skip_value_function
- "Wrong but loved" and "Missing but loved" now train the value
function, per the stronger-signal precedence requested.
Legacy single-net checkpoints migrate transparently on load (weights
remap into all ensemble members, rating buffer preserved intact).
The sampler-side toggle landed as a node widget only, with no Editor UI
control — caught after the fact. Editor UI takes priority over node
widgets for any new toggle going forward. No backend change needed: the
existing generic sampler_inputs bridge (builder.py:338) already forwards
any declared SAMPLER_KNOBS entry onto the Chain Sampler node.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New LatentValueFunction (value_function.py) mirrors OnlineValueFunction but is
trained on a pooled snapshot of the sampler's own predicted output (x0_hat)
instead of the input conditioning, borrowing NoiseTilt's core idea (reward
gradient from the model's own prediction) without its SDE noise-term trick,
which has no analogue in this deterministic-ODE sampler.
New output_guidance / output_guidance_strength toggle on the Chain Sampler
(off by default): per-step correction installed as the outermost
model_function_wrapper (on top of embed_guidance/score_slider), same
near-zero-forward-pass shape as embed_guidance. End of run writes a small
pooled snapshot per refinement key; the next rating cycle trains the value
function on it via a new call site in conditioning.py.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The composer was already being pre-selected on click, but nothing scrolled
it into view or signaled the change — on a fresh project all essentials
show as missing and push the composer off-screen, so the click looked
like a no-op. Now scrolls the composer into view and flashes it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Promotes the dev experimental features to a release: ALG and Momentum Guidance
on Distilled Flow, Bounded Attention, Auto Montage, Temp files browser,
reconnect-after-reload, Models bypass mode, Conditioning Adjustments in Engine
Settings, welcome-screen branch switch/update, plus the Blackwell sm_120
guide-scene crash fix and assorted Cutting Room fixes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract the git switch-branch and update-and-reload flows out of the FunPack
menu into a shared window.FunPackGit module (no duplication), and add both as
ghost buttons on the welcome screen with live branch/behind hints. Lets you
swap branches or update without first loading a project — no more racing the
welcome screen or opening a montage you don't want.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reconnect-after-reload: editor-queued prompts are now stamped with a stable
client_id and a funpack run-identity payload in ComfyUI's queue extra_data.
On boot the editor checks /api/active, re-attaches to any still-running run
(loading its project, blocking Generate, exposing Interrupt, recording the
result), and loops over queued montage runs until the queue drains. Run
identity lives in ComfyUI's ephemeral queue, not a persistent cache.
Temp files browser (Settings ▸ Temp files): media-bin-style grid over
ComfyUI's temp dir for scene previews and other transient outputs. Open in a
new tab (same-origin /view) or save to disk. No in-app player by design.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 0.5 ceiling was only on the widget bounds (samplers.py INPUT_TYPES +
the editor's Chain Sampler knob); the apply path (_append_joyai_memory_guides)
only enforces the 0.25 audio-safe FLOOR, no upper clamp, so higher values
already flow through. Raise max to 10.0 in both widgets so identity can be
pushed past the audio-safe band. Floor (0.25) kept — below it audio degrades
and identity drifts. Tooltip notes 0.25-0.5 is the audio-safe band; higher
pushes identity harder but may degrade audio / over-constrain motion.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Each generation run's preview is a ComfyUI temp file (VHS_VideoCombine
save_output=False; cleared only on ComfyUI restart). But the node had no
filename_prefix, so every run shared the default — a later run could reuse or
overwrite an earlier run's temp filename. The editor faithfully records the
filename each scene was tied to, so once a run clobbered a prior run's file,
the earlier scene stopped playing ("scene 1 stale, scene 2 plays; repeat for
3 and 4") even though temp files only really disappear on restart.
Give every build (= one run) a unique filename_prefix
(funpack_preview_<uuid12> on the run node, funpack_global_<uuid12> on the
lazily-synthesized global-combine node) so each run writes its own temp file
and every scene stays tied to its own file for the whole session. save_output
stays False — previews remain ephemeral and are still cleared on restart, as
intended; the editor records whatever filename ComfyUI returns, so no other
change is needed.
Test: movie_editor/tests/test_builder.py::test_run_output_gets_unique_temp_prefix
(two builds yield distinct prefixes; save_output stays False). Full builder
suite: 19 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Studio popup's per-phrase universal steering (the `adjustments` widget:
CLIP-encode each phrase and shift conditioning toward (+) / away (−) from it
on every generation, prompt-independent) was the one Studio control with no
Editor surface — engine_settings.js even said it "remains in the ComfyUI
Studio popup".
Add a "Conditioning adjustments" section to the Studio card: a phrase+strength
list with add/remove, stored in project.studio_inputs.adjustments (JSON
[{phrase, strength}]). Frontend-only — _run_studio_inputs already passes the
whole studio_inputs dict through and builder.py wires any studio_inputs key
onto the matching FunPackStudio widget, so this reaches FunPackStudio.adjust
with no backend change. Backend already ignores blank/zero rows, so the list
persists as-is while editing.
Also: count active adjustments in the Studio "changed" badge and drop the now-
stale "adjustments remain in the popup" hint.
Not ported: the popup's learned "session phrase bank" (needs a refinement key
+ the /funpack/phrase_memory route) — possible follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On compute capability > (9, 0) GPUs (Blackwell sm_120, e.g. RTX PRO 6000),
xformers' memory_efficient_attention has no kernel that accepts a tensor
attn_bias — fa3F/cutlassF require cap <= (9,0), fa2F/fa3F reject a tensor
bias. So unmasked attention works (i2v ANCHOR scenes generate) but MASKED
attention crashes: scenes generated from GUIDES hit the LTX guide path
(comfy/ldm/lightricks/model.py _attention_with_guide_mask) which passes a
tensor mask → "No operator found for memory_efficient_attention_forward".
This is the ComfyUI-wide issue normally worked around with
--disable-xformers --use-pytorch-cross-attention; when that flag isn't in
effect on an instance, the guide path still reaches xformers.
Fix: _funpack_install_mask_safe_attention(model), called at the top of the
chain sampler's sample(). When the active backend is xformers AND the device
capability is > (9,0), install ComfyUI's per-call optimized_attention_override
(the wrap_attn hook) so MASKED attention falls back to attention_pytorch
(SDPA handles masks on every GPU) while UNMASKED attention stays on the fast
xformers path. Threaded via transformer_options so it reaches every scene's
forward. No global monkeypatch; no recursion (wrap_attn stamps
_inside_attn_wrapper before calling the override). No-op on supported GPUs,
non-xformers backends, or when another override is already installed;
idempotent.
Tests: tests/test_mask_safe_attention.py — 9 cases covering override routing
(masked→pytorch via kwarg or positional, unmasked→original) and install
gating (xformers+Blackwell installs; supported cap / non-xformers / no-CUDA
skip; existing override respected; idempotent). Run with the ComfyUI venv.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extracts the pure global-prompt scene-matching kernel into scene_align.js
(dual browser/CommonJS export) and adds a node test suite so the "stale
scene anchor after reorder/insert/delete" class can't silently regress
again (it has recurred twice).
- scene_align.js: matchParsedToRoots(parsedTexts, rootTexts) — the two-pass
content-first matcher (exact text + nearest-position tiebreak, then
leftover roots in order), index-based and dependency-free. Exposed as
window.SceneAlign in the browser (same pattern as window.PipelineCaps)
and module.exports under node.
- store.js _alignScenesFromParsed now delegates the matching to the kernel
and maps indices back to roots — single source of truth, no duplicated
logic. Behaviour is identical to the previous inline two passes.
- index.html loads scene_align.js before store.js.
- scene_align.test.js: 11 cases via node's built-in runner — reorder,
edit-in-place, insert-at-front, delete-middle, reorder+edit-one,
edit+insert+keep, irreducible, duplicate-text tiebreak, empty-text,
no-roots, used-once. Run with `npm test` (node --test) in this dir.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of the "scene generates from a stale anchor/guide, survives
restart + key-clear" report. _alignScenesFromParsed matched parsed global-
prompt slots onto old scene roots BY POSITION first (oldRoots[i]), then
reused that root keeping its id + source/anchor + mapped render and
overwriting only .text. A reorder/insert/delete in the global prompt shifts
indices, so slot i's NEW text got glued onto a DIFFERENT scene's old
anchor+render. The wrong mapping is then SAVED into the project file, so it
survived ComfyUI restart and refinement-key/embed-guidance clears — it was
never a cache, it was persisted-wrong state.
Rewrite the matching priority to be CONTENT-first:
- Pass 1: exact normalized-text match (nearest original position breaks
ties) pins unchanged scenes to their own root through any reorder/insert/
delete.
- Pass 2: leftover slots inherit leftover roots in order (in-place text
edits keep their render/anchor; a slot with no root left becomes a fresh
scene with no inherited anchor).
The render lives in state.sceneRenders[id], so it follows the id
automatically — fixing which root a slot matches is the entire fix. The
only irreducible case (a scene both reordered AND fully rewritten with no
unchanged neighbour to anchor it) still falls back to a positional guess.
Validated with a standalone node test across reorder / edit-in-place /
insert-at-front / delete-middle / reorder+edit-one / edit+insert+keep /
irreducible — all pass; the old positional-first algorithm fails the first
five. Preserves ghost materialization, unused-root ghosting, timeline_order
tracking, and keep-render-on-minor-edit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
splitScene deep-clones the scene into the second half, then clears the
first half's transition_to_next and transition_frames so the internal
split seam reads as a hard cut (store.js). But it never cleared
video_transition (the NLE video blend). Two consequences:
- videoTransitionState (timeline.js:484) treats ANY non-empty
video_transition as active and falls back to 16 frames when
transition_frames is null, so a phantom transition renders on the
internal A1|A2 seam that should be a clean cut.
- the transition is duplicated: the deep clone kept it on the second
half (correct) AND it survived on the first half (wrong).
- on export _renderClips emits transition="<type>" with tdur=0 for the
first half (store.js:3434-3435), a malformed transition to the stitcher.
Clear video_transition on the first half alongside the other two
transition fields. The clone happens before the clear, so the second
half still correctly inherits the original transition.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cherry-picks the split-scene regression fix (9a50f0a) onto main and
bumps the version for the Comfy Registry release.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
splitScene correctly splices a fresh scene right after the original
(store.js:1826) and deletes the original's durationSec so both halves
fall back to plan-based (frames/fps) length (store.js:1831). But
_probeRenderDuration's async <video> metadata probe only guards against
the scene's media filename changing — a split keeps the same filename,
so a probe already in flight from page load can resolve AFTER the
split and silently rewrite durationSec back to the full original file
length. Since buildPreviewSegments lays out segments using durationSec,
the first half snapping back to full length pushes the second half
later — looks exactly like "a full-length scene got appended to the
end" even though the scene array itself was spliced correctly.
Fixed with a per-render edit token: _probeRenderDuration snapshots
r._editTok before starting the probe and bails if it no longer matches
at resolve time; splitScene bumps the token when it deletes durationSec.
splitScene correctly splices a fresh scene right after the original
(store.js:1826) and deletes the original's durationSec so both halves
fall back to plan-based (frames/fps) length (store.js:1831). But
_probeRenderDuration's async <video> metadata probe only guards against
the scene's media filename changing — a split keeps the same filename,
so a probe already in flight from page load can resolve AFTER the
split and silently rewrite durationSec back to the full original file
length. Since buildPreviewSegments lays out segments using durationSec,
the first half snapping back to full length pushes the second half
later — looks exactly like "a full-length scene got appended to the
end" even though the scene array itself was spliced correctly.
Fixed with a per-render edit token: _probeRenderDuration snapshots
r._editTok before starting the probe and bails if it no longer matches
at resolve time; splitScene bumps the token when it deletes durationSec.
Dragging an audio clip's right trim handle had no upper clamp in the
live preview (only resizeAudioTrack's commit path clamped to
audioTrackMaxDurSec), so the handle would visually drag past the real
end of the source and then silently snap back on mouseup with no
feedback — reads as "can't get audio to fit". Preview now clamps to
the same max and labels it when clamped.
Separately, the inline "Start (s)" number input on audio clips can
keep keyboard focus after a stray click (the controls bar can cover
most of a short clip's clickable area). Since i/o/s/Delete are inert
inside a number field anyway, the global keydown handler now blurs
that input and lets the clip-level shortcut through instead of
treating it as "user is typing".
autoMontage required source.type === "video" (explicit "Convert to
video" scenes only) — but most generations are plain generative scenes
with a render in sceneRenders, never converted. That's why "2+ scenes
selected" still failed: none of them satisfied isVideoClip.
Added _renderRef()/hasPlayableRender(), the same "has a render" check
used elsewhere in this file (removeFromPlan), covering both an
explicit video clip (source_in/source_dur) and a generative scene's
own render (sceneRenders[id].media + inSec, cloned per segment).
Dialog now also pre-fills lead/pool from the current timeline
selection instead of ignoring it.
Custom keys (e.g. a 'cs'-tagged cumshot shortcut) only ever trained
through the Movie Editor's per-scene multi-rating path, which requires
BOTH a multi-scene run AND explicit per-scene ratings from the editor.
Most real generations are single-scene with one overall rating, so
those keys were never created at all -- confirmed on a live instance:
zero custom key files existed despite weeks of use.
Single-scene runs already carry their fired keys in
previous_run["scene_refinement_keys"] (Studio always computes this,
the Movie Editor always forces split_by_transitions=True). Train any
non-default keys found there too, additively -- the default key keeps
learning exactly as before; this doesn't replicate the multi-scene
path's exclusivity (custom key trains instead of default), since that
would mean restructuring the heavily-relied-on single-scene rating
path without separately testing the exclusive behavior there.
It was silently installing or no-op'ing with zero console output,
unlike ALG and Momentum Guidance which both confirm activation. Added
the same style confirmation line on successful hook install, plus a
distinct skip reason for each no-op path (no sentence split detected,
bad latent shape, narrow frame, missing transformer blocks) so it's
possible to tell whether it's running at all without reading code.
Structured Diffusion Guidance (arXiv:2212.05032)-style upgrade: instead
of tokenizing each sentence-half just to COUNT tokens (an approximate
boundary within one shared tokenization, vulnerable to BPE merge drift
at the seam), encode each half SEPARATELY through the real text encoder
and concatenate the two conditioning tensors. The split point is now
exact by construction, and the two spans can never share a token even
before any masking is applied. Same toggle, same metadata field
(funpack_bound_split_tokens) the sampler already reads — no sampler
changes needed.
MultiDiffusion (the other planned "same family" technique) needs
multiple forward passes per region (~2x+ cost) and is skipped per the
project's low/no-overhead bar, pending explicit user sign-off.
_V2_PERSISTENT_ENCODE_CACHE survived for the entire ComfyUI process
lifetime, across every generation request, keyed partly by id(clip) —
a CPython object identity that can in principle be reused for an
unrelated object after garbage collection. Conditioning must always
trace back to what's live in the current request; nothing should be
served from a cache that outlives it. The per-call encode_cache dict
(created fresh per sample() invocation, never escapes it) already
covers legitimate within-request de-duplication and is unaffected.
Live testing found decay=0.9 produces garbage at any strength on our
8-step schedule: the EMA barely forgets, so it stays anchored to the
very first (near-pure-noise, wrong-scale) step's direction for nearly
the whole run. decay=0.5 forgets that stale step within 2-3 iterations
and tested clean even at strength=1.0. Paper's longer-schedule decay
assumptions don't transfer to a few-step distilled run.
Masks text cross-attention (attn2 only, never audio or self-attention)
so the left half of the frame can only attend to subject-1's prompt
tokens and the right half only to subject-2's, fighting attribute/
anatomy bleed between two figures in one frame. Pure attention masking
via PyTorch forward-pre-hooks, no extra forward pass, model-level so it
works on any sampler.
Studio always tags each scene with an approximate token-split boundary
(sentence-count heuristic + real tokenization via clip.tokenize) as
conditioning metadata — cheap, no-op unless the new
bounded_attention_enabled toggle on the Scene Chain Sampler is on.
Opt-in, off by default.
Keeps a running EMA of the per-step ODE direction and blends the current
step toward it once sigma drops below mg_sigma_threshold — the
complementary window to ALG's anchor blur, which only acts above its own
threshold. EMA accumulates every step (free) so there's no cold-start
when application kicks in; video-only, audio untouched. Opt-in, off by
default.
Splices an already-rendered lead clip into shrinking-length segments
(faster cuts toward the end) interleaved with randomly-drawn cutaway
clips from a pool. Cutaway selection is random per-clip but always
sequential within a clip, so later content in a clip can never surface
before its own earlier content. Pure timeline construction, reuses the
splitScene source_in/source_dur mechanism — no new generation involved.
Paper default strength=2.5 visibly changed character appearance, even
without JoyAI-Echo compensating. 2.0 holds i2v consistency noticeably
better while still breaking the static shortcut, confirmed independently
twice. 0.975 threshold (narrower blur window) paired with it in the
same tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a "bypass" toggle per node in the Models modal -- skips the node's
effect (pass its inputs straight through) without losing its saved
config, so testing without a LoRA/node no longer means deleting it or
zeroing a strength widget every time.
Backend (builder.py): _apply_bypass runs after auto-wire, drops a
bypassed slot's node and rewires its consumers to whatever already
feeds its matching-type input (model in -> MODEL out, clip in -> CLIP
out), same idea as ComfyUI's native node bypass. Only acts when the
output->input type mapping is unambiguous; otherwise leaves the node
active and reports why (e.g. a pure source node with no input to pass
through, like a checkpoint loader).
Frontend: bypass is a slot-level flag (slot.bypassed), kept out of
slot.inputs so it's never sent to ComfyUI as a fake widget value.
Exposable to the main editor via the same eye-button pattern as any
other exposed widget value (store.setModelBypass, inspector.js routes
the boolean control to slot.bypassed instead of slot.inputs).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
min=1.01 offset the whole step grid (1.01, 1.11, 1.21, ... 2.51, ...)
so every value the spinner snapped to ended in .01. Min is now a clean
1.0 -- kappa=1.0 is already just a harmless no-op blur, never needed
the 1.01 floor.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Generalizes the i2v-anchor blur (_alg_blur_video_frame0 -> _alg_blur_frames)
to optionally also blur newly-appended guide-attention frames each scene
(mid_scene_guide / carry_i2v_guides-as-guide / configured guides / JoyAI
memory), reusing the existing guide_tail bookkeeping rather than a mask-value
heuristic (frame_overlap continuation also uses mask=0 and must stay
untouched). New Scene Chain Sampler toggle "Blur i2v guides and JoyAI memory"
mutates the shared KSAMPLER's extra_options per-chunk since guide_tail varies
per scene; always resets so nothing leaks across chunks/samplers. Surfaced in
Engine Settings via the existing Experimental section, no backend/builder
changes needed beyond the new INPUT_TYPES field.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adaptive Low-Pass Guidance (arXiv:2506.08456): blurs the i2v anchor frame
while sigma is in the near-pure-noise region, then swaps back to the sharp
anchor, to stop the model shortcutting to a near-static video that just
matches the reference image. Opt-in, off by default, no-op without an i2v
anchor. Swaps model.latent_image per step (the same attribute ComfyUI's
inpaint compositing already re-pastes every step) instead of monkeypatching
comfy internals.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Project-scoped $name variables resolved DEAD LAST in refine_v2 — after
shortcut-expand and the transition split — so a value containing a comma or
trigger word can never move a scene cut (order: shortcuts -> splits -> vars).
Recursive (a value may reference other vars), cycle-safe (self-reference left
literal, no infinite loop), undefined -> literal $name.
- templates.resolve_variables: recursive, depth-guarded, cycle-safe resolver
- refine_v2: resolve base prompt (single-scene) + per-scene texts (post-split);
variables threaded via studio_settings.refiner (same channel as scenes)
- builder/server/timeline: carry project.variables into the build; persist
variables + prompt_templates on the project
- Composer: templates dropdown + Save above the prompt; collapsible Variables
editor (Add/Remove rows) below it, with live undeclared/cycle hint
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deleting a refinement key was non-atomic. A key's state spans <key>.json plus
sidecars (<key>.value_fn.pt, <key>.sampler_ctx.json, attn_maps/blessed maps +
K/V banks + creativity latent + velocity store), and there was no delete route —
users removed the visible <key>.json by hand, orphaning the rest. The orphans are
still applied: _v2_ascend_conditioning loads <key>.value_fn.pt purely by name
(not gated on the json), and blessed K/V inject at a 0.04 floor even at reward 0.
So a "cleared" key kept steering future generations, surviving both a ComfyUI
restart and the manual delete — the persistent half of the long-run quality drift
(the per-session half was the hook leak in the previous commit).
- templates.py: delete_refinement_key() sweeps the canonical json + every <key>.*
sidecar + clear_refinement_data (attn_maps banks, velocity disk+memory).
absolute_store_info()/clear_absolute_store() for the keyless Absolute taste store
(__funpack_absolute_global__): it learns from every rated gen across all prompts,
is invisible in the key list, and was otherwise only wiped by Session Reset.
Routes: POST refinement_keys/delete, GET/POST refinement_keys/absolute|clear_absolute.
- Refinement Key Loader node: "Delete Key" + "Clear Global Taste" buttons.
- Movie Editor: api deleteRefinementKey/absoluteStoreInfo/clearAbsoluteStore +
FunPack menu "Delete refinement key…" / "Clear global taste store…".
- tests: atomic-delete sweep + empty-key guard (real store, unique key, cleanup).
Pre-existing 16 failures unchanged (stale mocks); +2 new passing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
build_enhancements registered forward/forward-pre hooks directly on the
shared diffusion_model blocks but only removed them on scene-transition
sigma jumps, never at end-of-sampling. Single-scene runs (and the final
scene of multi-scene runs) thus left hooks installed; because ModelPatcher
clones share the underlying blocks, every subsequent generation stacked
another set, each lerp-ing block output toward a previous run's stale
reference tensor -> progressive corruption that survived Session Reset and
deleting refinements/ (the leak is in-process, not on disk) and only
cleared on a full restart.
Tag every installed hook (_FUNPACK_HOOK_TAG) and add strip_funpack_block_hooks
to pop only tagged entries from each block's hook stores (idempotent, leaves
foreign hooks intact). Strip at the top of build_enhancements and defensively
at FunPackLTXAVSceneChainSampler.sample() so accumulation is bounded to zero.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend the JoyAI cross-shot memory bank from video-only to paired audio-video,
the part that distinguishes JoyAI-Echo from a generic video-consistency trick.
- _JoyAIMemoryBank now stores (video, audio) pairs under the same pin/roll policy;
frames()/audio() stay 1:1. Audio half is None until joyai_audio_memory is on, so
the phase-1 video path is byte-identical.
- _append_joyai_audio_memory pins each banked prior-shot audio latent into the
chunk's audio stream as a fully-protected (mask=0) tail, cropped back off after
sampling (_crop_audio_tail). The audio analogue of the video guide bank; rides
native audio self-attention since LTX has no audio guide-attention API.
- _harvest_joyai_audio harvests the paired audio frame, position-matched to video.
- v2a_grad_scale scales the model's trained video->audio cross-attention via
reversible forward hooks on each AV block's video_to_audio_attn submodule
(JoyAI uses 2.0). 1.0 installs no hook -> native behavior, zero overhead.
- joyai_audio_memory + v2a_grad_scale wired through sample(), batch training,
refinement context, run_mechanisms, INPUT_TYPES (append-only), and editor knobs.
Deliberately breaks the audio pass-through invariant, so both are opt-in and
off/native by default. 3 new bank tests; zero regressions (same 12 pre-existing
stale-mock failures before/after).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The inputs were inserted mid-list (after mid_scene_guide_strength). The builder seeds
sampler widget values from the reference workflow's POSITIONAL widgets_values array
(extract_widgets maps it onto INPUT_TYPES order), so inserting widgets shifted every
following widget — landing a numeric on the joyai_frame_select combo ("0 not in
['center','first','random']") and corrupting the seeded defaults for embed_guidance /
score_slider / decode_*. Same class of bug the existing forceInput-ordering comment warns about.
Append-only fix: joyai widgets now sit at the END of the widget sequence (after
embed_guidance_source, before the forceInput sockets). They fall beyond the reference
array so extract_widgets leaves the object_info default ('center') in place, and all
pre-existing widgets realign. No logic change; behavior identical once on.
Requires a ComfyUI restart to reload object_info.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The phase-1 sampler inputs existed only on the raw node; the editor's sampler panel is a
hardcoded knob list, not object_info-driven, so they weren't reachable in the normal flow.
Add joyai_memory + its size/anchors/frame-select/strength knobs to SAMPLER_KNOBS and the
Experimental group (beside mid_scene_guide). They flow to the sampler node automatically via
builder.py's generic sampler_inputs pass-through — no backend change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
