A wired positive CONDITIONING means Studio does not encode the prompt.
That is not a preference to configure; it is what wiring the input says,
and it is what a wired negative CONDITIONING has always done. Wanting
CLIP to own the prompt means not wiring one.
Removes clip_owns_prompt from the node, both settings panels and the
settings JSON.
A new scale was invented where two calibrated ones existed.
The ceiling is now 1.25 — the exact boost _v2_build_attn2_patch has
applied to an emphasised phrase's K/V on LTX since long before this, on
a path that works. A different mechanism for the same intent should not
push harder.
The ramp toward it is _v2_auto_strength, which already answers "how much
evidence is there": base 0.030, scaled by bad_streak and reward EMA,
clamped to [0.008, 0.085]. So one Awful reaches about x1.13 and a
sustained bad streak x1.19, instead of the top phrase jumping straight to
the maximum off a single rating.
Two defects behind a reported x1.9.
Spans overlap — a phrase and one of its own words match the same tokens
and their biases ADD. Intended, so the word reinforces the phrase, but
unbounded three overlapping x1.5 spans reach an effective x3.4 on one
token. Bounded against the strongest SINGLE span, so an explicit
(word:2.0) still means exactly 2.0.
And it shipped on by default. It changes what the model sees on every
rated run, and it is gated on is_h3_clip — which is why disconnecting
CLIP made generations normal again. Now a toggle, off.
Also replaces prefer_wired_conditioning with clip_owns_prompt: a wired
conditioning owns the prompt by default, and the old order is the opt-in.
CLIP won whenever both were connected, so Studio replaced an R2V/I2V
node's conditioning with its own text-only encode. On H3 that discards
the reference outright: Qwen3-VL is causal and the vision block precedes
the prompt, so every prompt row is a hidden state that already read the
image. Studio cannot reproduce that — nothing wires h3_references or
source_image in the editor pipeline, so its encode never sees a picture.
Detected from minimax_token_tags (0 marks a vision block) rather than
guessed, so a text-only or non-H3 graph keeps the old precedence.
Studio reads studio_settings.refiner.negative_prompt whenever the wired
negative_prompt is blank. The editor writes the wired one and has no
field for the stored one, so clearing the visible box handed control to
a negative saved in an earlier session, invisibly — a 353-character
negative encoding on a run whose Negative prompt field was empty.
The editor's box now writes both, empty included. The log says which
source supplied the text instead of only how long it was.
Three problems, all visible in the first live run.
x1.03 on every phrase: category scores are 0..1 confidences and a phrase
rarely carries much of one axis, so `1 + strength * score` was applied
and inert. Weights are now relative to the strongest candidate, making
strength mean "how much the best phrase gets".
94 candidates capped to 8 by LENGTH kept an arbitrary eight. Ranking is
by weight; longest-first is now only the application order, so a phrase
still lands before one of its own words adds to it.
The prompt was assumed to be the tail of the conditioning. The same run
reported 367 conditioning rows against 368 modality tags, which is
exactly how that assumption breaks. minimax_token_tags marks text as 1
and vision blocks as 0, and references precede the prompt, so the prompt
is the trailing run of 1s — read it instead of inferring it. The status
line says which of the two placed it.
Studio has known which words to weight since 2.0 —
_v2_concept_units_for_run splits every run into phrases, single words and
n-grams; _v2_update_phrase_memory trains each against the rating's
missing/wrong axes. The only consumer was _v2_build_attn2_patch, and H3
skips it: one packed self-attention stream, no cross-attention. So the
learning ran on every generation and reached nothing.
It now becomes an attention-logit bias on the prompt's token positions in
the packed stream, which is the same statement the K/V x1.25 emphasis was
making, at a place H3 has. Boost only by default.
Located with the tokenizer's offset mapping (verified against ComfyUI's
H3 vocabulary), not by re-encoding each phrase through Qwen3-VL-32B the
way _v2_find_phrase_token_ranges does.
The bias is a mask and SLA routes masked calls to dense, so a weighted
run is a dense run. Said once, at install.
MiniMax H3 encodes with Qwen3-VL; the stand-in path fetched a 12B Gemma
tokenizer regardless. Local HF cache is now tried before any download,
and a download announces itself — it blocks the run with no timeout.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
On H3 it is a Qwen3-VL-32B forward whose output CFG 1.0 never reads.
It still has to happen, but it must be visible when accounting for
what a run loaded and when.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Erase the negative's pooled direction from the positive conditioning
(projection by default, never the vision span). Plus an option to let a
wired positive CONDITIONING own the prompt while CLIP keeps its jobs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pass 2 can run a different algorithm. Unwired it reuses pass 1; in the
Editor it is behind Second pass > Own sampler, and Studio mirrors the
high pass into that output when off, so existing projects are unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lift the unsharp out of the sampler loop through a denoiser proxy, the
same way ALG does. Window is a fraction of the schedule; audio excluded.
Off by default; an unwrappable sampler keeps sampling and says so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_refine_latent rejects nested latents at the door and every other caller
returns the clone unmodified, so the alias is unreachable. Noted so the next
audit does not re-open it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Unreferenced, and negatives do not exist at CFG=1 where most generation happens.
Drops _v2_negative_prompt_advisor and its two private helpers (131 lines);
_v2_negative_text_conflicts_with_request keeps 2 live callers and stays.
Tests: 968 passed, same as baseline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
14 more silent no-ops made audible: conditioning memory blend/repel/rollback,
Bounded Attention, ALG blur, guide image load, identity install, wired pins,
shortcut expansion. Both hook-removal leaks now name the restart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One mechanism for both: failed() names the attempt, the error and the effect on
the output, once per run; note_on_change states a standing condition once and
again only when it changes. 13 silent no-ops made audible, 4 repeats silenced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CLIP wins whenever it is connected and the wired CONDITIONING is never read;
the ownership label that knew this was computed and thrown away.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
LTX-2.5 encodes with Gemma4 where 2.3 used Gemma3. Concept steering finds a phrase
by matching its token ids inside the prompt's, so a tokenizer from the wrong family
still returns spans — pointing at the wrong tokens. Nothing raises; steering just
stops working.
The tokenizer now comes from the connected CLIP, which is by construction the one
that produced the conditioning. The download stays as the fallback for runs with no
CLIP, and says so rather than claiming to be Gemma3.
Learned state is stamped with its encoder and cleared when that changes, on both the
V1 and V2 paths. Clearing reuses Session Reset so the value function, x0 snapshots
and the keyless Absolute store go with it instead of being orphaned. An unstamped
key is adopted, never wiped: every key predating this has no stamp.
The vision probe no longer depends on the submodule being called gemma3_12b — it
still is on 2.5, where it holds Gemma4.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
LTX 2.5 reuses LTXAVModel/LTXBaseModel behind config flags, so the samplers,
the identity-transfer wrappers and the guide path all bind to it unchanged.
Two places assumed 2.3 specifics, and both would have gone wrong quietly.
The AV channel split was a hardcoded width table. From 2.5 on comfy reads the
video/audio projection widths off the checkpoint rather than assuming them, so
a checkpoint can ship a pair the table has never seen — and an unrecognised
width made ltxav_video_channels return the FULL width, which silently steers
the audio text context along with the video one. The only symptom would have
been degraded audio. The split is now measured off the live model when one is
on hand; the table stays as the fallback.
Decode-time noise only ever reached VideoVAE.decode. 2.5's diffusion decoder
takes no timestep and seeds itself, but it is an nn.Module, so stamping the
attributes onto it succeeded and was then ignored — a knob that reads as live
and does nothing. It is now capability-tested against the real contract
(hasattr decode_timestep, not a class name) and says so when it cannot apply.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three things that were each locked to one implementation detail:
- steps + schedule were a KSampler-branch feature, though every sampler here
walks a sigma list handed in from outside. They are now pass-level, so
Distilled Flow and Hybrid Euler 2S can run a named schedule (and the second
pass can be computed rather than typed) in both frontends and on the node.
- the project's Frames field snapped to the model's grid on every family. On
H3 that is required, but its arrows moved by 1 and the snap put the value
straight back; on LTX the builder rounds up by itself, so the snap only
fought the user. H3 now steps a whole 17 frames per arrow, LTX keeps what
you type and walks 8k+1 with its arrows.
- ALG lived inside the Distilled Flow step loop, so it was unreachable with a
stock KSampler. Its blurred/sharp swap is decided by the step's sigma, and
sigma is an argument of every model call, so a denoiser proxy drives it from
outside any sampler. New alg_anchor toggle on the Chain Sampler drives the
in-loop version too, so there is one control wherever you are.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An image prompt comes back one tag long: comfy builds minimax_token_tags from
the tokenizer's INPUT sequence (embeds.shape[1]) while the conditioning is what
the encoder returned, and the vision block's expansion does not survive 1:1.
The reconcile treated any surplus as corruption and dropped the whole vector —
which on exactly those prompts discards the modality-0 marks for the image span
and leaves the DiT modulating the picture as if it were text.
Position i still means position i, so the tail is trimmed instead. Growing
(appended tokens) is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two independent, silent breaks made an H3 i2v scene generate from the prompt
alone.
1. The Editor always runs split_by_transitions, and the per-scene conditionings
REPLACE the single entry Studio built — but they were re-encoded with no
reference_image and no h3_references. Both the frame-0 keyframe pin and the
Qwen vision presentation were gone before the sampler looked, even for a
one-scene project. The split now carries the anchor onto scene 0 and the
references onto every scene. Gated to H3: on LTX the anchor rides in the
latent, and adding Gemma3 vision to scene 0 would change existing output.
2. A MiniMax H3 Image to Video node hands its first/last frame pins out on its
CONDITIONING output, which had nowhere to go — the sampler's positive comes
from Studio. New optional h3_keyframes input on the Chain Sampler salvages
them: first frame onto the opening scene, last frame re-indexed onto the
closing scene's own final frame (the source node's frame_count is its own,
and a stale index would land mid-clip where PackedLayout refuses it). The
editor auto-wires it for H3 projects.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A SAMPLER object is only the step function, so picking KSampler used to leave
the pass with no schedule at all unless a sigma list was typed by hand.
steps + schedule now build one from the loaded model's model_sampling, exactly
as ComfyUI's BasicScheduler does. The schedule dropdown is the switch between
the two sources rather than a silent precedence rule: "use_user_sigmas" — the
default, so nothing changes for an existing config — runs the Sigmas field as
typed, and any real scheduler computes instead, leaving the typed schedule
parked in the field to switch back to without retyping it. Studio logs the fact
that the field is being ignored, and steps is only shown while it applies.
Both frontends, same change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
minimax_h3.py shipped tags_match / token_tags_length / extend_token_tags with a
docstring saying a mismatch is a hard IndexError mid-forward — and nothing ever
called them. Studio edits conditioning in many places, and H3 tokenizes reference
images into the same span (ref_image_size "match" scales them to the output
resolution), so the token count moves with the prompt AND with the resolution.
That is why the crash looked intermittent and why editing either one made it come
and go, reported only as a bare "list index out of range" at step 0.
Studio now reconciles before returning: extend the tags when tokens were appended,
drop them when tokens were removed (absent tags the DiT handles, wrong tags it does
not), and print which happened — losing the tags changes how the text span is
modulated, so it must not be silent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
H3 support was written against ComfyUI's PR; it has since merged (57500fc5, shipped in
v0.30.0) and the weights are out. Everything the adapter assumed still holds — audio time
axis, 17k+5 grid, batch-1 forward, the imported optimized_attention symbol, every payload
key — so this is what the release itself changed, not a correction of the reading. LTX-2 /
LTXAV is untouched throughout: a project picks a family and both are first-class.
* i2v anchors now work on H3. LTX writes the image into the starting latent
(LTXVImgToVideoInplace); H3 has no such path — its anchor is a condition row packed
beside the text — so a scene's anchor image becomes a frame-0 keyframe pin instead, and
the LTX node is no longer run against an H3 latent. Studio hands the source image's
pixels to the sampler (it owns the CLIP, the sampler owns the VAE). Pins now MERGE, so
a first-frame anchor and a last-frame guide coexist: H3's own first-and-last mode.
* Frame geometry follows the model: frames per scene snap to the family's grid (LTX 8k+1,
H3 17k+5) in both frontends, in the graph builder, and in the sampler, and H3 renders at
its fixed 24 fps. Picking a family moves the project onto its geometry and says so.
* The two checkpoints are named. fl2va (anchors) and ref2va (references) load identically
and never reject each other's conditioning, so the run reports which mode it is in and
the setup panel spells out the split.
* Pipeline setup is no longer a placeholder: released, with the published filenames.
* Reference images can be encoded at the reference pipeline's 2048px short edge.
Fixes found on the way:
* A video reference lost its soundtrack between Studio and the sampler — Studio had
already announced it to the text encoder as <Audio 1>, so every later audio ordinal in
the prompt pointed one reference earlier than written.
* Easy Gen never knew it was on H3: its model config lives on the project, which the
shared capability check did not read, so every H3 warning silently reported LTX.
* An off-grid scene length failed with arithmetic instead of an explanation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The graph builder is now parameterized by model family instead of assuming LTXAV. H3
keeps the core's shape but three of its nodes do not apply: LTXVConditioning (stamps LTX's
frame_rate onto conditioning, which H3 has no field for), LTXVConcatAVLatent (H3's own
EmptyMiniMaxH3LatentAV emits both streams, so the latent feeds the sampler directly) and
LTXVAudioVAEDecode (reads an output_sample_rate H3's audio VAE does not define — it would
raise). Requirements, wirable ports, role targets, guided-wiring hints and the port picker
all follow the family, so a loader can no longer be wired into a node the graph never emits.
The family is a Models setting, asked and never guessed from a checkpoint filename: a wrong
guess fails deep inside ComfyUI at generate time instead of in the panel that can fix it.
Video references now work, rather than being refused. FunPack owns the pipeline and already
requires ffmpeg for preview/export, so minimax_h3 decodes reference clips itself — once, at
24 fps — and derives both views from that single decode: the 2 fps timestamped frames Qwen
sees, and the full-rate latent block the DiT packs. A clip's own soundtrack rides with it,
presented as its <Audio j> label immediately before its <Video k>, which is what ties the
two together. Capped at 15s because reference tokens ride every sampling step.
Editor UI: a Reference media view (H3 only — there is no LTX equivalent to grey out) with
per-type 1-based tags shown exactly as the prompt must spell them, reorder controls that
renumber both sides at once, and a soundtrack picker on video entries. Warning chips beside
Generate for the LTX-only sampler settings H3 switches off, so that is known before a
generation is spent rather than after; Easy Gen gets the same warnings on its strip and the
same family selector (it already loads models.js and pipeline_caps.js).
Verified live in the Cutting Room: the family toggle flips requirements, ports and hints
both ways and persists per project; tags renumber on reorder; the H3 chips replace the
Best-FaceID chip on H3 and hand back to it on LTXAV.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
H3 packs reference blocks into the sequence itself (own position grid, never denoised,
never rendered), which is the thing Best-FaceID approximated on LTX with overlap tokens
plus a trained ArcFace projector. On H3 there is nothing to project and nothing to slice
back off, so this is the model's own feature, not a port of that one.
A reference has to reach the model twice and in the same order both times: as
presentation (Qwen sees "<Picture 1>: <vision block>" so the prompt can name it) and as
a packed latent block. Those encodes happen in different nodes, so the ordered spec
travels between them:
- Studio owns the CLIP: resolves the reference list, bakes the presentation via the
tokenizer's minimax_ref_items, and records what it presented as funpack_h3_refs on
the conditioning. Reference media and a first-frame anchor are separate conditioning
modes, so source_image steps aside (loudly) when references are present.
- Chain Sampler owns the VAE: encodes exactly that list, in that order, sized against
the scene's canvas, and attaches minimax_refs. Applied on every scene branch so an
identity holds across the whole chain, and cached per run so it is encoded once
rather than once per scene.
Order is the contract — drop or reorder an entry and "<Picture 2>" points at a
different reference than the text encoder saw. Anything that cannot be encoded is
dropped AND named, and an audio reference lost for want of an audio_vae says out loud
that later <Audio j> labels have shifted.
New optional audio_vae input on the Chain Sampler (audio references only; image
references need just the main vae). Video references are refused with a reason rather
than half-implemented — they need 2 fps frame sampling and a video loader FunPack does
not own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Day-0 groundwork for Comfy-Org/ComfyUI#15224 (MiniMax H3). New minimax_h3.py is the
one place that knows how H3 differs from LTXAV; everything else consults it.
Four divergences that failed silently before:
- Audio latent axis. LTXAV audio is [B,C,T,freq] (time on dim 2, same as video);
H3 audio is [B,32,stereo,T]. Both 4-D, so only the model family can say which
axis is time. The Chain Sampler now carries a per-stream time axis through
continuation, blending, tail crops and JoyAI audio memory.
- Frame grid. LTXAV 8k+1 -> k+1; H3 17k+5 -> 5k+2. H3's downscale_index_formula
is the index map, not the count map, so the old uniform formula under-counted
by ~20%. Frame counts now come from vae.downscale_ratio[0], which reproduces
LTXAV's answer exactly.
- Attention patches. LTX calls optimized_attention through its module; H3 binds
it at import, so FunPack's temperature/capture patch was inert on H3. It now
rebinds every module holding a reference, and position-indexed work (BachVid
K/V, attn importance) sits out H3's pre-reshaped [1,heads,S,D] layout.
- Batch size 1. H3's DiT refuses a batched forward and comfy batches cond+uncond
for cfg != 1.0. install_batch_split runs them one at a time, keeping cfg live.
Guides become keyframe pins: H3 packs condition rows into the sequence instead of
appending a masked latent frame, so no tail is appended and none is cropped. The
layout only accepts a first/last pin, and a mid-clip request is refused up front
instead of raising inside PackedLayout.
Studio: reference images route through Qwen3-VL's images= kwarg (the Gemma3 vision
probe returns False for H3 and was dropping them); the attn2 K/V direction patch
reports that H3 has no cross-attention to hook instead of installing a no-op.
Features that depend on LTX transformer structure (bounded attention, Best-FaceID
identity transfer, v2a_grad_scale, segmented detailing, second-pass latent ops) are
now switched off explicitly on H3 with the reason printed, rather than each one
discovering its own missing attribute mid-scene.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Punchy frame_rate mult held over the first/last slice of a scene's
denoise progress, easing to natural in the middle - same progress-
from-sigma mechanism as pulse, shaped as a single edge ramp instead
of repeated segments. Wired through build_enhancements (single-scene),
the Scene Chain Sampler's per-scene wrapper dispatch, and both Studio
and Editor temporal-style dropdowns.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds taste_nearest_prompt on the Scene Chain sampler. On every liked rating the
Refiner now records a (pooled-conditioning fingerprint -> liked direction) entry in a
ring buffer beside the global liked_dir. When the toggle is on, each scene retrieves
the similarity-weighted direction of its nearest rated prompts and uses that for
embed_guidance / score_slider instead of the one global average, falling back to the
global direction when nothing is close enough. Non-parametric (cosine lookup + vector
mean, no extra model forward, no value-function collapse risk). Wired into the Editor
Guidance panel. UNVALIDATED LIVE.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Editor ▸ Settings ▸ Editor gains a 'Shortcut revolver' toggle (+ 'Random' sub-option).
When on, each shortcut with several replacements draws from a per-shortcut queue —
sequential (first…last, then restart) or shuffled once per cycle — and nothing repeats
until the whole set has fired. Off (default) keeps the seeded random pick unchanged.
- State is a sidecar (shortcut_revolver.json) next to the shortcut DB: settings +
per-shortcut remaining queue + replacement-set fingerprint (editing a shortcut's
replacements restarts its cycle; changing either setting resets all cycles).
- Generation commits the draw (refine_v2 positive prompt); every preview path
(parse_timeline, split mapping, fragments) peeks the same stored queue without
saving, so previews show exactly what the next Generate will use.
- Batch Training variants each commit a draw: N variants sweep N consecutive
chambers instead of seed-rolled repeats.
- FunPackStudio.IS_CHANGED returns nan while the revolver is on so fixed-seed
re-runs aren't served from ComfyUI's node cache (the next chamber must fire).
Verified live on local ComfyUI: settings round-trip, seed-independent stable peeks,
sequential cycle REP_ONE→TWO→THREE→ONE across processes, peek-after-commit shows the
next chamber. 9 new unit tests; full suite has the same 20 pre-existing failures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'Awful' / 'Wrong appearance' banked on intrusion semantics; now any rating
whose profile reward marks the run a bad OUTCOME (<= -0.25: Missing
quality, Missing details+quality, Missing action+quality, Missing
details+action+quality) also feeds the negative bank - degradation is
visible in the latent, so steering off that attractor is meaningful.
Near-miss ratings with positive reward (Missing details +0.35, Missing
action +0.05) stay excluded: those latents are mostly-wanted content and
banking them would repel future runs from material the user asked for.
Decision moves into negative_memory.is_negative_profile (intrusion keys OR
reward threshold, robust to future profiles); promoted entries now carry
the rating key as provenance for severity-weighted steering later.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Store what went WRONG, not just what was asked for. Ratings 'Awful' and
'Wrong appearance' promote that run's final video latent (fp16, batch
squeezed) + mean-pooled conditioning into a per-key negative bank
(<key>.negative_latents.pt, ring buffer of 8). The Chain Sampler writes a
pending candidate at the end of every keyed run; the rating that scores
that run promotes or discards it - same pairing as the x0 snapshot, and
the stem glob in delete_refinement_key covers both new sidecars.
Sampling (dynashift toggle + strength/threshold, off by default):
per late step each current video frame is pooled and compared against
every banked negative frame (alignment-free in time, so chain position /
guide tails don't matter; other resolutions are skipped). Frames matching
above threshold get the matched negative frame's aligned component
subtracted (projection removal, coef clamped >= 0), scaled by
above-threshold margin - steering self-releases once the unwanted feature
is gone. Negatives are weighted by prompt similarity so bad gens from
unrelated prompts steer less. No uncond pass, no extra model forward;
audio byte-identical (video span slice + cat). Wrapper rides the tagged
per-scene lifecycle (leak-strip covered) and threads through batch
training.
Editor: DynaShift knobs land in Engine Settings > Guidance same-change
(sampler_inputs pass-through, no backend change), verified rendering live.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Block steering: run-averaged block activity is architecture-determined
(varies in the 4th decimal run-to-run), so no scoring scheme - pole EMAs
or reward correlation - could extract a rating signal from it. The idea
needs within-run per-step measurement to have a chance; that is a
research project, not a feature. Removed: module, sampler wiring, the
two trailing widgets (positional reference mapping unaffected), the
Refiner pairing hook, editor knobs, tests. Stale block_snapshot/
block_profile sidecars are inert and covered by the existing
delete_refinement_key sweep.
FunPackNVFP4ModelLoader: the FP4 GEMM path was real (3.35x at 16k
tokens) but end-to-end the user's runs are dominated by everything the
loader cannot touch (attention, prep, encode - the sampler node is
~23s of a ~60s run) while dynamic input quantization added a fixed
per-layer-call toll. Net observed: overhead. Users are better served
by dedicated NVFP4 loaders. Removed: module, registration, tests.
Kept (independently useful): the chain sampler's phase timing report +
quantized-layer census, the nested-AV x0 snapshot fix, and the sm_120
masked-attention work.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RLHF at the model level: which REGIONS of the transformer were doing the
work when a generation came out good or bad, and steer accordingly.
1. Measure: tagged forward hooks record each block's relative video-residual
delta per model call (near-zero cost; covered by the existing leak-strip).
2. Attribute: the end-of-run fingerprint is staged (block_snapshot.pt, same
stage-then-pair convention as x0_snapshot) and paired with the rating
reward on the next Refiner cycle into liked/disliked/all-runs EMA
profiles; when output_guidance's value function is trained, an innermost
model-function wrapper additionally credits blocks per step by
(value delta) x (centered activity) - converges in a handful of runs.
3. Steer: vx' = vx_in + g_b*(vx_out - vx_in) on the video stream only,
audio untouched; scores zero-mean/unit-max so emphasis is redistributed
rather than inflated, gains hard-capped at +/-10%. Exact no-op until the
profile has 2+ paired runs; blocks at gain 1.0 get no hook at all.
Chain Sampler toggle + strength (append-only widgets), exposed same-change
in the editor's Chain Sampler Guidance section via the sampler_inputs
bridge. New sidecars ride the defensive <key>.* sweep in
delete_refinement_key, so atomic key deletion covers them.
Tests: toy AV-block model driving the real hooks (recorder ranking, steer
video-only scaling + native path for unit gains, credit wrapper, snapshot/
profile/scores/gains roundtrip, mild-positive band, depth-mismatch reset,
leak-strip tagging). Full suite failure set identical to the pre-existing
baseline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The free video-only unsharp-on-x0 sharpening (temporal high-pass between
consecutive denoised predictions, no extra model eval) only existed on
Hybrid Euler 2S. Distilled Flow — the recommended sampler for this
pipeline (deterministic, clean audio) — had no equivalent, so the
"quality phase" of the actual production sampler had no sharpening path
at all.
Ports the same _apply_quality_sharpness call into
sample_funpack_distilled_flow, gated to the existing final-correction
(Heun) steps. Threaded through FunPackDistilledFlowSampler's
INPUT_TYPES/get_sampler, Studio's _build_one_sampler, and the Editor's
sampler_panel.js Distilled Flow section — verified live end-to-end
(UI -> studio_settings JSON -> backend build call).
Replace absolute reward regression with Bradley-Terry pairwise ranking
(DPO-style) so the value function learns relative preference between
sampled pairs rather than fitting noisy absolute reward floats.
Add a 3-member bootstrap ensemble; gradient() and ascend() scale by
inter-member agreement so guidance backs off in regions the ensemble
hasn't seen data for, instead of confidently extrapolating (the
static-character collapse this was meant to guard against).
Loosen the "Wrong but loved" exclusion in conditioning.py so a strong
loved_modifier signal is no longer filtered out by skip_value_function
- "Wrong but loved" and "Missing but loved" now train the value
function, per the stronger-signal precedence requested.
Legacy single-net checkpoints migrate transparently on load (weights
remap into all ensemble members, rating buffer preserved intact).
New LatentValueFunction (value_function.py) mirrors OnlineValueFunction but is
trained on a pooled snapshot of the sampler's own predicted output (x0_hat)
instead of the input conditioning, borrowing NoiseTilt's core idea (reward
gradient from the model's own prediction) without its SDE noise-term trick,
which has no analogue in this deterministic-ODE sampler.
New output_guidance / output_guidance_strength toggle on the Chain Sampler
(off by default): per-step correction installed as the outermost
model_function_wrapper (on top of embed_guidance/score_slider), same
near-zero-forward-pass shape as embed_guidance. End of run writes a small
pooled snapshot per refinement key; the next rating cycle trains the value
function on it via a new call site in conditioning.py.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Custom keys (e.g. a 'cs'-tagged cumshot shortcut) only ever trained
through the Movie Editor's per-scene multi-rating path, which requires
BOTH a multi-scene run AND explicit per-scene ratings from the editor.
Most real generations are single-scene with one overall rating, so
those keys were never created at all -- confirmed on a live instance:
zero custom key files existed despite weeks of use.
Single-scene runs already carry their fired keys in
previous_run["scene_refinement_keys"] (Studio always computes this,
the Movie Editor always forces split_by_transitions=True). Train any
non-default keys found there too, additively -- the default key keeps
learning exactly as before; this doesn't replicate the multi-scene
path's exclusivity (custom key trains instead of default), since that
would mean restructuring the heavily-relied-on single-scene rating
path without separately testing the exclusive behavior there.
Structured Diffusion Guidance (arXiv:2212.05032)-style upgrade: instead
of tokenizing each sentence-half just to COUNT tokens (an approximate
boundary within one shared tokenization, vulnerable to BPE merge drift
at the seam), encode each half SEPARATELY through the real text encoder
and concatenate the two conditioning tensors. The split point is now
exact by construction, and the two spans can never share a token even
before any masking is applied. Same toggle, same metadata field
(funpack_bound_split_tokens) the sampler already reads — no sampler
changes needed.
MultiDiffusion (the other planned "same family" technique) needs
multiple forward passes per region (~2x+ cost) and is skipped per the
project's low/no-overhead bar, pending explicit user sign-off.
_V2_PERSISTENT_ENCODE_CACHE survived for the entire ComfyUI process
lifetime, across every generation request, keyed partly by id(clip) —
a CPython object identity that can in principle be reused for an
unrelated object after garbage collection. Conditioning must always
trace back to what's live in the current request; nothing should be
served from a cache that outlives it. The per-call encode_cache dict
(created fresh per sample() invocation, never escapes it) already
covers legitimate within-request de-duplication and is unaffected.
Live testing found decay=0.9 produces garbage at any strength on our
8-step schedule: the EMA barely forgets, so it stays anchored to the
very first (near-pure-noise, wrong-scale) step's direction for nearly
the whole run. decay=0.5 forgets that stale step within 2-3 iterations
and tested clean even at strength=1.0. Paper's longer-schedule decay
assumptions don't transfer to a few-step distilled run.
Masks text cross-attention (attn2 only, never audio or self-attention)
so the left half of the frame can only attend to subject-1's prompt
tokens and the right half only to subject-2's, fighting attribute/
anatomy bleed between two figures in one frame. Pure attention masking
via PyTorch forward-pre-hooks, no extra forward pass, model-level so it
works on any sampler.
Studio always tags each scene with an approximate token-split boundary
(sentence-count heuristic + real tokenization via clip.tokenize) as
conditioning metadata — cheap, no-op unless the new
bounded_attention_enabled toggle on the Scene Chain Sampler is on.
Opt-in, off by default.
Keeps a running EMA of the per-step ODE direction and blends the current
step toward it once sigma drops below mg_sigma_threshold — the
complementary window to ALG's anchor blur, which only acts above its own
threshold. EMA accumulates every step (free) so there's no cold-start
when application kicks in; video-only, audio untouched. Opt-in, off by
default.
Adaptive Low-Pass Guidance (arXiv:2506.08456): blurs the i2v anchor frame
while sigma is in the near-pure-noise region, then swaps back to the sharp
anchor, to stop the model shortcutting to a near-static video that just
matches the reference image. Opt-in, off by default, no-op without an i2v
anchor. Swaps model.latent_image per step (the same attribute ComfyUI's
inpaint compositing already re-pastes every step) instead of monkeypatching
comfy internals.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Project-scoped $name variables resolved DEAD LAST in refine_v2 — after
shortcut-expand and the transition split — so a value containing a comma or
trigger word can never move a scene cut (order: shortcuts -> splits -> vars).
Recursive (a value may reference other vars), cycle-safe (self-reference left
literal, no infinite loop), undefined -> literal $name.
- templates.resolve_variables: recursive, depth-guarded, cycle-safe resolver
- refine_v2: resolve base prompt (single-scene) + per-scene texts (post-split);
variables threaded via studio_settings.refiner (same channel as scenes)
- builder/server/timeline: carry project.variables into the build; persist
variables + prompt_templates on the project
- Composer: templates dropdown + Save above the prompt; collapsible Variables
editor (Add/Remove rows) below it, with live undeclared/cycle hint
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Symmetric to the anchor (prepended): a project-level postfix appended to
each scene's conditioning. Lives next to the negative prompt in the project
inspector with its own enable toggle; never part of the verbatim global prompt.
- timeline: Project.postfix + postfix_enabled, effective_postfix(), carried in
build_generation_scene_segments and folded into the run fingerprint.
- conditioning: split_scenes_from_segments expands the postfix (keys fold into
every scene); the structural fold appends it after anchor+scene text.
- inspector: Postfix field with inline enable toggle after Negative prompt.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause the user pushed on: the editor flattened all scenes into ONE prompt
string and Studio re-derived boundaries by parsing text, so a carry scene with no
transition needed an injected 'scene N' delimiter. The editor already KNOWS every
boundary (each generative unit is a scene) — it never needed to smuggle markers
into the prompt.
- timeline.build_combined_prompt: no longer injects anything (verbatim anchor +
scene texts). New build_generation_scene_segments(project) lists the boundaries
the editor already has (each unit's raw text, prefixed with its REAL transition
so effects survive; carry = plain text).
- conditioning.split_scenes_from_segments: canonical split from an explicit scene
list (reuses split_scenes per item for expansion/keys/effect, folds the anchor),
same output shape as split_scenes. refine_v2 gains scene_segments and uses it
when present (count-preserving: a text-less i2v scene keeps its slot, no anchor
shift); falls back to text-splitting only for raw node-pack use.
- server/builder: editor sends scene_segments -> studio_settings.refiner.scenes.
Net: the generation prompt Studio encodes is now exactly what the user typed —
no 'scene N' anywhere, in the prompt or the conditioning. Tests updated to assert
the verbatim prompt + structural segments; new tests for split_scenes_from_segments
(boundaries, empty-scene preservation, anchor/key fold). Fallback path unchanged
(stash-verified zero regression). NOTE: montage generation needs GPU verification.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Multi-scene bug: the Movie Editor injects a generic 'scene N' separator before
any scene that lacks a leading transition trigger (build_combined_prompt), purely
to force a boundary. split_scenes was splitting on those markers with placement
'start', which KEEPS the marker word in the new scene's text -> 'scene 1',
'scene 2', ... got encoded into every scene's conditioning. With a text-less
leading i2v scene this also produced a label-only first scene (renders as the
anchor with no prompt) and shifted each scene's content down by one.
Fix in the one canonical splitter: a generic 'scene <N>' label (not a real
custom trigger in the transition DB) now cuts SILENTLY -> boundary only, dropped
from both expanded (generation) and raw (editing) text. Real user/DB triggers
keep their configured placement, so 'cut'/'qcut' etc. are unaffected.
Regression test asserts the markers split but never appear in any scene's
expanded/raw text, and that a real trigger still leads its scene.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
