The prior wording made the frame-index/anchor-eviction issue read as the
reason JoyAI is blocked on H3. It isn't: no JoyAI-Echo LoRA exists for H3, so
it's inaccessible by design regardless of interior pins. The frame-index issue
is real but secondary — it's what would also go wrong if someone forced it on.
Region lock: a keyframe pin can cover part of a frame. Transparent areas of an
i2v anchor are dropped from the packed sequence entirely — not blanked, not
noised, absent — so the model invents them. Row deletion re-derives the layout
from upstream's own output; two patches share one keep-mask so they cannot
disagree. Surfaced through the anchor's alpha, which needs no new wire and works
in the Editor. Excluded pixels are filled with the kept area's mean before the
VAE so no hard edge bleeds in. EXPERIMENTAL: fl2va saw whole condition frames.
Taste bias: the liked direction Studio already learns is now also added AFTER
H3's token refiner, where it lands as sent instead of being mixed across tokens
and normalized away. Relative to the span's own row norm, so one strength means
the same thing on every prompt. How far to push is a fifth learned gain,
centred on 0 and signed; h3_taste_bias is the manual override. Needs 3 liked
runs.
Gain state generalized to per-key centres, bounds and exploration width.
Tests: 1608 passed (+46). Untested on a rental.
The four H3 render strengths shipped as hand-set dials. They should not
have: four scalars is a SMALLER search than the sigma profile, which
already converges on ratings alone, so typing them is the user doing work
the loop can do.
Same ES update as the sigma profile, with one difference — that profile is
centred on 0, so its applied value IS its perturbation; these are centred
on 1.0, so the perturbation is (applied - value) and that is what the
reward is credited to. Exploration narrows on a like, widens otherwise,
and is seeded from the run seed so a repeat repeats.
Learned values reach the sampler through the conditioning meta, the bridge
h3 token weighting already uses: Studio owns the key, the sampler owns the
application. h3_gain_mode defaults to "learned" and ignores the widgets;
"manual" uses them as typed and still works with no Refiner in the graph.
Learning runs on every rating regardless of the toggle, the same rule
value_guidance follows.
Also fixes four note_on_change calls added earlier this session that were
missing the required tag argument — they would have raised on the non-H3
and unreadable-tags paths.
Qwen text reaches the DiT through condition_proj + a 2-block token
refiner ending in an RMSNorm. Everything steered today happens BEFORE
that, so the refiner mixes the edit across tokens and its final norm
renormalizes the magnitude away — what lands is not what was learned.
TokenRefinerEdit patches the refiner's output instead: a scale and/or a
bias over a row range, landing unchanged in the space the 50 blocks read.
project_into_refiner_space maps a conditioning-space direction across with
condition_proj's weight only (a direction is a difference of points, so
the bias would move the origin).
Ships h3_prompt_scale: how loudly the prompt is read through attention,
distinct from h3_gain_prompt (how much each block writes back into those
rows). Confined to the prompt rows via minimax_token_tags, so a wired
reference's vision block is left alone. Free, 1.0 = model not cloned.
The bias path is the mechanism for a learned direction and is not wired to
one yet: a single [hidden] vector added to every prompt row is
prompt-length-independent, which is the shape a learned direction needs
here since Qwen does not pad.
Each H3 block carries AdalnProj(t_dim, hidden, expand=6, modalities=3):
shift/scale/gate for attention and MLP, in three copies indexed
t_row*3 + tag (video 0, text 1, audio 2). The gate scales what a block
writes back into a row range, so scaling one modality's gate is per-block,
per-stream write strength — the only mechanism in this architecture that
separates prompt, picture and soundtrack at the point of writing.
Three sampler widgets, 1.0 = untouched and the model is not cloned. Free:
one vector multiply per block, no extra model calls.
Sampler-side and self-contained — reads no key, rating or learned
direction, so conditioning steering being off does not disable it and the
Refiner need not be wired at all.
Attached with add_object_patch, restored by ComfyUI on unpatch; nothing
survives the run. Pins and reference images carry the VIDEO tag, so the
video gain moves them too — stated on the console and in the tooltip.
It stepped the full distance to sigma_next and then added
sqrt(sigma^2 - sigma_next^2) of noise on top. That is the VP variance
formula, and adding noise without shortening the step leaves the latent
noisier at sigma_next than the schedule defines — so s_noise never behaved
like euler_ancestral on H3 or LTXAV at any value.
Lands short (sigma_down) and renoises with flow matching's alpha rescaling:
the arithmetic _sample_const_rf_full already carried, and comfy's
sample_euler_ancestral_RF. At s_noise=1.0 it IS euler_ancestral.
s_noise now means eta (fraction of the step taken stochastically); range
raised to 1.0 so euler_ancestral is expressible. A value carried over from
before does NOT mean what it did — the tooltip says so. Non-CONST models
keep the eps path unchanged. Audio still never gets ancestral noise.
The Heun corrector evaluates the model at sigma_next, and the terminal step
lands on sigma 0 — degenerate, and it returns early before the corrector is
reached. The window was measured back from the end of the schedule, so it
started on that step: final_correction_steps=1, the default, performed zero
corrections on any schedule ending at 0.
Counted over correctable steps instead, so N means N.
Default moves 1 -> 0: fixing the arithmetic alone would have added a model
call to every existing run (~14% on a 7-step schedule). 0 is what every run
has actually been doing. quality_sharpness shares the window by design.
Upstream PackedLayout computes a pin's time coordinate in two branches and
raises for anything between them. They are the two endpoints of one line:
the packed sequence advances FRAME_RESCALE per PIXEL frame, so
cond_t = text_len + FRAME_RESCALE * pixel_index reproduces both exactly at
every clip length, including the irregular (1,4,4,4,4) latent grouping.
Upstream still builds the layout; this rewrites the time column of the
condition rows it produced. Endpoint-only runs call straight through and are
bit-identical. The rule is re-derived from upstream's own constants before
patching, so a changed frame grid disables the feature instead of misplacing
pins.
Unlocks mid-clip guides on the Chain Sampler, which were refused outright.
EXPERIMENTAL: fl2va trained on first/last pins only, so the coordinate is
representable without being learned. Costs no extra model call.
The Chain Sampler reads one conditioning entry per SCENE
(scene_conditionings = positive[:max_scenes]). When Studio stopped discarding
the entries past the first of a wired CONDITIONING — the fix for a reference
surviving while the prompt did not — those entries reached the sampler
untagged, and the r2v reference block was counted as a scene of its own. The
run sampled the shot twice, which from outside is "the model ran a second pass
for no reason".
Studio tags every pass-through entry as a companion; the sampler excludes
companions when counting scenes and hands them to every scene instead, so they
still reach the model. Both halves of the original bug stay fixed: nothing is
dropped, and nothing is mistaken for a scene.
Also fixes the liked-conditioning blend, which was the last payload path still
doing a raw lerp: _v2_shape_compatible ignores sequence length by design, so a
direction learned at 586 positions met a 494-position prompt, threw, and took
the whole pull toward what the user rated well with it. Routed through
_v2_payload_like like every other payload.
Adds stage timers for the nine learning steps the first pass did not cover —
55% of a 26.9s run was accounted for, and the rest was unnamed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three problems, all visible in the first live run.
x1.03 on every phrase: category scores are 0..1 confidences and a phrase
rarely carries much of one axis, so `1 + strength * score` was applied
and inert. Weights are now relative to the strongest candidate, making
strength mean "how much the best phrase gets".
94 candidates capped to 8 by LENGTH kept an arbitrary eight. Ranking is
by weight; longest-first is now only the application order, so a phrase
still lands before one of its own words adds to it.
The prompt was assumed to be the tail of the conditioning. The same run
reported 367 conditioning rows against 368 modality tags, which is
exactly how that assumption breaks. minimax_token_tags marks text as 1
and vision blocks as 0, and references precede the prompt, so the prompt
is the trailing run of 1s — read it instead of inferring it. The status
line says which of the two placed it.
Studio has known which words to weight since 2.0 —
_v2_concept_units_for_run splits every run into phrases, single words and
n-grams; _v2_update_phrase_memory trains each against the rating's
missing/wrong axes. The only consumer was _v2_build_attn2_patch, and H3
skips it: one packed self-attention stream, no cross-attention. So the
learning ran on every generation and reached nothing.
It now becomes an attention-logit bias on the prompt's token positions in
the packed stream, which is the same statement the K/V x1.25 emphasis was
making, at a place H3 has. Boost only by default.
Located with the tokenizer's offset mapping (verified against ComfyUI's
H3 vocabulary), not by re-encoding each phrase through Qwen3-VL-32B the
way _v2_find_phrase_token_ranges does.
The bias is a mask and SLA routes masked calls to dense, so a weighted
run is a dense run. Said once, at install.
Unreadable modality tags widened the erase to every position including
an image span; a partly-failed erase left some scenes changed; an
unreadable sigma disabled sharpening while still reporting it active.
A malformed reference slot claimed nothing was marked.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Page-locked pages cannot be swapped or reclaimed, so a budget near
total RAM is what wedges a Linux host rather than killing the process.
Warn when ComfyUI's budget exceeds 80% of RAM.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RSS, host RAM, swap, VRAM and what ComfyUI is holding (with how much
sits on the CPU), before and after the decode — the call that evicts
model weights from VRAM into host RAM.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A decode is one uninterruptible call with no progress of its own, so a
slow decode looked identical to a hung one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Log latent shape, estimated CPU pixel size, free VRAM/RAM, and empty
torch's cache before decoding — a decode that kills the process leaves
no traceback. Sharpness proxy releases its retained prediction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pass 2 can run a different algorithm. Unwired it reuses pass 1; in the
Editor it is behind Second pass > Own sampler, and Studio mirrors the
high pass into that output when off, so existing projects are unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Lift the unsharp out of the sampler loop through a denoiser proxy, the
same way ALG does. Window is a fraction of the schedule; audio excluded.
Off by default; an unwrappable sampler keeps sampling and says so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both are structurally inert on H3; context windowing also mis-reported the
cause as an old ComfyUI. alg_anchor deliberately kept: a continuation scene
carries real latent frames for it to act on.
H3 keys keyframe pins by frame index and takes only first/last, so memory
frame 0 evicted the anchor and the rest were refused. Forced off with a
reason; hidden in Engine Settings. Tooltips now state the LoRA requirement.
14 more silent no-ops made audible: conditioning memory blend/repel/rollback,
Bounded Attention, ALG blur, guide image load, identity install, wired pins,
shortcut expansion. Both hook-removal leaks now name the restart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One mechanism for both: failed() names the attempt, the error and the effect on
the output, once per run; note_on_change states a standing condition once and
again only when it changes. 13 silent no-ops made audible, 4 repeats silenced.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
H3's shifted schedules keep sigma high to the end, so max(0, 1-2*sigma) gave
the rating wrappers 0 of 4 steps on turbo and 4 of 20 at shift 3 while they
reported active. Read the base grid off the schedule; report the window.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Only the fixed-direction fallback announced its lift, so a working value
function was indistinguishable from nothing happening.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fitted on raw conditioning, it was handed H3's refined tensor and threw every
step; the run fell back to the fixed direction while reporting VF steering.
Take the gradient at the raw cond, lift it, cache it per scene.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
H3 refines conditioning in extra_conds, so c_crossattn is 5376-dim hidden
state while the taste store holds 5120-dim raw text: embed_guidance raised,
score_slider swallowed it and steered nothing. Lift via the model's own
preprocessor; same bridge fixes DynaShift's prompt weighting.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A pin is packed as condition rows, so its token count belongs to the grid it was
encoded on and upscale_2x crashed the model placing it. LTX guides are dropped
here; on H3 the pin is the anchor, so it is brought along instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
H3's resizer interpolates to a requested size and takes 1.0-4.0; Lightricks' is a
fixed 2x network and now says it ignored the value instead of rounding quietly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both did LTX latent surgery unguarded: prepending frames off H3's 5k+2 grid and
attaching conditioning it never reads. Carry now pins the original reference at
frame 0; a mid-clip guide declines by the same rule as every other H3 pin.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
H3 has no latent conditioning, so a carried tail was noise it could overwrite:
the seam matched and the rest of the shot knew nothing about the scene before it.
The previous scene's last latent frame becomes this scene's frame-0 pin.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
v2a scale and bounded attention install on attn2 / video_to_audio_attn, which the
block-only sweep never reached and never tagged. The census prints what is still
installed before sampling, so a count that grows is proof rather than a hunch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both were switched off on H3 because Lightricks' upsampler is 128-channel — a fact about
the file, not the family. They now load any upsampler ComfyUI can (core's loader handles
the rest), refuse a mismatched one by name, and never download the LTX file for H3.
Three unrelated reports.
Wires lost or "(not allowed)" after loading a project. /api/pipeline-ports read the
family from the GLOBAL models.json, so it described whichever project was saved last.
Open one on the other family and its saved wires point at ports that family has no rule
for. It takes ?pid= now, and the panel asks for the config before the rules that depend
on it. Reconcile also dropped a port wire when core_overrides named a deleted slot — it
cleared every other slot's wire to that port, then failed to add the replacement.
Guided mode also filtered harder than the builder validates. A node added through
"Any node…" has role `custom`, which no rule table mentions, so a custom VAE loader
could not reach the pipeline at all. A role with no rule for an output type now reaches
any port of that type; a role that HAS one keeps it, and internal links stay closed.
Both sides read the same list, since the panel hiding what the builder accepts is what
produced "(not allowed)".
Frames / scene silently missed shots. Any timeline resize, trim or split moves a scene
off project length for good, and nothing said so — typing 305 and rendering 450 read as
the edit being ignored. The Project panel now says how many shots it doesn't reach and
offers to put them back. The panel's fingerprint had to learn about scene state, or the
warning outlived the fix.
second_pass_op=upscale_2x no longer drops the i2v pin. The pinned frames in the upscaled
latent ARE the anchor — carried through the same upsampler as the rest of the clip — so
only the mask needed scaling to the new grid. Pass 2 at 4x the pixels was the run most
likely to drift off the reference, and it was the one running unpinned. The pin now
holds the upscaled anchor rather than the encoded source image, and the report says so.
Guide keyframes are still dropped: those are token indices, not pixels.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
second_pass_op="upscale_2x" hands a scene back at twice the latent size while
every LATER scene is still built from latent_template at the original size.
Everything that crosses a scene boundary — carried overlap frames, the anchor's
continuation, the soft join, the JoyAI memory frame, per-scene guide sources —
is spliced into a chunk on the template's grid, so the second scene died on the
shape mismatch and the combination was simply unavailable.
One choke point, _match_template_resolution, brings the carried material back
before it is used, applied at the three places a finished scene crosses into the
next one. Each scene still samples and OUTPUTS at 2x: only what is carried gets
resampled, and only downwards, which is the direction that survives it — those
frames exist to say "continue from here", which a resample preserves far better
than invented detail would.
Identity no-op when the grids already agree, so any run without a resizing op is
bit-identical. Audio has no spatial axes and is never touched. A stale
noise_mask is dropped rather than reapplied at the wrong scale, and a resample
that throws costs continuity with a console line, not the run.
No grid guard on the chain blend: second_pass and second_pass_op are run-level
settings applied to every scene, so a chain cannot hold two resolutions — every
scene is upscaled or none is. Both tooltips said this combination dropped
multi-scene; they now say how it works.
Also: About reads "FunPack 3" with the codename on its own line, rather than
attaching the major to the codename.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The opening frame came out noisy because the cut was made in latent space. The
LTX video VAE is causal: latent frame 0 is the temporal origin and decodes to a
single pixel frame, while every later latent frame was generated as a
CONTINUATION of the frames before it. Slicing the front off the latent promoted
a continuation frame to position 0, and the decoder then applied the
temporal-origin handling that frame was never generated for. H3 already cut
decoded images for unrelated reasons and so never had the bug; LTX now does the
same thing.
The count was wrong too, quietly. A latent cut could only remove whole latent
frames, and the promoted frame changed from covering 1 pixel to covering
`scale`, so dropping k latent frames shortened the clip by k*scale rather than
the (k-1)*scale+1 it actually spanned. The value was also rounded up to a whole
latent frame on the way in. It is now exact: N means N, matching H3.
Each eligible scene records a (start, count) pixel span instead of cutting; the
spans are removed after decode, later ones first so earlier removals cannot
shift their indices. Mid-chain hard-cut scenes keep working — the span is
computed from the scene's position in the chain, which is exact because anchor
scenes append with zero blend overlap. The latent's audio stream loses the same
stretches of TIME, mapped by proportion on its own axis, or sound would run
ahead of picture with no error. The latent's VIDEO stream is deliberately left
whole (cutting it is the bug), so video comes from the IMAGES output — which the
Editor already wires. A run with IMAGES unconnected now skips with a reason
instead of silently doing nothing.
_cut_opening_latent is removed rather than left dead, since its docstring
documented the broken approach as correct.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two unrelated fixes.
**Guide keyframes vs a resolution-changing second_pass_op.** Guide keyframes are
recorded as TOKEN indices into pass 1's latent grid, so upscale_2x — which
doubles each spatial dim and thus quadruples the tokens per latent frame —
leaves them addressing the wrong tokens. Since the LTX-2.5 commit core rejects
that outright ("keyframe_idxs holds N tokens, which is not a whole number of
M-token latent frames"), which is how this surfaced; older cores silently
mis-placed the guides instead, which is worse. Pass 2 now drops them and says
so in the scene report, exactly as segmented detailing already does for its
crop — so detailing's stripper becomes public rather than being duplicated. Only
pass 2 is affected; pass 1 ran at the grid the guides were recorded against and
keeps every one. The pin loss and the guide loss now share one predicate
(_latent_spatial_changed) so they cannot drift apart, and an unreadable shape
answers "no change" so a probe failure never costs a guide.
**One ALG control instead of two.** The Distilled Flow panel had its own
alg_enabled while the chain sampler's alg_anchor did the same guidance on any
sampler and already drove the Distilled Flow one when set — two switches for one
behaviour, with precedence depending on which you had touched. The Editor now
shows only alg_anchor. The node input stays for hand-built graphs; what moved is
the Editor's control. A project still carrying the old switch is migrated once,
strength and threshold included, and told so, rather than left running ALG with
nothing on screen to turn it off.
The migration is deferred out of the render pass and guarded by a module-level
set of project ids: every store write re-renders the pane synchronously, so
migrating inline recursed until the stack blew (caught in the browser, not by
the suite).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
LTX 2.5 reuses LTXAVModel/LTXBaseModel behind config flags, so the samplers,
the identity-transfer wrappers and the guide path all bind to it unchanged.
Two places assumed 2.3 specifics, and both would have gone wrong quietly.
The AV channel split was a hardcoded width table. From 2.5 on comfy reads the
video/audio projection widths off the checkpoint rather than assuming them, so
a checkpoint can ship a pair the table has never seen — and an unrecognised
width made ltxav_video_channels return the FULL width, which silently steers
the audio text context along with the video one. The only symptom would have
been degraded audio. The split is now measured off the live model when one is
on hand; the table stays as the fallback.
Decode-time noise only ever reached VideoVAE.decode. 2.5's diffusion decoder
takes no timestep and seeds itself, but it is an nn.Module, so stamping the
attributes onto it succeeded and was then ignored — a knob that reads as live
and does nothing. It is now capability-tested against the real contract
(hasattr decode_timestep, not a class name) and says so when it cannot apply.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three things that were each locked to one implementation detail:
- steps + schedule were a KSampler-branch feature, though every sampler here
walks a sigma list handed in from outside. They are now pass-level, so
Distilled Flow and Hybrid Euler 2S can run a named schedule (and the second
pass can be computed rather than typed) in both frontends and on the node.
- the project's Frames field snapped to the model's grid on every family. On
H3 that is required, but its arrows moved by 1 and the snap put the value
straight back; on LTX the builder rounds up by itself, so the snap only
fought the user. H3 now steps a whole 17 frames per arrow, LTX keeps what
you type and walks 8k+1 with its arrows.
- ALG lived inside the Distilled Flow step loop, so it was unreachable with a
stock KSampler. Its blurred/sharp swap is decided by the step's sigma, and
sigma is an argument of every model call, so a denoiser proxy drives it from
outside any sampler. New alg_anchor toggle on the Chain Sampler drives the
in-loop version too, so there is one control wherever you are.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Engine settings were unreadable: every experimental knob carried a paragraph
about cost figures, sampler caveats and failure modes, so nobody read any of
it. The hint is now ONE sentence saying what the setting does, and the
paragraph moves behind a collapsed "Details" link — still the only place some
of those numbers exist, just not in the way of the checkbox.
cut_opening_frames also learns MiniMax H3. It could never engage there: H3's
anchor is a keyframe condition row rather than a pinned latent prefix, so the
t2v guard read every H3 scene as anchorless, and its video latent sits on a
5k+2 grid an arbitrary cut cannot land on. On H3 the frames now come off the
head of the DECODED batch instead — exact to the frame — with the latent's
audio stream cropped by the same proportion of time, because audio decodes
from this node's LATENT output and would otherwise start N frames early. The
latent keeps its full video stream (off-grid the moment it is cut) and the
scene report says so.
Fixes a latent bug on the way: extra streams were cropped on dim 2 with the
video stream's frame count. On H3 dim 2 is the stereo channel and time is
last, so that would have cropped the audio to mono and left its duration
untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit claimed the correction could not reach a stock
sampler. That was wrong. It can: the correction moves from the step to
the prediction, because a rectified-flow euler update is
x*r + denoised*(1-r), so scaling the audio part of (x - denoised) by the
same factor lands the audio on its own grid. A proxy denoiser wraps any
KSAMPLER-shaped SAMPLER and does exactly that.
Tying a model call to a step is the real constraint, and it is checked
rather than assumed: the proxy verifies on every call that the sigma it
was handed is this step's scheduled one, and switches itself off with a
console note the moment that stops holding. Samplers that evaluate twice
per step (heun, dpm_2, dpmpp_2s/sde, seeds_2) trip that on their first
corrector call and are left untouched.
Measured against comfy's real samplers with a perfect rectified-flow
predictor, so every number is pure schedule error (audio error as a % of
the stream's range, 4/8/20 steps):
euler 85/38/14 -> 0/0/0 exact at every step count
res_multistep 69/18/1 -> 21/23/15 helped at 4 steps, HURT at 20
dpmpp_2m 69/18/0.4 -> 20/22/14 same
heun, dpm_2 unchanged (proxy stands down)
The lesson worth keeping: this is a first-order integration error, so a
higher-order sampler already absorbs most of it once the schedule is
fine — which explains why res_multistep and er_sde behaved well on H3 at
20 steps while plain-euler paths did not. At 4 steps nothing absorbs it
(everything sits at 68-85%) and only the clock fixes it, which is the
regime turbo LoRAs put you in.
Not gated on any of this: the user picks the sampler, and the tooltip,
the Editor hint and a per-run console note carry the numbers so the
choice is informed. Also corrects two claims from the previous commit —
"no-op above 20 steps" (it is 14% on euler at 20) and "video is
bit-identical" (the streams share one attention sequence).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MiniMax H3 denoises video and audio on two different flow schedules
(shift 12 / shift 3) but hands the sampler a single sigma grid, so the
DiT reconciles them by scaling the audio velocity by d(sigma_a)/d(sigma_v)
measured at the START of each step. Exact in the continuous limit; on a
few-step schedule the tangent badly overshoots the chord it stands in
for. On a 4-step shift-12 grid the final step drives the audio ~2.5x past
where its own schedule puts it, which is heard as distortion — the reason
the turbo-LoRA community ships a custom sampler for it.
h3_audio_clock on the Chain Sampler swaps that start-of-step slope for
the chord that actually spans the step, so audio lands on its own grid.
One scalar multiply per step: no extra model call, no extra memory. The
video stream is untouched. Above ~20 steps the two agree and it does
nothing, so it is aimed at turbo / distilled schedules.
Wired into the two FunPack sampler functions H3 can reach (Distilled
Flow, and Hybrid Euler 2S via its CONST rectified-flow path). A stock
ComfyUI sampler cannot be reached from outside its loop and says so
rather than looking active. Off by default, UNVALIDATED LIVE.
Editor: Settings > Engine > Experimental, plus the mirror-image inert
chip — H3-only settings left on against an LTX pipeline now warn beside
Generate the same way H3-dead settings already do.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two independent, silent breaks made an H3 i2v scene generate from the prompt
alone.
1. The Editor always runs split_by_transitions, and the per-scene conditionings
REPLACE the single entry Studio built — but they were re-encoded with no
reference_image and no h3_references. Both the frame-0 keyframe pin and the
Qwen vision presentation were gone before the sampler looked, even for a
one-scene project. The split now carries the anchor onto scene 0 and the
references onto every scene. Gated to H3: on LTX the anchor rides in the
latent, and adding Gemma3 vision to scene 0 would change existing output.
2. A MiniMax H3 Image to Video node hands its first/last frame pins out on its
CONDITIONING output, which had nowhere to go — the sampler's positive comes
from Studio. New optional h3_keyframes input on the Chain Sampler salvages
them: first frame onto the opening scene, last frame re-indexed onto the
closing scene's own final frame (the source node's frame_count is its own,
and a stale index would land mid-clip where PackedLayout refuses it). The
editor auto-wires it for H3 projects.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mid_scene_guide_strength, the continuity pin/prior/mid-scene strengths, and
the per-guide strength field in the inspector were all clamped to 0.25-0.5 —
the measured audio-safe band, hardcoded as a limit. That band is advice, not
physics: it stays in the tooltips, the clamps go.
Backend clamps in timeline.py, the node input range, the JoyAI memory guide's
0.25 floor, and the three UI spinners all move to 0..1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two things, both from the same debugging session.
v2a_grad_scale is JoyAI's coupling knob and is never installed without
joyai_audio_memory (see the call site), so a left-over value with the feature
off was being reported as an H3 limitation — blaming the model for a knob that
would be equally inert on LTX. Both the sampler's pre-run notice and the
Editor's inert-setting hint now gate on joyai_audio_memory.
The NaN guard only saw the sampler's OUTPUT, which cannot tell "the model
computed garbage" from "the model was handed garbage and propagated it" — and
those have disjoint suspect lists. _sample_chunk now checks the incoming latent
and the schedule first, and says which it was. The schedule check is there
because it is hand-typed and our solvers divide by sigma (er = sigma_next /
sigma): an interior zero or a repeated value is an instant Inf that poisons
every value downstream. A trailing zero stays legal — it is only ever a step's
target, never a divisor.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A chunk that samples to completion and returns NaN/Inf currently sails through
everything: the scene blend spreads it into the already-finished scenes, the VAE
decodes it, and the first thing that objects is ffmpeg's AAC encoder — "Input
contains (near) NaN/+-Inf", naming nothing that produced it, after the whole
montage has been paid for. The video half never objects at all: NaN through
astype(np.uint8) is undefined but silent, so it degrades with a warning.
_sample_chunk now checks the returned latent and raises with the run-phase label
("scene 2/3 · pass 1 of 2"), which stream, how many values and what dtype, plus
the causes worth testing in order — mismatched LoRA, a mis-inferred checkpoint
layout, a dtype the model cannot hold. It raises rather than warns because a
non-finite latent is unrecoverable and blending it forward destroys the scenes
that were still good. One isfinite() per chunk against a multi-second sample is
not a cost worth weighing; a latent the helper cannot read is ignored, never
fatal.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
H3 support was written against ComfyUI's PR; it has since merged (57500fc5, shipped in
v0.30.0) and the weights are out. Everything the adapter assumed still holds — audio time
axis, 17k+5 grid, batch-1 forward, the imported optimized_attention symbol, every payload
key — so this is what the release itself changed, not a correction of the reading. LTX-2 /
LTXAV is untouched throughout: a project picks a family and both are first-class.
* i2v anchors now work on H3. LTX writes the image into the starting latent
(LTXVImgToVideoInplace); H3 has no such path — its anchor is a condition row packed
beside the text — so a scene's anchor image becomes a frame-0 keyframe pin instead, and
the LTX node is no longer run against an H3 latent. Studio hands the source image's
pixels to the sampler (it owns the CLIP, the sampler owns the VAE). Pins now MERGE, so
a first-frame anchor and a last-frame guide coexist: H3's own first-and-last mode.
* Frame geometry follows the model: frames per scene snap to the family's grid (LTX 8k+1,
H3 17k+5) in both frontends, in the graph builder, and in the sampler, and H3 renders at
its fixed 24 fps. Picking a family moves the project onto its geometry and says so.
* The two checkpoints are named. fl2va (anchors) and ref2va (references) load identically
and never reject each other's conditioning, so the run reports which mode it is in and
the setup panel spells out the split.
* Pipeline setup is no longer a placeholder: released, with the published filenames.
* Reference images can be encoded at the reference pipeline's 2048px short edge.
Fixes found on the way:
* A video reference lost its soundtrack between Studio and the sampler — Studio had
already announced it to the text encoder as <Audio 1>, so every later audio ordinal in
the prompt pointed one reference earlier than written.
* Easy Gen never knew it was on H3: its model config lives on the project, which the
shared capability check did not read, so every H3 warning silently reported LTX.
* An off-grid scene length failed with arithmetic instead of an explanation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The graph builder is now parameterized by model family instead of assuming LTXAV. H3
keeps the core's shape but three of its nodes do not apply: LTXVConditioning (stamps LTX's
frame_rate onto conditioning, which H3 has no field for), LTXVConcatAVLatent (H3's own
EmptyMiniMaxH3LatentAV emits both streams, so the latent feeds the sampler directly) and
LTXVAudioVAEDecode (reads an output_sample_rate H3's audio VAE does not define — it would
raise). Requirements, wirable ports, role targets, guided-wiring hints and the port picker
all follow the family, so a loader can no longer be wired into a node the graph never emits.
The family is a Models setting, asked and never guessed from a checkpoint filename: a wrong
guess fails deep inside ComfyUI at generate time instead of in the panel that can fix it.
Video references now work, rather than being refused. FunPack owns the pipeline and already
requires ffmpeg for preview/export, so minimax_h3 decodes reference clips itself — once, at
24 fps — and derives both views from that single decode: the 2 fps timestamped frames Qwen
sees, and the full-rate latent block the DiT packs. A clip's own soundtrack rides with it,
presented as its <Audio j> label immediately before its <Video k>, which is what ties the
two together. Capped at 15s because reference tokens ride every sampling step.
Editor UI: a Reference media view (H3 only — there is no LTX equivalent to grey out) with
per-type 1-based tags shown exactly as the prompt must spell them, reorder controls that
renumber both sides at once, and a soundtrack picker on video entries. Warning chips beside
Generate for the LTX-only sampler settings H3 switches off, so that is known before a
generation is spent rather than after; Easy Gen gets the same warnings on its strip and the
same family selector (it already loads models.js and pipeline_caps.js).
Verified live in the Cutting Room: the family toggle flips requirements, ports and hints
both ways and persists per project; tags renumber on reorder; the H3 chips replace the
Best-FaceID chip on H3 and hand back to it on LTXAV.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
H3 packs reference blocks into the sequence itself (own position grid, never denoised,
never rendered), which is the thing Best-FaceID approximated on LTX with overlap tokens
plus a trained ArcFace projector. On H3 there is nothing to project and nothing to slice
back off, so this is the model's own feature, not a port of that one.
A reference has to reach the model twice and in the same order both times: as
presentation (Qwen sees "<Picture 1>: <vision block>" so the prompt can name it) and as
a packed latent block. Those encodes happen in different nodes, so the ordered spec
travels between them:
- Studio owns the CLIP: resolves the reference list, bakes the presentation via the
tokenizer's minimax_ref_items, and records what it presented as funpack_h3_refs on
the conditioning. Reference media and a first-frame anchor are separate conditioning
modes, so source_image steps aside (loudly) when references are present.
- Chain Sampler owns the VAE: encodes exactly that list, in that order, sized against
the scene's canvas, and attaches minimax_refs. Applied on every scene branch so an
identity holds across the whole chain, and cached per run so it is encoded once
rather than once per scene.
Order is the contract — drop or reorder an entry and "<Picture 2>" points at a
different reference than the text encoder saw. Anything that cannot be encoded is
dropped AND named, and an audio reference lost for want of an audio_vae says out loud
that later <Audio j> labels have shifted.
New optional audio_vae input on the Chain Sampler (audio references only; image
references need just the main vae). Video references are refused with a reason rather
than half-implemented — they need 2 fps frame sampling and a video loader FunPack does
not own.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Day-0 groundwork for Comfy-Org/ComfyUI#15224 (MiniMax H3). New minimax_h3.py is the
one place that knows how H3 differs from LTXAV; everything else consults it.
Four divergences that failed silently before:
- Audio latent axis. LTXAV audio is [B,C,T,freq] (time on dim 2, same as video);
H3 audio is [B,32,stereo,T]. Both 4-D, so only the model family can say which
axis is time. The Chain Sampler now carries a per-stream time axis through
continuation, blending, tail crops and JoyAI audio memory.
- Frame grid. LTXAV 8k+1 -> k+1; H3 17k+5 -> 5k+2. H3's downscale_index_formula
is the index map, not the count map, so the old uniform formula under-counted
by ~20%. Frame counts now come from vae.downscale_ratio[0], which reproduces
LTXAV's answer exactly.
- Attention patches. LTX calls optimized_attention through its module; H3 binds
it at import, so FunPack's temperature/capture patch was inert on H3. It now
rebinds every module holding a reference, and position-indexed work (BachVid
K/V, attn importance) sits out H3's pre-reshaped [1,heads,S,D] layout.
- Batch size 1. H3's DiT refuses a batched forward and comfy batches cond+uncond
for cfg != 1.0. install_batch_split runs them one at a time, keeping cfg live.
Guides become keyframe pins: H3 packs condition rows into the sequence instead of
appending a masked latent frame, so no tail is appended and none is cropped. The
layout only accepts a first/last pin, and a mid-clip request is refused up front
instead of raising inside PackedLayout.
Studio: reference images route through Qwen3-VL's images= kwarg (the Gemma3 vision
probe returns False for H3 and was dropping them); the attn2 K/V direction patch
reports that H3 has no cross-attention to hook instead of installing a no-op.
Features that depend on LTX transformer structure (bounded attention, Best-FaceID
identity transfer, v2a_grad_scale, segmented detailing, second-pass latent ops) are
now switched off explicitly on H3 with the reason printed, rather than each one
discovering its own missing attribute mid-scene.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
