Found by asking how CFG=1.0 reaches the model. It reaches it through
comfy.sample.sample_custom -> sampling_function, which sets uncond_ = None at
1.0 so the negative branch is never evaluated. Nothing wrong there.
But the causal lane bypasses sample_custom entirely, and sample_custom's
prepare_sampling is also what calls load_models_gpu — which is in turn what
applies a patcher's OBJECT PATCHES (partially_load -> patch_model). So on that
path the weights could still be offloaded AND every object patch FunPack
installs would silently not be there: the per-modality AdaLN gains and the
token-refiner edit (prompt scale + learned taste bias) would all have been
absent, with nothing reporting it.
Loaded once for the whole rollout, before the first forward — not per chunk,
which would fight Comfy's own offload decision all the way down the clip. A
failed load degrades to whatever was already resident and says the patches may
be missing, rather than turning a placement problem into a dead run.
Tests: 1719 passed (+5).
A mode on the existing sampler, not a new node. h3_causal_chunks generates the
clip in time chunks that each attend to a KV cache of the ones before them, so
a long clip stays one continuous shot with real memory instead of separate
scenes stitched together. Sink and window are exposed as what they actually
are: chunks pinned from the start, and recent chunks still remembered.
Two defects found while auditing the bridge I had sketched earlier (which a
blanket `git add -A` had swept into an unrelated commit — it was dead code, but
it should not have gone in under that message):
- it read request.video_channels / audio_channels, which do not exist on
RAVEN's LatentRequest. First real call would have been an AttributeError.
- it handed BOTH streams the same sigma. H3 denoises audio on its own shifted
grid (3 against video's 12), so that is exactly the error h3_audio_clock
exists to correct, and it bites hardest on the few-step schedules this lane
is built for. Audio now goes through the same remap, transcribed from
comfy.ldm.minimax.model.time_shift_sigma and verified equal to the live
function across the range.
The lane REPLACES the sampler rather than wrapping it, so every reason it
cannot run falls through loudly: not H3, pack not installed, stock
bidirectional model. A rollout that fails mid-way raises rather than quietly
falling back to ordinary sampling, which would return a clip under a mode the
user believes ran. The run states that the wired sampler and CFG are unused on
this path, and that there is no per-chunk redo.
Editor: Settings > Engine > Sampler > Experimental, hidden entirely off H3.
Tests: 1714 passed (+18). Never run on hardware.
81 uses of --font-mono, two stray var(--mono, ...) fallbacks that were always
resolving to raw monospace because that token was never defined, and the boot
error banner's inline font. All now Inter. The --font-mono token is deleted
rather than repointed at a sans stack — a name saying "mono" that isn't would
be a trap for whoever reads it next.
JetBrains Mono is also dropped from the Google Fonts request. It was still
being fetched on every load for a family nothing used.
Kept the one thing monospace was earning: digits that do not jitter. A timecode
or countdown re-laying itself out every frame is why readouts reached for a
mono stack at all, so those selectors get font-variant-numeric: tabular-nums.
Inter carries tabular figures, so the alignment survives without the typewriter.
Courier New stays in the video text-overlay font picker. That is a creative
choice burned into the output, not interface chrome.
Verified in a browser against the real stylesheet: every sampled element
resolves to Inter, numeric readouts report tabular-nums.
The Movie Editor's builder forces split_by_transitions on for EVERY project, and
the per-scene conditionings replace the base entry — so enhancing the base
prompt as well was a second full generation whose result was then discarded. At
the ~59s per call measured on the rental that is a minute of pure waste per run,
and it looked like thinking mode being stuck on.
It was not thinking: that defaults off and is passed explicitly, so it can never
inherit the repair advisor's own default of True. It was two code paths both
doing the work.
Now the base is enhanced only when the run does NOT split, and the skip says so
on the console rather than being silent. One generation per prompt that is
actually encoded.
Tests: 1696 passed (+6).
The 2000-character cap was wrong. H3 has no padding, so every prompt token is a
real sequence row and a long prompt is a legitimate thing to want — an enhanced
one especially. Cutting at a fixed length contradicted the token limit the user
had just set.
Length cannot tell "long and detailed" from "stuck". What distinguishes them is
that a model which has stopped writing starts repeating itself, so that is what
is detected now: three occurrences of the same sentence and the prompt is cut
at the first repeat. A long prompt that never repeats is never touched.
The only remaining length bound is the one the user's own token limit already
implies (~4 chars/token), so there is one control rather than two fighting.
Short recurring lines are exempt — "She smiles." legitimately recurs in a shot
list.
Tests: 1690 passed.
Reported from a rental: with the length limit raised to 8192 the enhancer used
all of it. The H3 encoder has no trained decoder head — ComfyUI falls back to
tied word embeddings — so it may never emit a stop token, and max_length stops
being a ceiling and becomes the actual length. Nothing downstream noticed: it
just encoded thousands of tokens of rambling as the prompt.
Now capped at 2000 characters and cut back to the last COMPLETE sentence, never
mid-word, and the status says the model did not stop on its own.
Also reported: it seems to run on CPU. Every generation now reports the device
it executed on, how long it took and the cap that applied — a 32B encoder
sharing a card with the DiT can be placed on the CPU by ComfyUI's loader, which
is minutes per call and invisible from the returned text.
Tests: 1685 passed (+9 enhancer).
RAVEN's contribution is not a sampler, it is a way of shaping the sequence: cut
the clip into time chunks, carry each to completion, commit it into a KV cache
as clean context, generate the next one attending to that cache. The model gets
real memory of what it already drew, so a long clip stays one continuous shot
instead of a stitched chain.
That shaping is separable from the 4-step consistency sampler it ships with,
and this keeps them apart so each can be dissected:
- the chunk cut and the cache are RAVEN's (prefill_text / forward_chunk);
- the SCHEDULE is ours — any scheduler, any step count, bong_tangent included;
- the STEP RULE is a knob: consistency (what the LoRA is distilled for),
plain flow euler, or rectified-flow ancestral. So a chunked run can be
compared against an unchunked one without two variables moving.
Audio chunk boundaries are derived from the shared clock (40 latents/s against
a fixed 24 fps, 17 frames per chunk), not from a hard-coded 28/29 cadence — the
cadence comes out as a consequence and stays right if the grid moves. Verified
against the real H3 grid: 192 frames -> 12 chunks, 320 audio latents, matching
RAVEN's own validated run.
How i2v/r2v survives: `known_chunks` leading chunks are taken as given rather
than sampled, but are still COMMITTED. Chunk 0 comes from FunPack's ordinary
dense path — where pins, references and region locks already work — and only
then enters the cache. The causal lane is never asked to model conditioning
rows it has no layout for, which is exactly what RAVEN's own sampler refuses.
Every model interaction arrives as a callable, so the loop is tested with no
ComfyUI, no RAVEN package and no weights.
Core + 39 tests. Sampler wiring and the rental run are NOT done yet.
Runs after shortcuts and after $variables resolve, before encoding — the model
is handed exactly the text that would otherwise have been encoded, and the
result is what ComfyUI generates from, so nothing downstream knows it happened.
Not the repair advisor: that one is triggered by a rating and fixes a named
fault. This runs on every generation, needs no key, no rating and no previous
run, and only elaborates. Uses advisor_clip when one is wired (the Studio node
already has the socket), the generation CLIP otherwise.
Multi-scene enhances each scene separately. The editor's scene list is
authoritative on count, so rewriting the scenes as one paragraph and
re-splitting could come back with a different number and desync every clip from
its anchor. One generation per distinct text, cached within the run.
Every failure path returns the ORIGINAL prompt — an enhancer that empties the
prompt on a bad generation would render a blank video and blame the model.
Thinking traces, code fences, preambles and whole-answer quotes are stripped;
quotes inside the prompt survive, because speech is quoted on purpose.
Settings > Engine > Studio: toggle, editable system prompt (new textarea field
kind), length limit, temperature, top-p, thinking. Off by default.
Tests: 1637 passed (+24).
Reported from a rental: manual gain mode with the refinement keys deleted died
with "'TokenRefiner' object has no attribute 'inner'". Learned mode was fine,
because with no key the gains are neutral and the installer returns before
building anything — so only manual, which ignores keys, ever built the wrapper.
Wrapping a module in an nn.Module renames every weight beneath it
(token_refiner.blocks.* -> token_refiner.inner.blocks.*). patch_model applies
object patches BEFORE loading weights, so the names were recorded WITH .inner;
unpatch_model restores weights BEFORE unwrapping, and with several patchers
alive off one shared model — one clone per scene, plus one for the AdaLN gains
— another patcher unwraps first and the restore walks .inner on a real
TokenRefiner.
Both wrappers now patch `<module>.forward` through the same object-patch
mechanism instead of replacing the module, so the tree and the state dict stay
exactly as the checkpoint built them. Neither is an nn.Module any more, and
each captures the original bound method rather than re-reading `.forward` at
call time — ComfyUI restores a patch by writing the previous value back, which
can be a live wrapper, so re-reading would nest a scale per scene.
Reproduced against faithful patch semantics before and after. AdaLN gains had
the identical latent hazard and are fixed the same way.
dependsVals (an array) is the Studio-field convention; sampler knobs read a
singular dependsValue and fall back to a plain truthy check when it's absent —
which a non-empty combo string always passes. All five dials showed regardless
of h3_gain_mode. Switched to dependsValue: "manual" to match the mechanism
that's actually there.
The prior wording made the frame-index/anchor-eviction issue read as the
reason JoyAI is blocked on H3. It isn't: no JoyAI-Echo LoRA exists for H3, so
it's inaccessible by design regardless of interior pins. The frame-index issue
is real but secondary — it's what would also go wrong if someone forced it on.
h3_gain_mode and its five dials existed only as Chain Sampler widgets, so the
Editor could turn the learning on but never take it manual. They now sit under
Sampler > Experimental, hidden off H3 by the same rule as h3_audio_clock, and
the five dials depend on manual mode so learned mode shows one control.
Audited every Chain Sampler and Studio widget against Engine Settings while
here. One other real gap: carry_overlap_through_anchor was reachable from no
frontend at all. Added under Continuity.
Everything else unexposed is owned elsewhere by design (timeline, rating
picker, Models window, Batch Training) or is data plumbing, not a setting.
Region lock: a keyframe pin can cover part of a frame. Transparent areas of an
i2v anchor are dropped from the packed sequence entirely — not blanked, not
noised, absent — so the model invents them. Row deletion re-derives the layout
from upstream's own output; two patches share one keep-mask so they cannot
disagree. Surfaced through the anchor's alpha, which needs no new wire and works
in the Editor. Excluded pixels are filled with the kept area's mean before the
VAE so no hard edge bleeds in. EXPERIMENTAL: fl2va saw whole condition frames.
Taste bias: the liked direction Studio already learns is now also added AFTER
H3's token refiner, where it lands as sent instead of being mixed across tokens
and normalized away. Relative to the span's own row norm, so one strength means
the same thing on every prompt. How far to push is a fifth learned gain,
centred on 0 and signed; h3_taste_bias is the manual override. Needs 3 liked
runs.
Gain state generalized to per-key centres, bounds and exploration width.
Tests: 1608 passed (+46). Untested on a rental.
The gains read only the scalar reward, so "Missing concept" — reward +0.10
because quality was fine — nudged every gain slightly TOWARD the run that
missed the prompt, and taught the audio gain something about a complaint
that had nothing to do with audio.
Every rating already carries three per-axis signals in [-1, +1]. Route
them: concept -> prompt + prompt_scale, detail/quality -> video, and audio
keeps the overall reward because no rated axis covers it.
The combiner is min, not mean. These axes are conjunctive — the video gain
has to satisfy detail AND quality — so a satisfied axis must not mask a
failed one. Averaging let it: "Missing details" came out at -0.08 for the
gain that governs detail. It is -1.00 now.
Recovering four parameters from one scalar worked, but only by waiting for
random perturbations to decorrelate. Each rating now teaches the parameter
it names.
The four H3 render strengths shipped as hand-set dials. They should not
have: four scalars is a SMALLER search than the sigma profile, which
already converges on ratings alone, so typing them is the user doing work
the loop can do.
Same ES update as the sigma profile, with one difference — that profile is
centred on 0, so its applied value IS its perturbation; these are centred
on 1.0, so the perturbation is (applied - value) and that is what the
reward is credited to. Exploration narrows on a like, widens otherwise,
and is seeded from the run seed so a repeat repeats.
Learned values reach the sampler through the conditioning meta, the bridge
h3 token weighting already uses: Studio owns the key, the sampler owns the
application. h3_gain_mode defaults to "learned" and ignores the widgets;
"manual" uses them as typed and still works with no Refiner in the graph.
Learning runs on every rating regardless of the toggle, the same rule
value_guidance follows.
Also fixes four note_on_change calls added earlier this session that were
missing the required tag argument — they would have raised on the non-H3
and unreadable-tags paths.
Qwen text reaches the DiT through condition_proj + a 2-block token
refiner ending in an RMSNorm. Everything steered today happens BEFORE
that, so the refiner mixes the edit across tokens and its final norm
renormalizes the magnitude away — what lands is not what was learned.
TokenRefinerEdit patches the refiner's output instead: a scale and/or a
bias over a row range, landing unchanged in the space the 50 blocks read.
project_into_refiner_space maps a conditioning-space direction across with
condition_proj's weight only (a direction is a difference of points, so
the bias would move the origin).
Ships h3_prompt_scale: how loudly the prompt is read through attention,
distinct from h3_gain_prompt (how much each block writes back into those
rows). Confined to the prompt rows via minimax_token_tags, so a wired
reference's vision block is left alone. Free, 1.0 = model not cloned.
The bias path is the mechanism for a learned direction and is not wired to
one yet: a single [hidden] vector added to every prompt row is
prompt-length-independent, which is the shape a learned direction needs
here since Qwen does not pad.
Each H3 block carries AdalnProj(t_dim, hidden, expand=6, modalities=3):
shift/scale/gate for attention and MLP, in three copies indexed
t_row*3 + tag (video 0, text 1, audio 2). The gate scales what a block
writes back into a row range, so scaling one modality's gate is per-block,
per-stream write strength — the only mechanism in this architecture that
separates prompt, picture and soundtrack at the point of writing.
Three sampler widgets, 1.0 = untouched and the model is not cloned. Free:
one vector multiply per block, no extra model calls.
Sampler-side and self-contained — reads no key, rating or learned
direction, so conditioning steering being off does not disable it and the
Refiner need not be wired at all.
Attached with add_object_patch, restored by ComfyUI on unpatch; nothing
survives the run. Pins and reference images carry the VIDEO tag, so the
video gain moves them too — stated on the console and in the tooltip.
It stepped the full distance to sigma_next and then added
sqrt(sigma^2 - sigma_next^2) of noise on top. That is the VP variance
formula, and adding noise without shortening the step leaves the latent
noisier at sigma_next than the schedule defines — so s_noise never behaved
like euler_ancestral on H3 or LTXAV at any value.
Lands short (sigma_down) and renoises with flow matching's alpha rescaling:
the arithmetic _sample_const_rf_full already carried, and comfy's
sample_euler_ancestral_RF. At s_noise=1.0 it IS euler_ancestral.
s_noise now means eta (fraction of the step taken stochastically); range
raised to 1.0 so euler_ancestral is expressible. A value carried over from
before does NOT mean what it did — the tooltip says so. Non-CONST models
keep the eps path unchanged. Audio still never gets ancestral noise.
The Heun corrector evaluates the model at sigma_next, and the terminal step
lands on sigma 0 — degenerate, and it returns early before the corrector is
reached. The window was measured back from the end of the schedule, so it
started on that step: final_correction_steps=1, the default, performed zero
corrections on any schedule ending at 0.
Counted over correctable steps instead, so N means N.
Default moves 1 -> 0: fixing the arithmetic alone would have added a model
call to every existing run (~14% on a 7-step schedule). 0 is what every run
has actually been doing. quality_sharpness shares the window by design.
Upstream PackedLayout computes a pin's time coordinate in two branches and
raises for anything between them. They are the two endpoints of one line:
the packed sequence advances FRAME_RESCALE per PIXEL frame, so
cond_t = text_len + FRAME_RESCALE * pixel_index reproduces both exactly at
every clip length, including the irregular (1,4,4,4,4) latent grouping.
Upstream still builds the layout; this rewrites the time column of the
condition rows it produced. Endpoint-only runs call straight through and are
bit-identical. The rule is re-derived from upstream's own constants before
patching, so a changed frame grid disables the feature instead of misplacing
pins.
Unlocks mid-clip guides on the Chain Sampler, which were refused outright.
EXPERIMENTAL: fl2va trained on first/last pins only, so the coordinate is
representable without being learned. Costs no extra model call.
The picker is opened from a button in the timeline but mounts on document.body,
so moving onto it left the zone and the timeline shut underneath the thing that
had just been opened from it.
TimelinePeek gains hold(reason) / release(reason). A hold outranks the pointer:
whatever is holding it open is on screen because of the timeline. Reasons
rather than a boolean so two holders cannot release each other's hold, and a
hold taken while the timeline is shut opens it — the picker can be reached from
the strip's own row.
Any other panel that belongs to the timeline but lives outside it can use the
same two calls.
Verified in the browser: open by hover, hold, pointer leaves — stays open;
release — closes; hold while shut — opens; release — closes; turning the
preference off clears the holds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Crossing the strip on the way to the Dock opened the timeline for a frame. It
now waits 220ms with the pointer still on it before opening, and leaving
CANCELS a pending open rather than queueing a close — so a pass-through costs
nothing at all. A drag skips the wait entirely: dragging a file onto the
timeline is not an accident.
The swap animates. That meant giving up display:none for the closed state,
since display cannot be transitioned: closed is height 0 with overflow hidden,
which looks identical at rest — no sliver of lane, no cut-off clip — and on the
way open the same clipping is simply what a reveal looks like. The strip and
the timeline animate against each other so the zone's height stays continuous
rather than jumping. Reduced motion drops the slide and keeps the behaviour.
Verified in the browser: a touch-and-leave does not open it, staying does and
only after the delay, leaving closes it, and the transition is on the body.
Nine unit tests cover the intent timer with injected timers, including the two
that would bite — a drag cancelling a pending hover, and moving around inside
an already-open timeline not re-arming the delay.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collapsing the body to 18px showed the top of the real timeline — a lane label
and a cut-off clip, with dead space under them. It read as a broken timeline
rather than a closed one.
Closed is now display:none on the body and a strip that says "Timeline — hover
to show". Nothing of the timeline is visible while it is shut, so there is
nothing to look cut off and no leftover height.
Hover is driven from script rather than :hover: the two halves of "open" are
different elements — the strip you point at and the timeline that replaces it —
and a CSS rule on either alone flickers as the other takes the pointer. The
rule is one predicate: inside the zone and not inside the header. The header
stays inert, so reaching for Generate still does nothing.
The closed body has no width to lay out against, so opening dispatches one
resize (coalesced to a frame) rather than letting clips draw at a stale width
for the first frame.
Verified in the browser: strip present and captioned, body display none/flex
across closed/open/preference-off, and the pointer rules — header button and
header do not open it, strip does, the timeline itself holds it open, moving
back to the header closes it, leaving the zone closes it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It named the three strongest learned phrases as examples. Phrase memory holds
the user's prompt, so every paste of the log carried prompt text out with it —
stripped by hand each time, which is a thing that will eventually be forgotten.
The strongest weight replaces them: it answers the question the examples were
there for (is emphasis doing too much) without quoting anything. Counts stay.
Adds a test on the SHAPE, not on one message — joining phrase text into a log
line is how this would come back.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The header was the strip, so reaching for Generate, a pinned button or the
Composer threw the timeline open under the cursor — the opposite of what the
setting is for. The header now behaves exactly as it does with the setting off:
always visible, never a trigger. What collapses is the body below it, to an
18px strip with a grab-handle, and only that strip opens on hover.
focus-within moved to the body for the same reason: tabbing into a clip should
open the timeline, tabbing onto Generate should not.
Also fixes a real bug found while checking this in the browser: `flex: 0 0
var(--timeline-strip-h, 18px)` resolved flex-basis to 0%. var() inside a
shorthand is substituted at computed-value time and the whole declaration is
discarded if it cannot be resolved. Written as longhands, flex-basis resolves
to 18px as intended and a bad custom property can only cost the basis.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"Training is slow" and "training is slow ON THE CPU with four million
parameters" are different problems with different fixes, and nothing in the log
told them apart — leaving it to be inferred from timings, which is how the
first guess at this went to the disk instead.
One line per run per value function, naming the device and the parameter count.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Off by default. On, the timeline sits as a strip — its header, so the title,
transport and Composer stay reachable — and opens to full height while the
pointer is on it or something inside it has focus.
It also opens for the whole of a DRAG. During one the pointer is in drag mode
and a strip that stays shut is a drop target you cannot hit, so dragenter holds
it open and drop/dragend closes it. The tracker counts depth rather than
holding a boolean: dragleave for the element being left fires AFTER dragenter
for the one being entered, so a flag closes the strip mid-drag as the pointer
crosses from one clip to the next.
The strip height is MEASURED from the header rather than fixed: .zone-head
wraps when the zone is narrow (34px min-height, 138px on a normal window), so
any constant clips the very controls the strip exists to keep visible. It is
re-measured on resize. flex: 0 0 auto matters — .zone is a shrinkable column
child, and a height with min-height: 0 let the workspace squeeze the strip to
1px. The body collapses rather than being display:none so the lanes are not
torn down and rebuilt on every open.
A browser preference like the colour scheme, not a project one, and registered
in the same Settings section for that reason.
Also in this commit:
- vf: train wherever there is a GPU, not only when the sample sits on one. The
x0 value function's snapshot loads with map_location="cpu", so keying on the
sample left it training on the CPU — the run that named it measured
"x0 vf: train 10.1s" against 46ms for the same net locally.
- The stage line counted nested sub-stages twice and reported 153% of a run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured rather than assumed. The three value functions are ~16s of a 26.5s
rated run, and the phase that costs it is the training, not the disk:
text cond, dim 5120 3,982,467 params 16.2MB load 19ms TRAIN 1068ms save 26ms
x0 latent, dim 512 221,955 params 0.9MB load 3ms TRAIN 46ms save 2ms
The nets are tiny in FLOPs and not in parameters — a 5120-wide input makes the
first layer 94% of the weights — and twenty Adam steps over that ran on the
CPU because the checkpoint loads with map_location="cpu". Roughly 1.1s here,
about 7.5s on the rental's vCPU, three times per rated generation.
The conditioning is already on the GPU when there is one, so training moves
there and moves back before returning: the buffer and the checkpoint stay on
the CPU so the file remains portable, and inference elsewhere still finds a CPU
module. Adam's state does not follow module.to(), so it is dropped on the move
rather than migrated — it is rebuilt lazily and never persisted, so a run
starts from fresh moments either way.
Nothing about what training does changes; only where it runs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both value functions cost about eight seconds a run, and one has ten times
fewer parameters than the other — so it is not the training math, and the whole
per-call cost is somewhere else. The key file writes at roughly 3 MB/s on the
rental, which makes the checkpoint write the first suspect, but that is a guess
until the three phases are timed apart.
Also times loading the x0 snapshot, which is a full packed latent read from the
same disk.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Chain Sampler reads one conditioning entry per SCENE
(scene_conditionings = positive[:max_scenes]). When Studio stopped discarding
the entries past the first of a wired CONDITIONING — the fix for a reference
surviving while the prompt did not — those entries reached the sampler
untagged, and the r2v reference block was counted as a scene of its own. The
run sampled the shot twice, which from outside is "the model ran a second pass
for no reason".
Studio tags every pass-through entry as a companion; the sampler excludes
companions when counting scenes and hands them to every scene instead, so they
still reach the model. Both halves of the original bug stay fixed: nothing is
dropped, and nothing is mistaken for a scene.
Also fixes the liked-conditioning blend, which was the last payload path still
doing a raw lerp: _v2_shape_compatible ignores sequence length by design, so a
direction learned at 586 positions met a 494-position prompt, threw, and took
the whole pull toward what the user rated well with it. Routed through
_v2_payload_like like every other payload.
Adds stage timers for the nine learning steps the first pass did not cover —
55% of a 26.9s run was accounted for, and the rest was unnamed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The encoder tally answered "is it the encoder" — at 24.9s it is 2% — and the
next question is which learning stage the rest is. 2.4s at three rated samples
and 24.9s at five is not smooth growth, and the shape of that curve says
nothing about which stage draws it.
Times the six that can dominate a run: phrase memory, conditioning memory, the
three value functions, and applying the learned direction. One line, stages
over 0.05s only, costliest first.
Measurement only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One conditioning delta is 586x3584 floats — several megabytes of base64 inside
a document that is parsed and rewritten IN FULL on every generation. That is
the cost that grows with everything ever learned, and it is paid whether or not
any of those tensors are used.
Tensors now go to <key>.json.tensors.pt and the JSON keeps a named reference
whose id is the tensor's path in the tree ("state.global.liked_dir.direction"),
so the key stays readable and a reader can see what a reference points at.
No migration: a key written before this carries inline blobs, is read exactly
as it stands, and moves to the sidecar on its next save. If the sidecar cannot
be written the tensors stay inline — a slow key is fine, a JSON pointing at a
sidecar that does not exist is not. A reference the sidecar has lost reads as
no tensor, so a direction goes unapplied rather than breaking the run.
Deletion already sweeps it: delete_refinement_key globs every <key>.* sidecar
for exactly this reason. Session Reset removes it explicitly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The key is read and rewritten IN FULL on every run and holds base64
conditioning tensors, so its cost grows with everything ever learned — which
is what "2.0s, then 2.4s" looks like from outside. Timing the read and the
write, with the file size and the entry counts of the collections that can
grow without bound, says whether that is where the time goes and which
collection to bound first.
Measurement only. Nothing about what is stored or how changes here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A phrase's vector came from encoding that phrase alone — a full pass of the
text encoder per phrase, and on H3 that encoder is Qwen3-VL-32B. The phrases
are encoded together now, as one joined text, and each vector is the mean of
the rows its own tokens occupy; the tokenizer's offset mapping identifies those
rows exactly, so this is a lookup rather than an approximation.
Each phrase is located at the position it was joined at rather than by
searching the text, because one phrase can be a substring of another and a
search would hand both the same rows.
Falls back to one encode per phrase, unchanged, when the tokenizer offers no
offsets or the spans do not line up with the conditioning that came back. The
cosine step is shared by both paths so a phrase scores the same either way, and
the batched pass is labelled separately in the per-run encoder tally.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CATEGORY_DESCRIPTIONS is a class attribute of eight fixed strings, so the
vectors depend only on the encoder. They were re-embedded every run — eight
full passes of Qwen3-VL-32B on H3 — to arrive at the same eight vectors.
Held for the process, keyed by the encoder, behind a WEAK reference: pinning a
text encoder in VRAM to save eight passes would be the wrong trade. An identity
check sits on top of the id() key because ids are reused once a model is freed,
and a stale hit would classify phrases against another encoder's geometry.
Not a cache of generated content — nothing user-typed goes into it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The encoder is the most expensive thing Studio touches — on H3 it is
Qwen3-VL-32B — and most calls to it are bookkeeping rather than the prompt:
embedding eight category descriptions to classify a phrase by cosine
similarity, embedding a memory to ask whether it is about this scene.
Every call now records its purpose, its cost, and whether the per-run cache
served it. One line per run says where the time went and what share of Studio
it was, so "26 seconds before generation starts" becomes a list of candidates
rather than a guess. The tally is per run and never persisted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Locating a phrase's token range runs the text encoder once per phrase — on H3
that is the same multi-billion-parameter encoder the prompt goes through, up
to eight times. The ranges exist only to feed the attn2 direction patch, and
H3 has no cross-attention for it to hook, so the patch declined on the very
next line and the results were dropped.
The same question is now asked before the work instead of after it. LTX is
unaffected: it reads attn2, so the ranges are still computed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five commits: the unreachable v1 refiner entry point and the Lucky / Into the
Void machinery behind it, the v1 LoRA relevance scorer, filename-based LoRA
relevance, and per-block LoRA application on any model rather than LTX only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Per-block scaling is derived from the LoRA's own deltas — which blocks it put
energy into — so it is a property of the adapter, not of the architecture. Two
things made it LTX-only anyway:
The gate required image_model in {ltxv, ltxav} and mode ltx2, so a model
FunPack had not heard of got the global fallback without anything ever reading
its weights. Whether per-block can run is now discovered from the patches:
fewer than two numbered blocks and the existing fallback takes over, as before.
The key pattern only matched transformer_blocks.N. — MiniMax H3 names its
stack blocks.N., so even with the gate open every H3 patch counted as global
and per-block would have been a silent no-op. The pattern now covers blocks,
transformer_blocks, double_blocks, single_blocks, joint_blocks and layers.
The semantic anchors (PAG 14, STG 14/19) were absolute indices measured on
LTX's 28-block DiT and meant nothing on a stack of another depth. They are
positions now; a 28-block model still resolves to exactly 14 and 19, so LTX
behaviour is unchanged, and a deeper stack gets the equivalent blocks.
Declines, loudly, when a model numbers blocks in two separate stacks
(Flux-style double/single): both start at 0, so one scale would land on two
unrelated blocks.
Untested on a real model — local runs cannot load one. 13 new tests, 11 of
which fail against the previous code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
291 lines: _lora_words, _ensure_lora_memory, _score_lora_prompt_relation and
_update_lora_weight_suggestions. The scorer matched prompt words against each
LoRA's filename; the writer was the v1 half of the suggestion pair, superseded
by _v2_update_lora_suggestions, and unreachable since the v1 refiner entry
point was removed on this branch.
_lora_state_id stays — the live v2 writer still uses it.
Checked: no refs anywhere (grep -a, all file types, node_modules excluded),
not a route handler, not in NODE_CLASS_MAPPINGS, not an INPUT_TYPES widget,
no getattr dispatch. Tests: 1379 passed, baseline plus this branch's two new
ones.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The suggester split each LoRA's FILE NAME into words and matched them against
the prompt; the overlap scaled how far that LoRA's weight moved. That guessed
what a LoRA does from what someone happened to call it, which is a different
question from how much weight it has earned.
Relevance now comes from the type set on the LoRA and the categories present
in the prompt. Weight tuning itself is unchanged — ratings, axis feedback and
the learned offset all behave as before. Behaviour change: a LoRA whose name
echoes the prompt will move less per rating than it used to, and one with an
opaque name will move more.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
32 methods, 1317 lines: the co-occurrence memory, token/pair banks, phrase
placements, embedding canvas and prompt composer. Every entry point was
called only from the v1 refine() removed in the previous commit, or from
another method in the cluster — verified individually after that removal, all
clean. Lucky and Void are one commit because they share the token banks and
call into each other; splitting them would leave a commit referencing a
method that no longer exists.
Into the Void's toggle only ever existed on the unregistered v1 node, so no
UI could reach it. The Lucky TOGGLE stays: the Studio's Lucky is a separate
56-line composer over phrase_memory (_v2_compose_lucky_prompt), unrelated to
this code and still live.
The v2 state schema declares none of these keys, so refinement-key training
never touched them. Tests: 1377 passed, same as baseline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FunPackVideoRefiner is not in NODE_CLASS_MAPPINGS; only FunPackVideoRefinerV2
is, and it overrides FUNCTION, INPUT_TYPES and IS_CHANGED with no call to its
parent. Nothing in the repo calls refine(), so the v1 node contract and its
1263-line body could not execute.
The class itself stays — V2 inherits ~200 live helpers from it. What goes is
the node contract that ComfyUI never reads and the entry point nothing reaches.
Checked: no refs (grep -a, incl. the NUL-byte frontend files), not registered,
not a route handler, no getattr dispatch (the file's only two getattr sites
read literal flags), no JS or workflow-template reference, and the v2 state
schema declares none of its keys. Tests: 1377 passed, same as baseline.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The drop message named three keys and a count with no file attached, and
note_on_change keyed it globally, so in a stack only the first LoRA's shapes
were ever printed. Each file now carries its name into its own log key, and
the count is reported as dropped-of-matched: 51 of 400 is a variant mismatch,
51 of 51 is a file that contributes nothing.
Adds one stack-level line. Every LoRA dropping every weight is not a property
of the LoRAs but of the model they were all measured against, and that is the
reading worth making explicit.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
comfy merges with mm(...).reshape(weight.shape) and adds the result. The
reshape succeeds on any pair with the right element count, so loosening the
drop test to a count comparison let transposed and differently fused deltas
merge in silence. A correct adapter always produces exactly
(weight.shape[0], weight.shape[1:].numel()); anything else fitting by count
corrupts the weight at load and surfaces mid-render as an all-NaN latent.
The two abstentions kept: a locon mid and a padded reshape target are still
left to comfy rather than dropped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"conditioning memory failed (size of tensor a (492) must match tensor b
(499))" — the compatibility gate checks the channel width and the batch
and deliberately not the sequence length, so the payload passed and the
lerp threw. On LTX the encoder pads to a fixed length and this never
happened; Qwen does not pad, so on H3 the length moves with every prompt
edit and the entire learned direction was dropped for that run.
_resize_conditioning_sequence_like was already in the file for exactly
this, with zero callers. Both the blend and the repel now use it, and say
once when a payload had to be resampled.
An exact-length payload is untouched, so the common case — same prompt,
re-rolled — stays bit-exact.
Two bugs, one line apart.
Phrase memory accumulates everything the session has rated, including
phrases from prompts since edited away. The cap to 8 was applied BEFORE
checking whether a phrase is in the current prompt, so those could take
all eight slots on rank alone, match nothing, and leave phrases actually
in the prompt unweighted — while appearing in the status line as
"weighted", which is where they were seen.
And there was no reason for a cap. Locating a phrase here is a substring
find and a scan of the tokenizer's offset map. The 8 was carried over
from _v2_find_phrase_token_ranges, which locates a phrase by RE-ENCODING
it through the text encoder — eight of those is eight Qwen3-VL-32B
forward passes, and the cap is the whole point there.
The status line now names what was APPLIED, and counts how many learned
phrases are in this prompt rather than how many exist.
"<Picture n>: " and "<Audio n>: " are emitted by the H3 tokenizer, not
written by anyone. An AUDIO reference is the awkward one: it contributes
its label and NO vision block, so it lands inside the same run of text
tags as the prompt.
Placement is now anchored at the END of that run rather than requiring
the run to nearly equal the prompt, so any number of label tokens in
front no longer shifts the bias — or refuses it outright, which is what
the slack cap did.
Protection follows the same boundary: the whole tensor is stashed and
everything before the verified prompt start is restored, so a label is no
more steerable than the picture is. Without a tokenizer it falls back to
the end of the vision block — a label may stay steerable, the reference
never does.
An r2v conditioning is laid out reference-first: the "<Picture n>: "
label, the vision block, then the encoded prompt. So the boundary is the
only thing Studio needs — rows before it are the picture and are not
Studio's to touch, rows from it on are the prompt and are.
Protecting by the image tags alone left the label steerable, since it is
text sitting between the two.
prompt_region() is now the single place that walk lives; prompt_base and
choose_encoded_text both read it instead of repeating it.
