Prompt Markup strips it before the encoder; Apply Prompt Markup puts weights
and timed windows on attention and blends on the conditioning. Where the words
sit is the H3 module's (text_tokenizer, prompt_rows, video_time_rows). Wired
into both H3 presets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Loader-level attention option: mean-pool Q and a smoothed K into
blocks, score them with one small matmul, attend only the top
1-sparsity fraction of key blocks per query block, via a vendored
Triton kernel. Runs alongside the chosen attention backend rather than
replacing it (one override slot; the backend becomes SLA's dense
fall-through for the text refiner, masked calls, and trailing dense
steps), and refuses cleanly -- naming why -- on anything that is not
MiniMax H3 or a machine without CUDA+Triton.
modules/loaders/sla_block_map.py and sla_kernel.py are near-verbatim
ports of v4's vendored files (no FunPack-specific imports, so nothing
to adapt); sla_attention.py is the loader-facing glue (H3 gate,
override composition, per-step wrapper), same content as v4's with a
"not a module" note added to match gguf_support.py's convention.
Wired into FunPackDiffusionModelLoader as six new optional widgets,
following v4's loaders.py wiring but expressed in v5's io.Schema style.
The Triton kernel itself is unexercisable on this Mac (no CUDA) --
matches how real generation is verified in this project (rented GPU,
not local). Ported the CPU-testable half of v4's test suite (the
override contract, the H3 gate, the per-step wrapper); the three
kernel-correctness tests carry the same CUDA+Triton skip guard v4 used.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The bridge piece B and C (resolution source, reference images) both need:
Media bin picks have no way into a ComfyUI graph today. One node, media_id
in, IMAGE out, backed by core/media.py's existing store.
Also fixes a real .gitignore bug found along the way: a bare "media/"
pattern matched modules/media/ (this node's own source) as well as the
intended root-level user-data directory. Rooted both media/ and projects/.
v4 could not remove a built-in node from the pipeline and the reason was
structural: the wiring lived in code as a CORE dict nobody could reach, so
"remove this" had nothing to delete and "use a different one" had nothing to
change. Code is not a thing a user can edit.
The pipeline is now data. Three operations over it, each refusing with a
reason rather than half-doing the job:
build turns slots into the prompt ComfyUI queues, checking every link's type
and refusing a graph that cannot work. replace points a slot at any node
producing what its consumers read -- the built-in ones have no special
standing, which is what makes them replaceable rather than merely present.
remove takes a slot out and rewires what it fed to what fed it, when the
node has exactly one input matching each CONSUMED output; v4 required one
for every output a class could produce, so a spare FLOAT nobody wired made a
node unremovable.
Demonstrated against real schemas: FunPack's VAE loader swapped for core's
VAELoader, the modifiers node removed with the sampler rewired to read the
model directly, and removing the empty latent refused because nothing
upstream could feed what it fed.
Then run for real. The default pipeline, with three loader slots replaced by
one checkpoint slot, built from data, accepted by ComfyUI's own
validate_prompt, and executed by its own engine to execution_success and a
picture. Every wiring and type validated on the first attempt; the only
things ComfyUI objected to in the untouched default were the three unset
file pickers, which is correct -- a fresh install cannot know which model
you own.
Added a checkpoint loader, because the split loaders are right for LTX and
H3 and cannot read an SD-family single file at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
v4's Chain Sampler became a hub: 8183 lines, roughly seventy optional
inputs, a _sample_chunk that grew six ALG arguments, and a hardcoded list of
knobs silently inert on one model family. Every technique that needed the
loop was added to it, so it collected all of them.
This one picks an algorithm and a schedule from ComfyUI's own registries --
so a sampler it gains appears here without an edit -- and runs them. Three
things stop it becoming a hub again. It never learns what a model-side
modifier is. Sampler-side ones reach it through ONE call site,
chain.process(ctx, latent), which cannot grow because the step travels in a
context object rather than in the signature. And it declares the hook points
it offers, so core hands it only what fits and it never names a modifier.
Some modifiers genuinely cannot ride on a model: they need the step index,
the schedule, or a second pass. Those attach here instead, and one asking
for a hook this sampler does not offer is absent and said so, not
half-running.
Verified against ComfyUI's own guider path with real weights: bit-identical
output. That took correcting the method, not the code -- see below.
MEASUREMENT, corrected: sampling on this Mac's MPS backend is
NONDETERMINISTIC. The same node with the same seed and the same inputs
differs run to run: 2.42 by max latent difference, 0.0008 by mean pixel gap.
Comparing by MAX said this sampler diverged from core by 14.2 and it was
about to be "fixed"; by MEAN against a measured noise floor it is 0.00000
while core against itself is 0.00183. Every comparison here now runs a
control twice first. Re-checked the Detail modifier the same way: 145x the
floor, so that claim stands.
Also matched what core's sampler actually does and this did not:
fix_empty_latent_channels using the ratios our own empty-latent node reports,
the preview and progress callback, the move back to the intermediate device,
and dropping the ratio keys so nothing rescales twice.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Decode now takes the MODEL, so a provider answers "is this mine" by identity
instead of by shape. H3 was claiming any latent that happened to have two
parts -- right today because it is the only thing producing one, wrong the
moment a second AV model exists, and wrong in the way that reads as an
unrelated fault rather than an error. It declines when the model is not
wired, and the node says that is what is missing.
That completes the shape a model module should have: one folder saying how
this model behaves at each stage -- what it is, how its latent is built, how
its output is decoded -- instead of that knowledge spreading through the
conditioning and sampling code as special cases.
Added a modifier that runs on SD1.5, because a feature nobody can watch is a
feature nobody can check. It declares no traits, so it applies everywhere,
which is the other half of the rule ALG demonstrates by narrowing. Measured
on this machine at a fixed seed: off and on differ, 0.4 gives a visibly
richer picture that still reads correctly, and 1.2 blows the highlights --
so the range is -1..1, set from what was measured rather than from a round
number. A slider whose top half destroys the image lies about its range.
Two things found while building it. ComfyUI has a real circular import
between nodes_latent and nodes_post_processing that resolves only because
init_extra_nodes reaches one first; a custom node has to break it the same
way. And the accumulation bug returned through a third door: stripping
handled keyed wrappers but not the anonymous function lists on
model_options, so chaining doubled the effect. Tagging is now generic -- it
looks for our mark rather than for a list of hook names that would need
keeping current.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Core already turns images into a video and writes the file, and handles
containers, codecs and muxing properly. Writing our own would be a worse
copy of tested code, so there is no FunPack save node and no reason for VHS
either.
What core cannot generalise is decoding a latent that is not one tensor: an
AV latent has two branches wanting different VAEs, and their arrangement is
the model's business. Same rule as the empty latent -- a model module may
claim it, and H3 now does. A nested latent nobody claims is refused rather
than handed to vae.decode, which would raise somewhere confusing or quietly
decode one branch as the picture. A missing audio VAE is named, because
returning no audio looks exactly like a model that makes none.
With this the pipeline closes end to end using FunPack's eight nodes and
core's samplers and video nodes -- verified by loading both together. That
matters more than the node count: until now nothing FunPack produced could
be measured, only reasoned about.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two nodes. One carries the values a person picked, as a single validated
payload rather than a socket per setting -- a socket per setting means this
node's shape changes whenever any module gains an option, and every saved
workflow refers to sockets that moved.
The other reads what announced itself, filters it by what the model actually
is, orders it by declared relations, and installs each one through ComfyUI's
own patcher_extension wrappers. The consequence is checked against comfy's
real prepare_model_patcher and WrapperExecutor rather than against our own
dict: a model that has been through this node runs every per-step modifier
inside ComfyUI's own SamplerCustomAdvanced. The sampler is not consulted, so
it cannot own them.
Installing happens on a clone, and clone() copies the wrapper lists per key,
so the model everyone else holds is untouched. Fifty runs leave zero
wrappers behind. The accumulation that once degraded output across a hundred
generations and survived every reset short of a restart is structurally
impossible here rather than something to remember to undo.
The payload is validated against the same declaration that renders the
panel: an undeclared key, a wrong type, a value outside its own bounds and a
bool passed as a number are all refused, and all problems are reported at
once. v4 validated none of it, which is why it needed a hand-mirrored table
of every default and a list of keys that no longer meant anything.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Derived from latent_format where that is possible, which is every
single-tensor model: SDXL, Wan21 and LTXV all come out matching the numbers
core's own nodes compute, including models that ship after this.
H3 cannot be derived and is the reason modules exist. Its latent is a
NestedTensor of a video and an audio branch, its video channel count is not
the latent_channels its format reports, and its length snaps to a frame
grid. So modules/models/minimax_h3 claims it and builds it, importing the
grid from ComfyUI rather than copying a second definition of it.
Nothing outside that folder names a model, and nothing here assumes video:
a 2-D latent format yields an image latent with no branch anywhere.
Modules reach core through modules/_core.py. ComfyUI imports a pack by file
location, so an absolute `import core` in a module resolves under pytest and
not under ComfyUI -- the node then never registers, silently. That cost the
same bug twice, so there is now a test that loads the pack the way ComfyUI
does, in an isolated subprocess whose cwd is NOT the repo. The first version
of that test passed with the bug reintroduced, because cwd was on sys.path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ported from v4, minus two mechanisms that are no longer needed. The LoRA
loader takes one LoRA and chains, so the FlexibleOptionalInputType dict
subclass that lied to ComfyUI's validator has nothing left to do. The CLIP
loader uses numbered slots like core's DualCLIPLoader instead of a JSON
array in a string widget.
Autogrow looked like the right answer for the encoder list and is not: it
sets force_input on any widget input, so a list of combos renders as sockets
and the file picker stops being a picker. Every core use of it is a
connection type.
GGUF and SLA are deliberately not here. Both are capabilities rather than
loader options, and both belong in modules of their own.
The tests follow what the loaders PRODUCE rather than stopping at the call:
the attention override is carried into comfy's real dispatch and checked
against every backend this machine registers, because an override that only
fails at the first attention call fails at sampling, not at load.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ComfyUI's V3 entrypoint replaces NODE_CLASS_MAPPINGS, which must not exist
at all: nodes.py checks it first and reaches comfy_entrypoint only in an
elif, so an empty dict registers nothing and reports no error.
A module declares NODES for graph structure and SETTINGS for preferences,
and the two never mix -- a saved workflow stores class_type and input
names, so letting a new setting add a socket would rot existing graphs.
Traits now come from two places: facts core can read off any model, and
providers announced by a model's own module. Core names no model, so
supporting a new one is a folder rather than an edit here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
