Prompt Markup strips it before the encoder; Apply Prompt Markup puts weights
and timed windows on attention and blends on the conditioning. Where the words
sit is the H3 module's (text_tokenizer, prompt_rows, video_time_rows). Wired
into both H3 presets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both replace what a block allocates, which crashes aimdo's compiler (seen on a
rental with v4's shadow negative). Attached to the model, so any sampler.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Block repeat's video-only mode now really keeps audio single-pass: H3 blocks
write in place, so v4 restored audio from an already-repeated stream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
extensive_testing round 5 did a systematic audit rather than one more
spot check, and found the one remaining gap: row VALUES are wrapped to
card width before drawing (_wrap()), bounding each draw call's cost, but
"head" block titles (built from slot id/group/node, each individually
_cap()-allowed up to 400 chars per round 4) were drawn as one unwrapped
string. PIL's text-render cost scales with string length regardless of
visible width, so 2000 slots each with a 1200-char (id+group+node)
concatenated title -- every documented cap individually honored, nothing
near any threshold -- cost 20.5s of pure font-render CPU.
Fixed by truncating head-block text to the same rendered width row values
already get, before it's ever handed to the font renderer. Round 5's own
systematic pass (tracing every slots-derived value to its draw/embed
destination, checking the wired-link path, non-string id/group/node,
by_id construction cost, host reachability, and the render kwarg) found
nothing else -- everything else that reaches the raster or the JSON embed
is capped.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found while re-reading collect() after round 4's fix, same pattern as every
prior round: an unaudited site where caller data reaches the card uncapped.
- collect()'s per-input loop caps `value` via _short() but never capped
`name` -- the input dict's KEY is caller data too (the whole `inputs`
dict comes from the client), and flows verbatim into the drawn row label
and the embedded JSON.
- _rows_from_list's per-entry join (`f"{k}={_short(v)}"`) caps `v` but not
`k` -- a huge key still gets fully materialized into the string before
the joined result's own _short() call ever runs, so it's real per-row
cost even though the final printed string comes out short.
Both fixed with the same _cap() helper already used for slot id/group/node.
Added regression tests for both.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
extensive_testing round 4 found every cap added in rounds 1-3 bounded ROW
content only. A slot's id/group/node class, and project_name, were never
run through any length cap -- they reach render_png as "head"/"title"
block kinds, which the round-2 line-truncation loop skips entirely (it
only counted kind == "row"). Worse, collect()'s row-budget check never
fires for a slot with zero rows (empty inputs), so slot COUNT itself was
unbounded whenever each slot carries no rows.
Reproduced: 3000 slots with empty inputs and 4000-char id/group strings --
total_rows stayed at 0 the entire time, so none of the three existing caps
ever engaged -- cost 193.6s render time and produced a 12.1MB embedded
tEXt JSON, with zero truncation declared anywhere on the card.
Fixed by: capping slot id/group/node/project_name/wired-from-label length
via a new _cap() helper (same truncation as _short(), without its
display-value formatting so "no id"/"no group" fallbacks still work);
capping slot COUNT directly in collect() rather than relying on row count;
and making render_png's line-truncation loop count "head" blocks toward
the same budget as "row" blocks, as defense in depth against any other
block kind reaching render with no accompanying row. Added a test
reproducing the exact empty-input/huge-id shape, asserting on render_png's
wall-clock time and both the PNG size and the embedded JSON size.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
extensive_testing round 3 found round 2's fix was itself incomplete: it
bounded row count and per-key value length, but not the JOINED row string
_rows_from_list builds from a dict with many keys. A funpack_list entry
like {"k1": "x"*400, ..., "k150": "x"*400} produced one row whose value had
no length cap of its own -- row count and per-key length both stayed under
every existing budget while the report JSON embedded in the PNG's tEXt
chunk (which scales 1:1 with it) reached 123MB, reproducing round 2's exact
number through a different attack shape, with zero truncation declared
anywhere on the resulting card.
Fixed by running the joined row string back through _short() (closing the
length gap) and capping how many keys get processed per row at 50 before
that join even happens (closing a related CPU-amplification gap: an absurd
key count still cost one f-string+join per key before any length cap could
apply). Added a test reproducing the exact many-key-row shape and asserting
on render_png's actual wall-clock time and output size, not just collect()'s
data.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
extensive_testing round 2 found the round-1 render-cost fix was broken:
- _MAX_ROWS and _MAX_VALUE_CHARS each bounded one factor but never their
product -- 2000 rows of 400-char no-space values honored both caps
individually and still cost ~14s/~10MB per render.
- collect()'s card-wide row budget was checked once per SLOT, so one slot
with many funpack_list inputs (each independently capped at _MAX_ROWS)
could blow the total budget by 50x with no declared truncation -- 67.8s,
~123MB, and a PNG that trips Pillow's own decompression-bomb guard on
reopen. Worse than the original round-1 finding, not fixed by it.
- none of round 1's three new tests called render_png -- the function whose
cost was the actual finding -- so both regressions passed all 21 tests.
Fixed by bounding the actual cost driver directly: render_png now caps
total rendered lines at 1200 regardless of how collect() produced them
(defense in depth), and collect()'s row-budget check now runs per input,
not per slot, so one slot's inputs can no longer accumulate past the card
budget before the next slot's check fires. Added tests that call render_png
against both worst cases and assert on wall-clock time, not just collect()'s
output data.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Round 1 of extensive_testing on 6d49625 found two real defects:
- Unbounded server-side render cost/memory from a client-supplied
pipeline. render_png's image height grows with wrapped-line count,
so one crafted long input value amplified a small request into a
77,400px-tall, ~250MB render taking several seconds of single-
threaded PIL draw calls -- reproduced directly. This route is
unauthenticated, same as every sibling pipeline route, and reachable
over a rental's plain-http IP -- a real DoS surface on the box
actually doing GPU sampling. Fixed with two bounds: _short() now
truncates any one value past 400 chars (a card is a compact summary,
nobody reads a 200k-char row on it), and collect() caps the whole
card at 2000 rows, both declared on the card itself via a "…N more
omitted" note rather than silently dropped. Verified: the 200k-char
reproduction case drops from 3.2s/250MB to 0.09s/22KB; a 200,000-slot
pipeline renders bounded at ~1.7s/5MB with the truncation stated.
- Reopening "Export settings…" via the toolbar button (not the modal's
own ✕/Close) leaked the previous render's blob URL -- close()'s
revoke only runs on ITS OWN overlay, and the top-of-function DOM
cleanup only removed old .sc-overlay nodes, never revoked their
blobs. Carried over unfixed from v4's identical pattern. Fixed by
stashing the blob URL on its own overlay element so the cleanup line
can find and revoke it. Verified live: two clicks without ever
hitting Close produces two distinct blob URLs and exactly one
revokeObjectURL call, one overlay left in the DOM.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
MEDIUM-tier v4 feature ("Which model was that?" answered three days
later): a button in Models & Pipeline renders the currently loaded
pipeline as a PNG -- every slot's node class and typed-in values, wired
inputs marked as wired rather than printed as a stale value, and the
host facts (torch/CUDA/GPU/attention) that decide whether the setup
reproduces elsewhere. Downloadable, copyable to clipboard, and the
report is embedded in the PNG's own tEXt chunk for later machine
reading.
core/settings_card.py is a hard cut of v4's version, not a verbatim
port: v4 spent most of its length (_OWNED_BY, _EDITOR_DEFAULTS,
_pass_rows, _sampling_sections) describing FunPackStudio and the
LTX-AV Scene Chain Sampler's own hardcoded knob tables -- neither node
exists in v5's module-driven pipeline, so there is nothing left to
describe there. What's genuinely generic (walking a slot's real
inputs, detecting a wired one via core.graph.is_link, the PNG
renderer) is what's kept, adapted to v5's slot shape ({id, node,
group, inputs}, a wire arriving as [source_id, index] rather than a
separate input_sources/wires lookup) and v5's existing core/sysinfo.py
for host facts (whose field names already matched v4's needs).
New POST /api/settings-card route: stateless like /api/pipeline, the
CLIENT sends its own live pipeline (window.PipelineState.slots()) since
that's the one copy that includes any unsaved edit; the server supplies
only what it alone knows (host facts) and the render.
Frontend: app/legacy/api.js's settingsCard stub (present since the
Phase A port, explicitly marked "not built in v5 yet") replaced with a
real POST+blob call; app/legacy/models.js's Export-settings modal
ported from v4 near-verbatim, reading PS.slots()/project name instead
of a project id the old per-project models config needed. Verified
live: real pipeline, real host facts (this Mac's actual Apple MPS/
Python/ComfyUI versions), wired inputs correctly marked.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Loader-level attention option: mean-pool Q and a smoothed K into
blocks, score them with one small matmul, attend only the top
1-sparsity fraction of key blocks per query block, via a vendored
Triton kernel. Runs alongside the chosen attention backend rather than
replacing it (one override slot; the backend becomes SLA's dense
fall-through for the text refiner, masked calls, and trailing dense
steps), and refuses cleanly -- naming why -- on anything that is not
MiniMax H3 or a machine without CUDA+Triton.
modules/loaders/sla_block_map.py and sla_kernel.py are near-verbatim
ports of v4's vendored files (no FunPack-specific imports, so nothing
to adapt); sla_attention.py is the loader-facing glue (H3 gate,
override composition, per-step wrapper), same content as v4's with a
"not a module" note added to match gguf_support.py's convention.
Wired into FunPackDiffusionModelLoader as six new optional widgets,
following v4's loaders.py wiring but expressed in v5's io.Schema style.
The Triton kernel itself is unexercisable on this Mac (no CUDA) --
matches how real generation is verified in this project (rented GPU,
not local). Ported the CPU-testable half of v4's test suite (the
override contract, the H3 gate, the per-step wrapper); the three
kernel-correctness tests carry the same CUDA+Triton skip guard v4 used.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two real findings from adversarial review of the previous commit (the v4
UI port's route change), both fixed:
- app/tests-layout/*.spec.js (20 files, 133 tests): every one of these
navigated to /funpack/ and asserted on composer-specific DOM
(.cx-workspace*, window.FunPack) that no longer exists there now that
the ported v4 frontend is served at that URL instead. Left as-is they
would silently time out on every run. Each file now carries an explicit
test.skip(true, "...") with a comment explaining why and pointing at
the plan's pixel-exactness step, which is what will eventually rewrite
these against the new DOM -- declared and tracked, not quietly broken.
- core/tests/test_registry.py's fixture-based tests never exercised the
new category field through the real announcement_of()/scan() path --
only test_schema.py's direct validate() calls did, which bypass
registry.py's FIELDS name-mapping entirely. A typo there (e.g.
"Category" instead of "CATEGORY") would have passed every existing
test. The "good module" fixture gained CATEGORY = "guidance" and
test_the_good_module_loads now asserts on it, closing the gap.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two pieces of the "port v4's exact UI onto v5's backend" plan:
- core/contract.py: new optional ModuleSpec.category field (closed set:
continuity, guidance, conditioning, sampling, post, system), validated in
core/schema.py, threaded through core/registry.py's FIELDS and
to_manifest(). Presentation grouping, independent of STAGE (execution
order) -- a module can be stage="sampling" and category="guidance" at
once. Only the 4 existing modules that actually render a settings panel
(alg, sharpen, appearance, full_control) got a category assigned; the
other 12 have none to categorize (no SETTINGS block) and stay
uncategorised, which is valid.
- v4's frontend (movie_editor/frontend/*, ~29k lines, 61 files) copied
verbatim into app/legacy/. core/routes.py's "/" route now serves
legacy/index.html instead of the composer entry point; a <base> tag in
that file resolves its own relative script/link tags against
/funpack/app/legacy/ regardless of what URL it's served at, so every
file moved as a unit with zero per-file path edits. Verified live: all
61 assets load 200 OK, the real v4 onboarding screen renders pixel-for-
pixel (it's the actual DOM/CSS, not a reimplementation), and several v4
API calls (/api/media, /api/projects, /api/git/status) already succeed
against v5's real backend by coincidence of shared route naming -- the
rest 404/405 as expected, since api.js's ~90 functions haven't been
rewired to v5's actual routes yet (next piece of this plan).
The old composer entry point (app/index.html, app/composer/**) stays in
the tree, unreferenced, until a later cleanup pass once nothing imports it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pieces B and C: Scene gains source_image (aspect-ratio reference only --
resolution itself stays a project.video setting, ImageTransformKJ's
"total_pixels" keep_proportion fits the picture's aspect into that budget)
and references (media ids, in order). Inspector's Scene tab gets pick/clear/
remove controls; picking routes through the existing Media bin via a small
pending-pick handshake in boot.js (one bin, one onPick, so "which field
asked" has to be tracked between the click that asked and the click that
answered).
References wire into MiniMaxH3ReferenceToVideo's ref_images autogrow input
dynamically, per run, only for what a scene actually has -- an unwired slot,
not an empty image, is what "not in use" means to that node. The target is
data on the role itself (wireTo), not a slot id boot.js would otherwise have
to know, keeping the "core never names an implementation" contract intact
one layer up into the app.
Verifying this exposed a second real core gap (after the MatchType one):
core/graph.py had no idea what an Autogrow (V3's io.Autogrow) input even
is -- "ref_images" reported as one opaque COMFY_AUTOGROW_V3 slot, its real
per-instance sockets (ref_image_0, ref_image_1, ...) invisible to
declared()/build(), so a wire naming one was refused as an input the node
does not have. Fixed generically in comfy_types.py (autogrow_instances())
and graph.py, confirmed against ComfyUI's own _io.py (finalize_prefix's
dotted join, build_nested_inputs's runtime un-nesting) -- not guessed, and
proven end to end with a real override()+build() test using the exact edit
boot.js makes.
Piece A of the H3 workflow port: the loader/SigmaShift/ImageTransformKJ/
R2V/sampler-at-CFG-1/decode/video-mux chain the H3 user runs by hand today,
offered as a preset (GET /api/pipeline/presets) loadable into the same
editable pipeline core/graph.py already supports -- not a second "default",
which module-load order would make ambiguous.
References are deliberately NOT pre-wired into R2V's ref_images autogrow;
an unwired optional slot is what "not in use" means to that node (absent
from its ref_images dict), not an empty image degrading its <Picture i>
numbering. boot.js wires them per-run based on what's actually picked --
next commit.
Building this against a real ComfyUI+KJNodes registration (not a fake
schema) surfaced a genuine core bug: core/graph.py's from_comfyui() had no
idea what to do with a V3 MatchType socket (ImageTransformKJ's own image
input) -- its bare type name "COMFY_MATCHTYPE_V3" matched nothing, so a
completely legal IMAGE wire into it was refused. Fixed generically in
comfy_types.py (match_type_union()) and graph.py, for any node using
MatchType, not special-cased to this one.
The bridge piece B and C (resolution source, reference images) both need:
Media bin picks have no way into a ComfyUI graph today. One node, media_id
in, IMAGE out, backed by core/media.py's existing store.
Also fixes a real .gitignore bug found along the way: a bare "media/"
pattern matched modules/media/ (this node's own source) as well as the
intended root-level user-data directory. Rooted both media/ and projects/.
