Prompt Markup strips it before the encoder; Apply Prompt Markup puts weights
and timed windows on attention and blends on the conditioning. Where the words
sit is the H3 module's (text_tokenizer, prompt_rows, video_time_rows). Wired
into both H3 presets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both replace what a block allocates, which crashes aimdo's compiler (seen on a
rental with v4's shadow negative). Attached to the model, so any sampler.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Block repeat's video-only mode now really keeps audio single-pass: H3 blocks
write in place, so v4 restored audio from an already-repeated stream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
extensive_testing round 5 did a systematic audit rather than one more
spot check, and found the one remaining gap: row VALUES are wrapped to
card width before drawing (_wrap()), bounding each draw call's cost, but
"head" block titles (built from slot id/group/node, each individually
_cap()-allowed up to 400 chars per round 4) were drawn as one unwrapped
string. PIL's text-render cost scales with string length regardless of
visible width, so 2000 slots each with a 1200-char (id+group+node)
concatenated title -- every documented cap individually honored, nothing
near any threshold -- cost 20.5s of pure font-render CPU.
Fixed by truncating head-block text to the same rendered width row values
already get, before it's ever handed to the font renderer. Round 5's own
systematic pass (tracing every slots-derived value to its draw/embed
destination, checking the wired-link path, non-string id/group/node,
by_id construction cost, host reachability, and the render kwarg) found
nothing else -- everything else that reaches the raster or the JSON embed
is capped.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found while re-reading collect() after round 4's fix, same pattern as every
prior round: an unaudited site where caller data reaches the card uncapped.
- collect()'s per-input loop caps `value` via _short() but never capped
`name` -- the input dict's KEY is caller data too (the whole `inputs`
dict comes from the client), and flows verbatim into the drawn row label
and the embedded JSON.
- _rows_from_list's per-entry join (`f"{k}={_short(v)}"`) caps `v` but not
`k` -- a huge key still gets fully materialized into the string before
the joined result's own _short() call ever runs, so it's real per-row
cost even though the final printed string comes out short.
Both fixed with the same _cap() helper already used for slot id/group/node.
Added regression tests for both.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
extensive_testing round 4 found every cap added in rounds 1-3 bounded ROW
content only. A slot's id/group/node class, and project_name, were never
run through any length cap -- they reach render_png as "head"/"title"
block kinds, which the round-2 line-truncation loop skips entirely (it
only counted kind == "row"). Worse, collect()'s row-budget check never
fires for a slot with zero rows (empty inputs), so slot COUNT itself was
unbounded whenever each slot carries no rows.
Reproduced: 3000 slots with empty inputs and 4000-char id/group strings --
total_rows stayed at 0 the entire time, so none of the three existing caps
ever engaged -- cost 193.6s render time and produced a 12.1MB embedded
tEXt JSON, with zero truncation declared anywhere on the card.
Fixed by: capping slot id/group/node/project_name/wired-from-label length
via a new _cap() helper (same truncation as _short(), without its
display-value formatting so "no id"/"no group" fallbacks still work);
capping slot COUNT directly in collect() rather than relying on row count;
and making render_png's line-truncation loop count "head" blocks toward
the same budget as "row" blocks, as defense in depth against any other
block kind reaching render with no accompanying row. Added a test
reproducing the exact empty-input/huge-id shape, asserting on render_png's
wall-clock time and both the PNG size and the embedded JSON size.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
extensive_testing round 3 found round 2's fix was itself incomplete: it
bounded row count and per-key value length, but not the JOINED row string
_rows_from_list builds from a dict with many keys. A funpack_list entry
like {"k1": "x"*400, ..., "k150": "x"*400} produced one row whose value had
no length cap of its own -- row count and per-key length both stayed under
every existing budget while the report JSON embedded in the PNG's tEXt
chunk (which scales 1:1 with it) reached 123MB, reproducing round 2's exact
number through a different attack shape, with zero truncation declared
anywhere on the resulting card.
Fixed by running the joined row string back through _short() (closing the
length gap) and capping how many keys get processed per row at 50 before
that join even happens (closing a related CPU-amplification gap: an absurd
key count still cost one f-string+join per key before any length cap could
apply). Added a test reproducing the exact many-key-row shape and asserting
on render_png's actual wall-clock time and output size, not just collect()'s
data.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
extensive_testing round 2 found the round-1 render-cost fix was broken:
- _MAX_ROWS and _MAX_VALUE_CHARS each bounded one factor but never their
product -- 2000 rows of 400-char no-space values honored both caps
individually and still cost ~14s/~10MB per render.
- collect()'s card-wide row budget was checked once per SLOT, so one slot
with many funpack_list inputs (each independently capped at _MAX_ROWS)
could blow the total budget by 50x with no declared truncation -- 67.8s,
~123MB, and a PNG that trips Pillow's own decompression-bomb guard on
reopen. Worse than the original round-1 finding, not fixed by it.
- none of round 1's three new tests called render_png -- the function whose
cost was the actual finding -- so both regressions passed all 21 tests.
Fixed by bounding the actual cost driver directly: render_png now caps
total rendered lines at 1200 regardless of how collect() produced them
(defense in depth), and collect()'s row-budget check now runs per input,
not per slot, so one slot's inputs can no longer accumulate past the card
budget before the next slot's check fires. Added tests that call render_png
against both worst cases and assert on wall-clock time, not just collect()'s
output data.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Round 1 of extensive_testing on 6d49625 found two real defects:
- Unbounded server-side render cost/memory from a client-supplied
pipeline. render_png's image height grows with wrapped-line count,
so one crafted long input value amplified a small request into a
77,400px-tall, ~250MB render taking several seconds of single-
threaded PIL draw calls -- reproduced directly. This route is
unauthenticated, same as every sibling pipeline route, and reachable
over a rental's plain-http IP -- a real DoS surface on the box
actually doing GPU sampling. Fixed with two bounds: _short() now
truncates any one value past 400 chars (a card is a compact summary,
nobody reads a 200k-char row on it), and collect() caps the whole
card at 2000 rows, both declared on the card itself via a "…N more
omitted" note rather than silently dropped. Verified: the 200k-char
reproduction case drops from 3.2s/250MB to 0.09s/22KB; a 200,000-slot
pipeline renders bounded at ~1.7s/5MB with the truncation stated.
- Reopening "Export settings…" via the toolbar button (not the modal's
own ✕/Close) leaked the previous render's blob URL -- close()'s
revoke only runs on ITS OWN overlay, and the top-of-function DOM
cleanup only removed old .sc-overlay nodes, never revoked their
blobs. Carried over unfixed from v4's identical pattern. Fixed by
stashing the blob URL on its own overlay element so the cleanup line
can find and revoke it. Verified live: two clicks without ever
hitting Close produces two distinct blob URLs and exactly one
revokeObjectURL call, one overlay left in the DOM.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
MEDIUM-tier v4 feature ("Which model was that?" answered three days
later): a button in Models & Pipeline renders the currently loaded
pipeline as a PNG -- every slot's node class and typed-in values, wired
inputs marked as wired rather than printed as a stale value, and the
host facts (torch/CUDA/GPU/attention) that decide whether the setup
reproduces elsewhere. Downloadable, copyable to clipboard, and the
report is embedded in the PNG's own tEXt chunk for later machine
reading.
core/settings_card.py is a hard cut of v4's version, not a verbatim
port: v4 spent most of its length (_OWNED_BY, _EDITOR_DEFAULTS,
_pass_rows, _sampling_sections) describing FunPackStudio and the
LTX-AV Scene Chain Sampler's own hardcoded knob tables -- neither node
exists in v5's module-driven pipeline, so there is nothing left to
describe there. What's genuinely generic (walking a slot's real
inputs, detecting a wired one via core.graph.is_link, the PNG
renderer) is what's kept, adapted to v5's slot shape ({id, node,
group, inputs}, a wire arriving as [source_id, index] rather than a
separate input_sources/wires lookup) and v5's existing core/sysinfo.py
for host facts (whose field names already matched v4's needs).
New POST /api/settings-card route: stateless like /api/pipeline, the
CLIENT sends its own live pipeline (window.PipelineState.slots()) since
that's the one copy that includes any unsaved edit; the server supplies
only what it alone knows (host facts) and the render.
Frontend: app/legacy/api.js's settingsCard stub (present since the
Phase A port, explicitly marked "not built in v5 yet") replaced with a
real POST+blob call; app/legacy/models.js's Export-settings modal
ported from v4 near-verbatim, reading PS.slots()/project name instead
of a project id the old per-project models config needed. Verified
live: real pipeline, real host facts (this Mac's actual Apple MPS/
Python/ComfyUI versions), wired inputs correctly marked.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Loader-level attention option: mean-pool Q and a smoothed K into
blocks, score them with one small matmul, attend only the top
1-sparsity fraction of key blocks per query block, via a vendored
Triton kernel. Runs alongside the chosen attention backend rather than
replacing it (one override slot; the backend becomes SLA's dense
fall-through for the text refiner, masked calls, and trailing dense
steps), and refuses cleanly -- naming why -- on anything that is not
MiniMax H3 or a machine without CUDA+Triton.
modules/loaders/sla_block_map.py and sla_kernel.py are near-verbatim
ports of v4's vendored files (no FunPack-specific imports, so nothing
to adapt); sla_attention.py is the loader-facing glue (H3 gate,
override composition, per-step wrapper), same content as v4's with a
"not a module" note added to match gguf_support.py's convention.
Wired into FunPackDiffusionModelLoader as six new optional widgets,
following v4's loaders.py wiring but expressed in v5's io.Schema style.
The Triton kernel itself is unexercisable on this Mac (no CUDA) --
matches how real generation is verified in this project (rented GPU,
not local). Ported the CPU-testable half of v4's test suite (the
override contract, the H3 gate, the per-step wrapper); the three
kernel-correctness tests carry the same CUDA+Triton skip guard v4 used.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two real findings from adversarial review of the previous commit (the v4
UI port's route change), both fixed:
- app/tests-layout/*.spec.js (20 files, 133 tests): every one of these
navigated to /funpack/ and asserted on composer-specific DOM
(.cx-workspace*, window.FunPack) that no longer exists there now that
the ported v4 frontend is served at that URL instead. Left as-is they
would silently time out on every run. Each file now carries an explicit
test.skip(true, "...") with a comment explaining why and pointing at
the plan's pixel-exactness step, which is what will eventually rewrite
these against the new DOM -- declared and tracked, not quietly broken.
- core/tests/test_registry.py's fixture-based tests never exercised the
new category field through the real announcement_of()/scan() path --
only test_schema.py's direct validate() calls did, which bypass
registry.py's FIELDS name-mapping entirely. A typo there (e.g.
"Category" instead of "CATEGORY") would have passed every existing
test. The "good module" fixture gained CATEGORY = "guidance" and
test_the_good_module_loads now asserts on it, closing the gap.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two pieces of the "port v4's exact UI onto v5's backend" plan:
- core/contract.py: new optional ModuleSpec.category field (closed set:
continuity, guidance, conditioning, sampling, post, system), validated in
core/schema.py, threaded through core/registry.py's FIELDS and
to_manifest(). Presentation grouping, independent of STAGE (execution
order) -- a module can be stage="sampling" and category="guidance" at
once. Only the 4 existing modules that actually render a settings panel
(alg, sharpen, appearance, full_control) got a category assigned; the
other 12 have none to categorize (no SETTINGS block) and stay
uncategorised, which is valid.
- v4's frontend (movie_editor/frontend/*, ~29k lines, 61 files) copied
verbatim into app/legacy/. core/routes.py's "/" route now serves
legacy/index.html instead of the composer entry point; a <base> tag in
that file resolves its own relative script/link tags against
/funpack/app/legacy/ regardless of what URL it's served at, so every
file moved as a unit with zero per-file path edits. Verified live: all
61 assets load 200 OK, the real v4 onboarding screen renders pixel-for-
pixel (it's the actual DOM/CSS, not a reimplementation), and several v4
API calls (/api/media, /api/projects, /api/git/status) already succeed
against v5's real backend by coincidence of shared route naming -- the
rest 404/405 as expected, since api.js's ~90 functions haven't been
rewired to v5's actual routes yet (next piece of this plan).
The old composer entry point (app/index.html, app/composer/**) stays in
the tree, unreferenced, until a later cleanup pass once nothing imports it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pieces B and C: Scene gains source_image (aspect-ratio reference only --
resolution itself stays a project.video setting, ImageTransformKJ's
"total_pixels" keep_proportion fits the picture's aspect into that budget)
and references (media ids, in order). Inspector's Scene tab gets pick/clear/
remove controls; picking routes through the existing Media bin via a small
pending-pick handshake in boot.js (one bin, one onPick, so "which field
asked" has to be tracked between the click that asked and the click that
answered).
References wire into MiniMaxH3ReferenceToVideo's ref_images autogrow input
dynamically, per run, only for what a scene actually has -- an unwired slot,
not an empty image, is what "not in use" means to that node. The target is
data on the role itself (wireTo), not a slot id boot.js would otherwise have
to know, keeping the "core never names an implementation" contract intact
one layer up into the app.
Verifying this exposed a second real core gap (after the MatchType one):
core/graph.py had no idea what an Autogrow (V3's io.Autogrow) input even
is -- "ref_images" reported as one opaque COMFY_AUTOGROW_V3 slot, its real
per-instance sockets (ref_image_0, ref_image_1, ...) invisible to
declared()/build(), so a wire naming one was refused as an input the node
does not have. Fixed generically in comfy_types.py (autogrow_instances())
and graph.py, confirmed against ComfyUI's own _io.py (finalize_prefix's
dotted join, build_nested_inputs's runtime un-nesting) -- not guessed, and
proven end to end with a real override()+build() test using the exact edit
boot.js makes.
Piece A of the H3 workflow port: the loader/SigmaShift/ImageTransformKJ/
R2V/sampler-at-CFG-1/decode/video-mux chain the H3 user runs by hand today,
offered as a preset (GET /api/pipeline/presets) loadable into the same
editable pipeline core/graph.py already supports -- not a second "default",
which module-load order would make ambiguous.
References are deliberately NOT pre-wired into R2V's ref_images autogrow;
an unwired optional slot is what "not in use" means to that node (absent
from its ref_images dict), not an empty image degrading its <Picture i>
numbering. boot.js wires them per-run based on what's actually picked --
next commit.
Building this against a real ComfyUI+KJNodes registration (not a fake
schema) surfaced a genuine core bug: core/graph.py's from_comfyui() had no
idea what to do with a V3 MatchType socket (ImageTransformKJ's own image
input) -- its bare type name "COMFY_MATCHTYPE_V3" matched nothing, so a
completely legal IMAGE wire into it was refused. Fixed generically in
comfy_types.py (match_type_union()) and graph.py, for any node using
MatchType, not special-cased to this one.
The bridge piece B and C (resolution source, reference images) both need:
Media bin picks have no way into a ComfyUI graph today. One node, media_id
in, IMAGE out, backed by core/media.py's existing store.
Also fixes a real .gitignore bug found along the way: a bare "media/"
pattern matched modules/media/ (this node's own source) as well as the
intended root-level user-data directory. Rooted both media/ and projects/.
A variable referencing the same other variable more than once is not a
cycle, so memoization alone still leaves the OUTPUT exponential in size
(each level's re.sub concatenates the memoized child result multiple
times). Adds a shared output-size budget, checked before using either
a cached or freshly-computed result.
Also ignore the runtime shortcuts.json data file, and fix the fan-out
test's assertion, which was checking against a vacuous condition.
A trigger->replacement library (core/shortcuts.py, a close port of v4's
apply_prompt_shortcuts/resolve_variables), a Settings section to manage it,
autocomplete on the real prompt textarea (composer.autocomplete.default, so
it outranks the Constructor modal it lives inside), and a stateless
/api/prompt/expand that combines anchor + shortcuts + $variables + postfix
at the moment a run is queued -- scene.text itself stays exactly what was
typed. Project gained anchor/postfix/variables editing UI on its own tab.
Along the way: a real layout bug in the Properties column (every composer
stack can flex-shrink below its own content, and once the Project tab grew
past a couple of rows the column started squeezing its own children's
content into each other and, at the outer level, into the sibling Generation
section below) -- fixed with flex-shrink:0 at each level that needs it,
caught by a Playwright regression test.
Alongside the existing negative prompt: anchor (prepended to every scene)
and postfix + postfix_enabled (appended, with its own on/off switch --
v4 proved a postfix worth keeping needs to be turned off for a quick test
without losing it, which clearing the field would do).
Data layer only, deliberately: unlike negative, which maps 1:1 onto an
existing CLIPTextEncode "text" input via a role, anchor and postfix need
to be COMBINED with each scene's own prompt text, and nothing in the
pipeline does that yet -- that is Studio's job, not built. Wiring a UI
field to a mechanism that does not exist yet is the "half-assed port"
this project has already been steered away from once this session.
Also backfilled round-trip and never-trust-a-file tests for the existing
negative field, which had none at the core/projects.py level despite
video already having them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every node's own OUTPUTS were invisible on its own panel in the pipeline
editor -- wiring an input already let you pick a source, but opening the
node that PRODUCES something told you nothing about what it makes or
whether anything reads it. Found live, by a real user just looking at a
node, after two full review rounds that only ever checked the consuming
side. Fixed with a read-only "Outputs" section per node, naming each
output, its type, and who (if anyone) currently wires from it.
Also: model family detection. core/probe.py reads a safetensors header
(tensor names only, no weights) and asks every installed model module,
through the same capability-lookup mechanism already used for
empty_latent/decode, whether the file is its own -- core never names a
family itself. A real, adversarially-found bug in the ported signature:
one of MiniMax H3's three candidate prefixes doesn't correspond to any
prefix ComfyUI's own loader actually resolves to, traced through
comfy/model_detection.py and comfy/sd.py -- the probe could have called a
file H3 that ComfyUI's real loader would refuse. Removed, and pinned with
a dedicated test module. A second finding narrowed the folder search to
only the folders FunPack's own loaders actually read from.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pipeline wiring: core/graph.py gains wire()/unwire(), and the "Models and
pipeline" editor can now actually connect one node's output to another's
input -- sockets and widgets alike, which is what makes "linked inputs"
(several widgets sharing one Primitive node's value) buildable at all,
without a second mechanism. Three adversarial review rounds found and fixed
real bugs along the way: check()'s request builder silently dropped the
wire-specific fields; the widgets loop had no wiring UI at all despite the
backend supporting it, and a widget already holding a link rendered as an
editable box with the wrong (default) value; wiring an input that already
had an unsaved drafted value let the stale value silently win on Save. Also
fixed, found live rather than by review: the Add-node picker stacked two
disagreeing search boxes (an outer server search and filterList's own
redundant client-side one).
Updates window: "Restart ComfyUI" is now a plain, always-available button,
not just something offered when an update already landed. Rolling back or
switching branch on a dirty tree now auto-stashes instead of refusing --
updating still refuses, unchanged, since pulling new commits onto changes
in progress is not something to do without asking. Three review rounds
here too: stashing before reading the rollback target shifted what HEAD@{1}
meant (a real reflog hazard, not just reasoned about); a stash's disclosure
was lost if the step right after it failed; that fix was too narrow and
missed pull()'s own failures inside checkout(pull_after=True) -- the actual
default path -- and rollback()'s own reset failing; and finally, the
broadened fix still only caught GitUpdateError, missing subprocess-level
failures like a hung fetch timing out, which is the realistic failure mode
on a rented GPU box's network.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Files the user brings IN (a reference image, an imported clip) now have a
real store: core/media.py, /api/media routes, and a "Media" tab beside the
bin's "Results" tab in the Assets zone. Upload, list, fetch, delete all
round-trip through the real server; nothing wires into a pipeline input yet,
since no model module or Studio conditioning takes a reference image today.
Two adversarial review rounds found and fixed three real issues along the
way: MAX_BYTES (512MB) was dead code because aiohttp's own client_max_size
capped every upload far lower first in both prod and dev, with a JSON-parse
error leaking to the user on rejection instead of a clean message; the
index.json read-modify-write had no guard against concurrent writers; and
the devserver's bare Application didn't match ComfyUI's own 100MB default,
which also broke an unrelated e2e assertion once the review polluted the
real, unisolated media/ directory testing the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three things the unified Settings window was still missing next to v4's
own, all found by looking at it next to v4 rather than in isolation:
- The sidebar is a 52px icon rail now, expanding to a full list on
hover/focus and overlaying the content instead of a splitPane's
permanently-docked two-pane shape -- v4's own sidebar, not an
approximation of it. Fixing this surfaced two real CSS bugs along the
way: filterList's own min-width:220px silently defeated the collapse
(clipping the search box to "Sear" instead of showing an icon), and
.cx-filter-list's unset overflow-x (computed to auto once overflow-y
was set) put a horizontal scrollbar under it. Centering the icon in
the collapsed row needed the label's flex-basis AND the row's own gap
to both go to zero -- an invisible label still claims its share of
both otherwise, and the icon renders off-centre.
- The modal now has a real height (v4's own 660px) rather than sizing to
whichever section is showing, so switching between a short section
(About) and a long one (Node packs) doesn't visibly resize the whole
window. That exposed a THIRD latent bug: .cx-modal-body never had
flex:1 1 auto, invisible until a modal had a real height to fill --
every prior modal shrink-wrapped its content, so head+body+foot always
summed to exactly the card's height by accident. Fixed at the shared
rule, not just for Settings.
- About now shows what v4's does: commit and dirty state, a codename
when the checkout has one, and the machine's real hardware/software
facts (chip, memory, GPU or MPS, disk, OS, ComfyUI/Python/Torch/
attention-backend versions, hostname) -- a new core/sysinfo.py
(ported from v4's own, individually guarded so a missing probe never
takes the panel down) behind a new GET /funpack/api/system, injected
into settings_window.js the same way every other network call here is.
Also: a nav row's tone (the coloured icon chip) is now validated against
its closed set the same way button.js already validates its own tone --
an unknown one used to render silently wrong instead of failing loudly.
