Three holes left after the /result faststart remux (e296fe0), matching the
still-live symptoms (decode errors in console, blinking clips, playback
restarting from 0, stale previews):
1. Mid-write serving: /result hit while VHS is still writing the mp4 found no
moov, failed the remux, and served the raw moov-less bytes anyway —
undecodable, and the player's cache-busted retries kept blinking against it.
2. Byte-switching under one URL: once the write finished, the same URL switched
from raw to remuxed bytes with different offsets/length. A browser mid-Range
stream mixes offsets (PIPELINE_ERROR_DECODE) or fails If-Range revalidation
and falls back to a full 200 — resetting the element and restarting the clip
from the beginning.
3. Cache collisions/staleness: the remux cache was keyed by path only (a
rewritten source kept serving the old remux) and the remux output was named
by basename only (two sources sharing a basename overwrote each other's
remuxed copy with -y, possibly while it was being streamed).
Fix: _is_faststart becomes _moov_position (front/end/none) so "remux me"
(moov at end = complete VHS file) is distinguished from "come back later"
(no moov = still being written or aborted save). No-moov sources now get 503 +
Retry-After from both /result and preview-segment instead of undecodable bytes;
the player retry loop already handles that. The remux cache is keyed on the
source's (mtime_ns, size) signature, the output name is derived from
path+signature (no basename collisions), the remuxed file is validated
(nonzero, moov at front) before caching, and a stable-but-unremuxable file is
remembered so ffmpeg isn't re-run per request. The preview-segment cache key
also gains the source signature so a re-rendered source at the same path can't
serve segments encoded from the old bytes.
Verified against real ffmpeg files: end/front/none discrimination on complete,
faststart, truncated-mid-write, and empty mp4s; remux of a truncated file
raises instead of caching garbage. All 160 movie_editor tests pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ComfyUI's VHS_VideoCombine node writes moov at the end of the mp4 (no
+faststart). Deep-seeking that file is exactly what preview-segment already
routes multi-scene chain clips around, but a "shallow" single clip (inSec
<=0.5) or just opening a scene's own preview hits /result directly, gets
served the raw moov-at-end file, and fails to decode the moment the user
scrubs anywhere into it — reproduced live as repeated
NS_ERROR_DOM_MEDIA_DECODE_ERR/AppleVTDecoder failures on the same file even
after cache-busted retries, since the bytes on disk are permanently bad for
seeking, not transiently mid-write.
Fix at the source instead of extending the segment path: /result now
stream-copy remuxes (+faststart, no re-encode) any .mp4 it serves that
doesn't already have moov up front, caching the remuxed copy so it only
happens once per file. FunPack's own segment/export/final-render output
already sets +faststart, so this is a no-op for everything except the raw
VHS output — verified the atom-order detection and remux against real
ffmpeg-produced faststart/non-faststart files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
model=object() lacked model_options (needed by the per-scene wrapper
snapshot/restore added for the scene-wrapper-leak fix), and FakeVAE lacked
decode() (needed since the sampler always decodes IMAGES when the caller
doesn't declare that output as disconnected) — together these silently
killed every test in the file that calls sample(). Added FakeModel and
FakeVAE.decode, then fixed the resulting return-tuple arity (sample() now
returns 6 values, these tests unpacked 4) and three genuinely stale
assertions: the per-scene report line gained a timing segment, carry_i2v_guides
now prepends its guide frame instead of appending it, and _append_i2v_guides
no longer routes through keyframe_idxs/guide_attention_entries.
Verified in isolation: all 14 tests in this file now pass. Running the full
tests/ suite still shows the same file failing — that's a separate,
pre-existing cross-file sys.modules stubbing collision (confirmed via
git stash on a clean dev checkout) affecting this file plus
test_mask_safe_attention.py and test_v2_category_alignment.py; not fixed
here since it isn't a one-line root cause.
carry_overlap_through_anchor lets a scene keep the previous scene's latent
tail for background/environment continuity even when it switches to its own
i2v anchor, instead of the current hard cut. The anchor's own leading frame
stays untouched; only the frames after it carry the old tail.
Also fixes a real gap found along the way: the mixed-anchor branch never
called _apply_configured_guides, so Best-FaceID identity_transfer could
never resolve an identity_pin on an anchor-swap scene — exactly where it's
most useful. Added a standalone lookup so it resolves there too.
Today's preview-stability fix (4bc56e8) made the player pool fetch every
chain-shared clip through a trimmed, faststart preview segment instead of
deep-seeking the raw chain output (moov-at-end, stalls/blacks in the
browser). Two other consumers of the raw file were missed:
- appendFilmstrip built a <video preload=auto> pointed at the full shared
chain output and seeked into it per thumbnail — the same deep-seek
hazard the player fix eliminated, just in a different element.
- sceneAudioClip's waveform did a plain fetch()+arrayBuffer() of the
ENTIRE shared file, once per scene sharing it, to decode a waveform.
Repro (fake 3-scene chain sharing one moov-at-end mp4): opening the
project fired 15 aborted requests fighting the player pool over Chrome's
6-per-origin cap — exactly the stale/black/reset-to-start symptoms
reported live. Both now request the same cached preview-segment URL the
player uses; post-fix, scrubbing 11/11 test positions succeeded with no
black frames or resets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces the source-phase-RoPE-only approximation with a faithful native port of
ComfyUI-BFSNodes' LTX Identity Transfer (read upstream source directly): the reference
face is now injected as separate, non-rendered overlap tokens (clean timestep, RoPE
source-phase tag, trimmed before output) instead of blending into Continuity's identity-pin
keyframe, plus an optional ArcFace embedding -> IdentityProjector -> context-token channel
(identity_projector, source_id, phase_scale, id_strength, arcface_mode, debug_log). Stays
on Chain Sampler per the Studio/Sampler boundary law's guide-ops carve-out. New
identity_transfer.py module; insightface added as a soft pip dependency.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Models & Pipeline -> +New node: picking a node now collapses the search results
into a one-line "selected" row with a Change button, instead of leaving the full
list open above the Values/Wiring panel.
Prompt autocomplete: dropped the MAX_ITEMS=8 cap on matchTriggers — a prefix
shared by many shortcuts (e.g. 15 "lazy" triggers) now shows all of them; the
menu already scrolls (max-height in CSS), so nothing was gained by truncating.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reverts the earlier ComfyUI-BFSNodes graph-splice approach (builder.py/timeline.py/
server.py scaffolding) in favor of porting the missing piece natively: FunPack's own
guide-attention path (LTXVAddGuide.add_keyframe_index + guide_attention_entries)
already places a reference image on the frame-0 grid exactly like BFSNodes' overlap
technique, and Continuity's existing "Identity pin" already applies one reference
face to every scene. The only real gap was the LoRA's trained-time source_phase RoPE
tag, added as a tag+strip patch on FunPackLTXAVSceneChainSampler scoped to the exact
guide-entry token range (never co-active mid_scene/carry/prior-scene guides sharing
the same keyframe_idxs tensor). New "Best-FaceID compatibility" + "Source-phase id"
knobs live under Engine Settings -> Experimental, alongside every other unvalidated
technique. LoRA loading itself reuses the existing generic Models-menu lora role.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Detects ComfyUI-BFSNodes' LTX Identity Transfer node in object_info and, when
enabled, auto-loads the Best-FaceID LoRA onto the model path and splices the
identity node between Studio and the Chain Sampler (project-wide, one
reference face per project — a run's graph can't vary it mid-chain). New
Engine settings pane under Chain Sampler picks the reference face (media
picker) and LoRA file (live dropdown), with advanced ArcFace/phase knobs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Four compounding defects behind stale/black previews on live instances:
1. /preview-segment ran its ffmpeg x264 re-encode synchronously inside the async
handler — each encode froze ComfyUI's single aiohttp event loop for seconds,
stalling every streaming /result response mid-transfer (elements error out,
retries cache-bust and refetch, transport wedges). Worst right after a run,
when the pool requests every clip's segment at once. Now asyncio.to_thread
with a per-key lock (scrub storms don't spawn duplicate encodes) and
-preset veryfast for preview trims. /export-clip wrapped the same way.
2. Ghosts were excluded from the segment system, so scrubbing a removed scene
deep-seeked the raw moov-at-end chain MP4 — the exact stall segments were
built to avoid. Ghosts (and lone clips with a deep in-point, e.g. the last
surviving scene of a chain) now get segments; the trim window travels in the
query (render_in + dur) since the scene no longer exists server-side.
3. Real staleness: segment URLs lacked the clip duration while responses are
browser-cached for an hour — trimming a clip kept the same URL and replayed
the hour-old segment. dur is now part of every segment URL.
4. Every pooled <video> used preload=auto: the whole timeline downloaded
simultaneously, starving the browser's per-origin connections on tunneled
instances and aborting the stream actually being watched. Pool now idles at
preload=metadata; the current clip and its neighbours are promoted to auto.
Verified live on local ComfyUI with a synthetic 2-scene chain + ghost: ghost
segment 200/video-mp4, API stays responsive during an encode (17ms ping),
scrub across boundaries, seek into ghost (segment-relative time + badge),
play across the scene→ghost boundary, rapid pause/resume — all clean.
160 editor tests pass (new: ghost clip-spec parsing).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
alg_blur_guides no longer borrows the sampler's anchor knobs: the Scene Chain
Sampler gains alg_guide_blur_strength (default 2.0) and
alg_guide_blur_sigma_threshold (default 0.975), threaded per-chunk via
extra_options like the tail count. The sampler now precomputes one latent per
(anchor blurred?, tail blurred?) combination — the frame sets are disjoint, so
the both-blurred variant composes the two passes — and each region follows its
own sigma window per step. Anchor ALG (alg_enabled/alg_strength/
alg_sigma_threshold) is untouched and back to anchor-only semantics.
Editor: 'Guide blur strength' and 'Guide blur sigma threshold' rows appear
under Engine ▸ Experimental when the blur toggle is on (sampler_inputs bridge,
names match the new node widgets; inputs appended at the block end to keep the
builder's positional widget mapping aligned).
Verified live: object_info shows both widgets appended last; Editor rows
appear on toggle and persist to project.sampler_inputs. 11 ALG unit tests
pass (new: composed independent-strength blur); same 20 pre-existing suite
failures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Editor ▸ Settings ▸ Editor gains a 'Shortcut revolver' toggle (+ 'Random' sub-option).
When on, each shortcut with several replacements draws from a per-shortcut queue —
sequential (first…last, then restart) or shuffled once per cycle — and nothing repeats
until the whole set has fired. Off (default) keeps the seeded random pick unchanged.
- State is a sidecar (shortcut_revolver.json) next to the shortcut DB: settings +
per-shortcut remaining queue + replacement-set fingerprint (editing a shortcut's
replacements restarts its cycle; changing either setting resets all cycles).
- Generation commits the draw (refine_v2 positive prompt); every preview path
(parse_timeline, split mapping, fragments) peeks the same stored queue without
saving, so previews show exactly what the next Generate will use.
- Batch Training variants each commit a draw: N variants sweep N consecutive
chambers instead of seed-rolled repeats.
- FunPackStudio.IS_CHANGED returns nan while the revolver is on so fixed-seed
re-runs aren't served from ComfyUI's node cache (the next chamber must fire).
Verified live on local ComfyUI: settings round-trip, seed-independent stable peeks,
sequential cycle REP_ONE→TWO→THREE→ONE across processes, peek-after-commit shows the
next chamber. 9 new unit tests; full suite has the same 20 pre-existing failures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
alg_blur_guides now works with the sampler's alg_enabled off: the anchor
stays sharp and only the newly-appended guide-attention frames (mid-scene
guide, carried guides, per-scene guides, JoyAI memory) are blurred, reusing
the same alg_strength / alg_sigma_threshold parameters and sigma window.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bulb's popover now mixes what the editor knows about YOUR prompting habits
with library browsing, in four sections:
- You've paired these before — shortcuts you historically put in the same shot
as what's in the prompt, ranked by co-occurrence (reason chip: with «X» ×N)
- Your usual next shot — what usually follows the previous scene's shortcuts
in timeline order (scene inspector only; reason chip: after «X» ×N)
- Continue the thread — more from categories the prompt already uses
- New directions — the existing missing-category browser (counts, +N more)
Habits are mined on demand by GET /api/library/suggestion_stats: a text scan of
all saved projects' scene texts (triggers stay verbatim in scene texts, so no
new persistent state — project files remain the only source of truth). Pairs =
same scene, follows = consecutive scenes in timeline order; editorial cut
subclips and excluded scenes are skipped. One-off co-occurrences (<2) ignored.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
📌 hover action on image cards sets/unsets continuity_settings.identity_pin_ref
(same setting as Engine settings ▸ Continuity ▸ Identity pin, just quicker).
The pinned card shows a persistent 📌 badge on its thumbnail. Pinning while
Auto continuity is explicitly off re-enables it — a pin that does nothing
would be a hidden setting. Image cards only, chain-sampler pipelines only.
fpMediabrowser now fingerprints identity_pin_ref so pin changes repaint the
gallery zone (they were dropped by the ViewBus fingerprint before).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A 💡 next to the scene prompt and Composer global prompt lights up when whole
shortcut categories (camera moves, sub-actions, …) exist in the library but are
absent from the prompt. Click-only (never auto-opens): a category-grouped
popover lists every missing category with count, random insertable samples,
+N-more browse per category, and a shuffle. Chips insert at the caret through
the field's normal input path. Toggleable via Settings > Editor > Shortcut ideas.
Guards: no popover off a hidden/detached bulb (stranded top-left window),
dismiss listeners detach on every close path, popover closes when a re-render
removes the bulb.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fast pause/play over the network can abort the stream fetch and reset the
<video> element (the retry path's cache-busted load() does the same): the
element rewinds to 0, the tick stomped the playhead to the clip start, and the
next play restarted from the beginning. Now 'emptied' stashes the playhead as a
pending seek, loadedmetadata restores it (and resumes when playing), and the
tick treats readyState<1 as seeking so a resetting element can't feed a bogus
currentTime. Reproduced+verified via forced mid-playback load(): position holds
while playing, mid-burst, and while paused.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The main Settings sidebar collapses to a 52px icon rail and expands to full
width on hover/focus, overlaying the content (no reflow). Node values render in
one column instead of two, Engine's Conditioning/Sampler pickers stack, and the
wiring/fields shrink chain got min-width:0 everywhere a long select option used
to force horizontal overflow — the node page header actions (bypass/remove) now
sit in a wrapping cluster and stay reachable at any width (live-instance bug:
remove button was unreachable, no horizontal scroll).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
splitScene set frames on both halves but left frames_mode='project', which
sceneEffFrames (and backend eff_frames) IGNORE - so the first half kept full
project length (looked untouched) and the second half rendered as a whole
extra scene. Both halves now flip to 'timeline' mode, same as resizeScene.
Also: with a pinned custom cut order, _ensureTimelineOrder appended the new
half's id at the END of timeline_order - the 'split adds a scene to the end'
symptom. The second half is now spliced into the cut order right after its
first half.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The FunPack menu and the Settings dropdown are gone; the top bar is
File / Edit / View / Help / Settings, where Settings is a plain button that
opens the unified window (keeps .menu-btn[data-menu=Settings] for the tour).
Everything they held already lives in Settings sections, plus the two
stragglers migrated: Reset Studio session (Refinement & Taste, keeps the
armed/cancel two-step + confirm and Store-driven label) and Import ComfyUI
Workflow (Models & Pipeline header action next to Refresh). Tour/FAQ texts
updated to the new paths. FunPackMaintenance/FunPackGit stay the single
implementations behind the Settings rows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One Settings window replaces the four scattered modals: searchable sidebar with
colored icon chips grouped as Editor / Generation (Engine, Models & Pipeline) /
Learning (Refinement & Taste) / System (Updates & ComfyUI, Temp Files). Sections
self-register on a small shell (settings_window.js); the legacy modal facades
(EditorSettingsModal/EngineSettingsModal/ModelsModal/TempBrowserModal) deep-link
into their section so every existing entry point still works. Refinement-key and
restart/update actions get proper panels reusing the single menubar
implementations (window.FunPackMaintenance). Cmd+, opens it; tour's overlay
sweep now closes the window through its cleanup hooks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'Awful' / 'Wrong appearance' banked on intrusion semantics; now any rating
whose profile reward marks the run a bad OUTCOME (<= -0.25: Missing
quality, Missing details+quality, Missing action+quality, Missing
details+action+quality) also feeds the negative bank - degradation is
visible in the latent, so steering off that attractor is meaningful.
Near-miss ratings with positive reward (Missing details +0.35, Missing
action +0.05) stay excluded: those latents are mostly-wanted content and
banking them would repel future runs from material the user asked for.
Decision moves into negative_memory.is_negative_profile (intrusion keys OR
reward threshold, robust to future profiles); promoted entries now carry
the rating key as provenance for severity-weighted steering later.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Store what went WRONG, not just what was asked for. Ratings 'Awful' and
'Wrong appearance' promote that run's final video latent (fp16, batch
squeezed) + mean-pooled conditioning into a per-key negative bank
(<key>.negative_latents.pt, ring buffer of 8). The Chain Sampler writes a
pending candidate at the end of every keyed run; the rating that scores
that run promotes or discards it - same pairing as the x0 snapshot, and
the stem glob in delete_refinement_key covers both new sidecars.
Sampling (dynashift toggle + strength/threshold, off by default):
per late step each current video frame is pooled and compared against
every banked negative frame (alignment-free in time, so chain position /
guide tails don't matter; other resolutions are skipped). Frames matching
above threshold get the matched negative frame's aligned component
subtracted (projection removal, coef clamped >= 0), scaled by
above-threshold margin - steering self-releases once the unwanted feature
is gone. Negatives are weighted by prompt similarity so bad gens from
unrelated prompts steer less. No uncond pass, no extra model forward;
audio byte-identical (video span slice + cat). Wrapper rides the tagged
per-scene lifecycle (leak-strip covered) and threads through batch
training.
Editor: DynaShift knobs land in Engine Settings > Guidance same-change
(sampler_inputs pass-through, no backend change), verified rendering live.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two audit findings on the experimental feature stack:
1. Leak: an interrupt or OOM inside _sample_chunk skipped the per-feature
unwind blocks, leaking embed_guidance/score_slider/output_guidance/temporal
wrappers on the shared ModelPatcher - every later run then chained new
wrappers onto the leaked ones and double-steered (same in-process
progressive-corruption mode as the old block-hook leak).
- one base-wrapper snapshot per scene, restored in the same finally that
removes the v2a hooks; the four conditional unwind blocks are gone
- every per-scene wrapper is tagged with its predecessor; sample() strips
leaked tagged chains at run start (mirrors strip_funpack_block_hooks)
2. Inert feature: output_guidance applied the RAW value-fn gradient, whose
per-element magnitude is O(1/pool_window) - at video-latent sizes the
correction was ~1e-6 relative, i.e. numerically a no-op. It also fed the
full packed AV tensor to a value function trained on the video stream
alone (shifted pooling buckets + audio contamination).
- gradient now computed on the packed latent's video span only (matches
the training domain of _save_output_value_snapshot)
- delta norm-calibrated: strength = fraction of video-stream norm per
fully-ramped step (0.02 = 2%), same relative convention as the slider
- audio region byte-identical by construction (cat, not post-hoc mask)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-scene line now includes run_mechanisms, so a slow guide scene
attributes itself fully: which feature appended the tokens
(custom_guide_stack / carry_i2v_guides / joyai_memory / mid_scene_guide
/ soft_continuation) alongside the tail counts and sampling seconds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live report: guide scenes suddenly ~3.5x slower than the anchor scene
(6-scene 768x768x305 montage at 7 min vs ~2.5 expected), on a nightly
ComfyUI. Audit of the upstream delta found nothing touching the default
guide attention path (context windows is opt-in, int8 irrelevant, the
guide-strength mask machinery dates to February) - so before theorizing
further, make each run attribute itself: every scene line now carries
its sampling seconds plus what was appended to its sequence (carried
head frames, guide tail, audio tail), printed live and in the report.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Block steering: run-averaged block activity is architecture-determined
(varies in the 4th decimal run-to-run), so no scoring scheme - pole EMAs
or reward correlation - could extract a rating signal from it. The idea
needs within-run per-step measurement to have a chance; that is a
research project, not a feature. Removed: module, sampler wiring, the
two trailing widgets (positional reference mapping unaffected), the
Refiner pairing hook, editor knobs, tests. Stale block_snapshot/
block_profile sidecars are inert and covered by the existing
delete_refinement_key sweep.
FunPackNVFP4ModelLoader: the FP4 GEMM path was real (3.35x at 16k
tokens) but end-to-end the user's runs are dominated by everything the
loader cannot touch (attention, prep, encode - the sampler node is
~23s of a ~60s run) while dynamic input quantization added a fixed
per-layer-call toll. Net observed: overhead. Users are better served
by dedicated NVFP4 loaders. Removed: module, registration, tests.
Kept (independently useful): the chain sampler's phase timing report +
quantized-layer census, the nested-AV x0 snapshot fix, and the sm_120
masked-attention work.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Justified by live data: sampler phase timing shows sampling at 89% of
the chain node, all 792 quantized layers on the fast path, and the probe
puts dynamic input quantization at ~0.30 ms/layer-call vs ~0.11 ms with
a provided scale (amax dominates) - a top-line per-call cost.
New input_scales widget on the loader:
- "auto-calibrate", first load without a sidecar: forward-pre-hooks
record each quantized layer's running input amax (one generation is
enough; sidecar under refinements/nvfp4_scales/<checkpoint>.json,
per-machine and rebuilt in one run whenever absent).
- every later load: injects input_scale = amax * 1.25 / (448*6) into
the state dict next to each quantized weight - core's loader
registers it natively and skips the per-call amax from then on.
Margin leaves clip headroom (NVFP4 clips above the scale).
Tests: suffix-matched injection across checkpoint prefixes, value math,
core Linear registering input_scale and forwarding, sidecar read.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live report: "flat activity across runs" with a working profile. Root
cause is real, not a threshold tuning issue: the fingerprint is each
block's MEAN residual activity over ~1200 steps, which is practically a
property of the architecture - run-to-run variation sits in the 4th
decimal, so raw reward-covariance is numerically dead. Lowering the
epsilon would have "fixed" it by steering on noise.
Now: per-block Pearson CORRELATION between reward and activity (scale-
free - tiny variations count when they consistently track ratings),
fingerprints stored mean-normalized so run-global scale (step counts,
schedulers) can't masquerade as block signal, and a confidence ramp
(min(1,(runs-2)/6)) so a 3-run profile steers gently instead of not at
all, reaching full strength ~8 runs. MIN_RATED_RUNS 2->3 (correlation
over 2 runs is degenerate). Status lines now carry the evidence:
"ready (N runs, peak |corr| X, confidence Y)". Older profiles are
discarded on first pairing (fingerprint semantics changed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"The matmuls got faster" and "the video arrives sooner" are different
claims. The run report and log now show total = sampling + VAE decode +
other with percentages, so an optimization's real ceiling is visible
(at 768x768x305 the ~22k-token attention rivals all linears combined,
and decode/prep are NVFP4-blind - a 2x on quantized GEMMs can cap out
around seconds of a one-minute run).
Also logs a quantized-layer census at run start: layers forced OFF the
quantized matmul path by weight patches (LoRA / refiner attn deltas) or
emulation dequantize on EVERY call - silently slower than bf16.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live numbers from the rental (sm_120, cu130): FP4 GEMM wins 3.35x at
M=16384 (1.97x net) but loses at M=1024 to launch overhead, and input
quantization is a near-fixed ~0.2-0.3 ms per layer-call. With scope
'all blocks' (1232 layers incl. the small audio branch) that fixed toll
cancels the GEMM savings - the exact "speed didn't change" report.
The probe now: times input quantization with a precomputed static scale
too (tells whether input_scale calibration would reclaim the overhead or
it's construction/launch cost), and draws the verdict from the LARGE-M
run only, with explicit advice to prefer scope 'video blocks' when the
fast path is active but end-to-end looks flat.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First live diagnostics run showed the whole quantized path at 0.26x of
bf16 despite a healthy cuda backend and zero emulated layers. The single
combined timing cannot say WHICH half loses: the dynamic input
quantization wrapped around every matmul (a fixed per-call cost times
1232 layers/step) or the FP4 GEMM kernel not engaging. The probe now
times them separately at M=1024 and M=16384 (realistic video token
count) and names the culprit in the verdict.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'Loaded fine' and 'runs fast' are different claims. Every NVFP4 load now
prints a diagnostics block that settles which one you have:
- comfy_kitchen backend census (the big one: torch built for CUDA < 13
disables ck's CUDA backend, so quantized matmuls silently fall through
to the pure-PyTorch eager backend - correct output, ZERO speedup, with
an explicit warning line when detected)
- torch/cuda build + device capability
- loaded-model layer census: nvfp4 layers vs layers flagged
full-precision emulation (pre-Blackwell GPUs)
- a timed 4096x4096 GEMM probe (incl. dynamic input quantization):
bf16 vs nvfp4 wall time with a FAST PATH ACTIVE / NO SPEEDUP verdict
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three live-run findings from the rental:
1. Block steering said "profile not ready (needs 2+ rated runs)" with 16
paired runs. The v1 liked/disliked-EMA attribution had no contrast when
ratings cluster on one pole (pole-vs-all-runs-mean cancels to zero) and
dropped rewards in (0, 0.3) entirely. Attribution is now the per-block
COVARIANCE between reward and activity over a rolling 32-run history:
every admissible reward contributes with its actual value, contrast
comes from reward variance, and the sampler log now states the real
blocker ("all rated alike - rate both good AND bad runs") instead of a
misleading run count. v1 profiles are discarded on first pairing.
2. output_guidance stayed "not ready (needs 10+)" forever on LTX-AV: the
end-of-run x0 snapshot was gated on output samples being a plain
torch.Tensor, but AV chain output is a NESTED tensor (video+audio) - so
the output value function never received a single sample. The snapshot
now extracts the video stream from nested outputs.
3. FunPackNVFP4ModelLoader hit a size-mismatch wall (9-vs-6 ada params,
connector dims): it loaded the state dict without the safetensors
METADATA that carries the LTX-AV architecture config, so detection
built the wrong model variant. Metadata now flows through, mirroring
comfy.sd.load_diffusion_model.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'No, I'll use my own pipeline' in the missing-nodes prompt silently
PERSISTED disable_core on the project. From then on Engine settings…
greyed out with no reason and Generate died with ComfyUI's raw
"Prompt has no outputs" - both looked like breakage with no visible cause.
- The setup prompt button now confirms first, spelling out what it
disables, what Generate will need, and where to re-enable it.
- Engine settings… never greys over pipeline state anymore: the modal
itself already explains why knobs are unavailable and links to Models.
- Builder: disable_core with NO output node in the graph is now a
BLOCKING error with the fix in the message (wire 🌐 Global video
output, or Models → Enable built-in pipeline) instead of letting
ComfyUI reject the queue.
- GEN_ERROR_MAP: prompt_no_outputs mapped to the same actionable text
for custom pipelines that pass the builder but lack an output node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'What it looks like' section right under the intro: a clean full-workspace
hero shot plus Composer and Engine Settings side by side, generated from
the mocked tour demo (no ComfyUI/GPU needed) by a new
tools/showcase/capture_clean.mjs (npm run capture:clean) - same scaffolding
as capture.mjs but with the tour overlay stripped, so shots are undimmed
and card-free. Images committed under docs/images/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All three buttons are now same-shape shields.io for-the-badge badges
(Ko-fi's rounded SVG pill and Patreon's PNG clashed with the flat
Vast.ai badge), and an IMPORTANT callout right under them sets
expectations: one-person project, provided as is, bug reports welcome.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RLHF at the model level: which REGIONS of the transformer were doing the
work when a generation came out good or bad, and steer accordingly.
1. Measure: tagged forward hooks record each block's relative video-residual
delta per model call (near-zero cost; covered by the existing leak-strip).
2. Attribute: the end-of-run fingerprint is staged (block_snapshot.pt, same
stage-then-pair convention as x0_snapshot) and paired with the rating
reward on the next Refiner cycle into liked/disliked/all-runs EMA
profiles; when output_guidance's value function is trained, an innermost
model-function wrapper additionally credits blocks per step by
(value delta) x (centered activity) - converges in a handful of runs.
3. Steer: vx' = vx_in + g_b*(vx_out - vx_in) on the video stream only,
audio untouched; scores zero-mean/unit-max so emphasis is redistributed
rather than inflated, gains hard-capped at +/-10%. Exact no-op until the
profile has 2+ paired runs; blocks at gain 1.0 get no hook at all.
Chain Sampler toggle + strength (append-only widgets), exposed same-change
in the editor's Chain Sampler Guidance section via the sampler_inputs
bridge. New sidecars ride the defensive <key>.* sweep in
delete_refinement_key, so atomic key deletion covers them.
Tests: toy AV-block model driving the real hooks (recorder ranking, steer
video-only scaling + native path for unit gains, credit wrapper, snapshot/
profile/scores/gains roundtrip, mild-positive band, depth-mismatch reset,
leak-strip tagging). Full suite failure set identical to the pre-existing
baseline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FunPackNVFP4ModelLoader quantizes an existing bf16/fp16 checkpoint to NVFP4
at load time as a pure state-dict transform: selected 2D Linears are
re-serialized in ComfyUI core's quantized-checkpoint format (packed fp4
weights + fp8-e4m3 block scales + fp32 tensor scale + per-layer comfy_quant
metadata) and handed to the stock loading path. Core's MixedPrecisionOps
then does everything: real FP4 tensor-core GEMMs on sm_100+ (sm_120 incl.),
dynamic activation scales (no calibration pass), per-layer LoRA fallback,
and dequant emulation on older GPUs. No custom forward, no monkeypatching.
Zone-aware precision map for the joint AV transformer: default scope
quantizes only video-branch Linears (attn1/attn2/ff) of the middle blocks;
the audio branch, the audio<->video cross-modal bridges, the first/last N
blocks, and all non-matmul weights stay full precision. Scopes widen to
+cross-modal, all blocks, and all 2D layers for experimentation.
Tests drive the REAL comfy.quant_ops/comfy.ops via comfy_kitchen's eager
backend on CPU: scope selection, serialized-format contract, prefix
handling, refusal of pre-quantized checkpoints, and a load+forward
roundtrip through MixedPrecisionOps on both the NVFP4 and emulation paths.
The Movie Editor Models modal picks the node up automatically as a
Unet/Diffusion Model candidate (object_info-driven).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Composer search: filter keystrokes rebuilt the whole window and dropped
focus after every character; now only the list refreshes
- Global prompt: stale funpack-global-prompt-updated events (preview refresh
pinning the LAST completed apply) rewound the Compose textarea while newer
text was still debouncing/mid-parse; overlapping applies could also land
out of order. Added pending-apply guard + apply sequence token
- Composer crashed the editor when opened with no project (welcome screen)
- /result now streams from disk via FileResponse (Range/seek support) instead
of buffering whole videos through a loopback /view fetch that crawled or
aborted while ComfyUI was saving the next run; added '..' traversal guard
- Player keeps a URL-keyed video pool: one failed load left the clip dead
until page reload; now bounded cache-busted retries + recovery on rebuild
- Render validation deleted (and autosaved away!) sceneRenders on ANY HEAD
failure; now only a definitive 404 drops a render
- Temp browser / media bin / filmstrip / duration probes held stalled media
connections open, exhausting Chrome's 6-per-origin pool so /view tabs and
even generate calls hung until reboot; thumbnails snapshot to canvas and
every probe releases its src when done
- Inspector guide frame/apply/strength inputs lacked data-k typing protection
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The free video-only unsharp-on-x0 sharpening (temporal high-pass between
consecutive denoised predictions, no extra model eval) only existed on
Hybrid Euler 2S. Distilled Flow — the recommended sampler for this
pipeline (deterministic, clean audio) — had no equivalent, so the
"quality phase" of the actual production sampler had no sharpening path
at all.
Ports the same _apply_quality_sharpness call into
sample_funpack_distilled_flow, gated to the existing final-correction
(Heun) steps. Threaded through FunPackDistilledFlowSampler's
INPUT_TYPES/get_sampler, Studio's _build_one_sampler, and the Editor's
sampler_panel.js Distilled Flow section — verified live end-to-end
(UI -> studio_settings JSON -> backend build call).
Replace absolute reward regression with Bradley-Terry pairwise ranking
(DPO-style) so the value function learns relative preference between
sampled pairs rather than fitting noisy absolute reward floats.
Add a 3-member bootstrap ensemble; gradient() and ascend() scale by
inter-member agreement so guidance backs off in regions the ensemble
hasn't seen data for, instead of confidently extrapolating (the
static-character collapse this was meant to guard against).
Loosen the "Wrong but loved" exclusion in conditioning.py so a strong
loved_modifier signal is no longer filtered out by skip_value_function
- "Wrong but loved" and "Missing but loved" now train the value
function, per the stronger-signal precedence requested.
Legacy single-net checkpoints migrate transparently on load (weights
remap into all ensemble members, rating buffer preserved intact).
The sampler-side toggle landed as a node widget only, with no Editor UI
control — caught after the fact. Editor UI takes priority over node
widgets for any new toggle going forward. No backend change needed: the
existing generic sampler_inputs bridge (builder.py:338) already forwards
any declared SAMPLER_KNOBS entry onto the Chain Sampler node.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New LatentValueFunction (value_function.py) mirrors OnlineValueFunction but is
trained on a pooled snapshot of the sampler's own predicted output (x0_hat)
instead of the input conditioning, borrowing NoiseTilt's core idea (reward
gradient from the model's own prediction) without its SDE noise-term trick,
which has no analogue in this deterministic-ODE sampler.
New output_guidance / output_guidance_strength toggle on the Chain Sampler
(off by default): per-step correction installed as the outermost
model_function_wrapper (on top of embed_guidance/score_slider), same
near-zero-forward-pass shape as embed_guidance. End of run writes a small
pooled snapshot per refinement key; the next rating cycle trains the value
function on it via a new call site in conditioning.py.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The composer was already being pre-selected on click, but nothing scrolled
it into view or signaled the change — on a fresh project all essentials
show as missing and push the composer off-screen, so the click looked
like a no-op. Now scrolls the composer into view and flashes it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
