nodes/olivv-cs-ComfyUI-FunPack
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix: multi-scene chains work with a resolution-changing second pass

Browse Source 
second_pass_op="upscale_2x" hands a scene back at twice the latent size while
every LATER scene is still built from latent_template at the original size.
Everything that crosses a scene boundary — carried overlap frames, the anchor's
continuation, the soft join, the JoyAI memory frame, per-scene guide sources —
is spliced into a chunk on the template's grid, so the second scene died on the
shape mismatch and the combination was simply unavailable.

One choke point, _match_template_resolution, brings the carried material back
before it is used, applied at the three places a finished scene crosses into the
next one. Each scene still samples and OUTPUTS at 2x: only what is carried gets
resampled, and only downwards, which is the direction that survives it — those
frames exist to say "continue from here", which a resample preserves far better
than invented detail would.

Identity no-op when the grids already agree, so any run without a resizing op is
bit-identical. Audio has no spatial axes and is never touched. A stale
noise_mask is dropped rather than reapplied at the wrong scale, and a resample
that throws costs continuity with a console line, not the run.

No grid guard on the chain blend: second_pass and second_pass_op are run-level
settings applied to every scene, so a chain cannot hold two resolutions — every
scene is upscaled or none is. Both tooltips said this combination dropped
multi-scene; they now say how it works.

Also: About reads "FunPack 3" with the codename on its own line, rather than
attaching the major to the codename.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
digital-garbage
2026-08-12 00:05:52 +03:00
co-authored by Claude Opus 5
parent 06c3f75080
commit 7329cd9680
5 changed files with 267 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+14
View File
@@ -2,6 +2,20 @@
## [Unreleased]
### Fixed
- **Multi-scene chains work with a second pass again.** `second_pass_op="upscale_2x"` hands
back a scene at twice the latent size, but every later scene is still built from the latent
template at the original size — so the carried overlap frames, the anchor's continuation,
the soft join, the JoyAI memory frame and per-scene guide sources were all being spliced
into a chunk on a different grid, and the second scene died on the shape mismatch. Anything
crossing a scene boundary is now brought back to the template's grid on the way. Each scene
still samples and outputs at 2x; only the carried material is resampled, and only downwards
— the direction that survives it, since those frames exist to say "continue from here"
rather than to carry detail. Runs with no resolution-changing op are bit-identical.
- The About panel reads **FunPack 3 "Auspicious Asparagus"** rather than putting the major
on the codename line.
## [3.5.1] "Auspicious Asparagus" - 2026-08-11
FunPack picks up release codenames, Ubuntu-style: an adjective and a vegetable sharing an
movie_editor/frontend/engine_settings.js
+1 -1
View File
@@ -311,7 +311,7 @@
detail: "Let the i2v anchor do its work, then cut it out of the clip. The anchor is a pinned frame at position 0, so it carries character detail, style and composition better than anything that weakens it on the way in (ALG blurs it and loses detail; Best-FaceID tokens approximate it and lose some too) — but it is also literally the first frame you see. The scene is generated exactly as normal, with the anchor pinned at full strength the whole way and no extra sampling, and this many frames are then dropped off the FRONT of the finished clip: an i2v generation that reads as t2v. 0 = off. 8 (one latent frame) removes just the anchor itself, which is usually not enough — the anchor is followed by a settling-in stretch where the shot is still leaving the reference still and little is happening, and on a prompt asking for immediate action that dead time is exactly what you want gone. 48 was the value that worked on a 768×768×305@30 chain with a quick-cut prompt; treat it as a starting point for this pipeline, not a universal default. NOTHING IS REGROWN: the scene comes out that much SHORTER than the length you set, and the audio is cropped to match — every surviving frame was generated as part of one continuous shot, with no invented ending. Needs an anchor image; skipped (with the reason in the scene report) on continuation scenes and on scenes carrying guide frames or JoyAI audio memory. On MiniMax H3 the cut happens on the decoded frames instead of the latent (H3's anchor is a keyframe pin, and its latent grid can't express an arbitrary cut) — exact to the frame, audio cropped to match, and only the chain's opening is cut rather than each scene's." },
{ name: "second_pass_op", label: "Between-pass operation", kind: "combo", choices: ["none", "sharpen", "upscale_2x"], default: "none",
hint: "'sharpen' adds detail almost free; 'upscale_2x' doubles the output resolution at 3-5x the cost of pass 2.",
detail: "OPTIONAL operation applied to the latent between the two passes — 'none' by default, nothing runs unless you pick one. Both operations need the LTX 2.3 spatial upsampler in models/latent_upscale_models — the same ~1 GB file segmented detailing uses, found automatically or downloaded once on first use (watch the ComfyUI console). If it can't be obtained the second pass still runs, with the operation skipped and the reason in the scene report. 'sharpen': one forward of Lightricks' trained 2x latent upsampler, resampled straight back to the original size. No video-model calls, so it costs a fraction of a step; pass 2 then re-denoises the sharpened latent, which is what makes it stick. It adds detail consistent with what's already there and CANNOT fix wrong structure (an extra finger stays an extra finger, just sharper). 'upscale_2x': the same upsampler, but kept at 2x — pass 2 runs at four times the pixels and the scene decodes at double resolution. That's 3-5x the cost of the second half, and it drops the i2v pin (the anchor and its mask are the old size, and rescaling them would be inventing an anchor), so the scene can drift from the reference image; the scene report says when that happens. Both use the same upsampler file as segmented detailing. Video only — audio is never reshaped." },
detail: "OPTIONAL operation applied to the latent between the two passes — 'none' by default, nothing runs unless you pick one. Both operations need the LTX 2.3 spatial upsampler in models/latent_upscale_models — the same ~1 GB file segmented detailing uses, found automatically or downloaded once on first use (watch the ComfyUI console). If it can't be obtained the second pass still runs, with the operation skipped and the reason in the scene report. 'sharpen': one forward of Lightricks' trained 2x latent upsampler, resampled straight back to the original size. No video-model calls, so it costs a fraction of a step; pass 2 then re-denoises the sharpened latent, which is what makes it stick. It adds detail consistent with what's already there and CANNOT fix wrong structure (an extra finger stays an extra finger, just sharper). 'upscale_2x': the same upsampler, but kept at 2x — pass 2 runs at four times the pixels and the scene decodes at double resolution. That's 3-5x the cost of the second half, and it drops the i2v pin (the anchor and its mask are the old size, and rescaling them would be inventing an anchor), so the scene can drift from the reference image; the scene report says when that happens. Multi-scene projects work with it: a scene finishes at 2x while later scenes are still built from the template at the original size, so everything crossing a scene boundary — carried overlap, the anchor's continuation, JoyAI memory, guide sources — is brought back to the template's grid on the way. Scenes still sample and output at 2x; only the carried material is resampled. Both use the same upsampler file as segmented detailing. Video only — audio is never reshaped." },
{ name: "context_windows", label: "Context windows (long scenes)", kind: "bool", default: false,
hint: "Renders very long scenes as overlapping windows instead of one pass. Slower short, faster past ~300 frames.",
detail: "EXPERIMENTAL: denoise a scene that's LONGER than the model's comfortable window as overlapping windows instead of one giant pass — ComfyUI core's own mechanism, audio-aware on LTX (it maps each video window to its audio window and re-slices anchors, guides and JoyAI memory per window). Engages only on scenes longer than the window length below; shorter scenes are untouched and pay nothing. Cost at the defaults (145/40) is about 1.45× the per-frame work, since each window re-does its 40-frame overlap — offset by attention getting cheaper the longer the scene is (quadratic in one pass, near-flat when windowed). Roughly break-even around 200 frames, a net win past ~300. Needs ComfyUI v0.29.0 or newer; on older builds it's skipped with a note in the scene report. UNVALIDATED LIVE." },
movie_editor/frontend/settings_window.js
+8 -7
View File
@@ -245,15 +245,16 @@
clear(wrap);
const git = G ? G.get() : null;
wrap.append(el("div", "sw-about-mark", "◉"));
wrap.append(el("div", "sw-about-name", "FunPack"));
// The major belongs on the NAME — this is FunPack 3, and the codename is what 3 is
// called. Splitting them the other way ("FunPack" / "3 Auspicious Asparagus") read
// as if the number belonged to the codename.
const major = String(git?.version || "").split(".")[0];
wrap.append(el("div", "sw-about-name", "FunPack" + (major ? " " + major : "")));
// Ubuntu-style release codename, carried on the git status alongside the version.
// Keyed by major, so it reads "3 “Auspicious Asparagus”". Omitted entirely when the
// backend is older than the field or the major has no name yet — an empty quoted
// string would look like a bug.
// Omitted entirely when the backend predates the field or the major has no name
// yet — an empty pair of quotes would look like a bug.
if (git?.codename) {
const major = String(git.version || "").split(".")[0];
wrap.append(el("div", "sw-about-codename",
(major ? major + " " : "") + "“" + git.codename + "”"));
wrap.append(el("div", "sw-about-codename", "“" + git.codename + "”"));
}
// window.FunPackAppName lets a different frontend sharing this file (e.g. Easy
// Gen) relabel the app name without forking the whole section; Editor leaves it
samplers.py
+90 -7
View File
@@ -2863,7 +2863,7 @@ class FunPackLTXAVSceneChainSampler:
}),
"second_pass_op": (["none", "sharpen", "upscale_2x"], {
"default": "none",
"tooltip": "OPTIONAL latent-space operation applied between the two passes — 'none' by default, nothing runs unless you pick one. 'sharpen': one forward of Lightricks' trained 2x latent upsampler, resampled straight back to the original size. No video-model calls at all, so it costs a fraction of a step; pass 2 then re-denoises the sharpened latent, which is what makes it stick. It adds detail consistent with what is already there and CANNOT fix structure that is wrong (an extra finger stays an extra finger, just sharper) — the same limit segmented detailing's sharpen mode documents. 'upscale_2x': the same upsampler, but the result is KEPT at 2x, so pass 2 runs at four times the pixels and the scene decodes at double resolution. That is 3-5x the sampling cost of the second half and it drops the i2v pin (the anchor and its mask are the old size, and rescaling them would be inventing an anchor) — the scene report says so when it happens. Both use the same upsampler file as segmented detailing (detail_upsampler, 'auto' downloads the official one on first use). Video stream only; audio is never reshaped.",
"tooltip": "OPTIONAL latent-space operation applied between the two passes — 'none' by default, nothing runs unless you pick one. 'sharpen': one forward of Lightricks' trained 2x latent upsampler, resampled straight back to the original size. No video-model calls at all, so it costs a fraction of a step; pass 2 then re-denoises the sharpened latent, which is what makes it stick. It adds detail consistent with what is already there and CANNOT fix structure that is wrong (an extra finger stays an extra finger, just sharper) — the same limit segmented detailing's sharpen mode documents. 'upscale_2x': the same upsampler, but the result is KEPT at 2x, so pass 2 runs at four times the pixels and the scene decodes at double resolution. That is 3-5x the sampling cost of the second half and it drops the i2v pin (the anchor and its mask are the old size, and rescaling them would be inventing an anchor) — the scene report says so when it happens. MULTI-SCENE works with it: a scene finishes at 2x while every later scene is still built from the latent template at the original size, so everything that crosses a scene boundary (carried overlap frames, the anchor's continuation, the soft join, JoyAI memory, per-scene guide sources) is brought back to the template's grid on the way. Each scene still samples and OUTPUTS at 2x — only the carried material is resampled, and only downwards, which is the direction that survives it: those frames exist to say 'continue from here', which a resample preserves far better than invented detail would. Both ops use the same upsampler file as segmented detailing (detail_upsampler, 'auto' downloads the official one on first use). Video stream only; audio is never reshaped.",
}),
"h3_audio_clock": ("BOOLEAN", {
"default": False,
@@ -3135,6 +3135,68 @@ class FunPackLTXAVSceneChainSampler:
count += 1
return count
def _match_template_resolution(self, latent, template):
"""Resample a finished scene's VIDEO stream back onto `template`'s spatial grid.
This is what makes multi-scene work with a second pass. A resolution-changing
second_pass_op (upscale_2x) hands back a scene at twice the size, but every LATER
scene is still built from `latent_template` at the original size — so the carried
overlap frames, the anchor's continuation and the JoyAI memory frame no longer fit
the chunk they are spliced into, and the chain died on the shape mismatch. Bringing
the carried material back to the template's grid is the whole fix: each scene still
SAMPLES and OUTPUTS at the upscaled size, only what crosses a scene boundary is
brought back.
Downscaling is the right direction to lose information in: the carried frames exist
to say "continue from here", and that survives a resample far better than the detail
the second pass added would survive being invented at the wrong scale.
Audio has no spatial axes and is never touched. Returns the input object unchanged
when the grids already agree, so any run without a resolution-changing op is
bit-identical.
"""
try:
tensors = self._latent_tensors(latent)
video = tensors[0]
th, tw = self._latent_tensors(template)[0].shape[-2:]
except Exception:
return latent
if video.shape[-2:] == (th, tw):
return latent
try:
try:
from .detailing import _downscale_to
except ImportError:
from detailing import _downscale_to
if video.shape[-2] >= th and video.shape[-1] >= tw:
resized = _downscale_to(video, int(th), int(tw))
else:
# Not reachable from any current op (upscale_2x only grows), but a scene
# SMALLER than the template would corrupt the splice just as surely, so
# handle it rather than pass a wrong shape through. Antialiasing is a
# downscale-only concept, hence the separate call.
b, c, f, _h, _w = video.shape
resized = torch.nn.functional.interpolate(
video.permute(0, 2, 1, 3, 4).reshape(b * f, c, _h, _w),
size=(int(th), int(tw)), mode="bicubic", align_corners=False,
).reshape(b, f, c, int(th), int(tw)).permute(0, 2, 1, 3, 4)
except Exception as exc: # noqa: BLE001
print(f"[FunPackSceneChain] could not bring the previous scene back to the "
f"template's {th}x{tw} grid ({type(exc).__name__}: {exc}) — this scene's "
f"continuity (carried frames / anchor overlap / JoyAI memory) will fail.")
return latent
result = self._clone_latent(latent)
out = self._latent_tensors(result)
out[0] = resized.to(device=video.device, dtype=video.dtype)
if self._is_nested(result.get("samples")):
result["samples"] = comfy.nested_tensor.NestedTensor(out)
else:
result["samples"] = out[0]
# A mask describes the OLD grid; the caller rebuilds one for the chunk it is
# splicing into, and a stale one here would be applied at the wrong scale.
result.pop("noise_mask", None)
return result
def _build_continuation_chunk(self, template, previous, video_overlap):
chunk = self._clone_latent(template)
chunk_tensors = self._latent_tensors(chunk)
@@ -6438,6 +6500,17 @@ class FunPackLTXAVSceneChainSampler:
identity_ref_filename = None
run_mechanisms: list = []
anchor_meta = (scene_anchors or {}).get(str(scene_index))
# Everything that crosses a scene boundary (carried overlap, the anchor's
# continuation, the soft join) comes off the finished chain, which a
# resolution-changing second_pass_op leaves on a different grid than the
# template this scene is built from. Bring it back once, here, rather than at
# each splice — identical object when no op resized anything.
_carry_source = output if output is None else self._match_template_resolution(
output, latent_template)
if _carry_source is not output:
run_mechanisms.append(
"second_pass_op resized the previous scene — its carried frames were "
"brought back to the template grid for this scene's continuity")
if output is None:
chunk = self._clone_latent(latent_template)
custom_guides = None
@@ -6463,7 +6536,7 @@ class FunPackLTXAVSceneChainSampler:
"effect": effect if effect and transition_duration > 0 else None,
})
chunk = self._build_mixed_anchor_chunk(
vae, anchor_meta, latent_template, output, video_overlap,
vae, anchor_meta, latent_template, _carry_source, video_overlap,
carry_overlap=carry_overlap_through_anchor,
)
if carry_overlap_through_anchor and video_overlap > 0:
@@ -6488,9 +6561,9 @@ class FunPackLTXAVSceneChainSampler:
"pixel_frame": max(0, boundary_pixel),
"effect": effect if effect and transition_duration > 0 else None,
})
chunk = self._build_continuation_chunk(latent_template, output, video_overlap)
chunk = self._build_continuation_chunk(latent_template, _carry_source, video_overlap)
if video_overlap == 0:
chunk, soft_carried = self._prepend_soft_continuation(chunk, output)
chunk, soft_carried = self._prepend_soft_continuation(chunk, _carry_source)
if soft_carried > 0:
run_mechanisms.append(f"soft_continuation({soft_carried})")
elif video_overlap > 0:
@@ -6998,16 +7071,26 @@ class FunPackLTXAVSceneChainSampler:
# Harvest from the clean, fully-cropped scene so injected memory tails never re-enter
# the bank. Scene 0 seeds the pinned anchor (num_fix); later scenes roll in. The audio
# half is harvested only when audio memory is on, else stored as None (video-only).
v_frame = self._harvest_joyai_frame(sampled, joyai_frame_select)
a_frame = self._harvest_joyai_audio(sampled, joyai_frame_select) if joyai_audio_memory else None
# The bank is injected into LATER scenes, which are built from the template,
# so a scene left upscaled by second_pass_op must be brought back first —
# otherwise the memory frame is the one thing in the chunk on the wrong grid.
_joyai_src = self._match_template_resolution(sampled, latent_template)
v_frame = self._harvest_joyai_frame(_joyai_src, joyai_frame_select)
a_frame = self._harvest_joyai_audio(_joyai_src, joyai_frame_select) if joyai_audio_memory else None
joyai_bank.add(v_frame, a_frame)
scene_outputs.append(self._clone_latent(sampled))
# Kept on the TEMPLATE's grid: this list exists to be spliced back in as guide
# frames for later scenes, and those chunks are built from the template.
scene_outputs.append(self._clone_latent(
self._match_template_resolution(sampled, latent_template)))
# Stays 0 for anchor scenes even with carry_overlap_through_anchor on: the post-sample
# slerp blend would reach into the anchor image's own leading frame (position 0 of
# `sampled`) and fade it against the previous scene's tail, undermining the hard cut.
# The carried frames beyond it are already seeded pre-sample (see
# _build_mixed_anchor_chunk), so no post-hoc smoothing is needed there.
blend_overlap = 0 if anchor_meta else video_overlap
# No grid guard here on purpose: second_pass and second_pass_op are run-level
# settings applied to every scene, so the chain cannot end up holding two
# resolutions. Every scene is upscaled or none is.
output = sampled if output is None else self._blend_latents(output, sampled, blend_overlap)
cumulative_latent_frames = self._tensor_frames(self._latent_tensors(output)[0])
scene_meta = self._scene_meta(scene_cond, scene_index)
tests/test_second_pass_multiscene.py
+154
View File
@@ -0,0 +1,154 @@
"""Multi-scene chains survive a resolution-changing second_pass_op.
`second_pass_op="upscale_2x"` hands back a scene at twice the latent size, but every later
scene is still built from `latent_template` at the original size. Everything that crosses a
scene boundary — the carried overlap, the anchor's continuation, the soft join, the JoyAI
memory frame, per-scene guide sources — is spliced into a chunk on the template's grid, so
before this fix the second scene died on a shape mismatch and multi-scene + second pass was
simply unavailable.
`_match_template_resolution` is the single choke point. What matters:
1. It brings a resized scene back to the reference grid, and leaves audio alone.
2. It is an exact no-op (same object) when the grids already agree, so every run that does
not resize anything stays bit-identical.
3. It never raises — a failed resample must not take the whole chain down.
"""
import sys
import types
from pathlib import Path
import pytest
import torch
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
for _name in (
"comfy", "comfy.k_diffusion", "comfy.k_diffusion.sampling",
"comfy.model_sampling", "comfy.nested_tensor", "comfy.sample",
"comfy.samplers", "comfy.utils",
):
sys.modules.setdefault(_name, types.ModuleType(_name))
sys.modules["comfy.nested_tensor"].NestedTensor = object
import samplers # noqa: E402
class _FakeNested(list):
is_nested = True
def unbind(self):
return list(self)
@pytest.fixture
def nested_stub():
comfy_mod = sys.modules["comfy"]
nested_mod = sys.modules["comfy.nested_tensor"]
saved = (nested_mod.NestedTensor, getattr(comfy_mod, "nested_tensor", None))
nested_mod.NestedTensor = _FakeNested
comfy_mod.nested_tensor = nested_mod
yield
nested_mod.NestedTensor = saved[0]
if saved[1] is None:
delattr(comfy_mod, "nested_tensor")
else:
comfy_mod.nested_tensor = saved[1]
def _node():
return samplers.FunPackLTXAVSceneChainSampler()
def _latent(h=16, w=24, frames=5):
return {"samples": torch.randn(1, 128, frames, h, w)}
def _av_latent(h=16, w=24, frames=5, audio=40):
return {"samples": _FakeNested([torch.randn(1, 128, frames, h, w),
torch.randn(1, 8, audio)])}
# ── the resample ────────────────────────────────────────────────────────────
def test_an_upscaled_scene_comes_back_to_the_template_grid():
out = _node()._match_template_resolution(_latent(32, 48), _latent(16, 24))
assert out["samples"].shape[-2:] == (16, 24)
def test_frame_count_and_channels_are_preserved():
"""Only the spatial axes move — a resample that changed the duration would desync the
carried frames it exists to make usable."""
out = _node()._match_template_resolution(_latent(32, 48, frames=7), _latent(16, 24, frames=5))
assert out["samples"].shape[:3] == (1, 128, 7)
def test_matching_grids_are_an_exact_no_op():
"""Identity, not equality: every run without a resizing op must be bit-identical, and
the cheapest proof of that is that nothing was rebuilt."""
latent = _latent(16, 24)
assert _node()._match_template_resolution(latent, _latent(16, 24)) is latent
def test_audio_is_never_touched(nested_stub):
src = _av_latent(32, 48, audio=40)
before = src["samples"][1].clone()
out = _node()._match_template_resolution(src, _av_latent(16, 24, audio=40))
video, audio = out["samples"].unbind()
assert video.shape[-2:] == (16, 24)
assert torch.equal(audio, before), "audio has no spatial axes to resample"
def test_a_stale_mask_is_dropped():
"""A noise_mask describes the OLD grid; the caller builds a fresh one for the chunk it
splices into, and keeping this one would apply it at the wrong scale."""
src = _latent(32, 48)
src["noise_mask"] = torch.zeros(1, 1, 5, 32, 48)
out = _node()._match_template_resolution(src, _latent(16, 24))
assert "noise_mask" not in out
def test_the_source_latent_is_never_mutated():
src = _latent(32, 48)
before = src["samples"].clone()
_node()._match_template_resolution(src, _latent(16, 24))
assert torch.equal(src["samples"], before)
def test_a_smaller_scene_is_grown_to_the_template():
"""Not reachable from upscale_2x, which only grows — but a scene SMALLER than the
template would corrupt the splice just as surely, so it is handled rather than passed
through at the wrong shape."""
out = _node()._match_template_resolution(_latent(8, 12), _latent(16, 24))
assert out["samples"].shape[-2:] == (16, 24)
def test_constant_input_survives_the_resample():
"""The carried frames are spliced next to real generated ones, so the resample has to be
energy-preserving in the DC sense — a flat region must not come back brighter or darker."""
flat = {"samples": torch.full((1, 128, 3, 32, 48), 0.7)}
out = _node()._match_template_resolution(flat, _latent(16, 24))
assert torch.allclose(out["samples"], torch.full_like(out["samples"], 0.7), atol=1e-4)
# ── it must never take the chain down ───────────────────────────────────────
def test_an_unreadable_latent_returns_unchanged():
junk = {"samples": "not a tensor"}
assert _node()._match_template_resolution(junk, _latent(16, 24)) is junk
def test_an_unreadable_template_returns_unchanged():
latent = _latent(32, 48)
assert _node()._match_template_resolution(latent, {"samples": None}) is latent
def test_a_failing_resample_is_reported_and_survived(monkeypatch, capsys):
"""A resample that blows up must cost continuity, not the whole run — and must say so,
because silently carrying the wrong grid is how this became invisible in the first place."""
import detailing
monkeypatch.setattr(detailing, "_downscale_to",
lambda *a, **k: (_ for _ in ()).throw(RuntimeError("boom")))
latent = _latent(32, 48)
assert _node()._match_template_resolution(latent, _latent(16, 24)) is latent
assert "continuity" in capsys.readouterr().out