nodes/olivv-cs-ComfyUI-FunPack
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix anchor_shift under-denoising: pass 2 was re-noising a mid-trajectory latent

Browse Source 
Reported from the first live run: the idea works (opening reads as t2v,
character details preserved) but the output is full of noise artifacts, as if
under-denoised, with the last steps seemingly skipped. That was a real bug, not
the concept.

comfy's CONST scaling is x = sigma*noise + (1-sigma)*latent_image, and it
treats latent_image as a CLEAN image. Pass 2 was handed the PARTIALLY DENOISED
latent as latent_image, so at a 0.725 cut it entered at
0.725*fresh_noise + 0.275*picture — pass 1's content attenuated to a quarter
and buried under a full dose of new noise, with only the tail steps left to
recover. Exactly the reported symptom.

I cited segmented detailing's stage-2 tail as the precedent for this re-entry,
but that recipe feeds a FINISHED latent, where re-noising is correct. The
precedent does not transfer mid-trajectory. My error.

Fix — the two modes are now actually different things:
- CONTINUE (restart sigma 0, the default): pass 2 resumes from exactly the
  state pass 1 handed over. Passing that state as BOTH noise and latent_image
  collapses the CONST expression to an identity (s*x + (1-s)*x == x), so no
  noise is added at all. Cost still one full schedule.
- REWIND (restart sigma > 0): pass 1 now runs the WHOLE schedule to a finished
  clip first, the shift is applied to that, and pass 2 re-noises it upward and
  denoises down. That is the img2img recipe, and it is valid precisely because
  the input is finished. Costs a full schedule plus the tail.

Also fixed pass 2's progress offset, which restarted at pass 1's, so the bar
appeared to stall short of the end.

Two new tests reproduce the CONST arithmetic directly and assert the identity
holds in continue mode and does not in rewind mode. Their comfy stubs are now
snapshot/restored — the first version leaked into other modules and shifted the
suite off baseline (3 unrelated tests changed result). Back to baseline: 22
pre-existing failures, 272 passing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
digital-garbage
2026-07-29 22:32:33 +03:00
co-authored by Claude Opus 5
parent 83124a37ab
commit 9ed53c8346
3 changed files with 114 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
movie_editor/frontend/engine_settings.js
+1 -1
View File
@@ -233,7 +233,7 @@
{ name: "anchor_shift_frames", label: "Frames to drop", kind: "int", default: 8, min: 8, max: 512, step: 8, dependsOn: "anchor_shift",
hint: "How much of the front to drop at the shift, in real frames. 8 (one latent frame) removes just the anchor itself; more also removes the settling-in region where the shot is still visibly 'leaving' the reference still. Raise it if the opening still looks like it departed from a photo. The clip doesn't get shorter — whatever is dropped here is regrown at the end by pass 2." },
{ name: "anchor_shift_restart_sigma", label: "Pass 2 restart sigma", kind: "float", default: 0.0, min: 0.0, max: 0.999, step: 0.001, dependsOn: "anchor_shift",
hint: "Where pass 2 re-enters. 0 (default) continues from where pass 1 stopped, so the total cost stays one full schedule. Setting it higher than the shift sigma rewinds — pass 2 re-noises the shifted latent back up and re-denoises from there, giving more freedom to rebuild the regrown tail and blend the joint, at the cost of repeating those steps and partly overwriting what pass 1 established. Try it if the seam is visible or the new ending looks thin." },
hint: "Picks between the two modes. 0 (default) = CONTINUE: pass 1 stops at the shift sigma and pass 2 resumes from exactly that state with no re-noising — cost stays one full schedule. Anything above 0 = REWIND: pass 1 runs the WHOLE schedule to a finished clip, the shift is applied to that, then pass 2 re-noises it up to this sigma and denoises down. That's the standard img2img recipe and it is only valid on a finished latent, which is why pass 1 has to complete first. Rewind gives much more freedom to rebuild the regrown tail and blend the joint, and costs a full schedule plus the tail. Try it around 0.9 if continue leaves a visible seam or a thin ending." },
{ name: "anchor_shift_tail", label: "Regrown tail starts from", kind: "combo", choices: ["extend_last", "wrap", "empty"], default: "extend_last", dependsOn: "anchor_shift",
hint: "What fills the freed end after the slide. 'extend_last' (default) repeats the final frame so pass 2 grows the new ending out of something continuous with it. 'wrap' puts the dropped head back at the end — continuous, but it re-shows the reference image at the END of the clip. 'empty' zeroes it and lets pass 2 generate freely — most freedom, most likely to drift from the rest of the shot." },
{ name: "anchor_shift_fresh_audio", label: "Regenerate audio in pass 2", kind: "bool", default: true, dependsOn: "anchor_shift",
samplers.py
+29 -11
View File
@@ -2496,7 +2496,7 @@ class FunPackLTXAVSceneChainSampler:
}),
"anchor_shift_restart_sigma": ("FLOAT", {
"default": 0.0, "min": 0.0, "max": 0.999, "step": 0.001,
"tooltip": "Where pass 2 re-enters the schedule. 0 (default) means 'continue from where pass 1 stopped' — the cheapest option, total cost stays one full schedule. Setting it HIGHER than anchor_shift_sigma rewinds: pass 2 re-noises the shifted latent back up to that sigma and re-denoises from there, giving the model more freedom to rebuild the regrown tail and blend the seam, at the cost of repeating those steps (and of partly overwriting what pass 1 established). Try this if the joint is visible or the regrown tail looks thin at the default.",
"tooltip": "Picks between the two modes. 0 (default) = CONTINUE: pass 1 stops at the shift sigma and pass 2 resumes from exactly that state, no re-noising. Total cost stays one full schedule. Any value above 0 = REWIND: pass 1 instead runs the WHOLE schedule to a finished clip, the shift is applied to that, and pass 2 re-noises it up to this sigma and re-denoises down — the standard img2img recipe, which is only valid on a finished latent, which is why pass 1 has to complete first. Rewind gives the model far more freedom to rebuild the regrown tail and blend the joint, and costs a full schedule PLUS the tail. Try rewind (around 0.9) if continue leaves a visible seam or a thin ending.",
}),
"anchor_shift_tail": (["extend_last", "wrap", "empty"], {
"default": "extend_last",
@@ -2951,14 +2951,21 @@ class FunPackLTXAVSceneChainSampler:
def _sample_chunk(self, model, sampler, sigmas, seed, cfg, positive, negative, latent,
pbar=None, step_offset=0, alg_guide_tail_frames=0,
alg_guide_blur_strength=2.0, alg_guide_blur_sigma_threshold=0.975,
bounded_attention_enabled=False):
bounded_attention_enabled=False, continue_from_state=False):
if sampler is None:
raise ValueError("sampler input is required.")
if not isinstance(sigmas, torch.Tensor):
raise ValueError("sigmas input must be a SIGMAS tensor.")
latent = self._clone_latent(latent)
samples = latent["samples"]
noise = comfy.sample.prepare_noise(samples, int(seed))
# continue_from_state: `samples` is ALREADY the noisy x at sigmas[0] (a run resumed
# mid-trajectory), not a clean image. comfy's CONST scaling is
# x = s*noise + (1-s)*latent_image, which treats latent_image as CLEAN and re-noises
# it — on a mid-trajectory latent that scales the picture down by (1-s) and buries it
# under a fresh s of noise. Passing the state as BOTH terms makes the expression
# collapse to an identity (s*x + (1-s)*x == x), so sampling resumes from exactly the
# state we hand it. Only correct when the caller really is mid-trajectory.
noise = samples if continue_from_state else comfy.sample.prepare_noise(samples, int(seed))
def _progress_cb(step, _denoised, _x, _total_steps):
if pbar is not None:
@@ -3062,16 +3069,21 @@ class FunPackLTXAVSceneChainSampler:
return None # threshold above the first sigma, or not reached at all
restart = float(restart_sigma)
if restart <= 0:
# CONTINUE: pass 1 stops at the cut and pass 2 resumes from that exact state.
# Pass 2 must NOT re-noise (see _sample_chunk's continue_from_state).
first, second, resume = sigmas[:cut + 1], sigmas[cut:], True
start = cut
else:
# REWIND: pass 2 re-noises to a HIGHER sigma, which is only meaningful on a CLEAN
# latent — so pass 1 runs the whole schedule first and hands over a finished clip.
# Same shape as segmented detailing's stage-2 tail. Costs a full schedule + tail.
start = _first_at_or_below(restart)
if start is None:
return None
first = sigmas[:cut + 1]
second = sigmas[start:]
first, second, resume = sigmas, sigmas[start:], False
if first.numel() < 2 or second.numel() < 2:
return None
return first, second, vals[cut], vals[start]
return first, second, vals[cut], vals[start], resume
def _anchor_shift_latent(self, latent, drop, tail_mode, fresh_audio):
"""Slide the video stream left by `drop` latent frames and refill the freed tail.
@@ -5472,20 +5484,26 @@ class FunPackLTXAVSceneChainSampler:
if _shift_split is None:
run_mechanisms.append(f"anchor_shift(SKIPPED: {_reason})")
if _shift_split is not None:
_sig_a, _sig_b, _cut_at, _restart_at = _shift_split
_sig_a, _sig_b, _cut_at, _restart_at, _resume = _shift_split
_pass1 = self._sample_chunk(
model, sampler, _sig_a, scene_seed, cfg, scene_positive, scene_negative,
chunk, **_sample_kwargs)
_shifted, _dropped = self._anchor_shift_latent(
_pass1, _shift_drop, anchor_shift_tail, anchor_shift_fresh_audio)
# Fresh noise for pass 2: reusing scene_seed's field would re-apply the
# exact noise pass 1 just resolved, against content that has moved.
# Continue mode resumes from the handed-over state (no re-noise, so the
# seed is irrelevant); rewind mode re-noises a finished clip, where fresh
# noise is wanted — reusing scene_seed there would re-apply the exact field
# pass 1 just resolved, against content that has since moved.
_kw2 = dict(_sample_kwargs)
_kw2["step_offset"] = _sample_kwargs["step_offset"] + (_sig_a.numel() - 1)
_kw2["continue_from_state"] = _resume
sampled = self._sample_chunk(
model, sampler, _sig_b, scene_seed + 4242, cfg, scene_positive,
scene_negative, _shifted, **_sample_kwargs)
scene_negative, _shifted, **_kw2)
_pinned_n = self._anchor_pinned_frames(chunk)
run_mechanisms.append(
f"anchor_shift(cut@{_cut_at:g} restart@{_restart_at:g}, "
f"anchor_shift({'continue' if _resume else 'rewind'} "
f"cut@{_cut_at:g} restart@{_restart_at:g}, "
f"dropped {_dropped} of {_pinned_n} pinned latent frames, "
f"tail={anchor_shift_tail}"
f"{', fresh audio' if anchor_shift_fresh_audio else ''})")
tests/test_anchor_shift.py
+84 -4
View File
@@ -14,6 +14,7 @@ import sys
import types
from pathlib import Path
import pytest
import torch
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
@@ -47,19 +48,30 @@ def _latent(frames=10, audio=True):
# ── schedule splitting ──────────────────────────────────────────────────────
def test_split_cuts_at_the_requested_sigma_and_continues_by_default():
first, second, cut_at, restart_at = _node()._anchor_shift_split_sigmas(SCHEDULE, 0.909, 0.0)
def test_continue_mode_stops_at_the_cut_and_resumes_from_that_state():
first, second, cut_at, restart_at, resume = _node()._anchor_shift_split_sigmas(
SCHEDULE, 0.909, 0.0)
# float32: the schedule stores 0.909 as 0.90899997, so compare loosely.
assert abs(cut_at - 0.909) < 1e-5 and abs(restart_at - 0.909) < 1e-5
# pass 1 ends ON the cut sigma; pass 2 resumes from it, so no step is run twice.
assert [round(v, 3) for v in first.tolist()] == [1.0, 0.975, 0.909]
assert [round(v, 3) for v in second.tolist()] == [0.909, 0.725, 0.422, 0.0]
# resume=True is load-bearing: pass 2 receives a MID-TRAJECTORY latent, and comfy's
# CONST scaling (x = s*noise + (1-s)*latent) would otherwise treat it as a clean image
# and re-noise it — scaling the picture to (1-s) under a fresh s of noise. That is the
# bug that made the first live run come out under-denoised.
assert resume is True
def test_restart_sigma_above_the_cut_rewinds_pass_two():
first, second, cut_at, restart_at = _node()._anchor_shift_split_sigmas(SCHEDULE, 0.725, 0.975)
def test_rewind_mode_runs_pass_one_to_completion_first():
"""A re-noise to a HIGHER sigma is only meaningful on a clean latent, so rewind mode
must hand pass 2 a finished clip — not a mid-trajectory one."""
first, second, cut_at, restart_at, resume = _node()._anchor_shift_split_sigmas(
SCHEDULE, 0.725, 0.975)
assert abs(cut_at - 0.725) < 1e-5 and abs(restart_at - 0.975) < 1e-5
assert [round(v, 3) for v in first.tolist()] == [1.0, 0.975, 0.909, 0.725, 0.422, 0.0]
assert [round(v, 3) for v in second.tolist()] == [0.975, 0.909, 0.725, 0.422, 0.0]
assert resume is False # clean input -> re-noising is correct here
def test_split_refuses_when_a_pass_would_have_no_steps():
@@ -154,3 +166,71 @@ def test_pinned_frames_is_zero_for_a_real_t2v_scene():
assert n._anchor_pinned_frames({"samples": video, "noise_mask": torch.ones(1, 4, 6, 2, 2)}) == 0
assert n._anchor_pinned_frames({"samples": video}) == 0 # no mask at all
assert n._anchor_pinned_frames({"samples": None}) == 0 # unreadable -> refuse, don't raise
# ── the re-noise bug this feature shipped with, locked down ─────────────────
@pytest.fixture
def capture_sample_custom():
"""Stub comfy.sample so we can see exactly what noise/latent the sampler receives.
Snapshot/restore, because these comfy stubs are SHARED module objects across every test
module — leaving mutations behind changes other files' results depending on collection
order (the same trap test_detailing.py hit with comfy.nested_tensor).
"""
mod = sys.modules["comfy.sample"]
comfy_mod = sys.modules["comfy"]
saved = (getattr(mod, "prepare_noise", None), getattr(mod, "sample_custom", None),
getattr(comfy_mod, "sample", None))
seen = {}
def prepare_noise(samples, seed, *a, **k):
return torch.full_like(samples, 7.0) # unmistakably "fresh noise"
def sample_custom(model, noise, cfg, sampler, sigmas, pos, neg, latent_image, **kw):
seen["noise"] = noise
seen["latent_image"] = latent_image
return latent_image
mod.prepare_noise = prepare_noise
mod.sample_custom = sample_custom
sys.modules["comfy"].sample = mod # attribute access, not just sys.modules
yield seen
for name, val in (("prepare_noise", saved[0]), ("sample_custom", saved[1])):
if val is None:
delattr(mod, name)
else:
setattr(mod, name, val)
if saved[2] is None:
delattr(comfy_mod, "sample")
else:
comfy_mod.sample = saved[2]
def _const_noise_scaling(sigma, noise, latent_image):
"""comfy's CONST rule, verbatim: x = s*noise + (1-s)*latent_image."""
return sigma * noise + (1.0 - sigma) * latent_image
def test_continue_from_state_makes_const_scaling_an_identity(capture_sample_custom):
"""The real fix. Pass 2 gets a MID-TRAJECTORY latent; handing it as both terms makes
comfy's scaling collapse to x, so sampling resumes exactly where pass 1 stopped."""
seen = capture_sample_custom
state = torch.randn(1, 4, 5, 2, 2)
_node()._sample_chunk(object(), object(), torch.tensor([0.725, 0.422, 0.0]), 1, 1.0,
[], [], {"samples": state}, continue_from_state=True)
assert torch.equal(seen["noise"], seen["latent_image"])
resumed = _const_noise_scaling(0.725, seen["noise"], seen["latent_image"])
assert torch.allclose(resumed, state, atol=1e-6)
def test_without_continue_from_state_the_latent_is_renoised(capture_sample_custom):
"""The other mode, and the shape of the original bug: fresh noise is mixed in and the
picture is scaled to (1-sigma). Correct for a CLEAN input, wrong mid-trajectory."""
seen = capture_sample_custom
clean = torch.randn(1, 4, 5, 2, 2)
_node()._sample_chunk(object(), object(), torch.tensor([0.725, 0.422, 0.0]), 1, 1.0,
[], [], {"samples": clean}, continue_from_state=False)
assert not torch.equal(seen["noise"], seen["latent_image"])
entered = _const_noise_scaling(0.725, seen["noise"], seen["latent_image"])
assert not torch.allclose(entered, clean, atol=1e-3)