anchor_shift: add tail=crop — no regrown tail, the clip just ends earlier
User's idea, as an OPTION (default stays extend_last). The three existing tail modes all refill the freed end so the clip keeps its length, which means pass 2 has to invent an ending and blend a joint — the source of the "thin ending" and of most of the reason to reach for a rewind restart. 'crop' regrows nothing: the scene ends anchor_shift_frames earlier and every surviving frame was generated as part of one continuous shot. Checked the model side before assuming this was free, since a bad latent size would mismatch rather than degrade: - LTXAV's video patchifier is SymmetricPatchifier(1) -> temporal patch size 1, so there is no divisibility rule on video latent frames. - The streams reach the transformer as independent tensors (separate_audio_and_video_ latents just unbinds the list); no video:audio ratio is enforced there. - comfy re-packs per sample_custom call and re-derives latent_shapes, and the denoise masks off them (comfy/samplers.py outer_sample), so nothing is baked in from scene-build time and a shorter latent needs no conditioning rebuild. - VAE decode is (f-1)*time_scale+1 for any f >= 1. So no size is rejected. What IS real is proportion: audio timings come from the audio stream's own index and video's from its own RoPE, so cropping the two by different amounts of TIME desyncs the clip silently — no error, just drifting sound. Each extra stream's KEPT length is therefore derived from the new video length directly rather than by subtracting a separately-rounded drop, so rounding cannot leave them describing clips of different durations. fresh_audio zeroes after the crop, at the new length. Downstream needed nothing: _prepend_soft_continuation already clamps to the previous scene's real length and _derived_overlap clamps to the stream, so a shorter scene flows through the chain as-is. The scene report now states the cropped length in latent frames and in real frames, because a scene coming out shorter than the length that was set must never be something the user has to infer from the output. Tests: crop shortens instead of refilling and the surviving frames are the untouched originals; audio/video stay in proportion (including four rates that do not divide evenly); fresh_audio resets at the cropped length; the other three modes still keep the clip length. Suite at baseline: 22 pre-existing failures, 288 passing (+5). Verified live in the running app: the Engine ▸ Experimental dropdown offers extend_last/wrap/empty/crop, default unchanged, hint present. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
4c2a6b6b54
commit
9f7e8bb3c6
@@ -234,8 +234,8 @@
|
||||
hint: "How much of the front to drop at the shift, in real frames. 8 (one latent frame) removes just the anchor itself, which is usually not enough — the anchor is followed by a settling-in stretch where the shot is still leaving the reference still and little is happening, and on a prompt asking for immediate action that dead time is exactly what you want gone. 48 was the value that worked on a 768×768×305@30 chain with a quick-cut prompt; treat that as a starting point for this pipeline rather than a universal default. Raise it if the opening still looks like it departed from a photo. The clip doesn't get shorter — whatever is dropped here is regrown at the end by pass 2." },
|
||||
{ name: "anchor_shift_restart_sigma", label: "Pass 2 restart sigma", kind: "float", default: 0.0, min: 0.0, max: 0.999, step: 0.001, dependsOn: "anchor_shift",
|
||||
hint: "Where pass 2 re-enters the schedule after the shift. Pass 1 always stops at the shift sigma either way — this only decides where the second half picks up. 0 (default) = CONTINUE: resume from exactly the state pass 1 handed over, nothing added, cost stays one full schedule. Above the shift sigma = REWIND: the shifted latent is re-noised from the cut back up to this sigma, then denoised down again — much more freedom to rebuild the regrown tail and blend the joint, at the cost of repeating those steps and partly overwriting what pass 1 established. The re-noise is exact for a half-denoised latent (it rebuilds the state at the higher sigma from what pass 1 produced plus a matched top-up of fresh noise), not the naive img2img re-noise that caused the under-denoising in the first live run. BELOW the shift sigma is refused: noise can be added back, never removed. Try a step or two above the cut if continue leaves a visible seam or a thin ending." },
|
||||
{ name: "anchor_shift_tail", label: "Regrown tail starts from", kind: "combo", choices: ["extend_last", "wrap", "empty"], default: "extend_last", dependsOn: "anchor_shift",
|
||||
hint: "What fills the freed end after the slide. 'extend_last' (default) repeats the final frame so pass 2 grows the new ending out of something continuous with it. 'wrap' puts the dropped head back at the end — continuous, but it re-shows the reference image at the END of the clip. 'empty' zeroes it and lets pass 2 generate freely — most freedom, most likely to drift from the rest of the shot." },
|
||||
{ name: "anchor_shift_tail", label: "Regrown tail starts from", kind: "combo", choices: ["extend_last", "wrap", "empty", "crop"], default: "extend_last", dependsOn: "anchor_shift",
|
||||
hint: "What fills the freed end after the slide. 'extend_last' (default) repeats the final frame so pass 2 grows the new ending out of something continuous with it. 'wrap' puts the dropped head back at the end — continuous, but it re-shows the reference image at the END of the clip. 'empty' zeroes it and lets pass 2 generate freely — most freedom, most likely to drift from the rest of the shot. 'crop' regrows NOTHING: the scene simply ends 'Frames to drop' earlier and comes out SHORTER than the length you set. It is the only mode with no invented ending and no joint to blend — every surviving frame was generated as part of one continuous shot — so it removes the thin-ending failure the other three can produce, and makes a rewind restart much less necessary. Audio is cropped to match. Use it when the shot matters more than the exact duration; avoid it when a scene has to hit a specific length. The scene report states the new length." },
|
||||
{ name: "anchor_shift_fresh_audio", label: "Regenerate audio in pass 2", kind: "bool", default: true, dependsOn: "anchor_shift",
|
||||
hint: "Reset the audio to an empty latent before pass 2 so it's written fresh against the shifted picture. On by default because pass 1's audio was formed against frames that no longer sit at those positions. Turn off only to A/B whether the carried audio actually sounds better." },
|
||||
{ name: "context_windows", label: "Context windows (long scenes)", kind: "bool", default: false,
|
||||
|
||||
+59
-13
@@ -2536,9 +2536,9 @@ class FunPackLTXAVSceneChainSampler:
|
||||
"default": 0.0, "min": 0.0, "max": 0.999, "step": 0.001,
|
||||
"tooltip": "Where pass 2 re-enters the schedule after the shift. Pass 1 always stops at anchor_shift_sigma either way — this only decides where the second half picks up. 0 (default) = CONTINUE: resume from exactly the state pass 1 handed over, nothing added, total cost stays one full schedule. A value ABOVE anchor_shift_sigma = REWIND: the shifted latent is re-noised from the cut back up to this sigma, then denoised down again, giving the model far more freedom to rebuild the regrown tail and blend the joint — at the cost of repeating those steps (and of partly overwriting what pass 1 established). The re-noise is exact for a mid-trajectory latent (it rebuilds x at the higher sigma from the latent it already has plus a matched top-up of fresh noise), NOT the naive img2img re-noise, which assumes a finished image and is what caused the under-denoising in the first live run. A value BELOW anchor_shift_sigma is refused: noise can be added back, never removed. Try rewind (a step or two above the cut) if continue leaves a visible seam or a thin ending.",
|
||||
}),
|
||||
"anchor_shift_tail": (["extend_last", "wrap", "empty"], {
|
||||
"anchor_shift_tail": (["extend_last", "wrap", "empty", "crop"], {
|
||||
"default": "extend_last",
|
||||
"tooltip": "What the freed tail starts from after the slide. 'extend_last' (default) repeats the final frame, so pass 2 grows the new ending out of content that is already continuous with it. 'wrap' puts the dropped head back at the end (Mobius) — cheap and continuous, but it re-shows the reference image at the END of the clip, which is usually not what you want. 'empty' zeros the region and lets pass 2's noise scaling generate it from scratch — most freedom, most likely to drift from the rest of the shot.",
|
||||
"tooltip": "What the freed tail starts from after the slide. 'extend_last' (default) repeats the final frame, so pass 2 grows the new ending out of content that is already continuous with it. 'wrap' puts the dropped head back at the end (Mobius) — cheap and continuous, but it re-shows the reference image at the END of the clip, which is usually not what you want. 'empty' zeros the region and lets pass 2's noise scaling generate it from scratch — most freedom, most likely to drift from the rest of the shot. 'crop' regrows NOTHING: the clip simply ends anchor_shift_frames earlier, so the scene comes out shorter than the length you asked for. That is the only mode with no invented ending and no joint to blend — every frame that survives was generated as part of one continuous shot — so it removes the 'thin ending' failure the other three can produce, and it makes the extra freedom of a rewind restart much less necessary. Audio is cropped by the matching amount. Pick it when the shot mattering more than the exact duration is fine (the montage just gets a shorter clip); avoid it when a scene has to hit an exact length.",
|
||||
}),
|
||||
"anchor_shift_fresh_audio": ("BOOLEAN", {
|
||||
"default": True,
|
||||
@@ -3188,9 +3188,12 @@ class FunPackLTXAVSceneChainSampler:
|
||||
def _anchor_shift_latent(self, latent, drop, tail_mode, fresh_audio):
|
||||
"""Slide the video stream left by `drop` latent frames and refill the freed tail.
|
||||
|
||||
The head frames (anchor + settling-in region) are dropped, not cropped: the clip
|
||||
keeps its full length, so no extra frames are ever generated to pay for this. The
|
||||
freed tail is what pass 2 will grow into; `tail_mode` picks what it starts from.
|
||||
The head frames (anchor + settling-in region) are dropped. In the three refilling
|
||||
modes the clip keeps its full length, so no extra frames are ever generated to pay
|
||||
for this, and `tail_mode` picks what the freed tail starts from. In "crop" there is
|
||||
no tail at all: the clip is simply that much shorter, which is the only mode where
|
||||
pass 2 never has to invent an ending or blend a joint.
|
||||
|
||||
Audio is optionally reset to an empty latent so pass 2 writes it fresh against the
|
||||
shifted picture instead of inheriting pass 1's audio, which was scored against
|
||||
frames that no longer exist at those positions.
|
||||
@@ -3203,13 +3206,42 @@ class FunPackLTXAVSceneChainSampler:
|
||||
if drop <= 0:
|
||||
return result, 0
|
||||
head, body = video[:, :, :drop], video[:, :, drop:]
|
||||
if tail_mode == "wrap":
|
||||
tail = head # Mobius: the dropped head reappears at the end
|
||||
elif tail_mode == "extend_last":
|
||||
tail = body[:, :, -1:].repeat(1, 1, drop, 1, 1)
|
||||
else: # "empty": zeros, so pass 2's noise scaling starts that region from noise
|
||||
tail = torch.zeros_like(head)
|
||||
tensors[0] = torch.cat([body, tail], dim=2)
|
||||
if tail_mode == "crop":
|
||||
# No regrown tail — the clip just gets shorter.
|
||||
#
|
||||
# Nothing downstream constrains the new size: LTXAV's video patchifier has
|
||||
# temporal patch size 1 (SymmetricPatchifier(1) -> no divisibility rule on
|
||||
# latent frames), the streams reach the transformer as independent tensors with
|
||||
# no enforced video:audio ratio, and comfy re-packs and re-derives latent_shapes
|
||||
# (and the denoise masks off them) from whatever latent it is handed on the next
|
||||
# sample_custom call — so a shorter latent needs nothing rebuilt.
|
||||
#
|
||||
# What DOES matter is proportion. Audio timings come from the audio stream's own
|
||||
# index and video's from its own RoPE, so cropping the two by different amounts
|
||||
# of TIME desynchronises them silently — no error, just sound that drifts. Each
|
||||
# extra stream's kept length is therefore derived from the new VIDEO length
|
||||
# directly, not by subtracting a separately-rounded drop, so rounding cannot
|
||||
# leave the streams describing clips of different durations.
|
||||
tensors[0] = body
|
||||
kept = frames - drop
|
||||
for _idx in range(1, len(tensors)):
|
||||
_t = tensors[_idx]
|
||||
if _t is None:
|
||||
continue
|
||||
_n = self._tensor_frames(_t)
|
||||
_keep = max(1, min(_n, int(round(_n * kept / max(1, frames)))))
|
||||
if _keep < _n:
|
||||
tensors[_idx] = _t[:, :, _n - _keep:]
|
||||
else:
|
||||
if tail_mode == "wrap":
|
||||
tail = head # Mobius: the dropped head reappears at the end
|
||||
elif tail_mode == "extend_last":
|
||||
tail = body[:, :, -1:].repeat(1, 1, drop, 1, 1)
|
||||
else: # "empty": zeros, so pass 2's noise scaling starts that region from noise
|
||||
tail = torch.zeros_like(head)
|
||||
tensors[0] = torch.cat([body, tail], dim=2)
|
||||
# After any cropping: the reset writes zeros at whatever length the stream now has,
|
||||
# which is what defines the clip's audio duration.
|
||||
if fresh_audio and len(tensors) > 1 and tensors[1] is not None:
|
||||
tensors[1] = torch.zeros_like(tensors[1])
|
||||
if self._is_nested(result.get("samples")):
|
||||
@@ -5623,11 +5655,25 @@ class FunPackLTXAVSceneChainSampler:
|
||||
model, sampler, _sig_b, scene_seed + 4242, cfg, scene_positive,
|
||||
scene_negative, _shifted, **_kw2)
|
||||
_pinned_n = self._anchor_pinned_frames(chunk)
|
||||
# tail=crop makes the scene SHORTER than the length that was asked for.
|
||||
# That is the mode's whole point, but it must never be something the user
|
||||
# has to infer from the output — say it in the report, in latent frames
|
||||
# and in the real frames they set.
|
||||
_tail_note = ""
|
||||
if anchor_shift_tail == "crop":
|
||||
try:
|
||||
_before = self._tensor_frames(self._latent_tensors(_pass1)[0])
|
||||
_after = self._tensor_frames(self._latent_tensors(sampled)[0])
|
||||
_tail_note = (f", CROPPED to {_after} of {_before} latent frames "
|
||||
f"(~{int(anchor_shift_frames)} real frames shorter — "
|
||||
f"no tail regrown)")
|
||||
except Exception:
|
||||
_tail_note = ", CROPPED (no tail regrown)"
|
||||
run_mechanisms.append(
|
||||
f"anchor_shift({'continue' if _resume else 'rewind'} "
|
||||
f"cut@{_cut_at:g} restart@{_restart_at:g}, "
|
||||
f"dropped {_dropped} of {_pinned_n} pinned latent frames, "
|
||||
f"tail={anchor_shift_tail}"
|
||||
f"tail={anchor_shift_tail}{_tail_note}"
|
||||
f"{', fresh audio' if anchor_shift_fresh_audio else ''})")
|
||||
if _dropped < _pinned_n:
|
||||
# Part of the anchor survives the slide and stays visible in the clip —
|
||||
|
||||
@@ -323,6 +323,63 @@ def test_source_latent_is_never_mutated():
|
||||
assert torch.equal(latent["samples"], before)
|
||||
|
||||
|
||||
# ── tail=crop: no regrown tail, the clip just gets shorter ──────────────────
|
||||
|
||||
def _av_latent(video_frames=10, audio_frames=20):
|
||||
"""A packed-style AV latent: video [B,C,F,H,W] + audio [B,C,T] at its own rate."""
|
||||
video = torch.arange(video_frames, dtype=torch.float32).view(
|
||||
1, 1, video_frames, 1, 1).repeat(1, 4, 1, 2, 2)
|
||||
audio = torch.arange(audio_frames, dtype=torch.float32).view(1, 1, audio_frames).repeat(1, 4, 1)
|
||||
return {"samples": _FakeNested([video, audio]), "noise_mask": torch.zeros_like(video)}
|
||||
|
||||
|
||||
def test_crop_shortens_the_clip_instead_of_regrowing_a_tail(renoise_stubs):
|
||||
shifted, dropped = _node()._anchor_shift_latent(_latent(frames=10), 3, "crop", False)
|
||||
assert dropped == 3
|
||||
# 10 - 3, not 10: nothing is invented at the end, which is the whole point of the mode.
|
||||
assert shifted["samples"].shape[2] == 7
|
||||
# And what survives is the ORIGINAL frames 3..9, untouched — no blend, no joint.
|
||||
assert [int(v) for v in shifted["samples"][0, 0, :, 0, 0]] == [3, 4, 5, 6, 7, 8, 9]
|
||||
|
||||
|
||||
def test_crop_keeps_audio_and_video_describing_the_same_duration(renoise_stubs):
|
||||
"""Audio timing comes from the audio stream's own index and video's from its own RoPE,
|
||||
so cropping them by different amounts of TIME desyncs the clip silently — no error,
|
||||
just drifting sound. The kept lengths must stay in proportion."""
|
||||
shifted, _ = _node()._anchor_shift_latent(_av_latent(10, 20), 3, "crop", False)
|
||||
video, audio = shifted["samples"].unbind()
|
||||
assert video.shape[2] == 7 and audio.shape[2] == 14 # 7/10 == 14/20
|
||||
# Cropped from the HEAD, matching the video slide — the tail is what survives.
|
||||
assert [int(v) for v in audio[0, 0]] == list(range(6, 20))
|
||||
|
||||
|
||||
def test_crop_keeps_proportion_when_the_rate_does_not_divide_evenly(renoise_stubs):
|
||||
"""The rounding trap: deriving audio's KEPT length from the new video length keeps the
|
||||
two in proportion; subtracting a separately-rounded drop can leave them a frame apart."""
|
||||
for v_frames, a_frames, drop in ((10, 15, 3), (9, 14, 4), (13, 7, 5), (10, 21, 7)):
|
||||
shifted, _ = _node()._anchor_shift_latent(_av_latent(v_frames, a_frames), drop, "crop", False)
|
||||
video, audio = shifted["samples"].unbind()
|
||||
assert audio.shape[2] == max(1, round(a_frames * video.shape[2] / v_frames))
|
||||
assert audio.shape[2] >= 1 and video.shape[2] >= 1
|
||||
|
||||
|
||||
def test_crop_still_resets_audio_when_asked_at_the_cropped_length(renoise_stubs):
|
||||
"""fresh_audio zeroes the stream AFTER the crop, so the reset length is the new
|
||||
duration — zeroing first would leave the clip's audio length describing the old one."""
|
||||
shifted, _ = _node()._anchor_shift_latent(_av_latent(10, 20), 4, "crop", True)
|
||||
video, audio = shifted["samples"].unbind()
|
||||
assert video.shape[2] == 6 and audio.shape[2] == 12
|
||||
assert torch.count_nonzero(audio) == 0
|
||||
|
||||
|
||||
def test_the_other_tail_modes_still_keep_the_clip_length(renoise_stubs):
|
||||
"""crop is an OPTION — the three refilling modes must be untouched by it."""
|
||||
for mode in ("extend_last", "wrap", "empty"):
|
||||
shifted, _ = _node()._anchor_shift_latent(_av_latent(10, 20), 3, mode, False)
|
||||
video, audio = shifted["samples"].unbind()
|
||||
assert video.shape[2] == 10 and audio.shape[2] == 20, mode
|
||||
|
||||
|
||||
# ── the t2v guard ───────────────────────────────────────────────────────────
|
||||
|
||||
def test_pinned_frames_counts_the_anchor_prefix():
|
||||
|
||||
Reference in New Issue
Block a user