Merge dev into main - Release 3.3.0

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
digital-garbage
2026-07-15 21:29:44 +03:00
co-authored by Claude Fable 5
17 changed files with 2097 additions and 392 deletions
+68 -1
View File
@@ -1,6 +1,73 @@
# Changelog
## [Unreleased]
## [3.3.0] - 2026-07-15
### Added
- **Loop temporal style** - the "loop" option in Temporal style (inert since it was added)
now produces seamlessly looping clips, the LTX2.3 equivalent of WanVideoWrapper's Loop
Args: on eligible denoise steps the latent is cyclically rolled along time before the
model forward and the prediction unrolled after, so the model repeatedly smooths the
video's seam (last frame | first frame) as if it were an ordinary interior cut. Video and
audio roll by their own frame counts and stay time-aligned; i2v anchor masks roll in step.
Near-zero overhead (no extra forward passes). Also honored by the "auto" director when a
scene's prompt asks for a loop ("seamless loop", "cycling", "repeating"). On scenes that
carry guides (mid-scene guide, carry-i2v-guides, JoyAI memory, custom guide stacks) the
roll stays off - guide frames are pinned to absolute positions - and the run log shows
`temporal_loop_roll(inert: guides pin frames)`.
- **Plateau step-cache** (experimental, Engine ▸ Experimental) - reuses the base model's
forward across the near-noise plateau steps of the LTX 8-step schedule (sigma >= threshold,
default 0.975), skipping 3-4 of 8 forwards per scene for a negative-overhead speedup.
Guidance wrappers still post-process every step's prediction.
- **Wired widget locks** - node widget fields fed by an incoming wire are now locked in the
Models & Pipeline editor and show which node feeds them, instead of silently accepting
edits that the wire would overwrite on the next run.
- **Per-prompt taste direction** (`taste_nearest_prompt`, experimental, Engine ▸ Guidance) —
a retrieval layer over the learned taste steering. Instead of pushing Embed guidance /
Score slider along the single global liked-direction average, each liked rating now also
records a `(prompt fingerprint → that run's liked direction)` entry, and with this on each
scene retrieves the similarity-weighted direction of its nearest rated prompts (a forest
prompt pulls what worked on forests, not the mean across every prompt). Non-parametric
(a plain ring buffer keyed on pooled conditioning) — no extra model forward, just a cosine
lookup + vector mean, and no small value function to collapse into a spurious attractor.
Falls back to the global liked direction when nothing rated is close enough. Rides the same
refine_v2 clip-state file as `liked_dir`, so it's removed atomically with the key.
### Changed
- **Best-FaceID identity transfer** (experimental, Engine ▸ Experimental) is now a full native
port of ComfyUI-BFSNodes' LTX Identity Transfer overlap+source_phase+ArcFace conditioning,
replacing 3.2.1's source-phase-only approximation. The reference face is now injected as
separate, non-rendered overlap tokens (never blended into a real frame) instead of tagging
Continuity's Identity pin keyframe, plus an optional ArcFace projector channel (`identity_projector`,
`id_strength`, `arcface_mode`) for a second identity signal. Still keyed off Continuity's
existing Identity pin image — no new picker. Requires the `insightface` package (added to
requirements.txt; downloads the buffalo_l model on first use) only when an ArcFace projector
is selected — the overlap tokens alone work without it.
- Frame overlap now carries through anchor changes (hard scene cuts with a new anchor image
used to drop the latent overlap; `carry_overlap_through_anchor`), and an Identity pin
configured on an anchor-swap scene resolves again instead of being skipped.
### Fixed
- **Preview stability (rounds 3-5)** - the remaining causes of stale / black / undecodable
scene previews, especially "the second half stops loading":
- `/result` faststart-remuxes raw ComfyUI renders (VHS writes the MP4 index at the end of
the file; browsers cannot seek that) and never serves a video mid-write - a file whose
index isn't on disk yet answers 503 until the save finishes, instead of undecodable bytes.
- A failed remux answers 502 with rate-limited re-attempts instead of permanently falling
back to the raw file; temp-dir failures and the loopback fallback path can no longer
serve raw index-at-end video either (the fallback also gains Range/seek support).
- The player materializes video elements only in a window around the playhead instead of
one per timeline clip - a montage-scale timeline (100+ clips) used to create 100+
elements and fire ~180 preview-segment encodes at once, starving the browser's
per-origin connections (and hitting Chrome's hard 75-media-players-per-page cap), which
left the back half of the timeline permanently "loading". Timeline filmstrips load
through a small shared queue for the same reason.
- The monitor shows honest readiness slates ("Video is processing…" / "Video is loading…" /
"Preview unavailable") instead of a black or blinking frame, and a long GIL-bound save
can no longer exhaust the retry budget.
- Exposed controls silently reverting: the autosave snapshot was taken before edited model
values synced back to the project, so an exposed-widget edit could be undone by its own
autosave.
- ALG crash on multi-scene runs (tensor truthiness in the `alg_latents` fallback).
## [3.2.1] - 2026-07-06
+40
View File
@@ -9839,6 +9839,45 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner):
except Exception:
pass
def _v2_store_prompt_keyed_direction(self, global_state, payload, session_mean_payload,
max_entries=32):
"""Append a (prompt fingerprint -> liked direction) entry to a retrieval index.
Stores the SAME unit-delta direction that feeds the global liked_dir slot
(pool(payload) minus the session source mean), paired with this run's pooled
conditioning as the prompt fingerprint, so the sampler can later retrieve the
direction learned on the NEAREST prompts instead of the one global average (see
samplers._resolve_prompt_keyed_direction). The fingerprint is pool(payload) — the
processed conditioning, which is exactly what the Chain Sampler pools from its
scene input at retrieval time, so cosine matching lines up on the same domain.
Non-parametric memory: a plain ring buffer, no model to train, so it can't collapse
into a spurious attractor the way a small value function can. Oldest rolls off;
rides the same clip-state file as liked_dir, so delete_refinement_key removes it
with the key. Silent no-op on any shape/None issue."""
pooled = self._v2_pool_conditioning(payload)
session_mean = self._v2_pool_conditioning(session_mean_payload)
if pooled is None or session_mean is None:
return
try:
if list(session_mean.shape) != list(pooled.shape):
return
delta = pooled - session_mean.to(pooled.device)
magnitude = float(delta.norm().item())
if magnitude < 1e-6:
return
unit = delta / magnitude
index = global_state.get("prompt_dir_index")
if not isinstance(index, list):
index = []
index.append({
"prompt": tensor_to_serializable(pooled.cpu()),
"direction": tensor_to_serializable(unit.cpu()),
"direction_magnitude": magnitude,
})
global_state["prompt_dir_index"] = index[-int(max_entries):]
except Exception:
pass
def _v2_apply_direction(self, mixed, slot, strength, negate=False):
"""Apply stored unit direction to active tokens using NORM_SCALE calibration.
Requires direction_count >= 3. Falls back silently if not ready."""
@@ -9936,6 +9975,7 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner):
if key in {"like", "loved_it", "nailed_it"}:
liked_dir_slot = global_state.setdefault("liked_dir", {})
self._v2_store_direction(liked_dir_slot, payload, session_mean)
self._v2_store_prompt_keyed_direction(global_state, payload, session_mean)
count = int(global_state.get("liked_conditioning_count", 0))
if count <= 0 or not isinstance(global_state.get("liked_conditioning"), dict):
global_state["liked_conditioning"] = payload
+145
View File
@@ -0,0 +1,145 @@
"""Best-FaceID identity transfer — native port of ComfyUI-BFSNodes' LTX Identity Transfer
(alisson-anjos/ComfyUI-BFSNodes, LTXIdentityOverlapConditioning): the reference face is
injected as SEPARATE tokens that share the target's frame-0 RoPE grid (overlap) and are
tagged with a per-source RoPE phase (source_phase), exactly as ltx-trainer's flexible
strategy did at train/validation time. The ref tokens are clean (timestep 0), attend to the
target in self-attention, and are sliced off the output (never rendered) — unlike FunPack's
guide-attention path, they never blend into a real frame. Plus optional ArcFace IdentityProjector
tokens appended to the text context (as trained).
Requires `insightface` (+ buffalo_l, auto-downloaded on first use, CPU) for the ArcFace half;
soft dependency — the overlap tokens alone (identity_projector="None") work without it.
"""
from __future__ import annotations
import logging
import os
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from safetensors.torch import load_file
log = logging.getLogger("FunPackIdentityTransfer")
_USE_GPU = os.environ.get("FUNPACK_ARCFACE_GPU", "0") == "1"
_FACE_APP = None
def _get_face_app():
global _FACE_APP
if _FACE_APP is None:
from insightface.app import FaceAnalysis
providers = (["CUDAExecutionProvider", "CPUExecutionProvider"] if _USE_GPU else ["CPUExecutionProvider"])
app = FaceAnalysis(name="buffalo_l", providers=providers)
app.prepare(ctx_id=0 if _USE_GPU else -1, det_size=(640, 640))
_FACE_APP = app
return _FACE_APP
def arcface_embed(image_bhwc, mode="auto_adjust"):
"""Return the ArcFace embedding [512], or None if disabled / no face found.
mode: 'as_is' (detect on the image only), 'auto_adjust' (retry with border-pad
zoom-out + upscale when detection fails), 'disable' (skip ArcFace entirely)."""
if mode == "disable":
return None
import cv2
app = _get_face_app()
img = np.ascontiguousarray(
(np.clip(image_bhwc[0].detach().cpu().numpy(), 0.0, 1.0) * 255.0).astype(np.uint8)[:, :, ::-1]
)
attempts = [img]
if mode == "auto_adjust":
h, w = img.shape[:2]
pad = int(0.4 * max(h, w))
attempts.append(cv2.copyMakeBorder(img, pad, pad, pad, pad, cv2.BORDER_REPLICATE))
attempts.append(cv2.resize(img, None, fx=2.0, fy=2.0, interpolation=cv2.INTER_CUBIC))
attempts.append(cv2.resize(img, None, fx=0.5, fy=0.5, interpolation=cv2.INTER_AREA))
for a in attempts:
faces = app.get(a)
if faces:
f = max(faces, key=lambda x: (x.bbox[2] - x.bbox[0]) * (x.bbox[3] - x.bbox[1]))
return torch.from_numpy(f.normed_embedding.astype(np.float32))
return None
class IdentityProjector(nn.Module):
"""ArcFace embedding [in_dim] -> num_tokens context tokens [num_tokens, context_dim]."""
def __init__(self, in_dim=512, context_dim=4096, num_tokens=4, proj=None):
super().__init__()
self.in_dim = in_dim
self.context_dim = context_dim
self.num_tokens = num_tokens
self.proj = proj if proj is not None else nn.Sequential(
nn.Linear(in_dim, 1024), nn.GELU(), nn.Linear(1024, num_tokens * context_dim))
self.norm = nn.LayerNorm(context_dim)
def forward(self, e):
return self.norm(self.proj(e).reshape(-1, self.num_tokens, self.context_dim))
def load_identity_projector(path, device):
"""Load an IdentityProjector of ANY depth by rebuilding proj.N Linears from the
state_dict (old = 2 linears proj.0/proj.2; enhanced = 3 linears proj.0/proj.2/proj.4)."""
sd = load_file(path)
context_dim = sd["norm.weight"].shape[0]
idxs = sorted({int(k.split(".")[1]) for k in sd if k.startswith("proj.") and k.endswith(".weight")})
layers = []
for j, i in enumerate(idxs):
w = sd[f"proj.{i}.weight"]
layers.append(nn.Linear(w.shape[1], w.shape[0]))
if j < len(idxs) - 1:
layers.append(nn.GELU())
proj = nn.Sequential(*layers)
in_dim = sd["proj.0.weight"].shape[1]
num_tokens = sd[f"proj.{idxs[-1]}.weight"].shape[0] // context_dim
p = IdentityProjector(in_dim=in_dim, context_dim=context_dim, num_tokens=num_tokens, proj=proj)
p.load_state_dict(sd)
return p.to(device=device, dtype=torch.float32).eval()
def append_context_tokens(conditioning, tokens):
"""Append `tokens` [1 or B, N, D'] onto every entry's text context, padding/truncating
the last dim to match and extending any attention_mask with all-ones for the new tokens."""
out = []
for ce, cd in conditioning:
t = tokens.to(device=ce.device, dtype=ce.dtype)
if t.shape[0] != ce.shape[0]:
t = t.expand(ce.shape[0], -1, -1)
if t.shape[-1] < ce.shape[-1]:
t = F.pad(t, (0, ce.shape[-1] - t.shape[-1]))
elif t.shape[-1] > ce.shape[-1]:
t = t[..., :ce.shape[-1]]
nd = cd.copy()
am = nd.get("attention_mask")
if am is not None:
nd["attention_mask"] = torch.cat(
[am, torch.ones((*am.shape[:-1], t.shape[1]), device=am.device, dtype=am.dtype)], dim=-1)
out.append([torch.cat([ce, t], dim=1), nd])
return out
def rotate_overlap_freqs(pe, ref_len, seg_value, theta=10000.0):
"""Rotate the LAST ref_len tokens' RoPE freqs by phase = seg_value * theta^(-d/L)
(port of ltx_core rope.apply_segment_phase). pe = (cos, sin, [split_flag])."""
if ref_len <= 0 or seg_value == 0.0:
return pe
cos, sin = pe[0], pe[1]
rest = tuple(pe[2:])
L = cos.shape[-1]
d = torch.arange(L, device=cos.device, dtype=torch.float32)
rate = theta ** (-d / float(L))
phase = seg_value * rate
pc = phase.cos().to(cos.dtype)
ps = phase.sin().to(sin.dtype)
idx = [slice(None)] * cos.dim()
idx[-2] = slice(cos.shape[-2] - ref_len, cos.shape[-2])
idx = tuple(idx)
c0, s0 = cos[idx], sin[idx]
cos = cos.clone()
sin = sin.clone()
cos[idx] = c0 * pc - s0 * ps
sin[idx] = s0 * pc + c0 * ps
return (cos, sin, *rest)
+157 -2
View File
@@ -35,7 +35,7 @@ TEMPORAL_STYLES = ["natural", "auto", "accelerate", "decelerate", "loop", "freez
TEMPORAL_STYLE_MULT = {
"accelerate": 1.35,
"decelerate": 0.72,
"loop": 1.0, # loop is a separate RoPE coordinate trick (not yet wired); mult is a no-op
"loop": 1.0, # loop = Mobius latent roll (make_loop_temporal_wrapper), not a frame_rate mult
"freeze": 2.0,
}
@@ -188,6 +188,156 @@ def make_pulse_temporal_wrapper(old_wrapper, segment_count=None, peak_mult=None,
return _pulse_wrapper
# --- Loop temporal style: Mobius-style latent roll (arXiv:2502.20307) ---------------
# Seamless looping without training or extra forward passes: on eligible denoise steps
# the latent is cyclically rolled along its temporal dim before the forward and the
# prediction unrolled after. The model then repeatedly sees the video's seam
# (last frame | first frame) as an INTERIOR frame pair at a step-varying position and
# smooths it like any other cut — after enough steps the sequence is consistent under
# rotation, i.e. it loops. WanVideoWrapper's "Loop Args" is the same trick for WAN.
#
# Roll starts below the near-noise plateau: content is ~pure noise up there (nothing to
# smooth) and the plateau step-cache reuses base forwards, which a per-step roll would
# invalidate. Guides pin absolute frame positions (keyframe_idxs / attention entries
# reference appended frames inside the same T range), so calls carrying them are left
# canonical rather than rolled.
LOOP_ROLL_MAX_SIGMA = 0.95
LOOP_ROLL_MIN_FRAMES = 4 # below this many latent frames there is nothing to loop
def _van_der_corput(n):
"""Base-2 bit-reversal sequence in (0,1): 0.5, 0.25, 0.75, 0.125, 0.625… — every
prefix covers the unit interval near-uniformly, so however many steps end up
eligible, the seam visits well-spread temporal positions."""
v, denom = 0.0, 1.0
n = int(n)
while n:
denom *= 2.0
v += (n & 1) / denom
n >>= 1
return v
def _loop_stream_shapes(args):
"""Per-stream latent shapes for a packed [B,1,N] input, or None when the layout
can't be trusted. Mirrors the samplers' packed-latent handling (video first, then
audio; both streams keep their temporal dim at index 2)."""
c = args.get("c") or {}
shapes = c.get("latent_shapes")
if hasattr(shapes, "cond"):
shapes = shapes.cond
x = args.get("input")
if x is None:
return None
if not shapes:
# Single-stream (plain LTXV) fed unpacked: treat the input itself as one stream.
return [tuple(int(d) for d in x.shape)] if x.ndim == 5 else None
try:
shapes = [tuple(int(d) for d in s) for s in shapes]
import math as _math
if sum(_math.prod(s[1:]) for s in shapes) != int(x.shape[-1]):
return None # packed layout doesn't match — don't risk rolling blind
except (TypeError, ValueError):
return None
return shapes
def _loop_roll_packed(x, shapes, frac, direction):
"""Cyclically roll every stream's temporal dim (dim 2) of a packed [B,1,N] tensor by
round(frac * T_stream) frames (video and audio each by their own frame count, so the
two stay time-aligned). direction=+1 rolls, -1 unrolls. Returns a new tensor."""
import math as _math
if x.ndim == 5: # unpacked single stream
t = x.shape[2]
shift = int(round(frac * t)) % t
return torch.roll(x, shifts=direction * shift, dims=2) if shift else x
out = x.clone()
off = 0
for dims in shapes:
sz = _math.prod(dims[1:])
if len(dims) >= 3 and dims[2] >= 2:
t = dims[2]
shift = int(round(frac * t)) % t
if shift:
stream = x[..., off:off + sz].reshape([x.shape[0]] + list(dims[1:]))
out[..., off:off + sz] = torch.roll(
stream, shifts=direction * shift, dims=2).reshape(x.shape[0], 1, sz)
off += sz
return out
def _loop_roll_mask(mask, frac, direction):
"""Roll a denoise mask ([B,1,T,H,W] video / [B,C,T,F] audio) in step with its stream."""
if not isinstance(mask, torch.Tensor) or mask.ndim < 3 or mask.shape[2] < 2:
return mask
t = mask.shape[2]
shift = int(round(frac * t)) % t
return torch.roll(mask, shifts=direction * shift, dims=2) if shift else mask
def make_loop_temporal_wrapper(old_wrapper):
"""Build the loop-style model_function_wrapper. Installed INNERMOST (closest to
apply_model): prediction-modifying wrappers layered above it (dynashift, output
guidance, …) must see canonical-orientation inputs and outputs — the roll exists
only for the duration of the base forward. The per-step shift follows a van der
Corput sequence, reset whenever sigma jumps back up (a new scene/run)."""
state = {"count": 0, "last_sigma": None, "logged": False}
def _loop_wrapper(apply_fn, args, _old=old_wrapper):
def _call(a):
if _old is not None:
return _old(apply_fn, a)
return apply_fn(a["input"], a["timestep"], **a.get("c", {}))
try:
ts = args.get("timestep")
sigma = float(ts.max().item()) if ts is not None else 1.0
except Exception:
sigma = 1.0
if state["last_sigma"] is not None and sigma > state["last_sigma"] + 1e-4:
state["count"] = 0 # sigma went back up: new scene/run
state["last_sigma"] = sigma
c = args.get("c") or {}
if (
sigma > LOOP_ROLL_MAX_SIGMA
or c.get("keyframe_idxs") is not None
or c.get("guide_attention_entries") is not None
):
return _call(args)
shapes = _loop_stream_shapes(args)
if not shapes:
return _call(args)
# Gate on the VIDEO stream's frame count (the 5-dim shape); audio rides along.
t_video = max((s[2] for s in shapes if len(s) == 5),
default=max((s[2] for s in shapes if len(s) >= 3), default=0))
if t_video < LOOP_ROLL_MIN_FRAMES:
return _call(args)
state["count"] += 1
frac = _van_der_corput(state["count"])
if not state["logged"]:
state["logged"] = True
print(f"[FunPack] loop temporal style: Mobius latent roll active (T={t_video})")
try:
new_c = dict(c)
for key in ("denoise_mask", "audio_denoise_mask"):
if isinstance(new_c.get(key), torch.Tensor):
new_c[key] = _loop_roll_mask(new_c[key], frac, 1)
rolled = dict(args)
rolled["input"] = _loop_roll_packed(args["input"], shapes, frac, 1)
rolled["c"] = new_c
out = _call(rolled)
return _loop_roll_packed(out, shapes, frac, -1)
except Exception as e:
if state.get("roll_error") is None:
state["roll_error"] = True
print(f"[FunPack] loop roll failed, running canonical: {e}")
return _call(args)
return _loop_wrapper
# Confirmed semantic focal points (PAG default=14, STG defaults=14,19)
ANCHOR_BLOCKS = [14, 19]
@@ -1206,11 +1356,16 @@ def build_enhancements(model, rating_profile, temporal_style, refinement_key, re
# "auto" and "pulse" are per-scene and owned by the chain sampler (auto bakes a
# per-scene multiplier; pulse installs a segmented ease-down wrapper per scene).
# The global path here fires for concrete manual styles and pulse on single-scene
# workflows. "loop" stays a no-op until the RoPE coordinate trick is wired.
# workflows. "loop" = Mobius latent roll (not a frame_rate mult); installed here it
# sits innermost relative to the chain sampler's per-scene wrappers, which is
# required — prediction-modifying wrappers must see canonical orientation.
if temporal_style == "pulse":
wrapper = make_pulse_temporal_wrapper(model.model_options.get("model_function_wrapper"))
if wrapper is not None:
model.model_options["model_function_wrapper"] = wrapper
elif temporal_style == "loop":
model.model_options["model_function_wrapper"] = make_loop_temporal_wrapper(
model.model_options.get("model_function_wrapper"))
elif temporal_style not in ("natural", "auto"):
mult = TEMPORAL_STYLE_MULT.get(temporal_style, 1.0)
wrapper = make_temporal_wrapper(model.model_options.get("model_function_wrapper"), mult)
+62 -7
View File
@@ -5,6 +5,7 @@
(function () {
const { el, clear } = window.dom;
const S = window.Store;
const API = window.MovieEditorAPI;
let _mounted = null; // { scroller (pane, set per render), content (shell root) }
let unsub = null;
@@ -158,6 +159,8 @@
{ name: "embed_guidance_strength", label: "Embed strength", kind: "float", default: 0.02, min: 0.005, max: 0.1, step: 0.005, dependsOn: "embed_guidance" },
{ name: "score_slider", label: "Score slider", kind: "bool", default: false },
{ name: "score_slider_strength", label: "Slider strength (eta)", kind: "float", default: 1.0, min: 0, max: 3, step: 0.25, dependsOn: "score_slider" },
{ name: "taste_nearest_prompt", label: "Per-prompt taste direction", kind: "bool", default: false,
hint: "EXPERIMENTAL: source Embed guidance / Score slider from the taste direction learned on the prompts NEAREST this scene's prompt, instead of one global liked-direction average. Each liked rating records (prompt → its liked direction); this retrieves the closest matches per scene (a forest prompt pulls what worked on forests). No extra model pass — a cosine lookup + vector mean. Falls back to the global direction when nothing rated is close. Needs Embed guidance or Score slider on. UNVALIDATED LIVE." },
{ name: "output_guidance", label: "Output guidance", kind: "bool", default: false },
{ name: "output_guidance_strength", label: "Output guidance strength", kind: "float", default: 0.02, min: 0.005, max: 0.1, step: 0.005, dependsOn: "output_guidance" },
{ name: "decode_noise_scale", label: "Decode noise scale", kind: "float", default: 0.0, min: 0, max: 1, step: 0.01 },
@@ -180,9 +183,19 @@
{ name: "dynashift_strength", label: "DynaShift strength", kind: "float", default: 0.3, min: 0.05, max: 1.0, step: 0.05, dependsOn: "dynashift" },
{ name: "dynashift_threshold", label: "DynaShift match threshold", kind: "float", default: 0.6, min: 0.3, max: 0.95, step: 0.05, dependsOn: "dynashift" },
{ name: "identity_transfer_enabled", label: "Best-FaceID compatibility", kind: "bool", default: false,
hint: "Tags Continuity's Identity pin guide (Engine → Continuity) with the source-phase RoPE rotation Best-FaceID-style identity LoRAs were trained on. Load the LoRA itself the normal way — Models → add a LoRA loader onto the model path. No effect without an Identity pin image set." },
{ name: "identity_transfer_phase", label: "Source-phase id", kind: "float", default: 2.0, min: 0.0, max: 8.0, step: 0.5, dependsOn: "identity_transfer_enabled",
hint: "Matches the LoRA's training convention (ltx-trainer used 2)." },
hint: "Full native port of the overlap+source_phase+ArcFace conditioning Best-FaceID-style identity LoRAs were trained on. Replaces Continuity's Identity pin guide (Engine → Continuity) with separate, non-rendered reference tokens plus an optional ArcFace projector below. Load the LoRA itself the normal way — Models → add a LoRA loader onto the model path. No effect without an Identity pin image set." },
{ name: "source_id", label: "Source-phase id", kind: "float", default: 2.0, min: 0.0, max: 8.0, step: 1.0, dependsOn: "identity_transfer_enabled",
hint: "Matches the LoRA's training convention (ltx-trainer used 2). 0 disables the RoPE rotation." },
{ name: "phase_scale", label: "Phase scale", kind: "float", default: 1.0, min: 0.0, max: 4.0, step: 0.1, dependsOn: "identity_transfer_enabled" },
{ name: "id_strength", label: "ArcFace token strength", kind: "float", default: 1.0, min: 0.0, max: 50.0, step: 0.5, dependsOn: "identity_transfer_enabled",
hint: "Only applies when an ArcFace projector is selected below — weak channel, push high (5-20) to test." },
{ name: "arcface_mode", label: "ArcFace detection mode", kind: "combo", choices: ["auto_adjust", "as_is", "disable"], default: "auto_adjust", dependsOn: "identity_transfer_enabled" },
{ name: "debug_log", label: "Debug log", kind: "bool", default: false, dependsOn: "identity_transfer_enabled",
hint: "Print per-scene identity-transfer shape/status logs to the ComfyUI console." },
{ name: "plateau_cache", label: "Plateau step-cache (speed)", kind: "bool", default: false,
hint: "EXPERIMENTAL speed: the near-pure-noise early steps carry almost no signal, so the transformer output barely changes across them. Computes it once at the top of the plateau and reuses it for the rest, skipping ~3-4 of 8 transformer passes. Deterministic given seed (safe in Batch Training) but an approximation — A/B it before trusting on finals. Much of wall-clock time is outside the sampler, so total speedup is smaller than the forward count suggests. UNVALIDATED LIVE." },
{ name: "plateau_cache_threshold", label: "Plateau threshold (sigma)", kind: "float", default: 0.975, min: 0.5, max: 0.999, step: 0.005, dependsOn: "plateau_cache",
hint: "Steps with sigma at or above this count as the reusable plateau. Higher = fewer steps cached (safer); lower = more cached (faster, more approximation). 0.975 catches the documented noise plateau while leaving structure formation fully computed." },
];
const SAMPLER_KNOB_MAP = Object.fromEntries(SAMPLER_KNOBS.map((k) => [k.name, k]));
@@ -285,9 +298,9 @@
const CHAIN_VIEW_KNOBS = {
chain_continuity: ["carry_i2v_guides"],
chain_timing: ["frame_overlap", "transition_duration", "use_same_seed"],
chain_guidance: ["cfg", "embed_guidance", "embed_guidance_source", "embed_guidance_strength", "score_slider", "score_slider_strength", "output_guidance", "output_guidance_strength", "dynashift", "dynashift_strength", "dynashift_threshold"],
chain_guidance: ["cfg", "embed_guidance", "embed_guidance_source", "embed_guidance_strength", "score_slider", "score_slider_strength", "taste_nearest_prompt", "output_guidance", "output_guidance_strength", "dynashift", "dynashift_strength", "dynashift_threshold"],
chain_decode: ["decode_noise_scale", "decode_timestep", "decode_tile_size"],
chain_experimental: ["mid_scene_guide", "mid_scene_guide_strength", "joyai_memory", "joyai_memory_size", "joyai_fix_frames", "joyai_frame_select", "joyai_memory_strength", "joyai_audio_memory", "v2a_grad_scale", "alg_blur_guides", "alg_guide_blur_strength", "alg_guide_blur_sigma_threshold", "bounded_attention_enabled", "identity_transfer_enabled", "identity_transfer_phase"],
chain_experimental: ["plateau_cache", "plateau_cache_threshold", "mid_scene_guide", "mid_scene_guide_strength", "joyai_memory", "joyai_memory_size", "joyai_fix_frames", "joyai_frame_select", "joyai_memory_strength", "joyai_audio_memory", "v2a_grad_scale", "alg_blur_guides", "alg_guide_blur_strength", "alg_guide_blur_sigma_threshold", "bounded_attention_enabled", "identity_transfer_enabled", "source_id", "phase_scale", "id_strength", "arcface_mode", "debug_log"],
};
function countChainView(p, id) {
@@ -563,6 +576,49 @@
renderKnobList(g, st, CHAIN_VIEW_KNOBS[id]);
}
// ── Best-FaceID ArcFace projector: the one identity-transfer field that needs a
// live (server-fetched) choice list, so it can't live in the static SAMPLER_KNOBS combo
// shape. Reuses the same loras folder listing LoraLoaderModelOnly exposes.
let _loraChoices = null;
async function ensureLoraChoices() {
if (_loraChoices) return _loraChoices;
try {
const spec = await API.nodeSpec("LoraLoaderModelOnly");
const w = (spec?.inputs || []).find((i) => i.name === "lora_name");
_loraChoices = (w && w.choices) || [];
} catch (_) {
_loraChoices = [];
}
render();
return _loraChoices;
}
function renderChainExperimental(pane, st) {
renderChainKnobsView(pane, st, "chain_experimental", "Experimental",
"Research techniques — off by default; expect quality/overhead trade-offs.");
const si = st.project.sampler_inputs || {};
if (!si.identity_transfer_enabled) return;
const g = group(pane, "Best-FaceID: ArcFace projector");
if (!_loraChoices) {
g.append(hintEl("Loading projector list…"));
ensureLoraChoices();
return;
}
const sel = el("select"); sel.dataset.k = "si-identity_projector";
const noneOpt = el("option", null, "None"); noneOpt.value = "None";
if (!si.identity_projector || si.identity_projector === "None") noneOpt.selected = true;
sel.append(noneOpt);
_loraChoices.forEach((c) => {
const o = el("option", null, c); o.value = c;
if (c === si.identity_projector) o.selected = true;
sel.append(o);
});
sel.onchange = () => S.setSamplerInputNow("identity_projector", sel.value);
g.append(field("ArcFace projector", sel,
"Optional secondary identity channel — the overlap reference tokens above carry the bulk of identity even with this set to None."));
}
function renderPane(pane, st) {
const p = st.project;
if (!p) { pane.append(el("div", "pj-meta", "No project open.")); return; }
@@ -582,8 +638,7 @@
case "chain_timing": return renderChainTiming(pane, st);
case "chain_guidance": return renderChainKnobsView(pane, st, "chain_guidance", "Guidance");
case "chain_decode": return renderChainKnobsView(pane, st, "chain_decode", "Decode");
case "chain_experimental": return renderChainKnobsView(pane, st, "chain_experimental", "Experimental",
"Research techniques — off by default; expect quality/overhead trade-offs.");
case "chain_experimental": return renderChainExperimental(pane, st);
default: return renderOverview(pane, st);
}
}
+22
View File
@@ -685,7 +685,29 @@
wrap.append(field(`🔗 ${l.name} (${(l.members || []).length})`, ctrl));
}
});
// A widget input wired from another node's output is replaced by that connection at
// generation — an exposed control for it would edit a value generation ignores, so
// show the wire source instead (mirrors the locked field on the node's own page).
const wireInto = (slot, name) => {
const target = `node:${slot.id}:${name}`;
for (const s2 of (m.slots || [])) {
if (s2.id === slot.id) continue;
for (const [outName, raw] of Object.entries(s2.wires || {})) {
const arr = Array.isArray(raw) ? raw : (raw ? [raw] : []);
if (arr.includes(target)) return { slot: s2, out: outName };
}
}
return null;
};
slotItems.forEach(([slot, d]) => {
const iw = d.isBypass ? null : wireInto(slot, d.name);
if (iw) {
const note = el("div", "lib-sub");
note.textContent = `⇐ ${slotLabel(iw.slot)} · ${iw.out}`;
note.title = `This value comes from ${slotLabel(iw.slot)}'s "${iw.out}" output at generation.`;
wrap.append(field(`${slotLabel(slot)} · ${d.label || d.name}`, note));
return;
}
// Bypass is a slot-level flag (slot.bypassed), not a real node input — route it
// separately so it never gets sent to ComfyUI as a fake widget value.
const ctrl = d.isBypass
+27 -1
View File
@@ -459,13 +459,24 @@
const grid = el("div", "slot-fields");
cand.inputs.forEach((spec) => {
const lk = linkOf(slot.id, spec.name);
const iw = incomingWidgetWire(slot, spec.name);
const f = widgetField(spec, slot.inputs[spec.name], async (v) => {
slot.inputs[spec.name] = v;
const l2 = linkOf(slot.id, spec.name); if (l2) applyLinkValue(l2, v); // keep group in sync
await persist();
});
f.classList.add("with-eye");
if (lk) {
if (iw) {
// Wired from another node's output: the widget value is replaced by the
// connection at generation — lock the field and say where it comes from
// (same treatment as a linked control; the eye button is skipped because an
// exposed control would edit a value generation ignores).
f.classList.add("linked");
const ctrl = f.querySelector("input,select"); if (ctrl) ctrl.disabled = true;
const tag = el("span", "link-tag", `⇐ ${slotName(iw.slot)} · ${iw.out}`);
tag.title = `This value comes from ${slotFullLabel(iw.slot)}'s "${iw.out}" output at generation — the local field is ignored while wired.`;
f.append(tag);
} else if (lk) {
f.classList.add("linked");
const ctrl = f.querySelector("input,select"); if (ctrl) ctrl.disabled = true;
f.append(el("span", "link-tag", "🔗 " + lk.name));
@@ -571,6 +582,21 @@
return Array.isArray(raw) ? raw : [raw];
}
// The wire (if any) feeding a WIDGET input of `slot` from another slot's output —
// e.g. ImageTransform.width → EmptyLatent.width. The builder replaces the widget's
// value with that connection at generation, so the local field is dead while wired
// and must render locked (same treatment as a linked control).
function incomingWidgetWire(slot, inputName) {
const target = `node:${slot.id}:${inputName}`;
for (const s2 of config.slots) {
if (s2.id === slot.id) continue;
for (const [outName, raw] of Object.entries(s2.wires || {})) {
if (wireTargets(raw).includes(target)) return { slot: s2, out: outName };
}
}
return null;
}
// ── edge mirroring ────────────────────────────────────────────────────────────
// An output→input edge can be authored from either side: a slot's wires
// ("node:<destId>:<input>") OR the destination's input_sources ("out:<srcId>:<out>").
+133 -20
View File
@@ -463,7 +463,7 @@
// when previewing run 1's video while ComfyUI was GIL-bound saving run 2). Bounded
// backoff, cache-busted so a poisoned browser cache entry can't be replayed.
const MAX_VIDEO_RETRIES = 4;
function _scheduleVideoRetry(url, v) {
function _scheduleVideoRetry(url, v, delayMs) {
const n = (v._pmRetries || 0) + 1;
if (n > MAX_VIDEO_RETRIES) return;
v._pmRetries = n;
@@ -474,7 +474,37 @@
const sep = url.includes("?") ? "&" : "?";
v.src = url + sep + "r=" + Date.now();
v.load();
}, 1500 * n);
}, delayMs != null ? delayMs : 1500 * n);
}
// If the active clip's element changed readiness, repaint the slate/badge over it.
function _refreshClipStatusUi(url) {
if (_currentClip && !_currentClip.pending && _clipUrl(_currentClip) === url) {
_applyClipUi(_currentClip);
}
}
// Failed element → ask the server WHY (HEAD /result now answers 503 while the render is
// still being written to disk) so the slate can say "processing" vs "loading", and so a
// long GIL-bound save can't exhaust the retry budget: server-confirmed "processing"
// retries on a steady cadence with the budget refunded, everything else backs off.
function _classifyVideoError(url, v) {
const probe = url.split("&r=")[0].split("?r=")[0];
fetch(probe, { method: "HEAD", cache: "no-store" })
.then((r) => (r.status === 503 ? "processing" : "loading"))
.catch(() => "loading")
.then((status) => {
if (pool.get(url) !== v || !_urlClips.has(url)) return;
v._pmStatus = status;
if (status === "processing") {
v._pmRetries = 0;
_scheduleVideoRetry(url, v, 3000);
} else {
if ((v._pmRetries || 0) >= MAX_VIDEO_RETRIES) v._pmStatus = "failed";
_scheduleVideoRetry(url, v);
}
_refreshClipStatusUi(url);
});
}
function _ensureVideo(url, clip) {
@@ -494,10 +524,13 @@
// _updatePreloadHints() promotes the current clip and its neighbours to auto.
v.className = "pm-video"; v.preload = "metadata"; v.playsInline = true; v.src = url;
v._pmFx = { viewport, zoom };
v._pmStatus = "loading"; // → "ready" on loadedmetadata; "processing"/"failed" from _classifyVideoError
zoom.append(v);
viewport.append(zoom);
v.addEventListener("loadedmetadata", () => {
v._pmRetries = 0; // healthy again — restore the full retry budget
v._pmStatus = "ready";
_refreshClipStatusUi(url);
const pending = v._pmSeekPending != null ? v._pmSeekPending : (v === _active ? _seekPending : null);
if (pending != null) {
_applyVideoTime(v, pending, !_playing);
@@ -516,6 +549,9 @@
// to 0 and would restart the clip from its beginning. Stash the playhead position so
// loadedmetadata seeks back instead; _onVideoSeeked then resumes playback if playing.
v.addEventListener("emptied", () => {
// A media reset means the bytes are gone — downgrade so the slate covers the element
// instead of it blinking black while the retry reloads.
if (v._pmStatus === "ready") { v._pmStatus = "loading"; _refreshClipStatusUi(url); }
if (v !== _active || !_currentClip || _currentClip.pending) return;
if (_clipUrl(_currentClip) !== url) return;
if (v._pmSeekPending == null) {
@@ -526,8 +562,13 @@
const clips = _urlClips.get(url);
if (clips) [...clips].forEach((c) => _fallbackSegmentClip(c));
// Segment clips fell back to their base URL above (and unregistered); anything still
// registered here has no alternative URL — retry this one instead of staying dead.
if (_urlClips.has(url)) _scheduleVideoRetry(url, v);
// registered here has no alternative URL — classify the failure (processing vs dead)
// and retry this one instead of staying dead.
if (_urlClips.has(url)) {
v._pmStatus = "loading";
_refreshClipStatusUi(url);
_classifyVideoError(url, v);
}
});
v.addEventListener("ended", () => {
if (v !== _active || _isSeeking() || _clipBoundaryToken) return;
@@ -538,33 +579,87 @@
return v;
}
// Materializing a <video> per clip URL for the WHOLE timeline breaks big (montage)
// timelines: Chrome refuses new media players past ~75 per page, and every element's
// preload=metadata fetch of a per-clip preview segment (each a server-side ffmpeg
// encode) fights over the browser's ~6 per-origin connections on a tunneled instance —
// so clips in the back half of the timeline sit "loading" forever while the front half
// plays. Only clips in a window around the playhead get elements; the window keeps cut
// boundaries instant (upcoming clips are materialized ahead) while the element count
// stays flat no matter how long the timeline is.
const POOL_BEHIND = 2, POOL_AHEAD = 8, POOL_MAX = 16;
function _windowUrls() {
const urls = new Set();
let idx = _currentClip ? _clips.indexOf(_currentClip) : -1;
if (idx < 0) {
const at = _clipAt(_phSec);
idx = at ? _clips.indexOf(at) : -1;
}
if (idx < 0) idx = _clips.findIndex(_hasPlayableClipMedia);
if (idx >= 0) {
for (let i = Math.max(0, idx - POOL_BEHIND); i < _clips.length && i <= idx + POOL_AHEAD; i++) {
const c = _clips[i];
if (!c || c.pending || c.blank) continue;
const u = _clipUrl(c);
if (u) urls.add(u);
}
}
return urls;
}
function _releasePooledVideo(u, v) {
pool.delete(u);
if (_active === v) _active = null;
v.pause();
clearTimeout(v._pmRetryTimer);
// Release the socket BEFORE detaching: a detached media element that still has a
// src holds its per-origin connection until GC (media-connection-pool class).
try { v.removeAttribute("src"); v.load(); } catch (_) {}
v._pmFx?.viewport.remove();
}
// Materialize window members that are missing; evict far elements once the pool
// outgrows POOL_MAX (the slack keeps recently-visited clips warm for back-and-forth
// scrubbing instead of re-fetching on every jump). Runs at every playhead-affecting
// step via _updatePreloadHints, and is idempotent/cheap when nothing changed.
function _syncPoolWindow() {
const want = _windowUrls();
for (const u of want) {
if (pool.has(u)) continue;
const clip = _clips.find((c) => _clipUrl(c) === u);
if (clip) _ensureVideo(u, clip);
}
if (pool.size <= POOL_MAX) return;
for (const [u, v] of [...pool]) { // Map order = insertion order ⇒ oldest first
if (pool.size <= POOL_MAX) break;
if (want.has(u) || v === _active) continue;
_releasePooledVideo(u, v);
}
}
function _syncPool() {
_urlClips.clear();
const urls = new Set(_clips.filter((c) => c.media || c.binUrl || c.streamUrl).map((c) => _clipUrl(c)));
urls.forEach((u) => {
// Timeline rebuild: drop elements whose URL no longer exists on any clip at all
// (deleted/regenerated clips) regardless of the window.
const live = new Set(_clips.filter((c) => c.media || c.binUrl || c.streamUrl).map((c) => _clipUrl(c)));
for (const [u, v] of [...pool]) {
if (live.has(u)) continue;
_releasePooledVideo(u, v);
}
_syncPoolWindow();
for (const [u, v] of pool) {
const clip = _clips.find((c) => _clipUrl(c) === u);
const v = _ensureVideo(u, clip);
if (clip) _registerUrlClip(u, clip);
// Clip rebuild (e.g. a run just completed): give exhausted-but-errored elements a
// fresh recovery attempt — the failure was likely the server being busy back then.
if (v.error && !v._pmRetryTimer) { v._pmRetries = 0; _scheduleVideoRetry(u, v); }
});
for (const [u, v] of [...pool]) {
if (urls.has(u)) continue;
v.pause();
clearTimeout(v._pmRetryTimer);
v._pmFx?.viewport.remove();
pool.delete(u);
if (_active === v) { _active = null; }
}
}
function _resetPool() {
_stopTick();
for (const [, v] of pool) {
v.pause();
clearTimeout(v._pmRetryTimer);
v._pmFx?.viewport.remove();
}
for (const [u, v] of [...pool]) _releasePooledVideo(u, v);
pool.clear(); _active = null; _currentClip = null; _seekPending = null; _playPending = false;
_urlClips.clear();
_clipBoundaryToken = 0;
@@ -665,6 +760,23 @@
_showSlate(clip?.slate || "Not generated yet", clip?.slateSub);
return;
}
// A clip whose element hasn't decoded yet must say so instead of showing a black or
// half-loaded <video>: "processing" = the server answered 503 (render still being
// written to disk), "loading" = bytes on the way / retrying, "failed" = retry budget
// exhausted (re-armed on the next clip rebuild). Playback through _playPending already
// waits for loadedmetadata, so the preview only becomes live once the video really is.
const v = clip.blank ? null : pool.get(_clipUrl(clip));
if (v && v._pmStatus !== "ready") {
_showBadge("");
if (v._pmStatus === "processing") {
_showSlate("Video is processing…", "Preview isn't available yet");
} else if (v._pmStatus === "failed") {
_showSlate("Preview unavailable", "The video failed to load — it will retry when the timeline updates");
} else {
_showSlate("Video is loading…", "Preview isn't available yet");
}
return;
}
_showSlate("");
const badge = clip.ghost ? (clip.slate || "Removed — preview only")
: _segmentSourceBadge(clip.sceneId);
@@ -677,6 +789,7 @@
// stay instant); everything else idles at metadata so the pool doesn't saturate the
// browser's per-origin connections. A hint change never interrupts an in-flight load.
function _updatePreloadHints() {
_syncPoolWindow(); // playhead moved — materialize upcoming clips, evict far ones
if (!pool.size) return;
const hot = new Set();
const add = (c) => {
+5 -3
View File
@@ -841,10 +841,12 @@
if (_commitPromise) { _commitQueued = true; return _commitPromise; }
const skipPreviewRefresh = !!opts.skipPreviewRefresh;
_commitPromise = (async () => {
const snapshot = JSON.parse(JSON.stringify(state.project));
// Sync BEFORE snapshotting: the snapshot is what gets uploaded, and a snapshot taken
// first carries the models config (and renders/ghosts) from the PREVIOUS sync — every
// autosave then silently reverted exposed-control edits (bypass/input/link changes
// saved via /models) by overwriting p.models on the server with the stale copy.
_syncEditorStateToProject();
snapshot.scene_renders = state.project.scene_renders;
snapshot.scene_ghosts = state.project.scene_ghosts;
const snapshot = JSON.parse(JSON.stringify(state.project));
const previewKeyBefore = _promptPreviewKey(snapshot);
const selBefore = JSON.stringify(state.selectedSceneIds || []);
const metaBefore = { name: snapshot.name, scene_count: (snapshot.scenes || []).length };
+91 -39
View File
@@ -201,6 +201,23 @@
});
}
// Filmstrip probes run through a small shared queue: painting a big (montage) timeline
// used to start one preload=auto <video> per clip AT ONCE — each of them a server-side
// ffmpeg segment encode — saturating both the browser's ~6 per-origin connections and
// the server, and starving the player pool (clips stuck "loading" on tunneled
// instances). Two at a time keeps thumbnails flowing without wedging playback.
const _fsQueue = [];
let _fsActive = 0;
const FS_CONCURRENCY = 2;
function _fsPump() {
while (_fsActive < FS_CONCURRENCY && _fsQueue.length) {
const job = _fsQueue.shift();
_fsActive++;
job().catch(() => {}).then(() => { _fsActive--; _fsPump(); });
}
}
function _fsEnqueue(job) { _fsQueue.push(job); _fsPump(); }
function appendFilmstrip(clip, st, scene, widthPx) {
if (S.isVideoClip && S.isVideoClip(scene)) return;
if (!hasRender(st, scene.id)) return;
@@ -210,46 +227,70 @@
const strip = el("div", "clip-filmstrip");
clip.append(strip);
const n = Math.min(10, Math.max(3, Math.floor(widthPx / 28)));
const url = window.MovieEditorAPI.resultUrl(st.project.id, r.media);
const vid = document.createElement("video");
vid.muted = true; vid.preload = "auto"; vid.src = url;
const captureAt = (time) => new Promise((resolve) => {
if (!strip.isConnected) { resolve(); return; }
const cell = el("div", "clip-fs-cell");
const c = document.createElement("canvas");
c.width = 48; c.height = 27;
const ctx = c.getContext("2d");
const thumb = el("img", "clip-fs-thumb");
thumb.alt = "";
const done = () => {
if (!strip.isConnected) { resolve(); return; }
try { ctx.drawImage(vid, 0, 0, 48, 27); thumb.src = c.toDataURL("image/jpeg", 0.55); } catch (_) {}
cell.append(thumb);
strip.append(cell);
resolve();
const API = window.MovieEditorAPI;
const durSec = sDur(scene, st.project);
// Route through the same trimmed, faststart preview segment the player uses instead of
// the raw chain output: that file is moov-at-end (deep seeks stall/black in the browser)
// and shared across every scene in the run, so a filmstrip per clip meant N full-file
// downloads with preload=auto fighting the player pool over Chrome's 6-per-origin cap.
const segUrl = API.previewSegmentUrl && st.project.id
? API.previewSegmentUrl(st.project.id, scene.id, { media: r.media, renderIn: r.inSec || 0, dur: durSec })
: null;
const url = segUrl || API.resultUrl(st.project.id, r.media);
_fsEnqueue(() => new Promise((finish) => {
if (!strip.isConnected) { finish(); return; } // re-rendered before its turn came up
const vid = document.createElement("video");
vid.muted = true; vid.preload = "auto"; vid.src = url;
// Free the media connection once the strip is captured (or abandoned) — a detached
// <video> keeps its socket until GC, and those add up against Chrome's 6-per-origin
// cap. Also settles the queue job; the guard timer keeps a wedged load (server busy,
// dead tunnel) from clogging a queue slot forever.
let settled = false;
const release = () => {
if (settled) return;
settled = true;
clearTimeout(guard);
try { vid.removeAttribute("src"); vid.load(); } catch (_) {}
finish();
};
vid.onseeked = done;
const dur = vid.duration || sDur(scene, st.project);
vid.currentTime = Math.max(0, Math.min(time, dur - 0.01));
});
// Free the media connection once the strip is captured (or abandoned) — a detached
// <video> keeps its socket until GC, and those add up against Chrome's 6-per-origin cap.
const release = () => { try { vid.removeAttribute("src"); vid.load(); } catch (_) {} };
vid.onerror = release;
vid.onloadeddata = async () => {
if (!strip.isConnected) { release(); return; }
const dur = vid.duration || sDur(scene, st.project);
const inSec = (r.inSec || 0) + (scene.source_in || 0);
try {
for (let i = 0; i < n; i++) {
if (!strip.isConnected) return;
const t = inSec + (dur * (i + 0.5) / n);
await captureAt(t);
const guard = setTimeout(release, 20000);
const captureAt = (time) => new Promise((resolve) => {
if (!strip.isConnected) { resolve(); return; }
const cell = el("div", "clip-fs-cell");
const c = document.createElement("canvas");
c.width = 48; c.height = 27;
const ctx = c.getContext("2d");
const thumb = el("img", "clip-fs-thumb");
thumb.alt = "";
const done = () => {
if (!strip.isConnected) { resolve(); return; }
try { ctx.drawImage(vid, 0, 0, 48, 27); thumb.src = c.toDataURL("image/jpeg", 0.55); } catch (_) {}
cell.append(thumb);
strip.append(cell);
resolve();
};
vid.onseeked = done;
const dur = vid.duration || durSec;
vid.currentTime = Math.max(0, Math.min(time, dur - 0.01));
});
vid.onerror = release;
vid.onloadeddata = async () => {
if (!strip.isConnected) { release(); return; }
const dur = vid.duration || durSec;
// A segment starts at 0 (it's already trimmed to this scene); the raw-file fallback
// still needs the absolute in-point.
const inSec = segUrl ? 0 : (r.inSec || 0) + (scene.source_in || 0);
try {
for (let i = 0; i < n; i++) {
if (!strip.isConnected) return;
const t = inSec + (dur * (i + 0.5) / n);
await captureAt(t);
}
} finally {
release();
}
} finally {
release();
}
};
};
}));
}
const hasRender = (st, sceneId) => !!(st.sceneRenders && st.sceneRenders[sceneId] && st.sceneRenders[sceneId].media);
@@ -1259,7 +1300,18 @@
const binRef = scene.source?.media_ref;
const binAsset = binRef ? (st.mediaBin || []).find((m) => m.id === binRef) : null;
if (r?.media && st.project?.id) {
_attachWaveform(canvas, `scene-aud:${scene.id}:${inSec.toFixed(3)}:${durSec.toFixed(3)}`, window.MovieEditorAPI.resultUrl(st.project.id, r.media), {
// Same reasoning as appendFilmstrip: the raw render is the shared, moov-at-end chain
// output. A plain resultUrl fetch here downloaded and decoded the WHOLE multi-scene
// file once per scene that shares it (fighting the player pool for connections); the
// trimmed segment is this scene's span only, so no inSec slice is needed to paint it.
const API = window.MovieEditorAPI;
const segUrl = API.previewSegmentUrl
? API.previewSegmentUrl(st.project.id, scene.id, { media: r.media, renderIn: r.inSec || 0, dur: durSec })
: null;
_attachWaveform(canvas, `scene-aud:${scene.id}:${inSec.toFixed(3)}:${durSec.toFixed(3)}`, segUrl || API.resultUrl(st.project.id, r.media), segUrl ? {
width: w,
color: "rgba(45, 212, 191, 0.5)",
} : {
width: w,
inSec,
durationSec: durSec,
+222 -2
View File
@@ -209,6 +209,75 @@ def _ffmpeg_trim_clip(src_path: str, out_path: str, inn, dur, fast: bool = False
raise RuntimeError((proc.stderr or "ffmpeg failed")[-1000:])
_ISOBMFF_EXTS = (".mp4", ".m4v", ".mov") # containers where moov/mdat atom order applies
def _is_isobmff(path: str) -> bool:
import os
return os.path.splitext(path)[1].lower() in _ISOBMFF_EXTS
def _moov_position(path: str) -> str:
"""Where the MP4's moov atom sits: "front" (before mdat — browsers can decode from a
progressive/Range fetch), "end" (after mdat — ComfyUI's own VHS_VideoCombine output; a
complete file, but deep-seeking it raw is undecodable in the browser), or "none" (no moov
anywhere — the file is still being written, or a save was aborted; ffmpeg can't read it
either, so it must never be served as video). Distinguishing "end" from "none" is what
lets /result tell "remux me" apart from "come back later"."""
import os
try:
with open(path, "rb") as f:
size = os.fstat(f.fileno()).st_size
offset = 0
seen_mdat = False
while offset + 8 <= size:
f.seek(offset)
header = f.read(8)
if len(header) < 8:
break
box_size = int.from_bytes(header[:4], "big")
box_type = header[4:8]
if box_type == b"moov":
return "end" if seen_mdat else "front"
if box_type == b"mdat":
seen_mdat = True
if box_size == 1:
ext = f.read(8)
if len(ext) < 8:
break
box_size = int.from_bytes(ext, "big")
elif box_size == 0:
# Box runs to EOF without a size (mdat still being streamed to disk —
# ffmpeg backfills the size on finalize): nothing after it, no moov.
return "none"
if box_size < 8:
break
offset += box_size
except OSError:
pass
return "none"
def _file_sig(path: str) -> tuple:
"""Change detector for cache keys: same (mtime_ns, size) ⇒ same bytes for our purposes."""
import os
st = os.stat(path)
return (st.st_mtime_ns, st.st_size)
def _remux_faststart(src_path: str, out_path: str) -> None:
"""Stream-copy remux (no re-encode) that just relocates moov to the front."""
import shutil
import subprocess
ff = shutil.which("ffmpeg")
if not ff:
raise RuntimeError("ffmpeg not found on PATH.")
cmd = [ff, "-y", "-i", src_path, "-c", "copy", "-movflags", "+faststart", out_path]
proc = subprocess.run(cmd, capture_output=True, text=True)
if proc.returncode != 0:
raise RuntimeError((proc.stderr or "ffmpeg failed")[-1000:])
def _playback_render_from_query(query) -> Optional[dict]:
"""Optional live render override from preview-segment query params."""
filename = query.get("filename") if query else None
@@ -281,6 +350,11 @@ def _scene_playback_clip_spec(
_preview_segment_cache: dict[str, str] = {}
_preview_segment_locks: dict = {} # cache key -> asyncio.Lock (dedupe concurrent encodes)
_faststart_cache: dict[str, tuple] = {} # source path -> (source sig, remuxed faststart path)
_faststart_failed: dict[str, tuple] = {} # source path -> (sig, monotonic ts of last failed remux attempt)
_faststart_locks: dict = {} # source path -> asyncio.Lock (dedupe concurrent remuxes)
_FASTSTART_RETRY_SEC = 30.0 # min gap between remux re-attempts on a file that failed to remux
def _preview_segment_cache_key(pid: str, scene_id: str, clip: dict) -> str:
return (
@@ -1749,6 +1823,105 @@ if web is not None and PromptServer is not None:
async def _temp_list(_req):
return web.json_response({"files": _list_temp_media()})
async def _faststart_path(src_path: str) -> str:
"""Serve a stream-copy-remuxed (moov-at-front) copy in place of a raw ComfyUI render.
VHS_VideoCombine writes moov at the end; deep-seeking that directly in the browser is
exactly what preview-segment already routes chain clips around, but a "shallow" single
clip (or just opening one) hits /result directly and gets the same undecodable-on-seek
file. Cached + lock-deduped like the segment cache below; skipped entirely (no-op) for
anything that already has moov up front, e.g. FunPack's own segment/export/final-render
output, which already sets +faststart itself. MP4/MOV only (moov/mdat is an ISO-BMFF
concept — a non-.mp4 container falling through the caller's extension gate would just
get echoed back as-is here, not silently mis-remuxed into a mismatched container).
A file with NO moov (still being written, or an aborted save) raises 503 instead of
being served: raw moov-less bytes are undecodable, and once the write finishes the
same URL would start serving different bytes — a browser mid-Range-stream then mixes
offsets (PIPELINE decode errors) or fails If-Range and restarts the clip from 0.
Caches are keyed on the source's (mtime, size) signature and the remux output name is
derived from path+signature, so a rewritten source can't serve a stale copy and two
sources sharing a basename can't overwrite each other's remux mid-stream.
A moov-at-end file must NEVER be served raw, whatever goes wrong here: deep seeks
into it are undecodable in the browser (Firefox: NS_ERROR_DOM_MEDIA_DECODE_ERR /
AppleVTDecoder reference-missing), and the element's cache-busted retries then fail
identically forever. A remux failure answers 502 (rate-limited re-attempts) instead
of falling back to raw bytes — an honest "Preview unavailable" beats undecodable video."""
import asyncio
import hashlib
import os
import time
if not _is_isobmff(src_path):
return src_path
try:
sig = _file_sig(src_path)
except OSError:
return src_path # caller re-checks existence / falls through to 404
cached = _faststart_cache.get(src_path)
if cached and cached[0] == sig and os.path.isfile(cached[1]):
return cached[1]
lock = _faststart_locks.setdefault(src_path, asyncio.Lock())
async with lock:
try:
sig = _file_sig(src_path)
except OSError:
return src_path
cached = _faststart_cache.get(src_path)
if cached and cached[0] == sig and os.path.isfile(cached[1]):
return cached[1]
pos = _moov_position(src_path)
if pos == "front":
return src_path
if pos == "none":
# Mid-write (or truncated junk): never serve — undecodable, and the bytes
# will change under this URL. 503 lets the player's retry loop try again.
raise web.HTTPServiceUnavailable(
reason="Video file is still being written.", headers={"Retry-After": "2"})
failed = _faststart_failed.get(src_path)
if failed and failed[0] == sig and time.monotonic() - failed[1] < _FASTSTART_RETRY_SEC:
raise web.HTTPBadGateway(
reason="Video could not be prepared for playback (remux failed).")
try:
import folder_paths
tempdir = folder_paths.get_temp_directory()
except Exception:
# No temp dir means no seek-safe copy can be made; raw moov-at-end bytes
# would be undecodable, so tell the player to retry rather than serve them.
raise web.HTTPServiceUnavailable(
reason="Temp directory unavailable.", headers={"Retry-After": "5"})
digest = hashlib.md5(f"{src_path}|{sig[0]}|{sig[1]}".encode()).hexdigest()[:16]
ext = os.path.splitext(src_path)[1].lower()
out_path = os.path.join(tempdir, f"funpack_faststart_{digest}{ext}")
err = None
try:
# to_thread: same event-loop-freeze hazard as preview-segment encodes below —
# this is a stream copy (no re-encode) so it's fast, but still blocking I/O.
await asyncio.to_thread(_remux_faststart, src_path, out_path)
ok = os.path.getsize(out_path) > 0 and _moov_position(out_path) == "front"
except Exception as e: # noqa: BLE001
ok = False
err = e
try:
sig_now = _file_sig(src_path)
except OSError:
sig_now = None # deleted under us — retry will 404 definitively
if ok and sig_now == sig:
_faststart_cache[src_path] = (sig, out_path)
_faststart_failed.pop(src_path, None)
return out_path
try:
os.remove(out_path)
except OSError:
pass
if sig_now != sig:
# Source changed under the remux — it's being (re)written right now.
raise web.HTTPServiceUnavailable(
reason="Video file is still being written.", headers={"Retry-After": "2"})
print(f"[FunPack] faststart remux failed for {src_path}: {err}")
_faststart_failed[src_path] = (sig, time.monotonic())
raise web.HTTPBadGateway(
reason="Video could not be prepared for playback (remux failed).")
@routes.get(UI_PREFIX + "/api/projects/{pid}/result")
async def _result(req):
import os
@@ -1761,6 +1934,12 @@ if web is not None and PromptServer is not None:
_project_or_404(req.match_info["pid"])
if not path or not os.path.isfile(path):
raise web.HTTPNotFound()
# Mirror the GET path's mid-write gate: a moov-less mp4 would 503 on GET, so a
# HEAD claiming 200 would lie — the player probes HEAD after a media error to
# tell "render still being written" (show "processing") apart from a dead file.
if _is_isobmff(path) and _moov_position(path) == "none":
raise web.HTTPServiceUnavailable(
reason="Video file is still being written.", headers={"Retry-After": "2"})
return web.Response()
# Serve straight from disk: FileResponse streams with Range support, so the player
# can seek and a busy ComfyUI can't stall the transfer. The old path buffered the
@@ -1769,6 +1948,8 @@ if web is not None and PromptServer is not None:
if path and os.path.isfile(path):
import mimetypes as _mt
ctype = _mt.guess_type(path)[0] or "application/octet-stream"
if ctype.startswith("video/"):
path = await _faststart_path(path)
return web.FileResponse(path, headers={"Content-Type": ctype})
# Fallback (custom output dirs the resolver doesn't know): loopback /view fetch.
try:
@@ -1779,6 +1960,31 @@ if web is not None and PromptServer is not None:
)
except Exception as e: # noqa: BLE001
raise web.HTTPBadGateway(reason=f"could not fetch result: {e}")
fname = req.query.get("filename", "")
if _is_isobmff(fname):
# A buffered full-body mp4/mov response has no Range support and, for a raw
# ComfyUI render, moov at the end — both make seeking undecodable in the
# browser. Spool the fetched bytes to temp and serve through the same
# remux-and-FileResponse path as resolver hits.
import hashlib
try:
import folder_paths
tempdir = folder_paths.get_temp_directory()
except Exception:
raise web.HTTPServiceUnavailable(
reason="Temp directory unavailable.", headers={"Retry-After": "5"})
ext = os.path.splitext(fname)[1].lower()
digest = hashlib.md5(
f"{fname}|{req.query.get('subfolder', '')}|"
f"{req.query.get('type', 'output')}|{len(data)}".encode()).hexdigest()[:16]
spool = os.path.join(tempdir, f"funpack_viewspool_{digest}{ext}")
if not os.path.isfile(spool) or os.path.getsize(spool) != len(data):
tmp = spool + ".part"
with open(tmp, "wb") as f:
f.write(data)
os.replace(tmp, spool) # atomic: no reader ever sees a partial spool
path = await _faststart_path(spool)
return web.FileResponse(path, headers={"Content-Type": ctype.split(";")[0]})
return web.Response(body=data, content_type=ctype.split(";")[0])
# --- API: final render (async job — ffmpeg can exceed Cloudflare tunnel timeouts) ---
@@ -1831,7 +2037,22 @@ if web is not None and PromptServer is not None:
clip = _ghost_playback_clip_spec(req.query)
if clip is None:
return web.json_response({"detail": str(e)}, status=404)
key = _preview_segment_cache_key(pid, scene_id, clip)
try:
src_path = _resolve_clip_src_path(clip)
src_sig = _file_sig(src_path)
except FileNotFoundError as e:
return web.json_response({"detail": str(e)}, status=400)
except OSError as e:
return web.json_response({"detail": str(e)}, status=503)
if _is_isobmff(src_path) and _moov_position(src_path) == "none":
# No moov ⇒ still being written (VHS finalizes it last) or an aborted save:
# ffmpeg can't trim it and the segment would come out truncated/undecodable.
return web.json_response(
{"detail": "Source video is still being written."},
status=503, headers={"Retry-After": "2"})
# Source signature in the key: a re-rendered source at the SAME path must not keep
# serving the segment encoded from the old bytes (stale previews).
key = _preview_segment_cache_key(pid, scene_id, clip) + f":{src_sig[0]}:{src_sig[1]}"
# Serialize per segment: the pool preloads every clip at once after a run, and a
# scrub can re-request the same URL — without this lock each request would start
# its own duplicate encode of the same segment.
@@ -1851,7 +2072,6 @@ if web is not None and PromptServer is not None:
f"funpack_preview_{pid[:8]}_{scene_id}_{int(_time.time() * 1000)}.mp4",
)
try:
src_path = _resolve_clip_src_path(clip)
# to_thread: this encode takes seconds — run inline it would freeze the ONE
# aiohttp event loop ComfyUI has, stalling every playing /result stream and
# the whole editor API for its duration (stale/black previews on scrub).
+1 -1
View File
@@ -2,7 +2,7 @@
[project]
name = "ComfyUI-FunPack" # Unique identifier for your node. Immutable after creation.
description = "An AI movie studio inside ComfyUI: the Cutting Room browser-based video editor with timeline, per-scene prompts and one-click montage generation, plus a self-improving engine that learns from your ratings. Includes the full node stack (Studio, Refiner V2, LTXAV Scene Chain Sampler) for classic graph workflows."
version = "3.2.1" # Custom Node version. Must be semantically versioned.
version = "3.3.0" # Custom Node version. Must be semantically versioned.
license = { text = "GNU GPL v3" }
dependencies = [] # Filled in from requirements.txt
+2
View File
@@ -3,3 +3,5 @@ huggingface_hub
safetensors
numpy
Pillow
insightface
opencv-python-headless
+506 -149
View File
@@ -1869,7 +1869,15 @@ def sample_funpack_distilled_flow(model, x, sigmas, extra_args=None, callback=No
(False, False): alg_sharp_latent_image,
(True, False): anchor_blurred,
(False, True): tail_blurred,
(True, True): both_blurred if both_blurred is not None else (anchor_blurred or tail_blurred),
# both → anchor → tail → None. Explicit None checks, NOT `anchor_blurred or
# tail_blurred`: `or` calls bool() on the first operand, which is a multi-element
# tensor when anchor blur is on but tail blur is off (both_blurred is then None) —
# "Boolean value of Tensor is ambiguous". This value is only a defensive fallback:
# when (True, True) is actually indexed, alg_anchor_on and alg_tail_on are both
# True, so both_blurred is non-None and the chain never reaches the singles.
(True, True): (both_blurred if both_blurred is not None
else anchor_blurred if anchor_blurred is not None
else tail_blurred),
}
alg_active = alg_anchor_on or alg_tail_on
if alg_active:
@@ -2398,11 +2406,49 @@ class FunPackLTXAVSceneChainSampler:
}),
"identity_transfer_enabled": ("BOOLEAN", {
"default": False,
"tooltip": "EXPERIMENTAL Best-FaceID compatibility: tags Continuity's 'Identity pin' guide (Engine settings) with an extra source-phase RoPE rotation, matching the exact overlap+source_phase conditioning Best-FaceID-style identity LoRAs were trained on. No-op without an identity pin image set. Load the LoRA itself the normal way (Models -> add a LoRA loader onto the model path).",
"tooltip": "EXPERIMENTAL Best-FaceID compatibility: full native port of the overlap+source_phase+ArcFace conditioning Best-FaceID-style identity LoRAs were trained on. Replaces Continuity's 'Identity pin' guide (Engine settings) with separate, non-rendered reference tokens (never blended into frame 0) plus optional ArcFace projector tokens on the text context. No-op without an identity pin image set. Load the LoRA itself the normal way (Models -> add a LoRA loader onto the model path).",
}),
"identity_transfer_phase": ("FLOAT", {
"default": 2.0, "min": 0.0, "max": 8.0, "step": 0.5,
"tooltip": "Source-phase segment id for the identity-pin guide tokens (ltx-trainer's overlap+source_phase convention used 2). 0 disables the rotation while leaving the guide itself active.",
"identity_projector": (cls._identity_projector_choices(), {
"default": "None",
"tooltip": "ArcFace projector .safetensors (from models/loras). 'None' = overlap only (the projector is a weak secondary channel; the overlap latent carries the bulk of identity).",
}),
"source_id": ("FLOAT", {
"default": 2.0, "min": 0.0, "max": 8.0, "step": 1.0,
"tooltip": "source_phase segment id for the overlap reference tokens (ltx-trainer's overlap+source_phase convention used 2). 0 disables the RoPE rotation while leaving the overlap tokens active.",
}),
"phase_scale": ("FLOAT", {
"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.1,
"tooltip": "Multiplier on source_id before the RoPE rotation.",
}),
"id_strength": ("FLOAT", {
"default": 1.0, "min": 0.0, "max": 50.0, "step": 0.5,
"tooltip": "Multiplies the ArcFace projector tokens (only when identity_projector is set). Weak channel; push high (5-20) to test, very high may add artifacts.",
}),
"arcface_mode": (["auto_adjust", "as_is", "disable"], {
"default": "auto_adjust",
"tooltip": "auto_adjust: retry face detection with zoom-out/upscale, skip projector tokens if none found. as_is: detect on the image only. disable: skip ArcFace, use only the overlap latent.",
}),
"debug_log": ("BOOLEAN", {
"default": False,
"tooltip": "Print per-scene identity-transfer shape/status logs to the console.",
}),
# NOTE: append-only — keep new sampler widgets at the END of this block so the
# builder's positional reference-workflow mapping (extract_widgets) stays aligned.
"carry_overlap_through_anchor": ("BOOLEAN", {
"default": False,
"tooltip": "When a scene switches to its own i2v anchor (funpack_scene_anchors — a different reference image/character than the previous scene), still carry frame_overlap latent frames from the previous scene's tail into the frames right after the anchor image, instead of the default hard cut with no carried context. Preserves background/environment continuity through an anchor change (e.g. a Best-FaceID identity_transfer scene swapping the reference face mid-chain). The anchor image's own leading frame is never touched by the carried tail. No effect on scenes without a per-scene anchor.",
}),
"plateau_cache": ("BOOLEAN", {
"default": False,
"tooltip": "EXPERIMENTAL speed (MixCache/Chorus-family step-caching, adapted to LTX2.3's distilled schedule): the near-pure-noise plateau steps (sigma above plateau_cache_threshold) carry almost no signal, so the model's output barely changes across them. This computes the full transformer forward once at the top of the plateau, then REUSES that output for the remaining plateau steps instead of recomputing — skipping those transformer passes entirely. On the default 8-step schedule (sigmas 1.0→0.975 are the plateau) that's ~3-4 of 8 forwards skipped. DETERMINISTIC given seed (no diversity/rating impact, safe in Batch Training) but an APPROXIMATION — validate A/B before trusting on final renders. Note: much of wall-clock time is outside the sampler (encode/decode), so sampler speedup ≠ total speedup. Off by default. UNVALIDATED LIVE.",
}),
"plateau_cache_threshold": ("FLOAT", {
"default": 0.975, "min": 0.5, "max": 0.999, "step": 0.005,
"tooltip": "Steps whose sigma is at or above this value count as the reusable plateau (matches the alg_guide_blur_sigma_threshold convention). Higher = fewer steps cached (safer, less speedup); lower = more steps cached (faster, more approximation). 0.975 catches the documented near-pure-noise plateau (schedule steps 1-5) while leaving structure formation (sigma 0.909 onward) fully computed. Only used when plateau_cache is on.",
}),
"taste_nearest_prompt": ("BOOLEAN", {
"default": False,
"tooltip": "EXPERIMENTAL: source embed_guidance / score_slider from the taste direction learned on the prompts NEAREST this scene's prompt, instead of the single global liked-direction average. On every liked rating the Refiner records (prompt fingerprint -> that run's liked direction); with this on, each scene retrieves the similarity-weighted direction of its closest matches (a forest prompt pulls what worked on forests, not the mean across all prompts). Non-parametric retrieval — no extra model forward, just a cosine lookup + vector mean, and it can't collapse into a spurious attractor the way a value function can. Falls back to the global liked direction when no rated prompt is close enough (or the index is empty). Only affects embed_guidance / score_slider; needs refinement_key_input (or embed_guidance_source=absolute). UNVALIDATED LIVE.",
}),
},
"hidden": {
@@ -2420,6 +2466,14 @@ class FunPackLTXAVSceneChainSampler:
"Use with FunPack Studio split-by-transitions output."
)
@classmethod
def _identity_projector_choices(cls):
try:
import folder_paths
return ["None"] + folder_paths.get_filename_list("loras")
except Exception:
return ["None"]
def _is_nested(self, samples):
return bool(getattr(samples, "is_nested", False))
@@ -3100,6 +3154,17 @@ class FunPackLTXAVSceneChainSampler:
return None
return None
def _scene_temporal_loop(self, scene_conditioning):
"""Per-scene loop intent baked in by Studio's "auto" temporal director
(funpack_temporal_loop). True → install the Mobius latent-roll wrapper."""
if (
isinstance(scene_conditioning, (list, tuple))
and len(scene_conditioning) >= 2
and isinstance(scene_conditioning[1], dict)
):
return bool(scene_conditioning[1].get("funpack_temporal_loop"))
return False
def _scene_temporal_mode(self, scene_conditioning):
"""Per-scene temporal mode tag (e.g. pulse). None when unset."""
if (
@@ -3437,6 +3502,75 @@ class FunPackLTXAVSceneChainSampler:
"""Read the liked conditioning direction from the Refiner's state file."""
return self._load_taste_direction(refinement_key, "liked_dir")
def _load_prompt_dir_index(self, refinement_key):
"""Read the (prompt fingerprint -> liked direction) retrieval index from the taste
store — the per-prompt sibling of the single global liked_dir. Returns a list of
(prompt [dim] fp32, direction [dim] fp32, magnitude) tuples, or [] when the key has
no index yet. Written by conditioning._v2_store_prompt_keyed_direction on every
liked rating; lives in the same refine_v2 clip-state file as liked_dir."""
try:
try:
from .conditioning import refinement_state_path, serializable_to_tensor
except ImportError:
from conditioning import refinement_state_path, serializable_to_tensor
import json as _json
path = refinement_state_path(refinement_key, "clip", prefix="refine_v2")
with open(path, "r", encoding="utf-8") as f:
state = _json.load(f)
global_state = state.get("global", state)
raw = global_state.get("prompt_dir_index", [])
entries = []
for e in raw if isinstance(raw, list) else []:
if not isinstance(e, dict) or e.get("prompt") is None or e.get("direction") is None:
continue
try:
entries.append((serializable_to_tensor(e["prompt"]).float(),
serializable_to_tensor(e["direction"]).float(),
float(e.get("direction_magnitude", 0.0))))
except Exception:
continue
return entries
except Exception:
return []
def _resolve_prompt_keyed_direction(self, index, cond, k=3, min_sim=0.5):
"""Retrieve the liked direction learned on the prompts NEAREST this scene's prompt.
`cond` is the scene's c_crossattn ([.., seq, D]); it is mean-pooled to [D] and
cosine-matched against every indexed prompt fingerprint. Returns a unit [D]
direction = the similarity-weighted mean of the top-k neighbours scoring above
min_sim, or None when nothing is close enough (the caller then falls back to the
global liked_dir). This is the whole point of the feature: a prompt about a forest
pulls the direction that worked on forests, not the average across every prompt
ever rated. No extra model forward — one pooled cosine sweep + a vector mean."""
if not index or not isinstance(cond, torch.Tensor):
return None
try:
pooled = cond.detach().float()
while pooled.dim() > 1:
pooled = pooled.mean(dim=0) # [D], same pooling as _v2_pool_conditioning
pooled_u = torch.nn.functional.normalize(pooled, dim=0)
scored = []
for prompt, direction, _mag in index:
if list(prompt.shape) != list(pooled.shape) or list(direction.shape) != list(pooled.shape):
continue
sim = float(torch.nn.functional.cosine_similarity(
pooled_u, torch.nn.functional.normalize(prompt.to(pooled.device), dim=0), dim=0))
if sim >= min_sim:
scored.append((sim, direction.to(pooled.device)))
if not scored:
return None
scored.sort(key=lambda s: s[0], reverse=True)
acc = None
for sim, direction in scored[:max(1, int(k))]:
term = torch.nn.functional.normalize(direction, dim=0) * sim
acc = term if acc is None else acc + term
if acc is None or float(acc.norm()) <= 0.0:
return None
return torch.nn.functional.normalize(acc, dim=0)
except Exception:
return None
def _build_embed_guidance_wrapper(self, model, liked_dir, strength, value_fn=None):
"""Register a model_function_wrapper that nudges conditioning toward the
liked quality direction at each denoising step. Uses value function gradient
@@ -3576,34 +3710,8 @@ class FunPackLTXAVSceneChainSampler:
return 0, True
return 1 + (at - 1) * time_scale, False
def _tag_last_guide_entry(self, cond, key, value):
"""Best-effort: stamp `key=value` onto the guide_attention_entry _append_guide_attention_entry
just added (the last one), without touching any earlier entry. Used to mark which guide
(e.g. the Best-FaceID identity pin) is eligible for the source-phase RoPE patch — see
_install_identity_phase. No-op (returns cond unchanged) if node_helpers isn't importable
or no entries are present; never blocks generation."""
try:
import node_helpers
except ImportError:
return cond
try:
for t in cond:
entries = t[1].get("guide_attention_entries")
if entries:
tagged = [*entries[:-1], {**entries[-1], key: value}]
return node_helpers.conditioning_set_values(cond, {"guide_attention_entries": tagged})
except Exception:
pass
return cond
def _append_guide_latent(self, chunk, guide_frame, apply_at, strength, positive, negative, vae,
phase_tag=0.0):
"""Append one guide latent frame with LTX guide attention at apply_at.
`phase_tag`: when non-zero, marks this guide's entry with `funpack_identity_phase` so
_install_identity_phase can find its exact token range and apply the source-phase RoPE
rotation the Best-FaceID LoRA was trained against — see [[project_identity_transfer_native]].
"""
def _append_guide_latent(self, chunk, guide_frame, apply_at, strength, positive, negative, vae):
"""Append one guide latent frame with LTX guide attention at apply_at."""
try:
from comfy_extras.nodes_lt import LTXVAddGuide, _append_guide_attention_entry
except ImportError:
@@ -3629,9 +3737,6 @@ class FunPackLTXAVSceneChainSampler:
positive, negative = _append_guide_attention_entry(
positive, negative, pre_filter_count, guide_latent_shape, strength=float(strength)
)
if phase_tag:
positive = self._tag_last_guide_entry(positive, "funpack_identity_phase", float(phase_tag))
negative = self._tag_last_guide_entry(negative, "funpack_identity_phase", float(phase_tag))
result = self._clone_latent(chunk)
tensors = self._latent_tensors(result)
@@ -3795,18 +3900,44 @@ class FunPackLTXAVSceneChainSampler:
out = self._invoke_img2video_inplace(vae, image, chunk, strength)
return out if out is not None else chunk
def _build_mixed_anchor_chunk(self, vae, anchor_meta, latent_template, previous, video_overlap):
"""Mixed source: Img2Video anchor latent only — prior scene overlap is not used."""
def _build_mixed_anchor_chunk(self, vae, anchor_meta, latent_template, previous, video_overlap,
carry_overlap=False):
"""Mixed source: Img2Video anchor latent. By default prior scene overlap is not used
(hard cut). When carry_overlap is set and a previous scene exists, the chunk is first
seeded as a normal continuation (previous scene's tail copied into the leading
video_overlap frames, protected) and the anchor image is then written on top — since
LTXVImgToVideoInplace only overwrites the single encoded image frame's worth of samples
and preserves the rest of the incoming noise_mask, the anchor's own leading frame stays a
hard cut while the remaining carried frames keep the old scene's background/environment."""
filename = (anchor_meta or {}).get("filename")
strength = float((anchor_meta or {}).get("strength", 1.0))
image = self._load_image_tensor(filename) if filename else None
if image is None:
if previous is None:
return self._clone_latent(latent_template)
return self._build_continuation_chunk(latent_template, previous, video_overlap)
base = self._clone_latent(latent_template)
return self._build_continuation_chunk(latent_template, previous, 0)
if carry_overlap and previous is not None and video_overlap > 0:
base = self._build_continuation_chunk(latent_template, previous, video_overlap)
else:
base = self._clone_latent(latent_template)
return self._apply_img2video_to_video_latent(vae, image, base, strength)
def _identity_pin_filename(self, guide_list, scene_media_by_ref, identity_transfer_enabled):
"""First identity_pin-tagged image guide's filename, or None. Mirrors the matching
inside _apply_configured_guides, but callable standalone for branches (like the mixed
i2v anchor path) that don't run the rest of the guide stack against their chunk."""
if not identity_transfer_enabled:
return None
for g in guide_list or []:
if not g or not g.get("enabled", True):
continue
if str(g.get("source", "template")) == "image" and g.get("identity_pin"):
ref = g.get("media_ref")
fn = (scene_media_by_ref or {}).get(ref) if ref else None
if fn:
return fn
return None
def _encode_image_guide_frame(self, filename, vae, ref_tensor):
import os
try:
@@ -3845,7 +3976,7 @@ class FunPackLTXAVSceneChainSampler:
return None
def _append_media_guide_at(self, chunk, filename, frame_idx, apply_at, strength,
positive, negative, vae, phase_tag=0.0):
positive, negative, vae):
chunk_tensors = self._latent_tensors(chunk)
if not chunk_tensors:
return chunk, positive, negative, 0, 0
@@ -3856,15 +3987,15 @@ class FunPackLTXAVSceneChainSampler:
at = self._resolve_frame_index(total, frame_idx)
guide_slice = self._time_slice(guide_frame, at, at + 1)
return self._append_guide_latent(
chunk, guide_slice, apply_at, strength, positive, negative, vae, phase_tag=phase_tag,
chunk, guide_slice, apply_at, strength, positive, negative, vae,
) + (0,)
def _apply_configured_guides(self, chunk, scene_index, guide_list, latent_template,
scene_outputs, scene_media_by_ref, positive, negative, vae,
identity_transfer_enabled=False, identity_transfer_phase=2.0):
identity_transfer_enabled=False):
head_crop = 0
tail_crop = 0
identity_phase_applied = False
identity_ref_filename = None
for g in guide_list or []:
if not g or not g.get("enabled", True):
continue
@@ -3894,20 +4025,21 @@ class FunPackLTXAVSceneChainSampler:
fn = (scene_media_by_ref or {}).get(ref) if ref else None
if not fn:
continue
# The project's Best-FaceID reference face (Continuity's "Identity pin") is the
# ONLY guide eligible for the source-phase RoPE tag — never prior-scene/mid-scene
# guides, which serve motion continuity rather than a trained-LoRA identity lock.
phase_tag = identity_transfer_phase if (identity_transfer_enabled and g.get("identity_pin")) else 0.0
if identity_transfer_enabled and g.get("identity_pin"):
# Best-FaceID full port takes this entry over entirely: the LoRA was
# trained on separate overlap tokens + ArcFace projector tokens, not a
# blended frame-0 keyframe, so skip the plain guide-attention append and
# hand the reference filename back for _install_identity_overlap.
identity_ref_filename = fn
continue
chunk, positive, negative, head, tail = self._append_media_guide_at(
chunk, fn, frame_idx, apply_at, strength, positive, negative, vae, phase_tag=phase_tag,
chunk, fn, frame_idx, apply_at, strength, positive, negative, vae,
)
if phase_tag:
identity_phase_applied = True
else:
continue
head_crop += head
tail_crop += tail
return chunk, positive, negative, head_crop, tail_crop, identity_phase_applied
return chunk, positive, negative, head_crop, tail_crop, identity_ref_filename
def _append_mid_scene_guide(self, chunk, previous_output, positive, negative, vae, strength):
"""Append the middle frame of the previous scene as a guide for the current chunk
@@ -4110,128 +4242,269 @@ class FunPackLTXAVSceneChainSampler:
except Exception:
pass
def _build_plateau_cache_wrapper(self, model, threshold):
"""Plateau step-cache (MixCache/Chorus-family, adapted to LTX2.3's distilled schedule).
Installed INNERMOST so it caches the raw base-model forward; every other per-scene
wrapper (embed/score/dynashift/output guidance, temporal) still layers around it and
post-processes the (cached-or-fresh) prediction. On the near-pure-noise plateau
(sigma >= threshold) the transformer output barely changes step-to-step, so we compute
it once per distinct batch signature and reuse it for the remaining plateau steps,
skipping those full 48-block forwards. Below the threshold every step is computed for
real, and the cache is dropped (frees the held tensors) so structure formation is never
approximated. Deterministic given seed — no effect on batch-variant diversity.
Cache is keyed by (input.shape, cond_or_uncond) so a CFG>1 split cond/uncond pair each
gets its own slot instead of thrashing a single slot. Returns a stats dict the caller
reads after the scene for the run report (never used for control flow)."""
old_wrapper = model.model_options.get("model_function_wrapper")
thr = float(threshold)
stats = {"cache": {}, "reused": 0, "computed": 0}
def _plateau_wrapper(apply_fn, args, _ew=old_wrapper, _st=stats, _thr=thr):
def _run():
if _ew is not None:
return _ew(apply_fn, args)
return apply_fn(args["input"], args["timestep"], **(args.get("c") or {}))
ts = args.get("timestep")
x = args.get("input")
try:
sigma = float(ts.max().item()) if ts is not None else 0.0
key = (tuple(x.shape), tuple(args.get("cond_or_uncond") or ()))
except Exception:
return _run() # can't reason about it safely -> always compute
if sigma < _thr:
_st["cache"].clear() # past the plateau: compute every step, release cached tensors
return _run()
cached = _st["cache"].get(key)
if cached is not None:
_st["reused"] += 1
return cached
out = _run()
_st["cache"][key] = out
_st["computed"] += 1
return out
model.model_options["model_function_wrapper"] = _tag_scene_wrapper(_plateau_wrapper, old_wrapper)
return stats
# ---------------------------------------------------------------------------
# identity_transfer: source-phase RoPE tag for the Best-FaceID-style identity pin.
# identity_transfer: native port of ComfyUI-BFSNodes' LTX Identity Transfer
# (LTXIdentityOverlapConditioning) — see identity_transfer.py for the ArcFace/projector/
# RoPE-rotation pieces. The reference face is injected as SEPARATE tokens appended after
# the target's video tokens (an "overlap" placement, sharing the frame-0 pixel-coord grid),
# given a clean (timestep 0) denoise state, tagged with a source-phase RoPE rotation, and
# trimmed back off before unpatchify — never blended into a real rendered frame, unlike
# FunPack's guide-attention path. Optional ArcFace IdentityProjector tokens are appended to
# the text context on top.
#
# FunPack's guide-attention path (LTXVAddGuide.add_keyframe_index + causal_fix, see
# _append_guide_latent) already places a reference image on the frame-0 grid with a
# tunable attention strength — structurally the same "overlap" placement an identity
# LoRA like Best-FaceID expects. The one piece that path doesn't do is the LoRA's
# trained-time "source_phase" tag: an extra per-token RoPE rotation that marks those
# tokens as coming from a reference image, not the real timeline. This patches that in,
# scoped ONLY to the specific guide entry marked funpack_identity_phase (see
# _tag_last_guide_entry) — never to co-active guides (mid_scene_guide, carry, prior-
# scene continuity) that happen to share the same keyframe_idxs tensor.
#
# Two bound methods are wrapped on model.model.diffusion_model:
# _prepare_timestep (receives resolved_guide_entries / num_guide_tokens via
# kwargs already computed by the model's own guide-token
# accounting) -> locates the tagged entry's exact token
# range and stashes it on self for the very next call.
# _prepare_positional_embeddings (runs right after, same forward pass, no kwargs) ->
# rotates just that token range's RoPE freqs.
# Four bound methods are wrapped on model.model.diffusion_model, matching LTXBaseModel's own
# _forward pipeline: _process_input (append ref tokens) -> _prepare_timestep (clean timestep
# for ref tokens) -> _prepare_positional_embeddings (source-phase RoPE rotation) ->
# _process_output (trim ref tokens before unpatchify).
# Idempotent + tag/strip per scene (same discipline as _install_v2a_scale /
# _strip_funpack_scene_wrappers) so an interrupt mid-scene can never leave a stale patch
# steering a later, unrelated run — see [[project_hook_leak_bug]].
# ---------------------------------------------------------------------------
_IDENTITY_PHASE_TAG = "_funpack_identity_phase_patched"
_IDENTITY_OVERLAP_TAG = "_funpack_identity_overlap_patched"
def _install_identity_phase(self, model):
def _install_identity_overlap(self, model, ref_latent, seg_value):
try:
from .identity_transfer import rotate_overlap_freqs as _rotate_overlap_freqs
except ImportError:
from identity_transfer import rotate_overlap_freqs as _rotate_overlap_freqs
try:
ltxv = model.model.diffusion_model
except Exception:
return None
if getattr(ltxv, self._IDENTITY_PHASE_TAG, False):
if getattr(ltxv, self._IDENTITY_OVERLAP_TAG, False):
return None # already installed (idempotent across scenes/runs)
orig_process_input = ltxv._process_input
orig_prepare_timestep = ltxv._prepare_timestep
orig_prepare_pe = ltxv._prepare_positional_embeddings
orig_process_output = ltxv._process_output
def process_input(self_ltxv, x, keyframe_idxs, denoise_mask, **kw):
out = orig_process_input(x, keyframe_idxs, denoise_mask, **kw)
self_ltxv._funpack_id_ref_len = 0
try:
from comfy.ldm.lightricks.model import latent_to_pixel_coords
xx, pix, add = out
is_av = isinstance(xx, (list, tuple))
vx = xx[0] if is_av else xx
vco = pix[0] if is_av else pix
rt, rlc = self_ltxv.patchifier.patchify(ref_latent.to(dtype=vx.dtype, device=vx.device))
rpc = latent_to_pixel_coords(latent_coords=rlc, scale_factors=self_ltxv.vae_scale_factors,
causal_fix=self_ltxv.causal_temporal_positioning)
rt = self_ltxv.patchify_proj(rt)
if rt.shape[0] != vx.shape[0]:
rt = rt.expand(vx.shape[0], -1, -1)
if rpc.shape[0] != vco.shape[0]:
rpc = rpc.expand(vco.shape[0], *([-1] * (rpc.dim() - 1)))
ref_len = rt.shape[1]
self_ltxv._funpack_id_target_len = vx.shape[1]
vx = torch.cat([vx, rt], dim=1)
vco = torch.cat([vco, rpc.to(vco)], dim=2)
self_ltxv._funpack_id_ref_len = ref_len
if is_av:
xx = [vx, xx[1]]; pix = [vco, pix[1]]
else:
xx, pix = vx, vco
return xx, pix, add
except Exception:
self_ltxv._funpack_id_ref_len = 0
return out
def prepare_timestep(self_ltxv, timestep, batch_size, hidden_dtype, **kwargs):
self_ltxv._funpack_identity_range = None
self_ltxv._funpack_identity_phase_value = 0.0
try:
entries = kwargs.get("resolved_guide_entries")
num_guide_tokens = kwargs.get("num_guide_tokens", 0)
if entries and num_guide_tokens:
offset = 0
for entry in entries:
phase = entry.get("funpack_identity_phase")
surviving = int(entry.get("surviving_count", 0))
if phase:
self_ltxv._funpack_identity_range = (offset, surviving, int(num_guide_tokens))
self_ltxv._funpack_identity_phase_value = float(phase)
break
offset += surviving
except Exception:
self_ltxv._funpack_identity_range = None
ref_len = getattr(self_ltxv, "_funpack_id_ref_len", 0)
if ref_len:
target_len = getattr(self_ltxv, "_funpack_id_target_len", None)
if timestep.dim() <= 1 and target_len is not None:
timestep = timestep.view(-1, 1).expand(batch_size, target_len).contiguous()
if timestep.dim() >= 2:
ref_ts = torch.zeros(batch_size, ref_len, *timestep.shape[2:],
device=timestep.device, dtype=timestep.dtype)
timestep = torch.cat([timestep, ref_ts], dim=1)
return orig_prepare_timestep(timestep, batch_size, hidden_dtype, **kwargs)
def prepare_pe(self_ltxv, pixel_coords, frame_rate, x_dtype):
pe = orig_prepare_pe(pixel_coords, frame_rate, x_dtype)
rng = getattr(self_ltxv, "_funpack_identity_range", None)
phase = getattr(self_ltxv, "_funpack_identity_phase_value", 0.0)
if not rng or not phase:
ref_len = getattr(self_ltxv, "_funpack_id_ref_len", 0)
if not ref_len or not seg_value:
return pe
try:
offset, length, num_guide_tokens = rng
# AV models return [(v_pe, av_cross_video), (a_pe, av_cross_audio)]; plain
# video models return v_pe directly. Only the video PE carries guide tokens.
# video models return v_pe directly. Only the video PE carries the ref tokens.
if isinstance(pe, list) and len(pe) and isinstance(pe[0], (list, tuple)) and isinstance(pe[0][0], (list, tuple)):
v_pe, cross_v = pe[0][0], pe[0][1]
v_pe = self._rotate_identity_phase(v_pe, offset, length, num_guide_tokens, phase)
v_pe = _rotate_overlap_freqs(v_pe, ref_len, seg_value)
return [(v_pe, cross_v), pe[1]]
return self._rotate_identity_phase(pe, offset, length, num_guide_tokens, phase)
return _rotate_overlap_freqs(pe, ref_len, seg_value)
except Exception:
return pe
def process_output(self_ltxv, x, embedded_timestep, keyframe_idxs, **kw):
ref_len = getattr(self_ltxv, "_funpack_id_ref_len", 0)
if ref_len:
try:
from comfy.ldm.lightricks.av_model import CompressedTimestep
import copy
if isinstance(x, (list, tuple)):
x = [x[0][:, :x[0].shape[1] - ref_len], *x[1:]]
et_list = list(embedded_timestep) if isinstance(embedded_timestep, (list, tuple)) else [embedded_timestep]
v_et = et_list[0]
if isinstance(v_et, CompressedTimestep):
ppf = max(1, getattr(v_et, "patches_per_frame", 1) or 1)
n_ref_frames = max(1, ref_len // ppf)
v_et2 = copy.copy(v_et)
v_et2.data = v_et.data[:, : v_et.num_frames - n_ref_frames].contiguous()
v_et2.num_frames = v_et.num_frames - n_ref_frames
et_list[0] = v_et2
elif hasattr(v_et, "shape") and v_et.dim() >= 2 and v_et.shape[1] > 1:
et_list[0] = v_et[:, : v_et.shape[1] - ref_len]
embedded_timestep = et_list
else:
x = x[:, :x.shape[1] - ref_len]
if hasattr(embedded_timestep, "shape") and embedded_timestep.dim() >= 2 and embedded_timestep.shape[1] > 1:
embedded_timestep = embedded_timestep[:, : embedded_timestep.shape[1] - ref_len]
except Exception:
pass
return orig_process_output(x, embedded_timestep, keyframe_idxs, **kw)
ltxv._process_input = types.MethodType(process_input, ltxv)
ltxv._prepare_timestep = types.MethodType(prepare_timestep, ltxv)
ltxv._prepare_positional_embeddings = types.MethodType(prepare_pe, ltxv)
setattr(ltxv, self._IDENTITY_PHASE_TAG, True)
return (ltxv, orig_prepare_timestep, orig_prepare_pe)
ltxv._process_output = types.MethodType(process_output, ltxv)
setattr(ltxv, self._IDENTITY_OVERLAP_TAG, True)
return (ltxv, orig_process_input, orig_prepare_timestep, orig_prepare_pe, orig_process_output)
def _strip_identity_phase(self, handle):
def _strip_identity_overlap(self, handle):
if not handle:
return
ltxv, orig_prepare_timestep, orig_prepare_pe = handle
ltxv, orig_process_input, orig_prepare_timestep, orig_prepare_pe, orig_process_output = handle
try:
ltxv._process_input = orig_process_input
ltxv._prepare_timestep = orig_prepare_timestep
ltxv._prepare_positional_embeddings = orig_prepare_pe
setattr(ltxv, self._IDENTITY_PHASE_TAG, False)
for attr in ("_funpack_identity_range", "_funpack_identity_phase_value"):
ltxv._process_output = orig_process_output
setattr(ltxv, self._IDENTITY_OVERLAP_TAG, False)
for attr in ("_funpack_id_ref_len", "_funpack_id_target_len"):
if hasattr(ltxv, attr):
delattr(ltxv, attr)
except Exception:
pass
def _rotate_identity_phase(self, pe, offset, length, num_guide_tokens, phase, theta=10000.0):
"""Givens-rotate the RoPE freqs of exactly [guide_start+offset, guide_start+offset+length)
(a single guide entry's tokens, identified upstream — never the whole guide-token tail)
by a phase derived from `phase` — the same source_phase tag the LoRA saw at train time.
pe = (cos, sin, split_mode) with cos/sin shaped [..., T, L]. Returns a new pe tuple."""
if length <= 0 or not phase:
return pe
cos, sin = pe[0], pe[1]
rest = tuple(pe[2:])
T = cos.shape[-2]
guide_start = T - int(num_guide_tokens)
start = guide_start + int(offset)
end = start + int(length)
if start < 0 or end > T or start >= end:
return pe
L = cos.shape[-1]
d = torch.arange(L, device=cos.device, dtype=torch.float32)
rate = theta ** (-d / float(L))
rot = (float(phase) * rate)
pc = rot.cos().to(cos.dtype); ps = rot.sin().to(sin.dtype)
idx = [slice(None)] * cos.dim()
idx[-2] = slice(start, end)
idx = tuple(idx)
c0, s0 = cos[idx], sin[idx]
cos = cos.clone(); sin = sin.clone()
cos[idx] = c0 * pc - s0 * ps
sin[idx] = s0 * pc + c0 * ps
return (cos, sin, *rest)
def _resolve_identity_overlap(self, state, filename, vae, chunk, identity_projector, source_id,
phase_scale, id_strength, arcface_mode, debug_log):
"""Lazily encode the reference face + (optional) ArcFace projector tokens once per
run() call and cache them in `state` (a dict owned by the caller). Cheap on every
later scene — just returns the cached tuple. Returns
(ref_latent_or_None, seg_value, pos_tokens_or_None, neg_tokens_or_None)."""
if state.get("ready"):
return state["ref_latent"], state["seg_value"], state["pos_tokens"], state["neg_tokens"]
state.update(ready=True, ref_latent=None, seg_value=float(source_id) * float(phase_scale),
pos_tokens=None, neg_tokens=None)
image = self._load_image_tensor(filename)
if image is None:
print(f"[FunPackSceneChain] identity_transfer: couldn't load reference image '{filename}' — skipped.")
return None, 0.0, None, None
chunk_tensors = self._latent_tensors(chunk)
if not chunk_tensors:
return None, 0.0, None, None
try:
import comfy.utils
scale_factors = getattr(vae, "downscale_index_formula", [8, 8, 8])
_, w_sf, h_sf = scale_factors
_, _, _, lat_h, lat_w = chunk_tensors[0].shape
ref_px = comfy.utils.common_upscale(
image.movedim(-1, 1), lat_w * w_sf, lat_h * h_sf, "bilinear", "center",
).movedim(1, -1)[:1, :, :, :3]
state["ref_latent"] = vae.encode(ref_px)
if debug_log:
print(f"[FunPackSceneChain] identity_transfer: overlap ref latent "
f"{list(state['ref_latent'].shape)}, seg={state['seg_value']}")
except Exception as e:
print(f"[FunPackSceneChain] identity_transfer: overlap ref-latent encode failed ({e}) — skipped.")
return None, 0.0, None, None
if identity_projector in (None, "", "None"):
return state["ref_latent"], state["seg_value"], None, None
try:
try:
from .identity_transfer import arcface_embed, load_identity_projector
except ImportError:
from identity_transfer import arcface_embed, load_identity_projector
emb = arcface_embed(image, mode=arcface_mode)
if emb is None:
print(f"[FunPackSceneChain] identity_transfer: ArcFace mode={arcface_mode} found no "
"face — projector tokens skipped (overlap only).")
return state["ref_latent"], state["seg_value"], None, None
import folder_paths
path = folder_paths.get_full_path("loras", identity_projector) or identity_projector
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
projector = load_identity_projector(path, device)
emb = emb.to(device=device, dtype=torch.float32).unsqueeze(0)
with torch.no_grad():
state["pos_tokens"] = projector(emb) * float(id_strength)
state["neg_tokens"] = projector(torch.zeros(1, projector.in_dim, device=device))
if debug_log:
print(f"[FunPackSceneChain] identity_transfer: ArcFace projector '{identity_projector}' "
f"OK, id_strength={id_strength}")
except Exception as e:
print(f"[FunPackSceneChain] identity_transfer: ArcFace/projector failed ({e}) — overlap tokens only "
"(insightface installed? `pip install insightface`).")
return state["ref_latent"], state["seg_value"], None, None
return state["ref_latent"], state["seg_value"], state["pos_tokens"], state["neg_tokens"]
def _append_identity_context_tokens(self, positive, negative, pos_tokens, neg_tokens):
try:
from .identity_transfer import append_context_tokens
except ImportError:
from identity_transfer import append_context_tokens
return append_context_tokens(positive, pos_tokens), append_context_tokens(negative, neg_tokens)
def _bounded_attention_region_mask(self, t, h, w, device):
"""[T*H*W] region id per video token: 0 = left half (by width), 1 = right half.
@@ -4359,7 +4632,11 @@ class FunPackLTXAVSceneChainSampler:
output_guidance=False, output_guidance_strength=0.02,
dynashift=False, dynashift_strength=0.3, dynashift_threshold=0.6,
alg_guide_blur_strength=2.0, alg_guide_blur_sigma_threshold=0.975,
identity_transfer_enabled=False, identity_transfer_phase=2.0,
identity_transfer_enabled=False, identity_projector="None", source_id=2.0,
phase_scale=1.0, id_strength=1.0, arcface_mode="auto_adjust", debug_log=False,
carry_overlap_through_anchor=False,
plateau_cache=False, plateau_cache_threshold=0.975,
taste_nearest_prompt=False,
unique_id=None, prompt=None):
if not isinstance(positive, list) or not positive:
raise ValueError("positive conditioning must contain at least one scene entry.")
@@ -4417,6 +4694,7 @@ class FunPackLTXAVSceneChainSampler:
output_guidance=output_guidance, output_guidance_strength=output_guidance_strength,
dynashift=dynashift, dynashift_strength=dynashift_strength,
dynashift_threshold=dynashift_threshold,
plateau_cache=plateau_cache, plateau_cache_threshold=plateau_cache_threshold,
)
max_scene_count = max(1, int(max_scenes))
@@ -4438,6 +4716,7 @@ class FunPackLTXAVSceneChainSampler:
_liked_dir = None
_bad_dir = None
_value_fn = None
_prompt_dir_index = None # per-prompt taste retrieval index (taste_nearest_prompt)
_eg_source = str(embed_guidance_source or "relative").lower()
_eg_key = self._absolute_key() if _eg_source == "absolute" else refinement_key_input
if (embed_guidance or score_slider) and _eg_key:
@@ -4445,6 +4724,14 @@ class FunPackLTXAVSceneChainSampler:
if _liked_dir is None:
print(f"[FunPackSceneChain] taste steering ({_eg_source}): no liked direction found (need 3+ liked generations)")
else:
if taste_nearest_prompt:
_prompt_dir_index = self._load_prompt_dir_index(_eg_key) or None
if _prompt_dir_index:
print(f"[FunPackSceneChain] taste_nearest_prompt: on — retrieving per-scene "
f"direction from {len(_prompt_dir_index)} rated prompt(s), global liked_dir as fallback")
else:
print("[FunPackSceneChain] taste_nearest_prompt: on but index empty — "
"using global liked_dir (rate a few liked gens to build the index)")
_value_fn = self._load_value_function(_eg_key)
if embed_guidance:
if _value_fn:
@@ -4537,6 +4824,7 @@ class FunPackLTXAVSceneChainSampler:
scene_media_by_ref = self._parse_scene_anchors(funpack_scene_media_refs)
scene_runs: list = []
joyai_bank = _JoyAIMemoryBank(joyai_memory_size, joyai_fix_frames) if joyai_memory else None
_identity_overlap_state: dict = {}
for scene_index, scene_cond in enumerate(scene_conditionings):
scene_positive = [scene_cond]
@@ -4551,7 +4839,7 @@ class FunPackLTXAVSceneChainSampler:
soft_carried = 0
guide_tail = 0
audio_tail = 0
identity_phase_applied = False
identity_ref_filename = None
run_mechanisms: list = []
anchor_meta = (scene_anchors or {}).get(str(scene_index))
if output is None:
@@ -4561,11 +4849,10 @@ class FunPackLTXAVSceneChainSampler:
custom_guides = per_scene_guides[scene_index]
if custom_guides:
run_mechanisms.append("custom_guide_stack")
chunk, scene_positive, scene_negative, carried, guide_tail, identity_phase_applied = self._apply_configured_guides(
chunk, scene_positive, scene_negative, carried, guide_tail, identity_ref_filename = self._apply_configured_guides(
chunk, scene_index, custom_guides, latent_template, scene_outputs, scene_media_by_ref,
scene_positive, scene_negative, vae,
identity_transfer_enabled=identity_transfer_enabled,
identity_transfer_phase=identity_transfer_phase,
)
carried_guide_frames = max(carried_guide_frames, carried)
elif anchor_meta:
@@ -4580,8 +4867,20 @@ class FunPackLTXAVSceneChainSampler:
"effect": effect if effect and transition_duration > 0 else None,
})
chunk = self._build_mixed_anchor_chunk(
vae, anchor_meta, latent_template, output, 0,
vae, anchor_meta, latent_template, output, video_overlap,
carry_overlap=carry_overlap_through_anchor,
)
if carry_overlap_through_anchor and video_overlap > 0:
run_mechanisms.append(f"latent_overlap_through_anchor({frame_overlap}px)")
# The anchor branch skips _apply_configured_guides entirely, so an identity_pin
# guide configured for this scene would otherwise never resolve — Best-FaceID
# identity_transfer needs this to fire on the exact scenes that swap anchors.
if per_scene_guides and scene_index < len(per_scene_guides):
identity_ref_filename = self._identity_pin_filename(
per_scene_guides[scene_index], scene_media_by_ref, identity_transfer_enabled,
)
if identity_ref_filename:
run_mechanisms.append("identity_pin_on_anchor_scene")
else:
# Record boundary before blending
effect = self._scene_transition_effect(scene_cond)
@@ -4605,11 +4904,10 @@ class FunPackLTXAVSceneChainSampler:
custom_guides = per_scene_guides[scene_index]
if custom_guides:
run_mechanisms.append("custom_guide_stack")
chunk, scene_positive, scene_negative, carried, guide_tail, identity_phase_applied = self._apply_configured_guides(
chunk, scene_positive, scene_negative, carried, guide_tail, identity_ref_filename = self._apply_configured_guides(
chunk, scene_index, custom_guides, latent_template, scene_outputs, scene_media_by_ref,
scene_positive, scene_negative, vae,
identity_transfer_enabled=identity_transfer_enabled,
identity_transfer_phase=identity_transfer_phase,
)
carried_guide_frames = max(carried_guide_frames, carried)
elif carry_i2v_guides:
@@ -4644,20 +4942,55 @@ class FunPackLTXAVSceneChainSampler:
# mode as the block-hook leak; see _strip_funpack_scene_wrappers).
_scene_base_wrapper = model.model_options.get("model_function_wrapper")
_v2a_handles = []
_identity_phase_handle = None
_identity_overlap_handle = None
_plateau_stats = None
try:
# Innermost wrapper (installed first): caches the raw base-model forward on the
# near-noise plateau so later plateau steps reuse it. All guidance wrappers below
# layer around it and still post-process each step's (cached-or-fresh) prediction.
if plateau_cache:
_plateau_stats = self._build_plateau_cache_wrapper(model, plateau_cache_threshold)
# Loop temporal style (auto director's funpack_temporal_loop): Mobius latent
# roll. Installed BELOW the guidance wrappers (embed guidance / slider /
# dynashift / output guidance) so they see canonical-orientation inputs and
# predictions — the roll exists only around the base forward. Calls that
# carry guides (keyframe_idxs pins appended frames to absolute positions)
# are left canonical by the wrapper itself, so on guide-carrying scenes the
# roll is inert rather than corrupting.
if self._scene_temporal_loop(scene_cond):
try:
from .ltx_enhancements import make_loop_temporal_wrapper
except ImportError:
from ltx_enhancements import make_loop_temporal_wrapper
_loop_base = model.model_options.get("model_function_wrapper")
model.model_options["model_function_wrapper"] = _tag_scene_wrapper(
make_loop_temporal_wrapper(_loop_base), _loop_base)
_loop_pinned = guide_tail > 0 or carried > 0 or carried_guide_frames > 0
run_mechanisms.append(
"temporal_loop_roll(inert: guides pin frames)" if _loop_pinned
else "temporal_loop_roll")
# taste_nearest_prompt: swap the single global liked direction for the one
# learned on this scene's closest rated prompts (resolved from the ORIGINAL
# scene conditioning, before any value-fn ascent mutates it). Falls back to
# the global _liked_dir when nothing rated is close enough.
_scene_liked_dir = _liked_dir
if _prompt_dir_index and _liked_dir is not None and scene_positive:
_retrieved = self._resolve_prompt_keyed_direction(_prompt_dir_index, scene_positive[0][0])
if _retrieved is not None:
_scene_liked_dir = _retrieved
run_mechanisms.append("taste_nearest_prompt")
if embed_guidance and _value_fn is not None and _value_fn.is_ready():
run_mechanisms.append("embed_guidance_vf_ascend")
orig_cond, orig_extra = scene_positive[0][0], scene_positive[0][1]
ascended = self._protect_audio(_value_fn.ascend(orig_cond), orig_cond)
scene_positive = [[ascended, orig_extra]] + list(scene_positive[1:])
if embed_guidance and _liked_dir is not None:
if embed_guidance and _scene_liked_dir is not None:
run_mechanisms.append(f"embed_guidance({_eg_source},{embed_guidance_strength})")
self._build_embed_guidance_wrapper(model, _liked_dir, embed_guidance_strength, value_fn=_value_fn)
if score_slider and _liked_dir is not None:
self._build_embed_guidance_wrapper(model, _scene_liked_dir, embed_guidance_strength, value_fn=_value_fn)
if score_slider and _scene_liked_dir is not None:
_pole = "contrastive" if _bad_dir is not None else "symmetric"
run_mechanisms.append(f"score_slider({_eg_source},{score_slider_strength},{_pole})")
self._build_score_slider_wrapper(model, _liked_dir, score_slider_strength, bad_dir=_bad_dir)
self._build_score_slider_wrapper(model, _scene_liked_dir, score_slider_strength, bad_dir=_bad_dir)
if dynashift and _dynashift_negatives:
run_mechanisms.append(
f"dynashift({len(_dynashift_negatives)}neg,{dynashift_strength},thr={dynashift_threshold})")
@@ -4696,10 +5029,20 @@ class FunPackLTXAVSceneChainSampler:
_v2a_handles = self._install_v2a_scale(model, v2a_grad_scale)
if _v2a_handles:
run_mechanisms.append(f"v2a_grad_scale({v2a_grad_scale})")
if identity_transfer_enabled and identity_phase_applied:
_identity_phase_handle = self._install_identity_phase(model)
if _identity_phase_handle:
run_mechanisms.append(f"identity_transfer_phase({identity_transfer_phase})")
if identity_transfer_enabled and identity_ref_filename:
_id_ref_latent, _id_seg_value, _id_pos_tokens, _id_neg_tokens = self._resolve_identity_overlap(
_identity_overlap_state, identity_ref_filename, vae, chunk,
identity_projector, source_id, phase_scale, id_strength, arcface_mode, debug_log,
)
if _id_ref_latent is not None:
_identity_overlap_handle = self._install_identity_overlap(model, _id_ref_latent, _id_seg_value)
if _identity_overlap_handle:
run_mechanisms.append(f"identity_transfer_overlap(seg={_id_seg_value})")
if _id_pos_tokens is not None:
scene_positive, scene_negative = self._append_identity_context_tokens(
scene_positive, scene_negative, _id_pos_tokens, _id_neg_tokens,
)
run_mechanisms.append(f"identity_transfer_arcface(id_strength={id_strength})")
_t_sample0 = _time.perf_counter()
sampled = self._sample_chunk(
model, sampler, sigmas, scene_seed, cfg, scene_positive, scene_negative, chunk,
@@ -4711,9 +5054,15 @@ class FunPackLTXAVSceneChainSampler:
)
_scene_sample_s = _time.perf_counter() - _t_sample0
_phase_sampling += _scene_sample_s
if _plateau_stats is not None:
_reused, _computed = _plateau_stats["reused"], _plateau_stats["computed"]
_total = _reused + _computed
if _total > 0:
run_mechanisms.append(
f"plateau_cache(skipped {_reused}/{_total} plateau fwd, thr={plateau_cache_threshold})")
finally:
self._remove_v2a_scale(_v2a_handles)
self._strip_identity_phase(_identity_phase_handle)
self._strip_identity_overlap(_identity_overlap_handle)
if model.model_options.get("model_function_wrapper") is not _scene_base_wrapper:
if _scene_base_wrapper is not None:
model.model_options["model_function_wrapper"] = _scene_base_wrapper
@@ -4733,6 +5082,11 @@ class FunPackLTXAVSceneChainSampler:
a_frame = self._harvest_joyai_audio(sampled, joyai_frame_select) if joyai_audio_memory else None
joyai_bank.add(v_frame, a_frame)
scene_outputs.append(self._clone_latent(sampled))
# Stays 0 for anchor scenes even with carry_overlap_through_anchor on: the post-sample
# slerp blend would reach into the anchor image's own leading frame (position 0 of
# `sampled`) and fade it against the previous scene's tail, undermining the hard cut.
# The carried frames beyond it are already seeded pre-sample (see
# _build_mixed_anchor_chunk), so no post-hoc smoothing is needed there.
blend_overlap = 0 if anchor_meta else video_overlap
output = sampled if output is None else self._blend_latents(output, sampled, blend_overlap)
cumulative_latent_frames = self._tensor_frames(self._latent_tensors(output)[0])
@@ -4858,6 +5212,7 @@ class FunPackLTXAVSceneChainSampler:
"transitions_enabled": scene_count > 1,
"joyai_memory": bool(joyai_memory),
"joyai_audio_memory": bool(joyai_audio_memory),
"carry_overlap_through_anchor": bool(carry_overlap_through_anchor),
})
except Exception as e:
print(f"[FunPackLTXAVSceneChainSampler] Failed to write sampler context: {e}")
@@ -4988,7 +5343,8 @@ class FunPackLTXAVSceneChainSampler:
alg_blur_guides=False, bounded_attention_enabled=False,
output_guidance=False, output_guidance_strength=0.02,
dynashift=False, dynashift_strength=0.3, dynashift_threshold=0.6,
alg_guide_blur_strength=2.0, alg_guide_blur_sigma_threshold=0.975):
alg_guide_blur_strength=2.0, alg_guide_blur_sigma_threshold=0.975,
plateau_cache=False, plateau_cache_threshold=0.975):
"""Sample one chain per Studio-packed variant entry (seed + index), persisting each result
(latent + preview + per-entry cond + manifest) under ComfyUI temp for rating in Studio.
Reuses sample() per entry with only the seed changed, so each entry is a clean generation."""
@@ -5041,6 +5397,7 @@ class FunPackLTXAVSceneChainSampler:
output_guidance=output_guidance, output_guidance_strength=output_guidance_strength,
dynashift=dynashift, dynashift_strength=dynashift_strength,
dynashift_threshold=dynashift_threshold,
plateau_cache=plateau_cache, plateau_cache_threshold=plateau_cache_threshold,
unique_id=None, prompt=None,
)
last = out
+239 -136
View File
@@ -1,15 +1,16 @@
"""Unit tests for the native Best-FaceID identity_transfer port.
"""Unit tests for the native Best-FaceID identity_transfer port (overlap tokens + ArcFace
projector), a port of ComfyUI-BFSNodes' LTX Identity Transfer.
Covers the three pieces added to FunPackLTXAVSceneChainSampler:
- _tag_last_guide_entry: best-effort marks the most-recently-appended
guide_attention_entry, never touching earlier entries.
- _apply_configured_guides: routes the phase tag ONLY to the guide entry the
editor marked identity_pin (never prior-scene/mid-scene/template guides that
happen to share the same call).
- _install_identity_phase / _strip_identity_phase / _rotate_identity_phase: the
tag+strip RoPE patch that reproduces the LoRA's trained-time source_phase
tag, scoped to the exact token range of the tagged guide (not the whole
guide-token tail, which may include co-active unrelated guides).
Covers:
- identity_transfer.rotate_overlap_freqs / append_context_tokens: pure-tensor pieces,
no torch model needed.
- FunPackLTXAVSceneChainSampler._apply_configured_guides: routes the identity_pin entry
to the new mechanism (returns its filename, skips the keyframe-blend append) only when
identity_transfer_enabled; unaffected guides append normally either way.
- _install_identity_overlap / _strip_identity_overlap: tag+strip idempotency, ref-token
append (_process_input), clean timestep for ref tokens (_prepare_timestep), source-phase
RoPE rotation (_prepare_positional_embeddings), and ref-token trim (_process_output).
- _resolve_identity_overlap: lazy, memoized once per run() call.
"""
import sys
import types
@@ -23,217 +24,319 @@ for _name in (
"comfy", "comfy.k_diffusion", "comfy.k_diffusion.sampling",
"comfy.model_sampling", "comfy.nested_tensor", "comfy.sample",
"comfy.samplers", "comfy.utils",
"comfy.ldm", "comfy.ldm.lightricks", "comfy.ldm.lightricks.model", "comfy.ldm.lightricks.av_model",
):
sys.modules.setdefault(_name, types.ModuleType(_name))
sys.modules["comfy.nested_tensor"].NestedTensor = object
sys.modules["comfy.ldm.lightricks.model"].latent_to_pixel_coords = (
lambda latent_coords, scale_factors, causal_fix: latent_coords)
class _CompressedTimestep:
def __init__(self, data, num_frames, patches_per_frame=1):
self.data = data
self.num_frames = num_frames
self.patches_per_frame = patches_per_frame
sys.modules["comfy.ldm.lightricks.av_model"].CompressedTimestep = _CompressedTimestep
import samplers # noqa: E402
import identity_transfer as idt # noqa: E402
def _sampler():
return samplers.FunPackLTXAVSceneChainSampler()
def _install_node_helpers():
"""node_helpers.conditioning_set_values, the same real implementation shape
_append_guide_attention_entry itself relies on (copy metadata dict, overwrite
the given keys) — reimplemented here so the test runs without a full ComfyUI env."""
mod = types.ModuleType("node_helpers")
# ── identity_transfer.rotate_overlap_freqs ─────────────────────────────────────
def conditioning_set_values(conditioning, values=None, append=False):
values = values or {}
out = []
for t in conditioning:
n = [t[0], t[1].copy()]
for k, v in values.items():
n[1][k] = v
out.append(n)
return out
mod.conditioning_set_values = conditioning_set_values
sys.modules["node_helpers"] = mod
return mod
def test_rotate_overlap_freqs_noop_for_zero_ref_len_or_seg():
cos = torch.arange(20, dtype=torch.float32).reshape(5, 4)
sin = -cos.clone()
pe = (cos, sin, False)
assert idt.rotate_overlap_freqs(pe, 0, 2.0) is pe
assert idt.rotate_overlap_freqs(pe, 2, 0.0) is pe
def _cond_with_entries(entries):
return [[torch.zeros(1, 1, 4), {"guide_attention_entries": list(entries)}]]
def test_rotate_overlap_freqs_rotates_only_last_ref_len_tokens():
T, L = 10, 4
cos = torch.arange(T * L, dtype=torch.float32).reshape(T, L)
sin = torch.arange(T * L, dtype=torch.float32).reshape(T, L) * -1.0
pe = (cos, sin, False)
ref_len = 3
out_cos, out_sin, split = idt.rotate_overlap_freqs(pe, ref_len, 2.0)
untouched = list(range(0, T - ref_len))
for i in untouched:
assert torch.equal(out_cos[i], cos[i])
assert torch.equal(out_sin[i], sin[i])
rotated = list(range(T - ref_len, T))
changed = any(not torch.equal(out_cos[i], cos[i]) for i in rotated)
assert changed
assert split is False
# ── _tag_last_guide_entry ──────────────────────────────────────────────────────
# ── identity_transfer.append_context_tokens ────────────────────────────────────
def test_tag_last_guide_entry_marks_only_newest_entry():
_install_node_helpers()
s = _sampler()
cond = _cond_with_entries([{"pre_filter_count": 4}, {"pre_filter_count": 8}])
tagged = s._tag_last_guide_entry(cond, "funpack_identity_phase", 2.0)
entries = tagged[0][1]["guide_attention_entries"]
assert entries[0].get("funpack_identity_phase") is None
assert entries[1]["funpack_identity_phase"] == 2.0
# Original untouched (conditioning_set_values copies, doesn't mutate in place).
assert "funpack_identity_phase" not in cond[0][1]["guide_attention_entries"][1]
def test_append_context_tokens_pads_and_extends_mask():
ce = torch.zeros(2, 5, 8) # batch=2, tokens=5, dim=8
cond = [[ce, {"attention_mask": torch.ones(2, 5)}]]
tokens = torch.ones(1, 3, 4) # batch=1 (broadcast), 3 new tokens, smaller dim
out = idt.append_context_tokens(cond, tokens)
new_ce, new_extra = out[0]
assert new_ce.shape == (2, 8, 8) # 5+3 tokens, dim padded back to 8
assert torch.equal(new_ce[:, 5:, :4], torch.ones(2, 3, 4))
assert torch.equal(new_ce[:, 5:, 4:], torch.zeros(2, 3, 4))
assert new_extra["attention_mask"].shape == (2, 8)
assert torch.equal(new_extra["attention_mask"][:, 5:], torch.ones(2, 3))
# Original conditioning list untouched.
assert ce.shape == (2, 5, 8)
def test_tag_last_guide_entry_noop_without_entries():
_install_node_helpers()
s = _sampler()
cond = [[torch.zeros(1, 1, 4), {}]]
out = s._tag_last_guide_entry(cond, "funpack_identity_phase", 2.0)
assert out is cond # unchanged object, nothing to tag
def test_append_context_tokens_truncates_wider_tokens():
ce = torch.zeros(1, 2, 4)
cond = [[ce, {}]]
tokens = torch.ones(1, 1, 8) # wider than context dim
out = idt.append_context_tokens(cond, tokens)
new_ce, _ = out[0]
assert new_ce.shape == (1, 3, 4)
def test_tag_last_guide_entry_noop_without_node_helpers():
sys.modules.pop("node_helpers", None)
s = _sampler()
cond = _cond_with_entries([{"pre_filter_count": 4}])
out = s._tag_last_guide_entry(cond, "funpack_identity_phase", 2.0)
assert out is cond
# ── _apply_configured_guides identity-pin routing ──────────────────────────────
# ── _apply_configured_guides phase-tag routing ─────────────────────────────────
def test_phase_tag_applies_only_to_identity_pin_entry(monkeypatch):
def test_identity_pin_routes_to_overlap_when_enabled(monkeypatch):
s = _sampler()
calls = []
def fake_append_media(chunk, filename, frame_idx, apply_at, strength, positive, negative, vae,
phase_tag=0.0):
calls.append((filename, phase_tag))
return chunk, positive, negative, 0, 0
monkeypatch.setattr(s, "_append_media_guide_at", fake_append_media)
monkeypatch.setattr(s, "_append_media_guide_at",
lambda chunk, filename, *a, **kw: (calls.append(filename), chunk, [], [], 0, 0)[1:])
guide_list = [
{"enabled": True, "source": "image", "media_ref": "pin", "identity_pin": True, "strength": 0.35},
{"enabled": True, "source": "image", "media_ref": "prior", "strength": 0.35}, # NOT the pin
]
scene_media_by_ref = {"pin": "pin.png", "prior": "prior.png"}
chunk, pos, neg, head, tail, phase_applied = s._apply_configured_guides(
chunk, pos, neg, head, tail, identity_ref_filename = s._apply_configured_guides(
{}, 1, guide_list, {}, [], scene_media_by_ref, [], [], object(),
identity_transfer_enabled=True, identity_transfer_phase=3.0,
identity_transfer_enabled=True,
)
assert calls == [("pin.png", 3.0), ("prior.png", 0.0)]
assert phase_applied is True
assert calls == ["prior.png"] # pin entry never went through the plain append
assert identity_ref_filename == "pin.png"
def test_phase_tag_off_when_identity_transfer_disabled(monkeypatch):
def test_identity_pin_appends_normally_when_disabled(monkeypatch):
s = _sampler()
calls = []
monkeypatch.setattr(s, "_append_media_guide_at",
lambda chunk, filename, *a, phase_tag=0.0, **kw: (calls.append(phase_tag), chunk, [], [], 0, 0)[1:])
lambda chunk, filename, *a, **kw: (calls.append(filename), chunk, [], [], 0, 0)[1:])
guide_list = [{"enabled": True, "source": "image", "media_ref": "pin", "identity_pin": True, "strength": 0.35}]
_, _, _, _, _, phase_applied = s._apply_configured_guides(
_, _, _, _, _, identity_ref_filename = s._apply_configured_guides(
{}, 1, guide_list, {}, [], {"pin": "pin.png"}, [], [], object(),
identity_transfer_enabled=False, identity_transfer_phase=2.0,
identity_transfer_enabled=False,
)
assert calls == [0.0]
assert phase_applied is False
assert calls == ["pin.png"]
assert identity_ref_filename is None
# ── _install_identity_phase / _strip_identity_phase / _rotate_identity_phase ──
# ── _identity_pin_filename (standalone lookup for the mixed i2v anchor branch,
# which skips _apply_configured_guides entirely) ─────────────────────────────
def test_identity_pin_filename_finds_first_pin_entry():
s = _sampler()
guide_list = [
{"enabled": True, "source": "image", "media_ref": "prior", "strength": 0.35},
{"enabled": True, "source": "image", "media_ref": "pin", "identity_pin": True, "strength": 0.35},
]
scene_media_by_ref = {"pin": "pin.png", "prior": "prior.png"}
assert s._identity_pin_filename(guide_list, scene_media_by_ref, True) == "pin.png"
def test_identity_pin_filename_none_when_disabled_or_missing():
s = _sampler()
guide_list = [{"enabled": True, "source": "image", "media_ref": "pin", "identity_pin": True}]
assert s._identity_pin_filename(guide_list, {"pin": "pin.png"}, False) is None
assert s._identity_pin_filename(guide_list, {}, True) is None
assert s._identity_pin_filename(None, {"pin": "pin.png"}, True) is None
# ── _install_identity_overlap / _strip_identity_overlap ────────────────────────
class _FakePatchifier:
def patchify(self, latent):
# latent: [1, C, 1, H, W] -> pretend 2 ref tokens: rt=[B,N,C], rlc=[B,3,N] (coords).
b = latent.shape[0]
return torch.zeros(b, 2, 8), torch.zeros(b, 3, 2)
class _FakeLTXV:
def __init__(self):
self.patchifier = _FakePatchifier()
self.vae_scale_factors = [8, 8, 8]
self.causal_temporal_positioning = False
def _process_input(self, x, keyframe_idxs, denoise_mask, **kw):
# pixel_coords mirrors x's structure: [B, 3, T] (or a [video, audio] pair for AV).
if isinstance(x, (list, tuple)):
pix = [torch.zeros(t.shape[0], 3, t.shape[1]) for t in x]
else:
pix = torch.zeros(x.shape[0], 3, x.shape[1])
return x, pix, {}
def patchify_proj(self, rt):
return rt # identity, dim already matches x for the test
def _prepare_timestep(self, timestep, batch_size, hidden_dtype, **kwargs):
return ("timestep", batch_size, hidden_dtype)
return ("timestep", timestep, batch_size, hidden_dtype)
def _prepare_positional_embeddings(self, pixel_coords, frame_rate, x_dtype):
# cos/sin shaped [T=10, L=4]; identity token slice under test is [6:9).
T, L = 10, 4
cos = torch.arange(T * L, dtype=torch.float32).reshape(T, L)
sin = torch.arange(T * L, dtype=torch.float32).reshape(T, L) * -1.0
return (cos, sin, False)
def _process_output(self, x, embedded_timestep, keyframe_idxs, **kw):
return ("output", x, embedded_timestep)
def _fake_model(ltxv):
inner = types.SimpleNamespace(diffusion_model=ltxv)
return types.SimpleNamespace(model=inner)
def _ref_latent():
return torch.zeros(1, 8, 1, 4, 4)
def test_install_is_idempotent_and_strip_restores_originals():
s = _sampler()
ltxv = _FakeLTXV()
orig_pi = ltxv._process_input
orig_pt = ltxv._prepare_timestep
orig_pe = ltxv._prepare_positional_embeddings
orig_po = ltxv._process_output
handle1 = s._install_identity_phase(_fake_model(ltxv))
handle1 = s._install_identity_overlap(_fake_model(ltxv), _ref_latent(), 2.0)
assert handle1 is not None
assert ltxv._prepare_timestep is not orig_pt
handle2 = s._install_identity_phase(_fake_model(ltxv))
assert ltxv._process_input is not orig_pi
handle2 = s._install_identity_overlap(_fake_model(ltxv), _ref_latent(), 2.0)
assert handle2 is None # already installed — idempotent, no double-wrap
s._strip_identity_phase(handle1)
# Bound-method objects aren't identity-stable across separate attribute accesses in
# CPython (a fresh wrapper is created each time) — compare by equality (same __func__ +
# __self__) instead of `is`.
s._strip_identity_overlap(handle1)
assert ltxv._process_input == orig_pi
assert ltxv._prepare_timestep == orig_pt
assert ltxv._prepare_positional_embeddings == orig_pe
assert not hasattr(ltxv, "_funpack_identity_range")
assert ltxv._process_output == orig_po
assert not hasattr(ltxv, "_funpack_id_ref_len")
def test_prepare_timestep_stashes_range_for_tagged_entry_only():
def test_process_input_appends_ref_tokens_and_stashes_len():
s = _sampler()
ltxv = _FakeLTXV()
handle = s._install_identity_phase(_fake_model(ltxv))
entries = [
{"pre_filter_count": 16, "surviving_count": 2}, # unrelated guide, first
{"pre_filter_count": 16, "surviving_count": 3, "funpack_identity_phase": 2.5}, # the tagged one
{"pre_filter_count": 16, "surviving_count": 4}, # unrelated guide, after
]
ltxv._prepare_timestep(torch.zeros(1), 1, torch.float32,
resolved_guide_entries=entries, num_guide_tokens=9)
assert ltxv._funpack_identity_range == (2, 3, 9) # offset past the first entry's 2 tokens
assert ltxv._funpack_identity_phase_value == 2.5
s._strip_identity_phase(handle)
handle = s._install_identity_overlap(_fake_model(ltxv), _ref_latent(), 2.0)
x = torch.zeros(1, 5, 8) # 5 target tokens
xx, pix, add = ltxv._process_input(x, None, None)
assert xx.shape[1] == 7 # 5 target + 2 ref tokens
assert ltxv._funpack_id_ref_len == 2
assert ltxv._funpack_id_target_len == 5
s._strip_identity_overlap(handle)
def test_prepare_timestep_clears_range_without_tagged_entry():
def test_prepare_timestep_gives_ref_tokens_clean_timestep():
s = _sampler()
ltxv = _FakeLTXV()
handle = s._install_identity_phase(_fake_model(ltxv))
ltxv._funpack_identity_range = (0, 1, 1) # stale from a previous scene
entries = [{"pre_filter_count": 16, "surviving_count": 5}]
ltxv._prepare_timestep(torch.zeros(1), 1, torch.float32,
resolved_guide_entries=entries, num_guide_tokens=5)
assert ltxv._funpack_identity_range is None
s._strip_identity_phase(handle)
handle = s._install_identity_overlap(_fake_model(ltxv), _ref_latent(), 2.0)
ltxv._process_input(torch.zeros(1, 5, 8), None, None) # sets ref_len=2, target_len=5
_, timestep, batch_size, _ = ltxv._prepare_timestep(torch.full((1,), 0.7), 1, torch.float32)
assert timestep.shape == (1, 7)
assert torch.all(timestep[:, :5] == 0.7)
assert torch.all(timestep[:, 5:] == 0.0) # ref tokens: clean timestep
s._strip_identity_overlap(handle)
def test_prepare_pe_rotates_only_the_tagged_range():
def test_prepare_pe_rotates_only_the_ref_token_tail():
s = _sampler()
ltxv = _FakeLTXV()
handle = s._install_identity_phase(_fake_model(ltxv))
handle = s._install_identity_overlap(_fake_model(ltxv), _ref_latent(), 2.0)
baseline_cos, baseline_sin, _ = ltxv._prepare_positional_embeddings(None, 25, torch.float32)
entries = [
{"pre_filter_count": 16, "surviving_count": 2},
{"pre_filter_count": 16, "surviving_count": 3, "funpack_identity_phase": 2.5},
{"pre_filter_count": 16, "surviving_count": 4},
]
ltxv._prepare_timestep(torch.zeros(1), 1, torch.float32,
resolved_guide_entries=entries, num_guide_tokens=9)
ltxv._process_input(torch.zeros(1, 5, 8), None, None) # ref_len=2
cos, sin, split_mode = ltxv._prepare_positional_embeddings(None, 25, torch.float32)
# T=10, num_guide_tokens=9 -> guide_start=1; offset=2, length=3 -> rotated slice [3:6).
untouched = list(range(0, 3)) + list(range(6, 10))
untouched = list(range(0, 8))
for i in untouched:
assert torch.equal(cos[i], baseline_cos[i]), f"row {i} should be untouched"
assert torch.equal(sin[i], baseline_sin[i]), f"row {i} should be untouched"
rotated_rows = [3, 4, 5]
changed = any(not torch.equal(cos[i], baseline_cos[i]) for i in rotated_rows)
assert changed, "rotated slice should differ from baseline"
assert split_mode is False # passthrough element preserved
s._strip_identity_phase(handle)
assert torch.equal(cos[i], baseline_cos[i])
assert torch.equal(sin[i], baseline_sin[i])
rotated = [8, 9]
changed = any(not torch.equal(cos[i], baseline_cos[i]) for i in rotated)
assert changed
assert split_mode is False
s._strip_identity_overlap(handle)
def test_rotate_identity_phase_is_noop_for_zero_phase_or_length():
def test_prepare_pe_noop_when_ref_len_zero():
s = _sampler()
pe = (torch.ones(5, 4), torch.zeros(5, 4), False)
assert s._rotate_identity_phase(pe, 0, 0, 5, 2.0) is pe
assert s._rotate_identity_phase(pe, 0, 2, 5, 0.0) is pe
ltxv = _FakeLTXV()
handle = s._install_identity_overlap(_fake_model(ltxv), _ref_latent(), 2.0)
baseline = ltxv._prepare_positional_embeddings(None, 25, torch.float32)
# No _process_input call this "forward pass" -> _funpack_id_ref_len defaults to 0.
out = ltxv._prepare_positional_embeddings(None, 25, torch.float32)
assert torch.equal(out[0], baseline[0])
assert torch.equal(out[1], baseline[1])
s._strip_identity_overlap(handle)
def test_rotate_identity_phase_rejects_out_of_bounds_range():
def test_process_output_trims_ref_tokens_from_plain_tensor():
s = _sampler()
cos = torch.arange(20, dtype=torch.float32).reshape(5, 4)
sin = -cos.clone()
pe = (cos, sin, False)
out = s._rotate_identity_phase(pe, offset=10, length=3, num_guide_tokens=5, phase=2.0)
assert out is pe # start/end fall outside T -> safe no-op
ltxv = _FakeLTXV()
handle = s._install_identity_overlap(_fake_model(ltxv), _ref_latent(), 2.0)
ltxv._process_input(torch.zeros(1, 5, 8), None, None) # ref_len=2
x = torch.arange(7 * 3, dtype=torch.float32).reshape(1, 7, 3)
embedded_timestep = torch.zeros(1, 7, 1)
tag, out_x, out_et = ltxv._process_output(x, embedded_timestep, None)
assert out_x.shape[1] == 5
assert torch.equal(out_x, x[:, :5])
assert out_et.shape[1] == 5
s._strip_identity_overlap(handle)
def test_process_output_trims_compressed_timestep_frames():
s = _sampler()
ltxv = _FakeLTXV()
handle = s._install_identity_overlap(_fake_model(ltxv), _ref_latent(), 2.0)
ltxv._process_input([torch.zeros(1, 5, 8), torch.zeros(1, 3, 8)], None, None)
x = [torch.arange(7 * 3, dtype=torch.float32).reshape(1, 7, 3), torch.zeros(1, 3, 3)]
ct = _CompressedTimestep(data=torch.zeros(1, 7), num_frames=7, patches_per_frame=1)
tag, out_x, out_et = ltxv._process_output(x, [ct, "audio_et"], None)
assert out_x[0].shape[1] == 5
assert out_et[0].num_frames == 5
assert out_et[0].data.shape[1] == 5
assert out_et[1] == "audio_et"
s._strip_identity_overlap(handle)
# ── _resolve_identity_overlap memoization ──────────────────────────────────────
def test_resolve_identity_overlap_computes_once_per_state(monkeypatch):
s = _sampler()
monkeypatch.setattr(s, "_load_image_tensor", lambda filename: torch.zeros(1, 32, 32, 3))
monkeypatch.setattr(s, "_latent_tensors", lambda chunk: [torch.zeros(1, 8, 1, 4, 4)])
fake_comfy_utils = types.ModuleType("comfy.utils")
calls = []
def fake_upscale(img, w, h, mode, crop):
calls.append((w, h))
return img
fake_comfy_utils.common_upscale = fake_upscale
sys.modules["comfy.utils"] = fake_comfy_utils
sys.modules["comfy"].utils = fake_comfy_utils
class _FakeVAE:
downscale_index_formula = [8, 8, 8]
def encode(self, px):
return torch.zeros(1, 8, 1, 4, 4)
state = {}
r1 = s._resolve_identity_overlap(state, "pin.png", _FakeVAE(), {}, "None", 2.0, 1.0, 1.0, "auto_adjust", False)
assert len(calls) == 1
r2 = s._resolve_identity_overlap(state, "pin.png", _FakeVAE(), {}, "None", 2.0, 1.0, 1.0, "auto_adjust", False)
assert len(calls) == 1 # second call served from cache, no re-encode
assert r1 == r2
assert r1[2] is None and r1[3] is None # no projector selected -> no ArcFace tokens
+165
View File
@@ -0,0 +1,165 @@
"""Unit tests for the loop temporal style (Mobius latent roll, arXiv:2502.20307).
Covers: per-stream temporal roll of the packed [B,1,N] latent (video and audio each by
their own frame count), exact unroll round-trips, denoise-mask rolls staying in step,
the eligibility gates (near-noise plateau, guide-pinned calls, tiny clips, unreadable
packed layouts), van der Corput shift spreading with per-scene reset, and wrapper
chaining semantics (inner wrappers see rolled args, callers always get canonical
predictions back).
"""
import math
import sys
from pathlib import Path
import torch
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from ltx_enhancements import ( # noqa: E402
_loop_roll_mask,
_loop_roll_packed,
_loop_stream_shapes,
_van_der_corput,
make_loop_temporal_wrapper,
)
VSHAPE, ASHAPE = (1, 128, 16, 4, 6), (1, 8, 32, 16)
def _packed():
video = torch.arange(math.prod(VSHAPE), dtype=torch.float32).reshape(VSHAPE)
audio = torch.arange(math.prod(ASHAPE), dtype=torch.float32).reshape(ASHAPE) + 1e6
packed = torch.cat([video.reshape(1, 1, -1), audio.reshape(1, 1, -1)], dim=-1)
return video, audio, packed
def _args(sigma, packed, mask=None, extra_c=None):
c = {"latent_shapes": [torch.Size(VSHAPE), torch.Size(ASHAPE)]}
if mask is not None:
c["denoise_mask"] = mask
if extra_c:
c.update(extra_c)
return {"input": packed, "timestep": torch.tensor([sigma]), "c": c}
def test_van_der_corput_prefix():
assert [_van_der_corput(n) for n in range(1, 6)] == [0.5, 0.25, 0.75, 0.125, 0.625]
def test_roll_per_stream_and_round_trip():
video, audio, packed = _packed()
rolled = _loop_roll_packed(packed, [VSHAPE, ASHAPE], 0.5, 1)
assert torch.equal(rolled[..., : video.numel()].reshape(VSHAPE), torch.roll(video, 8, dims=2))
assert torch.equal(rolled[..., video.numel():].reshape(ASHAPE), torch.roll(audio, 16, dims=2))
assert torch.equal(_loop_roll_packed(rolled, [VSHAPE, ASHAPE], 0.5, -1), packed)
def test_roll_fraction_keeps_streams_time_aligned():
video, audio, packed = _packed()
rolled = _loop_roll_packed(packed, [VSHAPE, ASHAPE], 0.25, 1)
assert torch.equal(rolled[..., : video.numel()].reshape(VSHAPE), torch.roll(video, 4, dims=2))
assert torch.equal(rolled[..., video.numel():].reshape(ASHAPE), torch.roll(audio, 8, dims=2))
def test_mask_roll_round_trip():
mask = torch.rand(1, 1, 16, 4, 6)
rolled = _loop_roll_mask(mask, 0.5, 1)
assert torch.equal(rolled, torch.roll(mask, 8, dims=2))
assert torch.equal(_loop_roll_mask(rolled, 0.5, -1), mask)
def test_stream_shapes_refuses_mismatched_packing():
_, _, packed = _packed()
ok = _loop_stream_shapes(_args(0.7, packed))
assert ok == [VSHAPE, ASHAPE]
assert _loop_stream_shapes(_args(0.7, packed[..., :-5])) is None
def test_wrapper_rolls_in_unrolls_out():
video, _, packed = _packed()
mask = torch.rand(1, 1, 16, 4, 6)
seen = {}
def fake_apply(x, ts, **c):
seen["input"], seen["mask"] = x, c.get("denoise_mask")
return x * 1.0
w = make_loop_temporal_wrapper(None)
out = w(fake_apply, _args(0.909, packed, mask))
assert torch.equal(seen["input"][..., : video.numel()].reshape(VSHAPE), torch.roll(video, 8, dims=2))
assert torch.equal(seen["mask"], torch.roll(mask, 8, dims=2))
assert torch.equal(out, packed) # canonical on the way out
# second eligible step advances the shift sequence (0.25 -> 4 frames)
w(fake_apply, _args(0.725, packed, mask))
assert torch.equal(seen["input"][..., : video.numel()].reshape(VSHAPE), torch.roll(video, 4, dims=2))
def test_wrapper_gates_plateau_guides_and_tiny_clips():
video, _, packed = _packed()
seen = {}
def fake_apply(x, ts, **c):
seen["input"] = x
return x * 1.0
w = make_loop_temporal_wrapper(None)
w(fake_apply, _args(0.99, packed)) # near-noise plateau
assert torch.equal(seen["input"], packed)
w(fake_apply, _args(0.725, packed, extra_c={"keyframe_idxs": torch.zeros(1, 1)}))
assert torch.equal(seen["input"], packed)
w(fake_apply, _args(0.725, packed, extra_c={"guide_attention_entries": [object()]}))
assert torch.equal(seen["input"], packed)
tiny_shape = (1, 128, 2, 4, 6)
tiny = torch.rand(1, 1, math.prod(tiny_shape[1:]))
w2 = make_loop_temporal_wrapper(None)
w2(fake_apply, {"input": tiny, "timestep": torch.tensor([0.7]),
"c": {"latent_shapes": [torch.Size(tiny_shape)]}})
assert torch.equal(seen["input"], tiny)
def test_wrapper_resets_shift_sequence_on_new_scene():
video, _, packed = _packed()
seen = {}
def fake_apply(x, ts, **c):
seen["input"] = x
return x * 1.0
w = make_loop_temporal_wrapper(None)
w(fake_apply, _args(0.909, packed)) # frac 0.5
w(fake_apply, _args(0.725, packed)) # frac 0.25
w(fake_apply, _args(0.99, packed)) # sigma jumps back up: new scene
w(fake_apply, _args(0.909, packed)) # sequence restarts at 0.5
assert torch.equal(seen["input"][..., : video.numel()].reshape(VSHAPE), torch.roll(video, 8, dims=2))
def test_wrapper_unpacked_single_stream():
seen = {}
def fake_apply(x, ts, **c):
seen["input"] = x
return x * 1.0
five_d = torch.arange(math.prod(VSHAPE), dtype=torch.float32).reshape(VSHAPE)
w = make_loop_temporal_wrapper(None)
out = w(fake_apply, {"input": five_d, "timestep": torch.tensor([0.7]), "c": {}})
assert torch.equal(seen["input"], torch.roll(five_d, 8, dims=2))
assert torch.equal(out, five_d)
def test_wrapper_chain_inner_sees_rolled_caller_sees_canonical():
_, _, packed = _packed()
seen = {}
def fake_apply(x, ts, **c):
return x * 1.0
def inner(apply_fn, args):
seen["inner_input"] = args["input"]
return apply_fn(args["input"], args["timestep"], **args.get("c", {}))
w = make_loop_temporal_wrapper(inner)
out = w(fake_apply, _args(0.725, packed))
assert not torch.equal(seen["inner_input"], packed)
assert torch.equal(out, packed)
+212 -31
View File
@@ -117,6 +117,20 @@ from samplers import FunPackLTXAVSceneChainSampler
class FakeVAE:
downscale_index_formula = (1, 1, 1)
def decode(self, samples):
b, _c, t, _h, _w = samples.shape
return torch.zeros(b, t, 8, 8, 3)
class FakeModel:
"""Minimal stand-in with the one attribute the per-scene wrapper snapshot/restore
(samplers.py, around _scene_base_wrapper) needs. Plain object() lacks model_options
entirely, which pre-dates this feature (every sample()-calling test in this file
currently fails on that AttributeError — a known stale-mock gap, not something this
change introduces or fixes file-wide)."""
def __init__(self):
self.model_options = {}
def scene_cond(index):
return (
@@ -132,8 +146,8 @@ def test_scene_chain_detects_scene_count_and_increments_seed():
positive = [scene_cond(0), scene_cond(1), scene_cond(2)]
negative = [(torch.zeros(1, 2, 3), {})]
latent, status, scene_count, report = node.sample(
model=object(),
latent, _images, status, scene_count, report, _boundaries = node.sample(
model=FakeModel(),
vae=FakeVAE(),
positive=positive,
negative=negative,
@@ -165,8 +179,8 @@ def test_scene_chain_accepts_manual_combined_conditioning_without_metadata():
(torch.ones(1, 2, 3) * 3.0, {}),
]
_, status, scene_count, report = node.sample(
model=object(),
_, _images, status, scene_count, report, _boundaries = node.sample(
model=FakeModel(),
vae=FakeVAE(),
positive=positive,
negative=[],
@@ -184,8 +198,9 @@ def test_scene_chain_accepts_manual_combined_conditioning_without_metadata():
assert [call["seed"] for call in sample_calls] == [70, 71, 72]
assert [call["positive"][0][0].mean().item() for call in sample_calls] == [1.0, 2.0, 3.0]
assert "3 scene(s)" in status
assert "Scene 1: seed=70, text=Scene 1" in report
assert "Scene 3: seed=72, text=Scene 3" in report
# Per-scene report lines now include a "sampling {s}s" timing segment between seed and text.
assert "Scene 1: seed=70," in report and "text=Scene 1" in report
assert "Scene 3: seed=72," in report and "text=Scene 3" in report
def test_scene_chain_uses_scene_seed_metadata():
@@ -198,8 +213,8 @@ def test_scene_chain_uses_scene_seed_metadata():
(torch.ones(1, 2, 3), {"funpack_scene_text": "scene c", "funpack_scene_seed": 303}),
]
_, _, scene_count, report = node.sample(
model=object(),
_, _images, _status, scene_count, report, _boundaries = node.sample(
model=FakeModel(),
vae=FakeVAE(),
positive=positive,
negative=[],
@@ -228,8 +243,8 @@ def test_scene_chain_use_same_seed_forces_first_seed():
(torch.ones(1, 2, 3), {"funpack_scene_text": "scene c"}),
]
_, _, scene_count, _ = node.sample(
model=object(),
_, _images, _status, scene_count, _report, _boundaries = node.sample(
model=FakeModel(),
vae=FakeVAE(),
positive=positive,
negative=[],
@@ -256,8 +271,8 @@ def test_scene_chain_preserves_nested_av_structure_and_audio_length():
latent_template = {"samples": FakeNestedTensor([video, audio])}
positive = [scene_cond(0), scene_cond(1)]
latent, _, scene_count, _ = node.sample(
model=object(),
latent, _images, _status, scene_count, _report, _boundaries = node.sample(
model=FakeModel(),
vae=FakeVAE(),
positive=positive,
negative=[],
@@ -287,8 +302,8 @@ def test_scene_chain_default_max_is_eight_but_allows_more():
latent_template = {"samples": torch.zeros(1, 2, 3, 2, 2)}
positive = [scene_cond(index) for index in range(10)]
latent, status, scene_count, report = node.sample(
model=object(),
latent, _images, status, scene_count, report, _boundaries = node.sample(
model=FakeModel(),
vae=FakeVAE(),
positive=positive,
negative=[],
@@ -320,8 +335,8 @@ def test_scene_chain_can_append_i2v_template_as_hidden_guide():
latent_template = {"samples": samples, "noise_mask": mask}
positive = [scene_cond(0), scene_cond(1)]
latent, status, scene_count, _ = node.sample(
model=object(),
latent, _images, status, scene_count, _report, _boundaries = node.sample(
model=FakeModel(),
vae=FakeVAE(),
positive=positive,
negative=[],
@@ -336,16 +351,17 @@ def test_scene_chain_can_append_i2v_template_as_hidden_guide():
carry_i2v_guides=True,
)
# _append_i2v_guides prepends the guide frame (temporal pos 0) so it never collides with
# the overlap frames that follow: [guide, overlap0, overlap1, mid, mid, mid].
second_call = sample_calls[1]
assert scene_count == 2
assert second_call["latent_image"].shape[2] == 6
assert torch.all(second_call["latent_image"][:, :, 2:5] == 0.0)
assert torch.all(second_call["latent_image"][:, :, 5] == 7.0)
assert torch.all(second_call["noise_mask"][:, :, :2] == 0.0)
assert torch.all(second_call["noise_mask"][:, :, 2:5] == 1.0)
assert torch.all(second_call["noise_mask"][:, :, 5] == 0.0)
assert "keyframe_idxs" in second_call["positive"][0][1]
assert second_call["positive"][0][1]["guide_attention_entries"][0]["pre_filter_count"] == 1
assert torch.all(second_call["latent_image"][:, :, 0] == 7.0)
assert torch.all(second_call["latent_image"][:, :, 3:6] == 0.0)
assert torch.all(second_call["noise_mask"][:, :, :3] == 0.0)
assert torch.all(second_call["noise_mask"][:, :, 3:6] == 1.0)
# _append_i2v_guides is a plain protected-frame append (tensor + mask only) — unlike
# mid_scene_guide/joyai memory it does not add keyframe_idxs/guide_attention_entries.
assert latent["samples"].shape[2] == 8
assert "i2v guide tokens=1 latent frame(s)" in status
@@ -360,8 +376,8 @@ def test_scene_chain_expands_compact_i2v_guide_mask_to_spatial_chunk_mask():
latent_template = {"samples": samples, "noise_mask": mask}
positive = [scene_cond(0), scene_cond(1)]
_, status, scene_count, _ = node.sample(
model=object(),
_, _images, status, scene_count, _report, _boundaries = node.sample(
model=FakeModel(),
vae=FakeVAE(),
positive=positive,
negative=[],
@@ -376,10 +392,11 @@ def test_scene_chain_expands_compact_i2v_guide_mask_to_spatial_chunk_mask():
carry_i2v_guides=True,
)
# Guide frame is prepended (temporal pos 0) — see test_scene_chain_can_append_i2v_template_as_hidden_guide.
second_call = sample_calls[1]
assert scene_count == 2
assert second_call["noise_mask"].shape == second_call["latent_image"].shape
assert torch.all(second_call["noise_mask"][:, :, 5] == 0.0)
assert torch.all(second_call["noise_mask"][:, :, 0] == 0.0)
assert "i2v guide tokens=1 latent frame(s)" in status
@@ -396,8 +413,8 @@ def test_scene_chain_does_not_carry_i2v_guides_by_default():
latent_template = {"samples": samples, "noise_mask": mask}
positive = [scene_cond(0), scene_cond(1)]
_, status, scene_count, _ = node.sample(
model=object(),
_, _images, status, scene_count, _report, _boundaries = node.sample(
model=FakeModel(),
vae=FakeVAE(),
positive=positive,
negative=[],
@@ -459,7 +476,7 @@ def test_overlap_diagnostics_report_latent_blend_zone():
positive = [scene_cond(0), scene_cond(1)]
latent_template = {"samples": torch.zeros(1, 2, 5, 1, 1)}
_, _, status, _, _, boundaries_json = node.sample(
model=object(),
model=FakeModel(),
vae=FakeVAE(),
positive=positive,
negative=[],
@@ -496,7 +513,7 @@ def test_mixed_solo_applies_guides_on_first_chunk():
positive = [scene_cond(0)]
latent, _images, status, scene_count, _report, boundaries_json = node.sample(
model=object(),
model=FakeModel(),
vae=FakeVAE(),
positive=positive,
negative=[],
@@ -534,7 +551,7 @@ def test_mixed_anchor_skips_frame_overlap():
anchors = json.dumps({"1": {"filename": "anchor.png", "strength": 1.0}})
latent, _images, status, scene_count, _report, boundaries_json = node.sample(
model=object(),
model=FakeModel(),
vae=FakeVAE(),
positive=positive,
negative=[],
@@ -557,3 +574,167 @@ def test_mixed_anchor_skips_frame_overlap():
mechs = parsed["scenes"][1]["mechanisms"]
assert "mixed_i2v_anchor" in mechs
assert not any("latent_overlap" in m for m in mechs)
def test_mixed_anchor_carries_overlap_when_enabled():
"""carry_overlap_through_anchor=True: the chunk fed to scene 1's sampler call is seeded
with scene 0's tail (frame_overlap latent frames) instead of a bare template, even though
scene 1 has its own i2v anchor. The mocked _apply_img2video_to_video_latent is an identity
passthrough here, so whatever _build_mixed_anchor_chunk hands it is exactly what gets
sampled — letting us assert on the carried values and protected mask directly."""
import json
sample_calls.clear()
node = FunPackLTXAVSceneChainSampler()
node._load_image_tensor = lambda _fn: torch.ones(1, 8, 8, 3)
node._apply_img2video_to_video_latent = lambda _vae, _img, chunk, _strength: node._clone_latent(chunk)
positive = [scene_cond(0), scene_cond(1)]
latent_template = {"samples": torch.zeros(1, 2, 5, 3, 3)}
anchors = json.dumps({"1": {"filename": "anchor.png", "strength": 1.0}})
latent, _images, status, scene_count, _report, boundaries_json = node.sample(
model=FakeModel(),
vae=FakeVAE(),
positive=positive,
negative=[],
sampler=object(),
sigmas=torch.tensor([1.0, 0.0]),
seed=80,
latent_template=latent_template,
num_frames_per_scene=5,
frame_overlap=2,
cfg=1.0,
max_scenes=2,
funpack_scene_anchors=anchors,
carry_overlap_through_anchor=True,
prompt={"n1": {"inputs": {}}},
unique_id="test-node",
)
assert scene_count == 2
assert latent["samples"].shape[2] == 10
parsed = json.loads(boundaries_json)
mechs = parsed["scenes"][1]["mechanisms"]
assert "mixed_i2v_anchor" in mechs
assert "latent_overlap_through_anchor(2px)" in mechs
# Scene 0 (no anchor) samples with mask=None, so _sample_like adds its seed (80) uniformly:
# scene 0's full output is all 80s. Scene 1's chunk should carry the last 2 frames of that
# (80, 80) into its own leading frames, protected (mask=0), template's remaining 3 frames
# untouched (0) and free to denoise (mask=1).
scene1_chunk = sample_calls[1]["latent_image"]
scene1_mask = sample_calls[1]["noise_mask"]
samples = scene1_chunk["samples"] if isinstance(scene1_chunk, dict) else scene1_chunk
mask = scene1_mask["samples"] if isinstance(scene1_mask, dict) else scene1_mask
assert torch.allclose(samples[:, :, :2], torch.full_like(samples[:, :, :2], 80.0))
assert torch.allclose(samples[:, :, 2:], torch.zeros_like(samples[:, :, 2:]))
assert torch.allclose(mask[:, :, :2], torch.zeros_like(mask[:, :, :2]))
assert torch.allclose(mask[:, :, 2:], torch.ones_like(mask[:, :, 2:]))
def test_mixed_anchor_resolves_identity_pin_when_configured():
"""The mixed_i2v_anchor branch skips _apply_configured_guides entirely, so without the
explicit lookup an identity_pin guide configured for the anchor scene would never resolve
and Best-FaceID identity_transfer could never fire on an anchor-swap scene."""
import json
sample_calls.clear()
node = FunPackLTXAVSceneChainSampler()
node._load_image_tensor = lambda _fn: torch.ones(1, 8, 8, 3)
node._apply_img2video_to_video_latent = lambda _vae, _img, chunk, _strength: node._clone_latent(chunk)
positive = [scene_cond(0), scene_cond(1)]
latent_template = {"samples": torch.zeros(1, 2, 5, 3, 3)}
anchors = json.dumps({"1": {"filename": "anchor.png", "strength": 1.0}})
guides = json.dumps({
"stack_enabled": True,
"scenes": [
[],
[{"enabled": True, "source": "image", "media_ref": "pin", "identity_pin": True, "strength": 0.35}],
],
})
media_refs = json.dumps({"pin": "pin.png"})
_latent, _images, _status, scene_count, _report, boundaries_json = node.sample(
model=FakeModel(),
vae=FakeVAE(),
positive=positive,
negative=[],
sampler=object(),
sigmas=torch.tensor([1.0, 0.0]),
seed=80,
latent_template=latent_template,
num_frames_per_scene=5,
frame_overlap=2,
cfg=1.0,
max_scenes=2,
funpack_scene_anchors=anchors,
funpack_scene_guides=guides,
funpack_scene_media_refs=media_refs,
identity_transfer_enabled=True,
prompt={"n1": {"inputs": {}}},
unique_id="test-node",
)
assert scene_count == 2
parsed = json.loads(boundaries_json)
mechs = parsed["scenes"][1]["mechanisms"]
assert "identity_pin_on_anchor_scene" in mechs
def test_plateau_cache_reuses_forward_on_noise_plateau_and_recomputes_below():
"""On the near-noise plateau (sigma >= threshold) the base-model forward is computed once
and reused; once sigma drops below threshold every step recomputes."""
node = FunPackLTXAVSceneChainSampler()
model = FakeModel()
stats = node._build_plateau_cache_wrapper(model, 0.975)
wrapper = model.model_options["model_function_wrapper"]
calls = {"n": 0}
x = torch.zeros(1, 4, 2, 2, 2)
def apply_fn(inp, ts, **c):
calls["n"] += 1
return torch.full_like(inp, float(calls["n"]))
def run(sig):
args = {"input": x, "timestep": torch.tensor([sig]), "cond_or_uncond": [0], "c": {}}
return wrapper(apply_fn, args)
# Default 8-step distilled schedule: sigmas 1.0..0.975 are the plateau, 0.909 onward is not.
sigmas = [1.0, 0.99375, 0.9875, 0.98125, 0.975, 0.909375, 0.725, 0.421875]
outs = [run(s) for s in sigmas]
# Plateau: 1 real forward, 4 reuses. Below threshold: 3 real forwards. Total 4 forwards vs 8.
assert stats["computed"] == 1
assert stats["reused"] == 4
assert calls["n"] == 4
for o in outs[:5]:
assert torch.allclose(o, torch.ones_like(x)) # all reuse the first plateau output
assert torch.allclose(outs[5], torch.full_like(x, 2.0)) # structure steps recompute fresh
assert torch.allclose(outs[7], torch.full_like(x, 4.0))
def test_plateau_cache_keys_cond_and_uncond_separately():
"""A CFG>1 split cond/uncond pair must each get its own cache slot, not thrash one."""
node = FunPackLTXAVSceneChainSampler()
model = FakeModel()
stats = node._build_plateau_cache_wrapper(model, 0.975)
wrapper = model.model_options["model_function_wrapper"]
calls = {"n": 0}
x = torch.zeros(1, 4, 2, 2, 2)
def apply_fn(inp, ts, **c):
calls["n"] += 1
return torch.full_like(inp, float(calls["n"]))
def run(co):
args = {"input": x, "timestep": torch.tensor([1.0]), "cond_or_uncond": co, "c": {}}
return wrapper(apply_fn, args)
run([0]); run([1]); run([0]); run([1])
assert stats["computed"] == 2 # cond + uncond each computed once
assert stats["reused"] == 2 # then each reused once
assert calls["n"] == 2