diff --git a/docs/FunPackHybridEuler2SSampler.md b/docs/FunPackHybridEuler2SSampler.md index 39d631e1..82c56893 100644 --- a/docs/FunPackHybridEuler2SSampler.md +++ b/docs/FunPackHybridEuler2SSampler.md @@ -11,13 +11,9 @@ It is designed as a quality/speed compromise between: This sampler keeps classic Euler ancestral for the early "structure building" stage where motion, anatomy and main composition are forming. On the late denoise steps it switches to a deterministic Euler / DPM-Solver++(2S) ODE refinement path where fine detail and cleanup usually matter most. -It can also build an optional paper-style Restart schedule at the progress point you choose: +It can also apply optional early/mid **motion transition pulses** for single-clip image-to-video workflows. These pulses add monotonic noise kicks at selected normal denoise steps instead of inserting upward sigma jumps. This is intended to push LTX2.3 away from stale frame-1 reference behavior while avoiding the audio damage caused by Restart replay. -- detect a restart anchor across the full sigma schedule -- insert upward sigma jumps back to a higher sigma -- replay the chosen sigma interval one or more times - -Unlike the older implementation, Restart is now encoded directly into the outgoing `SIGMAS` schedule, so the visible schedule passed into `SamplerCustomAdvanced` is the actual schedule that gets executed. +Restart replay is disabled. Legacy restart widgets may still appear for workflow compatibility, but they are ignored. ## Recommended wiring @@ -43,15 +39,30 @@ Example: `0.35` means only the last 35% of steps use the ODE refinement path. - `0.0` = pure late-step Euler ODE - `1.0` = full late-step DPM++(2S)-style correction -**restart_steps**: Target size of the restarted sigma interval. The node interprets this against the actual sigma curve and expands the outgoing `SIGMAS` schedule accordingly. +**restart_steps**: Legacy compatibility control. Ignored. -**restart_repeats**: How many Restart loops to run. `0` disables Restart. +**restart_repeats**: Legacy compatibility control. Ignored. -**restart_trigger_pct**: Sampling progress point where Restart is triggered across the full schedule. This is independent from `high_quality_pct`, so values such as `0.30` can trigger an early restart instead of being clamped to the Euler-to-2S transition point. +**restart_trigger_pct**: Legacy compatibility control. Ignored. -**restart_noise**: Noise strength used when re-noising the latent up to the restart interval's higher sigma. +**restart_noise**: Legacy compatibility control. Ignored. -**sigmas**: Optional incoming sigma schedule. If connected, the node outputs a restart-expanded sigma schedule that matches the sampler settings. If not connected, the sampler output still works, but there is no schedule for the node to rewrite. +**transition_mode**: Motion pulse preset: + +- `off`: preserve legacy sampler behavior. +- `balanced`: one moderate early/mid pulse. +- `aggressive`: at least two stronger early/mid pulses for stale image-to-video generations. +- `custom`: use the transition count, spacing, and strength exactly as configured. + +**transition_start_pct**: Sampling progress point where the first motion pulse is applied. + +**transition_count**: Number of requested early/mid motion pulses. Pulses that would land in the late quality phase are skipped. + +**transition_spacing_pct**: Progress spacing between motion pulses. + +**transition_strength**: Strength of the monotonic noise kick. Higher values push harder against stale image references, with more drift risk. + +**sigmas**: Optional incoming sigma schedule. If connected, the node returns the same monotonic schedule plus sampler-side metadata for motion pulses. If not connected, the sampler computes pulse positions from the runtime schedule. ## Recommended starting values @@ -59,18 +70,19 @@ Example: `0.35` means only the last 35% of steps use the ODE refinement path. - `s_noise = 1.0` - `high_quality_pct = 0.30` to `0.40` - `correction_blend = 1.0` -- `restart_repeats = 0` for baseline testing +- `transition_mode = off` for baseline testing -For a first Restart test, try: +For an aggressive LTX2.3 image-to-video motion test, try: - `high_quality_pct = 0.35` - `correction_blend = 1.0` -- `restart_steps = 3` -- `restart_repeats = 1` -- `restart_trigger_pct = 0.85` -- `restart_noise = 1.0` +- `transition_mode = aggressive` +- `transition_start_pct = 0.30` +- `transition_count = 2` +- `transition_spacing_pct = 0.22` +- `transition_strength = 0.85` -Example: with `8` base sampling steps, `restart_steps = 3`, and `restart_repeats = 1`, a normal evenly spaced schedule will usually expand into a schedule with about `3` extra restart transitions. With custom sigmas, the exact replay step count can shift slightly because Restart is anchored to the real sigma values rather than raw step indices. +Keep the prompt/refiner conditioning explicit about both action and camera change, for example `orbiting camera`, `dolly in`, `zoom out`, `new side angle`, `turning`, or `dynamic pose change`. ## Expected behavior @@ -81,9 +93,9 @@ Compared to plain `euler_ancestral`, this sampler should usually: - keep the early motion/anatomy formation more lively - cost less than running a heavier deterministic solver for the whole schedule -With Restart enabled, it may further improve late cleanup and detail consistency, but it will increase runtime and may be too aggressive for some video workflows if overused. +With motion pulses enabled, it should more strongly encourage action, camera movement, and viewpoint changes inside a single clip. Stronger settings can increase subject drift or visual instability. -Because Restart is now represented in the outgoing `SIGMAS`, the total number of executed transitions should match what the expanded schedule shows in the downstream custom sampler. +The outgoing `SIGMAS` remain monotonic. Motion pulses happen inside the sampler at selected denoise steps rather than by expanding the schedule. ## Limitation diff --git a/samplers.py b/samplers.py index 5231efce..dfdd7012 100644 --- a/samplers.py +++ b/samplers.py @@ -8,6 +8,9 @@ import comfy.samplers import comfy.utils +TRANSITION_MODES = ["off", "balanced", "aggressive", "custom"] + + def _sigma_fn(t): return t.neg().exp() @@ -45,148 +48,161 @@ def _hybrid_ode_step(model, x, sigma, sigma_next, s_in, extra_args, correction_b return x_euler.lerp(x_2s, correction_blend), denoised -def _renoise_to_sigma(x, current_sigma, target_sigma, restart_noise, noise_sampler): - if target_sigma <= current_sigma or restart_noise <= 0.0: +def _apply_motion_pulse(x, sigma, sigma_next, pulse_noise, noise_sampler): + if pulse_noise <= 0.0 or sigma_next >= sigma: return x - sigma_delta_sq = max(0.0, float(target_sigma * target_sigma - current_sigma * current_sigma)) + sigma_delta_sq = max(0.0, float(sigma * sigma - sigma_next * sigma_next)) if sigma_delta_sq <= 0.0: return x sigma_delta = math.sqrt(sigma_delta_sq) - return x + noise_sampler(current_sigma, target_sigma) * (restart_noise * sigma_delta) + return x + noise_sampler(sigma, sigma_next) * (pulse_noise * sigma_delta) -def _find_restart_anchor_index(sigmas, total_steps, restart_trigger_pct): - schedule_sigmas = sigmas[:total_steps] - if schedule_sigmas.numel() <= 1: +def _find_schedule_anchor_index(sigmas, total_steps, schedule_progress): + if sigmas is None or total_steps <= 1: return 0 - high_sigma = float(schedule_sigmas[0].item()) - low_sigma = None - for idx in range(int(schedule_sigmas.shape[0]) - 1, -1, -1): - value = float(schedule_sigmas[idx].item()) - if value > 0.0: - low_sigma = value - break - if low_sigma is None: - return total_steps - 1 - - schedule_progress = max(0.0, min(1.0, restart_trigger_pct)) - - if high_sigma <= 0.0 or low_sigma <= 0.0 or abs(high_sigma - low_sigma) <= 1e-12: - return min(total_steps - 1, max(0, int(round(schedule_progress * max(0, total_steps - 1))))) - - log_high = math.log(high_sigma) - log_low = math.log(low_sigma) - target_log_sigma = log_high + (log_low - log_high) * schedule_progress - - best_index = 0 - best_distance = None - for idx in range(total_steps): - sigma_value = float(sigmas[idx].item()) - if sigma_value <= 0.0: - continue - distance = abs(math.log(sigma_value) - target_log_sigma) - if best_distance is None or distance < best_distance: - best_distance = distance - best_index = idx - return best_index + schedule_progress = max(0.0, min(1.0, schedule_progress)) + return min(total_steps - 1, max(0, int(round(schedule_progress * max(0, total_steps - 1))))) -def _build_restart_sigmas(sigmas, total_steps, restart_trigger_pct, restart_steps): - if total_steps <= 0: - return sigmas[:0], 0 +def _resolve_transition_options(transition_mode, transition_start_pct, + transition_count, transition_spacing_pct, + transition_strength): + mode = (transition_mode or "off").lower() + if mode not in TRANSITION_MODES: + mode = "off" - anchor_index = _find_restart_anchor_index(sigmas, total_steps, restart_trigger_pct) - anchor_index = min(total_steps - 1, max(0, anchor_index)) + if mode == "off": + return { + "enabled": False, + "start_pct": 0.30, + "count": 0, + "spacing_pct": 0.22, + "strength": 0.0, + "noise": 0.0, + } - if restart_steps <= 1: - return sigmas[anchor_index:anchor_index + 1], anchor_index + start_pct = 0.30 if transition_start_pct is None else float(transition_start_pct) + spacing_pct = 0.22 if transition_spacing_pct is None else float(transition_spacing_pct) + strength = 0.85 if transition_strength is None else float(transition_strength) + count = 2 if transition_count is None else int(transition_count) - schedule_sigmas = sigmas[:total_steps] - if schedule_sigmas.numel() <= 1: - return sigmas[anchor_index:anchor_index + 1], anchor_index + start_pct = max(0.02, min(0.90, start_pct)) + spacing_pct = max(0.04, min(0.45, spacing_pct)) + strength = max(0.0, min(1.0, strength)) + count = max(1, min(6, count)) - positive_logs = [] - for idx in range(int(schedule_sigmas.shape[0])): - sigma_value = float(schedule_sigmas[idx].item()) - if sigma_value > 0.0: - positive_logs.append(math.log(sigma_value)) + if mode == "balanced": + count = min(count, 1) + strength = 0.55 if transition_strength is None else min(strength, 0.70) + elif mode == "aggressive": + count = max(2, count) + strength = max(strength, 0.85) - if len(positive_logs) >= 2: - total_log_span = abs(positive_logs[0] - positive_logs[-1]) - avg_log_step = total_log_span / max(1, len(positive_logs) - 1) - target_log_span = avg_log_step * max(1, restart_steps - 1) - else: - target_log_span = 0.0 + noise = 0.10 + strength * 0.55 - restart_from = anchor_index - accumulated_span = 0.0 - for idx in range(anchor_index, 0, -1): - sigma_hi = float(sigmas[idx - 1].item()) - sigma_lo = float(sigmas[idx].item()) - restart_from = idx - 1 - if sigma_hi > 0.0 and sigma_lo > 0.0: - accumulated_span += abs(math.log(sigma_hi) - math.log(sigma_lo)) - if (anchor_index - restart_from) >= 1 and accumulated_span >= target_log_span: - break - - restart_from = max(0, restart_from) - return sigmas[restart_from:anchor_index + 1], anchor_index + return { + "enabled": True, + "start_pct": start_pct, + "count": count, + "spacing_pct": spacing_pct, + "strength": strength, + "noise": max(0.0, min(0.80, noise)), + } -def _expand_restart_sigmas(sigmas, high_quality_pct, restart_steps, restart_repeats, restart_trigger_pct): - if sigmas is None or not isinstance(sigmas, torch.Tensor): - return None, None - - expanded_sigmas = sigmas.detach().clone() - total_steps = max(0, int(expanded_sigmas.shape[0]) - 1) - if total_steps <= 0: - return expanded_sigmas, None - +def _get_late_start_index(total_steps, high_quality_pct): high_quality_pct = max(0.0, min(1.0, float(high_quality_pct))) - restart_steps = max(2, int(restart_steps)) - restart_repeats = max(0, int(restart_repeats)) - restart_trigger_pct = max(0.0, min(1.0, float(restart_trigger_pct))) - late_steps = max(1, int(math.ceil(total_steps * high_quality_pct))) if high_quality_pct > 0.0 else 0 - late_start = max(0, total_steps - late_steps) + return max(0, total_steps - late_steps) - quality_sigma_start = None - if late_start < expanded_sigmas.shape[0]: - quality_sigma_start = float(expanded_sigmas[late_start].item()) - if restart_repeats <= 0: - return expanded_sigmas, quality_sigma_start - - restart_sigmas, anchor_index = _build_restart_sigmas( - expanded_sigmas, - total_steps, - restart_trigger_pct, - restart_steps, +def _build_transition_pulse_steps(sigmas, total_steps, high_quality_pct, + transition_mode, transition_start_pct, + transition_count, transition_spacing_pct, + transition_strength): + options = _resolve_transition_options( + transition_mode, + transition_start_pct, + transition_count, + transition_spacing_pct, + transition_strength, ) - if restart_sigmas.numel() <= 1: - return expanded_sigmas, quality_sigma_start + if not options["enabled"] or total_steps <= 2: + return [], options - prefix = expanded_sigmas[:anchor_index + 1] - suffix = expanded_sigmas[anchor_index + 1:] - repeated = [restart_sigmas.clone() for _ in range(restart_repeats)] - expanded_sigmas = torch.cat([prefix] + repeated + [suffix], dim=0) - return expanded_sigmas, quality_sigma_start + late_start = _get_late_start_index(total_steps, high_quality_pct) + latest_anchor_pct = max(0.04, min(0.92, (late_start - 1) / max(1, total_steps - 1))) + pulse_steps = [] + used_anchors = set() + + for pulse_index in range(options["count"]): + trigger_pct = options["start_pct"] + options["spacing_pct"] * pulse_index + if trigger_pct >= latest_anchor_pct: + break + + anchor_index = _find_schedule_anchor_index(sigmas, total_steps, trigger_pct) + anchor_index = min(total_steps - 1, max(0, anchor_index)) + if anchor_index in used_anchors or anchor_index >= late_start: + continue + + used_anchors.add(anchor_index) + pulse_steps.append({ + "step_index": anchor_index, + "noise": options["noise"], + }) + + return pulse_steps, options + + +def _prepare_dynamic_sigmas(sigmas, high_quality_pct, transition_mode="off", + transition_start_pct=0.30, transition_count=2, + transition_spacing_pct=0.22, transition_strength=0.85): + if sigmas is None or not isinstance(sigmas, torch.Tensor): + return None, None, [], 0.0 + + base_sigmas = sigmas.detach().clone() + total_steps = max(0, int(base_sigmas.shape[0]) - 1) + if total_steps <= 0: + return base_sigmas, None, [], 0.0 + + late_start = _get_late_start_index(total_steps, high_quality_pct) + quality_sigma_start = None + if late_start < base_sigmas.shape[0]: + quality_sigma_start = float(base_sigmas[late_start].item()) + + pulse_steps, transition_options = _build_transition_pulse_steps( + base_sigmas, + total_steps, + high_quality_pct, + transition_mode, + transition_start_pct, + transition_count, + transition_spacing_pct, + transition_strength, + ) + return base_sigmas, quality_sigma_start, pulse_steps, transition_options["noise"] def sample_funpack_hybrid_euler_2s(model, x, sigmas, extra_args=None, callback=None, disable=None, eta=1.0, s_noise=1.0, high_quality_pct=0.35, correction_blend=1.0, - restart_steps=3, restart_repeats=0, - restart_trigger_pct=0.85, restart_noise=1.0, - quality_sigma_start=None): + quality_sigma_start=None, + transition_mode="off", + transition_start_pct=0.30, + transition_count=2, + transition_spacing_pct=0.22, + transition_strength=0.85, + transition_noise=0.0, + transition_pulse_steps=None): """ Hybrid sampler: - Early schedule: Euler ancestral for motion/anatomy buildup. - Late schedule: deterministic Euler / DPM-Solver++(2S) ODE refinement for detail. - - Restart: paper-style re-noise and short replay at the user-selected schedule point. + - Motion pulses: optional monotonic noise kicks before the late quality phase. """ if isinstance(model.inner_model.inner_model.model_sampling, comfy.model_sampling.CONST): return k_diffusion_sampling.sample_euler_ancestral( @@ -204,10 +220,23 @@ def sample_funpack_hybrid_euler_2s(model, x, sigmas, extra_args=None, callback=N high_quality_pct = max(0.0, min(1.0, float(high_quality_pct))) correction_blend = max(0.0, min(1.0, float(correction_blend))) - restart_steps = max(2, int(restart_steps)) - restart_repeats = max(0, int(restart_repeats)) - restart_trigger_pct = max(0.0, min(1.0, float(restart_trigger_pct))) - restart_noise = max(0.0, float(restart_noise)) + if not transition_pulse_steps: + _, _, transition_pulse_steps, computed_transition_noise = _prepare_dynamic_sigmas( + sigmas, + high_quality_pct, + transition_mode, + transition_start_pct, + transition_count, + transition_spacing_pct, + transition_strength, + ) + transition_noise = computed_transition_noise + transition_noise = max(0.0, float(transition_noise)) + transition_step_noise = { + int(item.get("step_index", -1)): max(0.0, float(item.get("noise", transition_noise))) + for item in (transition_pulse_steps or []) + if isinstance(item, dict) + } s_in = x.new_ones([x.shape[0]]) callback_step = 0 @@ -222,25 +251,13 @@ def sample_funpack_hybrid_euler_2s(model, x, sigmas, extra_args=None, callback=N for i in comfy.utils.model_trange(total_steps, disable=disable): sigma = sigmas[i] sigma_next = sigmas[i + 1] - is_restart_jump = sigma_next > sigma in_quality_phase = quality_sigma_start is not None and float(sigma.item()) <= quality_sigma_start - if is_restart_jump: - denoised = None - if callback is not None: - denoised = model(x, sigma * s_in, **extra_args) - callback({ - "x": x, - "i": callback_step, - "sigma": sigma, - "sigma_hat": sigma, - "denoised": denoised, - }) - callback_step += 1 - x = _renoise_to_sigma(x, sigma, sigma_next, restart_noise, noise_sampler) - continue - if not in_quality_phase: + pulse_noise = transition_step_noise.get(int(i), 0.0) + if pulse_noise > 0.0: + x = _apply_motion_pulse(x, sigma, sigma_next, pulse_noise, noise_sampler) + denoised = model(x, sigma * s_in, **extra_args) if callback is not None: @@ -329,28 +346,60 @@ class FunPackHybridEuler2SSampler: "min": 2, "max": 32, "step": 1, - "tooltip": "How many previous sigma steps are revisited during Restart replay." + "tooltip": "Legacy Restart control. Ignored because Restart replay is disabled for LTX audio safety." }), "restart_repeats": ("INT", { "default": 0, "min": 0, "max": 8, "step": 1, - "tooltip": "How many paper-style re-noise/replay loops to run at the chosen restart point. Set to 0 to disable Restart." + "tooltip": "Legacy Restart control. Ignored because Restart replay is disabled for LTX audio safety." }), "restart_trigger_pct": ("FLOAT", { "default": 0.85, "min": 0.0, "max": 1.0, "step": 0.01, - "tooltip": "Sampling progress point where Restart is triggered across the full schedule." + "tooltip": "Legacy Restart control. Ignored because Restart replay is disabled for LTX audio safety." }), "restart_noise": ("FLOAT", { "default": 1.0, "min": 0.0, "max": 2.0, "step": 0.01, - "tooltip": "Noise strength used when re-noising up to the Restart interval's higher sigma." + "tooltip": "Legacy Restart control. Ignored because Restart replay is disabled for LTX audio safety." + }), + "transition_mode": (TRANSITION_MODES, { + "default": "off", + "tooltip": "Adds early/mid motion pulses for single-clip image-to-video. Off preserves legacy sampler behavior." + }), + "transition_start_pct": ("FLOAT", { + "default": 0.30, + "min": 0.0, + "max": 0.95, + "step": 0.01, + "tooltip": "Sampling progress where the first motion pulse starts." + }), + "transition_count": ("INT", { + "default": 2, + "min": 1, + "max": 6, + "step": 1, + "tooltip": "How many early/mid motion pulses to request before the late quality phase." + }), + "transition_spacing_pct": ("FLOAT", { + "default": 0.22, + "min": 0.04, + "max": 0.45, + "step": 0.01, + "tooltip": "Progress spacing between motion pulses." + }), + "transition_strength": ("FLOAT", { + "default": 0.85, + "min": 0.0, + "max": 1.0, + "step": 0.01, + "tooltip": "How strongly motion pulses add monotonic noise kicks. Higher values push harder against stale image references." }), }, "optional": { @@ -362,16 +411,21 @@ class FunPackHybridEuler2SSampler: RETURN_NAMES = ("sampler", "sigmas") FUNCTION = "get_sampler" CATEGORY = "FunPack/Sampling" - DESCRIPTION = "Hybrid sampler: early Euler ancestral for motion, late DPM-Solver++(2S) ODE refinement for quality, with optional sigma-driven Restart replay." + DESCRIPTION = "Hybrid sampler: early Euler ancestral for motion, late DPM-Solver++(2S) ODE refinement for quality, with optional audio-safe motion pulses." def get_sampler(self, eta, s_noise, high_quality_pct, correction_blend, - restart_steps, restart_repeats, restart_trigger_pct, restart_noise, sigmas=None): - expanded_sigmas, quality_sigma_start = _expand_restart_sigmas( + restart_steps, restart_repeats, restart_trigger_pct, restart_noise, + transition_mode="off", transition_start_pct=0.30, + transition_count=2, transition_spacing_pct=0.22, + transition_strength=0.85, sigmas=None): + prepared_sigmas, quality_sigma_start, transition_pulse_steps, transition_noise = _prepare_dynamic_sigmas( sigmas, high_quality_pct, - restart_steps, - restart_repeats, - restart_trigger_pct, + transition_mode, + transition_start_pct, + transition_count, + transition_spacing_pct, + transition_strength, ) sampler = comfy.samplers.KSAMPLER( sample_funpack_hybrid_euler_2s, @@ -380,11 +434,14 @@ class FunPackHybridEuler2SSampler: "s_noise": s_noise, "high_quality_pct": high_quality_pct, "correction_blend": correction_blend, - "restart_steps": restart_steps, - "restart_repeats": restart_repeats, - "restart_trigger_pct": restart_trigger_pct, - "restart_noise": restart_noise, "quality_sigma_start": quality_sigma_start, + "transition_mode": transition_mode, + "transition_start_pct": transition_start_pct, + "transition_count": transition_count, + "transition_spacing_pct": transition_spacing_pct, + "transition_strength": transition_strength, + "transition_noise": transition_noise, + "transition_pulse_steps": transition_pulse_steps, } ) - return (sampler, expanded_sigmas) + return (sampler, prepared_sigmas)