commit 1a3aed8540a437db82a18cbf64caf6134b2aa0dd Author: m1_4k Date: Tue Feb 24 23:29:55 2026 +0900 Initial commit: Phase-based 2x frame interpolation ComfyUI node Implements Fourier-transform phase difference interpolation on a complex steerable pyramid. Doubles video frame rate while preserving sharpness and minimizing ghosting artifacts. Co-Authored-By: Claude Sonnet 4.6 diff --git a/README.md b/README.md new file mode 100644 index 0000000..477c4d5 --- /dev/null +++ b/README.md @@ -0,0 +1,138 @@ +# ComfyUI_FrameInterpolation + +A ComfyUI custom node for **2× video frame interpolation** using a Fourier-transform-based phase difference algorithm on a complex steerable pyramid. + +--- + +## Overview + +Standard frame interpolation blends pixel intensities linearly between two frames, which produces blurry or ghosted results whenever objects are in motion. This node takes a different approach: instead of blending intensities, it interpolates the **phase** of image content at each spatial scale. + +> "Image structure is encoded primarily by phase. Interpolating the phase of each pixel produces a much higher quality result." +> — [Phase-Based Frame Interpolation, exclav.es](https://exclav.es/2016/07/15/phase-based-frame-interpolation-a-julia-implementation/) + +The result is a synthesized intermediate frame that preserves sharp edges and fine structure with significantly less ghosting than naive linear blending. + +--- + +## Algorithm + +### 1. Color Space Separation + +Each input frame is converted from RGB to **CIE Lab** color space. The **L (luminance)** channel carries the structural information processed by phase interpolation. The **a and b (chrominance)** channels are handled with simple linear interpolation, since human vision is far less sensitive to chroma artifacts. + +### 2. Complex Analytic Pyramid + +A multi-scale complex pyramid is built from the luminance channel of each frame. At every pyramid level, a **2D analytic signal** is computed via FFT by zeroing negative-frequency components (equivalent to a 2D Hilbert transform). This yields complex coefficients where: + +- **Amplitude** encodes local contrast at that scale. +- **Phase** encodes the structural position of image features. + +### 3. Phase Difference + +For each corresponding pyramid level pair `(pyr1[k], pyr2[k])`, the per-pixel phase difference is computed as: + +``` +Δφ[k] = angle( pyr2[k] · conj(pyr1[k]) ) +``` + +This gives the wrapped phase shift between the two frames at scale `k`, in the range `(−π, π]`. + +### 4. Hierarchical Phase Unwrapping + +Raw phase differences are ambiguous modulo `2π`. The unwrapping resolves this by propagating motion estimates **coarse-to-fine**: + +1. The coarsest level is taken as-is (no ambiguity at low resolution). +2. For each finer level, the coarse unwrapped phase is upsampled and used as a prediction. +3. The residual between the fine difference and the coarse prediction is rounded to the nearest multiple of `2π` and subtracted — effectively choosing the physically consistent branch. + +### 5. Phase Consistency Adjustment + +A final pass ensures each unwrapped value agrees with the raw phase difference modulo `2π`, correcting any remaining inconsistencies introduced by the upsampling step. + +### 6. Interpolation + +For an interpolation factor `α ∈ [0, 1]` (midpoint = 0.5): + +| Component | Formula | +|-----------|---------| +| Amplitude | `(1 − α) · amp1 + α · amp2` (linear blend) | +| Phase | `φ1 + α · Δφ_unwrapped` | +| Complex coeff | `amp_interp · exp(i · phase_interp)` | + +### 7. Reconstruction + +The interpolated complex pyramid is collapsed back into a luminance image by summing real parts across scales. Chrominance channels are linearly blended. Both are then recombined into Lab and converted back to RGB. + +### 8. Video Assembly + +The output video contains the original frames interleaved with synthesized intermediate frames, doubling the frame count. The output frame rate is set to `2× input FPS` so playback duration is unchanged. + +--- + +## Node Reference + +**Category:** `video/interpolation` +**Display Name:** `Phase-Based Frame Interpolation (2x)` +**Internal Name:** `PhaseBasedFrameInterpolation` + +### Inputs + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `video_path` | STRING | — | Absolute path to the input video file (mp4, avi, mov, etc.) | +| `pyramid_levels` | INT (3–10) | 6 | Number of pyramid levels. More levels improve unwrapping accuracy at coarser scales but increase processing time. | +| `output_filename` | STRING | `phase_interp_output.mp4` | Filename for the output video, written to ComfyUI's output directory. | + +### Outputs + +| Parameter | Type | Description | +|-----------|------|-------------| +| `output_video_path` | STRING | Absolute path to the interpolated output video. | + +--- + +## Installation + +```bash +cd ComfyUI/custom_nodes +git clone ComfyUI_FrameInterpolation +pip install -r ComfyUI_FrameInterpolation/requirements.txt +``` + +Restart ComfyUI. The node will appear under **video/interpolation**. + +--- + +## Requirements + +| Package | Purpose | +|---------|---------| +| `numpy` | Array operations, FFT | +| `scipy` | (available for future extensions) | +| `opencv-python` | Video I/O, color space conversion, pyramid resizing | + +--- + +## Performance Notes + +- Processing is **CPU-bound**. A 1-minute 1080p clip at 30 fps may take several minutes depending on hardware. +- `pyramid_levels = 6` balances quality and speed well for most content. +- Increasing `pyramid_levels` improves phase unwrapping for large motions but scales processing time roughly linearly. +- Memory usage scales with the total number of pixels × pyramid levels. Very high-resolution videos should be tested with caution. + +--- + +## Limitations + +- The algorithm assumes **small-to-moderate motion** between frames. Large displacements (more than roughly a quarter of the image width per frame) may still produce artifacts, as the phase-unwrapping heuristic can fail at extreme shifts. +- Occlusions and dis-occlusions (objects appearing or disappearing) are not explicitly handled; these regions will blend toward the nearest available content. +- Only luminance is phase-interpolated; chroma is linearly blended, which may produce mild saturation artifacts on color edges under large motion. + +--- + +## References + +- [Phase-Based Frame Interpolation — A Julia Implementation](https://exclav.es/2016/07/15/phase-based-frame-interpolation-a-julia-implementation/) +- Simoncelli, E. P., & Freeman, W. T. (1995). *The steerable pyramid: A flexible architecture for multi-scale derivative computation.* ICIP 1995. +- Fleet, D. J., & Jepson, A. D. (1990). *Computation of component image velocity from local phase information.* IJCV 5(1). diff --git a/__init__.py b/__init__.py new file mode 100644 index 0000000..39a8c6b --- /dev/null +++ b/__init__.py @@ -0,0 +1,3 @@ +from .nodes import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS + +__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"] diff --git a/nodes.py b/nodes.py new file mode 100644 index 0000000..1966078 --- /dev/null +++ b/nodes.py @@ -0,0 +1,124 @@ +""" +ComfyUI custom node: Phase-Based Frame Interpolation (2x) + +Interpolates video frames using Fourier-transform-based phase differences +on a complex steerable pyramid, producing a 2x frame-rate video with +minimal ghosting compared to naive linear interpolation. +""" + +import os +import tempfile +import folder_paths + +from .phase_interpolation import interpolate_video_2x + + +class PhaseBasedFrameInterpolation: + """ + 2x Frame Interpolation via Phase-Based Fourier Method. + + Accepts a video file path, inserts one interpolated frame between + every pair of consecutive frames, and writes the result to an output + video file. + + Algorithm + --------- + 1. Convert each frame pair to Lab color space. + 2. Build complex analytic pyramids from the luminance channel. + 3. Compute hierarchically-unwrapped phase differences. + 4. Reconstruct interpolated luminance from amplitude + phase. + 5. Blend chrominance linearly (human vision is less sensitive there). + 6. Assemble output video at 2× the original frame rate. + """ + + @classmethod + def INPUT_TYPES(cls): + return { + "required": { + "video_path": ( + "STRING", + { + "default": "", + "multiline": False, + "tooltip": "Absolute path to the input video file (mp4, avi, mov, ...)", + }, + ), + "pyramid_levels": ( + "INT", + { + "default": 6, + "min": 3, + "max": 10, + "step": 1, + "tooltip": ( + "Number of complex pyramid levels. " + "More levels = finer phase unwrapping but slower." + ), + }, + ), + "output_filename": ( + "STRING", + { + "default": "phase_interp_output.mp4", + "multiline": False, + "tooltip": "Output filename (saved in ComfyUI output directory)", + }, + ), + } + } + + RETURN_TYPES = ("STRING",) + RETURN_NAMES = ("output_video_path",) + FUNCTION = "interpolate" + CATEGORY = "video/interpolation" + DESCRIPTION = ( + "2x frame interpolation using Fourier-transform phase differences. " + "Inserts one synthesized frame between every consecutive pair, " + "doubling the frame rate while preserving sharpness and structure." + ) + + def interpolate( + self, + video_path: str, + pyramid_levels: int, + output_filename: str, + ) -> tuple: + video_path = video_path.strip() + if not video_path: + raise ValueError("video_path must not be empty") + if not os.path.isfile(video_path): + raise FileNotFoundError(f"Video not found: {video_path}") + + # Resolve output directory + output_dir = folder_paths.get_output_directory() + os.makedirs(output_dir, exist_ok=True) + + # Sanitize filename + if not output_filename.strip(): + output_filename = "phase_interp_output.mp4" + base, ext = os.path.splitext(output_filename) + if not ext: + ext = ".mp4" + out_path = os.path.join(output_dir, base + ext) + + # Run interpolation + result = interpolate_video_2x( + input_path=video_path, + output_path=out_path, + num_levels=pyramid_levels, + ) + + return (result,) + + +# --------------------------------------------------------------------------- +# Node registration +# --------------------------------------------------------------------------- + +NODE_CLASS_MAPPINGS = { + "PhaseBasedFrameInterpolation": PhaseBasedFrameInterpolation, +} + +NODE_DISPLAY_NAME_MAPPINGS = { + "PhaseBasedFrameInterpolation": "Phase-Based Frame Interpolation (2x)", +} diff --git a/phase_interpolation.py b/phase_interpolation.py new file mode 100644 index 0000000..5b7c816 --- /dev/null +++ b/phase_interpolation.py @@ -0,0 +1,375 @@ +""" +Phase-Based Frame Interpolation using Fourier Transform and Complex Steerable Pyramid + +Algorithm reference: + https://exclav.es/2016/07/15/phase-based-frame-interpolation-a-julia-implementation/ + +Core concept: + Image structure is encoded primarily by phase. + Interpolating phase of each pixel (rather than pixel intensity) produces + higher quality results with less ghosting. + +Steps: + 1. Convert frames to Lab color space, extract luminance (L channel) + 2. Build complex steerable pyramid for each frame + 3. Compute phase differences between corresponding pyramid levels + 4. Unwrap phase differences hierarchically (coarse-to-fine) + 5. Interpolate: amplitude linearly, phase = original + alpha * unwrapped_diff + 6. Reconstruct pyramid -> luminance, blend chrominance linearly + 7. Convert back to RGB +""" + +import numpy as np +from scipy.ndimage import zoom +import cv2 + + +# --------------------------------------------------------------------------- +# Utility: color space conversions +# --------------------------------------------------------------------------- + +def rgb_to_lab_float(frame_rgb: np.ndarray) -> np.ndarray: + """ + Convert RGB uint8 -> Lab float32 using OpenCV float32 convention. + + frame_rgb : (H, W, 3) uint8, values [0, 255] + returns : (H, W, 3) float32 + L in [0, 100] + a in [-127, 127] + b in [-127, 127] + + Internally OpenCV's float32 Lab path expects BGR input in [0.0, 1.0]. + """ + bgr_f = cv2.cvtColor(frame_rgb, cv2.COLOR_RGB2BGR).astype(np.float32) / 255.0 + lab = cv2.cvtColor(bgr_f, cv2.COLOR_BGR2Lab) # L∈[0,100], a,b∈[-127,127] + return lab + + +def lab_to_rgb_uint8(lab: np.ndarray) -> np.ndarray: + """ + Convert Lab float32 -> RGB uint8. + + lab : (H, W, 3) float32 (L∈[0,100], a,b∈[-127,127]) + returns : (H, W, 3) uint8 [0, 255] + """ + lab_clipped = lab.copy() + lab_clipped[:, :, 0] = np.clip(lab_clipped[:, :, 0], 0.0, 100.0) + lab_clipped[:, :, 1] = np.clip(lab_clipped[:, :, 1], -127.0, 127.0) + lab_clipped[:, :, 2] = np.clip(lab_clipped[:, :, 2], -127.0, 127.0) + bgr_f = cv2.cvtColor(lab_clipped, cv2.COLOR_Lab2BGR) # [0.0, 1.0] + rgb_f = cv2.cvtColor(bgr_f, cv2.COLOR_BGR2RGB) + return np.clip(rgb_f * 255.0, 0, 255).astype(np.uint8) + + +# --------------------------------------------------------------------------- +# Complex Steerable Pyramid +# --------------------------------------------------------------------------- + +def _make_gaussian_kernel(sigma: float = 1.0) -> np.ndarray: + """Create a small Gaussian kernel for blurring.""" + size = int(6 * sigma + 1) | 1 # ensure odd + x = np.arange(size) - size // 2 + g = np.exp(-x ** 2 / (2 * sigma ** 2)) + g /= g.sum() + return np.outer(g, g) + + +def build_laplacian_pyramid(image: np.ndarray, num_levels: int = 6) -> list: + """ + Build a Laplacian pyramid. + + Each level contains the band-pass detail at that scale. + The last level is the low-pass residual. + + Returns list of (H_k, W_k) float32 arrays. + """ + pyramid = [] + current = image.astype(np.float32) + for _ in range(num_levels - 1): + blurred = cv2.GaussianBlur(current, (0, 0), sigmaX=1.0) + h, w = current.shape[:2] + downsampled = cv2.resize(blurred, (w // 2, h // 2), interpolation=cv2.INTER_LINEAR) + upsampled = cv2.resize(downsampled, (w, h), interpolation=cv2.INTER_LINEAR) + laplacian = current - upsampled + pyramid.append(laplacian) + current = downsampled + pyramid.append(current) # residual + return pyramid + + +def build_complex_pyramid(image: np.ndarray, num_levels: int = 6) -> list: + """ + Build a complex steerable pyramid approximation. + + For each Laplacian level, compute the complex analytic signal using + the Hilbert transform in 2D (via FFT), giving complex coefficients + where amplitude encodes local contrast and phase encodes structure. + + Returns list of complex128 arrays, same shape as Laplacian levels. + """ + lap_pyr = build_laplacian_pyramid(image, num_levels) + complex_pyr = [] + for band in lap_pyr: + # 2D analytic signal: zero out negative frequencies + F = np.fft.fft2(band) + H, W = band.shape + # Build half-plane filter (analytic signal in 2D) + # Use the standard 2D Hilbert / analytic approach: + # keep DC, double positive half, zero negative half + mask = np.zeros((H, W), dtype=np.float64) + # DC component + mask[0, 0] = 1.0 + # Positive frequencies + if H % 2 == 0: + mask[1:H // 2, :] = 2.0 + mask[H // 2, :] = 1.0 # Nyquist row + else: + mask[1:(H + 1) // 2, :] = 2.0 + if W % 2 == 0: + # Already handled via rows; but for W-direction keep symmetric + pass + analytic_F = F * mask + analytic = np.fft.ifft2(analytic_F) + complex_pyr.append(analytic) + return complex_pyr + + +# --------------------------------------------------------------------------- +# Phase operations +# --------------------------------------------------------------------------- + +def compute_phase_pyramid(pyr1: list, pyr2: list) -> list: + """ + Compute phase difference between two complex pyramids. + + phase_diff[k] = angle(pyr2[k] * conj(pyr1[k])) + = phase(pyr2[k]) - phase(pyr1[k]) + + Returns list of float arrays. + """ + phase_diffs = [] + for c1, c2 in zip(pyr1, pyr2): + diff = np.angle(c2 * np.conj(c1)) + phase_diffs.append(diff) + return phase_diffs + + +def _wrap_to_pi(x: np.ndarray) -> np.ndarray: + """Wrap values to [-pi, pi].""" + return (x + np.pi) % (2 * np.pi) - np.pi + + +def unwrap_phase_pyramid(phase_diffs: list) -> list: + """ + Hierarchical phase unwrapping (coarse to fine). + + For each level k (fine to coarse ordering, i.e. index 0=finest), + we unwrap by comparing to the next coarser level upsampled back. + + Algorithm: + 1. Start from coarsest level (last index) - no unwrapping needed. + 2. For each finer level k, upsample the unwrapped coarser level, + compute the residual: diff[k] - wrapped_coarse_up + round residual to nearest multiple of 2π and subtract. + + This ensures the fine-scale phase delta is consistent with the + coarse-scale motion estimate. + """ + num_levels = len(phase_diffs) + unwrapped = [None] * num_levels + + # Coarsest level: use as-is + unwrapped[-1] = phase_diffs[-1].copy() + + # Process from coarse to fine + for k in range(num_levels - 2, -1, -1): + fine_diff = phase_diffs[k] + coarse_unwrapped = unwrapped[k + 1] + + # Upsample coarse to fine resolution + h_fine, w_fine = fine_diff.shape + coarse_up = cv2.resize( + coarse_unwrapped.astype(np.float32), + (w_fine, h_fine), + interpolation=cv2.INTER_LINEAR + ).astype(np.float64) + + # Compute residual between fine diff and coarse prediction + residual = fine_diff - coarse_up + # Round residual to nearest multiple of 2π + correction = np.round(residual / (2 * np.pi)) * (2 * np.pi) + unwrapped[k] = fine_diff - correction + + return unwrapped + + +def adjust_phase_consistency(phase_diffs: list, unwrapped: list) -> list: + """ + Final phase consistency adjustment: + Ensure that (unwrapped mod 2π) == (original diff mod 2π). + Add/subtract 2π to bring unwrapped as close as possible to original. + """ + adjusted = [] + for orig, unwr in zip(phase_diffs, unwrapped): + residual = orig - unwr + correction = np.round(residual / (2 * np.pi)) * (2 * np.pi) + adjusted.append(unwr + correction) + return adjusted + + +# --------------------------------------------------------------------------- +# Reconstruction +# --------------------------------------------------------------------------- + +def reconstruct_from_pyramid(complex_pyr: list) -> np.ndarray: + """ + Reconstruct image from a complex pyramid by taking real part. + + Reverses the Laplacian pyramid: sum up from coarsest to finest. + """ + result = np.real(complex_pyr[-1]).astype(np.float32) + for band in reversed(complex_pyr[:-1]): + h, w = band.shape + result = cv2.resize(result, (w, h), interpolation=cv2.INTER_LINEAR) + result += np.real(band).astype(np.float32) + return result + + +# --------------------------------------------------------------------------- +# Main interpolation function +# --------------------------------------------------------------------------- + +def interpolate_frames(frame1: np.ndarray, frame2: np.ndarray, + alpha: float = 0.5, num_levels: int = 6) -> np.ndarray: + """ + Interpolate between two RGB frames using phase-based method. + + Parameters + ---------- + frame1, frame2 : (H, W, 3) uint8 RGB + alpha : interpolation position [0,1], 0=frame1, 1=frame2 + num_levels : number of pyramid levels + + Returns + ------- + interpolated : (H, W, 3) uint8 RGB + """ + # 1. Convert to Lab, separate L (luminance) from ab (chrominance) + lab1 = rgb_to_lab_float(frame1) # (H,W,3) + lab2 = rgb_to_lab_float(frame2) + + L1 = lab1[:, :, 0] / 100.0 # normalize L [0,100] -> [0,1] + L2 = lab2[:, :, 0] / 100.0 + ab1 = lab1[:, :, 1:] # (H,W,2) + ab2 = lab2[:, :, 1:] + + # 2. Build complex pyramids for luminance + pyr1 = build_complex_pyramid(L1, num_levels) + pyr2 = build_complex_pyramid(L2, num_levels) + + # 3. Compute phase differences + phase_diffs = compute_phase_pyramid(pyr1, pyr2) + + # 4. Unwrap phase differences hierarchically + unwrapped = unwrap_phase_pyramid(phase_diffs) + + # 5. Adjust phase consistency + adjusted = adjust_phase_consistency(phase_diffs, unwrapped) + + # 6. Build interpolated pyramid + interp_pyr = [] + for (c1, c2, adj_diff) in zip(pyr1, pyr2, adjusted): + # Linearly interpolate amplitude + amp1 = np.abs(c1) + amp2 = np.abs(c2) + amp_interp = (1 - alpha) * amp1 + alpha * amp2 + + # Phase of frame1 + scaled phase difference + phase1 = np.angle(c1) + phase_interp = phase1 + alpha * adj_diff + + # Reconstruct complex coefficients + c_interp = amp_interp * np.exp(1j * phase_interp) + interp_pyr.append(c_interp) + + # 7. Reconstruct luminance + L_interp = reconstruct_from_pyramid(interp_pyr) + # Clamp and rescale back to [0, 100] Lab L range + L_interp = np.clip(L_interp * 100.0, 0, 100) + + # 8. Linearly interpolate chrominance (ab channels) + ab_interp = (1 - alpha) * ab1 + alpha * ab2 + + # 9. Assemble Lab and convert back to RGB + lab_interp = np.zeros_like(lab1) + lab_interp[:, :, 0] = L_interp + lab_interp[:, :, 1:] = ab_interp + + rgb_interp = lab_to_rgb_uint8(lab_interp) + return rgb_interp + + +# --------------------------------------------------------------------------- +# Video processing +# --------------------------------------------------------------------------- + +def interpolate_video_2x(input_path: str, output_path: str, + num_levels: int = 6) -> str: + """ + Load video, insert one interpolated frame between each pair of frames (2x). + + Parameters + ---------- + input_path : path to input video + output_path : path to output video + num_levels : steerable pyramid levels + + Returns + ------- + output_path : same as output_path on success + """ + cap = cv2.VideoCapture(input_path) + if not cap.isOpened(): + raise RuntimeError(f"Cannot open video: {input_path}") + + fps = cap.get(cv2.CAP_PROP_FPS) + width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) + height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) + total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) + + # Output fps is 2x (more frames, same duration) + out_fps = fps * 2.0 + + fourcc = cv2.VideoWriter_fourcc(*"mp4v") + writer = cv2.VideoWriter(output_path, fourcc, out_fps, (width, height)) + + # Read all frames into memory (for moderate-length clips) + frames = [] + while True: + ret, frame = cap.read() + if not ret: + break + frames.append(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)) + cap.release() + + if len(frames) < 2: + raise RuntimeError("Video must have at least 2 frames") + + print(f"[PhaseInterp] Input: {len(frames)} frames @ {fps:.2f} fps") + print(f"[PhaseInterp] Output: {len(frames) * 2 - 1} frames @ {out_fps:.2f} fps") + + for i, frame in enumerate(frames): + # Write original frame + writer.write(cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)) + + # Write interpolated frame between current and next + if i < len(frames) - 1: + interp = interpolate_frames(frame, frames[i + 1], + alpha=0.5, num_levels=num_levels) + writer.write(cv2.cvtColor(interp, cv2.COLOR_RGB2BGR)) + if (i + 1) % 10 == 0 or i == 0: + print(f"[PhaseInterp] Processed {i + 1}/{len(frames) - 1} frame pairs") + + writer.release() + print(f"[PhaseInterp] Done -> {output_path}") + return output_path diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..0b4c431 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,3 @@ +numpy +scipy +opencv-python