Author SHA1 Message Date
POM ac1ea3a682 Update SteerableMotion.py 2025-10-03 21:34:07 +02:00
POM 3c28833342 It 2025-06-27 16:56:26 +01:00
POM 9dc2753916 Update README.md: Split usage section into Wan and Animatediff approaches 2025-06-27 16:52:15 +01:00
POM cf7c6aaafd Rename sequence option strings and update logic accordingly 2025-06-23 16:12:56 +02:00
POM e0ce6ad576 Simplify VideoContinuationGenerator: Improve parameter naming and remove complex replacement logic
- Rename parameters for maximum clarity:
  • when_to_start_control_images → how_to_use_control_images
  • when_to_start_inpaint_masks → how_to_use_inpaint_masks
- Update option names to clearly indicate input frame prioritization:
  • beginning_of_generation → start_sequence_at_beginning_but_prioritise_input_frames
  • after_overlap_frames → start_sequence_after_overlap_frames_prioritise_input_frames
- Remove instead_of_input_frames option and all complex replacement logic
- Simplify to two clear options that always prioritize input overlap frames
- Improve tooltips to clearly explain prioritization behavior
- Focus on user-friendly workflows that preserve input video structure
2025-06-23 16:06:38 +02:00
POM 76264adfc7 Enhance VideoContinuationGenerator: Add instead_of_input_frames option and improve parameter naming
- Add 'instead_of_input_frames' option to both when_to_start_control_images and when_to_start_inpaint_masks
- Rename parameters for clarity: when_to_start_control_frames → when_to_start_control_images, when_to_start_masks → when_to_start_inpaint_masks
- Enable full control replacement: control images can now replace overlap frames + middle frames + end frame
- Improve inpaint mask logic: when_to_start_inpaint_masks now affects custom masks properly
- Add comprehensive logic for insufficient control images with graceful fallbacks
- Maintain backward compatibility while adding powerful new functionality
2025-06-23 15:58:54 +02:00
POM 8e2a9a8130 Fix inverted logic and improve tooltips for VideoContinuationGenerator control frame and mask parameters 2025-06-23 11:47:58 +02:00
POM 12bdbfbc68 Refactor: Correct and rename VideoContinuationGenerator start logic
- Rename 'beginning' to 'beginning_of_generation' for clarity
- Rename 'after overlap_frames' to 'after_overlap_frames'
- Swap underlying logic to match intuitive naming
- Set default to 'beginning_of_generation' for both parameters
- Update tooltips to accurately reflect the new behavior
2025-06-23 11:34:47 +02:00
POM 08cd413f0b Refactor VideoContinuationGenerator for clarity
- Replaces confusing 'when_to_start' parameters with a single 'continuation_mode' dropdown.
- New modes ('Generate new content', 'Stitch to existing sequence') are goal-oriented for better UX.
- 'Generate new content' (default) uses control frames from C0 and masks the middle for inpainting.
- 'Stitch to existing sequence' skips overlapping control frames and treats them as known areas.
- This change makes the node's behavior more intuitive and easier to understand.
2025-06-23 11:27:08 +02:00
POM cfd1d8b256 Add when_to_start_masks parameter to VideoContinuationGenerator
- Adds dropdown with 'beginning' and 'after overlap_frames' options
- Controls when mask generation begins in the continuation video
- 'beginning' mode: masks start from frame 0 (standard behavior)
- 'after overlap_frames' mode: masks follow control frame timing logic
- Provides consistent mask/control frame alignment for better results
2025-06-23 11:11:07 +02:00
POM 9ed5f0b528 Add when_to_start_control_frames parameter to VideoContinuationGenerator
- Adds dropdown with 'beginning' and 'after overlap_frames' options
- Controls when control frames become active in the continuation video
- 'beginning' mode starts from first control frame regardless of overlap
- 'after overlap_frames' mode skips overlap count to avoid duplication
- Includes proper logic and tooltip explanation
2025-06-23 11:05:39 +02:00
POM a73c7b9759 Update SteerableMotion.py 2025-06-22 22:41:28 +02:00
POM b95e9eed09 Add files via upload 2024-06-16 01:01:54 +02:00
POM d747e46b21 Update README.md 2024-05-30 06:45:03 -07:00
POM 122f9778c6 Add files via upload 2024-05-30 15:44:22 +02:00
POM 53bfbd85c7 Delete demo/creative_interpolation_example.json 2024-05-30 06:43:14 -07:00
POM 13f6a58723 Delete demo/steerable-motion_liquidy_loop.json 2024-05-30 06:43:06 -07:00
POM f9250d62ee Add files via upload 2024-05-30 15:42:47 +02:00
POM 46742f3e4e Merge pull request #76 from banodoco/feature/interpolate
Fix SparseCtrl Issue
2024-05-30 13:49:53 +02:00
POM e9c92e78c5 Add files via upload 2024-05-27 01:30:09 +02:00
POM ab32b862c6 Add files via upload 2024-05-27 01:25:31 +02:00
POM 1e796ed917 Delete demo/steerable-moton_chocky-realistiche.json 2024-05-27 01:21:23 +02:00
POM ae350a6e7f Delete demo/steerable-motion_liquity_loop.json 2024-05-27 01:21:15 +02:00
POM a7050f41dd Add files via upload 2024-05-27 01:20:56 +02:00
POM 1311a0859d Add files via upload 2024-05-27 01:17:00 +02:00
POM 09e5797b0a Update README.md 2024-05-27 01:16:21 +02:00
POM e3c1031117 Add files via upload 2024-05-25 23:46:27 +02:00
POM 8339ca5368 Merge pull request #75 from banodoco/feature/interpolate
Feature/interpolate
2024-05-25 23:43:55 +02:00
11 changed files with 32903 additions and 1470 deletions
+35 -34
View File
@@ -1,55 +1,55 @@
# Steerable Motion, a ComfyUI custom node for steering videos with batches of images
# Steerable Motion, ComfyUI custom nodes & workflows node for steering videos with batches of images
Steerable Motion is a ComfyUI node for batch creative interpolation. Our goal is to feature the best quality and most precise and powerful methods for steering motion with images as video models evolve. This node is best used via [Dough](https://github.com/banodoco/dough) - a creative tool which simplifies the settings and provides a nice creative flow - or in Discord - by joining this channel.
![Main example](https://github.com/banodoco/steerable-motion/blob/main/demo/main_example.gif)
Steerable Motion is a set of ComfyUI nodes and workflows for travelling between images.
## Installation in Comfy
1. If you haven't already, install [ComfyUI](https://github.com/comfyanonymous/ComfyUI) and [Comfy Manager](https://github.com/ltdrdata/ComfyUI-Manager) - you can find instructions on their pages.
2. Download [this workflow](https://raw.githubusercontent.com/banodoco/steerable-motion/main/demo/creative_interpolation_example.json) and drop it into ComfyUI - or you can use one of the workflows others in the community made below.
3. When the workflow opens, download the dependent nodes by pressing "Install Missing Custom Nodes" in Comfy Manager. Search and download the required models from Comfy Manager also - make sure that the models you download have the same name as the ones in the workflow - or you're confident that they're the same.
2. When the workflow opens, download the dependent nodes by pressing "Install Missing Custom Nodes" in Comfy Manager. Search and download the required models from Comfy Manager also.
## Usage
## Wan
The main settings are:
The Wan approach uses VACE to create anchor images and continuations from previous images, which are chained together at the end:
- Key frame position: how many frames to generate between each main key frame you provide.
- Length of influence: what range of frames to apply the IP-Adapter (IPA) influence to.
- Strength of influence: what the low-point and high-point of each frame should be.
- Image adherence: how much we should force adherence to the input images.
![Main example](demo/wan_example.gif)
Other than image adherence which is set for the entire generation these are set linearly - the same for each frame - or dynamically - varying them for each frame - you can find detailed instructions on how to tweak these settings inside the workflow above.
Tweaking the settings can greatly influence the motion - for example, below you can see two examples of the same images animated - but with the one setting tweaked, the length of each frame's influence:
### Sample workflow for Wan
![Tweaking settings example](https://github.com/banodoco/steerable-motion/blob/main/demo/tweaking_settings.gif)
You can find a workflow [here](demo/Vace_Travel.json) to get started.
## Philosophy for getting the most from this
## Animatediff
This isn’t a tool like text to video that will perform well out of the box, it’s more like a paint brush - an artistic tool that you need to figure out how to get the best from.
The Animatediff approach uses a combination of IP-Adapter and SparseCtrl to travel between images:
![Main example](demo/main_example.gif)
### 5 basic workflows for Animatediff
Below are 5 basic workflows - each with their own weird and unique characteristics - all with differing levels of adherence and different types of motion - most of the changes come from tweaking the IPA configuration and switching out base models:
- [Smooth n' Steady](https://raw.githubusercontent.com/banodoco/steerable-motion/main/demo/steerable-motion_smooth-n-steady.json): tends to have nice smooth motion - good starting point
- [Rad Attack](https://raw.githubusercontent.com/banodoco/steerable-motion/main/demo/steerable-motion_rad-attack.json): probably the best for realistic motion
- [Slurshy Realistiche](https://raw.githubusercontent.com/banodoco/steerable-motion/main/demo/steerable-motion_slurshy-realistiche.json): moves in a slightly realistic manner but is a little bit slurshy
- [Chocky Realistiche](https://raw.githubusercontent.com/banodoco/steerable-motion/main/demo/steerable-motion_chocky-realistiche.json): realistic-ish but very blocky
- [Liquidy Loop](https://raw.githubusercontent.com/banodoco/steerable-motion/main/demo/steerable-motion_liquidy-loop.json): smooth and liquidy
You can see each in acton below:
![basic workflows](demo/basic_workflows.gif)
## Philosophy for getting the most from these
This isn't an approach like text to video that will perform well out of the box, it's more like a paint brush - an artistic tool that you need to figure out how to get the best from.
Through trial and error, you'll need to build an understanding of how the motion and settings work, what its limitations are, which inputs images work best with it, etc.
It won't work for everything but if you can figure out how to wield it, this approach can provide enough control for you to make beautiful things that match your imagination precisely.
## 2 examples of workflows others have built on top of this:
In both cases, tweaking the settings can greatly influence the motion - for example, below you can see two examples of the same images animated - but with the one setting tweaked, the length of each frame's influence:
The workflow I shared is just one way to use this - below are two other workflows people in our community have created on top of this node:
### Looped LCM by @idgallagher
First, [@idgallagher](https://twitter.com/idgallagher) uses LCM and different settings to achieve a really interesting realistic motion effect. You can grab it [here](https://github.com/IDGallagher/storage/blob/main/chiff_distilled_sm.json) and see an example output here:
![Flipping Sigmas](https://github.com/banodoco/steerable-motion/blob/main/demo/flipping_sigmas.gif)
### Smooth & Deep by @Superbeasts.ai:
Next, [Superbeasts.ai](https://www.instagram.com/superbeasts) uses depth maps to control the motion in different layers - creating a smoother motion effect. You can grab this workflow [here](https://github.com/banodoco/Steerable-Motion/blob/main/demo/SuperBeasts-POM-SmoothBatchCreative-V1.3.1.json) and see an example of it in action here:
![Superbeasts Example](https://github.com/banodoco/steerable-motion/blob/main/demo/superbeasts.gif)
I believe that that there are endless ways to expand upon and extend the ideas in this node - if you do anything cool, please share!
![Tweaking settings example](demo/tweaking_settings.gif)
## Want to give feedback, or join a community who are pushing open source models to their artistic and technical limits?
@@ -57,5 +57,6 @@ You're very welcome to drop into our Discord [here](https://discord.com/invite/8
## Credits
This code draws heavily from Cubiq's [IPAdapter_plus](https://github.com/cubiq/ComfyUI_IPAdapter_plus), while the workflow uses Kosinkadink's [Animatediff Evolved](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved) and [ComfyUI-Advanced-ControlNet](https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet), Fizzledorf's [Fizznodes](https://github.com/FizzleDorf/ComfyUI_FizzNodes), Fannovel16's [Frame Interpolation](https://github.com/Fannovel16/ComfyUI-Frame-Interpolation) and more. Thanks to all and of course the Animatediff team, Controlnet, others, and of course our supportive community!
For Animatediff, the code draws heavily from Cubiq's [IPAdapter_plus](https://github.com/cubiq/ComfyUI_IPAdapter_plus), while the workflow uses Kosinkadink's [Animatediff Evolved](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved) and [ComfyUI-Advanced-ControlNet](https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet), Fizzledorf's [Fizznodes](https://github.com/FizzleDorf/ComfyUI_FizzNodes), Fannovel16's [Frame Interpolation](https://github.com/Fannovel16/ComfyUI-Frame-Interpolation) and more.
For Wan, it's built on top of the work of Kijai's wonderful [ComfyUI-WanVideoWrapper](https://github.com/kijai/ComfyUI-WanVideoWrapper) and of course the VACE and Wan teams.
+422 -5
View File
@@ -1,17 +1,27 @@
# Standard library imports
from ast import literal_eval
from io import BytesIO
import numpy as np
import logging
import math
import gc
# Third-party library imports
import numpy as np
import torch
import torchvision.transforms as transforms
from PIL import Image
import matplotlib
import matplotlib.pyplot as plt
# Local application/library specific imports
from .imports.ComfyUI_IPAdapter_plus.IPAdapterPlus import IPAdapterBatchImport, IPAdapterTiledBatchImport, IPAdapterTiledImport, PrepImageForClipVisionImport, IPAdapterAdvancedImport, IPAdapterNoiseImport
from .imports.ComfyUI_Frame_Interpolation.vfi_models.film import FILM_VFIImport
import matplotlib
import gc
from comfy.utils import common_upscale
try:
from .utils import log # If your .utils has a log object
except ImportError:
log = logging.getLogger(__name__) # Fallback to standard logging
class BatchCreativeInterpolationNode:
@classmethod
@@ -642,15 +652,422 @@ class IpaConfigurationNode:
"noise_blur": noise_blur,
},
class VideoFrameExtractorAndMaskGenerator:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"input_video_frames": ("IMAGE", {"tooltip": "Input video frames (IMAGE batch) to extract from."}),
"total_output_frames": ("INT", {"default": 81, "min": 1, "max": 10000, "step": 4, "tooltip": "Total number of frames for the output guidance video and masks. Must satisfy: (frames - 1) divisible by 4."}),
"frame_selection_string": ("STRING", {"default": "0, 10:20", "multiline": False, "tooltip": "Comma-separated integers or ranges (e.g., 0, 5, 10:15, 20) of frames to extract from input video. Takes precedence over depth_frames."}),
"empty_frame_fill_level": ("FLOAT", {"default": 0.5, "min": 0.0, "max": 1.0, "step": 0.01, "tooltip": "Grayscale level (0.0 black, 1.0 white) for frames not explicitly selected or filled by depth."}),
},
"optional": {
"depth_video_frames": ("IMAGE", {"tooltip": "Optional depth frames (IMAGE batch). Placed if the slot is not already filled by frame_selection_string."}),
"master_inpaint_mask": ("MASK", {"tooltip": "Optional master inpaint mask. If provided, it defines the entire output mask, overriding masks for selected/depth frames."}),
},
}
RETURN_TYPES = ("IMAGE", "MASK",)
RETURN_NAMES = ("guidance_video_frames", "guidance_frame_masks",)
FUNCTION = "extract_frames_and_generate_masks"
CATEGORY = "Steerable-Motion"
DESCRIPTION = "Extracts/places frames from input/depth video into a new guidance video and generates corresponding masks. frame_selection_string takes precedence over depth frames."
def _parse_frame_selection_string(self, selection_string, max_frame_index_from_input):
selected_frame_indices = set()
selection_parts = selection_string.split(',')
for part in selection_parts:
part = part.strip()
if not part:
continue
if ':' in part:
try:
start_str, end_str = part.split(':')
start_frame = int(start_str)
end_frame = int(end_str)
if start_frame < 0 or end_frame < 0:
log.warning(f"Frame indices cannot be negative in '{part}'. Skipping.")
continue
if start_frame > end_frame:
log.warning(f"Range start {start_frame} is greater than end {end_frame} in '{part}'. Swapping.")
start_frame, end_frame = end_frame, start_frame
for frame_idx in range(start_frame, end_frame + 1): # Inclusive range
if 0 <= frame_idx <= max_frame_index_from_input:
selected_frame_indices.add(frame_idx)
else:
log.warning(f"Frame index {frame_idx} from range '{part}' is out of bounds for input video (0-{max_frame_index_from_input}). Skipping this specific index.")
except ValueError:
log.error(f"Invalid range format '{part}'. Skipping.")
else:
try:
frame_idx = int(part)
if frame_idx < 0:
log.warning(f"Frame index {frame_idx} cannot be negative. Skipping.")
continue
if 0 <= frame_idx <= max_frame_index_from_input:
selected_frame_indices.add(frame_idx)
else:
log.warning(f"Frame index {frame_idx} is out of bounds for input video (0-{max_frame_index_from_input}). Skipping.")
except ValueError:
log.error(f"Invalid frame index '{part}'. Skipping.")
return sorted(list(selected_frame_indices))
def extract_frames_and_generate_masks(self, input_video_frames, total_output_frames, frame_selection_string, empty_frame_fill_level, depth_video_frames=None, master_inpaint_mask=None):
# Convert string parameter to integer
total_output_frames = int(total_output_frames)
if (total_output_frames - 1) % 4 != 0:
raise ValueError("total_output_frames must satisfy (frames - 1) divisible by 4")
if input_video_frames is None or input_video_frames.shape[0] == 0:
log.error("Input video_frames is empty. Cannot proceed.")
dummy_height, dummy_width, dummy_channels = 64, 64, 3
return (torch.zeros((total_output_frames, dummy_height, dummy_width, dummy_channels), dtype=torch.float32),
torch.ones((total_output_frames, dummy_height, dummy_width), dtype=torch.float32))
device = input_video_frames.device
dtype = input_video_frames.dtype
batch_size_input, frame_height, frame_width, num_channels = input_video_frames.shape
max_input_frame_index = batch_size_input - 1
# Initialize guidance video with empty_frame_fill_level
guidance_video_output = torch.ones((total_output_frames, frame_height, frame_width, num_channels), device=device, dtype=dtype) * empty_frame_fill_level
# Initialize base masks: 1 for unknown/inpaint, 0 for known
base_frame_masks = torch.ones((total_output_frames, frame_height, frame_width), device=device, dtype=dtype)
# 1. Process frame_selection_string (highest priority)
selected_input_frame_indices = self._parse_frame_selection_string(frame_selection_string, max_input_frame_index)
log.info(f"Frames selected by 'frame_selection_string': {selected_input_frame_indices}")
for input_frame_index in selected_input_frame_indices:
# The selected_input_frame_indices are indices from the *input_video*.
# We place them at the *same index* in the output guidance_video if that index is valid.
target_output_frame_index = input_frame_index
if target_output_frame_index < total_output_frames:
guidance_video_output[target_output_frame_index] = input_video_frames[input_frame_index].clone()
base_frame_masks[target_output_frame_index] = 0.0 # This frame is now known and prioritized
log.debug(f"Placed frame {input_frame_index} from input_video_frames into guidance_video at index {target_output_frame_index}.")
else:
log.warning(f"Selected frame index {input_frame_index} from input video maps to target index {target_output_frame_index}, which is >= total_output_frames ({total_output_frames}). It won't be placed.")
# 2. Process depth_video_frames (second priority)
if depth_video_frames is not None and depth_video_frames.shape[0] > 0:
log.info(f"Processing {depth_video_frames.shape[0]} depth_video_frames.")
processed_depth_frames = depth_video_frames.clone().to(device=device, dtype=dtype)
# Resize depth_video_frames if their dimensions don't match input_video_frames
if processed_depth_frames.shape[1:] != (frame_height, frame_width, num_channels):
log.info(f"Resizing depth_video_frames from {processed_depth_frames.shape[1:]} to {(frame_height, frame_width, num_channels)} to match input_video_frames.")
resized_depth_frame_list = []
for frame_idx in range(processed_depth_frames.shape[0]):
# common_upscale expects (B, C, H, W) or (B, H, W)
# IMAGE is (B,H,W,C), so permute, upscale, permute back
frame_to_resize = processed_depth_frames[frame_idx:frame_idx+1].permute(0, 3, 1, 2) # (1, C, H_depth, W_depth)
resized_frame = common_upscale(frame_to_resize, frame_width, frame_height, "lanczos", "disabled") # (1, C, H, W)
resized_depth_frame_list.append(resized_frame.permute(0, 2, 3, 1)) # (1, H, W, C)
processed_depth_frames = torch.cat(resized_depth_frame_list, dim=0)
num_depth_frames_to_place = min(processed_depth_frames.shape[0], total_output_frames)
for frame_idx in range(num_depth_frames_to_place):
# Check if this slot in guidance_video is still an "empty" placeholder
# (i.e., its mask is still 1.0, meaning not filled by frame_selection_string)
if base_frame_masks[frame_idx].mean() > 0.99: # Check if it's still (mostly) 1.0
guidance_video_output[frame_idx] = processed_depth_frames[frame_idx].clone()
# Keep mask as 1.0 for depth frames (inpaint area) - don't set to 0.0
log.debug(f"Placed frame {frame_idx} from depth_video_frames into guidance_video at index {frame_idx} (keeping as inpaint area).")
else:
log.debug(f"Skipping depth_frame {frame_idx} as guidance_video index {frame_idx} was already filled by frame_selection_string.")
else:
log.info("No depth_video_frames provided or depth_video_frames is empty.")
# 3. Handle optional master_inpaint_mask (this will override base_frame_masks if provided)
final_frame_masks = base_frame_masks
if master_inpaint_mask is not None:
log.info("Processing provided master_inpaint_mask. This will override masks derived from frame/depth selection.")
processed_master_mask = master_inpaint_mask.clone().to(device=device, dtype=dtype)
if processed_master_mask.shape[1:] != (frame_height, frame_width):
log.info(f"Resizing master_inpaint_mask from {processed_master_mask.shape[1:]} to {(frame_height, frame_width)}.")
processed_master_mask = common_upscale(
processed_master_mask.unsqueeze(1),
frame_width, frame_height, "nearest-exact", "disabled"
).squeeze(1)
if processed_master_mask.shape[0] != total_output_frames:
log.info(f"Adjusting master_inpaint_mask frame count from {processed_master_mask.shape[0]} to {total_output_frames}.")
if processed_master_mask.shape[0] == 0:
log.error("Received an empty master_inpaint_mask after processing. Using base masks.")
elif processed_master_mask.shape[0] < total_output_frames:
num_mask_repeats = (total_output_frames + processed_master_mask.shape[0] - 1) // processed_master_mask.shape[0]
processed_master_mask = processed_master_mask.repeat(num_mask_repeats, 1, 1)[:total_output_frames]
else:
processed_master_mask = processed_master_mask[:total_output_frames]
final_frame_masks = processed_master_mask
return (guidance_video_output.cpu().float(), final_frame_masks.cpu().float())
class VideoContinuationGenerator:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"input_video_frames": ("IMAGE", {"tooltip": "Input video frames to create continuation from."}),
"total_output_frames": ("INT", {"default": 81, "min": 1, "max": 10000, "step": 4, "tooltip": "Total number of frames for the output continuation video. Must satisfy: (frames - 1) divisible by 4."}),
"overlap_frames": ("INT", {"default": 3, "min": 1, "max": 50, "step": 1, "tooltip": "Number of frames from the end of input video to use as overlap at the start."}),
"empty_frame_fill_level": ("FLOAT", {"default": 0.5, "min": 0.0, "max": 1.0, "step": 0.01, "tooltip": "Grayscale level (0.0 black, 1.0 white) for empty continuation frames."}),
},
"optional": {
"end_frame": ("IMAGE", {"tooltip": "Optional single frame to place at the end of the continuation video."}),
"control_images": ("IMAGE", {"tooltip": "Optional control images to fill the empty frames."}),
"inpaint_mask": ("MASK", {"tooltip": "Optional inpaint mask to use for the empty frames, overriding the default mask."}),
"how_to_use_control_images": (["start_sequence_at_beginning_and_prioritise_input_frames", "start_sequence_after_overlap_frames_and_prioritise_input_frames"], {"default": "start_sequence_at_beginning_and_prioritise_input_frames", "tooltip": "If start_sequence_at_beginning_and_prioritise_input_frames is selected, control images align with frame 0 but input overlap frames take priority, so control images become visible after the overlap period. If start_sequence_after_overlap_frames_and_prioritise_input_frames is selected, control images start being placed after the overlap frames from the input video."}),
"how_to_use_inpaint_masks": (["start_sequence_at_beginning_and_prioritise_input_frames", "start_sequence_after_overlap_frames_and_prioritise_input_frames"], {"default": "start_sequence_at_beginning_and_prioritise_input_frames", "tooltip": "If start_sequence_at_beginning_and_prioritise_input_frames is selected, inpaint masks align with frame 0 but preserve input overlap frames as known. If start_sequence_after_overlap_frames_and_prioritise_input_frames is selected, inpaint masks only affect frames after the overlap period."}),
},
}
RETURN_TYPES = ("IMAGE", "MASK",)
RETURN_NAMES = ("continuation_video_frames", "continuation_frame_masks",)
FUNCTION = "generate_continuation_video"
CATEGORY = "Steerable-Motion"
DESCRIPTION = "Creates a continuation video by placing overlap frames from the end of input video at the start, with optional end frame."
def generate_continuation_video(self, input_video_frames, total_output_frames, overlap_frames, empty_frame_fill_level, end_frame=None, control_images=None, inpaint_mask=None, how_to_use_control_images="start_sequence_at_beginning_and_prioritise_input_frames", how_to_use_inpaint_masks="start_sequence_at_beginning_and_prioritise_input_frames"):
# 1. Validation and Setup
total_output_frames = int(total_output_frames)
if (total_output_frames - 1) % 4 != 0:
raise ValueError("total_output_frames must satisfy (frames - 1) divisible by 4")
if input_video_frames is None or input_video_frames.shape[0] == 0:
log.error("Input video_frames is empty. Cannot proceed.")
dummy_height, dummy_width, dummy_channels = 64, 64, 3
return (torch.zeros((total_output_frames, dummy_height, dummy_width, dummy_channels), dtype=torch.float32),
torch.ones((total_output_frames, dummy_height, dummy_width), dtype=torch.float32))
device = input_video_frames.device
dtype = input_video_frames.dtype
batch_size_input, frame_height, frame_width, num_channels = input_video_frames.shape
# 2. Prepare Start Frames (from overlap)
actual_overlap_frames = min(overlap_frames, batch_size_input, total_output_frames)
if actual_overlap_frames < overlap_frames:
log.warning(f"Requested {overlap_frames} overlap frames but input video only has {batch_size_input} frames or total output is smaller. Using {actual_overlap_frames} instead.")
overlap_start_idx = batch_size_input - actual_overlap_frames
start_frames_part = input_video_frames[overlap_start_idx : overlap_start_idx + actual_overlap_frames].clone()
# 3. Prepare End Frame
end_frame_part = torch.empty((0, frame_height, frame_width, num_channels), device=device, dtype=dtype)
num_end_frames = 0
if end_frame is not None and end_frame.shape[0] > 0 and total_output_frames > actual_overlap_frames:
num_end_frames = 1
end_frame_processed = end_frame[0].clone().to(device=device, dtype=dtype)
if end_frame_processed.shape != (frame_height, frame_width, num_channels):
log.info(f"Resizing end_frame from {end_frame_processed.shape} to {(frame_height, frame_width, num_channels)}.")
frame_to_resize = end_frame_processed.unsqueeze(0).permute(0, 3, 1, 2)
resized_frame = common_upscale(frame_to_resize, frame_width, frame_height, "lanczos", "disabled")
end_frame_processed = resized_frame.permute(0, 2, 3, 1).squeeze(0)
end_frame_part = end_frame_processed.unsqueeze(0)
# 4. Prepare Middle Frames
num_middle_frames = total_output_frames - actual_overlap_frames - num_end_frames
middle_frames_part = torch.empty((0, frame_height, frame_width, num_channels), device=device, dtype=dtype)
if num_middle_frames > 0:
if control_images is not None:
log.info(f"Using 'control_images' to fill the {num_middle_frames} middle frames with '{how_to_use_control_images}' mode.")
control_images_resized = common_upscale(control_images.movedim(-1, 1), frame_width, frame_height, "lanczos", "disabled").movedim(1, -1)
if how_to_use_control_images == "start_sequence_at_beginning_and_prioritise_input_frames":
# Skip the first overlap_frames control images to avoid duplication
duplicate_count = min(actual_overlap_frames, control_images_resized.shape[0])
available_after_dup = control_images_resized.shape[0] - duplicate_count
if available_after_dup < num_middle_frames:
log.info(f"After skipping {duplicate_count} control frames, only {available_after_dup} remain; padding {num_middle_frames - available_after_dup} frames with 'empty_frame_fill_level'.")
selected_control = control_images_resized[duplicate_count:]
padding_needed = num_middle_frames - selected_control.shape[0]
padding = torch.ones((padding_needed, frame_height, frame_width, num_channels), device=device, dtype=dtype) * empty_frame_fill_level
middle_frames_part = torch.cat([selected_control, padding], dim=0)
else:
middle_frames_part = control_images_resized[duplicate_count:duplicate_count + num_middle_frames].clone()
else: # "start_sequence_after_overlap_frames_and_prioritise_input_frames"
# Use control frames from the beginning of the sequence (C0, C1, C2...)
if control_images_resized.shape[0] < num_middle_frames:
log.warning(f"Provided 'control_images' have {control_images_resized.shape[0]} frames, less than needed ({num_middle_frames}). Padding with 'empty_frame_fill_level'.")
padding_needed = num_middle_frames - control_images_resized.shape[0]
padding = torch.ones((padding_needed, frame_height, frame_width, num_channels), device=device, dtype=dtype) * empty_frame_fill_level
middle_frames_part = torch.cat([control_images_resized, padding], dim=0)
else:
middle_frames_part = control_images_resized[:num_middle_frames].clone()
else:
log.info(f"No 'control_images', filling {num_middle_frames} middle frames with level {empty_frame_fill_level}.")
middle_frames_part = torch.ones((num_middle_frames, frame_height, frame_width, num_channels), device=device, dtype=dtype) * empty_frame_fill_level
# 5. Assemble Final Video
continuation_video_output = torch.cat([start_frames_part, middle_frames_part, end_frame_part], dim=0)
# 6. Create Mask
continuation_frame_masks = torch.ones((total_output_frames, frame_height, frame_width), device=device, dtype=dtype)
# Apply mask logic based on how_to_use_inpaint_masks parameter
if how_to_use_inpaint_masks == "start_sequence_at_beginning_and_prioritise_input_frames":
# Set known frames (overlap and end) to 0.0, but also set middle section based on control frame logic
if actual_overlap_frames > 0:
continuation_frame_masks[0:actual_overlap_frames] = 0.0
if num_end_frames > 0:
continuation_frame_masks[-num_end_frames:] = 0.0
# For middle section, follow the same logic as control frames
if control_images is not None and num_middle_frames > 0:
duplicate_count = min(actual_overlap_frames, control_images.shape[0])
available_after_dup = control_images.shape[0] - duplicate_count
if available_after_dup >= num_middle_frames:
# If we have enough control frames after skipping, set those middle frames as known (0.0)
middle_start = actual_overlap_frames
middle_end = middle_start + num_middle_frames
continuation_frame_masks[middle_start:middle_end] = 0.0
else: # "start_sequence_after_overlap_frames_and_prioritise_input_frames"
# Set known frames (overlap and end) to 0.0, rest stay as 1.0 (inpaint)
if actual_overlap_frames > 0:
continuation_frame_masks[0:actual_overlap_frames] = 0.0
if num_end_frames > 0:
continuation_frame_masks[-num_end_frames:] = 0.0
# 7. Handle optional inpaint_mask with how_to_use_inpaint_masks logic
if inpaint_mask is not None:
log.info(f"Processing provided 'inpaint_mask' with '{how_to_use_inpaint_masks}' timing.")
processed_mask = common_upscale(inpaint_mask.unsqueeze(1), frame_width, frame_height, "nearest-exact", "disabled").squeeze(1).to(device)
if processed_mask.shape[0] != total_output_frames:
log.info(f"Adjusting inpaint_mask frame count from {processed_mask.shape[0]} to {total_output_frames}.")
if processed_mask.shape[0] < total_output_frames:
num_repeats = (total_output_frames + processed_mask.shape[0] - 1) // processed_mask.shape[0]
processed_mask = processed_mask.repeat(num_repeats, 1, 1)[:total_output_frames]
else:
processed_mask = processed_mask[:total_output_frames]
# Apply how_to_use_inpaint_masks logic to the provided mask
if how_to_use_inpaint_masks == "start_sequence_at_beginning_and_prioritise_input_frames":
# Use the provided mask as-is, but preserve known frames (overlap and end)
if actual_overlap_frames > 0:
processed_mask[0:actual_overlap_frames] = 0.0 # Keep overlap frames as known
if num_end_frames > 0:
processed_mask[-num_end_frames:] = 0.0 # Keep end frame as known
else: # "start_sequence_after_overlap_frames_and_prioritise_input_frames"
# Only apply the provided mask after overlap frames
if actual_overlap_frames > 0:
processed_mask[0:actual_overlap_frames] = 0.0 # Keep overlap frames as known
# The provided mask affects frames starting after overlap
if num_end_frames > 0:
processed_mask[-num_end_frames:] = 0.0 # Keep end frame as known
continuation_frame_masks = processed_mask.to(dtype=dtype)
log.info(f"Generated continuation video. Start: {actual_overlap_frames} frames, Middle: {num_middle_frames} frames, End: {num_end_frames} frames.")
return (continuation_video_output.cpu().float(), continuation_frame_masks.cpu().float())
class WanInputFrameNumber:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"frame_number": ("INT", {"default": 81, "min": 1, "max": 10000, "step": 4, "tooltip": "Frame number where (frames - 1) is divisible by 4."}),
},
}
RETURN_TYPES = ("INT",)
RETURN_NAMES = ("frame_number",)
FUNCTION = "get_frame_number"
CATEGORY = "Steerable-Motion"
DESCRIPTION = "Outputs a frame number that satisfies the WAN constraint: (frames - 1) divisible by 4."
def get_frame_number(self, frame_number):
frame_number = int(frame_number)
if (frame_number - 1) % 4 != 0:
raise ValueError("frame_number must satisfy (frame_number - 1) divisible by 4")
return (frame_number,)
class WanVideoBlenderNode:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"overlap_frames": ("INT", {"default": 10, "min": 1, "max": 1000, "step": 1}),
"video_1": ("IMAGE",),
"video_2": ("IMAGE",),
},
}
RETURN_TYPES = ("IMAGE",)
RETURN_NAMES = ("blended_video_frames",)
FUNCTION = "blend_videos"
CATEGORY = "Steerable-Motion"
DESCRIPTION = "Blends two input videos with a cross-fade. The resolution of the second clip is resized to match the first."
def _resize_video(self, video, target_height, target_width):
"""Resize a batch of frames (B,H,W,C) to (target_height,target_width) using Lanczos."""
if video.shape[1] == target_height and video.shape[2] == target_width:
return video
# (B, H, W, C) -> (B, C, H, W)
video_permuted = video.permute(0, 3, 1, 2)
resized = common_upscale(video_permuted, target_width, target_height, "lanczos", "disabled") # (B, C, H, W)
return resized.permute(0, 2, 3, 1)
def _cross_fade(self, tail, head, overlap_frames):
"""Blend two tensors of shape (overlap_frames,H,W,C) using linear alpha."""
device, dtype = tail.device, tail.dtype
alphas = torch.linspace(0, 1, overlap_frames, device=device, dtype=dtype).view(-1, 1, 1, 1)
blended = tail * (1 - alphas) + head * alphas
return blended
def blend_videos(self, overlap_frames, video_1, video_2):
if video_1 is None or video_2 is None:
raise ValueError("Both video_1 and video_2 are required.")
# Reference dimensions and properties from first video
ref_h, ref_w = video_1.shape[1:3]
# Ensure second video matches size
video_2_resized = self._resize_video(video_2, ref_h, ref_w)
if video_1.shape[0] < overlap_frames or video_2_resized.shape[0] < overlap_frames:
raise ValueError(f"One of the videos is shorter than overlap_frames={overlap_frames}.")
# Extract segments for blending
tail = video_1[-overlap_frames:]
head = video_2_resized[:overlap_frames]
blended = self._cross_fade(tail, head, overlap_frames)
# Assemble new timeline
final_video = torch.cat([
video_1[:-overlap_frames],
blended,
video_2_resized[overlap_frames:]
], dim=0)
return (final_video.cpu().float(),)
# NODE MAPPING
NODE_CLASS_MAPPINGS = {
"BatchCreativeInterpolation": BatchCreativeInterpolationNode,
"IpaConfiguration": IpaConfigurationNode,
"RemoveAndInterpolateFrames": RemoveAndInterpolateFramesNode,
"VideoFrameExtractorAndMaskGenerator": VideoFrameExtractorAndMaskGenerator,
"VideoContinuationGenerator": VideoContinuationGenerator,
"WanInputFrameNumber": WanInputFrameNumber,
"WanVideoBlender": WanVideoBlenderNode,
}
NODE_DISPLAY_NAME_MAPPINGS = {
"BatchCreativeInterpolation": "Batch Creative Interpolation 🎞️🅢🅜",
"IpaConfiguration": "IPA Configuration 🎞️🅢🅜",
"IpaConfiguration": "IP-Adapter Configuration 🎞️🅢🅜",
"RemoveAndInterpolateFrames": "Remove and Interpolate Frames 🎞️🅢🅜",
}
"VideoFrameExtractorAndMaskGenerator": "Video Frame Extractor & Mask Generator 🎞️🅢🅜",
"VideoContinuationGenerator": "Video Continuation Generator 🎞️🅢🅜",
"WanInputFrameNumber": "WAN Input Frame Number 🎞️🅢🅜",
"WanVideoBlender": "WAN Video Blender 🎞️🅢🅜",
}
File diff suppressed because it is too large Load Diff
Binary file not shown.

After

Width:  |  Height:  |  Size: 16 MiB

File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
Binary file not shown.

After

Width:  |  Height:  |  Size: 21 MiB