nodes/banodoco-steerable-motion
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Welcome to the new world

Browse Source
This commit is contained in:
peteromallet
2024-04-08 02:28:20 +02:00
parent 717e291247
commit a6ff35af46
7 changed files with 2843 additions and 5023 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+12 -23
View File
@@ -1,8 +1,8 @@
# Steerable Motion, a ComfyUI custom node for steering videos with batches of images
Steerable Motion is a ComfyUI node for batch creative interpolation. Our goal is to feature the best methods for steering motion with images as video models evolve.
Steerable Motion is a ComfyUI node for batch creative interpolation. Our goal is to feature the best quality and most precise and powerful methods for steering motion with images as video models evolve. This node is best used via [Dough](https://github.com/banodoco/dough) - a creative tool which simplifies the settings and provides a nice creative flow.
![Main example](https://github.com/peteromallet/ComfyUI-Creative-Interpolation/blob/main/demo/main_example.gif)
![Main example](https://github.com/banodoco/steerable-motion/blob/main/demo/main_example.gif)
## Installation
@@ -16,35 +16,23 @@ Steerable Motion is a ComfyUI node for batch creative interpolation. Our goal is
The main settings are:
- Key frame position: how many frames to generate between each main key frame you provide.
- Length of influence: what range of frames to apply ControlNet (CN) and IP-Adapter (IPA) to.
- Strength of influence: how strong the control of IPA and CN should be.
- Relative IPA strength & influence: whether to make IPA's influence stronger or weaker than CN's.
- Length of influence: what range of frames to apply the IP-Adapter (IPA) influence to.
- Strength of influence: what the low-point and high-point of each frame should be.
- Image adherence: how much we should force adherence to the input images.
These are set linearly - the same for each frame - or dynamically - varying them for each frame - you can find detailed instructions on how to tweak these settings inside the workflow above.
Other than image adherence which is set for the entire generation these are set linearly - the same for each frame - or dynamically - varying them for each frame - you can find detailed instructions on how to tweak these settings inside the workflow above.
Tweaking the settings can greatly influence the motion - you can see two examples of the same images with slightly different settings below to get an idea of this - the settings are visualised in the graph:
Tweaking the settings can greatly influence the motion - for example, below you can see two examples of the same images animated - but with the one setting tweaked, the length of each frame's influence:
![Tweaking settings example](https://github.com/peteromallet/ComfyUI-Creative-Interpolation/blob/main/demo/tweaking_settings.gif)
This also works well for moving between dramatically different images - like in the example below:
![Different images example](https://github.com/peteromallet/ComfyUI-Creative-Interpolation/blob/main/demo/different_images.gif)
![Tweaking settings example](https://github.com/banodoco/steerable-motion/blob/main/demo/tweaking_settings.gif)
## Philosophy for getting the most from this
This isn’t a tool like text to video that will perform well out of the box - it’s more like a paint brush, an artistic tool that you need to figure out how to get the best from.
This isn’t a tool like text to video that will perform well out of the box, it’s more like a paint brush - an artistic tool that you need to figure out how to get the best from.
Through trial and error, you'll need to build an understanding of how the motion and settings work, what its limitations are, which inputs images work best with it, etc.
If you can figure out how to wield it, this approach can provide enough control for you to make beautiful things that match your imagination precisely.
## Coming Soon
- Implement more powerful video models and approaches to increase lengh, coherence, etc.
- Implement video ControlNets (upcoming from both Stability and Animatediff authors) for greater adherence to input images and better motion.
- Increase frame count to unlimited - currently limited to 12 due to RAM constraints with current approach.
- Fix issues with colour drain, etc.
- Implement LCM, Turbo, etc. to increase speed.
It won't work for everything but if you can figure out how to wield it, this approach can provide enough control for you to make beautiful things that match your imagination precisely.
## Want to give feedback, or join a community who are pushing open source models to their artistic and technical limits?
@@ -52,4 +40,5 @@ You're very welcome to drop into our Discord [here](https://discord.com/invite/8
## Credits
This code draws heavily from [Kosinkadink's ComfyUI-Advanced-ControlNet](https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet) and Cubiq's [IPAdapter_plus](https://github.com/cubiq/ComfyUI_IPAdapter_plus), while the workflows uses [Kosinkadink's Animatediff Evolved](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved), Fizzledorf's [Fizznodes](https://github.com/FizzleDorf/ComfyUI_FizzNodes), Fannovel16's [Frame Interpolation](https://github.com/Fannovel16/ComfyUI-Frame-Interpolation) and more. Thanks to all and of course the Animatediff team, Controlnet, others, and of course our supportive community!
This code draws heavily from Cubiq's [IPAdapter_plus](https://github.com/cubiq/ComfyUI_IPAdapter_plus), while the workflow uses Kosinkadink's [Animatediff Evolved](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved) and [ComfyUI-Advanced-ControlNet](https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet), Fizzledorf's [Fizznodes](https://github.com/FizzleDorf/ComfyUI_FizzNodes), Fannovel16's [Frame Interpolation](https://github.com/Fannovel16/ComfyUI-Frame-Interpolation) and more. Thanks to all and of course the Animatediff team, Controlnet, others, and of course our supportive community!