Update README.md
This commit is contained in:
POM
@@ -1,66 +1,54 @@
|
||||
# Steerable Motion - ComfyUI node for creative interpolation and other methods for controlling Animatediff (Beta)
|
||||
# Steerable Motion, a ComfyUI custom node for steering videos with batches of images
|
||||
|
||||
This a ComfyUI node for batch creative interpolation. The goal is to allow you to input a batch of images, and to provide a range of simple settings to control how the images are interpolated between.
|
||||
Steerable Motion is a ComfyUI node for batch creative interpolation. Our goal is to feature the best methods for steering motion with images as video models evolve.
|
||||
|
||||
|
||||
|
||||
## Installation
|
||||
|
||||
1. If you haven't already, download [ComfyUI](https://github.com/comfyanonymous/ComfyUI) and [Comfy Manager](https://github.com/ltdrdata/ComfyUI-Manager).
|
||||
1. go to your custom_nodes folder and run: git clone https://github.com/peteromallet/ComfyUI-Creative-Interpolation.git
|
||||
2. Download Controlnet tile from Comfy Manager: control_v11f1e_sd15_tile_fp16.safetensors
|
||||
1. If you haven't already, install [ComfyUI](https://github.com/comfyanonymous/ComfyUI) and [Comfy Manager](https://github.com/ltdrdata/ComfyUI-Manager) - you can find instructions on their pages.
|
||||
2. Search "Steerable Motion" in Comfy Manager and download the node.
|
||||
3. Download [this workflow](https://raw.githubusercontent.com/banodoco/steerable-motion/main/demo/creative_interpolation_example.json) and drop it into ComfyUI.
|
||||
4. When the workflow opens, download the dependent nodes by pressing "Install Missing Custom Nodes" in Comfy Manager. Search and download the required models from Comfy Manager also - make sure that the models you download have the same name as the ones in the workflow - or you're confident that they're the same.
|
||||
|
||||
## Usage
|
||||
|
||||
Here's a workflow to get started with: https://raw.githubusercontent.com/peteromallet/ComfyUI-Creative-Interpolation/main/demo/creative_interpolation_example.json
|
||||
|
||||
You'll need to drop the input images into the 'creative_interpolation_input' folder in numerical order - 0.png, 1.png, etc.
|
||||
|
||||
Key frames are **distributed either linearly of dynamically**.
|
||||
|
||||
If you set type_of_frame_distribution to linear, you need to set linear_frames_per_keyframe to the gap you wish to have between each key frame - e.g. 16 would mean the frames are at 0, 16, 32, 48, etc.
|
||||
|
||||
Alternatively, if you set it to dynamic, you can input the positions of the key frames in the text box below this.
|
||||
|
||||
Other than this, please experiment with the different settings to achieve your desired effect:
|
||||
|
||||
The main settings are:
|
||||
|
||||
- frames_per_key_frame: How many frames to generate between each main key frame you provide.
|
||||
- length_of_key_frame_influence: How many frames to apply the ControlNet for after each key frame - the larger the number, the the wider range the input images will influence.
|
||||
- cn_strength: How strong the control of the ControlNet should overall.
|
||||
- buffer: the number of buffer frames places before your video - this is to prevent the end of the generation influencing the beginning due to how the context scheduler works.
|
||||
- Key frame position: how many frames to generate between each main key frame you provide.
|
||||
- Length of influence: what range of frames to apply ControlNet (CN) and IP-Adapter (IPA) to.
|
||||
- Strength of influence: how strong the control of IPA and CN should be.
|
||||
- Relative IPA strength & influence: whether to make IPA's influence stronger or weaker than CN's.
|
||||
|
||||
The **batch_size should be equal to the frames_per_key_frame * number_of_key_frames + 4** - the additional 4 is to accommodate for buffer frames that improve consistency.
|
||||
These are set linearly - the same for each frame - or dynamically - varying them for each frame - you can find detailed instructions on how to tweak these settings inside the workflow above.
|
||||
|
||||
Also, **there's currently a bug where it needs to be restarted after each batch**. This will be fixed soon.
|
||||
Tweaking the settings can greatly influence the motion - you can see two examples of the same images with slightly different settings below to get an idea of this - the settings are visualised in the graph:
|
||||
|
||||
As an example, here's a batch of input images:
|
||||
|
||||
|
||||
|
||||
This also works well for moving between dramatically different images - like in the example below:
|
||||
|
||||
And here's what it looks like when the length_of_key_frame_influence is set to 1.1:
|
||||
|
||||
|
||||
## Philosophy for getting the most from this
|
||||
|
||||
|
||||
This isn’t a tool like text to video that will perform well out of the box - it’s more like a paint brush, an artistic tool that you need to figure out how to get the best from.
|
||||
|
||||
Through trial and error, you'll need to build an understanding of how the motion and settings work, what its limitations are, which inputs images work best with it, etc.
|
||||
|
||||
While here's while that looks like at 0.8:
|
||||
|
||||
|
||||
|
||||
These settings can be so powerful I believe - please share what works for you and your results!
|
||||
If you can figure out how to wield it, this approach can provide enough control for you to make beautiful things that match your imagination precisely.
|
||||
|
||||
## Coming Soon
|
||||
|
||||
- Fix restarting bug
|
||||
- Clean up code
|
||||
- Simplify settings
|
||||
- Nuanced settings for each key frame
|
||||
- Improvements to structure and consistency
|
||||
- Better control over style
|
||||
- Implement more powerful video models and approaches to increase lengh, coherence, etc.
|
||||
- Implement video ControlNets (upcoming from both Stability and Animatediff authors) for greater adherence to input images and better motion.
|
||||
- Increase frame count to unlimited - currently limited to 12 due to RAM constraints with current approach.
|
||||
- Fix issues with colour drain, etc.
|
||||
- Implement LCM, Turbo, etc. to increase speed.
|
||||
|
||||
## Want to give feedback, share creations, or join our community?
|
||||
## Want to give feedback, or join a community who are pushing open source models to their artistic and technical limits?
|
||||
|
||||
You can drop into our Discord here: https://discord.com/invite/8Wx9dFu5tP
|
||||
You're very welcome to drop into our Discord [here](https://discord.com/invite/8Wx9dFu5tP).
|
||||
|
||||
## Credits
|
||||
|
||||
|
||||
Reference in New Issue
Block a user