nodes/Kosinkadink-ComfyUI-AnimateDiff-Evolved
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update README.md - motion_scale and Motion Model Settings

Browse Source
This commit is contained in:
Jedrzej Kosinski
2023-10-30 10:02:57 -05:00
committed by GitHub
parent 82b86a3c42
commit e2013cbf9f
1 changed files with 26 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+26 -6
View File
@@ -39,6 +39,7 @@ Examples shown here will also often make use of these helpful sets of nodes:
- Mixable Motion LoRAs from [original AnimateDiff repository](https://github.com/guoyww/animatediff/) implemented. Caveat: only really work on v2-based motion models like ```mm_sd_v15_v2```, ```mm-p_0.5.pth```, and ```mm-p_0.75.pth``` **(introduced 9/25/23)**
- Prompt travel using BatchPromptSchedule node from [ComfyUI_FizzNodes](https://github.com/FizzleDorf/ComfyUI_FizzNodes) **(working since 9/27/23)**
- [HotshotXL](https://huggingface.co/hotshotco/Hotshot-XL/tree/main) support (an SDXL motion module arch), ```hsxl_temporal_layers.safetensors``` **(working since 10/05/23)** NOTE: You will need to use ```linear``` beta_schedule, the sweetspot for context_length or total frames (when not using context) is 8 frames, and you will need to use an SDXL checkpoint. Will add more documentation and example workflows soon when I have some time between working on features/other nodes.
- Motion scaling and other motion model settings to influence motion amount **(introduced 10/30/23)**
# Upcoming features:
- Alternate context schedulers and context types (in progress)
@@ -47,7 +48,8 @@ Examples shown here will also often make use of these helpful sets of nodes:
# Core Nodes:
## AnimateDiff Loader
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/232ef170-30e0-4119-ace2-4cc9a842d1ac)
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/7dba5a1c-8c29-4d8c-9778-17e217af81b9)
The ***only required node to use AnimateDiff***, the Loader outputs a model that will perform AnimateDiff functionality when passed into a sampling node.
@@ -56,8 +58,10 @@ Inputs:
- model: model to setup for AnimateDiff usage. ***Must be a SD1.5-derived model.***
- context_options: optional context window to use while sampling; if passed in, total animation length has no limit. If not passed in, animation length will be limited to either 24 or 32 frames, depending on motion model.
- motion_lora: optional motion LoRA input; if passed in, can influence movement.
- motion_model_settings: optional settings to influence motion model.
- model_name: motion model to use with AnimateDiff.
- beta_schedule: noise scheduler for SD. ```sqrt_linear``` is the intended way to use AnimateDiff, with expected saturation. However, ```linear``` can give useful results as well, so feel free to experiment.
- motion_scale: change motion amount generated by motion model - if less than 1, less motion; if greater than 1, more motion.
Outputs:
- MODEL: model injected to perform AnimateDiff functions
@@ -69,14 +73,16 @@ The desired animation length is determined by the latents passed into the sample
You can also connect AnimateDiff LoRA Loader nodes to influence the overall movement in the image - currently, only works well on motion v2-based models.
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/524ba030-97aa-47a5-a0fd-ecffbbf5e439)
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/908d1848-13d4-4e86-bdb0-f6870fd28b06)
[Simplest Usage]
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/95c5edc1-3d48-4f03-9f04-4d4ce54c8602)
[All Possible Connections Usage]
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/22ffaf8a-bda9-428d-be11-163cb75feae4)
## Uniform Context Options
TODO: fill this out
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/426a2f07-fdcd-477c-bd87-48cf17d91cd0)
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/0327521c-15df-46d8-bee4-6c80b2d7d02d)
@@ -93,11 +99,25 @@ Inputs:
Outputs:
- MOTION_LORA: motion_lora object storing the names of all the LoRAs that were chained behind it - can be plugged into the back of another AnimateDiff LoRA Loader, or into AniamateDiff Loader's motion_lora input.
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/5e46261d-fe87-4daa-8ac3-3ef615f4619d)
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/21ec6fa6-4874-4312-bd41-477307c9ebf8)
[Simplest Usage]
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/520fb048-1d36-4ec0-a072-532817eafdc0)
[Chaining Multiple Motion LoRAs]
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/9c68fcb1-4573-4ce7-a9a5-4d2b1163d762)
## Motion Model Settings
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/12887803-8a4f-443b-9c73-82d4b8fc7c75)
Additional tweaks to the internals of the motion models. The Advanced settings will take a whole guide to explain, and I currently do not have the time for that. Instead, I'll focus on the simple settings.
Inputs:
- motion_pe_stretch: used to decrease the amount of motion by stretching (and interpolating) between the positional encoders (PEs). TL;DR: number go up, animation slow down. Number up too much, animation begins to vibrate (vibration artifacts).
Outputs:
- MOTION_MODEL_SETTINGS: motion_model_settings object to be plugged into an AnimateDiff Loader.
## Samples (download or drag images of the workflows into ComfyUI to instantly load the corresponding workflows!)
### txt2img