nodes/Kosinkadink-ComfyUI-AnimateDiff-Evolved
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update README.md - beginning of revamp

Browse Source
This commit is contained in:
Jedrzej Kosinski
2023-09-22 17:13:01 -05:00
committed by GitHub
parent 97ea3561bd
commit 8d58e992c9
1 changed files with 47 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+47 -30
View File
@@ -4,53 +4,62 @@ Improved [AnimateDiff](https://github.com/guoyww/AnimateDiff/) integration for C
Examples shown here will also often make use of two helpful set of nodes:
- [ComfyUI-Advanced-ControlNet](https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet) for loading files in batches and controlling which latents should be affected by the ControlNet inputs (work in progress, will include more advance workflows + features for AnimateDiff usage later).
- [comfy_controlnet_preprocessors](https://github.com/Fannovel16/comfy_controlnet_preprocessors) for ControlNet preprocessors not present in vanilla ComfyUI; this repo is archived, and future development by the dev will happen here: [comfyui_controlnet_aux](https://github.com/Fannovel16/comfyui_controlnet_aux). While most preprocessors are common between the two, some give different results. Workflows linked here use the archived version, comfy_controlnet_preprocessors.
- [comfy_controlnet_preprocessors](https://github.com/Fannovel16/comfy_controlnet_preprocessors) for ControlNet preprocessors not present in vanilla ComfyUI; this repo is archived, and future development by the dev will happen here: [comfyui_controlnet_aux](https://github.com/Fannovel16/comfyui_controlnet_aux). While most preprocessors are common between the two, some give different results. Workflows linked here use the archived version, comfy_controlnet_preprocessors. (TODO: I'll reinvestigate with more recent changes and update as needed)
## Installation
# Installation
🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥
***IMPORTANT: if you already have ArtVentureX's version of AnimateDiff installed, either remove the ```custom_nodes/comfyui-animatediff``` folder, uninstall or disable it using ComfyUI-Manager, or add .disabled to the end of that folder's name! Otherwise things will go wrong!***
🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥
### If using ComfyUI-Manager:
## If using Comfy Manager:
1. Look for ```AnimateDiff```, and be sure it is ```(Kosinkadink version)```. Install it.
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/10324120-aaee-460a-8ec9-57a46c1b9edc)
### If installing manually:
## If installing manually:
1. Clone this repo into `custom_nodes` folder.
## How to Use:
# How to Use:
1. Download motion modules. You will need at least 1. Different modules produce different results.
- Original research models available from [Google Drive](https://drive.google.com/drive/folders/1EqLC65eR1-W-sGD0Im7fkED6c8GkiNFI) | [HuggingFace](https://huggingface.co/guoyww/animatediff) | [CivitAI](https://civitai.com/models/108836) | [Baidu NetDisk](https://pan.baidu.com/s/18ZpcSM6poBqxWNHtnyMcxg?pwd=et8y).
- Stabilized finetunes of mm_sd_v14 by manshoety from [HuggingFace](https://huggingface.co/manshoety/AD_Stabilized_Motion/tree/main)
- Higher resolution finetune by CiaraRowles from [HuggingFace](https://huggingface.co/CiaraRowles/TemporalDiff/tree/main)
- Place models in ```ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/models```. They can be renamed if you want. More motion modules are being trained by the community - if I am made aware of any good ones, I will link here as well. (TODO: create .safetensor versions of the motion modules and share them here.)
- Original models ```mm_sd_v14```, ```mm_sd_v15```, and ```mm_sd_v15_v2```: [Google Drive](https://drive.google.com/drive/folders/1EqLC65eR1-W-sGD0Im7fkED6c8GkiNFI) | [HuggingFace](https://huggingface.co/guoyww/animatediff) | [CivitAI](https://civitai.com/models/108836) | [Baidu NetDisk](https://pan.baidu.com/s/18ZpcSM6poBqxWNHtnyMcxg?pwd=et8y).
- Stabilized finetunes of mm_sd_v14, ```mm-Stabilized_mid``` and ```mm-Stabilized_high```, by **manshoety**: [HuggingFace](https://huggingface.co/manshoety/AD_Stabilized_Motion/tree/main)
- Higher resolution finetune,```temporaldiff-v1-animatediff``` by **CiaraRowles**: [HuggingFace](https://huggingface.co/CiaraRowles/TemporalDiff/tree/main)
2. Place models in ```ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/models```. They can be renamed if you want.
3. Get creative! If it works for normal image generation, it (probably) will work for AnimateDiff generations. Latent upscales? Go for it. ControlNets, one or more stacked? You betcha. Masking the conditioning of ControlNets to only affect part of the animation? Sure. Try stuff and you will be surprised by what you can do. Samples with workflows are included below.
## Current Features:
- txt2img support; if it works for generating ComfyUI images, it will likely work for AnimateDiff. If something does not work, start a discussion or open an issue and I'll see if I can make it work
# Features:
- Compatible with a variety of samplers, vanilla KSampler nodes and KSampler (Effiecient) nodes.
- ControlNet support - both per-frame, or "interpolating" between frames; can kind of use this as img2video (see workflows below)
- Long animation lengths using sliding context windows {via AnimateDiff Loader (Advanced)}, allowing for longer coherent animations
- Infinite animation length support using sliding context windows (introduced 9/17/23)
## Upcoming features (aka TODO):
# Upcoming features:
- Prompt travel, and in general more control over per-frame conditioning
- Nodes for saving videos, saving generated files into a timestamped folder instead of all over ComfyUI output dir.
- Alternate context schedulers and context types
# Core Nodes:
## AnimateDiff Loader
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/eb07560b-5270-4fc0-9016-311a0327c413)
The ***only required node to use AnimateDiff***, the Loader outputs a model that will perform AnimateDiff functionality when passed into a sampling node.
Inputs:
- model: model to setup for AnimateDiff usage. ***Must be a SD1.5-derived model.***
- context_options: optional context window to use while sampling; if passed in, total animation length has no limit. If not passed in, animation length will be limited to either 24 or 32 frames, depending on motion model.
- model_name: motion model to use with AnimateDiff.
- beta_schedule: noise scheduler for SD. ```sqrt_linear``` is the intended way to use AnimateDiff, with expected saturation. However, ```linear``` can give useful results as well, so feel free to experiment.
Outputs:
- MODEL: model injected to perform AnimateDiff functions
### Usage
To use, just plug in your model into the AnimateDiff Loader. The output model (and any derivative of it in this pathway) is passed into a sampling node, AnimateDiff will function.
The desired animation length is determined by the latents passed into the sampler. **With context_options connected, there is no limit to the amount of latents you can pass in, AKA unlimited animation length.** When no context_options are connected, the sweetspot is 16 latents passed in for best results, with a limit of 24 or 32 based on motion model loaded. **These same rules apply to Uniform Context Option's context_length**.
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/08cc9da9-a21c-469b-8ed6-6153845f80b9)
![image](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/assets/7365912/7d9a21aa-59ee-47ec-8949-8f6a746e7bd7)
## Known Issues (and Solutions, please read!)
### Large resolutions may cause xformers to throw a CUDA error concerning a misconfigured value despite being within VRAM limitations.
It is an xformers bug accidentally triggered by the way the original AnimateDiff CrossAttention is passed in. Eventually either I will fix it, or xformers will. When encountered, the workaround is to boot ComfyUI with the "--disable-xformers" argument.
### GIF has Watermark (especially when using mm_sd_v15)
Training data used by the authors of the AnimateDiff paper contained Shutterstock watermarks. Since mm_sd_v15 was finetuned on finer, less drastic movement, the motion module attempts to replicate the transparency of that watermark and does not get blurred away like mm_sd_v14. Using other motion modules, or combinations of them using Advanced KSamplers should alleviate watermark issues.
## Uniform Context Options
TODO: fill this out
## Samples (download or drag images of the workflows into ComfyUI to instantly load the corresponding workflows!)
@@ -126,3 +135,11 @@ TODO: add generated image here (gif is too big for github)
<img width="1121" alt="Screenshot 2023-07-22 at 22 08 00" src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/600f96b0-df21-4437-917f-7eda35ab6363">
![AnimateDiff_00002](https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/c78d64b9-b308-41ec-9804-bbde654d0b47)
## Known Issues
### Some motion models have visible watermark on resulting images (especially when using mm_sd_v15)
Training data used by the authors of the AnimateDiff paper contained Shutterstock watermarks. Since mm_sd_v15 was finetuned on finer, less drastic movement, the motion module attempts to replicate the transparency of that watermark and does not get blurred away like mm_sd_v14. Using other motion modules, or combinations of them using Advanced KSamplers should alleviate watermark issues.