From 60d14a984022a6d8a826388e40aab6d820b1f47c Mon Sep 17 00:00:00 2001 From: Tung Nguyen Date: Sun, 17 Sep 2023 11:41:55 +0700 Subject: [PATCH] update README --- README.md | 67 +++++++++++++++++++++++-------------------------------- 1 file changed, 28 insertions(+), 39 deletions(-) diff --git a/README.md b/README.md index 66cecda..2180f9f 100644 --- a/README.md +++ b/README.md @@ -5,20 +5,34 @@ ## How to Use 1. Clone this repo into `custom_nodes` folder. -2. Download motion modules from [Google Drive](https://drive.google.com/drive/folders/1EqLC65eR1-W-sGD0Im7fkED6c8GkiNFI) | [HuggingFace](https://huggingface.co/guoyww/animatediff) | [CivitAI](https://civitai.com/models/108836) | [Baidu NetDisk](https://pan.baidu.com/s/18ZpcSM6poBqxWNHtnyMcxg?pwd=et8y). You only need to download one of `mm_sd_v14.ckpt` | `mm_sd_v15.ckpt`. Put the model weights under `comfyui-animatediff/models/`. DO NOT change model filename. +2. Download motion modules and put them under `comfyui-animatediff/models/`. + * Original modules: [Google Drive](https://drive.google.com/drive/folders/1EqLC65eR1-W-sGD0Im7fkED6c8GkiNFI) | [HuggingFace](https://huggingface.co/guoyww/animatediff) | [CivitAI](https://civitai.com/models/108836) | [Baidu NetDisk](https://pan.baidu.com/s/18ZpcSM6poBqxWNHtnyMcxg?pwd=et8y) + * Community modules: [manshoety/AD_Stabilized_Motion](https://huggingface.co/manshoety/AD_Stabilized_Motion) | [CiaraRowles/TemporalDiff](https://huggingface.co/CiaraRowles/TemporalDiff) + * AnimateDiff v2 [mm_sd_v15_v2.ckpt](https://huggingface.co/guoyww/animatediff/blob/main/mm_sd_v15_v2.ckpt) -#### Update 2023/09/15 +## Nodes -- You can now use community models from [manshoety/AD_Stabilized_Motion](https://huggingface.co/manshoety/AD_Stabilized_Motion) or [CiaraRowles/TemporalDiff](https://huggingface.co/CiaraRowles/TemporalDiff) -- Supports AnimateDiff v2 [mm_sd_v15_v2.ckpt](https://huggingface.co/guoyww/animatediff/blob/main/mm_sd_v15_v2.ckpt) model -- Fix image is grayed out. -- New node: **AnimateDiffSampler** and **AnimateDiffLoader** - - Mostly the same with `KSampler` - - Use `AnimateDiffLoader` to load the motion module - - `inject_method`: should left default. See [this issue](https://github.com/ArtVentureX/comfyui-animatediff#gif-has-wartermark-after-update-to-the-latest-version) for more details. - - `frame_number`: animation length +#### AnimateDiffLoader -image +image + +#### AnimateDiffSampler +- Mostly the same with `KSampler` +- Use `AnimateDiffLoader` to load the motion module +- `inject_method`: should left default +- `frame_number`: animation length +- `latent_image`: You can pass an `EmptyLatentImage` + +image + +#### AnimateDiffCombine +- Combine GIF frames and produce the GIF image +- `frame_rate`: number of frame per second +- `loop_count`: use 0 for infinite loop +- `save_image`: should GIF be saved to disk +- `format`: supports `image/gif`, `image/webp` (better compression) or `video/webm` (need `ffmpeg` installed and available in PATH) + +image #### Example Workflow @@ -39,39 +53,14 @@ Workflow file: https://github.com/ArtVentureX/comfyui-animatediff/blob/main/work ![AnimateDiff_00007_](https://github.com/ArtVentureX/comfyui-animatediff/assets/8894763/e6cd53cb-9878-45da-a58a-a15851882386) -See: https://github.com/continue-revolution/sd-webui-animatediff/issues/38 - -Main reasons: - -- Promt are too long (more than 75 tokens) -- Resolution are too high -- Number of frame too high - Work around: - Shorter your prompt and negative prompt - Reduce resolution. AnimateDiff is trained on 512x512 images so it works best with 512x512 output. -- Shouldn't generate longer than 16 frames. AnimateDiff is trained to output the best results with 16 frames. +- Disable xformers with `--disable-xformers` -### GIF has Wartermark after update to the latest version +### GIF has Wartermark (especially when using mm_sd_v15) See: https://github.com/continue-revolution/sd-webui-animatediff/issues/31 -As mentioned in the issue thread, it seems to be due to the training dataset. The new version is the correct implementation and produces smoother GIFs compared to the older version. - - - - - - - - - - - - - - -
Old revisionNew revision
- -I played around with both version and found that the watermark only present in some models, not always. To use the **old (legacy)** method, change `injection_method` to `legacy` in the `AnimateDiffSampler` node. +Training data used by the authors of the AnimateDiff paper contained Shutterstock watermarks. Since mm_sd_v15 was finetuned on finer, less drastic movement, the motion module attempts to replicate the transparency of that watermark and does not get blurred away like mm_sd_v14. Try other community finetuned modules.