AnimateDiff for ComfyUI

AnimateDiff integration for ComfyUI, adapts from sd-webui-animatediff. Please read the original repo README for more information.

How to Use

1. Clone this repo into custom_nodes folder.
2. Download motion modules and put them under comfyui-animatediff/models/.

• Original modules: Google Drive | HuggingFace | CivitAI | Baidu NetDisk
• Community modules: manshoety/AD_Stabilized_Motion | CiaraRowles/TemporalDiff
• AnimateDiff v2 mm_sd_v15_v2.ckpt

Nodes

AnimateDiffLoader

AnimateDiffSampler

• Mostly the same with KSampler
• Use AnimateDiffLoader to load the motion module
• inject_method: should left default
• frame_number: animation length
• latent_image: You can pass an EmptyLatentImage

AnimateDiffCombine

• Combine GIF frames and produce the GIF image
• frame_rate: number of frame per second
• loop_count: use 0 for infinite loop
• save_image: should GIF be saved to disk
• format: supports image/gif, image/webp (better compression) or video/webm (need ffmpeg installed and available in PATH)

Workflows

Simple txt2gif

Workflow: simple.json

Samples:

Latent upscale

Upscale latent output using LatentUpscale then do a 2nd pass with AnimateDiffSampler.

Workflow: latent-upscale.json

Samples:

Using with ControlNet

You will need following additional nodes:

• Kosinkadink/ComfyUI-Advanced-ControlNet: Apply different weight for each latent in batch
• Fannovel16/comfyui_controlnet_aux: ControlNet preprocessors

Animate with starting and ending images

• Use LatentKeyframe and TimestampKeyframe from ComfyUI-Advanced-ControlNet to apply diffrent weights for each latent index.
• Use 2 controlnet modules for two images with weights reverted.

Workflow: cn-2images.json

Samples:

Using GIF as ControlNet input

Using a GIF (or video, or a list of images) as ControlNet input.

Workflow: cn-vid2vid.json

Samples:

Known Issues

GIF split into multiple scenes

Work around:

• Shorter your prompt and negative prompt
• Reduce resolution. AnimateDiff is trained on 512x512 images so it works best with 512x512 output.
• Disable xformers with --disable-xformers

GIF has Wartermark (especially when using mm_sd_v15)

See: https://github.com/continue-revolution/sd-webui-animatediff/issues/31

Training data used by the authors of the AnimateDiff paper contained Shutterstock watermarks. Since mm_sd_v15 was finetuned on finer, less drastic movement, the motion module attempts to replicate the transparency of that watermark and does not get blurred away like mm_sd_v14. Try other community finetuned modules.