From 0cf8df11ce965583429449de551159c44acc42b6 Mon Sep 17 00:00:00 2001 From: ExponentialML <59846140+ExponentialML@users.noreply.github.com> Date: Wed, 20 Mar 2024 19:08:33 -0700 Subject: [PATCH] Update README.md --- README.md | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 433e3ca..a412e3e 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,65 @@ # ComfyUI_Native_DynamiCrafter DynamiCrafter that works natively with ComfyUI's nodes, optimizations, and more. -![image](https://github.com/ExponentialML/ComfyUI_Native_DynamiCrafter/assets/59846140/cc4c3dda-2568-4218-9890-a3ddaaf6354b) +![image](https://github.com/ExponentialML/ComfyUI_Native_DynamiCrafter/assets/59846140/fd1008ed-7660-454a-8253-1e032c9d054f) +https://github.com/ExponentialML/ComfyUI_Native_DynamiCrafter/assets/59846140/8a42cecf-06f6-4764-8f63-74c96be8578d -Cleaning code and final tests. Pushing soon. +> [!NOTE] +> While this is still considered WIP (or beta), everything should be fully functional and adaptable to various workflows. + +# Getting Started + +Go to your `custom_nodes` directory in ComfyUI, and install by: + +`git clone https://github.com/ExponentialML/ComfyUI_Native_DynamiCrafter.git` + +# Installation + +The pruned UNet checkpoints have been uploaded to HuggingFace. Each variant is working and fully functional. +**Model Links:** https://huggingface.co/ExponentialML/DynamiCrafterUNet + +You will also need a VAE, The CLIP model used with Stable Diffusion 2.1, and the Open CLIP Vision Model. All of the necessary model downloads are at that link. + +If you aready have the base SD models, you do not need to download them (just use the CheckpointSimpleLoader without the model part). + +Place the **DynamiCrafter** models inside `ComfyUI_Path/models/dynamicrafter_models` + +If you are downloading the CLIP and VAE models separately, place them under their respective paths in the `ComfyUI_Path/models/` directory. + +# Usage + +- **model**: The loaded DynamiCrafter model. + +- **clip_vision**: The CLIP Vision Checkpoint. + +- **vae**: A Stable Diffusion VAE. If it works with < SD 2.1, it will work with this. + +- **image_proj_model**: The Image Projection Model that is in the DynamiCrafter model file. + +- **images**: The input images necessary for inference. If you are doing interpolation, you can simply batch two images together, check the toggle (see below), and everything will be handled automatically. + +- **use_interpolation**: Use the interpolation mode with the interpolation model variant. You can interpolate any two frames (images), or predict the rest using one input. + +- **fps**: Controls the speed of the video. If you're using a 256 based model, the highest plausible value is **4** + +- **frames**: The amount of frames to use. If you're doing interpolation, the max is **16**. This is strictly enforced as it doesn't work properly (blurry results) if set higher. + +- **model (output)**: The output into the a Sampler. + +- **empty_latent**: An empty latent with the same size and frames as the processed ones. + +- **latent_img**: If you're doing Img2Img based workflows, this is the necessary one to use. + +> [!TIP] +> You don't have to use the latent outputs. As long as you use the same frame length (as your batch size) and same height and with as your image inputs, you can use your own latents. +> This means that you can experiment with inpainting and so on. + +# TODO +- [ ] Add various workflows. +- [ ] Ensure attention optimizations are working properly. +- [ ] Add autoregressive nodes (this may be a separate repository) + +# Credits + +Thanks to @Doubiiu for for open sourcing [DynamiCrafter](https://github.com/Doubiiu/DynamiCrafter)https://github.com/Doubiiu/DynamiCrafter! Please support their work, and please follow any license terms they may uphold.