# ComfyUI_EchoMimic
You can use EchoMimic & EchoMimic V2 & EchoMimic V3 in comfyui.
[Echomimic](https://github.com/antgroup/echomimic/tree/main)：Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditioning
[Echomimic_v2](https://github.com/antgroup/echomimic_v2): Towards Striking, Simplified, and Semi-Body Human Animation
[Echomimic_v3](https://github.com/antgroup/echomimic_v3)：1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animation
---
# New Updates
* V3正式上线，测试环境12G VRAM，OOM需要减少视频分块(partial_video_length)的数值，12G可以跑65，16可以试试97，更高可以试试113
* V3 is Done,you can try it now.. need 8G and more (use mmgp,LOW LOW,partial_video_length==65 or 33)
# 1. Installation
In the ./ComfyUI /custom_nodes directory, run the following:
```
git clone https://github.com/smthemex/ComfyUI_EchoMimic.git
```
---
# 2. Requirements
```
pip install -r requirements.txt
```
# Notice
---
If use v1 version 如果要使用V1版本：
```
pip install --no-deps facenet-pytorch
```
If use v3 version 如果要使用V3版本：
```
pip install retina-face==0.0.17 #使用须外网下载模型，待处理
pip install mmgp # optional 可选
```
* 如果ffmpeg 报错，if ffmpeg error：
```
pip uninstall ffmpeg
pip install ffmpeg-python
```
* 其他库缺啥装啥。。。
* If the module is missing, , pip install missing module.
# 3. Models Required
----
**3.1 V1 & V2 Shared model v1 和 v2 共用的模型**:
如果能直连抱脸,点击就会自动下载所需模型,不需要手动下载.
* unet [link](https://huggingface.co/lambdalabs/sd-image-variations-diffusers)
* V1 & V2 audio [link](https://huggingface.co/BadToBest/EchoMimic/tree/main)
* vae(stabilityai/sd-vae-ft-mse) [link](https://huggingface.co/stabilityai/sd-vae-ft-mse)
```
├── ComfyUI/models/ echo_mimic
| ├── unet
| ├── diffusion_pytorch_model.bin
| ├── config.json
| ├── audio_processor
| ├── whisper_tiny.pt
├── ComfyUI/models/vae
| ├── diffusion_pytorch_model.safetensors or rename sd-vae-ft-mse.safetensors
```
**3.2 V1 models V1使用以下模型**:
* V1 address [link](https://huggingface.co/BadToBest/EchoMimic/tree/main)
* Audio-Drived Algo Inference 音频驱动
```
├── ComfyUI/models/echo_mimic
| ├── denoising_unet.pth
| ├── face_locator.pth
| ├── motion_module.pth
| ├── reference_unet.pth
Audio-Drived Algo Inference acc 音频驱动加速版
| ├── denoising_unet_acc.pth
| ├── motion_module_acc.pth
```
* Using Pose-Drived Algo Inference 姿态驱动
```
├── ComfyUI/models/echo_mimic
| ├── denoising_unet_pose.pth
| ├── face_locator_pose.pth
| ├── motion_module_pose.pth
| ├── reference_unet_pose.pth
Using Pose-Drived Algo Inference ACC 姿态驱动加速版
| ├── denoising_unet_pose_acc.pth
| ├── motion_module_pose_acc.pth
```
**3.2 v2 version**
use model below V2, Automatic download, you can manually add it 使用以下模型,使用及自动下载,你可以手动添加:
模型地址address:[huggingface](https://huggingface.co/BadToBest/EchoMimicV2/tree/main)
```
├── ComfyUI/models/echo_mimic/v2
| ├── denoising_unet.pth
| ├── motion_module.pth
| ├── pose_encoder.pth
| ├── reference_unet.pth
if use acc 姿态驱动加速版
| ├── denoising_unet_acc.pth
| ├── motion_module_acc.pth
```
YOLOm8 [download link](https://huggingface.co/Ultralytics/YOLOv8/tree/main)
sapiens pose [download link](https://huggingface.co/facebook/sapiens-pose-1b-torchscript/tree/main)
sapiens的pose 模型可以量化为fp16的，详细见我的sapiens插件 [地址](https://github.com/smthemex/ComfyUI_Sapiens)
```
├── ComfyUI/models/echo_mimic
| ├── yolov8m.pt
| ├── sapiens_1b_goliath_best_goliath_AP_639_torchscript.pt2 or/或者 sapiens_1b_goliath_best_goliath_AP_639_torchscript_fp16.pt2
```
**3.3 v3 version**
3.3.1 from [Wan2.1-Fun-V1.1-1.3B-InP](https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-1.3B-InP/tree/main)downlaod Wan2.1_VAE.pth and diffusion_pytorch_model.safetensors
3.3.2 use comfyui ,[clipvison-h](https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/tree/main/split_files/clip_vision) and [umt5_xxl_fp8_e4m3fn_scaled.safetensors ](https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/tree/main/split_files/text_encoders)
3.3.3 [wav2vec2-base-960h](https://huggingface.co/facebook/wav2vec2-base-960h/tree/main)
3.3.4 [BadToBest/EchoMimicV3](https://huggingface.co/BadToBest/EchoMimicV3/tree/main) transformer
```
├── ComfyUI/models/echo_mimic/transformer
| ├── diffusion_pytorch_model.safetensors #Wan2.1-Fun-V1.1-1.3B-InP transformer #3.13G 务必注意模型同名。
| ├── config.json
├── ComfyUI/models/echo_mimic/wav2vec2-base-960h
| ├── all config json files
| ├── model.safetensors
├── ComfyUI/models/clip
| ├── umt5_xxl_fp8_e4m3fn_scaled.safetensors
├── ComfyUI/models/clip_vision
| ├──clipvison-h # 1.26G
├── ComfyUI/models/echo_mimic/
| ├──diffusion_pytorch_model.safetensors ##BadToBest/EchoMimicV3
├── ComfyUI/models/vae
| ├── Wan2.1_VAE.pth
```
# 4 Example
-----
* V3 version
* V2 version
* V2加载自定义视频驱动视频，V2 loads custom video driver videos
* Echomimic_v2 use default pose new version 使用官方默认的pose文件
* V1 version
* audio driver 音频驱动
* 示例的 VH node : [ComfyUI-VideoHelperSuite](https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite)
---
# 5 Function Description
---
* v1 版本的生成模式
a. 单纯音频驱动视频生成模式，infer_mode可选常规的“audio_drived” 和加速版"audio_drived_acc" 模型；
b. pose驱动生成模式，常规选项为pose_normal_sapiens/pose_normal_dwpose（等同） 加速版本为"pose_acc"模型；
----motion_sync：pose驱动时，如果打开且video_file有视频文件时，生成pkl文件，并生成参考视频的视频；pkl文件在input\tensorrt_lite 目录下，再次使用需要重启comfyUI。
----motion_sync：如果关闭且pose_dir不为none的时候，读取选定的pose_dir目录名的pkl文件，生成pose视频；如果pose_dir为空的时候，生成基于默认assets\test_pose_demo_pose的视频
* v2 版本的生成模式
a. infer_mode选择audio_drive,pose_dir 选择列表里的几个默认pose，则使用默认的npy pose文件;
b. infer_mode选择audio_drive,pose_dir 选择已有的npy文件夹（位于...ComfyUI/input/tensorrt_lite目录下);
c. infer_mode选择pose_normal_dwpose 或pose_normal_sapiens,video_images连接视频入口，确认...ComfyUI/models/echo_mimic 下有yolov8m.pt 和sapiens_1b_goliath_best_goliath_AP_639_torchscript.pt2 模型,根据输入视频生成npy文件（可以下次用）和视频
* v3 版本生成模式
a. 基于retina-face库生成
b. 如果retina-face调用失败，则以默认的女性face作为mask
**特别的选项**：
* save_video：如果不想使用VH节点时，可以开启，默认关闭；
* draw_mouse：你可以试试；
* length：帧数，时长等于length/fps；
* acc模型 ，6步就可以，但是质量略有下降；
* lowvram :低显存用户可以开启 lowvram users can enable it
* 内置内置图片等比例裁切。
* facecrop_ratio的放大系数为1/facecrop_ratio，如果设置为0.5，面部会得到2倍的放大，建议只在参考图片或者驱动视频中的人脸占比很小的时候，才将facecrop_ratio调整为较小的值.为1 或者0 时不裁切
* cfg数值设置为1，仅在turbo模式有效，其他会报错。V2推荐2.5 V3推荐3.5
* use_mmgp 仅V3版本有效
* partial_video_length 仅V3版本有效，数值越低显存占用越低；
* teacache 仅V3版本有效；
---
6 Citation
------
EchoMimici
``` python
@misc{chen2024echomimic,
title={EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditioning},
author={Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, Chenguang Ma},
year={2024},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```
EchoMimici-V2
``` python
@misc{meng2024echomimic,
title={EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animation},
author={Rang Meng, Xingyu Zhang, Yuming Li, Chenguang Ma},
year={2024},
eprint={2411.10061},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```
EchoMimici-V3
```
@misc{meng2025echomimicv3,
title={EchoMimicV3: 1.3B Parameters are All You Need for Unified Multi-Modal and Multi-Task Human Animation},
author={Rang Meng, Yan Wang, Weipeng Wu, Ruobing Zheng, Yuming Li, Chenguang Ma},
year={2025},
eprint={2507.03905},
archivePrefix={arXiv}
}
```
sapiens
```
@article{khirodkar2024sapiens,
title={Sapiens: Foundation for Human Vision Models},
author={Khirodkar, Rawal and Bagautdinov, Timur and Martinez, Julieta and Zhaoen, Su and James, Austin and Selednik, Peter and Anderson, Stuart and Saito, Shunsuke},
journal={arXiv preprint arXiv:2408.12569},
year={2024}
}
```