135 lines
6.3 KiB
Markdown
135 lines
6.3 KiB
Markdown
# ComfyUI_FoleyCrafter
|
||
FoleyCrafter is a video-to-audio generation framework which can produce realistic sound effects semantically relevant and synchronized with videos.
|
||
|
||
FoleyCrafter From: [FoleyCrafter](https://github.com/open-mmlab/FoleyCrafter)
|
||
|
||
1.Installation
|
||
-----
|
||
In the ./ComfyUI /custom_node directory, run the following:
|
||
```
|
||
git clone https://github.com/smthemex/ComfyUI_FoleyCrafter.git
|
||
```
|
||
|
||
2.requirements
|
||
----
|
||
按理是不需要装特别的库，如果还是库缺少，请单独安装。或者打开no need requirements.txt，查看缺失的库是否在里面。
|
||
秋叶包因为是沙箱模式，所以把缺失的库安装在系统的python库里，官方的便携包，用python -m pip install 库名。
|
||
If the module is missing, please open "no need requirements.txt" , pip install missing module.
|
||
|
||
可能会出现的问题，开启video_dubbing 是合成音视频，如果报错，打开控制台CMD，按以下步骤操作：
|
||
Possible issues may arise. Enabling "video_fubbing" produces synthesized audio and video. If an error occurs, open the CMD console and follow these steps:
|
||
|
||
```
|
||
python -m pip uninstall moviepy decorator
|
||
python -m pip install moviepy decorator
|
||
```
|
||
|
||
3 Need models
|
||
----
|
||
3.1
|
||
"ymzhang319/FoleyCrafter" [link](https://huggingface.co/ymzhang319/FoleyCrafter/tree/main) ,
|
||
全部下载，并按如下结构存放在ComfyUI/models/foleycrafter 文件夹下，联外网会自动下载：
|
||
Download all and store them in the "ComfyUI/models/foleycrafter folder" according to the following structure，online will auto download:
|
||
|
||
```
|
||
└── ComfyUI/models/foleycrafter/
|
||
├── semantic
|
||
│ ├── semantic_adapter.bin
|
||
├── vocoder
|
||
│ ├── vocoder.pt
|
||
│ ├── config.json
|
||
├── temporal_adapter.ckpt
|
||
│ │
|
||
└── timestamp_detector.pth.tar
|
||
```
|
||
3.2
|
||
"h94/IP-Adapter" [link](https://huggingface.co/h94/IP-Adapter/tree/main/models/image_encoder),
|
||
离线使用时，部分下载，文件结构如下,联外网会自动下载，if offline, Partial download, file structure as follows，online will auto download：
|
||
```
|
||
└── any_path/h94/IP-Adapter/models/
|
||
├── image_encoder
|
||
│ ├── model.safetensors
|
||
│ ├── config.json
|
||
|
||
```
|
||
3.3
|
||
"auffusion/auffusion-full-no-adapter" [link](https://huggingface.co/auffusion/auffusion-full-no-adapter/tree/main),
|
||
离线使用时，部分下载，文件结构如下,联外网会自动下载，if offline, Partial download, file structure as follows，online will auto download：
|
||
```
|
||
├── any_path/auffusion/auffusion-full-no-adapter/
|
||
| ├──model_index.json
|
||
| ├──vae
|
||
| ├── config.json
|
||
| ├── diffusion_pytorch_model.bin
|
||
| ├──unet
|
||
| ├── config.json
|
||
| ├── diffusion_pytorch_model.bin
|
||
| ├──tokenizer
|
||
| ├── merges.txt
|
||
| ├── special_tokens_map.json
|
||
| ├── tokenizer_config.json
|
||
| ├── vocab.json
|
||
| ├── text_encoder
|
||
| ├── config.json
|
||
| ├── pytorch_model.bin
|
||
| ├── scheduler
|
||
| ├── scheduler_config.json
|
||
| ├──feature_extractor
|
||
| ├──preprocessor_config.json
|
||
| ├──vocoder
|
||
| ├──config.json
|
||
| ├──vocoder.pt
|
||
```
|
||
4 Example
|
||
----
|
||
video_dubbing using prompt an uprompt (Latest version)
|
||
|
||
|
||
|
||
5 Function Description of Nodes
|
||
---
|
||
--semantic_scale :ip adatpter scale 调整音频的跟视频的相似度;
|
||
--max_frame ：audio length 音频长度，还未测试极限，目前设置500;
|
||
--controlnet_scale： another audio sacle 未测试;
|
||
--sample_width/sample_width： weights type, don't change it 模型尺寸，不要动，这个跟视频长宽无关;
|
||
--video_dubbing: save or not（if using example） 是否用内置的音视频合成，如果用示例的VH合成，可以关闭。
|
||
|
||
6 My ComfyUI node list：
|
||
-----
|
||
|
||
1、ParlerTTS node:[ComfyUI_ParlerTTS](https://github.com/smthemex/ComfyUI_ParlerTTS)
|
||
2、Llama3_8B node:[ComfyUI_Llama3_8B](https://github.com/smthemex/ComfyUI_Llama3_8B)
|
||
3、HiDiffusion node：[ComfyUI_HiDiffusion_Pro](https://github.com/smthemex/ComfyUI_HiDiffusion_Pro)
|
||
4、ID_Animator node： [ComfyUI_ID_Animator](https://github.com/smthemex/ComfyUI_ID_Animator)
|
||
5、StoryDiffusion node：[ComfyUI_StoryDiffusion](https://github.com/smthemex/ComfyUI_StoryDiffusion)
|
||
6、Pops node：[ComfyUI_Pops](https://github.com/smthemex/ComfyUI_Pops)
|
||
7、stable-audio-open-1.0 node ：[ComfyUI_StableAudio_Open](https://github.com/smthemex/ComfyUI_StableAudio_Open)
|
||
8、GLM4 node：[ComfyUI_ChatGLM_API](https://github.com/smthemex/ComfyUI_ChatGLM_API)
|
||
9、CustomNet node：[ComfyUI_CustomNet](https://github.com/smthemex/ComfyUI_CustomNet)
|
||
10、Pipeline_Tool node :[ComfyUI_Pipeline_Tool](https://github.com/smthemex/ComfyUI_Pipeline_Tool)
|
||
11、Pic2Story node :[ComfyUI_Pic2Story](https://github.com/smthemex/ComfyUI_Pic2Story)
|
||
12、PBR_Maker node:[ComfyUI_PBR_Maker](https://github.com/smthemex/ComfyUI_PBR_Maker)
|
||
13、ComfyUI_Streamv2v_Plus node:[ComfyUI_Streamv2v_Plus](https://github.com/smthemex/ComfyUI_Streamv2v_Plus)
|
||
14、ComfyUI_MS_Diffusion node:[ComfyUI_MS_Diffusion](https://github.com/smthemex/ComfyUI_MS_Diffusion)
|
||
15、ComfyUI_AnyDoor node: [ComfyUI_AnyDoor](https://github.com/smthemex/ComfyUI_AnyDoor)
|
||
16、ComfyUI_Stable_Makeup node: [ComfyUI_Stable_Makeup](https://github.com/smthemex/ComfyUI_Stable_Makeup)
|
||
17、ComfyUI_EchoMimic node: [ComfyUI_EchoMimic](https://github.com/smthemex/ComfyUI_EchoMimic)
|
||
18、ComfyUI_FollowYourEmoji node: [ComfyUI_FollowYourEmoji](https://github.com/smthemex/ComfyUI_FollowYourEmoji)
|
||
19、ComfyUI_Diffree node: [ComfyUI_Diffree](https://github.com/smthemex/ComfyUI_Diffree)
|
||
20、ComfyUI_FoleyCrafter node: [ComfyUI_FoleyCrafter](https://github.com/smthemex/ComfyUI_FoleyCrafter)
|
||
|
||
7 Citation
|
||
------
|
||
"open-mmlab/FoleyCrafter"
|
||
```
|
||
@misc{zhang2024pia,
|
||
title={FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds},
|
||
author={Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, Kai Chen},
|
||
year={2024},
|
||
eprint={2407.01494},
|
||
archivePrefix={arXiv},
|
||
primaryClass={cs.CV}
|
||
}
|
||
```
|
||
|