Update README.md
update
This commit is contained in:
smthemex
@@ -1,34 +1,35 @@
|
||||
# ComfyUI_StoryDiffusion
|
||||
You can using StoryDiffusion in ComfyUI
|
||||
|
||||
StoryDiffusion origin From: [link](https://github.com/HVision-NKU/StoryDiffusion)
|
||||
StoryDiffusion origin From: [link](https://github.com/HVision-NKU/StoryDiffusion)
|
||||
MS-Diffusion origin From: [link](https://github.com/MS-Diffusion/MS-Diffusion)
|
||||
----
|
||||
My ComfyUI node list：
|
||||
-----
|
||||
|
||||
1、ParlerTTS node:[ComfyUI_ParlerTTS](https://github.com/smthemex/ComfyUI_ParlerTTS)
|
||||
|
||||
2、Llama3_8B node:[ComfyUI_Llama3_8B](https://github.com/smthemex/ComfyUI_Llama3_8B)
|
||||
|
||||
3、HiDiffusion node：[ComfyUI_HiDiffusion_Pro](https://github.com/smthemex/ComfyUI_HiDiffusion_Pro)
|
||||
|
||||
3、HiDiffusion node：[ComfyUI_HiDiffusion_Pro](https://github.com/smthemex/ComfyUI_HiDiffusion_Pro)
|
||||
4、ID_Animator node： [ComfyUI_ID_Animator](https://github.com/smthemex/ComfyUI_ID_Animator)
|
||||
|
||||
5、StoryDiffusion node：[ComfyUI_StoryDiffusion](https://github.com/smthemex/ComfyUI_StoryDiffusion)
|
||||
|
||||
6、Pops node：[ComfyUI_Pops](https://github.com/smthemex/ComfyUI_Pops)
|
||||
|
||||
7、stable-audio-open-1.0 node ：[ComfyUI_StableAudio_Open](https://github.com/smthemex/ComfyUI_StableAudio_Open)
|
||||
|
||||
8、GLM4 node：[ComfyUI_ChatGLM_API](https://github.com/smthemex/ComfyUI_ChatGLM_API)
|
||||
|
||||
9、CustomNet node：[ComfyUI_CustomNet](https://github.com/smthemex/ComfyUI_CustomNet)
|
||||
|
||||
6、Pops node：[ComfyUI_Pops](https://github.com/smthemex/ComfyUI_Pops)
|
||||
7、stable-audio-open-1.0 node ：[ComfyUI_StableAudio_Open](https://github.com/smthemex/ComfyUI_StableAudio_Open)
|
||||
8、GLM4 node：[ComfyUI_ChatGLM_API](https://github.com/smthemex/ComfyUI_ChatGLM_API)
|
||||
9、CustomNet node：[ComfyUI_CustomNet](https://github.com/smthemex/ComfyUI_CustomNet)
|
||||
10、Pipeline_Tool node :[ComfyUI_Pipeline_Tool](https://github.com/smthemex/ComfyUI_Pipeline_Tool)
|
||||
11、Pic2Story node :[ComfyUI_Pic2Story](https://github.com/smthemex/ComfyUI_Pic2Story)
|
||||
12、PBR_Maker node:[ComfyUI_PBR_Maker](https://github.com/smthemex/ComfyUI_PBR_Maker)
|
||||
|
||||
11、Pic2Story node :[ComfyUI_Pic2Story](https://github.com/smthemex/ComfyUI_Pic2Story)
|
||||
NEW Update
|
||||
---
|
||||
1、添加双角色同框功能，使用方法：(A and B) have lunch...., A,B为角色名，中间的 and 和括号不能删除,括号为生效条件！！！
|
||||
2、因为调用了MS-diffusion的功能，所以要使用双角色同框，必须添加encoder模型（laion/CLIP-ViT-bigG-14-laion2B-39B-b160k,无法替换为其他的）和ip-adapeter微调模型（ms_adapter.bin,无法替换）；
|
||||
3、更改lora的代码，现在lora_scale应该正常生效了。
|
||||
|
||||
1. Add dual role same frame function, usage method: (A and B) have lunch...., where A and B are role names, and the middle "and" parentheses cannot be removed，The parentheses represent the effective conditions!!!
|
||||
2. Because the" MS diffusion" function is called, in order to use dual role same frame, it is necessary to add an encoder model (laion/CLIP-ViT-bigG-14-laion2B-39B-b160k,which cannot be replaced with others) and an ip adapet fine-tuning model (ms_adapter.bin,which cannot be replaced);
|
||||
3. Change Lora's code, "Lora_scale" should now take effect normally.
|
||||
|
||||
12、PBR_Maker node:[ComfyUI_PBR_Maker](https://github.com/smthemex/ComfyUI_PBR_Maker)
|
||||
|
||||
Notice（节点的特殊功能说明 Special Function Description of Nodes）
|
||||
---
|
||||
@@ -68,10 +69,12 @@ Notice（节点的特殊功能说明 Special Function Description of Nodes）
|
||||
|
||||
2.requirements
|
||||
----
|
||||
```
|
||||
pip install -r requirements.txt
|
||||
|
||||
```
|
||||
```
|
||||
pip install git+https://github.com/tencent-ailab/IP-Adapter.git
|
||||
|
||||
```
|
||||
如果缺失模块，请单独pip install
|
||||
If the module is missing, please pip install
|
||||
|
||||
@@ -81,26 +84,58 @@ If the module is missing, please pip install
|
||||
|
||||
例如：pip install -r requirements.txt --target= "你的路径/python_embeded/Lib/site-packages"
|
||||
|
||||
2.2 如果实在装不上，你需要安装python3.10环境，git，以及pip，然后运行pip install git+https://github.com/tencent-ailab/IP-Adapter.git。 安装好之后，在你独立安装的python目录下，找到lib/site-packages下的ip-Adapter文件夹，复制到你的/python_embeded/Lib/site-packages里去
|
||||
2.2 如果实在装不上，你需要安装python3.10环境，git，以及pip，然后运行pip install git+https://github.com/tencent-ailab/IP-Adapter.git 安装好之后，在你独立安装的python目录下，找到lib/site-packages下的ip-Adapter文件夹，复制到你的/python_embeded/Lib/site-packages里去
|
||||
|
||||
2.3秋叶包，理论上是按2.1的方法，如果不行，可以试试，2.2
|
||||
|
||||
|
||||
3 Need model
|
||||
----
|
||||
3.1 online
|
||||
在线模式，点击运行，会自动从huggingface 下载所需模型，请确保你的的网络通畅，默认可用的模型有G161222/RealVisXL_V4.0 ，stabilityai/stable-diffusion-xl-base-1.0 ， stablediffusionapi/sdxl-unstable-diffusers-y ，sd-community/sdxl-flash ；
|
||||
|
||||
3.1 online 在线模式
|
||||
点击运行，会自动从huggingface 下载所需模型，请确保你的的网络通畅，默认可用的模型有G161222/RealVisXL_V4.0 ，stabilityai/stable-diffusion-xl-base-1.0 ， stablediffusionapi/sdxl-unstable-diffusers-y ，sd-community/sdxl-flash ；
|
||||
选择'Use_Single_XL_Model',以及你本地的SDXL单体模型（例如：Juggernaut-XL_v9_RunDiffusionPhoto_v2.safetensors ），也会下载对应的config文件；
|
||||
|
||||
In online mode, click run and the required model will be automatically downloaded from the huggingface. Please ensure that your network is unobstructed. The default available models are G161222/RealVisXL_V4.0, stabilityai/stable-diffusion-xl-base-1.0 ， stablediffusionapi/sdxl-unstable-diffusers-y ，sd-community/sdxl-flash ；
|
||||
In online mode, click run and the required model will be automatically downloaded from the huggingface. Please ensure that your network is unobstructed. The default available models are G161222/RealVisXL_V4.0, stabilityai/stable-diffusion-xl-base-1.0 ， stablediffusionapi/sdxl-unstable-diffusers-y ，sd-community/sdxl-flash ；
|
||||
Select 'Use_Single_XL-Model', as well as your local SDXL monomer model (for example: Jumpernaut XL_v9-RunDiffusionPhoto_v2. safetensors), and the corresponding config file will also be downloaded;
|
||||
|
||||
Select 'Use_Single_XL-Model', as well as your local SDXL monomer model (for example: Jumpernaut XL_v9-RunDiffusionPhoto_v2. safetensors), and the corresponding config file will also be downloaded;
|
||||
--using dual role same frame function(使用双角色功能时):
|
||||
|
||||
Need download "ms_adapter.bin" : [link](https://huggingface.co/doge1516/MS-Diffusion/tree/main)
|
||||
Need encoder model "laion/CLIP-ViT-bigG-14-laion2B-39B-b160k":[link](https://huggingface.co/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k)
|
||||
|
||||
```
|
||||
├── ComfyUI/custom_nodes/ComfyUI_Pops/
|
||||
| ├──weights/
|
||||
| ├── photomaker-v1.bin
|
||||
| ├── ms_adapter.bin
|
||||
|
||||
```
|
||||
|
||||
3.2 offline
|
||||
打开ComfyUI_StoryDiffusion/config/models.yaml的models.yaml文件，如果有预下载的默认的扩散模型，可以不填，如果地址不在默认的C盘一类，可以在“path”一栏：填写扩散模型的绝对地址，须是“/” .
|
||||
Open the models.yaml file of ComfyUI_StoryDiffusion/config/models.yaml. If there is a pre downloaded default diffusion model, it can be left blank. If the address is not in the default C drive category, you can fill in the absolute address of the diffusion model in the "path" column, which must be "/"
|
||||
|
||||
--using dual role same frame function(使用双角色功能时):
|
||||
在“laion/CLIP-ViT-bigG-14-laion2B-39B-b160k” 一栏里填写你的本地clip模型的绝对路径，使用“/”，需求的文件看下面的文件结构演示。
|
||||
Fill in the absolute path of your local clip model in the "laion/CLIP ViT bigG-14-laion2B-39B-b160k" column, using "/". Please refer to the file structure demonstration below for the required files.
|
||||
```
|
||||
├── ComfyUI/custom_nodes/ComfyUI_Pops/
|
||||
| ├──weights/
|
||||
| ├── photomaker-v1.bin
|
||||
| ├── ms_adapter.bin
|
||||
├── local_path/
|
||||
| ├──CLIP ViT bigG-14-laion2B-39B-b160k/
|
||||
| ├── config.json
|
||||
| ├── preprocessor_config.json
|
||||
| ├──pytorch_model.bin.index.json
|
||||
| ├──pytorch_model-00001-of-00002.bin
|
||||
| ├──pytorch_model-00002-of-00002.bin
|
||||
| ├──special_tokens_map.json
|
||||
| ├──tokenizer.json
|
||||
| ├──tokenizer_config.json
|
||||
| ├──vocab.json
|
||||
```
|
||||
|
||||
3.3
|
||||
在comfyUI的models目录下，确认是否有photomaker 目录，没有会自己新建并下载 photomaker-v1.bin [link](https://huggingface.co/TencentARC/PhotoMaker/tree/main)
|
||||
如果有预下载，就把模型放进去。
|
||||
@@ -110,9 +145,8 @@ make sure ..models/photomaker/photomaker-v1.bin [link](https://huggingface.co
|
||||
4 Example
|
||||
----
|
||||
|
||||
add lora 加入lora的支持
|
||||
|
||||
|
||||
Dual role same frame and lora txt2img 双角色同框并加入Lora，文生图示例
|
||||
|
||||
|
||||
txt2img 文生图
|
||||
|
||||
@@ -120,8 +154,8 @@ txt2img 文生图
|
||||
img2img 图生图
|
||||
|
||||
|
||||
two character 双角色
|
||||
|
||||
two character 双角色及同框
|
||||
|
||||
|
||||
use single model 使用单模型
|
||||
|
||||
@@ -141,15 +175,25 @@ StoryDiffusion
|
||||
author={Zhou, Yupeng and Zhou, Daquan and Cheng, Ming-Ming and Feng, Jiashi and Hou, Qibin},
|
||||
journal={arXiv preprint arXiv:2405.01434},
|
||||
year={2024}
|
||||
|
||||
}
|
||||
```
|
||||
IP-Adapter
|
||||
```
|
||||
python
|
||||
@article{ye2023ip-adapter,
|
||||
title={IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models},
|
||||
author={Ye, Hu and Zhang, Jun and Liu, Sibo and Han, Xiao and Yang, Wei},
|
||||
booktitle={arXiv preprint arxiv:2308.06721},
|
||||
year={2023}
|
||||
|
||||
|
||||
}
|
||||
```
|
||||
MS-Diffusion
|
||||
```
|
||||
@misc{wang2024msdiffusion,
|
||||
title={MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance},
|
||||
author={X. Wang and Siming Fu and Qihan Huang and Wanggui He and Hao Jiang},
|
||||
year={2024},
|
||||
eprint={2406.07209},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CV}
|
||||
}
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user