1、增加 Split_prompt参数，默认是空，如果填入分割标点，推荐用";"，会规范化角色prompt和场景prompt（主要是加入换行符），方便规范化复制或者从其他节点引入的prompts；翻译节点的切分符号默认值改成中文的“；”
1、Add the Split prompt parameter, which is empty by default. If you fill in the split punctuation, it is recommended to use ";". This will standardize the character prompt and scene prompt (mainly by adding line breaks), making it easier to standardize copying or introducing prompts from other nodes;

Notice（节点的特殊功能说明 Special Function Description of Nodes）

1、添加双角色同框功能，使用方法：(A and B) have lunch...., A,B为角色名，中间的 and 和括号不能删除,括号为生效条件！！！
2、因为调用了MS-diffusion的功能，所以要使用双角色同框，必须添加encoder模型（laion/CLIP-ViT-bigG-14-laion2B-39B-b160k,无法替换为其他的）和ip-adapeter微调模型（ms_adapter.bin,无法替换）；
3、优化加载Lora的代码，使用加速Lora时，trigger_words不再加入prompt列表；
4、Playground v2.5可以在txt2img有效，没有Playground v2.5的风格Lora可用，当可以使用加速Lora;
5、因为可调参数太多，特意分出模型加载节点，并删除无用的3角色节点；
6、role_scale，mask_threshold，start_step主要调节双角色同框的随机性和风格一致性；
7、ip_adapter_strength和style_strength_ratio在img2img时，可以调节风格的一致性；
8、预处理翻译文本节点，使用方法可以参考示例图。 (中文或其他东亚文字注意更换字体)；
9、默认用每段文字末尾的";"来切分段落，翻译为中文后，有几率会被翻译为“；”，所以记得改成“；”，否则会是一句话。
10、编辑config/models.yaml文件，记住用同样的格式，可以加入你喜欢的基于SDXL的扩散模型。
11、示例的playground模型仅是测试，但是无法出图，请勿使用。
12、拼图节点支持自定义字体（把字体文件放在fonts目录下 .fonts/your_font.ttf）和字体大小，增加了双角色的支持（使用comfyUI的batch image）；
13、可以使用单体SDXL模型，方法是选择“Use_Single_XL_Model”，然后在ckpt_name菜单选择你想使用的XL模型； 14、加入Lora的支持，lora菜单选择"none"时，Lora无效；“trigger_words”须填入你选择的Lora模型的对应的trigger_words，无须在prompt中加入，插件会自动在每行的末尾加入； 15、支持diffuser 0.28以上版本；
16、图生图流程使用photomaker，角色prompt栏里，必须有img关键词，你可以使用a women img, a man img等；
17、图片不出现角色，场景prompt前面加入[NC] ；
18、分段prompt，用#，例如 AAAA#BBBB,将生成AAAA内容，但是文字只显示BBBB

Add dual role same frame function, usage method: (A and B) have lunch...., where A and B are role names, and the middle "and" parentheses cannot be removed，The parentheses represent the effective conditions!!!
Because the" MS diffusion" function is called, in order to use dual role same frame, it is necessary to add an encoder model (laion/CLIP-ViT-bigG-14-laion2B-39B-b160k,which cannot be replaced with others) and an ip adapet fine-tuning model (ms_adapter.bin,which cannot be replaced);
Optimize the loading of Lora's code, and when using accelerated Lora, trigger_words will no longer be added to the prompt list;
Playground v2.5 can be effective on txt2img, and there is no Playground v2.5 style Lora available when accelerated Lora can be used;
Due to too many adjustable parameters, the model loading node was deliberately separated and the useless 3 role nodes were deleted;
Role-scale, mask_threshold, and start_step mainly regulate the randomness and style consistency of two characters in the same frame;
The consistency of style can be adjusted between ip-adapter_strength and style_strength'ratio in img2img;
Preprocess translation text nodes, please refer to the example diagram for usage methods. (Pay attention to changing the font for Chinese or other East Asian characters);
By default, use the ";" at the end of each paragraph to divide the paragraph. After translation into Chinese, there is a chance that it will be translated as ";", so remember to change it to ";", otherwise it will be a sentence.
Edit the config/models. yaml file and remember to use the same format to include your favorite SDXL based diffusion model.
The playground model in the example is only for testing purposes, but cannot be illustrated. Please do not use it.
The jigsaw puzzle node supports custom fonts (placing font files in the fonts directory".fonts/you_font. ttf") and font sizes, and adds support for dual characters (using batch images in comfyUI);
You can use a single SDXL model by selecting "Use_Single_XL-Model" and then selecting the XL model you want to use from the ckptname menu;
Add support for Lora. When selecting "none" in the Lora menu, Lora becomes invalid; "Trigger_words" must be filled in with the corresponding trigger_words for the Lora model you have selected, without the need to add them in the prompt. The plugin will automatically add them at the end of each line;
Supports diffuser versions 0.28 and above;
The process of generating images using PhotosMaker requires the"img" keyword in the character prompt column. You can use keywords such as a woman img, a man img, etc;
No characters appear in the image, add [NC] in front of the scene prompt;
Segmented prompt, using #, such as AAAA # BBBB, will generate AAAA content, but the text will only display BBBB

1.Installation

In the ./ComfyUI /custom_node directory, run the following:

git clone https://github.com/smthemex/ComfyUI_StoryDiffusion.git

2.requirements

pip install -r requirements.txt

pip install git+https://github.com/tencent-ailab/IP-Adapter.git

如果缺失模块，请单独pip install
If the module is missing, please pip install

2.1如果使用的是comfyUI整合包提示ip_adapter库找不到，可以尝试以下方法：

在整合包的python_embeded目录下，复制插件的requirements.txt文件到这个目录，去掉这个文件里ip-adapter前面的#号，保存，再打开CMD，然后运行pip install -r requirements.txt --target= "你的路径/python_embeded/Lib/site-packages" ，只把你的路径改成你的实际路径，其他不要动

例如：pip install -r requirements.txt --target= "你的路径/python_embeded/Lib/site-packages"

2.2 如果实在装不上，你需要安装python3.10环境，git，以及pip，然后运行pip install git+https://github.com/tencent-ailab/IP-Adapter.git 安装好之后，在你独立安装的python目录下，找到lib/site-packages下的ip-Adapter文件夹，复制到你的/python_embeded/Lib/site-packages里去

2.3秋叶包，理论上是按2.1的方法，如果不行，可以试试，2.2

3 Need model

3.1 online 在线模式
点击运行，会自动从huggingface 下载所需模型，请确保你的的网络通畅，默认可用的模型有G161222/RealVisXL_V4.0 ，stabilityai/stable-diffusion-xl-base-1.0 ， stablediffusionapi/sdxl-unstable-diffusers-y ，sd-community/sdxl-flash ；
选择'Use_Single_XL_Model',以及你本地的SDXL单体模型（例如：Juggernaut-XL_v9_RunDiffusionPhoto_v2.safetensors ），也会下载对应的config文件；

In online mode, click run and the required model will be automatically downloaded from the huggingface. Please ensure that your network is unobstructed. The default available models are G161222/RealVisXL_V4.0, stabilityai/stable-diffusion-xl-base-1.0 ， stablediffusionapi/sdxl-unstable-diffusers-y ，sd-community/sdxl-flash ；
Select 'Use_Single_XL-Model', as well as your local SDXL monomer model (for example: Jumpernaut XL_v9-RunDiffusionPhoto_v2. safetensors), and the corresponding config file will also be downloaded;

--using dual role same frame function(使用双角色功能时):

Need download "ms_adapter.bin" : link Need encoder model "laion/CLIP-ViT-bigG-14-laion2B-39B-b160k":link

├── ComfyUI/custom_nodes/ComfyUI_Pops/
|      ├──weights/
|             ├── photomaker-v1.bin
|             ├── ms_adapter.bin

3.2 offline
打开ComfyUI_StoryDiffusion/config/models.yaml的models.yaml文件，如果有预下载的默认的扩散模型，可以不填，如果地址不在默认的C盘一类，可以在“path”一栏：填写扩散模型的绝对地址，须是“/” .
Open the models.yaml file of ComfyUI_StoryDiffusion/config/models.yaml. If there is a pre downloaded default diffusion model, it can be left blank. If the address is not in the default C drive category, you can fill in the absolute address of the diffusion model in the "path" column, which must be "/"

--using dual role same frame function(使用双角色功能时):
在“laion/CLIP-ViT-bigG-14-laion2B-39B-b160k” 一栏里填写你的本地clip模型的绝对路径，使用“/”，需求的文件看下面的文件结构演示。
Fill in the absolute path of your local clip model in the "laion/CLIP ViT bigG-14-laion2B-39B-b160k" column, using "/". Please refer to the file structure demonstration below for the required files.

├── ComfyUI/custom_nodes/ComfyUI_Pops/
|      ├──weights/
|             ├── photomaker-v1.bin
|             ├── ms_adapter.bin
├── local_path/
|     ├──CLIP ViT bigG-14-laion2B-39B-b160k/
|             ├── config.json
|             ├── preprocessor_config.json
|             ├──pytorch_model.bin.index.json
|             ├──pytorch_model-00001-of-00002.bin
|             ├──pytorch_model-00002-of-00002.bin
|             ├──special_tokens_map.json
|             ├──tokenizer.json
|             ├──tokenizer_config.json
|             ├──vocab.json

3.3 在comfyUI的models目录下，确认是否有photomaker 目录，没有会自己新建并下载 photomaker-v1.bin link
如果有预下载，就把模型放进去。

make sure ..models/photomaker/photomaker-v1.bin link

4 Example

txt2img lora and Dual role same fram 双角色同框并加入Lora，文生图示例

1img2img and lora 图生图加风格lora

two character lighting lora 双角色及闪电lora

using other language 使用其他语言的文本,翻译节点请换成你有的。

Citation

StoryDiffusion

@article{zhou2024storydiffusion,
  title={StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation},
  author={Zhou, Yupeng and Zhou, Daquan and Cheng, Ming-Ming and Feng, Jiashi and Hou, Qibin},
  journal={arXiv preprint arXiv:2405.01434},
  year={2024}
}

IP-Adapter

@article{ye2023ip-adapter,
  title={IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models},
  author={Ye, Hu and Zhang, Jun and Liu, Sibo and Han, Xiao and Yang, Wei},
  booktitle={arXiv preprint arxiv:2308.06721},
  year={2023}
}

MS-Diffusion

@misc{wang2024msdiffusion,
  title={MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance}, 
  author={X. Wang and Siming Fu and Qihan Huang and Wanggui He and Hao Jiang},
  year={2024},
  eprint={2406.07209},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}

README.md Unescape Escape

ComfyUI_StoryDiffusion

StoryDiffusion origin From: link MS-Diffusion origin From: link

My ComfyUI node list：

NEW Update