ComfyUI_StoryDiffusion

Using StoryDiffusion and other methods to make storys in ComfyUI ---- * Origin from [StoryDiffusion](https://github.com/HVision-NKU/StoryDiffusion) [MS-Diffusion](https://github.com/MS-Diffusion/MS-Diffusion),[StoryMaker](https://github.com/RedAIGC/StoryMaker)，[Consistory](https://github.com/NVlabs/consistory),[Kolor](https://github.com/Kwai-Kolors/Kolors),[Pulid](https://github.com/ToTheBeginning/PuLID),[Flux](https://github.com/black-forest-labs/flux),[photomaker](https://github.com/TencentARC/PhotoMaker),[IP-Adapter](https://github.com/tencent-ailab/IP-Adapter) [InfiniteYou](https://github.com/bytedance/InfiniteYou) ## Updates: **2025/04/05** * 1.修改了模型加载的流程，更新到V2版本，如果你喜欢旧的，可以下载V1.0版本的,2.请使用example.json，它集成了主要的工作流;3.剔除掉一些过时的功能; * 1.Modified the model loading process.Update to V2 version, If you like the old one, you can download version 1.0，2.Please use 'example.json', which integrates the main workflow，3.Remove some outdated features; 1.Installation ----- In the ./ComfyUI /custom_node directory, run the following: ``` git clone https://github.com/smthemex/ComfyUI_StoryDiffusion.git ``` 2.requirements ---- ``` pip install -r requirements.txt ``` * 使用story(photomaker V2)，pulid-flux，kolor，story-maker，infiniteyou，时需要安装insightface库。if using story(photomaker V2)，pulid-flux，kolor，story-make，infiniteyou: ``` pip install insightface ``` * If the module is missing, please pip install，缺什么库就装什么。 3 models ---- **3.1 stroy _diffusion mode （单纯故事）** * 3.1.1 any sdxl checkpoints 任意SDXL单体模型 ``` ├── ComfyUI/models/checkpoints/ | ├── juggernautXL_v8Rundiffusion.safetensors ``` * 3.1.2 如果使用图生图 if image to image 下载 download [photomaker-v1.bin](https://huggingface.co/TencentARC/PhotoMaker/tree/main) or 或者 [photomaker-v2.bin](https://huggingface.co/TencentARC/PhotoMaker-V2/tree/main) ``` ├── ComfyUI/models/photomaker/ | ├── photomaker-v1.bin or photomaker-v2.bin ``` **3.2 MS-diffusion mode（2 role in 1 imag 双角色同框）** * 3.2.1下载 download: [ms_adapter.bin](https://huggingface.co/doge1516/MS-Diffusion/tree/main) ``` ├── ComfyUI/models/ | ├── photomaker/ms_adapter.bin | ├── clip_vision/clip_vision_g.safetensors(2.35G) or CLIP-ViT-bigG-14-laion2B-39B-b160k.safetensors(3.43G) ``` * 3.2.2 用cn则需要对应的cn的controlnet模型。 if using controlnet in ms-diffusion(Control_img image preprocessing, please use other nodes ); ``` ├── ComfyUI/models/controlnet/ | ├──xinsir/controlnet-openpose-sdxl-1.0 | ├──... 其他类似的 ``` **3.3 kolors face mode（不再支持IP，已修复高版本错误）** * [Kwai-Kolors](https://huggingface.co/Kwai-Kolors/Kolors/tree/main) #不用全下，除了config文件，只需要下载unet和vae模型；下载 download [Kolors-IP-Adapter-FaceID-Plus](https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-FaceID-Plus) 下载 download[chatglm3-8bit.safetensors](https://huggingface.co/Kijai/ChatGLM3-safetensors/tree/main) or fp16 下载KJ的单体clip模型；自动下载"DIAMONIK7777/antelopev2"，will auto download "DIAMONIK7777/antelopev2" insightface models.... ``` ├── ComfyUI/models | ├── /photomaker/ipa-faceid-plus.bin | ├── clip/chatglm3-8bit.safetensors | ├── clip_vision/clip-vit-large-patch14.safetensors # Kolors-IP-Adapter-Plus or Kolors-IP-Adapter-FaceID-Plus using same checkpoints. ``` * kolors的repo文件结构 ``` ├── any path/Kwai-Kolors/Kolors | ├──model_index.json | ├──vae | ├── config.json | ├── diffusion_pytorch_model.safetensors (rename from diffusion_pytorch_model.fp16.safetensors ) | ├──unet | ├── config.json | ├── diffusion_pytorch_model.safetensors (rename from diffusion_pytorch_model.fp16.safetensors ) | ├──tokenizer | ├── tokenization_chatglm.py ##新版，修复高版本diffuser错误 | ├── ... #all 所有文件 | ├── text_encoder | ├── modeling_chatglm.py #新版，修复高版本diffuser错误 | ├── tokenization_chatglm.py ##新版，修复高版本diffuser错误 | ├── ... #all 所有文件 | ├── scheduler | ├── scheduler_config.json ``` **3.4 flux_pulid mode** . * torch must > 0.24.0 optimum-quanto must >=0.2.4 ``` pip install -U optimum-quanto ``` * 下载 download[EVA02_CLIP_L_336_psz14_s6B.pt](https://huggingface.co/QuanSun/EVA-CLIP/tree/main) and [pulid_flux_v0.9.0.safetensors](https://huggingface.co/guozinan/PuLID/tree/main) and [flux1-dev-fp8.safetensors](https://huggingface.co/Kijai/flux-fp8/tree/main) ， 自动下载 auto downlaod [DIAMONIK7777/antelopev2](https://huggingface.co/DIAMONIK7777/antelopev2/tree/main) ``` ├── ComfyUI/models/ | ├── photomaker/pulid_flux_v0.9.0.safetensors | ├── clip_vision/EVA02_CLIP_L_336_psz14_s6B.pt | ├── diffusion_models/flux1-dev-fp8.safetensors ├── ComfyUI/models/clip/ | ├── t5xxl_fp8_e4m3fn.safetensors | ├── clip_l.safetensors ``` **3.5 storymake mode** 下载 download [mask.bin](https://huggingface.co/RED-AIGC/StoryMaker/tree/main)#可以自动下载 [buffalo_l](https://huggingface.co/RED-AIGC/StoryMaker/tree/main)#自动下载 [RMBG-1.4](https://huggingface.co/briaai/RMBG-1.4/tree/main)#自动下载 ``` ├── ComfyUI/models/ | ├── photomaker/mask.bin | ├── clip_vision/clip_vision_H.safetensors #2.4G base in laion/CLIP-ViT-H-14-laion2B-s32B-b79K ├── ComfyUI/models/buffalo_l/ | ├── 1k3d68.onnx | ├── ... ``` **3.6 InfiniteYou mode** * 3.6.1 flux transformer repo or kj fp8 ``` ├── any_path/FLUX.1-dev/transformer | ├── config.json | ├──diffusion_pytorch_model-00001-of-00003.safetensors | ├──diffusion_pytorch_model-00002-of-00003.safetensors | ├──diffusion_pytorch_model-00003-of-00003.safetensors | ├── diffusion_pytorch_model.safetensors.index.json ``` or ``` ├── ComfyUI/models/ | ├── diffusion_models/flux1-dev-fp8.safetensors # ``` * 3.6.2 infinite controlnet from [here ](https://huggingface.co/ByteDance/InfiniteYou) ,you can use sim_stage1 or aes_stage2,必要模型，repo格式 ``` ├── any_path/sim_stage1/ | ├── image_proj_model.bin | ├── InfuseNetModel/ | ├── diffusion_pytorch_model-00001-of-00002.safetensors | ├── diffusion_pytorch_model-00002-of-00002.safetensors | ├── diffusion_pytorch_model.safetensors.index.json | ├── config.json ``` or ``` ├── any_path/aes_stage2/ | ├── ... ``` * 3.6.3 lora optional from [here](https://huggingface.co/ByteDance/InfiniteYou) * 3.6.4 insightface ``` ├── ComfyUI/models/antelopev2/ | ├──1k3d68.onnx | ├──... ``` * 3.6.5 recognition_arcface_ir_se50.pth from [here](https://github.com/xinntao/facexlib/releases/download/v0.1.0/recognition_arcface_ir_se50.pth) auto download,which embeded comfyui in "Lib\site-packages\facexlib\weights" dir * 3.6.6 if use gguf quatization (optional) download gguf from [here](https://huggingface.co/city96/FLUX.1-dev-gguf/tree/main),and fill local path in 'easyfunction_lite' node's 'select_method' ``` ├── ComfyUI/models/gguf | ├── flux1-dev-Q8_0.gguf #flux1-dev-Q6_K.gguf ``` * 3.6.7 if use svdquant(optional) download svdquant repo from [here](https://huggingface.co/mit-han-lab/svdq-fp4-flux.1-dev/tree/main) and fill local path in 'easyfunction_lite' node's 'select_method' 4 Example ---- **4.1 story-diffusion** * txt2img 文生图示例 * img2img 图生图示例 **4.2 ms-diffusion** * txt2img 文生图 双角色同框 * img2img 图生图 双角色同框 **4.3 story-maker or story-and-maker** * story-and-maker * story-maker **4.4 consistory** * only one role 只支持单角色 use example.json **4.5 kolor-face** * img2img kolor face,图生图 **4.6 pulid-flux** * 注意示例图片的repo模式已取消，使用 example.json的流程 **4.7 infiniteyou** * repo nf4 注意节点有修改，按example.json的流程 * gguf * svdq，注意能跑，但是ID无法迁移 **4.8 comfyUI classic（comfyUI经典模式，可以接任意适配CF的流程，主要是方便使用多角色的clip）** * any mode SD1.5 SDXL SD3.5 FLUX... 5 Citation ------ StoryDiffusion ``` python @article{zhou2024storydiffusion, title={StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation}, author={Zhou, Yupeng and Zhou, Daquan and Cheng, Ming-Ming and Feng, Jiashi and Hou, Qibin}, journal={arXiv preprint arXiv:2405.01434}, year={2024} } ``` IP-Adapter ``` @article{ye2023ip-adapter, title={IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models}, author={Ye, Hu and Zhang, Jun and Liu, Sibo and Han, Xiao and Yang, Wei}, booktitle={arXiv preprint arxiv:2308.06721}, year={2023} } ``` MS-Diffusion ``` @misc{wang2024msdiffusion, title={MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance}, author={X. Wang and Siming Fu and Qihan Huang and Wanggui He and Hao Jiang}, year={2024}, eprint={2406.07209}, archivePrefix={arXiv}, primaryClass={cs.CV} } ``` photomaker ``` @inproceedings{li2023photomaker, title={PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding}, author={Li, Zhen and Cao, Mingdeng and Wang, Xintao and Qi, Zhongang and Cheng, Ming-Ming and Shan, Ying}, booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, year={2024} } ``` kolors ``` @article{kolors, title={Kolors: Effective Training of Diffusion Model for Photorealistic Text-to-Image Synthesis}, author={Kolors Team}, journal={arXiv preprint}, year={2024} } ``` PuLID ``` @article{guo2024pulid, title={PuLID: Pure and Lightning ID Customization via Contrastive Alignment}, author={Guo, Zinan and Wu, Yanze and Chen, Zhuowei and Chen, Lang and He, Qian}, journal={arXiv preprint arXiv:2404.16022}, year={2024} } ``` Consistory ``` @article{tewel2024training, title={Training-free consistent text-to-image generation}, author={Tewel, Yoad and Kaduri, Omri and Gal, Rinon and Kasten, Yoni and Wolf, Lior and Chechik, Gal and Atzmon, Yuval}, journal={ACM Transactions on Graphics (TOG)}, volume={43}, number={4}, pages={1--18}, year={2024}, publisher={ACM New York, NY, USA} } ``` infiniteyou ``` @article{jiang2025infiniteyou, title={{InfiniteYou}: Flexible Photo Recrafting While Preserving Your Identity}, author={Jiang, Liming and Yan, Qing and Jia, Yumin and Liu, Zichuan and Kang, Hao and Lu, Xin}, journal={arXiv preprint}, volume={arXiv:2503.16418}, year={2025} } ``` svdquant ``` @inproceedings{ li2024svdquant, title={SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models}, author={Li*, Muyang and Lin*, Yujun and Zhang*, Zhekai and Cai, Tianle and Li, Xiuyu and Guo, Junxian and Xie, Enze and Meng, Chenlin and Zhu, Jun-Yan and Han, Song}, booktitle={The Thirteenth International Conference on Learning Representations}, year={2025} } ``` [GGUF](https://github.com/city96/ComfyUI-GGUF) [FLUX LICENSE](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md)