nodes/smthemex-ComfyUI_StoryDiffusion
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update README.md

Browse Source 
add
This commit is contained in:
smthemex
2024-06-27 09:41:59 +08:00
committed by GitHub
parent 2458d9e623
commit 2dcb3315ee
1 changed files with 117 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+117 -43
View File
@@ -22,50 +22,121 @@ My ComfyUI node list：
NEW Update
---
1、增加 Split_prompt参数，默认是空，如果填入分割标点，推荐用";"，会规范化角色prompt和场景prompt（主要是加入换行符），方便规范化复制或者从其他节点引入的prompts； 翻译节点的切分符号默认值改成中文的“；”
1、Add the Split prompt parameter, which is empty by default. If you fill in the split punctuation, it is recommended to use ";". This will standardize the character prompt and scene prompt (mainly by adding line breaks), making it easier to standardize copying or introducing prompts from other nodes;
--加入角色模型保存和加载功能
--优化模型加载逻辑
--已知文生图加双角色同图时，只能跑一次，再次跑，需要改一下模型加载的采样器或者别的选项的bug，暂时没时间修复；
--Add the function of saving and loading character models
--Optimize model loading logic
--It is known that when adding dual characters to the Wensheng diagram, it can only be run once. If you run it again, you need to fix the bug in the sampler or other options loaded on the model. There is currently no time to fix it;
Notice（节点的特殊功能说明 Special Function Description of Nodes）
Notice（节点的菜单功能说明 Special Function Description of Nodes Menu）
---
1、添加双角色同框功能，使用方法：(A and B) have lunch...., A,B为角色名，中间的 and 和括号不能删除,括号为生效条件！！！
2、因为调用了MS-diffusion的功能，所以要使用双角色同框，必须添加encoder模型（laion/CLIP-ViT-bigG-14-laion2B-39B-b160k,无法替换为其他的）和ip-adapeter微调模型（ms_adapter.bin,无法替换）；
3、优化加载Lora的代码，使用加速Lora时，trigger_words不再加入prompt列表；
4、Playground v2.5可以在txt2img有效，没有Playground v2.5的风格Lora可用，当可以使用加速Lora;
5、因为可调参数太多，特意分出模型加载节点，并删除无用的3角色节点；
6、role_scale，mask_threshold，start_step主要调节双角色同框的随机性和风格一致性；
7、ip_adapter_strength和style_strength_ratio在img2img时，可以调节风格的一致性；
8、预处理翻译文本节点，使用方法可以参考示例图。 (中文或其他东亚文字注意更换字体)；
9、默认用每段文字末尾的";"来切分段落，翻译为中文后，有几率会被翻译为“；”，所以记得改成“；”，否则会是一句话。
10、编辑config/models.yaml文件，记住用同样的格式，可以加入你喜欢的基于SDXL的扩散模型。
11、示例的playground模型仅是测试，但是无法出图，请勿使用。
12、拼图节点支持自定义字体（把字体文件放在fonts目录下 .fonts/your_font.ttf）和字体大小，增加了双角色的支持（使用comfyUI的batch image）；
13、可以使用单体SDXL模型，方法是 选择“Use_Single_XL_Model”，然后在ckpt_name菜单选择你想使用的XL模型；
14、加入Lora的支持，lora菜单选择"none"时，Lora无效；“trigger_words”须填入你选择的Lora模型的对应的trigger_words，无须在prompt中加入，插件会自动在每行的末尾加入；
15、支持diffuser 0.28以上版本；
16、图生图流程使用photomaker，角色prompt栏里，必须有img关键词，你可以使用a women img, a man img等；
17、图片不出现角色，场景prompt前面加入[NC] ；
18、分段prompt，用#，例如 AAAA#BBBB,将生成AAAA内容，但是文字只显示BBBB
--<Storydiffusion_Model_Loader>
-- sd_type：选择“Use_Single_XL_Model”时，可以使用社区SDXL模型，其他选项均为扩散模型；
-- ckpt_name：使用“Use_Single_XL_Model”时生效，社区SDLX模型选择；
-- character_weights：使用sampler节点的save_character 功能保存的角色权重。选择为“none/无”时不生效！（注意，保存的角色权重不能马上被识别，需要重启comfyUI）；
-- lora：选择SDXL lora，为“none”时不生效；
-- lora_scale： lora的权重，Lora生效时启用；
-- trigger_words： lora的关键词，会自动添加到prompt里，启用Lora时，请填写Lora对应的trigger_words；
-- scheduler： 采样器选择，文生图加角色同框时，如果连续跑，会报错，这时候，改一个采样器，就能继续跑，这个bug暂时没修复；
-- model_type： 选择txt2img 或者img2img模式，使用txt2img模式时采样器可以不接入图片；
-- id_number： 使用多少个角色，目前仅支持1个或者2个；
-- sa32_degree/sa64_degree： 注意力层的可调参数；
--img_width/img_height： 出图的高宽尺寸。
--<Storydiffusion_Sampler>
-- pipe/info： 必须链接的接口；
--image： 图生图才必须链接的接口，双角色请按示例，用comfyUI内置的image batch 节点；
--character_prompt： 角色的prompt，[角色名] 必须在开头，如果使用图生图模式，必须加入“img”关键词，例如 a man img；
--scene_prompts： 场景描述的prompt，[角色名] 必须在开头，2个角色最好在前两行各自出现一次，[NC]在开头时，角色不出现（适合无角色场景），(角色A and 角色B) 时开启MS-diffusion的双角色模式，and 和其前后空格不能忽略； #用于分段prompt，渲染整段，但是将只输出#后面的prompt；
--split_prompt： 切分prompt的符号,为空时不生效，用于prompt为外置时的规范化段落。比如你传入10行的文字时，分段符不一定正确，但是用切分符号，比如“；”就能很好的区分每一行。
--negative_prompt： 只在img_style为No_style时生效；
--seed/steps/cfg： 适用于comfyUI常用功能；
--ip_adapter_strength： img2img 图生图的ip_adapter权重控制；
--style_strength_ratio： 风格权重控制，控制风格从哪一步开始生效，风格一致性不好时，可以试着调高或者调低此参数；
--encoder_repo： 仅在双角色同图时有效，如果要使用本地模型，务必使用X:/XXX/XXX/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k 必须是“/”；
--role_scale： 仅在双角色同图时有效，控制角色在图片中的权重；
--mask_threshold： 仅在双角色同图时有效，控制角色在图片中的位置（MS系统自动根据prompt分配角色位置，所以prompt中可以加入适当的角色位置信息描写）；
-- start_step： 仅在双角色同图时有效，控制角色在图片中的位置的起始步数;
--save_character： 是否保存当前角色的角色权重，文件在./ComfyUI_StoryDiffusion/weigths/pt 下，以时间为文件名 ；
1. Add dual role same frame function, usage method: (A and B) have lunch...., where A and B are role names, and the middle "and" parentheses cannot be removed，The parentheses represent the effective conditions!!!
2. Because the" MS diffusion" function is called, in order to use dual role same frame, it is necessary to add an encoder model (laion/CLIP-ViT-bigG-14-laion2B-39B-b160k,which cannot be replaced with others) and an ip adapet fine-tuning model (ms_adapter.bin,which cannot be replaced);
3. Optimize the loading of Lora's code, and when using accelerated Lora, trigger_words will no longer be added to the prompt list;
4. Playground v2.5 can be effective on txt2img, and there is no Playground v2.5 style Lora available when accelerated Lora can be used;
5. Due to too many adjustable parameters, the model loading node was deliberately separated and the useless 3 role nodes were deleted;
6. Role-scale, mask_threshold, and start_step mainly regulate the randomness and style consistency of two characters in the same frame;
7. The consistency of style can be adjusted between ip-adapter_strength and style_strength'ratio in img2img;
8. Preprocess translation text nodes, please refer to the example diagram for usage methods. (Pay attention to changing the font for Chinese or other East Asian characters);
9. By default, use the ";" at the end of each paragraph to divide the paragraph. After translation into Chinese, there is a chance that it will be translated as ";", so remember to change it to ";", otherwise it will be a sentence.
10. Edit the config/models. yaml file and remember to use the same format to include your favorite SDXL based diffusion model.
11. The playground model in the example is only for testing purposes, but cannot be illustrated. Please do not use it.
12. The jigsaw puzzle node supports custom fonts (placing font files in the fonts directory".fonts/you_font. ttf") and font sizes, and adds support for dual characters (using batch images in comfyUI);
13. You can use a single SDXL model by selecting "Use_Single_XL-Model" and then selecting the XL model you want to use from the ckptname menu;
14. Add support for Lora. When selecting "none" in the Lora menu, Lora becomes invalid; "Trigger_words" must be filled in with the corresponding trigger_words for the Lora model you have selected, without the need to add them in the prompt. The plugin will automatically add them at the end of each line;
15. Supports diffuser versions 0.28 and above;
16. The process of generating images using PhotosMaker requires the"img" keyword in the character prompt column. You can use keywords such as a woman img, a man img, etc;
17. No characters appear in the image, add [NC] in front of the scene prompt;
18. Segmented prompt, using #, such as AAAA # BBBB, will generate AAAA content, but the text will only display BBBB
--<Comic_Type>
--fonts_list： 拼图节点支持自定义字体（把字体文件放在fonts目录下 .fonts/your_font.ttf）；
--text_size： 拼图文字的大小；
--comic_type： 拼图的风格展示；
--split_lines： 适用于非英语文字被其他翻译节点翻译后，换行符被删除，此时使用切分符号，可以正确地重新赋予prompt换行符，确保文字描述在正确的图片上显示；
--<Pre_Translate_prompt> ： 翻译节点的前置处理
--keep_character_name： 是否保留角色名在后续文字拼图上显示。
--<Storydiffusion_Model_Loader>
--Sd_type: When selecting "UseSingle_XL-Model", the community SDXL model can be used, and all other options are diffusion models;
--Ckptname: Effective when using "UsesSingle_XL-Model", community SDLX model selection;
--Character_weights: Character weights saved using the save_character feature of the sampler node. Selecting "none/none" does not take effect! (Note that the saved character weights cannot be immediately recognized and require a restart of comfyUI);
--Lora: Selecting SDXL Lora does not take effect when set to "none";
--Lora_scale: The weight of Lora, which is enabled when Lora takes effect;
--Trigger_words: The keyword for Lora will be automatically added to the prompt. When enabling Lora, please fill in the corresponding trigger_words for Lora;
--Scheduler: When selecting a sampler and adding characters to the same frame in the text and animation, if running continuously, an error will be reported. At this time, changing a sampler can continue running, but this bug has not been fixed yet;
--Model_type: Select either the txt2img or img2img mode, and when using the txt2img mode, the sampler may not be connected to the image;
--Idnumber: How many roles are used, currently only supporting 1 or 2;
--Sa32_degree/sa64_degree: an adjustable parameter for the attention layer;
--Img_width/img_height: The height and width dimensions of the drawing.
--<Storydiffusion_Sampler>
--Pipe/info: The interface that must be linked;
--Image: The interface that must be linked to the image generation diagram. For dual roles, please follow the example and use the built-in image batch node in comfyUI;
--Character prompt: The prompt for the character, [character name] must be at the beginning. If using the graphic mode, the keyword "img" must be added, such as a man img;
--Scene prompts: The prompt for the scene description, [character name], must start at the beginning. It is best for both characters to appear once in the first two lines. [NC] At the beginning, the character does not appear (suitable for non character scenes). When (character A and character B), MS diffusion's dual character mode is enabled, and and the spaces before and after it cannot be ignored# Used for segmented prompt, rendering the entire segment, but only outputting the prompt after #;
--Split prompt: The symbol for splitting the prompt, which does not take effect when it is empty. It is used to normalize paragraphs when the prompt is external. For example, when you pass in 10 lines of text, the hyphen may not be correct, but using a hyphen, such as ";", can effectively distinguish each line.
--Negative prompt: only effective when img_style is No_style;
--Seed/steps/cfg: suitable for commonly used functions in comfyUI;
--Ip-adapter_strength: img2img controls the weight of ip-adapter in graph generation;
--Style_strength'ratio: Style weight control, which controls from which step the style takes effect. When the style consistency is not good, you can try increasing or decreasing this parameter;
--Encoder'repo: Only valid when two characters are in the same image. If you want to use a local model, be sure to use X:/XXX/XXX/laion/CLIP ViT bigG-14-laion2B-39B-b160k, which must be "/";
--Role-scale: only effective when two characters are in the same image, controlling the weight of the characters in the image;
--Mask_threshold: It is only effective when two roles are in the same picture, and controls the position of the role in the picture (MS system automatically assigns the role position according to prompt, so appropriate role position information description can be added to prompt);
--Start_step: Only effective when two characters are in the same image, controlling the number of starting steps for the character's position in the image
--Save_character: Whether to save the character weights of the current character, file in/ Under ComfyUI_StoryDiffusion/weights/pt, use time as the file name;
--<Comic_Type>
--Fonts list: The puzzle node supports custom fonts (place the font file in the fonts directory. fonts/you_font. ttf);
--Text_size: The size of the puzzle text;
--Comic_type: Display the style of the puzzle;
--Split lines: Suitable for non English text that has been translated by other translation nodes and the line break is removed. In this case, using a split symbol can correctly reassign the prompt line break to ensure that the text description is displayed on the correct image;
--<Pre_Translate_prompt>: Pre processing of translation nodes
--Keep_charactername: Whether to keep the character name displayed on subsequent text puzzles.
Tips 提醒：
--添加双角色同框功能，使用方法：(A and B) have lunch...., A,B为角色名，中间的 and 和括号不能删除,括号为生效条件！！！
--因为调用了MS-diffusion的功能，所以要使用双角色同框，必须添加encoder模型（laion/CLIP-ViT-bigG-14-laion2B-39B-b160k,无法替换为其他的）和ip-adapeter微调模型（ms_adapter.bin,无法替换）；
--优化加载Lora的代码，使用加速Lora时，trigger_words不再加入prompt列表；
--Playground v2.5可以在txt2img有效，没有Playground v2.5的风格Lora可用，当可以使用加速Lora;
--role_scale，mask_threshold，start_step主要调节双角色同框的随机性和风格一致性；
--ip_adapter_strength和style_strength_ratio在img2img时，可以调节风格的一致性；
--预处理翻译文本节点，使用方法可以参考示例图。 (中文或其他东亚文字注意更换字体)；
--默认用每段文字末尾的";"来切分段落，翻译为中文后，有几率会被翻译为“；”，所以记得改成“；”，否则会是一句话。
--编辑config/models.yaml文件，记住用同样的格式，可以加入你喜欢的基于SDXL的扩散模型。
--支持diffuser 0.28以上版本；
--图生图流程使用photomaker，角色prompt栏里，必须有img关键词，你可以使用a women img, a man img等；
--图片不出现角色，场景prompt前面加入[NC] ；
--分段prompt，用#，例如 AAAA#BBBB,将生成AAAA内容，但是文字只显示BBBB
--Add dual character same frame function, usage method: (A and B) have lunch, A. B is the role name, and the middle and parentheses cannot be removed. The parentheses are the effective conditions!!!
--Because the MS diffusion function was called, in order to use dual role same frame, it is necessary to add an encoder model (laion/CLIP ViT bigG-14 laion2B-39B-b160k, which cannot be replaced with others) and an ip adapet fine-tuning model (ms-adapter.bin, which cannot be replaced);
--Optimize the loading of Lora's code, and when using accelerated Lora, trigger_words will no longer be added to the prompt list;
--Playground v2.5 can be effective on txt2img, and there is no Playground v2.5 style Lora available when accelerated Lora can be used;
--Role-scale, mask_threshold, and start_step mainly regulate the randomness and style consistency of two characters in the same frame;
--The consistency of style can be adjusted between ip-adapter_strength and style_strength'ratio in img2img;
--Preprocess translation text nodes, please refer to the example diagram for usage methods. (Pay attention to changing the font for Chinese or other East Asian characters);
--By default, use the ";" at the end of each paragraph to divide the paragraph. After translation into Chinese, there is a chance that it will be translated as ";", so remember to change it to ";", otherwise it will be a sentence.
--Edit the config/models. yaml file and remember to use the same format to include your favorite SDXL based diffusion model.
--Supports diffuser versions 0.28 and above;
--The process of generating images using PhotosMaker requires the IMG keyword in the character prompt column. You can use keywords such as a woman IMG, a man IMG, etc;
--No characters appear in the image, add [NC] in front of the scene prompt;
--Segmented prompt, using #, such as AAAA # BBBB, will generate AAAA content, but the text will only display BBBB
1.Installation
@@ -114,7 +185,7 @@ Need download "ms_adapter.bin" : [link](https://huggingface.co/doge1516/MS-Diffu
Need encoder model "laion/CLIP-ViT-bigG-14-laion2B-39B-b160k":[link](https://huggingface.co/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k)
```
├── ComfyUI/custom_nodes/ComfyUI_Pops/
├── ComfyUI/custom_nodes/ComfyUI_StoryDiffusion/
| ├──weights/
| ├── photomaker-v1.bin
| ├── ms_adapter.bin
@@ -129,11 +200,11 @@ Open the models.yaml file of ComfyUI_StoryDiffusion/config/models.yaml. If there
在“laion/CLIP-ViT-bigG-14-laion2B-39B-b160k” 一栏里填写你的本地clip模型的绝对路径，使用“/”，需求的文件看下面的文件结构演示。
Fill in the absolute path of your local clip model in the "laion/CLIP ViT bigG-14-laion2B-39B-b160k" column, using "/". Please refer to the file structure demonstration below for the required files.
```
├── ComfyUI/custom_nodes/ComfyUI_Pops/
├── ComfyUI/custom_nodes/ComfyUI_StoryDiffusion/
| ├──weights/
| ├── photomaker-v1.bin
| ├── ms_adapter.bin
├── local_path/
├── Any local_path/
| ├──CLIP ViT bigG-14-laion2B-39B-b160k/
| ├── config.json
| ├── preprocessor_config.json
@@ -154,6 +225,9 @@ make sure ..models/photomaker/photomaker-v1.bin [link](https://huggingface.co
4 Example
----
img2img 图生图最新的流程
![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/img2img.png)
txt2img lora and Dual role same fram 双角色同框并加入Lora，文生图示例
![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/txt2imglora2role.png)