From 7f2fb6ba962fe6ee2cd4c7b85d9b69bc4f57eb99 Mon Sep 17 00:00:00 2001 From: smthemex <138738845+smthemex@users.noreply.github.com> Date: Sun, 15 Sep 2024 19:56:29 +0800 Subject: [PATCH] Update README.md --- README.md | 167 ++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 113 insertions(+), 54 deletions(-) diff --git a/README.md b/README.md index eb6073c..3562225 100644 --- a/README.md +++ b/README.md @@ -6,26 +6,27 @@ You can using StoryDiffusion in ComfyUI. ## Updates: -**2024/09/06** -*fix runway error,when loader single model. +**2024/09/15** +* Happy Mid-autumn Festival！！！ +* Add " PulID FLUX " function, In my testing, a minimum of 12GB of VRAM can run normally,now you can fill "X:/xxx/xxx/black-forest-labs/FLUX.1-dev",and fill easy function "pilid,fp8,cpu"(if you Vram>30G,can remove cpu,and using Kijai/flux-fp8,if you Vram>45G,can remove fp8,cpu), although it is a bit slow if using cpu! ,Of course, some models need to be prepared, as detailed in the following text; + +* Add kolor FaceId function, now you can fill "xxx:/xxx/xxx/Kwai-Kolors/Kolors",and fill easy function "face",Of course, some models need to be prepared, as detailed in the following text; **Previous updates：** -*The loading speed of the NF4 model is many times faster than FP8, so I recommend using the NF4 model to run Flux. I have included the workflow of NF4 in the example， -* when easy_function fill in NF4 or nf4 ,can try NF4 FLUX ,need download weights at [link](https://huggingface.co/sayakpaul/flux.1-dev-nf4/tree/main),put weight in "comfyui/models/checkpoints/"; +* Add diffusers'img2img codes( Not commit diffusers yet),Now you can using flux img2img function. in flux img2img,"guidance_scale" is usually 3.5 ,you can change ip-adapter_strength's number to Control the noise of the output image, the closer the number is to 1, the less it looks like the original image, and the closer it is to 0, the more it looks like the original image. Correspondingly, your generated step count is a multiple of this value, which means that if you enter 50 steps, the actual number of steps taken is 50 * 0.8 (0.8 is the value of change ip-adapter_strength) #you can see exmaple img +* AWPortrait-FL-fp8.safetensors is support if using fp8 mode,.. +* using img crop to fix ms_diffusion only using square's error; +* change W and H global names,it cause some error; +* fix runway error,when loader single model. +* The loading speed of the NF4 model is many times faster than FP8, so I recommend using the NF4 model to run Flux. I have included the workflow of NF4 in the example， * Add an "easy_function" for debugging new function. This time, I have added support for "auraface" in "Photomake V2". You can enter "auraface" into the "easy_function" to test this method -* Flux fp8 single weights only support "Kijai/flux-fp8" or auto save's pt weights. -* fix some bug,Changing the loading method of Msdiffusion, reducing meaningless buttons, and removing the int4 mode of flux did not make it much faster; -* support mps again... -* Now clip checkpoints no need diffusers_repo,you can using "clip_g.safetensors" or other base from "CLIP-ViT-bigG-14-laion2B-39B-b160k"; -* Refactoring some code about MS-diffusion,controlnet still normal quality; +* support Flux ,1.only using fp8 repo,fill local "X:/xxx/xxx/black-forest-labs/FLUX.1-dev" and ckpt_name="none";if save .pt,fill easy function "save",2. using unet,fill local repo and choice a fp8 Unet(like Kijai/flux-fp8, AWPortrait-FL-fp8.safetensors. *.pt), 3. using fn4,need download weights at [link](https://huggingface.co/sayakpaul/flux.1-dev-nf4/tree/main),put weight in "comfyui/models/checkpoints/";fill local "X:/xxx/xxx/black-forest-labs/FLUX.1-dev" +* Now clip checkpoints no need diffusers_repo,you can using "clip_g.safetensors" or other base from "CLIP-ViT-bigG-14-laion2B-39B-b160k"; * 2 role in 1 img now using [A]...[B]... mode, -* if first using flux repo,will automatically save the PT file on checkpoint dir(name: transform_time.pt),So you only need to run the repo and flux once separately (without completing it) to obtain the PT model, Recommend using "repo+transfomer.pt" or "repo+other fp8.safetensors" or "repo+any_name.pt(rename from transfomer )", -* 2 ways to using flux，using repo like :"black-forest-labs/FLUX.1-dev" or "X:/xxx/xxx/black-forest-labs/FLUX.1-dev" and ckpt_name="none" in new loader node or old,or using repo like :"black-forest-labs/FLUX.1-dev" or "X:/xxx/xxx/black-forest-labs/FLUX.1-dev",and using single ckpt like "flux1-dev-fp8.safetensors"; * Support "kolors" text2img and "kolors"ipadapter img2img,using repo like :"xxx:/xxx/xxx/Kwai-Kolors/Kolors" (Please refer to the end of the article for detailed file combinations) * support photomaker V2; * ControlNet now uses community models. * The base model now has only two options: using repo input or selecting the community model... -* To fix the bug where MS diffusion cannot run continuously in the txt2img, it is necessary to enable the "reset_txt2img" of the loading model node to be Ture; * Introducing Controlnet for dual character co image, supporting multi image introduction, * Add the function of saving and loading character models @@ -49,7 +50,7 @@ If the module is missing, please pip install 3 Need model ---- -3.1.1base +3.1.1base: You can directly fill in the repo, such as:"stablityai/table diffusion xl base-1.0", or select the corresponding model in the local diffuser menu (provided that you have the model in the "models/diffuser" directory), or you can directly select a single SDXL community model. The priority of repo or local diffusers is higher than that of individual community models. Supports all SDXL based diffusion models (such as "G161222/RealVisXL_V4.0", "sd-community/sdxl-flash"）， It also supports non SD models, such as ("stablediffusionapi/sdxl-unstable-diffusers-y", playground-v2.5-1024px-aesthetic） @@ -79,6 +80,8 @@ Need clip_vision model "clip_g.safetensors" or other base from "CLIP-ViT-bigG-14 3.2 if using kolors: Kwai-Kolors [link](https://huggingface.co/Kwai-Kolors/Kolors/tree/main) Kolors-IP-Adapter-Plus [link](https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-Plus/tree/main) +Kolors-IP-Adapter-FaceID-Plus [link](https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-FaceID-Plus) + The file structure is shown in the following figure: ``` ├── any path/Kwai-Kolors/Kolors @@ -107,7 +110,7 @@ The file structure is shown in the following figure: | ├── pytorch_model-00001-of-00007.bin to pytorch_model-00007-of-00007.bin | ├── scheduler | ├── scheduler_config.json -| ├── Kolors-IP-Adapter-Plus +| ├── Kolors-IP-Adapter-Plus # if using Kolors-IP-Adapter-Plus | ├──model_index.json | ├──ip_adapter_plus_general.bin | ├──config.json @@ -118,6 +121,22 @@ The file structure is shown in the following figure: | ├──tokenizer.json | ├──tokenizer_config.json | ├──vocab.json +| ├── clip-vit-large-patch14-336 # if using Kolors-IP-Adapter-FaceID-Plus +| ├──config.json +| ├──merges.txt +| ├──preprocessor_config.json +| ├──pytorch_model.bin +| ├──special_tokens_map.json +| ├──tokenizer.json +| ├──tokenizer_config.json +| ├──vocab.json +``` +and if using Kolors-IP-Adapter-FaceID-Plus: +will auto download "DIAMONIK7777/antelopev2" insightface models.... +ipa-faceid-plus.bin :Kolors-IP-Adapter-FaceID-Plus [link](https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-FaceID-Plus) +``` +├── ComfyUI/models/photomaker/ +| ├── ipa-faceid-plus.bin ``` 3.3 The model file example for dual role controllnet is as follows, which only supports SDXL community controllnet @@ -130,42 +149,93 @@ The file structure is shown in the following figure: | ├──controlnet-zoe-depth-sdxl-1.0 | ├──TheMistoAI/MistoLine | ├──xinsir/controlnet-tile-sdxl-1.0 - ``` Control_img image preprocessing, please use other nodes +3.4 using flux pulid . +torch must > 0.24.0 +``` +pip install optimum-quanto==0.2.4 +``` +EVA02_CLIP_L_336_psz14_s6B.pt auto downlaod....[link](https://huggingface.co/QuanSun/EVA-CLIP/tree/main) #迟点改成不自动下载 +DIAMONIK7777/antelopev2 auto downlaod....[https://huggingface.co/DIAMONIK7777/antelopev2/tree/main) +"pulid_flux_v0.9.0.safetensors" download from [link](https://huggingface.co/guozinan/PuLID/tree/main) +fp8 using flux1-dev-fp8.safetensors from [link](https://huggingface.co/Kijai/flux-fp8/tree/main) +``` +├── ComfyUI/models/photomaker/ +| ├── pulid_flux_v0.9.0.safetensors +``` +make sure ae.safetensors in you FLUX.1-dev dir,example: +``` +├──any_path/black-forest-labs/FLUX.1-dev +| ├──model_index.json +| ├──ae.safetensors +| ├──vae +| ├── config.json +| ├── diffusion_pytorch_model.safetensors +| ├──transformer +| ├── config.json +| ├──diffusion_pytorch_model-00001-of-00003.safetensors +| ├──diffusion_pytorch_model-00002-of-00003.safetensors +| ├──diffusion_pytorch_model-00003-of-00003.safetensors +| ├── diffusion_pytorch_model.safetensors.index.json +| ├──tokenizer +| ├── special_tokens_map.json +| ├── tokenizer_config.json +| ├── vocab.json +| ├── merges.txt +| ├──tokenizer_2 +| ├── special_tokens_map.json +| ├── tokenizer_config.json +| ├── spiece.model +| ├── tokenizer.json +| ├── text_encoder +| ├── config.json +| ├── model.safetensors +| ├── text_encoder_2 +| ├── config.json +| ├── model-00001-of-00002.safetensors +| ├── model-00002-of-00002.safetensors +| ├── model.safetensors.index.json +| ├── scheduler +| ├── scheduler_config.json +``` + 4 Example ---- + +img2img mode use flux pulid 12G Vram,cpu(Latest version) +![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/flux_pulid_fp8_12GVR.png) + +img2img mode use nf4 flux (Latest version) +![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/flux_img2img.png) +![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/flux_img2img2role.png) + txt2img mode use NF4 FLUX (Latest version) ![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/nf4.png) img2img mode use auraface photomake V2 (Latest version) ![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/newest.png) -txt2img mode uses kolors model using chinese prompt (Outdated version examples) -![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/new.png) +img2img model use kolors ip adapter,and using chinese prompt (Latest version) +![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/kolor_ipadapter_use_chinese.png) +![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/kolor_ipadapter.png) -img2img mode, uses kolors model using chinese prompt (Outdated version examples) -![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/kolorsimg2img.png) +img2img model use kolors ip adapter face id (Latest version) +![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/kolor_faceid.png) -img2img mode, uses photomakeV1 (Outdated version examples) -![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/newimg2imgV1.png) +img2img sdxl mode, uses photomakeV1 (Latest version) +![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/photomakev1.png) -img2img mode, uses photomakeV2 (Outdated version examples) -![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/newimg2imgV2.png) +img2img sdxl mode, uses photomakeV2 (Latest version) +![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/photomakev2.png) -flux model,repo+pt txt2img/img2img(Outdated version examples) -![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/flux_transfomerpy.png) - -txt2img2role in 1 image (Outdated version examples) -![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/newtxt2img2role.png) +txt2img using lora and comic node (Latest version) +![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/txt2img_lora_comic.png) img2img2role in 1 image (Outdated version examples) ![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/2rolein1img.png) -img2img2role in 1 image lora (Outdated version examples) -![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/2rolein1imglora.png) - ControlNet added dual role co frame (Role 1 and Role 2) (Outdated version examples) ![](https://github.com/smthemex/ComfyUI_StoryDiffusion/blob/main/examples/controlnet.png) @@ -228,30 +298,7 @@ Function Description of Nodes * The process of generating images using PhotosMaker requires the IMG keyword in the character prompt column. You can use keywords such as a woman IMG, a man IMG, etc; * No characters appear in the image, add [NC] in front of the scene prompt; * Segmented prompt, using #, such as AAAA # BBBB, will generate AAAA content, but the text will only display BBBB - -My ComfyUI node list： ------ -1、ParlerTTS :[ComfyUI_ParlerTTS](https://github.com/smthemex/ComfyUI_ParlerTTS) -2、Llama3_8B :[ComfyUI_Llama3_8B](https://github.com/smthemex/ComfyUI_Llama3_8B) -3、HiDiffusion ：[ComfyUI_HiDiffusion_Pro](https://github.com/smthemex/ComfyUI_HiDiffusion_Pro) -4、ID_Animator ： [ComfyUI_ID_Animator](https://github.com/smthemex/ComfyUI_ID_Animator) -5、StoryDiffusion ：[ComfyUI_StoryDiffusion](https://github.com/smthemex/ComfyUI_StoryDiffusion) -6、Pops ：[ComfyUI_Pops](https://github.com/smthemex/ComfyUI_Pops) -7、stable-audio-open-1.0 ：[ComfyUI_StableAudio_Open](https://github.com/smthemex/ComfyUI_StableAudio_Open) -8、GLM4 ：[ComfyUI_ChatGLM_API](https://github.com/smthemex/ComfyUI_ChatGLM_API) -9、CustomNet ：[ComfyUI_CustomNet](https://github.com/smthemex/ComfyUI_CustomNet) -10、Pipeline_Tool :[ComfyUI_Pipeline_Tool](https://github.com/smthemex/ComfyUI_Pipeline_Tool) -11、Pic2Story :[ComfyUI_Pic2Story](https://github.com/smthemex/ComfyUI_Pic2Story) -12、ComfyUI_Streamv2v_Plus :[ComfyUI_Streamv2v_Plus](https://github.com/smthemex/ComfyUI_Streamv2v_Plus) -13、ComfyUI_MS_Diffusion :[ComfyUI_MS_Diffusion](https://github.com/smthemex/ComfyUI_MS_Diffusion) -14、ComfyUI_AnyDoor : [ComfyUI_AnyDoor](https://github.com/smthemex/ComfyUI_AnyDoor) -15、ComfyUI_Stable_Makeup : [ComfyUI_Stable_Makeup](https://github.com/smthemex/ComfyUI_Stable_Makeup) -16、ComfyUI_EchoMimic : [ComfyUI_EchoMimic](https://github.com/smthemex/ComfyUI_EchoMimic) -17、ComfyUI_FollowYourEmoji : [ComfyUI_FollowYourEmoji](https://github.com/smthemex/ComfyUI_FollowYourEmoji) -18、ComfyUI_Diffree : [ComfyUI_Diffree](https://github.com/smthemex/ComfyUI_Diffree) -19、ComfyUI_FoleyCrafter : [ComfyUI_FoleyCrafter](https://github.com/smthemex/ComfyUI_FoleyCrafter) -20、ComfyUI_MooER : [ComfyUI_MooER](https://github.com/smthemex/ComfyUI_MooER) Citation ------ @@ -303,3 +350,15 @@ kolors year={2024} } ``` +PuLID +``` +@article{guo2024pulid, + title={PuLID: Pure and Lightning ID Customization via Contrastive Alignment}, + author={Guo, Zinan and Wu, Yanze and Chen, Zhuowei and Chen, Lang and He, Qian}, + journal={arXiv preprint arXiv:2404.16022}, + year={2024} +} +``` + +FLUX +![LICENSE](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md)