diff --git a/README.md b/README.md index aec0108..65a7a6a 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,13 @@ # prompt-generator-comfyui -Custom AI prompt generator node for [ComfyUI](https://github.com/comfyanonymous/ComfyUI). With this node, you can use text generation models to generate prompts. Before using, text generation model has to be trained with prompt dataset or you can use the [pretrained models](#pretrained-prompt-models). +Custom AI prompt generator node for +[ComfyUI](https://github.com/comfyanonymous/ComfyUI). With this node, you can +use text generation models to generate prompts. Before using, text generation +model has to be trained with prompt dataset or you can use the +[pretrained models](#pretrained-prompt-models). # Table Of Contents + - [prompt-generator-comfyui](#prompt-generator-comfyui) - [Table Of Contents](#table-of-contents) - [Setup](#setup) @@ -25,158 +30,224 @@ Custom AI prompt generator node for [ComfyUI](https://github.com/comfyanonymous/ - [Package Version](#package-version) - [For Manual Installation of the ComfyUI](#for-manual-installation-of-the-comfyui-1) - [For Portable Installation of the ComfyUI](#for-portable-installation-of-the-comfyui-1) - - [Automatic Installation](#automatic-installation) - - [For Manual Installation of the ComfyUI](#for-manual-installation-of-the-comfyui-2) - - [For Portable Installation of the ComfyUI](#for-portable-installation-of-the-comfyui-2) - [New Updates On The Node](#new-updates-on-the-node) + - [Quantize Values Are Not Updated](#quantize-values-are-not-updated) - [Contributing](#contributing) - [Example Outputs](#example-outputs) # Setup + ## For Portable Installation of the ComfyUI -- [x] Automatic installation is added for portable version. -- Clone the repository with ```git clone https://github.com/alpertunga-bile/prompt-generator-comfyui.git``` command under ```custom_nodes``` folder. -- Go to the ```ComfyUI_windows_portable``` folder and run the **run_nvidia_gpu.bat** file -- Open the ```hires.fixWithPromptGenerator.json``` or ```basicWorkflowWithPromptGenerator.json``` workflow -- Put your generator under the ```models/prompt_generators``` folder. You can create your prompt generator with [this repository](https://github.com/alpertunga-bile/prompt-markdown-parser). You have to put generator as folder. Do not just put ```pytorch_model.bin``` file for example. -- Click ```Refresh``` button in ComfyUI + +- [x] Automatic package installation is added for portable version. +- Clone the repository with + `git clone https://github.com/alpertunga-bile/prompt-generator-comfyui.git` + command under `custom_nodes` folder. +- Go to the `ComfyUI_windows_portable` folder and run the **run_nvidia_gpu.bat** + file +- Open the `hires.fixWithPromptGenerator.json` or + `basicWorkflowWithPromptGenerator.json` workflow +- Put your generator under the `models/prompt_generators` folder. You have to + put generator as folder. Do not just put `pytorch_model.bin` file for example. +- Refresh or restart the ComfyUI ## For Manual Installation of the ComfyUI -- Clone the repository with ```git clone https://github.com/alpertunga-bile/prompt-generator-comfyui.git``` command under ```custom_nodes``` folder. + +- Clone the repository with + `git clone https://github.com/alpertunga-bile/prompt-generator-comfyui.git` + command under `custom_nodes` folder. - Run the ComfyUI -- Open the ```hires.fixWithPromptGenerator.json``` or ```basicWorkflowWithPromptGenerator.json``` workflow -- Put your generator under the ```models/prompt_generators``` folder. You can create your prompt generator with [this repository](https://github.com/alpertunga-bile/prompt-markdown-parser). You have to put generator as folder. Do not just put ```pytorch_model.bin``` file for example. -- Click ```Refresh``` button in ComfyUI +- Open the `hires.fixWithPromptGenerator.json` or + `basicWorkflowWithPromptGenerator.json` workflow +- Put your generator under the `models/prompt_generators` folder. You have to + put generator as folder. Do not just put `pytorch_model.bin` file for example. +- Refresh or restart the ComfyUI ## For ComfyUI Manager Users -- Download the node with [ComfyUI Manager](https://github.com/ltdrdata/ComfyUI-Manager) -- Restart the ComfyUI -- Open the ```hires.fixWithPromptGenerator.json``` or ```basicWorkflowWithPromptGenerator.json``` workflow -- Put your generator under the ```models/prompt_generators``` folder. You can create your prompt generator with [this repository](https://github.com/alpertunga-bile/prompt-markdown-parser). You have to put generator as folder. Do not just put ```pytorch_model.bin``` file for example. -- Click ```Refresh``` button in ComfyUI + +- Download the node with + [ComfyUI Manager](https://github.com/ltdrdata/ComfyUI-Manager) +- Open the `hires.fixWithPromptGenerator.json` or + `basicWorkflowWithPromptGenerator.json` workflow +- Put your generator under the `models/prompt_generators` folder. You have to + put generator as folder. Do not just put `pytorch_model.bin` file for example. +- Refresh or restart the ComfyUI # Features -- Multiple output generation is added. You can choose from 5 outputs with the index value. You can check the generated prompts from the log file and terminal. The prompts are logged and printed in order. + +- Multiple output generation is added. You can choose from 5 outputs with the + index value. You can check the generated prompts from the log file and + terminal. The prompts are logged and printed in order. - Randomness is added. See [this section](#random-generation). -- Quantization is added with [Quanto](https://github.com/huggingface/optimum-quanto) and [Bitsandbytes](https://huggingface.co/docs/bitsandbytes/main/en/index) packages. See [this section](#quantization). -- Lora adapter model loading is added with [Peft](https://huggingface.co/docs/peft/en/index) package. (The feature is not full tested in this repository because of my low VRAM but I am using the same implementation in Google Colab for training and inference and it is working there) -- Optimizations are done with [Optimum](https://github.com/huggingface/optimum) package. +- Quantization is added with + [Quanto](https://github.com/huggingface/optimum-quanto) and + [Bitsandbytes](https://huggingface.co/docs/bitsandbytes/main/en/index) + packages. See [this section](#quantization). +- Lora adapter model loading is added with + [Peft](https://huggingface.co/docs/peft/en/index) package. (The feature is not + full tested in this repository because of my low VRAM but I am using the same + implementation in Google Colab for training and inference and it is working + there) +- Optimizations are done with [Optimum](https://github.com/huggingface/optimum) + package. - ONNX and transformers models are supported. - Preprocessing outputs. See [this section](#how-preprocess-mode-works). - Recursive generation is supported. See [this section](#how-recursive-works). -- Print generated text to terminal and log the node's state under the ```generated_prompts``` folder with date as filename. +- Print generated text to terminal and log the node's state under the + `generated_prompts` folder with current date as filename. # Example Workflow + ![example_hires_workflow](images/hires_workflow.png) ![example_basic_workflow](images/basic_workflow.png) -- **Prompt Generator Node** may look different with final version but workflow in ComfyUI is not going to change +- **Prompt Generator Node** may look different with final version but workflow + in ComfyUI is not going to change # Pretrained Prompt Models -- You can find the models in [this link](https://drive.google.com/drive/folders/1c21kMH6FTaia5C8239okL3Q0wJnnWc1N?usp=share_link) +- You can find the models in + [this link](https://drive.google.com/drive/folders/1c21kMH6FTaia5C8239okL3Q0wJnnWc1N?usp=share_link) - For to use the pretrained model follow these steps: - - Download the model and unzip to ```models/prompt_generators``` folder. - - Click ```Refresh``` button in ComfyUI. - - Then select the generator with the node's ```model_name``` variable (If you can't see the generator, restart the ComfyUI). + - Download the model and unzip to `models/prompt_generators` folder. + - Refresh or restart the ComfyUI + - Then select the generator with the node's `model_name` variable (If you + can't see the generator, restart the ComfyUI). ## Dataset - [Huggingface Dataset](https://huggingface.co/datasets/WoWoWoWololo/stable_diffusion_female_prompts) -- Process of data cleaning and gathering can be found [here](https://github.com/alpertunga-bile/prompt-markdown-parser/blob/master/sources/CLI/CLICivitai.py) - +- Process of data cleaning and gathering can be found + [here](https://github.com/alpertunga-bile/prompt-markdown-parser/blob/master/sources/CLI/CLICivitai.py) + ## Models -- The model versions are used to differentiate models rather than showing which one is better. -- The v2 version is the latest trained model and the v4 and v5 models are experimental models. +- The model versions are used to differentiate models rather than showing which + one is better. +- The v2 version is the latest trained model and the v4 and v5 models are + experimental models. - female_positive_generator_v2 | **(Training In Process)** - Base model - using [distilgpt2](https://huggingface.co/distilgpt2) model - - ~500 MB - + - ~500 MB + - female_positive_generator_v3 | **(Training In Process)** - Base model - - using [bigscience/bloom-560m](https://huggingface.co/bigscience/bloom-560m) model + - using [bigscience/bloom-560m](https://huggingface.co/bigscience/bloom-560m) + model - ~1.3 GB - female_positive_generator_v4 | **Experimental** - Lora adapter model - - using [senseable/WestLake-7B-v2](https://huggingface.co/senseable/WestLake-7B-v2) model as the base model + - using + [senseable/WestLake-7B-v2](https://huggingface.co/senseable/WestLake-7B-v2) + model as the base model - Base model is ~14 GB # Variables -| Variable Names | Definitions | -| :-----------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **model_name** | Folder name that contains the model | -| **accelerate** | Open optimizations. Some of the models are not supported by BetterTransformer ([Check your model](https://huggingface.co/docs/optimum/bettertransformer/overview#supported-models)). If it is not supported switch this option to disable or convert your model to ONNX | -| **quantize** | Quantize the model. The quantize type is changed based on your OS and torch version. ```none``` value disables the quantization. Check [this section](#quantization) for more information | -| **token_healing** | Enable token healing algorithm which is used for fixing unintended bias. Read more [from this blog post](https://towardsdatascience.com/the-art-of-prompt-design-prompt-boundaries-and-token-healing-3b2448b0be38). To use it, the transformers package version has to be newer from or equal to **4.6 version** | -| **prompt** | Input prompt for the generator | -| **seed** | Seed value for the model | -| **lock** | Lock the generation and select from the last generated prompts with index value | -| **random_index** | Random index value in [1, 5]. If the value is __enable__, the __index__ value is not used | -| **index** | User specified index value for selecting prompt from the generated prompts. **random_index** variable must be __disable__ | -| **cfg** | CFG is enabled by setting guidance_scale > 1. Higher guidance scale encourages the model to generate samples that are more closely linked to the input prompt, usually at the expense of poorer quality | -| **min_new_tokens** | The minimum numbers of tokens to generate, ignoring the number of tokens in the prompt. | -| **max_new_tokens** | The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt. | -| **do_sample** | Whether or not to use sampling; use greedy decoding otherwise | -| **early_stopping** | Controls the stopping condition for beam-based methods, like beam-search | -| **num_beams** | Number of steps for each search path | -| **num_beam_groups** | Number of groups to divide num_beams into in order to ensure diversity among different groups of beams | -| **diversity_penalty** | This value is subtracted from a beam’s score if it generates a token same as any beam from other group at a particular time. Note that diversity_penalty is only effective if ```group beam search``` is enabled. | -| **temperature** | How sensitive the algorithm is to selecting low probability options | -| **top_k** | The number of highest probability vocabulary tokens to keep for top-k-filtering | -| **top_p** | If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for generation | -| **repetition_penalty** | The parameter for repetition penalty. 1.0 means no penalty | -| **no_repeat_ngram_size** | The size of an n-gram that cannot occur more than once. (0=infinity) | -| **remove_invalid_values** | Whether to remove possible nan and inf outputs of the model to prevent the generation method to crash. Note that using remove_invalid_values can slow down generation. | -| **self_recursive** | See [this section](#how-recursive-works) | -| **recursive_level** | See [this section](#how-recursive-works) | -| **preprocess_mode** | See [this section](#how-preprocess-mode-works) | +| Variable Names | Definitions | +| :-----------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **model_name** | Folder name that contains the model | +| **accelerate** | Open optimizations. Some of the models are not supported by BetterTransformer ([Check your model](https://huggingface.co/docs/optimum/bettertransformer/overview#supported-models)). If it is not supported switch this option to disable or convert your model to ONNX | +| **quantize** | Quantize the model. The quantize type is changed based on your OS and torch version. `none` value disables the quantization. Check [this section](#quantization) for more information | +| **token_healing** | Enable token healing algorithm which is used for fixing unintended bias. Read more [from this blog post](https://towardsdatascience.com/the-art-of-prompt-design-prompt-boundaries-and-token-healing-3b2448b0be38). To use it, the transformers package version has to be newer from or equal to **4.6 version** | +| **prompt** | Input prompt for the generator | +| **seed** | Seed value for the model | +| **lock** | Lock the generation and select from the last generated prompts with index value | +| **random_index** | Random index value in [1, 5]. If the value is **enable**, the **index** value is not used | +| **index** | User specified index value for selecting prompt from the generated prompts. **random_index** variable must be **disable** | +| **cfg** | CFG is enabled by setting guidance_scale > 1. Higher guidance scale encourages the model to generate samples that are more closely linked to the input prompt, usually at the expense of poorer quality | +| **min_new_tokens** | The minimum numbers of tokens to generate, ignoring the number of tokens in the prompt. | +| **max_new_tokens** | The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt. | +| **do_sample** | Whether or not to use sampling; use greedy decoding otherwise | +| **early_stopping** | Controls the stopping condition for beam-based methods, like beam-search | +| **num_beams** | Number of steps for each search path | +| **num_beam_groups** | Number of groups to divide num_beams into in order to ensure diversity among different groups of beams | +| **diversity_penalty** | This value is subtracted from a beam’s score if it generates a token same as any beam from other group at a particular time. Note that diversity_penalty is only effective if `group beam search` is enabled. | +| **temperature** | How sensitive the algorithm is to selecting low probability options | +| **top_k** | The number of highest probability vocabulary tokens to keep for top-k-filtering | +| **top_p** | If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for generation | +| **repetition_penalty** | The parameter for repetition penalty. 1.0 means no penalty | +| **no_repeat_ngram_size** | The size of an n-gram that cannot occur more than once. (0=infinity) | +| **remove_invalid_values** | Whether to remove possible nan and inf outputs of the model to prevent the generation method to crash. Note that using remove_invalid_values can slow down generation. | +| **self_recursive** | See [this section](#how-recursive-works) | +| **recursive_level** | See [this section](#how-recursive-works) | +| **preprocess_mode** | See [this section](#how-preprocess-mode-works) | -- For more information, follow [this link](https://huggingface.co/docs/transformers/v4.31.0/en/main_classes/text_generation#transformers.GenerationConfig). -- Check [this link](https://huggingface.co/docs/transformers/v4.31.0/en/generation_strategies#text-generation-strategies) for text generation strategies. +- For more information, follow + [this link](https://huggingface.co/docs/transformers/v4.31.0/en/main_classes/text_generation#transformers.GenerationConfig). +- Check + [this link](https://huggingface.co/docs/transformers/v4.31.0/en/generation_strategies#text-generation-strategies) + for text generation strategies. ## Quantization -- Quantization is added with [Quanto](https://github.com/huggingface/optimum-quanto) and [Bitsandbytes](https://huggingface.co/docs/bitsandbytes/main/en/index) packages. -- The Quanto package requires ```torch >= 2.4``` and Bitsandbytes package works out-of-box with Linux OS. So the node is checking which package to use: - - If requirements are not specified for this packages, you can not use the ```quantize``` variable and it has only ```none``` value. - - If the Quanto requirements are filled then you can choose between ```none, int8, float8, int4``` values. - - If the Bitsandbytes requirements are filled then you can choose between ```none, int8, int4``` values. -- If your environment can use the Quanto and Bitsandbytes packages, the node selects the Bitsandbytes package. - + +- Quantization is added with + [Quanto](https://github.com/huggingface/optimum-quanto) and + [Bitsandbytes](https://huggingface.co/docs/bitsandbytes/main/en/index) + packages. +- The Quanto package requires `torch >= 2.4` and Bitsandbytes package works + out-of-box with Linux OS. So the node is checking which package to use: + - If requirements are not specified for this packages, you can not use the + `quantize` variable and it has only `none` value. + - If the Quanto requirements are filled then you can choose between + `none, int8, float8, int4` values. + - If the Bitsandbytes requirements are filled then you can choose between + `none, int8, int4` values. +- If your environment can use the Quanto and Bitsandbytes packages, the node + selects the Bitsandbytes package. ## Random Generation + - For random generation: - Enable **do_sample** -- You can find this text generation strategy from the upper link. The strategy is called **Multinomial sampling**. -- Changing variable of **do_sample** to __disable__ gives deterministic generation. +- You can find this text generation strategy from the + [this link](https://huggingface.co/docs/transformers/v4.31.0/en/generation_strategies#text-generation-strategies). + The strategy is called **Multinomial sampling**. +- Changing variable of **do_sample** to **disable** gives deterministic + generation. - For more randomness, you can: - Set **num_beams** to 1 - Enable **random_index** variable - Increase **recursive_level** - Enable **self_recursive** - + ## Lock The Generation -- Enabling the **lock** variable skip the generation and let you choose from the last generated prompts. + +- Enabling the **lock** variable skip the generation and let you choose from the + last generated prompts. - You can choose from the **index** value or use the **random_index**. - If **random_index** is enabled, the **index** value is ignored. ## How Recursive Works? -- Let's say we give ```a, ``` as seed and recursive level is 1. I am going to use the same outputs for this example to explain the functionality more understandable. -- With self recursive, let's say generator's output is ```b```. So next seed is going to be ```b``` and generator's output is ```c```. Final output is ```a, c```. It can be used for generating random outputs. -- Without self recursive, let's say generator's output is ```b```. So next seed is going to be ```a, b``` and generator's output is ```c```. Final output is ```a, b, c```. It can be used for more accurate prompts. + +- Let's say we give `a,` as seed and recursive level is 1. I am going to use the + same outputs for this example to explain the functionality more + understandable. +- With self recursive, let's say generator's output is `b`. So next seed is + going to be `b` and generator's output is `c`. Final output is `a, c`. It can + be used for generating random outputs. +- Without self recursive, let's say generator's output is `b`. So next seed is + going to be `a, b` and generator's output is `c`. Final output is `a, b, c`. + It can be used for more accurate prompts. ## How Preprocess Mode Works? -- **exact_keyword** => ```(masterpiece), ((masterpiece))``` is not allowed. Checking the pure keyword without parantheses and weights. The algorithm is adding the prompts from the beginning of the generated text, so add important prompts to **prompt** variable. -- **exact_prompt** => ```(masterpiece), ((masterpiece))``` is allowed but ```(masterpiece), (masterpiece)``` is not. Checking the exact match of the prompt. + +- **exact_keyword** => `(masterpiece), ((masterpiece))` is not allowed. Checking + the pure keyword without parantheses and weights. The algorithm is adding the + prompts from the beginning of the generated text, so add important prompts to + **prompt** variable. +- **exact_prompt** => `(masterpiece), ((masterpiece))` is allowed but + `(masterpiece), (masterpiece)` is not. Checking the exact match of the prompt. - **none** => Everything is allowed even the repeated prompts. + ### Example + ``` # ---------------------------------------------------------------------- Original ---------------------------------------------------------------------- # ((masterpiece)), ((masterpiece:1.2)), (masterpiece), blahblah, blah, blah, ((blahblah)), (((((blah))))), ((same prompt)), same prompt, (masterpiece) @@ -188,51 +259,58 @@ Custom AI prompt generator node for [ComfyUI](https://github.com/comfyanonymous/ # Troubleshooting -- If the below solutions are not fixed your issue please create an issue with ```bug``` label +- If the below solutions are not fixed your issue please create an issue with + `bug` label ## Package Version -- The node is based on [transformers](https://github.com/huggingface/transformers) and [optimum](https://github.com/huggingface/optimum) packages. So most of the problems may be caused from these packages. For overcome these problems you can try to update these packages: +- The node is based on + [transformers](https://github.com/huggingface/transformers) and + [optimum](https://github.com/huggingface/optimum) packages. So most of the + problems may be caused from these packages. For overcome these problems you + can try to update these packages: ### For Manual Installation of the ComfyUI 1. Activate the virtual environment if there is one. -2. Run the ```pip install --upgrade transformers optimum optimum[onnxruntime-gpu]``` command. +2. Run the `pip install --upgrade transformers optimum optimum[onnxruntime-gpu]` + command. ### For Portable Installation of the ComfyUI -1. Go to the ```ComfyUI_windows_portable``` folder. +1. Go to the `ComfyUI_windows_portable` folder. 2. Open the command prompt in this folder. -3. Run the ```.\python_embeded\python.exe -s -m pip install --upgrade transformers optimum optimum[onnxruntime-gpu]``` command. - -## Automatic Installation - -### For Manual Installation of the ComfyUI - -- The users have to check if they activate the virtual environment if there is one. - -### For Portable Installation of the ComfyUI - -- The users have to check that they are starting the ComfyUI in the ```ComfyUI_windows_portable``` folder. -- Because the node is checking the ```python_embeded``` folder if it is exists and is using it to install the required packages. +3. Run the + `.\python_embeded\python.exe -s -m pip install --upgrade transformers optimum optimum[onnxruntime-gpu]` + command. ## New Updates On The Node -- Sometimes the variables are changed with updates, so it may broke the workflow. But don't worry, you have to just delete the node in the workflow and add it again. +- Sometimes the variables are changed with updates, so it may broke the + workflow. But don't worry, you have to just delete the node in the workflow + and add it again. + +## Quantize Values Are Not Updated + +- The provided workflows are saved with Quanto configuration. +- To update the `quantize` values for your environment, delete the node and add + it again in your workflow. # Contributing -- Contributions are welcome. If you have an idea and want to implement it by yourself please follow these steps: +- Contributions are welcome. If you have an idea and want to implement it by + yourself please follow these steps: 1. Create a fork 2. Pull request the fork with the description that explaining the new feature -- If you have an idea but don't know how to implement it, please create an issue with ```enhancement``` label. +- If you have an idea but don't know how to implement it, please create an issue + with `enhancement` label. -- [x] The contributing can be done in several ways. You can contribute to code or to README file. +- [x] The contributing can be done in several ways. You can contribute to code + or to README file. # Example Outputs -![first_example](images/first.png) -![second_example](images/second.png) -![third_example](images/third.png) +![first_example](images/first.png) ![second_example](images/second.png) +![third_example](images/third.png)