nodes/Zuellni-ComfyUI-ExLlama-Nodes
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add some info about model downloading

Browse Source
This commit is contained in:
Zuellni
2023-11-23 22:15:10 +01:00
committed by GitHub
parent 9cbbcb1e72
commit d881503d4e
1 changed files with 15 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+15 -3
View File
@@ -7,14 +7,26 @@ Clone the repository to `custom_nodes` and install dependencies:
git clone https://github.com/Zuellni/ComfyUI-ExLlama-Nodes
pip install -r requirements.txt
```
Optionally, you can install [flash-attention](https://github.com/Dao-AILab/flash-attention) by uncommenting the relevant lines in the requirements file.
If you see any ExLlama-related errors while loading the nodes, install it manually following the instructions [here](https://github.com/turboderp/exllamav2#method-2-install-from-release-with-prebuilt-extension).
Optionally, you can install [flash-attention](https://github.com/Dao-AILab/flash-attention) by uncommenting the relevant lines in the requirements file.<br>If you see any ExLlama-related errors while loading, install it manually following the instructions from [here](https://github.com/turboderp/exllamav2#method-2-install-from-release-with-prebuilt-extension).
## Usage
ExLlamaV2 supports EXL2 and 4-bit GPTQ models. You can find a lot of them on [Hugging Face](https://huggingface.co/TheBloke).
Refer to the model card in each repository for details about quant differences and instruction formats.
To use a model with the nodes, you should clone its repository with git or manually download all the files and place them in `models/llm`.
You can also add your own `llm` path to [extra_model_paths.yaml](https://github.com/comfyanonymous/ComfyUI/blob/master/extra_model_paths.yaml.example) and place the models there instead.
For instance, if you want to download the 4-bit 32g branch of [Zephyr 7B Beta](https://huggingface.co/TheBloke/zephyr-7B-beta-GPTQ), use the following command:
```
git clone https://huggingface.co/TheBloke/zephyr-7B-beta-GPTQ -b gptq-4bit-32g-actorder_True models/llm/zephyr-7b-gptq-32g
```
## Nodes
Name | Description
:--- | :---
Loader | Used to load EXL2/GPTQ Llama models. You can find a lot of them on [Hugging Face](https://huggingface.co/TheBloke).<br>Clone the model repository and place it in `models/llm` or specify your own `llm` path in `extra_model_paths.yaml`.
Generator | Generates a `string` based on the given input for use with other nodes.
Loader | Used to load EXL2/GPTQ Llama models.
Generator | Generates a `string` based on the given input.
Preview | Displays generated outputs in the UI.
Replace | Replaces variables enclosed in brackets, such as `[a]`, with their values.