nodes/Zuellni-ComfyUI-ExLlama-Nodes
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update to ExLlamaV2, loras currently not supported

Browse Source
This commit is contained in:
Zuellni
2023-09-27 15:55:24 +02:00
parent eaac644f6b
commit 89120f2474
4 changed files with 44 additions and 108 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+6 -7
View File
@@ -1,5 +1,5 @@
# ComfyUI ExLlama Nodes
A simple prompt generator for [ComfyUI](https://github.com/comfyanonymous/ComfyUI) utilizing [ExLlama](https://github.com/turboderp/exllama).
A simple prompt generator for [ComfyUI](https://github.com/comfyanonymous/ComfyUI) utilizing [ExLlamaV2](https://github.com/turboderp/exllamav2).
## Installation
Clone the repository to `custom_nodes` in your ComfyUI directory and install dependencies:
@@ -8,18 +8,17 @@ git clone https://github.com/Zuellni/ComfyUI-ExLlama-Nodes
python -m pip install -r requirements.txt
```
If you see any ExLlama errors while loading, install it manually from [here](https://github.com/jllllll/exllama/releases/latest).<br>
For example, on Windows with Python 3.10 and PyTorch CUDA 11.7:
If you see any ExLlama errors while loading, install it manually from [here](https://github.com/turboderp/exllamav2/releases/latest).<br>
For example, on Windows with Python 3.10 and CUDA 11.7:
```
python -m pip install https://github.com/jllllll/exllama/releases/download/0.0.17/exllama-0.0.17+cu117-cp310-cp310-win_amd64.whl
python -m pip install https://github.com/turboderp/exllamav2/releases/download/v0.0.4/exllamav2-0.0.4+cu117-cp310-cp310-win_amd64.whl
```
## Nodes
Name | Description
:--- | :---
Loader | Used to load 4-bit GPTQ Llama/2 models. You can find a lot of them on [Hugging Face](https://huggingface.co/TheBloke).<br>Clone the model repository or download all the files and place them in an empty directory, then specify the path in `model_dir`. The `model.safetensors` file won't work on its own.<br><br>ExLlama allocates memory based on `max_seq_len`. Lowering it is a good way to save on VRAM.<br>It's currently not possible to [offload](https://github.com/turboderp/exllama/issues/177) the model to RAM.
LoRA | Used to load LoRAs. The directory should contain `adapter_model.bin` and `adapter_config.json`.<br>LoRA parameter count has to match the model.
Generator | Generates a `string` based on the given `prompt` for use with other nodes.<br>Default values correspond to the `simple-1` preset from [text-generation-webui](https://github.com/oobabooga/text-generation-webui).<br><br>ExLlama isn't [deterministic](https://github.com/turboderp/exllama/issues/201), so the outputs may differ even with the same seed.
Loader | Used to load 4-bit GPTQ Llama/2 models. You can find a lot of them on [Hugging Face](https://huggingface.co/TheBloke).<br>Clone the model repository or download all the files and place them in an empty directory, then specify the path in `model_dir`. The `model.safetensors` file won't work on its own.<br><br>ExLlama allocates memory based on `max_seq_len`. Lowering it is a good way to save on VRAM.<br>It's currently not possible to offload the model to RAM.
Generator | Generates a `string` based on the given `prompt` for use with other nodes.<br>Default values correspond to the `simple-1` preset from [text-generation-webui](https://github.com/oobabooga/text-generation-webui).<br><br>ExLlama isn't deterministic, so the outputs may differ even with the same seed.
Previewer | Displays generated outputs in the UI and appends them to workflow metadata.
## Workflow