nodes/Zuellni-ComfyUI-ExLlama-Nodes
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add a note on model downloading

Browse Source
This commit is contained in:
Zuellni
2023-09-19 10:05:02 +02:00
committed by GitHub
parent 863bb143cc
commit e0201ed007
1 changed files with 5 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+5 -2
View File
@@ -20,9 +20,12 @@ pip install https://github.com/jllllll/exllama/releases/download/0.0.17/exllama-
Comes with the following nodes:
### Loader
Used to load 4-bit GPTQ Llama/2 models. You can find a lot of them over at [Hugging Face](https://huggingface.co/TheBloke).
Used to load 4-bit GPTQ Llama/2 models. You can find a lot of them over at [Hugging Face](https://huggingface.co/TheBloke).
You should either clone the model repository or download all the files in it manually, then point to the directory in `model_dir`. The `model.safetensors` file on its own is not enough to work.
ExLlama allocates [memory](https://github.com/turboderp/exllama/issues/259) according to `max_seq_len`. Lowering it is a good way to save on GPU RAM.
It's currently not possible to [offload](https://github.com/turboderp/exllama/issues/177) the model to CPU RAM.
It's currently not possible to [offload](https://github.com/turboderp/exllama/issues/177) the models to CPU RAM.
### Generator
Generates a `string` based on the given `prompt` for use with other nodes.