nodes/Zuellni-ComfyUI-ExLlama-Nodes
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'refs/remotes/origin/main'

Browse Source
This commit is contained in:
Zuellni
2023-09-14 11:16:33 +02:00
parent 523a881343 9e7a702f8d
commit f749ba0af2
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+2 -2
View File
@@ -20,13 +20,13 @@ pip install https://github.com/jllllll/exllama/releases/download/0.0.17/exllama-
Comes with the following nodes: Comes with the following nodes:
### ExLlama Loader ### ExLlama Loader
Used to load 4-bit GPTQ Llama/2 models. You can find a lot of them over at https://huggingface.co/TheBloke. Used to load 4-bit GPTQ Llama/2 models. You can find a lot of them over at [Hugging Face](https://huggingface.co/TheBloke).
ExLlama allocates memory according to `max_seq_len`. Lowering it is a good way to save on GPU RAM. ExLlama allocates memory according to `max_seq_len`. Lowering it is a good way to save on GPU RAM.
It's not possible to offload the model to CPU RAM currently. It's not possible to offload the model to CPU RAM currently.
### ExLlama Generator ### ExLlama Generator
Generates a `string` based on the given `prompt` for use with other nodes. Generates a `string` based on the given `prompt` for use with other nodes.
Default parameter values correspond to the `simple-1` preset from https://github.com/oobabooga/text-generation-webui. Default parameter values correspond to the `simple-1` preset from [text-generation-webui](https://github.com/oobabooga/text-generation-webui).
ExLlama isn't deterministic, so the outputs may differ even with the same seed. ExLlama isn't deterministic, so the outputs may differ even with the same seed.
## Example ## Example