nodes/Zuellni-ComfyUI-ExLlama-Nodes
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update readme

Browse Source
This commit is contained in:
Zuellni
2024-02-06 14:51:02 +01:00
committed by GitHub
parent 6831b951c5
commit ec6443816e
1 changed files with 6 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+6 -4
View File
@@ -7,12 +7,14 @@ Navigate to the root ComfyUI directory, clone the repository to `custom_nodes` a
git clone https://github.com/Zuellni/ComfyUI-ExLlama-Nodes custom_nodes/ComfyUI-ExLlama-Nodes
pip install -r custom_nodes/ComfyUI-ExLlama-Nodes/requirements.txt
```
Optionally, you can install [flash-attention](https://github.com/Dao-AILab/flash-attention) by uncommenting the relevant lines in the requirements file. It should lower VRAM usage but your mileage may vary.
> [!IMPORTANT]
> The wheels included in the requirements file should match the latest portable ComfyUI build. If you see any ExLlama-related errors while loading the nodes, try to install it manually following the [official instructions](https://github.com/turboderp/exllamav2#installation).
Optionally, you can install [flash-attention](https://github.com/Dao-AILab/flash-attention) by uncommenting relevant lines in the requirements file. It should lower VRAM usage but your mileage may vary.
> [!CAUTION]
> The wheels included in the requirements file should match the `stable pytorch 2.1 cu121` build of ComfyUI.<br>
> If you see any ExLlama-related errors while loading the nodes, try to install it following the [official instructions](https://github.com/turboderp/exllamav2#installation).<br>
> Keep in mind that wheels >= `0.0.13` require `pytorch 2.2`.
## Usage
Only EXL2 and 4-bit GPTQ models are supported. You can find a lot of them on [Hugging Face](https://huggingface.co/TheBloke). Refer to the model card in each repository for details about quant differences and instruction formats.
Only EXL2 and 4-bit GPTQ models are supported. You can find a lot of them on [Hugging](https://huggingface.co/LoneStriker) [Face](https://huggingface.co/TheBloke). Refer to the model card in each repository for details about quant differences and instruction formats.
To use a model with the nodes, you should clone its repository with git or manually download all the files and place them in `models/llm`. For example, if you'd like to download [Mistral-7B](https://huggingface.co/LoneStriker/Mistral-7B-Instruct-v0.2-5.0bpw-h6-exl2-2), use the following command:
```