ComfyUI ExLlama Nodes

A simple prompt generator for ComfyUI utilizing ExLlama.

Installation

Clone the repository to custom_nodes in your ComfyUI directory and install dependencies:

git clone https://github.com/Zuellni/ComfyUI-ExLlama-Nodes
pip install -r requirements.txt

If you see any ExLlama errors while loading, install it manually from here.
For example, on Windows with Python 3.10 and CUDA 11.7:

pip install https://github.com/turboderp/exllamav2/releases/download/v0.0.4/exllamav2-0.0.4+cu117-cp310-cp310-win_amd64.whl

Name	Description
Loader	Loads EXL2/GPTQ Llama models. You can find a lot of them on Hugging Face. Clone the model repository or download all the files and place them in an empty directory, then specify the path in `model_dir`. The `model.safetensors` file won't work on its own. ExLlama allocates memory based on `max_seq_len`. Lowering it is a good way to save on VRAM. It's currently not possible to offload the model to RAM.
Generator	Generates a `string` based on the given `prompt` for use with other nodes. Default values correspond to the `simple-1` preset from text-generation-webui. ExLlama isn't deterministic, so the outputs may differ even with the same seed.
Previewer	Displays generated outputs in the UI and appends them to workflow metadata.

The workflow below can be opened in ComfyUI. VRAM usage around 3GB before SDXL.
Model: Mistral-7B-instruct-exl2-2.5bpw.