ExLlama nodes for ComfyUI

A simple prompt generator for ComfyUI utilizing ExLlama.

Installation

Clone the repository to custom_nodes in your ComfyUI directory and install dependencies:

git clone https://github.com/Zuellni/ComfyUI-ExLlama-Nodes
python -m pip install -r requirements.txt

Next, install the latest pre-built ExLlama wheel from here.
Choose the version matching your platform, Python, and PyTorch CUDA/ROCm.

Example for Windows with Python 3.10 and CUDA 11.8 which should match the portable ComfyUI build:

python -m pip install https://github.com/jllllll/exllama/releases/download/0.0.17/exllama-0.0.17+cu118-cp310-cp310-win_amd64.whl

Name	Description
Loader	Loads 4-bit GPTQ Llama/2 models. You can find a lot of them on Hugging Face. Clone the model repository or download all the files in it to an empty directory, then point to it in `model_dir`. `model.safetensors` won't work on its own. To load a LoRA specify its directory in `lora_dir`. It should contain `adapter_model.bin` and `adapter_config.json`. ExLlama allocates memory based on `max_seq_len`. Lowering it is a good way to save on VRAM. It's currently not possible to offload the model to RAM.
Generator	Returns a `string` based on the given `prompt` for use with other nodes. Default values correspond to the `simple-1` preset from text-generation-webui. ExLlama isn't deterministic, so the outputs may differ even with the same seed.
Previewer	Displays generated outputs in the UI.

Can be opened directly in ComfyUI. Model used: MythoLogic-Mini-7B.