ComfyUI-GGUF

GGUF Quantization support for native ComfyUI models

This is currently very much WIP. These custom nodes provide support for model files stored in the GGUF format popularized by llama.cpp.

While quantization wasn't feasible for regular UNET models (conv2d), transformer/DiT models such as flux seem less affected by quantization. This allows running it in much lower bits per weight variable bitrate quants on low-end GPUs.

Installation

Important

Make sure your ComfyUI is on a recent-enough version to support custom ops when loading the UNET-only.

To install the custom node, clone it to your custom_nodes ComfyUI folder using the following command:

git clone https://github.com/city96/ComfyUI-GGUF ComfyUI/custom_nodes/ComfyUI-GGUF

To install the required gguf python package on the standalone ComfyUI, Open a CMD where your run_nvidia_gpu.bat file is and use the following command:

.\python_embeded\python.exe -s -m pip install -r .\ComfyUI\custom_nodes\ComfyUI-GGUF\requirements.txt

Usage

Simply use the GGUF Unet loader found under the bootleg category. You can find pre-quantized models for flux1-dev here - place these in your ComfyUI/models/unet folder.