20 lines
811 B
Markdown
20 lines
811 B
Markdown
This needs the llama.cpp version of gguf-py to work at the moment, not the pip one as that one does not have the python quantization code yet.
|
|
|
|
```
|
|
git clone https://github.com/ggerganov/llama.cpp
|
|
pip install llama.cpp/gguf-py
|
|
```
|
|
|
|
To quantize:
|
|
```
|
|
python convert.py --src ~/ComfyUI/models/unet/flux1-dev.safetensors --dst ~/ComfyUI/models/unet/flux1-dev-Q4_0.gguf --qtype Q4_0
|
|
```
|
|
|
|
Working quant types: Q4_0, Q5_0, Q8_0, F16
|
|
|
|
> [!WARNING]
|
|
> Do not use the diffusers UNET for flux, it won't work, use the default checkpoint that comes with the model or convert it.
|
|
|
|
> [!IMPORTANT]
|
|
> The model format is very much WIP. I don't recommend uploading the model files created with this method anywhere until proper metadata is added, although the key/quantization format is unlikely to change.
|