This needs the llama.cpp version of gguf-py to work at the moment, not the pip one as that one does not have the python quantization code yet.
git clone https://github.com/ggerganov/llama.cpp
pip install llama.cpp/gguf-py
To quantize:
python convert.py --src ~/ComfyUI/models/unet/flux1-dev.safetensors --dst ~/ComfyUI/models/unet/flux1-dev-Q4_0.gguf --qtype Q4_0
Working quant types: Q4_0, Q5_0, Q8_0, F16
Warning
Do not use the diffusers UNET for flux, it won't work, use the default checkpoint that comes with the model or convert it.
Important
The model format is very much WIP. I don't recommend uploading the model files created with this method anywhere until proper metadata is added, although the key/quantization format is unlikely to change.