68 lines
2.4 KiB
Markdown
68 lines
2.4 KiB
Markdown
# Configuration Guide
|
|
|
|
Complete configuration options, performance tuning, and model-specific settings.
|
|
|
|
## Model Paths
|
|
|
|
- **Relative paths**: `models/LLM/workflow-generator-q8_0.gguf` (relative to ComfyUI root)
|
|
- **Absolute paths**: `/path/to/model.gguf`
|
|
- System automatically resolves paths relative to ComfyUI's model directories
|
|
|
|
## GPU Settings
|
|
|
|
- **GPU Layers**: `"auto"` (default & recommended), `"all"`, `"none"`, or number (0-50)
|
|
- `"auto"`: Automatically determines optimal GPU layer allocation based on available VRAM to prevent OOM.
|
|
- `"all"`: Use all available GPU layers (fastest, but may cause OOM on lower VRAM).
|
|
- `"none"`: CPU only.
|
|
- Number: Specific count of GPU layers to use.
|
|
- **Device Preference**: `"auto"` (detects best), `"cuda"`, or `"cpu"`
|
|
|
|
## Model-Specific Settings
|
|
|
|
### GGUF Models
|
|
|
|
- Handle quantization internally (no dtype settings needed)
|
|
- Use `n_gpu_layers` to control GPU/CPU split
|
|
- Use `context_size` to set context window (default: 4096)
|
|
- Performance parameters:
|
|
- `use_mmap` (bool): Memory-mapped file access (default: True)
|
|
- `use_mlock` (bool): Lock memory pages (default: False)
|
|
- `n_batch` (int): Batch size for processing (default: 512)
|
|
- `n_threads` (int, optional): CPU thread count (auto-detected if not set)
|
|
|
|
### HuggingFace Models
|
|
|
|
- Support dtype options: `"auto"`, `"fp16"`, `"bf16"`, `"fp32"`, `"fp8"`, or `torch.dtype`
|
|
- Attention implementations:
|
|
- `"auto"`: Automatically selects best available
|
|
- `"flash_attention_2"`: Flash Attention 2 (requires flash-attn package)
|
|
- `"sdpa"`: Scaled Dot Product Attention (built into PyTorch 2.0+)
|
|
- Device map: Automatically handles device placement
|
|
|
|
## Performance Tuning
|
|
|
|
### For Lower VRAM Usage
|
|
|
|
- Use GGUF models (typically 4-8GB for q8_0 quantization vs 12-16GB+ for HuggingFace)
|
|
- Set `n_gpu_layers="all"` if VRAM allows
|
|
- Disable LLM refinement (`use_llm_refinement=False`)
|
|
- Use smaller context windows if possible
|
|
|
|
### For Lower VRAM Usage
|
|
|
|
- Reduce `n_gpu_layers` (try "auto" or lower number)
|
|
- Use smaller quantization (q4_0 instead of q8_0)
|
|
- Set `device_preference="cpu"` for some operations
|
|
- Use HuggingFace models with CPU offloading
|
|
|
|
### For Better Accuracy
|
|
|
|
- Enable LLM refinement (`use_llm_refinement=True`)
|
|
- Use larger models (q8_0 instead of q4_0)
|
|
- Increase `top_k` in NodeValidator for more candidate nodes
|
|
|
|
---
|
|
|
|
[← Back to Home](Home) | [← Node Reference](Node-Reference) | [Next: Troubleshooting →](Troubleshooting)
|
|
|