DanielPFlorian-ComfyUI-WorkflowGenerator/wiki/Configuration.md

Configuration Guide

Complete configuration options, performance tuning, and model-specific settings.

Model Paths

• Relative paths: models/LLM/workflow-generator-q8_0.gguf (relative to ComfyUI root)
• Absolute paths: /path/to/model.gguf
• System automatically resolves paths relative to ComfyUI's model directories

GPU Settings

• GPU Layers: "auto" (default & recommended), "all", "none", or number (0-50)
  • "auto": Automatically determines optimal GPU layer allocation based on available VRAM to prevent OOM.
  • "all": Use all available GPU layers (fastest, but may cause OOM on lower VRAM).
  • "none": CPU only.
  • Number: Specific count of GPU layers to use.
• Device Preference: "auto" (detects best), "cuda", or "cpu"

Model-Specific Settings

GGUF Models

• Handle quantization internally (no dtype settings needed)
• Use n_gpu_layers to control GPU/CPU split
• Use context_size to set context window (default: 4096)
• Performance parameters:
  • use_mmap (bool): Memory-mapped file access (default: True)
  • use_mlock (bool): Lock memory pages (default: False)
  • n_batch (int): Batch size for processing (default: 512)
  • n_threads (int, optional): CPU thread count (auto-detected if not set)

HuggingFace Models

• Support dtype options: "auto", "fp16", "bf16", "fp32", "fp8", or torch.dtype
• Attention implementations:
  • "auto": Automatically selects best available
  • "flash_attention_2": Flash Attention 2 (requires flash-attn package)
  • "sdpa": Scaled Dot Product Attention (built into PyTorch 2.0+)
• Device map: Automatically handles device placement

Performance Tuning

For Lower VRAM Usage

• Use GGUF models (typically 4-8GB for q8_0 quantization vs 12-16GB+ for HuggingFace)
• Set n_gpu_layers="all" if VRAM allows
• Disable LLM refinement (use_llm_refinement=False)
• Use smaller context windows if possible

For Lower VRAM Usage

• Reduce n_gpu_layers (try "auto" or lower number)
• Use smaller quantization (q4_0 instead of q8_0)
• Set device_preference="cpu" for some operations
• Use HuggingFace models with CPU offloading

For Better Accuracy

• Enable LLM refinement (use_llm_refinement=True)
• Use larger models (q8_0 instead of q4_0)
• Increase top_k in NodeValidator for more candidate nodes

---

← Back to Home | ← Node Reference | Next: Troubleshooting →