goodguy1963-ComfyUI-ThinkingLLM/docs/MODELS_AND_BACKENDS.md

Models and backends

This guide describes ThinkingLLM's inference paths, curated model families, local discovery, and platform requirements.

Choose a backend

Transformers / Hugging Face

The HF path is the recommended default for Windows and Linux when you want the fewest installation complications. It powers:

• ThinkingLLM
• ThinkingLLM (Advanced)
• ThinkingLLM Prompt Enhancer

Some Gemma and Qwen tokenizers require sentencepiece or tiktoken. ThinkingLLM declares both dependencies and attempts to install a missing tokenizer backend at first load.

GGUF / llama.cpp

The GGUF path powers:

• ThinkingLLM (GGUF)
• ThinkingLLM (GGUF Advanced)
• ThinkingLLM Gemma 4 Audio (GGUF)
• ThinkingLLM Prompt Enhancer (GGUF)

Multimodal GGUF inference requires a llama.cpp Python build with the chat handlers used by the selected architecture. The regular PyPI package may not include the required Qwen vision, Gemma 4, or MTMD audio support. Follow Install llama-cpp-python for platform-specific setup and verification.

Whisper ASR

ThinkingLLM Whisper ASR uses faster-whisper for speech-to-text. It is independent of the Gemma 4 GGUF audio path and requires FFmpeg for general audio-file decoding.

OpenAI-compatible API

The API node calls approved remote providers with credentials bound on the ComfyUI server. See ThinkingLLM API for setup and security boundaries.

Curated model families

ThinkingLLM ships model metadata in hf_models.json and gguf_models.json.

Qwen vision-language

• HF: Qwen3.5 4B/9B; Qwen3-VL 4B/8B/32B; Qwen2.5-VL 3B/7B
• GGUF: Qwen3.8 27B; Qwen3.5 4B/9B/27B; Qwen3-VL 4B/8B/32B; Qwen2.5-VL 3B/7B

Catalog variants include selected Instruct, thinking, abliterated, heretic, unredacted, and uncensored releases.

Qwen text-only

• HF: Qwen3 0.6B/4B/8B; Qwen3.5 4B/9B
• GGUF: Qwen3.8 27B; Qwen3 4B/8B; Qwen3.5 4B/9B/27B

Gemma 4 vision-language

• HF: Gemma 4 E2B/E4B, 12B, 26B-A4B, and 31B families
• GGUF: Gemma 4 E2B/E4B, 12B, 26B-A4B, and 31B families across selected quantizations

Gemma 4 text-only

Selected E2B, E4B, 12B, 26B-A4B, and 31B HF variants are available in text workflows.

Gemma 4 audio

The audio selector includes curated audio-capable Gemma 4 E2B, E4B, and 12B GGUF entries. Audio also depends on the selected mmproj and a compatible Gemma4ChatHandler; a model name alone does not guarantee audio support.

Whisper

Available sizes include tiny, base, small, medium, large-v3, and distil-large-v3.

The catalog files are the authoritative current list. Dropdown contents can evolve between releases as models and compatibility information change.

Model locations

ThinkingLLM searches configured ComfyUI model paths and common defaults, including:

• HF snapshots: ComfyUI/models/LLM/Qwen-VL/
• GGUF models: ComfyUI/models/LLM/GGUF/
• compatible single-file checkpoints: ComfyUI/models/text_encoders/

Keep a GGUF model and its matching mmproj together. Local GGUF files with an mmproj are discovered even when they are not in the bundled catalog.

Model labels

• [installed] means a catalog model was found in a configured LLM, GGUF, or Qwen-VL location.
• [local] means an uncatalogued compatible local model was discovered.

Labels are display-only. They do not alter saved model identities, and older saved display labels remain compatible where supported.

Shared ComfyUI text encoders

Compatible checkpoints in ComfyUI/models/text_encoders appear in applicable selectors and are loaded through ComfyUI rather than copied into a second Transformers snapshot.

Supported examples include:

• Qwen3-VL 4B and 8B .safetensors, including qwen3vl_8b_fp8_scaled.safetensors
• Gemma 3 12B .safetensors, including gemma-3-12b-it-heretic-v2_int8.safetensors

The MiniMax H3/Qwen3-VL 32B conditioning checkpoint is excluded because it is not a complete text-generation model. ComfyUI-native generation supports one-beam sampling and prints terminal output after generation rather than streaming token-by-token.

Community and commercial catalogs

Community dropdowns preserve normal model names and download behavior. Gated Hugging Face repositories still require acceptance of the model owner's terms and a read token.

Commercial-release dropdowns include only reviewed, locked models and group entries by rights status. This restriction is intentionally separate from Community behavior.

Platform notes

• Windows GGUF: install a matching win_amd64 multimodal wheel for the Python version bundled with ComfyUI.
• Linux GGUF: ThinkingLLM checks the backend at first use and can install a verified matching JamePeng wheel. Environment overrides are documented in the installation guide.
• Gemma 4 audio: requires multimodal llama-cpp-python 0.3.36 or newer and an mmproj that reports audio support.
• Whisper: requires faster-whisper; file-path decoding also requires FFmpeg.
• Hugging Face attention: set attention_mode to sage to request SageAttention instead of Flash Attention 2. auto tries SageAttention, then Flash Attention 2, then PyTorch SDPA. On ROCm, SageAttention 1.0.6 uses its Triton backend when available in the same Python environment as ComfyUI. Unsupported models or attention inputs use SDPA. The GGUF node's flash_attn setting belongs to llama.cpp and does not select SageAttention.
• Flash Attention: support is strongest on Linux and falls back when unavailable.
• Thinking controls: Qwen3/Qwen3.5-style models generally respond most consistently to explicit thinking steering; other architectures may interpret it differently.

Troubleshooting direction

• Missing or incompatible GGUF handlers: use the llama.cpp installation guide.
• Unexpected response versus reasoning output: inspect RAW_TRACE and Features and workflows.
• Remote provider setup or security errors: use the API guide.
• Duration or MiniMax structure questions: use Duration-aware video prompts.