ttulttul-ComfyUI-Modal/docs/MODAL_LLM.md

Resident Modal LLM Architecture

Modal LLM runs multimodal language-model inference inside the persistent RemoteEngine process. It is not an HTTP client and does not create an independently scaling inference service.

Request Path

1. The V3 node exposes text, IMAGE, native VIDEO, and OPENAI_INPUT_FILES inputs.
2. Queue rewriting places the marked node in a normal remote component.
3. The local dispatcher scans the complete payload, including split phases, for curated profiles or Hugging Face model IDs.
4. For a model ID, the CPU resolver inspects Hub metadata and config.json, validates architecture, quantization, safetensors inventory, security status, and download budget, then pins the exact commit in a content-addressed schema-v2 manifest.
5. The CPU ModelStager downloads only the approved revision-pinned safetensors, processor, and tokenizer assets to the shared Modal Volume.
6. The stager writes a completion marker and commits the Volume.
7. The dispatcher rewrites the model ID to the immutable generated profile ID, and the GPU worker reloads that exact Volume revision before executing the component.
8. The node converts ComfyUI tensors directly to processor inputs, samples video frames uniformly, and extracts bounded text from supported files.
9. ResidentLLMManager reuses or loads the profile, generates under a process lock, and reports progress and cancellation.
10. The response and JSON telemetry return through the normal Modal-Sync node-output transport.

Residency And Memory

The manager is module-global in the warm RemoteEngine process. Its LRU is keyed by immutable profile ID and retains up to COMFY_MODAL_LLM_MAX_RESIDENT_MODELS.

Before loading a new LLM, the manager:

1. asks ComfyUI's memory manager to release idle managed models if needed;
2. checks torch.cuda.mem_get_info();
3. evicts least-recently used LLMs until the profile estimate plus reserve fits;
4. fails with measured free/total VRAM rather than attempting an unsafe load.

ComfyUI sees the LLM's real CUDA allocation when calculating free memory, even though it does not own the Transformers object. The configurable reserve gives subsequent image/video nodes headroom. Executions are serialized within a worker; co-residency does not promise simultaneous CUDA kernels.

Security And Reproducibility

• Repository IDs are metadata-inspected on CPU before any weight download or GPU allocation.
• Every profile pins a 40-character repository commit.
• Generated profile IDs include the SHA-256 digest of all runtime-defining fields.
• trust_remote_code must remain false.
• The remote model loader uses local_files_only=true.
• The CPU stager is the only network download path.
• Snapshot allow-patterns omit Python source and pickle weight formats.
• Text and PDF input sizes are bounded before prompt construction.
• The compatibility policy participates in the remote runtime fingerprint.

Compatibility-policy changes remain deployment changes. New repositories using a reviewed architecture and quantization can generate immutable profiles without changing the image.

Current Backend Boundary

The compatibility registry selects Transformers for Muse-Glimmer and vLLM for Qwen3.5. Qwen block-FP8 and NVIDIA ModelOpt NVFP4 checkpoints share the vLLM adapter. Generated vLLM profiles set an explicit KV-cache byte budget and a conservative 32K default context instead of allowing the serving engine to reserve nearly all available B300 memory.

The B300 runtime pins Torch 2.13.0, torchvision 0.28.0, Transformers 5.15.0, and vLLM 0.27.1. vLLM is installed only in CUDA 13.2 B300/B200+ images; established CUDA 12.8 GPU profiles retain their existing Torch 2.10 stack. The initial vLLM policy uses a 12 GiB KV cache, one request at a time, and eager execution to keep graph-capture allocations predictable beside ComfyUI.