VrchStudio-comfyui-web-viewer/docs/model_nodes.md

Model Nodes

The model nodes support TensorRT workflows without forcing an unused PyTorch UNet to remain resident on the GPU. They are backend nodes and do not require custom frontend JavaScript.

Checkpoint CLIP-only Loader @ vrch.ai

Loads only the CLIP text encoder from a checkpoint. Use it when TensorRT owns the diffusion model but prompts must remain editable during a live session.

Inputs

• ckpt_name (CHECKPOINT, required): checkpoint selected from ComfyUI's registered checkpoints paths. The node does not hard-code a default; a new node uses the first option exposed by ComfyUI.

Outputs

• CLIP: the checkpoint's CLIP text encoder.

Behavior

• Requests CLIP with output_clip=True while explicitly disabling MODEL, VAE, and CLIP Vision output.
• Uses ComfyUI's configured embeddings directories.
• Keeps ordinary ComfyUI node caching. Changing prompt text re-runs downstream CLIP encoding without reconstructing the checkpoint loader.
• Fails the prompt when the checkpoint contains no supported CLIP encoder.

The node does not expose device, dtype, cache, VAE, UNet, or CLIP Vision controls. Device and offload behavior remain under ComfyUI model management.

ControlNet Loader (CPU Offload) @ vrch.ai

Loads a ControlNet checkpoint through ComfyUI while forcing construction onto CPU. This prevents --highvram from constructing another large model directly on CUDA before ComfyUI can offload or stream it beside a resident TensorRT Engine.

Inputs

• control_net_name (CONTROL_NET, required): checkpoint selected from ComfyUI's registered controlnet paths. There is no hard-coded default.

Outputs

• CONTROL_NET: the loaded ControlNet object.

Behavior

• Temporarily overrides ComfyUI's UNet offload device only for the synchronous ControlNet load call, then restores the original function even on failure.
• Serializes loads through a process lock so concurrent calls cannot observe the temporary device override.
• Uses ComfyUI's ordinary ControlNet loader and therefore supports any ControlNet checkpoint that the installed ComfyUI version supports. Union type selection, strength, start/end percentages, and conditioning remain in downstream nodes.
• Fails the prompt when the checkpoint contains no supported ControlNet model.

TAESD Memory Profile @ vrch.ai

Overrides ComfyUI's full-VAE memory estimate with a fixed estimate appropriate for a TAESD VAE. It does not change VAE math, reserve GPU memory, or unload CLIP.

Inputs

• vae (VAE, required): a TAESD VAE.
• memory_mib (INT): reported encode/decode memory requirement.
  • default: 256
  • minimum: 64
  • maximum: 1024
  • step: 64

Outputs

• VAE: the same VAE object with the memory profile applied.

Behavior

• Applies the configured fixed estimate to both encode and decode scheduling.
• Adds vrch_memory_profile metadata for diagnostics.
• Fails the prompt when the input is not a TAESD VAE.

64 MiB is the validated Simple workflow value. The node default remains the more conservative 256 MiB; lower values should be qualified with the actual resolution and live workload before deployment.

TensorRT Auto Loader @ vrch.ai

Loads a selected local TensorRT Engine and exposes the actual backend and status. PyTorch inputs are lazy, so a healthy TensorRT path does not construct an unused checkpoint UNet.

Required inputs

• load_mode: backend policy. Default: auto.
  • auto: try TensorRT; return a PyTorch fallback when Engine loading is unavailable or fails.
  • tensorrt: require TensorRT and fail the prompt instead of falling back.
  • pytorch: bypass TensorRT.
• engine_name: basename of an .engine file from ComfyUI's registered TensorRT paths, including output/tensorrt. There is no hard-coded Engine default; No TensorRT Engine Found is shown when none are registered.
• debug (BOOLEAN): concise loader, cache, and fallback diagnostics. Default: false.

Optional inputs

• model (MODEL, lazy): an existing PyTorch model used for model-family inference or fallback only when required.
• model_type: TensorRT model family. Default: auto.
  • auto
  • sdxl_base
  • sdxl_refiner
  • sd1.x
  • sd2.x-768v
  • svd
  • sd3
  • auraflow
  • flux_dev
  • flux_schnell
• fallback_checkpoint: checkpoint used to load only a diffusion MODEL when PyTorch fallback is needed. It does not load CLIP, VAE, or CLIP Vision. There is no hard-coded default.
• require_controlnet (BOOLEAN): require the installed TensorRT loader and selected Engine to implement the VRCH residual ControlNet contract. Default: false.

Outputs

• model (MODEL): TensorRT model or PyTorch fallback.
• backend (STRING): actual backend, tensorrt or pytorch.
• status (STRING): selected Engine, residual/control state, or fallback reason.

Lazy loading and fallback

• An explicit model_type lets the TensorRT path skip the lazy model input.
• model_type=auto evaluates model only to infer the model family.
• auto or pytorch evaluates model when no fallback_checkpoint is configured. When a fallback checkpoint is configured, fallback loads only its diffusion MODEL on demand.
• tensorrt mode never hides Engine selection, schema, deserialization, or compatibility errors behind a PyTorch fallback.
• auto fallback covers loader-time failures. TensorRT inference errors after the MODEL has been returned are not retried with PyTorch.

Engine choices are host-local. A workflow may contain an Engine name that was saved on another host or later removed; runtime validation handles that as a fallback in auto mode or an error in tensorrt mode. Engine paths are limited to safe basenames inside registered TensorRT roots.

Residual ControlNet contract

require_controlnet=true requires all of the following:

1. the installed TensorRTLoader advertises vrch-tensorrt-controlnet-residual-v1;
2. the selected Engine contains the residual input schema; and
3. an upstream ControlNet produces residuals at inference time.

A plain Engine is rejected when ControlNet is required. A residual Engine may also serve a ControlNet-OFF workflow when require_controlnet=false; missing residuals are zero-filled by the qualified TensorRT loader. Switching the same residual Engine between required and optional modes reuses the cached Engine instead of deserializing a second copy.

When ComfyUI's ControlNetApplyAdvanced receives exact strength=0, it may produce no control dictionary. A workflow with require_controlnet=true then fails loudly by design instead of silently generating an uncontrolled image. Product workflows should expose an explicit ControlNet ON/OFF state or keep the ON strength above zero.

Cache and diagnostics

The Engine cache key includes model type, Engine name, device/inode identity, size, and modification time. Replacing an Engine invalidates the cached MODEL. The status output records the Engine name, whether the residual schema is present, and whether ControlNet is required.

Refresh the ComfyUI frontend after adding or removing Engine files so native dropdown choices are rebuilt.

Recommended TensorRT workflow wiring

• Route Checkpoint CLIP-only Loader to the prompt encoders.
• Route TensorRT Auto Loader model to the sampler.
• For ControlNet workflows, route ControlNet Loader (CPU Offload) through the appropriate Union/specialized ControlNet configuration and conditioning nodes.
• Route a TAESD VAE through TAESD Memory Profile before VAE encode/decode.
• Use an explicit model_type and fallback_checkpoint so the healthy TensorRT path never evaluates a full checkpoint MODEL.
• Set require_controlnet=true only for workflows whose output must be controlled; leave it false for ordinary VJ workflows.

These nodes do not modify image size, prompt, seed, steps, CFG, denoise, ControlNet strength, or other generation defaults.