ComfyUI-LocateAnything

ComfyUI custom nodes for NVIDIA's LocateAnything-3B visual grounding model.

LocateAnything supports object detection, phrase grounding, text localization, document layout analysis, GUI grounding, and point-based localization. This integration processes ComfyUI IMAGE batches frame by frame and returns structured coordinates, annotated previews, and masks.

This node also applies runtime compatibility patches for recent transformers releases and can use the optional upstream batch runtime shipped inside the Hugging Face model snapshot.

Important Model License Notice

This repository contains integration code only. Model weights are downloaded separately from Hugging Face.

The NVIDIA LocateAnything-3B checkpoint is released under the NVIDIA License for non-commercial use. NVIDIA states that use, reproduction, and modification are permitted for academic and non-profit research purposes only. Review the official model license before downloading or using the checkpoint.

Nodes

LocateAnything Model Loader

Downloads and loads the checkpoint. The first run downloads approximately 7.8 GB into:

ComfyUI/models/LocateAnything/nvidia--LocateAnything-3B

Inputs:

Parameter	Description
model_source	Hugging Face repository ID or local snapshot directory. Defaults to nvidia/LocateAnything-3B.
download_model	Downloads missing model files automatically when enabled.
device	auto, cuda, cpu, or mps. NVIDIA CUDA is the recommended path.
dtype	auto, bfloat16, float16, or float32. auto selects an appropriate dtype for the device.
attention	sdpa, auto, or eager. SDPA is the broadly compatible default.
use_batch_runtime	Uses the official hybrid batch runtime from the model snapshot when available. Best for multi-frame batches on supported NVIDIA GPUs.
runtime_attention	Attention backend for the optional official batch runtime. la_flash is the upstream fast path.
vision_attention	Vision attention backend for the optional official batch runtime.
scheduler	Hybrid scheduler used by the official batch runtime.
group_size	Optional hybrid group size for the upstream batch runtime.
strict_attn	If enabled, the batch runtime refuses backend fallback and requires the configured attention mode.

The official checkpoint requires trust_remote_code=True. Loading it executes Python modeling files shipped in the Hugging Face model repository.

The node patches known API mismatches between the released model code and current transformers versions at load time. This keeps the integration working without editing the Hugging Face cache manually.

LocateAnything Grounding

Runs inference for every image in the connected batch. A video loaded as a ComfyUI IMAGE batch is processed frame by frame, with a native ComfyUI progress bar. The node includes a visible ? icon in its title bar that opens an in-app parameter reference popup.

Outputs:

Output	Description
answer	Raw model response. For batches, a JSON array of responses.
locations_json	Structured batch results with prompt, timing, normalized [0, 1000] coordinates, pixel coordinates, and batch_index.
annotated_image	Input batch with returned boxes and points drawn on each frame.
mask	Post-processed mask batch containing rectangular box masks and circular point masks after grow and blur.
mask_overlay	Original image or video frames with the post-processed mask blended using the selected color and opacity.

Task modes:

Mode	Description
ground_multi	Locates every instance matching a free-form description.
ground_single	Locates one instance matching a description.
detect	Detects objects matching categories or a description. Multiple categories may be separated with </c>.
ground_text	Locates a requested text phrase.
detect_text	Detects scene text. The query input is ignored.
gui_box	Locates a GUI region as a bounding box.
gui_point	Returns a point for a GUI target.
point	Returns a point for a described target.
custom	Sends query as the complete model prompt. General visual questions may return plain text, but VQA is experimental because the checkpoint is optimized for localization.

In custom mode, a question such as What is the color of the t-shirt? may return a plain-text answer. When the response contains no coordinates, annotated_image remains unchanged and mask is empty. General VQA quality is not guaranteed by this grounding-focused checkpoint.

Generation parameters:

Parameter	Description
generation_mode	hybrid, fast, or slow. See the table below.
max_new_tokens	Maximum generated tokens. Increase for dense detections; reduce for short grounding responses.
temperature	Sampling temperature. 0.0 is the safest path in this node. If a higher temperature fails, the node automatically retries with safe sampling settings.
top_p	Nucleus sampling cutoff. Relevant when temperature is above 0.
top_k	Top-k sampling cutoff. 0 disables top-k, matching the upstream worker.
repetition_penalty	Penalty for repeated tokens. The official worker uses 1.1.
point_radius	Radius in pixels used when points are drawn into the output mask.
mask_grow	Grow rectangular or circular masks by this many pixels. Negative values shrink them.
mask_blur	Gaussian blur radius applied after grow. Use 0 for hard edges.
overlay_color	Hex RGB color used for the overlay preview, such as #00ff66 or #ff0000.
overlay_opacity	Opacity of the colored overlay preview.
seed	Sampling seed. Relevant when temperature is above 0. For batches, frame N uses seed + N.
verbose	Prints the official step-by-step generation log in the terminal. Disable it for quieter runs.

Stability behavior:

• If sampling with temperature > 0 throws during generation, the node retries automatically with temperature=0.0, top_p=1.0, and top_k=0.
• If a non-custom task returns no coordinates, the node retries once with generation_mode="slow" before giving up.
• If the optional upstream batch runtime is unavailable or unsuitable for the current request, the node falls back to serial inference.

Mask geometry note: LocateAnything is a grounding model, not a segmentation model. Boxes generate rectangular masks and points generate circular masks. mask_grow and mask_blur modify these shapes but do not extract object silhouettes. Connect a segmentation node such as SAM downstream when a contour mask is required.

Generation modes:

Mode	Behavior	Tradeoff
hybrid	Uses fast parallel decoding and falls back to autoregressive decoding when needed.	Recommended balance.
fast	Uses MTP parallel decoding only.	Fastest; best for simpler scenes.
slow	Uses autoregressive decoding.	Slowest; most robust.

LocateAnything Unload Model

Releases model references and clears the CUDA cache.

Installation

ComfyUI Manager

Search for ComfyUI-LocateAnything in ComfyUI Manager after the package has been published to the Comfy Registry.

Manual Installation

cd ComfyUI/custom_nodes
git clone https://github.com/alisson-anjos/ComfyUI-LocateAnything.git
cd ComfyUI-LocateAnything
pip install -r requirements.txt

Restart ComfyUI after installation.

Basic Workflow

Load Image or Load Video
  -> LocateAnything Model Loader
  -> LocateAnything Grounding
  -> Preview Image

Connect the mask output to a mask preview or downstream crop/inpaint workflow.

For an initial test:

task: ground_multi
query: person
generation_mode: hybrid
temperature: 0.0
top_k: 0

If a simple ground_multi prompt still fails to produce coordinates, try:

task: ground_multi
query: person
generation_mode: slow
temperature: 0.0
top_p: 1.0
top_k: 0

Performance Notes

• The official model card recommends generation_mode="hybrid" and up to max_new_tokens=8192 to avoid truncating dense responses.
• The optional upstream batch runtime requires the newer Hugging Face snapshot files such as batch_utils/ and kernel_utils/. If they are not present, disable use_batch_runtime.
• Without optional MagiAttention, the checkpoint falls back to PyTorch SDPA. This remains functional but may be slower for MTP decoding.
• The model card documents optimized NVIDIA GPU execution. CPU execution is exposed for compatibility but will be substantially slower.

Official References

• NVIDIA LocateAnything-3B model card
• NVlabs Eagle Embodied source
• LocateAnything project page
• LocateAnything technical report

License

The integration code in this repository is licensed under GPL-3.0. The NVIDIA model weights have separate terms described above.