ComfyUI-Lance

Custom ComfyUI nodes for ByteDance Lance — a unified 3B multimodal model supporting image and video understanding, generation, and editing in a single framework.

A video I posted about unified multimodal, and this model : https://youtu.be/fvEeEW6trXA

Official sources

• Model weights: bytedance-research/Lance on Hugging Face
• Inference pipeline: bytedance/Lance (inference_lance.py, inference_lance.sh)

This pack bundles the upstream modeling/ code in lance_source/ and calls the same validate_on_fixed_batch path as the official CLI.

Features

Node	Description
Lance Model Loader	Load and cache the Lance model (LLM + ViT + VAE)
Lance Text-to-Image	Generate images from text prompts
Lance Text-to-Video	Generate videos from text prompts
Lance Image Edit	Edit images with text instructions
Lance Video Edit	Edit videos with text instructions
Lance Image Understanding	Ask questions about images (VQA)
Lance Video Understanding	Ask questions about videos (VQA)

Installation

1. Clone into ComfyUI custom nodes

cd ComfyUI/custom_nodes/
git clone https://github.com/benjiyaya/ComfyUI-Lance.git

2. Install dependencies

pip install -r ComfyUI-Lance/requirements.txt

3. Download model weights

Download the full Hugging Face repo (or individual folders) into ComfyUI/models/lance/.

The HF layout uses Lance_3B / Lance_3B_Video; the loader also accepts lance_3b / lance_3b_video (same as upstream downloads/ in the GitHub repo).

ComfyUI/models/lance/
├── Lance_3B/               # Image tasks (HF name) — or lance_3b/
│   ├── llm_config.json
│   ├── model.safetensors
│   └── tokenizer files...
├── Lance_3B_Video/         # Video tasks (HF name) — or lance_3b_video/
│   ├── llm_config.json
│   ├── model.safetensors
│   └── tokenizer files...
├── Qwen2.5-VL-ViT/         # Vision encoder
│   ├── vit.safetensors
│   └── config.json
└── Wan2.2_VAE.pth          # VAE weights (repo root on HF)

Upstream places the same files under downloads/; ComfyUI uses models/lance/ via folder_paths.

The Lance model source code is bundled in lance_source/ — no separate clone needed.

Advanced: Set LANCE_REPO_PATH env var to use a custom Lance repo instead of the bundled source.

Hardware Requirements

• GPU: NVIDIA GPU with ≥40GB VRAM (e.g., A100 40GB, A6000)
• CUDA: 12.4+
• Python: 3.10+

Usage

Basic Text-to-Video

[Lance Model Loader] → [Lance Text-to-Video] → [ComfyUI Save Video]

1. Load the model with lance_3b_video variant
2. Enter your prompt in the Text-to-Video node
3. Set resolution, frames, CFG, seed
4. Connect to ComfyUI's native Save Video to export as MP4

Image Understanding

[Load Image] → [Lance Image Understanding]

1. Load the model with lance_3b or lance_3b_video
2. Connect an image and type your question
3. Output is a text string

Video Edit / Video Understanding

[Load Video] → [Lance Video Edit] / [Lance Video Understanding]

Connect frame batches from ComfyUI's native Load Video node or VideoHelperSuite's Load Video. The node converts frames to a temp video file for Lance processing.

Key Parameters

Aligned with official inference defaults:

Parameter	Default	Description
num_timesteps	30	Denoising steps (30 or 50 recommended)
timestep_shift	3.5	Flow matching schedule shift
cfg_text_scale	4.0	Classifier-free guidance scale
num_frames	50	Video frames (max 121); T2I uses 1
resolution	video_480p / image_768res	Preset per task

Task ↔ model variant (official)

Task	ComfyUI node	Loader variant	resolution	Notes
t2i	Text-to-Image	lance_3b	image_768res	text_template=False, num_frames=1
t2v	Text-to-Video	lance_3b_video	video_* preset	text_template=True, frame count snapped to 4n+1
image_edit	Image Edit	lance_3b	image_768res	text_template=True, size from input
video_edit	Video Edit	lance_3b_video	video_480p	text_template=True, size from input
x2t_image	Image Understanding	either	—	No diffusion steps
x2t_video	Video Understanding	lance_3b_video	—	No diffusion steps

Model Caching

The model is loaded once and cached per GPU. Multiple nodes can share the same loaded model without re-loading.

License

This ComfyUI wrapper is provided as-is. The Lance model is licensed under Apache 2.0 by ByteDance. See https://github.com/bytedance/Lance for details.