nodes/EricRollei-Visual_Research
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
main
EricRollei-Visual_Research/docs/SERVER_SETUP.md
T

7.2 KiB
Raw Permalink Blame History

Eric Visual Research - Server Setup Guide

This package requires a running Gen-Searcher-8B model served via SGLang. Gen-Searcher-8B is a fine-tuned Qwen3-VL-8B that has been specifically trained to perform multi-hop web research for image generation - searching the web, finding visual references, and writing detailed grounded prompts.

Why SGLang? Why not LM Studio?

Gen-Searcher-8B is a vision-language model that makes structured multi-turn tool calls (search, image_search, browse). LM Studio does not currently support multi-turn tool calling for VL models. SGLang does, and is significantly faster.

Why Linux / WSL?

SGLang is developed for Linux. On Windows, several components fail:

  • deep_gemm (matrix kernel optimization) requires nvcc - the CUDA compiler - which is not easily available on bare Windows
  • SGLang's process management and async workers assume Linux process semantics
  • FlashInfer (attention backend) requires CUDA 12.9+ for Blackwell GPUs on Linux; the Windows support is incomplete

WSL2 (Windows Subsystem for Linux) gives you a full Ubuntu environment inside Windows with direct CUDA GPU access via your existing Windows NVIDIA driver. No dual-boot, no VM. Linux users skip WSL entirely.

Prerequisites

  • NVIDIA GPU with 20GB+ VRAM (for Gen-Searcher-8B)
  • NVIDIA driver 560+ (already installed if you're using ComfyUI with recent CUDA)
  • Windows 10/11 (build 19041+) for WSL2, or any Linux distro
  • ~30GB free disk space (17GB model + Python env)

Step 1 - Install WSL2 (Windows only)

Open PowerShell as Administrator:

# Enable WSL2 features
dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart
wsl --set-default-version 2

# Install Ubuntu 24.04
winget install --id Canonical.Ubuntu.2404 -e

Restart Windows.

If you get error 0x80370114 after restart:

  • Enter BIOS (F1/Del at startup)
  • Find SVM Mode (AMD) or VT-x (Intel) under CPU/Security settings
  • Enable it, save and exit

After restart, launch Ubuntu from Start Menu or run ubuntu2404 in PowerShell. Create a username (lowercase only, e.g. eric) and password when prompted.

Step 2 - Install CUDA Toolkit in WSL

Inside the Ubuntu terminal:

# Add NVIDIA CUDA repository
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update

# Install CUDA toolkit (compiler + headers - does NOT reinstall the driver)
sudo apt-get install -y cuda-toolkit-12-8

# Add to PATH permanently
echo 'export PATH=/usr/local/cuda-12.8/bin:$PATH' >> ~/.bashrc
echo 'export CUDA_HOME=/usr/local/cuda-12.8' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-12.8/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

# Verify
nvcc --version

You should see Cuda compilation tools, release 12.8.

Step 3 - Create Python Environment and Install SGLang

# Create a dedicated venv
python3 -m venv ~/sglang_env
source ~/sglang_env/bin/activate
pip install -U pip

# Install PyTorch with CUDA 12.8
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

# Verify GPU is visible
python3 -c "import torch; print(torch.cuda.get_device_name(0))"

# Install SGLang
pip install sglang[all]

# Install HuggingFace tools
pip install huggingface_hub transformers accelerate

Step 4 - Download Gen-Searcher-8B (~17GB)

Choose a destination. On Windows, your drives are accessible in WSL at /mnt/c/, /mnt/d/, etc. Example using an H: drive:

source ~/sglang_env/bin/activate

hf download GenSearcher/Gen-Searcher-8B \
    --local-dir /mnt/h/models/Gen-Searcher-8B

Supports automatic resume if the download is interrupted - just run again.

Step 5 - Start the Server

Save this as ~/start_gen_searcher.sh:

#!/bin/bash
source ~/sglang_env/bin/activate

export SGLANG_DISABLE_DEEP_GEMM=1

# CUDA_VISIBLE_DEVICES: use the GPU with 20GB+ VRAM
# 0 = first GPU, 1 = second GPU, etc.
# Run: python3 -c "import torch; [print(i, torch.cuda.get_device_name(i)) for i in range(torch.cuda.device_count())]"
# to find which device is which.

CUDA_VISIBLE_DEVICES=0 \
sglang serve \
    --model-path /mnt/h/models/Gen-Searcher-8B \
    --host 0.0.0.0 \
    --port 30000 \
    --trust-remote-code \
    --dtype bfloat16 \
    --mem-fraction-static 0.85 \
    --context-length 16384 \
    --attention-backend triton \
    --sampling-backend pytorch \
    --tool-call-parser qwen25

Run it:

chmod +x ~/start_gen_searcher.sh
~/start_gen_searcher.sh

Note on --attention-backend triton: Required for NVIDIA Blackwell (RTX 4090, RTX 5090, RTX PRO 6000, etc.) with CUDA 12.8. FlashInfer requires CUDA 12.9+ for Blackwell. If you have CUDA 12.9+ installed, you can remove this flag for ~20% better throughput.

Watch for: The server is fired up and ready to roll!

Keep this terminal open while using ComfyUI. The server runs in the foreground.

Step 6 - Verify the Server

In a second terminal:

curl http://localhost:30000/health
curl http://localhost:30000/v1/models

Or in Python:

import urllib.request, json
with urllib.request.urlopen("http://localhost:30000/v1/models") as r:
    print(json.loads(r.read()))

Step 7 - Configure API Keys

  1. Copy api_keys.ini.example to api_keys.ini in the package directory
  2. Get a free Serper API key at https://serper.dev (2,500 free queries/month)
  3. Add your key: 
    [api_keys]
serper = your-actual-key-here

Alternatively, set environment variables:

export SERPER_API_KEY=your-key-here   # on Linux/WSL
# or in ComfyUI's launch script on Windows

Linux (Native) Users

Skip Steps 1-2 entirely. Start from Step 3. If you have a different CUDA version, replace cu128 with your version in the pip URL.

Troubleshooting

0x80370114 on Windows: Enable SVM/VT-x virtualization in BIOS. See Step 1.

CUDA device not found in WSL: Your NVIDIA driver must be 560+. Check with nvidia-smi in PowerShell (not inside WSL - WSL uses the Windows driver).

FlashInfer requires GPUs with sm75 or higher: Add --attention-backend triton to your serve command. This is needed for Blackwell GPUs with CUDA 12.8.

deep_gemm errors on startup: Add export SGLANG_DISABLE_DEEP_GEMM=1 before the serve command. deep_gemm is an optional optimization that requires nvcc to be on the PATH at startup time.

Server runs but ComfyUI can't connect: WSL uses a different localhost from Windows. Try http://127.0.0.1:30000 instead of http://localhost:30000 in the node. If that fails, run hostname -I in WSL and use that IP.

Tool calls not working (searches not running): Ensure --tool-call-parser qwen25 is in your serve command. Without it, the model outputs tool calls as inline <tool_call> tags which the node parses as a fallback - but the server-side parser is more reliable.

Reference in New Issue View Git Blame Copy Permalink