nyueki-ComfyUI-RemoteCLIPLoader/README.md

# Remote CLIP for ComfyUI

Run the CLIP text encoder on one machine and use it from another. This is useful for offloading CLIP to a separate GPU or host to free up VRAM on the machine doing image generation.

As well as text encoding (conditioning), the remote CLIP also supports text generation, so LLM-style text encoders (such as Gemma) can drive nodes like **Generate Text** and **Generate LTX2 Prompt** from across the network. Image, video, and audio inputs are forwarded to the Sender for both encoding and generation.

There are two roles:

- **Sender** — the machine that holds the CLIP model and runs the encoder.
- **Loader** — the machine running the workflow, which connects to the Sender in place of a local CLIP.

## Installation

Clone the repository into your `ComfyUI/custom_nodes` folder on both machines:

```bash
git clone https://github.com/nyueki/ComfyUI-RemoteCLIPLoader.git
```

Restart ComfyUI on both machines.

## Usage

### Sender

On the machine holding the CLIP model:

1. Add the **Send Remote CLIP** node.
2. Connect your CLIP model to it.
3. Set `listen_port` (default `8181`).
4. Optionally set an `auth_token` to require a matching secret from Loaders, and a `bind_host` to restrict the listening interface (default `0.0.0.0`, all interfaces).
5. Queue the workflow to start listening.

### Loader

On the machine running the workflow:

1. Add the **Load Remote CLIP** node.
2. Set `worker_ip` to the Sender's IP address.
3. Set `port` to match the Sender (default `8181`).
4. Connect the output anywhere a normal CLIP is expected.

Optional inputs:

- `auth_token` — A shared secret. If the Sender was started with a token, the same value must be set here or requests are rejected. Leave blank when the Sender has no token. Can also be supplied through the `REMOTE_CLIP_TOKEN` environment variable.
- `transport_precision` — Precision of embeddings sent over the network:
  - `auto` (default) — sends fp16 to remote hosts to save bandwidth, but keeps full precision when the worker is on localhost, where bandwidth is not a constraint.
  - `fp16` — always send half precision. Halves bandwidth at the cost of a small precision loss; useful on slower links.
  - `fp32` — never downcast; full precision regardless of host.

The Sender and Loader must run the same plugin version. A protocol mismatch is rejected with a clear error rather than producing incorrect output.

## Text generation

The remote CLIP works as a drop-in for generation-capable text encoders, not just plain conditioning. Connect **Load Remote CLIP** to a generation node such as **Generate Text** or **Generate LTX2 Prompt** and the prompt, sampling settings, and any image/video/audio inputs are sent to the Sender, which runs the generation and returns the text.

- The model held by the Sender must actually support generation (for example a Gemma-class encoder). If it only does encoding, the generation nodes will error.
- Generation is not cached, since sampling and seeds make each run distinct.
- Generation can take much longer than encoding (autoregressive decoding runs token by token), so its network read uses an extended timeout and is not retried, to avoid kicking off a second expensive run on a brief hiccup. Generation speed is bound by the Sender's model and hardware, not by this plugin.

## LoRAs

The text encoder runs on the Sender, so any LoRA affecting CLIP is ultimately applied there. There are two ways to set this up.

### Apply on the Sender

Place the **LoraLoaderCLIPOnly** node between the Checkpoint or CLIP Loader and the **Send Remote CLIP** node on the Sender. Select the LoRA, set `strength_clip`, and connect the output to **Send Remote CLIP**.

### Apply from the Loader

You can also place **LoraLoaderCLIPOnly** on the Loader, right after **Load Remote CLIP**. When it detects a remote connection it does not patch locally; instead it forwards the LoRA name and strength to the Sender, which applies them.

- The LoRA file must exist in the Sender's `loras` folder. The Loader only sends the name and strength, so a missing file fails with a "LoRA not found on worker" error.
- Multiple `LoraLoaderCLIPOnly` nodes can be stacked; they accumulate and are all applied on the Sender.
- The standard ComfyUI LoRA Loader does not work against a remote CLIP. Use **LoraLoaderCLIPOnly**.

### Model-only LoRAs

If a LoRA only needs to affect image generation (the UNet) and not the text prompt, use the standard **LoraLoaderModelOnly** node on the Loader as usual.

## Performance

The Sender caches encoding results to keep repeated requests fast:

- Embedding cache — an identical prompt and settings return without re-encoding. Requests carrying image/video/audio inputs are not cached.
- Patched-CLIP cache — a given LoRA stack is applied once and reused, avoiding repeated patching.

The Loader reconnects automatically if the connection drops, so brief network interruptions do not break a running encode. Generation is the exception: it is not auto-retried, so a dropped connection mid-generation surfaces as an error rather than silently running twice.

## Notes

- Authentication: when exposing the Sender on a shared network, set the same `auth_token` on both nodes. If the Sender binds to `0.0.0.0` without a token, anyone on the network can use the model, and the Sender logs a warning.
- Bind host: the Sender binds to `0.0.0.0` by default. Set `bind_host` to a specific address to restrict access.
- Firewall: allow the listening port (default 8181) through the firewall on the Sender.
- Network: intended for use on a local network.
- Encryption: traffic is not encrypted. Do not expose this directly over the public internet; tunnel it through a VPN or SSH instead.

Node category: `Remote CLIP`

## Contributing

Contributions are welcome, whether bug fixes, improvements, or new features.

1. Fork the repository.
2. Create a feature branch: `git checkout -b feature/new-feature`
3. Commit your changes: `git commit -m 'Add a new feature'`
4. Push the branch: `git push origin feature/new-feature`
5. Open a pull request against the original repository.

If you find a bug or have a suggestion but aren't ready to implement it, please open an issue.