diff --git a/README.md b/README.md index cded4c6..60c3b64 100644 --- a/README.md +++ b/README.md @@ -1,142 +1,71 @@ # 🌐 Remote CLIP for ComfyUI -Run **CLIP text encoding on a different machine** and use it seamlessly inside **ComfyUI** πŸš€ -Perfect for offloading heavy CLIP models to another PC or server. +**Offload heavy CLIP models to a different machine to save VRAM.** + +These custom nodes allow you to run the **CLIP model** on one computer (The **Sender**) and use it wirelessly on another computer (The **Loader**). --- -## ✨ What is this? +## βš™οΈ Installation -This project adds **two custom ComfyUI nodes** that let you: +Run this command in your `ComfyUI/custom_nodes` folder on **both computers**: -* πŸ–₯️ Run a **CLIP Worker** on one machine -* πŸŽ›οΈ Connect to it from another machine using a **CLIP Loader** -* πŸ”— Send text prompts over the network -* πŸ“¦ Receive CLIP embeddings (`cond` + `pooled`) back in real time - -Both machines run **ComfyUI**, but only **one needs the CLIP model loaded**. - ---- - -## 🧠 How it works (simple view) - -``` -[ ComfyUI (Client) ] ---> text prompt ---> [ ComfyUI (Worker) ] -[ Remote CLIP Loader ] <--- embeddings <--- [ Remote CLIP Worker ] -``` - -* Uses **TCP sockets** -* Sends metadata as **JSON** -* Sends tensors as **binary blobs** -* Automatically handles tensor shapes & dtypes - ---- - -## 🧩 Nodes Included - -### πŸ”Ή Remote CLIP Worker - -πŸ–₯️ **Runs on the machine that has the CLIP model** - -* Listens on a TCP port -* Receives text prompts -* Runs CLIP encoding -* Sends embeddings back to clients - -**Inputs** - -* `clip` β†’ Any CLIP model -* `listen_port` β†’ Port to listen on (default: `8002`) - ---- - -### πŸ”Ή Remote CLIP Loader - -πŸŽ›οΈ **Runs on the client machine** - -* Connects to the worker -* Acts like a normal CLIP object in ComfyUI -* Transparently forwards encoding requests - -**Inputs** - -* `worker_ip` β†’ IP address of the worker machine -* `port` β†’ Worker port (default: `8002`) - ---- - -## βš™οΈ Setup Instructions - -### 1️⃣ Install on both machines - -Clone this repository into your ComfyUI `custom_nodes` directory: - -``` -cd ComfyUI/custom_nodes +```bash git clone https://github.com/nyueki/ComfyUI-RemoteCLIPLoader.git -``` - -Restart ComfyUI on **both machines**. - ---- - -### 2️⃣ Start the Worker (Machine A) - -1. Add **Remote CLIP Worker** node -2. Connect a CLIP model -3. Choose a port (e.g. `8002`) -4. Queue the workflow - -βœ… The worker is now listening for connections - ---- - -### 3️⃣ Connect from Client (Machine B) - -1. Add **Remote CLIP Loader** node -2. Enter the worker’s IP address -3. Match the port number -4. Use it anywhere a CLIP node is expected πŸŽ‰ - ---- - -## πŸ“¦ What gets transmitted? - -* πŸ“ Prompt text -* πŸ”’ Tokenization options -* 🧠 CLIP embeddings: - - * `cond` β†’ `(1, 77, 768)` - * `pooled` β†’ `(1, 768)` - -All tensors are: - -* Converted to CPU -* Sent as raw bytes -* Reconstructed safely on the client - ---- - -## πŸ›‘οΈ Notes & Tips - -* πŸ”Œ Make sure the port is open on the worker machine -* 🏠 Best used on local networks (LAN) -* ⚠️ No encryption β€” don’t expose to the public internet -* πŸ§ͺ Designed for simplicity & reliability - ---- - -## 🧰 Category in ComfyUI ``` -Remote CLIP -``` + +*Restart ComfyUI on both machines.* --- -## ❀️ Why use this? +## πŸš€ How to Use -* Save VRAM on your main machine -* Share one powerful CLIP server -* Experiment with distributed ComfyUI setups -* Clean, minimal, no external dependencies +### 1️⃣ On the "Sender" Machine + +*(The PC holding the CLIP model)* + +1. Add the **Send Remote CLIP** node. +2. Connect your desired **CLIP model** to it. +3. Set the `listen_port` (default `8181`). +4. **Queue the workflow** to start listening. + +### 2️⃣ On the "Loader" Machine + +*(The PC generating images)* + +1. Add the **Load Remote CLIP** node. +2. Enter the **IP address** of the Sender machine. +3. Match the `port` number (default `8181`). +4. Connect it to your workflow wherever a normal CLIP node is expected. + +--- + +## 🎨 How to use LoRAs + +Because the **Loader** machine does not have the actual CLIP model loaded (it only has a "remote connection"), you cannot use a standard LoRA Loader on that machine to patch the CLIP. You have two options: + +### Option A: Apply LoRA to CLIP (On the Sender) + +If you need the LoRA to affect the text encoding (CLIP), you must load it on the **Sender** machine *before* transmitting the connection. + +1. On the **Sender** machine, place the **LoraLoaderCLIPOnly** node between your Checkpoint/CLIP Loader and the **Send Remote CLIP** node. +2. Select your LoRA. +3. Connect the output to the **Send Remote CLIP** node. + +### Option B: Apply LoRA to Model Only (On the Loader) + +If you only need the LoRA to affect the image generation (UNet/Model) and not the text prompt, do this on the **Loader** machine. + +1. Use the standard **LoraLoaderModelOnly** node (built into ComfyUI). +2. Connect it to your Model/UNet on the Loader machine. + +--- + +## πŸ›‘οΈ Important Notes + +* **Firewall:** Ensure the port (default 8181) is allowed through the firewall on the Sender machine. +* **Network:** Works best on a local home network (LAN). +* **Security:** Do not use this over the public internet; it is not encrypted. + +**Node Category:** `Remote CLIP` \ No newline at end of file diff --git a/__init__.py b/__init__.py index 8086c77..bf468ed 100644 --- a/__init__.py +++ b/__init__.py @@ -4,6 +4,26 @@ import threading import json import torch import time +import folder_paths +import comfy.sd +import comfy.utils + +class TensorEncoder(json.JSONEncoder): + def default(self, obj): + if isinstance(obj, torch.Tensor): + return obj.cpu().tolist() + return super().default(obj) + +class TensorEncoder(json.JSONEncoder): + def default(self, obj): + if isinstance(obj, torch.Tensor): + return {"__tensor__": True, "value": obj.cpu().tolist()} + return super().default(obj) + +def tensor_hook(dct): + if "__tensor__" in dct: + return torch.tensor(dct["value"]) + return dct def log(msg): print(f"[RemoteCLIP {time.strftime('%H:%M:%S')}] {msg}", flush=True) @@ -18,14 +38,14 @@ def recv_exact(sock, size): return buf def send_packet(sock, meta, blob): - meta_bytes = json.dumps(meta).encode("utf-8") + meta_bytes = json.dumps(meta, cls=TensorEncoder).encode("utf-8") sock.sendall(struct.pack(">Q", len(meta_bytes))) sock.sendall(meta_bytes) sock.sendall(blob) def recv_packet(sock): meta_size = struct.unpack(">Q", recv_exact(sock, 8))[0] - meta = json.loads(recv_exact(sock, meta_size).decode("utf-8")) + meta = json.loads(recv_exact(sock, meta_size).decode("utf-8"), object_hook=tensor_hook) blob = recv_exact(sock, meta.get("blob_size", 0)) return meta, blob @@ -50,7 +70,7 @@ def unpack_tensors(meta, blob): offset = 0 for name, info in meta.items(): size = info["size"] - raw = blob[offset:offset + size] + raw = bytearray(blob[offset:offset + size]) offset += size dtype = getattr(torch, info["dtype"].split(".")[-1]) t = torch.frombuffer(raw, dtype=dtype).clone() @@ -98,17 +118,17 @@ class RemoteCLIPProxy: cond, pooled = self._infer(tokens) return [[cond, {"pooled_output": pooled}]] -class RemoteCLIPWorker: +class SendRemoteCLIP: @classmethod def INPUT_TYPES(cls): - return {"required": {"clip": ("CLIP",), "listen_port": ("INT", {"default": 8002})}} + return {"required": {"CLIP": ("CLIP",), "listen_port": ("INT", {"default": 8181})}} RETURN_TYPES = () FUNCTION = "start_worker" OUTPUT_NODE = True CATEGORY = "Remote CLIP" - def start_worker(self, clip, listen_port): + def start_worker(self, CLIP, listen_port): server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) server.bind(("0.0.0.0", listen_port)) @@ -126,8 +146,8 @@ class RemoteCLIPWorker: log("Warning: invalid request, skipping") continue log(f"Encoding prompt ({len(meta['text'])} chars)") - tokens = clip.tokenize(meta["text"], **meta["kwargs"]) - cond, pooled = clip.encode_from_tokens(tokens, return_pooled=True) + tokens = CLIP.tokenize(meta["text"], **meta["kwargs"]) + cond, pooled = CLIP.encode_from_tokens(tokens, return_pooled=True) tensor_meta, blob = pack_tensors({"cond": cond, "pooled": pooled}) reply_meta = {"tensors": tensor_meta, "blob_size": len(blob)} send_packet(conn, reply_meta, blob) @@ -140,10 +160,10 @@ class RemoteCLIPWorker: conn, _ = server.accept() threading.Thread(target=handle, args=(conn,), daemon=True).start() -class RemoteCLIPLoader: +class LoadRemoteCLIP: @classmethod def INPUT_TYPES(cls): - return {"required": {"worker_ip": ("STRING", {"default": "10.0.0.37"}), "port": ("INT", {"default": 8002})}} + return {"required": {"worker_ip": ("STRING", {"default": "127.0.0.1"}), "port": ("INT", {"default": 8181})}} RETURN_TYPES = ("CLIP",) FUNCTION = "load_remote" @@ -151,6 +171,27 @@ class RemoteCLIPLoader: def load_remote(self, worker_ip, port): return (RemoteCLIPProxy(worker_ip, port),) + +class LoraLoaderCLIPOnly: + @classmethod + def INPUT_TYPES(s): + return { + "required": { + "CLIP": ("CLIP",), + "lora_name": (folder_paths.get_filename_list("loras"), ), + "strength_clip": ("FLOAT", {"default": 1.0, "min": -10.0, "max": 10.0, "step": 0.01}), + } + } -NODE_CLASS_MAPPINGS = {"RemoteCLIPWorker": RemoteCLIPWorker, "RemoteCLIPLoader": RemoteCLIPLoader} -NODE_DISPLAY_NAME_MAPPINGS = {"RemoteCLIPWorker": "Remote CLIP Worker", "RemoteCLIPLoader": "Remote CLIP Loader"} \ No newline at end of file + RETURN_TYPES = ("CLIP",) + FUNCTION = "load_lora" + CATEGORY = "Remote CLIP" + + def load_lora(self, CLIP, lora_name, strength_clip): + lora_path = folder_paths.get_full_path("loras", lora_name) + lora = comfy.utils.load_torch_file(lora_path, safe_load=True) + _, clip_lora = comfy.sd.load_lora_for_models(None, CLIP, lora, 0, strength_clip) + return (clip_lora,) + +NODE_CLASS_MAPPINGS = {"SendRemoteCLIP": SendRemoteCLIP, "LoadRemoteCLIP": LoadRemoteCLIP, "LoraLoaderCLIPOnly": LoraLoaderCLIPOnly} +NODE_DISPLAY_NAME_MAPPINGS = {"SendRemoteCLIP": "Send Remote CLIP", "LoadRemoteCLIP": "Load Remote CLIP", "LoraLoaderCLIPOnly": "LoraLoaderCLIPOnly"} \ No newline at end of file