Bump version: 1.4.0 → 1.5.0

From experience elsewhere, there seem to be some providers that default
to streaming.

Also update dependencies in pyproject.toml to match requirements.txt.

.github/workflows/publish.yml

@@ -8,17 +8,19 @@ on:
     paths:
       - "pyproject.toml"
 
+permissions:
+  issues: write
+
 jobs:
   publish-node:
     name: Publish Custom Node to registry
     runs-on: ubuntu-latest
-    # if this is a forked repository. Skipping the workflow.
-    if: github.event.repository.fork == false 
+    if: ${{ github.repository_owner == 'asaddi' }}
     steps:
       - name: Check out code
         uses: actions/checkout@v4
       - name: Publish Custom Node
-        uses: Comfy-Org/publish-node-action@main
+        uses: Comfy-Org/publish-node-action@v1
         with:
           ## Add your own personal access token to your Github Repository secrets and reference it here.
           personal_access_token: ${{ secrets.REGISTRY_ACCESS_TOKEN }}

README.md

@@ -1,3 +1,5 @@
+
+
 # Yet Another LLM Node (for ComfyUI)
 
 Yet another set of LLM nodes for [ComfyUI](https://github.com/comfyanonymous/ComfyUI). This one only supports OpenAI-like APIs, but of course can be used with local LLM providers such as [llama.cpp](https://github.com/ggerganov/llama.cpp) (and ollama, LM Studio, among many others).
@@ -69,6 +71,10 @@ The input is optional, meaning you can mute the node leading into it or even dis
 
 The node will output whatever is currently being displayed in the textbox. The node text will also be saved in exported workflows and generated images.
 
+## "Prepend/Append" Node
+
+A simple utility node that takes a text input and optionally adds a prefix and/or suffix to it. This is useful for dynamically injecting instructions or context into prompts without altering the source text, as demonstrated in the `prompt-optimizer` example workflow.
+
 ## My Related Projects
 
 * https://github.com/asaddi/YALLM-LlamaVision Basic ComfyUI node for Llama 3.2 Vision. Sampler/model nodes are cross-compatible with this project.

examples/README.md

@@ -16,4 +16,4 @@ The YALLM nodes query a (vision) LLM, asking for a description of the loaded ima
 
 ## prompt-optimizer
 
-Asks the LLM to optimize a prompt by word count while keeping descriptive details. Uses the "String Function" node from <https://github.com/pythongosssss/ComfyUI-Custom-Scripts/> in order to prepend some text to the original prompt in order to use it as a system prompt.
+Asks the LLM to optimize a prompt by word count while keeping descriptive details. Uses the "Prepend/Append" node in order to prepend some text to the original prompt in order to use it as a system prompt.

examples/prompt-optimizer.json

@@ -1,54 +1,7 @@
 {
-  "last_node_id": 5,
-  "last_link_id": 4,
+  "last_node_id": 6,
+  "last_link_id": 7,
   "nodes": [
-    {
-      "id": 4,
-      "type": "StringFunction|pysssss",
-      "pos": {
-        "0": 0,
-        "1": 350
-      },
-      "size": {
-        "0": 400,
-        "1": 244
-      },
-      "flags": {},
-      "order": 2,
-      "mode": 0,
-      "inputs": [
-        {
-          "name": "text_b",
-          "type": "STRING",
-          "link": 2,
-          "widget": {
-            "name": "text_b"
-          },
-          "shape": 7
-        }
-      ],
-      "outputs": [
-        {
-          "name": "STRING",
-          "type": "STRING",
-          "links": [
-            3
-          ],
-          "slot_index": 0
-        }
-      ],
-      "properties": {
-        "Node name for S&R": "StringFunction|pysssss"
-      },
-      "widgets_values": [
-        "append",
-        "no",
-        "Use the following text as reference:\n",
-        "",
-        "",
-        "Use the following text as reference:\n\"A beautiful, serene landscape unfolds before us, with rolling hills and lush greenery stretching as far as the eye can see. A delicate, antique glass bottle lies abandoned on a rocky outcropping, its transparent walls glowing with an ethereal light that seems almost otherworldly. The bottle appears to be filled with a deep, rich purple liquid that perfectly captures the color of a distant galaxy, with swirling clouds of gas and dust visible within. The surrounding landscape is bathed in a soft, golden light, with the warm rays of the sun casting a gentle glow across the scene. A few wispy clouds drift lazily across the sky, adding a sense of depth and atmosphere to the already breathtaking view. The overall mood is one of peaceful contemplation, as if the viewer has stumbled upon a secret, hidden world that exists just beyond the edge of reality.\""
-      ]
-    },
     {
       "id": 5,
       "type": "LLMTextLatch",
@@ -90,50 +43,6 @@
         ""
       ]
     },
-    {
-      "id": 2,
-      "type": "LLMTextLatch",
-      "pos": {
-        "0": -450,
-        "1": 350
-      },
-      "size": {
-        "0": 400,
-        "1": 200
-      },
-      "flags": {},
-      "order": 0,
-      "mode": 0,
-      "inputs": [
-        {
-          "name": "text_input",
-          "type": "STRING",
-          "link": null,
-          "widget": {
-            "name": "text_input"
-          },
-          "shape": 7
-        }
-      ],
-      "outputs": [
-        {
-          "name": "text",
-          "type": "STRING",
-          "links": [
-            2
-          ],
-          "slot_index": 0
-        }
-      ],
-      "properties": {
-        "Node name for S&R": "LLMTextLatch"
-      },
-      "widgets_values": [
-        "\"A beautiful, serene landscape unfolds before us, with rolling hills and lush greenery stretching as far as the eye can see. A delicate, antique glass bottle lies abandoned on a rocky outcropping, its transparent walls glowing with an ethereal light that seems almost otherworldly. The bottle appears to be filled with a deep, rich purple liquid that perfectly captures the color of a distant galaxy, with swirling clouds of gas and dust visible within. The surrounding landscape is bathed in a soft, golden light, with the warm rays of the sun casting a gentle glow across the scene. A few wispy clouds drift lazily across the sky, adding a sense of depth and atmosphere to the already breathtaking view. The overall mood is one of peaceful contemplation, as if the viewer has stumbled upon a secret, hidden world that exists just beyond the edge of reality.\"",
-        true,
-        ""
-      ]
-    },
     {
       "id": 1,
       "type": "LLMProvider",
@@ -146,7 +55,7 @@
         "1": 106
       },
       "flags": {},
-      "order": 1,
+      "order": 0,
       "mode": 0,
       "inputs": [],
       "outputs": [
@@ -203,7 +112,7 @@
         {
           "name": "system_prompt",
           "type": "STRING",
-          "link": 3,
+          "link": 7,
           "widget": {
             "name": "system_prompt"
           },
@@ -230,6 +139,93 @@
         "Optimize the given text-to-image prompt for word count. Try to keep as many descriptive details as possible.\n\nOUTPUT ONLY THE MODIFIED PROMPT.",
         ""
       ]
+    },
+    {
+      "id": 2,
+      "type": "LLMTextLatch",
+      "pos": {
+        "0": -450,
+        "1": 350
+      },
+      "size": {
+        "0": 400,
+        "1": 200
+      },
+      "flags": {},
+      "order": 1,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "text_input",
+          "type": "STRING",
+          "link": null,
+          "widget": {
+            "name": "text_input"
+          },
+          "shape": 7
+        }
+      ],
+      "outputs": [
+        {
+          "name": "text",
+          "type": "STRING",
+          "links": [
+            6
+          ],
+          "slot_index": 0
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LLMTextLatch"
+      },
+      "widgets_values": [
+        "\"A beautiful, serene landscape unfolds before us, with rolling hills and lush greenery stretching as far as the eye can see. A delicate, antique glass bottle lies abandoned on a rocky outcropping, its transparent walls glowing with an ethereal light that seems almost otherworldly. The bottle appears to be filled with a deep, rich purple liquid that perfectly captures the color of a distant galaxy, with swirling clouds of gas and dust visible within. The surrounding landscape is bathed in a soft, golden light, with the warm rays of the sun casting a gentle glow across the scene. A few wispy clouds drift lazily across the sky, adding a sense of depth and atmosphere to the already breathtaking view. The overall mood is one of peaceful contemplation, as if the viewer has stumbled upon a secret, hidden world that exists just beyond the edge of reality.\"",
+        true,
+        ""
+      ]
+    },
+    {
+      "id": 6,
+      "type": "LLMPrependAppend",
+      "pos": {
+        "0": 0,
+        "1": 350
+      },
+      "size": [
+        400,
+        200
+      ],
+      "flags": {},
+      "order": 2,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "text_input",
+          "type": "STRING",
+          "link": 6,
+          "widget": {
+            "name": "text_input"
+          }
+        }
+      ],
+      "outputs": [
+        {
+          "name": "text",
+          "type": "STRING",
+          "links": [
+            7
+          ],
+          "slot_index": 0
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LLMPrependAppend"
+      },
+      "widgets_values": [
+        "",
+        "Use the following text as reference:\n",
+        ""
+      ]
     }
   ],
   "links": [
@@ -241,22 +237,6 @@
       0,
       "LLMMODEL"
     ],
-    [
-      2,
-      2,
-      0,
-      4,
-      0,
-      "STRING"
-    ],
-    [
-      3,
-      4,
-      0,
-      3,
-      3,
-      "STRING"
-    ],
     [
       4,
       3,
@@ -264,16 +244,32 @@
       5,
       0,
       "STRING"
+    ],
+    [
+      6,
+      2,
+      0,
+      6,
+      0,
+      "STRING"
+    ],
+    [
+      7,
+      6,
+      0,
+      3,
+      3,
+      "STRING"
     ]
   ],
   "groups": [],
   "config": {},
   "extra": {
     "ds": {
-      "scale": 0.9090909090909091,
+      "scale": 0.7513148009015777,
       "offset": [
-        111.69999999999979,
-        88.6999999999999
+        320.79927272727275,
+        220.6500000000004
       ]
     }
   },

models.yaml.example

@@ -6,6 +6,16 @@ models:
     api_key: none
     model: whatever
 
+  - name: "[LOCAL] LM Studio"
+    base_url: http://localhost:1234/v1
+    api_key: none
+    model: changeme
+
+  - name: "[LOCAL] Ollama"
+    base_url: http://localhost:11434/v1
+    api_key: none
+    model: changeme
+
 # Example of using a remote endpoint
 # If prefixed with "os.environ/" (similar to LiteLLM), base_url and/or api_key
 # will take their value from the named environment variable.

nodes.py

@@ -6,12 +6,14 @@ from typing import Any, Optional
 
 from aiohttp import web
 from aiohttp.web_request import Request
-import openai
 from PIL.Image import Image
 from pydantic import BaseModel
+import requests
 import torchvision.transforms.functional as F
 import yaml
 
+from .utils import PromptUtils, WorkflowUtils
+
 from server import PromptServer
 
 
@@ -114,6 +116,55 @@ MODELS = Models()
 MODELS.load()
 
 
+class LLMPrependAppend:
+    @classmethod
+    def INPUT_TYPES(cls):
+        return {
+            "required": {
+                "text_input": (
+                    "STRING",
+                    {
+                        "forceInput": True,
+                        "multiline": True,
+                    },
+                ),
+            },
+            "optional": {
+                "prefix": (
+                    "STRING",
+                    {
+                        "multiline": True,
+                    },
+                ),
+                "suffix": (
+                    "STRING",
+                    {
+                        "multiline": True,
+                    },
+                ),
+            },
+        }
+
+    TITLE = "Prepend/Append"
+
+    RETURN_TYPES = ("STRING",)
+    RETURN_NAMES = ("text",)
+
+    FUNCTION = "execute"
+
+    CATEGORY = "YALLM"
+
+    def execute(self, text_input, prefix=None, suffix=None):
+        if prefix is None:
+            prefix = ""
+        if suffix is None:
+            suffix = ""
+
+        text = prefix + text_input + suffix
+
+        return (text,)
+
+
 class LLMTextLatch:
     @classmethod
     def INPUT_TYPES(cls):
@@ -138,14 +189,21 @@ class LLMTextLatch:
                     {
                         "forceInput": True,
                         "multiline": True,
+                        "lazy": True,
                     },
                 ),
             },
+            "hidden": {
+                "unique_id": "UNIQUE_ID",
+                "prompt": "PROMPT",
+                "extra_pnginfo": "EXTRA_PNGINFO",
+            },
         }
 
     @classmethod
     def VALIDATE_INPUTS(cls, text):
         # Always valid, even if empty
+        # TODO I don't think this is necessary
         return True
 
     TITLE = "Text Latch"
@@ -159,13 +217,30 @@ class LLMTextLatch:
 
     CATEGORY = "YALLM"
 
-    def execute(self, text, replace, text_input=None):
+    def check_lazy_status(
+        self, text, replace, unique_id, prompt, extra_pnginfo, text_input=None
+    ):
+        # Require whatever is connected to text_input to be evaluated
+        # contingent on replace.
+        if replace and text_input is None:
+            # Do we have something connected to text_input?
+            pu = PromptUtils(prompt)
+            if pu.is_input_connected(unique_id, "text_input"):
+                return ["text_input"]
+        # All good, nothing else needs to be evaluated
+        return []
+
+    def execute(self, text, replace, unique_id, prompt, extra_pnginfo, text_input=None):
         if text_input is not None and replace:
             text = text_input
 
         if text is None:
             text = ""
 
+        # Save the current value in the outgoing metadata
+        wfu = WorkflowUtils(extra_pnginfo)
+        wfu.set_widget(unique_id, 0, text)
+
         # An oddity: the values of the UI dictionary have to be wrapped in
         # a tuple/list, just like the result?
         return {"ui": {"text": (text,)}, "result": (text,)}
@@ -272,20 +347,19 @@ SamplerSetting = tuple[str, Any]
 
 class LLMModel:
     def __init__(self, prov_def: ProviderDefinition, model: str | None = None):
-        self._llm = openai.OpenAI(
-            base_url=prov_def.resolve_base_url(),
-            api_key=(
-                prov_def.resolve_api_key() or "none"
-            ),  # openai package requires something for API key
-        )
+        self._base_url = prov_def.resolve_base_url()
+        self._headers = {}
+        if (api_key := prov_def.resolve_api_key()) and api_key != "none":
+            self._headers["authorization"] = f"Bearer {api_key}"
         if model is None:
             self._model = getattr(prov_def, "model", None)
         else:
             self._model = model
 
     def get_models(self):
-        models = list(self._llm.models.list())
-        return [m.id for m in models]
+        with requests.get(self._base_url + "/models", headers=self._headers) as r:
+            models_json = r.json()
+        return [m["id"] for m in models_json["data"]]
 
     def chat_completion(
         self,
@@ -329,34 +403,35 @@ class LLMModel:
 
         sampler_order = []
         extra_args = {}
-        extra_body = {}
         for k, v in samplers:
             sampler_order.append(k)
-            if k in (
-                "temperature",
-                "top_p",
-            ):  # The only ones supported directly by openai package
-                extra_args[k] = v
-            else:
-                # The rest have to go into "extra_body"
-                extra_body[k] = v
-        extra_body["samplers"] = (
+            extra_args[k] = v
+        extra_args["samplers"] = (
             sampler_order  # Hopefully won't cause an issue for non-llama.cpp endpoints
         )
 
         if seed is not None:
             extra_args["seed"] = seed
 
-        # print(f'extra_args = {repr(extra_args)}')
-        # print(f'extra_body = {repr(extra_body)}')
+        # print(f"extra_args = {repr(extra_args)}")
 
-        output = self._llm.chat.completions.create(
-            messages=messages, model=self._model, extra_body=extra_body, **extra_args
-        )
+        req_json = {
+            "model": self._model,
+            "messages": messages,
+            "stream": False,
+        }
+        req_json.update(extra_args)
 
-        # TODO Since we're not streaming, we'll need some sort of timeout. How to specify that?
+        # TODO Since we're not streaming, we'll need some sort of timeout. TODO specify that
+        with requests.post(
+            self._base_url + "/chat/completions", json=req_json, headers=self._headers
+        ) as r:
+            output = r.json()
 
-        return output.choices[0].message.content
+        if choices := output.get("choices", []):
+            return choices[0]["message"]["content"]
+        else:
+            return ""
 
 
 class LLMProvider:
@@ -528,6 +603,7 @@ async def llm_models(request: Request):
 
 
 NODE_CLASS_MAPPINGS = {
+    "LLMPrependAppend": LLMPrependAppend,
     "LLMTextLatch": LLMTextLatch,
     "LLMProvider": LLMProvider,
     "LLMModel": LLMModelNode,

providers.yaml.example

@@ -5,6 +5,14 @@ providers:
     base_url: http://localhost:8080/v1
     api_key: none
 
+  - name: "[LOCAL] LM Studio"
+    base_url: http://localhost:1234/v1
+    api_key: none
+
+  - name: "[LOCAL] Ollama"
+    base_url: http://localhost:11434/v1
+    api_key: none
+
 # Example of using a remote endpoint
 # If prefixed with "os.environ/" (similar to LiteLLM), base_url and/or api_key
 # will take their value from the named environment variable.

pyproject.toml

@@ -1,12 +1,14 @@
 [project]
 name = "comfyui-yallm-node"
 description = "Yet another set of LLM nodes for ComfyUI (for local/remote OpenAI-like APIs, multi-modal models supported)"
-version = "1.3.0"
+version = "1.5.0"
 license = {file = "LICENSE.txt"}
 dependencies = [
-    "openai",
+    "pillow",
     "pydantic",
-    "pyyaml"
+    "pyyaml",
+    "requests",
+    "torchvision"
 ]
 
 [project.urls]
@@ -19,7 +21,7 @@ DisplayName = "ComfyUI-YALLM-node"
 Icon = ""
 
 [tool.bumpversion]
-current_version = "1.3.0"
+current_version = "1.5.0"
 parse = "(?P<major>\\d+)\\.(?P<minor>\\d+)\\.(?P<patch>\\d+)"
 serialize = ["{major}.{minor}.{patch}"]
 search = "{current_version}"

requirements.txt

@@ -1,7 +1,6 @@
-openai
-pydantic
-
 # These should already be installed by ComfyUI
 pillow
+pydantic
 pyyaml
+requests
 torchvision

utils.py

@@ -0,0 +1,105 @@
+# Copyright (c) 2024 Allan Saddi <allan@saddi.com>
+class PromptUtils:
+    def __init__(self, prompt):
+        self.prompt = prompt
+
+    def get_node_info(self, node_id: int | str) -> dict:
+        node_id = str(node_id)
+        return self.prompt[node_id]
+
+    def is_input_connected(self, node_id: int | str, name: str) -> bool:
+        node_info = self.get_node_info(node_id)
+        input = node_info.get("inputs", {}).get(name)
+        # In reality, input will be a 2-elem list of source node ID + slot
+        # if it is connected.
+        return input is not None
+
+    @property
+    def output_map(self):
+        if hasattr(self, "_output_map"):
+            return self._output_map
+        # A map of (src node ID, slot) -> (dest node ID, "input name")
+        self._output_map: dict[tuple[str, int], list[tuple[str, str]]] = {}
+        for dest_id, dest_info in self.prompt.items():
+            for input_name, input in dest_info.get("inputs", {}).items():
+                if isinstance(input, list) and len(input) == 2:
+                    # This is apparently a link
+                    src_id, src_slot = input
+                    out_key = (src_id, src_slot)
+                    outputs = self._output_map.get(out_key)
+                    if outputs is None:
+                        self._output_map[out_key] = outputs = []
+                    outputs.append((dest_id, input_name))
+        return self._output_map
+
+    def is_output_connected(self, node_id: int | str, slot: int) -> bool:
+        out_key = (str(node_id), slot)
+        return out_key in self.output_map
+
+    # The assumption is that the given output slot can only be connected
+    # to nodes of the same type as the one being checked.
+    def get_downstream_nodes(self, node_id: int | str, slot: int) -> list[str]:
+        downstream: list[str] = []
+
+        to_check = [str(node_id)]
+        while to_check:
+            check_id = to_check.pop(0)
+
+            # Yes, consider the starting node downstream from itself as well
+            downstream.append(check_id)
+
+            outputs = self.output_map.get((check_id, slot))
+            if outputs is not None:
+                # NB There can be multiple outputs
+                out_ids: list[str] = [link_info[0] for link_info in outputs]
+                to_check.extend(out_ids)
+
+        return downstream
+
+
+class WorkflowUtils:
+    def __init__(self, extra_pnginfo):
+        # Note: API mode does not have extra_pnginfo (it is None)
+        self.extra_pnginfo = extra_pnginfo
+
+    def get_node_info(self, node_id: str | int) -> dict | None:
+        # Be somewhat paranoid since API mode offers much less access
+        if (
+            self.extra_pnginfo is not None
+            and (workflow := self.extra_pnginfo.get("workflow")) is not None
+            and (nodes := workflow.get("nodes")) is not None
+        ):
+            node_id = str(node_id)
+            # TODO For the time being, we aren't caching anything.
+            # Concievably, if there are a large number of nodes, a lot of time
+            # could be wasted searching.
+            my_node = [node_info for node_info in nodes if node_info["id"] == node_id]
+            if my_node:
+                return my_node[0]
+        return None
+
+    def set_property(self, node_id: str | int, property_name: str, value):
+        if (my_node := self.get_node_info(node_id)) is not None:
+            my_node["properties"][property_name] = value
+
+    def set_widget(self, node_id: str | int, slot: int, value):
+        if (my_node := self.get_node_info(node_id)) is not None:
+            values = my_node.get("widgets_values", [])
+            # TODO When would there be a mismatch?
+            if slot < len(values):
+                values[slot] = value
+
+
+class AnyType(str):
+    def __ne__(self, other):
+        if self == "*":
+            # If we're the wildcard, match anything
+            return False
+        return super().__ne__(other)
+
+
+class ComboType(str):
+    def __ne__(self, other):
+        if self == "*" and isinstance(other, list):
+            return False
+        return True