Bump version: 1.4.0 → 1.5.0

From experience elsewhere, there seem to be some providers that default
to streaming.

Also update dependencies in pyproject.toml to match requirements.txt.

.github/workflows/publish.yml

@@ -8,17 +8,19 @@ on:
     paths:
       - "pyproject.toml"
 
+permissions:
+  issues: write
+
 jobs:
   publish-node:
     name: Publish Custom Node to registry
     runs-on: ubuntu-latest
-    # if this is a forked repository. Skipping the workflow.
-    if: github.event.repository.fork == false 
+    if: ${{ github.repository_owner == 'asaddi' }}
     steps:
       - name: Check out code
         uses: actions/checkout@v4
       - name: Publish Custom Node
-        uses: Comfy-Org/publish-node-action@main
+        uses: Comfy-Org/publish-node-action@v1
         with:
           ## Add your own personal access token to your Github Repository secrets and reference it here.
           personal_access_token: ${{ secrets.REGISTRY_ACCESS_TOKEN }}

.gitignore

@@ -1,3 +1,4 @@
 __pycache__/
 *~
 /models.yaml
+/providers.yaml

README.md

@@ -1,13 +1,13 @@
+
+
 # Yet Another LLM Node (for ComfyUI)
 
-Yet another set of LLM nodes for [ComfyUI](https://github.com/comfyanonymous/ComfyUI). This one only supports OpenAI-like APIs, but of course can be used with local LLM providers such as [llama.cpp](https://github.com/ggerganov/llama.cpp) (and, I believe, ollama and LMStudio, among others).
+Yet another set of LLM nodes for [ComfyUI](https://github.com/comfyanonymous/ComfyUI). This one only supports OpenAI-like APIs, but of course can be used with local LLM providers such as [llama.cpp](https://github.com/ggerganov/llama.cpp) (and ollama, LM Studio, among many others).
 
 This project mainly served as an exercise in creating ComfyUI nodes.
 
 ![sample workflow](yallm-sample.png)
 
-(the "Show Text" node is from the very useful https://github.com/pythongosssss/ComfyUI-Custom-Scripts)
-
 ## Features
 
 * Connection profiles are defined externally in `models.yaml`, optionally pulling API URLs & API keys from environment variables.
@@ -21,6 +21,7 @@ This project mainly served as an exercise in creating ComfyUI nodes.
 
    Also note that the OpenAI API spec only officially supports temperature and top-p, so what's actually supported will depend on your provider.
 * Seed is exposed and configurable, allowing some degree of determinism.
+* Can optionally pass in an image and query the LLM about it. Of course this means the LLM at your remote API needs to support images. Also be mindful that some VLMs like Llama 3.2 Vision don't support a system prompt when prompting with an image. (Some providers will just silently fail!)
 
 Aside from adding more samplers (as requested/as I need them), I consider this project feature complete.
 
@@ -36,11 +37,7 @@ Then within the `ComfyUI-YALLM-node` directory:
 
 and edit `models.yaml` if you would like to use more than just the `http://localhost:8080/v1` endpoint.
 
-Finally, install the [openai](https://pypi.org/project/openai/) Python package (currently the only dependency):
-
-    pip install openai
-
-or
+Finally, install the dependencies:
 
     pip install -r requirements.txt
 
@@ -50,6 +47,38 @@ or
 
 Just make sure `pip`/`python` is the same one used for your ComfyUI installation.
 
+## "LLM Provider" Node
+
+As an alternative to the "LLM Model" node (where you map "virtual" models to API providers+models), there is also an "LLM Provider" node. This node is a low-setup alternative, as you only have to define providers/connections. The available models are fetched dynamically, allowing you to make a selection in the UI.
+
+To configure it, from within the `ComfyUI-YALLM-node` directory:
+
+    cp providers.yaml.example providers.yaml
+
+and then edit `providers.yaml` to taste. It is very similar to `models.yaml` and has the same features, such as environment variable substitution.
+
+You can then use the "LLM Provider" node in place of "LLM Model", like so:
+
+![example provider workflow](yallm-provider-example.png)
+
+When you click the "fetch models" button, or select a different provider, a query will be made to the currently selected provider's `/v1/models` API endpoint. Note: The node will error out (intentionally) if this hasn't been done successfully at least once.
+
+## "Text Latch" Node
+
+This is basically "Show Text" with memory. Whenever the node is executed, if there is an input and if the "replace" switch is true, the incoming string will replace the contents of the textbox.
+
+The input is optional, meaning you can mute the node leading into it or even disconnect it.
+
+The node will output whatever is currently being displayed in the textbox. The node text will also be saved in exported workflows and generated images.
+
+## "Prepend/Append" Node
+
+A simple utility node that takes a text input and optionally adds a prefix and/or suffix to it. This is useful for dynamically injecting instructions or context into prompts without altering the source text, as demonstrated in the `prompt-optimizer` example workflow.
+
+## My Related Projects
+
+* https://github.com/asaddi/YALLM-LlamaVision Basic ComfyUI node for Llama 3.2 Vision. Sampler/model nodes are cross-compatible with this project.
+
 ## License
 
 Licensed under [BSD-2-Clause-Patent](https://opensource.org/license/bsdpluspatent).

__init__.py

@@ -1,6 +1,8 @@
 # Copyright (c) 2024 Allan Saddi <allan@saddi.com>
 from .nodes import NODE_CLASS_MAPPINGS
 
-NODE_DISPLAY_NAME_MAPPINGS = { k: v.TITLE for k,v in NODE_CLASS_MAPPINGS.items() }
+NODE_DISPLAY_NAME_MAPPINGS = {k: v.TITLE for k, v in NODE_CLASS_MAPPINGS.items()}
 
-__all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS']
+WEB_DIRECTORY = "./js"
+
+__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS", "WEB_DIRECTORY"]

examples/README.md

@@ -0,0 +1,19 @@
+# Examples
+
+Note: Nowadays I prefer the "LLM Provider" node over "LLM Model". Be sure to customize your `providers.yaml`, or if you prefer it, swap back to the "LLM Model" node.
+
+Also note: In all examples (so far), the "Text Latch" node is completely optional and is just there to provide visibility to the generated prompts.
+
+## prompt-enhancer
+
+Uses a system prompt posted by a now-deleted account [on reddit](https://www.reddit.com/r/LocalLLaMA/comments/1fi0jkj/comment/lnef616/). Though helpfully, the comment still remains.
+
+## img2text2img
+
+This just takes the example [Stable Diffusion 3.5 Medium](https://huggingface.co/stabilityai/stable-diffusion-3.5-medium) workflow but replaces the prompt with LLM output.
+
+The YALLM nodes query a (vision) LLM, asking for a description of the loaded image.
+
+## prompt-optimizer
+
+Asks the LLM to optimize a prompt by word count while keeping descriptive details. Uses the "Prepend/Append" node in order to prepend some text to the original prompt in order to use it as a system prompt.

examples/img2text2img.json

@@ -0,0 +1,907 @@
+{
+  "last_node_id": 304,
+  "last_link_id": 609,
+  "nodes": [
+    {
+      "id": 67,
+      "type": "ConditioningZeroOut",
+      "pos": {
+        "0": -126,
+        "1": 126
+      },
+      "size": {
+        "0": 211.60000610351562,
+        "1": 26
+      },
+      "flags": {},
+      "order": 10,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "conditioning",
+          "type": "CONDITIONING",
+          "link": 597
+        }
+      ],
+      "outputs": [
+        {
+          "name": "CONDITIONING",
+          "type": "CONDITIONING",
+          "links": [
+            90
+          ],
+          "slot_index": 0,
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "ConditioningZeroOut"
+      },
+      "widgets_values": []
+    },
+    {
+      "id": 71,
+      "type": "CLIPTextEncode",
+      "pos": {
+        "0": -1010,
+        "1": 252
+      },
+      "size": {
+        "0": 351.8130798339844,
+        "1": 195.57545471191406
+      },
+      "flags": {},
+      "order": 7,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "clip",
+          "type": "CLIP",
+          "link": 94
+        }
+      ],
+      "outputs": [
+        {
+          "name": "CONDITIONING",
+          "type": "CONDITIONING",
+          "links": [
+            93,
+            597
+          ],
+          "slot_index": 0,
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "CLIPTextEncode"
+      },
+      "widgets_values": [
+        ""
+      ],
+      "color": "#322",
+      "bgcolor": "#533"
+    },
+    {
+      "id": 69,
+      "type": "ConditioningCombine",
+      "pos": {
+        "0": 504,
+        "1": 126
+      },
+      "size": {
+        "0": 228.39999389648438,
+        "1": 46
+      },
+      "flags": {},
+      "order": 14,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "conditioning_1",
+          "type": "CONDITIONING",
+          "link": 91
+        },
+        {
+          "name": "conditioning_2",
+          "type": "CONDITIONING",
+          "link": 92
+        }
+      ],
+      "outputs": [
+        {
+          "name": "CONDITIONING",
+          "type": "CONDITIONING",
+          "links": [
+            604
+          ],
+          "slot_index": 0,
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "ConditioningCombine"
+      },
+      "widgets_values": []
+    },
+    {
+      "id": 8,
+      "type": "VAEDecode",
+      "pos": {
+        "0": 1386,
+        "1": -504
+      },
+      "size": {
+        "0": 210,
+        "1": 46
+      },
+      "flags": {},
+      "order": 16,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "samples",
+          "type": "LATENT",
+          "link": 572
+        },
+        {
+          "name": "vae",
+          "type": "VAE",
+          "link": 605
+        }
+      ],
+      "outputs": [
+        {
+          "name": "IMAGE",
+          "type": "IMAGE",
+          "links": [
+            274
+          ],
+          "slot_index": 0,
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "VAEDecode"
+      },
+      "widgets_values": []
+    },
+    {
+      "id": 50,
+      "type": "PreviewImage",
+      "pos": {
+        "0": 1764,
+        "1": -504
+      },
+      "size": {
+        "0": 616.689697265625,
+        "1": 613.84130859375
+      },
+      "flags": {},
+      "order": 17,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "images",
+          "type": "IMAGE",
+          "link": 274
+        }
+      ],
+      "outputs": [],
+      "properties": {
+        "Node name for S&R": "PreviewImage"
+      },
+      "widgets_values": []
+    },
+    {
+      "id": 68,
+      "type": "ConditioningSetTimestepRange",
+      "pos": {
+        "0": 126,
+        "1": 126
+      },
+      "size": {
+        "0": 317.4000244140625,
+        "1": 82
+      },
+      "flags": {},
+      "order": 12,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "conditioning",
+          "type": "CONDITIONING",
+          "link": 90
+        }
+      ],
+      "outputs": [
+        {
+          "name": "CONDITIONING",
+          "type": "CONDITIONING",
+          "links": [
+            91
+          ],
+          "slot_index": 0,
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "ConditioningSetTimestepRange"
+      },
+      "widgets_values": [
+        0.2,
+        1
+      ]
+    },
+    {
+      "id": 70,
+      "type": "ConditioningSetTimestepRange",
+      "pos": {
+        "0": 126,
+        "1": 252
+      },
+      "size": {
+        "0": 317.4000244140625,
+        "1": 82
+      },
+      "flags": {},
+      "order": 9,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "conditioning",
+          "type": "CONDITIONING",
+          "link": 93,
+          "slot_index": 0
+        }
+      ],
+      "outputs": [
+        {
+          "name": "CONDITIONING",
+          "type": "CONDITIONING",
+          "links": [
+            92
+          ],
+          "slot_index": 0,
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "ConditioningSetTimestepRange"
+      },
+      "widgets_values": [
+        0,
+        0.2
+      ]
+    },
+    {
+      "id": 105,
+      "type": "Note",
+      "pos": {
+        "0": -1260,
+        "1": -252
+      },
+      "size": {
+        "0": 210,
+        "1": 110.18948364257812
+      },
+      "flags": {},
+      "order": 0,
+      "mode": 0,
+      "inputs": [],
+      "outputs": [],
+      "properties": {
+        "text": ""
+      },
+      "widgets_values": [
+        "Make sure the resolution is multiple of 64 pixels and adds up to around 1 megapixel. "
+      ],
+      "color": "#432",
+      "bgcolor": "#653"
+    },
+    {
+      "id": 135,
+      "type": "EmptySD3LatentImage",
+      "pos": {
+        "0": -1008,
+        "1": -252
+      },
+      "size": {
+        "0": 315,
+        "1": 106
+      },
+      "flags": {},
+      "order": 1,
+      "mode": 0,
+      "inputs": [],
+      "outputs": [
+        {
+          "name": "LATENT",
+          "type": "LATENT",
+          "links": [
+            598
+          ],
+          "slot_index": 0,
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "EmptySD3LatentImage"
+      },
+      "widgets_values": [
+        1280,
+        768,
+        1
+      ]
+    },
+    {
+      "id": 294,
+      "type": "KSampler",
+      "pos": {
+        "0": 882,
+        "1": -504
+      },
+      "size": {
+        "0": 378,
+        "1": 504
+      },
+      "flags": {},
+      "order": 15,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "model",
+          "type": "MODEL",
+          "link": 568
+        },
+        {
+          "name": "positive",
+          "type": "CONDITIONING",
+          "link": 569
+        },
+        {
+          "name": "negative",
+          "type": "CONDITIONING",
+          "link": 604
+        },
+        {
+          "name": "latent_image",
+          "type": "LATENT",
+          "link": 598
+        }
+      ],
+      "outputs": [
+        {
+          "name": "LATENT",
+          "type": "LATENT",
+          "links": [
+            572
+          ],
+          "slot_index": 0,
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "KSampler"
+      },
+      "widgets_values": [
+        13910517299447,
+        "randomize",
+        40,
+        5.5,
+        "dpmpp_2m",
+        "sgm_uniform",
+        1
+      ]
+    },
+    {
+      "id": 13,
+      "type": "ModelSamplingSD3",
+      "pos": {
+        "0": -192,
+        "1": -507
+      },
+      "size": {
+        "0": 315,
+        "1": 58
+      },
+      "flags": {
+        "collapsed": false
+      },
+      "order": 6,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "model",
+          "type": "MODEL",
+          "link": 445
+        }
+      ],
+      "outputs": [
+        {
+          "name": "MODEL",
+          "type": "MODEL",
+          "links": [
+            568
+          ],
+          "slot_index": 0,
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "ModelSamplingSD3"
+      },
+      "widgets_values": [
+        3
+      ]
+    },
+    {
+      "id": 4,
+      "type": "CheckpointLoaderSimple",
+      "pos": {
+        "0": -2016,
+        "1": -504
+      },
+      "size": {
+        "0": 632.6060180664062,
+        "1": 98
+      },
+      "flags": {},
+      "order": 2,
+      "mode": 0,
+      "inputs": [],
+      "outputs": [
+        {
+          "name": "MODEL",
+          "type": "MODEL",
+          "links": [
+            445
+          ],
+          "slot_index": 0,
+          "shape": 3
+        },
+        {
+          "name": "CLIP",
+          "type": "CLIP",
+          "links": null,
+          "shape": 3
+        },
+        {
+          "name": "VAE",
+          "type": "VAE",
+          "links": [
+            605
+          ],
+          "slot_index": 2,
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "CheckpointLoaderSimple"
+      },
+      "widgets_values": [
+        "sd3.5_medium.safetensors"
+      ]
+    },
+    {
+      "id": 11,
+      "type": "TripleCLIPLoader",
+      "pos": {
+        "0": -2016,
+        "1": -252
+      },
+      "size": {
+        "0": 315,
+        "1": 106
+      },
+      "flags": {},
+      "order": 3,
+      "mode": 0,
+      "inputs": [],
+      "outputs": [
+        {
+          "name": "CLIP",
+          "type": "CLIP",
+          "links": [
+            5,
+            94
+          ],
+          "slot_index": 0,
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "TripleCLIPLoader"
+      },
+      "widgets_values": [
+        "clip_g.safetensors",
+        "clip_l.safetensors",
+        "t5xxl_fp16.safetensors"
+      ]
+    },
+    {
+      "id": 6,
+      "type": "CLIPTextEncode",
+      "pos": {
+        "0": -1008,
+        "1": 2
+      },
+      "size": {
+        "0": 342.8335266113281,
+        "1": 177.20867919921875
+      },
+      "flags": {},
+      "order": 13,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "clip",
+          "type": "CLIP",
+          "link": 5
+        },
+        {
+          "name": "text",
+          "type": "STRING",
+          "link": 606,
+          "widget": {
+            "name": "text"
+          }
+        }
+      ],
+      "outputs": [
+        {
+          "name": "CONDITIONING",
+          "type": "CONDITIONING",
+          "links": [
+            569
+          ],
+          "slot_index": 0,
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "CLIPTextEncode"
+      },
+      "widgets_values": [
+        "beautiful scenery nature glass bottle landscape, purple galaxy bottle,"
+      ],
+      "color": "#232",
+      "bgcolor": "#353"
+    },
+    {
+      "id": 304,
+      "type": "LoadImage",
+      "pos": {
+        "0": -2350,
+        "1": 300
+      },
+      "size": {
+        "0": 315,
+        "1": 314
+      },
+      "flags": {},
+      "order": 4,
+      "mode": 0,
+      "inputs": [],
+      "outputs": [
+        {
+          "name": "IMAGE",
+          "type": "IMAGE",
+          "links": [
+            609
+          ],
+          "slot_index": 0
+        },
+        {
+          "name": "MASK",
+          "type": "MASK",
+          "links": null
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LoadImage"
+      },
+      "widgets_values": [
+        "example.png",
+        "image"
+      ]
+    },
+    {
+      "id": 301,
+      "type": "LLMTextLatch",
+      "pos": {
+        "0": -1500,
+        "1": 150
+      },
+      "size": {
+        "0": 400,
+        "1": 200
+      },
+      "flags": {},
+      "order": 11,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "text_input",
+          "type": "STRING",
+          "link": 607,
+          "widget": {
+            "name": "text_input"
+          },
+          "shape": 7
+        }
+      ],
+      "outputs": [
+        {
+          "name": "text",
+          "type": "STRING",
+          "links": [
+            606
+          ],
+          "slot_index": 0
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LLMTextLatch"
+      },
+      "widgets_values": [
+        "In the image, a person with blonde hair is captured in a moment of joy, their smile radiating warmth. They are standing against a backdrop of a serene landscape, composed of a rolling hill blanketed in lush green grass. The sky above them is a clear blue, dotted with fluffy white clouds, adding to the overall tranquility of the scene. The person's attire consists of a pink top and white pants, a color combination that stands out against the green and blue background. The image is simple yet vivid, with each element precisely located to create a harmonious composition.",
+        true,
+        ""
+      ]
+    },
+    {
+      "id": 303,
+      "type": "LLMProvider",
+      "pos": {
+        "0": -2350,
+        "1": 100
+      },
+      "size": {
+        "0": 318.2536315917969,
+        "1": 106
+      },
+      "flags": {},
+      "order": 5,
+      "mode": 0,
+      "inputs": [],
+      "outputs": [
+        {
+          "name": "llm_model",
+          "type": "LLMMODEL",
+          "links": [
+            608
+          ],
+          "slot_index": 0
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LLMProvider"
+      },
+      "widgets_values": [
+        "OpenRouter",
+        "openrouter/mistralai/pixtral-12b",
+        "models"
+      ]
+    },
+    {
+      "id": 302,
+      "type": "LLMChat",
+      "pos": {
+        "0": -1950,
+        "1": 150
+      },
+      "size": {
+        "0": 400,
+        "1": 214
+      },
+      "flags": {},
+      "order": 8,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "llm_model",
+          "type": "LLMMODEL",
+          "link": 608
+        },
+        {
+          "name": "llm_sampler",
+          "type": "LLMSAMPLER",
+          "link": null,
+          "shape": 7
+        },
+        {
+          "name": "image",
+          "type": "IMAGE",
+          "link": 609,
+          "shape": 7
+        }
+      ],
+      "outputs": [
+        {
+          "name": "completion",
+          "type": "STRING",
+          "links": [
+            607
+          ],
+          "slot_index": 0,
+          "shape": 6
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LLMChat"
+      },
+      "widgets_values": [
+        819376686150074,
+        "randomize",
+        "Describe the image in detail",
+        ""
+      ]
+    }
+  ],
+  "links": [
+    [
+      5,
+      11,
+      0,
+      6,
+      0,
+      "CLIP"
+    ],
+    [
+      90,
+      67,
+      0,
+      68,
+      0,
+      "CONDITIONING"
+    ],
+    [
+      91,
+      68,
+      0,
+      69,
+      0,
+      "CONDITIONING"
+    ],
+    [
+      92,
+      70,
+      0,
+      69,
+      1,
+      "CONDITIONING"
+    ],
+    [
+      93,
+      71,
+      0,
+      70,
+      0,
+      "CONDITIONING"
+    ],
+    [
+      94,
+      11,
+      0,
+      71,
+      0,
+      "CLIP"
+    ],
+    [
+      274,
+      8,
+      0,
+      50,
+      0,
+      "IMAGE"
+    ],
+    [
+      445,
+      4,
+      0,
+      13,
+      0,
+      "MODEL"
+    ],
+    [
+      568,
+      13,
+      0,
+      294,
+      0,
+      "MODEL"
+    ],
+    [
+      569,
+      6,
+      0,
+      294,
+      1,
+      "CONDITIONING"
+    ],
+    [
+      572,
+      294,
+      0,
+      8,
+      0,
+      "LATENT"
+    ],
+    [
+      597,
+      71,
+      0,
+      67,
+      0,
+      "CONDITIONING"
+    ],
+    [
+      598,
+      135,
+      0,
+      294,
+      3,
+      "LATENT"
+    ],
+    [
+      604,
+      69,
+      0,
+      294,
+      2,
+      "CONDITIONING"
+    ],
+    [
+      605,
+      4,
+      2,
+      8,
+      1,
+      "VAE"
+    ],
+    [
+      606,
+      301,
+      0,
+      6,
+      1,
+      "STRING"
+    ],
+    [
+      607,
+      302,
+      0,
+      301,
+      0,
+      "STRING"
+    ],
+    [
+      608,
+      303,
+      0,
+      302,
+      0,
+      "LLMMODEL"
+    ],
+    [
+      609,
+      304,
+      0,
+      302,
+      2,
+      "IMAGE"
+    ]
+  ],
+  "groups": [],
+  "config": {},
+  "extra": {
+    "ds": {
+      "scale": 0.8264462809917354,
+      "offset": [
+        2080.858098000003,
+        266.35047800000154
+      ]
+    }
+  },
+  "version": 0.4
+}

examples/prompt-enhancer.json

@@ -0,0 +1,259 @@
+{
+  "last_node_id": 5,
+  "last_link_id": 4,
+  "nodes": [
+    {
+      "id": 3,
+      "type": "LLMTemperature",
+      "pos": {
+        "0": 200,
+        "1": 450
+      },
+      "size": {
+        "0": 315,
+        "1": 58
+      },
+      "flags": {},
+      "order": 2,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "previous",
+          "type": "LLMSAMPLER",
+          "link": 1,
+          "shape": 7
+        }
+      ],
+      "outputs": [
+        {
+          "name": "llm_sampler",
+          "type": "LLMSAMPLER",
+          "links": [
+            2
+          ],
+          "slot_index": 0
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LLMTemperature"
+      },
+      "widgets_values": [
+        0.8
+      ]
+    },
+    {
+      "id": 4,
+      "type": "LLMMinP",
+      "pos": {
+        "0": 200,
+        "1": 550
+      },
+      "size": {
+        "0": 315,
+        "1": 58
+      },
+      "flags": {},
+      "order": 0,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "previous",
+          "type": "LLMSAMPLER",
+          "link": null,
+          "shape": 7
+        }
+      ],
+      "outputs": [
+        {
+          "name": "llm_sampler",
+          "type": "LLMSAMPLER",
+          "links": [
+            1
+          ],
+          "slot_index": 0
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LLMMinP"
+      },
+      "widgets_values": [
+        0.05
+      ]
+    },
+    {
+      "id": 5,
+      "type": "LLMTextLatch",
+      "pos": {
+        "0": 1000,
+        "1": 250
+      },
+      "size": {
+        "0": 400,
+        "1": 200
+      },
+      "flags": {},
+      "order": 4,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "text_input",
+          "type": "STRING",
+          "link": 4,
+          "widget": {
+            "name": "text_input"
+          },
+          "shape": 7
+        }
+      ],
+      "outputs": [
+        {
+          "name": "text",
+          "type": "STRING",
+          "links": null
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LLMTextLatch"
+      },
+      "widgets_values": [
+        "\"A surreal, dreamlike landscape unfolds with a striking juxtaposition of natural and manufactured elements. A serene, untouched landscape stretches out to the horizon, with rolling hills, towering trees, and a shimmering lake in the distance. However, in the foreground, a beautiful, delicate glass bottle rises organically from the earth, as if grown from the very land itself. The bottle's curves and contours blend seamlessly with the surrounding scenery, giving it an otherworldly, almost ethereal quality. But what truly sets this scene apart is the bottle's peculiar contents: a swirling, purple galaxy of colors, reminiscent of a distant nebula, fills the bottle, casting an enchanting, iridescent glow across the surrounding landscape. The soft, golden light of dawn or dusk casts a warm, gentle ambiance over the entire scene, imbuing the image with a sense of wonder, magic, and a deep connection to the mysteries of the universe.\"",
+        true,
+        ""
+      ]
+    },
+    {
+      "id": 1,
+      "type": "LLMProvider",
+      "pos": {
+        "0": 200,
+        "1": 250
+      },
+      "size": {
+        "0": 301,
+        "1": 106
+      },
+      "flags": {},
+      "order": 1,
+      "mode": 0,
+      "inputs": [],
+      "outputs": [
+        {
+          "name": "llm_model",
+          "type": "LLMMODEL",
+          "links": [
+            3
+          ],
+          "slot_index": 0
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LLMProvider"
+      },
+      "widgets_values": [
+        "[LOCAL] llama.cpp",
+        "Meta-Llama-3.1-8B-Instruct",
+        "models"
+      ]
+    },
+    {
+      "id": 2,
+      "type": "LLMChat",
+      "pos": {
+        "0": 550,
+        "1": 250
+      },
+      "size": {
+        "0": 400,
+        "1": 350
+      },
+      "flags": {},
+      "order": 3,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "llm_model",
+          "type": "LLMMODEL",
+          "link": 3
+        },
+        {
+          "name": "llm_sampler",
+          "type": "LLMSAMPLER",
+          "link": 2,
+          "shape": 7
+        },
+        {
+          "name": "image",
+          "type": "IMAGE",
+          "link": null,
+          "shape": 7
+        }
+      ],
+      "outputs": [
+        {
+          "name": "completion",
+          "type": "STRING",
+          "links": [
+            4
+          ],
+          "slot_index": 0,
+          "shape": 6
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LLMChat"
+      },
+      "widgets_values": [
+        475392180035454,
+        "randomize",
+        "beautiful scenery nature glass bottle landscape, purple galaxy bottle,",
+        "You are a text-to-image-prompt generator. You take simple user input and translate it into a detailed and eloquent simple natural language prompt meant for generating images.\n\nEXAMPLES:\nUser Input: \"young woman taking a selfie in a mirror with old iPhone\"\n\nOutput: \"Phone photo: A young woman stands in front of a mirror, capturing a selfie. The image quality is grainy, with a slight blur softening the details. The lighting is dim, casting shadows that obscure her features. The room is cluttered, with clothes strewn across the bed and an unmade blanket. Her expression is casual, full of concentration, while the old iPhone struggles to focus, giving the photo an authentic, unpolished feel. The mirror shows smudges and fingerprints, adding to the raw, everyday atmosphere of the scene.\"\n\nUser Input: \"1940s film noir jazz bar scene glamorous woman singing\"\n\nOutput: \"A black-and-white, grainy cine still from a 1940s noir film scene of a dimly lit jazz club. In the center of the frame, a glamorous femme fatale stands on a small stage, singing into a vintage microphone. She is wearing a sequined evening gown that catches the soft, moody spotlight, and her hair is styled in classic Hollywood waves. Her expression is sultry and captivating, her lips painted dark as she croons a melancholic tune. The room is filled with a thick haze of cigarette smoke, and the silhouettes of patrons sit at small, round tables, their faces partially obscured by shadows. The band behind her, blurred slightly to give depth to the scene. The grainy texture and high contrast lighting create dramatic shadows and highlights, enhancing the mysterious and seductive atmosphere typical of a noir film.\"\n\nUser Input: \"fox in a raincoat in a rainy moody forest\"\n\nOutput: \"A solitary fox stands in the middle of a deserted, rain-soaked forest clearing, wearing a bright yellow raincoat that contrasts with the dark, muted colors of the scene. Raindrops fall steadily, creating ripples in the puddles that surround the fox, and the damp ground is covered with fallen leaves, now heavy with rain. In the background, the trees loom tall and bare, their branches reaching out like skeletal fingers, adding to the somber mood. The gray sky is heavy with clouds, casting a shadowy gloom over the entire scene, with only the fox’s yellow raincoat providing a small but poignant splash of color in the otherwise bleak environment.\"\n\nOUTPUT ONLY THE GENERATED PROMPT.\n"
+      ]
+    }
+  ],
+  "links": [
+    [
+      1,
+      4,
+      0,
+      3,
+      0,
+      "LLMSAMPLER"
+    ],
+    [
+      2,
+      3,
+      0,
+      2,
+      1,
+      "LLMSAMPLER"
+    ],
+    [
+      3,
+      1,
+      0,
+      2,
+      0,
+      "LLMMODEL"
+    ],
+    [
+      4,
+      2,
+      0,
+      5,
+      0,
+      "STRING"
+    ]
+  ],
+  "groups": [],
+  "config": {},
+  "extra": {
+    "ds": {
+      "scale": 0.9090909090909091,
+      "offset": [
+        -148.99999999999997,
+        8.4
+      ]
+    }
+  },
+  "version": 0.4
+}

examples/prompt-optimizer.json

@@ -0,0 +1,277 @@
+{
+  "last_node_id": 6,
+  "last_link_id": 7,
+  "nodes": [
+    {
+      "id": 5,
+      "type": "LLMTextLatch",
+      "pos": {
+        "0": 900,
+        "1": 150
+      },
+      "size": {
+        "0": 400,
+        "1": 200
+      },
+      "flags": {},
+      "order": 4,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "text_input",
+          "type": "STRING",
+          "link": 4,
+          "widget": {
+            "name": "text_input"
+          },
+          "shape": 7
+        }
+      ],
+      "outputs": [
+        {
+          "name": "text",
+          "type": "STRING",
+          "links": null
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LLMTextLatch"
+      },
+      "widgets_values": [
+        "\"A serene landscape of rolling hills, lush greenery, and distant horizons unfolds, set against a soft, golden light. An antique glass bottle lies abandoned on a rocky outcropping, its transparent walls glowing with an ethereal light that fills it with a deep, rich purple liquid, reminiscent of a distant galaxy with swirling clouds of gas and dust. Wispy clouds drift lazily across a blue sky, adding depth and atmosphere to this breathtaking view, evoking a sense of peaceful contemplation and discovery of a hidden world.\"",
+        true,
+        ""
+      ]
+    },
+    {
+      "id": 1,
+      "type": "LLMProvider",
+      "pos": {
+        "0": 100,
+        "1": 150
+      },
+      "size": {
+        "0": 300,
+        "1": 106
+      },
+      "flags": {},
+      "order": 0,
+      "mode": 0,
+      "inputs": [],
+      "outputs": [
+        {
+          "name": "llm_model",
+          "type": "LLMMODEL",
+          "links": [
+            1
+          ],
+          "slot_index": 0
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LLMProvider"
+      },
+      "widgets_values": [
+        "[LOCAL] llama.cpp",
+        "Meta-Llama-3.1-8B-Instruct",
+        "models"
+      ]
+    },
+    {
+      "id": 3,
+      "type": "LLMChat",
+      "pos": {
+        "0": 450,
+        "1": 150
+      },
+      "size": {
+        "0": 400,
+        "1": 214
+      },
+      "flags": {},
+      "order": 3,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "llm_model",
+          "type": "LLMMODEL",
+          "link": 1
+        },
+        {
+          "name": "llm_sampler",
+          "type": "LLMSAMPLER",
+          "link": null,
+          "shape": 7
+        },
+        {
+          "name": "image",
+          "type": "IMAGE",
+          "link": null,
+          "shape": 7
+        },
+        {
+          "name": "system_prompt",
+          "type": "STRING",
+          "link": 7,
+          "widget": {
+            "name": "system_prompt"
+          },
+          "shape": 7
+        }
+      ],
+      "outputs": [
+        {
+          "name": "completion",
+          "type": "STRING",
+          "links": [
+            4
+          ],
+          "slot_index": 0,
+          "shape": 6
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LLMChat"
+      },
+      "widgets_values": [
+        1039007309412739,
+        "randomize",
+        "Optimize the given text-to-image prompt for word count. Try to keep as many descriptive details as possible.\n\nOUTPUT ONLY THE MODIFIED PROMPT.",
+        ""
+      ]
+    },
+    {
+      "id": 2,
+      "type": "LLMTextLatch",
+      "pos": {
+        "0": -450,
+        "1": 350
+      },
+      "size": {
+        "0": 400,
+        "1": 200
+      },
+      "flags": {},
+      "order": 1,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "text_input",
+          "type": "STRING",
+          "link": null,
+          "widget": {
+            "name": "text_input"
+          },
+          "shape": 7
+        }
+      ],
+      "outputs": [
+        {
+          "name": "text",
+          "type": "STRING",
+          "links": [
+            6
+          ],
+          "slot_index": 0
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LLMTextLatch"
+      },
+      "widgets_values": [
+        "\"A beautiful, serene landscape unfolds before us, with rolling hills and lush greenery stretching as far as the eye can see. A delicate, antique glass bottle lies abandoned on a rocky outcropping, its transparent walls glowing with an ethereal light that seems almost otherworldly. The bottle appears to be filled with a deep, rich purple liquid that perfectly captures the color of a distant galaxy, with swirling clouds of gas and dust visible within. The surrounding landscape is bathed in a soft, golden light, with the warm rays of the sun casting a gentle glow across the scene. A few wispy clouds drift lazily across the sky, adding a sense of depth and atmosphere to the already breathtaking view. The overall mood is one of peaceful contemplation, as if the viewer has stumbled upon a secret, hidden world that exists just beyond the edge of reality.\"",
+        true,
+        ""
+      ]
+    },
+    {
+      "id": 6,
+      "type": "LLMPrependAppend",
+      "pos": {
+        "0": 0,
+        "1": 350
+      },
+      "size": [
+        400,
+        200
+      ],
+      "flags": {},
+      "order": 2,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "text_input",
+          "type": "STRING",
+          "link": 6,
+          "widget": {
+            "name": "text_input"
+          }
+        }
+      ],
+      "outputs": [
+        {
+          "name": "text",
+          "type": "STRING",
+          "links": [
+            7
+          ],
+          "slot_index": 0
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LLMPrependAppend"
+      },
+      "widgets_values": [
+        "",
+        "Use the following text as reference:\n",
+        ""
+      ]
+    }
+  ],
+  "links": [
+    [
+      1,
+      1,
+      0,
+      3,
+      0,
+      "LLMMODEL"
+    ],
+    [
+      4,
+      3,
+      0,
+      5,
+      0,
+      "STRING"
+    ],
+    [
+      6,
+      2,
+      0,
+      6,
+      0,
+      "STRING"
+    ],
+    [
+      7,
+      6,
+      0,
+      3,
+      3,
+      "STRING"
+    ]
+  ],
+  "groups": [],
+  "config": {},
+  "extra": {
+    "ds": {
+      "scale": 0.7513148009015777,
+      "offset": [
+        320.79927272727275,
+        220.6500000000004
+      ]
+    }
+  },
+  "version": 0.4
+}

js/yallm-node.js

@@ -0,0 +1,86 @@
+import { app } from "../../scripts/app.js";
+import { api } from "../../scripts/api.js";
+app.registerExtension({
+	name: "YALLM.node",
+	async beforeRegisterNodeDef(nodeType, nodeData, app) {
+		if (nodeType.comfyClass === "LLMProvider") {
+			nodeType.prototype.myRefreshModels = function (name) {
+				const provWidget = this.widgets.find(
+					(widget) => widget.name === "provider",
+				);
+				const modelWidget = this.widgets.find(
+					(widget) => widget.name === "model",
+				);
+				const fetchWidget = this.widgets.find(
+					(widget) => widget.name === "fetch models",
+				);
+
+				const provName = name ?? provWidget.value;
+
+				const myDisableWidgets = (state) => {
+					provWidget.disabled = state;
+					modelWidget.disabled = state;
+					// Note: This is the "fetch models" button we add down below
+					fetchWidget.disabled = state;
+				};
+				myDisableWidgets(true);
+
+				api.fetchApi(`/llm_models?name=${provName}`).then((resp) => {
+					resp.json().then((data) => {
+						const widget = modelWidget;
+						widget.options.values = data;
+
+						if (
+							!widget.options.values.includes(widget.value) &&
+							widget.options.values.length > 0
+						) {
+							widget.value = widget.options.values[0];
+						}
+
+						myDisableWidgets(false);
+					});
+				});
+			};
+		}
+	},
+	async nodeCreated(node) {
+		if (node?.comfyClass === "LLMProvider") {
+			const provWidget = node.widgets.find(
+				(widget) => widget.name === "provider",
+			);
+			const modelWidget = node.widgets.find(
+				(widget) => widget.name === "model",
+			);
+
+			// Whenever provider changes value, fetch models
+			const original_callback = provWidget.callback;
+			provWidget.callback = function (...args) {
+				const name = args?.[0];
+				node.myRefreshModels(name);
+				return original_callback?.apply(this, args);
+			};
+
+			// Disable model combo widget until we fetch models
+			modelWidget.disabled = true;
+
+			// Add button to fetch models manually
+			const btn = node.addWidget("button", "fetch models", "models", () => {
+				node.myRefreshModels();
+			});
+			btn.serializeValue = () => void 0;
+		}
+	},
+	async setup() {
+		api.addEventListener("executed", (event) => {
+			const node = app.graph.getNodeById(event.detail.node);
+			if (node?.comfyClass === "LLMTextLatch") {
+				const textWidget = node.widgets.find(
+					(widget) => widget.name === "text",
+				);
+				const texts = event.detail.output.text;
+				textWidget.value = texts[texts.length - 1];
+				app.graph.setDirtyCanvas(true, false);
+			}
+		});
+	},
+});

models.yaml.example

@@ -6,11 +6,21 @@ models:
     api_key: none
     model: whatever
 
+  - name: "[LOCAL] LM Studio"
+    base_url: http://localhost:1234/v1
+    api_key: none
+    model: changeme
+
+  - name: "[LOCAL] Ollama"
+    base_url: http://localhost:11434/v1
+    api_key: none
+    model: changeme
+
 # Example of using a remote endpoint
-# If prefixed with "os-environ/" (similar to LiteLLM), base_url and/or api_key
+# If prefixed with "os.environ/" (similar to LiteLLM), base_url and/or api_key
 # will take their value from the named environment variable.
 
 #  - name: "[OpenRouter] Llama 3.1 8B Instruct"
 #    base_url: https://openrouter.ai/api/v1
-#    api_key: os-environ/OPENROUTER_API_KEY
+#    api_key: os.environ/OPENROUTER_API_KEY
 #    model: meta-llama/llama-3.1-8b-instruct:free

nodes.py

@@ -1,53 +1,41 @@
 # Copyright (c) 2024 Allan Saddi <allan@saddi.com>
+import base64
+from io import BytesIO
 import os
 from typing import Any, Optional
 
-import openai
+from aiohttp import web
+from aiohttp.web_request import Request
+from PIL.Image import Image
 from pydantic import BaseModel
+import requests
+import torchvision.transforms.functional as F
 import yaml
 
+from .utils import PromptUtils, WorkflowUtils
+
+from server import PromptServer
+
 
 BASE_PATH = os.path.dirname(os.path.realpath(__file__))
 
 
-class ModelDefinition(BaseModel):
+class ProviderDefinition(BaseModel):
     name: str
     base_url: str
     api_key: Optional[str]
-    model: str
 
+    def resolve_base_url(self) -> str:
+        return ProviderDefinition._get_env_or_value(self.base_url)
 
-class Models:
-    LIST: list[ModelDefinition] = []
-    BY_NAME: dict[str,ModelDefinition] = {}
-    CHOICES: list[str] = []
-
-    def load(self) -> None:
-        with open(os.path.join(BASE_PATH, 'models.yaml')) as inp:
-            d = yaml.load(inp, yaml.Loader)
-
-        for value in d['models']:
-            self.LIST.append(ModelDefinition.model_validate(value))
-        if not self.LIST:
-            raise RuntimeError('Need at least one model defined')
-        self.BY_NAME = { d.name: d for d in self.LIST }
-        self.CHOICES = [ d.name for d in self.LIST ]
-
-    def get_base_url(self, name: str) -> str:
-        base_url = self.BY_NAME[name].base_url
-        return Models._get_env_or_value(base_url)
-
-    def get_api_key(self, name: str) -> str | None:
-        api_key = self.BY_NAME[name].api_key
+    def resolve_api_key(self) -> str | None:
+        api_key = self.api_key
         if api_key:
-            api_key = Models._get_env_or_value(api_key)
-        if not api_key or api_key == 'none':
+            api_key = ProviderDefinition._get_env_or_value(api_key)
+        if not api_key or api_key == "none":
             return None
         return api_key
 
-    def get_model(self, name: str) -> str:
-        return self.BY_NAME[name].model
-
     @staticmethod
     def _get_env_or_value(value: str) -> str:
         """
@@ -60,39 +48,224 @@ class Models:
         Returns:
             str: The retrieved value.
         """
-        PREFIX = 'os.environ/'
+        PREFIX = "os.environ/"
         if value.startswith(PREFIX):
-            env_var = value[len(PREFIX):]
+            env_var = value[len(PREFIX) :]
             value = os.environ[env_var]
         return value
 
 
+class ModelDefinition(ProviderDefinition):
+    model: str
+
+
+class DefinitionsConfig:
+    _mtime = None
+
+    def load(self, def_type) -> None:
+        models_file = self._get_models_file()
+
+        with open(models_file) as inp:
+            d = yaml.load(inp, yaml.Loader)
+        self._mtime = (models_file, os.path.getmtime(models_file))
+
+        self.LIST = []
+        for value in d[self._root]:
+            self.LIST.append(def_type.model_validate(value))
+        if not self.LIST:
+            raise RuntimeError(f"{models_file}: Need at least one definition")
+        self.BY_NAME = {d.name: d for d in self.LIST}
+        self.CHOICES = [d.name for d in self.LIST]
+
+    def _get_models_file(self):
+        models_file = os.path.join(BASE_PATH, self._config)
+        if not os.path.exists(models_file):
+            # Just read from the example for now
+            # print(f'{models_file} does not exist; using default')
+            models_file = os.path.join(BASE_PATH, self._config_default)
+        return models_file
+
+    def refresh(self):
+        models_file = self._get_models_file()
+        if self._mtime != (models_file, os.path.getmtime(models_file)):
+            self.load()
+
+
+class Providers(DefinitionsConfig):
+    _root = "providers"
+    _config = "providers.yaml"
+    _config_default = "providers.yaml.example"
+
+    def load(self):
+        return super().load(ProviderDefinition)
+
+
+class Models(DefinitionsConfig):
+    _root = "models"
+    _config = "models.yaml"
+    _config_default = "models.yaml.example"
+
+    def load(self):
+        return super().load(ModelDefinition)
+
+
+PROVIDERS = Providers()
+PROVIDERS.load()
+
 MODELS = Models()
 MODELS.load()
 
 
+class LLMPrependAppend:
+    @classmethod
+    def INPUT_TYPES(cls):
+        return {
+            "required": {
+                "text_input": (
+                    "STRING",
+                    {
+                        "forceInput": True,
+                        "multiline": True,
+                    },
+                ),
+            },
+            "optional": {
+                "prefix": (
+                    "STRING",
+                    {
+                        "multiline": True,
+                    },
+                ),
+                "suffix": (
+                    "STRING",
+                    {
+                        "multiline": True,
+                    },
+                ),
+            },
+        }
+
+    TITLE = "Prepend/Append"
+
+    RETURN_TYPES = ("STRING",)
+    RETURN_NAMES = ("text",)
+
+    FUNCTION = "execute"
+
+    CATEGORY = "YALLM"
+
+    def execute(self, text_input, prefix=None, suffix=None):
+        if prefix is None:
+            prefix = ""
+        if suffix is None:
+            suffix = ""
+
+        text = prefix + text_input + suffix
+
+        return (text,)
+
+
+class LLMTextLatch:
+    @classmethod
+    def INPUT_TYPES(cls):
+        return {
+            "required": {
+                "text": (
+                    "STRING",
+                    {
+                        "multiline": True,
+                    },
+                ),
+                "replace": (
+                    "BOOLEAN",
+                    {
+                        "default": True,
+                    },
+                ),
+            },
+            "optional": {
+                "text_input": (
+                    "STRING",
+                    {
+                        "forceInput": True,
+                        "multiline": True,
+                        "lazy": True,
+                    },
+                ),
+            },
+            "hidden": {
+                "unique_id": "UNIQUE_ID",
+                "prompt": "PROMPT",
+                "extra_pnginfo": "EXTRA_PNGINFO",
+            },
+        }
+
+    @classmethod
+    def VALIDATE_INPUTS(cls, text):
+        # Always valid, even if empty
+        # TODO I don't think this is necessary
+        return True
+
+    TITLE = "Text Latch"
+
+    RETURN_TYPES = ("STRING",)
+    RETURN_NAMES = ("text",)
+
+    FUNCTION = "execute"
+
+    OUTPUT_NODE = True
+
+    CATEGORY = "YALLM"
+
+    def check_lazy_status(
+        self, text, replace, unique_id, prompt, extra_pnginfo, text_input=None
+    ):
+        # Require whatever is connected to text_input to be evaluated
+        # contingent on replace.
+        if replace and text_input is None:
+            # Do we have something connected to text_input?
+            pu = PromptUtils(prompt)
+            if pu.is_input_connected(unique_id, "text_input"):
+                return ["text_input"]
+        # All good, nothing else needs to be evaluated
+        return []
+
+    def execute(self, text, replace, unique_id, prompt, extra_pnginfo, text_input=None):
+        if text_input is not None and replace:
+            text = text_input
+
+        if text is None:
+            text = ""
+
+        # Save the current value in the outgoing metadata
+        wfu = WorkflowUtils(extra_pnginfo)
+        wfu.set_widget(unique_id, 0, text)
+
+        # An oddity: the values of the UI dictionary have to be wrapped in
+        # a tuple/list, just like the result?
+        return {"ui": {"text": (text,)}, "result": (text,)}
+
+
 class SamplerDefinition(BaseModel):
     name: str
     is_int: bool
-    min: Optional[float|int] = None
-    default: float|int
-    max: Optional[float|int] = None
+    min: Optional[float | int] = None
+    default: float | int
+    max: Optional[float | int] = None
 
     def to_input_type(self):
         constraints = {}
         if self.min is not None:
-            constraints['min'] = self.min
+            constraints["min"] = self.min
         if self.max is not None:
-            constraints['max'] = self.max
-        constraints['default'] = self.default
-        return {
-            self.name: ('INT' if self.is_int else 'FLOAT', constraints)
-        }
+            constraints["max"] = self.max
+        constraints["default"] = self.default
+        return {self.name: ("INT" if self.is_int else "FLOAT", constraints)}
 
 
 class LLMTemperature:
     _DEF = SamplerDefinition(
-        name='temperature',
+        name="temperature",
         is_int=False,
         min=0.0,
         default=0.8,
@@ -101,20 +274,20 @@ class LLMTemperature:
     @classmethod
     def INPUT_TYPES(cls):
         return {
-            'required': cls._DEF.to_input_type(),
-            'optional': {
-                'previous': ('LLMSAMPLER',),
+            "required": cls._DEF.to_input_type(),
+            "optional": {
+                "previous": ("LLMSAMPLER",),
             },
         }
 
-    TITLE = 'LLM Temperature'
+    TITLE = "LLM Temperature"
 
-    RETURN_TYPES = ('LLMSAMPLER',)
-    RETURN_NAMES = ('llm_sampler',)
+    RETURN_TYPES = ("LLMSAMPLER",)
+    RETURN_NAMES = ("llm_sampler",)
 
-    FUNCTION = 'execute'
+    FUNCTION = "execute"
 
-    CATEGORY = 'YALLM/samplers'
+    CATEGORY = "YALLM/samplers"
 
     def execute(self, previous=None, **args):
         value = args[self._DEF.name]
@@ -129,111 +302,204 @@ class LLMTemperature:
 
 class LLMTopP(LLMTemperature):
     _DEF = SamplerDefinition(
-        name='top_p',
+        name="top_p",
         is_int=False,
         min=0.0,
         default=0.95,
         max=1.0,
     )
 
-    TITLE = 'LLM Top-P'
+    TITLE = "LLM Top-P"
 
 
 class LLMTopK(LLMTemperature):
     _DEF = SamplerDefinition(
-        name='top_k',
+        name="top_k",
         is_int=True,
         min=1,
         default=40,
     )
 
-    TITLE = 'LLM Top-K'
+    TITLE = "LLM Top-K"
 
 
 class LLMMinP(LLMTemperature):
     _DEF = SamplerDefinition(
-        name='min_p',
+        name="min_p",
         is_int=False,
         min=0.0,
         default=0.05,
         max=1.0,
     )
 
-    TITLE = 'LLM Min-P'
+    TITLE = "LLM Min-P"
 
 
 # TODO typical_p? tfs_z? Any others? mirostat?
 
 
 # TODO Will we need a more complex definition? (e.g. multi-modal)
-ChatMessage = dict[str,str]
+ChatMessage = dict[str, str]
 ChatHistory = list[ChatMessage]
 
-SamplerSetting = tuple[str,Any]
+SamplerSetting = tuple[str, Any]
 
 
 class LLMModel:
-    def __init__(self, model: str):
-        self._llm = openai.OpenAI(
-            base_url=MODELS.get_base_url(model),
-            api_key=(MODELS.get_api_key(model) or 'none'), # openai package requires something for API key
-        )
-        self._model = MODELS.get_model(model)
+    def __init__(self, prov_def: ProviderDefinition, model: str | None = None):
+        self._base_url = prov_def.resolve_base_url()
+        self._headers = {}
+        if (api_key := prov_def.resolve_api_key()) and api_key != "none":
+            self._headers["authorization"] = f"Bearer {api_key}"
+        if model is None:
+            self._model = getattr(prov_def, "model", None)
+        else:
+            self._model = model
+
+    def get_models(self):
+        with requests.get(self._base_url + "/models", headers=self._headers) as r:
+            models_json = r.json()
+        return [m["id"] for m in models_json["data"]]
+
+    def chat_completion(
+        self,
+        messages: ChatHistory,
+        image: Image | None = None,
+        samplers: list[SamplerSetting] | None = None,
+        seed: int | None = None,
+    ) -> str:
+        assert self._model is not None
+
+        if image is not None:
+            # Work backwards through history
+            for msg in reversed(messages):
+                # And grab the very last user message we see
+                if msg.get("role") == "user":
+                    parts = msg.get("content", [])
+                    if isinstance(parts, str):
+                        parts = [{"type": "text", "text": parts}]
+
+                    # Convert image to data URI format
+                    png_io = BytesIO()
+                    image.save(png_io, "PNG")
+                    png_b64 = base64.b64encode(png_io.getbuffer()).decode("ascii")
+
+                    # And insert image just before the user prompt
+                    parts.insert(
+                        0,
+                        {
+                            "type": "image_url",
+                            "image_url": {"url": f"data:image/png;base64,{png_b64}"},
+                        },
+                    )
+
+                    msg["content"] = parts
+                    break
+
+        # print(messages)
 
-    def chat_completion(self, messages: ChatHistory, samplers: list[SamplerSetting]|None=None, seed: int|None=None) -> str:
         if samplers is None:
             samplers = []
 
         sampler_order = []
         extra_args = {}
-        extra_body = {}
-        for k,v in samplers:
+        for k, v in samplers:
             sampler_order.append(k)
-            if k in ('temperature', 'top_p'): # The only ones supported directly by openai package
-                extra_args[k] = v
-            else:
-                # The rest have to go into "extra_body"
-                extra_body[k] = v
-        extra_body['samplers'] = sampler_order # Hopefully won't cause an issue for non-llama.cpp endpoints
-
-        if seed is not None:
-            extra_args['seed'] = seed
-
-        # print(f'extra_args = {repr(extra_args)}')
-        # print(f'extra_body = {repr(extra_body)}')
-
-        output = self._llm.chat.completions.create(
-            messages=messages,
-            model=self._model,
-            extra_body=extra_body,
-            **extra_args
+            extra_args[k] = v
+        extra_args["samplers"] = (
+            sampler_order  # Hopefully won't cause an issue for non-llama.cpp endpoints
         )
 
-        # TODO Since we're not streaming, we'll need some sort of timeout. How to specify that?
+        if seed is not None:
+            extra_args["seed"] = seed
 
-        return output.choices[0].message.content
+        # print(f"extra_args = {repr(extra_args)}")
+
+        req_json = {
+            "model": self._model,
+            "messages": messages,
+            "stream": False,
+        }
+        req_json.update(extra_args)
+
+        # TODO Since we're not streaming, we'll need some sort of timeout. TODO specify that
+        with requests.post(
+            self._base_url + "/chat/completions", json=req_json, headers=self._headers
+        ) as r:
+            output = r.json()
+
+        if choices := output.get("choices", []):
+            return choices[0]["message"]["content"]
+        else:
+            return ""
+
+
+class LLMProvider:
+    @classmethod
+    def INPUT_TYPES(cls):
+        PROVIDERS.refresh()
+
+        return {
+            "required": {
+                "provider": (PROVIDERS.CHOICES,),
+                "model": (["fetch.models.first"],),
+            },
+        }
+
+    @classmethod
+    def IS_CHANGED(cls, provider, model):
+        PROVIDERS.refresh()
+        return PROVIDERS._mtime
+
+    @classmethod
+    def VALIDATE_INPUTS(cls, model):
+        if model in ("fetch.models.first", "failed.to.fetch"):
+            return "fetch models and make a selection first"
+
+        # Ass-u-me the frontend validated the choices
+        return True
+
+    TITLE = "LLM Provider (API)"
+
+    RETURN_TYPES = ("LLMMODEL",)
+    RETURN_NAMES = ("llm_model",)
+
+    FUNCTION = "execute"
+
+    CATEGORY = "YALLM"
+
+    def execute(self, provider, model):
+        llm = LLMModel(PROVIDERS.BY_NAME[provider], model=model)
+
+        return (llm,)
 
 
 class LLMModelNode:
     @classmethod
     def INPUT_TYPES(cls):
+        MODELS.refresh()
         return {
-            'required': {
-                'model': (MODELS.CHOICES,),
+            "required": {
+                "model": (MODELS.CHOICES,),
             },
         }
 
-    TITLE = 'LLM Model (API)'
+    @classmethod
+    def IS_CHANGED(cls, model):
+        MODELS.refresh()
+        return MODELS._mtime
 
-    RETURN_TYPES = ('LLMMODEL',)
-    RETURN_NAMES = ('llm_model',)
+    TITLE = "LLM Model (API)"
 
-    FUNCTION = 'execute'
+    RETURN_TYPES = ("LLMMODEL",)
+    RETURN_NAMES = ("llm_model",)
 
-    CATEGORY = 'YALLM'
+    FUNCTION = "execute"
+
+    CATEGORY = "YALLM"
 
     def execute(self, model):
-        llm = LLMModel(model)
+        llm = LLMModel(MODELS.BY_NAME[model])
 
         return (llm,)
 
@@ -242,52 +508,108 @@ class LLMChat:
     @classmethod
     def INPUT_TYPES(cls):
         return {
-            'required': {
-                'llm_model': ('LLMMODEL',),
-                'seed': ('INT', { # Thankfully, frontend has special handling for a widget of this name...
-                    # min/max/default from the KSampler nodes
-                    # llama.cpp is only 32-bit (with truncation, no doubt), OpenAI API spec doesn't specify a size
-                    'min': 0,
-                    'default': 0,
-                    'max': 0xffffffff_ffffffff,
-                }),
-                'user_prompt': ('STRING', {
-                    'multiline': True,
-                }),
+            "required": {
+                "llm_model": ("LLMMODEL",),
+                "seed": (
+                    "INT",
+                    {  # Thankfully, frontend has special handling for a widget of this name...
+                        # min/max/default from the KSampler nodes
+                        # llama.cpp is only 32-bit (with truncation, no doubt), OpenAI API spec doesn't specify a size
+                        "min": 0,
+                        "default": 0,
+                        "max": 0xFFFFFFFF_FFFFFFFF,
+                    },
+                ),
+                "user_prompt": (
+                    "STRING",
+                    {
+                        "multiline": True,
+                    },
+                ),
             },
-            'optional': {
-                'llm_sampler': ('LLMSAMPLER', ),
-                'system_prompt': ('STRING', {
-                    'multiline': True,
-                }),
+            "optional": {
+                "llm_sampler": ("LLMSAMPLER",),
+                "system_prompt": (
+                    "STRING",
+                    {
+                        "multiline": True,
+                    },
+                ),
+                "image": ("IMAGE",),
             },
         }
 
-    TITLE = 'LLM Chat'
+    TITLE = "LLM Chat"
 
-    RETURN_TYPES = ('STRING',)
-    RETURN_NAMES = ('completion',)
+    RETURN_TYPES = ("STRING",)
+    RETURN_NAMES = ("completion",)
+    OUTPUT_IS_LIST = (True,)
 
-    FUNCTION = 'execute'
+    FUNCTION = "execute"
 
-    CATEGORY = 'YALLM'
+    CATEGORY = "YALLM"
 
-    def execute(self, llm_model: LLMModel, seed, user_prompt, llm_sampler=None, system_prompt=None):
+    def execute(
+        self,
+        llm_model: LLMModel,
+        seed,
+        user_prompt,
+        llm_sampler=None,
+        system_prompt=None,
+        image=None,
+    ):
         messages = []
         if system_prompt:
-            messages.append({'role': 'system', 'content': system_prompt})
-        messages.append({'role': 'user', 'content': user_prompt})
+            messages.append({"role": "system", "content": system_prompt})
+        messages.append({"role": "user", "content": user_prompt})
 
-        result = llm_model.chat_completion(messages, samplers=llm_sampler, seed=seed)
+        result = []
+        if image is None:
+            text = llm_model.chat_completion(messages, samplers=llm_sampler, seed=seed)
+            result.append(text)
+        else:
+            image = image.permute(0, 3, 1, 2)  # Convert to [B,C,H,W]
+            for img in image:
+                pil_image = F.to_pil_image(img)
+                # Fortunately or unfortunately, I'm forced to have this conform
+                # to the conventions of my LlamaVision node.
+                # I think it would be cleaner to do the conversion here, but
+                # I have no idea how that would work for a local
+                # Transformers model.
+                text = llm_model.chat_completion(
+                    messages, image=pil_image, samplers=llm_sampler, seed=seed
+                )
+                result.append(text)
 
         return (result,)
 
 
+@PromptServer.instance.routes.get("/llm_models")
+async def llm_models(request: Request):
+    name = request.rel_url.query.get("name", None)
+    if name:
+        PROVIDERS.refresh()
+        if (prov_def := PROVIDERS.BY_NAME.get(name, None)) is not None:
+            try:
+                llm = LLMModel(prov_def)
+                models = llm.get_models()
+                return web.json_response(models)
+            except Exception as e:
+                print(f"Problem fetching models: {e}")
+                # Just eat it and fall through.
+
+    # Fail, but don't error out
+    return web.json_response(["failed.to.fetch"])
+
+
 NODE_CLASS_MAPPINGS = {
-    'LLMModel': LLMModelNode,
-    'LLMChat': LLMChat,
-    'LLMTemperature': LLMTemperature,
-    'LLMTopP': LLMTopP,
-    'LLMTopK': LLMTopK,
-    'LLMMinP': LLMMinP,
+    "LLMPrependAppend": LLMPrependAppend,
+    "LLMTextLatch": LLMTextLatch,
+    "LLMProvider": LLMProvider,
+    "LLMModel": LLMModelNode,
+    "LLMChat": LLMChat,
+    "LLMTemperature": LLMTemperature,
+    "LLMTopP": LLMTopP,
+    "LLMTopK": LLMTopK,
+    "LLMMinP": LLMMinP,
 }

providers.yaml.example

@@ -0,0 +1,22 @@
+providers:
+  # Note: "name" is what this connection will show up as in the UI
+
+  - name: "[LOCAL] llama.cpp"
+    base_url: http://localhost:8080/v1
+    api_key: none
+
+  - name: "[LOCAL] LM Studio"
+    base_url: http://localhost:1234/v1
+    api_key: none
+
+  - name: "[LOCAL] Ollama"
+    base_url: http://localhost:11434/v1
+    api_key: none
+
+# Example of using a remote endpoint
+# If prefixed with "os.environ/" (similar to LiteLLM), base_url and/or api_key
+# will take their value from the named environment variable.
+
+#  - name: "OpenRouter"
+#    base_url: https://openrouter.ai/api/v1
+#    api_key: os.environ/OPENROUTER_API_KEY

pyproject.toml

@@ -1,9 +1,15 @@
 [project]
 name = "comfyui-yallm-node"
-description = "Yet another set of LLM nodes for ComfyUI. This one only supports OpenAI-like APIs, but of course can be used with local LLM providers such as llama.cpp, ollama and LMStudio, among others."
-version = "1.0.0"
+description = "Yet another set of LLM nodes for ComfyUI (for local/remote OpenAI-like APIs, multi-modal models supported)"
+version = "1.5.0"
 license = {file = "LICENSE.txt"}
-dependencies = ["openai"]
+dependencies = [
+    "pillow",
+    "pydantic",
+    "pyyaml",
+    "requests",
+    "torchvision"
+]
 
 [project.urls]
 Repository = "https://github.com/asaddi/ComfyUI-YALLM-node"
@@ -13,3 +19,27 @@ Repository = "https://github.com/asaddi/ComfyUI-YALLM-node"
 PublisherId = "asaddi"
 DisplayName = "ComfyUI-YALLM-node"
 Icon = ""
+
+[tool.bumpversion]
+current_version = "1.5.0"
+parse = "(?P<major>\\d+)\\.(?P<minor>\\d+)\\.(?P<patch>\\d+)"
+serialize = ["{major}.{minor}.{patch}"]
+search = "{current_version}"
+replace = "{new_version}"
+regex = false
+ignore_missing_version = false
+ignore_missing_files = false
+tag = true
+sign_tags = true
+tag_name = "v{new_version}"
+tag_message = "Bump version: {current_version} → {new_version}"
+allow_dirty = false
+commit = true
+message = "Bump version: {current_version} → {new_version}"
+commit_args = "-S"
+setup_hooks = []
+pre_commit_hooks = []
+post_commit_hooks = []
+
+[[tool.bumpversion.files]]
+filename = "pyproject.toml"

requirements.txt

@@ -1 +1,6 @@
-openai
+# These should already be installed by ComfyUI
+pillow
+pydantic
+pyyaml
+requests
+torchvision

utils.py

@@ -0,0 +1,105 @@
+# Copyright (c) 2024 Allan Saddi <allan@saddi.com>
+class PromptUtils:
+    def __init__(self, prompt):
+        self.prompt = prompt
+
+    def get_node_info(self, node_id: int | str) -> dict:
+        node_id = str(node_id)
+        return self.prompt[node_id]
+
+    def is_input_connected(self, node_id: int | str, name: str) -> bool:
+        node_info = self.get_node_info(node_id)
+        input = node_info.get("inputs", {}).get(name)
+        # In reality, input will be a 2-elem list of source node ID + slot
+        # if it is connected.
+        return input is not None
+
+    @property
+    def output_map(self):
+        if hasattr(self, "_output_map"):
+            return self._output_map
+        # A map of (src node ID, slot) -> (dest node ID, "input name")
+        self._output_map: dict[tuple[str, int], list[tuple[str, str]]] = {}
+        for dest_id, dest_info in self.prompt.items():
+            for input_name, input in dest_info.get("inputs", {}).items():
+                if isinstance(input, list) and len(input) == 2:
+                    # This is apparently a link
+                    src_id, src_slot = input
+                    out_key = (src_id, src_slot)
+                    outputs = self._output_map.get(out_key)
+                    if outputs is None:
+                        self._output_map[out_key] = outputs = []
+                    outputs.append((dest_id, input_name))
+        return self._output_map
+
+    def is_output_connected(self, node_id: int | str, slot: int) -> bool:
+        out_key = (str(node_id), slot)
+        return out_key in self.output_map
+
+    # The assumption is that the given output slot can only be connected
+    # to nodes of the same type as the one being checked.
+    def get_downstream_nodes(self, node_id: int | str, slot: int) -> list[str]:
+        downstream: list[str] = []
+
+        to_check = [str(node_id)]
+        while to_check:
+            check_id = to_check.pop(0)
+
+            # Yes, consider the starting node downstream from itself as well
+            downstream.append(check_id)
+
+            outputs = self.output_map.get((check_id, slot))
+            if outputs is not None:
+                # NB There can be multiple outputs
+                out_ids: list[str] = [link_info[0] for link_info in outputs]
+                to_check.extend(out_ids)
+
+        return downstream
+
+
+class WorkflowUtils:
+    def __init__(self, extra_pnginfo):
+        # Note: API mode does not have extra_pnginfo (it is None)
+        self.extra_pnginfo = extra_pnginfo
+
+    def get_node_info(self, node_id: str | int) -> dict | None:
+        # Be somewhat paranoid since API mode offers much less access
+        if (
+            self.extra_pnginfo is not None
+            and (workflow := self.extra_pnginfo.get("workflow")) is not None
+            and (nodes := workflow.get("nodes")) is not None
+        ):
+            node_id = str(node_id)
+            # TODO For the time being, we aren't caching anything.
+            # Concievably, if there are a large number of nodes, a lot of time
+            # could be wasted searching.
+            my_node = [node_info for node_info in nodes if node_info["id"] == node_id]
+            if my_node:
+                return my_node[0]
+        return None
+
+    def set_property(self, node_id: str | int, property_name: str, value):
+        if (my_node := self.get_node_info(node_id)) is not None:
+            my_node["properties"][property_name] = value
+
+    def set_widget(self, node_id: str | int, slot: int, value):
+        if (my_node := self.get_node_info(node_id)) is not None:
+            values = my_node.get("widgets_values", [])
+            # TODO When would there be a mismatch?
+            if slot < len(values):
+                values[slot] = value
+
+
+class AnyType(str):
+    def __ne__(self, other):
+        if self == "*":
+            # If we're the wildcard, match anything
+            return False
+        return super().__ne__(other)
+
+
+class ComboType(str):
+    def __ne__(self, other):
+        if self == "*" and isinstance(other, list):
+            return False
+        return True