Merge pull request #8 from haohaocreates/publish

Add Github Action for Publishing to Comfy Registry

.github/workflows/publish.yml

@@ -0,0 +1,21 @@
+name: Publish to Comfy registry
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+    paths:
+      - "pyproject.toml"
+
+jobs:
+  publish-node:
+    name: Publish Custom Node to registry
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out code
+        uses: actions/checkout@v4
+      - name: Publish Custom Node
+        uses: Comfy-Org/publish-node-action@main
+        with:
+          ## Add your own personal access token to your Github Repository secrets and reference it here.
+          personal_access_token: ${{ secrets.REGISTRY_ACCESS_TOKEN }}

README.md

@@ -1,2 +1,131 @@
 # cyberdolphin
-Cyberdolphin Suite of ComfyUI nodes for wiring up things.
+
+The dolphin is wiring up OpenAI and local LLMs. OpenAI v1.2.3 is required.
+
+![cyberdolphin.png](examples/cyberdolphin.png)
+
+
+## Installation
+
+Git clone this repo into the `custom_nodes` folder.
+If necessary, check the pip requirements. It will be necessary.
+
+## Examples
+
+There are workflows in the [examples folder](./examples)
+![img.png](examples/img.png)
+---
+
+## Nodes
+
+The nodes all share a config file at `settings.yaml`. Provided with the repo is the
+`settings.yaml.example` which can be copied to a new file called `settings.yaml`
+for editing. The `settings.yaml` file is ignored by git.
+
+### OpenAI GPT Node
+
+**REQUIRES** STRING  `user_prompt`
+
+The text is the user portion of the gpt prompt.
+_Generates_ an engineered prompt
+from a user-editable template config file with the user text embedded.
+Dropdown select from available OpenAI models. Most of these will not work.
+The models that DO work at the time of writing are at least, including but not limited to
+**gpt-3.5-turbo** and **gpt-4**.
+Runs the prompt gpt-3.5-turbo (or a user-selected alternative) with the text.
+
+**PRODUCES** STRING.
+
+### OpenAI Compatible Node
+
+**REQUIRES** STRING `text`
+
+The text is embedded in the user prompt.
+Generates an "engineered" prompt from template.
+The user text is embedded in the engineered prompt.
+Calls for completion of the prompt to the user-defined URL.
+
+**PRODUCES** STRING
+
+### OpenAI DALL·E Node
+
+**REQUIRES** STRING `text`
+
+Calls OpenAI DALL·E with the text.
+
+**PRODUCES** IMAGE
+
+
+---
+
+### Pip requirements
+
+This collection has some extra requirements that are not present in the ComfyUI distribution.
+Things like openai, gradio-client and technologist tools.
+
+### Experimental
+
+This is an experimental collection of nodes. This project needs validation on MacOS, Windows and Linux.
+So far, it works on my machine which is a Linux distribution.
+
+## Contributions
+
+Looking for participants, happy to work on PRs!
+
+**Guidelines for the Dolphin:**
+
+* Keep it small - PRs should be quick and easy.
+* Large things must be compositions of smaller things.
+* Dependencies should be external - i.e. loaded by a node
+* For example:
+    * _the Llava loader node passes the Llava model to the recogniser node which uses the Llava model to emit a list of
+      objects_
+    * _and not, the "Llava node does everything"_
+
+**Keep it small**
+
+In the spirit of "Keep it small", I'm trying to make sure my big ideas for the dolphin
+stay within the realm of LLMs -
+
+Here are some big ideas that didn't make it into the roadmap for CyberDolphin:
+
+#### Big Ideas I have for future things that are not the dolphin:
+
+**Cam Nodes**
+
+* **Webcam Node** for phone/laptop
+* **Cam Node** for HDMI type input devices
+* **Live Stream Node** to capture vision from a _Thing of the Internet_
+
+**Speech to Text**
+
+* **Microphone node** Captures spoken instructions into **audio node**
+* Instructions are transcribed using
+    * **OpenAI-Whisper node** or
+    * _TTS model_ loaded by the **TTS node**
+
+**The Simple Storybook Production Kit**
+
+Where "LLM-node" is short for "LLM powered node":
+
+```text
+LLM-node dreams up the story type
+LLM-node dreams up the character names, their badge
+LLM-node dreams up the story title
+LLM-node dreams up chapter summaries
+
+LLM-node generates a page in "the story"
+
+LLM-node generates images of characters:
+    id badge,
+    smiling photo,
+    frowning photo,
+    'character' shot
+
+LLM-node generates prompt for page illustration
+LLM-node generates page text
+```
+
+## License
+
+GPL 3.

__init__.py

@@ -1,9 +1,21 @@
 from .cyberdolphin_gradio import CyberDolphinGradioApi
+from .cyberdolphin_openai_advanced import CyberdolphinOpenAIAdvanced
+from .cyberdolphin_openai_simple import CyberdolphinOpenAISimple
+from .cyberdolphin_openai_compatible import CyberdolphinOpenAICompatible
+from .cyberdolphin_imageneering import CyberDolphinImageneering
 
 NODE_CLASS_MAPPINGS = {
-    "🐬 Gradio ChatInterface": CyberDolphinGradioApi
+    "🐬 Gradio ChatInterface": CyberDolphinGradioApi,
+    "🐬 OpenAI Simple": CyberdolphinOpenAISimple,
+    "🐬 OpenAI Advanced": CyberdolphinOpenAIAdvanced,
+    "🐬 OpenAI Compatible": CyberdolphinOpenAICompatible,
+    "🐬 OpenAI DALL·E": CyberDolphinImageneering,
 }
 
 NODE_DISPLAY_NAME_MAPPINGS = {
-    "CyberDolphin Gradio": "🐬 CyberDolphin Gradio"
+    "CyberDolphin Gradio": "🐬 CyberDolphin Gradio",
+    "CyberDolphin GPT-3.5 (Simple)": "🐬 CyberDolphin GPT-3.5 (Simple)",
+    "CyberDolphin OpenAI (Advanced)": "🐬 CyberDolphin OpenAI (Advanced)",
+    "CyberDolphin OpenAI Compatible": "🐬 CyberDolphin OpenAI Compatible",
+    "CyberDolphin OpenAI DALL·E": "🐬 CyberDolphin OpenAI DALL·E",
 }

cyberdolphin_gradio.py

@@ -7,53 +7,31 @@ class CyberDolphinGradioApi:
 
     @classmethod
     def INPUT_TYPES(s):
-        """
-            Return a dictionary which contains config for all input fields.
-            Some types (string): "MODEL", "VAE", "CLIP", "CONDITIONING", "LATENT", "IMAGE", "INT", "STRING", "FLOAT".
-            Input types "INT", "STRING" or "FLOAT" are special values for fields on the node.
-            The type can be a list for selection.
-
-            Returns: `dict`:
-                - Key input_fields_group (`string`): Can be either required, hidden or optional. A node class must have property `required`
-                - Value input_fields (`dict`): Contains input fields config:
-                    * Key field_name (`string`): Name of a entry-point method's argument
-                    * Value field_config (`tuple`):
-                        + First value is a string indicate the type of field or a list for selection.
-                        + Second value is a config for type "INT", "STRING" or "FLOAT".
-        """
+        the_settings = load_settings()
+        prompt_templates = [p for p in the_settings['prompt_templates']]
+        example_user_prompt = the_settings['example_user_prompt']
         return {
             "required": {
-                "text": ("STRING", {
-                    "default": '',
+                "user_prompt": ("STRING", {
+                    "default": example_user_prompt,
                     "multiline": True,
-                    "forceInput": True
                 }),
-                "llm_prompt": ([p for p in load_settings()['prompts']], "STRING"),
+                "llm_prompt": (prompt_templates, "STRING"),
             },
         }
 
     RETURN_TYPES = ("STRING",)
-    # RETURN_NAMES = ("image_output_name",)
-
+    RETURN_NAMES = ("llm_response",)
     FUNCTION = "generate"
-
-    # OUTPUT_NODE = False
-
     CATEGORY = "🐬 CyberDolphin"
 
-    def generate(self, text="", llm_prompt: str = "default_prompt"):
+    def generate(self, user_prompt="", llm_prompt: str = "default_prompt"):
         settings = load_settings()
         client_src = settings['gradio_chat_interface']['src']
         client = Client(client_src)
-        prompt_prefix = settings['prompts'][llm_prompt]['prefix']
-        prompt_suffix = settings['prompts'][llm_prompt]['suffix']
-
-        prompt = f'{prompt_prefix} {text} {prompt_suffix}'
-        # prompt = f'{self.PREFIX_PROMPT}{string_field}{self.SUFFIX_PROMPT}'
+        prompt_prefix = settings['prompt_templates'][llm_prompt]['prefix']
+        prompt_suffix = settings['prompt_templates'][llm_prompt]['suffix']
+        prompt = f'{prompt_prefix} {user_prompt} {prompt_suffix}'
         result = client.predict(prompt, api_name="/chat")
         response = result
         return (f'{response}',)
-
-
-if __name__ == "__main__":
-    print("Hello there.")

cyberdolphin_imageneering.py

@@ -0,0 +1,31 @@
+from PIL import ImageOps
+import torch
+import numpy as np
+from .openai_client import OpenAiClient, DALL_E_SIZE
+
+
+class CyberDolphinImageneering:
+    @classmethod
+    def INPUT_TYPES(s):
+        return {"required": {
+            "prompt": ('STRING', {'default': 'darth vader with yoda ears', 'multiline': True}),
+            "size": (["256x256", "512x512", "1024x1024"], {'default': "1024x1024"}),
+        }}
+
+    CATEGORY = "🐬 CyberDolphin"
+    RETURN_TYPES = ("IMAGE", "MASK")
+    FUNCTION = "load_image"
+
+    def load_image(self, prompt: str, size: DALL_E_SIZE):
+        i = OpenAiClient.image_create(prompt=prompt, size=size)
+        # copy/pasted from class LoadImage:
+        i = ImageOps.exif_transpose(i)
+        image = i.convert("RGB")
+        image = np.array(image).astype(np.float32) / 255.0
+        image = torch.from_numpy(image)[None,]
+        if 'A' in i.getbands():
+            mask = np.array(i.getchannel('A')).astype(np.float32) / 255.0
+            mask = 1. - torch.from_numpy(mask)
+        else:
+            mask = torch.zeros((64, 64), dtype=torch.float32, device="cpu")
+        return image, mask.unsqueeze(0)

cyberdolphin_openai_advanced.py

@@ -0,0 +1,71 @@
+from .openai_client import OpenAiClient
+from .settings import load_settings
+
+
+class CyberdolphinOpenAIAdvanced:
+    the_settings = None
+
+    @classmethod
+    def INPUT_TYPES(s):
+        openai_model_list = OpenAiClient.model_list()
+        the_settings = load_settings()
+        gpt_prompt = the_settings['prompt_templates']['gpt-3.5-turbo']
+        example_system_prompt = gpt_prompt['system']
+        example_user_prompt = f"{gpt_prompt['prefix']}{the_settings['example_user_prompt']}{gpt_prompt['suffix']}"
+
+        return {
+            "required": {
+                "model": (openai_model_list, {
+                    "default": "gpt-3.5-turbo"}),
+                "system_prompt": ('STRING', {
+                    "multiline": True,
+                    "default": example_system_prompt
+                }),
+                "user_prompt": ("STRING", {
+                    "multiline": True,
+                    "default": example_user_prompt
+                }),
+                "temperature": ("FLOAT", {
+                    "default": 1.0, "min": 0.0, "max": 2.0, "step": 0.01,
+                    "help": """
+                            What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the
+                            output more random, while lower values like 0.2 will make it more focused and deterministic.
+                            
+                            We generally recommend altering this or top_p but not both.
+                    """
+
+                }),
+            },
+            "optional": {
+                "top_p": ("FLOAT", {
+                    "default": 1.0, "min": 0.001, "max": 1.0, "step": 0.01,
+                    "help": """
+                            An alternative to sampling with temperature, called nucleus sampling, where the model
+                            considers the results of the tokens with top_p probability mass.
+                            So 0.1 means only the tokens comprising the top 10% probability mass are considered.
+                            
+                            We generally recommend altering this or `temperature` but not both.
+                    """
+                }),
+            }
+        }
+
+    RETURN_TYPES = ("STRING",)
+    RETURN_NAMES = ("gpt_response",)
+    FUNCTION = "generate"
+    CATEGORY = "🐬 CyberDolphin"
+
+    def generate(self, model: str, system_prompt: str, user_prompt="",
+                 temperature: float | None = None, top_p: float | None = None):
+        system_content = system_prompt
+        user_content = user_prompt
+
+        response = OpenAiClient.complete(
+            key="openai",
+            model=model,
+            temperature=temperature,
+            top_p=top_p,
+            system_content=system_content,
+            user_content=user_content)
+
+        return (f'{response.choices[0].message.content}',)

cyberdolphin_openai_compatible.py

@@ -0,0 +1,61 @@
+from .openai_client import OpenAiClient
+from .settings import load_settings
+
+
+class CyberdolphinOpenAICompatible:
+
+    @classmethod
+    def INPUT_TYPES(s):
+        all_settings = load_settings()
+        prompt_templates = all_settings['prompt_templates']
+        default_user_prompt = all_settings['example_user_prompt']
+        available_apis = [a for a in all_settings['openai_compatible']]
+        available_templates = [t for t in prompt_templates]
+        default_model = all_settings['openai_compatible']['default']['model']
+
+        return {
+            "required": {
+                "api": (available_apis, {
+                    "default": "default"
+                }),
+                "prompt_template": (available_templates, {
+                    "default": 'default'
+                }),
+                "model": ("STRING", {
+                    "default": default_model
+                }),
+                "user_prompt": ("STRING", {
+                    "multiline": True,
+                    "default": default_user_prompt
+                }),
+                "temperature": ("FLOAT", {
+                    "default": 1.0, "min": 0.0, "max": 2.0, "step": 0.01,
+                }),
+            },
+            "optional": {
+                "top_p": ("FLOAT", {
+                    "default": 1.0, "min": 0.001, "max": 1.0, "step": 0.01,
+                }),
+            }
+        }
+
+    RETURN_TYPES = ("STRING",)
+    RETURN_NAMES = ("gpt_response",)
+    FUNCTION = "generate"
+    CATEGORY = "🐬 CyberDolphin"
+
+    def generate(self, api: str, prompt_template: str, model: str, temperature: float | None = None,
+                 top_p: float | None = None, user_prompt=""):
+        this_prompt = load_settings()['prompt_templates'][prompt_template]
+        system_content = this_prompt['system']
+        user_content = f"{this_prompt['prefix']} {user_prompt} {this_prompt['suffix']}"
+
+        response = OpenAiClient.complete(
+            key=api,
+            model=model,
+            temperature=temperature,
+            top_p=top_p,
+            system_content=system_content,
+            user_content=user_content)
+
+        return (f'{response.choices[0].message.content}',)

cyberdolphin_openai_simple.py

@@ -0,0 +1,41 @@
+from .openai_client import OpenAiClient
+from .settings import load_settings
+
+
+class CyberdolphinOpenAISimple:
+
+    @classmethod
+    def INPUT_TYPES(s):
+        settings = load_settings()
+        return {
+            'required': {
+                'user_prompt': ('STRING', {
+                    'multiline': True,
+                    'default': settings['example_user_prompt']
+                }),
+                'temperature': ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.75, "step": 0.01}),
+                'model': (['gpt-3.5-turbo', 'gpt-4'], {
+                    'default': settings['openai_compatible']['openai']['model']
+                }),
+            },
+        }
+
+    RETURN_TYPES = ('STRING',)
+    RETURN_NAMES = ('gpt_response',)
+    FUNCTION = 'generate'
+    CATEGORY = '🐬 CyberDolphin'
+
+    def generate(self, user_prompt="", temperature: float = 1.0, model: str = "gpt-3.5-turbo"):
+        settings = load_settings()
+        gpt_prompt = settings['prompt_templates']['gpt-3.5-turbo']
+        system_content = gpt_prompt['system']
+        user_content = f"{gpt_prompt['prefix']} {user_prompt} {gpt_prompt['suffix']}"
+        response = OpenAiClient.complete(
+            key='openai',
+            model=model,
+            temperature=temperature,
+            top_p=1.0,
+            system_content=system_content,
+            user_content=user_content)
+
+        return (f'{response.choices[0].message.content}',)

examples/README.md

@@ -0,0 +1,19 @@
+# Examples
+
+## Cyber Dolphin
+
+![cyberdolphin_example.png](cyberdolphin_example.png)
+
+---
+
+Get image prompt from OpenAI or a compatible API. By default `gpt-3.5-turbo` is used.
+
+![img.png](img.png)
+
+---
+
+Compare the results of DALL·E and local SD models to see which one is better at generating images from text.
+
+
+![img_1.png](img_1.png)
+---

openai_client.py

@@ -0,0 +1,82 @@
+from typing import Literal, TypeAlias
+
+import PIL.Image
+from openai import OpenAI
+
+from custom_nodes.cyberdolphin.settings import api_settings
+
+DALL_E_SIZE: TypeAlias = Literal["256x256", "512x512", "1024x1024", "1792x1024", "1024x1792"]
+
+
+def validation(temperature: float, top_p: float = None) -> list[str]:
+    errors_list = []
+    if temperature is None and top_p is None:
+        errors_list.append('Must contain a temperature or a top_p')
+    if top_p < 0 or top_p > 1:
+        errors_list.append('top_p must be a number between 0 and 1')
+    if temperature < 0 or temperature > 2:
+        errors_list.append(
+            """Temperature should be a value between 0.0 and 2.0
+             - openai says higher values like 0.8 will make the output more random,
+             lower values like 0.2 make it more focused and deterministic.
+             """)
+    return errors_list
+
+
+def convert_bson_to_image(image_bson: str) -> PIL.Image.Image:
+    import base64
+    import io
+    from PIL import Image
+    image_bytes = base64.b64decode(image_bson)
+    image = Image.open(io.BytesIO(image_bytes))
+    return image
+
+
+class OpenAiClient:
+
+    @staticmethod
+    def create_client(key: str = "openai"):
+        api_base, api_key, organization = api_settings(key)
+        the_client = OpenAI(
+            base_url=api_base,
+            api_key=api_key,
+            organization=organization,
+        )
+        return the_client
+
+    @staticmethod
+    def model_list():
+        the_client = OpenAiClient.create_client()
+        the_models = the_client.models.list()
+        return [m.id for m in the_models.data]
+
+    @staticmethod
+    def image_create(prompt: str, size: DALL_E_SIZE = "1024x1024", ) -> PIL.Image.Image:
+        the_client = OpenAiClient.create_client()
+        response = the_client.images.generate(
+            n=1,
+            size=size,
+            prompt=prompt,
+            response_format="b64_json"
+        )
+        image_bson = response.data[0].b64_json
+        i = convert_bson_to_image(image_bson)
+        return i
+
+    @staticmethod
+    def complete(key: str, model: str, temperature: float, top_p: float, system_content: str, user_content: str):
+        errors = validation(temperature, top_p)
+        if errors:
+            error_report = "\n".join([e for e in errors])
+            raise RuntimeError(f"There were problems with the parameters:\n{error_report}")
+        the_client = OpenAiClient.create_client(key)
+        response = the_client.chat.completions.create(
+            model=model,
+            temperature=temperature,
+            top_p=top_p,
+            messages=[
+                {"role": "system", "content": system_content},
+                {"role": "user", "content": user_content}
+            ]
+        )
+        return response

pyproject.toml

@@ -0,0 +1,15 @@
+[project]
+name = "cyberdolphin"
+description = "Cyberdolphin nodes for wiring up OpenAI and compatible LLM APIs."
+version = "1.0.0"
+license = "LICENSE"
+dependencies = ["openai ~= 1.2.3", "gradio_client", "numpy", "pillow"]
+
+[project.urls]
+Repository = "https://github.com/whatbirdisthat/cyberdolphin"
+#  Used by Comfy Registry https://comfyregistry.org
+
+[tool.comfy]
+PublisherId = "whatbirdisthat"
+DisplayName = "cyberdolphin"
+Icon = "dolphin"

requirements.txt

@@ -0,0 +1,4 @@
+openai ~= 1.2.3
+gradio_client
+numpy
+pillow

settings.py

@@ -1,10 +1,45 @@
 import os
+import pathlib
 
 import yaml
 
-def load_settings(section: str = "cyberdolphin"):
+DEFAULT_SETTINGS = {
+    "cyberdolphin": {
+        "openai": {
+            "organisation": "NO ORG",
+            "api_key": "NO API KEY",
+            "model": "gpt-3.5-turbo"
+        },
+        "openai_compatible": {
+            "organisation": "NO ORG",
+            "api_key": "NO API KEY",
+            "api_base": 'http://localhost:8000/v1'
+        },
+        "prompts": {
+            "example_user_prompt": "{Camel|goldfish|glowing orb},{moss|tree|fern|balloon},{space station|garden shed|glowing laser sword|bowl of petunias|orange taxi|neon sign}",
+            "default_prompt": {
+                "system": "You are deeply artistic, understanding of concepts like composition, pallete and color theory, and image psychology.",
+                "prefix": "Do use objective language. Do not add narrative. Do describe objects visually and in context. \
+                Do not describe the purpose of the objects, or any other explanations \"",
+                "suffix": '"'
+            }
+        }
+    }
+}
+
+
+def load_settings():
     path = os.path.join(os.path.dirname(__file__), "settings.yaml")
+    file_path = pathlib.Path(path)
+    if not file_path.exists():
+        return DEFAULT_SETTINGS['cyberdolphin']
+
     with open(path) as settings:
         the_yaml = yaml.safe_load(settings)
-    print(f"LOADED: {the_yaml[section]}")
-    return the_yaml[section]
+    # print(f'LOADED: {the_yaml["cyberdolphin"]}')
+    return the_yaml['cyberdolphin']
+
+
+def api_settings(section: str = "openai"):
+    openai_settings = load_settings()['openai_compatible'][section]
+    return openai_settings['api_base'], openai_settings['api_key'], openai_settings['organisation']

settings.yaml.example

@@ -1,8 +1,29 @@
 cyberdolphin:
   gradio_chat_interface:
-    src: "http://localhost:7860"
-  prompts:
-    default_prompt:
+    src: http://localhost:7860
+  openai_compatible:
+    openai:
+      api_base: 'https://api.openai.com/v1'
+      organisation: "ORG"
+      api_key: "KEY"
+      model: "gpt-3.5-turbo"
+    default:
+      api_base: "http://127.0.0.1:8000/v1"
+      organisation: "NONE"
+      api_key: "NONE"
+      model: "Llama-2-13b-chat"
+    another_openai_api:
+      api_base: "http://127.0.0.1:8001/v1"
+      organisation: "NONE"
+      api_key: "NONE"
+      model: "another_model"
+  example_user_prompt: "{Camel|goldfish|glowing orb},{moss|tree|fern|balloon},{space station|garden shed|glowing laser sword|bowl of petunias|orange taxi|neon sign}"
+  prompt_templates:
+#    the user prompt is by default a list of objects, such as what might be returned from a resnet node
+
+    default:
+      system: >-
+        You are deeply artistic, understanding of concepts like composition, palette and color theory, and image psychology.
       prefix: >-
         make a list of the things you see. do not explain why you see them.
         do explain what the things you see are doing. do explain where they are.
@@ -11,19 +32,31 @@ cyberdolphin:
         do use purely objective language: do not say "I see a..." but instead say "there is a..."
         SO:
         When I provide the list "
-
-
       suffix: >-
         ",
         what is this scene -
         do not simply repeat the list, do not say "when I provide the list" or similar just project the description.
         describe what you see in your mind.
-        
+    
 
-    prompt_two:
+    prompt_two_example:
+      system: >-
+        You are deeply artistic, understanding of concepts like composition, palette and color theory, and image psychology.
       prefix: >-
         describe a scene using the following list of objects: "
       suffix: >-
         " - there are many things
         and you must list them using a maximum of 25 words. be clear and specific.
         use present tense, objective language: do not say, "I see ..." rather say, "there is ..."
+    
+
+    gpt-3.5-turbo:
+      system: >-
+        You are deeply artistic, understanding of concepts like composition, palette and color theory, and image psychology.
+      prefix: >-
+        describe a scene using the following list of objects: "
+      suffix: >-
+        " - there are many things
+        and you must list them using a maximum of 25 words. be clear and specific.
+        use present tense, objective language: do not say, "I see ..." rather say, "there is ..."
+