This repo creates a series of nodes that enable you to utilize the Griptape Python Framework with ComfyUI, integrating LLMs (Large Language Models) and AI into your workflow.

Instructions and tutorials

Watch the trailer and all the instructional videos on our YouTube Playlist.

The repo currently has a subset of Griptape nodes, with more to come soon. Current nodes can:

Create Agents using these models:
- Local - via Ollama and LM Studio
  - Llama 3
  - Mistral
  - etc..
- Via Paid API Keys
  - OpenAI
  - Amazon Bedrock
  - Google Gemini
  - Anthropic Claude
Control agent behavior with access to [Rules and Rulesets][https://docs.griptape.ai/stable/griptape-framework/structures/rulesets/]
Give Agents access to Tools:
Run specific Agent Tasks:
- PromptTask
- TextSummaryTask
- ToolTask
- ToolkitTask
- PromptImageGenerationTask
- ImageQueryTask
- VariationImageGenerationTask (In Beta)
Generate Images using these models:
- OpenAI
- Amazon Bedrock Stable Diffusion
- Amazon Bedrock Titan
- Leonardo.AI
Transcribe Audio

Example

In this example, we're using three Image Description nodes to describe the given images. Those descriptions are then Merged into a single string which is used as inspiration for creating a new image using the Create Image from Text node, driven by an OpenAI Driver.

Using the nodes - Video Tutorials

Installation: https://youtu.be/L4-HnKH4BSI?si=Q7IqP-KnWug7JJ5s
Griptape Agents: https://youtu.be/wpQCciNel_A?si=WF_EogiZRGy0cQIm
Controlling which LLM your Agents use: https://youtu.be/JlPuyH5Ot5I?si=KMPjwN3wn4L4rUyg
Griptape Tools - Featuring Task Memory and Off-Prompt: https://youtu.be/TvEbv0vTZ5Q
Griptape Rulesets, and Image Creation: https://youtu.be/DK16ouQ_vSs
Image Generation with multiple drivers: https://youtu.be/Y4vxJmAZcho
Image Description, Parallel Image Description: https://youtu.be/OgYKIluSWWs?si=JUNxhvGohYM0YQaK
Audio Transcription: https://youtu.be/P4GVmm122B0?si=24b9c4v1LWk_n80T
Using Ollama as a Configuration Driver: https://youtu.be/jIq_TL5xmX0?si=ilfomN6Ka1G4hIEp
Combining Rulesets: https://youtu.be/3LDdiLwexp8?si=Oeb6ApEUTqIz6J6O
Integrating Text: https://youtu.be/_TVr2zZORnA?si=c6tN4pCEE3Qp0sBI
New Nodes & Quality of life improvements: https://youtu.be/M2YBxCfyPVo?si=pj3AFAhl2Tjpd_hw
Merge Input Data: https://youtu.be/wQ9lKaGWmZo?si=FgboU5iUg82pXRkC
Setting default agent configurations: https://youtu.be/IkioCcldEms?si=4uUu-y9UvIJWVBdE
Merge Text with dynamic inputs and custom separator: https://youtu.be/1fHAzKVPG4M?si=6JHe1NA2_a_nl9rG

Recent Changelog

July 9, 2024

Updated LMStudio and Ollama config nodes to use 127.0.0.1
Updated Create Agent and Run Agent nodes to no longer cache their knowledge between runs. Now if the agent input isn't connected to anything, it will create a new agent on each run.

July 2, 2024

All input nodes updated with dynamic inputs. Demonstration here: https://youtu.be/1fHAzKVPG4M?si=6JHe1NA2_a_nl9rG
Fixed bug with Text to Combo node

Installation

1. ComfyUI

Install ComfyUI using the instructions for your particular operating system.

2. Use Ollama

If you'd like to run with a local LLM, you can use Ollama and install a model like llama3.

Download and install Ollama from their website: https://ollama.com
Download a model by running ollama run <model>. For example:

ollama run llama3
You now have ollama available to you.

3. Add API Keys to your environment

For advanced features, it's recommended to use a more powerful model. These are available from the providers listed bellow, and will require API keys.

It's recommended to add these to a .env file in your base comfyUI folder before you start ComfyUI.

Note: Most nodes will work fine with just the OPENAI_API_KEY, so at least make sure you have that one.

OPENAI_API_KEY=
GOOGLE_API_KEY=
GOOGLE_API_SEARCH_ID=
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=
LEONARDO_API_KEY=
ANTHROPIC_API_KEY=
VOYAGE_API_KEY=
GRIPTAPE_API_KEY=

You can get the appropriate API keys from these respective sites:

OPENAI_API_KEY: https://platform.openai.com/api-keys
GOOGLE_API_KEY: https://makersuite.google.com/app/apikey
AWS_ACCESS_KEY_ID & SECURITY_ACCESS_KEY:
- Open the AWS Console
- Click on your username near the top right and select Security Credentials
- Click on Users in the sidebar
- Click on your username
- Click on the Security Credentials tab
- Click Create Access Key
- Click Show User Security Credentials
LEONARDO_API_KEY: https://docs.leonardo.ai/docs/create-your-api-key
ANTHROPIC_API_KEY: https://console.anthropic.com/settings/keys
VOYAGE_API_KEY: https://dash.voyageai.com/

4. Install Griptape-ComfyUI

There are two methods for installing the Griptape-ComfyUI repository. You can either download or git clone this repository inside the ComfyUI/custom_nodes, or use the ComfyUI Manager.

Option A - ComfyUI Manager (Recommended)
1. Install ComfyUI Manager by following the installation instructions.
2. Click Manager in ComfyUI to bring up the ComfyUI Manager
3. Search for "Griptape"
4. Find the ComfyUI-Griptape repo.
5. Click INSTALL
6. Follow the rest of the instructions.

Option B - Git Clone

Open a terminal and input the following commands:

cd /path/to/comfyUI
cd custom_nodes
git clone https://github.com/griptape-ai/ComfyUI-Griptape

5. Make sure libraries are loaded

Libraries should be installed automatically, but if you're having trouble, hopefully this can help.

There are certain libraries required for Griptape nodes that are called out in the requirements.txt file.

griptape[all]
python-dotenv
duckduckgo_search

These should get installed automatically if you used the ComfyUI Manager installation method. However, if you're running into issues, please install them yourself either using pip or poetry, depending on your installation method.

Option A - pip

pip install "griptape[all]" python-dotenv duckduckgo_search

Option B - poetry

poetry add "griptape[all]" python-dotenv duckduckgo_search

6. Restart ComfyUI

Now if you restart comfyUI, you should see the Griptape menu when you click with the Right Mouse button.

If you don't see the menu, please come to our Discord and let us know what kind of errors you're getting - we would like to resolve them as soon as possible!

Troubleshooting

Griptape Not Updating

Sometimes you'll find that the Griptape library didn't get updated properly. This seems to be especially happening when using the ComfyUI Manager. You might see an error like:

ImportError: cannot import name 'OllamaPromptDriver' from 'griptape.drivers' (C:\Users\evkou\Documents\Sci_Arc\Sci_Arc_Studio\ComfyUi\ComfyUI_windows_portable\python_embeded\Lib\site-packages\griptape\drivers\__init__.py)

To resolve this, you must make sure Griptape is runnig with the appropriate version. Things to try:

Update again via the ComfyUI Manager
Uninstall & Re-install the Griptape nodes via the ComfyUI Manager
In the terminal, go to your ComfyUI directory and type: python -m pip install griptape -U
Reach out on Discord and ask for help.

API Keys

When you run ComfyUI with the Griptape nodes installed, the installation method will check your environment variables and automatically install the appropriate API keys in a file called: griptape_config.json.

If for some reason your environment variables aren't being set properly, you can do this manually by simply copying over the griptape_config.json.default file to griptape_config.json and add the proper keys there.

If you ever need to change your API keys, go ahead and update that configuration file with the proper key.

Thank you

Massive thank you for help and inspiration from the following people and repos!

Jovieux from https://github.com/Amorano/Jovimetrix
rgthree https://github.com/rgthree/rgthree-comfy