Updated model lists for various nodes
This commit is contained in:
@@ -1,25 +1,25 @@
|
||||
# ComfyUI-ExternalAPI-Helpers
|
||||
|
||||
A collection of powerful custom nodes for ComfyUI that connect your local workflows to closed-source AI models via their APIs. Use Google's Gemini, Imagen, Veo, OpenAI's GPT-Image-1, and Black Forest Labs' FLUX models directly within ComfyUI.
|
||||
A collection of powerful custom nodes for ComfyUI that connect your local workflows to closed-source AI models via their APIs. Use Google's Gemini, Imagen, Veo 3.1, OpenAI's GPT-6 and GPT Image 2.5, ElevenLabs v4, and Black Forest Labs' FLUX models directly within ComfyUI.
|
||||
|
||||
## Key Features
|
||||
|
||||
* **FLUX Kontext Pro & Max:** Image-to-image transformations using the FLUX models via the Replicate API.
|
||||
* **Flux.2 (Replicate):** Generate images using the latest FLUX.2 models (Pro, Max, Dev) via Replicate.
|
||||
* **Gemini Chat:** Google's powerful multimodal AI. Ask questions about an image, generate detailed descriptions or create prompts for other models. Supports thinking budget controls for applicable models. Now supports audio input.
|
||||
* **Gemini Speaker Diarization:** Separate audio into different speaker tracks using Gemini.
|
||||
* **GPT Image Edit:** OpenAI's `gpt-image-1` for prompt-based image editing and inpainting. Simply mask an area and describe the change you want to see.
|
||||
* **OpenAI LLM:** Access OpenAI's powerful language models (GPT-4, GPT-5, o1, etc.) for text generation and reasoning.
|
||||
* **Gemini Chat:** Google's frontier multimodal models (Gemini 3.8 Flash, 3.6 Flash, 3.5 Flash-Lite, 3.1 Pro). Analyze images, process audio waveforms, and generate responses with dynamic thinking controls.
|
||||
* **Gemini Speaker Diarization:** Separate audio into individual speaker tracks with high-precision timestamping using Gemini 3.8.
|
||||
* **GPT Image Edit:** Prompt-based image editing and inpainting with OpenAI's latest `gpt-image-2.5-sunburst` and `gpt-image-2.5-flare` alongside GPT Image 2.
|
||||
* **OpenAI LLM:** Access OpenAI's frontier reasoning and generation models (GPT-6, GPT-5.6 family, GPT-5.5, GPT-5.4) using the native Responses API with selectable reasoning effort (`none`, `low`, `medium`, `high`).
|
||||
* **OpenAI Text-to-Speech:** Generate speech with OpenAI's TTS models (`gpt-4o-mini-tts`, `tts-1`, `tts-1-hd`) featuring all 13 built-in voices and natural language tone/style instructions.
|
||||
* **ElevenLabs TTS:** Generate speech using ElevenLabs' newest **Eleven v4** and **Eleven v4 Turbo** models with automatic parameter branching, alongside legacy v3 and multilingual engines.
|
||||
* **Gemini 3.8 TTS:** Next-gen speech synthesis powered by Gemini 3.8 with access to Google's Extended Voice Library (2,000+ voices like *Bodi*, *Lumi*, *Koda*), native `speech_metadata` style direction, and inline vocal tags (`<breath>`, `<laugh>`, `<short pause>`, `<sigh>`, `<cough>`).
|
||||
* **Gemini TTS (Legacy):** Dedicated backwards-compatible node for earlier Gemini 2.5 and 3.1 celestial voice endpoints.
|
||||
* **Nano Banana:** Google's native image generation and editing using the latest **`gemini-nano-banana-2.1`**, featuring 1K/2K/4K resolution control and multi-image consistency.
|
||||
* **Veo Video Generator (Gemini API & Vertex AI):** Generate video clips using Google's **Veo 3.1** family (`veo-3.1-generate-preview`, `veo-3.1-fast`, `veo-3.1-lite` / Vertex `001` releases) with native synchronized audio, 1080p HD support, and flexible durations.
|
||||
* **Google Imagen Generator & Edit:** Create and edit images with Google's Imagen models, with support for Vertex AI.
|
||||
* **Groq LLM:** Fast text generation using Groq-hosted models, with optional image input for supported vision models.
|
||||
* **Groq Orpheus TTS:** Generate speech using Groq-hosted Orpheus text-to-speech models.
|
||||
* **OpenAI Text-to-Speech:** Generate high-quality speech using OpenAI's TTS models.
|
||||
* **Google Imagen Generator & Edit:** Create and edit images with Google's Imagen models, with support for Vertex AI.
|
||||
* **Nano Banana:** A creative image generation node using a specialized Gemini model.
|
||||
* **Veo Video Generator:** Generate high-quality video clips from text prompts using Google's Veo model via Vertex AI or the Gemini API.
|
||||
* **ElevenLabs TTS:** Generate high-quality speech from text using ElevenLabs' diverse range of voices and models.
|
||||
* **Gemini TTS:** Create speech from text using Google's Gemini models.
|
||||
* **Tripo Text-to-3D:** Generate 3D models from text prompts using Tripo AI's advanced 3D generation API.
|
||||
* **Tripo Image-to-3D:** Convert images into detailed 3D models using Tripo AI's image-to-model capability.
|
||||
* **Tripo Text-to-3D & Image-to-3D:** Generate and convert images into production 3D models using Tripo AI.
|
||||
|
||||
---
|
||||
|
||||
@@ -34,208 +34,146 @@ A collection of powerful custom nodes for ComfyUI that connect your local workfl
|
||||
```bash
|
||||
git clone https://github.com/Aryan185/ComfyUI-ExternalAPI-Helpers.git
|
||||
```
|
||||
|
||||
4. Install the required Python packages. Navigate into the newly cloned directory and use pip to install the dependencies:
|
||||
4. Install the required Python packages:
|
||||
```bash
|
||||
cd ComfyUI-ExternalAPI-Helpers
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
5. **Restart ComfyUI.** After restarting, you should find the new nodes in the "Add Node" menu.
|
||||
5. **Restart ComfyUI.** The nodes will be available in the "Add Node" menu.
|
||||
|
||||
---
|
||||
|
||||
## 🔑 Prerequisites: API Keys
|
||||
|
||||
All nodes in this collection require API keys to function.
|
||||
|
||||
* **FLUX Nodes (Replicate):** You will need a [Replicate API Token](https://replicate.com/account/api-tokens).
|
||||
* **Gemini, Imagen, Nano Banana, Gemini TTS, Gemini Diarization, and Veo (Gemini API) Nodes:** You will need a [Google AI Studio API Key](https://aistudio.google.com/app/api-keys).
|
||||
* **OpenAI Nodes (GPT Image Edit, OpenAI LLM, OpenAI TTS):** You will need an [OpenAI API Key](https://platform.openai.com/api-keys).
|
||||
* **Groq Nodes (LLM, Orpheus TTS):** You will need a [Groq API Key](https://console.groq.com/keys).
|
||||
* **ElevenLabs TTS Node:** You will need an [ElevenLabs API Key](https://elevenlabs.io/).
|
||||
* **Tripo Nodes (Text-to-3D, Image-to-3D):** You will need a [Tripo API Key](https://tripo3d.ai/).
|
||||
* **Vertex AI Nodes (Imagen Edit, Veo Vertex AI):** You will need a Google Cloud Project ID, a service account with appropriate permissions, and the location for the resources.
|
||||
|
||||
You can paste your key directly into the `api_key` field on the corresponding node. For Vertex AI nodes, you will need to provide the project ID, location, and paste your service account JSON content.
|
||||
* **FLUX Nodes (Replicate):** [Replicate API Token](https://replicate.com/account/api-tokens).
|
||||
* **Google AI Studio Nodes (Gemini Chat, Diarization, Nano Banana, Gemini 3.8 TTS, Veo Gemini API):** [Google AI Studio API Key](https://aistudio.google.com/app/api-keys).
|
||||
* **OpenAI Nodes (GPT Image Edit, OpenAI LLM, OpenAI TTS):** [OpenAI API Key](https://platform.openai.com/api-keys).
|
||||
* **ElevenLabs TTS Node:** [ElevenLabs API Key](https://elevenlabs.io/).
|
||||
* **Groq Nodes (LLM, Orpheus TTS):** [Groq API Key](https://console.groq.com/keys).
|
||||
* **Tripo Nodes (Text-to-3D, Image-to-3D):** [Tripo API Key](https://tripo3d.ai/).
|
||||
* **Vertex AI Nodes (Imagen Edit, Veo Vertex AI):** Google Cloud Project ID, location, and service account JSON key.
|
||||
|
||||
### Using Environment Variables (.env file)
|
||||
|
||||
Instead of pasting API keys directly into nodes, you can store them in a `.env` file:
|
||||
Instead of pasting raw API keys into every node, store them in a `.env` file in the node directory:
|
||||
|
||||
1. Copy `.env.example` to `.env` in the node directory
|
||||
2. Fill in your API keys:
|
||||
```
|
||||
GEMINI_API_KEY=your_gemini_key_here
|
||||
OPENAI_API_KEY=your_openai_key_here
|
||||
GROQ_API_KEY=your_groq_key_here
|
||||
XI_API_KEY=your_elevenlabs_key_here
|
||||
REPLICATE_API_TOKEN=your_replicate_token_here
|
||||
TRIPO_API_KEY=your_tripo_api_key_here
|
||||
```
|
||||
3. In the node's `api_key` field, enter the variable name (e.g., `GEMINI_API_KEY`) instead of the actual key
|
||||
```env
|
||||
GEMINI_API_KEY=your_gemini_key_here
|
||||
OPENAI_API_KEY=your_openai_key_here
|
||||
XI_API_KEY=your_elevenlabs_key_here
|
||||
GROQ_API_KEY=your_groq_key_here
|
||||
REPLICATE_API_TOKEN=your_replicate_token_here
|
||||
TRIPO_API_KEY=your_tripo_api_key_here
|
||||
```
|
||||
|
||||
This keeps your API keys secure and makes it easy to switch between different keys or projects.
|
||||
In any node's `api_key` field, you can simply type the variable name (e.g. `GEMINI_API_KEY`) instead of exposing your actual secret.
|
||||
|
||||
---
|
||||
|
||||
## 📚 Node Guide
|
||||
|
||||
> **Note:** All nodes include standard inputs like `api_key`, `prompt`, `model`, `temperature`, and `seed` where applicable. Only unique or notable inputs are listed below.
|
||||
|
||||
### Flux Kontext Pro / Max
|
||||
|
||||
Transform images based on text prompts. Ideal for applying artistic styles or making conceptual changes.
|
||||
|
||||
* **Category:** `image/edit`
|
||||
* **Key Inputs:** `image`, `aspect_ratio` (use `match_input_image` to preserve composition)
|
||||
* **Output:** `image`
|
||||
|
||||
### Flux.2 (Replicate)
|
||||
|
||||
Generate images using FLUX.2 models (Pro, Max, Dev) via Replicate.
|
||||
|
||||
* **Category:** `image/generation`
|
||||
* **Key Inputs:** `image_1` to `image_5` (optional, for image-to-image tasks)
|
||||
* **Output:** `image`
|
||||
|
||||
### Gemini Chat
|
||||
|
||||
Multimodal text generation with image and audio analysis capabilities.
|
||||
|
||||
* **Category:** `text/generation`
|
||||
* **Key Inputs:** `thinking`, `thinking_budget`, `system_instruction`, `image` (optional), `audio` (optional)
|
||||
* **Models:** `gemini-3.8-flash` (default), `gemini-3.6-flash`, `gemini-3.5-flash-lite`, `gemini-3.1-pro-preview`, dynamic aliases.
|
||||
* **Key Inputs:** `prompt`, `thinking`, `thinking_budget`, `google_search`, `system_instruction`, `image` (optional), `audio` (optional)
|
||||
* **Output:** `response` (text)
|
||||
|
||||
### Gemini Speaker Diarization
|
||||
|
||||
Separate audio into different speaker tracks.
|
||||
|
||||
* **Category:** `audio/diarise`
|
||||
* **Key Inputs:** `audio`, `num_speakers`, `thinking`, `thinking_budget`
|
||||
* **Output:** `speaker_1` to `speaker_4` (audio tracks)
|
||||
* **Key Inputs:** `audio`, `num_speakers`, `model`, `thinking`, `thinking_budget`
|
||||
* **Output:** `speaker_1` to `speaker_4` (isolated audio tracks)
|
||||
|
||||
### GPT Image Edit
|
||||
|
||||
Prompt-based inpainting and image editing using OpenAI's API.
|
||||
|
||||
* **Category:** `image/edit`
|
||||
* **Key Inputs:** `image_1` to `image_5`, `mask` (optional, white area = edit region), `background`, `quality`, `size`
|
||||
* **Category:** `image/generation`
|
||||
* **Models:** `gpt-image-2.5-sunburst` (default), `gpt-image-2.5-flare`, `gpt-image-2`, `gpt-image-1.5`, `gpt-image-1`
|
||||
* **Key Inputs:** `prompt`, `image_1` to `image_5` (optional reference/edit inputs), `mask` (optional, white area = inpainting region), `background`, `quality`, `size`
|
||||
* **Output:** `image`
|
||||
|
||||
### OpenAI LLM
|
||||
|
||||
Access OpenAI language models (GPT-4, GPT-5, o1, etc.) for text generation.
|
||||
|
||||
* **Category:** `text/generation`
|
||||
* **Key Inputs:** `reasoning_effort` (low/medium/high), `max_output_tokens`, `system_instruction`, `image` (optional)
|
||||
* **Models:** `gpt-6` (default), `gpt-6.1-sol`, `gpt-6-luna`, `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-5.5`, `gpt-5.4`, `gpt-4.1`
|
||||
* **Key Inputs:** `prompt`, `reasoning_effort` (`none`, `low`, `medium`, `high`), `max_output_tokens`, `system_instruction`, `image` (optional)
|
||||
* **Output:** `response` (text)
|
||||
|
||||
### Groq LLM
|
||||
|
||||
Access Groq-hosted language models for fast text generation, with optional image understanding on supported vision models.
|
||||
|
||||
* **Category:** `text/generation`
|
||||
* **Key Inputs:** `max_completion_tokens`, `reasoning_effort`, `system_instruction`, `image` (optional)
|
||||
* **Output:** `response` (text)
|
||||
|
||||
### Groq Orpheus TTS
|
||||
|
||||
Generate speech using Groq-hosted Orpheus text-to-speech models.
|
||||
|
||||
* **Category:** `audio/generation`
|
||||
* **Key Inputs:** `text`, `voice`, `speed`
|
||||
* **Output:** `audio`
|
||||
|
||||
### OpenAI Text-to-Speech
|
||||
|
||||
Generate speech using OpenAI's TTS models.
|
||||
|
||||
* **Category:** `audio/generation`
|
||||
* **Key Inputs:** `text`, `voice` (alloy, echo, etc.), `response_format`, `speed`, `instructions` (optional)
|
||||
* **Models:** `gpt-4o-mini-tts` (default), `tts-1`, `tts-1-hd`
|
||||
* **Voices:** 13 voices (`alloy`, `ash`, `ballad`, `cedar`, `coral`, `echo`, `fable`, `marin`, `nova`, `onyx`, `sage`, `shimmer`, `verse`)
|
||||
* **Key Inputs:** `text`, `voice`, `response_format`, `speed`, `instructions` (natural-language tone/pacing direction on `gpt-4o-mini-tts`)
|
||||
* **Output:** `audio`
|
||||
|
||||
### Google Imagen Generator
|
||||
|
||||
Generate images from text using Google's Imagen models.
|
||||
|
||||
* **Category:** `image/generation`
|
||||
* **Key Inputs:** `number_of_images`, `aspect_ratio`, `image_size`, `guidance_scale`, `negative_prompt`
|
||||
* **Output:** `images`
|
||||
|
||||
### Google Imagen Edit (Vertex AI only)
|
||||
|
||||
Advanced image editing with inpainting, outpainting, and background swapping.
|
||||
|
||||
* **Category:** `image/edit`
|
||||
* **Key Inputs:** `image`, `mask`, `project_id`, `location`, `service_account` (JSON content), `edit_mode`
|
||||
* **Output:** `edited_images`
|
||||
|
||||
### Nano Banana
|
||||
|
||||
Creative image generation using a specialized Gemini model.
|
||||
|
||||
* **Category:** `image/generation`
|
||||
* **Key Inputs:** `image_1` to `image_5` (optional), `aspect_ratio`, `resolution`, `top_p`
|
||||
* **Output:** `image`
|
||||
|
||||
### Veo Video Generator (Vertex AI)
|
||||
|
||||
Generate video clips using Google's Veo model on Vertex AI.
|
||||
|
||||
* **Category:** `video/generation`
|
||||
* **Key Inputs:** `project_id`, `location`, `service_account` (JSON content), `resolution`, `aspect_ratio`, `duration_seconds`, `generate_audio`, `first_frame`/`last_frame` (optional)
|
||||
* **Output:** `frames`, `audio`
|
||||
|
||||
### Veo Video Generator (Gemini API)
|
||||
|
||||
Generate videos using Veo 2.0 via the Gemini API. Supports text-to-video and image-to-video.
|
||||
|
||||
* **Category:** `video/generation`
|
||||
* **Key Inputs:** `aspect_ratio`, `duration_seconds`, `negative_prompt`, `image` (optional, for image-to-video)
|
||||
* **Output:** `frames`
|
||||
|
||||
### ElevenLabs TTS
|
||||
|
||||
Generate speech using ElevenLabs' diverse voices and models.
|
||||
|
||||
* **Category:** `audio/generation`
|
||||
* **Key Inputs:** `text`, `voice_id`, `model_id`, `stability`, `similarity_boost`, `speed`, `style`, `use_speaker_boost`
|
||||
* **Models:** `eleven_v4` (default), `eleven_v4_turbo`, `eleven_v3`, `eleven_multilingual_v2`, `eleven_flash_v2_5`, `eleven_flash_v2`, `eleven_turbo_v2_5`
|
||||
* **Key Inputs:** `text`, `voice_id`, `stability`, `similarity_boost`, `speed`, `style`, `use_speaker_boost`, `previous_text` / `next_text` (automatically branches parameters to match v4 requirements)
|
||||
* **Output:** `audio`
|
||||
|
||||
### Gemini TTS
|
||||
|
||||
Generate speech using Google's Gemini TTS models.
|
||||
|
||||
### Gemini 3.8 TTS
|
||||
* **Category:** `audio/generation`
|
||||
* **Key Inputs:** `text`, `voice_id`, `system_prompt` (optional)
|
||||
* **Models:** `gemini-3.8-flash-tts` (default), `gemini-3.8-flash-lite-tts`
|
||||
* **Key Inputs:**
|
||||
* `text`: Supports point-in-time inline tags directly in the transcript (e.g. `<breath>`, `<laugh>`, `<short pause>`, `<long pause>`, `<sigh>`, `<cough>`, `<gasp>`, `<whispering>`).
|
||||
* `voice`: Open string targeting any of the 2,000+ Extended Voice Library voices from AI Studio (e.g. `Bodi`, `Lumi`, `Koda`, `Nika`, `Tavi`) or legacy celestial voices (`Puck`, `Zephyr`).
|
||||
* `style`: Global acting/delivery note passed straight to native `speech_metadata`.
|
||||
* **Output:** `audio`
|
||||
|
||||
### Tripo Text-to-3D
|
||||
### Gemini TTS (Legacy)
|
||||
* **Category:** `audio/generation`
|
||||
* **Models:** `gemini-2.5-flash-preview-tts`, `gemini-2.5-pro-preview-tts`, `gemini-3.1-flash-tts-preview`
|
||||
* **Key Inputs:** Fixed 30 celestial voices dropdown (`Puck`, `Zephyr`, etc.), `pace`, `accent`, `scene`, `audio_profile` prompt-director options.
|
||||
* **Output:** `audio`
|
||||
|
||||
Generate high-quality 3D models from text prompts using Tripo AI's powerful generation engine.
|
||||
### Nano Banana
|
||||
* **Category:** `image/generation`
|
||||
* **Models:** `gemini-nano-banana-2.1` (default), `gemini-3-pro-image`, `gemini-3.1-flash-lite-image`, `gemini-3.1-flash-image`, `gemini-2.5-flash-image`
|
||||
* **Key Inputs:** `prompt`, `aspect_ratio`, `resolution` (`1K`, `2K`, `4K`), `image_1` to `image_5` (multi-image consistency & editing), `google_search`
|
||||
* **Output:** `image`
|
||||
|
||||
### Veo Video Generator (Gemini API)
|
||||
* **Category:** `video/generation`
|
||||
* **Models:** `veo-3.1-generate-preview` (default), `veo-3.1-fast-generate-preview`, `veo-3.1-lite-generate-preview`, `veo-2.0-generate-001`
|
||||
* **Key Inputs:** `prompt`, `aspect_ratio` (16:9, 9:16), `resolution` (720p, 1080p on 3.1), `duration_seconds` (4 to 8s), `image` (optional, for image-to-video)
|
||||
* **Output:** `frames`
|
||||
|
||||
### Veo Video Generator (Vertex AI)
|
||||
* **Category:** `video/generation`
|
||||
* **Models:** `veo-3.1-generate-001` (default), `veo-3.1-fast-generate-001`, `veo-3.0-generate-001`, `veo-2.0-generate-001`
|
||||
* **Key Inputs:** `project_id`, `location`, `service_account`, `resolution`, `aspect_ratio`, `duration_seconds`, `generate_audio` (native synchronized audio on 3.x), `first_frame` / `last_frame`
|
||||
* **Output:** `frames`, `audio`
|
||||
|
||||
### Google Imagen Generator & Edit
|
||||
* **Category:** `image/generation` & `image/edit`
|
||||
* **Key Inputs:** Generation controls (`aspect_ratio`, `image_size`, `guidance_scale`) and Vertex AI edit modes (inpainting, outpainting, background swap).
|
||||
* **Output:** `images` / `edited_images`
|
||||
|
||||
### Groq LLM & Groq Orpheus TTS
|
||||
* **Category:** `text/generation` & `audio/generation`
|
||||
* **Key Inputs:** Fast inference using Groq Cloud endpoints for text/vision reasoning and Orpheus voice generation.
|
||||
* **Output:** `response` (text) / `audio`
|
||||
|
||||
### Tripo Text-to-3D & Image-to-3D
|
||||
* **Category:** `3D/generation`
|
||||
* **Key Inputs:** `prompt`, `model_version` (v3.0, v2.5, v2.0, v1.4, Turbo), `texture_quality` (standard/detailed), `save_name`
|
||||
* **Optional Inputs:** `seed`, `face_limit` (500-100000, default 20000), `texture_size` (512/1024/2048), `pbr` (realistic lighting), `negative_prompt`
|
||||
* **Output:** `glb` (3D model file)
|
||||
|
||||
### Tripo Image-to-3D
|
||||
|
||||
Convert images into detailed 3D models using Tripo AI's advanced image-to-model technology.
|
||||
|
||||
* **Category:** `3D/generation`
|
||||
* **Key Inputs:** `image`, `model_version` (v3.0, v2.5, v2.0, v1.4), `texture_quality` (standard/detailed), `save_name`
|
||||
* **Optional Inputs:** `seed`, `texture_seed` (randomness of colors/textures), `face_limit` (500-100000, default 20000), `texture_size` (512/1024/2048), `pbr` (realistic lighting)
|
||||
* **Key Inputs:** `prompt` or `image`, `model_version` (v3.0, v2.5, v2.0, Turbo), `texture_quality`, `face_limit`, `pbr`
|
||||
* **Output:** `glb` (3D model file)
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
* The [ComfyUI](https://github.com/comfyanonymous/ComfyUI) team for building the workflow platform that makes these nodes possible.
|
||||
* [Google](https://deepmind.google/technologies/gemini/) for Gemini, Imagen, and Veo.
|
||||
* [OpenAI](https://openai.com/) for GPT models and image and speech APIs.
|
||||
* [Groq](https://groq.com/) for fast inference and hosted text-to-speech support.
|
||||
* [xAI](https://x.ai/) for Grok image generation and editing models.
|
||||
* [Black Forest Labs](https://www.blackforestlabs.ai/) and [Replicate](https://replicate.com/) for FLUX model access.
|
||||
* [ElevenLabs](https://elevenlabs.io/) for speech synthesis.
|
||||
* [Tripo AI](https://www.tripo3d.ai/) for 3D generation APIs.
|
||||
* The [ComfyUI](https://github.com/comfyanonymous/ComfyUI) team for building the modular node platform.
|
||||
* [Google](https://deepmind.google/technologies/gemini/) for Gemini 3.8, Nano Banana, Imagen, and Veo 3.1.
|
||||
* [OpenAI](https://openai.com/) for GPT-6, GPT Image 2.5, and speech models.
|
||||
* [ElevenLabs](https://elevenlabs.io/) for Eleven v4 and conversational audio synthesis.
|
||||
* [Groq](https://groq.com/) for high-throughput LPU inference.
|
||||
* [Black Forest Labs](https://www.blackforestlabs.ai/) and [Replicate](https://replicate.com/) for FLUX model infrastructure.
|
||||
* [Tripo AI](https://www.tripo3d.ai/) for native text and image-to-3D APIs.
|
||||
+1
-1
@@ -18,13 +18,13 @@ modules = [
|
||||
"flux2_replicate",
|
||||
"openai_tts",
|
||||
"openai_node",
|
||||
"sora",
|
||||
"cleanup",
|
||||
"tripoTextToModel",
|
||||
"tripoImageToModel",
|
||||
"grok",
|
||||
"groq_node",
|
||||
"groq_orpheus",
|
||||
"gemini_tts3",
|
||||
]
|
||||
|
||||
NODE_CLASS_MAPPINGS = {}
|
||||
|
||||
+18
-10
@@ -12,6 +12,8 @@ class ElevenLabsTTSNode:
|
||||
"required": {
|
||||
"text": ("STRING", {"multiline": True, "default": ""}),
|
||||
"model_id": ([
|
||||
"eleven_v4",
|
||||
"eleven_v4_turbo",
|
||||
"eleven_v3",
|
||||
"eleven_multilingual_v2",
|
||||
"eleven_flash_v2_5",
|
||||
@@ -85,13 +87,19 @@ class ElevenLabsTTSNode:
|
||||
if stability != original_stability:
|
||||
print(f"For 'eleven_v3' model, stability value must be one of: [0.0, 0.5, 1.0] (0.0 = Creative, 0.5 = Natural, 1.0 = Robust). Rounding stability to {stability}.")
|
||||
|
||||
voice_settings = {
|
||||
"stability": stability,
|
||||
"similarity_boost": similarity_boost,
|
||||
"speed": speed,
|
||||
"style": style,
|
||||
"use_speaker_boost": use_speaker_boost
|
||||
}
|
||||
if model_id.startswith("eleven_v4"):
|
||||
voice_settings = {
|
||||
"stability": stability,
|
||||
"similarity_boost": similarity_boost
|
||||
}
|
||||
else:
|
||||
voice_settings = {
|
||||
"stability": stability,
|
||||
"similarity_boost": similarity_boost,
|
||||
"speed": speed,
|
||||
"style": style,
|
||||
"use_speaker_boost": use_speaker_boost
|
||||
}
|
||||
|
||||
data = {
|
||||
"text": text,
|
||||
@@ -101,9 +109,9 @@ class ElevenLabsTTSNode:
|
||||
"output_format": output_format
|
||||
}
|
||||
|
||||
if model_id == "eleven_v3":
|
||||
if model_id.startswith("eleven_v4") or model_id == "eleven_v3":
|
||||
if previous_text.strip() or next_text.strip():
|
||||
print("Providing previous_text or next_text is not yet supported with the 'eleven_v3' model. Ignoring these inputs.")
|
||||
print(f"Providing previous_text or next_text is not supported with '{model_id}'. Ignoring these inputs.")
|
||||
else:
|
||||
if previous_text.strip():
|
||||
data["previous_text"] = previous_text
|
||||
@@ -130,7 +138,7 @@ class ElevenLabsTTSNode:
|
||||
|
||||
@classmethod
|
||||
def IS_CHANGED(cls, **kwargs):
|
||||
return f"{kwargs.get('text', '')}-{kwargs.get('voice_id', '')}-{kwargs.get('seed', 40)}"
|
||||
return f"{kwargs.get('text', '')}-{kwargs.get('voice_id', '')}-{kwargs.get('seed', 40)}-{kwargs.get('model_id', '')}"
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"ElevenLabsTTSNode": ElevenLabsTTSNode}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"ElevenLabsTTSNode": "ElevenLabs TTS"}
|
||||
@@ -16,18 +16,13 @@ class GeminiDiarisationAPI:
|
||||
"audio": ("AUDIO",),
|
||||
"num_speakers": ("INT", {"default": 2, "min": 1, "max": 10, "step": 1}),
|
||||
"model": ([
|
||||
"gemini-3.8-flash",
|
||||
"gemini-3.6-flash",
|
||||
"gemini-3.5-flash",
|
||||
"gemini-3.5-flash-lite",
|
||||
"gemini-3.1-pro-preview",
|
||||
"gemini-3.1-flash-lite",
|
||||
"gemini-3-flash-preview",
|
||||
"gemini-2.5-flash",
|
||||
"gemini-2.5-pro",
|
||||
"gemini-2.5-flash-lite",
|
||||
"gemini-flash-latest",
|
||||
"gemini-flash-lite-latest"
|
||||
], {"default": "gemini-3.5-flash"}),
|
||||
], {"default": "gemini-3.8-flash"}),
|
||||
"api_key": ("STRING", {"default": "", "multiline": False, "tooltip": "Directly put Gemini API key or .env variable name (GEMINI_API_KEY)"}),
|
||||
"seed": ("INT", {"default": 69, "min": 0, "max": 2147483646, "step": 1}),
|
||||
"temperature": ("FLOAT", {"default": 0.2, "min": 0.0, "max": 2.0, "step": 0.1})
|
||||
|
||||
+2
-7
@@ -15,18 +15,13 @@ class GeminiChatNode:
|
||||
"prompt": ("STRING", {"multiline": True}),
|
||||
"api_key": ("STRING", {"default": "", "multiline": False, "tooltip": "Directly put Gemini API key or .env variable name (GEMINI_API_KEY)"}),
|
||||
"model": ([
|
||||
"gemini-3.8-flash",
|
||||
"gemini-3.6-flash",
|
||||
"gemini-3.5-flash",
|
||||
"gemini-3.5-flash-lite",
|
||||
"gemini-3.1-pro-preview",
|
||||
"gemini-3.1-flash-lite",
|
||||
"gemini-3-flash-preview",
|
||||
"gemini-2.5-flash",
|
||||
"gemini-2.5-pro",
|
||||
"gemini-2.5-flash-lite",
|
||||
"gemini-flash-latest",
|
||||
"gemini-flash-lite-latest"
|
||||
], {"default": "gemini-3.5-flash"}),
|
||||
], {"default": "gemini-3.8-flash"}),
|
||||
"temperature": ("FLOAT", {"default": 0.2, "min": 0.0, "max": 2.0, "step": 0.1}),
|
||||
"top_p": ("FLOAT", {"default": 0.95, "min": 0.0, "max": 1.0, "step": 0.01}),
|
||||
"thinking": ("BOOLEAN", {"default": False}),
|
||||
|
||||
+1
-1
@@ -97,4 +97,4 @@ class GeminiTTSNode:
|
||||
return f"{kwargs.get('text', '')}-{kwargs.get('voice_id', '')}-{kwargs.get('temperature', 1.0)}-{kwargs.get('model', '')}-{kwargs.get('seed', 69)}-{kwargs.get('audio_profile', '')}-{kwargs.get('style', '')}-{kwargs.get('pace', '')}-{kwargs.get('accent', '')}-{kwargs.get('scene', '')}"
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"GeminiTTSNode": GeminiTTSNode}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"GeminiTTSNode": "Gemini TTS"}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"GeminiTTSNode": "Gemini TTS (Legacy)"}
|
||||
+132
@@ -0,0 +1,132 @@
|
||||
import os
|
||||
import io
|
||||
import wave
|
||||
import torch
|
||||
from google import genai
|
||||
from google.genai import types
|
||||
|
||||
class GeminiTTS3Node:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"text": ("STRING", {
|
||||
"multiline": True,
|
||||
"default": "",
|
||||
"tooltip": "Transcript text. Supports inline tags: <argh>, <breath>, <cackle>, <cheer>, <chuckle>, <chuckles>, <cough>, <cry>, <exhales>, <gasp>, <giggle>, <groan>, <growl>, <grr>, <grunt>, <heavy breath>, <hiss>, <laugh>, <laughter>, <long pause>, <moan>, <pant>, <pff>, <phew>, <scream>, <short pause>, <shout>, <shriek>, <sigh>, <sighs>, <sneeze>, <snicker>, <snort>, <sob>, <throat-clearing>, <tsk>, <whimper>, <whispering>, <whispers>, <yawn>"
|
||||
}),
|
||||
"voice": ("STRING", {
|
||||
"default": "Bodi",
|
||||
"multiline": False,
|
||||
"tooltip": "Voice name from AI Studio (e.g. Bodi, Lumi, Sola, Koda, Nika, Tavi, Kira, Ludo, or legacy celestial names like Puck, Zephyr)"
|
||||
}),
|
||||
"model": ([
|
||||
"gemini-3.8-flash-tts",
|
||||
"gemini-3.8-flash-lite-tts",
|
||||
], {"default": "gemini-3.8-flash-tts"}),
|
||||
"api_key": ("STRING", {"default": "", "multiline": False, "tooltip": "Directly put Gemini API key or .env variable name (GEMINI_API_KEY)"}),
|
||||
"seed": ("INT", {"default": 69, "min": -1, "max": 2147483646, "step": 1}),
|
||||
"temperature": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 2.0, "step": 0.01}),
|
||||
},
|
||||
"optional": {
|
||||
"style": ("STRING", {
|
||||
"default": "",
|
||||
"multiline": True,
|
||||
"tooltip": "Acting style passed to speech_metadata (e.g. 'Bright, sunny morning greeting with a subtle vocal smile and crisp pacing.')"
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("AUDIO",)
|
||||
RETURN_NAMES = ("audio",)
|
||||
FUNCTION = "generate_speech"
|
||||
CATEGORY = "audio/generation"
|
||||
|
||||
def parse_audio_rate(self, mime_type: str) -> int:
|
||||
rate = 24000
|
||||
if not mime_type:
|
||||
return rate
|
||||
for param in mime_type.split(";"):
|
||||
param = param.strip()
|
||||
if param.lower().startswith("rate="):
|
||||
try:
|
||||
rate = int(param.split("=", 1)[1])
|
||||
except (ValueError, IndexError):
|
||||
pass
|
||||
return rate
|
||||
|
||||
def generate_speech(self, text, voice, model, api_key, seed, temperature, style=""):
|
||||
clean_text = text.strip()
|
||||
if not clean_text:
|
||||
raise ValueError("Text input cannot be empty.")
|
||||
|
||||
key = os.environ.get(api_key.strip(), api_key.strip()) or os.environ.get("GEMINI_API_KEY")
|
||||
if not key:
|
||||
raise ValueError("No API key provided.")
|
||||
|
||||
client = genai.Client(api_key=key)
|
||||
|
||||
transcript = clean_text if clean_text.startswith("## Transcript:") else f"## Transcript:\n{clean_text}"
|
||||
|
||||
metadata = None
|
||||
if style.strip():
|
||||
style_str = style.strip()
|
||||
if not style_str.lower().startswith("style:"):
|
||||
style_str = f"Style: {style_str}"
|
||||
metadata = types.SpeechMetadata(style=style_str)
|
||||
|
||||
part = types.Part(text=transcript, speech_metadata=metadata) if metadata else types.Part(text=transcript)
|
||||
|
||||
config = types.GenerateContentConfig(
|
||||
temperature=temperature,
|
||||
seed=seed,
|
||||
response_modalities=["audio"],
|
||||
speech_config=types.SpeechConfig(
|
||||
voice_config=types.VoiceConfig(
|
||||
prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name=voice.strip())
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
audio_data = bytearray()
|
||||
mime_type = ""
|
||||
|
||||
try:
|
||||
for chunk in client.models.generate_content_stream(
|
||||
model=model,
|
||||
contents=[types.Content(role="user", parts=[part])],
|
||||
config=config,
|
||||
):
|
||||
if chunk.parts is None:
|
||||
continue
|
||||
if chunk.parts[0].inline_data and chunk.parts[0].inline_data.data:
|
||||
inline_data = chunk.parts[0].inline_data
|
||||
audio_data.extend(inline_data.data)
|
||||
mime_type = inline_data.mime_type
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"Gemini API Error: {str(e)}")
|
||||
|
||||
if not audio_data:
|
||||
raise ValueError("API returned a response, but it contained no audio data.")
|
||||
|
||||
sample_rate = self.parse_audio_rate(mime_type)
|
||||
|
||||
if audio_data.startswith(b"RIFF"):
|
||||
with wave.open(io.BytesIO(audio_data), "rb") as wf:
|
||||
raw_pcm = wf.readframes(wf.getnframes())
|
||||
sample_rate = wf.getframerate()
|
||||
waveform = torch.frombuffer(bytearray(raw_pcm), dtype=torch.int16)
|
||||
else:
|
||||
waveform = torch.frombuffer(bytearray(audio_data), dtype=torch.int16)
|
||||
|
||||
waveform = waveform.to(torch.float32) / 32768.0
|
||||
waveform = waveform.unsqueeze(0).unsqueeze(0)
|
||||
|
||||
return ({"waveform": waveform, "sample_rate": sample_rate},)
|
||||
|
||||
@classmethod
|
||||
def IS_CHANGED(cls, **kwargs):
|
||||
return f"{kwargs.get('text', '')}-{kwargs.get('voice', '')}-{kwargs.get('model', '')}-{kwargs.get('seed', 69)}-{kwargs.get('temperature', 1.0)}-{kwargs.get('style', '')}"
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"GeminiTTS3Node": GeminiTTS3Node}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"GeminiTTS3Node": "Gemini TTS"}
|
||||
+8
-1
@@ -12,7 +12,14 @@ class GPTImageNode:
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"model": (["gpt-image-1", "gpt-image-1-mini", "gpt-image-1.5", "gpt-image-2"],),
|
||||
"model": ([
|
||||
"gpt-image-2.5-sunburst",
|
||||
"gpt-image-2.5-flare",
|
||||
"gpt-image-2",
|
||||
"gpt-image-1.5",
|
||||
"gpt-image-1",
|
||||
"gpt-image-1-mini"
|
||||
], {"default": "gpt-image-2.5-sunburst"}),
|
||||
"prompt": ("STRING", {"multiline": True}),
|
||||
"api_key": ("STRING", {"multiline": False, "default": "", "tooltip": "Directly put OpenAI API key or .env variable name (OPENAI_API_KEY)"}),
|
||||
"background": (["auto", "transparent", "opaque"], {"default": "auto"}),
|
||||
|
||||
+9
-3
@@ -14,7 +14,13 @@ class NanoBananaNode:
|
||||
"required": {
|
||||
"prompt": ("STRING", {"multiline": True, "default": ""}),
|
||||
"api_key": ("STRING", {"multiline": False, "default": "", "tooltip": "Directly put Gemini API key or .env variable name (GEMINI_API_KEY)"}),
|
||||
"model": (["gemini-3-pro-image", "gemini-2.5-flash-image", "gemini-3.1-flash-image", "gemini-3.1-flash-lite-image"]),
|
||||
"model": ([
|
||||
"gemini-nano-banana-2.1",
|
||||
"gemini-3-pro-image",
|
||||
"gemini-3.1-flash-lite-image",
|
||||
"gemini-3.1-flash-image",
|
||||
"gemini-2.5-flash-image",
|
||||
], {"default": "gemini-nano-banana-2.1"}),
|
||||
"aspect_ratio": (["1:1", "2:3", "3:2", "3:4", "4:3", "9:16", "16:9", "21:9"],),
|
||||
"resolution": (["1K", "2K", "4K"], {"default": "1K"}),
|
||||
"temperature": ("FLOAT", {"default": 0.5, "min": 0.0, "max": 1.0, "step": 0.01}),
|
||||
@@ -68,13 +74,13 @@ class NanoBananaNode:
|
||||
|
||||
tools = None
|
||||
if google_search:
|
||||
if "gemini-2.5" in model or "gemini-3.1-flash-lite-image" in model:
|
||||
if "gemini-2.5" in model or "flash-lite" in model:
|
||||
print(f"Ignoring google_search: {model} does not support it.")
|
||||
else:
|
||||
tools = [types.Tool(googleSearch=types.GoogleSearch())]
|
||||
|
||||
img_config_params = {"aspect_ratio": aspect_ratio}
|
||||
if "gemini-3-pro" in model:
|
||||
if "gemini-2.5" not in model and "flash-lite" not in model:
|
||||
img_config_params["image_size"] = resolution
|
||||
|
||||
config = types.GenerateContentConfig(
|
||||
|
||||
+23
-3
@@ -13,9 +13,27 @@ class OpenAILLMNode:
|
||||
return {
|
||||
"required": {
|
||||
"prompt": ("STRING", {"multiline": True, "default": ""}),
|
||||
"model": (["gpt-5.4", "gpt-5.4-pro", "gpt-4.1", "gpt-4.1-mini", "gpt-5", "gpt-5.2", "gpt-5-mini", "gpt-5-nano", "gpt-5.2-pro", "o1", "o3-mini"],),
|
||||
"model": ([
|
||||
"gpt-6",
|
||||
"gpt-6.1-sol",
|
||||
"gpt-6-luna",
|
||||
"gpt-5.6-sol",
|
||||
"gpt-5.6-terra",
|
||||
"gpt-5.6-luna",
|
||||
"gpt-5.5",
|
||||
"gpt-5.5-pro",
|
||||
"gpt-5.4",
|
||||
"gpt-5.4-pro",
|
||||
"gpt-5.2",
|
||||
"gpt-5.2-pro",
|
||||
"gpt-5",
|
||||
"gpt-5-mini",
|
||||
"gpt-5-nano",
|
||||
"gpt-4.1",
|
||||
"gpt-4.1-mini",
|
||||
], {"default": "gpt-6"}),
|
||||
"temperature": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 2.0, "step": 0.01}),
|
||||
"reasoning_effort": (["low", "medium", "high"],),
|
||||
"reasoning_effort": (["none", "low", "medium", "high"], {"default": "medium"}),
|
||||
"api_key": ("STRING", {"multiline": False, "default": "", "tooltip": "Directly put OpenAI API key or .env variable name (OPENAI_API_KEY)"}),
|
||||
"max_output_tokens": ("INT", {"default": 16384, "min": 1, "max": 32768, "step": 1})
|
||||
},
|
||||
@@ -61,8 +79,10 @@ class OpenAILLMNode:
|
||||
"max_output_tokens": max_output_tokens
|
||||
}
|
||||
|
||||
# Pass reasoning effort only if supported and not set to "none"
|
||||
if not model.startswith("gpt-4"):
|
||||
request_params["reasoning"] = {"effort": reasoning_effort}
|
||||
if reasoning_effort != "none":
|
||||
request_params["reasoning"] = {"effort": reasoning_effort}
|
||||
else:
|
||||
print(f"Skipping reasoning parameter for {model} (not supported)")
|
||||
|
||||
|
||||
+16
-8
@@ -15,18 +15,22 @@ class OpenAITTSNode:
|
||||
"gpt-4o-mini-tts",
|
||||
"tts-1",
|
||||
"tts-1-hd"
|
||||
],),
|
||||
], {"default": "gpt-4o-mini-tts"}),
|
||||
"voice": ([
|
||||
"alloy",
|
||||
"ash",
|
||||
"ballad",
|
||||
"cedar",
|
||||
"coral",
|
||||
"echo",
|
||||
"fable",
|
||||
"onyx",
|
||||
"marin",
|
||||
"nova",
|
||||
"onyx",
|
||||
"sage",
|
||||
"shimmer"
|
||||
],),
|
||||
"shimmer",
|
||||
"verse"
|
||||
], {"default": "alloy"}),
|
||||
"response_format": ([
|
||||
"mp3",
|
||||
"opus",
|
||||
@@ -34,12 +38,16 @@ class OpenAITTSNode:
|
||||
"flac",
|
||||
"wav",
|
||||
"pcm"
|
||||
],),
|
||||
], {"default": "mp3"}),
|
||||
"speed": ("FLOAT", {"default": 1.0, "min": 0.25, "max": 4.0, "step": 0.01}),
|
||||
"api_key": ("STRING", {"multiline": False, "default": "", "tooltip": "Directly put OpenAI API key or .env variable name (OPENAI_API_KEY)"}),
|
||||
},
|
||||
"optional": {
|
||||
"instructions": ("STRING", {"multiline": True, "default": ""}),
|
||||
"instructions": ("STRING", {
|
||||
"multiline": True,
|
||||
"default": "",
|
||||
"tooltip": "Supported on gpt-4o-mini-tts. Natural-language guidance for tone, emotion, or pacing (e.g. 'Whisper excitedly', 'Speak like a calm news anchor')."
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -78,7 +86,7 @@ class OpenAITTSNode:
|
||||
|
||||
# Add instructions only if provided and model supports it
|
||||
if instructions.strip() and model not in ["tts-1", "tts-1-hd"]:
|
||||
data["instructions"] = instructions
|
||||
data["instructions"] = instructions.strip()
|
||||
|
||||
response = requests.post(url, json=data, headers=headers)
|
||||
|
||||
@@ -100,7 +108,7 @@ class OpenAITTSNode:
|
||||
|
||||
@classmethod
|
||||
def IS_CHANGED(cls, **kwargs):
|
||||
return f"{kwargs.get('text', '')}-{kwargs.get('voice', '')}-{kwargs.get('model', '')}"
|
||||
return f"{kwargs.get('text', '')}-{kwargs.get('voice', '')}-{kwargs.get('model', '')}-{kwargs.get('speed', 1.0)}-{kwargs.get('instructions', '')}"
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"OpenAITTSNode": OpenAITTSNode}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"OpenAITTSNode": "OpenAI Text-to-Speech"}
|
||||
+1
-1
@@ -1,7 +1,7 @@
|
||||
[project]
|
||||
name = "externalapi-helpers"
|
||||
description = "ComfyUI nodes for FLUX.1 Kontext, FLUX.2, Gemini, Veo, Imagen, OpenAI GPT, GPT-Image, Sora, xAI Grok, Groq, ElevenLabs, Nano Banana, and Tripo"
|
||||
version = "1.1.7"
|
||||
version = "1.1.8"
|
||||
license = {file = "LICENSE"}
|
||||
# classifiers = [
|
||||
# # For OS-independent nodes (works on all operating systems)
|
||||
|
||||
@@ -1,90 +0,0 @@
|
||||
import time
|
||||
import io
|
||||
import torch
|
||||
import numpy as np
|
||||
import av
|
||||
from PIL import Image
|
||||
import os
|
||||
from openai import OpenAI
|
||||
|
||||
class SoraGen:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"prompt": ("STRING", {"multiline": True, "default": "A calico cat playing a piano on stage"}),
|
||||
"api_key": ("STRING", {"multiline": False, "default": "", "tooltip": "Directly put OpenAI API key or .env variable name (OPENAI_API_KEY)"}),
|
||||
"model": (["sora-2", "sora-2-pro"], {"default": "sora-2"}),
|
||||
"size": (["720x1280", "1280x720", "1024x1792", "1792x1024"], {"default": "1280x720"}),
|
||||
"duration": (["4", "8", "12"], {"default": "4"}),
|
||||
"seed": ("INT", {"default": 69, "min": 1, "max": 2147483646, "step": 1}),
|
||||
},
|
||||
"optional": {"input_image": ("IMAGE",)}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "AUDIO")
|
||||
RETURN_NAMES = ("frames", "audio")
|
||||
FUNCTION = "generate_video"
|
||||
CATEGORY = "video/generation"
|
||||
OUTPUT_IS_LIST = (True, False)
|
||||
|
||||
def generate_video(self, prompt, api_key, model, size, duration, seed, input_image=None):
|
||||
key = os.environ.get(api_key.strip(), api_key.strip()) or os.environ.get("OPENAI_API_KEY")
|
||||
if not key: raise ValueError("No API key provided.")
|
||||
client = OpenAI(api_key=key)
|
||||
api_args = {"prompt": prompt, "model": model, "size": size, "seconds": duration}
|
||||
|
||||
img_buf = None
|
||||
if input_image is not None:
|
||||
img_buf = io.BytesIO()
|
||||
# Convert Tensor (Batch, H, W, C) -> Numpy (H, W, C) -> PIL
|
||||
arr = (input_image.cpu().numpy()[0] * 255).astype(np.uint8)
|
||||
Image.fromarray(arr).save(img_buf, format="JPEG", quality=95)
|
||||
img_buf.seek(0)
|
||||
api_args["input_reference"] = ("ref.jpg", img_buf, "image/jpeg")
|
||||
|
||||
try:
|
||||
job = client.videos.create(**api_args)
|
||||
print(f"Job started: {job.id}")
|
||||
|
||||
while (status := client.videos.retrieve(job.id)).status not in ["completed", "failed"]:
|
||||
time.sleep(4)
|
||||
|
||||
if status.status == "failed":
|
||||
raise Exception(f"API Error: {status.error.message}")
|
||||
|
||||
video_bytes = io.BytesIO(client.videos.download_content(video_id=job.id).read())
|
||||
|
||||
container = av.open(video_bytes)
|
||||
frames = []
|
||||
for frame in container.decode(video=0):
|
||||
img = frame.to_rgb().to_ndarray().astype(np.float32) / 255.0
|
||||
frames.append(torch.from_numpy(img).unsqueeze(0))
|
||||
container.close()
|
||||
|
||||
video_bytes.seek(0)
|
||||
container = av.open(video_bytes)
|
||||
audio = None
|
||||
|
||||
if container.streams.audio:
|
||||
audio_data = [f.to_ndarray() for f in container.decode(audio=0)]
|
||||
if audio_data:
|
||||
waveform = torch.from_numpy(np.concatenate(audio_data, axis=1)).float()
|
||||
# Normalize Audio
|
||||
if audio_data[0].dtype == np.int16: waveform /= 32768.0
|
||||
elif audio_data[0].dtype == np.int32: waveform /= 2147483648.0
|
||||
|
||||
audio = {
|
||||
"waveform": waveform.unsqueeze(0),
|
||||
"sample_rate": container.streams.audio[0].rate
|
||||
}
|
||||
container.close()
|
||||
|
||||
return ([torch.cat(frames, dim=0)], audio)
|
||||
|
||||
finally:
|
||||
if img_buf:
|
||||
img_buf.close()
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"SoraGen": SoraGen}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"SoraGen": "Sora 2 (OpenAI)"}
|
||||
@@ -31,15 +31,19 @@ class VeoVertexVideoGenerator:
|
||||
], {"default": "us-central1"}),
|
||||
"service_account": ("STRING", {"multiline": True, "default": "", "tooltip": "Paste service account JSON content"}),
|
||||
"model": ([
|
||||
"veo-2.0-generate-001", "veo-2.0-generate-exp", "veo-2.0-generate-preview",
|
||||
"veo-3.0-generate-001", "veo-3.0-fast-generate-001",
|
||||
"veo-3.1-generate-001", "veo-3.1-fast-generate-001"
|
||||
], {"default": "veo-3.0-generate-001"}),
|
||||
"veo-3.1-generate-001",
|
||||
"veo-3.1-fast-generate-001",
|
||||
"veo-3.0-generate-001",
|
||||
"veo-3.0-fast-generate-001",
|
||||
"veo-2.0-generate-001",
|
||||
"veo-2.0-generate-exp",
|
||||
"veo-2.0-generate-preview",
|
||||
], {"default": "veo-3.1-generate-001"}),
|
||||
"resolution": (["720p", "1080p"], {"default": "720p"}),
|
||||
"aspect_ratio": (["16:9", "9:16"], {"default": "16:9"}),
|
||||
"duration_seconds": ("INT", {"default": 4, "min": 4, "max": 8, "step": 1}),
|
||||
"duration_seconds": ("INT", {"default": 8, "min": 4, "max": 8, "step": 1}),
|
||||
"seed": ("INT", {"default": 69, "min": 1, "max": 2147483646, "step": 1}),
|
||||
"generate_audio": ("BOOLEAN", {"default": False}),
|
||||
"generate_audio": ("BOOLEAN", {"default": True}),
|
||||
"fps": (["24"], {"default": "24"}),
|
||||
},
|
||||
"optional": {
|
||||
@@ -66,16 +70,29 @@ class VeoVertexVideoGenerator:
|
||||
|
||||
try:
|
||||
client = genai.Client(vertexai=True, project=project_id, location=location)
|
||||
|
||||
# Veo 2.0 requires minimum 5s duration and does not support audio generation
|
||||
dur = duration_seconds
|
||||
is_veo2 = "veo-2.0" in model
|
||||
if is_veo2 and dur < 5:
|
||||
dur = 5
|
||||
|
||||
should_gen_audio = False if is_veo2 else generate_audio
|
||||
|
||||
config = types.GenerateVideosConfig(
|
||||
resolution=resolution,
|
||||
aspect_ratio=aspect_ratio,
|
||||
duration_seconds=duration_seconds,
|
||||
generate_audio=generate_audio,
|
||||
fps=int(fps),
|
||||
seed=seed if seed != -1 else None,
|
||||
negative_prompt=negative_prompt.strip() if negative_prompt else None
|
||||
)
|
||||
config_kwargs = {
|
||||
"aspect_ratio": aspect_ratio,
|
||||
"duration_seconds": dur,
|
||||
"fps": int(fps),
|
||||
"seed": seed if seed != -1 else None,
|
||||
"negative_prompt": negative_prompt.strip() if negative_prompt else None
|
||||
}
|
||||
|
||||
# Only Veo 3.x accepts resolution and audio generation flags
|
||||
if not is_veo2:
|
||||
config_kwargs["resolution"] = resolution
|
||||
config_kwargs["generate_audio"] = should_gen_audio
|
||||
|
||||
config = types.GenerateVideosConfig(**config_kwargs)
|
||||
|
||||
# Helper: Tensor -> Bytes
|
||||
def tensor_to_bytes(t):
|
||||
@@ -112,7 +129,7 @@ class VeoVertexVideoGenerator:
|
||||
|
||||
# Decode Audio
|
||||
audio = None
|
||||
if generate_audio:
|
||||
if should_gen_audio:
|
||||
video_bytes.seek(0)
|
||||
container = av.open(video_bytes)
|
||||
if container.streams.audio:
|
||||
|
||||
+29
-12
@@ -14,9 +14,15 @@ class VeoGeminiVideoGenerator:
|
||||
return {
|
||||
"required": {
|
||||
"prompt": ("STRING", {"multiline": True, "default": "a cat reading a book"}),
|
||||
"model": (["veo-2.0-generate-001"], {"default": "veo-2.0-generate-001"}),
|
||||
"model": ([
|
||||
"veo-3.1-generate-preview",
|
||||
"veo-3.1-fast-generate-preview",
|
||||
"veo-3.1-lite",
|
||||
"veo-2.0-generate-001",
|
||||
], {"default": "veo-3.1-generate-preview"}),
|
||||
"aspect_ratio": (["16:9", "9:16"], {"default": "16:9"}),
|
||||
"duration_seconds": ("INT", {"default": 8, "min": 5, "max": 8, "step": 1}),
|
||||
"resolution": (["720p", "1080p"], {"default": "720p"}),
|
||||
"duration_seconds": ("INT", {"default": 8, "min": 4, "max": 8, "step": 1}),
|
||||
"api_key": ("STRING", {"default": "", "multiline": False, "tooltip": "Directly put Gemini API key or .env variable name (GEMINI_API_KEY)"}),
|
||||
"seed": ("INT", {"default": 69, "min": -1, "max": 2147483646, "step": 1}),
|
||||
},
|
||||
@@ -32,7 +38,7 @@ class VeoGeminiVideoGenerator:
|
||||
CATEGORY = "video/generation"
|
||||
OUTPUT_IS_LIST = (True,)
|
||||
|
||||
def generate_video(self, seed, prompt, model, aspect_ratio, duration_seconds, api_key,
|
||||
def generate_video(self, seed, prompt, model, aspect_ratio, resolution, duration_seconds, api_key,
|
||||
negative_prompt=None, image=None):
|
||||
|
||||
# 1. Setup Client
|
||||
@@ -40,24 +46,35 @@ class VeoGeminiVideoGenerator:
|
||||
if not key: raise ValueError("API Key required")
|
||||
client = genai.Client(http_options={"api_version": "v1beta"}, api_key=key)
|
||||
|
||||
# 2. Prepare Config (Seed used for ComfyUI caching only)
|
||||
# 2. Duration compatibility clamp (Veo 2.0 requires minimum 5s)
|
||||
dur = duration_seconds
|
||||
if "veo-2.0" in model and dur < 5:
|
||||
dur = 5
|
||||
|
||||
# 3. Prepare Config
|
||||
config_params = {
|
||||
"aspect_ratio": aspect_ratio,
|
||||
"duration_seconds": dur,
|
||||
"negative_prompt": negative_prompt.strip() if negative_prompt else None,
|
||||
}
|
||||
|
||||
# Resolution is supported on Veo 3.x
|
||||
if "veo-3" in model:
|
||||
config_params["resolution"] = resolution
|
||||
|
||||
gen_kwargs = {
|
||||
"model": model,
|
||||
"prompt": prompt,
|
||||
"config": types.GenerateVideosConfig(
|
||||
aspect_ratio=aspect_ratio,
|
||||
duration_seconds=duration_seconds,
|
||||
negative_prompt=negative_prompt.strip() if negative_prompt else None,
|
||||
)
|
||||
"config": types.GenerateVideosConfig(**config_params)
|
||||
}
|
||||
|
||||
# 3. Handle Image (In-Memory, ComfyUI Tensors are always [B,H,W,C])
|
||||
# 4. Handle Image Input (Image-to-Video)
|
||||
if image is not None:
|
||||
buf = io.BytesIO()
|
||||
Image.fromarray((image[0].cpu().numpy() * 255).astype(np.uint8)).save(buf, format="PNG")
|
||||
gen_kwargs["image"] = types.Image(image_bytes=buf.getvalue(), mime_type="image/png")
|
||||
|
||||
# 4. Generate & Poll
|
||||
# 5. Generate & Poll
|
||||
op = client.models.generate_videos(**gen_kwargs)
|
||||
print(f"Gemini Veo Operation: {op.name}")
|
||||
|
||||
@@ -68,7 +85,7 @@ class VeoGeminiVideoGenerator:
|
||||
if not op.response or not op.response.generated_videos:
|
||||
raise Exception("No videos generated")
|
||||
|
||||
# 5. Download & Decode (Direct Memory Stream)
|
||||
# 6. Download & Decode Video Frames
|
||||
video_bytes = io.BytesIO(client.files.download(file=op.response.generated_videos[0].video))
|
||||
|
||||
container = av.open(video_bytes)
|
||||
|
||||
Reference in New Issue
Block a user