add minimax video generate node

README.md

@@ -12,6 +12,12 @@ A collection of ComfyUI custom nodes that integrate with MiniMax API services.
 - **Voice Cloning**: Clone voices from audio samples
 - **Load Audio**: Load and preview audio files for voice cloning
 
+### Video Nodes (JM-MiniMax-API/Video)
+
+- **Video Generation**: Generate videos using MiniMax's unified video generation API (supports text-to-video, image-to-video, and subject-referenced video)
+- **Check Video Status**: Check the status of video generation tasks
+- **Download Video**: Download generated videos to local storage
+
 ## Installation
 
 1. Clone this repository to your ComfyUI custom_nodes folder:
@@ -99,6 +105,60 @@ This node uses MiniMax's API to convert text to speech.
 4. Enter text and configure other parameters in the Text to Speech node
 5. Run the workflow to generate speech using the cloned voice
 
+### Video Generation Node
+
+This node uses MiniMax's unified video generation API to create videos from text prompts, images, or subject references.
+
+#### Input Parameters:
+- **api_key**: MiniMax API key
+- **model**: Video generation model selection:
+  - **T2V-01-Director**, **T2V-01**: Text-to-video models (require text prompt)
+  - **I2V-01-Director**, **I2V-01**, **I2V-01-live**: Image-to-video models (require image input)
+  - **S2V-01**: Subject-referenced video model
+- **prompt**: Video generation description (max 2000 characters, supports camera movement instructions)
+- **prompt_optimizer**: Whether to auto-optimize prompt for better quality (true/false)
+- **image** (optional): Image input for I2V models (required for I2V-01-Director, I2V-01, I2V-01-live)
+- **callback_url** (optional): URL for status update callbacks
+
+#### Model Usage Guidelines:
+- **Text-to-Video (T2V models)**: Only requires a text prompt. Image input is optional.
+- **Image-to-Video (I2V models)**: Requires both image input and optionally a text prompt. If prompt is empty, the model will auto-generate video content based on the image.
+- **Subject-Referenced (S2V models)**: Currently S2V-01 model for subject-referenced video generation.
+
+#### Camera Movement Instructions:
+When using T2V-01-Director or I2V-01-Director models, you can use movement instructions in the prompt:
+- Movement: [左移], [右移], [推进], [拉远], [上升], [下降]
+- Rotation: [左摇], [右摇], [上摇], [下摇]
+- Zoom: [变焦推近], [变焦拉远]
+- Other: [晃动], [跟随], [固定]
+
+#### Image Requirements (for I2V models):
+- Format: JPG/JPEG/PNG
+- Aspect ratio: between 2:5 and 5:2
+- Short side: minimum 300px
+- File size: maximum 20MB
+
+#### Output:
+- **task_id**: Task ID for the video generation job
+
+## Video Workflow Examples
+
+### Text-to-Video Workflow:
+1. Use **Video Generation** node with T2V-01-Director or T2V-01 model
+2. Enter your API key and write a detailed text prompt
+3. Run to get a task_id
+4. Use **Check Video Status** node to monitor progress
+5. Once status is "success", use **Download Video** node to save the video
+
+### Image-to-Video Workflow:
+1. Load an image into ComfyUI (using Load Image node)
+2. Connect it to the **Video Generation** node
+3. Select an I2V model (I2V-01-Director recommended)
+4. Enter API key and optionally write a prompt with movement instructions
+5. Run to get a task_id
+6. Use **Check Video Status** node to monitor progress
+7. Once status is "success", use **Download Video** node to save the video
+
 ## License
 
 MIT License

README_CN.md

@@ -12,6 +12,91 @@
 - **声音克隆**: 从音频样本中克隆声音
 - **加载音频**: 加载和预览用于声音克隆的音频文件
 
+### 视频节点 (JM-MiniMax-API/Video)
+
+- **视频生成**: 使用 MiniMax 的统一视频生成 API 生成视频（支持文生视频、图生视频和视频主体参考）
+- **检查视频状态**: 检查视频生成任务的状态
+- **下载视频**: 将生成的视频下载到本地存储
+
+### 视频生成节点
+
+此节点使用 MiniMax 的统一视频生成 API 从文本提示词、图像或主体参考创建视频。
+
+#### 输入参数：
+- **api_key**: MiniMax API 密钥
+- **model**: 视频生成模型选择：
+  - **T2V-01-Director**, **T2V-01**: 文生视频模型（需要文本提示词）
+  - **I2V-01-Director**, **I2V-01**, **I2V-01-live**: 图生视频模型（需要图像输入）
+  - **S2V-01**: 视频主体参考模型
+- **prompt**: 视频生成描述（最多 2000 字符，支持运镜指令）
+- **prompt_optimizer**: 是否自动优化提示词以获得更好质量（true/false）
+- **image**（可选）: I2V 模型的图像输入（I2V-01-Director, I2V-01, I2V-01-live 必需）
+- **callback_url**（可选）: 状态更新回调 URL
+
+#### 模型使用指南：
+- **文生视频（T2V 模型）**: 只需要文本提示词。图像输入是可选的。
+- **图生视频（I2V 模型）**: 需要图像输入，文本提示词可选。如果提示词为空，模型将根据图像自动生成视频内容。
+- **视频主体参考（S2V 模型）**: 目前支持 S2V-01 模型进行主体参考视频生成。
+
+#### 运镜指令：
+使用 T2V-01-Director 或 I2V-01-Director 模型时，可以在提示词中使用运镜指令：
+- 移动：[左移], [右移], [推进], [拉远], [上升], [下降]
+- 旋转：[左摇], [右摇], [上摇], [下摇]
+- 变焦：[变焦推近], [变焦拉远]
+- 其他：[晃动], [跟随], [固定]
+
+#### 图像要求（适用于 I2V 模型）：
+- 格式：JPG/JPEG/PNG
+- 长宽比：2:5 到 5:2 之间
+- 短边：最少 300px
+- 文件大小：最大 20MB
+
+#### 输出：
+- **task_id**: 视频生成任务的任务 ID
+
+### 检查视频状态节点
+
+此节点检查视频生成任务的状态。
+
+#### 输入参数：
+- **api_key**: MiniMax API 密钥
+- **task_id**: 来自视频生成节点的任务 ID
+
+#### 输出：
+- **status**: 任务状态（processing, success, failed）
+- **file_id**: 生成视频的文件 ID
+- **video_url**: 生成视频的下载 URL
+- **cover_image_url**: 视频封面图像的 URL
+
+### 下载视频节点
+
+此节点将生成的视频从 URL 下载到本地存储。
+
+#### 输入参数：
+- **video_url**: 来自检查视频状态节点的视频 URL
+- **filename_prefix**: 下载视频文件的前缀
+
+#### 输出：
+- **video_path**: 下载视频文件的绝对路径
+
+## 视频工作流示例
+
+### 文生视频工作流：
+1. 使用**视频生成**节点选择 T2V-01-Director 或 T2V-01 模型
+2. 输入您的 API 密钥并编写详细的文本提示词
+3. 运行以获取 task_id
+4. 使用**检查视频状态**节点监控进度
+5. 状态为 "success" 后，使用**下载视频**节点保存视频
+
+### 图生视频工作流：
+1. 在 ComfyUI 中加载图像（使用加载图像节点）
+2. 将其连接到**视频生成**节点
+3. 选择 I2V 模型（推荐 I2V-01-Director）
+4. 输入 API 密钥，可选择性地编写带有运镜指令的提示词
+5. 运行以获取 task_id
+6. 使用**检查视频状态**节点监控进度
+7. 状态为 "success" 后，使用**下载视频**节点保存视频
+
 ## 安装
 
 1. 克隆此仓库到你的 ComfyUI custom_nodes 文件夹:

__init__.py

@@ -1,17 +1,26 @@
 from .nodes.text_to_speech import TextToSpeech
 from .nodes.voice_cloning import VoiceCloning
 from .nodes.load_audio import JM_LoadAudio
+from .nodes.video_generation import MiniMaxVideoGeneration
+from .nodes.check_video_status import CheckVideoStatus
+from .nodes.download_video import DownloadVideo
 
 NODE_CLASS_MAPPINGS = {
     "JM-MiniMax-API/text-to-speech": TextToSpeech,
     "JM-MiniMax-API/voice-cloning": VoiceCloning,
-    "JM-MiniMax-API/load-audio": JM_LoadAudio
+    "JM-MiniMax-API/load-audio": JM_LoadAudio,
+    "JM-MiniMax-API/video-generation": MiniMaxVideoGeneration,
+    "JM-MiniMax-API/check-video-status": CheckVideoStatus,
+    "JM-MiniMax-API/download-video": DownloadVideo
 }
 
 NODE_DISPLAY_NAME_MAPPINGS = {
     "JM-MiniMax-API/text-to-speech": "MiniMax Text to Speech",
     "JM-MiniMax-API/voice-cloning": "MiniMax Voice Cloning",
-    "JM-MiniMax-API/load-audio": "Load Audio"
+    "JM-MiniMax-API/load-audio": "Load Audio",
+    "JM-MiniMax-API/video-generation": "MiniMax Video Generation",
+    "JM-MiniMax-API/check-video-status": "Check Video Status",
+    "JM-MiniMax-API/download-video": "Download Video"
 }
 
 # Tell ComfyUI where to find web extensions

nodes/check_video_status.py

@@ -0,0 +1,105 @@
+import os
+import json
+import requests
+import folder_paths
+
+class CheckVideoStatus:
+    """
+    MiniMax Check Video Generation Status node for ComfyUI
+    """
+    def __init__(self):
+        self.api_url = "https://api.minimax.chat/v1/query/video_generation"
+        
+    @classmethod
+    def INPUT_TYPES(cls):
+        return {
+            "required": {
+                "api_key": ("STRING", {"multiline": False}),
+                "task_id": ("STRING", {"multiline": False, "placeholder": "Task ID from video generation"}),
+            }
+        }
+
+    RETURN_TYPES = ("STRING", "STRING", "STRING", "STRING")
+    RETURN_NAMES = ("status", "file_id", "video_url", "cover_image_url")
+    FUNCTION = "check_status"
+    CATEGORY = "JM-MiniMax-API/Video"
+
+    def check_status(self, api_key, task_id):
+        if not api_key or not task_id:
+            raise ValueError("API Key and Task ID must be provided")
+
+        try:
+            # Prepare API request
+            headers = {
+                'authorization': f'Bearer {api_key}',
+                'Content-Type': 'application/json'
+            }
+            
+            payload = {
+                "task_id": task_id
+            }
+            
+            print(f"Checking status for task_id: {task_id}")
+            
+            # Make API request
+            response = requests.post(
+                self.api_url,
+                headers=headers,
+                data=json.dumps(payload),
+                timeout=30
+            )
+            
+            print(f"Response status code: {response.status_code}")
+            
+            try:
+                response_data = response.json()
+                print(f"Response data: {json.dumps(response_data, indent=2)}")
+            except json.JSONDecodeError:
+                print(f"Raw response content: {response.content}")
+                raise RuntimeError("Failed to decode JSON response")
+            
+            # Check for API errors
+            base_resp = response_data.get("base_resp", {})
+            status_code = base_resp.get("status_code")
+            status_msg = base_resp.get("status_msg", "Unknown error")
+            
+            if status_code != 0:
+                error_messages = {
+                    1002: "Rate limit exceeded, please try again later",
+                    1004: "Authentication failed, please check your API key",
+                    1008: "Insufficient account balance",
+                    2013: "Invalid parameters, please check your input",
+                    2049: "Invalid API key, please check your API key"
+                }
+                error_msg = error_messages.get(status_code, f"API Error {status_code}: {status_msg}")
+                raise RuntimeError(error_msg)
+            
+            # Extract task information
+            status = response_data.get("status", "unknown")
+            file_id = response_data.get("file_id", "")
+            video_url = response_data.get("video_url", "")
+            cover_image_url = response_data.get("cover_image_url", "")
+            
+            # Additional information for logging
+            if "video_width" in response_data and "video_height" in response_data:
+                print(f"Video dimensions: {response_data['video_width']}x{response_data['video_height']}")
+            
+            if "video_length" in response_data:
+                print(f"Video length: {response_data['video_length']}s")
+            
+            print(f"Status: {status}")
+            if file_id:
+                print(f"File ID: {file_id}")
+            if video_url:
+                print(f"Video URL: {video_url}")
+            if cover_image_url:
+                print(f"Cover Image URL: {cover_image_url}")
+            
+            return (status, file_id, video_url, cover_image_url)
+
+        except requests.exceptions.RequestException as e:
+            print(f"Request error: {str(e)}")
+            raise RuntimeError(f"Failed to connect to MiniMax API: {str(e)}")
+        except Exception as e:
+            print(f"Unexpected error: {str(e)}")
+            raise RuntimeError(f"Status check failed: {str(e)}") 

nodes/download_video.py

@@ -0,0 +1,103 @@
+import os
+import requests
+import time
+import folder_paths
+from urllib.parse import urlparse
+
+class DownloadVideo:
+    """
+    Download Video from URL node for ComfyUI
+    """
+    def __init__(self):
+        pass
+        
+    @classmethod
+    def INPUT_TYPES(cls):
+        return {
+            "required": {
+                "video_url": ("STRING", {"multiline": False, "placeholder": "Video URL from status check"}),
+                "filename_prefix": ("STRING", {"default": "minimax_video", "multiline": False}),
+            }
+        }
+
+    RETURN_TYPES = ("STRING",)
+    RETURN_NAMES = ("video_path",)
+    FUNCTION = "download_video"
+    CATEGORY = "JM-MiniMax-API/Video"
+
+    def download_video(self, video_url, filename_prefix):
+        if not video_url or not video_url.strip():
+            raise ValueError("Video URL must be provided")
+        
+        video_url = video_url.strip()
+        
+        try:
+            # Validate URL
+            parsed_url = urlparse(video_url)
+            if not parsed_url.scheme or not parsed_url.netloc:
+                raise ValueError("Invalid video URL provided")
+            
+            print(f"Downloading video from: {video_url}")
+            
+            # Create output directory
+            output_dir = folder_paths.get_output_directory()
+            os.makedirs(output_dir, exist_ok=True)
+            
+            # Generate timestamp for filename
+            timestamp = time.strftime("%Y%m%d-%H%M%S")
+            
+            # Clean filename prefix (remove any invalid characters)
+            clean_prefix = "".join(c for c in filename_prefix if c.isalnum() or c in ('-', '_'))
+            if not clean_prefix:
+                clean_prefix = "minimax_video"
+            
+            # Try to get file extension from URL or default to mp4
+            url_path = parsed_url.path
+            if url_path and '.' in url_path:
+                file_extension = url_path.split('.')[-1].lower()
+                # Validate extension
+                if file_extension not in ['mp4', 'avi', 'mov', 'mkv', 'webm']:
+                    file_extension = 'mp4'
+            else:
+                file_extension = 'mp4'
+            
+            # Generate filename
+            video_filename = f"{clean_prefix}_{timestamp}.{file_extension}"
+            video_filepath = os.path.join(output_dir, video_filename)
+            
+            # Download video with progress tracking
+            with requests.get(video_url, stream=True, timeout=60) as response:
+                response.raise_for_status()
+                
+                # Get content length for progress tracking
+                content_length = response.headers.get('content-length')
+                if content_length:
+                    total_size = int(content_length)
+                    print(f"Video file size: {total_size / (1024*1024):.1f} MB")
+                else:
+                    total_size = None
+                    print("Video file size: Unknown")
+                
+                downloaded_size = 0
+                with open(video_filepath, 'wb') as f:
+                    for chunk in response.iter_content(chunk_size=8192):
+                        if chunk:
+                            f.write(chunk)
+                            downloaded_size += len(chunk)
+                            
+                            # Print progress every 1MB
+                            if total_size and downloaded_size % (1024*1024) == 0:
+                                progress = (downloaded_size / total_size) * 100
+                                print(f"Download progress: {progress:.1f}%")
+            
+            print(f"Video downloaded successfully to: {video_filepath}")
+            print(f"Final file size: {os.path.getsize(video_filepath) / (1024*1024):.1f} MB")
+            
+            return (os.path.abspath(video_filepath),)
+
+        except requests.exceptions.RequestException as e:
+            print(f"Download error: {str(e)}")
+            raise RuntimeError(f"Failed to download video: {str(e)}")
+        except Exception as e:
+            print(f"Unexpected error: {str(e)}")
+            raise RuntimeError(f"Video download failed: {str(e)}") 

nodes/video_generation.py

@@ -0,0 +1,199 @@
+import os
+import json
+import base64
+import requests
+from PIL import Image
+import folder_paths
+
+class MiniMaxVideoGeneration:
+    """
+    MiniMax Video Generation node for ComfyUI
+    Supports text-to-video, image-to-video, and subject-referenced video generation
+    """
+    def __init__(self):
+        self.api_url = "https://api.minimax.chat/v1/video_generation"
+        
+    @classmethod
+    def INPUT_TYPES(cls):
+        return {
+            "required": {
+                "api_key": ("STRING", {"multiline": False}),
+                "model": (["T2V-01-Director", "T2V-01", "I2V-01-Director", "I2V-01", "I2V-01-live", "S2V-01"], {
+                    "default": "I2V-01-Director"
+                }),
+                "prompt": ("STRING", {
+                    "multiline": True, 
+                    "default": "",
+                    "placeholder": "Describe the video generation (max 2000 characters). Leave empty for I2V models if you want auto-generation."
+                }),
+                "prompt_optimizer": ("BOOLEAN", {"default": True}),
+            },
+            "optional": {
+                "image": ("IMAGE", {
+                    "tooltip": "Required for I2V models (I2V-01-Director, I2V-01, I2V-01-live). Optional for other models."
+                }),
+                "callback_url": ("STRING", {
+                    "multiline": False, 
+                    "default": "",
+                    "placeholder": "Optional: Callback URL for status updates"
+                }),
+            }
+        }
+
+    RETURN_TYPES = ("STRING",)
+    RETURN_NAMES = ("task_id",)
+    FUNCTION = "generate_video"
+    CATEGORY = "JM-MiniMax-API/Video"
+
+    def generate_video(self, api_key, model, prompt, prompt_optimizer, image=None, callback_url=""):
+        if not api_key:
+            raise ValueError("API Key must be provided")
+
+        # Check model requirements
+        i2v_models = ["I2V-01-Director", "I2V-01", "I2V-01-live"]
+        t2v_models = ["T2V-01-Director", "T2V-01"]
+        s2v_models = ["S2V-01"]
+        
+        if model in i2v_models and image is None:
+            raise ValueError(f"Model {model} requires an image input. Please connect an image to the 'image' input.")
+        
+        if model in t2v_models and not prompt.strip():
+            raise ValueError(f"Model {model} requires a text prompt. Please provide a description in the 'prompt' field.")
+
+        try:
+            # Prepare API request
+            headers = {
+                'authorization': f'Bearer {api_key}',
+                'Content-Type': 'application/json'
+            }
+            
+            # Build payload
+            payload = {
+                "model": model,
+                "prompt": prompt,
+                "prompt_optimizer": prompt_optimizer
+            }
+            
+            # Handle image input for I2V models
+            if model in i2v_models and image is not None:
+                # Convert ComfyUI image tensor to PIL Image
+                if len(image.shape) == 4:  # Batch dimension
+                    # Take the first image from batch
+                    image_tensor = image[0]
+                else:
+                    image_tensor = image
+                
+                # Convert from tensor format (H, W, C) with values 0-1 to PIL Image
+                import torch
+                import numpy as np
+                
+                # Convert tensor to numpy array and scale to 0-255
+                if isinstance(image_tensor, torch.Tensor):
+                    image_np = image_tensor.cpu().numpy()
+                else:
+                    image_np = image_tensor
+                    
+                # Ensure values are in range 0-1, then scale to 0-255
+                image_np = np.clip(image_np, 0, 1)
+                image_np = (image_np * 255).astype(np.uint8)
+                
+                # Convert to PIL Image
+                pil_image = Image.fromarray(image_np)
+                
+                # Validate image dimensions
+                width, height = pil_image.size
+                aspect_ratio = width / height
+                
+                if aspect_ratio <= 2/5 or aspect_ratio >= 5/2:
+                    raise ValueError(f"Image aspect ratio ({aspect_ratio:.2f}) must be between 2:5 and 5:2")
+                
+                min_dimension = min(width, height)
+                if min_dimension < 300:
+                    raise ValueError(f"Image short side ({min_dimension}px) must be at least 300px")
+                
+                # Convert PIL Image to base64
+                import io
+                buffer = io.BytesIO()
+                
+                # Save as JPEG for API compatibility
+                if pil_image.mode in ('RGBA', 'LA', 'P'):
+                    # Convert to RGB for JPEG
+                    pil_image = pil_image.convert('RGB')
+                
+                pil_image.save(buffer, format='JPEG', quality=95)
+                buffer.seek(0)
+                
+                # Check file size (should be < 20MB)
+                file_size = buffer.getbuffer().nbytes
+                if file_size > 20 * 1024 * 1024:  # 20MB
+                    raise ValueError(f"Image file size ({file_size/1024/1024:.1f}MB) exceeds 20MB limit")
+                
+                # Convert to base64
+                image_base64 = base64.b64encode(buffer.getvalue()).decode('utf-8')
+                
+                # Add image to payload
+                payload["first_frame_image"] = f"data:image/jpeg;base64,{image_base64}"
+                
+                print(f"Image converted to base64, size: {len(image_base64)} characters")
+                print(f"Image dimensions: {width}x{height}, aspect ratio: {aspect_ratio:.2f}")
+            
+            # Add optional callback URL if provided
+            if callback_url.strip():
+                payload["callback_url"] = callback_url.strip()
+            
+            print(f"Sending request to {self.api_url}")
+            print(f"Model: {model}")
+            print(f"Prompt: {prompt}")
+            print(f"Prompt optimizer: {prompt_optimizer}")
+            if callback_url:
+                print(f"Callback URL: {callback_url}")
+            
+            # Make API request
+            response = requests.post(
+                self.api_url,
+                headers=headers,
+                data=json.dumps(payload),
+                timeout=30
+            )
+            
+            print(f"Response status code: {response.status_code}")
+            
+            try:
+                response_data = response.json()
+                print(f"Response data: {json.dumps(response_data, indent=2)}")
+            except json.JSONDecodeError:
+                print(f"Raw response content: {response.content}")
+                raise RuntimeError("Failed to decode JSON response")
+            
+            # Check for API errors
+            base_resp = response_data.get("base_resp", {})
+            status_code = base_resp.get("status_code")
+            status_msg = base_resp.get("status_msg", "Unknown error")
+            
+            if status_code != 0:
+                error_messages = {
+                    1002: "Rate limit exceeded, please try again later",
+                    1004: "Authentication failed, please check your API key",
+                    1008: "Insufficient account balance",
+                    1026: "Video description contains sensitive content, please adjust",
+                    2013: "Invalid parameters, please check your input",
+                    2049: "Invalid API key, please check your API key"
+                }
+                error_msg = error_messages.get(status_code, f"API Error {status_code}: {status_msg}")
+                raise RuntimeError(error_msg)
+            
+            # Extract task_id
+            task_id = response_data.get("task_id")
+            if not task_id:
+                raise RuntimeError("No task_id returned from API")
+            
+            print(f"Video generation task created successfully, task_id: {task_id}")
+            
+            return (task_id,)
+
+        except requests.exceptions.RequestException as e:
+            print(f"Request error: {str(e)}")
+            raise RuntimeError(f"Failed to connect to MiniMax API: {str(e)}")
+        except Exception as e:
+            print(f"Unexpected error: {str(e)}")
+            raise RuntimeError(f"Video generation failed: {str(e)}") 

requirements.txt

@@ -1 +1,2 @@
-requests>=2.31.0 
+requests>=2.31.0
+Pillow>=8.0.0