diff --git a/README.md b/README.md index f4dac9a..f462584 100644 --- a/README.md +++ b/README.md @@ -47,6 +47,13 @@ The following examples are provided in the `workflow` folder: ![Role2Story Workflow Example](imgs/role2story.jpg) +#### Video Caption Workflow +- File: [workflows/StepFunVideo2Caption.json](workflows/StepFunVideo2Caption.json) +- Features: + - 📽️ Native Video Upload Support + - 📝 Automatic Video Caption Generation + - 🤖 Intelligent Video Content Analysis + > 🚧 **More workflow examples are being updated...** ## Supported Features @@ -54,8 +61,33 @@ The following examples are provided in the `workflow` folder: - Image Content Recognition and Description - Video Scene Analysis - Intelligent Prompt Generation +- Native Video Upload Support - [Other Features] +## Video Processing Guide + +### Native Video Upload +You can now use the `StepFun Video File Uploader` node for direct video uploads. Simply provide the absolute path to your video file (e.g., `C:\Users\Administrator\Downloads\test.mp4`). The node will return a `stepfile://file-id` string that can be used as input for the `video_url` parameter in the `StepFun Video Chat` node. + +![Video Chat Example](imgs/videochat.png) + +### Video File Processing with FFmpeg +For videos larger than 128MB or in different formats, you can use FFmpeg for preprocessing: + +#### Splitting Video Files +To split a large video into smaller segments (120-second chunks): +```bash +ffmpeg -i sample.mp4 -acodec copy -f segment -segment_time 120 -vcodec copy -reset_timestamps 1 -map 0 output_time_%d.mp4 +``` + +#### Converting to MP4 Format +To convert videos from other formats to MP4: +```bash +ffmpeg -i sample.mkv -codec copy sample.mp4 +``` + +> Note: It's recommended to compress videos before uploading for better performance. + ## Notes - API calls require internet connection diff --git a/README_zh.md b/README_zh.md index 4558168..2b4d7fe 100644 --- a/README_zh.md +++ b/README_zh.md @@ -1,39 +1,40 @@ -# ComfyUI StepFun Nodes +# ComfyUI StepFun 节点 -这是一个基于[阶跃星辰(StepFun)](https://platform.stepfun.com/)API的ComfyUI自定义节点集合。通过这些节点，您可以轻松实现图片和视频的智能分析与处理。 -目前视频的上传暂未实现，需要传入视频url链接. +这是一个基于 [StepFun](https://platform.stepfun.com/) API 的 ComfyUI 自定义节点集合。这些节点可以轻松实现图像和视频的智能分析和处理。 +目前尚未实现视频上传功能，需要使用视频 URL 链接。 > 🚧 **更多节点正在更新中...** + ## 主要特点 -- 🖼️ 图片内容理解与分析 -- 🎬 视频内容理解与分析 -- ✨ 智能提示词(Prompt)生成 -- 💰 低成本的API定价 -- 🚀 无需本地GPU资源，云端处理 +- 🖼️ 图像内容理解和分析 +- 🎬 视频内容理解和分析 +- ✨ 智能提示生成 +- 💰 经济实惠的 API 定价 +- 🚀 无需本地 GPU 资源的云端处理 -## 安装说明 +## 安装 -1. 将本仓库克隆到ComfyUI的`custom_nodes`目录下: +1. 将此仓库克隆到你的 ComfyUI 的 `custom_nodes` 目录： ```bash cd custom_nodes git clone https://github.com/your-repo/ComfyUI_StepFun.git ``` -2. 安装依赖: +2. 安装依赖： ```bash cd ComfyUI_StepFun pip install -r requirements.txt ``` -## API密钥配置 +## API 密钥配置 -1. 访问[StepFun平台](https://platform.stepfun.com/account-overview)注册账号 -2. 在账户概览页面获取API密钥 -3. 将API密钥添加到配置文件中 +1. 在 [StepFun 平台](https://platform.stepfun.com/account-overview) 注册账号 +2. 从账户概览页面获取 API 密钥 +3. 将 API 密钥添加到配置文件中 ## 使用方法 -### 示例工作流 +### 工作流示例 在 `workflow` 文件夹中提供了以下示例： @@ -41,31 +42,61 @@ pip install -r requirements.txt - 文件：`workflow/role2story.json` - 功能：通过输入角色/场景/故事描述，自动生成： - 🎨 主题海报 - - 🎬 三个关键剧情镜头 -- 该工作流展示了模型对内容的理解能力和智能提示词生成能力 + - 🎬 三个关键情节镜头 +- 此工作流展示了模型的内容理解和智能提示生成能力 -![Role2Story工作流示例](imgs/role2story.jpg) +![Role2Story 工作流示例](imgs/role2story.jpg) + +#### 视频字幕工作流 +- 文件：[workflows/StepFunVideo2Caption.json](workflows/StepFunVideo2Caption.json) +- 功能： + - 📽️ 原生视频上传支持 + - 📝 自动视频字幕生成 + - 🤖 智能视频内容分析 > 🚧 **更多工作流示例正在更新中...** ## 支持的功能 -- 图片内容识别与描述 +- 图像内容识别和描述 - 视频场景分析 -- 智能提示词生成 +- 智能提示生成 +- 原生视频上传支持 - [其他功能] +## 视频处理指南 + +### 原生视频上传 +现在你可以使用 `StepFun Video File Uploader` 节点进行直接视频上传。只需提供视频文件的绝对路径（例如：`C:\Users\Administrator\Downloads\test.mp4`）。该节点将返回一个 `stepfile://file-id` 字符串，可用作 `StepFun Video Chat` 节点中 `video_url` 参数的输入。 + +![视频聊天示例](imgs/videochat.png) + +### 使用 FFmpeg 处理视频文件 +对于大于 128MB 或不同格式的视频，你可以使用 FFmpeg 进行预处理： + +#### 分割视频文件 +将大视频分割成较小的片段（120秒一段）： +```bash +ffmpeg -i sample.mp4 -acodec copy -f segment -segment_time 120 -vcodec copy -reset_timestamps 1 -map 0 output_time_%d.mp4 +``` + +#### 转换为 MP4 格式 +将其他格式的视频转换为 MP4： +```bash +ffmpeg -i sample.mkv -codec copy sample.mp4 +``` + +> 注意：建议在上传前压缩视频以获得更好的性能。 + ## 注意事项 -- API调用需要联网 -- 请确保API密钥配置正确 -- 遵守API使用限制和条款 - - +- API 调用需要互联网连接 +- 确保正确配置 API 密钥 +- 遵守 API 使用限制和条款 ## 联系方式 -- Email: 3354405250@qq.com +- 邮箱：3354405250@qq.com - [其他联系方式] -如果您在使用过程中遇到任何问题，欢迎通过以上方式联系我们。 +如果您在使用本项目时遇到任何问题，请随时通过以上方式与我们联系。 diff --git a/imgs/videochat.png b/imgs/videochat.png new file mode 100644 index 0000000..0de3ea3 Binary files /dev/null and b/imgs/videochat.png differ diff --git a/nodes.py b/nodes.py index 16f3ba7..28b4d05 100644 --- a/nodes.py +++ b/nodes.py @@ -27,25 +27,7 @@ def ensure_openai(): print("Failed to install 'openai'. Please, install manually.") from openai import OpenAI -def ensure_oss2(): - if "python_embeded" in sys.executable or "python_embedded" in sys.executable: - pip_install = [sys.executable, "-s", "-m", "pip", "install"] - else: - pip_install = [sys.executable, "-m", "pip", "install"] - try: - import oss2 - from oss2.models import Bucket - except Exception as e: - try: - subprocess.check_call(pip_install + ['oss2']) - import oss2 - from oss2.models import Bucket - except: - print("Failed to install 'oss2'. Please, install manually.") - -import oss2 -# from oss2.models import Bucket import json @@ -381,91 +363,6 @@ class JSONParser: except Exception as e: raise ValueError(f"JSON解析错误: {str(e)}") -class OSSClient: - @classmethod - def INPUT_TYPES(s): - return { - "required": { - "access_key_id": ("STRING", {"default": ""}), - "access_key_secret": ("STRING", {"default": ""}), - "endpoint": ("STRING", {"default": "oss-cn-hangzhou.aliyuncs.com"}), - "bucket_name": ("STRING", {"default": ""}), - "region": ("STRING", {"default": "cn-hangzhou"}) - } - } - - RETURN_TYPES = ("OSS_CLIENT",) - RETURN_NAMES = ("client",) - FUNCTION = "create_client" - CATEGORY = "StepFun" - - def create_client(self, access_key_id, access_key_secret, endpoint, bucket_name, region): - if endpoint.startswith('http://') or endpoint.startswith('https://'): - endpoint = endpoint.split('://')[-1] - - if '.' in bucket_name: - bucket_name = bucket_name.split('.')[0] - - auth = oss2.Auth(access_key_id, access_key_secret) - bucket = oss2.Bucket(auth, endpoint, bucket_name, region=region) - return (bucket,) - -class OSSUploader: - @classmethod - def INPUT_TYPES(s): - - return {"required": - {"client": ("OSS_CLIENT",), - "target_path": ("STRING", {"default": ""}), - "file_path":("STRING", {"multiline": True}), - } - } - - RETURN_TYPES = ("STRING",) - RETURN_NAMES = ("oss_url",) - FUNCTION = "upload_file" - CATEGORY = "StepFun" - - def upload_file(self, client, target_path, file_path): - try: - if not file_path: - raise ValueError("请选择要上传的文件路径") - - # 检查文件是否存在 - if not os.path.exists(file_path): - raise ValueError(f"文件不存在: {file_path}") - - # 使用原文件名(不包含路径) - filename = os.path.basename(file_path) - - # 构建目标路径 - if target_path: - # 移除开头的斜杠并规范化路径分隔符 - target_path = target_path.strip('/') - target_path = target_path.replace('\\', '/') - if target_path: - object_key = f"{target_path}/{filename}" - else: - object_key = filename - else: - object_key = filename - - client.put_object_from_file(object_key, file_path) - - # 构建OSS URL - bucket_endpoint = client.endpoint.replace('http://', '').replace('https://', '') - oss_url = f"https://{client.bucket_name}.{bucket_endpoint}/{object_key}" - - return { - "ui": { - "oss_url": oss_url - }, - "result": (oss_url,) - } - - except Exception as e: - error_msg = f"上传失败: {str(e)}" - raise Exception(error_msg) class CombineStrings: @classmethod @@ -492,15 +389,71 @@ class CombineStrings: return (all_strings, combined_string) +class VideoFileUploader: + @classmethod + def INPUT_TYPES(s): + return { + "required": { + "client": ("STEPFUN_CLIENT",), + "file_path": ("STRING", {"default": ""}), + }, + "hidden": { + "unique_id": "UNIQUE_ID" + } + } + + RETURN_TYPES = ("STRING",) + RETURN_NAMES = ("file_link",) + FUNCTION = "upload_video_file" + CATEGORY = "StepFun" + + def upload_video_file(self, client, file_path, unique_id): + if not os.path.exists(file_path): + raise ValueError(f"文件不存在: {file_path}") + + PromptServer.instance.send_sync("stepfun.file.upload", { + "node": unique_id, + "message": f"开始上传视频文件: {os.path.basename(file_path)}" + }) + + try: + response = client.files.create( + file=open(file_path, "rb"), + purpose="storage" + ) + + file_link = f"stepfile://{response.id}" + + PromptServer.instance.send_sync("stepfun.file.upload", { + "node": unique_id, + "message": f"视频文件上传成功: {response.filename}", + "type": "success" + }) + + return { + "ui": { + "file_link": file_link + }, + "result": (file_link,) + } + + except Exception as e: + error_msg = f"上传失败: {str(e)}" + PromptServer.instance.send_sync("stepfun.file.upload", { + "node": unique_id, + "message": error_msg, + "type": "error" + }) + raise Exception(error_msg) + # 更新节点映射 NODE_CLASS_MAPPINGS = { "StepFunClient": StepFunClient, "TextImageChat": TextImageChat, "VideoChat": VideoChat, "JSONParser": JSONParser, - "OSSClient": OSSClient, - "OSSUploader": OSSUploader, "CombineStrings": CombineStrings, + "VideoFileUploader": VideoFileUploader, } @@ -509,8 +462,7 @@ NODE_DISPLAY_NAME_MAPPINGS = { "TextImageChat": "StepFun Chat Completion", "VideoChat": "StepFun Video Chat", "JSONParser": "StepFun JSON Parser", - "OSSClient": "StepFun OSS Client", - "OSSUploader": "StepFun OSS Uploader", "CombineStrings": "StepFun String Combiner", + "VideoFileUploader": "StepFun Video File Uploader", } diff --git a/pyproject.toml b/pyproject.toml index 0681286..193fe82 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "comfyui-scstepfun" description = "To use stepfun's library, you need an official api that supports multimodal inputs such as video and pictures https://platform.stepfun.com/request-restriction" -version = "1.0.2" +version = "1.1.0" license = {file = "LICENSE"} dependencies = ["openai"] diff --git a/workflows/StepFunVideo2Caption.json b/workflows/StepFunVideo2Caption.json new file mode 100644 index 0000000..efc0467 --- /dev/null +++ b/workflows/StepFunVideo2Caption.json @@ -0,0 +1,277 @@ +{ + "last_node_id": 5, + "last_link_id": 5, + "nodes": [ + { + "id": 4, + "type": "VideoChat", + "pos": [ + -2003.46435546875, + 444.6044006347656 + ], + "size": [ + 400, + 310 + ], + "flags": { + "collapsed": false + }, + "order": 3, + "mode": 0, + "inputs": [ + { + "name": "client", + "type": "STEPFUN_CLIENT", + "link": 3 + }, + { + "name": "video_url", + "type": "STRING", + "link": 4, + "widget": { + "name": "video_url" + } + } + ], + "outputs": [ + { + "name": "response", + "type": "STRING", + "links": [ + 5 + ], + "slot_index": 0 + }, + { + "name": "total_tokens", + "type": "STRING", + "links": null + }, + { + "name": "finish_reason", + "type": "STRING", + "links": null + } + ], + "properties": { + "Node name for S&R": "VideoChat" + }, + "widgets_values": [ + "step-1.5v-mini", + "Description What happened to the uploaded video", + "", + "", + false, + 0.5, + 0.9, + 1024 + ] + }, + { + "id": 5, + "type": "ShowText|pysssss", + "pos": [ + -1557.027587890625, + 445.50140380859375 + ], + "size": [ + 321.4136047363281, + 270.87322998046875 + ], + "flags": {}, + "order": 4, + "mode": 0, + "inputs": [ + { + "name": "text", + "type": "STRING", + "link": 5, + "widget": { + "name": "text" + } + } + ], + "outputs": [ + { + "name": "STRING", + "type": "STRING", + "links": null, + "shape": 6 + } + ], + "properties": { + "Node name for S&R": "ShowText|pysssss" + }, + "widgets_values": [ + "", + "The image depicts a cartoon illustration of a sleeping animal, possibly a cat or a similar feline-like creature. The animal is shown lying on its side with its eyes closed and appears to be snoring, indicated by the \"zzz\" sound effect above its head. The background is a simple representation of the sky with some clouds. The illustration has a playful and whimsical appearance." + ] + }, + { + "id": 3, + "type": "StepFunClient", + "pos": [ + -2624.2880859375, + 108.11141967773438 + ], + "size": [ + 315, + 58 + ], + "flags": {}, + "order": 0, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "client", + "type": "STEPFUN_CLIENT", + "links": [ + 2, + 3 + ], + "slot_index": 0 + } + ], + "properties": { + "Node name for S&R": "StepFunClient" + }, + "widgets_values": [ + "" + ] + }, + { + "id": 2, + "type": "ShowText|pysssss", + "pos": [ + -1730.67626953125, + 256.4507141113281 + ], + "size": [ + 315, + 76 + ], + "flags": {}, + "order": 2, + "mode": 0, + "inputs": [ + { + "name": "text", + "type": "STRING", + "link": 1, + "widget": { + "name": "text" + } + } + ], + "outputs": [ + { + "name": "STRING", + "type": "STRING", + "links": null, + "shape": 6 + } + ], + "properties": { + "Node name for S&R": "ShowText|pysssss" + }, + "widgets_values": [ + "", + "stepfile://file-CzxiDg8Ey0" + ] + }, + { + "id": 1, + "type": "VideoFileUploader", + "pos": [ + -2175.044921875, + 271.3929443359375 + ], + "size": [ + 340.20001220703125, + 58 + ], + "flags": {}, + "order": 1, + "mode": 0, + "inputs": [ + { + "name": "client", + "type": "STEPFUN_CLIENT", + "link": 2 + } + ], + "outputs": [ + { + "name": "file_link", + "type": "STRING", + "links": [ + 1, + 4 + ], + "slot_index": 0 + } + ], + "properties": { + "Node name for S&R": "VideoFileUploader" + }, + "widgets_values": [ + "C:\\Users\\Administrator\\Downloads\\vidu--4-2024-11-21T20_47_47Z.mp4" + ] + } + ], + "links": [ + [ + 1, + 1, + 0, + 2, + 0, + "STRING" + ], + [ + 2, + 3, + 0, + 1, + 0, + "STEPFUN_CLIENT" + ], + [ + 3, + 3, + 0, + 4, + 0, + "STEPFUN_CLIENT" + ], + [ + 4, + 1, + 0, + 4, + 1, + "STRING" + ], + [ + 5, + 4, + 0, + 5, + 0, + "STRING" + ] + ], + "groups": [], + "config": {}, + "extra": { + "ds": { + "scale": 1.3513057093104115, + "offset": [ + 2805.166728388079, + 34.840790315249286 + ] + } + }, + "version": 0.4 +} \ No newline at end of file