Add README and requirements documentation

- Add English README with installation and usage instructions
- Add Chinese README (README_cn.md) for Chinese users
- Add requirements.txt with necessary dependencies
- Document Video-As-Prompt integration for motion-guided video generation
This commit is contained in:
wenjian
2025-10-30 18:20:11 +08:00
parent 9ecd90b037
commit 0c7a311a2e
3 changed files with 206 additions and 0 deletions
+98
View File
@@ -0,0 +1,98 @@
# ComfyUI Video-As-Prompt Node
A custom node for ComfyUI that integrates Video-As-Prompt for motion-guided video generation from image inputs.
## ✨ Features
* 🎬 **Motion-Guided Generation**: Use reference videos to control motion in generated videos
* 🖼️ **Image-to-Video**: Generate videos from image with reference motion guidance
* ⚙️ **Memory Optimization**: INT8 quantization + CPU offload for efficient inference
* 🚀 **CogVideoX-5B**: Based on powerful CogVideoX-5B model
## 🔧 Node List
* **RunningHub VideoAsPrompt Loader**: Load and initialize Video-As-Prompt pipeline
* **RunningHub VideoAsPrompt Sampler(CogVideoX)**: Generate video from image with reference motion
## 🚀 Quick Installation
### Step 1: Install the Node
```bash
# Navigate to ComfyUI custom_nodes directory
cd ComfyUI/custom_nodes/
# Clone the repository
git clone https://github.com/HM-RunningHub/ComfyUI_RH_VideoAsPrompt.git
cd ComfyUI_RH_VideoAsPrompt
# Install dependencies
pip install -r requirements.txt
```
### Step 2: Download Required Models
Download the CogVideoX-5B model and place it in the following structure:
```
ComfyUI/models/Video-As-Prompt/
└── CogVideoX-5B/
├── vae/
├── transformer/
└── ...
```
You can download from [Video-As-Prompt Dataset](https://huggingface.co/datasets/BianYx/VAP-Data) or use the pretrained CogVideoX-5B model.
### Step 3: Restart ComfyUI
## 📖 Usage
### Basic Workflow
```
[Load Image] → [Load Video] → [RunningHub VideoAsPrompt Loader] → [RunningHub VideoAsPrompt Sampler] → [Save Video]
```
### Generation Parameters
* **image**: Input image for video generation
* **ref_video**: Reference video for motion guidance
* **prompt**: Text description for the output video
* **prompt_mot_ref**: Text description for the reference motion
* **height/width**: Output video dimensions (default: 480x720)
* **num_frames**: Number of frames to generate (default: 49)
* **num_inference_steps**: Denoising steps (default: 50)
## 🛠️ Technical Requirements
* **GPU**: 12GB+ VRAM (with INT8 quantization + CPU offload)
* **RAM**: 16GB+ recommended
* **Storage**: ~20GB for CogVideoX-5B model
* **CUDA**: Required for optimal performance
## ⚠️ Important Notes
* **Model Paths**: Models must be placed in `ComfyUI/models/Video-As-Prompt/` directory
* **Memory Optimization**: INT8 quantization and CPU offload are enabled by default for memory efficiency
* All model files must be downloaded before first use
## 🔗 References
* [Video-As-Prompt Project](https://github.com/bytedance/Video-As-Prompt)
* [Video-As-Prompt Dataset](https://huggingface.co/datasets/BianYx/VAP-Data)
* [ComfyUI](https://github.com/comfyanonymous/ComfyUI)
## 📄 License
This project is based on the [Video-As-Prompt](https://github.com/bytedance/Video-As-Prompt) project.
## ⭐ Citation
If you find this project useful, please consider citing the original Video-As-Prompt paper.
---
**Developed by [HM-RunningHub](https://github.com/HM-RunningHub)**
+98
View File
@@ -0,0 +1,98 @@
# ComfyUI Video-As-Prompt 节点
ComfyUI 的自定义节点,集成 Video-As-Prompt 实现运动引导的视频生成。
## ✨ 功能特性
* 🎬 **运动引导生成**:使用参考视频控制生成视频的运动
* 🖼️ **图像转视频**:从图像生成视频,并通过参考视频引导运动
* ⚙️ **内存优化**:INT8量化 + CPU卸载,高效推理
* 🚀 **CogVideoX-5B**:基于强大的CogVideoX-5B模型
## 🔧 节点列表
* **RunningHub VideoAsPrompt Loader**:加载并初始化 Video-As-Prompt 管线
* **RunningHub VideoAsPrompt Sampler(CogVideoX)**:从图像生成带参考运动的视频
## 🚀 快速安装
### 步骤 1:安装节点
```bash
# 进入 ComfyUI 的 custom_nodes 目录
cd ComfyUI/custom_nodes/
# 克隆仓库
git clone https://github.com/HM-RunningHub/ComfyUI_RH_VideoAsPrompt.git
cd ComfyUI_RH_VideoAsPrompt
# 安装依赖
pip install -r requirements.txt
```
### 步骤 2:下载所需模型
下载 CogVideoX-5B 模型并放置在以下目录结构:
```
ComfyUI/models/Video-As-Prompt/
└── CogVideoX-5B/
├── vae/
├── transformer/
└── ...
```
可以从 [Video-As-Prompt 数据集](https://huggingface.co/datasets/BianYx/VAP-Data) 下载,或使用预训练的 CogVideoX-5B 模型。
### 步骤 3:重启 ComfyUI
## 📖 使用说明
### 基本工作流
```
[加载图像] → [加载视频] → [RunningHub VideoAsPrompt Loader] → [RunningHub VideoAsPrompt Sampler] → [保存视频]
```
### 生成参数
* **image**:用于视频生成的输入图像
* **ref_video**:用于运动引导的参考视频
* **prompt**:输出视频的文本描述
* **prompt_mot_ref**:参考运动的文本描述
* **height/width**:输出视频尺寸(默认:480x720)
* **num_frames**:生成帧数(默认:49)
* **num_inference_steps**:去噪步数(默认:50)
## 🛠️ 技术要求
* **GPU**:12GB+ 显存(使用 INT8 量化 + CPU 卸载)
* **内存**:建议 16GB+
* **存储**:CogVideoX-5B 模型约 20GB
* **CUDA**:需要 CUDA 以获得最佳性能
## ⚠️ 重要说明
* **模型路径**:模型必须放置在 `ComfyUI/models/Video-As-Prompt/` 目录下
* **内存优化**:默认启用 INT8 量化和 CPU 卸载以提高内存效率
* 首次使用前必须下载所有模型文件
## 🔗 参考链接
* [Video-As-Prompt 项目](https://github.com/bytedance/Video-As-Prompt)
* [Video-As-Prompt 数据集](https://huggingface.co/datasets/BianYx/VAP-Data)
* [ComfyUI](https://github.com/comfyanonymous/ComfyUI)
## 📄 许可证
本项目基于 [Video-As-Prompt](https://github.com/bytedance/Video-As-Prompt) 项目开发。
## ⭐ 引用
如果您觉得本项目有用,请考虑引用原始 Video-As-Prompt 论文。
---
**开发者:[HM-RunningHub](https://github.com/HM-RunningHub)**
+10
View File
@@ -0,0 +1,10 @@
diffusers>=0.30.0
transformers>=4.44.0
accelerate>=0.33.0
imageio>=2.34.0
torch>=2.1.0
torchvision
Pillow>=10.0.0
numpy
optimum-quanto>=0.2.0