Add README and requirements documentation
- Add English README with installation and usage instructions - Add Chinese README (README_cn.md) for Chinese users - Add requirements.txt with necessary dependencies - Document Video-As-Prompt integration for motion-guided video generation
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
# ComfyUI Video-As-Prompt Node
|
||||
|
||||
A custom node for ComfyUI that integrates Video-As-Prompt for motion-guided video generation from image inputs.
|
||||
|
||||
## ✨ Features
|
||||
|
||||
* 🎬 **Motion-Guided Generation**: Use reference videos to control motion in generated videos
|
||||
* 🖼️ **Image-to-Video**: Generate videos from image with reference motion guidance
|
||||
* ⚙️ **Memory Optimization**: INT8 quantization + CPU offload for efficient inference
|
||||
* 🚀 **CogVideoX-5B**: Based on powerful CogVideoX-5B model
|
||||
|
||||
## 🔧 Node List
|
||||
|
||||
* **RunningHub VideoAsPrompt Loader**: Load and initialize Video-As-Prompt pipeline
|
||||
* **RunningHub VideoAsPrompt Sampler(CogVideoX)**: Generate video from image with reference motion
|
||||
|
||||
## 🚀 Quick Installation
|
||||
|
||||
### Step 1: Install the Node
|
||||
|
||||
```bash
|
||||
# Navigate to ComfyUI custom_nodes directory
|
||||
cd ComfyUI/custom_nodes/
|
||||
|
||||
# Clone the repository
|
||||
git clone https://github.com/HM-RunningHub/ComfyUI_RH_VideoAsPrompt.git
|
||||
|
||||
cd ComfyUI_RH_VideoAsPrompt
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### Step 2: Download Required Models
|
||||
|
||||
Download the CogVideoX-5B model and place it in the following structure:
|
||||
|
||||
```
|
||||
ComfyUI/models/Video-As-Prompt/
|
||||
└── CogVideoX-5B/
|
||||
├── vae/
|
||||
├── transformer/
|
||||
└── ...
|
||||
```
|
||||
|
||||
You can download from [Video-As-Prompt Dataset](https://huggingface.co/datasets/BianYx/VAP-Data) or use the pretrained CogVideoX-5B model.
|
||||
|
||||
### Step 3: Restart ComfyUI
|
||||
|
||||
## 📖 Usage
|
||||
|
||||
### Basic Workflow
|
||||
|
||||
```
|
||||
[Load Image] → [Load Video] → [RunningHub VideoAsPrompt Loader] → [RunningHub VideoAsPrompt Sampler] → [Save Video]
|
||||
```
|
||||
|
||||
### Generation Parameters
|
||||
|
||||
* **image**: Input image for video generation
|
||||
* **ref_video**: Reference video for motion guidance
|
||||
* **prompt**: Text description for the output video
|
||||
* **prompt_mot_ref**: Text description for the reference motion
|
||||
* **height/width**: Output video dimensions (default: 480x720)
|
||||
* **num_frames**: Number of frames to generate (default: 49)
|
||||
* **num_inference_steps**: Denoising steps (default: 50)
|
||||
|
||||
## 🛠️ Technical Requirements
|
||||
|
||||
* **GPU**: 12GB+ VRAM (with INT8 quantization + CPU offload)
|
||||
* **RAM**: 16GB+ recommended
|
||||
* **Storage**: ~20GB for CogVideoX-5B model
|
||||
* **CUDA**: Required for optimal performance
|
||||
|
||||
## ⚠️ Important Notes
|
||||
|
||||
* **Model Paths**: Models must be placed in `ComfyUI/models/Video-As-Prompt/` directory
|
||||
* **Memory Optimization**: INT8 quantization and CPU offload are enabled by default for memory efficiency
|
||||
* All model files must be downloaded before first use
|
||||
|
||||
## 🔗 References
|
||||
|
||||
* [Video-As-Prompt Project](https://github.com/bytedance/Video-As-Prompt)
|
||||
* [Video-As-Prompt Dataset](https://huggingface.co/datasets/BianYx/VAP-Data)
|
||||
* [ComfyUI](https://github.com/comfyanonymous/ComfyUI)
|
||||
|
||||
## 📄 License
|
||||
|
||||
This project is based on the [Video-As-Prompt](https://github.com/bytedance/Video-As-Prompt) project.
|
||||
|
||||
## ⭐ Citation
|
||||
|
||||
If you find this project useful, please consider citing the original Video-As-Prompt paper.
|
||||
|
||||
---
|
||||
|
||||
**Developed by [HM-RunningHub](https://github.com/HM-RunningHub)**
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
# ComfyUI Video-As-Prompt 节点
|
||||
|
||||
ComfyUI 的自定义节点,集成 Video-As-Prompt 实现运动引导的视频生成。
|
||||
|
||||
## ✨ 功能特性
|
||||
|
||||
* 🎬 **运动引导生成**:使用参考视频控制生成视频的运动
|
||||
* 🖼️ **图像转视频**:从图像生成视频,并通过参考视频引导运动
|
||||
* ⚙️ **内存优化**:INT8量化 + CPU卸载,高效推理
|
||||
* 🚀 **CogVideoX-5B**:基于强大的CogVideoX-5B模型
|
||||
|
||||
## 🔧 节点列表
|
||||
|
||||
* **RunningHub VideoAsPrompt Loader**:加载并初始化 Video-As-Prompt 管线
|
||||
* **RunningHub VideoAsPrompt Sampler(CogVideoX)**:从图像生成带参考运动的视频
|
||||
|
||||
## 🚀 快速安装
|
||||
|
||||
### 步骤 1:安装节点
|
||||
|
||||
```bash
|
||||
# 进入 ComfyUI 的 custom_nodes 目录
|
||||
cd ComfyUI/custom_nodes/
|
||||
|
||||
# 克隆仓库
|
||||
git clone https://github.com/HM-RunningHub/ComfyUI_RH_VideoAsPrompt.git
|
||||
|
||||
cd ComfyUI_RH_VideoAsPrompt
|
||||
|
||||
# 安装依赖
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 步骤 2:下载所需模型
|
||||
|
||||
下载 CogVideoX-5B 模型并放置在以下目录结构:
|
||||
|
||||
```
|
||||
ComfyUI/models/Video-As-Prompt/
|
||||
└── CogVideoX-5B/
|
||||
├── vae/
|
||||
├── transformer/
|
||||
└── ...
|
||||
```
|
||||
|
||||
可以从 [Video-As-Prompt 数据集](https://huggingface.co/datasets/BianYx/VAP-Data) 下载,或使用预训练的 CogVideoX-5B 模型。
|
||||
|
||||
### 步骤 3:重启 ComfyUI
|
||||
|
||||
## 📖 使用说明
|
||||
|
||||
### 基本工作流
|
||||
|
||||
```
|
||||
[加载图像] → [加载视频] → [RunningHub VideoAsPrompt Loader] → [RunningHub VideoAsPrompt Sampler] → [保存视频]
|
||||
```
|
||||
|
||||
### 生成参数
|
||||
|
||||
* **image**:用于视频生成的输入图像
|
||||
* **ref_video**:用于运动引导的参考视频
|
||||
* **prompt**:输出视频的文本描述
|
||||
* **prompt_mot_ref**:参考运动的文本描述
|
||||
* **height/width**:输出视频尺寸(默认:480x720)
|
||||
* **num_frames**:生成帧数(默认:49)
|
||||
* **num_inference_steps**:去噪步数(默认:50)
|
||||
|
||||
## 🛠️ 技术要求
|
||||
|
||||
* **GPU**:12GB+ 显存(使用 INT8 量化 + CPU 卸载)
|
||||
* **内存**:建议 16GB+
|
||||
* **存储**:CogVideoX-5B 模型约 20GB
|
||||
* **CUDA**:需要 CUDA 以获得最佳性能
|
||||
|
||||
## ⚠️ 重要说明
|
||||
|
||||
* **模型路径**:模型必须放置在 `ComfyUI/models/Video-As-Prompt/` 目录下
|
||||
* **内存优化**:默认启用 INT8 量化和 CPU 卸载以提高内存效率
|
||||
* 首次使用前必须下载所有模型文件
|
||||
|
||||
## 🔗 参考链接
|
||||
|
||||
* [Video-As-Prompt 项目](https://github.com/bytedance/Video-As-Prompt)
|
||||
* [Video-As-Prompt 数据集](https://huggingface.co/datasets/BianYx/VAP-Data)
|
||||
* [ComfyUI](https://github.com/comfyanonymous/ComfyUI)
|
||||
|
||||
## 📄 许可证
|
||||
|
||||
本项目基于 [Video-As-Prompt](https://github.com/bytedance/Video-As-Prompt) 项目开发。
|
||||
|
||||
## ⭐ 引用
|
||||
|
||||
如果您觉得本项目有用,请考虑引用原始 Video-As-Prompt 论文。
|
||||
|
||||
---
|
||||
|
||||
**开发者:[HM-RunningHub](https://github.com/HM-RunningHub)**
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
diffusers>=0.30.0
|
||||
transformers>=4.44.0
|
||||
accelerate>=0.33.0
|
||||
imageio>=2.34.0
|
||||
torch>=2.1.0
|
||||
torchvision
|
||||
Pillow>=10.0.0
|
||||
numpy
|
||||
optimum-quanto>=0.2.0
|
||||
|
||||
Reference in New Issue
Block a user