wenjian
0c7a311a2e
- Add English README with installation and usage instructions - Add Chinese README (README_cn.md) for Chinese users - Add requirements.txt with necessary dependencies - Document Video-As-Prompt integration for motion-guided video generation
99 lines
3.1 KiB
Markdown
99 lines
3.1 KiB
Markdown
# ComfyUI Video-As-Prompt Node
|
|
|
|
A custom node for ComfyUI that integrates Video-As-Prompt for motion-guided video generation from image inputs.
|
|
|
|
## ✨ Features
|
|
|
|
* 🎬 **Motion-Guided Generation**: Use reference videos to control motion in generated videos
|
|
* 🖼️ **Image-to-Video**: Generate videos from image with reference motion guidance
|
|
* ⚙️ **Memory Optimization**: INT8 quantization + CPU offload for efficient inference
|
|
* 🚀 **CogVideoX-5B**: Based on powerful CogVideoX-5B model
|
|
|
|
## 🔧 Node List
|
|
|
|
* **RunningHub VideoAsPrompt Loader**: Load and initialize Video-As-Prompt pipeline
|
|
* **RunningHub VideoAsPrompt Sampler(CogVideoX)**: Generate video from image with reference motion
|
|
|
|
## 🚀 Quick Installation
|
|
|
|
### Step 1: Install the Node
|
|
|
|
```bash
|
|
# Navigate to ComfyUI custom_nodes directory
|
|
cd ComfyUI/custom_nodes/
|
|
|
|
# Clone the repository
|
|
git clone https://github.com/HM-RunningHub/ComfyUI_RH_VideoAsPrompt.git
|
|
|
|
cd ComfyUI_RH_VideoAsPrompt
|
|
|
|
# Install dependencies
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
### Step 2: Download Required Models
|
|
|
|
Download the CogVideoX-5B model and place it in the following structure:
|
|
|
|
```
|
|
ComfyUI/models/Video-As-Prompt/
|
|
└── CogVideoX-5B/
|
|
├── vae/
|
|
├── transformer/
|
|
└── ...
|
|
```
|
|
|
|
You can download from [Video-As-Prompt Dataset](https://huggingface.co/datasets/BianYx/VAP-Data) or use the pretrained CogVideoX-5B model.
|
|
|
|
### Step 3: Restart ComfyUI
|
|
|
|
## 📖 Usage
|
|
|
|
### Basic Workflow
|
|
|
|
```
|
|
[Load Image] → [Load Video] → [RunningHub VideoAsPrompt Loader] → [RunningHub VideoAsPrompt Sampler] → [Save Video]
|
|
```
|
|
|
|
### Generation Parameters
|
|
|
|
* **image**: Input image for video generation
|
|
* **ref_video**: Reference video for motion guidance
|
|
* **prompt**: Text description for the output video
|
|
* **prompt_mot_ref**: Text description for the reference motion
|
|
* **height/width**: Output video dimensions (default: 480x720)
|
|
* **num_frames**: Number of frames to generate (default: 49)
|
|
* **num_inference_steps**: Denoising steps (default: 50)
|
|
|
|
## 🛠️ Technical Requirements
|
|
|
|
* **GPU**: 12GB+ VRAM (with INT8 quantization + CPU offload)
|
|
* **RAM**: 16GB+ recommended
|
|
* **Storage**: ~20GB for CogVideoX-5B model
|
|
* **CUDA**: Required for optimal performance
|
|
|
|
## ⚠️ Important Notes
|
|
|
|
* **Model Paths**: Models must be placed in `ComfyUI/models/Video-As-Prompt/` directory
|
|
* **Memory Optimization**: INT8 quantization and CPU offload are enabled by default for memory efficiency
|
|
* All model files must be downloaded before first use
|
|
|
|
## 🔗 References
|
|
|
|
* [Video-As-Prompt Project](https://github.com/bytedance/Video-As-Prompt)
|
|
* [Video-As-Prompt Dataset](https://huggingface.co/datasets/BianYx/VAP-Data)
|
|
* [ComfyUI](https://github.com/comfyanonymous/ComfyUI)
|
|
|
|
## 📄 License
|
|
|
|
This project is based on the [Video-As-Prompt](https://github.com/bytedance/Video-As-Prompt) project.
|
|
|
|
## ⭐ Citation
|
|
|
|
If you find this project useful, please consider citing the original Video-As-Prompt paper.
|
|
|
|
---
|
|
|
|
**Developed by [HM-RunningHub](https://github.com/HM-RunningHub)**
|
|
|