Add files via upload

2025-03-23 09:46:28 -05:00
parent b2fc69343e
commit f547ed889d
1 changed files with 136 additions and 0 deletions
@@ -0,0 +1,136 @@
+# ComfyUI-Ovis2
+
+A ComfyUI custom node set for integrating [Ovis2](https://huggingface.co/AIDC-AI/Ovis2-34B), a powerful multimodal large language model designed to analyze images and videos.
+
+## Features
+
+- **Image Captioning**: Generate detailed descriptions of images
+- **Multi-Image Analysis**: Compare and analyze up to 4 images simultaneously
+- **Video Description**: Process video frames for scene understanding
+- **Auto-Download**: Automatically download models from Hugging Face
+- **Multiple Models**: Support for all Ovis2 model sizes (1B to 34B parameters)
+
+## Installation
+
+### Option 1: Using ComfyUI Manager (Recommended)
+
+1. Install ComfyUI Manager if you haven't already
+2. Open ComfyUI Manager
+3. Go to "Install Custom Nodes" tab
+4. Click "Install from Git URL"
+5. Enter the GitHub repository URL
+6. Click Install
+
+### Option 2: Manual Installation
+
+1. Navigate to your ComfyUI installation folder
+2. Go to the `custom_nodes` directory (create it if it doesn't exist)
+3. Clone this repository:
+   ```bash
+   git clone https://github.com/YourUsername/ComfyUI-Ovis2.git
+   ```
+4. Install the required dependencies:
+   ```bash
+   pip install -r custom_nodes/ComfyUI-Ovis2/requirements.txt
+   ```
+5. Restart ComfyUI
+
+## Dependencies
+
+- transformers>=4.46.2
+- huggingface-hub>=0.23.0
+- torch>=2.4.0
+- pillow>=10.3.0
+- flash-attn>=2.7.0
+- numpy>=1.25.0
+
+## Usage
+
+After installation, you'll find four new nodes in the "Ovis2" category:
+
+### Load Ovis2 Model
+
+Loads the Ovis2 model with configurable settings:
+- `model_name`: Choose which Ovis2 model to load
+- `precision`: Set numerical precision
+- `max_token_length`: Maximum context length
+- `device`: Choose CPU or CUDA for inference
+- `auto_download`: Enable or disable automatic model downloading
+
+### Ovis2 Image Caption
+
+Generates detailed descriptions of images:
+- `model`: Connect to the Ovis2 model
+- `image`: Connect to an image input
+- `prompt`: Instructions for the model
+- `max_new_tokens`: Maximum length of generated text
+- `temperature`: Controls randomness
+
+### Ovis2 Multi-Image Analysis
+
+Analyzes multiple images together:
+- Supports up to 4 images simultaneously
+- Great for comparison or sequence analysis
+
+### Ovis2 Video Frames Description
+
+Processes video frames:
+- Works with ComfyUI's standard video frame output format
+- Controls for frame_skip and max_frames to handle longer videos
+
+## Example Workflows
+
+### Basic Image Captioning
+
+1. Add a "Load Image" node and select an image
+2. Add a "Load Ovis2 Model" node and choose your preferred model size
+3. Add an "Ovis2 Image Caption" node
+4. Connect:
+   - The image output to the "image" input on the caption node
+   - The model output to the "model" input on the caption node
+5. Run the workflow to generate a detailed caption
+
+### Multi-Image Comparison
+
+1. Load two or more images using "Load Image" nodes
+2. Add a "Load Ovis2 Model" node
+3. Add an "Ovis2 Multi-Image Analysis" node
+4. Connect:
+   - The model output to the "model" input
+   - Each image to the corresponding image inputs
+   - Set a prompt like "Compare these images and describe their similarities and differences"
+5. Run the workflow to get a comparative analysis
+
+## Model Storage
+
+Models are stored in the `models/ovis` directory inside your ComfyUI installation. The nodes will automatically create this directory if it doesn't exist.
+
+## Troubleshooting
+
+### Memory Issues
+
+If you encounter CUDA out of memory errors, try:
+- Using a smaller model (Ovis2-1B or Ovis2-2B)
+- Reducing the image size before processing
+- Switching to "float16" precision
+- Reducing max_token_length
+
+### Model Loading Errors
+
+- Check if auto_download is enabled
+- Ensure you have a proper internet connection during first run
+- Check if the model files are already downloaded to the correct location
+
+### Import Errors
+
+- Verify that all dependencies are correctly installed
+- Check the ComfyUI console for specific error messages
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Acknowledgments
+
+- [AIDC-AI](https://huggingface.co/AIDC-AI) for creating the Ovis2 models
+- [ComfyUI](https://github.com/comfyanonymous/ComfyUI) for the amazing stable diffusion interface