diff --git a/README.md b/README.md index 9bcad5b..175758d 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,140 @@ -# ComfyUI-QwenVL -ComfyUI custom node for Qwen3-VL: Effortlessly handle text creation, image understanding, and video analysis with advanced multimodal capabilities. +# **QwenVL for ComfyUI** + +The ComfyUI-QwenVL custom node integrates the powerful Qwen-VL series of vision-language models (LVLMs) from Alibaba Cloud, including the latest Qwen3-VL and Qwen2.5-VL. This advanced node enables seamless multimodal AI capabilities within your ComfyUI workflows, allowing for efficient text generation, image understanding, and video analysis. + +## **πŸ“° News & Updates** + +* **2025/10/17**: **v1.0.0** Initial Release + * Support for Qwen3-VL and Qwen2.5-VL series models. + * Automatic model downloading from Hugging Face. + * On-the-fly quantization (4-bit, 8-bit, FP16). + * Preset and Custom Prompt system for flexible and easy use. + * **Includes both a standard and an advanced node** for users of all levels. + * Hardware-aware safeguards for FP8 model compatibility. + * Image and Video (frame sequence) input support. + * "Keep Model Loaded" option for improved performance on sequential runs. + * **Seed parameter** for reproducible generation. + +## **✨ Features** + +* **Standard & Advanced Nodes**: Includes a simple QwenVL node for quick use and a QwenVL (Advanced) node with fine-grained control over generation. +* **Preset & Custom Prompts**: Choose from a list of convenient preset prompts or write your own for full control. +* **Multi-Model Support**: Easily switch between various official Qwen-VL models. +* **Automatic Model Download**: Models are downloaded automatically on first use. +* **Smart Quantization**: Balance VRAM and performance with 4-bit, 8-bit, and FP16 options. +* **Hardware-Aware**: Automatically detects GPU capabilities and prevents errors with incompatible models (e.g., FP8). +* **Reproducible Generation**: Use the seed parameter to get consistent outputs. +* **Memory Management**: "Keep Model Loaded" option to retain the model in VRAM for faster processing. +* **Image & Video Support**: Accepts both single images and video frame sequences as input. +* **Robust Error Handling**: Provides clear error messages for hardware or memory issues. +* **Clean Console Output**: Minimal and informative console logs during operation. + +## **πŸš€ Installation** + +1. Clone this repository to your ComfyUI/custom\_nodes directory: + ``` + cd ComfyUI/custom\_nodes + git clone https://github.com/1038lab/ComfyUI-QwenVL.git + ``` +2. Install the required dependencies: + ``` + cd ComfyUI/custom\_nodes/ComfyUI-QwenVL + pip install \-r requirements.txt + ``` + +3. Restart ComfyUI. + +## **πŸ“₯ Download Models** + +The models will be automatically downloaded on first use. If you prefer to download them manually, place them in the ComfyUI/models/LLM/Qwen-VL/ directory. + +| Model | Link | +| :---- | :---- | +| Qwen3-VL-4B-Instruct | [Download](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) | +| Qwen3-VL-4B-Instruct-FP8 | [Download](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct-FP8) | +| Qwen3-VL-8B-Instruct | [Download](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) | +| Qwen3-VL-4B-Thinking | [Download](https://huggingface.co/Qwen/Qwen3-VL-4B-Thinking) | +| Qwen3-VL-4B-Thinking-FP8 | [Download](https://huggingface.co/Qwen/Qwen3-VL-4B-Thinking-FP8) | +| Qwen2.5-VL-3B-Instruct | [Download](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) | +| Qwen2.5-VL-7B-Instruct | [Download](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) | + +## **πŸ“– Usage** + +### **Basic Usage** + +1. Add the **"QwenVL"** node from the πŸ§ͺAILab/QwenVL category. +2. Select the **model\_name** you wish to use. +3. Connect an image or video (image sequence) source to the node. +4. Write your prompt using the preset or custom field. +5. Run the workflow. + +### **Advanced Usage** + +For more control, use the **"QwenVL (Advanced)"** node. This gives you access to detailed generation parameters like temperature, top\_p, beam search, and device selection. + +## **βš™οΈ Parameters** + +| Parameter | Description | Default | Range | Node(s) | +| :---- | :---- | :---- | :---- | :---- | +| **model\_name** | The Qwen-VL model to use. | Qwen3-VL-4B-Instruct | \- | Standard & Advanced | +| **quantization** | On-the-fly quantization. Ignored for pre-quantized models (e.g., FP8). | 8-bit (Balanced) | 4-bit, 8-bit, None | Standard & Advanced | +| **preset\_prompt** | A selection of pre-defined prompts for common tasks. | "Describe this..." | Any text | Standard & Advanced | +| **custom\_prompt** | Overrides the preset prompt if provided. | | Any text | Standard & Advanced | +| **max\_tokens** | Maximum number of new tokens to generate. | 1024 | 64-2048 | Standard & Advanced | +| **keep\_model\_loaded** | Keep the model in VRAM for faster subsequent runs. | True | True/False | Standard & Advanced | +| **seed** | A seed for reproducible results. | 1 | 1 \- 2^64-1 | Standard & Advanced | +| **temperature** | Controls randomness. Higher values \= more creative. (Used when num\_beams is 1). | 0.6 | 0.1-1.0 | Advanced Only | +| **top\_p** | Nucleus sampling threshold. (Used when num\_beams is 1). | 0.9 | 0.0-1.0 | Advanced Only | +| **num\_beams** | Number of beams for beam search. \> 1 disables temperature/top\_p sampling. | 1 | 1-10 | Advanced Only | +| **repetition\_penalty** | Discourages repeating tokens. | 1.2 | 0.0-2.0 | Advanced Only | +| **frame\_count** | Number of frames to sample from the video input. | 16 | 1-64 | Advanced Only | +| **device** | Override automatic device selection. | auto | auto, cuda, cpu | Advanced Only | + +### **πŸ’‘ Quantization Options** + +| Mode | Precision | Memory Usage | Speed | Quality | Recommended For | +| :---- | :---- | :---- | :---- | :---- | :---- | +| None (FP16) | 16-bit Float | High | Fastest | Best | High VRAM GPUs (16GB+) | +| 8-bit (Balanced) | 8-bit Integer | Medium | Fast | Very Good | Balanced performance (8GB+) | +| 4-bit (VRAM-friendly) | 4-bit Integer | Low | Slower\* | Good | Low VRAM GPUs (\<8GB) | + +\* **Note on 4-bit Speed**: 4-bit quantization significantly reduces VRAM usage but may result in slower performance on some systems due to the computational overhead of real-time dequantization. + +### **πŸ€” Setting Tips** + +| Setting | Recommendation | +| :---- | :---- | +| **Model Choice** | For most users, Qwen3-VL-4B-Instruct is a great starting point. If you have a 40-series GPU, try the \-FP8 version for better performance. | +| **Memory Mode** | Keep keep\_model\_loaded enabled (True) for the best performance if you plan to run the node multiple times. Disable it only if you are running out of VRAM for other nodes. | +| **Quantization** | Start with the default 8-bit. If you have plenty of VRAM (\>16GB), switch to None (FP16) for the best speed and quality. If you are low on VRAM, use 4-bit. | +| **Performance** | The first time a model is loaded with a specific quantization, it may be slow. Subsequent runs (with keep\_model\_loaded enabled) will be much faster. | + +## **🧠 About Model** + +This node utilizes the Qwen-VL series of models, developed by the Qwen Team at Alibaba Cloud. These are powerful, open-source large vision-language models (LVLMs) designed to understand and process both visual and textual information, making them ideal for tasks like detailed image and video description. + +## **πŸ—ΊοΈ Roadmap** + +### **βœ… Completed (v1.0.0)** + +* βœ… Support for Qwen3-VL and Qwen2.5-VL models. +* βœ… Automatic model downloading and management. +* βœ… On-the-fly 4-bit, 8-bit, and FP16 quantization. +* βœ… Hardware compatibility checks for FP8 models. +* βœ… Image and Video (frame sequence) input support. + +### **πŸ”„ Future Plans** +* GGUF format support for CPU and wider hardware compatibility. +* Integration of more vision-language models. +* Advanced parameter options for fine-tuning generation. +* Support for additional video processing features. + +## **πŸ™ Credits** + +* **Qwen Team**: [Alibaba Cloud](https://github.com/QwenLM) \- For developing and open-sourcing the powerful Qwen-VL models. +* **ComfyUI**: [comfyanonymous](https://github.com/comfyanonymous/ComfyUI) \- For the incredible and extensible ComfyUI platform. +* **ComfyUI Integration**: [1038lab](https://github.com/1038lab) \- Developer of this custom node. + +## **πŸ“œ License** + +This repository's code is released under the [GPL-3.0 License](https://www.gnu.org/licenses/gpl-3.0.en.html).