diff --git a/README.md b/README.md index de4e163..c9e6f64 100644 --- a/README.md +++ b/README.md @@ -6,11 +6,13 @@ ## Features -- Supports MiniCPM models including **MiniCPM-V-4 (GGUF)**. -- Multiple caption types to suit different use cases (Describe, Caption, Analyze, etc.). -- Memory management options to balance VRAM usage and speed. -- Auto-downloads model files on first use for easy setup. -- Customizable parameters: max tokens, temperature, top-p/k sampling, repetition penalty. +- Supports both **MiniCPM-V-4 (Transformers)** and **MiniCPM-V-4 (GGUF)** models +- Multiple caption types to suit different use cases (Describe, Caption, Analyze, etc.) +- Memory management options to balance VRAM usage and speed +- Auto-downloads model files on first use for easy setup +- Customizable parameters: max tokens, temperature, top-p/k sampling, repetition penalty +- Advanced node with full parameter control +- Legacy node for backward compatibility --- @@ -21,7 +23,7 @@ Clone the repo into your ComfyUI custom nodes folder: ```bash cd ComfyUI/custom_nodes git clone https://github.com/1038lab/comfyui-minicpm.git -```` +``` Install required dependencies: @@ -33,20 +35,55 @@ pip install -r ComfyUI/custom_nodes/comfyui-minicpm/requirements.txt ## Supported Models +### Transformers Models +| Model | Description | +| ------------------ | -------------------------------------- | +| MiniCPM-V-4-int4 | 4-bit quantized version, smaller memory footprint | +| MiniCPM-V-4 | Full precision version, higher quality | + +### GGUF Models | Model | Description | | ------------------ | -------------------------------------- | | MiniCPM-V-4 (GGUF) | Latest stable GGUF model, best quality | > The models will be automatically downloaded on first run. -> Manual download and placement into `models/LLM/GGUF` is also supported. +> Manual download and placement into `models/prompt_generator` (transformers) or `models/LLM/GGUF` (GGUF) is also supported. + +--- + +## Available Nodes + +### 1. MiniCPM-4-V-Transformers +- Basic transformers-based node with essential parameters +- Supports image and video input +- Memory management options +- Preset prompt types + +### 2. MiniCPM-4-V-Transformers Advanced +- Full-featured transformers-based node +- All parameters customizable +- System prompt support +- Advanced video processing options + +### 3. MiniCPM-4-V-GGUF +- GGUF-based node with essential parameters +- Optimized for performance + +### 4. MiniCPM-4-V-GGUF Advanced +- Full-featured GGUF-based node +- All parameters customizable + +### 5. MiniCPM (Legacy) +- Original node for backward compatibility +- Basic functionality --- ## Usage 1. Add the **MiniCPM** node from the `🧪AILab` category in ComfyUI. -2. Connect an image input node to the MiniCPM node. -3. Select the model variant (default is MiniCPM-V-4 GGUF). +2. Connect an image or video input node to the MiniCPM node. +3. Select the model variant (default is MiniCPM-V-4-int4 for transformers). 4. Choose caption type and adjust parameters as needed. 5. Execute your workflow to generate captions or analysis. @@ -60,11 +97,11 @@ pip install -r ComfyUI/custom_nodes/comfyui-minicpm/requirements.txt "gpu_layers": -1, "cpu_threads": 4, "default_max_tokens": 1024, - "default_temperature": 0.8, + "default_temperature": 0.7, "default_top_p": 0.9, - "default_top_k": 80, - "default_repetition_penalty": 1.05, - "default_system_prompt": "You are MiniCPM, a helpful, concise and knowledgeable vision-language assistant. Answer directly and stay on task." + "default_top_k": 100, + "default_repetition_penalty": 1.10, + "default_system_prompt": "You are MiniCPM-V, a helpful, concise and knowledgeable vision-language assistant. Answer directly and stay on task." } ``` @@ -80,6 +117,20 @@ pip install -r ComfyUI/custom_nodes/comfyui-minicpm/requirements.txt * **List:** List the main objects visible in this image. * **Scene:** Describe the scene and setting of this image. * **Details:** What are the key details in this image? +* **Summarize:** Summarize the key content of this image in 1-2 sentences. +* **Emotion:** Describe the emotions or mood conveyed by this image. +* **Style:** Describe the artistic or visual style of this image. +* **Location:** Where might this image be taken? Analyze the setting or location. +* **Question:** What question could be asked based on this image? +* **Creative:** Describe this image as if writing the beginning of a short story. + +--- + +## Memory Management Options + +* **Keep in Memory:** Model stays loaded for faster subsequent runs +* **Clear After Run:** Model is unloaded after each run to save memory +* **Global Cache:** Model is cached globally and shared between nodes --- @@ -89,6 +140,8 @@ pip install -r ComfyUI/custom_nodes/comfyui-minicpm/requirements.txt * Use top-p (0.9) and top-k (80) sampling for natural output diversity. * Lower max tokens or precision (bf16/fp16) for faster generation on less powerful GPUs. * Memory modes help optimize VRAM usage: default, balanced, max savings. +* Transformers models offer better quality but use more memory. +* GGUF models are more memory-efficient but may have slightly lower quality. ---