ymc_node_joy

🤖 comfyui custom nodes to caption image with joy

Why

• use joy to caption image for aigc.
• use joy to caption image files in diretory to do sth. (eg. lora training)

Features

Nodes

• nodes show in console:

• nodes show in right mouse menu:

Install

# cd to comfyui/custom_nodes
git clone https://github.com/ymc-github/ymc_node_joy

• deps will be installed automatically if deps in requirements.txt were not installed when comfyui up

Required Models

Essential components:

• google/siglip-so400m-patch14-384 (Vision model)
• unsloth/Meta-Llama-3.1-8B-bnb-4bit or meta-llama/Meta-Llama-3.1-8B (LLM)
• Joy_caption/image_adapter.pt (Custom adapter)

<comfyui_root>/
├── models/
│   ├── clip/                    # SigLIP Vision Model
│   │   └── siglip-so400m-patch14-384/
│   ├── llm/                     # Llama Language Model
│   │   ├── Meta-Llama-3.1-8B-bnb-4bit/
│   │   └── Meta-Llama-3.1-8B/
│   └── Joy_caption/             # Custom Components
│       └── image_adapter.pt     # Dimension Adapter

1. SigLIP Vision Model (google/siglip-so400m-patch14-384)

International: https://huggingface.co/google/siglip-so400m-patch14-384
China Mirror: https://hf-mirror.com/google/siglip-so400m-patch14-384

2. Llama Language Models

4bit Quantized (unsloth/Meta-Llama-3.1-8B-bnb-4bit)

International: https://huggingface.co/unsloth/Meta-Llama-3.1-8B-bnb-4bit
China Mirror: https://hf-mirror.com/unsloth/Meta-Llama-3.1-8B-bnb-4bit

Original (meta-llama/Meta-Llama-3.1-8B)

International: https://huggingface.co/meta-llama/Meta-Llama-3.1-8B (Access approval required)
China Mirror: https://hf-mirror.com/meta-llama/Meta-Llama-3.1-8B

3. Image Adapter (Joy_caption/image_adapter.pt)

International: https://huggingface.co/spaces/fancyfeast/joy-caption-pre-alpha/tree/main/wpkklhc6
China Mirror: https://www.modelscope.cn/models/fireicewolf/joy-caption-pre-alpha/files

Usage

• you can find it in search box : double click + typing keywords (eg: joy, caption)
• you can find it in right mouse menu : ymc/caption
• you can find it in right mouse menu : utils/ymc/caption (as alias)

Demo

• workflow demo:

Based-on

• pypi package yors_comfyui_node_setup - setup comfyui custom nodes easily
• pypi package yors_pano_ansi_color - info msg in console with color in your comfyui custom nodes easily
• pypi package yors_pano_zero_field - set nodes input field to be HQ in your comfyui custom nodes easily

Published to Comfy registry

• get more details in publish_to_comfy.yml

• docs for publishing to comfy registey

• installed with comfy-cli ? comfy node registry-install ymc_node_joy

Author

name	email	desciption
yemiancheng	ymc.github@gmail.com	Main developer and code maintainer
chenxinghua	455758525@qq.com	Code reference from StartHua/Comfyui_CXH_joy_caption

License

MIT