diff --git a/README.md b/README.md index 73057c0..d8f7205 100644 --- a/README.md +++ b/README.md @@ -29,11 +29,15 @@ The Load Vision Model node is able to load any of these model types, but it will The other model loading nodes are for specific model types and will filter the list to just that model type. -The text generation nodes are model-specific. Pixtral seems to be the only one out of these that currently supports repetition penalty. The prompts used in these nodes are raw, and need to include the special tokens to create an instruction or chat template. I plan to add some more nodes for creating prompts following a template like this, though. +The text generation nodes are model-specific. Pixtral seems to be the only one out of these that currently supports repetition penalty. I plan to add some more nodes for creating prompts following a chat sequence. + +The Generate Text with Pixtral node can take the `[IMG]` special token in the prompt, and should include it for as many images as you want to process in a single prompt. If these tags aren't added, they will be automatically added to the beginning of your prompt. The Llama and Molmo models will add the images to the beginning of the prompt automatically, and while they do support processing multiple images at once, they don't support including multiple images in different places in the prompt like this. + +System prompts are optional. I didn't include them for Pixtral because the current setup is already using the `[INST]` special token, so the pixtral prompting is already like a system prompt rather than a user conversation. I might change this later. Use `trust_remote_code` at your own risk. (I think Molmo looks safe, though) -I will probably rename this repo (again) pretty soon, considering it's expanded beyond just being for Pixtral. +I will probably rename this repo (again) pretty soon, considering it's expanded beyond just being for Pixtral. Is "PixtralLlamaMolmoVision" too much? ...Nah. ## Installation @@ -57,19 +61,26 @@ Unfortunately, the Pixtral nf4 model has considerably degraded performance on so Example Pixtral image captioning (not saving the output to a text file in this example): ![Example Pixtral image captioning workflow](examples/pixtral_caption_example.jpg) -Both models should work very well for image captioning, even in 4-bit quantization. You can also customize your captioning instructions. +All of these models should work very well for image captioning, even in 4-bit quantization. You can also customize your captioning instructions. Larger images might not work as well with Pixtral, so scaling them down to something like 512 x 512 before sending them to the text generation node might be a good idea. It's also worth noting that the nf4 Pixtral model has significantly degraded performance on images which consist of mainly text. + +Example Molmo dataset captioning for a LoRA: +![Example dataset captioning workflow](examples/dataset_captioning_example.jpg) + +This workflow sends a list of images to the image generation node to caption each of them sequentially, and creates images and text files in a folder with names `1.png`, `1.txt`, etc for easy LoRA training setup. + +Note that for captioning each image separately, this input should be a list, not a batch of images, because these models can take multiple images as input for a single generation. Currently these nodes don't support batched text generation, but I might add that in the future. Doing one text generation task at a time is probably better for people with normal amounts of VRAM though. Example Pixtral image comparison: ![Example Pixtral image comparison workflow](examples/pixtral_comparison_example.jpg) I haven't been able to get image comparison to work well at all with Llama Vision. It doesn't give any errors, but the multi-image understanding just isn't there. The image tokens have to be **before** the question/instruction and consecutive for the model to even be able to see both images at once (I found this out by looking at the image preprocessor cross-attention implementation), and even then, it seems to randomly mix up which is the first/second, left/right, the colors between them and other details. It doesn't seem usable for purposes involving two images in the same message, in my opinion. Not sure whether the non-quantized model is better at this. -Since Pixtral directly tokenizes the input images, it's able to handle them inline in the context, with any number of images of any aspect ratio, but it's limited by token lengths, since each image can be around 1000 tokens. +Since Pixtral directly tokenizes the input images, it's able to handle them inline in the context, with any number of images of any aspect ratio, but it's limited by token lengths, since each image can be around 1000 - 4000 tokens. Example Llama Vision object detection with bounding box: ![Example Llama Vision object detection with bounding box workflow](examples/llama_vision_bounding_box_example.jpg) -Both models kind of work for this, but not that well. They definitely have some understanding of the positions of objects in the image, though. Maybe it needs a better prompt. Or a non-quantized model. Or a finetune. But it does sometimes work. +Both Pixtral and Llama kind of work for this, but not that well. They definitely have some understanding of the positions of objects in the image, though. Maybe it needs a better prompt. Or a non-quantized model. Or a finetune. But it does sometimes work. Surprisingly, Molmo is pretty bad at this, though it is capable of pointing and counting. Example Molmo counting: ![Example Molmo counting workflow](examples/molmo_count_example.jpg) diff --git a/examples/dataset_captioning_example.jpg b/examples/dataset_captioning_example.jpg new file mode 100644 index 0000000..7b31bd6 Binary files /dev/null and b/examples/dataset_captioning_example.jpg differ diff --git a/examples/dataset_captioning_workflow.json b/examples/dataset_captioning_workflow.json new file mode 100644 index 0000000..cf861dd --- /dev/null +++ b/examples/dataset_captioning_workflow.json @@ -0,0 +1,499 @@ +{ + "last_node_id": 24, + "last_link_id": 24, + "nodes": [ + { + "id": 17, + "type": "Note", + "pos": { + "0": 1864, + "1": 871 + }, + "size": [ + 260.5903497028885, + 137.34932108462544 + ], + "flags": {}, + "order": 0, + "mode": 0, + "inputs": [], + "outputs": [], + "properties": {}, + "widgets_values": [ + "Folder within the ComfyUI output folder where you want the images + captions to be saved with the same name (1.png, 1.txt, etc.)" + ], + "color": "#432", + "bgcolor": "#653" + }, + { + "id": 16, + "type": "Note", + "pos": { + "0": 1864, + "1": 431 + }, + "size": [ + 282.40853152107, + 132.34932108462544 + ], + "flags": {}, + "order": 1, + "mode": 0, + "inputs": [], + "outputs": [], + "properties": {}, + "widgets_values": [ + "Path here should be the same as the path below, but as a relative path from where you are running ComfyUI, or as an absolute path" + ], + "color": "#432", + "bgcolor": "#653" + }, + { + "id": 18, + "type": "Note", + "pos": { + "0": 267, + "1": 795 + }, + "size": [ + 279.31762243016135, + 145.62204835735247 + ], + "flags": {}, + "order": 2, + "mode": 0, + "inputs": [], + "outputs": [], + "properties": {}, + "widgets_values": [ + "Path to a folder with the images you want to caption (filenames don't matter).\nimage_load_cap = 0 will load all the images from the folder." + ], + "color": "#432", + "bgcolor": "#653" + }, + { + "id": 24, + "type": "MolmoGenerateText", + "pos": { + "0": 990, + "1": 401 + }, + "size": { + "0": 400, + "1": 362 + }, + "flags": {}, + "order": 7, + "mode": 0, + "inputs": [ + { + "name": "molmo_model", + "type": "VISION_MODEL", + "link": 22 + }, + { + "name": "images", + "type": "IMAGE", + "link": 23 + } + ], + "outputs": [ + { + "name": "STRING", + "type": "STRING", + "links": [ + 24 + ], + "shape": 3, + "slot_index": 0 + } + ], + "properties": { + "Node name for S&R": "MolmoGenerateText" + }, + "widgets_values": [ + "", + "Describe this image in detail. ", + 256, + true, + 0.3, + 0.9, + 40, + "<|endoftext|>", + 3561946545, + "randomize", + false + ] + }, + { + "id": 23, + "type": "MolmoModelLoader", + "pos": { + "0": 572, + "1": 407 + }, + "size": { + "0": 315, + "1": 58 + }, + "flags": {}, + "order": 3, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "VISION_MODEL", + "type": "VISION_MODEL", + "links": [ + 22 + ], + "shape": 3, + "slot_index": 0 + } + ], + "properties": { + "Node name for S&R": "MolmoModelLoader" + }, + "widgets_values": [ + "molmo-7B-D-bnb-4bit" + ] + }, + { + "id": 19, + "type": "Note", + "pos": { + "0": 290, + "1": 368 + }, + "size": [ + 264.5454611197921, + 134.54546865653583 + ], + "flags": {}, + "order": 4, + "mode": 0, + "inputs": [], + "outputs": [], + "properties": {}, + "widgets_values": [ + "Llama 3.2 11B Vision and Molmo are probably better quality than Pixtral for this" + ], + "color": "#432", + "bgcolor": "#653" + }, + { + "id": 8, + "type": "Save Text File", + "pos": { + "0": 1537, + "1": 407 + }, + "size": [ + 303.044895157434, + 174 + ], + "flags": {}, + "order": 9, + "mode": 0, + "inputs": [ + { + "name": "text", + "type": "STRING", + "link": 24, + "widget": { + "name": "text" + } + }, + { + "name": "filename_prefix", + "type": "STRING", + "link": 9, + "widget": { + "name": "filename_prefix" + } + } + ], + "outputs": [], + "properties": { + "Node name for S&R": "Save Text File" + }, + "widgets_values": [ + "", + ".\\ComfyUI\\output\\images_with_captions", + "", + "", + 0, + ".txt", + "utf-8" + ] + }, + { + "id": 4, + "type": "ListCounter //Inspire", + "pos": { + "0": 954, + "1": 864 + }, + "size": { + "0": 210, + "1": 58 + }, + "flags": {}, + "order": 6, + "mode": 0, + "inputs": [ + { + "name": "signal", + "type": "*", + "link": 3 + } + ], + "outputs": [ + { + "name": "INT", + "type": "INT", + "links": [ + 1 + ], + "slot_index": 0, + "shape": 3 + } + ], + "properties": { + "Node name for S&R": "ListCounter //Inspire" + }, + "widgets_values": [ + 1 + ] + }, + { + "id": 6, + "type": "SomethingToString", + "pos": { + "0": 1223, + "1": 865 + }, + "size": [ + 210, + 82 + ], + "flags": {}, + "order": 8, + "mode": 0, + "inputs": [ + { + "name": "input", + "type": "*", + "link": 1 + } + ], + "outputs": [ + { + "name": "STRING", + "type": "STRING", + "links": [ + 9, + 14, + 15 + ], + "slot_index": 0, + "shape": 3 + } + ], + "properties": { + "Node name for S&R": "SomethingToString" + }, + "widgets_values": [ + "", + "" + ] + }, + { + "id": 1, + "type": "LoadImageListFromDir //Inspire", + "pos": { + "0": 570, + "1": 786 + }, + "size": { + "0": 315, + "1": 170 + }, + "flags": {}, + "order": 5, + "mode": 0, + "inputs": [], + "outputs": [ + { + "name": "IMAGE", + "type": "IMAGE", + "links": [ + 3, + 13, + 23 + ], + "slot_index": 0, + "shape": 6 + }, + { + "name": "MASK", + "type": "MASK", + "links": null, + "shape": 6 + }, + { + "name": "FILE PATH", + "type": "STRING", + "links": null, + "shape": 6 + } + ], + "properties": { + "Node name for S&R": "LoadImageListFromDir //Inspire" + }, + "widgets_values": [ + "E:\\datasets\\example", + 0, + 0, + false + ] + }, + { + "id": 13, + "type": "> Save Image", + "pos": { + "0": 1540, + "1": 790 + }, + "size": [ + 299.77216788470673, + 406 + ], + "flags": {}, + "order": 10, + "mode": 0, + "inputs": [ + { + "name": "images", + "type": "IMAGE", + "link": 13 + }, + { + "name": "filename_opt", + "type": "STRING", + "link": 14, + "widget": { + "name": "filename_opt" + } + }, + { + "name": "filename_prefix", + "type": "STRING", + "link": 15, + "widget": { + "name": "filename_prefix" + } + } + ], + "outputs": [], + "properties": { + "Node name for S&R": "> Save Image" + }, + "widgets_values": [ + "", + "images_with_captions", + false, + false, + "png", + 100, + "" + ] + } + ], + "links": [ + [ + 1, + 4, + 0, + 6, + 0, + "*" + ], + [ + 3, + 1, + 0, + 4, + 0, + "*" + ], + [ + 9, + 6, + 0, + 8, + 1, + "STRING" + ], + [ + 13, + 1, + 0, + 13, + 0, + "IMAGE" + ], + [ + 14, + 6, + 0, + 13, + 1, + "STRING" + ], + [ + 15, + 6, + 0, + 13, + 2, + "STRING" + ], + [ + 22, + 23, + 0, + 24, + 0, + "VISION_MODEL" + ], + [ + 23, + 1, + 0, + 24, + 1, + "IMAGE" + ], + [ + 24, + 24, + 0, + 8, + 0, + "STRING" + ] + ], + "groups": [], + "config": {}, + "extra": { + "ds": { + "scale": 1.1, + "offset": [ + -123.68125879379771, + -106.53113926644365 + ] + } + }, + "version": 0.4 +} \ No newline at end of file