diff --git a/README.MD b/README.MD index 1352970..b50219b 100644 --- a/README.MD +++ b/README.MD @@ -70,6 +70,9 @@ When this error has occurred, please check the network environment. ## Update **If the dependency package error after updating, please reinstall the relevant dependency packages.
+* Commit [QWenImage2Prompt](#QWenImage2Prompt) node, this node is repackage of the [ComfyUI_VLM_nodes](https://github.com/gokayfem/ComfyUI_VLM_nodes)'s ```UForm-Gen2 Qwen Node```, thanks to the original author. +* Commit [BooleanOperator](#BooleanOperator), [NumberCalculator](#NumberCalculator), [TextBox](#TextBox), [Integer](#Integer), [Float](#Float), [Boolean](#Boolean) nodes. These nodes can perform mathematical and logical operations. +* Commit [ExtendCanvasV2](#ExtendCanvasV2) node，support color value input. * Commit [AutoBrightness](#AutoBrightness) node，it can automatically adjust the brightness of image. * [CreateGradientMask](#CreateGradientMask) node add ```center``` option. * Commit [GetColorToneV2](#GetColorToneV2) node, can select the main and average colors for the background or body. @@ -562,6 +565,19 @@ The following changes have been made based on ImageScaleByAspectRatio: * scale_to_side: Allow scaling by specified size on long or short sides. * scale_to_length: When scale_by_side is set to "longest", this will be used as the length of the long edge of the image; When set to "shortest", it serves as the length of the short edge. + + +### QWenImage2Prompt +Inference the prompts based on the image. this node is repackage of the [ComfyUI_VLM_nodes](https://github.com/gokayfem/ComfyUI_VLM_nodes)'s ```UForm-Gen2 Qwen Node```, thanks to the original author. +Download model files from [huggingface](https://huggingface.co/unum-cloud/uform-gen2-qwen-500m) or [Baidu Netdisk](https://pan.baidu.com/s/1ztnVX_Sh800xsWZhMDe-Ww?pwd=esyt) to ```ComfyUI/models/LLavacheckpoints/files_for_uform_gen2_qwen``` folder. + +![image](image/qwen_image2prompt_example.png) + +Node Options: +* question: Prompt of UForm-Gen-QWen model. + + + ### PromptTagger Inference the prompts based on the image. it can replace key word for the prompt. This node currently uses Google Gemini API as the backend service. Please ensure that the network environment can use Gemini normally. Please apply for your API key on [Google AI Studio](https://makersuite.google.com/app/apikey), And fill it in ```api_key.ini```, this file is located in the root directory of the plug-in, and the default name is ```api_key.ini.example```. to use this file for the first time, you need to change the file suffix to ```.ini```. Open it using text editing software, fill in your API key after ```google_api_key=``` and save it. @@ -958,6 +974,38 @@ Used to provide assistance for workflow debugging. When running, the properties This node allows any type of input. + +### TextBox +![image](image/text_box_node.png) +Output a string. + +### Integer +![image](image/integer_node.png) +Output a integer value. + +### Float +![image](image/float_node.png) +Output a floating-point value with a precision of 5 decimal places. + +### Boolean +![image](image/boolean_node.png) +Output a boolean value. + +### NumberCalculator +![image](image/number_calculator_node.png) +Performs mathematical operations on two numeric values and outputs integer and floating point results*. Supported operations include```+```, ```-```, ```*```, ```/```, ```**```, ```//```, ```%```. + +* The input only supports boolean, integer, and floating point numbers, forcing in other data will result in error. + + +### BooleanOperator +![image](image/boolean_operator_node.png) +Perform a Boolean operation on two numeric values and output the result*. Supported operations include```==```, ```!=```, ```and```, ```or```, ```xor```, ```not```. + +* The input only supports boolean, integer, and floating point numbers, forcing in other data will result in error. The ```and``` operation between the values outputs a larger number, and the ```or``` operation outputs a smaller number. + + + # LayerMask ![image](image/layermask_nodes.png) diff --git a/README_CN.MD b/README_CN.MD index 2e612d4..a0cb55c 100644 --- a/README_CN.MD +++ b/README_CN.MD @@ -70,6 +70,9 @@ git clone https://github.com/chflame163/ComfyUI_LayerStyle.git ## 更新说明 **如果本插件更新后出现依赖包错误，请重新安装相关依赖包。 +* 添加 [QWenImage2Prompt](#QWenImage2Prompt)节点, 用本地模型反推提示词。(需要下载模型到models文件夹)。这个节点是[ComfyUI_VLM_nodes](https://github.com/gokayfem/ComfyUI_VLM_nodes)中的```UForm-Gen2 Qwen Node```节点的重新封装，感谢原作者。 +* 添加 [BooleanOperator](#BooleanOperator), [NumberCalculator](#NumberCalculator), [TextBox](#TextBox), [Integer](#Integer), [Float](#Float), [Boolean](#Boolean) 节点。这些节点可进行数学和逻辑运算。 +* 添加 [ExtendCanvasV2](#ExtendCanvasV2) 节点，支持color值输入。 * 添加 [AutoBrightness](#AutoBrightness) 节点，可自动调整图片亮度。 * [CreateGradientMask](#CreateGradientMask) 节点增加 ```center``` 选项。 * 添加 [GetColorToneV2](#GetColorToneV2) 节点, 可选择背景或主体的主色和平均色。 @@ -559,6 +562,17 @@ ImageScaleByAspectRatio的V2升级版 * scale_to_length: scale_by_side被设置为"longest"时，此项将作为是图像长边的长度; 设置为"shortest"时，作为短边的长度。 + +### QWenImage2Prompt +根据图片反推提示词。这个节点是[ComfyUI_VLM_nodes](https://github.com/gokayfem/ComfyUI_VLM_nodes)中的```UForm-Gen2 Qwen Node```节点的重新封装，感谢原作者。 +请从[huggingface](https://huggingface.co/unum-cloud/uform-gen2-qwen-500m)或者[百度网盘](https://pan.baidu.com/s/1ztnVX_Sh800xsWZhMDe-Ww?pwd=esyt)下载模型到```ComfyUI/models/LLavacheckpoints/files_for_uform_gen2_qwen```文件夹。 + +![image](image/qwen_image2prompt_example.png) + +节点选项说明: +* question: 对UForm-Gen-QWen模型的提示词。 + + ### PromptTagger 根据图片反推提示词，可以设置替换词。这个节点目前使用Google Gemini API作为后端服务，请确保网络环境可以正常使用Gemini。 请在[Google AI Studio](https://makersuite.google.com/app/apikey)申请你的API key, 并将其填到```api_key.ini```, 这个文件位于插件根目录下, 默认名字是```api_key.ini.example```, 初次使用这个文件需将文件后缀改为.ini。用文本编辑软件打开，在```google_api_key=```后面填入你的API key并保存。 @@ -955,6 +969,34 @@ cropped_mask: 裁切后的遮罩。 这个节点允许任意类型的输入。 +### TextBox +![image](image/text_box_node.png) +输出一段字符串。 + +### Integer +![image](image/integer_node.png) +输出一个整数。 + +### Float +![image](image/float_node.png) +输出一个浮点数，精度是小数点后5位。 + +### Boolean +![image](image/boolean_node.png) +输出一个布尔值。 + +### NumberCalculator +![image](image/number_calculator_node.png) +对两个数值进行数学运算并输出整数和浮点数结果*。支持的运算包括```+```、```-```、```*```、```/```、```**```、```//```、```%```。 + +* 输入仅支持布尔值、整数和浮点数，强行接入其他数据将导致错误。 + + +### BooleanOperator +![image](image/boolean_operator_node.png) +对两个数值进行布尔运算并输出结果*。支持的运算包括```==```、```!=```、```and```、```or```、```xor```、```not```。 + +* 输入仅支持布尔值、整数和浮点数，强行接入其他数据将导致错误。数值之间的```and```运算输出较大的数，```or```运算输出较小的数。 # LayerMask diff --git a/image/boolean_node.png b/image/boolean_node.png new file mode 100644 index 0000000..eb1e2ab Binary files /dev/null and b/image/boolean_node.png differ diff --git a/image/boolean_operator_node.png b/image/boolean_operator_node.png new file mode 100644 index 0000000..f3ff1e3 Binary files /dev/null and b/image/boolean_operator_node.png differ diff --git a/image/data_nodes_example.png b/image/data_nodes_example.png new file mode 100644 index 0000000..c17bc75 Binary files /dev/null and b/image/data_nodes_example.png differ diff --git a/image/float_node.png b/image/float_node.png new file mode 100644 index 0000000..fd9190e Binary files /dev/null and b/image/float_node.png differ diff --git a/image/integer_node.png b/image/integer_node.png new file mode 100644 index 0000000..fb4f4e4 Binary files /dev/null and b/image/integer_node.png differ diff --git a/image/number_calculator_node.png b/image/number_calculator_node.png new file mode 100644 index 0000000..92751ae Binary files /dev/null and b/image/number_calculator_node.png differ diff --git a/image/qwen_image2prompt_example.png b/image/qwen_image2prompt_example.png new file mode 100644 index 0000000..3264824 Binary files /dev/null and b/image/qwen_image2prompt_example.png differ diff --git a/image/text_box_node.png b/image/text_box_node.png new file mode 100644 index 0000000..aeec959 Binary files /dev/null and b/image/text_box_node.png differ diff --git a/py/Qwen_image2prompt.py b/py/Qwen_image2prompt.py new file mode 100644 index 0000000..712a00e --- /dev/null +++ b/py/Qwen_image2prompt.py @@ -0,0 +1,114 @@ +import os.path +from pathlib import Path +from transformers import AutoModel, AutoProcessor, StoppingCriteria, StoppingCriteriaList +import torch +from PIL import Image +from torchvision.transforms import ToPILImage +from huggingface_hub import snapshot_download +import folder_paths + +# Define the directory for saving files related to uform-gen2-qwen +# files_for_uform_gen2_qwen = Path(folder_paths.folder_names_and_paths["LLavacheckpoints"][0][0]) / "files_for_uform_gen2_qwen" +files_for_uform_gen2_qwen = Path(os.path.join(folder_paths.models_dir, "LLavacheckpoints", "files_for_uform_gen2_qwen")) +files_for_uform_gen2_qwen.mkdir(parents=True, exist_ok=True) # Ensure the directory exists + +class StopOnTokens(StoppingCriteria): + def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool: + stop_ids = [151645] # Define stop tokens as per your model's specifics + for stop_id in stop_ids: + if input_ids[0][-1] == stop_id: + return True + return False + +class UformGen2QwenChat: + def __init__(self): + self.model_path = snapshot_download("unum-cloud/uform-gen2-qwen-500m", + local_dir=files_for_uform_gen2_qwen, + force_download=False, # Set to True if you always want to download, regardless of local copy + local_files_only=False, # Set to False to allow downloading if not available locally + local_dir_use_symlinks="auto") # or set to True/False based on your symlink preference + self.device = "cuda:0" if torch.cuda.is_available() else "cpu" + self.model = AutoModel.from_pretrained(self.model_path, trust_remote_code=True).to(self.device) + self.processor = AutoProcessor.from_pretrained(self.model_path, trust_remote_code=True) + + def chat_response(self, message, history, image_path): + stop = StopOnTokens() + messages = [{"role": "system", "content": "You are a helpful Assistant."}] + + for user_msg, assistant_msg in history: + messages.append({"role": "user", "content": user_msg}) + messages.append({"role": "assistant", "content": assistant_msg}) + + if len(messages) == 1: + message = f" {message}" + + messages.append({"role": "user", "content": message}) + + model_inputs = self.processor.tokenizer.apply_chat_template( + messages, + add_generation_prompt=True, + return_tensors="pt" + ) + + image = Image.open(image_path) # Load image using PIL + image_tensor = ( + self.processor.feature_extractor(image) + .unsqueeze(0) + ) + + attention_mask = torch.ones( + 1, model_inputs.shape[1] + self.processor.num_image_latents - 1 + ) + + model_inputs = { + "input_ids": model_inputs, + "images": image_tensor, + "attention_mask": attention_mask + } + + model_inputs = {k: v.to(self.device) for k, v in model_inputs.items()} + + output = self.model.generate( + **model_inputs, + max_new_tokens=1024, + stopping_criteria=StoppingCriteriaList([stop]) + ) + + response_text = self.processor.tokenizer.decode(output[0], skip_special_tokens=True) + return response_text + +# Example of integrating UformGen2QwenChat into a node-like structure +class QWenImage2Prompt: + def __init__(self): + self.chat_model = UformGen2QwenChat() + + @classmethod + def INPUT_TYPES(cls): + return { + "required": { + "image": ("IMAGE",), + "question": ("STRING", {"multiline": False, "default": "describe this image",},), + }, + } + + RETURN_TYPES = ("STRING",) + RETURN_NAMES = ("text",) + FUNCTION = "uform_gen2_qwen_chat" + CATEGORY = '😺dzNodes/LayerUtility' + + def uform_gen2_qwen_chat(self, image, question): + history = [] # Example empty history + pil_image = ToPILImage()(image[0].permute(2, 0, 1)) + temp_path = files_for_uform_gen2_qwen / "temp.png" + pil_image.save(temp_path) + + response = self.chat_model.chat_response(question, history, temp_path) + return (response.split("assistant\n", 1)[1], ) + +NODE_CLASS_MAPPINGS = { + "LayerUtility: QWenImage2Prompt": QWenImage2Prompt +} + +NODE_DISPLAY_NAME_MAPPINGS = { + "LayerUtility: QWenImage2Prompt": "LayerUtility: QWenImage2Prompt" +} \ No newline at end of file diff --git a/py/data_nodes.py b/py/data_nodes.py new file mode 100644 index 0000000..892fe3e --- /dev/null +++ b/py/data_nodes.py @@ -0,0 +1,163 @@ +from .imagefunc import AnyType + +any = AnyType("*") + + +class BooleanOperator: + def __init__(self): + pass + @classmethod + def INPUT_TYPES(self): + operator_list = ["==", "!=", "and", "or", "xor", "not(a)"] + return {"required": { + "a": (any, {}), + "b": (any, {}), + "operator": (operator_list,), + },} + + RETURN_TYPES = ("BOOLEAN",) + RETURN_NAMES = ("boolean",) + FUNCTION = 'bool_operator_node' + CATEGORY = '😺dzNodes/LayerUtility' + + def bool_operator_node(self, a, b, operator): + ret_value = False + if operator == "==": + ret_value = a == b + if operator == "!=": + ret_value = a != b + if operator == "and": + ret_value = a and b + if operator == "or": + ret_value = a or b + if operator == "xor": + ret_value = not(a == b) + if operator == "not(a)": + ret_value = not a + + return (ret_value,) + +class NumberCalculator: + def __init__(self): + pass + @classmethod + def INPUT_TYPES(self): + operator_list = ["+", "-", "*", "/", "**", "//", "%" ] + return {"required": { + "a": (any, {}), + "b": (any, {}), + "operator": (operator_list,), + },} + + RETURN_TYPES = ("INT", "FLOAT",) + RETURN_NAMES = ("int", "float",) + FUNCTION = 'number_calculator_node' + CATEGORY = '😺dzNodes/LayerUtility' + + def number_calculator_node(self, a, b, operator): + ret_value = 0 + if operator == "+": + ret_value = a + b + if operator == "-": + ret_value = a - b + if operator == "*": + ret_value = a * b + if operator == "/": + ret_value = a / b + if operator == "**": + ret_value = a ** b + if operator == "//": + ret_value = a // b + if operator == "%": + ret_value = a % b + + return (int(ret_value), float(ret_value),) + +class TextBoxNode: + def __init__(self): + pass + @classmethod + def INPUT_TYPES(self): + return {"required": { + "text": ("STRING", {"multiline": True}), + },} + + RETURN_TYPES = ("STRING",) + RETURN_NAMES = ("text",) + FUNCTION = 'text_box_node' + CATEGORY = '😺dzNodes/LayerUtility' + + def text_box_node(self, text): + return (text,) + +class IntegerNode: + def __init__(self): + pass + @classmethod + def INPUT_TYPES(self): + return {"required": { + "int_value":("INT", {"default": 0, "min": -99999999999999999999, "max": 99999999999999999999, "step": 1}), + },} + + RETURN_TYPES = ("INT",) + RETURN_NAMES = ("int",) + FUNCTION = 'integer_node' + CATEGORY = '😺dzNodes/LayerUtility' + + def integer_node(self, int_value): + return (int_value,) + +class FloatNode: + def __init__(self): + pass + @classmethod + def INPUT_TYPES(self): + return {"required": { + "float_value": ("FLOAT", {"default": 0, "min": -99999999999999999999, "max": 99999999999999999999, "step": 0.00001}), + },} + + RETURN_TYPES = ("FLOAT",) + RETURN_NAMES = ("float",) + FUNCTION = 'float_node' + CATEGORY = '😺dzNodes/LayerUtility' + + def float_node(self, float_value): + return (float_value,) + +class BooleanNode: + def __init__(self): + pass + @classmethod + def INPUT_TYPES(self): + return {"required": { + "bool_value": ("BOOLEAN", {"default": False}), + },} + + RETURN_TYPES = ("BOOLEAN",) + RETURN_NAMES = ("boolean",) + FUNCTION = 'boolean_node' + CATEGORY = '😺dzNodes/LayerUtility' + + def boolean_node(self, bool_value): + return (bool_value,) + + + + +NODE_CLASS_MAPPINGS = { + "LayerUtility: BooleanOperator": BooleanOperator, + "LayerUtility: NumberCalculator": NumberCalculator, + "LayerUtility: TextBox": TextBoxNode, + "LayerUtility: Integer": IntegerNode, + "LayerUtility: Float": FloatNode, + "LayerUtility: Boolean": BooleanNode +} + +NODE_DISPLAY_NAME_MAPPINGS = { + "LayerUtility: BooleanOperator": "LayerUtility: BooleanOperator", + "LayerUtility: NumberCalculator": "LayerUtility: NumberCalculator", + "LayerUtility: TextBox": "LayerUtility: TextBox", + "LayerUtility: Integer": "LayerUtility: Integer", + "LayerUtility: Float": "LayerUtility: Float", + "LayerUtility: Boolean": "LayerUtility: Boolean" +} \ No newline at end of file diff --git a/py/print_info.py b/py/print_info.py index 9f57d17..f8c4843 100644 --- a/py/print_info.py +++ b/py/print_info.py @@ -16,11 +16,12 @@ class PrintInfo: CATEGORY = '😺dzNodes/LayerUtility' RETURN_TYPES = ("STRING",) - RETURN_NAMES = ("info",) + RETURN_NAMES = ("text",) FUNCTION = "print_info" + OUTPUT_NODE = True def print_info(self, anything=None): - value = f'PrintInfo: \n Input type = {type(anything)}' + value = f'PrintInfo:\nInput type = {type(anything)}' if isinstance(anything, torch.Tensor): value += f"\n Input dim = {anything.dim()}, shape[0] = {anything.shape[0]} \n" for i in range(anything.shape[0]): @@ -38,7 +39,11 @@ class PrintInfo: value = value + str(anything) + "\n" except Exception: value = 'source exists, but could not be serialized.' + else: + value = 'source does not exist.' + log(value) + return (value,) NODE_CLASS_MAPPINGS = { diff --git a/workflow/data_nodes_example.json b/workflow/data_nodes_example.json new file mode 100644 index 0000000..3978381 --- /dev/null +++ b/workflow/data_nodes_example.json @@ -0,0 +1,349 @@ +{ + "last_node_id": 11, + "last_link_id": 12, + "nodes": [ + { + "id": 1, + "type": "LayerUtility: Integer", + "pos": [ + 843, + 477 + ], + "size": { + "0": 247.19744873046875, + "1": 61.07960510253906 + }, + "flags": {}, + "order": 0, + "mode": 0, + "outputs": [ + { + "name": "int", + "type": "INT", + "links": [ + 1 + ], + "shape": 3, + "slot_index": 0 + } + ], + "properties": { + "Node name for S&R": "LayerUtility: Integer" + }, + "widgets_values": [ + 9 + ] + }, + { + "id": 7, + "type": "Display Any (rgthree)", + "pos": [ + 1498, + 452 + ], + "size": { + "0": 210, + "1": 76.0000228881836 + }, + "flags": {}, + "order": 5, + "mode": 0, + "inputs": [ + { + "name": "source", + "type": "*", + "link": 5, + "dir": 3 + } + ], + "properties": { + "Node name for S&R": "Display Any (rgthree)" + }, + "widgets_values": [ + "" + ] + }, + { + "id": 8, + "type": "Display Any (rgthree)", + "pos": [ + 1499, + 564 + ], + "size": { + "0": 210.36199951171875, + "1": 76 + }, + "flags": {}, + "order": 6, + "mode": 0, + "inputs": [ + { + "name": "source", + "type": "*", + "link": 6, + "dir": 3 + } + ], + "properties": { + "Node name for S&R": "Display Any (rgthree)" + }, + "widgets_values": [ + "" + ] + }, + { + "id": 5, + "type": "LayerUtility: NumberCalculator", + "pos": [ + 1168, + 534 + ], + "size": { + "0": 252, + "1": 83.83000183105469 + }, + "flags": {}, + "order": 3, + "mode": 0, + "inputs": [ + { + "name": "a", + "type": "*", + "link": 1 + }, + { + "name": "b", + "type": "*", + "link": 2 + } + ], + "outputs": [ + { + "name": "int", + "type": "INT", + "links": [ + 5 + ], + "shape": 3, + "slot_index": 0 + }, + { + "name": "float", + "type": "FLOAT", + "links": [ + 6 + ], + "shape": 3, + "slot_index": 1 + } + ], + "properties": { + "Node name for S&R": "LayerUtility: NumberCalculator" + }, + "widgets_values": [ + "*" + ] + }, + { + "id": 3, + "type": "LayerUtility: Float", + "pos": [ + 841, + 627 + ], + "size": { + "0": 245.2032928466797, + "1": 60.5140495300293 + }, + "flags": {}, + "order": 1, + "mode": 0, + "outputs": [ + { + "name": "float", + "type": "FLOAT", + "links": [ + 2, + 11 + ], + "shape": 3, + "slot_index": 0 + } + ], + "properties": { + "Node name for S&R": "LayerUtility: Float" + }, + "widgets_values": [ + 2.0331300000000003 + ] + }, + { + "id": 11, + "type": "LayerUtility: Float", + "pos": [ + 841, + 794 + ], + "size": { + "0": 245.2032928466797, + "1": 60.5140495300293 + }, + "flags": {}, + "order": 2, + "mode": 0, + "outputs": [ + { + "name": "float", + "type": "FLOAT", + "links": [ + 12 + ], + "shape": 3, + "slot_index": 0 + } + ], + "properties": { + "Node name for S&R": "LayerUtility: Float" + }, + "widgets_values": [ + 2.0331300000000003 + ] + }, + { + "id": 10, + "type": "LayerUtility: BooleanOperator", + "pos": [ + 1175, + 739 + ], + "size": { + "0": 250.57093811035156, + "1": 92.19929504394531 + }, + "flags": {}, + "order": 4, + "mode": 0, + "inputs": [ + { + "name": "a", + "type": "*", + "link": 11 + }, + { + "name": "b", + "type": "*", + "link": 12 + } + ], + "outputs": [ + { + "name": "boolean", + "type": "BOOLEAN", + "links": [ + 10 + ], + "shape": 3, + "slot_index": 0 + } + ], + "properties": { + "Node name for S&R": "LayerUtility: BooleanOperator" + }, + "widgets_values": [ + "==" + ] + }, + { + "id": 9, + "type": "Display Any (rgthree)", + "pos": [ + 1500, + 754 + ], + "size": { + "0": 210, + "1": 76.0000228881836 + }, + "flags": {}, + "order": 7, + "mode": 0, + "inputs": [ + { + "name": "source", + "type": "*", + "link": 10, + "dir": 3 + } + ], + "properties": { + "Node name for S&R": "Display Any (rgthree)" + }, + "widgets_values": [ + "" + ] + } + ], + "links": [ + [ + 1, + 1, + 0, + 5, + 0, + "*" + ], + [ + 2, + 3, + 0, + 5, + 1, + "*" + ], + [ + 5, + 5, + 0, + 7, + 0, + "*" + ], + [ + 6, + 5, + 1, + 8, + 0, + "*" + ], + [ + 10, + 10, + 0, + 9, + 0, + "*" + ], + [ + 11, + 3, + 0, + 10, + 0, + "*" + ], + [ + 12, + 11, + 0, + 10, + 1, + "*" + ] + ], + "groups": [], + "config": {}, + "extra": {}, + "version": 0.4 +} \ No newline at end of file