avatar-graph-comfyui

A custom nodes module for creating real-time interactive avatars powered by blender bpy mesh api + Avatech Shape Flow runtime.

Demo

➡️Open in Avatech Labs	➡️Open in Avatech Labs	➡️Open in Avatech Labs	➡️Open in Avatech Labs

➡️Open in Avatech Labs	➡️Open in Avatech Labs	➡️Open in Avatech Labs	➡️Open in Avatech Labs

Contents

• Workflow Template
  • Template01 - Simple Shape Flow
• Image Preprocess Guide
  • Character Gen Prompting Guide
  • Mouth Open Guide (Inpaint)
• Custom Nodes
  • Image Segmentation Nodes
  • Mesh Edit Nodes
  • Shape Keys Nodes
  • Avatar Output Nodes
• Shape Flow
• Installation
• Development

Workflow Template

Template01 - Simple Shape Flow

To enable the character to blink eyes and talking.

🎯Notice

For optimal results, please input a character image with an open mouth and a minimum resolution of 768x768. This higher resolution will enable the tool to accurately recognize and work with facial features.

💡Generate new image Guide

💡Make your character mouth open Guide

Download: 📂Template01 - Simple Shape Flow

Download: 📂Template01 - ControlNet Gen

If you don't want to modify any values in the custom nodes, you can download the ControlNet Gen Template to generate your own image.

Template01 - Nodes Value Setting Guide

Template01 - Nodes Value Setting Guide

Basic Eyeblink & Talking

1. Click [Segmentation (SAM)]/ Edit prompt button

2. Add new layer and rename

3. Drag layer to [Create Mesh Layer]/image

4. [Create Mesh Layer]/ face_threshold, shape_threshold, To control mesh threshold, recommend value: 0.6~0.7

5. [Create Mesh Layer]/ scale_x, scale_y, extrude_x, extrude_y, To control mesh threshold, recommend value: 1.2~1.4

6. [Modify Shape Key]/ rotate Setting Reference, If Head tilted to the left, set a positive number angle

0	5	-5	-15

Template01 - ControlNet Gen Guide

Place normal and openpose image with reference to images.

Image Preprocess Guide

💡If you want to generate a new character image

you can download this Template and refer to the Guide!

Character Gen Prompting Guide

Character Gen Prompting Guide

🎯Notice

We need a character image with an open mouth and enable the tool to easily recognize facial features, so please add to the prompt:

looking at viewer, detailed face, open mouth, [smile], solo,eye-level angle

Download: 📂Character Gen Template

Feel free to change any checkpoint model that suits your needs.

💡If you have a character image but it's not mouth open

you can download this Template and refer to the Guide!

Mouth Open Guide (Inpaint)

Mouth Open Guide (Inpaint)

To maintain consistency with the base image, it is recommended to utilize a checkpoint model that aligns with its style.

Download: 📂MouthOpen Template

Inpaint Demonstration

https://github.com/avatechai/avatar-graph-comfyui/assets/48451938/ff48c3d9-7292-4505-8993-8f117cee34ff

Recommend Checkpoint Model List

Anime Style SD1.5

• https://civitai.com/models/35960/flat-2d-animerge
• https://civitai.com/models/24149/mistoonanime
• https://civitai.com/models/22364/kizuki-anime-hentai-checkpoint

Realistic Style SD1.5

• https://civitai.com/models/4201/realistic-vision-v51
• https://civitai.com/models/49463/am-i-real
• https://civitai.com/models/43331/majicmix-realistic

Custom Nodes

Expand to see all the available nodes description Mesh Edit Nodes Shape Keys Nodes Avatar Output Nodes

Image Segmentation Nodes

Image Segmentation Nodes

Name	Description	Preview
Segmentation (SAM)	Integrative SAM node allowing you to directly select and create multiple image segment output.

Mesh Edit Nodes

Mesh Edit Nodes

Name	Description	Preview
Create Mesh Layer	Create a mesh object from the input images (usually a segmented part of the entire image)
Join Meshes	Combine multiple meshes into a single mesh object
Match Texture Aspect Ratio	Since the mesh is created in 1:1 aspect ratio, a re-scale is needed at the end of the operation
Plane Texture Unwrap	Will perform mesh face fill and UV Cube project on the target plane mesh, scaled to bounds.

Shape Keys Nodes

Shape Keys Nodes

Name	Description	Preview
Mesh Modify Shape Key	Given shape key name & target vertex_group, modify the vertex / all vertex’s transform
Create Shape Flow	Create runtime shape flow graph, allowing interactive inputs affecting shape keys value in runtime

Avatar Output Nodes

Avatar Output Nodes

Name	Description	Preview
Avatar Main Output	The primary output of the .ava file. The embeded Avatar View will auto update with this node's output

Shape Flow

Installation

Clone the repository to custom_nodes in your ComfyUI directory:

1. cd custom_nodes

2. git clone https://github.com/avatechgg/avatar-graph-comfyui.git

3. Install deps cd avatar-graph-comfyui && python -m pip install -r requirements.txt

4. Restart comfyui

5. Run comfyui with enable-cors-header python main.py --enable-cors-header or (mac)python main.py --force-fp16 --enable-cors-header

Development

If you are interested in contributing expand to see development details

For comfyui frontend extension, frontend js located at avatar-graph-comfyui/js

Web stack used: vanjs tailwindcss

Install deps

pnpm i

Run the dev command to start the tailwindcss watcher

pnpm dev

For each changes, simply refresh the comfyui page to see the changes.

p.s. For tailwind autocomplete, add the following to your vscode settings.json.

{
    "tailwindCSS.experimental.classRegex": [
        ["class\\s?:\\s?([\\s\\S]*)", "(?:\"|')([^\"']*)(?:\"|')"]
    ]
}