avatar-graph-comfyui

A custom nodes module for creating real-time interactive avatars powered by blender bpy mesh api + Avatech Shape Flow runtime.

Warning

We are still making changes to the nodes and demo templates, please stay tuned.

Demo

Interact 👆	Interact 👆	Interact 👆	Interact 👆
Interact 👆	Interact 👆	Interact 👆	Interact 👆

How to?

• Basic Rigging Workflow Template
• Best Practices for image input
• Custom Nodes List
• Shape Flow
• Installation
• Development
• Join Discord 💬

Basic Rigging Workflow Template

Creating an eye blink and lipsync avatar

For optimal results, please input a character image with an open mouth and a minimum resolution of 768x768. This higher resolution will enable the tool to accurately recognize and work with facial features.

Download: Template01 - Simple Shape Flow

Download: Template01 - ControlNet Gen

Template01 - Nodes Value Setting Guide

Template01 - Nodes Value Setting Guide

Basic Eyeblink & Talking

1. Click [Segmentation (SAM)]/ Edit prompt button

2. Add new layer and rename

3. Drag layer to [Create Mesh Layer]/image

4. [Create Mesh Layer]/ face_threshold, shape_threshold, To control mesh threshold, recommend value: 0.6~0.7

5. [Create Mesh Layer]/ scale_x, scale_y, extrude_x, extrude_y, To control mesh threshold, recommend value: 1.2~1.4

6. [Modify Shape Key]/ rotate Setting Reference, If Head tilted to the left, set a positive number angle

0	5	-5	-15

Best practices for image input

Generate a new character image

We need a character image with an open mouth and enable the tool to easily recognize facial features, so please add to the prompt:

looking at viewer, detailed face, open mouth, [smile], solo,eye-level angle

Download: 📂Character Gen Template

Make existing character image mouth open (Inpaint)

To maintain consistency with the base image, it is recommended to utilize a checkpoint model that aligns with its style.

Download: 📂Mouth Open Inpaint Template

Inpaint Demonstration

Pose Constraints (ControlNet)

Place normal and openpose image with reference to images.

Recommend Checkpoint Model List

Anime Style SD1.5

• https://civitai.com/models/35960/flat-2d-animerge
• https://civitai.com/models/24149/mistoonanime

Realistic Style SD1.5

• https://civitai.com/models/4201/realistic-vision-v51
• https://civitai.com/models/49463/am-i-real
• https://civitai.com/models/43331/majicmix-realistic

Custom Nodes

Expand to see all the available nodes description

Image Segmentation Nodes

Image Segmentation Nodes

Name	Description	Preview
Segmentation (SAM)	Integrative SAM node allowing you to directly select and create multiple image segment output.

Mesh Edit Nodes

Mesh Edit Nodes

Name	Description	Preview
Create Mesh Layer	Create a mesh object from the input images (usually a segmented part of the entire image)
Join Meshes	Combine multiple meshes into a single mesh object
Match Texture Aspect Ratio	Since the mesh is created in 1:1 aspect ratio, a re-scale is needed at the end of the operation
Plane Texture Unwrap	Will perform mesh face fill and UV Cube project on the target plane mesh, scaled to bounds.

Shape Keys Nodes

Shape Keys Nodes

Name	Description	Preview
Mesh Modify Shape Key	Given shape key name & target vertex_group, modify the vertex / all vertex’s transform
Create Shape Flow	Create runtime shape flow graph, allowing interactive inputs affecting shape keys value in runtime

Avatar Output Nodes

Avatar Output Nodes

Name	Description	Preview
Avatar Main Output	The primary output of the .ava file. The embeded Avatar View will auto update with this node's output

Shape Flow

Installation

Clone the repository to custom_nodes in your ComfyUI directory:

1. cd custom_nodes

2. git clone https://github.com/avatechgg/avatar-graph-comfyui.git

3. Install deps cd avatar-graph-comfyui && python -m pip install -r requirements.txt

4. Restart comfyui

5. Run comfyui with enable-cors-header python main.py --enable-cors-header or (mac)python main.py --force-fp16 --enable-cors-header

Development

If you are interested in contributing expand to see development details

For comfyui frontend extension, frontend js located at avatar-graph-comfyui/js

Web stack used: vanjs tailwindcss

Install deps

pnpm i

Run the dev command to start the tailwindcss watcher

pnpm dev

For each changes, simply refresh the comfyui page to see the changes.

p.s. For tailwind autocomplete, add the following to your vscode settings.json.

{
    "tailwindCSS.experimental.classRegex": [
        ["class\\s?:\\s?([\\s\\S]*)", "(?:\"|')([^\"']*)(?:\"|')"]
    ]
}