avatar-graph-comfyui

Wanna Animate? Join our Discord

A custom nodes module for creating real-time interactive avatars powered by blender bpy mesh api + Avatech Shape Flow runtime.

Warning

We are still making changes to the nodes and demo templates, please stay tuned.

Demo

Interact 👆	Interact 👆	Interact 👆	Interact 👆
Interact 👆	Interact 👆	Interact 👆	Interact 👆

How to?

• Basic Rigging Workflow Template
• Best Practices for image input
• Custom Nodes List
• Shape Flow
• Installation
• Development
• Join Discord 💬

Basic Rigging Workflow Template

1. Creating an eye blink and lipsync avatar

For optimal results, please input a character image with an open mouth and a minimum resolution of 768x768. This higher resolution will enable the tool to accurately recognize and work with facial features.

Download: Save the image, and drag into Comfyui or Simple Shape Flow

2. Creating an eye blink and lipsync emoji avatar

Download: Save the image, and drag into Comfyui

Download: Save the image, and drag into Comfyui or Dog Workflow

Best practices for image input

1. Generate a new character image

We need a character image with an open mouth and enable the tool to easily recognize facial features, so please add to the prompt: looking at viewer, detailed face, open mouth, [smile], solo,eye-level angle

Download: Character Gen Template

2. Make existing character image mouth open (Inpaint)

To maintain consistency with the base image, it is recommended to utilize a checkpoint model that aligns with its style.

Download: Mouth Open Inpaint Template

Inpaint Demonstration

3. Pose Constraints (ControlNet)

Place normal and openpose image with reference to images.

Download: ControlNet Gen

Recommend Checkpoint Model List

Anime Style SD1.5

• https://civitai.com/models/35960/flat-2d-animerge
• https://civitai.com/models/24149/mistoonanime

Realistic Style SD1.5

• https://civitai.com/models/4201/realistic-vision-v51
• https://civitai.com/models/49463/am-i-real
• https://civitai.com/models/43331/majicmix-realistic

Custom Nodes

Expand to see all the available nodes description

All Custom Nodes

Name	Description	Preview
Segmentation (SAM)	Integrative SAM node allowing you to directly select and create multiple image segment output.

Name	Description	Preview
Create Mesh Layer	Create a mesh object from the input images (usually a segmented part of the entire image)
Join Meshes	Combine multiple meshes into a single mesh object
Match Texture Aspect Ratio	Since the mesh is created in 1:1 aspect ratio, a re-scale is needed at the end of the operation
Plane Texture Unwrap	Will perform mesh face fill and UV Cube project on the target plane mesh, scaled to bounds.

Name	Description	Preview
Mesh Modify Shape Key	Given shape key name & target vertex_group, modify the vertex / all vertex’s transform
Create Shape Flow	Create runtime shape flow graph, allowing interactive inputs affecting shape keys value in runtime

Name	Description	Preview
Avatar Main Output	The primary output of the .ava file. The embedded Avatar View will auto update with this node's output

Shape Flow

Installation

Clone the repository to custom_nodes in your ComfyUI directory:

1. cd custom_nodes

2. git clone https://github.com/avatechgg/avatar-graph-comfyui.git

3. Install deps cd avatar-graph-comfyui && python -m pip install -r requirements.txt

4. Restart comfyui

5. Run comfyui with enable-cors-header python main.py --enable-cors-header or (mac)python main.py --force-fp16 --enable-cors-header

Development

If you are interested in contributing

For comfyui frontend extension, frontend js located at avatar-graph-comfyui/js

Web stack used: vanjs tailwindcss

Install deps

pnpm i

Run the dev command to start the tailwindcss watcher

pnpm dev

For each changes, simply refresh the comfyui page to see the changes.

p.s. For tailwind autocomplete, add the following to your vscode settings.json.

{
    "tailwindCSS.experimental.classRegex": [
        ["class\\s?:\\s?([\\s\\S]*)", "(?:\"|')([^\"']*)(?:\"|')"]
    ]
}

Update blender node types

To update blender operations input and output types (stored in blender/input_types.txt), run:

python generate_blender_types.py

Any question? Join our Discord