avatar-graph-comfyui

A custom_nodes module for creating real-time interactive avatars powered by blender bpy mesh api + Avatech Shape Flow runtime.

Demo

https://github.com/avatechai/avatar-graph-comfyui/assets/18395202/b229a4b3-88ba-4755-a8e0-9149260eef12

Custom Nodes

Preview	Node Name	Description
	Segmentation (SAM)	Integrative SAM node allowing you to directly select and create multiple image segment output.
	Create Mesh Layer	Create a mesh object from the input images (usually a segmented part of the entire image)
	Join Meshes	Combine multiple meshes into a single mesh object
	Mesh Modify Shape Key	Given shape key name & target vertex_group, modify the vertex / all vertex’s transform
	Create Shape Flow	Create runtime shape flow graph, allowing interactive inputs affecting shape keys value in runtime
	Match Texture Aspect Ratio	Since the mesh is created in 1:1 aspect ratio, a re-scale is needed at the end of the operation
	Plane Texture Unwrap	Will perform mesh face fill and UV Cube project on the target plane mesh, scaled to bounds.
	Avatar Main Output	The primary output of the .ava file. The embeded Avatar View will auto update with this node's output

Workflow

Workflow01

Character Gen Prompting Guide

Notice:

• We need a character image with an open mouth and enable the tool to easily recognize facial features, so please add to the prompt: looking at viewer, detailed face, open mouth, [smile], solo,eye-level angle

Workflow Download:

➡️Simple CharacterGen Workflow

• Feel free to change any checkpoint model that suits your needs.

Mouth Open(Inpaint) Guide

Notice:

• To maintain consistency with the base image, it is recommended to utilize a checkpoint model that aligns with its style.

Workflow Download:

➡️MouthOpen Workflow

Inpaint Demonstration

https://github.com/avatechai/avatar-graph-comfyui/assets/48451938/ff48c3d9-7292-4505-8993-8f117cee34ff

Recommend Checkpoint Model List:

Anime Style SD1.5

• https://civitai.com/models/35960/flat-2d-animerge
• https://civitai.com/models/24149/mistoonanime
• https://civitai.com/models/22364/kizuki-anime-hentai-checkpoint

Realistic Style SD1.5

• https://civitai.com/models/4201/realistic-vision-v51
• https://civitai.com/models/49463/am-i-real
• https://civitai.com/models/43331/majicmix-realistic

Shape Flow

Installation

1. cd custom_nodes

2. git clone https://github.com/avatechgg/avatar-graph-comfyui.git

3. Install deps cd avatar-graph-comfyui && python -m pip install -r requirements.txt

4. Navigate to in your comfyui installation folder, open comfyui/main.py and add the following lines:

def prompt_worker(q, server):
    e = execution.PromptExecutor(server)

    # This is a workaround since using blender as a python module, the bpy module has to be imported after in the custom thread, otherwise it will cause a segfault if imported in the custom nodes.

    # Add next line
    import global_bpy

    while True:
        item, item_id = q.get()
        # Add next line
        global_bpy.reset_bpy()

5. Restart comfyui

6. Run comfyui with enable-cors-header python main.py --enable-cors-header or (mac)python main.py --force-fp16 --enable-cors-header

Development

For comfyui frontend extension, frontend js located at avatar-graph-comfyui/js

Web stack used: vanjs tailwindcss

Install deps

pnpm i

Run the dev command to start the tailwindcss watcher

pnpm dev

For each changes, simply refresh the comfyui page to see the changes.

p.s. For tailwind autocomplete, add the following to your vscode settings.json.

{
    "tailwindCSS.experimental.classRegex": [
        ["class\\s?:\\s?([\\s\\S]*)", "(?:\"|')([^\"']*)(?:\"|')"]
    ]
}