avatechai-avatar-graph-comfyui/README.md

avatar-graph-comfyui

A custom_nodes module for creating real-time interactive avatars powered by blender bpy mesh api + Avatech Shape Flow runtime.

Demo

https://github.com/avatechai/avatar-graph-comfyui/assets/18395202/b229a4b3-88ba-4755-a8e0-9149260eef12

Custom Nodes

Category	Preview	Node Name	Description
Segmentation		Segmentation (SAM)
Mesh		Create Mesh Layer	Create a mesh object from the input images (usually a segmented part of the entire image)
		Join Meshes
Blendshapes		Mesh Modify Shape Key	Given shape key name & target vertex_group, modify the vertex / all vertex’s transform
		Create Shape Flow	Create runtime shape flow graph, allowing interactive inputs affecting shape keys value in runtime
UV & Texture		Match Texture Aspect Ratio	Since the mesh is created in 1:1 aspect ratio, a re-scale is needed at the end of the operation
		Plane Texture Unwrap	Will perform mesh face fill and UV Cube project on the target plane mesh, scaled to bounds.
Output		Avatar Main Output

Workflow

Workflow01

Character Gen Prompting Guide

Notice:

• We need a character image with an open mouth and enable the tool to easily recognize facial features, so please add to the prompt: looking at viewer, detailed face, open mouth, [smile], solo,eye-level angle

Workflow Download:

➡️Simple CharacterGen Workflow

• Feel free to change any checkpoint model that suits your needs.

Mouth Open(Inpaint) Guide

Notice:

• To maintain consistency with the base image, it is recommended to utilize a checkpoint model that aligns with its style.

Workflow Download:

➡️MouthOpen Workflow

Inpaint Demonstration

https://github.com/avatechai/avatar-graph-comfyui/assets/48451938/ff48c3d9-7292-4505-8993-8f117cee34ff

Recommend Checkpoint Model List:

Anime Style SD1.5

• https://civitai.com/models/35960/flat-2d-animerge
• https://civitai.com/models/24149/mistoonanime
• https://civitai.com/models/22364/kizuki-anime-hentai-checkpoint

Realistic Style SD1.5

• https://civitai.com/models/4201/realistic-vision-v51
• https://civitai.com/models/49463/am-i-real
• https://civitai.com/models/43331/majicmix-realistic

Shape Flow

Installation

1. cd custom_nodes

2. git clone https://github.com/avatechgg/avatar-graph-comfyui.git

3. Install deps cd avatar-graph-comfyui && python -m pip install -r requirements.txt

4. Navigate to in your comfyui installation folder, open comfyui/main.py and add the following lines:

def prompt_worker(q, server):
    e = execution.PromptExecutor(server)

    # This is a workaround since using blender as a python module, the bpy module has to be imported after in the custom thread, otherwise it will cause a segfault if imported in the custom nodes.

    # Add next line
    import global_bpy

    while True:
        item, item_id = q.get()
        # Add next line
        global_bpy.reset_bpy()

5. Restart comfyui

6. Run comfyui with enable-cors-header python main.py --enable-cors-header or (mac)python main.py --force-fp16 --enable-cors-header

Development

For comfyui frontend extension, frontend js located at avatar-graph-comfyui/js

Web stack used: vanjs tailwindcss

Install deps

pnpm i

Run the dev command to start the tailwindcss watcher

pnpm dev

For each changes, simply refresh the comfyui page to see the changes.

p.s. For tailwind autocomplete, add the following to your vscode settings.json.

{
    "tailwindCSS.experimental.classRegex": [
        ["class\\s?:\\s?([\\s\\S]*)", "(?:\"|')([^\"']*)(?:\"|')"]
    ]
}