138 lines
5.6 KiB
Markdown
138 lines
5.6 KiB
Markdown
# avatar-graph-comfyui
|
||
|
||
|
||
|
||
A custom_nodes module for creating real-time interactive avatars powered by blender bpy mesh api + Avatech Shape Flow runtime.
|
||
|
||
# Demo
|
||
|
||
https://github.com/avatechai/avatar-graph-comfyui/assets/18395202/b229a4b3-88ba-4755-a8e0-9149260eef12
|
||
|
||
# Custom Nodes
|
||
|
||
| Category | Preview | Node Name | Description |
|
||
| --- | --- | --- | --- |
|
||
| Segmentation |  | Segmentation (SAM) | |
|
||
| Mesh |  | Create Mesh Layer | Create a mesh object from the input images (usually a segmented part of the entire image) |
|
||
| | | Join Meshes | |
|
||
| Blendshapes |  | Mesh Modify Shape Key | Given shape key name & target vertex_group, modify the vertex / all vertex’s transform |
|
||
| |  | Create Shape Flow | Create runtime shape flow graph, allowing interactive inputs affecting shape keys value in runtime |
|
||
| UV & Texture |  | Match Texture Aspect Ratio | Since the mesh is created in 1:1 aspect ratio, a re-scale is needed at the end of the operation |
|
||
| |  | Plane Texture Unwrap | Will perform mesh face fill and UV Cube project on the target plane mesh, scaled to bounds.
|
||
| Output |  | Avatar Main Output | |
|
||
|
||
# Workflow
|
||
|
||
<details>
|
||
<summary> Workflow01 </summary>
|
||
</details>
|
||
|
||
<details>
|
||
<summary> Character Gen Prompting Guide </summary>
|
||
|
||
### Notice:
|
||
- We need a character image with an open mouth and enable the tool to easily recognize facial features, so please add to the prompt: ```looking at viewer, detailed face, open mouth, [smile], solo,eye-level angle```
|
||
|
||
### Workflow Download:
|
||
|
||
➡️[Simple CharacterGen Workflow](https://github.com/avatechai/avatar-graph-comfyui/blob/main/workflow_templates/SimpleCharacterGen.json)
|
||
- Feel free to change any checkpoint model that suits your needs.
|
||
|
||
|
||
|
||
|
||
</details>
|
||
|
||
<details>
|
||
<summary> Mouth Open(Inpaint) Guide </summary>
|
||
|
||
### Notice:
|
||
- To maintain consistency with the base image, it is recommended to utilize a checkpoint model that aligns with its style.
|
||
|
||
### Workflow Download:
|
||
|
||
➡️[MouthOpen Workflow](https://github.com/avatechai/avatar-graph-comfyui/blob/main/workflow_templates/MouthOpen_(inpaint).json)
|
||
|
||
|
||
|
||
### Inpaint Demonstration
|
||
|
||
https://github.com/avatechai/avatar-graph-comfyui/assets/48451938/ff48c3d9-7292-4505-8993-8f117cee34ff
|
||
|
||
### Recommend Checkpoint Model List:
|
||
##### Anime Style SD1.5
|
||
- https://civitai.com/models/35960/flat-2d-animerge
|
||
- https://civitai.com/models/24149/mistoonanime
|
||
- https://civitai.com/models/22364/kizuki-anime-hentai-checkpoint
|
||
##### Realistic Style SD1.5
|
||
- https://civitai.com/models/4201/realistic-vision-v51
|
||
- https://civitai.com/models/49463/am-i-real
|
||
- https://civitai.com/models/43331/majicmix-realistic
|
||
-
|
||
</details>
|
||
|
||
# Shape Flow
|
||
|
||
|
||
|
||
|
||
|
||
|
||
# Installation
|
||
|
||
1. `cd custom_nodes`
|
||
|
||
2. `git clone https://github.com/avatechgg/avatar-graph-comfyui.git`
|
||
|
||
3. Install deps `cd avatar-graph-comfyui && python -m pip install -r requirements.txt`
|
||
|
||
4. Navigate to in your comfyui installation folder, open `comfyui/main.py` and add the following lines:
|
||
```py
|
||
def prompt_worker(q, server):
|
||
e = execution.PromptExecutor(server)
|
||
|
||
# This is a workaround since using blender as a python module, the bpy module has to be imported after in the custom thread, otherwise it will cause a segfault if imported in the custom nodes.
|
||
|
||
# Add next line
|
||
import global_bpy
|
||
|
||
while True:
|
||
item, item_id = q.get()
|
||
# Add next line
|
||
global_bpy.reset_bpy()
|
||
```
|
||
|
||
5. Restart comfyui
|
||
|
||
6. Run comfyui with enable-cors-header `python main.py --enable-cors-header` or (mac)`python main.py --force-fp16 --enable-cors-header`
|
||
|
||
# Development
|
||
|
||
For comfyui frontend extension, frontend js located at `avatar-graph-comfyui/js`
|
||
|
||
Web stack used: [vanjs](https://github.com/vanjs-org/van) [tailwindcss](https://github.com/tailwindlabs/tailwindcss)
|
||
|
||
## Install deps
|
||
|
||
```
|
||
pnpm i
|
||
```
|
||
|
||
Run the dev command to start the tailwindcss watcher
|
||
|
||
```
|
||
pnpm dev
|
||
```
|
||
|
||
For each changes, simply refresh the comfyui page to see the changes.
|
||
|
||
p.s. For tailwind autocomplete, add the following to your vscode settings.json.
|
||
|
||
```json
|
||
{
|
||
"tailwindCSS.experimental.classRegex": [
|
||
["class\\s?:\\s?([\\s\\S]*)", "(?:\"|')([^\"']*)(?:\"|')"]
|
||
]
|
||
}
|
||
```
|