HelloVision-ComfyUI_HelloMeme/README.md

HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models

Shengkai Zhang, Nianhong Jiao, Tian Li, Chaojie Yang, Chenhui Xue^*, Boya Niu^*, Jun Gao

HelloVision | HelloGroup Inc.

^* Intern

🔆 New Features/Updates

• ☐ This update features significant changes, offering better compatibility with custom checkpoints and improved VRAM efficiency.

  

• ✅ 11/29/2024 a.Optimize the algorithm; b.Add VAE selection functionality; c.Introduce a super-resolution feature.

  

• ✅ 11/14/2024 Added the HMControlNet2 module, which uses the PD-FGC motion module to extract facial expression information (drive_exp2); restructured the ComfyUI interface; and optimized VRAM usage.

  

• ✅ 11/12/2024 Added a newly fine-tuned version of Animatediff with a patch size of 12, which uses less VRAM (Tested on 2080Ti).

• ✅ 11/11/2024 Optimized VRAM usage and added HMVideoSimplePipeline (workflows/hellomeme_video_simple_workflow.json), which doesn’t use Animatediff and can run on machines with less than 12G VRAM.

• ✅ 11/6/2024 The face proportion in the reference image significantly affects the generation quality. We have encapsulated the recommended image cropping method used during training into a CropReferenceImage Node. Refer to the workflows in the ComfyUI_HelloMeme/workflows directory: hellomeme_video_cropref_workflow.json and hellomeme_image_cropref_workflow.json.

Introduction

This repository is the official implementation of the HelloMeme ComfyUI interface, featuring both image and video generation functionalities. Example workflow files can be found in the ComfyUI_HelloMeme/workflows directory. Test images and videos are saved in the ComfyUI_HelloMeme/examples directory. Below are screenshots of the interfaces for image and video generation.

Note

Custom models should be placed in the directories listed below.

Checkpoints under: ComfyUI/models/checkpoints

Loras under: ComfyUI/models/checkpoints

workflows

workflow file	Video Generation	Image Generation	HMControlNet	HMControlNet2	Super-Resolution
hellomeme_image_workflow.json		✅	✅
hellomeme_video_workflow.json	✅		✅
hellomeme_image_sr_workflow.json		✅	✅		✅
hellomeme_video_sr_workflow.json	✅		✅		✅
hellomeme_image_v2_workflow.json		✅		✅
hellomeme_video_v2_workflow.json	✅			✅
hellomeme_image_v2_sr_workflow.json		✅		✅	✅
hellomeme_video_v2_sr_workflow.json	✅			✅	✅

Image Generation Interface

Video Generation Interface