HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models
HelloVision | HelloGroup Inc.
* Intern
## π New Features/Updates
- β
`11/29/2024` a.Optimize the algorithm; b.Add VAE selection functionality; c.Introduce a super-resolution feature.
[](https://www.youtube.com/watch?v=fM5nyn6q02Y)
- β
`11/14/2024` Added the `HMControlNet2` module, which uses the `PD-FGC` motion module to extract facial expression information (`drive_exp2`); restructured the ComfyUI interface; and optimized VRAM usage.
[](https://www.youtube.com/watch?v=ZvoMHyRm310)
- β
`11/12/2024` Added a newly fine-tuned version of [`Animatediff`](https://huggingface.co/songkey/hm_animatediff_frame12) with a patch size of 12, which uses less VRAM (Tested on 2080Ti).
- β
`11/11/2024` ~~Optimized VRAM usage and added `HMVideoSimplePipeline` (`workflows/hellomeme_video_simple_workflow.json`), which doesnβt use Animatediff and can run on machines with less than 12G VRAM.~~
- β
`11/6/2024` The face proportion in the reference image significantly affects the generation quality. We have encapsulated the **recommended image cropping method** used during training into a `CropReferenceImage` Node. Refer to the workflows in the `ComfyUI_HelloMeme/workflows directory`: `hellomeme_video_cropref_workflow.json` and `hellomeme_image_cropref_workflow.json`.
## Introduction
This repository is the official implementation of the [`HelloMeme`](https://arxiv.org/pdf/2410.22901) ComfyUI interface, featuring both image and video generation functionalities. Example workflow files can be found in the `ComfyUI_HelloMeme/workflows` directory. Test images and videos are saved in the `ComfyUI_HelloMeme/examples` directory. Below are screenshots of the interfaces for image and video generation.
> [!Note]
> [Custom models should be placed in the directories listed below.](https://github.com/HelloVision/ComfyUI_HelloMeme/issues/5#issuecomment-2461247829)
>
> **Checkpoints** under: `ComfyUI/models/checkpoints`
>
> **Loras** under: `ComfyUI/models/checkpoints`
### workflows
| workflow file | Video Generation | Image Generation | HMControlNet | HMControlNet2 | Super-Resolution |
|---------------|------------------|------------------|-----------|---------------|-----------------|
| hellomeme_image_workflow.json | | β
| β
| | |
| hellomeme_video_workflow.json | β
| | β
| | |
| hellomeme_image_sr_workflow.json | | β
| β
| | β
|
| hellomeme_video_sr_workflow.json | β
| | β
| | β
|
| hellomeme_image_v2_workflow.json | | β
| | β
| |
| hellomeme_video_v2_workflow.json | β
| | | β
| |
| hellomeme_image_v2_sr_workflow.json | | β
| | β
| β
|
| hellomeme_video_v2_sr_workflow.json | β
| | | β
| β
|
### Image Generation Interface
### Video Generation Interface
