ComfyUI-FirstOrderMM
A ComfyUI-native node to run First Order Motion Model for Image Animation
https://github.com/AliaksandrSiarohin/first-order-model
Now supports Face Swapping using Motion Supervised co-part Segmentation
https://github.com/AliaksandrSiarohin/motion-cosegmentation
Now also supports Motion Representations for Articulated Animation
https://github.com/snap-research/articulated-animation
Workflow:
https://github.com/FuouM/ComfyUI-FirstOrderMM/assets/57849916/e8adca97-b53c-48ce-a211-f336dbf7d7bc
FOMM
Part Swap
Articulate
Arguments
FOMM
relative_movement: Relative keypoint displacement (Inherit object proporions from the video)relative_jacobian: Only taken into effect whenrelative_movementis on, must also be on to avoid heavy deformation of the face (in a freaky way)adapt_movement_scale: If disabled, will heavily distort the source face to match the driving facefind_best_frame: Find driving frame that best match the source. Split the batch into two halves, with the first half reversed. Gives mixed results. Needs to installface-alignmentlibrary.
Part Swap
blend_scale: No idea, keeping at default = 1.0 seems to be fineuse_source_seg: Whether to use the source's segmentation or the target's. May help if some of the target's segmentation regions are missinghard_edges: Whether to make the edges hard, instead of featheringuse_face_parser: For Seg-based models, may help with cleaning up residual background (should only use15segwith this). TODO: Additional cleanup face_parser masks. Should definitely be used for FOMM modelsviz_alpha: Opacity of the segments in the visualization
Articulate
Doesn't need any
Installation
- Clone the repo to
ComfyUI/custom_nodes/
git clone https://github.com/FuouM/ComfyUI-FirstOrderMM.git
- Install required dependencies
pip install -r requirements.txt
Optional: Install face-alignment to use the find_best_frame feature:
pip install face-alignment
Model
FOMM currently supporting vox and vox-adv. Models can and must be manually downloaded from:
Part Swap currently supporting Seg-based models and FOMM (vox and vox-adv) models.
vox-5segmentsvox-10segmentsvox-15segmentsvox-first-order (partswap)
These models can be found in the original repository Motion Supervised co-part Segmentation
Place them in the checkpoints folder. It should look like this:
place_checkpoints_here.txt
vox-adv-cpk.pth.tar
vox-cpk.pth.tar
vox-5segments.pth.tar
vox-10segments.pth.tar
vox-15segments.pth.tar
vox-first-order.pth.tar
For Part Swap, Face-Parsing is also supported (Optional) (especially when using the FOMM or vox-first-order models)
- resnet18
resnet18-5c106cde: https://download.pytorch.org/models/resnet18-5c106cde.pth - face_parsing
79999_iter.pth: https://github.com/zllrunning/face-makeup.PyTorch/tree/master/cp
Place them in face_parsing folder:
face_parsing_model.py
...
resnet18-5c106cde.pth
79999_iter.pth
For Articulate, download the model from Pre-trained checkpoints section and place it here: articulate_module\models\vox256.pth


