174 lines
4.5 KiB
Markdown
174 lines
4.5 KiB
Markdown
# ComfyUI KeySync Wrapper
|
|
|
|
This custom node package wraps the official KeySync inference pipeline for ComfyUI.
|
|
|
|
It does not include KeySync weights and it does not rewrite KeySync model code. The first version calls the official `scripts/infer_raw.sh` script through `subprocess`.
|
|
|
|
## Nodes
|
|
|
|
- `KeySync Download Models`
|
|
- `KeySync Model Loader`
|
|
- `KeySync LipSync`
|
|
- `KeySync LipSync Advanced`
|
|
|
|
Typical workflow:
|
|
|
|
```text
|
|
Load Video (Path) -> KeySync LipSync.images
|
|
Load Audio (Path) -> KeySync LipSync.audio
|
|
KeySync LipSync.images/audio -> VideoHelperSuite Video Combine
|
|
```
|
|
|
|
`KeySync LipSync` returns:
|
|
|
|
- `images`: ComfyUI `IMAGE` frames
|
|
- `audio`: ComfyUI `AUDIO`
|
|
- `frame_rate`: usually `25`
|
|
- `output_video_path`: path to the intermediate `final.mp4`
|
|
|
|
## Install The Official KeySync Repo
|
|
|
|
Clone the original KeySync repository into this wrapper folder:
|
|
|
|
```bash
|
|
cd D:/ComfyUI/custom_nodes/ComfyUI-Keysync-wrapper
|
|
git clone https://github.com/antonibigata/keysync
|
|
```
|
|
|
|
Default repo path used by the loader:
|
|
|
|
```text
|
|
D:/ComfyUI/custom_nodes/ComfyUI-Keysync-wrapper/keysync
|
|
```
|
|
|
|
## Download Models
|
|
|
|
Recommended model directory:
|
|
|
|
```text
|
|
D:/ComfyUI/models/keysync
|
|
```
|
|
|
|
You can use the `KeySync Download Models` node with `download=True`.
|
|
|
|
Manual download is also supported:
|
|
|
|
```bash
|
|
git lfs install
|
|
git clone https://huggingface.co/toninio19/keysync D:/ComfyUI/models/keysync
|
|
```
|
|
|
|
The loader expects:
|
|
|
|
```text
|
|
D:/ComfyUI/models/keysync/checkpoints/keyframe_dub.pt
|
|
D:/ComfyUI/models/keysync/checkpoints/interpolation_dub.pt
|
|
```
|
|
|
|
The default UI intentionally hides repo and model directory fields. The wrapper checks these fixed paths:
|
|
|
|
```text
|
|
KeySync repo: D:/ComfyUI/custom_nodes/ComfyUI-Keysync-wrapper/keysync
|
|
Model dir: D:/ComfyUI/models/keysync
|
|
```
|
|
|
|
## Runtime Notes
|
|
|
|
KeySync has heavy dependencies. This wrapper intentionally does not install the full official KeySync dependency set into ComfyUI.
|
|
|
|
Recommended runtime:
|
|
|
|
- WSL2 or Linux
|
|
- `ffmpeg` available on `PATH`
|
|
- `bash` available on `PATH`
|
|
- official KeySync environment/dependencies installed according to the upstream README
|
|
|
|
If you run native Windows ComfyUI, install Git Bash or use WSL2. The official inference script depends on bash.
|
|
|
|
## Output Layout
|
|
|
|
Each run creates an isolated task folder:
|
|
|
|
```text
|
|
D:/ComfyUI/output/keysync/task_YYYYMMDD_HHMMSS_xxxxxxxx/
|
|
data/
|
|
videos/input.mp4
|
|
audios/input.wav
|
|
logs/
|
|
keysync_stdout.log
|
|
keysync_stderr.log
|
|
final.mp4
|
|
```
|
|
|
|
The official KeySync repo also receives its normal output under:
|
|
|
|
```text
|
|
<keysync_repo_path>/outputs/<task_id>/
|
|
```
|
|
|
|
Set `keep_intermediate_outputs=False` to remove the prepared input data and upstream output folder after `final.mp4` is copied.
|
|
|
|
## Parameters
|
|
|
|
The simple `KeySync LipSync` node exposes only the everyday parameters:
|
|
|
|
- `input_fps`
|
|
- `compute_until`
|
|
|
|
Use `KeySync LipSync Advanced` only when you need the official occlusion path:
|
|
|
|
- `fix_occlusion`: enables KeySync's official SAM2-based occlusion mask path.
|
|
- `position`: point prompt for the occluding object, for example `450,450`.
|
|
- `start_frame`: frame index where the point prompt applies.
|
|
|
|
Internal wrapper behavior is fixed for the simple node:
|
|
|
|
- video is normalized to 25 fps
|
|
- audio is normalized to 16 kHz mono WAV
|
|
- subprocess mode is always used
|
|
- intermediate task files are kept for debugging
|
|
- no subprocess timeout is enforced by default
|
|
|
|
Advanced KeySync parameters such as `resize_size`, `chunk_size`, `cond_aug`, and `decoding_t` are not exposed in v1 because the official `infer_raw.sh` script currently hardcodes them.
|
|
|
|
## Occlusion And SAM
|
|
|
|
The official KeySync code uses SAM2 for occlusion handling. The `position` field is a SAM2 point prompt used to track the object that blocks the face, such as a hand or microphone. This is powerful but awkward as a text field, so it is hidden from the simple node.
|
|
|
|
SAM3-based tracking may be added later as a separate preprocessing node that outputs an occlusion mask or point prompt. It is not wired into v1 because the official KeySync `infer_raw.sh` path expects its own SAM2 implementation and checkpoint layout.
|
|
|
|
## Common Errors
|
|
|
|
Missing KeySync repo:
|
|
|
|
```text
|
|
KeySync repo path does not exist
|
|
```
|
|
|
|
Missing official script:
|
|
|
|
```text
|
|
scripts/infer_raw.sh not found
|
|
```
|
|
|
|
Missing checkpoints:
|
|
|
|
```text
|
|
keyframes_ckpt does not exist
|
|
interpolation_ckpt does not exist
|
|
```
|
|
|
|
Missing ffmpeg:
|
|
|
|
```text
|
|
ffmpeg not found. Please install ffmpeg and add it to PATH.
|
|
```
|
|
|
|
Missing bash:
|
|
|
|
```text
|
|
KeySync official inference script requires bash.
|
|
```
|
|
|
|
When KeySync fails, the error message includes the stdout and stderr log paths.
|