72 lines
3.3 KiB
Markdown
72 lines
3.3 KiB
Markdown
# Core ML Suite for ComfyUI
|
|
|
|
## Overview
|
|
|
|
This repository contains a set of custom nodes for ComfyUI that allow you to use Core ML models in your ComfyUI
|
|
workflows. The models can be obtained [here](https://huggingface.co/coreml-community), or you can
|
|
convert your own models using [coremltools](https://github.com/apple/ml-stable-diffusion).
|
|
|
|
The main motivation behind using Core ML models in ComfyUI is to allow you to utilize the ANE (Apple Neural Engine)
|
|
on Apple Silicon (M1/M2) machines to improve performance.
|
|
|
|
While testing on M2 Pro 32GB, the ANE (`CPU_AND_NE` option) was able to speed up the inference by a factor
|
|
of ~1.5-2x.
|
|
|
|
### Features
|
|
- Loading Core ML Unet models
|
|
- Support for ControlNet
|
|
- Support for LoRA
|
|
- Support for ANE (Apple Neural Engine)
|
|
- Support for CPU and GPU
|
|
- Support for `mlmodelc` and `mlpackage` files
|
|
|
|
> [!NOTE]
|
|
> The main downside of using Core ML models is the initial compilation/loading time. For best results, please use the
|
|
compiled models (`.mlmodelc` files) instead of the `.mlpackage` files.
|
|
|
|
> [!NOTE]
|
|
> This repository is a work in progress and will be updated with more nodes and features in the future.
|
|
|
|
## Installation
|
|
|
|
To install the custom nodes, you can clone this repository ComfyUI into the `custom_nodes` directory of your ComfyUI.
|
|
Alternatively, you can download the repository as a zip file and extract it into the `custom_nodes` directory.
|
|
|
|
```bash
|
|
cd /path/to/comfyui/custom_nodes
|
|
git clone this-repo
|
|
```
|
|
Then, use pip or other package manager to install the dependencies:
|
|
```bash
|
|
cd /path/to/comfyui/custom_nodes/ComfyUI-CoreMLSuite
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
## Usage
|
|
### Available Nodes
|
|
|
|
#### CoreML Unet Loader (`CoreMLUnetLoader`)
|
|
This node allows you to load a Core ML Unet model and use it in your ComfyUI workflow. Place the converted
|
|
.mlpackage or .mlmodelc file in ComfyUI's models/unet directory and use the node to load the model. The output of the
|
|
node is a `MODEL` object similar to standard ComfyUI models.
|
|
Additionally, you can select the Compute Unit, that will be used to run the model. The default is `CPU_AND_NE`, which
|
|
gives the best results. You may, however, want to use `CPU_AND_GPU`, `ALL` or `CPU_ONLY` for experimentation.
|
|
> [!NOTE]
|
|
>To enable ControlNet in your workflow you need to use a Core ML model specifically converted for ControlNet.
|
|
> - If you use an unsupported model, the ControlNet input will be ignored.
|
|
> - If you use a model that supports ControlNet, but do not provide a ControlNet input (this includes setting
|
|
start_percent and end_percent to values other than 0 and 1 respectively), the model will use random noise
|
|
as ControlNet input.
|
|
|
|
|
|
## Limitations
|
|
- Due to the nature of Core ML models, the inputs and outputs of the models are fixed and cannot be changed[^1]. This
|
|
means
|
|
that the nodes in this repository are not as flexible as the standard ComfyUI nodes. You need to use
|
|
latent images of the same size as the input of the model (512x512 is the default for SD1.5)
|
|
- For now, only Stable Diffusion v1.5 is supported.
|
|
|
|
[^1]: Unless [EnumeratedShapes](https://apple.github.io/coremltools/docs-guides/source/flexible-inputs.html#select-from-predetermined-shapes) is used during conversion. Needs more testing.
|
|
|
|
## Support
|
|
Feel free to open an issue if you have any questions or suggestions. |