From 556f47bfc5ddef72bd717c07500d7df0c3ca6830 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adrian=20Szczepa=C5=84ski?= Date: Mon, 23 Oct 2023 14:50:32 +0200 Subject: [PATCH] Add README.md --- README.md | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..daaddae --- /dev/null +++ b/README.md @@ -0,0 +1,72 @@ +# Core ML Suite for ComfyUI + +## Overview + +This repository contains a set of custom nodes for ComfyUI that allow you to use Core ML models in your ComfyUI +workflows. The models can be obtained [here](https://huggingface.co/coreml-community), or you can +convert your own models using [coremltools](https://github.com/apple/ml-stable-diffusion). + +The main motivation behind using Core ML models in ComfyUI is to allow you to utilize the ANE (Apple Neural Engine) +on Apple Silicon (M1/M2) machines to improve performance. + +While testing on M2 Pro 32GB, the ANE (`CPU_AND_NE` option) was able to speed up the inference by a factor +of ~1.5-2x. + +### Features +- Loading Core ML Unet models +- Support for ControlNet +- Support for LoRA +- Support for ANE (Apple Neural Engine) +- Support for CPU and GPU +- Support for `mlmodelc` and `mlpackage` files + +> [!NOTE] +> The main downside of using Core ML models is the initial compilation/loading time. For best results, please use the +compiled models (`.mlmodelc` files) instead of the `.mlpackage` files. + +> [!NOTE] +> This repository is a work in progress and will be updated with more nodes and features in the future. + +## Installation + +To install the custom nodes, you can clone this repository ComfyUI into the `custom_nodes` directory of your ComfyUI. +Alternatively, you can download the repository as a zip file and extract it into the `custom_nodes` directory. + +```bash +cd /path/to/comfyui/custom_nodes +git clone this-repo +``` +Then, use pip or other package manager to install the dependencies: +```bash +cd /path/to/comfyui/custom_nodes/ComfyUI-CoreMLSuite +pip install -r requirements.txt +``` + +## Usage +### Available Nodes + +#### CoreML Unet Loader (`CoreMLUnetLoader`) +This node allows you to load a Core ML Unet model and use it in your ComfyUI workflow. Place the converted +.mlpackage or .mlmodelc file in ComfyUI's models/unet directory and use the node to load the model. The output of the +node is a `MODEL` object similar to standard ComfyUI models. +Additionally, you can select the Compute Unit, that will be used to run the model. The default is `CPU_AND_NE`, which +gives the best results. You may, however, want to use `CPU_AND_GPU`, `ALL` or `CPU_ONLY` for experimentation. +> [!NOTE] +>To enable ControlNet in your workflow you need to use a Core ML model specifically converted for ControlNet. +> - If you use an unsupported model, the ControlNet input will be ignored. +> - If you use a model that supports ControlNet, but do not provide a ControlNet input (this includes setting + start_percent and end_percent to values other than 0 and 1 respectively), the model will use random noise +as ControlNet input. + + +## Limitations +- Due to the nature of Core ML models, the inputs and outputs of the models are fixed and cannot be changed[^1]. This + means +that the nodes in this repository are not as flexible as the standard ComfyUI nodes. You need to use +latent images of the same size as the input of the model (512x512 is the default for SD1.5) +- For now, only Stable Diffusion v1.5 is supported. + +[^1]: Unless [EnumeratedShapes](https://apple.github.io/coremltools/docs-guides/source/flexible-inputs.html#select-from-predetermined-shapes) is used during conversion. Needs more testing. + +## Support +Feel free to open an issue if you have any questions or suggestions. \ No newline at end of file