diff --git a/README.md b/README.md index 760f6c0..92f8948 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,8 @@ That's it! You're now ready to start enhancing your ComfyUI workflows with Core - **VAE**: Variational Autoencoder. A model that learns a latent representation of images. It's used as a prior in Stable Diffusion. - **Checkpoint**: A file that contains the weights of a model. It's used to load models in Stable Diffusion. +- **LCM**: [Latent Consistency Model](https://latent-consistency-models.github.io/). A type of model designed to + generate images with as few steps as possible. > [!NOTE] > Note on Compute Units: @@ -64,6 +66,8 @@ These custom nodes come with a host of features, including: - Support for ANE (Apple Neural Engine) - Support for CPU and GPU - Support for `mlmodelc` and `mlpackage` files +- Support for LCM models +- LCM -> Core ML conversion > [!NOTE] > Please note that using Core ML models can take a bit longer to load initially. @@ -139,6 +143,30 @@ resulting latent as you normally would in your workflow. - **LATENT**: The latent image output by the Core ML model. This can be decoded using a VAE Decoder or used as input to the next node in your workflow. +#### LCM Converter + +![LCMConverter](./assets/lcm_converter.png?raw=true) + +This node converts [SimianLuo/LCM_Dreamshaper_v7](https://huggingface.co/SimianLuo/LCM_Dreamshaper_v7) model to Core +ML. The converted model is stored in the `models/unet` directory and can be used with the Core ML UNet Loader. The +conversion parameteres are encoded in the node name, so if the model already exists, the node will not convert it again. + +- **Inputs**: + - **height**: The desired height of the image generated by the model. The default is 512. Must be a multiple of 8. + - **width**: The desired width of the image generated by the model. The default is 512. Must be a multiple of 8. + - **batch_size**: The batch size of generated images. If you're planning to generate batches of images, you can try + increasing this value to speed up the generation process. The default is 1. + - **compute_unit**: The hardware on which the model should run. This is used only when loading the model and + doesn't affect the conversion process. + - **controlnet_support**: For the model to support ControlNet, it must be converted with this option set to True. + The default is False. + +> [!NOTE] +> The conversion process can take a while, so please be patient. + +> [!NOTE] +> When using the LCM model with Core ML Sampler, please set _sampler_name_ to `lcm` and _scheduler_ to `sgm_uniform`. + #### Core ML Adapter (Experimental) (`CoreMLModelAdapter`) ![CoreMLModelAdapter](./assets/adapter.png?raw=true) @@ -197,6 +225,13 @@ The ControlNet model used in this workflow is available Once downloaded, place the model in the `models/controlnet` directory. ![coreml-unet+controlnet](./assets/unet+sampler+controlnet.png?raw=true) +#### LCM conversion with ControlNet + +This workflow uses LCM converter to +convert [SimianLuo/LCM_Dreamshaper_v7](https://huggingface.co/SimianLuo/LCM_Dreamshaper_v7) +model to Core ML. The converted model can then be used with or without ControlNet to generate images. +![lcm+controlnet](./assets/lcm+controlnet.png?raw=true) + ## Limitations - Core ML models are fixed in terms of their inputs and outputs. diff --git a/assets/lcm+controlnet.png b/assets/lcm+controlnet.png new file mode 100644 index 0000000..d81f256 Binary files /dev/null and b/assets/lcm+controlnet.png differ diff --git a/assets/lcm_converter.png b/assets/lcm_converter.png new file mode 100644 index 0000000..49029e1 Binary files /dev/null and b/assets/lcm_converter.png differ