Add LCM info to readme

This commit is contained in:
aszc-dev
2023-11-13 13:47:15 +01:00
parent 4a1359b6b5
commit f9f25fbeb7
3 changed files with 35 additions and 0 deletions
+35
View File
@@ -48,6 +48,8 @@ That's it! You're now ready to start enhancing your ComfyUI workflows with Core
- **VAE**: Variational Autoencoder. A model that learns a latent representation of images. It's used as a prior in
Stable Diffusion.
- **Checkpoint**: A file that contains the weights of a model. It's used to load models in Stable Diffusion.
- **LCM**: [Latent Consistency Model](https://latent-consistency-models.github.io/). A type of model designed to
generate images with as few steps as possible.
> [!NOTE]
> Note on Compute Units:
@@ -64,6 +66,8 @@ These custom nodes come with a host of features, including:
- Support for ANE (Apple Neural Engine)
- Support for CPU and GPU
- Support for `mlmodelc` and `mlpackage` files
- Support for LCM models
- LCM -> Core ML conversion
> [!NOTE]
> Please note that using Core ML models can take a bit longer to load initially.
@@ -139,6 +143,30 @@ resulting latent as you normally would in your workflow.
- **LATENT**: The latent image output by the Core ML model. This can be decoded using a VAE Decoder or used as input
to the next node in your workflow.
#### LCM Converter
![LCMConverter](./assets/lcm_converter.png?raw=true)
This node converts [SimianLuo/LCM_Dreamshaper_v7](https://huggingface.co/SimianLuo/LCM_Dreamshaper_v7) model to Core
ML. The converted model is stored in the `models/unet` directory and can be used with the Core ML UNet Loader. The
conversion parameteres are encoded in the node name, so if the model already exists, the node will not convert it again.
- **Inputs**:
- **height**: The desired height of the image generated by the model. The default is 512. Must be a multiple of 8.
- **width**: The desired width of the image generated by the model. The default is 512. Must be a multiple of 8.
- **batch_size**: The batch size of generated images. If you're planning to generate batches of images, you can try
increasing this value to speed up the generation process. The default is 1.
- **compute_unit**: The hardware on which the model should run. This is used only when loading the model and
doesn't affect the conversion process.
- **controlnet_support**: For the model to support ControlNet, it must be converted with this option set to True.
The default is False.
> [!NOTE]
> The conversion process can take a while, so please be patient.
> [!NOTE]
> When using the LCM model with Core ML Sampler, please set _sampler_name_ to `lcm` and _scheduler_ to `sgm_uniform`.
#### Core ML Adapter (Experimental) (`CoreMLModelAdapter`)
![CoreMLModelAdapter](./assets/adapter.png?raw=true)
@@ -197,6 +225,13 @@ The ControlNet model used in this workflow is available
Once downloaded, place the model in the `models/controlnet` directory.
![coreml-unet+controlnet](./assets/unet+sampler+controlnet.png?raw=true)
#### LCM conversion with ControlNet
This workflow uses LCM converter to
convert [SimianLuo/LCM_Dreamshaper_v7](https://huggingface.co/SimianLuo/LCM_Dreamshaper_v7)
model to Core ML. The converted model can then be used with or without ControlNet to generate images.
![lcm+controlnet](./assets/lcm+controlnet.png?raw=true)
## Limitations
- Core ML models are fixed in terms of their inputs and outputs.
Binary file not shown.

After

Width:  |  Height:  |  Size: 476 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 51 KiB