city96-ComfyUI_ExtraModels/README.md

Extra Models for ComfyUI

This repository aims to add support for various random image diffusion models to ComfyUI.

Installation

Simply clone this repo to your custom_nodes folder using the following command: git clone https://github.com/city96/ComfyUI_ExtraModels custom_nodes/ComfyUI_ExtraModels.

DiT

Model info / implementation

• Uses class labels instead of prompts
• Limited to 256x256 or 512x512 images
• Same latent space as SD1.5 (works with the SD1.5 VAE)
• Works in FP16, but no other optimization (yet)

Usage

1. Download the original model weights from the DiT Repo or the converted FP16 safetensor ones from Huggingface.
2. Place them in ComfyUI\models\dit (created on first run after installing the extension)
3. Load the model and select the class labels as shown in the image below
4. Make sure to use the Empty label conditioning for the Negative input of the KSampler!

ConditioningCombine nodes should work for combining multiple labels. The area ones don't since the model currently can't handle dynamic input dimensions.

Image with sample workflow

VAE

A few custom VAE models are supported. The option to select a different dtype when loading is also possible, which can be useful for testing/comparisons.

AutoencoderKL / VQModel

kl-f4/8/16/32 from the compvis/latent diffusion repo.

vq-f4/8/16 from the taming transformers repo, weights for both vq and kl models available here

vq-f8 can accepts latents from the SD unet but just like xl with v1 latents, output largely garbage. The rest are completely useless without a matching UNET that uses the correct channel count.