SD-Latent-Interposer
A small neural network to provide interoperability between the latents generated by the different Stable Diffusion models.
I wanted to see if it was possible to pass latents generated by the new SDXL model directly into SDv1.5 models without decoding and re-encoding them using a VAE first.
Installation
To install it, simply download comfy_latent_interposer.py to your ComfyUI/custom_nodes folder. You may need to install hfhub using the command pip install huggingface-hub inside your venv.
If you need the model weights for something else, they are hosted on HF under the same Apache2 license as the rest of the repo.
Usage
See the image below for an example on how to use it. xl=>v1 conversion is almost flawless, v1=>xl seems to produce artifacts.
Without the interposer, the two latent spaces are incompatible:
Training
The training script should spit out a working model, nn layout is probably not optimal but I'm pretty short on VRAM to trial and error a better layout. PRs welcome.
Not sure why the training loss is so different, it might be due to the """highly curated""" dataset of 1000 random images from my Downloads folder that I used to train it.
I probably should've just grabbed LAION.
I also trained a v1-to-v2 mode, which