69 lines
4.2 KiB
Markdown
69 lines
4.2 KiB
Markdown
# SD-Latent-Interposer
|
|
A small neural network to provide interoperability between the latents generated by the different Stable Diffusion models.
|
|
|
|
I wanted to see if it was possible to pass latents generated by the new SDXL model directly into SDv1.5 models without decoding and re-encoding them using a VAE first.
|
|
|
|
## Installation
|
|
To install it, simply clone this repo to your custom_nodes folder using the following command: `git clone https://github.com/city96/SD-Latent-Interposer custom_nodes/SD-Latent-Interposer`.
|
|
|
|
Alternatively, you can download the [comfy_latent_interposer.py](https://github.com/city96/SD-Latent-Interposer/raw/main/comfy_latent_interposer.py) file to your `ComfyUI/custom_nodes` folder as well. You may need to install hfhub using the command `pip install huggingface-hub` inside your venv.
|
|
|
|
If you need the model weights for something else, they are [hosted on HF](https://huggingface.co/city96/SD-Latent-Interposer/tree/main) under the same Apache2 license as the rest of the repo.
|
|
|
|
## Usage
|
|
See the image below for an example on how to use it. xl=>v1 conversion is almost flawless, **v1=>xl seems to produce artifacts.**
|
|
|
|
|
|
|
|
Without the interposer, the two latent spaces are incompatible:
|
|
|
|
|
|
|
|
### Local models
|
|
The node pulls the required files from huggingface hub by default. You can create a `models` folder and place the modules there if you have a flaky connection or prefer to use it completely offline. The custom node will prefer local files over HF when available. The path should be: `ComfyUI/custom_nodes/SD-Latent-Interposer/models`
|
|
|
|
Alternatively, just clone the entire HF repo to it: `git clone https://huggingface.co/city96/SD-Latent-Interposer custom_nodes/SD-Latent-Interposer/models`
|
|
|
|
## Training
|
|
Most of the training/preprocessing code is a 1:1 mirror from my latent upscaler. The folder layout it expects is also the same.
|
|
|
|
### Interposer v3.1
|
|
This is basically a complete rewrite. Replaced the mediocre bunch of conv2d layers with something that looks more like a proper neural network. No VGG loss because I still don't have a better GPU.
|
|
|
|
Training was done on combined Flickr2K + DIV2K, with each image being processed into 6 1024x1024 segments. Padded with some of my random images for a total of 22,000 source images in the dataset.
|
|
|
|
I think I got rid of most of the XL artifacts, but the color/hue/saturation shift issues are still there. I actually saved the optimizer state this time so I might be able to do 100K steps with visual loss on my P40s. Hopefully they won't burn up.
|
|
|
|
v3.0 was 500k steps at a constant LR of 1e-4, v3.1 was 1M steps using a CosineAnnealingLR to drop the learning rate towards the end. Both used AdamW.
|
|
|
|
|
|
|
|
### Older versions
|
|
|
|
<details><summary>Interposer v1.1</summary>
|
|
|
|
### Interposer v1.1
|
|
This is the second release using the "spaceship" architecture. It was trained on the Flickr2K dataset and was continued from the v1.0 checkpoint.
|
|
Overall, it seems to perform a lot better, especially for real life photos. I also investigated the odd v1->xl artifacts but in the end it seems [inherent to the VAE decoder stage.](https://github.com/comfyanonymous/ComfyUI/issues/1116)
|
|
|
|
|
|
|
|
</details>
|
|
|
|
<details><summary>Interposer v1.0</summary>
|
|
|
|
### Interposer v1.0
|
|
Not sure why the training loss is so different, it might be due to the """highly curated""" dataset of 1000 random images from my Downloads folder that I used to train it.
|
|
|
|
I probably should've just grabbed LAION.
|
|
|
|
I also trained a v1-to-v2 mode, before realizing v1 and v2 shared the same latent space. Oh well.
|
|
|
|
|
|
|
|
|
|
|
|
</details>
|
|
|
|
</details>
|