nodes/city96-SD-Latent-Interposer
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
City
2023-07-30 06:34:50 +02:00
committed by GitHub
parent 1f6e20a422
commit 4e0979051d
1 changed files with 11 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+11 -2
View File
@@ -1,13 +1,20 @@
# SD-Latent-Interposer # SD-Latent-Interposer
A small neural network to provide interoperability between the latents generated by the different Stable Diffusion models. A small neural network to provide interoperability between the latents generated by the different Stable Diffusion models.
## Usage
I wanted to see if it was possible to pass latents generated by the new SDXL model directly into SDv1.5 models without decoding and re-encoding them using a VAE first. I wanted to see if it was possible to pass latents generated by the new SDXL model directly into SDv1.5 models without decoding and re-encoding them using a VAE first.
See below for an example on how to use it. xl=>v1 conversion is almost flawless, **v1=>xl seems to produce artifacts.** ## Installation
To install it, simply download [comfy_latent_interposer.py](https://github.com/city96/SD-Latent-Interposer/raw/main/comfy_latent_interposer.py) to your `ComfyUI/custom_nodes` folder. You may need to install hfhub using the command `pip install huggingface-hub` inside your venv.
If you need the model weights for something else, they are [hosted on HF](https://huggingface.co/city96/SD-Latent-Interposer/tree/main) under the same Apache2 license as the rest of the repo.
## Usage
See the image below for an example on how to use it. xl=>v1 conversion is almost flawless, **v1=>xl seems to produce artifacts.**
![LATENT_INTERPOSER_V3_TEST](https://github.com/city96/SD-Latent-Interposer/assets/125218114/4e15f7b6-e853-417d-ab58-205c1c99e507) ![LATENT_INTERPOSER_V3_TEST](https://github.com/city96/SD-Latent-Interposer/assets/125218114/4e15f7b6-e853-417d-ab58-205c1c99e507)
Without the interposer, the two latent spaces are incompatible:
![LATENT_INTERPOSER_V3 1](https://github.com/city96/SD-Latent-Interposer/assets/125218114/24e2864e-d20f-4977-b218-dff0bf0fdc9f) ![LATENT_INTERPOSER_V3 1](https://github.com/city96/SD-Latent-Interposer/assets/125218114/24e2864e-d20f-4977-b218-dff0bf0fdc9f)
## Training ## Training
@@ -17,6 +24,8 @@ Not sure why the training loss is so different, it might be due to the """highly
I probably should've just grabbed LAION. I probably should've just grabbed LAION.
I also trained a v1-to-v2 mode, which
### v1.0 Training loss/progress ### v1.0 Training loss/progress
![loss](https://github.com/city96/SD-Latent-Interposer/assets/125218114/89e996b9-3baa-4027-930c-38dc6ec5ec24) ![loss](https://github.com/city96/SD-Latent-Interposer/assets/125218114/89e996b9-3baa-4027-930c-38dc6ec5ec24)