nodes/SeargeDP-SeargeSDXL
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Test release v3.991 - first test on the road to v4.0

Browse Source
This commit is contained in:
SeargeDP
2023-08-27 10:07:33 +02:00
parent 86a0865777
commit 5c4ef0cfe7
126 changed files with 15450 additions and 41298 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+218 -272
View File
@@ -1,340 +1,286 @@
# Searge-SDXL v3.x - "Truly Reborn"
*Custom nodes extension* for [ComfyUI](https://github.com/comfyanonymous/ComfyUI)
including *a workflow* to use *SDXL 1.0* with both the *base and refiner* checkpoints.
# Searge-SDXL: EVOLVED ~~v4.x~~ v3.991 for ComfyUI
# Version 3.4
Instead of having separate workflows for different tasks, everything is now integrated in **one workflow file**.
*Custom nodes extension* for [ComfyUI](https://github.com/comfyanonymous/ComfyUI),
**including a workflow** to use *SDXL 1.0* with both the *base and refiner* checkpoints.
### Always use the latest version of the workflow json file with the latest version of the custom nodes!
# Public test version 3.991
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/Searge-SDXL-Example.png" width="768">
This version is the first public test version of the huge update to version 4.0 and is 95% feature complete.
## Missing features
## What's new in v3.4?
- Minor tweaks and fixes and the beginnings of some code restructuring, nothing user should notice in the workflows
- Preparations for more upcoming improvements in a compatible way
- Added compatibility with v1.x workflows, these have been used in some tutorials and did not work anymore with newer
versions of the extension
- *(backwards compatibility with v2.x and older v3.x version - before v3.3 - is unfortunately not possible)*
Right now the following features are still missing and are planned for the complete v4.0 release:
## What about v3.3?
- Starting from v3.3 the custom node extension will always be compatible with workflows created with v3.3 or later
- *(backwards compatibility with v2.x, v3.0, v3.1. and v3.2 workflows is unfortunately not possible)*
- Going forward, older versions of workflow will remain in the `workflow` folder, I still highly recommend to **always
use the latest version** and loading it **from the JSON file** instead of the example images
- *Version 3.3 has never been publicly released*
## What's new in v3.2?
- More prompting modes, including the "3-prompt" style that's common in other workflows
using separate prompts for the 2 CLIP models in SDXL (CLIP G & CLIP L) and a negative prompt
- **3-Prompt G+L-N** - Similar to simple mode, but cares about *a main, a secondary, and a negative prompt*
and **ignores** the *additional style prompting fields*, this is great to get similar results as on other
workflows and makes it easier to compare the images
- **Subject - Style** - The *subject focused* positives with the *style focused* negatives
- **Style - Subject** - The *style focused* positives with the *subject focused* negatives
- **Style Only** - **Only** the positive and negative **style prompts** are used and *main/secondary/negative are ignored*
- **Weighted - Overlay** - The positive prompts are *weighted* and the negative prompts are *overlaid*
- **Overlay - Weighted** - The positive prompts are *overlaid* and the negative prompts are *weighted*
- Better bug fix for the "exploding" the search box issue, should finally be fixed *(for real)* now
- Some additional node types to make it easier to still use my nodes in other custom workflows
- The custom node extension should now also work on **Python 3.9** again, it required 3.10 before
## What's new in v3.1?
- Fixed the issue with "exploding" the search box when this extension is installed
- Loading of Checkpoints, VAE, Upscalers, and Loras through custom nodes
- Updated workflow to make use of the added node types
- Adjusted the default settings for some parameters in the workflow
- Fixed some reported issues with the workflow and custom nodes
- Prepared the workflow for an upcoming feature
## What's new in v3.0?
- Completely overhauled **user interface**, now even easier to use than before
- More organized workflow graph - if you want to understand how it is designed "under the hood", it should now be
easier to figure out what is where and how things are connected
- New settings that help to tweak the generated images *without changing the composition*
- Quickly iterate between *sharper* results and *softer* results of the same image without changing the composition
or subject
- Easily make colors pop where needed, or render a softer image where it fits the mood better
- Three operating modes in **ONE** workflow
- **text-to-image**
- **image-to-image**
- **inpainting**
- Different prompting modes (**5 modes** available)
- **Simple** - Just cares about **a positive and a negative prompt** and *ignores the additional prompting fields*, this
is great to get started with SDXL, ComfyUI, and this workflow
- **Subject Focus** - In this mode the *main/secondary prompts* are more important than the *style prompts*
- **Style Focus** - In this mode the *style prompts* are more important than the *main/secondary prompts*
- **Weighted** - In this mode the balance between *main/secondary prompts* and *style prompts* can be influenced with
the *style prompt power* and *negative prompt power* option
- **Overlay** - In this mode the main*/secondary prompts* and the *style prompts* are competing with each other
- Greatly *improved Hires-Fix* - now with more options to influence the results
- A (rather limited for now) alpha test for *style templates*, this is work in progress and only includes one
style for now (called *test*)
- Options to change the **intensity of the refiner** when used together with the base model,
separate for *main pass* and *hires-fix pass*
- *(... many more things probably, since the workflow was almost completely re-made)*
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/Searge-SDXL-UI.png" width="768">
- **Prompt Styles** - loading and applying style templates from a file
- **More Prompting Modes** - many of the unique prompting modes from v3.x are still missing and need to be
re-implemented in the new architecture of this extension
- **Condition Mixing** - this is the foundation for re-introducing the v3.x prompting modes, but it's planned
to have an even more flexible system to design your own custom prompting modes
# Installing and Updating:
# Table of Content
### Recommended Installation:
- Navigate to your `ComfyUI/custom_nodes/` directory
- Open a command line window in the *custom_nodes* directory
- Run `git clone https://github.com/SeargeDP/SeargeSDXL.git`
- Restart ComfyUI
<!-- TOC -->
* [Searge-SDXL: EVOLVED ~~v4.x~~ v3.991 for ComfyUI](#searge-sdxl-evolved-v4x-v3991-for-comfyui)
* [Public test version 3.991](#public-test-version-3991)
* [Missing features](#missing-features)
* [Table of Content](#table-of-content)
* [Version ~~4.0~~ 3.991](#version-40-3991)
* [Always use the latest version of the workflow json file with the latest version of the custom nodes!](#always-use-the-latest-version-of-the-workflow-json-file-with-the-latest-version-of-the-custom-nodes)
* [Installing and Updating](#installing-and-updating)
* [Recommended Installation of the Test Version](#recommended-installation-of-the-test-version)
* [Recommended Update of the Test Version](#recommended-update-of-the-test-version)
* [Checkpoints and Models for these Workflows](#checkpoints-and-models-for-these-workflows)
* [Direct Downloads](#direct-downloads)
* [Updates](#updates)
* [What's new in ~~v4.0~~ 3.991?](#whats-new-in-v40-3991)
* [Major Highlights](#major-highlights)
* [Smaller Changes and Additions](#smaller-changes-and-additions-)
* [The Workflow File](#the-workflow-file)
* [Documentation](#documentation)
* [Workflow Details](#workflow-details)
* [Operating Modes](#operating-modes)
* [Text to Image Mode](#text-to-image-mode)
* [Image to Image Mode](#image-to-image-mode)
* [Inpainting Mode](#inpainting-mode)
* [More Example Images](#more-example-images)
<!-- TOC -->
### Alternative Installation (not recommended):
- Download and unpack the latest release from the [Searge SDXL CivitAI page](https://civitai.com/models/111463)
- Drop the `SeargeSDXL` folder into the `ComfyUI/custom_nodes` directory and restart ComfyUI.
### Updating an Existing Installation
- Navigate to your `ComfyUI/custom_nodes/` directory
- If you installed via `git clone` before
- Open a command line window in the *custom_nodes* directory
- Run `git pull`
- If you installed from a zip file
- Unpack the `SeargeSDXL` folder from the latest release into `ComfyUI/custom_nodes`, overwrite existing files
- Restart ComfyUI
# Version ~~4.0~~ 3.991
Instead of having separate workflows for different tasks, everything is integrated in **one workflow file**.
## Always use the latest version of the workflow json file with the latest version of the custom nodes!
<img src="docs/img/main_readme/banner.png" width="768">
# Installing and Updating
## Recommended Installation of the Test Version
- Download and unpack the latest test release from the [Searge SDXL CivitAI page](https://civitai.com/models/111463) or
the [Github releases page for this project](https://github.com/SeargeDP/SeargeSDXL/releases).
- Drop the `SeargeSDXL-Test` folder into the `ComfyUI/custom_nodes` directory and restart ComfyUI.
### Recommended Update of the Test Version
- When new test versions are released, before the final v4.0 update release, repeat the steps from
the [Recommended Installation of the Test Version](#recommended-installation-of-the-test-version) section
and overwrite existing files in the process.
## Checkpoints and Models for these Workflows
This workflow depends on certain checkpoint files to be installed in ComfyUI, here is a list of the necessary
files that the workflow expects to be available.
If any of the mentioned folders does not exist in `ComfyUI/models`, **create** the missing folder and put the
downloaded file into it.
I recommend to **download and copy all** these files *(the required, recommended, and optional ones)* to make
**full use of all features** included in the workflow!
### Direct Downloads
(from Huggingface)
- download [SDXL 1.0 base](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors) and copy it into `ComfyUI/models/checkpoints`
- download [SDXL 1.0 refiner](https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0/resolve/main/sd_xl_refiner_1.0.safetensors) and copy it into `ComfyUI/models/checkpoints`
- download [Fixed SDXL 0.9 vae](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix/resolve/main/sdxl_vae.safetensors) and copy it into `ComfyUI/models/vae`
- download [SDXL Offset Noise LoRA](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_offset_example-lora_1.0.safetensors)
and copy it into `ComfyUI/models/loras`
- download [4x_NMKD-Siax_200k upscaler](https://huggingface.co/uwg/upscaler/resolve/main/ESRGAN/4x_NMKD-Siax_200k.pth) and copy it into `ComfyUI/models/upscale_models`
- download [4x-UltraSharp upscaler](https://huggingface.co/uwg/upscaler/resolve/main/ESRGAN/4x-UltraSharp.pth) and copy it into `ComfyUI/models/upscale_models`
- **(required)** download [SDXL 1.0 Base with 0.9 VAE (7 GB)](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0_0.9vae.safetensors) and copy it into `ComfyUI/models/checkpoints`
- *(this should be pre-selected as the base model on the workflow already)*
- **(recommended)** download [SDXL 1.0 Refiner with 0.9 VAE (6 GB)](https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0/resolve/main/sd_xl_refiner_1.0_0.9vae.safetensors) and copy it into `ComfyUI/models/checkpoints`
- *(you should select this as the refiner model on the workflow)*
- *(optional)* download [Fixed SDXL 0.9 vae (335 MB)](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix/resolve/main/sdxl_vae.safetensors) and copy it into `ComfyUI/models/vae`
- *(instead of using the VAE that's embedded in SDXL 1.0, this one has been fixed to work in fp16 and should **fix the issue with generating black images**)*
- *(optional)* download [SDXL Offset Noise LoRA (50 MB)](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_offset_example-lora_1.0.safetensors) and copy it into `ComfyUI/models/loras`
- *(the example lora that was released alongside SDXL 1.0, it can add more contrast through offset-noise)*
- **(recommended)** download [4x-UltraSharp (67 MB)](https://huggingface.co/uwg/upscaler/resolve/main/ESRGAN/4x-UltraSharp.pth) and copy it into `ComfyUI/models/upscale_models`
- *(you should select this as the primary upscaler on the workflow)*
- **(recommended)** download [4x_NMKD-Siax_200k (67 MB)](https://huggingface.co/uwg/upscaler/resolve/main/ESRGAN/4x_NMKD-Siax_200k.pth) and copy it into `ComfyUI/models/upscale_models`
- *(you should select this as the secondary upscaler on the workflow)*
- **(recommended)** download [4x_Nickelback_70000G (67 MB)](https://huggingface.co/uwg/upscaler/resolve/main/ESRGAN/4x_Nickelback_70000G.pth) and copy it into `ComfyUI/models/upscale_models`
- *(you should select this as the high-res upscaler on the workflow)*
- *(optional)* download [1x-ITF-SkinDiffDetail-Lite-v1 (20 MB)](https://huggingface.co/uwg/upscaler/resolve/main/ESRGAN/4x_Nickelback_70000G.pth) and copy it into `ComfyUI/models/upscale_models`
- *(you can select this as the detail processor on the workflow)*
- **(required)** download [ControlNetHED (30 MB)](https://huggingface.co/lllyasviel/Annotators/resolve/main/ControlNetHED.pth) and copy it into `ComfyUI/models/annotators`
- *(this will be used by the controlnet nodes)*
- **(required)** download [res101 (531 MB)](https://huggingface.co/lllyasviel/Annotators/resolve/main/res101.pth) and copy it into `ComfyUI/models/annotators`
- *(this will be used by the controlnet nodes)*
- **(recommended)** download [clip_vision_g (3.7 GB)](https://huggingface.co/stabilityai/control-lora/resolve/main/revision/clip_vision_g.safetensors) and copy it into `ComfyUI/models/clip_vision`
- *(you should select this as the clip vision model on the workflow)*
- **(recommended)** download [control-lora-canny-rank256 (774 MB)](https://huggingface.co/stabilityai/control-lora/resolve/main/control-LoRAs-rank256/control-lora-canny-rank256.safetensors) and copy it into `ComfyUI/models/controlnet`
- *(you should select this as the canny checkpoint on the workflow)*
- **(recommended)** download [control-lora-depth-rank256 (774 MB)](https://huggingface.co/stabilityai/control-lora/resolve/main/control-LoRAs-rank256/control-lora-depth-rank256.safetensors) and copy it into `ComfyUI/models/controlnet`
- *(you should select this as the depth checkpoint on the workflow)*
- **(recommended)** download [control-lora-recolor-rank256 (774 MB)](https://huggingface.co/stabilityai/control-lora/resolve/main/control-LoRAs-rank256/control-lora-recolor-rank256.safetensors) and copy it into `ComfyUI/models/controlnet`
- *(you should select this as the recolor checkpoint on the workflow)*
- **(recommended)** download [control-lora-sketch-rank256 (774 MB)](https://huggingface.co/stabilityai/control-lora/resolve/main/control-LoRAs-rank256/control-lora-sketch-rank256.safetensors) and copy it into `ComfyUI/models/controlnet`
- *(you should select this as the sketch checkpoint on the workflow)*
- *(optional)* download [OpenPoseXL2 (5 GB)](https://huggingface.co/thibaud/controlnet-openpose-sdxl-1.0/resolve/main/OpenPoseXL2.safetensors) and copy it into `ComfyUI/models/controlnet`
- *(you can select this as the custom controlnet checkpoint on the workflow)*
Now everything should be prepared, but you may to have to adjust some file names in the different model selector boxes
on the workflow. Do so by clicking on the filename in the workflow UI and selecting the correct file from the list.
<img src="docs/img/main_readme/full_graph.png" width="768">
# More Information
Now **3** operating modes are included in the workflow, the *.json-file* for it is in the `workflow` folder.
They are called *text2image*, *image2image*, and *inpainting*.
# Updates
The simple workflow has not returned as a separate workflow, but is now also *fully integrated*.
Find information about the latest changes here.
To enable it, switch the **prompt mode** option to **simple** and it will only pay attention to the *main prompt*
and the *negative prompt*.
Or switch the **prompt mode** to **3 prompts** and only the *main prompt*, the *secondary prompt*, and the
*negative prompt* are used.
## What's new in ~~v4.0~~ 3.991?
### Major Highlights
- A **complete re-write** of the custom node extension and the SDXL workflow
- **Highly optimized** processing pipeline, now **up to 20% faster** than in older workflow versions
- Support for **Controlnet and Revision**, up to 5 can be applied together
- **Multi-LoRA** support with up to 5 LoRA's at once
- ... (TODO: list more major highlights)
### Smaller Changes and Additions
- Workflows created with this extension and metadata embeddings in generated images are forward-compatible with
future updates of this project
- The custom node extension included in this project is backward-compatible with every workflow since version v3.3
- ... (TODO: list more smaller changes)
<br><img src="docs/img/main_readme/ui-3.png" width="768">
*(5 multi-purpose image inputs for revision and controlnet)*
# The Workflow
The workflow is included in the `workflow` folder.
# The Workflow File
**After updating Searge SDXL, always make sure to load the latest version of the json file. Older versions of the
workflow are often not compatible anymore with the updated node extension.**
The workflow is included as a `.json` file in the `workflow` folder.
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/Searge-SDXL-Overview.png" width="768">
**After updating Searge SDXL, always make sure to load the latest version of the json file if you want to benefit
from the latest features, updates, and bugfixes.**
(you can check the version of the workflow that you are using by looking at the workflow information box)
![Workflow Version](docs/img/main_readme/workflow_version.png)
## Documentation
[Click this link to see the documentation](docs/readme.md)
<img src="docs/img/main_readme/ui-1.png" width="768">
*(the main UI of the workflow)*
# Searge SDXL Reborn Workflow Description
The **Reborn v3.x** workflow is a new workflow, created from scratch. It requires the latest additions to the
# Workflow Details
The **EVOLVED v4.x** workflow is a new workflow, created from scratch. It requires the latest additions to the
SeargeSDXL custom node extension, because it makes use of some new node types.
The interface for using this new workflow is also designed in a different way, with all parameters that
are usually tweaked to generate images tightly packed together. This should make it easier to have every
important element on the screen at the same time without scrolling.
Starting from version 3.0 all 3 operating modes (text-to-image, image-to-image, and inpainting) are available
from the same workflow and can be switched with an option.
<img src="docs/img/main_readme/ui-2.png" width="768">
## Video
[The amazing Youtube channel Nerdy Rodent has a video about this workflow](https://www.youtube.com/watch?v=_Qi0Dgrz1TM).
*(and while you are watching the video, don't forget to subscribe to their channel)*
*(more advanced UI elements right next to the main UI)*
## Reborn Workflow v3.x Operating Modes
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/UI-operation-mode.png" width="512">
## Operating Modes
![Workflow Version](docs/img/main_readme/operating_mode.png)
### Text to Image Mode
In this mode you can generate images from text descriptions. The source image and the mask (next to the prompt inputs)
are not used in this mode.
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/Searge-SDXL-workflow-1.png" width="768">
<img src="docs/img/main_readme/ui_txt2img.png" width="768">
*(example of using text-to-image in the workflow)*
<br>
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/Searge-SDXL-reborn.png" width="512">
<img src="docs/img/main_readme/result_txt2img.png" width="512">
*(result of the text-to-image example)*
### Image to Image Mode
In this mode you should first copy an image into the `ConfyUI/input` directory.
Alternatively you can change the option for the **save directory** to **input folder** when generating images, in that
case you have to press the ComfyUI *Refresh* button and it should show up in the image loader node.
Then select that image as the *Source Image* (next to the prompt inputs).
If it does not show up, press the *Refresh* button on the Comfy UI control box.
In this mode you can generate images from text descriptions and a source image. The mask (next to the prompt inputs)
is not used in this mode.
For image to image the parameter *Denoise* will determine how much the source image should be changed
according to the prompt.
Ranges are from *0.0* for "no change" to *1.0* for "completely change".
<img src="docs/img/main_readme/ui_img2img.png" width="768">
Good values to try are probably in the *0.2* to *0.8* range.
With examples of *0.25* for "very little change", *0.5* for "some changes", or *0.75* for "a lot of changes"
*(example of using image-to-image in the workflow)*
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/Searge-SDXL-workflow-2.png" width="768">
<br>
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/Searge-SDXL-img2img.png" width="512">
<img src="docs/img/main_readme/result_img2img.png" width="512">
*(result of the image-to-image example)*
### Inpainting Mode
This is similar to the image to image mode.
But it also lets you define a mask for selective changes of only parts of the image.
To use this mode, prepare a source image the same way as described in the image to image workflow.
Then **right click** on the *Inpainting Mask* image (the bottom one next to the input prompts) and select
**Open in Mask Editor**.
In this mode you can generate images from text descriptions and a source image. Both, the source image and the mask
(next to the prompt inputs) are used in this mode.
Paint your mask and then press the *Save to node* button when you are done.
The *Denoise* parameter works the same way as in image to image, but only masked areas will be changed.
This is similar to the image to image mode, but it also lets you define a mask for selective changes of only parts
of the image.
<img src="docs/img/main_readme/ui_inpainting.png" width="768">
*(example of using inpainting in the workflow)*
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/Searge-SDXL-workflow-3.png" width="768">
<br>
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/Searge-SDXL-inpaint.png" width="512">
<img src="docs/img/main_readme/result_inpainting.png" width="512">
*(result of the inpainting example)*
# Prompting Modes
# More Example Images
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/UI-prompt-style.png" width="512">
## Reborn Workflow v3.x Prompting Modes
### Simple
Just cares about the **main** and the **negative** prompt and **ignores** the *additional prompting fields*, this
is great to get started with SDXL, ComfyUI, and this workflow
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/01-simple.jpg" width="512">
### 3-Prompt G+L-N
Similar to simple mode, but cares about the **main & secondary** and the **negative** prompt
and **ignores** the *additional style prompting fields*, this is great to get similar results as on other
workflows and makes it easier to compare the images
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/02-3_prompts.jpg" width="512">
### Subject Focus
In this mode the *main & secondary* prompts are **more important** than the *style* prompts
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/03-subject_focus.jpg" width="512">
### Style Focus
In this mode the *style* prompts are **more important** than the *main & secondary* prompts
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/04-style_focus.jpg" width="512">
### Weighted
In this mode the **balance** between *main & secondary* prompts and *style prompts* can be influenced with
the **style prompt power** and **negative prompt power** option
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/05-weighted.jpg" width="512">
### Overlay
In this mode the *main & secondary* prompts and the *style* prompts are **competing with each other**
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/06-overlay.jpg" width="512">
### Subject - Style
The *main & secondary* positives with the *style* negatives
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/07-subject-style.jpg" width="512">
### Style - Subject
The *style* positives with the *main & secondary* negatives
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/08-style-subject.jpg" width="512">
### Style Only
**Only** the *style* prompt and *negative style* prompt are used, the *main & secondary* and *negative* are ignored
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/09-style_only.jpg" width="512">
### Weighted - Overlay
The *main & secondary* and *style* prompts are **weighted**, the *negative* and *negative style* prompts are **overlaid**
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/10-weighted-overlay.jpg" width="512">
### Overlay - Weighted
The *main & secondary* and *style* prompts are **overlaid**, the *negative* and *negative style* prompts are **weighted**
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/11-overlay-weighted.jpg" width="512">
# Custom Nodes
These custom node types are available in the extension.
The details about them are only important if you want to use them in your own workflow or if you want to
understand better how the included workflows work.
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/Searge-SDXL-Nodetypes.png" width="768">
## SDXL Sampler Node
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/Searge-SDXL-Node-1.png" width="407">
### Inputs
- **base_model** - connect the SDXL base model here, provided via a `Load Checkpoint` node
- **base_positive** - recommended to use a `CLIPTextEncodeSDXL` with 4096 for `width`, `height`,
`target_width`, and `target_height`
- **base_negative** - recommended to use a `CLIPTextEncodeSDXL` with 4096 for `width`, `height`,
`target_width`, and `target_height`
- **refiner_model** - connect the SDXL refiner model here, provided via a `Load Checkpoint` node
- **refiner_positive** - recommended to use a `CLIPTextEncodeSDXLRefiner` with 2048 for `width`, and `height`
- **refiner_negative** - recommended to use a `CLIPTextEncodeSDXLRefiner` with 2048 for `width`, and `height`
- **latent_image** - either an empty latent image or a VAE-encoded latent from a source image for img2img
- **noise_seed** - the random seed for generating the image
- **steps** - total steps for the sampler, it will internally be split into base steps and refiner steps
- **cfg** - CFG scale (classifier free guidance), values between 3.0 and 12.0 are most commonly used
- **sampler_name** - the noise sampler _(I prefer dpmpp_2m with the karras scheduler, sometimes ddim
with the ddim_uniform scheduler)_
- **scheduler** - the scheduler to use with the sampler selected in `sampler_name`
- **base_ratio** - the ratio between base model steps and refiner model steps _(0.8 = 80% base model and 20% refiner
model, with 30 total steps that's 24 base steps and 6 refiner steps)_
- **denoise** - denoising factor, keep this at 1.0 when creating new images from an empty latent and between 0.0-1.0 in the img2img workflow
### Outputs
- **LATENT** - the generated latent image
## SDXL Prompt Node
<img src="https://github.com/SeargeDP/SeargeSDXL/blob/main/example/Searge-SDXL-Node-2.png" width="434">
### Inputs
- **base_clip** - connect the SDXL base CLIP here, provided via a `Load Checkpoint` node
- **refiner_clip** - connect the SDXL refiner CLIP here, provided via a `Load Checkpoint` node
- **pos_g** - the text for the positive base prompt G
- **pos_l** - the text for the positive base prompt L
- **pos_r** - the text for the positive refiner prompt
- **neg_g** - the text for the negative base prompt G
- **neg_l** - the text for the negative base prompt L
- **neg_r** - the text for the negative refiner prompt
- **base_width** - the width for the base conditioning
- **base_height** - the height for the base conditioning
- **crop_w** - crop width for the base conditioning
- **crop_h** - crop height for the base conditioning
- **target_width** - the target width for the base conditioning
- **target_height** - the target height for the base conditioning
- **pos_ascore** - the positive aesthetic score for the refiner conditioning
- **neg_ascore** - the negative aesthetic score for the refiner conditioning
- **refiner_width** - the width for the refiner conditioning
- **refiner_height** - the height for the refiner conditioning
### Outputs
- **CONDITIONING** 1 - the positive base prompt conditioning
- **CONDITIONING** 2 - the negative base prompt conditioning
- **CONDITIONING** 3 - the positive refiner prompt conditioning
- **CONDITIONING** 4 - the negative refiner prompt conditioning
A small collection of example images (with embedded workflow) can be found in the `examples` folder. [Here is an
overview of the included images.](examples/readme.md)
__init__.py
+19 -1
View File
@@ -26,7 +26,25 @@ SOFTWARE.
"""
from .searge_sdxl_sampler_node import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
import os
import folder_paths
from .modules.ui import Defs
from .searge_sdxl import SEARGE_CLASS_MAPPINGS, SEARGE_DISPLAY_NAME_MAPPINGS
folder_paths.add_model_folder_path("annotators", os.path.join(folder_paths.models_dir, "annotators"))
if Defs.DEV_MODE:
NODE_CLASS_MAPPINGS = SEARGE_CLASS_MAPPINGS
NODE_DISPLAY_NAME_MAPPINGS = SEARGE_DISPLAY_NAME_MAPPINGS
else:
pass
# from .modules._legacy import LEGACY_CLASS_MAPPINGS, LEGACY_DISPLAY_NAME_MAPPINGS
# NODE_CLASS_MAPPINGS = SEARGE_CLASS_MAPPINGS | LEGACY_CLASS_MAPPINGS
# NODE_DISPLAY_NAME_MAPPINGS = SEARGE_DISPLAY_NAME_MAPPINGS | LEGACY_DISPLAY_NAME_MAPPINGS
__all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS']
docs/img/main_readme/banner.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.3 MiB

docs/img/main_readme/full_graph.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 110 KiB

docs/img/main_readme/operating_mode.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 10 KiB

docs/img/main_readme/result_img2img.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.4 MiB

docs/img/main_readme/result_inpainting.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.5 MiB

docs/img/main_readme/result_txt2img.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.3 MiB

docs/img/main_readme/ui-1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 612 KiB

docs/img/main_readme/ui-2.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.1 MiB

docs/img/main_readme/ui-3.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 481 KiB

docs/img/main_readme/ui_img2img.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 955 KiB

docs/img/main_readme/ui_inpainting.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.1 MiB

docs/img/main_readme/ui_txt2img.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 630 KiB

docs/img/main_readme/workflow_version.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

docs/img/model_selector.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

docs/img/queue_prompt.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 5.6 KiB

docs/readme.md
+44
View File
@@ -0,0 +1,44 @@
# Searge-SDXL: EVOLVED v4.x for ComfyUI
*(this documentation is work-in-progress and incomplete)*
<!-- TOC -->
* [Searge-SDXL: EVOLVED v4.x for ComfyUI](#searge-sdxl-evolved-v4x-for-comfyui)
* [Getting Started with the Workflow](#getting-started-with-the-workflow)
* [Testing the workflow](#testing-the-workflow)
* [Detailed Documentation](#detailed-documentation)
<!-- TOC -->
## Getting Started with the Workflow
After installing the required model files as described in the section *Checkpoints and Models* of the
[main readme file](../README.md#checkpoints-and-models-for-these-workflows)
for this project, follow these steps to test if everything has been installed correctly and is working properly.
**Also make sure you are using the latest `.json` file from the `workflow` folder.**
You can see the version information on the workflow and it should match the latest released version of this project.
![Model Selector](img/main_readme/workflow_version.png)
## Testing the workflow
To get started, select the base model, refiner model, and VAE in the model selector by clicking on the fields and
selecting the correct files. Using the **refiner is highly recommended** for best results. The recommended VAE is
a fixed version that works in fp16 mode without producing just black images, but if you don't want to use a separate
VAE file just select *from base model*.
![Model Selector](img/model_selector.png)
Once you selected the correct models press the Queue Prompt button in ComfyUI to test if everything is set up
correctly.
![Queue Prompt](img/queue_prompt.png)
# Detailed Documentation
*(TBD)*
example/01-simple.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.0 MiB

example/02-3_prompts.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.0 MiB

example/03-subject_focus.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.1 MiB

example/04-style_focus.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.2 MiB

example/05-weighted.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.1 MiB

example/06-overlay.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.1 MiB

example/07-subject-style.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.2 MiB

example/08-style-subject.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.1 MiB

example/09-style_only.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.1 MiB

example/10-weighted-overlay.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.0 MiB

example/11-overlay-weighted.jpg
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.1 MiB

example/Searge-SDXL-Example.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 5.5 MiB

example/Searge-SDXL-Node-1.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 30 KiB

example/Searge-SDXL-Node-2.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 24 KiB

example/Searge-SDXL-Nodetypes.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 146 KiB

example/Searge-SDXL-Overview.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 241 KiB

example/Searge-SDXL-UI.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 747 KiB

example/Searge-SDXL-img2img.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.5 MiB

example/Searge-SDXL-inpaint.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.5 MiB

example/Searge-SDXL-reborn.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.5 MiB

example/Searge-SDXL-workflow-1.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 709 KiB

example/Searge-SDXL-workflow-2.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 876 KiB

example/Searge-SDXL-workflow-3.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 809 KiB

example/UI-operation-mode.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 73 KiB

example/UI-prompt-style.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 73 KiB

examples/example-1-quick-iteration.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.3 MiB

examples/example-2-high-quality.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.9 MiB

examples/example-2-quality-settings.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 519 KiB

examples/example-3-watching-the-universe-end.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.2 MiB

examples/example-4-albert-einstein.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.2 MiB

examples/example-5-dragon.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.5 MiB

examples/example-6-revision-ghost.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.1 MiB

examples/example-7-controlnet-sketch-warrior.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.2 MiB

examples/readme.md
+57
View File
@@ -0,0 +1,57 @@
# Example Images
All images in this folder, except `example-2-quality-settings.png`, have the workflow embedded and can be loaded into
ComfyUI to re-create them.
To do that either load the images via the `Load` button or just drag one of the images
on the browser window that has ComfyUI loaded.
## Example 1 - Quick Iteration
This image uses the default settings from the `.json` file and should generate quickly. These settings are a great
starting point to quickly test a prompt or generate batches of images to pick from for further refinement.
<img src="example-1-quick-iteration.png" width="512">
## Example 2 - High Quality
This is a variation of example 1, but with some settings changed to produce a higher quality and higher resolution
result in the end.
<img src="example-2-high-quality.png" width="512">
Here is an overview of the main settings that can be used to get higher quality results.
<img src="example-2-quality-settings.png" width="512">
## Example 3 - Watching the Universe End
Using a prompt from one of the signature images that were used in an older version's release on CivitAI.
<img src="example-3-watching-the-universe-end.png" width="512">
## Example 4 - Albert Einstein Running a Marathon
This was an idea for some test images generated with an older version of this workflow during development.
<img src="example-4-albert-einstein.png" width="512">
# Example 5 - Dragon
Who doesn't like dragons? We had to have one in these examples.
<img src="example-5-dragon.png" width="512">
# Example 6 - Revision Ghost
Based on a prompt idea I found on one of the Discord servers. Re-creating a similar effect without the use of a LoRA,
but using one of the 5 included controlnet/revision stages instead.
<img src="example-6-revision-ghost.png" width="512">
# Example 7 - Controlnet Sketch Warrior
Using one of the 5 included controlnet/revision stages to create a realistic looking image of a warrior.
<img src="example-7-controlnet-sketch-warrior.png" width="512">
modules/__init__.py
+3 -1
View File
@@ -26,4 +26,6 @@ SOFTWARE.
"""
LegacyVersion = "v1.x"
from .ui import Defs
print("Searge-SDXL v" + Defs.VERSION + ("-dev" if Defs.DEV_MODE else "") + " in " + Defs.EXTENSION_PATH)
modules/_experimental.py
+80
View File
@@ -0,0 +1,80 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
import torch
def gaussian_latent_noise(width=128, height=128, seed=-1, fac=0.5, batch_size=1, nul=0.0, srnd=False, ver="xl"):
limit = {
"v1": {
"min": {"A": -5.5618, "B": -17.1368, "C": -10.3445, "D": -8.6218},
"max": {"A": 13.5369, "B": 11.1997, "C": 16.3043, "D": 10.6343},
"nul": {"A": -5.3870, "B": -14.2931, "C": 6.2738, "D": 7.1220},
},
"xl": {
"min": {"A": -22.2127, "B": -20.0131, "C": -17.7673, "D": -14.9434},
"max": {"A": 17.9334, "B": 26.3043, "C": 33.1648, "D": 8.9380},
"nul": {"A": -21.9287, "B": 3.8783, "C": 2.5879, "D": 2.5435},
}
}
# seed
if seed >= 0:
torch.manual_seed(seed)
limit = limit[ver]
out = []
for i in range(batch_size):
if srnd: # shared random
rand = torch.rand([height, width])
lat = torch.stack([
(limit["min"]["A"] + torch.clone(rand) * (limit["max"]["A"] - limit["min"]["A"])),
(limit["min"]["B"] + torch.clone(rand) * (limit["max"]["B"] - limit["min"]["B"])),
(limit["min"]["C"] + torch.clone(rand) * (limit["max"]["C"] - limit["min"]["C"])),
(limit["min"]["D"] + torch.clone(rand) * (limit["max"]["D"] - limit["min"]["D"])),
])
else: # separate random
lat = torch.stack([
(limit["min"]["A"] + torch.rand([height, width]) * (limit["max"]["A"] - limit["min"]["A"])),
(limit["min"]["B"] + torch.rand([height, width]) * (limit["max"]["B"] - limit["min"]["B"])),
(limit["min"]["C"] + torch.rand([height, width]) * (limit["max"]["C"] - limit["min"]["C"])),
(limit["min"]["D"] + torch.rand([height, width]) * (limit["max"]["D"] - limit["min"]["D"])),
])
tnul = torch.stack([ # black image
torch.ones([height, width]) * limit["nul"]["A"],
torch.ones([height, width]) * limit["nul"]["B"],
torch.ones([height, width]) * limit["nul"]["C"],
torch.ones([height, width]) * limit["nul"]["D"],
])
out.append(((lat * fac) * (1.0 - nul) + tnul * nul) / 2)
return {"samples": torch.stack(out)}
modules/after_upscaling.py
+62
View File
@@ -0,0 +1,62 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .data_utils import retrieve_parameter
from .names import Names
from .ui import UI
# ====================================================================================================
# Output from magic box for custom stage after a VAE decode
# ====================================================================================================
class SeargeCustomAfterUpscaling:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"custom_output": ("SRG_STAGE_OUTPUT",),
},
"optional": {
},
}
RETURN_TYPES = ("IMAGE",)
RETURN_NAMES = ("image",)
FUNCTION = "output"
CATEGORY = UI.CATEGORY_MAGIC_CUSTOM_STAGES
def output(self, custom_output):
if custom_output is None:
return (None,)
vae_decoded = retrieve_parameter(Names.S_UPSCALED, custom_output)
image = retrieve_parameter(Names.F_UPSCALED_IMAGE, vae_decoded)
return (image,)
modules/after_vae_decode.py
+64
View File
@@ -0,0 +1,64 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .data_utils import retrieve_parameter
from .names import Names
from .ui import UI
# ====================================================================================================
# Output from magic box for custom stage after a VAE decode
# ====================================================================================================
class SeargeCustomAfterVaeDecode:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"custom_output": ("SRG_STAGE_OUTPUT",),
},
"optional": {
},
}
RETURN_TYPES = ("IMAGE",)
RETURN_NAMES = ("image",)
FUNCTION = "output"
CATEGORY = UI.CATEGORY_MAGIC_CUSTOM_STAGES
def output(self, custom_output):
if custom_output is None:
return (None,)
vae_decoded = retrieve_parameter(Names.S_VAE_DECODED, custom_output)
image = retrieve_parameter(Names.F_DECODED_IMAGE, vae_decoded)
post_processed = retrieve_parameter(Names.F_POST_PROCESSED, vae_decoded)
result = image if post_processed is None else post_processed
return (result,)
modules/controlnet/ATTRIBUTION
+20
View File
@@ -0,0 +1,20 @@
This uses code from:
Leres Depth Map:
- Bob Thiry (https://github.com/thygate/stable-diffusion-webui-depthmap-script)
(released under MIT license)
Controlnet Aux:
- Fannovel16 (https://github.com/Fannovel16/comfyui_controlnet_aux)
(released under Apache 2.0 license)
Controlnet:
- lllyasviel (https://github.com/lllyasviel/ControlNet/tree/main/annotator)
(released under Apache 2.0 license)
Functions block_reduce and view_as_blocks:
- scikit-image (https://github.com/scikit-image/scikit-image)
(released under BSD 3-clause license)
Changes were made by Searge in August 2023 for the project https://github.com/SeargeDP/SeargeSDXL
modules/controlnet/LICENSE
+201
View File
@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
modules/controlnet/__init__.py
+48
View File
@@ -0,0 +1,48 @@
import comfy.model_management as model_management
from .canny import CannyDetector
from .hed import HEDdetector
from .leres import LeresDetector
from .utils import annotator_wrapper
def canny(image, low_threshold, high_threshold):
annotator = CannyDetector()
low_threshold = int(low_threshold * 255)
high_threshold = int(high_threshold * 255)
def annotate(np_image):
return annotator(np_image, low_threshold=low_threshold, high_threshold=high_threshold)
return annotator_wrapper(image, annotate)
def leres(image, rm_nearest, rm_background, annotator_model):
annotator = LeresDetector.from_pretrained(annotator_model).to(model_management.get_torch_device())
rm_nearest = rm_nearest * 100
rm_background = (1.0 - rm_background) * 100
def annotate(np_image):
return annotator(np_image, thr_a=rm_background, thr_b=rm_nearest)
out = annotator_wrapper(image, annotate)
del annotator
return out
def hed(image, annotator_model):
annotator = HEDdetector.from_pretrained(annotator_model).to(model_management.get_torch_device())
def annotate(np_image):
return annotator(np_image, safe=True)
out = annotator_wrapper(image, annotate)
del annotator
return out
modules/controlnet/canny/__init__.py
+32
View File
@@ -0,0 +1,32 @@
import cv2
import numpy as np
from PIL import Image
from ..utils import HWC3, resize_image
class CannyDetector:
def __call__(self, input_image=None, low_threshold=100, high_threshold=200, detect_resolution=512, image_resolution=512, output_type="np"):
if input_image is None:
raise ValueError("input_image must be defined.")
if not isinstance(input_image, np.ndarray):
input_image = np.array(input_image, dtype=np.uint8)
output_type = output_type or "pil"
else:
output_type = output_type or "np"
input_image = HWC3(input_image)
input_image = resize_image(input_image, detect_resolution)
detected_map = cv2.Canny(input_image, low_threshold, high_threshold)
detected_map = HWC3(detected_map)
img = resize_image(input_image, image_resolution)
H, W, C = img.shape
detected_map = cv2.resize(detected_map, (W, H), interpolation=cv2.INTER_LINEAR)
if output_type == "pil":
detected_map = Image.fromarray(detected_map)
return detected_map
modules/controlnet/hed/__init__.py
+119
View File
@@ -0,0 +1,119 @@
# This is an improved version and model of HED edge detection with Apache License, Version 2.0.
# Please use this implementation in your products
# This implementation may produce slightly different results from Saining Xie's official implementations,
# but it generates smoother edges and is more suitable for ControlNet as well as other image-to-image translations.
# Different from official models and other implementations, this is an RGB-input model (rather than BGR)
# and in this way it works better for gradio's RGB protocol
import os
import cv2
import numpy as np
import torch
from einops import rearrange
from PIL import Image
from ..utils import HWC3, resize_image, nms, safe_step
class DoubleConvBlock(torch.nn.Module):
def __init__(self, input_channel, output_channel, layer_number):
super().__init__()
self.convs = torch.nn.Sequential()
self.convs.append(torch.nn.Conv2d(in_channels=input_channel, out_channels=output_channel, kernel_size=(3, 3), stride=(1, 1), padding=1))
for i in range(1, layer_number):
self.convs.append(torch.nn.Conv2d(in_channels=output_channel, out_channels=output_channel, kernel_size=(3, 3), stride=(1, 1), padding=1))
self.projection = torch.nn.Conv2d(in_channels=output_channel, out_channels=1, kernel_size=(1, 1), stride=(1, 1), padding=0)
def __call__(self, x, down_sampling=False):
h = x
if down_sampling:
h = torch.nn.functional.max_pool2d(h, kernel_size=(2, 2), stride=(2, 2))
for conv in self.convs:
h = conv(h)
h = torch.nn.functional.relu(h)
return h, self.projection(h)
class ControlNetHED_Apache2(torch.nn.Module):
def __init__(self):
super().__init__()
self.norm = torch.nn.Parameter(torch.zeros(size=(1, 3, 1, 1)))
self.block1 = DoubleConvBlock(input_channel=3, output_channel=64, layer_number=2)
self.block2 = DoubleConvBlock(input_channel=64, output_channel=128, layer_number=2)
self.block3 = DoubleConvBlock(input_channel=128, output_channel=256, layer_number=3)
self.block4 = DoubleConvBlock(input_channel=256, output_channel=512, layer_number=3)
self.block5 = DoubleConvBlock(input_channel=512, output_channel=512, layer_number=3)
def __call__(self, x):
h = x - self.norm
h, projection1 = self.block1(h)
h, projection2 = self.block2(h, down_sampling=True)
h, projection3 = self.block3(h, down_sampling=True)
h, projection4 = self.block4(h, down_sampling=True)
h, projection5 = self.block5(h, down_sampling=True)
return projection1, projection2, projection3, projection4, projection5
class HEDdetector:
def __init__(self, netNetwork):
self.netNetwork = netNetwork
@classmethod
def from_pretrained(cls, pretrained_model_or_path, filename=None):
filename = filename or "ControlNetHED.pth"
if os.path.isdir(pretrained_model_or_path):
model_path = os.path.join(pretrained_model_or_path, filename)
else:
model_path = pretrained_model_or_path
netNetwork = ControlNetHED_Apache2()
netNetwork.load_state_dict(torch.load(model_path, map_location='cpu'))
netNetwork.float().eval()
return cls(netNetwork)
def to(self, device):
self.netNetwork.to(device)
return self
def __call__(self, input_image, detect_resolution=512, image_resolution=512, safe=False, output_type="np", scribble=False):
device = next(iter(self.netNetwork.parameters())).device
if not isinstance(input_image, np.ndarray):
input_image = np.array(input_image, dtype=np.uint8)
input_image = HWC3(input_image)
input_image = resize_image(input_image, detect_resolution)
assert input_image.ndim == 3
H, W, C = input_image.shape
with torch.no_grad():
image_hed = torch.from_numpy(input_image.copy()).float().to(device)
image_hed = rearrange(image_hed, 'h w c -> 1 c h w')
edges = self.netNetwork(image_hed)
edges = [e.detach().cpu().numpy().astype(np.float32)[0, 0] for e in edges]
edges = [cv2.resize(e, (W, H), interpolation=cv2.INTER_LINEAR) for e in edges]
edges = np.stack(edges, axis=2)
edge = 1 / (1 + np.exp(-np.mean(edges, axis=2).astype(np.float64)))
if safe:
edge = safe_step(edge)
edge = (edge * 255.0).clip(0, 255).astype(np.uint8)
detected_map = edge
detected_map = HWC3(detected_map)
img = resize_image(input_image, image_resolution)
H, W, C = img.shape
detected_map = cv2.resize(detected_map, (W, H), interpolation=cv2.INTER_LINEAR)
if scribble:
detected_map = nms(detected_map, 127, 3.0)
detected_map = cv2.GaussianBlur(detected_map, (0, 0), 3.0)
detected_map[detected_map > 4] = 255
detected_map[detected_map < 255] = 0
if output_type == "pil":
detected_map = Image.fromarray(detected_map)
return detected_map
modules/controlnet/leres/__init__.py
+95
View File
@@ -0,0 +1,95 @@
import os
import cv2
import numpy as np
import torch
from PIL import Image
from ..utils import HWC3, resize_image
from .leres.depthmap import estimateleres
from .leres.multi_depth_model_woauxi import RelDepthModel
from .leres.net_tools import strip_prefix_if_present
class LeresDetector:
def __init__(self, model):
self.model = model
@classmethod
def from_pretrained(cls, pretrained_model_or_path, filename=None):
filename = filename or "res101.pth"
if os.path.isdir(pretrained_model_or_path):
model_path = os.path.join(pretrained_model_or_path, filename)
else:
model_path = pretrained_model_or_path
checkpoint = torch.load(model_path, map_location=torch.device('cpu'))
model = RelDepthModel(backbone='resnext101')
model.load_state_dict(strip_prefix_if_present(checkpoint['depth_model'], "module."), strict=True)
del checkpoint
return cls(model)
def to(self, device):
self.model.to(device)
return self
def __call__(self, input_image, thr_a=0, thr_b=0, detect_resolution=512, image_resolution=512, output_type="np"):
if not isinstance(input_image, np.ndarray):
input_image = np.array(input_image, dtype=np.uint8)
input_image = HWC3(input_image)
input_image = resize_image(input_image, detect_resolution)
assert input_image.ndim == 3
height, width, dim = input_image.shape
with torch.no_grad():
depth = estimateleres(input_image, self.model, width, height)
numbytes=2
depth_min = depth.min()
depth_max = depth.max()
max_val = (2**(8*numbytes))-1
# check output before normalizing and mapping to 16 bit
if depth_max - depth_min > np.finfo("float").eps:
out = max_val * (depth - depth_min) / (depth_max - depth_min)
else:
out = np.zeros(depth.shape)
# single channel, 16 bit image
depth_image = out.astype("uint16")
# convert to uint8
depth_image = cv2.convertScaleAbs(depth_image, alpha=(255.0/65535.0))
# remove near
if thr_a != 0:
thr_a = ((thr_a/100)*255)
depth_image = cv2.threshold(depth_image, thr_a, 255, cv2.THRESH_TOZERO)[1]
# invert image
depth_image = cv2.bitwise_not(depth_image)
# remove bg
if thr_b != 0:
thr_b = ((thr_b/100)*255)
depth_image = cv2.threshold(depth_image, thr_b, 255, cv2.THRESH_TOZERO)[1]
detected_map = depth_image
detected_map = HWC3(detected_map)
img = resize_image(input_image, image_resolution)
H, W, C = img.shape
detected_map = cv2.resize(detected_map, (W, H), interpolation=cv2.INTER_LINEAR)
if output_type == "pil":
detected_map = Image.fromarray(detected_map)
return detected_map
modules/controlnet/leres/leres/LICENSE
+23
View File
@@ -0,0 +1,23 @@
https://github.com/thygate/stable-diffusion-webui-depthmap-script
MIT License
Copyright (c) 2023 Bob Thiry
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
modules/controlnet/leres/leres/Resnet.py
+199
View File
@@ -0,0 +1,199 @@
import torch.nn as nn
import torch.nn as NN
__all__ = ['ResNet', 'resnet18', 'resnet34', 'resnet50', 'resnet101',
'resnet152']
model_urls = {
'resnet18': 'https://download.pytorch.org/models/resnet18-5c106cde.pth',
'resnet34': 'https://download.pytorch.org/models/resnet34-333f7ec4.pth',
'resnet50': 'https://download.pytorch.org/models/resnet50-19c8e357.pth',
'resnet101': 'https://download.pytorch.org/models/resnet101-5d3b4d8f.pth',
'resnet152': 'https://download.pytorch.org/models/resnet152-b121ed2d.pth',
}
def conv3x3(in_planes, out_planes, stride=1):
"""3x3 convolution with padding"""
return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,
padding=1, bias=False)
class BasicBlock(nn.Module):
expansion = 1
def __init__(self, inplanes, planes, stride=1, downsample=None):
super(BasicBlock, self).__init__()
self.conv1 = conv3x3(inplanes, planes, stride)
self.bn1 = NN.BatchNorm2d(planes) #NN.BatchNorm2d
self.relu = nn.ReLU(inplace=True)
self.conv2 = conv3x3(planes, planes)
self.bn2 = NN.BatchNorm2d(planes) #NN.BatchNorm2d
self.downsample = downsample
self.stride = stride
def forward(self, x):
residual = x
out = self.conv1(x)
out = self.bn1(out)
out = self.relu(out)
out = self.conv2(out)
out = self.bn2(out)
if self.downsample is not None:
residual = self.downsample(x)
out += residual
out = self.relu(out)
return out
class Bottleneck(nn.Module):
expansion = 4
def __init__(self, inplanes, planes, stride=1, downsample=None):
super(Bottleneck, self).__init__()
self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False)
self.bn1 = NN.BatchNorm2d(planes) #NN.BatchNorm2d
self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride,
padding=1, bias=False)
self.bn2 = NN.BatchNorm2d(planes) #NN.BatchNorm2d
self.conv3 = nn.Conv2d(planes, planes * self.expansion, kernel_size=1, bias=False)
self.bn3 = NN.BatchNorm2d(planes * self.expansion) #NN.BatchNorm2d
self.relu = nn.ReLU(inplace=True)
self.downsample = downsample
self.stride = stride
def forward(self, x):
residual = x
out = self.conv1(x)
out = self.bn1(out)
out = self.relu(out)
out = self.conv2(out)
out = self.bn2(out)
out = self.relu(out)
out = self.conv3(out)
out = self.bn3(out)
if self.downsample is not None:
residual = self.downsample(x)
out += residual
out = self.relu(out)
return out
class ResNet(nn.Module):
def __init__(self, block, layers, num_classes=1000):
self.inplanes = 64
super(ResNet, self).__init__()
self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3,
bias=False)
self.bn1 = NN.BatchNorm2d(64) #NN.BatchNorm2d
self.relu = nn.ReLU(inplace=True)
self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
self.layer1 = self._make_layer(block, 64, layers[0])
self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
self.layer3 = self._make_layer(block, 256, layers[2], stride=2)
self.layer4 = self._make_layer(block, 512, layers[3], stride=2)
#self.avgpool = nn.AvgPool2d(7, stride=1)
#self.fc = nn.Linear(512 * block.expansion, num_classes)
for m in self.modules():
if isinstance(m, nn.Conv2d):
nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
elif isinstance(m, nn.BatchNorm2d):
nn.init.constant_(m.weight, 1)
nn.init.constant_(m.bias, 0)
def _make_layer(self, block, planes, blocks, stride=1):
downsample = None
if stride != 1 or self.inplanes != planes * block.expansion:
downsample = nn.Sequential(
nn.Conv2d(self.inplanes, planes * block.expansion,
kernel_size=1, stride=stride, bias=False),
NN.BatchNorm2d(planes * block.expansion), #NN.BatchNorm2d
)
layers = []
layers.append(block(self.inplanes, planes, stride, downsample))
self.inplanes = planes * block.expansion
for i in range(1, blocks):
layers.append(block(self.inplanes, planes))
return nn.Sequential(*layers)
def forward(self, x):
features = []
x = self.conv1(x)
x = self.bn1(x)
x = self.relu(x)
x = self.maxpool(x)
x = self.layer1(x)
features.append(x)
x = self.layer2(x)
features.append(x)
x = self.layer3(x)
features.append(x)
x = self.layer4(x)
features.append(x)
return features
def resnet18(pretrained=True, **kwargs):
"""Constructs a ResNet-18 model.
Args:
pretrained (bool): If True, returns a model pre-trained on ImageNet
"""
model = ResNet(BasicBlock, [2, 2, 2, 2], **kwargs)
return model
def resnet34(pretrained=True, **kwargs):
"""Constructs a ResNet-34 model.
Args:
pretrained (bool): If True, returns a model pre-trained on ImageNet
"""
model = ResNet(BasicBlock, [3, 4, 6, 3], **kwargs)
return model
def resnet50(pretrained=True, **kwargs):
"""Constructs a ResNet-50 model.
Args:
pretrained (bool): If True, returns a model pre-trained on ImageNet
"""
model = ResNet(Bottleneck, [3, 4, 6, 3], **kwargs)
return model
def resnet101(pretrained=True, **kwargs):
"""Constructs a ResNet-101 model.
Args:
pretrained (bool): If True, returns a model pre-trained on ImageNet
"""
model = ResNet(Bottleneck, [3, 4, 23, 3], **kwargs)
return model
def resnet152(pretrained=True, **kwargs):
"""Constructs a ResNet-152 model.
Args:
pretrained (bool): If True, returns a model pre-trained on ImageNet
"""
model = ResNet(Bottleneck, [3, 8, 36, 3], **kwargs)
return model
modules/controlnet/leres/leres/Resnext_torch.py
+237
View File
@@ -0,0 +1,237 @@
#!/usr/bin/env python
# coding: utf-8
import torch.nn as nn
try:
from urllib import urlretrieve
except ImportError:
from urllib.request import urlretrieve
__all__ = ['resnext101_32x8d']
model_urls = {
'resnext50_32x4d': 'https://download.pytorch.org/models/resnext50_32x4d-7cdf4587.pth',
'resnext101_32x8d': 'https://download.pytorch.org/models/resnext101_32x8d-8ba56ff5.pth',
}
def conv3x3(in_planes, out_planes, stride=1, groups=1, dilation=1):
"""3x3 convolution with padding"""
return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,
padding=dilation, groups=groups, bias=False, dilation=dilation)
def conv1x1(in_planes, out_planes, stride=1):
"""1x1 convolution"""
return nn.Conv2d(in_planes, out_planes, kernel_size=1, stride=stride, bias=False)
class BasicBlock(nn.Module):
expansion = 1
def __init__(self, inplanes, planes, stride=1, downsample=None, groups=1,
base_width=64, dilation=1, norm_layer=None):
super(BasicBlock, self).__init__()
if norm_layer is None:
norm_layer = nn.BatchNorm2d
if groups != 1 or base_width != 64:
raise ValueError('BasicBlock only supports groups=1 and base_width=64')
if dilation > 1:
raise NotImplementedError("Dilation > 1 not supported in BasicBlock")
# Both self.conv1 and self.downsample layers downsample the input when stride != 1
self.conv1 = conv3x3(inplanes, planes, stride)
self.bn1 = norm_layer(planes)
self.relu = nn.ReLU(inplace=True)
self.conv2 = conv3x3(planes, planes)
self.bn2 = norm_layer(planes)
self.downsample = downsample
self.stride = stride
def forward(self, x):
identity = x
out = self.conv1(x)
out = self.bn1(out)
out = self.relu(out)
out = self.conv2(out)
out = self.bn2(out)
if self.downsample is not None:
identity = self.downsample(x)
out += identity
out = self.relu(out)
return out
class Bottleneck(nn.Module):
# Bottleneck in torchvision places the stride for downsampling at 3x3 convolution(self.conv2)
# while original implementation places the stride at the first 1x1 convolution(self.conv1)
# according to "Deep residual learning for image recognition"https://arxiv.org/abs/1512.03385.
# This variant is also known as ResNet V1.5 and improves accuracy according to
# https://ngc.nvidia.com/catalog/model-scripts/nvidia:resnet_50_v1_5_for_pytorch.
expansion = 4
def __init__(self, inplanes, planes, stride=1, downsample=None, groups=1,
base_width=64, dilation=1, norm_layer=None):
super(Bottleneck, self).__init__()
if norm_layer is None:
norm_layer = nn.BatchNorm2d
width = int(planes * (base_width / 64.)) * groups
# Both self.conv2 and self.downsample layers downsample the input when stride != 1
self.conv1 = conv1x1(inplanes, width)
self.bn1 = norm_layer(width)
self.conv2 = conv3x3(width, width, stride, groups, dilation)
self.bn2 = norm_layer(width)
self.conv3 = conv1x1(width, planes * self.expansion)
self.bn3 = norm_layer(planes * self.expansion)
self.relu = nn.ReLU(inplace=True)
self.downsample = downsample
self.stride = stride
def forward(self, x):
identity = x
out = self.conv1(x)
out = self.bn1(out)
out = self.relu(out)
out = self.conv2(out)
out = self.bn2(out)
out = self.relu(out)
out = self.conv3(out)
out = self.bn3(out)
if self.downsample is not None:
identity = self.downsample(x)
out += identity
out = self.relu(out)
return out
class ResNet(nn.Module):
def __init__(self, block, layers, num_classes=1000, zero_init_residual=False,
groups=1, width_per_group=64, replace_stride_with_dilation=None,
norm_layer=None):
super(ResNet, self).__init__()
if norm_layer is None:
norm_layer = nn.BatchNorm2d
self._norm_layer = norm_layer
self.inplanes = 64
self.dilation = 1
if replace_stride_with_dilation is None:
# each element in the tuple indicates if we should replace
# the 2x2 stride with a dilated convolution instead
replace_stride_with_dilation = [False, False, False]
if len(replace_stride_with_dilation) != 3:
raise ValueError("replace_stride_with_dilation should be None "
"or a 3-element tuple, got {}".format(replace_stride_with_dilation))
self.groups = groups
self.base_width = width_per_group
self.conv1 = nn.Conv2d(3, self.inplanes, kernel_size=7, stride=2, padding=3,
bias=False)
self.bn1 = norm_layer(self.inplanes)
self.relu = nn.ReLU(inplace=True)
self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
self.layer1 = self._make_layer(block, 64, layers[0])
self.layer2 = self._make_layer(block, 128, layers[1], stride=2,
dilate=replace_stride_with_dilation[0])
self.layer3 = self._make_layer(block, 256, layers[2], stride=2,
dilate=replace_stride_with_dilation[1])
self.layer4 = self._make_layer(block, 512, layers[3], stride=2,
dilate=replace_stride_with_dilation[2])
#self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
#self.fc = nn.Linear(512 * block.expansion, num_classes)
for m in self.modules():
if isinstance(m, nn.Conv2d):
nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
elif isinstance(m, (nn.BatchNorm2d, nn.GroupNorm)):
nn.init.constant_(m.weight, 1)
nn.init.constant_(m.bias, 0)
# Zero-initialize the last BN in each residual branch,
# so that the residual branch starts with zeros, and each residual block behaves like an identity.
# This improves the model by 0.2~0.3% according to https://arxiv.org/abs/1706.02677
if zero_init_residual:
for m in self.modules():
if isinstance(m, Bottleneck):
nn.init.constant_(m.bn3.weight, 0)
elif isinstance(m, BasicBlock):
nn.init.constant_(m.bn2.weight, 0)
def _make_layer(self, block, planes, blocks, stride=1, dilate=False):
norm_layer = self._norm_layer
downsample = None
previous_dilation = self.dilation
if dilate:
self.dilation *= stride
stride = 1
if stride != 1 or self.inplanes != planes * block.expansion:
downsample = nn.Sequential(
conv1x1(self.inplanes, planes * block.expansion, stride),
norm_layer(planes * block.expansion),
)
layers = []
layers.append(block(self.inplanes, planes, stride, downsample, self.groups,
self.base_width, previous_dilation, norm_layer))
self.inplanes = planes * block.expansion
for _ in range(1, blocks):
layers.append(block(self.inplanes, planes, groups=self.groups,
base_width=self.base_width, dilation=self.dilation,
norm_layer=norm_layer))
return nn.Sequential(*layers)
def _forward_impl(self, x):
# See note [TorchScript super()]
features = []
x = self.conv1(x)
x = self.bn1(x)
x = self.relu(x)
x = self.maxpool(x)
x = self.layer1(x)
features.append(x)
x = self.layer2(x)
features.append(x)
x = self.layer3(x)
features.append(x)
x = self.layer4(x)
features.append(x)
#x = self.avgpool(x)
#x = torch.flatten(x, 1)
#x = self.fc(x)
return features
def forward(self, x):
return self._forward_impl(x)
def resnext101_32x8d(pretrained=True, **kwargs):
"""Constructs a ResNet-152 model.
Args:
pretrained (bool): If True, returns a model pre-trained on ImageNet
"""
kwargs['groups'] = 32
kwargs['width_per_group'] = 8
model = ResNet(Bottleneck, [3, 4, 23, 3], **kwargs)
return model
modules/controlnet/leres/leres/__init__.py
View File
modules/controlnet/leres/leres/depthmap.py
+349
View File
@@ -0,0 +1,349 @@
# Author: thygate
# https://github.com/thygate/stable-diffusion-webui-depthmap-script
import gc
from operator import getitem
import cv2
import numpy as np
import torch
from torchvision.transforms import transforms
whole_size_threshold = 1600 # R_max from the paper
pix2pixsize = 1024
def scale_torch(img):
"""
Scale the image and output it in torch.tensor.
:param img: input rgb is in shape [H, W, C], input depth/disp is in shape [H, W]
:param scale: the scale factor. float
:return: img. [C, H, W]
"""
if len(img.shape) == 2:
img = img[np.newaxis, :, :]
if img.shape[2] == 3:
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.485, 0.456, 0.406) , (0.229, 0.224, 0.225) )])
img = transform(img.astype(np.float32))
else:
img = img.astype(np.float32)
img = torch.from_numpy(img)
return img
def estimateleres(img, model, w, h):
device = next(iter(model.parameters())).device
# leres transform input
rgb_c = img[:, :, ::-1].copy()
A_resize = cv2.resize(rgb_c, (w, h))
img_torch = scale_torch(A_resize)[None, :, :, :]
# compute
with torch.no_grad():
img_torch = img_torch.to(device)
prediction = model.depth_model(img_torch)
prediction = prediction.squeeze().cpu().numpy()
prediction = cv2.resize(prediction, (img.shape[1], img.shape[0]), interpolation=cv2.INTER_CUBIC)
return prediction
def generatemask(size):
# Generates a Guassian mask
mask = np.zeros(size, dtype=np.float32)
sigma = int(size[0]/16)
k_size = int(2 * np.ceil(2 * int(size[0]/16)) + 1)
mask[int(0.15*size[0]):size[0] - int(0.15*size[0]), int(0.15*size[1]): size[1] - int(0.15*size[1])] = 1
mask = cv2.GaussianBlur(mask, (int(k_size), int(k_size)), sigma)
mask = (mask - mask.min()) / (mask.max() - mask.min())
mask = mask.astype(np.float32)
return mask
def resizewithpool(img, size):
i_size = img.shape[0]
n = int(np.floor(i_size/size))
out = block_reduce(img, (n, n), np.max)
return out
def rgb2gray(rgb):
# Converts rgb to gray
return np.dot(rgb[..., :3], [0.2989, 0.5870, 0.1140])
def calculateprocessingres(img, basesize, confidence=0.1, scale_threshold=3, whole_size_threshold=3000):
# Returns the R_x resolution described in section 5 of the main paper.
# Parameters:
# img :input rgb image
# basesize : size the dilation kernel which is equal to receptive field of the network.
# confidence: value of x in R_x; allowed percentage of pixels that are not getting any contextual cue.
# scale_threshold: maximum allowed upscaling on the input image ; it has been set to 3.
# whole_size_threshold: maximum allowed resolution. (R_max from section 6 of the main paper)
# Returns:
# outputsize_scale*speed_scale :The computed R_x resolution
# patch_scale: K parameter from section 6 of the paper
# speed scale parameter is to process every image in a smaller size to accelerate the R_x resolution search
speed_scale = 32
image_dim = int(min(img.shape[0:2]))
gray = rgb2gray(img)
grad = np.abs(cv2.Sobel(gray, cv2.CV_64F, 0, 1, ksize=3)) + np.abs(cv2.Sobel(gray, cv2.CV_64F, 1, 0, ksize=3))
grad = cv2.resize(grad, (image_dim, image_dim), cv2.INTER_AREA)
# thresholding the gradient map to generate the edge-map as a proxy of the contextual cues
m = grad.min()
M = grad.max()
middle = m + (0.4 * (M - m))
grad[grad < middle] = 0
grad[grad >= middle] = 1
# dilation kernel with size of the receptive field
kernel = np.ones((int(basesize/speed_scale), int(basesize/speed_scale)), float)
# dilation kernel with size of the a quarter of receptive field used to compute k
# as described in section 6 of main paper
kernel2 = np.ones((int(basesize / (4*speed_scale)), int(basesize / (4*speed_scale))), float)
# Output resolution limit set by the whole_size_threshold and scale_threshold.
threshold = min(whole_size_threshold, scale_threshold * max(img.shape[:2]))
outputsize_scale = basesize / speed_scale
for p_size in range(int(basesize/speed_scale), int(threshold/speed_scale), int(basesize / (2*speed_scale))):
grad_resized = resizewithpool(grad, p_size)
grad_resized = cv2.resize(grad_resized, (p_size, p_size), cv2.INTER_NEAREST)
grad_resized[grad_resized >= 0.5] = 1
grad_resized[grad_resized < 0.5] = 0
dilated = cv2.dilate(grad_resized, kernel, iterations=1)
meanvalue = (1-dilated).mean()
if meanvalue > confidence:
break
else:
outputsize_scale = p_size
grad_region = cv2.dilate(grad_resized, kernel2, iterations=1)
patch_scale = grad_region.mean()
return int(outputsize_scale*speed_scale), patch_scale
# Generate a double-input depth estimation
def doubleestimate(img, size1, size2, pix2pixsize, model, net_type, pix2pixmodel):
# Generate the low resolution estimation
estimate1 = singleestimate(img, size1, model, net_type)
# Resize to the inference size of merge network.
estimate1 = cv2.resize(estimate1, (pix2pixsize, pix2pixsize), interpolation=cv2.INTER_CUBIC)
# Generate the high resolution estimation
estimate2 = singleestimate(img, size2, model, net_type)
# Resize to the inference size of merge network.
estimate2 = cv2.resize(estimate2, (pix2pixsize, pix2pixsize), interpolation=cv2.INTER_CUBIC)
# Inference on the merge model
pix2pixmodel.set_input(estimate1, estimate2)
pix2pixmodel.test()
visuals = pix2pixmodel.get_current_visuals()
prediction_mapped = visuals['fake_B']
prediction_mapped = (prediction_mapped+1)/2
prediction_mapped = (prediction_mapped - torch.min(prediction_mapped)) / (
torch.max(prediction_mapped) - torch.min(prediction_mapped))
prediction_mapped = prediction_mapped.squeeze().cpu().numpy()
return prediction_mapped
# Generate a single-input depth estimation
def singleestimate(img, msize, model, net_type):
# if net_type == 0:
return estimateleres(img, model, msize, msize)
# else:
# return estimatemidasBoost(img, model, msize, msize)
def applyGridpatch(blsize, stride, img, box):
# Extract a simple grid patch.
counter1 = 0
patch_bound_list = {}
for k in range(blsize, img.shape[1] - blsize, stride):
for j in range(blsize, img.shape[0] - blsize, stride):
patch_bound_list[str(counter1)] = {}
patchbounds = [j - blsize, k - blsize, j - blsize + 2 * blsize, k - blsize + 2 * blsize]
patch_bound = [box[0] + patchbounds[1], box[1] + patchbounds[0], patchbounds[3] - patchbounds[1],
patchbounds[2] - patchbounds[0]]
patch_bound_list[str(counter1)]['rect'] = patch_bound
patch_bound_list[str(counter1)]['size'] = patch_bound[2]
counter1 = counter1 + 1
return patch_bound_list
def getGF_fromintegral(integralimage, rect):
# Computes the gradient density of a given patch from the gradient integral image.
x1 = rect[1]
x2 = rect[1]+rect[3]
y1 = rect[0]
y2 = rect[0]+rect[2]
value = integralimage[x2, y2]-integralimage[x1, y2]-integralimage[x2, y1]+integralimage[x1, y1]
return value
def impatch(image, rect):
# Extract the given patch pixels from a given image.
w1 = rect[0]
h1 = rect[1]
w2 = w1 + rect[2]
h2 = h1 + rect[3]
image_patch = image[h1:h2, w1:w2]
return image_patch
class ImageandPatchs:
def __init__(self, root_dir, name, patchsinfo, rgb_image, scale=1):
self.root_dir = root_dir
self.patchsinfo = patchsinfo
self.name = name
self.patchs = patchsinfo
self.scale = scale
self.rgb_image = cv2.resize(rgb_image, (round(rgb_image.shape[1]*scale), round(rgb_image.shape[0]*scale)),
interpolation=cv2.INTER_CUBIC)
self.do_have_estimate = False
self.estimation_updated_image = None
self.estimation_base_image = None
def __len__(self):
return len(self.patchs)
def set_base_estimate(self, est):
self.estimation_base_image = est
if self.estimation_updated_image is not None:
self.do_have_estimate = True
def set_updated_estimate(self, est):
self.estimation_updated_image = est
if self.estimation_base_image is not None:
self.do_have_estimate = True
def __getitem__(self, index):
patch_id = int(self.patchs[index][0])
rect = np.array(self.patchs[index][1]['rect'])
msize = self.patchs[index][1]['size']
## applying scale to rect:
rect = np.round(rect * self.scale)
rect = rect.astype('int')
msize = round(msize * self.scale)
patch_rgb = impatch(self.rgb_image, rect)
if self.do_have_estimate:
patch_whole_estimate_base = impatch(self.estimation_base_image, rect)
patch_whole_estimate_updated = impatch(self.estimation_updated_image, rect)
return {'patch_rgb': patch_rgb, 'patch_whole_estimate_base': patch_whole_estimate_base,
'patch_whole_estimate_updated': patch_whole_estimate_updated, 'rect': rect,
'size': msize, 'id': patch_id}
else:
return {'patch_rgb': patch_rgb, 'rect': rect, 'size': msize, 'id': patch_id}
def print_options(self, opt):
"""Print and save options
It will print both current options and default values(if different).
It will save options into a text file / [checkpoints_dir] / opt.txt
"""
message = ''
message += '----------------- Options ---------------\n'
for k, v in sorted(vars(opt).items()):
comment = ''
default = self.parser.get_default(k)
if v != default:
comment = '\t[default: %s]' % str(default)
message += '{:>25}: {:<30}{}\n'.format(str(k), str(v), comment)
message += '----------------- End -------------------'
print(message)
# save to the disk
"""
expr_dir = os.path.join(opt.checkpoints_dir, opt.name)
util.mkdirs(expr_dir)
file_name = os.path.join(expr_dir, '{}_opt.txt'.format(opt.phase))
with open(file_name, 'wt') as opt_file:
opt_file.write(message)
opt_file.write('\n')
"""
def parse(self):
"""Parse our options, create checkpoints directory suffix, and set up gpu device."""
opt = self.gather_options()
opt.isTrain = self.isTrain # train or test
# process opt.suffix
if opt.suffix:
suffix = ('_' + opt.suffix.format(**vars(opt))) if opt.suffix != '' else ''
opt.name = opt.name + suffix
#self.print_options(opt)
# set gpu ids
str_ids = opt.gpu_ids.split(',')
opt.gpu_ids = []
for str_id in str_ids:
id = int(str_id)
if id >= 0:
opt.gpu_ids.append(id)
#if len(opt.gpu_ids) > 0:
# torch.cuda.set_device(opt.gpu_ids[0])
self.opt = opt
return self.opt
# --------------------====================--------------------
from numpy.lib.stride_tricks import as_strided
def view_as_blocks(arr_in, block_shape):
if not isinstance(block_shape, tuple):
raise TypeError('block needs to be a tuple')
block_shape = np.array(block_shape)
if (block_shape <= 0).any():
raise ValueError("'block_shape' elements must be strictly positive")
if block_shape.size != arr_in.ndim:
raise ValueError("'block_shape' must have the same length "
"as 'arr_in.shape'")
arr_shape = np.array(arr_in.shape)
if (arr_shape % block_shape).sum() != 0:
raise ValueError("'block_shape' is not compatible with 'arr_in'")
# -- restride the array to build the block view
new_shape = tuple(arr_shape // block_shape) + tuple(block_shape)
new_strides = tuple(arr_in.strides * block_shape) + arr_in.strides
arr_out = as_strided(arr_in, shape=new_shape, strides=new_strides)
return arr_out
def block_reduce(image, block_size=2, func=np.sum, cval=0, func_kwargs=None):
if np.isscalar(block_size):
block_size = (block_size,) * image.ndim
elif len(block_size) != image.ndim:
raise ValueError("`block_size` must be a scalar or have "
"the same length as `image.shape`")
if func_kwargs is None:
func_kwargs = {}
pad_width = []
for i in range(len(block_size)):
if block_size[i] < 1:
raise ValueError("Down-sampling factors must be >= 1. Use "
"`skimage.transform.resize` to up-sample an "
"image.")
if image.shape[i] % block_size[i] != 0:
after_width = block_size[i] - (image.shape[i] % block_size[i])
else:
after_width = 0
pad_width.append((0, after_width))
image = np.pad(image, pad_width=pad_width, mode='constant',
constant_values=cval)
blocked = view_as_blocks(image, block_size)
return func(blocked, axis=tuple(range(image.ndim, blocked.ndim)), **func_kwargs)
modules/controlnet/leres/leres/multi_depth_model_woauxi.py
+40
View File
@@ -0,0 +1,40 @@
import torch
import torch.nn as nn
from . import network_auxi as network
from .net_tools import get_func
class RelDepthModel(nn.Module):
def __init__(self, backbone='resnet50'):
super(RelDepthModel, self).__init__()
if backbone == 'resnet50':
encoder = 'resnet50_stride32'
elif backbone == 'resnext101':
encoder = 'resnext101_stride32x8d'
self.depth_model = DepthModel(encoder)
def inference(self, rgb):
with torch.no_grad():
input = rgb.to(self.depth_model.device)
depth = self.depth_model(input)
#pred_depth_out = depth - depth.min() + 0.01
return depth #pred_depth_out
class DepthModel(nn.Module):
def __init__(self, encoder):
super(DepthModel, self).__init__()
if encoder == "resnet50_stride32":
self.encoder_modules = network.resnet50_stride32()
elif encoder == "resnext101_stride32x8d":
self.encoder_modules = network.resnext101_stride32x8d()
else:
backbone = network.__name__.split('.')[-1] + '.' + encoder
self.encoder_modules = get_func(backbone)()
self.decoder_modules = network.Decoder()
def forward(self, x):
lateral_out = self.encoder_modules(x)
out_logit = self.decoder_modules(lateral_out)
return out_logit
modules/controlnet/leres/leres/net_tools.py
+54
View File
@@ -0,0 +1,54 @@
import importlib
import torch
import os
from collections import OrderedDict
def get_func(func_name):
"""Helper to return a function object by name. func_name must identify a
function in this module or the path to a function relative to the base
'modeling' module.
"""
if func_name == '':
return None
try:
parts = func_name.split('.')
# Refers to a function in this module
if len(parts) == 1:
return globals()[parts[0]]
# Otherwise, assume we're referencing a module under modeling
module_name = 'controlnet_aux.leres.leres.' + '.'.join(parts[:-1])
module = importlib.import_module(module_name)
return getattr(module, parts[-1])
except Exception:
print('Failed to f1ind function: %s', func_name)
raise
def load_ckpt(args, depth_model, shift_model, focal_model):
"""
Load checkpoint.
"""
if os.path.isfile(args.load_ckpt):
print("loading checkpoint %s" % args.load_ckpt)
checkpoint = torch.load(args.load_ckpt)
if shift_model is not None:
shift_model.load_state_dict(strip_prefix_if_present(checkpoint['shift_model'], 'module.'),
strict=True)
if focal_model is not None:
focal_model.load_state_dict(strip_prefix_if_present(checkpoint['focal_model'], 'module.'),
strict=True)
depth_model.load_state_dict(strip_prefix_if_present(checkpoint['depth_model'], "module."),
strict=True)
del checkpoint
if torch.cuda.is_available():
torch.cuda.empty_cache()
def strip_prefix_if_present(state_dict, prefix):
keys = sorted(state_dict.keys())
if not all(key.startswith(prefix) for key in keys):
return state_dict
stripped_state_dict = OrderedDict()
for key, value in state_dict.items():
stripped_state_dict[key.replace(prefix, "")] = value
return stripped_state_dict
modules/controlnet/leres/leres/network_auxi.py
+417
View File
@@ -0,0 +1,417 @@
import torch
import torch.nn as nn
import torch.nn.init as init
from . import Resnet, Resnext_torch
def resnet50_stride32():
return DepthNet(backbone='resnet', depth=50, upfactors=[2, 2, 2, 2])
def resnext101_stride32x8d():
return DepthNet(backbone='resnext101_32x8d', depth=101, upfactors=[2, 2, 2, 2])
class Decoder(nn.Module):
def __init__(self):
super(Decoder, self).__init__()
self.inchannels = [256, 512, 1024, 2048]
self.midchannels = [256, 256, 256, 512]
self.upfactors = [2,2,2,2]
self.outchannels = 1
self.conv = FTB(inchannels=self.inchannels[3], midchannels=self.midchannels[3])
self.conv1 = nn.Conv2d(in_channels=self.midchannels[3], out_channels=self.midchannels[2], kernel_size=3, padding=1, stride=1, bias=True)
self.upsample = nn.Upsample(scale_factor=self.upfactors[3], mode='bilinear', align_corners=True)
self.ffm2 = FFM(inchannels=self.inchannels[2], midchannels=self.midchannels[2], outchannels = self.midchannels[2], upfactor=self.upfactors[2])
self.ffm1 = FFM(inchannels=self.inchannels[1], midchannels=self.midchannels[1], outchannels = self.midchannels[1], upfactor=self.upfactors[1])
self.ffm0 = FFM(inchannels=self.inchannels[0], midchannels=self.midchannels[0], outchannels = self.midchannels[0], upfactor=self.upfactors[0])
self.outconv = AO(inchannels=self.midchannels[0], outchannels=self.outchannels, upfactor=2)
self._init_params()
def _init_params(self):
for m in self.modules():
if isinstance(m, nn.Conv2d):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.ConvTranspose2d):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.BatchNorm2d): #NN.BatchNorm2d
init.constant_(m.weight, 1)
init.constant_(m.bias, 0)
elif isinstance(m, nn.Linear):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
def forward(self, features):
x_32x = self.conv(features[3]) # 1/32
x_32 = self.conv1(x_32x)
x_16 = self.upsample(x_32) # 1/16
x_8 = self.ffm2(features[2], x_16) # 1/8
x_4 = self.ffm1(features[1], x_8) # 1/4
x_2 = self.ffm0(features[0], x_4) # 1/2
#-----------------------------------------
x = self.outconv(x_2) # original size
return x
class DepthNet(nn.Module):
__factory = {
18: Resnet.resnet18,
34: Resnet.resnet34,
50: Resnet.resnet50,
101: Resnet.resnet101,
152: Resnet.resnet152
}
def __init__(self,
backbone='resnet',
depth=50,
upfactors=[2, 2, 2, 2]):
super(DepthNet, self).__init__()
self.backbone = backbone
self.depth = depth
self.pretrained = False
self.inchannels = [256, 512, 1024, 2048]
self.midchannels = [256, 256, 256, 512]
self.upfactors = upfactors
self.outchannels = 1
# Build model
if self.backbone == 'resnet':
if self.depth not in DepthNet.__factory:
raise KeyError("Unsupported depth:", self.depth)
self.encoder = DepthNet.__factory[depth](pretrained=self.pretrained)
elif self.backbone == 'resnext101_32x8d':
self.encoder = Resnext_torch.resnext101_32x8d(pretrained=self.pretrained)
else:
self.encoder = Resnext_torch.resnext101(pretrained=self.pretrained)
def forward(self, x):
x = self.encoder(x) # 1/32, 1/16, 1/8, 1/4
return x
class FTB(nn.Module):
def __init__(self, inchannels, midchannels=512):
super(FTB, self).__init__()
self.in1 = inchannels
self.mid = midchannels
self.conv1 = nn.Conv2d(in_channels=self.in1, out_channels=self.mid, kernel_size=3, padding=1, stride=1,
bias=True)
# NN.BatchNorm2d
self.conv_branch = nn.Sequential(nn.ReLU(inplace=True), \
nn.Conv2d(in_channels=self.mid, out_channels=self.mid, kernel_size=3,
padding=1, stride=1, bias=True), \
nn.BatchNorm2d(num_features=self.mid), \
nn.ReLU(inplace=True), \
nn.Conv2d(in_channels=self.mid, out_channels=self.mid, kernel_size=3,
padding=1, stride=1, bias=True))
self.relu = nn.ReLU(inplace=True)
self.init_params()
def forward(self, x):
x = self.conv1(x)
x = x + self.conv_branch(x)
x = self.relu(x)
return x
def init_params(self):
for m in self.modules():
if isinstance(m, nn.Conv2d):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.ConvTranspose2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.BatchNorm2d): # NN.BatchNorm2d
init.constant_(m.weight, 1)
init.constant_(m.bias, 0)
elif isinstance(m, nn.Linear):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
class ATA(nn.Module):
def __init__(self, inchannels, reduction=8):
super(ATA, self).__init__()
self.inchannels = inchannels
self.avg_pool = nn.AdaptiveAvgPool2d(1)
self.fc = nn.Sequential(nn.Linear(self.inchannels * 2, self.inchannels // reduction),
nn.ReLU(inplace=True),
nn.Linear(self.inchannels // reduction, self.inchannels),
nn.Sigmoid())
self.init_params()
def forward(self, low_x, high_x):
n, c, _, _ = low_x.size()
x = torch.cat([low_x, high_x], 1)
x = self.avg_pool(x)
x = x.view(n, -1)
x = self.fc(x).view(n, c, 1, 1)
x = low_x * x + high_x
return x
def init_params(self):
for m in self.modules():
if isinstance(m, nn.Conv2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
# init.normal(m.weight, std=0.01)
init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.ConvTranspose2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
# init.normal_(m.weight, std=0.01)
init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.BatchNorm2d): # NN.BatchNorm2d
init.constant_(m.weight, 1)
init.constant_(m.bias, 0)
elif isinstance(m, nn.Linear):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
class FFM(nn.Module):
def __init__(self, inchannels, midchannels, outchannels, upfactor=2):
super(FFM, self).__init__()
self.inchannels = inchannels
self.midchannels = midchannels
self.outchannels = outchannels
self.upfactor = upfactor
self.ftb1 = FTB(inchannels=self.inchannels, midchannels=self.midchannels)
# self.ata = ATA(inchannels = self.midchannels)
self.ftb2 = FTB(inchannels=self.midchannels, midchannels=self.outchannels)
self.upsample = nn.Upsample(scale_factor=self.upfactor, mode='bilinear', align_corners=True)
self.init_params()
def forward(self, low_x, high_x):
x = self.ftb1(low_x)
x = x + high_x
x = self.ftb2(x)
x = self.upsample(x)
return x
def init_params(self):
for m in self.modules():
if isinstance(m, nn.Conv2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.ConvTranspose2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.BatchNorm2d): # NN.Batchnorm2d
init.constant_(m.weight, 1)
init.constant_(m.bias, 0)
elif isinstance(m, nn.Linear):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
class AO(nn.Module):
# Adaptive output module
def __init__(self, inchannels, outchannels, upfactor=2):
super(AO, self).__init__()
self.inchannels = inchannels
self.outchannels = outchannels
self.upfactor = upfactor
self.adapt_conv = nn.Sequential(
nn.Conv2d(in_channels=self.inchannels, out_channels=self.inchannels // 2, kernel_size=3, padding=1,
stride=1, bias=True), \
nn.BatchNorm2d(num_features=self.inchannels // 2), \
nn.ReLU(inplace=True), \
nn.Conv2d(in_channels=self.inchannels // 2, out_channels=self.outchannels, kernel_size=3, padding=1,
stride=1, bias=True), \
nn.Upsample(scale_factor=self.upfactor, mode='bilinear', align_corners=True))
self.init_params()
def forward(self, x):
x = self.adapt_conv(x)
return x
def init_params(self):
for m in self.modules():
if isinstance(m, nn.Conv2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.ConvTranspose2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.BatchNorm2d): # NN.Batchnorm2d
init.constant_(m.weight, 1)
init.constant_(m.bias, 0)
elif isinstance(m, nn.Linear):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
# ==============================================================================================================
class ResidualConv(nn.Module):
def __init__(self, inchannels):
super(ResidualConv, self).__init__()
# NN.BatchNorm2d
self.conv = nn.Sequential(
# nn.BatchNorm2d(num_features=inchannels),
nn.ReLU(inplace=False),
# nn.Conv2d(in_channels=inchannels, out_channels=inchannels, kernel_size=3, padding=1, stride=1, groups=inchannels,bias=True),
# nn.Conv2d(in_channels=inchannels, out_channels=inchannels, kernel_size=1, padding=0, stride=1, groups=1,bias=True)
nn.Conv2d(in_channels=inchannels, out_channels=inchannels / 2, kernel_size=3, padding=1, stride=1,
bias=False),
nn.BatchNorm2d(num_features=inchannels / 2),
nn.ReLU(inplace=False),
nn.Conv2d(in_channels=inchannels / 2, out_channels=inchannels, kernel_size=3, padding=1, stride=1,
bias=False)
)
self.init_params()
def forward(self, x):
x = self.conv(x) + x
return x
def init_params(self):
for m in self.modules():
if isinstance(m, nn.Conv2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.ConvTranspose2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.BatchNorm2d): # NN.BatchNorm2d
init.constant_(m.weight, 1)
init.constant_(m.bias, 0)
elif isinstance(m, nn.Linear):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
class FeatureFusion(nn.Module):
def __init__(self, inchannels, outchannels):
super(FeatureFusion, self).__init__()
self.conv = ResidualConv(inchannels=inchannels)
# NN.BatchNorm2d
self.up = nn.Sequential(ResidualConv(inchannels=inchannels),
nn.ConvTranspose2d(in_channels=inchannels, out_channels=outchannels, kernel_size=3,
stride=2, padding=1, output_padding=1),
nn.BatchNorm2d(num_features=outchannels),
nn.ReLU(inplace=True))
def forward(self, lowfeat, highfeat):
return self.up(highfeat + self.conv(lowfeat))
def init_params(self):
for m in self.modules():
if isinstance(m, nn.Conv2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.ConvTranspose2d):
# init.kaiming_normal_(m.weight, mode='fan_out')
init.normal_(m.weight, std=0.01)
# init.xavier_normal_(m.weight)
if m.bias is not None:
init.constant_(m.bias, 0)
elif isinstance(m, nn.BatchNorm2d): # NN.BatchNorm2d
init.constant_(m.weight, 1)
init.constant_(m.bias, 0)
elif isinstance(m, nn.Linear):
init.normal_(m.weight, std=0.01)
if m.bias is not None:
init.constant_(m.bias, 0)
class SenceUnderstand(nn.Module):
def __init__(self, channels):
super(SenceUnderstand, self).__init__()
self.channels = channels
self.conv1 = nn.Sequential(nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, padding=1),
nn.ReLU(inplace=True))
self.pool = nn.AdaptiveAvgPool2d(8)
self.fc = nn.Sequential(nn.Linear(512 * 8 * 8, self.channels),
nn.ReLU(inplace=True))
self.conv2 = nn.Sequential(
nn.Conv2d(in_channels=self.channels, out_channels=self.channels, kernel_size=1, padding=0),
nn.ReLU(inplace=True))
self.initial_params()
def forward(self, x):
n, c, h, w = x.size()
x = self.conv1(x)
x = self.pool(x)
x = x.view(n, -1)
x = self.fc(x)
x = x.view(n, self.channels, 1, 1)
x = self.conv2(x)
x = x.repeat(1, 1, h, w)
return x
def initial_params(self, dev=0.01):
for m in self.modules():
if isinstance(m, nn.Conv2d):
# print torch.sum(m.weight)
m.weight.data.normal_(0, dev)
if m.bias is not None:
m.bias.data.fill_(0)
elif isinstance(m, nn.ConvTranspose2d):
# print torch.sum(m.weight)
m.weight.data.normal_(0, dev)
if m.bias is not None:
m.bias.data.fill_(0)
elif isinstance(m, nn.Linear):
m.weight.data.normal_(0, dev)
if __name__ == '__main__':
net = DepthNet(depth=50, pretrained=True)
print(net)
inputs = torch.ones(4,3,128,128)
out = net(inputs)
print(out.size())
modules/controlnet/utils.py
+83
View File
@@ -0,0 +1,83 @@
import torch
import numpy as np
import cv2
def annotator_wrapper(tensor_images, preprocessor_lambda):
out_list = []
for image in tensor_images:
H, W, C = image.shape
np_image = np.asarray(image * 255., dtype=np.uint8)
np_result = preprocessor_lambda(np_image)
np_result = cv2.resize(np_result, (W, H), interpolation=cv2.INTER_AREA)
out_list.append(torch.from_numpy(np_result.astype(np.float32) / 255.0))
return torch.stack(out_list, dim=0)
def HWC3(x):
assert x.dtype == np.uint8
if x.ndim == 2:
x = x[:, :, None]
assert x.ndim == 3
H, W, C = x.shape
assert C == 1 or C == 3 or C == 4
if C == 3:
return x
if C == 1:
return np.concatenate([x, x, x], axis=2)
if C == 4:
color = x[:, :, 0:3].astype(np.float32)
alpha = x[:, :, 3:4].astype(np.float32) / 255.0
y = color * alpha + 255.0 * (1.0 - alpha)
y = y.clip(0, 255).astype(np.uint8)
return y
def nms(x, t, s):
x = cv2.GaussianBlur(x.astype(np.float32), (0, 0), s)
f1 = np.array([[0, 0, 0], [1, 1, 1], [0, 0, 0]], dtype=np.uint8)
f2 = np.array([[0, 1, 0], [0, 1, 0], [0, 1, 0]], dtype=np.uint8)
f3 = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]], dtype=np.uint8)
f4 = np.array([[0, 0, 1], [0, 1, 0], [1, 0, 0]], dtype=np.uint8)
y = np.zeros_like(x)
for f in [f1, f2, f3, f4]:
np.putmask(y, cv2.dilate(x, kernel=f) == x, x)
z = np.zeros_like(y, dtype=np.uint8)
z[y > t] = 255
return z
def safe_step(x, step=2):
y = x.astype(np.float32) * float(step + 1)
y = y.astype(np.int32).astype(np.float32) / float(step)
return y
def resize_image(input_image, resolution):
H, W, C = input_image.shape
H = float(H)
W = float(W)
k = float(resolution) / min(H, W)
H *= k
W *= k
H = int(np.round(H / 64.0)) * 64
W = int(np.round(W / 64.0)) * 64
return cv2.resize(input_image, (W, H), interpolation=cv2.INTER_LANCZOS4 if k > 1 else cv2.INTER_AREA)
modules/controlnet_adapter.py
+165
View File
@@ -0,0 +1,165 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from folder_paths import get_full_path
from .controlnet import canny
from .controlnet import leres
from .controlnet import hed
from .data_utils import retrieve_parameter
from .ui import UI
# ====================================================================================================
# Adapter for image inputs
# ====================================================================================================
class SeargeControlnetAdapterV2:
def __init__(self):
self.expected_size = None
self.hed_annotator = "ControlNetHED.pth"
self.leres_annotator = "res101.pth"
self.hed_annotator_full_path = get_full_path("annotators", self.hed_annotator)
self.leres_annotator_full_path = get_full_path("annotators", self.leres_annotator)
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"controlnet_mode": (UI.CONTROLNET_MODES, {"default": UI.NONE},),
"controlnet_preprocessor": ("BOOLEAN", {"default": False},),
"strength": ("FLOAT", {"default": 0.5, "min": 0.0, "max": 1.0, "step": 0.05},),
"low_threshold": ("FLOAT", {"default": 0.25, "min": 0.0, "max": 1.0, "step": 0.05},),
"high_threshold": ("FLOAT", {"default": 0.5, "min": 0.0, "max": 1.0, "step": 0.05},),
"start_percent": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.05},),
"end_percent": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.05},),
"noise_augmentation": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.05},),
"revision_enhancer": ("BOOLEAN", {"default": False},),
},
"optional": {
"data": ("SRG_DATA_STREAM",),
"source_image": ("IMAGE",),
},
}
RETURN_TYPES = ("SRG_DATA_STREAM", "IMAGE",)
RETURN_NAMES = ("data", "preview",)
FUNCTION = "get_value"
CATEGORY = UI.CATEGORY_UI_PROMPTING
def process_image(self, image, mode, low_threshold, high_threshold):
if mode == UI.CN_MODE_CANNY:
image = canny(image, low_threshold, high_threshold)
elif mode == UI.CN_MODE_DEPTH:
image = leres(image, low_threshold, high_threshold, self.leres_annotator_full_path)
elif mode == UI.CN_MODE_SKETCH:
image = hed(image, self.hed_annotator_full_path)
else:
# do nothing for any other mode, just use the provided image unchanged
pass
return image
def create_dict(self, stack, source_image, controlnet_mode, controlnet_preprocessor, strength,
low_threshold, high_threshold, start, end, noise_augmentation, revision_enhancer):
if controlnet_mode is None or controlnet_mode == UI.NONE:
cn_image = None
else:
cn_image = source_image
low_threshold = round(low_threshold, 3)
high_threshold = round(high_threshold, 3)
# NOTE: for the modes "revision" and "custom" no image pre-processing is needed
if controlnet_mode == UI.CN_MODE_REVISION or controlnet_mode == UI.CUSTOM:
controlnet_preprocessor = False
if controlnet_preprocessor and cn_image is not None:
cn_image = self.process_image(cn_image, controlnet_mode, low_threshold, high_threshold)
stack += [
{
UI.F_REV_CN_IMAGE: cn_image,
UI.F_REV_CN_IMAGE_CHANGED: True,
UI.F_REV_CN_MODE: controlnet_mode,
UI.F_CN_PRE_PROCESSOR: controlnet_preprocessor,
UI.F_REV_CN_STRENGTH: round(strength, 3),
UI.F_CN_LOW_THRESHOLD: low_threshold,
UI.F_CN_HIGH_THRESHOLD: high_threshold,
UI.F_CN_START: round(start, 3),
UI.F_CN_END: round(end, 3),
UI.F_REV_NOISE_AUGMENTATION: round(noise_augmentation, 3),
UI.F_REV_ENHANCER: revision_enhancer,
}
]
return (
{
UI.F_CN_STACK: stack,
},
cn_image,
)
def get_value(self, controlnet_mode, controlnet_preprocessor, strength, low_threshold, high_threshold,
start_percent, end_percent, noise_augmentation, revision_enhancer, source_image=None, data=None):
if data is None:
data = {}
stack = retrieve_parameter(UI.F_CN_STACK, retrieve_parameter(UI.S_CONTROLNET_INPUTS, data), [])
if self.expected_size is None:
self.expected_size = len(stack)
elif self.expected_size == 0:
stack = []
elif len(stack) > self.expected_size:
stack = stack[:self.expected_size]
(stack_entry, image) = self.create_dict(
stack,
source_image,
controlnet_mode,
controlnet_preprocessor,
strength,
low_threshold,
high_threshold,
start_percent,
end_percent,
noise_augmentation,
revision_enhancer,
)
data[UI.S_CONTROLNET_INPUTS] = stack_entry
return (data, image,)
modules/custom_sdxl_ksampler.py
+314
View File
@@ -0,0 +1,314 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
import torch
import warnings
import comfy.model_management
import comfy.sample
import comfy.samplers
import comfy.utils
import latent_preview
from .utils import slerp_latents
# --------------------------------------------------------------------------------
class CfgMethods:
INTERPOLATE = "interpolate"
RESCALE = "rescale"
TONEMAP = "tonemap"
# --------------------------------------------------------------------------------
def sdxl_sample(base_model, refiner_model, noise, base_steps, refiner_steps, cfg, sampler_name, scheduler,
base_positive, base_negative, refiner_positive, refiner_negative, latent_image, batch_inds,
denoise=1.0, start_step=None, last_step=None, force_full_denoise=False, noise_mask=None, sigmas=None,
base_callback=None, refiner_callback=None, disable_pbar=False, seed=None, cfg_method=None,
dynamic_base_cfg=0.0, dynamic_refiner_cfg=0.0, refiner_detail_boost=0.0):
device = comfy.model_management.get_torch_device()
if noise_mask is not None:
noise_mask = comfy.sample.prepare_mask(noise_mask, noise.shape, device)
steps = base_steps + refiner_steps
def base_cfg_callback(args):
(cond, uncond, cond_scale, timestep) = (args["cond"], args["uncond"], args["cond_scale"], args["timestep"])
dyn_cfg = dynamic_base_cfg
if dyn_cfg < 0.0:
dyn_cfg = -dyn_cfg
ts = 1.0 - float(timestep) / 999.0
else:
ts = float(timestep) / 999.0
if dyn_cfg > 0.0999:
cond_scale = cond_scale * ts + (cond_scale * (1.0 - dyn_cfg) + dyn_cfg) * (1.0 - ts)
return uncond + (cond - uncond) * cond_scale
def base_rescale_cfg(args):
multiplier = dynamic_base_cfg if dynamic_base_cfg >= 0.0 else -dynamic_base_cfg
cond = args["cond"]
uncond = args["uncond"]
cond_scale = args["cond_scale"]
x_cfg = uncond + cond_scale * (cond - uncond)
ro_pos = torch.std(cond, dim=(1, 2, 3), keepdim=True)
ro_cfg = torch.std(x_cfg, dim=(1, 2, 3), keepdim=True)
x_rescaled = x_cfg * (ro_pos / ro_cfg)
x_final = multiplier * x_rescaled + (1.0 - multiplier) * x_cfg
return x_final
def base_tonemap_reinhard(args):
multiplier = dynamic_base_cfg if dynamic_base_cfg >= 0.0 else -dynamic_base_cfg
cond = args["cond"]
uncond = args["uncond"]
cond_scale = args["cond_scale"]
noise_pred = (cond - uncond)
noise_pred_vector_magnitude = (torch.linalg.vector_norm(noise_pred, dim=(1)) + 0.0000000001)[:, None]
noise_pred /= noise_pred_vector_magnitude
mean = torch.mean(noise_pred_vector_magnitude, dim=(1, 2, 3), keepdim=True)
std = torch.std(noise_pred_vector_magnitude, dim=(1, 2, 3), keepdim=True)
top = (std * 3 + mean) * multiplier
noise_pred_vector_magnitude *= (1.0 / top)
new_magnitude = noise_pred_vector_magnitude / (noise_pred_vector_magnitude + 1.0)
new_magnitude *= top
return uncond + noise_pred * new_magnitude * cond_scale
if cfg_method is not None:
base_model = base_model.clone()
if cfg_method == CfgMethods.INTERPOLATE:
base_model.set_model_sampler_cfg_function(base_cfg_callback)
elif cfg_method == CfgMethods.RESCALE and dynamic_base_cfg > 0.0:
base_model.set_model_sampler_cfg_function(base_rescale_cfg)
elif cfg_method == CfgMethods.TONEMAP and dynamic_base_cfg > 0.0:
base_model.set_model_sampler_cfg_function(base_tonemap_reinhard)
base_models = comfy.sample.get_additional_models(base_positive, base_negative)
comfy.model_management.load_models_gpu([base_model] + base_models, comfy.model_management.batch_area_memory(noise.shape[0] * noise.shape[2] * noise.shape[3]))
real_base_model = base_model.model
original_latent = latent_image
noise = noise.to(device)
latent_image = latent_image.to(device)
pos_base_copy = comfy.sample.broadcast_cond(base_positive, noise.shape[0], device)
neg_base_copy = comfy.sample.broadcast_cond(base_negative, noise.shape[0], device)
base_sampler = comfy.samplers.KSampler(real_base_model, steps=steps, device=device, sampler=sampler_name,
scheduler=scheduler, denoise=denoise, model_options=base_model.model_options)
base_samples = base_sampler.sample(noise, pos_base_copy, neg_base_copy, cfg=cfg, latent_image=latent_image,
start_step=start_step, last_step=base_steps, force_full_denoise=False,
denoise_mask=noise_mask, sigmas=sigmas, callback=base_callback,
disable_pbar=disable_pbar, seed=seed)
comfy.sample.cleanup_additional_models(base_models)
noise = torch.zeros(base_samples.size(), dtype=base_samples.dtype, layout=base_samples.layout, device=device)
if refiner_steps < 1:
return base_samples
if refiner_detail_boost > 0.0:
new_noise = comfy.sample.prepare_noise(original_latent, seed + 1, batch_inds).to(device)
new_noise /= real_base_model.latent_format.scale_factor
factor = base_sampler.sigmas[-refiner_steps - 1]
new_noise = new_noise * factor
noised_samples = base_samples + new_noise
base_samples = slerp_latents(base_samples, noised_samples, refiner_detail_boost)
if noise_mask is not None:
latent_from_base = base_samples * noise_mask + latent_image * (1.0 - noise_mask)
else:
latent_from_base = base_samples
def refiner_cfg_callback(args):
(cond, uncond, cond_scale, timestep) = (args["cond"], args["uncond"], args["cond_scale"], args["timestep"])
dyn_cfg = dynamic_refiner_cfg
if dyn_cfg < 0.0:
dyn_cfg = -dyn_cfg
ts = 1.0 - float(timestep) / 999.0
else:
ts = float(timestep) / 999.0
if dyn_cfg > 0.0999:
cond_scale = cond_scale * ts + (cond_scale * (1.0 - dyn_cfg) + dyn_cfg) * (1.0 - ts)
return uncond + (cond - uncond) * cond_scale
def refiner_rescale_cfg(args):
multiplier = dynamic_refiner_cfg if dynamic_refiner_cfg >= 0.0 else -dynamic_refiner_cfg
cond = args["cond"]
uncond = args["uncond"]
cond_scale = args["cond_scale"]
x_cfg = uncond + cond_scale * (cond - uncond)
ro_pos = torch.std(cond, dim=(1, 2, 3), keepdim=True)
ro_cfg = torch.std(x_cfg, dim=(1, 2, 3), keepdim=True)
x_rescaled = x_cfg * (ro_pos / ro_cfg)
x_final = multiplier * x_rescaled + (1.0 - multiplier) * x_cfg
return x_final
def refiner_tonemap_reinhard(args):
multiplier = dynamic_refiner_cfg if dynamic_refiner_cfg >= 0.0 else -dynamic_refiner_cfg
cond = args["cond"]
uncond = args["uncond"]
cond_scale = args["cond_scale"]
noise_pred = (cond - uncond)
noise_pred_vector_magnitude = (torch.linalg.vector_norm(noise_pred, dim=(1)) + 0.0000000001)[:, None]
noise_pred /= noise_pred_vector_magnitude
mean = torch.mean(noise_pred_vector_magnitude, dim=(1, 2, 3), keepdim=True)
std = torch.std(noise_pred_vector_magnitude, dim=(1, 2, 3), keepdim=True)
top = (std * 3 + mean) * multiplier
noise_pred_vector_magnitude *= (1.0 / top)
new_magnitude = noise_pred_vector_magnitude / (noise_pred_vector_magnitude + 1.0)
new_magnitude *= top
return uncond + noise_pred * new_magnitude * cond_scale
if cfg_method is not None:
refiner_model = refiner_model.clone()
if cfg_method == CfgMethods.INTERPOLATE:
refiner_model.set_model_sampler_cfg_function(refiner_cfg_callback)
elif cfg_method == CfgMethods.RESCALE and dynamic_refiner_cfg > 0.0:
refiner_model.set_model_sampler_cfg_function(refiner_rescale_cfg)
elif cfg_method == CfgMethods.TONEMAP and dynamic_refiner_cfg > 0.0:
refiner_model.set_model_sampler_cfg_function(refiner_tonemap_reinhard)
refiner_models = comfy.sample.get_additional_models(refiner_positive, refiner_negative)
comfy.model_management.load_models_gpu([refiner_model] + refiner_models, comfy.model_management.batch_area_memory(noise.shape[0] * noise.shape[2] * noise.shape[3]))
real_refiner_model = refiner_model.model
pos_refiner_copy = comfy.sample.broadcast_cond(refiner_positive, noise.shape[0], device)
neg_refiner_copy = comfy.sample.broadcast_cond(refiner_negative, noise.shape[0], device)
refiner_sampler = comfy.samplers.KSampler(real_refiner_model, steps=steps, device=device, sampler=sampler_name,
scheduler=scheduler, denoise=denoise, model_options=refiner_model.model_options)
refiner_samples = refiner_sampler.sample(noise, pos_refiner_copy, neg_refiner_copy, cfg=cfg, latent_image=latent_from_base,
start_step=base_steps, last_step=last_step, force_full_denoise=force_full_denoise,
denoise_mask=noise_mask, sigmas=sigmas, callback=refiner_callback,
disable_pbar=disable_pbar, seed=seed)
refiner_samples = refiner_samples.cpu()
comfy.sample.cleanup_additional_models(refiner_models)
return refiner_samples
# --------------------------------------------------------------------------------
def sdxl_ksampler(base_model, refiner_model, seed, base_steps, refiner_steps, cfg, sampler_name, scheduler,
base_positive, base_negative, refiner_positive, refiner_negative, latent, denoise=1.0,
disable_noise=False, start_step=None, last_step=None, force_full_denoise=False, cfg_method=None,
dynamic_base_cfg=0.0, dynamic_refiner_cfg=0.0, refiner_detail_boost=0.0):
device = comfy.model_management.get_torch_device()
latent_image = latent["samples"]
batch_inds = None
if disable_noise:
noise = torch.zeros(latent_image.size(), dtype=latent_image.dtype, layout=latent_image.layout, device="cpu")
else:
batch_inds = latent["batch_index"] if "batch_index" in latent else None
noise = comfy.sample.prepare_noise(latent_image, seed, batch_inds)
noise_mask = None
if "noise_mask" in latent:
noise_mask = latent["noise_mask"]
preview_format = "JPEG"
if preview_format not in ["JPEG", "PNG"]:
preview_format = "JPEG"
base_previewer = latent_preview.get_previewer(device, base_model.model.latent_format)
refiner_previewer = None
if refiner_model is not None:
refiner_previewer = latent_preview.get_previewer(device, refiner_model.model.latent_format)
steps = base_steps + refiner_steps
pbar = comfy.utils.ProgressBar(steps)
def base_callback(step, x0, x, total_steps):
preview_bytes = None
if base_previewer:
preview_bytes = base_previewer.decode_latent_to_preview_image(preview_format, x0)
pbar.update_absolute(step + 1, total_steps, preview_bytes)
def refiner_callback(step, x0, x, total_steps):
preview_bytes = None
if refiner_previewer:
preview_bytes = refiner_previewer.decode_latent_to_preview_image(preview_format, x0)
pbar.update_absolute(step + 1, total_steps, preview_bytes)
with warnings.catch_warnings():
warnings.simplefilter("ignore")
samples = sdxl_sample(base_model, refiner_model, noise, base_steps, refiner_steps, cfg, sampler_name, scheduler,
base_positive, base_negative, refiner_positive, refiner_negative, latent_image,
batch_inds, denoise=denoise, start_step=start_step, last_step=last_step,
force_full_denoise=force_full_denoise, noise_mask=noise_mask,
base_callback=base_callback, refiner_callback=refiner_callback, seed=seed,
dynamic_base_cfg=dynamic_base_cfg, dynamic_refiner_cfg=dynamic_refiner_cfg,
cfg_method=cfg_method, refiner_detail_boost=refiner_detail_boost)
out = latent.copy()
out["samples"] = samples
return (out,)
modules/data_utils.py
+53
View File
@@ -0,0 +1,53 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
# --------------------------------------------------------------------------------
def retrieve_input(name, data, stage_input):
if stage_input is not None and name in stage_input:
result = stage_input[name]
elif data is not None and name in data:
result = data[name]
else:
result = None
return result
# --------------------------------------------------------------------------------
def retrieve_parameter(name, structure, default=None):
if structure is None:
result = default
elif name in structure and structure[name] is not None:
result = structure[name]
else:
result = default
return result
modules/debug_printer.py
+126
View File
@@ -0,0 +1,126 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .names import Names
from .ui import UI
# ====================================================================================================
# Print state of a data stream
# ====================================================================================================
class SeargeDebugPrinter:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"enabled": ("BOOLEAN", {"default": True},),
},
"optional": {
"data": ("SRG_DATA_STREAM",),
"prefix": ("STRING", {"multiline": False, "default": ""},),
},
}
RETURN_TYPES = ("SRG_DATA_STREAM",)
RETURN_NAMES = ("data",)
FUNCTION = "output"
OUTPUT_NODE = True
CATEGORY = UI.CATEGORY_DEBUG
def output(self, enabled, data=None, prefix=None):
if data is None or not enabled:
return (data,)
prefix = "" if prefix is None or len(prefix) < 1 else prefix + ": "
indent_spaces = "· "
test_data = False
if test_data:
data["test_dict"] = {"k1": 1.0, "k2": 2, "k3": True}
data["test_list"] = ["l1", 2.0, 3]
data["test_tuple"] = (1, "t2", 3.0)
def print_dict(coll, ind=0, kp='"', pk=True):
spaces = indent_spaces * ind
for (k, v) in coll.items():
print_val(k, v, ind, kp, pk)
def print_coll(coll, ind=0, kp='', pk=False):
spaces = indent_spaces * ind
cl = len(coll)
for i in range(0, cl):
v = coll[i]
print_val(i, v, ind, kp, pk)
def print_val(k, v, ind=0, kp='"', pk=True):
spaces = indent_spaces * ind
key = kp + str(k) + kp + ': ' if pk else ''
if ind > 10:
print(prefix + spaces + key + '<max recursion depth>')
return
if v is None:
print(prefix + spaces + key + 'None,')
elif isinstance(v, int) or isinstance(v, float):
print(prefix + spaces + key + str(v) + ',')
elif isinstance(v, str):
print(prefix + spaces + key + '"' + v + '",')
elif isinstance(v, dict):
# dirty hack: we don't need to print the whole workflow and prompt
if k != Names.MAGIC_BOX_HIDDEN:
print(prefix + spaces + key + '{')
print_dict(v, ind + 1, '"', True)
print(prefix + spaces + '},')
else:
print(prefix + spaces + key + '{ ... printing skipped ... }')
elif isinstance(v, list):
print(prefix + spaces + key + '[')
print_coll(v, ind + 1, '', True)
print(prefix + spaces + '],')
elif isinstance(v, tuple):
print(prefix + spaces + key + '(')
print_coll(v, ind + 1, '', False)
print(prefix + spaces + '),')
else:
print(prefix + spaces + key + str(type(v)))
print(prefix + "===============================================================================")
if not isinstance(data, dict):
print(prefix + " ! invalid data stream !")
else:
print(prefix + "* DATA STREAM *")
print(prefix + "---------------")
print_val("data", data)
print(prefix + "===============================================================================")
return (data,)
modules/image_adapter.py
+77
View File
@@ -0,0 +1,77 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .ui import UI
# ====================================================================================================
# Adapter for image inputs
# ====================================================================================================
class SeargeImageAdapterV2:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
},
"optional": {
"data": ("SRG_DATA_STREAM",),
"source_image": ("IMAGE",),
"image_mask": ("MASK",),
"uploaded_mask": ("MASK",),
},
}
RETURN_TYPES = ("SRG_DATA_STREAM", "SRG_DATA_STREAM",)
RETURN_NAMES = ("data", UI.S_IMAGE_INPUTS,)
FUNCTION = "get_value"
CATEGORY = UI.CATEGORY_UI_PROMPTING
@staticmethod
def create_dict(source_image, image_mask, uploaded_mask):
return {
UI.F_SOURCE_IMAGE_CHANGED: True,
UI.F_SOURCE_IMAGE: source_image,
UI.F_IMAGE_MASK_CHANGED: True,
UI.F_IMAGE_MASK: image_mask,
UI.F_UPLOADED_MASK_CHANGED: True,
UI.F_UPLOADED_MASK: uploaded_mask,
}
def get_value(self, source_image=None, image_mask=None, uploaded_mask=None, data=None):
if data is None:
data = {}
data[UI.S_IMAGE_INPUTS] = self.create_dict(
source_image,
image_mask,
uploaded_mask,
)
return (data, data[UI.S_IMAGE_INPUTS],)
modules/legacy.py
-162
View File
@@ -1,162 +0,0 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
import comfy.samplers
import comfy_extras.nodes_post_processing
import comfy_extras.nodes_upscale_model
import nodes
# SDXL Sampler with base and refiner support
class SeargeSDXLSampler:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"base_model": ("MODEL",),
"base_positive": ("CONDITIONING", ),
"base_negative": ("CONDITIONING", ),
"refiner_model": ("MODEL",),
"refiner_positive": ("CONDITIONING", ),
"refiner_negative": ("CONDITIONING", ),
"latent_image": ("LATENT", ),
"noise_seed": ("INT", {"default": 0, "min": 0, "max": 0xffffffffffffffff}),
"steps": ("INT", {"default": 30, "min": 1, "max": 1000}),
"cfg": ("FLOAT", {"default": 7.0, "min": 0.0, "max": 100.0, "step": 0.5}),
"sampler_name": (comfy.samplers.KSampler.SAMPLERS, {"default": "dpmpp_2m"}),
"scheduler": (comfy.samplers.KSampler.SCHEDULERS, {"default": "karras"}),
"base_ratio": ("FLOAT", {"default": 0.8, "min": 0.0, "max": 1.0, "step": 0.01}),
"denoise": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.01}),
},
}
RETURN_TYPES = ("LATENT", )
RETURN_NAMES = ("", )
FUNCTION = "sample"
CATEGORY = "Searge/Legacy"
def sample(self, base_model, base_positive, base_negative, refiner_model, refiner_positive, refiner_negative, latent_image, noise_seed, steps, cfg, sampler_name, scheduler, base_ratio, denoise):
base_steps = int(steps * base_ratio)
if denoise < 0.01:
return (latent_image, )
if base_steps >= steps:
return nodes.common_ksampler(base_model, noise_seed, steps, cfg, sampler_name, scheduler, base_positive, base_negative, latent_image, denoise=denoise, disable_noise=False, start_step=0, last_step=steps, force_full_denoise=True)
base_result = nodes.common_ksampler(base_model, noise_seed, steps, cfg, sampler_name, scheduler, base_positive, base_negative, latent_image, denoise=denoise, disable_noise=False, start_step=0, last_step=base_steps, force_full_denoise=False)
return nodes.common_ksampler(refiner_model, noise_seed, steps, cfg, sampler_name, scheduler, refiner_positive, refiner_negative, base_result[0], denoise=1.0, disable_noise=True, start_step=base_steps, last_step=steps, force_full_denoise=True)
# SDXL Image2Image Sampler (incl. HiRes Fix)
class SeargeSDXLImage2ImageSampler:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"base_model": ("MODEL",),
"base_positive": ("CONDITIONING", ),
"base_negative": ("CONDITIONING", ),
"refiner_model": ("MODEL",),
"refiner_positive": ("CONDITIONING",),
"refiner_negative": ("CONDITIONING",),
"image": ("IMAGE", ),
"vae": ("VAE",),
"noise_seed": ("INT", {"default": 0, "min": 0, "max": 0xfffffffffffffff0}),
"steps": ("INT", {"default": 20, "min": 0, "max": 200}),
"cfg": ("FLOAT", {"default": 7.0, "min": 0.0, "max": 30.0, "step": 0.5}),
"sampler_name": ("SAMPLER_NAME", {"default": "ddim"}),
"scheduler": ("SCHEDULER_NAME", {"default": "ddim_uniform"}),
"base_ratio": ("FLOAT", {"default": 0.8, "min": 0.0, "max": 1.0, "step": 0.01}),
"denoise": ("FLOAT", {"default": 0.25, "min": 0.0, "max": 1.0, "step": 0.01}),
},
"optional": {
"upscale_model": ("UPSCALE_MODEL",),
"scaled_width": ("INT", {"default": 1536, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"scaled_height": ("INT", {"default": 1536, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"noise_offset": ("INT", {"default": 1, "min": 0, "max": 1}),
"refiner_strength": ("FLOAT", {"default": 1.0, "min": 0.1, "max": 1.0, "step": 0.05}),
"softness": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.05}),
},
}
RETURN_TYPES = ("IMAGE", )
FUNCTION = "sample"
CATEGORY = "Searge/Legacy"
def sample(self, base_model, base_positive, base_negative, refiner_model, refiner_positive, refiner_negative, image, vae, noise_seed, steps, cfg, sampler_name, scheduler, base_ratio, denoise, softness, upscale_model=None, scaled_width=None, scaled_height=None, noise_offset=None, refiner_strength=None):
base_steps = int(steps * (base_ratio + 0.0001))
if noise_offset is None:
noise_offset = 1
if refiner_strength is None:
refiner_strength = 1.0
if refiner_strength < 0.01:
refiner_strength = 0.01
if steps < 1:
return (image, )
scaled_image = image
use_upscale_model = upscale_model is not None and softness < 0.9999
if use_upscale_model:
upscale_result = comfy_extras.nodes_upscale_model.ImageUpscaleWithModel().upscale(upscale_model, image)
scaled_image = upscale_result[0]
if scaled_width is not None and scaled_height is not None:
upscale_result = nodes.ImageScale().upscale(scaled_image, "bicubic", scaled_width, scaled_height, "center")
scaled_image = upscale_result[0]
if use_upscale_model and softness > 0.0001:
upscale_result = nodes.ImageScale().upscale(image, "bicubic", scaled_width, scaled_height, "center")
scaled_original = upscale_result[0]
blend_result = comfy_extras.nodes_post_processing.Blend().blend_images(scaled_image, scaled_original, softness, "normal")
scaled_image = blend_result[0]
if denoise < 0.01:
return (scaled_image, )
vae_encode_result = nodes.VAEEncode().encode(vae, scaled_image)
input_latent = vae_encode_result[0]
if base_steps >= steps:
result_latent = nodes.common_ksampler(base_model, noise_seed, steps, cfg, sampler_name, scheduler, base_positive, base_negative, input_latent, denoise=denoise, disable_noise=False, start_step=0, last_step=steps, force_full_denoise=True)
else:
base_result = nodes.common_ksampler(base_model, noise_seed, steps, cfg, sampler_name, scheduler, base_positive, base_negative, input_latent, denoise=denoise, disable_noise=False, start_step=0, last_step=base_steps, force_full_denoise=True)
result_latent = nodes.common_ksampler(refiner_model, noise_seed + noise_offset, steps, cfg, sampler_name, scheduler, refiner_positive, refiner_negative, base_result[0], denoise=denoise * refiner_strength, disable_noise=False, start_step=base_steps, last_step=steps, force_full_denoise=True)
vae_decode_result = nodes.VAEDecode().decode(vae, result_latent[0])
output_image = vae_decode_result[0]
return (output_image, )
modules/magic_box.py
+268
View File
@@ -0,0 +1,268 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .mb_pipeline import PipelineAccess
from .stage import SeargeStage
from .stage_pre_processing import SeargePreProcessData
from .stage_load_checkpoints import SeargeStageLoadCheckpoints
from .stage_apply_loras import SeargeStageApplyLoras
from .stage_clip_conditioning import SeargeStageClipConditioning
from .stage_apply_controlnet import SeargeStageApplyControlnet
from .stage_latent_inputs import SeargeStageLatentInputs
from .stage_sampling import SeargeStageSampling
from .stage_latent_detailer import SeargeStageLatentDetailer
from .stage_vae_decode_sampled import SeargeStageVAEDecodeSampled
from .stage_high_resolution import SeargeStageHighResolution
from .stage_vae_decode_hires import SeargeStageVAEDecodeHires
from .stage_upscaling import SeargeStageUpscaling
from .stage_image_saving import SeargeStageImageSaving
from .ui import UI
# ====================================================================================================
# UI: Model Selector Input
# ====================================================================================================
class SeargeMagicBox:
# processing stages supported by the magic box
NONE = "none - skip"
PRE_PROCESS_DATA = "pre-process data"
LOAD_CHECKPOINTS = "load checkpoints"
APPLY_LORAS = "apply loras"
PROMPT_STYLING = "prompt styling"
CLIP_CONDITIONING = "clip conditioning"
CLIP_MIXING = "clip mixing"
APPLY_CONTROLNET = "apply controlnet"
LATENT_INPUTS = "latent inputs"
SAMPLING = "sampling"
LATENT_DETAILER = "latent detailer"
VAE_DECODE_SAMPLED = "vae decode sampled"
HIGH_RESOLUTION = "high resolution"
VAE_DECODE_HI_RES = "vae decode hi-res"
UPSCALING = "upscaling"
IMAGE_SAVING = "image saving"
STAGES = [
NONE,
PRE_PROCESS_DATA,
LOAD_CHECKPOINTS,
APPLY_LORAS,
PROMPT_STYLING,
CLIP_CONDITIONING,
CLIP_MIXING,
APPLY_CONTROLNET,
LATENT_INPUTS,
SAMPLING,
LATENT_DETAILER,
VAE_DECODE_SAMPLED,
HIGH_RESOLUTION,
VAE_DECODE_HI_RES,
UPSCALING,
IMAGE_SAVING,
]
# option to take inputs from a custom stage instead of the data stream
DATA = "data stream"
CUSTOM_AND_DATA = "custom stage & data stream"
INPUT_OUTPUT = [
DATA,
CUSTOM_AND_DATA,
]
def __init__(self):
self.stage_pre_process_data = None
self.stage_load_checkpoints = None
self.stage_apply_loras = None
self.stage_prompt_styling = None
self.stage_clip_conditioning = None
self.stage_clip_mixing = None
self.stage_latent_inputs = None
self.stage_apply_controlnet = None
self.stage_sampling = None
self.stage_vae_decode_sampled = None
self.stage_latent_detailer = None
self.stage_high_resolution = None
self.stage_vae_decode_hi_res = None
self.stage_upscaling = None
self.stage_image_saving = None
self.stage_ = None
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"stage": (s.STAGES, {"default": s.NONE},),
"input_from": (s.INPUT_OUTPUT,),
"output_to": (s.INPUT_OUTPUT,),
},
"optional": {
"data": ("SRG_DATA_STREAM",),
"custom_input": ("SRG_STAGE_INPUT",),
},
}
RETURN_TYPES = ("SRG_DATA_STREAM", "SRG_STAGE_OUTPUT",)
RETURN_NAMES = ("data", "custom_output",)
FUNCTION = "process"
CATEGORY = UI.CATEGORY_MAGIC
def run_stage(self, stage, data, stage_input=None):
stage_processor = None
has_data = data is not None
# clear old stage output from data stream
stage_output = stage_input
if has_data:
data["stage_output"] = None
# stage: "none - skip" - does nothing, skip processing in this magic box
if stage == self.NONE:
pass
elif stage == self.PRE_PROCESS_DATA:
if self.stage_pre_process_data is None:
self.stage_pre_process_data = SeargePreProcessData()
stage_processor = self.stage_pre_process_data
elif stage == self.LOAD_CHECKPOINTS:
if self.stage_load_checkpoints is None:
self.stage_load_checkpoints = SeargeStageLoadCheckpoints()
stage_processor = self.stage_load_checkpoints
elif stage == self.APPLY_LORAS:
if self.stage_apply_loras is None:
self.stage_apply_loras = SeargeStageApplyLoras()
stage_processor = self.stage_apply_loras
elif stage == self.PROMPT_STYLING:
if self.stage_prompt_styling is None:
print("TODO: implement stage " + stage)
self.stage_prompt_styling = SeargeStage()
stage_processor = self.stage_prompt_styling
elif stage == self.CLIP_CONDITIONING:
if self.stage_clip_conditioning is None:
self.stage_clip_conditioning = SeargeStageClipConditioning()
stage_processor = self.stage_clip_conditioning
elif stage == self.CLIP_MIXING:
if self.stage_clip_mixing is None:
print("TODO: implement stage " + stage)
self.stage_clip_mixing = SeargeStage()
stage_processor = self.stage_clip_mixing
elif stage == self.APPLY_CONTROLNET:
if self.stage_apply_controlnet is None:
self.stage_apply_controlnet = SeargeStageApplyControlnet()
stage_processor = self.stage_apply_controlnet
elif stage == self.LATENT_INPUTS:
if self.stage_latent_inputs is None:
self.stage_latent_inputs = SeargeStageLatentInputs()
stage_processor = self.stage_latent_inputs
elif stage == self.SAMPLING:
if self.stage_sampling is None:
self.stage_sampling = SeargeStageSampling()
stage_processor = self.stage_sampling
elif stage == self.LATENT_DETAILER:
if self.stage_latent_detailer is None:
self.stage_latent_detailer = SeargeStageLatentDetailer()
stage_processor = self.stage_latent_detailer
elif stage == self.VAE_DECODE_SAMPLED:
if self.stage_vae_decode_sampled is None:
self.stage_vae_decode_sampled = SeargeStageVAEDecodeSampled()
stage_processor = self.stage_vae_decode_sampled
elif stage == self.HIGH_RESOLUTION:
if self.stage_high_resolution is None:
self.stage_high_resolution = SeargeStageHighResolution()
stage_processor = self.stage_high_resolution
elif stage == self.VAE_DECODE_HI_RES:
if self.stage_vae_decode_hi_res is None:
self.stage_vae_decode_hi_res = SeargeStageVAEDecodeHires()
stage_processor = self.stage_vae_decode_hi_res
elif stage == self.UPSCALING:
if self.stage_upscaling is None:
self.stage_upscaling = SeargeStageUpscaling()
stage_processor = self.stage_upscaling
elif stage == self.IMAGE_SAVING:
if self.stage_image_saving is None:
self.stage_image_saving = SeargeStageImageSaving()
stage_processor = self.stage_image_saving
else:
print("WARNING: implementation for stage " + stage + " is missing!")
# no stage processor exists, so no processing can happen and no result exists
if stage_processor is None:
return (data, None,)
# get the stage input data that is relevant to the selected stage
stage_input = stage_processor.get_input(data, stage_output)
# process the selected stage
stage_result = None
if stage_input is not None:
(data, stage_result) = stage_processor.process(data, stage_input)
# if we got a result from this stage, put it on the data stream
if has_data:
data["stage_output"] = stage_result
return (data, stage_result,)
def process(self, stage, input_from, output_to, data=None, custom_input=None):
if data is None:
data = {}
stage_input = None
custom_output = None
# input from custom stage ?
if input_from == self.CUSTOM_AND_DATA:
stage_input = custom_input
# if no stage data is provided, the stage will take it from the data stream
if PipelineAccess(data).is_pipeline_enabled(): # or stage == self.LOAD_CHECKPOINTS:
(data, stage_result) = self.run_stage(stage, data, stage_input)
else:
stage_result = None
# output to custom stage ?
if output_to == self.CUSTOM_AND_DATA:
custom_output = stage_result
# the result will always be on the data stream, so even without custom output it will be passed on
return (data, custom_output,)
modules/mb_pipeline.py
+282
View File
@@ -0,0 +1,282 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .data_utils import retrieve_parameter
from .names import Names
from .ui import UI
# ====================================================================================================
# Pipeline
# ====================================================================================================
class Pipeline:
def __init__(self):
self.data = None
self.old_settings = {}
self.new_settings = {}
self.old_overrides = {}
self.new_overrides = {}
self.cache = {}
self.pipeline = {}
self.enabled = True
def enable(self, enabled=True):
if self.data is not None:
self.data[Names.B_MAGIC_BOX_ENABLED] = enabled
self.enabled = enabled
def start(self, data):
self.data = data
if data is None:
print("Warning: no data stream for pipeline at start")
return
if self.enabled:
self.old_settings = self.new_settings
self.old_overrides = self.new_overrides
for k in self.cache.keys():
self.cache[k]["changed"] = False
self.new_settings = {}
for k in UI.ALL_UI_INPUTS:
self.new_settings[k] = retrieve_parameter(k, data)
self.new_overrides = {}
self.pipeline = {
"old_settings": self.old_settings,
"new_settings": self.new_settings,
"old_overrides": self.old_overrides,
"new_overrides": self.new_overrides,
"cache": self.cache,
"stream": {}
}
if data is not None:
data[PipelineAccess.NAME] = self.pipeline
# ====================================================================================================
# Pipeline Access
# ====================================================================================================
class PipelineAccess:
NAME = "pipeline"
def __init__(self, data):
self.data = data
self.pipeline = data[PipelineAccess.NAME] if PipelineAccess.NAME in data else None
if self.pipeline is None:
print("Warning: pipeline access could not find data")
def terminate_pipeline(self):
if self.data is None:
print("Warning: no data stream for pipeline to terminate")
return
if self.pipeline is None:
print("Warning: no pipeline to terminate in data stream")
return
self.pipeline["stream"] = {}
def is_pipeline_enabled(self):
return retrieve_parameter(Names.B_MAGIC_BOX_ENABLED, self.data, True)
# -----===== settings =====-----
def has_structure(self, name):
if self.pipeline is None:
return False
if name in retrieve_parameter("new_overrides", self.pipeline, {}):
return True
if name in retrieve_parameter("new_settings", self.pipeline, {}):
return True
return False
def get_effective_structure(self, name):
if not self.has_structure(name):
return None
new_settings = retrieve_parameter("new_settings", self.pipeline)
new_overrides = retrieve_parameter("new_overrides", self.pipeline, {})
new_structure = retrieve_parameter(name, new_settings)
new_structure_overrides = retrieve_parameter(name, new_overrides, {})
return new_structure | new_structure_overrides if new_structure is not None else new_structure_overrides
def override_setting(self, structure_name, field_name, value):
if self.pipeline is None:
return False
new_overrides = retrieve_parameter("new_overrides", self.pipeline, {})
if structure_name in new_overrides:
new_overrides[structure_name][field_name] = value
return True
new_overrides[structure_name] = {
field_name: value,
}
self.pipeline["new_overrides"] = new_overrides
return True
def get_active_setting(self, structure_name, field_name, default=None):
structure = retrieve_parameter(structure_name, retrieve_parameter("new_overrides", self.pipeline), {})
if field_name not in structure:
structure = retrieve_parameter(structure_name, retrieve_parameter("new_settings", self.pipeline))
return retrieve_parameter(field_name, structure, default)
def get_old_setting(self, structure_name, field_name):
structure = retrieve_parameter(structure_name, retrieve_parameter("old_overrides", self.pipeline, {}), {})
if field_name not in structure:
structure = retrieve_parameter(structure_name, retrieve_parameter("old_settings", self.pipeline))
return retrieve_parameter(field_name, structure)
def has_setting(self, structure_name, field_name):
return self.get_active_setting(structure_name, field_name) is not None
def setting_changed(self, structure_name, field_name):
old_value = self.get_old_setting(structure_name, field_name)
new_value = self.get_active_setting(structure_name, field_name)
return old_value != new_value
# -----===== pipeline stream =====-----
def update_in_pipeline(self, name, value):
if self.pipeline is None or "stream" not in self.pipeline:
return False
if value is None:
return False
self.pipeline["stream"][name] = {
"changed": True,
"data": value,
}
return True
def restore_in_pipeline(self, name, value):
if self.pipeline is None or "stream" not in self.pipeline:
return False
if value is None:
return False
self.pipeline["stream"][name] = {
"changed": False,
"data": value,
}
return True
def has_in_pipeline(self, name):
if self.pipeline is not None and "stream" in self.pipeline:
cache = self.pipeline["stream"]
if name in cache:
return True
return False
def get_from_pipeline(self, name):
if self.has_in_pipeline(name):
cache = self.pipeline["stream"]
cached = cache[name]
if "data" in cached:
return cached["data"]
return None
def changed_in_pipeline(self, name):
if self.has_in_pipeline(name):
cache = self.pipeline["stream"]
cached = cache[name]
return "changed" in cached and cached["changed"]
return False
# -----===== cache =====-----
def update_in_cache(self, name, key, value):
if self.pipeline is None or "cache" not in self.pipeline:
return False
self.pipeline["cache"][name] = {
"key": key,
"data": value,
}
return True
def has_in_cache(self, name):
if self.pipeline is not None and "cache" in self.pipeline:
cache = self.pipeline["cache"]
if name in cache:
return True
return False
def get_from_cache(self, name):
if self.has_in_cache(name):
cache = self.pipeline["cache"]
cached = cache[name]
if "data" in cached:
return cached["data"]
return None
def remove_from_cache(self, name):
if self.has_in_cache(name):
cache = self.pipeline["cache"]
cache.pop(name)
return True
return False
def changed_in_cache(self, name, key):
if self.has_in_cache(name):
cache = self.pipeline["cache"]
cached = cache[name]
return "key" not in cached or cached["key"] != key
return True
modules/mb_pipeline_start.py
+98
View File
@@ -0,0 +1,98 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .mb_pipeline import Pipeline
from .mb_pipeline import PipelineAccess
from .names import Names
from .ui import Defs
from .ui import UI
# ====================================================================================================
# Magic Box Pipeline Terminator
# ====================================================================================================
class SeargePipelineStart:
def __init__(self):
self.pipeline = Pipeline()
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"wf_version": (Defs.WORKFLOW_VERSIONS,),
},
"optional": {
"data": ("SRG_DATA_STREAM",),
"additional_data": ("SRG_DATA_STREAM",),
},
"hidden": {
"prompt": "PROMPT",
"extra_pnginfo": "EXTRA_PNGINFO",
},
}
RETURN_TYPES = ("SRG_DATA_STREAM",)
RETURN_NAMES = ("data",)
FUNCTION = "trigger"
OUTPUT_NODE = True
CATEGORY = UI.CATEGORY_MAGIC
def trigger(self, wf_version, data=None, additional_data=None, prompt=None, extra_pnginfo=None):
if data is None:
print("Warning: Pipeline Start - missing data stream")
else:
if additional_data is not None:
data = data | additional_data
self.pipeline.start(data)
access = PipelineAccess(data)
self.pipeline.enable(access.get_active_setting(UI.S_OPERATING_MODE, UI.F_WORKFLOW_MODE) != UI.NONE)
mb_hidden = {
Names.F_MAGIC_BOX_PROMPT: prompt,
Names.F_MAGIC_BOX_EXTRA_PNGINFO: extra_pnginfo,
}
mb_version = {
Names.F_MAGIC_BOX_EXTENSION: Defs.VERSION,
Names.F_MAGIC_BOX_WORKFLOW: wf_version,
}
access.update_in_pipeline(Names.S_MAGIC_BOX_HIDDEN, mb_hidden)
access.update_in_pipeline(Names.S_MAGIC_BOX_VERSION, mb_version)
if data is not None:
data[Names.S_MAGIC_BOX_HIDDEN] = mb_hidden
data[Names.S_MAGIC_BOX_VERSION] = mb_version
return (data,)
modules/mb_pipeline_terminator.py
+58
View File
@@ -0,0 +1,58 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .mb_pipeline import PipelineAccess
from .ui import UI
# ====================================================================================================
# Magic Box Pipeline Terminator
# ====================================================================================================
class SeargePipelineTerminator:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
},
"optional": {
"data": ("SRG_DATA_STREAM",),
},
}
RETURN_TYPES = ()
FUNCTION = "trigger"
OUTPUT_NODE = True
CATEGORY = UI.CATEGORY_MAGIC
def trigger(self, data=None):
access = PipelineAccess(data)
access.terminate_pipeline()
return {}
modules/names.py
+246
View File
@@ -0,0 +1,246 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
# ====================================================================================================
# Names to be used for data streams and the structures & fields in them
# ====================================================================================================
class Names:
PLACEHOLDER = "placeholder"
S_EXAMPLE_STRUCTURE = "example_structure"
F_EXAMPLE_FIELD = "example_field"
# ----------------------------------------
# structures and fields
# ----------------------------------------
# magic box
B_MAGIC_BOX_ENABLED = "magic_box_enabled"
# pre-processor
S_MAGIC_BOX_HIDDEN = "hidden_fields"
F_MAGIC_BOX_PROMPT = "prompt"
F_MAGIC_BOX_EXTRA_PNGINFO = "pnginfo"
S_MAGIC_BOX_VERSION = "version_info"
F_MAGIC_BOX_EXTENSION = "extension_version"
F_MAGIC_BOX_WORKFLOW = "workflow_version"
# vae decoder stage outputs
S_VAE_DECODED = "vae_decoded"
F_DECODED_IMAGE = "image"
F_POST_PROCESSED = "post_processed"
# checkpoint loader
S_LOADED_MODELS = "loaded_models"
F_BASE_MODEL = "base_model"
F_BASE_CLIP = "base_clip"
F_BASE_VAE = "base_vae"
F_REFINER_MODEL = "refiner_model"
F_REFINER_CLIP = "refiner_clip"
F_REFINER_VAE = "refiner_vae"
F_VAE_MODEL = "vae_model"
F_HIRES_UPSCALER = "hires_upscaler"
F_PRIMARY_UPSCALER = "primary_upscaler"
F_SECONDARY_UPSCALER = "secondary_upscaler"
F_DETAIL_PROCESSOR = "detail_processor"
F_CLIP_VISION_MODEL = "clip_vision_model"
F_CN_CANNY_MODEL = "cn_canny_model"
F_CN_DEPTH_MODEL = "cn_depth_model"
F_CN_RECOLOR_MODEL = "cn_recolor_model"
F_CN_SKETCH_MODEL = "cn_sketch_model"
F_CN_CUSTOM_MODEL = "cn_custom_model"
# apply loras
S_LOADED_LORAS = "loaded_loras"
F_LORA_NAMES = "lora_names"
# clip conditioning
S_PROCESSED_PROMPTS = "processed_prompts"
F_BASE_POSITIVE_MAIN_PROMPT = "base_positive_main_prompt"
F_BASE_POSITIVE_SECONDARY_PROMPT = "base_positive_secondary_prompt"
F_BASE_POSITIVE_STYLE_PROMPT = "base_positive_style_prompt"
F_BASE_NEGATIVE_MAIN_PROMPT = "base_negative_main_prompt"
F_BASE_NEGATIVE_SECONDARY_PROMPT = "base_negative_secondary_prompt"
F_BASE_NEGATIVE_STYLE_PROMPT = "base_negative_style_prompt"
F_REFINER_POSITIVE_PROMPT = "refiner_positive_prompt"
F_REFINER_POSITIVE_STYLE_PROMPT = "refiner_positive_style_prompt"
F_REFINER_NEGATIVE_PROMPT = "refiner_negative_prompt"
F_REFINER_NEGATIVE_STYLE_PROMPT = "refiner_negative_style_prompt"
S_CONDITIONING = "conditioning"
F_BASE_POSITIVE = "base_positive"
F_BASE_POSITIVE_STYLE = "base_positive_style"
F_BASE_NEGATIVE = "base_negative"
F_BASE_NEGATIVE_STYLE = "base_negative_style"
F_REFINER_POSITIVE = "refiner_positive"
F_REFINER_POSITIVE_STYLE = "refiner_positive_style"
F_REFINER_NEGATIVE = "refiner_negative"
F_REFINER_NEGATIVE_STYLE = "refiner_negative_style"
# apply controlnet
S_CONTROLNET_OUTPUT = "controlnet_output"
F_CN_BASE_POSITIVE = "cn_base_positive"
F_CN_BASE_NEGATIVE = "cn_base_negative"
# latent inputs
S_LATENT_INPUTS = "latent_inputs"
F_LATENT_IMAGE = "latent_image"
# sampler
S_SAMPLED_IMAGE = "sampled_image"
F_LATENT_SAMPLED = "latent_sampled"
# latent detailer
S_LATENT_DETAILED = "latent_detailed"
F_DETAILED_SAMPLED = "detailed_sampled"
# vae decode sampled
S_VAE_DECODED_SAMPLED = "vae_decoded_sampled"
F_DECODED_SAMPLED_IMAGE = "sampled_image"
F_SAMPLED_POST_PROCESSED = "sampled_post_processed"
# high resolution
S_HIRES_OUTPUT = "hires_output"
F_LATENT_HIRES = "latent_hires"
# vae decode hires
S_VAE_DECODED_HIRES = "vae_decoded_hires"
F_DECODED_HIRES_IMAGE = "hires_image"
F_HIRES_POST_PROCESSED = "hires_post_processed"
# upscaling
S_UPSCALED = "upscaled"
F_UPSCALED_IMAGE = "upscaled_image"
# image saving
S_SAVED_FILES = "saved_files"
F_GENERATED_IMAGE_PATH = "generated_image_path"
F_HIGH_RES_IMAGE_PATH = "high_res_image_path"
F_UPSCALED_IMAGE_PATH = "upscaled_image_path"
F_PARAMETER_FILE_PATH = "parameter_file_path"
# ----------------------------------------
# cache names
# ----------------------------------------
# pre-processor
C_SOURCE_IMAGE = "source_image"
C_IMAGE_SIZE = "image_size"
C_SOURCE_MASK = "source_mask"
C_BLURRY_MASK = "blurry_mask"
# checkpoint loader
C_BASE_CHECKPOINT = "base_checkpoint"
C_REFINER_CHECKPOINT = "refiner_checkpoint"
C_VAE_CHECKPOINT = "vae_checkpoint"
C_HIRES_UPSCALE_MODEL = "hires_upscale_checkpoint"
C_PRIMARY_UPSCALE_MODEL = "primary_upscale_checkpoint"
C_SECONDARY_UPSCALE_MODEL = "secondary_upscale_checkpoint"
C_DETAIL_PROCESSOR_MODEL = "detail_processor_checkpoint"
C_CLIP_VISION_MODEL = "clip_vision_checkpoint"
C_CN_CANNY_MODEL = "cn_canny_checkpoint"
C_CN_DEPTH_MODEL = "cn_depth_checkpoint"
C_CN_RECOLOR_MODEL = "cn_recolor_checkpoint"
C_CN_SKETCH_MODEL = "cn_sketch_checkpoint"
C_CN_CUSTOM_MODEL = "cn_custom_checkpoint"
# apply loras
C_APPLIED_LORAS = "applied_loras"
# clip conditioning
C_PROCESSED_PROMPTS = "processed_prompts"
C_BASE_CONDITIONING = "base_conditioning"
C_REFINER_CONDITIONING = "refiner_conditioning"
# apply controlnet
C_APPLIED_CONTROLNET = "applied_controlnet"
# latent inputs
C_LATENT_FROM_IMAGE = "latent_from_image"
C_IMAGE_MASK = "image_mask"
C_LATENT_WITH_MASK = "latent_with_mask"
C_EMPTY_LATENT = "empty_latent"
# sampler
C_SAMPLED = "sampled"
# latent detailer
C_SAMPLED_DETAILER = "sampled_detailer"
# vae decode sampled
C_VAE_DECODED = "vae_decoded"
C_POST_PROCESSED = "post_processed"
# high resolution
C_HIRES_LATENT = "hires_latent"
C_HIRES_LATENT_SIMPLE = "hires_latent_simple"
C_HIRES_LATENT_NORMAL = "hires_latent_normal"
# vae decode sampled
C_VAE_DECODED_HIRES = "vae_decoded_hires"
C_POST_PROCESSED_HIRES = "post_processed_hires"
# upscaling
C_UPSCALED_IMAGE = "upscaled_image"
# ----------------------------------------
# pipeline stream names
# ----------------------------------------
P_IMAGE = "image"
P_MASK = "mask"
P_LATENT = "latent"
P_BASE_MODEL = "base_model"
P_BASE_CLIP = "base_clip"
P_BASE_VAE = "base_vae"
P_REFINER_MODEL = "refiner_model"
P_REFINER_CLIP = "refiner_clip"
P_REFINER_VAE = "refiner_vae"
P_VAE_MODEL = "vae_model"
P_HIRES_UPSCALER = "hires_upscaler"
P_PRIMARY_UPSCALER = "primary_upscaler"
P_SECONDARY_UPSCALER = "secondary_upscaler"
P_DETAIL_PROCESSOR = "detail_processor"
P_CLIP_VISION_MODEL = "clip_vision_model"
P_CN_CANNY_MODEL = "cn_canny_model"
P_CN_DEPTH_MODEL = "cn_depth_model"
P_CN_RECOLOR_MODEL = "cn_recolor_model"
P_CN_SKETCH_MODEL = "cn_sketch_model"
P_CN_CUSTOM_MODEL = "cn_custom_model"
P_PROCESSED_PROMPTS = "processed_prompts"
P_BASE_CONDITIONING = "base_conditioning"
P_REFINER_CONDITIONING = "refiner_conditioning"
modules/node_wrapper.py
+108
View File
@@ -0,0 +1,108 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
import nodes
import comfy_extras.nodes_mask
import comfy_extras.nodes_post_processing
import comfy_extras.nodes_clip_sdxl
import comfy_extras.nodes_upscale_model
from .custom_sdxl_ksampler import sdxl_ksampler
from .ui import UI
# ====================================================================================================
# Wrapper for other ComfyUI nodes
# ====================================================================================================
class NodeWrapper:
checkpoint_loader = nodes.CheckpointLoaderSimple()
clipvision_encoder = nodes.CLIPVisionEncode()
clipvision_loader = nodes.CLIPVisionLoader()
controlnet_advanced = nodes.ControlNetApplyAdvanced()
controlnet_loader = nodes.ControlNetLoader()
empty_latent = nodes.EmptyLatentImage()
image_blend = comfy_extras.nodes_post_processing.Blend()
image_blur = comfy_extras.nodes_post_processing.Blur()
image_composite = comfy_extras.nodes_mask.ImageCompositeMasked()
image_scale = nodes.ImageScale()
image_to_mask = comfy_extras.nodes_mask.ImageToMask()
latent_repeater = nodes.RepeatLatentBatch()
latent_selector = nodes.LatentFromBatch()
latent_upscale_by = nodes.LatentUpscaleBy()
lora_loader = nodes.LoraLoader()
mask_to_image = comfy_extras.nodes_mask.MaskToImage()
scale_with_model = comfy_extras.nodes_upscale_model.ImageUpscaleWithModel()
sdxl_clip_base_encoder = comfy_extras.nodes_clip_sdxl.CLIPTextEncodeSDXL()
sdxl_clip_refiner_encoder = comfy_extras.nodes_clip_sdxl.CLIPTextEncodeSDXLRefiner()
set_latent_mask = nodes.SetLatentNoiseMask()
unclip_conditioning = nodes.unCLIPConditioning()
upscale_loader = comfy_extras.nodes_upscale_model.UpscaleModelLoader()
vae_decoder = nodes.VAEDecode()
vae_encoder = nodes.VAEEncode()
vae_loader = nodes.VAELoader()
@staticmethod
def sdxl_sampler(base_model, base_positive, base_negative, latent_image, noise_seed, steps, cfg,
sampler_name, scheduler, refiner_model=None, refiner_positive=None, refiner_negative=None,
base_ratio=0.8, denoise=1.0, cfg_method=None, dynamic_base_cfg=0.0, dynamic_refiner_cfg=0.0,
refiner_detail_boost=0.0):
has_refiner_model = refiner_model is not None
base_steps = int(steps * (base_ratio + 0.0001)) if has_refiner_model else steps
refiner_steps = max(0, steps - base_steps)
if cfg_method == UI.NONE:
cfg_method = None
if denoise < 0.005:
return (latent_image,)
if refiner_steps == 0 or not has_refiner_model:
result = sdxl_ksampler(base_model, None, noise_seed, base_steps, 0, cfg, sampler_name,
scheduler, base_positive, base_negative, None, None,
latent_image, denoise=denoise, disable_noise=False, start_step=0, last_step=steps,
force_full_denoise=True, dynamic_base_cfg=dynamic_base_cfg, cfg_method=cfg_method)
else:
result = sdxl_ksampler(base_model, refiner_model, noise_seed, base_steps, refiner_steps, cfg, sampler_name,
scheduler, base_positive, base_negative, refiner_positive, refiner_negative,
latent_image, denoise=denoise, disable_noise=False,
start_step=0, last_step=steps, force_full_denoise=True,
dynamic_base_cfg=dynamic_base_cfg, dynamic_refiner_cfg=dynamic_refiner_cfg,
cfg_method=cfg_method, refiner_detail_boost=refiner_detail_boost)
return result[0]
@staticmethod
def common_sampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent,
denoise=1.0, disable_noise=False, start_step=None, last_step=None, force_full_denoise=False):
result = nodes.common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative,
latent, denoise=denoise, disable_noise=disable_noise, start_step=start_step,
last_step=last_step, force_full_denoise=force_full_denoise)
return result[0]
modules/processing.py
-239
View File
@@ -1,239 +0,0 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
# UI: Parameter Processor
class SeargeParameterProcessor:
# hard - refiner uses same seed as base | soft - refiner uses different seed from base
REFINER_INTENSITY = ["hard", "soft"]
# same - HRF uses same seed as image generation | distinct - HRF uses different seed than image generation
HRF_SEED_OFFSET = ["same", "distinct"]
# simple "boolean"-like type
STATES = ["disabled", "enabled"]
# operating modes, determine if the latent source is empty or from a source image, inpainting also uses mask
OPERATION_MODE = ["text to image", "image to image", "inpainting"]
# sorted from easy-to-use to harder-to-use
PROMPT_STYLE = ["simple", "3 prompts G+L-N", "subject focus", "style focus", "weighted", "overlay", "subject - style", "style - subject", "style only", "weighted - overlay", "overlay - weighted"]
# (work in progress)
STYLE_TEMPLATE = ["none", "from preprocessor", "test"]
# save folder for generated images
SAVE_TO = ["output folder", "input folder"]
@classmethod
def INPUT_TYPES(s):
return {"required": {
"inputs": ("PARAMETER_INPUTS", ),
},
}
RETURN_TYPES = ("PARAMETERS", )
RETURN_NAMES = ("parameters", )
FUNCTION = "process"
CATEGORY = "Searge/UI"
def process(self, inputs):
if inputs is None:
parameters = {}
else:
parameters = inputs
if parameters["denoise"] is None:
parameters["denoise"] = 1.0
saturation = parameters["refiner_intensity"]
if saturation is not None:
# "soft"
if saturation == SeargeParameterProcessor.REFINER_INTENSITY[1]:
parameters["noise_offset"] = 1
# incl. SeargeParameterProcessor.REFINER_INTENSITY[1] -> "hard"
else:
parameters["noise_offset"] = 0
hires_fix = parameters["hires_fix"]
# "disabled"
if hires_fix is not None and hires_fix == SeargeParameterProcessor.STATES[0]:
parameters["hrf_steps"] = 0
hrf_saturation = parameters["hrf_intensity"]
if hrf_saturation is not None:
# "soft"
if hrf_saturation == SeargeParameterProcessor.REFINER_INTENSITY[1]:
parameters["hrf_noise_offset"] = 1
# incl. SeargeParameterProcessor.REFINER_INTENSITY[0] -> "hard"
else:
parameters["hrf_noise_offset"] = 0
seed_offset = parameters["hrf_seed_offset"]
if seed_offset is not None:
seed = parameters["seed"] if parameters["seed"] is not None else 0
# "distinct"
if seed_offset == SeargeParameterProcessor.HRF_SEED_OFFSET[1]:
parameters["hrf_seed"] = seed + 3
# incl. SeargeParameterProcessor.HRF_SEED_OFFSET[0] -> "same"
else:
parameters["hrf_seed"] = seed
style_template = parameters["style_template"]
if style_template is not None:
# "from preprocessor"
if style_template == SeargeParameterProcessor.STYLE_TEMPLATE[1]:
# this does nothing here, but will be used in the preprocessor
pass
# "test"
if style_template == SeargeParameterProcessor.STYLE_TEMPLATE[2]:
if parameters["noise_offset"] is not None:
parameters["noise_offset"] = 1 - parameters["hrf_noise_offset"]
if parameters["hrf_noise_offset"] is not None:
parameters["hrf_noise_offset"] = 1 - parameters["hrf_noise_offset"]
# incl. SeargeParameterProcessor.STYLE_TEMPLATE[0] -> "none"
else:
# TODO: apply style based on its name here...
pass
operation_mode = parameters["operation_mode"]
if operation_mode is not None:
# "image to image":
if operation_mode == SeargeParameterProcessor.OPERATION_MODE[1]:
parameters["operation_selector"] = 1
# "inpainting":
elif operation_mode == SeargeParameterProcessor.OPERATION_MODE[2]:
parameters["operation_selector"] = 2
# incl. SeargeParameterProcessor.OPERATION_MODE[0] -> "text to image":
else:
parameters["operation_selector"] = 0
# always fully denoise in img2img mode
parameters["denoise"] = 1.0
prompt_style = parameters["prompt_style"]
if prompt_style is not None:
# "simple"
if prompt_style == SeargeParameterProcessor.PROMPT_STYLE[0]:
parameters["prompt_style_selector"] = 0
parameters["prompt_style_group"] = 0
main_prompt = parameters["main_prompt"]
parameters["secondary_prompt"] = main_prompt
parameters["style_prompt"] = ""
parameters["negative_style"] = ""
# "subject focus"
elif prompt_style == SeargeParameterProcessor.PROMPT_STYLE[2]:
parameters["prompt_style_selector"] = 1
parameters["prompt_style_group"] = 0
# "style focus"
elif prompt_style == SeargeParameterProcessor.PROMPT_STYLE[3]:
parameters["prompt_style_selector"] = 2
parameters["prompt_style_group"] = 0
# "weighted"
elif prompt_style == SeargeParameterProcessor.PROMPT_STYLE[4]:
parameters["prompt_style_selector"] = 3
parameters["prompt_style_group"] = 0
# "overlay"
elif prompt_style == SeargeParameterProcessor.PROMPT_STYLE[5]:
parameters["prompt_style_selector"] = 4
parameters["prompt_style_group"] = 0
# "subject - style"
elif prompt_style == SeargeParameterProcessor.PROMPT_STYLE[6]:
parameters["prompt_style_selector"] = 0
parameters["prompt_style_group"] = 1
# "style - subject"
elif prompt_style == SeargeParameterProcessor.PROMPT_STYLE[7]:
parameters["prompt_style_selector"] = 1
parameters["prompt_style_group"] = 1
# "style only"
elif prompt_style == SeargeParameterProcessor.PROMPT_STYLE[8]:
parameters["prompt_style_selector"] = 2
parameters["prompt_style_group"] = 1
# "weighted - overlay"
elif prompt_style == SeargeParameterProcessor.PROMPT_STYLE[9]:
parameters["prompt_style_selector"] = 3
parameters["prompt_style_group"] = 1
# "overlay - weighted"
elif prompt_style == SeargeParameterProcessor.PROMPT_STYLE[10]:
parameters["prompt_style_selector"] = 4
parameters["prompt_style_group"] = 1
# incl. SeargeParameterProcessor.PROMPT_STYLE[1] -> "3 prompts G+L-N"
else:
parameters["prompt_style_selector"] = 0
parameters["prompt_style_group"] = 0
parameters["style_prompt"] = ""
parameters["negative_style"] = ""
# TODO: replace this special logic and the dirty hacks by creating new generated parameters for saving
save_image = parameters["save_image"]
if save_image is not None:
# "disabled"
if save_image == SeargeParameterProcessor.STATES[0]:
# when image saving is disabled, we also don't want to save the upscaled image, even if that's enabled
parameters["save_upscaled_image"] = SeargeParameterProcessor.STATES[0]
# HACK: this is a bit dirty, but the variable hires_fix determines if the image should be saved
# but when image saving is disabled, we don't want that to happen
parameters["hires_fix"] = SeargeParameterProcessor.STATES[0]
# "enabled"
else:
# in case we are saving to the input folder, we need to enable saving after the hires fix, even
# if that's disabled in the settings
if parameters["save_directory"] == SeargeParameterProcessor.SAVE_TO[1]:
parameters["hires_fix"] = SeargeParameterProcessor.STATES[1]
return (parameters, )
# UI: Style Processor
class SeargeStylePreprocessor:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"inputs": ("PARAMETER_INPUTS", ),
"active_style_name": ("STRING", {"multiline": False, "default": ""}),
"style_definitions": ("STRING", {"multiline": True, "default": "[unfinished work in progress]"}),
},
}
RETURN_TYPES = ("PARAMETER_INPUTS", )
RETURN_NAMES = ("inputs", )
FUNCTION = "process"
CATEGORY = "Searge/UI"
def process(self, inputs, active_style_name, style_definitions):
if inputs is None:
inputs = {}
style_template = inputs["style_template"]
# not "from preprocessor"
if style_template is None or style_template != SeargeParameterProcessor.STYLE_TEMPLATE[1]:
return (inputs,)
# TODO: do what needs to be done to apply the selected style
return (inputs, )
modules/prompt_adapter.py
+85
View File
@@ -0,0 +1,85 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .ui import UI
# ====================================================================================================
# Adapter for prompt text inputs
# ====================================================================================================
class SeargePromptAdapterV2:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
},
"optional": {
"data": ("SRG_DATA_STREAM",),
"main_prompt": ("SRG_PROMPT_TEXT",),
"secondary_prompt": ("SRG_PROMPT_TEXT",),
"style_prompt": ("SRG_PROMPT_TEXT",),
"negative_main_prompt": ("SRG_PROMPT_TEXT",),
"negative_secondary_prompt": ("SRG_PROMPT_TEXT",),
"negative_style_prompt": ("SRG_PROMPT_TEXT",),
},
}
RETURN_TYPES = ("SRG_DATA_STREAM", "SRG_DATA_STREAM",)
RETURN_NAMES = ("data", UI.S_PROMPTS,)
FUNCTION = "get_value"
CATEGORY = UI.CATEGORY_UI_PROMPTING
@staticmethod
def create_dict(main_prompt=None, secondary_prompt=None, style_prompt=None,
negative_main_prompt=None, negative_secondary_prompt=None, negative_style_prompt=None):
return {
UI.F_MAIN_PROMPT: main_prompt,
UI.F_SECONDARY_PROMPT: secondary_prompt,
UI.F_STYLE_PROMPT: style_prompt,
UI.F_NEGATIVE_MAIN_PROMPT: negative_main_prompt,
UI.F_NEGATIVE_SECONDARY_PROMPT: negative_secondary_prompt,
UI.F_NEGATIVE_STYLE_PROMPT: negative_style_prompt,
}
def get_value(self, main_prompt=None, secondary_prompt=None, style_prompt=None,
negative_main_prompt=None, negative_secondary_prompt=None, negative_style_prompt=None, data=None):
if data is None:
data = {}
data[UI.S_PROMPTS] = self.create_dict(
main_prompt,
secondary_prompt,
style_prompt,
negative_main_prompt,
negative_secondary_prompt,
negative_style_prompt
)
return (data, data[UI.S_PROMPTS],)
modules/prompt_adapter_output.py
+82
View File
@@ -0,0 +1,82 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .data_utils import retrieve_parameter
from .ui import UI
# ====================================================================================================
# UI: Prompt Adapter Output
# ====================================================================================================
class SeargePromptAdapterV2Output:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
},
"optional": {
"data": ("SRG_DATA_STREAM",),
"prompts": ("SRG_DATA_STREAM",),
},
}
RETURN_TYPES = ("SRG_DATA_STREAM", "STRING", "STRING", "STRING",
"STRING", "STRING", "STRING",)
RETURN_NAMES = ("data", UI.F_MAIN_PROMPT, UI.F_SECONDARY_PROMPT, UI.F_STYLE_PROMPT,
UI.F_NEGATIVE_MAIN_PROMPT, UI.F_NEGATIVE_SECONDARY_PROMPT, UI.F_NEGATIVE_STYLE_PROMPT,)
FUNCTION = "output"
CATEGORY = UI.CATEGORY_UI_PROMPTING
@staticmethod
def get_data(data=None, prompts=None):
if prompts is None:
prompts = retrieve_parameter("prompts", data)
if prompts is None:
return (False, None,)
return (True, {
UI.F_MAIN_PROMPT: retrieve_parameter(UI.F_MAIN_PROMPT, prompts),
UI.F_SECONDARY_PROMPT: retrieve_parameter(UI.F_SECONDARY_PROMPT, prompts),
UI.F_STYLE_PROMPT: retrieve_parameter(UI.F_STYLE_PROMPT, prompts),
UI.F_NEGATIVE_MAIN_PROMPT: retrieve_parameter(UI.F_NEGATIVE_MAIN_PROMPT, prompts),
UI.F_NEGATIVE_SECONDARY_PROMPT: retrieve_parameter(UI.F_NEGATIVE_SECONDARY_PROMPT, prompts),
UI.F_NEGATIVE_STYLE_PROMPT: retrieve_parameter(UI.F_NEGATIVE_STYLE_PROMPT, prompts),
})
def output(self, data=None, prompts=None):
(has_data, output) = self.get_data(data, prompts)
if not has_data:
return (data, None, None, None, None, None, None,)
return (data, output[UI.F_MAIN_PROMPT], output[UI.F_SECONDARY_PROMPT],
output[UI.F_STYLE_PROMPT],
output[UI.F_NEGATIVE_MAIN_PROMPT], output[UI.F_NEGATIVE_SECONDARY_PROMPT],
output[UI.F_NEGATIVE_STYLE_PROMPT],)
modules/prompt_text_input.py
+52
View File
@@ -0,0 +1,52 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .ui import UI
# ====================================================================================================
# Text input node for prompt text
# ====================================================================================================
class SeargeTextInputV2:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"prompt": ("STRING", {"default": "", "multiline": True},),
},
}
RETURN_TYPES = ("SRG_PROMPT_TEXT",)
RETURN_NAMES = ("prompt_text",)
FUNCTION = "get_value"
CATEGORY = UI.CATEGORY_UI_PROMPTING
def get_value(self, prompt):
return (prompt,)
modules/prompting.py
-251
View File
@@ -1,251 +0,0 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
import nodes
# SDXL CLIP Text Encoder for prompts with base and refiner support
class SeargeSDXLPromptEncoder:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"base_clip": ("CLIP", ),
"refiner_clip": ("CLIP", ),
"pos_g": ("STRING", {"multiline": True, "default": "POS_G"}),
"pos_l": ("STRING", {"multiline": True, "default": "POS_L"}),
"pos_r": ("STRING", {"multiline": True, "default": "POS_R"}),
"neg_g": ("STRING", {"multiline": True, "default": "NEG_G"}),
"neg_l": ("STRING", {"multiline": True, "default": "NEG_L"}),
"neg_r": ("STRING", {"multiline": True, "default": "NEG_R"}),
"base_width": ("INT", {"default": 4096, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"base_height": ("INT", {"default": 4096, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"crop_w": ("INT", {"default": 0, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"crop_h": ("INT", {"default": 0, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"target_width": ("INT", {"default": 4096, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"target_height": ("INT", {"default": 4096, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"pos_ascore": ("FLOAT", {"default": 6.0, "min": 0.0, "max": 1000.0, "step": 0.01}),
"neg_ascore": ("FLOAT", {"default": 2.5, "min": 0.0, "max": 1000.0, "step": 0.01}),
"refiner_width": ("INT", {"default": 2048, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"refiner_height": ("INT", {"default": 2048, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
},
}
RETURN_TYPES = ("CONDITIONING", "CONDITIONING", "CONDITIONING", "CONDITIONING", )
RETURN_NAMES = ("base_positive", "base_negative", "refiner_positive", "refiner_negative", )
FUNCTION = "encode"
CATEGORY = "Searge/ClipEncoding"
def encode(self, base_clip, refiner_clip, pos_g, pos_l, pos_r, neg_g, neg_l, neg_r, base_width, base_height, crop_w, crop_h, target_width, target_height, pos_ascore, neg_ascore, refiner_width, refiner_height, ):
empty = base_clip.tokenize("")
# positive base prompt
tokens1 = base_clip.tokenize(pos_g)
tokens1["l"] = base_clip.tokenize(pos_l)["l"]
if len(tokens1["l"]) != len(tokens1["g"]):
while len(tokens1["l"]) < len(tokens1["g"]):
tokens1["l"] += empty["l"]
while len(tokens1["l"]) > len(tokens1["g"]):
tokens1["g"] += empty["g"]
cond1, pooled1 = base_clip.encode_from_tokens(tokens1, return_pooled=True)
res1 = [[cond1, {"pooled_output": pooled1, "width": base_width, "height": base_height, "crop_w": crop_w, "crop_h": crop_h, "target_width": target_width, "target_height": target_height}]]
# negative base prompt
tokens2 = base_clip.tokenize(neg_g)
tokens2["l"] = base_clip.tokenize(neg_l)["l"]
if len(tokens2["l"]) != len(tokens2["g"]):
while len(tokens2["l"]) < len(tokens2["g"]):
tokens2["l"] += empty["l"]
while len(tokens2["l"]) > len(tokens2["g"]):
tokens2["g"] += empty["g"]
cond2, pooled2 = base_clip.encode_from_tokens(tokens2, return_pooled=True)
res2 = [[cond2, {"pooled_output": pooled2, "width": base_width, "height": base_height, "crop_w": crop_w, "crop_h": crop_h, "target_width": target_width, "target_height": target_height}]]
# positive refiner prompt
tokens3 = refiner_clip.tokenize(pos_r)
cond3, pooled3 = refiner_clip.encode_from_tokens(tokens3, return_pooled=True)
res3 = [[cond3, {"pooled_output": pooled3, "aesthetic_score": pos_ascore, "width": refiner_width, "height": refiner_height}]]
# negative refiner prompt
tokens4 = refiner_clip.tokenize(neg_r)
cond4, pooled4 = refiner_clip.encode_from_tokens(tokens4, return_pooled=True)
res4 = [[cond4, {"pooled_output": pooled4, "aesthetic_score": neg_ascore, "width": refiner_width, "height": refiner_height}]]
return (res1, res2, res3, res4, )
# SDXL CLIP Text Encoder for base prompts
class SeargeSDXLBasePromptEncoder:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"base_clip": ("CLIP", ),
"pos_g": ("STRING", {"multiline": True, "default": "POS_G"}),
"pos_l": ("STRING", {"multiline": True, "default": "POS_L"}),
"neg_g": ("STRING", {"multiline": True, "default": "NEG_G"}),
"neg_l": ("STRING", {"multiline": True, "default": "NEG_L"}),
"base_width": ("INT", {"default": 4096, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"base_height": ("INT", {"default": 4096, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"crop_w": ("INT", {"default": 0, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"crop_h": ("INT", {"default": 0, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"target_width": ("INT", {"default": 4096, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"target_height": ("INT", {"default": 4096, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
},
}
RETURN_TYPES = ("CONDITIONING", "CONDITIONING", )
RETURN_NAMES = ("base_positive", "base_negative", )
FUNCTION = "encode"
CATEGORY = "Searge/ClipEncoding"
def encode(self, base_clip, pos_g, pos_l, neg_g, neg_l, base_width, base_height, crop_w, crop_h, target_width, target_height, ):
empty = base_clip.tokenize("")
# positive base prompt
tokens1 = base_clip.tokenize(pos_g)
tokens1["l"] = base_clip.tokenize(pos_l)["l"]
if len(tokens1["l"]) != len(tokens1["g"]):
while len(tokens1["l"]) < len(tokens1["g"]):
tokens1["l"] += empty["l"]
while len(tokens1["l"]) > len(tokens1["g"]):
tokens1["g"] += empty["g"]
cond1, pooled1 = base_clip.encode_from_tokens(tokens1, return_pooled=True)
res1 = [[cond1, {"pooled_output": pooled1, "width": base_width, "height": base_height, "crop_w": crop_w, "crop_h": crop_h, "target_width": target_width, "target_height": target_height}]]
# negative base prompt
tokens2 = base_clip.tokenize(neg_g)
tokens2["l"] = base_clip.tokenize(neg_l)["l"]
if len(tokens2["l"]) != len(tokens2["g"]):
while len(tokens2["l"]) < len(tokens2["g"]):
tokens2["l"] += empty["l"]
while len(tokens2["l"]) > len(tokens2["g"]):
tokens2["g"] += empty["g"]
cond2, pooled2 = base_clip.encode_from_tokens(tokens2, return_pooled=True)
res2 = [[cond2, {"pooled_output": pooled2, "width": base_width, "height": base_height, "crop_w": crop_w, "crop_h": crop_h, "target_width": target_width, "target_height": target_height}]]
return (res1, res2, )
# SDXL CLIP Text Encoder for prompts with base and refiner support
class SeargeSDXLRefinerPromptEncoder:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"refiner_clip": ("CLIP", ),
"pos_r": ("STRING", {"multiline": True, "default": "POS_R"}),
"neg_r": ("STRING", {"multiline": True, "default": "NEG_R"}),
"pos_ascore": ("FLOAT", {"default": 6.0, "min": 0.0, "max": 1000.0, "step": 0.01}),
"neg_ascore": ("FLOAT", {"default": 2.5, "min": 0.0, "max": 1000.0, "step": 0.01}),
"refiner_width": ("INT", {"default": 2048, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"refiner_height": ("INT", {"default": 2048, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
},
}
RETURN_TYPES = ("CONDITIONING", "CONDITIONING", )
RETURN_NAMES = ("refiner_positive", "refiner_negative", )
FUNCTION = "encode"
CATEGORY = "Searge/ClipEncoding"
def encode(self, refiner_clip, pos_r, neg_r, pos_ascore, neg_ascore, refiner_width, refiner_height, ):
# positive refiner prompt
tokens1 = refiner_clip.tokenize(pos_r)
cond1, pooled1 = refiner_clip.encode_from_tokens(tokens1, return_pooled=True)
res1 = [[cond1, {"pooled_output": pooled1, "aesthetic_score": pos_ascore, "width": refiner_width, "height": refiner_height}]]
# negative refiner prompt
tokens2 = refiner_clip.tokenize(neg_r)
cond2, pooled2 = refiner_clip.encode_from_tokens(tokens2, return_pooled=True)
res2 = [[cond2, {"pooled_output": pooled2, "aesthetic_score": neg_ascore, "width": refiner_width, "height": refiner_height}]]
return (res1, res2, )
# Tool: text input node for prompt text
class SeargePromptText:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"prompt": ("STRING", {"default": "", "multiline": True}),
},
}
RETURN_TYPES = ("STRING", )
RETURN_NAMES = ("prompt", )
FUNCTION = "get_value"
CATEGORY = "Searge/Prompting"
def get_value(self, prompt):
return (prompt,)
# Tool: text input node for prompt text
class SeargePromptCombiner:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"prompt1": ("STRING", {"default": "", "multiline": True}),
"separator": ("STRING", {"default": ", ", "multiline": False}),
"prompt2": ("STRING", {"default": "", "multiline": True}),
},
}
RETURN_TYPES = ("STRING", )
RETURN_NAMES = ("combined prompt", )
FUNCTION = "get_value"
CATEGORY = "Searge/Prompting"
def get_value(self, prompt1, separator, prompt2, ):
len1 = len(prompt1)
len2 = len(prompt2)
prompt = ""
if len1 > 0 and len2 > 0:
prompt = prompt1 + separator + prompt2
elif len1 > 0:
prompt = prompt1
elif len2 > 0:
prompt = prompt2
return (prompt,)
modules/sampling.py
-448
View File
@@ -1,448 +0,0 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
import comfy.model_management
import comfy.sample
import comfy.samplers
import comfy.utils
import comfy_extras.nodes_post_processing
import comfy_extras.nodes_upscale_model
import latent_preview
import nodes
import torch
def sdxl_sample(modelB, modelR, noiseO, stepsB, stepsR, cfg, sampler_name, scheduler, positiveB, negativeB, positiveR, negativeR, latent_image, denoise=1.0, start_step=None, last_step=None, force_full_denoise=False, noise_mask=None, sigmas=None, callbackB=None, callbackR=None, disable_pbar=False, seed=None):
device = comfy.model_management.get_torch_device()
if noise_mask is not None:
noise_mask = comfy.sample.prepare_mask(noise_mask, noiseO.shape, device)
steps = stepsB + stepsR
comfy.model_management.load_model_gpu(modelB)
real_modelB = modelB.model
noise = noiseO.to(device)
latent_image = latent_image.to(device)
positive_copyB = comfy.sample.broadcast_cond(positiveB, noise.shape[0], device)
negative_copyB = comfy.sample.broadcast_cond(negativeB, noise.shape[0], device)
modelsB = comfy.sample.load_additional_models(positiveB, negativeB, modelB.model_dtype())
samplerB = comfy.samplers.KSampler(real_modelB, steps=steps, device=device, sampler=sampler_name, scheduler=scheduler, denoise=denoise, model_options=modelB.model_options)
samplesB = samplerB.sample(noise, positive_copyB, negative_copyB, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=stepsB, force_full_denoise=False, denoise_mask=noise_mask, sigmas=sigmas, callback=callbackB, disable_pbar=disable_pbar, seed=seed)
comfy.sample.cleanup_additional_models(modelsB)
noise = torch.zeros(samplesB.size(), dtype=samplesB.dtype, layout=samplesB.layout, device=device)
if noise_mask is not None:
latent_for_refiner = samplesB * noise_mask + latent_image * (1.0 - noise_mask)
else:
latent_for_refiner = samplesB
comfy.model_management.load_model_gpu(modelR)
real_modelR = modelR.model
positive_copyR = comfy.sample.broadcast_cond(positiveR, noise.shape[0], device)
negative_copyR = comfy.sample.broadcast_cond(negativeR, noise.shape[0], device)
modelsR = comfy.sample.load_additional_models(positiveR, negativeR, modelR.model_dtype())
samplerR = comfy.samplers.KSampler(real_modelR, steps=steps, device=device, sampler=sampler_name, scheduler=scheduler, denoise=denoise, model_options=modelR.model_options)
samples = samplerR.sample(noise, positive_copyR, negative_copyR, cfg=cfg, latent_image=latent_for_refiner, start_step=stepsB, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callbackR, disable_pbar=disable_pbar, seed=seed)
samples = samples.cpu()
comfy.sample.cleanup_additional_models(modelsR)
return samples
def sdxl_ksampler(modelB, modelR, seed, stepsB, stepsR, cfg, sampler_name, scheduler, positiveB, negativeB, positiveR, negativeR, latent, denoise=1.0, disable_noise=False, start_step=None, last_step=None, force_full_denoise=False):
device = comfy.model_management.get_torch_device()
latent_image = latent["samples"]
if disable_noise:
noise = torch.zeros(latent_image.size(), dtype=latent_image.dtype, layout=latent_image.layout, device="cpu")
else:
batch_inds = latent["batch_index"] if "batch_index" in latent else None
noise = comfy.sample.prepare_noise(latent_image, seed, batch_inds)
noise_mask = None
if "noise_mask" in latent:
noise_mask = latent["noise_mask"]
preview_format = "JPEG"
if preview_format not in ["JPEG", "PNG"]:
preview_format = "JPEG"
previewerB = latent_preview.get_previewer(device, modelB.model.latent_format)
previewerR = latent_preview.get_previewer(device, modelR.model.latent_format)
steps = stepsB + stepsR
pbar = comfy.utils.ProgressBar(steps)
def callbackB(step, x0, x, total_steps):
preview_bytes = None
if previewerB:
preview_bytes = previewerB.decode_latent_to_preview_image(preview_format, x0)
pbar.update_absolute(step + 1, total_steps, preview_bytes)
def callbackR(step, x0, x, total_steps):
preview_bytes = None
if previewerR:
preview_bytes = previewerR.decode_latent_to_preview_image(preview_format, x0)
pbar.update_absolute(step + 1, total_steps, preview_bytes)
samples = sdxl_sample(modelB, modelR, noise, stepsB, stepsR, cfg, sampler_name, scheduler, positiveB, negativeB, positiveR, negativeR, latent_image,
denoise=denoise, start_step=start_step, last_step=last_step,
force_full_denoise=force_full_denoise, noise_mask=noise_mask, callbackB=callbackB, callbackR=callbackR, seed=seed)
out = latent.copy()
out["samples"] = samples
return (out, )
# SDXL Sampler with base and refiner support
class SeargeSDXLSampler2:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"base_model": ("MODEL",),
"base_positive": ("CONDITIONING", ),
"base_negative": ("CONDITIONING", ),
"refiner_model": ("MODEL",),
"refiner_positive": ("CONDITIONING", ),
"refiner_negative": ("CONDITIONING", ),
"latent_image": ("LATENT", ),
"noise_seed": ("INT", {"default": 0, "min": 0, "max": 0xfffffffffffffff0}),
"steps": ("INT", {"default": 20, "min": 1, "max": 200}),
"cfg": ("FLOAT", {"default": 7.0, "min": 0.0, "max": 30.0, "step": 0.5}),
"sampler_name": ("SAMPLER_NAME", {"default": "ddim"}),
"scheduler": ("SCHEDULER_NAME", {"default": "ddim_uniform"}),
"base_ratio": ("FLOAT", {"default": 0.8, "min": 0.0, "max": 1.0, "step": 0.01}),
"denoise": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.01}),
},
"optional": {
"refiner_prep_steps": ("INT", {"default": 0, "min": 0, "max": 10}),
"noise_offset": ("INT", {"default": 1, "min": 0, "max": 1}),
"refiner_strength": ("FLOAT", {"default": 1.0, "min": 0.01, "max": 1.0, "step": 0.05}),
},
}
RETURN_TYPES = ("LATENT", )
FUNCTION = "sample"
CATEGORY = "Searge/Sampling"
def sample(self, base_model, base_positive, base_negative, refiner_model, refiner_positive, refiner_negative, latent_image, noise_seed, steps, cfg, sampler_name, scheduler, base_ratio, denoise, refiner_prep_steps=None, noise_offset=None, refiner_strength=None):
base_steps = int(steps * (base_ratio + 0.0001))
if noise_offset is None:
noise_offset = 1
if refiner_strength is None:
refiner_strength = 1.0
if refiner_strength < 0.01:
refiner_strength = 0.01
if denoise < 0.01:
return (latent_image, )
start_at_step = 0
input_latent = latent_image
if refiner_prep_steps is not None:
if refiner_prep_steps >= base_steps:
refiner_prep_steps = base_steps - 1
if refiner_prep_steps > 0:
start_at_step = refiner_prep_steps
precondition_result = nodes.common_ksampler(refiner_model, noise_seed + 2, steps, cfg, sampler_name, scheduler, refiner_positive, refiner_negative, latent_image, denoise=denoise, disable_noise=False, start_step=steps - refiner_prep_steps, last_step=steps, force_full_denoise=False)
input_latent = precondition_result[0]
if base_steps >= steps:
return nodes.common_ksampler(base_model, noise_seed, steps, cfg, sampler_name, scheduler, base_positive, base_negative, input_latent, denoise=denoise, disable_noise=False, start_step=start_at_step, last_step=steps, force_full_denoise=True)
base_result = nodes.common_ksampler(base_model, noise_seed, steps, cfg, sampler_name, scheduler, base_positive, base_negative, input_latent, denoise=denoise, disable_noise=False, start_step=start_at_step, last_step=base_steps, force_full_denoise=True)
return nodes.common_ksampler(refiner_model, noise_seed + noise_offset, steps, cfg, sampler_name, scheduler, refiner_positive, refiner_negative, base_result[0], denoise=denoise * refiner_strength, disable_noise=False, start_step=base_steps, last_step=steps, force_full_denoise=True)
# SDXL Image2Image Sampler (incl. HiRes Fix)
class SeargeSDXLImage2ImageSampler2:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"base_model": ("MODEL",),
"base_positive": ("CONDITIONING", ),
"base_negative": ("CONDITIONING", ),
"refiner_model": ("MODEL",),
"refiner_positive": ("CONDITIONING",),
"refiner_negative": ("CONDITIONING",),
"image": ("IMAGE", ),
"vae": ("VAE",),
"noise_seed": ("INT", {"default": 0, "min": 0, "max": 0xfffffffffffffff0}),
"steps": ("INT", {"default": 20, "min": 0, "max": 200}),
"cfg": ("FLOAT", {"default": 7.0, "min": 0.0, "max": 30.0, "step": 0.5}),
"sampler_name": ("SAMPLER_NAME", {"default": "ddim"}),
"scheduler": ("SCHEDULER_NAME", {"default": "ddim_uniform"}),
"base_ratio": ("FLOAT", {"default": 0.8, "min": 0.0, "max": 1.0, "step": 0.01}),
"denoise": ("FLOAT", {"default": 0.25, "min": 0.0, "max": 1.0, "step": 0.01}),
},
"optional": {
"upscale_model": ("UPSCALE_MODEL",),
"scaled_width": ("INT", {"default": 1536, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"scaled_height": ("INT", {"default": 1536, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"noise_offset": ("INT", {"default": 1, "min": 0, "max": 1}),
"refiner_strength": ("FLOAT", {"default": 1.0, "min": 0.1, "max": 1.0, "step": 0.05}),
"softness": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.05}),
},
}
RETURN_TYPES = ("IMAGE", )
FUNCTION = "sample"
CATEGORY = "Searge/Sampling"
def sample(self, base_model, base_positive, base_negative, refiner_model, refiner_positive, refiner_negative, image, vae, noise_seed, steps, cfg, sampler_name, scheduler, base_ratio, denoise, softness, upscale_model=None, scaled_width=None, scaled_height=None, noise_offset=None, refiner_strength=None):
base_steps = int(steps * (base_ratio + 0.0001))
if noise_offset is None:
noise_offset = 1
if refiner_strength is None:
refiner_strength = 1.0
if refiner_strength < 0.01:
refiner_strength = 0.01
if steps < 1:
return (image, )
scaled_image = image
use_upscale_model = upscale_model is not None and softness < 0.9999
if use_upscale_model:
upscale_result = comfy_extras.nodes_upscale_model.ImageUpscaleWithModel().upscale(upscale_model, image)
scaled_image = upscale_result[0]
if scaled_width is not None and scaled_height is not None:
upscale_result = nodes.ImageScale().upscale(scaled_image, "bicubic", scaled_width, scaled_height, "center")
scaled_image = upscale_result[0]
if use_upscale_model and softness > 0.0001:
upscale_result = nodes.ImageScale().upscale(image, "bicubic", scaled_width, scaled_height, "center")
scaled_original = upscale_result[0]
blend_result = comfy_extras.nodes_post_processing.Blend().blend_images(scaled_image, scaled_original, softness, "normal")
scaled_image = blend_result[0]
if denoise < 0.01:
return (scaled_image, )
vae_encode_result = nodes.VAEEncode().encode(vae, scaled_image)
input_latent = vae_encode_result[0]
if base_steps >= steps:
result_latent = nodes.common_ksampler(base_model, noise_seed, steps, cfg, sampler_name, scheduler, base_positive, base_negative, input_latent, denoise=denoise, disable_noise=False, start_step=0, last_step=steps, force_full_denoise=True)
else:
base_result = nodes.common_ksampler(base_model, noise_seed, steps, cfg, sampler_name, scheduler, base_positive, base_negative, input_latent, denoise=denoise, disable_noise=False, start_step=0, last_step=base_steps, force_full_denoise=True)
result_latent = nodes.common_ksampler(refiner_model, noise_seed + noise_offset, steps, cfg, sampler_name, scheduler, refiner_positive, refiner_negative, base_result[0], denoise=denoise * refiner_strength, disable_noise=False, start_step=base_steps, last_step=steps, force_full_denoise=True)
vae_decode_result = nodes.VAEDecode().decode(vae, result_latent[0])
output_image = vae_decode_result[0]
return (output_image, )
# SDXL Sampler with base and refiner support
class SeargeSDXLSamplerV3:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"base_model": ("MODEL",),
"base_positive": ("CONDITIONING", ),
"base_negative": ("CONDITIONING", ),
"refiner_model": ("MODEL",),
"refiner_positive": ("CONDITIONING", ),
"refiner_negative": ("CONDITIONING", ),
"latent_image": ("LATENT", ),
"noise_seed": ("INT", {"default": 0, "min": 0, "max": 0xfffffffffffffff0}),
"steps": ("INT", {"default": 20, "min": 1, "max": 200}),
"cfg": ("FLOAT", {"default": 7.0, "min": 0.0, "max": 30.0, "step": 0.5}),
"sampler_name": ("SAMPLER_NAME", {"default": "ddim"}),
"scheduler": ("SCHEDULER_NAME", {"default": "ddim_uniform"}),
"base_ratio": ("FLOAT", {"default": 0.8, "min": 0.0, "max": 1.0, "step": 0.01}),
"denoise": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.01}),
},
"optional": {
"refiner_prep_steps": ("INT", {"default": 0, "min": 0, "max": 10}),
# "noise_offset": ("INT", {"default": 1, "min": 0, "max": 1}),
# "refiner_strength": ("FLOAT", {"default": 1.0, "min": 0.01, "max": 1.0, "step": 0.05}),
},
}
RETURN_TYPES = ("LATENT", )
FUNCTION = "sample"
CATEGORY = "Searge/Sampling"
def sample(self, base_model, base_positive, base_negative, refiner_model, refiner_positive, refiner_negative, latent_image, noise_seed, steps, cfg, sampler_name, scheduler, base_ratio, denoise, refiner_prep_steps=None, noise_offset=None, refiner_strength=None):
base_steps = int(steps * (base_ratio + 0.0001))
refiner_steps = max(0, steps - base_steps)
# if noise_offset is None:
# noise_offset = 1
# if refiner_strength is None:
# refiner_strength = 1.0
# if refiner_strength < 0.01:
# refiner_strength = 0.01
if denoise < 0.01:
return (latent_image, )
start_at_step = 0
input_latent = latent_image
if refiner_prep_steps is not None:
if refiner_prep_steps >= base_steps:
refiner_prep_steps = base_steps - 1
if refiner_prep_steps > 0:
start_at_step = refiner_prep_steps
precondition_result = nodes.common_ksampler(refiner_model, noise_seed + 2, steps, cfg, sampler_name, scheduler, refiner_positive, refiner_negative, latent_image, denoise=denoise, disable_noise=False, start_step=steps - refiner_prep_steps, last_step=steps, force_full_denoise=False)
input_latent = precondition_result[0]
if base_steps >= steps:
return nodes.common_ksampler(base_model, noise_seed, steps, cfg, sampler_name, scheduler, base_positive, base_negative, input_latent, denoise=denoise, disable_noise=False, start_step=start_at_step, last_step=steps, force_full_denoise=True)
return sdxl_ksampler(base_model, refiner_model, noise_seed, base_steps, refiner_steps, cfg, sampler_name, scheduler, base_positive, base_negative, refiner_positive, refiner_negative, input_latent, denoise=denoise, disable_noise=False, start_step=start_at_step, last_step=steps, force_full_denoise=True)
# base_result = nodes.common_ksampler(base_model, noise_seed, steps, cfg, sampler_name, scheduler, base_positive, base_negative, input_latent, denoise=denoise, disable_noise=False, start_step=start_at_step, last_step=base_steps, force_full_denoise=True)
# return nodes.common_ksampler(refiner_model, noise_seed + noise_offset, steps, cfg, sampler_name, scheduler, refiner_positive, refiner_negative, base_result[0], denoise=denoise * refiner_strength, disable_noise=False, start_step=base_steps, last_step=steps, force_full_denoise=True)
# SDXL Image2Image Sampler (incl. HiRes Fix)
class SeargeSDXLImage2ImageSamplerV3:
@classmethod
def INPUT_TYPES(s):
return {"required": {
"base_model": ("MODEL",),
"base_positive": ("CONDITIONING", ),
"base_negative": ("CONDITIONING", ),
"refiner_model": ("MODEL",),
"refiner_positive": ("CONDITIONING",),
"refiner_negative": ("CONDITIONING",),
"image": ("IMAGE", ),
"vae": ("VAE",),
"noise_seed": ("INT", {"default": 0, "min": 0, "max": 0xfffffffffffffff0}),
"steps": ("INT", {"default": 20, "min": 0, "max": 200}),
"cfg": ("FLOAT", {"default": 7.0, "min": 0.0, "max": 30.0, "step": 0.5}),
"sampler_name": ("SAMPLER_NAME", {"default": "ddim"}),
"scheduler": ("SCHEDULER_NAME", {"default": "ddim_uniform"}),
"base_ratio": ("FLOAT", {"default": 0.8, "min": 0.0, "max": 1.0, "step": 0.01}),
"denoise": ("FLOAT", {"default": 0.25, "min": 0.0, "max": 1.0, "step": 0.01}),
},
"optional": {
"upscale_model": ("UPSCALE_MODEL",),
"scaled_width": ("INT", {"default": 1536, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
"scaled_height": ("INT", {"default": 1536, "min": 0, "max": nodes.MAX_RESOLUTION, "step": 8}),
# "noise_offset": ("INT", {"default": 1, "min": 0, "max": 1}),
# "refiner_strength": ("FLOAT", {"default": 1.0, "min": 0.1, "max": 1.0, "step": 0.05}),
"softness": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 1.0, "step": 0.05}),
},
}
RETURN_TYPES = ("IMAGE", )
FUNCTION = "sample"
CATEGORY = "Searge/Sampling"
def sample(self, base_model, base_positive, base_negative, refiner_model, refiner_positive, refiner_negative, image, vae, noise_seed, steps, cfg, sampler_name, scheduler, base_ratio, denoise, softness, upscale_model=None, scaled_width=None, scaled_height=None, noise_offset=None, refiner_strength=None):
base_steps = int(steps * (base_ratio + 0.0001))
refiner_steps = max(0, steps - base_steps)
# if noise_offset is None:
# noise_offset = 1
# if refiner_strength is None:
# refiner_strength = 1.0
# if refiner_strength < 0.01:
# refiner_strength = 0.01
if steps < 1:
return (image, )
scaled_image = image
use_upscale_model = upscale_model is not None and softness < 0.9999
if use_upscale_model:
upscale_result = comfy_extras.nodes_upscale_model.ImageUpscaleWithModel().upscale(upscale_model, image)
scaled_image = upscale_result[0]
if scaled_width is not None and scaled_height is not None:
upscale_result = nodes.ImageScale().upscale(scaled_image, "bicubic", scaled_width, scaled_height, "center")
scaled_image = upscale_result[0]
if use_upscale_model and softness > 0.0001:
upscale_result = nodes.ImageScale().upscale(image, "bicubic", scaled_width, scaled_height, "center")
scaled_original = upscale_result[0]
blend_result = comfy_extras.nodes_post_processing.Blend().blend_images(scaled_image, scaled_original, softness, "normal")
scaled_image = blend_result[0]
if denoise < 0.01:
return (scaled_image, )
vae_encode_result = nodes.VAEEncode().encode(vae, scaled_image)
input_latent = vae_encode_result[0]
if base_steps >= steps:
result_latent = nodes.common_ksampler(base_model, noise_seed, steps, cfg, sampler_name, scheduler, base_positive, base_negative, input_latent, denoise=denoise, disable_noise=False, start_step=0, last_step=steps, force_full_denoise=True)
else:
result_latent = sdxl_ksampler(base_model, refiner_model, noise_seed, base_steps, refiner_steps, cfg, sampler_name, scheduler, base_positive, base_negative, refiner_positive, refiner_negative, input_latent, denoise=denoise, disable_noise=False, start_step=0, last_step=steps, force_full_denoise=True)
# base_result = nodes.common_ksampler(base_model, noise_seed, steps, cfg, sampler_name, scheduler, base_positive, base_negative, input_latent, denoise=denoise, disable_noise=False, start_step=0, last_step=base_steps, force_full_denoise=True)
# result_latent = nodes.common_ksampler(refiner_model, noise_seed + noise_offset, steps, cfg, sampler_name, scheduler, refiner_positive, refiner_negative, base_result[0], denoise=denoise * refiner_strength, disable_noise=False, start_step=base_steps, last_step=steps, force_full_denoise=True)
vae_decode_result = nodes.VAEDecode().decode(vae, result_latent[0])
output_image = vae_decode_result[0]
return (output_image, )
modules/stage.py
+70
View File
@@ -0,0 +1,70 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .data_utils import retrieve_parameter
from .mb_pipeline import PipelineAccess
from .names import Names
from .ui import UI
# --------------------------------------------------------------------------------
# Stage: TemplateForNewStages
# --------------------------------------------------------------------------------
class SeargeStage:
def __init__(self):
pass
def get_input(self, data, stage_data):
# if we still don't have stage data,
if stage_data is None and data is not None:
stage_data = {
PipelineAccess.NAME: retrieve_parameter(PipelineAccess.NAME, data),
}
return stage_data
def process(self, data, stage_input):
access = PipelineAccess(stage_input)
example = access.get_active_setting(UI.S_EXAMPLE_STRUCTURE, UI.F_EXAMPLE_FIELD, 4815162342)
example += 1337
example_structure = {
Names.F_EXAMPLE_FIELD: example,
}
if data is not None:
data[Names.S_EXAMPLE_STRUCTURE] = example_structure
stage_output = {
Names.S_EXAMPLE_STRUCTURE: example_structure,
}
return (data, stage_output,)
modules/stage_apply_controlnet.py
+204
View File
@@ -0,0 +1,204 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .data_utils import retrieve_parameter
from .mb_pipeline import PipelineAccess
from .names import Names
from .node_wrapper import NodeWrapper
from .ui import UI
# --------------------------------------------------------------------------------
# Stage: Apply Controlnet
# --------------------------------------------------------------------------------
class SeargeStageApplyControlnet:
def __init__(self):
pass
def get_input(self, data, stage_data):
# if we still don't have stage data,
if stage_data is None and data is not None:
stage_data = {
PipelineAccess.NAME: retrieve_parameter(PipelineAccess.NAME, data),
}
return stage_data
def process(self, data, stage_input):
access = PipelineAccess(stage_input)
stack = access.get_active_setting(UI.S_CONTROLNET_INPUTS, UI.F_CN_STACK, [])
base_cond = access.get_from_pipeline(Names.P_BASE_CONDITIONING)
base_cond_changed = access.changed_in_pipeline(Names.P_BASE_CONDITIONING)
clip_vision_changed = access.changed_in_pipeline(Names.P_CLIP_VISION_MODEL)
canny_changed = access.changed_in_pipeline(Names.P_CN_CANNY_MODEL)
depth_changed = access.changed_in_pipeline(Names.P_CN_DEPTH_MODEL)
recolor_changed = access.changed_in_pipeline(Names.P_CN_RECOLOR_MODEL)
sketch_changed = access.changed_in_pipeline(Names.P_CN_SKETCH_MODEL)
custom_changed = access.changed_in_pipeline(Names.P_CN_CUSTOM_MODEL)
(cn_stack, images_changed) = self.comparable_stack(stack)
any_changes = (
images_changed or
base_cond_changed or
clip_vision_changed or
canny_changed or
depth_changed or
recolor_changed or
sketch_changed or
custom_changed
)
applied_controlnet_changed = access.changed_in_cache(Names.C_APPLIED_CONTROLNET, cn_stack)
if any_changes or applied_controlnet_changed:
(base_positive, base_negative, changed_cond) = self.apply_controlnet(access, stack, base_cond)
access.update_in_cache(Names.C_APPLIED_CONTROLNET, cn_stack, (base_positive, base_negative,
base_cond, changed_cond))
access.update_in_pipeline(Names.P_BASE_CONDITIONING, base_cond)
else:
(base_positive, base_negative, base_cond, changed_cond) = access.get_from_cache(Names.C_APPLIED_CONTROLNET)
if changed_cond:
access.restore_in_pipeline(Names.P_BASE_CONDITIONING, base_cond)
controlnet_output = {
Names.F_CN_BASE_POSITIVE: base_positive,
Names.F_CN_BASE_NEGATIVE: base_negative,
}
if data is not None:
data[Names.S_CONTROLNET_OUTPUT] = controlnet_output
stage_output = {
Names.S_CONTROLNET_OUTPUT: controlnet_output,
}
return (data, stage_output,)
def comparable_stack(self, stack):
new_stack = []
images_changed = False
for controlnet in stack:
entry = controlnet.copy()
if UI.F_REV_CN_IMAGE in entry:
entry.pop(UI.F_REV_CN_IMAGE)
if UI.F_REV_CN_IMAGE_CHANGED in entry:
if entry[UI.F_REV_CN_IMAGE_CHANGED]:
images_changed = True
controlnet[UI.F_REV_CN_IMAGE_CHANGED] = False
entry.pop(UI.F_REV_CN_IMAGE_CHANGED)
new_stack.append(entry)
return (new_stack, images_changed,)
def apply_controlnet(self, access, stack, base_cond):
base_positive = retrieve_parameter(Names.F_BASE_POSITIVE, base_cond)
base_negative = retrieve_parameter(Names.F_BASE_NEGATIVE, base_cond)
changed_cond = False
for controlnet in stack:
mode = retrieve_parameter(UI.F_REV_CN_MODE, controlnet, UI.NONE)
strength = retrieve_parameter(UI.F_REV_CN_STRENGTH, controlnet, 0.0)
cn_image = retrieve_parameter(UI.F_REV_CN_IMAGE, controlnet)
base_positive = retrieve_parameter(Names.F_BASE_POSITIVE, base_cond)
base_negative = retrieve_parameter(Names.F_BASE_NEGATIVE, base_cond)
controlnet_model = None
if mode == UI.NONE:
continue
elif mode == UI.CN_MODE_REVISION:
clipvision_model = access.get_from_pipeline(Names.P_CLIP_VISION_MODEL)
if clipvision_model is not None and cn_image is not None:
clip_vision = NodeWrapper.clipvision_encoder.encode(clipvision_model, cn_image)[0]
else:
clip_vision = None
if clip_vision is not None and base_positive is not None and strength != 0.0:
noise_aug = retrieve_parameter(UI.F_REV_NOISE_AUGMENTATION, controlnet, 0.0)
enhancer = retrieve_parameter(UI.F_REV_ENHANCER, controlnet, False)
base_positive = NodeWrapper.unclip_conditioning.apply_adm(base_positive, clip_vision,
strength, noise_aug)[0]
base_cond[Names.F_BASE_POSITIVE] = base_positive
if base_negative is not None and strength > 0.0 and enhancer:
base_negative = NodeWrapper.unclip_conditioning.apply_adm(base_negative, clip_vision,
-strength, noise_aug)[0]
base_cond[Names.F_BASE_NEGATIVE] = base_negative
changed_cond = True
elif mode == UI.CN_MODE_CANNY:
controlnet_model = access.get_from_pipeline(Names.P_CN_CANNY_MODEL)
elif mode == UI.CN_MODE_DEPTH:
controlnet_model = access.get_from_pipeline(Names.P_CN_DEPTH_MODEL)
elif mode == UI.CN_MODE_RECOLOR:
controlnet_model = access.get_from_pipeline(Names.P_CN_RECOLOR_MODEL)
elif mode == UI.CN_MODE_SKETCH:
controlnet_model = access.get_from_pipeline(Names.P_CN_SKETCH_MODEL)
elif mode == UI.CUSTOM:
controlnet_model = access.get_from_pipeline(Names.P_CN_CUSTOM_MODEL)
if controlnet_model is None:
continue
if cn_image is not None and base_positive is not None and base_negative is not None:
start = retrieve_parameter(UI.F_CN_START, controlnet, 0.0)
end = retrieve_parameter(UI.F_CN_END, controlnet, 1.0)
result = NodeWrapper.controlnet_advanced.apply_controlnet(base_positive, base_negative,
controlnet_model, cn_image, strength,
start, end)
base_positive = result[0]
base_negative = result[1]
base_cond[Names.F_BASE_POSITIVE] = base_positive
base_cond[Names.F_BASE_NEGATIVE] = base_negative
changed_cond = True
return (base_positive, base_negative, changed_cond)
modules/stage_apply_loras.py
+101
View File
@@ -0,0 +1,101 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .data_utils import retrieve_parameter
from .mb_pipeline import PipelineAccess
from .names import Names
from .node_wrapper import NodeWrapper
from .ui import UI
# --------------------------------------------------------------------------------
# Stage: Apply Loras
# --------------------------------------------------------------------------------
class SeargeStageApplyLoras:
def __init__(self):
pass
def get_input(self, data, stage_data):
# if we still don't have stage data,
if stage_data is None and data is not None:
stage_data = {
PipelineAccess.NAME: retrieve_parameter(PipelineAccess.NAME, data),
}
return stage_data
def process(self, data, stage_input):
access = PipelineAccess(stage_input)
base_model = access.get_from_pipeline(Names.P_BASE_MODEL)
base_clip = access.get_from_pipeline(Names.P_BASE_CLIP)
base_model_changed = access.changed_in_pipeline(Names.P_BASE_MODEL)
base_clip_changed = access.changed_in_pipeline(Names.P_BASE_CLIP)
lora_stack = access.get_active_setting(UI.S_LORAS, UI.F_LORA_STACK, [])
any_changes = (
base_model_changed or
base_clip_changed
)
applied_loras = []
loras_changed = access.changed_in_cache(Names.C_APPLIED_LORAS, lora_stack)
if loras_changed or any_changes:
for lora in lora_stack:
lora_name = retrieve_parameter(UI.F_LORA_NAME, lora)
lora_strength = retrieve_parameter(UI.F_LORA_STRENGTH, lora, 0.0)
if lora_name is not None and lora_name != UI.NONE and lora_strength != 0.0:
(base_model, base_clip) = NodeWrapper.lora_loader.load_lora(base_model, base_clip, lora_name,
lora_strength, lora_strength)
applied_loras.append(lora_name)
access.update_in_cache(Names.C_APPLIED_LORAS, lora_stack, (base_model, base_clip))
access.update_in_pipeline(Names.P_BASE_MODEL, base_model)
access.update_in_pipeline(Names.P_BASE_CLIP, base_clip)
else:
(base_model, base_clip) = access.get_from_cache(Names.C_APPLIED_LORAS)
access.restore_in_pipeline(Names.P_BASE_MODEL, base_model)
access.restore_in_pipeline(Names.P_BASE_CLIP, base_clip)
loaded_loras = {
Names.F_LORA_NAMES: applied_loras,
}
if data is not None:
data[Names.S_LOADED_LORAS] = loaded_loras
stage_output = {
Names.S_EXAMPLE_STRUCTURE: loaded_loras,
}
return (data, stage_output,)
modules/stage_clip_conditioning.py
+392
View File
@@ -0,0 +1,392 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .data_utils import retrieve_parameter
from .mb_pipeline import PipelineAccess
from .names import Names
from .node_wrapper import NodeWrapper
from .ui import UI
from .utils import next_multiple_of
# --------------------------------------------------------------------------------
# Stage: Clip Conditioning
# --------------------------------------------------------------------------------
class SeargeStageClipConditioning:
PROMPT_PLACEHOLDER = "<prompt>"
CONDITIONING_ROUNDING = 16
def __init__(self):
pass
def get_input(self, data, stage_data):
# if we still don't have stage data,
if stage_data is None and data is not None:
stage_data = {
PipelineAccess.NAME: retrieve_parameter(PipelineAccess.NAME, data),
}
return stage_data
def process(self, data, stage_input):
access = PipelineAccess(stage_input)
base_clip_changed = access.changed_in_pipeline(Names.P_BASE_CLIP)
refiner_clip_changed = access.changed_in_pipeline(Names.P_REFINER_CLIP)
base_clip = access.get_from_pipeline(Names.P_BASE_CLIP)
refiner_clip = access.get_from_pipeline(Names.P_REFINER_CLIP)
has_refiner_clip = refiner_clip is not None
main_prompt = access.get_active_setting(UI.S_PROMPTS, UI.F_MAIN_PROMPT, "")
secondary_prompt = access.get_active_setting(UI.S_PROMPTS, UI.F_SECONDARY_PROMPT, "")
style_prompt = access.get_active_setting(UI.S_PROMPTS, UI.F_STYLE_PROMPT, "")
neg_main_prompt = access.get_active_setting(UI.S_PROMPTS, UI.F_NEGATIVE_MAIN_PROMPT, "")
neg_secondary_prompt = access.get_active_setting(UI.S_PROMPTS, UI.F_NEGATIVE_SECONDARY_PROMPT, "")
neg_style_prompt = access.get_active_setting(UI.S_PROMPTS, UI.F_NEGATIVE_STYLE_PROMPT, "")
prompting_mode = access.get_active_setting(UI.S_OPERATING_MODE, UI.F_PROMPTING_MODE, UI.PROMPTING_DEFAULT)
image_width = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_IMAGE_WIDTH, 1024)
image_height = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_IMAGE_HEIGHT, 1024)
base_cond_scale = access.get_active_setting(UI.S_CONDITIONING_PARAMETERS, UI.F_BASE_CONDITIONING_SCALE, 1)
refiner_cond_scale = access.get_active_setting(UI.S_CONDITIONING_PARAMETERS, UI.F_REFINER_CONDITIONING_SCALE, 1)
target_cond_scale = access.get_active_setting(UI.S_CONDITIONING_PARAMETERS, UI.F_TARGET_CONDITIONING_SCALE, 1)
pos_cond_scale = access.get_active_setting(UI.S_CONDITIONING_PARAMETERS, UI.F_POSITIVE_CONDITIONING_SCALE, 1.5)
neg_cond_scale = access.get_active_setting(UI.S_CONDITIONING_PARAMETERS, UI.F_NEGATIVE_CONDITIONING_SCALE, 0.75)
pos_ascore = access.get_active_setting(UI.S_CONDITIONING_PARAMETERS, UI.F_POSITIVE_AESTHETIC_SCORE, 6.0)
neg_ascore = access.get_active_setting(UI.S_CONDITIONING_PARAMETERS, UI.F_NEGATIVE_AESTHETIC_SCORE, 2.5)
prompts = [
prompting_mode,
main_prompt,
secondary_prompt,
style_prompt,
neg_main_prompt,
neg_secondary_prompt,
neg_style_prompt,
]
def pack_prompts(processed):
(base_pos_main, base_pos_sec, base_pos_style, base_neg_main, base_neg_sec, base_neg_style,
ref_pos, ref_pos_style, ref_neg, ref_neg_style) = processed
return {
Names.F_BASE_POSITIVE_MAIN_PROMPT: base_pos_main,
Names.F_BASE_POSITIVE_SECONDARY_PROMPT: base_pos_sec,
Names.F_BASE_POSITIVE_STYLE_PROMPT: base_pos_style,
Names.F_BASE_NEGATIVE_MAIN_PROMPT: base_neg_main,
Names.F_BASE_NEGATIVE_SECONDARY_PROMPT: base_neg_sec,
Names.F_BASE_NEGATIVE_STYLE_PROMPT: base_neg_style,
Names.F_REFINER_POSITIVE_PROMPT: ref_pos,
Names.F_REFINER_POSITIVE_STYLE_PROMPT: ref_pos_style,
Names.F_REFINER_NEGATIVE_PROMPT: ref_neg,
Names.F_REFINER_NEGATIVE_STYLE_PROMPT: ref_neg_style,
}
prompts_changed = access.changed_in_cache(Names.C_PROCESSED_PROMPTS, prompts)
if prompts_changed:
empty = ""
if prompting_mode == UI.PROMPTING_DEFAULT:
processed_prompts = self.create_standard_prompts(
main_prompt, secondary_prompt, style_prompt,
neg_main_prompt, neg_secondary_prompt, neg_style_prompt)
elif prompting_mode == UI.PROMPTING_MAIN_AND_NEGATIVE_ONLY:
processed_prompts = self.create_standard_prompts(
main_prompt, main_prompt, empty,
neg_main_prompt, neg_main_prompt, empty)
elif prompting_mode == UI.PROMPTING_MAIN_SECONDARY_AND_NEGATIVE:
processed_prompts = self.create_standard_prompts(
main_prompt, secondary_prompt, empty,
neg_main_prompt, neg_main_prompt, empty)
elif prompting_mode == UI.PROMPTING_MAIN_ALL_EXCEPT_SECONDARY:
processed_prompts = self.create_standard_prompts(
main_prompt, main_prompt, style_prompt,
neg_main_prompt, neg_secondary_prompt, neg_style_prompt)
else:
processed_prompts = self.create_pass_through_prompts(
main_prompt, secondary_prompt, style_prompt,
neg_main_prompt, neg_secondary_prompt, neg_style_prompt)
access.update_in_cache(Names.C_PROCESSED_PROMPTS, prompts, processed_prompts)
access.update_in_pipeline(Names.P_PROCESSED_PROMPTS, pack_prompts(processed_prompts))
else:
processed_prompts = access.get_from_cache(Names.C_PROCESSED_PROMPTS)
access.restore_in_pipeline(Names.P_PROCESSED_PROMPTS, pack_prompts(processed_prompts))
(base_pos_main, base_pos_sec, base_pos_style, base_neg_main, base_neg_sec, base_neg_style,
ref_pos, ref_pos_style, ref_neg, ref_neg_style) = processed_prompts
base_prompts = [
base_cond_scale,
target_cond_scale,
pos_cond_scale,
neg_cond_scale,
base_pos_main,
base_pos_sec,
base_pos_style,
base_neg_main,
base_neg_sec,
base_neg_style,
]
refiner_prompts = [
refiner_cond_scale,
pos_cond_scale,
neg_cond_scale,
pos_ascore,
neg_ascore,
ref_pos,
ref_pos_style,
ref_neg,
ref_neg_style,
]
def pack_base_cond(encoded):
(pos, pos_style, neg, neg_style) = encoded
return {
Names.F_BASE_POSITIVE: pos,
Names.F_BASE_POSITIVE_STYLE: pos_style,
Names.F_BASE_NEGATIVE: neg,
Names.F_BASE_NEGATIVE_STYLE: neg_style,
}
def pack_ref_cond(encoded):
(pos, pos_style, neg, neg_style) = encoded
return {
Names.F_REFINER_POSITIVE: pos,
Names.F_REFINER_POSITIVE_STYLE: pos_style,
Names.F_REFINER_NEGATIVE: neg,
Names.F_REFINER_NEGATIVE_STYLE: neg_style,
}
base_cond_changed = access.changed_in_cache(Names.C_BASE_CONDITIONING, base_prompts)
if base_cond_changed or base_clip_changed:
encoded_base = self.encode_base(base_clip, processed_prompts, image_width, image_height,
base_cond_scale, target_cond_scale, pos_cond_scale, neg_cond_scale)
access.update_in_cache(Names.C_BASE_CONDITIONING, base_prompts, encoded_base)
access.update_in_pipeline(Names.P_BASE_CONDITIONING, pack_base_cond(encoded_base))
else:
encoded_base = access.get_from_cache(Names.C_BASE_CONDITIONING)
access.restore_in_pipeline(Names.P_BASE_CONDITIONING, pack_base_cond(encoded_base))
if has_refiner_clip:
ref_cond_changed = access.changed_in_cache(Names.C_REFINER_CONDITIONING, refiner_prompts)
if ref_cond_changed or refiner_clip_changed:
encoded_ref = self.encode_ref(refiner_clip, processed_prompts, image_width, image_height,
refiner_cond_scale, pos_cond_scale, neg_cond_scale,
pos_ascore, neg_ascore)
access.update_in_cache(Names.C_REFINER_CONDITIONING, refiner_prompts, encoded_ref)
access.update_in_pipeline(Names.P_REFINER_CONDITIONING, pack_ref_cond(encoded_ref))
else:
encoded_ref = access.get_from_cache(Names.C_REFINER_CONDITIONING)
access.restore_in_pipeline(Names.P_REFINER_CONDITIONING, pack_ref_cond(encoded_ref))
else:
encoded_ref = (None, None, None, None,)
if refiner_clip_changed:
access.update_in_pipeline(Names.P_REFINER_CONDITIONING, pack_ref_cond(encoded_ref))
else:
access.restore_in_pipeline(Names.P_REFINER_CONDITIONING, pack_ref_cond(encoded_ref))
(base_positive, base_positive_style, base_negative, base_negative_style) = encoded_base
(refiner_positive, refiner_positive_style, refiner_negative, refiner_negative_style) = encoded_ref
processed_prompts = pack_prompts(processed_prompts)
conditioning = {
Names.F_BASE_POSITIVE: base_positive,
Names.F_BASE_POSITIVE_STYLE: base_positive_style,
Names.F_BASE_NEGATIVE: base_negative,
Names.F_BASE_NEGATIVE_STYLE: base_negative_style,
Names.F_REFINER_POSITIVE: refiner_positive,
Names.F_REFINER_POSITIVE_STYLE: refiner_positive_style,
Names.F_REFINER_NEGATIVE: refiner_negative,
Names.F_REFINER_NEGATIVE_STYLE: refiner_negative_style,
}
if data is not None:
data[Names.S_CONDITIONING] = conditioning
data[Names.S_PROCESSED_PROMPTS] = processed_prompts
stage_output = {
Names.S_CONDITIONING: conditioning,
Names.S_PROCESSED_PROMPTS: processed_prompts,
}
return (data, stage_output,)
def create_standard_prompts(self, main, secondary, pos_style, neg_main, neg_secondary, neg_style):
main = "" if main is None else main
secondary = main if secondary is None else secondary
neg_main = "" if neg_main is None else neg_main
neg_secondary = neg_main if neg_secondary is None else neg_secondary
if pos_style is not None and len(pos_style) > 0:
if pos_style.find(self.PROMPT_PLACEHOLDER) >= 0:
base_pos_main = pos_style.replace(self.PROMPT_PLACEHOLDER, main)
base_pos_sec = pos_style.replace(self.PROMPT_PLACEHOLDER, secondary)
else:
if len(main) > 0:
base_pos_main = main + ". " + pos_style
else:
base_pos_main = pos_style
if len(secondary) > 0:
base_pos_sec = secondary + ". " + pos_style
else:
base_pos_sec = pos_style
else:
base_pos_main = main
base_pos_sec = secondary
base_neg_main = neg_main
base_neg_sec = neg_secondary
if pos_style is not None and len(pos_style) > 0:
if pos_style.find(self.PROMPT_PLACEHOLDER) >= 0:
ref_pos = pos_style.replace(self.PROMPT_PLACEHOLDER, main)
else:
if len(main) > 0:
ref_pos = main + ". " + pos_style
else:
ref_pos = pos_style
else:
ref_pos = main
if len(neg_main) > 0 and len(neg_secondary) > 0:
ref_neg = neg_main + ". " + neg_secondary
elif len(neg_main) > 0:
ref_neg = neg_main
elif len(neg_secondary) > 0:
ref_neg = neg_secondary
else:
ref_neg = ""
base_pos_style = pos_style.replace(self.PROMPT_PLACEHOLDER, "")
base_neg_style = neg_style.replace(self.PROMPT_PLACEHOLDER, "")
ref_pos_style = base_pos_style
ref_neg_style = base_neg_style
return (base_pos_main, base_pos_sec, base_pos_style, base_neg_main, base_neg_sec, base_neg_style,
ref_pos, ref_pos_style, ref_neg, ref_neg_style)
def create_pass_through_prompts(self, main, secondary, style_prompt, neg_main, neg_secondary, neg_style):
base_pos_main = main
base_pos_sec = secondary
base_pos_style = style_prompt
base_neg_main = neg_main
base_neg_sec = neg_secondary
base_neg_style = neg_style
if len(main) > 0 and len(secondary) > 0:
ref_pos = main + ". " + secondary
elif len(main) > 0:
ref_pos = main
elif len(secondary) > 0:
ref_pos = secondary
else:
ref_pos = ""
ref_pos_style = style_prompt
if len(neg_main) > 0 and len(neg_secondary) > 0:
ref_neg = neg_main + ". " + neg_secondary
elif len(neg_main) > 0:
ref_neg = neg_main
elif len(neg_secondary) > 0:
ref_neg = neg_secondary
else:
ref_neg = ""
ref_neg_style = neg_style
return (base_pos_main, base_pos_sec, base_pos_style, base_neg_main, base_neg_sec, base_neg_style,
ref_pos, ref_pos_style, ref_neg, ref_neg_style)
def encode_base(self, base_clip, std_prompts, image_width, image_height, cond_scale=1.0, target_scale=1.0,
pos_scale=1.0, neg_scale=1.0):
encoder = NodeWrapper.sdxl_clip_base_encoder
(pos_main, pos_sec, pos_style, neg_main, neg_sec, neg_style, _, _, _, _) = std_prompts
base_width = next_multiple_of(image_width * cond_scale, self.CONDITIONING_ROUNDING)
base_height = next_multiple_of(image_height * cond_scale, self.CONDITIONING_ROUNDING)
target_width = next_multiple_of(image_width * target_scale, self.CONDITIONING_ROUNDING)
target_height = next_multiple_of(image_height * target_scale, self.CONDITIONING_ROUNDING)
pos_width = next_multiple_of(base_width * pos_scale, self.CONDITIONING_ROUNDING)
pos_height = next_multiple_of(base_height * pos_scale, self.CONDITIONING_ROUNDING)
neg_width = next_multiple_of(base_width * neg_scale, self.CONDITIONING_ROUNDING)
neg_height = next_multiple_of(base_height * neg_scale, self.CONDITIONING_ROUNDING)
base_positive = encoder.encode(base_clip, pos_width, pos_height, 0, 0, target_width, target_height,
pos_main, pos_sec)[0]
base_positive_style = encoder.encode(base_clip, pos_width, pos_height, 0, 0, target_width, target_height,
pos_style, pos_style)[0]
base_negative = encoder.encode(base_clip, neg_width, neg_height, 0, 0, target_width, target_height,
neg_main, neg_sec)[0]
base_negative_style = encoder.encode(base_clip, neg_width, neg_height, 0, 0, target_width, target_height,
neg_style, neg_style)[0]
return (base_positive, base_positive_style, base_negative, base_negative_style)
def encode_ref(self, refiner_clip, std_prompts, image_width, image_height, cond_scale=1.0,
pos_scale=1.0, neg_scale=1.0, pos_ascore=6.0, neg_ascore=2.5):
encoder = NodeWrapper.sdxl_clip_refiner_encoder
(_, _, _, _, _, _, pos, pos_style, neg, neg_style) = std_prompts
refiner_width = next_multiple_of(image_width * cond_scale, self.CONDITIONING_ROUNDING)
refiner_height = next_multiple_of(image_height * cond_scale, self.CONDITIONING_ROUNDING)
pos_width = next_multiple_of(refiner_width * pos_scale, self.CONDITIONING_ROUNDING)
pos_height = next_multiple_of(refiner_height * pos_scale, self.CONDITIONING_ROUNDING)
neg_width = next_multiple_of(refiner_width * neg_scale, self.CONDITIONING_ROUNDING)
neg_height = next_multiple_of(refiner_height * neg_scale, self.CONDITIONING_ROUNDING)
refiner_positive = encoder.encode(refiner_clip, pos_ascore, pos_width, pos_height, pos)[0]
refiner_positive_style = encoder.encode(refiner_clip, pos_ascore, pos_width, pos_height, pos_style)[0]
refiner_negative = encoder.encode(refiner_clip, neg_ascore, neg_width, neg_height, neg)[0]
refiner_negative_style = encoder.encode(refiner_clip, neg_ascore, neg_width, neg_height, neg_style)[0]
return (refiner_positive, refiner_positive_style, refiner_negative, refiner_negative_style)
modules/stage_high_resolution.py
+332
View File
@@ -0,0 +1,332 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .data_utils import retrieve_parameter
from .mb_pipeline import PipelineAccess
from .names import Names
from .node_wrapper import NodeWrapper
from .ui import UI
from .utils import get_image_size
from .utils import next_multiple_of
# --------------------------------------------------------------------------------
# Stage: Sampling
# --------------------------------------------------------------------------------
class SeargeStageHighResolution:
SIZE_MULTIPLE_OF = 8
def __init__(self):
pass
def get_input(self, data, stage_data):
# if we still don't have stage data,
if stage_data is None and data is not None:
stage_data = {
PipelineAccess.NAME: retrieve_parameter(PipelineAccess.NAME, data),
}
return stage_data
def process(self, data, stage_input):
access = PipelineAccess(stage_input)
base_changed = access.changed_in_pipeline(Names.P_BASE_MODEL)
refiner_changed = access.changed_in_pipeline(Names.P_REFINER_MODEL)
base_model = access.get_from_pipeline(Names.P_BASE_MODEL)
refiner_model = access.get_from_pipeline(Names.P_REFINER_MODEL)
has_refiner = refiner_model is not None
vae_changed = access.changed_in_pipeline(Names.P_VAE_MODEL)
vae_model = access.get_from_pipeline(Names.P_VAE_MODEL)
upscaler_changed = access.changed_in_pipeline(Names.P_HIRES_UPSCALER)
upscale_model = access.get_from_pipeline(Names.P_HIRES_UPSCALER)
detail_processor_changed = access.changed_in_pipeline(Names.P_DETAIL_PROCESSOR)
detail_processor = access.get_from_pipeline(Names.P_DETAIL_PROCESSOR)
base_cond_changed = access.changed_in_pipeline(Names.P_BASE_CONDITIONING)
refiner_cond_changed = access.changed_in_pipeline(Names.P_REFINER_CONDITIONING)
base_cond = access.get_from_pipeline(Names.P_BASE_CONDITIONING)
refiner_cond = access.get_from_pipeline(Names.P_REFINER_CONDITIONING)
base_positive = retrieve_parameter(Names.F_BASE_POSITIVE, base_cond)
base_negative = retrieve_parameter(Names.F_BASE_NEGATIVE, base_cond)
refiner_positive = retrieve_parameter(Names.F_REFINER_POSITIVE, refiner_cond)
refiner_negative = retrieve_parameter(Names.F_REFINER_NEGATIVE, refiner_cond)
# for now these are here to prepare for the future addition of latent upscaling
latent_changed = access.changed_in_pipeline(Names.P_LATENT)
latent = access.get_from_pipeline(Names.P_LATENT)
image_changed = access.changed_in_pipeline(Names.P_IMAGE)
image = access.get_from_pipeline(Names.P_IMAGE)
seed = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_SEED, 4815162342)
steps = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_STEPS, 25)
cfg = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_CFG, 7.0)
sampler_name = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_SAMPLER_NAME, "dpmpp_2m")
scheduler = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_SCHEDULER, "karras")
base_ratio = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_BASE_VS_REFINER_RATIO, 0.8)
hires_mode = access.get_active_setting(UI.S_HIGH_RESOLUTION, UI.F_HIRES_MODE, UI.NONE)
hires_mode_changed = access.setting_changed(UI.S_HIGH_RESOLUTION, UI.F_HIRES_MODE)
hires_mode_enabled = hires_mode != UI.NONE
hires_scale = access.get_active_setting(UI.S_HIGH_RESOLUTION, UI.F_HIRES_SCALE, 1.5)
hires_denoise = access.get_active_setting(UI.S_HIGH_RESOLUTION, UI.F_HIRES_DENOISE, 0.2)
hires_softness = access.get_active_setting(UI.S_HIGH_RESOLUTION, UI.F_HIRES_SOFTNESS, 0.5)
hires_detail_boost = access.get_active_setting(UI.S_HIGH_RESOLUTION, UI.F_HIRES_DETAIL_BOOST, 0.0)
if not has_refiner:
refiner_model = None
refiner_positive = None
refiner_negative = None
base_ratio = 1.0
hires_detail_boost = 0.0
def run_sampler(latent, refiner_model, steps, denoise, cfg_method, dynamic_cfg, detail_boost):
sampler = NodeWrapper.sdxl_sampler
latent = sampler(base_model, base_positive, base_negative, latent, seed, steps, cfg,
sampler_name, scheduler, refiner_model=refiner_model,
refiner_positive=refiner_positive, refiner_negative=refiner_negative,
base_ratio=base_ratio, denoise=denoise, cfg_method=cfg_method,
dynamic_base_cfg=dynamic_cfg, dynamic_refiner_cfg=dynamic_cfg,
refiner_detail_boost=detail_boost)
return latent
upscale_factor = 1.0
if hires_scale == UI.HIRES_SCALE_1_25:
upscale_factor = 1.25
if hires_scale == UI.HIRES_SCALE_1_5:
upscale_factor = 1.5
if hires_scale == UI.HIRES_SCALE_2_0:
upscale_factor = 2.0
(image_width, image_height) = get_image_size(image)
new_image_width = next_multiple_of(image_width * upscale_factor, self.SIZE_MULTIPLE_OF)
new_image_height = next_multiple_of(image_height * upscale_factor, self.SIZE_MULTIPLE_OF)
# use this to make sure old cached latents are not kept when the high resolution mode changes
def cleanup_cache():
# do this for all types of upscaled latents that we cache before the sampler at the end
access.remove_from_cache(Names.C_HIRES_LATENT_SIMPLE)
access.remove_from_cache(Names.C_HIRES_LATENT_NORMAL)
larger = upscale_factor > 1.0
if larger and hires_mode == UI.HIRES_MODE_SIMPLE:
parameters = [
hires_mode,
image_width,
image_height,
hires_scale,
upscale_factor,
new_image_width,
new_image_height,
hires_softness,
]
any_changes = (
vae_changed or
image_changed)
hires_latent_changed = access.changed_in_cache(Names.C_HIRES_LATENT_SIMPLE, parameters)
if any_changes or hires_latent_changed:
need_nearest = hires_softness < 0.999
need_bicubic = hires_softness > 0.001
if need_nearest:
nearest = NodeWrapper.image_scale.upscale(image, "nearest-exact", new_image_width,
new_image_height, "center")[0]
else:
nearest = None
if need_bicubic:
bicubic = NodeWrapper.image_scale.upscale(image, "bicubic", new_image_width,
new_image_height, "center")[0]
else:
bicubic = None
if need_nearest and need_bicubic and nearest is not None and bicubic is not None:
softened = NodeWrapper.image_blend.blend_images(nearest, bicubic, hires_softness, "normal")[0]
elif need_nearest and nearest is not None:
softened = nearest
elif need_bicubic and bicubic is not None:
softened = bicubic
else:
softened = None
if softened is not None:
latent = NodeWrapper.vae_encoder.encode(vae_model, softened)[0]
cleanup_cache()
access.update_in_cache(Names.C_HIRES_LATENT_SIMPLE, parameters, latent)
access.update_in_pipeline(Names.P_LATENT, latent)
else:
latent = access.get_from_cache(Names.C_HIRES_LATENT_SIMPLE)
access.restore_in_pipeline(Names.P_LATENT, latent)
elif larger and hires_mode == UI.HIRES_MODE_NORMAL:
parameters = [
hires_mode,
image_width,
image_height,
hires_scale,
upscale_factor,
new_image_width,
new_image_height,
hires_softness,
]
any_changes = (
vae_changed or
upscaler_changed or
detail_processor_changed or
image_changed)
hires_latent_changed = access.changed_in_cache(Names.C_HIRES_LATENT_NORMAL, parameters)
if any_changes or hires_latent_changed:
need_upscaled = hires_softness < 0.999
need_bicubic = hires_softness > 0.001
if upscale_model is not None or detail_processor is not None:
if need_upscaled:
upscaled = image
(scaled_width, scaled_height) = (image_width, image_height)
if upscale_model is not None:
upscaled = NodeWrapper.scale_with_model.upscale(upscale_model, upscaled)[0]
(scaled_width, scaled_height) = get_image_size(upscaled)
if scaled_width != 4 * image_width or scaled_height != 4 * image_height:
print("Warning: high res upscaler should be a 4x ESRGAN model")
if detail_processor is not None:
upscaled = NodeWrapper.scale_with_model.upscale(detail_processor, upscaled)[0]
(detailed_width, detailed_height) = get_image_size(upscaled)
if detailed_width != scaled_width or detailed_height != scaled_height:
print("Warning: detail processor should be a 1x ESRGAN model")
upscaled = NodeWrapper.image_scale.upscale(upscaled, "bicubic", new_image_width,
new_image_height, "center")[0]
else:
upscaled = None
if need_bicubic:
bicubic = NodeWrapper.image_scale.upscale(image, "bicubic", new_image_width,
new_image_height, "center")[0]
else:
bicubic = None
if need_upscaled and need_bicubic and upscaled is not None and bicubic is not None:
softened = NodeWrapper.image_blend.blend_images(upscaled, bicubic, hires_softness, "normal")[0]
elif need_upscaled and upscaled is not None:
softened = upscaled
elif need_bicubic and bicubic is not None:
softened = bicubic
else:
softened = None
if softened is not None:
latent = NodeWrapper.vae_encoder.encode(vae_model, softened)[0]
else:
latent = None
cleanup_cache()
access.update_in_cache(Names.C_HIRES_LATENT_NORMAL, parameters, latent)
access.update_in_pipeline(Names.P_LATENT, latent)
else:
latent = access.get_from_cache(Names.C_HIRES_LATENT_NORMAL)
access.restore_in_pipeline(Names.P_LATENT, latent)
else:
latent = None
if hires_mode_changed:
cleanup_cache()
access.update_in_pipeline(Names.P_LATENT, latent)
else:
access.restore_in_pipeline(Names.P_LATENT, latent)
parameters = [
seed,
steps,
cfg,
sampler_name,
scheduler,
base_ratio,
hires_softness,
hires_denoise,
hires_detail_boost,
]
latent_changed = access.changed_in_pipeline(Names.P_LATENT)
# DON'T DO THIS HERE (keep using the current latent variable): latent = access.get_from_pipeline(Names.P_LATENT)
any_changes = (
base_changed or
refiner_changed or
base_cond_changed or
refiner_cond_changed or
latent_changed)
hires_latent_changed = access.changed_in_cache(Names.C_HIRES_LATENT, parameters)
if any_changes or hires_latent_changed:
if hires_mode_enabled and latent is not None:
if "noise_mask" in latent:
latent = latent.clone()
latent.pop("noise_mask")
hires_steps = int((steps * 2 + 2) // 3)
latent = run_sampler(latent, refiner_model, hires_steps, hires_denoise, cfg_method=None,
dynamic_cfg=0.0, detail_boost=hires_detail_boost)
else:
latent = None
# NOTE: it's important NOT to call the cleanup cache function here, because it's unrelated to this cache
access.update_in_cache(Names.C_HIRES_LATENT, parameters, latent)
access.update_in_pipeline(Names.P_LATENT, latent)
else:
latent = access.get_from_cache(Names.C_HIRES_LATENT)
access.restore_in_pipeline(Names.P_LATENT, latent)
high_res_output = {
Names.F_LATENT_HIRES: latent,
}
if data is not None:
data[Names.S_HIRES_OUTPUT] = high_res_output
stage_output = {
Names.S_HIRES_OUTPUT: high_res_output,
}
return (data, stage_output,)
modules/stage_image_saving.py
+236
View File
@@ -0,0 +1,236 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
import json
import numpy as np
import os
from datetime import datetime
from PIL import Image
from PIL.PngImagePlugin import PngInfo
import folder_paths
from .data_utils import retrieve_parameter
from .mb_pipeline import PipelineAccess
from .names import Names
from .ui import UI
# --------------------------------------------------------------------------------
# Stage: TemplateForNewStages
# --------------------------------------------------------------------------------
class SeargeStageImageSaving:
def __init__(self):
pass
def get_input(self, data, stage_data):
# if we still don't have stage data,
if stage_data is None and data is not None:
stage_data = {
PipelineAccess.NAME: retrieve_parameter(PipelineAccess.NAME, data),
}
return stage_data
def process(self, data, stage_input):
access = PipelineAccess(stage_input)
save_parameters_file = access.get_active_setting(UI.S_IMAGE_SAVING, UI.F_SAVE_PARAMETERS_FILE, False)
save_folder = access.get_active_setting(UI.S_IMAGE_SAVING, UI.F_SAVE_FOLDER, UI.SAVE_TO_OUTPUT_DATE)
save_generated_image = access.get_active_setting(UI.S_IMAGE_SAVING, UI.F_SAVE_GENERATED_IMAGE, True)
embed_wf_in_generated = access.get_active_setting(UI.S_IMAGE_SAVING, UI.F_EMBED_WORKFLOW_IN_GENERATED, True)
generated_image_name = access.get_active_setting(UI.S_IMAGE_SAVING, UI.F_GENERATED_IMAGE_NAME, "generated")
save_high_res_image = access.get_active_setting(UI.S_IMAGE_SAVING, UI.F_SAVE_HIGH_RES_IMAGE, True)
embed_wf_in_high_res = access.get_active_setting(UI.S_IMAGE_SAVING, UI.F_EMBED_WORKFLOW_IN_HIGH_RES, True)
high_res_image_name = access.get_active_setting(UI.S_IMAGE_SAVING, UI.F_HIGH_RES_IMAGE_NAME, "hires")
save_upscaled_image = access.get_active_setting(UI.S_IMAGE_SAVING, UI.F_SAVE_UPSCALED_IMAGE, True)
embed_wf_in_upscaled = access.get_active_setting(UI.S_IMAGE_SAVING, UI.F_EMBED_WORKFLOW_IN_UPSCALED, True)
upscaled_image_name = access.get_active_setting(UI.S_IMAGE_SAVING, UI.F_UPSCALED_IMAGE_NAME, "upscaled")
magic_box_hidden = access.get_from_pipeline(Names.S_MAGIC_BOX_HIDDEN)
hidden_prompt = retrieve_parameter(Names.F_MAGIC_BOX_PROMPT, magic_box_hidden)
hidden_extra_pnginfo = retrieve_parameter(Names.F_MAGIC_BOX_EXTRA_PNGINFO, magic_box_hidden)
# get the images from the data stream instead of the pipeline, this is intentional
vae_decoded_sampled = retrieve_parameter(Names.S_VAE_DECODED_SAMPLED, data)
generated_images = retrieve_parameter(Names.F_DECODED_SAMPLED_IMAGE, vae_decoded_sampled)
post_processed_images = retrieve_parameter(Names.F_SAMPLED_POST_PROCESSED, vae_decoded_sampled)
vae_decoded_hires = retrieve_parameter(Names.S_VAE_DECODED_HIRES, data)
high_res_images = retrieve_parameter(Names.F_DECODED_HIRES_IMAGE, vae_decoded_hires)
post_processed_hires = retrieve_parameter(Names.F_HIRES_POST_PROCESSED, vae_decoded_hires)
upscaling_output = retrieve_parameter(Names.S_UPSCALED, data)
upscaled_images = retrieve_parameter(Names.F_UPSCALED_IMAGE, upscaling_output)
save_to_input = save_folder == UI.SAVE_TO_INPUT
output_folder = folder_paths.get_input_directory() if save_to_input else folder_paths.get_output_directory()
if save_folder == UI.SAVE_TO_OUTPUT:
sub_folder = ""
elif save_folder == UI.SAVE_TO_OUTPUT_DATE:
sub_folder = "%date%"
elif save_folder == UI.SAVE_TO_OUTPUT_SEARGE_SDXL_DATE:
sub_folder = "Searge-SDXL-%date%"
elif save_folder == UI.SAVE_TO_INPUT:
sub_folder = ""
else:
return (data, None,)
sub_folder = sub_folder.replace("%date%", datetime.now().strftime("%Y-%m-%d"))
full_path = os.path.join(output_folder, sub_folder)
try:
files = [fn for fn in os.listdir(full_path) if os.path.isfile(os.path.join(full_path, fn))]
except FileNotFoundError:
os.makedirs(full_path, exist_ok=True)
files = []
num = 0
for filenum in [fn[0:5] for fn in files if fn[5] == '-' and fn[0:5].isnumeric()]:
test = int(filenum)
if test > num:
num = test
num = num + 1
generated_image_path = False
high_res_image_path = False
upscaled_image_path = False
parameter_file_path = False
anything_saved = False
if save_generated_image and generated_images is not None:
generated_image_name = generated_image_name.replace("\\", "_").replace("/", "_").replace(".", "_")
filename = f"{num:05}-{generated_image_name}"
generated_image_path = os.path.join(sub_folder, filename)
images_to_save = generated_images if post_processed_images is None else post_processed_images
self.save_images(images_to_save, full_path, filename, embed_wf_in_generated,
hidden_prompt, hidden_extra_pnginfo)
anything_saved = True
if save_high_res_image and high_res_images is not None:
high_res_image_name = high_res_image_name.replace("\\", "_").replace("/", "_").replace(".", "_")
filename = f"{num:05}-{high_res_image_name}"
high_res_image_path = os.path.join(sub_folder, filename)
images_to_save = high_res_images if post_processed_hires is None else post_processed_hires
self.save_images(images_to_save, full_path, filename, embed_wf_in_high_res,
hidden_prompt, hidden_extra_pnginfo)
anything_saved = True
if save_upscaled_image and upscaled_images is not None:
upscaled_image_name = upscaled_image_name.replace("\\", "_").replace("/", "_").replace(".", "_")
filename = f"{num:05}-{upscaled_image_name}"
upscaled_image_path = os.path.join(sub_folder, filename)
self.save_images(upscaled_images, full_path, filename, embed_wf_in_upscaled,
hidden_prompt, hidden_extra_pnginfo)
anything_saved = True
if save_parameters_file and anything_saved:
filename = f"{num:05}-param.txt"
parameter_file_path = os.path.join(sub_folder, filename)
full_filename = os.path.join(full_path, filename)
parameters = {
Names.S_MAGIC_BOX_VERSION: access.get_from_pipeline(Names.S_MAGIC_BOX_VERSION),
UI.S_PROMPTS: access.get_effective_structure(UI.S_PROMPTS),
UI.S_OPERATING_MODE: access.get_effective_structure(UI.S_OPERATING_MODE),
UI.S_GENERATION_PARAMETERS: access.get_effective_structure(UI.S_GENERATION_PARAMETERS),
UI.S_CONDITIONING_PARAMETERS: access.get_effective_structure(UI.S_CONDITIONING_PARAMETERS),
UI.S_ADVANCED_PARAMETERS: access.get_effective_structure(UI.S_ADVANCED_PARAMETERS),
UI.S_IMG2IMG_INPAINTING: access.get_effective_structure(UI.S_IMG2IMG_INPAINTING),
UI.S_HIGH_RESOLUTION: access.get_effective_structure(UI.S_HIGH_RESOLUTION),
UI.S_CHECKPOINTS: access.get_effective_structure(UI.S_CHECKPOINTS),
UI.S_UPSCALE_MODELS: access.get_effective_structure(UI.S_UPSCALE_MODELS), # TODO
UI.S_LORAS: access.get_effective_structure(UI.S_LORAS), # TODO
UI.S_PROMPT_STYLING: access.get_effective_structure(UI.S_PROMPT_STYLING), # TODO
UI.S_CUSTOM_PROMPTING: access.get_effective_structure(UI.S_CUSTOM_PROMPTING), # TODO
UI.S_CONDITION_MIXING: access.get_effective_structure(UI.S_CONDITION_MIXING), # TODO
"debug_information": {
Names.S_PROCESSED_PROMPTS: retrieve_parameter(Names.S_PROCESSED_PROMPTS, data, {"info": "missing"})
}
}
parameters_json = json.dumps(parameters, indent=4)
with open(full_filename, "w", encoding="utf-8") as f:
f.write(parameters_json)
saved_files = {
Names.F_GENERATED_IMAGE_PATH: generated_image_path,
Names.F_HIGH_RES_IMAGE_PATH: high_res_image_path,
Names.F_UPSCALED_IMAGE_PATH: upscaled_image_path,
Names.F_PARAMETER_FILE_PATH: parameter_file_path,
}
if data is not None:
data[Names.S_SAVED_FILES] = saved_files
stage_output = {
Names.S_SAVED_FILES: saved_files,
}
return (data, stage_output,)
@staticmethod
def save_images(images, full_path, filename, embed_metadata, prompt, extra_pnginfo):
if images is None:
print(f"Warning: trying to save {filename}, but no images were provided")
return
counter = 1
for image in images:
i = 255. * image.cpu().numpy()
img = Image.fromarray(np.clip(i, 0, 255).astype(np.uint8))
metadata = None
if embed_metadata:
metadata = PngInfo()
if prompt is not None:
metadata.add_text("prompt", json.dumps(prompt))
if extra_pnginfo is not None:
for x in extra_pnginfo:
metadata.add_text(x, json.dumps(extra_pnginfo[x]))
file = f"{filename}-{counter}.png" if counter > 1 else f"{filename}.png"
counter = counter + 1
img.save(os.path.join(full_path, file), pnginfo=metadata, compress_level=4)
modules/stage_latent_detailer.py
+147
View File
@@ -0,0 +1,147 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from comfy.sample import prepare_mask
from .data_utils import retrieve_parameter
from .mb_pipeline import PipelineAccess
from .names import Names
from .node_wrapper import NodeWrapper
from .ui import UI
# --------------------------------------------------------------------------------
# Stage: Latent Detailer
# --------------------------------------------------------------------------------
class SeargeStageLatentDetailer:
def __init__(self):
pass
def get_input(self, data, stage_data):
# if we still don't have stage data,
if stage_data is None and data is not None:
stage_data = {
PipelineAccess.NAME: retrieve_parameter(PipelineAccess.NAME, data),
}
return stage_data
def process(self, data, stage_input):
access = PipelineAccess(stage_input)
base_changed = access.changed_in_pipeline(Names.P_BASE_MODEL)
base_model = access.get_from_pipeline(Names.P_BASE_MODEL)
base_cond_changed = access.changed_in_pipeline(Names.P_BASE_CONDITIONING)
base_cond = access.get_from_pipeline(Names.P_BASE_CONDITIONING)
base_positive = retrieve_parameter(Names.F_BASE_POSITIVE, base_cond)
base_negative = retrieve_parameter(Names.F_BASE_NEGATIVE, base_cond)
latent_changed = access.changed_in_pipeline(Names.P_LATENT)
latent = access.get_from_pipeline(Names.P_LATENT)
seed = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_SEED, 4815162342)
cfg = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_CFG, 7.0)
latent_detailer = access.get_active_setting(UI.S_ADVANCED_PARAMETERS, UI.F_LATENT_DETAILER, UI.NONE)
parameters = [
seed,
cfg,
latent_detailer,
]
any_changes = (
base_changed or
base_cond_changed or
latent_changed)
sampled_changed = access.changed_in_cache(Names.C_SAMPLED_DETAILER, parameters)
if any_changes or sampled_changed:
latent_original = latent
if latent_detailer == UI.DETAILER_NORMAL:
latent = self.detailer(latent, 5, "nearest-exact", base_model, base_positive, base_negative, seed, cfg)
elif latent_detailer == UI.DETAILER_SOFT:
latent = self.detailer(latent, 5, "bicubic", base_model, base_positive, base_negative, seed, cfg)
elif latent_detailer == UI.DETAILER_BLURRY:
latent = self.detailer(latent, 10, "nearest-exact", base_model, base_positive, base_negative, seed, cfg)
elif latent_detailer == UI.DETAILER_SOFT_BLURRY:
latent = self.detailer(latent, 10, "bicubic", base_model, base_positive, base_negative, seed, cfg)
if "noise_mask" in latent_original and "samples" in latent_original and "samples" in latent:
old_samples = latent_original["samples"]
new_samples = latent["samples"]
noise_mask = latent_original["noise_mask"]
noise_mask = prepare_mask(noise_mask, old_samples.shape, "cpu")
latent["samples"] = new_samples * noise_mask + old_samples * (1.0 - noise_mask)
access.update_in_cache(Names.C_SAMPLED_DETAILER, parameters, latent)
access.update_in_pipeline(Names.P_LATENT, latent)
else:
latent = access.get_from_cache(Names.C_SAMPLED_DETAILER)
access.restore_in_pipeline(Names.P_LATENT, latent)
detailed_output = {
Names.F_DETAILED_SAMPLED: latent,
}
if data is not None:
data[Names.S_LATENT_DETAILED] = detailed_output
stage_output = {
Names.S_LATENT_DETAILED: detailed_output,
}
return (data, stage_output,)
def detailer(self, latent, percent, method, base_model, base_positive, base_negative, seed, cfg):
sampler = NodeWrapper.common_sampler
scaler = NodeWrapper.latent_upscale_by
sampler_name = "dpmpp_2m"
scheduler = "karras"
latent = scaler.upscale(latent, method, 2.0)[0]
latent = sampler(base_model, seed, 100, cfg, sampler_name, scheduler,
base_positive, base_negative, latent, denoise=1.0, disable_noise=False,
start_step=int(100 - percent * 2), last_step=int(100 - percent),
force_full_denoise=False)
latent = scaler.upscale(latent, method, 0.5)[0]
latent = sampler(base_model, seed, 100, cfg, sampler_name, scheduler,
base_positive, base_negative, latent, denoise=1.0, disable_noise=True,
start_step=int(100 - percent * 2), last_step=100,
force_full_denoise=True)
return latent
modules/stage_latent_inputs.py
+206
View File
@@ -0,0 +1,206 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from ._experimental import gaussian_latent_noise
from .data_utils import retrieve_parameter
from .mb_pipeline import PipelineAccess
from .names import Names
from .node_wrapper import NodeWrapper
from .ui import UI
from .utils import get_image_size
from .utils import get_mask_size
# --------------------------------------------------------------------------------
# Stage: Latent Inputs
# --------------------------------------------------------------------------------
class SeargeStageLatentInputs:
def __init__(self):
pass
def get_input(self, data, stage_data):
# if we still don't have stage data,
if stage_data is None and data is not None:
stage_data = {
PipelineAccess.NAME: retrieve_parameter(PipelineAccess.NAME, data),
}
return stage_data
def process(self, data, stage_input):
access = PipelineAccess(stage_input)
vae_changed = access.changed_in_pipeline(Names.P_VAE_MODEL)
vae_model = access.get_from_pipeline(Names.P_VAE_MODEL)
image_width = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_IMAGE_WIDTH, 1024)
image_height = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_IMAGE_HEIGHT, 1024)
seed = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_SEED, 4815162342)
workflow_mode = access.get_active_setting(UI.S_OPERATING_MODE, UI.F_WORKFLOW_MODE, UI.WF_MODE_TEXT_TO_IMAGE)
image_changed = access.changed_in_pipeline(Names.P_IMAGE)
mask_changed = access.changed_in_pipeline(Names.P_MASK)
image = access.get_from_pipeline(Names.P_IMAGE)
mask = access.get_from_pipeline(Names.P_MASK)
batch_size = access.get_active_setting(UI.S_OPERATING_MODE, UI.F_BATCH_SIZE, 1)
precondition_mode = access.get_active_setting(UI.S_CONDITIONING_PARAMETERS, UI.F_PRECONDITION_MODE, UI.NONE)
precondition_strength = access.get_active_setting(UI.S_CONDITIONING_PARAMETERS, UI.F_PRECONDITION_STRENGTH, 0.1)
latent = None
if workflow_mode == UI.WF_MODE_IMAGE_TO_IMAGE or workflow_mode == UI.WF_MODE_IN_PAINTING:
parameters = [
image_width,
image_height,
batch_size,
workflow_mode,
]
latent_changed = access.changed_in_cache(Names.C_LATENT_FROM_IMAGE, parameters)
if latent_changed or image_changed or vae_changed:
(width, height) = get_image_size(image)
if width != image_width or height != image_height:
image = NodeWrapper.image_scale.upscale(image, "bicubic", image_width, image_height, "center")[0]
access.update_in_pipeline(Names.P_IMAGE, image)
latent = NodeWrapper.vae_encoder.encode(vae_model, image)[0]
if batch_size > 1:
# NOTE: only repeat with batch size here if we are not in inpainting mode (optimization)
if workflow_mode == UI.WF_MODE_IMAGE_TO_IMAGE:
latent = NodeWrapper.latent_repeater.repeat(latent, batch_size)[0]
image = image.repeat(batch_size, 1, 1, 1)
access.update_in_pipeline(Names.P_IMAGE, image)
access.remove_from_cache(Names.C_EMPTY_LATENT)
access.remove_from_cache(Names.C_IMAGE_MASK)
access.remove_from_cache(Names.C_LATENT_WITH_MASK)
access.update_in_cache(Names.C_LATENT_FROM_IMAGE, parameters, (latent, image))
access.update_in_pipeline(Names.P_LATENT, latent)
else:
(latent, image) = access.get_from_cache(Names.C_LATENT_FROM_IMAGE)
access.restore_in_pipeline(Names.P_LATENT, latent)
access.restore_in_pipeline(Names.P_IMAGE, image)
if workflow_mode == UI.WF_MODE_IN_PAINTING:
parameters = [
image_width,
image_height,
batch_size,
workflow_mode,
]
image_mask_changed = access.changed_in_cache(Names.C_IMAGE_MASK, parameters)
if mask_changed or image_mask_changed:
(width, height) = get_mask_size(mask)
if width != image_width or height != image_height:
image_scale = NodeWrapper.image_scale
mask_image = NodeWrapper.mask_to_image.mask_to_image(mask)[0]
mask_image = image_scale.upscale(mask_image, "bicubic", image_width, image_height, "center")[0]
mask = NodeWrapper.image_to_mask.image_to_mask(mask_image, "green")[0]
access.remove_from_cache(Names.C_EMPTY_LATENT)
access.remove_from_cache(Names.C_LATENT_WITH_MASK)
access.update_in_cache(Names.C_IMAGE_MASK, parameters, mask)
access.update_in_pipeline(Names.P_MASK, mask)
else:
mask = access.get_from_cache(Names.C_IMAGE_MASK)
access.restore_in_pipeline(Names.P_MASK, mask)
latent_changed = access.changed_in_pipeline(Names.P_LATENT)
mask_changed = access.changed_in_pipeline(Names.P_MASK)
if latent_changed or mask_changed:
latent = access.get_from_pipeline(Names.P_LATENT)
# in case we are using an older cached latent that was already repeated with batch size, take only first
latent = NodeWrapper.latent_selector.frombatch(latent, 0, 1)[0]
latent = NodeWrapper.set_latent_mask.set_mask(latent, mask)[0]
# repeat with batch size, will also repeat the mask in addition to the latent
if batch_size > 1:
latent = NodeWrapper.latent_repeater.repeat(latent, batch_size)[0]
access.remove_from_cache(Names.C_EMPTY_LATENT)
access.update_in_cache(Names.C_LATENT_WITH_MASK, parameters, latent)
access.update_in_pipeline(Names.P_LATENT, latent)
else:
latent = access.get_from_cache(Names.C_LATENT_WITH_MASK)
access.restore_in_pipeline(Names.P_LATENT, latent)
if workflow_mode == UI.WF_MODE_TEXT_TO_IMAGE:
parameters = [
seed,
image_width,
image_height,
batch_size,
precondition_mode,
precondition_strength,
]
empty_latent_changed = access.changed_in_cache(Names.C_EMPTY_LATENT, parameters)
if empty_latent_changed:
if precondition_mode == UI.NONE or precondition_strength < 0.001:
latent = NodeWrapper.empty_latent.generate(image_width, image_height, batch_size)[0]
elif precondition_mode == UI.PRECONDITION_MODE_GAUSSIAN:
latent = gaussian_latent_noise(image_width // 8, image_height // 8, seed, precondition_strength,
batch_size)
else:
latent = NodeWrapper.empty_latent.generate(image_width, image_height, batch_size)[0]
access.remove_from_cache(Names.C_LATENT_FROM_IMAGE)
access.remove_from_cache(Names.C_IMAGE_MASK)
access.remove_from_cache(Names.C_LATENT_WITH_MASK)
access.update_in_cache(Names.C_EMPTY_LATENT, parameters, latent)
access.update_in_pipeline(Names.P_LATENT, latent)
else:
latent = access.get_from_cache(Names.C_EMPTY_LATENT)
access.restore_in_pipeline(Names.P_LATENT, latent)
latent_input = {
Names.F_LATENT_IMAGE: latent,
}
if data is not None:
data[Names.S_LATENT_INPUTS] = latent_input
stage_output = {
Names.S_LATENT_INPUTS: latent_input,
}
return (data, stage_output,)
modules/stage_load_checkpoints.py
+330
View File
@@ -0,0 +1,330 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .data_utils import retrieve_parameter
from .mb_pipeline import PipelineAccess
from .names import Names
from .node_wrapper import NodeWrapper
from .ui import UI
# --------------------------------------------------------------------------------
# Stage: Load Checkpoints
# --------------------------------------------------------------------------------
class SeargeStageLoadCheckpoints:
def __init__(self):
pass
def get_input(self, data, stage_data):
# if we still don't have stage data,
if stage_data is None and data is not None:
stage_data = {
PipelineAccess.NAME: retrieve_parameter(PipelineAccess.NAME, data),
}
return stage_data
def process(self, data, stage_input):
access = PipelineAccess(stage_input)
# TODO: this stage will always execute, even if the pipeline is disabled and in that case unload all models
if not access.is_pipeline_enabled():
pass
base_changed = access.setting_changed(UI.S_CHECKPOINTS, UI.F_BASE_CHECKPOINT)
if base_changed:
base_name = access.get_active_setting(UI.S_CHECKPOINTS, UI.F_BASE_CHECKPOINT)
base_checkpoint = NodeWrapper.checkpoint_loader.load_checkpoint(base_name)
access.update_in_cache(Names.C_BASE_CHECKPOINT, [base_name], base_checkpoint)
else:
base_checkpoint = access.get_from_cache(Names.C_BASE_CHECKPOINT)
base_model = base_checkpoint[0]
base_clip = base_checkpoint[1]
base_vae = base_checkpoint[2]
if base_changed:
access.update_in_pipeline(Names.P_BASE_MODEL, base_model)
access.update_in_pipeline(Names.P_BASE_CLIP, base_clip)
access.update_in_pipeline(Names.P_BASE_VAE, base_vae)
else:
access.restore_in_pipeline(Names.P_BASE_MODEL, base_model)
access.restore_in_pipeline(Names.P_BASE_CLIP, base_clip)
access.restore_in_pipeline(Names.P_BASE_VAE, base_vae)
refiner_changed = access.setting_changed(UI.S_CHECKPOINTS, UI.F_REFINER_CHECKPOINT)
if refiner_changed:
refiner_name = access.get_active_setting(UI.S_CHECKPOINTS, UI.F_REFINER_CHECKPOINT, UI.NONE)
if refiner_name == UI.NONE:
refiner_checkpoint = (None, None, None,)
else:
refiner_checkpoint = NodeWrapper.checkpoint_loader.load_checkpoint(refiner_name)
access.update_in_cache(Names.C_REFINER_CHECKPOINT, [refiner_name], refiner_checkpoint)
else:
refiner_checkpoint = access.get_from_cache(Names.C_REFINER_CHECKPOINT)
refiner_model = refiner_checkpoint[0]
refiner_clip = refiner_checkpoint[1]
refiner_vae = refiner_checkpoint[2]
if refiner_changed:
access.update_in_pipeline(Names.P_REFINER_MODEL, refiner_model)
access.update_in_pipeline(Names.P_REFINER_CLIP, refiner_clip)
access.update_in_pipeline(Names.P_REFINER_VAE, refiner_vae)
else:
access.restore_in_pipeline(Names.P_REFINER_MODEL, refiner_model)
access.restore_in_pipeline(Names.P_REFINER_CLIP, refiner_clip)
access.restore_in_pipeline(Names.P_REFINER_VAE, refiner_vae)
vae_changed = access.setting_changed(UI.S_CHECKPOINTS, UI.F_VAE_CHECKPOINT)
if vae_changed:
vae_name = access.get_active_setting(UI.S_CHECKPOINTS, UI.F_VAE_CHECKPOINT, UI.VAE_FROM_BASE_MODEL)
if vae_name == UI.VAE_FROM_REFINER_MODEL:
if refiner_vae is not None:
vae_checkpoint = refiner_vae
else:
vae_checkpoint = base_vae
elif vae_name == UI.VAE_FROM_BASE_MODEL:
vae_checkpoint = base_vae
else:
vae_checkpoint = NodeWrapper.vae_loader.load_vae(vae_name)[0]
access.update_in_cache(Names.C_VAE_CHECKPOINT, [vae_name], vae_checkpoint)
access.update_in_pipeline(Names.P_VAE_MODEL, vae_checkpoint)
else:
vae_checkpoint = access.get_from_cache(Names.C_VAE_CHECKPOINT)
access.restore_in_pipeline(Names.P_VAE_MODEL, vae_checkpoint)
vae_model = vae_checkpoint
hires_upscaler_changed = access.setting_changed(UI.S_UPSCALE_MODELS, UI.F_HIGH_RES_UPSCALER)
if hires_upscaler_changed:
hires_upscaler_name = access.get_active_setting(UI.S_UPSCALE_MODELS, UI.F_HIGH_RES_UPSCALER, UI.NONE)
if hires_upscaler_name != UI.NONE:
hires_upscaler_model = NodeWrapper.upscale_loader.load_model(hires_upscaler_name)[0]
else:
hires_upscaler_model = None
access.update_in_cache(Names.C_HIRES_UPSCALE_MODEL, [hires_upscaler_name], hires_upscaler_model)
access.update_in_pipeline(Names.P_HIRES_UPSCALER, hires_upscaler_model)
else:
hires_upscaler_model = access.get_from_cache(Names.C_HIRES_UPSCALE_MODEL)
access.restore_in_pipeline(Names.P_HIRES_UPSCALER, hires_upscaler_model)
hires_upscaler = hires_upscaler_model
primary_upscaler_changed = access.setting_changed(UI.S_UPSCALE_MODELS, UI.F_PRIMARY_UPSCALER)
if primary_upscaler_changed:
primary_upscaler_name = access.get_active_setting(UI.S_UPSCALE_MODELS, UI.F_PRIMARY_UPSCALER, UI.NONE)
if primary_upscaler_name != UI.NONE:
primary_upscaler_model = NodeWrapper.upscale_loader.load_model(primary_upscaler_name)[0]
else:
primary_upscaler_model = None
access.update_in_cache(Names.C_PRIMARY_UPSCALE_MODEL, [primary_upscaler_name], primary_upscaler_model)
access.update_in_pipeline(Names.P_PRIMARY_UPSCALER, primary_upscaler_model)
else:
primary_upscaler_model = access.get_from_cache(Names.C_PRIMARY_UPSCALE_MODEL)
access.restore_in_pipeline(Names.P_PRIMARY_UPSCALER, primary_upscaler_model)
primary_upscaler = primary_upscaler_model
secondary_upscaler_changed = access.setting_changed(UI.S_UPSCALE_MODELS, UI.F_SECONDARY_UPSCALER)
if secondary_upscaler_changed:
secondary_upscaler_name = access.get_active_setting(UI.S_UPSCALE_MODELS, UI.F_SECONDARY_UPSCALER, UI.NONE)
if secondary_upscaler_name != UI.NONE:
secondary_upscaler_model = NodeWrapper.upscale_loader.load_model(secondary_upscaler_name)[0]
else:
secondary_upscaler_model = None
access.update_in_cache(Names.C_SECONDARY_UPSCALE_MODEL, [secondary_upscaler_name], secondary_upscaler_model)
access.update_in_pipeline(Names.P_SECONDARY_UPSCALER, secondary_upscaler_model)
else:
secondary_upscaler_model = access.get_from_cache(Names.C_SECONDARY_UPSCALE_MODEL)
access.restore_in_pipeline(Names.P_SECONDARY_UPSCALER, secondary_upscaler_model)
secondary_upscaler = secondary_upscaler_model
detail_processor_changed = access.setting_changed(UI.S_UPSCALE_MODELS, UI.F_DETAIL_PROCESSOR)
if detail_processor_changed:
detail_processor_name = access.get_active_setting(UI.S_UPSCALE_MODELS, UI.F_DETAIL_PROCESSOR, UI.NONE)
if detail_processor_name != UI.NONE:
detail_processor_model = NodeWrapper.upscale_loader.load_model(detail_processor_name)[0]
else:
detail_processor_model = None
access.update_in_cache(Names.C_DETAIL_PROCESSOR_MODEL, [detail_processor_name], detail_processor_model)
access.update_in_pipeline(Names.P_DETAIL_PROCESSOR, detail_processor_model)
else:
detail_processor_model = access.get_from_cache(Names.C_DETAIL_PROCESSOR_MODEL)
access.restore_in_pipeline(Names.P_DETAIL_PROCESSOR, detail_processor_model)
detail_processor = detail_processor_model
clip_vision_checkpoint_changed = access.setting_changed(UI.S_CONTROLNET_MODELS, UI.F_CLIP_VISION_CHECKPOINT)
if clip_vision_checkpoint_changed:
clip_vision_checkpoint_name = access.get_active_setting(UI.S_CONTROLNET_MODELS, UI.F_CLIP_VISION_CHECKPOINT, UI.NONE)
if clip_vision_checkpoint_name != UI.NONE:
clip_vision_model = NodeWrapper.clipvision_loader.load_clip(clip_vision_checkpoint_name)[0]
else:
clip_vision_model = None
access.update_in_cache(Names.C_CLIP_VISION_MODEL, [clip_vision_checkpoint_name], clip_vision_model)
access.update_in_pipeline(Names.P_CLIP_VISION_MODEL, clip_vision_model)
else:
clip_vision_model = access.get_from_cache(Names.C_CLIP_VISION_MODEL)
access.restore_in_pipeline(Names.P_CLIP_VISION_MODEL, clip_vision_model)
clip_vision = clip_vision_model
canny_checkpoint_changed = access.setting_changed(UI.S_CONTROLNET_MODELS, UI.F_CANNY_CHECKPOINT)
if canny_checkpoint_changed:
canny_checkpoint_name = access.get_active_setting(UI.S_CONTROLNET_MODELS, UI.F_CANNY_CHECKPOINT, UI.NONE)
if canny_checkpoint_name != UI.NONE:
canny_model = NodeWrapper.controlnet_loader.load_controlnet(canny_checkpoint_name)[0]
else:
canny_model = None
access.update_in_cache(Names.C_CN_CANNY_MODEL, [canny_checkpoint_name], canny_model)
access.update_in_pipeline(Names.P_CN_CANNY_MODEL, canny_model)
else:
canny_model = access.get_from_cache(Names.C_CN_CANNY_MODEL)
access.restore_in_pipeline(Names.P_CN_CANNY_MODEL, canny_model)
cn_canny = canny_model
depth_checkpoint_changed = access.setting_changed(UI.S_CONTROLNET_MODELS, UI.F_DEPTH_CHECKPOINT)
if depth_checkpoint_changed:
depth_checkpoint_name = access.get_active_setting(UI.S_CONTROLNET_MODELS, UI.F_DEPTH_CHECKPOINT, UI.NONE)
if depth_checkpoint_name != UI.NONE:
depth_model = NodeWrapper.controlnet_loader.load_controlnet(depth_checkpoint_name)[0]
else:
depth_model = None
access.update_in_cache(Names.C_CN_DEPTH_MODEL, [depth_checkpoint_name], depth_model)
access.update_in_pipeline(Names.P_CN_DEPTH_MODEL, depth_model)
else:
depth_model = access.get_from_cache(Names.C_CN_DEPTH_MODEL)
access.restore_in_pipeline(Names.P_CN_DEPTH_MODEL, depth_model)
cn_depth = depth_model
recolor_checkpoint_changed = access.setting_changed(UI.S_CONTROLNET_MODELS, UI.F_RECOLOR_CHECKPOINT)
if recolor_checkpoint_changed:
recolor_checkpoint_name = access.get_active_setting(UI.S_CONTROLNET_MODELS, UI.F_RECOLOR_CHECKPOINT, UI.NONE)
if recolor_checkpoint_name != UI.NONE:
recolor_model = NodeWrapper.controlnet_loader.load_controlnet(recolor_checkpoint_name)[0]
else:
recolor_model = None
access.update_in_cache(Names.C_CN_RECOLOR_MODEL, [recolor_checkpoint_name], recolor_model)
access.update_in_pipeline(Names.P_CN_RECOLOR_MODEL, recolor_model)
else:
recolor_model = access.get_from_cache(Names.C_CN_RECOLOR_MODEL)
access.restore_in_pipeline(Names.P_CN_RECOLOR_MODEL, recolor_model)
cn_recolor = recolor_model
sketch_checkpoint_changed = access.setting_changed(UI.S_CONTROLNET_MODELS, UI.F_SKETCH_CHECKPOINT)
if sketch_checkpoint_changed:
sketch_checkpoint_name = access.get_active_setting(UI.S_CONTROLNET_MODELS, UI.F_SKETCH_CHECKPOINT, UI.NONE)
if sketch_checkpoint_name != UI.NONE:
sketch_model = NodeWrapper.controlnet_loader.load_controlnet(sketch_checkpoint_name)[0]
else:
sketch_model = None
access.update_in_cache(Names.C_CN_SKETCH_MODEL, [sketch_checkpoint_name], sketch_model)
access.update_in_pipeline(Names.P_CN_SKETCH_MODEL, sketch_model)
else:
sketch_model = access.get_from_cache(Names.C_CN_SKETCH_MODEL)
access.restore_in_pipeline(Names.P_CN_SKETCH_MODEL, sketch_model)
cn_sketch = sketch_model
custom_checkpoint_changed = access.setting_changed(UI.S_CONTROLNET_MODELS, UI.F_CUSTOM_CHECKPOINT)
if custom_checkpoint_changed:
custom_checkpoint_name = access.get_active_setting(UI.S_CONTROLNET_MODELS, UI.F_CUSTOM_CHECKPOINT, UI.NONE)
if custom_checkpoint_name != UI.NONE:
custom_model = NodeWrapper.controlnet_loader.load_controlnet(custom_checkpoint_name)[0]
else:
custom_model = None
access.update_in_cache(Names.C_CN_CUSTOM_MODEL, [custom_checkpoint_name], custom_model)
access.update_in_pipeline(Names.P_CN_CUSTOM_MODEL, custom_model)
else:
custom_model = access.get_from_cache(Names.C_CN_CUSTOM_MODEL)
access.restore_in_pipeline(Names.P_CN_CUSTOM_MODEL, custom_model)
cn_custom = custom_model
loaded_models = {
Names.F_BASE_MODEL: base_model,
Names.F_BASE_CLIP: base_clip,
Names.F_BASE_VAE: base_vae,
Names.F_REFINER_MODEL: refiner_model,
Names.F_REFINER_CLIP: refiner_clip,
Names.F_REFINER_VAE: refiner_vae,
Names.F_VAE_MODEL: vae_model,
Names.F_HIRES_UPSCALER: hires_upscaler,
Names.F_PRIMARY_UPSCALER: primary_upscaler,
Names.F_SECONDARY_UPSCALER: secondary_upscaler,
Names.F_DETAIL_PROCESSOR: detail_processor,
Names.F_CLIP_VISION_MODEL: clip_vision,
Names.F_CN_CANNY_MODEL: cn_canny,
Names.F_CN_DEPTH_MODEL: cn_depth,
Names.F_CN_RECOLOR_MODEL: cn_recolor,
Names.F_CN_SKETCH_MODEL: cn_sketch,
Names.F_CN_CUSTOM_MODEL: cn_custom,
}
if data is not None:
data[Names.S_LOADED_MODELS] = loaded_models
stage_output = {
Names.S_LOADED_MODELS: loaded_models,
}
return (data, stage_output,)
modules/stage_pre_processing.py
+196
View File
@@ -0,0 +1,196 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .data_utils import retrieve_input
from .data_utils import retrieve_parameter
from .mb_pipeline import PipelineAccess
from .names import Names
from .node_wrapper import NodeWrapper
from .ui import UI
from .utils import get_image_size
# --------------------------------------------------------------------------------
# Stage: Pre Process Data
# --------------------------------------------------------------------------------
class SeargePreProcessData:
def __init__(self):
self.UI_OUTPUT_KEYS = None
self.STAGE_OUTPUT_KEYS = None
def get_input(self, data, stage_data):
# if we still don't have stage data,
if stage_data is None and data is not None:
stage_data = {
PipelineAccess.NAME: retrieve_parameter(PipelineAccess.NAME, data),
UI.S_IMAGE_INPUTS: retrieve_parameter(UI.S_IMAGE_INPUTS, data),
}
return stage_data
def process(self, data, stage_input):
access = PipelineAccess(stage_input)
denoise = access.get_active_setting(UI.S_IMG2IMG_INPAINTING, UI.F_DENOISE, 0.5)
workflow_mode = access.get_active_setting(UI.S_OPERATING_MODE, UI.F_WORKFLOW_MODE, UI.WF_MODE_TEXT_TO_IMAGE)
if denoise is not None and workflow_mode == UI.WF_MODE_TEXT_TO_IMAGE:
denoise = 1.0
access.override_setting(UI.S_IMG2IMG_INPAINTING, UI.F_DENOISE, denoise)
size_preset = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_IMAGE_SIZE_PRESET, UI.USE_SETTINGS)
need_image = (
workflow_mode == UI.WF_MODE_IMAGE_TO_IMAGE or
workflow_mode == UI.WF_MODE_IN_PAINTING or
size_preset == UI.RESOLUTION_FROM_IMAGE
)
if need_image:
image_inputs = retrieve_input(UI.S_IMAGE_INPUTS, data, stage_input)
image_changed = retrieve_parameter(UI.F_SOURCE_IMAGE_CHANGED, image_inputs, False)
image_inputs[UI.F_SOURCE_IMAGE_CHANGED] = False
if image_changed:
image = retrieve_parameter(UI.F_SOURCE_IMAGE, image_inputs)
access.update_in_cache(Names.C_SOURCE_IMAGE, [], image)
access.update_in_pipeline(Names.P_IMAGE, image)
else:
image = access.get_from_cache(Names.C_SOURCE_IMAGE)
access.restore_in_pipeline(Names.P_IMAGE, image)
if size_preset == UI.RESOLUTION_1024x1024:
(image_width, image_height) = (1024, 1024)
elif size_preset == UI.RESOLUTION_1152x896:
(image_width, image_height) = (1152, 896)
elif size_preset == UI.RESOLUTION_1216x832:
(image_width, image_height) = (1216, 832)
elif size_preset == UI.RESOLUTION_1344x768:
(image_width, image_height) = (1344, 768)
elif size_preset == UI.RESOLUTION_1536x640:
(image_width, image_height) = (1536, 640)
elif size_preset == UI.RESOLUTION_896x1152:
(image_width, image_height) = (896, 1152)
elif size_preset == UI.RESOLUTION_832x1216:
(image_width, image_height) = (832, 1216)
elif size_preset == UI.RESOLUTION_768x1344:
(image_width, image_height) = (768, 1344)
elif size_preset == UI.RESOLUTION_640x1536:
(image_width, image_height) = (640, 1536)
else:
(image_width, image_height) = (
access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_IMAGE_WIDTH, 1024),
access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_IMAGE_HEIGHT, 1024),
)
image_changed = access.changed_in_pipeline(Names.P_IMAGE)
changed_in_cache = access.changed_in_cache(Names.C_IMAGE_SIZE, [size_preset])
any_changes = (
image_changed or
changed_in_cache
)
if size_preset == UI.RESOLUTION_FROM_IMAGE:
if any_changes:
image = access.get_from_pipeline(Names.P_IMAGE)
if image is not None:
(image_width, image_height) = get_image_size(image)
access.update_in_cache(Names.C_IMAGE_SIZE, [size_preset], (image_width, image_height))
elif access.has_in_cache(Names.C_IMAGE_SIZE):
(image_width, image_height) = access.get_from_cache(Names.C_IMAGE_SIZE)
access.override_setting(UI.S_GENERATION_PARAMETERS, UI.F_IMAGE_WIDTH, image_width)
access.override_setting(UI.S_GENERATION_PARAMETERS, UI.F_IMAGE_HEIGHT, image_height)
mask_mode = access.get_active_setting(UI.S_IMG2IMG_INPAINTING, UI.F_INPAINT_MASK_MODE)
mask_mode_changed = access.setting_changed(UI.S_IMG2IMG_INPAINTING, UI.F_INPAINT_MASK_MODE)
if workflow_mode == UI.WF_MODE_IN_PAINTING:
image_inputs = retrieve_input(UI.S_IMAGE_INPUTS, data, stage_input)
if mask_mode == UI.MASK_MODE_UPLOADED_FULL:
mask_changed = retrieve_parameter(UI.F_UPLOADED_MASK_CHANGED, image_inputs, False)
image_inputs[UI.F_UPLOADED_MASK_CHANGED] = False
else:
mask_changed = retrieve_parameter(UI.F_IMAGE_MASK_CHANGED, image_inputs, False)
image_inputs[UI.F_IMAGE_MASK_CHANGED] = False
if mask_changed or mask_mode_changed:
if mask_mode == UI.MASK_MODE_UPLOADED_FULL:
mask = retrieve_parameter(UI.F_UPLOADED_MASK, image_inputs)
else:
mask = retrieve_parameter(UI.F_IMAGE_MASK, image_inputs)
access.update_in_cache(Names.C_SOURCE_MASK, [], mask)
access.update_in_pipeline(Names.P_MASK, mask)
else:
mask = access.get_from_cache(Names.C_SOURCE_MASK)
access.restore_in_pipeline(Names.P_MASK, mask)
mask_blur = access.get_active_setting(UI.S_IMG2IMG_INPAINTING, UI.F_INPAINT_MASK_BLUR, 8)
parameters = [
mask_blur,
mask_mode,
]
mask_changed = access.changed_in_pipeline(Names.P_MASK)
changed_in_cache = access.changed_in_cache(Names.C_BLURRY_MASK, parameters)
any_changes = (
mask_changed or
changed_in_cache
)
if any_changes:
mask = access.get_from_pipeline(Names.P_MASK)
if mask is not None and mask_blur > 0 and changed_in_cache:
mask = NodeWrapper.mask_to_image.mask_to_image(mask)[0]
mask = NodeWrapper.image_blur.blur(mask, mask_blur, 3.0)[0]
mask = NodeWrapper.image_to_mask.image_to_mask(mask, "green")[0]
access.update_in_cache(Names.C_BLURRY_MASK, parameters, mask)
access.update_in_pipeline(Names.P_MASK, mask)
elif access.has_in_cache(Names.C_BLURRY_MASK):
mask = access.get_from_cache(Names.C_BLURRY_MASK)
access.restore_in_pipeline(Names.P_MASK, mask)
stage_results = {
}
stage_output = {
Names.PLACEHOLDER: stage_results,
}
return (data, stage_output,)
modules/stage_sampling.py
+150
View File
@@ -0,0 +1,150 @@
"""
Custom nodes for SDXL in ComfyUI
MIT License
Copyright (c) 2023 Searge
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from .data_utils import retrieve_parameter
from .mb_pipeline import PipelineAccess
from .names import Names
from .node_wrapper import NodeWrapper
from .ui import UI
# --------------------------------------------------------------------------------
# Stage: Sampling
# --------------------------------------------------------------------------------
class SeargeStageSampling:
def __init__(self):
pass
def get_input(self, data, stage_data):
# if we still don't have stage data,
if stage_data is None and data is not None:
stage_data = {
PipelineAccess.NAME: retrieve_parameter(PipelineAccess.NAME, data),
}
return stage_data
def process(self, data, stage_input):
access = PipelineAccess(stage_input)
base_changed = access.changed_in_pipeline(Names.P_BASE_MODEL)
refiner_changed = access.changed_in_pipeline(Names.P_REFINER_MODEL)
base_model = access.get_from_pipeline(Names.P_BASE_MODEL)
refiner_model = access.get_from_pipeline(Names.P_REFINER_MODEL)
has_refiner = refiner_model is not None
base_cond_changed = access.changed_in_pipeline(Names.P_BASE_CONDITIONING)
refiner_cond_changed = access.changed_in_pipeline(Names.P_REFINER_CONDITIONING)
base_cond = access.get_from_pipeline(Names.P_BASE_CONDITIONING)
refiner_cond = access.get_from_pipeline(Names.P_REFINER_CONDITIONING)
base_positive = retrieve_parameter(Names.F_BASE_POSITIVE, base_cond)
base_negative = retrieve_parameter(Names.F_BASE_NEGATIVE, base_cond)
refiner_positive = retrieve_parameter(Names.F_REFINER_POSITIVE, refiner_cond)
refiner_negative = retrieve_parameter(Names.F_REFINER_NEGATIVE, refiner_cond)
latent_changed = access.changed_in_pipeline(Names.P_LATENT)
latent = access.get_from_pipeline(Names.P_LATENT)
seed = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_SEED, 4815162342)
steps = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_STEPS, 25)
cfg = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_CFG, 7.0)
sampler_name = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_SAMPLER_NAME, "dpmpp_2m")
scheduler = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_SCHEDULER, "karras")
base_ratio = access.get_active_setting(UI.S_GENERATION_PARAMETERS, UI.F_BASE_VS_REFINER_RATIO, 0.8)
denoise = access.get_active_setting(UI.S_IMG2IMG_INPAINTING, UI.F_DENOISE, 0.5)
cfg_method = access.get_active_setting(UI.S_ADVANCED_PARAMETERS, UI.F_DYNAMIC_CFG_METHOD)
dynamic_cfg = access.get_active_setting(UI.S_ADVANCED_PARAMETERS, UI.F_DYNAMIC_CFG_FACTOR, 0.0)
refiner_detail_boost = access.get_active_setting(UI.S_ADVANCED_PARAMETERS, UI.F_REFINER_DETAIL_BOOST)
dynamic_base_cfg = dynamic_cfg
dynamic_refiner_cfg = dynamic_cfg
if not has_refiner:
refiner_model = None
refiner_positive = None
refiner_negative = None
base_ratio = 1.0
dynamic_refiner_cfg = 0.0
refiner_detail_boost = None
parameters = [
has_refiner,
seed,
steps,
cfg,
sampler_name,
scheduler,
base_ratio,
denoise,
cfg_method,
dynamic_base_cfg,
dynamic_refiner_cfg,
refiner_detail_boost,
]
any_changes = (
base_changed or
refiner_changed or
base_cond_changed or
refiner_cond_changed or
latent_changed)
sampled_changed = access.changed_in_cache(Names.C_SAMPLED, parameters)
if any_changes or sampled_changed:
sampler = NodeWrapper.sdxl_sampler
latent = sampler(base_model, base_positive, base_negative, latent, seed, steps, cfg,
sampler_name, scheduler, refiner_model=refiner_model,
refiner_positive=refiner_positive, refiner_negative=refiner_negative,
base_ratio=base_ratio, denoise=denoise, cfg_method=cfg_method,
dynamic_base_cfg=dynamic_base_cfg, dynamic_refiner_cfg=dynamic_refiner_cfg,
refiner_detail_boost=refiner_detail_boost)
access.update_in_cache(Names.C_SAMPLED, parameters, latent)
access.update_in_pipeline(Names.P_LATENT, latent)
else:
latent = access.get_from_cache(Names.C_SAMPLED)
access.restore_in_pipeline(Names.P_LATENT, latent)
sampled_image = {
Names.F_LATENT_SAMPLED: latent,
}
if data is not None:
data[Names.S_SAMPLED_IMAGE] = sampled_image
stage_output = {
Names.S_SAMPLED_IMAGE: sampled_image,
}
return (data, stage_output,)

Some files were not shown because too many files have changed in this diff Show More