Tiled Diffusion & VAE for ComfyUI

Check out the SD-WebUI extension for more information.

This extension enables large image drawing & upscaling with limited VRAM via the following techniques:

1. SOTA diffusion tiling algorithms: Mixture of Diffusers , MultiDiffusion , and SpotDiffusion
2. pkuliyi2015 & Kahsolt's Tiled VAE algorithm.
3. pkuliyi2015 & Kahsolt's TIled Noise Inversion for better upscaling.

Note

Sizes/dimensions are in pixels and then converted to latent-space sizes.

Features

• Model support
  • SD1.x, SD2.x, SDXL, SD3
  • Stable Cascade
  • FLUX
• ControlNet support
• StableSR support
• Tiled Noise Inversion
• Tiled VAE
• Regional Prompt Control
• Img2img upscale
• Ultra-Large image generation

Tiled Diffusion

Tip

• Set tile_overlap to 0 and denoise to 1 to see the tile seams and then adjust the options to your needs.
• Increase tile_batch_size to increase speed (if your machine can handle it).
• Use the colorfix node if your colors look off.

Options

Name	Description
method	Tiling strategy.
tile_width	Tile's width
tile_height	Tile's height
tile_overlap	Tile's overlap
tile_batch_size	The number of tiles to process in a batch

How can I specify the tiles' arrangement?

If you have the Math Expression node (or something similar), you can use that to pass in the latent that's passed in your KSampler and divide the tile_height/tile_width by the number of rows/columns you want.

C = number of columns you want
R = number of rows you want

pixel width of input image or latent // C = tile_width
pixel height of input image or latent // R = tile_height

SpotDiffusion

A fast tiling algortihm that eliminates seams by shifting the denoise window per timestep.

To get the most out of SpotDiffusion, you'll want nice, clean tiles with no "overlaps". Overlaps are treated as another tile. Set tile_overlap to 0 and adjust tile_width and tile_height so your input image/latent is nicely divisible by it. That should result in faster gens.

Tiled VAE

The recommended tile sizes are given upon the creation of the node based on the available VRAM.

Note

Enabling fast for the decoder may produce images with slightly higher contrast and brightness.

Options

Name	Description
tile_size	The image is split into tiles, which are then padded with 11/32 pixels' in the decoder/encoder.
fast	When Fast Mode is disabled: The original VAE forward is decomposed into a task queue and a task worker, which starts to process each tile. When GroupNorm is needed, it suspends, stores current GroupNorm mean and var, send everything to RAM, and turns to the next tile. After all GroupNorm means and vars are summarized, it applies group norm to tiles and continues. A zigzag execution order is used to reduce unnecessary data transfer. When Fast Mode is enabled: The original input is downsampled and passed to a separate task queue. Its group norm parameters are recorded and used by all tiles' task queues. Each tile is separately processed without any RAM-VRAM data transfer. After all tiles are processed, tiles are written to a result buffer and returned.
color_fix	Only estimate GroupNorm before downsampling, i.e., run in a semi-fast mode. Only for the encoder. Can restore colors if tiles are too small.

Workflows

The following images can be loaded in ComfyUI.

Simple upscale.

4x upscale. 3 passes.

Citation

@article{jimenez2023mixtureofdiffusers,
  title={Mixture of Diffusers for scene composition and high resolution image generation},
  author={Álvaro Barbero Jiménez},
  journal={arXiv preprint arXiv:2302.02412},
  year={2023}
}

@article{bar2023multidiffusion,
  title={MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation},
  author={Bar-Tal, Omer and Yariv, Lior and Lipman, Yaron and Dekel, Tali},
  journal={arXiv preprint arXiv:2302.08113},
  year={2023}
}