From 3a534e7ce3c5eb45d9861ef029d464835cf05b8d Mon Sep 17 00:00:00 2001 From: ControlAltAI <6172927+gseth@users.noreply.github.com> Date: Fri, 13 Dec 2024 10:18:38 +0530 Subject: [PATCH 1/6] Update README.md --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 5230502..3d6552d 100644 --- a/README.md +++ b/README.md @@ -217,7 +217,11 @@ The node pipeline is as follows: Region Mask Generator --> Region Mask Processor ![ComfyUI Screenshot](https://gseth.com/images/SNAG-4957.png)
-**Xformers & Token Limits:** The pipeline here needs VRAM to be stored in the same GPU while Flux, LoRA, and Dual Clip are loaded. Due to this, XFormers is required to optimize VRAM Usage. The total tensor size is approximately 5000x5000 for a 24GB VRAM GPU. This limit is set on purpose for consumer-grade hardware and will be increased with the release of a 5090 GPU. Due to this tensor size limitation, token size is also limited when using Dual Clip. Single Clip is not recommended. Use only Dual Clip. When Base Plus all 3 Regions are connected, a max token for each clip l and t5xxl should be 80 or less. 160 Tokens when combined. In other words, you can use (160 x 4 = 640) tokens in total before running into a tensor size error. Reduce prompt tokens when you get a tensor size error. This system was implemented to avoid any OOM errors (out-of-memory errors) on a 24GB VRAM GPU. All testing has been done on Flux 1.dev model with FP8 and t5xxl with FP 16 without any OOM on a 4090. +**Xformers & Token/Attention Limits:** The pipeline uses an advanced attention mechanism that combines text tokens from your prompts with spatial information from defined regions. As you increase prompt length or add multiple, complex regions, you create larger attention matrices. While xFormers helps optimize memory usage, there is still a practical limit on how many tokens and spatial positions the model can handle without causing dimension or shape alignment errors. + +Example Error: 'Invalid shape for attention bias: torch.Size([1, 24, 5264, 5264]) (expected (1, 24, 5118, 5118))' + +This limit isn’t about a fixed “5,000 x 5,000” size or a strict VRAM cap. Instead, it’s determined by the model’s architecture and how tokens are combined with spatial positions. Extremely long prompts or too many intricate regions can produce attention shapes that the model’s code cannot process, resulting in shape mismatch errors rather than running out of memory. If you encounter these errors, try shortening your prompt or reducing the complexity of your regional conditioning. There isn’t a simple formula linking VRAM size directly to token count. Instead, it’s about balancing your prompt length and region definitions to keep the attention mechanism within workable limits. Testing with the Flux model and T5-XXL in FP16 on a 4090 shows that keeping prompts relatively short and regions manageable helps avoid such issues. **GGUF & CivitAI fine-tune models:** The Flux Region Pipeline was tested with GGUF models without issues. Third-party CivitAI Copax Timeless XPlus 3 Flux models also worked without problems. From 5edcde1d1ee24b633010d54b22924ebec31cbb3e Mon Sep 17 00:00:00 2001 From: ControlAltAI <6172927+gseth@users.noreply.github.com> Date: Fri, 13 Dec 2024 10:22:15 +0530 Subject: [PATCH 2/6] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 3d6552d..6107495 100644 --- a/README.md +++ b/README.md @@ -221,7 +221,7 @@ The node pipeline is as follows: Region Mask Generator --> Region Mask Processor Example Error: 'Invalid shape for attention bias: torch.Size([1, 24, 5264, 5264]) (expected (1, 24, 5118, 5118))' -This limit isn’t about a fixed “5,000 x 5,000” size or a strict VRAM cap. Instead, it’s determined by the model’s architecture and how tokens are combined with spatial positions. Extremely long prompts or too many intricate regions can produce attention shapes that the model’s code cannot process, resulting in shape mismatch errors rather than running out of memory. If you encounter these errors, try shortening your prompt or reducing the complexity of your regional conditioning. There isn’t a simple formula linking VRAM size directly to token count. Instead, it’s about balancing your prompt length and region definitions to keep the attention mechanism within workable limits. Testing with the Flux model and T5-XXL in FP16 on a 4090 shows that keeping prompts relatively short and regions manageable helps avoid such issues. +This limit isn’t about a fixed “5,000 x 5,000” size or a strict VRAM cap. Instead, it’s determined by the model’s architecture and how tokens are combined with spatial positions. Extremely long prompts or too many intricate regions can produce attention shapes that the model’s code cannot process, resulting in shape mismatch errors rather than running out of memory. If you encounter these errors, try shortening your prompt or reducing the complexity of your regional conditioning. There isn’t a simple formula linking VRAM size directly to token count. Instead, it’s about balancing your prompt length and region definitions to keep the attention mechanism within workable limits. Testing with the Flux model and T5-XXL in FP16 on a 4090 shows that keeping prompts relatively short (each clip under 80 tokens) and regions manageable helps avoid such issues. **GGUF & CivitAI fine-tune models:** The Flux Region Pipeline was tested with GGUF models without issues. Third-party CivitAI Copax Timeless XPlus 3 Flux models also worked without problems. From e19abda3f3d5b2c106d68cb535538c29a28a2adf Mon Sep 17 00:00:00 2001 From: ControlAltAI <6172927+gseth@users.noreply.github.com> Date: Wed, 18 Dec 2024 12:49:22 +0530 Subject: [PATCH 3/6] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6107495..e161e0f 100644 --- a/README.md +++ b/README.md @@ -265,7 +265,7 @@ Color Manipulation: Base Conditioning (ignored) + 2 Regions ![ComfyUI Screenshot](https://gseth.com/images/region_control_15.png) ![ComfyUI Screenshot](https://gseth.com/images/region_control_16.png) -**YouTube tutorial Flux Region Usage: Coming Soon** +**YouTube tutorial Flux Region Usage: Flux Region Spatial Control Tutorial** ### YouTube ComfyUI Tutorials From 47e882d61e5ba54873aa6287c0ae09eb42f83191 Mon Sep 17 00:00:00 2001 From: ControlAltAI <6172927+gseth@users.noreply.github.com> Date: Wed, 18 Dec 2024 15:02:51 +0530 Subject: [PATCH 4/6] Update README.md --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index e161e0f..3317cf7 100644 --- a/README.md +++ b/README.md @@ -278,6 +278,10 @@ Black Forest Labs, a pioneering AI research organization, has developed the Flux For more details on these models, their capabilities, and licensing information, you can visit the Black Forest Labs website +### Flux Regional Spatial Control Acknowledgment + +Inspired from: Flux Region Attention by Attashe + ### License This project is licensed under the MIT License. From ce7559609becc8423855d664ec19d1cd3737ddab Mon Sep 17 00:00:00 2001 From: ControlAltAI <6172927+gseth@users.noreply.github.com> Date: Fri, 20 Dec 2024 09:36:18 +0530 Subject: [PATCH 5/6] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 3317cf7..c5c9a47 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -### Requirements Update: 8 Dec 2024: Flux Attention Control Node requires XFormers. Check your version of PyTorch and install a compatible version of XFormers. Please follow the instructions here: xformers_instructions +### Update: 20 Dec 2024: Due to the addition of attention mask in Flux and change of environment here: https://github.com/comfyanonymous/ComfyUI/pull/5942, Flux Region Spatial Control code is now redundant and will only work with older versions of comfy UI. This will not work in ComfyUI v0.3.8-4-g3cacd3f, ComfyUI_frontend v1.5.19 and above. # ComfyUI ControlAltAI Nodes From fb91a72e949a7b331b386b74216e42342665e70c Mon Sep 17 00:00:00 2001 From: ControlAltAI <6172927+gseth@users.noreply.github.com> Date: Sat, 21 Dec 2024 02:46:34 +0530 Subject: [PATCH 6/6] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c5c9a47..3317cf7 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -### Update: 20 Dec 2024: Due to the addition of attention mask in Flux and change of environment here: https://github.com/comfyanonymous/ComfyUI/pull/5942, Flux Region Spatial Control code is now redundant and will only work with older versions of comfy UI. This will not work in ComfyUI v0.3.8-4-g3cacd3f, ComfyUI_frontend v1.5.19 and above. +### Requirements Update: 8 Dec 2024: Flux Attention Control Node requires XFormers. Check your version of PyTorch and install a compatible version of XFormers. Please follow the instructions here: xformers_instructions # ComfyUI ControlAltAI Nodes