updated docs on node limitations with persisting CLIP models
This commit is contained in:
mkim87404
@@ -7,13 +7,13 @@
|
||||
|
||||
[](https://github.com/user-attachments/assets/792c727e-ad57-488f-b827-7e5ceccee32e)
|
||||
|
||||
`🔷 Control Order & Free Memory` is a dependency-free ComfyUI custom node for controlling the execution order of nodes by routing any type/number of data through it, ensuring all input connections resolve first before being passed to outputs. Optionally free VRAM & RAM at any point in a workflow using device-agnostic & robust memory management utilities managed by ComfyUI that safely unload all models (except any models passing though this node) and release as much VRAM & RAM as possible, while preserving all connected passthrough data (e.g. latents, conditioning, images, masks, models, etc.) through to the next node. This node will also print a descriptive message on the ComfyUI session terminal showing how much VRAM & RAM has been freed.
|
||||
`🔷 Control Order & Free Memory` is a dependency-free ComfyUI custom node for controlling the execution order of nodes by routing any type/number of data through it, ensuring all input connections resolve first before being passed to outputs. Optionally free VRAM & RAM at any point in a workflow using device-agnostic & robust memory management utilities managed by ComfyUI that safely unload all models (except any live models passing though this node) and release as much VRAM & RAM as possible, while preserving all connected passthrough data (e.g. latents, conditioning, images, masks, models, etc.) through to the next node. This node will also print a descriptive message on the ComfyUI session terminal showing how much VRAM & RAM has been freed.
|
||||
|
||||
All input & output slots are AnyType (*). They can hook onto any node, including loader nodes like "Load Model", "Load VAE", "Load CLIP", etc. that usually don't have any connectable input slots, making linear connection/chaining of all node types possible - enforcing a single, sequential & deterministic flow of execution. This node adaptively expands its input & output slots to support infinitely many passthrough data (persist_any_N), passing every input to their corresponding "N"th output slot unchanged.
|
||||
|
||||
The memory clean up + linear passthrough of data facilitated by this node can help to **speed up workflows** involving multiple models (e.g. CLIP, VAE, Diffusion Model, etc.) that compete for the same VRAM/RAM but don't all need to be loaded at the same time, by separating each model's operation into its own segment and linearly chaining them together with multiple instances of this node, where only the necessary outputs of each segment (e.g. latents, conditioning, images, models, etc.) are passed on to the next segment while unloading all/some models that are no longer needed in the workflow.
|
||||
|
||||
This becomes especially effective in freeing significant amount of VRAM/RAM between multiple KSamplers (e.g. Wan 2.2 High & Low KSamplers), or before & after VAE Encode / VAE Decode / Load CLIP / Load Model / etc. that allow more of the models' layers to be loaded onto the GPU VRAM and less offloaded to the slower System RAM, resulting in faster overall inference times and letting the user push to higher resolution media outputs before hitting OOM. I have personally seen great reductions in total execution time of my workflows using this node.
|
||||
This becomes especially effective in freeing VRAM/RAM between multiple KSamplers (e.g. Wan 2.2 High & Low KSamplers), or before & after VAE Encode / VAE Decode / Load CLIP / Load Model / etc. that allow more of the models' layers to be loaded onto the GPU VRAM and less offloaded to the slower System RAM, resulting in faster overall inference times and letting the user push to higher resolution media outputs before hitting OOM. I have personally seen great reductions in total execution time of my workflows using this node.
|
||||
|
||||
## 🛠️ Installation
|
||||
|
||||
@@ -41,7 +41,7 @@ In the `example_workflows/` folder of this repo, I have included example Wan 2.2
|
||||
|
||||
- **Control execution order & free memory:**
|
||||
|
||||
The `🔷 Control Order & Free Memory` node ensures that all of the input-connected nodes finish executing first before the output-connected nodes start executing. Hence to control the execution order of nodes, you can split your workflow into multiple groups of nodes with differing execution priorities and connect all nodes that should execute first to the input slots of this node, and all nodes that should execute later to the output slots of this node, ensuring that the correct data is being passed from the "persist_any_N" input slot to its corresponding "N"th output slot. You can then chain additional groups of nodes with as many `🔷 Control Order & Free Memory` nodes as you need, until all groups of nodes are connected together to form a single, sequential flow of execution from start to finish. Optionally, toggle "free_memory" ON on this node to unload all models and free as much VRAM & RAM as possible at the node's position, while preserving all the passthrough data. You can connect as many passthrough data as you need through this node, and each of them can be a different data type (e.g. latents, conditioning, images, models, etc.). If any of the passthrough data are models, they will stay loaded and survive the "free_memory" operation. Also, when "free_memory" is ON, this node will print a descriptive message on the ComfyUI session terminal where you can check how much VRAM & RAM has been freed.
|
||||
The `🔷 Control Order & Free Memory` node ensures that all of the input-connected nodes finish executing first before the output-connected nodes start executing. Hence to control the execution order of nodes, you can split your workflow into multiple groups of nodes with differing execution priorities and connect all nodes that should execute first to the input slots of this node, and all nodes that should execute later to the output slots of this node, ensuring that the correct data is being passed from the "persist_any_N" input slot to its corresponding "N"th output slot. You can then chain additional groups of nodes with as many `🔷 Control Order & Free Memory` nodes as you need, until all groups of nodes are connected together to form a single, sequential flow of execution from start to finish. Optionally, toggle "free_memory" ON on this node to unload all models and free as much VRAM & RAM as possible at the node's position, while preserving all the passthrough data. You can connect as many passthrough data as you need through this node, and each of them can be a different data type (e.g. latents, conditioning, images, models, etc.). If any of the passthrough data are models and they were not already unloaded by the sender nodes, they will stay loaded and survive the "free_memory" operation (Please refer to [Limitations](#limitations) for more details). Also, when "free_memory" is ON, this node will print a descriptive message on the ComfyUI session terminal where you can check how much VRAM & RAM has been freed.
|
||||
|
||||
- **Connecting this node into loader-type nodes:**
|
||||
|
||||
@@ -61,13 +61,15 @@ In the `example_workflows/` folder of this repo, I have included example Wan 2.2
|
||||
- Adaptively expands input & output slots to support infinitely many dynamic passthrough data of any type (*).
|
||||
- Linear passthrough design that focuses on one group of related nodes at a time and routes only necessary outputs to the next group of nodes helps with "cable management", tidying up the connections that can get messy as the workflow grows large, making them easier to understand and maintain.
|
||||
|
||||
## ℹ️ Notes
|
||||
## ℹ️ Limitations
|
||||
|
||||
- Please note that this node is built upon the current ComfyUI implementation of the "comfy.model_management.py" module for memory management, and ComfyUI extension hooks & prototype hijacking patterns for making dynamic updates to the node's UI elements. Hence, it is possible that some features of the node may stop working in future updates of ComfyUI. For more information, you may find these Docs useful:
|
||||
- [ComfyUI model management](https://github.com/Comfy-Org/ComfyUI/blob/master/comfy/model_management.py)
|
||||
- [ComfyUI custom extension hooks](https://docs.comfy.org/custom-nodes/js/javascript_hooks)
|
||||
- [LiteGraph.js](https://tamats.com/projects/litegraph/doc/classes/LGraphNode.html)
|
||||
- You may notice that system RAM usage often does not drop as significantly as GPU VRAM does after this node's "free_memory" operation runs. This is expected behavior in the current implementation of ComfyUI, as ComfyUI intentionally caches the output of every executed node in RAM (e.g. latents, images, conditionings, tensors, etc.) with strong references that survive any py garbage collection invoked from inside the custom node's logic, and only releases those references after the entire "prompt execution" finishes (see [execution.py](https://github.com/Comfy-Org/ComfyUI/blob/master/execution.py), [comfy_execution/caching.py](https://github.com/Comfy-Org/ComfyUI/blob/master/comfy_execution/caching.py)). This is an intentional optimization design by ComfyUI so that downstream nodes can reuse upstream outputs instantly without re-computation. The "free_memory" operation of this custom node already does the maximum possible cleanup with (model unloading + explicit gc.collect() + double device cache clearing before and after the garbage collection to collect as much lingering tensors as possible, etc.). Hence, any RAM that remains allocated after this node runs is ComfyUI's deliberate output caching, which is best to leave up to ComfyUI to manage & optimize behind the scenes across multiple workflow runs.
|
||||
- CLIP models (and other text encoders) behave differently from diffusion models. In most workflows, ComfyUI automatically unloads the CLIP model immediately after the conditioning nodes (CLIPTextEncode, CLIPTextEncodeFlux, etc.) finish (refer to https://github.com/Comfy-Org/ComfyUI/blob/master/comfy/model_management.py). Hence, if you route the CLIP model through this node, the node will often receive an already-unloaded “husk” object, and passing this object to a later conditioning node can cause malformed conditioning outputs or OutOfMemory errors when you feed them into KSamplers. This is an inherent behaviour of the `Load CLIP (Comfy Core)` node which my custom node (that executes later in the graph) cannot prevent. As a workaround, do not rely on passing the CLIP model itself through this node. Instead, you could generate the conditioning immediately after loading CLIP, and only pass the resulting conditioning outputs through to persist_any_N. If you need the CLIP model later, you can always reload it with a new `Load CLIP (Comfy Core)` node in the subsequent chains. Diffusion models, VAEs, and most other large models managed by ComfyUI do persist correctly when routed through this node.
|
||||
- GGUF / Quantized Models: Some GGUF loaders have custom unloading logic that may not always respect the keep_loaded feature of this node's free_memory operation, hence may drop itself before entering the node much like how CLIP models behave. But again, this can be mitigated by loading the model back in on the subsequent chains whenever you need them.
|
||||
|
||||
## ❤️ Inspired By
|
||||
|
||||
|
||||
@@ -22,12 +22,12 @@ class ControlOrderFreeMemory:
|
||||
"required": {
|
||||
"free_memory": ("BOOLEAN", {
|
||||
"default": False,
|
||||
"tooltip": "Unload all models and release as much VRAM & RAM as possible while routing & preserving all 'persist_any' passthrough data. Any models passed into 'persist_any' will stay loaded. Prints how much VRAM & RAM has been freed on the ComfyUI session terminal.",
|
||||
"tooltip": "Unload all models and release as much VRAM & RAM as possible while routing & preserving all 'persist_any' passthrough data. Any models passed into 'persist_any' will stay loaded if they were not already unloaded by the sender nodes (e.g. CLIP / some GGUF loaders). Prints how much VRAM & RAM has been freed on the ComfyUI session terminal.",
|
||||
}),
|
||||
},
|
||||
"optional": {
|
||||
"persist_any_1": (anyType, {
|
||||
"tooltip": "Persist any type of data through to the next node e.g. models, latents, conditioning, images, masks, etc. This data survives the 'free_memory' operation. I/O slots expand adaptively",
|
||||
"tooltip": "Persist any type of data through to the next node e.g. latents, conditioning, images, masks, models (except CLIP / some GGUF models already unloaded by the sender nodes), etc. This data survives the 'free_memory' operation. I/O slots expand adaptively",
|
||||
}),
|
||||
}
|
||||
}
|
||||
@@ -39,14 +39,14 @@ class ControlOrderFreeMemory:
|
||||
OUTPUT_NODE = True # Let this be a hybrid node that can both be in the middle or at the end of a workflow (with no outputs connected).
|
||||
RETURN_TYPES = MatryoshkaTuple((anyType, )) # Let this node's output slot cardinality grow dynamically from JS hooks, while ComfyUI type validation check believes this is just a tuple of 1 "AnyType" element.
|
||||
RETURN_NAMES = MatryoshkaTuple(("persist_any_1", )) # Same trick
|
||||
OUTPUT_TOOLTIPS = MatryoshkaTuple(("Persist any type of data through to the next node e.g. models, latents, conditioning, images, masks, etc. This data survives the 'free_memory' operation. I/O slots expand adaptively", ))
|
||||
OUTPUT_TOOLTIPS = MatryoshkaTuple(("Persist any type of data through to the next node e.g. latents, conditioning, images, masks, models (except CLIP / some GGUF models already unloaded by the sender nodes), etc. This data survives the 'free_memory' operation. I/O slots expand adaptively", ))
|
||||
FUNCTION = "passthrough"
|
||||
CATEGORY = "Control Order & Free Memory"
|
||||
DESCRIPTION = """Control the execution order of nodes by routing any data (as many as you need - I/O slots expand adaptively) through this node. Ensures all input-connected nodes finish executing first before the output-connected nodes start executing.
|
||||
|
||||
All input & output slots are AnyType (*). They can hook onto any node, including loader-type nodes like "Load Model", "Load VAE", "Load CLIP", etc. For connecting into loader-type nodes, you can use the "📁 Filename Selector" helper node (already installed with this custom node) to select & feed the filename into the input "persist_any_N" slot that corresponds to the output "persist_any_N" slot that is connecting into the loader-type node.
|
||||
|
||||
Optionally unload all models and release as much VRAM & RAM as possible while routing & preserving all 'persist_any' passthrough data. Any models passed into 'persist_any' will stay loaded. This node will also print how much VRAM & RAM has been freed on the ComfyUI session terminal.
|
||||
Optionally unload all models and release as much VRAM & RAM as possible while routing & preserving all 'persist_any' passthrough data. Any models passed into 'persist_any' will stay loaded if they were not already unloaded by the sender nodes (e.g. Load CLIP / some GGUF loaders, etc.) This node will also print how much VRAM & RAM has been freed on the ComfyUI session terminal.
|
||||
"""
|
||||
|
||||
def passthrough(self, **kwargs):
|
||||
@@ -85,7 +85,7 @@ Optionally unload all models and release as much VRAM & RAM as possible while ro
|
||||
# Take snapshot of RAM before cleanup
|
||||
initial_ram = psutil.virtual_memory().used # This is fully cross-platform & included in all ComfyUI installations.
|
||||
|
||||
# If any of the inputs were models, keep them loaded
|
||||
# If any of the inputs were models (and were not already unloaded by the sender nodes, e.g. CLIP / some GGUF loaders), keep them loaded
|
||||
loaded_models = model_management.loaded_models() # returns a list of the actual model objects currently tracked by ComfyUI in current_loaded_models
|
||||
keep_loaded = []
|
||||
for key in input_keys:
|
||||
|
||||
Reference in New Issue
Block a user