LoRA Power-Merger ComfyUI

Advanced LoRA merging for ComfyUI with Mergekit integration, supporting 8+ merge algorithms including TIES, DARE, SLERP, and more. Features modular architecture, SVD decomposition, selective layer filtering, and comprehensive validation.

[IMAGE: Overview of merger nodes - PLACEHOLDER]

This is an enhanced fork of laksjdjf's LoRA Merger with extensive refactoring and new features. Core merging algorithms from Mergekit by Arcee AI.

Features

• 8+ Merge Algorithms: Task Arithmetic, TIES, DARE, DELLA, Breadcrumbs, SLERP, and more
• Mergekit Integration: Production-grade merge methods from Arcee AI's Mergekit library
• SVD Support: Full, randomized, and energy-based SVD decomposition with dynamic rank selection
• Modular Architecture: Clean separation of concerns with focused, single-responsibility modules
• DiT Architecture Support: Automatic layer grouping for Diffusion Transformer models
• Selective Layer Merging: Filter by attention layers, MLP layers, or custom patterns
• Comprehensive Validation: Runtime type checking and structured error reporting
• Thread-Safe Processing: Parallel processing with device-aware workload distribution

Installation

cd ComfyUI/custom_nodes
git clone https://github.com/YourUsername/LoRA-Merger-ComfyUI.git
cd LoRA-Merger-ComfyUI
pip install -r requirements.txt

Requirements:

• PyTorch
• Mergekit (git+https://github.com/arcee-ai/mergekit.git)
• lxml

Quick Start

Basic Two-LoRA Merge

[IMAGE: Basic workflow - PM LoRA Stacker → PM LoRA Decompose → PM LoRA Merger → PM LoRA Apply - PLACEHOLDER]

1. Stack LoRAs with PM LoRA Stacker or PM LoRA Power Stacker
2. Decompose using PM LoRA Stack Decompose
3. Choose a merge method (e.g., PM TIES, PM DARE)
4. Merge with PM LoRA Merger (Mergekit)
5. Apply with PM LoRA Apply or save with PM LoRA Save

Batch Directory Merge

[IMAGE: Directory merge workflow - PLACEHOLDER]

Use PM LoRA Stacker (Directory) to load all LoRAs from a folder and merge them in one operation.

Node Reference

Core Workflow Nodes

PM LoRA Stacker

Combine multiple LoRAs into a stack for merging. Dynamically adds connection points as you connect LoRAs.

[IMAGE: PM LoRA Stacker node - PLACEHOLDER]

Inputs:

• lora_1, lora_2, ... lora_N: LoRABundle inputs (unlimited)

Output:

• LoRAStack: Dictionary mapping LoRA names to their patch dictionaries

PM LoRA Stacker (Directory)

Load all LoRAs from a directory automatically.

[IMAGE: PM LoRA Stacker Directory node - PLACEHOLDER]

Parameters:

• directory_path: Path to folder containing LoRA files
• strength_model: Default model strength for all LoRAs
• strength_clip: Default CLIP strength for all LoRAs

PM LoRA Stack Decompose

Decompose LoRA stack into (up, down, alpha) tensor components for merging.

[IMAGE: PM LoRA Stack Decompose node - PLACEHOLDER]

Features:

• Hash-based caching: Skips expensive decomposition if inputs unchanged
• Architecture detection: Automatically identifies SD vs DiT LoRAs
• Layer filtering: Apply preset or custom layer filters

Parameters:

• lora_stack: Input LoRAStack
• layer_filter: Preset filters ("full", "attn-only", "attn-mlp", "mlp-only", "dit-attn", "dit-mlp") or custom

Outputs:

• components: LoRATensors (decomposed tensors by layer)
• strengths: LoRAWeights (strength_model/strength_clip per LoRA)

PM LoRA Merger (Mergekit)

Main merging node using Mergekit algorithms. Processes layers in parallel with thread-safe progress tracking.

[IMAGE: PM LoRA Merger node - PLACEHOLDER]

Parameters:

• components: Decomposed LoRATensors from decompose node
• strengths: LoRAWeights from decompose node
• merge_method: MergeMethod configuration from method nodes
• lambda_scale: Final scaling factor (default: 1.0)
• device: Processing device ("cpu", "cuda")
• dtype: Computation precision ("float32", "float16", "bfloat16")

Features:

• Parallel processing: ThreadPoolExecutor with max_workers=8
• Comprehensive validation: Input structure, tensor shapes, strength presence
• Smart strength application: Uses strength_model for UNet, strength_clip for CLIP layers
• Device-aware distribution: Balances CPU and GPU workload

Output:

• Merged LoRA as LoRAAdapter state dictionary

PM LoRA Apply

Apply merged LoRA to a model.

[IMAGE: PM LoRA Apply node - PLACEHOLDER]

Inputs:

• model: ComfyUI model to patch
• clip: ComfyUI CLIP model
• lora: Merged LoRA from merger

Outputs:

• model: Patched model
• clip: Patched CLIP

PM LoRA Save

Save merged LoRA to disk in standard format.

[IMAGE: PM LoRA Save node - PLACEHOLDER]

Parameters:

• lora: Merged LoRA to save
• filename: Output filename (without extension)

Merge Method Nodes

Each method node configures algorithm-specific parameters. Connect to the merge_method input of PM LoRA Merger.

PM Linear

Simple weighted linear combination.

[IMAGE: PM Linear node - PLACEHOLDER]

Parameters:

• normalize (bool): Normalize by number of LoRAs (default: True)

PM TIES

Task Arithmetic with Interference Elimination and Sign consensus.

[IMAGE: PM TIES node - PLACEHOLDER]

Parameters:

• density (float): Fraction of values to keep (0.0-1.0, default: 0.9)
• normalize (bool): Normalize merged result (default: True)

Reference: TIES-Merging Paper

PM DARE

Drop And REscale for efficient model merging.

[IMAGE: PM DARE node - PLACEHOLDER]

Parameters:

• density (float): Probability of keeping each parameter (default: 0.9)
• normalize (bool): Normalize after rescaling (default: True)

Reference: DARE Paper

PM DELLA

Depth-Enhanced Low-rank adaptation with Layer-wise Averaging.

[IMAGE: PM DELLA node - PLACEHOLDER]

Parameters:

• density (float): Layer density parameter (default: 0.9)
• epsilon (float): Small value for numerical stability (default: 1e-8)
• lambda_factor (float): Scaling factor (default: 1.0)

PM Breadcrumbs

Breadcrumb-based merging strategy.

[IMAGE: PM Breadcrumbs node - PLACEHOLDER]

Parameters:

• density (float): Path density (default: 0.9)
• tie_method ("sum" or "mean"): How to combine tied parameters

PM SLERP

Spherical Linear Interpolation for smooth model interpolation.

[IMAGE: PM SLERP node - PLACEHOLDER]

Parameters:

• t (float): Interpolation factor (0.0-1.0, default: 0.5)

Note: SLERP requires exactly 2 LoRAs. For multiple LoRAs, use PM NuSLERP or PM Karcher.

PM NuSLERP

Normalized Spherical Linear Interpolation for multiple models.

[IMAGE: PM NuSLERP node - PLACEHOLDER]

Parameters:

• normalize (bool): Normalize result to unit sphere (default: True)

PM Karcher

Karcher mean on the manifold (generalized SLERP for N models).

[IMAGE: PM Karcher node - PLACEHOLDER]

Parameters:

• max_iterations (int): Maximum optimization iterations (default: 100)
• tolerance (float): Convergence threshold (default: 1e-6)

PM Task Arithmetic

Standard task vector arithmetic (delta merging).

[IMAGE: PM Task Arithmetic node - PLACEHOLDER]

Parameters:

• normalize (bool): Normalize by number of models (default: False)

PM SCE (Selective Consensus Ensemble)

Selective consensus with threshold-based parameter selection.

[IMAGE: PM SCE node - PLACEHOLDER]

Parameters:

• threshold (float): Consensus threshold (default: 0.5)

PM NearSwap

Nearest neighbor parameter swapping.

[IMAGE: PM NearSwap node - PLACEHOLDER]

Parameters:

• distance_metric ("cosine" or "euclidean"): Distance measure

PM Arcee Fusion

Arcee's proprietary fusion method for high-quality merges.

[IMAGE: PM Arcee Fusion node - PLACEHOLDER]

Parameters:

• Various advanced parameters (see node UI)

Utility Nodes

PM LoRA Resizer

Adjust LoRA rank using SVD decomposition.

[IMAGE: PM LoRA Resizer node - PLACEHOLDER]

Parameters:

• lora: Input LoRA
• rank_mode: Rank selection strategy
  • target_rank specifies exact rank
  • sv_ratio: Keep singular values above ratio threshold
  • sv_cumulative: Keep top N% of cumulative energy
  • sv_fro: Frobenius norm-based truncation
• target_rank (int): Target rank for fixed mode
• dynamic_param (float): Parameter for dynamic modes (ratio/cumulative/fro)
• device: Processing device
• dtype: Computation precision

Features:

• Dynamic rank selection: Automatically choose optimal rank based on singular value distribution
• Statistical reporting: Shows Frobenius norm retention and singular value retention
• Conv and linear support: Handles 2D, 3D, and 4D tensors
• Decomposer selection: Choose from Standard SVD, Randomized SVD, or QR factorization

Output:

• Resized LoRA with adjusted rank

PM LoRA Block Sampler

Sample different block configurations for layer-wise experiments.

[IMAGE: PM LoRA Block Sampler node - PLACEHOLDER]

PM LoRA Stack Sampler

Sample subsets of LoRAs from a stack.

[IMAGE: PM LoRA Stack Sampler node - PLACEHOLDER]

PM Parameter Sweep Sampler

Systematically sweep through parameter combinations for merge optimization.

[IMAGE: PM Parameter Sweep Sampler node - PLACEHOLDER]

Features:

• Cartesian product of parameter ranges
• Support for strength, density, rank variations
• Export results for analysis

Power Stacker Node

PM LoRA Power Stacker

Advanced stacking with per-LoRA configuration and dynamic input management.

[IMAGE: PM LoRA Power Stacker node - PLACEHOLDER]

Features:

• Dynamic LoRA inputs: Add unlimited LoRAs with individual strength controls
• Per-LoRA strengths: Separate strength_model and strength_clip for each LoRA
• Architecture detection: Automatically identifies SD vs DiT LoRAs
• Layer filtering: Built-in preset and custom layer filters

SVD and Decomposition

The project includes a comprehensive tensor decomposition system with multiple strategies:

Decomposition Methods

Standard SVD

Full singular value decomposition for exact low-rank approximation.

Use case: High accuracy, small to medium tensors

Randomized SVD

Fast approximate SVD using randomized linear algebra.

Use case: Large tensors where speed is critical

Energy-Based Randomized SVD

Adaptive SVD that automatically selects rank based on energy threshold.

Use case: Automatic rank selection with quality guarantees

Error Handling

All decomposers include:

• GPU failure fallback: Automatically retries on CPU if GPU decomposition fails
• Zero matrix detection: Gracefully handles degenerate cases
• Shape validation: Automatic reshape for 2D/3D/4D tensors (conv and linear layers)
• Numerical stability: Epsilon regularization for near-singular matrices

Layer Filtering

Selective merging allows targeting specific layer types:

Preset Filters

• "full": All layers (no filter)
• "attn-only": Only attention layers (attn1, attn2)
• "attn-mlp": Attention + feed-forward (attn1, attn2, ff)
• "mlp-only": Only feed-forward layers
• "dit-attn": DiT attention layers
• "dit-mlp": DiT MLP layers

Custom Filters

Specify layer keys as comma-separated string or set:

"attn1, attn2, ff.net.0"

Architecture Support

Stable Diffusion LoRAs

Automatic detection and handling of:

• UNet blocks: Input, middle, output blocks
• Attention layers: attn1 (self-attention), attn2 (cross-attention)
• Feed-forward: ff.net layers
• CLIP text encoder: text_model layers

DiT (Diffusion Transformer) LoRAs

Automatic layer grouping for:

• Joint blocks: Unified transformer blocks
• Attention: Multi-head attention layers
• MLP: Feed-forward networks
• Positional encoding: Learned position embeddings

The system automatically detects architecture and applies appropriate decomposition strategies.

License

MIT License

Copyright (c) 2024 LoRA Power-Merger Contributors

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Credits

• Original LoRA Merger by laksjdjf
• Mergekit by Arcee AI
• TIES-Merging: Paper
• DARE: Paper
• ComfyUI by comfyanonymous