ethanfel-ComfyUI-LoRA-Optimizer/docs/wiki/Merge-Algorithms.md

Merge Algorithms

Deep dive into each merge algorithm: the math, when each is selected, and how they interact with sparsification and quality enhancements.

For the overall pipeline and composition order, see How It Works.

---

Overview

The optimizer has six merge algorithms. Each prefix in the model can use a different one based on local conflict data.

Algorithm	Best For	Properties
Weighted Sum	Single-LoRA prefixes	Lossless, fully SVD-compressible
Weighted Average	Opposing / partially-aligned overlapping prefixes	Lossless, fully SVD-compressible
Sum-Preserve	Orthogonal prefixes (style + content)	Magnitude-preserving, energy-capped
TIES	High-conflict prefixes (>25% sign conflict)	Nonlinear, lossy SVD
SLERP	Low-conflict, similar-magnitude LoRAs	Magnitude-preserving
Consensus	Highly similar LoRAs (high cosine similarity)	Importance-weighted, spectral cleanup

---

Weighted Sum

The simplest merge: plain scaled addition.

Formula

output[i] = Σ_k (strength_k × diff_k[i])

Each LoRA's contribution is scaled by its strength and summed. No normalization, no conflict resolution.

When Selected

• Automatically used for prefixes where only one LoRA has weights (no overlap, no conflict)
• Forced everywhere when optimization_mode=additive

Properties

• Preserves every contribution exactly
• No information loss — output is the exact linear combination
• Fully compressible by SVD (rank = sum of input ranks)
• Can oversaturate when LoRAs reinforce each other at high strength

---

Weighted Average

Normalized sum that prevents magnitude inflation.

Formula

total_weight = Σ_k |strength_k|

output[i] = Σ_k (strength_k / total_weight × diff_k[i])

Same as weighted sum, but divided by the sum of strengths. This ensures the output magnitude stays proportional to individual LoRAs rather than growing with the number of LoRAs.

When Selected

• Per-prefix: 2+ LoRAs present, low conflict, and the directions are opposing (cos < 0) — the division keeps the directional cancellation
• Global: aggregate conflict ≤ 25% AND cosine similarity between 0.3–0.7
• Orthogonal, non-opposing prefixes go to Sum-Preserve instead (no_slerp/basic) or SLERP (full) — see below

Properties

• Fair blending — each LoRA contributes proportionally
• Prevents magnitude inflation from stacking
• Fully compressible by SVD (linear operation)
• Caveat: because it divides by Σ|strength|, a LoRA's absolute contribution is capped at a convex fraction — a style LoRA set to strength 2 still lands below 1× its delta (s/(s+s') → 1.0). For orthogonal stacks this washes out the weaker signal; that is why orthogonal prefixes now route to Sum-Preserve.

---

Sum-Preserve (Bounded Additive)

Keeps each LoRA at its full strength like Weighted Sum, but adds a per-group energy cap so orthogonal stacks don't oversaturate. Introduced to stop a style LoRA from disappearing when merged with a content/character LoRA.

Formula

result   = Σ_k (strength_k × diff_k)                 # full strength, like weighted_sum
max_single = max_k ‖strength_k × diff_k‖             # strongest single contribution
cap        = 1.5 × max_single                        # _ORTHO_SUM_HEADROOM
if ‖result‖ > cap:  result ×= cap / ‖result‖

Two orthogonal LoRAs combine to ≈1.41× the strongest single contribution (< 1.5×) and pass through un-capped — each keeps its full delta, so the style survives. Three equal orthogonal LoRAs hit ≈1.73× and are scaled back to exactly 1.5× — bounded, no oversaturation. The cap is per-group and independent of the global auto-strength floor, so it can't be defeated by the orthogonal-floor clamp the way a raw weighted-sum flip would be.

When Selected

• Per-prefix: 2+ LoRAs, orthogonal (low cosine similarity), non-opposing (cos ≥ 0), in the no_slerp or basic strategy sets
• The full strategy set keeps SLERP for these prefixes (the AutoTuner ranks both as candidates)

Properties

• Preserves each LoRA's emphasis — strength behaves as an amount, not a ratio
• Energy-capped per group — safe for 3+ orthogonal LoRAs
• Fully compressible by SVD (the cap folds into the factors as a scalar)
• This is what "additive looks better than per-prefix" was pointing at — now available inside per_prefix while real conflicts still get TIES

---

TIES Merging

Trim, Interfere, Elect Sign — specifically designed to resolve sign conflicts.

Reference: Yadav et al., NeurIPS 2023

Algorithm

TIES has three steps applied at each position independently:

Step 1 — Trim (Sparsification)

Keep only the top-k% of weights by absolute magnitude. The rest are zeroed out.

for each LoRA k:
    threshold_k = quantile(|diff_k|, 1 - density)
    trimmed_k[i] = diff_k[i] if |diff_k[i]| ≥ threshold_k else 0

The density parameter controls how aggressive the trim is.

Interaction with DARE/DELLA: If sparsification is enabled, it replaces this trim step entirely. Both achieve the same goal (zeroing out low-value weights), so running both would be redundant.

Step 2 — Elect Sign

For each position, vote on the majority sign direction across all LoRAs.

Frequency method (equal votes):

elected_sign[i] = sign(Σ_k sign(trimmed_k[i]))

Each LoRA gets one vote per position.

Total method (magnitude-weighted votes):

elected_sign[i] = sign(Σ_k trimmed_k[i])

Larger magnitudes carry more weight in the vote.

The optimizer picks the method based on magnitude ratio:

• Ratio > 2× → total (stronger LoRA dominates)
• Ratio ≤ 2× → frequency (equal votes)

Step 3 — Disjoint Merge

Average only the LoRAs that agree with the elected sign. Contributors opposing the majority are excluded.

agreeing_k = {k : sign(trimmed_k[i]) == elected_sign[i]}

output[i] = Σ_{k ∈ agreeing_k} (strength_k × trimmed_k[i]) / Σ_{k ∈ agreeing_k} |strength_k|

When Selected

• Per-prefix: 2+ LoRAs present AND sign conflict ratio > 25%
• Global: aggregate conflict ratio > 25%

Properties

• Resolves sign conflicts without cancellation — opposing LoRAs don't interfere
• Nonlinear operations (trim + sign election) produce full-rank output
• SVD compression is lossy on TIES prefixes
• Best for genuinely conflicting LoRAs

---

SLERP

Spherical Linear intERPolation — magnitude-preserving directional blend along the geodesic on a hypersphere.

Formula (2 LoRAs)

Ω = arccos(cos_sim(diff_A, diff_B))

output = sin((1-t)Ω)/sin(Ω) × diff_A + sin(tΩ)/sin(Ω) × diff_B

Where t is the interpolation weight derived from LoRA strengths.

For near-parallel vectors (Ω ≈ 0), falls back to linear interpolation to avoid numerical instability.

Formula (3+ LoRAs)

Iterative pairwise SLERP, sorted by descending weight:

1. Sort diffs by strength (descending)
2. result = diff_strongest
3. for each remaining diff_k (by decreasing strength):
       t_k = adjusted interpolation weight
       result = slerp(result, diff_k, t_k)
4. Correct final magnitude to match weighted average of input norms

The strongest LoRA anchors the direction; subsequent LoRAs blend in.

When Selected

• full strategy set: auto-selected when conflict ≤ 25% AND LoRAs are detected as orthogonal (low cosine similarity)
• Not auto-selected in the no_slerp or basic strategy sets
• Can be forced via merge_strategy_override

Properties

• Preserves weight magnitude (no inflation or deflation)
• Smooth directional interpolation on the unit sphere
• Magnitude correction ensures output norm matches expected value
• Best for low-conflict LoRAs of similar magnitude

---

Consensus Merging

Three-stage algorithm designed for highly similar LoRAs where importance-weighting and spectral cleanup produce cleaner results than simple averaging.

Stage 1 — Fisher-Proxy Importance Weighting

Weight each parameter by |diff|² as a proxy for Fisher information (parameter importance):

numerator[i] = Σ_k (diff_k[i] × strength_k × |diff_k[i]|²)

denominator[i] = Σ_k (|strength_k| × |diff_k[i]|²)

output[i] = numerator[i] / denominator[i]

Parameters where LoRAs make large changes get more influence. Positions with near-zero changes contribute little.

Stage 2 — MAGIC Calibration

Rescale the merged result so its L2 norm matches the weighted average of input norms:

target_norm = Σ_k (|strength_k| × ||diff_k||₂) / Σ_k |strength_k|

output = output × (target_norm / ||output||₂)

This prevents magnitude drift from the importance weighting.

Stage 3 — MonoSoup Spectral Cleanup

SVD-based entropy filtering on 2D+ tensors:

1. Compute SVD: output = U × S × Vᵀ
2. Compute entropy of singular values: H = -Σ (p_i × log(p_i))
   where p_i = s_i / Σ s_j
3. Effective rank = exp(H)
4. Keep only top effective_rank singular values
5. Reconstruct: output = U[:,:r] × S[:r] × Vᵀ[:r,:]

This removes noise — singular values with negligible contribution are filtered out, producing a cleaner merge.

When Selected

• full strategy set: auto-selected when conflict ≤ 25% AND cosine similarity > 0.7 (highly similar LoRAs)
• Not available in the basic strategy set
• Can be forced via merge_strategy_override

Properties

• Importance-weighted — parameters that change most get the most influence
• Magnitude-calibrated — no drift from weighting
• Spectral cleanup — removes merge noise via SVD filtering
• Best for similar LoRAs (same concept, different epochs/seeds)

---

Sparsification Methods

Sparsification is a preprocessing step that reduces parameter interference by zeroing out weights before merging. Two algorithms with two variants each.

DARE (Drop And REscale)

Reference: Yu et al., ICML 2024

mask[i] ~ Bernoulli(density)              # each position kept with probability = density
sparse_diff[i] = diff[i] × mask[i]        # zero out dropped positions
sparse_diff[i] = sparse_diff[i] / density  # rescale survivors to preserve expected value

DAREx enhancement (ICLR 2025): The dare_dampening parameter λ interpolates rescaling:

rescale_factor = 1 / (density^(1-λ))

At λ=0: standard DARE. Higher λ: less noise amplification at low density.

DELLA (magnitude-aware dropout)

Reference: Deep et al., 2024

for each row of diff:
    rank positions by |value| (ascending)
    drop_probability[i] = (1 - density) × (rank[i] / num_positions)
    mask[i] ~ Bernoulli(1 - drop_probability[i])
    sparse_diff[i] = diff[i] × mask[i] / (1 - drop_probability[i])

Low-magnitude weights have higher drop probability. High-magnitude weights are preserved. More surgical than DARE.

Conflict-Aware Variants

Standard sparsification operates everywhere. Conflict-aware variants (dare_conflict, della_conflict) only sparsify positions where LoRAs actually interfere:

conflict_mask[i] = (count of LoRAs with opposing signs at position i) ≥ 2

if conflict_mask[i]:
    apply DARE or DELLA sparsification
else:
    keep full weight (no sparsification)

This preserves unique contributions and same-sign reinforcements while still reducing interference at contested positions.

Interaction with Merge Strategies

Merge Strategy	Sparsification Role
TIES	Replaces the TIES trim step (Step 1)
Weighted Sum	Preprocessing before sum
Weighted Average	Preprocessing before average
SLERP	Preprocessing before interpolation
Consensus	Preprocessing before importance weighting

---

Quality Enhancements

Quality enhancements compose with all merge strategies. They modify the diffs or the merge process to improve results. See How It Works#quality-enhancements for the pipeline position.

DO-Merging (refine+)

Reference: arXiv 2505.15875

Orthogonalizes LoRA direction vectors via Modified Gram-Schmidt:

1. Flatten each diff to a 1D vector
2. Apply Gram-Schmidt orthogonalization (order: by descending norm)
3. Record original magnitudes
4. Reconstruct: new_diff = orthogonal_direction × original_magnitude

After orthogonalization, LoRAs that were partially aligned become truly independent. This reduces directional interference while preserving each LoRA's strength.

Column-Wise Conflict Resolution (refine+)

Inspired by ZipLoRA, Shah et al., 2025

Instead of each element voting on sign direction independently, entire output neurons (rows of the weight matrix) vote as a unit:

for each row j of the weight matrix:
    row_sign_votes = Σ_k sign(diff_k[j, :]) × |diff_k[j, :]|
    elected_row_sign = sign(Σ row_sign_votes)
    apply elected_row_sign to entire row

Preserves structural coherence — a neuron's input weights work together as a functional unit. Falls back to element-wise for 1D tensors (biases, layer norms).

TALL-Masks (refine+)

Reference: Wang et al., 2024

Identifies "selfish" weights — positions where one LoRA dominates:

for each position i:
    contributions = [|strength_k × diff_k[i]| for each LoRA k]
    dominant = argmax(contributions)
    if contributions[dominant] ≥ sum(others) × lambda:
        mark as selfish (belongs to LoRA dominant)

1. Separate selfish weights from consensus pool
2. Run merge algorithm on consensus weights only
3. Add selfish weights back to the result

Threshold lambda=1.0 — a LoRA's contribution must exceed all others combined to be considered selfish. This protects unique features from being averaged away during consensus merging.

KnOTS SVD Alignment (full)

Reference: Ramé et al., 2024

Projects all diffs into a shared singular value basis for better comparability:

1. Concatenate diffs column-wise: M = [diff_1 | diff_2 | ... | diff_N]
2. Compute truncated SVD: M ≈ U × S × Vᵀ  (rank ≤ 256)
3. For each diff k:
       proj_k = Uᵀ × diff_k         # project into shared basis
       aligned_k = U × proj_k        # reconstruct in aligned space
4. Use aligned diffs for all subsequent operations

This makes diffs more directly comparable by representing them in a shared coordinate system. Falls back to CPU on GPU OOM; skips gracefully if both fail.