ethanfel-ComfyUI-LoRA-Optimizer/docs/wiki/How-It-Works.md

# How the LoRA Optimizer Works

This page explains the full pipeline — from building a LoRA stack through analysis, strategy selection, merge execution, and post-processing. Each section builds on the previous one to show how algorithms interact and why each step exists.

---

## Table of Contents

- [The Problem](#the-problem)
- [Pipeline Overview](#pipeline-overview)
- [Step 1 — Building the LoRA Stack](#step-1--building-the-lora-stack)
- [Step 2 — Key Normalization](#step-2--key-normalization)
- [Step 3 — Pass 1: Analysis](#step-3--pass-1-analysis)
  - [Computing Weight Diffs](#computing-weight-diffs)
  - [Conflict Detection](#conflict-detection)
  - [Cosine Similarity](#cosine-similarity)
  - [Auto-Strength Calibration](#auto-strength-calibration)
  - [Suggested Max Output Strength](#suggested-max-output-strength)
- [Step 4 — Strategy Selection](#step-4--strategy-selection)
  - [Per-Prefix Decision Tree](#per-prefix-decision-tree)
  - [Global Merge Mode Selection](#global-merge-mode-selection)
  - [Architecture Presets](#architecture-presets)
  - [Behavior Profiles](#behavior-profiles)
  - [Sign Method Selection](#sign-method-selection)
- [Step 5 — Pass 2: Merge Execution](#step-5--pass-2-merge-execution)
  - [Preprocessing: Sparsification](#preprocessing-sparsification)
  - [Quality Enhancements](#quality-enhancements)
  - [Merge Algorithms](#merge-algorithms)
  - [How Algorithms Interact](#how-algorithms-interact)
- [Step 6 — Post-Processing](#step-6--post-processing)
  - [SVD Patch Compression](#svd-patch-compression)
  - [Dtype Downcasting](#dtype-downcasting)
  - [VRAM Budget & Patch Placement](#vram-budget--patch-placement)
  - [Patch Application](#patch-application)
- [Algorithm Interaction Map](#algorithm-interaction-map)
- [Node Reference](#node-reference)
- [Appendix: AutoTuner](#appendix-autotuner)
- [Appendix: Conflict Editor](#appendix-conflict-editor)
- [Appendix: Per-LoRA Controls](#appendix-per-lora-controls)
- [Appendix: Output Nodes](#appendix-output-nodes)
- [Appendix: Supported Architectures](#appendix-supported-architectures)

---

## The Problem

When you stack multiple LoRAs in ComfyUI, each one is applied independently — their effects simply add together on the base model weights. This works fine when LoRAs touch different parts of the model, but breaks down when they overlap:

- **Oversaturation** — Two style LoRAs at strength 1.0 can produce blown-out results because their effects compound in the same weight regions
- **Sign conflicts** — LoRA A wants to push a weight positive while LoRA B pushes it negative. Simple addition cancels out both contributions
- **Noise amplification** — Minor disagreements across thousands of parameters accumulate into visible artifacts

The LoRA Optimizer solves this by analyzing _where_ LoRAs conflict and applying the right merge strategy to each region independently.

---

## Pipeline Overview

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                          LORA OPTIMIZER PIPELINE                            │
│                                                                             │
│  LoRA Stack ──► Key Normalization ──► Pass 1: Analysis ──► Strategy Selection│
│                                           │                      │          │
│                                           ▼                      ▼          │
│                                      Pass 2: Merge Execution                │
│                                           │                                 │
│                                           ▼                                 │
│                                    Post-Processing                          │
│                              (SVD compress, dtype cast,                     │
│                               VRAM budget placement)                        │
│                                           │                                 │
│                                           ▼                                 │
│                            MODEL + CLIP + Report + LORA_DATA                │
└─────────────────────────────────────────────────────────────────────────────┘
```

The pipeline has two physical passes through the weights but several logical stages. This two-pass design keeps peak memory close to “one target group at a time,” but the exact peak still depends on the largest layer being processed, how many LoRAs overlap on it, and whether extra quality/compression steps are enabled.

---

## Step 1 — Building the LoRA Stack

Before anything is analyzed or merged, you assemble a list of LoRAs using **LoRA Stack** or **LoRA Stack (Dynamic)** nodes.

Each entry in the stack contains:
- **LoRA file** — the `.safetensors` file to load
- **Strength** — how much this LoRA should contribute (default 1.0)
- **Conflict mode** — controls where this LoRA's contributions are kept (`all`, `low_conflict`, `high_conflict`)
- **Key filter** — controls which weight prefixes this LoRA contributes to (`all`, `shared_only`, `unique_only`)

The stack is just a list — no analysis happens yet. LoRAs are loaded lazily when the optimizer needs them.

### LoRA Stack (Dynamic)

The Dynamic variant provides 1–10 adjustable LoRA slots in a single node, replacing the need to chain multiple Stack nodes. It has two visibility modes controlled by `settings_visibility`:

| Mode | Per-LoRA Controls |
|------|-------------------|
| **Simple** | Single `strength` slider per LoRA — clean and beginner-friendly |
| **Advanced** | Separate `model_strength` and `clip_strength`, plus `conflict_mode` and `key_filter` per LoRA |

It also supports two input modes via `input_mode`:
- **Dropdown** — standard file picker from your LoRAs folder
- **Text** — type a name or path; auto-matches against installed LoRAs

An optional `base_model_filter` integrates with ComfyUI-Lora-Manager to filter LoRAs by base model (when that extension is installed).

### Stack Formats

The optimizer accepts stacks from multiple sources:
- **Native format** — dict-based entries from LoRA Stack / LoRA Stack (Dynamic)
- **Tuple format** — `(lora_name, model_strength, clip_strength)` from Efficiency Nodes, Comfyroll, and other packs

Both are normalized internally before analysis begins.

---

## Step 2 — Key Normalization

> _Controlled by: `normalize_keys` (disabled / enabled)_

Different LoRA trainers name their weights differently. Kohya uses underscores (`lora_transformer_double_blocks_0_...`), AI-Toolkit uses dots (`transformer.transformer_blocks.0...`), diffusers uses yet another format. If you mix LoRAs from different trainers, the optimizer sees _no key overlap_ — it thinks they touch completely different parts of the model and can't merge them.

Key normalization solves this by:

1. **Detecting the model architecture** from LoRA key patterns (FLUX, SDXL, Wan, Z-Image, LTX, Qwen)
2. **Remapping all keys** to a canonical format so LoRAs from any trainer can be compared

### Special Case: Z-Image QKV Fusion

Z-Image LoRAs often fuse Q, K, and V projections into a single `attention.qkv` weight. The normalizer:
1. Detects fused QKV (lora_up shape = `[3×out_features, rank]`)
2. **Splits** into separate `to_q`, `to_k`, `to_v` components
3. Allows per-component conflict analysis during the merge
4. **Re-fuses** back to native `attention.qkv` format after merging

This splitting is critical — Q, K, and V weights serve different purposes. Two LoRAs might agree on Q weights but conflict on V weights. Without splitting, the optimizer would average their conflicts and get a suboptimal result for both.

### Special Case: WanVideo `_orig_mod` Keys

WanVideo models compiled with `torch.compile` prefix their keys with `_orig_mod.`. The normalizer detects this mismatch and strips the prefix so LoRA keys can match correctly against model weights.

---

## Step 3 — Pass 1: Analysis

Pass 1 iterates through every unique weight prefix (e.g., `input_blocks.4`, `middle_block.1`) across all LoRAs. For each prefix, it computes statistics and then **immediately discards the full weight diffs**. Only lightweight scalars survive — this is what keeps memory usage low.

### Computing Weight Diffs

For each LoRA at each prefix, the optimizer computes the **full-rank diff** — the actual change this LoRA would make to the base model weights:

```
diff = lora_up @ lora_down × (alpha / rank) × strength
```

For LoCon layers (with `lora_mid`), the reconstruction handles the additional intermediate matrix.

These diffs are the ground truth — they show exactly what each LoRA wants to do to each weight. All analysis operates on these diffs, not on the raw low-rank factors.

### Conflict Detection

For each pair of LoRAs that overlap at a given prefix, the optimizer measures **sign conflict ratio**: the fraction of weight positions where the two LoRAs push in opposite directions.

```
conflict_ratio = count(sign(diff_A) ≠ sign(diff_B)) / total_overlapping_positions
```

A high conflict ratio (>25%) means the LoRAs are fighting each other at this prefix — they want opposite things. A low ratio means they mostly agree.

The conflict ratio is measured per-prefix, not globally. Two LoRAs might have 5% conflict in attention layers but 40% conflict in feed-forward layers. The optimizer handles each independently.

### Cosine Similarity

Beyond sign conflicts, the optimizer measures **cosine similarity** between LoRA diff vectors:

```
cos_sim = dot(diff_A, diff_B) / (||diff_A|| × ||diff_B||)
```

This captures directional alignment:
- **cos ≈ 1** → LoRAs point the same direction (reinforcing)
- **cos ≈ 0** → LoRAs are orthogonal (independent)
- **cos ≈ -1** → LoRAs point opposite directions (canceling)

Cosine similarity is used by:
- **Auto-strength calibration** — to compute interference energy
- **Global mode selection** — consensus and SLERP are preferred when similarity is high
- **Behavior profiles** — orthogonal detection triggers SLERP upgrade

### Auto-Strength Calibration

> _Controlled by: `auto_strength` (enabled / disabled)_

When enabled, the optimizer reduces all LoRA strengths proportionally to prevent oversaturation from stacking. The algorithm is **interference-aware energy normalization**:

1. Measure pairwise cosine similarity between all LoRAs
2. Compute total combined energy accounting for directional alignment:
   ```
   ||Σ v_i||² = Σ||v_i||² + 2 × Σ ||v_i|| × ||v_j|| × cos(v_i, v_j)
   ```
3. Scale all strengths so combined energy = energy of the strongest single LoRA alone

The key insight is that aligned LoRAs (cos ≈ 1) produce more combined energy than orthogonal ones (cos ≈ 0), which produce more than opposing ones (cos ≈ -1). The calibration adapts to the actual geometry:

| Scenario | Scale Factor |
|----------|-------------|
| 2 aligned LoRAs (cos≈1) | ~0.50 each |
| 2 orthogonal LoRAs (cos≈0) | ~0.71 each |
| 2 opposing LoRAs (cos≈-1) | ~1.0 each |

Original strength _ratios_ are always preserved — only the overall scale changes.

### Suggested Max Output Strength

The analysis report now includes a **suggested maximum output strength** — automatically calculated from the magnitude ratio of the merged LoRAs:

```
suggested_max = min(1.0 / norm_ratio, architecture_cap)
```

The cap varies by architecture preset (3.0 for UNet, 5.0 for DiT, 3.0 for LLM). This helps you avoid oversaturation by showing a safe strength ceiling. It appears in the report alongside the current `output_strength` value.

---

## Step 4 — Strategy Selection

After analysis, the optimizer decides how to merge each prefix. This is where `optimization_mode` matters:

| Mode | Behavior |
|------|----------|
| `per_prefix` (default) | Each weight prefix gets its own strategy based on local conflict data |
| `global` | One strategy for all prefixes, chosen from aggregate statistics |
| `additive` | Force simple addition everywhere |

### Per-Prefix Decision Tree

For each prefix independently:

```
                    How many LoRAs touch this prefix?
                              │
                    ┌─────────┴─────────┐
                    │                   │
                 Only 1              2 or more
                    │                   │
              weighted_sum         Sign conflict ratio?
           (full strength,              │
            no dilution)      ┌─────────┴─────────┐
                              │                   │
                           ≤ 25%               > 25%
                              │                   │
                      weighted_average          TIES
                       (compatible,        (resolve conflicts
                        blend fairly)    via trim/elect/merge)
```

This is the fundamental innovation: non-overlapping regions get 100% of their LoRA's effect (no dilution from averaging), while genuinely conflicting regions get proper TIES conflict resolution.

### Global Merge Mode Selection

When using `global` optimization (or as a fallback), the optimizer picks a single mode based on aggregate statistics. The `full` strategy set adds more nuanced selection:

```
                        Aggregate conflict ratio
                              │
                    ┌─────────┴─────────┐
                    │                   │
                 ≤ 25%               > 25%
                    │                   │
           Cosine similarity?        TIES
                    │
          ┌─────┬──┴──┬─────┐
          │     │     │     │
       > 0.7  0.3-0.7  < 0.3  orthogonal
          │     │     │     │
      consensus  weighted   SLERP
                 average   (full)
```

### Architecture Presets

> _Controlled by: `architecture_preset` (auto / sd_unet / dit / llm)_

Different model architectures have different numerical characteristics. The architecture preset tunes the optimizer's internal thresholds — density ranges, noise floors, and strength caps — to match:

| Preset | Architectures | Density Range | Noise Floor | Max Strength Cap |
|--------|--------------|---------------|-------------|-----------------|
| `sd_unet` | SD 1.5, SDXL | 0.1 – 0.9 | 10% | 3.0 |
| `dit` | Flux, Wan, Z-Image, LTX, HunyuanVideo | 0.4 – 0.95 | 5% | 5.0 |
| `llm` | Qwen, LLaMA | 0.1 – 0.8 | 15% | 3.0 |
| `auto` (default) | Auto-detected from LoRA keys | Selected automatically | — | — |

DiT models are denser and tolerate higher strengths. UNet models are sparser. LLM models need tighter density bounds and lower strength caps.

The architecture preset is **orthogonal** to the behavior profile — the preset controls _numeric thresholds_, while the profile controls _strategy selection logic_.

### Behavior Profiles

> _Controlled by: `strategy_set` (full / no_slerp / basic)_

The behavior profile controls _which_ merge strategies are considered during auto-selection:

- **`full`** (default) — Full detection: consensus for high similarity, SLERP for orthogonal, weighted_average otherwise
- **`no_slerp`** — Same detection but SLERP never auto-selected; stays as weighted_average
- **`basic`** — Pre-1.2 behavior: only TIES vs weighted_average, no SLERP or consensus

### Sign Method Selection

For TIES merging, the optimizer picks how to resolve sign conflicts:

| Condition | Method | Logic |
|-----------|--------|-------|
| Magnitude ratio > 2× | `total` | Stronger LoRA dominates the vote |
| Magnitude ratio ≤ 2× | `frequency` | Equal vote per LoRA |

---

## Step 5 — Pass 2: Merge Execution

Pass 2 walks through every prefix again, recomputes the diffs, and applies the selected strategy. Several processing stages happen in sequence for each prefix:

```
┌──────────────────────────────────────────────────────────────────┐
│                    PER-PREFIX MERGE PIPELINE                      │
│                                                                  │
│  Recompute      Quality           Sparsify       Apply Merge     │
│    Diffs    ──► Enhancements  ──► (if enabled) ──► Algorithm     │
│                (if enhanced/                         │            │
│                 maximum)                             ▼            │
│                                              Merged Patch        │
│                                                  │               │
│                                                  ▼               │
│                                           SVD Compress           │
│                                          (if enabled)            │
│                                                  │               │
│                                                  ▼               │
│                                        Dtype Downcast            │
│                                                  │               │
│                                                  ▼               │
│                                      VRAM Budget Placement       │
│                                                  │               │
│                                                  ▼               │
│                                           Free & Continue        │
└──────────────────────────────────────────────────────────────────┘
```

### Preprocessing: Sparsification

> _Controlled by: `sparsification`, `sparsification_density`, `dare_dampening`_

Sparsification reduces parameter interference by zeroing out a fraction of each LoRA's diff before merging. Two algorithms are available, each with a standard and conflict-aware variant:

#### DARE (Drop And REscale)

1. Generate a random binary mask where each weight has probability `density` of being kept
2. Zero out masked positions
3. Rescale survivors by `1/density` to preserve expected magnitude

**DAREx enhancement:** The `dare_dampening` parameter (0.0–1.0) interpolates the rescaling factor toward 1.0, reducing noise amplification from aggressive rescaling. At 0.0: standard DARE. Higher values: less noise at the cost of slightly biased magnitudes.

#### DELLA (magnitude-aware dropout)

1. Rank weights per-row by absolute magnitude
2. Low-magnitude weights get higher drop probability
3. High-magnitude weights are preserved
4. Rescale survivors to maintain expected value

DELLA is more surgical than DARE — it preferentially preserves the weights that matter most.

#### Conflict-Aware Variants

Standard sparsification drops weights everywhere, including positions where only one LoRA contributes or where LoRAs agree. This destroys useful signal.

Conflict-aware variants (`dare_conflict`, `della_conflict`) compute a sign-conflict mask first:
```
For each position:
  if 2+ LoRAs have opposite signs → sparsify (this is interference)
  if LoRAs agree or only 1 present → keep full weight (this is useful signal)
```

Result: interference is reduced without sacrificing unique features.

#### Interaction with TIES

When the merge strategy is TIES:
- **Sparsification replaces the TIES trim step.** Both achieve sparsification (removing low-value weights), so running both would be redundant. The sparsification density parameter controls how aggressive the combined effect is.

When the merge strategy is anything else:
- **Sparsification runs as preprocessing** before the merge algorithm sees the diffs.

### Quality Enhancements

> _Controlled by: `merge_refinement` (none / refine / full)_

Quality enhancements are additional processing steps applied before or during the merge. They compose with sparsification — sparsification runs first, then quality enhancements process the (possibly sparsified) diffs.

#### None (baseline)

No additional processing. Element-wise sign voting in TIES mode.

#### Refine (adds three techniques)

**1. DO-Merging (Decouple & Orthogonalize)**
- Orthogonalizes LoRA direction vectors via Modified Gram-Schmidt
- Preserves original magnitudes while reducing directional interference
- LoRAs that were partially aligned become truly independent

**2. Column-Wise Conflict Resolution** (replaces element-wise)
- Instead of each weight position voting on sign direction independently, entire output neurons (rows) vote as a unit
- Preserves structural coherence — a neuron's weights work together, so their signs should be resolved together
- Falls back to element-wise for 1D tensors (biases, norms)

**3. TALL-Masks (Task-Aware weight protection)**
- Identifies "selfish" weights — positions where one LoRA dominates and others contribute little
- Separates selfish contributions from the consensus
- After merging, adds selfish contributions back, protecting unique features from being averaged away

#### Full (adds SVD alignment)

Everything from Refine, plus:

**KnOTS SVD Alignment**
- Concatenates all LoRA diffs column-wise: `M = [diff_1 | diff_2 | ... | diff_N]`
- Computes truncated SVD (rank ≤ 256)
- Reconstructs each diff in the shared singular value basis
- Makes diffs more directly comparable by aligning their representation spaces
- Falls back to CPU on GPU OOM; skips gracefully if both fail

### Merge Algorithms

After preprocessing and quality enhancements, the actual merge algorithm runs. The algorithm was selected in Step 4.

#### Weighted Sum

```
output = Σ (strength_i × diff_i)
```

The simplest merge: plain addition. Used when only one LoRA touches a prefix (no conflict possible). Also available as `additive` optimization mode to force it everywhere.

**Properties:**
- Preserves every contribution exactly
- No information loss
- Fully compressible by SVD (linear operation)

#### Weighted Average

```
output = Σ (strength_i / Σ_strengths × diff_i)
```

Normalized addition. Fair blending when LoRAs agree on direction.

**Properties:**
- Prevents magnitude inflation from stacking
- Works well for compatible LoRAs (low conflict)
- Fully compressible by SVD (linear operation)

#### TIES (Trim, Elect Sign, Disjoint Merge)

A three-step process specifically designed for sign conflicts:

```
Step 1 — Trim: Keep only top-k% weights by magnitude
         (or: use DARE/DELLA sparsification instead)
              │
              ▼
Step 2 — Elect Sign: For each position, vote on the majority sign direction
         ┌─ frequency: each LoRA gets one vote
         └─ total: votes weighted by magnitude
              │
              ▼
Step 3 — Disjoint Merge: Average only the LoRAs that agree with the elected sign
         Contributors opposing the majority are excluded for this position
```

**Properties:**
- Resolves sign conflicts without cancellation
- Nonlinear (trim + sign election) → produces full-rank results
- Lossy SVD compression (can't perfectly capture nonlinear output)

#### SLERP (Spherical Linear Interpolation)

Magnitude-preserving directional blend:

1. For 2 LoRAs: standard SLERP formula along the geodesic on a hypersphere
2. For 3+ LoRAs: iterative pairwise SLERP, sorted by descending weight (strongest anchors direction)
3. Falls back to linear interpolation for near-parallel vectors (where SLERP is numerically unstable)
4. Final magnitude corrected to match weighted average of input norms

**Properties:**
- Preserves weight magnitude (no inflation or deflation)
- Smooth directional interpolation
- Best for low-conflict, similar-magnitude LoRAs

#### Consensus Merging

Three-stage algorithm for highly similar LoRAs:

```
Stage 1 — Fisher-Proxy Importance Weighting:
    weight each parameter by |diff|² as importance proxy
    numerator = Σ(diff_i × strength_i × |diff_i|²)
    denominator = Σ(|strength_i| × |diff_i|²)
    output = numerator / denominator

Stage 2 — MAGIC Calibration:
    rescale merged result so L2 norm matches
    weighted average of input norms

Stage 3 — MonoSoup Spectral Cleanup:
    SVD-based entropy filtering on 2D+ tensors
    keeps effective rank based on singular value entropy
    removes noise while preserving signal
```

**Properties:**
- Best for highly similar LoRAs (high cosine similarity)
- Importance-weighted → parameters that matter more get more influence
- Spectral cleanup removes merge noise

### How Algorithms Interact

The key to understanding the optimizer is knowing that these algorithms **compose in a fixed order**, not as alternatives:

```
                     ┌──────────────────────────────────────────────┐
                     │            COMPOSITION ORDER                 │
                     │                                              │
                     │  1. Key Normalization (if enabled)           │
                     │        ↓                                     │
                     │  2. Architecture Preset applied              │
                     │        ↓ (sets density/strength thresholds)  │
                     │  3. Auto-Strength (if enabled)               │
                     │        ↓ (adjusts strength scalars)          │
                     │  4. Quality: KnOTS alignment (if maximum)    │
                     │        ↓ (transforms diff vectors)           │
                     │  5. Quality: DO-Merging (if enhanced+)       │
                     │        ↓ (orthogonalizes directions)         │
                     │  6. Sparsification (if enabled)              │
                     │        ↓ (zeros out weights)                 │
                     │        ├─ TIES mode? → replaces trim step    │
                     │        └─ Other modes? → preprocessing       │
                     │  7. Quality: TALL-masks (if enhanced+)       │
                     │        ↓ (separates selfish weights)         │
                     │  8. Merge Algorithm                          │
                     │        ↓ (combines diffs)                    │
                     │  9. TALL-masks re-add (if enhanced+)         │
                     │        ↓ (restores selfish weights)          │
                     │  10. SVD Compression (if enabled)            │
                     │        ↓ (reduces to low-rank)               │
                     │  11. Dtype downcast to native precision      │
                     │        ↓ (reduces memory footprint)          │
                     │  12. VRAM budget placement (GPU or CPU)      │
                     │        ↓                                     │
                     │  13. Patch application to model              │
                     └──────────────────────────────────────────────┘
```

Important interactions:

| Combination | Interaction |
|-------------|-------------|
| DARE/DELLA + TIES | Sparsification **replaces** TIES trim (no double-sparsification) |
| DARE/DELLA + other modes | Sparsification runs as **preprocessing** before merge |
| Conflict-aware + per_prefix | Conflict mask computed per-prefix using local conflict data |
| KnOTS + TIES | KnOTS aligns diffs in shared basis _before_ TIES sign voting |
| DO-Merging + any mode | Orthogonalization happens _after_ KnOTS but _before_ sparsification |
| TALL-masks + any mode | Selfish weights separated _before_ merge, added back _after_ |
| Column-wise + TIES | Replaces element-wise sign voting (structural coherence) |
| Auto-strength + everything | Only adjusts strength scalars — all algorithms see adjusted strengths |
| Architecture preset + behavior profile | Preset sets _numeric thresholds_, profile sets _strategy logic_ — independent |

---

## Step 6 — Post-Processing

### SVD Patch Compression

> _Controlled by: `patch_compression` (smart / aggressive / disabled), `svd_device` (gpu / cpu)_

After merging, each prefix produces a full-rank diff tensor. For a 4096×4096 weight, that's ~64MB per key (vs ~0.5MB for a rank-32 LoRA patch). SVD compression re-factors these back to low-rank:

```
full_rank_diff ──► truncated SVD ──► lora_up (rank R) + lora_down (rank R)
```

The compression rank is automatically set to the **sum of all input LoRA ranks**. Three rank-32 LoRAs → rank-96 compressed patch. This is enough to represent the full merge without quality loss for linear operations.

| Mode | Compresses | Quality |
|------|-----------|---------|
| `smart` (default) | weighted_sum and weighted_average prefixes only | **Lossless** — linear operations are exactly representable |
| `aggressive` | Everything including TIES | **Lossy on TIES** — nonlinear ops (trim, sign election) produce full-rank results |
| `disabled` | Nothing | No loss, but ~32× more RAM |

### Dtype Downcasting

After merge and optional SVD compression, patches are automatically downcast to the **native dtype** of the input LoRA weights (typically `float16` or `bfloat16`). Merging internally uses `float32` for numerical precision, but the final patches don't need that precision. This reduces memory footprint significantly — a `float32` patch is 2× the size of a `float16` patch.

### VRAM Budget & Patch Placement

> _Controlled by: `vram_budget` (0.0 – 1.0)_

By default (`vram_budget=0.0`), all merged patches live on CPU. The model patcher moves them to GPU on demand during sampling, which adds latency.

When `vram_budget` > 0, the optimizer keeps merged patches on GPU up to a fraction of available free VRAM:

```
usable_vram = free_vram - current_model_size
gpu_budget = usable_vram × vram_budget
```

Patches are placed on GPU until the budget is exhausted; remaining patches stay on CPU. This trades GPU memory for faster sampling — especially useful for iterative workflows where you sample repeatedly with the same merge.

### Patch Application

The merged patches (compressed or full-rank) are applied to the model via ComfyUI's model patcher system. CLIP patches are applied separately with the `clip_strength_multiplier` scaling.

The optimizer outputs:
- **MODEL** — the patched model ready for sampling
- **CLIP** — the patched text encoder (if CLIP input was provided)
- **STRING** — detailed analysis report with per-prefix strategy map and suggested max strength
- **LORA_DATA** — the raw merged patches for use with Save Merged LoRA, Merged LoRA to Hook, or Merged LoRA to WanVideo

---

## Algorithm Interaction Map

This diagram shows every algorithm in the system and how they relate:

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                                                                             │
│  INPUT                    ANALYSIS                    STRATEGY              │
│  ─────                    ────────                    ────────              │
│  LoRA Stack ────► Key Normalization                                         │
│                         │                                                   │
│                         ▼                                                   │
│               Architecture Detection ──► Architecture Preset                │
│                         │                (density/strength thresholds)       │
│                         ▼                                                   │
│                   Compute Diffs ──────► Sign Conflict ──► Per-Prefix        │
│                         │                  Ratio          Decision          │
│                         │                                    │              │
│                         ├──────► Cosine ─────► Auto-        │              │
│                         │       Similarity     Strength      │              │
│                         │           │             │          │              │
│                         │           └──► Behavior  │          │              │
│                         │               Profile    │          │              │
│                         │               (strategy  │          │              │
│                         │                logic)    │          │              │
│                         │                          │          │              │
│                         ├──────► Magnitude ──► Sign Method   │              │
│                         │         Ratio       Selection      │              │
│                         │                                    │              │
│                         ├──────► Norms ──► Suggested Max     │              │
│                         │                  Strength          │              │
│                         │                                    │              │
│                   Discard Diffs                              │              │
│                                                              │              │
│  ═══════════════════════════════════════════════════════════════════════     │
│                                                              │              │
│  MERGE (per prefix)                                          │              │
│  ──────────────────                                          ▼              │
│                                                                             │
│  Recompute Diffs ──► KnOTS ──► DO-Merge ──► DARE/DELLA ──► ┌──────────┐   │
│  (with adjusted       (if       (if          (if            │  Select  │   │
│   strengths)        maximum)  enhanced+)    enabled)        │  Merge:  │   │
│                                                  │          │          │   │
│                                    ┌─────────────┤          │ wt_sum   │   │
│                                    │             │          │ wt_avg   │   │
│                                    │  TALL-mask  │          │ TIES     │   │
│                                    │  separate   │          │ SLERP    │   │
│                                    │  (if enh+)  │          │ consensus│   │
│                                    │             │          └────┬─────┘   │
│                                    │             ▼               │         │
│                                    │        Merge Execute ◄──────┘         │
│                                    │             │                         │
│                                    │  TALL-mask  │                         │
│                                    │  re-add ────┘                         │
│                                    │             │                         │
│                                    └─────────────┘                         │
│                                                  │                         │
│                                                  ▼                         │
│                                           SVD Compress                     │
│                                                  │                         │
│                                                  ▼                         │
│                                         Dtype Downcast                     │
│                                                  │                         │
│                                                  ▼                         │
│                                       VRAM Budget Placement                │
│                                                  │                         │
│                                                  ▼                         │
│                                           Apply to Model                   │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

## Node Reference

All nodes registered by the LoRA Optimizer:

| Node | Display Name | Purpose |
|------|-------------|---------|
| `LoRAStack` | LoRA Stack | Build LoRA stacks by chaining |
| `LoRAStackDynamic` | LoRA Stack (Dynamic) | Single node with 1–10 adjustable slots, simple/advanced modes |
| `LoRAMergeSettings` | LoRA Merge Settings | Shared config: normalization, architecture preset, smoothing, VRAM budget |
| `LoRAOptimizerSettings` | LoRA Optimizer Settings | Optimizer-specific settings: auto-strength, sparsification, compression, etc. |
| `LoRAAutoTunerSettings` | LoRA AutoTuner Settings | Tuner-specific settings: top_n, scoring, diff cache, etc. |
| `LoRAOptimizerSimple` | LoRA Optimizer | Simplified optimizer with sensible defaults, accepts optional `settings` and `tuner_data` |
| `LoRAOptimizer` | LoRA Optimizer (Legacy) | All parameters on one node — for bridge workflow with `settings_source` |
| `LoRAAutoTuner` | LoRA AutoTuner | Automated parameter sweep to rank merge configs |
| `LoRAMergeSelector` | Merge Selector | Select alternative configs from AutoTuner results |
| `LoRAConflictEditor` | LoRA Conflict Editor | Interactive conflict analysis with per-LoRA overrides |
| `SaveMergedLoRA` | Save Merged LoRA | Export merged patches as standalone `.safetensors` |
| `MergedLoRAToHook` | Merged LoRA to Hook | Wrap merged patches as conditioning hooks |
| `LoRACompatibilityAnalyzer` | LoRA Compatibility Analyzer | Pre-merge planning: overlap analysis, grouping, optional node creation |
| `WanVideoLoRAOptimizer` | WanVideo LoRA Optimizer (WIP) | Optimizer variant for WanVideo models |
| `MergedLoRAToWanVideo` | Merged LoRA → WanVideo (WIP) | Bridge merged LORA_DATA to WanVideo wrapper models |

### Node Variants

**LoRA Optimizer** (recommended):
Exposes `model`, `lora_stack`, `output_strength`, `clip` (optional), and `clip_strength_multiplier`. Accepts optional `settings` (from a Settings node) and `tuner_data` (from AutoTuner) inputs. When no settings node is connected, uses built-in defaults: `auto_strength=enabled`, `optimization_mode=per_prefix`, `merge_refinement=none`, `patch_compression=smart`, `vram_budget=0.0`.

**LoRA Optimizer (Legacy):**
All parameters on one node. Has `settings_source` for the AutoTuner ↔ Optimizer bridge workflow. Superseded by LoRA Optimizer + Settings nodes for new workflows.

**Settings nodes** (LoRA Merge Settings, LoRA Optimizer Settings, LoRA AutoTuner Settings):
Separate shared and node-specific configuration from the main optimizer. Merge Settings feeds into Optimizer Settings or AutoTuner Settings, which feed into the optimizer's `settings` input.

**WanVideo LoRA Optimizer:**
Accepts `WANVIDEOMODEL` instead of `MODEL`, skips CLIP. Defaults differ from the standard optimizer: `normalize_keys=enabled` (WanVideo LoRAs come from many trainers), `cache_patches=disabled` (video models are large), `architecture_preset=dit`.

---

## Appendix: AutoTuner

The **LoRA AutoTuner** node automates parameter selection by sweeping all combinations and ranking them with internal merge metrics, plus an optional external evaluator hook.

### Architecture

The AutoTuner runs a **single Pass 1 analysis** and caches the results. All parameter combinations are scored against this cached analysis without re-analyzing — making the sweep fast regardless of how many combinations are tested.

### Phase 1 — Fast Heuristic Scoring

Generates a full parameter grid across all configurable dimensions:

| Parameter | Values Swept |
|-----------|-------------|
| Merge modes | `weighted_average`, `slerp`, `consensus`, `ties` |
| Sparsification | `disabled`, `dare`, `della`, `dare_conflict`, `della_conflict` |
| Density | `0.5`, `0.7`, `0.9` |
| DARE dampening | `0.0`, `0.3`, `0.6` |
| Quality | `none`, `refine`, `full` |
| Auto-strength | `enabled`, `disabled` |
| Optimization mode | `per_prefix`, `global` |

This produces **2,000+ combinations**. Each is scored heuristically in ~2-5 seconds total using the analysis data from Pass 1 — no actual merges are performed. Scoring considers conflict, excess conflict, cosine similarity, subspace overlap, and magnitude distribution.

### Phase 2 — Merge & Measure Top-N

The top-N configs from Phase 1 are actually merged and scored by measuring:
- **Norm consistency** across patches
- **Effective rank** (SVD-based entropy) — when `scoring_svd=enabled`
- **Sparsity distribution** (estimated via column-sampling for efficiency)
- **Composite score** combining all metrics, optionally blended with an external evaluator

Progress is tracked via ComfyUI's progress bar (spanning both Phase 1 analysis and Phase 2 merges). Each candidate logs merge time and scoring time for diagnostics.

### Memory Management

Phase 2 is designed to avoid RAM exhaustion:
- Only the **current best candidate** is kept in memory
- Non-best candidates' model/clip patches are freed immediately after scoring
- `gc.collect()` is forced after freeing each non-best candidate
- Magnitude samples from Phase 1 are freed before Phase 2 begins

### AutoTuner Parameters

| Parameter | Default | Effect |
|-----------|---------|--------|
| `top_n` | 3 | How many candidates to actually merge and score |
| `scoring_svd` | disabled | Enable SVD-based effective rank scoring (slower, more thorough) |
| `scoring_device` | gpu | Where to run SVD scoring (`gpu` is 10-50x faster) |
| `scoring_speed` | turbo | Subsample prefix scoring for faster sweeps (`full`, `fast`, `turbo`, `turbo+`) |
| `auto_strength_floor` | -1.0 | Minimum auto-strength floor for orthogonal LoRAs (`-1` = architecture default) |
| `decision_smoothing` | 0.25 | Smooth per-prefix decision metrics toward block averages |
| `evaluator` | — | External evaluator hook for prompt/reference scoring |
| `diff_cache_mode` | auto | Reuse raw LoRA diffs across candidates (`disabled`, `auto`, `ram`, `disk`) |
| `diff_cache_ram_pct` | 0.5 | RAM budget for `auto` diff caching before spilling to disk |
| `cache_patches` | enabled | Keep the final AutoTuner merge cached in RAM for fast re-execution |
| `vram_budget` | 0.0 | Fraction of free VRAM for patch placement |
| `record_dataset` | disabled | Save analysis metrics to JSONL for threshold tuning research |

### Dataset Recording

When `record_dataset=enabled`, the AutoTuner saves a JSONL entry to `lora_optimizer_reports/autotuner_dataset.jsonl` after each run. Each entry records:
- Timestamp and detected architecture
- LoRA names and analysis summary
- Per-prefix strategy distribution
- All top-N configs with their measured scores

This data is used for refining architecture presets and threshold tuning.

### AutoTuner Outputs

| Output | Type | Purpose |
|--------|------|---------|
| `MODEL` | MODEL | Patched model from the top-ranked config |
| `CLIP` | CLIP | Patched text encoder |
| `report` | STRING | Ranked results with scores and parameters |
| `analysis_report` | STRING | Full optimizer analysis report for the top-ranked config |
| `TUNER_DATA` | TUNER_DATA | All ranked configs for Merge Selector |
| `LORA_DATA` | LORA_DATA | Merged patches for Save Merged LoRA / Merged LoRA to Hook |

Both AutoTuner and Merge Selector are marked as `OUTPUT_NODE = True` for ComfyUI's output recording and re-execution optimization.

### AutoTuner Workflow

```
LoRA Stack ──► LoRA AutoTuner ──► MODEL + CLIP + Report + Analysis Report + TUNER_DATA + LORA_DATA
                                                                                │            │
                                                                                ▼            ▼
                                                                         Merge Selector   Save Merged LoRA
                                                                      (pick alternative)
                                                                                │
                                                                                ▼
                                                                    MODEL + CLIP + LORA_DATA
```

---

## Appendix: Conflict Editor

The **LoRA Conflict Editor** node provides interactive analysis and override control:

1. Loads all LoRAs and computes pairwise conflict ratios
2. Auto-suggests per-LoRA conflict modes:
   - < 15% average conflict → `all`
   - 15–40% → `low_conflict`
   - \> 40% → `high_conflict`
3. Allows manual override of conflict modes and merge strategy
4. Outputs enriched stack + merge strategy override for the optimizer

### Conflict Editor Workflow

```
LoRA Stack ──► LoRA Conflict Editor ──► Enriched LORA_STACK ──► LoRA Optimizer
                     │                                                ▲
                     └── merge_strategy (STRING) ─────────────────────┘
                                                   (strategy override)
```

---

## Appendix: Per-LoRA Controls

### Conflict Modes

Per-LoRA control over where each LoRA's contributions are applied:

| Mode | Behavior | Use Case |
|------|----------|----------|
| `all` (default) | Apply everywhere | Normal merging |
| `low_conflict` | Only where this LoRA agrees with the majority | Conservative — keep only safe contributions |
| `high_conflict` | Only where this LoRA disagrees with the majority | Force dominance in contested regions |

### Key Filters

Per-LoRA control over which weight prefixes each LoRA contributes to:

| Filter | Behavior | Use Case |
|--------|----------|----------|
| `all` (default) | All prefixes | Normal merging |
| `shared_only` | Only prefixes present in 2+ LoRAs | Strip variant-specific keys (e.g., I2V keys from a mixed LoRA) |
| `unique_only` | Only prefixes present in exactly 1 LoRA | Extract lightweight variant-specific adapters |

---

## Appendix: Output Nodes

### Save Merged LoRA

Exports the optimizer's merged result as a standalone `.safetensors` file. Connects to `LORA_DATA` from LoRA Optimizer, AutoTuner, or Merge Selector.

| Option | Default | Effect |
|--------|---------|--------|
| `save_folder` | first configured LoRA folder | Choose which configured ComfyUI LoRA directory to save into |
| `filename` | `merged_lora` | File name relative to `save_folder`. Subdirectories allowed; traversal blocked |
| `save_rank` | 0 (auto) | 0 = use each layer's existing rank. Non-zero = force this rank via SVD compression |
| `bake_strength` | enabled | Bakes `output_strength` into saved weights so the LoRA reproduces your merge at strength 1.0 |

### Merged LoRA to Hook

Wraps merged patches as a **conditioning hook** (`HOOKS`) for per-conditioning LoRA application:

- **Per-prompt LoRA** — apply different merged LoRAs to positive vs negative conditioning
- **Scheduled application** — combine with hook keyframes for step-specific LoRA
- **Regional conditioning** — apply LoRA to specific image regions
- **Preserving the base model** — keep MODEL unpatched while using the merge through hooks

### Merged LoRA → WanVideo

Bridges `LORA_DATA` to a `WANVIDEOMODEL`. Handles the `_orig_mod.` key mismatch from `torch.compile` and injects patches in the format WanVideo's model loader expects.

---

## Appendix: Supported Architectures

| Architecture | Key Detection Patterns | Special Handling |
|-------------|----------------------|-----------------|
| **FLUX** | `double_blocks`, `single_blocks` | Sliced QKV offsets, multi-trainer unification |
| **SDXL** | `lora_te1_`, `input_blocks` | Text encoder + UNet unified |
| **Z-Image** (Lumina2) | `diffusion_model.layers.N.attention` | Fused QKV split/re-fuse, Musubi Tuner format |
| **Wan** 2.1/2.2 | `blocks.N` with `self_attn`/`ffn` | LyCORIS / diffusers / Musubi / Fun LoRA / finetrainer unified, RS-LoRA alpha fix |
| **LTX Video** | `adaln_single`, `attn1`/`attn2` | Trainer format unification |
| **Qwen-Image** | `img_mlp`/`txt_mlp`/`img_mod`/`txt_mod` | Dual-stream key unification |

**Supported trainers** (auto-normalized when `normalize_keys=enabled`):
Kohya, AI-Toolkit, LyCORIS, Musubi Tuner, diffusers/PEFT, Fun LoRA, finetrainer

**Supported LoRA formats:**
Standard LoRA, LoCon (with `lora_mid`), LyCORIS, diffusers/PEFT format, RS-LoRA (with alpha fix)

---

## References

- **TIES-Merging:** [Yadav et al., NeurIPS 2023](https://arxiv.org/abs/2306.01708) — Trim, Elect Sign, Disjoint Merge
- **DARE:** [Yu et al., ICML 2024](https://arxiv.org/abs/2311.03099) — Drop And REscale
- **DAREx:** ICLR 2025 — Dampened DARE rescaling
- **DELLA:** [Deep et al., 2024](https://arxiv.org/abs/2406.11617) — Magnitude-aware dropout
- **KnOTS:** [Ramé et al., 2024](https://arxiv.org/abs/2407.09095) — SVD alignment for model merging
- **TALL-masks:** [Wang et al., 2024](https://arxiv.org/abs/2406.12832) — Task-aware selfish weight protection
- **ZipLoRA:** [Shah et al., 2025](https://arxiv.org/abs/2311.13600) — Column-wise structural sparsity (inspiration)
- **DO-Merging:** [arXiv 2505.15875](https://arxiv.org/abs/2505.15875) — Decouple & Orthogonalize