EthanfelandClaude Opus 4.6
4ae79d0c9e
- Add example_workflows/LoRa Optimizer v2.json for the new node layout - Rename LoRA Optimizer (Advanced) → (Legacy) across all wiki pages - Document Settings nodes (Merge, Optimizer, AutoTuner) in Nodes.md - Add Settings and Compatibility Analyzer workflows to Workflows.md - Add decision_smoothing, smooth_slerp_gate, scoring_speed/formula, and Settings node architecture to Configuration-Guide.md - Update How-It-Works.md node registry and variant descriptions - Rename patch_compression non_ties → smart in all references - Update Home.md feature table (8 architectures, settings nodes) - Add zit_example_1.png asset - Update SVG assets (timeline, pipeline, bridge, ties, merge-use-cases) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
50 KiB
How the LoRA Optimizer Works
This page explains the full pipeline — from building a LoRA stack through analysis, strategy selection, merge execution, and post-processing. Each section builds on the previous one to show how algorithms interact and why each step exists.
Table of Contents
- The Problem
- Pipeline Overview
- Step 1 — Building the LoRA Stack
- Step 2 — Key Normalization
- Step 3 — Pass 1: Analysis
- Step 4 — Strategy Selection
- Step 5 — Pass 2: Merge Execution
- Step 6 — Post-Processing
- Algorithm Interaction Map
- Node Reference
- Appendix: AutoTuner
- Appendix: Conflict Editor
- Appendix: Per-LoRA Controls
- Appendix: Output Nodes
- Appendix: Supported Architectures
The Problem
When you stack multiple LoRAs in ComfyUI, each one is applied independently — their effects simply add together on the base model weights. This works fine when LoRAs touch different parts of the model, but breaks down when they overlap:
- Oversaturation — Two style LoRAs at strength 1.0 can produce blown-out results because their effects compound in the same weight regions
- Sign conflicts — LoRA A wants to push a weight positive while LoRA B pushes it negative. Simple addition cancels out both contributions
- Noise amplification — Minor disagreements across thousands of parameters accumulate into visible artifacts
The LoRA Optimizer solves this by analyzing where LoRAs conflict and applying the right merge strategy to each region independently.
Pipeline Overview
┌─────────────────────────────────────────────────────────────────────────────┐
│ LORA OPTIMIZER PIPELINE │
│ │
│ LoRA Stack ──► Key Normalization ──► Pass 1: Analysis ──► Strategy Selection│
│ │ │ │
│ ▼ ▼ │
│ Pass 2: Merge Execution │
│ │ │
│ ▼ │
│ Post-Processing │
│ (SVD compress, dtype cast, │
│ VRAM budget placement) │
│ │ │
│ ▼ │
│ MODEL + CLIP + Report + LORA_DATA │
└─────────────────────────────────────────────────────────────────────────────┘
The pipeline has two physical passes through the weights but several logical stages. This two-pass design keeps peak memory close to “one target group at a time,” but the exact peak still depends on the largest layer being processed, how many LoRAs overlap on it, and whether extra quality/compression steps are enabled.
Step 1 — Building the LoRA Stack
Before anything is analyzed or merged, you assemble a list of LoRAs using LoRA Stack or LoRA Stack (Dynamic) nodes.
Each entry in the stack contains:
- LoRA file — the
.safetensorsfile to load - Strength — how much this LoRA should contribute (default 1.0)
- Conflict mode — controls where this LoRA's contributions are kept (
all,low_conflict,high_conflict) - Key filter — controls which weight prefixes this LoRA contributes to (
all,shared_only,unique_only)
The stack is just a list — no analysis happens yet. LoRAs are loaded lazily when the optimizer needs them.
LoRA Stack (Dynamic)
The Dynamic variant provides 1–10 adjustable LoRA slots in a single node, replacing the need to chain multiple Stack nodes. It has two visibility modes controlled by settings_visibility:
| Mode | Per-LoRA Controls |
|---|---|
| Simple | Single strength slider per LoRA — clean and beginner-friendly |
| Advanced | Separate model_strength and clip_strength, plus conflict_mode and key_filter per LoRA |
It also supports two input modes via input_mode:
- Dropdown — standard file picker from your LoRAs folder
- Text — type a name or path; auto-matches against installed LoRAs
An optional base_model_filter integrates with ComfyUI-Lora-Manager to filter LoRAs by base model (when that extension is installed).
Stack Formats
The optimizer accepts stacks from multiple sources:
- Native format — dict-based entries from LoRA Stack / LoRA Stack (Dynamic)
- Tuple format —
(lora_name, model_strength, clip_strength)from Efficiency Nodes, Comfyroll, and other packs
Both are normalized internally before analysis begins.
Step 2 — Key Normalization
Controlled by:
normalize_keys(disabled / enabled)
Different LoRA trainers name their weights differently. Kohya uses underscores (lora_transformer_double_blocks_0_...), AI-Toolkit uses dots (transformer.transformer_blocks.0...), diffusers uses yet another format. If you mix LoRAs from different trainers, the optimizer sees no key overlap — it thinks they touch completely different parts of the model and can't merge them.
Key normalization solves this by:
- Detecting the model architecture from LoRA key patterns (FLUX, SDXL, Wan, Z-Image, LTX, Qwen)
- Remapping all keys to a canonical format so LoRAs from any trainer can be compared
Special Case: Z-Image QKV Fusion
Z-Image LoRAs often fuse Q, K, and V projections into a single attention.qkv weight. The normalizer:
- Detects fused QKV (lora_up shape =
[3×out_features, rank]) - Splits into separate
to_q,to_k,to_vcomponents - Allows per-component conflict analysis during the merge
- Re-fuses back to native
attention.qkvformat after merging
This splitting is critical — Q, K, and V weights serve different purposes. Two LoRAs might agree on Q weights but conflict on V weights. Without splitting, the optimizer would average their conflicts and get a suboptimal result for both.
Special Case: WanVideo _orig_mod Keys
WanVideo models compiled with torch.compile prefix their keys with _orig_mod.. The normalizer detects this mismatch and strips the prefix so LoRA keys can match correctly against model weights.
Step 3 — Pass 1: Analysis
Pass 1 iterates through every unique weight prefix (e.g., input_blocks.4, middle_block.1) across all LoRAs. For each prefix, it computes statistics and then immediately discards the full weight diffs. Only lightweight scalars survive — this is what keeps memory usage low.
Computing Weight Diffs
For each LoRA at each prefix, the optimizer computes the full-rank diff — the actual change this LoRA would make to the base model weights:
diff = lora_up @ lora_down × (alpha / rank) × strength
For LoCon layers (with lora_mid), the reconstruction handles the additional intermediate matrix.
These diffs are the ground truth — they show exactly what each LoRA wants to do to each weight. All analysis operates on these diffs, not on the raw low-rank factors.
Conflict Detection
For each pair of LoRAs that overlap at a given prefix, the optimizer measures sign conflict ratio: the fraction of weight positions where the two LoRAs push in opposite directions.
conflict_ratio = count(sign(diff_A) ≠ sign(diff_B)) / total_overlapping_positions
A high conflict ratio (>25%) means the LoRAs are fighting each other at this prefix — they want opposite things. A low ratio means they mostly agree.
The conflict ratio is measured per-prefix, not globally. Two LoRAs might have 5% conflict in attention layers but 40% conflict in feed-forward layers. The optimizer handles each independently.
Cosine Similarity
Beyond sign conflicts, the optimizer measures cosine similarity between LoRA diff vectors:
cos_sim = dot(diff_A, diff_B) / (||diff_A|| × ||diff_B||)
This captures directional alignment:
- cos ≈ 1 → LoRAs point the same direction (reinforcing)
- cos ≈ 0 → LoRAs are orthogonal (independent)
- cos ≈ -1 → LoRAs point opposite directions (canceling)
Cosine similarity is used by:
- Auto-strength calibration — to compute interference energy
- Global mode selection — consensus and SLERP are preferred when similarity is high
- Behavior profiles — orthogonal detection triggers SLERP upgrade
Auto-Strength Calibration
Controlled by:
auto_strength(enabled / disabled)
When enabled, the optimizer reduces all LoRA strengths proportionally to prevent oversaturation from stacking. The algorithm is interference-aware energy normalization:
- Measure pairwise cosine similarity between all LoRAs
- Compute total combined energy accounting for directional alignment:
||Σ v_i||² = Σ||v_i||² + 2 × Σ ||v_i|| × ||v_j|| × cos(v_i, v_j) - Scale all strengths so combined energy = energy of the strongest single LoRA alone
The key insight is that aligned LoRAs (cos ≈ 1) produce more combined energy than orthogonal ones (cos ≈ 0), which produce more than opposing ones (cos ≈ -1). The calibration adapts to the actual geometry:
| Scenario | Scale Factor |
|---|---|
| 2 aligned LoRAs (cos≈1) | ~0.50 each |
| 2 orthogonal LoRAs (cos≈0) | ~0.71 each |
| 2 opposing LoRAs (cos≈-1) | ~1.0 each |
Original strength ratios are always preserved — only the overall scale changes.
Suggested Max Output Strength
The analysis report now includes a suggested maximum output strength — automatically calculated from the magnitude ratio of the merged LoRAs:
suggested_max = min(1.0 / norm_ratio, architecture_cap)
The cap varies by architecture preset (3.0 for UNet, 5.0 for DiT, 3.0 for LLM). This helps you avoid oversaturation by showing a safe strength ceiling. It appears in the report alongside the current output_strength value.
Step 4 — Strategy Selection
After analysis, the optimizer decides how to merge each prefix. This is where optimization_mode matters:
| Mode | Behavior |
|---|---|
per_prefix (default) |
Each weight prefix gets its own strategy based on local conflict data |
global |
One strategy for all prefixes, chosen from aggregate statistics |
additive |
Force simple addition everywhere |
Per-Prefix Decision Tree
For each prefix independently:
How many LoRAs touch this prefix?
│
┌─────────┴─────────┐
│ │
Only 1 2 or more
│ │
weighted_sum Sign conflict ratio?
(full strength, │
no dilution) ┌─────────┴─────────┐
│ │
≤ 25% > 25%
│ │
weighted_average TIES
(compatible, (resolve conflicts
blend fairly) via trim/elect/merge)
This is the fundamental innovation: non-overlapping regions get 100% of their LoRA's effect (no dilution from averaging), while genuinely conflicting regions get proper TIES conflict resolution.
Global Merge Mode Selection
When using global optimization (or as a fallback), the optimizer picks a single mode based on aggregate statistics. The full strategy set adds more nuanced selection:
Aggregate conflict ratio
│
┌─────────┴─────────┐
│ │
≤ 25% > 25%
│ │
Cosine similarity? TIES
│
┌─────┬──┴──┬─────┐
│ │ │ │
> 0.7 0.3-0.7 < 0.3 orthogonal
│ │ │ │
consensus weighted SLERP
average (full)
Architecture Presets
Controlled by:
architecture_preset(auto / sd_unet / dit / llm)
Different model architectures have different numerical characteristics. The architecture preset tunes the optimizer's internal thresholds — density ranges, noise floors, and strength caps — to match:
| Preset | Architectures | Density Range | Noise Floor | Max Strength Cap |
|---|---|---|---|---|
sd_unet |
SD 1.5, SDXL | 0.1 – 0.9 | 10% | 3.0 |
dit |
Flux, Wan, Z-Image, LTX, HunyuanVideo | 0.4 – 0.95 | 5% | 5.0 |
llm |
Qwen, LLaMA | 0.1 – 0.8 | 15% | 3.0 |
auto (default) |
Auto-detected from LoRA keys | Selected automatically | — | — |
DiT models are denser and tolerate higher strengths. UNet models are sparser. LLM models need tighter density bounds and lower strength caps.
The architecture preset is orthogonal to the behavior profile — the preset controls numeric thresholds, while the profile controls strategy selection logic.
Behavior Profiles
Controlled by:
strategy_set(full / no_slerp / basic)
The behavior profile controls which merge strategies are considered during auto-selection:
full(default) — Full detection: consensus for high similarity, SLERP for orthogonal, weighted_average otherwiseno_slerp— Same detection but SLERP never auto-selected; stays as weighted_averagebasic— Pre-1.2 behavior: only TIES vs weighted_average, no SLERP or consensus
Sign Method Selection
For TIES merging, the optimizer picks how to resolve sign conflicts:
| Condition | Method | Logic |
|---|---|---|
| Magnitude ratio > 2× | total |
Stronger LoRA dominates the vote |
| Magnitude ratio ≤ 2× | frequency |
Equal vote per LoRA |
Step 5 — Pass 2: Merge Execution
Pass 2 walks through every prefix again, recomputes the diffs, and applies the selected strategy. Several processing stages happen in sequence for each prefix:
┌──────────────────────────────────────────────────────────────────┐
│ PER-PREFIX MERGE PIPELINE │
│ │
│ Recompute Quality Sparsify Apply Merge │
│ Diffs ──► Enhancements ──► (if enabled) ──► Algorithm │
│ (if enhanced/ │ │
│ maximum) ▼ │
│ Merged Patch │
│ │ │
│ ▼ │
│ SVD Compress │
│ (if enabled) │
│ │ │
│ ▼ │
│ Dtype Downcast │
│ │ │
│ ▼ │
│ VRAM Budget Placement │
│ │ │
│ ▼ │
│ Free & Continue │
└──────────────────────────────────────────────────────────────────┘
Preprocessing: Sparsification
Controlled by:
sparsification,sparsification_density,dare_dampening
Sparsification reduces parameter interference by zeroing out a fraction of each LoRA's diff before merging. Two algorithms are available, each with a standard and conflict-aware variant:
DARE (Drop And REscale)
- Generate a random binary mask where each weight has probability
densityof being kept - Zero out masked positions
- Rescale survivors by
1/densityto preserve expected magnitude
DAREx enhancement: The dare_dampening parameter (0.0–1.0) interpolates the rescaling factor toward 1.0, reducing noise amplification from aggressive rescaling. At 0.0: standard DARE. Higher values: less noise at the cost of slightly biased magnitudes.
DELLA (magnitude-aware dropout)
- Rank weights per-row by absolute magnitude
- Low-magnitude weights get higher drop probability
- High-magnitude weights are preserved
- Rescale survivors to maintain expected value
DELLA is more surgical than DARE — it preferentially preserves the weights that matter most.
Conflict-Aware Variants
Standard sparsification drops weights everywhere, including positions where only one LoRA contributes or where LoRAs agree. This destroys useful signal.
Conflict-aware variants (dare_conflict, della_conflict) compute a sign-conflict mask first:
For each position:
if 2+ LoRAs have opposite signs → sparsify (this is interference)
if LoRAs agree or only 1 present → keep full weight (this is useful signal)
Result: interference is reduced without sacrificing unique features.
Interaction with TIES
When the merge strategy is TIES:
- Sparsification replaces the TIES trim step. Both achieve sparsification (removing low-value weights), so running both would be redundant. The sparsification density parameter controls how aggressive the combined effect is.
When the merge strategy is anything else:
- Sparsification runs as preprocessing before the merge algorithm sees the diffs.
Quality Enhancements
Controlled by:
merge_refinement(none / refine / full)
Quality enhancements are additional processing steps applied before or during the merge. They compose with sparsification — sparsification runs first, then quality enhancements process the (possibly sparsified) diffs.
None (baseline)
No additional processing. Element-wise sign voting in TIES mode.
Refine (adds three techniques)
1. DO-Merging (Decouple & Orthogonalize)
- Orthogonalizes LoRA direction vectors via Modified Gram-Schmidt
- Preserves original magnitudes while reducing directional interference
- LoRAs that were partially aligned become truly independent
2. Column-Wise Conflict Resolution (replaces element-wise)
- Instead of each weight position voting on sign direction independently, entire output neurons (rows) vote as a unit
- Preserves structural coherence — a neuron's weights work together, so their signs should be resolved together
- Falls back to element-wise for 1D tensors (biases, norms)
3. TALL-Masks (Task-Aware weight protection)
- Identifies "selfish" weights — positions where one LoRA dominates and others contribute little
- Separates selfish contributions from the consensus
- After merging, adds selfish contributions back, protecting unique features from being averaged away
Full (adds SVD alignment)
Everything from Refine, plus:
KnOTS SVD Alignment
- Concatenates all LoRA diffs column-wise:
M = [diff_1 | diff_2 | ... | diff_N] - Computes truncated SVD (rank ≤ 256)
- Reconstructs each diff in the shared singular value basis
- Makes diffs more directly comparable by aligning their representation spaces
- Falls back to CPU on GPU OOM; skips gracefully if both fail
Merge Algorithms
After preprocessing and quality enhancements, the actual merge algorithm runs. The algorithm was selected in Step 4.
Weighted Sum
output = Σ (strength_i × diff_i)
The simplest merge: plain addition. Used when only one LoRA touches a prefix (no conflict possible). Also available as additive optimization mode to force it everywhere.
Properties:
- Preserves every contribution exactly
- No information loss
- Fully compressible by SVD (linear operation)
Weighted Average
output = Σ (strength_i / Σ_strengths × diff_i)
Normalized addition. Fair blending when LoRAs agree on direction.
Properties:
- Prevents magnitude inflation from stacking
- Works well for compatible LoRAs (low conflict)
- Fully compressible by SVD (linear operation)
TIES (Trim, Elect Sign, Disjoint Merge)
A three-step process specifically designed for sign conflicts:
Step 1 — Trim: Keep only top-k% weights by magnitude
(or: use DARE/DELLA sparsification instead)
│
▼
Step 2 — Elect Sign: For each position, vote on the majority sign direction
┌─ frequency: each LoRA gets one vote
└─ total: votes weighted by magnitude
│
▼
Step 3 — Disjoint Merge: Average only the LoRAs that agree with the elected sign
Contributors opposing the majority are excluded for this position
Properties:
- Resolves sign conflicts without cancellation
- Nonlinear (trim + sign election) → produces full-rank results
- Lossy SVD compression (can't perfectly capture nonlinear output)
SLERP (Spherical Linear Interpolation)
Magnitude-preserving directional blend:
- For 2 LoRAs: standard SLERP formula along the geodesic on a hypersphere
- For 3+ LoRAs: iterative pairwise SLERP, sorted by descending weight (strongest anchors direction)
- Falls back to linear interpolation for near-parallel vectors (where SLERP is numerically unstable)
- Final magnitude corrected to match weighted average of input norms
Properties:
- Preserves weight magnitude (no inflation or deflation)
- Smooth directional interpolation
- Best for low-conflict, similar-magnitude LoRAs
Consensus Merging
Three-stage algorithm for highly similar LoRAs:
Stage 1 — Fisher-Proxy Importance Weighting:
weight each parameter by |diff|² as importance proxy
numerator = Σ(diff_i × strength_i × |diff_i|²)
denominator = Σ(|strength_i| × |diff_i|²)
output = numerator / denominator
Stage 2 — MAGIC Calibration:
rescale merged result so L2 norm matches
weighted average of input norms
Stage 3 — MonoSoup Spectral Cleanup:
SVD-based entropy filtering on 2D+ tensors
keeps effective rank based on singular value entropy
removes noise while preserving signal
Properties:
- Best for highly similar LoRAs (high cosine similarity)
- Importance-weighted → parameters that matter more get more influence
- Spectral cleanup removes merge noise
How Algorithms Interact
The key to understanding the optimizer is knowing that these algorithms compose in a fixed order, not as alternatives:
┌──────────────────────────────────────────────┐
│ COMPOSITION ORDER │
│ │
│ 1. Key Normalization (if enabled) │
│ ↓ │
│ 2. Architecture Preset applied │
│ ↓ (sets density/strength thresholds) │
│ 3. Auto-Strength (if enabled) │
│ ↓ (adjusts strength scalars) │
│ 4. Quality: KnOTS alignment (if maximum) │
│ ↓ (transforms diff vectors) │
│ 5. Quality: DO-Merging (if enhanced+) │
│ ↓ (orthogonalizes directions) │
│ 6. Sparsification (if enabled) │
│ ↓ (zeros out weights) │
│ ├─ TIES mode? → replaces trim step │
│ └─ Other modes? → preprocessing │
│ 7. Quality: TALL-masks (if enhanced+) │
│ ↓ (separates selfish weights) │
│ 8. Merge Algorithm │
│ ↓ (combines diffs) │
│ 9. TALL-masks re-add (if enhanced+) │
│ ↓ (restores selfish weights) │
│ 10. SVD Compression (if enabled) │
│ ↓ (reduces to low-rank) │
│ 11. Dtype downcast to native precision │
│ ↓ (reduces memory footprint) │
│ 12. VRAM budget placement (GPU or CPU) │
│ ↓ │
│ 13. Patch application to model │
└──────────────────────────────────────────────┘
Important interactions:
| Combination | Interaction |
|---|---|
| DARE/DELLA + TIES | Sparsification replaces TIES trim (no double-sparsification) |
| DARE/DELLA + other modes | Sparsification runs as preprocessing before merge |
| Conflict-aware + per_prefix | Conflict mask computed per-prefix using local conflict data |
| KnOTS + TIES | KnOTS aligns diffs in shared basis before TIES sign voting |
| DO-Merging + any mode | Orthogonalization happens after KnOTS but before sparsification |
| TALL-masks + any mode | Selfish weights separated before merge, added back after |
| Column-wise + TIES | Replaces element-wise sign voting (structural coherence) |
| Auto-strength + everything | Only adjusts strength scalars — all algorithms see adjusted strengths |
| Architecture preset + behavior profile | Preset sets numeric thresholds, profile sets strategy logic — independent |
Step 6 — Post-Processing
SVD Patch Compression
Controlled by:
patch_compression(smart / aggressive / disabled),svd_device(gpu / cpu)
After merging, each prefix produces a full-rank diff tensor. For a 4096×4096 weight, that's ~64MB per key (vs ~0.5MB for a rank-32 LoRA patch). SVD compression re-factors these back to low-rank:
full_rank_diff ──► truncated SVD ──► lora_up (rank R) + lora_down (rank R)
The compression rank is automatically set to the sum of all input LoRA ranks. Three rank-32 LoRAs → rank-96 compressed patch. This is enough to represent the full merge without quality loss for linear operations.
| Mode | Compresses | Quality |
|---|---|---|
smart (default) |
weighted_sum and weighted_average prefixes only | Lossless — linear operations are exactly representable |
aggressive |
Everything including TIES | Lossy on TIES — nonlinear ops (trim, sign election) produce full-rank results |
disabled |
Nothing | No loss, but ~32× more RAM |
Dtype Downcasting
After merge and optional SVD compression, patches are automatically downcast to the native dtype of the input LoRA weights (typically float16 or bfloat16). Merging internally uses float32 for numerical precision, but the final patches don't need that precision. This reduces memory footprint significantly — a float32 patch is 2× the size of a float16 patch.
VRAM Budget & Patch Placement
Controlled by:
vram_budget(0.0 – 1.0)
By default (vram_budget=0.0), all merged patches live on CPU. The model patcher moves them to GPU on demand during sampling, which adds latency.
When vram_budget > 0, the optimizer keeps merged patches on GPU up to a fraction of available free VRAM:
usable_vram = free_vram - current_model_size
gpu_budget = usable_vram × vram_budget
Patches are placed on GPU until the budget is exhausted; remaining patches stay on CPU. This trades GPU memory for faster sampling — especially useful for iterative workflows where you sample repeatedly with the same merge.
Patch Application
The merged patches (compressed or full-rank) are applied to the model via ComfyUI's model patcher system. CLIP patches are applied separately with the clip_strength_multiplier scaling.
The optimizer outputs:
- MODEL — the patched model ready for sampling
- CLIP — the patched text encoder (if CLIP input was provided)
- STRING — detailed analysis report with per-prefix strategy map and suggested max strength
- LORA_DATA — the raw merged patches for use with Save Merged LoRA, Merged LoRA to Hook, or Merged LoRA to WanVideo
Algorithm Interaction Map
This diagram shows every algorithm in the system and how they relate:
┌─────────────────────────────────────────────────────────────────────────────┐
│ │
│ INPUT ANALYSIS STRATEGY │
│ ───── ──────── ──────── │
│ LoRA Stack ────► Key Normalization │
│ │ │
│ ▼ │
│ Architecture Detection ──► Architecture Preset │
│ │ (density/strength thresholds) │
│ ▼ │
│ Compute Diffs ──────► Sign Conflict ──► Per-Prefix │
│ │ Ratio Decision │
│ │ │ │
│ ├──────► Cosine ─────► Auto- │ │
│ │ Similarity Strength │ │
│ │ │ │ │ │
│ │ └──► Behavior │ │ │
│ │ Profile │ │ │
│ │ (strategy │ │ │
│ │ logic) │ │ │
│ │ │ │ │
│ ├──────► Magnitude ──► Sign Method │ │
│ │ Ratio Selection │ │
│ │ │ │
│ ├──────► Norms ──► Suggested Max │ │
│ │ Strength │ │
│ │ │ │
│ Discard Diffs │ │
│ │ │
│ ═══════════════════════════════════════════════════════════════════════ │
│ │ │
│ MERGE (per prefix) │ │
│ ────────────────── ▼ │
│ │
│ Recompute Diffs ──► KnOTS ──► DO-Merge ──► DARE/DELLA ──► ┌──────────┐ │
│ (with adjusted (if (if (if │ Select │ │
│ strengths) maximum) enhanced+) enabled) │ Merge: │ │
│ │ │ │ │
│ ┌─────────────┤ │ wt_sum │ │
│ │ │ │ wt_avg │ │
│ │ TALL-mask │ │ TIES │ │
│ │ separate │ │ SLERP │ │
│ │ (if enh+) │ │ consensus│ │
│ │ │ └────┬─────┘ │
│ │ ▼ │ │
│ │ Merge Execute ◄──────┘ │
│ │ │ │
│ │ TALL-mask │ │
│ │ re-add ────┘ │
│ │ │ │
│ └─────────────┘ │
│ │ │
│ ▼ │
│ SVD Compress │
│ │ │
│ ▼ │
│ Dtype Downcast │
│ │ │
│ ▼ │
│ VRAM Budget Placement │
│ │ │
│ ▼ │
│ Apply to Model │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
Node Reference
All nodes registered by the LoRA Optimizer:
| Node | Display Name | Purpose |
|---|---|---|
LoRAStack |
LoRA Stack | Build LoRA stacks by chaining |
LoRAStackDynamic |
LoRA Stack (Dynamic) | Single node with 1–10 adjustable slots, simple/advanced modes |
LoRAMergeSettings |
LoRA Merge Settings | Shared config: normalization, architecture preset, smoothing, VRAM budget |
LoRAOptimizerSettings |
LoRA Optimizer Settings | Optimizer-specific settings: auto-strength, sparsification, compression, etc. |
LoRAAutoTunerSettings |
LoRA AutoTuner Settings | Tuner-specific settings: top_n, scoring, diff cache, etc. |
LoRAOptimizerSimple |
LoRA Optimizer | Simplified optimizer with sensible defaults, accepts optional settings and tuner_data |
LoRAOptimizer |
LoRA Optimizer (Legacy) | All parameters on one node — for bridge workflow with settings_source |
LoRAAutoTuner |
LoRA AutoTuner | Automated parameter sweep to rank merge configs |
LoRAMergeSelector |
Merge Selector | Select alternative configs from AutoTuner results |
LoRAConflictEditor |
LoRA Conflict Editor | Interactive conflict analysis with per-LoRA overrides |
SaveMergedLoRA |
Save Merged LoRA | Export merged patches as standalone .safetensors |
MergedLoRAToHook |
Merged LoRA to Hook | Wrap merged patches as conditioning hooks |
LoRACompatibilityAnalyzer |
LoRA Compatibility Analyzer | Pre-merge planning: overlap analysis, grouping, optional node creation |
WanVideoLoRAOptimizer |
WanVideo LoRA Optimizer (WIP) | Optimizer variant for WanVideo models |
MergedLoRAToWanVideo |
Merged LoRA → WanVideo (WIP) | Bridge merged LORA_DATA to WanVideo wrapper models |
Node Variants
LoRA Optimizer (recommended):
Exposes model, lora_stack, output_strength, clip (optional), and clip_strength_multiplier. Accepts optional settings (from a Settings node) and tuner_data (from AutoTuner) inputs. When no settings node is connected, uses built-in defaults: auto_strength=enabled, optimization_mode=per_prefix, merge_refinement=none, patch_compression=smart, vram_budget=0.0.
LoRA Optimizer (Legacy):
All parameters on one node. Has settings_source for the AutoTuner ↔ Optimizer bridge workflow. Superseded by LoRA Optimizer + Settings nodes for new workflows.
Settings nodes (LoRA Merge Settings, LoRA Optimizer Settings, LoRA AutoTuner Settings):
Separate shared and node-specific configuration from the main optimizer. Merge Settings feeds into Optimizer Settings or AutoTuner Settings, which feed into the optimizer's settings input.
WanVideo LoRA Optimizer:
Accepts WANVIDEOMODEL instead of MODEL, skips CLIP. Defaults differ from the standard optimizer: normalize_keys=enabled (WanVideo LoRAs come from many trainers), cache_patches=disabled (video models are large), architecture_preset=dit.
Appendix: AutoTuner
The LoRA AutoTuner node automates parameter selection by sweeping all combinations and ranking them with internal merge metrics, plus an optional external evaluator hook.
Architecture
The AutoTuner runs a single Pass 1 analysis and caches the results. All parameter combinations are scored against this cached analysis without re-analyzing — making the sweep fast regardless of how many combinations are tested.
Phase 1 — Fast Heuristic Scoring
Generates a full parameter grid across all configurable dimensions:
| Parameter | Values Swept |
|---|---|
| Merge modes | weighted_average, slerp, consensus, ties |
| Sparsification | disabled, dare, della, dare_conflict, della_conflict |
| Density | 0.5, 0.7, 0.9 |
| DARE dampening | 0.0, 0.3, 0.6 |
| Quality | none, refine, full |
| Auto-strength | enabled, disabled |
| Optimization mode | per_prefix, global |
This produces 2,000+ combinations. Each is scored heuristically in ~2-5 seconds total using the analysis data from Pass 1 — no actual merges are performed. Scoring considers conflict, excess conflict, cosine similarity, subspace overlap, and magnitude distribution.
Phase 2 — Merge & Measure Top-N
The top-N configs from Phase 1 are actually merged and scored by measuring:
- Norm consistency across patches
- Effective rank (SVD-based entropy) — when
scoring_svd=enabled - Sparsity distribution (estimated via column-sampling for efficiency)
- Composite score combining all metrics, optionally blended with an external evaluator
Progress is tracked via ComfyUI's progress bar (spanning both Phase 1 analysis and Phase 2 merges). Each candidate logs merge time and scoring time for diagnostics.
Memory Management
Phase 2 is designed to avoid RAM exhaustion:
- Only the current best candidate is kept in memory
- Non-best candidates' model/clip patches are freed immediately after scoring
gc.collect()is forced after freeing each non-best candidate- Magnitude samples from Phase 1 are freed before Phase 2 begins
AutoTuner Parameters
| Parameter | Default | Effect |
|---|---|---|
top_n |
3 | How many candidates to actually merge and score |
scoring_svd |
disabled | Enable SVD-based effective rank scoring (slower, more thorough) |
scoring_device |
gpu | Where to run SVD scoring (gpu is 10-50x faster) |
scoring_speed |
turbo | Subsample prefix scoring for faster sweeps (full, fast, turbo, turbo+) |
auto_strength_floor |
-1.0 | Minimum auto-strength floor for orthogonal LoRAs (-1 = architecture default) |
decision_smoothing |
0.25 | Smooth per-prefix decision metrics toward block averages |
evaluator |
— | External evaluator hook for prompt/reference scoring |
diff_cache_mode |
auto | Reuse raw LoRA diffs across candidates (disabled, auto, ram, disk) |
diff_cache_ram_pct |
0.5 | RAM budget for auto diff caching before spilling to disk |
cache_patches |
enabled | Keep the final AutoTuner merge cached in RAM for fast re-execution |
vram_budget |
0.0 | Fraction of free VRAM for patch placement |
record_dataset |
disabled | Save analysis metrics to JSONL for threshold tuning research |
Dataset Recording
When record_dataset=enabled, the AutoTuner saves a JSONL entry to lora_optimizer_reports/autotuner_dataset.jsonl after each run. Each entry records:
- Timestamp and detected architecture
- LoRA names and analysis summary
- Per-prefix strategy distribution
- All top-N configs with their measured scores
This data is used for refining architecture presets and threshold tuning.
AutoTuner Outputs
| Output | Type | Purpose |
|---|---|---|
MODEL |
MODEL | Patched model from the top-ranked config |
CLIP |
CLIP | Patched text encoder |
report |
STRING | Ranked results with scores and parameters |
analysis_report |
STRING | Full optimizer analysis report for the top-ranked config |
TUNER_DATA |
TUNER_DATA | All ranked configs for Merge Selector |
LORA_DATA |
LORA_DATA | Merged patches for Save Merged LoRA / Merged LoRA to Hook |
Both AutoTuner and Merge Selector are marked as OUTPUT_NODE = True for ComfyUI's output recording and re-execution optimization.
AutoTuner Workflow
LoRA Stack ──► LoRA AutoTuner ──► MODEL + CLIP + Report + Analysis Report + TUNER_DATA + LORA_DATA
│ │
▼ ▼
Merge Selector Save Merged LoRA
(pick alternative)
│
▼
MODEL + CLIP + LORA_DATA
Appendix: Conflict Editor
The LoRA Conflict Editor node provides interactive analysis and override control:
- Loads all LoRAs and computes pairwise conflict ratios
- Auto-suggests per-LoRA conflict modes:
- < 15% average conflict →
all - 15–40% →
low_conflict - > 40% →
high_conflict
- < 15% average conflict →
- Allows manual override of conflict modes and merge strategy
- Outputs enriched stack + merge strategy override for the optimizer
Conflict Editor Workflow
LoRA Stack ──► LoRA Conflict Editor ──► Enriched LORA_STACK ──► LoRA Optimizer
│ ▲
└── merge_strategy (STRING) ─────────────────────┘
(strategy override)
Appendix: Per-LoRA Controls
Conflict Modes
Per-LoRA control over where each LoRA's contributions are applied:
| Mode | Behavior | Use Case |
|---|---|---|
all (default) |
Apply everywhere | Normal merging |
low_conflict |
Only where this LoRA agrees with the majority | Conservative — keep only safe contributions |
high_conflict |
Only where this LoRA disagrees with the majority | Force dominance in contested regions |
Key Filters
Per-LoRA control over which weight prefixes each LoRA contributes to:
| Filter | Behavior | Use Case |
|---|---|---|
all (default) |
All prefixes | Normal merging |
shared_only |
Only prefixes present in 2+ LoRAs | Strip variant-specific keys (e.g., I2V keys from a mixed LoRA) |
unique_only |
Only prefixes present in exactly 1 LoRA | Extract lightweight variant-specific adapters |
Appendix: Output Nodes
Save Merged LoRA
Exports the optimizer's merged result as a standalone .safetensors file. Connects to LORA_DATA from LoRA Optimizer, AutoTuner, or Merge Selector.
| Option | Default | Effect |
|---|---|---|
save_folder |
first configured LoRA folder | Choose which configured ComfyUI LoRA directory to save into |
filename |
merged_lora |
File name relative to save_folder. Subdirectories allowed; traversal blocked |
save_rank |
0 (auto) | 0 = use each layer's existing rank. Non-zero = force this rank via SVD compression |
bake_strength |
enabled | Bakes output_strength into saved weights so the LoRA reproduces your merge at strength 1.0 |
Merged LoRA to Hook
Wraps merged patches as a conditioning hook (HOOKS) for per-conditioning LoRA application:
- Per-prompt LoRA — apply different merged LoRAs to positive vs negative conditioning
- Scheduled application — combine with hook keyframes for step-specific LoRA
- Regional conditioning — apply LoRA to specific image regions
- Preserving the base model — keep MODEL unpatched while using the merge through hooks
Merged LoRA → WanVideo
Bridges LORA_DATA to a WANVIDEOMODEL. Handles the _orig_mod. key mismatch from torch.compile and injects patches in the format WanVideo's model loader expects.
Appendix: Supported Architectures
| Architecture | Key Detection Patterns | Special Handling |
|---|---|---|
| FLUX | double_blocks, single_blocks |
Sliced QKV offsets, multi-trainer unification |
| SDXL | lora_te1_, input_blocks |
Text encoder + UNet unified |
| Z-Image (Lumina2) | diffusion_model.layers.N.attention |
Fused QKV split/re-fuse, Musubi Tuner format |
| Wan 2.1/2.2 | blocks.N with self_attn/ffn |
LyCORIS / diffusers / Musubi / Fun LoRA / finetrainer unified, RS-LoRA alpha fix |
| LTX Video | adaln_single, attn1/attn2 |
Trainer format unification |
| Qwen-Image | img_mlp/txt_mlp/img_mod/txt_mod |
Dual-stream key unification |
Supported trainers (auto-normalized when normalize_keys=enabled):
Kohya, AI-Toolkit, LyCORIS, Musubi Tuner, diffusers/PEFT, Fun LoRA, finetrainer
Supported LoRA formats:
Standard LoRA, LoCon (with lora_mid), LyCORIS, diffusers/PEFT format, RS-LoRA (with alpha fix)
References
- TIES-Merging: Yadav et al., NeurIPS 2023 — Trim, Elect Sign, Disjoint Merge
- DARE: Yu et al., ICML 2024 — Drop And REscale
- DAREx: ICLR 2025 — Dampened DARE rescaling
- DELLA: Deep et al., 2024 — Magnitude-aware dropout
- KnOTS: Ramé et al., 2024 — SVD alignment for model merging
- TALL-masks: Wang et al., 2024 — Task-aware selfish weight protection
- ZipLoRA: Shah et al., 2025 — Column-wise structural sparsity (inspiration)
- DO-Merging: arXiv 2505.15875 — Decouple & Orthogonalize