ethanfel-ComfyUI-LoRA-Optimizer/docs/wiki/Tips-and-Troubleshooting.md

# Tips and Troubleshooting

Best practices, common pitfalls, and solutions for working with the LoRA Optimizer.

---

## What to Merge (and What NOT to Merge)

### Good Candidates for the Optimizer

- **Style LoRAs** — art styles, aesthetic LoRAs, color grading
- **Character LoRAs** — character appearance, costume, features
- **Concept LoRAs** — objects, poses, compositions
- **Detail LoRAs** — texture, sharpness, quality enhancement

These LoRAs modify the model's creative outputs and benefit from intelligent conflict resolution.

### Do NOT Put These in the Optimizer Stack

| LoRA Type | Why Not | What to Do Instead |
|-----------|---------|-------------------|
| **Distillation** (LCM, Lightning, Turbo, Hyper) | Precisely calibrated for step reduction; merging breaks the calibration | Apply via standard **Load LoRA** node upstream |
| **DPO LoRAs** | Trained for preference alignment; weights are calibrated as a unit | Apply via **Load LoRA** upstream |
| **Edit/Instruction LoRAs** (Qwen edit, Klein edit) | Modify fundamental model behavior, not style | Apply via **Load LoRA** upstream |
| **ControlNet LoRAs** | Separate control mechanism, not a style merge | Use ControlNet nodes |

If you must include an edit LoRA in the stack, use `additive` optimization mode and disable sparsification to avoid weight trimming.

---

## Understanding the Analysis Report

### Block Strategy Map

```
--- Block Strategy Map ---
  input_blocks.0   ====  sum  1 LoRA (6x)
  input_blocks.4   ----  avg  12% conflict (6x)
  middle_block.1   ####  TIES 42% conflict (6x)
  output_blocks.3  ----  avg  8% conflict (6x)
  output_blocks.8  ====  sum  1 LoRA (6x)
  Legend: ==== sum (single LoRA)  ---- avg (compatible)  #### TIES (conflict)
```

- `====` (sum) — Only one LoRA touches these layers. Full strength, no dilution.
- `----` (avg) — Multiple LoRAs, but they mostly agree. Fair blending.
- `####` (TIES) — Significant sign conflicts. TIES resolves them via trim/elect/merge.

A healthy merge shows a mix of all three. If everything is `####`, your LoRAs are very different — consider whether they should be merged at all.

### Suggested Max Output Strength

The report shows a suggested maximum for `output_strength`. Going above this value may cause oversaturation. The ceiling is calculated from the magnitude ratio of the merged LoRAs and capped per architecture:
- UNet models: max 3.0
- DiT models: max 5.0
- LLM models: max 3.0

### Auto-Strength Adjustments

When `auto_strength=enabled`, the report shows original vs adjusted strengths:
```
--- Auto-Strength Adjustment ---
  style_lora: 1.0 -> 0.63
  detail_lora: 0.8 -> 0.51
  Scale factor: 0.63
  Avg pairwise cosine similarity: 0.312
```

The scale factor tells you how much the optimizer reduced strengths. A low factor means the LoRAs reinforce each other heavily.

---

## Memory Optimization

### For Large Models (Video, High-Res)

| Setting | Recommended Value | Why |
|---------|------------------|-----|
| `cache_patches` | disabled | Don't keep merge in RAM between runs |
| `patch_compression` | aggressive | Compress everything, even TIES (slight quality loss) |
| `free_vram_between_passes` | enabled | Release GPU cache between analysis and merge |
| `vram_budget` | 0.0 | Keep all patches on CPU |

### For Iterative Workflows (Quick Experimentation)

| Setting | Recommended Value | Why |
|---------|------------------|-----|
| `cache_patches` | enabled | Instant re-execution with same settings |
| `patch_compression` | smart | Exact low-rank path on linear ops; only nonlinear paths use SVD compression |
| `vram_budget` | 0.3–0.5 | Keep some patches on GPU for faster sampling |

### Peak VRAM Usage

The optimizer's two-pass architecture usually keeps peak memory near one active target group at a time, but the exact peak still depends on layer size, overlap count, and enabled quality/compression steps. The merged result can still dominate memory. Use `patch_compression` and `vram_budget` to control where the result lives.

---

## Common Issues

### "My merge looks worse than simple stacking"

1. **Check auto-strength** — if enabled, it may be reducing strengths too aggressively for your use case. Try disabling it.
2. **Check the strategy map** — if most prefixes use TIES, the LoRAs may conflict too much for a clean merge. Consider using fewer LoRAs or adjusting strengths.
3. **Try enhanced quality** — `merge_refinement=refine` adds DO-orthogonalization and TALL-mask protection.
4. **Try the AutoTuner** — let it rank parameter combinations by merge metrics or your external evaluator.

### "Results are too subtle / LoRA effect is weak"

1. **Increase `output_strength`** — but stay below the suggested max
2. **Disable auto-strength** — if it's reducing strengths too much
3. **Check key overlap** — if LoRAs don't share keys, there's nothing to merge and the optimizer passes them through at full strength already
4. **Check key filter** — `shared_only` or `unique_only` may be filtering out keys you want

### "Results are oversaturated / blown out"

1. **Enable auto-strength** — prevents compounding from stacking
2. **Lower `output_strength`** — check the suggested max in the report
3. **Lower individual LoRA strengths** — especially for LoRAs trained at high learning rates
4. **Use `optimization_mode=per_prefix`** — ensures non-conflicting regions aren't over-merged

### "Key normalization picks the wrong architecture"

This can happen if LoRA keys are ambiguous. Set `architecture_preset` manually to the correct value (`sd_unet`, `dit`, or `llm`) to override auto-detection.

### "LoRAs from different trainers don't merge"

Enable `normalize_keys` — this remaps keys from different trainers to a canonical format. Without it, the optimizer sees no key overlap between differently-named weights.

### "AutoTuner runs out of memory"

- Lower `top_n` (fewer candidates to merge)
- Disable `scoring_svd` (skip SVD-based effective rank scoring)
- Set `patch_compression=aggressive`
- Set `vram_budget=0.0`

### "Merge Selector says stack has changed"

The Merge Selector validates that the LoRA stack hasn't changed since the AutoTuner ran (via hash comparison). If you modified the stack (changed LoRAs, strengths, or conflict modes), re-run the AutoTuner.

### "AutoTuner memory hit but I updated a LoRA file"

The memory cache key uses filename + modification time + file size. If you replaced a LoRA file in place without the timestamp changing (e.g., via some tools that preserve mtime), the cache won't detect the change. Set `memory_mode=clear_and_run` to force a fresh sweep and update the cache.

### "AutoTuner is slow / I want to reuse analysis from a previous run"

Enable `memory_mode=auto` (the default). After the first sweep completes, results are cached in `models/autotuner_memory/`. Subsequent runs with the same LoRAs and settings replay the cached result in a single merge pass (~2–5s).

The analysis cache (`.analysis.json` files) also persists pairwise conflict metrics between runs — even without a full memory hit, Phase 1 analysis is skipped if the LoRA set matches.

### "AutoTuner crashed or was interrupted mid-run"

The in-progress analysis is checkpointed automatically. On the next run with the same LoRA set, analysis resumes from where it left off. The checkpoint file is deleted automatically after a successful run.

---

## Performance Tips

### Speed

- `svd_device=gpu` is 10–50x faster than CPU for SVD compression
- `cache_patches=enabled` gives instant re-execution when only changing `output_strength`
- The AutoTuner caches Pass 1 analysis — only Phase 2 merges take time on repeated runs with the same LoRA set
- Enable `memory_mode=auto` to skip the entire sweep on subsequent runs with the same LoRAs and settings
- `scoring_device=gpu` speeds up AutoTuner quality scoring

### Quality

- `merge_refinement=refine` with `della_conflict` sparsification is a strong default for quality
- `merge_refinement=full` adds KnOTS-inspired SVD alignment — strongest quality mode for critical merges
- `strategy_set=full` gives the optimizer the full algorithm repertoire
- Use the Conflict Editor to understand pairwise conflicts before choosing settings

---

## Limitations

1. **The optimizer only sees its own stack.** LoRAs applied via upstream Load LoRA nodes are invisible — they stack additively on top. This can cause overexposure if you have upstream LoRAs + optimizer LoRAs touching the same weights.

2. **Fully baked checkpoints are invisible.** If a checkpoint already has LoRA weights baked in, the optimizer can't detect or account for them.

3. **TIES produces full-rank results.** The nonlinear operations (trim + sign election) mean TIES output can't be perfectly captured by low-rank SVD compression. Use `patch_compression=smart` to only compress linear operations losslessly.

4. **Conflict-aware sparsification needs overlap.** If LoRAs don't share any keys, there are no conflicts to detect and conflict-aware variants behave the same as disabled.