# 2-Model Merger Documentation

---

## Weight-Sum
> `A * (1 - α) + B * α`. A simple linear interpolation between two models.

**Models Used:** A, B
**Parameters:**
- **Alpha:** Interpolation weight. `0.0` is 100% Model A, `1.0` is 100% Model B.

---

## Comparative-Interpolation
> Interpolates between A and B based on the relative differences in their tensor values. Creates a unique blend by deciding how much of each model to use for each individual weight.

**Models Used:** A, B
**Parameters:**
- **Alpha:** Controls the interpolation curve's strength and shape.
- **Beta:** Switches the interpolation style. `0.0` focuses on similarity, `1.0` focuses on difference.
- **Gamma:** Mixes between a randomized (binomial) interpolation at `0.0` and a smooth (linear) one at `1.0`.

---

## Power-Up (DARE)
> Adds the unique capabilities of Model B to Model A using the Drop and Rescale (DARE) technique. This implementation handles shape mismatches between models by padding and uses a randomized dropout mask.

**Models Used:** A, B
**Parameters:**
- **Alpha:** The dropout rate ($p$). This is the proportion of delta parameters from Model B that are randomly set to zero.
- **Beta:** A final multiplier for the rescaled difference before it's added to Model A.
- **Rescaling Logic:** Remaining weights are automatically rescaled by $1/(1-p)$ as per the DARE paper to approximate the original embeddings.

---

## Enhanced Man Interp
> Sophisticated interpolation between values from A and B depending on their difference relative to other values, with manual threshold control.

**Models Used:** A, B
**Parameters:**
- **Alpha:** Interpolation strength.
- **Beta:** Lower mean threshold for filtering differences.
- **Gamma:** Upper mean threshold for filtering differences.
- **Delta:** Smoothness factor (mix between randomized mask and powered differences).

---

## Enhanced Auto Interp
> Automated version of the enhanced interpolation mode that dynamically calculates thresholds based on mean differences.

**Models Used:** A, B
**Parameters:**
- **Alpha:** Interpolation strength.
- **Beta:** Threshold adjustment factor.
- **Gamma:** Smoothness factor.

---

## Weight-Sum Cutoff
> A linear interpolation mode that only applies the merge to weights whose differences fall within a specific threshold range.

**Models Used:** A, B
**Parameters:**
- **Alpha:** Interpolation weight (multiplier for the difference).
- **Beta:** Upper threshold for the difference cutoff.
- **Gamma:** Lower threshold for the difference cutoff.

---

## SVD LoRA Extraction
> Extracts the difference between Model B and Model A and saves it as a new LoRA file. **This mode outputs a LoRA, not a full model.** The output file will be saved to your `loras` directory.

**Models Used:** A (Tuned Model), B (Base Model)
**Parameters:**
- **Alpha:** The Rank (dimension) for the LoRA's standard layers.
- **Beta:** The Rank (dimension) specifically for 3x3 convolution layers.
- **Gamma:** The clamp quantile for weight values. `0.99` is a typical value to prevent outlier weights from dominating.

---

## Layer Mismatch Handling

When merging models with different layer structures (e.g., fine-tuned models with added/removed layers, or LoRAs with partial layer coverage), the `mismatch_mode` parameter controls behavior:

| Mode | Behavior |
|------|----------|
| `skip` | Missing layers in B use A's values **(default)** |
| `zeros` | Missing layers in B are treated as zeros |
| `error` | Fail if any layer is missing |

**Common scenarios:**
- Merging a fine-tuned model with added/removed layers
- Combining LoRA-extracted differences with base models
- Cross-architecture experiments

**Note:** Extra layers in Model B that don't exist in Model A are currently ignored. The output will always have the same layer structure as Model A.

---

## Layer Filtering (Regex Patterns)

Use regex patterns to control which layers are merged:

### `exclude_patterns`
Layers matching any pattern will **keep Model A's values only** (no merge).

### `discard_patterns`
Layers matching any pattern will be **removed entirely** from the output.

**Pattern format:**
- Whitespace-separated regex patterns
- Patterns use **substring matching** (not full match)
- Example: `text_model lora` matches any key containing "text_model" OR "lora"
- Example: `layer\.[0-5]\.` matches layers 0-5 using regex syntax

**Pattern Examples:**
| Pattern | Matches |
|---------|---------|
| `text_model` | All text encoder layers |
| `\.norm` | All normalization layers |
| `attn\.(q\|k\|v)` | Query, key, value attention weights |
| `block\.[0-9]\.` | Blocks 0-9 |