silveroxides-ComfyUI-ModelUtils/docs/merger_2_model_modes.md

2-Model Merger Documentation

---

Weight-Sum

A * (1 - α) + B * α. A simple linear interpolation between two models.

Models Used: A, B Parameters:

• Alpha: Interpolation weight. 0.0 is 100% Model A, 1.0 is 100% Model B.

---

Comparative-Interpolation

Interpolates between A and B based on the relative differences in their tensor values. Creates a unique blend by deciding how much of each model to use for each individual weight.

Models Used: A, B Parameters:

• Alpha: Controls the interpolation curve's strength and shape.
• Beta: Switches the interpolation style. 0.0 focuses on similarity, 1.0 focuses on difference.
• Gamma: Mixes between a randomized (binomial) interpolation at 0.0 and a smooth (linear) one at 1.0.

---

Power-Up (DARE)

Adds the unique capabilities of Model B to Model A using the Drop and Rescale (DARE) technique. This implementation handles shape mismatches between models by padding and uses a randomized dropout mask.

Models Used: A, B Parameters:

• Alpha: The dropout rate (p). This is the proportion of delta parameters from Model B that are randomly set to zero.
• Beta: A final multiplier for the rescaled difference before it's added to Model A.
• Rescaling Logic: Remaining weights are automatically rescaled by 1/(1-p) as per the DARE paper to approximate the original embeddings.

---

Enhanced Man Interp

Sophisticated interpolation between values from A and B depending on their difference relative to other values, with manual threshold control.

Models Used: A, B Parameters:

• Alpha: Interpolation strength.
• Beta: Lower mean threshold for filtering differences.
• Gamma: Upper mean threshold for filtering differences.
• Delta: Smoothness factor (mix between randomized mask and powered differences).

---

Enhanced Auto Interp

Automated version of the enhanced interpolation mode that dynamically calculates thresholds based on mean differences.

Models Used: A, B Parameters:

• Alpha: Interpolation strength.
• Beta: Threshold adjustment factor.
• Gamma: Smoothness factor.

---

Weight-Sum Cutoff

A linear interpolation mode that only applies the merge to weights whose differences fall within a specific threshold range.

Models Used: A, B Parameters:

• Alpha: Interpolation weight (multiplier for the difference).
• Beta: Upper threshold for the difference cutoff.
• Gamma: Lower threshold for the difference cutoff.

---

SVD LoRA Extraction

Extracts the difference between Model B and Model A and saves it as a new LoRA file. This mode outputs a LoRA, not a full model. The output file will be saved to your loras directory.

Models Used: A (Tuned Model), B (Base Model) Parameters:

• Alpha: The Rank (dimension) for the LoRA's standard layers.
• Beta: The Rank (dimension) specifically for 3x3 convolution layers.
• Gamma: The clamp quantile for weight values. 0.99 is a typical value to prevent outlier weights from dominating.

---

Layer Mismatch Handling

When merging models with different layer structures (e.g., fine-tuned models with added/removed layers, or LoRAs with partial layer coverage), the mismatch_mode parameter controls behavior:

Mode	Behavior
skip	Missing layers in B use A's values (default)
zeros	Missing layers in B are treated as zeros
error	Fail if any layer is missing

Common scenarios:

• Merging a fine-tuned model with added/removed layers
• Combining LoRA-extracted differences with base models
• Cross-architecture experiments

Note: Extra layers in Model B that don't exist in Model A are currently ignored. The output will always have the same layer structure as Model A.

---

Layer Filtering (Regex Patterns)

Use regex patterns to control which layers are merged:

exclude_patterns

Layers matching any pattern will keep Model A's values only (no merge).

discard_patterns

Layers matching any pattern will be removed entirely from the output.

Pattern format:

• Whitespace-separated regex patterns
• Patterns use substring matching (not full match)
• Example: text_model lora matches any key containing "text_model" OR "lora"
• Example: layer\.[0-5]\. matches layers 0-5 using regex syntax

Pattern Examples:

Pattern	Matches
text_model	All text encoder layers
\.norm	All normalization layers
attn\.(q|k|v)	Query, key, value attention weights
block\.[0-9]\.	Blocks 0-9