Refactor loading strategies in mergers and LoRA resize nodes.

This commit is contained in:
silveroxides
2026-07-11 11:21:40 +02:00
parent 314f5ef67b
commit 9375ced5cf
4 changed files with 133 additions and 43 deletions
+9
View File
@@ -121,6 +121,15 @@ When merging models with different layer structures (e.g., fine-tuned models wit
---
## Dtype Preservation & Override
By default, the merger preserves the highest precision `dtype` for each key found across all active source models (Model A and Model B) and the requested `save_dtype`.
- **Preserve Higher Precision**: If a key is `float32` in any of the source models, but you requested `save_dtype="bf16"` or `"fp16"`, that key will be saved as `float32` anyway. This prevents degrading critical high-precision layers (such as normalization layers or layers with restored outliers).
- **Explicit Override**: If you prefer to force the entire model to be saved strictly as the selected `save_dtype` (e.g. for complete downcasting to `bf16`), toggle `override_dtype` to `True`.
---
## Layer Filtering (Regex Patterns)
Use regex patterns to control which layers are merged:
+9
View File
@@ -67,6 +67,15 @@ When merging models with different layer structures, the `mismatch_mode` paramet
---
## Dtype Preservation & Override
By default, the merger preserves the highest precision `dtype` for each key found across all active source models (Model A, B, and C) and the requested `save_dtype`.
- **Preserve Higher Precision**: If a key is `float32` in any of the active source models, but you requested `save_dtype="bf16"` or `"fp16"`, that key will be saved as `float32` anyway. This prevents degrading critical high-precision layers (such as normalization layers or layers with restored outliers).
- **Explicit Override**: If you prefer to force the entire model to be saved strictly as the selected `save_dtype` (e.g. for complete downcasting to `bf16`), toggle `override_dtype` to `True`.
---
## Layer Filtering (Regex Patterns)
Use regex patterns to control which layers are merged: