silveroxides-ComfyUI-ModelUtils/docs/DEVELOPMENT.md

# ComfyUI-ModelUtils - Complete Project History

## Version Timeline

| Version | Commits | Key Features |
|---------|---------|--------------|
| 0.1.1 | `70d6c13`-`d67cbca` | Initial nodes (MetaKeys) |
| 0.1.2 | `6b662a4` | Icon assets |
| 0.1.4 | `cbe3502` | Key renaming/pruning nodes |
| 0.1.5 | `78b3198` | Model merging nodes |
| 0.1.6 | `31989be` | Embedding nodes, namespace fixes |
| 0.1.7 | `b6fae8d` | Mismatch handling, layer filtering, V3 migration |
| **0.2.0** | `3df073f` | LoRA extraction, memory optimization, pinned memory |

---

## Phase 1: Initial Release (v0.1.1)

**Commit:** `70d6c13` - "Add first nodes, toml and publish"

### Files Created
- `nodes/metakeys.py` - Get model metadata and layer keys
- `nodes/utils.py` - Shared utilities
- `__init__.py` - Node registration
- `pyproject.toml` - Package configuration

### Nodes
- `ModelMetaKeys` - Get diffusion model metadata
- `CheckpointMetaKeys` - Get checkpoint metadata
- `TextEncoderMetaKeys` - Get text encoder metadata
- `LoRAMetaKeys` - Get LoRA metadata
- `EmbeddingMetaKeys` - Get embedding metadata

---

## Phase 2: Key Manipulation (v0.1.4)

**Commit:** `4f08fe6` - "add key renaming and key pruning nodes"

### Files Created
- `nodes/renamekeys.py` - Rename tensor keys
- `nodes/prunekeys.py` - Remove tensor keys by pattern

### Nodes (5 variants each for model types)
- `ModelRenameKeys`, `CheckpointRenameKeys`, `TextEncoderRenameKeys`, `LoRARenameKeys`, `EmbeddingRenameKeys`
- `ModelPruneKeys`, `CheckpointPruneKeys`, `TextEncoderPruneKeys`, `LoRAPruneKeys`, `EmbeddingPruneKeys`

---

## Phase 3: Model Merging (v0.1.5)

**Commit:** `7959370` - "Add model merging nodes"

### Files Created
- `nodes/merger.py` - Main merging logic
- `nodes/merger_ops.py` - Merge operation modes
- `nodes/merger_utils.py` - `MemoryEfficientSafeOpen` class
- `docs/merger_2_model_modes.md` - Documentation
- `docs/merger_3_model_modes.md` - Documentation

### Nodes
- `CheckpointTwoMerger`, `DiffusionModelTwoMerger`, `TextEncoderTwoMerger`, `LoRATwoMerger`
- `CheckpointThreeMerger`, `DiffusionModelThreeMerger`, `TextEncoderThreeMerger`, `LoRAThreeMerger`

### Merge Modes (TWO_MODEL_MODES)
- Weighted Sum, Add Difference, Smooth Add Difference
- Train Difference, Sum Twice, Triple Sum
- Euclidean Add Difference, Distribution Crossover, Ties
- Slerp, Lerp, SVD Supermerger, SVD LoRA Extraction

### Merge Modes (THREE_MODEL_MODES)
- Add Difference (A + (B - C) × α)

### MemoryEfficientSafeOpen (initial)
```python
class MemoryEfficientSafeOpen:
    def __init__(self, filename, device='cpu', mmap_mode=True)
    def keys() -> List[str]
    def get_tensor(key) -> Tensor
    # mmap for zero-copy reads
```

---

## Phase 4: Embedding Nodes (v0.1.6)

**Commit:** `0d035ef` - "Add embedding nodes"

### Additions
- Added embedding variants to all node types
- Namespace conflict fixes (`68f22fa`, `5ea63ab`)
- Refactored names for universality (`232dc76`)

---

## Phase 5: Mismatch Handling (v0.1.7)

**Commits:** `19bc047` → `de27186`

### Features Added
- `mismatch_mode` parameter: skip/zeros/error
- Layer filtering with regex patterns
- `exclude_patterns` - Keep model A only (no merge)
- `discard_patterns` - Remove from output entirely

### Files Modified
- `merger.py` - Added mismatch handling logic
- `merger_ops.py` - Added `MissingTensorBehavior` enum, `MissingTensorError`

---

## Phase 6: V3 API Migration (v0.1.7)

**Commit:** `a3116cb` - "migrate to v3 schema and add LoRA extraction node"

### Changes
- Migrated all nodes from legacy API to `comfy_api.latest.io`
- `io.Schema`, `io.ComfyNode`, `io.NodeOutput`
- Added initial LoRA extraction node (diff-based)

---

## Phase 7: LyCORIS Extraction (pre-0.2.0)

**Commits:** `92d3c55` → `18679cc`

### Files Created
- `nodes/lycoris_extract.py` - LyCORIS-based extraction
- `nodes/lora_extract.py` - Simple diff extraction

### Initial Implementation
- Used `lycoris-lora` external dependency
- `lycoris.utils.extract_linear`, `extract_conv`
- Added `requirements.txt` with `lycoris-lora`

### Nodes Split (`18679cc`)
- `LyCORISExtractFixed`, `LyCORISExtractThreshold`
- `LyCORISExtractRatio`, `LyCORISExtractQuantile`

---

## Phase 8: Native SVD Port (0.2.0)

**Commits:** `6d7b2d1` → `eef7377`

### Removed
- `lycoris-lora` dependency
- `lycoris_extract.py`
- `lora_extract.py`

### NEW: [lora_extract_svd.py](file:///f:/ComfyUI/custom_nodes/ComfyUI-ModelUtils/nodes/lora_extract_svd.py)

**8 Rank Selection Functions:**
```python
_index_sv_fixed(S, dim)           # Fixed rank
_index_sv_threshold(S, threshold) # Absolute threshold
_index_sv_ratio(S, ratio)         # Relative to max
_index_sv_cumulative(S, target)   # Cumulative sum %
_index_sv_fro(S, target)          # Frobenius norm
_index_sv_knee(S)                 # Knee detection
_index_sv_cumulative_knee(S)      # Cumsum knee
_index_sv_rel_decrease(S, tau)    # Ratio drop
```

**SVD Functions:**
```python
_svd_extract_linear_lowrank(weight, rank, device, clamp_quantile)
_svd_extract_linear(weight, mode, mode_param, device, max_rank, clamp_quantile)
_svd_extract_conv(weight, mode, mode_param, device, max_rank, clamp_quantile)
_detect_fused_layer(key, shape)  # QKV/MLP detection
_extract_chunked_layer(...)      # Chunked extraction
```

**Nodes:**
- `LoRAExtractFixed` - Uses `svd_lowrank` (10x faster)
- `LoRAExtractRatio` - Relative threshold
- `LoRAExtractQuantile` - Cumulative target
- `LoRAExtractKnee` - Auto knee detection
- `LoRAExtractFrobenius` - Norm preservation

---

## Phase 9: Memory Management (0.2.0)

**Commit:** `6d7b2d1` - "refactor nodes and implement memory management"

### NEW: [device_utils.py](file:///f:/ComfyUI/custom_nodes/ComfyUI-ModelUtils/nodes/device_utils.py)

Integrates with `comfy.model_management`:

```python
@dataclass
class DeviceCapabilities:
    device_type, total_vram_gb, free_vram_gb, total_ram_gb, free_ram_gb
    cpu_cores, recommended_workers, max_tensor_size_gb
    supports_pinned_memory, supports_async_streams

get_device_capabilities(device) -> DeviceCapabilities
estimate_model_size(path) -> float  # GB
can_fit_in_memory(paths, device, safety_factor) -> (bool, str)
prepare_for_large_operation(gb, device) -> bool  # Calls free_memory()
cleanup_after_operation()  # gc.collect() + soft_empty_cache()
get_optimal_workers(size_gb) -> int
```

### Updates to Existing Nodes
- `metakeys.py` - Header-only reading (no tensor loading)
- `renamekeys.py` - `MemoryEfficientSafeOpen` + memory mgmt
- `prunekeys.py` - `MemoryEfficientSafeOpen` + memory mgmt
- `merger.py` - Memory prep/cleanup

---

## Phase 10: Pinned Memory (0.2.0)

**Commit:** `b770786` - "Add pinned memory for faster merging and extraction"

### [merger_utils.py](file:///f:/ComfyUI/custom_nodes/ComfyUI-ModelUtils/nodes/merger_utils.py) Enhancements

```python
# Context manager
class PinnedTensor:
    def __enter__(self) -> Tensor  # Pins memory
    def __exit__(...)              # Unpins

# Transfer helper (2-3x faster)
def transfer_to_gpu_pinned(tensor, device='cuda', dtype=None) -> Tensor

# MemoryEfficientSafeOpen additions
def get_tensor_to_gpu(key, device, dtype)  # Pinned transfer
def keys_sorted_by_offset() -> List[str]   # Sequential I/O
def get_tensors_parallel(keys, workers, sort_by_offset) -> Dict  # Parallel
def get_tensor_as_dict(key) -> Dict        # JSON config parsing
```

---

## Phase 11: SVD Optimization (0.2.0)

**Commit:** `65c1ae3` - "add svd_lowrank for fixed rank extraction"

### Changes
- Added `_svd_extract_linear_lowrank()` using `torch.svd_lowrank`
- Routes fixed mode to lowrank for **10x speedup**
- Changed `chunk_large_layers` default to `False`

---

## Phase 12: Bug Fixes (0.2.0)

**Commit:** `eef7377` - "Added .contiguous() to saves"

### Fix
Non-contiguous tensor error from SVD slices:
```python
tensor.to(dtype).cpu().contiguous()  # Added .contiguous()
```

---

## Current File Structure

```
ComfyUI-ModelUtils/
├── __init__.py
├── pyproject.toml
├── requirements.txt (empty - no external deps)
├── README.md
├── docs/
│   ├── merger_2_model_modes.md
│   └── merger_3_model_modes.md
├── nodes/
│   ├── utils.py
│   ├── metakeys.py
│   ├── renamekeys.py
│   ├── prunekeys.py
│   ├── merger.py
│   ├── merger_ops.py
│   ├── merger_utils.py
│   ├── device_utils.py
│   └── lora_extract_svd.py
└── assets/
    └── icon.png
```

---

## Performance Summary

| Feature | Improvement |
|---------|-------------|
| Fixed-rank SVD | **10x** (svd_lowrank) |
| CPU→GPU transfer | **2-3x** (pinned memory) |
| Metadata reading | **100x+** (header only) |
| Parallel I/O | **2-4x** (ThreadPoolExecutor) |
| Memory management | Auto-unload models before heavy ops |
| **LoRA Resize Fast** | **~91x** (merge-to-base extraction) |

---

## Phase 13: Fast LoRA Resize (2026-01-11)

### Summary
Added fast LoRA resize via base model extraction path, achieving ~91x speedup.

### Changes
- Added `resize_lora_via_base()` function using extraction code path
- Added 4 new fast resize nodes requiring base model input

### Files Modified

| File | Changes |
|------|---------|
| `nodes/lora_resize.py` | Added `resize_lora_via_base()`, 4 new node classes |
| `__init__.py` | Registered new nodes |

### New Nodes
- `LoRAResizeViaBaseFixed` - Fast fixed rank resize
- `LoRAResizeViaBaseRatio` - Fast SV ratio resize  
- `LoRAResizeViaBaseFrobenius` - Fast Frobenius resize
- `LoRAResizeViaBaseCumulative` - Fast cumulative resize