Adds module-level _extract_lora_svd() that SVD-decomposes a weight
delta into (lora_up, lora_down, alpha), supporting fixed-rank and
auto-rank (energy threshold) modes with a singular value noise floor.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The diff cache was using 50% of available RAM with no absolute cap,
causing 54GB+ usage on WAN. Now capped at 16GB — excess spills to
disk automatically. First candidate is still slow but subsequent
ones benefit from disk-cached diffs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Log process RSS and diff cache size after each Phase 2 candidate
to diagnose RAM growth on large models. Also del measured dict
to free metric temporaries sooner.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Free tensor references eagerly in scoring loop, reorder sparsity
before SVD, avoid unnecessary intermediate allocations. Helps
prevent RAM accumulation during Phase 2 on video models.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Avoid keeping a full fp32 copy of each diff tensor for the entire
scoring iteration. Norm and sparsity now work on original dtype,
fp32 conversion only happens inside the SVD block and is freed
immediately after Gram matrix computation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use Gram matrix + eigvalsh instead of full SVD for tensors with
min dimension > 32. Same effective rank result but much smaller
memory footprint, critical for video models like WAN.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When LoRAs have significantly different ranks (2x+ ratio), the analysis
report now suggests trying scoring_svd='lora_rank' or 'full'.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Procrustes alignment as faster alternative to KnOTS SVD alignment
(pure rotation via batched_procrustes from kernel.py, whiten=False)
- Batched deferred eigvalsh for LoRA effective rank computation
(group by rank, single eigvalsh per group vs 240 individual SVDs)
- Unified scoring_svd dropdown: disabled/merge_quality/lora_rank/full
- Import batched_procrustes from kernel.py with graceful fallback
Based on PR #4.
Co-Authored-By: Ethanfel <ethan.fel@ts-pc.fr>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Integrate kernel.py into scoring and rank estimation pipelines.
Based on PR #4 approach, with hardening:
- _triton_svdvals helper with internal transpose and kernel dispatch
- Cap at N<=32 to avoid eigh serialization cliff
- Warning log when kernel.py exists but fails to load
- NameError guard for frozen/zipimport environments
- Fix >2D tensor fallback slice axis
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When switching memory_mode (e.g. auto → disabled), the in-memory
cache key was unchanged, causing stale memory-hit results to be
returned instead of running a fresh sweep.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allows picking which ranked config to apply (default=1, top-ranked)
directly on the AutoTuner node, without needing a separate Merge
Selector node. Changing selection skips the full sweep — only the
final merge is re-run.
Also bumps version to 1.4.7.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Skip expensive parameter sweeps (30-120s) by caching tuning results
to disk and replaying the best config in a single merge (~2-5s).
Memory is keyed by LoRA combo (order-independent hash) + tuning
settings + algorithm version. Stored as individual JSON files in
models/autotuner_memory/ with atomic writes for crash safety.
Adds memory_mode input (disabled/auto/read_only/clear_and_run) to
LoRAAutoTuner, LoRAAutoTunerSettings, and LoRAOptimizerSimple nodes.
Defaults to disabled for backward compatibility.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The _build_exact_linear_patch fast path concatenates LoRA factors
directly without expanding to dense diffs. It looks up patches via
aliases (string prefixes), but virtual LoRAs from sub-merges store
patches keyed by target keys (tuples). The alias lookup silently
missed virtual LoRAs, producing patches containing only the non-virtual
LoRA — making sub-merged characters invisible in the output.
Also removes diagnostic logging added in the previous commit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Traces sub-merge patch extraction, target group resolution, and diff
lookup to diagnose why character LoRAs have zero visible effect in
(1+2)+3 formula output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- _detected_arch was not restored when formula resolved to single item,
causing wrong architecture preset in the fall-through path
- saved_arch moved before _autotune_resolve_tree so both >= 2 and == 1
paths can restore it
- Single-LoRA path uses resolved preset_key when coming from formula
- Sub-merge auto_tune no longer overwrites _autotuner_cache, preventing
loss of cached results from prior runs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- _skip_qkv_refusion=True from sub-merges was leaking into the final
merge in _execute_merge_tree, causing Z-Image models to output
unfused QKV patches that ComfyUI cannot apply
- Sub-merges may overwrite _detected_arch (e.g. all-virtual stacks);
save and restore it around _resolve_tree_to_stack
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a target group has multiple aliases (e.g., virtual LoRA target key +
real LoRA prefix), a miss on one alias but hit on another was incorrectly
counted as a skip. Now skip_count only increments when ALL aliases fail
for a given LoRA, fixing inflated "Skipped keys" in merge formula reports.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sub-merge virtual LoRA patches must stay unfused (to_q/to_k/to_v) so
the outer merge can pair them with other LoRAs' unfused keys. Re-fusion
only happens in the final merge. Fixes 150 skipped keys from shape
mismatch when merging fused virtual patches with unfused style LoRA.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sub-merge patches are stored on CPU but the final merge's analysis runs
on CUDA. Move precomputed diffs to the compute device in all three diff
lookup paths (_prepare_group_diffs, _process_prefix, _analyze_prefix)
to prevent device mismatch in subspace overlap computation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use raw tensor dtype (e.g. fp16) instead of post-.float() dtype (fp32)
to avoid 2x memory overhead in final merge patches.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prevents all-virtual final merges (e.g. (1+2)+(3+4)) from falling back
to sd_unet preset when auto-detection finds no real LoRAs to detect from.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- When sub-merge hits single-LoRA fast path (lora_data=None), pass
sub-stack items through directly instead of creating empty virtual LoRA
- Generate canonical prefix from target_key when all aliases are tuples
(Flux/WAN all-virtual merges) to prevent group collision at ordered[""]
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Guard aggregated[i] assignment in _prepare_group_diffs to skip when
virtual LoRA has no matching key (prevents None diff crashes)
- Keep tuple target keys intact in _collect_lora_prefixes for correct
Flux/sliced-weight group resolution
- Handle mixed string/tuple sorting in prefix collection and alias lists
- Restore original cache_patches setting for the final merge while
keeping sub-merges uncached
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Parse formula against full normalized stack, not filtered active_loras,
so indices match user-specified positions regardless of zero-strength items
- Add string components of tuple target keys to _build_target_groups sets
for Flux/WanVideo sliced-weight compatibility
- Preserve inner weights in parenthesized expressions (parser fix)
- Handle virtual LoRAs in _analyze_prefix with precomputed diff lookup
- Deep-copy metadata dict in _resolve_tree_to_stack to prevent mutation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Handle 3-tuple vs 5-tuple from WanVideoLoRAOptimizer.optimize_merge
- Normalize tuple target keys to strings in _collect_lora_prefixes
to prevent mixed-type sorting crashes on Flux architectures
- Add tuple fallback in precomputed diff lookups
- Preserve clip_strength through re-normalization
- Free sub-merge model clones after extracting patches
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Preserve _precomputed_diffs flag through _normalize_stack
- Handle virtual LoRA model-space keys in _collect_lora_prefixes
and _build_target_groups
- Use full target_key (including tuples) for virtual LoRA lookup
- Skip _normalize_keys for virtual LoRAs (model-space keys)
- Skip architecture detection for virtual LoRAs
- Avoid dense expansion of compressed patches in _model_to_virtual_lora
- Disable cache_patches for sub-merges to prevent cache thrashing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add _execute_merge_tree, _resolve_tree_to_stack, and
_model_to_virtual_lora methods to LoRAOptimizer. Modify optimize_merge
to detect formula metadata in the stack and delegate to the tree
executor for hierarchical sub-group merging. Add safety filter in
_normalize_stack to strip formula metadata entries.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add LoKr/LoHa key suffixes to _collect_lora_prefixes for discovery
- New _get_lokr_diff/_get_loha_diff methods compute full diffs from
Kronecker/Hadamard factors (including Tucker decomposition)
- Update _prepare_group_diffs, _process_prefix, _analyze_prefix to
fall back to LoKr/LoHa when standard LoRA extraction returns None
- Fix _build_exact_linear_patch to detect LoKr/LoHa keys and fall
through to dense path instead of silently dropping contributions
- Update analyze_and_enrich (Conflict Editor) with LoKr/LoHa support
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Single LoRA outputs raw prompt; multiple LoRAs prefix each with
[lora_name] so you know which is which.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove '# filename' prefix from prompt/description outputs
- Exclude prompt/description from metadata_info (they have dedicated outputs)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SaveMergedLoRA now writes safetensors metadata (source LoRAs, merge
settings, architecture) plus optional user prompt/description
- SaveTunerData accepts optional prompt/description, embeds in JSON
- LoadTunerData outputs prompt, description, and metadata_info strings
- AutoTuner records source_loras in tuner_data for provenance
- New LoRAMetadataReader passthrough node extracts metadata from any
safetensors LoRA in a stack (prompt, description, merge info)
- LoRAStack attaches file metadata at load time; _normalize_stack
preserves it through both tuple and dict paths
- Graceful degradation: missing metadata produces empty strings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolve conflict in lora_optimizer.py: keep delegate cache clearing
from this branch (clears AutoTuner delegate caches on model change).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The monolithic node is now legacy — the simple optimizer with
settings nodes is the recommended workflow.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The +0.1 compatibility bonus was applied to all non-opposing pairs,
including LoRAs that share no weights (like DPO LoRAs touching
different layers). This caused them to be grouped instead of solo.
Now the bonus only applies when overlap > 0, so LoRAs with no
shared weights correctly end up as solo.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
No longer an end node — user will trigger via connected outputs.
Remove ui.text and ui.images (no longer needed without OUTPUT_NODE).
Report and heatmap flow through normal STRING/IMAGE outputs.
Node creation via ui.groups still works for any executed node.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Save heatmap to temp dir and return via ui["images"] so ComfyUI
renders it directly on the node alongside the text report.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add "text" key to all ui dict returns so ComfyUI renders the
report directly on the node surface.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Solo LoRAs now create LoraLoader/LoraLoaderModelOnly nodes (based on CLIP)
- Merge groups continue creating LoRAStackDynamic nodes
- Use original strengths instead of auto-strengths for created nodes
- Add create_nodes toggle to disable automatic node creation
- Move smooth_slerp_gate from AutoTunerSettings to shared MergeSettings
- Add smooth_slerp_gate to main LoRAOptimizer and _SIMPLE_DEFAULTS
- Add decision_smoothing (0.25) to _SIMPLE_DEFAULTS
- Remove output_mode from AutoTunerSettings (hardcode "merge")
- Rewrite all settings node tooltips for clarity
- Update tests for new settings structure
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create a new LoRAMergeSettings node holding the 6 shared parameters
(normalize_keys, architecture_preset, auto_strength_floor,
decision_smoothing, vram_budget, cache_patches) that were duplicated
across LoRAOptimizerSettings and LoRAAutoTunerSettings.
Both mode nodes now accept an optional MERGE_SETTINGS input and fall
back to identical defaults when unconnected — output dicts are unchanged
so execute_simple needs zero modifications.
Also fixes:
- v1 scoring: de-dent composite_score recomputation so it always runs
under scoring_formula=="v1", not only when sparsification is enabled
- RAM leak: clear _autotuner_delegate caches on model change and when
switching between advanced/autotuner modes in LoRAOptimizerSimple
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
