Adds module-level _extract_lora_svd() that SVD-decomposes a weight
delta into (lora_up, lora_down, alpha), supporting fixed-rank and
auto-rank (energy threshold) modes with a singular value noise floor.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds TestExtractLoRAFromDelta test class and _load_module() helper
to support TDD for the upcoming LoRAExtractFromModel node.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The diff cache was using 50% of available RAM with no absolute cap,
causing 54GB+ usage on WAN. Now capped at 16GB — excess spills to
disk automatically. First candidate is still slow but subsequent
ones benefit from disk-cached diffs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Log process RSS and diff cache size after each Phase 2 candidate
to diagnose RAM growth on large models. Also del measured dict
to free metric temporaries sooner.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Free tensor references eagerly in scoring loop, reorder sparsity
before SVD, avoid unnecessary intermediate allocations. Helps
prevent RAM accumulation during Phase 2 on video models.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Avoid keeping a full fp32 copy of each diff tensor for the entire
scoring iteration. Norm and sparsity now work on original dtype,
fp32 conversion only happens inside the SVD block and is freed
immediately after Gram matrix computation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use Gram matrix + eigvalsh instead of full SVD for tensors with
min dimension > 32. Same effective rank result but much smaller
memory footprint, critical for video models like WAN.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When LoRAs have significantly different ranks (2x+ ratio), the analysis
report now suggests trying scoring_svd='lora_rank' or 'full'.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Procrustes alignment as faster alternative to KnOTS SVD alignment
(pure rotation via batched_procrustes from kernel.py, whiten=False)
- Batched deferred eigvalsh for LoRA effective rank computation
(group by rank, single eigvalsh per group vs 240 individual SVDs)
- Unified scoring_svd dropdown: disabled/merge_quality/lora_rank/full
- Import batched_procrustes from kernel.py with graceful fallback
Based on PR #4.
Co-Authored-By: Ethanfel <ethan.fel@ts-pc.fr>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Log info message when Triton is not available so users know
fused N=2/N=3 kernels are disabled.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Override _displayValue getter on combo widgets to strip directory
paths, showing only the filename for readability.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Integrate kernel.py into scoring and rank estimation pipelines.
Based on PR #4 approach, with hardening:
- _triton_svdvals helper with internal transpose and kernel dispatch
- Cap at N<=32 to avoid eigh serialization cliff
- Warning log when kernel.py exists but fails to load
- NameError guard for frozen/zipimport environments
- Fix >2D tensor fallback slice axis
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fused Triton kernels for N=2,3 (3850-5488x faster than torch),
Gram-eigh hybrid for N>=4, Newton-Schulz inverse sqrt, and
subspace-preserving Procrustes alignment.
From PR #4.
When switching memory_mode (e.g. auto → disabled), the in-memory
cache key was unchanged, causing stale memory-hit results to be
returned instead of running a fresh sweep.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allows picking which ranked config to apply (default=1, top-ranked)
directly on the AutoTuner node, without needing a separate Merge
Selector node. Changing selection skips the full sweep — only the
final merge is re-run.
Also bumps version to 1.4.7.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Skip expensive parameter sweeps (30-120s) by caching tuning results
to disk and replaying the best config in a single merge (~2-5s).
Memory is keyed by LoRA combo (order-independent hash) + tuning
settings + algorithm version. Stored as individual JSON files in
models/autotuner_memory/ with atomic writes for crash safety.
Adds memory_mode input (disabled/auto/read_only/clear_and_run) to
LoRAAutoTuner, LoRAAutoTunerSettings, and LoRAOptimizerSimple nodes.
Defaults to disabled for backward compatibility.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Additional notes (Optimizer): rewrote strategy_set and merge_refinement
tips to explain they're different toolboxes, not quality rankings
- Merge Formula note (782): added formula syntax explanation
- Merge Formula note (783): added description of what the node does
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The _build_exact_linear_patch fast path concatenates LoRA factors
directly without expanding to dense diffs. It looks up patches via
aliases (string prefixes), but virtual LoRAs from sub-merges store
patches keyed by target keys (tuples). The alias lookup silently
missed virtual LoRAs, producing patches containing only the non-virtual
LoRA — making sub-merged characters invisible in the output.
Also removes diagnostic logging added in the previous commit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Traces sub-merge patch extraction, target group resolution, and diff
lookup to diagnose why character LoRAs have zero visible effect in
(1+2)+3 formula output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- _detected_arch was not restored when formula resolved to single item,
causing wrong architecture preset in the fall-through path
- saved_arch moved before _autotune_resolve_tree so both >= 2 and == 1
paths can restore it
- Single-LoRA path uses resolved preset_key when coming from formula
- Sub-merge auto_tune no longer overwrites _autotuner_cache, preventing
loss of cached results from prior runs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- _skip_qkv_refusion=True from sub-merges was leaking into the final
merge in _execute_merge_tree, causing Z-Image models to output
unfused QKV patches that ComfyUI cannot apply
- Sub-merges may overwrite _detected_arch (e.g. all-virtual stacks);
save and restore it around _resolve_tree_to_stack
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a target group has multiple aliases (e.g., virtual LoRA target key +
real LoRA prefix), a miss on one alias but hit on another was incorrectly
counted as a skip. Now skip_count only increments when ALL aliases fail
for a given LoRA, fixing inflated "Skipped keys" in merge formula reports.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sub-merge virtual LoRA patches must stay unfused (to_q/to_k/to_v) so
the outer merge can pair them with other LoRAs' unfused keys. Re-fusion
only happens in the final merge. Fixes 150 skipped keys from shape
mismatch when merging fused virtual patches with unfused style LoRA.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sub-merge patches are stored on CPU but the final merge's analysis runs
on CUDA. Move precomputed diffs to the compute device in all three diff
lookup paths (_prepare_group_diffs, _process_prefix, _analyze_prefix)
to prevent device mismatch in subspace overlap computation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use raw tensor dtype (e.g. fp16) instead of post-.float() dtype (fp32)
to avoid 2x memory overhead in final merge patches.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prevents all-virtual final merges (e.g. (1+2)+(3+4)) from falling back
to sd_unet preset when auto-detection finds no real LoRAs to detect from.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- When sub-merge hits single-LoRA fast path (lora_data=None), pass
sub-stack items through directly instead of creating empty virtual LoRA
- Generate canonical prefix from target_key when all aliases are tuples
(Flux/WAN all-virtual merges) to prevent group collision at ordered[""]
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Guard aggregated[i] assignment in _prepare_group_diffs to skip when
virtual LoRA has no matching key (prevents None diff crashes)
- Keep tuple target keys intact in _collect_lora_prefixes for correct
Flux/sliced-weight group resolution
- Handle mixed string/tuple sorting in prefix collection and alias lists
- Restore original cache_patches setting for the final merge while
keeping sub-merges uncached
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Parse formula against full normalized stack, not filtered active_loras,
so indices match user-specified positions regardless of zero-strength items
- Add string components of tuple target keys to _build_target_groups sets
for Flux/WanVideo sliced-weight compatibility
- Preserve inner weights in parenthesized expressions (parser fix)
- Handle virtual LoRAs in _analyze_prefix with precomputed diff lookup
- Deep-copy metadata dict in _resolve_tree_to_stack to prevent mutation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Handle 3-tuple vs 5-tuple from WanVideoLoRAOptimizer.optimize_merge
- Normalize tuple target keys to strings in _collect_lora_prefixes
to prevent mixed-type sorting crashes on Flux architectures
- Add tuple fallback in precomputed diff lookups
- Preserve clip_strength through re-normalization
- Free sub-merge model clones after extracting patches
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Preserve _precomputed_diffs flag through _normalize_stack
- Handle virtual LoRA model-space keys in _collect_lora_prefixes
and _build_target_groups
- Use full target_key (including tuples) for virtual LoRA lookup
- Skip _normalize_keys for virtual LoRAs (model-space keys)
- Skip architecture detection for virtual LoRAs
- Avoid dense expansion of compressed patches in _model_to_virtual_lora
- Disable cache_patches for sub-merges to prevent cache thrashing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
