Previously the partial file was only deleted at post-analysis cleanup,
so early returns (memory-cache hit, no target groups) left it on disk
permanently. Now the partial is deleted as soon as its contents are
absorbed into cached_analysis; the on_prefix_done callback recreates
it incrementally for any new prefixes computed in the same run.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Correct log prefix in _analysis_partial_save except block from
[AutoTuner Analysis Cache] to [AutoTuner Analysis Partial]
- Add using_partial flag so partial is promoted to full cache even
when all prefixes were already cached (stuck-state fix)
- Move _analysis_partial_delete outside the new_entries guard so it
is always called (safe no-op when file doesn't exist)
- Add test_partial_promoted_to_full_cache_when_all_prefixes_already_done
to TestAnalysisPartialLifecycle exercising the stuck-state fix
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On startup, fall back to partial checkpoint when no full cache exists.
Define _on_prefix_done callback that writes partial file after each
fresh prefix. Pass callback to _run_group_analysis. Delete partial on
successful full cache write. Delete partial on clear_and_run. Also fix
log prefixes in _analysis_partial_load/save (carry-forward from Task 1
review).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Callers can now pass on_prefix_done=lambda prefix, entry: ... to receive
a callback after each freshly-computed prefix is extracted. The callback
fires only on cache misses and only when track_new_entries=True, so all
existing call sites (which pass neither argument) are unaffected.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On first run cached_analysis is None, causing new_analysis_entries to
stay empty and the cache to never be saved. Replace the
'cached_analysis is not None' guard with an explicit 'track_new_entries'
flag so auto_tune opts in while other callers of _run_group_analysis
avoid the extraction overhead.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove trailing {} from _analyze_target_group early return (was 9 elements,
_collect_analysis_result only unpacks 8)
- Use clip_strength as effective strength in _extract_for_analysis_cache when
is_clip is True, matching the scaling used in _analyze_target_group
- Remove unused _strength_signs variable in auto_tune
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The fast linear merge path (_build_exact_linear_patch) produces LoRAAdapters
with rank = sum of all input ranks. SaveMergedLoRA was extracting these
weights directly without applying any SVD compression, making save_rank and
patch_compression settings ineffective for weighted_sum/weighted_average merges.
Fix: compress LoRAAdapter patches when current_rank > target_rank (where
target = save_rank when explicit, or fallback_rank in auto mode). Skip
LoCon patches (mid != None) which require special reshape handling.
Also improve auto-mode logging to explain rank growth (n_loras × avg_rank)
and direct users to set save_rank explicitly to control output file size.
Fix test discovery: __init__.py relative import fails when pytest imports it
as a standalone module. Guard with try/except to prevent all tests from
failing during collection/setup. Add pyproject.toml pytest config and
conftest.py to configure test discovery.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- _extract_lora_svd: return None early for 1D tensors (bias vectors) to
prevent torch.linalg.svd crash; add test_1d_delta_returns_none
- LoRAExtractFromModel: add optional base_clip/finetuned_clip inputs;
fix model_lora_keys_clip to use cond_stage_model instead of model;
merge CLIP state dicts into base_sd/fine_sd when both CLIPs provided;
warn when only one CLIP is connected; remove redundant import comfy.lora
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds the LoRAExtractFromModel ComfyUI node that extracts a baked-in LoRA
from a finetuned model by SVD-decomposing per-layer weight deltas against
a clean base model. Outputs both LORA_STACK and LORA_DATA for downstream use.
Also registers the node in NODE_CLASS_MAPPINGS and NODE_DISPLAY_NAME_MAPPINGS.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds module-level _extract_lora_svd() that SVD-decomposes a weight
delta into (lora_up, lora_down, alpha), supporting fixed-rank and
auto-rank (energy threshold) modes with a singular value noise floor.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The diff cache was using 50% of available RAM with no absolute cap,
causing 54GB+ usage on WAN. Now capped at 16GB — excess spills to
disk automatically. First candidate is still slow but subsequent
ones benefit from disk-cached diffs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Log process RSS and diff cache size after each Phase 2 candidate
to diagnose RAM growth on large models. Also del measured dict
to free metric temporaries sooner.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Free tensor references eagerly in scoring loop, reorder sparsity
before SVD, avoid unnecessary intermediate allocations. Helps
prevent RAM accumulation during Phase 2 on video models.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Avoid keeping a full fp32 copy of each diff tensor for the entire
scoring iteration. Norm and sparsity now work on original dtype,
fp32 conversion only happens inside the SVD block and is freed
immediately after Gram matrix computation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use Gram matrix + eigvalsh instead of full SVD for tensors with
min dimension > 32. Same effective rank result but much smaller
memory footprint, critical for video models like WAN.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When LoRAs have significantly different ranks (2x+ ratio), the analysis
report now suggests trying scoring_svd='lora_rank' or 'full'.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Procrustes alignment as faster alternative to KnOTS SVD alignment
(pure rotation via batched_procrustes from kernel.py, whiten=False)
- Batched deferred eigvalsh for LoRA effective rank computation
(group by rank, single eigvalsh per group vs 240 individual SVDs)
- Unified scoring_svd dropdown: disabled/merge_quality/lora_rank/full
- Import batched_procrustes from kernel.py with graceful fallback
Based on PR #4.
Co-Authored-By: Ethanfel <ethan.fel@ts-pc.fr>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Integrate kernel.py into scoring and rank estimation pipelines.
Based on PR #4 approach, with hardening:
- _triton_svdvals helper with internal transpose and kernel dispatch
- Cap at N<=32 to avoid eigh serialization cliff
- Warning log when kernel.py exists but fails to load
- NameError guard for frozen/zipimport environments
- Fix >2D tensor fallback slice axis
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When switching memory_mode (e.g. auto → disabled), the in-memory
cache key was unchanged, causing stale memory-hit results to be
returned instead of running a fresh sweep.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allows picking which ranked config to apply (default=1, top-ranked)
directly on the AutoTuner node, without needing a separate Merge
Selector node. Changing selection skips the full sweep — only the
final merge is re-run.
Also bumps version to 1.4.7.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Skip expensive parameter sweeps (30-120s) by caching tuning results
to disk and replaying the best config in a single merge (~2-5s).
Memory is keyed by LoRA combo (order-independent hash) + tuning
settings + algorithm version. Stored as individual JSON files in
models/autotuner_memory/ with atomic writes for crash safety.
Adds memory_mode input (disabled/auto/read_only/clear_and_run) to
LoRAAutoTuner, LoRAAutoTunerSettings, and LoRAOptimizerSimple nodes.
Defaults to disabled for backward compatibility.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The _build_exact_linear_patch fast path concatenates LoRA factors
directly without expanding to dense diffs. It looks up patches via
aliases (string prefixes), but virtual LoRAs from sub-merges store
patches keyed by target keys (tuples). The alias lookup silently
missed virtual LoRAs, producing patches containing only the non-virtual
LoRA — making sub-merged characters invisible in the output.
Also removes diagnostic logging added in the previous commit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Traces sub-merge patch extraction, target group resolution, and diff
lookup to diagnose why character LoRAs have zero visible effect in
(1+2)+3 formula output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- _detected_arch was not restored when formula resolved to single item,
causing wrong architecture preset in the fall-through path
- saved_arch moved before _autotune_resolve_tree so both >= 2 and == 1
paths can restore it
- Single-LoRA path uses resolved preset_key when coming from formula
- Sub-merge auto_tune no longer overwrites _autotuner_cache, preventing
loss of cached results from prior runs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- _skip_qkv_refusion=True from sub-merges was leaking into the final
merge in _execute_merge_tree, causing Z-Image models to output
unfused QKV patches that ComfyUI cannot apply
- Sub-merges may overwrite _detected_arch (e.g. all-virtual stacks);
save and restore it around _resolve_tree_to_stack
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a target group has multiple aliases (e.g., virtual LoRA target key +
real LoRA prefix), a miss on one alias but hit on another was incorrectly
counted as a skip. Now skip_count only increments when ALL aliases fail
for a given LoRA, fixing inflated "Skipped keys" in merge formula reports.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sub-merge virtual LoRA patches must stay unfused (to_q/to_k/to_v) so
the outer merge can pair them with other LoRAs' unfused keys. Re-fusion
only happens in the final merge. Fixes 150 skipped keys from shape
mismatch when merging fused virtual patches with unfused style LoRA.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sub-merge patches are stored on CPU but the final merge's analysis runs
on CUDA. Move precomputed diffs to the compute device in all three diff
lookup paths (_prepare_group_diffs, _process_prefix, _analyze_prefix)
to prevent device mismatch in subspace overlap computation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
