nodes/siraxe-ComfyUI-WanVideoWrapper_QQ

Fork 0

Files

T

siraxe 77bb15a9f8 experimental_node

2025-10-20 02:32:15 +01:00

29 KiB

Raw Permalink Blame History

LoraExtractKJ Comprehensive Documentation

📖 Overview

The LoraExtractKJ class is a ComfyUI node designed to extract LoRA (Low-Rank Adaptation) weights from the differences between a fine-tuned model and an original base model. This node provides multiple extraction algorithms and adaptive methods to create efficient LoRA files that can be used to apply model modifications with minimal storage overhead.

📍 Location

File: ComfyUI-KJNodes/nodes/lora_nodes.py
Line: 153-205
Category: KJNodes/lora

⚙️ Input Parameters

Parameter	Type	Default	Range/Options	Description
`finetuned_model`	MODEL	-	-	The fine-tuned model from which to extract LoRA weights
`original_model`	MODEL	-	-	The original base model to compare against
`filename_prefix`	STRING	"loras/ComfyUI_extracted_lora"	-	Output file path and name prefix
`rank`	INT	8	1-4096	Target rank for standard LoRA or maximum rank limit for adaptive methods
`lora_type`	ENUM	"standard"	6 options	Extraction method type (see details below)
`algorithm`	ENUM	"svd_linalg"	2 options	SVD algorithm to use
`lowrank_iters`	INT	7	1-100	Number of subspace iterations for lowrank SVD
`output_dtype`	ENUM	"fp16"	3 options	Output data type for the LoRA file
`bias_diff`	BOOLEAN	True	-	Whether to extract bias differences
`adaptive_param`	FLOAT	0.15	0.0-1.0	Parameter for adaptive extraction methods
`clamp_quantile`	BOOLEAN	True	-	Whether to clamp values using quantile method

🎨 LoRA Types and Methods

1. Standard LoRA (`"standard"`)

Description: Traditional fixed-rank LoRA extraction
Use Case: When you want consistent, predictable file sizes and performance
Algorithm: Uses SVD to decompose weight differences into low-rank matrices
Parameters: rank directly controls the LoRA rank

2. Full LoRA (`"full"`)

Description: Stores complete weight differences without rank reduction
Use Case: Maximum quality preservation, larger file sizes
Algorithm: No decomposition, stores raw weight differences
Parameters: rank parameter is ignored

3. Adaptive Ratio (`"adaptive_ratio"`)

Description: Dynamically selects rank based on singular value ratios
Algorithm: Keeps singular values > adaptive_param * max(singular_value)
Use Case: Balanced quality and size based on importance thresholds
Default: 0.15 (15% of maximum singular value)

4. Adaptive Quantile (`"adaptive_quantile"`)

Description: Selects rank based on cumulative distribution of singular values
Algorithm: Keeps singular values until cumulative sum reaches adaptive_param * total_sum
Use Case: Statistical approach to rank selection
Default: 0.15 (15% of total singular value sum)

5. Adaptive Energy (`"adaptive_energy"`)

Description: Rank selection based on energy preservation
Algorithm: Keeps components until cumulative energy reaches adaptive_param * total_energy
Use Case: Preserving specific percentage of mathematical energy
Default: 0.15 (15% of total energy)

6. Adaptive Frobenius (`"adaptive_fro"`)

Description: Rank selection based on Frobenius norm preservation
Algorithm: Keeps components until Frobenius norm reaches adaptive_param² * total_norm
Use Case: Maintaining specific percentage of matrix information
Default: 0.15 (15% Frobenius norm retention)

🔬 SVD Algorithms

1. Full SVD (`"svd_linalg"`)

Method: torch.linalg.svd()
Pros: Most accurate, complete decomposition
Cons: Slower, higher memory usage
Recommended: For high-quality extractions and adaptive methods

2. Low-rank SVD (`"svd_lowrank"`)

Method: torch.svd_lowrank()
Pros: Faster, lower memory usage
Cons: Less accurate, approximation
Limitation: Only works with standard LoRA type
Recommended: For quick extractions when speed is prioritized

💾 Data Types

Type	Description	Use Case
`fp16`	16-bit floating point	Balance of quality and file size
`bf16`	16-bit bfloat floating point	Better range than fp16, similar size
`fp32`	32-bit floating point	Maximum precision, larger files

🎛️ Advanced Parameters

`adaptive_param` Usage by LoRA Type

LoRA Type	`adaptive_param` Meaning	Recommended Range
`adaptive_ratio`	Ratio to maximum singular value	0.01-0.3
`adaptive_quantile`	Quantile of singular values	0.1-0.9
`adaptive_energy`	Energy preservation percentage	0.8-0.99
`adaptive_fro`	Frobenius norm retention	0.8-0.99

`clamp_quantile`

Purpose: Prevents extreme values in LoRA weights
Method: Clips values to [-99th percentile, +99th percentile]
Effect: More stable training and inference
Recommendation: Keep enabled unless you need full dynamic range

`lowrank_iters`

Purpose: Number of iterations for low-rank SVD approximation
Effect: Higher values = better accuracy, slower processing
Range: 1-100
Default: 7 (good balance)

📁 Output File Naming

The node generates files with this naming convention:

{filename}_rank_{rank/type}_{dtype}_{counter:05}_.safetensors

Examples:

my_lora_rank_8_fp16_00001_.safetensors
my_lora_rank_adaptive_ratio_0.15_fp16_00001_.safetensors

⭐ Recommended Configurations

🏆 High Quality Extraction

lora_type: "adaptive_fro"
adaptive_param: 0.95
algorithm: "svd_linalg"
output_dtype: "fp32"
clamp_quantile: true

⚖️ Balanced Quality/Size

lora_type: "adaptive_ratio"
adaptive_param: 0.1
algorithm: "svd_linalg"
output_dtype: "fp16"
clamp_quantile: true

⚡ Fast Extraction

lora_type: "standard"
rank: 16
algorithm: "svd_lowrank"
lowrank_iters: 4
output_dtype: "fp16"

💎 Maximum Quality

lora_type: "full"
output_dtype: "fp32"

📦 Small File Size

lora_type: "adaptive_ratio"
adaptive_param: 0.02
algorithm: "svd_linalg"
output_dtype: "fp16"
clamp_quantile: true

📋 Usage Guidelines

🎯 When to Use Each LoRA Type

Character/Stylistic LoRAs: adaptive_ratio with 0.05-0.1
Concept LoRAs: adaptive_fro with 0.85-0.95
Technical LoRAs: adaptive_energy with 0.9-0.95
Testing/Prototyping: standard with rank 4-8
Archive/Backup: full with fp32

⚡ Performance Considerations

Memory Usage: Low-rank SVD uses ~50% less memory
Processing Time: Low-rank SVD is 2-3x faster
File Size: Adaptive methods can reduce size by 40-80%
Quality Loss: Usually <5% for well-chosen parameters

🔧 Common Issues and Solutions

Issue	Cause	Solution
Large file sizes	`adaptive_param` too high	Reduce to 0.05-0.15
Quality loss	`adaptive_param` too low	Increase to 0.15-0.3
Slow processing	Using full SVD on large models	Switch to `svd_lowrank`
Training instability	Extreme weight values	Enable `clamp_quantile`
Memory errors	Large model with full SVD	Use `svd_lowrank` or reduce model size

📊 Comparison of LoRA Types

📈 Quality vs File Size Trade-offs

Quality
  ▲
  │   Full LoRA ──────────────────────●
  │                                   │
  │   Adaptive Fro (0.95) ────────────●
  │                                   │
  │   Adaptive Energy (0.9) ──────────●
  │                                   │
  │   Adaptive Quantile (0.5) ────────●
  │                                   │
  │   Adaptive Ratio (0.15) ────────●
  │                                   │
  │   Standard (Rank 16) ────────────●
  │                                   │
  │   Standard (Rank 8) ─────────────●
  │                                   │
  │   Standard (Rank 4) ────────────●
  └───────────────────────────────────► File Size
    Small                           Large

Memory Usage (MB) for 1B Model:
Full LoRA:       ████████████████████████████████████ ~4000MB
Standard R8:     ████████████ ~500MB
Adaptive Ratio:  ████████ ~300MB
Adaptive Energy: █████████ ~400MB

⏱️ Processing Speed Comparison

Full LoRA
┌─────────────────────────────────────────────────────────┐
│ Direct weight difference extraction                     │
│ Time: ████████████████████████████████████████ 10x     │
│ Memory: ████████████████████████████████████████ High   │
└─────────────────────────────────────────────────────────┘

Standard LoRA (SVD Low-rank)
┌─────────────────────────────────────────────────────────┐
│ Fast SVD approximation                                 │
│ Time: ████████████ 1x (Fastest)                       │
│ Memory: ████████ Low                                   │
└─────────────────────────────────────────────────────────┘

Adaptive Methods (Full SVD)
┌─────────────────────────────────────────────────────────┐
│ Complete SVD with rank analysis                        │
│ Time: ████████████████████████ 3-5x                    │
│ Memory: ████████████████ Medium                        │
└─────────────────────────────────────────────────────────┘

🔬 Algorithm Details

📊 SVD Decomposition Process

Step 1: Compute Weight Difference
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Fine-tuned     │    │   Original      │    │   Weight Diff   │
│  Model Weights  │ ── │  Model Weights  │ =  │  (W_diff)       │
│   (W_finetuned) │    │   (W_original)  │    │                 │
└─────────────────┘    └─────────────────┘    └─────────────────┘

Step 2: Apply SVD Decomposition
                      ┌─────────────────────────────────┐
                      │          W_diff Matrix          │
                      │     [out_dim × in_dim]          │
                      └─────────────────────────────────┘
                                     │
                                     ▼ SVD
┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
│   U Matrix      │  │   S Matrix      │  │   Vh Matrix     │
│ [out_dim × rank]│  │  [rank × rank]  │  │ [rank × in_dim] │
│   (Left)        │  │ (Singular Values)│  │   (Right)       │
└─────────────────┘  └─────────────────┘  └─────────────────┘

Step 3: Create LoRA Matrices
┌─────────────────┐           ┌─────────────────┐
│   U × √S         │           │   √S × Vh        │
│                 │           │                 │
│  lora_up        │           │  lora_down      │
│ [out_dim × rank]│           │ [rank × in_dim] │
└─────────────────┘           └─────────────────┘
         │                               │
         └──────────────┬────────────────┘
                        ▼
               ┌─────────────────┐
               │    LoRA Pair    │
               │ (Efficient Low  │
               │  Rank Approx.)  │
               └─────────────────┘

Compute weight difference: W_diff = W_finetuned - W_original
Apply SVD: W_diff = U @ S @ Vh
For LoRA: lora_up = U @ sqrt(S), lora_down = sqrt(S) @ Vh
Clamp values if enabled
Save with proper data type conversion

🎯 Adaptive Rank Selection

The adaptive methods automatically determine the optimal rank based on different criteria:

Singular Values Spectrum
┌─────────────────────────────────────────────────────────────────┐
│ S₁ ● S₂ ● S₃ ● S₄ ● S₅ ● S₆ ● S₇ ● S₈ ● S₉ ● S₁₀ ● ... ● Sₙ      │
│ │   │   │   │   │   │   │   │   │    │          │              │
│ └───┘   └───┘   └───┘   └───┘   └───┘    └────────────────┘      │
│   ●       ●       ●       ●       ●            negligible        │
│ important  important  important  important  important    values    │
└─────────────────────────────────────────────────────────────────┘
            ▲           ▲           ▲           ▲
            │           │           │           │
         Standard    Ratio     Quantile    Energy/Fro
          Rank R    Threshold  Distribution  Preservation

📏 Adaptive Ratio Method

Threshold = adaptive_param × max(singular_value)

Singular Values: [100, 95, 88, 72, 45, 23, 8, 3, 1, 0.5, ...]
                │││││││││││││││││││││││││││││││││││││││││
Threshold:      15 (0.15 × 100) ────────────────────────┘
Selected:        ↑↑↑↑↑↑↑  (rank = 7)
Discarded:                    ↑↑↑↑↑↑↑↑↑↑↑↑↑

📊 Adaptive Quantile Method

Cumulative Sum Distribution: ████████████░░░░░░░░░░░░░░░░░░░░
                              └────────────┘
                          0.15 × total_sum

Singular Values: [100, 95, 88, 72, 45, 23, 8, 3, 1, 0.5, ...]
Cumulative:     [100, 195, 283, 355, 400, 423, 431, 434, 435, ...]
Target:         0.15 × 450 = 67.5
Selected:        ↑↑↑ (rank = 3, since 283 > 67.5)

⚡ Adaptive Energy Method

Energy (S²): [10000, 9025, 7744, 5184, 2025, 529, 64, 9, 1, 0.25, ...]
Cumulative:  ████████████████████████████████████████████████░░░░░░
             └────────────────────────────────┘
         0.95 × total_energy (95% energy preservation)

Selected:  ↑↑↑↑↑↑↑ (rank = 7, preserves 95% of total energy)

🎯 Adaptive Frobenius Method

Frobenius Norm: √(S₁² + S₂² + ... + Sᵣ²) = 0.95 × √(S₁² + ... + Sₙ²)

Progressive:   ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
               └──────────────────────────────────────┘
           95% of total Frobenius norm

Selected:      ↑↑↑↑↑↑↑↑↑ (rank = 9, preserves 95% of matrix information)

Ratio: Keeps components above relative importance threshold
Quantile: Statistical distribution-based selection
Energy: Preserves mathematical energy content
Frobenius: Maintains matrix information content

Each method provides feedback showing the actual rank selected and preservation metrics.

⚙️ Technical Implementation

🔑 Key Functions

extract_lora(): Core SVD extraction logic (lines 14-99)
calc_lora_model(): Model-wide processing (lines 102-151)
LoraExtractKJ.save(): Main execution method (lines 181-205)

💾 Memory Management

Models are moved to CPU after processing
GPU memory is cleared between operations
Progress bars show extraction status
Batch processing for large models

🛡️ Error Handling

Skips 5D tensors (patch embeddings)
Handles failed SVD decompositions gracefully
Validates algorithm compatibility
Provides detailed logging

🔗 Integration Notes

🎨 ComfyUI Integration

Located in "KJNodes/lora" category
Output node (saves to disk)
Supports standard ComfyUI model handling
Compatible with all model formats

📁 File Format

Outputs in SafeTensors format
Includes metadata about extraction parameters
Compatible with standard LoRA loaders
Supports automatic dtype conversion

🎯 Decision Tree for Choosing LoRA Type

┌─────────────────────────────────────────────────────────────────┐
│                    What's your priority?                       │
└─────────────────────────────────────────────────────────────────┘
                                │
                ┌───────────────┼───────────────┐
                │               │               │
        ┌───────▼──────┐ ┌──────▼──────┐ ┌─────▼───────┐
        │   Quality     │ │    Speed    │ │ File Size   │
        └───────┬──────┘ └──────┬──────┘ └─────┬───────┘
                │               │               │
        ┌───────▼──────┐ ┌──────▼──────┐ ┌─────▼───────┐
        │ Full LoRA     │ │ Standard +  │ │ Adaptive    │
        │ (fp32)        │ │ Low-rank    │ │ Ratio       │
        │               │ │ SVD         │ │ (0.02-0.05) │
        └───────────────┘ └─────────────┘ └─────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                   What type of content?                        │
└─────────────────────────────────────────────────────────────────┘
                                │
                ┌───────────────┼───────────────┐
                │               │               │
        ┌───────▼──────┐ ┌──────▼──────┐ ┌─────▼───────┐
        │   Characters  │ │   Concepts  │ │   Style     │
        └───────┬──────┘ └──────┬──────┘ └─────┬───────┘
                │               │               │
        ┌───────▼──────┐ ┌──────▼──────┐ ┌─────▼───────┐
        │ Adaptive     │ │ Adaptive    │ │ Adaptive    │
        │ Ratio        │ │ Fro (0.95)  │ │ Energy      │
        │ (0.05-0.1)   │ │             │ │ (0.9-0.95)  │
        └───────────────┘ └─────────────┘ └─────────────┘

🎯 Memory Flow Diagram

GPU Memory Management During Extraction:

┌─────────────────────────────────────────────────────────────────┐
│                    Initial State                               │
│  ┌─────────────┐    ┌─────────────┐                           │
│  │ Fine-tuned  │    │   Original  │                           │
│  │   Model     │    │    Model    │                           │
│  └─────────────┘    └─────────────┘                           │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Weight Difference                            │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────────┐     │
│  │ Fine-tuned  │ ── │   Original  │ =  │   W_diff        │     │
│  │   Model     │    │    Model    │    │   (GPU)         │     │
│  └─────────────┘    └─────────────┘    └─────────────────┘     │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼ SVD Processing
┌─────────────────────────────────────────────────────────────────┐
│                 LoRA Extraction (GPU)                          │
│  ┌─────────────┐    ┌─────────────┐                           │
│  │  lora_up    │    │ lora_down   │                           │
│  │   (GPU)     │    │   (GPU)     │                           │
│  └─────────────┘    └─────────────┘                           │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼ Memory Optimization
┌─────────────────────────────────────────────────────────────────┐
│                  Transfer to CPU & Save                        │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────────┐     │
│  │  lora_up    │    │ lora_down   │    │   SafeTensors   │     │
│  │   (CPU)     │    │   (CPU)     │    │     File        │     │
│  └─────────────┘    └─────────────┘    └─────────────────┘     │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                  GPU Memory Cleared                            │
│                     Ready for Next Layer                       │
└─────────────────────────────────────────────────────────────────┘

🎯 Conclusion

The LoraExtractKJ node provides a comprehensive solution for LoRA extraction with multiple algorithms and adaptive methods. The choice of parameters depends on your specific use case, balancing quality, file size, and processing speed. The adaptive methods are particularly useful for automatically optimizing the rank selection based on the actual content of the weight differences.

Key Takeaways:

🎯 Quality matters: Use adaptive methods for better preservation
⚡ Speed matters: Use low-rank SVD for faster processing
💾 Size matters: Adaptive ratio can significantly reduce file sizes
🔧 Flexibility: Different methods suit different content types

29 KiB Raw Permalink Blame History Unescape Escape