Remove unnecessary files: filter scripts, temp files, and old release notes
This commit is contained in:
ussoewwin
@@ -1,364 +0,0 @@
|
||||
# Applying LoRA to Qwen Image (AWQ) img_mod.1 / txt_mod.1 (modulation) Layers - Complete Explanation
|
||||
|
||||
## Overview
|
||||
|
||||
In Qwen Image models, each transformer block has `img_mod.1` and `txt_mod.1` modulation layers. These layers are linear layers that generate shift/scale/gate parameters for AdaNorm.
|
||||
|
||||
In Nunchaku's AWQ quantized models, these layers are implemented as quantized linear layers called `AWQW4A16Linear`. Since the standard LoRA application method (directly modifying weights) cannot be applied to quantized layers, we adopt a method of adding LoRA during the forward pass.
|
||||
|
||||
However, as a result of implementation and testing, applying LoRA to `img_mod.1` and `txt_mod.1` causes image noise. Therefore, we have implemented a **safety mechanism that skips LoRA application to these layers by default**.
|
||||
|
||||
## Detailed Code Explanation
|
||||
|
||||
### 1. Environment Variable Loading
|
||||
|
||||
**File**: `nunchaku_code/lora_qwen.py`
|
||||
**Line Numbers**: 23-28
|
||||
|
||||
```python
|
||||
# Safety switch:
|
||||
# QwenImage's modulation linears (`img_mod.1` / `txt_mod.1`) are extremely sensitive because their
|
||||
# output is reshaped into shift/scale/gate parameters. With AWQ quantization (AWQW4A16Linear),
|
||||
# applying LoRA here often results in severe noise. Default to skipping these two layers.
|
||||
# Users can override by setting env var `QWENIMAGE_LORA_APPLY_AWQ_MOD=1`.
|
||||
_APPLY_AWQ_MOD = str(os.getenv("QWENIMAGE_LORA_APPLY_AWQ_MOD", "0")).strip().lower() in ("1", "true", "yes", "y", "on")
|
||||
```
|
||||
|
||||
**Explanation**:
|
||||
|
||||
`_APPLY_AWQ_MOD` is a global variable that controls whether to apply LoRA to AWQ modulation layers.
|
||||
|
||||
1. `os.getenv("QWENIMAGE_LORA_APPLY_AWQ_MOD", "0")` loads the environment variable. The default value is "0" (disabled).
|
||||
2. `str(...).strip().lower()` normalizes the string (removes leading/trailing whitespace and converts to lowercase).
|
||||
3. `in ("1", "true", "yes", "y", "on")` checks if it's a valid value (values considered true).
|
||||
4. As a result, if the environment variable is not set, or has values like "0" / "false" / "no", it becomes `False` and is skipped by default.
|
||||
|
||||
**Debug Log Display**:
|
||||
|
||||
This variable itself is not displayed in logs, but logs are output in the skip logic described later.
|
||||
|
||||
### 2. AWQW4A16Linear Module Detection
|
||||
|
||||
**File**: `nunchaku_code/lora_qwen.py`
|
||||
**Line Numbers**: 1385-1392
|
||||
|
||||
```python
|
||||
is_awq_w4a16 = (
|
||||
module.__class__.__name__ == "AWQW4A16Linear"
|
||||
and hasattr(module, "qweight")
|
||||
and hasattr(module, "wscales")
|
||||
and hasattr(module, "wzeros")
|
||||
and hasattr(module, "in_features")
|
||||
and hasattr(module, "out_features")
|
||||
)
|
||||
```
|
||||
|
||||
**Explanation**:
|
||||
|
||||
Within the `compose_loras_v2` function, this checks whether each module is `AWQW4A16Linear`.
|
||||
|
||||
1. `module.__class__.__name__ == "AWQW4A16Linear"` checks the class name. We use class name comparison instead of `isinstance()` to account for possible differences in import paths.
|
||||
2. `hasattr(module, "qweight")` checks if the quantized weight tensor exists. In AWQ, weights are quantized to INT4 and stored in `qweight`.
|
||||
3. `hasattr(module, "wscales")` and `hasattr(module, "wzeros")` check if quantization scales and zero points exist.
|
||||
4. `hasattr(module, "in_features")` and `hasattr(module, "out_features")` check if standard linear layer attributes exist.
|
||||
|
||||
If all these conditions are met, `is_awq_w4a16 = True` and it is treated as an AWQ quantized layer.
|
||||
|
||||
**Debug Log Display**:
|
||||
|
||||
This detection process itself is not displayed in logs, but logs are output in subsequent processing.
|
||||
|
||||
### 3. AWQ Modulation Layer Skip Logic
|
||||
|
||||
**File**: `nunchaku_code/lora_qwen.py`
|
||||
**Line Numbers**: 1394-1400
|
||||
|
||||
```python
|
||||
# Default-skip AWQ modulation layers to avoid catastrophic noise.
|
||||
if is_awq_w4a16 and (".img_mod.1" in resolved_name or ".txt_mod.1" in resolved_name) and not _APPLY_AWQ_MOD:
|
||||
logger.warning(
|
||||
f"[SKIP] {resolved_name}: AWQ modulation layer LoRA is disabled by default (prevents noise). "
|
||||
f"Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable."
|
||||
)
|
||||
continue
|
||||
```
|
||||
|
||||
**Explanation**:
|
||||
|
||||
This processing skips LoRA application to AWQ quantized modulation layers (`img_mod.1` or `txt_mod.1`) by default.
|
||||
|
||||
If:
|
||||
1. `is_awq_w4a16` is `True`, and
|
||||
2. `resolved_name` contains `.img_mod.1` or `.txt_mod.1`, and
|
||||
3. `_APPLY_AWQ_MOD` is `False` (environment variable is disabled)
|
||||
|
||||
then LoRA application is skipped and processing continues to the next module (`continue`).
|
||||
|
||||
**Debug Log Display**:
|
||||
|
||||
When skipped, a WARNING level log like the following is output:
|
||||
|
||||
```
|
||||
[SKIP] transformer_blocks.0.img_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
[SKIP] transformer_blocks.0.txt_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
[SKIP] transformer_blocks.1.img_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
[SKIP] transformer_blocks.1.txt_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
...
|
||||
```
|
||||
|
||||
This log is output for each `img_mod.1` and `txt_mod.1` of each transformer block (typically 0 to 59, 60 blocks total). In other words, up to 120 skip logs may be output with default settings.
|
||||
|
||||
### 4. LoRA Application Processing to AWQW4A16Linear
|
||||
|
||||
**File**: `nunchaku_code/lora_qwen.py`
|
||||
**Line Numbers**: 1018-1043
|
||||
|
||||
```python
|
||||
elif module.__class__.__name__ == "AWQW4A16Linear" and hasattr(module, "qweight") and hasattr(module, "wscales") and hasattr(module, "wzeros"):
|
||||
if not hasattr(model, "_lora_slots"):
|
||||
model._lora_slots = {}
|
||||
|
||||
# Save original forward once
|
||||
if not hasattr(module, "_lora_original_forward"):
|
||||
module._lora_original_forward = module.forward
|
||||
|
||||
# Store LoRA weights on the module (already scaled/concatenated by caller)
|
||||
module._lora_A = A
|
||||
module._lora_B = B
|
||||
|
||||
def _forward_with_lora(x: torch.Tensor) -> torch.Tensor:
|
||||
# Base quantized forward
|
||||
y = module._lora_original_forward(x)
|
||||
# LoRA branch (dtype/device should already match caller cast)
|
||||
# A: [rank, in], B: [out, rank]
|
||||
lora_out = (x @ module._lora_A.T) @ module._lora_B.T
|
||||
return y + lora_out.to(dtype=y.dtype, device=y.device)
|
||||
|
||||
module.forward = _forward_with_lora
|
||||
|
||||
# Track for reset
|
||||
model._lora_slots[module_name] = {
|
||||
"type": "awq_w4a16",
|
||||
}
|
||||
```
|
||||
|
||||
**Explanation**:
|
||||
|
||||
This processing implements a method of adding LoRA terms during the forward pass without modifying quantized weights when applying LoRA to AWQ quantized layers.
|
||||
|
||||
1. **LoRA Slot Initialization**: If `model._lora_slots` doesn't exist, it is initialized. This dictionary is needed for resetting later.
|
||||
|
||||
2. **Save Original Forward Method**: The original `forward` method is saved to `module._lora_original_forward`. This allows restoring the original behavior during reset. This processing is executed only once (via `hasattr` check).
|
||||
|
||||
3. **Store LoRA Weights**: LoRA A and B matrices are stored in `module._lora_A` and `module._lora_B`. These weights have already been scaled and concatenated by the caller (`compose_loras_v2`).
|
||||
|
||||
4. **Implement LoRA-enabled Forward**: An internal function `_forward_with_lora` is defined. This function:
|
||||
- Executes normal forward with quantized weights via `module._lora_original_forward(x)`.
|
||||
- Calculates LoRA terms via `(x @ module._lora_A.T) @ module._lora_B.T`. Here, `A` has shape `[rank, in_features]` and `B` has shape `[out_features, rank]`. `A.T` becomes `[in_features, rank]` and `B.T` becomes `[rank, out_features]`, so `x @ A.T @ B.T` becomes `[batch, in_features] @ [in_features, rank] @ [rank, out_features] = [batch, out_features]`.
|
||||
- Adds LoRA terms to the quantized forward result via `y + lora_out.to(dtype=y.dtype, device=y.device)`. `to(dtype=..., device=...)` aligns data types and devices.
|
||||
|
||||
5. **Replace Forward Method**: `module.forward = _forward_with_lora` replaces the module's forward method with the LoRA-enabled version.
|
||||
|
||||
6. **Record Slot Information**: `{"type": "awq_w4a16"}` is recorded in `model._lora_slots[module_name]`. This allows identifying that this module is an AWQ quantized layer during reset.
|
||||
|
||||
**Debug Log Display**:
|
||||
|
||||
When this processing is executed, the following log is output in subsequent processing of the `compose_loras_v2` function (lines 1460-1461):
|
||||
|
||||
```
|
||||
[APPLY] LoRA applied to: transformer_blocks.0.img_mod.1
|
||||
```
|
||||
|
||||
However, with default settings (`_APPLY_AWQ_MOD = False`), `img_mod.1` and `txt_mod.1` are skipped, so this log is not output.
|
||||
|
||||
### 5. B Matrix Reordering Processing for Modulation Layers
|
||||
|
||||
**File**: `nunchaku_code/lora_qwen.py`
|
||||
**Line Numbers**: 1417-1433
|
||||
|
||||
```python
|
||||
# QwenImage modulation layers (img_mod.1 / txt_mod.1) are AWQ-quantized in Nunchaku
|
||||
# and their output is *interleaved* as [B, dim, 6] flattened, then later reordered by:
|
||||
# view(B, dim, 6).transpose(1, 2).reshape(B, 6*dim)
|
||||
#
|
||||
# Most LoRAs are trained against the *standard* (non-interleaved) layout (6 blocks of dim).
|
||||
# Because our AWQ LoRA path adds the LoRA output BEFORE that reorder, we must convert the
|
||||
# LoRA "up" matrix B from standard->interleaved order (permute output channels).
|
||||
if (".img_mod.1" in resolved_name) or (".txt_mod.1" in resolved_name):
|
||||
if B.ndim == 2 and (B.shape[0] % 6 == 0):
|
||||
dim = B.shape[0] // 6
|
||||
# standard [6, dim, r] -> interleaved [dim, 6, r] -> [6*dim, r]
|
||||
B = B.contiguous().view(6, dim, B.shape[1]).transpose(0, 1).reshape(B.shape[0], B.shape[1])
|
||||
else:
|
||||
logger.warning(
|
||||
f"{resolved_name}: expected mod up-matrix with out_features divisible by 6, got B{tuple(B.shape)}; "
|
||||
f"skipping mod-channel reorder"
|
||||
)
|
||||
```
|
||||
|
||||
**Explanation**:
|
||||
|
||||
In Qwen Image modulation layers, the output is used as shift/scale/gate parameters (6 blocks total). In Nunchaku's implementation, this output is processed in "interleaved" format.
|
||||
|
||||
1. **Problem Background**: Most LoRAs are trained against the standard (non-interleaved) layout (6 blocks × dim). However, in the AWQ LoRA path, LoRA output is added **before** the reordering process, so the B matrix must be converted from standard format to interleaved format.
|
||||
|
||||
2. **Shape Check**: `B.ndim == 2 and (B.shape[0] % 6 == 0)` checks if B is 2-dimensional and if `out_features` is divisible by 6. This assumes a structure of 6 blocks (shift/scale/gate × 2).
|
||||
|
||||
3. **Reordering Processing**:
|
||||
- `dim = B.shape[0] // 6` calculates the dimension of each block.
|
||||
- `B.contiguous().view(6, dim, B.shape[1])` reshapes B to shape `[6, dim, rank]`. `contiguous()` is needed to ensure contiguous memory layout.
|
||||
- `.transpose(0, 1)` converts it to `[dim, 6, rank]`. This converts from standard format `[6, dim, rank]` to interleaved format `[dim, 6, rank]`.
|
||||
- `.reshape(B.shape[0], B.shape[1])` finally reshapes it back to shape `[6*dim, rank]`.
|
||||
|
||||
4. **Error Handling**: If B's shape differs from expectations (e.g., not divisible by 6), reordering is skipped and a warning log is output.
|
||||
|
||||
**Debug Log Display**:
|
||||
|
||||
When reordering is skipped (shape differs from expectations), a WARNING log like the following is output:
|
||||
|
||||
```
|
||||
transformer_blocks.0.img_mod.1: expected mod up-matrix with out_features divisible by 6, got B(1234, 16); skipping mod-channel reorder
|
||||
```
|
||||
|
||||
However, with default settings, `img_mod.1` and `txt_mod.1` are skipped, so this processing itself is not executed (only executed when enabled via environment variable).
|
||||
|
||||
### 6. LoRA Reset Processing (AWQ Quantized Layer Restoration)
|
||||
|
||||
**File**: `nunchaku_code/lora_qwen.py`
|
||||
**Line Numbers**: 1568-1581
|
||||
|
||||
```python
|
||||
elif module_type == "awq_w4a16":
|
||||
# Restore original forward and remove attached LoRA tensors
|
||||
if hasattr(module, "_lora_original_forward"):
|
||||
try:
|
||||
module.forward = module._lora_original_forward
|
||||
except Exception:
|
||||
# Safety: never fail reset
|
||||
pass
|
||||
for attr in ("_lora_A", "_lora_B", "_lora_original_forward"):
|
||||
if hasattr(module, attr):
|
||||
try:
|
||||
delattr(module, attr)
|
||||
except Exception:
|
||||
pass
|
||||
```
|
||||
|
||||
**Explanation**:
|
||||
|
||||
This is the reset (restore) processing for LoRA applied to AWQ quantized layers within the `reset_lora_v2` function.
|
||||
|
||||
1. **Restore Forward Method**: `hasattr(module, "_lora_original_forward")` checks if the original forward method is saved. If saved, `module.forward = module._lora_original_forward` restores the original forward method.
|
||||
|
||||
2. **Delete LoRA Attributes**: Each attribute `_lora_A`, `_lora_B`, and `_lora_original_forward` is deleted. This restores the module to its original state.
|
||||
|
||||
3. **Error Handling**: Each operation is wrapped in `try-except` to ensure the reset process doesn't fail even if errors occur (as the comment "Safety: never fail reset" indicates, reset must always succeed).
|
||||
|
||||
**Debug Log Display**:
|
||||
|
||||
At the end of the `reset_lora_v2` function (line 1585), the following INFO level log is output:
|
||||
|
||||
```
|
||||
All LoRA weights have been reset from the model.
|
||||
```
|
||||
|
||||
This log is output after reset of all LoRA types (Nunchaku LoRA-ready, standard Linear, AWQ quantized layers) is complete.
|
||||
|
||||
## Log Appearance (Default Settings: Skip Enabled)
|
||||
|
||||
With default settings (`QWENIMAGE_LORA_APPLY_AWQ_MOD` not set or 0), logs are output as follows:
|
||||
|
||||
```
|
||||
[SKIP] transformer_blocks.0.img_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
[SKIP] transformer_blocks.0.txt_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
[SKIP] transformer_blocks.1.img_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
[SKIP] transformer_blocks.1.txt_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
...
|
||||
[SKIP] transformer_blocks.59.img_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
[SKIP] transformer_blocks.59.txt_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
```
|
||||
|
||||
A WARNING level skip log is output for each `img_mod.1` and `txt_mod.1` of each transformer block (0 to 59, 60 blocks total). In other words, up to 120 skip logs are output.
|
||||
|
||||
After that, a log of the number of applied modules is output:
|
||||
|
||||
```
|
||||
Applied LoRA compositions to 480 modules.
|
||||
```
|
||||
|
||||
This number 480 is the total number of modules (600) minus the number of skipped modules (120). Each transformer block typically has 10 LoRA application points, but 2 of them (`img_mod.1` and `txt_mod.1`) are skipped, so 60 blocks × 8 points = 480 modules are applied.
|
||||
|
||||
## Log Appearance (Environment Variable Enabled: Skip Disabled)
|
||||
|
||||
When the environment variable `QWENIMAGE_LORA_APPLY_AWQ_MOD=1` is set, skip logs are not output, and instead application logs are output:
|
||||
|
||||
```
|
||||
[APPLY] LoRA applied to: transformer_blocks.0.img_mod.1
|
||||
[APPLY] LoRA applied to: transformer_blocks.0.txt_mod.1
|
||||
[APPLY] LoRA applied to: transformer_blocks.1.img_mod.1
|
||||
[APPLY] LoRA applied to: transformer_blocks.1.txt_mod.1
|
||||
...
|
||||
[APPLY] LoRA applied to: transformer_blocks.59.img_mod.1
|
||||
[APPLY] LoRA applied to: transformer_blocks.59.txt_mod.1
|
||||
```
|
||||
|
||||
After that, a log of the number of applied modules is output:
|
||||
|
||||
```
|
||||
Applied LoRA compositions to 600 modules.
|
||||
```
|
||||
|
||||
This number 600 is the total number of modules (60 blocks × 10 points).
|
||||
|
||||
However, **enabling the environment variable may cause image noise**, so this setting is not recommended.
|
||||
|
||||
## Why `Applied LoRA compositions to 480 modules.` Appears
|
||||
|
||||
Qwen Image (Double Block) typically has approximately 10 application points per `transformer_blocks.X` (when LoRA keys exist):
|
||||
|
||||
1. `attn.to_qkv` (fused Q/K/V attention)
|
||||
2. `attn.to_out.0`
|
||||
3. `attn.add_qkv_proj` (additional Q/K/V projection)
|
||||
4. `attn.to_add_out`
|
||||
5. `img_mlp.net.0.proj`
|
||||
6. `img_mlp.net.2`
|
||||
7. `img_mod.1` ← **Skip target**
|
||||
8. `txt_mlp.net.0.proj`
|
||||
9. `txt_mlp.net.2`
|
||||
10. `txt_mod.1` ← **Skip target**
|
||||
|
||||
With default settings, 2 points (`img_mod.1` and `txt_mod.1`) are skipped.
|
||||
|
||||
For `transformer_blocks.0` to `59` (60 blocks):
|
||||
|
||||
- Skip count: 60 × 2 = 120
|
||||
- Application count: 600 - 120 = 480
|
||||
|
||||
This matches the log.
|
||||
|
||||
## If You Still Want to Apply to Modulation Layers (Not Recommended)
|
||||
|
||||
In conclusion, **there is no guarantee that it can be applied without noise** (varies by LoRA/strength/model instance, reproducibility is low).
|
||||
If you still want to try, you can force enable it with an environment variable.
|
||||
|
||||
### PowerShell (Before Startup)
|
||||
|
||||
```powershell
|
||||
setx QWENIMAGE_LORA_APPLY_AWQ_MOD 1
|
||||
```
|
||||
|
||||
To revert (OFF):
|
||||
|
||||
```powershell
|
||||
setx QWENIMAGE_LORA_APPLY_AWQ_MOD 0
|
||||
```
|
||||
|
||||
**ComfyUI restart is required** after setting (does not apply to already running processes).
|
||||
|
||||
## Summary
|
||||
|
||||
- **Default Behavior**: LoRA application to `img_mod.1` and `txt_mod.1` is skipped (to prevent noise).
|
||||
- **Skip Logs**: With default settings, a WARNING log is output for each modulation layer (up to 120 entries).
|
||||
- **Application Logs**: When enabled via environment variable, an INFO log is output for each modulation layer (up to 120 entries).
|
||||
- **Application Count**: 480 modules with default settings, 600 modules when environment variable is enabled.
|
||||
- **Environment Variable**: `QWENIMAGE_LORA_APPLY_AWQ_MOD=1` can force enable, but not recommended due to noise risk.
|
||||
@@ -1,117 +0,0 @@
|
||||
# v1.56 — Robust LoRA re-compose after device transitions; safe retry when 0 targets detected
|
||||
|
||||
## Summary
|
||||
- Fixes a frequent edge case where, after a crash or CPU offloading transitions, LoRA composition could end up applying to 0 target modules.
|
||||
- Adds two minimal, backward-compatible safety mechanisms:
|
||||
- Device transition detection → force re-compose
|
||||
- One-time reset+re-compose retry when composition returns 0 targets
|
||||
- Normal flows remain unchanged.
|
||||
|
||||
## Problem
|
||||
- Some users see “Applied LoRA compositions to 0 module” after a crash or when toggling CPU offload (enable/disable/auto).
|
||||
- Root cause: the underlying transformer’s device/offload-manager state may change, making the LoRA target discovery return an empty set, or the previous change detection logic fails to re-apply when the device changed but the LoRA list did not.
|
||||
|
||||
## Technical Solution
|
||||
- Device transition detection:
|
||||
- Track last known device on the wrapped transformer.
|
||||
- If device changes (e.g., CPU⇄GPU), treat it as a change and force re-compose.
|
||||
- Safe retry on zero targets:
|
||||
- After compose, if the model reports no LoRA target slots and there are LoRAs to apply, perform one reset+compose retry.
|
||||
- Limited to a single retry to avoid loops.
|
||||
|
||||
## Code Changes
|
||||
- File: `wrappers/qwenimage.py`
|
||||
- Key edits:
|
||||
- Added `self._last_device` to track device signature.
|
||||
- Extended change detection with `device_changed`.
|
||||
- After compose, validate target slots; if zero (while LoRAs exist), run one forced reset→re-compose retry.
|
||||
- The rest of the pipeline (reset→compose→offload manager rebuild) remains unchanged.
|
||||
|
||||
### Code-level Explanation (with references)
|
||||
|
||||
1) Track last device to detect CPU/GPU moves
|
||||
|
||||
```60:66:D:\USERFILES\GitHub\ComfyUI-QwenImageLoraLoader\wrappers\qwenimage.py
|
||||
self._linspace_cache_w = {}
|
||||
|
||||
# Track last seen device to detect CPU/GPU moves that require re-compose
|
||||
self._last_device = None
|
||||
```
|
||||
|
||||
2) Compute current device and detect change
|
||||
|
||||
```101:110:D:\USERFILES\GitHub\ComfyUI-QwenImageLoraLoader\wrappers\qwenimage.py
|
||||
# Detect device transition (e.g., CPU offload on/off) and force re-compose
|
||||
try:
|
||||
current_device = next(self.model.parameters()).device
|
||||
except Exception:
|
||||
current_device = None
|
||||
device_changed = (self._last_device != current_device)
|
||||
```
|
||||
|
||||
3) Include `device_changed` in re-compose trigger
|
||||
|
||||
```110:116:D:\USERFILES\GitHub\ComfyUI-QwenImageLoraLoader\wrappers\qwenimage.py
|
||||
if loras_changed or model_is_dirty or device_changed:
|
||||
# The compose function handles resetting before applying the new stack
|
||||
reset_lora_v2(self.model)
|
||||
self._applied_loras = self.loras.copy()
|
||||
```
|
||||
|
||||
4) Retry once if compose produced 0 targets after crash/transition
|
||||
|
||||
```150:170:D:\USERFILES\GitHub\ComfyUI-QwenImageLoraLoader\wrappers\qwenimage.py
|
||||
# 4. Compose LoRAs. This changes internal tensor shapes.
|
||||
compose_loras_v2(self.model, self.loras)
|
||||
|
||||
# Validate composition result; if 0 targets after a crash/transition, retry once
|
||||
try:
|
||||
has_slots = hasattr(self.model, "_lora_slots") and bool(self.model._lora_slots)
|
||||
except Exception:
|
||||
has_slots = True
|
||||
if self.loras and not has_slots:
|
||||
logger.warning("LoRA composition reported 0 target modules. Forcing reset and one retry.")
|
||||
try:
|
||||
reset_lora_v2(self.model)
|
||||
compose_loras_v2(self.model, self.loras)
|
||||
except Exception as e:
|
||||
logger.error(f"LoRA re-compose retry failed: {e}")
|
||||
```
|
||||
|
||||
5) Update device signature after composition (success path)
|
||||
|
||||
```176:181:D:\USERFILES\GitHub\ComfyUI-QwenImageLoraLoader\wrappers\qwenimage.py
|
||||
# --- END MODIFIED SECTION ---
|
||||
|
||||
# Update last known device signature after (re)composition
|
||||
self._last_device = current_device
|
||||
```
|
||||
|
||||
Rationale
|
||||
- Device changes (CPU↔GPU or offload rebuild) can invalidate prior target discovery. By detecting transitions, we ensure reset→compose runs even when the LoRA list is unchanged.
|
||||
- A single retry on 0-target avoids false negatives immediately after crashes or manager rebuilds, without masking persistent problems.
|
||||
- No change to normal execution: when device is stable and target discovery succeeds, additional logic is dormant.
|
||||
|
||||
## Expected Impact
|
||||
- After crashes/offload toggles, the loader will auto-recover instead of silently applying to 0 modules.
|
||||
- No changes to UI, node I/O, or normal execution paths.
|
||||
- Backward compatible. No behavior change when things are healthy.
|
||||
|
||||
## How to Verify
|
||||
- Single LoRA:
|
||||
- Apply a LoRA, tweak strength ±0.01, re-run → still applied.
|
||||
- Stack LoRA:
|
||||
- Set `lora_count=3`, assign 2–3 LoRAs, toggle `lora_count` 1→3→1, re-run → applied consistently.
|
||||
- Offload:
|
||||
- Switch offload disable/enable/auto and re-run → still applied.
|
||||
- Edge:
|
||||
- If composition ever reports 0 targets, a single retry is attempted automatically (warning is logged once).
|
||||
|
||||
## Acknowledgments
|
||||
- This update is derived from real-world reports that became more visible post-crash scenarios and offload transitions. Thank you to everyone raising and discussing similar patterns.
|
||||
|
||||
## Links
|
||||
- Tags page: https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/tags
|
||||
- v1.56 release: https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/releases/tag/v1.56
|
||||
|
||||
|
||||
@@ -1,205 +0,0 @@
|
||||
## Summary
|
||||
- **Fixed**: Addressed [Issue #21](https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/issues/21) – CPU offload setting was hardcoded to `"auto"`, causing unnecessary slowdowns
|
||||
- **Problem**: Users could not control CPU offload behavior, and the setting was always `"auto"`, which automatically enabled CPU offload when VRAM was low, causing slowdowns even when VRAM was sufficient
|
||||
- **Solution**: Added `cpu_offload` parameter to `INPUT_TYPES` allowing users to select from `["auto", "enable", "disable"]` with default `"disable"` for performance
|
||||
|
||||
## Problem
|
||||
- CPU offload setting was hardcoded to `"auto"` in the initialization code
|
||||
- When `"auto"` mode detected low VRAM (< 4.0GB), CPU offload was automatically enabled
|
||||
- This caused slowdowns even when users had sufficient VRAM
|
||||
- Users had no way to explicitly disable CPU offload for better performance
|
||||
- Reported by: @G7b9 (GitHub Issue #21)
|
||||
|
||||
## Technical Solution
|
||||
- Added `cpu_offload` parameter to `INPUT_TYPES()` for both `NunchakuQwenImageLoraLoader` and `NunchakuQwenImageLoraStack`
|
||||
- Options: `["auto", "enable", "disable"]`
|
||||
- Default: `"disable"` (performance-first)
|
||||
- Updated method signatures to accept `cpu_offload` parameter
|
||||
- Pass user-selected value to `ComfyQwenImageWrapper` instead of hardcoded `"auto"`
|
||||
- Added `cpu_offload` to `IS_CHANGED()` method for proper cache management
|
||||
- Added comprehensive logging to track CPU offload settings
|
||||
|
||||
## Code Changes
|
||||
|
||||
### Files Modified
|
||||
1. `nodes/lora/qwenimage.py` - Main LoRA loader node implementation
|
||||
2. `wrappers/qwenimage.py` - Added initialization logging
|
||||
|
||||
### Key Changes
|
||||
|
||||
#### 1. Added `cpu_offload` Parameter to `INPUT_TYPES()`
|
||||
|
||||
**NunchakuQwenImageLoraLoader:**
|
||||
```python
|
||||
"cpu_offload": (
|
||||
["auto", "enable", "disable"],
|
||||
{
|
||||
"default": "disable",
|
||||
"tooltip": "CPU offload setting. 'auto' enables offload when VRAM is low, 'enable' forces offload, 'disable' disables offload.",
|
||||
},
|
||||
),
|
||||
```
|
||||
|
||||
**NunchakuQwenImageLoraStack:**
|
||||
- Same implementation as `NunchakuQwenImageLoraLoader`
|
||||
|
||||
#### 2. Updated Method Signatures
|
||||
|
||||
**Before:**
|
||||
```python
|
||||
def load_lora(self, model, lora_name: str, lora_strength: float):
|
||||
```
|
||||
|
||||
**After:**
|
||||
```python
|
||||
def load_lora(self, model, lora_name: str, lora_strength: float, cpu_offload: str):
|
||||
```
|
||||
|
||||
#### 3. Updated `ComfyQwenImageWrapper` Creation
|
||||
|
||||
**Before:**
|
||||
```python
|
||||
wrapped_model = ComfyQwenImageWrapper(
|
||||
model_wrapper,
|
||||
getattr(model_wrapper, 'config', {}),
|
||||
None,
|
||||
{},
|
||||
"auto", # ← Hardcoded
|
||||
4.0,
|
||||
)
|
||||
```
|
||||
|
||||
**After:**
|
||||
```python
|
||||
wrapped_model = ComfyQwenImageWrapper(
|
||||
model_wrapper,
|
||||
getattr(model_wrapper, 'config', {}),
|
||||
None,
|
||||
{},
|
||||
cpu_offload, # ← User-selected value
|
||||
4.0,
|
||||
)
|
||||
```
|
||||
|
||||
#### 4. Added Cache Management
|
||||
|
||||
**Updated `IS_CHANGED()` method:**
|
||||
```python
|
||||
@classmethod
|
||||
def IS_CHANGED(s, model, lora_name, lora_strength, cpu_offload, *args, **kwargs):
|
||||
import hashlib
|
||||
m = hashlib.sha256()
|
||||
m.update(lora_name.encode())
|
||||
m.update(str(lora_strength).encode())
|
||||
m.update(str(model).encode())
|
||||
m.update(cpu_offload.encode()) # ← Added
|
||||
return m.digest().hex()
|
||||
```
|
||||
|
||||
#### 5. Added Logging
|
||||
|
||||
**Wrapper Initialization (`wrappers/qwenimage.py`):**
|
||||
```python
|
||||
logger.info(f"🔧 CPU offload setting: '{cpu_offload_setting}' (VRAM margin: {vram_margin_gb}GB)")
|
||||
```
|
||||
|
||||
**Wrapper Creation (`nodes/lora/qwenimage.py`):**
|
||||
```python
|
||||
logger.info(f"📦 Creating ComfyQwenImageWrapper with cpu_offload='{cpu_offload}'")
|
||||
```
|
||||
|
||||
**Setting Update:**
|
||||
```python
|
||||
if model_wrapper.cpu_offload_setting != cpu_offload:
|
||||
logger.info(f"🔄 Updating CPU offload setting from '{model_wrapper.cpu_offload_setting}' to '{cpu_offload}'")
|
||||
model_wrapper.cpu_offload_setting = cpu_offload
|
||||
```
|
||||
|
||||
## Behavior Changes
|
||||
|
||||
### CPU Offload Options
|
||||
|
||||
1. **`"auto"`** - Automatically enables CPU offload when VRAM is below 4.0GB margin
|
||||
- Same behavior as before when `"auto"` was hardcoded
|
||||
- Recommended for users with limited VRAM
|
||||
|
||||
2. **`"enable"`** - Always enables CPU offload
|
||||
- Forces CPU offload regardless of VRAM availability
|
||||
- Useful for very low VRAM systems
|
||||
|
||||
3. **`"disable"`** - Always disables CPU offload (default)
|
||||
- Keeps all processing on GPU for maximum performance
|
||||
- Recommended for users with sufficient VRAM
|
||||
|
||||
### Default Behavior
|
||||
|
||||
- **Previous**: Always `"auto"` (could cause slowdowns)
|
||||
- **Current**: `"disable"` (performance-first, users can change if needed)
|
||||
|
||||
## Benefits
|
||||
|
||||
- ✅ **User Control**: Users can now explicitly control CPU offload behavior
|
||||
- ✅ **Performance**: Default `"disable"` setting provides better performance for users with sufficient VRAM
|
||||
- ✅ **Flexibility**: Users with low VRAM can still use `"auto"` or `"enable"` modes
|
||||
- ✅ **Visibility**: Comprehensive logging helps users verify CPU offload settings
|
||||
- ✅ **Cache Management**: Proper cache invalidation when CPU offload setting changes
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
- ⚠️ **Breaking Change**: Existing workflows need to be updated
|
||||
- The `cpu_offload` parameter is now required
|
||||
- Default value is `"disable"` (different from previous `"auto"` behavior)
|
||||
- Users should review their workflows and adjust the `cpu_offload` setting if needed
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### For Users with Sufficient VRAM (Recommended)
|
||||
- No action required - default `"disable"` setting provides best performance
|
||||
- If you experience VRAM issues, change to `"auto"` or `"enable"`
|
||||
|
||||
### For Users with Limited VRAM
|
||||
- Change `cpu_offload` setting to `"auto"` or `"enable"` in the node UI
|
||||
- This restores the previous automatic behavior
|
||||
|
||||
### For Existing Workflows
|
||||
- Open your workflow in ComfyUI
|
||||
- The `cpu_offload` parameter will appear in LoRA loader nodes
|
||||
- Select your preferred setting (`"disable"` for performance, `"auto"` for automatic, `"enable"` for forced offload)
|
||||
- Save your workflow
|
||||
|
||||
## Testing
|
||||
|
||||
### Verified Scenarios
|
||||
- ✅ New wrapper creation with different `cpu_offload` settings
|
||||
- ✅ Existing wrapper setting update
|
||||
- ✅ Cache invalidation when `cpu_offload` setting changes
|
||||
- ✅ Logging output for all scenarios
|
||||
- ✅ Both `NunchakuQwenImageLoraLoader` and `NunchakuQwenImageLoraStack` nodes
|
||||
|
||||
### Log Output Examples
|
||||
|
||||
**New Wrapper Creation:**
|
||||
```
|
||||
🔧 Wrapping NunchakuQwenImageTransformer2DModel with ComfyQwenImageWrapper
|
||||
📦 Creating ComfyQwenImageWrapper with cpu_offload='disable'
|
||||
🔧 CPU offload setting: 'disable' (VRAM margin: 4.0GB)
|
||||
```
|
||||
|
||||
**Existing Wrapper:**
|
||||
```
|
||||
✅ Model is already wrapped (detected via attributes)
|
||||
📦 Current CPU offload setting: 'auto'
|
||||
🔄 Updating CPU offload setting from 'auto' to 'disable'
|
||||
```
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
- **Reported by**: @G7b9 (GitHub Issue #21)
|
||||
- **Special Thanks**: This improvement was made possible by @G7b9's detailed problem report and feature request
|
||||
|
||||
## Links
|
||||
|
||||
- **Issue #21**: https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/issues/21
|
||||
- **Tags page**: https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/tags
|
||||
- **v1.63 release**: https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/releases/tag/v1.63
|
||||
|
||||
@@ -1,487 +0,0 @@
|
||||
## Summary
|
||||
|
||||
- **Added**: V2 loader with ComfyUI Nodes 2.0 (Beta) support
|
||||
- **New Node**: `NunchakuQwenImageLoraStackV2` - V2 loader node added
|
||||
- **Fixed**: Resolved [Issue #9](https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/issues/9) – The 10th LoRA control row no longer displays when `lora_count` is set to less than 10. Dynamic UI now correctly hides unused LoRA slots and adjusts node height automatically
|
||||
- **Problem**: ComfyUI Nodes 2.0 (Beta) compatibility required a new approach to widget management, as traditional widget hiding methods no longer work
|
||||
- **Solution**: Implemented V2 loader using `optional` inputs and Flux V2-style widget array reconstruction for complete ComfyUI Nodes 2.0 (Beta) compatibility
|
||||
|
||||
## Problem
|
||||
|
||||
### ComfyUI Nodes 2.0 (Beta) Compatibility Issue
|
||||
|
||||
- ComfyUI Nodes 2.0 (Beta) introduced changes to the widget system that broke traditional widget hiding methods
|
||||
- The original `NunchakuQwenImageLoraStack` (V1) node used `required` inputs for all LoRA slots, which caused display issues
|
||||
- JavaScript-based widget hiding techniques (like `widgethider.js`) no longer work reliably with ComfyUI Nodes 2.0 (Beta)
|
||||
- The 10th LoRA control row always displayed regardless of the selected `lora_count` value ([Issue #9](https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/issues/9))
|
||||
- Reported by: @xingewh (GitHub Issue #9)
|
||||
|
||||
### Root Cause
|
||||
|
||||
**V1 Implementation (`nodes/lora/qwenimage.py`):**
|
||||
- All LoRA inputs were defined as `required` in `INPUT_TYPES()`
|
||||
- JavaScript attempted to hide widgets using `widgethider.js` with negative height values
|
||||
- ComfyUI Nodes 2.0 (Beta) changed how widgets are rendered, making this approach ineffective
|
||||
- The widget array structure could not be dynamically modified after node creation
|
||||
|
||||
**V1 Code Structure:**
|
||||
```python
|
||||
inputs = {
|
||||
"required": {
|
||||
"model": ("MODEL", {...}),
|
||||
"lora_count": ("INT", {...}),
|
||||
"cpu_offload": (["auto", "enable", "disable"], {...}),
|
||||
},
|
||||
}
|
||||
# All LoRA slots added as required
|
||||
for i in range(1, 11):
|
||||
inputs["required"][f"lora_name_{i}"] = (loras, {...})
|
||||
inputs["required"][f"lora_strength_{i}"] = ("FLOAT", {...})
|
||||
```
|
||||
|
||||
## Technical Solution
|
||||
|
||||
### Architecture Overview
|
||||
|
||||
The V2 loader implements a completely new architecture based on ComfyUI Nodes 2.0 (Beta) best practices:
|
||||
|
||||
1. **Optional Inputs**: All LoRA slots are defined as `optional` inputs instead of `required`
|
||||
2. **Widget Array Reconstruction**: JavaScript physically reconstructs the `widgets` array based on `lora_count`
|
||||
3. **Hidden Widget Sync**: V1's `lora_count` widget is hidden but kept in sync with V2's control widget
|
||||
4. **Dynamic Height Calculation**: Node height automatically adjusts based on visible LoRA slots
|
||||
5. **Flux V2-Style Implementation**: Follows the same pattern used in Flux V2 LoRA stacker nodes
|
||||
|
||||
### Key Design Decisions
|
||||
|
||||
- **Backward Compatibility**: V1 node (`NunchakuQwenImageLoraStack`) remains available for users who haven't upgraded to ComfyUI Nodes 2.0 (Beta)
|
||||
- **Feature Parity**: V2 node provides complete feature parity with V1, including CPU offload support
|
||||
- **Clean UI**: Only visible LoRA slots are included in the widget array, eliminating display issues
|
||||
|
||||
## Code Changes
|
||||
|
||||
### Files Added
|
||||
|
||||
1. `nodes/lora/qwenimage_v2.py` - New V2 node implementation
|
||||
2. `js/z_qwen_lora_dynamic.js` - JavaScript widget management for V2 node
|
||||
|
||||
### Files Modified
|
||||
|
||||
1. `__init__.py` - Added V2 node registration using `GENERATED_NODES` pattern
|
||||
|
||||
### Key Changes
|
||||
|
||||
#### 1. V2 Node Implementation (`nodes/lora/qwenimage_v2.py`)
|
||||
|
||||
**New Class Structure:**
|
||||
```python
|
||||
class NunchakuQwenImageLoraStackV2:
|
||||
"""
|
||||
Node for loading and applying multiple LoRAs to a Nunchaku Qwen Image model with dynamic UI.
|
||||
"""
|
||||
```
|
||||
|
||||
**Key Difference: Optional Inputs**
|
||||
|
||||
**V1 (Required Inputs):**
|
||||
```python
|
||||
inputs = {
|
||||
"required": {
|
||||
"model": ("MODEL", {...}),
|
||||
"lora_count": ("INT", {...}),
|
||||
"cpu_offload": (["auto", "enable", "disable"], {...}),
|
||||
},
|
||||
}
|
||||
# All LoRA slots as required
|
||||
for i in range(1, 11):
|
||||
inputs["required"][f"lora_name_{i}"] = (loras, {...})
|
||||
inputs["required"][f"lora_strength_{i}"] = ("FLOAT", {...})
|
||||
```
|
||||
|
||||
**V2 (Optional Inputs):**
|
||||
```python
|
||||
inputs = {
|
||||
"required": {
|
||||
"model": ("MODEL", {...}),
|
||||
"lora_count": ("INT", {...}),
|
||||
"cpu_offload": (["auto", "enable", "disable"], {...}),
|
||||
},
|
||||
"optional": {}, # ← New optional section
|
||||
}
|
||||
|
||||
# All LoRA slots as optional
|
||||
for i in range(1, 11):
|
||||
inputs["optional"][f"lora_name_{i}"] = ( # ← Changed to optional
|
||||
loras,
|
||||
{"tooltip": f"The file name of LoRA {i}. Select 'None' to skip this slot."},
|
||||
)
|
||||
inputs["optional"][f"lora_strength_{i}"] = ( # ← Changed to optional
|
||||
"FLOAT",
|
||||
{
|
||||
"default": 1.0,
|
||||
"min": -100.0,
|
||||
"max": 100.0,
|
||||
"step": 0.01,
|
||||
"tooltip": f"Strength for LoRA {i}.",
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
**Processing Logic:**
|
||||
```python
|
||||
def load_lora_stack(self, model, lora_count, cpu_offload, **kwargs):
|
||||
loras_to_apply = []
|
||||
|
||||
# Process only the number of LoRAs specified by lora_count
|
||||
for i in range(1, lora_count + 1): # ← Only process up to lora_count
|
||||
lora_name = kwargs.get(f"lora_name_{i}")
|
||||
lora_strength = kwargs.get(f"lora_strength_{i}", 1.0)
|
||||
|
||||
# Skip if lora_name is None or strength is negligible
|
||||
if lora_name and lora_name != "None" and abs(lora_strength) > 1e-5:
|
||||
loras_to_apply.append((lora_name, lora_strength))
|
||||
|
||||
# ... rest of implementation
|
||||
```
|
||||
|
||||
#### 2. JavaScript Widget Management (`js/z_qwen_lora_dynamic.js`)
|
||||
|
||||
**Widget Caching System:**
|
||||
```javascript
|
||||
const initCache = () => {
|
||||
if (cacheReady) return;
|
||||
const all = [...node.widgets];
|
||||
|
||||
// Cache lora_count widget (required for Python backend, but hidden in UI)
|
||||
const loraCountWidget = all.find(w => w.name === "lora_count");
|
||||
if (loraCountWidget) {
|
||||
node.cachedLoraCount = loraCountWidget;
|
||||
// Hide V1's lora_count widget using HIDDEN_TAG
|
||||
loraCountWidget.type = HIDDEN_TAG;
|
||||
loraCountWidget.computeSize = () => [0, -4];
|
||||
}
|
||||
|
||||
// Cache cpu_offload widget
|
||||
const cpuOffloadWidget = all.find(w => w.name === "cpu_offload");
|
||||
if (cpuOffloadWidget) {
|
||||
node.cachedCpuOffload = cpuOffloadWidget;
|
||||
}
|
||||
|
||||
// Cache all LoRA widgets (1-10)
|
||||
for (let i = 1; i <= 10; i++) {
|
||||
const wName = all.find(w => w.name === `lora_name_${i}`);
|
||||
const wStrength = all.find(w => w.name === `lora_strength_${i}`);
|
||||
if (wName && wStrength) {
|
||||
node.cachedWidgets[i] = [wName, wStrength];
|
||||
}
|
||||
}
|
||||
cacheReady = true;
|
||||
};
|
||||
```
|
||||
|
||||
**Control Widget Creation:**
|
||||
```javascript
|
||||
const ensureControlWidget = () => {
|
||||
const name = "🔢 LoRA Count";
|
||||
|
||||
// Remove old button widgets
|
||||
for (let i = node.widgets.length - 1; i >= 0; i--) {
|
||||
const w = node.widgets[i];
|
||||
if (w.name === "🔢 Set LoRA Count" || w.type === "button") {
|
||||
node.widgets.splice(i, 1);
|
||||
}
|
||||
}
|
||||
|
||||
let w = node.widgets.find(x => x.name === name);
|
||||
if (!w) {
|
||||
const values = ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10"];
|
||||
w = node.addWidget("combo", name, "1", (v) => {
|
||||
const num = parseInt(v);
|
||||
if (!isNaN(num)) {
|
||||
node.properties["visibleLoraCount"] = num;
|
||||
// Sync with lora_count widget for Python backend
|
||||
if (node.cachedLoraCount) {
|
||||
node.cachedLoraCount.value = num;
|
||||
}
|
||||
node.updateLoraSlots();
|
||||
}
|
||||
}, { values });
|
||||
}
|
||||
w.value = node.properties["visibleLoraCount"].toString();
|
||||
// Sync lora_count widget value
|
||||
if (node.cachedLoraCount) {
|
||||
node.cachedLoraCount.value = node.properties["visibleLoraCount"];
|
||||
}
|
||||
return w;
|
||||
};
|
||||
```
|
||||
|
||||
**Widget Array Reconstruction (Flux V2-Style):**
|
||||
```javascript
|
||||
node.updateLoraSlots = function() {
|
||||
if (!cacheReady) initCache();
|
||||
|
||||
const count = parseInt(this.properties["visibleLoraCount"] || 1);
|
||||
const controlWidget = ensureControlWidget();
|
||||
|
||||
// Physical widget reconstruction for clean layout (like Flux V2)
|
||||
this.widgets = [controlWidget]; // ← Start fresh with control widget
|
||||
|
||||
// Add lora_count widget (required for Python backend, but hidden in UI)
|
||||
if (node.cachedLoraCount) {
|
||||
node.cachedLoraCount.type = HIDDEN_TAG;
|
||||
node.cachedLoraCount.computeSize = () => [0, -4];
|
||||
node.cachedLoraCount.value = count; // ← Sync value
|
||||
this.widgets.push(node.cachedLoraCount); // ← Add to array but hidden
|
||||
}
|
||||
|
||||
// Add cpu_offload widget from cache
|
||||
if (node.cachedCpuOffload) {
|
||||
this.widgets.push(node.cachedCpuOffload);
|
||||
}
|
||||
|
||||
// Add only visible LoRA slots (non-visible widgets are removed from array)
|
||||
for (let i = 1; i <= count; i++) { // ← Only add up to count
|
||||
const pair = this.cachedWidgets[i];
|
||||
if (pair) {
|
||||
this.widgets.push(pair[0]); // lora_name
|
||||
this.widgets.push(pair[1]); // lora_strength
|
||||
}
|
||||
}
|
||||
|
||||
// Height calculation
|
||||
const HEADER_H = 60;
|
||||
const SLOT_H = 54;
|
||||
const CPU_OFFLOAD_H = node.cachedCpuOffload ? 40 : 0;
|
||||
const PADDING = 20;
|
||||
const targetH = HEADER_H + CPU_OFFLOAD_H + (count * SLOT_H) + PADDING;
|
||||
|
||||
this.setSize([this.size[0], targetH]); // ← Dynamic height adjustment
|
||||
|
||||
if (app.canvas) app.canvas.setDirty(true, true);
|
||||
};
|
||||
```
|
||||
|
||||
**Key Features:**
|
||||
- **Physical Array Reconstruction**: `this.widgets = [controlWidget]` creates a fresh array
|
||||
- **Selective Widget Addition**: Only widgets for visible slots (1 to `count`) are added
|
||||
- **Hidden Widget Sync**: V1's `lora_count` widget is hidden but kept in sync for Python backend
|
||||
- **Dynamic Height**: Node height calculated based on visible slots: `HEADER_H + CPU_OFFLOAD_H + (count * SLOT_H) + PADDING`
|
||||
|
||||
#### 3. Node Registration (`__init__.py`)
|
||||
|
||||
**V2 Node Registration:**
|
||||
```python
|
||||
from nodes.lora.qwenimage_v2 import GENERATED_NODES, GENERATED_DISPLAY_NAMES
|
||||
|
||||
# Register V2 node
|
||||
NODE_CLASS_MAPPINGS.update(GENERATED_NODES)
|
||||
NODE_DISPLAY_NAME_MAPPINGS.update(GENERATED_DISPLAY_NAMES)
|
||||
```
|
||||
|
||||
**V2 Node Export (`nodes/lora/qwenimage_v2.py`):**
|
||||
```python
|
||||
GENERATED_NODES = {
|
||||
"NunchakuQwenImageLoraStackV2": NunchakuQwenImageLoraStackV2
|
||||
}
|
||||
|
||||
GENERATED_DISPLAY_NAMES = {
|
||||
"NunchakuQwenImageLoraStackV2": "Nunchaku Qwen Image LoRA Stack V2"
|
||||
}
|
||||
```
|
||||
|
||||
## Behavior Changes
|
||||
|
||||
### UI Behavior
|
||||
|
||||
**V1 (Before):**
|
||||
- All 10 LoRA slots always visible in UI
|
||||
- 10th slot always displayed regardless of `lora_count` setting
|
||||
- Node height fixed regardless of visible slots
|
||||
- JavaScript attempted to hide widgets but failed with ComfyUI Nodes 2.0 (Beta)
|
||||
|
||||
**V2 (After):**
|
||||
- Only LoRA slots up to `lora_count` are visible
|
||||
- 10th slot correctly hidden when `lora_count < 10`
|
||||
- Node height automatically adjusts based on visible slots
|
||||
- Clean widget array with only visible widgets included
|
||||
|
||||
### Widget Management
|
||||
|
||||
**V1 Approach:**
|
||||
- All widgets defined as `required`
|
||||
- JavaScript tried to hide widgets using `widgethider.js`
|
||||
- Widget array structure fixed after node creation
|
||||
- Failed with ComfyUI Nodes 2.0 (Beta)
|
||||
|
||||
**V2 Approach:**
|
||||
- All LoRA widgets defined as `optional`
|
||||
- JavaScript physically reconstructs widget array
|
||||
- Only visible widgets included in array
|
||||
- Works perfectly with ComfyUI Nodes 2.0 (Beta)
|
||||
|
||||
### Node Height Calculation
|
||||
|
||||
**V2 Dynamic Height:**
|
||||
```javascript
|
||||
const HEADER_H = 60; // Node header height
|
||||
const SLOT_H = 54; // Height per LoRA slot
|
||||
const CPU_OFFLOAD_H = 40; // CPU offload widget height (if present)
|
||||
const PADDING = 20; // Bottom padding
|
||||
const targetH = HEADER_H + CPU_OFFLOAD_H + (count * SLOT_H) + PADDING;
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
- `lora_count = 1`: Height = 60 + 40 + (1 × 54) + 20 = 174px
|
||||
- `lora_count = 3`: Height = 60 + 40 + (3 × 54) + 20 = 282px
|
||||
- `lora_count = 10`: Height = 60 + 40 + (10 × 54) + 20 = 660px
|
||||
|
||||
## Benefits
|
||||
|
||||
- ✅ **ComfyUI Nodes 2.0 (Beta) Compatibility**: Full support for ComfyUI Nodes 2.0 (Beta) widget system
|
||||
- ✅ **Issue #9 Fixed**: 10th LoRA control row no longer displays when `lora_count < 10`
|
||||
- ✅ **Clean UI**: Only visible LoRA slots are shown, eliminating clutter
|
||||
- ✅ **Dynamic Height**: Node height automatically adjusts based on `lora_count`
|
||||
- ✅ **Backward Compatibility**: V1 node remains available for non-Beta users
|
||||
- ✅ **Feature Parity**: V2 provides complete feature parity with V1, including CPU offload support
|
||||
- ✅ **Future-Proof**: Uses ComfyUI Nodes 2.0 (Beta) best practices for long-term compatibility
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
- ✅ **V1 Node Preserved**: The original `NunchakuQwenImageLoraStack` (V1) node remains available
|
||||
- ✅ **No Breaking Changes**: Existing workflows using V1 node continue to work
|
||||
- ✅ **Optional Migration**: Users can migrate to V2 node when ready
|
||||
- ⚠️ **V1 Limitation**: V1 node still exhibits Issue #9 behavior (10th row always visible)
|
||||
- ✅ **Recommendation**: Use V2 node for ComfyUI Nodes 2.0 (Beta) or to fix Issue #9
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### For New Users
|
||||
|
||||
- Use `NunchakuQwenImageLoraStackV2` node directly
|
||||
- No migration needed
|
||||
|
||||
### For Existing V1 Users
|
||||
|
||||
**Option 1: Keep Using V1 (Not Recommended)**
|
||||
- V1 node continues to work
|
||||
- Issue #9 (10th row always visible) will persist
|
||||
- May have compatibility issues with future ComfyUI Nodes 2.0 updates
|
||||
|
||||
**Option 2: Migrate to V2 (Recommended)**
|
||||
1. Open your workflow in ComfyUI
|
||||
2. Replace `NunchakuQwenImageLoraStack` nodes with `NunchakuQwenImageLoraStackV2`
|
||||
3. Configure LoRA slots using the "🔢 LoRA Count" combo box
|
||||
4. Verify that only the selected number of LoRA slots are visible
|
||||
5. Save your workflow
|
||||
|
||||
**Migration Benefits:**
|
||||
- Issue #9 fixed (10th row no longer always visible)
|
||||
- Cleaner UI with dynamic height adjustment
|
||||
- Full ComfyUI Nodes 2.0 (Beta) compatibility
|
||||
- Future-proof implementation
|
||||
|
||||
### For ComfyUI Nodes 2.0 (Beta) Users
|
||||
|
||||
- **Required**: Use `NunchakuQwenImageLoraStackV2` node
|
||||
- V1 node may not work correctly with ComfyUI Nodes 2.0 (Beta)
|
||||
- V2 node provides full compatibility
|
||||
|
||||
## Testing
|
||||
|
||||
### Verified Scenarios
|
||||
|
||||
- ✅ V2 node creation and initialization
|
||||
- ✅ Dynamic UI updates when `lora_count` changes
|
||||
- ✅ 10th LoRA slot correctly hidden when `lora_count < 10`
|
||||
- ✅ Node height adjustment based on `lora_count`
|
||||
- ✅ LoRA loading and application with V2 node
|
||||
- ✅ CPU offload functionality with V2 node
|
||||
- ✅ Widget array reconstruction on node configure
|
||||
- ✅ Hidden `lora_count` widget sync with control widget
|
||||
- ✅ ComfyUI Nodes 2.0 (Beta) compatibility
|
||||
- ✅ Backward compatibility with V1 node
|
||||
|
||||
### Test Cases
|
||||
|
||||
**Test 1: Basic Functionality**
|
||||
- Create V2 node
|
||||
- Set `lora_count = 3`
|
||||
- Verify only 3 LoRA slots are visible
|
||||
- Verify node height is correct
|
||||
- Load LoRAs and verify they apply correctly
|
||||
|
||||
**Test 2: Issue #9 Fix**
|
||||
- Create V2 node
|
||||
- Set `lora_count = 5`
|
||||
- Verify 10th LoRA slot is NOT visible
|
||||
- Change `lora_count` to 10
|
||||
- Verify 10th LoRA slot becomes visible
|
||||
- Change `lora_count` back to 5
|
||||
- Verify 10th LoRA slot is hidden again
|
||||
|
||||
**Test 3: Dynamic Height**
|
||||
- Create V2 node
|
||||
- Set `lora_count = 1`
|
||||
- Record node height
|
||||
- Change `lora_count` to 10
|
||||
- Verify node height increases correctly
|
||||
- Change `lora_count` to 3
|
||||
- Verify node height decreases correctly
|
||||
|
||||
**Test 4: Widget Sync**
|
||||
- Create V2 node
|
||||
- Change "🔢 LoRA Count" combo box
|
||||
- Verify hidden `lora_count` widget value syncs
|
||||
- Verify Python backend receives correct value
|
||||
|
||||
**Test 5: ComfyUI Nodes 2.0 (Beta) Compatibility**
|
||||
- Test V2 node with ComfyUI Nodes 2.0 (Beta)
|
||||
- Verify all widgets display correctly
|
||||
- Verify dynamic UI updates work
|
||||
- Verify no console errors
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Why Optional Inputs?
|
||||
|
||||
ComfyUI Nodes 2.0 (Beta) changed how widgets are managed. Using `optional` inputs allows JavaScript to:
|
||||
1. Control which widgets appear in the `widgets` array
|
||||
2. Physically reconstruct the array without breaking the node
|
||||
3. Hide widgets by simply not including them in the array
|
||||
|
||||
### Why Widget Array Reconstruction?
|
||||
|
||||
Traditional widget hiding (using `widgethider.js` or negative heights) no longer works reliably with ComfyUI Nodes 2.0 (Beta). The Flux V2-style approach:
|
||||
1. Creates a fresh widget array on each update
|
||||
2. Only includes widgets for visible slots
|
||||
3. Ensures clean UI without hidden widgets causing layout issues
|
||||
|
||||
### Why Keep V1's lora_count Widget?
|
||||
|
||||
The Python backend still expects a `lora_count` parameter. V2:
|
||||
1. Keeps V1's `lora_count` widget in the array (for Python backend)
|
||||
2. Hides it using `HIDDEN_TAG` and `computeSize = () => [0, -4]`
|
||||
3. Syncs its value with V2's "🔢 LoRA Count" combo box
|
||||
4. Ensures Python backend receives correct value
|
||||
|
||||
### Flux V2-Style Implementation
|
||||
|
||||
V2 follows the same pattern used in Flux V2 LoRA stacker nodes:
|
||||
- `ComfyUI-NunchakuFluxLoraStacker/js/z_flux_lora_dynamic.js`
|
||||
- `ComfyUI-NunchakuFluxLoraStacker/js/lora_stacker_v2.js`
|
||||
|
||||
This ensures consistency and compatibility with ComfyUI Nodes 2.0 (Beta) best practices.
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
- **Reported by**: @xingewh (GitHub Issue #9)
|
||||
- **Special Thanks**: This fix was made possible by @xingewh's detailed bug report about the 10th LoRA control row always displaying
|
||||
- **Inspiration**: Flux V2 LoRA stacker implementation for ComfyUI Nodes 2.0 (Beta) compatibility patterns
|
||||
|
||||
## Links
|
||||
|
||||
- **Issue #9**: https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/issues/9
|
||||
- **Tags page**: https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/tags
|
||||
- **v1.70 release**: https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/releases/tag/v1.70
|
||||
- **ComfyUI Nodes 2.0 (Beta)**: https://github.com/comfyanonymous/ComfyUI
|
||||
|
||||
@@ -1,439 +0,0 @@
|
||||
## Summary
|
||||
|
||||
* **Fixed**: Attempted fix for [Issue #25](https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/issues/25) – `AttributeError: 'NunchakuModelPatcher' object has no attribute 'pinned'` and deepcopy errors with `model_config`
|
||||
* **Reported by**: [@LacklusterOpsec](https://github.com/LacklusterOpsec) ([Issue #25](https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/issues/25))
|
||||
* **Problem**: In certain environments, `copy.deepcopy(model)` fails when `model.model.model_config` exists but cannot be deepcopied due to missing `__setstate__` method, causing `TypeError: 'NoneType' object is not callable`
|
||||
* **Solution**: Added logic to temporarily save and restore `model_config` before deepcopy operations to avoid errors
|
||||
* **Current Status**: ⚠️ **This error does not occur in our stable ComfyUI environment** - The fix was implemented based on the reported issue, but we cannot guarantee it will completely resolve the issue as we cannot reproduce it in our environment. If you encounter this error, please report with your ComfyUI version and environment details.
|
||||
|
||||
## Problem
|
||||
|
||||
### Deepcopy Error with model_config
|
||||
|
||||
* When using LoRA loader nodes, `copy.deepcopy(model)` operation fails with `TypeError: 'NoneType' object is not callable`
|
||||
* Error occurs during the Flux-style deepcopy process when copying the model object
|
||||
* The error is triggered by `model.model.model_config` object that lacks proper pickle serialization support
|
||||
* Warning messages appear: `WARNING, you accessed __deepcopy__ from the model config object which doesn't exist` and `WARNING, you accessed __setstate__ from the model config object which doesn't exist`
|
||||
* Reported by: @LacklusterOpsec (GitHub Issue #25)
|
||||
|
||||
### Error Details
|
||||
|
||||
**Error Traceback:**
|
||||
|
||||
```
|
||||
TypeError: 'NoneType' object is not callable
|
||||
|
||||
Traceback (most recent call last):
|
||||
File ".../execution.py", line 515, in execute
|
||||
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(...)
|
||||
...
|
||||
File ".../qwenimage.py", line 161, in load_lora
|
||||
ret_model = copy.deepcopy(model)
|
||||
...
|
||||
File ".../copy.py", line 273, in _reconstruct
|
||||
y.__setstate__(state)
|
||||
TypeError: 'NoneType' object is not callable
|
||||
```
|
||||
|
||||
**Warning Messages:**
|
||||
|
||||
```
|
||||
WARNING, you accessed __deepcopy__ from the model config object which doesn't exist. Please fix your code.
|
||||
WARNING, you accessed __setstate__ from the model config object which doesn't exist. Please fix your code.
|
||||
```
|
||||
|
||||
### Root Cause
|
||||
|
||||
**Model Structure:**
|
||||
|
||||
The model object has the following structure:
|
||||
|
||||
```
|
||||
NunchakuModelPatcher
|
||||
└── model (NunchakuQwenImage)
|
||||
├── diffusion_model (NunchakuQwenImageTransformer2DModel or ComfyQwenImageWrapper)
|
||||
├── model_config ← This attribute causes the problem
|
||||
└── other parameters
|
||||
```
|
||||
|
||||
**Why Deepcopy Fails:**
|
||||
|
||||
1. **Flux-style Deepcopy Process**: The LoRA loader uses a "Flux-style deepcopy" technique to create a copy of the model while temporarily setting `model_wrapper.model = None` to avoid copying the heavy transformer weights
|
||||
2. **model_config Object**: The `model.model.model_config` attribute contains configuration data that Python's `copy.deepcopy` attempts to serialize
|
||||
3. **Missing Methods**: When `copy.deepcopy` tries to serialize `model_config`, it calls `__getstate__()` to get the object's state, then `__setstate__()` to restore it. However, the `model_config` object (likely a Hugging Face `PretrainedConfig` or similar) has `__setstate__` set to `None` instead of a proper method
|
||||
4. **Error Trigger**: When deepcopy attempts to call `__setstate__` on the copied object, it tries to call `None()`, which raises `TypeError: 'NoneType' object is not callable`
|
||||
|
||||
**Code Before Fix:**
|
||||
|
||||
```python
|
||||
# Flux-style deepcopy
|
||||
model_wrapper.model = None
|
||||
ret_model = copy.deepcopy(model) # ← Fails here if model_config cannot be deepcopied
|
||||
ret_model_wrapper = ret_model.model.diffusion_model
|
||||
model_wrapper.model = transformer
|
||||
ret_model_wrapper.model = transformer
|
||||
```
|
||||
|
||||
**Potential Connection to ComfyUI Updates:**
|
||||
|
||||
A significant ComfyUI commit ([fd10932](https://github.com/comfyanonymous/ComfyUI/commit/fd109325db7126f92c2dfb7e6b25310eded8c1f8)) added Kandinsky5 model support by introducing a new `Kandinsky5` class that inherits from `BaseModel`. This commit modified `model_base.py` to add:
|
||||
|
||||
```python
|
||||
class Kandinsky5(BaseModel):
|
||||
def __init__(self, model_config, model_type=ModelType.FLOW, device=None):
|
||||
super().__init__(model_config, model_type, device=device, ...)
|
||||
```
|
||||
|
||||
**Potential Impact:**
|
||||
|
||||
1. **BaseModel Structure Changes**: The addition of new model types (Kandinsky5) to ComfyUI's `BaseModel` may have introduced changes to how `model_config` is handled across all model types
|
||||
2. **Config Object Evolution**: As ComfyUI adds support for more model architectures, the `model_config` attribute structure may have evolved, potentially affecting pickle serialization behavior
|
||||
3. **Version-Dependent Behavior**: Different ComfyUI versions may have different implementations of `BaseModel` and how it stores/uses `model_config`, which could explain why the error occurs in some environments but not others
|
||||
4. **New Model Type Influence**: The way new model types (like Kandinsky5) are integrated may have influenced how `model_config` objects are structured or initialized, potentially leading to pickle serialization issues
|
||||
|
||||
**Note**: While this connection is speculative based on the timing and nature of the ComfyUI commit, it provides a possible explanation for why this error appears in certain ComfyUI versions but not in stable builds.
|
||||
|
||||
**Why It Doesn't Happen in All Environments:**
|
||||
|
||||
* Different ComfyUI versions may have different model structures or `model_config` implementations
|
||||
* Different model types (Qwen Image standard vs Edit) may have different config structures
|
||||
* Different ComfyUI-nunchaku versions may handle `model_config` differently
|
||||
* Some environments may not have `model_config` attribute at all, avoiding the issue entirely
|
||||
|
||||
## Technical Solution
|
||||
|
||||
### Approach: Temporarily Remove model_config Before Deepcopy
|
||||
|
||||
The solution is to temporarily remove `model_config` before performing deepcopy, then restore it to both the original and copied models after deepcopy completes.
|
||||
|
||||
**Key Design Decisions:**
|
||||
|
||||
1. **Safe Attribute Check**: Only attempt to save/restore `model_config` if it exists using `hasattr()`
|
||||
2. **Exception Safety**: Use `try/finally` to ensure `model_config` is always restored, even if deepcopy fails
|
||||
3. **Bidirectional Restoration**: Restore `model_config` to both the original model and the copied model to maintain consistency
|
||||
4. **No Side Effects**: The temporary removal of `model_config` does not affect model functionality since it's immediately restored
|
||||
|
||||
### Implementation Details
|
||||
|
||||
**Step 1: Check and Save model_config**
|
||||
|
||||
```python
|
||||
saved_config = None
|
||||
if hasattr(model, 'model') and hasattr(model.model, 'model_config'):
|
||||
saved_config = model.model.model_config
|
||||
model.model.model_config = None
|
||||
```
|
||||
|
||||
**Step 2: Perform Deepcopy Safely**
|
||||
|
||||
```python
|
||||
model_wrapper.model = None
|
||||
try:
|
||||
ret_model = copy.deepcopy(model) # Now safe since model_config is None
|
||||
finally:
|
||||
# Always restore, even if deepcopy fails
|
||||
if saved_config is not None:
|
||||
model.model.model_config = saved_config
|
||||
model_wrapper.model = transformer
|
||||
```
|
||||
|
||||
**Step 3: Restore model_config to Copied Model**
|
||||
|
||||
```python
|
||||
ret_model_wrapper = ret_model.model.diffusion_model
|
||||
if saved_config is not None:
|
||||
ret_model.model.model_config = saved_config # Restore to copied model too
|
||||
ret_model_wrapper.model = transformer
|
||||
```
|
||||
|
||||
### Why This Works
|
||||
|
||||
1. **Bypasses Pickle Issue**: By temporarily setting `model_config` to `None`, we avoid the pickle serialization problem entirely
|
||||
2. **Maintains Functionality**: Since `model_config` is restored immediately after deepcopy, the model functionality is not affected
|
||||
3. **Safe for All Environments**: The `hasattr()` check ensures this code works even when `model_config` doesn't exist
|
||||
4. **Exception Safe**: The `try/finally` block guarantees restoration even if errors occur
|
||||
|
||||
## Code Changes
|
||||
|
||||
### Files Modified
|
||||
|
||||
1. `nodes/lora/qwenimage.py` - Modified `NunchakuQwenImageLoraLoader.load_lora()` and `NunchakuQwenImageLoraStack.load_lora_stack()`
|
||||
2. `nodes/lora/qwenimage_v2.py` - Modified `NunchakuQwenImageLoraStackV2.load_lora_stack()`
|
||||
|
||||
### Code Changes in Detail
|
||||
|
||||
#### 1. NunchakuQwenImageLoraLoader.load_lora() (nodes/lora/qwenimage.py)
|
||||
|
||||
**Before (Lines 159-164):**
|
||||
|
||||
```python
|
||||
# Flux-style deepcopy
|
||||
model_wrapper.model = None
|
||||
ret_model = copy.deepcopy(model)
|
||||
ret_model_wrapper = ret_model.model.diffusion_model
|
||||
model_wrapper.model = transformer
|
||||
ret_model_wrapper.model = transformer
|
||||
```
|
||||
|
||||
**After (Lines 159-179):**
|
||||
|
||||
```python
|
||||
# Flux-style deepcopy
|
||||
# Save config before deepcopy to avoid __setstate__ errors
|
||||
saved_config = None
|
||||
if hasattr(model, 'model') and hasattr(model.model, 'model_config'):
|
||||
saved_config = model.model.model_config
|
||||
model.model.model_config = None
|
||||
|
||||
model_wrapper.model = None
|
||||
try:
|
||||
ret_model = copy.deepcopy(model)
|
||||
finally:
|
||||
# Restore config and model
|
||||
if saved_config is not None:
|
||||
model.model.model_config = saved_config
|
||||
model_wrapper.model = transformer
|
||||
|
||||
ret_model_wrapper = ret_model.model.diffusion_model
|
||||
# Restore config in copied model if it was saved
|
||||
if saved_config is not None:
|
||||
ret_model.model.model_config = saved_config
|
||||
ret_model_wrapper.model = transformer
|
||||
```
|
||||
|
||||
#### 2. NunchakuQwenImageLoraStack.load_lora_stack() (nodes/lora/qwenimage.py)
|
||||
|
||||
**Before (Lines 335-340):**
|
||||
|
||||
```python
|
||||
# Flux-style deepcopy
|
||||
model_wrapper.model = None
|
||||
ret_model = copy.deepcopy(model)
|
||||
ret_model_wrapper = ret_model.model.diffusion_model
|
||||
model_wrapper.model = transformer
|
||||
ret_model_wrapper.model = transformer
|
||||
```
|
||||
|
||||
**After (Lines 350-375):**
|
||||
|
||||
```python
|
||||
# Flux-style deepcopy
|
||||
# Save config before deepcopy to avoid __setstate__ errors
|
||||
saved_config = None
|
||||
if hasattr(model, 'model') and hasattr(model.model, 'model_config'):
|
||||
saved_config = model.model.model_config
|
||||
model.model.model_config = None
|
||||
|
||||
model_wrapper.model = None
|
||||
try:
|
||||
ret_model = copy.deepcopy(model)
|
||||
finally:
|
||||
# Restore config and model
|
||||
if saved_config is not None:
|
||||
model.model.model_config = saved_config
|
||||
model_wrapper.model = transformer
|
||||
|
||||
ret_model_wrapper = ret_model.model.diffusion_model
|
||||
# Restore config in copied model if it was saved
|
||||
if saved_config is not None:
|
||||
ret_model.model.model_config = saved_config
|
||||
ret_model_wrapper.model = transformer
|
||||
```
|
||||
|
||||
#### 3. NunchakuQwenImageLoraStackV2.load_lora_stack() (nodes/lora/qwenimage_v2.py)
|
||||
|
||||
**Before (Lines 192-197):**
|
||||
|
||||
```python
|
||||
# Flux-style deepcopy
|
||||
model_wrapper.model = None
|
||||
ret_model = copy.deepcopy(model)
|
||||
ret_model_wrapper = ret_model.model.diffusion_model
|
||||
model_wrapper.model = transformer
|
||||
ret_model_wrapper.model = transformer
|
||||
```
|
||||
|
||||
**After (Lines 192-211):**
|
||||
|
||||
```python
|
||||
# Flux-style deepcopy
|
||||
# Save config before deepcopy to avoid __setstate__ errors
|
||||
saved_config = None
|
||||
if hasattr(model, 'model') and hasattr(model.model, 'model_config'):
|
||||
saved_config = model.model.model_config
|
||||
model.model.model_config = None
|
||||
|
||||
model_wrapper.model = None
|
||||
try:
|
||||
ret_model = copy.deepcopy(model)
|
||||
finally:
|
||||
# Restore config and model
|
||||
if saved_config is not None:
|
||||
model.model.model_config = saved_config
|
||||
model_wrapper.model = transformer
|
||||
|
||||
ret_model_wrapper = ret_model.model.diffusion_model
|
||||
# Restore config in copied model if it was saved
|
||||
if saved_config is not None:
|
||||
ret_model.model.model_config = saved_config
|
||||
ret_model_wrapper.model = transformer
|
||||
```
|
||||
|
||||
### Code Change Summary
|
||||
|
||||
**Lines Changed:**
|
||||
|
||||
* `nodes/lora/qwenimage.py`:
|
||||
- `NunchakuQwenImageLoraLoader.load_lora()`: Lines 159-179 (added 15 lines)
|
||||
- `NunchakuQwenImageLoraStack.load_lora_stack()`: Lines 350-375 (added 15 lines)
|
||||
* `nodes/lora/qwenimage_v2.py`:
|
||||
- `NunchakuQwenImageLoraStackV2.load_lora_stack()`: Lines 192-211 (added 15 lines)
|
||||
|
||||
**Total Changes:** 45 lines added across 3 methods in 2 files
|
||||
|
||||
## Benefits
|
||||
|
||||
* ✅ **Fixes Deepcopy Error**: Prevents `TypeError: 'NoneType' object is not callable` during model copying
|
||||
* ✅ **Safe for All Environments**: Uses `hasattr()` checks to work with or without `model_config`
|
||||
* ✅ **Exception Safe**: `try/finally` ensures `model_config` is always restored
|
||||
* ✅ **No Functional Impact**: Temporary removal of `model_config` does not affect model behavior
|
||||
* ✅ **Backward Compatible**: Works with existing workflows and model structures
|
||||
* ✅ **Universal Fix**: Applied to all three LoRA loader node classes for consistency
|
||||
|
||||
## Testing
|
||||
|
||||
### Verified Scenarios
|
||||
|
||||
* ✅ Deepcopy succeeds when `model_config` exists and cannot be deepcopied
|
||||
* ✅ Deepcopy succeeds when `model_config` does not exist (backward compatibility)
|
||||
* ✅ `model_config` is correctly restored to original model after deepcopy
|
||||
* ✅ `model_config` is correctly restored to copied model after deepcopy
|
||||
* ✅ Model functionality remains unchanged after deepcopy
|
||||
* ✅ LoRA loading works correctly after fix
|
||||
* ✅ All three node classes (Loader, Stack, StackV2) work correctly
|
||||
|
||||
### Test Cases
|
||||
|
||||
**Test 1: Environment with model_config (Issue #25 Scenario)**
|
||||
|
||||
* Use ComfyUI version that includes `model_config` attribute
|
||||
* Load Qwen Image model
|
||||
* Apply LoRA using `NunchakuQwenImageLoraLoader`
|
||||
* Verify: Deepcopy succeeds without errors
|
||||
* Verify: `model_config` is preserved in both original and copied models
|
||||
* Verify: LoRA is applied correctly
|
||||
|
||||
**Test 2: Environment without model_config (Backward Compatibility)**
|
||||
|
||||
* Use ComfyUI version without `model_config` attribute
|
||||
* Load Qwen Image model
|
||||
* Apply LoRA using any LoRA loader node
|
||||
* Verify: Deepcopy succeeds (hasattr check returns False, no modification)
|
||||
* Verify: LoRA is applied correctly
|
||||
|
||||
**Test 3: Exception Handling**
|
||||
|
||||
* Simulate deepcopy failure
|
||||
* Verify: `model_config` is still restored in `finally` block
|
||||
* Verify: Original model is not corrupted
|
||||
|
||||
**Test 4: Multiple LoRA Stacking**
|
||||
|
||||
* Use `NunchakuQwenImageLoraStack` or `NunchakuQwenImageLoraStackV2`
|
||||
* Apply multiple LoRAs
|
||||
* Verify: Each deepcopy operation succeeds
|
||||
* Verify: All LoRAs are applied correctly
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Why model_config Cannot Be Deepcopied
|
||||
|
||||
The `model_config` object (typically a Hugging Face `PretrainedConfig` or similar) is designed to be serialized using pickle, but some implementations may have `__setstate__` set to `None` for various reasons:
|
||||
|
||||
1. **Lazy Initialization**: Some config objects delay full initialization until needed
|
||||
2. **Version Compatibility**: Different versions of transformers library may handle config serialization differently
|
||||
3. **Custom Implementations**: Custom model configs may not fully implement pickle protocol
|
||||
4. **ComfyUI Integration**: ComfyUI's model loading process may modify config objects in ways that break pickle compatibility
|
||||
|
||||
### Why Temporary Removal Works
|
||||
|
||||
1. **Pickle Bypass**: Setting `model_config` to `None` means pickle doesn't attempt to serialize it
|
||||
2. **Immediate Restoration**: Since restoration happens immediately after deepcopy, the temporary removal is transparent
|
||||
3. **Reference Safety**: We save a reference to the original `model_config` object, not a copy, so restoration is safe
|
||||
4. **No Side Effects**: The `None` assignment is temporary and local to the deepcopy operation
|
||||
|
||||
### Flux-Style Deepcopy Context
|
||||
|
||||
The "Flux-style deepcopy" is a technique used to efficiently copy model objects:
|
||||
|
||||
1. **Heavy Object Removal**: `model_wrapper.model = None` removes the heavy transformer weights before deepcopy
|
||||
2. **Lightweight Copy**: Deepcopy only copies the lightweight wrapper structure
|
||||
3. **Weight Restoration**: After deepcopy, the original transformer weights are assigned to both models (shared reference)
|
||||
4. **Memory Efficiency**: Avoids copying large tensor weights, saving memory
|
||||
|
||||
Our fix integrates seamlessly with this technique by handling `model_config` before and after the deepcopy operation.
|
||||
|
||||
### Exception Safety Design
|
||||
|
||||
The use of `try/finally` ensures that:
|
||||
|
||||
1. **Always Restored**: `model_config` is always restored, even if deepcopy raises an exception
|
||||
2. **Model Integrity**: The original model is never left in a corrupted state
|
||||
3. **Resource Safety**: No memory leaks or orphaned references
|
||||
|
||||
### Compatibility Considerations
|
||||
|
||||
**Works With:**
|
||||
|
||||
* ✅ Environments with `model_config` that cannot be deepcopied (Issue #25)
|
||||
* ✅ Environments without `model_config` attribute (backward compatible)
|
||||
* ✅ Environments with `model_config` that can be deepcopied (no negative impact)
|
||||
* ✅ All ComfyUI versions (hasattr check is universal)
|
||||
* ✅ All model types (standard Qwen Image, Edit version, etc.)
|
||||
|
||||
**Potential Limitations:**
|
||||
|
||||
* ⚠️ **Cannot Verify Fix**: We cannot reproduce the error in our environment, so we cannot verify the fix works for the reported scenario
|
||||
* ⚠️ **Edge Cases**: There may be edge cases where `model_config` structure differs significantly
|
||||
* ⚠️ **Unknown Environments**: Fix is based on reported error traceback, actual environment details unknown
|
||||
|
||||
## Limitations and Future Work
|
||||
|
||||
### Current Status
|
||||
|
||||
* ⚠️ **Not Fully Tested**: The fix cannot be tested in our environment since the error does not occur there
|
||||
* ⚠️ **Based on Error Report**: Fix is implemented based on error traceback and warning messages from Issue #25
|
||||
* ⚠️ **Environment Unknown**: We do not know the exact ComfyUI version, model type, or environment configuration where the error occurs
|
||||
|
||||
### Recommendations for Users
|
||||
|
||||
If you encounter this error:
|
||||
|
||||
1. **Report Details**: Please report your ComfyUI version, ComfyUI-nunchaku version, and Python version
|
||||
2. **Model Type**: Specify whether you're using Qwen Image standard or Edit version
|
||||
3. **Error Context**: Provide full error traceback and any warning messages
|
||||
4. **Workaround**: If the fix doesn't work, you may need to use a different ComfyUI version or model configuration
|
||||
|
||||
### Future Improvements
|
||||
|
||||
* **Better Detection**: Add logging to detect when `model_config` is being temporarily removed
|
||||
* **Config Validation**: Validate `model_config` structure before attempting deepcopy
|
||||
* **Alternative Approaches**: Explore alternative deepcopy strategies if current fix doesn't work in all cases
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
* **Reported by**: [@LacklusterOpsec](https://github.com/LacklusterOpsec) ([Issue #25](https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/issues/25))
|
||||
* **Special Thanks**: This fix was made possible by @LacklusterOpsec's detailed error report and traceback
|
||||
* **Reference**: Error traceback and warning messages from Issue #25 provided the necessary information to implement the fix
|
||||
|
||||
## Links
|
||||
|
||||
* **Issue #25**: https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/issues/25
|
||||
* **Tags page**: https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/tags
|
||||
* **v1.71 release**: https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/releases/tag/v1.71
|
||||
* **ComfyUI Repository**: https://github.com/comfyanonymous/ComfyUI
|
||||
|
||||
### Contributors
|
||||
|
||||
* @LacklusterOpsec
|
||||
|
||||
@@ -1,132 +0,0 @@
|
||||
# v2.1.0 Release Notes
|
||||
|
||||
## Summary
|
||||
|
||||
- **Fixed**: Resolved [Issue #33](https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/issues/33) – Fixed `AttributeError: 'NoneType' object has no attribute 'to'` by adding None checks to `to_safely` and `forward` methods in `ComfyQwenImageWrapper`
|
||||
- **Reported by**: [@vvhitevvizard](https://github.com/vvhitevvizard) ([Issue #33](https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/issues/33))
|
||||
- **Problem**: In certain environments, `ComfyQwenImageWrapper.to_safely()` method fails when `self.model` is `None` (after garbage collection), causing `AttributeError: 'NoneType' object has no attribute 'to'`
|
||||
- **Solution**: Added `None` checks to `to_safely` and `forward` methods to safely handle cases where the model has been garbage collected or unloaded
|
||||
|
||||
## Problem
|
||||
|
||||
### AttributeError: 'NoneType' object has no attribute 'to'
|
||||
|
||||
When using LoRA loader nodes, `AttributeError: 'NoneType' object has no attribute 'to'` error occurs during model loading/unloading operations.
|
||||
|
||||
**Error Traceback:**
|
||||
|
||||
```
|
||||
AttributeError: 'NoneType' object has no attribute 'to'
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "D:_ai\ComfyUI\custom_nodes\ComfyUI-nunchaku\model_patcher.py", line 29, in load
|
||||
self.model.diffusion_model.to_safely(device_to)
|
||||
File "D:_ai\ComfyUI\custom_nodes\ComfyUI-QwenImageLoraLoader\wrappers\qwenimage.py", line 65, in to_safely
|
||||
self.model.to(device)
|
||||
```
|
||||
|
||||
**Error occurs in two scenarios:**
|
||||
1. During model loading (`model_patcher.py` line 29: `self.model.diffusion_model.to_safely(device_to)`)
|
||||
2. During model detaching (`model_patcher.py` line 41: `self.model.diffusion_model.to_safely(self.offload_device)`)
|
||||
|
||||
### Root Cause
|
||||
|
||||
**ComfyUI 0.4.0 Model Management Changes:**
|
||||
|
||||
1. **Weak Reference-Based Model Management**
|
||||
- ComfyUI 0.4.0 introduced `weakref.ref(ModelPatcher)` in `model_management.py`
|
||||
- When a model is garbage collected, weak references return `None`
|
||||
|
||||
2. **Earlier Automatic Unloading**
|
||||
- Compared to ComfyUI 0.3.x, automatic model unloading occurs earlier
|
||||
- Garbage collection timing has changed
|
||||
|
||||
3. **Missing None Checks**
|
||||
- The `to_safely` method in `ComfyQwenImageWrapper` did not check if `self.model` is `None` before calling `self.model.to(device)`
|
||||
- When `self.model` becomes `None` after GC, calling `None.to(device)` raises `AttributeError`
|
||||
|
||||
**Why `hasattr` check alone is insufficient:**
|
||||
|
||||
```python
|
||||
# Original code (problematic)
|
||||
def to_safely(self, device):
|
||||
if hasattr(self.model, "to_safely"):
|
||||
self.model.to_safely(device)
|
||||
else:
|
||||
self.model.to(device) # ← AttributeError if self.model is None
|
||||
```
|
||||
|
||||
When `self.model` is `None`, `hasattr(None, "to_safely")` returns `False`, so the code enters the `else` block and tries to call `None.to(device)`, causing the error.
|
||||
|
||||
## Solution
|
||||
|
||||
### Fix: Add None Checks
|
||||
|
||||
**1. Added None check to `to_safely` method:**
|
||||
|
||||
```python
|
||||
def to_safely(self, device):
|
||||
"""Safely move the model to the specified device."""
|
||||
if self.model is None:
|
||||
return self
|
||||
if hasattr(self.model, "to_safely"):
|
||||
self.model.to_safely(device)
|
||||
else:
|
||||
self.model.to(device)
|
||||
return self
|
||||
```
|
||||
|
||||
**2. Added None check to `forward` method (defensive programming):**
|
||||
|
||||
```python
|
||||
def forward(self, x, timestep, context=None, y=None, guidance=None, control=None, transformer_options={}, **kwargs):
|
||||
"""Forward pass for the wrapped model."""
|
||||
# ... (preprocessing code) ...
|
||||
|
||||
# Guard against None model (can happen during GC/unload)
|
||||
if self.model is None:
|
||||
raise RuntimeError("Model has been unloaded or garbage collected. Cannot perform forward pass.")
|
||||
|
||||
# ... (remaining processing) ...
|
||||
```
|
||||
|
||||
### Why This Fix Works
|
||||
|
||||
1. **Safety Improvement**
|
||||
- `to_safely` now safely handles `None` model without crashing
|
||||
- Early return when model is `None` is appropriate (no object to move)
|
||||
|
||||
2. **Defensive Programming**
|
||||
- `forward` method explicitly detects and reports invalid state
|
||||
- Clear error messages aid debugging
|
||||
|
||||
3. **Compatibility with ComfyUI 0.4.0**
|
||||
- Works with weak reference-based model management
|
||||
- Handles GC timing differences across environments
|
||||
|
||||
## Relationship with Issue #25
|
||||
|
||||
Issue #33 shares the same root cause as [Issue #25](https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/issues/25): ComfyUI 0.4.0's weak reference-based model management.
|
||||
|
||||
**Differences:**
|
||||
- **Issue #25**: Primarily `copy.deepcopy` and `pinned` attribute errors (fixed by temporarily setting `model_config` to `None`)
|
||||
- **Issue #33**: Primarily `to_safely` method `AttributeError` (fixed by adding `None` checks)
|
||||
|
||||
Both fixes complement each other to provide robust handling of ComfyUI 0.4.0's model management changes.
|
||||
|
||||
## Testing
|
||||
|
||||
### Verified Scenarios
|
||||
|
||||
- ✅ `to_safely` succeeds when `self.model` is `None` (early return)
|
||||
- ✅ `to_safely` succeeds when `self.model` exists normally
|
||||
- ✅ `forward` raises clear error when `self.model` is `None`
|
||||
- ✅ Model loading/unloading operations complete without errors
|
||||
- ✅ Workflow execution is stable
|
||||
|
||||
## Related Issues
|
||||
|
||||
- [Issue #33](https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/issues/33) - AttributeError: 'NoneType' object has no attribute 'to'
|
||||
- [Issue #25](https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/issues/25) - AttributeError: 'NunchakuModelPatcher' object has no attribute 'pinned'
|
||||
- [COMFYUI_0.4.0_UPDATE_ERROR_FIXES.md](https://github.com/ussoewwin/ComfyUI-QwenImageLoraLoader/blob/main/COMFYUI_0.4.0_UPDATE_ERROR_FIXES.md) - ComfyUI core fixes
|
||||
|
||||
@@ -1,208 +0,0 @@
|
||||
# ComfyUI-QwenImageLoraLoader側のZ-Image-Turbo ControlNet対応修正の詳細解説
|
||||
|
||||
## 追加修正が必要になった原因
|
||||
|
||||
非公式Z-Image-Turboローダー側（`ComfyUI-nunchaku-unofficial-z-image-turbo-loader`）で`NunchakuZImageTransformer2DModel.forward`をモンキーパッチし、`transformer_options`と`control`パラメータを受け付けるようにした。
|
||||
|
||||
しかし、ComfyUI-QwenImageLoraLoader側の`ComfyZImageTurboWrapper`クラスが、これらのパラメータを下位の`NunchakuZImageTransformer2DModel.forward`に正しく伝達していなかった。
|
||||
|
||||
`ComfyZImageTurboWrapper`はLoRA合成とフォワードパスを分離するラッパーで、`forward`メソッドで受け取った引数を`_execute_model`メソッド経由で`NunchakuZImageTransformer2DModel.forward`に渡す。この伝達経路に`transformer_options`と`control`を追加する必要があった。
|
||||
|
||||
## 修正したファイル
|
||||
|
||||
`ComfyUI-QwenImageLoraLoader/wrappers/zimageturbo.py`
|
||||
|
||||
## 修正内容
|
||||
|
||||
### 修正1: forwardメソッドのシグネチャ（76行目から85行目）
|
||||
|
||||
#### 修正前
|
||||
|
||||
```python
|
||||
def forward(
|
||||
self,
|
||||
x,
|
||||
timestep,
|
||||
context=None,
|
||||
y=None,
|
||||
guidance=None,
|
||||
**kwargs,
|
||||
):
|
||||
```
|
||||
|
||||
#### 修正後
|
||||
|
||||
```python
|
||||
def forward(
|
||||
self,
|
||||
x,
|
||||
timestep,
|
||||
context=None,
|
||||
y=None,
|
||||
guidance=None,
|
||||
control=None,
|
||||
transformer_options={},
|
||||
**kwargs,
|
||||
):
|
||||
```
|
||||
|
||||
#### 意味の説明
|
||||
|
||||
`forward`メソッドのシグネチャに`control`と`transformer_options`を追加。ComfyUIのModelPatcherからこれらを受け取れるようにした。
|
||||
|
||||
### 修正2: forwardメソッド内のtransformer_optionsマージ処理（93行目から100行目）
|
||||
|
||||
#### 修正内容
|
||||
|
||||
```python
|
||||
# Remove guidance, transformer_options, and attention_mask from kwargs
|
||||
if "guidance" in kwargs:
|
||||
kwargs.pop("guidance")
|
||||
if "transformer_options" in kwargs:
|
||||
if isinstance(transformer_options, dict) and isinstance(kwargs["transformer_options"], dict):
|
||||
transformer_options = {**transformer_options, **kwargs.pop("transformer_options")}
|
||||
else:
|
||||
kwargs.pop("transformer_options")
|
||||
if "attention_mask" in kwargs:
|
||||
kwargs.pop("attention_mask")
|
||||
```
|
||||
|
||||
#### 意味の説明
|
||||
|
||||
`kwargs`に`transformer_options`が含まれている場合、明示的なパラメータとマージ。明示的なパラメータが優先。
|
||||
|
||||
### 修正3: _execute_modelメソッドへの引数伝達（219行目、221行目）
|
||||
|
||||
#### 修正内容
|
||||
|
||||
`forward`メソッド内で`_execute_model`を呼び出す際に、`control`と`transformer_options`を渡すように変更：
|
||||
|
||||
```python
|
||||
with cache_context(self._cache_context):
|
||||
out = self._execute_model(x, timestep, context, y, guidance, control, transformer_options, **kwargs)
|
||||
else:
|
||||
out = self._execute_model(x, timestep, context, y, guidance, control, transformer_options, **kwargs)
|
||||
```
|
||||
|
||||
#### 意味の説明
|
||||
|
||||
`_execute_model`に`control`と`transformer_options`を渡すことで、下位の`NunchakuZImageTransformer2DModel.forward`まで伝達できるようにした。
|
||||
|
||||
### 修正4: _execute_modelメソッドのシグネチャ（239行目）
|
||||
|
||||
#### 修正前
|
||||
|
||||
```python
|
||||
def _execute_model(self, x, timestep, context, y, guidance, **kwargs):
|
||||
```
|
||||
|
||||
#### 修正後
|
||||
|
||||
```python
|
||||
def _execute_model(self, x, timestep, context, y, guidance, control, transformer_options, **kwargs):
|
||||
```
|
||||
|
||||
#### 意味の説明
|
||||
|
||||
`_execute_model`のシグネチャに`control`と`transformer_options`を追加。
|
||||
|
||||
### 修正5: customized_forward使用時の伝達（282行目から300行目）
|
||||
|
||||
#### 修正内容
|
||||
|
||||
`customized_forward`が設定されている場合（非公式ZITローダー側のカスタムフォワードを使用する場合）の処理：
|
||||
|
||||
```python
|
||||
if self.customized_forward:
|
||||
with torch.inference_mode():
|
||||
forward_kwargs_cleaned = {k: v for k, v in self.forward_kwargs.items() if k not in ("guidance", "ref_latents", "transformer_options", "attention_mask")}
|
||||
transformer_options_cleaned = dict(transformer_options) if transformer_options else {}
|
||||
transformer_options_cleaned.pop("guidance", None)
|
||||
transformer_options_cleaned.pop("ref_latents", None)
|
||||
|
||||
return self.customized_forward(
|
||||
self.model,
|
||||
hidden_states=x,
|
||||
encoder_hidden_states=context,
|
||||
timestep=timestep,
|
||||
guidance=guidance if self.config.get("guidance_embed", False) else None,
|
||||
ref_latents=ref_latents_value,
|
||||
control=control,
|
||||
transformer_options=transformer_options_cleaned,
|
||||
**forward_kwargs_cleaned,
|
||||
**kwargs,
|
||||
)
|
||||
```
|
||||
|
||||
#### 意味の説明
|
||||
|
||||
`customized_forward`呼び出し時に`control`と`transformer_options`を渡す。`transformer_options`から`guidance`と`ref_latents`を除去（重複を避けるため）。
|
||||
|
||||
### 修正6: 直接forward呼び出し時の伝達（389行目から409行目）
|
||||
|
||||
#### 修正内容
|
||||
|
||||
`customized_forward`が設定されていない場合（直接`NunchakuZImageTransformer2DModel.forward`を呼び出す場合）の処理：
|
||||
|
||||
```python
|
||||
# Call Z-Image-Turbo forward with correct signature
|
||||
# Pass control and transformer_options to allow Model Patcher (double_block patches) to work
|
||||
# Check if the model's forward method accepts control parameter
|
||||
import inspect
|
||||
forward_sig = inspect.signature(self.model.forward)
|
||||
forward_params = set(forward_sig.parameters.keys())
|
||||
|
||||
zimage_kwargs_clean = {k: v for k, v in zimage_kwargs.items() if k not in ('control', 'transformer_options')}
|
||||
|
||||
# Build kwargs based on what the forward method accepts
|
||||
forward_kwargs = zimage_kwargs_clean.copy()
|
||||
if 'control' in forward_params:
|
||||
forward_kwargs['control'] = control
|
||||
if 'transformer_options' in forward_params:
|
||||
forward_kwargs['transformer_options'] = transformer_options
|
||||
|
||||
model_output = self.model(
|
||||
x_list,
|
||||
t_zimage,
|
||||
cap_feats=cap_feats,
|
||||
**forward_kwargs
|
||||
)
|
||||
```
|
||||
|
||||
#### 意味の説明
|
||||
|
||||
- `inspect.signature`で`self.model.forward`のシグネチャを確認
|
||||
- `control`と`transformer_options`が受け付けられる場合のみ、それらを`forward_kwargs`に追加
|
||||
- これにより、非公式ZITローダー側の`_patched_forward`（モンキーパッチされたforwardメソッド）が正しく`transformer_options`と`control`を受け取れる
|
||||
|
||||
## 重要な設計判断
|
||||
|
||||
### inspect.signatureによる動的パラメータチェック
|
||||
|
||||
`inspect.signature`を使って、`forward`メソッドが`control`と`transformer_options`を受け付けるかどうかを動的に確認する。これにより：
|
||||
|
||||
1. 後方互換性が保たれる（これらのパラメータを受け付けない旧バージョンでもエラーにならない）
|
||||
2. 非公式ZITローダー側のモンキーパッチが適用されているかどうかに関係なく、適切に動作する
|
||||
3. 将来的な拡張に対応しやすい
|
||||
|
||||
### 非公式ZITローダー側との連携
|
||||
|
||||
この修正により、以下の連携フローが実現される：
|
||||
|
||||
1. ComfyUIのModelPatcherが`transformer_options`と`control`を`ComfyZImageTurboWrapper.forward`に渡す
|
||||
2. `ComfyZImageTurboWrapper.forward`がこれらを`_execute_model`に渡す
|
||||
3. `_execute_model`が`inspect.signature`で確認してから`NunchakuZImageTransformer2DModel.forward`に渡す
|
||||
4. 非公式ZITローダー側の`_patched_forward`（モンキーパッチされたforward）が`transformer_options`と`control`を受け取り、`double_block`パッチを適用する
|
||||
|
||||
## まとめ
|
||||
|
||||
ComfyUI-QwenImageLoraLoader側の`ComfyZImageTurboWrapper`クラスを修正し、`control`と`transformer_options`パラメータを正しく伝達するようにした。
|
||||
|
||||
主な変更点は：
|
||||
|
||||
1. `forward`メソッドのシグネチャに`control`と`transformer_options`を追加
|
||||
2. `_execute_model`メソッドのシグネチャに`control`と`transformer_options`を追加
|
||||
3. `inspect.signature`を使って、下位の`forward`メソッドがこれらのパラメータを受け付ける場合のみ渡すように実装
|
||||
|
||||
これにより、非公式ZITローダー側で追加したControlNet対応のモンキーパッチが正しく動作し、ControlNetとLoRAの両立が実現される。
|
||||
|
||||
@@ -1,260 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Analyze SNOFs LoRA file structure"""
|
||||
import sys
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Optional, Tuple
|
||||
from collections import defaultdict
|
||||
|
||||
# Key mapping from nunchaku_code/lora_qwen.py
|
||||
KEY_MAPPING = [
|
||||
# Fused QKV (Double Block)
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._]attn[._]to[._]qkv$"), r"\1.\2.attn.to_qkv", "qkv", None),
|
||||
# Decomposed QKV (Double Block)
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._]attn[._]to[._](q|k|v)$"), r"\1.\2.attn.to_qkv", "qkv",
|
||||
lambda m: m.group(3).upper()),
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._]attn[._](q|k|v)[._]proj$"), r"\1.\2.attn.to_qkv", "qkv",
|
||||
lambda m: m.group(3).upper()),
|
||||
# Fused Add_QKV (Double Block)
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._]attn[._]add[._]qkv[._]proj$"), r"\1.\2.attn.add_qkv_proj",
|
||||
"add_qkv", None),
|
||||
# Decomposed Add_QKV (Double Block)
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._]attn[._]add[._](q|k|v)[._]proj$"), r"\1.\2.attn.add_qkv_proj",
|
||||
"add_qkv",
|
||||
lambda m: m.group(3).upper()),
|
||||
# Fused QKV (Single Block)
|
||||
(re.compile(r"^(single_transformer_blocks)[._](\d+)[._]attn[._]to[._]qkv$"), r"\1.\2.attn.to_qkv", "qkv", None),
|
||||
# Decomposed QKV (Single Block)
|
||||
(re.compile(r"^(single_transformer_blocks)[._](\d+)[._]attn[._]to[._](q|k|v)$"), r"\1.\2.attn.to_qkv", "qkv",
|
||||
lambda m: m.group(3).upper()),
|
||||
# Output Projections
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._]out[._]proj[._]context$"), r"\1.\2.attn.to_add_out", "regular",
|
||||
None),
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._]attn[._]to[._]add[._]out$"), r"\1.\2.attn.to_add_out",
|
||||
"regular", None),
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._]out[._]proj$"), r"\1.\2.attn.to_out.0", "regular", None),
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._]attn[._]to[._]out$"), r"\1.\2.attn.to_out.0", "regular", None),
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._]attn[._]to[._]out[._]0$"), r"\1.\2.attn.to_out.0", "regular",
|
||||
None),
|
||||
(re.compile(r"^(single_transformer_blocks)[._](\d+)[._]attn[._]to[._]out$"), r"\1.\2.attn.to_out", "regular",
|
||||
None),
|
||||
# Feed-Forward / MLP Layers (Standard)
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._]ff[._]net[._]0(?:[._]proj)?$"), r"\1.\2.mlp_fc1", "regular",
|
||||
None),
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._]ff[._]net[._]2$"), r"\1.\2.mlp_fc2", "regular", None),
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._]ff_context[._]net[._]0(?:[._]proj)?$"),
|
||||
r"\1.\2.mlp_context_fc1", "regular", None),
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._]ff_context[._]net[._]2$"), r"\1.\2.mlp_context_fc2", "regular",
|
||||
None),
|
||||
# Feed-Forward / MLP Layers (img/txt)
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._](img_mlp)[._](net)[._](0)[._](proj)$"), r"\1.\2.\3.\4.\5.\6",
|
||||
"regular", None),
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._](img_mlp)[._](net)[._](0)$"), r"\1.\2.\3.\4.\5", "regular",
|
||||
None),
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._](img_mlp)[._](net)[._](2)$"), r"\1.\2.\3.\4.\5", "regular",
|
||||
None),
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._](txt_mlp)[._](net)[._](0)[._](proj)$"), r"\1.\2.\3.\4.\5.\6",
|
||||
"regular", None),
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._](txt_mlp)[._](net)[._](0)$"), r"\1.\2.\3.\4.\5", "regular",
|
||||
None),
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._](txt_mlp)[._](net)[._](2)$"), r"\1.\2.\3.\4.\5", "regular",
|
||||
None),
|
||||
# Mod Layers (img/txt)
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._](img_mod)[._](1)$"), r"\1.\2.\3.\4", "regular", None),
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._](txt_mod)[._](1)$"), r"\1.\2.\3.\4", "regular", None),
|
||||
# Single Block Projections
|
||||
(re.compile(r"^(single_transformer_blocks)[._](\d+)[._]proj[._]out$"), r"\1.\2.proj_out", "single_proj_out",
|
||||
None),
|
||||
(re.compile(r"^(single_transformer_blocks)[._](\d+)[._]proj[._]mlp$"), r"\1.\2.mlp_fc1", "regular", None),
|
||||
# Normalization Layers
|
||||
(re.compile(r"^(single_transformer_blocks)[._](\d+)[._]norm[._]linear$"), r"\1.\2.norm.linear", "regular", None),
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._]norm1[._]linear$"), r"\1.\2.norm1.linear", "regular", None),
|
||||
(re.compile(r"^(transformer_blocks)[._](\d+)[._]norm1_context[._]linear$"), r"\1.\2.norm1_context.linear",
|
||||
"regular", None),
|
||||
# Mappings for top-level diffusion_model modules
|
||||
(re.compile(r"^(img_in)$"), r"\1", "regular", None),
|
||||
(re.compile(r"^(txt_in)$"), r"\1", "regular", None),
|
||||
(re.compile(r"^(proj_out)$"), r"\1", "regular", None),
|
||||
(re.compile(r"^(norm_out)[._](linear)$"), r"\1.\2", "regular", None),
|
||||
(re.compile(r"^(time_text_embed)[._](timestep_embedder)[._](linear_1)$"), r"\1.\2.\3", "regular", None),
|
||||
(re.compile(r"^(time_text_embed)[._](timestep_embedder)[._](linear_2)$"), r"\1.\2.\3", "regular", None),
|
||||
]
|
||||
|
||||
_RE_LORA_SUFFIX = re.compile(r"\.(?P<tag>lora(?:[._](?:A|B|down|up)))(?:\.[^.]+)*\.weight$")
|
||||
_RE_ALPHA_SUFFIX = re.compile(r"\.(?:alpha|lora_alpha)(?:\.[^.]+)*$")
|
||||
|
||||
def _rename_layer_underscore_layer_name(old_name: str) -> str:
|
||||
rules = [
|
||||
(r'_(\d+)_attn_to_out_(\d+)', r'.\1.attn.to_out.\2'),
|
||||
(r'_(\d+)_img_mlp_net_(\d+)_proj', r'.\1.img_mlp.net.\2.proj'),
|
||||
(r'_(\d+)_txt_mlp_net_(\d+)_proj', r'.\1.txt_mlp.net.\2.proj'),
|
||||
(r'_(\d+)_img_mlp_net_(\d+)', r'.\1.img_mlp.net.\2'),
|
||||
(r'_(\d+)_txt_mlp_net_(\d+)', r'.\1.txt_mlp.net.\2'),
|
||||
(r'_(\d+)_img_mod_(\d+)', r'.\1.img_mod.\2'),
|
||||
(r'_(\d+)_txt_mod_(\d+)', r'.\1.txt_mod.\2'),
|
||||
(r'_(\d+)_attn_', r'.\1.attn.'),
|
||||
]
|
||||
new_name = old_name
|
||||
for pattern, replacement in rules:
|
||||
new_name = re.sub(pattern, replacement, new_name)
|
||||
return new_name
|
||||
|
||||
def _classify_and_map_key(key: str) -> Optional[Tuple[str, str, Optional[str], str]]:
|
||||
k = key
|
||||
if k.startswith("transformer."):
|
||||
k = k[len("transformer."):]
|
||||
if k.startswith("diffusion_model."):
|
||||
k = k[len("diffusion_model."):]
|
||||
if k.startswith("lora_unet_"):
|
||||
k = k[len("lora_unet_"):]
|
||||
k = _rename_layer_underscore_layer_name(k)
|
||||
|
||||
base = None
|
||||
ab = None
|
||||
|
||||
m = _RE_LORA_SUFFIX.search(k)
|
||||
if m:
|
||||
tag = m.group("tag")
|
||||
base = k[: m.start()]
|
||||
if "lora_A" in tag or tag.endswith(".A") or "down" in tag:
|
||||
ab = "A"
|
||||
elif "lora_B" in tag or tag.endswith(".B") or "up" in tag:
|
||||
ab = "B"
|
||||
else:
|
||||
m = _RE_ALPHA_SUFFIX.search(k)
|
||||
if m:
|
||||
ab = "alpha"
|
||||
base = k[: m.start()]
|
||||
|
||||
if base is None or ab is None:
|
||||
return None
|
||||
|
||||
for pattern, template, group, comp_fn in KEY_MAPPING:
|
||||
match = pattern.match(base)
|
||||
if match:
|
||||
final_key = match.expand(template)
|
||||
component = comp_fn(match) if comp_fn else None
|
||||
return group, final_key, component, ab
|
||||
|
||||
return None
|
||||
|
||||
lora_path = r"D:\USERFILES\GitHub\Qwen_Snofs_1_3.safetensors"
|
||||
|
||||
print(f"Analyzing LoRA file: {lora_path}")
|
||||
print("=" * 100)
|
||||
|
||||
try:
|
||||
from safetensors import safe_open
|
||||
|
||||
with safe_open(lora_path, framework="pt") as f:
|
||||
keys = list(f.keys())
|
||||
print(f"\nTotal keys: {len(keys)}")
|
||||
|
||||
# Test key mapping
|
||||
print("\n" + "=" * 100)
|
||||
print("Key Mapping Analysis:")
|
||||
print("=" * 100)
|
||||
|
||||
mapped_keys = []
|
||||
unmapped_keys = []
|
||||
mapped_details = defaultdict(list)
|
||||
|
||||
for key in keys:
|
||||
parsed = _classify_and_map_key(key)
|
||||
if parsed is None:
|
||||
unmapped_keys.append(key)
|
||||
else:
|
||||
group, base_key, comp, ab = parsed
|
||||
mapped_keys.append(key)
|
||||
mapped_details[base_key].append((key, group, comp, ab))
|
||||
|
||||
print(f"\n✓ Mapped keys: {len(mapped_keys)}")
|
||||
print(f"✗ Unmapped keys: {len(unmapped_keys)}")
|
||||
print(f" Mapping rate: {len(mapped_keys)/len(keys)*100:.1f}%")
|
||||
|
||||
if unmapped_keys:
|
||||
print("\n" + "=" * 100)
|
||||
print("UNMAPPED KEYS (First 50):")
|
||||
print("=" * 100)
|
||||
for i, key in enumerate(unmapped_keys[:50], 1):
|
||||
print(f"{i:3d}. {key}")
|
||||
|
||||
# Analyze unmapped key patterns
|
||||
print("\n" + "=" * 100)
|
||||
print("Unmapped Key Pattern Analysis:")
|
||||
print("=" * 100)
|
||||
|
||||
unmapped_prefixes = defaultdict(int)
|
||||
|
||||
for key in unmapped_keys:
|
||||
if key.startswith("transformer."):
|
||||
p = "transformer.*"
|
||||
elif key.startswith("diffusion_model."):
|
||||
p = "diffusion_model.*"
|
||||
elif key.startswith("model."):
|
||||
p = "model.*"
|
||||
elif key.startswith("unet."):
|
||||
p = "unet.*"
|
||||
elif "." in key:
|
||||
p = key.split(".")[0] + ".*"
|
||||
else:
|
||||
p = key[:20] + "*"
|
||||
unmapped_prefixes[p] += 1
|
||||
|
||||
print("\nUnmapped key prefixes:")
|
||||
for prefix, count in sorted(unmapped_prefixes.items(), key=lambda x: x[1], reverse=True)[:10]:
|
||||
print(f" {prefix:40s}: {count:4d}")
|
||||
|
||||
# Check for specific patterns in unmapped keys
|
||||
print("\n" + "=" * 100)
|
||||
print("Unmapped Key Details (Sample):")
|
||||
print("=" * 100)
|
||||
for key in unmapped_keys[:20]:
|
||||
# Try to understand why it's unmapped
|
||||
k = key
|
||||
prefix_info = ""
|
||||
if k.startswith("transformer."):
|
||||
k = k[len("transformer."):]
|
||||
prefix_info = "[transformer.]"
|
||||
elif k.startswith("diffusion_model."):
|
||||
k = k[len("diffusion_model."):]
|
||||
prefix_info = "[diffusion_model.]"
|
||||
elif k.startswith("model."):
|
||||
k = k[len("model."):]
|
||||
prefix_info = "[model.]"
|
||||
|
||||
# Check suffix
|
||||
has_lora_suffix = _RE_LORA_SUFFIX.search(k) is not None
|
||||
has_alpha_suffix = _RE_ALPHA_SUFFIX.search(k) is not None
|
||||
|
||||
print(f" {key:70s}")
|
||||
print(f" {prefix_info} processed: '{k}' | lora_suffix: {has_lora_suffix} | alpha_suffix: {has_alpha_suffix}")
|
||||
|
||||
if mapped_keys:
|
||||
print("\n" + "=" * 100)
|
||||
print("MAPPED KEYS (Sample by base_key):")
|
||||
print("=" * 100)
|
||||
for base_key in sorted(mapped_details.keys())[:10]:
|
||||
details = mapped_details[base_key]
|
||||
print(f"\n{base_key}:")
|
||||
for orig_key, group, comp, ab in details[:3]:
|
||||
print(f" {orig_key} -> group={group}, comp={comp}, ab={ab}")
|
||||
|
||||
# Show first 30 keys with mapping status
|
||||
print("\n" + "=" * 100)
|
||||
print("First 30 Keys with Mapping Status:")
|
||||
print("=" * 100)
|
||||
for i, key in enumerate(keys[:30], 1):
|
||||
parsed = _classify_and_map_key(key)
|
||||
status = "✓ MAPPED" if parsed else "✗ UNMAPPED"
|
||||
group_info = f" -> {parsed[1]}" if parsed else ""
|
||||
print(f"{i:3d}. [{status:12s}] {key:60s}{group_info}")
|
||||
|
||||
except ImportError:
|
||||
print("Error: safetensors library not available")
|
||||
sys.exit(1)
|
||||
except Exception as e:
|
||||
print(f"Error: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
@@ -1,98 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Rewrite git history to remove Japanese characters from README.md in all commits.
|
||||
"""
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
def remove_japanese_from_content(content):
|
||||
"""Remove lines containing Japanese characters from content."""
|
||||
japanese_pattern = re.compile(r'[あ-ん]|[ア-ン]|[一-龯]')
|
||||
lines = content.split('\n')
|
||||
filtered_lines = [line for line in lines if not japanese_pattern.search(line)]
|
||||
return '\n'.join(filtered_lines)
|
||||
|
||||
def get_all_commits():
|
||||
"""Get all commit hashes."""
|
||||
result = subprocess.run(
|
||||
['git', 'log', '--all', '--format=%H'],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=os.getcwd()
|
||||
)
|
||||
return result.stdout.strip().split('\n') if result.stdout.strip() else []
|
||||
|
||||
def process_commit(commit_hash):
|
||||
"""Process a single commit to remove Japanese from README.md."""
|
||||
# Check if README.md exists in this commit
|
||||
result = subprocess.run(
|
||||
['git', 'cat-file', '-e', f'{commit_hash}:README.md'],
|
||||
capture_output=True,
|
||||
cwd=os.getcwd()
|
||||
)
|
||||
if result.returncode != 0:
|
||||
return False # README.md doesn't exist in this commit
|
||||
|
||||
# Get README.md content
|
||||
result = subprocess.run(
|
||||
['git', 'show', f'{commit_hash}:README.md'],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=os.getcwd()
|
||||
)
|
||||
if result.returncode != 0:
|
||||
return False
|
||||
|
||||
original_content = result.stdout
|
||||
filtered_content = remove_japanese_from_content(original_content)
|
||||
|
||||
# If content changed, we need to rewrite this commit
|
||||
if original_content != filtered_content:
|
||||
return True
|
||||
return False
|
||||
|
||||
def main():
|
||||
print("Getting all commits...")
|
||||
commits = get_all_commits()
|
||||
print(f"Found {len(commits)} commits")
|
||||
|
||||
# Use git filter-branch with a custom script
|
||||
filter_script = '''#!/bin/bash
|
||||
git rm --cached --ignore-unmatch README.md
|
||||
git checkout HEAD -- README.md 2>/dev/null || true
|
||||
if [ -f README.md ]; then
|
||||
python remove_japanese.py README.md
|
||||
git add README.md
|
||||
fi
|
||||
'''
|
||||
|
||||
with tempfile.NamedTemporaryFile(mode='w', suffix='.sh', delete=False) as f:
|
||||
f.write(filter_script)
|
||||
script_path = f.name
|
||||
|
||||
try:
|
||||
os.chmod(script_path, 0o755)
|
||||
env = os.environ.copy()
|
||||
env['FILTER_BRANCH_SQUELCH_WARNING'] = '1'
|
||||
|
||||
result = subprocess.run(
|
||||
['git', 'filter-branch', '--force', '--index-filter', f'bash {script_path}',
|
||||
'--prune-empty', '--tag-name-filter', 'cat', '--', '--all'],
|
||||
cwd=os.getcwd(),
|
||||
env=env
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
print("Successfully rewrote git history")
|
||||
else:
|
||||
print(f"Error: git filter-branch returned {result.returncode}")
|
||||
sys.exit(1)
|
||||
finally:
|
||||
os.unlink(script_path)
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
||||
@@ -1,363 +0,0 @@
|
||||
## Overview
|
||||
|
||||
Qwen Image models have modulation layers `img_mod.1` and `txt_mod.1` in each transformer block. These layers are linear layers that generate shift/scale/gate parameters for AdaNorm.
|
||||
|
||||
In Nunchaku's AWQ quantized models, these layers are implemented as quantized linear layers called `AWQW4A16Linear`. Standard LoRA application methods (directly modifying weights) cannot be applied to quantized layers, so we adopted a method that adds LoRA in the forward path.
|
||||
|
||||
However, as a result of implementation and testing, applying LoRA to `img_mod.1` and `txt_mod.1` caused image noise. Therefore, we implemented a safety measure to **skip LoRA application to these layers by default**.
|
||||
|
||||
## Detailed Code Implementation Explanation
|
||||
|
||||
### 1. Environment Variable Loading
|
||||
|
||||
**File**: `nunchaku_code/lora_qwen.py`
|
||||
**Line Numbers**: 23-28
|
||||
|
||||
```python
|
||||
# Safety switch:
|
||||
# QwenImage's modulation linears (`img_mod.1` / `txt_mod.1`) are extremely sensitive because their
|
||||
# output is reshaped into shift/scale/gate parameters. With AWQ quantization (AWQW4A16Linear),
|
||||
# applying LoRA here often results in severe noise. Default to skipping these two layers.
|
||||
# Users can override by setting env var `QWENIMAGE_LORA_APPLY_AWQ_MOD=1`.
|
||||
_APPLY_AWQ_MOD = str(os.getenv("QWENIMAGE_LORA_APPLY_AWQ_MOD", "0")).strip().lower() in ("1", "true", "yes", "y", "on")
|
||||
```
|
||||
|
||||
**Explanation**:
|
||||
|
||||
`_APPLY_AWQ_MOD` is a global variable that controls whether to apply LoRA to AWQ modulation layers.
|
||||
|
||||
1. `os.getenv("QWENIMAGE_LORA_APPLY_AWQ_MOD", "0")` reads the environment variable. The default value is "0" (disabled).
|
||||
2. `str(...).strip().lower()` normalizes the string (removes leading/trailing whitespace and converts to lowercase).
|
||||
3. `in ("1", "true", "yes", "y", "on")` checks if it is a valid value (values considered true).
|
||||
4. As a result, if the environment variable is not set, or has values like "0" / "false" / "no", it becomes `False` and is skipped by default.
|
||||
|
||||
**Debug Log Display**:
|
||||
|
||||
This variable itself is not displayed in logs, but logs are output by the skip logic described later.
|
||||
|
||||
### 2. AWQW4A16Linear Module Detection
|
||||
|
||||
**File**: `nunchaku_code/lora_qwen.py`
|
||||
**Line Numbers**: 1385-1392
|
||||
|
||||
```python
|
||||
is_awq_w4a16 = (
|
||||
module.__class__.__name__ == "AWQW4A16Linear"
|
||||
and hasattr(module, "qweight")
|
||||
and hasattr(module, "wscales")
|
||||
and hasattr(module, "wzeros")
|
||||
and hasattr(module, "in_features")
|
||||
and hasattr(module, "out_features")
|
||||
)
|
||||
```
|
||||
|
||||
**Explanation**:
|
||||
|
||||
Within the `compose_loras_v2` function, this determines whether each module is `AWQW4A16Linear`.
|
||||
|
||||
1. `module.__class__.__name__ == "AWQW4A16Linear"` checks the class name. We use class name instead of `isinstance()` because import paths may differ.
|
||||
2. `hasattr(module, "qweight")` checks if the quantized weight tensor exists. In AWQ, weights are quantized to INT4 and stored in `qweight`.
|
||||
3. `hasattr(module, "wscales")` and `hasattr(module, "wzeros")` check if quantization scales and zero points exist.
|
||||
4. `hasattr(module, "in_features")` and `hasattr(module, "out_features")` check if standard linear layer attributes exist.
|
||||
|
||||
If all these conditions are met, `is_awq_w4a16 = True` and the module is treated as an AWQ quantized layer.
|
||||
|
||||
**Debug Log Display**:
|
||||
|
||||
This detection process itself is not displayed in logs, but logs are output by subsequent processing.
|
||||
|
||||
### 3. AWQ Modulation Layer Skip Logic
|
||||
|
||||
**File**: `nunchaku_code/lora_qwen.py`
|
||||
**Line Numbers**: 1394-1400
|
||||
|
||||
```python
|
||||
# Default-skip AWQ modulation layers to avoid catastrophic noise.
|
||||
if is_awq_w4a16 and (".img_mod.1" in resolved_name or ".txt_mod.1" in resolved_name) and not _APPLY_AWQ_MOD:
|
||||
logger.warning(
|
||||
f"[SKIP] {resolved_name}: AWQ modulation layer LoRA is disabled by default (prevents noise). "
|
||||
f"Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable."
|
||||
)
|
||||
continue
|
||||
```
|
||||
|
||||
**Explanation**:
|
||||
|
||||
This processing skips LoRA application to AWQ quantized modulation layers (`img_mod.1` or `txt_mod.1`) by default.
|
||||
|
||||
When:
|
||||
1. `is_awq_w4a16` is `True`, and
|
||||
2. `resolved_name` contains `.img_mod.1` or `.txt_mod.1`, and
|
||||
3. `_APPLY_AWQ_MOD` is `False` (environment variable is disabled)
|
||||
|
||||
it skips LoRA application and proceeds to the next module (`continue`).
|
||||
|
||||
**Debug Log Display**:
|
||||
|
||||
When skipped, a WARNING level log like the following is output:
|
||||
|
||||
```
|
||||
[SKIP] transformer_blocks.0.img_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
[SKIP] transformer_blocks.0.txt_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
[SKIP] transformer_blocks.1.img_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
[SKIP] transformer_blocks.1.txt_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
...
|
||||
```
|
||||
|
||||
This log is output for each `img_mod.1` and `txt_mod.1` in each transformer block (typically 0 to 59, 60 blocks total). That is, up to 120 skip logs may be output with default settings.
|
||||
|
||||
### 4. LoRA Application to AWQW4A16Linear
|
||||
|
||||
**File**: `nunchaku_code/lora_qwen.py`
|
||||
**Line Numbers**: 1018-1043
|
||||
|
||||
```python
|
||||
elif module.__class__.__name__ == "AWQW4A16Linear" and hasattr(module, "qweight") and hasattr(module, "wscales") and hasattr(module, "wzeros"):
|
||||
if not hasattr(model, "_lora_slots"):
|
||||
model._lora_slots = {}
|
||||
|
||||
# Save original forward once
|
||||
if not hasattr(module, "_lora_original_forward"):
|
||||
module._lora_original_forward = module.forward
|
||||
|
||||
# Store LoRA weights on the module (already scaled/concatenated by caller)
|
||||
module._lora_A = A
|
||||
module._lora_B = B
|
||||
|
||||
def _forward_with_lora(x: torch.Tensor) -> torch.Tensor:
|
||||
# Base quantized forward
|
||||
y = module._lora_original_forward(x)
|
||||
# LoRA branch (dtype/device should already match caller cast)
|
||||
# A: [rank, in], B: [out, rank]
|
||||
lora_out = (x @ module._lora_A.T) @ module._lora_B.T
|
||||
return y + lora_out.to(dtype=y.dtype, device=y.device)
|
||||
|
||||
module.forward = _forward_with_lora
|
||||
|
||||
# Track for reset
|
||||
model._lora_slots[module_name] = {
|
||||
"type": "awq_w4a16",
|
||||
}
|
||||
```
|
||||
|
||||
**Explanation**:
|
||||
|
||||
This processing implements a method that adds LoRA terms in the forward path without modifying quantized weights when applying LoRA to AWQ quantized layers.
|
||||
|
||||
1. **LoRA Slot Initialization**: Initializes `model._lora_slots` if it doesn't exist. This dictionary is needed for resetting later.
|
||||
|
||||
2. **Original Forward Method Storage**: Saves the original `forward` method to `module._lora_original_forward`. This allows restoring the original behavior during reset. This processing runs only once (due to `hasattr` check).
|
||||
|
||||
3. **LoRA Weight Storage**: Stores LoRA A and B matrices to `module._lora_A` and `module._lora_B`. These weights have already been scaled and concatenated by the caller (`compose_loras_v2`).
|
||||
|
||||
4. **LoRA-Enabled Forward Implementation**: Defines an internal function `_forward_with_lora`. This function:
|
||||
- Executes normal forward with quantized weights via `module._lora_original_forward(x)`.
|
||||
- Calculates LoRA terms via `(x @ module._lora_A.T) @ module._lora_B.T`. Here, `A` is shape `[rank, in_features]`, `B` is shape `[out_features, rank]`. `A.T` is `[in_features, rank]`, `B.T` is `[rank, out_features]`, so `x @ A.T @ B.T` is `[batch, in_features] @ [in_features, rank] @ [rank, out_features] = [batch, out_features]`.
|
||||
- Adds LoRA terms to quantized forward results via `y + lora_out.to(dtype=y.dtype, device=y.device)`. `to(dtype=..., device=...)` aligns data type and device.
|
||||
|
||||
5. **Forward Method Replacement**: Replaces the module's forward method with the LoRA-enabled version via `module.forward = _forward_with_lora`.
|
||||
|
||||
6. **Slot Information Recording**: Records `{"type": "awq_w4a16"}` to `model._lora_slots[module_name]`. This allows identifying that this module is an AWQ quantized layer during reset.
|
||||
|
||||
**Debug Log Display**:
|
||||
|
||||
When this processing is executed, the following log is output by subsequent processing in the `compose_loras_v2` function (lines 1460-1461):
|
||||
|
||||
```
|
||||
[APPLY] LoRA applied to: transformer_blocks.0.img_mod.1
|
||||
```
|
||||
|
||||
However, with default settings (`_APPLY_AWQ_MOD = False`), `img_mod.1` and `txt_mod.1` are skipped, so this log is not output.
|
||||
|
||||
### 5. B Matrix Reordering for Modulation Layers
|
||||
|
||||
**File**: `nunchaku_code/lora_qwen.py`
|
||||
**Line Numbers**: 1417-1433
|
||||
|
||||
```python
|
||||
# QwenImage modulation layers (img_mod.1 / txt_mod.1) are AWQ-quantized in Nunchaku
|
||||
# and their output is *interleaved* as [B, dim, 6] flattened, then later reordered by:
|
||||
# view(B, dim, 6).transpose(1, 2).reshape(B, 6*dim)
|
||||
#
|
||||
# Most LoRAs are trained against the *standard* (non-interleaved) layout (6 blocks of dim).
|
||||
# Because our AWQ LoRA path adds the LoRA output BEFORE that reorder, we must convert the
|
||||
# LoRA "up" matrix B from standard->interleaved order (permute output channels).
|
||||
if (".img_mod.1" in resolved_name) or (".txt_mod.1" in resolved_name):
|
||||
if B.ndim == 2 and (B.shape[0] % 6 == 0):
|
||||
dim = B.shape[0] // 6
|
||||
# standard [6, dim, r] -> interleaved [dim, 6, r] -> [6*dim, r]
|
||||
B = B.contiguous().view(6, dim, B.shape[1]).transpose(0, 1).reshape(B.shape[0], B.shape[1])
|
||||
else:
|
||||
logger.warning(
|
||||
f"{resolved_name}: expected mod up-matrix with out_features divisible by 6, got B{tuple(B.shape)}; "
|
||||
f"skipping mod-channel reorder"
|
||||
)
|
||||
```
|
||||
|
||||
**Explanation**:
|
||||
|
||||
In Qwen Image's modulation layers, output is used as shift/scale/gate parameters (6 blocks total). In Nunchaku's implementation, this output is processed in "interleaved" format.
|
||||
|
||||
1. **Problem Background**: Most LoRAs are trained against the standard (non-interleaved) layout (6 blocks × dim). However, in the AWQ LoRA path, since LoRA output is added **before** the reordering process, the B matrix must be converted from standard to interleaved format.
|
||||
|
||||
2. **Shape Check**: `B.ndim == 2 and (B.shape[0] % 6 == 0)` checks if B is 2-dimensional and if `out_features` is divisible by 6. This assumes a 6-block structure (shift/scale/gate × 2).
|
||||
|
||||
3. **Reordering Processing**:
|
||||
- `dim = B.shape[0] // 6` calculates the dimension count for each block.
|
||||
- `B.contiguous().view(6, dim, B.shape[1])` reshapes B to shape `[6, dim, rank]`. `contiguous()` is needed to ensure contiguous memory layout.
|
||||
- `.transpose(0, 1)` converts to `[dim, 6, rank]`. This converts from standard format `[6, dim, rank]` to interleaved format `[dim, 6, rank]`.
|
||||
- `.reshape(B.shape[0], B.shape[1])` finally returns to shape `[6*dim, rank]`.
|
||||
|
||||
4. **Error Handling**: If B's shape differs from expectations (e.g., not divisible by 6), reordering is skipped and a warning log is output.
|
||||
|
||||
**Debug Log Display**:
|
||||
|
||||
When reordering is skipped (shape differs from expectations), a WARNING log like the following is output:
|
||||
|
||||
```
|
||||
transformer_blocks.0.img_mod.1: expected mod up-matrix with out_features divisible by 6, got B(1234, 16); skipping mod-channel reorder
|
||||
```
|
||||
|
||||
However, with default settings, `img_mod.1` and `txt_mod.1` are skipped, so this processing itself is not executed (only executed when enabled via environment variable).
|
||||
|
||||
### 6. LoRA Reset Processing (AWQ Quantized Layer Restoration)
|
||||
|
||||
**File**: `nunchaku_code/lora_qwen.py`
|
||||
**Line Numbers**: 1568-1581
|
||||
|
||||
```python
|
||||
elif module_type == "awq_w4a16":
|
||||
# Restore original forward and remove attached LoRA tensors
|
||||
if hasattr(module, "_lora_original_forward"):
|
||||
try:
|
||||
module.forward = module._lora_original_forward
|
||||
except Exception:
|
||||
# Safety: never fail reset
|
||||
pass
|
||||
for attr in ("_lora_A", "_lora_B", "_lora_original_forward"):
|
||||
if hasattr(module, attr):
|
||||
try:
|
||||
delattr(module, attr)
|
||||
except Exception:
|
||||
pass
|
||||
```
|
||||
|
||||
**Explanation**:
|
||||
|
||||
Reset processing for LoRA applied to AWQ quantized layers (restoring to original state) within the `reset_lora_v2` function.
|
||||
|
||||
1. **Forward Method Restoration**: Checks if the original forward method is saved via `hasattr(module, "_lora_original_forward")`. If saved, restores the original forward method via `module.forward = module._lora_original_forward`.
|
||||
|
||||
2. **LoRA Attribute Deletion**: Deletes each attribute `_lora_A`, `_lora_B`, `_lora_original_forward`. This restores the module to its original state.
|
||||
|
||||
3. **Error Handling**: Each operation is wrapped in `try-except` to ensure the reset process doesn't fail even if errors occur (as the comment "Safety: never fail reset" indicates, reset must always succeed).
|
||||
|
||||
**Debug Log Display**:
|
||||
|
||||
At the end of the `reset_lora_v2` function (line 1585), the following INFO level log is output:
|
||||
|
||||
```
|
||||
All LoRA weights have been reset from the model.
|
||||
```
|
||||
|
||||
This log is output after reset of all LoRA types (Nunchaku LoRA-ready, standard Linear, AWQ quantized layers) is completed.
|
||||
|
||||
## Log Appearance (Default Settings: Skip Enabled)
|
||||
|
||||
With default settings (`QWENIMAGE_LORA_APPLY_AWQ_MOD` not set or 0), logs are output as follows:
|
||||
|
||||
```
|
||||
[SKIP] transformer_blocks.0.img_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
[SKIP] transformer_blocks.0.txt_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
[SKIP] transformer_blocks.1.img_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
[SKIP] transformer_blocks.1.txt_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
...
|
||||
[SKIP] transformer_blocks.59.img_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
[SKIP] transformer_blocks.59.txt_mod.1: AWQ modulation layer LoRA is disabled by default (prevents noise). Set QWENIMAGE_LORA_APPLY_AWQ_MOD=1 to force-enable.
|
||||
```
|
||||
|
||||
For each `img_mod.1` and `txt_mod.1` in each transformer block (0 to 59, 60 blocks total), a WARNING level skip log is output. That is, up to 120 skip logs are output.
|
||||
|
||||
After that, a log of the number of applied modules is output:
|
||||
|
||||
```
|
||||
Applied LoRA compositions to 480 modules.
|
||||
```
|
||||
|
||||
This number 480 is the total number of modules (600) minus the number of skipped modules (120). Each transformer block typically has 10 LoRA application locations, but 2 of them (`img_mod.1` and `txt_mod.1`) are skipped, so 60 blocks × 8 locations = 480 modules are applied.
|
||||
|
||||
## Log Appearance (Environment Variable Enabled: Skip Disabled)
|
||||
|
||||
When environment variable `QWENIMAGE_LORA_APPLY_AWQ_MOD=1` is set, skip logs are not output, and instead apply logs are output:
|
||||
|
||||
```
|
||||
[APPLY] LoRA applied to: transformer_blocks.0.img_mod.1
|
||||
[APPLY] LoRA applied to: transformer_blocks.0.txt_mod.1
|
||||
[APPLY] LoRA applied to: transformer_blocks.1.img_mod.1
|
||||
[APPLY] LoRA applied to: transformer_blocks.1.txt_mod.1
|
||||
...
|
||||
[APPLY] LoRA applied to: transformer_blocks.59.img_mod.1
|
||||
[APPLY] LoRA applied to: transformer_blocks.59.txt_mod.1
|
||||
```
|
||||
|
||||
After that, a log of the number of applied modules is output:
|
||||
|
||||
```
|
||||
Applied LoRA compositions to 600 modules.
|
||||
```
|
||||
|
||||
This number 600 is the total number of modules (60 blocks × 10 locations).
|
||||
|
||||
However, **enabling the environment variable has a high probability of causing image noise**, so this setting is not recommended.
|
||||
|
||||
## Reason for `Applied LoRA compositions to 480 modules.`
|
||||
|
||||
Qwen Image (Double Block) typically applies to approximately 10 locations in each `transformer_blocks.X` (when LoRA keys exist):
|
||||
|
||||
1. `attn.to_qkv` (fused Q/K/V attention)
|
||||
2. `attn.to_out.0`
|
||||
3. `attn.add_qkv_proj` (additional Q/K/V projection)
|
||||
4. `attn.to_add_out`
|
||||
5. `img_mlp.net.0.proj`
|
||||
6. `img_mlp.net.2`
|
||||
7. `img_mod.1` ← **Skip target**
|
||||
8. `txt_mlp.net.0.proj`
|
||||
9. `txt_mlp.net.2`
|
||||
10. `txt_mod.1` ← **Skip target**
|
||||
|
||||
With default settings, 2 locations (`img_mod.1` and `txt_mod.1`) are skipped.
|
||||
|
||||
For **60 blocks** from `transformer_blocks.0` to `59`:
|
||||
|
||||
- Skip count: 60 × 2 = 120
|
||||
- Apply count: 600 - 120 = 480
|
||||
|
||||
This matches the log.
|
||||
|
||||
## If You Still Want to Apply to Modulation Layers (Not Recommended)
|
||||
|
||||
In conclusion, **there is no guarantee that application will not cause noise** (varies by LoRA/strength/model instance, low reproducibility).
|
||||
If you still want to try, you can force it ON with an environment variable.
|
||||
|
||||
### PowerShell (Before Startup)
|
||||
|
||||
```powershell
|
||||
setx QWENIMAGE_LORA_APPLY_AWQ_MOD 1
|
||||
```
|
||||
|
||||
To restore (OFF):
|
||||
|
||||
```powershell
|
||||
setx QWENIMAGE_LORA_APPLY_AWQ_MOD 0
|
||||
```
|
||||
|
||||
After setting, **ComfyUI restart is required** (does not reflect in already running processes).
|
||||
|
||||
## Summary
|
||||
|
||||
- **Default Behavior**: LoRA application to `img_mod.1` and `txt_mod.1` is skipped (to prevent noise).
|
||||
- **Skip Logs**: With default settings, a WARNING log is output for each modulation layer (up to 120 entries).
|
||||
- **Apply Logs**: When enabled via environment variable, an INFO log is output for each modulation layer (up to 120 entries).
|
||||
- **Apply Count**: 480 modules with defaults, 600 modules when environment variable is enabled.
|
||||
- **Environment Variable**: Can be force-enabled with `QWENIMAGE_LORA_APPLY_AWQ_MOD=1`, but this is not recommended due to noise risk.
|
||||
|
||||
Reference in New Issue
Block a user