Update node display name, README docs, and add diffsynth node image
This commit is contained in:
ussoewwin
@@ -77,7 +77,9 @@ By default, detailed debug logs are **muted**. If you want detailed debug output
|
||||
|
||||
<img src="images/zitv4_stack.png" alt="NunchakuZImageTurboLoraStackV4: Z-Image-Turbo LoRA stacker with dynamic UI - Standard ComfyUI LoRA loader format (CLIP input/output) - ComfyUI Nodes 2.0 compatible" width="400">
|
||||
|
||||
- **NunchakuQI&ZITDiffsynthControlnet**: DiffSynth ControlNet support node for Nunchaku Qwen Image, Z-ImageTurbo & Krea2 (depth / openpose)
|
||||
- **Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet** (class: `NunchakuQwenImageDiffsynthControlnet`): DiffSynth ControlNet support node for **Nunchaku Z-ImageTurbo**, standard (non-Nunchaku) Qwen Image, & Krea2 (depth / openpose). **Nunchaku Qwen Image is NOT supported** by the DiffSynth route (quantized hidden-state scale mismatch produces broken output); use a standard bf16 Qwen Image model for the DiffSynth ControlNet route, or Nunchaku Z-Image-Turbo for the Z-Image route.
|
||||
|
||||
<img src="images/diffsynth_controlnet.png" alt="Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet" width="420">
|
||||
|
||||
- **Krea2ControlNetLoraLoader**: Krea2 controlnet-lora loader (depth & openpose)
|
||||
- Auto-detects the Krea2 control sub-type from the LoRA file: **depth** (`first.weight` expansion route) or **openpose** (pure block LoRA + native reference-latent route). Routing is strictly exclusive.
|
||||
@@ -103,12 +105,12 @@ By default, detailed debug logs are **muted**. If you want detailed debug output
|
||||
|
||||
1. Load your diffsynth ControlNet model patch using `Model Patch Loader` from [ComfyUI-NunchakuFluxLoraStacker](https://github.com/ussoewwin/ComfyUI-NunchakuFluxLoraStacker)
|
||||
2. The `Model Patch Loader` (`ModelPatchLoaderCustom`) supports CPU offload, allowing you to load ControlNet patches to CPU memory to save VRAM
|
||||
3. Connect the `MODEL_PATCH` output to the `model_patch` input of `NunchakuQI&ZITDiffsynthControlnet` node
|
||||
3. Connect the `MODEL_PATCH` output to the `model_patch` input of `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet` node
|
||||
4. Connect your Nunchaku Qwen Image model, VAE, and control image
|
||||
5. Set the ControlNet strength and connect to your workflow
|
||||
6. **Krea2 control support** (depth & openpose): add `Krea2ControlNetLoraLoader`, select
|
||||
your Krea2 controlnet-lora file, then connect its `MODEL_PATCH` output to the `model_patch`
|
||||
input of `NunchakuQI&ZITDiffsynthControlnet`. The node auto-detects the Krea2 sub-type:
|
||||
input of `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet`. The node auto-detects the Krea2 sub-type:
|
||||
- **Depth** (e.g. `krea2-depth-control-lora.safetensors`): expanded first-projection route
|
||||
with control-latent injection. This is the original Krea2 depth behavior, unchanged.
|
||||
- **OpenPose** (e.g. `krea2_turbo_openpose_controlnet.safetensors`): pure block-LoRA route on
|
||||
@@ -119,13 +121,13 @@ By default, detailed debug logs are **muted**. If you want detailed debug output
|
||||
|
||||
### Krea2 Control (Depth & OpenPose)
|
||||
|
||||
`NunchakuQI&ZITDiffsynthControlnet` auto-detects the Krea2 control sub-type from the loaded
|
||||
`Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet` auto-detects the Krea2 control sub-type from the loaded
|
||||
LoRA file and applies a dedicated, self-contained route - no external nodes required.
|
||||
|
||||
#### Krea2 Depth Control
|
||||
|
||||
1. Load the depth control LoRA with `Krea2ControlNetLoraLoader` (e.g. `krea2-depth-control-lora.safetensors`)
|
||||
2. Connect its `MODEL_PATCH` to `NunchakuQI&ZITDiffsynthControlnet`'s `model_patch`
|
||||
2. Connect its `MODEL_PATCH` to `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet`'s `model_patch`
|
||||
3. Connect your Krea2 model, VAE and the depth image; set `strength`
|
||||
|
||||
**How it works:**
|
||||
@@ -138,10 +140,10 @@ LoRA file and applies a dedicated, self-contained route - no external nodes requ
|
||||
#### Krea2 OpenPose Control
|
||||
|
||||
The `krea2_turbo_openpose_controlnet.safetensors` control LoRA is applied through a dedicated,
|
||||
self-contained route in `NunchakuQI&ZITDiffsynthControlnet`:
|
||||
self-contained route in `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet`:
|
||||
|
||||
1. Load the openpose control LoRA with `Krea2ControlNetLoraLoader`
|
||||
2. Connect its `MODEL_PATCH` to `NunchakuQI&ZITDiffsynthControlnet`'s `model_patch`
|
||||
2. Connect its `MODEL_PATCH` to `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet`'s `model_patch`
|
||||
3. Connect your Krea2 model, VAE and the pose image; set `strength`
|
||||
4. Keep `use_kv_cache = True` (default) for the isolated reference K/V mode
|
||||
|
||||
|
||||
+1
-1
@@ -143,7 +143,7 @@ NODE_DISPLAY_NAME_MAPPINGS["Krea2ControlNetLoraLoader"] = "Krea2 controlnet lora
|
||||
|
||||
# Add ControlNet display name only if the node was successfully loaded
|
||||
if "NunchakuQwenImageDiffsynthControlnet" in NODE_CLASS_MAPPINGS:
|
||||
NODE_DISPLAY_NAME_MAPPINGS["NunchakuQwenImageDiffsynthControlnet"] = "NunchakuQI&ZITDiffsynthControlnet"
|
||||
NODE_DISPLAY_NAME_MAPPINGS["NunchakuQwenImageDiffsynthControlnet"] = "Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet"
|
||||
|
||||
# Register JavaScript extensions
|
||||
WEB_DIRECTORY = "js"
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 177 KiB |
+34
-7
@@ -51,7 +51,7 @@ def _classify_controlnet_target(model, model_patch):
|
||||
dm_name = dm.__class__.__name__
|
||||
dm_module = getattr(dm.__class__, "__module__", "")
|
||||
|
||||
if dm_name in ("ComfyQwenImageWrapper", "NunchakuQwenImageTransformer2DModel"):
|
||||
if dm_name in ("ComfyQwenImageWrapper", "NunchakuQwenImageTransformer2DModel", "QwenImageTransformer2DModel"):
|
||||
return "qwenimage_standard"
|
||||
|
||||
if dm_name == "SingleStreamDiT" or "comfy.ldm.krea2" in dm_module:
|
||||
@@ -942,6 +942,11 @@ class ZImageControlPatch:
|
||||
self.temp_data = (next_layer, self.model_patch.model.forward_control_block(next_layer, self.temp_data[1][1], img_input[:, :self.temp_data[1][1].shape[1]], None, pe, vec))
|
||||
|
||||
if cnet_index_float == self.temp_data[0]:
|
||||
try:
|
||||
with open(r"D:\USERFILES\ComfyUI\ComfyUI\custom_nodes\ComfyUI-nunchaku\_zimg_debug.log", "a", encoding="utf-8") as _f:
|
||||
_f.write(f"ZCNET block={block_index} res_mean={float(self.temp_data[1][0].abs().mean()):.4f} res_max={float(self.temp_data[1][0].abs().max()):.4f} img_mean={float(img.abs().mean()):.4f} str={self.strength}\n")
|
||||
except Exception:
|
||||
pass
|
||||
img[:, :self.temp_data[1][0].shape[1]] += (self.temp_data[1][0] * self.strength)
|
||||
if cnet_blocks == self.temp_data[0] + 1:
|
||||
self.temp_data = None
|
||||
@@ -971,14 +976,18 @@ class DiffSynthCnetBlockReplace:
|
||||
self.block_index = block_index
|
||||
|
||||
def __call__(self, args, extra_options):
|
||||
try:
|
||||
with open(r"D:\USERFILES\ComfyUI\ComfyUI\custom_nodes\ComfyUI-nunchaku\_cnet_debug.log", "a", encoding="utf-8") as _dbgf:
|
||||
_dbgf.write(f"BlockReplace CALLED block={self.block_index}\n")
|
||||
except Exception:
|
||||
pass
|
||||
# Run the original transformer block first
|
||||
out = extra_options["original_block"](args)
|
||||
|
||||
# Apply DiffSynth ControlNet residual
|
||||
img = out["img"]
|
||||
|
||||
# We need `x` to determine target resolution for dynamic resizing, but Nunchaku doesn't pass it to patches_replace.
|
||||
# Find `x` in the call stack (BaseModel.apply_model or ComfyQwenImageWrapper.forward):
|
||||
|
||||
# We need `x` to determine target resolution for dynamic resizing
|
||||
import sys
|
||||
x = None
|
||||
frame = sys._getframe()
|
||||
@@ -989,12 +998,12 @@ class DiffSynthCnetBlockReplace:
|
||||
x = candidate_x
|
||||
break
|
||||
frame = frame.f_back
|
||||
|
||||
|
||||
if x is not None:
|
||||
spacial_compression = self.cnet_patch.vae.spacial_compression_encode()
|
||||
target_h = x.shape[-2] * spacial_compression
|
||||
target_w = x.shape[-1] * spacial_compression
|
||||
|
||||
|
||||
if self.cnet_patch.encoded_image is None or self.cnet_patch.encoded_image_size != (target_h, target_w):
|
||||
logger.info(f"[ControlNet Block {self.block_index}] Resizing condition image to {target_w}x{target_h}")
|
||||
image_scaled = comfy.utils.common_upscale(self.cnet_patch.image.movedim(-1, 1), target_w, target_h, "area", "center")
|
||||
@@ -1002,7 +1011,7 @@ class DiffSynthCnetBlockReplace:
|
||||
self.cnet_patch.encoded_image = self.cnet_patch.model_patch.model.process_input_latent_image(self.cnet_patch.encode_latent_cond(image_scaled.movedim(1, -1)))
|
||||
self.cnet_patch.encoded_image_size = (target_h, target_w)
|
||||
comfy.model_management.load_models_gpu(loaded_models)
|
||||
|
||||
|
||||
encoded_image = self.cnet_patch.encoded_image
|
||||
if encoded_image is not None:
|
||||
control_residual = self.cnet_patch.model_patch.model.control_block(
|
||||
@@ -1010,6 +1019,19 @@ class DiffSynthCnetBlockReplace:
|
||||
encoded_image.to(img.dtype),
|
||||
self.block_index
|
||||
)
|
||||
if self.block_index == 0:
|
||||
try:
|
||||
with open(r"D:\USERFILES\ComfyUI\ComfyUI\custom_nodes\ComfyUI-nunchaku\_cnet_debug.log", "a", encoding="utf-8") as _dbgf:
|
||||
_dbgf.write(
|
||||
"STATS img_mean={:.4f} res_mean={:.4f} res_max={:.4f} str={} enc_shape={} img_shape={}\n".format(
|
||||
float(img.abs().mean()),
|
||||
float(control_residual.abs().mean()), float(control_residual.abs().max()),
|
||||
float(self.cnet_patch.strength),
|
||||
tuple(encoded_image.shape), tuple(img.shape),
|
||||
)
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
img[:, :encoded_image.shape[1]] += control_residual * self.cnet_patch.strength
|
||||
out["img"] = img
|
||||
|
||||
@@ -1045,6 +1067,11 @@ class NunchakuQwenImageDiffsynthControlnet:
|
||||
|
||||
route = _classify_controlnet_target(model, model_patch)
|
||||
logger.info(f"[ControlNet] route={route}")
|
||||
try:
|
||||
with open(r"D:\USERFILES\ComfyUI\ComfyUI\custom_nodes\ComfyUI-nunchaku\_cnet_debug.log", "a", encoding="utf-8") as _dbgf:
|
||||
_dbgf.write(f"route={route} model_base={model.model.__class__.__name__} model_patch_model={type(model_patch.model).__name__}\n")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if route == "zimage":
|
||||
# ZImage ControlNet (works for both standard and Nunchaku Z-Image)
|
||||
|
||||
+9
-7
@@ -152,7 +152,9 @@
|
||||
|
||||
|
||||
|
||||
- **NunchakuQI&ZITDiffsynthControlnet**: 用于 Nunchaku Qwen Image、Z-ImageTurbo 与 Krea2（depth / openpose）的 DiffSynth ControlNet 支持节点
|
||||
- **Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet**（类名: `NunchakuQwenImageDiffsynthControlnet`）: 为 **Nunchaku Z-ImageTurbo**、标准（非 Nunchaku）Qwen Image 与 Krea2（depth / openpose）提供 DiffSynth ControlNet 支持。**Nunchaku Qwen Image 不受支持**（量化后的 hidden-state 尺度不匹配会导致输出损坏）；DiffSynth ControlNet 路由请使用标准 bf16 Qwen Image 模型，Z-Image 路由请使用 Nunchaku Z-Image-Turbo。
|
||||
|
||||
<img src="../images/diffsynth_controlnet.png" alt="Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet" width="420">
|
||||
|
||||
|
||||
- **Krea2ControlNetLoraLoader**: Krea2 controlnet-lora 加载器（depth & openpose）
|
||||
@@ -201,7 +203,7 @@
|
||||
|
||||
2. `Model Patch Loader`（即 `ModelPatchLoaderCustom` 节点）支持 CPU 卸载，允许您将 ControlNet 补丁加载到 CPU 内存以节省 VRAM
|
||||
|
||||
3. 将 `MODEL_PATCH` 输出连接到 `NunchakuQI&ZITDiffsynthControlnet` 节点的 `model_patch` 输入端
|
||||
3. 将 `MODEL_PATCH` 输出连接到 `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet` 节点的 `model_patch` 输入端
|
||||
|
||||
4. 连接您的 Nunchaku Qwen Image 模型、VAE 和控制图像
|
||||
|
||||
@@ -209,7 +211,7 @@
|
||||
|
||||
6. **Krea2 控制支持**（depth & openpose）：添加 `Krea2ControlNetLoraLoader`，选择你的
|
||||
Krea2 controlnet-lora 文件，然后将其 `MODEL_PATCH` 输出连接到
|
||||
`NunchakuQI&ZITDiffsynthControlnet` 的 `model_patch` 输入端。节点会自动检测 Krea2 子类型：
|
||||
`Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet` 的 `model_patch` 输入端。节点会自动检测 Krea2 子类型：
|
||||
- **Depth**（例如 `krea2-depth-control-lora.safetensors`）：扩展 first 投影路由 + 控制潜变量
|
||||
注入。这是原有的 Krea2 depth 行为，完全未改动。
|
||||
- **OpenPose**（例如 `krea2_turbo_openpose_controlnet.safetensors`）：纯块 LoRA 路由，作用于
|
||||
@@ -219,13 +221,13 @@
|
||||
|
||||
### Krea2 控制（Depth & OpenPose）
|
||||
|
||||
`NunchakuQI&ZITDiffsynthControlnet` 会根据加载的 LoRA 文件自动检测 Krea2 控制子类型，并应用
|
||||
`Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet` 会根据加载的 LoRA 文件自动检测 Krea2 控制子类型，并应用
|
||||
独立、自包含的路由——无需任何外部节点。
|
||||
|
||||
#### Krea2 Depth 控制
|
||||
|
||||
1. 使用 `Krea2ControlNetLoraLoader` 加载 depth 控制 LoRA（例如 `krea2-depth-control-lora.safetensors`）
|
||||
2. 将其 `MODEL_PATCH` 输出连接到 `NunchakuQI&ZITDiffsynthControlnet` 的 `model_patch` 输入端
|
||||
2. 将其 `MODEL_PATCH` 输出连接到 `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet` 的 `model_patch` 输入端
|
||||
3. 连接你的 Krea2 模型、VAE 与深度图，并设置 `strength`
|
||||
|
||||
**工作原理：**
|
||||
@@ -237,10 +239,10 @@
|
||||
#### Krea2 OpenPose 控制
|
||||
|
||||
`krea2_turbo_openpose_controlnet.safetensors` 控制 LoRA 通过
|
||||
`NunchakuQI&ZITDiffsynthControlnet` 中独立、自包含的路由应用：
|
||||
`Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet` 中独立、自包含的路由应用：
|
||||
|
||||
1. 使用 `Krea2ControlNetLoraLoader` 加载 openpose 控制 LoRA
|
||||
2. 将其 `MODEL_PATCH` 输出连接到 `NunchakuQI&ZITDiffsynthControlnet` 的 `model_patch` 输入端
|
||||
2. 将其 `MODEL_PATCH` 输出连接到 `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet` 的 `model_patch` 输入端
|
||||
3. 连接你的 Krea2 模型、VAE 与姿态图，并设置 `strength`
|
||||
4. 保持 `use_kv_cache = True`（默认值）以启用隔离参考 K/V 模式
|
||||
|
||||
|
||||
Reference in New Issue
Block a user