diff --git a/README.md b/README.md index f3a544a..5e62919 100644 --- a/README.md +++ b/README.md @@ -77,7 +77,9 @@ By default, detailed debug logs are **muted**. If you want detailed debug output NunchakuZImageTurboLoraStackV4: Z-Image-Turbo LoRA stacker with dynamic UI - Standard ComfyUI LoRA loader format (CLIP input/output) - ComfyUI Nodes 2.0 compatible -- **NunchakuQI&ZITDiffsynthControlnet**: DiffSynth ControlNet support node for Nunchaku Qwen Image, Z-ImageTurbo & Krea2 (depth / openpose) +- **Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet** (class: `NunchakuQwenImageDiffsynthControlnet`): DiffSynth ControlNet support node for **Nunchaku Z-ImageTurbo**, standard (non-Nunchaku) Qwen Image, & Krea2 (depth / openpose). **Nunchaku Qwen Image is NOT supported** by the DiffSynth route (quantized hidden-state scale mismatch produces broken output); use a standard bf16 Qwen Image model for the DiffSynth ControlNet route, or Nunchaku Z-Image-Turbo for the Z-Image route. + +Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet - **Krea2ControlNetLoraLoader**: Krea2 controlnet-lora loader (depth & openpose) - Auto-detects the Krea2 control sub-type from the LoRA file: **depth** (`first.weight` expansion route) or **openpose** (pure block LoRA + native reference-latent route). Routing is strictly exclusive. @@ -103,12 +105,12 @@ By default, detailed debug logs are **muted**. If you want detailed debug output 1. Load your diffsynth ControlNet model patch using `Model Patch Loader` from [ComfyUI-NunchakuFluxLoraStacker](https://github.com/ussoewwin/ComfyUI-NunchakuFluxLoraStacker) 2. The `Model Patch Loader` (`ModelPatchLoaderCustom`) supports CPU offload, allowing you to load ControlNet patches to CPU memory to save VRAM -3. Connect the `MODEL_PATCH` output to the `model_patch` input of `NunchakuQI&ZITDiffsynthControlnet` node +3. Connect the `MODEL_PATCH` output to the `model_patch` input of `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet` node 4. Connect your Nunchaku Qwen Image model, VAE, and control image 5. Set the ControlNet strength and connect to your workflow 6. **Krea2 control support** (depth & openpose): add `Krea2ControlNetLoraLoader`, select your Krea2 controlnet-lora file, then connect its `MODEL_PATCH` output to the `model_patch` - input of `NunchakuQI&ZITDiffsynthControlnet`. The node auto-detects the Krea2 sub-type: + input of `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet`. The node auto-detects the Krea2 sub-type: - **Depth** (e.g. `krea2-depth-control-lora.safetensors`): expanded first-projection route with control-latent injection. This is the original Krea2 depth behavior, unchanged. - **OpenPose** (e.g. `krea2_turbo_openpose_controlnet.safetensors`): pure block-LoRA route on @@ -119,13 +121,13 @@ By default, detailed debug logs are **muted**. If you want detailed debug output ### Krea2 Control (Depth & OpenPose) -`NunchakuQI&ZITDiffsynthControlnet` auto-detects the Krea2 control sub-type from the loaded +`Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet` auto-detects the Krea2 control sub-type from the loaded LoRA file and applies a dedicated, self-contained route - no external nodes required. #### Krea2 Depth Control 1. Load the depth control LoRA with `Krea2ControlNetLoraLoader` (e.g. `krea2-depth-control-lora.safetensors`) -2. Connect its `MODEL_PATCH` to `NunchakuQI&ZITDiffsynthControlnet`'s `model_patch` +2. Connect its `MODEL_PATCH` to `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet`'s `model_patch` 3. Connect your Krea2 model, VAE and the depth image; set `strength` **How it works:** @@ -138,10 +140,10 @@ LoRA file and applies a dedicated, self-contained route - no external nodes requ #### Krea2 OpenPose Control The `krea2_turbo_openpose_controlnet.safetensors` control LoRA is applied through a dedicated, -self-contained route in `NunchakuQI&ZITDiffsynthControlnet`: +self-contained route in `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet`: 1. Load the openpose control LoRA with `Krea2ControlNetLoraLoader` -2. Connect its `MODEL_PATCH` to `NunchakuQI&ZITDiffsynthControlnet`'s `model_patch` +2. Connect its `MODEL_PATCH` to `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet`'s `model_patch` 3. Connect your Krea2 model, VAE and the pose image; set `strength` 4. Keep `use_kv_cache = True` (default) for the isolated reference K/V mode diff --git a/__init__.py b/__init__.py index 923eef5..57935f8 100644 --- a/__init__.py +++ b/__init__.py @@ -143,7 +143,7 @@ NODE_DISPLAY_NAME_MAPPINGS["Krea2ControlNetLoraLoader"] = "Krea2 controlnet lora # Add ControlNet display name only if the node was successfully loaded if "NunchakuQwenImageDiffsynthControlnet" in NODE_CLASS_MAPPINGS: - NODE_DISPLAY_NAME_MAPPINGS["NunchakuQwenImageDiffsynthControlnet"] = "NunchakuQI&ZITDiffsynthControlnet" + NODE_DISPLAY_NAME_MAPPINGS["NunchakuQwenImageDiffsynthControlnet"] = "Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet" # Register JavaScript extensions WEB_DIRECTORY = "js" diff --git a/images/diffsynth_controlnet.png b/images/diffsynth_controlnet.png new file mode 100644 index 0000000..78adc54 Binary files /dev/null and b/images/diffsynth_controlnet.png differ diff --git a/nodes/controlnet.py b/nodes/controlnet.py index 6d59416..aaef22a 100644 --- a/nodes/controlnet.py +++ b/nodes/controlnet.py @@ -51,7 +51,7 @@ def _classify_controlnet_target(model, model_patch): dm_name = dm.__class__.__name__ dm_module = getattr(dm.__class__, "__module__", "") - if dm_name in ("ComfyQwenImageWrapper", "NunchakuQwenImageTransformer2DModel"): + if dm_name in ("ComfyQwenImageWrapper", "NunchakuQwenImageTransformer2DModel", "QwenImageTransformer2DModel"): return "qwenimage_standard" if dm_name == "SingleStreamDiT" or "comfy.ldm.krea2" in dm_module: @@ -942,6 +942,11 @@ class ZImageControlPatch: self.temp_data = (next_layer, self.model_patch.model.forward_control_block(next_layer, self.temp_data[1][1], img_input[:, :self.temp_data[1][1].shape[1]], None, pe, vec)) if cnet_index_float == self.temp_data[0]: + try: + with open(r"D:\USERFILES\ComfyUI\ComfyUI\custom_nodes\ComfyUI-nunchaku\_zimg_debug.log", "a", encoding="utf-8") as _f: + _f.write(f"ZCNET block={block_index} res_mean={float(self.temp_data[1][0].abs().mean()):.4f} res_max={float(self.temp_data[1][0].abs().max()):.4f} img_mean={float(img.abs().mean()):.4f} str={self.strength}\n") + except Exception: + pass img[:, :self.temp_data[1][0].shape[1]] += (self.temp_data[1][0] * self.strength) if cnet_blocks == self.temp_data[0] + 1: self.temp_data = None @@ -971,14 +976,18 @@ class DiffSynthCnetBlockReplace: self.block_index = block_index def __call__(self, args, extra_options): + try: + with open(r"D:\USERFILES\ComfyUI\ComfyUI\custom_nodes\ComfyUI-nunchaku\_cnet_debug.log", "a", encoding="utf-8") as _dbgf: + _dbgf.write(f"BlockReplace CALLED block={self.block_index}\n") + except Exception: + pass # Run the original transformer block first out = extra_options["original_block"](args) # Apply DiffSynth ControlNet residual img = out["img"] - - # We need `x` to determine target resolution for dynamic resizing, but Nunchaku doesn't pass it to patches_replace. - # Find `x` in the call stack (BaseModel.apply_model or ComfyQwenImageWrapper.forward): + + # We need `x` to determine target resolution for dynamic resizing import sys x = None frame = sys._getframe() @@ -989,12 +998,12 @@ class DiffSynthCnetBlockReplace: x = candidate_x break frame = frame.f_back - + if x is not None: spacial_compression = self.cnet_patch.vae.spacial_compression_encode() target_h = x.shape[-2] * spacial_compression target_w = x.shape[-1] * spacial_compression - + if self.cnet_patch.encoded_image is None or self.cnet_patch.encoded_image_size != (target_h, target_w): logger.info(f"[ControlNet Block {self.block_index}] Resizing condition image to {target_w}x{target_h}") image_scaled = comfy.utils.common_upscale(self.cnet_patch.image.movedim(-1, 1), target_w, target_h, "area", "center") @@ -1002,7 +1011,7 @@ class DiffSynthCnetBlockReplace: self.cnet_patch.encoded_image = self.cnet_patch.model_patch.model.process_input_latent_image(self.cnet_patch.encode_latent_cond(image_scaled.movedim(1, -1))) self.cnet_patch.encoded_image_size = (target_h, target_w) comfy.model_management.load_models_gpu(loaded_models) - + encoded_image = self.cnet_patch.encoded_image if encoded_image is not None: control_residual = self.cnet_patch.model_patch.model.control_block( @@ -1010,6 +1019,19 @@ class DiffSynthCnetBlockReplace: encoded_image.to(img.dtype), self.block_index ) + if self.block_index == 0: + try: + with open(r"D:\USERFILES\ComfyUI\ComfyUI\custom_nodes\ComfyUI-nunchaku\_cnet_debug.log", "a", encoding="utf-8") as _dbgf: + _dbgf.write( + "STATS img_mean={:.4f} res_mean={:.4f} res_max={:.4f} str={} enc_shape={} img_shape={}\n".format( + float(img.abs().mean()), + float(control_residual.abs().mean()), float(control_residual.abs().max()), + float(self.cnet_patch.strength), + tuple(encoded_image.shape), tuple(img.shape), + ) + ) + except Exception: + pass img[:, :encoded_image.shape[1]] += control_residual * self.cnet_patch.strength out["img"] = img @@ -1045,6 +1067,11 @@ class NunchakuQwenImageDiffsynthControlnet: route = _classify_controlnet_target(model, model_patch) logger.info(f"[ControlNet] route={route}") + try: + with open(r"D:\USERFILES\ComfyUI\ComfyUI\custom_nodes\ComfyUI-nunchaku\_cnet_debug.log", "a", encoding="utf-8") as _dbgf: + _dbgf.write(f"route={route} model_base={model.model.__class__.__name__} model_patch_model={type(model_patch.model).__name__}\n") + except Exception: + pass if route == "zimage": # ZImage ControlNet (works for both standard and Nunchaku Z-Image) diff --git a/zhmd/README.md b/zhmd/README.md index a59f988..b9b50e4 100644 --- a/zhmd/README.md +++ b/zhmd/README.md @@ -152,7 +152,9 @@ -- **NunchakuQI&ZITDiffsynthControlnet**: 用于 Nunchaku Qwen Image、Z-ImageTurbo 与 Krea2（depth / openpose）的 DiffSynth ControlNet 支持节点 +- **Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet**（类名: `NunchakuQwenImageDiffsynthControlnet`）: 为 **Nunchaku Z-ImageTurbo**、标准（非 Nunchaku）Qwen Image 与 Krea2（depth / openpose）提供 DiffSynth ControlNet 支持。**Nunchaku Qwen Image 不受支持**（量化后的 hidden-state 尺度不匹配会导致输出损坏）；DiffSynth ControlNet 路由请使用标准 bf16 Qwen Image 模型，Z-Image 路由请使用 Nunchaku Z-Image-Turbo。 + +Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet - **Krea2ControlNetLoraLoader**: Krea2 controlnet-lora 加载器（depth & openpose） @@ -201,7 +203,7 @@ 2. `Model Patch Loader`（即 `ModelPatchLoaderCustom` 节点）支持 CPU 卸载，允许您将 ControlNet 补丁加载到 CPU 内存以节省 VRAM -3. 将 `MODEL_PATCH` 输出连接到 `NunchakuQI&ZITDiffsynthControlnet` 节点的 `model_patch` 输入端 +3. 将 `MODEL_PATCH` 输出连接到 `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet` 节点的 `model_patch` 输入端 4. 连接您的 Nunchaku Qwen Image 模型、VAE 和控制图像 @@ -209,7 +211,7 @@ 6. **Krea2 控制支持**（depth & openpose）：添加 `Krea2ControlNetLoraLoader`，选择你的 Krea2 controlnet-lora 文件，然后将其 `MODEL_PATCH` 输出连接到 - `NunchakuQI&ZITDiffsynthControlnet` 的 `model_patch` 输入端。节点会自动检测 Krea2 子类型： + `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet` 的 `model_patch` 输入端。节点会自动检测 Krea2 子类型： - **Depth**（例如 `krea2-depth-control-lora.safetensors`）：扩展 first 投影路由 + 控制潜变量 注入。这是原有的 Krea2 depth 行为，完全未改动。 - **OpenPose**（例如 `krea2_turbo_openpose_controlnet.safetensors`）：纯块 LoRA 路由，作用于 @@ -219,13 +221,13 @@ ### Krea2 控制（Depth & OpenPose） -`NunchakuQI&ZITDiffsynthControlnet` 会根据加载的 LoRA 文件自动检测 Krea2 控制子类型，并应用 +`Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet` 会根据加载的 LoRA 文件自动检测 Krea2 控制子类型，并应用 独立、自包含的路由——无需任何外部节点。 #### Krea2 Depth 控制 1. 使用 `Krea2ControlNetLoraLoader` 加载 depth 控制 LoRA（例如 `krea2-depth-control-lora.safetensors`） -2. 将其 `MODEL_PATCH` 输出连接到 `NunchakuQI&ZITDiffsynthControlnet` 的 `model_patch` 输入端 +2. 将其 `MODEL_PATCH` 输出连接到 `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet` 的 `model_patch` 输入端 3. 连接你的 Krea2 模型、VAE 与深度图，并设置 `strength` **工作原理：** @@ -237,10 +239,10 @@ #### Krea2 OpenPose 控制 `krea2_turbo_openpose_controlnet.safetensors` 控制 LoRA 通过 -`NunchakuQI&ZITDiffsynthControlnet` 中独立、自包含的路由应用： +`Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet` 中独立、自包含的路由应用： 1. 使用 `Krea2ControlNetLoraLoader` 加载 openpose 控制 LoRA -2. 将其 `MODEL_PATCH` 输出连接到 `NunchakuQI&ZITDiffsynthControlnet` 的 `model_patch` 输入端 +2. 将其 `MODEL_PATCH` 输出连接到 `Nunchaku ZI Diffsynth Controlnet&Krea2 LoRA ControlNet` 的 `model_patch` 输入端 3. 连接你的 Krea2 模型、VAE 与姿态图，并设置 `strength` 4. 保持 `use_kv_cache = True`（默认值）以启用隔离参考 K/V 模式