nodes/Kosinkadink-ComfyUI-Advanced-ControlNet
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
382 Commits 12 Branches 0 Tags
f900f14b0b365b48c4c4c30c9ab2a63c87504d78
Commit Graph
63 Commits
Author SHA1 Message Date
Jedrzej KosinskiandAmp f900f14b0b Support Anima LLLite v2 models 
Amp-Thread-ID: https://ampcode.com/threads/T-019f7088-85bf-7703-87c3-f9c0ca5d711d
Co-authored-by: Amp <amp@ampcode.com>
 2026-07-17 15:44:56 -07:00
Jedrzej KosinskiandAmp 9538b054cd Cast SparseCtrl/SVD control models to unet dtype after loading 
Setting controlnet_config["dtype"] alone is not enough: comfy's lazy/zero-copy
state_dict loading (Windows + aimdo path in disable_weight_init) assigns the
on-disk fp32 tensors directly as parameters and ignores the configured dtype.
With the disable_weight_init ops (no runtime weight casting), the control model
then runs fp32 weights against fp16 activations, raising "mat1 and mat2 must have
the same dtype, but got Half and Float" at the first time_embed Linear.

Explicitly cast the control model to unet_dtype after load_state_dict (mirroring
the motion model load and AnimateDiff-Evolved #573) so the weights always match
the activation dtype at runtime.

Verified end-to-end with v3_sd15_sparsectrl_rgb.ckpt on an fp16 SD1.5 model.

Amp-Thread-ID: https://ampcode.com/threads/T-019e947a-9fd3-76df-a847-5eb68d7f18de
Co-authored-by: Amp <amp@ampcode.com>
 2026-06-04 15:10:48 -07:00
Jedrzej KosinskiandAmp 71b536c47f Fix dtype mismatch for SparseCtrl and SVD-ControlNet models 
load_sparsectrl and load_svdcontrolnet never set controlnet_config["dtype"],
so the control model was built with dtype=None (float32) while the UNet runs
in fp16. This caused "mat1 and mat2 must have the same dtype, but got Half and
Float" at sampling time (notably for .pth/.ckpt SparseCtrl models).

Set controlnet_config["dtype"] = unet_dtype before building the model, matching
the existing ControlNet++ and CtrLoRA loaders and ComfyUI's own controlnet loader.

Fixes #574

Amp-Thread-ID: https://ampcode.com/threads/T-019e947a-9fd3-76df-a847-5eb68d7f18de
Co-authored-by: Amp <amp@ampcode.com>
 2026-06-04 14:26:17 -07:00
Jedrzej Kosinski 79f8a8aa8b Fixed device mismatch issue for ControlNet++, SparseCtrl, and SVDControlNet 2025-08-06 12:59:25 -07:00
Jedrzej Kosinski 0da13ac0e2 Updated the remaining context= controlnet code to use comfy.model_management.cast_to_device calls 2025-08-05 13:43:52 -07:00
German Chaverri f980ae7c7b Apply fix for ControlNet conditioning to match changes from ComfyUI 140ffc7fdc53e810030f060e421c1f528c2d2ab9 2025-08-05 11:00:08 -06:00
Jedrzej Kosinski 3ad082b691 Refactored SparseCtrlAdvanced to not save a reference to the actual ModelPatcher, just in case it's messing with memory management 2024-12-01 20:35:32 -06:00
Jedrzej Kosinski 4618d8006f Just in case make sure cleanup is called on motion model for SparseCtrl 2024-12-01 20:06:38 -06:00
Jedrzej Kosinski 37b318debc Refacotr SparseCtrl to depend on AnimateDiff-Evolved for AnimateDiffModel definition/creation; allows for code to not be duplicated and any feature that works for AD can be resued/exposed for SparseCtrl 2024-12-01 19:46:47 -06:00
Jedrzej Kosinski c59f99efbc Add ControlNetSD35Advanced, port over changes from vanilla ControlNet to ControlNetAdvanced to support ControlNetSD35 2024-11-27 00:52:52 -06:00
Jedrzej Kosinski dc91f47aed Add middle_mult extra via Middle Weight Extras node 2024-11-26 20:21:53 -06:00
Jedrzej Kosinski 999763ce69 Add real_compression_ratio so can be adjusted to properly check for expected cond_hint size 2024-11-25 21:52:51 -06:00
Jedrzej Kosinski 72f0e6dbac Add initial CtrLoRA support, fix issue with vae not being set, add a way to disable multiplying compression_ratio by vae.downscale_ratio when not needed 2024-11-25 20:33:53 -06:00
Jedrzej Kosinski a2430fb880 Add clarifying comment so I don't accidentally cause regression in the future with new memory management system 2024-11-14 20:02:51 -06:00
Jedrzej Kosinski b9c53b79c9 Fix from_vanilla call for ControlNetAdvanced so it does not break ComfyUI memory management system; parent model patcher has to be tracked properly now 2024-11-14 11:49:56 -06:00
Jedrzej Kosinski 7f84e973d6 Converted sampler_sample_wrapper to a outer_sample_wrapper so that controlnet prerun can be ran properly 2024-11-14 10:53:59 -06:00
Jedrzej Kosinski 3120c14e0a Added checks for None control_net to produce more helpful error messages for users 2024-11-14 03:48:42 -06:00
Jedrzej Kosinski f42c902fd4 Fixed checks for special preprocessor input (ReferenceCN and RGB SparseCtrl), continued work on DinkLink 2024-11-14 03:42:29 -06:00
Jedrzej Kosinski 74320a78e3 Replaced SparseModelPatcher with native ModelPatcher and callbacks 2024-11-12 18:14:21 -06:00
Jedrzej Kosinski 767cbf3dcc Merge branch 'main' into rework-modelpatcher 2024-10-23 22:02:47 -05:00
Jedrzej Kosinski 95e26b9b79 Made code work with new ComfyUI changes (device var removed from ControlBase) 2024-10-23 21:57:39 -05:00
Kosinkadink f778bb2a70 Modified code to accept transformer_options to match ComfyUI's get_control change and removed need to a terribly ugly hack that was required before to get the cond_or_uncond; now can just be gotten from transformer_options 2024-09-25 20:50:20 +09:00
Jedrzej Kosinski 476da0e724 Implemented changes from ComfyUI's InstantX depth flux support commit 2024-08-30 07:09:26 -05:00
Jedrzej Kosinski a9e9ec7b92 Fix inconsistent variable name causing reference before assignment error 2024-08-30 03:28:01 -05:00
Jedrzej Kosinski d3ceadbaba Attempted to add masking support for flux controlnet (not working yet, but logic is partway there 2024-08-29 09:57:23 -05:00
Jedrzej Kosinski 49609d318f Refactored cn weights to separate out input, middle, and output blocks, improved soft weight behavior to generalize math to be architecture agnostic (for the most part) 2024-08-24 07:31:26 -05:00
Jedrzej Kosinski 1807cd64e4 Updated ControlNetAdvanced to work with Flux controlnet, beginning of rework of weights 2024-08-24 00:43:21 -05:00
Jedrzej Kosinski ee3490d4b0 Initial working implementation of ContextRef 2024-07-27 21:08:46 -05:00
Jedrzej Kosinski cdabf60fca Automatic conversion of vanilla ControlNets to Advanced-ControlNet equivalents when context_options are detected, extracting refcn's sample wrapper to its own function, refactoring of ControlLLLite code to work without a model_optional input 2024-07-19 18:17:24 -05:00
Jedrzej Kosinski 906ef7f2c6 Remove exception when ControlNet++ is detected and not using ControlNet++ nodes 2024-07-18 14:05:40 -05:00
Jedrzej Kosinski e7434e9fa0 fixed msg typo and unfinished comment 2024-07-15 02:47:18 -05:00
Jedrzej Kosinski 55da9bb2d1 Initial progress on initial support of ControlNet++ model 2024-07-08 20:09:20 -05:00
Jedrzej Kosinski 4454c47707 Added sd3 support from upcoming ComfyUI update 2024-06-27 18:23:10 -05:00
Jedrzej Kosinski 96d957e9ec Fixed get_calc_pow to work the same as before (only current difference would be with SDXL T2IAdapter models with custom/soft weights), automatically resize T2IAdapter control tensors to match batch_size to make my life easier 2024-06-27 17:43:21 -05:00
Jedrzej Kosinski 7e7fdf0b83 Also reverse T2IAdapter's get_calc_pow 2024-06-27 13:59:43 -05:00
Jedrzej Kosinski 4646bb3a2b Reverse T2IAdapter weights to account for ComfyUI changes (means SDXL T2IAdapter softweights have breaking changes due to extra middle weight) 2024-06-27 13:51:18 -05:00
Jedrzej Kosinski 59e7d1a7af Make require_vae something for me to work on in the future for SparseCtrl and ReferenceCN 2024-06-27 12:24:18 -05:00
Jedrzej Kosinski e800d52acf Made all controlnets works with upcoming ComfyUI changes, started work on making latent preprocs obsolete, still need to make control weights apply the same way as before 2024-06-27 05:03:10 -05:00
Jedrzej Kosinski e6085cc111 Change what I previously called latent_format to model_latent_format to avoid conflicts with upcoming ComfyUI changes 2024-06-27 00:46:55 -05:00
Jedrzej Kosinski 949e1acf7e Turned context_aware into a dropdown in case I add other context_aware types in future, renamed sparse_mask_strength to sparse_mask_mult 2024-05-29 01:23:21 -05:00
Jedrzej Kosinski 74b6cf576f Implemented context_aware 2024-05-28 22:32:19 -05:00
Jedrzej Kosinski 20d54c0660 Added support for weight extras, added sparse_hint_mult, sparse_nonhint_mult, and sparse_mask_strength for SparseCtrl, renamed Force Default Weights node to Default Weights 2024-05-27 18:49:28 -05:00
Jedrzej Kosinski 1f95fac964 Fixed odd behavior when context options are active and there are less condhints than latents, condhints/masks are extended (last image/mask) instead of repeated, starting to work on uncond_mask to extend uncond_multiplier feature, modified motion_strength > 1.0 behavior 2024-05-24 00:11:26 -05:00
Jedrzej Kosinski 5660a110df Made SparseCtrl work with fp8 and cleaner lowvram support 2024-05-23 18:12:28 -05:00
Jedrzej Kosinski 4ed14e9ae0 Make T2IAdapter work with uncond_multiplier (results are not great when < 1.0, but match up exactly with auto1111 results) 2024-05-16 17:40:46 -05:00
Jedrzej Kosinski db0bf14dac Add support for uncond_multiplier in all Control Weights (identical to auto1111's 'controlnet is more important when set to 0.0) 2024-05-16 04:15:36 -05:00
Jedrzej Kosinski 0556fd2265 Fixed SparseCtrl not working with lowvram 2024-04-04 14:35:28 -05:00
Jedrzej Kosinski 7696628fdd Simplified RefCN injection, model_opt no longer needed and should fix some intermittent bugs 2024-03-30 19:16:21 -05:00
Jedrzej Kosinski 5c3786e1a7 Merge branch 'main' into develop 2024-03-19 10:55:07 -05:00
Jedrzej Kosinski 6e05ca03fe Fixed ModelPatcher import in control.py 2024-03-19 10:40:44 -05:00