xmarre
d29ab5e65e
Updated README for clarity and consistency in instructions regarding model loading and AutoGuidance parameters.
243 lines
8.7 KiB
Markdown
243 lines
8.7 KiB
Markdown
# ComfyUI-AutoGuidance
|
||
|
||
Adds an **AutoGuidance + CFG** **GUIDER** node for **ComfyUI**, based on:
|
||
|
||
- **Guiding a Diffusion Model with a Bad Version of Itself** (arXiv:2406.02507)
|
||
https://arxiv.org/abs/2406.02507
|
||
|
||
This node lets you run normal CFG on a “good” model while also using a “bad” model to compute an additional **AutoGuidance delta** that is added on top of the CFG output (optionally capped and ramped over steps).
|
||
|
||
You provide:
|
||
- a **good_model** (what you actually want),
|
||
- a **bad_model** (intentionally different/worse),
|
||
- and this guider mixes them according to the selected **delta mode**, **cap**, and **ramp**.
|
||
|
||
---
|
||
|
||
## Install
|
||
|
||
Clone into:
|
||
|
||
`ComfyUI/custom_nodes/ComfyUI-AutoGuidance`
|
||
|
||
Restart ComfyUI.
|
||
|
||
---
|
||
|
||
## Node
|
||
|
||
- **AutoGuidance CFG Guider (good+bad)** → outputs a **GUIDER**
|
||
Plug it into **SamplerCustomAdvanced** (GUIDER input).
|
||
|
||
---
|
||
|
||
## Quick start (how to wire it)
|
||
|
||
1) Load your two models:
|
||
- `good_model`: checkpoint + your normal LoRA stack
|
||
- `bad_model`: intentionally different/worse setup (examples below)
|
||
|
||
2) Add:
|
||
- **AutoGuidance CFG Guider (good+bad)**
|
||
|
||
3) Connect:
|
||
- its **GUIDER** output → **SamplerCustomAdvanced** → **GUIDER**
|
||
|
||
That’s it.
|
||
|
||
---
|
||
|
||
## Extremely important: `dual_models_2x_vram` requires a second checkpoint file path
|
||
|
||
If you pick **`swap_mode = dual_models_2x_vram`**, you must ensure ComfyUI loads **two distinct model instances**.
|
||
|
||
If both loaders point to the same checkpoint path, ComfyUI’s caching can reuse the same underlying model object. In that situation, `dual_models_2x_vram` cannot behave as intended.
|
||
|
||
### Do this (reliable)
|
||
|
||
Make a second checkpoint file path and load each one:
|
||
|
||
- Copy the checkpoint file (simple + reliable), e.g.
|
||
`model.safetensors` → `model_copy.safetensors`
|
||
|
||
Then in ComfyUI:
|
||
- `CheckpointLoaderSimple` → load `model.safetensors` (good path)
|
||
- `CheckpointLoaderSimple` → load `model_copy.safetensors` (bad path)
|
||
|
||
**VRAM note:** dual-model mode uses ~2× the base checkpoint VRAM (plus LoRAs, plus activations).
|
||
|
||
---
|
||
|
||
## Choosing a useful “bad_model”
|
||
|
||
AutoGuidance only has leverage if **good_model and bad_model produce meaningfully different denoiser predictions**. If they’re too similar, the effect will be subtle regardless of ramp settings.
|
||
|
||
Ways to make `bad_model` meaningfully different (examples, not promises):
|
||
- Remove some LoRAs on the bad path (good has them, bad doesn’t)
|
||
- Use a different LoRA stack on the bad path (style/character/etc.)
|
||
- Use the same LoRA at a deliberately different weight (including negative if your workflow supports it)
|
||
- Change conditioning strategy on the bad path (e.g. different prompt/conditioning pipeline)
|
||
|
||
The point isn’t “worse” aesthetically—it’s **different** in a way that changes predictions.
|
||
|
||
---
|
||
|
||
## Node parameters
|
||
|
||
### Core knobs
|
||
|
||
#### `cfg`
|
||
Normal CFG scale.
|
||
|
||
#### `w_autoguide`
|
||
Paper-style parameterization:
|
||
- `w_autoguide = 1.0` → AutoGuidance delta effectively off
|
||
- `w_autoguide > 1.0` → strength increases with `(w_autoguide - 1)`
|
||
|
||
Internally this is used as a multiplier for the AutoGuidance delta direction (added on top of CFG).
|
||
|
||
---
|
||
|
||
## Swap modes (performance + correctness)
|
||
|
||
### `shared_safe_low_vram` (default)
|
||
- Lowest VRAM overhead
|
||
- Most compatibility / correctness across Comfy variants
|
||
- Can be **very slow** because it must swap LoRA stacks on a shared model safely
|
||
|
||
### `shared_fast_extra_vram`
|
||
- Faster swapping than safe mode (uses inplace updates / device-friendly behavior)
|
||
- Still shares one model object, so there’s still unavoidable overhead
|
||
- Uses extra VRAM
|
||
|
||
### `dual_models_2x_vram`
|
||
- Fastest **when you truly load two separate checkpoint instances** (see the “second checkpoint path” section above)
|
||
- Uses ~2× checkpoint VRAM
|
||
- Avoids the expensive shared swapping path entirely
|
||
|
||
---
|
||
|
||
## AutoGuidance delta mode (`ag_delta_mode`)
|
||
|
||
### `bad_conditional` (recommended default)
|
||
Uses the most “LoRA-sensitive” direction in practice:
|
||
- compare **good conditional output** vs **bad conditional output**
|
||
|
||
This tends to show the biggest differences when you change LoRAs on the bad path.
|
||
|
||
### `raw_delta`
|
||
Uses a raw difference direction between guided outputs. Can be harsher / less predictable.
|
||
|
||
### `project_cfg`
|
||
Projects the “push-away-from-bad” direction onto the **actual CFG update direction**.
|
||
This keeps changes more aligned with CFG, often more conservative.
|
||
|
||
### `reject_cfg`
|
||
Removes the component of the “push-away-from-bad” direction that is parallel to the **actual CFG update direction**.
|
||
This can increase composition changes when AG otherwise behaves like “CFG++”.
|
||
|
||
---
|
||
|
||
## Safety cap: `ag_max_ratio`
|
||
|
||
AutoGuidance is capped relative to the magnitude of the **actual CFG update** (`cfg_out - uncond_good`):
|
||
|
||
- Higher `ag_max_ratio` = stronger visible effect (up to destabilization if extreme)
|
||
- Lower `ag_max_ratio` = subtler
|
||
|
||
If your output looks “basically like normal CFG”, this cap (and/or the ramp) is the first place to inspect.
|
||
|
||
> This repo includes `debug_metrics` specifically so you can see whether you are being clamped (look at `scale` and `n_delta_applied`).
|
||
|
||
---
|
||
|
||
## Ramp over steps (this controls “composition vs detail”)
|
||
|
||
These parameters scale **the cap** (`ag_max_ratio`) over denoise progress.
|
||
|
||
Important concept:
|
||
- Progress is defined as **0 early (high sigma)** → **1 late (low sigma)**
|
||
- In this implementation, progress is tracked by step index: `prog = step / (total_steps - 1)`
|
||
|
||
### `ag_ramp_mode`
|
||
- `flat`
|
||
Constant cap across all steps.
|
||
**Use this if you want composition changes.**
|
||
- `detail_late`
|
||
Weak early, strong late → mostly affects fine detail.
|
||
If `ag_ramp_floor = 0`, early steps can get near-zero AG and the image composition can stay close to normal CFG.
|
||
- `compose_early`
|
||
Strong early, weaker late → pushes structure/composition more than detail.
|
||
- `mid_peak`
|
||
Strongest mid-steps, weaker at ends.
|
||
|
||
### `ag_ramp_power`
|
||
Controls steepness of the selected ramp curve.
|
||
|
||
### `ag_ramp_floor`
|
||
Minimum always-on fraction of the cap.
|
||
- `0.0` means the ramp can reduce AG close to zero in parts of the schedule.
|
||
- If you use `detail_late` but still want some early influence, raise `ag_ramp_floor`.
|
||
|
||
> This README intentionally does not give “magic” preset numbers. Use `debug_metrics` to confirm what’s actually being applied in your workflow.
|
||
|
||
---
|
||
|
||
## Advanced / safety toggles
|
||
|
||
### `safe_force_clean_swap`
|
||
Only relevant in shared-model modes. Forces clean swaps between patch stacks to avoid state leakage / washed-out results on some Comfy builds.
|
||
- Safer
|
||
- Slower
|
||
|
||
### `uuid_only_noop`
|
||
Debug option. Treat “same patches_uuid” as a no-op even if ownership tracking is imperfect.
|
||
Use only for debugging.
|
||
|
||
### `debug_swap`
|
||
Prints swap/patch diagnostics (patch counts, uuids, etc.).
|
||
|
||
### `debug_metrics`
|
||
Prints direction / magnitude diagnostics (useful for confirming AG is actually being applied).
|
||
It also logs active `sampler_post_cfg_function` hooks at step 0, which helps diagnose post-CFG rescale/clamp behavior that can suppress visible AG effects.
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
### “It looks almost the same as normal CFG”
|
||
Common causes:
|
||
|
||
1) **Your ramp/cap is effectively zero early**, so composition won’t move.
|
||
- If you used `detail_late` with `ag_ramp_floor = 0`, early AG can be near-zero by design.
|
||
- Fix: use `flat` or `compose_early`, or raise `ag_ramp_floor`.
|
||
|
||
2) **Bad model is not actually “bad enough”**
|
||
- Make the bad path deliberately different for one test (remove key LoRAs, use an intentionally different stack, etc.).
|
||
- Confirm with `debug_metrics` that `n_good_minus_bad_cond` is meaningfully non-zero.
|
||
|
||
3) **You’re not truly in dual-model mode**
|
||
- If you selected `dual_models_2x_vram` but did not load the checkpoint via two different file paths, you may not actually have two model instances.
|
||
|
||
4) **Post-CFG hooks are suppressing the visible effect**
|
||
- `debug_metrics` prints active `sampler_post_cfg_function` hooks at step 0.
|
||
- Some workflows rescale/normalize/clamp after CFG, which can reduce the visible impact of AG.
|
||
|
||
### “dual_models_2x_vram is insanely slow”
|
||
That can happen if both loaders resolve to the same underlying model instance (same checkpoint path / caching). Fix it by creating a second checkpoint file path and loading both.
|
||
|
||
### “My LoRA changes on the bad path do nothing”
|
||
Check `debug_swap` / `patch_info`:
|
||
- If patch counts change when you edit bad LoRAs, the patches are applied.
|
||
- If the image still doesn’t move, inspect `debug_metrics`:
|
||
- are you being heavily clamped (`scale` near 0)?
|
||
- is `n_good_minus_bad_cond` meaningfully > 0?
|
||
- do you have post-CFG hooks that might normalize the change?
|
||
|
||
---
|
||
|
||
## Credits
|
||
|
||
- Paper: https://arxiv.org/abs/2406.02507
|
||
- This repository implements an AutoGuidance-style guider node for ComfyUI’s custom sampler pipeline.
|