nodes/xmarre-ComfyUI-TIDE
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
xmarre
2026-05-01 17:31:58 +02:00
committed by GitHub
parent f15c200b65
commit a8df2d4360
1 changed files with 175 additions and 115 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+175 -115
View File
@@ -1,60 +1,132 @@
# ComfyUI-TIDE
ComfyUI-TIDE is a ComfyUI custom node implementation of **TIDE: Text-Informed Dynamic Extrapolation with Step-Aware Temperature Control for Diffusion Transformers**.
ComfyUI-TIDE is a ComfyUI custom node for applying the core inference-time mechanisms from **TIDE: Text-Informed Dynamic Extrapolation with Step-Aware Temperature Control for Diffusion Transformers** to Flux-style DiT attention in ComfyUI.
The node is intended for FLUX-family DiT models in ComfyUI, including FLUX.2-style model paths that use the same joint text/image attention structure as ComfyUI's FLUX implementation. It patches the `MODEL` object and does not add sampling steps.
The node patches a cloned ComfyUI `MODEL` object. It does **not** add extra sampling steps, replace the sampler, replace the scheduler, or fork ComfyUI core.
## Credits and attribution
The algorithmic method implemented here is based on the TIDE paper:
**TIDE: Text-Informed Dynamic Extrapolation with Step-Aware Temperature Control for Diffusion Transformers**
Yihua Liu, Fanjiang Ye, Bowen Lin, Rongyu Fang, Chengming Zhang
arXiv:2603.08928, 2026
<https://arxiv.org/abs/2603.08928>
Credit for the TIDE method, including **Text Anchoring**, **Dynamic Temperature Control**, and the paper's analysis of attention dilution in high-resolution Diffusion Transformer generation, belongs to the paper authors.
This repository is an independent ComfyUI custom-node implementation. It is not the official TIDE repository, not affiliated with the paper authors, and should not be cited as the original method. If this node is useful in your work, cite the TIDE paper.
```bibtex
@misc{liu2026tide,
title = {TIDE: Text-Informed Dynamic Extrapolation with Step-Aware Temperature Control for Diffusion Transformers},
author = {Yihua Liu and Fanjiang Ye and Bowen Lin and Rongyu Fang and Chengming Zhang},
year = {2026},
eprint = {2603.08928},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2603.08928}
}
```
## Scope
This repository implements the practical ComfyUI attention-side parts of TIDE for Flux-style joint text/image attention:
- text-token additive bias for Text Anchoring;
- step-aware RoPE temperature scaling for Dynamic Temperature Control;
- a lightweight ComfyUI model wrapper to pass the current denoising timestep into the attention patch;
- a small PyTorch SDPA fallback used only when an additive TIDE attention mask is active.
It does **not** implement the full official Diffusers pipeline, benchmark harness, datasets, metric evaluation scripts, Qwen pipeline, or complete YaRN/DyPE/NTK positional interpolation stack.
## Supported model path
The target path is FLUX-family DiT models in ComfyUI, including FLUX.2-style paths if they use the same practical structure as ComfyUI's Flux implementation:
- joint text/image attention;
- text tokens before image tokens;
- `attn1_patch` support;
- `extra_options["img_slice"]` available at the attention patch site;
- RoPE matrix passed as `pe`.
Other DiT models may require model-specific patch paths. They should not be assumed to work unless their ComfyUI implementation exposes the same attention-patch contract.
## What is implemented
TIDE has two core inference-time mechanisms:
### 1. Text Anchoring
1. **Text Anchoring**
- Adds a positive additive bias to attention logits whose keys are text tokens.
- The default adaptive bias is:
TIDE identifies text-token influence decay as a core failure mode at high resolution: image token count grows with resolution while text token count stays fixed. Text Anchoring counteracts this by adding a positive bias to attention logits whose keys are text tokens.
```text
beta = log((target_width * target_height) / (base_width * base_height))
```
This node computes the default adaptive bias as:
- With the default base resolution of 1024x1024, this matches the official FLUX script's `log(width / 1024) + log(height / 1024)`.
```text
beta = log((target_width * target_height) / (base_width * base_height))
```
2. **Dynamic Temperature Control**
- Applies the paper/official-code YaRN temperature concept as a per-frequency multiplier on ComfyUI's RoPE matrix.
- The default curve follows the released implementation's `dyheating` form:
With the default `base_width=1024` and `base_height=1024`, this is equivalent to:
```text
tau(t, f) = tau_max - (tau_max - tau_min) * t ** alpha(f)
alpha(f) = alpha_low + (alpha_high - alpha_low) * f
```
```text
log(width / 1024) + log(height / 1024)
```
- Defaults: `tau_max=1.0`, `alpha_low=0.6`, `alpha_high=0.2`.
- `frequency_mode=official_raw` uses raw RoPE frequencies, matching the released code. `paper_normalized` is available for comparison because the paper text describes a normalized frequency variable.
The final applied value is:
## Implementation plan
```text
applied_beta = text_anchor_strength * beta
```
1. Patch ComfyUI attention rather than cloning a full model implementation.
2. Use ComfyUI's existing `attn1_patch` hook to modify joint DiT attention inputs.
3. Use a model function wrapper to expose the current denoising timestep to the attention patch.
4. Keep TIDE state local to the cloned `MODEL` object.
5. Avoid global monkey-patching and avoid changing the scheduler or sampler.
6. Force a small PyTorch SDPA attention path only when an additive TIDE mask is present, preventing backends from rejecting or materializing a dense high-resolution mask.
By default, the node applies no Text Anchoring at native-or-smaller token counts unless `apply_to_native_or_smaller=True`.
## Paper / official-code / local-module mapping
### 2. Dynamic Temperature Control
| Paper component | Official implementation component | This repository |
TIDE uses a step-aware temperature curve so attention sharpening is stronger in the early/global part of denoising and relaxes toward the late/detail part of denoising. This repository applies that idea as a RoPE temperature multiplier, matching the reference implementation strategy rather than inserting a new attention kernel for every backend.
Default curve:
```text
tau(t, f) = tau_max - (tau_max - tau_min) * t ** alpha(f)
alpha(f) = alpha_low + (alpha_high - alpha_low) * f
```
Defaults:
| Parameter | Default |
|---|---:|
| `tau_max` | `1.0` |
| `alpha_low` | `0.6` |
| `alpha_high` | `0.2` |
| `temperature_strength` | `1.0` |
| `frequency_mode` | `official_raw` |
`frequency_mode=official_raw` uses raw RoPE frequencies, matching the released implementation behavior this port was written against. `paper_normalized` is exposed for comparison because the paper notation describes a normalized frequency variable.
## Implementation notes
1. The node clones and patches the incoming ComfyUI `MODEL`.
2. TIDE state is stored on the cloned model through ComfyUI model options.
3. `TIDEModelWrapper` injects current timestep metadata into `transformer_options["tide"]`.
4. `TIDEAttentionPatch` applies Text Anchoring and Dynamic Temperature Control through ComfyUI's `attn1_patch` hook.
5. `TIDEAttentionOverride` forces a compact PyTorch SDPA path only when an additive TIDE mask is active. This avoids attention backends that reject additive masks or try to materialize a dense high-resolution mask.
6. No global monkey-patching is used.
7. No sampler or scheduler rewrite is performed.
## Paper / reference-code / local-module mapping
| Paper component | Reference implementation behavior | This repository |
|---|---|---|
| Text-token influence decay analysis | `run.py::get_attn_mask()` creates an additive text-token mask | `tide_core.math.adaptive_text_bias`, `tide_core.patches.TIDEAttentionPatch` |
| Text Anchoring, beta = log(lambda) | `log(width / 1024) + log(height / 1024)` for the first 512 text tokens | Adaptive beta computed from node `width`, `height`, `base_width`, `base_height`; text-token count is inferred from ComfyUI `img_slice` instead of hard-coded to 512 |
| YaRN temperature baseline | `get_mscale`, `get_default_temperature` | `tide_core.math.get_mscale`, `get_default_temperature` |
| Dynamic Temperature Control | `dyheating()` and `tuning_temperature()` multiply RoPE cos/sin by `1/sqrt(tau)` | `tide_core.math.rope_temperature_scale`, applied to ComfyUI's RoPE matrix in `TIDEAttentionPatch` |
| Denoising-step-aware RoPE | Official `FluxPosEmbed.update_timestep(timestep.item())` | `TIDEModelWrapper` injects normalized current timestep into `transformer_options["tide"]` |
| FLUX attention integration | Official fork modifies Diffusers `FluxTransformer2DModel` / processor | ComfyUI `attn1_patch` and `optimized_attention_override`; no fork of ComfyUI core |
| Logarithmic FLUX scheduler shift | Official `FluxPipeline(..., shift_mode="log")` | Not implemented as a node-level sampler change; see limitations |
| DyPE / NTK-by-parts / YaRN positional interpolation | Official fork implements custom RoPE interpolation | Not fully implemented; this node implements TIDE's attention-side mechanisms and optional frequency interpretation only |
| Text-token influence decay | Additive text-token attention mask | `tide_core.math.adaptive_text_bias`, `tide_core.patches.TIDEAttentionPatch` |
| Text Anchoring, `beta = log(lambda)` | `log(width / 1024) + log(height / 1024)` for FLUX-sized text prefix | Adaptive beta from node `width`, `height`, `base_width`, `base_height`; text-token count inferred from ComfyUI `img_slice` |
| YaRN temperature baseline | `get_mscale`, default temperature from extrapolation scale | `tide_core.math.get_mscale`, `get_default_temperature` |
| Dynamic Temperature Control | `dyheating()` / temperature-aware RoPE scaling | `tide_core.math.rope_temperature_scale`, applied to ComfyUI `pe` |
| Denoising-step-aware behavior | Update position embedding state from current timestep | `TIDEModelWrapper` injects normalized timestep into `transformer_options` |
| FLUX attention integration | Modified Diffusers FLUX transformer/processor | ComfyUI `attn1_patch` plus optional `optimized_attention_override` |
| Logarithmic FLUX scheduler shift | Scheduler/pipeline-level change | Not implemented by this node |
| DyPE / NTK-by-parts / YaRN positional interpolation | Custom positional interpolation stack | Not fully implemented; this node implements the TIDE attention-side mechanisms only |
## Installation
Clone or copy this repository into ComfyUI's custom node directory:
Clone this repository into ComfyUI's custom node directory:
```bash
cd ComfyUI/custom_nodes
@@ -63,56 +135,63 @@ git clone <this-repo-url> ComfyUI-TIDE
Restart ComfyUI.
No extra Python dependency is required beyond the PyTorch/ComfyUI environment. `requirements.txt` lists `torch` only for standalone unit tests.
No extra runtime dependency is required beyond the PyTorch/ComfyUI environment. `requirements.txt` lists `torch` for standalone tests.
## Usage
1. Add **TIDE High-Resolution Extrapolation** after your model loader.
2. Connect its `model` output to the sampler model input.
3. Set `width` and `height` to the final generation size used by your latent node.
4. Use a FLUX-family DiT model.
1. Load a FLUX-family model as usual.
2. Add **TIDE High-Resolution Extrapolation** after the model loader.
3. Connect the patched `model` output to your sampler.
4. Set `width` and `height` to the final generated image dimensions used by your latent node.
5. Keep `base_width=1024` and `base_height=1024` for FLUX-family models unless you know the model's native training target differs.
Recommended starting values:
| Setting | Value |
|---|---:|
| `width` / `height` | final latent/image size |
| `base_width` / `base_height` | `1024` / `1024` for FLUX-family models |
| `width` / `height` | final image dimensions |
| `base_width` / `base_height` | `1024` / `1024` |
| `text_anchor_strength` | `1.0` |
| `temperature_strength` | `1.0` |
| `alpha_low` | `0.6` |
| `alpha_high` | `0.2` |
| `tau_max` | `1.0` |
| `frequency_mode` | `official_raw` |
| `force_pytorch_attention_with_mask` | `True` |
For ablations:
Ablation settings:
- Text Anchoring only: `temperature_strength=0.0`, `text_anchor_strength=1.0`
- Dynamic Temperature only: `text_anchor_strength=0.0`, `temperature_strength=1.0`
- Disable the node behavior without removing it: set both strengths to `0.0`
| Test | Settings |
|---|---|
| Text Anchoring only | `text_anchor_strength=1.0`, `temperature_strength=0.0` |
| Dynamic Temperature only | `text_anchor_strength=0.0`, `temperature_strength=1.0` |
| Disabled | `text_anchor_strength=0.0`, `temperature_strength=0.0` |
## Node inputs
### Required
- `model`: ComfyUI MODEL to patch.
- `width`, `height`: final target generation dimensions in pixels.
- `text_anchor_strength`: multiplier on beta. `1.0` follows paper/official behavior.
- `temperature_strength`: multiplier on the RoPE temperature scale. `1.0` follows official behavior; `0.0` disables DTC.
| Input | Description |
|---|---|
| `model` | ComfyUI `MODEL` object to patch. |
| `width`, `height` | Final target generation dimensions in pixels. Must match the latent/image size used by the workflow. |
| `text_anchor_strength` | Multiplier on adaptive beta. `1.0` follows the paper/reference behavior. `0.0` disables Text Anchoring. |
| `temperature_strength` | Multiplier on Dynamic Temperature Control. `1.0` follows the reference curve. `0.0` disables DTC. |
### Optional
- `base_width`, `base_height`: training/native resolution used for adaptive scaling. Default is 1024x1024.
- `alpha_low`, `alpha_high`: Dynamic Temperature Control exponents.
- `tau_max`: maximum temperature. Default 1.0.
- `frequency_mode`:
- `official_raw`: matches the released code's raw RoPE-frequency use.
- `paper_normalized`: normalizes frequency to [0, 1] as a literal reading of the paper notation.
- `apply_to_double_blocks`, `apply_to_single_blocks`: choose which FLUX block types receive the patch.
- `apply_to_native_or_smaller`: default false. Prevents non-positive or native-resolution anchoring from changing normal-resolution behavior.
- `force_pytorch_attention_with_mask`: default true. Uses an internal PyTorch SDPA path only when the TIDE additive mask is active.
- `preserve_existing_wrapper`: default true. Delegates to an existing model wrapper after injecting TIDE timestep metadata.
- `debug`: logs skipped dynamic-temperature shape mismatches and exceptions.
| Input | Default | Description |
|---|---:|---|
| `base_width`, `base_height` | `1024`, `1024` | Native/training resolution used for adaptive scaling. |
| `alpha_low`, `alpha_high` | `0.6`, `0.2` | DTC exponents for low/high RoPE frequency behavior. |
| `tau_max` | `1.0` | Maximum temperature reached near the end of denoising. |
| `frequency_mode` | `official_raw` | `official_raw` or `paper_normalized`. |
| `apply_to_double_blocks` | `True` | Apply patch to FLUX double-stream blocks. |
| `apply_to_single_blocks` | `True` | Apply patch to FLUX single-stream blocks. |
| `apply_to_native_or_smaller` | `False` | Allow patching even when target token count is not above base token count. |
| `force_pytorch_attention_with_mask` | `True` | Use internal PyTorch SDPA only when TIDE's additive mask is active. |
| `preserve_existing_wrapper` | `True` | Delegate to an existing ComfyUI model wrapper after injecting TIDE metadata. |
| `debug` | `False` | Log skipped DTC shape mismatches and exceptions. |
## Repository structure
@@ -136,85 +215,66 @@ ComfyUI-TIDE/
## Tests
Standalone math/patch tests:
Run the standalone tests from the repository root:
```bash
cd ComfyUI-TIDE
python -m pip install pytest torch
python -m pytest -q
```
These tests validate:
The tests cover:
- adaptive beta computation,
- YaRN temperature formula,
- RoPE scale shape and timestep progression,
- attention-mask creation,
- adaptive Text Anchoring beta computation;
- YaRN/default temperature formula;
- RoPE temperature scale shape and timestep progression;
- additive attention-mask creation;
- masked SDPA override behavior.
They do not validate visual quality or live ComfyUI model execution.
The tests do not validate visual quality or live ComfyUI execution.
## Paper vs official code mismatches and resolutions
## Paper vs implementation differences
### 1. Frequency variable in Dynamic Temperature Control
### Scheduler time shifting
- Paper: describes `f` as frequency normalized into a range used by `alpha(f)`.
- Official code: multiplies `(alpha_high - alpha_low)` by raw RoPE frequencies.
- Resolution: default `frequency_mode=official_raw` to match official behavior; `paper_normalized` is exposed for controlled comparison.
The paper appendix describes a logarithmic FLUX time-shift schedule for high resolutions. This node receives an already-built ComfyUI sampler schedule and does not silently rewrite it. Use a scheduler setup that does not over-shift high-resolution FLUX timesteps.
### 2. Dynamic Temperature implementation site
### Positional interpolation
- Paper: formulates the final attention with a temperature term in the softmax denominator.
- Official FLUX code: implements temperature by multiplying RoPE cos/sin by `1/sqrt(tau)` inside YaRN RoPE generation.
- Resolution: ComfyUI exposes RoPE matrices through `pe`; this repo applies the official-code equivalent multiplier to `pe`.
The paper evaluates TIDE in combination with positional extrapolation/interpolation methods such as YaRN/DyPE-style handling. This repository does not port the full positional interpolation stack. It applies the TIDE attention-side mechanisms through ComfyUI's patch system.
### 3. Text token count
### Frequency variable
- Paper: text-token length is abstract `L_T`.
- Official FLUX script: hard-codes 512 text tokens for FLUX.1.
- Resolution: this repo infers text-token count from ComfyUI's `img_slice`, avoiding hard-coding 512 and making FLUX-family variants more likely to work.
The paper notation describes a normalized frequency variable for `alpha(f)`. The reference implementation behavior used by this port applies the curve using raw RoPE frequencies. This mismatch is exposed as `frequency_mode` so the default can follow the reference behavior while still allowing controlled comparison.
### 4. Scheduler time shifting
### Text-token count
- Paper appendix and official pipeline use a logarithmic FLUX time-shift schedule for high resolutions.
- This custom node receives an already-built sampler schedule and should not silently alter the user's sampler.
- Resolution: no sampler schedule rewrite is performed. This is a deliberate deviation. Use a ComfyUI sampler/scheduler setup that does not over-shift high-resolution FLUX timesteps.
The paper writes the text-token length abstractly as `L_T`. Some FLUX implementations use a fixed text-token prefix length. This repository infers the text prefix from ComfyUI's `img_slice` rather than hard-coding a token count.
### 5. Positional interpolation / DyPE / YaRN
### General DiT support
- Paper experiments combine TIDE with YaRN/DyPE-style positional handling.
- Official code includes a Diffusers FLUX fork with NTK, NTK-by-parts, YaRN, and DyPE RoPE logic.
- Resolution: this repo implements TIDE's attention-side contribution in ComfyUI and does not clone the full official Diffusers transformer. This avoids replacing ComfyUI internals but means it is not a complete official YaRN/DyPE port.
### 6. Qwen/general DiT support
- Official code includes Qwen-Image support.
- ComfyUI Qwen's current patch path does not propagate a patch-returned additive attention mask to the final attention call in the same way as FLUX.
- Resolution: this repo is implemented for Flux-style ComfyUI joint attention first. Other DiTs may work only if their ComfyUI implementation exposes compatible `attn1_patch`, `img_slice`, and additive-mask propagation.
The method is architecture-relevant to DiTs, but this implementation is tied to ComfyUI's Flux-style patch interface. Non-Flux DiTs need compatible attention hooks and token-layout metadata.
## Assumptions
- The model uses ComfyUI's Flux-style joint attention with text tokens before image tokens.
- ComfyUI provides `extra_options["img_slice"]` in attention patches.
- `width` and `height` passed to this node match the actual generated latent/image dimensions.
- The timestep seen by the wrapper is already normalized or sigma-like in [0, 1]. Values outside the interval are clamped.
- FLUX-family token granularity is 16 image pixels per transformer token.
- The model uses Flux-style joint attention with text tokens before image tokens.
- `extra_options["img_slice"]` identifies the split between text and image tokens.
- The ComfyUI attention patch receives a RoPE matrix through `pe`.
- The node `width` and `height` match the actual generated dimensions.
- The timestep passed through the model wrapper is normalized or sigma-like in `[0, 1]`; values outside the interval are clamped.
- FLUX-family image token granularity is 16 pixels per transformer token.
## Limitations
- Not a full official repository clone.
- Does not implement official Diffusers pipeline scripts, benchmark code, metric evaluation, datasets, or training code.
- Not an official TIDE release.
- Not a full reproduction of the paper's complete experimental pipeline.
- Does not include paper benchmark scripts, datasets, generated result images, or metric evaluation.
- Does not modify the sampler's high-resolution time-shift schedule.
- Does not implement full NTK-by-parts, YaRN positional interpolation, or DyPE positional interpolation.
- Visual quality is unverified in this static repository export.
- Very large resolutions still require enough VRAM for the chosen model, sampler, attention backend, and VAE path.
## Unresolved uncertainties
- Exact FLUX.2 internal token layout may differ from FLUX.1 depending on the ComfyUI model implementation. If it still uses Flux-style `img_slice` with text tokens first, this patch should apply.
- Whether ComfyUI's current default scheduler for FLUX.2 already avoids the extreme high-resolution time-shift problem described in the paper needs live workflow verification.
- The paper notation and official code differ on frequency normalization; defaulting to official code is the safest reproducibility choice, but it is still a documented mismatch.
- Does not fully implement NTK-by-parts, YaRN positional interpolation, or DyPE positional interpolation.
- Visual quality still requires live workflow testing.
- Very large resolutions still require sufficient VRAM for the selected model, sampler, attention path, latent size, and VAE path.
## License
This repository is an implementation scaffold for ComfyUI. ComfyUI itself is GPL-licensed. Review license compatibility before redistributing as part of a larger package.
This repository is released under the MIT License; see [`LICENSE`](LICENSE).
ComfyUI is GPL-3.0 licensed. This custom node is distributed as a separate plugin, but it imports and runs inside ComfyUI. Review license compatibility before redistributing this node as part of a larger bundled package.