nodes/malkuthro-ComfyUI-Koolook
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

refactor: per-task automation modules + WhatDreamsCost Koolook fork

Browse Source 
Pivot docs/automations/ from "one folder per model" to "one folder per
automation module" so the same diffusion model can host several
independent tasks side by side. Replace the parallel docs/investigations/
tree with a fork of WhatDreamsCost-ComfyUI under forks/.

Automation modules (new layout)
- docs/automations/LTX-2.3/base-1step/         the existing single-stage
                                                automation, moved from
                                                docs/automations/LTX-2.3/
                                                with its README, findings,
                                                and handoff-checklist
                                                rescoped to the task. The
                                                4k-transitions backstory
                                                that produced the module
                                                moves here as
                                                backstory/4k-transitions.md.
                                                The earlier
                                                docs/investigations/experiments/
                                                card PNGs/HTML land in
                                                example-cards/.
- docs/automations/LTX-2.3/audio-lipsync/      the audio-file lip-sync
                                                module previously living
                                                under docs/investigations/01_…/.
                                                README, handoff, findings
                                                stub, backstory/, and
                                                runs/{LOOP,log}.md, all
                                                referencing the new fork.
- docs/automations/README.md + CHEATSHEET.md   updated to describe the
                                                module-per-task pivot and
                                                the new tree shape.
- docs/automations/CONVENTIONS.md               unchanged (cross-cutting
                                                loop contract still holds).

WhatDreamsCost Koolook fork (new)
- forks/whatdreamscost_koolook/versions/v1_3_2/
    UPSTREAM_PIN.yaml                          pinned to upstream e81223a
                                                (v1.3.2), GPL-3.0
                                                verified 2026-05-28.
    ltx_director.py                            MODIFIED — adds the
                                                relay_overrides multiline
                                                JSON widget on LTXDirector.
                                                Moved from
                                                docs/investigations/01_…/patches/.
    prompt_relay.py                            MODIFIED — per-segment σ per
                                                the Prompt-Relay paper
                                                formula σ = (L − w_eff) /
                                                (2 · √ln(1/ε)). Moved from
                                                docs/investigations/01_…/patches/.
    patches.py                                 UNMODIFIED — vendored
                                                verbatim from upstream
                                                because ltx_director.py
                                                imports detect_model_type
                                                and apply_patches from it
                                                (same precedent as
                                                radiance_koolook's
                                                color_helpers.py).
- Registered as LTXDirector__koolook_v1_3_2 ("LTX Director (Koolook v1.3.2)")
  so it coexists with an installed copy of upstream WhatDreamsCost-ComfyUI.
  Wired into the root __init__.py NODE_CLASS_MAPPINGS merge.
- forks/forks_manifest.yaml + forks/THIRD_PARTY.md updated with the new
  entry (attribution, change log, license verified date, exposed node IDs).

Removed
- docs/investigations/                          entire tree retired; its
                                                contents either moved to
                                                automation modules or to
                                                forks/.
- scripts/sync_investigation_patches.py         no longer needed — fork
                                                code lives in the Koolook
                                                tree and is synced via the
                                                standard dev-sync flow.
- KOLOOK_WHATDREAMSCOST_PATH in .env.example    same reason.
- docs/investigations in pyproject.toml's ruff
  extend-exclude                                added in 96bf8ad to silence
                                                lint on vendored patches;
                                                no longer needed now that
                                                vendored code is under
                                                forks/ (already excluded).

Docs
- docs/reference/glossary.md                    Investigation folder,
                                                Investigation patches,
                                                JSON file folder, and the
                                                sync_investigation_patches.py
                                                entries replaced by
                                                Automation module, Backstory,
                                                Working folder, and Card.
                                                Loop and Run rewritten to
                                                point at the new structure.
- CHANGELOG.md [Unreleased]                     describes the restructure +
                                                the new fork.

Verification
- ruff check . → all checks passed
- tools/preflight_release.py --skip manager-meta → 42 files parsed; 11
  unique node IDs collected including LTXDirector; VAE dispatch branches
  all OK; fixture workflows still reference all registered Koolook IDs.
- pytest → 173 passed, 0 failed.

Workflow JSON heads-up
The active audio-lipsync workflow LTX-23-audio_tests_v01.json continues to
reference the upstream LTXDirector ID. To pick up the new fork's behaviour,
swap that node to "LTX Director (Koolook v1.3.2)" (registered ID
LTXDirector__koolook_v1_3_2) — sockets and wiring are preserved. The
relay_overrides widget appears at the bottom of the new node. Documented
in the audio-lipsync README's "Workflow JSON — Koolook node ID" section.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
malkuthro
2026-05-28 21:45:16 +02:00
co-authored by Claude Opus 4.7
parent 96bf8ad515
commit e7f3ab4fe8
37 changed files with 1015 additions and 499 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
-14
View File
@@ -106,17 +106,3 @@ KOLOOK_AUTOMATIONS_WORK_DIR=
# KOLOOK_COMFYUI_DEV_PATH (assumes <comfyui-root>/user/comfyui.log). Set this
# only if your install writes logs elsewhere.
KOLOOK_COMFYUI_LOG=
# --- Investigation patch sync targets ---------------------------------------
# Per-investigation absolute path to a third-party node's live install dir.
# Used by scripts/sync_investigation_patches.py to push patched files from
# docs/investigations/<NN>_<topic>/patches/ into the running ComfyUI.
# Each upstream gets its own env var, declared in the investigation's
# upstream.yaml under sync_target_env.
# WhatDreamsCost-ComfyUI install path. Required for investigation 01
# (LTX23-audio-file-lipsync). Forward slashes OK on Windows.
#
# Example:
# Windows: E:/_AI/portable/ComfyUI_windows_312/ComfyUI/custom_nodes/WhatDreamsCost-ComfyUI
KOLOOK_WHATDREAMSCOST_PATH=
CHANGELOG.md
+40
View File
@@ -6,6 +6,46 @@ The format is inspired by Keep a Changelog and SemVer.
## [Unreleased]
### Changed
- **Automations restructured around modules instead of models.**
`docs/automations/` now pivots on one folder per *task* (each its own
workflow + iteration loop + findings) rather than one folder per model.
The existing `LTX-2.3/{README,findings,handoff-checklist}.md` content
moved into `LTX-2.3/base-1step/` (single-stage render task) and the new
sibling `LTX-2.3/audio-lipsync/` (audio-file lip-sync task) was created.
`docs/automations/{README,CHEATSHEET}.md` updated to reflect the new
layout; `docs/automations/CONVENTIONS.md` unchanged.
- **`docs/investigations/` removed.** Its two children migrated to their
natural homes: narrative content into the new automation modules under
`docs/automations/LTX-2.3/{base-1step,audio-lipsync}/`; the modified
upstream Python code into the new `forks/whatdreamscost_koolook/`
fork (see *Added*). The retired `scripts/sync_investigation_patches.py`
and its `KOLOOK_WHATDREAMSCOST_PATH` env-var section in `.env.example`
go with it — iteration now uses the standard `dev-sync` flow against
the in-repo fork.
- **Glossary updated.** The *Investigation folder*, *Investigation patches*,
*JSON file folder*, and `sync_investigation_patches.py` entries in
[`docs/reference/glossary.md`](docs/reference/glossary.md) are replaced
by *Automation module*, *Backstory*, *Working folder*, and *Card*; the
*Loop* and *Run* entries are rewritten to point at the new structure.
### Added
- **`forks/whatdreamscost_koolook/v1_3_2/` — Koolook fork of WhatDreamsCost-ComfyUI's
`LTXDirector`** (upstream `e81223a`, GPL-3.0). Two upstream files modified
(`ltx_director.py` adds a `relay_overrides` multiline-JSON widget;
`prompt_relay.py` uses the Prompt-Relay paper's per-segment σ formula
instead of upstream's length-independent constant) plus the
unmodified `patches.py` vendored verbatim because the modified file
imports from it. Registered as
`LTXDirector__koolook_v1_3_2` (display name *"LTX Director (Koolook
v1.3.2)"*) so it coexists with an installed copy of upstream
WhatDreamsCost-ComfyUI in the node picker. Drives the
[`docs/automations/LTX-2.3/audio-lipsync/`](docs/automations/LTX-2.3/audio-lipsync/)
iteration loop. License attribution + change log in
[`forks/THIRD_PARTY.md`](forks/THIRD_PARTY.md); pin metadata in
[`forks/whatdreamscost_koolook/versions/v1_3_2/UPSTREAM_PIN.yaml`](forks/whatdreamscost_koolook/versions/v1_3_2/UPSTREAM_PIN.yaml)
and [`forks/forks_manifest.yaml`](forks/forks_manifest.yaml).
## [0.3.7] - 2026-05-23
### Added
__init__.py
+5
View File
@@ -25,6 +25,9 @@ try:
from .forks.radiance_koolook import NODE_CLASS_MAPPINGS as radiance_koolook_mappings
from .forks.radiance_koolook import NODE_DISPLAY_NAME_MAPPINGS as radiance_koolook_display
from .forks.whatdreamscost_koolook import NODE_CLASS_MAPPINGS as whatdreamscost_koolook_mappings
from .forks.whatdreamscost_koolook import NODE_DISPLAY_NAME_MAPPINGS as whatdreamscost_koolook_display
except ImportError as _node_import_exc:
# The relative imports above require a parent-package context that
# only ComfyUI's custom-node loader provides. Pytest's package
@@ -68,6 +71,7 @@ else:
**pattern_mappings,
**video_combine_mappings,
**radiance_koolook_mappings,
**whatdreamscost_koolook_mappings,
}
NODE_DISPLAY_NAME_MAPPINGS = {
@@ -79,6 +83,7 @@ else:
**pattern_display,
**video_combine_display,
**radiance_koolook_display,
**whatdreamscost_koolook_display,
}
WEB_DIRECTORY = "./web"
docs/automations/CHEATSHEET.md
+13 -7
View File
@@ -1,12 +1,12 @@
# Cheat sheet
For someone who already knows the loop. Full docs: [`README.md`](README.md), [`CONVENTIONS.md`](CONVENTIONS.md), [`LTX-2.3/README.md`](LTX-2.3/README.md).
For someone who already knows the loop. Full docs: [`README.md`](README.md), [`CONVENTIONS.md`](CONVENTIONS.md), and the per-module READMEs (e.g. [`LTX-2.3/base-1step/README.md`](LTX-2.3/base-1step/README.md), [`LTX-2.3/audio-lipsync/README.md`](LTX-2.3/audio-lipsync/README.md)).
---
## Bootstrap
Bare-bones — what's needed to get the loop running on a fresh machine. Model-specific setup (checkpoints, LoRAs, custom nodes for a particular model) lives in each model's handoff checklist, e.g. [`LTX-2.3/handoff-checklist.md`](LTX-2.3/handoff-checklist.md).
Bare-bones — what's needed to get the loop running on a fresh machine. Module-specific setup (checkpoints, LoRAs, custom nodes for a particular task, optional fork code) lives in each module's handoff checklist, e.g. [`LTX-2.3/base-1step/handoff-checklist.md`](LTX-2.3/base-1step/handoff-checklist.md).
**Repo side**
- [ ] Python 3.11+ with `pip install Pillow`. On Windows, the absolute path `C:/Python313/python.exe` sidesteps the Microsoft Store stub.
@@ -85,10 +85,16 @@ ComfyUI-Koolook/
├── README.md overview + start-here
├── CHEATSHEET.md this file
├── CONVENTIONS.md full contract
└── LTX-2.3/ one folder per model
├── README.md
├── handoff-checklist.md
└── findings.md
└── LTX-2.3/ model grouping (no shared docs of its own)
├── base-1step/ ← one automation module
│ ├── README.md
│ ├── handoff-checklist.md
│ └── findings.md
└── audio-lipsync/ ← another module on the same model
├── README.md
├── handoff-checklist.md
├── findings.md
└── runs/ in-repo run snapshots for fork-touching iterations
```
### Working-folder-side (per project — pointed at by `KOLOOK_AUTOMATIONS_WORK_DIR`)
@@ -113,7 +119,7 @@ Wipe `_AI/` to reset the agent's tracking history without touching renders. Wipe
- **`loop` in the filename = post-card output, skipped.** When ComfyUI saves the composited-with-card video back into the working folder, name it `<base>_loop_<seq>.mp4` (and pair JSON). Auto-discovery ignores anything with `loop` in the stem, so the script never feeds its own output back as input.
- **Card filename is stable.** Wire your NLE to `_AI/card.png` once.
- **Seed = 12 (fixed)** while sweeping any other knob. Vary seed only after a setting stabilises.
- **Scheduler = `linear_quadratic` (8 steps)** — locked-in finding. See [`LTX-2.3/findings.md`](LTX-2.3/findings.md).
- **Scheduler = `linear_quadratic` (8 steps)** — locked-in finding. See [`LTX-2.3/base-1step/findings.md`](LTX-2.3/base-1step/findings.md).
- **Promote** a hypothesis to a finding when it's stable across ≥ 3 runs in `_AI/iterations.md`.
- **Run label** on the card: prefers `v01`-style; falls back to `_00001` sequence numbers; else `?`.
- **Switching projects:** update `Working_Folder_PATH` node + `.env`. Node wins on mismatch.
docs/automations/LTX-2.3/audio-lipsync/README.md
+123
View File
@@ -0,0 +1,123 @@
# LTX 2.3 audio-file lip-sync — iteration loop
AI-assisted iteration loop for getting audio-file-driven lip-sync working
on the WhatDreamsCost `LTXDirector` (LTX 2.3). The vanilla `use_custom_audio = True`
path produces broken video — no lip-sync, no meaningful animation, prompt
adherence collapses. This automation iterates on a Koolook-modified
`LTXDirector` that adds a `relay_overrides` widget and a per-segment-σ
formula in `build_segments`; the loop sweeps those knobs to find a setting
that preserves video coherence with the external audio.
**Tracking issue:** [#177](https://github.com/malkuthro/ComfyUI-Koolook/issues/177)
**Sibling automation:** [`../base-1step/`](../base-1step/) — same model, the single-stage path with model-generated audio. Locked-in LTX-2.3 architecture facts live in its [`findings.md`](../base-1step/findings.md).
**Backstory:** [`backstory/audio-lipsync-rationale.md`](backstory/audio-lipsync-rationale.md) — why the audio path fails by default, the Prompt-Relay paper, what the two patches are doing mechanically.
**Started:** 2026-05-28
## What the Koolook fork ships
Lives at [`../../../../forks/whatdreamscost_koolook/versions/v1_3_2/`](../../../../forks/whatdreamscost_koolook/versions/v1_3_2/),
loaded by ComfyUI as `LTX Director (Koolook v1.3.2)` in the picker. Two upstream
modifications:
### Modification 1 — per-segment σ in `prompt_relay.py`
`build_segments` now computes σ per-segment using the Prompt-Relay paper
formula:
```
σ = (L − w_eff) / (2 · √ln(1/ε))
```
Upstream used a length-independent `σ = 1/ln(1/ε)` (≈ 0.1448 at ε=0.001) —
a razor-sharp boundary that did not scale with segment length L. The new
formula calibrates the penalty so it hits threshold ε exactly at the
segment boundary regardless of L. A `SIGMA_FALLBACK = 0.1448` preserves
the prior constant for the degenerate `L ≤ w_eff` corner. Per-segment σ
is logged as `[PromptRelay] seg L=… w_v=… sigma_v=…`.
### Modification 2 — `relay_overrides` widget on `LTXDirector`
A new optional multiline-string input on the LTXDirector canvas widget.
The maintainer pastes a JSON dict of Prompt-Relay knobs directly into the
node — values live entirely inside the workflow JSON, no disk file, no
env var.
Supported keys (all optional):
| Key | Type | Default | Effect |
|---|---|---|---|
| `video_strength` | float | 1.0 | Multiplies `attn2` penalty (video × text). Raise to tighten per-segment text routing. |
| `video_window_scale` | float | 1.0 | Scales the flat-top window width on the video side. < 1.0 = penalty kicks in closer to segment center. |
| `audio_strength` | float | 1.0 | Same for `audio_attn2`. **Note:** no-op for video output when `use_custom_audio=true` (audio latent is frozen at `noise_mask=0`). |
| `audio_window_scale` | float | 1.0 | Audio-side window. |
| `audio_epsilon` | float | inherit | Independent σ for the audio path. |
Underscore-prefixed keys are ignored (use for inline JSON comments). Example value to paste:
```json
{"video_strength": 10.0}
```
Empty field → upstream Prompt-Relay defaults (Modification 1 still active —
the new σ formula is unconditional).
## Folder map
```
audio-lipsync/
├── README.md ← this file
├── handoff-checklist.md ← bootstrap for picking up this automation cold
├── findings.md ← locked-in (currently empty; populated as runs validate)
├── backstory/
│ └── audio-lipsync-rationale.md
└── runs/ ← per-render snapshots
├── LOOP.md ← the iteration protocol
├── log.md ← rolling table of runs
└── run-NNN_<label>/
├── workflow.json
├── relay_overrides.txt
├── patch_state.txt
└── notes.md
```
## Iteration loop
1. **Edit a knob.** Either:
- **Widget-only swap** — change `relay_overrides` JSON on the LTXDirector node in ComfyUI. No restart needed; just queue.
- **Code-level change** — edit
[`../../../../forks/whatdreamscost_koolook/versions/v1_3_2/prompt_relay.py`](../../../../forks/whatdreamscost_koolook/versions/v1_3_2/prompt_relay.py)
or [`ltx_director.py`](../../../../forks/whatdreamscost_koolook/versions/v1_3_2/ltx_director.py)
in this repo. Then sync into the maintainer's live ComfyUI via the standard `dev-sync` (user-initiated; see project `CLAUDE.md`). Restart Comfy for Python module reload.
2. **Save workflow** — `Workflow → Save (API Format)` into the working folder.
3. **Render** — queue.
4. **Report** in chat — verbal feedback on sync state, motion, prompt adherence.
5. **Agent snapshots** the run into `runs/run-NNN_<label>/` and appends a row to [`runs/log.md`](runs/log.md).
See [`runs/LOOP.md`](runs/LOOP.md) for the full per-render protocol.
## Workflow JSON — Koolook node ID
The test workflow uses `LTXDirector__koolook_v1_3_2` (display name *"LTX
Director (Koolook v1.3.2)"*), not the upstream `LTXDirector`. Both appear
in ComfyUI's node picker side-by-side — upstream stays vanilla, the Koolook
variant carries our two modifications.
If you're starting from an existing workflow that wires the upstream
`LTXDirector`:
1. Open the workflow in ComfyUI.
2. Right-click the `LTX Director` node → **Convert / Replace** with `LTX Director (Koolook v1.3.2)`. Same input/output socket layout — wires are preserved.
3. The new `relay_overrides` widget appears at the bottom of the node — paste your JSON there (empty field is fine).
4. `Workflow → Save (API Format)` into the working folder.
Reverting to upstream behaviour for an A/B comparison: drop in the upstream
`LTXDirector` node next to the Koolook one and toggle which feeds the
downstream `LTXDirectorGuide`. No file backups needed — the upstream
install is untouched.
## Reverting changes to the fork
Edits to `forks/whatdreamscost_koolook/versions/v1_3_2/*.py` are normal
git-tracked changes — `git restore` / `git stash` work as usual. Once
`dev-sync` is run, the running ComfyUI install is updated; restart Comfy
so the Python module reimports.
docs/automations/LTX-2.3/audio-lipsync/backstory/audio-lipsync-rationale.md
+96
View File
@@ -0,0 +1,96 @@
# Why we forked LTXDirector for audio-file lip-sync
Backstory for the [LTX 2.3 audio-lipsync automation](../README.md): the
symptom, the mechanism, and what the two fork modifications are doing.
The running iteration narrative lives in
[`../runs/log.md`](../runs/log.md); locked findings in
[`../findings.md`](../findings.md).
## The symptom
When `use_custom_audio: true` and an external audio file is provided to
upstream `LTXDirector`, the resulting video is broken: no lip-sync,
no meaningful animation, prompt adherence collapses. The only mode that
produces good output is `use_custom_audio: false` with `[Audio]: …` in
the prompt (model-generated audio).
We want audio-file mode to work — the model-generated path can't be
art-directed against an existing soundtrack.
## Why upstream fails
Two mechanisms compounded:
### 1. Length-independent σ in `build_segments`
Upstream `prompt_relay.py` computes the segment cost penalty with:
```python
sigma = 1.0 / math.log(1.0 / epsilon) if 0 < epsilon < 1 else 0.1448
```
A constant ≈ 0.1448 at the paper-default ε=0.001. The Prompt-Relay paper
(*"Prompt Relay: Inference-Time Temporal Control for Multi-Event Video
Generation"*, [arXiv:2604.10030](https://arxiv.org/abs/2604.10030))
defines:
```
σ = (L − w) / (2 · √ln(1/ε))
```
— scaling σ with segment length L so the penalty hits threshold ε
exactly at the segment boundary. Upstream's constant-σ approximation
holds when all segments are roughly the same length, but breaks down when
segment lengths vary widely (which is what happens when audio drives the
segment boundaries — speech segments are not uniform).
### 2. No knob for per-render Prompt-Relay tuning
The Prompt-Relay penalty has knobs (`video_strength`, `video_window_scale`,
`audio_strength`, `audio_window_scale`, `audio_epsilon`) that the paper
varies depending on the data. Upstream hardcodes them, so the only way to
tune was edit-and-restart of `prompt_relay.py`. For an iteration loop
sweeping these knobs across runs, that's heavyweight.
## What the fork does
Two upstream files modified, one vendored verbatim:
| File | Change | Why |
|---|---|---|
| `prompt_relay.py` | Per-segment σ per the paper formula, with `SIGMA_FALLBACK = 0.1448` for the `L ≤ w_eff` degenerate corner. | Tracks the paper precisely — same behaviour as upstream when L is uniform; differs when L varies. |
| `ltx_director.py` | Adds a `relay_overrides` multiline JSON input on the LTXDirector node. Underscore-prefixed keys ignored (inline JSON comments). | Per-render knob sweeps without editing the fork file. Workflow JSON carries the override; reproducible across machines. |
| `patches.py` | **Unmodified**. Vendored verbatim because `ltx_director.py` imports `detect_model_type, apply_patches` from it. | Self-contained fork; no runtime dependency on the user's upstream WhatDreamsCost install for these two functions. |
The fork is loaded as `LTXDirector__koolook_v1_3_2` (display name
*"LTX Director (Koolook v1.3.2)"*). Upstream `LTXDirector` continues to
exist in the node picker for A/B comparison.
## Hypothesis space being swept
What's not yet validated and is what the iteration loop is for:
1. **Whether the paper σ formula alone fixes the symptom.** Empty
`relay_overrides` → fork's σ change is active, all other knobs at
upstream defaults. If sync recovers, σ was the dominant cause.
2. **Whether `video_strength` > 1 helps.** Tightening per-segment text
routing should reduce the model's freedom to ignore the prompt under
audio pressure. Sweep `1, 2.5, 10`.
3. **Whether window scaling helps.** Narrowing the flat-top makes the
penalty kick in closer to segment center, biasing the model to commit
to one prompt segment at a time. Sweep `video_window_scale` `0.7 / 1.0`.
4. **Whether audio-side knobs matter at all when `use_custom_audio=true`.**
The audio latent is frozen at `noise_mask=0` in that mode, so
`audio_attn2`'s penalty has no effect on the video output. Predict
`audio_strength` is a no-op; confirm empirically.
Each of these turns into one or more rows in
[`../runs/log.md`](../runs/log.md). Once a pattern shows up across
≥ 3 confirming runs, it graduates to [`../findings.md`](../findings.md).
## Related
- [Prompt-Relay paper](https://arxiv.org/abs/2604.10030) — Chen, Huang, Liu (S-Lab, NTU, April 2026).
- [Reference implementation by paper authors](https://github.com/GordonChen19/Prompt-Relay).
- [Kijai's ComfyUI port](https://github.com/kijai/ComfyUI-PromptRelay) — the basis of WhatDreamsCost's LTXDirector.
- Sibling automation: [`../../base-1step/`](../../base-1step/) — same model, the model-generated-audio path. Locked-in LTX-2.3 architecture facts (spatial upscaler ×2 hardcoding, attention scaling) live in its [`findings.md`](../../base-1step/findings.md) and apply here too.
docs/automations/LTX-2.3/audio-lipsync/findings.md
+26
View File
@@ -0,0 +1,26 @@
# LTX 2.3 audio-lipsync — promoted findings
Settled conclusions from the iteration loop. Entries graduate here from the
running [`runs/log.md`](runs/log.md) when a pattern shows up across multiple
renders.
This file is scoped to the **audio-file lip-sync** automation. Model-level
LTX-2.3 architecture facts (spatial upscaler ×2 hardcoding, brief path,
attention quadratic in token count) live in the sibling
[`../base-1step/findings.md`](../base-1step/findings.md) and apply here too.
Anything in this file is **locked in** — don't sweep it again without a reason.
---
*No findings yet. First entry appears once a knob's effect is stable across
≥ 3 confirming runs.*
---
## How to add a new finding
1. Pattern is stable across ≥ 3 runs in [`runs/log.md`](runs/log.md).
2. You can describe it as a single declarative sentence ("X causes Y" or "do X, don't do Y").
3. Move it here under an appropriate heading, with a one-paragraph "why" and links to the run(s) that confirmed it.
4. Remove the corresponding hypothesis from the open list in [`README.md`](README.md) (and from [`backstory/audio-lipsync-rationale.md`](backstory/audio-lipsync-rationale.md) if it was raised there) so those files shrink toward truth.
docs/automations/LTX-2.3/audio-lipsync/handoff-checklist.md
+70
View File
@@ -0,0 +1,70 @@
# Handoff checklist — LTX 2.3 audio-lipsync automation
Use this when picking up this automation on a different machine, or handing
over to someone else. Target time-to-first-render: ~5 minutes after the
sibling [`../base-1step/handoff-checklist.md`](../base-1step/handoff-checklist.md)
prerequisites are met.
The base-1step handoff covers shared prerequisites (Python + Pillow + the
working folder + Text Multiline nodes + most custom-node packs). Do that
first, then come back here.
## 1. Extra ComfyUI deps
This automation drives a modified `LTXDirector`, so beyond the base-1step
custom-node list you also need:
- [ ] **`WhatDreamsCost-ComfyUI` installed.** The Koolook fork only ports
`LTXDirector`; the companion `LTXDirectorGuide`, `LTXKeyframer`,
`MultiImageLoader`, `LTXSequencer`, `SpeechLengthCalculator`,
`LoadAudioUI`, `LoadVideoUI` come from upstream. Install via
ComfyUI-Manager or `git clone https://github.com/WhatDreamsCost/WhatDreamsCost-ComfyUI` into `custom_nodes/`.
- [ ] **The Koolook fork is loaded automatically.** No extra step — it ships
inside this repo at
[`../../../../forks/whatdreamscost_koolook/`](../../../../forks/whatdreamscost_koolook/)
and is wired into [`../../../../__init__.py`](../../../../__init__.py). On
ComfyUI startup you should see both `LTX Director` (upstream) and
`LTX Director (Koolook v1.3.2)` in the node picker.
## 2. Workflow file
- [ ] Save the audio-lipsync workflow as
`<workflow root>/LTX-23-audio_tests_v01.json` (or whatever filename your
working folder uses). The Director node must be
**`LTX Director (Koolook v1.3.2)`** (registered ID
`LTXDirector__koolook_v1_3_2`), not the upstream `LTX Director`.
- [ ] On the Koolook Director, the `relay_overrides` widget sits at the
bottom of the node. Start with an empty string for a regression check
against upstream-default behaviour, then iterate.
## 3. First render
- [ ] Queue once. Confirm in chat with the maintainer what the audio /
sync / motion looked like.
- [ ] Agent will capture the snapshot into `runs/run-001_<label>/` and
append a row to [`runs/log.md`](runs/log.md).
If the Director node still shows the upstream `LTXDirector` after loading
the workflow:
- Check the ComfyUI startup log for a `[Koolook] node registry skipped` or
`from .forks.whatdreamscost_koolook import …` ImportError — that means
the fork didn't load (transitive import failure). Run pytest at the repo
root to see the failure outside ComfyUI.
- Verify `pip show -f WhatDreamsCost-ComfyUI` or equivalent — the Koolook
fork imports through `comfy_api`, which only exists in a real ComfyUI
process.
## 4. Iteration
See [`README.md`](README.md) (this folder) for the iteration loop and
[`runs/LOOP.md`](runs/LOOP.md) for the per-render protocol.
## Off-boarding
- Commit + push `findings.md` updates and any fork-code edits you made
this session.
- The per-project working folder (workflow JSON + MP4 + `_AI/`) stays
where it is, outside the repo. Make sure
`KOLOOK_AUTOMATIONS_WORK_DIR` in `.env` points at it so the next
machine picks up the same iterations log.
docs/automations/LTX-2.3/audio-lipsync/runs/LOOP.md
+68
View File
@@ -0,0 +1,68 @@
# LTX 2.3 audio-lipsync — per-render iteration protocol
**Automation:** [LTX 2.3 audio-lipsync](../README.md)
**Tracking issue:** [malkuthro/ComfyUI-Koolook#177](https://github.com/malkuthro/ComfyUI-Koolook/issues/177)
**Started:** 2026-05-28
---
## The loop
| Step | Who | What |
|---|---|---|
| 1 | Maintainer | Either edit `relay_overrides` JSON on the **`LTX Director (Koolook v1.3.2)`** node in ComfyUI, OR edit a `forks/whatdreamscost_koolook/versions/v1_3_2/*.py` source in this repo + run `dev-sync` + restart Comfy. |
| 2 | Maintainer | **Save** workflow → overwrites the current workflow file at the working folder. |
| 3 | Maintainer | Queue render in ComfyUI. |
| 4 | Maintainer | Report result in chat (verbal feedback — sync state, motion state, prompt adherence). |
| 5 | Agent | Snapshot current state into a new `run-NNN_<label>/` folder here + append a row to [`log.md`](log.md). |
| 6 | Agent | Confirm what was captured. |
**Save-before-render** is the anchor: the workflow file on disk at submission time = exactly what produced the render being described. No copy-paste, no version drift.
## Per-run snapshot contents
```
run-NNN_<label>/
├── workflow.json ← copy of the working-folder workflow at submission
├── relay_overrides.txt ← the LTX Director (Koolook) node's relay_overrides widget value
├── patch_state.txt ← MAIN SHA + dev-sync SHA + whether forks/.../v1_3_2/*.py differs from MAIN
└── notes.md ← maintainer's verbal feedback + agent's one-line interp
```
The `relay_overrides` value is also stored inside `workflow.json` (Koolook
LTXDirector node widget), but the `.txt` copy makes it diff-friendly in the
run folder.
## Trigger detection
No magic phrase. Whenever the maintainer describes a render result in chat, the agent treats it as a feedback signal and creates a new run folder. The agent confirms in chat what was captured; the maintainer can rename, delete, or correct.
To share a workflow file *without* logging it as a run, say "no log" or "don't log this" in the chat message.
## Run labels
Auto-generated from key knob state + feedback summary, e.g.:
- `run-001_no-override_baseline-fork-only`
- `run-002_vstr10_no-change`
- `run-003_vstr25_some-motion-recovered`
- `run-004_vstr10_astr0.5_sync-partial`
- `run-005_codechange_sigma-bumped_sync-clean`
The maintainer can override any label by stating it explicitly.
## Syncing fork-code edits to the live install
When the iteration step requires editing a `forks/whatdreamscost_koolook/`
source (not just the widget), the standard
[`dev-sync`](../../../../scripts/sync_to_dev.py) flow applies — it's a
user-initiated, never-automatic action per the project `CLAUDE.md`. Quick
recipe:
```
python scripts/sync_to_dev.py --scope "audio-lipsync fork edit"
```
Then restart Comfy so the Python module re-imports. Widget-only changes
(editing `relay_overrides` JSON on the canvas) do **not** require sync or
restart — just queue the next render.
docs/investigations/01_LTX23-audio-file-lipsync/runs/log.md → docs/automations/LTX-2.3/audio-lipsync/runs/log.md
View File
docs/automations/LTX-2.3/README.md → docs/automations/LTX-2.3/base-1step/README.md
+26 -14
View File
@@ -1,16 +1,25 @@
# LTX 2.3 — evaluation loop
# LTX 2.3 base — single-stage iteration loop
AI-assisted iteration loop for evaluating **Lightricks LTX 2.3** video generation behaviour. Currently focused on the WhatDreamsCost `LTXDirector` node — multi-segment prompts, keyframe transitions, audio sync, and high-resolution behaviour.
AI-assisted iteration loop for the **single-stage LTX 2.3** rendering path: brief
the WhatDreamsCost `LTXDirector` once, sample 8 steps at the chosen resolution,
no Phase-2 upscale. Each render produces a side-by-side card; the maintainer
scores it; locked-in conclusions graduate to [`findings.md`](findings.md). The
sibling [`../audio-lipsync/`](../audio-lipsync/) automation iterates on the
same model with audio-file lip-sync instead.
The shared loop pattern + node contract is in [`../CONVENTIONS.md`](../CONVENTIONS.md). Anything LTX-specific is here.
The shared loop pattern + Text-Multiline node contract is in
[`../../CONVENTIONS.md`](../../CONVENTIONS.md); anything specific to *this*
automation lives here.
## What this loop is for
- Sweep one knob at a time (Phase 2 denoise, scheduler shape, resolution selector, etc.), render at the chosen seed, capture a tracking card next to the video.
- Sweep one knob at a time (resolution selector, scheduler shape, denoise, etc.) at a fixed seed; capture a tracking card next to the video.
- Build a body of evidence over multiple runs about *what each knob actually does at this model's training boundaries*.
- Promote stable findings to [`findings.md`](findings.md) so they don't get re-discovered.
The original investigation that produced this loop: [`../../investigations/ltx-director-4k-transitions.md`](../../investigations/ltx-director-4k-transitions.md).
How this automation got here — what the LTX Director actually does, the 4K
dissolve / 2K morph experiments, scheduler-shape lock-in — is captured in the
backstory: [`backstory/4k-transitions.md`](backstory/4k-transitions.md).
## The loop (per iteration)
@@ -50,23 +59,26 @@ Free-text fields (filled by you in the workflow):
| Feedback | `OVERLAY - FEEDBACK` (non-score lines) | Post-render observations. |
| Outcome scores | `OVERLAY - FEEDBACK` (lines like `motion: 4/5`) | Each axis 0–5; missing axes show as `?/5`. |
## Active investigation
## Open questions (still iterating)
The 4K dissolve / audio drift problem. See [`findings.md`](findings.md) for what's already settled and [`../../investigations/ltx-director-4k-transitions.md`](../../investigations/ltx-director-4k-transitions.md) for the running narrative.
The 4K dissolve / audio drift behaviour. See [`findings.md`](findings.md) for
what's already settled and
[`backstory/4k-transitions.md`](backstory/4k-transitions.md) for the running
narrative that produced this automation.
Open hypotheses being tested:
- Phase 2 denoise = 0.80 is too aggressive for "preserve stage 1" semantics; sweep down toward 0.30.
- Phase 2 denoise = 0.80 is too aggressive for "preserve stage 1" semantics; sweep down toward 0.30. *(Two-stage variant — out of scope for this base 1-step automation; tracked here only because the hypothesis predates the split.)*
- LTX 2.3 prior weakens at 4K → 5K, producing dissolves in inter-keyframe motion.
- `LTXVLatentUpsampler [BETA]` may inject mouth-region detail that breaks audio sync.
- `LTXVLatentUpsampler [BETA]` may inject mouth-region detail that breaks audio sync. *(Same — two-stage-specific.)*
## Where to find everything
| Thing | Path |
|---|---|
| Card renderer | [`scripts/make_card.py`](../../../scripts/make_card.py) |
| Watcher | [`scripts/watch_cards.py`](../../../scripts/watch_cards.py) |
| `/make-card` skill | [`.claude/skills/make-card/`](../../../.claude/skills/make-card/) |
| Cross-cutting conventions | [`../CONVENTIONS.md`](../CONVENTIONS.md) |
| Original investigation narrative | [`../../investigations/ltx-director-4k-transitions.md`](../../investigations/ltx-director-4k-transitions.md) |
| Card renderer | [`scripts/make_card.py`](../../../../scripts/make_card.py) |
| Watcher | [`scripts/watch_cards.py`](../../../../scripts/watch_cards.py) |
| `/make-card` skill | [`.claude/skills/make-card/`](../../../../.claude/skills/make-card/) |
| Cross-cutting conventions | [`../../CONVENTIONS.md`](../../CONVENTIONS.md) |
| Backstory narrative | [`backstory/4k-transitions.md`](backstory/4k-transitions.md) |
| Working folder | `$KOLOOK_AUTOMATIONS_WORK_DIR` (set in `.env` — see `.env.example`) |
| Handoff to another machine / person | [`handoff-checklist.md`](handoff-checklist.md) |
docs/investigations/00_LTX23-base-1-step/README.md → docs/automations/LTX-2.3/base-1step/backstory/4k-transitions.md
+7 -3
View File
@@ -1,9 +1,13 @@
# LTX Director — 4K multi-keyframe transition breakdown
**Branch:** `investigate/ltx-director-4k-transitions`
The running narrative that produced the [LTX 2.3 base 1-step automation](../README.md).
Architectural mechanics, observations across four rounds (HD vs 2K vs 4K,
two-stage workflow, scheduler-shape lock-in), and the open questions that
graduated to [`../findings.md`](../findings.md).
**Target node:** WhatDreamsCost-ComfyUI → `LTXDirector` (v1.3.2)
**Status:** investigation kickoff — no code changes yet.
**Date:** 2026-05-16
**Status:** narrative complete; live iteration continues per [`../README.md`](../README.md). Audio-file lip-sync work split into the sibling [`../../audio-lipsync/`](../../audio-lipsync/) automation.
**Started:** 2026-05-16
---
docs/investigations/experiments/baseline.html → docs/automations/LTX-2.3/base-1step/example-cards/baseline.html
View File
docs/investigations/experiments/combined-card.html → docs/automations/LTX-2.3/base-1step/example-cards/combined-card.html
View File
docs/investigations/experiments/run-001-card.html → docs/automations/LTX-2.3/base-1step/example-cards/run-001-card.html
View File
docs/investigations/experiments/run-001-card.png → docs/automations/LTX-2.3/base-1step/example-cards/run-001-card.png
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 76 KiB

After

Width:  |  Height:  |  Size: 76 KiB

docs/investigations/experiments/run-v04-card.png → docs/automations/LTX-2.3/base-1step/example-cards/run-v04-card.png
View File
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 68 KiB

After

Width:  |  Height:  |  Size: 68 KiB

docs/automations/LTX-2.3/findings.md → docs/automations/LTX-2.3/base-1step/findings.md
+14 -4
View File
@@ -1,6 +1,16 @@
# LTX 2.3 — promoted findings
# LTX 2.3 base — promoted findings
Settled conclusions from the iteration loop. Entries graduate here from the
running per-project `_AI/iterations.md` log (lives outside the repo, in your
working folder; see [`../README.md`](README.md) for the contract).
This file is scoped to the **single-stage LTX 2.3 base** automation. The
spatial-upscaler architecture facts at the bottom of this file are model-level
truths that any LTX 2.3 automation can rely on (including the sibling
[`../audio-lipsync/`](../audio-lipsync/) module); they sit here because this
is where they were first established. Lift to a shared model-architecture doc
if a third LTX module ever needs to reference them.
Settled conclusions from the iteration loop. Entries graduate here from the running [iterations log](../../../).<br>
Anything in this file is **locked in** — don't sweep it again without a reason.
## Sampling — scheduler shape
@@ -11,7 +21,7 @@ The sigma curve it produces — flat plateau near σ ≈ 1.0 for the first ~5 st
- Other scheduler shapes degrade fast motion and keyframe transitions.
- Both Phase 1 and Phase 2 keep this scheduler. Phase 2 starts slightly lower on σ because the lower denoise value shifts the entry point down the curve.
- Source: [investigation Round 4](../../investigations/ltx-director-4k-transitions.md#round-4--scheduler-finding--tracking-system-plan).
- Source: [backstory Round 4](backstory/4k-transitions.md#round-4--scheduler-finding--tracking-system-plan).
## Sampling — locked knobs
@@ -101,7 +111,7 @@ Two-stage = render at half-res Phase 1, upscale + resample at Phase 2 via `LTXVL
Current working hypothesis (Round 3 of investigation): #1 is dominant; #2 is a secondary contributor; #3 is the floor.
See [`../../investigations/ltx-director-4k-transitions.md`](../../investigations/ltx-director-4k-transitions.md) for the full hypothesis tree and test plan.
See [`backstory/4k-transitions.md`](backstory/4k-transitions.md) for the full hypothesis tree and test plan.
## How to add a new finding
docs/automations/LTX-2.3/handoff-checklist.md → docs/automations/LTX-2.3/base-1step/handoff-checklist.md
+7 -5
View File
@@ -1,12 +1,14 @@
# Handoff checklist — LTX 2.3 loop
# Handoff checklist — LTX 2.3 base (single-stage) automation
Use this when picking up the loop on a different machine, or handing over to someone else. Target time-to-first-card: ~5 minutes.
Use this when picking up this automation on a different machine, or handing over to someone else. Target time-to-first-card: ~5 minutes.
The sibling [`../audio-lipsync/`](../audio-lipsync/) automation has its own handoff checklist — that one needs the WhatDreamsCost Koolook fork loaded and a workflow JSON that uses the Koolook-namespaced node. This one does not.
## 1. Repo + branch
- [ ] Clone `ComfyUI-Koolook`.
- [ ] Check out the branch carrying the loop tooling — currently `investigate/ltx-director-4k-transitions`. (Eventually expected to merge to `main`.)
- [ ] Read [`README.md`](README.md) (this folder) and [`../CONVENTIONS.md`](../CONVENTIONS.md).
- [ ] Use `main` (the loop tooling is part of the standard tree).
- [ ] Read [`README.md`](README.md) (this folder) and [`../../CONVENTIONS.md`](../../CONVENTIONS.md).
## 2. Python + Pillow
@@ -24,7 +26,7 @@ Use this when picking up the loop on a different machine, or handing over to som
- `was-node-suite-comfyui` (`Text Multiline`).
- `rgthree-comfy` (helpful for finding nodes by ID).
- `pythongosssss/ComfyUI-Custom-Scripts` (optional — workflow auto-save).
- [ ] An LTX-2.3 workflow with the three required Text Multiline nodes (see `../CONVENTIONS.md` §2): `Working_Folder_PATH`, `OVERLAY - INFO`, `OVERLAY - FEEDBACK`.
- [ ] An LTX-2.3 workflow with the three required Text Multiline nodes (see `../../CONVENTIONS.md` §2): `Working_Folder_PATH`, `OVERLAY - INFO`, `OVERLAY - FEEDBACK`.
## 4. Working folder
docs/automations/README.md
+65 -20
View File
@@ -1,8 +1,15 @@
# Automations
AI-assisted iteration loops for evaluating generative models inside ComfyUI.
AI-managed ComfyUI iteration loops. Each automation module is one
self-contained workflow: own ComfyUI session, own workflow JSON, own
narrative, own findings — typically focused on one specific generation
task (a single-stage render path, an audio-driven lip-sync, an upscale
recipe, etc.). The same diffusion model can host several modules side
by side; one folder per **task**, not one per model.
Each subfolder is one model under study. The loop pattern is the same across all of them; the model-specific knobs, conventions, and findings live in the subfolder.
The agent's role inside each module is the same: watch the maintainer's
workflow edits, score feedback, render tracking cards via `/make-card`,
and — when the task needs it — modify code in `forks/` or `scripts/`.
> **Want the visual walkthrough?** → [`index.html`](index.html) presents the setup and loop as an HTML overview.
>
@@ -22,18 +29,40 @@ ComfyUI-Koolook/
├── scripts/
│ ├── make_card.py renders a card PNG from workflow JSON
│ └── watch_cards.py optional polling watcher
├── forks/ ← any modified upstream node code an
│ automation needs (e.g. whatdreamscost_koolook/)
└── docs/automations/
├── README.md you are here
├── CONVENTIONS.md cross-cutting rules
└── LTX-2.3/ one folder per model under study
├── README.md loop entry point for this model
├── handoff-checklist.md 5-minute onboarding
└── findings.md locked-in conclusions
├── CONVENTIONS.md cross-cutting loop contract
├── CHEATSHEET.md one-page reminder
└── LTX-2.3/ grouping by diffusion model
├── base-1step/ ← one automation module
│ ├── README.md loop entry point for this task
│ ├── handoff-checklist.md
│ ├── findings.md locked-in conclusions for THIS task
│ ├── backstory/ narrative that produced the module
│ └── example-cards/ rendered cards from past runs
└── audio-lipsync/ ← another automation module, same model
├── README.md
├── handoff-checklist.md
├── findings.md
├── backstory/
└── runs/ per-render snapshots in the repo
├── LOOP.md per-task iteration protocol
└── log.md rolling table of runs
```
The model-name folder (`LTX-2.3/`) is **only a grouping** — it doesn't
carry shared docs of its own. Each automation under it is independent
(own README, own findings). Model-architecture facts that genuinely
apply to every module on that model can live in any one module's
`findings.md` and be cross-linked from siblings (or lifted to a shared
file later if a third module materialises and the duplication is real).
### Per-project working folder (lives outside the repo)
Pointed at by `KOLOOK_AUTOMATIONS_WORK_DIR` in `.env`. One folder per project. Holds:
Pointed at by `KOLOOK_AUTOMATIONS_WORK_DIR` in `.env`. One folder per
project. Holds:
```
<working folder>/ e.g. JOBS/<client>/<scene>/ComfyUI-working-folder
@@ -45,29 +74,45 @@ Pointed at by `KOLOOK_AUTOMATIONS_WORK_DIR` in `.env`. One folder per project. H
└── iterations.md append-only log, one row per render
```
The `_AI/` split lets you wipe agent artifacts without touching user content (or vice-versa). You can delete `_AI/` to reset the history; you can delete the JSONs and MP4s without losing the iteration log.
The `_AI/` split lets you wipe agent artifacts without touching user
content (or vice-versa). You can delete `_AI/` to reset the history;
you can delete the JSONs and MP4s without losing the iteration log.
For automations that ship in-repo `runs/` snapshots (the
audio-lipsync module does this — because the iteration touches
fork code, not just widgets), the snapshots live inside the module
folder. The `_AI/iterations.md` outside-the-repo log is still the
authoritative running record; the in-repo `runs/log.md` is the
condensed cross-machine summary.
## What lives here
| Subfolder | Status | Scope |
|---|---|---|
| [`LTX-2.3/`](LTX-2.3/README.md) | active | Lightricks LTX 2.3 video diffusion — Director node behaviour, 4K transitions, audio sync |
| Module | Model | Status | Task |
|---|---|---|---|
| [`LTX-2.3/base-1step/`](LTX-2.3/base-1step/README.md) | LTX 2.3 | active | Single-stage render: brief once, sample 8 steps, no Phase-2 upscale. Model-generated audio. |
| [`LTX-2.3/audio-lipsync/`](LTX-2.3/audio-lipsync/README.md) | LTX 2.3 | active | Audio-file-driven lip-sync. Modified `LTXDirector` (Prompt-Relay paper σ + `relay_overrides` widget). Ships a fork at [`forks/whatdreamscost_koolook/`](../../forks/whatdreamscost_koolook/). |
## The pattern (every loop looks like this)
## The pattern (every module looks like this)
1. **One working folder per project** holds everything for one iteration: workflow JSON + rendered video at the root (user content), plus an `_AI/` subfolder for the agent-managed card + iterations log.
1. **One working folder per project** holds everything for the iteration: workflow JSON + rendered video at the root (user content), plus an `_AI/` subfolder for the agent-managed card + iterations log.
2. **Three named Text Multiline nodes** inside the ComfyUI workflow form the data contract:
- `Working_Folder_PATH` — authoritative output path.
- `OVERLAY - INFO` — the run params summary + free-text `BASE (notes):` for what changed this run.
- `OVERLAY - FEEDBACK` — observations on the rendered video + score lines (`motion: 4/5`, `sync: 5/5`, `sharp: 4/5`).
3. **The `/make-card` skill** (or `scripts/make_card.py` directly) reads the latest JSON in the working folder, extracts every parameter, renders `_AI/card.png`, and appends a row to `_AI/iterations.md`. Cards travel alongside the video into the NLE for side-by-side comparison.
4. **Findings get promoted** from per-run notes into the model's `findings.md` when a knob's behaviour is stable enough to lock in.
4. **Code modifications go in `forks/`** when a module needs to alter an upstream node. The agent edits the fork file in the repo; the maintainer runs `dev-sync` to copy it into the live install. See [`../../forks/README.md`](../../forks/README.md).
5. **Findings get promoted** from per-run notes into the module's `findings.md` when a knob's behaviour is stable enough to lock in (≥ 3 confirming runs).
Full cross-cutting contract: [`CONVENTIONS.md`](CONVENTIONS.md).
## Starting a new loop (new model)
## Starting a new automation module
1. Pick a folder name — short, lowercase, hyphen-separated (e.g. `wan-2.1`, `audio-ldm-3`).
2. Copy `LTX-2.3/README.md` and `LTX-2.3/handoff-checklist.md` to the new folder as templates.
3. Replace LTX-specific node names, parameter sets, and finding placeholders with the new model's.
4. The cross-cutting tooling (`scripts/make_card.py`, `.claude/skills/make-card/`) is model-agnostic — no changes needed there unless the new model needs different fields extracted.
1. Pick a folder name — short, lowercase, hyphen-separated, descriptive of the *task* (e.g. `LTX-2.3/two-stage-upscale/`, `wan-2.1/i2v-from-still/`). The model-name grouping is optional but recommended once there are 2+ modules on the same model.
2. Copy `LTX-2.3/base-1step/README.md` and `handoff-checklist.md` to the new folder as templates.
3. Rewrite the *task scope* paragraph at the top. Replace task-specific node names, parameter sets, and finding placeholders with the new module's.
4. Add a row to the "What lives here" table above.
5. If the module needs to modify upstream node code, run the
[`add-external-fork`](../../.claude/skills/add-external-fork/) skill
first to register the fork properly; then the module can reference
`forks/<name>_koolook/`.
6. The cross-cutting tooling (`scripts/make_card.py`, `.claude/skills/make-card/`) is module-agnostic — no changes needed there unless the new module needs different fields extracted from its workflow.
docs/automations/index.html
+1 -1
View File
@@ -978,7 +978,7 @@ Card ready:
</section>
<aside class="callout">
<p><strong>Reference layer:</strong> use <a href="./README.md">README.md</a> for the overview, <a href="./CONVENTIONS.md">CONVENTIONS.md</a> for the shared contract, <a href="./CHEATSHEET.md">CHEATSHEET.md</a> during a live session, and <a href="./LTX-2.3/README.md">LTX-2.3/README.md</a> for the active model loop.</p>
<p><strong>Reference layer:</strong> use <a href="./README.md">README.md</a> for the overview, <a href="./CONVENTIONS.md">CONVENTIONS.md</a> for the shared contract, <a href="./CHEATSHEET.md">CHEATSHEET.md</a> during a live session, and the per-module READMEs — e.g. <a href="./LTX-2.3/base-1step/README.md">LTX-2.3/base-1step/README.md</a> and <a href="./LTX-2.3/audio-lipsync/README.md">LTX-2.3/audio-lipsync/README.md</a>.</p>
</aside>
</main>
</body>
docs/investigations/01_LTX23-audio-file-lipsync/README.md
-103
View File
@@ -1,103 +0,0 @@
# 01 — LTX 2.3 Director: audio-file lip-sync investigation
**Tracking issue:** [#177](https://github.com/malkuthro/ComfyUI-Koolook/issues/177)
**Related:** [`../00_LTX23-base-1-step/README.md`](../00_LTX23-base-1-step/README.md) (the foundational LTX 2.3 investigation — 4K dissolves, 2-stage workflow, scheduler).
**Started:** 2026-05-28
**Status:** investigation-phase patches applied to live install; sync via `sync-audio`.
---
## Problem
When `use_custom_audio: true` and an external audio file is provided to `LTXDirector`, the resulting video is broken: no lip-sync, no meaningful animation, prompt adherence collapses. The only mode that produces good output is `use_custom_audio: false` with `[Audio]: …` in the prompt (model-generated audio).
Goal: investigate whether the Prompt Relay cross-attention math can be tuned to give the text prompt enough pull on the video to overcome the audio-driven prior baked into LTX 2.3's training distribution.
## Folder map
```
01_LTX23-audio-file-lipsync/
├── README.md ← this file
├── upstream.json ← upstream version pin + sync target + file list
├── patches/ ← source of truth for modified upstream code
│ ├── ltx_director.py
│ └── prompt_relay.py
└── runs/ ← per-render snapshots
├── LOOP.md ← the iteration protocol
├── log.md ← rolling table of runs
└── run-NNN_<label>/
├── workflow.json
├── relay_overrides.txt
├── patch_state.txt
└── notes.md
```
## Patch summary
### Patch 1 — `prompt_relay.py`
`build_segments` now computes σ per-segment using the Prompt-Relay paper formula:
```
σ = (L − w_eff) / (2 · √ln(1/ε))
```
Original code used a length-independent `σ = 1/ln(1/ε)` (≈ 0.1448 at ε=0.001), producing a razor-sharp boundary that did not scale with L. The new formula calibrates the penalty so it hits threshold ε exactly at the segment boundary regardless of segment length.
A `SIGMA_FALLBACK = 0.1448` preserves the prior constant for the degenerate `L ≤ w_eff` corner. Per-segment σ is now logged: `[PromptRelay] seg L=… w_v=… sigma_v=…`.
### Patch 2 — `ltx_director.py`
The `LTXDirector` node gains a new optional input `relay_overrides` (multiline string, default `""`). The maintainer types a JSON dict of Prompt Relay knobs directly into the node's canvas widget. The values live entirely inside the workflow JSON — no disk file, no env var.
Supported keys (all optional):
| Key | Type | Default | Effect |
|---|---|---|---|
| `video_strength` | float | 1.0 | Multiplies `attn2` penalty (video × text). Raise to tighten per-segment text routing. |
| `video_window_scale` | float | 1.0 | Scales the flat-top window width on the video side. < 1.0 = penalty kicks in closer to segment center. |
| `audio_strength` | float | 1.0 | Same for `audio_attn2`. **Note:** no-op for video output when `use_custom_audio=true` (audio latent is frozen at `noise_mask=0`). |
| `audio_window_scale` | float | 1.0 | Audio-side window. |
| `audio_epsilon` | float | inherit | Independent σ for the audio path. |
Underscore-prefixed keys are ignored (use for inline JSON comments). Example value to paste:
```json
{"video_strength": 10.0}
```
Empty field → upstream Prompt Relay behaviour.
## Sync — pushing patches to the live install
MAIN is the source of truth. The live install at `$KOLOOK_WHATDREAMSCOST_PATH` is a destination only.
**Trigger phrase: `sync-audio`** (declared in [`upstream.json`](upstream.json)). Equivalent CLI:
```
python scripts/sync_investigation_patches.py 01_LTX23-audio-file-lipsync
```
The script reads [`upstream.json`](upstream.json), backs up the target's current state (`.bak.<YYYYMMDD>`, once per day), copies `patches/*.py` over, prints a 2-line report.
## Test workflow
`E:/_AI/portable/ComfyUI_windows_312/ComfyUI/user/default/workflows/LTX-23-audio_tests_v01.json` — the maintainer overwrites this single file from inside ComfyUI between iterations.
See [`runs/LOOP.md`](runs/LOOP.md) for the per-render iteration protocol.
## Reversion (any one works)
1. Restore `.bak.<date>` files in the live install dir (created automatically by the sync script).
2. `git -C "$KOLOOK_WHATDREAMSCOST_PATH" checkout -- prompt_relay.py ltx_director.py` — upstream is a git clone (verified: `.git` present).
3. ComfyUI-Manager → Try Update — re-clones from upstream.
## Promotion path
If the patches validate:
1. Run [`license-pre-check`](../../../.claude/skills/license-pre-check/) — upstream is GPL-3.0, Koolook is GPL-3.0 (compatible).
2. Run [`add-external-fork`](../../../.claude/skills/add-external-fork/) to register `WhatDreamsCost-ComfyUI` in [`../../../forks/forks_manifest.yaml`](../../../forks/forks_manifest.yaml).
3. Create `forks/ltx_director_koolook/` with namespaced node ID (`LTXDirector_Koolook`) so it coexists with installed upstream.
4. Update [`../../../forks/THIRD_PARTY.md`](../../../forks/THIRD_PARTY.md).
5. Promote validated findings to [`../../automations/LTX-2.3/findings.md`](../../automations/LTX-2.3/findings.md) per its "How to add a new finding" rule (≥3 confirming runs).
docs/investigations/01_LTX23-audio-file-lipsync/runs/LOOP.md
-67
View File
@@ -1,67 +0,0 @@
# Audio-file lip-sync investigation — iteration loop
**Investigation:** [01_LTX23-audio-file-lipsync](../README.md)
**Tracking issue:** [malkuthro/ComfyUI-Koolook#177](https://github.com/malkuthro/ComfyUI-Koolook/issues/177)
**Started:** 2026-05-28
---
## The loop
| Step | Who | What |
|---|---|---|
| 1 | Maintainer | Edit workflow in ComfyUI (canvas — node widgets, including the `relay_overrides` JSON field on the LTX Director) |
| 2 | Maintainer | **Save** workflow → overwrites `E:\_AI\portable\ComfyUI_windows_312\ComfyUI\user\default\workflows\LTX-23-audio_tests_v01.json` |
| 3 | Maintainer | Queue render in ComfyUI |
| 4 | Maintainer | Report result in chat (verbal feedback — sync state, motion state, prompt adherence) |
| 5 | Agent | Snapshot current state into a new `run-NNN_<label>/` folder here + append a row to [`log.md`](log.md) |
| 6 | Agent | Confirm what was captured |
**Save-before-render** is the anchor: the file on disk at the JSON file folder = exactly what produced the render being described. No copy-paste, no version drift.
## Per-run snapshot contents
```
run-NNN_<label>/
├── workflow.json ← copy of LTX-23-audio_tests_v01.json at submission
├── relay_overrides.txt ← the LTX Director node's relay_overrides widget value
├── patch_state.txt ← MAIN SHA + upstream SHA + which patches were synced
└── notes.md ← maintainer's verbal feedback + agent's one-line interp
```
The `relay_overrides` value is also stored inside `workflow.json` (LTXDirector node widget), but the `.txt` copy makes it diff-friendly in the run folder.
## Trigger detection
No magic phrase. Whenever the maintainer describes a render result in chat, the agent treats it as a feedback signal and creates a new run folder. The agent confirms in chat what was captured; the maintainer can rename, delete, or correct.
To share a workflow file *without* logging it as a run, say "no log" or "don't log this" in the chat message.
## Run labels
Auto-generated from key knob state + feedback summary, e.g.:
- `run-001_no-patch_broken-baseline`
- `run-002_vstr10_no-change`
- `run-003_vstr25_some-motion-recovered`
- `run-004_vstr10_astr0.5_sync-partial`
The maintainer can override any label by stating it explicitly.
## Sync to live install — `sync-audio`
Before the first render with new patches, sync MAIN → live install:
```
python scripts/sync_investigation_patches.py 01_LTX23-audio-file-lipsync
```
Or just say "**sync-audio**" in chat and the agent runs it. Backs up the target's current state to `.bak.<YYYYMMDD>` before overwriting. See the investigation [README](../README.md) and [`upstream.json`](../upstream.json) for details.
## Asking "what's the loop?"
Point future-session-me at this file: `docs/investigations/01_LTX23-audio-file-lipsync/runs/LOOP.md`. Everything we agreed lives here.
## Glossary
See [`../../../reference/glossary.md`](../../../reference/glossary.md) for definitions of *JSON file folder*, *Loop*, *Run*, *Investigation patches*, *Investigation folder*.
docs/investigations/01_LTX23-audio-file-lipsync/upstream.json
-22
View File
@@ -1,22 +0,0 @@
{
"_purpose": "Source-of-truth pin for the upstream files this investigation modifies. Read by scripts/sync_investigation_patches.py.",
"_underscore_keys": "All keys starting with underscore are ignored by the loader. Use them for inline documentation.",
"investigation": "01_LTX23-audio-file-lipsync",
"trigger": "sync-audio",
"upstream": {
"repo": "https://github.com/WhatDreamsCost/WhatDreamsCost-ComfyUI",
"version": "v1.3.2",
"pinned_commit": "e81223a"
},
"_sync_target_env_doc": "Name of the env var (in .env) that holds the absolute path to the live install dir. Script errors cleanly if unset.",
"sync_target_env": "KOLOOK_WHATDREAMSCOST_PATH",
"_files_doc": "Filenames under patches/ to copy into the target dir. Each gets a per-day backup (<filename>.bak.<YYYYMMDD>) before overwrite.",
"files": [
"ltx_director.py",
"prompt_relay.py"
]
}
docs/reference/glossary.md
+61 -55
View File
@@ -73,65 +73,71 @@ Short, plain-English definitions for this repo workflow.
(`comfyui-easy-use/web_version/v1/js/getset.js`) and simulating the relevant
LiteGraph shape locally when the live `app.graph` object is not exposed.
## Investigation folder
- A subfolder under `docs/investigations/` that holds everything related to
one investigation — the narrative README, an `upstream.yaml` pin file,
modified upstream code under `patches/`, and per-render snapshots under
`runs/`.
- Numbered prefix for ordering and short topic identifier:
`docs/investigations/<NN>_<topic-slug>/`.
- Examples: `00_LTX23-base-1-step/`, `01_LTX23-audio-file-lipsync/`.
- Created when a piece of investigative work needs its own code patches and
iteration log; promoted to a proper `forks/<name>_koolook/` fork when the
patches validate.
## JSON file folder
- ComfyUI's workflow library directory where the maintainer overwrites a
single workflow file between iterations.
- Current example (investigation 01): `E:\_AI\portable\ComfyUI_windows_312\ComfyUI\user\default\workflows\LTX-23-audio_tests_v01.json`.
- Read-only from the agent's perspective — never edited by the agent
(preserves the ComfyUI canvas layout). Agent copies it into the
investigation's `runs/run-NNN_*/` folder on each render.
## Automation module
- A self-contained AI-managed ComfyUI iteration task. Lives under
`docs/automations/<grouping>/<task>/`, with the grouping typically being
the diffusion model (`LTX-2.3/`) and the task being a specific generation
goal (`base-1step/`, `audio-lipsync/`).
- Each module owns: a `README.md` (loop entry + workflow contract), a
`handoff-checklist.md` (5-minute bootstrap), a `findings.md` (locked-in
conclusions for that task), optionally a `backstory/` folder (the
narrative that produced the module), optionally a `runs/` folder for
in-repo run snapshots when the iteration touches code (not just widget
values).
- Modules are independent: own workflow JSON, own working folder, own
findings. The model grouping is only structural — it does not carry
shared docs unless duplication forces a future `_shared/` extraction.
- When a module needs to modify an upstream node's code, the modification
lives under `forks/<package>_koolook/` per the fork pattern (see
*Wrapper* and *Version Folder* above), and the module's README
cross-references the fork.
## Loop
- The save → render → feedback → snapshot cycle for an investigation.
- Step 1: maintainer edits and **saves** workflow in ComfyUI (overwrites the
JSON file folder target).
- Step 2: maintainer queues render.
- Step 3: maintainer reports result in chat.
- Step 4: agent snapshots current state (workflow + relay_overrides + active
patches) into a new run folder under
`docs/investigations/<NN>_<topic>/runs/`.
- Per-investigation protocol lives at
`docs/investigations/<NN>_<topic>/runs/LOOP.md`.
- The save → render → feedback → snapshot cycle inside an automation
module.
- Step 1: maintainer edits a knob in ComfyUI (or in a `forks/` source
file) and **saves** the workflow.
- Step 2: maintainer queues the render.
- Step 3: maintainer reports the result in chat.
- Step 4: agent snapshots current state into the module's
`runs/run-NNN_<label>/` (when the module ships in-repo runs) or just
appends to the working-folder `_AI/iterations.md` (when card-and-log
is all that's needed).
- Module-specific protocol lives at `<module>/runs/LOOP.md` for the
fork-touching modules; lighter modules document the loop inline in
their `README.md`.
## Run
- One iteration of the loop.
- Folder named `run-NNN_<short-knob-summary>_<short-result-tag>` inside
`docs/investigations/<NN>_<topic>/runs/`.
- Contains: `workflow.json` (snapshot), `relay_overrides.txt`,
`patch_state.txt` (MAIN SHA + upstream SHA + synced files), and `notes.md`
(maintainer feedback + agent interpretation).
- For modules that ship in-repo run snapshots, a `run-NNN_<short-knob-summary>_<short-result-tag>`
folder under `<module>/runs/` containing at minimum a `workflow.json`
copy and a `notes.md` (maintainer feedback + agent interpretation).
- For all modules, also a row in the per-project working-folder
`_AI/iterations.md` log (auto-rendered by `/make-card`).
## Investigation patches
- Modifications to a third-party node's source code, applied during the
investigation phase before a fork is formally registered under `forks/`.
- Source of truth lives in
`docs/investigations/<NN>_<topic>/patches/<file>` — versioned by git.
- Pushed to the live install via
`python scripts/sync_investigation_patches.py <NN>_<topic>`, which reads
`upstream.yaml` from the investigation folder for the target path and
file list.
- Each investigation declares its own short chat-trigger phrase via
`trigger:` in `upstream.yaml` (e.g. `sync-audio` for investigation 01).
- Promoted to a proper `forks/<name>_koolook/` fork once the approach
validates (≥ 3 confirming runs per the LTX-2.3 findings rule).
## Backstory
- The running narrative that produced an automation module — the
problem, the mechanism, the hypotheses considered, the rationale for
why the module exists. Lives at `<module>/backstory/<topic>.md`.
- Distinct from `findings.md` (locked-in conclusions) and
`runs/log.md` (rolling table of renders). The backstory is a
reference for *why* and *how it got here*; findings are *what's
true now*.
## `sync_investigation_patches.py`
- Helper script that copies an investigation's `patches/*` files to the
live install path declared in its `upstream.yaml`.
- Mirrors the pattern of `scripts/sync_to_dev.py` but for third-party node
modifications outside `KOLOOK_COMFYUI_DEV_PATH`.
- Creates a per-day backup of the target's current state
(`<filename>.bak.<YYYYMMDD>`) before overwriting.
- User-initiated only (never automatic) — same rule as `dev-sync`.
## Working folder
- The outside-the-repo per-project folder pointed at by
`KOLOOK_AUTOMATIONS_WORK_DIR` in `.env`. Holds workflow JSON,
rendered video, and the agent-managed `_AI/` subfolder with
`card.png` + append-only `iterations.md`.
- One working folder per project. Multiple automation modules can
point at the same working folder if they share workflow files;
more commonly, each module's iteration uses its own working folder.
## Card
- `_AI/card.png` — the per-render tracking visual rendered by the
`/make-card` skill (or `scripts/make_card.py` directly). Stable
filename, overwritten each run; the maintainer's NLE points at it
once.
- Renders from the latest workflow JSON in the working folder. Format
and palette mirror `docs/designs/snapshot-dialogs.html`. See
`docs/automations/CONVENTIONS.md` §3.
forks/THIRD_PARTY.md
+60
View File
@@ -146,6 +146,66 @@ below — GPL-3.0 §5(c) requires the whole work to be GPL-3.0.
pipeline and routes color-prepped frames directly to `vae.encode()`.
- **Last reviewed:** 2026-05-03
### WhatDreamsCost/WhatDreamsCost-ComfyUI — v1.3.2 LTX Director subset (Koolook fork)
- **Name:** WhatDreamsCost-ComfyUI
- **Upstream repo URL:** https://github.com/WhatDreamsCost/WhatDreamsCost-ComfyUI
- **Upstream commit/tag used:** `e81223a2add687555a6371e311b325880bf7c3c9`
(v1.3.2; the latest README-only update on the release commit chain).
- **License:** GPL-3.0 (verified 2026-05-28 from the upstream `LICENSE`
file — *GNU General Public License Version 3, 29 June 2007*).
- **Local path(s):** [`forks/whatdreamscost_koolook/versions/v1_3_2/`](whatdreamscost_koolook/versions/v1_3_2/)
- **What changed locally:**
- **`ltx_director.py` (modified).** The `LTXDirector` node gains a new
optional `relay_overrides` multiline-string input. The maintainer pastes
a JSON dict of Prompt-Relay knobs (`video_strength`,
`video_window_scale`, `audio_strength`, `audio_window_scale`,
`audio_epsilon`) directly into the canvas widget; underscore-prefixed
keys are ignored so the field can carry inline JSON comments. The
parsed overrides flow through to `build_segments` per render — no
disk file, no env var, the values live entirely inside the workflow
JSON. Empty field → upstream Prompt-Relay defaults.
- **`prompt_relay.py` (modified).** `build_segments` now computes σ
per-segment using the Prompt-Relay paper formula
`σ = (L − w_eff) / (2 · √ln(1/ε))` instead of the upstream's
length-independent `σ = 1/ln(1/ε)` (≈ 0.1448 at ε=0.001). The new
formula calibrates the penalty so it hits threshold ε exactly at
the segment boundary regardless of segment length. A
`SIGMA_FALLBACK = 0.1448` preserves the prior constant for the
degenerate `L ≤ w_eff` corner. Per-segment σ is logged.
- **`patches.py` (verbatim vendored).** `ltx_director.py` imports
`detect_model_type` and `apply_patches` from this file. Carried
along unmodified so the fork is self-contained — same precedent as
`forks/radiance_koolook/versions/v2_3_3/color_helpers.py`.
- **What was *not* forked:** The companion `LTXDirectorGuide` node, plus
`LTXKeyframer`, `MultiImageLoader`, `LTXSequencer`,
`SpeechLengthCalculator`, `LoadAudioUI`, and `LoadVideoUI`. These are
unmodified upstream nodes; users need an installed copy of upstream
WhatDreamsCost-ComfyUI to use them. The Koolook variant of LTXDirector
is namespaced (`LTXDirector__koolook_v1_3_2`) so the two coexist in
the node picker.
- **Why namespaced (`__koolook_v1_3_2`):** per the convention in
`forks/README.md`, ports of upstream-named classes carry the version
suffix to avoid colliding with an installed copy of the same upstream
package. Graphs wiring the Koolook variant pick
`LTX Director (Koolook v1.3.2)` from the picker; the unsuffixed
`LTX Director` continues to refer to the user's upstream install.
- **Why a partial fork rather than a subclass:** the two upstream
modifications cross multiple call sites inside the `LTXDirector` body
and inside the `build_segments` function in `prompt_relay.py`. A
subclass override would have to either reach into upstream internals
(brittle) or duplicate the bodies (more code than the verbatim fork).
The partial fork was the smaller surface.
- **Maintenance loop:** the audio-lipsync iteration that drives changes
here lives at
[`../docs/automations/LTX-2.3/audio-lipsync/`](../docs/automations/LTX-2.3/audio-lipsync/).
Patches were originally synced into the user's upstream install via
`scripts/sync_investigation_patches.py`; with v1.3.2 promoted to a
fork, the iteration uses the standard
[`../scripts/sync_to_dev.py`](../scripts/sync_to_dev.py) flow and the
workflow JSON references the Koolook-suffixed node ID.
- **Last reviewed:** 2026-05-28
## De-vendored upstream code (untracked in v0.1.4 / v0.1.5)
Six third-party trees were untracked from MAIN's git index in the
forks/forks_manifest.yaml
+38 -1
View File
@@ -1,5 +1,5 @@
version: 1
updated_at: "2026-05-04"
updated_at: "2026-05-28"
external_forks:
root_relative_to_main: "../ComfyUI-Forks"
root_env_var: "KOLOOK_FORKS_DIR"
@@ -39,6 +39,43 @@ entries:
license_verified_at: "2026-05-03"
notes: "Mirrors only the RadianceVAE4KEncode/Decode interface surface — Koolook ships a slimmer rank-agnostic reimplementation that handles image sequences (video frames) without the upstream's 4K cosine-blend tile engine, which conflicts with video VAEs (Wan 2.2 etc.) that already do their own temporal stitching."
- id: "whatdreamscost_koolook_v1_3_2_ltx_director"
name: "WhatDreamsCost Koolook v1.3.2 LTX Director subset"
source_repo: "https://github.com/WhatDreamsCost/WhatDreamsCost-ComfyUI.git"
source_ref: "v1.3.2"
pinned_commit: "e81223a2add687555a6371e311b325880bf7c3c9"
external_checkout:
relative_path_from_forks_root: "whatdreamscost-v1.3.2-koolook"
upstream_tracking_path_from_forks_root: "whatdreamscost-main-upstream"
default_branch: "main"
local_paths:
- "forks/whatdreamscost_koolook/__init__.py"
- "forks/whatdreamscost_koolook/versions/v1_3_2/__init__.py"
- "forks/whatdreamscost_koolook/versions/v1_3_2/UPSTREAM_PIN.yaml"
- "forks/whatdreamscost_koolook/versions/v1_3_2/ltx_director.py"
- "forks/whatdreamscost_koolook/versions/v1_3_2/prompt_relay.py"
- "forks/whatdreamscost_koolook/versions/v1_3_2/patches.py"
status: "modified"
sync_state: "ltx-director-modified-plus-deps"
license: "GPL-3.0"
license_verified_at: "2026-05-28"
exposed_node_ids:
- "LTXDirector__koolook_v1_3_2"
notes: |
Ships a Koolook variant of upstream's `LTXDirector` only — the companion
`LTXDirectorGuide` is consumed from the user's upstream install via the
`guide_data -> latent` wiring. The Koolook variant adds:
1) a `relay_overrides` multiline JSON input on the LTX Director canvas
widget so per-render Prompt-Relay knob overrides live inside the
workflow JSON (no disk file / no env var);
2) per-segment sigma in `build_segments` matching the Prompt-Relay paper
formula σ = (L − w_eff) / (2 · sqrt(ln(1/ε))) instead of upstream's
length-independent constant (≈ 0.1448 at ε=0.001).
`patches.py` is vendored verbatim from upstream e81223a — `ltx_director.py`
imports `detect_model_type` and `apply_patches` from it. The maintainer-
facing iteration loop driving changes here lives at
`docs/automations/LTX-2.3/audio-lipsync/`.
- id: "easy_resize_inspiration"
name: "Easy Resize (Koolook reimplementation, inspired by KJ Nodes)"
source_repo: "https://github.com/kijai/ComfyUI-KJNodes"
forks/whatdreamscost_koolook/README.md
+21
View File
@@ -0,0 +1,21 @@
# WhatDreamsCost Koolook
Versioned local fork of WhatDreamsCost-ComfyUI nodes (LTX Director + supporting
Prompt-Relay implementation).
- Package entrypoint: `forks/whatdreamscost_koolook/__init__.py`
- Version folders: `forks/whatdreamscost_koolook/versions/<version>/`
- Each version folder keeps:
- modified node files tracked in MAIN
- vendored unmodified upstream deps the modified files import
- `UPSTREAM_PIN.yaml` for parity with pinned external raw checkout
Current active version:
- `v1_3_2` — see [`forks/THIRD_PARTY.md`](../THIRD_PARTY.md) for upstream
attribution and what changed locally. Tracks
[WhatDreamsCost/WhatDreamsCost-ComfyUI](https://github.com/WhatDreamsCost/WhatDreamsCost-ComfyUI)
at commit `e81223a`. License: GPL-3.0 (compatible with this pack).
The maintainer-facing iteration loop that drives changes here lives at
[`docs/automations/LTX-2.3/audio-lipsync/`](../../docs/automations/LTX-2.3/audio-lipsync/).
forks/whatdreamscost_koolook/__init__.py
+19
View File
@@ -0,0 +1,19 @@
"""WhatDreamsCost Koolook package entrypoint for versioned node sets.
Each `versions/vX_Y_Z/` subpackage owns a namespaced slice of nodes from a
pinned upstream commit. This entrypoint merges all version mappings into a
single dict for the root loader.
Adding a new version: see `.claude/skills/add-external-fork/SKILL.md`.
"""
from .versions.v1_3_2 import NODE_CLASS_MAPPINGS as V1_3_2_NODE_CLASS_MAPPINGS
from .versions.v1_3_2 import NODE_DISPLAY_NAME_MAPPINGS as V1_3_2_NODE_DISPLAY_NAME_MAPPINGS
NODE_CLASS_MAPPINGS = {}
NODE_DISPLAY_NAME_MAPPINGS = {}
NODE_CLASS_MAPPINGS.update(V1_3_2_NODE_CLASS_MAPPINGS)
NODE_DISPLAY_NAME_MAPPINGS.update(V1_3_2_NODE_DISPLAY_NAME_MAPPINGS)
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
forks/whatdreamscost_koolook/versions/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Versioned WhatDreamsCost Koolook node sets."""
forks/whatdreamscost_koolook/versions/v1_3_2/UPSTREAM_PIN.yaml
+22
View File
@@ -0,0 +1,22 @@
source_repo: "https://github.com/WhatDreamsCost/WhatDreamsCost-ComfyUI.git"
source_ref: "v1.3.2"
pinned_commit: "e81223a2add687555a6371e311b325880bf7c3c9"
external_checkout_relative_to_forks_root: "whatdreamscost-v1.3.2-koolook"
upstream_tracking_relative_to_forks_root: "whatdreamscost-main-upstream"
external_work_branch: "main"
license: "GPL-3.0"
license_verified_at: "2026-05-28"
notes: |
This version folder ships:
- ltx_director.py (MODIFIED — adds `relay_overrides` multiline input)
- prompt_relay.py (MODIFIED — per-segment sigma per Prompt-Relay paper)
- patches.py (UNMODIFIED — vendored verbatim; ltx_director.py
imports detect_model_type / apply_patches from it)
Only one node is registered: `LTXDirector` (suffixed `__koolook_v1_3_2`
to avoid colliding with installed upstream WhatDreamsCost-ComfyUI). The
companion `LTXDirectorGuide` is unmodified and consumed from the user's
upstream install — graphs wire
`LTXDirector__koolook_v1_3_2.guide_data -> LTXDirectorGuide.latent`.
See forks/THIRD_PARTY.md for the full attribution + diff against upstream.
forks/whatdreamscost_koolook/versions/v1_3_2/__init__.py
+64
View File
@@ -0,0 +1,64 @@
# SPDX-License-Identifier: GPL-3.0-or-later
"""
WhatDreamsCost Koolook v1.3.2 mappings.
Loads the modified `LTXDirector` node (with the added `relay_overrides`
multiline input + per-segment-sigma Prompt-Relay implementation) and
namespaces its ID with the suffix `__koolook_v1_3_2` to avoid collisions
with an installed copy of upstream WhatDreamsCost-ComfyUI v1.3.2.
The companion `LTXDirectorGuide` node is intentionally NOT included here:
it is upstream code we did not modify, and graphs are expected to wire
`LTXDirector__koolook_v1_3_2.guide_data` into the user's upstream
`LTXDirectorGuide` (or our future Koolook variant).
See `forks/THIRD_PARTY.md` and `UPSTREAM_PIN.yaml` for upstream
attribution and the full change log.
"""
from .ltx_director import LTXDirector
NAMESPACE_SUFFIX = "__koolook_v1_3_2"
DISPLAY_SUFFIX = " (Koolook v1.3.2)"
# IDs in this set are exposed verbatim (no suffix). Empty for this version —
# the only registered node is LTXDirector, whose plain ID collides with the
# upstream class of the same name. Future Koolook-original additions here
# (with no upstream counterpart) can opt out via this set, same as
# `forks/radiance_koolook/versions/v2_3_3/__init__.py`.
SKIP_VERSION_SUFFIX: set[str] = set()
_BASE_CLASS_MAPPINGS = {
"LTXDirector": LTXDirector,
}
_BASE_DISPLAY_MAPPINGS = {
"LTXDirector": "LTX Director",
}
def _namespace_mappings(class_mappings, display_mappings):
namespaced_classes = {}
namespaced_display = {}
for node_id, node_cls in class_mappings.items():
if node_id in SKIP_VERSION_SUFFIX:
namespaced_id = node_id
namespaced_display[namespaced_id] = display_mappings.get(node_id, node_id)
else:
namespaced_id = f"{node_id}{NAMESPACE_SUFFIX}"
base_display = display_mappings.get(node_id, node_id)
namespaced_display[namespaced_id] = f"{base_display}{DISPLAY_SUFFIX}"
namespaced_classes[namespaced_id] = node_cls
return namespaced_classes, namespaced_display
NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS = _namespace_mappings(
_BASE_CLASS_MAPPINGS,
_BASE_DISPLAY_MAPPINGS,
)
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
docs/investigations/01_LTX23-audio-file-lipsync/patches/ltx_director.py → forks/whatdreamscost_koolook/versions/v1_3_2/ltx_director.py
View File
forks/whatdreamscost_koolook/versions/v1_3_2/patches.py
+167
View File
@@ -0,0 +1,167 @@
import types
import torch
import comfy.ldm.modules.attention
def _masked_attention(q, k, v, heads, mask, transformer_options={}, **kwargs):
# Bypass wrap_attn (sage/etc may ignore masks) by calling attention_pytorch directly.
return comfy.ldm.modules.attention.attention_pytorch(
q, k, v, heads, mask=mask,
_inside_attn_wrapper=True,
transformer_options=transformer_options,
**kwargs,
)
def _wan_t2v_forward(self, mask_fn, x, context, transformer_options={}, **kwargs):
q = self.norm_q(self.q(x))
k = self.norm_k(self.k(context))
v = self.v(context)
mask = mask_fn(q, k, transformer_options)
if mask is not None:
x = _masked_attention(q, k, v, heads=self.num_heads, mask=mask,
transformer_options=transformer_options)
else:
x = comfy.ldm.modules.attention.optimized_attention(
q, k, v, heads=self.num_heads, transformer_options=transformer_options,
)
return self.o(x)
def _wan_i2v_forward(self, mask_fn, x, context, context_img_len, transformer_options={}, **kwargs):
context_img = context[:, :context_img_len]
context_text = context[:, context_img_len:]
q = self.norm_q(self.q(x))
k_img = self.norm_k_img(self.k_img(context_img))
v_img = self.v_img(context_img)
img_x = comfy.ldm.modules.attention.optimized_attention(
q, k_img, v_img, heads=self.num_heads, transformer_options=transformer_options,
)
k = self.norm_k(self.k(context_text))
v = self.v(context_text)
mask = mask_fn(q, k, transformer_options)
if mask is not None:
x = _masked_attention(q, k, v, heads=self.num_heads, mask=mask,
transformer_options=transformer_options)
else:
x = comfy.ldm.modules.attention.optimized_attention(
q, k, v, heads=self.num_heads, transformer_options=transformer_options,
)
return self.o(x + img_x)
def _ltx_forward(self, mask_fn, x, context=None, mask=None, pe=None, k_pe=None, transformer_options={}):
from comfy.ldm.lightricks.model import apply_rotary_emb
is_self_attn = context is None
context = x if is_self_attn else context
q = self.q_norm(self.to_q(x))
k = self.k_norm(self.to_k(context))
v = self.to_v(context)
if pe is not None:
q = apply_rotary_emb(q, pe)
k = apply_rotary_emb(k, pe if k_pe is None else k_pe)
if not is_self_attn:
temporal_mask = mask_fn(q, k, transformer_options)
if temporal_mask is not None:
mask = temporal_mask if mask is None else mask + temporal_mask
if mask is None:
out = comfy.ldm.modules.attention.optimized_attention(
q, k, v, self.heads, attn_precision=self.attn_precision,
transformer_options=transformer_options,
)
else:
out = _masked_attention(q, k, v, self.heads, mask=mask,
attn_precision=self.attn_precision,
transformer_options=transformer_options)
if self.to_gate_logits is not None:
gate_logits = self.to_gate_logits(x)
b, t, _ = out.shape
out = out.view(b, t, self.heads, self.dim_head)
out = out * (2.0 * torch.sigmoid(gate_logits)).unsqueeze(-1)
out = out.view(b, t, self.heads * self.dim_head)
return self.to_out(out)
class _CrossAttnPatch:
"""Descriptor that binds (impl, mask_fn) as a method onto a cross-attn module."""
def __init__(self, impl, mask_fn):
self.impl = impl
self.mask_fn = mask_fn
def __get__(self, obj, objtype=None):
impl, mask_fn = self.impl, self.mask_fn
def wrapped(self_module, *args, **kwargs):
return impl(self_module, mask_fn, *args, **kwargs)
return types.MethodType(wrapped, obj)
def detect_model_type(model):
"""Return (arch, patch_size, temporal_stride) for latent geometry.
temporal_stride is the VAE's pixel→latent temporal compression factor,
used to convert user-facing pixel frame counts to latent frames.
"""
diff_model = model.model.diffusion_model
if hasattr(diff_model, "patch_size") and not hasattr(diff_model, "patchifier"):
return "wan", tuple(diff_model.patch_size), 4
if hasattr(diff_model, "patchifier"):
return "ltx", (1, 1, 1), int(diff_model.vae_scale_factors[0])
raise ValueError(
f"Unsupported model type: {type(diff_model).__name__}. "
f"Currently supports Wan and LTX models."
)
def _check_unpatched(model_clone, key):
if key in getattr(model_clone, "object_patches", {}):
raise RuntimeError(
f"PromptRelay: cross-attention forward at '{key}' is already patched by "
"another node (e.g. KJNodes NAG). Stacking is not supported — remove the "
"conflicting node."
)
def apply_patches(model_clone, arch, mask_fn):
diffusion_model = model_clone.get_model_object("diffusion_model")
if arch == "wan":
from comfy.ldm.wan.model import WanI2VCrossAttention
for idx, block in enumerate(diffusion_model.blocks):
key = f"diffusion_model.blocks.{idx}.cross_attn.forward"
_check_unpatched(model_clone, key)
cross_attn = block.cross_attn
impl = _wan_i2v_forward if isinstance(cross_attn, WanI2VCrossAttention) else _wan_t2v_forward
model_clone.add_object_patch(key, _CrossAttnPatch(impl, mask_fn).__get__(cross_attn, cross_attn.__class__))
return
if arch == "ltx":
for idx, block in enumerate(diffusion_model.transformer_blocks):
for attr in ("attn2", "audio_attn2"):
module = getattr(block, attr, None)
if module is None:
continue
key = f"diffusion_model.transformer_blocks.{idx}.{attr}.forward"
_check_unpatched(model_clone, key)
model_clone.add_object_patch(key, _CrossAttnPatch(_ltx_forward, mask_fn).__get__(module, module.__class__))
return
raise ValueError(f"Unknown model arch: {arch}")
docs/investigations/01_LTX23-audio-file-lipsync/patches/prompt_relay.py → forks/whatdreamscost_koolook/versions/v1_3_2/prompt_relay.py
View File
pyproject.toml
+1 -1
View File
@@ -35,7 +35,7 @@ includes = []
# "requires-comfyui" = ">=1.0.0" # ComfyUI version compatibility
[tool.ruff]
extend-exclude = [".venv", "forks", "upscaler_FIX", "nuke_CAM_exporter", "docs/investigations"]
extend-exclude = [".venv", "forks", "upscaler_FIX", "nuke_CAM_exporter"]
[tool.setuptools]
py-modules = [
scripts/sync_investigation_patches.py
-182
View File
@@ -1,182 +0,0 @@
#!/usr/bin/env python3
"""sync_investigation_patches.py — push modified upstream files from MAIN to a live install.
Mirrors the `scripts/sync_to_dev.py` pattern but for third-party node modifications
under `docs/investigations/<NN>_<topic>/patches/`. MAIN is the source of truth;
the live install is a destination only.
Usage:
python scripts/sync_investigation_patches.py <investigation-folder> [--dry-run]
Example:
python scripts/sync_investigation_patches.py 01_LTX23-audio-file-lipsync
The investigation folder must contain `upstream.yaml` declaring:
- sync_target_env: name of the env var holding the absolute target path
- files: list of filenames to copy from patches/ into the target
Errors cleanly when the env var is unset, the target path is missing,
or upstream.yaml is malformed.
Backup convention: <filename>.bak.<YYYYMMDD> created in the target dir
before the first overwrite of the day. Subsequent same-day syncs do not
re-create the backup (preserves the day's earliest pre-sync state).
"""
from __future__ import annotations
import argparse
import datetime as dt
import os
import shutil
import subprocess
import sys
from pathlib import Path
import json
REPO_ROOT = Path(__file__).resolve().parent.parent
INVESTIGATIONS_DIR = REPO_ROOT / "docs" / "investigations"
def _load_dotenv() -> None:
"""Load .env from REPO_ROOT, falling back to the main worktree's .env when
REPO_ROOT is itself a git worktree (e.g. running from .claude/worktrees/<name>/).
First-set-wins per env-var.
"""
candidates = [REPO_ROOT / ".env"]
try:
common_dir = subprocess.check_output(
["git", "rev-parse", "--git-common-dir"],
cwd=str(REPO_ROOT),
text=True,
stderr=subprocess.DEVNULL,
).strip()
if common_dir:
main_root = Path(common_dir).resolve().parent
if main_root.resolve() != REPO_ROOT.resolve():
candidates.append(main_root / ".env")
except (subprocess.CalledProcessError, FileNotFoundError):
pass
for env_path in candidates:
if not env_path.is_file():
continue
for line in env_path.read_text(encoding="utf-8").splitlines():
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, _, val = line.partition("=")
os.environ.setdefault(key.strip(), val.strip().strip('"').strip("'"))
def _short_sha(path: Path) -> str:
try:
out = subprocess.check_output(
["git", "rev-parse", "--short", "HEAD"],
cwd=str(path),
text=True,
stderr=subprocess.DEVNULL,
)
return out.strip()
except (subprocess.CalledProcessError, FileNotFoundError):
return "no-git"
def _fail(msg: str) -> int:
print(f"ERROR: {msg}", file=sys.stderr)
return 1
def main() -> int:
_load_dotenv()
parser = argparse.ArgumentParser(
description="Sync modified upstream files from a Koolook investigation folder to a live ComfyUI install."
)
parser.add_argument(
"folder",
help="Investigation folder name under docs/investigations/ (e.g. 01_LTX23-audio-file-lipsync)",
)
parser.add_argument("--dry-run", action="store_true", help="Preview without copying or backing up")
args = parser.parse_args()
inv_dir = INVESTIGATIONS_DIR / args.folder
if not inv_dir.is_dir():
return _fail(f"Investigation folder not found: {inv_dir}")
upstream_json = inv_dir / "upstream.json"
if not upstream_json.is_file():
return _fail(f"Missing upstream.json in {inv_dir}")
try:
spec = json.loads(upstream_json.read_text(encoding="utf-8"))
except json.JSONDecodeError as e:
return _fail(f"upstream.json is not valid JSON: {e}")
if not isinstance(spec, dict):
return _fail("upstream.json must be a top-level object")
env_var = spec.get("sync_target_env")
if not env_var:
return _fail("upstream.json is missing `sync_target_env`")
target_root = os.environ.get(env_var)
if not target_root:
return _fail(f"${env_var} is unset. Add it to .env (see .env.example).")
target_root_p = Path(target_root)
if not target_root_p.is_dir():
return _fail(f"Target directory does not exist: {target_root_p}")
files = spec.get("files") or []
if not files:
return _fail("upstream.json has no `files` array")
patches_dir = inv_dir / "patches"
if not patches_dir.is_dir():
return _fail(f"Missing patches/ directory in {inv_dir}")
# Two-line header — first line is parsed by chat reports verbatim
main_sha = _short_sha(REPO_ROOT)
trigger = spec.get("trigger") or args.folder
print(f"{main_sha} - {args.folder}")
print(f"sync-investigation: {trigger}")
today = dt.date.today().strftime("%Y%m%d")
copied = 0
skipped = 0
for fname in files:
src = patches_dir / fname
dst = target_root_p / fname
if not src.is_file():
print(f" SKIP {fname}: not in patches/")
skipped += 1
continue
if not dst.is_file():
print(f" SKIP {fname}: target does not exist at destination")
skipped += 1
continue
bak = dst.with_suffix(dst.suffix + f".bak.{today}")
if args.dry_run:
print(f" DRY {fname} -> {dst}")
continue
if not bak.exists():
shutil.copy2(dst, bak)
shutil.copy2(src, dst)
print(f" OK {fname} -> {dst}")
copied += 1
if args.dry_run:
print(f"(dry-run) would have synced {len(files)} files; skipped {skipped}")
else:
print(f"synced {copied} files; skipped {skipped}")
return 0
if __name__ == "__main__":
sys.exit(main())