From e7f3ab4fe8aff95d291be3c204f8eada5c14a423 Mon Sep 17 00:00:00 2001 From: malkuthro <166633396+malkuthro@users.noreply.github.com> Date: Thu, 28 May 2026 21:45:16 +0200 Subject: [PATCH] refactor: per-task automation modules + WhatDreamsCost Koolook fork MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pivot docs/automations/ from "one folder per model" to "one folder per automation module" so the same diffusion model can host several independent tasks side by side. Replace the parallel docs/investigations/ tree with a fork of WhatDreamsCost-ComfyUI under forks/. Automation modules (new layout) - docs/automations/LTX-2.3/base-1step/ the existing single-stage automation, moved from docs/automations/LTX-2.3/ with its README, findings, and handoff-checklist rescoped to the task. The 4k-transitions backstory that produced the module moves here as backstory/4k-transitions.md. The earlier docs/investigations/experiments/ card PNGs/HTML land in example-cards/. - docs/automations/LTX-2.3/audio-lipsync/ the audio-file lip-sync module previously living under docs/investigations/01_…/. README, handoff, findings stub, backstory/, and runs/{LOOP,log}.md, all referencing the new fork. - docs/automations/README.md + CHEATSHEET.md updated to describe the module-per-task pivot and the new tree shape. - docs/automations/CONVENTIONS.md unchanged (cross-cutting loop contract still holds). WhatDreamsCost Koolook fork (new) - forks/whatdreamscost_koolook/versions/v1_3_2/ UPSTREAM_PIN.yaml pinned to upstream e81223a (v1.3.2), GPL-3.0 verified 2026-05-28. ltx_director.py MODIFIED — adds the relay_overrides multiline JSON widget on LTXDirector. Moved from docs/investigations/01_…/patches/. prompt_relay.py MODIFIED — per-segment σ per the Prompt-Relay paper formula σ = (L − w_eff) / (2 · √ln(1/ε)). Moved from docs/investigations/01_…/patches/. patches.py UNMODIFIED — vendored verbatim from upstream because ltx_director.py imports detect_model_type and apply_patches from it (same precedent as radiance_koolook's color_helpers.py). - Registered as LTXDirector__koolook_v1_3_2 ("LTX Director (Koolook v1.3.2)") so it coexists with an installed copy of upstream WhatDreamsCost-ComfyUI. Wired into the root __init__.py NODE_CLASS_MAPPINGS merge. - forks/forks_manifest.yaml + forks/THIRD_PARTY.md updated with the new entry (attribution, change log, license verified date, exposed node IDs). Removed - docs/investigations/ entire tree retired; its contents either moved to automation modules or to forks/. - scripts/sync_investigation_patches.py no longer needed — fork code lives in the Koolook tree and is synced via the standard dev-sync flow. - KOLOOK_WHATDREAMSCOST_PATH in .env.example same reason. - docs/investigations in pyproject.toml's ruff extend-exclude added in 96bf8ad to silence lint on vendored patches; no longer needed now that vendored code is under forks/ (already excluded). Docs - docs/reference/glossary.md Investigation folder, Investigation patches, JSON file folder, and the sync_investigation_patches.py entries replaced by Automation module, Backstory, Working folder, and Card. Loop and Run rewritten to point at the new structure. - CHANGELOG.md [Unreleased] describes the restructure + the new fork. Verification - ruff check . → all checks passed - tools/preflight_release.py --skip manager-meta → 42 files parsed; 11 unique node IDs collected including LTXDirector; VAE dispatch branches all OK; fixture workflows still reference all registered Koolook IDs. - pytest → 173 passed, 0 failed. Workflow JSON heads-up The active audio-lipsync workflow LTX-23-audio_tests_v01.json continues to reference the upstream LTXDirector ID. To pick up the new fork's behaviour, swap that node to "LTX Director (Koolook v1.3.2)" (registered ID LTXDirector__koolook_v1_3_2) — sockets and wiring are preserved. The relay_overrides widget appears at the bottom of the new node. Documented in the audio-lipsync README's "Workflow JSON — Koolook node ID" section. Co-Authored-By: Claude Opus 4.7 (1M context) --- .env.example | 14 -- CHANGELOG.md | 40 ++++ __init__.py | 5 + docs/automations/CHEATSHEET.md | 20 +- .../LTX-2.3/audio-lipsync/README.md | 123 ++++++++++++ .../backstory/audio-lipsync-rationale.md | 96 +++++++++ .../LTX-2.3/audio-lipsync/findings.md | 26 +++ .../audio-lipsync/handoff-checklist.md | 70 +++++++ .../LTX-2.3/audio-lipsync/runs/LOOP.md | 68 +++++++ .../LTX-2.3/audio-lipsync}/runs/log.md | 0 .../LTX-2.3/{ => base-1step}/README.md | 40 ++-- .../base-1step/backstory/4k-transitions.md} | 10 +- .../base-1step/example-cards}/baseline.html | 0 .../example-cards}/combined-card.html | 0 .../example-cards}/run-001-card.html | 0 .../example-cards}/run-001-card.png | Bin .../example-cards}/run-v04-card.png | Bin .../LTX-2.3/{ => base-1step}/findings.md | 18 +- .../{ => base-1step}/handoff-checklist.md | 12 +- docs/automations/README.md | 85 ++++++-- docs/automations/index.html | 2 +- .../01_LTX23-audio-file-lipsync/README.md | 103 ---------- .../01_LTX23-audio-file-lipsync/runs/LOOP.md | 67 ------- .../01_LTX23-audio-file-lipsync/upstream.json | 22 --- docs/reference/glossary.md | 116 +++++------ forks/THIRD_PARTY.md | 60 ++++++ forks/forks_manifest.yaml | 39 +++- forks/whatdreamscost_koolook/README.md | 21 ++ forks/whatdreamscost_koolook/__init__.py | 19 ++ .../versions/__init__.py | 1 + .../versions/v1_3_2/UPSTREAM_PIN.yaml | 22 +++ .../versions/v1_3_2/__init__.py | 64 ++++++ .../versions/v1_3_2}/ltx_director.py | 0 .../versions/v1_3_2/patches.py | 167 ++++++++++++++++ .../versions/v1_3_2}/prompt_relay.py | 0 pyproject.toml | 2 +- scripts/sync_investigation_patches.py | 182 ------------------ 37 files changed, 1015 insertions(+), 499 deletions(-) create mode 100644 docs/automations/LTX-2.3/audio-lipsync/README.md create mode 100644 docs/automations/LTX-2.3/audio-lipsync/backstory/audio-lipsync-rationale.md create mode 100644 docs/automations/LTX-2.3/audio-lipsync/findings.md create mode 100644 docs/automations/LTX-2.3/audio-lipsync/handoff-checklist.md create mode 100644 docs/automations/LTX-2.3/audio-lipsync/runs/LOOP.md rename docs/{investigations/01_LTX23-audio-file-lipsync => automations/LTX-2.3/audio-lipsync}/runs/log.md (100%) rename docs/automations/LTX-2.3/{ => base-1step}/README.md (65%) rename docs/{investigations/00_LTX23-base-1-step/README.md => automations/LTX-2.3/base-1step/backstory/4k-transitions.md} (97%) rename docs/{investigations/experiments => automations/LTX-2.3/base-1step/example-cards}/baseline.html (100%) rename docs/{investigations/experiments => automations/LTX-2.3/base-1step/example-cards}/combined-card.html (100%) rename docs/{investigations/experiments => automations/LTX-2.3/base-1step/example-cards}/run-001-card.html (100%) rename docs/{investigations/experiments => automations/LTX-2.3/base-1step/example-cards}/run-001-card.png (100%) rename docs/{investigations/experiments => automations/LTX-2.3/base-1step/example-cards}/run-v04-card.png (100%) rename docs/automations/LTX-2.3/{ => base-1step}/findings.md (90%) rename docs/automations/LTX-2.3/{ => base-1step}/handoff-checklist.md (85%) delete mode 100644 docs/investigations/01_LTX23-audio-file-lipsync/README.md delete mode 100644 docs/investigations/01_LTX23-audio-file-lipsync/runs/LOOP.md delete mode 100644 docs/investigations/01_LTX23-audio-file-lipsync/upstream.json create mode 100644 forks/whatdreamscost_koolook/README.md create mode 100644 forks/whatdreamscost_koolook/__init__.py create mode 100644 forks/whatdreamscost_koolook/versions/__init__.py create mode 100644 forks/whatdreamscost_koolook/versions/v1_3_2/UPSTREAM_PIN.yaml create mode 100644 forks/whatdreamscost_koolook/versions/v1_3_2/__init__.py rename {docs/investigations/01_LTX23-audio-file-lipsync/patches => forks/whatdreamscost_koolook/versions/v1_3_2}/ltx_director.py (100%) create mode 100644 forks/whatdreamscost_koolook/versions/v1_3_2/patches.py rename {docs/investigations/01_LTX23-audio-file-lipsync/patches => forks/whatdreamscost_koolook/versions/v1_3_2}/prompt_relay.py (100%) delete mode 100644 scripts/sync_investigation_patches.py diff --git a/.env.example b/.env.example index e28928a..6a801bf 100644 --- a/.env.example +++ b/.env.example @@ -106,17 +106,3 @@ KOLOOK_AUTOMATIONS_WORK_DIR= # KOLOOK_COMFYUI_DEV_PATH (assumes /user/comfyui.log). Set this # only if your install writes logs elsewhere. KOLOOK_COMFYUI_LOG= - -# --- Investigation patch sync targets --------------------------------------- -# Per-investigation absolute path to a third-party node's live install dir. -# Used by scripts/sync_investigation_patches.py to push patched files from -# docs/investigations/_/patches/ into the running ComfyUI. -# Each upstream gets its own env var, declared in the investigation's -# upstream.yaml under sync_target_env. - -# WhatDreamsCost-ComfyUI install path. Required for investigation 01 -# (LTX23-audio-file-lipsync). Forward slashes OK on Windows. -# -# Example: -# Windows: E:/_AI/portable/ComfyUI_windows_312/ComfyUI/custom_nodes/WhatDreamsCost-ComfyUI -KOLOOK_WHATDREAMSCOST_PATH= diff --git a/CHANGELOG.md b/CHANGELOG.md index af31d72..6a938c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,46 @@ The format is inspired by Keep a Changelog and SemVer. ## [Unreleased] +### Changed +- **Automations restructured around modules instead of models.** + `docs/automations/` now pivots on one folder per *task* (each its own + workflow + iteration loop + findings) rather than one folder per model. + The existing `LTX-2.3/{README,findings,handoff-checklist}.md` content + moved into `LTX-2.3/base-1step/` (single-stage render task) and the new + sibling `LTX-2.3/audio-lipsync/` (audio-file lip-sync task) was created. + `docs/automations/{README,CHEATSHEET}.md` updated to reflect the new + layout; `docs/automations/CONVENTIONS.md` unchanged. +- **`docs/investigations/` removed.** Its two children migrated to their + natural homes: narrative content into the new automation modules under + `docs/automations/LTX-2.3/{base-1step,audio-lipsync}/`; the modified + upstream Python code into the new `forks/whatdreamscost_koolook/` + fork (see *Added*). The retired `scripts/sync_investigation_patches.py` + and its `KOLOOK_WHATDREAMSCOST_PATH` env-var section in `.env.example` + go with it — iteration now uses the standard `dev-sync` flow against + the in-repo fork. +- **Glossary updated.** The *Investigation folder*, *Investigation patches*, + *JSON file folder*, and `sync_investigation_patches.py` entries in + [`docs/reference/glossary.md`](docs/reference/glossary.md) are replaced + by *Automation module*, *Backstory*, *Working folder*, and *Card*; the + *Loop* and *Run* entries are rewritten to point at the new structure. + +### Added +- **`forks/whatdreamscost_koolook/v1_3_2/` — Koolook fork of WhatDreamsCost-ComfyUI's + `LTXDirector`** (upstream `e81223a`, GPL-3.0). Two upstream files modified + (`ltx_director.py` adds a `relay_overrides` multiline-JSON widget; + `prompt_relay.py` uses the Prompt-Relay paper's per-segment σ formula + instead of upstream's length-independent constant) plus the + unmodified `patches.py` vendored verbatim because the modified file + imports from it. Registered as + `LTXDirector__koolook_v1_3_2` (display name *"LTX Director (Koolook + v1.3.2)"*) so it coexists with an installed copy of upstream + WhatDreamsCost-ComfyUI in the node picker. Drives the + [`docs/automations/LTX-2.3/audio-lipsync/`](docs/automations/LTX-2.3/audio-lipsync/) + iteration loop. License attribution + change log in + [`forks/THIRD_PARTY.md`](forks/THIRD_PARTY.md); pin metadata in + [`forks/whatdreamscost_koolook/versions/v1_3_2/UPSTREAM_PIN.yaml`](forks/whatdreamscost_koolook/versions/v1_3_2/UPSTREAM_PIN.yaml) + and [`forks/forks_manifest.yaml`](forks/forks_manifest.yaml). + ## [0.3.7] - 2026-05-23 ### Added diff --git a/__init__.py b/__init__.py index e07bd78..bc6e85b 100644 --- a/__init__.py +++ b/__init__.py @@ -25,6 +25,9 @@ try: from .forks.radiance_koolook import NODE_CLASS_MAPPINGS as radiance_koolook_mappings from .forks.radiance_koolook import NODE_DISPLAY_NAME_MAPPINGS as radiance_koolook_display + + from .forks.whatdreamscost_koolook import NODE_CLASS_MAPPINGS as whatdreamscost_koolook_mappings + from .forks.whatdreamscost_koolook import NODE_DISPLAY_NAME_MAPPINGS as whatdreamscost_koolook_display except ImportError as _node_import_exc: # The relative imports above require a parent-package context that # only ComfyUI's custom-node loader provides. Pytest's package @@ -68,6 +71,7 @@ else: **pattern_mappings, **video_combine_mappings, **radiance_koolook_mappings, + **whatdreamscost_koolook_mappings, } NODE_DISPLAY_NAME_MAPPINGS = { @@ -79,6 +83,7 @@ else: **pattern_display, **video_combine_display, **radiance_koolook_display, + **whatdreamscost_koolook_display, } WEB_DIRECTORY = "./web" diff --git a/docs/automations/CHEATSHEET.md b/docs/automations/CHEATSHEET.md index 1bc35c7..985c731 100644 --- a/docs/automations/CHEATSHEET.md +++ b/docs/automations/CHEATSHEET.md @@ -1,12 +1,12 @@ # Cheat sheet -For someone who already knows the loop. Full docs: [`README.md`](README.md), [`CONVENTIONS.md`](CONVENTIONS.md), [`LTX-2.3/README.md`](LTX-2.3/README.md). +For someone who already knows the loop. Full docs: [`README.md`](README.md), [`CONVENTIONS.md`](CONVENTIONS.md), and the per-module READMEs (e.g. [`LTX-2.3/base-1step/README.md`](LTX-2.3/base-1step/README.md), [`LTX-2.3/audio-lipsync/README.md`](LTX-2.3/audio-lipsync/README.md)). --- ## Bootstrap -Bare-bones — what's needed to get the loop running on a fresh machine. Model-specific setup (checkpoints, LoRAs, custom nodes for a particular model) lives in each model's handoff checklist, e.g. [`LTX-2.3/handoff-checklist.md`](LTX-2.3/handoff-checklist.md). +Bare-bones — what's needed to get the loop running on a fresh machine. Module-specific setup (checkpoints, LoRAs, custom nodes for a particular task, optional fork code) lives in each module's handoff checklist, e.g. [`LTX-2.3/base-1step/handoff-checklist.md`](LTX-2.3/base-1step/handoff-checklist.md). **Repo side** - [ ] Python 3.11+ with `pip install Pillow`. On Windows, the absolute path `C:/Python313/python.exe` sidesteps the Microsoft Store stub. @@ -85,10 +85,16 @@ ComfyUI-Koolook/ ├── README.md overview + start-here ├── CHEATSHEET.md this file ├── CONVENTIONS.md full contract - └── LTX-2.3/ one folder per model - ├── README.md - ├── handoff-checklist.md - └── findings.md + └── LTX-2.3/ model grouping (no shared docs of its own) + ├── base-1step/ ← one automation module + │ ├── README.md + │ ├── handoff-checklist.md + │ └── findings.md + └── audio-lipsync/ ← another module on the same model + ├── README.md + ├── handoff-checklist.md + ├── findings.md + └── runs/ in-repo run snapshots for fork-touching iterations ``` ### Working-folder-side (per project — pointed at by `KOLOOK_AUTOMATIONS_WORK_DIR`) @@ -113,7 +119,7 @@ Wipe `_AI/` to reset the agent's tracking history without touching renders. Wipe - **`loop` in the filename = post-card output, skipped.** When ComfyUI saves the composited-with-card video back into the working folder, name it `_loop_.mp4` (and pair JSON). Auto-discovery ignores anything with `loop` in the stem, so the script never feeds its own output back as input. - **Card filename is stable.** Wire your NLE to `_AI/card.png` once. - **Seed = 12 (fixed)** while sweeping any other knob. Vary seed only after a setting stabilises. -- **Scheduler = `linear_quadratic` (8 steps)** — locked-in finding. See [`LTX-2.3/findings.md`](LTX-2.3/findings.md). +- **Scheduler = `linear_quadratic` (8 steps)** — locked-in finding. See [`LTX-2.3/base-1step/findings.md`](LTX-2.3/base-1step/findings.md). - **Promote** a hypothesis to a finding when it's stable across ≥ 3 runs in `_AI/iterations.md`. - **Run label** on the card: prefers `v01`-style; falls back to `_00001` sequence numbers; else `?`. - **Switching projects:** update `Working_Folder_PATH` node + `.env`. Node wins on mismatch. diff --git a/docs/automations/LTX-2.3/audio-lipsync/README.md b/docs/automations/LTX-2.3/audio-lipsync/README.md new file mode 100644 index 0000000..6da934e --- /dev/null +++ b/docs/automations/LTX-2.3/audio-lipsync/README.md @@ -0,0 +1,123 @@ +# LTX 2.3 audio-file lip-sync — iteration loop + +AI-assisted iteration loop for getting audio-file-driven lip-sync working +on the WhatDreamsCost `LTXDirector` (LTX 2.3). The vanilla `use_custom_audio = True` +path produces broken video — no lip-sync, no meaningful animation, prompt +adherence collapses. This automation iterates on a Koolook-modified +`LTXDirector` that adds a `relay_overrides` widget and a per-segment-σ +formula in `build_segments`; the loop sweeps those knobs to find a setting +that preserves video coherence with the external audio. + +**Tracking issue:** [#177](https://github.com/malkuthro/ComfyUI-Koolook/issues/177) +**Sibling automation:** [`../base-1step/`](../base-1step/) — same model, the single-stage path with model-generated audio. Locked-in LTX-2.3 architecture facts live in its [`findings.md`](../base-1step/findings.md). +**Backstory:** [`backstory/audio-lipsync-rationale.md`](backstory/audio-lipsync-rationale.md) — why the audio path fails by default, the Prompt-Relay paper, what the two patches are doing mechanically. +**Started:** 2026-05-28 + +## What the Koolook fork ships + +Lives at [`../../../../forks/whatdreamscost_koolook/versions/v1_3_2/`](../../../../forks/whatdreamscost_koolook/versions/v1_3_2/), +loaded by ComfyUI as `LTX Director (Koolook v1.3.2)` in the picker. Two upstream +modifications: + +### Modification 1 — per-segment σ in `prompt_relay.py` + +`build_segments` now computes σ per-segment using the Prompt-Relay paper +formula: + +``` +σ = (L − w_eff) / (2 · √ln(1/ε)) +``` + +Upstream used a length-independent `σ = 1/ln(1/ε)` (≈ 0.1448 at ε=0.001) — +a razor-sharp boundary that did not scale with segment length L. The new +formula calibrates the penalty so it hits threshold ε exactly at the +segment boundary regardless of L. A `SIGMA_FALLBACK = 0.1448` preserves +the prior constant for the degenerate `L ≤ w_eff` corner. Per-segment σ +is logged as `[PromptRelay] seg L=… w_v=… sigma_v=…`. + +### Modification 2 — `relay_overrides` widget on `LTXDirector` + +A new optional multiline-string input on the LTXDirector canvas widget. +The maintainer pastes a JSON dict of Prompt-Relay knobs directly into the +node — values live entirely inside the workflow JSON, no disk file, no +env var. + +Supported keys (all optional): + +| Key | Type | Default | Effect | +|---|---|---|---| +| `video_strength` | float | 1.0 | Multiplies `attn2` penalty (video × text). Raise to tighten per-segment text routing. | +| `video_window_scale` | float | 1.0 | Scales the flat-top window width on the video side. < 1.0 = penalty kicks in closer to segment center. | +| `audio_strength` | float | 1.0 | Same for `audio_attn2`. **Note:** no-op for video output when `use_custom_audio=true` (audio latent is frozen at `noise_mask=0`). | +| `audio_window_scale` | float | 1.0 | Audio-side window. | +| `audio_epsilon` | float | inherit | Independent σ for the audio path. | + +Underscore-prefixed keys are ignored (use for inline JSON comments). Example value to paste: + +```json +{"video_strength": 10.0} +``` + +Empty field → upstream Prompt-Relay defaults (Modification 1 still active — +the new σ formula is unconditional). + +## Folder map + +``` +audio-lipsync/ +├── README.md ← this file +├── handoff-checklist.md ← bootstrap for picking up this automation cold +├── findings.md ← locked-in (currently empty; populated as runs validate) +├── backstory/ +│ └── audio-lipsync-rationale.md +└── runs/ ← per-render snapshots + ├── LOOP.md ← the iteration protocol + ├── log.md ← rolling table of runs + └── run-NNN_