diff --git a/.env.example b/.env.example index e28928a..6a801bf 100644 --- a/.env.example +++ b/.env.example @@ -106,17 +106,3 @@ KOLOOK_AUTOMATIONS_WORK_DIR= # KOLOOK_COMFYUI_DEV_PATH (assumes /user/comfyui.log). Set this # only if your install writes logs elsewhere. KOLOOK_COMFYUI_LOG= - -# --- Investigation patch sync targets --------------------------------------- -# Per-investigation absolute path to a third-party node's live install dir. -# Used by scripts/sync_investigation_patches.py to push patched files from -# docs/investigations/_/patches/ into the running ComfyUI. -# Each upstream gets its own env var, declared in the investigation's -# upstream.yaml under sync_target_env. - -# WhatDreamsCost-ComfyUI install path. Required for investigation 01 -# (LTX23-audio-file-lipsync). Forward slashes OK on Windows. -# -# Example: -# Windows: E:/_AI/portable/ComfyUI_windows_312/ComfyUI/custom_nodes/WhatDreamsCost-ComfyUI -KOLOOK_WHATDREAMSCOST_PATH= diff --git a/CHANGELOG.md b/CHANGELOG.md index af31d72..6a938c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,46 @@ The format is inspired by Keep a Changelog and SemVer. ## [Unreleased] +### Changed +- **Automations restructured around modules instead of models.** + `docs/automations/` now pivots on one folder per *task* (each its own + workflow + iteration loop + findings) rather than one folder per model. + The existing `LTX-2.3/{README,findings,handoff-checklist}.md` content + moved into `LTX-2.3/base-1step/` (single-stage render task) and the new + sibling `LTX-2.3/audio-lipsync/` (audio-file lip-sync task) was created. + `docs/automations/{README,CHEATSHEET}.md` updated to reflect the new + layout; `docs/automations/CONVENTIONS.md` unchanged. +- **`docs/investigations/` removed.** Its two children migrated to their + natural homes: narrative content into the new automation modules under + `docs/automations/LTX-2.3/{base-1step,audio-lipsync}/`; the modified + upstream Python code into the new `forks/whatdreamscost_koolook/` + fork (see *Added*). The retired `scripts/sync_investigation_patches.py` + and its `KOLOOK_WHATDREAMSCOST_PATH` env-var section in `.env.example` + go with it — iteration now uses the standard `dev-sync` flow against + the in-repo fork. +- **Glossary updated.** The *Investigation folder*, *Investigation patches*, + *JSON file folder*, and `sync_investigation_patches.py` entries in + [`docs/reference/glossary.md`](docs/reference/glossary.md) are replaced + by *Automation module*, *Backstory*, *Working folder*, and *Card*; the + *Loop* and *Run* entries are rewritten to point at the new structure. + +### Added +- **`forks/whatdreamscost_koolook/v1_3_2/` — Koolook fork of WhatDreamsCost-ComfyUI's + `LTXDirector`** (upstream `e81223a`, GPL-3.0). Two upstream files modified + (`ltx_director.py` adds a `relay_overrides` multiline-JSON widget; + `prompt_relay.py` uses the Prompt-Relay paper's per-segment σ formula + instead of upstream's length-independent constant) plus the + unmodified `patches.py` vendored verbatim because the modified file + imports from it. Registered as + `LTXDirector__koolook_v1_3_2` (display name *"LTX Director (Koolook + v1.3.2)"*) so it coexists with an installed copy of upstream + WhatDreamsCost-ComfyUI in the node picker. Drives the + [`docs/automations/LTX-2.3/audio-lipsync/`](docs/automations/LTX-2.3/audio-lipsync/) + iteration loop. License attribution + change log in + [`forks/THIRD_PARTY.md`](forks/THIRD_PARTY.md); pin metadata in + [`forks/whatdreamscost_koolook/versions/v1_3_2/UPSTREAM_PIN.yaml`](forks/whatdreamscost_koolook/versions/v1_3_2/UPSTREAM_PIN.yaml) + and [`forks/forks_manifest.yaml`](forks/forks_manifest.yaml). + ## [0.3.7] - 2026-05-23 ### Added diff --git a/__init__.py b/__init__.py index e07bd78..bc6e85b 100644 --- a/__init__.py +++ b/__init__.py @@ -25,6 +25,9 @@ try: from .forks.radiance_koolook import NODE_CLASS_MAPPINGS as radiance_koolook_mappings from .forks.radiance_koolook import NODE_DISPLAY_NAME_MAPPINGS as radiance_koolook_display + + from .forks.whatdreamscost_koolook import NODE_CLASS_MAPPINGS as whatdreamscost_koolook_mappings + from .forks.whatdreamscost_koolook import NODE_DISPLAY_NAME_MAPPINGS as whatdreamscost_koolook_display except ImportError as _node_import_exc: # The relative imports above require a parent-package context that # only ComfyUI's custom-node loader provides. Pytest's package @@ -68,6 +71,7 @@ else: **pattern_mappings, **video_combine_mappings, **radiance_koolook_mappings, + **whatdreamscost_koolook_mappings, } NODE_DISPLAY_NAME_MAPPINGS = { @@ -79,6 +83,7 @@ else: **pattern_display, **video_combine_display, **radiance_koolook_display, + **whatdreamscost_koolook_display, } WEB_DIRECTORY = "./web" diff --git a/docs/automations/CHEATSHEET.md b/docs/automations/CHEATSHEET.md index 1bc35c7..985c731 100644 --- a/docs/automations/CHEATSHEET.md +++ b/docs/automations/CHEATSHEET.md @@ -1,12 +1,12 @@ # Cheat sheet -For someone who already knows the loop. Full docs: [`README.md`](README.md), [`CONVENTIONS.md`](CONVENTIONS.md), [`LTX-2.3/README.md`](LTX-2.3/README.md). +For someone who already knows the loop. Full docs: [`README.md`](README.md), [`CONVENTIONS.md`](CONVENTIONS.md), and the per-module READMEs (e.g. [`LTX-2.3/base-1step/README.md`](LTX-2.3/base-1step/README.md), [`LTX-2.3/audio-lipsync/README.md`](LTX-2.3/audio-lipsync/README.md)). --- ## Bootstrap -Bare-bones — what's needed to get the loop running on a fresh machine. Model-specific setup (checkpoints, LoRAs, custom nodes for a particular model) lives in each model's handoff checklist, e.g. [`LTX-2.3/handoff-checklist.md`](LTX-2.3/handoff-checklist.md). +Bare-bones — what's needed to get the loop running on a fresh machine. Module-specific setup (checkpoints, LoRAs, custom nodes for a particular task, optional fork code) lives in each module's handoff checklist, e.g. [`LTX-2.3/base-1step/handoff-checklist.md`](LTX-2.3/base-1step/handoff-checklist.md). **Repo side** - [ ] Python 3.11+ with `pip install Pillow`. On Windows, the absolute path `C:/Python313/python.exe` sidesteps the Microsoft Store stub. @@ -85,10 +85,16 @@ ComfyUI-Koolook/ ├── README.md overview + start-here ├── CHEATSHEET.md this file ├── CONVENTIONS.md full contract - └── LTX-2.3/ one folder per model - ├── README.md - ├── handoff-checklist.md - └── findings.md + └── LTX-2.3/ model grouping (no shared docs of its own) + ├── base-1step/ ← one automation module + │ ├── README.md + │ ├── handoff-checklist.md + │ └── findings.md + └── audio-lipsync/ ← another module on the same model + ├── README.md + ├── handoff-checklist.md + ├── findings.md + └── runs/ in-repo run snapshots for fork-touching iterations ``` ### Working-folder-side (per project — pointed at by `KOLOOK_AUTOMATIONS_WORK_DIR`) @@ -113,7 +119,7 @@ Wipe `_AI/` to reset the agent's tracking history without touching renders. Wipe - **`loop` in the filename = post-card output, skipped.** When ComfyUI saves the composited-with-card video back into the working folder, name it `_loop_.mp4` (and pair JSON). Auto-discovery ignores anything with `loop` in the stem, so the script never feeds its own output back as input. - **Card filename is stable.** Wire your NLE to `_AI/card.png` once. - **Seed = 12 (fixed)** while sweeping any other knob. Vary seed only after a setting stabilises. -- **Scheduler = `linear_quadratic` (8 steps)** — locked-in finding. See [`LTX-2.3/findings.md`](LTX-2.3/findings.md). +- **Scheduler = `linear_quadratic` (8 steps)** — locked-in finding. See [`LTX-2.3/base-1step/findings.md`](LTX-2.3/base-1step/findings.md). - **Promote** a hypothesis to a finding when it's stable across ≥ 3 runs in `_AI/iterations.md`. - **Run label** on the card: prefers `v01`-style; falls back to `_00001` sequence numbers; else `?`. - **Switching projects:** update `Working_Folder_PATH` node + `.env`. Node wins on mismatch. diff --git a/docs/automations/LTX-2.3/audio-lipsync/README.md b/docs/automations/LTX-2.3/audio-lipsync/README.md new file mode 100644 index 0000000..6da934e --- /dev/null +++ b/docs/automations/LTX-2.3/audio-lipsync/README.md @@ -0,0 +1,123 @@ +# LTX 2.3 audio-file lip-sync — iteration loop + +AI-assisted iteration loop for getting audio-file-driven lip-sync working +on the WhatDreamsCost `LTXDirector` (LTX 2.3). The vanilla `use_custom_audio = True` +path produces broken video — no lip-sync, no meaningful animation, prompt +adherence collapses. This automation iterates on a Koolook-modified +`LTXDirector` that adds a `relay_overrides` widget and a per-segment-σ +formula in `build_segments`; the loop sweeps those knobs to find a setting +that preserves video coherence with the external audio. + +**Tracking issue:** [#177](https://github.com/malkuthro/ComfyUI-Koolook/issues/177) +**Sibling automation:** [`../base-1step/`](../base-1step/) — same model, the single-stage path with model-generated audio. Locked-in LTX-2.3 architecture facts live in its [`findings.md`](../base-1step/findings.md). +**Backstory:** [`backstory/audio-lipsync-rationale.md`](backstory/audio-lipsync-rationale.md) — why the audio path fails by default, the Prompt-Relay paper, what the two patches are doing mechanically. +**Started:** 2026-05-28 + +## What the Koolook fork ships + +Lives at [`../../../../forks/whatdreamscost_koolook/versions/v1_3_2/`](../../../../forks/whatdreamscost_koolook/versions/v1_3_2/), +loaded by ComfyUI as `LTX Director (Koolook v1.3.2)` in the picker. Two upstream +modifications: + +### Modification 1 — per-segment σ in `prompt_relay.py` + +`build_segments` now computes σ per-segment using the Prompt-Relay paper +formula: + +``` +σ = (L − w_eff) / (2 · √ln(1/ε)) +``` + +Upstream used a length-independent `σ = 1/ln(1/ε)` (≈ 0.1448 at ε=0.001) — +a razor-sharp boundary that did not scale with segment length L. The new +formula calibrates the penalty so it hits threshold ε exactly at the +segment boundary regardless of L. A `SIGMA_FALLBACK = 0.1448` preserves +the prior constant for the degenerate `L ≤ w_eff` corner. Per-segment σ +is logged as `[PromptRelay] seg L=… w_v=… sigma_v=…`. + +### Modification 2 — `relay_overrides` widget on `LTXDirector` + +A new optional multiline-string input on the LTXDirector canvas widget. +The maintainer pastes a JSON dict of Prompt-Relay knobs directly into the +node — values live entirely inside the workflow JSON, no disk file, no +env var. + +Supported keys (all optional): + +| Key | Type | Default | Effect | +|---|---|---|---| +| `video_strength` | float | 1.0 | Multiplies `attn2` penalty (video × text). Raise to tighten per-segment text routing. | +| `video_window_scale` | float | 1.0 | Scales the flat-top window width on the video side. < 1.0 = penalty kicks in closer to segment center. | +| `audio_strength` | float | 1.0 | Same for `audio_attn2`. **Note:** no-op for video output when `use_custom_audio=true` (audio latent is frozen at `noise_mask=0`). | +| `audio_window_scale` | float | 1.0 | Audio-side window. | +| `audio_epsilon` | float | inherit | Independent σ for the audio path. | + +Underscore-prefixed keys are ignored (use for inline JSON comments). Example value to paste: + +```json +{"video_strength": 10.0} +``` + +Empty field → upstream Prompt-Relay defaults (Modification 1 still active — +the new σ formula is unconditional). + +## Folder map + +``` +audio-lipsync/ +├── README.md ← this file +├── handoff-checklist.md ← bootstrap for picking up this automation cold +├── findings.md ← locked-in (currently empty; populated as runs validate) +├── backstory/ +│ └── audio-lipsync-rationale.md +└── runs/ ← per-render snapshots + ├── LOOP.md ← the iteration protocol + ├── log.md ← rolling table of runs + └── run-NNN_