* feat(investigations): scaffold #177 LTX23 audio-file lip-sync investigation
Reorganize docs/investigations/ into numbered topic folders, each
self-contained with narrative + upstream pin + patches + per-run snapshots.
Add a generic sync helper that mirrors the dev-sync pattern for any
investigation's third-party node modifications.
Structure (new):
docs/investigations/
00_LTX23-base-1-step/README.md (rename of ltx-director-4k-transitions.md)
01_LTX23-audio-file-lipsync/
README.md -- problem, mechanism, patches, reversion, promotion
upstream.json -- repo + version + pinned commit + sync target env + files
patches/ -- source of truth for modified upstream files
ltx_director.py -- adds `relay_overrides` multiline JSON input
prompt_relay.py -- per-segment sigma per Prompt-Relay paper formula
runs/
LOOP.md -- iteration protocol
log.md -- rolling table (empty header for now)
Sync helper:
scripts/sync_investigation_patches.py <NN>_<topic> [--dry-run]
- Reads upstream.json for target env var + file list
- Backs up the target's current state once per day (<file>.bak.<YYYYMMDD>)
- Worktree-aware: falls back to MAIN's .env via git rev-parse --git-common-dir
- Prints a two-line report matching scripts/sync_to_dev.py shape
- Per-investigation trigger phrase via `trigger:` key (sync-audio for #177)
Env:
.env.example -- documents new KOLOOK_WHATDREAMSCOST_PATH
.env -- local path set
Glossary updates (docs/reference/glossary.md):
+ Investigation folder
+ JSON file folder
+ Loop
+ Run
+ Investigation patches
+ sync_investigation_patches.py
Closes the workflow gap from #177: MAIN is now the source of truth for
investigation patches; the live install is a sync destination only. No
manual file copying between iterations — `sync-audio` does it.
* chore(ruff): exclude docs/investigations from lint
The investigation patches under docs/investigations/*/patches/ are
vendored upstream source (WhatDreamsCost-ComfyUI v1.3.2) carried so
sync_investigation_patches.py can mirror them into the live install
and so iterations can be diff'd against the pinned upstream commit.
Linting them would force divergence from upstream and defeat that
purpose. Mirrors the existing forks/ exclude precedent.
Fixes the CI Lint & Smoke failure on PR #178.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: per-task automation modules + WhatDreamsCost Koolook fork
Pivot docs/automations/ from "one folder per model" to "one folder per
automation module" so the same diffusion model can host several
independent tasks side by side. Replace the parallel docs/investigations/
tree with a fork of WhatDreamsCost-ComfyUI under forks/.
Automation modules (new layout)
- docs/automations/LTX-2.3/base-1step/ the existing single-stage
automation, moved from
docs/automations/LTX-2.3/
with its README, findings,
and handoff-checklist
rescoped to the task. The
4k-transitions backstory
that produced the module
moves here as
backstory/4k-transitions.md.
The earlier
docs/investigations/experiments/
card PNGs/HTML land in
example-cards/.
- docs/automations/LTX-2.3/audio-lipsync/ the audio-file lip-sync
module previously living
under docs/investigations/01_…/.
README, handoff, findings
stub, backstory/, and
runs/{LOOP,log}.md, all
referencing the new fork.
- docs/automations/README.md + CHEATSHEET.md updated to describe the
module-per-task pivot and
the new tree shape.
- docs/automations/CONVENTIONS.md unchanged (cross-cutting
loop contract still holds).
WhatDreamsCost Koolook fork (new)
- forks/whatdreamscost_koolook/versions/v1_3_2/
UPSTREAM_PIN.yaml pinned to upstream e81223a
(v1.3.2), GPL-3.0
verified 2026-05-28.
ltx_director.py MODIFIED — adds the
relay_overrides multiline
JSON widget on LTXDirector.
Moved from
docs/investigations/01_…/patches/.
prompt_relay.py MODIFIED — per-segment σ per
the Prompt-Relay paper
formula σ = (L − w_eff) /
(2 · √ln(1/ε)). Moved from
docs/investigations/01_…/patches/.
patches.py UNMODIFIED — vendored
verbatim from upstream
because ltx_director.py
imports detect_model_type
and apply_patches from it
(same precedent as
radiance_koolook's
color_helpers.py).
- Registered as LTXDirector__koolook_v1_3_2 ("LTX Director (Koolook v1.3.2)")
so it coexists with an installed copy of upstream WhatDreamsCost-ComfyUI.
Wired into the root __init__.py NODE_CLASS_MAPPINGS merge.
- forks/forks_manifest.yaml + forks/THIRD_PARTY.md updated with the new
entry (attribution, change log, license verified date, exposed node IDs).
Removed
- docs/investigations/ entire tree retired; its
contents either moved to
automation modules or to
forks/.
- scripts/sync_investigation_patches.py no longer needed — fork
code lives in the Koolook
tree and is synced via the
standard dev-sync flow.
- KOLOOK_WHATDREAMSCOST_PATH in .env.example same reason.
- docs/investigations in pyproject.toml's ruff
extend-exclude added in 96bf8ad to silence
lint on vendored patches;
no longer needed now that
vendored code is under
forks/ (already excluded).
Docs
- docs/reference/glossary.md Investigation folder,
Investigation patches,
JSON file folder, and the
sync_investigation_patches.py
entries replaced by
Automation module, Backstory,
Working folder, and Card.
Loop and Run rewritten to
point at the new structure.
- CHANGELOG.md [Unreleased] describes the restructure +
the new fork.
Verification
- ruff check . → all checks passed
- tools/preflight_release.py --skip manager-meta → 42 files parsed; 11
unique node IDs collected including LTXDirector; VAE dispatch branches
all OK; fixture workflows still reference all registered Koolook IDs.
- pytest → 173 passed, 0 failed.
Workflow JSON heads-up
The active audio-lipsync workflow LTX-23-audio_tests_v01.json continues to
reference the upstream LTXDirector ID. To pick up the new fork's behaviour,
swap that node to "LTX Director (Koolook v1.3.2)" (registered ID
LTXDirector__koolook_v1_3_2) — sockets and wiring are preserved. The
relay_overrides widget appears at the bottom of the new node. Documented
in the audio-lipsync README's "Workflow JSON — Koolook node ID" section.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(scripts): add dev-sync-audio — scoped sync for audio-lipsync iteration
`dev-sync-audio` is a scoped wrapper around `dev-sync` for the
[`docs/automations/LTX-2.3/audio-lipsync/`](docs/automations/LTX-2.3/audio-lipsync/)
iteration loop. It copies *only* the paths that module touches — the
`forks/whatdreamscost_koolook/` subtree and the root `__init__.py` —
instead of `rmtree`-ing the whole dest `forks/` and re-copying every
Koolook fork on every render. Same `KOLOOK_COMFYUI_DEV_PATH` target,
same ComfyUI-Manager restart, same two-line chat-report shape.
Changes
- `scripts/sync_to_dev_audio.py` (new). Thin wrapper that reuses
`sync_to_dev`'s env loader, target validator, restart helper, and
chat-report builder, supplying its own `AUDIO_PATHS` tuple. Defaults
the `--scope` tag to `"audio-lipsync fork edit"` so the build
identifier names this module even when invoked without an explicit
scope.
- `scripts/sync_to_dev.py` — `sync()` gains an optional `paths` kwarg
(defaults to the existing `RUNTIME_PATHS`). Backwards-compatible with
every existing call site.
- `docs/automations/LTX-2.3/audio-lipsync/README.md` and
`runs/LOOP.md` updated to point at `dev-sync-audio` instead of plain
`dev-sync` for the iteration step.
- Project `CLAUDE.md` — new "Per-module scoped variants" subsection
under the existing `dev-sync` section documents the
`dev-sync-<module>` pattern, the trigger-name → script-name mapping,
and reiterates that the user-initiated-only rule applies verbatim to
every `dev-sync-*` variant.
Verification
- ruff check . → all passed
- python scripts/sync_to_dev_audio.py --dry-run -v →
would copy: __init__.py
would copy: forks/whatdreamscost_koolook
e7f3ab4 - beautiful-mahavira-ae6da6
would sync 2 entries -> <KOLOOK_COMFYUI_DEV_PATH> (dev-sync-audio)
(Confirms scope is correct and the existing dev-sync chat-report
shape is preserved.)
User-initiated only — never automatic. Trigger phrase: `dev-sync-audio`
in chat (also: "copy audio fork", "sync audio fork"). Same rule as
`dev-sync` — see project CLAUDE.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(audio-lipsync): only snapshot runs when explicitly kept
Every render adds a row to runs/log.md (always). The heavier
run-NNN_<label>/ snapshot folders (workflow JSON copy + relay_overrides
+ patch state + notes) are now created only when the maintainer says
"keep", "save run", "save this", or "snapshot" in chat. Without that
trigger, the render is just a log row.
Why: an active knob sweep can do 10+ renders/hour. Saving a folder
per render bloats runs/ fast and most of those folders are noise the
moment the next render lands. Cheap log row everywhere, expensive
snapshot only for runs worth pinning (knob landed, surprising failure,
something heading for findings.md).
Changes
- docs/automations/LTX-2.3/audio-lipsync/runs/LOOP.md
Step 5 split into "always log" + "snapshot only on request".
Trigger-detection table updated. Run-labels section reframed
(NNN matches log.md row number). Added a one-paragraph "why this
retention policy" rationale.
- docs/automations/LTX-2.3/audio-lipsync/README.md
Folder-map flags run-NNN_<label>/ as conditional. Iteration loop
step 5 names the keep / save-run / save-this triggers explicitly.
Not a code change. No script edits — the retention rule lives in the
agent protocol, not in dev-sync-audio (which stays scoped to code
sync, not run management).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(review): harden relay_overrides parser + polish (PR #178 review)
Addresses verified findings from the /review-pr pass on PR #178.
H1 — relay_overrides values are now type-checked
- Allowlist of recognised knobs (`video_strength`, `video_window_scale`,
`audio_strength`, `audio_window_scale`, `audio_epsilon`) with explicit
float-coercion. Numeric strings ("10") coerce silently; non-numeric
values (e.g. "ten") now raise a ValueError naming the offending knob
and showing a minimal correct example — instead of a TypeError deep
inside `build_segments` arithmetic, or a SUCCESS render at upstream
defaults that polluted the A/B iteration log.
H2 — malformed JSON no longer silently falls back to defaults
- `json.JSONDecodeError`, non-dict JSON (arrays, strings, bare numbers,
null), and non-numeric values for recognised knobs all now raise
`ValueError` with operator-actionable messages. The only silent
fallback is the explicit empty/whitespace case — "use upstream
defaults" is the operator's intent there.
- Typo'd keys (e.g. `viedo_strength`) log a warning naming the unknown
keys + the recognised set, then drop the typo. Previously they were
silently retained in the cleaned dict but never queried by
`build_segments`, indistinguishable from a default render.
Architectural side-effect — shrinks the diff vs vendored upstream
- `_parse_relay_overrides` extracted into a stdlib-only sibling module
`_relay_overrides.py`. `ltx_director.py` shrinks closer to upstream
baseline (one import + one alias instead of a 30-line function body)
and is easier to diff against `e81223a`. The parser is also now
unit-testable without ComfyUI's `comfy_api` runtime.
L5 — tests added (31 cases, all passing)
- New `tests/forks/__init__.py` + `tests/forks/test_whatdreamscost_v1_3_2_relay_overrides.py`.
- Covers every confirmed failure mode (H1 + H2), the happy path, the
comment-key convention, the unknown-key warning, and an allowlist-
consistency assertion that fails loudly if `build_segments` ever
grows a knob without the parser being updated.
- Test module uses `importlib.util.spec_from_file_location` so it can
load the parser without walking through the package hierarchy (which
would trigger the `comfy_api` import inside `ltx_director.py`).
L3 — `_build_line` promoted to public `build_line`
- `scripts/sync_to_dev.py` renames the private helper. The underscore
alias is kept for one release cycle as a deprecation slot.
- `scripts/sync_to_dev_audio.py` switches to `_dev.build_line()`. No
behaviour change; chat-report shape is bit-identical to before.
L2 — `source_ref` clarified in pin metadata
- `forks/whatdreamscost_koolook/versions/v1_3_2/UPSTREAM_PIN.yaml` and
`forks/forks_manifest.yaml` both now spell out that `e81223a` is the
post-v1.3.2 README-only commit on upstream's `main`, not the v1.3.2
release tag itself. Code is byte-identical to the tagged release; the
pin differs only in trailing docs.
L1, L4 are explicitly left alone (L1 is dead upstream cruft, kept for
minimum-diff; L4 is helper duplication tracked for a separate follow-up
issue when a third fork lands).
Verification
- ruff check . → clean
- pytest → 204 passed (173 prior + 31 new parser tests)
- tools/preflight_release.py --skip manager-meta → all PASS (static AST
extraction still picks up LTXDirector from the fork)
- scripts/sync_to_dev_audio.py --dry-run → two-line chat report still
emits via the renamed `build_line` helper
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(audio-lipsync): config-driven loop-audio script + module cheat sheet
End-to-end automation for one render cycle of the audio-lipsync
iteration loop. The chat phrase `loop-audio` invokes a single script
that does the whole chain — no manual artifact-shuffling.
Scripts (cross-cutting infrastructure at the repo root, like make_card
and sync_to_dev; they get lifted into the module folder in the
`docs/automations/` → `automations/` move queued for the follow-up PR)
- scripts/loop_audio.py
Reads its settings from loop_audio.config.json (alongside),
auto-discovers the most recent workflow JSON in the ComfyUI user
workflows dir (derived from KOLOOK_COMFYUI_DEV_PATH or overridden
via KOLOOK_COMFYUI_WORKFLOWS_DIR), extracts the five tracked Text
Multilines + LTX Director state + sampler chain, captures
fork-state (HEAD SHA + last dev-sync-audio sync info + fork-dir
git status), writes the run-NNN_<auto-label>/ snapshot, appends a
row to runs/log.md, and chains the card renderer in. Flags
mirror sync_to_dev: --dry-run, --label, --no-log, --no-card,
--workflow, --config.
- scripts/loop_audio.config.json
Single source of truth for the customisable settings (job_name,
module_path, workflow_pattern, comfyui_workflows_subpath,
skip_filename_substring, tracked_multilines, fork_to_track,
render_card). Underscore-prefixed keys are inline documentation
and ignored by the loader. When the convention shifts — new
multiline title, new fork path, different filename pattern —
edit the JSON, not the Python.
- scripts/make_card_audio.py
Audio-lipsync-specific card renderer. Five sections (KNOB STATE,
FORK STATE, SAMPLER, BASE notes, OUTCOME) plus an INERT warning
band when the workflow's Director is upstream `LTXDirector` with
a non-empty relay_overrides. Same palette + font fallback chain
as scripts/make_card.py so the base-1step and audio-lipsync card
families read as a set. PIL/Pillow added to the `test` optional
deps so CI doesn't hit ImportError if any future test touches
this script.
Docs
- docs/automations/LTX-2.3/audio-lipsync/CHEATSHEET.md
Module-scoped operating instructions per the cheat-sheet pattern:
what the module is for, the chat triggers, the required workflow
shape (5 multilines + the Koolook Director variant), the render
cycle steps, what lands in each run-NNN_ folder, findings-
promotion rule, where every customisable setting lives, and a
card-section legend.
- docs/reference/glossary.md
`Card` rewritten to describe the per-module renderer fan-out
(`make_card.py` for base-1step, `make_card_audio.py` for
audio-lipsync) — cards no longer live at a single stable
`_AI/card.png` path. New `Loop config (per-module)` entry
documenting the `<script>.config.json` convention + the
required-keys list.
- docs/automations/LTX-2.3/audio-lipsync/runs/log.md
Header rewritten with column set matching the actual log-row
output (Director / relay_overrides / Custom audio / Phase 1 /
Phase 2 / Scores / Notes).
First real run captured
- docs/automations/LTX-2.3/audio-lipsync/runs/run-001_bear_3x-fr_w.audio_upstream_audio-off_vstr10.0/
Baseline reference render — `Bear_3x-FR_w.Audio`, upstream
`LTXDirector` still wired into the workflow, use_custom_audio
off. Motion 4/5 · Sync 5/5 · Sharp 4/5 per maintainer feedback.
The relay_overrides `{"video_strength": 10.0}` is set but inert
(Director not yet swapped to the Koolook variant); the card
surfaces this with the amber INERT band at the bottom. Captures:
workflow.json (185KB), relay_overrides.txt (with INERT note),
patch_state.txt (3ffb7b7 / 15d5688 / clean), notes.md (feedback
+ scores + mechanical interp), card.png (61KB).
Verification
- ruff check . → clean
- pytest -q → 204 passed, 0 failed
- tools/preflight_release.py --skip manager-meta → all PASS
- scripts/loop_audio.py --dry-run → identifies the latest workflow,
emits the two-line chat-report header, doesn't touch disk
- scripts/loop_audio.py (real) → wrote the run-001 folder + card +
log row in a single invocation
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(audio-lipsync card): match base-1step visual style 1:1
The first cut of the audio-lipsync card used solid colored header bands
and block section panels — visually heavier than base-1step's renderer.
Refactored to copy make_card.py's primitives directly:
- small uppercase accent-colored section labels (no header bars)
- subtle BG_SECTION panel outlines instead of full-width fills
- left-accented note boxes (4px coloured bar + dark BG + label inside)
for Base · notes, Feedback (video), and the conditional Director
note warning
- same font fallback chain + sizes as make_card.py (F_SECTION 17pt,
F_MONO 19pt, F_NOTE italic 18pt)
- oversized canvas + post-render crop matches make_card.py's pattern
so cards size to actual content
- palette imported in lockstep (BG_OUTER/CARD/SECTION, BORDER, the
three accent triples plus the new ACCENT_FORK violet)
Two tuning fixes spotted during re-render:
- _wrap_text gained a keep_blank_lines flag so prose blocks can
collapse the empty separator lines maintainer canvases often have
- max_lines bumped on the prose blocks: Base · notes 18 (was 6,
truncated the OVERLAY - INFO body), Feedback 6 (was 5), Director
note 8 (was 4, truncated mid-sentence)
- make_card_audio.py's standalone CLI now loads .env via
loop_audio.load_dotenv before rebuilding state, so re-renders show
the same FORK STATE lines (MAIN sha + sync sha + scope) that the
live loop_audio run captures
Visual reference: docs/automations/LTX-2.3/base-1step/example-cards/
run-001-card.png is the layout target. The audio-lipsync card now
reads as a sibling of that family — KNOB STATE in amber, FORK STATE in
violet, SAMPLER + Base · notes in sky, FEEDBACK + OUTCOME in green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(fork): ship LTX Director timeline-editor JS to the Koolook variant
The Koolook variant of LTXDirector was loading as a bare schema — no
image strip, no audio strip, no canvas timeline editor — because the
upstream timeline-editor JS bundle never came across with the fork.
The widget upstream paints onto the node lives entirely in
`js/ltx_director.js`; the Python class only declares the inputs.
Compounding it, the JS's binding check is hard-coded
``if (nodeData.name === "LTXDirector")``, so even if the bundle had
been loaded, it would have skipped our suffixed registry ID
(`LTXDirector__koolook_v1_3_2`).
Fix: vendor the JS into the fork and rebind it.
- web/whatdreamscost_koolook_v1_3_2/ltx_director.js
Verbatim copy of upstream's
custom_nodes/WhatDreamsCost-ComfyUI/js/ltx_director.js at the
pinned commit (e81223a). Two string changes vs upstream:
* `name: "LTXDirector"` -> `"LTXDirector_Koolook_v1_3_2"`
so the registered ComfyUI extension doesn't collide with the
upstream extension (both load in parallel).
* `if (nodeData.name === "LTXDirector")` ->
`if (nodeData.name === "LTXDirector__koolook_v1_3_2")`
so the canvas widget mounts on the Koolook variant only;
the upstream extension keeps owning the upstream node.
A short comment block at the top documents both deltas.
- scripts/sync_to_dev_audio.py
AUDIO_PATHS gains `web/whatdreamscost_koolook_v1_3_2` so
dev-sync-audio carries the JS into the live install alongside the
fork's Python files + root __init__.py.
After dev-sync-audio + ComfyUI restart, the Koolook variant mounts the
same timeline editor as upstream: image/audio strips, segment dragging,
the visual preview row, the [Audio] tag handling, the
img_compression/divisible_by widgets. The `relay_overrides` field
declared in the Python schema renders at the bottom of the node
alongside everything else.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
malkuthro