Files
ba37ec3d41 feat: LTX Director 2.0.2 fork, keyframe grid-snap, and audio→video bind schedule (#259)
* feat(forks): LTX Director v2_0_2 fork + keyframe latent-grid snap (#258)

Bump the WhatDreamsCost Koolook fork to upstream 2.0.2 (pinned fe09f73) in a
new versions/v2_0_2 namespace, and add `snap_keyframes_to_grid` (default on)
to fix hard-pinned image keyframes "jumping" at latent-bucket edges.

Why: a Director image keyframe is injected as a single latent frame at the
segment's left edge, quantized to the model temporal stride s
(L(f)=0 if f==0 else 1+(f-1)//s). Pins near a bucket boundary round
ambiguously and two pins in one bucket collide. Snapping each pin to its
bucket center removes the ambiguity without changing which latent frame it
lands on; colliding pins are bumped to the next free bucket with a warning.

- forks/whatdreamscost_koolook/versions/v2_0_2/: vendored 2.0.2 sources with
  the v1.3.9 Koolook deltas carried forward (relay_overrides,
  audio_transcript_json, per-segment sigma — build_segments is byte-identical
  1.3.9->2.0.2 so the sigma patch drops in verbatim), plus new pure
  keyframe_grid.py snapper wired into the insert_frames build.
- Canonical LTXDirector__koolook rolls to 2.0.2 (entrypoint loads v2_0_2
  last); LTXDirector__koolook_v1_3_2 alias preserved; v1.3.9 kept on disk.
- LICENSE re-verified GPL-3.0 (2026-06-22). Manifest + THIRD_PARTY + CHANGELOG
  + audio-lipsync findings/CHEATSHEET updated.
- 37 new unit tests for the snapper; 107 fork tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(forks): append Koolook LTX Director widgets last for clean load

Move the v2_0_2 Koolook widgets (relay_overrides, audio_transcript_json,
snap_keyframes_to_grid) to the end of the schema, after override_audio. They
were inserted after img_compression, which shifted the last upstream widget
when an upstream-2.0.x-saved workflow is loaded onto LTXDirector__koolook.
With them last, an upstream graph's positional widgets_values map onto the
upstream widgets untouched and only the Koolook widgets append with defaults
(snap_keyframes_to_grid=on). Execution is unaffected (kwargs by name).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(forks): make v2_0_2 a clean upstream-2.0.2 replica + grid-snap only

Per maintainer direction: the v2_0_2 Koolook LTX Director should faithfully
replicate upstream 2.0.2 (so all latest features work) rather than carry the
older customizations, which are unused.

- Drop relay_overrides, audio_transcript_json, and the per-segment sigma patch.
  prompt_relay.py and patches.py are now vendored VERBATIM from upstream 2.0.2;
  _relay_overrides.py removed.
- ltx_director.py is upstream 2.0.2 verbatim except two additive changes:
  the namespaced node ID LTXDirector__koolook, and the snap_keyframes_to_grid
  latent-bucket keyframe snap (issue #258), appended last in the schema.
- v2_0_2 registers only LTXDirector__koolook; the legacy
  LTXDirector__koolook_v1_3_2 ID stays backed by the v1_3_9 package. v1.3.9
  (with the older customizations) remains on disk for rollback.
- Manifest / THIRD_PARTY / UPSTREAM_PIN / CHANGELOG updated to the clean scope.
  grid-snap unit tests unchanged (107 fork tests pass).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(forks): vendor upstream 2.0.2 timeline editor JS for LTXDirector__koolook

The v2_0_2 Python was upstream-exact but the node still rendered with the old
1.3.9-era timeline editor (fewer toolbar actions, no IC-LoRA track, leaked
*_second widgets, frame_rate NaN). Replace web/whatdreamscost_koolook/
ltx_director.js with upstream 2.0.2's js/ltx_director.js vendored verbatim,
retargeted with only two edits: unique extension name 'LTXDirector_Koolook'
and node guard nodeData.name === 'LTXDirector__koolook'. UX now matches
upstream 2.0.2 exactly; only the Python differs (grid-snap).

The old koolook base64-scrub deltas are dropped as obsolete — upstream 2.0.2
persists timeline media via imageFile/audioFile + /api/view URLs (saved
timelines are already lean), and the localStorage quota guard stays in the
global web/koolook_draft_guard.js (upstream does not override localStorage).

Rewrote tests/sidebar/test_ltx_director_draft_guard.py to the new contract
(no embedded guard, retargeted to the koolook node, pinned-upstream
provenance). 110 fork+contract tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(scripts): resolve .env from main repo when dev-sync runs in a worktree

Unrelated to the LTX work: plain `dev-sync` failed with "KOLOOK_COMFYUI_DEV_PATH
not set" from any worktree because scripts/sync_to_dev.py only loaded
`.env` from the worktree root, where the committed `.env` does not exist
(it lives in the main checkout). sync_to_dev_audio.py already had the
worktree->main fallback, which is why dev-sync-audio worked.

Move that fallback into a shared sync_to_dev.find_dotenv() and use it in both
scripts so they can't drift. Adds regression tests for the worktree fallback,
the worktree-own-.env preference, and the not-a-worktree case.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: gitignore root-level generated workflow JSONs

Node-swapped/converted workflow copies (e.g. *.koolook.json) are dropped at
the repo root during dev iteration; they're scratch deliverables, not package
content. Ignore root-level *.json only (leading `/`), with a `!config.json`
exception, so the 30+ tracked JSON in subdirs (test fixtures, web/, .claude/,
docs/runs/*/workflow.json) stay tracked.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat: LTX A/V Bind Schedule node — gate audio→video binding by sigma

New Koolook node LTXAVBindSchedule decouples big motion from lip-sync in LTX
2.3. The AV transformer (comfy/ldm/lightricks/av_model.py, BasicAVTransformerBlock)
adds audio into video via a gated `audio_to_video_attn` per block, run only when
transformer_options["a2v_cross_attn"] is set. This node patches each block's
audio_to_video_attn.forward to scale its output by a per-step gain, and a
unet-function-wrapper computes that gain from the current sigma: early/high-sigma
steps run near `early_gain` (coarse motion settles audio-blind), ramping
(smoothstep) to full at low sigma (lips bind onto settled motion). One pass, no
re-noise — the fix for keyframe transitions jumping on audio peaks.

early_gain=1.0 reproduces stock behavior; a graph without the node is unchanged.
Registered as a root node + added to RUNTIME_PATHS so dev-sync ships it. Pure
ramp (a2v_gain) unit-tested (9 cases); model wiring validated by live render.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat: LTX A/V Bind Schedule — add transition-window mode (lips synced + smooth)

The global sigma ramp decoupled motion from lips but desynced the mouth (lip
timing is coarse motion too, so holding audio out globally moved the mouth
audio-blind). Add a WINDOWED mode: audio binds at full strength everywhere
(lips sync across the whole clip) EXCEPT a +/-transition_window band around each
keyframe transition, where the sigma ramp applies — so only those moments are
motion-protected early and rejoined late.

Implemented as a per-latent-frame gain on a2v_out: read grid_sizes from
transformer_options to map frame-major video tokens -> latent frames, build a
gain vector (1.0 outside bands, smoothstep-tapered to sigma_gain at each
transition center), expand per-frame -> per-token, scale. New optional inputs:
transition_frames, total_frames, transition_window. Empty transition_frames =
prior global-ramp behavior. windowed_frame_gains unit-tested (7 cases).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat: LTXAVBindSchedule — one-shot diagnostic log (shapes/grid/mode)

Logs out.shape, grid_sizes, transition_fracs, and whether the per-frame mask
fired vs fell back to scalar — so a single render reveals whether the a2v patch
hooks the model and whether the windowed token->frame mapping aligns, instead
of guessing blind.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(audio-lipsync): record validated a2v bind-schedule knee (0.35/0.25/0.55)

Promote the finding that LTX 2.3's audio->video coupling must be sigma-ramped
(not full-on) to keep motion smooth AND lips synced, with the validated knee
early_gain=0.35 / bind_start=0.25 / bind_end=0.55 on the 8-step base pass, plus
the trade-axis table and the per-frame-mode caveat (grid_sizes None + appended
guide frames -> scalar fallback).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(forks): keyframe ease — strength-ramped neighbor pins to de-robotic transitions

Adds `keyframe_ease` / `ease_falloff` to the Koolook LTX Director (default off).
Each hard image pin keeps its exact center pose, but emits same-pose neighbor
pins one latent bucket apart at decreasing strength (center*falloff**k), so the
model glides into and out of the locked pose instead of snapping to a dead stop
— softening the "stop/dissolve/stop" robotic feel without softening the pose.

Implemented as a pure list expansion (`expand_keyframe_ease` in keyframe_grid.py,
6 unit tests) applied to guide_data after grid-snap, so the upstream guide node
is untouched (no fork). Center stays exact; neighbors that fall off-clip or onto
another center are skipped. Costs extra guide frames (more compute), hence opt-in.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(audio-lipsync): downgrade a2v knee to 'promising' — note lows-placement confound

The 0.35/0.25/0.55 win was measured on a clip whose transitions sit on audio
lows, so 'no jumps' may be the placement, not the node. Mark the controlled
on-peak test (early_gain 1.0 vs 0.35 on the same on-peak layout) as still pending.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: mark vendored WhatDreamsCost fork files linguist-vendored

The koolook Director ships upstream 2.0.2 code (the 8.5k-line timeline-editor JS
+ verbatim prompt_relay/patches) so the pack runs and distributes, but it's
upstream, not authored here. Mark those linguist-vendored (drop from language
stats) and the JS -diff (collapse the large blob in PR diffs). Our edited/new
files stay attributed to us.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ltx-av): drop unused `import math` (ruff F401)

a2v_gain uses an explicit smoothstep polynomial, not math.*, so the
import was dead. Unblocks the Lint & Smoke CI gate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* ci(codeql): advanced setup that excludes vendored upstream code

CodeQL default setup cannot read a config file, so it scanned the
verbatim-vendored WhatDreamsCost editor (web/whatdreamscost_koolook/
ltx_director.js) and raised 4 high-sev DOM-XSS alerts in upstream code we
do not own. Switch to advanced setup with a committed config whose
paths-ignore mirrors the .gitattributes linguist-vendored block, so the
scanner treats vendored upstream as out-of-scope while still fully
scanning our own nodes and sidebar JS.

Languages match the prior default-setup matrix (actions,
javascript-typescript, python).

NOTE: requires disabling "default setup" once in repo Settings before
this workflow can run (GitHub rejects advanced runs while default setup
is enabled).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ci): scope vendored fork exclusions to v2_0_2

* fix(forks): preserve final LTX keyframe snap bucket

---------

Co-authored-by: AI Machine <ai.machine@x.baconcph.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-24 11:15:41 +02:00

214 lines
7.5 KiB
Python

#!/usr/bin/env python3
"""
Scoped variant of ``dev-sync`` for the LTX 2.3 audio-lipsync automation.
Copies only what that automation's iteration loop touches:
- forks/whatdreamscost_koolook/ the Koolook fork of LTXDirector +
its modified prompt_relay + the
verbatim-vendored patches.py
- __init__.py root loader (registers the fork's
node mappings; needs to be re-synced
when the fork's NODE_CLASS_MAPPINGS
set changes)
- web/whatdreamscost_koolook/ Director timeline-editor extension
- web/koolook_draft_guard.js global Comfy draft-quota guard
(formerly embedded in the Director
extension; shipped so a scoped sync
never strands a guard-less install)
It also ships ``koolook_install_guard.py`` + ``koolook_versioning.py`` — the
loader gates ``__init__.py`` imports at load — so a scoped sync never leaves
the loader pointing at a module the target lacks (#198 / #183).
Everything else in the live install - ``forks/radiance_koolook/``, the
root ``k_*.py`` nodes, unrelated ``web/`` assets, ``video_formats/`` -
is left untouched.
USER-INITIATED ONLY. Same rule as plain ``dev-sync`` (see project
``CLAUDE.md`` -> ``dev-sync`` section). Never run automatically:
* after a commit
* after a PR merge or ``/ship-pr``
* at session end / wrap-up
* on hook completion or "task complete" cleanup
* from any agent skill that doesn't explicitly require it
Run only on the explicit user trigger phrase ``dev-sync-audio`` (or
``copy audio fork``, ``sync audio``, etc.). The maintainer typically has
multiple parallel sessions across worktrees; an unsolicited sync from one
silently destroys what another is reviewing.
The target path comes from ``KOLOOK_COMFYUI_DEV_PATH`` in ``.env`` -
identical to plain ``dev-sync``. No new env var.
Usage:
python scripts/sync_to_dev_audio.py
python scripts/sync_to_dev_audio.py --dry-run
python scripts/sync_to_dev_audio.py --scope "vstr=10 trial"
Exit codes mirror ``sync_to_dev.py`` exactly:
0 success
2 KOLOOK_COMFYUI_DEV_PATH unset or target missing without --init
3 --init refused
After copying Python files, restart ComfyUI manually so custom nodes are
re-imported. This scoped sync only copies files.
"""
from __future__ import annotations
import argparse
import os
import shutil
import sys
from pathlib import Path
# Reuse the dev-sync infrastructure for the .env loader, target validation,
# and chat-report shape. This scoped wrapper differs only in which paths get
# copied, and it never tries to manage the running ComfyUI process.
_SCRIPT_DIR = Path(__file__).resolve().parent
if str(_SCRIPT_DIR) not in sys.path:
sys.path.insert(0, str(_SCRIPT_DIR))
import sync_to_dev as _dev # noqa: E402
AUDIO_PATHS: tuple[str, ...] = (
"__init__.py",
# __init__.py imports these at load, before any node group, and a missing
# gate is catastrophic (not per-group-guarded): the install guard's
# absolute-import fallback raises uncaught, and the koolook_versioning
# context probe mislabels its ImportError as a non-Comfy context and
# registers nothing. Ship them with __init__.py so a scoped sync never
# leaves the loader pointing at a module the target lacks (#198 / #183).
"koolook_install_guard.py",
"koolook_versioning.py",
"forks/whatdreamscost_koolook",
"web/whatdreamscost_koolook",
# The Comfy draft-quota guard used to be embedded in the Director web
# extension above; it is global now. Ship it with every scoped audio
# sync so replacing web/whatdreamscost_koolook/ can never leave a dev
# install without the guard.
"web/koolook_draft_guard.js",
)
STALE_AUDIO_PATHS: tuple[str, ...] = (
"web/whatdreamscost_koolook_v1_3_2",
)
def target_is_repo_root(target: Path) -> bool:
"""Return True when the configured dev target is this source repo."""
try:
return target.resolve() == _dev.REPO_ROOT.resolve()
except OSError:
return False
def remove_stale_paths(target: Path, *, dry_run: bool, verbose: bool) -> int:
"""Remove old audio-sync paths that were renamed.
The v1.3.9 upgrade moved the Director web extension from a versioned
folder to a stable one. Existing dev installs can still have the old
folder on disk, causing ComfyUI to load two timeline extensions for
legacy workflows. Keep this scoped to explicit known paths.
"""
removed = 0
for rel in STALE_AUDIO_PATHS:
stale = target / rel
if not stale.exists() and not stale.is_symlink():
continue
if dry_run:
if verbose:
print(f"would remove stale: {rel}")
removed += 1
continue
if stale.is_symlink() or stale.is_file():
stale.unlink()
elif stale.is_dir():
shutil.rmtree(stale)
else:
stale.unlink()
if verbose:
print(f"removed stale: {rel}")
removed += 1
return removed
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--dry-run",
action="store_true",
help="Print what would be copied without touching the target.",
)
parser.add_argument(
"--init",
action="store_true",
help=(
"Create the target directory if it doesn't exist. The parent "
"(typically a ComfyUI custom_nodes/ folder) must already be "
"in place. Use on first-run only."
),
)
parser.add_argument(
"--verbose",
"-v",
action="store_true",
help="Print each file/dir as it's copied.",
)
parser.add_argument(
"--scope",
type=str,
default="audio-lipsync fork edit",
help=(
"Short (<=10 word) description for the chat report's second line "
"and the in-browser footer in `web/_dev_build.json`. Defaults to "
"'audio-lipsync fork edit' so the build identifier always names "
"this module even when the maintainer doesn't pass a scope."
),
)
args = parser.parse_args()
env_path = _dev.find_dotenv()
if env_path is not None:
_dev.load_dotenv(env_path)
target_str = os.environ.get("KOLOOK_COMFYUI_DEV_PATH")
if not target_str:
print(
"KOLOOK_COMFYUI_DEV_PATH not set. Add it to .env (see .env.example).",
file=sys.stderr,
)
return 2
target = Path(target_str).expanduser()
if target_is_repo_root(target):
print(
"KOLOOK_COMFYUI_DEV_PATH points at this source repo; refusing dev-sync.",
file=sys.stderr,
)
return 2
err = _dev.ensure_target(target, init=args.init)
if err is not None:
return err
n = _dev.sync(
target,
dry_run=args.dry_run,
verbose=args.verbose,
paths=AUDIO_PATHS,
)
removed = remove_stale_paths(target, dry_run=args.dry_run, verbose=args.verbose)
verb = "would sync" if args.dry_run else "synced"
print(_dev.build_line())
stale_note = f"; removed {removed} stale" if removed else ""
print(f"{verb} {n} entries{stale_note} -> {target} (dev-sync-audio)")
if not args.dry_run:
_dev.write_build_info(target, args.scope)
return 0
if __name__ == "__main__":
sys.exit(main())