* feat(forks): LTX Director v2_0_2 fork + keyframe latent-grid snap (#258) Bump the WhatDreamsCost Koolook fork to upstream 2.0.2 (pinned fe09f73) in a new versions/v2_0_2 namespace, and add `snap_keyframes_to_grid` (default on) to fix hard-pinned image keyframes "jumping" at latent-bucket edges. Why: a Director image keyframe is injected as a single latent frame at the segment's left edge, quantized to the model temporal stride s (L(f)=0 if f==0 else 1+(f-1)//s). Pins near a bucket boundary round ambiguously and two pins in one bucket collide. Snapping each pin to its bucket center removes the ambiguity without changing which latent frame it lands on; colliding pins are bumped to the next free bucket with a warning. - forks/whatdreamscost_koolook/versions/v2_0_2/: vendored 2.0.2 sources with the v1.3.9 Koolook deltas carried forward (relay_overrides, audio_transcript_json, per-segment sigma — build_segments is byte-identical 1.3.9->2.0.2 so the sigma patch drops in verbatim), plus new pure keyframe_grid.py snapper wired into the insert_frames build. - Canonical LTXDirector__koolook rolls to 2.0.2 (entrypoint loads v2_0_2 last); LTXDirector__koolook_v1_3_2 alias preserved; v1.3.9 kept on disk. - LICENSE re-verified GPL-3.0 (2026-06-22). Manifest + THIRD_PARTY + CHANGELOG + audio-lipsync findings/CHEATSHEET updated. - 37 new unit tests for the snapper; 107 fork tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(forks): append Koolook LTX Director widgets last for clean load Move the v2_0_2 Koolook widgets (relay_overrides, audio_transcript_json, snap_keyframes_to_grid) to the end of the schema, after override_audio. They were inserted after img_compression, which shifted the last upstream widget when an upstream-2.0.x-saved workflow is loaded onto LTXDirector__koolook. With them last, an upstream graph's positional widgets_values map onto the upstream widgets untouched and only the Koolook widgets append with defaults (snap_keyframes_to_grid=on). Execution is unaffected (kwargs by name). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(forks): make v2_0_2 a clean upstream-2.0.2 replica + grid-snap only Per maintainer direction: the v2_0_2 Koolook LTX Director should faithfully replicate upstream 2.0.2 (so all latest features work) rather than carry the older customizations, which are unused. - Drop relay_overrides, audio_transcript_json, and the per-segment sigma patch. prompt_relay.py and patches.py are now vendored VERBATIM from upstream 2.0.2; _relay_overrides.py removed. - ltx_director.py is upstream 2.0.2 verbatim except two additive changes: the namespaced node ID LTXDirector__koolook, and the snap_keyframes_to_grid latent-bucket keyframe snap (issue #258), appended last in the schema. - v2_0_2 registers only LTXDirector__koolook; the legacy LTXDirector__koolook_v1_3_2 ID stays backed by the v1_3_9 package. v1.3.9 (with the older customizations) remains on disk for rollback. - Manifest / THIRD_PARTY / UPSTREAM_PIN / CHANGELOG updated to the clean scope. grid-snap unit tests unchanged (107 fork tests pass). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(forks): vendor upstream 2.0.2 timeline editor JS for LTXDirector__koolook The v2_0_2 Python was upstream-exact but the node still rendered with the old 1.3.9-era timeline editor (fewer toolbar actions, no IC-LoRA track, leaked *_second widgets, frame_rate NaN). Replace web/whatdreamscost_koolook/ ltx_director.js with upstream 2.0.2's js/ltx_director.js vendored verbatim, retargeted with only two edits: unique extension name 'LTXDirector_Koolook' and node guard nodeData.name === 'LTXDirector__koolook'. UX now matches upstream 2.0.2 exactly; only the Python differs (grid-snap). The old koolook base64-scrub deltas are dropped as obsolete — upstream 2.0.2 persists timeline media via imageFile/audioFile + /api/view URLs (saved timelines are already lean), and the localStorage quota guard stays in the global web/koolook_draft_guard.js (upstream does not override localStorage). Rewrote tests/sidebar/test_ltx_director_draft_guard.py to the new contract (no embedded guard, retargeted to the koolook node, pinned-upstream provenance). 110 fork+contract tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(scripts): resolve .env from main repo when dev-sync runs in a worktree Unrelated to the LTX work: plain `dev-sync` failed with "KOLOOK_COMFYUI_DEV_PATH not set" from any worktree because scripts/sync_to_dev.py only loaded `.env` from the worktree root, where the committed `.env` does not exist (it lives in the main checkout). sync_to_dev_audio.py already had the worktree->main fallback, which is why dev-sync-audio worked. Move that fallback into a shared sync_to_dev.find_dotenv() and use it in both scripts so they can't drift. Adds regression tests for the worktree fallback, the worktree-own-.env preference, and the not-a-worktree case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: gitignore root-level generated workflow JSONs Node-swapped/converted workflow copies (e.g. *.koolook.json) are dropped at the repo root during dev iteration; they're scratch deliverables, not package content. Ignore root-level *.json only (leading `/`), with a `!config.json` exception, so the 30+ tracked JSON in subdirs (test fixtures, web/, .claude/, docs/runs/*/workflow.json) stay tracked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: LTX A/V Bind Schedule node — gate audio→video binding by sigma New Koolook node LTXAVBindSchedule decouples big motion from lip-sync in LTX 2.3. The AV transformer (comfy/ldm/lightricks/av_model.py, BasicAVTransformerBlock) adds audio into video via a gated `audio_to_video_attn` per block, run only when transformer_options["a2v_cross_attn"] is set. This node patches each block's audio_to_video_attn.forward to scale its output by a per-step gain, and a unet-function-wrapper computes that gain from the current sigma: early/high-sigma steps run near `early_gain` (coarse motion settles audio-blind), ramping (smoothstep) to full at low sigma (lips bind onto settled motion). One pass, no re-noise — the fix for keyframe transitions jumping on audio peaks. early_gain=1.0 reproduces stock behavior; a graph without the node is unchanged. Registered as a root node + added to RUNTIME_PATHS so dev-sync ships it. Pure ramp (a2v_gain) unit-tested (9 cases); model wiring validated by live render. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: LTX A/V Bind Schedule — add transition-window mode (lips synced + smooth) The global sigma ramp decoupled motion from lips but desynced the mouth (lip timing is coarse motion too, so holding audio out globally moved the mouth audio-blind). Add a WINDOWED mode: audio binds at full strength everywhere (lips sync across the whole clip) EXCEPT a +/-transition_window band around each keyframe transition, where the sigma ramp applies — so only those moments are motion-protected early and rejoined late. Implemented as a per-latent-frame gain on a2v_out: read grid_sizes from transformer_options to map frame-major video tokens -> latent frames, build a gain vector (1.0 outside bands, smoothstep-tapered to sigma_gain at each transition center), expand per-frame -> per-token, scale. New optional inputs: transition_frames, total_frames, transition_window. Empty transition_frames = prior global-ramp behavior. windowed_frame_gains unit-tested (7 cases). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: LTXAVBindSchedule — one-shot diagnostic log (shapes/grid/mode) Logs out.shape, grid_sizes, transition_fracs, and whether the per-frame mask fired vs fell back to scalar — so a single render reveals whether the a2v patch hooks the model and whether the windowed token->frame mapping aligns, instead of guessing blind. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(audio-lipsync): record validated a2v bind-schedule knee (0.35/0.25/0.55) Promote the finding that LTX 2.3's audio->video coupling must be sigma-ramped (not full-on) to keep motion smooth AND lips synced, with the validated knee early_gain=0.35 / bind_start=0.25 / bind_end=0.55 on the 8-step base pass, plus the trade-axis table and the per-frame-mode caveat (grid_sizes None + appended guide frames -> scalar fallback). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(forks): keyframe ease — strength-ramped neighbor pins to de-robotic transitions Adds `keyframe_ease` / `ease_falloff` to the Koolook LTX Director (default off). Each hard image pin keeps its exact center pose, but emits same-pose neighbor pins one latent bucket apart at decreasing strength (center*falloff**k), so the model glides into and out of the locked pose instead of snapping to a dead stop — softening the "stop/dissolve/stop" robotic feel without softening the pose. Implemented as a pure list expansion (`expand_keyframe_ease` in keyframe_grid.py, 6 unit tests) applied to guide_data after grid-snap, so the upstream guide node is untouched (no fork). Center stays exact; neighbors that fall off-clip or onto another center are skipped. Costs extra guide frames (more compute), hence opt-in. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(audio-lipsync): downgrade a2v knee to 'promising' — note lows-placement confound The 0.35/0.25/0.55 win was measured on a clip whose transitions sit on audio lows, so 'no jumps' may be the placement, not the node. Mark the controlled on-peak test (early_gain 1.0 vs 0.35 on the same on-peak layout) as still pending. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: mark vendored WhatDreamsCost fork files linguist-vendored The koolook Director ships upstream 2.0.2 code (the 8.5k-line timeline-editor JS + verbatim prompt_relay/patches) so the pack runs and distributes, but it's upstream, not authored here. Mark those linguist-vendored (drop from language stats) and the JS -diff (collapse the large blob in PR diffs). Our edited/new files stay attributed to us. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ltx-av): drop unused `import math` (ruff F401) a2v_gain uses an explicit smoothstep polynomial, not math.*, so the import was dead. Unblocks the Lint & Smoke CI gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(codeql): advanced setup that excludes vendored upstream code CodeQL default setup cannot read a config file, so it scanned the verbatim-vendored WhatDreamsCost editor (web/whatdreamscost_koolook/ ltx_director.js) and raised 4 high-sev DOM-XSS alerts in upstream code we do not own. Switch to advanced setup with a committed config whose paths-ignore mirrors the .gitattributes linguist-vendored block, so the scanner treats vendored upstream as out-of-scope while still fully scanning our own nodes and sidebar JS. Languages match the prior default-setup matrix (actions, javascript-typescript, python). NOTE: requires disabling "default setup" once in repo Settings before this workflow can run (GitHub rejects advanced runs while default setup is enabled). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): scope vendored fork exclusions to v2_0_2 * fix(forks): preserve final LTX keyframe snap bucket --------- Co-authored-by: AI Machine <ai.machine@x.baconcph.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
214 lines
7.5 KiB
Python
214 lines
7.5 KiB
Python
#!/usr/bin/env python3
|
|
"""
|
|
Scoped variant of ``dev-sync`` for the LTX 2.3 audio-lipsync automation.
|
|
|
|
Copies only what that automation's iteration loop touches:
|
|
|
|
- forks/whatdreamscost_koolook/ the Koolook fork of LTXDirector +
|
|
its modified prompt_relay + the
|
|
verbatim-vendored patches.py
|
|
- __init__.py root loader (registers the fork's
|
|
node mappings; needs to be re-synced
|
|
when the fork's NODE_CLASS_MAPPINGS
|
|
set changes)
|
|
- web/whatdreamscost_koolook/ Director timeline-editor extension
|
|
- web/koolook_draft_guard.js global Comfy draft-quota guard
|
|
(formerly embedded in the Director
|
|
extension; shipped so a scoped sync
|
|
never strands a guard-less install)
|
|
|
|
It also ships ``koolook_install_guard.py`` + ``koolook_versioning.py`` — the
|
|
loader gates ``__init__.py`` imports at load — so a scoped sync never leaves
|
|
the loader pointing at a module the target lacks (#198 / #183).
|
|
|
|
Everything else in the live install - ``forks/radiance_koolook/``, the
|
|
root ``k_*.py`` nodes, unrelated ``web/`` assets, ``video_formats/`` -
|
|
is left untouched.
|
|
|
|
USER-INITIATED ONLY. Same rule as plain ``dev-sync`` (see project
|
|
``CLAUDE.md`` -> ``dev-sync`` section). Never run automatically:
|
|
|
|
* after a commit
|
|
* after a PR merge or ``/ship-pr``
|
|
* at session end / wrap-up
|
|
* on hook completion or "task complete" cleanup
|
|
* from any agent skill that doesn't explicitly require it
|
|
|
|
Run only on the explicit user trigger phrase ``dev-sync-audio`` (or
|
|
``copy audio fork``, ``sync audio``, etc.). The maintainer typically has
|
|
multiple parallel sessions across worktrees; an unsolicited sync from one
|
|
silently destroys what another is reviewing.
|
|
|
|
The target path comes from ``KOLOOK_COMFYUI_DEV_PATH`` in ``.env`` -
|
|
identical to plain ``dev-sync``. No new env var.
|
|
|
|
Usage:
|
|
python scripts/sync_to_dev_audio.py
|
|
python scripts/sync_to_dev_audio.py --dry-run
|
|
python scripts/sync_to_dev_audio.py --scope "vstr=10 trial"
|
|
|
|
Exit codes mirror ``sync_to_dev.py`` exactly:
|
|
0 success
|
|
2 KOLOOK_COMFYUI_DEV_PATH unset or target missing without --init
|
|
3 --init refused
|
|
|
|
After copying Python files, restart ComfyUI manually so custom nodes are
|
|
re-imported. This scoped sync only copies files.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import os
|
|
import shutil
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
# Reuse the dev-sync infrastructure for the .env loader, target validation,
|
|
# and chat-report shape. This scoped wrapper differs only in which paths get
|
|
# copied, and it never tries to manage the running ComfyUI process.
|
|
_SCRIPT_DIR = Path(__file__).resolve().parent
|
|
if str(_SCRIPT_DIR) not in sys.path:
|
|
sys.path.insert(0, str(_SCRIPT_DIR))
|
|
import sync_to_dev as _dev # noqa: E402
|
|
|
|
|
|
AUDIO_PATHS: tuple[str, ...] = (
|
|
"__init__.py",
|
|
# __init__.py imports these at load, before any node group, and a missing
|
|
# gate is catastrophic (not per-group-guarded): the install guard's
|
|
# absolute-import fallback raises uncaught, and the koolook_versioning
|
|
# context probe mislabels its ImportError as a non-Comfy context and
|
|
# registers nothing. Ship them with __init__.py so a scoped sync never
|
|
# leaves the loader pointing at a module the target lacks (#198 / #183).
|
|
"koolook_install_guard.py",
|
|
"koolook_versioning.py",
|
|
"forks/whatdreamscost_koolook",
|
|
"web/whatdreamscost_koolook",
|
|
# The Comfy draft-quota guard used to be embedded in the Director web
|
|
# extension above; it is global now. Ship it with every scoped audio
|
|
# sync so replacing web/whatdreamscost_koolook/ can never leave a dev
|
|
# install without the guard.
|
|
"web/koolook_draft_guard.js",
|
|
)
|
|
|
|
STALE_AUDIO_PATHS: tuple[str, ...] = (
|
|
"web/whatdreamscost_koolook_v1_3_2",
|
|
)
|
|
|
|
|
|
def target_is_repo_root(target: Path) -> bool:
|
|
"""Return True when the configured dev target is this source repo."""
|
|
try:
|
|
return target.resolve() == _dev.REPO_ROOT.resolve()
|
|
except OSError:
|
|
return False
|
|
|
|
|
|
def remove_stale_paths(target: Path, *, dry_run: bool, verbose: bool) -> int:
|
|
"""Remove old audio-sync paths that were renamed.
|
|
|
|
The v1.3.9 upgrade moved the Director web extension from a versioned
|
|
folder to a stable one. Existing dev installs can still have the old
|
|
folder on disk, causing ComfyUI to load two timeline extensions for
|
|
legacy workflows. Keep this scoped to explicit known paths.
|
|
"""
|
|
removed = 0
|
|
for rel in STALE_AUDIO_PATHS:
|
|
stale = target / rel
|
|
if not stale.exists() and not stale.is_symlink():
|
|
continue
|
|
if dry_run:
|
|
if verbose:
|
|
print(f"would remove stale: {rel}")
|
|
removed += 1
|
|
continue
|
|
if stale.is_symlink() or stale.is_file():
|
|
stale.unlink()
|
|
elif stale.is_dir():
|
|
shutil.rmtree(stale)
|
|
else:
|
|
stale.unlink()
|
|
if verbose:
|
|
print(f"removed stale: {rel}")
|
|
removed += 1
|
|
return removed
|
|
|
|
|
|
def main() -> int:
|
|
parser = argparse.ArgumentParser(description=__doc__)
|
|
parser.add_argument(
|
|
"--dry-run",
|
|
action="store_true",
|
|
help="Print what would be copied without touching the target.",
|
|
)
|
|
parser.add_argument(
|
|
"--init",
|
|
action="store_true",
|
|
help=(
|
|
"Create the target directory if it doesn't exist. The parent "
|
|
"(typically a ComfyUI custom_nodes/ folder) must already be "
|
|
"in place. Use on first-run only."
|
|
),
|
|
)
|
|
parser.add_argument(
|
|
"--verbose",
|
|
"-v",
|
|
action="store_true",
|
|
help="Print each file/dir as it's copied.",
|
|
)
|
|
parser.add_argument(
|
|
"--scope",
|
|
type=str,
|
|
default="audio-lipsync fork edit",
|
|
help=(
|
|
"Short (<=10 word) description for the chat report's second line "
|
|
"and the in-browser footer in `web/_dev_build.json`. Defaults to "
|
|
"'audio-lipsync fork edit' so the build identifier always names "
|
|
"this module even when the maintainer doesn't pass a scope."
|
|
),
|
|
)
|
|
args = parser.parse_args()
|
|
|
|
env_path = _dev.find_dotenv()
|
|
if env_path is not None:
|
|
_dev.load_dotenv(env_path)
|
|
|
|
target_str = os.environ.get("KOLOOK_COMFYUI_DEV_PATH")
|
|
if not target_str:
|
|
print(
|
|
"KOLOOK_COMFYUI_DEV_PATH not set. Add it to .env (see .env.example).",
|
|
file=sys.stderr,
|
|
)
|
|
return 2
|
|
|
|
target = Path(target_str).expanduser()
|
|
if target_is_repo_root(target):
|
|
print(
|
|
"KOLOOK_COMFYUI_DEV_PATH points at this source repo; refusing dev-sync.",
|
|
file=sys.stderr,
|
|
)
|
|
return 2
|
|
|
|
err = _dev.ensure_target(target, init=args.init)
|
|
if err is not None:
|
|
return err
|
|
|
|
n = _dev.sync(
|
|
target,
|
|
dry_run=args.dry_run,
|
|
verbose=args.verbose,
|
|
paths=AUDIO_PATHS,
|
|
)
|
|
removed = remove_stale_paths(target, dry_run=args.dry_run, verbose=args.verbose)
|
|
verb = "would sync" if args.dry_run else "synced"
|
|
print(_dev.build_line())
|
|
stale_note = f"; removed {removed} stale" if removed else ""
|
|
print(f"{verb} {n} entries{stale_note} -> {target} (dev-sync-audio)")
|
|
if not args.dry_run:
|
|
_dev.write_build_info(target, args.scope)
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|