diff --git a/CHANGELOG.md b/CHANGELOG.md index 915a8c5..5af5a0e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,18 @@ The format is inspired by Keep a Changelog and SemVer. ## [Unreleased] ### Added +- **LTX Keyframe Soften Schedule node (`LTXKeyframeSoftenSchedule`).** Smooths + motion between keyframes without adding any guide frames: a model patcher that + sigma-schedules the keyframe denoise-mask pins — loosening them early (high + sigma) so the model forms smooth motion between poses, then restoring them by a + crossover point so the poses sharpen for the back half of denoise. Uses the + sampler's per-step `denoise_mask_function` hook; works with **no IC-LoRA**. + Audio-safe: the A/V mask is a `NestedTensor((video, audio))`, so only the video + keyframe pins are softened and the audio mask passes through untouched (lip-sync + unaffected). Two normalized 0-1 widgets map onto the (very sensitive) useful + range: `max_soften` (ease amount, 0-1 → internal 0-0.20) and `crossover` + (invention window, 0-1 → internal 0-0.05). Pure `soften_gain` ramp unit-tested; + model wiring render-validated. - **LTX Director fork bumped to upstream v2.0.2** with a new `forks/whatdreamscost_koolook/versions/v2_0_2/` namespace (pinned to upstream commit `fe09f73`). This version is a **faithful replica of upstream 2.0.2** — @@ -39,7 +51,66 @@ The format is inspired by Keep a Changelog and SemVer. reproduces stock behavior. Ramp math is unit-tested; the model wiring is validated by rendering against a live LTX 2.3 AV model. +- **LTX Director Ghost Mask reference (REF image / character sheet).** New + optional `reference_images` (IMAGE) + `reference_strength` inputs on + `LTXDirector__koolook` (plus informational `clean_latent_frames` / + `clean_pixel_frames` outputs). Reference image(s) are added as guide frames + just past the clean timeline, so the model anchors identity (face / mouth + shape) the way a WAN VACE reference does — without the references appearing in + the output and **without a LoRA**. Targets mouth/identity drift when keyframes + show a closed (or different) mouth than the audio-driven motion needs. The + references ride the **same append-and-crop path as the timeline keyframes**: + `LTXDirectorGuide` appends each as an extra latent frame and + `LTXDirectorCropGuides` removes it, so the video latent is never pre-grown and + the audio track stays in sync (pre-growing it desyncs A/V and breaks motion). + A general-purpose root node **`CleanLatentSlice`** ("Clean Latent Slice + (Koolook)") is also included for bare pipelines that don't use + `LTXDirectorCropGuides`. The trailing-reference *approach* is adapted (idea + only, reimplemented) from CGlide's WhatDreamsCost-CSGlide (GPL-3.0); see + `forks/THIRD_PARTY.md`. Geometry is unit-tested; model behavior is validated + by rendering against a live LTX 2.3 model. (Phase 1: Ghost Mask only — the + LoRA-backed Licon MSR prefix mode is intentionally not ported.) + +- **LTX Guide Reference Strength node (`LTXGuideReferenceStrength`).** Companion + to the Ghost Mask reference: rewrites the strength of *only* the reference + frames in `guide_data` (the trailing entries the Director tags with + `reference_count`), leaving the keyframe pins untouched. Intended for + two-stage pipelines — feed stage 1 the Director's `guide_data` unchanged + (light reference, motion forms freely) and route stage 2's guide_data through + this node at a higher reference strength, so identity locks during the + low-denoise refinement without disturbing stage-1 motion. Strength math is + unit-tested. + +- **LTX Reference Bind Schedule node (`LTXReferenceBindSchedule`).** Ramps the + Ghost Mask reference's **attention** up as sigma falls — neutral early (motion + forms freely) rising to `peak_strength` late (identity locks during + refinement). The inverse of the motion curve and the mirror of + `LTXAVBindSchedule`. Mechanism: a `model_function_wrapper` captures the + per-step sigma and an object-patch on the core LTX + `_build_guide_self_attention_mask` (which is rebuilt every denoise step and + supports `strength > 1.0` to amplify) scales **only the trailing + `num_references` guide entries'** attention strength — keyframe pins and the + noise-mask freezing are untouched, so it's orthogonal to `reference_strength`. + Single pass, no re-noise. The `ref_gain` ramp is unit-tested; model behavior is + validated by rendering against a live LTX 2.3 model. **Works without an + IC-LoRA:** the attention-bias machinery is base-model functionality, but + `LTXDirectorGuide` only *populates* the per-guide entries when an IC-LoRA is + active. So when the entries are missing this node **fabricates them itself** + for the trailing reference tokens (set `num_keyframes` so it knows the + keyframe/reference split of `num_guide_tokens`), giving the reference the same + amplified-attention channel an IC-LoRA would use — inside the Director, no + guide-node changes, no graph rewire. When an IC-LoRA *is* active it scales the + real entries instead. The split/fabrication math is unit-tested. + ### Fixed +- **Keyframe guidance now defaults to 0.8 (smooth), not 1.0 (robotic).** The + timeline editor auto-generated the `guide_strength` string from each clip's + per-segment `guideStrength`, filling **1.0** for any clip left unset — and the + Director's Python fallback did the same. 1.0 is a hard pin, so any clip you + forgot to lower snapped instead of transitioning smoothly. Unset clips now + default to **0.8** in both the editor (display + serialization) and the node; + per-clip values you set explicitly still override (e.g. a hard 1.0 pin on a + critical pose). Clips with no explicit strength shift from 1.0 → 0.8 on load. - **`dev-sync` from a git worktree.** `scripts/sync_to_dev.py` now resolves `.env` with the worktree→main-repo fallback (the committed `.env` lives only in the main checkout), matching `sync_to_dev_audio.py` and the documented diff --git a/__init__.py b/__init__.py index 2ec2c0f..d8ed732 100644 --- a/__init__.py +++ b/__init__.py @@ -132,6 +132,10 @@ else: _merge_node_group("Koolook Loop Status", ".k_loop_status") _merge_node_group("Koolook Publish Contract", ".k_publish_contract") _merge_node_group("LTX A/V Bind Schedule", ".k_ltx_av_bind_schedule") + _merge_node_group("Clean Latent Slice", ".k_clean_latent_slice") + _merge_node_group("LTX Guide Reference Strength", ".k_ltx_guide_reference_strength") + _merge_node_group("LTX Reference Bind Schedule", ".k_ltx_reference_bind_schedule") + _merge_node_group("LTX Keyframe Soften Schedule", ".k_ltx_keyframe_soften_schedule") # The VHS-dependent video wrappers degrade to empty mappings when # VideoHelperSuite is absent (they catch the import internally); the # guard also covers a hard ImportError should one ever slip through. diff --git a/forks/THIRD_PARTY.md b/forks/THIRD_PARTY.md index 3adcf1e..13ed491 100644 --- a/forks/THIRD_PARTY.md +++ b/forks/THIRD_PARTY.md @@ -279,24 +279,35 @@ below — GPL-3.0 §5(c) requires the whole work to be GPL-3.0. Koolook customizations (`relay_overrides`, `audio_transcript_json`, per-segment sigma) are **intentionally dropped** as unused, so every latest 2.0.2 feature (Prompt Relay, audio, motion) behaves exactly as upstream. - - **`ltx_director.py` (upstream verbatim + 3 additive changes).** (1) the - node is namespaced as `LTXDirector__koolook` so it coexists with the + - **`ltx_director.py` (upstream verbatim + Koolook additive changes).** (1) + the node is namespaced as `LTXDirector__koolook` so it coexists with the installed upstream copy; (2) **issue #258:** a `snap_keyframes_to_grid` - toggle (default on, appended last in the schema) snaps each image - keyframe's pixel position to the center of its LTX latent-time bucket - before the guide is built, so hard pins land on a single latent frame and - two pins never collide in one bucket (the later is bumped, with a logged - warning). Pixel positions in, pixel positions out — the upstream - `LTXDirectorGuide` re-derives latent indices, so it needs no change; and - (3) `keyframe_ease` / `ease_falloff` optionally emit strength-ramped - neighbor pins of the same pose to smooth into and out of hard keyframes. - Nothing else is modified. + toggle (default on) snaps each image keyframe's pixel position to the + center of its LTX latent-time bucket before the guide is built, so hard + pins land on a single latent frame and two pins never collide in one + bucket (the later is bumped, with a logged warning); (3) `keyframe_ease` / + `ease_falloff` emit strength-ramped neighbor pins to soften the robotic + stop/dissolve at each hard pin; (4) **Ghost Mask reference** — + `reference_images` (IMAGE) + `reference_strength` inputs append the + reference image(s) as masked guide frames *after* the clean video region + so the model anchors identity (face / mouth shape) without the references + appearing in the output, plus `clean_latent_frames` / `clean_pixel_frames` + outputs to drive the slice-off (see CGlide attribution below). All inputs + are optional and appended last, so saved workflows are unaffected; pixel + positions in, pixel positions out — the upstream `LTXDirectorGuide` + re-derives latent indices and needs no change. - **`prompt_relay.py` (verbatim vendored).** Unmodified upstream 2.0.2; imported by `ltx_director.py` for its Prompt Relay helpers. - **`patches.py` (verbatim vendored).** Unmodified upstream `fe09f73`. - **`keyframe_grid.py` (new, Koolook-original).** Pure stdlib latent-grid snap/ease helpers; unit-tested in [`tests/forks/test_whatdreamscost_v2_0_2_keyframe_grid.py`](../tests/forks/test_whatdreamscost_v2_0_2_keyframe_grid.py). + - **`reference_ghost.py` (new, Koolook-original).** Pure stdlib geometry for + the Ghost Mask trailing-reference layout (insert frames + grown latent + length + slice bounds); unit-tested in + [`tests/forks/test_whatdreamscost_v2_0_2_reference_ghost.py`](../tests/forks/test_whatdreamscost_v2_0_2_reference_ghost.py). + The trailing-reference *approach* is adapted from CGlide (see entry below); + the code is Koolook-original. - **What was *not* forked:** Same policy as v1.3.9 — `LTXDirectorGuide` and other WhatDreamsCost helper nodes remain runtime dependencies from the user's installed upstream package. @@ -306,7 +317,37 @@ below — GPL-3.0 §5(c) requires the whole work to be GPL-3.0. is not registered here; it stays backed by the v1.3.9 package for workflows saved against it. v1.3.9 (with the older customizations) remains on disk for rollback. -- **Last reviewed:** 2026-06-22 +- **Last reviewed:** 2026-06-24 + +### CGlide/WhatDreamsCost-CSGlide — Ghost Mask reference approach (Koolook reimplementation) + +- **Name:** WhatDreamsCost-CSGlide (a fork of WhatDreamsCost-ComfyUI) +- **Upstream repo URL:** https://github.com/CGlide/WhatDreamsCost-CSGlide +- **Upstream commit/tag used:** `main_cs` @ `21dd0766898f6c3fed34dc9d2e9f87e67c4d7607` + (2026-06-23). Upstream does not tag releases. +- **License:** GPL-3.0 (verified 2026-06-24 from the upstream `LICENSE` file; + `pyproject.toml` declares `license = {file = "LICENSE"}`). Compatible with + this package's GPL-3.0 — no relicense required. +- **Scope of import:** clean-room reimplementation of one feature — *not* a + vendor. We read CGlide's `ltx_director.py` (Ghost Mask branch) and + `latent_slice.py` to learn the mechanism, then wrote our own code. +- **What we took (idea, not code):** the **Ghost Mask** technique for + reference-image identity anchoring without a LoRA — append the reference + image(s) as guide frames *after* the clean video region (so the transformer + attends to them for identity), generate, then slice the trailing reference + frames off the output latent. Implemented in Koolook as: the + `reference_images` / `reference_strength` inputs + `clean_latent_frames` / + `clean_pixel_frames` outputs on `LTXDirector__koolook` + ([`versions/v2_0_2/ltx_director.py`](whatdreamscost_koolook/versions/v2_0_2/ltx_director.py)), + the pure geometry helper + [`versions/v2_0_2/reference_ghost.py`](whatdreamscost_koolook/versions/v2_0_2/reference_ghost.py), + and the new root node `CleanLatentSlice` ("Clean Latent Slice (Koolook)") in + [`k_clean_latent_slice.py`](../k_clean_latent_slice.py). +- **What we did NOT take:** the Licon MSR (Prefix) mode and its companion + LiconStudio LoRA dependency, the `@charN` prompt-tag / character-slot editor + UI, the Ollama character-analysis endpoint, and CGlide's editor JS rewrite. + Ghost Mask here is fed by a plain IMAGE socket, not the timeline editor. +- **Last reviewed:** 2026-06-24 ## De-vendored upstream code (untracked in v0.1.4 / v0.1.5) diff --git a/forks/forks_manifest.yaml b/forks/forks_manifest.yaml index d8d6d76..a30583a 100644 --- a/forks/forks_manifest.yaml +++ b/forks/forks_manifest.yaml @@ -1,5 +1,5 @@ version: 1 -updated_at: "2026-06-22" +updated_at: "2026-06-25" external_forks: root_relative_to_main: "../ComfyUI-Forks" root_env_var: "KOLOOK_FORKS_DIR" @@ -130,9 +130,10 @@ entries: - "forks/whatdreamscost_koolook/versions/v2_0_2/prompt_relay.py" - "forks/whatdreamscost_koolook/versions/v2_0_2/patches.py" - "forks/whatdreamscost_koolook/versions/v2_0_2/keyframe_grid.py" + - "forks/whatdreamscost_koolook/versions/v2_0_2/reference_ghost.py" - "web/whatdreamscost_koolook/ltx_director.js" status: "modified" - sync_state: "upstream-verbatim-plus-keyframe-snap" + sync_state: "upstream-verbatim-plus-koolook-additions" license: "GPL-3.0" license_verified_at: "2026-06-22" exposed_node_ids: @@ -161,6 +162,48 @@ entries: so the editor UX matches upstream 2.0.2 exactly; the old 1.3.9-era editor and its now-obsolete base64-scrub deltas are replaced. localStorage quota guarding stays in the global `web/koolook_draft_guard.js`. + Beyond the keyframe snap, `ltx_director.py` also carries the Koolook + `keyframe_ease`/`ease_falloff` neighbour-pin softening and the Ghost Mask + reference feature (`reference_images`/`reference_strength` inputs + + `clean_latent_frames`/`clean_pixel_frames` outputs); all are optional + inputs / appended outputs (saved workflows unaffected). The Ghost Mask + geometry lives in the new Koolook-original `reference_ghost.py`; its + companion root node is `k_clean_latent_slice.py` ("Clean Latent Slice + (Koolook)", node id `CleanLatentSlice`). The Ghost Mask APPROACH (append + references as trailing guide frames, then slice them off) is adapted — + idea only, reimplemented — from CGlide/WhatDreamsCost-CSGlide (main_cs @ + 21dd076, GPL-3.0, verified 2026-06-24); see forks/THIRD_PARTY.md. The + LoRA-backed Licon MSR prefix mode is intentionally not ported. The CGlide + Ghost Mask adaptation has its own formal entry below + (`csglide_ghost_mask_reimplementation`). + + - id: "csglide_ghost_mask_reimplementation" + name: "WhatDreamsCost-CSGlide Ghost Mask approach (Koolook reimplementation)" + source_repo: "https://github.com/CGlide/WhatDreamsCost-CSGlide" + source_ref: "main_cs (idea-only adaptation — no source code copied)" + pinned_commit: "21dd076" + local_paths: + - "forks/whatdreamscost_koolook/versions/v2_0_2/reference_ghost.py" + - "k_clean_latent_slice.py" + status: "inspired_reimplementation" + sync_state: "no-upstream-tracking-needed" + license: "GPL-3.0" + license_verified_at: "2026-06-24" + exposed_node_ids: + - "CleanLatentSlice" + notes: | + Idea-level adaptation of CSGlide's "Ghost Mask" reference approach — append + reference images as trailing guide frames, let the transformer attend to + them for identity, then slice them off the output latent so they never + appear. NO source code was copied: the integer geometry is a fresh + Koolook write in `forks/whatdreamscost_koolook/versions/v2_0_2/reference_ghost.py` + (pure-stdlib, unit-tested), the in-Director plumbing lives in that fork's + `ltx_director.py` (Ghost Mask block), and the tensor slice is the + Koolook-original `k_clean_latent_slice.py` (node id `CleanLatentSlice`). + CSGlide is itself a GPL-3.0 fork of WhatDreamsCost-ComfyUI (already vendored + here under GPL-3.0), so no relicense pressure. `pinned_commit` is the short + hash on record (`main_cs @ 21dd076`); full attribution in + forks/THIRD_PARTY.md. The LoRA-backed Licon MSR prefix mode is not ported. - id: "easy_resize_inspiration" name: "Easy Resize (Koolook reimplementation, inspired by KJ Nodes)" diff --git a/forks/whatdreamscost_koolook/versions/v2_0_2/UPSTREAM_PIN.yaml b/forks/whatdreamscost_koolook/versions/v2_0_2/UPSTREAM_PIN.yaml index 5fdae60..8c7bdf5 100644 --- a/forks/whatdreamscost_koolook/versions/v2_0_2/UPSTREAM_PIN.yaml +++ b/forks/whatdreamscost_koolook/versions/v2_0_2/UPSTREAM_PIN.yaml @@ -14,14 +14,23 @@ notes: | motion) behave exactly as upstream. This version folder ships: - - ltx_director.py (upstream 2.0.2 VERBATIM + three additive changes only: - the namespaced node ID `LTXDirector__koolook`, and the - `snap_keyframes_to_grid` latent-bucket keyframe snap from issue #258, - and `keyframe_ease` / `ease_falloff` strength-ramped neighbor pins). + - ltx_director.py (upstream 2.0.2 VERBATIM + Koolook additive changes only: + the namespaced node ID `LTXDirector__koolook`; the + `snap_keyframes_to_grid` latent-bucket keyframe snap from issue #258; + the `keyframe_ease`/`ease_falloff` strength-ramped neighbor pins; and the + Ghost Mask reference feature (`reference_images`/`reference_strength` + inputs + `clean_latent_frames`/`clean_pixel_frames` outputs). All + additions are optional inputs / appended outputs, so saved workflows are + unaffected.). - prompt_relay.py (VERBATIM upstream 2.0.2; imported by ltx_director.py). - patches.py (VERBATIM upstream 2.0.2; imported by ltx_director.py). - keyframe_grid.py (NEW, Koolook-original) pure latent-grid snap/ease helpers used by ltx_director.py; unit-tested in tests/forks/. + - reference_ghost.py (NEW, Koolook-original) pure geometry for the Ghost + Mask trailing-reference layout; unit-tested in tests/forks/. The Ghost + Mask APPROACH is adapted from CGlide's WhatDreamsCost-CSGlide (main_cs @ + 21dd076, GPL-3.0) — idea only, reimplemented; see forks/THIRD_PARTY.md. + Companion root node: k_clean_latent_slice.py (CleanLatentSlice). Node IDs: the canonical `LTXDirector__koolook` rolls to this clean 2.0.2 implementation (package entrypoint loads v2_0_2 last). The legacy diff --git a/forks/whatdreamscost_koolook/versions/v2_0_2/ltx_director.py b/forks/whatdreamscost_koolook/versions/v2_0_2/ltx_director.py index 39ec948..4cf4626 100644 --- a/forks/whatdreamscost_koolook/versions/v2_0_2/ltx_director.py +++ b/forks/whatdreamscost_koolook/versions/v2_0_2/ltx_director.py @@ -45,6 +45,7 @@ from .patches import detect_model_type, apply_patches from .keyframe_grid import latent_count_for_duration as _latent_count_for_duration from .keyframe_grid import snap_keyframes_to_grid as _snap_keyframes_to_grid from .keyframe_grid import expand_keyframe_ease as _expand_keyframe_ease +from .reference_ghost import ghost_insert_frames as _ghost_insert_frames log = logging.getLogger(__name__) @@ -994,6 +995,29 @@ class LTXDirector(io.ComfyNode): "(less dwell), higher = gentler/longer ease." ), ), + io.Image.Input( + "reference_images", optional=True, + tooltip=( + "Koolook (Ghost Mask): optional reference image(s) / " + "character sheet to anchor identity (face, mouth shape) " + "the way a WAN VACE reference does. Each image is added as " + "a guide frame just past the clean timeline, so the model " + "attends to it for identity but it never appears in the " + "output. It rides the SAME guide path as your keyframes — " + "LTXDirectorGuide appends it and LTXDirectorCropGuides " + "removes it, so no extra wiring and the audio stays in " + "sync. No LoRA required. Lower reference_strength if the " + "reference suppresses motion." + ), + ), + io.Float.Input( + "reference_strength", default=1.0, min=0.0, max=5.0, step=0.05, optional=True, + tooltip=( + "Guide strength for the reference image(s). 1.0 = full " + "identity pull; lower it if the references bleed pose or " + "lighting into the video." + ), + ), ], outputs=[ io.Model.Output(display_name="model"), @@ -1004,6 +1028,18 @@ class LTXDirector(io.ComfyNode): MotionGuideData.Output(display_name="motion_guide_data"), io.Float.Output(display_name="frame_rate", tooltip="The frame rate used for the timeline."), io.Audio.Output(display_name="combined_audio", tooltip="Combined timeline audio layout."), + io.Int.Output( + display_name="clean_latent_frames", + tooltip=( + "Latent frame count of the clean video region (excludes " + "any appended reference frames). Wire to 'Clean Latent " + "Slice (Koolook)' length with start=0 to drop the refs." + ), + ), + io.Int.Output( + display_name="clean_pixel_frames", + tooltip="Pixel frame count of the clean video region.", + ), ], ) @@ -1014,7 +1050,8 @@ class LTXDirector(io.ComfyNode): custom_width=768, custom_height=512, resize_method="maintain aspect ratio", divisible_by=32, img_compression=0, audio_vae=None, optional_latent=None, use_custom_audio=False, inpaint_audio=True, use_custom_motion=True, override_audio=False, - snap_keyframes_to_grid=True, keyframe_ease=0, ease_falloff=0.5) -> io.NodeOutput: + snap_keyframes_to_grid=True, keyframe_ease=0, ease_falloff=0.5, + reference_images=None, reference_strength=1.0) -> io.NodeOutput: # Parse timeline data try: @@ -1101,7 +1138,9 @@ class LTXDirector(io.ComfyNode): insert_frame = max(0, seg_start + int(seg.get("length", 1)) - 1 - start_frame) else: insert_frame = max(0, seg_start - start_frame) - strength = strengths[idx] if idx < len(strengths) else 1.0 + # Koolook: unset guidance defaults to 0.8 (smooth transitions), + # not 1.0 (hard pin -> robotic snapping between keyframes). + strength = strengths[idx] if idx < len(strengths) else 0.8 guide_data["images"].append(tensor) guide_data["insert_frames"].append(insert_frame) guide_data["strengths"].append(float(strength)) @@ -1227,16 +1266,82 @@ class LTXDirector(io.ComfyNode): except Exception as _e: log.warning("[LTXDirector] keyframe ease skipped: %s", _e) - # --- Auto-generate LTXV latent if none was provided --- - # Apply the community 8n+1 rule directly to the timeline's duration_frames: - # int(ceil(((duration_frames) - 1) / 8) * 8) + 1 - # This ensures we get AT LEAST the requested frames, snapped to LTXV's requirements. + # --- Latent geometry --- + # Community 8n+1 rule on the timeline duration: ceil((duration-1)/8)*8 + 1 + # yields AT LEAST the requested pixel frames, snapped to LTXV requirements. ltxv_length = int(math.ceil((duration_frames - 1) / 8.0) * 8) + 1 - + # When the caller overrides the latent, the clean length (and therefore the + # Ghost-Mask reference geometry below AND the clean_*_frames outputs) must + # follow the ACTUAL latent, not the timeline duration — otherwise references + # are placed at the wrong frames and the reported counts are stale. + if optional_latent is not None: + try: + _lat_t = int(optional_latent["samples"].shape[2]) + ltxv_length = (_lat_t - 1) * 8 + 1 + except Exception as _e: + log.warning( + "[LTXDirector] optional_latent shape unreadable (%s); " + "clean length falls back to the timeline duration.", _e, + ) + clean_pixel_frames = int(ltxv_length) + clean_latent_frames = ((ltxv_length - 1) // 8) + 1 + + # --- Koolook (Ghost Mask): add reference image(s) as guide frames so the + # model anchors identity (face / mouth shape) without the references + # appearing in the output. They are added to guide_data exactly like the + # timeline keyframes and therefore ride the SAME append-and-crop path: + # the downstream LTXDirectorGuide appends each as an extra latent frame + # and LTXDirectorCropGuides removes them again (and the audio track is + # untouched, so A/V stays in sync). We must NOT pre-grow the video latent + # here — doing that leaves the references uncropped AND desyncs the audio + # track, which breaks motion. Positioned just past the clean timeline so + # they anchor identity without pinning a visible pose. Added AFTER + # snap/ease so refs are never grid-snapped or ease-expanded. The trick is + # adapted from CGlide's WhatDreamsCost-CSGlide (GPL-3.0). --- + n_refs = 0 + _ref_shape = getattr(reference_images, "shape", None) + if reference_images is not None and _ref_shape is not None and int(_ref_shape[0]) > 0: + try: + n_refs = int(_ref_shape[0]) + # Match references to the established output dimensions; fall back + # to the references' own size when nothing else set them. + ref_w = derived_w if derived_w > 0 else int(_ref_shape[2]) + ref_h = derived_h if derived_h > 0 else int(_ref_shape[1]) + _ref_inserts = _ghost_insert_frames(clean_latent_frames, n_refs) + for i in range(n_refs): + ref = reference_images[i:i + 1] + ref = _resize_image(ref, ref_w, ref_h, resize_method, divisible_by) + if img_compression > 0: + ref = _compress_image(ref, img_compression) + guide_data["images"].append(ref) + guide_data["insert_frames"].append(int(_ref_inserts[i])) + guide_data["strengths"].append(float(reference_strength)) + if derived_w <= 0: + derived_w = ref_w + if derived_h <= 0: + derived_h = ref_h + log.info( + "[LTXDirector] Ghost Mask: added %d reference guide frame(s) " + "past frame %d (strength=%.2f); they ride the guide " + "append-and-crop path (LTXDirectorCropGuides removes them).", + n_refs, clean_pixel_frames, float(reference_strength), + ) + except Exception as _e: + log.warning("[LTXDirector] reference Ghost Mask skipped: %s", _e) + n_refs = 0 + + # Tag how many of the trailing guide frames are references so a + # downstream node ("LTX Guide Reference Strength") can boost ONLY the + # references (e.g. harder pin in a stage-2 refinement) without touching + # the keyframe pins. References are always the LAST n_refs entries. + guide_data["reference_count"] = int(n_refs) + + # --- Auto-generate LTXV latent if none was provided --- + # Sized to the CLEAN length only — references are appended (and cropped) + # by the downstream guide pipeline, never by growing this latent. if optional_latent is None: latent_w = max(32, (derived_w // 32) * 32) latent_h = max(32, (derived_h // 32) * 32) - # LTXV temporal: ((length - 1) // 8) + 1 latent frames; invert to get pixel frames -> length latent_t = ((ltxv_length - 1) // 8) + 1 samples = torch.zeros( [1, 128, latent_t, latent_h // 32, latent_w // 32], @@ -1421,7 +1526,7 @@ class LTXDirector(io.ComfyNode): guide_data["duration_frames"] = duration_frames guide_data["resize_method"] = resize_method - return io.NodeOutput(patched, conditioning, latent, audio_latent, guide_data, motion_guide_data, float(frame_rate), audio_out) + return io.NodeOutput(patched, conditioning, latent, audio_latent, guide_data, motion_guide_data, float(frame_rate), audio_out, int(clean_latent_frames), int(clean_pixel_frames)) NODE_CLASS_MAPPINGS = { diff --git a/forks/whatdreamscost_koolook/versions/v2_0_2/reference_ghost.py b/forks/whatdreamscost_koolook/versions/v2_0_2/reference_ghost.py new file mode 100644 index 0000000..cb7cfdd --- /dev/null +++ b/forks/whatdreamscost_koolook/versions/v2_0_2/reference_ghost.py @@ -0,0 +1,74 @@ +# SPDX-License-Identifier: GPL-3.0-or-later +""" +Koolook-original pure helpers for the LTX Director "Ghost Mask" reference +feature (REF image / character sheet identity anchor). + +The trailing-reference trick — append the reference image(s) as guide frames +*after* the clean video region, let the transformer attend to them for +identity, then slice them off the output latent — is adapted from CGlide's +WhatDreamsCost-CSGlide (branch main_cs @ 21dd076, GPL-3.0). Only the integer +geometry lives here; the tensor work stays in the Director / the Clean Latent +Slice node. Pure stdlib so it unit-tests without torch or ComfyUI. + +Geometry recap (LTXV stride-8 temporal latent): + latent index of pixel frame f>0 : L(f) = 1 + (f - 1) // 8 + so pixel f = (clean_latent_frames + i) * 8 -> L(f) = clean_latent_frames + i +which is exactly the i-th trailing latent slot. We append one reference per +trailing slot, then the video latent must be grown to make room for them. +""" +from __future__ import annotations + +# LTXV temporal downscale stride. The Director's clean-latent math hardcodes +# the same 8, so the Ghost Mask geometry must agree with it. +LTXV_STRIDE = 8 + + +def ghost_insert_frames(clean_latent_frames: int, n_refs: int, stride: int = LTXV_STRIDE) -> list[int]: + """Pixel-space insert positions for ``n_refs`` Ghost Mask reference frames. + + Each reference pins the trailing latent slot ``clean_latent_frames + i``. + Returns ``[(clean_latent_frames + i) * stride for i in range(n_refs)]``. + """ + clf = int(clean_latent_frames) + n = int(n_refs) + s = int(stride) + if clf < 0: + raise ValueError(f"clean_latent_frames must be >= 0, got {clf}") + if n < 0: + raise ValueError(f"n_refs must be >= 0, got {n}") + if s < 1: + raise ValueError(f"stride must be >= 1, got {s}") + return [(clf + i) * s for i in range(n)] + + +def ghost_total_latent_frames(clean_latent_frames: int, n_refs: int) -> int: + """Total latent frame count once the reference slots are appended. + + The auto-generated video latent must be this long so the sampler produces + the trailing reference slots the guide node pins; they are removed + afterwards with the Clean Latent Slice node. + """ + clf = int(clean_latent_frames) + n = int(n_refs) + if clf < 0 or n < 0: + raise ValueError(f"counts must be >= 0, got clf={clf}, n_refs={n}") + return clf + n + + +def clean_slice_bounds(size: int, start: int, length: int) -> tuple[int, int]: + """Clamp a (start, length) request to a tensor axis of ``size`` frames. + + Mirrors the safety logic of the Clean Latent Slice node so it can be + unit-tested without torch. Returns ``(actual_start, actual_length)`` such + that ``0 <= actual_start < size`` (or 0 when empty) and the window never + runs past the end. For Ghost Mask the caller passes start=0, + length=clean_latent_frames to drop the trailing reference frames. + """ + sz = int(size) + if sz <= 0: + return 0, 0 + st = max(0, int(start)) + st = min(st, sz - 1) + ln = max(0, int(length)) + ln = min(ln, sz - st) + return st, ln diff --git a/k_clean_latent_slice.py b/k_clean_latent_slice.py new file mode 100644 index 0000000..84d0c5b --- /dev/null +++ b/k_clean_latent_slice.py @@ -0,0 +1,104 @@ +# SPDX-License-Identifier: GPL-3.0-or-later +# +# ComfyUI-Koolook — Clean Latent Slice node. +# Copyright (C) 2026 ComfyUI-Koolook contributors (kforgelabs). +"""Clean Latent Slice (Koolook) — trim frames off a video latent. + +Companion to the LTX Director "Ghost Mask" reference feature. That mode appends +reference image(s) as guide frames *after* the clean video region so the model +anchors identity (face / mouth shape) without the references appearing in the +output. After sampling, the trailing reference frames must be removed: wire the +Director's ``clean_latent_frames`` output to ``length`` (with ``start=0``) and +this node slices the latent back down to just the video. + +The trick — and the use of ``torch.narrow`` to avoid PyTorch NestedTensor +slicing issues — is adapted from CGlide's CleanLatentSliceCS +(WhatDreamsCost-CSGlide, branch main_cs @ 21dd076, GPL-3.0). + +The clamp math mirrors ``reference_ghost.clean_slice_bounds`` (unit-tested in +tests/forks); torch is imported lazily so this module loads without it. +""" +from __future__ import annotations + +import logging + +log = logging.getLogger(__name__) + + +def _clamp_slice(size: int, start: int, length: int) -> tuple[int, int]: + """Clamp (start, length) to a frame axis of ``size``. Mirrors + ``forks.whatdreamscost_koolook.versions.v2_0_2.reference_ghost.clean_slice_bounds``. + """ + sz = int(size) + if sz <= 0: + return 0, 0 + st = min(max(0, int(start)), sz - 1) + ln = min(max(0, int(length)), sz - st) + return st, ln + + +class CleanLatentSlice: + @classmethod + def INPUT_TYPES(cls): + return { + "required": { + "latent": ("LATENT",), + "start": ("INT", { + "default": 0, "min": 0, "max": 100000, "step": 1, + "tooltip": "First latent frame to keep. 0 for Ghost Mask (refs are at the end).", + }), + "length": ("INT", { + "default": 1, "min": 1, "max": 100000, "step": 1, + "tooltip": "Number of latent frames to keep. Wire the Director's 'clean_latent_frames'.", + }), + } + } + + RETURN_TYPES = ("LATENT",) + RETURN_NAMES = ("latent",) + FUNCTION = "slice_latent" + CATEGORY = "Koolook/LTX" + DESCRIPTION = ( + "Slice a video latent to [start, start+length) along the temporal axis. " + "Use it to drop the trailing reference frames added by LTX Director's " + "Ghost Mask reference feature (start=0, length=clean_latent_frames)." + ) + + def slice_latent(self, latent, start, length): + import torch # lazy: keep module importable without torch + + out = latent.copy() + + def _narrow(tensor): + dims = tensor.ndim if hasattr(tensor, "ndim") else len(tensor.shape) + # 5D [B, C, F, H, W] -> temporal dim 2; 4D/3D [F, ...] -> dim 0. + axis = 2 if dims == 5 else 0 + size = tensor.size(axis) + st, ln = _clamp_slice(size, start, length) + if ln <= 0: + return tensor + try: + return torch.narrow(tensor, axis, st, ln) + except Exception as e: + log.warning("[CleanLatentSlice] torch.narrow failed (%s); using python slice", e) + idx = [slice(None)] * dims + idx[axis] = slice(st, st + ln) + return tensor[tuple(idx)] + + if "samples" in out: + out["samples"] = _narrow(out["samples"]) + # Only slice a real per-frame mask, not a broadcasted (size-1) one. + if "noise_mask" in out: + nm = out["noise_mask"] + nm_dims = nm.ndim if hasattr(nm, "ndim") else len(nm.shape) + nm_axis = 2 if nm_dims == 5 else 0 + if nm.size(nm_axis) > 1: + out["noise_mask"] = _narrow(nm) + + return (out,) + + +NODE_CLASS_MAPPINGS = {"CleanLatentSlice": CleanLatentSlice} +NODE_DISPLAY_NAME_MAPPINGS = {"CleanLatentSlice": "Clean Latent Slice (Koolook)"} + +__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"] diff --git a/k_ltx_guide_reference_strength.py b/k_ltx_guide_reference_strength.py new file mode 100644 index 0000000..80969c9 --- /dev/null +++ b/k_ltx_guide_reference_strength.py @@ -0,0 +1,90 @@ +# SPDX-License-Identifier: GPL-3.0-or-later +# +# ComfyUI-Koolook — LTX Guide Reference Strength node. +# Copyright (C) 2026 ComfyUI-Koolook contributors (kforgelabs). +"""LTX Guide Reference Strength (Koolook) — boost ONLY the reference frames. + +Companion to LTX Director's Ghost Mask reference feature. The Director appends +the reference image(s) as the LAST entries in ``guide_data`` (tagged with +``reference_count``) so they ride the same append-and-crop path as the timeline +keyframes. Every guide frame's strength becomes a noise-mask pin +(``noise_mask = 1 - strength``) in the downstream LTXDirectorGuide. + +This node rewrites the strength of just those trailing reference entries, +leaving the keyframe pins untouched. The intended use is a two-stage pipeline: +feed stage 1 the Director's ``guide_data`` unchanged (light reference, so motion +forms freely), and route stage 2's guide_data through this node at a higher +reference strength — so the identity locks during the low-denoise refinement +without disturbing the motion already set in stage 1. + +Pure-python; no torch. The strength math is unit-tested. +""" +from __future__ import annotations + +import logging + +log = logging.getLogger(__name__) + + +def scaled_reference_strengths(strengths, reference_count, value, mode="absolute"): + """Return a new strengths list with the last ``reference_count`` entries + rewritten. ``mode='absolute'`` sets them to ``value``; ``mode='multiply'`` + multiplies the existing strength by ``value``. Other entries (keyframes) are + left untouched. Clamps to [0, 1] (guide strength -> noise-mask range).""" + out = [float(s) for s in strengths] + n = len(out) + k = max(0, min(int(reference_count), n)) + for i in range(n - k, n): + v = float(value) if mode == "absolute" else out[i] * float(value) + out[i] = max(0.0, min(1.0, v)) + return out + + +class LTXGuideReferenceStrength: + @classmethod + def INPUT_TYPES(cls): + return { + "required": { + "guide_data": ("GUIDE_DATA",), + "reference_strength": ("FLOAT", { + "default": 0.9, "min": 0.0, "max": 1.0, "step": 0.01, + "tooltip": "New strength for the reference frames only. Higher = harder identity pin. Use ~0.9 on a stage-2 refinement guide.", + }), + "mode": (["absolute", "multiply"], { + "default": "absolute", + "tooltip": "absolute: set reference strength to this value. multiply: scale the Director's reference strength by it.", + }), + } + } + + RETURN_TYPES = ("GUIDE_DATA",) + RETURN_NAMES = ("guide_data",) + FUNCTION = "apply" + CATEGORY = "Koolook/LTX" + DESCRIPTION = ( + "Rewrite the strength of just the Ghost Mask reference frames in " + "guide_data (the trailing entries tagged by LTX Director), leaving the " + "keyframe pins untouched. Insert before a stage-2 refinement guide to " + "lock identity harder without disturbing stage-1 motion." + ) + + def apply(self, guide_data, reference_strength, mode): + ref_count = int((guide_data or {}).get("reference_count", 0) or 0) + if not guide_data or ref_count <= 0: + log.info("[LTXGuideReferenceStrength] no reference frames in guide_data; passthrough.") + return (guide_data,) + out = dict(guide_data) + out["strengths"] = scaled_reference_strengths( + guide_data.get("strengths", []), ref_count, reference_strength, mode, + ) + log.info( + "[LTXGuideReferenceStrength] set %d reference frame(s) -> %.2f (%s).", + ref_count, float(reference_strength), mode, + ) + return (out,) + + +NODE_CLASS_MAPPINGS = {"LTXGuideReferenceStrength": LTXGuideReferenceStrength} +NODE_DISPLAY_NAME_MAPPINGS = {"LTXGuideReferenceStrength": "LTX Guide Reference Strength (Koolook)"} + +__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"] diff --git a/k_ltx_keyframe_soften_schedule.py b/k_ltx_keyframe_soften_schedule.py new file mode 100644 index 0000000..9ab0ebd --- /dev/null +++ b/k_ltx_keyframe_soften_schedule.py @@ -0,0 +1,208 @@ +# SPDX-License-Identifier: GPL-3.0-or-later +# +# ComfyUI-Koolook — LTX Keyframe Soften Schedule node. +# Copyright (C) 2026 ComfyUI-Koolook contributors (kforgelabs). +"""LTX Keyframe Soften Schedule (Koolook) — smooth motion early, sharp poses late. + +The LTX Director pins each keyframe by setting that latent frame's denoise mask +to ``1 - strength`` (a value < 1). The sampler applies it EVERY step +(``comfy/samplers.py``):: + + if "denoise_mask_function" in model_options: + denoise_mask = model_options["denoise_mask_function"](sigma, denoise_mask, ...) + x = x * denoise_mask + guide * (1 - denoise_mask) # the pin + +Hard pins held from step 0 force the model to hit exact poses immediately, so a +transition between two different poses resolves as a CUT. This node schedules the +pin by sigma instead: it **softens** the keyframe pins early (high sigma), when +the model is deciding coarse motion — letting it form a smooth trajectory between +the poses instead of snapping — then **restores** them by a configurable +``crossover`` measured in denoise *progress* (``1 - sigma/sigma_max``) so the poses +sharpen as denoise finishes. Because LTX's sigma schedule is front-loaded (sigma +held high, dropping late), a SMALL progress fraction already covers the early +motion-forming steps — hence the ``crossover`` widget maps its 0-1 range onto a +small internal ``0-0.05`` progress band (see ``_CROSSOVER_SCALE``). + +It adds NOTHING (no extra guide frames / poses — that approach backfires; every +guide is a pose target). It only loosens/tightens the keyframes you already have. + +Audio-safe: the A/V latent mask is a ``NestedTensor((video_mask, audio_mask))`` +(``LTXVConcatAVLatent``); we soften ONLY the video mask and pass the audio mask +through untouched, so lip-sync timing is never disturbed. + +NOTE: the pure ``soften_gain`` ramp is unit-tested; the model wiring (the +denoise-mask function) is validated by rendering against a live LTX 2.3 model. +""" +from __future__ import annotations + +import logging + +log = logging.getLogger(__name__) + +# The node widgets are normalized 0-1 sliders that map onto the *useful* internal +# range (the effect is very sensitive, so the raw values bunch up near 0). Widget +# value * SCALE = the internal value passed to soften_gain(). +_MAX_SOFTEN_SCALE = 0.2 # widget 0..1 -> internal max_soften 0..0.2 +_CROSSOVER_SCALE = 0.05 # widget 0..1 -> internal crossover 0..0.05 + + +def soften_gain(sigma, sigma_max, crossover=0.55, max_soften=0.6): + """How much to soften the keyframe pins at the current ``sigma``. + + Denoise progress runs 0 at the start (``sigma == sigma_max``) to 1 at the end + (``sigma -> 0``). Returns ``max_soften`` at progress 0, smoothsteps down to 0 + by ``crossover`` progress, and stays 0 afterwards (pins fully restored). + + A returned gain ``g`` raises a pinned mask value ``m`` toward 1 (unpinned) + via ``m + (1 - m) * g``: g=0 keeps the exact pin, g=1 fully frees it. + """ + sigma_max = float(sigma_max) + if sigma_max <= 0.0: + return 0.0 + progress = 1.0 - (float(sigma) / sigma_max) + progress = 0.0 if progress < 0.0 else (1.0 if progress > 1.0 else progress) + crossover = float(crossover) + if crossover <= 0.0 or progress >= crossover: + return 0.0 + t = progress / crossover # 0 at start -> 1 at crossover + s = t * t * (3.0 - 2.0 * t) # smoothstep 0 -> 1 + return float(max_soften) * (1.0 - s) # max_soften at start -> 0 at crossover + + +def soften_mask_value(mask_value, gain): + """Raise a single denoise-mask value toward 1 (unpinned) by ``gain``.""" + return mask_value + (1.0 - mask_value) * float(gain) + + +def soften_denoise_mask(denoise_mask, gain, nested_ctor=None): + """Soften a denoise mask, sparing the audio half of an A/V mask. + + The A/V latent mask is a ``NestedTensor((video_mask, audio_mask))`` produced by + ``LTXVConcatAVLatent`` — video FIRST, audio SECOND. We soften ONLY part 0 + (video keyframe pins) and pass part 1 (audio) through untouched, so lip-sync + timing is never disturbed. A plain (video-only) mask is softened whole. + + ``nested_ctor`` rebuilds the nested mask; defaults to + ``comfy.nested_tensor.NestedTensor`` and is injectable so the audio-safety + behavior can be unit-tested with a stub. ``gain<=0`` / ``None`` is a no-op. + """ + if gain <= 0.0 or denoise_mask is None: + return denoise_mask + if getattr(denoise_mask, "is_nested", False): + parts = list(denoise_mask.unbind()) + parts[0] = soften_mask_value(parts[0], gain) # video only; audio untouched + if nested_ctor is None: + import comfy.nested_tensor + nested_ctor = comfy.nested_tensor.NestedTensor + return nested_ctor(tuple(parts)) + return soften_mask_value(denoise_mask, gain) + + +class LTXKeyframeSoftenSchedule: + """Sigma-schedule the LTX keyframe pins: soft early (smooth motion), sharp late.""" + + @classmethod + def INPUT_TYPES(cls): + return { + "required": { + "model": ("MODEL", {"tooltip": "LTX 2.3 model (place after the other model patchers)."}), + "max_soften": ("FLOAT", { + "default": 0.5, "min": 0.0, "max": 1.0, "step": 0.01, + "tooltip": "EASE amount (normalized 0-1; maps to internal 0-0.20). " + "How gently the keyframe pins loosen at the START of " + "denoise — the smooth approach into each key. 0 = stock " + "(no ease). Higher = smoother, but also more invented " + "in-between motion. Default 0.5 = internal 0.10.", + }), + "crossover": ("FLOAT", { + "default": 0.65, "min": 0.0, "max": 1.0, "step": 0.01, + "tooltip": "INVENTION window (normalized 0-1; maps to internal " + "0-0.05). How long the pins stay loose before re-locking. " + "LOWER = poses lock back sooner = FEWER invented in-between " + "frames (truer to your keyframes). Higher = more invented " + "motion (and risk of a noisy/under-resolved image). " + "Default 0.65 = internal ~0.033.", + }), + }, + } + + RETURN_TYPES = ("MODEL",) + RETURN_NAMES = ("model",) + FUNCTION = "patch" + CATEGORY = "Koolook/LTX" + DESCRIPTION = ( + "Smooth motion early, sharp poses late: sigma-schedule the LTX keyframe " + "pins — loosen them at high sigma so the model forms smooth motion between " + "poses, restore them by a crossover (~step 9) so the poses sharpen. Adds no " + "frames; audio mask untouched (lip-sync safe)." + ) + + def patch(self, model, max_soften, crossover): + m = model.clone() + + # Widgets are normalized 0-1; map onto the sensitive internal range. + max_soften_w, crossover_w = float(max_soften), float(crossover) + max_soften = max_soften_w * _MAX_SOFTEN_SCALE + crossover = crossover_w * _CROSSOVER_SCALE + + try: + sigma_max = float(m.get_model_object("model_sampling").sigma_max) + except Exception as e: # pragma: no cover - live model + log.warning("[LTXKeyframeSoftenSchedule] no model_sampling.sigma_max: %s", e) + sigma_max = 1.0 + + if max_soften <= 0.0: + log.info("[LTXKeyframeSoftenSchedule] max_soften=0 — model passed through.") + return (m,) + + # Chain any pre-existing denoise_mask_function instead of clobbering it + # (mirrors how the sibling LTX patchers chain model_function_wrapper). + prev_fn = m.model_options.get("denoise_mask_function") + diag = [False] + warned = [False] + + def denoise_mask_function(sigma, denoise_mask, extra_options=None): + if prev_fn is not None: + denoise_mask = prev_fn(sigma, denoise_mask, extra_options=extra_options) + if denoise_mask is None: + return denoise_mask + try: + s = float(sigma.max()) if hasattr(sigma, "max") else float(sigma) + except Exception as _e: + s = sigma_max + if not warned[0]: + warned[0] = True + log.warning( + "[LTXKeyframeSoftenSchedule] could not read sigma (%s); " + "softening at max this step.", _e, + ) + gain = soften_gain(s, sigma_max, crossover, max_soften) + if gain <= 0.0: + return denoise_mask + if not diag[0]: + diag[0] = True + nested = getattr(denoise_mask, "is_nested", False) + log.info( + "[LTXKeyframeSoftenSchedule] active (%s): softening video keyframe " + "pins%s. max_soften=%.3f crossover=%.3f", + "A/V" if nested else "video-only", + ", audio untouched" if nested else "", max_soften, crossover, + ) + return soften_denoise_mask(denoise_mask, gain) + + m.set_model_denoise_mask_function(denoise_mask_function) + log.info( + "[LTXKeyframeSoftenSchedule] patched: max_soften=%.2f (->%.3f) " + "crossover=%.2f (->%.3f) sigma_max=%.3f", + max_soften_w, max_soften, crossover_w, crossover, sigma_max, + ) + return (m,) + + +NODE_CLASS_MAPPINGS = {"LTXKeyframeSoftenSchedule": LTXKeyframeSoftenSchedule} +NODE_DISPLAY_NAME_MAPPINGS = {"LTXKeyframeSoftenSchedule": "LTX Keyframe Soften Schedule (Koolook)"} + +__all__ = [ + "NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS", + "soften_gain", "soften_mask_value", "soften_denoise_mask", +] diff --git a/k_ltx_reference_bind_schedule.py b/k_ltx_reference_bind_schedule.py new file mode 100644 index 0000000..0dde9c3 --- /dev/null +++ b/k_ltx_reference_bind_schedule.py @@ -0,0 +1,286 @@ +# SPDX-License-Identifier: GPL-3.0-or-later +# +# ComfyUI-Koolook — LTX Reference Bind Schedule node. +# Copyright (C) 2026 ComfyUI-Koolook contributors (kforgelabs). +"""LTX Reference Bind Schedule (Koolook) — ramp reference identity up at low sigma. + +Companion to the LTX Director Ghost Mask reference. The reference image is added +as a guide frame; the core LTX transformer attends to it with a per-guide +**attention strength** (`guide_attention_entries[].strength`) which it turns into +a log-space additive self-attention bias, rebuilt **every denoise step** in +`comfy/ldm/lightricks/model.py::_build_guide_self_attention_mask`. That builder +explicitly supports `strength > 1.0` to *amplify* a guide's attention. + +This node schedules that amplification over the denoise so identity locks late +without disturbing motion that forms early: + +* **Early steps (high sigma)** — gain ~1.0: the reference attention is neutral, + big motion resolves freely. +* **Late steps (low sigma)** — gain ramps up to `peak_strength`: the refinement + conforms harder to the reference's identity (face / mouth). + +It's the inverse of the motion curve, and the mirror of `LTXAVBindSchedule` +(which ramps audio->video up at low sigma for lip-sync). It touches ONLY the +attention bias of the trailing `num_references` guide entries (references are +appended after the keyframes) — the keyframe pins and the noise-mask freezing +are untouched. + +WORKS WITHOUT AN IC-LoRA. The attention-bias machinery is BASE-model +functionality — `_build_guide_self_attention_mask` builds the mask from +`resolved_guide_entries` regardless of any LoRA. The catch is that +LTXDirectorGuide only *populates* those entries `if is_lora_active`, so in the +no-LoRA Ghost Mask path the list is empty. This node closes that gap: when the +entries are missing it **fabricates them itself** for the trailing reference +tokens (splitting `num_guide_tokens` across the `num_keyframes + num_references` +guide frames and assigning the ramped strength to the reference share). So the +reference gets the same amplified-attention channel an IC-LoRA would use, inside +the Director, with no LoRA, no guide-node vendor, and no graph change. When an +IC-LoRA *is* active and real entries exist, it scales those instead. + +Mechanics (same as the A/V node): a `model_function_wrapper` captures the +current sigma each step and stashes the gain in `transformer_options`; an +object-patch on `_build_guide_self_attention_mask` reads it and fabricates (or +scales) the reference entries' `strength` before the mask is built. + +NOTE: the `ref_gain` ramp is unit-tested; the model wiring is validated by +rendering against a live LTX 2.3 model (not in CI). +""" +from __future__ import annotations + +import types +import logging + +log = logging.getLogger(__name__) + +_REF_GAIN_KEY = "nghtdrp_ref_gain" # per-step reference-attention multiplier + + +def ref_gain(sigma, sigma_max, ramp_start=0.5, ramp_end=0.9, peak=2.0): + """Reference-attention multiplier for the current ``sigma``. + + Progress runs 0 at the start of denoise (``sigma == sigma_max``) to 1 at the + end (``sigma -> 0``). Holds at 1.0 (neutral) until ``ramp_start``, then + smoothsteps up to ``peak`` by ``ramp_end`` (progress fractions in [0, 1]). + Returning 1.0 means "no change" — the mask builder then skips entirely. + """ + if sigma_max <= 0.0: + return float(peak) + progress = 1.0 - (float(sigma) / float(sigma_max)) + progress = 0.0 if progress < 0.0 else (1.0 if progress > 1.0 else progress) + if ramp_end <= ramp_start: + return float(peak) if progress >= ramp_end else 1.0 + if progress <= ramp_start: + return 1.0 + if progress >= ramp_end: + return float(peak) + t = (progress - ramp_start) / (ramp_end - ramp_start) + s = t * t * (3.0 - 2.0 * t) # smoothstep + return 1.0 + (float(peak) - 1.0) * s + + +def _boosted_entries(resolved_entries, ref_count, gain): + """Return a copy of ``resolved_entries`` with the LAST ``ref_count`` entries' + ``strength`` multiplied by ``gain`` (references are appended after keyframes). + Returns the original list unchanged when there's nothing to do.""" + if not resolved_entries or gain == 1.0 or ref_count <= 0: + return resolved_entries + n = len(resolved_entries) + k = min(int(ref_count), n) + if k <= 0: + return resolved_entries + out = list(resolved_entries) + for i in range(n - k, n): + e = dict(out[i]) + e["strength"] = float(e.get("strength", 1.0)) * float(gain) + out[i] = e + return out + + +def fabricate_reference_entries(num_guide_tokens, num_references, num_keyframes, strength): + """Build ``resolved_guide_entries`` for the trailing reference tokens when the + guide node created none (the no-IC-LoRA path). + + Splits ``num_guide_tokens`` evenly across the ``num_keyframes + num_references`` + guide frames and gives the trailing reference share ``strength`` while the + keyframe share stays 1.0 (neutral). The entries match the shape the core + builder consumes (``strength`` + ``surviving_count`` + optional pixel_mask / + latent_shape). When ``num_keyframes`` is 0 the whole guide region is treated + as reference. Returns ``[]`` when there's nothing to amplify. + """ + n_guide = int(num_guide_tokens) + n_ref = int(num_references) + n_kf = max(0, int(num_keyframes)) + if n_guide <= 0 or n_ref <= 0: + return [] + total_frames = n_ref + n_kf + ref_tokens = round(n_guide * n_ref / total_frames) if total_frames > 0 else n_guide + ref_tokens = max(1, min(int(ref_tokens), n_guide)) + kf_tokens = n_guide - ref_tokens + entries = [] + if kf_tokens > 0: + entries.append({"strength": 1.0, "surviving_count": kf_tokens, "pixel_mask": None, "latent_shape": None}) + entries.append({"strength": float(strength), "surviving_count": ref_tokens, "pixel_mask": None, "latent_shape": None}) + return entries + + +def _make_guide_mask_wrapper(underlying, ref_count, kf_count, peak, diag): + """Bound-method wrapper around ``_build_guide_self_attention_mask``. + + If the guide node already populated entries (IC-LoRA path) scale the trailing + ``ref_count`` of them by the per-step gain. If not (no-LoRA Ghost Mask path) + fabricate entries for the trailing reference tokens at the gain strength. Then + hand off to the original builder, which turns it into the additive bias. + """ + + def wrapped(_self_module, x, transformer_options, merged_args, *args, **kwargs): + to = transformer_options or {} + gain = float(to.get(_REF_GAIN_KEY, 1.0)) + ma = merged_args if isinstance(merged_args, dict) else {} + entries = ma.get("resolved_guide_entries") + n_guide = int(ma.get("num_guide_tokens") or 0) + + mode = "passthrough" + if entries: # IC-LoRA path — scale real entries + if gain != 1.0 and ref_count > 0: + boosted = _boosted_entries(entries, ref_count, gain) + if boosted is not entries: + merged_args = {**ma, "resolved_guide_entries": boosted} + mode = "scaled" + elif n_guide > 0 and ref_count > 0 and gain != 1.0: # no-LoRA path — fabricate + fab = fabricate_reference_entries(n_guide, ref_count, kf_count, gain) + if fab: + merged_args = {**ma, "resolved_guide_entries": fab} + mode = "fabricated" + + if not diag.get("first"): + diag["first"] = True + log.info( + "[LTXReferenceBindSchedule] active: num_guide_tokens=%s entries=%s " + "ref_count=%d num_keyframes=%d peak=%.2f (amplifies the trailing " + "reference tokens at low sigma).", + n_guide, (len(entries) if entries else 0), int(ref_count), + int(kf_count), float(peak), + ) + if mode in ("scaled", "fabricated") and not diag.get(mode): + diag[mode] = True + log.info("[LTXReferenceBindSchedule] %s reference attention (gain x%.2f).", mode, gain) + return underlying(x, transformer_options, merged_args, *args, **kwargs) + + return wrapped + + +class LTXReferenceBindSchedule: + """Ramp the LTX Ghost Mask reference's attention up as sigma falls.""" + + @classmethod + def INPUT_TYPES(cls): + return { + "required": { + "model": ("MODEL", {"tooltip": "LTX 2.3 model (the same one feeding the guide pipeline)."}), + "num_references": ("INT", { + "default": 1, "min": 0, "max": 64, "step": 1, + "tooltip": "How many trailing guide frames are references (= how many images you fed LTX Director's reference_images). They're appended after the keyframes, so the LAST N guide frames are the references.", + }), + "peak_strength": ("FLOAT", { + "default": 2.0, "min": 1.0, "max": 8.0, "step": 0.05, + "tooltip": "Reference attention multiplier at the end of denoise (low sigma). 1.0 = no boost; 2.0 = double the identity pull during refinement. Try 2-4.", + }), + }, + "optional": { + # NOTE: ramp_start/ramp_end stay first to preserve widget order + # for workflows saved before num_keyframes existed; num_keyframes + # is appended last so old [num_references, peak_strength, ramp_start, + # ramp_end] values keep aligning. + "ramp_start": ("FLOAT", { + "default": 0.5, "min": 0.0, "max": 1.0, "step": 0.01, + "tooltip": "Denoise progress (0=start, 1=end) where the reference boost begins. Keep above where big motion finalizes (~0.5-0.6) so motion forms first.", + }), + "ramp_end": ("FLOAT", { + "default": 0.9, "min": 0.0, "max": 1.0, "step": 0.01, + "tooltip": "Denoise progress where the boost reaches peak_strength.", + }), + "num_keyframes": ("INT", { + "default": 0, "min": 0, "max": 256, "step": 1, + "tooltip": "No-LoRA path only: how many timeline image keyframes precede the references, so the node knows the keyframe/reference token split. Set this = your number of timeline image segments (e.g. 4). Leave 0 and the WHOLE guide region is treated as reference (also boosts keyframes). Ignored when an IC-LoRA populated real entries.", + }), + }, + } + + RETURN_TYPES = ("MODEL",) + RETURN_NAMES = ("model",) + FUNCTION = "patch" + CATEGORY = "Koolook/LTX" + DESCRIPTION = ( + "Ramp the reference's guide-attention UP at low sigma so identity locks " + "during refinement without disturbing early motion. Works WITHOUT an " + "IC-LoRA: it fabricates the per-guide attention entries for the trailing " + "reference tokens itself (set num_keyframes so it knows the split), so the " + "reference gets the same amplified-attention channel an IC-LoRA would use " + "— inside the Director, no guide-node changes. Mirror of LTX A/V Bind " + "Schedule. One pass, no re-noise." + ) + + def patch(self, model, num_references, peak_strength, num_keyframes=0, ramp_start=0.5, ramp_end=0.9): + m = model.clone() + + if int(num_references) <= 0: + log.info("[LTXReferenceBindSchedule] num_references=0 — model passed through.") + return (m,) + + try: + sigma_max = float(m.get_model_object("model_sampling").sigma_max) + except Exception as e: # pragma: no cover - live model + log.warning("[LTXReferenceBindSchedule] no model_sampling.sigma_max: %s", e) + sigma_max = 1.0 + + key = "diffusion_model._build_guide_self_attention_mask" + try: + underlying = m.get_model_object(key) + dm = m.get_model_object("diffusion_model") + except Exception as e: # pragma: no cover - live model + log.warning( + "[LTXReferenceBindSchedule] no %s on this model (not an LTX guide " + "model?): %s — passed through unchanged.", key, e, + ) + return (m,) + + diag = {} + wrapper = _make_guide_mask_wrapper( + underlying, int(num_references), int(num_keyframes), float(peak_strength), diag, + ) + try: + m.add_object_patch(key, types.MethodType(wrapper, dm)) + except Exception as e: # pragma: no cover - live model + log.warning("[LTXReferenceBindSchedule] guide-mask patch failed: %s", e) + return (m,) + + prev = m.model_options.get("model_function_wrapper") + + def unet_wrapper(model_function, params): + try: + sigma = float(params["timestep"].max()) + except Exception: + sigma = sigma_max + gain = ref_gain(sigma, sigma_max, ramp_start, ramp_end, peak_strength) + c = params["c"] + to = dict(c.get("transformer_options", {})) + to[_REF_GAIN_KEY] = gain + c = {**c, "transformer_options": to} + params = {**params, "c": c} + if prev is not None: + return prev(model_function, params) + return model_function(params["input"], params["timestep"], **params["c"]) + + m.set_model_unet_function_wrapper(unet_wrapper) + log.info( + "[LTXReferenceBindSchedule] scheduling %d reference(s): peak x%.2f, " + "ramp %.2f..%.2f of denoise.", int(num_references), float(peak_strength), + float(ramp_start), float(ramp_end), + ) + return (m,) + + +NODE_CLASS_MAPPINGS = {"LTXReferenceBindSchedule": LTXReferenceBindSchedule} +NODE_DISPLAY_NAME_MAPPINGS = {"LTXReferenceBindSchedule": "LTX Reference Bind Schedule (Koolook)"} + +__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"] diff --git a/scripts/sync_to_dev.py b/scripts/sync_to_dev.py index e4c257c..dfb4b11 100644 --- a/scripts/sync_to_dev.py +++ b/scripts/sync_to_dev.py @@ -155,6 +155,7 @@ RUNTIME_PATHS: tuple[str, ...] = ( "__init__.py", "config.json", "k_ai_pipeline.py", + "k_clean_latent_slice.py", "k_easy_image_batch.py", "k_easy_pattern.py", "k_easy_resize.py", @@ -163,6 +164,9 @@ RUNTIME_PATHS: tuple[str, ...] = ( "k_easy_wan22_prompt.py", "k_loop_status.py", "k_ltx_av_bind_schedule.py", + "k_ltx_guide_reference_strength.py", + "k_ltx_keyframe_soften_schedule.py", + "k_ltx_reference_bind_schedule.py", "k_publish_contract.py", "k_video_combine.py", "k_video_load.py", diff --git a/tests/forks/test_whatdreamscost_v2_0_2_reference_ghost.py b/tests/forks/test_whatdreamscost_v2_0_2_reference_ghost.py new file mode 100644 index 0000000..56b40d7 --- /dev/null +++ b/tests/forks/test_whatdreamscost_v2_0_2_reference_ghost.py @@ -0,0 +1,99 @@ +"""Tests for the v2_0_2 Ghost Mask reference geometry helpers. + +Lives in +``forks/whatdreamscost_koolook/versions/v2_0_2/reference_ghost.py`` and is +pure stdlib, so we load it by file path — the same isolation trick the other +fork tests use to avoid triggering the package ``__init__`` (which imports +``ltx_director`` -> ``comfy_api``, unavailable inside pytest). +""" +from __future__ import annotations + +import importlib.util +from pathlib import Path + +import pytest + + +_MOD_PATH = ( + Path(__file__).resolve().parents[2] + / "forks" + / "whatdreamscost_koolook" + / "versions" + / "v2_0_2" + / "reference_ghost.py" +) +_spec = importlib.util.spec_from_file_location( + "whatdreamscost_v2_0_2_reference_ghost", _MOD_PATH +) +_mod = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(_mod) + +ghost_insert_frames = _mod.ghost_insert_frames +ghost_total_latent_frames = _mod.ghost_total_latent_frames +clean_slice_bounds = _mod.clean_slice_bounds + + +# --- ghost_insert_frames --------------------------------------------------- + +def test_insert_frames_pin_trailing_latent_slots(): + # clean region is 10 latent frames; two refs -> latent slots 10 and 11. + # pixel = slot * 8, and L(pixel) = 1 + (pixel-1)//8 must equal the slot. + frames = ghost_insert_frames(10, 2) + assert frames == [80, 88] + for i, f in enumerate(frames): + latent_idx = 1 + (f - 1) // 8 + assert latent_idx == 10 + i + + +def test_insert_frames_empty_when_no_refs(): + assert ghost_insert_frames(10, 0) == [] + + +def test_insert_frames_custom_stride(): + assert ghost_insert_frames(3, 2, stride=4) == [12, 16] + + +def test_insert_frames_rejects_negative(): + with pytest.raises(ValueError): + ghost_insert_frames(-1, 1) + with pytest.raises(ValueError): + ghost_insert_frames(1, -1) + with pytest.raises(ValueError): + ghost_insert_frames(1, 1, stride=0) + + +# --- ghost_total_latent_frames --------------------------------------------- + +def test_total_latent_frames_adds_refs(): + assert ghost_total_latent_frames(10, 0) == 10 + assert ghost_total_latent_frames(10, 3) == 13 + + +def test_total_latent_frames_rejects_negative(): + with pytest.raises(ValueError): + ghost_total_latent_frames(-1, 0) + with pytest.raises(ValueError): + ghost_total_latent_frames(0, -1) + + +# --- clean_slice_bounds ---------------------------------------------------- + +def test_slice_bounds_drops_trailing_refs(): + # total 13 frames (10 clean + 3 refs); keep [0, 10). + assert clean_slice_bounds(13, 0, 10) == (0, 10) + + +def test_slice_bounds_clamps_overlong_length(): + assert clean_slice_bounds(10, 0, 999) == (0, 10) + + +def test_slice_bounds_clamps_start_into_range(): + assert clean_slice_bounds(10, 50, 5) == (9, 1) + + +def test_slice_bounds_empty_tensor(): + assert clean_slice_bounds(0, 0, 5) == (0, 0) + + +def test_slice_bounds_negative_inputs_floored(): + assert clean_slice_bounds(10, -3, -2) == (0, 0) diff --git a/tests/nodes/test_clean_latent_slice.py b/tests/nodes/test_clean_latent_slice.py new file mode 100644 index 0000000..43f0345 --- /dev/null +++ b/tests/nodes/test_clean_latent_slice.py @@ -0,0 +1,79 @@ +"""Unit tests for the Clean Latent Slice node. + +The clamp math (`_clamp_slice`) is stdlib and always tested. The torch.narrow +slice is exercised only when torch is importable (skipped in the stdlib CI +env, validated locally / on a torch machine). Loaded by file path to avoid +importing the package __init__ (which pulls in ComfyUI). +""" +from __future__ import annotations + +import importlib.util +from pathlib import Path + +import pytest + + +_MOD = Path(__file__).resolve().parents[2] / "k_clean_latent_slice.py" +_spec = importlib.util.spec_from_file_location("k_clean_latent_slice_under_test", _MOD) +_m = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(_m) +clamp = _m._clamp_slice + + +def _real_torch(): + """Skip unless a real torch is installed. + + Other node tests inject a bare ``types.ModuleType("torch")`` into + ``sys.modules`` that persists for the session, so ``importorskip`` alone + would find that hollow stub. Real torch carries ``__version__``; the stub + does not. + """ + torch = pytest.importorskip("torch") + if not getattr(torch, "__version__", None): + pytest.skip("real torch not installed (hollow stub present)") + return torch + + +# --- _clamp_slice ---------------------------------------------------------- + +def test_clamp_keeps_clean_region(): + # 13 frames (10 clean + 3 refs), drop the refs. + assert clamp(13, 0, 10) == (0, 10) + + +def test_clamp_overlong_length(): + assert clamp(10, 0, 999) == (0, 10) + + +def test_clamp_start_past_end(): + assert clamp(10, 50, 5) == (9, 1) + + +def test_clamp_empty_axis(): + assert clamp(0, 0, 5) == (0, 0) + + +def test_clamp_negatives_floored(): + assert clamp(10, -3, -2) == (0, 0) + + +# --- slice_latent (torch) -------------------------------------------------- + +def test_slice_latent_drops_trailing_frames(): + torch = _real_torch() + node = _m.CleanLatentSlice() + # [B, C, F, H, W] with F = 5 (3 clean + 2 refs) + samples = torch.arange(5).float().reshape(1, 1, 5, 1, 1) + (out,) = node.slice_latent({"samples": samples}, 0, 3) + assert out["samples"].shape[2] == 3 + assert out["samples"].flatten().tolist() == [0.0, 1.0, 2.0] + + +def test_slice_latent_skips_broadcast_noise_mask(): + torch = _real_torch() + node = _m.CleanLatentSlice() + samples = torch.zeros(1, 1, 5, 1, 1) + bmask = torch.ones(1, 1, 1, 1, 1) # broadcasted, size-1 temporal axis + (out,) = node.slice_latent({"samples": samples, "noise_mask": bmask}, 0, 3) + assert out["samples"].shape[2] == 3 + assert out["noise_mask"].shape[2] == 1 # untouched diff --git a/tests/nodes/test_ltx_guide_reference_strength.py b/tests/nodes/test_ltx_guide_reference_strength.py new file mode 100644 index 0000000..158fb33 --- /dev/null +++ b/tests/nodes/test_ltx_guide_reference_strength.py @@ -0,0 +1,64 @@ +"""Unit tests for the LTX Guide Reference Strength helper. + +Pure stdlib. Loaded by file path to avoid importing the package __init__ +(which pulls in ComfyUI). +""" +from __future__ import annotations + +import importlib.util +from pathlib import Path + + +_MOD = Path(__file__).resolve().parents[2] / "k_ltx_guide_reference_strength.py" +_spec = importlib.util.spec_from_file_location("k_ltx_guide_reference_strength_under_test", _MOD) +_m = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(_m) +scale = _m.scaled_reference_strengths + + +def test_absolute_rewrites_only_trailing_refs(): + # 4 keyframes (0.8) + 2 refs (0.5); boost refs to 0.9, keyframes untouched. + s = [0.8, 0.8, 0.8, 0.8, 0.5, 0.5] + assert scale(s, 2, 0.9, "absolute") == [0.8, 0.8, 0.8, 0.8, 0.9, 0.9] + + +def test_multiply_scales_trailing_refs(): + s = [0.8, 0.8, 0.4, 0.4] + assert scale(s, 2, 2.0, "multiply") == [0.8, 0.8, 0.8, 0.8] + + +def test_multiply_clamps_to_one(): + s = [0.8, 0.5] + assert scale(s, 1, 5.0, "multiply") == [0.8, 1.0] + + +def test_zero_reference_count_is_noop(): + s = [0.8, 0.8, 0.8] + assert scale(s, 0, 0.9, "absolute") == [0.8, 0.8, 0.8] + + +def test_reference_count_capped_at_length(): + s = [0.5, 0.5] + assert scale(s, 9, 0.9, "absolute") == [0.9, 0.9] + + +def test_does_not_mutate_input(): + s = [0.8, 0.5] + _ = scale(s, 1, 0.9, "absolute") + assert s == [0.8, 0.5] + + +def test_node_passthrough_when_no_refs(): + node = _m.LTXGuideReferenceStrength() + gd = {"strengths": [0.8, 0.8], "reference_count": 0} + (out,) = node.apply(gd, 0.9, "absolute") + assert out["strengths"] == [0.8, 0.8] + + +def test_node_boosts_refs_and_copies(): + node = _m.LTXGuideReferenceStrength() + gd = {"strengths": [0.8, 0.5], "reference_count": 1, "images": ["x", "y"]} + (out,) = node.apply(gd, 0.9, "absolute") + assert out["strengths"] == [0.8, 0.9] + assert gd["strengths"] == [0.8, 0.5] # original untouched + assert out["images"] == ["x", "y"] # other keys preserved diff --git a/tests/nodes/test_ltx_keyframe_soften_schedule.py b/tests/nodes/test_ltx_keyframe_soften_schedule.py new file mode 100644 index 0000000..97970e5 --- /dev/null +++ b/tests/nodes/test_ltx_keyframe_soften_schedule.py @@ -0,0 +1,121 @@ +"""Unit tests for the LTX Keyframe Soften Schedule ramp. + +The pure ``soften_gain`` / ``soften_mask_value`` helpers are stdlib-only and +always tested; the model wiring (the denoise_mask_function set on the patcher, +which softens the video keyframe pins early and restores them by the crossover) +is validated by rendering against a live LTX 2.3 model. Loaded by file path to +avoid importing the package __init__. +""" +from __future__ import annotations + +import importlib.util +from pathlib import Path + +import pytest + + +_MOD = Path(__file__).resolve().parents[2] / "k_ltx_keyframe_soften_schedule.py" +_spec = importlib.util.spec_from_file_location("k_ltx_keyframe_soften_schedule_under_test", _MOD) +_m = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(_m) +gain = _m.soften_gain +soften = _m.soften_mask_value +soften_mask = _m.soften_denoise_mask + +SMAX = 10.0 + + +class _FakeNested: + """Minimal NestedTensor stub: holds (video, audio) parts, exposes is_nested + unbind.""" + is_nested = True + + def __init__(self, parts): + self.parts = tuple(parts) + + def unbind(self): + return self.parts + + +# --- soften_gain ramp (max early, 0 by crossover) -------------------------- + +def test_max_soften_at_denoise_start(): + # sigma == sigma_max -> progress 0 -> full max_soften + assert gain(SMAX, SMAX, 0.5, 0.6) == pytest.approx(0.6) + + +def test_zero_after_crossover(): + # sigma -> 0 (progress 1) is past the crossover -> fully restored (0) + assert gain(0.0, SMAX, 0.5, 0.6) == 0.0 + # exactly at the crossover -> 0 + assert gain(SMAX * 0.5, SMAX, 0.5, 0.6) == 0.0 + + +def test_midpoint_of_ramp(): + # progress 0.25 is the midpoint of [0, 0.5] -> smoothstep 0.5 -> half max + assert gain(SMAX * 0.75, SMAX, 0.5, 0.6) == pytest.approx(0.3) + + +def test_monotonic_non_increasing_as_sigma_falls(): + prev = 1.0 + for i in range(0, 21): + s = SMAX * (1.0 - i / 20.0) + g = gain(s, SMAX, 0.55, 0.6) + assert g <= prev + 1e-9 + prev = g + + +def test_max_soften_zero_is_noop(): + assert gain(SMAX, SMAX, 0.55, 0.0) == 0.0 + + +def test_crossover_zero_is_noop(): + assert gain(SMAX, SMAX, 0.0, 0.6) == 0.0 + + +def test_degenerate_sigma_max(): + assert gain(1.0, 0.0, 0.5, 0.6) == 0.0 + + +# --- soften_mask_value (raise a pin toward 1) ------------------------------ + +def test_gain_zero_keeps_exact_pin(): + assert soften(0.2, 0.0) == pytest.approx(0.2) + + +def test_gain_one_fully_frees(): + assert soften(0.2, 1.0) == pytest.approx(1.0) + + +def test_free_frame_stays_free(): + # a video frame already at 1.0 is unaffected by any gain + assert soften(1.0, 0.7) == pytest.approx(1.0) + + +def test_partial_soften(): + # 0.2 pin, gain 0.5 -> 0.2 + 0.8*0.5 = 0.6 + assert soften(0.2, 0.5) == pytest.approx(0.6) + + +# --- soften_denoise_mask: audio-safety on the A/V nested mask --------------- + +def test_nested_mask_softens_video_only_audio_untouched(): + # NestedTensor((video_pin=0.2, audio_pin=0.3)). Only the video half moves. + nested = _FakeNested((0.2, 0.3)) + out = soften_mask(nested, 0.5, nested_ctor=_FakeNested) + video, audio = out.unbind() + assert video == pytest.approx(0.6) # 0.2 -> 0.2 + 0.8*0.5 + assert audio == pytest.approx(0.3) # audio mask untouched (lip-sync safe) + + +def test_video_only_mask_softened_whole(): + # plain (non-nested) mask -> softened directly + assert soften_mask(0.2, 0.5) == pytest.approx(0.6) + + +def test_soften_mask_noop_when_gain_zero(): + nested = _FakeNested((0.2, 0.3)) + assert soften_mask(nested, 0.0, nested_ctor=_FakeNested) is nested + + +def test_soften_mask_noop_when_none(): + assert soften_mask(None, 0.5) is None diff --git a/tests/nodes/test_ltx_reference_bind_schedule.py b/tests/nodes/test_ltx_reference_bind_schedule.py new file mode 100644 index 0000000..d3357c6 --- /dev/null +++ b/tests/nodes/test_ltx_reference_bind_schedule.py @@ -0,0 +1,137 @@ +"""Unit tests for the LTX Reference Bind Schedule ramp + entry boosting. + +The pure `ref_gain` ramp and `_boosted_entries` helper are stdlib-only and +always tested; the model wiring (unet wrapper + object-patch on +_build_guide_self_attention_mask) is validated by rendering against a live LTX +2.3 model. Loaded by file path to avoid importing the package __init__. +""" +from __future__ import annotations + +import importlib.util +from pathlib import Path + +import pytest + + +_MOD = Path(__file__).resolve().parents[2] / "k_ltx_reference_bind_schedule.py" +_spec = importlib.util.spec_from_file_location("k_ltx_reference_bind_schedule_under_test", _MOD) +_m = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(_m) +gain = _m.ref_gain +boost = _m._boosted_entries +fab = _m.fabricate_reference_entries + +SMAX = 10.0 + + +# --- ref_gain ramp --------------------------------------------------------- + +def test_early_steps_neutral(): + # high sigma == start of denoise -> progress 0 -> neutral 1.0 + assert gain(SMAX, SMAX, 0.5, 0.9, 2.0) == 1.0 + # still before ramp_start + assert gain(SMAX * 0.6, SMAX, 0.5, 0.9, 2.0) == 1.0 # progress 0.4 < 0.5 + + +def test_late_steps_reach_peak(): + # sigma -> 0 -> progress 1 -> peak + assert gain(0.0, SMAX, 0.5, 0.9, 2.0) == 2.0 + assert gain(0.05 * SMAX, SMAX, 0.5, 0.9, 2.0) == 2.0 # progress 0.95 >= 0.9 + + +def test_midpoint_between_one_and_peak(): + # progress 0.7 is the midpoint of [0.5, 0.9] -> smoothstep 0.5 -> 1.5 + g = gain(SMAX * 0.3, SMAX, 0.5, 0.9, 2.0) + assert g == pytest.approx(1.5) + + +def test_monotonic_non_decreasing_as_sigma_falls(): + prev = 0.0 + for i in range(0, 21): + s = SMAX * (1.0 - i / 20.0) + g = gain(s, SMAX, 0.5, 0.9, 2.0) + assert g >= prev - 1e-9 + prev = g + + +def test_degenerate_ramp_window(): + # ramp_end <= ramp_start -> step at ramp_end + assert gain(0.0, SMAX, 0.9, 0.9, 3.0) == 3.0 + assert gain(SMAX, SMAX, 0.9, 0.9, 3.0) == 1.0 + + +# --- _boosted_entries ------------------------------------------------------ + +def test_boost_scales_only_trailing_refs(): + entries = [{"strength": 1.0}, {"strength": 1.0}, {"strength": 1.0}] # 2 kf + 1 ref + out = boost(entries, 1, 2.0) + assert [e["strength"] for e in out] == [1.0, 1.0, 2.0] + + +def test_boost_multiplies_existing_strength(): + entries = [{"strength": 0.8}, {"strength": 0.5}] + out = boost(entries, 1, 2.0) + assert out[0]["strength"] == 0.8 + assert out[1]["strength"] == 1.0 + + +def test_boost_noop_when_gain_one(): + entries = [{"strength": 1.0}, {"strength": 1.0}] + assert boost(entries, 1, 1.0) is entries # untouched object + + +def test_boost_noop_when_no_refs(): + entries = [{"strength": 1.0}] + assert boost(entries, 0, 2.0) is entries + + +def test_boost_does_not_mutate_input(): + entries = [{"strength": 1.0}, {"strength": 1.0}] + _ = boost(entries, 1, 3.0) + assert entries[1]["strength"] == 1.0 + + +def test_boost_count_capped_at_length(): + entries = [{"strength": 1.0}, {"strength": 1.0}] + out = boost(entries, 9, 2.0) + assert [e["strength"] for e in out] == [2.0, 2.0] + + +# --- fabricate_reference_entries (no-LoRA path) ---------------------------- + +def test_fabricate_splits_keyframes_and_refs(): + # 5 guide frames (4 kf + 1 ref), 3275 guide tokens -> ref share ~ 1/5. + out = fab(3275, 1, 4, 2.0) + assert len(out) == 2 + kf, ref = out + assert kf["strength"] == 1.0 + assert ref["strength"] == 2.0 + assert kf["surviving_count"] + ref["surviving_count"] == 3275 + assert ref["surviving_count"] == round(3275 * 1 / 5) + + +def test_fabricate_no_keyframes_is_all_reference(): + out = fab(1000, 1, 0, 3.0) + assert len(out) == 1 + assert out[0]["strength"] == 3.0 + assert out[0]["surviving_count"] == 1000 + + +def test_fabricate_entry_shape_matches_core_builder(): + # core reads strength, surviving_count, and .get(pixel_mask)/.get(latent_shape) + out = fab(800, 2, 2, 2.0) + for e in out: + assert set(("strength", "surviving_count", "pixel_mask", "latent_shape")) <= set(e) + + +def test_fabricate_empty_when_no_refs_or_tokens(): + assert fab(0, 1, 4, 2.0) == [] + assert fab(3275, 0, 4, 2.0) == [] + + +def test_fabricate_ref_tokens_clamped_at_least_one(): + # tiny ref share rounds toward >=1 + out = fab(10, 1, 100, 2.0) + ref = out[-1] + assert ref["surviving_count"] >= 1 + assert sum(e["surviving_count"] for e in out) == 10 diff --git a/web/sidebar/tree.js b/web/sidebar/tree.js index 362cc93..9e85966 100644 --- a/web/sidebar/tree.js +++ b/web/sidebar/tree.js @@ -3847,20 +3847,25 @@ export function renderPanel(container, options = {}) { if (showUpdateFooter) { const starFooter = document.createElement("div"); starFooter.className = "koolook-star-footer"; + // One inline text flow (not separate flex children) so the footer's + // `gap` doesn't space out the words; the star glyph and GitHub link sit + // inline inside the sentence. The action ("Give a ★") leads the ask. + const starText = document.createElement("span"); + starText.append("Help creators find us! Give a "); const starGlyph = document.createElement("span"); starGlyph.className = "koolook-star-glyph"; starGlyph.textContent = "★"; - starGlyph.setAttribute("aria-hidden", "true"); - const starText = document.createElement("span"); - starText.textContent = "Help creators find us on "; + starGlyph.setAttribute("role", "img"); + starGlyph.setAttribute("aria-label", "star"); + starText.append(starGlyph, " on "); const starLink = document.createElement("a"); starLink.href = GITHUB_REPO_URL; starLink.target = "_blank"; starLink.rel = "noopener noreferrer"; starLink.textContent = "GitHub"; - starLink.title = "Help creators find Kforge Labs on GitHub"; + starLink.title = "Star Kforge Labs on GitHub so more creators find it"; starText.append(starLink); - starFooter.append(starGlyph, starText); + starFooter.append(starText); container.appendChild(starFooter); } diff --git a/web/whatdreamscost_koolook/ltx_director.js b/web/whatdreamscost_koolook/ltx_director.js index 51b221d..8500981 100644 --- a/web/whatdreamscost_koolook/ltx_director.js +++ b/web/whatdreamscost_koolook/ltx_director.js @@ -5545,7 +5545,8 @@ class TimelineEditor { this.promptInput.style.opacity = "1.0"; const isImage = (this.selectionType === "image") && (seg.type === "image" || seg.type === "video"); - const strength = isImage ? (seg.guideStrength ?? 1.0) : 1.0; + // Koolook: default image-guide strength 0.8 (smooth), not 1.0 (hard pin). + const strength = isImage ? (seg.guideStrength ?? 0.8) : 1.0; this.strengthValue.value = strength.toFixed(2); this.strengthValue.disabled = !isImage; this.strengthValue.style.opacity = isImage ? "1.0" : "0.35"; @@ -8931,7 +8932,8 @@ class TimelineEditor { const strList = sortedSegments .filter(s => s.type !== "text") .filter(s => s.start + s.length > startFrames && s.start < endFrames) - .map(s => (s.guideStrength !== undefined ? s.guideStrength : 1.0).toFixed(2)); + // Koolook: unset clips default to 0.8 (smooth) instead of 1.0 (hard pin / robotic transitions). + .map(s => (s.guideStrength !== undefined ? s.guideStrength : 0.8).toFixed(2)); val = strList.join(","); } updateWidgetValue(this.guideStrengthWidget, val);