feat(ltx-director): reference-image identity — Ghost Mask + schedulable guide attention (#264)
* feat(ltx-director): Ghost Mask reference image (REF / character sheet) Add a no-LoRA reference-image identity anchor to LTXDirector__koolook, the way a WAN VACE reference works: attach reference image(s) and the model keeps the character's face / mouth consistent even when keyframes show a closed (or different) mouth than the audio-driven motion needs. Targets the mouth / identity drift seen in the talking-goat lip-sync workflow. Mechanism (Ghost Mask): the reference image(s) are appended as masked guide frames *after* the clean video region, so the transformer attends to them for identity but they never appear in the output; the trailing reference frames are then sliced off the result. - LTXDirector__koolook: new optional `reference_images` (IMAGE) + `reference_strength` inputs, and appended `clean_latent_frames` / `clean_pixel_frames` outputs. All additive/optional -> saved workflows unaffected. Refs are appended AFTER snap/ease so they are never grid-snapped or ease-expanded; the auto latent is grown to hold them. ltx_director.py is under ruff's forks/ exclude. - reference_ghost.py (new, Koolook-original, pure stdlib): trailing-reference geometry (insert frames, grown latent length, slice bounds). Unit-tested. - k_clean_latent_slice.py (new root node `CleanLatentSlice`): torch.narrow slice to drop the trailing reference frames (start=0, length=clean_latent_frames). torch imported lazily; clamp math unit-tested, slice path tested when real torch is present (skips under the CI stub). - Registered the node; added to sync_to_dev RUNTIME_PATHS. - Attribution: the Ghost Mask APPROACH is adapted (idea only, reimplemented) from CGlide/WhatDreamsCost-CSGlide (main_cs @ 21dd076, GPL-3.0, license verified 2026-06-24, GPL->GPL OK). THIRD_PARTY.md, forks_manifest.yaml, UPSTREAM_PIN.yaml, CHANGELOG updated. Phase 1: Ghost Mask only. The LoRA-backed Licon MSR prefix mode and the editor's character-slot UI are intentionally NOT ported. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(sidebar): action-led star footer ("Give a ★ on GitHub") Reword the GitHub discovery footer from the passive "Help creators find us on GitHub" to an action-led "Help creators find us! Give a ★ on GitHub", with the star as the explicit ask. Render the sentence as one inline text flow (not separate flex children) so the footer's `gap` doesn't space out the words; the ★ glyph keeps its gold styling inline and now carries role="img"/aria-label="star" so it is announced. Verified rendering in an isolated harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ltx-director): Ghost Mask broke motion in the append-and-crop pipeline The upstream guide pipeline (LTXDirectorGuide + LTXDirectorCropGuides) appends every guide_data image as an extra latent frame and crops them back off via `exact_crop_frames = latent.shape[2] - initial_latent_length`. The Ghost Mask code (modeled on CGlide's CleanLatentSlice flow) ALSO pre-grew the video latent by n_refs, so initial_latent_length became clean+n_refs: CropGuides then left n_refs frames uncropped AND the audio latent stayed at the clean length, desyncing the A/V joint transformer and breaking motion. Fix: don't pre-grow the latent. References are added to guide_data exactly like timeline keyframes and ride the same append-and-crop path — the guide node appends them and CropGuides removes them, audio untouched. Auto latent is sized to the clean length again. CleanLatentSlice/clean_latent_frames remain available for bare pipelines that don't use LTXDirectorCropGuides. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ltx): LTX Guide Reference Strength — per-stage reference boost Adds a small Koolook node that rewrites the strength of ONLY the Ghost Mask reference frames in guide_data (the trailing entries the Director now tags with `reference_count`), leaving the keyframe pins untouched. Insert it before a stage-2 refinement guide and set ~0.9 to lock identity during the low-denoise pass without disturbing the motion set in stage 1 — the deterministic form of "more reference in stage 2". Chosen over vendoring the 600-line upstream LTXDirectorGuide (which would force a node-ID rename + graph rewire): this is one small node inserted on the stage-2 guide_data wire. The sigma-ramp alternative (ramp reference up at low sigma) is deferred — the reference hook is noise_mask + a core-model attention term, so it needs a live-model spike, not a clean static patch. - Director tags guide_data["reference_count"] = n_refs. - k_ltx_guide_reference_strength.py: pure strength rewrite (absolute / multiply, clamped), unit-tested; registered; added to sync paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ltx): LTX Reference Bind Schedule — sigma-ramp the reference attention Confirmed against core (comfy/ldm/lightricks/model.py): the reference's guide attention strength is turned into a log-space additive self-attention bias in `_build_guide_self_attention_mask`, which is rebuilt EVERY denoise step and explicitly supports strength > 1.0 to amplify. That's a clean per-step hook — so the "inverse sigma curve" idea is buildable, not just a stage knob. New node mirrors LTXAVBindSchedule: a model_function_wrapper captures the per-step sigma and stashes a gain in transformer_options; an object-patch on diffusion_model._build_guide_self_attention_mask multiplies the strength of ONLY the trailing `num_references` guide entries (references are appended after keyframes) by a smoothstep ramp — neutral early (motion forms), rising to peak_strength late (identity locks during refinement). Keyframe pins and noise-mask freezing untouched; orthogonal to reference_strength. One pass, no re-noise, no LoRA. - ref_gain ramp + _boosted_entries unit-tested (11 tests); model wiring validated by live render. - Registered; added to sync paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ltx): mark Reference Bind Schedule inert without an IC-LoRA Live diagnostic on the AV model showed the builder is reached and the sigma gain flows in, but resolved_guide_entries=None despite 3276 guide tokens. Root cause: LTXDirectorGuide only records guide_attention_entries when an IC-LoRA is active (`if is_lora_active:` gates _append_guide_attention_entry). With no LoRA there is no per-guide attention bias to scale, so the attention-ramp lever does not exist in the Ghost Mask path — it can only attenuate/amplify entries that an IC-LoRA created. Rather than silently no-op, the node now logs a clear INERT warning when the entry list is empty, and the docstring/DESCRIPTION/CHANGELOG state the IC-LoRA requirement. The node remains correct for the IC-LoRA path (its partner). No-LoRA reference strengthening stays on the noise-mask lever (reference_strength / LTX Guide Reference Strength). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ltx): Reference Bind Schedule works without an IC-LoRA (fabricates entries) The attention-bias identity channel (_build_guide_self_attention_mask) is base-model functionality, but LTXDirectorGuide only populates the per-guide entries when an IC-LoRA is active, so the node was inert in the no-LoRA Ghost Mask path. Now, when entries are missing, the node FABRICATES them for the trailing reference tokens: it splits num_guide_tokens across (num_keyframes + num_references) guide frames and assigns the sigma-ramped strength to the reference share, then hands off to the core builder. This gives the reference the same amplified-attention channel an IC-LoRA would use — inside the Director, no guide-node vendor, no graph rewire (the node is already in the user's setup). When an IC-LoRA IS active it scales the real entries as before. - New `fabricate_reference_entries(num_guide_tokens, num_references, num_keyframes, strength)` (pure, unit-tested) + `num_keyframes` input. - Wrapper now does scale-or-fabricate; clearer one-shot diagnostics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ltx): keep num_keyframes last so pre-existing RefBind widgets stay aligned num_keyframes was inserted mid widget-order, which shifted ramp_start/ramp_end for workflows saved before it existed. Move it to the end of the optional inputs so saved [num_references, peak_strength, ramp_start, ramp_end] values keep mapping correctly; num_keyframes just defaults to 0 on reload. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(test): drop unused pytest import (ruff F401) The guide-reference-strength test uses plain asserts; the stray `import pytest` tripped ruff F401 and reddened the Lint & Smoke CI job. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ltx-director): default keyframe guidance 0.8 (smooth), not 1.0 The timeline editor auto-built the guide_strength string from each clip's per-segment guideStrength, filling 1.0 for any clip left unset (and the Python fallback matched). 1.0 is a hard pin, so a clip you forgot to lower snapped instead of transitioning smoothly. Unset clips now default to 0.8 in the editor (display + serialization) and the node; explicit per-clip values still override. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ltx): Keyframe Soften Schedule — smooth motion early, sharp poses late New model-patcher node LTXKeyframeSoftenSchedule: sigma-schedules the LTX keyframe denoise-mask pins so the model forms smooth motion between poses early (pins loosened at high sigma) and resolves sharp poses late (pins restored by a crossover). Uses the sampler's per-step denoise_mask_function hook — no IC-LoRA required — and adds no guide frames (so it can't fabricate extra poses). Audio-safe: only the video half of the NestedTensor A/V mask is softened; the audio mask is untouched, so lip-sync is unaffected. Two normalized 0-1 widgets map onto the sensitive useful range: max_soften (ease, ->0-0.20) and crossover (invention window, ->0-0.05). Pure soften_gain ramp unit-tested (11 tests); model wiring render-validated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(review): chain denoise hook, optional_latent clean-length, audio test, CGlide manifest Addresses PR #264 review findings: - MEDIUM (Soften node): chain a pre-existing denoise_mask_function instead of clobbering it (mirrors the sibling LTX patchers' wrapper-chaining), so the node composes with anything else that sets that hook. - HIGH #2 (Director): clean_pixel_frames/clean_latent_frames now derive from the ACTUAL optional_latent shape when one is connected, so Ghost-Mask reference geometry and the reported counts follow the real latent instead of the timeline duration (with a safe fallback + warning). - LOW (Soften node): reconcile docstring (crossover is sigma-progress, maps to internal 0-0.05), one-time warn on the sigma-read fallback, and extract a testable soften_denoise_mask() helper. - LOW: lock the audio-safety guarantee with NestedTensor-stub tests — video mask softened, audio mask untouched, part-ordering asserted. - #6: add a formal forks_manifest.yaml entry for the CSGlide Ghost Mask idea-only adaptation (was prose-only in another entry's notes; THIRD_PARTY.md already carries the full attribution). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: AI Machine <ai.machine@x.baconcph.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
malkuthro
co-authored by
Claude Opus 4.8
AI Machine
Claude Opus 4.8
AI Machine
parent
ba37ec3d41
commit
961a95b9bd
@@ -7,6 +7,18 @@ The format is inspired by Keep a Changelog and SemVer.
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- **LTX Keyframe Soften Schedule node (`LTXKeyframeSoftenSchedule`).** Smooths
|
||||
motion between keyframes without adding any guide frames: a model patcher that
|
||||
sigma-schedules the keyframe denoise-mask pins — loosening them early (high
|
||||
sigma) so the model forms smooth motion between poses, then restoring them by a
|
||||
crossover point so the poses sharpen for the back half of denoise. Uses the
|
||||
sampler's per-step `denoise_mask_function` hook; works with **no IC-LoRA**.
|
||||
Audio-safe: the A/V mask is a `NestedTensor((video, audio))`, so only the video
|
||||
keyframe pins are softened and the audio mask passes through untouched (lip-sync
|
||||
unaffected). Two normalized 0-1 widgets map onto the (very sensitive) useful
|
||||
range: `max_soften` (ease amount, 0-1 → internal 0-0.20) and `crossover`
|
||||
(invention window, 0-1 → internal 0-0.05). Pure `soften_gain` ramp unit-tested;
|
||||
model wiring render-validated.
|
||||
- **LTX Director fork bumped to upstream v2.0.2** with a new
|
||||
`forks/whatdreamscost_koolook/versions/v2_0_2/` namespace (pinned to upstream
|
||||
commit `fe09f73`). This version is a **faithful replica of upstream 2.0.2** —
|
||||
@@ -39,7 +51,66 @@ The format is inspired by Keep a Changelog and SemVer.
|
||||
reproduces stock behavior. Ramp math is unit-tested; the model wiring is
|
||||
validated by rendering against a live LTX 2.3 AV model.
|
||||
|
||||
- **LTX Director Ghost Mask reference (REF image / character sheet).** New
|
||||
optional `reference_images` (IMAGE) + `reference_strength` inputs on
|
||||
`LTXDirector__koolook` (plus informational `clean_latent_frames` /
|
||||
`clean_pixel_frames` outputs). Reference image(s) are added as guide frames
|
||||
just past the clean timeline, so the model anchors identity (face / mouth
|
||||
shape) the way a WAN VACE reference does — without the references appearing in
|
||||
the output and **without a LoRA**. Targets mouth/identity drift when keyframes
|
||||
show a closed (or different) mouth than the audio-driven motion needs. The
|
||||
references ride the **same append-and-crop path as the timeline keyframes**:
|
||||
`LTXDirectorGuide` appends each as an extra latent frame and
|
||||
`LTXDirectorCropGuides` removes it, so the video latent is never pre-grown and
|
||||
the audio track stays in sync (pre-growing it desyncs A/V and breaks motion).
|
||||
A general-purpose root node **`CleanLatentSlice`** ("Clean Latent Slice
|
||||
(Koolook)") is also included for bare pipelines that don't use
|
||||
`LTXDirectorCropGuides`. The trailing-reference *approach* is adapted (idea
|
||||
only, reimplemented) from CGlide's WhatDreamsCost-CSGlide (GPL-3.0); see
|
||||
`forks/THIRD_PARTY.md`. Geometry is unit-tested; model behavior is validated
|
||||
by rendering against a live LTX 2.3 model. (Phase 1: Ghost Mask only — the
|
||||
LoRA-backed Licon MSR prefix mode is intentionally not ported.)
|
||||
|
||||
- **LTX Guide Reference Strength node (`LTXGuideReferenceStrength`).** Companion
|
||||
to the Ghost Mask reference: rewrites the strength of *only* the reference
|
||||
frames in `guide_data` (the trailing entries the Director tags with
|
||||
`reference_count`), leaving the keyframe pins untouched. Intended for
|
||||
two-stage pipelines — feed stage 1 the Director's `guide_data` unchanged
|
||||
(light reference, motion forms freely) and route stage 2's guide_data through
|
||||
this node at a higher reference strength, so identity locks during the
|
||||
low-denoise refinement without disturbing stage-1 motion. Strength math is
|
||||
unit-tested.
|
||||
|
||||
- **LTX Reference Bind Schedule node (`LTXReferenceBindSchedule`).** Ramps the
|
||||
Ghost Mask reference's **attention** up as sigma falls — neutral early (motion
|
||||
forms freely) rising to `peak_strength` late (identity locks during
|
||||
refinement). The inverse of the motion curve and the mirror of
|
||||
`LTXAVBindSchedule`. Mechanism: a `model_function_wrapper` captures the
|
||||
per-step sigma and an object-patch on the core LTX
|
||||
`_build_guide_self_attention_mask` (which is rebuilt every denoise step and
|
||||
supports `strength > 1.0` to amplify) scales **only the trailing
|
||||
`num_references` guide entries'** attention strength — keyframe pins and the
|
||||
noise-mask freezing are untouched, so it's orthogonal to `reference_strength`.
|
||||
Single pass, no re-noise. The `ref_gain` ramp is unit-tested; model behavior is
|
||||
validated by rendering against a live LTX 2.3 model. **Works without an
|
||||
IC-LoRA:** the attention-bias machinery is base-model functionality, but
|
||||
`LTXDirectorGuide` only *populates* the per-guide entries when an IC-LoRA is
|
||||
active. So when the entries are missing this node **fabricates them itself**
|
||||
for the trailing reference tokens (set `num_keyframes` so it knows the
|
||||
keyframe/reference split of `num_guide_tokens`), giving the reference the same
|
||||
amplified-attention channel an IC-LoRA would use — inside the Director, no
|
||||
guide-node changes, no graph rewire. When an IC-LoRA *is* active it scales the
|
||||
real entries instead. The split/fabrication math is unit-tested.
|
||||
|
||||
### Fixed
|
||||
- **Keyframe guidance now defaults to 0.8 (smooth), not 1.0 (robotic).** The
|
||||
timeline editor auto-generated the `guide_strength` string from each clip's
|
||||
per-segment `guideStrength`, filling **1.0** for any clip left unset — and the
|
||||
Director's Python fallback did the same. 1.0 is a hard pin, so any clip you
|
||||
forgot to lower snapped instead of transitioning smoothly. Unset clips now
|
||||
default to **0.8** in both the editor (display + serialization) and the node;
|
||||
per-clip values you set explicitly still override (e.g. a hard 1.0 pin on a
|
||||
critical pose). Clips with no explicit strength shift from 1.0 → 0.8 on load.
|
||||
- **`dev-sync` from a git worktree.** `scripts/sync_to_dev.py` now resolves
|
||||
`.env` with the worktree→main-repo fallback (the committed `.env` lives only
|
||||
in the main checkout), matching `sync_to_dev_audio.py` and the documented
|
||||
|
||||
@@ -132,6 +132,10 @@ else:
|
||||
_merge_node_group("Koolook Loop Status", ".k_loop_status")
|
||||
_merge_node_group("Koolook Publish Contract", ".k_publish_contract")
|
||||
_merge_node_group("LTX A/V Bind Schedule", ".k_ltx_av_bind_schedule")
|
||||
_merge_node_group("Clean Latent Slice", ".k_clean_latent_slice")
|
||||
_merge_node_group("LTX Guide Reference Strength", ".k_ltx_guide_reference_strength")
|
||||
_merge_node_group("LTX Reference Bind Schedule", ".k_ltx_reference_bind_schedule")
|
||||
_merge_node_group("LTX Keyframe Soften Schedule", ".k_ltx_keyframe_soften_schedule")
|
||||
# The VHS-dependent video wrappers degrade to empty mappings when
|
||||
# VideoHelperSuite is absent (they catch the import internally); the
|
||||
# guard also covers a hard ImportError should one ever slip through.
|
||||
|
||||
+53
-12
@@ -279,24 +279,35 @@ below — GPL-3.0 §5(c) requires the whole work to be GPL-3.0.
|
||||
Koolook customizations (`relay_overrides`, `audio_transcript_json`,
|
||||
per-segment sigma) are **intentionally dropped** as unused, so every latest
|
||||
2.0.2 feature (Prompt Relay, audio, motion) behaves exactly as upstream.
|
||||
- **`ltx_director.py` (upstream verbatim + 3 additive changes).** (1) the
|
||||
node is namespaced as `LTXDirector__koolook` so it coexists with the
|
||||
- **`ltx_director.py` (upstream verbatim + Koolook additive changes).** (1)
|
||||
the node is namespaced as `LTXDirector__koolook` so it coexists with the
|
||||
installed upstream copy; (2) **issue #258:** a `snap_keyframes_to_grid`
|
||||
toggle (default on, appended last in the schema) snaps each image
|
||||
keyframe's pixel position to the center of its LTX latent-time bucket
|
||||
before the guide is built, so hard pins land on a single latent frame and
|
||||
two pins never collide in one bucket (the later is bumped, with a logged
|
||||
warning). Pixel positions in, pixel positions out — the upstream
|
||||
`LTXDirectorGuide` re-derives latent indices, so it needs no change; and
|
||||
(3) `keyframe_ease` / `ease_falloff` optionally emit strength-ramped
|
||||
neighbor pins of the same pose to smooth into and out of hard keyframes.
|
||||
Nothing else is modified.
|
||||
toggle (default on) snaps each image keyframe's pixel position to the
|
||||
center of its LTX latent-time bucket before the guide is built, so hard
|
||||
pins land on a single latent frame and two pins never collide in one
|
||||
bucket (the later is bumped, with a logged warning); (3) `keyframe_ease` /
|
||||
`ease_falloff` emit strength-ramped neighbor pins to soften the robotic
|
||||
stop/dissolve at each hard pin; (4) **Ghost Mask reference** —
|
||||
`reference_images` (IMAGE) + `reference_strength` inputs append the
|
||||
reference image(s) as masked guide frames *after* the clean video region
|
||||
so the model anchors identity (face / mouth shape) without the references
|
||||
appearing in the output, plus `clean_latent_frames` / `clean_pixel_frames`
|
||||
outputs to drive the slice-off (see CGlide attribution below). All inputs
|
||||
are optional and appended last, so saved workflows are unaffected; pixel
|
||||
positions in, pixel positions out — the upstream `LTXDirectorGuide`
|
||||
re-derives latent indices and needs no change.
|
||||
- **`prompt_relay.py` (verbatim vendored).** Unmodified upstream 2.0.2;
|
||||
imported by `ltx_director.py` for its Prompt Relay helpers.
|
||||
- **`patches.py` (verbatim vendored).** Unmodified upstream `fe09f73`.
|
||||
- **`keyframe_grid.py` (new, Koolook-original).** Pure stdlib latent-grid
|
||||
snap/ease helpers; unit-tested in
|
||||
[`tests/forks/test_whatdreamscost_v2_0_2_keyframe_grid.py`](../tests/forks/test_whatdreamscost_v2_0_2_keyframe_grid.py).
|
||||
- **`reference_ghost.py` (new, Koolook-original).** Pure stdlib geometry for
|
||||
the Ghost Mask trailing-reference layout (insert frames + grown latent
|
||||
length + slice bounds); unit-tested in
|
||||
[`tests/forks/test_whatdreamscost_v2_0_2_reference_ghost.py`](../tests/forks/test_whatdreamscost_v2_0_2_reference_ghost.py).
|
||||
The trailing-reference *approach* is adapted from CGlide (see entry below);
|
||||
the code is Koolook-original.
|
||||
- **What was *not* forked:** Same policy as v1.3.9 — `LTXDirectorGuide` and
|
||||
other WhatDreamsCost helper nodes remain runtime dependencies from the
|
||||
user's installed upstream package.
|
||||
@@ -306,7 +317,37 @@ below — GPL-3.0 §5(c) requires the whole work to be GPL-3.0.
|
||||
is not registered here; it stays backed by the v1.3.9 package for workflows
|
||||
saved against it. v1.3.9 (with the older customizations) remains on disk for
|
||||
rollback.
|
||||
- **Last reviewed:** 2026-06-22
|
||||
- **Last reviewed:** 2026-06-24
|
||||
|
||||
### CGlide/WhatDreamsCost-CSGlide — Ghost Mask reference approach (Koolook reimplementation)
|
||||
|
||||
- **Name:** WhatDreamsCost-CSGlide (a fork of WhatDreamsCost-ComfyUI)
|
||||
- **Upstream repo URL:** https://github.com/CGlide/WhatDreamsCost-CSGlide
|
||||
- **Upstream commit/tag used:** `main_cs` @ `21dd0766898f6c3fed34dc9d2e9f87e67c4d7607`
|
||||
(2026-06-23). Upstream does not tag releases.
|
||||
- **License:** GPL-3.0 (verified 2026-06-24 from the upstream `LICENSE` file;
|
||||
`pyproject.toml` declares `license = {file = "LICENSE"}`). Compatible with
|
||||
this package's GPL-3.0 — no relicense required.
|
||||
- **Scope of import:** clean-room reimplementation of one feature — *not* a
|
||||
vendor. We read CGlide's `ltx_director.py` (Ghost Mask branch) and
|
||||
`latent_slice.py` to learn the mechanism, then wrote our own code.
|
||||
- **What we took (idea, not code):** the **Ghost Mask** technique for
|
||||
reference-image identity anchoring without a LoRA — append the reference
|
||||
image(s) as guide frames *after* the clean video region (so the transformer
|
||||
attends to them for identity), generate, then slice the trailing reference
|
||||
frames off the output latent. Implemented in Koolook as: the
|
||||
`reference_images` / `reference_strength` inputs + `clean_latent_frames` /
|
||||
`clean_pixel_frames` outputs on `LTXDirector__koolook`
|
||||
([`versions/v2_0_2/ltx_director.py`](whatdreamscost_koolook/versions/v2_0_2/ltx_director.py)),
|
||||
the pure geometry helper
|
||||
[`versions/v2_0_2/reference_ghost.py`](whatdreamscost_koolook/versions/v2_0_2/reference_ghost.py),
|
||||
and the new root node `CleanLatentSlice` ("Clean Latent Slice (Koolook)") in
|
||||
[`k_clean_latent_slice.py`](../k_clean_latent_slice.py).
|
||||
- **What we did NOT take:** the Licon MSR (Prefix) mode and its companion
|
||||
LiconStudio LoRA dependency, the `@charN` prompt-tag / character-slot editor
|
||||
UI, the Ollama character-analysis endpoint, and CGlide's editor JS rewrite.
|
||||
Ghost Mask here is fed by a plain IMAGE socket, not the timeline editor.
|
||||
- **Last reviewed:** 2026-06-24
|
||||
|
||||
## De-vendored upstream code (untracked in v0.1.4 / v0.1.5)
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
version: 1
|
||||
updated_at: "2026-06-22"
|
||||
updated_at: "2026-06-25"
|
||||
external_forks:
|
||||
root_relative_to_main: "../ComfyUI-Forks"
|
||||
root_env_var: "KOLOOK_FORKS_DIR"
|
||||
@@ -130,9 +130,10 @@ entries:
|
||||
- "forks/whatdreamscost_koolook/versions/v2_0_2/prompt_relay.py"
|
||||
- "forks/whatdreamscost_koolook/versions/v2_0_2/patches.py"
|
||||
- "forks/whatdreamscost_koolook/versions/v2_0_2/keyframe_grid.py"
|
||||
- "forks/whatdreamscost_koolook/versions/v2_0_2/reference_ghost.py"
|
||||
- "web/whatdreamscost_koolook/ltx_director.js"
|
||||
status: "modified"
|
||||
sync_state: "upstream-verbatim-plus-keyframe-snap"
|
||||
sync_state: "upstream-verbatim-plus-koolook-additions"
|
||||
license: "GPL-3.0"
|
||||
license_verified_at: "2026-06-22"
|
||||
exposed_node_ids:
|
||||
@@ -161,6 +162,48 @@ entries:
|
||||
so the editor UX matches upstream 2.0.2 exactly; the old 1.3.9-era editor
|
||||
and its now-obsolete base64-scrub deltas are replaced. localStorage quota
|
||||
guarding stays in the global `web/koolook_draft_guard.js`.
|
||||
Beyond the keyframe snap, `ltx_director.py` also carries the Koolook
|
||||
`keyframe_ease`/`ease_falloff` neighbour-pin softening and the Ghost Mask
|
||||
reference feature (`reference_images`/`reference_strength` inputs +
|
||||
`clean_latent_frames`/`clean_pixel_frames` outputs); all are optional
|
||||
inputs / appended outputs (saved workflows unaffected). The Ghost Mask
|
||||
geometry lives in the new Koolook-original `reference_ghost.py`; its
|
||||
companion root node is `k_clean_latent_slice.py` ("Clean Latent Slice
|
||||
(Koolook)", node id `CleanLatentSlice`). The Ghost Mask APPROACH (append
|
||||
references as trailing guide frames, then slice them off) is adapted —
|
||||
idea only, reimplemented — from CGlide/WhatDreamsCost-CSGlide (main_cs @
|
||||
21dd076, GPL-3.0, verified 2026-06-24); see forks/THIRD_PARTY.md. The
|
||||
LoRA-backed Licon MSR prefix mode is intentionally not ported. The CGlide
|
||||
Ghost Mask adaptation has its own formal entry below
|
||||
(`csglide_ghost_mask_reimplementation`).
|
||||
|
||||
- id: "csglide_ghost_mask_reimplementation"
|
||||
name: "WhatDreamsCost-CSGlide Ghost Mask approach (Koolook reimplementation)"
|
||||
source_repo: "https://github.com/CGlide/WhatDreamsCost-CSGlide"
|
||||
source_ref: "main_cs (idea-only adaptation — no source code copied)"
|
||||
pinned_commit: "21dd076"
|
||||
local_paths:
|
||||
- "forks/whatdreamscost_koolook/versions/v2_0_2/reference_ghost.py"
|
||||
- "k_clean_latent_slice.py"
|
||||
status: "inspired_reimplementation"
|
||||
sync_state: "no-upstream-tracking-needed"
|
||||
license: "GPL-3.0"
|
||||
license_verified_at: "2026-06-24"
|
||||
exposed_node_ids:
|
||||
- "CleanLatentSlice"
|
||||
notes: |
|
||||
Idea-level adaptation of CSGlide's "Ghost Mask" reference approach — append
|
||||
reference images as trailing guide frames, let the transformer attend to
|
||||
them for identity, then slice them off the output latent so they never
|
||||
appear. NO source code was copied: the integer geometry is a fresh
|
||||
Koolook write in `forks/whatdreamscost_koolook/versions/v2_0_2/reference_ghost.py`
|
||||
(pure-stdlib, unit-tested), the in-Director plumbing lives in that fork's
|
||||
`ltx_director.py` (Ghost Mask block), and the tensor slice is the
|
||||
Koolook-original `k_clean_latent_slice.py` (node id `CleanLatentSlice`).
|
||||
CSGlide is itself a GPL-3.0 fork of WhatDreamsCost-ComfyUI (already vendored
|
||||
here under GPL-3.0), so no relicense pressure. `pinned_commit` is the short
|
||||
hash on record (`main_cs @ 21dd076`); full attribution in
|
||||
forks/THIRD_PARTY.md. The LoRA-backed Licon MSR prefix mode is not ported.
|
||||
|
||||
- id: "easy_resize_inspiration"
|
||||
name: "Easy Resize (Koolook reimplementation, inspired by KJ Nodes)"
|
||||
|
||||
@@ -14,14 +14,23 @@ notes: |
|
||||
motion) behave exactly as upstream.
|
||||
|
||||
This version folder ships:
|
||||
- ltx_director.py (upstream 2.0.2 VERBATIM + three additive changes only:
|
||||
the namespaced node ID `LTXDirector__koolook`, and the
|
||||
`snap_keyframes_to_grid` latent-bucket keyframe snap from issue #258,
|
||||
and `keyframe_ease` / `ease_falloff` strength-ramped neighbor pins).
|
||||
- ltx_director.py (upstream 2.0.2 VERBATIM + Koolook additive changes only:
|
||||
the namespaced node ID `LTXDirector__koolook`; the
|
||||
`snap_keyframes_to_grid` latent-bucket keyframe snap from issue #258;
|
||||
the `keyframe_ease`/`ease_falloff` strength-ramped neighbor pins; and the
|
||||
Ghost Mask reference feature (`reference_images`/`reference_strength`
|
||||
inputs + `clean_latent_frames`/`clean_pixel_frames` outputs). All
|
||||
additions are optional inputs / appended outputs, so saved workflows are
|
||||
unaffected.).
|
||||
- prompt_relay.py (VERBATIM upstream 2.0.2; imported by ltx_director.py).
|
||||
- patches.py (VERBATIM upstream 2.0.2; imported by ltx_director.py).
|
||||
- keyframe_grid.py (NEW, Koolook-original) pure latent-grid snap/ease helpers
|
||||
used by ltx_director.py; unit-tested in tests/forks/.
|
||||
- reference_ghost.py (NEW, Koolook-original) pure geometry for the Ghost
|
||||
Mask trailing-reference layout; unit-tested in tests/forks/. The Ghost
|
||||
Mask APPROACH is adapted from CGlide's WhatDreamsCost-CSGlide (main_cs @
|
||||
21dd076, GPL-3.0) — idea only, reimplemented; see forks/THIRD_PARTY.md.
|
||||
Companion root node: k_clean_latent_slice.py (CleanLatentSlice).
|
||||
|
||||
Node IDs: the canonical `LTXDirector__koolook` rolls to this clean 2.0.2
|
||||
implementation (package entrypoint loads v2_0_2 last). The legacy
|
||||
|
||||
@@ -45,6 +45,7 @@ from .patches import detect_model_type, apply_patches
|
||||
from .keyframe_grid import latent_count_for_duration as _latent_count_for_duration
|
||||
from .keyframe_grid import snap_keyframes_to_grid as _snap_keyframes_to_grid
|
||||
from .keyframe_grid import expand_keyframe_ease as _expand_keyframe_ease
|
||||
from .reference_ghost import ghost_insert_frames as _ghost_insert_frames
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
@@ -994,6 +995,29 @@ class LTXDirector(io.ComfyNode):
|
||||
"(less dwell), higher = gentler/longer ease."
|
||||
),
|
||||
),
|
||||
io.Image.Input(
|
||||
"reference_images", optional=True,
|
||||
tooltip=(
|
||||
"Koolook (Ghost Mask): optional reference image(s) / "
|
||||
"character sheet to anchor identity (face, mouth shape) "
|
||||
"the way a WAN VACE reference does. Each image is added as "
|
||||
"a guide frame just past the clean timeline, so the model "
|
||||
"attends to it for identity but it never appears in the "
|
||||
"output. It rides the SAME guide path as your keyframes — "
|
||||
"LTXDirectorGuide appends it and LTXDirectorCropGuides "
|
||||
"removes it, so no extra wiring and the audio stays in "
|
||||
"sync. No LoRA required. Lower reference_strength if the "
|
||||
"reference suppresses motion."
|
||||
),
|
||||
),
|
||||
io.Float.Input(
|
||||
"reference_strength", default=1.0, min=0.0, max=5.0, step=0.05, optional=True,
|
||||
tooltip=(
|
||||
"Guide strength for the reference image(s). 1.0 = full "
|
||||
"identity pull; lower it if the references bleed pose or "
|
||||
"lighting into the video."
|
||||
),
|
||||
),
|
||||
],
|
||||
outputs=[
|
||||
io.Model.Output(display_name="model"),
|
||||
@@ -1004,6 +1028,18 @@ class LTXDirector(io.ComfyNode):
|
||||
MotionGuideData.Output(display_name="motion_guide_data"),
|
||||
io.Float.Output(display_name="frame_rate", tooltip="The frame rate used for the timeline."),
|
||||
io.Audio.Output(display_name="combined_audio", tooltip="Combined timeline audio layout."),
|
||||
io.Int.Output(
|
||||
display_name="clean_latent_frames",
|
||||
tooltip=(
|
||||
"Latent frame count of the clean video region (excludes "
|
||||
"any appended reference frames). Wire to 'Clean Latent "
|
||||
"Slice (Koolook)' length with start=0 to drop the refs."
|
||||
),
|
||||
),
|
||||
io.Int.Output(
|
||||
display_name="clean_pixel_frames",
|
||||
tooltip="Pixel frame count of the clean video region.",
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
@@ -1014,7 +1050,8 @@ class LTXDirector(io.ComfyNode):
|
||||
custom_width=768, custom_height=512, resize_method="maintain aspect ratio",
|
||||
divisible_by=32, img_compression=0, audio_vae=None, optional_latent=None,
|
||||
use_custom_audio=False, inpaint_audio=True, use_custom_motion=True, override_audio=False,
|
||||
snap_keyframes_to_grid=True, keyframe_ease=0, ease_falloff=0.5) -> io.NodeOutput:
|
||||
snap_keyframes_to_grid=True, keyframe_ease=0, ease_falloff=0.5,
|
||||
reference_images=None, reference_strength=1.0) -> io.NodeOutput:
|
||||
|
||||
# Parse timeline data
|
||||
try:
|
||||
@@ -1101,7 +1138,9 @@ class LTXDirector(io.ComfyNode):
|
||||
insert_frame = max(0, seg_start + int(seg.get("length", 1)) - 1 - start_frame)
|
||||
else:
|
||||
insert_frame = max(0, seg_start - start_frame)
|
||||
strength = strengths[idx] if idx < len(strengths) else 1.0
|
||||
# Koolook: unset guidance defaults to 0.8 (smooth transitions),
|
||||
# not 1.0 (hard pin -> robotic snapping between keyframes).
|
||||
strength = strengths[idx] if idx < len(strengths) else 0.8
|
||||
guide_data["images"].append(tensor)
|
||||
guide_data["insert_frames"].append(insert_frame)
|
||||
guide_data["strengths"].append(float(strength))
|
||||
@@ -1227,16 +1266,82 @@ class LTXDirector(io.ComfyNode):
|
||||
except Exception as _e:
|
||||
log.warning("[LTXDirector] keyframe ease skipped: %s", _e)
|
||||
|
||||
# --- Auto-generate LTXV latent if none was provided ---
|
||||
# Apply the community 8n+1 rule directly to the timeline's duration_frames:
|
||||
# int(ceil(((duration_frames) - 1) / 8) * 8) + 1
|
||||
# This ensures we get AT LEAST the requested frames, snapped to LTXV's requirements.
|
||||
# --- Latent geometry ---
|
||||
# Community 8n+1 rule on the timeline duration: ceil((duration-1)/8)*8 + 1
|
||||
# yields AT LEAST the requested pixel frames, snapped to LTXV requirements.
|
||||
ltxv_length = int(math.ceil((duration_frames - 1) / 8.0) * 8) + 1
|
||||
|
||||
# When the caller overrides the latent, the clean length (and therefore the
|
||||
# Ghost-Mask reference geometry below AND the clean_*_frames outputs) must
|
||||
# follow the ACTUAL latent, not the timeline duration — otherwise references
|
||||
# are placed at the wrong frames and the reported counts are stale.
|
||||
if optional_latent is not None:
|
||||
try:
|
||||
_lat_t = int(optional_latent["samples"].shape[2])
|
||||
ltxv_length = (_lat_t - 1) * 8 + 1
|
||||
except Exception as _e:
|
||||
log.warning(
|
||||
"[LTXDirector] optional_latent shape unreadable (%s); "
|
||||
"clean length falls back to the timeline duration.", _e,
|
||||
)
|
||||
clean_pixel_frames = int(ltxv_length)
|
||||
clean_latent_frames = ((ltxv_length - 1) // 8) + 1
|
||||
|
||||
# --- Koolook (Ghost Mask): add reference image(s) as guide frames so the
|
||||
# model anchors identity (face / mouth shape) without the references
|
||||
# appearing in the output. They are added to guide_data exactly like the
|
||||
# timeline keyframes and therefore ride the SAME append-and-crop path:
|
||||
# the downstream LTXDirectorGuide appends each as an extra latent frame
|
||||
# and LTXDirectorCropGuides removes them again (and the audio track is
|
||||
# untouched, so A/V stays in sync). We must NOT pre-grow the video latent
|
||||
# here — doing that leaves the references uncropped AND desyncs the audio
|
||||
# track, which breaks motion. Positioned just past the clean timeline so
|
||||
# they anchor identity without pinning a visible pose. Added AFTER
|
||||
# snap/ease so refs are never grid-snapped or ease-expanded. The trick is
|
||||
# adapted from CGlide's WhatDreamsCost-CSGlide (GPL-3.0). ---
|
||||
n_refs = 0
|
||||
_ref_shape = getattr(reference_images, "shape", None)
|
||||
if reference_images is not None and _ref_shape is not None and int(_ref_shape[0]) > 0:
|
||||
try:
|
||||
n_refs = int(_ref_shape[0])
|
||||
# Match references to the established output dimensions; fall back
|
||||
# to the references' own size when nothing else set them.
|
||||
ref_w = derived_w if derived_w > 0 else int(_ref_shape[2])
|
||||
ref_h = derived_h if derived_h > 0 else int(_ref_shape[1])
|
||||
_ref_inserts = _ghost_insert_frames(clean_latent_frames, n_refs)
|
||||
for i in range(n_refs):
|
||||
ref = reference_images[i:i + 1]
|
||||
ref = _resize_image(ref, ref_w, ref_h, resize_method, divisible_by)
|
||||
if img_compression > 0:
|
||||
ref = _compress_image(ref, img_compression)
|
||||
guide_data["images"].append(ref)
|
||||
guide_data["insert_frames"].append(int(_ref_inserts[i]))
|
||||
guide_data["strengths"].append(float(reference_strength))
|
||||
if derived_w <= 0:
|
||||
derived_w = ref_w
|
||||
if derived_h <= 0:
|
||||
derived_h = ref_h
|
||||
log.info(
|
||||
"[LTXDirector] Ghost Mask: added %d reference guide frame(s) "
|
||||
"past frame %d (strength=%.2f); they ride the guide "
|
||||
"append-and-crop path (LTXDirectorCropGuides removes them).",
|
||||
n_refs, clean_pixel_frames, float(reference_strength),
|
||||
)
|
||||
except Exception as _e:
|
||||
log.warning("[LTXDirector] reference Ghost Mask skipped: %s", _e)
|
||||
n_refs = 0
|
||||
|
||||
# Tag how many of the trailing guide frames are references so a
|
||||
# downstream node ("LTX Guide Reference Strength") can boost ONLY the
|
||||
# references (e.g. harder pin in a stage-2 refinement) without touching
|
||||
# the keyframe pins. References are always the LAST n_refs entries.
|
||||
guide_data["reference_count"] = int(n_refs)
|
||||
|
||||
# --- Auto-generate LTXV latent if none was provided ---
|
||||
# Sized to the CLEAN length only — references are appended (and cropped)
|
||||
# by the downstream guide pipeline, never by growing this latent.
|
||||
if optional_latent is None:
|
||||
latent_w = max(32, (derived_w // 32) * 32)
|
||||
latent_h = max(32, (derived_h // 32) * 32)
|
||||
# LTXV temporal: ((length - 1) // 8) + 1 latent frames; invert to get pixel frames -> length
|
||||
latent_t = ((ltxv_length - 1) // 8) + 1
|
||||
samples = torch.zeros(
|
||||
[1, 128, latent_t, latent_h // 32, latent_w // 32],
|
||||
@@ -1421,7 +1526,7 @@ class LTXDirector(io.ComfyNode):
|
||||
guide_data["duration_frames"] = duration_frames
|
||||
guide_data["resize_method"] = resize_method
|
||||
|
||||
return io.NodeOutput(patched, conditioning, latent, audio_latent, guide_data, motion_guide_data, float(frame_rate), audio_out)
|
||||
return io.NodeOutput(patched, conditioning, latent, audio_latent, guide_data, motion_guide_data, float(frame_rate), audio_out, int(clean_latent_frames), int(clean_pixel_frames))
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
# SPDX-License-Identifier: GPL-3.0-or-later
|
||||
"""
|
||||
Koolook-original pure helpers for the LTX Director "Ghost Mask" reference
|
||||
feature (REF image / character sheet identity anchor).
|
||||
|
||||
The trailing-reference trick — append the reference image(s) as guide frames
|
||||
*after* the clean video region, let the transformer attend to them for
|
||||
identity, then slice them off the output latent — is adapted from CGlide's
|
||||
WhatDreamsCost-CSGlide (branch main_cs @ 21dd076, GPL-3.0). Only the integer
|
||||
geometry lives here; the tensor work stays in the Director / the Clean Latent
|
||||
Slice node. Pure stdlib so it unit-tests without torch or ComfyUI.
|
||||
|
||||
Geometry recap (LTXV stride-8 temporal latent):
|
||||
latent index of pixel frame f>0 : L(f) = 1 + (f - 1) // 8
|
||||
so pixel f = (clean_latent_frames + i) * 8 -> L(f) = clean_latent_frames + i
|
||||
which is exactly the i-th trailing latent slot. We append one reference per
|
||||
trailing slot, then the video latent must be grown to make room for them.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
# LTXV temporal downscale stride. The Director's clean-latent math hardcodes
|
||||
# the same 8, so the Ghost Mask geometry must agree with it.
|
||||
LTXV_STRIDE = 8
|
||||
|
||||
|
||||
def ghost_insert_frames(clean_latent_frames: int, n_refs: int, stride: int = LTXV_STRIDE) -> list[int]:
|
||||
"""Pixel-space insert positions for ``n_refs`` Ghost Mask reference frames.
|
||||
|
||||
Each reference pins the trailing latent slot ``clean_latent_frames + i``.
|
||||
Returns ``[(clean_latent_frames + i) * stride for i in range(n_refs)]``.
|
||||
"""
|
||||
clf = int(clean_latent_frames)
|
||||
n = int(n_refs)
|
||||
s = int(stride)
|
||||
if clf < 0:
|
||||
raise ValueError(f"clean_latent_frames must be >= 0, got {clf}")
|
||||
if n < 0:
|
||||
raise ValueError(f"n_refs must be >= 0, got {n}")
|
||||
if s < 1:
|
||||
raise ValueError(f"stride must be >= 1, got {s}")
|
||||
return [(clf + i) * s for i in range(n)]
|
||||
|
||||
|
||||
def ghost_total_latent_frames(clean_latent_frames: int, n_refs: int) -> int:
|
||||
"""Total latent frame count once the reference slots are appended.
|
||||
|
||||
The auto-generated video latent must be this long so the sampler produces
|
||||
the trailing reference slots the guide node pins; they are removed
|
||||
afterwards with the Clean Latent Slice node.
|
||||
"""
|
||||
clf = int(clean_latent_frames)
|
||||
n = int(n_refs)
|
||||
if clf < 0 or n < 0:
|
||||
raise ValueError(f"counts must be >= 0, got clf={clf}, n_refs={n}")
|
||||
return clf + n
|
||||
|
||||
|
||||
def clean_slice_bounds(size: int, start: int, length: int) -> tuple[int, int]:
|
||||
"""Clamp a (start, length) request to a tensor axis of ``size`` frames.
|
||||
|
||||
Mirrors the safety logic of the Clean Latent Slice node so it can be
|
||||
unit-tested without torch. Returns ``(actual_start, actual_length)`` such
|
||||
that ``0 <= actual_start < size`` (or 0 when empty) and the window never
|
||||
runs past the end. For Ghost Mask the caller passes start=0,
|
||||
length=clean_latent_frames to drop the trailing reference frames.
|
||||
"""
|
||||
sz = int(size)
|
||||
if sz <= 0:
|
||||
return 0, 0
|
||||
st = max(0, int(start))
|
||||
st = min(st, sz - 1)
|
||||
ln = max(0, int(length))
|
||||
ln = min(ln, sz - st)
|
||||
return st, ln
|
||||
@@ -0,0 +1,104 @@
|
||||
# SPDX-License-Identifier: GPL-3.0-or-later
|
||||
#
|
||||
# ComfyUI-Koolook — Clean Latent Slice node.
|
||||
# Copyright (C) 2026 ComfyUI-Koolook contributors (kforgelabs).
|
||||
"""Clean Latent Slice (Koolook) — trim frames off a video latent.
|
||||
|
||||
Companion to the LTX Director "Ghost Mask" reference feature. That mode appends
|
||||
reference image(s) as guide frames *after* the clean video region so the model
|
||||
anchors identity (face / mouth shape) without the references appearing in the
|
||||
output. After sampling, the trailing reference frames must be removed: wire the
|
||||
Director's ``clean_latent_frames`` output to ``length`` (with ``start=0``) and
|
||||
this node slices the latent back down to just the video.
|
||||
|
||||
The trick — and the use of ``torch.narrow`` to avoid PyTorch NestedTensor
|
||||
slicing issues — is adapted from CGlide's CleanLatentSliceCS
|
||||
(WhatDreamsCost-CSGlide, branch main_cs @ 21dd076, GPL-3.0).
|
||||
|
||||
The clamp math mirrors ``reference_ghost.clean_slice_bounds`` (unit-tested in
|
||||
tests/forks); torch is imported lazily so this module loads without it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _clamp_slice(size: int, start: int, length: int) -> tuple[int, int]:
|
||||
"""Clamp (start, length) to a frame axis of ``size``. Mirrors
|
||||
``forks.whatdreamscost_koolook.versions.v2_0_2.reference_ghost.clean_slice_bounds``.
|
||||
"""
|
||||
sz = int(size)
|
||||
if sz <= 0:
|
||||
return 0, 0
|
||||
st = min(max(0, int(start)), sz - 1)
|
||||
ln = min(max(0, int(length)), sz - st)
|
||||
return st, ln
|
||||
|
||||
|
||||
class CleanLatentSlice:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"latent": ("LATENT",),
|
||||
"start": ("INT", {
|
||||
"default": 0, "min": 0, "max": 100000, "step": 1,
|
||||
"tooltip": "First latent frame to keep. 0 for Ghost Mask (refs are at the end).",
|
||||
}),
|
||||
"length": ("INT", {
|
||||
"default": 1, "min": 1, "max": 100000, "step": 1,
|
||||
"tooltip": "Number of latent frames to keep. Wire the Director's 'clean_latent_frames'.",
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("LATENT",)
|
||||
RETURN_NAMES = ("latent",)
|
||||
FUNCTION = "slice_latent"
|
||||
CATEGORY = "Koolook/LTX"
|
||||
DESCRIPTION = (
|
||||
"Slice a video latent to [start, start+length) along the temporal axis. "
|
||||
"Use it to drop the trailing reference frames added by LTX Director's "
|
||||
"Ghost Mask reference feature (start=0, length=clean_latent_frames)."
|
||||
)
|
||||
|
||||
def slice_latent(self, latent, start, length):
|
||||
import torch # lazy: keep module importable without torch
|
||||
|
||||
out = latent.copy()
|
||||
|
||||
def _narrow(tensor):
|
||||
dims = tensor.ndim if hasattr(tensor, "ndim") else len(tensor.shape)
|
||||
# 5D [B, C, F, H, W] -> temporal dim 2; 4D/3D [F, ...] -> dim 0.
|
||||
axis = 2 if dims == 5 else 0
|
||||
size = tensor.size(axis)
|
||||
st, ln = _clamp_slice(size, start, length)
|
||||
if ln <= 0:
|
||||
return tensor
|
||||
try:
|
||||
return torch.narrow(tensor, axis, st, ln)
|
||||
except Exception as e:
|
||||
log.warning("[CleanLatentSlice] torch.narrow failed (%s); using python slice", e)
|
||||
idx = [slice(None)] * dims
|
||||
idx[axis] = slice(st, st + ln)
|
||||
return tensor[tuple(idx)]
|
||||
|
||||
if "samples" in out:
|
||||
out["samples"] = _narrow(out["samples"])
|
||||
# Only slice a real per-frame mask, not a broadcasted (size-1) one.
|
||||
if "noise_mask" in out:
|
||||
nm = out["noise_mask"]
|
||||
nm_dims = nm.ndim if hasattr(nm, "ndim") else len(nm.shape)
|
||||
nm_axis = 2 if nm_dims == 5 else 0
|
||||
if nm.size(nm_axis) > 1:
|
||||
out["noise_mask"] = _narrow(nm)
|
||||
|
||||
return (out,)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"CleanLatentSlice": CleanLatentSlice}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"CleanLatentSlice": "Clean Latent Slice (Koolook)"}
|
||||
|
||||
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
|
||||
@@ -0,0 +1,90 @@
|
||||
# SPDX-License-Identifier: GPL-3.0-or-later
|
||||
#
|
||||
# ComfyUI-Koolook — LTX Guide Reference Strength node.
|
||||
# Copyright (C) 2026 ComfyUI-Koolook contributors (kforgelabs).
|
||||
"""LTX Guide Reference Strength (Koolook) — boost ONLY the reference frames.
|
||||
|
||||
Companion to LTX Director's Ghost Mask reference feature. The Director appends
|
||||
the reference image(s) as the LAST entries in ``guide_data`` (tagged with
|
||||
``reference_count``) so they ride the same append-and-crop path as the timeline
|
||||
keyframes. Every guide frame's strength becomes a noise-mask pin
|
||||
(``noise_mask = 1 - strength``) in the downstream LTXDirectorGuide.
|
||||
|
||||
This node rewrites the strength of just those trailing reference entries,
|
||||
leaving the keyframe pins untouched. The intended use is a two-stage pipeline:
|
||||
feed stage 1 the Director's ``guide_data`` unchanged (light reference, so motion
|
||||
forms freely), and route stage 2's guide_data through this node at a higher
|
||||
reference strength — so the identity locks during the low-denoise refinement
|
||||
without disturbing the motion already set in stage 1.
|
||||
|
||||
Pure-python; no torch. The strength math is unit-tested.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def scaled_reference_strengths(strengths, reference_count, value, mode="absolute"):
|
||||
"""Return a new strengths list with the last ``reference_count`` entries
|
||||
rewritten. ``mode='absolute'`` sets them to ``value``; ``mode='multiply'``
|
||||
multiplies the existing strength by ``value``. Other entries (keyframes) are
|
||||
left untouched. Clamps to [0, 1] (guide strength -> noise-mask range)."""
|
||||
out = [float(s) for s in strengths]
|
||||
n = len(out)
|
||||
k = max(0, min(int(reference_count), n))
|
||||
for i in range(n - k, n):
|
||||
v = float(value) if mode == "absolute" else out[i] * float(value)
|
||||
out[i] = max(0.0, min(1.0, v))
|
||||
return out
|
||||
|
||||
|
||||
class LTXGuideReferenceStrength:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"guide_data": ("GUIDE_DATA",),
|
||||
"reference_strength": ("FLOAT", {
|
||||
"default": 0.9, "min": 0.0, "max": 1.0, "step": 0.01,
|
||||
"tooltip": "New strength for the reference frames only. Higher = harder identity pin. Use ~0.9 on a stage-2 refinement guide.",
|
||||
}),
|
||||
"mode": (["absolute", "multiply"], {
|
||||
"default": "absolute",
|
||||
"tooltip": "absolute: set reference strength to this value. multiply: scale the Director's reference strength by it.",
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("GUIDE_DATA",)
|
||||
RETURN_NAMES = ("guide_data",)
|
||||
FUNCTION = "apply"
|
||||
CATEGORY = "Koolook/LTX"
|
||||
DESCRIPTION = (
|
||||
"Rewrite the strength of just the Ghost Mask reference frames in "
|
||||
"guide_data (the trailing entries tagged by LTX Director), leaving the "
|
||||
"keyframe pins untouched. Insert before a stage-2 refinement guide to "
|
||||
"lock identity harder without disturbing stage-1 motion."
|
||||
)
|
||||
|
||||
def apply(self, guide_data, reference_strength, mode):
|
||||
ref_count = int((guide_data or {}).get("reference_count", 0) or 0)
|
||||
if not guide_data or ref_count <= 0:
|
||||
log.info("[LTXGuideReferenceStrength] no reference frames in guide_data; passthrough.")
|
||||
return (guide_data,)
|
||||
out = dict(guide_data)
|
||||
out["strengths"] = scaled_reference_strengths(
|
||||
guide_data.get("strengths", []), ref_count, reference_strength, mode,
|
||||
)
|
||||
log.info(
|
||||
"[LTXGuideReferenceStrength] set %d reference frame(s) -> %.2f (%s).",
|
||||
ref_count, float(reference_strength), mode,
|
||||
)
|
||||
return (out,)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"LTXGuideReferenceStrength": LTXGuideReferenceStrength}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"LTXGuideReferenceStrength": "LTX Guide Reference Strength (Koolook)"}
|
||||
|
||||
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
|
||||
@@ -0,0 +1,208 @@
|
||||
# SPDX-License-Identifier: GPL-3.0-or-later
|
||||
#
|
||||
# ComfyUI-Koolook — LTX Keyframe Soften Schedule node.
|
||||
# Copyright (C) 2026 ComfyUI-Koolook contributors (kforgelabs).
|
||||
"""LTX Keyframe Soften Schedule (Koolook) — smooth motion early, sharp poses late.
|
||||
|
||||
The LTX Director pins each keyframe by setting that latent frame's denoise mask
|
||||
to ``1 - strength`` (a value < 1). The sampler applies it EVERY step
|
||||
(``comfy/samplers.py``)::
|
||||
|
||||
if "denoise_mask_function" in model_options:
|
||||
denoise_mask = model_options["denoise_mask_function"](sigma, denoise_mask, ...)
|
||||
x = x * denoise_mask + guide * (1 - denoise_mask) # the pin
|
||||
|
||||
Hard pins held from step 0 force the model to hit exact poses immediately, so a
|
||||
transition between two different poses resolves as a CUT. This node schedules the
|
||||
pin by sigma instead: it **softens** the keyframe pins early (high sigma), when
|
||||
the model is deciding coarse motion — letting it form a smooth trajectory between
|
||||
the poses instead of snapping — then **restores** them by a configurable
|
||||
``crossover`` measured in denoise *progress* (``1 - sigma/sigma_max``) so the poses
|
||||
sharpen as denoise finishes. Because LTX's sigma schedule is front-loaded (sigma
|
||||
held high, dropping late), a SMALL progress fraction already covers the early
|
||||
motion-forming steps — hence the ``crossover`` widget maps its 0-1 range onto a
|
||||
small internal ``0-0.05`` progress band (see ``_CROSSOVER_SCALE``).
|
||||
|
||||
It adds NOTHING (no extra guide frames / poses — that approach backfires; every
|
||||
guide is a pose target). It only loosens/tightens the keyframes you already have.
|
||||
|
||||
Audio-safe: the A/V latent mask is a ``NestedTensor((video_mask, audio_mask))``
|
||||
(``LTXVConcatAVLatent``); we soften ONLY the video mask and pass the audio mask
|
||||
through untouched, so lip-sync timing is never disturbed.
|
||||
|
||||
NOTE: the pure ``soften_gain`` ramp is unit-tested; the model wiring (the
|
||||
denoise-mask function) is validated by rendering against a live LTX 2.3 model.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# The node widgets are normalized 0-1 sliders that map onto the *useful* internal
|
||||
# range (the effect is very sensitive, so the raw values bunch up near 0). Widget
|
||||
# value * SCALE = the internal value passed to soften_gain().
|
||||
_MAX_SOFTEN_SCALE = 0.2 # widget 0..1 -> internal max_soften 0..0.2
|
||||
_CROSSOVER_SCALE = 0.05 # widget 0..1 -> internal crossover 0..0.05
|
||||
|
||||
|
||||
def soften_gain(sigma, sigma_max, crossover=0.55, max_soften=0.6):
|
||||
"""How much to soften the keyframe pins at the current ``sigma``.
|
||||
|
||||
Denoise progress runs 0 at the start (``sigma == sigma_max``) to 1 at the end
|
||||
(``sigma -> 0``). Returns ``max_soften`` at progress 0, smoothsteps down to 0
|
||||
by ``crossover`` progress, and stays 0 afterwards (pins fully restored).
|
||||
|
||||
A returned gain ``g`` raises a pinned mask value ``m`` toward 1 (unpinned)
|
||||
via ``m + (1 - m) * g``: g=0 keeps the exact pin, g=1 fully frees it.
|
||||
"""
|
||||
sigma_max = float(sigma_max)
|
||||
if sigma_max <= 0.0:
|
||||
return 0.0
|
||||
progress = 1.0 - (float(sigma) / sigma_max)
|
||||
progress = 0.0 if progress < 0.0 else (1.0 if progress > 1.0 else progress)
|
||||
crossover = float(crossover)
|
||||
if crossover <= 0.0 or progress >= crossover:
|
||||
return 0.0
|
||||
t = progress / crossover # 0 at start -> 1 at crossover
|
||||
s = t * t * (3.0 - 2.0 * t) # smoothstep 0 -> 1
|
||||
return float(max_soften) * (1.0 - s) # max_soften at start -> 0 at crossover
|
||||
|
||||
|
||||
def soften_mask_value(mask_value, gain):
|
||||
"""Raise a single denoise-mask value toward 1 (unpinned) by ``gain``."""
|
||||
return mask_value + (1.0 - mask_value) * float(gain)
|
||||
|
||||
|
||||
def soften_denoise_mask(denoise_mask, gain, nested_ctor=None):
|
||||
"""Soften a denoise mask, sparing the audio half of an A/V mask.
|
||||
|
||||
The A/V latent mask is a ``NestedTensor((video_mask, audio_mask))`` produced by
|
||||
``LTXVConcatAVLatent`` — video FIRST, audio SECOND. We soften ONLY part 0
|
||||
(video keyframe pins) and pass part 1 (audio) through untouched, so lip-sync
|
||||
timing is never disturbed. A plain (video-only) mask is softened whole.
|
||||
|
||||
``nested_ctor`` rebuilds the nested mask; defaults to
|
||||
``comfy.nested_tensor.NestedTensor`` and is injectable so the audio-safety
|
||||
behavior can be unit-tested with a stub. ``gain<=0`` / ``None`` is a no-op.
|
||||
"""
|
||||
if gain <= 0.0 or denoise_mask is None:
|
||||
return denoise_mask
|
||||
if getattr(denoise_mask, "is_nested", False):
|
||||
parts = list(denoise_mask.unbind())
|
||||
parts[0] = soften_mask_value(parts[0], gain) # video only; audio untouched
|
||||
if nested_ctor is None:
|
||||
import comfy.nested_tensor
|
||||
nested_ctor = comfy.nested_tensor.NestedTensor
|
||||
return nested_ctor(tuple(parts))
|
||||
return soften_mask_value(denoise_mask, gain)
|
||||
|
||||
|
||||
class LTXKeyframeSoftenSchedule:
|
||||
"""Sigma-schedule the LTX keyframe pins: soft early (smooth motion), sharp late."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"model": ("MODEL", {"tooltip": "LTX 2.3 model (place after the other model patchers)."}),
|
||||
"max_soften": ("FLOAT", {
|
||||
"default": 0.5, "min": 0.0, "max": 1.0, "step": 0.01,
|
||||
"tooltip": "EASE amount (normalized 0-1; maps to internal 0-0.20). "
|
||||
"How gently the keyframe pins loosen at the START of "
|
||||
"denoise — the smooth approach into each key. 0 = stock "
|
||||
"(no ease). Higher = smoother, but also more invented "
|
||||
"in-between motion. Default 0.5 = internal 0.10.",
|
||||
}),
|
||||
"crossover": ("FLOAT", {
|
||||
"default": 0.65, "min": 0.0, "max": 1.0, "step": 0.01,
|
||||
"tooltip": "INVENTION window (normalized 0-1; maps to internal "
|
||||
"0-0.05). How long the pins stay loose before re-locking. "
|
||||
"LOWER = poses lock back sooner = FEWER invented in-between "
|
||||
"frames (truer to your keyframes). Higher = more invented "
|
||||
"motion (and risk of a noisy/under-resolved image). "
|
||||
"Default 0.65 = internal ~0.033.",
|
||||
}),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("MODEL",)
|
||||
RETURN_NAMES = ("model",)
|
||||
FUNCTION = "patch"
|
||||
CATEGORY = "Koolook/LTX"
|
||||
DESCRIPTION = (
|
||||
"Smooth motion early, sharp poses late: sigma-schedule the LTX keyframe "
|
||||
"pins — loosen them at high sigma so the model forms smooth motion between "
|
||||
"poses, restore them by a crossover (~step 9) so the poses sharpen. Adds no "
|
||||
"frames; audio mask untouched (lip-sync safe)."
|
||||
)
|
||||
|
||||
def patch(self, model, max_soften, crossover):
|
||||
m = model.clone()
|
||||
|
||||
# Widgets are normalized 0-1; map onto the sensitive internal range.
|
||||
max_soften_w, crossover_w = float(max_soften), float(crossover)
|
||||
max_soften = max_soften_w * _MAX_SOFTEN_SCALE
|
||||
crossover = crossover_w * _CROSSOVER_SCALE
|
||||
|
||||
try:
|
||||
sigma_max = float(m.get_model_object("model_sampling").sigma_max)
|
||||
except Exception as e: # pragma: no cover - live model
|
||||
log.warning("[LTXKeyframeSoftenSchedule] no model_sampling.sigma_max: %s", e)
|
||||
sigma_max = 1.0
|
||||
|
||||
if max_soften <= 0.0:
|
||||
log.info("[LTXKeyframeSoftenSchedule] max_soften=0 — model passed through.")
|
||||
return (m,)
|
||||
|
||||
# Chain any pre-existing denoise_mask_function instead of clobbering it
|
||||
# (mirrors how the sibling LTX patchers chain model_function_wrapper).
|
||||
prev_fn = m.model_options.get("denoise_mask_function")
|
||||
diag = [False]
|
||||
warned = [False]
|
||||
|
||||
def denoise_mask_function(sigma, denoise_mask, extra_options=None):
|
||||
if prev_fn is not None:
|
||||
denoise_mask = prev_fn(sigma, denoise_mask, extra_options=extra_options)
|
||||
if denoise_mask is None:
|
||||
return denoise_mask
|
||||
try:
|
||||
s = float(sigma.max()) if hasattr(sigma, "max") else float(sigma)
|
||||
except Exception as _e:
|
||||
s = sigma_max
|
||||
if not warned[0]:
|
||||
warned[0] = True
|
||||
log.warning(
|
||||
"[LTXKeyframeSoftenSchedule] could not read sigma (%s); "
|
||||
"softening at max this step.", _e,
|
||||
)
|
||||
gain = soften_gain(s, sigma_max, crossover, max_soften)
|
||||
if gain <= 0.0:
|
||||
return denoise_mask
|
||||
if not diag[0]:
|
||||
diag[0] = True
|
||||
nested = getattr(denoise_mask, "is_nested", False)
|
||||
log.info(
|
||||
"[LTXKeyframeSoftenSchedule] active (%s): softening video keyframe "
|
||||
"pins%s. max_soften=%.3f crossover=%.3f",
|
||||
"A/V" if nested else "video-only",
|
||||
", audio untouched" if nested else "", max_soften, crossover,
|
||||
)
|
||||
return soften_denoise_mask(denoise_mask, gain)
|
||||
|
||||
m.set_model_denoise_mask_function(denoise_mask_function)
|
||||
log.info(
|
||||
"[LTXKeyframeSoftenSchedule] patched: max_soften=%.2f (->%.3f) "
|
||||
"crossover=%.2f (->%.3f) sigma_max=%.3f",
|
||||
max_soften_w, max_soften, crossover_w, crossover, sigma_max,
|
||||
)
|
||||
return (m,)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"LTXKeyframeSoftenSchedule": LTXKeyframeSoftenSchedule}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"LTXKeyframeSoftenSchedule": "LTX Keyframe Soften Schedule (Koolook)"}
|
||||
|
||||
__all__ = [
|
||||
"NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS",
|
||||
"soften_gain", "soften_mask_value", "soften_denoise_mask",
|
||||
]
|
||||
@@ -0,0 +1,286 @@
|
||||
# SPDX-License-Identifier: GPL-3.0-or-later
|
||||
#
|
||||
# ComfyUI-Koolook — LTX Reference Bind Schedule node.
|
||||
# Copyright (C) 2026 ComfyUI-Koolook contributors (kforgelabs).
|
||||
"""LTX Reference Bind Schedule (Koolook) — ramp reference identity up at low sigma.
|
||||
|
||||
Companion to the LTX Director Ghost Mask reference. The reference image is added
|
||||
as a guide frame; the core LTX transformer attends to it with a per-guide
|
||||
**attention strength** (`guide_attention_entries[].strength`) which it turns into
|
||||
a log-space additive self-attention bias, rebuilt **every denoise step** in
|
||||
`comfy/ldm/lightricks/model.py::_build_guide_self_attention_mask`. That builder
|
||||
explicitly supports `strength > 1.0` to *amplify* a guide's attention.
|
||||
|
||||
This node schedules that amplification over the denoise so identity locks late
|
||||
without disturbing motion that forms early:
|
||||
|
||||
* **Early steps (high sigma)** — gain ~1.0: the reference attention is neutral,
|
||||
big motion resolves freely.
|
||||
* **Late steps (low sigma)** — gain ramps up to `peak_strength`: the refinement
|
||||
conforms harder to the reference's identity (face / mouth).
|
||||
|
||||
It's the inverse of the motion curve, and the mirror of `LTXAVBindSchedule`
|
||||
(which ramps audio->video up at low sigma for lip-sync). It touches ONLY the
|
||||
attention bias of the trailing `num_references` guide entries (references are
|
||||
appended after the keyframes) — the keyframe pins and the noise-mask freezing
|
||||
are untouched.
|
||||
|
||||
WORKS WITHOUT AN IC-LoRA. The attention-bias machinery is BASE-model
|
||||
functionality — `_build_guide_self_attention_mask` builds the mask from
|
||||
`resolved_guide_entries` regardless of any LoRA. The catch is that
|
||||
LTXDirectorGuide only *populates* those entries `if is_lora_active`, so in the
|
||||
no-LoRA Ghost Mask path the list is empty. This node closes that gap: when the
|
||||
entries are missing it **fabricates them itself** for the trailing reference
|
||||
tokens (splitting `num_guide_tokens` across the `num_keyframes + num_references`
|
||||
guide frames and assigning the ramped strength to the reference share). So the
|
||||
reference gets the same amplified-attention channel an IC-LoRA would use, inside
|
||||
the Director, with no LoRA, no guide-node vendor, and no graph change. When an
|
||||
IC-LoRA *is* active and real entries exist, it scales those instead.
|
||||
|
||||
Mechanics (same as the A/V node): a `model_function_wrapper` captures the
|
||||
current sigma each step and stashes the gain in `transformer_options`; an
|
||||
object-patch on `_build_guide_self_attention_mask` reads it and fabricates (or
|
||||
scales) the reference entries' `strength` before the mask is built.
|
||||
|
||||
NOTE: the `ref_gain` ramp is unit-tested; the model wiring is validated by
|
||||
rendering against a live LTX 2.3 model (not in CI).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import types
|
||||
import logging
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
_REF_GAIN_KEY = "nghtdrp_ref_gain" # per-step reference-attention multiplier
|
||||
|
||||
|
||||
def ref_gain(sigma, sigma_max, ramp_start=0.5, ramp_end=0.9, peak=2.0):
|
||||
"""Reference-attention multiplier for the current ``sigma``.
|
||||
|
||||
Progress runs 0 at the start of denoise (``sigma == sigma_max``) to 1 at the
|
||||
end (``sigma -> 0``). Holds at 1.0 (neutral) until ``ramp_start``, then
|
||||
smoothsteps up to ``peak`` by ``ramp_end`` (progress fractions in [0, 1]).
|
||||
Returning 1.0 means "no change" — the mask builder then skips entirely.
|
||||
"""
|
||||
if sigma_max <= 0.0:
|
||||
return float(peak)
|
||||
progress = 1.0 - (float(sigma) / float(sigma_max))
|
||||
progress = 0.0 if progress < 0.0 else (1.0 if progress > 1.0 else progress)
|
||||
if ramp_end <= ramp_start:
|
||||
return float(peak) if progress >= ramp_end else 1.0
|
||||
if progress <= ramp_start:
|
||||
return 1.0
|
||||
if progress >= ramp_end:
|
||||
return float(peak)
|
||||
t = (progress - ramp_start) / (ramp_end - ramp_start)
|
||||
s = t * t * (3.0 - 2.0 * t) # smoothstep
|
||||
return 1.0 + (float(peak) - 1.0) * s
|
||||
|
||||
|
||||
def _boosted_entries(resolved_entries, ref_count, gain):
|
||||
"""Return a copy of ``resolved_entries`` with the LAST ``ref_count`` entries'
|
||||
``strength`` multiplied by ``gain`` (references are appended after keyframes).
|
||||
Returns the original list unchanged when there's nothing to do."""
|
||||
if not resolved_entries or gain == 1.0 or ref_count <= 0:
|
||||
return resolved_entries
|
||||
n = len(resolved_entries)
|
||||
k = min(int(ref_count), n)
|
||||
if k <= 0:
|
||||
return resolved_entries
|
||||
out = list(resolved_entries)
|
||||
for i in range(n - k, n):
|
||||
e = dict(out[i])
|
||||
e["strength"] = float(e.get("strength", 1.0)) * float(gain)
|
||||
out[i] = e
|
||||
return out
|
||||
|
||||
|
||||
def fabricate_reference_entries(num_guide_tokens, num_references, num_keyframes, strength):
|
||||
"""Build ``resolved_guide_entries`` for the trailing reference tokens when the
|
||||
guide node created none (the no-IC-LoRA path).
|
||||
|
||||
Splits ``num_guide_tokens`` evenly across the ``num_keyframes + num_references``
|
||||
guide frames and gives the trailing reference share ``strength`` while the
|
||||
keyframe share stays 1.0 (neutral). The entries match the shape the core
|
||||
builder consumes (``strength`` + ``surviving_count`` + optional pixel_mask /
|
||||
latent_shape). When ``num_keyframes`` is 0 the whole guide region is treated
|
||||
as reference. Returns ``[]`` when there's nothing to amplify.
|
||||
"""
|
||||
n_guide = int(num_guide_tokens)
|
||||
n_ref = int(num_references)
|
||||
n_kf = max(0, int(num_keyframes))
|
||||
if n_guide <= 0 or n_ref <= 0:
|
||||
return []
|
||||
total_frames = n_ref + n_kf
|
||||
ref_tokens = round(n_guide * n_ref / total_frames) if total_frames > 0 else n_guide
|
||||
ref_tokens = max(1, min(int(ref_tokens), n_guide))
|
||||
kf_tokens = n_guide - ref_tokens
|
||||
entries = []
|
||||
if kf_tokens > 0:
|
||||
entries.append({"strength": 1.0, "surviving_count": kf_tokens, "pixel_mask": None, "latent_shape": None})
|
||||
entries.append({"strength": float(strength), "surviving_count": ref_tokens, "pixel_mask": None, "latent_shape": None})
|
||||
return entries
|
||||
|
||||
|
||||
def _make_guide_mask_wrapper(underlying, ref_count, kf_count, peak, diag):
|
||||
"""Bound-method wrapper around ``_build_guide_self_attention_mask``.
|
||||
|
||||
If the guide node already populated entries (IC-LoRA path) scale the trailing
|
||||
``ref_count`` of them by the per-step gain. If not (no-LoRA Ghost Mask path)
|
||||
fabricate entries for the trailing reference tokens at the gain strength. Then
|
||||
hand off to the original builder, which turns it into the additive bias.
|
||||
"""
|
||||
|
||||
def wrapped(_self_module, x, transformer_options, merged_args, *args, **kwargs):
|
||||
to = transformer_options or {}
|
||||
gain = float(to.get(_REF_GAIN_KEY, 1.0))
|
||||
ma = merged_args if isinstance(merged_args, dict) else {}
|
||||
entries = ma.get("resolved_guide_entries")
|
||||
n_guide = int(ma.get("num_guide_tokens") or 0)
|
||||
|
||||
mode = "passthrough"
|
||||
if entries: # IC-LoRA path — scale real entries
|
||||
if gain != 1.0 and ref_count > 0:
|
||||
boosted = _boosted_entries(entries, ref_count, gain)
|
||||
if boosted is not entries:
|
||||
merged_args = {**ma, "resolved_guide_entries": boosted}
|
||||
mode = "scaled"
|
||||
elif n_guide > 0 and ref_count > 0 and gain != 1.0: # no-LoRA path — fabricate
|
||||
fab = fabricate_reference_entries(n_guide, ref_count, kf_count, gain)
|
||||
if fab:
|
||||
merged_args = {**ma, "resolved_guide_entries": fab}
|
||||
mode = "fabricated"
|
||||
|
||||
if not diag.get("first"):
|
||||
diag["first"] = True
|
||||
log.info(
|
||||
"[LTXReferenceBindSchedule] active: num_guide_tokens=%s entries=%s "
|
||||
"ref_count=%d num_keyframes=%d peak=%.2f (amplifies the trailing "
|
||||
"reference tokens at low sigma).",
|
||||
n_guide, (len(entries) if entries else 0), int(ref_count),
|
||||
int(kf_count), float(peak),
|
||||
)
|
||||
if mode in ("scaled", "fabricated") and not diag.get(mode):
|
||||
diag[mode] = True
|
||||
log.info("[LTXReferenceBindSchedule] %s reference attention (gain x%.2f).", mode, gain)
|
||||
return underlying(x, transformer_options, merged_args, *args, **kwargs)
|
||||
|
||||
return wrapped
|
||||
|
||||
|
||||
class LTXReferenceBindSchedule:
|
||||
"""Ramp the LTX Ghost Mask reference's attention up as sigma falls."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"model": ("MODEL", {"tooltip": "LTX 2.3 model (the same one feeding the guide pipeline)."}),
|
||||
"num_references": ("INT", {
|
||||
"default": 1, "min": 0, "max": 64, "step": 1,
|
||||
"tooltip": "How many trailing guide frames are references (= how many images you fed LTX Director's reference_images). They're appended after the keyframes, so the LAST N guide frames are the references.",
|
||||
}),
|
||||
"peak_strength": ("FLOAT", {
|
||||
"default": 2.0, "min": 1.0, "max": 8.0, "step": 0.05,
|
||||
"tooltip": "Reference attention multiplier at the end of denoise (low sigma). 1.0 = no boost; 2.0 = double the identity pull during refinement. Try 2-4.",
|
||||
}),
|
||||
},
|
||||
"optional": {
|
||||
# NOTE: ramp_start/ramp_end stay first to preserve widget order
|
||||
# for workflows saved before num_keyframes existed; num_keyframes
|
||||
# is appended last so old [num_references, peak_strength, ramp_start,
|
||||
# ramp_end] values keep aligning.
|
||||
"ramp_start": ("FLOAT", {
|
||||
"default": 0.5, "min": 0.0, "max": 1.0, "step": 0.01,
|
||||
"tooltip": "Denoise progress (0=start, 1=end) where the reference boost begins. Keep above where big motion finalizes (~0.5-0.6) so motion forms first.",
|
||||
}),
|
||||
"ramp_end": ("FLOAT", {
|
||||
"default": 0.9, "min": 0.0, "max": 1.0, "step": 0.01,
|
||||
"tooltip": "Denoise progress where the boost reaches peak_strength.",
|
||||
}),
|
||||
"num_keyframes": ("INT", {
|
||||
"default": 0, "min": 0, "max": 256, "step": 1,
|
||||
"tooltip": "No-LoRA path only: how many timeline image keyframes precede the references, so the node knows the keyframe/reference token split. Set this = your number of timeline image segments (e.g. 4). Leave 0 and the WHOLE guide region is treated as reference (also boosts keyframes). Ignored when an IC-LoRA populated real entries.",
|
||||
}),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("MODEL",)
|
||||
RETURN_NAMES = ("model",)
|
||||
FUNCTION = "patch"
|
||||
CATEGORY = "Koolook/LTX"
|
||||
DESCRIPTION = (
|
||||
"Ramp the reference's guide-attention UP at low sigma so identity locks "
|
||||
"during refinement without disturbing early motion. Works WITHOUT an "
|
||||
"IC-LoRA: it fabricates the per-guide attention entries for the trailing "
|
||||
"reference tokens itself (set num_keyframes so it knows the split), so the "
|
||||
"reference gets the same amplified-attention channel an IC-LoRA would use "
|
||||
"— inside the Director, no guide-node changes. Mirror of LTX A/V Bind "
|
||||
"Schedule. One pass, no re-noise."
|
||||
)
|
||||
|
||||
def patch(self, model, num_references, peak_strength, num_keyframes=0, ramp_start=0.5, ramp_end=0.9):
|
||||
m = model.clone()
|
||||
|
||||
if int(num_references) <= 0:
|
||||
log.info("[LTXReferenceBindSchedule] num_references=0 — model passed through.")
|
||||
return (m,)
|
||||
|
||||
try:
|
||||
sigma_max = float(m.get_model_object("model_sampling").sigma_max)
|
||||
except Exception as e: # pragma: no cover - live model
|
||||
log.warning("[LTXReferenceBindSchedule] no model_sampling.sigma_max: %s", e)
|
||||
sigma_max = 1.0
|
||||
|
||||
key = "diffusion_model._build_guide_self_attention_mask"
|
||||
try:
|
||||
underlying = m.get_model_object(key)
|
||||
dm = m.get_model_object("diffusion_model")
|
||||
except Exception as e: # pragma: no cover - live model
|
||||
log.warning(
|
||||
"[LTXReferenceBindSchedule] no %s on this model (not an LTX guide "
|
||||
"model?): %s — passed through unchanged.", key, e,
|
||||
)
|
||||
return (m,)
|
||||
|
||||
diag = {}
|
||||
wrapper = _make_guide_mask_wrapper(
|
||||
underlying, int(num_references), int(num_keyframes), float(peak_strength), diag,
|
||||
)
|
||||
try:
|
||||
m.add_object_patch(key, types.MethodType(wrapper, dm))
|
||||
except Exception as e: # pragma: no cover - live model
|
||||
log.warning("[LTXReferenceBindSchedule] guide-mask patch failed: %s", e)
|
||||
return (m,)
|
||||
|
||||
prev = m.model_options.get("model_function_wrapper")
|
||||
|
||||
def unet_wrapper(model_function, params):
|
||||
try:
|
||||
sigma = float(params["timestep"].max())
|
||||
except Exception:
|
||||
sigma = sigma_max
|
||||
gain = ref_gain(sigma, sigma_max, ramp_start, ramp_end, peak_strength)
|
||||
c = params["c"]
|
||||
to = dict(c.get("transformer_options", {}))
|
||||
to[_REF_GAIN_KEY] = gain
|
||||
c = {**c, "transformer_options": to}
|
||||
params = {**params, "c": c}
|
||||
if prev is not None:
|
||||
return prev(model_function, params)
|
||||
return model_function(params["input"], params["timestep"], **params["c"])
|
||||
|
||||
m.set_model_unet_function_wrapper(unet_wrapper)
|
||||
log.info(
|
||||
"[LTXReferenceBindSchedule] scheduling %d reference(s): peak x%.2f, "
|
||||
"ramp %.2f..%.2f of denoise.", int(num_references), float(peak_strength),
|
||||
float(ramp_start), float(ramp_end),
|
||||
)
|
||||
return (m,)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"LTXReferenceBindSchedule": LTXReferenceBindSchedule}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"LTXReferenceBindSchedule": "LTX Reference Bind Schedule (Koolook)"}
|
||||
|
||||
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
|
||||
@@ -155,6 +155,7 @@ RUNTIME_PATHS: tuple[str, ...] = (
|
||||
"__init__.py",
|
||||
"config.json",
|
||||
"k_ai_pipeline.py",
|
||||
"k_clean_latent_slice.py",
|
||||
"k_easy_image_batch.py",
|
||||
"k_easy_pattern.py",
|
||||
"k_easy_resize.py",
|
||||
@@ -163,6 +164,9 @@ RUNTIME_PATHS: tuple[str, ...] = (
|
||||
"k_easy_wan22_prompt.py",
|
||||
"k_loop_status.py",
|
||||
"k_ltx_av_bind_schedule.py",
|
||||
"k_ltx_guide_reference_strength.py",
|
||||
"k_ltx_keyframe_soften_schedule.py",
|
||||
"k_ltx_reference_bind_schedule.py",
|
||||
"k_publish_contract.py",
|
||||
"k_video_combine.py",
|
||||
"k_video_load.py",
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
"""Tests for the v2_0_2 Ghost Mask reference geometry helpers.
|
||||
|
||||
Lives in
|
||||
``forks/whatdreamscost_koolook/versions/v2_0_2/reference_ghost.py`` and is
|
||||
pure stdlib, so we load it by file path — the same isolation trick the other
|
||||
fork tests use to avoid triggering the package ``__init__`` (which imports
|
||||
``ltx_director`` -> ``comfy_api``, unavailable inside pytest).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
_MOD_PATH = (
|
||||
Path(__file__).resolve().parents[2]
|
||||
/ "forks"
|
||||
/ "whatdreamscost_koolook"
|
||||
/ "versions"
|
||||
/ "v2_0_2"
|
||||
/ "reference_ghost.py"
|
||||
)
|
||||
_spec = importlib.util.spec_from_file_location(
|
||||
"whatdreamscost_v2_0_2_reference_ghost", _MOD_PATH
|
||||
)
|
||||
_mod = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(_mod)
|
||||
|
||||
ghost_insert_frames = _mod.ghost_insert_frames
|
||||
ghost_total_latent_frames = _mod.ghost_total_latent_frames
|
||||
clean_slice_bounds = _mod.clean_slice_bounds
|
||||
|
||||
|
||||
# --- ghost_insert_frames ---------------------------------------------------
|
||||
|
||||
def test_insert_frames_pin_trailing_latent_slots():
|
||||
# clean region is 10 latent frames; two refs -> latent slots 10 and 11.
|
||||
# pixel = slot * 8, and L(pixel) = 1 + (pixel-1)//8 must equal the slot.
|
||||
frames = ghost_insert_frames(10, 2)
|
||||
assert frames == [80, 88]
|
||||
for i, f in enumerate(frames):
|
||||
latent_idx = 1 + (f - 1) // 8
|
||||
assert latent_idx == 10 + i
|
||||
|
||||
|
||||
def test_insert_frames_empty_when_no_refs():
|
||||
assert ghost_insert_frames(10, 0) == []
|
||||
|
||||
|
||||
def test_insert_frames_custom_stride():
|
||||
assert ghost_insert_frames(3, 2, stride=4) == [12, 16]
|
||||
|
||||
|
||||
def test_insert_frames_rejects_negative():
|
||||
with pytest.raises(ValueError):
|
||||
ghost_insert_frames(-1, 1)
|
||||
with pytest.raises(ValueError):
|
||||
ghost_insert_frames(1, -1)
|
||||
with pytest.raises(ValueError):
|
||||
ghost_insert_frames(1, 1, stride=0)
|
||||
|
||||
|
||||
# --- ghost_total_latent_frames ---------------------------------------------
|
||||
|
||||
def test_total_latent_frames_adds_refs():
|
||||
assert ghost_total_latent_frames(10, 0) == 10
|
||||
assert ghost_total_latent_frames(10, 3) == 13
|
||||
|
||||
|
||||
def test_total_latent_frames_rejects_negative():
|
||||
with pytest.raises(ValueError):
|
||||
ghost_total_latent_frames(-1, 0)
|
||||
with pytest.raises(ValueError):
|
||||
ghost_total_latent_frames(0, -1)
|
||||
|
||||
|
||||
# --- clean_slice_bounds ----------------------------------------------------
|
||||
|
||||
def test_slice_bounds_drops_trailing_refs():
|
||||
# total 13 frames (10 clean + 3 refs); keep [0, 10).
|
||||
assert clean_slice_bounds(13, 0, 10) == (0, 10)
|
||||
|
||||
|
||||
def test_slice_bounds_clamps_overlong_length():
|
||||
assert clean_slice_bounds(10, 0, 999) == (0, 10)
|
||||
|
||||
|
||||
def test_slice_bounds_clamps_start_into_range():
|
||||
assert clean_slice_bounds(10, 50, 5) == (9, 1)
|
||||
|
||||
|
||||
def test_slice_bounds_empty_tensor():
|
||||
assert clean_slice_bounds(0, 0, 5) == (0, 0)
|
||||
|
||||
|
||||
def test_slice_bounds_negative_inputs_floored():
|
||||
assert clean_slice_bounds(10, -3, -2) == (0, 0)
|
||||
@@ -0,0 +1,79 @@
|
||||
"""Unit tests for the Clean Latent Slice node.
|
||||
|
||||
The clamp math (`_clamp_slice`) is stdlib and always tested. The torch.narrow
|
||||
slice is exercised only when torch is importable (skipped in the stdlib CI
|
||||
env, validated locally / on a torch machine). Loaded by file path to avoid
|
||||
importing the package __init__ (which pulls in ComfyUI).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
_MOD = Path(__file__).resolve().parents[2] / "k_clean_latent_slice.py"
|
||||
_spec = importlib.util.spec_from_file_location("k_clean_latent_slice_under_test", _MOD)
|
||||
_m = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(_m)
|
||||
clamp = _m._clamp_slice
|
||||
|
||||
|
||||
def _real_torch():
|
||||
"""Skip unless a real torch is installed.
|
||||
|
||||
Other node tests inject a bare ``types.ModuleType("torch")`` into
|
||||
``sys.modules`` that persists for the session, so ``importorskip`` alone
|
||||
would find that hollow stub. Real torch carries ``__version__``; the stub
|
||||
does not.
|
||||
"""
|
||||
torch = pytest.importorskip("torch")
|
||||
if not getattr(torch, "__version__", None):
|
||||
pytest.skip("real torch not installed (hollow stub present)")
|
||||
return torch
|
||||
|
||||
|
||||
# --- _clamp_slice ----------------------------------------------------------
|
||||
|
||||
def test_clamp_keeps_clean_region():
|
||||
# 13 frames (10 clean + 3 refs), drop the refs.
|
||||
assert clamp(13, 0, 10) == (0, 10)
|
||||
|
||||
|
||||
def test_clamp_overlong_length():
|
||||
assert clamp(10, 0, 999) == (0, 10)
|
||||
|
||||
|
||||
def test_clamp_start_past_end():
|
||||
assert clamp(10, 50, 5) == (9, 1)
|
||||
|
||||
|
||||
def test_clamp_empty_axis():
|
||||
assert clamp(0, 0, 5) == (0, 0)
|
||||
|
||||
|
||||
def test_clamp_negatives_floored():
|
||||
assert clamp(10, -3, -2) == (0, 0)
|
||||
|
||||
|
||||
# --- slice_latent (torch) --------------------------------------------------
|
||||
|
||||
def test_slice_latent_drops_trailing_frames():
|
||||
torch = _real_torch()
|
||||
node = _m.CleanLatentSlice()
|
||||
# [B, C, F, H, W] with F = 5 (3 clean + 2 refs)
|
||||
samples = torch.arange(5).float().reshape(1, 1, 5, 1, 1)
|
||||
(out,) = node.slice_latent({"samples": samples}, 0, 3)
|
||||
assert out["samples"].shape[2] == 3
|
||||
assert out["samples"].flatten().tolist() == [0.0, 1.0, 2.0]
|
||||
|
||||
|
||||
def test_slice_latent_skips_broadcast_noise_mask():
|
||||
torch = _real_torch()
|
||||
node = _m.CleanLatentSlice()
|
||||
samples = torch.zeros(1, 1, 5, 1, 1)
|
||||
bmask = torch.ones(1, 1, 1, 1, 1) # broadcasted, size-1 temporal axis
|
||||
(out,) = node.slice_latent({"samples": samples, "noise_mask": bmask}, 0, 3)
|
||||
assert out["samples"].shape[2] == 3
|
||||
assert out["noise_mask"].shape[2] == 1 # untouched
|
||||
@@ -0,0 +1,64 @@
|
||||
"""Unit tests for the LTX Guide Reference Strength helper.
|
||||
|
||||
Pure stdlib. Loaded by file path to avoid importing the package __init__
|
||||
(which pulls in ComfyUI).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
_MOD = Path(__file__).resolve().parents[2] / "k_ltx_guide_reference_strength.py"
|
||||
_spec = importlib.util.spec_from_file_location("k_ltx_guide_reference_strength_under_test", _MOD)
|
||||
_m = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(_m)
|
||||
scale = _m.scaled_reference_strengths
|
||||
|
||||
|
||||
def test_absolute_rewrites_only_trailing_refs():
|
||||
# 4 keyframes (0.8) + 2 refs (0.5); boost refs to 0.9, keyframes untouched.
|
||||
s = [0.8, 0.8, 0.8, 0.8, 0.5, 0.5]
|
||||
assert scale(s, 2, 0.9, "absolute") == [0.8, 0.8, 0.8, 0.8, 0.9, 0.9]
|
||||
|
||||
|
||||
def test_multiply_scales_trailing_refs():
|
||||
s = [0.8, 0.8, 0.4, 0.4]
|
||||
assert scale(s, 2, 2.0, "multiply") == [0.8, 0.8, 0.8, 0.8]
|
||||
|
||||
|
||||
def test_multiply_clamps_to_one():
|
||||
s = [0.8, 0.5]
|
||||
assert scale(s, 1, 5.0, "multiply") == [0.8, 1.0]
|
||||
|
||||
|
||||
def test_zero_reference_count_is_noop():
|
||||
s = [0.8, 0.8, 0.8]
|
||||
assert scale(s, 0, 0.9, "absolute") == [0.8, 0.8, 0.8]
|
||||
|
||||
|
||||
def test_reference_count_capped_at_length():
|
||||
s = [0.5, 0.5]
|
||||
assert scale(s, 9, 0.9, "absolute") == [0.9, 0.9]
|
||||
|
||||
|
||||
def test_does_not_mutate_input():
|
||||
s = [0.8, 0.5]
|
||||
_ = scale(s, 1, 0.9, "absolute")
|
||||
assert s == [0.8, 0.5]
|
||||
|
||||
|
||||
def test_node_passthrough_when_no_refs():
|
||||
node = _m.LTXGuideReferenceStrength()
|
||||
gd = {"strengths": [0.8, 0.8], "reference_count": 0}
|
||||
(out,) = node.apply(gd, 0.9, "absolute")
|
||||
assert out["strengths"] == [0.8, 0.8]
|
||||
|
||||
|
||||
def test_node_boosts_refs_and_copies():
|
||||
node = _m.LTXGuideReferenceStrength()
|
||||
gd = {"strengths": [0.8, 0.5], "reference_count": 1, "images": ["x", "y"]}
|
||||
(out,) = node.apply(gd, 0.9, "absolute")
|
||||
assert out["strengths"] == [0.8, 0.9]
|
||||
assert gd["strengths"] == [0.8, 0.5] # original untouched
|
||||
assert out["images"] == ["x", "y"] # other keys preserved
|
||||
@@ -0,0 +1,121 @@
|
||||
"""Unit tests for the LTX Keyframe Soften Schedule ramp.
|
||||
|
||||
The pure ``soften_gain`` / ``soften_mask_value`` helpers are stdlib-only and
|
||||
always tested; the model wiring (the denoise_mask_function set on the patcher,
|
||||
which softens the video keyframe pins early and restores them by the crossover)
|
||||
is validated by rendering against a live LTX 2.3 model. Loaded by file path to
|
||||
avoid importing the package __init__.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
_MOD = Path(__file__).resolve().parents[2] / "k_ltx_keyframe_soften_schedule.py"
|
||||
_spec = importlib.util.spec_from_file_location("k_ltx_keyframe_soften_schedule_under_test", _MOD)
|
||||
_m = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(_m)
|
||||
gain = _m.soften_gain
|
||||
soften = _m.soften_mask_value
|
||||
soften_mask = _m.soften_denoise_mask
|
||||
|
||||
SMAX = 10.0
|
||||
|
||||
|
||||
class _FakeNested:
|
||||
"""Minimal NestedTensor stub: holds (video, audio) parts, exposes is_nested + unbind."""
|
||||
is_nested = True
|
||||
|
||||
def __init__(self, parts):
|
||||
self.parts = tuple(parts)
|
||||
|
||||
def unbind(self):
|
||||
return self.parts
|
||||
|
||||
|
||||
# --- soften_gain ramp (max early, 0 by crossover) --------------------------
|
||||
|
||||
def test_max_soften_at_denoise_start():
|
||||
# sigma == sigma_max -> progress 0 -> full max_soften
|
||||
assert gain(SMAX, SMAX, 0.5, 0.6) == pytest.approx(0.6)
|
||||
|
||||
|
||||
def test_zero_after_crossover():
|
||||
# sigma -> 0 (progress 1) is past the crossover -> fully restored (0)
|
||||
assert gain(0.0, SMAX, 0.5, 0.6) == 0.0
|
||||
# exactly at the crossover -> 0
|
||||
assert gain(SMAX * 0.5, SMAX, 0.5, 0.6) == 0.0
|
||||
|
||||
|
||||
def test_midpoint_of_ramp():
|
||||
# progress 0.25 is the midpoint of [0, 0.5] -> smoothstep 0.5 -> half max
|
||||
assert gain(SMAX * 0.75, SMAX, 0.5, 0.6) == pytest.approx(0.3)
|
||||
|
||||
|
||||
def test_monotonic_non_increasing_as_sigma_falls():
|
||||
prev = 1.0
|
||||
for i in range(0, 21):
|
||||
s = SMAX * (1.0 - i / 20.0)
|
||||
g = gain(s, SMAX, 0.55, 0.6)
|
||||
assert g <= prev + 1e-9
|
||||
prev = g
|
||||
|
||||
|
||||
def test_max_soften_zero_is_noop():
|
||||
assert gain(SMAX, SMAX, 0.55, 0.0) == 0.0
|
||||
|
||||
|
||||
def test_crossover_zero_is_noop():
|
||||
assert gain(SMAX, SMAX, 0.0, 0.6) == 0.0
|
||||
|
||||
|
||||
def test_degenerate_sigma_max():
|
||||
assert gain(1.0, 0.0, 0.5, 0.6) == 0.0
|
||||
|
||||
|
||||
# --- soften_mask_value (raise a pin toward 1) ------------------------------
|
||||
|
||||
def test_gain_zero_keeps_exact_pin():
|
||||
assert soften(0.2, 0.0) == pytest.approx(0.2)
|
||||
|
||||
|
||||
def test_gain_one_fully_frees():
|
||||
assert soften(0.2, 1.0) == pytest.approx(1.0)
|
||||
|
||||
|
||||
def test_free_frame_stays_free():
|
||||
# a video frame already at 1.0 is unaffected by any gain
|
||||
assert soften(1.0, 0.7) == pytest.approx(1.0)
|
||||
|
||||
|
||||
def test_partial_soften():
|
||||
# 0.2 pin, gain 0.5 -> 0.2 + 0.8*0.5 = 0.6
|
||||
assert soften(0.2, 0.5) == pytest.approx(0.6)
|
||||
|
||||
|
||||
# --- soften_denoise_mask: audio-safety on the A/V nested mask ---------------
|
||||
|
||||
def test_nested_mask_softens_video_only_audio_untouched():
|
||||
# NestedTensor((video_pin=0.2, audio_pin=0.3)). Only the video half moves.
|
||||
nested = _FakeNested((0.2, 0.3))
|
||||
out = soften_mask(nested, 0.5, nested_ctor=_FakeNested)
|
||||
video, audio = out.unbind()
|
||||
assert video == pytest.approx(0.6) # 0.2 -> 0.2 + 0.8*0.5
|
||||
assert audio == pytest.approx(0.3) # audio mask untouched (lip-sync safe)
|
||||
|
||||
|
||||
def test_video_only_mask_softened_whole():
|
||||
# plain (non-nested) mask -> softened directly
|
||||
assert soften_mask(0.2, 0.5) == pytest.approx(0.6)
|
||||
|
||||
|
||||
def test_soften_mask_noop_when_gain_zero():
|
||||
nested = _FakeNested((0.2, 0.3))
|
||||
assert soften_mask(nested, 0.0, nested_ctor=_FakeNested) is nested
|
||||
|
||||
|
||||
def test_soften_mask_noop_when_none():
|
||||
assert soften_mask(None, 0.5) is None
|
||||
@@ -0,0 +1,137 @@
|
||||
"""Unit tests for the LTX Reference Bind Schedule ramp + entry boosting.
|
||||
|
||||
The pure `ref_gain` ramp and `_boosted_entries` helper are stdlib-only and
|
||||
always tested; the model wiring (unet wrapper + object-patch on
|
||||
_build_guide_self_attention_mask) is validated by rendering against a live LTX
|
||||
2.3 model. Loaded by file path to avoid importing the package __init__.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
_MOD = Path(__file__).resolve().parents[2] / "k_ltx_reference_bind_schedule.py"
|
||||
_spec = importlib.util.spec_from_file_location("k_ltx_reference_bind_schedule_under_test", _MOD)
|
||||
_m = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(_m)
|
||||
gain = _m.ref_gain
|
||||
boost = _m._boosted_entries
|
||||
fab = _m.fabricate_reference_entries
|
||||
|
||||
SMAX = 10.0
|
||||
|
||||
|
||||
# --- ref_gain ramp ---------------------------------------------------------
|
||||
|
||||
def test_early_steps_neutral():
|
||||
# high sigma == start of denoise -> progress 0 -> neutral 1.0
|
||||
assert gain(SMAX, SMAX, 0.5, 0.9, 2.0) == 1.0
|
||||
# still before ramp_start
|
||||
assert gain(SMAX * 0.6, SMAX, 0.5, 0.9, 2.0) == 1.0 # progress 0.4 < 0.5
|
||||
|
||||
|
||||
def test_late_steps_reach_peak():
|
||||
# sigma -> 0 -> progress 1 -> peak
|
||||
assert gain(0.0, SMAX, 0.5, 0.9, 2.0) == 2.0
|
||||
assert gain(0.05 * SMAX, SMAX, 0.5, 0.9, 2.0) == 2.0 # progress 0.95 >= 0.9
|
||||
|
||||
|
||||
def test_midpoint_between_one_and_peak():
|
||||
# progress 0.7 is the midpoint of [0.5, 0.9] -> smoothstep 0.5 -> 1.5
|
||||
g = gain(SMAX * 0.3, SMAX, 0.5, 0.9, 2.0)
|
||||
assert g == pytest.approx(1.5)
|
||||
|
||||
|
||||
def test_monotonic_non_decreasing_as_sigma_falls():
|
||||
prev = 0.0
|
||||
for i in range(0, 21):
|
||||
s = SMAX * (1.0 - i / 20.0)
|
||||
g = gain(s, SMAX, 0.5, 0.9, 2.0)
|
||||
assert g >= prev - 1e-9
|
||||
prev = g
|
||||
|
||||
|
||||
def test_degenerate_ramp_window():
|
||||
# ramp_end <= ramp_start -> step at ramp_end
|
||||
assert gain(0.0, SMAX, 0.9, 0.9, 3.0) == 3.0
|
||||
assert gain(SMAX, SMAX, 0.9, 0.9, 3.0) == 1.0
|
||||
|
||||
|
||||
# --- _boosted_entries ------------------------------------------------------
|
||||
|
||||
def test_boost_scales_only_trailing_refs():
|
||||
entries = [{"strength": 1.0}, {"strength": 1.0}, {"strength": 1.0}] # 2 kf + 1 ref
|
||||
out = boost(entries, 1, 2.0)
|
||||
assert [e["strength"] for e in out] == [1.0, 1.0, 2.0]
|
||||
|
||||
|
||||
def test_boost_multiplies_existing_strength():
|
||||
entries = [{"strength": 0.8}, {"strength": 0.5}]
|
||||
out = boost(entries, 1, 2.0)
|
||||
assert out[0]["strength"] == 0.8
|
||||
assert out[1]["strength"] == 1.0
|
||||
|
||||
|
||||
def test_boost_noop_when_gain_one():
|
||||
entries = [{"strength": 1.0}, {"strength": 1.0}]
|
||||
assert boost(entries, 1, 1.0) is entries # untouched object
|
||||
|
||||
|
||||
def test_boost_noop_when_no_refs():
|
||||
entries = [{"strength": 1.0}]
|
||||
assert boost(entries, 0, 2.0) is entries
|
||||
|
||||
|
||||
def test_boost_does_not_mutate_input():
|
||||
entries = [{"strength": 1.0}, {"strength": 1.0}]
|
||||
_ = boost(entries, 1, 3.0)
|
||||
assert entries[1]["strength"] == 1.0
|
||||
|
||||
|
||||
def test_boost_count_capped_at_length():
|
||||
entries = [{"strength": 1.0}, {"strength": 1.0}]
|
||||
out = boost(entries, 9, 2.0)
|
||||
assert [e["strength"] for e in out] == [2.0, 2.0]
|
||||
|
||||
|
||||
# --- fabricate_reference_entries (no-LoRA path) ----------------------------
|
||||
|
||||
def test_fabricate_splits_keyframes_and_refs():
|
||||
# 5 guide frames (4 kf + 1 ref), 3275 guide tokens -> ref share ~ 1/5.
|
||||
out = fab(3275, 1, 4, 2.0)
|
||||
assert len(out) == 2
|
||||
kf, ref = out
|
||||
assert kf["strength"] == 1.0
|
||||
assert ref["strength"] == 2.0
|
||||
assert kf["surviving_count"] + ref["surviving_count"] == 3275
|
||||
assert ref["surviving_count"] == round(3275 * 1 / 5)
|
||||
|
||||
|
||||
def test_fabricate_no_keyframes_is_all_reference():
|
||||
out = fab(1000, 1, 0, 3.0)
|
||||
assert len(out) == 1
|
||||
assert out[0]["strength"] == 3.0
|
||||
assert out[0]["surviving_count"] == 1000
|
||||
|
||||
|
||||
def test_fabricate_entry_shape_matches_core_builder():
|
||||
# core reads strength, surviving_count, and .get(pixel_mask)/.get(latent_shape)
|
||||
out = fab(800, 2, 2, 2.0)
|
||||
for e in out:
|
||||
assert set(("strength", "surviving_count", "pixel_mask", "latent_shape")) <= set(e)
|
||||
|
||||
|
||||
def test_fabricate_empty_when_no_refs_or_tokens():
|
||||
assert fab(0, 1, 4, 2.0) == []
|
||||
assert fab(3275, 0, 4, 2.0) == []
|
||||
|
||||
|
||||
def test_fabricate_ref_tokens_clamped_at_least_one():
|
||||
# tiny ref share rounds toward >=1
|
||||
out = fab(10, 1, 100, 2.0)
|
||||
ref = out[-1]
|
||||
assert ref["surviving_count"] >= 1
|
||||
assert sum(e["surviving_count"] for e in out) == 10
|
||||
+10
-5
@@ -3847,20 +3847,25 @@ export function renderPanel(container, options = {}) {
|
||||
if (showUpdateFooter) {
|
||||
const starFooter = document.createElement("div");
|
||||
starFooter.className = "koolook-star-footer";
|
||||
// One inline text flow (not separate flex children) so the footer's
|
||||
// `gap` doesn't space out the words; the star glyph and GitHub link sit
|
||||
// inline inside the sentence. The action ("Give a ★") leads the ask.
|
||||
const starText = document.createElement("span");
|
||||
starText.append("Help creators find us! Give a ");
|
||||
const starGlyph = document.createElement("span");
|
||||
starGlyph.className = "koolook-star-glyph";
|
||||
starGlyph.textContent = "★";
|
||||
starGlyph.setAttribute("aria-hidden", "true");
|
||||
const starText = document.createElement("span");
|
||||
starText.textContent = "Help creators find us on ";
|
||||
starGlyph.setAttribute("role", "img");
|
||||
starGlyph.setAttribute("aria-label", "star");
|
||||
starText.append(starGlyph, " on ");
|
||||
const starLink = document.createElement("a");
|
||||
starLink.href = GITHUB_REPO_URL;
|
||||
starLink.target = "_blank";
|
||||
starLink.rel = "noopener noreferrer";
|
||||
starLink.textContent = "GitHub";
|
||||
starLink.title = "Help creators find Kforge Labs on GitHub";
|
||||
starLink.title = "Star Kforge Labs on GitHub so more creators find it";
|
||||
starText.append(starLink);
|
||||
starFooter.append(starGlyph, starText);
|
||||
starFooter.append(starText);
|
||||
container.appendChild(starFooter);
|
||||
}
|
||||
|
||||
|
||||
+4
-2
@@ -5545,7 +5545,8 @@ class TimelineEditor {
|
||||
this.promptInput.style.opacity = "1.0";
|
||||
|
||||
const isImage = (this.selectionType === "image") && (seg.type === "image" || seg.type === "video");
|
||||
const strength = isImage ? (seg.guideStrength ?? 1.0) : 1.0;
|
||||
// Koolook: default image-guide strength 0.8 (smooth), not 1.0 (hard pin).
|
||||
const strength = isImage ? (seg.guideStrength ?? 0.8) : 1.0;
|
||||
this.strengthValue.value = strength.toFixed(2);
|
||||
this.strengthValue.disabled = !isImage;
|
||||
this.strengthValue.style.opacity = isImage ? "1.0" : "0.35";
|
||||
@@ -8931,7 +8932,8 @@ class TimelineEditor {
|
||||
const strList = sortedSegments
|
||||
.filter(s => s.type !== "text")
|
||||
.filter(s => s.start + s.length > startFrames && s.start < endFrames)
|
||||
.map(s => (s.guideStrength !== undefined ? s.guideStrength : 1.0).toFixed(2));
|
||||
// Koolook: unset clips default to 0.8 (smooth) instead of 1.0 (hard pin / robotic transitions).
|
||||
.map(s => (s.guideStrength !== undefined ? s.guideStrength : 0.8).toFixed(2));
|
||||
val = strList.join(",");
|
||||
}
|
||||
updateWidgetValue(this.guideStrengthWidget, val);
|
||||
|
||||
Reference in New Issue
Block a user