Files
malkuthro-ComfyUI-Koolook/k_ltx_guide_reference_strength.py
T
961a95b9bd feat(ltx-director): reference-image identity — Ghost Mask + schedulable guide attention (#264)
* feat(ltx-director): Ghost Mask reference image (REF / character sheet)

Add a no-LoRA reference-image identity anchor to LTXDirector__koolook, the
way a WAN VACE reference works: attach reference image(s) and the model keeps
the character's face / mouth consistent even when keyframes show a closed (or
different) mouth than the audio-driven motion needs. Targets the mouth /
identity drift seen in the talking-goat lip-sync workflow.

Mechanism (Ghost Mask): the reference image(s) are appended as masked guide
frames *after* the clean video region, so the transformer attends to them for
identity but they never appear in the output; the trailing reference frames
are then sliced off the result.

- LTXDirector__koolook: new optional `reference_images` (IMAGE) +
  `reference_strength` inputs, and appended `clean_latent_frames` /
  `clean_pixel_frames` outputs. All additive/optional -> saved workflows
  unaffected. Refs are appended AFTER snap/ease so they are never grid-snapped
  or ease-expanded; the auto latent is grown to hold them. ltx_director.py is
  under ruff's forks/ exclude.
- reference_ghost.py (new, Koolook-original, pure stdlib): trailing-reference
  geometry (insert frames, grown latent length, slice bounds). Unit-tested.
- k_clean_latent_slice.py (new root node `CleanLatentSlice`): torch.narrow
  slice to drop the trailing reference frames (start=0,
  length=clean_latent_frames). torch imported lazily; clamp math unit-tested,
  slice path tested when real torch is present (skips under the CI stub).
- Registered the node; added to sync_to_dev RUNTIME_PATHS.
- Attribution: the Ghost Mask APPROACH is adapted (idea only, reimplemented)
  from CGlide/WhatDreamsCost-CSGlide (main_cs @ 21dd076, GPL-3.0, license
  verified 2026-06-24, GPL->GPL OK). THIRD_PARTY.md, forks_manifest.yaml,
  UPSTREAM_PIN.yaml, CHANGELOG updated.

Phase 1: Ghost Mask only. The LoRA-backed Licon MSR prefix mode and the
editor's character-slot UI are intentionally NOT ported.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(sidebar): action-led star footer ("Give a ★ on GitHub")

Reword the GitHub discovery footer from the passive "Help creators find
us on GitHub" to an action-led "Help creators find us! Give a ★ on
GitHub", with the star as the explicit ask. Render the sentence as one
inline text flow (not separate flex children) so the footer's `gap`
doesn't space out the words; the ★ glyph keeps its gold styling inline and
now carries role="img"/aria-label="star" so it is announced. Verified
rendering in an isolated harness.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ltx-director): Ghost Mask broke motion in the append-and-crop pipeline

The upstream guide pipeline (LTXDirectorGuide + LTXDirectorCropGuides)
appends every guide_data image as an extra latent frame and crops them
back off via `exact_crop_frames = latent.shape[2] - initial_latent_length`.
The Ghost Mask code (modeled on CGlide's CleanLatentSlice flow) ALSO
pre-grew the video latent by n_refs, so initial_latent_length became
clean+n_refs: CropGuides then left n_refs frames uncropped AND the audio
latent stayed at the clean length, desyncing the A/V joint transformer and
breaking motion.

Fix: don't pre-grow the latent. References are added to guide_data exactly
like timeline keyframes and ride the same append-and-crop path — the guide
node appends them and CropGuides removes them, audio untouched. Auto latent
is sized to the clean length again. CleanLatentSlice/clean_latent_frames
remain available for bare pipelines that don't use LTXDirectorCropGuides.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(ltx): LTX Guide Reference Strength — per-stage reference boost

Adds a small Koolook node that rewrites the strength of ONLY the Ghost
Mask reference frames in guide_data (the trailing entries the Director now
tags with `reference_count`), leaving the keyframe pins untouched. Insert
it before a stage-2 refinement guide and set ~0.9 to lock identity during
the low-denoise pass without disturbing the motion set in stage 1 — the
deterministic form of "more reference in stage 2".

Chosen over vendoring the 600-line upstream LTXDirectorGuide (which would
force a node-ID rename + graph rewire): this is one small node inserted on
the stage-2 guide_data wire. The sigma-ramp alternative (ramp reference up
at low sigma) is deferred — the reference hook is noise_mask + a core-model
attention term, so it needs a live-model spike, not a clean static patch.

- Director tags guide_data["reference_count"] = n_refs.
- k_ltx_guide_reference_strength.py: pure strength rewrite (absolute /
  multiply, clamped), unit-tested; registered; added to sync paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(ltx): LTX Reference Bind Schedule — sigma-ramp the reference attention

Confirmed against core (comfy/ldm/lightricks/model.py): the reference's
guide attention strength is turned into a log-space additive self-attention
bias in `_build_guide_self_attention_mask`, which is rebuilt EVERY denoise
step and explicitly supports strength > 1.0 to amplify. That's a clean
per-step hook — so the "inverse sigma curve" idea is buildable, not just a
stage knob.

New node mirrors LTXAVBindSchedule: a model_function_wrapper captures the
per-step sigma and stashes a gain in transformer_options; an object-patch on
diffusion_model._build_guide_self_attention_mask multiplies the strength of
ONLY the trailing `num_references` guide entries (references are appended
after keyframes) by a smoothstep ramp — neutral early (motion forms), rising
to peak_strength late (identity locks during refinement). Keyframe pins and
noise-mask freezing untouched; orthogonal to reference_strength. One pass, no
re-noise, no LoRA.

- ref_gain ramp + _boosted_entries unit-tested (11 tests); model wiring
  validated by live render.
- Registered; added to sync paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ltx): mark Reference Bind Schedule inert without an IC-LoRA

Live diagnostic on the AV model showed the builder is reached and the sigma
gain flows in, but resolved_guide_entries=None despite 3276 guide tokens.
Root cause: LTXDirectorGuide only records guide_attention_entries when an
IC-LoRA is active (`if is_lora_active:` gates _append_guide_attention_entry).
With no LoRA there is no per-guide attention bias to scale, so the
attention-ramp lever does not exist in the Ghost Mask path — it can only
attenuate/amplify entries that an IC-LoRA created.

Rather than silently no-op, the node now logs a clear INERT warning when the
entry list is empty, and the docstring/DESCRIPTION/CHANGELOG state the
IC-LoRA requirement. The node remains correct for the IC-LoRA path (its
partner). No-LoRA reference strengthening stays on the noise-mask lever
(reference_strength / LTX Guide Reference Strength).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(ltx): Reference Bind Schedule works without an IC-LoRA (fabricates entries)

The attention-bias identity channel (_build_guide_self_attention_mask) is
base-model functionality, but LTXDirectorGuide only populates the per-guide
entries when an IC-LoRA is active, so the node was inert in the no-LoRA Ghost
Mask path. Now, when entries are missing, the node FABRICATES them for the
trailing reference tokens: it splits num_guide_tokens across
(num_keyframes + num_references) guide frames and assigns the sigma-ramped
strength to the reference share, then hands off to the core builder. This
gives the reference the same amplified-attention channel an IC-LoRA would use
— inside the Director, no guide-node vendor, no graph rewire (the node is
already in the user's setup). When an IC-LoRA IS active it scales the real
entries as before.

- New `fabricate_reference_entries(num_guide_tokens, num_references,
  num_keyframes, strength)` (pure, unit-tested) + `num_keyframes` input.
- Wrapper now does scale-or-fabricate; clearer one-shot diagnostics.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ltx): keep num_keyframes last so pre-existing RefBind widgets stay aligned

num_keyframes was inserted mid widget-order, which shifted ramp_start/ramp_end
for workflows saved before it existed. Move it to the end of the optional
inputs so saved [num_references, peak_strength, ramp_start, ramp_end] values
keep mapping correctly; num_keyframes just defaults to 0 on reload.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(test): drop unused pytest import (ruff F401)

The guide-reference-strength test uses plain asserts; the stray
`import pytest` tripped ruff F401 and reddened the Lint & Smoke CI job.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ltx-director): default keyframe guidance 0.8 (smooth), not 1.0

The timeline editor auto-built the guide_strength string from each clip's
per-segment guideStrength, filling 1.0 for any clip left unset (and the
Python fallback matched). 1.0 is a hard pin, so a clip you forgot to lower
snapped instead of transitioning smoothly. Unset clips now default to 0.8
in the editor (display + serialization) and the node; explicit per-clip
values still override.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(ltx): Keyframe Soften Schedule — smooth motion early, sharp poses late

New model-patcher node LTXKeyframeSoftenSchedule: sigma-schedules the LTX
keyframe denoise-mask pins so the model forms smooth motion between poses
early (pins loosened at high sigma) and resolves sharp poses late (pins
restored by a crossover). Uses the sampler's per-step denoise_mask_function
hook — no IC-LoRA required — and adds no guide frames (so it can't fabricate
extra poses). Audio-safe: only the video half of the NestedTensor A/V mask is
softened; the audio mask is untouched, so lip-sync is unaffected.

Two normalized 0-1 widgets map onto the sensitive useful range: max_soften
(ease, ->0-0.20) and crossover (invention window, ->0-0.05). Pure soften_gain
ramp unit-tested (11 tests); model wiring render-validated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(review): chain denoise hook, optional_latent clean-length, audio test, CGlide manifest

Addresses PR #264 review findings:

- MEDIUM (Soften node): chain a pre-existing denoise_mask_function instead of
  clobbering it (mirrors the sibling LTX patchers' wrapper-chaining), so the
  node composes with anything else that sets that hook.
- HIGH #2 (Director): clean_pixel_frames/clean_latent_frames now derive from the
  ACTUAL optional_latent shape when one is connected, so Ghost-Mask reference
  geometry and the reported counts follow the real latent instead of the
  timeline duration (with a safe fallback + warning).
- LOW (Soften node): reconcile docstring (crossover is sigma-progress, maps to
  internal 0-0.05), one-time warn on the sigma-read fallback, and extract a
  testable soften_denoise_mask() helper.
- LOW: lock the audio-safety guarantee with NestedTensor-stub tests — video
  mask softened, audio mask untouched, part-ordering asserted.
- #6: add a formal forks_manifest.yaml entry for the CSGlide Ghost Mask
  idea-only adaptation (was prose-only in another entry's notes; THIRD_PARTY.md
  already carries the full attribution).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: AI Machine <ai.machine@x.baconcph.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 21:30:46 +02:00

91 lines
3.9 KiB
Python

# SPDX-License-Identifier: GPL-3.0-or-later
#
# ComfyUI-Koolook — LTX Guide Reference Strength node.
# Copyright (C) 2026 ComfyUI-Koolook contributors (kforgelabs).
"""LTX Guide Reference Strength (Koolook) — boost ONLY the reference frames.
Companion to LTX Director's Ghost Mask reference feature. The Director appends
the reference image(s) as the LAST entries in ``guide_data`` (tagged with
``reference_count``) so they ride the same append-and-crop path as the timeline
keyframes. Every guide frame's strength becomes a noise-mask pin
(``noise_mask = 1 - strength``) in the downstream LTXDirectorGuide.
This node rewrites the strength of just those trailing reference entries,
leaving the keyframe pins untouched. The intended use is a two-stage pipeline:
feed stage 1 the Director's ``guide_data`` unchanged (light reference, so motion
forms freely), and route stage 2's guide_data through this node at a higher
reference strength — so the identity locks during the low-denoise refinement
without disturbing the motion already set in stage 1.
Pure-python; no torch. The strength math is unit-tested.
"""
from __future__ import annotations
import logging
log = logging.getLogger(__name__)
def scaled_reference_strengths(strengths, reference_count, value, mode="absolute"):
"""Return a new strengths list with the last ``reference_count`` entries
rewritten. ``mode='absolute'`` sets them to ``value``; ``mode='multiply'``
multiplies the existing strength by ``value``. Other entries (keyframes) are
left untouched. Clamps to [0, 1] (guide strength -> noise-mask range)."""
out = [float(s) for s in strengths]
n = len(out)
k = max(0, min(int(reference_count), n))
for i in range(n - k, n):
v = float(value) if mode == "absolute" else out[i] * float(value)
out[i] = max(0.0, min(1.0, v))
return out
class LTXGuideReferenceStrength:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"guide_data": ("GUIDE_DATA",),
"reference_strength": ("FLOAT", {
"default": 0.9, "min": 0.0, "max": 1.0, "step": 0.01,
"tooltip": "New strength for the reference frames only. Higher = harder identity pin. Use ~0.9 on a stage-2 refinement guide.",
}),
"mode": (["absolute", "multiply"], {
"default": "absolute",
"tooltip": "absolute: set reference strength to this value. multiply: scale the Director's reference strength by it.",
}),
}
}
RETURN_TYPES = ("GUIDE_DATA",)
RETURN_NAMES = ("guide_data",)
FUNCTION = "apply"
CATEGORY = "Koolook/LTX"
DESCRIPTION = (
"Rewrite the strength of just the Ghost Mask reference frames in "
"guide_data (the trailing entries tagged by LTX Director), leaving the "
"keyframe pins untouched. Insert before a stage-2 refinement guide to "
"lock identity harder without disturbing stage-1 motion."
)
def apply(self, guide_data, reference_strength, mode):
ref_count = int((guide_data or {}).get("reference_count", 0) or 0)
if not guide_data or ref_count <= 0:
log.info("[LTXGuideReferenceStrength] no reference frames in guide_data; passthrough.")
return (guide_data,)
out = dict(guide_data)
out["strengths"] = scaled_reference_strengths(
guide_data.get("strengths", []), ref_count, reference_strength, mode,
)
log.info(
"[LTXGuideReferenceStrength] set %d reference frame(s) -> %.2f (%s).",
ref_count, float(reference_strength), mode,
)
return (out,)
NODE_CLASS_MAPPINGS = {"LTXGuideReferenceStrength": LTXGuideReferenceStrength}
NODE_DISPLAY_NAME_MAPPINGS = {"LTXGuideReferenceStrength": "LTX Guide Reference Strength (Koolook)"}
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]