fix: H3 image conditioning reaches the model

Two independent, silent breaks made an H3 i2v scene generate from the prompt
alone.

1. The Editor always runs split_by_transitions, and the per-scene conditionings
   REPLACE the single entry Studio built — but they were re-encoded with no
   reference_image and no h3_references. Both the frame-0 keyframe pin and the
   Qwen vision presentation were gone before the sampler looked, even for a
   one-scene project. The split now carries the anchor onto scene 0 and the
   references onto every scene. Gated to H3: on LTX the anchor rides in the
   latent, and adding Gemma3 vision to scene 0 would change existing output.

2. A MiniMax H3 Image to Video node hands its first/last frame pins out on its
   CONDITIONING output, which had nowhere to go — the sampler's positive comes
   from Studio. New optional h3_keyframes input on the Chain Sampler salvages
   them: first frame onto the opening scene, last frame re-indexed onto the
   closing scene's own final frame (the source node's frame_count is its own,
   and a stale index would land mid-clip where PackedLayout refuses it). The
   editor auto-wires it for H3 projects.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
digital-garbage
2026-08-05 15:40:23 +03:00
co-authored by Claude Opus 5
parent cd6639e7a7
commit fdf39bbd03
8 changed files with 289 additions and 10 deletions
+23
View File
@@ -2,6 +2,29 @@
## [Unreleased]
### Fixed
- **H3 image conditioning actually reaches the model.** Two independent breaks, both silent,
both ending with a generation that ignored the input image entirely and left the prompt to
carry the whole shot. The Movie Editor always splits a run into per-scene conditionings,
and that split re-encoded every scene *without* the anchor image or the ref2va references —
so neither the keyframe pin nor the `<Picture i>` presentation ever reached the sampler,
even for a single-scene project. The split now carries the anchor onto the opening scene
and the references onto every scene. LTX projects are untouched: there the anchor travels
in the latent, and adding vision to scene 1 would change existing output.
- **A wired `MiniMax H3 Image to Video` node no longer throws its image away.** That node
carries its first/last frame pins on its CONDITIONING output, which this pipeline dropped
because the sampler's positive comes from Studio. New optional **Chain Sampler ·
h3_keyframes** input (auto-wired for H3 projects) salvages the pins — first frame onto the
opening scene, last frame re-indexed onto the closing scene's own final frame.
### Changed
- **Guide strengths span the full 0..1 range.** Mid-scene guide, identity pin, prior-scene
guide, the per-guide strength field, and the JoyAI memory floor were clamped to 0.25–0.5.
That band is the measured audio-safe sweet spot, not a physical limit; it stays in the
tooltips and out of the code.
- The H3 `v2a_grad_scale` warning chip is gone. The knob does nothing without JoyAI audio
memory on any model, so flagging it as an H3 limitation blamed the wrong thing.
## [3.5.0] - 2026-08-04
A second model family. FunPack was built around LTX-2 / LTXAV and now supports MiniMax H3
+34 -2
View File
@@ -6938,7 +6938,25 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner):
scene_effects.append(effect)
return scene_texts, scene_effects
def _v2_transition_scene_conditionings(self, clip, scene_texts, scene_effects=None, encode_cache=None):
def _v2_transition_scene_conditionings(self, clip, scene_texts, scene_effects=None, encode_cache=None,
reference_image=None, h3_references=None):
"""Per-scene conditionings for the Chain Sampler, one entry per scene.
These REPLACE the single-entry conditioning built above, so anything that lived on
that entry has to be re-established here or it never reaches the sampler. That is
why the visual conditioning travels in:
* ``reference_image`` — the i2v anchor, H3 only. Scene 0 only, matching how the chain
reads it (later scenes continue from the previous scene's output unless they carry
their own anchor). It has to be re-encoded, not copied: on H3 the anchor is presented
to Qwen inside the tokenize call, so a scene encoded without it is textually blind to
the image no matter what metadata is attached afterwards. LTX is left alone — there
the anchor reaches the model through the latent, and adding Gemma3 vision to scene 0
would change what every existing multi-scene LTX project generates.
* ``h3_references`` — ref2va reference media. EVERY scene: a reference identity holds
across the whole chain, and the ``<Picture i>`` ordinals in a scene's prompt only
resolve if that scene's own encode presented them.
"""
if clip is None:
return None
scene_texts = list(scene_texts or [])
@@ -6946,6 +6964,12 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner):
return None
scene_effects = list(scene_effects or [])
try:
from . import minimax_h3 as h3mod
except ImportError:
import minimax_h3 as h3mod
anchor_image = reference_image if h3mod.is_h3_clip(clip) else None
conditionings = []
scene_count = len(scene_texts)
seen_texts = {}
@@ -6953,7 +6977,11 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner):
seen_count = seen_texts.get(scene_text, 0)
seen_texts[scene_text] = seen_count + 1
encode_text = f"Returning to an earlier scene: {scene_text}" if seen_count > 0 else scene_text
cond, meta, _ = self._v2_encode_prompt(clip, encode_text, encode_cache=encode_cache)
scene_image = anchor_image if scene_index == 0 else None
cond, meta, _ = self._v2_encode_prompt(
clip, encode_text, encode_cache=encode_cache,
reference_image=scene_image, h3_references=h3_references,
)
if not isinstance(cond, torch.Tensor):
return None
scene_meta = dict(meta) if isinstance(meta, dict) else {"pooled_output": None}
@@ -12866,6 +12894,10 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner):
if scene_texts:
scene_conditionings = self._v2_transition_scene_conditionings(
clip, scene_texts, scene_effects=split_scene_effects, encode_cache=encode_cache,
# The per-scene entries replace `output_conditioning` wholesale, so H3's
# visual conditioning has to be re-established here or the anchor image
# and the ref2va references never reach the sampler at all.
reference_image=source_image, h3_references=_h3_references,
)
if scene_conditionings is not None:
output_conditioning = self._v2_apply_scene_seed_metadata(
+18 -2
View File
@@ -681,8 +681,24 @@ without complaint: `minimax_h3_fl2va_*` for text-to-video and anchor/keyframe sc
using both at once in one scene means one of them is untrained conditioning. Text encoder
and both VAEs are shared. Needs ComfyUI v0.30.0 or newer.
**Inert on H3:** Best-FaceID identity transfer, Bounded Attention, segmented detailing, the
and the second pass's latent op all depend on LTX transformer structure H3 does not have.
**Getting an image into an H3 scene — two routes, both supported:**
1. **Timeline image → `Studio · source_image`** (the default wiring for an image_processing
loader). Studio presents it to Qwen and hands the pixels to the Chain Sampler, which pins
it as the opening scene's frame-0 keyframe. This is the route the Editor's own image
scenes use.
2. **A `MiniMax H3 Image to Video` node's `first_frame` / `last_frame`.** That node emits its
pins on its **CONDITIONING** output — which used to go nowhere, because the sampler's
positive comes from Studio, so the image was silently discarded. Wire that output to
**Chain Sampler · h3_keyframes** (auto-wired for new projects) and the pins survive: a
first_frame lands on scene 1, a last_frame on the last scene's final frame. Only the pins
are read — write the prompt in Studio as usual.
The run prints `h3_keyframe_anchor(...)` or `h3_wired_keyframes(...)` in its scene mechanics.
If neither appears, the model never saw the image.
**Inert on H3:** Best-FaceID identity transfer, Bounded Attention, segmented detailing, and
the second pass's latent op all depend on LTX transformer structure H3 does not have.
The Chain Sampler switches them off and says why; the main window warns before the run.
**Wrong:** Install stuck → **Cancel install**. Pip security policy errors → install packs manually in terminal. After Manager install, restart before pack install.
+4
View File
@@ -136,6 +136,10 @@ FAMILIES: dict[str, dict] = {
("audiodec", "vae", "VAE", True),
# optional: only needed to encode AUDIO ref2va references
("sampler", "audio_vae", "VAE", False),
# optional: a MiniMax H3 Image to Video node emits its first/last frame pins
# on its CONDITIONING output, which this pipeline otherwise drops (the
# sampler's positive comes from Studio). Wiring it here keeps the pins.
("sampler", "h3_keyframes", "CONDITIONING", False),
),
},
},
+18 -5
View File
@@ -103,6 +103,10 @@ PORT_TO_OPEN_CORE: dict[str, tuple[str, str]] = {
#
# Everything not listed is inherited, so a rule added for LTXAV reaches H3 too.
_H3_LATENT_PORT = "FunPackLTXAVSceneChainSampler.latent_template"
# A MiniMax H3 Image to Video node in the latent role also emits CONDITIONING carrying its
# first_frame / last_frame pins. Studio owns the sampler's positive, so that conditioning has
# nowhere to go and the image is silently lost; this port is where the pins are salvaged.
_H3_KEYFRAME_PORT = "FunPackLTXAVSceneChainSampler.h3_keyframes"
FAMILY_WIRING: dict[str, dict] = {
"ltxav": {},
@@ -111,19 +115,27 @@ FAMILY_WIRING: dict[str, dict] = {
"audio_vae": [("VAE", None, "VAEDecodeAudio.vae"),
("VAE", None, "FunPackLTXAVSceneChainSampler.audio_vae")],
"audio_encoder": [],
"empty_latent": [("LATENT", None, _H3_LATENT_PORT)],
"video_latent": [("LATENT", None, _H3_LATENT_PORT)],
"empty_latent": [("LATENT", None, _H3_LATENT_PORT),
("CONDITIONING", None, _H3_KEYFRAME_PORT)],
"video_latent": [("LATENT", None, _H3_LATENT_PORT),
("CONDITIONING", None, _H3_KEYFRAME_PORT)],
"image_processing": [("IMAGE", None, "FunPackStudio.source_image")],
},
"type_chain_terminals": {"LATENT": [_H3_LATENT_PORT]},
"type_chain_terminals": {"LATENT": [_H3_LATENT_PORT],
"CONDITIONING": [_H3_KEYFRAME_PORT]},
"default_wires": {
"audio_vae": {"VAE": "port:VAEDecodeAudio.vae"},
"audio_encoder": {},
"empty_latent": {"LATENT": "port:" + _H3_LATENT_PORT},
"video_latent": {"LATENT": "port:" + _H3_LATENT_PORT},
# both outputs of the H3 latent node are wired by default: the AV latent, and the
# keyframe pins that come with it when the node has a first/last frame image
"empty_latent": {"LATENT": "port:" + _H3_LATENT_PORT,
"CONDITIONING": "port:" + _H3_KEYFRAME_PORT},
"video_latent": {"LATENT": "port:" + _H3_LATENT_PORT,
"CONDITIONING": "port:" + _H3_KEYFRAME_PORT},
},
"port_labels": {
_H3_LATENT_PORT: "Chain Sampler · latent_template (H3 AV latent)",
_H3_KEYFRAME_PORT: "Chain Sampler · h3_keyframes (first/last frame pins)",
"VAEDecodeAudio.vae": "VAE Decode Audio · vae",
"FunPackLTXAVSceneChainSampler.audio_vae": "Chain Sampler · audio_vae (ref2va audio)",
},
@@ -134,6 +146,7 @@ FAMILY_WIRING: dict[str, dict] = {
"FunPackStudio.source_image": ("studio", "source_image"),
"FunPackLTXAVSceneChainSampler.vae": ("sampler", "vae"),
_H3_LATENT_PORT: ("sampler", "latent_template"),
_H3_KEYFRAME_PORT: ("sampler", "h3_keyframes"),
"FunPackLTXAVSceneChainSampler.audio_vae": ("sampler", "audio_vae"),
"VAEDecodeAudio.vae": ("audiodec", "vae"),
},
+72 -1
View File
@@ -2351,6 +2351,9 @@ class FunPackLTXAVSceneChainSampler:
"audio_vae": ("VAE", {
"tooltip": "MiniMax H3 only: the audio VAE, needed to encode AUDIO reference media for ref2va (a voice or ambience clip the generation should sound like). Studio lists the references and bakes their <Audio j> labels into the prompt; this VAE turns them into the latent blocks the DiT packs. Image references need only the main vae and work without this. Ignored entirely on LTX.",
}),
"h3_keyframes": ("CONDITIONING", {
"tooltip": "MiniMax H3 only: the CONDITIONING output of a MiniMax H3 Image to Video node, wired here purely so its first_frame / last_frame pins survive. That node's conditioning is otherwise discarded (the sampler's positive comes from Studio), which silently drops the image. Only the keyframe pins are read — the prompt encoded by that node is ignored, so write your prompt in Studio as usual. A first-frame pin lands on scene 1, a last-frame pin on the last scene's final frame. Ignored entirely on LTX.",
}),
"refinement_key_input": ("STRING", {
"default": "",
"multiline": False,
@@ -4267,6 +4270,59 @@ class FunPackLTXAVSceneChainSampler:
aug=None if aug >= 1.0 else aug)
return positive, True
def _h3_external_pins(self, conditioning):
"""Rescue the keyframe pins from a MiniMax H3 Image to Video node's conditioning.
That node encodes first_frame/last_frame into `minimax_keyframes` and hands them out
on its CONDITIONING output — which this pipeline drops, because the sampler's positive
comes from Studio. Wiring the node's conditioning into `h3_keyframes` lets the pins
travel anyway. Only the pins are taken; the node's own prompt encode is not used.
Returns ``{"first": [...], "last": [...], "aug": float|None}``. The node writes a
first_frame at index 0 and a last_frame at ITS frame_count - 1, which is a per-node
length that need not match this run's scene length — so pins are classified here and
re-indexed against the scene they land on, never trusted as absolute positions.
"""
pins = self._conditioning_value(conditioning, "minimax_keyframes") or []
if not pins:
return None
source_count = self._conditioning_value(conditioning, "minimax_frame_count")
first, last = [], []
for pin in pins:
try:
index = int(pin["resolved_frame_index"])
except (KeyError, TypeError, ValueError):
continue
(first if index == 0 else last).append(pin)
if not first and not last:
return None
aug = self._conditioning_value(conditioning, "minimax_visual_cond_noise_aug")
return {
"first": first,
"last": last,
"aug": float(aug) if aug is not None else None,
"source_count": int(source_count) if source_count else None,
}
def _apply_h3_external_pins(self, positive, pins, scene_index, scene_count):
"""Place rescued pins on the scene they belong to: first on the opening scene, last
on the closing one. A one-scene run gets both, which is the plain single-clip case.
Returns (positive, applied_labels)."""
frame_count = max(1, int(self._h3_frame_count))
placed, labels = [], []
if scene_index == 0 and pins["first"]:
placed.extend({**p, "resolved_frame_index": 0} for p in pins["first"])
labels.append("first")
if scene_index == scene_count - 1 and pins["last"]:
# re-indexed onto THIS scene's last frame — the source node's frame_count is its
# own, and a pin at a stale index would land mid-clip, where the layout refuses it
placed.extend({**p, "resolved_frame_index": frame_count - 1} for p in pins["last"])
labels.append("last")
if not placed:
return positive, []
return self._h3_add_keyframes(positive, placed, frame_count, aug=pins["aug"]), labels
def _append_h3_keyframe(self, guide_frame, apply_at, strength, positive, negative):
"""H3's equivalent of an LTX guide: a keyframe pin carried on the conditioning.
@@ -5490,7 +5546,7 @@ class FunPackLTXAVSceneChainSampler:
context_window_freenoise=True, context_window_retain_first=False,
cut_opening_frames=0,
second_pass=False, second_pass_op="none", second_pass_sigmas=None,
audio_vae=None,
audio_vae=None, h3_keyframes=None,
unique_id=None, prompt=None):
if not isinstance(positive, list) or not positive:
raise ValueError("positive conditioning must contain at least one scene entry.")
@@ -5645,6 +5701,13 @@ class FunPackLTXAVSceneChainSampler:
max_scene_count = max(1, int(max_scenes))
scene_conditionings = positive[:max_scene_count]
scene_count = len(scene_conditionings)
# Keyframe pins wired in from a MiniMax H3 Image to Video node, read once for the run.
_h3_wired_pins = self._h3_external_pins(h3_keyframes) if self._is_h3 else None
if h3_keyframes and not self._is_h3:
print("[FunPackSceneChain] h3_keyframes is wired but this model is not MiniMax H3 — ignored.")
elif h3_keyframes and _h3_wired_pins is None:
print("[FunPackSceneChain] h3_keyframes is wired but carries no keyframe pins — the "
"source node had no first_frame/last_frame image.")
time_scale = self._time_scale(vae)
video_frames = self._validate_template_length(latent_template, num_frames_per_scene, time_scale, vae=vae)
video_overlap = self._overlap_frames(latent_template, frame_overlap, time_scale, vae=vae)
@@ -5916,6 +5979,14 @@ class FunPackLTXAVSceneChainSampler:
scene_positive, chunk, vae, _anchor_image, _anchor_strength)
if _pinned:
run_mechanisms.append(f"h3_keyframe_anchor(strength={_anchor_strength:g})")
# Pins rescued from a wired MiniMax H3 Image to Video node. Applied AFTER the
# anchor so an explicitly wired first_frame wins the frame-0 slot — the user
# drew that wire, the anchor is inferred from the timeline.
if _h3_wired_pins:
scene_positive, _pin_labels = self._apply_h3_external_pins(
scene_positive, _h3_wired_pins, scene_index, scene_count)
if _pin_labels:
run_mechanisms.append(f"h3_wired_keyframes({'+'.join(_pin_labels)})")
# ref2va reference blocks. Applied after the chunk is final because the blocks are
# sized against this scene's canvas, and to every scene branch (fresh / anchored /
+77
View File
@@ -479,3 +479,80 @@ def test_mixing_pins_and_references_is_called_out(capsys):
}]])
out = capsys.readouterr().out
assert "BOTH" in out and "fl2va" in out and "ref2va" in out
# ── pins rescued from a wired MiniMax H3 Image to Video node ─────────────────
# That node hands its first_frame/last_frame pins out on its CONDITIONING output, which this
# pipeline drops (Studio owns the sampler's positive). Wiring it into h3_keyframes is the
# difference between the image conditioning the generation and being silently discarded.
def _wired_conditioning(indices, frame_count=124, aug=None):
pins = [{"resolved_frame_index": i, "latent": torch.full((1, 24, 1, 48, 84), float(i))}
for i in indices]
meta = {"minimax_keyframes": pins, "minimax_frame_count": frame_count}
if aug is not None:
meta["minimax_visual_cond_noise_aug"] = aug
return [[torch.zeros(1, 12, 5120), meta]]
def test_wired_pins_are_split_into_first_and_last():
node = h3_node()
pins = node._h3_external_pins(_wired_conditioning([0, 123], aug=0.8))
assert [p["resolved_frame_index"] for p in pins["first"]] == [0]
assert [p["resolved_frame_index"] for p in pins["last"]] == [123]
assert pins["aug"] == pytest.approx(0.8)
assert pins["source_count"] == 124
def test_conditioning_without_pins_is_reported_as_nothing_to_rescue():
node = h3_node()
assert node._h3_external_pins([[torch.zeros(1, 12, 5120), {}]]) is None
assert node._h3_external_pins(None) is None
def test_a_first_frame_pin_lands_on_the_opening_scene_only():
node = h3_node(frame_count=124)
pins = node._h3_external_pins(_wired_conditioning([0]))
positive = [[torch.zeros(1, 12, 5120), {}]]
out, labels = node._apply_h3_external_pins(positive, pins, scene_index=0, scene_count=3)
assert labels == ["first"]
assert [kf["resolved_frame_index"] for kf in out[0][1]["minimax_keyframes"]] == [0]
out, labels = node._apply_h3_external_pins(positive, pins, scene_index=1, scene_count=3)
assert labels == [] and out is positive # untouched mid-chain
def test_a_last_frame_pin_is_reindexed_onto_the_closing_scenes_own_length():
"""The source node's frame_count is its own — a stale index would land mid-clip, where
PackedLayout refuses it."""
node = h3_node(frame_count=209)
pins = node._h3_external_pins(_wired_conditioning([0, 123], frame_count=124))
out, labels = node._apply_h3_external_pins(
[[torch.zeros(1, 12, 5120), {}]], pins, scene_index=2, scene_count=3)
assert labels == ["last"]
assert [kf["resolved_frame_index"] for kf in out[0][1]["minimax_keyframes"]] == [208]
def test_a_single_scene_run_gets_both_pins():
node = h3_node(frame_count=124)
pins = node._h3_external_pins(_wired_conditioning([0, 123]))
out, labels = node._apply_h3_external_pins(
[[torch.zeros(1, 12, 5120), {}]], pins, scene_index=0, scene_count=1)
assert labels == ["first", "last"]
assert [kf["resolved_frame_index"] for kf in out[0][1]["minimax_keyframes"]] == [0, 123]
def test_a_wired_first_frame_overrides_the_timeline_anchor_on_frame_zero():
"""Both target frame 0; the explicit wire is the one the user drew."""
node = h3_node(frame_count=124)
anchor = torch.zeros(1, 24, 1, 48, 84)
positive = node._h3_add_keyframes(
[[torch.zeros(1, 12, 5120), {}]],
[{"resolved_frame_index": 0, "latent": anchor}], 124)
pins = node._h3_external_pins(_wired_conditioning([0]))
out, _labels = node._apply_h3_external_pins(positive, pins, scene_index=0, scene_count=1)
kfs = out[0][1]["minimax_keyframes"]
assert len(kfs) == 1 and kfs[0]["latent"] is not anchor
+43
View File
@@ -113,3 +113,46 @@ def test_the_reference_size_mode_reaches_the_sampler(h3_media):
H3Clip(), "a shot",
h3_references=[{"kind": "image", "filename": "face.png", "size": "max"}])
assert meta["funpack_h3_refs"][0]["size"] == "max"
# ── the multi-scene split must not drop the visual conditioning ──────────────
# The Movie Editor always runs split_by_transitions, and the per-scene conditionings REPLACE
# the single entry Studio built. Anything that lived on that entry — the anchor, the resolved
# references — has to be re-established per scene or it never reaches the sampler at all.
def test_the_anchor_survives_the_multi_scene_split_on_the_opening_scene():
node = FunPackVideoRefinerV2()
clip = H3Clip()
scenes = node._v2_transition_scene_conditionings(
clip, ["a room", "a street", "a roof"], reference_image=IMAGE)
assert [m["funpack_scene_index"] for _c, m in scenes] == [0, 1, 2]
# scene 0 carries the pin AND was encoded with the image in front of Qwen
assert scenes[0][1]["funpack_h3_anchor"]["image"] is IMAGE
assert clip.tokenize_kwargs[0]["images"] == [IMAGE]
# later scenes continue from the previous scene's output, so no anchor and no vision
assert all("funpack_h3_anchor" not in m for _c, m in scenes[1:])
assert all(not kw.get("images") for kw in clip.tokenize_kwargs[1:])
def test_references_survive_the_split_on_every_scene(h3_media):
"""A reference identity holds across the whole chain, and <Picture 1> only resolves in a
scene whose own encode presented it."""
node = FunPackVideoRefinerV2()
clip = H3Clip()
scenes = node._v2_transition_scene_conditionings(
clip, ["a room", "a street"],
h3_references=[{"kind": "image", "filename": "face.png"}])
assert all([r["filename"] for r in m["funpack_h3_refs"]] == ["face.png"] for _c, m in scenes)
assert all(kw.get("minimax_ref_items") for kw in clip.tokenize_kwargs)
def test_an_ltx_split_is_left_alone():
"""On LTX the anchor reaches the model through the latent. Adding Gemma3 vision to scene 0
here would change what every existing multi-scene LTX project generates."""
node = FunPackVideoRefinerV2()
node._gemma3_has_vision = lambda clip: True
scenes = node._v2_transition_scene_conditionings(LTXClip(), ["a room", "a street"],
reference_image=IMAGE)
assert all("funpack_h3_anchor" not in m for _c, m in scenes)