feat: MiniMax-H3 against the merged release — real i2v anchors, model-owned frame geometry
H3 support was written against ComfyUI's PR; it has since merged (57500fc5, shipped in v0.30.0) and the weights are out. Everything the adapter assumed still holds — audio time axis, 17k+5 grid, batch-1 forward, the imported optimized_attention symbol, every payload key — so this is what the release itself changed, not a correction of the reading. LTX-2 / LTXAV is untouched throughout: a project picks a family and both are first-class. * i2v anchors now work on H3. LTX writes the image into the starting latent (LTXVImgToVideoInplace); H3 has no such path — its anchor is a condition row packed beside the text — so a scene's anchor image becomes a frame-0 keyframe pin instead, and the LTX node is no longer run against an H3 latent. Studio hands the source image's pixels to the sampler (it owns the CLIP, the sampler owns the VAE). Pins now MERGE, so a first-frame anchor and a last-frame guide coexist: H3's own first-and-last mode. * Frame geometry follows the model: frames per scene snap to the family's grid (LTX 8k+1, H3 17k+5) in both frontends, in the graph builder, and in the sampler, and H3 renders at its fixed 24 fps. Picking a family moves the project onto its geometry and says so. * The two checkpoints are named. fl2va (anchors) and ref2va (references) load identically and never reject each other's conditioning, so the run reports which mode it is in and the setup panel spells out the split. * Pipeline setup is no longer a placeholder: released, with the published filenames. * Reference images can be encoded at the reference pipeline's 2048px short edge. Fixes found on the way: * A video reference lost its soundtrack between Studio and the sampler — Studio had already announced it to the text encoder as <Audio 1>, so every later audio ordinal in the prompt pointed one reference earlier than written. * Easy Gen never knew it was on H3: its model config lives on the project, which the shared capability check did not read, so every H3 warning silently reported LTX. * An off-grid scene length failed with arithmetic instead of an explanation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
digital-garbage
co-authored by
Claude Opus 5
Claude Opus 5
parent
349cce2be8
commit
4aec53e01b
@@ -2,6 +2,39 @@
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- **MiniMax H3 is live.** H3 support was written against ComfyUI's pull request; the model
|
||||
has since merged (ComfyUI v0.30.0) and the weights are published, so the pipeline setup
|
||||
now offers it as a real, released family with the actual filenames to download. Nothing
|
||||
about LTX-2 / LTXAV changes — a project picks a family, and both are first-class.
|
||||
- **i2v anchors work on H3.** LTX anchors an image by writing it into the starting latent;
|
||||
H3 has no such path — its anchor is a condition row packed beside the text. A scene's
|
||||
anchor image (and the Cutting Room's per-scene anchors) now becomes H3's own frame-0
|
||||
keyframe pin, so image-to-video actually conditions instead of quietly generating from
|
||||
noise. A first-frame anchor and a last-frame guide now coexist rather than overwriting
|
||||
each other, which is exactly H3's first-and-last-frame mode.
|
||||
- **Reference images can be encoded at full detail** (Settings ▸ Engine ▸ References):
|
||||
*Match output size* as before, or *Max detail (2048px)*, the reference pipeline's own
|
||||
sizing. Reference rows ride through every sampling step, so max detail is not free.
|
||||
- **Frame geometry follows the model.** Frames per scene snap to the chosen family's grid
|
||||
(LTX 8k+1, H3 17k+5) in both frontends and in the graph builder, and H3 renders at its
|
||||
fixed 24 fps. Picking a model family brings the project onto its geometry and says so.
|
||||
|
||||
### Fixed
|
||||
- **A video reference lost its soundtrack between Studio and the sampler.** Studio had
|
||||
already announced the track to the text encoder as `<Audio 1>`, but the reference list it
|
||||
handed on dropped the field — so no audio rows were packed for it and every later
|
||||
`<Audio j>` in the prompt pointed one reference earlier than written.
|
||||
- **Easy Gen never knew it was on H3.** Its model config lives on the project rather than in
|
||||
a separate slot, which the shared capability check did not read — so every H3-specific
|
||||
warning silently reported LTX and never appeared.
|
||||
- **An off-grid scene length on H3 failed with arithmetic instead of an explanation.** The
|
||||
length is now snapped to the model's grid, and a genuine template mismatch names the
|
||||
latent node and the number it needs.
|
||||
- **The run says which H3 checkpoint its conditioning needs.** H3 ships as two DiTs —
|
||||
`fl2va` (anchors/keyframes) and `ref2va` (reference media) — that load identically and
|
||||
never reject each other's conditioning, so a mismatch only shows up as a poor generation.
|
||||
|
||||
## [3.4.1] - 2026-07-31
|
||||
|
||||
### Added
|
||||
|
||||
+16
-2
@@ -6800,10 +6800,24 @@ class FunPackVideoRefinerV2(FunPackVideoRefiner):
|
||||
if resolved_refs:
|
||||
# The resolved order travels with the conditioning: the sampler must encode these
|
||||
# exact references, in this exact order, or "<Picture 2>" points at the wrong one.
|
||||
# Every field that shaped the presentation travels with it — a video's soundtrack
|
||||
# ("audio") especially: Studio already emitted its "<Audio j>" label, so a sampler
|
||||
# that doesn't know about the track packs no audio rows for it and shifts every
|
||||
# later audio ordinal by one.
|
||||
meta = dict(meta)
|
||||
meta["funpack_h3_refs"] = [{"kind": r["kind"], "filename": r["filename"]}
|
||||
for r in resolved_refs]
|
||||
meta["funpack_h3_refs"] = [
|
||||
{k: v for k, v in r.items() if k in ("kind", "filename", "audio", "size")}
|
||||
for r in resolved_refs
|
||||
]
|
||||
vision_tag = f" +{len(resolved_refs)} ref"
|
||||
elif h3 and use_vision:
|
||||
# fl2va: Studio owns the CLIP and has just presented this image to Qwen, but the
|
||||
# anchor also has to reach the DiT as a frame-0 keyframe pin — and that needs the
|
||||
# VAE, which lives on the Chain Sampler. Hand the pixels over so the sampler can
|
||||
# encode them at the scene's canvas. On LTX this never happens: there the anchor
|
||||
# is written into the latent by the graph, not carried on the conditioning.
|
||||
meta = dict(meta)
|
||||
meta["funpack_h3_anchor"] = {"image": reference_image}
|
||||
if use_vision:
|
||||
print(f"[FunPackStudio] Vision encoding done — {self._get_conditioning_seq_len(cond)} positions")
|
||||
result = (cond, meta, f"encoded {self._get_conditioning_seq_len(cond)} positions{vision_tag}")
|
||||
|
||||
@@ -657,6 +657,35 @@ When you open a project using the **built-in pipeline**, the editor checks Comfy
|
||||
| **No, I'll use my own pipeline** | Sets `disable_core`; dismisses future prompts (localStorage) |
|
||||
| **Close** | Dismisses modal without installing |
|
||||
|
||||
### Model family
|
||||
|
||||
The first question in Pipeline setup, because it decides which nodes and model files the
|
||||
project needs. A project is built for **one** family; both are fully supported.
|
||||
|
||||
| | LTX-2 / LTXAV | MiniMax H3 (Hailuo) |
|
||||
|---|---|---|
|
||||
| Text encoder | Gemma3 | Qwen3-VL-32B |
|
||||
| Latents | separate video + audio | one joint AV latent |
|
||||
| Frame grid | 8k+1, project fps | 17k+5, fixed 24 fps |
|
||||
| Image conditioning | i2v anchor written into the latent | frame-0 **keyframe pin** on the conditioning |
|
||||
| Reference media | Best-FaceID identity transfer | native **ref2va** `<Picture i>` / `<Video k>` / `<Audio j>` blocks |
|
||||
|
||||
Choosing a family snaps the project's frames per scene onto that model's grid, and sets the
|
||||
frame rate when the model has a fixed one. Frames per scene and the AV latent node's
|
||||
`length` must agree — bind the node's `length` to **Project · Frames** in Models & Pipeline
|
||||
to keep them in step.
|
||||
|
||||
**H3 ships two diffusion checkpoints** and they are not interchangeable, though both load
|
||||
without complaint: `minimax_h3_fl2va_*` for text-to-video and anchor/keyframe scenes,
|
||||
`minimax_h3_ref2va_*` for reference media. The run prints which mode its conditioning is in;
|
||||
using both at once in one scene means one of them is untrained conditioning. Text encoder
|
||||
and both VAEs are shared. Needs ComfyUI v0.30.0 or newer.
|
||||
|
||||
**Inert on H3:** Best-FaceID identity transfer, Bounded Attention, segmented detailing, the
|
||||
second pass's latent op, and `v2a_grad_scale` all depend on LTX transformer structure H3
|
||||
does not have. The Chain Sampler switches them off and says why; the main window warns
|
||||
before the run.
|
||||
|
||||
**Wrong:** Install stuck → **Cancel install**. Pip security policy errors → install packs manually in terminal. After Manager install, restart before pack install.
|
||||
|
||||
---
|
||||
|
||||
@@ -56,9 +56,20 @@
|
||||
const out = el("div", "sw-rows");
|
||||
out.append(field("Width", numberInput(p.width, (v) => commit((pr) => pr.width = v), { min: 64, step: 16 })));
|
||||
out.append(field("Height", numberInput(p.height, (v) => commit((pr) => pr.height = v), { min: 64, step: 16 })));
|
||||
out.append(field("Frames", numberInput(p.num_frames_per_scene, (v) => commit((pr) => pr.num_frames_per_scene = v), { min: 9, step: 8 }),
|
||||
"Length of the generated clip, in frames."));
|
||||
out.append(field("Frame rate", numberInput(p.frame_rate, (v) => commit((pr) => pr.frame_rate = v), { min: 1 })));
|
||||
// Frame geometry is per model family: LTX generates on an 8k+1 grid at the project's
|
||||
// fps, MiniMax H3 on a 17k+5 grid at a fixed 24 fps. Off-grid lengths fail the run
|
||||
// outright on H3, so the input snaps to the family's grid rather than warning after.
|
||||
const grid = window.PipelineCaps?.frameGrid ? window.PipelineCaps.frameGrid(st)
|
||||
: { step: 8, base: 1, fps: null, label: "8k+1" };
|
||||
const snapTo = (v) => (window.PipelineCaps?.snapFramesTo
|
||||
? window.PipelineCaps.snapFramesTo(v, st, "round")
|
||||
: Math.max(9, Math.round((v - 1) / 8) * 8 + 1));
|
||||
out.append(field("Frames",
|
||||
numberInput(p.num_frames_per_scene, (v) => commit((pr) => pr.num_frames_per_scene = snapTo(v)),
|
||||
{ min: grid.step + grid.base, step: grid.step }),
|
||||
"Length of the generated clip, in frames — snapped to this model's " + grid.label + " grid."));
|
||||
out.append(field("Frame rate", numberInput(p.frame_rate, (v) => commit((pr) => pr.frame_rate = v), { min: 1 }),
|
||||
grid.fps ? "MiniMax H3 always generates at " + grid.fps + " fps; the render is muxed at that rate." : null));
|
||||
wrap.append(out);
|
||||
|
||||
wrap.append(el("div", "sw-rows-label", "Prompt"));
|
||||
|
||||
+79
-15
@@ -1,8 +1,10 @@
|
||||
"""MiniMax-H3 adapter — the one place that knows how H3 differs from LTXAV.
|
||||
|
||||
FunPack was built against LTX2.3/LTXAV. H3 (Comfy-Org/ComfyUI#15224) is close
|
||||
enough that most of the stack works untouched, and different in four ways that
|
||||
break silently if nobody accounts for them:
|
||||
FunPack was built against LTX2.3/LTXAV, and supports BOTH families side by side:
|
||||
nothing here changes an LTX run, and every entry point answers "not H3" when the
|
||||
model in hand is an LTX one. H3 (merged as ComfyUI 57500fc5, PR #15224, shipped
|
||||
in ComfyUI v0.30.0) is close enough that most of the stack works untouched, and
|
||||
different in four ways that break silently if nobody accounts for them:
|
||||
|
||||
1. **Audio latent axis.** LTXAV's audio latent is ``[B, C, T, freq]`` — time on
|
||||
dim 2, the same axis as the video latent's frames, which is why the Chain
|
||||
@@ -26,8 +28,17 @@ break silently if nobody accounts for them:
|
||||
alone that makes every CFG generation crash; :func:`install_batch_split`
|
||||
splits the batch back apart around the model call.
|
||||
|
||||
Nothing here imports the H3 core modules at module scope — the PR is not in
|
||||
ComfyUI master yet, so every entry point degrades to "not H3" when it is absent.
|
||||
Nothing here imports the H3 core modules at module scope: an older ComfyUI (or an
|
||||
LTX-only install) has none of them, and every entry point has to degrade to "not
|
||||
H3" rather than raise on import.
|
||||
|
||||
WEIGHTS COME IN TWO CHECKPOINTS, and they are not interchangeable:
|
||||
``minimax_h3_fl2va_*`` is the t2v / first-last-frame model (keyframe pins) and
|
||||
``minimax_h3_ref2va_*`` is the reference model (reference blocks). Both load
|
||||
through the same detection and produce a structurally valid graph either way, so
|
||||
nothing crashes when the wrong one is loaded — it just conditions badly. The DiT
|
||||
state dict carries no variant marker, so :func:`checkpoint_mode_note` reports
|
||||
which mode a run is actually using and lets the user match the file.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -47,9 +58,17 @@ SPATIAL_DOWNSCALE = 16
|
||||
VIDEO_LATENT_CHANNELS = 24
|
||||
AUDIO_LATENT_CHANNELS = 32
|
||||
CANVAS_MULTIPLE = 32 # width/height must be a multiple of this
|
||||
REF_IMAGE_SHORT_EDGE = 2048 # the reference pipeline's own "max" sizing
|
||||
DEFAULT_SHIFT_VIDEO = 12.0
|
||||
DEFAULT_SHIFT_AUDIO = 3.0
|
||||
|
||||
# transformer_options keys MiniMaxH3SigmaShift sets. The DiT inverts the VIDEO shift to
|
||||
# its base grid to derive the audio timestep AND the audio velocity slope, so a schedule
|
||||
# built on a different shift has to say so here or the audio stream is integrated against
|
||||
# the wrong clock. Absent, the DiT falls back to its own 12.0 / 3.0.
|
||||
SHIFT_VIDEO_OPTION = "minimax_h3_sigma_shift_video"
|
||||
SHIFT_AUDIO_OPTION = "minimax_h3_sigma_shift_audio"
|
||||
|
||||
# Audio rows are [B, 32, stereo, T]: time is the LAST axis, not dim 2.
|
||||
AUDIO_TIME_DIM = -1
|
||||
VIDEO_TIME_DIM = 2
|
||||
@@ -277,11 +296,13 @@ def _split_cond_batch(value, index, batch):
|
||||
def install_batch_split(model):
|
||||
"""Run the H3 DiT one sample at a time when comfy hands it a batched forward.
|
||||
|
||||
``MiniMaxH3Model._forward`` raises on ``x.shape[0] != 1``. comfy batches the
|
||||
positive and negative conds into a single forward whenever ``cfg != 1.0``, so
|
||||
without this every CFG>1 generation dies on the first step. Splitting here
|
||||
costs exactly what CFG already costs (one forward per cond) and keeps the
|
||||
``cfg`` knob live instead of pinning it to 1.0.
|
||||
``MiniMaxH3Model._forward`` raises on ``x.shape[0] != 1``. In practice comfy does
|
||||
NOT batch the positive and negative conds together on H3 — each carries its own
|
||||
``minimax_payload`` (different token tags, different packed layout), and an unequal
|
||||
``CONDConstant`` blocks the concat — so the cfg>1 path survives on its own. What
|
||||
does reach the DiT batched is a latent whose batch size is >1 (several videos in one
|
||||
request). Splitting here keeps that working at exactly the cost the batch already
|
||||
implies, and stays correct if comfy's batching rules change.
|
||||
|
||||
Returns the previous ``model_function_wrapper`` so the caller can restore it.
|
||||
"""
|
||||
@@ -354,6 +375,29 @@ def set_refs(conditioning, ref_blocks):
|
||||
return _set_values(conditioning, {"minimax_refs": list(ref_blocks)})
|
||||
|
||||
|
||||
def checkpoint_mode_note(has_keyframes: bool, has_refs: bool):
|
||||
"""One line naming the checkpoint this run's conditioning actually needs, or None.
|
||||
|
||||
H3 ships as two separate DiTs — ``minimax_h3_fl2va_*`` (t2v / first-last frame) and
|
||||
``minimax_h3_ref2va_*`` (references). Both load through the same detection and neither
|
||||
rejects the other's conditioning, so the only failure signal is a bad generation. There
|
||||
is no variant marker in the state dict to check against, so the run says what it is
|
||||
doing and leaves matching the file to the user.
|
||||
"""
|
||||
if has_keyframes and has_refs:
|
||||
return ("this run uses BOTH keyframe pins and reference blocks, which live in "
|
||||
"DIFFERENT checkpoints (fl2va pins / ref2va references) — whichever model "
|
||||
"is loaded, one of the two is untrained conditioning. Drop the anchor image "
|
||||
"or the references, or run them as separate scenes.")
|
||||
if has_keyframes:
|
||||
return ("keyframe pin conditioning (fl2va) — this needs a minimax_h3_fl2va_* "
|
||||
"checkpoint; ref2va weights will read it, but were not trained on it.")
|
||||
if has_refs:
|
||||
return ("reference conditioning (ref2va) — this needs a minimax_h3_ref2va_* "
|
||||
"checkpoint; fl2va weights will read it, but were not trained on it.")
|
||||
return None
|
||||
|
||||
|
||||
def keyframe_indices_supported(frame_index, frame_count):
|
||||
"""H3's packed layout only places a pin at the FIRST or LAST pixel frame.
|
||||
|
||||
@@ -372,17 +416,28 @@ def encode_keyframe(vae, image, width, height, frame_index, crop="disabled"):
|
||||
canvas (it defines the geometry), a last-frame follower is cover-cropped.
|
||||
"""
|
||||
import comfy.utils
|
||||
samples = image[..., :3].movedim(-1, 1)
|
||||
# one frame only: a pin is a single condition frame, and an IMAGE batch here would
|
||||
# otherwise encode into a latent with a batch dimension the packed layout cannot place.
|
||||
samples = image[:1, ..., :3].movedim(-1, 1)
|
||||
samples = comfy.utils.common_upscale(samples, int(width), int(height), "lanczos", crop)
|
||||
pixels = samples.movedim(1, -1)
|
||||
return {"resolved_frame_index": int(frame_index), "latent": vae.encode(pixels)}
|
||||
|
||||
|
||||
def image_ref_block(vae, image, width, height):
|
||||
"""Encode a pixel image into a ref2va IMAGE block (identity / style reference)."""
|
||||
def image_ref_block(vae, image, width, height, size_mode="match"):
|
||||
"""Encode a pixel image into a ref2va IMAGE block (identity / style reference).
|
||||
|
||||
`size_mode` mirrors the reference node's own control: "match" scales the reference
|
||||
(down only, keeping aspect) to the generation's pixel area; "max" uses the reference
|
||||
pipeline's 2048px short edge for the best identity fidelity. Reference rows ride
|
||||
through every sampling step, so "max" costs real time on every step of every scene.
|
||||
"""
|
||||
import comfy.utils
|
||||
h, w = int(image.shape[1]), int(image.shape[2])
|
||||
scale = min(1.0, math.sqrt((int(width) * int(height)) / max(1, w * h)))
|
||||
if str(size_mode).lower() == "max":
|
||||
scale = min(1.0, REF_IMAGE_SHORT_EDGE / max(1, min(w, h)))
|
||||
else:
|
||||
scale = min(1.0, math.sqrt((int(width) * int(height)) / max(1, w * h)))
|
||||
snap = lambda v: max(CANVAS_MULTIPLE, int(round(v / CANVAS_MULTIPLE)) * CANVAS_MULTIPLE)
|
||||
tw, th = snap(w * scale), snap(h * scale)
|
||||
samples = image[:1, ..., :3].movedim(-1, 1)
|
||||
@@ -449,6 +504,9 @@ def normalize_ref_spec(raw):
|
||||
entry = {"kind": kind, "filename": filename}
|
||||
if kind == "video" and item.get("audio"):
|
||||
entry["audio"] = str(item["audio"]).strip()
|
||||
if kind == "image":
|
||||
size = str(item.get("size") or "match").strip().lower()
|
||||
entry["size"] = size if size in ("match", "max") else "match"
|
||||
out.append(entry)
|
||||
return out
|
||||
|
||||
@@ -702,12 +760,18 @@ def ref_blocks_from_spec(resolved, vae, width, height, audio_vae=None):
|
||||
`skipped` — the caller must say so out loud, because the presentation was already
|
||||
baked at encode time and every later "<Audio j>" in the prompt now points one
|
||||
reference earlier than the user wrote it.
|
||||
|
||||
Reference videos are capped by :func:`load_input_video`'s own `max_seconds`, NOT by
|
||||
the generation's length (which the reference node uses): the presentation is baked by
|
||||
Studio, which never sees a frame count, so a sampler-side cap would silently show Qwen
|
||||
more of the clip than the packed rows contain.
|
||||
"""
|
||||
blocks, skipped = [], []
|
||||
for entry in resolved:
|
||||
kind = entry["kind"]
|
||||
if kind == "image":
|
||||
_item, block = image_ref_block(vae, entry["image"], width, height)
|
||||
_item, block = image_ref_block(vae, entry["image"], width, height,
|
||||
size_mode=entry.get("size", "match"))
|
||||
blocks.append(block)
|
||||
elif kind == "audio":
|
||||
if audio_vae is None:
|
||||
|
||||
@@ -143,6 +143,35 @@ FAMILIES: dict[str, dict] = {
|
||||
}
|
||||
|
||||
|
||||
# Pixel-frame grid and generation frame rate per family. LTX: 8k+1 at the project's fps.
|
||||
# MiniMax H3: 17k+5 at a fixed 24 fps (both are properties of the model, not settings).
|
||||
# The frontends snap to the same numbers (PipelineCaps.frameGrid); this is the authority,
|
||||
# so a project built by an older UI — or by the API directly — still gets a valid graph.
|
||||
FAMILY_FRAME_GRID: dict[str, dict] = {
|
||||
"ltxav": {"step": 8, "base": 1, "fps": None},
|
||||
"minimax_h3": {"step": 17, "base": 5, "fps": 24},
|
||||
}
|
||||
|
||||
|
||||
def family_frames(family: str, frames) -> int:
|
||||
"""`frames` snapped UP to `family`'s pixel-frame grid (never below one whole step)."""
|
||||
grid = FAMILY_FRAME_GRID.get(family) or FAMILY_FRAME_GRID[DEFAULT_FAMILY]
|
||||
try:
|
||||
n = int(frames)
|
||||
except (TypeError, ValueError):
|
||||
return grid["step"] + grid["base"]
|
||||
n = max(grid["step"] + grid["base"], n)
|
||||
while (n - grid["base"]) % grid["step"] != 0:
|
||||
n += 1
|
||||
return n
|
||||
|
||||
|
||||
def family_frame_rate(family: str, frame_rate):
|
||||
"""The frame rate the graph must use — the model's own when it has a fixed one."""
|
||||
grid = FAMILY_FRAME_GRID.get(family) or FAMILY_FRAME_GRID[DEFAULT_FAMILY]
|
||||
return grid["fps"] if grid["fps"] is not None else frame_rate
|
||||
|
||||
|
||||
def family_of(models_config: Optional[dict]) -> str:
|
||||
"""Which model family this project's graph is built for.
|
||||
|
||||
@@ -357,9 +386,12 @@ def build(object_info: dict, models_config: dict, params: dict, media: dict | No
|
||||
if params.get("negative_prompt") is not None:
|
||||
graph["neg"]["inputs"]["value"] = params["negative_prompt"]
|
||||
if params.get("num_frames_per_scene") is not None:
|
||||
graph["frames"]["inputs"]["value"] = params["num_frames_per_scene"]
|
||||
graph["frames"]["inputs"]["value"] = family_frames(family, params["num_frames_per_scene"])
|
||||
if params.get("frame_rate") is not None:
|
||||
graph["fps"]["inputs"]["value"] = params["frame_rate"]
|
||||
# H3 generates at a fixed 24 fps. The project's rate still drives the container
|
||||
# the clip is muxed into, so anything else plays the generated frames at the
|
||||
# wrong speed — pin it here rather than render something that looks fast/slow.
|
||||
graph["fps"]["inputs"]["value"] = family_frame_rate(family, params["frame_rate"])
|
||||
if params.get("seed") is not None:
|
||||
graph["sampler"]["inputs"]["seed"] = params["seed"]
|
||||
if params.get("max_scenes") is not None:
|
||||
@@ -513,7 +545,15 @@ def build(object_info: dict, models_config: dict, params: dict, media: dict | No
|
||||
# An "editor"-sourced link pulls its value from a project setting (params) instead.
|
||||
for link in (models_config or {}).get("links") or []:
|
||||
if link.get("source") == "editor":
|
||||
val = params.get(link.get("editor_key"))
|
||||
key = link.get("editor_key")
|
||||
val = params.get(key)
|
||||
# A latent node driven by "Project · Frames" has to receive the SAME number the
|
||||
# sampler is given, or the two disagree about the scene length and the run dies
|
||||
# on the mismatch. Same for a fixed-rate family's fps.
|
||||
if val is not None and key == "num_frames_per_scene":
|
||||
val = family_frames(family, val)
|
||||
elif val is not None and key == "frame_rate":
|
||||
val = family_frame_rate(family, val)
|
||||
else:
|
||||
val = link.get("value")
|
||||
if val is None:
|
||||
|
||||
@@ -53,11 +53,9 @@ _ALL_MAPPED = frozenset().union(*(p["classes"] for p in PIPELINE_PACKS))
|
||||
|
||||
|
||||
# ── per-family setup: what a project needs before it can generate ─────────────
|
||||
# Declarative on purpose. MiniMax H3 is not released yet — its nodes are in an unmerged
|
||||
# ComfyUI PR and no weights exist — so the entries below are PLACEHOLDERS: they describe
|
||||
# what to look for and say plainly that it is not out. When it ships, the only thing that
|
||||
# changes here is the data (flip `released`, fill in the real filename hints); the setup
|
||||
# flow, the panel and the tests already work against it.
|
||||
# Declarative on purpose: both families are real pipelines here, and a project picks one.
|
||||
# MiniMax H3 shipped in ComfyUI v0.30.0 (PR #15224) with weights on Comfy-Org/MiniMax-H3,
|
||||
# so its entries below are the real files, not placeholders.
|
||||
#
|
||||
# `nodes` are slot node classes (not core graph nodes — those are covered by PIPELINE_PACKS).
|
||||
# `models` are files the user has to download; `folder` is the ComfyUI models/ subdirectory.
|
||||
@@ -76,34 +74,43 @@ FAMILY_SETUP: dict[str, dict[str, Any]] = {
|
||||
},
|
||||
"minimax_h3": {
|
||||
"label": "MiniMax H3 (Hailuo)",
|
||||
"released": False,
|
||||
"released": True,
|
||||
"summary": "MiniMax H3. Qwen3-VL text encoder, one joint AV latent, 24 fps, "
|
||||
"native reference conditioning (ref2va).",
|
||||
"not_released_note":
|
||||
"MiniMax H3 is not available yet. Its nodes are in an unmerged ComfyUI pull "
|
||||
"request and no weights have been published. You can select it now to set the "
|
||||
"project up — the pipeline is already wired for it, and it will start working "
|
||||
"as soon as ComfyUI ships the nodes and the weights appear in your models "
|
||||
"folders. Nothing here needs another FunPack update.",
|
||||
"source_url": "https://github.com/Comfy-Org/ComfyUI/pull/15224",
|
||||
"source_title": "ComfyUI PR #15224 — MiniMax H3 support",
|
||||
"native reference conditioning (ref2va). Needs ComfyUI v0.30.0 or newer.",
|
||||
"note":
|
||||
"MiniMax H3 ships as TWO diffusion checkpoints and they are not interchangeable: "
|
||||
"minimax_h3_fl2va_* is the text-to-video / first-last-frame model (it is the one "
|
||||
"that uses a scene's anchor image), minimax_h3_ref2va_* is the reference model "
|
||||
"(<Picture 1> / <Video 1> / <Audio 1> media). Load the one that matches how you "
|
||||
"generate — the other reads the same graph without complaining and just conditions "
|
||||
"badly. Everything else (text encoder, both VAEs) is shared.",
|
||||
"source_url": "https://huggingface.co/Comfy-Org/MiniMax-H3",
|
||||
"source_title": "Comfy-Org/MiniMax-H3 — repackaged weights",
|
||||
"nodes": [
|
||||
{"class": "EmptyMiniMaxH3LatentAV", "label": "Empty MiniMax H3 AV Latent",
|
||||
"role": "empty_latent",
|
||||
"why": "Makes the joint video + audio latent that feeds the Chain Sampler."},
|
||||
"why": "Makes the joint video + audio latent that feeds the Chain Sampler. Its "
|
||||
"`length` must match the project's frames per scene (both snap to 17k+5)."},
|
||||
{"class": "MiniMaxH3SigmaShift", "label": "MiniMax H3 Sigma Shift",
|
||||
"role": None, "optional": True,
|
||||
"why": "Optional — sets the video/audio flow shifts (defaults 12.0 / 3.0)."},
|
||||
"why": "Optional — sets the video/audio flow shifts (defaults 12.0 / 3.0). It also "
|
||||
"tells the DiT which shift the schedule uses, which is how the audio stream "
|
||||
"stays on its own clock."},
|
||||
],
|
||||
"models": [
|
||||
{"role": "unet", "label": "MiniMax H3 diffusion model", "folder": "diffusion_models",
|
||||
"hint": "single-stream packed AV DiT"},
|
||||
{"role": "unet", "label": "MiniMax H3 diffusion model (fl2va OR ref2va)",
|
||||
"folder": "diffusion_models",
|
||||
"hint": "minimax_h3_fl2va_pruned_int8_convrot.safetensors (~21 GB) for t2v/i2v, or "
|
||||
"minimax_h3_ref2va_pruned_int8_convrot.safetensors for reference media. "
|
||||
"_int8_convrot (~34 GB) and _bf16 (~66 GB) are the higher-precision cuts"},
|
||||
{"role": "clip", "label": "Qwen3-VL-32B text encoder (50 layers)", "folder": "text_encoders",
|
||||
"hint": "H3 consumes the unnormalized hidden state after layer 50"},
|
||||
"hint": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors (~16 GB); _int8_convrot and "
|
||||
"_bf16 also published. H3 consumes the unnormalized hidden state after layer 50"},
|
||||
{"role": "video_vae", "label": "MiniMax H3 video VAE", "folder": "vae",
|
||||
"hint": "24-channel latent, 16x spatial"},
|
||||
"hint": "minimax_h3_video_vae_fp16.safetensors — 24-channel latent, 16x spatial"},
|
||||
{"role": "audio_vae", "label": "MiniMax H3 audio VAE", "folder": "vae",
|
||||
"hint": "32 kHz stereo; also encodes audio references for ref2va"},
|
||||
"hint": "minimax_h3_audio_vae_fp32.safetensors — 32 kHz stereo; also encodes "
|
||||
"audio references for ref2va"},
|
||||
],
|
||||
},
|
||||
}
|
||||
@@ -121,7 +128,7 @@ def families_payload() -> list[dict]:
|
||||
return [
|
||||
{"key": key, "label": spec["label"], "released": bool(spec.get("released")),
|
||||
"summary": spec.get("summary", ""),
|
||||
"note": spec.get("not_released_note"),
|
||||
"note": spec.get("note"),
|
||||
"source_url": spec.get("source_url"), "source_title": spec.get("source_title")}
|
||||
for key, spec in FAMILY_SETUP.items()
|
||||
]
|
||||
@@ -144,7 +151,7 @@ def family_readiness(object_info: dict | None, family: str) -> dict:
|
||||
"label": spec["label"],
|
||||
"released": bool(spec.get("released")),
|
||||
"summary": spec.get("summary", ""),
|
||||
"note": spec.get("not_released_note"),
|
||||
"note": spec.get("note"),
|
||||
"source_url": spec.get("source_url"),
|
||||
"source_title": spec.get("source_title"),
|
||||
"nodes": nodes,
|
||||
@@ -223,8 +230,9 @@ def status_payload(
|
||||
unmapped = unmapped_missing_classes(object_info, family)
|
||||
on_disk = manager_dir_on_disk() if manager_on_disk is None else manager_on_disk
|
||||
readiness = family_readiness(object_info, family or DEFAULT_FAMILY)
|
||||
# A family whose nodes are not out yet is a setup problem too — the modal has to open
|
||||
# and say so, rather than reporting a clean pipeline that cannot actually generate.
|
||||
# A family whose own nodes are missing (an older ComfyUI, say) is a setup problem too —
|
||||
# the modal has to open and say so, rather than reporting a clean pipeline that cannot
|
||||
# actually generate.
|
||||
needs_setup = bool(packs) or bool(readiness["missing_nodes"])
|
||||
needs_manager = bool(packs) and not manager_available
|
||||
return {
|
||||
|
||||
@@ -106,8 +106,15 @@
|
||||
issues.push(...(window.PipelineCaps?.h3InertSettings(st) || []));
|
||||
issues.forEach((issue) => {
|
||||
const chip = el("button", "btn ghost compact action-warn", "⚠ " + issue.short);
|
||||
chip.title = issue.detail + "\n\nClick to open Engine settings.";
|
||||
chip.onclick = () => window.SettingsWindow?.open("engine");
|
||||
// most of these are engine settings; a few (frame rate) are project fields, and
|
||||
// sending the user to the Engine window for those would be a dead end
|
||||
const toProject = issue.target === "project";
|
||||
chip.title = issue.detail + (toProject
|
||||
? "\n\nClick to open the project settings."
|
||||
: "\n\nClick to open Engine settings.");
|
||||
chip.onclick = toProject
|
||||
? () => S.selectScene(null)
|
||||
: () => window.SettingsWindow?.open("engine");
|
||||
mount.append(chip);
|
||||
});
|
||||
}
|
||||
|
||||
@@ -738,6 +738,26 @@
|
||||
});
|
||||
row.append(picker);
|
||||
|
||||
// How large this reference is encoded. "Match" scales it to the generation's pixel
|
||||
// area; "Max" uses the reference pipeline's own 2048px short edge, which holds identity
|
||||
// best and costs real time on EVERY sampling step of every scene.
|
||||
if (ref.kind === "image") {
|
||||
const size = el("select", "sw-input");
|
||||
[["match", "Match output size"], ["max", "Max detail (2048px)"]].forEach(([v, label]) => {
|
||||
const o = new Option(label, v);
|
||||
if ((ref.size || "match") === v) o.selected = true;
|
||||
size.append(o);
|
||||
});
|
||||
size.title = "Max keeps more identity detail, but reference rows ride through every "
|
||||
+ "sampling step — it can be several times slower.";
|
||||
size.onchange = () => {
|
||||
const next = refs.slice();
|
||||
next[i] = { ...ref, size: size.value };
|
||||
patchH3Refs(next);
|
||||
};
|
||||
row.append(size);
|
||||
}
|
||||
|
||||
// a video's own soundtrack: presented as "<Audio j>" immediately before its
|
||||
// "<Video k>", which is what tells the model the two belong together
|
||||
if (ref.kind === "video") {
|
||||
|
||||
@@ -423,9 +423,30 @@
|
||||
name.oninput = () => S.patchProjectQuiet({ name: name.value });
|
||||
body.append(field("Project name", name));
|
||||
|
||||
// Frames snap to the model family's grid on blur (LTX 8k+1, MiniMax H3 17k+5). Typing is
|
||||
// left alone — snapping mid-keystroke fights the user — but the value that reaches the
|
||||
// project is always one the model can actually generate.
|
||||
const grid = window.PipelineCaps?.frameGrid ? window.PipelineCaps.frameGrid(st) : null;
|
||||
const row1 = el("div", "fields-row");
|
||||
row1.append(numberField("Frames / scene", p.num_frames_per_scene, (v) => S.patchProjectQuiet({ num_frames_per_scene: v }), "pj-frames"));
|
||||
row1.append(numberField("FPS", p.frame_rate, (v) => S.patchProjectQuiet({ frame_rate: v }), "pj-fps"));
|
||||
const framesField = numberField("Frames / scene", p.num_frames_per_scene,
|
||||
(v) => S.patchProjectQuiet({ num_frames_per_scene: v }), "pj-frames");
|
||||
const framesInput = framesField.querySelector("input");
|
||||
if (framesInput) {
|
||||
if (grid) framesInput.title = "Snaps to the model's " + grid.label + " frame grid.";
|
||||
framesInput.onchange = () => {
|
||||
const snapped = S.snapFrames(parseInt(framesInput.value || "0", 10));
|
||||
framesInput.value = snapped;
|
||||
S.patchProjectQuiet({ num_frames_per_scene: snapped });
|
||||
};
|
||||
}
|
||||
row1.append(framesField);
|
||||
const fpsField = numberField("FPS", p.frame_rate, (v) => S.patchProjectQuiet({ frame_rate: v }), "pj-fps");
|
||||
if (grid && grid.fps) {
|
||||
const fpsInput = fpsField.querySelector("input");
|
||||
if (fpsInput) fpsInput.title = "MiniMax H3 always generates at " + grid.fps
|
||||
+ " fps — the render is muxed at that rate whatever this says.";
|
||||
}
|
||||
row1.append(fpsField);
|
||||
body.append(row1);
|
||||
const row2 = el("div", "fields-row");
|
||||
row2.append(numberField("Width", p.width != null ? p.width : 768, (v) => S.patchProjectQuiet({ width: v }), "pj-w"));
|
||||
|
||||
@@ -2,8 +2,12 @@
|
||||
(function () {
|
||||
const CHAIN_ONLY = new Set(["carry", "mixed", "generated_frame", "v2v", "anchor_guide"]);
|
||||
|
||||
// The Editor keeps the live model config at state.models (synced into project.models on
|
||||
// commit); Easy Gen has no separate copy and keeps it on the project. Read both, or every
|
||||
// family-dependent answer here silently reads as LTX in Easy Gen.
|
||||
function models(st) {
|
||||
return (st && st.models) || {};
|
||||
if (!st) return {};
|
||||
return st.models || (st.project && st.project.models) || {};
|
||||
}
|
||||
|
||||
function usesFunpackStudio(st) {
|
||||
@@ -27,6 +31,49 @@
|
||||
return modelFamily(st) === "minimax_h3";
|
||||
}
|
||||
|
||||
// Frame geometry per family. LTX generates on an 8k+1 pixel-frame grid at whatever fps
|
||||
// the project says; MiniMax H3 generates on a 17k+5 grid and ALWAYS at 24 fps (the rate
|
||||
// is baked into the model, not a conditioning field). Both frontends snap through here so
|
||||
// a project can never ask for a length the chosen model cannot produce.
|
||||
const FRAME_GRIDS = {
|
||||
ltxav: { step: 8, base: 1, fps: null, label: "8k+1" },
|
||||
minimax_h3: { step: 17, base: 5, fps: 24, label: "17k+5" },
|
||||
};
|
||||
|
||||
function frameGrid(st) {
|
||||
return FRAME_GRIDS[modelFamily(st)] || FRAME_GRIDS.ltxav;
|
||||
}
|
||||
|
||||
// round | floor | ceil onto the family's grid. Always >= one whole grid step.
|
||||
function snapFramesTo(n, st, mode) {
|
||||
const g = frameGrid(st);
|
||||
const min = g.step + g.base;
|
||||
const k = (Math.round(Number(n) || 0) - g.base) / g.step;
|
||||
const rounded = mode === "floor" ? Math.floor(k) : mode === "ceil" ? Math.ceil(k) : Math.round(k);
|
||||
return Math.max(min, rounded * g.step + g.base);
|
||||
}
|
||||
|
||||
// H3 renders at a fixed 24 fps. The project's frame rate still drives the container the
|
||||
// clip is muxed into, so anything else plays the generated frames at the wrong speed —
|
||||
// the builder pins it, and this is what says so before the run.
|
||||
function frameRateIssue(st) {
|
||||
const p = st && st.project;
|
||||
const g = frameGrid(st);
|
||||
if (!p || !g.fps) return null;
|
||||
const fps = Number(p.frame_rate);
|
||||
if (!fps || Math.abs(fps - g.fps) < 0.01) return null;
|
||||
return {
|
||||
short: "H3: frame rate is fixed at " + g.fps,
|
||||
detail: "This project is set to " + fps + " fps, but MiniMax H3 always generates at "
|
||||
+ g.fps + " fps — the rate is part of the model, not a setting. The render is muxed at "
|
||||
+ g.fps + " fps regardless, so set FPS to " + g.fps + " in the project settings to keep "
|
||||
+ "the timeline honest.",
|
||||
// a project field, not an engine setting: the chip must not send the user to the
|
||||
// Engine window, where there is no frame rate to change
|
||||
target: "project",
|
||||
};
|
||||
}
|
||||
|
||||
function caps(st) {
|
||||
const m = models(st);
|
||||
return {
|
||||
@@ -36,6 +83,7 @@
|
||||
imported_workflow: !!m.workflow_import,
|
||||
family: modelFamily(st),
|
||||
h3: isH3(st),
|
||||
frame_grid: frameGrid(st),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -81,10 +129,13 @@
|
||||
if (si[key]) issues.push([key, H3_DEAD_SAMPLER_INPUTS[key]]);
|
||||
});
|
||||
issues.push(...h3DeadValueIssues(si));
|
||||
return issues.map(([key, detail]) => ({
|
||||
const out = issues.map(([key, detail]) => ({
|
||||
short: "H3: " + key.replace(/_enabled$/, "").replace(/_/g, " ") + " can't run",
|
||||
detail: detail + " Turn it off in Settings → Engine to stop it showing here.",
|
||||
}));
|
||||
const fps = frameRateIssue(st);
|
||||
if (fps) out.push(fps);
|
||||
return out;
|
||||
}
|
||||
|
||||
function effectiveSourceType(scene, st) {
|
||||
@@ -176,6 +227,9 @@
|
||||
caps,
|
||||
modelFamily,
|
||||
isH3,
|
||||
frameGrid,
|
||||
snapFramesTo,
|
||||
frameRateIssue,
|
||||
h3InertSettings,
|
||||
effectiveSourceType,
|
||||
isChainOnlySource,
|
||||
|
||||
@@ -8,6 +8,9 @@
|
||||
let overlay = null;
|
||||
let _pollTimer = null;
|
||||
let _activeJobId = null;
|
||||
// Set when picking a family moved the project onto that model's frame grid / frame rate,
|
||||
// so the modal can say what changed rather than leaving it to be noticed later.
|
||||
let geometryNote = "";
|
||||
|
||||
function dismissed() {
|
||||
try { return localStorage.getItem(LS_KEY) === "1"; } catch (_) { return false; }
|
||||
@@ -186,6 +189,28 @@
|
||||
models.model_family = key;
|
||||
await API.saveModels(S.get().project.id, models);
|
||||
await S.loadModels();
|
||||
// Frame geometry belongs to the model, not to taste: the new family may generate on a
|
||||
// different frame grid (LTX 8k+1 / H3 17k+5) and at a fixed rate (H3 is always 24 fps).
|
||||
// Bring the project onto them now, while the user is looking at the choice that caused
|
||||
// it — the alternative is a run that fails, or plays back at the wrong speed, later.
|
||||
const st = S.get();
|
||||
const grid = window.PipelineCaps?.frameGrid ? window.PipelineCaps.frameGrid(st) : null;
|
||||
if (grid && st.project) {
|
||||
const patch = {};
|
||||
const frames = Number(st.project.num_frames_per_scene);
|
||||
const snapped = window.PipelineCaps.snapFramesTo(frames, st, "round");
|
||||
if (frames !== snapped) patch.num_frames_per_scene = snapped;
|
||||
if (grid.fps && Number(st.project.frame_rate) !== grid.fps) patch.frame_rate = grid.fps;
|
||||
if (Object.keys(patch).length) {
|
||||
S.patchProject(patch);
|
||||
// Shown in the modal below; if setup is already complete and the modal closes, the
|
||||
// new values are visible in the inspector's Frames / FPS fields either way.
|
||||
geometryNote = "Project moved onto this model's frame geometry: "
|
||||
+ (patch.num_frames_per_scene || frames) + " frames per scene"
|
||||
+ (patch.frame_rate ? " at " + patch.frame_rate + " fps" : "")
|
||||
+ " (grid " + grid.label + ").";
|
||||
}
|
||||
}
|
||||
let deps;
|
||||
try { deps = await API.pipelineDeps(); } catch (_) { closeModal(); return; }
|
||||
if (!deps?.needs_setup) { closeModal(); return; }
|
||||
@@ -209,9 +234,9 @@
|
||||
return box;
|
||||
}
|
||||
|
||||
// What the chosen family still needs. For a family that is not out yet this is the
|
||||
// whole point of the panel: say what is missing and why, instead of offering an
|
||||
// install button that cannot possibly succeed.
|
||||
// What the chosen family still needs, plus anything about it a user cannot read off
|
||||
// their own graph — MiniMax H3's two interchangeable-looking diffusion checkpoints
|
||||
// above all. Shown whenever there is something to say, not only when it is missing.
|
||||
function readinessBlock(r) {
|
||||
const box = el("div", "pipe-setup-readiness");
|
||||
if (r.note) box.append(el("div", "pipe-setup-hint", r.note));
|
||||
@@ -246,6 +271,10 @@
|
||||
}
|
||||
|
||||
function openModal(deps) {
|
||||
// One-shot, and read AFTER closeModal(): reopening is how a family switch gets here, so
|
||||
// clearing it on close would drop the very message that switch produced.
|
||||
const geometryLine = geometryNote;
|
||||
geometryNote = "";
|
||||
closeModal();
|
||||
overlay = el("div", "modal-overlay pipe-setup-overlay");
|
||||
const modal = el("div", "modal pipe-setup-modal");
|
||||
@@ -255,9 +284,10 @@
|
||||
|
||||
const content = el("div", "modal-content");
|
||||
content.append(familyStep(deps));
|
||||
if (geometryLine) content.append(el("div", "pipe-setup-hint", geometryLine));
|
||||
const r = deps.readiness;
|
||||
const unreleased = r && !r.released;
|
||||
if (unreleased || (r && (r.missing_nodes || []).length)) {
|
||||
if (unreleased || (r && (r.note || (r.missing_nodes || []).length))) {
|
||||
content.append(readinessBlock(r));
|
||||
}
|
||||
// Only talk about installing when there is something Manager can actually install.
|
||||
|
||||
@@ -1685,17 +1685,20 @@
|
||||
return (p.overlay_tracks || []).some((ov) => (ov.duration_sec || 0) > 0);
|
||||
}
|
||||
|
||||
// LTX-valid frame counts are 8k+1 (so (frames-1) % 8 === 0); snap to the nearest.
|
||||
function snapFrames(n) { return Math.max(9, Math.round((Math.round(n) - 1) / 8) * 8 + 1); }
|
||||
function snapFramesFloor(n) {
|
||||
n = Math.round(n);
|
||||
if (n < 9) return 9;
|
||||
return Math.floor((n - 1) / 8) * 8 + 1;
|
||||
}
|
||||
function snapFramesCeil(n) {
|
||||
n = Math.round(n);
|
||||
return Math.max(9, Math.ceil((n - 1) / 8) * 8 + 1);
|
||||
// Valid frame counts are per model family — LTX 8k+1, MiniMax H3 17k+5 — so the grid comes
|
||||
// from PipelineCaps rather than being hardcoded here. Off-grid lengths do not fail softly:
|
||||
// the H3 latent node snaps its own length up while the sampler expects the number the
|
||||
// project asked for, and the run dies on the mismatch.
|
||||
function _snap(n, mode) {
|
||||
if (window.PipelineCaps?.snapFramesTo) return window.PipelineCaps.snapFramesTo(n, state, mode);
|
||||
n = Math.round(Number(n) || 0);
|
||||
const k = (n - 1) / 8;
|
||||
const r = mode === "floor" ? Math.floor(k) : mode === "ceil" ? Math.ceil(k) : Math.round(k);
|
||||
return Math.max(9, r * 8 + 1);
|
||||
}
|
||||
function snapFrames(n) { return _snap(n, "round"); }
|
||||
function snapFramesFloor(n) { return _snap(n, "floor"); }
|
||||
function snapFramesCeil(n) { return _snap(n, "ceil"); }
|
||||
|
||||
function _framesFromDuration(durationSec, fps, curFrames) {
|
||||
const raw = Math.max(1, durationSec) * fps;
|
||||
@@ -1705,7 +1708,9 @@
|
||||
else if (raw > cur + 0.01) frames = snapFramesCeil(raw);
|
||||
else frames = cur;
|
||||
if (frames === cur && Math.abs(durationSec - cur / fps) > 0.02) {
|
||||
frames = raw < cur ? Math.max(9, cur - 8) : cur + 8;
|
||||
// one whole grid step in the direction the drag went (8 frames on LTX, 17 on H3)
|
||||
const step = window.PipelineCaps?.frameGrid ? window.PipelineCaps.frameGrid(state).step : 8;
|
||||
frames = raw < cur ? snapFramesFloor(cur - step) : cur + step;
|
||||
}
|
||||
return frames;
|
||||
}
|
||||
|
||||
@@ -137,6 +137,8 @@
|
||||
h3V2a: st.project?.sampler_inputs?.v2a_grad_scale,
|
||||
h3Pass2: st.project?.sampler_inputs?.second_pass,
|
||||
h3Pass2Op: st.project?.sampler_inputs?.second_pass_op,
|
||||
// ... including the project's frame rate, which the H3 fixed-24-fps chip reads.
|
||||
h3Fps: st.project?.frame_rate,
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -507,6 +507,44 @@ def test_the_ltxav_graph_is_unchanged_by_the_family_split():
|
||||
assert not report["blocking"], report["blocking"]
|
||||
|
||||
|
||||
# ── frame geometry is a property of the model, not a setting ──────────────────
|
||||
# LTX: 8k+1 frames at the project's fps. H3: 17k+5 frames at a fixed 24 fps. An off-grid
|
||||
# length on H3 is not a rounding nuisance — the latent node snaps its own length up while
|
||||
# the sampler is told the raw number, and the run dies on the mismatch.
|
||||
|
||||
def test_h3_frame_counts_snap_to_the_17k_plus_5_grid():
|
||||
graph, _ = builder.build(H3_OI, H3_MODELS, {"prompt": "a shot", "num_frames_per_scene": 121})
|
||||
assert graph["frames"]["inputs"]["value"] == 124
|
||||
# a value already on the grid is left exactly as it is
|
||||
graph, _ = builder.build(H3_OI, H3_MODELS, {"prompt": "a shot", "num_frames_per_scene": 124})
|
||||
assert graph["frames"]["inputs"]["value"] == 124
|
||||
|
||||
|
||||
def test_h3_renders_at_the_models_own_frame_rate():
|
||||
graph, _ = builder.build(H3_OI, H3_MODELS, {"prompt": "a shot", "frame_rate": 30})
|
||||
assert graph["fps"]["inputs"]["value"] == 24
|
||||
|
||||
|
||||
def test_ltx_frame_geometry_is_untouched():
|
||||
graph, _ = builder.build(OI, {"slots": []},
|
||||
{"prompt": "a shot", "num_frames_per_scene": 121, "frame_rate": 30})
|
||||
assert graph["frames"]["inputs"]["value"] == 121 # already 8k+1
|
||||
assert graph["fps"]["inputs"]["value"] == 30 # LTX has no fixed rate
|
||||
graph, _ = builder.build(OI, {"slots": []}, {"prompt": "a shot", "num_frames_per_scene": 100})
|
||||
assert graph["frames"]["inputs"]["value"] == 105 # snapped up to 8k+1
|
||||
|
||||
|
||||
def test_a_latent_node_driven_by_project_frames_gets_the_same_snapped_number():
|
||||
"""A slot widget bound to "Project · Frames" must not receive the raw value while the
|
||||
sampler receives the snapped one — that disagreement is the mismatch itself."""
|
||||
models = dict(H3_MODELS)
|
||||
models["links"] = [{"id": "l1", "source": "editor", "editor_key": "num_frames_per_scene",
|
||||
"members": [{"slotId": "lat", "input": "length"}]}]
|
||||
graph, _ = builder.build(H3_OI, models, {"prompt": "a shot", "num_frames_per_scene": 121})
|
||||
assert graph["slot_lat"]["inputs"]["length"] == 124
|
||||
assert graph["frames"]["inputs"]["value"] == 124
|
||||
|
||||
|
||||
def test_an_unknown_family_falls_back_to_ltxav_rather_than_emitting_nothing():
|
||||
assert builder.family_of({"model_family": "hailuo-9000"}) == "ltxav"
|
||||
assert builder.family_of({}) == "ltxav"
|
||||
|
||||
@@ -82,47 +82,50 @@ def test_custom_nodes_dir_uses_folder_paths(tmp_path, monkeypatch):
|
||||
assert pipeline_deps.custom_nodes_dir() == cn
|
||||
|
||||
|
||||
# ── per-family setup placeholders ─────────────────────────────────────────────
|
||||
# MiniMax H3 is not released — no merged nodes, no published weights. The setup flow has
|
||||
# to say that plainly rather than either hiding the family or offering an install that
|
||||
# cannot succeed. These pin the shape the placeholders must keep, so shipping the real
|
||||
# thing is a data edit (flip `released`, fill the hints) and not a code change.
|
||||
# ── per-family setup ──────────────────────────────────────────────────────────
|
||||
# Both families are real, selectable pipelines. H3 shipped in ComfyUI v0.30.0 with weights
|
||||
# on Comfy-Org/MiniMax-H3, and it ships as TWO diffusion checkpoints (fl2va / ref2va) that
|
||||
# load interchangeably but condition differently — the setup panel is the only place that
|
||||
# can warn about that before a generation is spent, so these pin that it says so.
|
||||
|
||||
def test_both_families_are_offered_and_h3_is_marked_unreleased():
|
||||
def test_both_families_are_offered_and_released():
|
||||
from movie_editor.backend import pipeline_deps as pd
|
||||
fams = {f["key"]: f for f in pd.families_payload()}
|
||||
assert set(fams) == {"ltxav", "minimax_h3"}
|
||||
assert fams["ltxav"]["released"] is True
|
||||
assert fams["minimax_h3"]["released"] is False
|
||||
# a user picking it must be told why nothing generates yet, and where it comes from
|
||||
assert "not available yet" in fams["minimax_h3"]["note"]
|
||||
assert "15224" in fams["minimax_h3"]["source_url"]
|
||||
assert fams["minimax_h3"]["released"] is True
|
||||
# the two-checkpoint split is the one thing a user cannot discover from the graph
|
||||
note = fams["minimax_h3"]["note"]
|
||||
assert "fl2va" in note and "ref2va" in note
|
||||
assert "MiniMax-H3" in fams["minimax_h3"]["source_url"]
|
||||
assert fams["ltxav"]["note"] is None
|
||||
|
||||
|
||||
def test_h3_readiness_lists_the_nodes_and_model_files_it_is_waiting_on():
|
||||
def test_h3_readiness_lists_the_nodes_and_model_files_it_needs():
|
||||
from movie_editor.backend import pipeline_deps as pd
|
||||
r = pd.family_readiness({}, "minimax_h3")
|
||||
assert r["released"] is False
|
||||
assert r["released"] is True
|
||||
# the AV latent node is required; the sigma-shift node is optional and must not block
|
||||
missing = {n["class"] for n in r["missing_nodes"]}
|
||||
assert missing == {"EmptyMiniMaxH3LatentAV"}
|
||||
assert {m["role"] for m in r["models"]} == {"unet", "clip", "video_vae", "audio_vae"}
|
||||
assert all(m["folder"] for m in r["models"])
|
||||
# the hints name real files, including both diffusion variants
|
||||
unet_hint = next(m["hint"] for m in r["models"] if m["role"] == "unet")
|
||||
assert "minimax_h3_fl2va" in unet_hint and "minimax_h3_ref2va" in unet_hint
|
||||
|
||||
# once ComfyUI ships the node, it stops being listed as missing — no code change
|
||||
# on an older ComfyUI without the node, it is listed as missing — no code change
|
||||
r2 = pd.family_readiness({"EmptyMiniMaxH3LatentAV": {}}, "minimax_h3")
|
||||
assert r2["missing_nodes"] == []
|
||||
|
||||
|
||||
def test_an_unreleased_family_still_counts_as_needing_setup():
|
||||
def test_a_family_missing_its_own_nodes_still_counts_as_needing_setup():
|
||||
"""Otherwise the modal never opens and the project looks ready when it cannot generate."""
|
||||
from movie_editor.backend import pipeline_deps as pd
|
||||
full_oi = {cls: {} for cls in pd.required_core_classes("minimax_h3")}
|
||||
s = pd.status_payload(full_oi, manager_available=True, family="minimax_h3")
|
||||
assert s["missing_packs"] == [] # nothing for Manager to install
|
||||
assert s["needs_setup"] is True # ... but setup is not finished
|
||||
assert s["readiness"]["released"] is False
|
||||
assert s["needs_setup"] is True # ... but the H3 latent node is not installed
|
||||
assert s["family"] == "minimax_h3"
|
||||
|
||||
# with the H3 node present too, setup is genuinely done
|
||||
|
||||
+147
-7
@@ -2685,9 +2685,21 @@ class FunPackLTXAVSceneChainSampler:
|
||||
video_frames = self._tensor_frames(self._latent_tensors(latent_template)[0])
|
||||
expected = self._expected_latent_frames(num_frames_per_scene, time_scale, vae=vae)
|
||||
if video_frames != expected:
|
||||
detail = ""
|
||||
if self._is_h3:
|
||||
# By this point num_frames_per_scene is already snapped to 17k+5, so a
|
||||
# mismatch means the latent node was built for a different length entirely.
|
||||
try:
|
||||
from .minimax_h3 import FRAME_GRID, FRAME_BASE
|
||||
except ImportError:
|
||||
from minimax_h3 import FRAME_GRID, FRAME_BASE
|
||||
detail = (f" On MiniMax H3 the latent node's `length` must be the SAME "
|
||||
f"{FRAME_GRID}k+{FRAME_BASE} frame count as num_frames_per_scene "
|
||||
f"({num_frames_per_scene}) — set Empty MiniMax H3 AV Latent to that.")
|
||||
raise ValueError(
|
||||
f"latent_template has {video_frames} video latent frames, expected {expected} "
|
||||
f"from num_frames_per_scene={num_frames_per_scene} and time scale={time_scale}."
|
||||
+ detail
|
||||
)
|
||||
return video_frames
|
||||
|
||||
@@ -4089,6 +4101,64 @@ class FunPackLTXAVSceneChainSampler:
|
||||
_h3_frame_count = 0
|
||||
_is_h3 = False
|
||||
_h3_ref_cache: dict = {}
|
||||
_h3_mode_noted = False
|
||||
|
||||
def _h3_add_keyframes(self, conditioning, pins, frame_count, aug=None):
|
||||
"""Merge keyframe pins into a conditioning, keeping any already attached.
|
||||
|
||||
H3's payload takes a LIST of pins (a first-frame anchor and a last-frame target can
|
||||
coexist), so every producer here — the i2v anchor and the guide path — has to add to
|
||||
the list rather than overwrite it. Pins are keyed by resolved frame index; writing the
|
||||
same index twice replaces it, which is the only sane reading of two anchors on one
|
||||
frame. ``minimax_visual_cond_noise_aug`` stays payload-wide as the DiT defines it, so
|
||||
the last explicit strength written still wins — stated, not silently averaged.
|
||||
"""
|
||||
if not conditioning or not pins:
|
||||
return conditioning
|
||||
existing = self._conditioning_value(conditioning, "minimax_keyframes") or []
|
||||
merged = {int(p["resolved_frame_index"]): p for p in existing}
|
||||
for pin in pins:
|
||||
merged[int(pin["resolved_frame_index"])] = pin
|
||||
values = {
|
||||
"minimax_keyframes": [merged[k] for k in sorted(merged)],
|
||||
"minimax_frame_count": int(frame_count),
|
||||
}
|
||||
if aug is not None:
|
||||
values["minimax_visual_cond_noise_aug"] = float(aug)
|
||||
return self._condition_with_values(conditioning, values)
|
||||
|
||||
def _apply_h3_anchor(self, positive, chunk, vae, image, strength=1.0):
|
||||
"""Pin an i2v anchor image as H3's frame-0 keyframe (the fl2va conditioning path).
|
||||
|
||||
LTX anchors an image by writing it into the starting latent (LTXVImgToVideoInplace)
|
||||
and masking that frame out of denoising. H3 does not condition that way: the anchor
|
||||
is a CONDITION ROW packed beside the text, never denoised and never rendered, and a
|
||||
latent written frame-0 is just noise the model is free to overwrite. So on H3 the
|
||||
same user intent — "this scene starts from this image" — becomes a keyframe pin.
|
||||
|
||||
Returns (positive, applied). The image is encoded at the scene's own canvas, read
|
||||
off the chunk, so it matches whatever the latent template was built for.
|
||||
"""
|
||||
if image is None:
|
||||
return positive, False
|
||||
try:
|
||||
from . import minimax_h3 as h3mod
|
||||
except ImportError:
|
||||
import minimax_h3 as h3mod
|
||||
try:
|
||||
video = self._latent_tensors(chunk)[0]
|
||||
if video.dim() != 5:
|
||||
raise ValueError("video latent is not 5D")
|
||||
width = int(video.shape[4]) * h3mod.SPATIAL_DOWNSCALE
|
||||
height = int(video.shape[3]) * h3mod.SPATIAL_DOWNSCALE
|
||||
pin = h3mod.encode_keyframe(vae, image, width, height, 0, crop="disabled")
|
||||
except Exception as error:
|
||||
print(f"[FunPackSceneChain] H3: i2v anchor skipped — could not encode it ({error}).")
|
||||
return positive, False
|
||||
aug = max(0.0, min(1.0, float(strength)))
|
||||
positive = self._h3_add_keyframes(positive, [pin], self._h3_frame_count,
|
||||
aug=None if aug >= 1.0 else aug)
|
||||
return positive, True
|
||||
|
||||
def _append_h3_keyframe(self, guide_frame, apply_at, strength, positive, negative):
|
||||
"""H3's equivalent of an LTX guide: a keyframe pin carried on the conditioning.
|
||||
@@ -4114,19 +4184,39 @@ class FunPackLTXAVSceneChainSampler:
|
||||
f"image (ref2va) for mid-clip guidance instead.")
|
||||
return positive, negative, 0
|
||||
|
||||
keyframes = [{"resolved_frame_index": int(at), "latent": guide_frame}]
|
||||
values = {"minimax_keyframes": keyframes, "minimax_frame_count": int(frame_count)}
|
||||
pins = [{"resolved_frame_index": int(at), "latent": guide_frame}]
|
||||
# strength maps onto the DiT's condition noise augmentation: 1.0 pins the clean latent,
|
||||
# lower values mix noise into the condition rows. It is a single payload-wide value, so
|
||||
# with several pins in one scene the last one written wins — stated here rather than
|
||||
# silently averaged.
|
||||
aug = max(0.0, min(1.0, float(strength)))
|
||||
if aug < 1.0:
|
||||
values["minimax_visual_cond_noise_aug"] = aug
|
||||
positive = self._condition_with_values(positive, values)
|
||||
negative = self._condition_with_values(negative, values) if negative else negative
|
||||
aug = None if aug >= 1.0 else aug
|
||||
positive = self._h3_add_keyframes(positive, pins, frame_count, aug=aug)
|
||||
negative = self._h3_add_keyframes(negative, pins, frame_count, aug=aug) if negative else negative
|
||||
return positive, negative, 0
|
||||
|
||||
def _report_h3_checkpoint_mode(self, positive):
|
||||
"""Say once per run which H3 checkpoint this run's conditioning actually needs.
|
||||
|
||||
H3 ships two DiTs (fl2va for keyframe pins, ref2va for reference blocks) and neither
|
||||
rejects the other's conditioning — a mismatch costs quality, silently. Nothing in the
|
||||
state dict identifies the variant, so the run reports the mode it is in and leaves
|
||||
matching the file to the user.
|
||||
"""
|
||||
if self._h3_mode_noted:
|
||||
return
|
||||
try:
|
||||
from .minimax_h3 import checkpoint_mode_note
|
||||
except ImportError:
|
||||
from minimax_h3 import checkpoint_mode_note
|
||||
note = checkpoint_mode_note(
|
||||
bool(self._conditioning_value(positive, "minimax_keyframes")),
|
||||
bool(self._conditioning_value(positive, "minimax_refs")),
|
||||
)
|
||||
self._h3_mode_noted = True
|
||||
if note:
|
||||
print(f"[FunPackSceneChain] H3: {note}")
|
||||
|
||||
def _apply_h3_references(self, positive, chunk, vae, audio_vae=None):
|
||||
"""Turn Studio's resolved ref2va order into the DiT's `minimax_refs` blocks.
|
||||
|
||||
@@ -4385,7 +4475,12 @@ class FunPackLTXAVSceneChainSampler:
|
||||
video_overlap frames, protected) and the anchor image is then written on top — since
|
||||
LTXVImgToVideoInplace only overwrites the single encoded image frame's worth of samples
|
||||
and preserves the rest of the incoming noise_mask, the anchor's own leading frame stays a
|
||||
hard cut while the remaining carried frames keep the old scene's background/environment."""
|
||||
hard cut while the remaining carried frames keep the old scene's background/environment.
|
||||
|
||||
On MiniMax H3 the latent is left alone: the anchor is applied as a keyframe pin on the
|
||||
conditioning instead (_apply_h3_anchor), which is the only image conditioning that model
|
||||
was trained on. The carried-overlap behaviour is identical either way, so a mixed scene
|
||||
keeps its prior-scene context on both families."""
|
||||
filename = (anchor_meta or {}).get("filename")
|
||||
strength = float((anchor_meta or {}).get("strength", 1.0))
|
||||
image = self._load_image_tensor(filename) if filename else None
|
||||
@@ -4397,6 +4492,8 @@ class FunPackLTXAVSceneChainSampler:
|
||||
base = self._build_continuation_chunk(latent_template, previous, video_overlap)
|
||||
else:
|
||||
base = self._clone_latent(latent_template)
|
||||
if self._is_h3:
|
||||
return base
|
||||
return self._apply_img2video_to_video_latent(vae, image, base, strength)
|
||||
|
||||
def _identity_pin_filename(self, guide_list, scene_media_by_ref, identity_transfer_enabled):
|
||||
@@ -5297,10 +5394,28 @@ class FunPackLTXAVSceneChainSampler:
|
||||
# its time axis needs to know before it touches a tensor, because LTXAV and MiniMax
|
||||
# H3 disagree about which axis that is on the AUDIO stream and both are 4-D.
|
||||
self._is_h3 = self._set_stream_axes(model)
|
||||
if self._is_h3:
|
||||
# H3 only generates on a 17k+5 pixel-frame grid. Empty MiniMax H3 AV Latent snaps
|
||||
# its own `length` up silently, so an off-grid scene length produces a template
|
||||
# that is LONGER than the count asked for here and the length check below fails
|
||||
# with an arithmetic complaint that never mentions the grid. Snap to the same
|
||||
# number the latent node would, so the two always agree. LTX is untouched: its
|
||||
# grid is 8k+1 and the count comes from the VAE either way.
|
||||
try:
|
||||
from .minimax_h3 import align_frame_count
|
||||
except ImportError:
|
||||
from minimax_h3 import align_frame_count
|
||||
_aligned = align_frame_count(num_frames_per_scene)
|
||||
if _aligned != int(num_frames_per_scene):
|
||||
print(f"[FunPackSceneChain] H3: num_frames_per_scene {int(num_frames_per_scene)} "
|
||||
f"-> {_aligned} (the model's 17k+5 frame grid at 24 fps). Every scene is "
|
||||
f"that much longer than requested.")
|
||||
num_frames_per_scene = _aligned
|
||||
self._h3_frame_count = int(num_frames_per_scene)
|
||||
# Fresh per run: the encoded references must always trace back to the media that is
|
||||
# live in THIS request (see [[feedback_no_persistent_state_caches]]).
|
||||
self._h3_ref_cache = {}
|
||||
self._h3_mode_noted = False
|
||||
if self._is_h3:
|
||||
print("[FunPackSceneChain] MiniMax H3 detected — audio stream time axis is the last "
|
||||
"dim, frame grid is 17k+5, conditioning is a single packed self-attention "
|
||||
@@ -5666,6 +5781,30 @@ class FunPackLTXAVSceneChainSampler:
|
||||
elif not custom_guides:
|
||||
guide_tail = 0
|
||||
|
||||
# H3 i2v anchor (fl2va). On LTX the anchor is already IN the chunk — either written
|
||||
# by LTXVImgToVideoInplace above, or, for the opening scene, baked into the latent
|
||||
# template by whatever produced it. H3 has no latent i2v path at all, so the anchor
|
||||
# image becomes a frame-0 keyframe pin here, after the chunk is final (the pin is
|
||||
# encoded at this scene's canvas) and for every branch that carries an anchor.
|
||||
if self._is_h3:
|
||||
_anchor_image, _anchor_strength = None, 1.0
|
||||
_anchor_file = (anchor_meta or {}).get("filename")
|
||||
if _anchor_file:
|
||||
_anchor_image = self._load_image_tensor(_anchor_file)
|
||||
_anchor_strength = float(anchor_meta.get("strength", 1.0))
|
||||
elif output is None:
|
||||
# Studio owns source_image (it presents it to Qwen); it hands the pixels on
|
||||
# so this side can encode them. Opening scene only — later scenes continue
|
||||
# from the previous one unless they carry their own anchor.
|
||||
_studio_anchor = self._conditioning_value(scene_positive, "funpack_h3_anchor")
|
||||
if isinstance(_studio_anchor, dict):
|
||||
_anchor_image = _studio_anchor.get("image")
|
||||
if _anchor_image is not None:
|
||||
scene_positive, _pinned = self._apply_h3_anchor(
|
||||
scene_positive, chunk, vae, _anchor_image, _anchor_strength)
|
||||
if _pinned:
|
||||
run_mechanisms.append(f"h3_keyframe_anchor(strength={_anchor_strength:g})")
|
||||
|
||||
# ref2va reference blocks. Applied after the chunk is final because the blocks are
|
||||
# sized against this scene's canvas, and to every scene branch (fresh / anchored /
|
||||
# continuation) because a reference identity is meant to hold across the whole
|
||||
@@ -5675,6 +5814,7 @@ class FunPackLTXAVSceneChainSampler:
|
||||
scene_positive, chunk, vae, audio_vae=audio_vae)
|
||||
if h3_ref_count:
|
||||
run_mechanisms.append(f"h3_ref2va({h3_ref_count})")
|
||||
self._report_h3_checkpoint_mode(scene_positive)
|
||||
|
||||
# Everything from here through sampling installs per-scene state on the SHARED
|
||||
# model (function wrappers, forward hooks). One snapshot + one finally guarantees
|
||||
|
||||
@@ -365,6 +365,45 @@ def test_image_ref_blocks_snap_to_32_and_never_upscale():
|
||||
assert small["latent_h"] * 16 == 128
|
||||
|
||||
|
||||
def test_max_sizing_uses_the_reference_pipelines_2048_short_edge():
|
||||
"""The reference node's other sizing mode: identity fidelity over speed."""
|
||||
class FakeVAE:
|
||||
def encode(self, pixels):
|
||||
return torch.zeros(1, 24, 1, pixels.shape[1] // 16, pixels.shape[2] // 16)
|
||||
|
||||
big = torch.zeros(1, 3000, 4000, 3)
|
||||
_item, matched = h3.image_ref_block(FakeVAE(), big, 768, 768, size_mode="match")
|
||||
_item, maxed = h3.image_ref_block(FakeVAE(), big, 768, 768, size_mode="max")
|
||||
assert maxed["latent_h"] * 16 == h3.REF_IMAGE_SHORT_EDGE # short edge pinned to 2048
|
||||
assert maxed["latent_h"] > matched["latent_h"]
|
||||
# never upscaled, whatever the mode
|
||||
_item, small = h3.image_ref_block(FakeVAE(), torch.zeros(1, 128, 128, 3), 768, 768, size_mode="max")
|
||||
assert small["latent_h"] * 16 == 128
|
||||
|
||||
|
||||
def test_the_size_mode_survives_normalization_and_defaults_to_match():
|
||||
spec = h3.normalize_ref_spec([
|
||||
{"kind": "image", "filename": "a.png", "size": "max"},
|
||||
{"kind": "image", "filename": "b.png"},
|
||||
{"kind": "image", "filename": "c.png", "size": "nonsense"},
|
||||
{"kind": "audio", "filename": "v.wav"},
|
||||
])
|
||||
assert [e.get("size") for e in spec] == ["max", "match", "match", None]
|
||||
|
||||
|
||||
# ── the two checkpoints ──────────────────────────────────────────────────────
|
||||
# fl2va (pins) and ref2va (references) are separate weights that load identically and
|
||||
# never reject each other's conditioning. Nothing in the state dict says which is which,
|
||||
# so the only honest thing is to report the mode the run is in.
|
||||
|
||||
def test_checkpoint_note_names_the_variant_each_mode_needs():
|
||||
assert "fl2va" in h3.checkpoint_mode_note(True, False)
|
||||
assert "ref2va" in h3.checkpoint_mode_note(False, True)
|
||||
assert h3.checkpoint_mode_note(False, False) is None
|
||||
both = h3.checkpoint_mode_note(True, True)
|
||||
assert "BOTH" in both and "fl2va" in both and "ref2va" in both
|
||||
|
||||
|
||||
# ── 6. video references ──────────────────────────────────────────────────────
|
||||
# A reference video enters the model as TWO things: frames at 2 fps with timestamps for
|
||||
# Qwen, and a full-rate latent block for the DiT. Both come from one decode, so the clip
|
||||
|
||||
@@ -382,3 +382,100 @@ def test_references_are_encoded_once_per_run_not_once_per_scene(fake_media):
|
||||
smaller = {"samples": NT([torch.zeros(1, 24, 37, 32, 32), torch.zeros(1, 32, 2, 207)])}
|
||||
node._apply_h3_references(positive, smaller, vae)
|
||||
assert len(encodes) == 2
|
||||
|
||||
|
||||
# ── i2v anchors become keyframe pins ─────────────────────────────────────────
|
||||
# LTX anchors an image by writing it into the starting latent (LTXVImgToVideoInplace) and
|
||||
# masking that frame out. H3 has no latent i2v path at all — an anchor written into the
|
||||
# latent is just noise the model may overwrite — so the same intent has to become a
|
||||
# frame-0 keyframe pin. Both families must keep their own path.
|
||||
|
||||
def test_an_anchor_image_becomes_a_frame_zero_keyframe_pin():
|
||||
node = h3_node(frame_count=124)
|
||||
positive = [[torch.zeros(1, 12, 5120), {"funpack_scene_text": "shot 1"}]]
|
||||
out, applied = node._apply_h3_anchor(
|
||||
positive, av_latent(), RefVAE(), torch.zeros(1, 256, 256, 3), strength=1.0)
|
||||
|
||||
assert applied is True
|
||||
meta = out[0][1]
|
||||
assert [kf["resolved_frame_index"] for kf in meta["minimax_keyframes"]] == [0]
|
||||
assert meta["minimax_frame_count"] == 124
|
||||
assert "minimax_visual_cond_noise_aug" not in meta # full-strength pin stays clean
|
||||
assert meta["funpack_scene_text"] == "shot 1"
|
||||
|
||||
|
||||
def test_a_weakened_anchor_maps_onto_condition_noise_augmentation():
|
||||
node = h3_node()
|
||||
out, applied = node._apply_h3_anchor(
|
||||
[[torch.zeros(1, 12, 5120), {}]], av_latent(), RefVAE(),
|
||||
torch.zeros(1, 256, 256, 3), strength=0.6)
|
||||
assert applied is True
|
||||
assert out[0][1]["minimax_visual_cond_noise_aug"] == pytest.approx(0.6)
|
||||
|
||||
|
||||
def test_an_anchor_and_a_last_frame_guide_coexist_instead_of_overwriting():
|
||||
"""H3's payload takes a LIST of pins — the anchor must not erase the guide, or vice
|
||||
versa. This is exactly the fl2va first-AND-last-frame case."""
|
||||
node = h3_node(frame_count=124)
|
||||
positive = [[torch.zeros(1, 12, 5120), {}]]
|
||||
positive, _neg, _tail = node._append_h3_keyframe(
|
||||
torch.ones(1, 24, 1, 48, 84), apply_at=-1, strength=1.0,
|
||||
positive=positive, negative=[])
|
||||
positive, applied = node._apply_h3_anchor(
|
||||
positive, av_latent(), RefVAE(), torch.zeros(1, 256, 256, 3))
|
||||
|
||||
assert applied is True
|
||||
assert [kf["resolved_frame_index"] for kf in positive[0][1]["minimax_keyframes"]] == [0, 123]
|
||||
|
||||
|
||||
def test_a_mixed_anchor_chunk_leaves_the_h3_latent_alone():
|
||||
"""On LTX the anchor is written into the chunk; on H3 the chunk must come back
|
||||
untouched (the pin does the work) — otherwise the latent carries an anchor the model
|
||||
was never trained to read, on top of the pin."""
|
||||
node = h3_node()
|
||||
template = av_latent()
|
||||
calls = []
|
||||
node._apply_img2video_to_video_latent = lambda *a, **k: calls.append(a) or template
|
||||
node._load_image_tensor = lambda f: torch.zeros(1, 256, 256, 3)
|
||||
|
||||
chunk = node._build_mixed_anchor_chunk(
|
||||
RefVAE(), {"filename": "anchor.png", "strength": 1.0}, template, None, 0)
|
||||
assert calls == [] # the LTX node is never invoked on H3
|
||||
assert chunk["samples"].unbind()[0].shape == template["samples"].unbind()[0].shape
|
||||
|
||||
|
||||
def test_ltx_anchors_still_go_through_img2video_inplace():
|
||||
node = FunPackLTXAVSceneChainSampler()
|
||||
assert node._is_h3 is False
|
||||
template = {"samples": torch.zeros(1, 128, 16, 4, 4)}
|
||||
calls = []
|
||||
node._load_image_tensor = lambda f: torch.zeros(1, 256, 256, 3)
|
||||
node._apply_img2video_to_video_latent = lambda vae, image, base, strength: (
|
||||
calls.append(strength) or base)
|
||||
node._build_mixed_anchor_chunk(
|
||||
LTXVAE(), {"filename": "anchor.png", "strength": 0.9}, template, None, 0)
|
||||
assert calls == [0.9]
|
||||
|
||||
|
||||
# ── the two checkpoints ──────────────────────────────────────────────────────
|
||||
|
||||
def test_the_run_names_the_checkpoint_its_conditioning_needs(capsys):
|
||||
node = h3_node()
|
||||
node._h3_mode_noted = False
|
||||
node._report_h3_checkpoint_mode([[torch.zeros(1, 12, 5120), {"minimax_refs": [{"kind": "image"}]}]])
|
||||
out = capsys.readouterr().out
|
||||
assert "ref2va" in out
|
||||
# once per run, not once per scene
|
||||
node._report_h3_checkpoint_mode([[torch.zeros(1, 12, 5120), {"minimax_refs": [{"kind": "image"}]}]])
|
||||
assert capsys.readouterr().out == ""
|
||||
|
||||
|
||||
def test_mixing_pins_and_references_is_called_out(capsys):
|
||||
node = h3_node()
|
||||
node._h3_mode_noted = False
|
||||
node._report_h3_checkpoint_mode([[torch.zeros(1, 12, 5120), {
|
||||
"minimax_keyframes": [{"resolved_frame_index": 0}],
|
||||
"minimax_refs": [{"kind": "image"}],
|
||||
}]])
|
||||
out = capsys.readouterr().out
|
||||
assert "BOTH" in out and "fl2va" in out and "ref2va" in out
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
"""Studio's half of the MiniMax H3 split, and the LTX path it must not disturb.
|
||||
|
||||
Studio owns the CLIP; the Chain Sampler owns the VAE. Two things therefore have to travel
|
||||
between them on the conditioning, and both are silent when they go missing:
|
||||
|
||||
1. The i2v anchor image. H3 has no latent i2v path — the anchor is a frame-0 keyframe pin,
|
||||
which needs the VAE — so Studio presents the image to Qwen and hands the PIXELS on. On
|
||||
LTX nothing of the sort happens: the anchor is written into the latent by the graph.
|
||||
2. The resolved reference list, with every field that shaped the presentation. A video's
|
||||
soundtrack especially: Studio already emitted its "<Audio j>" label, so a sampler that
|
||||
never hears about the track packs no audio rows and every later audio ordinal shifts.
|
||||
"""
|
||||
import sys
|
||||
import types
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
sys.modules.setdefault("folder_paths", types.SimpleNamespace(models_dir=""))
|
||||
|
||||
from conditioning import FunPackVideoRefinerV2 # noqa: E402
|
||||
|
||||
|
||||
class H3Clip:
|
||||
"""What comfy builds for H3: an SD1Tokenizer named after the Qwen3-VL TE."""
|
||||
|
||||
def __init__(self):
|
||||
self.tokenizer = types.SimpleNamespace(qwen3vl_32b=object())
|
||||
self.tokenize_kwargs = []
|
||||
|
||||
def tokenize(self, text, **kwargs):
|
||||
self.tokenize_kwargs.append(kwargs)
|
||||
return text
|
||||
|
||||
def encode_from_tokens_scheduled(self, tokens):
|
||||
return [(torch.ones(1, 12, 5120), {"pooled_output": None})]
|
||||
|
||||
|
||||
class LTXClip:
|
||||
def __init__(self):
|
||||
self.tokenizer = types.SimpleNamespace(gemma3=object())
|
||||
|
||||
def tokenize(self, text, **kwargs):
|
||||
return text
|
||||
|
||||
def encode_from_tokens_scheduled(self, tokens):
|
||||
return [(torch.ones(1, 12, 4096), {"pooled_output": None})]
|
||||
|
||||
|
||||
IMAGE = torch.zeros(1, 64, 64, 3)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def h3_media(monkeypatch):
|
||||
import minimax_h3 as h3mod
|
||||
monkeypatch.setattr(h3mod, "load_input_image", lambda f: torch.zeros(1, 64, 64, 3))
|
||||
monkeypatch.setattr(h3mod, "load_input_audio",
|
||||
lambda f: {"waveform": torch.zeros(1, 2, 800), "sample_rate": 32000})
|
||||
monkeypatch.setattr(h3mod, "load_input_video", lambda f, **k: torch.zeros(39, 64, 64, 3))
|
||||
|
||||
|
||||
def test_the_anchor_image_is_handed_to_the_sampler_as_pixels():
|
||||
node = FunPackVideoRefinerV2()
|
||||
clip = H3Clip()
|
||||
_cond, meta, _status = node._v2_encode_prompt(clip, "a shot", reference_image=IMAGE)
|
||||
|
||||
# presented to Qwen as <Picture 1> ...
|
||||
assert clip.tokenize_kwargs[0]["images"] == [IMAGE]
|
||||
# ... and handed on, because pinning it needs the VAE the sampler owns
|
||||
assert meta["funpack_h3_anchor"]["image"] is IMAGE
|
||||
|
||||
|
||||
def test_an_ltx_anchor_is_not_carried_on_the_conditioning():
|
||||
"""LTX writes the anchor into the latent in the graph. Carrying it here too would put a
|
||||
second, competing anchor mechanism on a family that already has one."""
|
||||
node = FunPackVideoRefinerV2()
|
||||
node._gemma3_has_vision = lambda clip: True
|
||||
_cond, meta, _status = node._v2_encode_prompt(LTXClip(), "a shot", reference_image=IMAGE)
|
||||
assert "funpack_h3_anchor" not in meta
|
||||
|
||||
|
||||
def test_references_take_over_from_the_anchor_and_say_so(h3_media, capsys):
|
||||
"""ref2va and fl2va are separate conditioning modes — an anchor image cannot ride along
|
||||
with reference media, and the user has to be told which one won."""
|
||||
node = FunPackVideoRefinerV2()
|
||||
_cond, meta, _status = node._v2_encode_prompt(
|
||||
H3Clip(), "a shot", reference_image=IMAGE,
|
||||
h3_references=[{"kind": "image", "filename": "face.png"}])
|
||||
|
||||
assert "funpack_h3_anchor" not in meta
|
||||
assert [r["filename"] for r in meta["funpack_h3_refs"]] == ["face.png"]
|
||||
assert "source_image is NOT" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_a_video_references_soundtrack_reaches_the_sampler(h3_media):
|
||||
"""Studio emits "<Audio 1>" for the track it resolved. If the sampler never learns the
|
||||
track exists it packs no audio rows for it, and every later <Audio j> points one
|
||||
reference earlier than the prompt says."""
|
||||
node = FunPackVideoRefinerV2()
|
||||
_cond, meta, _status = node._v2_encode_prompt(
|
||||
H3Clip(), "a shot",
|
||||
h3_references=[{"kind": "video", "filename": "clip.mp4", "audio": "clip.wav"}])
|
||||
|
||||
assert meta["funpack_h3_refs"] == [{"kind": "video", "filename": "clip.mp4",
|
||||
"audio": "clip.wav"}]
|
||||
|
||||
|
||||
def test_the_reference_size_mode_reaches_the_sampler(h3_media):
|
||||
node = FunPackVideoRefinerV2()
|
||||
_cond, meta, _status = node._v2_encode_prompt(
|
||||
H3Clip(), "a shot",
|
||||
h3_references=[{"kind": "image", "filename": "face.png", "size": "max"}])
|
||||
assert meta["funpack_h3_refs"][0]["size"] == "max"
|
||||
Reference in New Issue
Block a user