Add experimental H3 native temporal node and Planner conditioning mode (1.115.9)
This commit is contained in:
alisson-anjos
@@ -0,0 +1,53 @@
|
||||
# Experimental H3 native temporal alignment
|
||||
|
||||
Matches ai-toolkit branch [`feat/h3-native-temporal-motion`](https://github.com/alisson-anjos/ai-toolkit/tree/feat/h3-native-temporal-motion), implementation `native_temporal_v1` (commit e0ecf3a). This mode retains a native video reference, aligns its temporal RoPE coordinates with the target, and softly penalizes target queries attending to distant guide times. It preserves native image-reference coordinates and native spatial geometry. It does not add learned weights or load a LoRA.
|
||||
|
||||
Use the matching native-temporal LoRA when available. Existing Add Guide motion-transfer LoRAs were trained under a different conditioning contract. Quality is experimental; CPU math/routing tests are not a full video validation.
|
||||
|
||||
## Standalone native workflow
|
||||
|
||||
1. Load the native MiniMax H3 base and the native-temporal LoRA.
|
||||
2. Pass the guide as `<Video 1>` to **MiniMax H3 Reference to Video**, alongside any number of identity reference pictures supported by that native node. Picture and video numbering are independent.
|
||||
3. Place **BFS MiniMax H3 Native Temporal Alignment (Experimental)** on the MODEL path, after LoRA loading. Route its MODEL output to your guider/sampler. Keep Reference to Video's positive and AV latent outputs connected normally.
|
||||
4. Select `motion_video_index = 1` (or the actual `<Video N>` supplying motion).
|
||||
5. Use the same 24 fps frame window for guide and output. H3 may trim frame counts to its supported cadence; use a matching valid count such as 107.
|
||||
|
||||
Do not add Add Guide, first-frame guides, VSA, or other attention/block replacements. The node raises a clear error for incompatible layouts and temporal lengths instead of silently stretching motion. Source video audio is supported by native packing but is not temporally biased by this patch. For CFG paths, every conditioning branch evaluated by the patched model must retain the native motion reference; an empty negative branch without references cannot provide its timeline. Training preview uses CFG 1.
|
||||
|
||||
## Planner
|
||||
|
||||
In **BFS Shot H3 Conditioning** (Advanced or Simple), choose:
|
||||
|
||||
```text
|
||||
guide_mode = native temporal alignment (experimental)
|
||||
```
|
||||
|
||||
Connect MODEL from your LoRA loader to the conditioning node, then connect the conditioning node's MODEL output to the guider. Advanced keeps its existing output order. Simple appends MODEL after positive/latent/prompt; old links keep their indices. Reload older nodes or recreate them if the new socket does not appear.
|
||||
|
||||
The shot is encoded as native `<Video 1>`, with identity pictures unchanged. The patch is applied separately to the returned model for each shot, and the current shot's packed temporal grid is used at sampling. Use 24 fps, first_frame none, mask_guide off and continuity off. Full-frame or ordinary native generation masks may still be used; masks do not create a temporal guide. Colored source overlays are retained if enabled, but color/mesh datasets have not yet been validated with this fresh experiment.
|
||||
|
||||
The Planner option uses the training defaults. To tune the policy manually, instead choose the ordinary `native reference video (Video 1)` conditioning mode and place the standalone node on the model path. Never install both patches on the same model path.
|
||||
|
||||
## Exact policy
|
||||
|
||||
Only target-video query to selected-motion-video key edges receive:
|
||||
|
||||
```text
|
||||
score -= min(strength * (max(abs(t_target - t_guide) - window_seconds, 0) / falloff_seconds)^2, max_penalty)
|
||||
```
|
||||
|
||||
Defaults: strength 2.0, window 0.20 s, falloff 0.25 s, maximum penalty 6.0. Set strength 0 to test temporal RoPE alignment alone. Bypass the node to test fully native baseline. Text, image references, target self-attention, reference audio, and token-refiner attention retain their normal behavior.
|
||||
|
||||
CUDA uses PyTorch compiled FlexAttention without allocating an S×S bias matrix. Requires a PyTorch build with FlexAttention and a compatible native ComfyUI H3 PackedLayout/attention override API. The tiny dense CPU implementation is a test oracle only and refuses long sequences. Start with model compilation/malloc graph optimizations disabled until validated on your local ComfyUI build.
|
||||
|
||||
Runtime confirmation:
|
||||
|
||||
```text
|
||||
[BFS H3 native temporal] native_temporal_v1: <Video 1>, ... motion rows; temporal RoPE aligned; soft window=0.2s strength=2.0 cap=6.0; images remain native; Add Guide OFF
|
||||
```
|
||||
|
||||
This log confirms the patch ran, not that visual motion or identity quality is correct.
|
||||
|
||||
## Validation
|
||||
|
||||
Tests compare an audited native ComfyUI layout with positions exported by the actual training implementation, including image references and reference audio. They verify original conditioning/layout immutability, temporal-length rejection, attention score math/output shapes, token-refiner fallback, per-model routing and incompatible patches. Planner tests verify native-video conditioning and returned MODEL routing. Full ComfyUI GPU generation remains pending local validation; the active training process is not restarted or modified by this source change.
|
||||
@@ -195,3 +195,5 @@ See [setup, workflow template, compatibility and validation](MINIMAX_H3_GUIDES.m
|
||||
|
||||
BFSNodes **1.47.0** adds optional MiniMax-H3 overlap/sidecar source-phase RoPE and
|
||||
**MiniMax-H3 Identity Reference + RoPE (BFS)**. See [the H3 guide](MINIMAX_H3_GUIDES.md#experimental-reference-layouts-and-source-phase-1470).
|
||||
|
||||
Experimental native temporal motion-transfer inference (standalone node and Shot Conditioning mode): see [H3_NATIVE_TEMPORAL.md](H3_NATIVE_TEMPORAL.md). Requires the matching experimental LoRA; preserves native image references and applies temporal RoPE alignment plus soft temporal attention.
|
||||
|
||||
@@ -92,6 +92,12 @@ try:
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] BFS Shot Noise node not loaded: {_e!r}")
|
||||
SNOISE_NODE_CLASS_MAPPINGS, SNOISE_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
try:
|
||||
from .minimax_h3_native_temporal import NODE_CLASS_MAPPINGS as H3TEMP_NODE_CLASS_MAPPINGS
|
||||
from .minimax_h3_native_temporal import NODE_DISPLAY_NAME_MAPPINGS as H3TEMP_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e:
|
||||
print(f"[BFSNodes] H3 native temporal nodes not loaded: {_e!r}")
|
||||
H3TEMP_NODE_CLASS_MAPPINGS, H3TEMP_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
try:
|
||||
from .minimax_h3_downscaled_guide import NODE_CLASS_MAPPINGS as H3GUIDE_NODE_CLASS_MAPPINGS
|
||||
from .minimax_h3_downscaled_guide import NODE_DISPLAY_NAME_MAPPINGS as H3GUIDE_NODE_DISPLAY_NAME_MAPPINGS
|
||||
@@ -129,6 +135,7 @@ except Exception as _e: # noqa
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
**H3CLIP_NODE_CLASS_MAPPINGS,
|
||||
**H3GUIDE_NODE_CLASS_MAPPINGS,
|
||||
**H3TEMP_NODE_CLASS_MAPPINGS,
|
||||
**SHOT_NODE_CLASS_MAPPINGS,
|
||||
**PANEL_NODE_CLASS_MAPPINGS,
|
||||
**GV_NODE_CLASS_MAPPINGS,
|
||||
@@ -159,6 +166,7 @@ NODE_CLASS_MAPPINGS = {
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
**H3CLIP_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**H3GUIDE_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**H3TEMP_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**SHOT_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**PANEL_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**GV_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
|
||||
+31
-11
@@ -4217,7 +4217,7 @@ def inpaint_latent(latent: dict, positive, vae, frames: torch.Tensor, mask: torc
|
||||
class BFSShotH3Conditioning:
|
||||
"""Native MiniMax H3 conditioning for one shot: references, prompt and the shot as a guide."""
|
||||
|
||||
GUIDE_MODES = ["aligned guide (Add Guide)", "native reference video (Video 1)", "both", "none", "aligned guide + CLIP video context"]
|
||||
GUIDE_MODES = ["aligned guide (Add Guide)", "native reference video (Video 1)", "both", "none", "aligned guide + CLIP video context", "native temporal alignment (experimental)"]
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
@@ -4230,7 +4230,8 @@ class BFSShotH3Conditioning:
|
||||
"aligned guide: the shot's frames sit on the generated frames (MiniMax H3 Add Guide), "
|
||||
"frame by frame. native reference video: the shot enters as <Video 1> like the reference "
|
||||
"video input. both: the two together. aligned guide + CLIP video context: Qwen sees "
|
||||
"the shot at clip_video_fps, while its latents enter only as an aligned guide."}),
|
||||
"the shot at clip_video_fps, while its latents enter only as an aligned guide. Native temporal alignment: "
|
||||
"experimental mode matching native_temporal_v1 training; requires MODEL routing, 24 fps and no Add Guide/VSA."}),
|
||||
"first_frame": (["none", "shot's first frame"], {"default": "none", "tooltip":
|
||||
"Anchor the shot's own first frame as an extra aligned image guide. Leave this at none when "
|
||||
"replacing the source location: its first frame also anchors the original background."}),
|
||||
@@ -4240,7 +4241,7 @@ class BFSShotH3Conditioning:
|
||||
"audio_vae": ("VAE",),
|
||||
"with_audio": ("BOOLEAN", {"default": False, "tooltip":
|
||||
"Attach the shot's soundtrack to the guide / reference video (needs audio_vae)."}),
|
||||
"model": ("MODEL", {"tooltip": "Optional: passed through to the model output unchanged (workflows "
|
||||
"model": ("MODEL", {"tooltip": "Optional: passed through, or patched in native temporal mode. Route MODEL output to the guider (workflows "
|
||||
"that routed the model through this node keep working)."}),
|
||||
"vlm": ("CLIP", {"tooltip": "Optional VLM (CLIPLoader with a Qwen3-VL text encoder). Fills {details} "
|
||||
"when the planner left it (no 📝 Describe refs, no VLM on the planner)."}),
|
||||
@@ -4335,7 +4336,17 @@ class BFSShotH3Conditioning:
|
||||
if prev is not None and shot.get("chain") == "reference":
|
||||
refs[f"ref_image_{len(refs)}"] = prev # one more <Picture n>, after the shot's own
|
||||
audio = shot["audio"] if (with_audio and audio_vae is not None) else None
|
||||
native = guide_mode in (self.GUIDE_MODES[1], self.GUIDE_MODES[2])
|
||||
temporal = guide_mode == self.GUIDE_MODES[5]
|
||||
if temporal:
|
||||
if model is None:
|
||||
raise ValueError("Native temporal alignment needs MODEL connected; route this node's MODEL output to the guider.")
|
||||
if first_frame != "none" or shot.get("chain") not in (None, "off") or shot.get("head") or shot.get("history"):
|
||||
raise ValueError("Native temporal mode requires first-frame and shot continuity guides disabled.")
|
||||
if mask_guide != "off":
|
||||
raise ValueError("Native temporal mode requires mask_guide off; use only the shot as native motion video.")
|
||||
if abs(float(shot["fps"])-24.0) > 1e-6:
|
||||
raise ValueError("Native temporal experiment requires 24 fps shots. Set the Planner output to 24 fps.")
|
||||
native = guide_mode in (self.GUIDE_MODES[1], self.GUIDE_MODES[2]) or temporal
|
||||
clip_context = guide_mode == self.GUIDE_MODES[4]
|
||||
aligned = guide_mode in (self.GUIDE_MODES[0], self.GUIDE_MODES[2]) or clip_context
|
||||
text = shot["prompt"] or ""
|
||||
@@ -4455,6 +4466,13 @@ class BFSShotH3Conditioning:
|
||||
+ f", the rest is kept ({int(nm.shape[2])} latent frames)", flush=True)
|
||||
positive = self._history(shot, positive, vae, gtag)
|
||||
latent = self._head(shot, latent, vae, gtag)
|
||||
if temporal:
|
||||
try:
|
||||
from .minimax_h3_native_temporal import patch_temporal_model
|
||||
except ImportError:
|
||||
from minimax_h3_native_temporal import patch_temporal_model
|
||||
model = patch_temporal_model(model)
|
||||
print(f"{gtag}: native temporal alignment enabled; use the MODEL output for sampling", flush=True)
|
||||
return (positive, latent, model, text)
|
||||
|
||||
@staticmethod
|
||||
@@ -4517,7 +4535,8 @@ class BFSShotH3ConditioningSimple:
|
||||
"aligned guide: preserve the shot frame by frame with MiniMax H3 Add Guide. "
|
||||
"native reference video: provide it as <Video 1> context. both: use both paths. "
|
||||
"aligned guide + CLIP video context: Qwen sees the shot at clip_video_fps without native "
|
||||
"shot reference latents; motion still uses Add Guide."}),
|
||||
"shot reference latents; motion still uses Add Guide. Native temporal alignment: experimental native "
|
||||
"video mode for its matching LoRA; MODEL input and output must be connected, no Add Guide or VSA."}),
|
||||
"ref_image_size": (["match", "max"], {"default": "match"}),
|
||||
},
|
||||
"optional": {
|
||||
@@ -4544,11 +4563,12 @@ class BFSShotH3ConditioningSimple:
|
||||
"tooltip": "Only for aligned guide + CLIP video context. Qwen sampling rate for this shot; "
|
||||
"try 2, 4 or 8 fps. Higher values cost more context/encoding time, not output fps. "
|
||||
"Console confirms sent frames, video vision tokens and completed encoding."}),
|
||||
"model": ("MODEL", {"tooltip": "Required for native temporal mode. Route MODEL output to the guider."}),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("CONDITIONING", "LATENT", "STRING")
|
||||
RETURN_NAMES = ("positive", "latent", "prompt")
|
||||
RETURN_TYPES = ("CONDITIONING", "LATENT", "STRING", "MODEL")
|
||||
RETURN_NAMES = ("positive", "latent", "prompt", "model")
|
||||
FUNCTION = "condition"
|
||||
CATEGORY = "BFS/shot loop"
|
||||
DESCRIPTION = ("Build the usual MiniMax H3 shot conditioning with references, prompt and an aligned guide "
|
||||
@@ -4556,14 +4576,14 @@ class BFSShotH3ConditioningSimple:
|
||||
|
||||
def condition(self, shot, clip, vae, guide_mode, ref_image_size,
|
||||
audio_vae=None, with_audio=False, vlm=None, inpaint=INPAINT_MODES[0],
|
||||
source_blur="off", source_blur_strength=1.0, source_blur_segmenter="SAM 3.1", source_blur_effect="blur", clip_video_fps=2.0):
|
||||
positive, latent, _, prompt = BFSShotH3Conditioning().condition(
|
||||
source_blur="off", source_blur_strength=1.0, source_blur_segmenter="SAM 3.1", source_blur_effect="blur", clip_video_fps=2.0, model=None):
|
||||
positive, latent, patched_model, prompt = BFSShotH3Conditioning().condition(
|
||||
shot, clip, vae, guide_mode, "none", ref_image_size,
|
||||
audio_vae=audio_vae, with_audio=with_audio, vlm=vlm, inpaint=inpaint,
|
||||
source_blur=source_blur, source_blur_strength=source_blur_strength,
|
||||
source_blur_segmenter=source_blur_segmenter, source_blur_effect=source_blur_effect,
|
||||
clip_video_fps=clip_video_fps)
|
||||
return positive, latent, prompt
|
||||
clip_video_fps=clip_video_fps, model=model)
|
||||
return positive, latent, prompt, patched_model
|
||||
|
||||
|
||||
class BFSShotJoin:
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
"""Experimental native H3 temporal inference, matching ai-toolkit native_temporal_v1.
|
||||
|
||||
Per-model wrappers only: no global ComfyUI monkey patches or extra weights.
|
||||
"""
|
||||
import copy
|
||||
import math
|
||||
import torch
|
||||
try:
|
||||
from . import minimax_h3_native_temporal_core as core
|
||||
except ImportError:
|
||||
import minimax_h3_native_temporal_core as core
|
||||
|
||||
MARKER = 'bfs_h3_native_temporal_v1'
|
||||
|
||||
|
||||
def temporal_layout(native, refs, keyframes, video_index=1):
|
||||
if keyframes:
|
||||
raise ValueError('Native temporal mode cannot be combined with Add Guide, first-frame guides or history. Remove those guides.')
|
||||
required = ('position_ids', 'segments', 'seq_len', 'signature')
|
||||
if any(not hasattr(native, x) for x in required):
|
||||
raise RuntimeError('Unsupported H3 PackedLayout API; update ComfyUI.')
|
||||
result = copy.copy(native)
|
||||
result.position_ids = native.position_ids.clone()
|
||||
source = torch.zeros(native.seq_len, dtype=torch.bool)
|
||||
target = torch.zeros_like(source)
|
||||
ta, tb, _ = next(s for s in native.segments if s[2] == 'video')
|
||||
target[ta:tb] = True
|
||||
# Native packing is text, each reference's optional audio then video, targets.
|
||||
cursor = next(s[1] for s in native.segments if s[2] == 'text')
|
||||
videos = 0
|
||||
for ref in refs:
|
||||
kind = ref['kind']
|
||||
if kind == 'image':
|
||||
cursor += (ref['latent_h']//2)*(ref['latent_w']//2)
|
||||
elif kind == 'audio':
|
||||
cursor += 2*ref['ref_audio_t']
|
||||
elif kind in ('video', 'video_audio'):
|
||||
videos += 1
|
||||
cursor += 2*ref.get('ref_audio_t', 0)
|
||||
n = ref['latent_t']*(ref['latent_h']//2)*(ref['latent_w']//2)
|
||||
if videos == video_index:
|
||||
source[cursor:cursor+n] = True
|
||||
result.position_ids[cursor:cursor+n, 0] += native.position_ids[ta, 0] - native.position_ids[cursor, 0]
|
||||
cursor += n
|
||||
else:
|
||||
raise ValueError('Unknown native H3 reference kind: '+str(kind))
|
||||
if not source.any():
|
||||
raise ValueError(f'Native temporal mode requires <Video {video_index}> as a native video reference. Connect Reference to Video positive conditioning.')
|
||||
st = torch.unique_consecutive(result.position_ids[source, 0])
|
||||
tt = torch.unique_consecutive(result.position_ids[target, 0])
|
||||
if st.shape != tt.shape or not torch.allclose(st, tt, atol=1e-9, rtol=0):
|
||||
raise ValueError('Native temporal guide and output must have identical temporal grids. Use the same frame window, 24 fps and matching H3 frame count (e.g. 107). No automatic motion stretching is applied.')
|
||||
return result, source, target
|
||||
|
||||
|
||||
def attention_override(ctx, previous=None):
|
||||
def override(original, q, k, v, heads, mask=None, skip_reshape=False, skip_output_reshape=False, **kwargs):
|
||||
# Token refiner and other attention paths retain their original backend.
|
||||
n = q.shape[-2] if skip_reshape else q.shape[1]
|
||||
if n != ctx.times.shape[1] or k.shape[-2 if skip_reshape else 1] != n:
|
||||
if previous:
|
||||
return previous(original, q, k, v, heads, mask=mask, skip_reshape=skip_reshape,
|
||||
skip_output_reshape=skip_output_reshape, **kwargs)
|
||||
return original(q, k, v, heads, mask=mask, skip_reshape=skip_reshape,
|
||||
skip_output_reshape=skip_output_reshape, **kwargs)
|
||||
if mask is not None:
|
||||
raise ValueError('Native temporal attention does not support another attention mask override.')
|
||||
if not skip_reshape:
|
||||
q, k, v = [x.view(x.shape[0], x.shape[1], heads, -1).transpose(1, 2) for x in (q,k,v)]
|
||||
out = core.attention(q, k, v, ctx)
|
||||
return out if skip_output_reshape else out.transpose(1, 2).reshape(out.shape[0], -1, out.shape[1]*out.shape[3])
|
||||
return override
|
||||
|
||||
|
||||
def patch_temporal_model(model, video_index=1, strength=2.0, window=0.2, falloff=0.25, cap=6.0):
|
||||
cfg = core.options(dict(native_temporal_alignment=True, native_temporal_strength=strength,
|
||||
native_temporal_window_seconds=window, native_temporal_falloff_seconds=falloff,
|
||||
native_temporal_max_penalty=cap))
|
||||
if isinstance(video_index, bool) or not isinstance(video_index, int) or video_index < 1:
|
||||
raise ValueError('motion_video_index must be a positive integer')
|
||||
if model is None:
|
||||
raise ValueError('Native temporal mode needs the MODEL input. Route this node MODEL output to the sampler/guider.')
|
||||
if model.model_options.get(MARKER):
|
||||
raise ValueError('Native temporal alignment is already installed. Use only one native temporal node per model path.')
|
||||
net = model.get_model_object('diffusion_model')
|
||||
if net.__class__.__name__ != 'MiniMaxH3Model':
|
||||
raise ValueError('Native temporal alignment requires a native ComfyUI MiniMaxH3 model.')
|
||||
opts = model.model_options.get('transformer_options', {})
|
||||
if opts.get('optimized_attention_override') or opts.get('patches_replace', {}).get('dit'):
|
||||
raise ValueError('Remove VSA and other attention/block replacement patches before native temporal alignment.')
|
||||
patched = model.clone()
|
||||
previous = patched.model_options.get('model_function_wrapper')
|
||||
cache = [None, None]
|
||||
logged = [False]
|
||||
|
||||
def wrapper(model_function, args):
|
||||
conds = args.get('c', {})
|
||||
payload = conds.get('minimax_payload') or {}
|
||||
native = payload.get('layout')
|
||||
if native is None:
|
||||
raise RuntimeError('Native H3 layout missing. Update ComfyUI and use native AV latent conditioning.')
|
||||
if cache[0] is not native:
|
||||
cache[:] = [native, temporal_layout(native, payload.get('refs', []), payload.get('keyframes'), video_index)]
|
||||
layout, source, target = cache[1]
|
||||
device = args['input'][0].device if isinstance(args['input'], (list, tuple)) else args['input'].device
|
||||
to = dict(conds.get('transformer_options', {}))
|
||||
if to.get('optimized_attention_override') or to.get('patches_replace', {}).get('dit'):
|
||||
raise ValueError('Native temporal alignment conflicts with another attention/block replacement patch.')
|
||||
times = layout.position_ids[:, 0].float()[None].to(device)/40.0
|
||||
ctx = core.TemporalContext(times, source[None].to(device), target[None].to(device),
|
||||
torch.ones_like(source[None], device=device), strength, window, falloff, cap,
|
||||
torch.tensor([strength,window,falloff,cap],device=device,dtype=torch.float32))
|
||||
to['optimized_attention_override'] = attention_override(ctx)
|
||||
copied = dict(args, c=dict(conds, minimax_payload=dict(payload, layout=layout), transformer_options=to))
|
||||
if not logged[0]:
|
||||
print(f'[BFS H3 native temporal] {core.VERSION}: <Video {video_index}>, {int(source.sum())} motion rows; temporal RoPE aligned; soft window={window}s strength={strength} cap={cap}; images remain native; Add Guide OFF', flush=True)
|
||||
logged[0] = True
|
||||
if previous:
|
||||
return previous(model_function, copied)
|
||||
return model_function(copied['input'], copied['timestep'], **copied['c'])
|
||||
|
||||
patched.set_model_unet_function_wrapper(wrapper)
|
||||
patched.model_options[MARKER] = dict(version=core.VERSION, video_index=video_index, **cfg)
|
||||
return patched
|
||||
|
||||
|
||||
class BFSMiniMaxH3NativeTemporal:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {'required': {'model': ('MODEL',),
|
||||
'motion_video_index': ('INT', {'default':1, 'min':1, 'max':100,
|
||||
'tooltip':'Native <Video N> supplying motion. Picture numbering is independent.'}),
|
||||
'strength': ('FLOAT', {'default':2.0,'min':0.0,'max':20.0,'step':0.1}),
|
||||
'window_seconds': ('FLOAT', {'default':0.2,'min':0.0,'max':10.0,'step':0.01}),
|
||||
'falloff_seconds': ('FLOAT', {'default':0.25,'min':0.01,'max':10.0,'step':0.01}),
|
||||
'max_penalty': ('FLOAT', {'default':6.0,'min':0.0,'max':30.0,'step':0.1})}}
|
||||
RETURN_TYPES = ('MODEL',)
|
||||
FUNCTION = 'apply'
|
||||
CATEGORY = 'BFS/MiniMax H3'
|
||||
DESCRIPTION = ('Experimental native_temporal_v1 inference for the matching motion-transfer LoRA. '
|
||||
'Keep the motion video in MiniMax H3 Reference to Video positive conditioning; route this MODEL '
|
||||
'to the guider. No Add Guide or VSA. Guide/output must share a 24fps temporal grid. '
|
||||
'Does not encode video or load the LoRA. Defaults match training.')
|
||||
|
||||
def apply(self, model, motion_video_index=1, strength=2.0, window_seconds=0.2, falloff_seconds=0.25, max_penalty=6.0):
|
||||
return (patch_temporal_model(model,motion_video_index,strength,window_seconds,falloff_seconds,max_penalty),)
|
||||
|
||||
NODE_CLASS_MAPPINGS = {'BFSMiniMaxH3NativeTemporal': BFSMiniMaxH3NativeTemporal}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {'BFSMiniMaxH3NativeTemporal': 'BFS MiniMax H3 Native Temporal Alignment (Experimental)'}
|
||||
@@ -0,0 +1,122 @@
|
||||
"""Native-reference temporal alignment v1. No new learned model weights.
|
||||
|
||||
Only source channel 1's VIDEO latent rows are moved onto the target clock.
|
||||
Image-reference spatial/temporal positions remain native. Target-to-motion
|
||||
attention receives a bounded, soft temporal penalty; all other edges stay dense.
|
||||
The CUDA implementation uses FlexAttention, never a quadratic dense bias tensor.
|
||||
"""
|
||||
from dataclasses import dataclass
|
||||
import math
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
|
||||
VERSION = 'native_temporal_v1'
|
||||
_FLEX = None
|
||||
|
||||
def options(kwargs):
|
||||
enabled = kwargs.get('native_temporal_alignment', False)
|
||||
if not isinstance(enabled, bool):raise ValueError('native_temporal_alignment must be boolean')
|
||||
result = dict(enabled=enabled, source_channel=kwargs.get('native_motion_source_channel',1),
|
||||
strength=kwargs.get('native_temporal_strength',2.0),
|
||||
window=kwargs.get('native_temporal_window_seconds',0.2),
|
||||
falloff=kwargs.get('native_temporal_falloff_seconds',0.25),
|
||||
cap=kwargs.get('native_temporal_max_penalty',6.0))
|
||||
if isinstance(result['source_channel'],bool) or not isinstance(result['source_channel'],int) or result['source_channel']<1:raise ValueError('Invalid native motion source channel')
|
||||
for name in ['strength','window','falloff','cap']:
|
||||
x=result[name]
|
||||
if isinstance(x,bool) or not isinstance(x,(int,float)) or not math.isfinite(x) or x<0 or (name=='falloff' and x==0):raise ValueError('Invalid native temporal '+name)
|
||||
if enabled and (kwargs.get('align_video_refs') or kwargs.get('align_image_refs') or kwargs.get('reference_source_phase') or kwargs.get('reference_rope_layout','native')!='native'):
|
||||
raise ValueError('Native temporal mode requires native references, no AddGuide alignment or source-phase patch')
|
||||
return result
|
||||
|
||||
def align_layout(layout, ref_blocks, source_ids, cfg):
|
||||
"""Return a sequence-row mask, aligning only the explicit motion video's clock.
|
||||
|
||||
Native video latents remain in ref_blocks and keep their own spatial grid.
|
||||
Reference audio is not temporally biased by this video-only experiment.
|
||||
"""
|
||||
mask=torch.zeros(layout.sequence_length,dtype=torch.bool)
|
||||
if not cfg['enabled'] or not ref_blocks:return mask
|
||||
if len(source_ids)!=len(ref_blocks):raise ValueError('Native temporal mode requires exact reference source IDs')
|
||||
cursor=int(layout.text_indices.numel());found=0
|
||||
# target rows are always the final packed rows; condition video indices precede them.
|
||||
target_start=int(layout.audio_indices[-1])+1 if layout.audio_indices.numel() else layout.sequence_length
|
||||
# For zero audio use the count of conditioning video rows to locate the target.
|
||||
target_idx=layout.video_indices[layout.num_condition_video_rows:]
|
||||
target_start=int(target_idx[0]);target_origin=layout.position_ids[target_start,0].clone()
|
||||
for block, source in zip(ref_blocks,source_ids):
|
||||
t,h,w=block[:3];a=block[3] if len(block)>3 else 0
|
||||
cursor+=int(a)*2
|
||||
rows=int(t)*(int(h)//2)*(int(w)//2)
|
||||
if source==cfg['source_channel']:
|
||||
if t<=1:raise ValueError('Motion channel must be a video, not an image')
|
||||
if len(block)>4 and block[4]:raise ValueError('Native motion video was accidentally converted to aligned guide')
|
||||
found+=1;mask[cursor:cursor+rows]=True
|
||||
layout.position_ids=layout.position_ids.clone()
|
||||
layout.position_ids[cursor:cursor+rows,0]+=target_origin-layout.position_ids[cursor,0].clone()
|
||||
# A common temporal extent is required; never silently stretch motion to fit.
|
||||
src_times=torch.unique_consecutive(layout.position_ids[cursor:cursor+rows,0])
|
||||
dst_times=torch.unique_consecutive(layout.position_ids[target_idx,0])
|
||||
if src_times.shape!=dst_times.shape or not torch.allclose(src_times,dst_times,atol=1e-9,rtol=0):raise ValueError('Native motion and target must use identical temporal grids; resample both to matching frame windows')
|
||||
cursor+=rows
|
||||
if found!=1:raise ValueError(f'Expected one motion video in channel {cfg["source_channel"]}, found {found}')
|
||||
assert not mask[target_idx].any()
|
||||
return mask
|
||||
|
||||
def pad_motion_masks(layouts, masks):
|
||||
max_text=max(int(x.text_indices.numel()) for x in layouts)
|
||||
total=max_text+layouts[0].sequence_length-int(layouts[0].text_indices.numel())
|
||||
result=torch.zeros(len(layouts),total,dtype=torch.bool)
|
||||
for i,(layout,mask) in enumerate(zip(layouts,masks)):
|
||||
n=int(layout.text_indices.numel());result[i,:n]=mask[:n];result[i,max_text:]=mask[n:]
|
||||
return result
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TemporalContext:
|
||||
times: torch.Tensor
|
||||
source: torch.Tensor
|
||||
target: torch.Tensor
|
||||
live: torch.Tensor
|
||||
strength: float
|
||||
window: float
|
||||
falloff: float
|
||||
cap: float
|
||||
policy: torch.Tensor | None = None
|
||||
|
||||
def context(position_ids, source_mask, tags, video_indices, target_grid, cfg):
|
||||
if not cfg['enabled'] or source_mask is None or not bool(source_mask.any()):return None
|
||||
if source_mask.shape!=tags.shape:raise ValueError('Motion mask must match packed token rows')
|
||||
target=torch.zeros_like(source_mask);n=math.prod(target_grid)
|
||||
target[:,video_indices[-n:]]=True
|
||||
return TemporalContext(position_ids[:,:,0].float()/40.0,source_mask,target,tags>=0,
|
||||
float(cfg['strength']),float(cfg['window']),float(cfg['falloff']),float(cfg['cap']),
|
||||
torch.tensor([cfg['strength'],cfg['window'],cfg['falloff'],cfg['cap']],device=position_ids.device,dtype=torch.float32))
|
||||
|
||||
def score_modifier(ctx):
|
||||
# Captured tensors are O(sequence length). No SxS matrix is materialized on CUDA.
|
||||
times,source,target,live=ctx.times,ctx.source,ctx.target,ctx.live
|
||||
policy=ctx.policy if ctx.policy is not None else torch.tensor([ctx.strength,ctx.window,ctx.falloff,ctx.cap],device=times.device,dtype=torch.float32)
|
||||
def modify(score,b,h,q,k):
|
||||
qi=q.clamp(max=times.shape[1]-1);ki=k.clamp(max=times.shape[1]-1)
|
||||
dt=(times[b,qi]-times[b,ki]).abs()
|
||||
penalty=torch.minimum(policy[0]*(torch.relu(dt-policy[1])/policy[2]).square(),policy[3])
|
||||
edited=torch.where(target[b,qi]&source[b,ki],score-penalty,score)
|
||||
return torch.where(live[b,ki],edited,float('-inf'))
|
||||
return modify
|
||||
|
||||
def dense_reference(q,k,v,ctx):
|
||||
"""Tiny test oracle only; forbidden at production sequence lengths."""
|
||||
if q.shape[-2]>256:raise ValueError('Dense temporal reference is only for tiny tests')
|
||||
delta=(ctx.times[:,:,None]-ctx.times[:,None,:]).abs()
|
||||
penalty=(ctx.strength*(F.relu(delta-ctx.window)/ctx.falloff).square()).clamp(max=ctx.cap)
|
||||
bias=torch.where(ctx.target[:,:,None]&ctx.source[:,None,:],-penalty,0.0)
|
||||
bias=bias.masked_fill(~ctx.live[:,None,:],float('-inf'))
|
||||
return F.scaled_dot_product_attention(q,k,v,attn_mask=bias[:,None].to(q.dtype))
|
||||
|
||||
def attention(q,k,v,ctx):
|
||||
global _FLEX
|
||||
if q.device.type!='cuda':return dense_reference(q,k,v,ctx)
|
||||
if _FLEX is None:
|
||||
from torch.nn.attention.flex_attention import flex_attention
|
||||
_FLEX=torch.compile(flex_attention,dynamic=True,fullgraph=True)
|
||||
return _FLEX(q,k,v,score_mod=score_modifier(ctx))
|
||||
+1
-1
@@ -1,7 +1,7 @@
|
||||
[project]
|
||||
name = "bfsnodes"
|
||||
description = "Custom ComfyUI nodes for reference-guided video: BFS Sampler (guide-driven IC-LoRAs, auto crop, mask inpainting, temporal chunking), LTX Multiple Controls / Identity Transfer / Edit Anything, Head-Swap and Anime2Real conditioning, MiniMax-H3 latent guides, RoPE layouts and loaders, LoRA Surgery, and Shot Planner / Shot Loop (tabbed panel): split long videos into shots at camera cuts, face-tracked cast, SAM 3 mask per shot with full frame, paste, mask, crop and crop + mask generation modes, optional translucent Colored Mark overlay for manually selected regions, optional Qwen3-VL to suggest settings, describe references ({details}) and the person to replace ({target}), continuity between shots, H3 conditioning with an optional duet and prompt writer, join with the original timing, audio and a side-by-side comparison. H3 Duet: pin a source clip beside the video, training-free."
|
||||
version = "1.115.8"
|
||||
version = "1.115.9"
|
||||
license = {file = "LICENSE"}
|
||||
dependencies = [
|
||||
"torch",
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -484,6 +484,20 @@ class ConditioningPromptTest(unittest.TestCase):
|
||||
self.assertEqual(out[2], "m")
|
||||
self.assertNotIn("duet", SL.BFSShotH3Conditioning.INPUT_TYPES()["optional"])
|
||||
|
||||
def test_native_temporal_requires_model_before_encoding(self):
|
||||
shot = self._shot()
|
||||
with self.assertRaisesRegex(ValueError, "MODEL connected"):
|
||||
SL.BFSShotH3Conditioning().condition(shot, "clip", "vae", SL.BFSShotH3Conditioning.GUIDE_MODES[5], "none", "match")
|
||||
self.assertEqual(self.seen, {})
|
||||
|
||||
def test_native_temporal_passes_shot_as_video_and_returns_patched_model(self):
|
||||
import minimax_h3_native_temporal as temporal
|
||||
shot = self._shot(); shot["fps"] = 24
|
||||
with patch.object(temporal, "patch_temporal_model", return_value="patched"):
|
||||
out = SL.BFSShotH3Conditioning().condition(shot, "clip", "vae", SL.BFSShotH3Conditioning.GUIDE_MODES[5], "none", "match", model="model")
|
||||
self.assertEqual(out[2], "patched")
|
||||
self.assertIs(self.seen["ref_videos"]["ref_video_1"], shot["frames"])
|
||||
|
||||
def test_setting_picture_is_the_last_reference_with_static(self):
|
||||
mask = torch.zeros(22, 8, 8)
|
||||
mask[:, 2:4, 2:4] = 1
|
||||
@@ -505,11 +519,11 @@ class SimpleConditioningTest(unittest.TestCase):
|
||||
self.assertIn("with_audio", names)
|
||||
self.assertIn("inpaint", names)
|
||||
self.assertNotIn("duet", names)
|
||||
self.assertNotIn("model", names)
|
||||
self.assertIn("model", names)
|
||||
self.assertNotIn("rope_gap", names)
|
||||
self.assertNotIn("task", names)
|
||||
|
||||
def test_forwards_safe_advanced_defaults_and_hides_model_output(self):
|
||||
def test_forwards_safe_advanced_defaults_and_appends_model_output(self):
|
||||
seen = {}
|
||||
original = SL.BFSShotH3Conditioning.condition
|
||||
|
||||
@@ -525,7 +539,7 @@ class SimpleConditioningTest(unittest.TestCase):
|
||||
finally:
|
||||
SL.BFSShotH3Conditioning.condition = original
|
||||
|
||||
self.assertEqual(out, ("positive", "latent", "prompt"))
|
||||
self.assertEqual(out, ("positive", "latent", "prompt", "unchanged model"))
|
||||
self.assertEqual(seen["args"][4], "none")
|
||||
self.assertTrue(seen["kwargs"]["with_audio"])
|
||||
self.assertEqual(seen["kwargs"]["inpaint"], SL.INPAINT_MODES[0])
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
import copy
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from types import SimpleNamespace
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0,str(ROOT))
|
||||
import minimax_h3_native_temporal as node
|
||||
|
||||
def load(name,path):
|
||||
s=importlib.util.spec_from_file_location(name,path);m=importlib.util.module_from_spec(s);sys.modules[name]=m;s.loader.exec_module(m);return m
|
||||
CORE=load('native_temporal_comfy_fixture',ROOT/'tests/fixtures/comfy_h3_layout_8cfe5e1.py')
|
||||
|
||||
def example(vt=7):
|
||||
refs=[dict(kind='image',latent_h=4,latent_w=8),dict(kind='video_audio',latent_h=8,latent_w=4,latent_t=vt,ref_audio_t=3)]
|
||||
l=CORE.PackedLayout(3,7,8,8,2,refs=refs)
|
||||
return l,refs
|
||||
|
||||
def test_layout_matches_actual_training_coordinates():
|
||||
l,refs=example();before=l.position_ids.clone()
|
||||
aligned,src,tgt=node.temporal_layout(l,refs,None)
|
||||
golden=json.loads((ROOT/'tests/fixtures/h3_native_temporal_training_positions.json').read_text())
|
||||
torch.testing.assert_close(aligned.position_ids,torch.tensor(golden['position_ids'],dtype=torch.float64),rtol=0,atol=1e-12)
|
||||
assert src.tolist()==golden['source']
|
||||
assert torch.equal(l.position_ids,before)
|
||||
assert torch.equal(aligned.position_ids[~src],before[~src])
|
||||
assert torch.equal(aligned.position_ids[src,1:],before[src,1:])
|
||||
assert not (src&tgt).any()
|
||||
|
||||
def test_wrong_video_index_and_temporal_mismatch_rejected():
|
||||
l,r=example()
|
||||
with pytest.raises(ValueError,match='Video 2'):node.temporal_layout(l,r,None,2)
|
||||
l,r=example(6)
|
||||
with pytest.raises(ValueError,match='identical temporal'):node.temporal_layout(l,r,None)
|
||||
with pytest.raises(ValueError,match='Add Guide'):node.temporal_layout(l,r,[{}])
|
||||
|
||||
def context():
|
||||
l,r=example();a,s,t=node.temporal_layout(l,r,None)
|
||||
return node.core.TemporalContext(a.position_ids[:,0].float()[None]/40,s[None],t[None],torch.ones_like(s[None]),2,.2,.25,6)
|
||||
|
||||
def test_attention_adapter_matches_dense_score_policy_and_shapes():
|
||||
ctx=context();torch.manual_seed(12)
|
||||
q,k,v=[torch.randn(1,2,ctx.times.shape[1],8) for _ in range(3)]
|
||||
def forbidden(*args,**kwargs):raise AssertionError('wrong backend')
|
||||
out=node.attention_override(ctx)(forbidden,q,k,v,2,skip_reshape=True)
|
||||
expected=node.core.dense_reference(q,k,v,ctx).transpose(1,2).flatten(2)
|
||||
torch.testing.assert_close(out,expected)
|
||||
raw=[x.transpose(1,2).flatten(2) for x in (q,k,v)]
|
||||
torch.testing.assert_close(node.attention_override(ctx)(forbidden,*raw,2),expected)
|
||||
|
||||
def test_token_refiner_uses_existing_attention():
|
||||
ctx=context();q=torch.randn(1,2,3,8)
|
||||
def original(*args,**kwargs):return 'original'
|
||||
assert node.attention_override(ctx)(original,q,q,q,2,skip_reshape=True)=='original'
|
||||
|
||||
class Model:
|
||||
def __init__(self):self.model_options={}
|
||||
def clone(self):m=Model();m.model_options=copy.deepcopy(self.model_options);return m
|
||||
def get_model_object(self,key):return type('MiniMaxH3Model',(),{})()
|
||||
def set_model_unet_function_wrapper(self,w):self.model_options['model_function_wrapper']=w
|
||||
|
||||
def test_per_model_wrapper_clones_payload_and_applies_attention():
|
||||
m=Model();patched=node.patch_temporal_model(m);assert not m.model_options
|
||||
l,r=example();payload=dict(layout=l,refs=r)
|
||||
def fn(x,t,**c):
|
||||
assert c['minimax_payload']['layout'] is not l
|
||||
assert 'optimized_attention_override' in c['transformer_options']
|
||||
return 'ok'
|
||||
args=dict(input=torch.zeros(1),timestep=torch.ones(1),c=dict(minimax_payload=payload))
|
||||
assert patched.model_options['model_function_wrapper'](fn,args)=='ok'
|
||||
assert payload['layout'] is l and 'transformer_options' not in args['c']
|
||||
with pytest.raises(ValueError,match='already installed'):node.patch_temporal_model(patched)
|
||||
|
||||
def test_incompatible_attention_rejected():
|
||||
m=Model();m.model_options={'transformer_options':{'optimized_attention_override':object()}}
|
||||
with pytest.raises(ValueError,match='VSA'):node.patch_temporal_model(m)
|
||||
with pytest.raises(ValueError,match='MODEL input'):node.patch_temporal_model(None)
|
||||
|
||||
|
||||
def test_second_video_selection_preserves_first_video_and_images():
|
||||
refs=[dict(kind='image',latent_h=4,latent_w=8),
|
||||
dict(kind='video',latent_h=4,latent_w=4,latent_t=3,ref_audio_t=0),
|
||||
dict(kind='video',latent_h=8,latent_w=4,latent_t=7,ref_audio_t=0)]
|
||||
native=CORE.PackedLayout(3,7,8,8,2,refs=refs)
|
||||
aligned,source,target=node.temporal_layout(native,refs,None,2)
|
||||
assert int(source.sum())==56
|
||||
assert torch.equal(aligned.position_ids[~source],native.position_ids[~source])
|
||||
assert not source[3:23].any()
|
||||
@@ -953,7 +953,15 @@ function upgradeConditioningWidgets(node, info) {
|
||||
|
||||
function upgradeLegacySlots(node, info) {
|
||||
const cls = node.comfyClass;
|
||||
if (cls === "BFSShotH3Conditioning" || cls === "BFSShotH3ConditioningSimple") { upgradeConditioningWidgets(node, info); return; }
|
||||
if (cls === "BFSShotH3Conditioning" || cls === "BFSShotH3ConditioningSimple") {
|
||||
upgradeConditioningWidgets(node, info);
|
||||
// 1.115.9 appends MODEL routing to Simple; preserve all existing slot indices/links.
|
||||
if (cls === "BFSShotH3ConditioningSimple") {
|
||||
if (!node.inputs?.some(x => x.name === "model")) node.addInput("model", "MODEL");
|
||||
if (!node.outputs?.some(x => x.name === "model")) node.addOutput("model", "MODEL");
|
||||
}
|
||||
return;
|
||||
}
|
||||
// ref_image / ref_image_2 outputs at slots `at`, at+1 -> refs (list) / ref_count
|
||||
const listOutputs = at => {
|
||||
const a = node.outputs?.[at], b = node.outputs?.[at + 1];
|
||||
|
||||
@@ -754,3 +754,7 @@ After marking a stored shot for redo, click **Undo redo** to keep its previous t
|
||||
Once a shot has been regenerated, its previous take cannot be restored through this control.
|
||||
**Refresh status** rereads the cache status; it cannot recover files deleted by older versions of Reset loop.
|
||||
Changing the seed, model or other render settings still invalidates stored shots even after cancelling redo.
|
||||
|
||||
### Experimental native temporal motion transfer
|
||||
|
||||
`BFS Shot H3 Conditioning` now has **native temporal alignment (experimental)** in `guide_mode`. Connect MODEL in and route its MODEL output to the guider. This uses the matching experimental LoRA, the shot as native `<Video 1>`, temporal RoPE alignment and soft temporal attention; it does not use Add Guide. Use 24 fps and disable continuity/first-frame/mask guides and VSA. Simple appends a MODEL output without moving existing outputs. The standalone **BFS MiniMax H3 Native Temporal Alignment (Experimental)** node provides the same patch for native workflows. See `H3_NATIVE_TEMPORAL.md` for the full contract and validation limits.
|
||||
|
||||
Reference in New Issue
Block a user