Add standalone native H3 shot loop with optional image continuity and workflow examples
This commit is contained in:
@@ -0,0 +1,69 @@
|
||||
# Native H3 shot loop without Planner (experimental)
|
||||
|
||||
Requires BFSNodes **1.115.27**. Update, restart ComfyUI and refresh the browser.
|
||||
|
||||
Load [the auto-prompt loop workflow](examples/h3_tara_native_auto_prompt_loop.comfy.json)
|
||||
(or [API graph](examples/h3_tara_native_auto_prompt_loop.api.json)). It is a separate
|
||||
variant of the single-clip workflow, with the same models, reference description,
|
||||
per-segment prompt generation and native H3 sampling. No Planner or Planner run
|
||||
state is involved.
|
||||
|
||||
## Wiring
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Load["Load Video · force 24fps"] --> Start["Native Shot Loop Start"]
|
||||
Start --> Prepare["Prepare Native Video"]
|
||||
Prepare --> Prompt["Guide analysis + H3 prompt generation"]
|
||||
Refs["Describe References · original Pictures 1..N"] --> Continuity
|
||||
Prompt --> Continuity["Native Shot Continuity"]
|
||||
Start --> Continuity
|
||||
Continuity --> Encode["Native Reference Set to Video"]
|
||||
Prepare --> Encode
|
||||
Encode --> Sample["Sampler + decode + trim padding"]
|
||||
Sample --> End["Native Shot Loop End / Join"]
|
||||
Start --> End
|
||||
End --> Save["Create Video + Save Video"]
|
||||
Load --> Audio["Original audio"] --> Save
|
||||
```
|
||||
|
||||
- **Loop Start:** default chunks of 107 source frames. `camera_cuts` accepts
|
||||
comma-separated 24fps frame indices relative to the loaded clip. A cut always
|
||||
starts a new segment. Empty means one continuous take divided into chunks;
|
||||
**there is no automatic scene detection in these nodes**. Set VHS `force_rate`
|
||||
to 24 and `select_every_nth` to 1. Set cuts after any loader trim/skip.
|
||||
- **Continuity:** enabled by default. After the first segment, append its last
|
||||
generated frame as the next native Picture. All user Picture indices remain
|
||||
unchanged. The appended instruction limits that image to appearance/scene
|
||||
continuity while the current guide supplies movement. Disable at camera cuts
|
||||
automatically; switch `enabled` off to disable it entirely. At most **8 user
|
||||
images** when continuity is active (H3 node limit: 9 including the anchor).
|
||||
- **Loop End:** stores each decoded segment in the ComfyUI temp directory,
|
||||
expands only the start-to-end body, then concatenates the segments. Model
|
||||
loading/LoRA/TARA nodes stay shared outside the loop. Connect the final images
|
||||
to Create Video; connect original loader audio separately. Do not add final
|
||||
Save Video inside the repeated body.
|
||||
|
||||
Keep Prepare Native Video's `max_frames=0` and `frame_rounding=pad last frame`.
|
||||
The example trims H3's `17n+5` padding before the join. A final segment under five
|
||||
frames repeats its last source frame internally to meet H3's minimum; Loop End
|
||||
keeps only its actual frame count. No overlap or crossfade is applied, so output
|
||||
length remains the original loaded clip length at 24fps.
|
||||
|
||||
This is **soft native-image continuity**, not Add Guide, first-frame latent
|
||||
locking or a guarantee of seamless boundaries. TARA can remain enabled because
|
||||
no latent head/history guide is added. Continuity may still bias composition or
|
||||
propagate errors; disable it to compare. There is no performer tracking in these
|
||||
nodes: ambiguous source assignments after crossings/cuts still require suitable
|
||||
prompts or bindings.
|
||||
|
||||
The temp manifest records the segment ranges and stored frame tensors. They are
|
||||
not durable checkpoints or an automatic restart/resume mechanism. Float image
|
||||
caches can be large; the final concatenation also uses CPU RAM. Remove the
|
||||
corresponding `temp/bfs_h3_native_loop/<session>` folder after saving the final
|
||||
video if it is no longer needed. Do not remove it while that run is active.
|
||||
|
||||
Validation: four CPU checks cover exact frame ranges/cuts, anchor ordering and
|
||||
cut resets, short-tail trimming, actual ComfyUI GraphBuilder expansion and
|
||||
bidirectional links in the example. No end-to-end GPU render, browser execution,
|
||||
identity quality or seamlessness claim has been validated for this feature yet.
|
||||
@@ -222,3 +222,7 @@ attached reference in Picture 1..N order. Original sizes and pixels are preserve
|
||||
images of different sizes are not resized into a batch. The existing reference
|
||||
bundle and text outputs retain their indices. This preview connection does not
|
||||
change which images are sent to the encoder.
|
||||
|
||||
### Native shot loop without Planner (1.115.27, experimental)
|
||||
|
||||
Separate Start, Continuity and End/Join nodes now have a [native auto-prompt loop example](examples/h3_tara_native_auto_prompt_loop.comfy.json). See [wiring, manual cuts, continuity and validation limits](H3_NATIVE_SHOT_LOOP.md).
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
from .bfs_h3_native_shot_loop import NODE_CLASS_MAPPINGS as H3LOOP_NODE_CLASS_MAPPINGS
|
||||
from .bfs_h3_native_shot_loop import NODE_DISPLAY_NAME_MAPPINGS as H3LOOP_NODE_DISPLAY_NAME_MAPPINGS
|
||||
from .nodes import NODE_CLASS_MAPPINGS as BFS_NODE_CLASS_MAPPINGS
|
||||
from .nodes import NODE_DISPLAY_NAME_MAPPINGS as BFS_NODE_DISPLAY_NAME_MAPPINGS
|
||||
from .ltxv_editanything import NODE_CLASS_MAPPINGS as LTXV_EA_NODE_CLASS_MAPPINGS
|
||||
@@ -140,6 +142,7 @@ except Exception as _e: # noqa
|
||||
H3WORKFLOW_NODE_CLASS_MAPPINGS, H3WORKFLOW_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
**H3LOOP_NODE_CLASS_MAPPINGS,
|
||||
**H3WORKFLOW_NODE_CLASS_MAPPINGS,
|
||||
**H3CLIP_NODE_CLASS_MAPPINGS,
|
||||
**H3GUIDE_NODE_CLASS_MAPPINGS,
|
||||
@@ -172,6 +175,7 @@ NODE_CLASS_MAPPINGS = {
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
**H3LOOP_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**H3WORKFLOW_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**H3CLIP_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**H3GUIDE_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
|
||||
@@ -0,0 +1,160 @@
|
||||
"""Sequential native H3 workflow expansion, independent of Shot Planner.
|
||||
|
||||
The loop body is expanded after each decoded segment. Models stay outside it.
|
||||
Decoded segments are cached on disk; no mutable global session state is used.
|
||||
"""
|
||||
import json
|
||||
import math
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
|
||||
CATEGORY = 'BFS/MiniMax H3/Native Shot Loop'
|
||||
|
||||
|
||||
def ranges(count, chunk_frames, cuts):
|
||||
if count < 1 or chunk_frames < 5:
|
||||
raise ValueError('Load a nonempty video and choose at least 5 frames per segment')
|
||||
try:
|
||||
cut_list = sorted(set(int(x.strip()) for x in cuts.split(',') if x.strip()))
|
||||
except ValueError as exc:
|
||||
raise ValueError('Camera cuts must be comma-separated source-frame indices') from exc
|
||||
if any(x <= 0 or x >= count for x in cut_list):
|
||||
raise ValueError('Camera cuts must be inside the loaded video frame range')
|
||||
result = []
|
||||
for a, b in zip([0] + cut_list, cut_list + [count]):
|
||||
for start in range(a, b, chunk_frames):
|
||||
result.append(dict(start=start, end=min(b, start+chunk_frames),
|
||||
camera_cut=start in cut_list))
|
||||
return result
|
||||
|
||||
|
||||
class BFSH3NativeShotLoopStart:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {'required': {'images': ('IMAGE',),
|
||||
'source_fps': ('FLOAT', {'default':24., 'min':.01}),
|
||||
'chunk_frames': ('INT', {'default':107, 'min':5, 'max':3600}),
|
||||
'camera_cuts': ('STRING', {'default':'', 'tooltip':'Comma-separated SOURCE frame indices, relative to the loaded clip. Empty = one continuous take split by chunk_frames. No automatic cut detection.'})},
|
||||
'optional': {'loop_state': ('BFS_H3_NATIVE_LOOP',)}}
|
||||
RETURN_TYPES = ('IMAGE', 'BFS_H3_NATIVE_LOOP', 'FLOAT', 'INT', 'INT')
|
||||
RETURN_NAMES = ('shot_images', 'loop_state', 'source_fps', 'shot_index', 'shot_count')
|
||||
FUNCTION = 'start'; CATEGORY = CATEGORY
|
||||
|
||||
def start(self, images, source_fps=24., chunk_frames=107, camera_cuts='', loop_state=None):
|
||||
if not math.isfinite(source_fps) or abs(source_fps-24.)>1e-6:
|
||||
raise ValueError('Native H3 shot loop requires a 24 fps loaded video. Set VHS Load Video force_rate=24 before this node to keep boundaries and audio aligned.')
|
||||
if loop_state is None:
|
||||
import folder_paths
|
||||
folder = Path(folder_paths.get_temp_directory()) / 'bfs_h3_native_loop' / uuid.uuid4().hex
|
||||
folder.mkdir(parents=True)
|
||||
loop_state = dict(index=0, ranges=ranges(len(images),chunk_frames,camera_cuts),
|
||||
folder=str(folder), segments=[], fps=24.,
|
||||
source_frames=len(images), source_fps=source_fps)
|
||||
state = dict(loop_state)
|
||||
if len(images) != state['source_frames'] or source_fps != state['source_fps']:
|
||||
raise ValueError('Source video changed during the native shot loop')
|
||||
shot = state['ranges'][state['index']]
|
||||
print(f"[BFS H3 native loop] segment {state['index']+1}/{len(state['ranges'])}: source frames {shot['start']}..{shot['end']-1}",flush=True)
|
||||
frames=images[shot['start']:shot['end']]
|
||||
if len(frames)<5:
|
||||
frames=torch.cat((frames,frames[-1:].expand(5-len(frames),-1,-1,-1)))
|
||||
return frames,state,source_fps,state['index']+1,len(state['ranges'])
|
||||
|
||||
|
||||
class BFSH3NativeShotContinuity:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {'required': {'references': ('BFS_H3_REFERENCE_SET',),
|
||||
'prompt': ('STRING', {'forceInput':True}), 'loop_state': ('BFS_H3_NATIVE_LOOP',),
|
||||
'enabled': ('BOOLEAN', {'default':True})}}
|
||||
RETURN_TYPES = ('BFS_H3_REFERENCE_SET', 'STRING', 'STRING')
|
||||
RETURN_NAMES = ('references', 'prompt', 'continuity_status')
|
||||
FUNCTION = 'prepare'; CATEGORY = CATEGORY
|
||||
DESCRIPTION = 'Experimental native still-image continuity, not first-frame latent locking. Appends the previous generated last frame after all user pictures, only within a continuous take. Original Picture numbers stay fixed.'
|
||||
|
||||
def prepare(self, references, prompt, loop_state, enabled=True):
|
||||
state=loop_state
|
||||
shot=state['ranges'][state['index']]
|
||||
if not enabled or not state['segments'] or shot['camera_cut']:
|
||||
reason='disabled' if not enabled else 'camera cut' if shot['camera_cut'] else 'first segment'
|
||||
return references,prompt,'Continuity OFF: '+reason
|
||||
if len(references)>=9:
|
||||
raise ValueError('Native continuity needs one free Picture slot: use at most 8 user reference images, or disable continuity')
|
||||
previous=torch.load(state['segments'][-1]['path'],map_location='cpu',weights_only=True)
|
||||
anchor=previous[-1:].clone()
|
||||
index=len(references)+1
|
||||
text=(f' Continuity anchor: <Picture {index}> is the last generated frame of the preceding segment of this same take. '
|
||||
'Use it only for consistent scene appearance, character identity and wardrobe at the boundary. '
|
||||
'The original assigned identity and environment pictures and user edits remain authoritative. '
|
||||
'Continue the current <Video 1> performance and timing; do not freeze or repeat the anchor pose, '
|
||||
'copy its earlier action, add another performer or introduce a transition.')
|
||||
return tuple(references)+(anchor,),prompt+text,f'Continuity ON: previous final frame is Picture {index} (native image reference, not a latent head guide)'
|
||||
|
||||
|
||||
class BFSH3NativeShotLoopEnd:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {'required': {'images': ('IMAGE',), 'loop_state': ('BFS_H3_NATIVE_LOOP',)},
|
||||
'hidden': {'dynprompt':'DYNPROMPT','unique_id':'UNIQUE_ID'}}
|
||||
RETURN_TYPES=('IMAGE','FLOAT','STRING')
|
||||
RETURN_NAMES=('joined_images','fps','manifest')
|
||||
FUNCTION='finish'; CATEGORY=CATEGORY
|
||||
DESCRIPTION='Wire trimmed decoded shot frames here. Expands only the start-to-end workflow body until all segments are generated, then joins at 24 fps. Connect audio separately from the original video loader to Create Video.'
|
||||
|
||||
def finish(self,images,loop_state,dynprompt=None,unique_id=None):
|
||||
state=dict(loop_state); index=state['index']; shot=state['ranges'][index]
|
||||
expected=shot['end']-shot['start']
|
||||
# Prepare Native Video pads tiny tail segments to H3's minimum grid.
|
||||
if len(images)<expected:
|
||||
raise ValueError(f'Segment {index+1} has {len(images)} decoded frames but needs {expected}; use pad last frame and remove video/frame caps')
|
||||
path=Path(state['folder'])/f'{index:05d}.pt'
|
||||
frames=images[:expected].detach().cpu().contiguous()
|
||||
tmp=path.with_suffix('.tmp');torch.save(frames,tmp);tmp.replace(path)
|
||||
segments=list(state['segments'])+[dict(path=str(path),frames=expected)]
|
||||
state.update(segments=segments,index=index+1)
|
||||
(path.parent/'manifest.json').write_text(json.dumps(state,indent=2))
|
||||
if state['index'] < len(state['ranges']):
|
||||
if dynprompt is None:
|
||||
raise ValueError('Native shot loop requires ComfyUI dynamic graph execution')
|
||||
return self.expand(dynprompt,str(unique_id),state)
|
||||
parts=[torch.load(x['path'],map_location='cpu',weights_only=True) for x in segments]
|
||||
shapes={tuple(x.shape[1:]) for x in parts}
|
||||
if len(shapes)!=1:
|
||||
raise ValueError('All segments must decode at the same resolution')
|
||||
return torch.cat(parts),24.,str(path.parent/'manifest.json')
|
||||
|
||||
@staticmethod
|
||||
def expand(dynprompt,end,state):
|
||||
# Same start/end graph-expansion pattern used by BFS Shot Join, adapted
|
||||
# from Ethanfel's MiniMax H3 Context Loop / SxCP loop nodes (GPL-3.0).
|
||||
from comfy_execution.graph_utils import GraphBuilder,is_link
|
||||
feeds={};seen=set()
|
||||
def visit(nid):
|
||||
if nid in seen:return
|
||||
seen.add(nid)
|
||||
for value in dynprompt.get_node(nid).get('inputs',{}).values():
|
||||
if is_link(value):
|
||||
parent=str(value[0]);feeds.setdefault(parent,[]).append(nid);visit(parent)
|
||||
visit(end)
|
||||
starts=[n for n in seen if dynprompt.get_node(n)['class_type']=='BFSH3NativeShotLoopStart']
|
||||
if len(starts)!=1:
|
||||
raise ValueError('Connect exactly one Native Shot Loop Start upstream of this Loop End')
|
||||
start=starts[0];body={start};pending=[start]
|
||||
while pending:
|
||||
for child in feeds.get(pending.pop(),[]):
|
||||
if child not in body:body.add(child);pending.append(child)
|
||||
graph=GraphBuilder();clones={n:graph.node(dynprompt.get_node(n)['class_type'], 'Next' if n==end else n) for n in body}
|
||||
for nid,node in clones.items():
|
||||
node.set_override_display_id(dynprompt.get_display_node_id(nid))
|
||||
for key,value in dynprompt.get_node(nid).get('inputs',{}).items():
|
||||
node.set_input(key,clones[str(value[0])].out(value[1]) if is_link(value) and str(value[0]) in body else value)
|
||||
clones[start].set_input('loop_state',state)
|
||||
return {'result':tuple(clones[end].out(i) for i in range(3)),'expand':graph.finalize()}
|
||||
|
||||
|
||||
CLASSES=(BFSH3NativeShotLoopStart,BFSH3NativeShotContinuity,BFSH3NativeShotLoopEnd)
|
||||
NODE_CLASS_MAPPINGS={c.__name__:c for c in CLASSES}
|
||||
NODE_DISPLAY_NAME_MAPPINGS=dict(zip(NODE_CLASS_MAPPINGS,(
|
||||
'BFS H3 Native Shot Loop Start','BFS H3 Native Shot Continuity (Experimental)','BFS H3 Native Shot Loop End / Join')))
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+1
-1
@@ -1,7 +1,7 @@
|
||||
[project]
|
||||
name = "bfsnodes"
|
||||
description = "Custom ComfyUI nodes for reference-guided video: BFS Sampler (guide-driven IC-LoRAs, auto crop, mask inpainting, temporal chunking), LTX Multiple Controls / Identity Transfer / Edit Anything, Head-Swap and Anime2Real conditioning, MiniMax-H3 latent guides, RoPE layouts and loaders, LoRA Surgery, and Shot Planner / Shot Loop (tabbed panel): split long videos into shots at camera cuts, face-tracked cast, SAM 3 mask per shot with full frame, paste, mask, crop and crop + mask generation modes, optional translucent Colored Mark overlay for manually selected regions, optional Qwen3-VL to suggest settings, describe references ({details}) and the person to replace ({target}), continuity between shots, H3 conditioning with an optional duet and prompt writer, join with the original timing, audio and a side-by-side comparison. H3 Duet: pin a source clip beside the video, training-free."
|
||||
version = "1.115.26"
|
||||
version = "1.115.27"
|
||||
license = {file = "LICENSE"}
|
||||
dependencies = [
|
||||
"torch",
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
import importlib.util
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
spec=importlib.util.spec_from_file_location('native_shot_loop_test',ROOT/'bfs_h3_native_shot_loop.py')
|
||||
m=importlib.util.module_from_spec(spec);spec.loader.exec_module(m)
|
||||
|
||||
|
||||
def test_ranges_respect_camera_cuts_and_cover_every_frame():
|
||||
spans=m.ranges(220,107,'110')
|
||||
assert [(s['start'],s['end'],s['camera_cut']) for s in spans]==[(0,107,False),(107,110,False),(110,217,True),(217,220,False)]
|
||||
assert sum(s['end']-s['start'] for s in spans)==220
|
||||
with pytest.raises(ValueError):m.ranges(220,107,'220')
|
||||
|
||||
|
||||
def test_tail_padding_and_continuity_do_not_renumber_references(tmp_path,monkeypatch):
|
||||
monkeypatch.setitem(sys.modules,'folder_paths',SimpleNamespace(get_temp_directory=lambda:str(tmp_path)))
|
||||
video=torch.zeros(8,8,8,3)
|
||||
start=m.BFSH3NativeShotLoopStart()
|
||||
_,state,*_=start.start(video,chunk_frames=5)
|
||||
previous=torch.ones(5,8,8,3);p=tmp_path/'previous.pt';torch.save(previous,p)
|
||||
state.update(index=1,segments=[{'path':str(p),'frames':5}])
|
||||
tail,state,*_=start.start(video,chunk_frames=5,loop_state=state)
|
||||
assert len(tail)==5
|
||||
refs=(torch.zeros(1,4,8,3),)
|
||||
output,prompt,status=m.BFSH3NativeShotContinuity().prepare(refs,'motion_transfer: test',state)
|
||||
assert output[0] is refs[0] and torch.equal(output[1],previous[-1:])
|
||||
assert '<Picture 2>' in prompt and 'ON' in status
|
||||
state['ranges'][1]['camera_cut']=True
|
||||
output,prompt,status=m.BFSH3NativeShotContinuity().prepare(refs,'original',state)
|
||||
assert output is refs and prompt=='original' and 'camera cut' in status
|
||||
state['ranges'][1]['camera_cut']=False
|
||||
with pytest.raises(ValueError,match='free Picture slot'):
|
||||
m.BFSH3NativeShotContinuity().prepare(refs*9,'original',state)
|
||||
# Join trims the padded tail and preserves exactly the original 8 frames.
|
||||
joined,fps,_=m.BFSH3NativeShotLoopEnd().finish(tail,state)
|
||||
assert len(joined)==8 and fps==24.
|
||||
|
||||
|
||||
def test_graph_expansion_clones_only_loop_body_and_carries_state():
|
||||
# Use the actual ComfyUI graph builder when available; no GPU/model execution.
|
||||
comfy=Path('/data/comfyui')
|
||||
if not (comfy/'comfy_execution/graph_utils.py').exists():pytest.skip('ComfyUI graph utility not available')
|
||||
sys.path.insert(0,str(comfy))
|
||||
graph={'source':{'class_type':'Load','inputs':{}},
|
||||
'model':{'class_type':'Model','inputs':{}},
|
||||
'start':{'class_type':'BFSH3NativeShotLoopStart','inputs':{'images':['source',0]}},
|
||||
'sampler':{'class_type':'Sample','inputs':{'images':['start',0],'model':['model',0]}},
|
||||
'end':{'class_type':'BFSH3NativeShotLoopEnd','inputs':{'images':['sampler',0],'loop_state':['start',1]}}}
|
||||
dyn=SimpleNamespace(get_node=lambda n:graph[n],get_display_node_id=lambda n:n)
|
||||
out=m.BFSH3NativeShotLoopEnd.expand(dyn,'end',{'index':1})
|
||||
expanded=list(out['expand'].values())
|
||||
assert {n['class_type'] for n in expanded}=={'BFSH3NativeShotLoopStart','Sample','BFSH3NativeShotLoopEnd'}
|
||||
start=next(n for n in expanded if n['class_type']=='BFSH3NativeShotLoopStart')
|
||||
assert start['inputs']['images']==['source',0] and start['inputs']['loop_state']=={'index':1}
|
||||
assert next(n for n in expanded if n['class_type']=='Sample')['inputs']['model']==['model',0]
|
||||
|
||||
|
||||
def test_example_ui_links_are_bidirectional_and_match_api_loop():
|
||||
import json
|
||||
d=json.loads((ROOT/'examples/h3_tara_native_auto_prompt_loop.comfy.json').read_text());nodes={n['id']:n for n in d['nodes']}
|
||||
for lid,src,slot,dst,input_slot,kind in d['links']:
|
||||
assert lid in nodes[src]['outputs'][slot]['links']
|
||||
assert nodes[dst]['inputs'][input_slot]['link']==lid
|
||||
api=json.loads((ROOT/'examples/h3_tara_native_auto_prompt_loop.api.json').read_text())
|
||||
assert api['22']['inputs']['images']==['42',0]
|
||||
assert api['22']['inputs']['audio']==['10',2]
|
||||
Reference in New Issue
Block a user