Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a6cae2b9c1 | ||
|
|
e6a9d384b7 | ||
|
|
6d434b246e | ||
|
|
33fd830581 | ||
|
|
22bb235c1d |
@@ -0,0 +1,79 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to **ComfyUI-BFSNodes** are documented in this file.
|
||||
|
||||
## [1.119.3] - 2026-11-09
|
||||
|
||||
### Changed
|
||||
- **BFS Video Segment Picker (`BFSVideoSegmentPicker`) & BFS H3 Character Swap Conditioning (`BFSH3KeyframeConditioning`)**:
|
||||
- Limited selectable subjects and reference pictures to 3 (`1 · red`, `2 · blue`, `3 · green` / `picture_1`..`picture_3`) to match the 3-subject training distribution of the Normal Map and DWPose Character Swap LoRAs.
|
||||
|
||||
---
|
||||
|
||||
## [1.119.2] - 2026-11-09
|
||||
|
||||
### Fixed
|
||||
- **BFS H3 Character Swap Conditioning (`BFSH3KeyframeConditioning`)**:
|
||||
- Added runtime VRAM estimation and peak activation patching (`_patch_vram_estimate`) onto the `MiniMaxH3` model instance so ComfyUI accounts for `minimax_keyframes` (`guide_video`), `minimax_refs` (`Picture 1`–`Picture 4`), and `cross_attn` tokens when computing `minimum_memory_required`, and releases intermediate `q, k, v` and `h` buffers early in `DiTBlock` / `Attention` without modifying ComfyUI core.
|
||||
- Cleared cached DWPose detector/pose sessions (`global_cached_dwpose`) safely in `_free_gpu()` without triggering lazy-import errors on unrelated modules in `sys.modules`.
|
||||
- **BFS Video Segment Picker (`BFSVideoSegmentPicker`)**:
|
||||
- Separated multi-subject DWPose guide rendering into two passes (erase all subject regions to neutral gray first, then draw all subject skeletons on top) so overlapping gray regions of adjacent/hugging subjects never erase each other's skeletons.
|
||||
- Added per-frame DWPose matching exclusion (`exclude`), relaxed overlap fallback (`min_inside=0.25`), and neighbor-frame pose fallback (`±2` frames) so closely interacting subjects do not lose their skeleton on intermediate frames.
|
||||
|
||||
---
|
||||
|
||||
## [1.119.1] - 2026-11-09
|
||||
|
||||
### Fixed
|
||||
- **BFS Video Segment Picker (`BFSVideoSegmentPicker`)**:
|
||||
- Prevented subjects picked on later frames from being falsely propagated or re-anchored onto other people in earlier frames before `first_pick`.
|
||||
- Restricted Molmo2 tracking (`molmo_relocate`) to search strictly forward from `first_pick` (`image[first_pick:]`), highlighting the selected person on the initial frame (`t=0.0s`) and dims background distractions.
|
||||
- Added visual candidate verification (`molmo_verify_candidate`): compares the reference person crop (`Picture 1`) against each candidate person crop (`Picture 2`) with Molmo2, checks RGB/chroma histogram similarity (`person_color_sim`), and rejects candidates that overlap masks already claimed by other subjects (`claimed_at`).
|
||||
- Bounded backward SAM 3 tracking before `first_pick` to the immediately preceding strip frame (`prev_strip`) with frame-to-frame spatial (`IoU >= 0.15`), non-overlap, and color continuity checks.
|
||||
- Enforced `gc.collect()` and `mm.soft_empty_cache(True)` when freeing GPU models after segmentation/tracking.
|
||||
- **BFS H3 Character Swap Conditioning (`BFSH3KeyframeConditioning`)**:
|
||||
- Added `_free_gpu()` to explicitly unload `MiniMaxH3TEModel_` (15 GB) and `MiniMaxH3VideoVAE` (5 GB) and flush the CUDA cache after `MiniMaxH3ReferenceToVideo` and `MiniMaxH3AddGuide`, preventing partial-unload VRAM retention and CUDA OOM when `MiniMaxH3` (20 GB) starts sampling.
|
||||
|
||||
---
|
||||
|
||||
## [1.119.0] - 2026-11-09
|
||||
|
||||
### Added
|
||||
- **BFS Video Segment Picker (`BFSVideoSegmentPicker`)**:
|
||||
- Interactive multi-subject (1–4) SAM 3.1 video segmentation and shot-by-shot propagation with automatic camera cut detection.
|
||||
- Optional Molmo2 re-anchoring across camera cuts and lost stretches, with per-shot and per-subject caching.
|
||||
- Interactive timeline scrubber displaying per-frame subject coverage and lost stretches.
|
||||
- Hybrid guide video output supporting `normal map`, `openpose + face` (DWPose skeleton + face landmarks over neutral gray), and texture-free styles (`flat color`, `shading`, `blur`, `pixelate`, `gray detail`, `edges`).
|
||||
- Configurable `pose_region` (`convex hull` or `person outline`) and `pose_gray_opacity`.
|
||||
- **BFS H3 Character Swap Conditioning (`BFSH3KeyframeConditioning`)**:
|
||||
- Unified conditioning node supporting `keyframe pins`, `normal map guide`, and `DWPose guide` modes with up to 4 reference pictures (`Picture 1`–`Picture 4`) mapped to red, blue, green, and yellow subject marks.
|
||||
- **Keyframe & Loop Helpers**:
|
||||
- Added `BFSKeyframePlanner`, `BFSKeyframeMarks`, `BFSTrackFromFrame`, `BFSWorkflowLoop`, `BFSXYSweep`, and `BFSFrameLoop`.
|
||||
|
||||
---
|
||||
|
||||
## [1.118.1] - 2026-11-07
|
||||
|
||||
### Fixed
|
||||
- **Panel Server Routes**: Wrapped interactive panel button model executions under `torch.inference_mode()` to match standard ComfyUI node execution.
|
||||
|
||||
---
|
||||
|
||||
## [1.118.0] - 2026-11-04
|
||||
|
||||
### Added
|
||||
- **BFS Segment Picker (`BFSSegmentPicker`)**: Added interactive brush strokes, negative/remove points, and eraser mode on the segmentation canvas.
|
||||
|
||||
---
|
||||
|
||||
## [1.117.0] - 2026-11-03
|
||||
|
||||
### Added
|
||||
- **BFS Video Compare (`BFSVideoCompare`)**: Added current-frame PNG download button in the comparison UI and automatic PNG output when comparing single images.
|
||||
|
||||
---
|
||||
|
||||
## [1.116.0] - 2026-11-03
|
||||
|
||||
### Added
|
||||
- **BFS Segment Picker (`BFSSegmentPicker`)**: Interactive SAM 3.1 object picker canvas with automatic concept detection, text search, and point prompts.
|
||||
@@ -272,3 +272,50 @@ paints what to cut out. Each becomes a chip; unselect it to switch it off. Brush
|
||||
eraser strokes belong to the image they were painted on and are dropped when the image
|
||||
changes. Fixed: clicking inside the node no longer shrinks its canvas while the side
|
||||
panel is open.
|
||||
|
||||
### Character swap with marked guides (1.119.0)
|
||||
|
||||
Mark who is replaced in the guide video, then condition MiniMax H3 Character Swap on it.
|
||||
|
||||
**BFS Video Segment Picker (SAM 3)** — the Segment Picker for a video. A strip shows
|
||||
frames of the batch; click a frame, pick the subject tab (1 red, 2 blue, 3 green,
|
||||
4 yellow = Picture 1–4) and click the person. **↔ Propagate** tracks every subject from
|
||||
its picked frames to the whole video with SAM 3, shot by shot (camera cuts are detected
|
||||
and never tracked across). Optional `molmo_clip` (Molmo2 H3 encoder): Molmo2 describes
|
||||
the picked person, follows them through the video and re-anchors shots that have no
|
||||
pick, so a person is found again after a cut. Picking runs only segment the strip;
|
||||
tracking, Molmo2, MoGe and DWPose run on Propagate. Repeated runs reuse what did not
|
||||
change (per subject and per shot). After the first Propagate, picking on a frame
|
||||
propagates by itself (**auto**). The **Whole video** scrubber shows every frame with
|
||||
its masks, the lost stretches in red; **+ add this frame** puts any frame in the strip
|
||||
to fix it. A status line with a progress bar and a log shows what runs.
|
||||
|
||||
Outputs: `masks`, `marked` (tinted), `overlay`, `report`, and `hybrid`: the guide
|
||||
with each subject drawn in its color, by `mark_style`:
|
||||
- `normal map` — its MoGe normal map in gray detail, tinted (connect `normals`:
|
||||
Run MoGe Inference → Render MoGe Geometry, a normal mode);
|
||||
- `openpose + face` — erased to gray (`pose_region`: convex hull or person outline,
|
||||
`pose_gray_opacity`) and drawn as its DWPose skeleton with the eyes and the mouth
|
||||
opening (connect `pose_keypoint`: DWPose Estimator with body, hands and face);
|
||||
- texture-free alternatives: flat color, shading, blur, pixelate, gray detail, edges.
|
||||
|
||||
`hybrid_grow` sets the margin around each subject; `hybrid_base` stacks pickers.
|
||||
`free_after` unloads the models from the GPU when the node is done (default: all).
|
||||
|
||||
**BFS H3 Character Swap Conditioning** — `mode`:
|
||||
- `keyframe pins`: the source video plus one Add Guide per edited keyframe at its own
|
||||
frame (`keyframes` + `keyframe_spec` from BFS Keyframe Planner, or `keyframe_N` /
|
||||
`time_N`). `keyframe_until` drops the keyframes (or every guide) below that sigma on
|
||||
the returned `model`.
|
||||
- `normal map guide` / `DWPose guide`: the Video Segment Picker's `hybrid` as the guide.
|
||||
|
||||
Pictures 1–4 are the references in color order; the prompt is written from the mode
|
||||
and the number of Pictures (or `prompt_override`). Outputs: model, positive, latent,
|
||||
prompt, keyframe_video and `pictures` (the Pictures as a list, e.g. for BFS Video
|
||||
Compare's `reference`).
|
||||
|
||||
Also: **BFS Keyframe Planner** (choose the keyframe frames on the video and export
|
||||
them with their spec), **BFS Keyframe Marks** / **BFS Track From Frame (SAM 3)** (a
|
||||
tracked person's mask at each keyframe), **BFS Workflow Loop** (run a part of the
|
||||
graph once per item, injecting the item and changing widgets per iteration),
|
||||
**BFS XY Sweep** (a grid of any two widgets' values) and **BFS Frame Loop Start/End**.
|
||||
|
||||
+12
@@ -227,6 +227,18 @@ NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
**SEGPICK_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
}
|
||||
|
||||
# Character swap with marked guides (Video Segment Picker -> H3 Character Swap Conditioning), keyframe planning, and
|
||||
# the loop / sweep helpers. Each module loads on its own: a missing optional dependency only skips that module.
|
||||
for _mod in ("bfs_keyframe_marks", "bfs_video_segment_picker", "bfs_h3_swap_conditioning", "bfs_keyframe_planner",
|
||||
"bfs_frame_loop", "bfs_xy_sweep", "bfs_workflow_loop"):
|
||||
try:
|
||||
import importlib as _il
|
||||
_m = _il.import_module(f".{_mod}", __package__)
|
||||
NODE_CLASS_MAPPINGS.update(_m.NODE_CLASS_MAPPINGS)
|
||||
NODE_DISPLAY_NAME_MAPPINGS.update(_m.NODE_DISPLAY_NAME_MAPPINGS)
|
||||
except Exception as _e: # noqa: BLE001
|
||||
print(f"[BFSNodes] {_mod} not loaded: {_e!r}")
|
||||
|
||||
WEB_DIRECTORY = "./web/js"
|
||||
|
||||
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS", "WEB_DIRECTORY"]
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
"""BFS Frame Loop: run any part of a workflow once per frame, one after another, and collect the results.
|
||||
|
||||
Wire a batch of frames (e.g. the Keyframe Planner's source_frames, or BFS Keyframe Marks' marked_frames) into
|
||||
BFS Frame Loop Start; it gives one frame (plus its mask, index and count). Build the per-frame work after it (an image
|
||||
edit: Qwen 2.1 body swap, Klein, ...), and wire its result into BFS Frame Loop End. The End node expands the body
|
||||
between Start and End again for the next frame until all are done, then returns them as one batch (same order).
|
||||
Models stay outside the loop body. Results are kept on disk between iterations; no global state.
|
||||
|
||||
The expansion follows the BFS H3 Native Shot Loop pattern (start/end graph expansion).
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
|
||||
try:
|
||||
import folder_paths
|
||||
except Exception: # tests run without ComfyUI
|
||||
folder_paths = None
|
||||
|
||||
CATEGORY = "BFS/Loop"
|
||||
|
||||
|
||||
class BFSFrameLoopStart:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {"images": ("IMAGE", {"tooltip": "The frames to process one by one."})},
|
||||
"optional": {"masks": ("MASK", {"tooltip": "One mask per frame (optional), handed out with its frame."}),
|
||||
"loop_state": ("BFS_FRAME_LOOP", {"tooltip": "Set by Frame Loop End; leave unconnected."})}}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "MASK", "INT", "INT", "BFS_FRAME_LOOP")
|
||||
RETURN_NAMES = ("image", "mask", "index", "count", "loop_state")
|
||||
OUTPUT_TOOLTIPS = ("The current frame.", "Its mask (zeros when no masks are given).", "Index of the current frame (0-based).",
|
||||
"How many frames the loop runs.", "Connect to BFS Frame Loop End.")
|
||||
FUNCTION = "start"
|
||||
CATEGORY = CATEGORY
|
||||
DESCRIPTION = "Hands out one frame per loop iteration; pair it with BFS Frame Loop End."
|
||||
|
||||
def start(self, images, masks=None, loop_state=None):
|
||||
n = int(images.shape[0])
|
||||
if loop_state is None:
|
||||
temp = folder_paths.get_temp_directory() if folder_paths else "."
|
||||
folder = os.path.join(temp, "bfs_frame_loop", uuid.uuid4().hex[:12])
|
||||
os.makedirs(folder, exist_ok=True)
|
||||
loop_state = {"folder": folder, "index": 0, "count": n, "done": []}
|
||||
state = dict(loop_state)
|
||||
if state["count"] != n:
|
||||
raise ValueError(f"The loop started with {state['count']} frames but now gets {n}.")
|
||||
i = int(state["index"])
|
||||
h, w = images.shape[1:3]
|
||||
if masks is not None:
|
||||
m = masks if masks.ndim == 3 else masks[None]
|
||||
if m.shape[0] != n:
|
||||
raise ValueError(f"{m.shape[0]} masks for {n} frames.")
|
||||
mask = m[i:i + 1]
|
||||
if mask.shape[1:] != (h, w):
|
||||
mask = F.interpolate(mask[:, None].float(), size=(h, w), mode="bilinear", align_corners=False)[:, 0]
|
||||
else:
|
||||
mask = torch.zeros(1, h, w)
|
||||
print(f"[BFS Frame Loop] frame {i + 1}/{n}", flush=True)
|
||||
return images[i:i + 1], mask, i, n, state
|
||||
|
||||
|
||||
class BFSFrameLoopEnd:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {"image": ("IMAGE", {"tooltip": "The processed frame of this iteration."}),
|
||||
"loop_state": ("BFS_FRAME_LOOP", {"tooltip": "From BFS Frame Loop Start."})},
|
||||
"optional": {"resize_to_first": ("BOOLEAN", {"default": True,
|
||||
"tooltip": "Resize every result to the first result's size so they form one batch."})},
|
||||
"hidden": {"dynprompt": "DYNPROMPT", "unique_id": "UNIQUE_ID"}}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "INT")
|
||||
RETURN_NAMES = ("images", "count")
|
||||
OUTPUT_TOOLTIPS = ("All processed frames, in order, as one batch.", "How many frames.")
|
||||
FUNCTION = "finish"
|
||||
CATEGORY = CATEGORY
|
||||
DESCRIPTION = "Collects one processed frame per iteration and expands the loop body until all frames are done."
|
||||
|
||||
def finish(self, image, loop_state, resize_to_first=True, dynprompt=None, unique_id=None):
|
||||
state = dict(loop_state)
|
||||
i = int(state["index"])
|
||||
path = Path(state["folder"]) / f"{i:05d}.pt"
|
||||
tmp = path.with_suffix(".tmp"); torch.save(image[:1].detach().cpu().contiguous(), tmp); tmp.replace(path)
|
||||
state["done"] = list(state["done"]) + [str(path)]
|
||||
state["index"] = i + 1
|
||||
if state["index"] < state["count"]:
|
||||
if dynprompt is None:
|
||||
raise ValueError("BFS Frame Loop needs ComfyUI's dynamic graph execution")
|
||||
return self.expand(dynprompt, str(unique_id), state)
|
||||
parts = [torch.load(p, map_location="cpu", weights_only=True) for p in state["done"]]
|
||||
if resize_to_first:
|
||||
h, w = parts[0].shape[1:3]
|
||||
parts = [p if p.shape[1:3] == (h, w) else
|
||||
F.interpolate(p.movedim(-1, 1), size=(h, w), mode="bilinear", align_corners=False).movedim(1, -1) for p in parts]
|
||||
if len({tuple(p.shape[1:]) for p in parts}) != 1:
|
||||
raise ValueError("The processed frames have different sizes; turn on resize_to_first.")
|
||||
(Path(state["folder"]) / "manifest.json").write_text(json.dumps(state, indent=1))
|
||||
return torch.cat(parts), len(parts)
|
||||
|
||||
@staticmethod
|
||||
def expand(dynprompt, end, state):
|
||||
from comfy_execution.graph_utils import GraphBuilder, is_link
|
||||
feeds, seen = {}, set()
|
||||
|
||||
def visit(nid):
|
||||
if nid in seen:
|
||||
return
|
||||
seen.add(nid)
|
||||
for value in dynprompt.get_node(nid).get("inputs", {}).values():
|
||||
if is_link(value):
|
||||
parent = str(value[0]); feeds.setdefault(parent, []).append(nid); visit(parent)
|
||||
visit(end)
|
||||
starts = [n for n in seen if dynprompt.get_node(n)["class_type"] == "BFSFrameLoopStart"]
|
||||
if len(starts) != 1:
|
||||
raise ValueError("Connect exactly one BFS Frame Loop Start upstream of this Frame Loop End")
|
||||
start = starts[0]; body = {start}; pending = [start]
|
||||
while pending:
|
||||
for child in feeds.get(pending.pop(), []):
|
||||
if child not in body:
|
||||
body.add(child); pending.append(child)
|
||||
graph = GraphBuilder()
|
||||
clones = {n: graph.node(dynprompt.get_node(n)["class_type"], "Next" if n == end else n) for n in body}
|
||||
for nid, node in clones.items():
|
||||
node.set_override_display_id(dynprompt.get_display_node_id(nid))
|
||||
for key, value in dynprompt.get_node(nid).get("inputs", {}).items():
|
||||
node.set_input(key, clones[str(value[0])].out(value[1]) if is_link(value) and str(value[0]) in body else value)
|
||||
clones[start].set_input("loop_state", state)
|
||||
return {"result": tuple(clones[end].out(i) for i in range(2)), "expand": graph.finalize()}
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"BFSFrameLoopStart": BFSFrameLoopStart, "BFSFrameLoopEnd": BFSFrameLoopEnd}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"BFSFrameLoopStart": "BFS Frame Loop Start", "BFSFrameLoopEnd": "BFS Frame Loop End"}
|
||||
@@ -0,0 +1,372 @@
|
||||
"""BFS H3 Character Swap Conditioning: MiniMax H3 Character Swap where the guide says who is replaced.
|
||||
|
||||
Three ways to mark the subjects (`mode`):
|
||||
- "keyframe pins": the source video as an aligned guide (Add Guide at frame 0) plus one Add Guide per edited keyframe
|
||||
at its own frame index (frames of the result with the new people in place, e.g. from BFS Keyframe Planner).
|
||||
- "normal map guide": the guide video with each subject to replace drawn as its normal map in gray detail, tinted with
|
||||
the subject's color (1 red, 2 blue, 3 green = Picture 1..3).
|
||||
- "DWPose guide": the guide video with each subject erased to gray and drawn as its DWPose skeleton (body, hands and
|
||||
face, with the eyes and the mouth opening) in the subject's color.
|
||||
The marked guides come from BFS Video Segment Picker (`hybrid` output). The Pictures are native references and the
|
||||
prompt is written from the mode and the number of Pictures (or prompt_override).
|
||||
"""
|
||||
import json
|
||||
|
||||
import torch
|
||||
|
||||
GRAY = 128.0 / 255.0 # background of the keyframe_video preview
|
||||
MAX_KEYFRAMES = 8
|
||||
MAX_PICTURES = 3
|
||||
PINS, NORMALS, POSE = "keyframe pins", "normal map guide", "DWPose guide"
|
||||
MODES = [PINS, NORMALS, POSE]
|
||||
SUBJECT_COLORS = ["red", "blue", "green"]
|
||||
|
||||
|
||||
def pose_caption(k):
|
||||
"""The DWPose guide prompt (k Pictures)."""
|
||||
C = SUBJECT_COLORS
|
||||
defs = '\n'.join(f'<Subject {j}> is the subject marked by the {C[j - 1]} pose skeleton in the guide video; its appearance comes '
|
||||
f'from <Picture {j}>, its pose, motion, facial expression and mouth movement from the {C[j - 1]} skeleton.'
|
||||
for j in range(1, k + 1))
|
||||
ret = '\n'.join(f'<Subject {j}>: attribute_transfer - keeps the pose, motion, expression and lip movement of the {C[j - 1]} skeleton '
|
||||
f'and takes the face, hair, body shape and whole outfit of <Picture {j}> in every frame.' for j in range(1, k + 1))
|
||||
who = ', '.join(f'the {C[j - 1]} subject replaced by <Picture {j}>' for j in range(1, k + 1))
|
||||
return ('character_swap:\nsubject_definitions:\nThe guide video is the source of the edit. In it, each subject to replace is erased to '
|
||||
'plain gray and shown only as its pose skeleton (body, hands and face, with the eyes and the mouth opening) drawn in one '
|
||||
'color; the mark gives the pose and the expression, not the body shape, and everything else in the guide is the real video.\n'
|
||||
+ defs + f'\nsummary: [video editing + reference generation] The target video is the guide video with {who}, from the first '
|
||||
'frame to the last, following each skeleton\'s pose, expression and mouth. Every other person, the layout, the backgrounds, '
|
||||
'the camera and the motion stay exactly as in the guide video.\nretention_analysis:\n' + ret +
|
||||
'\nEveryone else, the layout and the backgrounds: fully_preserved.')
|
||||
|
||||
|
||||
def normals_caption(k):
|
||||
"""The normal map guide prompt (k Pictures)."""
|
||||
C = SUBJECT_COLORS
|
||||
defs = '\n'.join(f'<Subject {j}> is the subject marked by the {C[j - 1]} normal map in the guide video; its appearance comes '
|
||||
f'from <Picture {j}>, its pose and motion from the {C[j - 1]} mark.' for j in range(1, k + 1))
|
||||
ret = '\n'.join(f'<Subject {j}>: attribute_transfer - keeps the pose and motion of the {C[j - 1]} mark and takes the face, '
|
||||
f'hair, body and whole outfit of <Picture {j}> in every frame.' for j in range(1, k + 1))
|
||||
who = ', '.join(f'the {C[j - 1]} subject replaced by <Picture {j}>' for j in range(1, k + 1))
|
||||
return ('character_swap:\nsubject_definitions:\nThe guide video is the source of the edit. In it, each subject to replace is marked '
|
||||
'with its surface-normal map colorized in one solid color (its 3D surface, without its real colors); the mark says who '
|
||||
'changes, not where the new look must end, and everything else in the guide is the real video.\n' + defs +
|
||||
f'\nsummary: [video editing + reference generation] The target video is the guide video with {who}, from the first frame '
|
||||
'to the last, following each mark\'s pose and motion. Every other person, the layout, the backgrounds, the camera and the '
|
||||
'motion stay exactly as in the guide video.\nretention_analysis:\n' + ret + '\nEveryone else, the layout and the backgrounds: fully_preserved.')
|
||||
|
||||
|
||||
def _tail(k):
|
||||
subs = '\n'.join(f'<Subject {j}> is the person of the guide video who appears with the look of <Picture {j}> in the '
|
||||
f'keyframes; the appearance comes from <Picture {j}>, the motion from the guide video.' for j in range(1, k + 1))
|
||||
ret = '\n'.join(f'<Subject {j}>: attribute_transfer - keeps the motion of the guide video and takes the face, hair, body and '
|
||||
f'whole outfit of <Picture {j}> in every frame.' for j in range(1, k + 1))
|
||||
summary = (f'\nsummary: [video editing + keyframe completion + reference generation] The target video is the guide video with '
|
||||
f'{k} {"people" if k > 1 else "person"} replaced: exactly the people that differ between the keyframes and the guide video, '
|
||||
f'replaced by {", ".join(f"<Picture {j}>" for j in range(1, k + 1))} from the first frame to the last. Every other '
|
||||
'person, the layout, the backgrounds, the camera and the motion stay exactly as in the guide video.\n'
|
||||
'retention_analysis:\n' + ret + '\nEveryone else, the layout and the backgrounds: fully_preserved.')
|
||||
return subs, summary
|
||||
|
||||
|
||||
def pins_caption(k, times):
|
||||
"""The keyframe pins prompt (k Pictures, keyframes at `times` seconds)."""
|
||||
when = ', '.join(f'{t:.1f}s' for t in times); n = len(times)
|
||||
subs, summary = _tail(k)
|
||||
return ('character_swap:\nsubject_definitions:\nThe guide video is the source of the edit. '
|
||||
f'{n} keyframe{"s" if n > 1 else ""} of the target video {"are" if n > 1 else "is"} pinned at {when}; at those times '
|
||||
'the target video matches the keyframes exactly.\n' + subs + summary)
|
||||
|
||||
|
||||
def _fit(image, h, w):
|
||||
"""[1,H,W,C] -> [h,w,3], center crop to the guide's aspect, then resize."""
|
||||
import comfy.utils
|
||||
x = image[:1, ..., :3].movedim(-1, 1)
|
||||
x = comfy.utils.common_upscale(x, w, h, "lanczos", "center")
|
||||
return x.movedim(1, -1)[0].clamp(0, 1)
|
||||
|
||||
|
||||
class BFSH3KeyframeConditioning:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
opt = {f"picture_{j}": ("IMAGE", {"tooltip": f"Subject {j} ({SUBJECT_COLORS[j - 1]} mark in the guide modes)."})
|
||||
for j in range(2, MAX_PICTURES + 1)}
|
||||
opt.update({
|
||||
"model": ("MODEL", {"tooltip": "keyframe pins: connect to get it back with keyframe_until applied. Guide modes: passed through."}),
|
||||
"keyframes": ("IMAGE", {"tooltip": "keyframe pins: edited frames, one per keyframe (BFS Keyframe Planner keyframes)."}),
|
||||
"keyframe_spec": ("STRING", {"forceInput": True, "tooltip": "keyframe pins: BFS Keyframe Planner keyframe_spec (the frame of every keyframe)."}),
|
||||
})
|
||||
for i in range(1, MAX_KEYFRAMES + 1):
|
||||
opt[f"keyframe_{i}"] = ("IMAGE", {"tooltip": f"keyframe pins: edited frame {i}, a full frame of the RESULT (swapped person in place)."})
|
||||
opt[f"time_{i}"] = ("FLOAT", {"default": float(i - 1), "min": 0.0, "max": 600.0, "step": 0.05,
|
||||
"tooltip": f"keyframe pins: seconds into the guide where keyframe {i} sits."})
|
||||
return {"required": {
|
||||
"clip": ("CLIP", {"tooltip": "MiniMax H3's text encoder (Qwen3-VL)."}),
|
||||
"vae": ("VAE",),
|
||||
"guide_video": ("IMAGE", {"tooltip": "keyframe pins: the prepared 24-fps source video. Guide modes: the marked guide "
|
||||
"(BFS Video Segment Picker `hybrid`)."}),
|
||||
"width": ("INT", {"forceInput": True}), "height": ("INT", {"forceInput": True}), "length": ("INT", {"forceInput": True}),
|
||||
"picture_1": ("IMAGE", {"tooltip": "Subject 1 (red mark in the guide modes)."}),
|
||||
"mode": (MODES, {"default": PINS, "tooltip": "How the subjects to replace are marked: edited keyframes pinned at their "
|
||||
"frames, or a guide with each subject drawn as a colored normal map or "
|
||||
"DWPose skeleton (BFS Video Segment Picker)."}),
|
||||
"keyframe_until": ("FLOAT", {"default": 0.7, "min": 0.0, "max": 1.0, "step": 0.01,
|
||||
"tooltip": "keyframe pins only, on the returned model: the guides condition every step whose "
|
||||
"sigma is at least this; below it only the Pictures remain. 0.5 = more faithful to "
|
||||
"the source, 0 = guides in every step. The guide modes keep the guide on every step."}),
|
||||
"keyframe_until_drops": (["keyframes only", "all guides"], {"default": "keyframes only",
|
||||
"tooltip": "keyframe pins only: below keyframe_until drop only the keyframes (the source video "
|
||||
"keeps guiding every step) or every guide."}),
|
||||
"prompt_override": ("STRING", {"multiline": True, "default": "", "tooltip": "Non-empty: used instead of the generated prompt."}),
|
||||
}, "optional": opt}
|
||||
|
||||
RETURN_TYPES = ("MODEL", "CONDITIONING", "LATENT", "STRING", "IMAGE", "IMAGE")
|
||||
RETURN_NAMES = ("model", "positive", "latent", "prompt", "keyframe_video", "pictures")
|
||||
OUTPUT_IS_LIST = (False, False, False, False, False, True)
|
||||
OUTPUT_TOOLTIPS = ("", "", "", "The prompt used.", "keyframe pins: the keyframes at their frames on gray. Guide modes: the guide.",
|
||||
"The Pictures in order (Picture 1..4, any sizes), e.g. for BFS Video Compare's reference.")
|
||||
FUNCTION = "encode"
|
||||
CATEGORY = "BFS/MiniMax H3"
|
||||
DESCRIPTION = ("MiniMax H3 Character Swap conditioning. keyframe pins: the source video plus one Add Guide per edited "
|
||||
"keyframe. normal map guide / DWPose guide: the marked guide from BFS Video Segment Picker, subjects 1-4 = "
|
||||
"red, blue, green, yellow = Picture 1-4. The Pictures as references and the matching prompt.")
|
||||
|
||||
@staticmethod
|
||||
def _collect(kw, fps=24.0):
|
||||
"""[(frame, image[1,H,W,C])] from the Planner batch + spec and/or the keyframe_N / time_N inputs."""
|
||||
out = []
|
||||
batch, spec = kw.get("keyframes"), kw.get("keyframe_spec")
|
||||
if batch is not None:
|
||||
frames = json.loads(spec).get("frames", []) if spec else []
|
||||
if len(frames) != len(batch):
|
||||
raise ValueError(f"{len(batch)} keyframes but keyframe_spec lists {len(frames)} frames: connect the Planner's keyframe_spec.")
|
||||
out += [(int(f), batch[i:i + 1]) for i, f in enumerate(frames)]
|
||||
for i in range(1, MAX_KEYFRAMES + 1):
|
||||
if kw.get(f"keyframe_{i}") is not None:
|
||||
out.append((round(float(kw.get(f"time_{i}", 0.0)) * fps), kw[f"keyframe_{i}"]))
|
||||
return sorted(out, key=lambda x: x[0])
|
||||
|
||||
@staticmethod
|
||||
def _free_gpu():
|
||||
"""Unload TE, VAE, and cached preprocessor models (e.g. DWPose) from GPU so MiniMaxH3 has full VRAM."""
|
||||
try:
|
||||
import gc
|
||||
import sys
|
||||
import comfy.model_management as mm
|
||||
mm.unload_all_models()
|
||||
for k, mod in list(sys.modules.items()):
|
||||
if "dwpose" not in k or mod is None:
|
||||
continue
|
||||
try:
|
||||
dw = getattr(mod, "global_cached_dwpose", None)
|
||||
if dw is not None:
|
||||
dw.det = None
|
||||
dw.pose = None
|
||||
dw.det_filename = None
|
||||
dw.pose_filename = None
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
gc.collect()
|
||||
mm.soft_empty_cache(True)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"[BFS H3 keyframe] could not free GPU models: {e!r}", flush=True)
|
||||
|
||||
@staticmethod
|
||||
def _patch_vram_estimate(model):
|
||||
"""Ensure MiniMaxH3 accounts for guide_video/keyframes/refs in ComfyUI's VRAM estimate so it doesn't OOM."""
|
||||
if model is None:
|
||||
return None
|
||||
inner = getattr(model, "model", None)
|
||||
if inner is None:
|
||||
return model
|
||||
import math
|
||||
import types
|
||||
|
||||
def _extra_conds_shapes(_self, **kwargs):
|
||||
extra = 0
|
||||
payload = kwargs.get("minimax_payload", None)
|
||||
if hasattr(payload, "cond") and isinstance(payload.cond, dict):
|
||||
payload = payload.cond
|
||||
keyframes = kwargs.get("minimax_keyframes", None)
|
||||
if keyframes is None and isinstance(payload, dict):
|
||||
keyframes = payload.get("keyframes", None)
|
||||
refs = kwargs.get("minimax_refs", None)
|
||||
if refs is None and isinstance(payload, dict):
|
||||
refs = payload.get("refs", None)
|
||||
for item in (keyframes or []) + (refs or []):
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
lat = item.get("latent", None)
|
||||
if lat is not None and hasattr(lat, "shape"):
|
||||
extra += math.prod(lat.shape[1:])
|
||||
aud = item.get("audio_latent", None)
|
||||
if aud is not None and hasattr(aud, "shape"):
|
||||
extra += math.prod(aud.shape[1:])
|
||||
cross_attn = kwargs.get("cross_attn", kwargs.get("c_crossattn", None))
|
||||
if hasattr(cross_attn, "cond"):
|
||||
cross_attn = cross_attn.cond
|
||||
if cross_attn is not None and hasattr(cross_attn, "shape") and len(cross_attn.shape) >= 2:
|
||||
extra += int(cross_attn.shape[1]) * 96
|
||||
return {"minimax_cond": [1, 1, extra]} if extra > 0 else {}
|
||||
|
||||
cond_keys = tuple(getattr(inner, "memory_usage_factor_conds", ()) or ())
|
||||
if "minimax_cond" not in cond_keys:
|
||||
inner.memory_usage_factor_conds = cond_keys + ("minimax_cond",)
|
||||
inner.extra_conds_shapes = types.MethodType(_extra_conds_shapes, inner)
|
||||
if getattr(inner, "memory_usage_factor", 0.0) < 0.135:
|
||||
inner.memory_usage_factor = 0.135
|
||||
|
||||
dm = getattr(inner, "diffusion_model", None)
|
||||
if dm is not None and not getattr(dm, "_bfs_low_peak_patched", False):
|
||||
try:
|
||||
import comfy.ldm.minimax.model as mm_mod
|
||||
import comfy.model_management as cmm
|
||||
import comfy.quant_ops
|
||||
from comfy.ldm.modules.attention import AttentionTensorContainer, optimized_attention
|
||||
|
||||
def _attn_forward(self, x, rope_freqs=None, transformer_options={}):
|
||||
s = x.shape[0]
|
||||
dev = x.device
|
||||
q, k, v = self.qkv_proj(x).split(self.heads * self.head_dim, dim=-1)
|
||||
del x
|
||||
v = v.view(s, self.heads, self.head_dim)
|
||||
if rope_freqs is not None:
|
||||
q = q.view(1, s, self.heads, self.head_dim)
|
||||
k = k.view(1, s, self.heads, self.head_dim)
|
||||
qw = cmm.cast_to(self.q_norm.weight, device=dev)
|
||||
kw = cmm.cast_to(self.k_norm.weight, device=dev)
|
||||
rot = rope_freqs.shape[-3] * 2
|
||||
if cmm.in_training:
|
||||
q, k = comfy.quant_ops.ck.rms_rope_split_half(
|
||||
q, k, rope_freqs, qw, kw, epsilon=self.q_norm.eps, rot_dim=rot)
|
||||
else:
|
||||
comfy.quant_ops.ck.rms_rope_split_half_(
|
||||
q, k, rope_freqs, qw, kw, epsilon=self.q_norm.eps, rot_dim=rot)
|
||||
del qw, kw
|
||||
q, k = q[0], k[0]
|
||||
else:
|
||||
q = self.q_norm(q.view(s, self.heads, self.head_dim))
|
||||
k = self.k_norm(k.view(s, self.heads, self.head_dim))
|
||||
q = AttentionTensorContainer(q.transpose(0, 1).unsqueeze(0))
|
||||
k = AttentionTensorContainer(k.transpose(0, 1).unsqueeze(0))
|
||||
v = AttentionTensorContainer(v.transpose(0, 1).unsqueeze(0))
|
||||
out = optimized_attention(
|
||||
q, k, v, self.heads, preferred_attention=self.comfy_attention,
|
||||
mask=None, skip_reshape=True, skip_output_reshape=True,
|
||||
transformer_options=transformer_options,
|
||||
)
|
||||
del q, k, v
|
||||
out = out.transpose(1, 2).reshape(s, self.heads * self.head_dim)
|
||||
return self.out_proj(out)
|
||||
|
||||
def _block_forward(self, x, t_emb, mod_segments, rope_freqs, transformer_options={}, attention=None):
|
||||
attention = self.attn if attention is None else attention
|
||||
shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = self.adaln_proj(t_emb)
|
||||
attn_out = attention(
|
||||
mm_mod._mod_scale_shift(self.norm1(x), shift_msa, scale_msa, mod_segments),
|
||||
rope_freqs=rope_freqs, transformer_options=transformer_options,
|
||||
)
|
||||
del shift_msa, scale_msa
|
||||
x = mm_mod._mod_gate(x, gate_msa, attn_out, mod_segments)
|
||||
del gate_msa, attn_out
|
||||
mlp_out = self.mlp(mm_mod._mod_scale_shift(self.norm2(x), shift_mlp, scale_mlp, mod_segments))
|
||||
del shift_mlp, scale_mlp
|
||||
return mm_mod._mod_gate(x, gate_mlp, mlp_out, mod_segments)
|
||||
|
||||
for blk in getattr(dm, "blocks", ()):
|
||||
blk.forward = types.MethodType(_block_forward, blk)
|
||||
if hasattr(blk, "attn"):
|
||||
blk.attn.forward = types.MethodType(_attn_forward, blk.attn)
|
||||
dm._bfs_low_peak_patched = True
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"[BFS H3 keyframe] could not patch block peak VRAM: {e!r}", flush=True)
|
||||
return model
|
||||
|
||||
def encode(self, clip, vae, guide_video, width, height, length, picture_1, mode=PINS, keyframe_until=0.7,
|
||||
keyframe_until_drops="keyframes only", prompt_override="", model=None, **kw):
|
||||
if mode not in MODES:
|
||||
raise ValueError(f"Unknown mode {mode!r}: use one of {MODES}.")
|
||||
from comfy_extras.nodes_minimax_h3 import MiniMaxH3AddGuide, MiniMaxH3ReferenceToVideo
|
||||
pictures = [picture_1] + [kw[f"picture_{j}"] for j in range(2, MAX_PICTURES + 1) if kw.get(f"picture_{j}") is not None]
|
||||
kfs = self._collect(kw)
|
||||
if mode != PINS:
|
||||
return self._guide(clip, vae, guide_video, width, height, length, pictures, prompt_override, model, kw, pose=mode == POSE)
|
||||
if not kfs:
|
||||
raise ValueError("No keyframes: connect the Planner's keyframes + keyframe_spec, or keyframe_1 / time_1.")
|
||||
if len(guide_video) != length:
|
||||
raise ValueError(f"guide has {len(guide_video)} frames, length is {length}: connect Prepare's length.")
|
||||
h, w = guide_video.shape[1:3]
|
||||
video = torch.full((length, h, w, 3), GRAY, dtype=guide_video.dtype)
|
||||
placed, frames = [], []
|
||||
for f, img in kfs:
|
||||
f = min(length - 1, max(0, int(f)))
|
||||
if f in placed:
|
||||
print(f"[BFS H3 keyframe] two keyframes at frame {f}", flush=True)
|
||||
frame = _fit(img, h, w).to(video.dtype)
|
||||
video[f] = frame
|
||||
placed.append(f); frames.append(frame)
|
||||
times = [f / 24.0 for f in placed]
|
||||
prompt = prompt_override.strip() or pins_caption(len(pictures), times)
|
||||
print("[BFS H3 keyframe] prompt:\n" + prompt, flush=True)
|
||||
try:
|
||||
positive, latent = MiniMaxH3ReferenceToVideo.execute(
|
||||
clip=clip, vae=vae, prompt=prompt, width=width, height=height, length=length, ref_image_size="match",
|
||||
ref_images={f"ref_image_{i}": p for i, p in enumerate(pictures, 1)}).args[:2]
|
||||
self._free_gpu()
|
||||
positive = MiniMaxH3AddGuide.execute(positive=positive, latent=latent, frame_idx=0, vae=vae, image=guide_video).args[0]
|
||||
for f, frame in zip(placed, frames):
|
||||
positive = MiniMaxH3AddGuide.execute(positive=positive, latent=latent, frame_idx=f, vae=vae, image=frame[None]).args[0]
|
||||
finally:
|
||||
self._free_gpu()
|
||||
print(f"[BFS H3 keyframe] {mode}: {len(placed)} keyframes at frames {placed}; {len(pictures)} Pictures", flush=True)
|
||||
return (self._until(model, keyframe_until, keyframe_until_drops.startswith("keyframes")), positive, latent, prompt, video,
|
||||
pictures)
|
||||
|
||||
def _guide(self, clip, vae, guide_video, width, height, length, pictures, prompt_override, model, kw, pose=False):
|
||||
"""Normal map / DWPose guide: the marked guide (subjects drawn as color-coded marks) as the only aligned guide, every
|
||||
step; the Pictures in color order; no keyframes."""
|
||||
from comfy_extras.nodes_minimax_h3 import MiniMaxH3AddGuide, MiniMaxH3ReferenceToVideo
|
||||
if len(guide_video) != length:
|
||||
raise ValueError(f"guide has {len(guide_video)} frames, length is {length}: connect Prepare's length.")
|
||||
prompt = prompt_override.strip() or (pose_caption if pose else normals_caption)(len(pictures))
|
||||
print("[BFS H3 keyframe] guide prompt:\n" + prompt, flush=True)
|
||||
try:
|
||||
positive, latent = MiniMaxH3ReferenceToVideo.execute(
|
||||
clip=clip, vae=vae, prompt=prompt, width=width, height=height, length=length, ref_image_size="match",
|
||||
ref_images={f"ref_image_{i}": p for i, p in enumerate(pictures, 1)}).args[:2]
|
||||
self._free_gpu()
|
||||
positive = MiniMaxH3AddGuide.execute(positive=positive, latent=latent, frame_idx=0, vae=vae, image=guide_video).args[0]
|
||||
finally:
|
||||
self._free_gpu()
|
||||
# the guide carries the background: it stays in every step (keyframe_until does not apply)
|
||||
return self._patch_vram_estimate(model), positive, latent, prompt, guide_video, pictures
|
||||
|
||||
@staticmethod
|
||||
def _until(model, keyframe_until, keep_source=False):
|
||||
"""The model with the guides dropped below sigma keyframe_until (as Viggle Remake's RemakeKeyframeUntil).
|
||||
keep_source: drop only the keyframes; the source video (the first Add Guide, see encode) stays."""
|
||||
if model is None:
|
||||
return None
|
||||
BFSH3KeyframeConditioning._patch_vram_estimate(model)
|
||||
import comfy.patcher_extension
|
||||
until = float(keyframe_until)
|
||||
|
||||
def release(executor, x, timestep, context, transformer_options, minimax_payload=None, **kwargs):
|
||||
p = minimax_payload or {}
|
||||
if p.get("keyframes") and float(timestep.flatten()[0]) / 1000 < until:
|
||||
refs = p.get("refs") or []
|
||||
kept = list(p["keyframes"][:1]) if keep_source else []
|
||||
minimax_payload = dict(p, keyframes=kept or None, layout=None,
|
||||
cond_video_latents=[k["latent"] for k in kept if k.get("latent") is not None] +
|
||||
[r["latent"] for r in refs if "latent" in r],
|
||||
cond_audio_latents=[k["audio_latent"] for k in kept if k.get("audio_latent") is not None] +
|
||||
[r["audio_latent"] for r in refs if r.get("audio_latent") is not None])
|
||||
return executor(x, timestep, context, transformer_options, minimax_payload=minimax_payload, **kwargs)
|
||||
|
||||
m = model.clone()
|
||||
m.add_wrapper_with_key(comfy.patcher_extension.WrappersMP.DIFFUSION_MODEL, "bfs_h3_keyframe_until", release)
|
||||
return m
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"BFSH3KeyframeConditioning": BFSH3KeyframeConditioning}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"BFSH3KeyframeConditioning": "BFS H3 Character Swap Conditioning"}
|
||||
@@ -0,0 +1,151 @@
|
||||
"""BFS Keyframe Marks: the tracked person's mask at every keyframe, and the keyframe frames marked for an image edit.
|
||||
|
||||
Pick the person once (BFS Segment Picker on a guide frame), track it over the whole guide (Run SAM3 Video Track ->
|
||||
SAM3 Track to Mask), then this node takes the mask at each keyframe of a BFS Keyframe Planner keyframe_spec and tints
|
||||
that frame (the Colored Mark convention: magenta at 30 % marks the person an image edit replaces). A keyframe whose mask is empty (the track lost the person, e.g. after a camera cut) is reported, so
|
||||
it can be marked again by hand.
|
||||
"""
|
||||
import json
|
||||
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
|
||||
|
||||
def parse_color(text):
|
||||
s = str(text or "").strip().lstrip("#")
|
||||
if len(s) == 3:
|
||||
s = "".join(ch * 2 for ch in s)
|
||||
try:
|
||||
return int(s[0:2], 16), int(s[2:4], 16), int(s[4:6], 16)
|
||||
except (ValueError, IndexError):
|
||||
return 255, 0, 255
|
||||
|
||||
|
||||
def grow(mask, pixels):
|
||||
"""[N,H,W] mask dilated (pixels > 0) or eroded (pixels < 0) by a square of that radius."""
|
||||
if not pixels:
|
||||
return mask
|
||||
r = abs(int(pixels)); k = 2 * r + 1
|
||||
m = mask[:, None].float()
|
||||
if pixels > 0:
|
||||
return F.max_pool2d(m, k, 1, r)[:, 0]
|
||||
return -F.max_pool2d(-m, k, 1, r)[:, 0]
|
||||
|
||||
|
||||
class BFSKeyframeMarks:
|
||||
DESCRIPTION = __doc__.split("\n\n")[0]
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {
|
||||
"guide_video": ("IMAGE", {"tooltip": "The guide frames the track was made on (same frames as the Planner's guide)."}),
|
||||
"masks": ("MASK", {"tooltip": "The person to replace: one mask per guide frame, or one per keyframe (Segment Picker batch_masks on the Planner's source_frames)."}),
|
||||
"keyframe_spec": ("STRING", {"forceInput": True, "tooltip": "BFS Keyframe Planner keyframe_spec: which frames are keyframes."}),
|
||||
"mark_color": ("STRING", {"default": "#FF00FF", "tooltip": "Tint of the person in marked_frames (hex)."}),
|
||||
"mark_opacity": ("FLOAT", {"default": 0.3, "min": 0.0, "max": 1.0, "step": 0.05,
|
||||
"tooltip": "How strongly the person is tinted (0.3 = the Colored Mark default)."}),
|
||||
"grow_pixels": ("INT", {"default": 0, "min": -64, "max": 64,
|
||||
"tooltip": "Grow (positive) or shrink (negative) each mask before tinting, e.g. to cover hair."}),
|
||||
"threshold": ("FLOAT", {"default": 0.5, "min": 0.0, "max": 1.0, "step": 0.05, "tooltip": "Mask values above this count as the person."}),
|
||||
}}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "MASK", "IMAGE", "STRING")
|
||||
RETURN_NAMES = ("marked_frames", "masks", "source_frames", "report")
|
||||
OUTPUT_TOOLTIPS = ("The keyframe frames with the person tinted, one per keyframe (in keyframe_spec order).",
|
||||
"The person's mask at each keyframe.",
|
||||
"The untouched keyframe frames.",
|
||||
"JSON: per keyframe its frame, time, mask area and whether the mask is empty.")
|
||||
FUNCTION = "mark"
|
||||
CATEGORY = "BFS/MiniMax H3"
|
||||
|
||||
def mark(self, guide_video, masks, keyframe_spec, mark_color, mark_opacity, grow_pixels, threshold):
|
||||
spec = json.loads(keyframe_spec or "{}")
|
||||
frames = [int(f) for f in spec.get("frames", [])]
|
||||
if not frames:
|
||||
raise ValueError("keyframe_spec lists no keyframes: add them in BFS Keyframe Planner.")
|
||||
n, h, w = guide_video.shape[:3]
|
||||
if masks.ndim == 2:
|
||||
masks = masks[None]
|
||||
per_keyframe = masks.shape[0] == len(frames) and masks.shape[0] != n # masks made on the Planner's source_frames
|
||||
if not per_keyframe and masks.shape[0] != n:
|
||||
raise ValueError(f"{masks.shape[0]} masks: give one per guide frame ({n}) or one per keyframe ({len(frames)}).")
|
||||
if masks.shape[1:] != (h, w):
|
||||
masks = F.interpolate(masks[:, None].float(), size=(h, w), mode="bilinear", align_corners=False)[:, 0]
|
||||
color = torch.tensor(parse_color(mark_color), dtype=guide_video.dtype) / 255.0
|
||||
fps = float(spec.get("fps", 24.0))
|
||||
src, out, kept, report = [], [], [], []
|
||||
for i, f in enumerate(frames):
|
||||
f = max(0, min(n - 1, f))
|
||||
j = i if per_keyframe else f
|
||||
m = grow((masks[j:j + 1] > threshold).float(), grow_pixels).clamp(0, 1)
|
||||
img = guide_video[f:f + 1, ..., :3]
|
||||
a = (m[0].to(img.dtype) * float(mark_opacity))[..., None]
|
||||
out.append(img * (1 - a) + color * a)
|
||||
src.append(img); kept.append(m)
|
||||
area = float(m.mean())
|
||||
report.append({"frame": f, "time": round(f / fps, 3), "area": round(area, 4), "empty": area < 1e-4})
|
||||
lost = [r["frame"] for r in report if r["empty"]]
|
||||
if len(lost) == len(report):
|
||||
raise ValueError("No person selected: click the person to replace in the Video Segment Picker, then run again.")
|
||||
if lost:
|
||||
print(f"[BFS Keyframe Marks] no mask at keyframe frame(s) {lost}: mark the person there by hand", flush=True)
|
||||
return torch.cat(out), torch.cat(kept), torch.cat(src), json.dumps({"keyframes": report, "lost": lost})
|
||||
|
||||
|
||||
class BFSTrackFromFrame:
|
||||
"""Track a person over the whole guide from the frame they were picked on (not only frame 0): SAM 3 tracks
|
||||
forward from that frame and backward (the frames before it, reversed), and the two halves are joined. With
|
||||
anchor_frame -1 the anchor is the first keyframe of keyframe_spec, so feed the Planner's source_frames to the
|
||||
Segment Picker and its mask lines up."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {
|
||||
"guide_video": ("IMAGE", {"tooltip": "The guide frames (the same the Planner gets)."}),
|
||||
"sam3_model": ("MODEL", {"tooltip": "SAM 3.1 MODEL (Load Checkpoint with sam3.1_multiplex_fp16.safetensors)."}),
|
||||
"mask": ("MASK", {"tooltip": "The person on the anchor frame (BFS Segment Picker mask)."}),
|
||||
"anchor_frame": ("INT", {"default": -1, "min": -1, "max": 100000,
|
||||
"tooltip": "Guide frame the mask was made on; -1 = the first keyframe of keyframe_spec."}),
|
||||
}, "optional": {
|
||||
"keyframe_spec": ("STRING", {"forceInput": True, "tooltip": "BFS Keyframe Planner keyframe_spec (for anchor_frame -1)."}),
|
||||
}}
|
||||
|
||||
RETURN_TYPES = ("MASK", "INT")
|
||||
RETURN_NAMES = ("masks", "anchor_frame")
|
||||
OUTPUT_TOOLTIPS = ("One mask per guide frame.", "The anchor frame used.")
|
||||
FUNCTION = "track"
|
||||
CATEGORY = "BFS/MiniMax H3"
|
||||
|
||||
@staticmethod
|
||||
def _run(frames, model, mask):
|
||||
from comfy_extras.nodes_sam3 import SAM3_VideoTrack, SAM3_TrackToMask
|
||||
data = SAM3_VideoTrack.execute(images=frames, model=model, initial_mask=mask, max_objects=1).args[0]
|
||||
return SAM3_TrackToMask.execute(track_data=data, object_indices="").args[0]
|
||||
|
||||
def track(self, guide_video, sam3_model, mask, anchor_frame, keyframe_spec=None):
|
||||
n, h, w = guide_video.shape[:3]
|
||||
k = int(anchor_frame)
|
||||
if k < 0:
|
||||
frames = json.loads(keyframe_spec or "{}").get("frames", [])
|
||||
if not frames:
|
||||
raise ValueError("anchor_frame -1 needs a keyframe_spec with at least one keyframe.")
|
||||
k = int(frames[0])
|
||||
k = max(0, min(n - 1, k))
|
||||
m = mask[:1] if mask.ndim == 3 else mask[None]
|
||||
if m.shape[1:] != (h, w):
|
||||
m = F.interpolate(m[:, None].float(), size=(h, w), mode="bilinear", align_corners=False)[:, 0]
|
||||
m = (m > 0.5).float()
|
||||
fwd = self._run(guide_video[k:], sam3_model, m) # frames k .. n-1
|
||||
parts = [fwd]
|
||||
if k > 0:
|
||||
bwd = self._run(guide_video[:k + 1].flip(0), sam3_model, m) # frames k .. 0
|
||||
parts = [bwd.flip(0)[:k], fwd]
|
||||
out = torch.cat([p.to(guide_video.device, torch.float32) for p in parts])
|
||||
if out.shape[1:] != (h, w):
|
||||
out = F.interpolate(out[:, None], size=(h, w), mode="bilinear", align_corners=False)[:, 0]
|
||||
print(f"[BFS Track From Frame] anchor frame {k}: tracked {k} frames back, {n - k} forward", flush=True)
|
||||
return out, k
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"BFSKeyframeMarks": BFSKeyframeMarks, "BFSTrackFromFrame": BFSTrackFromFrame}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"BFSKeyframeMarks": "BFS Keyframe Marks", "BFSTrackFromFrame": "BFS Track From Frame (SAM 3)"}
|
||||
@@ -0,0 +1,148 @@
|
||||
"""BFS Keyframe Planner: pick keyframe times on the guide video, drop in edited frames and reference pictures.
|
||||
|
||||
The node shows the guide on a small player with a timeline (frames come from its last run: queue it once to load
|
||||
them). "+ Keyframe" marks the current frame; each keyframe card holds the source frame and, optionally, the edited
|
||||
frame for that time (drop an image on the card or upload it). Reference pictures (up to 4) are dropped or uploaded in
|
||||
the node too, or connected as inputs.
|
||||
|
||||
Two ways to use it:
|
||||
- with edited frames: wire keyframes + keyframe_spec + pictures into BFS H3 Keyframe Swap Conditioning;
|
||||
- without: use it to choose the times; source_frames are the original frames there, ready for an image edit
|
||||
(e.g. Segment Picker + the Qwen 2.1 magenta body swap), and the edits come back into the conditioning node.
|
||||
|
||||
All state lives in the hidden `plan` STRING widget (JSON), saved with the workflow:
|
||||
{"keyframes": [{"frame": 53, "edit": "bfs_keyframes/abc.png"}, {"frame": 96, "edit": null}],
|
||||
"pictures": ["bfs_keyframes/ref1.png"]}
|
||||
Image paths are relative to ComfyUI's input folder (the panel uploads there).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from PIL import Image
|
||||
|
||||
try:
|
||||
import folder_paths
|
||||
except Exception: # tests run without ComfyUI
|
||||
folder_paths = None
|
||||
|
||||
MAX_PICTURES = 4
|
||||
STRIP_MAX_FRAMES = 480 # the panel's filmstrip: at most this many frames (evenly spread over longer guides)
|
||||
STRIP_WIDTH = 384
|
||||
|
||||
|
||||
def parse_plan(text):
|
||||
try:
|
||||
p = json.loads(text or "{}") or {}
|
||||
except Exception:
|
||||
p = {}
|
||||
kfs = []
|
||||
for k in p.get("keyframes", []) if isinstance(p.get("keyframes"), list) else []:
|
||||
try:
|
||||
kfs.append({"frame": int(k["frame"]), "edit": k.get("edit") or None})
|
||||
except Exception:
|
||||
continue
|
||||
pics = [x for x in p.get("pictures", []) if isinstance(x, str) and x] if isinstance(p.get("pictures"), list) else []
|
||||
return {"keyframes": kfs, "pictures": pics[:MAX_PICTURES]}
|
||||
|
||||
|
||||
def _input_path(rel):
|
||||
base = folder_paths.get_input_directory() if folder_paths else "."
|
||||
path = os.path.normpath(os.path.join(base, rel))
|
||||
if not path.startswith(os.path.normpath(base) + os.sep):
|
||||
raise ValueError(f"image path outside the input folder: {rel}")
|
||||
if not os.path.isfile(path):
|
||||
raise FileNotFoundError(f"keyframe planner image not found in input/: {rel}")
|
||||
return path
|
||||
|
||||
|
||||
def load_image(rel):
|
||||
im = Image.open(_input_path(rel)).convert("RGB")
|
||||
return torch.from_numpy(np.asarray(im, np.float32) / 255.0)[None]
|
||||
|
||||
|
||||
def fit(image, h, w):
|
||||
"""[1,H,W,C] -> [1,h,w,3]: center crop to the guide's aspect, then resize."""
|
||||
import comfy.utils
|
||||
x = image[:1, ..., :3].movedim(-1, 1)
|
||||
return comfy.utils.common_upscale(x, w, h, "lanczos", "center").movedim(1, -1).clamp(0, 1)
|
||||
|
||||
|
||||
def filmstrip(video, fps):
|
||||
"""Small JPEGs of the guide in the temp folder, for the panel. Same video -> same files (hashed)."""
|
||||
n = video.shape[0]
|
||||
idx = np.unique(np.linspace(0, n - 1, min(n, STRIP_MAX_FRAMES)).round().astype(int))
|
||||
h, w = video.shape[1:3]
|
||||
sw = min(STRIP_WIDTH, w); sh = max(1, round(h * sw / w))
|
||||
sample = video[:: max(1, n // 16), ::16, ::16, :3].float().cpu().numpy()
|
||||
key = hashlib.sha1(np.ascontiguousarray(sample).tobytes() + f"{n}x{h}x{w}".encode()).hexdigest()[:16]
|
||||
temp = folder_paths.get_temp_directory() if folder_paths else "."
|
||||
sub = os.path.join("bfs_kfplan", key); out = os.path.join(temp, sub); os.makedirs(out, exist_ok=True)
|
||||
files = []
|
||||
for i in idx:
|
||||
name = f"f{i:05d}.jpg"; path = os.path.join(out, name)
|
||||
if not os.path.exists(path):
|
||||
arr = (video[int(i), ..., :3].float().cpu().numpy() * 255).clip(0, 255).astype(np.uint8)
|
||||
Image.fromarray(arr).resize((sw, sh), Image.BILINEAR).save(path, quality=82)
|
||||
files.append(name)
|
||||
return {"subfolder": sub, "type": "temp", "files": files, "frames": [int(i) for i in idx],
|
||||
"count": int(n), "fps": float(fps), "width": int(w), "height": int(h)}
|
||||
|
||||
|
||||
class BFSKeyframePlanner:
|
||||
DESCRIPTION = __doc__.split("\n\n")[0]
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
opt = {f"picture_{j}": ("IMAGE", {"tooltip": f"Reference picture {j}; overrides the one dropped in the panel."})
|
||||
for j in range(1, MAX_PICTURES + 1)}
|
||||
return {
|
||||
"required": {
|
||||
"guide_video": ("IMAGE", {"tooltip": "The guide frames (e.g. BFS H3 Prepare Native Video)."}),
|
||||
"fps": ("FLOAT", {"default": 24.0, "min": 1.0, "max": 120.0, "step": 0.5}),
|
||||
"plan": ("STRING", {"default": "{}", "multiline": True, "tooltip": "Set by the panel: keyframe frames, edited frames and pictures."}),
|
||||
},
|
||||
"optional": opt,
|
||||
"hidden": {"unique_id": "UNIQUE_ID"},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "IMAGE", "IMAGE", "STRING") + ("IMAGE",) * MAX_PICTURES
|
||||
RETURN_NAMES = ("guide_video", "source_frames", "keyframes", "keyframe_spec") + tuple(f"picture_{j}" for j in range(1, MAX_PICTURES + 1))
|
||||
OUTPUT_TOOLTIPS = ("The guide, unchanged.",
|
||||
"The original guide frames at the keyframe times (one per keyframe), e.g. to edit them.",
|
||||
"The edited frame of every keyframe, at the guide size; a keyframe without one gives its source frame.",
|
||||
"JSON: frames, times (s), fps and which keyframes have an edited frame.") + \
|
||||
tuple(f"Reference picture {j} (None when not set)." for j in range(1, MAX_PICTURES + 1))
|
||||
FUNCTION = "plan_keyframes"
|
||||
CATEGORY = "BFS/MiniMax H3"
|
||||
OUTPUT_NODE = True
|
||||
|
||||
def plan_keyframes(self, guide_video, fps, plan, unique_id=None, **kw):
|
||||
n, h, w = guide_video.shape[0], guide_video.shape[1], guide_video.shape[2]
|
||||
state = parse_plan(plan)
|
||||
ui = {"bfs_kfplan": [filmstrip(guide_video, fps)]}
|
||||
kfs = sorted({max(0, min(n - 1, k["frame"])): k for k in state["keyframes"]}.items())
|
||||
src, edit, spec = [], [], {"frames": [], "times": [], "edited": [], "fps": float(fps), "length": int(n)}
|
||||
for f, k in kfs:
|
||||
frame = guide_video[f:f + 1, ..., :3]
|
||||
src.append(frame)
|
||||
edit.append(fit(load_image(k["edit"]), h, w).to(frame) if k["edit"] else frame)
|
||||
spec["frames"].append(int(f)); spec["times"].append(round(f / float(fps), 4)); spec["edited"].append(bool(k["edit"]))
|
||||
if not kfs:
|
||||
print("[BFS Keyframe Planner] no keyframes yet: add them in the panel (outputs hold the first frame)", flush=True)
|
||||
src = edit = [guide_video[:1, ..., :3]]
|
||||
pictures = []
|
||||
for j in range(1, MAX_PICTURES + 1):
|
||||
p = kw.get(f"picture_{j}")
|
||||
if p is None and j <= len(state["pictures"]):
|
||||
p = load_image(state["pictures"][j - 1])
|
||||
pictures.append(p)
|
||||
return {"ui": ui, "result": (guide_video, torch.cat(src), torch.cat(edit), json.dumps(spec), *pictures)}
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"BFSKeyframePlanner": BFSKeyframePlanner}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"BFSKeyframePlanner": "BFS Keyframe Planner"}
|
||||
@@ -0,0 +1,984 @@
|
||||
"""BFS Video Segment Picker (SAM 3): pick who to replace in a video or in a set of keyframes.
|
||||
|
||||
The image Segment Picker shows one image; this one shows every frame of the batch (the Keyframe Planner's
|
||||
source_frames, or a short clip) as a strip: click a frame, then click the person on it, exactly like the image picker.
|
||||
A long video shows an evenly spread strip of up to 32 frames. Select on one frame or on several (e.g. one per camera shot); frames without a selection take the mask of the nearest
|
||||
selected frame, tracked with SAM 3 from there (forward and backward). One run segments every frame, so you can pick on
|
||||
any of them without running again.
|
||||
|
||||
All state lives in the hidden `selection` STRING widget (JSON): one image-picker selection per frame,
|
||||
{"frames": {"0": {"selected": ["a2"], "added": [...], "next": 3}, "4": {...}}}
|
||||
The image Segment Picker node and its workflows are untouched; its helpers are reused.
|
||||
"""
|
||||
import json
|
||||
import time
|
||||
from collections import OrderedDict
|
||||
|
||||
import torch
|
||||
|
||||
from . import bfs_segment_picker as sp
|
||||
from .bfs_keyframe_marks import BFSTrackFromFrame
|
||||
|
||||
MAX_FRAMES = 32 # frames segmented for the panel; a longer video shows an evenly spread strip
|
||||
# the hybrid guide: subject k drawn without its texture (e.g. its normal map in gray detail), tinted with its color
|
||||
HYBRID_COLORS = [(235, 40, 40), (35, 110, 235), (60, 205, 60)] # RGB: red, blue, green
|
||||
HYBRID_TINT = 0.75
|
||||
|
||||
|
||||
def hybrid_tint(normal_rgb, color, alpha=HYBRID_TINT):
|
||||
"""normal_rgb [H,W,3] in 0..1 (MoGe OpenCV normals * 0.5 + 0.5) -> gray detail tinted with the subject color, 0..1."""
|
||||
gray = normal_rgb[..., 0] * 0.299 + normal_rgb[..., 1] * 0.587 + normal_rgb[..., 2] * 0.114
|
||||
gray = (gray - gray.min()) / (gray.max() - gray.min() + 1e-6)
|
||||
detail = (0.25 + 0.75 * gray)[..., None]
|
||||
c = torch.tensor(color, dtype=normal_rgb.dtype, device=normal_rgb.device) / 255.0
|
||||
return (detail * c * alpha + detail * (1 - alpha)).clamp(0, 1)
|
||||
|
||||
|
||||
FREE_MODES = ["unload all models from the GPU", "unload SAM 3 from the GPU", "unload all models + clear the picker caches",
|
||||
"keep everything loaded"]
|
||||
LEGACY_FREE = {"unload SAM 3 + clear the picker caches": "unload all models + clear the picker caches"}
|
||||
MARK_STYLES = ["normal map (needs normals)", "openpose + face (needs pose_keypoint)", "flat color", "shading (needs normals)", "blur",
|
||||
"pixelate", "gray detail", "edges"]
|
||||
|
||||
|
||||
def mark_fill(style, frame, normal_rgb, mask, color, alpha=HYBRID_TINT):
|
||||
"""The subject's region drawn without its texture, tinted with the subject's color. frame/normal_rgb [H,W,3] 0..1.
|
||||
'normal map' and 'openpose + face' are the guide modes of BFS H3 Character Swap Conditioning; the others are
|
||||
texture-free alternatives (no MoGe needed for most)."""
|
||||
import torch.nn.functional as F
|
||||
c = torch.tensor(color, dtype=frame.dtype, device=frame.device) / 255.0
|
||||
if style.startswith("normal"):
|
||||
return hybrid_tint(normal_rgb, color, alpha)
|
||||
if style.startswith("shading"):
|
||||
n = normal_rgb * 2 - 1
|
||||
light = n.new_tensor([-0.35, -0.5, 0.8]); light = light / light.norm()
|
||||
lum = (0.30 + 0.70 * (n * light).sum(-1).clamp(0, 1))[..., None]
|
||||
return (lum * c).clamp(0, 1)
|
||||
if style == "flat color":
|
||||
return c.expand_as(frame).clone()
|
||||
gray = (frame[..., 0] * 0.299 + frame[..., 1] * 0.587 + frame[..., 2] * 0.114)
|
||||
ys, xs = torch.nonzero(mask, as_tuple=True)
|
||||
size = float(((ys.max() - ys.min() + 1) * (xs.max() - xs.min() + 1)) ** 0.5) if len(ys) else 64.0
|
||||
if style == "blur":
|
||||
k = max(3, int(size * 0.12) | 1)
|
||||
g = F.avg_pool2d(F.avg_pool2d(gray[None, None], k, 1, k // 2), k, 1, k // 2)[0, 0]
|
||||
detail = g
|
||||
elif style == "pixelate":
|
||||
b = max(2, int(size / 12))
|
||||
h, w = gray.shape
|
||||
small = F.interpolate(gray[None, None], size=(max(1, h // b), max(1, w // b)), mode="area")
|
||||
detail = F.interpolate(small, size=(h, w), mode="nearest")[0, 0]
|
||||
elif style == "edges":
|
||||
gx = gray[:, 2:] - gray[:, :-2]; gy = gray[2:, :] - gray[:-2, :]
|
||||
e = torch.zeros_like(gray); e[1:-1, 1:-1] = (gx[1:-1] ** 2 + gy[:, 1:-1] ** 2).sqrt()
|
||||
detail = 1 - (e / (e.max() + 1e-6)).clamp(0, 1) * 0.9
|
||||
else: # gray detail: the original's own luminance, posterized to 4 levels
|
||||
detail = (gray * 4).floor().clamp(0, 3) / 3
|
||||
m = mask.bool()
|
||||
if m.any():
|
||||
lo, hi = detail[m].min(), detail[m].max()
|
||||
detail = (detail - lo) / (hi - lo + 1e-6)
|
||||
detail = (0.25 + 0.75 * detail.clamp(0, 1))[..., None]
|
||||
return (detail * c * alpha + detail * (1 - alpha)).clamp(0, 1)
|
||||
|
||||
|
||||
DESCRIBE_SOURCE = ('Describe the foreground person or costumed character in this image as ONE short English noun '
|
||||
'phrase of 6 to 18 words that uniquely identifies their visual appearance in a video: outfit or '
|
||||
'costume colors and pieces, mask/helmet/glasses if any, hair, and build. '
|
||||
'No leading article, no background, no pose, no full sentence.')
|
||||
VERIFY_SAME_PERSON = ('Picture 1 shows the reference person selected by the user. '
|
||||
'Picture 2 shows a candidate person from another frame of the video. '
|
||||
'Do Picture 1 and Picture 2 show the exact same individual wearing the exact same outfit or '
|
||||
'costume? Answer with a single word: yes or no.')
|
||||
|
||||
|
||||
def detect_cuts(image, thresh=0.22):
|
||||
"""Camera cuts: frame indices where a new shot starts (mean abs diff of small gray frames, plus a histogram check)."""
|
||||
import torch.nn.functional as F
|
||||
g = image[..., :3].float().mean(-1)[:, None]
|
||||
small = F.interpolate(g, size=(36, 64), mode="area")[:, 0]
|
||||
cuts = []
|
||||
for i in range(1, small.shape[0]):
|
||||
d = (small[i] - small[i - 1]).abs().mean().item()
|
||||
h0 = torch.histc(small[i - 1], 16, 0, 1); h1 = torch.histc(small[i], 16, 0, 1)
|
||||
hd = (h0 / h0.sum() - h1 / h1.sum()).abs().sum().item() / 2
|
||||
if d > thresh * 0.6 and hd > 0.25 or d > thresh:
|
||||
cuts.append(i)
|
||||
return cuts
|
||||
|
||||
|
||||
def shots_of(n, cuts):
|
||||
edges = [0] + [c for c in cuts if 0 < c < n] + [n]
|
||||
return [(edges[k], edges[k + 1]) for k in range(len(edges) - 1)]
|
||||
|
||||
|
||||
def person_at(sam, frame, x, y, threshold):
|
||||
"""The whole person under Molmo's point: SAM 3 'person' instances, the one containing the point (else the nearest);
|
||||
falls back to a point prompt."""
|
||||
h, w = frame.shape[1:3]
|
||||
px, py = min(w - 1, int(x * w)), min(h - 1, int(y * h))
|
||||
inst = [m for m, _ in sp.detect_text(sam, frame, "person", threshold, 12)]
|
||||
best, bd = None, (0.06 * max(h, w)) ** 2
|
||||
for m in inst:
|
||||
if m[py, px]:
|
||||
return m
|
||||
ys, xs = torch.nonzero(m, as_tuple=True)
|
||||
if len(xs):
|
||||
d = float(((xs - px) ** 2 + (ys - py) ** 2).min())
|
||||
if d < bd: best, bd = m, d
|
||||
if best is not None:
|
||||
return best
|
||||
return sp.detect_points(sam, frame, [{"x": x, "y": y, "label": 1}], threshold)
|
||||
|
||||
|
||||
_MOLMO_CACHE = OrderedDict() # (video, subject, anchor frame, anchor mask) -> (description, points): Molmo2 runs once per pick
|
||||
_SHOT_CACHE = OrderedDict() # (video, subject, shot, its anchors + masks, threshold) -> (masks, report) of that shot
|
||||
|
||||
|
||||
def mask_key(m):
|
||||
import hashlib
|
||||
import numpy as np
|
||||
return hashlib.sha1(np.packbits((m > 0.5).cpu().numpy()).tobytes()).hexdigest()[:16]
|
||||
|
||||
|
||||
def _remember(cache, key, value, cap):
|
||||
cache[key] = value
|
||||
cache.move_to_end(key)
|
||||
while len(cache) > cap:
|
||||
cache.popitem(last=False)
|
||||
|
||||
|
||||
def person_crop(frame, mask):
|
||||
"""Crop [H,W,3] around mask > 0.5 with a small context margin and non-mask pixels dimmed toward neutral gray."""
|
||||
m = mask > 0.5
|
||||
ys, xs = torch.nonzero(m, as_tuple=True)
|
||||
if not len(ys):
|
||||
return None
|
||||
h, w = frame.shape[:2]
|
||||
y0, y1, x0, x1 = int(ys.min()), int(ys.max()) + 1, int(xs.min()), int(xs.max()) + 1
|
||||
py = max(4, int((y1 - y0) * 0.08)); px = max(4, int((x1 - x0) * 0.08))
|
||||
y0, y1 = max(0, y0 - py), min(h, y1 + py)
|
||||
x0, x1 = max(0, x0 - px), min(w, x1 + px)
|
||||
crop = frame[y0:y1, x0:x1, :3].float().clone()
|
||||
cm = m[y0:y1, x0:x1]
|
||||
crop[~cm] = crop[~cm] * 0.25 + 0.375
|
||||
return crop
|
||||
|
||||
|
||||
def highlight_person_frame(frame, mask):
|
||||
"""Full [H,W,3] frame with the picked person in full color and everything else dimmed toward neutral gray."""
|
||||
m = mask > 0.5
|
||||
out = frame[..., :3].float() * 0.20 + 0.40
|
||||
out[m] = frame[..., :3].float()[m]
|
||||
return out
|
||||
|
||||
|
||||
def person_color_sim(frame_a, mask_a, frame_b, mask_b):
|
||||
"""Histogram intersection similarity (0..1) of RGB + chroma inside two person masks."""
|
||||
ma, mb = mask_a > 0.5, mask_b > 0.5
|
||||
if int(ma.sum()) < 16 or int(mb.sum()) < 16:
|
||||
return 0.0
|
||||
pa = frame_a[..., :3].float()[ma].clamp(0, 1)
|
||||
pb = frame_b[..., :3].float()[mb].clamp(0, 1)
|
||||
sims = []
|
||||
for c in range(3):
|
||||
ha = torch.histc(pa[:, c], 16, 0.0, 1.0); hb = torch.histc(pb[:, c], 16, 0.0, 1.0)
|
||||
sims.append(float(torch.minimum(ha / (ha.sum() + 1e-6), hb / (hb.sum() + 1e-6)).sum()))
|
||||
ca = pa.max(-1).values - pa.min(-1).values
|
||||
cb = pb.max(-1).values - pb.min(-1).values
|
||||
ha = torch.histc(ca, 12, 0.0, 1.0); hb = torch.histc(cb, 12, 0.0, 1.0)
|
||||
sims.append(float(torch.minimum(ha / (ha.sum() + 1e-6), hb / (hb.sum() + 1e-6)).sum()))
|
||||
return sum(sims) / len(sims)
|
||||
|
||||
|
||||
def mask_overlap(mask_a, mask_b):
|
||||
"""(IoU, containment_in_smaller) between two 2D masks."""
|
||||
a, b = mask_a > 0.5, mask_b > 0.5
|
||||
inter = float((a & b).sum())
|
||||
if inter <= 0:
|
||||
return 0.0, 0.0
|
||||
union = float((a | b).sum())
|
||||
small = float(min(a.sum(), b.sum()))
|
||||
return inter / max(1.0, union), inter / max(1.0, small)
|
||||
|
||||
|
||||
def molmo_verify_candidate(molmo, ref_crop, ref_frame, ref_mask, cand_frame, cand_mask, claimed_masks=None, memo=None):
|
||||
"""Verify a candidate re-anchor mask against the user's clicked reference person using:
|
||||
1) non-overlap with masks already claimed by other subjects on that frame,
|
||||
2) RGB + chroma appearance similarity, and
|
||||
3) Molmo2 visual comparison of [Picture 1: clicked person crop, Picture 2: candidate person crop]."""
|
||||
if not bool((cand_mask > 0.5).any()):
|
||||
return False
|
||||
for cm in (claimed_masks or []):
|
||||
iou, contain = mask_overlap(cand_mask, cm)
|
||||
if iou > 0.40 or contain > 0.55:
|
||||
return False
|
||||
sim = person_color_sim(ref_frame, ref_mask, cand_frame, cand_mask)
|
||||
if sim < 0.38:
|
||||
return False
|
||||
if molmo is None or not hasattr(molmo, "runtime") or ref_crop is None:
|
||||
return sim >= 0.55
|
||||
cand_crop = person_crop(cand_frame, cand_mask)
|
||||
if cand_crop is None:
|
||||
return False
|
||||
vkey = memo and ("verify_v2",) + tuple(memo) + (mask_key(ref_mask), mask_key(cand_mask))
|
||||
if vkey and vkey in _MOLMO_CACHE:
|
||||
ok = _MOLMO_CACHE[vkey]
|
||||
_MOLMO_CACHE.move_to_end(vkey)
|
||||
return bool(ok)
|
||||
text = molmo.runtime.generate(
|
||||
VERIFY_SAME_PERSON,
|
||||
[{"type": "image", "data": ref_crop[None]}, {"type": "image", "data": cand_crop[None]}],
|
||||
16, molmo.phase_b_max_frames, do_sample=False, repetition_penalty=1.0
|
||||
)["text"]
|
||||
ans = str(text or "").strip().lower()
|
||||
ok = ans.startswith("yes") or ("yes" in ans.split()[:2] and "no" not in ans.split()[:2])
|
||||
if vkey:
|
||||
_remember(_MOLMO_CACHE, vkey, bool(ok), 64)
|
||||
return bool(ok)
|
||||
|
||||
|
||||
def molmo_relocate(molmo, sam, image, anchor_mask, anchor, shots, has_anchor, threshold, fps=24.0, memo=None, claimed_by_frame=None):
|
||||
"""Find the subject in later shots that have no anchor yet: Molmo2 describes the picked person (masked crop), tracks
|
||||
forward from the picked frame (with the picked person visually highlighted in the first frame of the clip), and in
|
||||
each later shot without an anchor verifies the candidate person crop against the clicked reference crop with Molmo2.
|
||||
Returns {frame: mask}, report, pts."""
|
||||
import re
|
||||
m = anchor_mask > 0.5
|
||||
ys, xs = torch.nonzero(m, as_tuple=True)
|
||||
if not len(ys):
|
||||
return {}, "no anchor mask", []
|
||||
n, h, w = image.shape[:3]
|
||||
crop = person_crop(image[anchor, ..., :3], m)
|
||||
cx = float(xs.float().mean()) / max(1, w) * 100.0
|
||||
cy = float(ys.float().mean()) / max(1, h) * 100.0
|
||||
key = memo and (("relocate_v2",) + tuple(memo) + (anchor, mask_key(anchor_mask)))
|
||||
reused = key in _MOLMO_CACHE if key else False
|
||||
if reused: # same first pick of this subject: the description and Molmo2's forward track are the same
|
||||
desc, pts = _MOLMO_CACHE[key]
|
||||
_MOLMO_CACHE.move_to_end(key)
|
||||
else:
|
||||
text = molmo.runtime.generate(DESCRIBE_SOURCE, [{"type": "image", "data": crop[None]}], 96, molmo.phase_b_max_frames,
|
||||
do_sample=False, repetition_penalty=1.0)["text"]
|
||||
desc = re.sub(r"^(the|a|an)\s+", "", " ".join(str(text).strip().strip('"').split()).rstrip(". "), flags=re.I)
|
||||
if anchor < n - 1:
|
||||
sub = image[anchor:, ..., :3].clone()
|
||||
sub[0] = highlight_person_frame(image[anchor, ..., :3], m).to(sub.dtype)
|
||||
query = f"Track the {desc} (the person highlighted in color at ({cx:.0f}%, {cy:.0f}%) in the first frame)."
|
||||
ans = molmo.runtime.track_subjects([query], sub, fps, max_fps=4.0)[0]
|
||||
t_off = float(anchor) / float(fps)
|
||||
pts = sorted((float(t) + t_off, int(o), float(x), float(y))
|
||||
for t, o, x, y in ans.get("points", [])
|
||||
if int(round((float(t) + t_off) * fps)) > anchor)
|
||||
else:
|
||||
pts = []
|
||||
if key:
|
||||
_remember(_MOLMO_CACHE, key, (desc, pts), 16)
|
||||
found = {}
|
||||
for si, (s0, s1) in enumerate(shots):
|
||||
if has_anchor[si] or s1 <= anchor:
|
||||
continue
|
||||
lo = max(s0, anchor + 1)
|
||||
inside = [p for p in pts if lo <= int(round(float(p[0]) * fps)) < s1]
|
||||
if not inside:
|
||||
continue
|
||||
mid = (lo + s1 - 1) / 2.0 # try points nearest the shot's middle first: steadiest framing
|
||||
seen_f = set()
|
||||
for t, _o, x, y in sorted(inside, key=lambda p: abs(float(p[0]) * fps - mid)):
|
||||
f = min(s1 - 1, max(lo, int(round(float(t) * fps))))
|
||||
if f in seen_f:
|
||||
continue
|
||||
seen_f.add(f)
|
||||
mk = person_at(sam, image[f:f + 1, ..., :3].float().cpu(), x, y, threshold)
|
||||
if bool(mk.any()) and molmo_verify_candidate(
|
||||
molmo, crop, image[anchor, ..., :3], m, image[f, ..., :3], mk,
|
||||
claimed_masks=(claimed_by_frame or {}).get(f), memo=(memo or ()) + (anchor, f)
|
||||
):
|
||||
found[f] = mk.float()
|
||||
break
|
||||
if len(seen_f) >= 3:
|
||||
break
|
||||
return found, (f"Molmo2{' (reused)' if reused else ''}: '{desc}', {len(pts)} points from f{anchor}, {len(shots)} shots, "
|
||||
f"{len(found)} shots re-anchored"), pts
|
||||
|
||||
|
||||
# OpenPose 18-point body (COCO order of controlnet_aux DWPose) and the 70-point face (68 + 2 pupils)
|
||||
BODY_LIMBS = [(1, 2), (1, 5), (2, 3), (3, 4), (5, 6), (6, 7), (1, 8), (8, 9), (9, 10), (1, 11), (11, 12), (12, 13), (1, 0),
|
||||
(0, 14), (14, 16), (0, 15), (15, 17)]
|
||||
HAND_LIMBS = [(0, 1), (1, 2), (2, 3), (3, 4), (0, 5), (5, 6), (6, 7), (7, 8), (0, 9), (9, 10), (10, 11), (11, 12),
|
||||
(0, 13), (13, 14), (14, 15), (15, 16), (0, 17), (17, 18), (18, 19), (19, 20)]
|
||||
|
||||
|
||||
def _pts(flat, W, H, thr=0.3):
|
||||
"""[x, y, c, ...] -> list of (x, y) pixels or None; DWPose writes 0..1 coordinates (pixels are handled too). Points
|
||||
outside the frame (guessed limbs of a close-up) are dropped."""
|
||||
if not flat:
|
||||
return []
|
||||
arr = [(flat[i], flat[i + 1], flat[i + 2]) for i in range(0, len(flat) - 2, 3)]
|
||||
norm = max((max(abs(x), abs(y)) for x, y, c in arr if c > 0), default=0) <= 1.5
|
||||
out = []
|
||||
for x, y, c in arr:
|
||||
if c <= 0 or (x <= 0 and y <= 0):
|
||||
out.append(None); continue
|
||||
px, py = (x * W, y * H) if norm else (x, y)
|
||||
out.append((px, py) if (-0.01 * W <= px <= 1.01 * W and -0.01 * H <= py <= 1.01 * H) else None)
|
||||
return out
|
||||
|
||||
|
||||
def _limb_ok(a, b, cap):
|
||||
return a is not None and b is not None and ((a[0] - b[0]) ** 2 + (a[1] - b[1]) ** 2) ** 0.5 <= cap
|
||||
|
||||
|
||||
def pick_person(people, mask, W, H, min_inside=0.5, min_diag=0.3, exclude=()):
|
||||
"""The DWPose person of this subject: most of its body/face points inside the subject's OWN mask (not the grown gray
|
||||
region, which can swallow someone in the background) and a size that matches it. None when DWPose missed the subject
|
||||
in this frame (e.g. an upside-down dancer): then no skeleton is drawn rather than somebody else's."""
|
||||
import numpy as np
|
||||
ys, xs = np.nonzero(mask)
|
||||
if not len(xs):
|
||||
return None
|
||||
mdiag = float(np.hypot(xs.max() - xs.min(), ys.max() - ys.min())) + 1
|
||||
best, score = None, 0.0
|
||||
excl_ids = {id(p) for p in (exclude or ())}
|
||||
for p in people or []:
|
||||
if id(p) in excl_ids:
|
||||
continue
|
||||
pts = [q for q in _pts(p.get("pose_keypoints_2d"), W, H) + _pts(p.get("face_keypoints_2d"), W, H) if q]
|
||||
if len(pts) < 4:
|
||||
continue
|
||||
inside = sum(1 for x, y in pts if 0 <= int(y) < H and 0 <= int(x) < W and mask[int(y), int(x)]) / len(pts)
|
||||
px, py = [q[0] for q in pts], [q[1] for q in pts]
|
||||
pdiag = float(np.hypot(max(px) - min(px), max(py) - min(py)))
|
||||
if inside < min_inside or pdiag < min_diag * mdiag:
|
||||
continue
|
||||
sc = inside + min(pdiag / mdiag, 1.0)
|
||||
if sc > score: best, score = p, sc
|
||||
return best
|
||||
|
||||
|
||||
def draw_pose(person, color, W, H):
|
||||
"""The skeleton (body + hands) and a face whose mouth opening and eyes are FILLED areas, so speech and blinks survive
|
||||
the video VAE; line widths scale with the person's size. Returns an [H, W, 3] uint8 RGB canvas and its drawn mask."""
|
||||
import cv2
|
||||
import numpy as np
|
||||
canvas = np.zeros((H, W, 3), np.uint8); col = tuple(int(c) for c in color)
|
||||
body = _pts(person.get("pose_keypoints_2d"), W, H); face = _pts(person.get("face_keypoints_2d"), W, H)
|
||||
allp = [q for q in body + face if q]
|
||||
if not allp:
|
||||
return canvas
|
||||
xs, ys = [q[0] for q in allp], [q[1] for q in allp]
|
||||
size = max(max(xs) - min(xs), max(ys) - min(ys), 16.0)
|
||||
lw = max(2, int(size * 0.025))
|
||||
has_face = len(face) >= 68 and sum(1 for q in face[:68] if q) > 40
|
||||
head = {0, 14, 15, 16, 17} # with a face, the face draws the head (body head points would cover it)
|
||||
cap = 0.6 * (W * W + H * H) ** 0.5 # a limb longer than this is a wrong guess (close-ups)
|
||||
fbox, closeup = None, False
|
||||
if has_face:
|
||||
fq = [q for q in face[:68] if q]
|
||||
fx0, fx1 = min(q[0] for q in fq), max(q[0] for q in fq); fy0, fy1 = min(q[1] for q in fq), max(q[1] for q in fq)
|
||||
mx, my = (fx1 - fx0) * 0.1, (fy1 - fy0) * 0.1
|
||||
fbox = (fx0 - mx, fy0 - my, fx1 + mx, fy1 + my)
|
||||
closeup = (fy1 - fy0) > 0.3 * H # the face fills the frame: arms and legs are guesses
|
||||
shoulders = {(1, 2), (1, 5)}
|
||||
for a, b in BODY_LIMBS:
|
||||
if has_face and (a in head or b in head):
|
||||
continue
|
||||
if closeup and (a, b) not in shoulders:
|
||||
continue
|
||||
if a < len(body) and b < len(body) and _limb_ok(body[a], body[b], cap):
|
||||
if fbox is not None: # never across the face
|
||||
cx, cy = (body[a][0] + body[b][0]) / 2, (body[a][1] + body[b][1]) / 2
|
||||
if fbox[0] <= cx <= fbox[2] and fbox[1] <= cy <= fbox[3]:
|
||||
continue
|
||||
cv2.line(canvas, tuple(map(int, body[a])), tuple(map(int, body[b])), col, lw, cv2.LINE_AA)
|
||||
for k, q in enumerate(body):
|
||||
if not q or (has_face and k in head) or (closeup and k not in (1, 2, 5)):
|
||||
continue
|
||||
if fbox is not None and fbox[0] <= q[0] <= fbox[2] and fbox[1] <= q[1] <= fbox[3]:
|
||||
continue
|
||||
cv2.circle(canvas, tuple(map(int, q)), lw, col, -1, cv2.LINE_AA)
|
||||
if has_face and len(body) > 1 and body[1]: # neck to the chin
|
||||
chin = face[8] if face[8] else None
|
||||
if chin: cv2.line(canvas, tuple(map(int, body[1])), tuple(map(int, chin)), col, max(1, lw // 2), cv2.LINE_AA)
|
||||
for key in ("hand_left_keypoints_2d", "hand_right_keypoints_2d"):
|
||||
hand = _pts(person.get(key), W, H)
|
||||
hp = [q for q in hand if q]
|
||||
if len(hp) < 12: # a hand is drawn only when most of it was found
|
||||
continue
|
||||
hs = max(max(q[0] for q in hp) - min(q[0] for q in hp), max(q[1] for q in hp) - min(q[1] for q in hp))
|
||||
if hs > 0.5 * size: # a "hand" as big as the person is noise
|
||||
continue
|
||||
for a, b in HAND_LIMBS:
|
||||
if a < len(hand) and b < len(hand) and _limb_ok(hand[a], hand[b], 0.35 * size):
|
||||
cv2.line(canvas, tuple(map(int, hand[a])), tuple(map(int, hand[b])), col, max(1, lw // 2), cv2.LINE_AA)
|
||||
if has_face:
|
||||
fx = [q[0] for q in face[:68] if q]; fy = [q[1] for q in face[:68] if q]
|
||||
fs = max(max(fx) - min(fx), max(fy) - min(fy), 8.0)
|
||||
fw = max(2, int(fs * 0.04))
|
||||
light = tuple(min(255, int(c * 0.55 + 255 * 0.45)) for c in col)
|
||||
def poly(idx, closed):
|
||||
q = [face[i] for i in idx if face[i]]
|
||||
return np.array([[int(x), int(y)] for x, y in q], np.int32) if len(q) >= 3 else None
|
||||
for idx, closed in ((range(0, 17), False), (range(17, 22), False), (range(22, 27), False), (range(27, 31), False), (range(31, 36), False)):
|
||||
pl = poly(idx, closed)
|
||||
if pl is not None: cv2.polylines(canvas, [pl], closed, col, fw, cv2.LINE_AA)
|
||||
for idx in (range(36, 42), range(42, 48)): # eyes: filled, so a blink is an area closing
|
||||
pl = poly(idx, True)
|
||||
if pl is not None: cv2.fillPoly(canvas, [pl], light, cv2.LINE_AA); cv2.polylines(canvas, [pl], True, col, max(1, fw // 2), cv2.LINE_AA)
|
||||
outer, inner = poly(range(48, 60), True), poly(range(60, 68), True)
|
||||
if outer is not None:
|
||||
cv2.fillPoly(canvas, [outer], col, cv2.LINE_AA) # lips: solid color
|
||||
if inner is not None:
|
||||
cv2.fillPoly(canvas, [inner], (20, 20, 20), cv2.LINE_AA) # mouth opening: dark area, its size = how open
|
||||
for q in face[68:70]:
|
||||
if q: cv2.circle(canvas, tuple(map(int, q)), max(1, fw // 2), (20, 20, 20), -1, cv2.LINE_AA)
|
||||
return canvas
|
||||
|
||||
|
||||
def to_opencv_normals(img, convention):
|
||||
"""Render MoGe Geometry draws normals flipped (Z, and Y for OpenGL); undo it to get the raw MoGe normals as RGB."""
|
||||
n = img[..., :3] * 2 - 1
|
||||
n = n * n.new_tensor([1.0, -1.0 if convention == "normal_opengl" else 1.0, -1.0])
|
||||
return (n * 0.5 + 0.5).clamp(0, 1)
|
||||
|
||||
|
||||
def grow_masks(masks, rel):
|
||||
"""Dilate each mask by rel * sqrt(its mean area) pixels (the margin around the subject)."""
|
||||
if rel <= 0:
|
||||
return masks
|
||||
import torch.nn.functional as F
|
||||
area = float(masks.sum(dim=(1, 2)).mean())
|
||||
r = max(1, int(round((area ** 0.5) * rel)))
|
||||
return F.max_pool2d(masks[:, None].float(), 2 * r + 1, 1, r)[:, 0]
|
||||
|
||||
|
||||
from collections import OrderedDict
|
||||
_TRACK_CACHE = OrderedDict() # (video, subject, its picks, cuts, options) -> (masks, report), most recent last
|
||||
|
||||
|
||||
def video_key(image):
|
||||
"""A cheap fingerprint of the frames (subsampled pixels + shape) for the track cache."""
|
||||
import hashlib
|
||||
smp = image[:: max(1, image.shape[0] // 12), ::24, ::24, :3].float().cpu().numpy()
|
||||
return hashlib.sha1(smp.tobytes() + str(tuple(image.shape)).encode()).hexdigest()[:16]
|
||||
|
||||
|
||||
def timeline_strip(image, per_subject, width=256, max_frames=400):
|
||||
"""Small JPEGs of every frame with each subject's mask tinted in its color, for the panel's whole-video scrubber
|
||||
(so a wrong mask can be found on any frame, not only on the strip)."""
|
||||
import os, uuid
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
n, h, w = image.shape[:3]
|
||||
idx = sorted(set(int(round(x)) for x in torch.linspace(0, n - 1, min(n, max_frames)).tolist()))
|
||||
sw = min(width, w); sh = max(1, round(h * sw / w))
|
||||
out_dir = os.path.join(sp._temp_dir(), "tl_" + uuid.uuid4().hex[:8]); os.makedirs(out_dir, exist_ok=True)
|
||||
files = []
|
||||
for i in idx:
|
||||
fr = image[i, ..., :3].float().clone()
|
||||
for sj, m in per_subject.items():
|
||||
c = torch.tensor(HYBRID_COLORS[(int(sj) - 1) % len(HYBRID_COLORS)], dtype=fr.dtype) / 255.0
|
||||
mk = (m[i] > 0.5)[..., None].to(fr.dtype) * 0.55
|
||||
fr = fr * (1 - mk) + c * mk
|
||||
arr = (fr.clamp(0, 1).cpu().numpy() * 255).astype(np.uint8)
|
||||
name = f"f{i:05d}.jpg"
|
||||
Image.fromarray(arr).resize((sw, sh), Image.BILINEAR).save(os.path.join(out_dir, name), quality=80)
|
||||
files.append({"filename": name, "subfolder": "bfs_segpick/" + os.path.basename(out_dir), "type": "temp", "frame": i})
|
||||
return files
|
||||
|
||||
|
||||
class Status:
|
||||
"""Live status for the panel (message 'bfs_vsegpick_status' to this node) + ComfyUI's own progress bar + console log."""
|
||||
def __init__(self, uid):
|
||||
import time
|
||||
self.uid, self.t0, self.last = (str(uid) if uid is not None else None), time.time(), 0.0
|
||||
try:
|
||||
import comfy.utils
|
||||
self.bar = comfy.utils.ProgressBar(1000)
|
||||
except Exception:
|
||||
self.bar = None
|
||||
|
||||
def __call__(self, text, progress=None, log=False):
|
||||
import time
|
||||
now = time.time()
|
||||
if progress is not None and self.bar is not None:
|
||||
self.bar.update_absolute(int(max(0.0, min(1.0, progress)) * 1000))
|
||||
if log:
|
||||
print(f"[BFS Video Segment Picker] {text}", flush=True)
|
||||
if not log and now - self.last < 0.25: # throttle the frequent updates
|
||||
return
|
||||
self.last = now
|
||||
try:
|
||||
from server import PromptServer
|
||||
PromptServer.instance.send_sync("bfs_vsegpick_status", {"node": self.uid, "text": text,
|
||||
"progress": progress, "log": bool(log), "elapsed": round(now - self.t0, 1)})
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def parse_frames(text):
|
||||
"""{"frames": {"<frame>": {"<subject>": selection}}}; a frame holding a bare selection is subject 1."""
|
||||
try:
|
||||
s = json.loads(text or "{}") or {}
|
||||
except Exception:
|
||||
s = {}
|
||||
frames = s.get("frames") if isinstance(s.get("frames"), dict) else {}
|
||||
out = {}
|
||||
for k, v in frames.items():
|
||||
if not isinstance(v, dict):
|
||||
continue
|
||||
out[str(k)] = {"1": v} if ("selected" in v or "added" in v) else {
|
||||
str(sk): sv for sk, sv in v.items()
|
||||
if isinstance(sv, dict) and str(sk).isdigit() and 1 <= int(sk) <= len(HYBRID_COLORS)
|
||||
}
|
||||
return out
|
||||
|
||||
|
||||
class BFSVideoSegmentPicker:
|
||||
DESCRIPTION = __doc__.split("\n\n")[0]
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
t = sp.BFSSegmentPicker.INPUT_TYPES()
|
||||
req = dict(t["required"])
|
||||
req["image"] = ("IMAGE", {"tooltip": f"The frames: the Keyframe Planner's source_frames or a short clip (at most {MAX_FRAMES})."})
|
||||
req["selection"] = ("STRING", {"default": "{}", "multiline": True, "tooltip": "Set by the panel: the selection on each frame."})
|
||||
req["track"] = ("BOOLEAN", {"default": True, "tooltip": "Frames without a selection take the nearest selected frame's mask, tracked with SAM 3."})
|
||||
opt = {
|
||||
"normals": ("IMAGE", {"lazy": True, "tooltip": "Optional: MoGe normals of the same frames (Run MoGe Inference -> Render MoGe Geometry, a normal mode). "
|
||||
"With it the node also returns the normal-map hybrid guide."}),
|
||||
"normals_convention": (["normal_directx", "normal_opengl"], {"default": "normal_directx",
|
||||
"tooltip": "The Render MoGe Geometry output mode the normals come from."}),
|
||||
"hybrid_base": ("IMAGE", {"tooltip": "Optional: the hybrid output of another Video Segment Picker, to add this subject on top (one picker per subject)."}),
|
||||
"pose_keypoint": ("POSE_KEYPOINT", {"lazy": True, "tooltip": "Optional: DWPose Estimator keypoints of the same frames (body + hands + face), "
|
||||
"for mark_style 'openpose + face'."}),
|
||||
"molmo_clip": ("CLIP", {"lazy": True, "tooltip": "Optional: Molmo2 as H3 encoder. It sees the whole video, finds the picked person again after "
|
||||
"camera cuts (describe -> track) and SAM 3.1 segments them there from Molmo's point."}),
|
||||
"mark_style": (MARK_STYLES, {"default": MARK_STYLES[0],
|
||||
"tooltip": "How each subject is drawn in the hybrid output, always in its color: the normal map, the DWPose "
|
||||
"skeleton with the face, or texture-free alternatives that need no MoGe (flat, blur, pixelate, gray "
|
||||
"detail, edges). With a style that needs no normals, 'hybrid' works without the normals input."}),
|
||||
"hybrid_grow": ("FLOAT", {"default": 0.08, "min": 0.0, "max": 0.5, "step": 0.01,
|
||||
"tooltip": "Margin around the subject in the hybrid guide, relative to its size (normal map: 0.04-0.12; "
|
||||
"openpose: 0.08-0.18 around the person's convex hull)."}),
|
||||
"strip_frames": ("INT", {"default": 16, "min": 2, "max": MAX_FRAMES,
|
||||
"tooltip": "How many frames the panel shows to pick on (SAM segments each one): fewer is faster."}),
|
||||
"free_after": (FREE_MODES, {"default": FREE_MODES[0],
|
||||
"tooltip": "After the node: free the GPU for the next nodes (H3). 'all models' also unloads H3 / the text "
|
||||
"encoder left from the last generation (they load again when needed). Clearing the caches also "
|
||||
"frees RAM, but the next run segments and tracks again. Before MoGe / DWPose / Molmo2 run "
|
||||
"(only on Propagate) every model is unloaded anyway, so a second run does not run out of memory."}),
|
||||
"pose_region": (["person outline", "convex hull"], {"default": "person outline",
|
||||
"tooltip": "openpose mark: the gray region around each subject. The outline (hybrid_grow ~0.02-0.05) hides "
|
||||
"the least background and usually looks best; the convex hull (with hybrid_grow margin) "
|
||||
"hides the original body shape (the new one stays free); the outline follows the person "
|
||||
"(less background hidden, but the original silhouette shows)."}),
|
||||
"pose_gray_opacity": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 1.0, "step": 0.05,
|
||||
"tooltip": "openpose mark: how opaque the gray region is (1.0 = fully hidden). Lower lets the "
|
||||
"background show through, but also the original person (clothes, body), which the "
|
||||
"result may then copy."}),
|
||||
}
|
||||
return {"required": req, "optional": opt, "hidden": {**t.get("hidden", {}), "prompt": "PROMPT"}}
|
||||
|
||||
RETURN_TYPES = ("MASK", "IMAGE", "IMAGE", "STRING", "IMAGE")
|
||||
RETURN_NAMES = ("masks", "marked", "overlay", "report", "hybrid")
|
||||
OUTPUT_TOOLTIPS = ("One mask per frame: the person to replace.", "Every frame with the person tinted (mark_color, mark_opacity).",
|
||||
"The strip frames with all objects found, numbered.", "JSON per frame: picked by hand, tracked from frame N, or empty.",
|
||||
"With normals: the frames (or hybrid_base) with this subject drawn as its tinted normal map; else hybrid_base or the frames.")
|
||||
FUNCTION = "pick"
|
||||
CATEGORY = "BFS/Mask"
|
||||
OUTPUT_NODE = True
|
||||
|
||||
def pick(self, image, sam3_model, sam3_clip, concepts, threshold, max_per_concept, min_area, mark_color, mark_opacity,
|
||||
selection, track=True, strip_frames=16, normals=None, normals_convention="normal_directx", hybrid_base=None, subject=None,
|
||||
mark_style=MARK_STYLES[0], molmo_clip=None, pose_keypoint=None,
|
||||
hybrid_grow=0.08, free_after=FREE_MODES[0], pose_region="person outline",
|
||||
pose_gray_opacity=1.0, unique_id=None, prompt=None):
|
||||
try:
|
||||
return self._pick(image, sam3_model, sam3_clip, concepts, threshold, max_per_concept, min_area, mark_color, mark_opacity,
|
||||
selection, track, strip_frames, normals, normals_convention, hybrid_base, subject, mark_style,
|
||||
molmo_clip, pose_keypoint, hybrid_grow, unique_id, pose_region, pose_gray_opacity)
|
||||
finally:
|
||||
self._free(sam3_model, free_after)
|
||||
|
||||
def check_lazy_status(self, selection="{}", mark_style=MARK_STYLES[0], track=True, normals=None, pose_keypoint=None,
|
||||
molmo_clip=None, unique_id=None, prompt=None, **kw):
|
||||
"""The heavy optional inputs (MoGe normals, DWPose keypoints, Molmo2) are only needed when propagating, not while
|
||||
picking; before they run, every model leaves the GPU (e.g. H3 from the last generation), or MoGe runs out of memory."""
|
||||
try:
|
||||
segment_only = bool(json.loads(selection or "{}").get("segment_only"))
|
||||
except Exception:
|
||||
segment_only = False
|
||||
if segment_only:
|
||||
return []
|
||||
need = []
|
||||
if normals is None and not mark_style.startswith("openpose"):
|
||||
need.append("normals")
|
||||
if pose_keypoint is None and mark_style.startswith("openpose"):
|
||||
need.append("pose_keypoint")
|
||||
if molmo_clip is None and track:
|
||||
need.append("molmo_clip")
|
||||
links = ((prompt or {}).get(str(unique_id)) or {}).get("inputs", {})
|
||||
need = [x for x in need if isinstance(links.get(x), list)] # only connected inputs can be requested
|
||||
if need:
|
||||
try:
|
||||
import comfy.model_management as mm
|
||||
mm.unload_all_models(); mm.soft_empty_cache()
|
||||
print(f"[BFS Video Segment Picker] all models unloaded before {', '.join(need)}", flush=True)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"[BFS Video Segment Picker] could not unload models: {e!r}", flush=True)
|
||||
return need
|
||||
|
||||
@staticmethod
|
||||
def _free(sam3_model, mode):
|
||||
mode = LEGACY_FREE.get(mode, mode)
|
||||
if mode.startswith("keep"):
|
||||
return
|
||||
try:
|
||||
import gc
|
||||
import comfy.model_management as mm
|
||||
if mode.startswith("unload all"):
|
||||
mm.unload_all_models()
|
||||
else:
|
||||
mm.unload_model_and_clones(sam3_model)
|
||||
if "caches" in mode:
|
||||
sp._SESSIONS.clear(); _TRACK_CACHE.clear(); _SHOT_CACHE.clear(); _MOLMO_CACHE.clear()
|
||||
gc.collect()
|
||||
mm.soft_empty_cache(True)
|
||||
print(f"[BFS Video Segment Picker] memory freed ({mode})", flush=True)
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f"[BFS Video Segment Picker] could not free memory: {e!r}", flush=True)
|
||||
|
||||
def _pick(self, image, sam3_model, sam3_clip, concepts, threshold, max_per_concept, min_area, mark_color, mark_opacity,
|
||||
selection, track, strip_frames, normals, normals_convention, hybrid_base, subject, mark_style, molmo_clip,
|
||||
pose_keypoint, hybrid_grow, unique_id, pose_region="person outline", pose_gray_opacity=1.0):
|
||||
st = Status(unique_id)
|
||||
n = int(image.shape[0])
|
||||
st(f"start: {n} frames", 0.0, log=True)
|
||||
strip = sorted(set(int(round(x)) for x in torch.linspace(0, n - 1, min(n, int(strip_frames), MAX_FRAMES)).tolist()))
|
||||
for sel_frame in parse_frames(selection): # frames picked on earlier stay in the strip
|
||||
if sel_frame.isdigit() and int(sel_frame) < n and int(sel_frame) not in strip:
|
||||
strip = sorted(strip + [int(sel_frame)])
|
||||
sp.MAX_SESSIONS = max(sp.MAX_SESSIONS, MAX_FRAMES + 8) # every frame keeps its session for the panel's "add"
|
||||
h, w = image.shape[1:3]
|
||||
spec = {"concepts": ", ".join(sp.split_concepts(concepts)), "threshold": round(float(threshold), 4),
|
||||
"max_per_concept": int(max_per_concept), "min_area": round(float(min_area), 5)}
|
||||
states = parse_frames(selection)
|
||||
try:
|
||||
segment_only = bool(json.loads(selection or "{}").get("segment_only"))
|
||||
except Exception:
|
||||
segment_only = False
|
||||
if segment_only:
|
||||
st("segment only (picking): no tracking, no Molmo2 - press Propagate to track", None, log=True)
|
||||
subjects = sorted({int(sk) for st in states.values() for sk, sv in st.items() if (sv.get("selected") or [])} or {1})
|
||||
panels, overlays = [], []
|
||||
own = {sj: {} for sj in subjects}
|
||||
for k_strip, i in enumerate(strip):
|
||||
st(f"SAM 3: segmenting frame {i} ({k_strip + 1}/{len(strip)})", 0.02 + 0.28 * k_strip / max(1, len(strip)))
|
||||
frame = image[i:i + 1, ..., :3].float().cpu()
|
||||
sid, sess = sp.session_for((sam3_model, sam3_clip), frame, spec)
|
||||
chosen_all = set()
|
||||
for sj in subjects:
|
||||
state = sp.parse_state(json.dumps(states.get(str(i), {}).get(str(sj), {})))
|
||||
objects = sp.all_objects(sess, state)
|
||||
by_id = {o["id"]: o for o in objects}
|
||||
chosen = [by_id[k] for k in state["selected"] if k in by_id]
|
||||
chosen_all |= {o["id"] for o in chosen}
|
||||
cut = [o["mask"] for o in chosen if o.get("remove")]
|
||||
cut = torch.stack(cut).any(0) if cut else torch.zeros(h, w, dtype=torch.bool)
|
||||
mask, _ = sp.compose([o["mask"] & ~cut for o in chosen if not o.get("remove")], h, w)
|
||||
own[sj][i] = mask.reshape(h, w).float()
|
||||
objects = sp.all_objects(sess, sp.parse_state(json.dumps({"added": [a for st in states.get(str(i), {}).values() for a in st.get("added", [])]})))
|
||||
panels.append(sp.panel_data(sid, sess, objects))
|
||||
overlays.append(sp.overlay(frame, objects, chosen_all))
|
||||
cuts = detect_cuts(image) if n > 1 else []
|
||||
if cuts:
|
||||
st(f"camera cuts at frames {cuts}", 0.3, log=True)
|
||||
per_subject, report_s = {}, {}
|
||||
vkey = video_key(image)
|
||||
for sj in subjects:
|
||||
mine = {k: v[str(sj)] for k, v in sorted(states.items()) if v.get(str(sj))} # only this subject's picks
|
||||
ckey = ("v2", vkey, sj, json.dumps(mine, sort_keys=True), tuple(cuts), bool(track), molmo_clip is not None, float(threshold))
|
||||
hit = _TRACK_CACHE.get(ckey)
|
||||
if hit is not None: # same video, same picks for this subject: reuse its masks (style / margin changes are instant)
|
||||
st(f"subject {sj}: masks reused from the last run (same video and picks)", 0.3 + 0.4 * subjects.index(sj) / len(subjects), log=True)
|
||||
per_subject[sj], report_s[sj] = hit[0].clone(), [dict(r) for r in hit[1]]
|
||||
_TRACK_CACHE.move_to_end(ckey)
|
||||
continue
|
||||
anchors = [i for i in strip if bool(own[sj][i].any())]
|
||||
user_anchors = list(anchors)
|
||||
first_pick = user_anchors[0] if user_anchors else 0
|
||||
prev_strip = max([s for s in strip if s < first_pick], default=0)
|
||||
min_valid_frame = 0 if first_pick == 0 else prev_strip
|
||||
if segment_only: # picking only: the picked frames' masks (the last propagated masks are kept by the cache)
|
||||
m = torch.zeros(n, h, w)
|
||||
for i in anchors: m[i] = own[sj][i]
|
||||
per_subject[sj] = m
|
||||
report_s[sj] = [{"frame": i, "source": "picked" if i in anchors else "not tracked yet (Propagate)"} for i in range(n)]
|
||||
continue
|
||||
m = torch.zeros(n, h, w)
|
||||
for i in anchors:
|
||||
m[i] = own[sj][i]
|
||||
rep = [{"frame": i, "source": "picked" if i in anchors else "empty"} for i in range(n)]
|
||||
if anchors and track and n > 1:
|
||||
def claimed_at(f):
|
||||
out_c = [per_subject[ok][f] for ok in per_subject if ok != sj and bool((per_subject[ok][f] > 0.5).any())]
|
||||
for ok in subjects:
|
||||
if ok != sj and f in own.get(ok, {}) and bool((own[ok][f] > 0.5).any()):
|
||||
out_c.append(own[ok][f])
|
||||
return out_c
|
||||
|
||||
shots = shots_of(n, cuts)
|
||||
shot_of = lambda f: next(k for k, (s0, s1) in enumerate(shots) if s0 <= f < s1)
|
||||
has = [any(s0 <= a < s1 for a in anchors) for s0, s1 in shots]
|
||||
use_molmo = molmo_clip is not None and hasattr(molmo_clip, "runtime")
|
||||
pts = []
|
||||
if use_molmo:
|
||||
st(f"Molmo2: describing and tracking subject {sj} from frame {first_pick} forward", 0.3 + 0.4 * subjects.index(sj) / len(subjects))
|
||||
t0 = time.time()
|
||||
claimed_map = {f: claimed_at(f) for f in range(first_pick + 1, n)}
|
||||
extra, note, pts = molmo_relocate(
|
||||
molmo_clip, (sam3_model, sam3_clip), image, own[sj][first_pick], first_pick,
|
||||
shots, has, float(threshold), memo=(vkey, sj, float(threshold)), claimed_by_frame=claimed_map
|
||||
)
|
||||
note += f" ({time.time() - t0:.0f}s)"
|
||||
st(f"subject {sj}: {note}", None, log=True)
|
||||
for f, mk in extra.items():
|
||||
own[sj][f] = mk; anchors = sorted(set(anchors) | {f})
|
||||
|
||||
def track_shots(anchors):
|
||||
out, rp = {}, {}
|
||||
kept = 0
|
||||
for si_, (s0, s1) in enumerate(shots): # each shot is tracked only from anchors inside it: never across a cut
|
||||
if s1 <= min_valid_frame:
|
||||
for i in range(s0, s1):
|
||||
out[i] = torch.zeros(h, w); rp[i] = "before first pick"
|
||||
continue
|
||||
sa = [a for a in anchors if s0 <= a < s1]
|
||||
skey = ("v2", vkey, sj, s0, s1, min_valid_frame, tuple((a, mask_key(own[sj][a])) for a in sa), float(threshold))
|
||||
if skey in _SHOT_CACHE: # same anchors in this shot as a previous run: its tracking is the same
|
||||
cm, cr = _SHOT_CACHE[skey]
|
||||
_SHOT_CACHE.move_to_end(skey)
|
||||
for i in range(s0, s1):
|
||||
out[i] = cm[i - s0].float(); rp[i] = cr[i - s0]
|
||||
kept += 1
|
||||
continue
|
||||
st(f"SAM 3: tracking subject {sj}, shot {si_ + 1}/{len(shots)} (frames {s0}-{s1 - 1})",
|
||||
0.3 + 0.4 * (subjects.index(sj) + si_ / max(1, len(shots))) / len(subjects))
|
||||
if not sa:
|
||||
for i in range(s0, s1):
|
||||
out[i] = torch.zeros(h, w)
|
||||
rp[i] = "before first pick" if i < first_pick else "lost (no anchor in this shot)"
|
||||
continue
|
||||
seg_s0 = max(s0, min_valid_frame)
|
||||
seg = image[seg_s0:s1, ..., :3]
|
||||
tr = {a: (BFSTrackFromFrame().track(seg, sam3_model, own[sj][a][None], a - seg_s0)[0] if s1 - seg_s0 > 1
|
||||
else own[sj][a][None]) for a in sa if a >= seg_s0}
|
||||
for i in range(s0, seg_s0):
|
||||
out[i] = torch.zeros(h, w); rp[i] = "before first pick"
|
||||
for i in range(seg_s0, s1):
|
||||
if i in sa:
|
||||
out[i] = own[sj][i]; rp[i] = "picked" if i in user_anchors else "Molmo2 + SAM"; continue
|
||||
a = min(tr, key=lambda k: (abs(k - i), k))
|
||||
out[i] = (tr[a][i - seg_s0] > 0.5).float()
|
||||
rp[i] = f"tracked from {a}" if bool(out[i].any()) else (
|
||||
"before first pick" if i < first_pick else f"lost (tracked from {a})")
|
||||
if first_pick > seg_s0 and seg_s0 <= first_pick < s1:
|
||||
for i in range(first_pick - 1, seg_s0 - 1, -1):
|
||||
ok_bwd = (
|
||||
bool(out[i].any())
|
||||
and mask_overlap(out[i], out[i + 1])[0] >= 0.15
|
||||
and not any(mask_overlap(out[i], cm)[1] > 0.55 for cm in claimed_at(i))
|
||||
and person_color_sim(image[first_pick, ..., :3], own[sj][first_pick], image[i, ..., :3], out[i]) >= 0.48
|
||||
)
|
||||
if not ok_bwd:
|
||||
for j in range(seg_s0, i + 1):
|
||||
out[j] = torch.zeros(h, w); rp[j] = "before first pick"
|
||||
break
|
||||
for i in range(seg_s0, s1):
|
||||
if bool(out[i].any()) and i not in user_anchors:
|
||||
cl = claimed_at(i)
|
||||
if cl:
|
||||
cu = torch.stack([cm > 0.5 for cm in cl]).any(0)
|
||||
out[i] = out[i] * (~cu).float()
|
||||
if not bool(out[i].any()):
|
||||
rp[i] = "before first pick" if i < first_pick else rp[i].replace("tracked from", "lost (tracked from") + (")" if not rp[i].endswith(")") else "")
|
||||
if sa:
|
||||
_remember(_SHOT_CACHE, skey, (torch.stack([out[i] > 0.5 for i in range(s0, s1)]),
|
||||
[rp[i] for i in range(s0, s1)]), 64)
|
||||
if kept:
|
||||
st(f"subject {sj}: {kept}/{len(shots)} shots unchanged, reused", None, log=True)
|
||||
return out, rp
|
||||
|
||||
out, rp = track_shots(anchors)
|
||||
if pts: # second pass: stretches after first_pick where the SAM mask is missing OR misses Molmo's point
|
||||
ref_crop = person_crop(image[first_pick, ..., :3], own[sj][first_pick])
|
||||
off = set()
|
||||
for t_, _o, x_, y_ in pts:
|
||||
f_ = min(n - 1, max(0, int(round(float(t_) * 24))))
|
||||
if f_ <= first_pick:
|
||||
continue
|
||||
px_, py_ = min(w - 1, int(x_ * w)), min(h - 1, int(y_ * h))
|
||||
mk_ = out[f_] > 0.5
|
||||
if bool(mk_.any()) and not bool(mk_[max(0, py_ - 6):py_ + 7, max(0, px_ - 6):px_ + 7].any()):
|
||||
if person_color_sim(image[first_pick, ..., :3], own[sj][first_pick], image[f_, ..., :3], mk_) < 0.42:
|
||||
off.add(f_)
|
||||
# Molmo points about every 6 frames: two in a row off the mask -> the frames between them count as lost
|
||||
offs = sorted(off); wrong = set()
|
||||
for a_, b_ in zip(offs, offs[1:]):
|
||||
if b_ - a_ <= 8: wrong.update(range(a_, b_ + 1))
|
||||
lost = sorted({i for i in range(first_pick + 1, n) if not bool(out[i].any())} | wrong)
|
||||
runs, cur = [], []
|
||||
for i in lost:
|
||||
if cur and i != cur[-1] + 1: runs.append(cur); cur = []
|
||||
cur.append(i)
|
||||
if cur: runs.append(cur)
|
||||
added = 0
|
||||
for run in [r for r in runs if len(r) >= 6]:
|
||||
inside = [q for q in pts if run[0] <= int(round(float(q[0]) * 24)) <= run[-1]]
|
||||
if not inside: continue
|
||||
mid = (run[0] + run[-1]) / 2.0
|
||||
seen_f = set()
|
||||
for t, _o, x, y in sorted(inside, key=lambda q: abs(float(q[0]) * 24 - mid)):
|
||||
f = min(run[-1], max(run[0], int(round(float(t) * 24))))
|
||||
if f in seen_f:
|
||||
continue
|
||||
seen_f.add(f)
|
||||
mk = person_at((sam3_model, sam3_clip), image[f:f + 1, ..., :3].float().cpu(), x, y, float(threshold))
|
||||
if bool(mk.any()) and molmo_verify_candidate(
|
||||
molmo_clip, ref_crop, image[first_pick, ..., :3], own[sj][first_pick],
|
||||
image[f, ..., :3], mk, claimed_masks=claimed_at(f), memo=(vkey, sj, float(threshold), first_pick, f)
|
||||
):
|
||||
own[sj][f] = mk.float(); anchors = sorted(set(anchors) | {f}); added += 1
|
||||
break
|
||||
if len(seen_f) >= 3:
|
||||
break
|
||||
if added:
|
||||
st(f"subject {sj}: {added} lost stretches re-anchored by Molmo2 (visually verified), tracking again", None, log=True)
|
||||
prev_out, prev_rp = dict(out), dict(rp)
|
||||
out, rp = track_shots(anchors)
|
||||
for i in range(n):
|
||||
if bool(prev_out[i].any()) and i not in wrong:
|
||||
out[i], rp[i] = prev_out[i], prev_rp[i]
|
||||
for i in range(n):
|
||||
m[i] = out[i]; rep[i] = {"frame": i, "source": rp[i]}
|
||||
per_subject[sj] = m; report_s[sj] = rep
|
||||
_TRACK_CACHE[ckey] = (m.clone(), [dict(r) for r in rep])
|
||||
while len(_TRACK_CACHE) > 24:
|
||||
_TRACK_CACHE.popitem(last=False)
|
||||
masks = torch.stack(list(per_subject.values())).amax(0)
|
||||
anchors = sorted({i for sj in subjects for i in strip if bool(own[sj][i].any())})
|
||||
report = [{"frame": i, "subjects": {str(sj): report_s[sj][i]["source"] for sj in subjects}} for i in range(n)]
|
||||
if not anchors:
|
||||
st("nothing selected yet: click a frame, then the person, then run again", None, log=True)
|
||||
color = torch.tensor(sp.parse_color(mark_color), dtype=image.dtype) / 255.0
|
||||
a = (masks.to(image.dtype) * float(mark_opacity))[..., None]
|
||||
marked = image[..., :3] * (1 - a) + color * a
|
||||
base = (hybrid_base if hybrid_base is not None else image)[..., :3].clone()
|
||||
if mark_style.startswith("openpose") and pose_keypoint is None and segment_only:
|
||||
st("picking: DWPose runs on Propagate, the hybrid shows the marks then", None, log=True)
|
||||
elif mark_style.startswith("openpose"):
|
||||
if pose_keypoint is None:
|
||||
raise ValueError("mark_style 'openpose + face' needs the pose_keypoint input (DWPose Estimator on the same frames).")
|
||||
frames_kp = pose_keypoint if isinstance(pose_keypoint, list) else [pose_keypoint]
|
||||
if len(frames_kp) != n:
|
||||
raise ValueError(f"{len(frames_kp)} pose frames for {n} frames: run DWPose on the same frames.")
|
||||
import numpy as np
|
||||
import cv2
|
||||
st("drawing the DWPose marks", 0.75)
|
||||
ga = float(pose_gray_opacity)
|
||||
subj_data = {}
|
||||
# Pass 1: compute hulls and erase all subjects to neutral gray FIRST so no subject's gray region erases another subject's skeleton
|
||||
for sj, m in per_subject.items():
|
||||
mk = (m > 0.5).numpy().astype(np.uint8)
|
||||
area = float(mk.sum(axis=(1, 2)).mean()) + 1
|
||||
hull = pose_region.startswith("convex")
|
||||
r = max(3, int((area ** 0.5) * (max(float(hybrid_grow), 0.08) if hull else float(hybrid_grow))))
|
||||
se = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (2 * r + 1, 2 * r + 1))
|
||||
hulls = []
|
||||
for i in range(n):
|
||||
hm_i = np.zeros_like(mk[i])
|
||||
cs, _ = cv2.findContours(mk[i], cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
|
||||
if cs:
|
||||
cv2.fillPoly(hm_i, [cv2.convexHull(np.concatenate(cs))] if hull else cs, 1)
|
||||
region = torch.from_numpy(cv2.dilate(hm_i, se) > 0)
|
||||
hulls.append(region)
|
||||
if bool(region.any()):
|
||||
base[i][region] = base[i][region] * (1 - ga) + 0.5 * ga
|
||||
subj_data[sj] = (mk, hulls)
|
||||
|
||||
# Pass 2: match DWPose skeletons per frame (with relaxed fallback + neighbor frame fallback) and draw all skeletons on top
|
||||
used_per_frame = {i: [] for i in range(n)}
|
||||
for sj, (mk, hulls) in subj_data.items():
|
||||
col = HYBRID_COLORS[(int(sj) - 1) % len(HYBRID_COLORS)]
|
||||
matched = [None] * n
|
||||
for i in range(n):
|
||||
if not bool(hulls[i].any()):
|
||||
continue
|
||||
fk = frames_kp[i] or {}
|
||||
ppl = fk.get("people")
|
||||
tight = cv2.dilate(mk[i], cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (7, 7))) > 0
|
||||
person = pick_person(ppl, tight, w, h, exclude=used_per_frame[i])
|
||||
if person is None:
|
||||
grow_t = cv2.dilate(mk[i], cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (19, 19))) > 0
|
||||
person = pick_person(ppl, grow_t, w, h, min_inside=0.25, min_diag=0.2, exclude=used_per_frame[i])
|
||||
if person is not None:
|
||||
matched[i] = person
|
||||
used_per_frame[i].append(person)
|
||||
for i in range(n):
|
||||
if not bool(hulls[i].any()):
|
||||
continue
|
||||
person = matched[i]
|
||||
if person is None:
|
||||
for delta in (-1, 1, -2, 2):
|
||||
j = i + delta
|
||||
if 0 <= j < n and matched[j] is not None:
|
||||
person = matched[j]
|
||||
break
|
||||
if person is None:
|
||||
continue
|
||||
cv = torch.from_numpy(draw_pose(person, col, w, h)).to(base.dtype) / 255.0
|
||||
drawn = cv.amax(-1) > 0
|
||||
base[i][drawn] = cv[drawn] # the skeleton may leave the region: the body is free
|
||||
needs_normals = "needs normals" in mark_style and not mark_style.startswith("openpose")
|
||||
if not mark_style.startswith("openpose") and (normals is not None or not needs_normals):
|
||||
import torch.nn.functional as F
|
||||
if normals is not None:
|
||||
if normals.shape[0] != n:
|
||||
raise ValueError(f"{normals.shape[0]} normal maps for {n} frames: run MoGe on the same frames.")
|
||||
nm = normals[..., :3].float()
|
||||
if nm.shape[1:3] != (h, w):
|
||||
nm = F.interpolate(nm.movedim(-1, 1), size=(h, w), mode="bilinear", align_corners=False).movedim(1, -1)
|
||||
nm = to_opencv_normals(nm, normals_convention)
|
||||
elif needs_normals:
|
||||
nm = None
|
||||
else:
|
||||
nm = torch.full((n, h, w, 3), 0.5)
|
||||
for sj, m in per_subject.items(): # subject j: the j-th color (= Picture j)
|
||||
hm = grow_masks((m > 0.5).float(), float(hybrid_grow)) > 0.5
|
||||
col = HYBRID_COLORS[(int(sj) - 1) % len(HYBRID_COLORS)]
|
||||
for i in range(n):
|
||||
if bool(hm[i].any()):
|
||||
fill = mark_fill(mark_style, image[i, ..., :3].float(), nm[i], hm[i], col)
|
||||
base[i][hm[i]] = fill.to(base.dtype)[hm[i]]
|
||||
lost_runs = [] # stretches where a picked subject has no mask after its first appearance: the panel offers them to add to the strip
|
||||
for sj in subjects:
|
||||
if not any(r["source"] == "picked" for r in report_s[sj]):
|
||||
continue
|
||||
first_active = min([i for i in range(n) if bool(per_subject[sj][i].any())] or [0])
|
||||
run = None
|
||||
for i in range(first_active, n):
|
||||
bad = not bool(per_subject[sj][i].any())
|
||||
if bad and run is None: run = [i, i]
|
||||
elif bad: run[1] = i
|
||||
elif run is not None: lost_runs.append({"subject": sj, "start": run[0], "end": run[1]}); run = None
|
||||
if run is not None: lost_runs.append({"subject": sj, "start": run[0], "end": run[1]})
|
||||
ui = {"bfs_vsegpick": [{"frames": panels, "indices": strip, "report": [report[i] for i in strip], "subjects": subjects,
|
||||
"count": n, "lost": lost_runs, "timeline": (st("building the timeline previews", 0.9) or timeline_strip(image, per_subject))}]}
|
||||
lost_frames = sum(r["end"] - r["start"] + 1 for r in lost_runs)
|
||||
st(f"done: {len(subjects)} subject(s), {lost_frames} frames without a mask", 1.0, log=True)
|
||||
return {"ui": ui, "result": (masks, marked, torch.cat(overlays), json.dumps({"frames": report, "anchors": anchors, "subjects": subjects}), base)}
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"BFSVideoSegmentPicker": BFSVideoSegmentPicker}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"BFSVideoSegmentPicker": "BFS Video Segment Picker (SAM 3)"}
|
||||
@@ -0,0 +1,227 @@
|
||||
"""BFS Workflow Loop: run any part of any workflow once per item, injecting the item (and per-iteration parameters)
|
||||
into nodes you pick, and collect the results in order.
|
||||
|
||||
Connect the IMAGE the loop should produce to `result`, and the batch to iterate over to `items` (frames, keyframes,
|
||||
marked frames...) and optionally `item_masks`. In the panel pick, among the nodes upstream of `result`:
|
||||
- inject: a node whose output is replaced, on iteration i, by item i (e.g. a Load Image that stands for "the frame",
|
||||
or a mask loader for "its mask"),
|
||||
- params: widgets that take one value per iteration (a seed, a prompt line, a strength...), cycled if shorter.
|
||||
On run the node does not compute `result` itself (lazy input): it expands one copy per iteration of the part of the graph
|
||||
that depends on the injected nodes or the changed widgets; models and everything else are shared and run once.
|
||||
|
||||
State lives in the hidden `loop` STRING widget (JSON), saved with the workflow:
|
||||
{"inject": [{"node": "6", "slot": 0, "source": "items"}],
|
||||
"params": [{"node": "34", "widget": "noise_seed", "values": [1, 2, 3]}]}
|
||||
"""
|
||||
import json
|
||||
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
|
||||
from .bfs_xy_sweep import _cast
|
||||
|
||||
CATEGORY = "BFS/Loop"
|
||||
MAX_ITEMS = 256
|
||||
|
||||
|
||||
def parse_loop(text):
|
||||
try:
|
||||
s = json.loads(text or "{}") or {}
|
||||
except Exception:
|
||||
s = {}
|
||||
inject = [{"node": str(i["node"]), "slot": int(i.get("slot", 0)), "source": i.get("source", "items")}
|
||||
for i in s.get("inject", []) if isinstance(i, dict) and i.get("node") not in (None, "")]
|
||||
params = [{"node": str(p["node"]), "widget": str(p["widget"]), "values": list(p["values"])}
|
||||
for p in s.get("params", []) if isinstance(p, dict) and p.get("node") not in (None, "") and p.get("widget")
|
||||
and isinstance(p.get("values"), list) and p["values"]]
|
||||
return inject, params
|
||||
|
||||
|
||||
class BFSWorkflowLoop:
|
||||
DESCRIPTION = __doc__.split("\n\n")[0]
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {
|
||||
"result": ("IMAGE", {"lazy": True, "tooltip": "What one iteration produces; everything upstream of it can be looped."}),
|
||||
"loop": ("STRING", {"default": "{}", "multiline": True, "tooltip": "Set by the panel: injected nodes and per-iteration widgets."}),
|
||||
}, "optional": {
|
||||
"items": ("IMAGE", {"tooltip": "The batch to iterate over: item i is injected on iteration i."}),
|
||||
"items_b": ("IMAGE", {"tooltip": "A second batch, same length (e.g. the marked frames next to the plain ones)."}),
|
||||
"item_masks": ("MASK", {"tooltip": "One mask per item, injected where the panel says (source 'item_masks')."}),
|
||||
"count": ("INT", {"default": 0, "min": 0, "max": MAX_ITEMS,
|
||||
"tooltip": "Iterations when there are no items (params only); 0 = as many as the longest value list."}),
|
||||
}, "hidden": {"dynprompt": "DYNPROMPT", "unique_id": "UNIQUE_ID"}}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "INT")
|
||||
RETURN_NAMES = ("images", "count")
|
||||
OUTPUT_TOOLTIPS = ("The result of every iteration, in order, as one batch (resized to the first one's size).", "Iterations run.")
|
||||
FUNCTION = "run_loop"
|
||||
CATEGORY = CATEGORY
|
||||
|
||||
def check_lazy_status(self, result=None, **kw):
|
||||
return []
|
||||
|
||||
def run_loop(self, loop="{}", result=None, items=None, items_b=None, item_masks=None, count=0, dynprompt=None, unique_id=None):
|
||||
inject, params = parse_loop(loop)
|
||||
if not inject and not params:
|
||||
raise ValueError("BFS Workflow Loop: pick a node to inject the items into, or a widget to change, in the panel.")
|
||||
if dynprompt is None:
|
||||
raise ValueError("BFS Workflow Loop needs ComfyUI's dynamic graph execution")
|
||||
from comfy_execution.graph_utils import GraphBuilder, is_link
|
||||
me = str(unique_id); mine = dynprompt.get_node(me)["inputs"]
|
||||
link = mine.get("result")
|
||||
if not is_link(link):
|
||||
raise ValueError("Connect what one iteration produces to `result`.")
|
||||
n = int(items.shape[0]) if items is not None else (int(count) or max(len(p["values"]) for p in params) if params else 0)
|
||||
if any(i["source"] == "items" for i in inject) and items is None:
|
||||
raise ValueError("The panel injects `items`, but nothing is connected to items.")
|
||||
if any(i["source"] == "items_b" for i in inject):
|
||||
if items_b is None:
|
||||
raise ValueError("The panel injects `items_b`, but nothing is connected to items_b.")
|
||||
if items is not None and items_b.shape[0] != n:
|
||||
raise ValueError(f"{items_b.shape[0]} items_b for {n} items.")
|
||||
n = n or int(items_b.shape[0])
|
||||
if any(i["source"] == "item_masks" for i in inject):
|
||||
if item_masks is None:
|
||||
raise ValueError("The panel injects `item_masks`, but nothing is connected to item_masks.")
|
||||
if items is not None and item_masks.shape[0] != n:
|
||||
raise ValueError(f"{item_masks.shape[0]} masks for {n} items.")
|
||||
if n < 1:
|
||||
raise ValueError("Nothing to iterate over: connect items or give parameter values.")
|
||||
if n > MAX_ITEMS:
|
||||
raise ValueError(f"{n} iterations; at most {MAX_ITEMS}.")
|
||||
root = str(link[0])
|
||||
up, feeds, stack = set(), {}, [root]
|
||||
while stack:
|
||||
nid = stack.pop()
|
||||
if nid in up:
|
||||
continue
|
||||
up.add(nid)
|
||||
for v in dynprompt.get_node(nid).get("inputs", {}).values():
|
||||
if is_link(v):
|
||||
p = str(v[0]); feeds.setdefault(p, set()).add(nid); stack.append(p)
|
||||
for i in inject:
|
||||
if i["node"] not in up:
|
||||
raise ValueError(f"Injected node {i['node']} is not upstream of `result` (pick it again in the panel).")
|
||||
for p in params:
|
||||
if p["node"] not in up:
|
||||
raise ValueError(f"Node {p['node']} is not upstream of `result` (pick it again in the panel).")
|
||||
if p["widget"] not in dynprompt.get_node(p["node"]).get("inputs", {}):
|
||||
raise ValueError(f"Node {p['node']} has no widget '{p['widget']}' in this prompt.")
|
||||
injected = {(i["node"], i["slot"]): i["source"] for i in inject}
|
||||
# the body: everything that depends on an injected output or a changed widget and leads to `result`
|
||||
seeds = [p["node"] for p in params] + [c for i in inject for c in feeds.get(i["node"], ()) if c in up]
|
||||
body, stack = set(), list(seeds)
|
||||
while stack:
|
||||
nid = stack.pop()
|
||||
if nid in body:
|
||||
continue
|
||||
body.add(nid)
|
||||
stack.extend(c for c in feeds.get(nid, ()) if c in up)
|
||||
body -= {i["node"] for i in inject if i["node"] not in {p["node"] for p in params}}
|
||||
graph = GraphBuilder()
|
||||
outs = []
|
||||
for it in range(n):
|
||||
picks = {}
|
||||
for src in set(injected.values()):
|
||||
pk = graph.node("BFSLoopPick", f"pick{it}_{src}")
|
||||
if src == "item_masks":
|
||||
pk.set_input("masks", mine["item_masks"])
|
||||
else:
|
||||
pk.set_input("images", mine[src])
|
||||
pk.set_input("index", it)
|
||||
picks[src] = pk
|
||||
over = {(p["node"], p["widget"]): p["values"][it % len(p["values"])] for p in params}
|
||||
clones = {nid: graph.node(dynprompt.get_node(nid)["class_type"], f"i{it}_{nid}") for nid in body}
|
||||
for nid, node in clones.items():
|
||||
node.set_override_display_id(dynprompt.get_display_node_id(nid))
|
||||
for key, value in dynprompt.get_node(nid).get("inputs", {}).items():
|
||||
if (nid, key) in over:
|
||||
node.set_input(key, _cast(over[(nid, key)], value))
|
||||
elif is_link(value) and (str(value[0]), int(value[1])) in injected:
|
||||
src = injected[(str(value[0]), int(value[1]))]
|
||||
node.set_input(key, picks[src].out(1 if src == "item_masks" else 0))
|
||||
elif is_link(value) and str(value[0]) in body:
|
||||
node.set_input(key, clones[str(value[0])].out(value[1]))
|
||||
else:
|
||||
node.set_input(key, value)
|
||||
if root in clones:
|
||||
outs.append(clones[root].out(link[1]))
|
||||
elif (root, int(link[1])) in injected:
|
||||
src = injected[(root, int(link[1]))]; outs.append(picks[src].out(1 if src == "item_masks" else 0))
|
||||
else:
|
||||
outs.append(link)
|
||||
coll = graph.node("BFSLoopCollect", "collect")
|
||||
for i, o in enumerate(outs):
|
||||
coll.set_input(f"image_{i}", o)
|
||||
coll.set_input("n", n)
|
||||
return {"result": (coll.out(0), coll.out(1)), "expand": graph.finalize()}
|
||||
|
||||
|
||||
class BFSLoopPick:
|
||||
"""Internal: item `index` of a batch (image and/or mask)."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {"index": ("INT", {"default": 0, "min": 0, "max": 100000})},
|
||||
"optional": {"images": ("IMAGE",), "masks": ("MASK",)}}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "MASK")
|
||||
FUNCTION = "pick"
|
||||
CATEGORY = CATEGORY
|
||||
|
||||
def pick(self, index, images=None, masks=None):
|
||||
img = images[index:index + 1] if images is not None else torch.zeros(1, 64, 64, 3)
|
||||
if masks is not None:
|
||||
m = masks if masks.ndim == 3 else masks[None]
|
||||
msk = m[index:index + 1]
|
||||
else:
|
||||
msk = torch.zeros(1, img.shape[1], img.shape[2])
|
||||
return img, msk
|
||||
|
||||
|
||||
class BFSLoopCollect:
|
||||
"""Internal: every iteration's result, in order, as one batch."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {"n": ("INT", {"default": 1, "min": 1, "max": MAX_ITEMS})},
|
||||
"optional": {f"image_{i}": ("IMAGE",) for i in range(MAX_ITEMS)}}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "INT")
|
||||
FUNCTION = "collect"
|
||||
CATEGORY = CATEGORY
|
||||
|
||||
def collect(self, n, **kw):
|
||||
parts = [kw[f"image_{i}"] for i in range(n)]
|
||||
h, w = parts[0].shape[1:3]
|
||||
parts = [p if p.shape[1:3] == (h, w) else
|
||||
F.interpolate(p.movedim(-1, 1), size=(h, w), mode="bilinear", align_corners=False).movedim(1, -1) for p in parts]
|
||||
return torch.cat(parts), n
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"BFSWorkflowLoop": BFSWorkflowLoop, "BFSLoopPick": BFSLoopPick, "BFSLoopCollect": BFSLoopCollect}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"BFSWorkflowLoop": "BFS Workflow Loop", "BFSLoopPick": "BFS Loop Pick (internal)",
|
||||
"BFSLoopCollect": "BFS Loop Collect (internal)"}
|
||||
|
||||
|
||||
class BFSLoopSlot:
|
||||
"""A stand-in for "the current item" inside a BFS Workflow Loop body: pick it in the loop's Inject list and it gets
|
||||
item i on iteration i. Outside a loop it gives a small gray image (no file to load, so validation never fails)."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {"label": ("STRING", {"default": "item", "tooltip": "Only a name, to tell slots apart in the loop's list."})}}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "MASK")
|
||||
RETURN_NAMES = ("image", "mask")
|
||||
FUNCTION = "slot"
|
||||
CATEGORY = CATEGORY
|
||||
|
||||
def slot(self, label="item"):
|
||||
return torch.full((1, 64, 64, 3), 0.5), torch.zeros(1, 64, 64)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS["BFSLoopSlot"] = BFSLoopSlot
|
||||
NODE_DISPLAY_NAME_MAPPINGS["BFSLoopSlot"] = "BFS Loop Slot"
|
||||
+241
@@ -0,0 +1,241 @@
|
||||
"""BFS XY Sweep: run any workflow over a grid of widget values and get one labeled grid back.
|
||||
|
||||
Connect the IMAGE you want to compare (a decoded video or an image) to `result`. In the panel, pick for X (and
|
||||
optionally Y) any widget of any node upstream of that input (the panel lists them, like TinyTerra's advanced xyPlot)
|
||||
and type the values. On run the node does not compute `result` itself (lazy input): it expands one copy of the part of
|
||||
the graph that depends on the swept widgets per combination (everything else, models included, is shared and runs
|
||||
once), then a collector builds the grid.
|
||||
|
||||
All state lives in the hidden `sweep` STRING widget (JSON), saved with the workflow:
|
||||
{"x": {"node": "44", "widget": "lora_name", "values": ["a.safetensors", "b.safetensors"]},
|
||||
"y": {"node": "101", "widget": "keyframe_until", "values": [0.5, 0.7, 0.9]}}
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
|
||||
try:
|
||||
import folder_paths
|
||||
except Exception: # tests run without ComfyUI
|
||||
folder_paths = None
|
||||
|
||||
CATEGORY = "BFS/Loop"
|
||||
MAX_CELLS = 64
|
||||
|
||||
|
||||
def parse_sweep(text):
|
||||
try:
|
||||
s = json.loads(text or "{}") or {}
|
||||
except Exception:
|
||||
s = {}
|
||||
axes = []
|
||||
for key in ("x", "y"):
|
||||
a = s.get(key) or {}
|
||||
if a.get("node") not in (None, "") and a.get("widget") and isinstance(a.get("values"), list) and a["values"]:
|
||||
axes.append({"node": str(a["node"]), "widget": str(a["widget"]), "values": list(a["values"]), "title": a.get("title", "")})
|
||||
return axes
|
||||
|
||||
|
||||
def _cast(value, like):
|
||||
"""A typed value from the panel, cast to the type the widget had in the prompt."""
|
||||
if isinstance(like, bool):
|
||||
return value if isinstance(value, bool) else str(value).strip().lower() in ("1", "true", "yes", "on")
|
||||
if isinstance(like, int) and not isinstance(like, bool):
|
||||
return int(round(float(value)))
|
||||
if isinstance(like, float):
|
||||
return float(value)
|
||||
return value if isinstance(value, str) else str(value)
|
||||
|
||||
|
||||
def _short(v):
|
||||
s = str(v)
|
||||
if "/" in s or "\\" in s:
|
||||
s = s.replace("\\", "/").rsplit("/", 1)[-1]
|
||||
for ext in (".safetensors", ".ckpt", ".pt", ".png", ".jpg", ".mp4"):
|
||||
if s.endswith(ext):
|
||||
s = s[: -len(ext)]
|
||||
return s if len(s) <= 40 else "…" + s[-39:]
|
||||
|
||||
|
||||
def _font(size):
|
||||
for name in ("DejaVuSans.ttf", "arial.ttf", "LiberationSans-Regular.ttf"):
|
||||
try:
|
||||
return ImageFont.truetype(name, size)
|
||||
except Exception:
|
||||
continue
|
||||
return ImageFont.load_default()
|
||||
|
||||
|
||||
class BFSXYSweep:
|
||||
DESCRIPTION = __doc__.split("\n\n")[0]
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {
|
||||
"result": ("IMAGE", {"lazy": True, "tooltip": "The image or decoded video to compare; the sweep runs everything upstream of it."}),
|
||||
"frames_per_cell": ("INT", {"default": 4, "min": 1, "max": 16, "tooltip": "For videos: how many frames (evenly spread) each cell shows."}),
|
||||
"cell_height": ("INT", {"default": 256, "min": 64, "max": 1024, "step": 16}),
|
||||
"save_videos": ("BOOLEAN", {"default": True, "tooltip": "Save every multi-frame result as an mp4 in output/bfs_sweep/, named by its values."}),
|
||||
"fps": ("FLOAT", {"default": 24.0, "min": 1.0, "max": 120.0}),
|
||||
"sweep": ("STRING", {"default": "{}", "multiline": True, "tooltip": "Set by the panel: the swept widgets and their values."}),
|
||||
}, "hidden": {"dynprompt": "DYNPROMPT", "unique_id": "UNIQUE_ID"}}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "IMAGE", "STRING")
|
||||
RETURN_NAMES = ("grid", "results", "labels")
|
||||
OUTPUT_TOOLTIPS = ("The labeled grid.", "Every result (the cells' frames), resized to one size.", "JSON: the values of every cell, in order.")
|
||||
FUNCTION = "sweep_run"
|
||||
CATEGORY = CATEGORY
|
||||
OUTPUT_NODE = True
|
||||
|
||||
def check_lazy_status(self, result=None, **kw):
|
||||
return [] # never evaluate `result` here: the expansion evaluates one copy per combination
|
||||
|
||||
def sweep_run(self, result=None, frames_per_cell=4, cell_height=256, save_videos=True, fps=24.0, sweep="{}",
|
||||
dynprompt=None, unique_id=None):
|
||||
axes = parse_sweep(sweep)
|
||||
if not axes:
|
||||
raise ValueError("BFS XY Sweep: choose at least the X widget and its values in the panel.")
|
||||
if dynprompt is None:
|
||||
raise ValueError("BFS XY Sweep needs ComfyUI's dynamic graph execution")
|
||||
from comfy_execution.graph_utils import GraphBuilder, is_link
|
||||
me = str(unique_id)
|
||||
link = dynprompt.get_node(me)["inputs"].get("result")
|
||||
if not is_link(link):
|
||||
raise ValueError("Connect the image to compare to `result`.")
|
||||
root = str(link[0])
|
||||
# nodes upstream of `result` and who feeds whom inside that set
|
||||
up, feeds, stack = set(), {}, [root]
|
||||
while stack:
|
||||
nid = stack.pop()
|
||||
if nid in up:
|
||||
continue
|
||||
up.add(nid)
|
||||
for v in dynprompt.get_node(nid).get("inputs", {}).values():
|
||||
if is_link(v):
|
||||
p = str(v[0]); feeds.setdefault(p, set()).add(nid); stack.append(p)
|
||||
for a in axes:
|
||||
if a["node"] not in up:
|
||||
raise ValueError(f"Swept node {a['node']} is not upstream of `result` (pick it again in the panel).")
|
||||
if a["widget"] not in dynprompt.get_node(a["node"]).get("inputs", {}):
|
||||
raise ValueError(f"Node {a['node']} has no widget '{a['widget']}' in this prompt.")
|
||||
# the body: what depends on a swept node and leads to `result` (cloned per cell); the rest is shared
|
||||
body, stack = set(), [a["node"] for a in axes]
|
||||
while stack:
|
||||
nid = stack.pop()
|
||||
if nid in body:
|
||||
continue
|
||||
body.add(nid)
|
||||
stack.extend(c for c in feeds.get(nid, ()) if c in up)
|
||||
xs = axes[0]["values"]; ys = axes[1]["values"] if len(axes) > 1 else [None]
|
||||
cells = [(x, y) for y in ys for x in xs]
|
||||
if len(cells) > MAX_CELLS:
|
||||
raise ValueError(f"{len(cells)} combinations; at most {MAX_CELLS}.")
|
||||
graph = GraphBuilder()
|
||||
outs, labels = [], []
|
||||
for ci, (x, y) in enumerate(cells):
|
||||
over = {(axes[0]["node"], axes[0]["widget"]): x}
|
||||
if y is not None:
|
||||
over[(axes[1]["node"], axes[1]["widget"])] = y
|
||||
clones = {n: graph.node(dynprompt.get_node(n)["class_type"], f"c{ci}_{n}") for n in body}
|
||||
for nid, node in clones.items():
|
||||
node.set_override_display_id(dynprompt.get_display_node_id(nid))
|
||||
for key, value in dynprompt.get_node(nid).get("inputs", {}).items():
|
||||
if (nid, key) in over:
|
||||
node.set_input(key, _cast(over[(nid, key)], value))
|
||||
elif is_link(value) and str(value[0]) in body:
|
||||
node.set_input(key, clones[str(value[0])].out(value[1]))
|
||||
else:
|
||||
node.set_input(key, value)
|
||||
outs.append(clones[root].out(link[1]) if root in clones else link)
|
||||
labels.append({"x": x, "y": y})
|
||||
meta = {"axes": [{k: a[k] for k in ("node", "widget", "title")} for a in axes], "cells": labels,
|
||||
"xs": xs, "ys": ys if len(axes) > 1 else [], "frames_per_cell": frames_per_cell, "cell_height": cell_height,
|
||||
"save_videos": save_videos, "fps": fps}
|
||||
coll = graph.node("BFSXYSweepCollect", "collect")
|
||||
coll.set_input("meta", json.dumps(meta))
|
||||
for i, o in enumerate(outs):
|
||||
coll.set_input(f"image_{i}", o)
|
||||
return {"result": (coll.out(0), coll.out(1), coll.out(2)), "expand": graph.finalize()}
|
||||
|
||||
|
||||
class BFSXYSweepCollect:
|
||||
"""Internal: receives every cell's result from BFS XY Sweep and draws the grid."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {"meta": ("STRING", {"default": "{}"})},
|
||||
"optional": {f"image_{i}": ("IMAGE",) for i in range(MAX_CELLS)}}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "IMAGE", "STRING")
|
||||
RETURN_NAMES = ("grid", "results", "labels")
|
||||
FUNCTION = "collect"
|
||||
CATEGORY = CATEGORY
|
||||
OUTPUT_NODE = True
|
||||
|
||||
def collect(self, meta, **kw):
|
||||
m = json.loads(meta)
|
||||
cells = m["cells"]; k = int(m["frames_per_cell"]); ch = int(m["cell_height"])
|
||||
imgs = [kw.get(f"image_{i}") for i in range(len(cells))]
|
||||
strips, saved = [], []
|
||||
for i, (img, lab) in enumerate(zip(imgs, cells)):
|
||||
idx = np.unique(np.linspace(0, img.shape[0] - 1, min(k, img.shape[0])).round().astype(int))
|
||||
fr = img[idx, ..., :3].float()
|
||||
w = max(1, round(fr.shape[2] * ch / fr.shape[1]))
|
||||
fr = F.interpolate(fr.movedim(-1, 1), size=(ch, w), mode="bilinear", align_corners=False).movedim(1, -1)
|
||||
strips.append(torch.cat(list(fr), dim=1))
|
||||
if m.get("save_videos") and img.shape[0] > 1 and folder_paths:
|
||||
saved.append(self._save(img, lab, m, i))
|
||||
sw = max(s.shape[1] for s in strips)
|
||||
strips = [F.pad(s, (0, 0, 0, sw - s.shape[1])) for s in strips]
|
||||
xs, ys = m["xs"], m["ys"] or [None]
|
||||
ax = m["axes"]
|
||||
xl = f"{ax[0]['title'] or ax[0]['node']}.{ax[0]['widget']}"
|
||||
yl = f"{ax[1]['title'] or ax[1]['node']}.{ax[1]['widget']}" if len(ax) > 1 else ""
|
||||
pad, top, left = 6, 34, (170 if len(ax) > 1 else 0)
|
||||
H = top + len(ys) * (ch + pad); W = left + len(xs) * (sw + pad)
|
||||
canvas = Image.new("RGB", (W, H), (24, 24, 24)); d = ImageDraw.Draw(canvas); f = _font(16)
|
||||
for c, x in enumerate(xs):
|
||||
d.text((left + c * (sw + pad) + 4, 8), f"{xl} = {_short(x)}", fill=(230, 230, 230), font=f)
|
||||
for r, y in enumerate(ys):
|
||||
if y is not None:
|
||||
d.text((6, top + r * (ch + pad) + ch // 2 - 10), f"{yl} =\n{_short(y)}", fill=(230, 230, 230), font=f)
|
||||
for c in range(len(xs)):
|
||||
s = strips[r * len(xs) + c]
|
||||
arr = (s.clamp(0, 1).cpu().numpy() * 255).astype(np.uint8)
|
||||
canvas.paste(Image.fromarray(arr), (left + c * (sw + pad), top + r * (ch + pad)))
|
||||
grid = torch.from_numpy(np.asarray(canvas, np.float32) / 255.0)[None]
|
||||
results = torch.stack(strips)
|
||||
info = {"cells": cells, "videos": saved}
|
||||
ui = {}
|
||||
if folder_paths:
|
||||
out = os.path.join(folder_paths.get_output_directory(), "bfs_sweep"); os.makedirs(out, exist_ok=True)
|
||||
name = f"grid_{len(os.listdir(out)):05d}.png"; canvas.save(os.path.join(out, name))
|
||||
ui = {"images": [{"filename": name, "subfolder": "bfs_sweep", "type": "output"}]}
|
||||
return {"ui": ui, "result": (grid, results, json.dumps(info))}
|
||||
|
||||
@staticmethod
|
||||
def _save(img, lab, m, i):
|
||||
import av
|
||||
out = os.path.join(folder_paths.get_output_directory(), "bfs_sweep"); os.makedirs(out, exist_ok=True)
|
||||
tag = "_".join(_short(v).replace(" ", "") for v in (lab["x"], lab["y"]) if v is not None)
|
||||
tag = "".join(ch if ch.isalnum() or ch in "._-" else "-" for ch in tag)[:80]
|
||||
path = os.path.join(out, f"{i:02d}_{tag}.mp4")
|
||||
frames = (img[..., :3].clamp(0, 1).cpu().numpy() * 255).astype(np.uint8)
|
||||
h, w = frames.shape[1] // 2 * 2, frames.shape[2] // 2 * 2
|
||||
with av.open(path, "w") as c:
|
||||
st = c.add_stream("libx264", rate=int(round(float(m.get("fps", 24)))))
|
||||
st.width, st.height, st.pix_fmt = w, h, "yuv420p"; st.options = {"crf": "16"}
|
||||
for fr in frames:
|
||||
for p in st.encode(av.VideoFrame.from_ndarray(fr[:h, :w], format="rgb24")):
|
||||
c.mux(p)
|
||||
for p in st.encode():
|
||||
c.mux(p)
|
||||
return path
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"BFSXYSweep": BFSXYSweep, "BFSXYSweepCollect": BFSXYSweepCollect}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"BFSXYSweep": "BFS XY Sweep", "BFSXYSweepCollect": "BFS XY Sweep Collect (internal)"}
|
||||
+2
-2
@@ -1,7 +1,7 @@
|
||||
[project]
|
||||
name = "bfsnodes"
|
||||
description = "Custom ComfyUI nodes for reference-guided video: BFS Sampler (guide-driven IC-LoRAs, auto crop, mask inpainting, temporal chunking), LTX Multiple Controls / Identity Transfer / Edit Anything, Head-Swap and Anime2Real conditioning, MiniMax-H3 latent guides, RoPE layouts and loaders, LoRA Surgery, and Shot Planner / Shot Loop (tabbed panel): split long videos into shots at camera cuts, face-tracked cast, SAM 3 mask per shot with full frame, paste, mask, crop and crop + mask generation modes, optional translucent Colored Mark overlay for manually selected regions, optional Qwen3-VL to suggest settings, describe references ({details}) and the person to replace ({target}), continuity between shots, H3 conditioning with an optional duet and prompt writer, join with the original timing, audio and a side-by-side comparison. H3 Duet: pin a source clip beside the video, training-free."
|
||||
version = "1.118.1"
|
||||
description = "Custom ComfyUI nodes for reference-guided video: BFS Video Segment Picker (SAM 3 multi-subject video segmentation, shot-aware propagation, Molmo2 visual re-anchoring, and normal-map / DWPose hybrid guides), BFS H3 Character Swap Conditioning (keyframe pins, normal map guide, and DWPose guide), BFS Sampler (guide-driven IC-LoRAs, auto crop, mask inpainting, temporal chunking), LTX Multiple Controls / Identity Transfer / Edit Anything, Head-Swap and Anime2Real conditioning, MiniMax-H3 latent guides, RoPE layouts and loaders, LoRA Surgery, and Shot Planner / Shot Loop (tabbed panel): split long videos into shots at camera cuts, face-tracked cast, SAM 3 mask per shot with full frame, paste, mask, crop and crop + mask generation modes, optional translucent Colored Mark overlay for manually selected regions, optional Qwen3-VL to suggest settings, describe references ({details}) and the person to replace ({target}), continuity between shots, H3 conditioning with an optional duet and prompt writer, join with the original timing, audio and a side-by-side comparison. H3 Duet: pin a source clip beside the video, training-free."
|
||||
version = "1.119.3"
|
||||
license = {file = "LICENSE"}
|
||||
dependencies = [
|
||||
"torch",
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
import importlib.util
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SPEC = importlib.util.spec_from_file_location("bfs_h3_swap_conditioning", ROOT / "bfs_h3_swap_conditioning.py")
|
||||
SC = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(SC)
|
||||
NODE = SC.BFSH3KeyframeConditioning
|
||||
|
||||
|
||||
class InputsTest(unittest.TestCase):
|
||||
def test_modes(self):
|
||||
self.assertEqual(SC.MODES, ["keyframe pins", "normal map guide", "DWPose guide"])
|
||||
|
||||
def test_pictures_grouped_first(self):
|
||||
t = NODE.INPUT_TYPES()
|
||||
self.assertEqual(list(t["required"])[:7], ["clip", "vae", "guide_video", "width", "height", "length", "picture_1"])
|
||||
self.assertEqual(list(t["optional"])[:5], ["picture_2", "picture_3", "model", "keyframes", "keyframe_spec"])
|
||||
|
||||
def test_outputs(self):
|
||||
self.assertEqual(len(NODE.RETURN_TYPES), len(NODE.RETURN_NAMES))
|
||||
self.assertEqual(len(NODE.OUTPUT_IS_LIST), len(NODE.RETURN_TYPES))
|
||||
self.assertEqual(NODE.RETURN_NAMES[-1], "pictures")
|
||||
self.assertTrue(NODE.OUTPUT_IS_LIST[-1])
|
||||
|
||||
def test_unknown_mode(self):
|
||||
import torch
|
||||
with self.assertRaises(ValueError):
|
||||
NODE().encode(None, None, torch.zeros(1, 8, 8, 3), 8, 8, 1, torch.zeros(1, 8, 8, 3), mode="gray video")
|
||||
|
||||
|
||||
class PromptTest(unittest.TestCase):
|
||||
def test_guide_prompts_name_each_subject_once(self):
|
||||
for fn, word in ((SC.normals_caption, "normal map"), (SC.pose_caption, "pose skeleton")):
|
||||
for k in range(1, SC.MAX_PICTURES + 1):
|
||||
p = fn(k)
|
||||
self.assertTrue(p.startswith("character_swap:\nsubject_definitions:\n"))
|
||||
for j in range(1, k + 1):
|
||||
self.assertIn(f"<Subject {j}> is the subject marked by the {SC.SUBJECT_COLORS[j - 1]} {word}", p)
|
||||
self.assertNotIn(f"<Picture {k + 1}>", p)
|
||||
|
||||
def test_pins_prompt(self):
|
||||
p = SC.pins_caption(2, [0.0, 1.5])
|
||||
self.assertIn("2 keyframes of the target video are pinned at 0.0s, 1.5s", p)
|
||||
self.assertIn("<Picture 2>", p)
|
||||
self.assertIn("1 keyframe of the target video is pinned at 2.0s", SC.pins_caption(1, [2.0]))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,302 @@
|
||||
/**
|
||||
* BFS Keyframe Planner: the panel of the node. A small player of the guide (frames from the node's last run), a timeline
|
||||
* with the keyframes on it, one card per keyframe (source frame + an optional edited frame, dropped or uploaded) and up
|
||||
* to 4 reference pictures.
|
||||
*
|
||||
* All state lives in the hidden `plan` STRING widget (JSON: keyframe frames, edited frame and picture paths in input/),
|
||||
* so a workflow saves, reloads and shares the plan. Uploaded images go to input/bfs_keyframes/.
|
||||
*/
|
||||
import { app } from "../../scripts/app.js";
|
||||
import { api } from "../../scripts/api.js";
|
||||
import { wheelToCanvas } from "./bfs_wheel.js";
|
||||
import { createApp, h, ref, reactive, computed, watch, onBeforeUnmount } from "./vendor/vue.esm-browser.prod.mjs";
|
||||
|
||||
const DEFAULT_W = 560, DEFAULT_H = 640, MAX_PICTURES = 4, SUBFOLDER = "bfs_keyframes";
|
||||
|
||||
const tempUrl = (d, f) => api.apiURL(`/view?filename=${encodeURIComponent(f)}&subfolder=${encodeURIComponent(d.subfolder)}&type=${d.type || "temp"}`);
|
||||
const inputUrl = p => {
|
||||
const i = p.lastIndexOf("/");
|
||||
return api.apiURL(`/view?filename=${encodeURIComponent(p.slice(i + 1))}&subfolder=${encodeURIComponent(i < 0 ? "" : p.slice(0, i))}&type=input`);
|
||||
};
|
||||
|
||||
function parsePlan(text) {
|
||||
let s = {};
|
||||
try { s = JSON.parse(text || "{}") || {}; } catch { s = {}; }
|
||||
const keyframes = (Array.isArray(s.keyframes) ? s.keyframes : []).filter(k => Number.isFinite(+k.frame))
|
||||
.map(k => ({ frame: Math.round(+k.frame), edit: k.edit || null }));
|
||||
const pictures = (Array.isArray(s.pictures) ? s.pictures : []).slice(0, MAX_PICTURES).map(p => p || null);
|
||||
while (pictures.length < MAX_PICTURES) pictures.push(null);
|
||||
return { keyframes, pictures };
|
||||
}
|
||||
|
||||
async function upload(file) {
|
||||
const body = new FormData();
|
||||
body.append("image", file);
|
||||
body.append("subfolder", SUBFOLDER);
|
||||
body.append("type", "input");
|
||||
const r = await api.fetchApi("/upload/image", { method: "POST", body });
|
||||
if (r.status !== 200) throw new Error(`upload failed (${r.status})`);
|
||||
const j = await r.json();
|
||||
return j.subfolder ? `${j.subfolder}/${j.name}` : j.name;
|
||||
}
|
||||
|
||||
function pickFile() {
|
||||
return new Promise(resolve => {
|
||||
const inp = document.createElement("input");
|
||||
inp.type = "file"; inp.accept = "image/*";
|
||||
inp.onchange = () => resolve(inp.files?.[0] || null);
|
||||
inp.click();
|
||||
});
|
||||
}
|
||||
|
||||
let styled = false;
|
||||
function styles() {
|
||||
if (styled) return;
|
||||
styled = true;
|
||||
const css = document.createElement("style");
|
||||
css.textContent = `
|
||||
.bkp{position:absolute;inset:0;display:flex;flex-direction:column;gap:6px;padding:6px;box-sizing:border-box;font:12px system-ui,sans-serif;color:#ddd}
|
||||
.bkp button{background:#333;color:#ddd;border:1px solid #555;border-radius:4px;padding:3px 8px;cursor:pointer;font:inherit;line-height:1.2}
|
||||
.bkp button:hover:not(:disabled){background:#444}
|
||||
.bkp button:disabled{opacity:.4;cursor:default}
|
||||
.bkp button.pri{background:#2d5a8a;border-color:#3d6fa5;color:#fff}
|
||||
.bkp button.danger:hover:not(:disabled){background:#6a2a2a}
|
||||
.bkp .stage{flex:1;min-height:150px;position:relative;display:flex;align-items:center;justify-content:center;background:#121212;border-radius:6px;overflow:hidden}
|
||||
.bkp .stage img{max-width:100%;max-height:100%;object-fit:contain;user-select:none;-webkit-user-drag:none}
|
||||
.bkp .stage.onkf{box-shadow:inset 0 0 0 2px #e0a030}
|
||||
.bkp .badge{position:absolute;left:6px;top:6px;background:rgba(0,0,0,.7);padding:2px 7px;border-radius:10px;font-variant-numeric:tabular-nums}
|
||||
.bkp .badge.kf{left:auto;right:6px;background:#e0a030;color:#111;font-weight:600}
|
||||
.bkp .empty{color:#888;text-align:center;padding:16px;line-height:1.7}
|
||||
.bkp .ctl{display:flex;gap:4px;align-items:center}
|
||||
.bkp .ctl .sp{flex:1}
|
||||
.bkp .time{font-variant-numeric:tabular-nums;color:#bbb;min-width:110px;text-align:center}
|
||||
.bkp .tl{position:relative;height:26px;background:#1d1d1d;border:1px solid #333;border-radius:5px;cursor:pointer;touch-action:none}
|
||||
.bkp .tl .fill{position:absolute;left:0;top:0;bottom:0;background:#2a3a4d;border-radius:5px 0 0 5px;pointer-events:none}
|
||||
.bkp .tl .head{position:absolute;top:-2px;bottom:-2px;width:2px;margin-left:-1px;background:#fff;pointer-events:none}
|
||||
.bkp .tl .mk{position:absolute;top:3px;width:10px;height:18px;margin-left:-5px;border-radius:3px;background:#e0a030;border:1px solid #111;cursor:pointer}
|
||||
.bkp .tl .mk.done{background:#3fbf6a}
|
||||
.bkp .tl .mk.cur{outline:2px solid #fff}
|
||||
.bkp .sec{display:flex;align-items:center;gap:6px;color:#999;font-size:11px;text-transform:uppercase;letter-spacing:.05em}
|
||||
.bkp .sec .n{color:#ccc}
|
||||
.bkp .row{display:flex;gap:6px;overflow-x:auto;padding-bottom:3px;min-height:96px}
|
||||
.bkp .card{flex:0 0 auto;display:flex;flex-direction:column;gap:3px;background:#1d1d1d;border:1px solid #383838;border-radius:6px;padding:4px}
|
||||
.bkp .card.cur{border-color:#e0a030}
|
||||
.bkp .card .pair{display:flex;gap:3px}
|
||||
.bkp .thumb{width:72px;height:54px;border-radius:4px;background:#111 center/cover no-repeat;position:relative;cursor:pointer;flex:0 0 auto}
|
||||
.bkp .thumb .lbl{position:absolute;left:2px;bottom:2px;font-size:9px;background:rgba(0,0,0,.7);padding:0 3px;border-radius:2px}
|
||||
.bkp .drop{white-space:pre-line;width:72px;height:54px;border:1px dashed #555;border-radius:4px;display:flex;align-items:center;justify-content:center;text-align:center;color:#888;font-size:10px;cursor:pointer;line-height:1.2;flex:0 0 auto}
|
||||
.bkp .drop:hover,.bkp .drop.over{border-color:#3fbf6a;color:#cfc}
|
||||
.bkp .card .foot{display:flex;align-items:center;gap:4px;font-size:11px;font-variant-numeric:tabular-nums}
|
||||
.bkp .card .foot .sp{flex:1}
|
||||
.bkp .x{color:#999;cursor:pointer;padding:0 3px;border-radius:3px}.bkp .x:hover{color:#f66;background:#2a1a1a}
|
||||
.bkp .pics{display:flex;gap:6px}
|
||||
.bkp .pic{position:relative}
|
||||
.bkp .pic .thumb{width:60px;height:60px}
|
||||
.bkp .pic .drop{width:60px;height:60px}
|
||||
.bkp .pic .x{position:absolute;right:1px;top:1px;background:rgba(0,0,0,.7)}
|
||||
.bkp .pic .tag{font-size:10px;color:#888;text-align:center}
|
||||
.bkp .hint{color:#777;font-size:11px}.bkp .err{color:#f77;font-size:11px}
|
||||
`;
|
||||
document.head.appendChild(css);
|
||||
}
|
||||
|
||||
function Panel(io) {
|
||||
const plan = reactive(parsePlan(io.get()));
|
||||
const strip = ref(null); // {subfolder, type, files, frames, count, fps, width, height} from the last run
|
||||
const pos = ref(0); // index into strip.frames
|
||||
const playing = ref(false);
|
||||
const err = ref("");
|
||||
const over = ref(""); // drop target being hovered
|
||||
let timer = null;
|
||||
|
||||
const save = () => io.set(JSON.stringify({
|
||||
keyframes: [...plan.keyframes].sort((a, b) => a.frame - b.frame),
|
||||
pictures: plan.pictures.map(p => p || null),
|
||||
}));
|
||||
watch(plan, save, { deep: true });
|
||||
|
||||
const count = computed(() => strip.value?.count || 0);
|
||||
const fps = computed(() => strip.value?.fps || io.fps() || 24);
|
||||
const frame = computed(() => strip.value ? strip.value.frames[pos.value] : 0);
|
||||
const sorted = computed(() => [...plan.keyframes].sort((a, b) => a.frame - b.frame));
|
||||
const curKf = computed(() => plan.keyframes.find(k => k.frame === frame.value) || null);
|
||||
const nearest = f => { // strip index of the frame nearest to f
|
||||
const fr = strip.value?.frames || [];
|
||||
let best = 0;
|
||||
for (let i = 1; i < fr.length; i++) if (Math.abs(fr[i] - f) < Math.abs(fr[best] - f)) best = i;
|
||||
return best;
|
||||
};
|
||||
const frameUrl = f => strip.value ? tempUrl(strip.value, strip.value.files[nearest(f)]) : "";
|
||||
const fmt = f => `${(f / fps.value).toFixed(2)}s`;
|
||||
|
||||
const stop = () => { playing.value = false; if (timer) { clearInterval(timer); timer = null; } };
|
||||
const play = () => {
|
||||
if (!strip.value) return;
|
||||
if (playing.value) return stop();
|
||||
playing.value = true;
|
||||
const fr = strip.value.frames;
|
||||
const step = fr.length > 1 ? (fr[fr.length - 1] - fr[0]) / (fr.length - 1) : 1;
|
||||
timer = setInterval(() => { pos.value = (pos.value + 1) % fr.length; }, 1000 * step / fps.value);
|
||||
};
|
||||
const seekIdx = i => { if (strip.value) pos.value = Math.max(0, Math.min(strip.value.frames.length - 1, i)); };
|
||||
const seekFrame = f => seekIdx(nearest(f));
|
||||
onBeforeUnmount(stop);
|
||||
|
||||
const addHere = () => {
|
||||
if (!strip.value || curKf.value) return;
|
||||
plan.keyframes.push({ frame: frame.value, edit: null });
|
||||
};
|
||||
const removeKf = k => { plan.keyframes.splice(plan.keyframes.indexOf(k), 1); };
|
||||
const jumpKf = d => {
|
||||
const list = sorted.value;
|
||||
if (!list.length) return;
|
||||
const f = frame.value;
|
||||
const t = d > 0 ? list.find(k => k.frame > f) || list[0] : [...list].reverse().find(k => k.frame < f) || list[list.length - 1];
|
||||
seekFrame(t.frame);
|
||||
};
|
||||
|
||||
const withFile = async (file, apply) => {
|
||||
if (!file) return;
|
||||
err.value = "";
|
||||
try { apply(await upload(file)); } catch (e) { err.value = String(e.message || e); }
|
||||
};
|
||||
const dropProps = (key, apply) => ({
|
||||
class: ["drop", over.value === key ? "over" : ""],
|
||||
onDragover: e => { e.preventDefault(); e.stopPropagation(); over.value = key; },
|
||||
onDragleave: () => { if (over.value === key) over.value = ""; },
|
||||
onDrop: e => { e.preventDefault(); e.stopPropagation(); over.value = ""; withFile(e.dataTransfer?.files?.[0], apply); },
|
||||
onClick: async e => { e.stopPropagation(); withFile(await pickFile(), apply); },
|
||||
});
|
||||
|
||||
// timeline: click / drag to seek
|
||||
const tlDown = e => {
|
||||
if (!strip.value) return;
|
||||
const el = e.currentTarget, r = el.getBoundingClientRect();
|
||||
const go = ev => {
|
||||
const x = Math.max(0, Math.min(1, (ev.clientX - r.left) / r.width));
|
||||
seekFrame(Math.round(x * Math.max(0, count.value - 1)));
|
||||
};
|
||||
stop(); go(e);
|
||||
el.setPointerCapture?.(e.pointerId);
|
||||
const move = ev => go(ev);
|
||||
const up = () => { el.removeEventListener("pointermove", move); el.removeEventListener("pointerup", up); };
|
||||
el.addEventListener("pointermove", move); el.addEventListener("pointerup", up);
|
||||
};
|
||||
const pct = f => `${count.value > 1 ? (100 * f / (count.value - 1)) : 0}%`;
|
||||
|
||||
io.expose({
|
||||
executed: d => {
|
||||
if (!d) return;
|
||||
const keep = frame.value;
|
||||
strip.value = d;
|
||||
seekFrame(keep);
|
||||
for (const f of d.files) { const im = new Image(); im.src = tempUrl(d, f); } // preload for smooth playback
|
||||
},
|
||||
reload: () => { Object.assign(plan, parsePlan(io.get())); },
|
||||
});
|
||||
|
||||
return () => {
|
||||
const s = strip.value;
|
||||
const cards = sorted.value.map((k, i) => h("div", { class: ["card", k.frame === frame.value ? "cur" : ""], key: k.frame }, [
|
||||
h("div", { class: "pair" }, [
|
||||
h("div", { class: "thumb", title: "source frame: click to go there", style: s ? { backgroundImage: `url("${frameUrl(k.frame)}")` } : {},
|
||||
onClick: () => { stop(); seekFrame(k.frame); } }, [h("span", { class: "lbl" }, "source")]),
|
||||
k.edit
|
||||
? h("div", { class: "thumb", title: "edited frame: click to replace", style: { backgroundImage: `url("${inputUrl(k.edit)}")` },
|
||||
onClick: async () => withFile(await pickFile(), p => { k.edit = p; }) }, [h("span", { class: "lbl" }, "edited")])
|
||||
: h("div", { ...dropProps(`kf${k.frame}`, p => { k.edit = p; }), title: "drop or click: the edited frame for this time (optional)" },
|
||||
"drop edited\nframe"),
|
||||
]),
|
||||
h("div", { class: "foot" }, [
|
||||
h("span", null, `#${i + 1} · ${fmt(k.frame)}`), h("span", { class: "sp" }),
|
||||
k.edit ? h("span", { class: "x", title: "clear the edited frame", onClick: () => { k.edit = null; } }, "⟲") : null,
|
||||
h("span", { class: "x", title: "remove this keyframe", onClick: () => removeKf(k) }, "✕"),
|
||||
]),
|
||||
]));
|
||||
const linked = j => !!io.node.inputs?.find(x => x.name === `picture_${j}`)?.link;
|
||||
const pics = plan.pictures.map((p, j) => h("div", { class: "pic", key: j }, [
|
||||
linked(j + 1)
|
||||
? h("div", { class: "drop", title: "this picture comes from the connected input" }, "input")
|
||||
: p
|
||||
? h("div", { class: "thumb", style: { backgroundImage: `url("${inputUrl(p)}")` }, title: "click to replace",
|
||||
onClick: async () => withFile(await pickFile(), q => { plan.pictures[j] = q; }) })
|
||||
: h("div", { ...dropProps(`pic${j}`, q => { plan.pictures[j] = q; }), title: "drop or click: a reference picture" }, "+ picture"),
|
||||
p && !linked(j + 1) ? h("span", { class: "x", title: "remove", onClick: () => { plan.pictures[j] = null; } }, "✕") : null,
|
||||
h("div", { class: "tag" }, `Picture ${j + 1}`),
|
||||
]));
|
||||
return h("div", { class: "bkp" }, [
|
||||
h("div", { class: ["stage", curKf.value ? "onkf" : ""] }, s
|
||||
? [h("img", { src: tempUrl(s, s.files[pos.value]), draggable: false }),
|
||||
h("span", { class: "badge" }, `${fmt(frame.value)} · frame ${frame.value}/${count.value - 1}`),
|
||||
curKf.value ? h("span", { class: "badge kf" }, curKf.value.edit ? "keyframe ✓" : "keyframe") : null]
|
||||
: [h("div", { class: "empty" }, ["Queue this node once to load the guide frames.", h("br"),
|
||||
"Then scrub, press + Keyframe, and drop the edited frames on the cards."])]),
|
||||
h("div", { class: "ctl" }, [
|
||||
h("button", { disabled: !s, title: "previous keyframe", onClick: () => { stop(); jumpKf(-1); } }, "⏮"),
|
||||
h("button", { disabled: !s, title: "previous frame", onClick: () => { stop(); seekIdx(pos.value - 1); } }, "◀"),
|
||||
h("button", { disabled: !s, title: playing.value ? "pause" : "play", onClick: play }, playing.value ? "⏸" : "▶"),
|
||||
h("button", { disabled: !s, title: "next frame", onClick: () => { stop(); seekIdx(pos.value + 1); } }, "▶|"),
|
||||
h("button", { disabled: !s, title: "next keyframe", onClick: () => { stop(); jumpKf(1); } }, "⏭"),
|
||||
h("span", { class: "time" }, s ? `${fmt(frame.value)} / ${fmt(count.value - 1)}` : "--"),
|
||||
h("span", { class: "sp" }),
|
||||
curKf.value
|
||||
? h("button", { class: "danger", onClick: () => removeKf(curKf.value) }, "- Keyframe")
|
||||
: h("button", { class: "pri", disabled: !s, onClick: addHere, title: "mark the current frame as a keyframe" }, "+ Keyframe"),
|
||||
]),
|
||||
h("div", { class: "tl", onPointerdown: tlDown }, s ? [
|
||||
h("div", { class: "fill", style: { width: pct(frame.value) } }),
|
||||
...sorted.value.map(k => h("div", { class: ["mk", k.edit ? "done" : "", k.frame === frame.value ? "cur" : ""], key: k.frame,
|
||||
style: { left: pct(k.frame) }, title: `${fmt(k.frame)}${k.edit ? " (edited)" : " (no edited frame yet)"}`,
|
||||
onPointerdown: e => { e.stopPropagation(); stop(); seekFrame(k.frame); } })),
|
||||
h("div", { class: "head", style: { left: pct(frame.value) } }),
|
||||
] : []),
|
||||
h("div", { class: "sec" }, ["Keyframes ", h("span", { class: "n" }, String(plan.keyframes.length)),
|
||||
plan.keyframes.length ? h("span", null, ` · ${plan.keyframes.filter(k => k.edit).length} edited`) : null]),
|
||||
h("div", { class: "row" }, cards.length ? cards : [h("div", { class: "hint" }, "No keyframes yet.")]),
|
||||
h("div", { class: "sec" }, "References"),
|
||||
h("div", { class: "pics" }, pics),
|
||||
err.value ? h("div", { class: "err" }, err.value) : h("div", { class: "hint" }, "Outputs follow the plan on the next run. Without edited frames, source_frames are ready to edit."),
|
||||
]);
|
||||
};
|
||||
}
|
||||
|
||||
app.registerExtension({
|
||||
name: "BFSNodes.KeyframePlanner",
|
||||
async nodeCreated(node) {
|
||||
if (node.comfyClass !== "BFSKeyframePlanner") return;
|
||||
styles();
|
||||
const widget = node.widgets?.find(w => w.name === "plan");
|
||||
if (widget) { // the JSON stays a widget (saved with the workflow) but is never shown
|
||||
widget.type = "hidden"; widget.hidden = true; widget.computeSize = () => [0, -4];
|
||||
widget.options = { ...(widget.options || {}), hidden: true, serialize: true };
|
||||
widget.serializeValue = () => widget.value;
|
||||
}
|
||||
const fpsW = node.widgets?.find(w => w.name === "fps");
|
||||
const host = document.createElement("div");
|
||||
host.style.cssText = "position:relative;width:100%;height:100%;min-width:440px;min-height:520px";
|
||||
node.addDOMWidget("bfs_keyframe_planner", "div", host, { serialize: false, hideOnZoom: false, hideInPanel: true });
|
||||
wheelToCanvas(host);
|
||||
const handle = {};
|
||||
createApp({
|
||||
setup: () => Panel({
|
||||
node,
|
||||
get: () => widget?.value ?? "{}",
|
||||
set: v => { if (widget) widget.value = v; node.graph?.setDirtyCanvas(true); },
|
||||
fps: () => Number(fpsW?.value) || 24,
|
||||
expose: o => Object.assign(handle, o),
|
||||
}),
|
||||
}).mount(host);
|
||||
const onExecuted = node.onExecuted;
|
||||
node.onExecuted = function (output) {
|
||||
onExecuted?.apply(this, arguments);
|
||||
handle.executed?.(output?.bfs_kfplan?.[0]);
|
||||
};
|
||||
const onConfigure = node.onConfigure;
|
||||
node.onConfigure = function () {
|
||||
const r = onConfigure?.apply(this, arguments);
|
||||
setTimeout(() => handle.reload?.(), 0);
|
||||
return r;
|
||||
};
|
||||
node.size = [Math.max(node.size[0], DEFAULT_W), Math.max(node.size[1], DEFAULT_H)];
|
||||
},
|
||||
});
|
||||
@@ -29,7 +29,7 @@ function parseState(text) {
|
||||
}
|
||||
|
||||
const STYLE_ID = "bfs-segpick-style";
|
||||
function styles() {
|
||||
export function styles() {
|
||||
if (document.getElementById(STYLE_ID)) return;
|
||||
const css = document.createElement("style");
|
||||
css.id = STYLE_ID;
|
||||
@@ -101,7 +101,7 @@ function loadMask(o, color) {
|
||||
});
|
||||
}
|
||||
|
||||
function Panel(io) {
|
||||
export function Panel(io) { // exported: BFS Video Segment Picker shows one per frame
|
||||
const state = reactive(parseState(io.get()));
|
||||
const data = ref(null); // {session, width, height, image, objects} from the last run
|
||||
const loaded = new Map(); // object id -> loadMask result (not reactive: canvases and big arrays)
|
||||
|
||||
@@ -0,0 +1,317 @@
|
||||
/**
|
||||
* BFS Video Segment Picker: a strip with every frame of the batch on top; click a frame to show it in the image
|
||||
* Segment Picker's canvas (the same Panel, one selection per frame). The strip marks each frame: ✓ picked by hand,
|
||||
* ↔ tracked from a picked frame, ⚠ lost or nothing. State: the hidden `selection` widget, {"frames": {"i": selection}}.
|
||||
*/
|
||||
import { app } from "../../scripts/app.js";
|
||||
import { api } from "../../scripts/api.js";
|
||||
import { wheelToCanvas } from "./bfs_wheel.js";
|
||||
import { Panel, styles as pickerStyles } from "./bfs_segment_picker.js";
|
||||
import { createApp, h, ref, reactive } from "./vendor/vue.esm-browser.prod.mjs";
|
||||
|
||||
const fileUrl = f => api.apiURL(`/view?filename=${encodeURIComponent(f.filename)}&subfolder=${encodeURIComponent(f.subfolder || "")}&type=${f.type || "temp"}`);
|
||||
|
||||
const SUBJECTS = [["1", "#eb2828", "red"], ["2", "#236eeb", "blue"], ["3", "#3ccd3c", "green"]];
|
||||
function parse(text) { // {frame: {subject: selection}}; a bare selection (older state) is subject 1
|
||||
let s = {};
|
||||
try { s = JSON.parse(text || "{}") || {}; } catch { s = {}; }
|
||||
const f = (s.frames && typeof s.frames === "object") ? s.frames : {};
|
||||
const out = {};
|
||||
for (const [k, v] of Object.entries(f)) {
|
||||
if (v && (v.selected || v.added)) out[k] = { "1": v };
|
||||
else if (v && typeof v === "object") {
|
||||
out[k] = Object.fromEntries(Object.entries(v).filter(([sk]) => +sk >= 1 && +sk <= SUBJECTS.length));
|
||||
} else out[k] = {};
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
let styled = false;
|
||||
function styles() {
|
||||
if (styled) return;
|
||||
styled = true;
|
||||
const css = document.createElement("style");
|
||||
css.textContent = `
|
||||
.bvsp{--card:#1b1d21;--line:#2e3138;--muted:#8b919c;--text:#e3e6eb;--accent:#3b82f6;--ok:#22c55e;--trk:#60a5fa;--bad:#f43f5e;
|
||||
position:absolute;inset:0;display:flex;flex-direction:column;gap:6px;padding:6px;box-sizing:border-box;font:12px system-ui,sans-serif;color:var(--text);overflow:hidden}
|
||||
.bvsp .card{background:var(--card);border:1px solid var(--line);border-radius:8px;padding:6px 8px;display:flex;flex-direction:column;gap:6px;flex:0 0 auto}
|
||||
.bvsp .row{display:flex;align-items:center;gap:8px;min-width:0}
|
||||
.bvsp .sp{flex:1}
|
||||
.bvsp .muted{color:var(--muted);font-size:11px}
|
||||
.bvsp .btn{background:#2a2d34;border:1px solid #3a3e47;color:var(--text);border-radius:6px;padding:4px 10px;cursor:pointer;font:inherit;white-space:nowrap}
|
||||
.bvsp .btn:hover{background:#343843}
|
||||
.bvsp .btn.pri{background:var(--accent);border-color:#4b8ff7;color:#fff;font-weight:600}
|
||||
.bvsp .btn.pri:hover{background:#4b8ff7}
|
||||
.bvsp .chk{display:flex;align-items:center;gap:4px;color:var(--muted);font-size:11px;cursor:pointer;user-select:none}
|
||||
.bvsp .subj{display:flex;gap:4px}
|
||||
.bvsp .subj .t{display:flex;align-items:center;gap:5px;border:1px solid var(--line);background:#23262c;border-radius:999px;padding:3px 10px 3px 6px;cursor:pointer;color:var(--muted);font-weight:600}
|
||||
.bvsp .subj .t i{width:10px;height:10px;border-radius:50%;display:block}
|
||||
.bvsp .subj .t.on{color:#fff;border-color:currentColor;background:#2b2f37}
|
||||
.bvsp .stat{display:flex;align-items:center;gap:8px;font-size:11px;color:#cfd6e0}
|
||||
.bvsp .stat .pb{flex:0 0 110px;height:5px;background:#2a2d34;border-radius:3px;overflow:hidden}
|
||||
.bvsp .stat .pb i{display:block;height:100%}
|
||||
.bvsp .stat .txt{flex:1;white-space:nowrap;overflow:hidden;text-overflow:ellipsis}
|
||||
.bvsp .stat .tog{cursor:pointer;color:var(--trk);user-select:none}
|
||||
.bvsp .log{max-height:90px;overflow:auto;background:#141518;border:1px solid var(--line);border-radius:6px;padding:4px 6px;font:11px ui-monospace,monospace;color:#aab}
|
||||
.bvsp .counts{display:flex;gap:10px;font-size:11px;color:var(--muted)}
|
||||
.bvsp .counts b{font-weight:700}
|
||||
.bvsp .strip{display:flex;gap:5px;overflow-x:auto;padding-bottom:2px}
|
||||
.bvsp .fr{position:relative;flex:0 0 auto;width:96px;height:58px;border-radius:6px;background:#111 center/cover no-repeat;cursor:pointer;border:2px solid transparent;outline:1px solid var(--line)}
|
||||
.bvsp .fr.on{border-color:#f5b041}
|
||||
.bvsp .fr .b{position:absolute;left:3px;bottom:3px;font-size:10px;background:rgba(0,0,0,.7);padding:0 4px;border-radius:3px}
|
||||
.bvsp .fr .s{position:absolute;right:3px;top:3px;font-size:11px;font-weight:700;padding:0 4px;border-radius:3px;background:rgba(0,0,0,.7)}
|
||||
.bvsp .fr .s.picked{color:var(--ok)}.bvsp .fr .s.tracked{color:var(--trk)}.bvsp .fr .s.lost{color:var(--bad)}
|
||||
.bvsp .fr .dots{position:absolute;left:3px;top:3px;display:flex;gap:2px}
|
||||
.bvsp .fr .dots i{width:8px;height:8px;border-radius:50%;border:1px solid #000;display:block}
|
||||
.bvsp .tl{display:flex;gap:10px;align-items:center}
|
||||
.bvsp .tl img{height:72px;border-radius:6px;border:1px solid var(--line);flex:0 0 auto}
|
||||
.bvsp .tl .scrub{flex:1;display:flex;flex-direction:column;gap:6px;min-width:140px}
|
||||
.bvsp .tl .track{position:relative;height:18px}
|
||||
.bvsp .tl .track .bg{position:absolute;left:0;right:0;top:7px;height:4px;border-radius:2px}
|
||||
.bvsp .tl .track input{position:absolute;inset:0;width:100%;margin:0;background:transparent;-webkit-appearance:none;appearance:none}
|
||||
.bvsp .tl .track input::-webkit-slider-runnable-track{background:transparent;height:18px}
|
||||
.bvsp .tl .track input::-moz-range-track{background:transparent}
|
||||
.bvsp .tl .track input::-webkit-slider-thumb{-webkit-appearance:none;width:14px;height:14px;margin-top:2px;border-radius:50%;background:#fff;border:2px solid var(--accent);cursor:pointer}
|
||||
.bvsp .tl .track input::-moz-range-thumb{width:12px;height:12px;border-radius:50%;background:#fff;border:2px solid var(--accent);cursor:pointer}
|
||||
.bvsp .tl .fn{font-variant-numeric:tabular-nums;color:var(--text);font-weight:600}
|
||||
.bvsp .chips{display:flex;flex-wrap:wrap;gap:4px;align-items:center}
|
||||
.bvsp .chips .c{display:flex;align-items:center;gap:4px;border:1px solid #6b2a37;background:#2e1a1f;color:#fecdd3;border-radius:999px;padding:1px 8px;cursor:pointer;font-size:11px}
|
||||
.bvsp .chips .c:hover{background:#45222b}
|
||||
.bvsp .chips .c i{width:8px;height:8px;border-radius:50%;display:block}
|
||||
.bvsp .chips .okk{color:var(--ok);font-size:11px}
|
||||
.bvsp .host{position:absolute;inset:0}
|
||||
.bvsp .empty{color:var(--muted);text-align:center;padding:12px}
|
||||
`;
|
||||
document.head.appendChild(css);
|
||||
}
|
||||
|
||||
// A run the panel starts for picking ("Segment again", "+ frame") only segments the strip: the selection it sends is marked
|
||||
// segment_only, so the node skips tracking and Molmo2. "Propagate" and a normal workflow run track as usual.
|
||||
const propagating = new Set();
|
||||
let wrapped = false;
|
||||
function wrapQueue() {
|
||||
if (wrapped) return;
|
||||
wrapped = true;
|
||||
const orig = api.queuePrompt.bind(api);
|
||||
api.queuePrompt = async function (number, p, opts) {
|
||||
const targets = opts?.partialExecutionTargets || [];
|
||||
for (const id of targets) {
|
||||
const out = p?.output?.[String(id)];
|
||||
if (out?.class_type !== "BFSVideoSegmentPicker" || propagating.has(String(id))) continue;
|
||||
try {
|
||||
const sel = JSON.parse(out.inputs.selection || "{}");
|
||||
sel.segment_only = true;
|
||||
out.inputs.selection = JSON.stringify(sel);
|
||||
} catch { /* leave it */ }
|
||||
}
|
||||
return orig(number, p, opts);
|
||||
};
|
||||
}
|
||||
|
||||
app.registerExtension({
|
||||
name: "BFSNodes.VideoSegmentPicker",
|
||||
async nodeCreated(node) {
|
||||
if (node.comfyClass !== "BFSVideoSegmentPicker") return;
|
||||
pickerStyles(); styles();
|
||||
const widget = node.widgets?.find(w => w.name === "selection");
|
||||
if (widget) {
|
||||
widget.type = "hidden"; widget.hidden = true; widget.computeSize = () => [0, -4];
|
||||
widget.options = { ...(widget.options || {}), hidden: true, serialize: true };
|
||||
widget.serializeValue = () => widget.value;
|
||||
}
|
||||
const frames = reactive({ v: parse(widget?.value) });
|
||||
const save = () => { if (widget) widget.value = JSON.stringify({ frames: frames.v }); node.graph?.setDirtyCanvas(true); };
|
||||
const run = reactive({ data: null }); // {frames: [panel data per frame], report: [...]}
|
||||
const active = ref(0);
|
||||
const subject = ref("1"); // which subject the clicks select (its color = Picture number)
|
||||
|
||||
const root = document.createElement("div");
|
||||
root.style.cssText = "position:relative;width:100%;height:100%;min-width:440px;min-height:420px";
|
||||
node.addDOMWidget("bfs_video_segment_picker", "div", root, { serialize: false, hideOnZoom: false, hideInPanel: true });
|
||||
wheelToCanvas(root);
|
||||
|
||||
// the image picker's Panel for the active frame, remounted when the frame changes
|
||||
let panelApp = null, handle = {};
|
||||
const panelHost = document.createElement("div");
|
||||
panelHost.className = "host";
|
||||
const mountPanel = () => {
|
||||
panelApp?.unmount(); panelHost.innerHTML = ""; handle = {};
|
||||
const i = run.data?.indices?.[active.value] ?? active.value; // the real frame index of the strip entry
|
||||
const sj = subject.value;
|
||||
const el = document.createElement("div"); el.style.cssText = "position:absolute;inset:0"; panelHost.appendChild(el);
|
||||
panelApp = createApp({ setup: () => Panel({
|
||||
node,
|
||||
get: () => JSON.stringify(frames.v[String(i)]?.[sj] || {}),
|
||||
set: v => {
|
||||
const before = JSON.stringify(frames.v[String(i)]?.[sj]?.selected || []);
|
||||
try { frames.v[String(i)] = { ...(frames.v[String(i)] || {}), [sj]: JSON.parse(v) }; } catch { /* keep */ }
|
||||
save();
|
||||
if (JSON.stringify(frames.v[String(i)]?.[sj]?.selected || []) !== before) autoPropagate();
|
||||
},
|
||||
expose: o => Object.assign(handle, o),
|
||||
}) });
|
||||
panelApp.mount(el);
|
||||
const d = run.data?.frames?.[active.value];
|
||||
if (d) setTimeout(() => handle.executed?.(d), 0);
|
||||
};
|
||||
|
||||
const pickedSubjects = i => Object.entries(frames.v[String(run.data?.indices?.[i] ?? i)] || {}).filter(([, v]) => (v?.selected || []).length).map(([k]) => k);
|
||||
const status = i => {
|
||||
const own = (frames.v[String(run.data?.indices?.[i] ?? i)]?.[subject.value]?.selected || []).length > 0;
|
||||
const r = run.data?.report?.[i]?.subjects?.[subject.value] || run.data?.report?.[i]?.source || "";
|
||||
if (own) return ["picked", "✓"];
|
||||
if (r.startsWith("tracked")) return ["tracked", "↔"];
|
||||
if (r.startsWith("lost")) return ["lost", "⚠"];
|
||||
return ["", ""];
|
||||
};
|
||||
const busy = ref("");
|
||||
const live = reactive({ text: "", progress: null, elapsed: 0, log: [], open: false, running: false });
|
||||
api.addEventListener("bfs_vsegpick_status", e => {
|
||||
const d = e.detail || {};
|
||||
if (String(d.node) !== String(node.id)) return;
|
||||
live.text = d.text; live.elapsed = d.elapsed || 0; live.running = (d.progress ?? 0) < 1;
|
||||
if (d.progress !== null && d.progress !== undefined) live.progress = d.progress;
|
||||
if (d.log) { live.log.push(`[${(d.elapsed || 0).toFixed(1)}s] ${d.text}`); if (live.log.length > 60) live.log.shift(); }
|
||||
});
|
||||
wrapQueue();
|
||||
const segmentOnly = async () => { // a picking run: the new frame is segmented, nothing is tracked
|
||||
busy.value = "segmenting…";
|
||||
try { const p = await app.graphToPrompt(); await api.queuePrompt(0, p, { partialExecutionTargets: [String(node.id)] }); }
|
||||
catch (e) { busy.value = String(e.message || e); }
|
||||
};
|
||||
const propagate = async () => {
|
||||
busy.value = "propagating…";
|
||||
try {
|
||||
const p = await app.graphToPrompt();
|
||||
propagating.add(String(node.id));
|
||||
try { await api.queuePrompt(0, p, { partialExecutionTargets: [String(node.id)] }); } finally { propagating.delete(String(node.id)); }
|
||||
} catch (e) { busy.value = String(e.message || e); }
|
||||
};
|
||||
// after the first Propagate, a pick that changes a frame's selection propagates by itself (1.5 s after the last
|
||||
// click): only the shots whose picks changed are tracked again, the rest comes from the cache
|
||||
const auto = ref(localStorage.getItem("bfs_vsegpick_auto") !== "0");
|
||||
let autoTimer = null;
|
||||
const propagatedOnce = () => (run.data?.report || []).some(r => /tracked|lost|Molmo/.test(JSON.stringify(r)));
|
||||
const autoPropagate = () => {
|
||||
if (!auto.value || !propagatedOnce()) return;
|
||||
clearTimeout(autoTimer);
|
||||
busy.value = "auto-propagate in 1.5 s…";
|
||||
autoTimer = setTimeout(propagate, 1500);
|
||||
};
|
||||
const addFrame = f => { // put a frame in the strip (an empty pick) and run, so it can be picked on
|
||||
const n = run.data?.count || 0;
|
||||
f = Math.max(0, Math.min(n - 1, Math.round(+f)));
|
||||
if (!Number.isFinite(f)) return;
|
||||
if (!frames.v[String(f)]) { frames.v[String(f)] = {}; save(); }
|
||||
segmentOnly();
|
||||
};
|
||||
const SUBJ = { "1": "red", "2": "blue", "3": "green" };
|
||||
const tlPos = ref(0); // index into run.data.timeline (whole-video scrubber)
|
||||
const counts = () => {
|
||||
const r = run.data?.report || []; const n = run.data?.frames?.length || 0;
|
||||
let picked = 0, tracked = 0, lost = 0;
|
||||
for (let i = 0; i < n; i++) { const [c] = status(i); if (c === "picked") picked++; else if (c === "tracked") tracked++; else if (c === "lost") lost++; }
|
||||
return { picked, tracked, lost, n };
|
||||
};
|
||||
const tlAt = () => run.data.timeline[Math.min(tlPos.value, run.data.timeline.length - 1)];
|
||||
const lostTrack = () => { // the scrubber's track: lost stretches in red over the video's length
|
||||
const n = Math.max(1, (run.data?.count || 1) - 1);
|
||||
const runs = [...(run.data?.lost || [])];
|
||||
if (!runs.length) return "#3a3e47";
|
||||
const stops = ["#3a3e47 0%"];
|
||||
for (const r of runs.sort((a, b) => a.start - b.start)) {
|
||||
const a = (100 * r.start / n).toFixed(2), b = (100 * Math.max(r.end, r.start + 0.5) / n).toFixed(2);
|
||||
stops.push(`#3a3e47 ${a}%`, `#f43f5e ${a}%`, `#f43f5e ${b}%`, `#3a3e47 ${b}%`);
|
||||
}
|
||||
return `linear-gradient(90deg, ${stops.join(", ")}, #3a3e47 100%)`;
|
||||
};
|
||||
createApp({
|
||||
setup: () => () => h("div", { class: "bvsp" }, [
|
||||
h("div", { class: "card" }, [
|
||||
h("div", { class: "row" }, [
|
||||
h("div", { class: "subj" }, SUBJECTS.map(([k, col, name]) => h("span", {
|
||||
class: ["t", subject.value === k ? "on" : ""], style: subject.value === k ? { color: col } : null,
|
||||
title: `Subject ${k} (${name}) = Picture ${k}: clicks select this subject`,
|
||||
onClick: () => { if (subject.value !== k) { subject.value = k; mountPanel(); } } },
|
||||
[h("i", { style: { background: col } }), h("span", { style: "color:#fff" }, `${k} · ${name}`)]))),
|
||||
h("span", { class: "sp" }),
|
||||
h("label", { class: "chk", title: "After the first Propagate: picking a person on a frame propagates by itself (1.5 s after the last click)" }, [
|
||||
h("input", { type: "checkbox", checked: auto.value, onChange: e => { auto.value = e.target.checked;
|
||||
try { localStorage.setItem("bfs_vsegpick_auto", e.target.checked ? "1" : "0"); } catch { /* ignore */ } } }), "auto"]),
|
||||
h("button", { class: "btn pri", onClick: propagate, title: "Track the picks to every frame (Molmo2 finds the person after camera cuts)" }, "↔ Propagate"),
|
||||
]),
|
||||
live.text || busy.value ? h("div", { class: "stat" }, [
|
||||
h("span", { class: "pb" }, [h("i", { style: { width: `${Math.round(100 * (live.progress ?? 0))}%`,
|
||||
background: live.running ? "#3b82f6" : "#22c55e" } })]),
|
||||
h("span", { class: "txt", title: live.text }, busy.value && !live.running ? busy.value
|
||||
: `${live.running ? "⏳" : "✓"} ${live.text} · ${live.elapsed}s`),
|
||||
h("span", { class: "tog", onClick: () => { live.open = !live.open; } }, live.open ? "▾ log" : "▸ log"),
|
||||
]) : h("div", { class: "muted" }, "Pick a subject, click a frame below, then click its person. Subject N = Picture N."),
|
||||
live.open ? h("div", { class: "log" }, live.log.length ? live.log.map((l, i) => h("div", { key: i }, l)) : "no log yet") : null,
|
||||
]),
|
||||
h("div", { class: "card" }, [
|
||||
h("div", { class: "row" }, [
|
||||
h("span", { style: "font-weight:600" }, "Frames"),
|
||||
run.data ? h("span", { class: "counts" }, (() => { const c = counts(); return [
|
||||
h("span", null, [h("b", { style: "color:#22c55e" }, "✓ "), `${c.picked} picked`]),
|
||||
h("span", null, [h("b", { style: "color:#60a5fa" }, "↔ "), `${c.tracked} tracked`]),
|
||||
h("span", null, [h("b", { style: "color:#f43f5e" }, "⚠ "), `${c.lost} lost`])]; })()) : null,
|
||||
]),
|
||||
run.data?.frames?.length
|
||||
? h("div", { class: "strip" }, run.data.frames.map((d, i) => {
|
||||
const [cls, mark] = status(i);
|
||||
return h("div", { class: ["fr", i === active.value ? "on" : ""], key: i, title: run.data.report?.[i]?.source || "",
|
||||
style: { backgroundImage: `url("${fileUrl(d.image)}")` },
|
||||
onClick: () => { if (active.value !== i) { active.value = i; mountPanel(); } } },
|
||||
[h("span", { class: "b" }, `f${run.data.indices?.[i] ?? i}`), mark ? h("span", { class: ["s", cls] }, mark) : null,
|
||||
h("span", { class: "dots" }, pickedSubjects(i).map(k => h("i", { style: { background: SUBJECTS[(+k - 1) % SUBJECTS.length][1] } })))]);
|
||||
}))
|
||||
: h("div", { class: "empty" }, "Run the node once: the frames appear here. Click a frame, then the person."),
|
||||
]),
|
||||
run.data?.timeline?.length ? h("div", { class: "card" }, [
|
||||
h("div", { class: "tl" }, [
|
||||
h("img", { src: fileUrl(tlAt()), title: "every frame with its masks: scrub to find a wrong or missing mark" }),
|
||||
h("div", { class: "scrub" }, [
|
||||
h("div", { class: "row" }, [
|
||||
h("span", { style: "font-weight:600" }, "Whole video"),
|
||||
h("span", { class: "fn" }, `f${tlAt().frame}`),
|
||||
h("span", { class: "muted" }, `/ ${(run.data.count || 1) - 1}`),
|
||||
h("span", { class: "sp" }),
|
||||
h("button", { class: "btn", onClick: () => addFrame(tlAt().frame),
|
||||
title: "put this frame in the strip to pick or fix the person on it" }, "+ add this frame"),
|
||||
]),
|
||||
h("div", { class: "track" }, [
|
||||
h("div", { class: "bg", style: { background: lostTrack() } }),
|
||||
h("input", { type: "range", min: 0, max: run.data.timeline.length - 1, value: tlPos.value,
|
||||
onInput: e => { tlPos.value = +e.target.value; } }),
|
||||
]),
|
||||
h("div", { class: "chips" }, (run.data.lost || []).length
|
||||
? (run.data.lost || []).slice(0, 12).map(r => h("span", { class: "c", title: "add the middle frame of this stretch to the strip",
|
||||
onClick: () => addFrame((r.start + r.end) / 2) },
|
||||
[h("i", { style: { background: SUBJECTS[(+r.subject - 1) % SUBJECTS.length]?.[1] || "#999" } }), `f${r.start}–${r.end}`]))
|
||||
: [h("span", { class: "okk" }, "✓ no lost stretches")]),
|
||||
]),
|
||||
]),
|
||||
]) : null,
|
||||
h("div", { ref: el => { if (el && !el.contains(panelHost)) { el.appendChild(panelHost); if (!panelApp) mountPanel(); } }, style: "flex:1;position:relative;min-height:260px;border:1px solid #2e3138;border-radius:8px;overflow:hidden" }),
|
||||
]),
|
||||
}).mount(root);
|
||||
|
||||
const onExecuted = node.onExecuted;
|
||||
node.onExecuted = function (output) {
|
||||
onExecuted?.apply(this, arguments);
|
||||
const d = output?.bfs_vsegpick?.[0];
|
||||
if (!d) return;
|
||||
run.data = d; busy.value = "";
|
||||
if (active.value >= d.frames.length) active.value = 0;
|
||||
mountPanel();
|
||||
};
|
||||
const onConfigure = node.onConfigure;
|
||||
node.onConfigure = function () {
|
||||
const r = onConfigure?.apply(this, arguments);
|
||||
setTimeout(() => { frames.v = parse(widget?.value); mountPanel(); }, 0);
|
||||
return r;
|
||||
};
|
||||
node.size = [Math.max(node.size[0], 780), Math.max(node.size[1], 1180)];
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,167 @@
|
||||
/**
|
||||
* BFS Workflow Loop: the panel. Lists the nodes upstream of `result`. "Inject" picks a node whose output is replaced
|
||||
* by the current item (or its mask) on every iteration; "Change per iteration" picks widgets with one value per
|
||||
* iteration (one per line, or "a, b, c" / "start:stop:step" for numbers). State lives in the hidden `loop` widget.
|
||||
*/
|
||||
import { app } from "../../scripts/app.js";
|
||||
import { wheelToCanvas } from "./bfs_wheel.js";
|
||||
import { createApp, h, reactive, computed, watch } from "./vendor/vue.esm-browser.prod.mjs";
|
||||
|
||||
const SKIP_TYPES = new Set(["hidden", "converted-widget", "button"]);
|
||||
const SKIP_NAMES = new Set(["control_after_generate", "upload", "image_upload", "videopreview", "loop"]);
|
||||
|
||||
function upstreamNodes(node) {
|
||||
const graph = node.graph || app.graph;
|
||||
const getLink = id => graph.links[id] ?? graph.links.get?.(id);
|
||||
const inp = node.inputs?.find(i => i.name === "result");
|
||||
const link = inp?.link != null ? getLink(inp.link) : null;
|
||||
if (!link) return [];
|
||||
const seen = new Set(), order = [], stack = [link.origin_id];
|
||||
while (stack.length) {
|
||||
const id = stack.pop();
|
||||
if (seen.has(id)) continue;
|
||||
seen.add(id);
|
||||
const n = graph.getNodeById(id);
|
||||
if (!n) continue;
|
||||
order.push(n);
|
||||
for (const i of n.inputs || []) if (i.link != null) { const l = getLink(i.link); if (l) stack.push(l.origin_id); }
|
||||
}
|
||||
return order.sort((a, b) => a.id - b.id);
|
||||
}
|
||||
const widgetsOf = n => (n?.widgets || []).filter(w => w.name && !SKIP_TYPES.has(w.type) && !SKIP_NAMES.has(w.name) && !w.hidden);
|
||||
const kindOf = w => (w?.type === "combo" || Array.isArray(w?.options?.values)) ? "combo"
|
||||
: (w?.type === "toggle" || typeof w?.value === "boolean") ? "bool" : (w?.type === "number" || typeof w?.value === "number") ? "number" : "text";
|
||||
|
||||
function parseValues(text, kind) {
|
||||
if (kind === "number") {
|
||||
const out = [];
|
||||
for (const part of String(text).split(/[,\n]/).map(s => s.trim()).filter(Boolean)) {
|
||||
const r = part.split(":").map(Number);
|
||||
if (r.length === 3 && r.every(Number.isFinite) && r[2] !== 0) {
|
||||
for (let v = r[0], i = 0; (r[2] > 0 ? v <= r[1] + 1e-9 : v >= r[1] - 1e-9) && i < 256; v += r[2], i++) out.push(+v.toFixed(6));
|
||||
} else if (Number.isFinite(+part)) out.push(+part);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
if (kind === "bool") return String(text).split(/[,\n]/).map(s => s.trim().toLowerCase()).filter(Boolean).map(s => ["1", "true", "yes", "on"].includes(s));
|
||||
return String(text).split("\n").map(s => s.trim()).filter(Boolean);
|
||||
}
|
||||
|
||||
function parseState(text) {
|
||||
let s = {};
|
||||
try { s = JSON.parse(text || "{}") || {}; } catch { s = {}; }
|
||||
return {
|
||||
inject: (s.inject || []).map(i => ({ node: String(i.node ?? ""), slot: +i.slot || 0, source: i.source || "items" })),
|
||||
params: (s.params || []).map(p => ({ node: String(p.node ?? ""), widget: p.widget || "", text: p.text ?? (p.values || []).join("\n") })),
|
||||
};
|
||||
}
|
||||
|
||||
let styled = false;
|
||||
function styles() {
|
||||
if (styled) return;
|
||||
styled = true;
|
||||
const css = document.createElement("style");
|
||||
css.textContent = `
|
||||
.bwl{position:absolute;inset:0;display:flex;flex-direction:column;gap:8px;padding:6px;box-sizing:border-box;font:12px system-ui,sans-serif;color:#ddd;overflow:auto}
|
||||
.bwl .sec{display:flex;align-items:center;gap:6px;color:#9ab;font-size:11px;text-transform:uppercase;letter-spacing:.05em}
|
||||
.bwl .row{background:#1d1d1d;border:1px solid #363636;border-radius:6px;padding:5px;display:flex;flex-direction:column;gap:4px}
|
||||
.bwl .line{display:flex;gap:4px;align-items:center}
|
||||
.bwl select,.bwl textarea{background:#262626;color:#ddd;border:1px solid #4a4a4a;border-radius:4px;padding:3px 5px;font:inherit;min-width:0;flex:1}
|
||||
.bwl textarea{resize:vertical;min-height:38px}
|
||||
.bwl button{background:#333;color:#ddd;border:1px solid #555;border-radius:4px;padding:2px 8px;cursor:pointer;font:inherit}
|
||||
.bwl button:hover{background:#444}
|
||||
.bwl .x{color:#999;cursor:pointer;padding:0 4px}.bwl .x:hover{color:#f66}
|
||||
.bwl .hint{color:#888;font-size:11px}.bwl .warn{color:#f7a}
|
||||
.bwl .foot{color:#bbb}.bwl .foot b{color:#fff}
|
||||
`;
|
||||
document.head.appendChild(css);
|
||||
}
|
||||
|
||||
function Panel(io) {
|
||||
const st = reactive(parseState(io.get()));
|
||||
const tick = reactive({ n: 0 });
|
||||
const nodes = computed(() => (tick.n, upstreamNodes(io.node)));
|
||||
const byId = id => nodes.value.find(n => String(n.id) === String(id));
|
||||
const save = () => io.set(JSON.stringify({
|
||||
inject: st.inject.filter(i => i.node).map(i => ({ node: i.node, slot: i.slot, source: i.source })),
|
||||
params: st.params.filter(p => p.node && p.widget).map(p => {
|
||||
const w = widgetsOf(byId(p.node)).find(x => x.name === p.widget);
|
||||
return { node: p.node, widget: p.widget, values: parseValues(p.text, kindOf(w)), text: p.text };
|
||||
}),
|
||||
}));
|
||||
watch(st, save, { deep: true });
|
||||
io.expose({ refresh: () => { tick.n++; }, reload: () => { Object.assign(st, parseState(io.get())); tick.n++; } });
|
||||
|
||||
const nodeSelect = (obj, onPick) => h("select", { value: obj.node, onChange: e => { obj.node = e.target.value; onPick?.(); } },
|
||||
[h("option", { value: "" }, "— pick a node —"), ...nodes.value.map(n => h("option", { value: String(n.id) }, `[${n.id}] ${n.title || n.type}`))]);
|
||||
|
||||
const injectRow = (i, idx) => {
|
||||
const n = byId(i.node);
|
||||
const outs = (n?.outputs || []).map((o, k) => ({ k, label: `${o.name || o.type} (${o.type})` }));
|
||||
return h("div", { class: "row", key: "i" + idx }, [
|
||||
h("div", { class: "line" }, [nodeSelect(i, () => { i.slot = 0; }), h("span", { class: "x", onClick: () => st.inject.splice(idx, 1) }, "✕")]),
|
||||
n ? h("div", { class: "line" }, [
|
||||
h("select", { value: String(i.slot), onChange: e => { i.slot = +e.target.value; } }, outs.map(o => h("option", { value: String(o.k) }, `output: ${o.label}`))),
|
||||
h("select", { value: i.source, onChange: e => { i.source = e.target.value; } },
|
||||
[h("option", { value: "items" }, "← item i (items)"), h("option", { value: "items_b" }, "← item i (items_b)"),
|
||||
h("option", { value: "item_masks" }, "← mask i (item_masks)")]),
|
||||
]) : null,
|
||||
]);
|
||||
};
|
||||
const paramRow = (p, idx) => {
|
||||
const n = byId(p.node);
|
||||
const w = widgetsOf(n).find(x => x.name === p.widget);
|
||||
const kind = kindOf(w);
|
||||
const vals = parseValues(p.text, kind);
|
||||
return h("div", { class: "row", key: "p" + idx }, [
|
||||
h("div", { class: "line" }, [nodeSelect(p, () => { p.widget = ""; p.text = ""; }), h("span", { class: "x", onClick: () => st.params.splice(idx, 1) }, "✕")]),
|
||||
n ? h("select", { value: p.widget, onChange: e => { p.widget = e.target.value; const ww = widgetsOf(n).find(x => x.name === p.widget); p.text = ww ? String(ww.value ?? "") : ""; } },
|
||||
[h("option", { value: "" }, "— pick a widget —"), ...widgetsOf(n).map(x => h("option", { value: x.name }, `${x.name} (${String(x.value).split(/[\\/]/).pop().slice(0, 36)})`))]) : null,
|
||||
w ? h("textarea", { value: p.text, placeholder: kind === "number" ? "1, 2, 3 or 1:10:1" : "one value per line (iteration 1, 2, 3...)",
|
||||
onInput: e => { p.text = e.target.value; } }) : null,
|
||||
w ? h("div", { class: "hint" }, kind === "combo" ? `${vals.length} values; options: ${(w.options?.values || []).slice(0, 6).map(v => String(v).split(/[\\/]/).pop()).join(", ")}…`
|
||||
: `${vals.length} values${vals.length ? ": " + vals.slice(0, 8).join(", ") + (vals.length > 8 ? "…" : "") : ""}`) : null,
|
||||
]);
|
||||
};
|
||||
const itemsLinked = () => !!io.node.inputs?.find(x => x.name === "items")?.link;
|
||||
|
||||
return () => h("div", { class: "bwl" }, [
|
||||
nodes.value.length ? null : h("div", { class: "warn" }, "Connect what one iteration produces to `result`; its upstream nodes appear here."),
|
||||
h("div", { class: "sec" }, ["Inject the current item into", h("span", { style: "flex:1" }),
|
||||
h("button", { onClick: () => st.inject.push({ node: "", slot: 0, source: "items" }) }, "+")]),
|
||||
...st.inject.map(injectRow),
|
||||
st.inject.length ? null : h("div", { class: "hint" }, "e.g. the Load Image that stands for “the frame”: on iteration i it gets item i."),
|
||||
h("div", { class: "sec" }, ["Change per iteration", h("span", { style: "flex:1" }),
|
||||
h("button", { onClick: () => st.params.push({ node: "", widget: "", text: "" }) }, "+")]),
|
||||
...st.params.map(paramRow),
|
||||
st.params.length ? null : h("div", { class: "hint" }, "e.g. a seed or a prompt line per iteration (cycled when shorter than the items)."),
|
||||
h("div", { class: "foot" }, [itemsLinked() ? "Iterations: one per item." : "No items connected: iterations = the longest value list (or count).",
|
||||
h("span", { style: "flex:1" }), h("button", { onClick: () => { tick.n++; }, title: "re-read the graph" }, "↻")]),
|
||||
]);
|
||||
}
|
||||
|
||||
app.registerExtension({
|
||||
name: "BFSNodes.WorkflowLoop",
|
||||
async nodeCreated(node) {
|
||||
if (node.comfyClass !== "BFSWorkflowLoop") return;
|
||||
styles();
|
||||
const widget = node.widgets?.find(w => w.name === "loop");
|
||||
if (widget) {
|
||||
widget.type = "hidden"; widget.hidden = true; widget.computeSize = () => [0, -4];
|
||||
widget.options = { ...(widget.options || {}), hidden: true, serialize: true };
|
||||
widget.serializeValue = () => widget.value;
|
||||
}
|
||||
const host = document.createElement("div");
|
||||
host.style.cssText = "position:relative;width:100%;height:100%;min-width:340px;min-height:360px";
|
||||
node.addDOMWidget("bfs_workflow_loop", "div", host, { serialize: false, hideOnZoom: false, hideInPanel: true });
|
||||
wheelToCanvas(host);
|
||||
const handle = {};
|
||||
createApp({ setup: () => Panel({ node, get: () => widget?.value ?? "{}",
|
||||
set: v => { if (widget) widget.value = v; node.graph?.setDirtyCanvas(true); }, expose: o => Object.assign(handle, o) }) }).mount(host);
|
||||
const onConn = node.onConnectionsChange;
|
||||
node.onConnectionsChange = function () { const r = onConn?.apply(this, arguments); handle.refresh?.(); return r; };
|
||||
const onConfigure = node.onConfigure;
|
||||
node.onConfigure = function () { const r = onConfigure?.apply(this, arguments); setTimeout(() => handle.reload?.(), 0); return r; };
|
||||
node.size = [Math.max(node.size[0], 380), Math.max(node.size[1], 440)];
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,213 @@
|
||||
/**
|
||||
* BFS XY Sweep: the panel. Lists every node upstream of the `result` input and its widgets (like TinyTerra's
|
||||
* advanced xyPlot); for X and Y pick a node + widget and its values. Combo widgets (LoRA names, samplers...) get a
|
||||
* filterable checklist; numbers take "0.5, 0.7, 0.9" or a range "start:stop:step"; text takes one value per line.
|
||||
* State lives in the hidden `sweep` STRING widget (JSON), saved with the workflow.
|
||||
*/
|
||||
import { app } from "../../scripts/app.js";
|
||||
import { wheelToCanvas } from "./bfs_wheel.js";
|
||||
import { createApp, h, reactive, computed, watch } from "./vendor/vue.esm-browser.prod.mjs";
|
||||
|
||||
const SKIP_TYPES = new Set(["hidden", "converted-widget", "button"]);
|
||||
const SKIP_NAMES = new Set(["control_after_generate", "upload", "image_upload", "videopreview", "sweep"]);
|
||||
|
||||
function upstreamNodes(node) {
|
||||
const graph = node.graph || app.graph;
|
||||
const inp = node.inputs?.find(i => i.name === "result");
|
||||
const link = inp?.link != null ? graph.links[inp.link] ?? graph.links.get?.(inp.link) : null;
|
||||
if (!link) return [];
|
||||
const seen = new Set(), order = [], stack = [link.origin_id];
|
||||
while (stack.length) {
|
||||
const id = stack.pop();
|
||||
if (seen.has(id)) continue;
|
||||
seen.add(id);
|
||||
const n = graph.getNodeById(id);
|
||||
if (!n) continue;
|
||||
order.push(n);
|
||||
for (const i of n.inputs || []) {
|
||||
if (i.link == null) continue;
|
||||
const l = graph.links[i.link] ?? graph.links.get?.(i.link);
|
||||
if (l) stack.push(l.origin_id);
|
||||
}
|
||||
}
|
||||
return order.sort((a, b) => a.id - b.id);
|
||||
}
|
||||
|
||||
function widgetsOf(n) {
|
||||
return (n.widgets || []).filter(w => w.name && !SKIP_TYPES.has(w.type) && !SKIP_NAMES.has(w.name) && !w.hidden
|
||||
&& !(w.options && w.options.hidden));
|
||||
}
|
||||
|
||||
function kindOf(w) {
|
||||
if (!w) return "text";
|
||||
if (w.type === "combo" || Array.isArray(w.options?.values)) return "combo";
|
||||
if (w.type === "toggle" || typeof w.value === "boolean") return "bool";
|
||||
if (w.type === "number" || typeof w.value === "number") return "number";
|
||||
return "text";
|
||||
}
|
||||
|
||||
function parseNumbers(text) {
|
||||
const out = [];
|
||||
for (const part of String(text).split(/[,\n]/).map(s => s.trim()).filter(Boolean)) {
|
||||
const r = part.split(":").map(Number);
|
||||
if (r.length === 3 && r.every(Number.isFinite) && r[2] !== 0) {
|
||||
for (let v = r[0], i = 0; (r[2] > 0 ? v <= r[1] + 1e-9 : v >= r[1] - 1e-9) && i < 256; v += r[2], i++) out.push(+v.toFixed(6));
|
||||
} else if (Number.isFinite(+part)) out.push(+part);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function parseState(text) {
|
||||
let s = {};
|
||||
try { s = JSON.parse(text || "{}") || {}; } catch { s = {}; }
|
||||
const axis = a => ({ node: a?.node != null ? String(a.node) : "", widget: a?.widget || "", values: Array.isArray(a?.values) ? a.values : [],
|
||||
text: a?.text ?? (Array.isArray(a?.values) ? a.values.join(", ") : ""), title: a?.title || "", filter: "" });
|
||||
return { x: axis(s.x), y: axis(s.y) };
|
||||
}
|
||||
|
||||
let styled = false;
|
||||
function styles() {
|
||||
if (styled) return;
|
||||
styled = true;
|
||||
const css = document.createElement("style");
|
||||
css.textContent = `
|
||||
.bxy{position:absolute;inset:0;display:flex;flex-direction:column;gap:8px;padding:6px;box-sizing:border-box;font:12px system-ui,sans-serif;color:#ddd;overflow:auto}
|
||||
.bxy .ax{background:#1d1d1d;border:1px solid #363636;border-radius:6px;padding:6px;display:flex;flex-direction:column;gap:5px}
|
||||
.bxy .hd{display:flex;align-items:center;gap:6px}
|
||||
.bxy .tag{font-weight:700;background:#2d5a8a;color:#fff;border-radius:4px;padding:1px 7px}
|
||||
.bxy .tag.y{background:#7a3d8a}
|
||||
.bxy select,.bxy input,.bxy textarea{background:#262626;color:#ddd;border:1px solid #4a4a4a;border-radius:4px;padding:3px 5px;font:inherit;min-width:0}
|
||||
.bxy select{flex:1}
|
||||
.bxy textarea{resize:vertical;min-height:40px}
|
||||
.bxy .opts{max-height:120px;overflow:auto;display:flex;flex-direction:column;gap:1px;border:1px solid #333;border-radius:4px;padding:3px}
|
||||
.bxy .opts label{display:flex;gap:5px;align-items:center;white-space:nowrap;overflow:hidden;text-overflow:ellipsis;cursor:pointer}
|
||||
.bxy .opts label:hover{background:#2a2a2a}
|
||||
.bxy .chips{display:flex;flex-wrap:wrap;gap:3px}
|
||||
.bxy .chip{background:#2a2a2a;border:1px solid #444;border-radius:9px;padding:0 6px;font-size:11px}
|
||||
.bxy .hint{color:#888;font-size:11px}
|
||||
.bxy .foot{display:flex;align-items:center;gap:6px;color:#bbb}
|
||||
.bxy .foot .n{font-weight:700;color:#fff}
|
||||
.bxy .warn{color:#f7a}
|
||||
.bxy button{background:#333;color:#ddd;border:1px solid #555;border-radius:4px;padding:2px 8px;cursor:pointer;font:inherit}
|
||||
.bxy button:hover{background:#444}
|
||||
`;
|
||||
document.head.appendChild(css);
|
||||
}
|
||||
|
||||
function Panel(io) {
|
||||
const st = reactive(parseState(io.get()));
|
||||
const tick = reactive({ n: 0 }); // bumps on graph changes so the node lists refresh
|
||||
const nodes = computed(() => (tick.n, upstreamNodes(io.node)));
|
||||
const nodeById = id => nodes.value.find(n => String(n.id) === String(id));
|
||||
const widgetOf = a => widgetsOf(nodeById(a.node) || {}).find(w => w.name === a.widget);
|
||||
|
||||
const save = () => {
|
||||
const out = {};
|
||||
for (const k of ["x", "y"]) {
|
||||
const a = st[k];
|
||||
if (!a.node || !a.widget) continue;
|
||||
const kind = kindOf(widgetOf(a));
|
||||
let values = a.values;
|
||||
if (kind === "number") values = parseNumbers(a.text);
|
||||
else if (kind === "text") values = String(a.text).split("\n").map(s => s.trim()).filter(Boolean);
|
||||
else if (kind === "bool") values = [false, true].filter(v => a.values.includes(v));
|
||||
const n = nodeById(a.node);
|
||||
out[k] = { node: a.node, widget: a.widget, values, text: a.text, title: n ? (n.title || n.type) : a.title };
|
||||
}
|
||||
io.set(JSON.stringify(out));
|
||||
};
|
||||
watch(st, save, { deep: true });
|
||||
|
||||
const count = computed(() => {
|
||||
const len = a => {
|
||||
if (!a.node || !a.widget) return 0;
|
||||
const kind = kindOf(widgetOf(a));
|
||||
if (kind === "number") return parseNumbers(a.text).length;
|
||||
if (kind === "text") return String(a.text).split("\n").filter(s => s.trim()).length;
|
||||
return a.values.length;
|
||||
};
|
||||
const x = len(st.x), y = len(st.y);
|
||||
return x * (y || 1);
|
||||
});
|
||||
|
||||
io.expose({ refresh: () => { tick.n++; }, reload: () => { Object.assign(st, parseState(io.get())); tick.n++; } });
|
||||
|
||||
const axisView = (k, label) => {
|
||||
const a = st[k];
|
||||
const n = nodeById(a.node);
|
||||
const ws = n ? widgetsOf(n) : [];
|
||||
const w = widgetOf(a);
|
||||
const kind = kindOf(w);
|
||||
const pickNode = e => { a.node = e.target.value; a.widget = ""; a.values = []; a.text = ""; };
|
||||
const pickWidget = e => {
|
||||
a.widget = e.target.value; a.values = []; const ww = widgetOf(a);
|
||||
a.text = ww && kindOf(ww) !== "combo" ? String(ww.value ?? "") : "";
|
||||
};
|
||||
let editor = null;
|
||||
if (w && kind === "combo") {
|
||||
const all = (w.options?.values || []).map(String);
|
||||
const shown = a.filter ? all.filter(v => v.toLowerCase().includes(a.filter.toLowerCase())) : all;
|
||||
editor = [
|
||||
h("input", { placeholder: "filter options…", value: a.filter, onInput: e => { a.filter = e.target.value; } }),
|
||||
h("div", { class: "opts" }, shown.slice(0, 400).map(v => h("label", { title: v }, [
|
||||
h("input", { type: "checkbox", checked: a.values.includes(v),
|
||||
onChange: e => { a.values = e.target.checked ? [...a.values, v] : a.values.filter(x => x !== v); } }), v]))),
|
||||
a.values.length ? h("div", { class: "chips" }, a.values.map(v => h("span", { class: "chip", title: v }, v.split(/[\\/]/).pop()))) : null,
|
||||
];
|
||||
} else if (w && kind === "bool") {
|
||||
editor = h("div", { class: "chips" }, [false, true].map(v => h("label", null, [
|
||||
h("input", { type: "checkbox", checked: a.values.includes(v),
|
||||
onChange: e => { a.values = e.target.checked ? [...a.values, v] : a.values.filter(x => x !== v); } }), String(v)])));
|
||||
} else if (w && kind === "number") {
|
||||
editor = [h("input", { value: a.text, placeholder: "0.5, 0.7, 0.9 or 0.5:0.9:0.2", onInput: e => { a.text = e.target.value; } }),
|
||||
h("div", { class: "hint" }, `→ ${parseNumbers(a.text).join(", ") || "no values"} (now ${w.value})`)];
|
||||
} else if (w) {
|
||||
editor = [h("textarea", { value: a.text, placeholder: "one value per line", onInput: e => { a.text = e.target.value; } })];
|
||||
}
|
||||
return h("div", { class: "ax" }, [
|
||||
h("div", { class: "hd" }, [h("span", { class: ["tag", k] }, label),
|
||||
h("select", { value: a.node, onChange: pickNode }, [h("option", { value: "" }, k === "y" ? "— none —" : "— pick a node —"),
|
||||
...nodes.value.map(nd => h("option", { value: String(nd.id) }, `[${nd.id}] ${nd.title || nd.type}`))]),
|
||||
]),
|
||||
n ? h("select", { value: a.widget, onChange: pickWidget }, [h("option", { value: "" }, "— pick a widget —"),
|
||||
...ws.map(ww => h("option", { value: ww.name }, `${ww.name} (${String(ww.value).split(/[\\/]/).pop().slice(0, 40)})`))]) : null,
|
||||
...(Array.isArray(editor) ? editor : [editor]),
|
||||
]);
|
||||
};
|
||||
|
||||
return () => h("div", { class: "bxy" }, [
|
||||
nodes.value.length ? null : h("div", { class: "warn" }, "Connect the image to compare to `result`; its upstream nodes appear here."),
|
||||
axisView("x", "X"),
|
||||
axisView("y", "Y"),
|
||||
h("div", { class: "foot" }, [h("span", null, ["cells: ", h("span", { class: "n" }, String(count.value))]),
|
||||
count.value > 64 ? h("span", { class: "warn" }, "max 64") : null, h("span", { style: "flex:1" }),
|
||||
h("button", { onClick: () => { tick.n++; }, title: "re-read the graph" }, "↻")]),
|
||||
h("div", { class: "hint" }, "Only the nodes that depend on the swept widgets run once per cell; models and the rest run once."),
|
||||
]);
|
||||
}
|
||||
|
||||
app.registerExtension({
|
||||
name: "BFSNodes.XYSweep",
|
||||
async nodeCreated(node) {
|
||||
if (node.comfyClass !== "BFSXYSweep") return;
|
||||
styles();
|
||||
const widget = node.widgets?.find(w => w.name === "sweep");
|
||||
if (widget) {
|
||||
widget.type = "hidden"; widget.hidden = true; widget.computeSize = () => [0, -4];
|
||||
widget.options = { ...(widget.options || {}), hidden: true, serialize: true };
|
||||
widget.serializeValue = () => widget.value;
|
||||
}
|
||||
const host = document.createElement("div");
|
||||
host.style.cssText = "position:relative;width:100%;height:100%;min-width:340px;min-height:380px";
|
||||
node.addDOMWidget("bfs_xy_sweep", "div", host, { serialize: false, hideOnZoom: false, hideInPanel: true });
|
||||
wheelToCanvas(host);
|
||||
const handle = {};
|
||||
createApp({ setup: () => Panel({ node, get: () => widget?.value ?? "{}",
|
||||
set: v => { if (widget) widget.value = v; node.graph?.setDirtyCanvas(true); }, expose: o => Object.assign(handle, o) }) }).mount(host);
|
||||
const onConn = node.onConnectionsChange;
|
||||
node.onConnectionsChange = function () { const r = onConn?.apply(this, arguments); handle.refresh?.(); return r; };
|
||||
const onConfigure = node.onConfigure;
|
||||
node.onConfigure = function () { const r = onConfigure?.apply(this, arguments); setTimeout(() => handle.reload?.(), 0); return r; };
|
||||
node.size = [Math.max(node.size[0], 380), Math.max(node.size[1], 520)];
|
||||
},
|
||||
});
|
||||
Reference in New Issue
Block a user