Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Alisson AnjosandClaude Opus 5.5
|
6df4276ae9 |
@@ -144,7 +144,7 @@ workflow once per shot, and join the results back with the original timing and s
|
||||
guide at 2 fps without native guide video-reference latents; use Add Guide separately for motion.
|
||||
See [H3 guide modes](MINIMAX_H3_GUIDES.md#video-context-in-clip-with-aligned-guide-latents).
|
||||
- **BFS Shot Join** — concatenates, trims to the true lengths, cross-fades soft joins, returns audio. With `comparison` on, its `guide` output is the original video frame for frame.
|
||||
- **BFS Video Compare** — one player for two videos (Join's `guide` and `images`): wipe ↔ / ↕, overlay or side by side, any of guide / target / difference on top, plus a per-pixel difference map and the difference per frame (click to jump). 🖼 PNG downloads the frame on screen as shown; with two images (one frame) ⬇ save PNG downloads it from lossless PNGs at the compared size.
|
||||
- **BFS Video Compare** — one player for two videos (Join's `guide` and `images`): wipe ↔ / ↕, overlay or side by side, any of guide / target / difference on top, plus a per-pixel difference map and the difference per frame (click to jump).
|
||||
|
||||
**Cast (1.49.0)**: *Find people* tracks faces through the video (InsightFace) and groups them
|
||||
into people. Link each person to a reference and every shot uses the reference of the person
|
||||
@@ -252,23 +252,3 @@ The Token / Word Weight Editor accepts `(phrase:1.3)` directives in
|
||||
[One workflow](examples/h3_tara_optional_sam_loop.comfy.json) now has independent
|
||||
SAM 3.1 and shot-loop switches. Both default OFF. See
|
||||
[setup, disabled branches and status](H3_CAST_AND_TOKEN_CONTROLS.md#one-workflow-with-optional-sam-and-loop-111531).
|
||||
|
||||
### BFS Segment Picker (1.116.0)
|
||||
|
||||
Segment an image into objects with SAM 3.1 and pick them with the mouse. Connect a
|
||||
Load Checkpoint node with `sam3.1_multiplex_fp16.safetensors` (its MODEL and CLIP),
|
||||
press **▶ Segment** (runs only this node) and the objects appear on a canvas in the
|
||||
node: hover to see one, click to select it. The automatic pass looks for the
|
||||
`concepts` (default `person, animal`); add what it missed with **Find** (a SAM 3 text
|
||||
prompt) or with keep / exclude **points**. Outputs: the selection's mask, one mask per
|
||||
selected object, the image with the selection tinted (`mark_color` at `mark_opacity`,
|
||||
magenta at 0.3 by default, the Colored Mark convention), a numbered overlay and the
|
||||
objects as JSON. The selection and the added objects are saved with the workflow and
|
||||
segmented again on load.
|
||||
|
||||
Hand edits (1.118.0): **🖌 Brush** paints extra area into the mark (e.g. hair SAM left
|
||||
out), **➖ Remove** cuts out what SAM segments at the points you place, and **🧽 Eraser**
|
||||
paints what to cut out. Each becomes a chip; unselect it to switch it off. Brush and
|
||||
eraser strokes belong to the image they were painted on and are dropped when the image
|
||||
changes. Fixed: clicking inside the node no longer shrinks its canvas while the side
|
||||
panel is open.
|
||||
|
||||
@@ -132,12 +132,6 @@ try:
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] Video Compare node not loaded: {_e!r}")
|
||||
VCMP_NODE_CLASS_MAPPINGS, VCMP_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
try:
|
||||
from .bfs_segment_picker import NODE_CLASS_MAPPINGS as SEGPICK_NODE_CLASS_MAPPINGS
|
||||
from .bfs_segment_picker import NODE_DISPLAY_NAME_MAPPINGS as SEGPICK_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] Segment Picker node not loaded: {_e!r}")
|
||||
SEGPICK_NODE_CLASS_MAPPINGS, SEGPICK_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
|
||||
try:
|
||||
from .minimax_h3_clip_video import NODE_CLASS_MAPPINGS as H3CLIP_NODE_CLASS_MAPPINGS
|
||||
@@ -187,7 +181,6 @@ NODE_CLASS_MAPPINGS = {
|
||||
**HSMS_NODE_CLASS_MAPPINGS,
|
||||
**LSURG_NODE_CLASS_MAPPINGS,
|
||||
**VCMP_NODE_CLASS_MAPPINGS,
|
||||
**SEGPICK_NODE_CLASS_MAPPINGS,
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
@@ -224,7 +217,6 @@ NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
**HSMS_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**LSURG_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**VCMP_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**SEGPICK_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
}
|
||||
|
||||
WEB_DIRECTORY = "./web/js"
|
||||
|
||||
@@ -1,426 +0,0 @@
|
||||
"""BFS Segment Picker: segment an image into objects with SAM 3.1, pick them with the mouse, get their masks.
|
||||
|
||||
Connect SAM 3.1 from a Load Checkpoint node (sam3.1_multiplex_fp16.safetensors gives its MODEL and CLIP). The node
|
||||
segments the first image of its input automatically (one SAM 3 text prompt per concept: "person, animal" by
|
||||
default) and shows every object on a canvas in the node. Click objects to select them; the outputs are the mask of the
|
||||
selection, one mask per selected object, the image with the selection tinted (the Colored Mark convention: magenta at
|
||||
30 % marks the subject a body swap replaces) and a numbered overlay of everything found.
|
||||
|
||||
When the automatic pass misses something, add it in the panel: type what to find (a SAM 3 text prompt, e.g. "red
|
||||
umbrella"), place keep / exclude points on it, or paint it with the brush (to grow the mark past what SAM found, e.g.
|
||||
over hair). Added objects appear at once (the panel asks the server through
|
||||
/bfs/segpick/*), and the workflow stores how each one was made, so a reload or another machine segments them again.
|
||||
|
||||
All state lives in the hidden `selection` STRING widget (JSON):
|
||||
{"selected": ["a3", "t1.0"], "added": [{"id": "t1", "text": "red umbrella"},
|
||||
{"id": "p2", "points": [{"x": .41, "y": .62, "label": 1}]},
|
||||
{"id": "b3", "image": "9f2c...", "strokes": [{"r": .02, "pts": [[.4, .3], [.45, .32]]}]}], "next": 4}
|
||||
An entry with "remove": true (r<n> points, e<n> eraser strokes) is cut out of the selection instead of added to it.
|
||||
Points are relative to the image (0..1), a stroke radius to its width. A brush entry belongs to the image it was painted
|
||||
on (`image`, see image_key): on another image it is dropped, while text and point entries are segmented again. Ids: a<k> automatic objects (left to right),
|
||||
t<n>.<k> the k-th match of a text entry, p<n> a point entry, b<n> a brush entry.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import uuid
|
||||
from collections import OrderedDict
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
|
||||
try:
|
||||
import folder_paths
|
||||
except Exception: # tests run without ComfyUI
|
||||
folder_paths = None
|
||||
|
||||
PALETTE = [(230, 70, 60), (45, 125, 225), (50, 180, 90), (235, 205, 35), (205, 55, 180), (30, 200, 205),
|
||||
(240, 130, 35), (135, 75, 210), (120, 200, 60), (225, 95, 150), (80, 160, 230), (190, 150, 90)]
|
||||
DEFAULT_CONCEPTS = "person, animal"
|
||||
PREVIEW_SIDE = 1024 # long side of the images the panel shows (outputs keep full resolution)
|
||||
MAX_SESSIONS = 6
|
||||
_SESSIONS: "OrderedDict[str, dict]" = OrderedDict()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------------------------- state
|
||||
|
||||
def parse_state(text: str | None) -> dict:
|
||||
"""The selection widget's JSON, tolerant of empty or hand-edited values."""
|
||||
try:
|
||||
s = json.loads(text) if text and text.strip() else {}
|
||||
except (TypeError, ValueError):
|
||||
s = {}
|
||||
if not isinstance(s, dict):
|
||||
s = {}
|
||||
added = []
|
||||
for e in s.get("added") or []:
|
||||
if not isinstance(e, dict) or not e.get("id"):
|
||||
continue
|
||||
n = len(added)
|
||||
if str(e.get("text") or "").strip():
|
||||
added.append({"id": str(e["id"]), "text": str(e["text"]).strip()})
|
||||
elif e.get("points"):
|
||||
pts = [{"x": float(p["x"]), "y": float(p["y"]), "label": 1 if p.get("label", 1) else 0}
|
||||
for p in e["points"] if isinstance(p, dict) and "x" in p and "y" in p]
|
||||
if any(p["label"] for p in pts):
|
||||
added.append({"id": str(e["id"]), "points": pts})
|
||||
elif e.get("strokes"):
|
||||
strokes = [{"r": float(st["r"]), "pts": [[float(x), float(y)] for x, y in st["pts"]]}
|
||||
for st in e["strokes"] if isinstance(st, dict) and st.get("pts")]
|
||||
if strokes:
|
||||
added.append({"id": str(e["id"]), "image": str(e.get("image") or ""), "strokes": strokes})
|
||||
if e.get("remove") and len(added) > n:
|
||||
added[-1]["remove"] = True
|
||||
selected = [str(i) for i in s.get("selected") or [] if isinstance(i, (str, int))]
|
||||
return {"selected": list(dict.fromkeys(selected)), "added": added, "next": int(s.get("next") or len(added) + 1)}
|
||||
|
||||
|
||||
def split_concepts(text: str) -> list[str]:
|
||||
return list(dict.fromkeys(c.strip() for c in str(text or "").replace("\n", ",").split(",") if c.strip()))
|
||||
|
||||
|
||||
def entry_key(entry: dict) -> str:
|
||||
"""What an added entry asks SAM for: an entry whose text or points change is segmented again."""
|
||||
return json.dumps({k: entry[k] for k in ("text", "points", "strokes", "remove") if k in entry}, sort_keys=True)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------------------ geometry
|
||||
|
||||
def box_of(mask: torch.Tensor) -> list[int]:
|
||||
ys, xs = torch.nonzero(mask, as_tuple=True)
|
||||
if len(xs) == 0:
|
||||
return [0, 0, 0, 0]
|
||||
x0, y0, x1, y1 = int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1
|
||||
return [x0, y0, x1 - x0, y1 - y0]
|
||||
|
||||
|
||||
def dedupe(found: list[tuple[torch.Tensor, dict]], overlap: float = .8) -> list[tuple[torch.Tensor, dict]]:
|
||||
"""Drop a mask that mostly repeats an earlier one (the animal prompt can find a person again, and the reverse)."""
|
||||
kept: list[tuple[torch.Tensor, dict]] = []
|
||||
for m, info in found:
|
||||
a = int(m.sum())
|
||||
if any(int((m & o).sum()) > overlap * min(a, int(o.sum())) for o, _ in kept):
|
||||
continue
|
||||
kept.append((m, info))
|
||||
return kept
|
||||
|
||||
|
||||
def compose(masks: list[torch.Tensor], h: int, w: int) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""(union [1, h, w], individual [n, h, w]) as float masks; an empty selection gives zero masks."""
|
||||
if not masks:
|
||||
z = torch.zeros(1, h, w)
|
||||
return z, z.clone()
|
||||
stack = torch.stack([m.float() for m in masks])
|
||||
return (stack.amax(0, keepdim=True)), stack
|
||||
|
||||
|
||||
def paint(strokes: list[dict], h: int, w: int) -> torch.Tensor:
|
||||
"""Brush strokes (relative points, radius relative to the width) as a bool mask [h, w]."""
|
||||
img = Image.new("L", (w, h), 0)
|
||||
draw = ImageDraw.Draw(img)
|
||||
for st in strokes:
|
||||
r = max(1., st["r"] * w)
|
||||
pts = [(x * w, y * h) for x, y in st["pts"]]
|
||||
if len(pts) > 1:
|
||||
draw.line(pts, fill=255, width=round(2 * r), joint="curve")
|
||||
for x, y in pts:
|
||||
draw.ellipse([x - r, y - r, x + r, y + r], fill=255)
|
||||
return torch.from_numpy(np.array(img) > 127)
|
||||
|
||||
|
||||
def tint(image: torch.Tensor, mask: torch.Tensor, color: tuple[int, int, int], opacity: float) -> torch.Tensor:
|
||||
"""image [1, h, w, 3] with the mask's pixels blended toward the color."""
|
||||
c = torch.tensor(color, dtype=image.dtype) / 255.
|
||||
a = (mask[0].to(image.dtype) * float(opacity))[..., None]
|
||||
out = image.clone()
|
||||
out[0, ..., :3] = image[0, ..., :3] * (1 - a) + c * a
|
||||
return out
|
||||
|
||||
|
||||
def parse_color(text: str) -> tuple[int, int, int]:
|
||||
s = str(text or "").strip().lstrip("#")
|
||||
if len(s) == 3:
|
||||
s = "".join(ch * 2 for ch in s)
|
||||
try:
|
||||
return int(s[0:2], 16), int(s[2:4], 16), int(s[4:6], 16)
|
||||
except (ValueError, IndexError):
|
||||
return 255, 0, 255
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------------------------------------------------- SAM
|
||||
|
||||
def _text_cond(clip, text: str, max_detections: int):
|
||||
cond = clip.encode_from_tokens_scheduled(clip.tokenize(text))
|
||||
# SAM3_Detect keeps one detection per text prompt unless the prompt asks for more (the multi-prompt form)
|
||||
return [[cond[0][0], {**cond[0][1], "sam3_multi_cond": [{"cond": cond[0][0], "attention_mask": cond[0][1].get("attention_mask"),
|
||||
"max_detections": int(max_detections)}]}]]
|
||||
|
||||
|
||||
def detect_text(sam: tuple, frame: torch.Tensor, text: str, threshold: float, max_detections: int) -> list[tuple[torch.Tensor, dict]]:
|
||||
from comfy_extras.nodes_sam3 import SAM3_Detect
|
||||
model, clip = sam
|
||||
h, w = frame.shape[1:3]
|
||||
masks, boxes = SAM3_Detect.execute(model=model, image=frame, conditioning=_text_cond(clip, text, max_detections),
|
||||
threshold=float(threshold), refine_iterations=1, individual_masks=True).args
|
||||
masks = masks.cpu() > .5 if masks.numel() else torch.zeros(0, h, w, dtype=torch.bool)
|
||||
boxes = boxes[0] if boxes else []
|
||||
return [(m, {"score": round(float(b.get("score", 0)), 3)}) for m, b in zip(masks, boxes)]
|
||||
|
||||
|
||||
def detect_points(sam: tuple, frame: torch.Tensor, points: list[dict], threshold: float) -> torch.Tensor:
|
||||
from comfy_extras.nodes_sam3 import SAM3_Detect
|
||||
model, _ = sam
|
||||
h, w = frame.shape[1:3]
|
||||
pos = [{"x": p["x"] * w, "y": p["y"] * h} for p in points if p["label"]]
|
||||
neg = [{"x": p["x"] * w, "y": p["y"] * h} for p in points if not p["label"]]
|
||||
masks = SAM3_Detect.execute(model=model, image=frame, positive_coords=json.dumps(pos), negative_coords=json.dumps(neg),
|
||||
threshold=float(threshold), refine_iterations=2).args[0]
|
||||
return (masks[0].cpu() > .5) if masks.numel() else torch.zeros(h, w, dtype=torch.bool)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------------------- session
|
||||
|
||||
def image_key(frame: torch.Tensor) -> str:
|
||||
"""Which image this is, from a 64x64 thumbnail and the size: what a brush entry is tied to."""
|
||||
small = torch.nn.functional.interpolate(frame[..., :3].movedim(-1, 1).float(), size=(64, 64), mode="area")
|
||||
digest = hashlib.sha1((small * 255).round().byte().numpy().tobytes())
|
||||
digest.update(str(tuple(frame.shape)).encode())
|
||||
return digest.hexdigest()[:12]
|
||||
|
||||
|
||||
def _session_key(frame: torch.Tensor, spec: dict, sam: tuple) -> str:
|
||||
digest = hashlib.sha1(image_key(frame).encode())
|
||||
digest.update(json.dumps(spec, sort_keys=True).encode())
|
||||
digest.update(str(id(getattr(sam[0], "model", sam[0]))).encode()) # another SAM checkpoint segments again
|
||||
return digest.hexdigest()[:16]
|
||||
|
||||
|
||||
def _remember(sid: str, sess: dict) -> dict:
|
||||
_SESSIONS[sid] = sess
|
||||
_SESSIONS.move_to_end(sid)
|
||||
while len(_SESSIONS) > MAX_SESSIONS:
|
||||
_SESSIONS.popitem(last=False)
|
||||
return sess
|
||||
|
||||
|
||||
def auto_objects(sam: tuple, frame: torch.Tensor, concepts: list[str], threshold: float, max_per_concept: int, min_area: float) -> list[dict]:
|
||||
h, w = frame.shape[1:3]
|
||||
found = []
|
||||
for concept in concepts:
|
||||
for m, info in detect_text(sam, frame, concept, threshold, max_per_concept):
|
||||
if int(m.sum()) >= max(64, min_area * h * w):
|
||||
found.append((m, {**info, "label": concept}))
|
||||
found = dedupe(found)
|
||||
found.sort(key=lambda f: box_of(f[0])[0]) # left to right, so "a1" reads naturally
|
||||
return [{"id": f"a{k + 1}", "label": info["label"], "source": "auto", "score": info["score"], "mask": m}
|
||||
for k, (m, info) in enumerate(found)]
|
||||
|
||||
|
||||
def entry_objects(sam: tuple, frame: torch.Tensor, entry: dict, threshold: float, max_per_concept: int) -> list[dict]:
|
||||
objs = _entry_objects(sam, frame, entry, threshold, max_per_concept)
|
||||
if entry.get("remove"):
|
||||
for o in objs:
|
||||
o["remove"], o["label"] = True, "eraser" if entry.get("strokes") else "remove"
|
||||
return objs
|
||||
|
||||
|
||||
def _entry_objects(sam: tuple, frame: torch.Tensor, entry: dict, threshold: float, max_per_concept: int) -> list[dict]:
|
||||
if entry.get("text"):
|
||||
hits = dedupe(detect_text(sam, frame, entry["text"], threshold, max_per_concept))
|
||||
hits.sort(key=lambda f: box_of(f[0])[0])
|
||||
return [{"id": f"{entry['id']}.{k}", "label": entry["text"], "source": "text", "entry": entry["id"],
|
||||
"score": info["score"], "mask": m} for k, (m, info) in enumerate(hits)]
|
||||
if entry.get("strokes"):
|
||||
m = paint(entry["strokes"], *frame.shape[1:3])
|
||||
return [{"id": entry["id"], "label": "brush", "source": "brush", "entry": entry["id"], "score": None, "mask": m}] if m.any() else []
|
||||
m = detect_points(sam, frame, entry["points"], threshold)
|
||||
if int(m.sum()) < 16:
|
||||
return []
|
||||
return [{"id": entry["id"], "label": "points", "source": "points", "entry": entry["id"], "score": None, "mask": m}]
|
||||
|
||||
|
||||
def session_for(sam: tuple, frame: torch.Tensor, spec: dict) -> tuple[str, dict]:
|
||||
sid = _session_key(frame, spec, sam)
|
||||
if sid in _SESSIONS:
|
||||
_SESSIONS.move_to_end(sid)
|
||||
_SESSIONS[sid]["sam"] = sam
|
||||
return sid, _SESSIONS[sid]
|
||||
objs = auto_objects(sam, frame, split_concepts(spec["concepts"]), spec["threshold"], spec["max_per_concept"], spec["min_area"])
|
||||
# the panel's /bfs/segpick/add segments with the SAM the node last ran with
|
||||
return sid, _remember(sid, {"frame": frame.cpu(), "image_key": image_key(frame), "spec": spec, "auto": objs, "entries": {}, "sam": sam})
|
||||
|
||||
|
||||
def ensure_entry(sess: dict, entry: dict) -> list[dict]:
|
||||
"""The objects of an added entry, segmented once per (entry id, what it asks for)."""
|
||||
key = entry_key(entry)
|
||||
have = sess["entries"].get(entry["id"])
|
||||
if have is None or have["key"] != key:
|
||||
spec = sess["spec"]
|
||||
have = {"key": key, "objects": entry_objects(sess["sam"], sess["frame"], entry, spec["threshold"], spec["max_per_concept"])}
|
||||
sess["entries"][entry["id"]] = have
|
||||
return have["objects"]
|
||||
|
||||
|
||||
def all_objects(sess: dict, state: dict) -> list[dict]:
|
||||
objs = list(sess["auto"])
|
||||
for e in state["added"]:
|
||||
if e.get("strokes") and e["image"] and e["image"] != sess["image_key"]:
|
||||
continue # painted on another image
|
||||
objs += ensure_entry(sess, e)
|
||||
return objs
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------------------------- panel files
|
||||
|
||||
def _temp_dir() -> str:
|
||||
base = folder_paths.get_temp_directory() if folder_paths else os.path.join(os.path.dirname(__file__), ".temp")
|
||||
d = os.path.join(base, "bfs_segpick")
|
||||
os.makedirs(d, exist_ok=True)
|
||||
return d
|
||||
|
||||
|
||||
def _preview_size(h: int, w: int) -> tuple[int, int]:
|
||||
s = min(1.0, PREVIEW_SIDE / max(h, w))
|
||||
return max(1, round(w * s)), max(1, round(h * s))
|
||||
|
||||
|
||||
def _save(img: Image.Image, prefix: str) -> dict:
|
||||
name = f"{prefix}_{uuid.uuid4().hex[:10]}.png"
|
||||
img.save(os.path.join(_temp_dir(), name), compress_level=1)
|
||||
return {"filename": name, "subfolder": "bfs_segpick", "type": "temp"}
|
||||
|
||||
|
||||
def panel_object(o: dict, size: tuple[int, int], h: int, w: int) -> dict:
|
||||
"""What the panel needs about one object: its mask as a small PNG (white on transparent) and where it is."""
|
||||
m = Image.fromarray(o["mask"].numpy().astype(np.uint8) * 255).resize(size, Image.NEAREST)
|
||||
rgba = Image.new("RGBA", size, (255, 255, 255, 0))
|
||||
rgba.putalpha(m)
|
||||
x, y, bw, bh = box_of(o["mask"])
|
||||
if "_file" not in o:
|
||||
o["_file"] = _save(rgba, f"m_{o['id']}")
|
||||
return {"id": o["id"], "label": o["label"], "source": o["source"], "entry": o.get("entry"), "score": o["score"],
|
||||
"remove": bool(o.get("remove")), "area": round(float(o["mask"].float().mean()), 5), "box": [round(x / w, 4), round(y / h, 4), round(bw / w, 4), round(bh / h, 4)],
|
||||
"mask": o["_file"]}
|
||||
|
||||
|
||||
def panel_data(sid: str, sess: dict, objects: list[dict]) -> dict:
|
||||
frame = sess["frame"]
|
||||
h, w = frame.shape[1:3]
|
||||
size = _preview_size(h, w)
|
||||
if "_image" not in sess:
|
||||
rgb = (frame[0, ..., :3].clamp(0, 1) * 255).byte().numpy()
|
||||
sess["_image"] = _save(Image.fromarray(rgb).resize(size, Image.LANCZOS), "img")
|
||||
return {"session": sid, "image_key": sess["image_key"], "width": int(w), "height": int(h), "image": sess["_image"],
|
||||
"objects": [panel_object(o, size, h, w) for o in objects]}
|
||||
|
||||
|
||||
def overlay(frame: torch.Tensor, objects: list[dict], selected: set[str]) -> torch.Tensor:
|
||||
rgb = (frame[0, ..., :3].clamp(0, 1) * 255).byte().numpy().astype(np.float32)
|
||||
h, w = rgb.shape[:2]
|
||||
for n, o in enumerate(objects):
|
||||
m = o["mask"].numpy()
|
||||
a = .6 if o["id"] in selected else .3
|
||||
rgb[m] = rgb[m] * (1 - a) + np.array(PALETTE[n % len(PALETTE)], np.float32) * a
|
||||
canvas = Image.fromarray(rgb.clip(0, 255).astype(np.uint8))
|
||||
draw = ImageDraw.Draw(canvas)
|
||||
size = max(14, h // 28)
|
||||
try:
|
||||
font = ImageFont.load_default(size=size)
|
||||
except TypeError:
|
||||
font = ImageFont.load_default()
|
||||
for n, o in enumerate(objects):
|
||||
x, y, bw, bh = box_of(o["mask"])
|
||||
color = PALETTE[n % len(PALETTE)]
|
||||
if o["id"] in selected:
|
||||
draw.rectangle([x, y, x + bw, y + bh], outline=color, width=max(2, h // 300))
|
||||
draw.text((x + 4, y + 4), o["id"], fill=(255, 255, 255), font=font, stroke_width=2, stroke_fill=(0, 0, 0))
|
||||
return torch.from_numpy(np.asarray(canvas, np.float32) / 255.)[None]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------------------------------------------- node
|
||||
|
||||
class BFSSegmentPicker:
|
||||
DESCRIPTION = __doc__.split("\n\n")[0] + " Click objects on the canvas to select them; add missed ones by text, points or the brush."
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"image": ("IMAGE",),
|
||||
"sam3_model": ("MODEL", {"tooltip": "SAM 3.1: MODEL of a Load Checkpoint node with sam3.1_multiplex_fp16.safetensors."}),
|
||||
"sam3_clip": ("CLIP", {"tooltip": "SAM 3.1: CLIP of the same Load Checkpoint node; turns the concepts and Find texts into SAM prompts."}),
|
||||
"concepts": ("STRING", {"default": DEFAULT_CONCEPTS, "tooltip": "What the automatic pass looks for, comma separated (one SAM 3 text prompt each), e.g. person, animal, car, chair. Empty = no automatic pass, only what you add."}),
|
||||
"threshold": ("FLOAT", {"default": .4, "min": .05, "max": .95, "step": .05, "tooltip": "SAM 3 detection score threshold: lower finds more (and more false hits)."}),
|
||||
"max_per_concept": ("INT", {"default": 8, "min": 1, "max": 32, "tooltip": "Most objects one concept or text prompt can return."}),
|
||||
"min_area": ("FLOAT", {"default": .002, "min": 0., "max": .5, "step": .001, "tooltip": "Automatic objects smaller than this share of the image are dropped."}),
|
||||
"mark_color": ("STRING", {"default": "#FF00FF", "tooltip": "Tint of the selection in the marked output (hex)."}),
|
||||
"mark_opacity": ("FLOAT", {"default": .3, "min": 0., "max": 1., "step": .05, "tooltip": "How strongly the marked output tints the selection (0.3 = the Colored Mark default)."}),
|
||||
"selection": ("STRING", {"default": "{}", "multiline": True, "tooltip": "Set by the canvas: the selected objects and the ones added by text or points."}),
|
||||
},
|
||||
"hidden": {"unique_id": "UNIQUE_ID"},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("MASK", "MASK", "IMAGE", "IMAGE", "STRING")
|
||||
RETURN_NAMES = ("mask", "masks", "marked", "overlay", "objects")
|
||||
OUTPUT_TOOLTIPS = ("Union of the selected objects, minus the selected remove / eraser ones (one mask).", "One mask per selected object (remove ones already cut out), in selection order.",
|
||||
"The image with the selection tinted (mark_color, mark_opacity).", "Every object found, numbered; the selection stronger.",
|
||||
"JSON: size, selected ids and every object's id, label, score, box (relative x, y, w, h) and area.")
|
||||
FUNCTION = "pick"
|
||||
CATEGORY = "BFS/Mask"
|
||||
OUTPUT_NODE = True
|
||||
|
||||
def pick(self, image, sam3_model, sam3_clip, concepts, threshold, max_per_concept, min_area, mark_color, mark_opacity, selection, unique_id=None):
|
||||
frame = image[:1, ..., :3].float().cpu()
|
||||
h, w = frame.shape[1:3]
|
||||
spec = {"concepts": ", ".join(split_concepts(concepts)), "threshold": round(float(threshold), 4),
|
||||
"max_per_concept": int(max_per_concept), "min_area": round(float(min_area), 5)}
|
||||
state = parse_state(selection)
|
||||
sid, sess = session_for((sam3_model, sam3_clip), frame, spec)
|
||||
objects = all_objects(sess, state)
|
||||
by_id = {o["id"]: o for o in objects}
|
||||
chosen = [by_id[i] for i in state["selected"] if i in by_id]
|
||||
cut = [o["mask"] for o in chosen if o.get("remove")]
|
||||
cut = torch.stack(cut).any(0) if cut else torch.zeros(h, w, dtype=torch.bool)
|
||||
mask, masks = compose([o["mask"] & ~cut for o in chosen if not o.get("remove")], h, w)
|
||||
marked = tint(frame, mask, parse_color(mark_color), mark_opacity)
|
||||
info = {"width": int(w), "height": int(h), "selected": [o["id"] for o in chosen],
|
||||
"objects": [{k: v for k, v in panel_object(o, _preview_size(h, w), h, w).items() if k != "mask"} for o in objects]}
|
||||
return {"ui": {"bfs_segpick": [panel_data(sid, sess, objects)]},
|
||||
"result": (mask, masks, marked, overlay(frame, objects, set(info["selected"])), json.dumps(info))}
|
||||
|
||||
|
||||
# -------------------------------------------------------------------------------------------------------------- routes
|
||||
|
||||
def _routes():
|
||||
try:
|
||||
from aiohttp import web
|
||||
from server import PromptServer
|
||||
except Exception:
|
||||
return
|
||||
|
||||
@PromptServer.instance.routes.post("/bfs/segpick/add")
|
||||
async def add(request):
|
||||
"""Segment one added entry ({session, entry: {id, text} | {id, points}}) on the session's image now."""
|
||||
body = await request.json()
|
||||
sess = _SESSIONS.get(str(body.get("session") or ""))
|
||||
if sess is None:
|
||||
return web.json_response({"error": "This image is no longer in memory: run the node once, then add again."}, status=404)
|
||||
state = parse_state(json.dumps({"added": [body.get("entry") or {}]}))
|
||||
if not state["added"]:
|
||||
return web.json_response({"error": "Give a text prompt, at least one keep point or a brush stroke."}, status=400)
|
||||
try:
|
||||
objs = ensure_entry(sess, state["added"][0])
|
||||
except Exception as e: # noqa: BLE001 - shown in the panel
|
||||
return web.json_response({"error": f"SAM 3 failed: {e}"}, status=500)
|
||||
h, w = sess["frame"].shape[1:3]
|
||||
size = _preview_size(h, w)
|
||||
return web.json_response({"objects": [panel_object(o, size, h, w) for o in objs]})
|
||||
|
||||
|
||||
_routes()
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"BFSSegmentPicker": BFSSegmentPicker}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"BFSSegmentPicker": "BFS Segment Picker (SAM 3)"}
|
||||
+5
-14
@@ -1685,16 +1685,7 @@ async def _off_loop(fn):
|
||||
"""Run slow work in a thread so the server keeps answering (and the panel gets progress events)."""
|
||||
import asyncio
|
||||
_outside_prompt()
|
||||
return await asyncio.get_running_loop().run_in_executor(None, lambda: _like_a_node(fn))
|
||||
|
||||
|
||||
def _like_a_node(fn):
|
||||
"""Run a panel button's model work under torch.inference_mode(), as ComfyUI runs every node (execution.py). After
|
||||
the workflow ran once, the VLM and SAM keep tensors made in inference mode; using them outside it raises
|
||||
"Inference tensors do not track version counter" (issue #14)."""
|
||||
import torch
|
||||
with torch.inference_mode():
|
||||
return fn()
|
||||
return await asyncio.get_running_loop().run_in_executor(None, fn)
|
||||
|
||||
|
||||
def _outside_prompt() -> None:
|
||||
@@ -5859,10 +5850,10 @@ try:
|
||||
preview_color = (mask_override or {}).get("preview_color") or (
|
||||
seg.get("color_guide_color") if seg.get("color_guide") else "#ff283c")
|
||||
out = await asyncio.get_running_loop().run_in_executor(
|
||||
None, lambda: _like_a_node(lambda: mask_preview(path, a, seg["start"], seg["end"] - seg["start"], spec,
|
||||
None, lambda: mask_preview(path, a, seg["start"], seg["end"] - seg["start"], spec,
|
||||
int(body.get("count", 6)),
|
||||
preview_color,
|
||||
seg.get("color_guide_opacity") if seg.get("color_guide") else 0.55)))
|
||||
seg.get("color_guide_opacity") if seg.get("color_guide") else 0.55))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return web.json_response({"error": f"{type(exc).__name__}: {exc}"}, status=500)
|
||||
return web.json_response(out)
|
||||
@@ -5887,8 +5878,8 @@ try:
|
||||
import asyncio
|
||||
_outside_prompt()
|
||||
out = await asyncio.get_running_loop().run_in_executor(
|
||||
None, lambda: _like_a_node(lambda: [dict(vlm_shot(clip, path, a, s["start"], s["end"], cfg),
|
||||
start=s["start"], end=s["end"]) for s in pick]))
|
||||
None, lambda: [dict(vlm_shot(clip, path, a, s["start"], s["end"], cfg), start=s["start"], end=s["end"])
|
||||
for s in pick])
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return web.json_response({"error": f"{type(exc).__name__}: {exc}"}, status=500)
|
||||
return web.json_response({"segs": out})
|
||||
|
||||
@@ -67,17 +67,6 @@ def ref_pictures(reference) -> list:
|
||||
return out[:MAX_REFS]
|
||||
|
||||
|
||||
def save_stills(frames: dict, out_dir: str) -> dict:
|
||||
"""{source: [1, H, W, 3] in 0..1} as lossless PNGs -> {source: file name}."""
|
||||
from PIL import Image
|
||||
names = {}
|
||||
for k, v in frames.items():
|
||||
names[k] = f"{k}.png"
|
||||
Image.fromarray((v[0, ..., :3].float().clamp(0, 1).cpu().numpy() * 255).round().astype(np.uint8)).save(
|
||||
os.path.join(out_dir, names[k]))
|
||||
return names
|
||||
|
||||
|
||||
def save_refs(reference, out_dir: str, height: int) -> list:
|
||||
"""The reference pictures as PNGs no taller than the player (no upscale) -> file names."""
|
||||
from PIL import Image
|
||||
@@ -206,9 +195,6 @@ class BFSVideoCompare:
|
||||
w.close()
|
||||
diff_map = torch.cat(maps, 0)
|
||||
mae, mse = torch.cat(mae), torch.cat(mse)
|
||||
# a single frame is a picture: lossless PNGs at the compared size, which the player's ⬇ save PNG downloads
|
||||
stills = save_stills({"guide": video_chunk(guide, 0, 1, H, W), "target": video_chunk(target, 0, 1, H, W),
|
||||
"difference": diff_map[:1]}, out_dir) if n == 1 else {}
|
||||
mean = float(mae.mean())
|
||||
worst, best = int(mae.argmax()), int(mae.argmin())
|
||||
report = "\n".join([
|
||||
@@ -224,7 +210,6 @@ class BFSVideoCompare:
|
||||
ui = {"bfs_compare": [{**files, "fps": float(fps), "frames": n, "width": PW, "height": PH,
|
||||
"labels": [label_guide, label_target], "mae": [round(float(x), 2) for x in mae],
|
||||
"refs": refs,
|
||||
"stills": {k: {"filename": f, "subfolder": sub, "type": "temp"} for k, f in stills.items()},
|
||||
"report": report, "notes": notes}]}
|
||||
return {"ui": ui, "result": (diff_map, report, mean)}
|
||||
|
||||
|
||||
@@ -48,8 +48,9 @@ def dense_backend_selection(override):
|
||||
|
||||
|
||||
def temporal_layout(native, refs, keyframes, video_index=1):
|
||||
if keyframes:
|
||||
raise ValueError('Native temporal mode cannot be combined with Add Guide, first-frame guides or history. Remove those guides.')
|
||||
# Add Guide keyframes (e.g. the previous shot's last frames as history) pack as 'cond' rows right after the text,
|
||||
# before the references, at the target's own timeline positions and never denoised. They stay as they are: no
|
||||
# temporal bias touches them, and the reference rows are located after them.
|
||||
required = ('position_ids', 'segments', 'seq_len', 'signature')
|
||||
if any(not hasattr(native, x) for x in required):
|
||||
raise RuntimeError('Unsupported H3 PackedLayout API; update ComfyUI.')
|
||||
@@ -59,8 +60,9 @@ def temporal_layout(native, refs, keyframes, video_index=1):
|
||||
target = torch.zeros_like(source)
|
||||
ta, tb, _ = next(s for s in native.segments if s[2] == 'video')
|
||||
target[ta:tb] = True
|
||||
# Native packing is text, each reference's optional audio then video, targets.
|
||||
# Native packing is text, keyframe cond rows, each reference's optional audio then video, targets.
|
||||
cursor = next(s[1] for s in native.segments if s[2] == 'text')
|
||||
cursor = max([cursor] + [s[1] for s in native.segments if s[2] in ('cond', 'cond_audio')])
|
||||
videos = 0
|
||||
for ref in refs:
|
||||
kind = ref['kind']
|
||||
|
||||
+1
-1
@@ -1,7 +1,7 @@
|
||||
[project]
|
||||
name = "bfsnodes"
|
||||
description = "Custom ComfyUI nodes for reference-guided video: BFS Sampler (guide-driven IC-LoRAs, auto crop, mask inpainting, temporal chunking), LTX Multiple Controls / Identity Transfer / Edit Anything, Head-Swap and Anime2Real conditioning, MiniMax-H3 latent guides, RoPE layouts and loaders, LoRA Surgery, and Shot Planner / Shot Loop (tabbed panel): split long videos into shots at camera cuts, face-tracked cast, SAM 3 mask per shot with full frame, paste, mask, crop and crop + mask generation modes, optional translucent Colored Mark overlay for manually selected regions, optional Qwen3-VL to suggest settings, describe references ({details}) and the person to replace ({target}), continuity between shots, H3 conditioning with an optional duet and prompt writer, join with the original timing, audio and a side-by-side comparison. H3 Duet: pin a source clip beside the video, training-free."
|
||||
version = "1.118.1"
|
||||
version = "1.115.33"
|
||||
license = {file = "LICENSE"}
|
||||
dependencies = [
|
||||
"torch",
|
||||
|
||||
@@ -1,217 +0,0 @@
|
||||
import importlib.util
|
||||
import json
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SPEC = importlib.util.spec_from_file_location("bfs_segment_picker", ROOT / "bfs_segment_picker.py")
|
||||
SP = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(SP)
|
||||
|
||||
|
||||
def rect(h, w, x0, y0, x1, y1):
|
||||
m = torch.zeros(h, w, dtype=torch.bool)
|
||||
m[y0:y1, x0:x1] = True
|
||||
return m
|
||||
|
||||
|
||||
class StateTest(unittest.TestCase):
|
||||
def test_empty_and_broken_values(self):
|
||||
for v in (None, "", "{}", "not json", "[1, 2]"):
|
||||
self.assertEqual(SP.parse_state(v), {"selected": [], "added": [], "next": 1})
|
||||
|
||||
def test_keeps_valid_entries_only(self):
|
||||
s = SP.parse_state(json.dumps({
|
||||
"selected": ["a1", "a1", "t1.0", 3],
|
||||
"added": [{"id": "t1", "text": " red car "}, {"id": "p2", "points": [{"x": .5, "y": .5, "label": 1}]},
|
||||
{"id": "p3", "points": [{"x": .1, "y": .1, "label": 0}]}, # only exclude points: nothing to keep
|
||||
{"text": "no id"}, {"id": "t4", "text": " "}],
|
||||
"next": 5}))
|
||||
self.assertEqual(s["selected"], ["a1", "t1.0", "3"])
|
||||
self.assertEqual(s["added"], [{"id": "t1", "text": "red car"}, {"id": "p2", "points": [{"x": .5, "y": .5, "label": 1}]}])
|
||||
self.assertEqual(s["next"], 5)
|
||||
|
||||
def test_concepts(self):
|
||||
self.assertEqual(SP.split_concepts("person, animal,\ncar, person ,,"), ["person", "animal", "car"])
|
||||
self.assertEqual(SP.split_concepts(""), [])
|
||||
|
||||
def test_entry_key_changes_with_what_it_asks(self):
|
||||
a = SP.entry_key({"id": "t1", "text": "car"})
|
||||
self.assertEqual(a, SP.entry_key({"id": "t9", "text": "car"}))
|
||||
self.assertNotEqual(a, SP.entry_key({"id": "t1", "text": "bus"}))
|
||||
|
||||
|
||||
class GeometryTest(unittest.TestCase):
|
||||
def test_box(self):
|
||||
self.assertEqual(SP.box_of(rect(10, 20, 3, 2, 8, 6)), [3, 2, 5, 4])
|
||||
self.assertEqual(SP.box_of(torch.zeros(4, 4, dtype=torch.bool)), [0, 0, 0, 0])
|
||||
|
||||
def test_dedupe_drops_repeats_keeps_neighbours(self):
|
||||
a = rect(20, 20, 0, 0, 10, 10)
|
||||
repeat = rect(20, 20, 0, 0, 10, 9) # 90 % inside a
|
||||
other = rect(20, 20, 8, 8, 18, 18) # touches a a little
|
||||
kept = SP.dedupe([(a, {"n": 1}), (repeat, {"n": 2}), (other, {"n": 3})])
|
||||
self.assertEqual([i["n"] for _, i in kept], [1, 3])
|
||||
|
||||
def test_compose(self):
|
||||
u, each = SP.compose([], 4, 5)
|
||||
self.assertEqual(tuple(u.shape), (1, 4, 5)); self.assertEqual(float(u.sum()), 0.)
|
||||
a, b = rect(4, 5, 0, 0, 2, 2), rect(4, 5, 1, 1, 3, 3)
|
||||
u, each = SP.compose([a, b], 4, 5)
|
||||
self.assertEqual(tuple(each.shape), (2, 4, 5))
|
||||
self.assertEqual(float(u.sum()), float((a | b).sum()))
|
||||
|
||||
def test_tint_only_inside(self):
|
||||
img = torch.zeros(1, 4, 4, 3)
|
||||
m = rect(4, 4, 0, 0, 2, 2).float()[None]
|
||||
out = SP.tint(img, m, (255, 0, 255), .3)
|
||||
self.assertAlmostEqual(float(out[0, 0, 0, 0]), .3, places=5)
|
||||
self.assertAlmostEqual(float(out[0, 0, 0, 1]), 0., places=5)
|
||||
self.assertEqual(float(out[0, 3, 3].sum()), 0.)
|
||||
|
||||
def test_color(self):
|
||||
self.assertEqual(SP.parse_color("#FF00FF"), (255, 0, 255))
|
||||
self.assertEqual(SP.parse_color("0f0"), (0, 255, 0))
|
||||
self.assertEqual(SP.parse_color("nope"), (255, 0, 255))
|
||||
|
||||
|
||||
class SessionTest(unittest.TestCase):
|
||||
"""The node with SAM replaced: what is segmented once, what is segmented again, what the outputs hold."""
|
||||
|
||||
def setUp(self):
|
||||
self.calls = []
|
||||
h, w = 20, 40
|
||||
people = [rect(h, w, 25, 2, 35, 18), rect(h, w, 2, 2, 12, 18)]
|
||||
|
||||
def detect_text(sam, frame, text, threshold, max_detections):
|
||||
self.calls.append(("text", text))
|
||||
if text == "person":
|
||||
return [(m, {"score": .9}) for m in people]
|
||||
if text == "dog":
|
||||
return [(rect(h, w, 15, 10, 20, 15), {"score": .7})]
|
||||
return []
|
||||
|
||||
def detect_points(sam, frame, points, threshold):
|
||||
self.calls.append(("points", len(points)))
|
||||
return rect(h, w, 14, 0, 19, 6)
|
||||
|
||||
SP.detect_text, SP.detect_points = detect_text, detect_points
|
||||
SP._SESSIONS.clear()
|
||||
self.frame = torch.rand(1, h, w, 3)
|
||||
self.sam = (object(), object())
|
||||
self.tmp = Path(__file__).resolve().parent / ".segpick_tmp"
|
||||
SP._temp_dir = lambda: str(self.tmp.mkdir(exist_ok=True) or self.tmp)
|
||||
|
||||
def tearDown(self):
|
||||
for f in self.tmp.glob("*.png"):
|
||||
f.unlink()
|
||||
if self.tmp.exists():
|
||||
self.tmp.rmdir()
|
||||
|
||||
def run_node(self, selection, concepts="person"):
|
||||
return SP.BFSSegmentPicker().pick(self.frame, self.sam[0], self.sam[1], concepts, .4, 8, 0., "#FF00FF", .3, json.dumps(selection))
|
||||
|
||||
def test_auto_ids_left_to_right_and_selection(self):
|
||||
out = self.run_node({"selected": ["a2"]})
|
||||
mask, masks, marked, overlay, info = out["result"]
|
||||
info = json.loads(info)
|
||||
self.assertEqual([o["id"] for o in info["objects"]], ["a1", "a2"])
|
||||
self.assertLess(info["objects"][0]["box"][0], info["objects"][1]["box"][0])
|
||||
self.assertEqual(info["selected"], ["a2"])
|
||||
self.assertEqual(float(mask[0, 10, 30]), 1.) # a2 is the person on the right
|
||||
self.assertEqual(float(mask[0, 10, 5]), 0.)
|
||||
self.assertEqual(tuple(masks.shape), (1, 20, 40))
|
||||
self.assertGreater(float(marked[0, 10, 30, 0]), float(self.frame[0, 10, 30, 0]) * .7 - 1e-6)
|
||||
panel = out["ui"]["bfs_segpick"][0]
|
||||
self.assertEqual(len(panel["objects"]), 2)
|
||||
self.assertTrue((self.tmp / panel["objects"][0]["mask"]["filename"]).exists())
|
||||
|
||||
def test_segments_once_then_only_new_entries(self):
|
||||
self.run_node({"selected": ["a1"]})
|
||||
self.run_node({"selected": ["a2"]}) # only the selection changed
|
||||
self.assertEqual(self.calls, [("text", "person")])
|
||||
sel = {"selected": ["t1.0", "p2"], "added": [{"id": "t1", "text": "dog"},
|
||||
{"id": "p2", "points": [{"x": .37, "y": .05, "label": 1}]}]}
|
||||
_, masks, *_ , info = self.run_node(sel)["result"]
|
||||
self.assertEqual(json.loads(info)["selected"], ["t1.0", "p2"])
|
||||
self.assertEqual(tuple(masks.shape), (2, 20, 40))
|
||||
self.run_node(sel)
|
||||
self.assertEqual(self.calls, [("text", "person"), ("text", "dog"), ("points", 1)])
|
||||
sel["added"][0]["text"] = "cat" # an edited entry is segmented again
|
||||
_, _, _, _, info = self.run_node(sel)["result"]
|
||||
self.assertEqual(self.calls[-1], ("text", "cat"))
|
||||
self.assertEqual(json.loads(info)["selected"], ["p2"]) # "cat" found nothing, so t1.0 is gone
|
||||
|
||||
def test_brush_is_dropped_on_another_image(self):
|
||||
key = SP.image_key(self.frame)
|
||||
sel = {"selected": ["a1", "b1"], "added": [{"id": "b1", "image": key, "strokes": [{"r": .05, "pts": [[.5, .5]]}]}]}
|
||||
out = self.run_node(sel)
|
||||
self.assertEqual(out["ui"]["bfs_segpick"][0]["image_key"], key)
|
||||
self.assertEqual(json.loads(out["result"][4])["selected"], ["a1", "b1"])
|
||||
self.frame = torch.rand(1, 20, 40, 3) # another image: the strokes no longer apply
|
||||
mask, _, _, _, info = self.run_node(sel)["result"]
|
||||
self.assertEqual(json.loads(info)["selected"], ["a1"])
|
||||
self.assertEqual(float(mask[0, 10, 20]), 0.)
|
||||
|
||||
def test_remove_entries_cut_the_selection(self):
|
||||
key = SP.image_key(self.frame)
|
||||
sel = {"selected": ["a1", "e1", "r2"], "added": [
|
||||
{"id": "e1", "image": key, "remove": True, "strokes": [{"r": .05, "pts": [[.175, .5]]}]},
|
||||
{"id": "r2", "remove": True, "points": [{"x": .4, "y": .1, "label": 1}]}]}
|
||||
out = self.run_node(sel)
|
||||
mask, masks, _, _, info = out["result"]
|
||||
self.assertEqual(float(mask[0, 10, 7]), 0.) # erased out of a1
|
||||
self.assertEqual(float(mask[0, 10, 3]), 1.) # the rest of a1 stays
|
||||
self.assertEqual(tuple(masks.shape), (1, 20, 40)) # remove entries are not masks of their own
|
||||
self.assertEqual(float(masks[0, 10, 7]), 0.)
|
||||
self.assertEqual(json.loads(info)["selected"], ["a1", "e1", "r2"])
|
||||
panel = {o["id"]: o for o in out["ui"]["bfs_segpick"][0]["objects"]}
|
||||
self.assertEqual((panel["e1"]["label"], panel["e1"]["remove"]), ("eraser", True))
|
||||
self.assertEqual((panel["r2"]["label"], panel["r2"]["remove"]), ("remove", True))
|
||||
self.assertFalse(panel["a1"]["remove"])
|
||||
sel["selected"] = ["a1", "r2"] # unselecting the eraser gives a1 back whole
|
||||
self.assertEqual(float(self.run_node(sel)["result"][0][0, 10, 7]), 1.)
|
||||
|
||||
def test_unknown_ids_are_ignored(self):
|
||||
mask, masks, *_ = self.run_node({"selected": ["a9", "t5.0"]})["result"]
|
||||
self.assertEqual(float(mask.sum()), 0.)
|
||||
self.assertEqual(tuple(masks.shape), (1, 20, 40))
|
||||
|
||||
def test_another_sam_is_another_session(self):
|
||||
self.run_node({})
|
||||
self.sam = (object(), object())
|
||||
self.run_node({})
|
||||
self.assertEqual(self.calls, [("text", "person"), ("text", "person")])
|
||||
|
||||
def test_other_concepts_are_another_session(self):
|
||||
self.run_node({}, concepts="person")
|
||||
self.run_node({}, concepts="person, dog")
|
||||
self.assertEqual(self.calls, [("text", "person"), ("text", "person"), ("text", "dog")])
|
||||
|
||||
|
||||
class BrushTest(unittest.TestCase):
|
||||
def test_state_keeps_strokes(self):
|
||||
s = SP.parse_state(json.dumps({"added": [{"id": "b1", "strokes": [{"r": .1, "pts": [[.2, .5], [.8, .5]]}]},
|
||||
{"id": "b2", "strokes": [{"r": .1, "pts": []}]}]}))
|
||||
self.assertEqual(s["added"], [{"id": "b1", "image": "", "strokes": [{"r": .1, "pts": [[.2, .5], [.8, .5]]}]}])
|
||||
self.assertNotEqual(SP.entry_key(s["added"][0]), SP.entry_key({"id": "b1", "strokes": [{"r": .2, "pts": [[.2, .5]]}]}))
|
||||
r = SP.parse_state(json.dumps({"added": [{"id": "r1", "remove": True, "points": [{"x": .5, "y": .5, "label": 1}]},
|
||||
{"id": "p2", "remove": False, "points": [{"x": .5, "y": .5, "label": 1}]}]}))
|
||||
self.assertEqual([e.get("remove") for e in r["added"]], [True, None])
|
||||
|
||||
def test_paint_covers_the_stroke_only(self):
|
||||
m = SP.paint([{"r": .1, "pts": [[.2, .5], [.8, .5]]}], 50, 100)
|
||||
self.assertTrue(m[25, 20] and m[25, 50] and m[25, 80])
|
||||
self.assertFalse(m[5, 50] or m[25, 5] or m[25, 95])
|
||||
|
||||
def test_brush_entry_needs_no_sam(self):
|
||||
frame = torch.zeros(1, 50, 100, 3)
|
||||
objs = SP.entry_objects(None, frame, {"id": "b3", "strokes": [{"r": .05, "pts": [[.5, .5]]}]}, .4, 8)
|
||||
self.assertEqual([(o["id"], o["source"]) for o in objs], [("b3", "brush")])
|
||||
self.assertTrue(objs[0]["mask"][25, 50])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1759,19 +1759,3 @@ class PromptReferenceContractTest(unittest.TestCase):
|
||||
["fox wearing blue pants", "city street"], "Use Picture 2 for the setting", {},
|
||||
style="captions", source_facts="A dancer is moving in a street.")
|
||||
self.assertEqual(generator.call_count, 2)
|
||||
|
||||
|
||||
class PanelInferenceModeTest(unittest.TestCase):
|
||||
"""Issue #14: panel buttons reuse models a workflow run left with inference tensors; they must run like a node."""
|
||||
|
||||
def test_panel_work_runs_in_inference_mode(self):
|
||||
import asyncio
|
||||
with torch.inference_mode():
|
||||
cache = torch.zeros(3) # e.g. a VLM buffer created while the workflow ran
|
||||
|
||||
def work():
|
||||
self.assertTrue(torch.is_inference_mode_enabled())
|
||||
cache.add_(1)
|
||||
return cache.sum().item()
|
||||
|
||||
self.assertEqual(asyncio.run(SL._off_loop(work)), 3.0)
|
||||
|
||||
@@ -107,19 +107,6 @@ class CompareNodeTest(unittest.TestCase):
|
||||
with self.assertRaisesRegex(ValueError, "nonempty"):
|
||||
VC.BFSVideoCompare().compare([], t)
|
||||
|
||||
def test_a_single_frame_also_sends_lossless_pngs_at_the_compared_size(self):
|
||||
from PIL import Image
|
||||
guide, target = torch.rand(1, 40, 60, 3), torch.rand(1, 20, 30, 3)
|
||||
ui = VC.BFSVideoCompare().compare(guide, target, preview_height=16)["ui"]["bfs_compare"][0]
|
||||
self.assertEqual(sorted(ui["stills"]), ["difference", "guide", "target"])
|
||||
for k, f in ui["stills"].items():
|
||||
im = Image.open(os.path.join(_TMP, f["subfolder"], f["filename"]))
|
||||
self.assertEqual(im.size, (60, 40)) # the guide's size, not the 16 px preview
|
||||
g = Image.open(os.path.join(_TMP, ui["stills"]["guide"]["subfolder"], ui["stills"]["guide"]["filename"]))
|
||||
self.assertEqual(g.getpixel((7, 5)), tuple(int(round(float(c) * 255)) for c in guide[0, 5, 7])) # lossless
|
||||
video = VC.BFSVideoCompare().compare(torch.rand(3, 8, 8, 3), torch.rand(3, 8, 8, 3), preview_height=8)
|
||||
self.assertEqual(video["ui"]["bfs_compare"][0]["stills"], {})
|
||||
|
||||
def test_reference_pictures_go_to_the_player_no_taller_than_it(self):
|
||||
from PIL import Image
|
||||
guide, target = torch.rand(4, 32, 48, 3), torch.rand(4, 32, 48, 3)
|
||||
|
||||
@@ -36,7 +36,24 @@ def test_wrong_video_index_and_temporal_mismatch_rejected():
|
||||
with pytest.raises(ValueError,match='Video 2'):node.temporal_layout(l,r,None,2)
|
||||
l,r=example(6)
|
||||
with pytest.raises(ValueError,match='identical temporal'):node.temporal_layout(l,r,None)
|
||||
with pytest.raises(ValueError,match='Add Guide'):node.temporal_layout(l,r,[{}])
|
||||
|
||||
def test_add_guide_history_keyframes_keep_reference_alignment():
|
||||
# The previous shot's last frames as an Add Guide history (cond rows after the text) must not shift the reference.
|
||||
l,refs=example();plain,src0,tgt0=node.temporal_layout(l,refs,None)
|
||||
kfs=[dict(resolved_frame_index=0,latent=torch.zeros(1,4,2,8,8))]
|
||||
k=CORE.PackedLayout(3,7,8,8,2,keyframes=kfs,refs=refs)
|
||||
aligned,src,tgt=node.temporal_layout(k,refs,kfs)
|
||||
cond=[s for s in k.segments if s[2]=='cond'];assert cond
|
||||
a,b,_=cond[0];n=b-a
|
||||
assert src.sum()==src0.sum() and tgt.sum()==tgt0.sum()
|
||||
assert not src[a:b].any() and not tgt[a:b].any()
|
||||
# Same rows as without history, shifted by the cond rows; same aligned positions for the reference and the target.
|
||||
assert torch.equal(src[b:],src0[a:]) and torch.equal(tgt[b:],tgt0[a:])
|
||||
torch.testing.assert_close(aligned.position_ids[src],plain.position_ids[src0],rtol=0,atol=1e-12)
|
||||
torch.testing.assert_close(aligned.position_ids[tgt],plain.position_ids[tgt0],rtol=0,atol=1e-12)
|
||||
# History rows sit on the target's first frames and are left untouched.
|
||||
torch.testing.assert_close(aligned.position_ids[a:b],k.position_ids[a:b],rtol=0,atol=0)
|
||||
assert aligned.position_ids[a,0]==aligned.position_ids[tgt][0,0]
|
||||
|
||||
def context():
|
||||
l,r=example();a,s,t=node.temporal_layout(l,r,None)
|
||||
|
||||
@@ -1,425 +0,0 @@
|
||||
/**
|
||||
* BFS Segment Picker: the canvas of the node. It shows the objects SAM 3 found in the image, colored; hover one to
|
||||
* see it, click to select it. Objects the automatic pass missed are added by text (a SAM 3 prompt), by points
|
||||
* (keep / exclude), segmented at once by /bfs/segpick/add, or painted with the brush (to grow the mark by hand).
|
||||
* ➖ Remove (points) and 🧽 Eraser (brush) make the same kinds of objects, cut out of the mark instead of added to it.
|
||||
*
|
||||
* All state lives in the hidden `selection` STRING widget (JSON: selected ids and the added entries), so a workflow
|
||||
* saves, reloads and shares the selection. The objects themselves come from the node's last run (onExecuted).
|
||||
*/
|
||||
import { app } from "../../scripts/app.js";
|
||||
import { api } from "../../scripts/api.js";
|
||||
import { wheelToCanvas } from "./bfs_wheel.js";
|
||||
import { createApp, h, ref, reactive, computed, watch, nextTick, onMounted, onBeforeUnmount } from "./vendor/vue.esm-browser.prod.mjs";
|
||||
|
||||
// same order as PALETTE in bfs_segment_picker.py, so the panel and the overlay output agree
|
||||
const PALETTE = [[230, 70, 60], [45, 125, 225], [50, 180, 90], [235, 205, 35], [205, 55, 180], [30, 200, 205],
|
||||
[240, 130, 35], [135, 75, 210], [120, 200, 60], [225, 95, 150], [80, 160, 230], [190, 150, 90]];
|
||||
const CUT = [25, 25, 25]; // remove / eraser objects: a dark hole, unlike any PALETTE color
|
||||
const rgb = (c, a = 1) => `rgba(${c[0]},${c[1]},${c[2]},${a})`;
|
||||
const DEFAULT_W = 760, DEFAULT_H = 820;
|
||||
|
||||
const fileUrl = f => api.apiURL(`/view?filename=${encodeURIComponent(f.filename)}&subfolder=${encodeURIComponent(f.subfolder || "")}&type=${f.type || "temp"}`);
|
||||
|
||||
function parseState(text) {
|
||||
let s = {};
|
||||
try { s = JSON.parse(text || "{}") || {}; } catch { s = {}; }
|
||||
return { selected: Array.isArray(s.selected) ? s.selected.map(String) : [], added: Array.isArray(s.added) ? s.added : [],
|
||||
next: Number(s.next) || (Array.isArray(s.added) ? s.added.length + 1 : 1) };
|
||||
}
|
||||
|
||||
const STYLE_ID = "bfs-segpick-style";
|
||||
function styles() {
|
||||
if (document.getElementById(STYLE_ID)) return;
|
||||
const css = document.createElement("style");
|
||||
css.id = STYLE_ID;
|
||||
css.textContent = `
|
||||
.bsp{position:absolute;inset:0;display:flex;flex-direction:column;gap:6px;padding:6px;box-sizing:border-box;font:12px system-ui,sans-serif;color:#ddd}
|
||||
.bsp .bar{display:flex;flex-wrap:wrap;gap:6px;align-items:center}
|
||||
.bsp button{background:#333;color:#ddd;border:1px solid #555;border-radius:4px;padding:3px 8px;cursor:pointer;font:inherit}
|
||||
.bsp button:hover:not(:disabled){background:#444}
|
||||
.bsp button:disabled{opacity:.45;cursor:default}
|
||||
.bsp button.pri{background:#2d5a8a;border-color:#3d6fa5;color:#fff}
|
||||
.bsp button.on{background:#7a3d8a;border-color:#9a55aa;color:#fff}
|
||||
.bsp input[type=text]{background:#222;color:#ddd;border:1px solid #555;border-radius:4px;padding:3px 6px;font:inherit;flex:1;min-width:120px}
|
||||
.bsp .stage{flex:1;min-height:200px;position:relative;display:flex;align-items:center;justify-content:center;background:#151515;border-radius:4px;overflow:hidden}
|
||||
.bsp .frame{position:relative}
|
||||
.bsp .frame img,.bsp .frame canvas{position:absolute;inset:0;width:100%;height:100%;user-select:none}
|
||||
.bsp .frame canvas{cursor:pointer}
|
||||
.bsp .frame.points canvas,.bsp .frame.remove canvas{cursor:crosshair}
|
||||
.bsp .frame.brush canvas,.bsp .frame.eraser canvas{cursor:none}
|
||||
.bsp .chip.cut{border-style:dashed}
|
||||
.bsp .frame.remove .pt.pos{background:#e74c3c}.bsp .frame.remove .pt.neg{background:#2ecc71}
|
||||
.bsp input[type=range]{width:90px}
|
||||
.bsp .pt{position:absolute;width:12px;height:12px;margin:-6px 0 0 -6px;border-radius:50%;border:2px solid #fff;pointer-events:auto;cursor:pointer;box-shadow:0 0 3px #000}
|
||||
.bsp .pt.pos{background:#2ecc71}.bsp .pt.neg{background:#e74c3c}
|
||||
.bsp .empty{color:#888;text-align:center;padding:20px;line-height:1.6}
|
||||
.bsp .tip{position:absolute;left:6px;top:6px;background:rgba(0,0,0,.75);padding:2px 6px;border-radius:3px;pointer-events:none}
|
||||
.bsp .list{display:flex;flex-wrap:wrap;gap:4px;max-height:96px;overflow:auto}
|
||||
.bsp .chip{display:flex;align-items:center;gap:4px;padding:2px 6px;border-radius:10px;border:1px solid #444;background:#222;cursor:pointer}
|
||||
.bsp .chip.sel{border-color:#fff;background:#3a3a3a}
|
||||
.bsp .chip .sw{width:10px;height:10px;border-radius:2px}
|
||||
.bsp .chip .x{color:#999;padding:0 2px}.bsp .chip .x:hover{color:#f66}
|
||||
.bsp .hint{color:#888}.bsp .err{color:#f77}
|
||||
`;
|
||||
document.head.appendChild(css);
|
||||
}
|
||||
|
||||
/** One loaded object: a colored canvas to draw and its mask alpha for hit tests, at the preview resolution. */
|
||||
function loadMask(o, color) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const img = new Image();
|
||||
img.crossOrigin = "anonymous";
|
||||
img.onload = () => {
|
||||
const w = img.naturalWidth, hh = img.naturalHeight;
|
||||
const c = document.createElement("canvas"); c.width = w; c.height = hh;
|
||||
const g = c.getContext("2d");
|
||||
g.drawImage(img, 0, 0);
|
||||
const alpha = g.getImageData(0, 0, w, hh).data;
|
||||
const hit = new Uint8Array(w * hh);
|
||||
for (let i = 0; i < hit.length; i++) hit[i] = alpha[i * 4 + 3] > 127 ? 1 : 0;
|
||||
g.globalCompositeOperation = "source-in";
|
||||
g.fillStyle = rgb(color); g.fillRect(0, 0, w, hh);
|
||||
// the outline: mask pixels with a background pixel within r, so it stays visible once scaled down
|
||||
const r = Math.max(1, Math.round(Math.max(w, hh) / 400));
|
||||
const e = document.createElement("canvas"); e.width = w; e.height = hh;
|
||||
const eg = e.getContext("2d"), px = eg.createImageData(w, hh), [cr, cg, cb] = color;
|
||||
for (let y = 0; y < hh; y++) for (let x = 0; x < w; x++) {
|
||||
if (!hit[y * w + x]) continue;
|
||||
let edge = false;
|
||||
for (let dy = -r; dy <= r && !edge; dy++) for (let dx = -r; dx <= r; dx++) {
|
||||
const xx = x + dx, yy = y + dy;
|
||||
if (xx < 0 || yy < 0 || xx >= w || yy >= hh || !hit[yy * w + xx]) { edge = true; break; }
|
||||
}
|
||||
if (edge) { const i = (y * w + x) * 4; px.data[i] = cr; px.data[i + 1] = cg; px.data[i + 2] = cb; px.data[i + 3] = 255; }
|
||||
}
|
||||
eg.putImageData(px, 0, 0);
|
||||
resolve({ canvas: c, edge: e, hit, w, h: hh });
|
||||
};
|
||||
img.onerror = reject;
|
||||
img.src = fileUrl(o.mask);
|
||||
});
|
||||
}
|
||||
|
||||
function Panel(io) {
|
||||
const state = reactive(parseState(io.get()));
|
||||
const data = ref(null); // {session, width, height, image, objects} from the last run
|
||||
const loaded = new Map(); // object id -> loadMask result (not reactive: canvases and big arrays)
|
||||
const ready = ref(0); // bumps when masks finish loading, to redraw
|
||||
const hover = ref(null);
|
||||
const mode = ref("select"); // "select" | "points" | "brush" | "remove" (points) | "eraser" (brush)
|
||||
const points = ref([]); // the point entry being built
|
||||
const strokes = ref([]); // the brush entry being built: [{r, pts: [[x, y], ...]}], relative to the image
|
||||
const brush = ref(.02); // brush radius, relative to the image width
|
||||
let painting = false, cursor = null;
|
||||
const text = ref("");
|
||||
const busy = ref(""); const error = ref("");
|
||||
const stageEl = ref(null); const canvasEl = ref(null);
|
||||
const fit = reactive({ w: 0, h: 0 });
|
||||
|
||||
const save = () => io.set(JSON.stringify({ selected: state.selected, added: state.added, next: state.next }));
|
||||
const objects = computed(() => data.value?.objects || []);
|
||||
const colorOf = id => {
|
||||
const i = Math.max(0, objects.value.findIndex(o => o.id === id));
|
||||
return objects.value[i]?.remove ? CUT : PALETTE[i % PALETTE.length];
|
||||
};
|
||||
const isSel = id => state.selected.includes(id);
|
||||
const pointing = () => mode.value === "points" || mode.value === "remove";
|
||||
const brushing = () => mode.value === "brush" || mode.value === "eraser";
|
||||
const cutting = () => mode.value === "remove" || mode.value === "eraser";
|
||||
function setMode(m) {
|
||||
if (m !== mode.value) { points.value = []; strokes.value = []; } // unfinished points / strokes mean something else in another mode
|
||||
mode.value = m; hover.value = null;
|
||||
}
|
||||
|
||||
async function loadAll() {
|
||||
const objs = objects.value;
|
||||
await Promise.all(objs.map(async (o, i) => {
|
||||
const key = `${o.id}|${o.mask.filename}`;
|
||||
if (loaded.get(o.id)?.key === key) return;
|
||||
try { loaded.set(o.id, { key, ...(await loadMask(o, o.remove ? CUT : PALETTE[i % PALETTE.length])) }); } catch { /* shown as missing */ }
|
||||
}));
|
||||
ready.value++;
|
||||
}
|
||||
|
||||
function relayout() {
|
||||
const el = stageEl.value, d = data.value;
|
||||
if (!el || !d) return;
|
||||
const W = el.clientWidth - 2, H = el.clientHeight - 2, s = Math.min(W / d.width, H / d.height);
|
||||
fit.w = Math.max(1, Math.floor(d.width * s)); fit.h = Math.max(1, Math.floor(d.height * s));
|
||||
}
|
||||
|
||||
function draw() {
|
||||
const c = canvasEl.value, d = data.value;
|
||||
if (!c || !d) return;
|
||||
const first = loaded.values().next().value;
|
||||
const W = first?.w || fit.w, H = first?.h || fit.h;
|
||||
if (c.width !== W || c.height !== H) { c.width = W; c.height = H; }
|
||||
const g = c.getContext("2d");
|
||||
g.clearRect(0, 0, W, H);
|
||||
// biggest first, so small objects stay visible on top of the ones around them
|
||||
const order = [...objects.value].sort((a, b) => b.area - a.area);
|
||||
// the image itself is never tinted: an object is an outline, filled only while hovered (light) or selected
|
||||
for (const o of order) {
|
||||
const m = loaded.get(o.id);
|
||||
if (!m) continue;
|
||||
const sel = isSel(o.id), hov = hover.value === o.id;
|
||||
if (sel || hov) { g.globalAlpha = sel ? (hov ? .6 : .45) : .3; g.drawImage(m.canvas, 0, 0, W, H); }
|
||||
if (mode.value !== "select" && !sel && !hov) continue;
|
||||
g.globalAlpha = sel || hov ? 1 : .7;
|
||||
g.drawImage(m.edge, 0, 0, W, H);
|
||||
}
|
||||
g.globalAlpha = 1;
|
||||
if (!brushing()) return;
|
||||
g.strokeStyle = g.fillStyle = cutting() ? "rgba(20,20,20,.6)" : "rgba(255,0,255,.55)"; g.lineCap = g.lineJoin = "round";
|
||||
for (const st of strokes.value) {
|
||||
const [x0, y0] = st.pts[0];
|
||||
g.beginPath(); g.arc(x0 * W, y0 * H, st.r * W, 0, 2 * Math.PI); g.fill();
|
||||
if (st.pts.length < 2) continue;
|
||||
g.lineWidth = 2 * st.r * W; g.beginPath(); g.moveTo(x0 * W, y0 * H);
|
||||
for (const [x, y] of st.pts.slice(1)) g.lineTo(x * W, y * H);
|
||||
g.stroke();
|
||||
}
|
||||
if (cursor) {
|
||||
g.lineWidth = Math.max(1, W / 500); g.strokeStyle = "#fff";
|
||||
g.beginPath(); g.arc(cursor.x * W, cursor.y * H, brush.value * W, 0, 2 * Math.PI); g.stroke();
|
||||
}
|
||||
}
|
||||
watch([ready, hover, mode, brush, () => strokes.value.length, () => [...state.selected], () => fit.w, () => fit.h], () => nextTick(draw));
|
||||
|
||||
function at(e) {
|
||||
const r = canvasEl.value.getBoundingClientRect();
|
||||
return { x: Math.min(1, Math.max(0, (e.clientX - r.left) / r.width)), y: Math.min(1, Math.max(0, (e.clientY - r.top) / r.height)) };
|
||||
}
|
||||
function objectAt(p) {
|
||||
let best = null;
|
||||
for (const o of objects.value) {
|
||||
const m = loaded.get(o.id);
|
||||
if (!m) continue;
|
||||
const i = Math.min(m.h - 1, Math.floor(p.y * m.h)) * m.w + Math.min(m.w - 1, Math.floor(p.x * m.w));
|
||||
if (m.hit[i] && (!best || o.area < best.area)) best = o; // the smallest one under the mouse: inner objects stay pickable
|
||||
}
|
||||
return best;
|
||||
}
|
||||
|
||||
function toggle(id) {
|
||||
state.selected = isSel(id) ? state.selected.filter(i => i !== id) : [...state.selected, id];
|
||||
save();
|
||||
}
|
||||
function onPointer(e) {
|
||||
if (!data.value) return;
|
||||
e.preventDefault(); e.stopPropagation();
|
||||
const p = at(e);
|
||||
if (pointing()) {
|
||||
points.value = [...points.value, { x: +p.x.toFixed(4), y: +p.y.toFixed(4), label: (e.button === 2 || e.shiftKey) ? 0 : 1 }];
|
||||
return;
|
||||
}
|
||||
if (brushing()) {
|
||||
canvasEl.value.setPointerCapture(e.pointerId);
|
||||
painting = true;
|
||||
strokes.value = [...strokes.value, { r: +brush.value.toFixed(4), pts: [[+p.x.toFixed(4), +p.y.toFixed(4)]] }];
|
||||
return;
|
||||
}
|
||||
const o = objectAt(p);
|
||||
if (o) toggle(o.id);
|
||||
}
|
||||
function onMove(e) {
|
||||
if (data.value && brushing()) {
|
||||
cursor = at(e);
|
||||
const pts = strokes.value.at(-1)?.pts, last = pts?.at(-1);
|
||||
// a point every ~quarter radius keeps the stroke smooth without bloating the saved JSON
|
||||
if (painting && Math.hypot(cursor.x - last[0], cursor.y - last[1]) > brush.value / 4) pts.push([+cursor.x.toFixed(4), +cursor.y.toFixed(4)]);
|
||||
draw();
|
||||
return;
|
||||
}
|
||||
if (!data.value || pointing()) { hover.value = null; return; }
|
||||
hover.value = objectAt(at(e))?.id ?? null;
|
||||
}
|
||||
|
||||
async function add(entry) {
|
||||
if (!data.value) { error.value = "Run the node first (▶ Segment)."; return; }
|
||||
busy.value = entry.text ? `Finding “${entry.text}”…` : entry.strokes ? "Adding the painted area…" : "Segmenting the points…"; error.value = "";
|
||||
try {
|
||||
const r = await api.fetchApi("/bfs/segpick/add", { method: "POST", headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ session: data.value.session, entry }) });
|
||||
const j = await r.json();
|
||||
if (!r.ok || j.error) throw new Error(j.error || r.statusText);
|
||||
if (!j.objects.length) { error.value = entry.text ? `Nothing found for “${entry.text}”: try other words or a lower threshold.` : "The points gave an empty mask."; return; }
|
||||
state.added = [...state.added.filter(a => a.id !== entry.id), entry];
|
||||
state.next = Math.max(state.next, parseInt(entry.id.slice(1)) + 1);
|
||||
data.value = { ...data.value, objects: [...data.value.objects.filter(o => o.entry !== entry.id), ...j.objects] };
|
||||
state.selected = [...state.selected, ...j.objects.map(o => o.id).filter(id => !isSel(id))]; // what you add is what you want
|
||||
save();
|
||||
await loadAll();
|
||||
} catch (err) {
|
||||
error.value = String(err.message || err);
|
||||
} finally { busy.value = ""; }
|
||||
}
|
||||
const addText = () => { const t = text.value.trim(); if (t) { add({ id: `t${state.next}`, text: t }); text.value = ""; } };
|
||||
const addPoints = async () => {
|
||||
if (!points.value.some(p => p.label)) return;
|
||||
await add(cutting() ? { id: `r${state.next}`, remove: true, points: points.value } : { id: `p${state.next}`, points: points.value });
|
||||
if (!error.value) setMode("select");
|
||||
};
|
||||
const addBrush = async () => {
|
||||
if (!strokes.value.length) return;
|
||||
const stroke = { image: data.value.image_key, strokes: strokes.value };
|
||||
await add(cutting() ? { id: `e${state.next}`, remove: true, ...stroke } : { id: `b${state.next}`, ...stroke });
|
||||
if (!error.value) setMode("select");
|
||||
};
|
||||
function removeEntry(entryId) {
|
||||
state.added = state.added.filter(a => a.id !== entryId);
|
||||
const gone = objects.value.filter(o => o.entry === entryId).map(o => o.id);
|
||||
state.selected = state.selected.filter(i => !gone.includes(i));
|
||||
if (data.value) data.value = { ...data.value, objects: data.value.objects.filter(o => o.entry !== entryId) };
|
||||
gone.forEach(i => loaded.delete(i));
|
||||
save(); ready.value++;
|
||||
}
|
||||
|
||||
async function runNode() {
|
||||
error.value = "";
|
||||
try {
|
||||
const p = await app.graphToPrompt();
|
||||
await api.queuePrompt(0, p, { partialExecutionTargets: [String(io.node.id)] });
|
||||
busy.value = "Segmenting…";
|
||||
} catch (err) { error.value = String(err.message || err); }
|
||||
}
|
||||
|
||||
io.expose({
|
||||
executed: d => {
|
||||
busy.value = "";
|
||||
if (!d) return;
|
||||
if (data.value?.session !== d.session) loaded.clear();
|
||||
if (data.value && data.value.image_key !== d.image_key) strokes.value = [];
|
||||
// brush entries belong to the image they were painted on: another image drops them (text and points segment again)
|
||||
const stale = state.added.filter(a => a.strokes && a.image && a.image !== d.image_key).map(a => a.id);
|
||||
if (stale.length) { state.added = state.added.filter(a => !stale.includes(a.id)); save(); }
|
||||
data.value = d;
|
||||
// ids of objects that no longer exist (another image, other concepts) leave the selection
|
||||
const ids = new Set(d.objects.map(o => o.id));
|
||||
const keep = state.selected.filter(i => ids.has(i));
|
||||
if (keep.length !== state.selected.length) { state.selected = keep; save(); }
|
||||
nextTick(() => { relayout(); loadAll(); });
|
||||
},
|
||||
reload: () => { Object.assign(state, parseState(io.get())); ready.value++; },
|
||||
});
|
||||
|
||||
let ro;
|
||||
const done = () => { busy.value = ""; }; // a failed or fully cached run sends no output to this node
|
||||
onMounted(() => {
|
||||
ro = new ResizeObserver(() => relayout()); ro.observe(stageEl.value);
|
||||
api.addEventListener("execution_error", done); api.addEventListener("execution_success", done);
|
||||
});
|
||||
onBeforeUnmount(() => {
|
||||
ro?.disconnect();
|
||||
api.removeEventListener("execution_error", done); api.removeEventListener("execution_success", done);
|
||||
});
|
||||
watch(data, () => nextTick(relayout));
|
||||
|
||||
return () => {
|
||||
const d = data.value, objs = objects.value;
|
||||
const hovered = objs.find(o => o.id === hover.value);
|
||||
return h("div", { class: "bsp", onKeydown: e => e.stopPropagation() }, [
|
||||
h("div", { class: "bar" }, [
|
||||
h("button", { class: "pri", onClick: runNode, title: "Run this node only (and what feeds it) to segment the image" }, d ? "↻ Segment again" : "▶ Segment"),
|
||||
h("button", { class: mode.value === "select" ? "on" : "", onClick: () => setMode("select") }, "👆 Select"),
|
||||
h("button", { class: mode.value === "points" ? "on" : "", disabled: !d, onClick: () => setMode("points"),
|
||||
title: "Place points on something the automatic pass missed" }, "📍 Points"),
|
||||
h("button", { class: mode.value === "brush" ? "on" : "", disabled: !d, onClick: () => setMode("brush"),
|
||||
title: "Paint extra area into the mark by hand (e.g. hair SAM left out)" }, "🖌 Brush"),
|
||||
h("button", { class: mode.value === "remove" ? "on" : "", disabled: !d, onClick: () => setMode("remove"),
|
||||
title: "Place points on something to cut out of the mark (SAM segments it)" }, "➖ Remove"),
|
||||
h("button", { class: mode.value === "eraser" ? "on" : "", disabled: !d, onClick: () => setMode("eraser"),
|
||||
title: "Paint what to cut out of the mark by hand" }, "🧽 Eraser"),
|
||||
h("span", { class: "hint" }, d ? `${objs.length} objects · ${state.selected.length} selected` : ""),
|
||||
h("span", { style: "flex:1" }),
|
||||
h("button", { disabled: !objs.length, onClick: () => { state.selected = objs.map(o => o.id); save(); } }, "All"),
|
||||
h("button", { disabled: !state.selected.length, onClick: () => { state.selected = []; save(); } }, "None"),
|
||||
]),
|
||||
h("div", { class: "bar" }, [
|
||||
h("input", { type: "text", value: text.value, placeholder: "Find by text (SAM 3 prompt), e.g. red umbrella, the man on the left",
|
||||
disabled: !d, onInput: e => { text.value = e.target.value; }, onKeydown: e => { e.stopPropagation(); if (e.key === "Enter") addText(); } }),
|
||||
h("button", { disabled: !d || !text.value.trim() || !!busy.value, onClick: addText }, "🔎 Find"),
|
||||
pointing() ? [
|
||||
h("button", { class: "pri", disabled: !points.value.some(p => p.label) || !!busy.value, onClick: addPoints },
|
||||
`${cutting() ? "✓ Remove this" : "✓ Create object"} (${points.value.length})`),
|
||||
h("button", { disabled: !points.value.length, onClick: () => { points.value = []; } }, "Clear points"),
|
||||
] : null,
|
||||
brushing() ? [
|
||||
h("input", { type: "range", min: .003, max: .1, step: .001, value: brush.value, title: "Brush size",
|
||||
onInput: e => { brush.value = +e.target.value; } }),
|
||||
h("button", { class: "pri", disabled: !strokes.value.length || !!busy.value, onClick: addBrush },
|
||||
`${cutting() ? "✓ Erase painted area" : "✓ Add painted area"} (${strokes.value.length})`),
|
||||
h("button", { disabled: !strokes.value.length, onClick: () => { strokes.value = strokes.value.slice(0, -1); } }, "Undo"),
|
||||
h("button", { disabled: !strokes.value.length, onClick: () => { strokes.value = []; } }, "Clear"),
|
||||
] : null,
|
||||
]),
|
||||
h("div", { class: "stage", ref: stageEl, onContextmenu: e => e.preventDefault() }, d ? [
|
||||
h("div", { class: ["frame", mode.value !== "select" && mode.value], style: `width:${fit.w}px;height:${fit.h}px` }, [
|
||||
h("img", { src: fileUrl(d.image), draggable: false }),
|
||||
h("canvas", { ref: canvasEl, onPointerdown: onPointer, onPointermove: onMove, onPointerup: () => { painting = false; },
|
||||
onPointerleave: () => { hover.value = null; cursor = null; if (!painting) draw(); } }),
|
||||
...points.value.map((p, k) => h("div", { class: ["pt", p.label ? "pos" : "neg"], style: `left:${p.x * 100}%;top:${p.y * 100}%`,
|
||||
title: "click to remove", onPointerdown: e => { e.stopPropagation(); points.value = points.value.filter((_, j) => j !== k); } })),
|
||||
]),
|
||||
hovered ? h("div", { class: "tip" }, `${hovered.id} · ${hovered.label}${hovered.score != null ? ` · ${hovered.score}` : ""}`) : null,
|
||||
mode.value === "points" ? h("div", { class: "tip" }, "click = keep · right-click or shift+click = exclude · click a point to remove it") : null,
|
||||
mode.value === "remove" ? h("div", { class: "tip" }, "click on what to cut out of the mark · right-click or shift+click = keep it in · then ✓ Remove this") : null,
|
||||
mode.value === "brush" ? h("div", { class: "tip" }, "paint what the mark should also cover · then ✓ Add painted area") : null,
|
||||
mode.value === "eraser" ? h("div", { class: "tip" }, "paint what to cut out of the mark · then ✓ Erase painted area") : null,
|
||||
] : h("div", { class: "empty" }, ["Connect an image and press ▶ Segment (or run the workflow).", h("br"),
|
||||
"Then click objects to select them; add what was missed by text or 📍 points."])),
|
||||
busy.value ? h("div", { class: "hint" }, busy.value) : null,
|
||||
error.value ? h("div", { class: "err" }, error.value) : null,
|
||||
objs.length ? h("div", { class: "list" }, objs.map(o => h("div", {
|
||||
class: ["chip", isSel(o.id) && "sel", o.remove && "cut"], title: `${o.label}${o.score != null ? ` · score ${o.score}` : ""} · ${(o.area * 100).toFixed(1)}% of the image`,
|
||||
onClick: () => toggle(o.id), onPointerenter: () => { hover.value = o.id; }, onPointerleave: () => { hover.value = null; },
|
||||
}, [
|
||||
h("span", { class: "sw", style: `background:${rgb(colorOf(o.id))}` }),
|
||||
`${o.remove ? "➖ " : ""}${o.id} ${o.label}`,
|
||||
o.entry ? h("span", { class: "x", title: "remove this added object", onClick: e => { e.stopPropagation(); removeEntry(o.entry); } }, "×") : null,
|
||||
]))) : null,
|
||||
d ? h("div", { class: "hint" }, "The outputs follow the selection on the next run.") : null,
|
||||
]);
|
||||
};
|
||||
}
|
||||
|
||||
app.registerExtension({
|
||||
name: "BFSNodes.SegmentPicker",
|
||||
async nodeCreated(node) {
|
||||
if (node.comfyClass !== "BFSSegmentPicker") return;
|
||||
styles();
|
||||
const widget = node.widgets?.find(w => w.name === "selection");
|
||||
if (widget) { // the JSON stays a widget (saved with the workflow) but is never shown
|
||||
widget.type = "hidden"; widget.hidden = true; widget.computeSize = () => [0, -4];
|
||||
widget.options = { ...(widget.options || {}), hidden: true, serialize: true };
|
||||
widget.serializeValue = () => widget.value;
|
||||
}
|
||||
const host = document.createElement("div");
|
||||
host.style.cssText = "position:relative;width:100%;height:100%;min-width:420px;min-height:460px";
|
||||
// hideInPanel: the side panel's legacy renderer writes its own width into the widget, shrinking the canvas in the node
|
||||
node.addDOMWidget("bfs_segment_picker", "div", host, { serialize: false, hideOnZoom: false, hideInPanel: true });
|
||||
wheelToCanvas(host);
|
||||
const handle = {};
|
||||
createApp({
|
||||
setup: () => Panel({
|
||||
node,
|
||||
get: () => widget?.value ?? "{}",
|
||||
set: v => { if (widget) widget.value = v; node.graph?.setDirtyCanvas(true); },
|
||||
expose: o => Object.assign(handle, o),
|
||||
}),
|
||||
}).mount(host);
|
||||
const onExecuted = node.onExecuted;
|
||||
node.onExecuted = function (output) {
|
||||
onExecuted?.apply(this, arguments);
|
||||
handle.executed?.(output?.bfs_segpick?.[0]);
|
||||
};
|
||||
const onConfigure = node.onConfigure;
|
||||
node.onConfigure = function () {
|
||||
const r = onConfigure?.apply(this, arguments);
|
||||
setTimeout(() => handle.reload?.(), 0);
|
||||
return r;
|
||||
};
|
||||
node.size = [Math.max(node.size[0], DEFAULT_W), Math.max(node.size[1], DEFAULT_H)];
|
||||
},
|
||||
});
|
||||
@@ -9,8 +9,6 @@
|
||||
* whatever you do in the player (dragging the divider, pausing, scrubbing) until ■ stop. Both draw the two videos into
|
||||
* a canvas and record it in the browser, in real time. The browser's file (WebM, or a fragmented MP4 many apps won't
|
||||
* play) is then converted by ComfyUI to the chosen format: MP4 (H.264), a small MP4 (under 10 MB) or a GIF.
|
||||
* 🖼 PNG downloads the frame on screen as a picture, drawn the same way. With a single frame (two images) the node
|
||||
* also sends lossless PNGs of guide / target / difference: ⬇ save PNG draws from those, at the compared size.
|
||||
*/
|
||||
import { app } from "../../scripts/app.js";
|
||||
import { api } from "../../scripts/api.js";
|
||||
@@ -111,7 +109,6 @@ function buildPlayer(host) {
|
||||
selFormat.title = "Format of what save / rec download. MP4: H.264 + AAC, plays and previews anywhere. MP4 small: "
|
||||
+ "the same, sized to stay under 10 MB. GIF: no sound, under 10 MB. browser file: the recording as it is, no conversion.";
|
||||
const btnRec = el("button", { title: "Record what you do in the player until stop, then download it", onclick: () => (rec.mr ? stopRec() : startRec()) }, "● rec");
|
||||
const btnShot = el("button", { title: "Download the frame on screen as a PNG, as shown (layout, divider, labels, refs)", onclick: () => snapshot() }, "🖼 PNG");
|
||||
const chkLabels = el("input", { type: "checkbox" });
|
||||
chkLabels.checked = true;
|
||||
chkLabels.addEventListener("change", () => { st.labels = chkLabels.checked; });
|
||||
@@ -134,7 +131,7 @@ function buildPlayer(host) {
|
||||
stage,
|
||||
refBox,
|
||||
el("div", { class: "row" }, [btnPrev, btnPlay, btnNext, scrub, btnMute, info]),
|
||||
el("div", { class: "row" }, [btnSave, btnShot, btnRec, selFormat, el("label", { title: "Draw the names on the saved / recorded video" }, [chkLabels, "labels"]), lblRefs]),
|
||||
el("div", { class: "row" }, [btnSave, btnRec, selFormat, el("label", { title: "Draw the names on the saved / recorded video" }, [chkLabels, "labels"]), lblRefs]),
|
||||
strip,
|
||||
notes,
|
||||
]));
|
||||
@@ -270,9 +267,6 @@ function buildPlayer(host) {
|
||||
}
|
||||
|
||||
const last = new Map(); // video -> its last whole frame, drawn while it seeks (a seeking video draws blank)
|
||||
const stills = {}; // source -> loaded lossless PNG (single-frame comparisons only)
|
||||
const single = () => st.data?.frames === 1;
|
||||
const still = s => (stills[s]?.complete && stills[s].naturalWidth ? stills[s] : null);
|
||||
function pic(v) {
|
||||
if (v.readyState < 2 || v.seeking) return last.get(v) || v;
|
||||
let c = last.get(v);
|
||||
@@ -312,8 +306,8 @@ function buildPlayer(host) {
|
||||
|
||||
function drawVideos(ctx, area) {
|
||||
const cw = ctx.canvas.width, ch = area ?? ctx.canvas.height, m = st.mode, p = st.pos;
|
||||
const B = still(st.bottom) || pic(vBot), T = still(st.top) || pic(vTop);
|
||||
const aw = B.naturalWidth || vBot.videoWidth || st.data.width, ah = B.naturalHeight || vBot.videoHeight || st.data.height;
|
||||
const B = pic(vBot), T = pic(vTop);
|
||||
const aw = vBot.videoWidth || st.data.width, ah = vBot.videoHeight || st.data.height;
|
||||
const name = s => s === "guide" ? (st.data?.labels?.[0] || "guide") : s === "target" ? (st.data?.labels?.[1] || "target") : "difference";
|
||||
ctx.fillStyle = "#000"; ctx.fillRect(0, 0, cw, ch);
|
||||
if (m === "side") { // bottom left, top right, as on the stage
|
||||
@@ -384,29 +378,6 @@ function buildPlayer(host) {
|
||||
}
|
||||
}
|
||||
|
||||
// the canvas for save / rec / PNG: the videos at their size (side by side twice as wide), the reference strip below
|
||||
function sheet(aw, ah) {
|
||||
const canvas = el("canvas");
|
||||
canvas.width = (st.mode === "side" ? 2 * aw + Math.round(2 * aw * 0.004) : aw) & ~1;
|
||||
canvas.videoHeight = ah & ~1;
|
||||
canvas.height = canvas.videoHeight + stripHeight(canvas.videoHeight);
|
||||
return canvas;
|
||||
}
|
||||
|
||||
function snapshot() { // the frame on screen -> PNG (a single frame draws from the lossless stills)
|
||||
if (!st.data) return;
|
||||
if (st.playing) toggle();
|
||||
const s = still(st.bottom);
|
||||
const canvas = sheet(s?.naturalWidth || vBot.videoWidth || st.data.width, s?.naturalHeight || vBot.videoHeight || st.data.height);
|
||||
compose(canvas.getContext("2d"));
|
||||
canvas.toBlob(blob => {
|
||||
if (!blob) { alert("BFS Video Compare: could not make the PNG."); return; }
|
||||
const href = URL.createObjectURL(blob);
|
||||
download(href, `${stem()}${single() ? "" : `_f${st.frame}`}.png`);
|
||||
setTimeout(() => URL.revokeObjectURL(href), 60000);
|
||||
}, "image/png");
|
||||
}
|
||||
|
||||
function buttons() {
|
||||
if (rec.busy) { // uploading / converting the finished recording
|
||||
btnRec.classList.remove("rec");
|
||||
@@ -420,17 +391,18 @@ function buildPlayer(host) {
|
||||
btnRec.classList.toggle("rec", busy);
|
||||
btnRec.textContent = busy ? (rec.full ? "■ stop" : `■ stop ${((performance.now() - rec.t0) / 1000).toFixed(0)}s`) : "● rec";
|
||||
btnSave.disabled = busy;
|
||||
btnSave.textContent = busy && rec.full ? `⬇ ${Math.round(100 * (vBot.currentTime / (vBot.duration || 1)))}%` : single() ? "⬇ save PNG" : "⬇ save";
|
||||
btnShot.disabled = busy;
|
||||
// one frame is a picture: save downloads it as PNG, a video recording has nothing to show
|
||||
for (const b of [btnRec, selFormat, btnShot]) b.style.display = single() ? "none" : "";
|
||||
btnSave.textContent = busy && rec.full ? `⬇ ${Math.round(100 * (vBot.currentTime / (vBot.duration || 1)))}%` : "⬇ save";
|
||||
}
|
||||
|
||||
function startRec(full = false) {
|
||||
if (!st.data || rec.mr || rec.busy) return false;
|
||||
if (!window.MediaRecorder || !HTMLCanvasElement.prototype.captureStream) { alert("This browser can't record a canvas (MediaRecorder)."); return false; }
|
||||
const aw = vBot.videoWidth || st.data.width, ah = vBot.videoHeight || st.data.height;
|
||||
const side = st.mode === "side", canvas = el("canvas");
|
||||
// the size of the layout it starts in (side by side is twice as wide); a later mode change is fitted into it
|
||||
const canvas = sheet(vBot.videoWidth || st.data.width, vBot.videoHeight || st.data.height);
|
||||
canvas.width = (side ? 2 * aw + Math.round(2 * aw * 0.004) : aw) & ~1;
|
||||
canvas.videoHeight = ah & ~1; // the videos; the reference strip goes below them
|
||||
canvas.height = canvas.videoHeight + stripHeight(canvas.videoHeight);
|
||||
const ctx = canvas.getContext("2d");
|
||||
compose(ctx);
|
||||
const stream = canvas.captureStream(Math.max(30, Math.round(fps())));
|
||||
@@ -465,7 +437,6 @@ function buildPlayer(host) {
|
||||
|
||||
async function saveFull() { // once from the first frame to the last, then stop and download
|
||||
if (!st.data || rec.mr || rec.busy) return;
|
||||
if (single()) { snapshot(); return; }
|
||||
if (st.playing) toggle();
|
||||
for (const v of videos()) v.loop = false;
|
||||
const seeked = v => new Promise(res => { const t = setTimeout(res, 2000); v.addEventListener("seeked", () => { clearTimeout(t); res(); }, { once: true }); });
|
||||
@@ -487,9 +458,6 @@ function buildPlayer(host) {
|
||||
lblRefs.style.display = d.refs?.length ? "flex" : "none";
|
||||
showRefs();
|
||||
st.data = d;
|
||||
for (const k of Object.keys(stills)) delete stills[k];
|
||||
for (const [k, f] of Object.entries(d.stills || {})) { const im = new Image(); im.crossOrigin = "anonymous"; im.src = url(f); stills[k] = im; }
|
||||
buttons();
|
||||
notes.textContent = (d.notes || []).join("\n");
|
||||
empty.style.display = "none";
|
||||
scrub.max = String(Math.max(0, d.frames - 1));
|
||||
|
||||
Reference in New Issue
Block a user