Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2e4d197314 | ||
|
|
1ff7c1c9e1 | ||
|
|
5a4191f56c | ||
|
|
3ce2eb1ea0 | ||
|
|
27872290fe |
@@ -142,3 +142,84 @@ Sidecar and phase have CPU geometry/rotary parity tests, including mixed image,
|
||||
video and audio, but no full pretrained H3 quality validation yet. They distinguish
|
||||
sources; they do not guarantee the same face. See the
|
||||
[training configuration guide](https://github.com/alisson-anjos/ai-toolkit/blob/minimax-h3-latent-guides/H3_REFERENCE_ROPE.md).
|
||||
|
||||
|
||||
## Video context in CLIP with aligned guide latents
|
||||
|
||||
Select **aligned guide + CLIP video context** in **BFS Shot H3 Conditioning**
|
||||
(or its Simple variant). Qwen receives only the CURRENT SHOT as `<Video 1>` at
|
||||
2 fps with real timestamps, using the same shot interval passed to Add Guide.
|
||||
It does not receive the full source video on every shot. The original shot frames
|
||||
provide visual context even when source blur modifies the aligned latent guide.
|
||||
Images retain their `<Picture 1>` through `<Picture N>` order and native image
|
||||
reference latents. Planner and the new standalone node allow up to 100 pictures
|
||||
(the Comfy autogrow capacity); available model context and memory can limit this.
|
||||
The original H3 node keeps its own unchanged input limits.
|
||||
|
||||
The shot itself enters the diffusion model only as aligned `minimax_keyframes`
|
||||
through Add Guide; the new mode does not create a native `minimax_refs` video block
|
||||
for it. Its Qwen embeddings still travel in `positive`. Explicitly enabling the
|
||||
separate native masked-region video reference adds that as `<Video 2>`.
|
||||
|
||||
For a native workflow, use **MiniMax H3 Reference to Video — Guide 0 CLIP Context
|
||||
(BFS)** in place of **MiniMax H3 Reference to Video**:
|
||||
|
||||
1. Connect your source guide to **guide_video_0**, and pictures to `ref_images`.
|
||||
`0` identifies the input's special role; its prompt token is `<Video 1>` because
|
||||
H3 numbers video tokens from 1. Only this guide is CLIP-only.
|
||||
2. Optional `ref_videos` remain ACTUAL native video references, receiving both Qwen
|
||||
encoding and native reference latents. With guide 0 connected they are
|
||||
`<Video 2>`, `<Video 3>`, `<Video 4>`; without it native numbering starts at 1.
|
||||
Up to 3 actual reference videos are available, as in the original node.
|
||||
3. Connect the returned `positive` and target `latent` to **MiniMax H3 Add Guide**,
|
||||
with the SAME guide frames, VAE and `frame_idx = 0`.
|
||||
4. Feed Add Guide's `positive` and the target `latent` to your sampler.
|
||||
|
||||
Video inputs use the native 24-fps convention, are trimmed to the requested target
|
||||
length, and round down to the native 17*k+5 frame grid. There is no separate 15s
|
||||
hard cap in this helper; practical context/memory and checkpoint training limits
|
||||
still apply. Native reference soundtracks remain paired to their `ref_video_N`
|
||||
input, not to guide 0. Supply guide audio through Add Guide.
|
||||
The helper resizes only the guide frames sampled for Qwen and never VAE-encodes
|
||||
that guide as a native video reference. Images, actual reference videos and audio
|
||||
retain the native implementation. Target latent starts empty; Add Guide supplies
|
||||
aligned motion. The BFS mode performs the context and aligned-guide steps
|
||||
internally; Shot Join only assembles/composites the generated outputs.
|
||||
|
||||
Example: `<Subject 1> comes from <Picture 3> and follows the blond performer in
|
||||
the yellow shirt in <Video 1>. <Subject 2> comes from <Picture 1> and <Picture 2>
|
||||
and follows the orange-haired performer in the white shirt in <Video 1>. Keep
|
||||
these assignments through crossings and occlusions.` A picture need not mean a
|
||||
separate person; multiple images can define one identity, and there can be N
|
||||
assigned subjects/reference assets within the available context.
|
||||
|
||||
Existing modes and defaults are unchanged. Qwen encoding cost and text context
|
||||
length increase. Checkpoints trained with `guide_latent_only` did not see the
|
||||
source video in Qwen: this is an experimental inference option, not a demonstrated
|
||||
identity-binding improvement. GPU and visual validation remain for the user's
|
||||
local workflow.
|
||||
|
||||
|
||||
### Guide sampling rate and console evidence
|
||||
|
||||
`clip_video_fps` is available on both BFS conditioning variants and the Guide 0
|
||||
node. Default is 2 fps; range is 0.5–24 fps, capped to the source frame rate
|
||||
without duplicating frames. It changes guide sampling in Qwen only, not the
|
||||
aligned guide, output fps, or native reference-video sampling. Try 4 or 8 fps
|
||||
against 2 fps with the same shot, prompt, references, seed and sampler settings.
|
||||
Native H3 timestamps are retained; higher sampling rates are an experiment, not
|
||||
a guarantee of stronger identity binding.
|
||||
|
||||
Console messages with `[BFS H3 CLIP context]` report the input/used frame count,
|
||||
requested/effective fps, sent RGB frames, dimensions and first/last timestamps.
|
||||
A second message confirms that the tokenizer actually emitted H3 video vision
|
||||
blocks; an incompatible CLIP fails explicitly rather than silently ignoring the
|
||||
video. The final message appears only after Qwen encoding returns, with elapsed
|
||||
time and conditioning shapes. Qwen can pad an odd sample count to pairs.
|
||||
These logs confirm the conditioning path ran, not visual quality.
|
||||
|
||||
Compare `aligned guide (Add Guide)` against `aligned guide + CLIP video context`
|
||||
with all other settings fixed to assess whether video context changes the result.
|
||||
Use a fresh sampler execution (Planner cached renders may otherwise return old
|
||||
results); the CLIP-context logs only appear when conditioning actually executes.
|
||||
Compare 2/4/8 fps separately to isolate the sampling-rate effect.
|
||||
|
||||
@@ -139,7 +139,10 @@ workflow once per shot, and join the results back with the original timing and s
|
||||
every node downstream runs once per shot.
|
||||
- **BFS Shot Unpack** — guide frames, references, prompt and length of one shot, for any model.
|
||||
- **BFS Shot H3 Conditioning** — MiniMax H3 conditioning for one shot with the native nodes
|
||||
(aligned guide and/or native reference video).
|
||||
(aligned guide, native reference video, or aligned guide + CLIP video context).
|
||||
- **MiniMax H3 Reference to Video — Guide 0 CLIP Context (BFS)** — lets Qwen see the video
|
||||
guide at 2 fps without native guide video-reference latents; use Add Guide separately for motion.
|
||||
See [H3 guide modes](MINIMAX_H3_GUIDES.md#video-context-in-clip-with-aligned-guide-latents).
|
||||
- **BFS Shot Join** — concatenates, trims to the true lengths, cross-fades soft joins, returns audio. With `comparison` on, its `guide` output is the original video frame for frame.
|
||||
- **BFS Video Compare** — one player for two videos (Join's `guide` and `images`): wipe ↔ / ↕, overlay or side by side, any of guide / target / difference on top, plus a per-pixel difference map and the difference per frame (click to jump).
|
||||
|
||||
|
||||
+18
@@ -119,7 +119,23 @@ except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] Video Compare node not loaded: {_e!r}")
|
||||
VCMP_NODE_CLASS_MAPPINGS, VCMP_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
|
||||
try:
|
||||
from .minimax_h3_clip_video import NODE_CLASS_MAPPINGS as H3CLIP_NODE_CLASS_MAPPINGS
|
||||
from .minimax_h3_clip_video import NODE_DISPLAY_NAME_MAPPINGS as H3CLIP_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] MiniMax H3 CLIP video context not loaded: {_e!r}")
|
||||
H3CLIP_NODE_CLASS_MAPPINGS, H3CLIP_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
|
||||
try:
|
||||
from .minimax_h3_embedding_bridge import NODE_CLASS_MAPPINGS as H3BRIDGE_NODE_CLASS_MAPPINGS
|
||||
from .minimax_h3_embedding_bridge import NODE_DISPLAY_NAME_MAPPINGS as H3BRIDGE_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] Experimental H3 embedding bridge not loaded: {_e!r}")
|
||||
H3BRIDGE_NODE_CLASS_MAPPINGS, H3BRIDGE_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
**H3BRIDGE_NODE_CLASS_MAPPINGS,
|
||||
**H3CLIP_NODE_CLASS_MAPPINGS,
|
||||
**H3GUIDE_NODE_CLASS_MAPPINGS,
|
||||
**SHOT_NODE_CLASS_MAPPINGS,
|
||||
**PANEL_NODE_CLASS_MAPPINGS,
|
||||
@@ -149,6 +165,8 @@ NODE_CLASS_MAPPINGS = {
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
**H3BRIDGE_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**H3CLIP_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**H3GUIDE_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**SHOT_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**PANEL_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
|
||||
+95
-71
@@ -504,7 +504,7 @@ def _load_plan(plan_json: str) -> dict:
|
||||
return out
|
||||
|
||||
|
||||
MAX_REFS = 12 # reference pictures per shot, <Picture 1..12>; the continuity and setting pictures come after them
|
||||
MAX_REFS = 100 # matches Comfy autogrow input capacity; continuity/setting pictures follow the shot refs
|
||||
|
||||
|
||||
def ref_list(refs, ref="", ref2="") -> list[str]:
|
||||
@@ -1002,19 +1002,10 @@ SAM3_URL = "https://huggingface.co/Comfy-Org/sam3.1/resolve/main/checkpoints/sam
|
||||
_SAM3: dict[str, Any] = {}
|
||||
_MASK_CACHE: dict[tuple, dict] = {}
|
||||
# global mask settings (the planner's Mask card); a shot only says what to segment (text / points / key frame)
|
||||
DEFAULT_MASK = {"text": "", "points": [], "key": 0, "keys": [], "threshold": 0.5, "max_objects": 4, "invert": False,
|
||||
DEFAULT_MASK = {"text": "", "points": [], "key": 0, "threshold": 0.5, "max_objects": 4, "invert": False,
|
||||
"fill_holes": True, "temporal_expand": 2, "blockify": 0, "padding": 0.15, "expand": 16,
|
||||
"feather": 12, "paste": "box"}
|
||||
SHOT_MASK_KEYS = ("text", "points", "key", "keys")
|
||||
|
||||
|
||||
def point_keys(spec: dict) -> list[tuple[int, list]]:
|
||||
"""The shot's key frames with their points, in frame order: spec['keys'] ([{key, points}], 1.116), else the single
|
||||
key frame of older plans (points + key)."""
|
||||
keys = {int(k.get("key") or 0): list(k.get("points") or []) for k in spec.get("keys") or [] if k.get("points")}
|
||||
if not keys and spec.get("points"):
|
||||
keys = {int(spec.get("key") or 0): list(spec["points"])}
|
||||
return sorted(keys.items())
|
||||
SHOT_MASK_KEYS = ("text", "points", "key")
|
||||
|
||||
|
||||
def _sam3_path() -> str:
|
||||
@@ -1083,7 +1074,7 @@ def plan_spec(p: dict, m: dict | None, path: str = "") -> dict:
|
||||
|
||||
|
||||
def spec_has_mask(spec: dict) -> bool:
|
||||
return bool(spec.get("ext") or point_keys(spec) or str(spec.get("text") or "").strip())
|
||||
return bool(spec.get("ext") or spec.get("points") or str(spec.get("text") or "").strip())
|
||||
|
||||
|
||||
def ext_masks(ext: dict, analysis: dict, start: int, length: int, w: int, h: int) -> torch.Tensor:
|
||||
@@ -1216,34 +1207,28 @@ def _track(model, imgs: torch.Tensor, init: torch.Tensor | None, cond, spec: dic
|
||||
|
||||
|
||||
def segment_frames(imgs: torch.Tensor, spec: dict) -> torch.Tensor:
|
||||
"""[N,H,W,3] -> [N,H,W] masks with SAM 3, from the text or from the points of the key frames: each key frame's
|
||||
points select the subject there and it is tracked from that frame up to the next key frame (before the first one,
|
||||
backward from it), so a later key frame corrects the mask from its frame on."""
|
||||
"""[N,H,W,3] -> [N,H,W] masks with SAM 3: tracked from the points on the key frame, or from the text."""
|
||||
from comfy_extras.nodes_sam3 import SAM3_Detect
|
||||
model, clip = _sam3()
|
||||
text = str(spec.get("text") or "").strip()
|
||||
cond = clip.encode_from_tokens_scheduled(clip.tokenize(text)) if text else None
|
||||
pts = spec.get("points") or []
|
||||
N, H, W = imgs.shape[:3]
|
||||
keys = [(max(0, min(N - 1, k)), pts) for k, pts in point_keys(spec)]
|
||||
if not keys:
|
||||
if not pts:
|
||||
if cond is None:
|
||||
raise ValueError("the shot's mask needs a text prompt or points")
|
||||
return _track(model, imgs, None, cond, spec)
|
||||
|
||||
def detect(k, pts):
|
||||
pos = [{"x": p["x"] * W, "y": p["y"] * H} for p in pts if p.get("label", 1)]
|
||||
neg = [{"x": p["x"] * W, "y": p["y"] * H} for p in pts if not p.get("label", 1)]
|
||||
return SAM3_Detect.execute(model=model, image=imgs[k:k + 1], positive_coords=json.dumps(pos),
|
||||
negative_coords=json.dumps(neg), threshold=float(spec["threshold"]),
|
||||
refine_iterations=2).args[0][:1].float()
|
||||
out = torch.zeros(N, H, W)
|
||||
for i, (k, pts) in enumerate(keys):
|
||||
seed = detect(k, pts)
|
||||
if i == 0 and k > 0: # first key frame -> start
|
||||
out[:k + 1] = _track(model, imgs[:k + 1].flip(0), seed, None, spec).flip(0)
|
||||
end = keys[i + 1][0] if i + 1 < len(keys) else N # this key frame -> the next one
|
||||
out[k:end] = _track(model, imgs[k:end], seed, None, spec)
|
||||
return out
|
||||
key = max(0, min(N - 1, int(spec.get("key") or 0)))
|
||||
pos = [{"x": p["x"] * W, "y": p["y"] * H} for p in pts if p.get("label", 1)]
|
||||
neg = [{"x": p["x"] * W, "y": p["y"] * H} for p in pts if not p.get("label", 1)]
|
||||
first = SAM3_Detect.execute(model=model, image=imgs[key:key + 1], positive_coords=json.dumps(pos),
|
||||
negative_coords=json.dumps(neg), threshold=float(spec["threshold"]),
|
||||
refine_iterations=2).args[0][:1].float()
|
||||
fwd = _track(model, imgs[key:], first, None, spec) # key frame -> end
|
||||
if key == 0:
|
||||
return fwd
|
||||
back = _track(model, imgs[:key + 1].flip(0), first, None, spec).flip(0) # key frame -> start
|
||||
return torch.cat([back[:-1], fwd], 0)
|
||||
|
||||
|
||||
def _mask_key(path: str, analysis: dict, start: int, length: int, spec: dict) -> tuple:
|
||||
@@ -1259,7 +1244,7 @@ def _mask_key(path: str, analysis: dict, start: int, length: int, spec: dict) ->
|
||||
else:
|
||||
e = None
|
||||
return (path, os.path.getmtime(path), float(analysis["fps"]), int(start), int(length),
|
||||
json.dumps({"keys": point_keys(spec), **{k: spec[k] for k in ("text", "threshold", "max_objects")}}, sort_keys=True), e)
|
||||
json.dumps({k: spec[k] for k in ("text", "points", "key", "threshold", "max_objects")}, sort_keys=True), e)
|
||||
|
||||
|
||||
def shot_mask(path: str, analysis: dict, start: int, length: int, spec: dict) -> dict:
|
||||
@@ -1283,8 +1268,8 @@ def shot_mask(path: str, analysis: dict, start: int, length: int, spec: dict) ->
|
||||
frames = _read_frames(path, src[idx], (w, h), stage="Reading the shot")
|
||||
imgs = torch.from_numpy(np.stack(frames).astype(np.float32) / 255.0)
|
||||
_status(f"Segmenting {len(frames)} frames with SAM 3", force=True)
|
||||
keys = point_keys(spec)
|
||||
what = (", ".join(f"{len(p)} points on frame {k}" for k, p in keys)) if keys else f"'{spec.get('text', '')}'"
|
||||
what = f"{len(spec.get('points') or [])} points on frame {int(spec.get('key') or 0)}" if spec.get("points") \
|
||||
else f"'{spec.get('text', '')}'"
|
||||
print(f"[BFS Shot] SAM 3: segmenting {what} on {len(frames)} frames ({w}x{h}, timeline {start}-{start + length - 1})…",
|
||||
flush=True)
|
||||
t0 = time.time()
|
||||
@@ -1495,15 +1480,13 @@ def uncrop(result: torch.Tensor, shot: dict, edge_hardness: float = 0.0, match_c
|
||||
|
||||
def mask_preview(path: str, analysis: dict, start: int, length: int, spec: dict, count: int = 6,
|
||||
color: str = "#ff283c", opacity: float = 0.55) -> dict:
|
||||
"""`count` frames of the shot (0 = every frame, smaller, for the points window's player) with the selected region
|
||||
tinted and the crop box, as data URLs."""
|
||||
"""A few frames of the shot with the selected region tinted and the crop box, as data URLs."""
|
||||
import cv2
|
||||
r = shot_mask(path, analysis, start, length, spec)
|
||||
masks, (w, h), box = r["masks"], r["size"], r["box"]
|
||||
src = _timeline(analysis["n_src"], analysis["fps_src"], float(analysis["fps"]))
|
||||
picks = list(range(length)) if count <= 0 else sorted(set(int(round(i)) for i in np.linspace(0, length - 1, min(count, length))))
|
||||
picks = sorted(set(int(round(i)) for i in np.linspace(0, length - 1, min(count, length))))
|
||||
frames = _read_frames(path, src[np.clip(np.array(picks) + start, 0, analysis["n"] - 1)], (w, h))
|
||||
small = 384 / max(w, h) if count <= 0 and max(w, h) > 384 else 1.0 # every frame: ~10 KB each
|
||||
try:
|
||||
tint = np.array([int(color[i:i + 2], 16) for i in (1, 3, 5)], dtype=np.float32)
|
||||
alpha = min(0.8, max(0.05, float(opacity)))
|
||||
@@ -1517,9 +1500,7 @@ def mask_preview(path: str, analysis: dict, start: int, length: int, spec: dict,
|
||||
if box:
|
||||
x0, y0, x1, y1 = box_px(box, w, h)
|
||||
cv2.rectangle(img, (x0, y0), (x1 - 1, y1 - 1), (255, 210, 90), 2)
|
||||
if small < 1.0:
|
||||
img = cv2.resize(img, (int(w * small) // 2 * 2, int(h * small) // 2 * 2), interpolation=cv2.INTER_AREA)
|
||||
ok, buf = cv2.imencode(".jpg", cv2.cvtColor(img, cv2.COLOR_RGB2BGR), [cv2.IMWRITE_JPEG_QUALITY, 80 if small == 1.0 else 70])
|
||||
ok, buf = cv2.imencode(".jpg", cv2.cvtColor(img, cv2.COLOR_RGB2BGR), [cv2.IMWRITE_JPEG_QUALITY, 80])
|
||||
out.append({"f": start + i, "src": "data:image/jpeg;base64," + base64.b64encode(buf.tobytes()).decode()})
|
||||
cover = float(masks.float().mean()) if masks.numel() else 0.0
|
||||
return {"box": box, "frames": out, "coverage": cover, "empty": box is None}
|
||||
@@ -1538,8 +1519,7 @@ TARGET_Q = ("Describe the main subject in this picture (a person, an animal, a c
|
||||
def target_crop(path: str, analysis: dict, start: int, length: int, spec: dict, pad: float = 0.12):
|
||||
"""The selected subject (the shot's SAM 3 points / text, on its key frame) cropped from that frame, RGB uint8."""
|
||||
src = _timeline(analysis["n_src"], analysis["fps_src"], float(analysis["fps"]))
|
||||
keys = point_keys(spec)
|
||||
key = max(0, min(length - 1, keys[0][0] if keys else int(spec.get("key") or length // 2)))
|
||||
key = max(0, min(length - 1, int(spec.get("key") or length // 2)))
|
||||
fr = _read_frames(path, src[np.clip(np.array([start + key]), 0, analysis["n"] - 1)], None)[0]
|
||||
if spec.get("ext"):
|
||||
sm = shot_mask(path, analysis, start, length, spec)["masks"]
|
||||
@@ -3363,7 +3343,7 @@ class BFSShotPlanner:
|
||||
# the ref_image / ref_image_2 inputs were removed in 1.113 (the references live in the panel, up to
|
||||
# 12); a workflow saved with them connected still links them, and they are ignored
|
||||
print("[BFS Shot Planner] ref_image / ref_image_2 are no longer planner inputs: the connected images "
|
||||
"are ignored. Set the references in the panel (Prompts & refs tab, up to 12).", flush=True)
|
||||
"are ignored. Set the references in the panel (Prompts & refs tab, up to 100).", flush=True)
|
||||
if vlm is not None:
|
||||
_VLM["clip"] = vlm # the panel's Analyse button uses it too
|
||||
vcfg = vlm_cfg(p.get("vlm_cfg"))
|
||||
@@ -3985,8 +3965,8 @@ def color_marks(plan: dict, m: dict) -> list[dict]:
|
||||
"key": m.get("color_guide_key2")}] if m.get("color_guide_text2") or m.get("color_guide_points2") else [])
|
||||
marks = [{"picture": max(1, int(m.get("color_guide_picture") or 1))}]
|
||||
marks += [{"picture": max(1, int(r.get("picture") or 2)), "text": str(r.get("text") or "").strip(),
|
||||
"points": list(r.get("points") or []), "key": int(r.get("key") or 0), "keys": list(r.get("keys") or [])}
|
||||
for r in extra if isinstance(r, dict) and (str(r.get("text") or "").strip() or r.get("points") or r.get("keys"))]
|
||||
"points": list(r.get("points") or []), "key": int(r.get("key") or 0)}
|
||||
for r in extra if isinstance(r, dict) and (str(r.get("text") or "").strip() or r.get("points"))]
|
||||
for x in marks:
|
||||
x["color"] = picture_color(plan, x["picture"])
|
||||
return marks
|
||||
@@ -4005,7 +3985,6 @@ def extra_mark_masks(shot: dict, log: str = "") -> list[tuple]:
|
||||
out = []
|
||||
for x in extra:
|
||||
spec = dict(DEFAULT_MASK, text=x.get("text") or "", points=x.get("points") or [], key=x.get("key") or 0,
|
||||
keys=x.get("keys") or [],
|
||||
max_objects=1)
|
||||
out.append((shot_mask(ms["path"], ms["analysis"], ms["start"], ms["length"], spec)["masks"], x))
|
||||
return out
|
||||
@@ -4175,7 +4154,7 @@ def inpaint_latent(latent: dict, positive, vae, frames: torch.Tensor, mask: torc
|
||||
class BFSShotH3Conditioning:
|
||||
"""Native MiniMax H3 conditioning for one shot: references, prompt and the shot as a guide."""
|
||||
|
||||
GUIDE_MODES = ["aligned guide (Add Guide)", "native reference video (Video 1)", "both", "none"]
|
||||
GUIDE_MODES = ["aligned guide (Add Guide)", "native reference video (Video 1)", "both", "none", "aligned guide + CLIP video context"]
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
@@ -4187,7 +4166,8 @@ class BFSShotH3Conditioning:
|
||||
"guide_mode": (cls.GUIDE_MODES, {"default": cls.GUIDE_MODES[0], "tooltip":
|
||||
"aligned guide: the shot's frames sit on the generated frames (MiniMax H3 Add Guide), "
|
||||
"frame by frame. native reference video: the shot enters as <Video 1> like the reference "
|
||||
"video input. both: the two together."}),
|
||||
"video input. both: the two together. aligned guide + CLIP video context: Qwen sees "
|
||||
"the shot at clip_video_fps, while its latents enter only as an aligned guide."}),
|
||||
"first_frame": (["none", "shot's first frame"], {"default": "none", "tooltip":
|
||||
"Anchor the shot's own first frame as an extra aligned image guide."}),
|
||||
"ref_image_size": (["match", "max"], {"default": "match"}),
|
||||
@@ -4254,6 +4234,10 @@ class BFSShotH3Conditioning:
|
||||
"mask_guide '+ reference video' only: size of that reference video (fraction of the generation "
|
||||
"size). The native reference channel keeps a smaller video as it is, so 1/2 costs ~1/4 of the "
|
||||
"tokens; it only has to show the pose and outline."}),
|
||||
"clip_video_fps": ("FLOAT", {"default": 2.0, "min": 0.5, "max": 24.0, "step": 0.5,
|
||||
"tooltip": "Only for aligned guide + CLIP video context. Qwen sampling rate for this shot; "
|
||||
"try 2, 4 or 8 fps. Higher values cost more context/encoding time, not output fps. "
|
||||
"Console confirms sent frames, video vision tokens and completed encoding."}),
|
||||
},
|
||||
}
|
||||
|
||||
@@ -4262,11 +4246,12 @@ class BFSShotH3Conditioning:
|
||||
FUNCTION = "condition"
|
||||
CATEGORY = "BFS/shot loop"
|
||||
DESCRIPTION = ("Build MiniMax H3 conditioning for one shot with the native nodes: Reference to Video "
|
||||
"(prompt, references, length) plus the shot as an aligned guide and/or reference video.")
|
||||
"(prompt, references, length) plus the shot as an aligned guide and/or reference video. "
|
||||
"CLIP video context lets Qwen see the shot without native shot reference latents.")
|
||||
|
||||
def condition(self, shot, clip, vae, guide_mode, first_frame, ref_image_size,
|
||||
audio_vae=None, with_audio=False, model=None, vlm=None, setting_ref="off", setting_mask=None, inpaint="off", mask_guide="off", mask_ref_size="1/2", mask_guide_look="grey blurred",
|
||||
source_blur="off", source_blur_strength=1.0, source_blur_segmenter="SAM 3.1", source_blur_effect="blur"):
|
||||
source_blur="off", source_blur_strength=1.0, source_blur_segmenter="SAM 3.1", source_blur_effect="blur", clip_video_fps=2.0):
|
||||
from comfy_extras.nodes_minimax_h3 import MiniMaxH3AddGuide, MiniMaxH3ReferenceToVideo
|
||||
# every picture of the shot, <Picture 1> first (chosen in the planner; use_ref_2 left in 1.113)
|
||||
refs = {f"ref_image_{k}": r for k, r in enumerate(shot_refs(shot)[:MAX_REFS])}
|
||||
@@ -4286,7 +4271,8 @@ class BFSShotH3Conditioning:
|
||||
refs[f"ref_image_{len(refs)}"] = prev # one more <Picture n>, after the shot's own
|
||||
audio = shot["audio"] if (with_audio and audio_vae is not None) else None
|
||||
native = guide_mode in (self.GUIDE_MODES[1], self.GUIDE_MODES[2])
|
||||
aligned = guide_mode in (self.GUIDE_MODES[0], self.GUIDE_MODES[2])
|
||||
clip_context = guide_mode == self.GUIDE_MODES[4]
|
||||
aligned = guide_mode in (self.GUIDE_MODES[0], self.GUIDE_MODES[2]) or clip_context
|
||||
text = shot["prompt"] or ""
|
||||
setting = setting_picture(shot, setting_ref, setting_mask) if setting_ref and setting_ref != "off" else None
|
||||
if setting is not None:
|
||||
@@ -4308,9 +4294,6 @@ class BFSShotH3Conditioning:
|
||||
text = add_setting(text, len(refs), "<Subject 1>" in text)
|
||||
text = fill_details_now(text, vlm, shot_refs(shot), f"[BFS Shot H3 Conditioning] shot {shot.get('index', 0) + 1}",
|
||||
shot.get("ref_names"))
|
||||
if len(refs) > 9: # H3's own node stops at 9; OpenJutsu-style LoRAs were trained with up to 9 pictures
|
||||
print(f"[BFS Shot H3 Conditioning] shot {shot.get('index', 0) + 1}: {len(refs)} reference pictures "
|
||||
"(more than the 9 H3 and its LoRAs usually see)", flush=True)
|
||||
kwargs = dict(clip=clip, prompt=text, width=shot["width"], height=shot["height"],
|
||||
length=shot["gen_length"], ref_image_size=ref_image_size, vae=vae,
|
||||
audio_vae=audio_vae, ref_images=refs or None)
|
||||
@@ -4362,10 +4345,15 @@ class BFSShotH3Conditioning:
|
||||
print(f"{gtag}: masked-region reference video at {guide_px.shape[2]}x{guide_px.shape[1]} ({mask_ref_size})",
|
||||
flush=True)
|
||||
vids[f"ref_video_{len(vids) + 1}"] = guide_px
|
||||
kwargs["prompt"] = add_mask_video(kwargs["prompt"], len(vids))
|
||||
print(f"{gtag}: the masked region is <Video {len(vids)}> (write {{mask_video}} in the prompt to place "
|
||||
kwargs["prompt"] = add_mask_video(kwargs["prompt"], len(vids) + int(clip_context))
|
||||
print(f"{gtag}: the masked region is <Video {len(vids) + int(clip_context)}> (write {{mask_video}} in the prompt to place "
|
||||
f"its tag, otherwise a sentence is added)", flush=True)
|
||||
kwargs["ref_videos"] = vids
|
||||
if clip_context:
|
||||
from .minimax_h3_clip_video import clip_with_video_context
|
||||
kwargs["clip"] = clip_with_video_context(clip, {"shot": shot["frames"]}, shot["gen_length"], source_fps=float(shot["fps"]), context_fps=clip_video_fps)
|
||||
print(f"{gtag}: shot is <Video 1> in Qwen/CLIP at {clip_video_fps:g} fps; motion uses Add Guide, "
|
||||
"without native shot reference latents", flush=True)
|
||||
positive, latent = MiniMaxH3ReferenceToVideo.execute(**kwargs).args[:2]
|
||||
if pg in (MASK_GUIDES[1], MASK_GUIDES[2]):
|
||||
positive = MiniMaxH3AddGuide.execute(positive=positive, latent=latent, frame_idx=0, vae=vae,
|
||||
@@ -4462,7 +4450,9 @@ class BFSShotH3ConditioningSimple:
|
||||
"vae": ("VAE",),
|
||||
"guide_mode": (cls.GUIDE_MODES, {"default": cls.GUIDE_MODES[0], "tooltip":
|
||||
"aligned guide: preserve the shot frame by frame with MiniMax H3 Add Guide. "
|
||||
"native reference video: provide it as <Video 1> context. both: use both paths."}),
|
||||
"native reference video: provide it as <Video 1> context. both: use both paths. "
|
||||
"aligned guide + CLIP video context: Qwen sees the shot at clip_video_fps without native "
|
||||
"shot reference latents; motion still uses Add Guide."}),
|
||||
"ref_image_size": (["match", "max"], {"default": "match"}),
|
||||
},
|
||||
"optional": {
|
||||
@@ -4485,6 +4475,10 @@ class BFSShotH3ConditioningSimple:
|
||||
"source_blur_effect": (BLUR_EFFECTS, {"default": BLUR_EFFECTS[0], "tooltip":
|
||||
"blur: no detail left. invert colours: keeps the shapes and the expression but not the skin tone "
|
||||
"or the colours."}),
|
||||
"clip_video_fps": ("FLOAT", {"default": 2.0, "min": 0.5, "max": 24.0, "step": 0.5,
|
||||
"tooltip": "Only for aligned guide + CLIP video context. Qwen sampling rate for this shot; "
|
||||
"try 2, 4 or 8 fps. Higher values cost more context/encoding time, not output fps. "
|
||||
"Console confirms sent frames, video vision tokens and completed encoding."}),
|
||||
},
|
||||
}
|
||||
|
||||
@@ -4497,12 +4491,13 @@ class BFSShotH3ConditioningSimple:
|
||||
|
||||
def condition(self, shot, clip, vae, guide_mode, ref_image_size,
|
||||
audio_vae=None, with_audio=False, vlm=None, inpaint=INPAINT_MODES[0],
|
||||
source_blur="off", source_blur_strength=1.0, source_blur_segmenter="SAM 3.1", source_blur_effect="blur"):
|
||||
source_blur="off", source_blur_strength=1.0, source_blur_segmenter="SAM 3.1", source_blur_effect="blur", clip_video_fps=2.0):
|
||||
positive, latent, _, prompt = BFSShotH3Conditioning().condition(
|
||||
shot, clip, vae, guide_mode, "none", ref_image_size,
|
||||
audio_vae=audio_vae, with_audio=with_audio, vlm=vlm, inpaint=inpaint,
|
||||
source_blur=source_blur, source_blur_strength=source_blur_strength,
|
||||
source_blur_segmenter=source_blur_segmenter, source_blur_effect=source_blur_effect)
|
||||
source_blur_segmenter=source_blur_segmenter, source_blur_effect=source_blur_effect,
|
||||
clip_video_fps=clip_video_fps)
|
||||
return positive, latent, prompt
|
||||
|
||||
|
||||
@@ -4723,6 +4718,7 @@ class BFSShotJoin:
|
||||
shot.get("shot_storage") or "lossless")
|
||||
st = run_state(rid)
|
||||
i = int(shot["index"])
|
||||
st.get("redo", {}).pop(str(i), None)
|
||||
if i not in st["done"]:
|
||||
st["done"].append(i)
|
||||
st["count"] = shot["count"]
|
||||
@@ -5023,15 +5019,16 @@ class BFSShotJoin:
|
||||
|
||||
|
||||
def cache_status(plan: dict, reset: bool = False) -> dict:
|
||||
"""The panel's view of the shot cache of a plan's video (as of the last time the planner ran)."""
|
||||
"""Read stored-shot status. Legacy reset requests only cancel pending redos; never delete files."""
|
||||
path = _input_path(plan["video"])
|
||||
rid = run_key(plan, path)
|
||||
d = run_dir(rid, create=False)
|
||||
if reset:
|
||||
import shutil
|
||||
shutil.rmtree(d, ignore_errors=True)
|
||||
if reset and os.path.isdir(d):
|
||||
st = run_state(rid)
|
||||
_cancel_redo(st, list(st.get("redo", {})))
|
||||
save_state(rid, st)
|
||||
if not os.path.isdir(d): # nothing stored for this video (no folder is made just to look)
|
||||
return {"run_id": rid, "folder": d, "count": 0, "done": 0, "ok": [], "stored": [], "mb": 0.0,
|
||||
return {"run_id": rid, "folder": d, "count": 0, "done": 0, "ok": [], "stored": [], "redo": [], "mb": 0.0,
|
||||
"temp": rid.startswith("tmp_")}
|
||||
st = run_state(rid)
|
||||
ok = shot_status(st, d) if st.get("cur") else []
|
||||
@@ -5039,15 +5036,41 @@ def cache_status(plan: dict, reset: bool = False) -> dict:
|
||||
stored = sorted(int(f[5:9]) for f in files if f.startswith("shot_") and f.endswith(".pt"))
|
||||
size = sum(os.path.getsize(os.path.join(d, f)) for f in files if f.startswith("shot_"))
|
||||
return {"run_id": rid, "folder": d, "count": len(ok) or st.get("count", 0), "done": sum(ok),
|
||||
"ok": ok, "stored": stored, "mb": round(size / 1e6, 1), "temp": rid.startswith("tmp_")}
|
||||
"ok": ok, "stored": stored, "redo": sorted(int(i) for i in st.get("redo", {})),
|
||||
"mb": round(size / 1e6, 1), "temp": rid.startswith("tmp_")}
|
||||
|
||||
|
||||
def redo_shots(plan: dict, shots: list[int], cascade: bool = True) -> dict:
|
||||
def _cancel_redo(st: dict, shots: list) -> None:
|
||||
for i in shots:
|
||||
key = str(int(i))
|
||||
saved = st.get("redo", {}).pop(key, None)
|
||||
if saved is None or saved.get("gen") != st.get("gen", {}).get(key):
|
||||
continue # a new take has already replaced this selection
|
||||
if saved["done"] and int(i) not in st.setdefault("done", []):
|
||||
st["done"].append(int(i))
|
||||
if saved.get("keep_next"):
|
||||
st.setdefault("keep_next", {})[key] = True
|
||||
else:
|
||||
st.get("keep_next", {}).pop(key, None)
|
||||
|
||||
|
||||
def redo_shots(plan: dict, shots: list[int], cascade: bool = True, cancel: bool = False) -> dict:
|
||||
"""Mark stored shots to run again on the next Queue. cascade: the shots that continue from them are redone too
|
||||
(automatically, once the new take exists); off: they keep their take (the join may show a seam there)."""
|
||||
(automatically, once the new take exists); off: they keep their take (the join may show a seam there).
|
||||
cancel restores the selection's previous state only while its stored take has not been replaced."""
|
||||
rid = run_key(plan, _input_path(plan["video"]))
|
||||
st = run_state(rid)
|
||||
if cancel:
|
||||
_cancel_redo(st, shots)
|
||||
save_state(rid, st)
|
||||
return cache_status(plan)
|
||||
for i in shots:
|
||||
key = str(int(i))
|
||||
st.setdefault("redo", {}).setdefault(key, {
|
||||
"done": int(i) in st.get("done", []),
|
||||
"gen": st.get("gen", {}).get(key),
|
||||
"keep_next": bool(st.get("keep_next", {}).get(key)),
|
||||
})
|
||||
if int(i) in st.get("done", []):
|
||||
st["done"].remove(int(i))
|
||||
if cascade:
|
||||
@@ -5829,7 +5852,8 @@ try:
|
||||
body = await request.json()
|
||||
try:
|
||||
return web.json_response(redo_shots(_load_plan(json.dumps(body.get("plan", {}))),
|
||||
[int(x) for x in body.get("shots", [])], bool(body.get("cascade", True))))
|
||||
[int(x) for x in body.get("shots", [])], bool(body.get("cascade", True)),
|
||||
bool(body.get("cancel", False))))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return web.json_response({"error": f"{type(exc).__name__}: {exc}"}, status=500)
|
||||
|
||||
|
||||
+45
-14
@@ -80,6 +80,28 @@ def save_refs(reference, out_dir: str, height: int) -> list:
|
||||
return names
|
||||
|
||||
|
||||
def video_batches(value, name):
|
||||
parts = value if isinstance(value, list) else [value]
|
||||
if not parts or any(not isinstance(x, torch.Tensor) or x.ndim != 4 or
|
||||
x.shape[0] < 1 or x.shape[-1] < 3 or min(x.shape[1:3]) < 1 for x in parts):
|
||||
raise ValueError(f"{name} must contain nonempty IMAGE batches")
|
||||
return parts
|
||||
|
||||
|
||||
def video_chunk(parts, start, end, h, w):
|
||||
"""Read a bounded frame interval across shot batches without concatenating full videos."""
|
||||
result, pos = [], 0
|
||||
for part in parts:
|
||||
count = int(part.shape[0])
|
||||
a, b = max(0, start - pos), min(count, end - pos)
|
||||
if a < b:
|
||||
result.append(_scale(part[a:b, ..., :3].float(), h, w))
|
||||
pos += count
|
||||
if pos >= end:
|
||||
break
|
||||
return result[0] if len(result) == 1 else torch.cat(result, 0)
|
||||
|
||||
|
||||
class BFSVideoCompare:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
@@ -110,8 +132,7 @@ class BFSVideoCompare:
|
||||
},
|
||||
}
|
||||
|
||||
# lists in: the reference can be the refs list of the planner / Unpack (pictures of different sizes); the other
|
||||
# inputs take their first item
|
||||
# IMAGE lists contain every shot, in order. Only scalar settings take their first item.
|
||||
INPUT_IS_LIST = True
|
||||
RETURN_TYPES = ("IMAGE", "STRING", "FLOAT")
|
||||
RETURN_NAMES = ("difference", "report", "mean_difference")
|
||||
@@ -127,18 +148,28 @@ class BFSVideoCompare:
|
||||
def compare(self, guide, target, fps=24.0, audio=None, gain=2.0, size="guide", preview_height=720,
|
||||
label_guide="guide", label_target="target", reference=None):
|
||||
one = lambda v: v[0] if isinstance(v, list) and v else v
|
||||
guide, target, fps, audio, gain, size, preview_height, label_guide, label_target = map(
|
||||
one, (guide, target, fps, audio, gain, size, preview_height, label_guide, label_target))
|
||||
fps, audio, gain, size, preview_height, label_guide, label_target = map(
|
||||
one, (fps, audio, gain, size, preview_height, label_guide, label_target))
|
||||
guide, target = video_batches(guide, "guide"), video_batches(target, "target")
|
||||
ng, nt = sum(x.shape[0] for x in guide), sum(x.shape[0] for x in target)
|
||||
if len(guide) > 1 or len(target) > 1:
|
||||
if ng != nt or (len(guide) == len(target) and any(g.shape[0] != t.shape[0] for g, t in zip(guide, target))):
|
||||
raise ValueError("Guide/target shot lists have different timing. Connect BFS Shot Join's guide and images outputs from the SAME Join; Unpack guide frames can include continuity heads and generation padding.")
|
||||
notes = []
|
||||
n = min(guide.shape[0], target.shape[0])
|
||||
if guide.shape[0] != target.shape[0]:
|
||||
notes.append(f"lengths differ (guide {guide.shape[0]}, target {target.shape[0]} frames): "
|
||||
n = min(ng, nt)
|
||||
if ng != nt:
|
||||
notes.append(f"lengths differ (guide {ng}, target {nt} frames): "
|
||||
f"compared the first {n}")
|
||||
ref = guide if size == "guide" else target
|
||||
H, W = int(ref.shape[1]), int(ref.shape[2])
|
||||
for name, v in (("guide", guide), ("target", target)):
|
||||
if tuple(v.shape[1:3]) != (H, W):
|
||||
notes.append(f"{name} scaled from {v.shape[2]}x{v.shape[1]} to {W}x{H}")
|
||||
H, W = int(ref[0].shape[1]), int(ref[0].shape[2])
|
||||
for name, parts in (("guide", guide), ("target", target)):
|
||||
if len(parts) > 1:
|
||||
notes.append(f"{name}: compared all {len(parts)} shot batches in input order")
|
||||
for dims in sorted(set(tuple(v.shape[1:3]) for v in parts)):
|
||||
if dims != (H, W):
|
||||
notes.append(f"{name} scaled from {dims[1]}x{dims[0]} to {W}x{H}")
|
||||
if ng != nt:
|
||||
notes.append("Timing may be misaligned. For Planner results, connect guide and images from the SAME BFS Shot Join.")
|
||||
ph = min(H, int(preview_height)) if preview_height and preview_height > 0 else H
|
||||
PH, PW = max(16, ph // 2 * 2), max(16, int(round(W * ph / H)) // 2 * 2)
|
||||
|
||||
@@ -151,8 +182,8 @@ class BFSVideoCompare:
|
||||
try:
|
||||
for a in range(0, n, CHUNK):
|
||||
e = min(a + CHUNK, n) # the longer video stops where the shorter one ends
|
||||
g = _scale(guide[a:e, ..., :3].float(), H, W)
|
||||
t = _scale(target[a:e, ..., :3].float(), H, W)
|
||||
g = video_chunk(guide, a, e, H, W)
|
||||
t = video_chunk(target, a, e, H, W)
|
||||
m, e, s = difference(g, t, gain)
|
||||
maps.append(m)
|
||||
mae.append(e)
|
||||
@@ -179,7 +210,7 @@ class BFSVideoCompare:
|
||||
ui = {"bfs_compare": [{**files, "fps": float(fps), "frames": n, "width": PW, "height": PH,
|
||||
"labels": [label_guide, label_target], "mae": [round(float(x), 2) for x in mae],
|
||||
"refs": refs,
|
||||
"report": report}]}
|
||||
"report": report, "notes": notes}]}
|
||||
return {"ui": ui, "result": (diff_map, report, mean)}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
# Experimental Qwen3.5 9B thinking states -> H3 conditioning
|
||||
|
||||
Branch: `experiment/h3-9b-thinking-embeddings`. No changes to main or the live
|
||||
OpenJutsu training run. This is a prepared research prototype, not a trained
|
||||
encoder replacement. GPU execution and real workflow validation are pending.
|
||||
|
||||
## What is actually tested
|
||||
|
||||
The full Qwen3.5-9B sees numbered pictures and chronological guide frames, plus
|
||||
an unchanged copy of the request. It generates with thinking off/on. A forward
|
||||
hook captures its last decoder-block activations BEFORE final normalization:
|
||||
one prefill bank and a separate bank of generated-token states. The sampled
|
||||
last token may not have been forwarded. Reasoning text is not rewritten into an
|
||||
H3 prompt, not saved, and not fed through the teacher. Token states themselves
|
||||
are the bridge input. A completed `</think>` is required in thinking mode;
|
||||
truncated thinking is not silently labelled successful.
|
||||
|
||||
A shared trainable bridge consumes these states. Position/modality queries
|
||||
produce a variable-length sequence with the original H3 teacher's exact layout,
|
||||
5120-wide unnormalized features and visual/text tags. The bridge never sees
|
||||
teacher feature values as INPUT. Only its reconstruction loss uses them.
|
||||
The 9B and H3 encoders stay frozen; H3 DiT/VAE are not loaded for calibration.
|
||||
Teacher is the existing converted H3 encoder (50 layers), so no 66-GB full
|
||||
teacher download is needed.
|
||||
|
||||
This prototype requires a cached teacher layout for each evaluation case. It is
|
||||
NOT yet a general online encoder, a speedup demonstration, or an identity/motion
|
||||
quality improvement. Distilling ordinary H3 embeddings may suppress useful
|
||||
thinking information. Embedding error alone cannot prove thinking benefits.
|
||||
|
||||
## What is ready and what is not
|
||||
|
||||
Prepared: disjoint input manifest, teacher/student cache scripts, a modality-
|
||||
balanced shared resampling bridge, train/validation/test reconstruction metrics,
|
||||
held-out conditioning exports, and a guarded Comfy cached-case embedding swap.
|
||||
Only syntax and CPU file preflight are checked. No tensor, GPU, generation or
|
||||
quality tests have been run, per the user's local testing preference. No bridge
|
||||
weights have been trained/exported. No new model download or environment upgrade
|
||||
is performed automatically. GPU entrypoints refuse to run while the dataset
|
||||
reviewer is active or GPU memory is occupied; there is no auto-start watcher.
|
||||
|
||||
Full local Qwen3.5-9B weights and a compatible Transformers environment are
|
||||
required. Existing Comfy caption-model support does not guarantee a matching
|
||||
Transformers model directory. Keep the experimental environment separate from
|
||||
Comfy/training; use current official Qwen requirements rather than upgrading
|
||||
the active environments. Generation uses a complete HF model, not GGUF and not
|
||||
the headless H3 encoder.
|
||||
|
||||
## CPU preparation (safe during dataset processing)
|
||||
|
||||
```bash
|
||||
cd /data/bfs-dev-vlm-reasoning/experiments/h3_embedding_bridge
|
||||
python make_manifest.py --output /data/reports/h3_9b_embedding_bridge/manifest.jsonl
|
||||
python preflight.py --manifest /data/reports/h3_9b_embedding_bridge/manifest.jsonl \
|
||||
--output /data/reports/h3_9b_embedding_bridge/preflight.json
|
||||
```
|
||||
|
||||
Default: 32 calibration-train, 8 validation and 8 held-out cases; no exact
|
||||
reference-asset hash reuse across these cases. Inputs are original training
|
||||
split captions/guides, capped at four pictures for the initial cost experiment.
|
||||
They are UNFILTERED calibration requests, NOT approved H3 training examples and
|
||||
NOT a clean identity-quality benchmark. No targets, original captions or
|
||||
production caches are modified. Their source captions may be inaccurate; this
|
||||
limits downstream identity conclusions. Reference-image hashes prevent exact
|
||||
asset overlap, not all appearances of the same real identity.
|
||||
|
||||
Each probe uses at most 73 native 24-fps frames (~3.04 s), source-time resampling,
|
||||
384-pixel width and 2-fps Qwen context. Teacher and student use the same media
|
||||
interval and picture order, with their respective native vision processors.
|
||||
This small probe cannot establish full-video identity stability.
|
||||
|
||||
## Deferred GPU stages (after the dataset reviewer exits)
|
||||
|
||||
Use absolute paths or set `PROBE`, `CALIB`, `H3PY`, `QWENPY` in your shell.
|
||||
`H3PY` is a compatible Comfy Python environment; `QWENPY` is an isolated
|
||||
Transformers environment supporting the complete Qwen3.5 model. Use an empty
|
||||
output directory. Models are loaded sequentially, not concurrently.
|
||||
|
||||
```bash
|
||||
PROBE=/data/bfs-dev-vlm-reasoning/experiments/h3_embedding_bridge
|
||||
CALIB=/data/reports/h3_9b_embedding_bridge
|
||||
H3PY=/data/comfyui/.venv/bin/python
|
||||
# Set QWENPY to your separate compatible environment, and QWEN_MODEL to local weights.
|
||||
PYTHONPATH=/tmp/caption-runtime-site "$H3PY" "$PROBE/cache_teacher.py" \
|
||||
--manifest "$CALIB/manifest.jsonl" --cache "$CALIB/cache"
|
||||
"$QWENPY" "$PROBE/cache_student.py" --model "$QWEN_MODEL" \
|
||||
--manifest "$CALIB/manifest.jsonl" --cache "$CALIB/cache" --max-new-tokens 512
|
||||
"$QWENPY" "$PROBE/train_bridge.py" --manifest "$CALIB/manifest.jsonl" \
|
||||
--cache "$CALIB/cache" --output "$CALIB/run_001" --steps 500
|
||||
```
|
||||
|
||||
Cache only complete, matching input keys. Teacher weights stay in their native
|
||||
conversion/quantization format. If the local Comfy Python cannot load that format,
|
||||
fix the isolated runtime first. The teacher stage has not been executed here.
|
||||
Student HF paths are local-only; partial thinking halts with a metadata record
|
||||
and can be retried with a larger token budget. The scripts do not resume training
|
||||
optimizer state; choose a new output directory for each trial.
|
||||
|
||||
Validation selects one shared bridge using both thinking modes. Test outputs
|
||||
contain `thinking_0`, `thinking_1`, and `thinking_ablated` (same thinking-on
|
||||
prefill, generated states zeroed, same state-bank quota). Never use test teacher
|
||||
features to fit the bridge. The per-channel output scale is estimated from the
|
||||
training split only. There is no video/identity/reward objective in this first
|
||||
calibration. Production use might require H3 adaptation, broader calibration,
|
||||
and a trained task-specific objective.
|
||||
|
||||
## Local Comfy experiment after calibration
|
||||
|
||||
Copy held-out exported `*.thinking_*.safetensors` and
|
||||
`*.thinking_ablated.safetensors` into `ComfyUI/models/h3_embedding_bridge/`.
|
||||
Do not select `bridge.safetensors` there: it contains weights, not conditioning.
|
||||
Restart Comfy with this experimental BFS branch. The node is:
|
||||
|
||||
**BFS H3 Experimental 9B Embedding Bridge (Cached Case)**
|
||||
|
||||
First build native H3 conditioning with the exact SAME encoder, prompt, ordered
|
||||
pictures, source interval/resampling, dimensions and 2-fps guide context used in
|
||||
that export. Feed its `positive` through the experimental node before the
|
||||
sampler; retain the normal latent/Add Guide/reference-VAE path. The node replaces
|
||||
only embeddings and retains existing metadata, native reference latents and
|
||||
aligned guide latents. A teacher-baseline comparison rejects mismatched cached
|
||||
cases/tags; do not raise tolerance to bypass wrong inputs. This baseline pass
|
||||
means the prototype does not save encoder compute.
|
||||
|
||||
Render four matched arms with identical seed, guide, references, LoRA, CFG,
|
||||
steps and sampler: native teacher, 9B thinking off, 9B thinking on, generated-state
|
||||
ablation. Judge wrong-person assignments, identity exchanges after crossings or
|
||||
occlusions, reference wardrobe fidelity and motion separately. A difference in
|
||||
output, or a lower embedding reconstruction error, does not establish a benefit.
|
||||
Use curated longer multi-person clips before changing the production encoder.
|
||||
|
||||
Official Qwen model/config: https://huggingface.co/Qwen/Qwen3.5-9B
|
||||
Original H3 encoder: https://huggingface.co/MiniMaxAI/MiniMax-H3/tree/main/FL2VA/text_encoder
|
||||
@@ -0,0 +1,78 @@
|
||||
"""Experimental 9B hidden-state -> H3 layer-50 bridge, not an encoder replacement yet."""
|
||||
import math
|
||||
from dataclasses import asdict, dataclass
|
||||
import torch
|
||||
from torch import nn
|
||||
from torch.nn import functional as F
|
||||
|
||||
|
||||
@dataclass
|
||||
class BridgeConfig:
|
||||
student_dim: int = 4096
|
||||
teacher_dim: int = 5120
|
||||
width: int = 512
|
||||
heads: int = 8
|
||||
source_slots: int = 512
|
||||
query_chunk: int = 256
|
||||
|
||||
|
||||
class HiddenStateBridge(nn.Module):
|
||||
"""Queries use only target positions/modality tags, NEVER teacher features.
|
||||
|
||||
Teacher layout is needed for this cached-case prototype. Arbitrary-input
|
||||
layout prediction is deliberately not claimed or implemented.
|
||||
"""
|
||||
def __init__(self, config=BridgeConfig()):
|
||||
super().__init__()
|
||||
self.config = config
|
||||
self.source = nn.Sequential(nn.LayerNorm(config.student_dim), nn.Linear(config.student_dim, config.width))
|
||||
self.modality = nn.Embedding(2, config.width)
|
||||
self.attention = nn.MultiheadAttention(config.width, config.heads, batch_first=True)
|
||||
self.norm = nn.LayerNorm(config.width)
|
||||
self.output = nn.Linear(config.width, config.teacher_dim)
|
||||
self.register_buffer('teacher_channel_scale', torch.ones(config.teacher_dim))
|
||||
|
||||
def position(self, length, device):
|
||||
# Normalized positions do not depend on teacher feature values.
|
||||
pos = torch.linspace(0, 1, length, device=device)[:, None]
|
||||
freq = torch.exp(torch.arange(0, self.config.width, 2, device=device) * (-math.log(10000) / self.config.width))
|
||||
angles = pos * 1000 * freq
|
||||
return torch.stack((angles.sin(), angles.cos()), -1).flatten(-2)
|
||||
|
||||
def memory(self, prefix, generated):
|
||||
# Separate quotas keep thinking states from being washed out by image tokens.
|
||||
banks = [self.source(prefix)]
|
||||
if generated.shape[0]:
|
||||
banks.append(self.source(generated))
|
||||
quota = max(1, self.config.source_slots // len(banks))
|
||||
pooled = [F.adaptive_avg_pool1d(x.T[None], min(quota, len(x))).transpose(1, 2) for x in banks]
|
||||
result = torch.cat(pooled, 1)
|
||||
return result + self.position(result.shape[1], result.device)[None].to(result.dtype)
|
||||
|
||||
def forward(self, prefix, generated, target_tags):
|
||||
assert prefix.ndim == generated.ndim == 2
|
||||
assert prefix.shape[-1] == generated.shape[-1] == self.config.student_dim
|
||||
assert target_tags.ndim == 1 and len(target_tags) > 0
|
||||
memory = self.memory(prefix, generated)
|
||||
queries = self.modality(target_tags.long()) + self.position(len(target_tags), target_tags.device).to(memory.dtype)
|
||||
output = []
|
||||
for start in range(0, len(queries), self.config.query_chunk):
|
||||
q = queries[None, start:start + self.config.query_chunk]
|
||||
x, _ = self.attention(q, memory, memory, need_weights=False)
|
||||
output.append(self.output(self.norm(x + q))[0])
|
||||
return torch.cat(output) * self.teacher_channel_scale
|
||||
|
||||
def config_dict(self):
|
||||
return asdict(self.config)
|
||||
|
||||
|
||||
def alignment_loss(prediction, teacher, tags):
|
||||
# Equal modality weighting prevents the longer visual prefix dominating text.
|
||||
scale = teacher.float().square().mean(0).sqrt().clamp_min(1e-3)
|
||||
losses = []
|
||||
for kind in (0, 1):
|
||||
mask = tags == kind
|
||||
if mask.any():
|
||||
p, t = prediction[mask].float(), teacher[mask].float()
|
||||
losses.append(((p - t) / scale).square().mean() + .1 * (1 - F.cosine_similarity(p, t).mean()))
|
||||
return torch.stack(losses).mean()
|
||||
@@ -0,0 +1,86 @@
|
||||
"""GPU stage 2. Capture actual 9B decoder activations, not rewritten prompts.
|
||||
|
||||
A forward hook retains one selected layer only, avoiding generate's all-layer
|
||||
hidden-state cache. No generated reasoning text is persisted or fed to H3.
|
||||
"""
|
||||
import argparse, json, sys
|
||||
from pathlib import Path
|
||||
from common import load_manifest, require_idle_gpu, visual_inputs
|
||||
|
||||
|
||||
def decoder_layer(model):
|
||||
for path in ('model.language_model.layers', 'language_model.layers', 'model.layers'):
|
||||
obj = model
|
||||
try:
|
||||
for name in path.split('.'):
|
||||
obj = getattr(obj, name)
|
||||
return obj[-1]
|
||||
except AttributeError:
|
||||
pass
|
||||
raise RuntimeError('Unsupported Transformers decoder layout; no hidden states captured')
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(); ap.add_argument('--manifest', required=True); ap.add_argument('--cache', required=True)
|
||||
ap.add_argument('--model', required=True, help='Local full Qwen3.5-9B Transformers directory; no automatic download')
|
||||
ap.add_argument('--max-new-tokens', type=int, default=512)
|
||||
args = ap.parse_args(); require_idle_gpu()
|
||||
if not Path(args.model).is_dir():
|
||||
raise RuntimeError('Full 9B weights are not present locally. Download in an isolated environment first.')
|
||||
import numpy as np
|
||||
import torch
|
||||
from safetensors.torch import save_file
|
||||
from transformers import AutoModelForImageTextToText, AutoProcessor
|
||||
processor = AutoProcessor.from_pretrained(args.model, local_files_only=True)
|
||||
model = AutoModelForImageTextToText.from_pretrained(args.model, dtype=torch.bfloat16, device_map='auto', local_files_only=True)
|
||||
model.eval().requires_grad_(False)
|
||||
layer = decoder_layer(model); cache = Path(args.cache); cache.mkdir(parents=True, exist_ok=True)
|
||||
for row in load_manifest(args.manifest):
|
||||
refs, frames, picks, info = visual_inputs(row)
|
||||
content = []
|
||||
for i, image in enumerate(refs, 1):
|
||||
content.extend([{'type': 'text', 'text': f'<Picture {i}>:'}, {'type': 'image', 'image': image}])
|
||||
content.append({'type': 'text', 'text': '<Video 1> chronological motion guide:'})
|
||||
for i in picks:
|
||||
content.extend([{'type': 'text', 'text': f'<{i / 24:.3f} seconds>:'}, {'type': 'image', 'image': frames[i]}])
|
||||
content.append({'type': 'text', 'text': 'Analyze the requested reference-to-performer assignments across these guide moments. '
|
||||
'Keep reference identity separate from guide identity and motion. Do not invent unseen details. '
|
||||
'Conclude with a short mapping summary. Request: ' + row['prompt']})
|
||||
messages = [{'role': 'user', 'content': content}]
|
||||
for thinking in (False, True):
|
||||
dest = cache / f'{row["id"]}.student_{int(thinking)}.safetensors'
|
||||
if dest.exists():
|
||||
continue
|
||||
inputs = processor.apply_chat_template(messages, tokenize=True, add_generation_prompt=True,
|
||||
return_dict=True, return_tensors='pt', enable_thinking=thinking).to(model.device)
|
||||
captured = []
|
||||
def capture(_module, _inputs, output):
|
||||
x = output[0] if isinstance(output, tuple) else output
|
||||
assert x.ndim == 3 and x.shape[0] == 1 and x.shape[-1] == 4096
|
||||
captured.append(x[0].detach().cpu().to(torch.bfloat16).contiguous())
|
||||
handle = layer.register_forward_hook(capture)
|
||||
try:
|
||||
with torch.inference_mode():
|
||||
generated = model.generate(**inputs, max_new_tokens=args.max_new_tokens, do_sample=False,
|
||||
use_cache=True, return_dict_in_generate=False)
|
||||
finally:
|
||||
handle.remove()
|
||||
assert captured, 'No decoder activations captured'
|
||||
prefix = captured[0]; tail = torch.cat(captured[1:]) if len(captured) > 1 else prefix[:0]
|
||||
text = processor.tokenizer.decode(generated[0, inputs['input_ids'].shape[1]:], skip_special_tokens=False)
|
||||
completed = not thinking or '</think>' in text
|
||||
meta = dict(input_key=row['input_key'], model=str(Path(args.model).resolve()), thinking=thinking,
|
||||
thinking_completed=completed, prefix_tokens=len(prefix), generated_state_tokens=len(tail),
|
||||
generated_tokens=generated.shape[1] - inputs['input_ids'].shape[1],
|
||||
layer='last decoder block before final norm', visual_inputs=info,
|
||||
extraction='Prefill activations plus generated-token activations; final sampled token may not be forwarded')
|
||||
if not completed:
|
||||
(cache / f'{row["id"]}.student_1.incomplete.json').write_text(json.dumps(meta, indent=2))
|
||||
raise RuntimeError('Thinking did not finish within token budget. Increase --max-new-tokens; no valid thinking cache saved.')
|
||||
save_file({'prefix': prefix, 'generated': tail}, str(dest), metadata={'experiment': json.dumps(meta)})
|
||||
print(json.dumps(dict(id=row['id'], thinking=thinking, completed=completed,
|
||||
prefix_tokens=len(prefix), generated_state_tokens=len(tail))), flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,49 @@
|
||||
"""GPU stage 1. Reuses the existing H3 encoder; does not load the H3 DiT/VAE."""
|
||||
import argparse, importlib.util, json, sys
|
||||
from pathlib import Path
|
||||
from common import load_manifest, require_idle_gpu, visual_inputs
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--manifest', required=True); ap.add_argument('--cache', required=True)
|
||||
ap.add_argument('--comfy', default='/data/comfyui')
|
||||
ap.add_argument('--teacher', default='/data/models/minimax-h3/text_encoders/qwen3vl_32b_minimax_h3_int8_convrot.safetensors')
|
||||
args = ap.parse_args(); require_idle_gpu(); sys.argv = [sys.argv[0]]
|
||||
import numpy as np
|
||||
import torch
|
||||
from safetensors.torch import save_file
|
||||
sys.path.insert(0, args.comfy)
|
||||
import comfy.options
|
||||
comfy.options.enable_args_parsing()
|
||||
import folder_paths, nodes
|
||||
teacher = Path(args.teacher); folder_paths.add_model_folder_path('text_encoders', str(teacher.parent))
|
||||
clip = nodes.CLIPLoader().load_clip(teacher.name, 'stable_diffusion')[0]
|
||||
path = Path(__file__).resolve().parents[2] / 'minimax_h3_clip_video.py'
|
||||
spec = importlib.util.spec_from_file_location('bridge_native_context', path)
|
||||
module = importlib.util.module_from_spec(spec); spec.loader.exec_module(module)
|
||||
cache = Path(args.cache); cache.mkdir(parents=True, exist_ok=True)
|
||||
for row in load_manifest(args.manifest):
|
||||
dest = cache / (row['id'] + '.teacher.safetensors')
|
||||
if dest.exists():
|
||||
continue
|
||||
refs, frames, picks, info = visual_inputs(row)
|
||||
images = {f'ref_image_{i}': torch.from_numpy(np.asarray(im).copy()).float()[None] / 255 for i, im in enumerate(refs)}
|
||||
guide = torch.from_numpy(np.stack([np.asarray(im) for im in frames]).copy()).float() / 255
|
||||
positive = module.BFSMiniMaxH3ReferenceToVideoGuideContext.execute(
|
||||
clip=clip, prompt=row['prompt'], width=info['width'], height=info['height'], length=info['length'],
|
||||
ref_image_size='match', ref_images=images, guide_video_0=guide, clip_video_fps=2,
|
||||
).args[0]
|
||||
assert len(positive) == 1
|
||||
embeds, extra = positive[0]; tags = extra['minimax_token_tags']
|
||||
embeds = embeds[0] if embeds.ndim == 3 else embeds
|
||||
tags = tags.reshape(-1)
|
||||
assert embeds.shape == (len(tags), 5120) and torch.isfinite(embeds).all()
|
||||
save_file({'teacher': embeds.cpu().to(torch.bfloat16).contiguous(), 'tags': tags.cpu().long().contiguous()},
|
||||
str(dest), metadata={'input_key': row['input_key'], 'teacher': str(teacher), 'visual_inputs': json.dumps(info)})
|
||||
(cache / (row['id'] + '.input.json')).write_text(json.dumps(row, indent=2, ensure_ascii=False))
|
||||
print(json.dumps(dict(id=row['id'], teacher_shape=list(embeds.shape), split=row['split'])), flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,53 @@
|
||||
import json, subprocess
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def require_idle_gpu():
|
||||
# No background polling/auto launch: fail before loading a model.
|
||||
active = subprocess.check_output(['ps', '-eo', 'args'], text=True)
|
||||
if any('audit_temporal_identity_binding.py' in line and not line.lstrip().startswith(('ps ', 'rg '))
|
||||
for line in active.splitlines()):
|
||||
raise RuntimeError('Dataset reviewer is active. No experiment GPU work was started.')
|
||||
usage = subprocess.check_output(['nvidia-smi', '--query-gpu=memory.used', '--format=csv,noheader,nounits'], text=True)
|
||||
if any(int(v) > 1024 for v in usage.split()):
|
||||
raise RuntimeError('GPU is occupied. No experiment GPU work was started.')
|
||||
|
||||
|
||||
def load_manifest(path):
|
||||
rows = [json.loads(line) for line in Path(path).read_text().splitlines() if line.strip()]
|
||||
assert rows and len({r['id'] for r in rows}) == len(rows)
|
||||
return rows
|
||||
|
||||
|
||||
def visual_inputs(row, length=73, width=384, context_fps=2.0):
|
||||
"""Identical 24-fps source interval and Picture order for teacher/student."""
|
||||
import cv2
|
||||
import numpy as np
|
||||
from PIL import Image, ImageOps
|
||||
cap = cv2.VideoCapture(row['guide']); fps = cap.get(cv2.CAP_PROP_FPS); count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
|
||||
if fps <= 0 or count <= 0:
|
||||
raise ValueError('Unreadable guide ' + row['guide'])
|
||||
n = min(length, int(count / fps * 24)); n -= (n - 5) % 17
|
||||
if n < 5:
|
||||
raise ValueError('Guide shorter than H3 grid')
|
||||
frames = []
|
||||
# Sequential decoding avoids failed random seeks on some original files.
|
||||
wanted = [min(count - 1, round(i / 24 * fps)) for i in range(n)]; cursor = -1; bgr = None
|
||||
for index in wanted:
|
||||
while cursor < index:
|
||||
ok, bgr = cap.read(); cursor += 1
|
||||
if not ok:
|
||||
raise ValueError(f'Guide decoding failed at source frame {cursor}')
|
||||
frame = Image.fromarray(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB))
|
||||
w = max(32, round(width / 32) * 32); h = max(32, round(frame.height / frame.width * w / 32) * 32)
|
||||
frames.append(frame.resize((w, h), Image.Resampling.LANCZOS))
|
||||
cap.release()
|
||||
picks = sorted({round(i * 24 / context_fps) for i in range(int((n - 1) / 24 * context_fps) + 1)})
|
||||
refs = []
|
||||
for path in row['references']:
|
||||
with Image.open(path) as raw:
|
||||
im = raw.convert('RGB'); scale = min(1.0, (w * h / (im.width * im.height)) ** .5)
|
||||
rw = max(32, round(im.width * scale / 32) * 32); rh = max(32, round(im.height * scale / 32) * 32)
|
||||
refs.append(im.resize((rw, rh), Image.Resampling.LANCZOS))
|
||||
return refs, frames, picks, dict(length=n, width=w, height=h, timestamps=[i / 24 for i in picks],
|
||||
context_fps=context_fps, input_key=row['input_key'])
|
||||
@@ -0,0 +1,51 @@
|
||||
"""CPU-only input selection. These calibration inputs are NOT approved H3 training samples."""
|
||||
import argparse, hashlib, json, random
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--output', required=True)
|
||||
parser.add_argument('--train', type=int, default=32)
|
||||
parser.add_argument('--validation', type=int, default=8)
|
||||
parser.add_argument('--test', type=int, default=8)
|
||||
args = parser.parse_args()
|
||||
rows = []
|
||||
for family, root in [('genjutsu', Path('/data/datasets/openjutsu/train_mt_multi')),
|
||||
('animate', Path('/data/datasets/animate_motion_transfer_v1/train'))]:
|
||||
for caption in sorted((root / 'targets').glob('*.txt')):
|
||||
refs = []
|
||||
for i in range(1, 17):
|
||||
files = sorted((root / f'refs_{i}').glob(caption.stem + '.*'))
|
||||
if files:
|
||||
refs.append(str(files[0]))
|
||||
if not 1 <= len(refs) <= 4:
|
||||
continue
|
||||
guide = root / ('guides_original' if family == 'animate' else 'guides') / (caption.stem + '.mp4')
|
||||
if not guide.is_file():
|
||||
continue
|
||||
rows.append(dict(id=family + '__' + caption.stem, guide=str(guide), references=refs,
|
||||
prompt=caption.read_text().strip(), original_caption_sha256=hashlib.sha256(caption.read_bytes()).hexdigest(),
|
||||
calibration_only=True, approved_for_h3_training=False))
|
||||
random.Random(42).shuffle(rows)
|
||||
selected = []; seen_assets = set()
|
||||
# No exact reference-asset reuse across selected cases/splits.
|
||||
for row in rows:
|
||||
hashes = {hashlib.sha256(Path(p).read_bytes()).hexdigest() for p in row['references']}
|
||||
if hashes & seen_assets:
|
||||
continue
|
||||
seen_assets.update(hashes); row['reference_sha256'] = sorted(hashes); selected.append(row)
|
||||
if len(selected) >= args.train + args.validation + args.test:
|
||||
break
|
||||
assert len(selected) == args.train + args.validation + args.test, 'Insufficient disjoint reference assets'
|
||||
for i, row in enumerate(selected):
|
||||
row['split'] = 'train' if i < args.train else 'validation' if i < args.train + args.validation else 'test'
|
||||
row['input_key'] = hashlib.sha256(json.dumps(row, sort_keys=True).encode()).hexdigest()
|
||||
out = Path(args.output); out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(''.join(json.dumps(r, ensure_ascii=False) + '\n' for r in selected))
|
||||
print(json.dumps(dict(output=str(out), total=len(selected), train=args.train, validation=args.validation, test=args.test,
|
||||
purpose='Embedding reconstruction only; unfiltered input captions, not a curated identity-quality benchmark.')))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,36 @@
|
||||
"""Read-only CPU preflight: does not import torch, download weights or launch GPU jobs."""
|
||||
import argparse, ast, json
|
||||
from pathlib import Path
|
||||
from common import load_manifest
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(); ap.add_argument('--manifest', required=True)
|
||||
ap.add_argument('--student', default='/data/models/Qwen3.5-9B')
|
||||
ap.add_argument('--output', required=True); args = ap.parse_args()
|
||||
rows = load_manifest(args.manifest)
|
||||
missing = []
|
||||
for row in rows:
|
||||
for path in [row['guide'], *row['references']]:
|
||||
if not Path(path).is_file():
|
||||
missing.append(path)
|
||||
for path in Path(__file__).parent.glob('*.py'):
|
||||
ast.parse(path.read_text(), filename=str(path))
|
||||
student = Path(args.student)
|
||||
ready = (student / 'config.json').exists() and any(student.glob('*.safetensors'))
|
||||
report = dict(phase='prepared_gpu_execution_pending', samples=len(rows),
|
||||
splits={s: sum(r['split'] == s for r in rows) for s in ['train', 'validation', 'test']},
|
||||
missing_input_files=missing, student_weights_found=ready, student_path=str(student),
|
||||
syntax_checked=True, tensor_tests_run=False, gpu_tests_run=False, adapter_trained=False,
|
||||
h3_quality_validated=False, active_dataset_review_untouched=True,
|
||||
pending=['Provide full local Qwen3.5-9B Transformers weights in an isolated compatible environment.',
|
||||
'After dataset GPU review exits, cache teacher then student activations sequentially.',
|
||||
'Train shared bridge and evaluate held-out embedding reconstruction.',
|
||||
'Render native baseline / thinking off / thinking on / generated-state ablation with fixed inputs.'])
|
||||
if missing:
|
||||
raise RuntimeError('Calibration input files missing: ' + str(missing))
|
||||
Path(args.output).write_text(json.dumps(report, indent=2)); print(json.dumps(report, indent=2))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,101 @@
|
||||
"""GPU stage 3: one shared bridge, balanced thinking on/off, frozen cached encoders.
|
||||
Embedding reconstruction is calibration only, NOT evidence that thinking helps H3.
|
||||
"""
|
||||
import argparse, json, random
|
||||
from pathlib import Path
|
||||
from common import load_manifest, require_idle_gpu
|
||||
from bridge import BridgeConfig, HiddenStateBridge, alignment_loss
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(); ap.add_argument('--manifest', required=True); ap.add_argument('--cache', required=True)
|
||||
ap.add_argument('--output', required=True); ap.add_argument('--steps', type=int, default=500)
|
||||
ap.add_argument('--eval-every', type=int, default=50); ap.add_argument('--lr', type=float, default=0.0001)
|
||||
args = ap.parse_args(); require_idle_gpu()
|
||||
import torch
|
||||
from safetensors import safe_open
|
||||
from safetensors.torch import load_file, save_file
|
||||
torch.manual_seed(42); rng = random.Random(42)
|
||||
rows = load_manifest(args.manifest); cache = Path(args.cache); out = Path(args.output)
|
||||
if out.exists() and any(out.iterdir()):
|
||||
raise RuntimeError('Refusing to overwrite an existing bridge run')
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
for row in rows:
|
||||
for suffix in ['teacher', 'student_0', 'student_1']:
|
||||
path = cache / f'{row["id"]}.{suffix}.safetensors'
|
||||
with safe_open(str(path), framework='pt') as f:
|
||||
md = f.metadata() or {}
|
||||
key = md.get('input_key') if suffix == 'teacher' else json.loads(md['experiment'])['input_key']
|
||||
assert key == row['input_key'], 'Cache belongs to different inputs: ' + str(path)
|
||||
train = [r for r in rows if r['split'] == 'train']; val = [r for r in rows if r['split'] == 'validation']
|
||||
test = [r for r in rows if r['split'] == 'test']; assert train and val and test
|
||||
def case(row, thinking):
|
||||
teacher = load_file(str(cache / f'{row["id"]}.teacher.safetensors'))
|
||||
student = load_file(str(cache / f'{row["id"]}.student_{thinking}.safetensors'))
|
||||
return {k: v.cuda().float() for k, v in {**teacher, **student}.items()}
|
||||
model = HiddenStateBridge(BridgeConfig()).cuda()
|
||||
# Training-only per-channel scale; no held-out target features in model inputs.
|
||||
squares = torch.zeros(5120, dtype=torch.float64); tokens = 0
|
||||
for row in train:
|
||||
t = load_file(str(cache / f'{row["id"]}.teacher.safetensors'))['teacher'].float()
|
||||
squares += t.double().square().sum(0); tokens += len(t)
|
||||
model.teacher_channel_scale.copy_((squares / tokens).sqrt().clamp_min(1e-3).float().cuda())
|
||||
optimizer = torch.optim.AdamW(model.parameters(), lr=args.lr, weight_decay=.01)
|
||||
def predict(c, ablate=False):
|
||||
generated = torch.zeros_like(c['generated']) if ablate else c['generated']
|
||||
return model(c['prefix'], generated, c['tags'].long())
|
||||
@torch.no_grad()
|
||||
def measure(subset, export=False):
|
||||
model.eval(); results = []
|
||||
for row in subset:
|
||||
for thinking in (0, 1):
|
||||
c = case(row, thinking)
|
||||
for ablate in ([False, True] if thinking and export else [False]):
|
||||
pred = predict(c, ablate); t = c['teacher']
|
||||
assert torch.isfinite(pred).all()
|
||||
relative = float((pred - t).norm() / t.norm().clamp_min(1e-8))
|
||||
cosine = float(torch.nn.functional.cosine_similarity(pred.flatten(), t.flatten(), dim=0))
|
||||
results.append(dict(id=row['id'], split=row['split'], thinking=bool(thinking),
|
||||
generated_states_zeroed=ablate, relative_error=relative, cosine=cosine,
|
||||
loss=float(alignment_loss(pred, t, c['tags']))))
|
||||
if export:
|
||||
label = 'thinking_ablated' if ablate else f'thinking_{thinking}'
|
||||
name = row['id'] + '.' + label + '.safetensors'
|
||||
metadata = dict(experiment='h3_9b_hidden_state_bridge_v1', input_key=row['input_key'],
|
||||
source_id=row['id'], thinking=str(bool(thinking)), generated_states_zeroed=str(ablate),
|
||||
prompt=row['prompt'], validated_h3_quality='false', teacher_layout_required='true')
|
||||
save_file({'conditioning': pred.detach().cpu().to(torch.bfloat16).contiguous(),
|
||||
'teacher_baseline': t.cpu().to(torch.bfloat16).contiguous(),
|
||||
'minimax_token_tags': c['tags'].long().cpu().contiguous()},
|
||||
str(out / name), metadata=metadata)
|
||||
return results
|
||||
best = float('inf'); history = []
|
||||
for step in range(1, args.steps + 1):
|
||||
model.train(); row = rng.choice(train); thinking = step % 2
|
||||
c = case(row, thinking); optimizer.zero_grad(set_to_none=True)
|
||||
prediction = predict(c); loss = alignment_loss(prediction, c['teacher'], c['tags'])
|
||||
assert torch.isfinite(loss)
|
||||
loss.backward(); torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0); optimizer.step()
|
||||
if step % args.eval_every == 0 or step == args.steps:
|
||||
metrics = measure(val); score = sum(r['loss'] for r in metrics) / len(metrics)
|
||||
entry = dict(step=step, train_loss=float(loss), validation_loss=score); history.append(entry)
|
||||
print(json.dumps(entry), flush=True)
|
||||
(out / 'history.json').write_text(json.dumps(history, indent=2))
|
||||
if score < best:
|
||||
best = score
|
||||
save_file({k: v.detach().cpu().contiguous() for k, v in model.state_dict().items()},
|
||||
str(out / 'bridge.safetensors'), metadata={'config': json.dumps(model.config_dict()),
|
||||
'selected_by': 'validation reconstruction loss',
|
||||
'h3_quality_validated': 'false'})
|
||||
model.load_state_dict(load_file(str(out / 'bridge.safetensors')))
|
||||
metrics = measure(test, export=True)
|
||||
report = dict(test_metrics=metrics, h3_quality_validated=False, thinking_benefit_proven=False,
|
||||
next_step='Compare native teacher, thinking off, thinking on and generated-state ablation in H3 with fixed seed/guide/references.',
|
||||
limitation='Teacher-supervised reconstruction can suppress thinking-specific information; lower embedding error is not identity/motion quality.')
|
||||
(out / 'report.json').write_text(json.dumps(report, indent=2))
|
||||
(out / 'manifest.jsonl').write_text(Path(args.manifest).read_text())
|
||||
print(json.dumps(dict(phase='embedding_calibration_complete_gpu_video_validation_pending', output=str(out))), flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,159 @@
|
||||
"""H3 video context in Qwen embeddings without a native video-latent reference."""
|
||||
import math
|
||||
import time
|
||||
from dataclasses import replace
|
||||
|
||||
from comfy_extras.nodes_minimax_h3 import MiniMaxH3ReferenceToVideo
|
||||
from comfy_api.latest import io
|
||||
|
||||
|
||||
class _VideoContextClip:
|
||||
"""Request-local adapter; never changes the user's CLIP or cached conditioning."""
|
||||
|
||||
def __init__(self, clip, items):
|
||||
self._clip = clip
|
||||
self._items = items
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(self._clip, name)
|
||||
|
||||
def tokenize(self, text, **kwargs):
|
||||
items = list(kwargs.get("minimax_ref_items") or [])
|
||||
# Pictures keep their numbering. Context videos precede any explicit
|
||||
# native video/audio references, so the shot remains <Video 1>.
|
||||
index = 0
|
||||
while index < len(items) and items[index].get("type") == "image":
|
||||
index += 1
|
||||
kwargs["minimax_ref_items"] = items[:index] + self._items + items[index:]
|
||||
tokens = self._clip.tokenize(text, **kwargs)
|
||||
def count_video_blocks(value):
|
||||
if isinstance(value, dict):
|
||||
if value.get("minimax_video_block") is True:
|
||||
return 1
|
||||
return sum(count_video_blocks(v) for k, v in value.items() if k != "data")
|
||||
if isinstance(value, (list, tuple)):
|
||||
return sum(count_video_blocks(v) for v in value)
|
||||
return 0
|
||||
expected = sum((item["data"].shape[0] + 1) // 2 for item in self._items)
|
||||
actual = count_video_blocks(tokens)
|
||||
if actual < expected:
|
||||
raise ValueError("CLIP did not produce the H3 video vision blocks. Use the native MiniMax H3 CLIP loader.")
|
||||
print(f"[BFS H3 CLIP context] tokenizer confirmed {actual} video vision blocks "
|
||||
f"(guide requires {expected}); encoding next", flush=True)
|
||||
return tokens
|
||||
|
||||
def encode_from_tokens_scheduled(self, tokens, *args, **kwargs):
|
||||
start = time.perf_counter()
|
||||
result = self._clip.encode_from_tokens_scheduled(tokens, *args, **kwargs)
|
||||
shapes = [tuple(entry[0].shape) for entry in result]
|
||||
print(f"[BFS H3 CLIP context] Qwen encoding complete in {time.perf_counter() - start:.2f}s; "
|
||||
f"conditioning shapes={shapes}. Guide native video reference blocks: 0", flush=True)
|
||||
return result
|
||||
|
||||
|
||||
def clip_with_video_context(clip, videos, length, source_fps=24.0, context_fps=2.0):
|
||||
"""Native H3 24-fps frame convention, presented to Qwen at a selectable sampling rate.
|
||||
|
||||
Only selected frames are resized. No video VAE encoding or minimax_refs
|
||||
video block is created here. Real image reference blocks are unaffected.
|
||||
"""
|
||||
from comfy_extras.nodes_minimax_h3 import (
|
||||
CANVAS_MULTIPLE, _resize, adapt_canvas, temporal_shape,
|
||||
)
|
||||
|
||||
if not math.isfinite(source_fps) or source_fps <= 0:
|
||||
raise ValueError("CLIP video context needs a positive source frame rate")
|
||||
if not math.isfinite(context_fps) or not 0.5 <= context_fps <= 24:
|
||||
raise ValueError("clip_video_fps must be between 0.5 and 24")
|
||||
effective_fps = min(context_fps, source_fps)
|
||||
frame_count = temporal_shape(length)[0]
|
||||
items = []
|
||||
for frames in (videos or {}).values():
|
||||
if frames is None:
|
||||
continue
|
||||
if frames.ndim != 4 or frames.shape[-1] < 3:
|
||||
raise ValueError("CLIP video context needs IMAGE frames [frames, height, width, channels]")
|
||||
n = min(frames.shape[0], frame_count)
|
||||
if n < 5:
|
||||
raise ValueError("H3 CLIP video context needs at least 5 frames at the native 24 fps")
|
||||
n -= (n - 5) % 17
|
||||
height, width = frames.shape[1:3]
|
||||
cw, ch = adapt_canvas(width, height)
|
||||
if width * height < cw * ch:
|
||||
cw = max(CANVAS_MULTIPLE, round(width / CANVAS_MULTIPLE) * CANVAS_MULTIPLE)
|
||||
ch = max(CANVAS_MULTIPLE, round(height / CANVAS_MULTIPLE) * CANVAS_MULTIPLE)
|
||||
indices = sorted({round(i * source_fps / effective_fps)
|
||||
for i in range(math.ceil(n / source_fps * effective_fps))
|
||||
if round(i * source_fps / effective_fps) < n})
|
||||
selected = _resize(frames[indices], cw, ch, "disabled")
|
||||
times = [index / source_fps for index in indices]
|
||||
preview = times if len(times) <= 6 else times[:3] + times[-3:]
|
||||
print(f"[BFS H3 CLIP context] guide: input={frames.shape[0]} frames at {source_fps:g} fps; "
|
||||
f"used={n}; requested={context_fps:g} fps, effective={effective_fps:g} fps; "
|
||||
f"sent={len(indices)} RGB frames at {cw}x{ch}; "
|
||||
f"timestamps={[round(t, 3) for t in preview]} seconds (first/last); "
|
||||
"native guide video reference latents: OFF", flush=True)
|
||||
items.append({"type": "video", "data": selected,
|
||||
"timestamps": [index / source_fps for index in indices]})
|
||||
return _VideoContextClip(clip, items) if items else clip
|
||||
|
||||
|
||||
class BFSMiniMaxH3ReferenceToVideoGuideContext(MiniMaxH3ReferenceToVideo):
|
||||
@classmethod
|
||||
def define_schema(cls):
|
||||
schema = MiniMaxH3ReferenceToVideo.define_schema()
|
||||
# Keep native image/video/audio reference semantics. The separately
|
||||
# named guide is the only video excluded from native reference latents.
|
||||
inputs = list(schema.inputs)
|
||||
inputs.insert(1, io.Image.Input(
|
||||
"guide_video_0", optional=True,
|
||||
tooltip=("The motion guide / current shot, as native 24-fps frames. Qwen sees it at clip_video_fps "
|
||||
"as <Video 1>; no native guide video reference latents are added. Connect the "
|
||||
"same frames to Add Guide for motion. Other ref_videos start at <Video 2>."),
|
||||
))
|
||||
inputs.append(io.Float.Input(
|
||||
"clip_video_fps", default=2.0, min=0.5, max=24.0, step=0.5,
|
||||
tooltip=("Guide 0 sampling rate for Qwen only. Default/native presentation is 2 fps; "
|
||||
"try 4 or 8 fps. Higher values increase context and encoding cost, not output fps. "
|
||||
"Never duplicates frames above the source rate. Actual reference videos stay native 2 fps."),
|
||||
))
|
||||
for item in inputs:
|
||||
if item.id == "ref_images":
|
||||
item.template = io.Autogrow.TemplatePrefix(
|
||||
input=io.Image.Input("ref_image", tooltip="Reference image, in <Picture N> order"),
|
||||
prefix="ref_image_", min=0, max=100,
|
||||
)
|
||||
elif item.id == "ref_videos":
|
||||
item.tooltip = ("Actual native video references, separate from guide_video_0. With a guide "
|
||||
"connected, these are <Video 2>, <Video 3>, etc. They enter both CLIP and "
|
||||
"native reference latents, as in the original node. Up to 3 reference videos.")
|
||||
item.template.input.tooltip = item.tooltip
|
||||
return replace(
|
||||
schema, node_id="BFSMiniMaxH3ReferenceToVideoGuideContext",
|
||||
display_name="MiniMax H3 Reference to Video — Guide 0 CLIP Context (BFS)",
|
||||
description=("guide_video_0 enters Qwen/CLIP only as <Video 1>, without native guide video "
|
||||
"reference latents. Pictures and other reference videos retain native semantics. "
|
||||
"Use Add Guide separately for aligned motion. The target latent starts empty."),
|
||||
inputs=inputs,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def execute(cls, clip, prompt, width, height, length, ref_image_size="match", vae=None,
|
||||
audio_vae=None, guide_video_0=None, ref_images=None, ref_videos=None,
|
||||
ref_video_audios=None, ref_audios=None, clip_video_fps=2.0):
|
||||
if guide_video_0 is not None:
|
||||
clip = clip_with_video_context(clip, {"guide_video_0": guide_video_0}, length, context_fps=clip_video_fps)
|
||||
return MiniMaxH3ReferenceToVideo.execute(
|
||||
clip=clip, prompt=prompt, width=width, height=height, length=length,
|
||||
ref_image_size=ref_image_size, vae=vae, audio_vae=audio_vae,
|
||||
ref_images=ref_images, ref_videos=ref_videos,
|
||||
ref_video_audios=ref_video_audios, ref_audios=ref_audios,
|
||||
)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"BFSMiniMaxH3ReferenceToVideoGuideContext": BFSMiniMaxH3ReferenceToVideoGuideContext,
|
||||
}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"BFSMiniMaxH3ReferenceToVideoGuideContext": "MiniMax H3 Reference to Video — Guide 0 CLIP Context (BFS)",
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Experimental cached-case embedding swap, preserving existing H3 latent conditioning."""
|
||||
import os
|
||||
import torch
|
||||
import folder_paths
|
||||
from safetensors import safe_open
|
||||
from safetensors.torch import load_file
|
||||
|
||||
folder_paths.add_model_folder_path('h3_embedding_bridge', os.path.join(folder_paths.models_dir, 'h3_embedding_bridge'))
|
||||
|
||||
|
||||
class BFSH3ExperimentalEmbeddingBridge:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {'required': {
|
||||
'positive': ('CONDITIONING',),
|
||||
'embedding_file': (folder_paths.get_filename_list('h3_embedding_bridge') or ['(no calibrated export)'],),
|
||||
'baseline_tolerance': ('FLOAT', {'default': .02, 'min': 0.0, 'max': .05, 'step': .001,
|
||||
'tooltip': 'Reject cached embeddings for a different input/encoder. Use the matching calibration case, prompt, media interval and encoder. Do not raise this to bypass mismatched inputs.'}),
|
||||
}}
|
||||
|
||||
RETURN_TYPES = ('CONDITIONING', 'STRING')
|
||||
RETURN_NAMES = ('positive', 'experiment_info')
|
||||
FUNCTION = 'apply'
|
||||
CATEGORY = 'BFS/experimental'
|
||||
DESCRIPTION = ('Replace only text/vision embeddings with a calibrated 9B hidden-state bridge export for the SAME cached case. '
|
||||
'Preserves native reference latents and aligned guides. Requires trained bridge exports; this is not a general encoder replacement.')
|
||||
|
||||
def apply(self, positive, embedding_file, baseline_tolerance=.02):
|
||||
if len(positive) != 1:
|
||||
raise ValueError('Experimental bridge supports one native H3 conditioning entry')
|
||||
path = folder_paths.get_full_path_or_raise('h3_embedding_bridge', embedding_file)
|
||||
with safe_open(path, framework='pt') as f:
|
||||
md = f.metadata() or {}
|
||||
if md.get('experiment') != 'h3_9b_hidden_state_bridge_v1':
|
||||
raise ValueError('Select a held-out conditioning export, not bridge weights or another LoRA')
|
||||
data = load_file(path); base = positive[0][0]; source = base[0] if base.ndim == 3 else base
|
||||
expected = data['teacher_baseline'].float(); actual = source.detach().cpu().float()
|
||||
if actual.shape != expected.shape:
|
||||
raise ValueError('Cached case does not match current prompt/media token layout')
|
||||
relative = float((actual - expected).norm() / expected.norm().clamp_min(1e-8))
|
||||
if relative > baseline_tolerance:
|
||||
raise ValueError(f'Cached case mismatch: baseline relative error {relative:.5f}; use exact matching inputs')
|
||||
replacement = data['conditioning']; tags = data['minimax_token_tags']
|
||||
if replacement.shape != expected.shape or len(tags) != len(replacement) or not torch.isfinite(replacement).all():
|
||||
raise ValueError('Invalid bridge conditioning export')
|
||||
current_tags = positive[0][1].get('minimax_token_tags')
|
||||
if current_tags is None or not torch.equal(current_tags.detach().cpu().reshape(-1).long(), tags.reshape(-1).long()):
|
||||
raise ValueError('Cached case has different modality tags')
|
||||
values = dict(positive[0][1])
|
||||
values['minimax_token_tags'] = tags.to(current_tags.device)
|
||||
replacement = replacement.to(device=base.device, dtype=base.dtype)
|
||||
if base.ndim == 3:
|
||||
replacement = replacement[None]
|
||||
info = (f"EXPERIMENTAL {md['source_id']}: thinking={md['thinking']}, "
|
||||
f"generated states zeroed={md['generated_states_zeroed']}; "
|
||||
f"baseline error={relative:.5f}; H3 quality not validated")
|
||||
print('[BFS H3 embedding bridge] ' + info, flush=True)
|
||||
return [[replacement, values]], info
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {'BFSH3ExperimentalEmbeddingBridge': BFSH3ExperimentalEmbeddingBridge}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {'BFSH3ExperimentalEmbeddingBridge': 'BFS H3 Experimental 9B Embedding Bridge (Cached Case)'}
|
||||
+1
-1
@@ -1,7 +1,7 @@
|
||||
[project]
|
||||
name = "bfsnodes"
|
||||
description = "Custom ComfyUI nodes for reference-guided video: BFS Sampler (guide-driven IC-LoRAs, auto crop, mask inpainting, temporal chunking), LTX Multiple Controls / Identity Transfer / Edit Anything, Head-Swap and Anime2Real conditioning, MiniMax-H3 latent guides, RoPE layouts and loaders, LoRA Surgery, and Shot Planner / Shot Loop (tabbed panel): split long videos into shots at camera cuts, face-tracked cast, SAM 3 mask per shot with full frame, paste, mask, crop and crop + mask generation modes, optional translucent Colored Mark overlay for manually selected regions, optional Qwen3-VL to suggest settings, describe references ({details}) and the person to replace ({target}), continuity between shots, H3 conditioning with an optional duet and prompt writer, join with the original timing, audio and a side-by-side comparison. H3 Duet: pin a source clip beside the video, training-free."
|
||||
version = "1.116.0"
|
||||
version = "1.115.3"
|
||||
license = {file = "LICENSE"}
|
||||
dependencies = [
|
||||
"torch",
|
||||
|
||||
+55
-46
@@ -1088,52 +1088,6 @@ class MultiRefTest(unittest.TestCase):
|
||||
self.assertNotIn("ref_image", SL.BFSShotPlanner.INPUT_TYPES()["optional"])
|
||||
|
||||
|
||||
class KeyFrameSegmentTest(unittest.TestCase):
|
||||
"""Points on several frames of a shot: each key frame starts the track up to the next one."""
|
||||
|
||||
def setUp(self):
|
||||
import json as _json
|
||||
|
||||
class Detect: # the mask of a key frame carries its first point's x (in pixels), to see who filled a frame
|
||||
@staticmethod
|
||||
def execute(model, image, positive_coords, **kw):
|
||||
return types.SimpleNamespace(args=(torch.full((1, 4, 4), _json.loads(positive_coords)[0]["x"]),))
|
||||
mod = types.ModuleType("comfy_extras.nodes_sam3"); mod.SAM3_Detect = Detect
|
||||
sys.modules.setdefault("comfy_extras", types.ModuleType("comfy_extras"))
|
||||
self._old = (sys.modules.get("comfy_extras.nodes_sam3"), SL._sam3, SL._track)
|
||||
sys.modules["comfy_extras.nodes_sam3"] = mod
|
||||
SL._sam3 = lambda: (None, None)
|
||||
self.tracks = []
|
||||
|
||||
def track(model, imgs, seed, cond, spec):
|
||||
self.tracks.append((imgs.shape[0], float(seed.flatten()[0])))
|
||||
return seed.expand(imgs.shape[0], -1, -1).clone()
|
||||
SL._track = track
|
||||
|
||||
def tearDown(self):
|
||||
mod, SL._sam3, SL._track = self._old
|
||||
if mod is None:
|
||||
sys.modules.pop("comfy_extras.nodes_sam3", None)
|
||||
else:
|
||||
sys.modules["comfy_extras.nodes_sam3"] = mod
|
||||
|
||||
def test_each_key_frame_corrects_the_mask_from_its_frame_on(self):
|
||||
pt = lambda x: [{"x": x, "y": 0.5, "label": 1}]
|
||||
spec = dict(SL.DEFAULT_MASK, keys=[{"key": 6, "points": pt(0.75)}, {"key": 2, "points": pt(0.25)}])
|
||||
out = SL.segment_frames(torch.zeros(10, 4, 4, 3), spec)
|
||||
self.assertEqual([float(out[f, 0, 0]) for f in range(10)], [1.0] * 6 + [3.0] * 4) # frames 0-5 from key 2, 6-9 from key 6
|
||||
self.assertEqual(self.tracks, [(3, 1.0), (4, 1.0), (4, 3.0)]) # back 2->0, forward 2->5, forward 6->9
|
||||
self.assertTrue(SL.spec_has_mask(dict(SL.DEFAULT_MASK, keys=[{"key": 3, "points": pt(0.5)}])))
|
||||
self.assertFalse(SL.spec_has_mask(dict(SL.DEFAULT_MASK, keys=[{"key": 3, "points": []}])))
|
||||
|
||||
def test_a_plan_with_one_key_frame_works_as_before(self):
|
||||
spec = dict(SL.DEFAULT_MASK, points=[{"x": 0.5, "y": 0.5, "label": 1}], key=4)
|
||||
self.assertEqual(SL.point_keys(spec), [(4, spec["points"])])
|
||||
out = SL.segment_frames(torch.zeros(8, 4, 4, 3), spec)
|
||||
self.assertEqual(self.tracks, [(5, 2.0), (4, 2.0)])
|
||||
self.assertEqual(float(out.sum()), 2.0 * 8 * 16)
|
||||
|
||||
|
||||
class DescribeShortTest(unittest.TestCase):
|
||||
def test_short_details_fit_inside_the_prompt(self):
|
||||
self.assertEqual(SL.tidy_details('"The green goblin in a torn leather vest."\nextra', {"describe_preset": "short"}),
|
||||
@@ -1621,3 +1575,58 @@ class FillDetailsTest(unittest.TestCase):
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
|
||||
class RedoRecoveryTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.tmp.cleanup)
|
||||
self.directory = Path(self.tmp.name)
|
||||
self.plan = {"video": "test.mp4"}
|
||||
self.state = {"done": [0, 1], "count": 2, "sig": {"0": "a", "1": "b"},
|
||||
"gen": {"0": "old", "1": "next"}, "from": {"1": "old"},
|
||||
"cur": {"sigs": ["a", "b"], "prev": [False, True]}}
|
||||
for i in range(2):
|
||||
(self.directory / f"shot_{i:04d}.pt").write_bytes(b"saved take")
|
||||
for name, value in (("_input_path", lambda _: "video"), ("run_key", lambda *_: "test"),
|
||||
("run_dir", lambda *a, **kw: str(self.directory))):
|
||||
patcher = patch.object(SL, name, value)
|
||||
patcher.start()
|
||||
self.addCleanup(patcher.stop)
|
||||
SL.save_state("test", self.state)
|
||||
|
||||
def test_redo_can_be_cancelled_with_continuity_restored(self):
|
||||
self.assertEqual(SL.redo_shots(self.plan, [0])["ok"], [False, False])
|
||||
result = SL.redo_shots(self.plan, [0], cancel=True)
|
||||
self.assertEqual(result["ok"], [True, True])
|
||||
self.assertEqual(result["redo"], [])
|
||||
|
||||
def test_reset_preserves_files_and_state(self):
|
||||
SL.redo_shots(self.plan, [0], cascade=False)
|
||||
result = SL.cache_status(self.plan, reset=True)
|
||||
self.assertEqual(result["ok"], [True, True])
|
||||
self.assertEqual((self.directory / "shot_0000.pt").read_bytes(), b"saved take")
|
||||
self.assertNotIn("0", SL.run_state("test").get("keep_next", {}))
|
||||
self.assertEqual(SL.run_state("test")["gen"], self.state["gen"])
|
||||
|
||||
def test_repeated_selection_does_not_overwrite_undo(self):
|
||||
SL.redo_shots(self.plan, [0])
|
||||
SL.redo_shots(self.plan, [0])
|
||||
self.assertEqual(SL.redo_shots(self.plan, [0], cancel=True)["ok"], [True, True])
|
||||
|
||||
def test_cancel_does_not_make_changed_settings_valid(self):
|
||||
SL.redo_shots(self.plan, [0])
|
||||
st = SL.run_state("test")
|
||||
st["cur"]["sigs"][0] = "changed seed"
|
||||
SL.save_state("test", st)
|
||||
self.assertEqual(SL.redo_shots(self.plan, [0], cancel=True)["ok"], [False, False])
|
||||
|
||||
def test_cancel_does_not_restore_replaced_take(self):
|
||||
SL.redo_shots(self.plan, [0])
|
||||
st = SL.run_state("test")
|
||||
st["gen"]["0"] = "new"
|
||||
SL.save_state("test", st)
|
||||
self.assertEqual(SL.redo_shots(self.plan, [0], cancel=True)["ok"], [False, False])
|
||||
|
||||
def test_reset_without_selection_preserves_completed_run(self):
|
||||
self.assertEqual(SL.cache_status(self.plan, reset=True)["ok"], [True, True])
|
||||
|
||||
@@ -6,6 +6,7 @@ import sys
|
||||
import tempfile
|
||||
import types
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
@@ -65,6 +66,47 @@ class CompareNodeTest(unittest.TestCase):
|
||||
n += 1
|
||||
self.assertEqual(n, 18, k)
|
||||
|
||||
def test_all_shots_keep_guide_and_target_order_across_chunk_boundaries(self):
|
||||
class Writer:
|
||||
videos = {}
|
||||
def __init__(self, path, *args, **kwargs):
|
||||
self.frames = self.videos.setdefault(Path(path).name, [])
|
||||
def write(self, frames):
|
||||
self.frames.append(frames.clone())
|
||||
def close(self):
|
||||
pass
|
||||
# Scene 1 red, scene 2 green: 22 frames span the 16-frame encoder chunk.
|
||||
guide = [torch.zeros(10, 32, 48, 3), torch.zeros(12, 32, 48, 3)]
|
||||
guide[0][..., 0] = 1
|
||||
guide[1][..., 1] = 1
|
||||
target = [x.clone() for x in guide]
|
||||
with patch.object(VC.VIO, "Mp4Writer", Writer):
|
||||
out = VC.BFSVideoCompare().compare(guide, target, preview_height=32)
|
||||
self.assertEqual(out["ui"]["bfs_compare"][0]["frames"], 22)
|
||||
self.assertEqual(out["result"][2], 0)
|
||||
for name in ("guide.mp4", "target.mp4"):
|
||||
frames = torch.cat(Writer.videos[name])
|
||||
self.assertEqual(frames.shape[0], 22)
|
||||
self.assertEqual(frames[:10, 0, 0].tolist(), [[1., 0., 0.]] * 10)
|
||||
self.assertEqual(frames[10:, 0, 0].tolist(), [[0., 1., 0.]] * 12)
|
||||
|
||||
def test_joined_video_can_match_multiple_equal_timing_shots(self):
|
||||
g = [torch.zeros(3, 32, 48, 3), torch.ones(4, 32, 48, 3)]
|
||||
self.assertEqual(VC.video_chunk(g, 2, 6, 32, 48)[:, 0, 0, 0].tolist(), [0., 1., 1., 1.])
|
||||
out = VC.BFSVideoCompare().compare(g, torch.cat(g), preview_height=32)
|
||||
self.assertEqual(out["ui"]["bfs_compare"][0]["frames"], 7)
|
||||
self.assertEqual(out["result"][2], 0)
|
||||
|
||||
def test_misaligned_shot_lists_require_join_timing_instead_of_silent_truncation(self):
|
||||
g = [torch.zeros(5, 32, 48, 3), torch.zeros(7, 32, 48, 3)]
|
||||
t = [torch.zeros(6, 32, 48, 3), torch.zeros(6, 32, 48, 3)]
|
||||
with self.assertRaisesRegex(ValueError, "SAME Join"):
|
||||
VC.BFSVideoCompare().compare(g, t)
|
||||
with self.assertRaisesRegex(ValueError, "SAME Join"):
|
||||
VC.BFSVideoCompare().compare(g, torch.zeros(8, 32, 48, 3))
|
||||
with self.assertRaisesRegex(ValueError, "nonempty"):
|
||||
VC.BFSVideoCompare().compare([], t)
|
||||
|
||||
def test_reference_pictures_go_to_the_player_no_taller_than_it(self):
|
||||
from PIL import Image
|
||||
guide, target = torch.rand(4, 32, 48, 3), torch.rand(4, 32, 48, 3)
|
||||
|
||||
+25
-56
@@ -70,8 +70,7 @@ function Panel(io) {
|
||||
const maskPrev = ref({}); // "start-end" -> {frames, box, coverage, empty} from /bfs/shotloop/mask
|
||||
const showMasks = ref(true); // overlay the mask preview on the shot cards
|
||||
const vlmSug = ref({}); // "start-end" -> {segment, shot, people, recommend, reason} from the VLM
|
||||
const modal = reactive({ open: false, idx: -1, key: 0, points: [], text: "", region: 1, groups: {}, prev: null, stale: false,
|
||||
showMask: true, playing: 0, busy: "" });
|
||||
const modal = reactive({ open: false, idx: -1, key: 0, points: [], text: "", region: 1, groups: {}, prev: null, busy: "" });
|
||||
const segPeople = ref({}); // "start-end" -> {people, main} from /bfs/shotloop/plan
|
||||
const targetCrop = ref({}); // "start-end" -> data URL of the subject the description was made from
|
||||
const tab = ref(store.get("bfs.shotloop.tab", "shots"));
|
||||
@@ -109,7 +108,7 @@ function Panel(io) {
|
||||
plan.auto_prompts = { ...(p.auto_prompts || {}) };
|
||||
if (!Array.isArray(plan.bounds)) plan.bounds = [];
|
||||
if (!Array.isArray(plan.segs)) plan.segs = [];
|
||||
// plans before 1.113 had two references (ref / ref2): now a list of up to 12 pictures
|
||||
// plans before 1.113 had two references (ref / ref2): now a list of up to MAX_REFS pictures
|
||||
plan.global_refs = refList(p.global_refs, p.global_ref, p.global_ref2);
|
||||
plan.segs.forEach(m => { if (m) m.refs = refList(m.refs, m.ref, m.ref2); });
|
||||
// Colored Mark: a color per picture (was: two marker colors), regions tied to a picture (was: region 2)
|
||||
@@ -353,17 +352,15 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
} catch (e) { error.value = String(e.message || e); }
|
||||
busy.value = "";
|
||||
}
|
||||
// intoModal: every frame of the shot, for the points window's player (count 0)
|
||||
async function previewMask(i, spec, intoModal = false) {
|
||||
const s = segs.value[i]; if (!s) return;
|
||||
const label = "Segmenting with SAM 3… (the first time downloads/loads the model)";
|
||||
if (intoModal) { modal.busy = label; busySince.value = Date.now(); Object.assign(status, { stage: "", done: 0, total: 0 }); } else busy.value = label;
|
||||
error.value = "";
|
||||
try {
|
||||
const body = { plan: { ...plan }, index: i, mask: spec || null, ...(intoModal ? { count: 0 } : {}) };
|
||||
const r = await api.fetchApi("/bfs/shotloop/mask", { method: "POST", body: JSON.stringify(body) });
|
||||
const r = await api.fetchApi("/bfs/shotloop/mask", { method: "POST", body: JSON.stringify({ plan: { ...plan }, index: i, mask: spec || null }) });
|
||||
const j = await r.json(); if (j.error) throw new Error(j.error);
|
||||
if (intoModal) { modal.prev = j; modal.stale = false; modal.showMask = true; } else maskPrev.value = { ...maskPrev.value, [segKey(s)]: j };
|
||||
if (intoModal) modal.prev = j; else maskPrev.value = { ...maskPrev.value, [segKey(s)]: j };
|
||||
} catch (e) { error.value = String(e.message || e); }
|
||||
if (intoModal) modal.busy = ""; else busy.value = "";
|
||||
}
|
||||
@@ -374,52 +371,29 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
if (s && maskPrev.value[segKey(s)]) { const mp = { ...maskPrev.value }; delete mp[segKey(s)]; maskPrev.value = mp; }
|
||||
};
|
||||
const setMaskCfg = (k, v) => { plan.mask_cfg = { ...plan.mask_cfg, [k]: v }; save(); maskPrev.value = {}; };
|
||||
// The points window. A selection (region 1 = the shot's own mask, 2.. = its extra Colored Mark regions) has key
|
||||
// frames, each with its own points: SAM 3 starts from a key frame and tracks up to the next one, so points added on a
|
||||
// later frame correct the mask from there on. modal.key is the frame shown, modal.points its points (being edited).
|
||||
const keyFramesOf = m => (Array.isArray(m?.keys) && m.keys.length ? m.keys.map(k => ({ key: +k.key || 0, points: [...(k.points || [])] }))
|
||||
: (m?.points || []).length ? [{ key: +m.key || 0, points: [...m.points] }] : []);
|
||||
const pointGroup = () => modal.groups[modal.region];
|
||||
const keepFrame = () => { // the frame being edited back into its selection's key frames
|
||||
const g = pointGroup(); if (!g) return;
|
||||
g.keys = g.keys.filter(k => k.key !== modal.key);
|
||||
if (modal.points.length) g.keys = [...g.keys, { key: modal.key, points: [...modal.points] }].sort((a, b) => a.key - b.key);
|
||||
g.text = modal.text;
|
||||
};
|
||||
const showFrame = f => {
|
||||
keepFrame();
|
||||
modal.key = Math.max(0, Math.min((segs.value[modal.idx]?.len || 1) - 1, f));
|
||||
modal.points = [...(pointGroup()?.keys.find(k => k.key === modal.key)?.points || [])];
|
||||
};
|
||||
const dropKeyFrame = f => {
|
||||
const g = pointGroup(); g.keys = g.keys.filter(k => k.key !== f);
|
||||
if (f === modal.key) modal.points = [];
|
||||
modal.stale = !!modal.prev;
|
||||
};
|
||||
// Colored Mark regions in the points window: 1 = the shot's own selection (its mask), 2.. = its extra regions
|
||||
const openPoints = (i, region = 1) => {
|
||||
const s = segs.value[i]; if (!s) return;
|
||||
const groups = { 1: { keys: keyFramesOf(s.mask), text: s.mask.text || "" } };
|
||||
if (s.color_guide) s.regions.forEach((r, j) => { groups[j + 2] = { keys: keyFramesOf(r), text: r.text || "" }; });
|
||||
const r0 = groups[region] ? region : 1, key = groups[r0].keys[0]?.key ?? Math.floor(s.len / 2);
|
||||
Object.assign(modal, { open: true, idx: i, region: r0, groups, key, points: [...(groups[r0].keys[0]?.points || [])],
|
||||
text: groups[r0].text, prev: null, stale: false, showMask: true, busy: "" });
|
||||
const g1 = { key: s.mask.key ?? Math.floor(s.len / 2), points: [...(s.mask.points || [])], text: s.mask.text || "" };
|
||||
const groups = { 1: g1 };
|
||||
if (s.color_guide) s.regions.forEach((r, j) => { groups[j + 2] = { key: r.key ?? g1.key, points: [...(r.points || [])], text: r.text || "" }; });
|
||||
const g = groups[region] || g1;
|
||||
Object.assign(modal, { open: true, idx: i, region: groups[region] ? region : 1, groups, key: g.key, points: [...g.points],
|
||||
text: g.text, prev: null, busy: "" });
|
||||
};
|
||||
const switchPointRegion = region => {
|
||||
keepFrame();
|
||||
modal.region = region; modal.text = pointGroup()?.text || "";
|
||||
modal.points = [...(pointGroup()?.keys.find(k => k.key === modal.key)?.points || [])]; modal.prev = null;
|
||||
modal.groups[modal.region] = { key: modal.key, points: [...modal.points], text: modal.text };
|
||||
const next = modal.groups[region] || { key: modal.key, points: [], text: "" };
|
||||
modal.region = region; modal.points = [...next.points]; modal.text = next.text; modal.prev = null;
|
||||
};
|
||||
// the selection being edited as a mask spec (what Segment sends)
|
||||
const pointSpec = () => { keepFrame(); const g = pointGroup(); return { keys: g.keys, points: g.keys[0]?.points || [], key: g.keys[0]?.key ?? modal.key, text: g.text }; };
|
||||
const commitPointGroups = i => {
|
||||
keepFrame();
|
||||
modal.groups[modal.region] = { key: modal.key, points: [...modal.points], text: modal.text };
|
||||
const m = meta(i), g1 = modal.groups[1];
|
||||
// keys is what 1.116 reads; points + key (the first key frame) for older versions
|
||||
m.mask = { ...(m.mask || {}), keys: g1.keys, points: g1.keys[0]?.points || [], key: g1.keys[0]?.key ?? 0, text: g1.text };
|
||||
m.mask = { ...(m.mask || {}), points: [...g1.points], key: g1.key, text: g1.text };
|
||||
if (segs.value[i]?.color_guide) {
|
||||
m.color_guide_regions = markRegions(m).map((r, j) => {
|
||||
const g = modal.groups[j + 2];
|
||||
return g ? { ...r, keys: g.keys, points: g.keys[0]?.points || [], key: g.keys[0]?.key ?? 0, text: g.text } : r;
|
||||
return g ? { ...r, points: [...g.points], key: g.key, text: g.text } : r;
|
||||
});
|
||||
}
|
||||
};
|
||||
@@ -447,11 +421,7 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
};
|
||||
const savePoints = () => {
|
||||
const i = modal.idx; commitPointGroups(i); save();
|
||||
if (modal.prev && !modal.stale) { // the card's strip shows 6 of the window's frames
|
||||
const fr = modal.prev.frames, pick = [...new Set(Array.from({ length: Math.min(6, fr.length) }, (_, k) => Math.round(k * (fr.length - 1) / Math.max(1, Math.min(6, fr.length) - 1))))];
|
||||
maskPrev.value = { ...maskPrev.value, [segKey(segs.value[modal.idx])]: { ...modal.prev, frames: pick.map(k => fr[k]) } };
|
||||
}
|
||||
clearInterval(modal.playing); modal.playing = 0;
|
||||
if (modal.prev) maskPrev.value = { ...maskPrev.value, [segKey(segs.value[modal.idx])]: modal.prev };
|
||||
modal.open = false;
|
||||
};
|
||||
const targets = (scope, from) => segs.value.map((_, i) => i).filter(i => i !== from && (scope === "all" || i > from));
|
||||
@@ -461,8 +431,7 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
const src = segs.value[fromIdx]; const rel = src ? key / Math.max(1, src.len - 1) : 0.5;
|
||||
[fromIdx, ...targets(scope, fromIdx)].forEach(i => {
|
||||
const x = segs.value[i], m = meta(i);
|
||||
const key = Math.round(rel * Math.max(0, x.len - 1));
|
||||
m.mask = { ...(m.mask || {}), keys: [{ key, points: points.map(p => ({ ...p })) }], points: points.map(p => ({ ...p })), key, text: text ?? (m.mask || {}).text };
|
||||
m.mask = { ...(m.mask || {}), points: points.map(p => ({ ...p })), key: Math.round(rel * Math.max(0, x.len - 1)), text: text ?? (m.mask || {}).text };
|
||||
});
|
||||
maskPrev.value = {}; save();
|
||||
};
|
||||
@@ -475,13 +444,13 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
// part of that track, so the subject is found wherever it moves (copied points can fall on the background)
|
||||
const saveTrack = () => {
|
||||
const s0 = segs.value[modal.idx]; if (!s0 || !modal.points.length) return;
|
||||
const own = segs.value.filter((x, i) => i !== modal.idx && (x.mask.text || (x.mask.points || []).length || (x.mask.keys || []).length)).length;
|
||||
const own = segs.value.filter((x, i) => i !== modal.idx && (x.mask.text || (x.mask.points || []).length)).length;
|
||||
if (own && !window.confirm(`${own} other shot(s) have their own mask text or points, which would win over the track. Clear them so every shot uses the track?`)) {
|
||||
// keep them: those shots keep their own selection
|
||||
} else {
|
||||
segs.value.forEach((_, i) => { const m = meta(i); if (m.mask) { m.mask = { ...m.mask, points: [], keys: [], text: "" }; } });
|
||||
segs.value.forEach((_, i) => { const m = meta(i); if (m.mask) { m.mask = { ...m.mask, points: [], text: "" }; } });
|
||||
}
|
||||
const m = meta(modal.idx); m.mask = { ...(m.mask || {}), points: [], keys: [], text: "" };
|
||||
const m = meta(modal.idx); m.mask = { ...(m.mask || {}), points: [], text: "" };
|
||||
plan.track = { points: modal.points.map(p => ({ ...p })), frame: s0.start + modal.key, text: modal.text || "",
|
||||
threshold: plan.mask_cfg?.threshold ?? 0.5 };
|
||||
maskPrev.value = {}; save(); modal.open = false;
|
||||
@@ -670,9 +639,9 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
} catch { /* ignore */ }
|
||||
}
|
||||
// ---- stored shots: redo some, or combine what is stored into an MP4 (low memory, on the server)
|
||||
async function redo(shots, cascade = true) {
|
||||
async function redo(shots, cascade = true, cancel = false) {
|
||||
try {
|
||||
const r = await api.fetchApi("/bfs/shotloop/redo", { method: "POST", body: JSON.stringify({ plan: { ...plan }, shots, cascade }) });
|
||||
const r = await api.fetchApi("/bfs/shotloop/redo", { method: "POST", body: JSON.stringify({ plan: { ...plan }, shots, cascade, cancel }) });
|
||||
const j = await r.json(); if (j.error) error.value = j.error; else { Object.assign(cache, j); prog.done = j.done; }
|
||||
} catch (e) { error.value = `${e}`; }
|
||||
}
|
||||
@@ -904,7 +873,7 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
refreshFiles, analyze, autoSplit, analyzeContent, findPeople, setCast, describeTarget, previewMask, setMask, setMaskCfg,
|
||||
openPoints, switchPointRegion, savePoints, savePointsAll, saveTrack, clearTrack, pointsTo, clearMask, copyFrom, analyseVLM, setVlmCfg,
|
||||
applySug, applyAllSug, setCastOpt, setGlobalMaskVideo, setFilter, pickFile, progress, setVideo, splitAt, mergeNext, removeCut, select, drag,
|
||||
frameAt, thumbFor, playShot, playAll, stop, seek, setRef, putRefs, uploadRef, dropRefs, showFrame, dropKeyFrame, pointSpec, keyFramesOf, pictures, describePictures, setPictureDetail, setPictureCfg,
|
||||
frameAt, thumbFor, playShot, playAll, stop, seek, setRef, putRefs, uploadRef, dropRefs, pictures, describePictures, setPictureDetail, setPictureCfg,
|
||||
setMarkPicture, addMarkRegion, setMarkRegion, removeMarkRegion, setPictureColor,
|
||||
autoFor, writtenOf, setAuto, setWritten, writePrompts,
|
||||
};
|
||||
|
||||
@@ -125,6 +125,7 @@ function buildPlayer(host) {
|
||||
const r = strip.getBoundingClientRect(), n = st.data?.frames || 0;
|
||||
if (n) seek(Math.min(n - 1, Math.max(0, Math.floor((e.clientX - r.left) / r.width * n))));
|
||||
});
|
||||
const notes = el("div", {style:"white-space:pre-wrap;font-size:12px;padding:6px;color:#e6b566"});
|
||||
host.append(el("div", { class: "bfsvc" }, [
|
||||
el("div", { class: "row" }, [el("span", {}, "top"), selTop, btnSwap, el("span", {}, "bottom"), selBot, selMode, btnFull]),
|
||||
stage,
|
||||
@@ -132,6 +133,7 @@ function buildPlayer(host) {
|
||||
el("div", { class: "row" }, [btnPrev, btnPlay, btnNext, scrub, btnMute, info]),
|
||||
el("div", { class: "row" }, [btnSave, btnRec, selFormat, el("label", { title: "Draw the names on the saved / recorded video" }, [chkLabels, "labels"]), lblRefs]),
|
||||
strip,
|
||||
notes,
|
||||
]));
|
||||
|
||||
const fps = () => st.data?.fps || 24;
|
||||
@@ -456,6 +458,7 @@ function buildPlayer(host) {
|
||||
lblRefs.style.display = d.refs?.length ? "flex" : "none";
|
||||
showRefs();
|
||||
st.data = d;
|
||||
notes.textContent = (d.notes || []).join("\n");
|
||||
empty.style.display = "none";
|
||||
scrub.max = String(Math.max(0, d.frames - 1));
|
||||
refreshSelects();
|
||||
|
||||
@@ -61,14 +61,14 @@ Three run modes:
|
||||
stored on disk; then BFS Shot Join expands the part of the graph between the planner and itself (ComfyUI node
|
||||
expansion) for the next shot, so that shot is conditioned *after* the previous one exists and continuity
|
||||
(*head* / *history*) works. No browser re-queue (the tab can be closed), Save Video fires once with the full video,
|
||||
and after a failure a new Queue resumes from the stored shots (*Reset loop* starts over). Previews connected inside
|
||||
and after a failure a new Queue resumes from the stored shots (*Redo all* regenerates every shot). Previews connected inside
|
||||
the shot's part of the graph show every shot. Needs exactly one BFS Shot Planner before the join. Same idea as the
|
||||
recursive loop of Ethanfel's MiniMax H3 Context Loop.
|
||||
- **Queue loop (one shot per run)**: each run generates the next pending shot and stores it on
|
||||
disk. The join blocks every node after it until the last shot, then outputs the full video,
|
||||
so a Save Video node only fires once. With *Auto-queue the next shot* on (default), the panel re-queues
|
||||
the workflow after each shot, so one click still renders the whole video (keep the ComfyUI tab open: the panel
|
||||
does the re-queue). *Reset loop* starts over. Changing the plan starts a new run. It resumes from the stored
|
||||
does the re-queue). *Redo all* regenerates every shot. Changing the plan starts a new run. It resumes from the stored
|
||||
shots after a failure, and continuity *head* / *history* needs it: each shot reads the result of the one before.
|
||||
- **Auto loop (all shots at once)**: every shot in one queue run, each node runs for every shot before the next node.
|
||||
Holds all decoded shots in memory. Continuity *head* / *history* does not work here (all shots are conditioned
|
||||
@@ -721,3 +721,11 @@ or *Crop + mask*.
|
||||
| Only the face or the hair is described | the click was on the head: click the body |
|
||||
| The other person is described | add an exclude point (right-click) on the other person, **👁 Segment**, save, describe again |
|
||||
| The swap still hits the wrong person | make the description more specific (position: "on the left", clothing colours) |
|
||||
|
||||
### Cancel a pending redo
|
||||
|
||||
After marking a stored shot for redo, click **Undo redo** to keep its previous take.
|
||||
**Clear redo selection** cancels all pending redo selections and preserves stored files and loop progress.
|
||||
Once a shot has been regenerated, its previous take cannot be restored through this control.
|
||||
**Refresh status** rereads the cache status; it cannot recover files deleted by older versions of Reset loop.
|
||||
Changing the seed, model or other render settings still invalidates stored shots even after cancelling redo.
|
||||
|
||||
@@ -472,7 +472,7 @@ export const secondPassSize = (w, h, sp) => {
|
||||
};
|
||||
|
||||
// ---- reference pictures (<Picture 1..12>)
|
||||
export const MAX_REFS = 12;
|
||||
export const MAX_REFS = 100;
|
||||
// a reference list of the plan: `refs` (plans from 1.113 on), else the old ref / ref2 pair; empty slots at the end dropped
|
||||
export const refList = (refs, a = "", b = "") => {
|
||||
const out = (Array.isArray(refs) ? refs : [a, b]).map(x => (typeof x === "string" ? x : "")).slice(0, MAX_REFS);
|
||||
|
||||
@@ -209,11 +209,6 @@ const CSS = `
|
||||
.bsl .pt.pos{background:#3ccf6b}.bsl .pt.neg{background:#e8455a}
|
||||
.bsl .pt.cg.pos{background:var(--point-color,#00ff00)}
|
||||
.bsl .pt.other{opacity:.5}
|
||||
.bsl .kslider{position:relative;flex:1;min-width:160px;padding-bottom:8px}
|
||||
.bsl .kmarks{position:absolute;left:8px;right:8px;bottom:0;height:8px}
|
||||
.bsl .kmark{position:absolute;width:8px;height:8px;margin-left:-4px;border-radius:50%;background:var(--mark,#3ccf6b);opacity:.55;cursor:pointer;border:1px solid #000}
|
||||
.bsl .kmark.on{opacity:1}
|
||||
.bsl .kchip{display:inline-flex;gap:1px}.bsl .kchip.on button:first-child{border-color:var(--acc)}
|
||||
.bsl .mreg{display:flex;gap:6px;align-items:center;flex-wrap:wrap;margin-top:6px}
|
||||
`;
|
||||
|
||||
|
||||
@@ -15,106 +15,68 @@ export function busyBar(c, msg) {
|
||||
}
|
||||
|
||||
export function pointsModal(c) {
|
||||
const { modal, segs, plan, api } = c;
|
||||
const { modal, segs, plan, api, setMeta } = c;
|
||||
const ms = segs.value[modal.idx];
|
||||
if (!modal.open || !ms) return null;
|
||||
// Colored Mark: region 1 = the shot's own selection, 2.. = its extra regions; each one is a picture, in its color
|
||||
const regionPicture = r => (r === 1 ? ms.color_guide_picture : ms.regions[r - 2]?.picture) || 1;
|
||||
const regionColor = r => (ms.color_guide ? pictureColor(plan, regionPicture(r)) : "#3ccf6b");
|
||||
const regionColor = r => pictureColor(plan, regionPicture(r));
|
||||
const regionsN = ms.color_guide ? 1 + ms.regions.length : 1;
|
||||
const regions = Array.from({ length: regionsN }, (_, q) => q + 1);
|
||||
// the points of a region on the frame shown (the one being edited are in modal.points)
|
||||
const pointsAt = r => (r === modal.region ? modal.points : modal.groups[r]?.keys.find(k => k.key === modal.key)?.points || []);
|
||||
const keysOf = r => {
|
||||
const ks = new Set((modal.groups[r]?.keys || []).map(k => k.key));
|
||||
if (r === modal.region) { if (modal.points.length) ks.add(modal.key); else ks.delete(modal.key); }
|
||||
return [...ks].sort((a, b) => a - b);
|
||||
};
|
||||
const edited = () => { modal.stale = !!modal.prev; };
|
||||
const pointOverlays = regions.flatMap(r => pointsAt(r).map((p, k) => h("div", {
|
||||
class: ["pt", ms.color_guide && "cg", r !== modal.region && "other", p.label ? "pos" : "neg"],
|
||||
style: `left:${p.x * 100}%;top:${p.y * 100}%;--point-color:${regionColor(r)}`,
|
||||
title: r === modal.region ? "click to remove" : `Picture ${regionPicture(r)}`,
|
||||
onPointerdown: e => { if (r !== modal.region) return; e.stopPropagation(); modal.points = modal.points.filter((_, j) => j !== k); edited(); },
|
||||
})));
|
||||
const pointOverlays = ms.color_guide
|
||||
? Array.from({ length: regionsN }, (_, q) => q + 1).flatMap(region => (region === modal.region ? modal.points : modal.groups[region]?.points || []).map((p, k) => h("div", {
|
||||
class: ["pt", "cg", region !== modal.region && "other", p.label ? "pos" : "neg"],
|
||||
style: `left:${p.x * 100}%;top:${p.y * 100}%;--point-color:${regionColor(region)}`,
|
||||
title: `Region ${region} · Picture ${regionPicture(region)}${region === modal.region ? " · click to remove" : ""}`,
|
||||
onPointerdown: e => { if (region !== modal.region) return; e.stopPropagation(); modal.points = modal.points.filter((_, j) => j !== k); modal.prev = null; },
|
||||
})))
|
||||
: modal.points.map((p, k) => h("div", { class: ["pt", p.label ? "pos" : "neg"], style: `left:${p.x * 100}%;top:${p.y * 100}%`,
|
||||
title: "click to remove", onPointerdown: e => { e.stopPropagation(); modal.points = modal.points.filter((_, j) => j !== k); modal.prev = null; } }));
|
||||
const regionButton = r => {
|
||||
const k = regionPicture(r), name = ms.names[k - 1];
|
||||
return h("button", { class: modal.region === r ? "pri" : "", onClick: () => c.switchPointRegion(r),
|
||||
title: `Region ${r}: becomes <Picture ${k}> (${markName(regionColor(r))})` }, [
|
||||
name ? h("img", { class: "mthumb", src: c.viewUrl(name), style: `--mark:${regionColor(r)}` }) : null,
|
||||
` Picture ${k} · ${markName(regionColor(r))} (${keysOf(r).length} key frame${keysOf(r).length === 1 ? "" : "s"})`]);
|
||||
` Picture ${k} · ${markName(regionColor(r))} (${(r === modal.region ? modal.points : modal.groups[r]?.points || []).length})`]);
|
||||
};
|
||||
// after Segment: the mask over every frame of the shot (modal.prev.frames, one per frame)
|
||||
const masked = modal.prev && modal.showMask ? modal.prev.frames.find(f => f.f === ms.start + modal.key) : null;
|
||||
const play = () => {
|
||||
if (modal.playing) { clearInterval(modal.playing); modal.playing = 0; return; }
|
||||
modal.playing = setInterval(() => {
|
||||
if (modal.key >= ms.len - 1 || !modal.open) { clearInterval(modal.playing); modal.playing = 0; return; }
|
||||
c.showFrame(modal.key + 1);
|
||||
}, 1000 / Math.min(24, plan.fps || 24));
|
||||
};
|
||||
const marks = h("div", { class: "kmarks" }, regions.flatMap(r => keysOf(r).map(f => h("span", {
|
||||
class: ["kmark", r === modal.region && "on"], style: `left:${(f / Math.max(1, ms.len - 1)) * 100}%;--mark:${regionColor(r)}`,
|
||||
title: `key frame ${ms.start + f}`, onClick: () => c.showFrame(f) }))));
|
||||
return h(Teleport, { to: "body" }, h("div", { class: "bsl", style: "background:none;padding:0" }, h("div", { class: "mmodal",
|
||||
onKeydown: e => {
|
||||
e.stopPropagation();
|
||||
if (e.target.tagName === "INPUT" && e.target.type === "text") return;
|
||||
if (e.key === "ArrowLeft") c.showFrame(modal.key - 1);
|
||||
if (e.key === "ArrowRight") c.showFrame(modal.key + 1);
|
||||
}, onPointerdown: e => { if (e.target === e.currentTarget) modal.open = false; } }, [
|
||||
h("div", { class: "mbox", tabindex: 0 }, [
|
||||
onKeydown: e => e.stopPropagation(), onPointerdown: e => { if (e.target === e.currentTarget) modal.open = false; } }, [
|
||||
h("div", { class: "mbox" }, [
|
||||
h("div", { class: "row", style: "margin-bottom:6px" }, [h("span", { class: "sect" }, `Shot #${modal.idx + 1} · points`),
|
||||
ms.color_guide ? h("div", { class: "row" }, regions.map(regionButton)) : null,
|
||||
hint("click = keep · right-click or shift+click = exclude · click a point to remove it · points on another frame = a new key frame: the mask is tracked from each key frame up to the next one")]),
|
||||
ms.color_guide ? h("div", { class: "row" }, [hint("One point set per marked person; switch to edit each. All use the frame selected below."),
|
||||
...Array.from({ length: regionsN }, (_, q) => regionButton(q + 1))]) : null,
|
||||
hint("click = keep · right-click or shift+click = exclude · click a point to remove it")]),
|
||||
h("div", { class: "mstage" }, [
|
||||
h("div", { class: "mimg", onContextmenu: e => e.preventDefault(), onPointerdown: e => {
|
||||
const box = e.currentTarget.getBoundingClientRect();
|
||||
const x = (e.clientX - box.left) / box.width, y = (e.clientY - box.top) / box.height;
|
||||
modal.points = [...modal.points, { x: +x.toFixed(4), y: +y.toFixed(4), label: (e.button === 2 || e.shiftKey) ? 0 : 1 }];
|
||||
edited();
|
||||
modal.prev = null;
|
||||
} }, [
|
||||
h("img", { draggable: false, src: masked ? masked.src : api.apiURL(`/bfs/shotloop/frame?video=${encodeURIComponent(plan.video)}&fps=${plan.fps}&f=${ms.start + modal.key}&w=900`) }),
|
||||
h("img", { draggable: false, src: api.apiURL(`/bfs/shotloop/frame?video=${encodeURIComponent(plan.video)}&fps=${plan.fps}&f=${ms.start + modal.key}&w=900`) }),
|
||||
...pointOverlays,
|
||||
])]),
|
||||
h("div", { class: "row", style: "margin-top:6px;align-items:center" }, [
|
||||
h("button", { title: "play the shot (with the mask after Segment)", onClick: play }, modal.playing ? "⏸" : "▶"),
|
||||
h("button", { title: "previous frame (←)", onClick: () => c.showFrame(modal.key - 1) }, "◀"),
|
||||
h("button", { title: "next frame (→)", onClick: () => c.showFrame(modal.key + 1) }, "▶|"),
|
||||
h("div", { class: "row", style: "margin-top:6px" }, [
|
||||
hint(`frame ${ms.start + modal.key} (${modal.key + 1}/${ms.len})`),
|
||||
h("div", { class: "kslider" }, [
|
||||
h("input", { type: "range", min: 0, max: ms.len - 1, step: 1, value: modal.key, style: "width:100%",
|
||||
onInput: e => c.showFrame(parseInt(e.target.value)) }),
|
||||
marks,
|
||||
]),
|
||||
modal.prev ? h("label", { class: "row", style: "gap:4px" }, [h("input", { type: "checkbox", checked: modal.showMask,
|
||||
onChange: e => { modal.showMask = e.target.checked; } }), "mask"]) : null,
|
||||
]),
|
||||
h("div", { class: "row", style: "margin-top:4px;gap:4px;flex-wrap:wrap" }, [
|
||||
hint("key frames:"),
|
||||
...(keysOf(modal.region).length ? keysOf(modal.region).map(f => h("span", { class: ["kchip", f === modal.key && "on"] }, [
|
||||
h("button", { class: "sm", onClick: () => c.showFrame(f) }, `frame ${ms.start + f}`),
|
||||
h("button", { class: "sm dng", title: "remove this key frame's points", onClick: () => { c.dropKeyFrame(f); edited(); } }, "✕")]))
|
||||
: [hint("none yet: click on the frame to add points")]),
|
||||
h("input", { type: "range", min: 0, max: ms.len - 1, step: 1, value: modal.key, style: "flex:1",
|
||||
onInput: e => { modal.key = parseInt(e.target.value); modal.prev = null; } }),
|
||||
]),
|
||||
h("div", { class: "row", style: "margin-top:6px" }, [
|
||||
h("input", { type: "text", value: modal.text, style: "flex:1", placeholder: "text prompt (used when there are no points)",
|
||||
onChange: e => { modal.text = e.target.value; edited(); } }),
|
||||
h("button", { title: "remove the points of this frame", onClick: () => { modal.points = []; edited(); } }, "Clear frame"),
|
||||
h("button", { class: "pri", disabled: !!modal.busy || !(keysOf(modal.region).length || modal.text),
|
||||
onClick: () => c.previewMask(modal.idx, { ...c.pointSpec(), ...(modal.region > 1 ? { max_objects: 1 } : {}),
|
||||
preview_color: regionColor(modal.region) }, true) }, modal.prev && modal.stale ? "👁 Segment again" : "👁 Segment"),
|
||||
onChange: e => { modal.text = e.target.value; } }),
|
||||
h("button", { onClick: () => { modal.points = []; modal.prev = null; } }, "Clear points"),
|
||||
h("button", { class: "pri", disabled: !!modal.busy || !(modal.points.length || modal.text),
|
||||
onClick: () => c.previewMask(modal.idx, { points: modal.points, key: modal.key, text: modal.text,
|
||||
preview_color: ms.color_guide ? regionColor(modal.region) : undefined }, true) }, "👁 Segment"),
|
||||
]),
|
||||
modal.busy ? busyBar(c, modal.busy) : null,
|
||||
modal.prev ? hint(modal.prev.empty ? "nothing found" : `covers ${(modal.prev.coverage * 100).toFixed(1)}% of the frames`
|
||||
+ (modal.stale ? " · the points changed: Segment again to see the new mask" : " · play or scrub to check every frame; where it slips, add points on that frame"),
|
||||
"display:block;margin-top:4px") : null,
|
||||
modal.prev ? h("div", { class: "mstrip" }, [...modal.prev.frames.map(f => h("img", { src: f.src, title: `frame ${f.f}` })),
|
||||
hint(modal.prev.empty ? "nothing found" : `covers ${(modal.prev.coverage * 100).toFixed(1)}%`)]) : null,
|
||||
h("div", { class: "row", style: "margin-top:8px;justify-content:flex-end" }, [
|
||||
h("button", { onClick: () => { modal.open = false; } }, "Cancel"),
|
||||
!ms.color_guide ? h("button", { title: "Apply this frame's points to EVERY shot, each on its frame at the same relative position (a subject that stays in place)",
|
||||
!ms.color_guide ? h("button", { title: "Apply these points to EVERY shot, each on its frame at the same relative position (a subject that stays in place)",
|
||||
onClick: c.savePointsAll }, "Save → all shots") : null,
|
||||
h("button", { disabled: !modal.points.length || ms.color_guide,
|
||||
title: "Track this frame's selection with SAM 3 through the whole video (or the range), forward and backward from this frame; every shot without its own selection takes its part of the track. Also type a short text (e.g. 'woman in pink top') so the subject is found again after camera cuts.",
|
||||
title: "Track this selection with SAM 3 through the whole video (or the range), forward and backward from this frame; every shot without its own selection takes its part of the track. Also type a short text (e.g. 'woman in pink top') so the subject is found again after camera cuts.",
|
||||
onClick: c.saveTrack }, "🎯 Track → whole video"),
|
||||
h("button", { class: "pri", onClick: c.savePoints }, "Save (this shot)"),
|
||||
]),
|
||||
|
||||
@@ -40,7 +40,7 @@ function loopSection(c) {
|
||||
h("div", { class: "prog" }, [h("div", { style: `width:${(100 * prog.done / Math.max(1, prog.count || active.value.length)).toFixed(1)}%` })]),
|
||||
h("div", { class: "row", style: "margin-top:6px" }, [
|
||||
plan.run === "queue" ? check(plan.auto_continue, v => c.setPlan("auto_continue", v), "Auto-queue the next shot") : null,
|
||||
h("button", { onClick: () => c.progress(false) }, "↻ Status"), h("button", { class: "dng", onClick: () => c.progress(true) }, "⟲ Reset loop"),
|
||||
h("button", { onClick: () => c.progress(false) }, "↻ Refresh status"), h("button", { title: "Cancel pending redo selections. Stored shots and loop progress are preserved.", onClick: () => c.progress(true) }, "Clear redo selection"),
|
||||
]),
|
||||
hint(plan.run === "queue"
|
||||
? "Each run generates one shot and stores it. Nodes after BFS Shot Join only run on the last shot, with the full video."
|
||||
@@ -91,7 +91,8 @@ function cacheSection(c) {
|
||||
const i = c.segs.value.indexOf(active.value[k]);
|
||||
return h("span", { class: ["pill", g ? "ok" : ""], title: g ? "stored and up to date" : "runs on the next Queue" }, [
|
||||
`#${(i >= 0 ? i : k) + 1} ${g ? "💾" : "⏳"}`,
|
||||
g ? h("button", { class: "chip", style: "margin-left:4px", title: "generate this shot again on the next Queue"
|
||||
(cache.redo || []).includes(k) ? h("button", { class: "chip", title: "Cancel redo and keep the stored take",
|
||||
onClick: () => c.redo([k], true, true) }, "Undo redo") : g ? h("button", { class: "chip", style: "margin-left:4px", title: "generate this shot again on the next Queue"
|
||||
+ (plan.redo_only ? " (only this shot)" : " (and the shots that continue from it)"),
|
||||
onClick: () => c.redo([k], !plan.redo_only) }, "↻") : null]);
|
||||
}) : [hint(n ? "Nothing stored yet for this video: Queue the workflow." : "")]),
|
||||
|
||||
@@ -162,7 +162,7 @@ function editor(c, cur) {
|
||||
h("button", { disabled: !!busy.value || !(r.text || (r.points || []).length),
|
||||
onClick: () => c.previewMask(i, { text: r.text || "", points: r.points || [], key: r.key || 0, max_objects: 1,
|
||||
preview_color: pictureColor(plan, r.picture) }) }, "👁 Preview"),
|
||||
h("button", { onClick: () => c.openPoints(i, j + 2) }, c.keyFramesOf(r).length ? `🎯 Points (${c.keyFramesOf(r).length} key frame${c.keyFramesOf(r).length > 1 ? "s" : ""})` : "🎯 Points…"),
|
||||
h("button", { onClick: () => c.openPoints(i, j + 2) }, (r.points || []).length ? `🎯 Points (${r.points.length})` : "🎯 Points…"),
|
||||
h("button", { class: "dng", title: "remove this person's mark", onClick: () => c.removeMarkRegion(i)(j) }, "✕"),
|
||||
])),
|
||||
h("div", { class: "row", style: "margin-top:6px;align-items:center;gap:8px" }, [
|
||||
@@ -180,7 +180,7 @@ function editor(c, cur) {
|
||||
onInput: e => c.setMask(i, "text", e.target.value),
|
||||
onKeydown: e => { if (e.key === "Enter") { e.preventDefault(); c.setMask(i, "text", e.target.value); if (e.target.value || (cur.mask.points || []).length) c.previewMask(i); } } }),
|
||||
h("button", { title: "Pick positive / negative points on a frame of this shot (more precise than text when several people look alike)",
|
||||
onClick: () => c.openPoints(i) }, c.keyFramesOf(cur.mask).length ? `🎯 Points (${c.keyFramesOf(cur.mask).length} key frame${c.keyFramesOf(cur.mask).length > 1 ? "s" : ""})` : "🎯 Points…"),
|
||||
onClick: () => c.openPoints(i) }, (cur.mask.points || []).length ? `🎯 Points (${cur.mask.points.length})` : "🎯 Points…"),
|
||||
h("button", { class: hasMask(cur) && !mp ? "pri" : "", disabled: !!busy.value || !hasMask(cur),
|
||||
title: (cur.mask.points || []).length ? "Segment this shot (the points win over the text)" : "Segment this shot with the text and show a few frames",
|
||||
onClick: () => c.previewMask(i) }, "👁 Preview"),
|
||||
|
||||
Reference in New Issue
Block a user