Compare commits

...
15 Commits
Author SHA1 Message Date
Alisson Anjos 9dcb9e4d08 Add H3 Image character swap nodes and MolmoPoint picker support 2026-11-10 21:15:50 +00:00
Alisson Anjos b29bb93dd4 Shorten registry description and release v1.119.5 2026-11-09 19:46:21 +00:00
Alisson Anjos 194690985a Improve video segment picker repair workflow 2026-11-09 18:30:04 +00:00
Alisson Anjos 0fc2f69257 feat(vsegpick): auto-unmark extra subjects, one-click clear button, and explicit Molmo text tracking (v1.119.4) 2026-11-09 16:08:24 +00:00
Alisson Anjos 93bcf3806d Merge branch 'feat/swap-guide-nodes' 2026-11-09 13:24:38 +00:00
Alisson Anjos a6cae2b9c1 feat(vsegpick,h3-swap): limit subjects and reference pictures to 3 (red, blue, green) (v1.119.3) 2026-11-09 13:07:25 +00:00
Alisson Anjos e6a9d384b7 fix(h3-swap,vsegpick): runtime MiniMaxH3 VRAM estimate/peak patch and two-pass DWPose rendering (v1.119.2) 2026-11-09 12:25:46 +00:00
Alisson Anjos 6d434b246e Fix multi-subject propagation & unload H3 conditioning models from VRAM (1.119.1)
- BFSVideoSegmentPicker: track Molmo2 strictly forward from first_pick,
  highlight the clicked person on frame 0, visually verify candidate
  re-anchors (Picture 1 crop vs Picture 2 crop + color histogram + non-overlap
  with already-claimed subjects), bound backward SAM 3 tracking to the
  previous strip frame with spatial/color continuity checks, and force GC +
  CUDA cache flush on _free.
- BFSH3KeyframeConditioning: unload TE and Video VAE from GPU after
  ReferenceToVideo and AddGuide so MiniMaxH3 has full VRAM during sampling.
- Add CHANGELOG.md and bump pyproject.toml to 1.119.1.
2026-11-09 09:57:24 +00:00
NRDX 9b41672085 Merge pull request #18 from alisson-anjos/feat/swap-guide-nodes
Character swap guide nodes: Video Segment Picker + H3 Character Swap Conditioning (1.119.0)
2026-11-09 02:09:10 +00:00
Alisson AnjosandClaude Opus 5.5 33fd830581 Add character swap guide nodes: Video Segment Picker, H3 Character Swap Conditioning (1.119.0)
- BFS Video Segment Picker (SAM 3): pick each subject on frames of a video, propagate
  shot by shot (camera cuts detected), optional Molmo2 relocation after cuts, per-shot
  and per-subject caches, status/log, whole-video scrubber with lost stretches, and a
  hybrid guide output (normal map, DWPose skeleton with face, texture-free styles).
  MoGe / DWPose / Molmo2 inputs are lazy (only on Propagate); models are unloaded
  before them and after the node.
- BFS H3 Character Swap Conditioning: keyframe pins, normal map guide or DWPose guide,
  Pictures 1-4 in color order, pictures list output.
- BFS Keyframe Planner, Keyframe Marks, Track From Frame, Workflow Loop, XY Sweep and
  Frame Loop helpers.
- Segment Picker: export Panel/styles for the video picker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 02:07:56 +00:00
NRDX 22bb235c1d Merge pull request #16 from alisson-anjos/fix/panel-inference-mode
Panel buttons run models under inference_mode (fixes #14)
2026-11-07 19:04:25 +00:00
Alisson AnjosandClaude Opus 5.5 04d8012020 Panel buttons run models under inference_mode like a node; bump to 1.118.1
Describe refs / Analyse / Auto prompt / target / mask preview ran the VLM and SAM in a server thread
outside torch.inference_mode(). After a workflow run those models hold inference tensors, so the
buttons failed with "Inference tensors do not track version counter". Fixes #14.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 18:27:25 +00:00
Alisson AnjosandClaude Opus 5.5 1ebfd0c9b6 BFS Segment Picker: brush, remove points and eraser; bump to 1.118.0
The mark can now be edited by hand. 🖌 Brush paints extra area into it (e.g. hair
SAM left out), ➖ Remove cuts out what SAM segments at the points placed and
🧽 Eraser paints what to cut out. Brush and eraser strokes are stored with the
image they were painted on and dropped when the image changes; text and point
entries are segmented again as before.

The canvas is hidden from the side panel (hideInPanel): its legacy renderer wrote
its own width into the widget and shrank the canvas in the node on click.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 20:26:38 +00:00
Alisson AnjosandClaude Opus 5.5 4c8711374f BFS Video Compare: download the frame as PNG; single images save as PNG; bump to 1.117.0
🖼 PNG downloads the frame on screen as shown (layout, divider, labels, reference
strip). With one frame (two images) the node also writes lossless PNGs of guide,
target and difference at the compared size; ⬇ save becomes ⬇ save PNG and draws from
them, and rec / the video format are hidden.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 12:05:58 +00:00
Alisson AnjosandClaude Opus 5.5 d83b9ed0af Add BFS Segment Picker: pick SAM 3.1 objects on a canvas; bump to 1.116.0
The node segments an image into objects with SAM 3.1 (MODEL and CLIP from a Load
Checkpoint node, one text prompt per concept) and shows them on a Vue canvas in the
node: hover highlights an object, click selects it. Objects the automatic pass misses
are added by a SAM 3 text prompt or by keep / exclude points through /bfs/segpick/add.
Outputs the selection's mask, one mask per object, the image with the selection
tinted (magenta at 0.3 by default), a numbered overlay and the objects as JSON. The
selection lives in a hidden widget, so it is saved and shared with the workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 11:56:05 +00:00
30 changed files with 5727 additions and 20 deletions
+111
View File
@@ -0,0 +1,111 @@
# Changelog
All notable changes to **ComfyUI-BFSNodes** are documented in this file.
## [1.120.0] - 2026-11-10
### Added
- **BFS H3 Image Character Swap Conditioning**: native H3 Image conditioning and automatic still-image prompts. The guide is Picture 1; red, blue and green references are Pictures 2, 3 and 4. Supports normal maps, DWPose, overlays and outlines. Requires ComfyUI 0.45.0 or later.
- **Add Guide for MiniMax H3 Image (BFS)**: experimental frozen guide reference aligned to the target image positions through a cloned ModelPatcher. Uses native H3 Image latents without an audio stream.
- **Video Segment Picker**: optional `point_model` input and MolmoPoint-8B tracking method. Connect the loader from ComfyUI-Molmo2-H3.
- Protected-object selections, a normal-map-inside-mask mode with feathered edges, translucent overlays and colored outlines.
### Fixed
- Bound picker mask caches by memory size, keep cached masks as boolean tensors and preserve existing masks when filling empty stretches.
- Track empty stretches and nearest-anchor windows instead of repeatedly recomputing the full clip.
- Correct SAM inference tensor handling for point edits, rendering invalidation, normal-map lazy input requests and Apply fix messages.
- Stabilize compare preview sizing; expose memory cleanup after comparison.
## [1.119.5] - 2026-11-09
### Changed
- **BFS Video Segment Picker (`BFSVideoSegmentPicker`)**: Restored the classic Molmo2 full-video text tracking as the default, with the current SAM 3 tracking available in a dropdown.
- Added repair scopes for one frame, the active frame onward, recomputing all frames, and filling only empty masks. Filling empty masks is the default and preserves existing non-empty results.
- Preserve the last propagated preview while adding selections, disable auto-propagation by default, and hide the manual eraser in the video picker while retaining SAM-guided remove points.
- Shortened the package description to fit Comfy Registry metadata validation.
## [1.119.4] - 2026-11-09
### Added
- **BFS Video Segment Picker (`BFSVideoSegmentPicker`)**:
- **Automatic Unmarking / Subject Limiting (`max_subjects`)**: Added `"max_subjects"` parameter (`["auto", "1", "2", "3"]`, default `"auto"`). In `"auto"` mode, the node dynamically inspects the execution workflow graph (`prompt`) to count connected reference pictures (`picture_N`) on downstream conditioning nodes (e.g. `BFSH3KeyframeConditioning`). If only 2 pictures are connected, Subject 3 (`3 · green`) is automatically excluded from tracking, masking, and hybrid guide rendering, preventing unmatched green skeletons or normal maps from reaching MiniMax H3.
- **One-Click Subject Clear (`✕`)**: Added a clear button `✕` on each subject badge (`1 · red`, `2 · blue`, `3 · green`) in the UI header. Clicking it instantly removes all picks for that subject across all frames and triggers auto-propagation.
- **Explicit Text Tracking (`molmo_descriptions`)**: Added optional `"molmo_descriptions"` multiline input (line $k$ = Subject $k$). When provided, Molmo2 tracks using the exact text descriptions across camera cuts and shots instead of guessing from low-resolution crops, avoiding tracking errors.
---
## [1.119.3] - 2026-11-09
### Changed
- **BFS Video Segment Picker (`BFSVideoSegmentPicker`) & BFS H3 Character Swap Conditioning (`BFSH3KeyframeConditioning`)**:
- Limited selectable subjects and reference pictures to 3 (`1 · red`, `2 · blue`, `3 · green` / `picture_1`..`picture_3`) to match the 3-subject training distribution of the Normal Map and DWPose Character Swap LoRAs.
---
## [1.119.2] - 2026-11-09
### Fixed
- **BFS H3 Character Swap Conditioning (`BFSH3KeyframeConditioning`)**:
- Added runtime VRAM estimation and peak activation patching (`_patch_vram_estimate`) onto the `MiniMaxH3` model instance so ComfyUI accounts for `minimax_keyframes` (`guide_video`), `minimax_refs` (`Picture 1`–`Picture 4`), and `cross_attn` tokens when computing `minimum_memory_required`, and releases intermediate `q, k, v` and `h` buffers early in `DiTBlock` / `Attention` without modifying ComfyUI core.
- Cleared cached DWPose detector/pose sessions (`global_cached_dwpose`) safely in `_free_gpu()` without triggering lazy-import errors on unrelated modules in `sys.modules`.
- **BFS Video Segment Picker (`BFSVideoSegmentPicker`)**:
- Separated multi-subject DWPose guide rendering into two passes (erase all subject regions to neutral gray first, then draw all subject skeletons on top) so overlapping gray regions of adjacent/hugging subjects never erase each other's skeletons.
- Added per-frame DWPose matching exclusion (`exclude`), relaxed overlap fallback (`min_inside=0.25`), and neighbor-frame pose fallback (`±2` frames) so closely interacting subjects do not lose their skeleton on intermediate frames.
---
## [1.119.1] - 2026-11-09
### Fixed
- **BFS Video Segment Picker (`BFSVideoSegmentPicker`)**:
- Prevented subjects picked on later frames from being falsely propagated or re-anchored onto other people in earlier frames before `first_pick`.
- Restricted Molmo2 tracking (`molmo_relocate`) to search strictly forward from `first_pick` (`image[first_pick:]`), highlighting the selected person on the initial frame (`t=0.0s`) and dims background distractions.
- Added visual candidate verification (`molmo_verify_candidate`): compares the reference person crop (`Picture 1`) against each candidate person crop (`Picture 2`) with Molmo2, checks RGB/chroma histogram similarity (`person_color_sim`), and rejects candidates that overlap masks already claimed by other subjects (`claimed_at`).
- Bounded backward SAM 3 tracking before `first_pick` to the immediately preceding strip frame (`prev_strip`) with frame-to-frame spatial (`IoU >= 0.15`), non-overlap, and color continuity checks.
- Enforced `gc.collect()` and `mm.soft_empty_cache(True)` when freeing GPU models after segmentation/tracking.
- **BFS H3 Character Swap Conditioning (`BFSH3KeyframeConditioning`)**:
- Added `_free_gpu()` to explicitly unload `MiniMaxH3TEModel_` (15 GB) and `MiniMaxH3VideoVAE` (5 GB) and flush the CUDA cache after `MiniMaxH3ReferenceToVideo` and `MiniMaxH3AddGuide`, preventing partial-unload VRAM retention and CUDA OOM when `MiniMaxH3` (20 GB) starts sampling.
---
## [1.119.0] - 2026-11-09
### Added
- **BFS Video Segment Picker (`BFSVideoSegmentPicker`)**:
- Interactive multi-subject (1–4) SAM 3.1 video segmentation and shot-by-shot propagation with automatic camera cut detection.
- Optional Molmo2 re-anchoring across camera cuts and lost stretches, with per-shot and per-subject caching.
- Interactive timeline scrubber displaying per-frame subject coverage and lost stretches.
- Hybrid guide video output supporting `normal map`, `openpose + face` (DWPose skeleton + face landmarks over neutral gray), and texture-free styles (`flat color`, `shading`, `blur`, `pixelate`, `gray detail`, `edges`).
- Configurable `pose_region` (`convex hull` or `person outline`) and `pose_gray_opacity`.
- **BFS H3 Character Swap Conditioning (`BFSH3KeyframeConditioning`)**:
- Unified conditioning node supporting `keyframe pins`, `normal map guide`, and `DWPose guide` modes with up to 4 reference pictures (`Picture 1`–`Picture 4`) mapped to red, blue, green, and yellow subject marks.
- **Keyframe & Loop Helpers**:
- Added `BFSKeyframePlanner`, `BFSKeyframeMarks`, `BFSTrackFromFrame`, `BFSWorkflowLoop`, `BFSXYSweep`, and `BFSFrameLoop`.
---
## [1.118.1] - 2026-11-07
### Fixed
- **Panel Server Routes**: Wrapped interactive panel button model executions under `torch.inference_mode()` to match standard ComfyUI node execution.
---
## [1.118.0] - 2026-11-04
### Added
- **BFS Segment Picker (`BFSSegmentPicker`)**: Added interactive brush strokes, negative/remove points, and eraser mode on the segmentation canvas.
---
## [1.117.0] - 2026-11-03
### Added
- **BFS Video Compare (`BFSVideoCompare`)**: Added current-frame PNG download button in the comparison UI and automatic PNG output when comparing single images.
---
## [1.116.0] - 2026-11-03
### Added
- **BFS Segment Picker (`BFSSegmentPicker`)**: Interactive SAM 3.1 object picker canvas with automatic concept detection, text search, and point prompts.
+78 -1
View File
@@ -144,7 +144,7 @@ workflow once per shot, and join the results back with the original timing and s
guide at 2 fps without native guide video-reference latents; use Add Guide separately for motion. guide at 2 fps without native guide video-reference latents; use Add Guide separately for motion.
See [H3 guide modes](MINIMAX_H3_GUIDES.md#video-context-in-clip-with-aligned-guide-latents). See [H3 guide modes](MINIMAX_H3_GUIDES.md#video-context-in-clip-with-aligned-guide-latents).
- **BFS Shot Join** — concatenates, trims to the true lengths, cross-fades soft joins, returns audio. With `comparison` on, its `guide` output is the original video frame for frame. - **BFS Shot Join** — concatenates, trims to the true lengths, cross-fades soft joins, returns audio. With `comparison` on, its `guide` output is the original video frame for frame.
- **BFS Video Compare** — one player for two videos (Join's `guide` and `images`): wipe ↔ / ↕, overlay or side by side, any of guide / target / difference on top, plus a per-pixel difference map and the difference per frame (click to jump). - **BFS Video Compare** — one player for two videos (Join's `guide` and `images`): wipe ↔ / ↕, overlay or side by side, any of guide / target / difference on top, plus a per-pixel difference map and the difference per frame (click to jump). 🖼 PNG downloads the frame on screen as shown; with two images (one frame) ⬇ save PNG downloads it from lossless PNGs at the compared size.
**Cast (1.49.0)**: *Find people* tracks faces through the video (InsightFace) and groups them **Cast (1.49.0)**: *Find people* tracks faces through the video (InsightFace) and groups them
into people. Link each person to a reference and every shot uses the reference of the person into people. Link each person to a reference and every shot uses the reference of the person
@@ -252,3 +252,80 @@ The Token / Word Weight Editor accepts `(phrase:1.3)` directives in
[One workflow](examples/h3_tara_optional_sam_loop.comfy.json) now has independent [One workflow](examples/h3_tara_optional_sam_loop.comfy.json) now has independent
SAM 3.1 and shot-loop switches. Both default OFF. See SAM 3.1 and shot-loop switches. Both default OFF. See
[setup, disabled branches and status](H3_CAST_AND_TOKEN_CONTROLS.md#one-workflow-with-optional-sam-and-loop-111531). [setup, disabled branches and status](H3_CAST_AND_TOKEN_CONTROLS.md#one-workflow-with-optional-sam-and-loop-111531).
### BFS Segment Picker (1.116.0)
Segment an image into objects with SAM 3.1 and pick them with the mouse. Connect a
Load Checkpoint node with `sam3.1_multiplex_fp16.safetensors` (its MODEL and CLIP),
press **▶ Segment** (runs only this node) and the objects appear on a canvas in the
node: hover to see one, click to select it. The automatic pass looks for the
`concepts` (default `person, animal`); add what it missed with **Find** (a SAM 3 text
prompt) or with keep / exclude **points**. Outputs: the selection's mask, one mask per
selected object, the image with the selection tinted (`mark_color` at `mark_opacity`,
magenta at 0.3 by default, the Colored Mark convention), a numbered overlay and the
objects as JSON. The selection and the added objects are saved with the workflow and
segmented again on load.
Hand edits (1.118.0): **🖌 Brush** paints extra area into the mark (e.g. hair SAM left
out), **➖ Remove** cuts out what SAM segments at the points you place, and **🧽 Eraser**
paints what to cut out. Each becomes a chip; unselect it to switch it off. Brush and
eraser strokes belong to the image they were painted on and are dropped when the image
changes. Fixed: clicking inside the node no longer shrinks its canvas while the side
panel is open.
### Character swap with marked guides (1.119.0)
Mark who is replaced in the guide video, then condition MiniMax H3 Character Swap on it.
**BFS Video Segment Picker (SAM 3)** — the Segment Picker for a video. A strip shows
frames of the batch; click a frame, pick the subject tab (1 red, 2 blue, 3 green,
4 yellow = Picture 1–4) and click the person. **↔ Propagate** tracks every subject from
its picked frames to the whole video with SAM 3, shot by shot (camera cuts are detected
and never tracked across). Optional `molmo_clip` (Molmo2 H3 encoder): Molmo2 describes
the picked person, follows them through the video and re-anchors shots that have no
pick, so a person is found again after a cut. Picking runs only segment the strip;
tracking, Molmo2, MoGe and DWPose run on Propagate. Repeated runs reuse what did not
change (per subject and per shot). After the first Propagate, picking on a frame
propagates by itself (**auto**). The **Whole video** scrubber shows every frame with
its masks, the lost stretches in red; **+ add this frame** puts any frame in the strip
to fix it. A status line with a progress bar and a log shows what runs.
Outputs: `masks`, `marked` (tinted), `overlay`, `report`, and `hybrid`: the guide
with each subject drawn in its color, by `mark_style`:
- `normal map` — its MoGe normal map in gray detail, tinted (connect `normals`:
Run MoGe Inference → Render MoGe Geometry, a normal mode);
- `openpose + face` — erased to gray (`pose_region`: convex hull or person outline,
`pose_gray_opacity`) and drawn as its DWPose skeleton with the eyes and the mouth
opening (connect `pose_keypoint`: DWPose Estimator with body, hands and face);
- texture-free alternatives: flat color, shading, blur, pixelate, gray detail, edges.
`hybrid_grow` sets the margin around each subject; `hybrid_base` stacks pickers.
`free_after` unloads the models from the GPU when the node is done (default: all).
**BFS H3 Character Swap Conditioning** — `mode`:
- `keyframe pins`: the source video plus one Add Guide per edited keyframe at its own
frame (`keyframes` + `keyframe_spec` from BFS Keyframe Planner, or `keyframe_N` /
`time_N`). `keyframe_until` drops the keyframes (or every guide) below that sigma on
the returned `model`.
- `normal map guide` / `DWPose guide`: the Video Segment Picker's `hybrid` as the guide.
Pictures 1–4 are the references in color order; the prompt is written from the mode
and the number of Pictures (or `prompt_override`). Outputs: model, positive, latent,
prompt, keyframe_video and `pictures` (the Pictures as a list, e.g. for BFS Video
Compare's `reference`).
Also: **BFS Keyframe Planner** (choose the keyframe frames on the video and export
them with their spec), **BFS Keyframe Marks** / **BFS Track From Frame (SAM 3)** (a
tracked person's mask at each keyframe), **BFS Workflow Loop** (run a part of the
graph once per item, injecting the item and changing widgets per iteration),
**BFS XY Sweep** (a grid of any two widgets' values) and **BFS Frame Loop Start/End**.
### MiniMax H3 Image character swap
Requires **ComfyUI 0.45.0+** and the H3 Image model, image VAE and matching text encoder.
Use **BFS H3 Image Character Swap Conditioning** with the Picker's single-image `hybrid` as `guide_image`. Connect character references in red, blue and green order. The generated prompt treats the guide as **Picture 1** and the characters as **Pictures 2–4**. Connect `positive` and `latent` to the sampler; the node also outputs the prompt for inspection. Normal map, DWPose, translucent overlay and outline guides are supported.
**Add Guide for MiniMax H3 Image (BFS)** is an optional experimental alignment step. Connect the model, positive conditioning, native image latent, image VAE and marked guide; use its returned MODEL and positive with the sampler. It aligns the guide's frozen reference positions to the target canvas. It does not use the video/audio Add Guide latent format. Structural checks passed; visual equivalence to a video-trained guide LoRA is not guaranteed.
For **MolmoPoint-8B** in Video Segment Picker, connect `point_model` from the loader in [ComfyUI-Molmo2-H3](https://github.com/alisson-anjos/ComfyUI-Molmo2-H3), select `MolmoPoint-8B` in `propagation_method`, and use **Apply fix** with the desired repair scope. SAM 3 still produces the masks.
+27
View File
@@ -132,6 +132,12 @@ try:
except Exception as _e: # noqa except Exception as _e: # noqa
print(f"[BFSNodes] Video Compare node not loaded: {_e!r}") print(f"[BFSNodes] Video Compare node not loaded: {_e!r}")
VCMP_NODE_CLASS_MAPPINGS, VCMP_NODE_DISPLAY_NAME_MAPPINGS = {}, {} VCMP_NODE_CLASS_MAPPINGS, VCMP_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
try:
from .bfs_segment_picker import NODE_CLASS_MAPPINGS as SEGPICK_NODE_CLASS_MAPPINGS
from .bfs_segment_picker import NODE_DISPLAY_NAME_MAPPINGS as SEGPICK_NODE_DISPLAY_NAME_MAPPINGS
except Exception as _e: # noqa
print(f"[BFSNodes] Segment Picker node not loaded: {_e!r}")
SEGPICK_NODE_CLASS_MAPPINGS, SEGPICK_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
try: try:
from .minimax_h3_clip_video import NODE_CLASS_MAPPINGS as H3CLIP_NODE_CLASS_MAPPINGS from .minimax_h3_clip_video import NODE_CLASS_MAPPINGS as H3CLIP_NODE_CLASS_MAPPINGS
@@ -181,6 +187,7 @@ NODE_CLASS_MAPPINGS = {
**HSMS_NODE_CLASS_MAPPINGS, **HSMS_NODE_CLASS_MAPPINGS,
**LSURG_NODE_CLASS_MAPPINGS, **LSURG_NODE_CLASS_MAPPINGS,
**VCMP_NODE_CLASS_MAPPINGS, **VCMP_NODE_CLASS_MAPPINGS,
**SEGPICK_NODE_CLASS_MAPPINGS,
} }
NODE_DISPLAY_NAME_MAPPINGS = { NODE_DISPLAY_NAME_MAPPINGS = {
@@ -217,8 +224,28 @@ NODE_DISPLAY_NAME_MAPPINGS = {
**HSMS_NODE_DISPLAY_NAME_MAPPINGS, **HSMS_NODE_DISPLAY_NAME_MAPPINGS,
**LSURG_NODE_DISPLAY_NAME_MAPPINGS, **LSURG_NODE_DISPLAY_NAME_MAPPINGS,
**VCMP_NODE_DISPLAY_NAME_MAPPINGS, **VCMP_NODE_DISPLAY_NAME_MAPPINGS,
**SEGPICK_NODE_DISPLAY_NAME_MAPPINGS,
} }
# Character swap with marked guides (Video Segment Picker -> H3 Character Swap Conditioning), keyframe planning, and
# the loop / sweep helpers. Each module loads on its own: a missing optional dependency only skips that module.
for _mod in ("bfs_keyframe_marks", "bfs_video_segment_picker", "bfs_h3_swap_conditioning", "bfs_keyframe_planner",
"bfs_frame_loop", "bfs_xy_sweep", "bfs_workflow_loop"):
try:
import importlib as _il
_m = _il.import_module(f".{_mod}", __package__)
NODE_CLASS_MAPPINGS.update(_m.NODE_CLASS_MAPPINGS)
NODE_DISPLAY_NAME_MAPPINGS.update(_m.NODE_DISPLAY_NAME_MAPPINGS)
except Exception as _e: # noqa: BLE001
print(f"[BFSNodes] {_mod} not loaded: {_e!r}")
WEB_DIRECTORY = "./web/js" WEB_DIRECTORY = "./web/js"
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS", "WEB_DIRECTORY"] __all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS", "WEB_DIRECTORY"]
try:
from .bfs_h3_image_swap_conditioning import BFSH3ImageCharacterSwapConditioning
NODE_CLASS_MAPPINGS["BFSH3ImageCharacterSwapConditioning"] = BFSH3ImageCharacterSwapConditioning
NODE_DISPLAY_NAME_MAPPINGS["BFSH3ImageCharacterSwapConditioning"] = "BFS H3 Image Character Swap Conditioning"
except ImportError as _e:
print(f"[BFSNodes] H3 Image conditioning requires updated ComfyUI: {_e}")
+136
View File
@@ -0,0 +1,136 @@
"""BFS Frame Loop: run any part of a workflow once per frame, one after another, and collect the results.
Wire a batch of frames (e.g. the Keyframe Planner's source_frames, or BFS Keyframe Marks' marked_frames) into
BFS Frame Loop Start; it gives one frame (plus its mask, index and count). Build the per-frame work after it (an image
edit: Qwen 2.1 body swap, Klein, ...), and wire its result into BFS Frame Loop End. The End node expands the body
between Start and End again for the next frame until all are done, then returns them as one batch (same order).
Models stay outside the loop body. Results are kept on disk between iterations; no global state.
The expansion follows the BFS H3 Native Shot Loop pattern (start/end graph expansion).
"""
import json
import os
import uuid
from pathlib import Path
import torch
import torch.nn.functional as F
try:
import folder_paths
except Exception: # tests run without ComfyUI
folder_paths = None
CATEGORY = "BFS/Loop"
class BFSFrameLoopStart:
@classmethod
def INPUT_TYPES(cls):
return {"required": {"images": ("IMAGE", {"tooltip": "The frames to process one by one."})},
"optional": {"masks": ("MASK", {"tooltip": "One mask per frame (optional), handed out with its frame."}),
"loop_state": ("BFS_FRAME_LOOP", {"tooltip": "Set by Frame Loop End; leave unconnected."})}}
RETURN_TYPES = ("IMAGE", "MASK", "INT", "INT", "BFS_FRAME_LOOP")
RETURN_NAMES = ("image", "mask", "index", "count", "loop_state")
OUTPUT_TOOLTIPS = ("The current frame.", "Its mask (zeros when no masks are given).", "Index of the current frame (0-based).",
"How many frames the loop runs.", "Connect to BFS Frame Loop End.")
FUNCTION = "start"
CATEGORY = CATEGORY
DESCRIPTION = "Hands out one frame per loop iteration; pair it with BFS Frame Loop End."
def start(self, images, masks=None, loop_state=None):
n = int(images.shape[0])
if loop_state is None:
temp = folder_paths.get_temp_directory() if folder_paths else "."
folder = os.path.join(temp, "bfs_frame_loop", uuid.uuid4().hex[:12])
os.makedirs(folder, exist_ok=True)
loop_state = {"folder": folder, "index": 0, "count": n, "done": []}
state = dict(loop_state)
if state["count"] != n:
raise ValueError(f"The loop started with {state['count']} frames but now gets {n}.")
i = int(state["index"])
h, w = images.shape[1:3]
if masks is not None:
m = masks if masks.ndim == 3 else masks[None]
if m.shape[0] != n:
raise ValueError(f"{m.shape[0]} masks for {n} frames.")
mask = m[i:i + 1]
if mask.shape[1:] != (h, w):
mask = F.interpolate(mask[:, None].float(), size=(h, w), mode="bilinear", align_corners=False)[:, 0]
else:
mask = torch.zeros(1, h, w)
print(f"[BFS Frame Loop] frame {i + 1}/{n}", flush=True)
return images[i:i + 1], mask, i, n, state
class BFSFrameLoopEnd:
@classmethod
def INPUT_TYPES(cls):
return {"required": {"image": ("IMAGE", {"tooltip": "The processed frame of this iteration."}),
"loop_state": ("BFS_FRAME_LOOP", {"tooltip": "From BFS Frame Loop Start."})},
"optional": {"resize_to_first": ("BOOLEAN", {"default": True,
"tooltip": "Resize every result to the first result's size so they form one batch."})},
"hidden": {"dynprompt": "DYNPROMPT", "unique_id": "UNIQUE_ID"}}
RETURN_TYPES = ("IMAGE", "INT")
RETURN_NAMES = ("images", "count")
OUTPUT_TOOLTIPS = ("All processed frames, in order, as one batch.", "How many frames.")
FUNCTION = "finish"
CATEGORY = CATEGORY
DESCRIPTION = "Collects one processed frame per iteration and expands the loop body until all frames are done."
def finish(self, image, loop_state, resize_to_first=True, dynprompt=None, unique_id=None):
state = dict(loop_state)
i = int(state["index"])
path = Path(state["folder"]) / f"{i:05d}.pt"
tmp = path.with_suffix(".tmp"); torch.save(image[:1].detach().cpu().contiguous(), tmp); tmp.replace(path)
state["done"] = list(state["done"]) + [str(path)]
state["index"] = i + 1
if state["index"] < state["count"]:
if dynprompt is None:
raise ValueError("BFS Frame Loop needs ComfyUI's dynamic graph execution")
return self.expand(dynprompt, str(unique_id), state)
parts = [torch.load(p, map_location="cpu", weights_only=True) for p in state["done"]]
if resize_to_first:
h, w = parts[0].shape[1:3]
parts = [p if p.shape[1:3] == (h, w) else
F.interpolate(p.movedim(-1, 1), size=(h, w), mode="bilinear", align_corners=False).movedim(1, -1) for p in parts]
if len({tuple(p.shape[1:]) for p in parts}) != 1:
raise ValueError("The processed frames have different sizes; turn on resize_to_first.")
(Path(state["folder"]) / "manifest.json").write_text(json.dumps(state, indent=1))
return torch.cat(parts), len(parts)
@staticmethod
def expand(dynprompt, end, state):
from comfy_execution.graph_utils import GraphBuilder, is_link
feeds, seen = {}, set()
def visit(nid):
if nid in seen:
return
seen.add(nid)
for value in dynprompt.get_node(nid).get("inputs", {}).values():
if is_link(value):
parent = str(value[0]); feeds.setdefault(parent, []).append(nid); visit(parent)
visit(end)
starts = [n for n in seen if dynprompt.get_node(n)["class_type"] == "BFSFrameLoopStart"]
if len(starts) != 1:
raise ValueError("Connect exactly one BFS Frame Loop Start upstream of this Frame Loop End")
start = starts[0]; body = {start}; pending = [start]
while pending:
for child in feeds.get(pending.pop(), []):
if child not in body:
body.add(child); pending.append(child)
graph = GraphBuilder()
clones = {n: graph.node(dynprompt.get_node(n)["class_type"], "Next" if n == end else n) for n in body}
for nid, node in clones.items():
node.set_override_display_id(dynprompt.get_display_node_id(nid))
for key, value in dynprompt.get_node(nid).get("inputs", {}).items():
node.set_input(key, clones[str(value[0])].out(value[1]) if is_link(value) and str(value[0]) in body else value)
clones[start].set_input("loop_state", state)
return {"result": tuple(clones[end].out(i) for i in range(2)), "expand": graph.finalize()}
NODE_CLASS_MAPPINGS = {"BFSFrameLoopStart": BFSFrameLoopStart, "BFSFrameLoopEnd": BFSFrameLoopEnd}
NODE_DISPLAY_NAME_MAPPINGS = {"BFSFrameLoopStart": "BFS Frame Loop Start", "BFSFrameLoopEnd": "BFS Frame Loop End"}
+81
View File
@@ -0,0 +1,81 @@
"""Aligned image guide for the native MiniMax H3 Image reference stream."""
import copy
import comfy.utils
from comfy_api.latest import io
MARKER = 'bfs_h3_image_guide'
def align_guide_layout(layout, refs):
result = copy.copy(layout)
result.position_ids = layout.position_ids.clone()
target = next((a, b) for a, b, kind in layout.segments if kind == 'video')
image_refs = iter(ref for ref in refs if ref['kind'] == 'image')
for a, b, kind in layout.segments:
if kind != 'ref_img':
continue
ref = next(image_refs)
if ref.get(MARKER):
ta, tb = target
if b - a != tb - ta:
raise ValueError('H3 Image guide must match the target canvas')
result.position_ids[a:b] = layout.position_ids[ta:tb]
return result
def patch_image_guide(model):
patched = model.clone()
if patched.model_options.get(MARKER):
return patched
previous = patched.model_options.get('model_function_wrapper')
def wrapper(model_function, args):
conds = args['c']
payload = conds.get('minimax_payload', {})
refs = payload.get('refs', [])
if any(ref.get(MARKER) for ref in refs):
layout = align_guide_layout(payload['layout'], refs)
args = dict(args, c=dict(conds, minimax_payload=dict(payload, layout=layout)))
if previous is not None:
return previous(model_function, args)
return model_function(args['input'], args['timestep'], **args['c'])
patched.set_model_unet_function_wrapper(wrapper)
patched.model_options[MARKER] = True
return patched
class BFSH3ImageAddGuide(io.ComfyNode):
@classmethod
def define_schema(cls):
return io.Schema(node_id='BFSH3ImageAddGuide', display_name='Add Guide for MiniMax H3 Image (BFS)',
category='BFS/MiniMax H3',
description='Experimental aligned guide for H3 Image. Uses a frozen image reference with target-aligned positions. Connect the returned MODEL and positive to the sampler; use the native H3 Image latent.',
inputs=[io.Model.Input('model'), io.Conditioning.Input('positive'), io.Vae.Input('vae'),
io.Image.Input('guide'), io.Latent.Input('latent')],
outputs=[io.Model.Output(), io.Conditioning.Output(display_name='positive')])
@classmethod
def execute(cls, model, positive, vae, guide, latent):
samples = latent['samples']
if samples.is_nested or samples.ndim != 5 or samples.shape[1:3] != (24, 1):
raise ValueError('Use the LATENT from MiniMax H3 Image Conditioning or Empty MiniMax H3 Image Latent')
if model.model.diffusion_model.__class__.__name__ != 'MiniMaxH3ImageModel':
raise ValueError('This guide requires the native MiniMax H3 Image model')
h, w = samples.shape[-2:]
pixels = comfy.utils.common_upscale(guide[:1, ..., :3].movedim(-1, 1), w * 16, h * 16, 'bicubic', 'center').movedim(1, -1)
z = vae.encode(pixels)
if z.ndim != 5 or tuple(z.shape[1:]) != (24, 1, h, w):
raise ValueError('Guide VAE must produce native H3 Image latents matching the target')
guide_ref = {'kind': 'image', 'latent_h': h, 'latent_w': w, 'latent': z, MARKER: True}
out = []
for cond, options in positive:
refs = [r for r in options.get('minimax_refs', []) if not r.get(MARKER)]
updated = dict(options, minimax_refs=refs + [guide_ref])
out.append([cond, updated])
return io.NodeOutput(patch_image_guide(model), out)
NODE_CLASS_MAPPINGS = {'BFSH3ImageAddGuide': BFSH3ImageAddGuide}
NODE_DISPLAY_NAME_MAPPINGS = {'BFSH3ImageAddGuide': 'Add Guide for MiniMax H3 Image (BFS)'}
+76
View File
@@ -0,0 +1,76 @@
"""Native MiniMax H3 Image conditioning with color-marked character references."""
from comfy_api.latest import io
from comfy_extras.nodes_minimax_h3 import MiniMaxH3ImageConditioning
COLORS = ('red', 'blue', 'green')
MODES = ('normal map guide', 'DWPose guide', 'translucent overlay guide', 'outline guide')
def image_swap_prompt(count, mode):
if mode not in MODES:
raise ValueError(f'Unknown guide mode: {mode}')
if not 1 <= count <= len(COLORS):
raise ValueError('Connect one to three character reference images')
marks = {
'normal map guide': ('normal map', 'each subject to replace is marked with its surface-normal map colorized in one solid color (its 3D surface, without its real colors)'),
'DWPose guide': ('pose skeleton', 'each subject to replace is shown as its pose skeleton (body, hands and face) drawn in one color'),
'translucent overlay guide': ('translucent overlay', 'each subject to replace is identified by a translucent color overlay; the appearance beneath the tint is the source appearance to replace'),
'outline guide': ('outline', 'each subject to replace is identified by a thin colored outline around that subject'),
}
mark, explanation = marks[mode]
definitions = '\n'.join(
f'<Subject {j}> is the subject marked by the {COLORS[j-1]} {mark} in <Picture 1>; its appearance comes from <Picture {j+1}>, its position, pose, head orientation, gaze and facial expression from the {COLORS[j-1]} mark.'
for j in range(1, count+1))
replacements = ', '.join(f'the {COLORS[j-1]} subject replaced by <Picture {j+1}>' for j in range(1, count+1))
retention = '\n'.join(
f'<Subject {j}>: attribute_transfer - keeps the position, pose, head orientation, gaze and facial expression of the {COLORS[j-1]} mark and takes the face, hair, body and whole outfit of <Picture {j+1}>.'
for j in range(1, count+1))
return (
'character_swap:\nsubject_definitions:\n'
f'<Picture 1> is the source of the edit. In it, {explanation}; the mark says who changes, not where the new look must end, and everything else in the guide is the original image.\n'
f'{definitions}\n'
f'summary: [image editing + reference generation] The target image is <Picture 1> with {replacements}, following each mark\'s pose and expression. '
'Every other person, the layout, the background, lighting, camera perspective and framing stay exactly as in <Picture 1>. '
'Keep held objects and foreground occluders in their original positions, shapes and colors; they are separate from the subjects being replaced. '
'Remove the normal-map colors, skeletons, overlays and outlines from the final image.\n'
f'retention_analysis:\n{retention}\n'
'Everyone else, held objects, foreground occluders, the layout and the background: fully_preserved.')
class BFSH3ImageCharacterSwapConditioning(io.ComfyNode):
@classmethod
def define_schema(cls):
return io.Schema(
node_id='BFSH3ImageCharacterSwapConditioning',
display_name='BFS H3 Image Character Swap Conditioning',
category='BFS/MiniMax H3',
description='Native H3 Image character swap: the marked guide is Picture 1; red, blue and green subjects take their appearance from Pictures 2, 3 and 4. Produces native image reference latents and a still-image prompt.',
inputs=[
io.Clip.Input('clip'), io.Vae.Input('vae'),
io.Image.Input('guide_image', tooltip='Marked hybrid image from BFS Video Segment Picker; becomes Picture 1.'),
io.Autogrow.Input('references', template=io.Autogrow.TemplatePrefix(
input=io.Image.Input('reference', tooltip='Character appearance in color order: red, blue, green; becomes Picture 2, 3, 4.'),
prefix='reference_', min=1, max=3)),
io.Combo.Input('mode', options=MODES, default=MODES[0]),
io.Int.Input('width', default=0, min=0, max=16384, step=32, tooltip='0 uses the native H3 Image sizing based on the guide.'),
io.Int.Input('height', default=0, min=0, max=16384, step=32),
io.String.Input('prompt_override', default='', multiline=True, optional=True,
tooltip='Optional complete replacement prompt. Guide remains Picture 1; character references start at Picture 2.'),
],
outputs=[io.Conditioning.Output(display_name='positive'), io.Latent.Output(display_name='latent'),
io.String.Output(display_name='prompt')])
@classmethod
def execute(cls, clip, vae, guide_image, references, mode=MODES[0], width=0, height=0, prompt_override=''):
names = sorted(references, key=lambda name: int(name.rsplit('_', 1)[-1]))
pictures = [references[name] for name in names if references[name] is not None]
prompt = prompt_override.strip() or image_swap_prompt(len(pictures), mode)
images = {'image_1': guide_image}
images.update({f'image_{j}': picture for j, picture in enumerate(pictures, 2)})
positive, latent = MiniMaxH3ImageConditioning.execute(
clip=clip, vae=vae, prompt=prompt, width=width, height=height, images=images).args[:2]
return io.NodeOutput(positive, latent, prompt)
NODE_CLASS_MAPPINGS = {'BFSH3ImageCharacterSwapConditioning': BFSH3ImageCharacterSwapConditioning}
NODE_DISPLAY_NAME_MAPPINGS = {'BFSH3ImageCharacterSwapConditioning': 'BFS H3 Image Character Swap Conditioning'}
+419
View File
@@ -0,0 +1,419 @@
"""BFS H3 Character Swap Conditioning: MiniMax H3 Character Swap where the guide says who is replaced.
Five ways to mark the subjects (`mode`):
- "keyframe pins": the source video as an aligned guide (Add Guide at frame 0) plus one Add Guide per edited keyframe
at its own frame index (frames of the result with the new people in place, e.g. from BFS Keyframe Planner).
- "normal map guide": the guide video with each subject to replace drawn as its normal map in gray detail, tinted with
the subject's color (1 red, 2 blue, 3 green = Picture 1..3).
- "DWPose guide": the guide video with each subject erased to gray and drawn as its DWPose skeleton (body, hands and
face, with the eyes and the mouth opening) in the subject's color.
- "translucent overlay guide": the source video with each subject's mask tinted in its color at the Picker's
mark_opacity, so the original pose and motion remain visible under the tint.
- "outline guide": the source video with a thin colored contour inside each subject's mask; the scene and held
objects remain visible.
The marked guides come from BFS Video Segment Picker (`hybrid` output). The Pictures are native references and the
prompt is written from the mode and the number of Pictures (or prompt_override).
"""
import json
import torch
GRAY = 128.0 / 255.0 # background of the keyframe_video preview
MAX_KEYFRAMES = 8
MAX_PICTURES = 3
PINS, NORMALS, POSE, OVERLAY, OUTLINE = ("keyframe pins", "normal map guide", "DWPose guide",
"translucent overlay guide", "outline guide")
MODES = [PINS, NORMALS, POSE, OVERLAY, OUTLINE]
SUBJECT_COLORS = ["red", "blue", "green"]
def pose_caption(k):
"""The DWPose guide prompt (k Pictures)."""
C = SUBJECT_COLORS
defs = '\n'.join(f'<Subject {j}> is the subject marked by the {C[j - 1]} pose skeleton in the guide video; its appearance comes '
f'from <Picture {j}>, its pose, motion, facial expression and mouth movement from the {C[j - 1]} skeleton.'
for j in range(1, k + 1))
ret = '\n'.join(f'<Subject {j}>: attribute_transfer - keeps the pose, motion, expression and lip movement of the {C[j - 1]} skeleton '
f'and takes the face, hair, body shape and whole outfit of <Picture {j}> in every frame.' for j in range(1, k + 1))
who = ', '.join(f'the {C[j - 1]} subject replaced by <Picture {j}>' for j in range(1, k + 1))
return ('character_swap:\nsubject_definitions:\nThe guide video is the source of the edit. In it, each subject to replace is erased to '
'plain gray and shown only as its pose skeleton (body, hands and face, with the eyes and the mouth opening) drawn in one '
'color; the mark gives the pose and the expression, not the body shape, and everything else in the guide is the real video.\n'
+ defs + f'\nsummary: [video editing + reference generation] The target video is the guide video with {who}, from the first '
'frame to the last, following each skeleton\'s pose, expression and mouth. Every other person, the layout, the backgrounds, '
'the camera and the motion stay exactly as in the guide video.\nretention_analysis:\n' + ret +
'\nEveryone else, the layout and the backgrounds: fully_preserved.')
def normals_caption(k):
"""The normal map guide prompt (k Pictures)."""
C = SUBJECT_COLORS
defs = '\n'.join(f'<Subject {j}> is the subject marked by the {C[j - 1]} normal map in the guide video; its appearance comes '
f'from <Picture {j}>, its pose and motion from the {C[j - 1]} mark.' for j in range(1, k + 1))
ret = '\n'.join(f'<Subject {j}>: attribute_transfer - keeps the pose and motion of the {C[j - 1]} mark and takes the face, '
f'hair, body and whole outfit of <Picture {j}> in every frame.' for j in range(1, k + 1))
who = ', '.join(f'the {C[j - 1]} subject replaced by <Picture {j}>' for j in range(1, k + 1))
return ('character_swap:\nsubject_definitions:\nThe guide video is the source of the edit. In it, each subject to replace is marked '
'with its surface-normal map colorized in one solid color (its 3D surface, without its real colors); the mark says who '
'changes, not where the new look must end, and everything else in the guide is the real video.\n' + defs +
f'\nsummary: [video editing + reference generation] The target video is the guide video with {who}, from the first frame '
'to the last, following each mark\'s pose and motion. Every other person, the layout, the backgrounds, the camera and the '
'motion stay exactly as in the guide video. Keep held objects and foreground occluders in their original '
'positions, shapes and colors; they are separate from the subject being replaced.\nretention_analysis:\n' + ret +
'\nEveryone else, held objects, foreground occluders, the layout and the backgrounds: fully_preserved.')
def overlay_caption(k):
"""Prompt for the Picker's translucent color overlay, where source appearance is still visible underneath."""
C = SUBJECT_COLORS
defs = '\n'.join(f'<Subject {j}> is the person under the translucent {C[j - 1]} overlay in the guide video. '
f'Replace that person with <Picture {j}> while keeping their position, pose, motion and timing.'
for j in range(1, k + 1))
ret = '\n'.join(f'<Subject {j}>: attribute_transfer - use the face, hair, body and full outfit of <Picture {j}>; '
f'use the {C[j - 1]} marked person only for position, pose, motion and timing.'
for j in range(1, k + 1))
return ('character_swap:\nsubject_definitions:\nThe guide is the original video with a translucent color overlay on each '
'person to replace. The overlay identifies the person; the original appearance visible beneath it is not the target '
'appearance. The tint must disappear in the result.\n' + defs +
'\nsummary: [video editing + reference generation] Replace every marked person throughout the video with their '
'matching Picture. Keep their actions, camera motion, all unmarked people and the background as in the guide. '
'Do not render the colored overlays.\nretention_analysis:\n' + ret +
'\nUnmarked people and background: fully_preserved.')
def outline_caption(k):
"""Prompt for a person contour that leaves the original scene and interactions unobscured."""
C = SUBJECT_COLORS
defs = '\n'.join(f'<Subject {j}> is the person inside the {C[j - 1]} outline in the guide video. '
f'Take their new appearance from <Picture {j}> while retaining their position, pose, motion and timing.'
for j in range(1, k + 1))
ret = '\n'.join(f'<Subject {j}>: attribute_transfer - use the face, hair, body and full outfit of <Picture {j}>; '
f'keep the movement and interactions of the {C[j - 1]} outlined person.'
for j in range(1, k + 1))
return ('character_swap:\nsubject_definitions:\nThe guide is the original video with a thin colored outline on each '
'person to replace. The outline is only an identity marker and must disappear in the result. The source person '
'inside the outline provides pose and motion, while their new appearance comes from the matching Picture.\n' + defs +
'\nsummary: [video editing + reference generation] Replace only the outlined people throughout the video. '
'Preserve the background, other people, and all held or interacting objects with their original colors, positions, '
'shapes and contact with the subjects. Preserve foreground occlusions and the camera motion. Do not render the '
'colored outlines.\nretention_analysis:\n' + ret +
'\nBackground, other people, held objects and occlusions: fully_preserved.')
def _tail(k):
subs = '\n'.join(f'<Subject {j}> is the person of the guide video who appears with the look of <Picture {j}> in the '
f'keyframes; the appearance comes from <Picture {j}>, the motion from the guide video.' for j in range(1, k + 1))
ret = '\n'.join(f'<Subject {j}>: attribute_transfer - keeps the motion of the guide video and takes the face, hair, body and '
f'whole outfit of <Picture {j}> in every frame.' for j in range(1, k + 1))
summary = (f'\nsummary: [video editing + keyframe completion + reference generation] The target video is the guide video with '
f'{k} {"people" if k > 1 else "person"} replaced: exactly the people that differ between the keyframes and the guide video, '
f'replaced by {", ".join(f"<Picture {j}>" for j in range(1, k + 1))} from the first frame to the last. Every other '
'person, the layout, the backgrounds, the camera and the motion stay exactly as in the guide video.\n'
'retention_analysis:\n' + ret + '\nEveryone else, the layout and the backgrounds: fully_preserved.')
return subs, summary
def pins_caption(k, times):
"""The keyframe pins prompt (k Pictures, keyframes at `times` seconds)."""
when = ', '.join(f'{t:.1f}s' for t in times); n = len(times)
subs, summary = _tail(k)
return ('character_swap:\nsubject_definitions:\nThe guide video is the source of the edit. '
f'{n} keyframe{"s" if n > 1 else ""} of the target video {"are" if n > 1 else "is"} pinned at {when}; at those times '
'the target video matches the keyframes exactly.\n' + subs + summary)
def _fit(image, h, w):
"""[1,H,W,C] -> [h,w,3], center crop to the guide's aspect, then resize."""
import comfy.utils
x = image[:1, ..., :3].movedim(-1, 1)
x = comfy.utils.common_upscale(x, w, h, "lanczos", "center")
return x.movedim(1, -1)[0].clamp(0, 1)
class BFSH3KeyframeConditioning:
@classmethod
def INPUT_TYPES(cls):
opt = {f"picture_{j}": ("IMAGE", {"tooltip": f"Subject {j} ({SUBJECT_COLORS[j - 1]} mark in the guide modes)."})
for j in range(2, MAX_PICTURES + 1)}
opt.update({
"model": ("MODEL", {"tooltip": "keyframe pins: connect to get it back with keyframe_until applied. Guide modes: passed through."}),
"keyframes": ("IMAGE", {"tooltip": "keyframe pins: edited frames, one per keyframe (BFS Keyframe Planner keyframes)."}),
"keyframe_spec": ("STRING", {"forceInput": True, "tooltip": "keyframe pins: BFS Keyframe Planner keyframe_spec (the frame of every keyframe)."}),
})
for i in range(1, MAX_KEYFRAMES + 1):
opt[f"keyframe_{i}"] = ("IMAGE", {"tooltip": f"keyframe pins: edited frame {i}, a full frame of the RESULT (swapped person in place)."})
opt[f"time_{i}"] = ("FLOAT", {"default": float(i - 1), "min": 0.0, "max": 600.0, "step": 0.05,
"tooltip": f"keyframe pins: seconds into the guide where keyframe {i} sits."})
return {"required": {
"clip": ("CLIP", {"tooltip": "MiniMax H3's text encoder (Qwen3-VL)."}),
"vae": ("VAE",),
"guide_video": ("IMAGE", {"tooltip": "keyframe pins: the prepared 24-fps source video. Guide modes: the marked guide "
"(BFS Video Segment Picker `hybrid`)."}),
"width": ("INT", {"forceInput": True}), "height": ("INT", {"forceInput": True}), "length": ("INT", {"forceInput": True}),
"picture_1": ("IMAGE", {"tooltip": "Subject 1 (red mark in the guide modes)."}),
"mode": (MODES, {"default": PINS, "tooltip": "How the subjects to replace are marked: edited keyframes pinned at their "
"frames, or a guide with each subject drawn as a colored normal map, "
"DWPose skeleton, translucent color overlay, or colored outline "
"(BFS Video Segment Picker)."}),
"keyframe_until": ("FLOAT", {"default": 0.7, "min": 0.0, "max": 1.0, "step": 0.01,
"tooltip": "keyframe pins only, on the returned model: the guides condition every step whose "
"sigma is at least this; below it only the Pictures remain. 0.5 = more faithful to "
"the source, 0 = guides in every step. The guide modes keep the guide on every step."}),
"keyframe_until_drops": (["keyframes only", "all guides"], {"default": "keyframes only",
"tooltip": "keyframe pins only: below keyframe_until drop only the keyframes (the source video "
"keeps guiding every step) or every guide."}),
"prompt_override": ("STRING", {"multiline": True, "default": "", "tooltip": "Non-empty: used instead of the generated prompt."}),
}, "optional": opt}
RETURN_TYPES = ("MODEL", "CONDITIONING", "LATENT", "STRING", "IMAGE", "IMAGE")
RETURN_NAMES = ("model", "positive", "latent", "prompt", "keyframe_video", "pictures")
OUTPUT_IS_LIST = (False, False, False, False, False, True)
OUTPUT_TOOLTIPS = ("", "", "", "The prompt used.", "keyframe pins: the keyframes at their frames on gray. Guide modes: the guide.",
"The Pictures in order (Picture 1..4, any sizes), e.g. for BFS Video Compare's reference.")
FUNCTION = "encode"
CATEGORY = "BFS/MiniMax H3"
DESCRIPTION = ("MiniMax H3 Character Swap conditioning. keyframe pins: the source video plus one Add Guide per edited "
"keyframe. Guide modes use the BFS Video Segment Picker hybrid output: normal map, DWPose, translucent "
"overlay or colored outline. Red, blue and green marks correspond to Pictures 1, 2 and 3. "
"The node writes the matching prompt automatically.")
@staticmethod
def _collect(kw, fps=24.0):
"""[(frame, image[1,H,W,C])] from the Planner batch + spec and/or the keyframe_N / time_N inputs."""
out = []
batch, spec = kw.get("keyframes"), kw.get("keyframe_spec")
if batch is not None:
frames = json.loads(spec).get("frames", []) if spec else []
if len(frames) != len(batch):
raise ValueError(f"{len(batch)} keyframes but keyframe_spec lists {len(frames)} frames: connect the Planner's keyframe_spec.")
out += [(int(f), batch[i:i + 1]) for i, f in enumerate(frames)]
for i in range(1, MAX_KEYFRAMES + 1):
if kw.get(f"keyframe_{i}") is not None:
out.append((round(float(kw.get(f"time_{i}", 0.0)) * fps), kw[f"keyframe_{i}"]))
return sorted(out, key=lambda x: x[0])
@staticmethod
def _free_gpu():
"""Unload TE, VAE, and cached preprocessor models (e.g. DWPose) from GPU so MiniMaxH3 has full VRAM."""
try:
import gc
import sys
import comfy.model_management as mm
mm.unload_all_models()
for k, mod in list(sys.modules.items()):
if "dwpose" not in k or mod is None:
continue
try:
dw = getattr(mod, "global_cached_dwpose", None)
if dw is not None:
dw.det = None
dw.pose = None
dw.det_filename = None
dw.pose_filename = None
except Exception: # noqa: BLE001
pass
gc.collect()
mm.soft_empty_cache(True)
except Exception as e: # noqa: BLE001
print(f"[BFS H3 keyframe] could not free GPU models: {e!r}", flush=True)
@staticmethod
def _patch_vram_estimate(model):
"""Ensure MiniMaxH3 accounts for guide_video/keyframes/refs in ComfyUI's VRAM estimate so it doesn't OOM."""
if model is None:
return None
inner = getattr(model, "model", None)
if inner is None:
return model
import math
import types
def _extra_conds_shapes(_self, **kwargs):
extra = 0
payload = kwargs.get("minimax_payload", None)
if hasattr(payload, "cond") and isinstance(payload.cond, dict):
payload = payload.cond
keyframes = kwargs.get("minimax_keyframes", None)
if keyframes is None and isinstance(payload, dict):
keyframes = payload.get("keyframes", None)
refs = kwargs.get("minimax_refs", None)
if refs is None and isinstance(payload, dict):
refs = payload.get("refs", None)
for item in (keyframes or []) + (refs or []):
if not isinstance(item, dict):
continue
lat = item.get("latent", None)
if lat is not None and hasattr(lat, "shape"):
extra += math.prod(lat.shape[1:])
aud = item.get("audio_latent", None)
if aud is not None and hasattr(aud, "shape"):
extra += math.prod(aud.shape[1:])
cross_attn = kwargs.get("cross_attn", kwargs.get("c_crossattn", None))
if hasattr(cross_attn, "cond"):
cross_attn = cross_attn.cond
if cross_attn is not None and hasattr(cross_attn, "shape") and len(cross_attn.shape) >= 2:
extra += int(cross_attn.shape[1]) * 96
return {"minimax_cond": [1, 1, extra]} if extra > 0 else {}
cond_keys = tuple(getattr(inner, "memory_usage_factor_conds", ()) or ())
if "minimax_cond" not in cond_keys:
inner.memory_usage_factor_conds = cond_keys + ("minimax_cond",)
inner.extra_conds_shapes = types.MethodType(_extra_conds_shapes, inner)
if getattr(inner, "memory_usage_factor", 0.0) < 0.135:
inner.memory_usage_factor = 0.135
dm = getattr(inner, "diffusion_model", None)
if dm is not None and not getattr(dm, "_bfs_low_peak_patched", False):
try:
import comfy.ldm.minimax.model as mm_mod
import comfy.model_management as cmm
import comfy.quant_ops
from comfy.ldm.modules.attention import AttentionTensorContainer, optimized_attention
def _attn_forward(self, x, rope_freqs=None, transformer_options={}):
s = x.shape[0]
dev = x.device
q, k, v = self.qkv_proj(x).split(self.heads * self.head_dim, dim=-1)
del x
v = v.view(s, self.heads, self.head_dim)
if rope_freqs is not None:
q = q.view(1, s, self.heads, self.head_dim)
k = k.view(1, s, self.heads, self.head_dim)
qw = cmm.cast_to(self.q_norm.weight, device=dev)
kw = cmm.cast_to(self.k_norm.weight, device=dev)
rot = rope_freqs.shape[-3] * 2
if cmm.in_training:
q, k = comfy.quant_ops.ck.rms_rope_split_half(
q, k, rope_freqs, qw, kw, epsilon=self.q_norm.eps, rot_dim=rot)
else:
comfy.quant_ops.ck.rms_rope_split_half_(
q, k, rope_freqs, qw, kw, epsilon=self.q_norm.eps, rot_dim=rot)
del qw, kw
q, k = q[0], k[0]
else:
q = self.q_norm(q.view(s, self.heads, self.head_dim))
k = self.k_norm(k.view(s, self.heads, self.head_dim))
q = AttentionTensorContainer(q.transpose(0, 1).unsqueeze(0))
k = AttentionTensorContainer(k.transpose(0, 1).unsqueeze(0))
v = AttentionTensorContainer(v.transpose(0, 1).unsqueeze(0))
out = optimized_attention(
q, k, v, self.heads, preferred_attention=self.comfy_attention,
mask=None, skip_reshape=True, skip_output_reshape=True,
transformer_options=transformer_options,
)
del q, k, v
out = out.transpose(1, 2).reshape(s, self.heads * self.head_dim)
return self.out_proj(out)
def _block_forward(self, x, t_emb, mod_segments, rope_freqs, transformer_options={}, attention=None):
attention = self.attn if attention is None else attention
shift_msa, scale_msa, gate_msa, shift_mlp, scale_mlp, gate_mlp = self.adaln_proj(t_emb)
attn_out = attention(
mm_mod._mod_scale_shift(self.norm1(x), shift_msa, scale_msa, mod_segments),
rope_freqs=rope_freqs, transformer_options=transformer_options,
)
del shift_msa, scale_msa
x = mm_mod._mod_gate(x, gate_msa, attn_out, mod_segments)
del gate_msa, attn_out
mlp_out = self.mlp(mm_mod._mod_scale_shift(self.norm2(x), shift_mlp, scale_mlp, mod_segments))
del shift_mlp, scale_mlp
return mm_mod._mod_gate(x, gate_mlp, mlp_out, mod_segments)
for blk in getattr(dm, "blocks", ()):
blk.forward = types.MethodType(_block_forward, blk)
if hasattr(blk, "attn"):
blk.attn.forward = types.MethodType(_attn_forward, blk.attn)
dm._bfs_low_peak_patched = True
except Exception as e: # noqa: BLE001
print(f"[BFS H3 keyframe] could not patch block peak VRAM: {e!r}", flush=True)
return model
def encode(self, clip, vae, guide_video, width, height, length, picture_1, mode=PINS, keyframe_until=0.7,
keyframe_until_drops="keyframes only", prompt_override="", model=None, **kw):
if mode not in MODES:
raise ValueError(f"Unknown mode {mode!r}: use one of {MODES}.")
from comfy_extras.nodes_minimax_h3 import MiniMaxH3AddGuide, MiniMaxH3ReferenceToVideo
pictures = [picture_1] + [kw[f"picture_{j}"] for j in range(2, MAX_PICTURES + 1) if kw.get(f"picture_{j}") is not None]
kfs = self._collect(kw)
if mode != PINS:
return self._guide(clip, vae, guide_video, width, height, length, pictures, prompt_override, model, kw, mode=mode)
if not kfs:
raise ValueError("No keyframes: connect the Planner's keyframes + keyframe_spec, or keyframe_1 / time_1.")
if len(guide_video) != length:
raise ValueError(f"guide has {len(guide_video)} frames, length is {length}: connect Prepare's length.")
h, w = guide_video.shape[1:3]
video = torch.full((length, h, w, 3), GRAY, dtype=guide_video.dtype)
placed, frames = [], []
for f, img in kfs:
f = min(length - 1, max(0, int(f)))
if f in placed:
print(f"[BFS H3 keyframe] two keyframes at frame {f}", flush=True)
frame = _fit(img, h, w).to(video.dtype)
video[f] = frame
placed.append(f); frames.append(frame)
times = [f / 24.0 for f in placed]
prompt = prompt_override.strip() or pins_caption(len(pictures), times)
print("[BFS H3 keyframe] prompt:\n" + prompt, flush=True)
try:
positive, latent = MiniMaxH3ReferenceToVideo.execute(
clip=clip, vae=vae, prompt=prompt, width=width, height=height, length=length, ref_image_size="match",
ref_images={f"ref_image_{i}": p for i, p in enumerate(pictures, 1)}).args[:2]
self._free_gpu()
positive = MiniMaxH3AddGuide.execute(positive=positive, latent=latent, frame_idx=0, vae=vae, image=guide_video).args[0]
for f, frame in zip(placed, frames):
positive = MiniMaxH3AddGuide.execute(positive=positive, latent=latent, frame_idx=f, vae=vae, image=frame[None]).args[0]
finally:
self._free_gpu()
print(f"[BFS H3 keyframe] {mode}: {len(placed)} keyframes at frames {placed}; {len(pictures)} Pictures", flush=True)
return (self._until(model, keyframe_until, keyframe_until_drops.startswith("keyframes")), positive, latent, prompt, video,
pictures)
def _guide(self, clip, vae, guide_video, width, height, length, pictures, prompt_override, model, kw, mode=NORMALS):
"""Marked guide (subjects drawn as color-coded marks) as the only aligned guide, every
step; the Pictures in color order; no keyframes."""
from comfy_extras.nodes_minimax_h3 import MiniMaxH3AddGuide, MiniMaxH3ReferenceToVideo
if len(guide_video) != length:
raise ValueError(f"guide has {len(guide_video)} frames, length is {length}: connect Prepare's length.")
captions = {NORMALS: normals_caption, POSE: pose_caption, OVERLAY: overlay_caption, OUTLINE: outline_caption}
prompt = prompt_override.strip() or captions[mode](len(pictures))
print("[BFS H3 keyframe] guide prompt:\n" + prompt, flush=True)
try:
positive, latent = MiniMaxH3ReferenceToVideo.execute(
clip=clip, vae=vae, prompt=prompt, width=width, height=height, length=length, ref_image_size="match",
ref_images={f"ref_image_{i}": p for i, p in enumerate(pictures, 1)}).args[:2]
self._free_gpu()
positive = MiniMaxH3AddGuide.execute(positive=positive, latent=latent, frame_idx=0, vae=vae, image=guide_video).args[0]
finally:
self._free_gpu()
# the guide carries the background: it stays in every step (keyframe_until does not apply)
return self._patch_vram_estimate(model), positive, latent, prompt, guide_video, pictures
@staticmethod
def _until(model, keyframe_until, keep_source=False):
"""The model with the guides dropped below sigma keyframe_until (as Viggle Remake's RemakeKeyframeUntil).
keep_source: drop only the keyframes; the source video (the first Add Guide, see encode) stays."""
if model is None:
return None
BFSH3KeyframeConditioning._patch_vram_estimate(model)
import comfy.patcher_extension
until = float(keyframe_until)
def release(executor, x, timestep, context, transformer_options, minimax_payload=None, **kwargs):
p = minimax_payload or {}
if p.get("keyframes") and float(timestep.flatten()[0]) / 1000 < until:
refs = p.get("refs") or []
kept = list(p["keyframes"][:1]) if keep_source else []
minimax_payload = dict(p, keyframes=kept or None, layout=None,
cond_video_latents=[k["latent"] for k in kept if k.get("latent") is not None] +
[r["latent"] for r in refs if "latent" in r],
cond_audio_latents=[k["audio_latent"] for k in kept if k.get("audio_latent") is not None] +
[r["audio_latent"] for r in refs if r.get("audio_latent") is not None])
return executor(x, timestep, context, transformer_options, minimax_payload=minimax_payload, **kwargs)
m = model.clone()
m.add_wrapper_with_key(comfy.patcher_extension.WrappersMP.DIFFUSION_MODEL, "bfs_h3_keyframe_until", release)
return m
NODE_CLASS_MAPPINGS = {"BFSH3KeyframeConditioning": BFSH3KeyframeConditioning}
NODE_DISPLAY_NAME_MAPPINGS = {"BFSH3KeyframeConditioning": "BFS H3 Character Swap Conditioning"}
+151
View File
@@ -0,0 +1,151 @@
"""BFS Keyframe Marks: the tracked person's mask at every keyframe, and the keyframe frames marked for an image edit.
Pick the person once (BFS Segment Picker on a guide frame), track it over the whole guide (Run SAM3 Video Track ->
SAM3 Track to Mask), then this node takes the mask at each keyframe of a BFS Keyframe Planner keyframe_spec and tints
that frame (the Colored Mark convention: magenta at 30 % marks the person an image edit replaces). A keyframe whose mask is empty (the track lost the person, e.g. after a camera cut) is reported, so
it can be marked again by hand.
"""
import json
import torch
import torch.nn.functional as F
def parse_color(text):
s = str(text or "").strip().lstrip("#")
if len(s) == 3:
s = "".join(ch * 2 for ch in s)
try:
return int(s[0:2], 16), int(s[2:4], 16), int(s[4:6], 16)
except (ValueError, IndexError):
return 255, 0, 255
def grow(mask, pixels):
"""[N,H,W] mask dilated (pixels > 0) or eroded (pixels < 0) by a square of that radius."""
if not pixels:
return mask
r = abs(int(pixels)); k = 2 * r + 1
m = mask[:, None].float()
if pixels > 0:
return F.max_pool2d(m, k, 1, r)[:, 0]
return -F.max_pool2d(-m, k, 1, r)[:, 0]
class BFSKeyframeMarks:
DESCRIPTION = __doc__.split("\n\n")[0]
@classmethod
def INPUT_TYPES(cls):
return {"required": {
"guide_video": ("IMAGE", {"tooltip": "The guide frames the track was made on (same frames as the Planner's guide)."}),
"masks": ("MASK", {"tooltip": "The person to replace: one mask per guide frame, or one per keyframe (Segment Picker batch_masks on the Planner's source_frames)."}),
"keyframe_spec": ("STRING", {"forceInput": True, "tooltip": "BFS Keyframe Planner keyframe_spec: which frames are keyframes."}),
"mark_color": ("STRING", {"default": "#FF00FF", "tooltip": "Tint of the person in marked_frames (hex)."}),
"mark_opacity": ("FLOAT", {"default": 0.3, "min": 0.0, "max": 1.0, "step": 0.05,
"tooltip": "How strongly the person is tinted (0.3 = the Colored Mark default)."}),
"grow_pixels": ("INT", {"default": 0, "min": -64, "max": 64,
"tooltip": "Grow (positive) or shrink (negative) each mask before tinting, e.g. to cover hair."}),
"threshold": ("FLOAT", {"default": 0.5, "min": 0.0, "max": 1.0, "step": 0.05, "tooltip": "Mask values above this count as the person."}),
}}
RETURN_TYPES = ("IMAGE", "MASK", "IMAGE", "STRING")
RETURN_NAMES = ("marked_frames", "masks", "source_frames", "report")
OUTPUT_TOOLTIPS = ("The keyframe frames with the person tinted, one per keyframe (in keyframe_spec order).",
"The person's mask at each keyframe.",
"The untouched keyframe frames.",
"JSON: per keyframe its frame, time, mask area and whether the mask is empty.")
FUNCTION = "mark"
CATEGORY = "BFS/MiniMax H3"
def mark(self, guide_video, masks, keyframe_spec, mark_color, mark_opacity, grow_pixels, threshold):
spec = json.loads(keyframe_spec or "{}")
frames = [int(f) for f in spec.get("frames", [])]
if not frames:
raise ValueError("keyframe_spec lists no keyframes: add them in BFS Keyframe Planner.")
n, h, w = guide_video.shape[:3]
if masks.ndim == 2:
masks = masks[None]
per_keyframe = masks.shape[0] == len(frames) and masks.shape[0] != n # masks made on the Planner's source_frames
if not per_keyframe and masks.shape[0] != n:
raise ValueError(f"{masks.shape[0]} masks: give one per guide frame ({n}) or one per keyframe ({len(frames)}).")
if masks.shape[1:] != (h, w):
masks = F.interpolate(masks[:, None].float(), size=(h, w), mode="bilinear", align_corners=False)[:, 0]
color = torch.tensor(parse_color(mark_color), dtype=guide_video.dtype) / 255.0
fps = float(spec.get("fps", 24.0))
src, out, kept, report = [], [], [], []
for i, f in enumerate(frames):
f = max(0, min(n - 1, f))
j = i if per_keyframe else f
m = grow((masks[j:j + 1] > threshold).float(), grow_pixels).clamp(0, 1)
img = guide_video[f:f + 1, ..., :3]
a = (m[0].to(img.dtype) * float(mark_opacity))[..., None]
out.append(img * (1 - a) + color * a)
src.append(img); kept.append(m)
area = float(m.mean())
report.append({"frame": f, "time": round(f / fps, 3), "area": round(area, 4), "empty": area < 1e-4})
lost = [r["frame"] for r in report if r["empty"]]
if len(lost) == len(report):
raise ValueError("No person selected: click the person to replace in the Video Segment Picker, then run again.")
if lost:
print(f"[BFS Keyframe Marks] no mask at keyframe frame(s) {lost}: mark the person there by hand", flush=True)
return torch.cat(out), torch.cat(kept), torch.cat(src), json.dumps({"keyframes": report, "lost": lost})
class BFSTrackFromFrame:
"""Track a person over the whole guide from the frame they were picked on (not only frame 0): SAM 3 tracks
forward from that frame and backward (the frames before it, reversed), and the two halves are joined. With
anchor_frame -1 the anchor is the first keyframe of keyframe_spec, so feed the Planner's source_frames to the
Segment Picker and its mask lines up."""
@classmethod
def INPUT_TYPES(cls):
return {"required": {
"guide_video": ("IMAGE", {"tooltip": "The guide frames (the same the Planner gets)."}),
"sam3_model": ("MODEL", {"tooltip": "SAM 3.1 MODEL (Load Checkpoint with sam3.1_multiplex_fp16.safetensors)."}),
"mask": ("MASK", {"tooltip": "The person on the anchor frame (BFS Segment Picker mask)."}),
"anchor_frame": ("INT", {"default": -1, "min": -1, "max": 100000,
"tooltip": "Guide frame the mask was made on; -1 = the first keyframe of keyframe_spec."}),
}, "optional": {
"keyframe_spec": ("STRING", {"forceInput": True, "tooltip": "BFS Keyframe Planner keyframe_spec (for anchor_frame -1)."}),
}}
RETURN_TYPES = ("MASK", "INT")
RETURN_NAMES = ("masks", "anchor_frame")
OUTPUT_TOOLTIPS = ("One mask per guide frame.", "The anchor frame used.")
FUNCTION = "track"
CATEGORY = "BFS/MiniMax H3"
@staticmethod
def _run(frames, model, mask):
from comfy_extras.nodes_sam3 import SAM3_VideoTrack, SAM3_TrackToMask
data = SAM3_VideoTrack.execute(images=frames, model=model, initial_mask=mask, max_objects=1).args[0]
return SAM3_TrackToMask.execute(track_data=data, object_indices="").args[0]
def track(self, guide_video, sam3_model, mask, anchor_frame, keyframe_spec=None):
n, h, w = guide_video.shape[:3]
k = int(anchor_frame)
if k < 0:
frames = json.loads(keyframe_spec or "{}").get("frames", [])
if not frames:
raise ValueError("anchor_frame -1 needs a keyframe_spec with at least one keyframe.")
k = int(frames[0])
k = max(0, min(n - 1, k))
m = mask[:1] if mask.ndim == 3 else mask[None]
if m.shape[1:] != (h, w):
m = F.interpolate(m[:, None].float(), size=(h, w), mode="bilinear", align_corners=False)[:, 0]
m = (m > 0.5).float()
fwd = self._run(guide_video[k:], sam3_model, m) # frames k .. n-1
parts = [fwd]
if k > 0:
bwd = self._run(guide_video[:k + 1].flip(0), sam3_model, m) # frames k .. 0
parts = [bwd.flip(0)[:k], fwd]
out = torch.cat([p.to(guide_video.device, torch.float32) for p in parts])
if out.shape[1:] != (h, w):
out = F.interpolate(out[:, None], size=(h, w), mode="bilinear", align_corners=False)[:, 0]
print(f"[BFS Track From Frame] anchor frame {k}: tracked {k} frames back, {n - k} forward", flush=True)
return out, k
NODE_CLASS_MAPPINGS = {"BFSKeyframeMarks": BFSKeyframeMarks, "BFSTrackFromFrame": BFSTrackFromFrame}
NODE_DISPLAY_NAME_MAPPINGS = {"BFSKeyframeMarks": "BFS Keyframe Marks", "BFSTrackFromFrame": "BFS Track From Frame (SAM 3)"}
+148
View File
@@ -0,0 +1,148 @@
"""BFS Keyframe Planner: pick keyframe times on the guide video, drop in edited frames and reference pictures.
The node shows the guide on a small player with a timeline (frames come from its last run: queue it once to load
them). "+ Keyframe" marks the current frame; each keyframe card holds the source frame and, optionally, the edited
frame for that time (drop an image on the card or upload it). Reference pictures (up to 4) are dropped or uploaded in
the node too, or connected as inputs.
Two ways to use it:
- with edited frames: wire keyframes + keyframe_spec + pictures into BFS H3 Keyframe Swap Conditioning;
- without: use it to choose the times; source_frames are the original frames there, ready for an image edit
(e.g. Segment Picker + the Qwen 2.1 magenta body swap), and the edits come back into the conditioning node.
All state lives in the hidden `plan` STRING widget (JSON), saved with the workflow:
{"keyframes": [{"frame": 53, "edit": "bfs_keyframes/abc.png"}, {"frame": 96, "edit": null}],
"pictures": ["bfs_keyframes/ref1.png"]}
Image paths are relative to ComfyUI's input folder (the panel uploads there).
"""
from __future__ import annotations
import hashlib
import json
import os
import numpy as np
import torch
from PIL import Image
try:
import folder_paths
except Exception: # tests run without ComfyUI
folder_paths = None
MAX_PICTURES = 4
STRIP_MAX_FRAMES = 480 # the panel's filmstrip: at most this many frames (evenly spread over longer guides)
STRIP_WIDTH = 384
def parse_plan(text):
try:
p = json.loads(text or "{}") or {}
except Exception:
p = {}
kfs = []
for k in p.get("keyframes", []) if isinstance(p.get("keyframes"), list) else []:
try:
kfs.append({"frame": int(k["frame"]), "edit": k.get("edit") or None})
except Exception:
continue
pics = [x for x in p.get("pictures", []) if isinstance(x, str) and x] if isinstance(p.get("pictures"), list) else []
return {"keyframes": kfs, "pictures": pics[:MAX_PICTURES]}
def _input_path(rel):
base = folder_paths.get_input_directory() if folder_paths else "."
path = os.path.normpath(os.path.join(base, rel))
if not path.startswith(os.path.normpath(base) + os.sep):
raise ValueError(f"image path outside the input folder: {rel}")
if not os.path.isfile(path):
raise FileNotFoundError(f"keyframe planner image not found in input/: {rel}")
return path
def load_image(rel):
im = Image.open(_input_path(rel)).convert("RGB")
return torch.from_numpy(np.asarray(im, np.float32) / 255.0)[None]
def fit(image, h, w):
"""[1,H,W,C] -> [1,h,w,3]: center crop to the guide's aspect, then resize."""
import comfy.utils
x = image[:1, ..., :3].movedim(-1, 1)
return comfy.utils.common_upscale(x, w, h, "lanczos", "center").movedim(1, -1).clamp(0, 1)
def filmstrip(video, fps):
"""Small JPEGs of the guide in the temp folder, for the panel. Same video -> same files (hashed)."""
n = video.shape[0]
idx = np.unique(np.linspace(0, n - 1, min(n, STRIP_MAX_FRAMES)).round().astype(int))
h, w = video.shape[1:3]
sw = min(STRIP_WIDTH, w); sh = max(1, round(h * sw / w))
sample = video[:: max(1, n // 16), ::16, ::16, :3].float().cpu().numpy()
key = hashlib.sha1(np.ascontiguousarray(sample).tobytes() + f"{n}x{h}x{w}".encode()).hexdigest()[:16]
temp = folder_paths.get_temp_directory() if folder_paths else "."
sub = os.path.join("bfs_kfplan", key); out = os.path.join(temp, sub); os.makedirs(out, exist_ok=True)
files = []
for i in idx:
name = f"f{i:05d}.jpg"; path = os.path.join(out, name)
if not os.path.exists(path):
arr = (video[int(i), ..., :3].float().cpu().numpy() * 255).clip(0, 255).astype(np.uint8)
Image.fromarray(arr).resize((sw, sh), Image.BILINEAR).save(path, quality=82)
files.append(name)
return {"subfolder": sub, "type": "temp", "files": files, "frames": [int(i) for i in idx],
"count": int(n), "fps": float(fps), "width": int(w), "height": int(h)}
class BFSKeyframePlanner:
DESCRIPTION = __doc__.split("\n\n")[0]
@classmethod
def INPUT_TYPES(cls):
opt = {f"picture_{j}": ("IMAGE", {"tooltip": f"Reference picture {j}; overrides the one dropped in the panel."})
for j in range(1, MAX_PICTURES + 1)}
return {
"required": {
"guide_video": ("IMAGE", {"tooltip": "The guide frames (e.g. BFS H3 Prepare Native Video)."}),
"fps": ("FLOAT", {"default": 24.0, "min": 1.0, "max": 120.0, "step": 0.5}),
"plan": ("STRING", {"default": "{}", "multiline": True, "tooltip": "Set by the panel: keyframe frames, edited frames and pictures."}),
},
"optional": opt,
"hidden": {"unique_id": "UNIQUE_ID"},
}
RETURN_TYPES = ("IMAGE", "IMAGE", "IMAGE", "STRING") + ("IMAGE",) * MAX_PICTURES
RETURN_NAMES = ("guide_video", "source_frames", "keyframes", "keyframe_spec") + tuple(f"picture_{j}" for j in range(1, MAX_PICTURES + 1))
OUTPUT_TOOLTIPS = ("The guide, unchanged.",
"The original guide frames at the keyframe times (one per keyframe), e.g. to edit them.",
"The edited frame of every keyframe, at the guide size; a keyframe without one gives its source frame.",
"JSON: frames, times (s), fps and which keyframes have an edited frame.") + \
tuple(f"Reference picture {j} (None when not set)." for j in range(1, MAX_PICTURES + 1))
FUNCTION = "plan_keyframes"
CATEGORY = "BFS/MiniMax H3"
OUTPUT_NODE = True
def plan_keyframes(self, guide_video, fps, plan, unique_id=None, **kw):
n, h, w = guide_video.shape[0], guide_video.shape[1], guide_video.shape[2]
state = parse_plan(plan)
ui = {"bfs_kfplan": [filmstrip(guide_video, fps)]}
kfs = sorted({max(0, min(n - 1, k["frame"])): k for k in state["keyframes"]}.items())
src, edit, spec = [], [], {"frames": [], "times": [], "edited": [], "fps": float(fps), "length": int(n)}
for f, k in kfs:
frame = guide_video[f:f + 1, ..., :3]
src.append(frame)
edit.append(fit(load_image(k["edit"]), h, w).to(frame) if k["edit"] else frame)
spec["frames"].append(int(f)); spec["times"].append(round(f / float(fps), 4)); spec["edited"].append(bool(k["edit"]))
if not kfs:
print("[BFS Keyframe Planner] no keyframes yet: add them in the panel (outputs hold the first frame)", flush=True)
src = edit = [guide_video[:1, ..., :3]]
pictures = []
for j in range(1, MAX_PICTURES + 1):
p = kw.get(f"picture_{j}")
if p is None and j <= len(state["pictures"]):
p = load_image(state["pictures"][j - 1])
pictures.append(p)
return {"ui": ui, "result": (guide_video, torch.cat(src), torch.cat(edit), json.dumps(spec), *pictures)}
NODE_CLASS_MAPPINGS = {"BFSKeyframePlanner": BFSKeyframePlanner}
NODE_DISPLAY_NAME_MAPPINGS = {"BFSKeyframePlanner": "BFS Keyframe Planner"}
+438
View File
@@ -0,0 +1,438 @@
"""BFS Segment Picker: segment an image into objects with SAM 3.1, pick them with the mouse, get their masks.
Connect SAM 3.1 from a Load Checkpoint node (sam3.1_multiplex_fp16.safetensors gives its MODEL and CLIP). The node
segments the first image of its input automatically (one SAM 3 text prompt per concept: "person, animal" by
default) and shows every object on a canvas in the node. Click objects to select them; the outputs are the mask of the
selection, one mask per selected object, the image with the selection tinted (the Colored Mark convention: magenta at
30 % marks the subject a body swap replaces) and a numbered overlay of everything found.
When the automatic pass misses something, add it in the panel: type what to find (a SAM 3 text prompt, e.g. "red
umbrella"), place keep / exclude points on it, or paint it with the brush (to grow the mark past what SAM found, e.g.
over hair). Added objects appear at once (the panel asks the server through
/bfs/segpick/*), and the workflow stores how each one was made, so a reload or another machine segments them again.
All state lives in the hidden `selection` STRING widget (JSON):
{"selected": ["a3", "t1.0"], "added": [{"id": "t1", "text": "red umbrella"},
{"id": "p2", "points": [{"x": .41, "y": .62, "label": 1}]},
{"id": "b3", "image": "9f2c...", "strokes": [{"r": .02, "pts": [[.4, .3], [.45, .32]]}]}], "next": 4}
An entry with "remove": true (r<n> points, e<n> eraser strokes) is cut out of the selection instead of added to it.
Points are relative to the image (0..1), a stroke radius to its width. A brush entry belongs to the image it was painted
on (`image`, see image_key): on another image it is dropped, while text and point entries are segmented again. Ids: a<k> automatic objects (left to right),
t<n>.<k> the k-th match of a text entry, p<n> a point entry, b<n> a brush entry.
"""
from __future__ import annotations
import hashlib
import json
import os
import uuid
from collections import OrderedDict
import numpy as np
import torch
from PIL import Image, ImageDraw, ImageFont
try:
import folder_paths
except Exception: # tests run without ComfyUI
folder_paths = None
PALETTE = [(230, 70, 60), (45, 125, 225), (50, 180, 90), (235, 205, 35), (205, 55, 180), (30, 200, 205),
(240, 130, 35), (135, 75, 210), (120, 200, 60), (225, 95, 150), (80, 160, 230), (190, 150, 90)]
DEFAULT_CONCEPTS = "person, animal"
PREVIEW_SIDE = 1024 # long side of the images the panel shows (outputs keep full resolution)
MAX_SESSIONS = 6
_SESSIONS: "OrderedDict[str, dict]" = OrderedDict()
# ---------------------------------------------------------------------------------------------------------------- state
def parse_state(text: str | None) -> dict:
"""The selection widget's JSON, tolerant of empty or hand-edited values."""
try:
s = json.loads(text) if text and text.strip() else {}
except (TypeError, ValueError):
s = {}
if not isinstance(s, dict):
s = {}
added = []
for e in s.get("added") or []:
if not isinstance(e, dict) or not e.get("id"):
continue
n = len(added)
if str(e.get("text") or "").strip():
added.append({"id": str(e["id"]), "text": str(e["text"]).strip()})
elif e.get("points"):
pts = [{"x": float(p["x"]), "y": float(p["y"]), "label": 1 if p.get("label", 1) else 0}
for p in e["points"] if isinstance(p, dict) and "x" in p and "y" in p]
if any(p["label"] for p in pts):
added.append({"id": str(e["id"]), "points": pts})
elif e.get("strokes"):
strokes = [{"r": float(st["r"]), "pts": [[float(x), float(y)] for x, y in st["pts"]]}
for st in e["strokes"] if isinstance(st, dict) and st.get("pts")]
if strokes:
added.append({"id": str(e["id"]), "image": str(e.get("image") or ""), "strokes": strokes})
if e.get("remove") and len(added) > n:
added[-1]["remove"] = True
selected = [str(i) for i in s.get("selected") or [] if isinstance(i, (str, int))]
return {"selected": list(dict.fromkeys(selected)), "added": added, "next": int(s.get("next") or len(added) + 1)}
def split_concepts(text: str) -> list[str]:
return list(dict.fromkeys(c.strip() for c in str(text or "").replace("\n", ",").split(",") if c.strip()))
def entry_key(entry: dict) -> str:
"""What an added entry asks SAM for: an entry whose text or points change is segmented again."""
return json.dumps({k: entry[k] for k in ("text", "points", "strokes", "remove") if k in entry}, sort_keys=True)
# ------------------------------------------------------------------------------------------------------------ geometry
def box_of(mask: torch.Tensor) -> list[int]:
ys, xs = torch.nonzero(mask, as_tuple=True)
if len(xs) == 0:
return [0, 0, 0, 0]
x0, y0, x1, y1 = int(xs.min()), int(ys.min()), int(xs.max()) + 1, int(ys.max()) + 1
return [x0, y0, x1 - x0, y1 - y0]
def dedupe(found: list[tuple[torch.Tensor, dict]], overlap: float = .8) -> list[tuple[torch.Tensor, dict]]:
"""Drop a mask that mostly repeats an earlier one (the animal prompt can find a person again, and the reverse)."""
kept: list[tuple[torch.Tensor, dict]] = []
for m, info in found:
a = int(m.sum())
if any(int((m & o).sum()) > overlap * min(a, int(o.sum())) for o, _ in kept):
continue
kept.append((m, info))
return kept
def compose(masks: list[torch.Tensor], h: int, w: int) -> tuple[torch.Tensor, torch.Tensor]:
"""(union [1, h, w], individual [n, h, w]) as float masks; an empty selection gives zero masks."""
if not masks:
z = torch.zeros(1, h, w)
return z, z.clone()
stack = torch.stack([m.float() for m in masks])
return (stack.amax(0, keepdim=True)), stack
def paint(strokes: list[dict], h: int, w: int) -> torch.Tensor:
"""Brush strokes (relative points, radius relative to the width) as a bool mask [h, w]."""
img = Image.new("L", (w, h), 0)
draw = ImageDraw.Draw(img)
for st in strokes:
r = max(1., st["r"] * w)
pts = [(x * w, y * h) for x, y in st["pts"]]
if len(pts) > 1:
draw.line(pts, fill=255, width=round(2 * r), joint="curve")
for x, y in pts:
draw.ellipse([x - r, y - r, x + r, y + r], fill=255)
return torch.from_numpy(np.array(img) > 127)
def tint(image: torch.Tensor, mask: torch.Tensor, color: tuple[int, int, int], opacity: float) -> torch.Tensor:
"""image [1, h, w, 3] with the mask's pixels blended toward the color."""
c = torch.tensor(color, dtype=image.dtype) / 255.
a = (mask[0].to(image.dtype) * float(opacity))[..., None]
out = image.clone()
out[0, ..., :3] = image[0, ..., :3] * (1 - a) + c * a
return out
def parse_color(text: str) -> tuple[int, int, int]:
s = str(text or "").strip().lstrip("#")
if len(s) == 3:
s = "".join(ch * 2 for ch in s)
try:
return int(s[0:2], 16), int(s[2:4], 16), int(s[4:6], 16)
except (ValueError, IndexError):
return 255, 0, 255
# ----------------------------------------------------------------------------------------------------------------- SAM
def _text_cond(clip, text: str, max_detections: int):
cond = clip.encode_from_tokens_scheduled(clip.tokenize(text))
# SAM3_Detect keeps one detection per text prompt unless the prompt asks for more (the multi-prompt form)
return [[cond[0][0], {**cond[0][1], "sam3_multi_cond": [{"cond": cond[0][0], "attention_mask": cond[0][1].get("attention_mask"),
"max_detections": int(max_detections)}]}]]
def detect_text(sam: tuple, frame: torch.Tensor, text: str, threshold: float, max_detections: int) -> list[tuple[torch.Tensor, dict]]:
from comfy_extras.nodes_sam3 import SAM3_Detect
model, clip = sam
h, w = frame.shape[1:3]
# ComfyUI can materialize SAM weights as inference tensors. Its detector must then
# run under inference_mode too; otherwise autograd asks those weights for a version counter.
with torch.inference_mode():
masks, boxes = SAM3_Detect.execute(model=model, image=frame, conditioning=_text_cond(clip, text, max_detections),
threshold=float(threshold), refine_iterations=1, individual_masks=True).args
masks = masks.cpu() > .5 if masks.numel() else torch.zeros(0, h, w, dtype=torch.bool)
boxes = boxes[0] if boxes else []
return [(m, {"score": round(float(b.get("score", 0)), 3)}) for m, b in zip(masks, boxes)]
def detect_points(sam: tuple, frame: torch.Tensor, points: list[dict], threshold: float) -> torch.Tensor:
from comfy_extras.nodes_sam3 import SAM3_Detect
model, _ = sam
h, w = frame.shape[1:3]
pos = [{"x": p["x"] * w, "y": p["y"] * h} for p in points if p["label"]]
neg = [{"x": p["x"] * w, "y": p["y"] * h} for p in points if not p["label"]]
with torch.inference_mode():
masks = SAM3_Detect.execute(model=model, image=frame, positive_coords=json.dumps(pos), negative_coords=json.dumps(neg),
threshold=float(threshold), refine_iterations=2).args[0]
return (masks[0].cpu() > .5) if masks.numel() else torch.zeros(h, w, dtype=torch.bool)
# ------------------------------------------------------------------------------------------------------------- session
def image_key(frame: torch.Tensor) -> str:
"""Which image this is, from a 64x64 thumbnail and the size: what a brush entry is tied to."""
small = torch.nn.functional.interpolate(frame[..., :3].movedim(-1, 1).float(), size=(64, 64), mode="area")
digest = hashlib.sha1((small * 255).round().byte().numpy().tobytes())
digest.update(str(tuple(frame.shape)).encode())
return digest.hexdigest()[:12]
def _session_key(frame: torch.Tensor, spec: dict, sam: tuple) -> str:
digest = hashlib.sha1(image_key(frame).encode())
digest.update(json.dumps(spec, sort_keys=True).encode())
digest.update(str(id(getattr(sam[0], "model", sam[0]))).encode()) # another SAM checkpoint segments again
return digest.hexdigest()[:16]
def _remember(sid: str, sess: dict) -> dict:
_SESSIONS[sid] = sess
_SESSIONS.move_to_end(sid)
while len(_SESSIONS) > MAX_SESSIONS:
_SESSIONS.popitem(last=False)
return sess
def auto_objects(sam: tuple, frame: torch.Tensor, concepts: list[str], threshold: float, max_per_concept: int, min_area: float) -> list[dict]:
h, w = frame.shape[1:3]
found = []
for concept in concepts:
for m, info in detect_text(sam, frame, concept, threshold, max_per_concept):
area = int(m.sum())
if area < max(64, min_area * h * w):
continue
# Generic "objects" may segment the entire scene. Such a mask is
# unsafe as an auto subject: it paints the background into the guide.
if concept.strip().casefold() in {"object", "objects"} and area >= .25 * h * w:
_, _, bw, bh = box_of(m)
if bw >= .98 * w and bh >= .98 * h:
continue
found.append((m, {**info, "label": concept}))
found = dedupe(found)
found.sort(key=lambda f: box_of(f[0])[0]) # left to right, so "a1" reads naturally
return [{"id": f"a{k + 1}", "label": info["label"], "source": "auto", "score": info["score"], "mask": m}
for k, (m, info) in enumerate(found)]
def entry_objects(sam: tuple, frame: torch.Tensor, entry: dict, threshold: float, max_per_concept: int) -> list[dict]:
objs = _entry_objects(sam, frame, entry, threshold, max_per_concept)
if entry.get("remove"):
for o in objs:
o["remove"], o["label"] = True, "eraser" if entry.get("strokes") else "remove"
return objs
def _entry_objects(sam: tuple, frame: torch.Tensor, entry: dict, threshold: float, max_per_concept: int) -> list[dict]:
if entry.get("text"):
hits = dedupe(detect_text(sam, frame, entry["text"], threshold, max_per_concept))
hits.sort(key=lambda f: box_of(f[0])[0])
return [{"id": f"{entry['id']}.{k}", "label": entry["text"], "source": "text", "entry": entry["id"],
"score": info["score"], "mask": m} for k, (m, info) in enumerate(hits)]
if entry.get("strokes"):
m = paint(entry["strokes"], *frame.shape[1:3])
return [{"id": entry["id"], "label": "brush", "source": "brush", "entry": entry["id"], "score": None, "mask": m}] if m.any() else []
m = detect_points(sam, frame, entry["points"], threshold)
if int(m.sum()) < 16:
return []
return [{"id": entry["id"], "label": "points", "source": "points", "entry": entry["id"], "score": None, "mask": m}]
def session_for(sam: tuple, frame: torch.Tensor, spec: dict) -> tuple[str, dict]:
sid = _session_key(frame, spec, sam)
if sid in _SESSIONS:
_SESSIONS.move_to_end(sid)
_SESSIONS[sid]["sam"] = sam
return sid, _SESSIONS[sid]
objs = auto_objects(sam, frame, split_concepts(spec["concepts"]), spec["threshold"], spec["max_per_concept"], spec["min_area"])
# the panel's /bfs/segpick/add segments with the SAM the node last ran with
return sid, _remember(sid, {"frame": frame.cpu(), "image_key": image_key(frame), "spec": spec, "auto": objs, "entries": {}, "sam": sam})
def ensure_entry(sess: dict, entry: dict) -> list[dict]:
"""The objects of an added entry, segmented once per (entry id, what it asks for)."""
key = entry_key(entry)
have = sess["entries"].get(entry["id"])
if have is None or have["key"] != key:
spec = sess["spec"]
have = {"key": key, "objects": entry_objects(sess["sam"], sess["frame"], entry, spec["threshold"], spec["max_per_concept"])}
sess["entries"][entry["id"]] = have
return have["objects"]
def all_objects(sess: dict, state: dict) -> list[dict]:
objs = list(sess["auto"])
for e in state["added"]:
if e.get("strokes") and e["image"] and e["image"] != sess["image_key"]:
continue # painted on another image
objs += ensure_entry(sess, e)
return objs
# --------------------------------------------------------------------------------------------------------- panel files
def _temp_dir() -> str:
base = folder_paths.get_temp_directory() if folder_paths else os.path.join(os.path.dirname(__file__), ".temp")
d = os.path.join(base, "bfs_segpick")
os.makedirs(d, exist_ok=True)
return d
def _preview_size(h: int, w: int) -> tuple[int, int]:
s = min(1.0, PREVIEW_SIDE / max(h, w))
return max(1, round(w * s)), max(1, round(h * s))
def _save(img: Image.Image, prefix: str) -> dict:
name = f"{prefix}_{uuid.uuid4().hex[:10]}.png"
img.save(os.path.join(_temp_dir(), name), compress_level=1)
return {"filename": name, "subfolder": "bfs_segpick", "type": "temp"}
def panel_object(o: dict, size: tuple[int, int], h: int, w: int) -> dict:
"""What the panel needs about one object: its mask as a small PNG (white on transparent) and where it is."""
m = Image.fromarray(o["mask"].numpy().astype(np.uint8) * 255).resize(size, Image.NEAREST)
rgba = Image.new("RGBA", size, (255, 255, 255, 0))
rgba.putalpha(m)
x, y, bw, bh = box_of(o["mask"])
if "_file" not in o:
o["_file"] = _save(rgba, f"m_{o['id']}")
return {"id": o["id"], "label": o["label"], "source": o["source"], "entry": o.get("entry"), "score": o["score"],
"remove": bool(o.get("remove")), "area": round(float(o["mask"].float().mean()), 5), "box": [round(x / w, 4), round(y / h, 4), round(bw / w, 4), round(bh / h, 4)],
"mask": o["_file"]}
def panel_data(sid: str, sess: dict, objects: list[dict]) -> dict:
frame = sess["frame"]
h, w = frame.shape[1:3]
size = _preview_size(h, w)
if "_image" not in sess:
rgb = (frame[0, ..., :3].clamp(0, 1) * 255).byte().numpy()
sess["_image"] = _save(Image.fromarray(rgb).resize(size, Image.LANCZOS), "img")
return {"session": sid, "image_key": sess["image_key"], "width": int(w), "height": int(h), "image": sess["_image"],
"objects": [panel_object(o, size, h, w) for o in objects]}
def overlay(frame: torch.Tensor, objects: list[dict], selected: set[str]) -> torch.Tensor:
rgb = (frame[0, ..., :3].clamp(0, 1) * 255).byte().numpy().astype(np.float32)
h, w = rgb.shape[:2]
for n, o in enumerate(objects):
m = o["mask"].numpy()
a = .6 if o["id"] in selected else .3
rgb[m] = rgb[m] * (1 - a) + np.array(PALETTE[n % len(PALETTE)], np.float32) * a
canvas = Image.fromarray(rgb.clip(0, 255).astype(np.uint8))
draw = ImageDraw.Draw(canvas)
size = max(14, h // 28)
try:
font = ImageFont.load_default(size=size)
except TypeError:
font = ImageFont.load_default()
for n, o in enumerate(objects):
x, y, bw, bh = box_of(o["mask"])
color = PALETTE[n % len(PALETTE)]
if o["id"] in selected:
draw.rectangle([x, y, x + bw, y + bh], outline=color, width=max(2, h // 300))
draw.text((x + 4, y + 4), o["id"], fill=(255, 255, 255), font=font, stroke_width=2, stroke_fill=(0, 0, 0))
return torch.from_numpy(np.asarray(canvas, np.float32) / 255.)[None]
# ---------------------------------------------------------------------------------------------------------------- node
class BFSSegmentPicker:
DESCRIPTION = __doc__.split("\n\n")[0] + " Click objects on the canvas to select them; add missed ones by text, points or the brush."
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"image": ("IMAGE",),
"sam3_model": ("MODEL", {"tooltip": "SAM 3.1: MODEL of a Load Checkpoint node with sam3.1_multiplex_fp16.safetensors."}),
"sam3_clip": ("CLIP", {"tooltip": "SAM 3.1: CLIP of the same Load Checkpoint node; turns the concepts and Find texts into SAM prompts."}),
"concepts": ("STRING", {"default": DEFAULT_CONCEPTS, "tooltip": "What the automatic pass looks for, comma separated (one SAM 3 text prompt each), e.g. person, animal, car, chair. Empty = no automatic pass, only what you add."}),
"threshold": ("FLOAT", {"default": .4, "min": .05, "max": .95, "step": .05, "tooltip": "SAM 3 detection score threshold: lower finds more (and more false hits)."}),
"max_per_concept": ("INT", {"default": 8, "min": 1, "max": 32, "tooltip": "Most objects one concept or text prompt can return."}),
"min_area": ("FLOAT", {"default": .002, "min": 0., "max": .5, "step": .001, "tooltip": "Automatic objects smaller than this share of the image are dropped."}),
"mark_color": ("STRING", {"default": "#FF00FF", "tooltip": "Tint of the selection in the marked output (hex)."}),
"mark_opacity": ("FLOAT", {"default": .3, "min": 0., "max": 1., "step": .05, "tooltip": "How strongly the marked output tints the selection (0.3 = the Colored Mark default)."}),
"selection": ("STRING", {"default": "{}", "multiline": True, "tooltip": "Set by the canvas: the selected objects and the ones added by text or points."}),
},
"hidden": {"unique_id": "UNIQUE_ID"},
}
RETURN_TYPES = ("MASK", "MASK", "IMAGE", "IMAGE", "STRING")
RETURN_NAMES = ("mask", "masks", "marked", "overlay", "objects")
OUTPUT_TOOLTIPS = ("Union of the selected objects, minus the selected remove / eraser ones (one mask).", "One mask per selected object (remove ones already cut out), in selection order.",
"The image with the selection tinted (mark_color, mark_opacity).", "Every object found, numbered; the selection stronger.",
"JSON: size, selected ids and every object's id, label, score, box (relative x, y, w, h) and area.")
FUNCTION = "pick"
CATEGORY = "BFS/Mask"
OUTPUT_NODE = True
def pick(self, image, sam3_model, sam3_clip, concepts, threshold, max_per_concept, min_area, mark_color, mark_opacity, selection, unique_id=None):
frame = image[:1, ..., :3].float().cpu()
h, w = frame.shape[1:3]
spec = {"concepts": ", ".join(split_concepts(concepts)), "threshold": round(float(threshold), 4),
"max_per_concept": int(max_per_concept), "min_area": round(float(min_area), 5)}
state = parse_state(selection)
sid, sess = session_for((sam3_model, sam3_clip), frame, spec)
objects = all_objects(sess, state)
by_id = {o["id"]: o for o in objects}
chosen = [by_id[i] for i in state["selected"] if i in by_id]
cut = [o["mask"] for o in chosen if o.get("remove")]
cut = torch.stack(cut).any(0) if cut else torch.zeros(h, w, dtype=torch.bool)
mask, masks = compose([o["mask"] & ~cut for o in chosen if not o.get("remove")], h, w)
marked = tint(frame, mask, parse_color(mark_color), mark_opacity)
info = {"width": int(w), "height": int(h), "selected": [o["id"] for o in chosen],
"objects": [{k: v for k, v in panel_object(o, _preview_size(h, w), h, w).items() if k != "mask"} for o in objects]}
return {"ui": {"bfs_segpick": [panel_data(sid, sess, objects)]},
"result": (mask, masks, marked, overlay(frame, objects, set(info["selected"])), json.dumps(info))}
# -------------------------------------------------------------------------------------------------------------- routes
def _routes():
try:
from aiohttp import web
from server import PromptServer
except Exception:
return
@PromptServer.instance.routes.post("/bfs/segpick/add")
async def add(request):
"""Segment one added entry ({session, entry: {id, text} | {id, points}}) on the session's image now."""
body = await request.json()
sess = _SESSIONS.get(str(body.get("session") or ""))
if sess is None:
return web.json_response({"error": "This image is no longer in memory: run the node once, then add again."}, status=404)
state = parse_state(json.dumps({"added": [body.get("entry") or {}]}))
if not state["added"]:
return web.json_response({"error": "Give a text prompt, at least one keep point or a brush stroke."}, status=400)
try:
objs = ensure_entry(sess, state["added"][0])
except Exception as e: # noqa: BLE001 - shown in the panel
return web.json_response({"error": f"SAM 3 failed: {e}"}, status=500)
h, w = sess["frame"].shape[1:3]
size = _preview_size(h, w)
return web.json_response({"objects": [panel_object(o, size, h, w) for o in objs]})
_routes()
NODE_CLASS_MAPPINGS = {"BFSSegmentPicker": BFSSegmentPicker}
NODE_DISPLAY_NAME_MAPPINGS = {"BFSSegmentPicker": "BFS Segment Picker (SAM 3)"}
+14 -5
View File
@@ -1685,7 +1685,16 @@ async def _off_loop(fn):
"""Run slow work in a thread so the server keeps answering (and the panel gets progress events).""" """Run slow work in a thread so the server keeps answering (and the panel gets progress events)."""
import asyncio import asyncio
_outside_prompt() _outside_prompt()
return await asyncio.get_running_loop().run_in_executor(None, fn) return await asyncio.get_running_loop().run_in_executor(None, lambda: _like_a_node(fn))
def _like_a_node(fn):
"""Run a panel button's model work under torch.inference_mode(), as ComfyUI runs every node (execution.py). After
the workflow ran once, the VLM and SAM keep tensors made in inference mode; using them outside it raises
"Inference tensors do not track version counter" (issue #14)."""
import torch
with torch.inference_mode():
return fn()
def _outside_prompt() -> None: def _outside_prompt() -> None:
@@ -5850,10 +5859,10 @@ try:
preview_color = (mask_override or {}).get("preview_color") or ( preview_color = (mask_override or {}).get("preview_color") or (
seg.get("color_guide_color") if seg.get("color_guide") else "#ff283c") seg.get("color_guide_color") if seg.get("color_guide") else "#ff283c")
out = await asyncio.get_running_loop().run_in_executor( out = await asyncio.get_running_loop().run_in_executor(
None, lambda: mask_preview(path, a, seg["start"], seg["end"] - seg["start"], spec, None, lambda: _like_a_node(lambda: mask_preview(path, a, seg["start"], seg["end"] - seg["start"], spec,
int(body.get("count", 6)), int(body.get("count", 6)),
preview_color, preview_color,
seg.get("color_guide_opacity") if seg.get("color_guide") else 0.55)) seg.get("color_guide_opacity") if seg.get("color_guide") else 0.55)))
except Exception as exc: # noqa: BLE001 except Exception as exc: # noqa: BLE001
return web.json_response({"error": f"{type(exc).__name__}: {exc}"}, status=500) return web.json_response({"error": f"{type(exc).__name__}: {exc}"}, status=500)
return web.json_response(out) return web.json_response(out)
@@ -5878,8 +5887,8 @@ try:
import asyncio import asyncio
_outside_prompt() _outside_prompt()
out = await asyncio.get_running_loop().run_in_executor( out = await asyncio.get_running_loop().run_in_executor(
None, lambda: [dict(vlm_shot(clip, path, a, s["start"], s["end"], cfg), start=s["start"], end=s["end"]) None, lambda: _like_a_node(lambda: [dict(vlm_shot(clip, path, a, s["start"], s["end"], cfg),
for s in pick]) start=s["start"], end=s["end"]) for s in pick]))
except Exception as exc: # noqa: BLE001 except Exception as exc: # noqa: BLE001
return web.json_response({"error": f"{type(exc).__name__}: {exc}"}, status=500) return web.json_response({"error": f"{type(exc).__name__}: {exc}"}, status=500)
return web.json_response({"segs": out}) return web.json_response({"segs": out})
+28 -3
View File
@@ -67,6 +67,17 @@ def ref_pictures(reference) -> list:
return out[:MAX_REFS] return out[:MAX_REFS]
def save_stills(frames: dict, out_dir: str) -> dict:
"""{source: [1, H, W, 3] in 0..1} as lossless PNGs -> {source: file name}."""
from PIL import Image
names = {}
for k, v in frames.items():
names[k] = f"{k}.png"
Image.fromarray((v[0, ..., :3].float().clamp(0, 1).cpu().numpy() * 255).round().astype(np.uint8)).save(
os.path.join(out_dir, names[k]))
return names
def save_refs(reference, out_dir: str, height: int) -> list: def save_refs(reference, out_dir: str, height: int) -> list:
"""The reference pictures as PNGs no taller than the player (no upscale) -> file names.""" """The reference pictures as PNGs no taller than the player (no upscale) -> file names."""
from PIL import Image from PIL import Image
@@ -129,6 +140,8 @@ class BFSVideoCompare:
"size) or a batch; up to 12."}), "size) or a batch; up to 12."}),
"label_guide": ("STRING", {"default": "guide"}), "label_guide": ("STRING", {"default": "guide"}),
"label_target": ("STRING", {"default": "target"}), "label_target": ("STRING", {"default": "target"}),
"free_memory_after": ("BOOLEAN", {"default": True, "tooltip":
"After the workflow finishes, release cached node outputs and unloaded model weights from RAM. The next run reloads models."}),
}, },
} }
@@ -146,10 +159,10 @@ class BFSVideoCompare:
"where they differ, pixel by pixel.") "where they differ, pixel by pixel.")
def compare(self, guide, target, fps=24.0, audio=None, gain=2.0, size="guide", preview_height=720, def compare(self, guide, target, fps=24.0, audio=None, gain=2.0, size="guide", preview_height=720,
label_guide="guide", label_target="target", reference=None): label_guide="guide", label_target="target", reference=None, free_memory_after=True):
one = lambda v: v[0] if isinstance(v, list) and v else v one = lambda v: v[0] if isinstance(v, list) and v else v
fps, audio, gain, size, preview_height, label_guide, label_target = map( fps, audio, gain, size, preview_height, label_guide, label_target, free_memory_after = map(
one, (fps, audio, gain, size, preview_height, label_guide, label_target)) one, (fps, audio, gain, size, preview_height, label_guide, label_target, free_memory_after))
guide, target = video_batches(guide, "guide"), video_batches(target, "target") guide, target = video_batches(guide, "guide"), video_batches(target, "target")
ng, nt = sum(x.shape[0] for x in guide), sum(x.shape[0] for x in target) ng, nt = sum(x.shape[0] for x in guide), sum(x.shape[0] for x in target)
if len(guide) > 1 or len(target) > 1: if len(guide) > 1 or len(target) > 1:
@@ -195,6 +208,9 @@ class BFSVideoCompare:
w.close() w.close()
diff_map = torch.cat(maps, 0) diff_map = torch.cat(maps, 0)
mae, mse = torch.cat(mae), torch.cat(mse) mae, mse = torch.cat(mae), torch.cat(mse)
# a single frame is a picture: lossless PNGs at the compared size, which the player's ⬇ save PNG downloads
stills = save_stills({"guide": video_chunk(guide, 0, 1, H, W), "target": video_chunk(target, 0, 1, H, W),
"difference": diff_map[:1]}, out_dir) if n == 1 else {}
mean = float(mae.mean()) mean = float(mae.mean())
worst, best = int(mae.argmax()), int(mae.argmin()) worst, best = int(mae.argmax()), int(mae.argmin())
report = "\n".join([ report = "\n".join([
@@ -210,7 +226,16 @@ class BFSVideoCompare:
ui = {"bfs_compare": [{**files, "fps": float(fps), "frames": n, "width": PW, "height": PH, ui = {"bfs_compare": [{**files, "fps": float(fps), "frames": n, "width": PW, "height": PH,
"labels": [label_guide, label_target], "mae": [round(float(x), 2) for x in mae], "labels": [label_guide, label_target], "mae": [round(float(x), 2) for x in mae],
"refs": refs, "refs": refs,
"stills": {k: {"filename": f, "subfolder": sub, "type": "temp"} for k, f in stills.items()},
"report": report, "notes": notes}]} "report": report, "notes": notes}]}
if free_memory_after:
try:
from server import PromptServer
except ImportError: # the comparison can also run outside ComfyUI
PromptServer = None
queue = getattr(getattr(PromptServer, "instance", None), "prompt_queue", None)
if queue is not None:
queue.set_flag("free_memory", True)
return {"ui": ui, "result": (diff_map, report, mean)} return {"ui": ui, "result": (diff_map, report, mean)}
File diff suppressed because it is too large Load Diff
+227
View File
@@ -0,0 +1,227 @@
"""BFS Workflow Loop: run any part of any workflow once per item, injecting the item (and per-iteration parameters)
into nodes you pick, and collect the results in order.
Connect the IMAGE the loop should produce to `result`, and the batch to iterate over to `items` (frames, keyframes,
marked frames...) and optionally `item_masks`. In the panel pick, among the nodes upstream of `result`:
- inject: a node whose output is replaced, on iteration i, by item i (e.g. a Load Image that stands for "the frame",
or a mask loader for "its mask"),
- params: widgets that take one value per iteration (a seed, a prompt line, a strength...), cycled if shorter.
On run the node does not compute `result` itself (lazy input): it expands one copy per iteration of the part of the graph
that depends on the injected nodes or the changed widgets; models and everything else are shared and run once.
State lives in the hidden `loop` STRING widget (JSON), saved with the workflow:
{"inject": [{"node": "6", "slot": 0, "source": "items"}],
"params": [{"node": "34", "widget": "noise_seed", "values": [1, 2, 3]}]}
"""
import json
import torch
import torch.nn.functional as F
from .bfs_xy_sweep import _cast
CATEGORY = "BFS/Loop"
MAX_ITEMS = 256
def parse_loop(text):
try:
s = json.loads(text or "{}") or {}
except Exception:
s = {}
inject = [{"node": str(i["node"]), "slot": int(i.get("slot", 0)), "source": i.get("source", "items")}
for i in s.get("inject", []) if isinstance(i, dict) and i.get("node") not in (None, "")]
params = [{"node": str(p["node"]), "widget": str(p["widget"]), "values": list(p["values"])}
for p in s.get("params", []) if isinstance(p, dict) and p.get("node") not in (None, "") and p.get("widget")
and isinstance(p.get("values"), list) and p["values"]]
return inject, params
class BFSWorkflowLoop:
DESCRIPTION = __doc__.split("\n\n")[0]
@classmethod
def INPUT_TYPES(cls):
return {"required": {
"result": ("IMAGE", {"lazy": True, "tooltip": "What one iteration produces; everything upstream of it can be looped."}),
"loop": ("STRING", {"default": "{}", "multiline": True, "tooltip": "Set by the panel: injected nodes and per-iteration widgets."}),
}, "optional": {
"items": ("IMAGE", {"tooltip": "The batch to iterate over: item i is injected on iteration i."}),
"items_b": ("IMAGE", {"tooltip": "A second batch, same length (e.g. the marked frames next to the plain ones)."}),
"item_masks": ("MASK", {"tooltip": "One mask per item, injected where the panel says (source 'item_masks')."}),
"count": ("INT", {"default": 0, "min": 0, "max": MAX_ITEMS,
"tooltip": "Iterations when there are no items (params only); 0 = as many as the longest value list."}),
}, "hidden": {"dynprompt": "DYNPROMPT", "unique_id": "UNIQUE_ID"}}
RETURN_TYPES = ("IMAGE", "INT")
RETURN_NAMES = ("images", "count")
OUTPUT_TOOLTIPS = ("The result of every iteration, in order, as one batch (resized to the first one's size).", "Iterations run.")
FUNCTION = "run_loop"
CATEGORY = CATEGORY
def check_lazy_status(self, result=None, **kw):
return []
def run_loop(self, loop="{}", result=None, items=None, items_b=None, item_masks=None, count=0, dynprompt=None, unique_id=None):
inject, params = parse_loop(loop)
if not inject and not params:
raise ValueError("BFS Workflow Loop: pick a node to inject the items into, or a widget to change, in the panel.")
if dynprompt is None:
raise ValueError("BFS Workflow Loop needs ComfyUI's dynamic graph execution")
from comfy_execution.graph_utils import GraphBuilder, is_link
me = str(unique_id); mine = dynprompt.get_node(me)["inputs"]
link = mine.get("result")
if not is_link(link):
raise ValueError("Connect what one iteration produces to `result`.")
n = int(items.shape[0]) if items is not None else (int(count) or max(len(p["values"]) for p in params) if params else 0)
if any(i["source"] == "items" for i in inject) and items is None:
raise ValueError("The panel injects `items`, but nothing is connected to items.")
if any(i["source"] == "items_b" for i in inject):
if items_b is None:
raise ValueError("The panel injects `items_b`, but nothing is connected to items_b.")
if items is not None and items_b.shape[0] != n:
raise ValueError(f"{items_b.shape[0]} items_b for {n} items.")
n = n or int(items_b.shape[0])
if any(i["source"] == "item_masks" for i in inject):
if item_masks is None:
raise ValueError("The panel injects `item_masks`, but nothing is connected to item_masks.")
if items is not None and item_masks.shape[0] != n:
raise ValueError(f"{item_masks.shape[0]} masks for {n} items.")
if n < 1:
raise ValueError("Nothing to iterate over: connect items or give parameter values.")
if n > MAX_ITEMS:
raise ValueError(f"{n} iterations; at most {MAX_ITEMS}.")
root = str(link[0])
up, feeds, stack = set(), {}, [root]
while stack:
nid = stack.pop()
if nid in up:
continue
up.add(nid)
for v in dynprompt.get_node(nid).get("inputs", {}).values():
if is_link(v):
p = str(v[0]); feeds.setdefault(p, set()).add(nid); stack.append(p)
for i in inject:
if i["node"] not in up:
raise ValueError(f"Injected node {i['node']} is not upstream of `result` (pick it again in the panel).")
for p in params:
if p["node"] not in up:
raise ValueError(f"Node {p['node']} is not upstream of `result` (pick it again in the panel).")
if p["widget"] not in dynprompt.get_node(p["node"]).get("inputs", {}):
raise ValueError(f"Node {p['node']} has no widget '{p['widget']}' in this prompt.")
injected = {(i["node"], i["slot"]): i["source"] for i in inject}
# the body: everything that depends on an injected output or a changed widget and leads to `result`
seeds = [p["node"] for p in params] + [c for i in inject for c in feeds.get(i["node"], ()) if c in up]
body, stack = set(), list(seeds)
while stack:
nid = stack.pop()
if nid in body:
continue
body.add(nid)
stack.extend(c for c in feeds.get(nid, ()) if c in up)
body -= {i["node"] for i in inject if i["node"] not in {p["node"] for p in params}}
graph = GraphBuilder()
outs = []
for it in range(n):
picks = {}
for src in set(injected.values()):
pk = graph.node("BFSLoopPick", f"pick{it}_{src}")
if src == "item_masks":
pk.set_input("masks", mine["item_masks"])
else:
pk.set_input("images", mine[src])
pk.set_input("index", it)
picks[src] = pk
over = {(p["node"], p["widget"]): p["values"][it % len(p["values"])] for p in params}
clones = {nid: graph.node(dynprompt.get_node(nid)["class_type"], f"i{it}_{nid}") for nid in body}
for nid, node in clones.items():
node.set_override_display_id(dynprompt.get_display_node_id(nid))
for key, value in dynprompt.get_node(nid).get("inputs", {}).items():
if (nid, key) in over:
node.set_input(key, _cast(over[(nid, key)], value))
elif is_link(value) and (str(value[0]), int(value[1])) in injected:
src = injected[(str(value[0]), int(value[1]))]
node.set_input(key, picks[src].out(1 if src == "item_masks" else 0))
elif is_link(value) and str(value[0]) in body:
node.set_input(key, clones[str(value[0])].out(value[1]))
else:
node.set_input(key, value)
if root in clones:
outs.append(clones[root].out(link[1]))
elif (root, int(link[1])) in injected:
src = injected[(root, int(link[1]))]; outs.append(picks[src].out(1 if src == "item_masks" else 0))
else:
outs.append(link)
coll = graph.node("BFSLoopCollect", "collect")
for i, o in enumerate(outs):
coll.set_input(f"image_{i}", o)
coll.set_input("n", n)
return {"result": (coll.out(0), coll.out(1)), "expand": graph.finalize()}
class BFSLoopPick:
"""Internal: item `index` of a batch (image and/or mask)."""
@classmethod
def INPUT_TYPES(cls):
return {"required": {"index": ("INT", {"default": 0, "min": 0, "max": 100000})},
"optional": {"images": ("IMAGE",), "masks": ("MASK",)}}
RETURN_TYPES = ("IMAGE", "MASK")
FUNCTION = "pick"
CATEGORY = CATEGORY
def pick(self, index, images=None, masks=None):
img = images[index:index + 1] if images is not None else torch.zeros(1, 64, 64, 3)
if masks is not None:
m = masks if masks.ndim == 3 else masks[None]
msk = m[index:index + 1]
else:
msk = torch.zeros(1, img.shape[1], img.shape[2])
return img, msk
class BFSLoopCollect:
"""Internal: every iteration's result, in order, as one batch."""
@classmethod
def INPUT_TYPES(cls):
return {"required": {"n": ("INT", {"default": 1, "min": 1, "max": MAX_ITEMS})},
"optional": {f"image_{i}": ("IMAGE",) for i in range(MAX_ITEMS)}}
RETURN_TYPES = ("IMAGE", "INT")
FUNCTION = "collect"
CATEGORY = CATEGORY
def collect(self, n, **kw):
parts = [kw[f"image_{i}"] for i in range(n)]
h, w = parts[0].shape[1:3]
parts = [p if p.shape[1:3] == (h, w) else
F.interpolate(p.movedim(-1, 1), size=(h, w), mode="bilinear", align_corners=False).movedim(1, -1) for p in parts]
return torch.cat(parts), n
NODE_CLASS_MAPPINGS = {"BFSWorkflowLoop": BFSWorkflowLoop, "BFSLoopPick": BFSLoopPick, "BFSLoopCollect": BFSLoopCollect}
NODE_DISPLAY_NAME_MAPPINGS = {"BFSWorkflowLoop": "BFS Workflow Loop", "BFSLoopPick": "BFS Loop Pick (internal)",
"BFSLoopCollect": "BFS Loop Collect (internal)"}
class BFSLoopSlot:
"""A stand-in for "the current item" inside a BFS Workflow Loop body: pick it in the loop's Inject list and it gets
item i on iteration i. Outside a loop it gives a small gray image (no file to load, so validation never fails)."""
@classmethod
def INPUT_TYPES(cls):
return {"required": {"label": ("STRING", {"default": "item", "tooltip": "Only a name, to tell slots apart in the loop's list."})}}
RETURN_TYPES = ("IMAGE", "MASK")
RETURN_NAMES = ("image", "mask")
FUNCTION = "slot"
CATEGORY = CATEGORY
def slot(self, label="item"):
return torch.full((1, 64, 64, 3), 0.5), torch.zeros(1, 64, 64)
NODE_CLASS_MAPPINGS["BFSLoopSlot"] = BFSLoopSlot
NODE_DISPLAY_NAME_MAPPINGS["BFSLoopSlot"] = "BFS Loop Slot"
+241
View File
@@ -0,0 +1,241 @@
"""BFS XY Sweep: run any workflow over a grid of widget values and get one labeled grid back.
Connect the IMAGE you want to compare (a decoded video or an image) to `result`. In the panel, pick for X (and
optionally Y) any widget of any node upstream of that input (the panel lists them, like TinyTerra's advanced xyPlot)
and type the values. On run the node does not compute `result` itself (lazy input): it expands one copy of the part of
the graph that depends on the swept widgets per combination (everything else, models included, is shared and runs
once), then a collector builds the grid.
All state lives in the hidden `sweep` STRING widget (JSON), saved with the workflow:
{"x": {"node": "44", "widget": "lora_name", "values": ["a.safetensors", "b.safetensors"]},
"y": {"node": "101", "widget": "keyframe_until", "values": [0.5, 0.7, 0.9]}}
"""
import json
import os
import numpy as np
import torch
import torch.nn.functional as F
from PIL import Image, ImageDraw, ImageFont
try:
import folder_paths
except Exception: # tests run without ComfyUI
folder_paths = None
CATEGORY = "BFS/Loop"
MAX_CELLS = 64
def parse_sweep(text):
try:
s = json.loads(text or "{}") or {}
except Exception:
s = {}
axes = []
for key in ("x", "y"):
a = s.get(key) or {}
if a.get("node") not in (None, "") and a.get("widget") and isinstance(a.get("values"), list) and a["values"]:
axes.append({"node": str(a["node"]), "widget": str(a["widget"]), "values": list(a["values"]), "title": a.get("title", "")})
return axes
def _cast(value, like):
"""A typed value from the panel, cast to the type the widget had in the prompt."""
if isinstance(like, bool):
return value if isinstance(value, bool) else str(value).strip().lower() in ("1", "true", "yes", "on")
if isinstance(like, int) and not isinstance(like, bool):
return int(round(float(value)))
if isinstance(like, float):
return float(value)
return value if isinstance(value, str) else str(value)
def _short(v):
s = str(v)
if "/" in s or "\\" in s:
s = s.replace("\\", "/").rsplit("/", 1)[-1]
for ext in (".safetensors", ".ckpt", ".pt", ".png", ".jpg", ".mp4"):
if s.endswith(ext):
s = s[: -len(ext)]
return s if len(s) <= 40 else "…" + s[-39:]
def _font(size):
for name in ("DejaVuSans.ttf", "arial.ttf", "LiberationSans-Regular.ttf"):
try:
return ImageFont.truetype(name, size)
except Exception:
continue
return ImageFont.load_default()
class BFSXYSweep:
DESCRIPTION = __doc__.split("\n\n")[0]
@classmethod
def INPUT_TYPES(cls):
return {"required": {
"result": ("IMAGE", {"lazy": True, "tooltip": "The image or decoded video to compare; the sweep runs everything upstream of it."}),
"frames_per_cell": ("INT", {"default": 4, "min": 1, "max": 16, "tooltip": "For videos: how many frames (evenly spread) each cell shows."}),
"cell_height": ("INT", {"default": 256, "min": 64, "max": 1024, "step": 16}),
"save_videos": ("BOOLEAN", {"default": True, "tooltip": "Save every multi-frame result as an mp4 in output/bfs_sweep/, named by its values."}),
"fps": ("FLOAT", {"default": 24.0, "min": 1.0, "max": 120.0}),
"sweep": ("STRING", {"default": "{}", "multiline": True, "tooltip": "Set by the panel: the swept widgets and their values."}),
}, "hidden": {"dynprompt": "DYNPROMPT", "unique_id": "UNIQUE_ID"}}
RETURN_TYPES = ("IMAGE", "IMAGE", "STRING")
RETURN_NAMES = ("grid", "results", "labels")
OUTPUT_TOOLTIPS = ("The labeled grid.", "Every result (the cells' frames), resized to one size.", "JSON: the values of every cell, in order.")
FUNCTION = "sweep_run"
CATEGORY = CATEGORY
OUTPUT_NODE = True
def check_lazy_status(self, result=None, **kw):
return [] # never evaluate `result` here: the expansion evaluates one copy per combination
def sweep_run(self, result=None, frames_per_cell=4, cell_height=256, save_videos=True, fps=24.0, sweep="{}",
dynprompt=None, unique_id=None):
axes = parse_sweep(sweep)
if not axes:
raise ValueError("BFS XY Sweep: choose at least the X widget and its values in the panel.")
if dynprompt is None:
raise ValueError("BFS XY Sweep needs ComfyUI's dynamic graph execution")
from comfy_execution.graph_utils import GraphBuilder, is_link
me = str(unique_id)
link = dynprompt.get_node(me)["inputs"].get("result")
if not is_link(link):
raise ValueError("Connect the image to compare to `result`.")
root = str(link[0])
# nodes upstream of `result` and who feeds whom inside that set
up, feeds, stack = set(), {}, [root]
while stack:
nid = stack.pop()
if nid in up:
continue
up.add(nid)
for v in dynprompt.get_node(nid).get("inputs", {}).values():
if is_link(v):
p = str(v[0]); feeds.setdefault(p, set()).add(nid); stack.append(p)
for a in axes:
if a["node"] not in up:
raise ValueError(f"Swept node {a['node']} is not upstream of `result` (pick it again in the panel).")
if a["widget"] not in dynprompt.get_node(a["node"]).get("inputs", {}):
raise ValueError(f"Node {a['node']} has no widget '{a['widget']}' in this prompt.")
# the body: what depends on a swept node and leads to `result` (cloned per cell); the rest is shared
body, stack = set(), [a["node"] for a in axes]
while stack:
nid = stack.pop()
if nid in body:
continue
body.add(nid)
stack.extend(c for c in feeds.get(nid, ()) if c in up)
xs = axes[0]["values"]; ys = axes[1]["values"] if len(axes) > 1 else [None]
cells = [(x, y) for y in ys for x in xs]
if len(cells) > MAX_CELLS:
raise ValueError(f"{len(cells)} combinations; at most {MAX_CELLS}.")
graph = GraphBuilder()
outs, labels = [], []
for ci, (x, y) in enumerate(cells):
over = {(axes[0]["node"], axes[0]["widget"]): x}
if y is not None:
over[(axes[1]["node"], axes[1]["widget"])] = y
clones = {n: graph.node(dynprompt.get_node(n)["class_type"], f"c{ci}_{n}") for n in body}
for nid, node in clones.items():
node.set_override_display_id(dynprompt.get_display_node_id(nid))
for key, value in dynprompt.get_node(nid).get("inputs", {}).items():
if (nid, key) in over:
node.set_input(key, _cast(over[(nid, key)], value))
elif is_link(value) and str(value[0]) in body:
node.set_input(key, clones[str(value[0])].out(value[1]))
else:
node.set_input(key, value)
outs.append(clones[root].out(link[1]) if root in clones else link)
labels.append({"x": x, "y": y})
meta = {"axes": [{k: a[k] for k in ("node", "widget", "title")} for a in axes], "cells": labels,
"xs": xs, "ys": ys if len(axes) > 1 else [], "frames_per_cell": frames_per_cell, "cell_height": cell_height,
"save_videos": save_videos, "fps": fps}
coll = graph.node("BFSXYSweepCollect", "collect")
coll.set_input("meta", json.dumps(meta))
for i, o in enumerate(outs):
coll.set_input(f"image_{i}", o)
return {"result": (coll.out(0), coll.out(1), coll.out(2)), "expand": graph.finalize()}
class BFSXYSweepCollect:
"""Internal: receives every cell's result from BFS XY Sweep and draws the grid."""
@classmethod
def INPUT_TYPES(cls):
return {"required": {"meta": ("STRING", {"default": "{}"})},
"optional": {f"image_{i}": ("IMAGE",) for i in range(MAX_CELLS)}}
RETURN_TYPES = ("IMAGE", "IMAGE", "STRING")
RETURN_NAMES = ("grid", "results", "labels")
FUNCTION = "collect"
CATEGORY = CATEGORY
OUTPUT_NODE = True
def collect(self, meta, **kw):
m = json.loads(meta)
cells = m["cells"]; k = int(m["frames_per_cell"]); ch = int(m["cell_height"])
imgs = [kw.get(f"image_{i}") for i in range(len(cells))]
strips, saved = [], []
for i, (img, lab) in enumerate(zip(imgs, cells)):
idx = np.unique(np.linspace(0, img.shape[0] - 1, min(k, img.shape[0])).round().astype(int))
fr = img[idx, ..., :3].float()
w = max(1, round(fr.shape[2] * ch / fr.shape[1]))
fr = F.interpolate(fr.movedim(-1, 1), size=(ch, w), mode="bilinear", align_corners=False).movedim(1, -1)
strips.append(torch.cat(list(fr), dim=1))
if m.get("save_videos") and img.shape[0] > 1 and folder_paths:
saved.append(self._save(img, lab, m, i))
sw = max(s.shape[1] for s in strips)
strips = [F.pad(s, (0, 0, 0, sw - s.shape[1])) for s in strips]
xs, ys = m["xs"], m["ys"] or [None]
ax = m["axes"]
xl = f"{ax[0]['title'] or ax[0]['node']}.{ax[0]['widget']}"
yl = f"{ax[1]['title'] or ax[1]['node']}.{ax[1]['widget']}" if len(ax) > 1 else ""
pad, top, left = 6, 34, (170 if len(ax) > 1 else 0)
H = top + len(ys) * (ch + pad); W = left + len(xs) * (sw + pad)
canvas = Image.new("RGB", (W, H), (24, 24, 24)); d = ImageDraw.Draw(canvas); f = _font(16)
for c, x in enumerate(xs):
d.text((left + c * (sw + pad) + 4, 8), f"{xl} = {_short(x)}", fill=(230, 230, 230), font=f)
for r, y in enumerate(ys):
if y is not None:
d.text((6, top + r * (ch + pad) + ch // 2 - 10), f"{yl} =\n{_short(y)}", fill=(230, 230, 230), font=f)
for c in range(len(xs)):
s = strips[r * len(xs) + c]
arr = (s.clamp(0, 1).cpu().numpy() * 255).astype(np.uint8)
canvas.paste(Image.fromarray(arr), (left + c * (sw + pad), top + r * (ch + pad)))
grid = torch.from_numpy(np.asarray(canvas, np.float32) / 255.0)[None]
results = torch.stack(strips)
info = {"cells": cells, "videos": saved}
ui = {}
if folder_paths:
out = os.path.join(folder_paths.get_output_directory(), "bfs_sweep"); os.makedirs(out, exist_ok=True)
name = f"grid_{len(os.listdir(out)):05d}.png"; canvas.save(os.path.join(out, name))
ui = {"images": [{"filename": name, "subfolder": "bfs_sweep", "type": "output"}]}
return {"ui": ui, "result": (grid, results, json.dumps(info))}
@staticmethod
def _save(img, lab, m, i):
import av
out = os.path.join(folder_paths.get_output_directory(), "bfs_sweep"); os.makedirs(out, exist_ok=True)
tag = "_".join(_short(v).replace(" ", "") for v in (lab["x"], lab["y"]) if v is not None)
tag = "".join(ch if ch.isalnum() or ch in "._-" else "-" for ch in tag)[:80]
path = os.path.join(out, f"{i:02d}_{tag}.mp4")
frames = (img[..., :3].clamp(0, 1).cpu().numpy() * 255).astype(np.uint8)
h, w = frames.shape[1] // 2 * 2, frames.shape[2] // 2 * 2
with av.open(path, "w") as c:
st = c.add_stream("libx264", rate=int(round(float(m.get("fps", 24)))))
st.width, st.height, st.pix_fmt = w, h, "yuv420p"; st.options = {"crf": "16"}
for fr in frames:
for p in st.encode(av.VideoFrame.from_ndarray(fr[:h, :w], format="rgb24")):
c.mux(p)
for p in st.encode():
c.mux(p)
return path
NODE_CLASS_MAPPINGS = {"BFSXYSweep": BFSXYSweep, "BFSXYSweepCollect": BFSXYSweepCollect}
NODE_DISPLAY_NAME_MAPPINGS = {"BFSXYSweep": "BFS XY Sweep", "BFSXYSweepCollect": "BFS XY Sweep Collect (internal)"}
+4
View File
@@ -420,3 +420,7 @@ NODE_DISPLAY_NAME_MAPPINGS = {
'BFSMiniMaxH3GuideTarget': 'MiniMax-H3 Guide Target — Image / Video (BFS)', 'BFSMiniMaxH3GuideTarget': 'MiniMax-H3 Guide Target — Image / Video (BFS)',
'BFSMiniMaxH3IdentityReference': 'MiniMax-H3 Identity Reference + RoPE (BFS)', 'BFSMiniMaxH3IdentityReference': 'MiniMax-H3 Identity Reference + RoPE (BFS)',
} }
from .bfs_h3_image_guide import BFSH3ImageAddGuide
NODE_CLASS_MAPPINGS['BFSH3ImageAddGuide'] = BFSH3ImageAddGuide
NODE_DISPLAY_NAME_MAPPINGS['BFSH3ImageAddGuide'] = 'Add Guide for MiniMax H3 Image (BFS)'
+2 -2
View File
@@ -1,7 +1,7 @@
[project] [project]
name = "bfsnodes" name = "bfsnodes"
description = "Custom ComfyUI nodes for reference-guided video: BFS Sampler (guide-driven IC-LoRAs, auto crop, mask inpainting, temporal chunking), LTX Multiple Controls / Identity Transfer / Edit Anything, Head-Swap and Anime2Real conditioning, MiniMax-H3 latent guides, RoPE layouts and loaders, LoRA Surgery, and Shot Planner / Shot Loop (tabbed panel): split long videos into shots at camera cuts, face-tracked cast, SAM 3 mask per shot with full frame, paste, mask, crop and crop + mask generation modes, optional translucent Colored Mark overlay for manually selected regions, optional Qwen3-VL to suggest settings, describe references ({details}) and the person to replace ({target}), continuity between shots, H3 conditioning with an optional duet and prompt writer, join with the original timing, audio and a side-by-side comparison. H3 Duet: pin a source clip beside the video, training-free." description = "ComfyUI nodes for video workflows: SAM 3 segmentation and Molmo2 tracking; MiniMax H3 character swapping and sampling; shot planning and looping; LTX video controls and identity transfer; model loaders, LoRA tools, and image/video utilities."
version = "1.115.33" version = "1.120.0"
license = {file = "LICENSE"} license = {file = "LICENSE"}
dependencies = [ dependencies = [
"torch", "torch",
+34
View File
@@ -0,0 +1,34 @@
import importlib.util
import sys
import os
import unittest
from pathlib import Path
import torch
if os.environ.get('COMFYUI_PATH'):
sys.path.insert(0, os.environ['COMFYUI_PATH'])
from comfy.ldm.minimax.model import PackedLayout
spec = importlib.util.spec_from_file_location('image_guide', Path(__file__).resolve().parents[1] / 'bfs_h3_image_guide.py')
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
class ImageGuideTest(unittest.TestCase):
def test_only_guide_positions_are_aligned_and_inputs_stay_unchanged(self):
refs = [{'kind': 'image', 'latent_h': 8, 'latent_w': 8},
{'kind': 'image', 'latent_h': 8, 'latent_w': 8, module.MARKER: True}]
original = PackedLayout(4, 1, 8, 8, 0, refs=refs)
positions = original.position_ids.clone()
result = module.align_guide_layout(original, refs)
ref_segments = [(a,b) for a,b,k in result.segments if k == 'ref_img']
a,b = ref_segments[1]
ta,tb = next((a,b) for a,b,k in result.segments if k == 'video')
self.assertTrue(torch.equal(result.position_ids[a:b], result.position_ids[ta:tb]))
a,b = ref_segments[0]
self.assertTrue(torch.equal(result.position_ids[a:b], positions[a:b]))
self.assertTrue(torch.equal(original.position_ids, positions))
self.assertTrue(torch.equal(result.img_update, original.img_update))
def test_incompatible_guide_grid_is_rejected(self):
refs = [{'kind': 'image', 'latent_h': 4, 'latent_w': 4, module.MARKER: True}]
layout = PackedLayout(4, 1, 8, 8, 0, refs=refs)
with self.assertRaisesRegex(ValueError, 'match the target'):
module.align_guide_layout(layout, refs)
@@ -0,0 +1,60 @@
import importlib.util
import sys
import os
import unittest
from pathlib import Path
import torch
if os.environ.get('COMFYUI_PATH'):
sys.path.insert(0, os.environ['COMFYUI_PATH'])
spec = importlib.util.spec_from_file_location('image_swap', Path(__file__).resolve().parents[1] / 'bfs_h3_image_swap_conditioning.py')
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
class Clip:
def tokenize(self, prompt, **kwargs):
self.prompt, self.images = prompt, kwargs['images']
return {'text': prompt}
def encode_from_tokens_scheduled(self, tokens):
return [[torch.zeros(1, 4, 8), {}]]
class Vae:
def encode(self, pixels):
return torch.zeros(1, 24, 1, pixels.shape[1] // 16, pixels.shape[2] // 16)
class ImageSwapTest(unittest.TestCase):
def test_modes_use_still_image_language_and_shifted_picture_numbers(self):
for mode in module.MODES:
text = module.image_swap_prompt(3, mode)
for word in ('video', 'motion', 'frame', 'timing'):
self.assertNotIn(word, text.lower())
for j, color in enumerate(module.COLORS, 1):
self.assertIn(f'<Subject {j}> is the subject marked by the {color}', text)
self.assertIn(f'appearance comes from <Picture {j+1}>', text)
self.assertIn('foreground occluders', text)
def test_native_conditioning_order_and_image_latent(self):
clip = Clip()
guide = torch.full((1, 32, 32, 3), .1)
red = torch.full_like(guide, .2)
blue = torch.full_like(guide, .3)
result = module.BFSH3ImageCharacterSwapConditioning.execute(
clip, Vae(), guide, {'reference_2': blue, 'reference_0': red}, width=128, height=96)
positive, latent, prompt = result.args
self.assertEqual(tuple(latent['samples'].shape), (1, 24, 1, 6, 8))
self.assertEqual(len(positive[0][1]['minimax_refs']), 3)
self.assertEqual([round(float(i.mean()), 1) for i in clip.images], [.1, .2, .3])
self.assertIn('red subject replaced by <Picture 2>', prompt)
self.assertIn('blue subject replaced by <Picture 3>', prompt)
def test_override_preserved(self):
clip = Clip()
image = torch.zeros(1, 32, 32, 3)
result = module.BFSH3ImageCharacterSwapConditioning.execute(
clip, Vae(), image, {'reference_0': image}, width=64, height=64, prompt_override='Custom edit')
self.assertEqual(result.args[2], 'Custom edit')
self.assertEqual(clip.prompt, 'Custom edit')
+72
View File
@@ -0,0 +1,72 @@
import importlib.util
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
SPEC = importlib.util.spec_from_file_location("bfs_h3_swap_conditioning", ROOT / "bfs_h3_swap_conditioning.py")
SC = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(SC)
NODE = SC.BFSH3KeyframeConditioning
class InputsTest(unittest.TestCase):
def test_modes(self):
self.assertEqual(SC.MODES, ["keyframe pins", "normal map guide", "DWPose guide", "translucent overlay guide", "outline guide"])
def test_pictures_grouped_first(self):
t = NODE.INPUT_TYPES()
self.assertEqual(list(t["required"])[:7], ["clip", "vae", "guide_video", "width", "height", "length", "picture_1"])
self.assertEqual(list(t["optional"])[:5], ["picture_2", "picture_3", "model", "keyframes", "keyframe_spec"])
def test_outputs(self):
self.assertEqual(len(NODE.RETURN_TYPES), len(NODE.RETURN_NAMES))
self.assertEqual(len(NODE.OUTPUT_IS_LIST), len(NODE.RETURN_TYPES))
self.assertEqual(NODE.RETURN_NAMES[-1], "pictures")
self.assertTrue(NODE.OUTPUT_IS_LIST[-1])
def test_unknown_mode(self):
import torch
with self.assertRaises(ValueError):
NODE().encode(None, None, torch.zeros(1, 8, 8, 3), 8, 8, 1, torch.zeros(1, 8, 8, 3), mode="gray video")
class PromptTest(unittest.TestCase):
def test_guide_prompts_name_each_subject_once(self):
for fn, word in ((SC.normals_caption, "normal map"), (SC.pose_caption, "pose skeleton")):
for k in range(1, SC.MAX_PICTURES + 1):
p = fn(k)
self.assertTrue(p.startswith("character_swap:\nsubject_definitions:\n"))
for j in range(1, k + 1):
self.assertIn(f"<Subject {j}> is the subject marked by the {SC.SUBJECT_COLORS[j - 1]} {word}", p)
self.assertNotIn(f"<Picture {k + 1}>", p)
def test_pins_prompt(self):
p = SC.pins_caption(2, [0.0, 1.5])
self.assertIn("2 keyframes of the target video are pinned at 0.0s, 1.5s", p)
self.assertIn("<Picture 2>", p)
self.assertIn("1 keyframe of the target video is pinned at 2.0s", SC.pins_caption(1, [2.0]))
def test_overlay_prompt(self):
prompt = SC.overlay_caption(2)
self.assertIn("translucent red overlay", prompt)
self.assertIn("translucent blue overlay", prompt)
self.assertIn("original appearance visible beneath it is not the target appearance", prompt)
self.assertIn("The tint must disappear in the result", prompt)
self.assertNotIn("<Picture 3>", prompt)
def test_outline_prompt_preserves_props(self):
prompt = SC.outline_caption(2)
self.assertIn("red outline", prompt)
self.assertIn("blue outline", prompt)
self.assertIn("held or interacting objects", prompt)
self.assertIn("foreground occlusions", prompt)
self.assertNotIn("<Picture 3>", prompt)
def test_normal_map_prompt_preserves_occluders(self):
prompt = SC.normals_caption(1)
self.assertIn("held objects and foreground occluders", prompt)
self.assertIn("original positions, shapes and colors", prompt)
if __name__ == "__main__":
unittest.main()
+265
View File
@@ -0,0 +1,265 @@
import importlib.util
import json
import sys
import types
import unittest
from pathlib import Path
from unittest.mock import patch
import torch
ROOT = Path(__file__).resolve().parents[1]
SPEC = importlib.util.spec_from_file_location("bfs_segment_picker", ROOT / "bfs_segment_picker.py")
SP = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(SP)
ORIGINAL_DETECT_TEXT = SP.detect_text
ORIGINAL_DETECT_POINTS = SP.detect_points
def rect(h, w, x0, y0, x1, y1):
m = torch.zeros(h, w, dtype=torch.bool)
m[y0:y1, x0:x1] = True
return m
class StateTest(unittest.TestCase):
def test_empty_and_broken_values(self):
for v in (None, "", "{}", "not json", "[1, 2]"):
self.assertEqual(SP.parse_state(v), {"selected": [], "added": [], "next": 1})
def test_keeps_valid_entries_only(self):
s = SP.parse_state(json.dumps({
"selected": ["a1", "a1", "t1.0", 3],
"added": [{"id": "t1", "text": " red car "}, {"id": "p2", "points": [{"x": .5, "y": .5, "label": 1}]},
{"id": "p3", "points": [{"x": .1, "y": .1, "label": 0}]}, # only exclude points: nothing to keep
{"text": "no id"}, {"id": "t4", "text": " "}],
"next": 5}))
self.assertEqual(s["selected"], ["a1", "t1.0", "3"])
self.assertEqual(s["added"], [{"id": "t1", "text": "red car"}, {"id": "p2", "points": [{"x": .5, "y": .5, "label": 1}]}])
self.assertEqual(s["next"], 5)
def test_concepts(self):
self.assertEqual(SP.split_concepts("person, animal,\ncar, person ,,"), ["person", "animal", "car"])
self.assertEqual(SP.split_concepts(""), [])
def test_entry_key_changes_with_what_it_asks(self):
a = SP.entry_key({"id": "t1", "text": "car"})
self.assertEqual(a, SP.entry_key({"id": "t9", "text": "car"}))
self.assertNotEqual(a, SP.entry_key({"id": "t1", "text": "bus"}))
class GeometryTest(unittest.TestCase):
def test_box(self):
self.assertEqual(SP.box_of(rect(10, 20, 3, 2, 8, 6)), [3, 2, 5, 4])
self.assertEqual(SP.box_of(torch.zeros(4, 4, dtype=torch.bool)), [0, 0, 0, 0])
def test_dedupe_drops_repeats_keeps_neighbours(self):
a = rect(20, 20, 0, 0, 10, 10)
repeat = rect(20, 20, 0, 0, 10, 9) # 90 % inside a
other = rect(20, 20, 8, 8, 18, 18) # touches a a little
kept = SP.dedupe([(a, {"n": 1}), (repeat, {"n": 2}), (other, {"n": 3})])
self.assertEqual([i["n"] for _, i in kept], [1, 3])
def test_compose(self):
u, each = SP.compose([], 4, 5)
self.assertEqual(tuple(u.shape), (1, 4, 5)); self.assertEqual(float(u.sum()), 0.)
a, b = rect(4, 5, 0, 0, 2, 2), rect(4, 5, 1, 1, 3, 3)
u, each = SP.compose([a, b], 4, 5)
self.assertEqual(tuple(each.shape), (2, 4, 5))
self.assertEqual(float(u.sum()), float((a | b).sum()))
def test_tint_only_inside(self):
img = torch.zeros(1, 4, 4, 3)
m = rect(4, 4, 0, 0, 2, 2).float()[None]
out = SP.tint(img, m, (255, 0, 255), .3)
self.assertAlmostEqual(float(out[0, 0, 0, 0]), .3, places=5)
self.assertAlmostEqual(float(out[0, 0, 0, 1]), 0., places=5)
self.assertEqual(float(out[0, 3, 3].sum()), 0.)
def test_color(self):
self.assertEqual(SP.parse_color("#FF00FF"), (255, 0, 255))
self.assertEqual(SP.parse_color("0f0"), (0, 255, 0))
self.assertEqual(SP.parse_color("nope"), (255, 0, 255))
class InferenceModeTest(unittest.TestCase):
def test_text_and_point_sam_calls_run_in_inference_mode(self):
calls = []
class FakeDetect:
@staticmethod
def execute(**kwargs):
calls.append(torch.is_inference_mode_enabled())
return types.SimpleNamespace(args=(torch.ones(1, 4, 4), [[{"score": 0.9}]]))
parent = types.ModuleType("comfy_extras")
parent.__path__ = []
child = types.ModuleType("comfy_extras.nodes_sam3")
child.SAM3_Detect = FakeDetect
frame = torch.zeros(1, 4, 4, 3)
with patch.dict(sys.modules, {"comfy_extras": parent, "comfy_extras.nodes_sam3": child}), \
patch.object(SP, "_text_cond", return_value=[]):
self.assertEqual(len(ORIGINAL_DETECT_TEXT((None, None), frame, "person", 0.4, 1)), 1)
self.assertTrue(ORIGINAL_DETECT_POINTS((None, None), frame, [{"x": 0.5, "y": 0.5, "label": 1}], 0.4).any())
self.assertEqual(calls, [True, True])
class AutoObjectsTest(unittest.TestCase):
def test_rejects_scene_sized_generic_object_but_keeps_person(self):
h, w = 20, 40
scene = torch.zeros(h, w, dtype=torch.bool)
scene[::2, :] = True
scene[-1, :] = True
person = rect(h, w, 10, 2, 28, 18)
def detect_text(sam, frame, concept, threshold, max_detections):
mask = person if concept == "person" else scene
return [(mask, {"score": .9})]
frame = torch.zeros(1, h, w, 3)
with patch.object(SP, "detect_text", side_effect=detect_text):
objects = SP.auto_objects((None, None), frame, ["objects", "person"], .4, 8, 0.)
with patch.object(SP, "detect_text", return_value=[(scene, {"score": .9})]):
large_person = SP.auto_objects((None, None), frame, ["person"], .4, 8, 0.)
self.assertEqual([(o["id"], o["label"]) for o in objects], [("a1", "person")])
self.assertEqual(len(large_person), 1)
class SessionTest(unittest.TestCase):
"""The node with SAM replaced: what is segmented once, what is segmented again, what the outputs hold."""
def setUp(self):
self.calls = []
h, w = 20, 40
people = [rect(h, w, 25, 2, 35, 18), rect(h, w, 2, 2, 12, 18)]
def detect_text(sam, frame, text, threshold, max_detections):
self.calls.append(("text", text))
if text == "person":
return [(m, {"score": .9}) for m in people]
if text == "dog":
return [(rect(h, w, 15, 10, 20, 15), {"score": .7})]
return []
def detect_points(sam, frame, points, threshold):
self.calls.append(("points", len(points)))
return rect(h, w, 14, 0, 19, 6)
SP.detect_text, SP.detect_points = detect_text, detect_points
SP._SESSIONS.clear()
self.frame = torch.rand(1, h, w, 3)
self.sam = (object(), object())
self.tmp = Path(__file__).resolve().parent / ".segpick_tmp"
SP._temp_dir = lambda: str(self.tmp.mkdir(exist_ok=True) or self.tmp)
def tearDown(self):
for f in self.tmp.glob("*.png"):
f.unlink()
if self.tmp.exists():
self.tmp.rmdir()
def run_node(self, selection, concepts="person"):
return SP.BFSSegmentPicker().pick(self.frame, self.sam[0], self.sam[1], concepts, .4, 8, 0., "#FF00FF", .3, json.dumps(selection))
def test_auto_ids_left_to_right_and_selection(self):
out = self.run_node({"selected": ["a2"]})
mask, masks, marked, overlay, info = out["result"]
info = json.loads(info)
self.assertEqual([o["id"] for o in info["objects"]], ["a1", "a2"])
self.assertLess(info["objects"][0]["box"][0], info["objects"][1]["box"][0])
self.assertEqual(info["selected"], ["a2"])
self.assertEqual(float(mask[0, 10, 30]), 1.) # a2 is the person on the right
self.assertEqual(float(mask[0, 10, 5]), 0.)
self.assertEqual(tuple(masks.shape), (1, 20, 40))
self.assertGreater(float(marked[0, 10, 30, 0]), float(self.frame[0, 10, 30, 0]) * .7 - 1e-6)
panel = out["ui"]["bfs_segpick"][0]
self.assertEqual(len(panel["objects"]), 2)
self.assertTrue((self.tmp / panel["objects"][0]["mask"]["filename"]).exists())
def test_segments_once_then_only_new_entries(self):
self.run_node({"selected": ["a1"]})
self.run_node({"selected": ["a2"]}) # only the selection changed
self.assertEqual(self.calls, [("text", "person")])
sel = {"selected": ["t1.0", "p2"], "added": [{"id": "t1", "text": "dog"},
{"id": "p2", "points": [{"x": .37, "y": .05, "label": 1}]}]}
_, masks, *_ , info = self.run_node(sel)["result"]
self.assertEqual(json.loads(info)["selected"], ["t1.0", "p2"])
self.assertEqual(tuple(masks.shape), (2, 20, 40))
self.run_node(sel)
self.assertEqual(self.calls, [("text", "person"), ("text", "dog"), ("points", 1)])
sel["added"][0]["text"] = "cat" # an edited entry is segmented again
_, _, _, _, info = self.run_node(sel)["result"]
self.assertEqual(self.calls[-1], ("text", "cat"))
self.assertEqual(json.loads(info)["selected"], ["p2"]) # "cat" found nothing, so t1.0 is gone
def test_brush_is_dropped_on_another_image(self):
key = SP.image_key(self.frame)
sel = {"selected": ["a1", "b1"], "added": [{"id": "b1", "image": key, "strokes": [{"r": .05, "pts": [[.5, .5]]}]}]}
out = self.run_node(sel)
self.assertEqual(out["ui"]["bfs_segpick"][0]["image_key"], key)
self.assertEqual(json.loads(out["result"][4])["selected"], ["a1", "b1"])
self.frame = torch.rand(1, 20, 40, 3) # another image: the strokes no longer apply
mask, _, _, _, info = self.run_node(sel)["result"]
self.assertEqual(json.loads(info)["selected"], ["a1"])
self.assertEqual(float(mask[0, 10, 20]), 0.)
def test_remove_entries_cut_the_selection(self):
key = SP.image_key(self.frame)
sel = {"selected": ["a1", "e1", "r2"], "added": [
{"id": "e1", "image": key, "remove": True, "strokes": [{"r": .05, "pts": [[.175, .5]]}]},
{"id": "r2", "remove": True, "points": [{"x": .4, "y": .1, "label": 1}]}]}
out = self.run_node(sel)
mask, masks, _, _, info = out["result"]
self.assertEqual(float(mask[0, 10, 7]), 0.) # erased out of a1
self.assertEqual(float(mask[0, 10, 3]), 1.) # the rest of a1 stays
self.assertEqual(tuple(masks.shape), (1, 20, 40)) # remove entries are not masks of their own
self.assertEqual(float(masks[0, 10, 7]), 0.)
self.assertEqual(json.loads(info)["selected"], ["a1", "e1", "r2"])
panel = {o["id"]: o for o in out["ui"]["bfs_segpick"][0]["objects"]}
self.assertEqual((panel["e1"]["label"], panel["e1"]["remove"]), ("eraser", True))
self.assertEqual((panel["r2"]["label"], panel["r2"]["remove"]), ("remove", True))
self.assertFalse(panel["a1"]["remove"])
sel["selected"] = ["a1", "r2"] # unselecting the eraser gives a1 back whole
self.assertEqual(float(self.run_node(sel)["result"][0][0, 10, 7]), 1.)
def test_unknown_ids_are_ignored(self):
mask, masks, *_ = self.run_node({"selected": ["a9", "t5.0"]})["result"]
self.assertEqual(float(mask.sum()), 0.)
self.assertEqual(tuple(masks.shape), (1, 20, 40))
def test_another_sam_is_another_session(self):
self.run_node({})
self.sam = (object(), object())
self.run_node({})
self.assertEqual(self.calls, [("text", "person"), ("text", "person")])
def test_other_concepts_are_another_session(self):
self.run_node({}, concepts="person")
self.run_node({}, concepts="person, dog")
self.assertEqual(self.calls, [("text", "person"), ("text", "person"), ("text", "dog")])
class BrushTest(unittest.TestCase):
def test_state_keeps_strokes(self):
s = SP.parse_state(json.dumps({"added": [{"id": "b1", "strokes": [{"r": .1, "pts": [[.2, .5], [.8, .5]]}]},
{"id": "b2", "strokes": [{"r": .1, "pts": []}]}]}))
self.assertEqual(s["added"], [{"id": "b1", "image": "", "strokes": [{"r": .1, "pts": [[.2, .5], [.8, .5]]}]}])
self.assertNotEqual(SP.entry_key(s["added"][0]), SP.entry_key({"id": "b1", "strokes": [{"r": .2, "pts": [[.2, .5]]}]}))
r = SP.parse_state(json.dumps({"added": [{"id": "r1", "remove": True, "points": [{"x": .5, "y": .5, "label": 1}]},
{"id": "p2", "remove": False, "points": [{"x": .5, "y": .5, "label": 1}]}]}))
self.assertEqual([e.get("remove") for e in r["added"]], [True, None])
def test_paint_covers_the_stroke_only(self):
m = SP.paint([{"r": .1, "pts": [[.2, .5], [.8, .5]]}], 50, 100)
self.assertTrue(m[25, 20] and m[25, 50] and m[25, 80])
self.assertFalse(m[5, 50] or m[25, 5] or m[25, 95])
def test_brush_entry_needs_no_sam(self):
frame = torch.zeros(1, 50, 100, 3)
objs = SP.entry_objects(None, frame, {"id": "b3", "strokes": [{"r": .05, "pts": [[.5, .5]]}]}, .4, 8)
self.assertEqual([(o["id"], o["source"]) for o in objs], [("b3", "brush")])
self.assertTrue(objs[0]["mask"][25, 50])
if __name__ == "__main__":
unittest.main()
+16
View File
@@ -1759,3 +1759,19 @@ class PromptReferenceContractTest(unittest.TestCase):
["fox wearing blue pants", "city street"], "Use Picture 2 for the setting", {}, ["fox wearing blue pants", "city street"], "Use Picture 2 for the setting", {},
style="captions", source_facts="A dancer is moving in a street.") style="captions", source_facts="A dancer is moving in a street.")
self.assertEqual(generator.call_count, 2) self.assertEqual(generator.call_count, 2)
class PanelInferenceModeTest(unittest.TestCase):
"""Issue #14: panel buttons reuse models a workflow run left with inference tensors; they must run like a node."""
def test_panel_work_runs_in_inference_mode(self):
import asyncio
with torch.inference_mode():
cache = torch.zeros(3) # e.g. a VLM buffer created while the workflow ran
def work():
self.assertTrue(torch.is_inference_mode_enabled())
cache.add_(1)
return cache.sum().item()
self.assertEqual(asyncio.run(SL._off_loop(work)), 3.0)
+13
View File
@@ -107,6 +107,19 @@ class CompareNodeTest(unittest.TestCase):
with self.assertRaisesRegex(ValueError, "nonempty"): with self.assertRaisesRegex(ValueError, "nonempty"):
VC.BFSVideoCompare().compare([], t) VC.BFSVideoCompare().compare([], t)
def test_a_single_frame_also_sends_lossless_pngs_at_the_compared_size(self):
from PIL import Image
guide, target = torch.rand(1, 40, 60, 3), torch.rand(1, 20, 30, 3)
ui = VC.BFSVideoCompare().compare(guide, target, preview_height=16)["ui"]["bfs_compare"][0]
self.assertEqual(sorted(ui["stills"]), ["difference", "guide", "target"])
for k, f in ui["stills"].items():
im = Image.open(os.path.join(_TMP, f["subfolder"], f["filename"]))
self.assertEqual(im.size, (60, 40)) # the guide's size, not the 16 px preview
g = Image.open(os.path.join(_TMP, ui["stills"]["guide"]["subfolder"], ui["stills"]["guide"]["filename"]))
self.assertEqual(g.getpixel((7, 5)), tuple(int(round(float(c) * 255)) for c in guide[0, 5, 7])) # lossless
video = VC.BFSVideoCompare().compare(torch.rand(3, 8, 8, 3), torch.rand(3, 8, 8, 3), preview_height=8)
self.assertEqual(video["ui"]["bfs_compare"][0]["stills"], {})
def test_reference_pictures_go_to_the_player_no_taller_than_it(self): def test_reference_pictures_go_to_the_player_no_taller_than_it(self):
from PIL import Image from PIL import Image
guide, target = torch.rand(4, 32, 48, 3), torch.rand(4, 32, 48, 3) guide, target = torch.rand(4, 32, 48, 3), torch.rand(4, 32, 48, 3)
+132
View File
@@ -0,0 +1,132 @@
import importlib
import sys
import types
import unittest
from pathlib import Path
from unittest.mock import patch
import torch
ROOT = Path(__file__).resolve().parents[1]
pkg = types.ModuleType("bfsnodes")
pkg.__path__ = [str(ROOT)]
sys.modules["bfsnodes"] = pkg
VSP = importlib.import_module("bfsnodes.bfs_video_segment_picker")
class MaskQualityTest(unittest.TestCase):
def test_rendered_edge_is_soft_and_stays_inside_selection(self):
mask = torch.zeros(20, 20, dtype=torch.bool)
mask[3:17, 3:17] = True
alpha = VSP.feather_inside_mask(mask, 4)
self.assertEqual(float(alpha[2, 10]), 0.0)
self.assertGreater(float(alpha[10, 10]), 0.95)
self.assertGreater(float(alpha[3, 10]), 0.0)
self.assertLess(float(alpha[3, 10]), 0.5)
def test_anchor_windows_cover_each_frame_once_and_keep_earlier_ties(self):
windows = list(VSP.anchor_windows([0, 53, 69, 125], 0, 209))
assigned = {i: a for a, lo, hi in windows for i in range(lo, hi)}
self.assertEqual(len(assigned), 209)
self.assertEqual(assigned[26], 0)
self.assertEqual(assigned[27], 53)
self.assertEqual(assigned[61], 53)
self.assertEqual(assigned[62], 69)
def test_fill_empty_tracks_only_gaps_and_preserves_existing_masks(self):
frames = torch.zeros(8, 4, 4, 3)
masks = torch.zeros(8, 4, 4, dtype=torch.bool)
masks[[0, 3, 7], 1, 1] = True
report = [{"frame": i, "source": "empty"} for i in range(8)]
calls = []
class FakeTracker:
def track(self, video, model, mask, anchor_frame):
calls.append((len(video), anchor_frame))
out = torch.zeros(len(video), 4, 4)
out[:, 0, 0] = 1
return out, anchor_frame
with patch.object(VSP, "BFSTrackFromFrame", FakeTracker):
filled = VSP.fill_empty_mask_runs(frames, None, masks, report, [(0, 8)], active_frame=3)
self.assertEqual(filled, 5)
self.assertEqual(calls, [(3, 2), (4, 0)])
self.assertTrue(bool(masks[3, 1, 1]))
self.assertFalse(bool(masks[3, 0, 0]))
self.assertTrue(bool(masks[5, 0, 0]))
def test_fill_scope_does_not_request_molmo_when_masks_are_cached(self):
selection = '{"frames":{"1":{"1":{"selected":["a1"]}}}}'
prompt = {"7": {"inputs": {"molmo_clip": ["6", 0]}}}
with patch.object(VSP, "video_key", return_value="video"), \
patch.object(VSP, "_LAST_TRACK_CACHE", {("video", 1): (None, None, None)}):
need = VSP.BFSVideoSegmentPicker().check_lazy_status(
selection=selection, mark_style="flat color", track=True, image=torch.zeros(2, 4, 4, 3),
molmo_clip=None, propagation_method="Molmo2 (classic)",
propagate_scope="fill only empty masks", unique_id="7", prompt=prompt)
self.assertEqual(need, [])
class AutoUnmarkingTest(unittest.TestCase):
def test_find_connected_picture_count(self):
# 1. 2 pictures connected downstream
mock_prompt = {
"100": {
"class_type": "BFSH3KeyframeConditioning",
"inputs": {
"guide_video": ["10", 4],
"picture_1": ["20", 0],
"picture_2": ["21", 0],
"picture_3": None,
}
},
"10": {
"class_type": "BFSVideoSegmentPicker",
"inputs": {}
}
}
self.assertEqual(VSP.find_connected_picture_count(mock_prompt, "10"), 2)
# 2. 3 pictures connected downstream
mock_prompt["100"]["inputs"]["picture_3"] = ["22", 0]
self.assertEqual(VSP.find_connected_picture_count(mock_prompt, "10"), 3)
# 3. Only 1 picture connected
mock_prompt["100"]["inputs"]["picture_2"] = None
mock_prompt["100"]["inputs"]["picture_3"] = None
self.assertEqual(VSP.find_connected_picture_count(mock_prompt, "10"), 1)
# 4. No downstream node connected
mock_prompt["100"]["inputs"]["guide_video"] = ["99", 0]
self.assertIsNone(VSP.find_connected_picture_count(mock_prompt, "10"))
# 5. Empty prompt or None
self.assertIsNone(VSP.find_connected_picture_count(None, "10"))
self.assertIsNone(VSP.find_connected_picture_count({}, "10"))
def test_input_types_controls(self):
t = VSP.BFSVideoSegmentPicker.INPUT_TYPES()
opt = t["optional"]
self.assertIn("max_subjects", opt)
self.assertEqual(opt["max_subjects"][0], ["auto", "1", "2", "3"])
self.assertEqual(opt["max_subjects"][1]["default"], "auto")
self.assertIn("molmo_descriptions", opt)
self.assertEqual(opt["molmo_descriptions"][0], "STRING")
self.assertFalse(opt["molmo_descriptions"][1].get("multiline"))
self.assertEqual(opt["protect_mask"][0], "MASK")
self.assertIn(VSP.PROTECTED_NORMAL_STYLE, opt["mark_style"][0])
def test_locked_objects_survive_selection_round_trip(self):
import json
state = {"frames": {"7": {"1": {"selected": ["a1"]},
"lock": {"selected": ["lock_p1"], "added": [
{"id": "lock_p1", "points": [{"x": 0.4, "y": 0.5, "label": 1}]}]}}}}
frames = VSP.parse_frames(json.dumps(state))
self.assertEqual(frames["7"]["1"]["selected"], ["a1"])
self.assertEqual(frames["7"]["lock"]["selected"], ["lock_p1"])
if __name__ == "__main__":
unittest.main()
+302
View File
@@ -0,0 +1,302 @@
/**
* BFS Keyframe Planner: the panel of the node. A small player of the guide (frames from the node's last run), a timeline
* with the keyframes on it, one card per keyframe (source frame + an optional edited frame, dropped or uploaded) and up
* to 4 reference pictures.
*
* All state lives in the hidden `plan` STRING widget (JSON: keyframe frames, edited frame and picture paths in input/),
* so a workflow saves, reloads and shares the plan. Uploaded images go to input/bfs_keyframes/.
*/
import { app } from "../../scripts/app.js";
import { api } from "../../scripts/api.js";
import { wheelToCanvas } from "./bfs_wheel.js";
import { createApp, h, ref, reactive, computed, watch, onBeforeUnmount } from "./vendor/vue.esm-browser.prod.mjs";
const DEFAULT_W = 560, DEFAULT_H = 640, MAX_PICTURES = 4, SUBFOLDER = "bfs_keyframes";
const tempUrl = (d, f) => api.apiURL(`/view?filename=${encodeURIComponent(f)}&subfolder=${encodeURIComponent(d.subfolder)}&type=${d.type || "temp"}`);
const inputUrl = p => {
const i = p.lastIndexOf("/");
return api.apiURL(`/view?filename=${encodeURIComponent(p.slice(i + 1))}&subfolder=${encodeURIComponent(i < 0 ? "" : p.slice(0, i))}&type=input`);
};
function parsePlan(text) {
let s = {};
try { s = JSON.parse(text || "{}") || {}; } catch { s = {}; }
const keyframes = (Array.isArray(s.keyframes) ? s.keyframes : []).filter(k => Number.isFinite(+k.frame))
.map(k => ({ frame: Math.round(+k.frame), edit: k.edit || null }));
const pictures = (Array.isArray(s.pictures) ? s.pictures : []).slice(0, MAX_PICTURES).map(p => p || null);
while (pictures.length < MAX_PICTURES) pictures.push(null);
return { keyframes, pictures };
}
async function upload(file) {
const body = new FormData();
body.append("image", file);
body.append("subfolder", SUBFOLDER);
body.append("type", "input");
const r = await api.fetchApi("/upload/image", { method: "POST", body });
if (r.status !== 200) throw new Error(`upload failed (${r.status})`);
const j = await r.json();
return j.subfolder ? `${j.subfolder}/${j.name}` : j.name;
}
function pickFile() {
return new Promise(resolve => {
const inp = document.createElement("input");
inp.type = "file"; inp.accept = "image/*";
inp.onchange = () => resolve(inp.files?.[0] || null);
inp.click();
});
}
let styled = false;
function styles() {
if (styled) return;
styled = true;
const css = document.createElement("style");
css.textContent = `
.bkp{position:absolute;inset:0;display:flex;flex-direction:column;gap:6px;padding:6px;box-sizing:border-box;font:12px system-ui,sans-serif;color:#ddd}
.bkp button{background:#333;color:#ddd;border:1px solid #555;border-radius:4px;padding:3px 8px;cursor:pointer;font:inherit;line-height:1.2}
.bkp button:hover:not(:disabled){background:#444}
.bkp button:disabled{opacity:.4;cursor:default}
.bkp button.pri{background:#2d5a8a;border-color:#3d6fa5;color:#fff}
.bkp button.danger:hover:not(:disabled){background:#6a2a2a}
.bkp .stage{flex:1;min-height:150px;position:relative;display:flex;align-items:center;justify-content:center;background:#121212;border-radius:6px;overflow:hidden}
.bkp .stage img{max-width:100%;max-height:100%;object-fit:contain;user-select:none;-webkit-user-drag:none}
.bkp .stage.onkf{box-shadow:inset 0 0 0 2px #e0a030}
.bkp .badge{position:absolute;left:6px;top:6px;background:rgba(0,0,0,.7);padding:2px 7px;border-radius:10px;font-variant-numeric:tabular-nums}
.bkp .badge.kf{left:auto;right:6px;background:#e0a030;color:#111;font-weight:600}
.bkp .empty{color:#888;text-align:center;padding:16px;line-height:1.7}
.bkp .ctl{display:flex;gap:4px;align-items:center}
.bkp .ctl .sp{flex:1}
.bkp .time{font-variant-numeric:tabular-nums;color:#bbb;min-width:110px;text-align:center}
.bkp .tl{position:relative;height:26px;background:#1d1d1d;border:1px solid #333;border-radius:5px;cursor:pointer;touch-action:none}
.bkp .tl .fill{position:absolute;left:0;top:0;bottom:0;background:#2a3a4d;border-radius:5px 0 0 5px;pointer-events:none}
.bkp .tl .head{position:absolute;top:-2px;bottom:-2px;width:2px;margin-left:-1px;background:#fff;pointer-events:none}
.bkp .tl .mk{position:absolute;top:3px;width:10px;height:18px;margin-left:-5px;border-radius:3px;background:#e0a030;border:1px solid #111;cursor:pointer}
.bkp .tl .mk.done{background:#3fbf6a}
.bkp .tl .mk.cur{outline:2px solid #fff}
.bkp .sec{display:flex;align-items:center;gap:6px;color:#999;font-size:11px;text-transform:uppercase;letter-spacing:.05em}
.bkp .sec .n{color:#ccc}
.bkp .row{display:flex;gap:6px;overflow-x:auto;padding-bottom:3px;min-height:96px}
.bkp .card{flex:0 0 auto;display:flex;flex-direction:column;gap:3px;background:#1d1d1d;border:1px solid #383838;border-radius:6px;padding:4px}
.bkp .card.cur{border-color:#e0a030}
.bkp .card .pair{display:flex;gap:3px}
.bkp .thumb{width:72px;height:54px;border-radius:4px;background:#111 center/cover no-repeat;position:relative;cursor:pointer;flex:0 0 auto}
.bkp .thumb .lbl{position:absolute;left:2px;bottom:2px;font-size:9px;background:rgba(0,0,0,.7);padding:0 3px;border-radius:2px}
.bkp .drop{white-space:pre-line;width:72px;height:54px;border:1px dashed #555;border-radius:4px;display:flex;align-items:center;justify-content:center;text-align:center;color:#888;font-size:10px;cursor:pointer;line-height:1.2;flex:0 0 auto}
.bkp .drop:hover,.bkp .drop.over{border-color:#3fbf6a;color:#cfc}
.bkp .card .foot{display:flex;align-items:center;gap:4px;font-size:11px;font-variant-numeric:tabular-nums}
.bkp .card .foot .sp{flex:1}
.bkp .x{color:#999;cursor:pointer;padding:0 3px;border-radius:3px}.bkp .x:hover{color:#f66;background:#2a1a1a}
.bkp .pics{display:flex;gap:6px}
.bkp .pic{position:relative}
.bkp .pic .thumb{width:60px;height:60px}
.bkp .pic .drop{width:60px;height:60px}
.bkp .pic .x{position:absolute;right:1px;top:1px;background:rgba(0,0,0,.7)}
.bkp .pic .tag{font-size:10px;color:#888;text-align:center}
.bkp .hint{color:#777;font-size:11px}.bkp .err{color:#f77;font-size:11px}
`;
document.head.appendChild(css);
}
function Panel(io) {
const plan = reactive(parsePlan(io.get()));
const strip = ref(null); // {subfolder, type, files, frames, count, fps, width, height} from the last run
const pos = ref(0); // index into strip.frames
const playing = ref(false);
const err = ref("");
const over = ref(""); // drop target being hovered
let timer = null;
const save = () => io.set(JSON.stringify({
keyframes: [...plan.keyframes].sort((a, b) => a.frame - b.frame),
pictures: plan.pictures.map(p => p || null),
}));
watch(plan, save, { deep: true });
const count = computed(() => strip.value?.count || 0);
const fps = computed(() => strip.value?.fps || io.fps() || 24);
const frame = computed(() => strip.value ? strip.value.frames[pos.value] : 0);
const sorted = computed(() => [...plan.keyframes].sort((a, b) => a.frame - b.frame));
const curKf = computed(() => plan.keyframes.find(k => k.frame === frame.value) || null);
const nearest = f => { // strip index of the frame nearest to f
const fr = strip.value?.frames || [];
let best = 0;
for (let i = 1; i < fr.length; i++) if (Math.abs(fr[i] - f) < Math.abs(fr[best] - f)) best = i;
return best;
};
const frameUrl = f => strip.value ? tempUrl(strip.value, strip.value.files[nearest(f)]) : "";
const fmt = f => `${(f / fps.value).toFixed(2)}s`;
const stop = () => { playing.value = false; if (timer) { clearInterval(timer); timer = null; } };
const play = () => {
if (!strip.value) return;
if (playing.value) return stop();
playing.value = true;
const fr = strip.value.frames;
const step = fr.length > 1 ? (fr[fr.length - 1] - fr[0]) / (fr.length - 1) : 1;
timer = setInterval(() => { pos.value = (pos.value + 1) % fr.length; }, 1000 * step / fps.value);
};
const seekIdx = i => { if (strip.value) pos.value = Math.max(0, Math.min(strip.value.frames.length - 1, i)); };
const seekFrame = f => seekIdx(nearest(f));
onBeforeUnmount(stop);
const addHere = () => {
if (!strip.value || curKf.value) return;
plan.keyframes.push({ frame: frame.value, edit: null });
};
const removeKf = k => { plan.keyframes.splice(plan.keyframes.indexOf(k), 1); };
const jumpKf = d => {
const list = sorted.value;
if (!list.length) return;
const f = frame.value;
const t = d > 0 ? list.find(k => k.frame > f) || list[0] : [...list].reverse().find(k => k.frame < f) || list[list.length - 1];
seekFrame(t.frame);
};
const withFile = async (file, apply) => {
if (!file) return;
err.value = "";
try { apply(await upload(file)); } catch (e) { err.value = String(e.message || e); }
};
const dropProps = (key, apply) => ({
class: ["drop", over.value === key ? "over" : ""],
onDragover: e => { e.preventDefault(); e.stopPropagation(); over.value = key; },
onDragleave: () => { if (over.value === key) over.value = ""; },
onDrop: e => { e.preventDefault(); e.stopPropagation(); over.value = ""; withFile(e.dataTransfer?.files?.[0], apply); },
onClick: async e => { e.stopPropagation(); withFile(await pickFile(), apply); },
});
// timeline: click / drag to seek
const tlDown = e => {
if (!strip.value) return;
const el = e.currentTarget, r = el.getBoundingClientRect();
const go = ev => {
const x = Math.max(0, Math.min(1, (ev.clientX - r.left) / r.width));
seekFrame(Math.round(x * Math.max(0, count.value - 1)));
};
stop(); go(e);
el.setPointerCapture?.(e.pointerId);
const move = ev => go(ev);
const up = () => { el.removeEventListener("pointermove", move); el.removeEventListener("pointerup", up); };
el.addEventListener("pointermove", move); el.addEventListener("pointerup", up);
};
const pct = f => `${count.value > 1 ? (100 * f / (count.value - 1)) : 0}%`;
io.expose({
executed: d => {
if (!d) return;
const keep = frame.value;
strip.value = d;
seekFrame(keep);
for (const f of d.files) { const im = new Image(); im.src = tempUrl(d, f); } // preload for smooth playback
},
reload: () => { Object.assign(plan, parsePlan(io.get())); },
});
return () => {
const s = strip.value;
const cards = sorted.value.map((k, i) => h("div", { class: ["card", k.frame === frame.value ? "cur" : ""], key: k.frame }, [
h("div", { class: "pair" }, [
h("div", { class: "thumb", title: "source frame: click to go there", style: s ? { backgroundImage: `url("${frameUrl(k.frame)}")` } : {},
onClick: () => { stop(); seekFrame(k.frame); } }, [h("span", { class: "lbl" }, "source")]),
k.edit
? h("div", { class: "thumb", title: "edited frame: click to replace", style: { backgroundImage: `url("${inputUrl(k.edit)}")` },
onClick: async () => withFile(await pickFile(), p => { k.edit = p; }) }, [h("span", { class: "lbl" }, "edited")])
: h("div", { ...dropProps(`kf${k.frame}`, p => { k.edit = p; }), title: "drop or click: the edited frame for this time (optional)" },
"drop edited\nframe"),
]),
h("div", { class: "foot" }, [
h("span", null, `#${i + 1} · ${fmt(k.frame)}`), h("span", { class: "sp" }),
k.edit ? h("span", { class: "x", title: "clear the edited frame", onClick: () => { k.edit = null; } }, "⟲") : null,
h("span", { class: "x", title: "remove this keyframe", onClick: () => removeKf(k) }, "✕"),
]),
]));
const linked = j => !!io.node.inputs?.find(x => x.name === `picture_${j}`)?.link;
const pics = plan.pictures.map((p, j) => h("div", { class: "pic", key: j }, [
linked(j + 1)
? h("div", { class: "drop", title: "this picture comes from the connected input" }, "input")
: p
? h("div", { class: "thumb", style: { backgroundImage: `url("${inputUrl(p)}")` }, title: "click to replace",
onClick: async () => withFile(await pickFile(), q => { plan.pictures[j] = q; }) })
: h("div", { ...dropProps(`pic${j}`, q => { plan.pictures[j] = q; }), title: "drop or click: a reference picture" }, "+ picture"),
p && !linked(j + 1) ? h("span", { class: "x", title: "remove", onClick: () => { plan.pictures[j] = null; } }, "✕") : null,
h("div", { class: "tag" }, `Picture ${j + 1}`),
]));
return h("div", { class: "bkp" }, [
h("div", { class: ["stage", curKf.value ? "onkf" : ""] }, s
? [h("img", { src: tempUrl(s, s.files[pos.value]), draggable: false }),
h("span", { class: "badge" }, `${fmt(frame.value)} · frame ${frame.value}/${count.value - 1}`),
curKf.value ? h("span", { class: "badge kf" }, curKf.value.edit ? "keyframe ✓" : "keyframe") : null]
: [h("div", { class: "empty" }, ["Queue this node once to load the guide frames.", h("br"),
"Then scrub, press + Keyframe, and drop the edited frames on the cards."])]),
h("div", { class: "ctl" }, [
h("button", { disabled: !s, title: "previous keyframe", onClick: () => { stop(); jumpKf(-1); } }, "⏮"),
h("button", { disabled: !s, title: "previous frame", onClick: () => { stop(); seekIdx(pos.value - 1); } }, "◀"),
h("button", { disabled: !s, title: playing.value ? "pause" : "play", onClick: play }, playing.value ? "⏸" : "▶"),
h("button", { disabled: !s, title: "next frame", onClick: () => { stop(); seekIdx(pos.value + 1); } }, "▶|"),
h("button", { disabled: !s, title: "next keyframe", onClick: () => { stop(); jumpKf(1); } }, "⏭"),
h("span", { class: "time" }, s ? `${fmt(frame.value)} / ${fmt(count.value - 1)}` : "--"),
h("span", { class: "sp" }),
curKf.value
? h("button", { class: "danger", onClick: () => removeKf(curKf.value) }, "- Keyframe")
: h("button", { class: "pri", disabled: !s, onClick: addHere, title: "mark the current frame as a keyframe" }, "+ Keyframe"),
]),
h("div", { class: "tl", onPointerdown: tlDown }, s ? [
h("div", { class: "fill", style: { width: pct(frame.value) } }),
...sorted.value.map(k => h("div", { class: ["mk", k.edit ? "done" : "", k.frame === frame.value ? "cur" : ""], key: k.frame,
style: { left: pct(k.frame) }, title: `${fmt(k.frame)}${k.edit ? " (edited)" : " (no edited frame yet)"}`,
onPointerdown: e => { e.stopPropagation(); stop(); seekFrame(k.frame); } })),
h("div", { class: "head", style: { left: pct(frame.value) } }),
] : []),
h("div", { class: "sec" }, ["Keyframes ", h("span", { class: "n" }, String(plan.keyframes.length)),
plan.keyframes.length ? h("span", null, ` · ${plan.keyframes.filter(k => k.edit).length} edited`) : null]),
h("div", { class: "row" }, cards.length ? cards : [h("div", { class: "hint" }, "No keyframes yet.")]),
h("div", { class: "sec" }, "References"),
h("div", { class: "pics" }, pics),
err.value ? h("div", { class: "err" }, err.value) : h("div", { class: "hint" }, "Outputs follow the plan on the next run. Without edited frames, source_frames are ready to edit."),
]);
};
}
app.registerExtension({
name: "BFSNodes.KeyframePlanner",
async nodeCreated(node) {
if (node.comfyClass !== "BFSKeyframePlanner") return;
styles();
const widget = node.widgets?.find(w => w.name === "plan");
if (widget) { // the JSON stays a widget (saved with the workflow) but is never shown
widget.type = "hidden"; widget.hidden = true; widget.computeSize = () => [0, -4];
widget.options = { ...(widget.options || {}), hidden: true, serialize: true };
widget.serializeValue = () => widget.value;
}
const fpsW = node.widgets?.find(w => w.name === "fps");
const host = document.createElement("div");
host.style.cssText = "position:relative;width:100%;height:100%;min-width:440px;min-height:520px";
node.addDOMWidget("bfs_keyframe_planner", "div", host, { serialize: false, hideOnZoom: false, hideInPanel: true });
wheelToCanvas(host);
const handle = {};
createApp({
setup: () => Panel({
node,
get: () => widget?.value ?? "{}",
set: v => { if (widget) widget.value = v; node.graph?.setDirtyCanvas(true); },
fps: () => Number(fpsW?.value) || 24,
expose: o => Object.assign(handle, o),
}),
}).mount(host);
const onExecuted = node.onExecuted;
node.onExecuted = function (output) {
onExecuted?.apply(this, arguments);
handle.executed?.(output?.bfs_kfplan?.[0]);
};
const onConfigure = node.onConfigure;
node.onConfigure = function () {
const r = onConfigure?.apply(this, arguments);
setTimeout(() => handle.reload?.(), 0);
return r;
};
node.size = [Math.max(node.size[0], DEFAULT_W), Math.max(node.size[1], DEFAULT_H)];
},
});
+430
View File
@@ -0,0 +1,430 @@
/**
* BFS Segment Picker: the canvas of the node. It shows the objects SAM 3 found in the image, colored; hover one to
* see it, click to select it. Objects the automatic pass missed are added by text (a SAM 3 prompt), by points
* (keep / exclude), segmented at once by /bfs/segpick/add, or painted with the brush (to grow the mark by hand).
* ➖ Remove (points) and 🧽 Eraser (brush) make the same kinds of objects, cut out of the mark instead of added to it.
*
* All state lives in the hidden `selection` STRING widget (JSON: selected ids and the added entries), so a workflow
* saves, reloads and shares the selection. The objects themselves come from the node's last run (onExecuted).
*/
import { app } from "../../scripts/app.js";
import { api } from "../../scripts/api.js";
import { wheelToCanvas } from "./bfs_wheel.js";
import { createApp, h, ref, reactive, computed, watch, nextTick, onMounted, onBeforeUnmount } from "./vendor/vue.esm-browser.prod.mjs";
// same order as PALETTE in bfs_segment_picker.py, so the panel and the overlay output agree
const PALETTE = [[230, 70, 60], [45, 125, 225], [50, 180, 90], [235, 205, 35], [205, 55, 180], [30, 200, 205],
[240, 130, 35], [135, 75, 210], [120, 200, 60], [225, 95, 150], [80, 160, 230], [190, 150, 90]];
const CUT = [25, 25, 25]; // remove / eraser objects: a dark hole, unlike any PALETTE color
const rgb = (c, a = 1) => `rgba(${c[0]},${c[1]},${c[2]},${a})`;
const DEFAULT_W = 760, DEFAULT_H = 820;
const fileUrl = f => api.apiURL(`/view?filename=${encodeURIComponent(f.filename)}&subfolder=${encodeURIComponent(f.subfolder || "")}&type=${f.type || "temp"}`);
function parseState(text) {
let s = {};
try { s = JSON.parse(text || "{}") || {}; } catch { s = {}; }
return { selected: Array.isArray(s.selected) ? s.selected.map(String) : [], added: Array.isArray(s.added) ? s.added : [],
next: Number(s.next) || (Array.isArray(s.added) ? s.added.length + 1 : 1) };
}
const STYLE_ID = "bfs-segpick-style";
export function styles() {
if (document.getElementById(STYLE_ID)) return;
const css = document.createElement("style");
css.id = STYLE_ID;
css.textContent = `
.bsp{position:absolute;inset:0;display:flex;flex-direction:column;gap:6px;padding:6px;box-sizing:border-box;font:12px system-ui,sans-serif;color:#ddd}
.bsp .bar{display:flex;flex-wrap:wrap;gap:6px;align-items:center}
.bsp button{background:#333;color:#ddd;border:1px solid #555;border-radius:4px;padding:3px 8px;cursor:pointer;font:inherit}
.bsp button:hover:not(:disabled){background:#444}
.bsp button:disabled{opacity:.45;cursor:default}
.bsp button.pri{background:#2d5a8a;border-color:#3d6fa5;color:#fff}
.bsp button.on{background:#7a3d8a;border-color:#9a55aa;color:#fff}
.bsp input[type=text]{background:#222;color:#ddd;border:1px solid #555;border-radius:4px;padding:3px 6px;font:inherit;flex:1;min-width:120px}
.bsp .stage{flex:1;min-height:200px;position:relative;display:flex;align-items:center;justify-content:center;background:#151515;border-radius:4px;overflow:hidden}
.bsp .frame{position:relative}
.bsp .frame img,.bsp .frame canvas{position:absolute;inset:0;width:100%;height:100%;user-select:none}
.bsp .frame canvas{cursor:pointer}
.bsp .frame.points canvas,.bsp .frame.remove canvas{cursor:crosshair}
.bsp .frame.brush canvas,.bsp .frame.eraser canvas{cursor:none}
.bsp .chip.cut{border-style:dashed}
.bsp .frame.remove .pt.pos{background:#e74c3c}.bsp .frame.remove .pt.neg{background:#2ecc71}
.bsp input[type=range]{width:90px}
.bsp .pt{position:absolute;width:12px;height:12px;margin:-6px 0 0 -6px;border-radius:50%;border:2px solid #fff;pointer-events:auto;cursor:pointer;box-shadow:0 0 3px #000}
.bsp .pt.pos{background:#2ecc71}.bsp .pt.neg{background:#e74c3c}
.bsp .empty{color:#888;text-align:center;padding:20px;line-height:1.6}
.bsp .tip{position:absolute;left:6px;top:6px;background:rgba(0,0,0,.75);padding:2px 6px;border-radius:3px;pointer-events:none}
.bsp .list{display:flex;flex-wrap:wrap;gap:4px;max-height:96px;overflow:auto}
.bsp .chip{display:flex;align-items:center;gap:4px;padding:2px 6px;border-radius:10px;border:1px solid #444;background:#222;cursor:pointer}
.bsp .chip.sel{border-color:#fff;background:#3a3a3a}
.bsp .chip .sw{width:10px;height:10px;border-radius:2px}
.bsp .chip .x{color:#999;padding:0 2px}.bsp .chip .x:hover{color:#f66}
.bsp .hint{color:#888}.bsp .err{color:#f77}
`;
document.head.appendChild(css);
}
/** One loaded object: a colored canvas to draw and its mask alpha for hit tests, at the preview resolution. */
function loadMask(o, color) {
return new Promise((resolve, reject) => {
const img = new Image();
img.crossOrigin = "anonymous";
img.onload = () => {
const w = img.naturalWidth, hh = img.naturalHeight;
const c = document.createElement("canvas"); c.width = w; c.height = hh;
const g = c.getContext("2d");
g.drawImage(img, 0, 0);
const alpha = g.getImageData(0, 0, w, hh).data;
const hit = new Uint8Array(w * hh);
for (let i = 0; i < hit.length; i++) hit[i] = alpha[i * 4 + 3] > 127 ? 1 : 0;
g.globalCompositeOperation = "source-in";
g.fillStyle = rgb(color); g.fillRect(0, 0, w, hh);
// the outline: mask pixels with a background pixel within r, so it stays visible once scaled down
const r = Math.max(1, Math.round(Math.max(w, hh) / 400));
const e = document.createElement("canvas"); e.width = w; e.height = hh;
const eg = e.getContext("2d"), px = eg.createImageData(w, hh), [cr, cg, cb] = color;
for (let y = 0; y < hh; y++) for (let x = 0; x < w; x++) {
if (!hit[y * w + x]) continue;
let edge = false;
for (let dy = -r; dy <= r && !edge; dy++) for (let dx = -r; dx <= r; dx++) {
const xx = x + dx, yy = y + dy;
if (xx < 0 || yy < 0 || xx >= w || yy >= hh || !hit[yy * w + xx]) { edge = true; break; }
}
if (edge) { const i = (y * w + x) * 4; px.data[i] = cr; px.data[i + 1] = cg; px.data[i + 2] = cb; px.data[i + 3] = 255; }
}
eg.putImageData(px, 0, 0);
resolve({ canvas: c, edge: e, hit, w, h: hh });
};
img.onerror = reject;
img.src = fileUrl(o.mask);
});
}
export function Panel(io, options = {}) { // exported: BFS Video Segment Picker shows one per frame
const showEraser = options.showEraser !== false;
const showRemove = options.showRemove !== false;
const state = reactive(parseState(io.get()));
const data = ref(null); // {session, width, height, image, objects} from the last run
const loaded = new Map(); // object id -> loadMask result (not reactive: canvases and big arrays)
const ready = ref(0); // bumps when masks finish loading, to redraw
const hover = ref(null);
const mode = ref("select"); // "select" | "points" | "brush" | "remove" (points) | "eraser" (brush)
const points = ref([]); // the point entry being built
const strokes = ref([]); // the brush entry being built: [{r, pts: [[x, y], ...]}], relative to the image
const brush = ref(.02); // brush radius, relative to the image width
let painting = false, cursor = null;
const text = ref("");
const busy = ref(""); const error = ref("");
const stageEl = ref(null); const canvasEl = ref(null);
const fit = reactive({ w: 0, h: 0 });
const save = () => io.set(JSON.stringify({ selected: state.selected, added: state.added, next: state.next }));
const entryId = kind => `${options.entryPrefix || ""}${kind}${state.next}`;
const objects = computed(() => data.value?.objects || []);
const colorOf = id => {
const i = Math.max(0, objects.value.findIndex(o => o.id === id));
return objects.value[i]?.remove ? CUT : PALETTE[i % PALETTE.length];
};
const isSel = id => state.selected.includes(id);
const pointing = () => mode.value === "points" || mode.value === "remove";
const brushing = () => mode.value === "brush" || mode.value === "eraser";
const cutting = () => mode.value === "remove" || mode.value === "eraser";
function setMode(m) {
if (m !== mode.value) { points.value = []; strokes.value = []; } // unfinished points / strokes mean something else in another mode
mode.value = m; hover.value = null;
}
async function loadAll() {
const objs = objects.value;
await Promise.all(objs.map(async (o, i) => {
const key = `${o.id}|${o.mask.filename}`;
if (loaded.get(o.id)?.key === key) return;
try { loaded.set(o.id, { key, ...(await loadMask(o, o.remove ? CUT : (options.maskColor || PALETTE[i % PALETTE.length]))) }); } catch { /* shown as missing */ }
}));
ready.value++;
}
function relayout() {
const el = stageEl.value, d = data.value;
if (!el || !d) return;
const W = el.clientWidth - 2, H = el.clientHeight - 2, s = Math.min(W / d.width, H / d.height);
fit.w = Math.max(1, Math.floor(d.width * s)); fit.h = Math.max(1, Math.floor(d.height * s));
}
function draw() {
const c = canvasEl.value, d = data.value;
if (!c || !d) return;
const first = loaded.values().next().value;
const W = first?.w || fit.w, H = first?.h || fit.h;
if (c.width !== W || c.height !== H) { c.width = W; c.height = H; }
const g = c.getContext("2d");
g.clearRect(0, 0, W, H);
// biggest first, so small objects stay visible on top of the ones around them
const order = [...objects.value].sort((a, b) => b.area - a.area);
// the image itself is never tinted: an object is an outline, filled only while hovered (light) or selected
for (const o of order) {
const m = loaded.get(o.id);
if (!m) continue;
const sel = isSel(o.id), hov = hover.value === o.id;
if (sel || hov) { g.globalAlpha = sel ? (hov ? .6 : .45) : .3; g.drawImage(m.canvas, 0, 0, W, H); }
if (mode.value !== "select" && !sel && !hov) continue;
g.globalAlpha = sel || hov ? 1 : .7;
g.drawImage(m.edge, 0, 0, W, H);
}
g.globalAlpha = 1;
if (!brushing()) return;
g.strokeStyle = g.fillStyle = cutting() ? "rgba(20,20,20,.6)" : "rgba(255,0,255,.55)"; g.lineCap = g.lineJoin = "round";
for (const st of strokes.value) {
const [x0, y0] = st.pts[0];
g.beginPath(); g.arc(x0 * W, y0 * H, st.r * W, 0, 2 * Math.PI); g.fill();
if (st.pts.length < 2) continue;
g.lineWidth = 2 * st.r * W; g.beginPath(); g.moveTo(x0 * W, y0 * H);
for (const [x, y] of st.pts.slice(1)) g.lineTo(x * W, y * H);
g.stroke();
}
if (cursor) {
g.lineWidth = Math.max(1, W / 500); g.strokeStyle = "#fff";
g.beginPath(); g.arc(cursor.x * W, cursor.y * H, brush.value * W, 0, 2 * Math.PI); g.stroke();
}
}
watch([ready, hover, mode, brush, () => strokes.value.length, () => [...state.selected], () => fit.w, () => fit.h], () => nextTick(draw));
function at(e) {
const r = canvasEl.value.getBoundingClientRect();
return { x: Math.min(1, Math.max(0, (e.clientX - r.left) / r.width)), y: Math.min(1, Math.max(0, (e.clientY - r.top) / r.height)) };
}
function objectAt(p) {
let best = null;
for (const o of objects.value) {
const m = loaded.get(o.id);
if (!m) continue;
const i = Math.min(m.h - 1, Math.floor(p.y * m.h)) * m.w + Math.min(m.w - 1, Math.floor(p.x * m.w));
if (m.hit[i] && (!best || o.area < best.area)) best = o; // the smallest one under the mouse: inner objects stay pickable
}
return best;
}
function toggle(id) {
state.selected = isSel(id) ? state.selected.filter(i => i !== id) : [...state.selected, id];
save();
}
function onPointer(e) {
if (!data.value) return;
e.preventDefault(); e.stopPropagation();
const p = at(e);
if (pointing()) {
points.value = [...points.value, { x: +p.x.toFixed(4), y: +p.y.toFixed(4), label: (e.button === 2 || e.shiftKey) ? 0 : 1 }];
return;
}
if (brushing()) {
canvasEl.value.setPointerCapture(e.pointerId);
painting = true;
strokes.value = [...strokes.value, { r: +brush.value.toFixed(4), pts: [[+p.x.toFixed(4), +p.y.toFixed(4)]] }];
return;
}
const o = objectAt(p);
if (options.autoPointOnClick && (!o || !o.entry)) {
add({ id: entryId("p"), points: [{ x: +p.x.toFixed(4), y: +p.y.toFixed(4), label: 1 }] });
} else if (o) toggle(o.id);
}
function onMove(e) {
if (data.value && brushing()) {
cursor = at(e);
const pts = strokes.value.at(-1)?.pts, last = pts?.at(-1);
// a point every ~quarter radius keeps the stroke smooth without bloating the saved JSON
if (painting && Math.hypot(cursor.x - last[0], cursor.y - last[1]) > brush.value / 4) pts.push([+cursor.x.toFixed(4), +cursor.y.toFixed(4)]);
draw();
return;
}
if (!data.value || pointing()) { hover.value = null; return; }
hover.value = objectAt(at(e))?.id ?? null;
}
async function add(entry) {
if (!data.value) { error.value = "Run the node first (▶ Segment)."; return; }
busy.value = entry.text ? `Finding “${entry.text}”…` : entry.strokes ? "Adding the painted area…" : "Segmenting the points…"; error.value = "";
try {
const r = await api.fetchApi("/bfs/segpick/add", { method: "POST", headers: { "Content-Type": "application/json" },
body: JSON.stringify({ session: data.value.session, entry }) });
const j = await r.json();
if (!r.ok || j.error) throw new Error(j.error || r.statusText);
if (!j.objects.length) { error.value = entry.text ? `Nothing found for “${entry.text}”: try other words or a lower threshold.` : "The points gave an empty mask."; return; }
state.added = [...state.added.filter(a => a.id !== entry.id), entry];
state.next = Math.max(state.next, Number(entry.id.match(/(\d+)$/)?.[1] || 0) + 1);
data.value = { ...data.value, objects: [...data.value.objects.filter(o => o.entry !== entry.id), ...j.objects] };
state.selected = [...state.selected, ...j.objects.map(o => o.id).filter(id => !isSel(id))]; // what you add is what you want
save();
await loadAll();
} catch (err) {
error.value = String(err.message || err);
} finally { busy.value = ""; }
}
const addText = () => { const t = text.value.trim(); if (t) { add({ id: entryId("t"), text: t }); text.value = ""; } };
const addPoints = async () => {
if (!points.value.some(p => p.label)) return;
await add(cutting() ? { id: entryId("r"), remove: true, points: points.value } : { id: entryId("p"), points: points.value });
if (!error.value) setMode("select");
};
const addBrush = async () => {
if (!strokes.value.length) return;
const stroke = { image: data.value.image_key, strokes: strokes.value };
await add(cutting() ? { id: entryId("e"), remove: true, ...stroke } : { id: entryId("b"), ...stroke });
if (!error.value) setMode("select");
};
function removeEntry(entryId) {
state.added = state.added.filter(a => a.id !== entryId);
const gone = objects.value.filter(o => o.entry === entryId).map(o => o.id);
state.selected = state.selected.filter(i => !gone.includes(i));
if (data.value) data.value = { ...data.value, objects: data.value.objects.filter(o => o.entry !== entryId) };
gone.forEach(i => loaded.delete(i));
save(); ready.value++;
}
async function runNode() {
error.value = "";
try {
const p = await app.graphToPrompt();
await api.queuePrompt(0, p, { partialExecutionTargets: [String(io.node.id)] });
busy.value = "Segmenting…";
} catch (err) { error.value = String(err.message || err); }
}
io.expose({
executed: d => {
busy.value = "";
if (!d) return;
if (data.value?.session !== d.session) loaded.clear();
if (data.value && data.value.image_key !== d.image_key) strokes.value = [];
// brush entries belong to the image they were painted on: another image drops them (text and points segment again)
const stale = state.added.filter(a => a.strokes && a.image && a.image !== d.image_key).map(a => a.id);
if (stale.length) { state.added = state.added.filter(a => !stale.includes(a.id)); save(); }
data.value = d;
// ids of objects that no longer exist (another image, other concepts) leave the selection
const ids = new Set(d.objects.map(o => o.id));
const keep = state.selected.filter(i => ids.has(i));
if (keep.length !== state.selected.length) { state.selected = keep; save(); }
nextTick(() => { relayout(); loadAll(); });
},
reload: () => { Object.assign(state, parseState(io.get())); ready.value++; },
});
let ro;
const done = () => { busy.value = ""; }; // a failed or fully cached run sends no output to this node
onMounted(() => {
ro = new ResizeObserver(() => relayout()); ro.observe(stageEl.value);
api.addEventListener("execution_error", done); api.addEventListener("execution_success", done);
});
onBeforeUnmount(() => {
ro?.disconnect();
api.removeEventListener("execution_error", done); api.removeEventListener("execution_success", done);
});
watch(data, () => nextTick(relayout));
return () => {
const d = data.value, objs = objects.value;
const hovered = objs.find(o => o.id === hover.value);
return h("div", { class: "bsp", onKeydown: e => e.stopPropagation() }, [
h("div", { class: "bar" }, [
h("button", { class: "pri", onClick: runNode, title: "Run this node only (and what feeds it) to segment the image" }, d ? "↻ Segment again" : "▶ Segment"),
h("button", { class: mode.value === "select" ? "on" : "", onClick: () => setMode("select") }, "👆 Select"),
h("button", { class: mode.value === "points" ? "on" : "", disabled: !d, onClick: () => setMode("points"),
title: "Place points on something the automatic pass missed" }, "📍 Points"),
h("button", { class: mode.value === "brush" ? "on" : "", disabled: !d, onClick: () => setMode("brush"),
title: "Paint extra area into the mark by hand (e.g. hair SAM left out)" }, "🖌 Brush"),
showRemove ? h("button", { class: mode.value === "remove" ? "on" : "", disabled: !d, onClick: () => setMode("remove"),
title: "Place points on something to cut out of the mark (SAM segments it)" }, "➖ Remove") : null,
showEraser ? h("button", { class: mode.value === "eraser" ? "on" : "", disabled: !d, onClick: () => setMode("eraser"),
title: "Paint an area to subtract manually from the selected mask" }, "🧽 Eraser") : null,
h("span", { class: "hint" }, d ? `${objs.length} objects · ${state.selected.length} selected` : ""),
h("span", { style: "flex:1" }),
h("button", { disabled: !objs.length, onClick: () => { state.selected = objs.map(o => o.id); save(); } }, "All"),
h("button", { disabled: !state.selected.length, onClick: () => { state.selected = []; save(); } }, "None"),
]),
h("div", { class: "bar" }, [
h("input", { type: "text", value: text.value, placeholder: "Find by text (SAM 3 prompt), e.g. red umbrella, the man on the left",
disabled: !d, onInput: e => { text.value = e.target.value; }, onKeydown: e => { e.stopPropagation(); if (e.key === "Enter") addText(); } }),
h("button", { disabled: !d || !text.value.trim() || !!busy.value, onClick: addText }, "🔎 Find"),
pointing() ? [
h("button", { class: "pri", disabled: !points.value.some(p => p.label) || !!busy.value, onClick: addPoints },
`${cutting() ? "✓ Remove this" : "✓ Create object"} (${points.value.length})`),
h("button", { disabled: !points.value.length, onClick: () => { points.value = []; } }, "Clear points"),
] : null,
brushing() ? [
h("input", { type: "range", min: .003, max: .1, step: .001, value: brush.value, title: "Brush size",
onInput: e => { brush.value = +e.target.value; } }),
h("button", { class: "pri", disabled: !strokes.value.length || !!busy.value, onClick: addBrush },
`${cutting() ? "✓ Erase painted area" : "✓ Add painted area"} (${strokes.value.length})`),
h("button", { disabled: !strokes.value.length, onClick: () => { strokes.value = strokes.value.slice(0, -1); } }, "Undo"),
h("button", { disabled: !strokes.value.length, onClick: () => { strokes.value = []; } }, "Clear"),
] : null,
]),
h("div", { class: "stage", ref: stageEl, onContextmenu: e => e.preventDefault() }, d ? [
h("div", { class: ["frame", mode.value !== "select" && mode.value], style: `width:${fit.w}px;height:${fit.h}px` }, [
h("img", { src: fileUrl(d.image), draggable: false }),
h("canvas", { ref: canvasEl, onPointerdown: onPointer, onPointermove: onMove, onPointerup: () => { painting = false; },
onPointerleave: () => { hover.value = null; cursor = null; if (!painting) draw(); } }),
...points.value.map((p, k) => h("div", { class: ["pt", p.label ? "pos" : "neg"], style: `left:${p.x * 100}%;top:${p.y * 100}%`,
title: "click to remove", onPointerdown: e => { e.stopPropagation(); points.value = points.value.filter((_, j) => j !== k); } })),
]),
hovered ? h("div", { class: "tip" }, `${hovered.id} · ${hovered.label}${hovered.score != null ? ` · ${hovered.score}` : ""}`) : null,
mode.value === "points" ? h("div", { class: "tip" }, "click = keep · right-click or shift+click = exclude · click a point to remove it") : null,
mode.value === "remove" ? h("div", { class: "tip" }, "click on what to cut out of the mark · right-click or shift+click = keep it in · then ✓ Remove this") : null,
mode.value === "brush" ? h("div", { class: "tip" }, "paint what the mark should also cover · then ✓ Add painted area") : null,
mode.value === "eraser" ? h("div", { class: "tip" }, "paint what to cut out of the mark · then ✓ Erase painted area") : null,
] : h("div", { class: "empty" }, ["Connect an image and press ▶ Segment (or run the workflow).", h("br"),
"Then click objects to select them; add what was missed by text or 📍 points."])),
busy.value ? h("div", { class: "hint" }, busy.value) : null,
error.value ? h("div", { class: "err" }, error.value) : null,
objs.length ? h("div", { class: "list" }, objs.map(o => h("div", {
class: ["chip", isSel(o.id) && "sel", o.remove && "cut"], title: `${o.label}${o.score != null ? ` · score ${o.score}` : ""} · ${(o.area * 100).toFixed(1)}% of the image`,
onClick: () => toggle(o.id), onPointerenter: () => { hover.value = o.id; }, onPointerleave: () => { hover.value = null; },
}, [
h("span", { class: "sw", style: `background:${rgb(colorOf(o.id))}` }),
`${o.remove ? "➖ " : ""}${o.id} ${o.label}`,
o.entry ? h("span", { class: "x", title: "remove this added object", onClick: e => { e.stopPropagation(); removeEntry(o.entry); } }, "×") : null,
]))) : null,
d ? h("div", { class: "hint" }, "The outputs follow the selection on the next run.") : null,
]);
};
}
app.registerExtension({
name: "BFSNodes.SegmentPicker",
async nodeCreated(node) {
if (node.comfyClass !== "BFSSegmentPicker") return;
styles();
const widget = node.widgets?.find(w => w.name === "selection");
if (widget) { // the JSON stays a widget (saved with the workflow) but is never shown
widget.type = "hidden"; widget.hidden = true; widget.computeSize = () => [0, -4];
widget.options = { ...(widget.options || {}), hidden: true, serialize: true };
widget.serializeValue = () => widget.value;
}
const host = document.createElement("div");
host.style.cssText = "position:relative;width:100%;height:100%;min-width:420px;min-height:460px";
// hideInPanel: the side panel's legacy renderer writes its own width into the widget, shrinking the canvas in the node
node.addDOMWidget("bfs_segment_picker", "div", host, { serialize: false, hideOnZoom: false, hideInPanel: true });
wheelToCanvas(host);
const handle = {};
createApp({
setup: () => Panel({
node,
get: () => widget?.value ?? "{}",
set: v => { if (widget) widget.value = v; node.graph?.setDirtyCanvas(true); },
expose: o => Object.assign(handle, o),
}),
}).mount(host);
const onExecuted = node.onExecuted;
node.onExecuted = function (output) {
onExecuted?.apply(this, arguments);
handle.executed?.(output?.bfs_segpick?.[0]);
};
const onConfigure = node.onConfigure;
node.onConfigure = function () {
const r = onConfigure?.apply(this, arguments);
setTimeout(() => handle.reload?.(), 0);
return r;
};
node.size = [Math.max(node.size[0], DEFAULT_W), Math.max(node.size[1], DEFAULT_H)];
},
});
+46 -9
View File
@@ -9,6 +9,8 @@
* whatever you do in the player (dragging the divider, pausing, scrubbing) until ■ stop. Both draw the two videos into * whatever you do in the player (dragging the divider, pausing, scrubbing) until ■ stop. Both draw the two videos into
* a canvas and record it in the browser, in real time. The browser's file (WebM, or a fragmented MP4 many apps won't * a canvas and record it in the browser, in real time. The browser's file (WebM, or a fragmented MP4 many apps won't
* play) is then converted by ComfyUI to the chosen format: MP4 (H.264), a small MP4 (under 10 MB) or a GIF. * play) is then converted by ComfyUI to the chosen format: MP4 (H.264), a small MP4 (under 10 MB) or a GIF.
* 🖼 PNG downloads the frame on screen as a picture, drawn the same way. With a single frame (two images) the node
* also sends lossless PNGs of guide / target / difference: ⬇ save PNG draws from those, at the compared size.
*/ */
import { app } from "../../scripts/app.js"; import { app } from "../../scripts/app.js";
import { api } from "../../scripts/api.js"; import { api } from "../../scripts/api.js";
@@ -109,6 +111,7 @@ function buildPlayer(host) {
selFormat.title = "Format of what save / rec download. MP4: H.264 + AAC, plays and previews anywhere. MP4 small: " selFormat.title = "Format of what save / rec download. MP4: H.264 + AAC, plays and previews anywhere. MP4 small: "
+ "the same, sized to stay under 10 MB. GIF: no sound, under 10 MB. browser file: the recording as it is, no conversion."; + "the same, sized to stay under 10 MB. GIF: no sound, under 10 MB. browser file: the recording as it is, no conversion.";
const btnRec = el("button", { title: "Record what you do in the player until stop, then download it", onclick: () => (rec.mr ? stopRec() : startRec()) }, "● rec"); const btnRec = el("button", { title: "Record what you do in the player until stop, then download it", onclick: () => (rec.mr ? stopRec() : startRec()) }, "● rec");
const btnShot = el("button", { title: "Download the frame on screen as a PNG, as shown (layout, divider, labels, refs)", onclick: () => snapshot() }, "🖼 PNG");
const chkLabels = el("input", { type: "checkbox" }); const chkLabels = el("input", { type: "checkbox" });
chkLabels.checked = true; chkLabels.checked = true;
chkLabels.addEventListener("change", () => { st.labels = chkLabels.checked; }); chkLabels.addEventListener("change", () => { st.labels = chkLabels.checked; });
@@ -131,7 +134,7 @@ function buildPlayer(host) {
stage, stage,
refBox, refBox,
el("div", { class: "row" }, [btnPrev, btnPlay, btnNext, scrub, btnMute, info]), el("div", { class: "row" }, [btnPrev, btnPlay, btnNext, scrub, btnMute, info]),
el("div", { class: "row" }, [btnSave, btnRec, selFormat, el("label", { title: "Draw the names on the saved / recorded video" }, [chkLabels, "labels"]), lblRefs]), el("div", { class: "row" }, [btnSave, btnShot, btnRec, selFormat, el("label", { title: "Draw the names on the saved / recorded video" }, [chkLabels, "labels"]), lblRefs]),
strip, strip,
notes, notes,
])); ]));
@@ -267,6 +270,9 @@ function buildPlayer(host) {
} }
const last = new Map(); // video -> its last whole frame, drawn while it seeks (a seeking video draws blank) const last = new Map(); // video -> its last whole frame, drawn while it seeks (a seeking video draws blank)
const stills = {}; // source -> loaded lossless PNG (single-frame comparisons only)
const single = () => st.data?.frames === 1;
const still = s => (stills[s]?.complete && stills[s].naturalWidth ? stills[s] : null);
function pic(v) { function pic(v) {
if (v.readyState < 2 || v.seeking) return last.get(v) || v; if (v.readyState < 2 || v.seeking) return last.get(v) || v;
let c = last.get(v); let c = last.get(v);
@@ -306,8 +312,8 @@ function buildPlayer(host) {
function drawVideos(ctx, area) { function drawVideos(ctx, area) {
const cw = ctx.canvas.width, ch = area ?? ctx.canvas.height, m = st.mode, p = st.pos; const cw = ctx.canvas.width, ch = area ?? ctx.canvas.height, m = st.mode, p = st.pos;
const B = pic(vBot), T = pic(vTop); const B = still(st.bottom) || pic(vBot), T = still(st.top) || pic(vTop);
const aw = vBot.videoWidth || st.data.width, ah = vBot.videoHeight || st.data.height; const aw = B.naturalWidth || vBot.videoWidth || st.data.width, ah = B.naturalHeight || vBot.videoHeight || st.data.height;
const name = s => s === "guide" ? (st.data?.labels?.[0] || "guide") : s === "target" ? (st.data?.labels?.[1] || "target") : "difference"; const name = s => s === "guide" ? (st.data?.labels?.[0] || "guide") : s === "target" ? (st.data?.labels?.[1] || "target") : "difference";
ctx.fillStyle = "#000"; ctx.fillRect(0, 0, cw, ch); ctx.fillStyle = "#000"; ctx.fillRect(0, 0, cw, ch);
if (m === "side") { // bottom left, top right, as on the stage if (m === "side") { // bottom left, top right, as on the stage
@@ -378,6 +384,29 @@ function buildPlayer(host) {
} }
} }
// the canvas for save / rec / PNG: the videos at their size (side by side twice as wide), the reference strip below
function sheet(aw, ah) {
const canvas = el("canvas");
canvas.width = (st.mode === "side" ? 2 * aw + Math.round(2 * aw * 0.004) : aw) & ~1;
canvas.videoHeight = ah & ~1;
canvas.height = canvas.videoHeight + stripHeight(canvas.videoHeight);
return canvas;
}
function snapshot() { // the frame on screen -> PNG (a single frame draws from the lossless stills)
if (!st.data) return;
if (st.playing) toggle();
const s = still(st.bottom);
const canvas = sheet(s?.naturalWidth || vBot.videoWidth || st.data.width, s?.naturalHeight || vBot.videoHeight || st.data.height);
compose(canvas.getContext("2d"));
canvas.toBlob(blob => {
if (!blob) { alert("BFS Video Compare: could not make the PNG."); return; }
const href = URL.createObjectURL(blob);
download(href, `${stem()}${single() ? "" : `_f${st.frame}`}.png`);
setTimeout(() => URL.revokeObjectURL(href), 60000);
}, "image/png");
}
function buttons() { function buttons() {
if (rec.busy) { // uploading / converting the finished recording if (rec.busy) { // uploading / converting the finished recording
btnRec.classList.remove("rec"); btnRec.classList.remove("rec");
@@ -391,18 +420,17 @@ function buildPlayer(host) {
btnRec.classList.toggle("rec", busy); btnRec.classList.toggle("rec", busy);
btnRec.textContent = busy ? (rec.full ? "■ stop" : `■ stop ${((performance.now() - rec.t0) / 1000).toFixed(0)}s`) : "● rec"; btnRec.textContent = busy ? (rec.full ? "■ stop" : `■ stop ${((performance.now() - rec.t0) / 1000).toFixed(0)}s`) : "● rec";
btnSave.disabled = busy; btnSave.disabled = busy;
btnSave.textContent = busy && rec.full ? `⬇ ${Math.round(100 * (vBot.currentTime / (vBot.duration || 1)))}%` : "⬇ save"; btnSave.textContent = busy && rec.full ? `⬇ ${Math.round(100 * (vBot.currentTime / (vBot.duration || 1)))}%` : single() ? "⬇ save PNG" : "⬇ save";
btnShot.disabled = busy;
// one frame is a picture: save downloads it as PNG, a video recording has nothing to show
for (const b of [btnRec, selFormat, btnShot]) b.style.display = single() ? "none" : "";
} }
function startRec(full = false) { function startRec(full = false) {
if (!st.data || rec.mr || rec.busy) return false; if (!st.data || rec.mr || rec.busy) return false;
if (!window.MediaRecorder || !HTMLCanvasElement.prototype.captureStream) { alert("This browser can't record a canvas (MediaRecorder)."); return false; } if (!window.MediaRecorder || !HTMLCanvasElement.prototype.captureStream) { alert("This browser can't record a canvas (MediaRecorder)."); return false; }
const aw = vBot.videoWidth || st.data.width, ah = vBot.videoHeight || st.data.height;
const side = st.mode === "side", canvas = el("canvas");
// the size of the layout it starts in (side by side is twice as wide); a later mode change is fitted into it // the size of the layout it starts in (side by side is twice as wide); a later mode change is fitted into it
canvas.width = (side ? 2 * aw + Math.round(2 * aw * 0.004) : aw) & ~1; const canvas = sheet(vBot.videoWidth || st.data.width, vBot.videoHeight || st.data.height);
canvas.videoHeight = ah & ~1; // the videos; the reference strip goes below them
canvas.height = canvas.videoHeight + stripHeight(canvas.videoHeight);
const ctx = canvas.getContext("2d"); const ctx = canvas.getContext("2d");
compose(ctx); compose(ctx);
const stream = canvas.captureStream(Math.max(30, Math.round(fps()))); const stream = canvas.captureStream(Math.max(30, Math.round(fps())));
@@ -437,6 +465,7 @@ function buildPlayer(host) {
async function saveFull() { // once from the first frame to the last, then stop and download async function saveFull() { // once from the first frame to the last, then stop and download
if (!st.data || rec.mr || rec.busy) return; if (!st.data || rec.mr || rec.busy) return;
if (single()) { snapshot(); return; }
if (st.playing) toggle(); if (st.playing) toggle();
for (const v of videos()) v.loop = false; for (const v of videos()) v.loop = false;
const seeked = v => new Promise(res => { const t = setTimeout(res, 2000); v.addEventListener("seeked", () => { clearTimeout(t); res(); }, { once: true }); }); const seeked = v => new Promise(res => { const t = setTimeout(res, 2000); v.addEventListener("seeked", () => { clearTimeout(t); res(); }, { once: true }); });
@@ -458,6 +487,9 @@ function buildPlayer(host) {
lblRefs.style.display = d.refs?.length ? "flex" : "none"; lblRefs.style.display = d.refs?.length ? "flex" : "none";
showRefs(); showRefs();
st.data = d; st.data = d;
for (const k of Object.keys(stills)) delete stills[k];
for (const [k, f] of Object.entries(d.stills || {})) { const im = new Image(); im.crossOrigin = "anonymous"; im.src = url(f); stills[k] = im; }
buttons();
notes.textContent = (d.notes || []).join("\n"); notes.textContent = (d.notes || []).join("\n");
empty.style.display = "none"; empty.style.display = "none";
scrub.max = String(Math.max(0, d.frames - 1)); scrub.max = String(Math.max(0, d.frames - 1));
@@ -478,6 +510,11 @@ app.registerExtension({
const r = created?.apply(this, arguments); const r = created?.apply(this, arguments);
styles(); styles();
const host = el("div", { style: "width:100%;height:100%;min-height:320px" }); const host = el("div", { style: "width:100%;height:100%;min-height:320px" });
// Keep clicks/drags inside the interactive player from reaching LiteGraph. Otherwise a
// pointer event on a video/control can be interpreted as a canvas gesture and the node
// may recalculate its DOM-widget height, making the compare panel collapse after use.
for (const type of ["pointerdown", "pointerup", "pointermove", "click", "dblclick", "contextmenu"])
host.addEventListener(type, e => e.stopPropagation());
this._bfsCompare = buildPlayer(host); this._bfsCompare = buildPlayer(host);
wheelToCanvas(host); // the wheel zooms the canvas over the player too wheelToCanvas(host); // the wheel zooms the canvas over the player too
this.addDOMWidget("bfs_video_compare", "div", host, { serialize: false, hideOnZoom: false, getMinHeight: () => 320 }); this.addDOMWidget("bfs_video_compare", "div", host, { serialize: false, hideOnZoom: false, getMinHeight: () => 320 });
+385
View File
@@ -0,0 +1,385 @@
/**
* BFS Video Segment Picker: a strip with every frame of the batch on top; click a frame to show it in the image
* Segment Picker's canvas (the same Panel, one selection per frame). The strip marks each frame: ✓ picked by hand,
* ↔ tracked from a picked frame, ⚠ lost or nothing. State: the hidden `selection` widget, {"frames": {"i": selection}}.
*/
import { app } from "../../scripts/app.js";
import { api } from "../../scripts/api.js";
import { wheelToCanvas } from "./bfs_wheel.js";
import { Panel, styles as pickerStyles } from "./bfs_segment_picker.js";
import { createApp, h, ref, reactive } from "./vendor/vue.esm-browser.prod.mjs";
const fileUrl = f => api.apiURL(`/view?filename=${encodeURIComponent(f.filename)}&subfolder=${encodeURIComponent(f.subfolder || "")}&type=${f.type || "temp"}`);
const SUBJECTS = [["1", "#eb2828", "red"], ["2", "#236eeb", "blue"], ["3", "#3ccd3c", "green"]];
function parse(text) { // {frame: {subject: selection}}; a bare selection (older state) is subject 1
let s = {};
try { s = JSON.parse(text || "{}") || {}; } catch { s = {}; }
const f = (s.frames && typeof s.frames === "object") ? s.frames : {};
const out = {};
for (const [k, v] of Object.entries(f)) {
if (v && (v.selected || v.added)) out[k] = { "1": v };
else if (v && typeof v === "object") {
out[k] = Object.fromEntries(Object.entries(v).filter(([sk]) => sk === "lock" || (+sk >= 1 && +sk <= SUBJECTS.length)));
} else out[k] = {};
}
return out;
}
let styled = false;
function styles() {
if (styled) return;
styled = true;
const css = document.createElement("style");
css.textContent = `
.bvsp{--card:#1b1d21;--line:#2e3138;--muted:#8b919c;--text:#e3e6eb;--accent:#3b82f6;--ok:#22c55e;--trk:#60a5fa;--bad:#f43f5e;
position:absolute;inset:0;display:flex;flex-direction:column;gap:6px;padding:6px;box-sizing:border-box;font:12px system-ui,sans-serif;color:var(--text);overflow:hidden}
.bvsp .card{background:var(--card);border:1px solid var(--line);border-radius:8px;padding:6px 8px;display:flex;flex-direction:column;gap:6px;flex:0 0 auto}
.bvsp .row{display:flex;align-items:center;gap:8px;min-width:0}
.bvsp .sp{flex:1}
.bvsp .muted{color:var(--muted);font-size:11px}
.bvsp .btn{background:#2a2d34;border:1px solid #3a3e47;color:var(--text);border-radius:6px;padding:4px 10px;cursor:pointer;font:inherit;white-space:nowrap}
.bvsp .btn:hover{background:#343843}
.bvsp .btn.pri{background:var(--accent);border-color:#4b8ff7;color:#fff;font-weight:600}
.bvsp .btn.pri:hover{background:#4b8ff7}
.bvsp .chk{display:flex;align-items:center;gap:4px;color:var(--muted);font-size:11px;cursor:pointer;user-select:none}
.bvsp .subj{display:flex;gap:4px}
.bvsp .subj .t{display:flex;align-items:center;gap:5px;border:1px solid var(--line);background:#23262c;border-radius:999px;padding:3px 10px 3px 6px;cursor:pointer;color:var(--muted);font-weight:600}
.bvsp .subj .t .clr{display:inline-flex;align-items:center;justify-content:center;width:14px;height:14px;border-radius:50%;margin-left:4px;font-size:10px;line-height:1;background:rgba(255,255,255,.15);color:#fff;cursor:pointer}
.bvsp .subj .t .clr:hover{background:#f43f5e;color:#fff}
.bvsp .subj .t i{width:10px;height:10px;border-radius:50%;display:block}
.bvsp .subj .t.on{color:#fff;border-color:currentColor;background:#2b2f37}
.bvsp .subj .t.lock.on{color:#fbbf24;background:#44351b}
.bvsp .stat{display:flex;align-items:center;gap:8px;font-size:11px;color:#cfd6e0}
.bvsp .stat .pb{flex:0 0 110px;height:5px;background:#2a2d34;border-radius:3px;overflow:hidden}
.bvsp .stat .pb i{display:block;height:100%}
.bvsp .stat .txt{flex:1;white-space:nowrap;overflow:hidden;text-overflow:ellipsis}
.bvsp .stat .tog{cursor:pointer;color:var(--trk);user-select:none}
.bvsp .log{max-height:90px;overflow:auto;background:#141518;border:1px solid var(--line);border-radius:6px;padding:4px 6px;font:11px ui-monospace,monospace;color:#aab}
.bvsp .counts{display:flex;gap:10px;font-size:11px;color:var(--muted)}
.bvsp .counts b{font-weight:700}
.bvsp .strip{display:flex;gap:5px;overflow-x:auto;padding-bottom:2px}
.bvsp .fr{position:relative;flex:0 0 auto;width:96px;height:58px;border-radius:6px;background:#111 center/cover no-repeat;cursor:pointer;border:2px solid transparent;outline:1px solid var(--line)}
.bvsp .fr.on{border-color:#f5b041}
.bvsp .fr .b{position:absolute;left:3px;bottom:3px;font-size:10px;background:rgba(0,0,0,.7);padding:0 4px;border-radius:3px}
.bvsp .fr .s{position:absolute;right:3px;top:3px;font-size:11px;font-weight:700;padding:0 4px;border-radius:3px;background:rgba(0,0,0,.7)}
.bvsp .fr .s.picked{color:var(--ok)}.bvsp .fr .s.tracked{color:var(--trk)}.bvsp .fr .s.lost{color:var(--bad)}
.bvsp .fr .dots{position:absolute;left:3px;top:3px;display:flex;gap:2px}
.bvsp .fr .dots i{width:8px;height:8px;border-radius:50%;border:1px solid #000;display:block}
.bvsp .tl{display:flex;gap:10px;align-items:center}
.bvsp .tl img{height:72px;border-radius:6px;border:1px solid var(--line);flex:0 0 auto}
.bvsp .tl .scrub{flex:1;display:flex;flex-direction:column;gap:6px;min-width:140px}
.bvsp .tl .track{position:relative;height:18px}
.bvsp .tl .track .bg{position:absolute;left:0;right:0;top:7px;height:4px;border-radius:2px}
.bvsp .tl .track input{position:absolute;inset:0;width:100%;margin:0;background:transparent;-webkit-appearance:none;appearance:none}
.bvsp .tl .track input::-webkit-slider-runnable-track{background:transparent;height:18px}
.bvsp .tl .track input::-moz-range-track{background:transparent}
.bvsp .tl .track input::-webkit-slider-thumb{-webkit-appearance:none;width:14px;height:14px;margin-top:2px;border-radius:50%;background:#fff;border:2px solid var(--accent);cursor:pointer}
.bvsp .tl .track input::-moz-range-thumb{width:12px;height:12px;border-radius:50%;background:#fff;border:2px solid var(--accent);cursor:pointer}
.bvsp .tl .fn{font-variant-numeric:tabular-nums;color:var(--text);font-weight:600}
.bvsp .chips{display:flex;flex-wrap:wrap;gap:4px;align-items:center}
.bvsp .chips .c{display:flex;align-items:center;gap:4px;border:1px solid #6b2a37;background:#2e1a1f;color:#fecdd3;border-radius:999px;padding:1px 8px;cursor:pointer;font-size:11px}
.bvsp .chips .c:hover{background:#45222b}
.bvsp .chips .c i{width:8px;height:8px;border-radius:50%;display:block}
.bvsp .chips .okk{color:var(--ok);font-size:11px}
.bvsp .host{position:absolute;inset:0}
.bvsp .empty{color:var(--muted);text-align:center;padding:12px}
`;
document.head.appendChild(css);
}
// A run the panel starts for picking ("Segment again", "+ frame") only segments the strip: the selection it sends is marked
// segment_only, so the node skips tracking and Molmo2. "Propagate" and a normal workflow run track as usual.
const propagating = new Set();
let wrapped = false;
function wrapQueue() {
if (wrapped) return;
wrapped = true;
const orig = api.queuePrompt.bind(api);
api.queuePrompt = async function (number, p, opts) {
const targets = opts?.partialExecutionTargets || [];
for (const id of targets) {
const out = p?.output?.[String(id)];
if (out?.class_type !== "BFSVideoSegmentPicker" || propagating.has(String(id))) continue;
try {
const sel = JSON.parse(out.inputs.selection || "{}");
sel.segment_only = true;
out.inputs.selection = JSON.stringify(sel);
} catch { /* leave it */ }
}
return orig(number, p, opts);
};
}
app.registerExtension({
name: "BFSNodes.VideoSegmentPicker",
async nodeCreated(node) {
if (node.comfyClass !== "BFSVideoSegmentPicker") return;
pickerStyles(); styles();
const widget = node.widgets?.find(w => w.name === "selection");
if (widget) {
widget.type = "hidden"; widget.hidden = true; widget.computeSize = () => [0, -4];
widget.options = { ...(widget.options || {}), hidden: true, serialize: true };
widget.serializeValue = () => widget.value;
}
const frames = reactive({ v: parse(widget?.value) });
let focusFrame = 0;
const scopeWidget = node.widgets?.find(w => w.name === "propagate_scope");
const validScopes = ["all frames", "only this frame", "from active frame onward", "from first selected frame onward", "fill only empty masks"];
const scope = ref(validScopes.includes(scopeWidget?.value) ? scopeWidget.value : "fill only empty masks");
if (scopeWidget) scopeWidget.value = scope.value;
const save = () => { if (widget) widget.value = JSON.stringify({ frames: frames.v, active_frame: focusFrame }); node.graph?.setDirtyCanvas(true); };
const hasPicks = k => Object.values(frames.v).some(f => f?.[k]?.selected?.length);
const clearSubject = k => {
let changed = false;
for (const f of Object.keys(frames.v)) {
if (frames.v[f]?.[k]) {
delete frames.v[f][k];
changed = true;
}
}
if (changed) {
scope.value = "all frames";
if (scopeWidget) scopeWidget.value = "all frames";
save();
mountPanel();
if (propagatedOnce()) propagate();
}
};
const run = reactive({ data: null }); // {frames: [panel data per frame], report: [...]}
const active = ref(0);
const subject = ref("1"); // which subject the clicks select (its color = Picture number)
const root = document.createElement("div");
root.style.cssText = "position:relative;width:100%;height:100%;min-width:440px;min-height:420px";
node.addDOMWidget("bfs_video_segment_picker", "div", root, { serialize: false, hideOnZoom: false, hideInPanel: true });
wheelToCanvas(root);
// the image picker's Panel for the active frame, remounted when the frame changes
let panelApp = null, handle = {};
const panelHost = document.createElement("div");
panelHost.className = "host";
const mountPanel = () => {
panelApp?.unmount(); panelHost.innerHTML = ""; handle = {};
const i = run.data?.indices?.[active.value] ?? active.value; // the real frame index of the strip entry
focusFrame = i;
const sj = subject.value;
const el = document.createElement("div"); el.style.cssText = "position:absolute;inset:0"; panelHost.appendChild(el);
panelApp = createApp({ setup: () => Panel({
node,
get: () => JSON.stringify(frames.v[String(i)]?.[sj] || {}),
set: v => {
focusFrame = i;
const before = JSON.stringify(frames.v[String(i)]?.[sj]?.selected || []);
try { frames.v[String(i)] = { ...(frames.v[String(i)] || {}), [sj]: JSON.parse(v) }; } catch { /* keep */ }
save();
if (JSON.stringify(frames.v[String(i)]?.[sj]?.selected || []) !== before) autoPropagate();
},
expose: o => Object.assign(handle, o),
}, { showEraser: false, showRemove: false, autoPointOnClick: sj === "lock", entryPrefix: sj === "lock" ? "lock_" : "",
maskColor: sj === "lock" ? [251, 191, 36] : null }) });
panelApp.mount(el);
const d = run.data?.frames?.[active.value];
if (d) setTimeout(() => handle.executed?.(d), 0);
};
const pickedSubjects = i => Object.entries(frames.v[String(run.data?.indices?.[i] ?? i)] || {}).filter(([, v]) => (v?.selected || []).length).map(([k]) => k);
const status = i => {
const own = (frames.v[String(run.data?.indices?.[i] ?? i)]?.[subject.value]?.selected || []).length > 0;
const r = subject.value === "lock"
? (run.data?.protected_frames?.[i] ? "tracked protected" : "")
: (run.data?.report?.[i]?.subjects?.[subject.value] || run.data?.report?.[i]?.source || "");
if (own) return ["picked", "✓"];
if (r.startsWith("tracked")) return ["tracked", "↔"];
if (r.startsWith("lost")) return ["lost", "⚠"];
return ["", ""];
};
const busy = ref("");
const live = reactive({ text: "", progress: null, elapsed: 0, log: [], open: false, running: false });
api.addEventListener("bfs_vsegpick_status", e => {
const d = e.detail || {};
if (String(d.node) !== String(node.id)) return;
live.text = d.text; live.elapsed = d.elapsed || 0; live.running = (d.progress ?? 0) < 1;
if (d.progress !== null && d.progress !== undefined) live.progress = d.progress;
if (d.log) { live.log.push(`[${(d.elapsed || 0).toFixed(1)}s] ${d.text}`); if (live.log.length > 60) live.log.shift(); }
});
wrapQueue();
const segmentOnly = async () => { // a picking run: the new frame is segmented, nothing is tracked
busy.value = "segmenting…";
try { const p = await app.graphToPrompt(); await api.queuePrompt(0, p, { partialExecutionTargets: [String(node.id)] }); }
catch (e) { busy.value = String(e.message || e); }
};
const propagate = async () => {
busy.value = "propagating…";
try {
const p = await app.graphToPrompt();
propagating.add(String(node.id));
try { await api.queuePrompt(0, p, { partialExecutionTargets: [String(node.id)] }); } finally { propagating.delete(String(node.id)); }
} catch (e) { busy.value = String(e.message || e); }
};
// after the first Propagate, a pick that changes a frame's selection propagates by itself (1.5 s after the last
// click): only the shots whose picks changed are tracked again, the rest comes from the cache
const auto = ref(localStorage.getItem("bfs_vsegpick_auto") === "1");
let autoTimer = null;
const propagatedOnce = () => (run.data?.report || []).some(r => /tracked|lost|Molmo/.test(JSON.stringify(r)));
const autoPropagate = () => {
if (!auto.value || !propagatedOnce()) return;
clearTimeout(autoTimer);
busy.value = "auto-propagate in 1.5 s…";
autoTimer = setTimeout(propagate, 1500);
};
const addFrame = f => { // put a frame in the strip (an empty pick) and run, so it can be picked on
const n = run.data?.count || 0;
f = Math.max(0, Math.min(n - 1, Math.round(+f)));
if (!Number.isFinite(f)) return;
focusFrame = f;
if (!frames.v[String(f)]) { frames.v[String(f)] = {}; save(); }
else save();
segmentOnly();
};
const SUBJ = { "1": "red", "2": "blue", "3": "green" };
const tlPos = ref(0); // index into run.data.timeline (whole-video scrubber)
const counts = () => {
const r = run.data?.report || []; const n = run.data?.frames?.length || 0;
let picked = 0, tracked = 0, lost = 0;
for (let i = 0; i < n; i++) { const [c] = status(i); if (c === "picked") picked++; else if (c === "tracked") tracked++; else if (c === "lost") lost++; }
return { picked, tracked, lost, n };
};
const tlAt = () => run.data.timeline[Math.min(tlPos.value, run.data.timeline.length - 1)];
const lostTrack = () => { // the scrubber's track: lost stretches in red over the video's length
const n = Math.max(1, (run.data?.count || 1) - 1);
const runs = [...(run.data?.lost || [])];
if (!runs.length) return "#3a3e47";
const stops = ["#3a3e47 0%"];
for (const r of runs.sort((a, b) => a.start - b.start)) {
const a = (100 * r.start / n).toFixed(2), b = (100 * Math.max(r.end, r.start + 0.5) / n).toFixed(2);
stops.push(`#3a3e47 ${a}%`, `#f43f5e ${a}%`, `#f43f5e ${b}%`, `#3a3e47 ${b}%`);
}
return `linear-gradient(90deg, ${stops.join(", ")}, #3a3e47 100%)`;
};
createApp({
setup: () => () => h("div", { class: "bvsp" }, [
h("div", { class: "card" }, [
h("div", { class: "row" }, [
h("div", { class: "subj" }, [...SUBJECTS.map(([k, col, name]) => h("span", {
class: ["t", subject.value === k ? "on" : ""], style: subject.value === k ? { color: col } : null,
title: `Subject ${k} (${name}) = Picture ${k}: clicks select this subject`,
onClick: () => { if (subject.value !== k) { subject.value = k; mountPanel(); } } },
[
h("i", { style: { background: col } }),
h("span", { style: "color:#fff" }, `${k} · ${name}`),
hasPicks(k) ? h("span", {
class: "clr",
title: `Clear all picks for Subject ${k} (${name})`,
onClick: (e) => { e.stopPropagation(); clearSubject(k); }
}, "✕") : null,
])), h("span", {
class: ["t", "lock", subject.value === "lock" ? "on" : ""],
title: "Everything outside the selected subject is already protected. Use this padlock only for an object inside that subject's mask; click it to segment with SAM 3.",
onClick: () => { if (subject.value !== "lock") { subject.value = "lock"; mountPanel(); } },
}, [
h("span", null, "🔒"),
hasPicks("lock") ? h("span", {
class: "clr", title: "Clear all protected-object picks",
onClick: e => { e.stopPropagation(); clearSubject("lock"); },
}, "✕") : null,
])]),
h("span", { class: "sp" }),
h("select", { value: scope.value, title: "Choose how a new frame selection changes the propagated masks",
style: "max-width:190px;background:#23262c;color:#e3e6eb;border:1px solid #3a3e47;border-radius:6px;padding:4px;font:inherit",
onChange: e => { scope.value = e.target.value; if (scopeWidget) scopeWidget.value = e.target.value; node.graph?.setDirtyCanvas(true); } },
["from active frame onward", "only this frame", "fill only empty masks", "all frames", "from first selected frame onward"].map(v => h("option", { value: v },
v === "from active frame onward" ? "From this frame onward" : v === "only this frame" ? "Only this frame"
: v === "fill only empty masks" ? "Fill only empty masks" : v === "all frames" ? "Recompute all frames" : "From earliest selected frame"))),
h("label", { class: "chk", title: "After a selection changes, apply the selected fix mode automatically after 1.5 seconds" }, [
h("input", { type: "checkbox", checked: auto.value, onChange: e => { auto.value = e.target.checked;
try { localStorage.setItem("bfs_vsegpick_auto", e.target.checked ? "1" : "0"); } catch { /* ignore */ } } }), "auto"]),
h("button", { class: "btn pri", onClick: propagate, title: "Apply the selected fix mode to the current selection" }, "↔ Apply fix"),
]),
live.text || busy.value ? h("div", { class: "stat" }, [
h("span", { class: "pb" }, [h("i", { style: { width: `${Math.round(100 * (live.progress ?? 0))}%`,
background: live.running ? "#3b82f6" : "#22c55e" } })]),
h("span", { class: "txt", title: live.text }, busy.value && !live.running ? busy.value
: `${live.running ? "⏳" : "✓"} ${live.text} · ${live.elapsed}s`),
h("span", { class: "tog", onClick: () => { live.open = !live.open; } }, live.open ? "▾ log" : "▸ log"),
]) : h("div", { class: "muted" }, subject.value === "lock"
? "🔒 Outside the subject is already protected. Click an object inside the subject mask to keep it unchanged, then Apply fix."
: "Pick a subject, click a frame below, then click its person. Subject N = Picture N."),
live.open ? h("div", { class: "log" }, live.log.length ? live.log.map((l, i) => h("div", { key: i }, l)) : "no log yet") : null,
]),
h("div", { class: "card" }, [
h("div", { class: "row" }, [
h("span", { style: "font-weight:600" }, "Frames"),
run.data ? h("span", { class: "counts" }, (() => { const c = counts(); return [
h("span", null, [h("b", { style: "color:#22c55e" }, "✓ "), `${c.picked} picked`]),
h("span", null, [h("b", { style: "color:#60a5fa" }, "↔ "), `${c.tracked} tracked`]),
h("span", null, [h("b", { style: "color:#f43f5e" }, "⚠ "), `${c.lost} lost`])]; })()) : null,
]),
run.data?.frames?.length
? h("div", { class: "strip" }, run.data.frames.map((d, i) => {
const [cls, mark] = status(i);
return h("div", { class: ["fr", i === active.value ? "on" : ""], key: i, title: run.data.report?.[i]?.source || "",
style: { backgroundImage: `url("${fileUrl(d.image)}")` },
onClick: () => { if (active.value !== i) { active.value = i; focusFrame = run.data.indices?.[i] ?? i; save(); mountPanel(); } } },
[h("span", { class: "b" }, `f${run.data.indices?.[i] ?? i}`), mark ? h("span", { class: ["s", cls] }, mark) : null,
h("span", { class: "dots" }, pickedSubjects(i).map(k => k === "lock"
? h("span", { title: "Protected object picked", style: "color:#fbbf24;background:#181818;border-radius:3px;padding:0 2px" }, "🔒")
: h("i", { style: { background: SUBJECTS[(+k - 1) % SUBJECTS.length][1] } })))]);
}))
: h("div", { class: "empty" }, "Run the node once: the frames appear here. Click a frame, then the person."),
]),
run.data?.timeline?.length ? h("div", { class: "card" }, [
h("div", { class: "tl" }, [
h("img", { src: fileUrl(tlAt()), title: "every frame with its masks: scrub to find a wrong or missing mark" }),
h("div", { class: "scrub" }, [
h("div", { class: "row" }, [
h("span", { style: "font-weight:600" }, "Whole video"),
h("span", { class: "fn" }, `f${tlAt().frame}`),
h("span", { class: "muted" }, `/ ${(run.data.count || 1) - 1}`),
h("span", { class: "sp" }),
h("button", { class: "btn", onClick: () => addFrame(tlAt().frame),
title: "put this frame in the strip to pick or fix the person on it" }, "+ add this frame"),
]),
h("div", { class: "track" }, [
h("div", { class: "bg", style: { background: lostTrack() } }),
h("input", { type: "range", min: 0, max: run.data.timeline.length - 1, value: tlPos.value,
onInput: e => { tlPos.value = +e.target.value; } }),
]),
h("div", { class: "chips" }, (run.data.lost || []).length
? (run.data.lost || []).slice(0, 12).map(r => h("span", { class: "c", title: "add the middle frame of this stretch to the strip",
onClick: () => addFrame((r.start + r.end) / 2) },
[h("i", { style: { background: SUBJECTS[(+r.subject - 1) % SUBJECTS.length]?.[1] || "#999" } }), `f${r.start}–${r.end}`]))
: [h("span", { class: "okk" }, "✓ no lost stretches")]),
]),
]),
]) : null,
h("div", { ref: el => { if (el && !el.contains(panelHost)) { el.appendChild(panelHost); if (!panelApp) mountPanel(); } }, style: "flex:1;position:relative;min-height:260px;border:1px solid #2e3138;border-radius:8px;overflow:hidden" }),
]),
}).mount(root);
const onExecuted = node.onExecuted;
node.onExecuted = function (output) {
onExecuted?.apply(this, arguments);
const d = output?.bfs_vsegpick?.[0];
if (!d) return;
run.data = d; busy.value = "";
const focusIndex = d.indices?.indexOf?.(focusFrame) ?? -1;
if (focusIndex >= 0) active.value = focusIndex;
if (active.value >= d.frames.length) active.value = 0;
mountPanel();
};
const onConfigure = node.onConfigure;
node.onConfigure = function () {
const r = onConfigure?.apply(this, arguments);
setTimeout(() => {
let saved = {};
try { saved = JSON.parse(widget?.value || "{}"); } catch { /* ignore */ }
frames.v = parse(widget?.value); focusFrame = Number.isInteger(+saved.active_frame) ? +saved.active_frame : 0;
if (scopeWidget?.value) scope.value = scopeWidget.value;
mountPanel();
}, 0);
return r;
};
node.size = [Math.max(node.size[0], 780), Math.max(node.size[1], 1180)];
},
});
+167
View File
@@ -0,0 +1,167 @@
/**
* BFS Workflow Loop: the panel. Lists the nodes upstream of `result`. "Inject" picks a node whose output is replaced
* by the current item (or its mask) on every iteration; "Change per iteration" picks widgets with one value per
* iteration (one per line, or "a, b, c" / "start:stop:step" for numbers). State lives in the hidden `loop` widget.
*/
import { app } from "../../scripts/app.js";
import { wheelToCanvas } from "./bfs_wheel.js";
import { createApp, h, reactive, computed, watch } from "./vendor/vue.esm-browser.prod.mjs";
const SKIP_TYPES = new Set(["hidden", "converted-widget", "button"]);
const SKIP_NAMES = new Set(["control_after_generate", "upload", "image_upload", "videopreview", "loop"]);
function upstreamNodes(node) {
const graph = node.graph || app.graph;
const getLink = id => graph.links[id] ?? graph.links.get?.(id);
const inp = node.inputs?.find(i => i.name === "result");
const link = inp?.link != null ? getLink(inp.link) : null;
if (!link) return [];
const seen = new Set(), order = [], stack = [link.origin_id];
while (stack.length) {
const id = stack.pop();
if (seen.has(id)) continue;
seen.add(id);
const n = graph.getNodeById(id);
if (!n) continue;
order.push(n);
for (const i of n.inputs || []) if (i.link != null) { const l = getLink(i.link); if (l) stack.push(l.origin_id); }
}
return order.sort((a, b) => a.id - b.id);
}
const widgetsOf = n => (n?.widgets || []).filter(w => w.name && !SKIP_TYPES.has(w.type) && !SKIP_NAMES.has(w.name) && !w.hidden);
const kindOf = w => (w?.type === "combo" || Array.isArray(w?.options?.values)) ? "combo"
: (w?.type === "toggle" || typeof w?.value === "boolean") ? "bool" : (w?.type === "number" || typeof w?.value === "number") ? "number" : "text";
function parseValues(text, kind) {
if (kind === "number") {
const out = [];
for (const part of String(text).split(/[,\n]/).map(s => s.trim()).filter(Boolean)) {
const r = part.split(":").map(Number);
if (r.length === 3 && r.every(Number.isFinite) && r[2] !== 0) {
for (let v = r[0], i = 0; (r[2] > 0 ? v <= r[1] + 1e-9 : v >= r[1] - 1e-9) && i < 256; v += r[2], i++) out.push(+v.toFixed(6));
} else if (Number.isFinite(+part)) out.push(+part);
}
return out;
}
if (kind === "bool") return String(text).split(/[,\n]/).map(s => s.trim().toLowerCase()).filter(Boolean).map(s => ["1", "true", "yes", "on"].includes(s));
return String(text).split("\n").map(s => s.trim()).filter(Boolean);
}
function parseState(text) {
let s = {};
try { s = JSON.parse(text || "{}") || {}; } catch { s = {}; }
return {
inject: (s.inject || []).map(i => ({ node: String(i.node ?? ""), slot: +i.slot || 0, source: i.source || "items" })),
params: (s.params || []).map(p => ({ node: String(p.node ?? ""), widget: p.widget || "", text: p.text ?? (p.values || []).join("\n") })),
};
}
let styled = false;
function styles() {
if (styled) return;
styled = true;
const css = document.createElement("style");
css.textContent = `
.bwl{position:absolute;inset:0;display:flex;flex-direction:column;gap:8px;padding:6px;box-sizing:border-box;font:12px system-ui,sans-serif;color:#ddd;overflow:auto}
.bwl .sec{display:flex;align-items:center;gap:6px;color:#9ab;font-size:11px;text-transform:uppercase;letter-spacing:.05em}
.bwl .row{background:#1d1d1d;border:1px solid #363636;border-radius:6px;padding:5px;display:flex;flex-direction:column;gap:4px}
.bwl .line{display:flex;gap:4px;align-items:center}
.bwl select,.bwl textarea{background:#262626;color:#ddd;border:1px solid #4a4a4a;border-radius:4px;padding:3px 5px;font:inherit;min-width:0;flex:1}
.bwl textarea{resize:vertical;min-height:38px}
.bwl button{background:#333;color:#ddd;border:1px solid #555;border-radius:4px;padding:2px 8px;cursor:pointer;font:inherit}
.bwl button:hover{background:#444}
.bwl .x{color:#999;cursor:pointer;padding:0 4px}.bwl .x:hover{color:#f66}
.bwl .hint{color:#888;font-size:11px}.bwl .warn{color:#f7a}
.bwl .foot{color:#bbb}.bwl .foot b{color:#fff}
`;
document.head.appendChild(css);
}
function Panel(io) {
const st = reactive(parseState(io.get()));
const tick = reactive({ n: 0 });
const nodes = computed(() => (tick.n, upstreamNodes(io.node)));
const byId = id => nodes.value.find(n => String(n.id) === String(id));
const save = () => io.set(JSON.stringify({
inject: st.inject.filter(i => i.node).map(i => ({ node: i.node, slot: i.slot, source: i.source })),
params: st.params.filter(p => p.node && p.widget).map(p => {
const w = widgetsOf(byId(p.node)).find(x => x.name === p.widget);
return { node: p.node, widget: p.widget, values: parseValues(p.text, kindOf(w)), text: p.text };
}),
}));
watch(st, save, { deep: true });
io.expose({ refresh: () => { tick.n++; }, reload: () => { Object.assign(st, parseState(io.get())); tick.n++; } });
const nodeSelect = (obj, onPick) => h("select", { value: obj.node, onChange: e => { obj.node = e.target.value; onPick?.(); } },
[h("option", { value: "" }, "— pick a node —"), ...nodes.value.map(n => h("option", { value: String(n.id) }, `[${n.id}] ${n.title || n.type}`))]);
const injectRow = (i, idx) => {
const n = byId(i.node);
const outs = (n?.outputs || []).map((o, k) => ({ k, label: `${o.name || o.type} (${o.type})` }));
return h("div", { class: "row", key: "i" + idx }, [
h("div", { class: "line" }, [nodeSelect(i, () => { i.slot = 0; }), h("span", { class: "x", onClick: () => st.inject.splice(idx, 1) }, "✕")]),
n ? h("div", { class: "line" }, [
h("select", { value: String(i.slot), onChange: e => { i.slot = +e.target.value; } }, outs.map(o => h("option", { value: String(o.k) }, `output: ${o.label}`))),
h("select", { value: i.source, onChange: e => { i.source = e.target.value; } },
[h("option", { value: "items" }, "← item i (items)"), h("option", { value: "items_b" }, "← item i (items_b)"),
h("option", { value: "item_masks" }, "← mask i (item_masks)")]),
]) : null,
]);
};
const paramRow = (p, idx) => {
const n = byId(p.node);
const w = widgetsOf(n).find(x => x.name === p.widget);
const kind = kindOf(w);
const vals = parseValues(p.text, kind);
return h("div", { class: "row", key: "p" + idx }, [
h("div", { class: "line" }, [nodeSelect(p, () => { p.widget = ""; p.text = ""; }), h("span", { class: "x", onClick: () => st.params.splice(idx, 1) }, "✕")]),
n ? h("select", { value: p.widget, onChange: e => { p.widget = e.target.value; const ww = widgetsOf(n).find(x => x.name === p.widget); p.text = ww ? String(ww.value ?? "") : ""; } },
[h("option", { value: "" }, "— pick a widget —"), ...widgetsOf(n).map(x => h("option", { value: x.name }, `${x.name} (${String(x.value).split(/[\\/]/).pop().slice(0, 36)})`))]) : null,
w ? h("textarea", { value: p.text, placeholder: kind === "number" ? "1, 2, 3 or 1:10:1" : "one value per line (iteration 1, 2, 3...)",
onInput: e => { p.text = e.target.value; } }) : null,
w ? h("div", { class: "hint" }, kind === "combo" ? `${vals.length} values; options: ${(w.options?.values || []).slice(0, 6).map(v => String(v).split(/[\\/]/).pop()).join(", ")}…`
: `${vals.length} values${vals.length ? ": " + vals.slice(0, 8).join(", ") + (vals.length > 8 ? "…" : "") : ""}`) : null,
]);
};
const itemsLinked = () => !!io.node.inputs?.find(x => x.name === "items")?.link;
return () => h("div", { class: "bwl" }, [
nodes.value.length ? null : h("div", { class: "warn" }, "Connect what one iteration produces to `result`; its upstream nodes appear here."),
h("div", { class: "sec" }, ["Inject the current item into", h("span", { style: "flex:1" }),
h("button", { onClick: () => st.inject.push({ node: "", slot: 0, source: "items" }) }, "+")]),
...st.inject.map(injectRow),
st.inject.length ? null : h("div", { class: "hint" }, "e.g. the Load Image that stands for “the frame”: on iteration i it gets item i."),
h("div", { class: "sec" }, ["Change per iteration", h("span", { style: "flex:1" }),
h("button", { onClick: () => st.params.push({ node: "", widget: "", text: "" }) }, "+")]),
...st.params.map(paramRow),
st.params.length ? null : h("div", { class: "hint" }, "e.g. a seed or a prompt line per iteration (cycled when shorter than the items)."),
h("div", { class: "foot" }, [itemsLinked() ? "Iterations: one per item." : "No items connected: iterations = the longest value list (or count).",
h("span", { style: "flex:1" }), h("button", { onClick: () => { tick.n++; }, title: "re-read the graph" }, "↻")]),
]);
}
app.registerExtension({
name: "BFSNodes.WorkflowLoop",
async nodeCreated(node) {
if (node.comfyClass !== "BFSWorkflowLoop") return;
styles();
const widget = node.widgets?.find(w => w.name === "loop");
if (widget) {
widget.type = "hidden"; widget.hidden = true; widget.computeSize = () => [0, -4];
widget.options = { ...(widget.options || {}), hidden: true, serialize: true };
widget.serializeValue = () => widget.value;
}
const host = document.createElement("div");
host.style.cssText = "position:relative;width:100%;height:100%;min-width:340px;min-height:360px";
node.addDOMWidget("bfs_workflow_loop", "div", host, { serialize: false, hideOnZoom: false, hideInPanel: true });
wheelToCanvas(host);
const handle = {};
createApp({ setup: () => Panel({ node, get: () => widget?.value ?? "{}",
set: v => { if (widget) widget.value = v; node.graph?.setDirtyCanvas(true); }, expose: o => Object.assign(handle, o) }) }).mount(host);
const onConn = node.onConnectionsChange;
node.onConnectionsChange = function () { const r = onConn?.apply(this, arguments); handle.refresh?.(); return r; };
const onConfigure = node.onConfigure;
node.onConfigure = function () { const r = onConfigure?.apply(this, arguments); setTimeout(() => handle.reload?.(), 0); return r; };
node.size = [Math.max(node.size[0], 380), Math.max(node.size[1], 440)];
},
});
+213
View File
@@ -0,0 +1,213 @@
/**
* BFS XY Sweep: the panel. Lists every node upstream of the `result` input and its widgets (like TinyTerra's
* advanced xyPlot); for X and Y pick a node + widget and its values. Combo widgets (LoRA names, samplers...) get a
* filterable checklist; numbers take "0.5, 0.7, 0.9" or a range "start:stop:step"; text takes one value per line.
* State lives in the hidden `sweep` STRING widget (JSON), saved with the workflow.
*/
import { app } from "../../scripts/app.js";
import { wheelToCanvas } from "./bfs_wheel.js";
import { createApp, h, reactive, computed, watch } from "./vendor/vue.esm-browser.prod.mjs";
const SKIP_TYPES = new Set(["hidden", "converted-widget", "button"]);
const SKIP_NAMES = new Set(["control_after_generate", "upload", "image_upload", "videopreview", "sweep"]);
function upstreamNodes(node) {
const graph = node.graph || app.graph;
const inp = node.inputs?.find(i => i.name === "result");
const link = inp?.link != null ? graph.links[inp.link] ?? graph.links.get?.(inp.link) : null;
if (!link) return [];
const seen = new Set(), order = [], stack = [link.origin_id];
while (stack.length) {
const id = stack.pop();
if (seen.has(id)) continue;
seen.add(id);
const n = graph.getNodeById(id);
if (!n) continue;
order.push(n);
for (const i of n.inputs || []) {
if (i.link == null) continue;
const l = graph.links[i.link] ?? graph.links.get?.(i.link);
if (l) stack.push(l.origin_id);
}
}
return order.sort((a, b) => a.id - b.id);
}
function widgetsOf(n) {
return (n.widgets || []).filter(w => w.name && !SKIP_TYPES.has(w.type) && !SKIP_NAMES.has(w.name) && !w.hidden
&& !(w.options && w.options.hidden));
}
function kindOf(w) {
if (!w) return "text";
if (w.type === "combo" || Array.isArray(w.options?.values)) return "combo";
if (w.type === "toggle" || typeof w.value === "boolean") return "bool";
if (w.type === "number" || typeof w.value === "number") return "number";
return "text";
}
function parseNumbers(text) {
const out = [];
for (const part of String(text).split(/[,\n]/).map(s => s.trim()).filter(Boolean)) {
const r = part.split(":").map(Number);
if (r.length === 3 && r.every(Number.isFinite) && r[2] !== 0) {
for (let v = r[0], i = 0; (r[2] > 0 ? v <= r[1] + 1e-9 : v >= r[1] - 1e-9) && i < 256; v += r[2], i++) out.push(+v.toFixed(6));
} else if (Number.isFinite(+part)) out.push(+part);
}
return out;
}
function parseState(text) {
let s = {};
try { s = JSON.parse(text || "{}") || {}; } catch { s = {}; }
const axis = a => ({ node: a?.node != null ? String(a.node) : "", widget: a?.widget || "", values: Array.isArray(a?.values) ? a.values : [],
text: a?.text ?? (Array.isArray(a?.values) ? a.values.join(", ") : ""), title: a?.title || "", filter: "" });
return { x: axis(s.x), y: axis(s.y) };
}
let styled = false;
function styles() {
if (styled) return;
styled = true;
const css = document.createElement("style");
css.textContent = `
.bxy{position:absolute;inset:0;display:flex;flex-direction:column;gap:8px;padding:6px;box-sizing:border-box;font:12px system-ui,sans-serif;color:#ddd;overflow:auto}
.bxy .ax{background:#1d1d1d;border:1px solid #363636;border-radius:6px;padding:6px;display:flex;flex-direction:column;gap:5px}
.bxy .hd{display:flex;align-items:center;gap:6px}
.bxy .tag{font-weight:700;background:#2d5a8a;color:#fff;border-radius:4px;padding:1px 7px}
.bxy .tag.y{background:#7a3d8a}
.bxy select,.bxy input,.bxy textarea{background:#262626;color:#ddd;border:1px solid #4a4a4a;border-radius:4px;padding:3px 5px;font:inherit;min-width:0}
.bxy select{flex:1}
.bxy textarea{resize:vertical;min-height:40px}
.bxy .opts{max-height:120px;overflow:auto;display:flex;flex-direction:column;gap:1px;border:1px solid #333;border-radius:4px;padding:3px}
.bxy .opts label{display:flex;gap:5px;align-items:center;white-space:nowrap;overflow:hidden;text-overflow:ellipsis;cursor:pointer}
.bxy .opts label:hover{background:#2a2a2a}
.bxy .chips{display:flex;flex-wrap:wrap;gap:3px}
.bxy .chip{background:#2a2a2a;border:1px solid #444;border-radius:9px;padding:0 6px;font-size:11px}
.bxy .hint{color:#888;font-size:11px}
.bxy .foot{display:flex;align-items:center;gap:6px;color:#bbb}
.bxy .foot .n{font-weight:700;color:#fff}
.bxy .warn{color:#f7a}
.bxy button{background:#333;color:#ddd;border:1px solid #555;border-radius:4px;padding:2px 8px;cursor:pointer;font:inherit}
.bxy button:hover{background:#444}
`;
document.head.appendChild(css);
}
function Panel(io) {
const st = reactive(parseState(io.get()));
const tick = reactive({ n: 0 }); // bumps on graph changes so the node lists refresh
const nodes = computed(() => (tick.n, upstreamNodes(io.node)));
const nodeById = id => nodes.value.find(n => String(n.id) === String(id));
const widgetOf = a => widgetsOf(nodeById(a.node) || {}).find(w => w.name === a.widget);
const save = () => {
const out = {};
for (const k of ["x", "y"]) {
const a = st[k];
if (!a.node || !a.widget) continue;
const kind = kindOf(widgetOf(a));
let values = a.values;
if (kind === "number") values = parseNumbers(a.text);
else if (kind === "text") values = String(a.text).split("\n").map(s => s.trim()).filter(Boolean);
else if (kind === "bool") values = [false, true].filter(v => a.values.includes(v));
const n = nodeById(a.node);
out[k] = { node: a.node, widget: a.widget, values, text: a.text, title: n ? (n.title || n.type) : a.title };
}
io.set(JSON.stringify(out));
};
watch(st, save, { deep: true });
const count = computed(() => {
const len = a => {
if (!a.node || !a.widget) return 0;
const kind = kindOf(widgetOf(a));
if (kind === "number") return parseNumbers(a.text).length;
if (kind === "text") return String(a.text).split("\n").filter(s => s.trim()).length;
return a.values.length;
};
const x = len(st.x), y = len(st.y);
return x * (y || 1);
});
io.expose({ refresh: () => { tick.n++; }, reload: () => { Object.assign(st, parseState(io.get())); tick.n++; } });
const axisView = (k, label) => {
const a = st[k];
const n = nodeById(a.node);
const ws = n ? widgetsOf(n) : [];
const w = widgetOf(a);
const kind = kindOf(w);
const pickNode = e => { a.node = e.target.value; a.widget = ""; a.values = []; a.text = ""; };
const pickWidget = e => {
a.widget = e.target.value; a.values = []; const ww = widgetOf(a);
a.text = ww && kindOf(ww) !== "combo" ? String(ww.value ?? "") : "";
};
let editor = null;
if (w && kind === "combo") {
const all = (w.options?.values || []).map(String);
const shown = a.filter ? all.filter(v => v.toLowerCase().includes(a.filter.toLowerCase())) : all;
editor = [
h("input", { placeholder: "filter options…", value: a.filter, onInput: e => { a.filter = e.target.value; } }),
h("div", { class: "opts" }, shown.slice(0, 400).map(v => h("label", { title: v }, [
h("input", { type: "checkbox", checked: a.values.includes(v),
onChange: e => { a.values = e.target.checked ? [...a.values, v] : a.values.filter(x => x !== v); } }), v]))),
a.values.length ? h("div", { class: "chips" }, a.values.map(v => h("span", { class: "chip", title: v }, v.split(/[\\/]/).pop()))) : null,
];
} else if (w && kind === "bool") {
editor = h("div", { class: "chips" }, [false, true].map(v => h("label", null, [
h("input", { type: "checkbox", checked: a.values.includes(v),
onChange: e => { a.values = e.target.checked ? [...a.values, v] : a.values.filter(x => x !== v); } }), String(v)])));
} else if (w && kind === "number") {
editor = [h("input", { value: a.text, placeholder: "0.5, 0.7, 0.9 or 0.5:0.9:0.2", onInput: e => { a.text = e.target.value; } }),
h("div", { class: "hint" }, `→ ${parseNumbers(a.text).join(", ") || "no values"} (now ${w.value})`)];
} else if (w) {
editor = [h("textarea", { value: a.text, placeholder: "one value per line", onInput: e => { a.text = e.target.value; } })];
}
return h("div", { class: "ax" }, [
h("div", { class: "hd" }, [h("span", { class: ["tag", k] }, label),
h("select", { value: a.node, onChange: pickNode }, [h("option", { value: "" }, k === "y" ? "— none —" : "— pick a node —"),
...nodes.value.map(nd => h("option", { value: String(nd.id) }, `[${nd.id}] ${nd.title || nd.type}`))]),
]),
n ? h("select", { value: a.widget, onChange: pickWidget }, [h("option", { value: "" }, "— pick a widget —"),
...ws.map(ww => h("option", { value: ww.name }, `${ww.name} (${String(ww.value).split(/[\\/]/).pop().slice(0, 40)})`))]) : null,
...(Array.isArray(editor) ? editor : [editor]),
]);
};
return () => h("div", { class: "bxy" }, [
nodes.value.length ? null : h("div", { class: "warn" }, "Connect the image to compare to `result`; its upstream nodes appear here."),
axisView("x", "X"),
axisView("y", "Y"),
h("div", { class: "foot" }, [h("span", null, ["cells: ", h("span", { class: "n" }, String(count.value))]),
count.value > 64 ? h("span", { class: "warn" }, "max 64") : null, h("span", { style: "flex:1" }),
h("button", { onClick: () => { tick.n++; }, title: "re-read the graph" }, "↻")]),
h("div", { class: "hint" }, "Only the nodes that depend on the swept widgets run once per cell; models and the rest run once."),
]);
}
app.registerExtension({
name: "BFSNodes.XYSweep",
async nodeCreated(node) {
if (node.comfyClass !== "BFSXYSweep") return;
styles();
const widget = node.widgets?.find(w => w.name === "sweep");
if (widget) {
widget.type = "hidden"; widget.hidden = true; widget.computeSize = () => [0, -4];
widget.options = { ...(widget.options || {}), hidden: true, serialize: true };
widget.serializeValue = () => widget.value;
}
const host = document.createElement("div");
host.style.cssText = "position:relative;width:100%;height:100%;min-width:340px;min-height:380px";
node.addDOMWidget("bfs_xy_sweep", "div", host, { serialize: false, hideOnZoom: false, hideInPanel: true });
wheelToCanvas(host);
const handle = {};
createApp({ setup: () => Panel({ node, get: () => widget?.value ?? "{}",
set: v => { if (widget) widget.value = v; node.graph?.setDirtyCanvas(true); }, expose: o => Object.assign(handle, o) }) }).mount(host);
const onConn = node.onConnectionsChange;
node.onConnectionsChange = function () { const r = onConn?.apply(this, arguments); handle.refresh?.(); return r; };
const onConfigure = node.onConfigure;
node.onConfigure = function () { const r = onConfigure?.apply(this, arguments); setTimeout(() => handle.reload?.(), 0); return r; };
node.size = [Math.max(node.size[0], 380), Math.max(node.size[1], 520)];
},
});