- bg_color options: transparent (default), green, blue, black, white
- Transparent: previews saved as PNG with alpha, video as WebM (VP9 alpha)
- Colored: previews saved as JPG, video as MP4
- SelectActorPreview auto-detects PNG (RGBA) vs JPG (RGB) previews
- Interpolated frames still reuse previous composited frame (no redundant copy)
- FaceAnalysis now tries CUDAExecutionProvider first, falls back to CPU
- ctx_id changed from -1 (forced CPU) to 0 (GPU when available)
- det_size reduced from (640,640) to (320,320) — 4x fewer pixels to process
- requirements.txt: onnxruntime-gpu on Linux, onnxruntime on other platforms
- Previous optimization still active: 10 frame samples, single crop attempt
- Max samples per track reduced from 30 to 10 (8 area + 3 temporal)
- Crop expansion values reduced from [0.6, 1.0, 1.5, 2.0] to [0.5]
(1 crop + 1 full-frame fallback instead of 4 crops + fallback)
- Each face detection call was ~0.4s on CPU; this cuts calls by ~70%
- 200-frame test: 130s (down from ~247s projected before optimization)
The encoding loop was doing frame_lookup + copy + mask write for every
frame including interpolated gap-filler frames. Now only real
detections trigger a full composite; interpolated frames reuse the
previous frame's result (cv2.VideoWriter copies internally on write).
Preview images are now always saved as PNG with BGRA (person pixels
alpha=255, background alpha=0), regardless of bg_color setting.
bg_color only affects the output video.
SelectActorPreview updated to read PNG with alpha, output [N,H,W,4]
RGBA tensor, with fallback to legacy .jpg previews.
Default is transparent (WebM with VP9 alpha). Colored backgrounds output
MP4 with the selected solid color filling non-person pixels.
Transparent previews saved as PNG, colored as JPG.
Only frames where InsightFace detected a face are considered for
preview selection. No fallback to mask bbox area — if no face was
detected, that frame is simply not a preview candidate.
IdentityCluster now records face bbox areas during clustering (when
InsightFace already detects faces). Preview image selection uses
these face areas for sorting, with fallback to mask bbox area when
no face was detected.
- Optimization 1: Downscale frames to 640px max-side before YOLOv8-seg
inference in segmenter.py, resize masks back to original resolution
(~13% speedup from reduced post-processing)
- Optimization 2: Store bool masks instead of composited BGR frames in
MaskActor.frames (~97% memory reduction per tracked frame, from ~6MB
to ~200KB per entry). Green screen now applied on-the-fly during
encoding rather than during tracking.
- Optimization 3: Add skip_every_n parameter to VideoActorExtractor
(default 1=all, 2=every other frame, up to 30). Users can trade
precision for speed.
- Add per-substep timing (Detection vs Tracking) printed after frame loop
- Set PublisherId to ahkimkoo
- Add DisplayName, Repository URL
- Bump version to 1.0.0 for initial registry release
- Switch license from Apache-2.0 to MIT
- Add core/model_utils.py with ensure_models_exist() function
- Auto-download yolov8n.pt and yolov8n-seg.pt from GitHub if missing
- Auto-download InsightFace buffalo_l via InsightFace API if missing
- Handle InsightFace path quirk (models/buffalo_l/ -> buffalo_l/)
- Call ensure_models_exist() at start of VideoActorExtractor.extract()
- Sort preview images by mask bbox area (proxy for face size) instead of mask pixel area
- MaskActor.frames now stores 4-tuple (frame_idx, masked_frame, mask_area, bbox_area)
- SelectActorPreview outputs indexes as INT list (not JSON string)
- Add OUTPUT_IS_LIST for indexes output
- Save frame indexes JSON alongside preview JPGs for each actor
- Update README with full node I/O documentation, workflow screenshot, and workflow file link
- Update examples: add screenshot, update workflow JSON
- Remove photo_index parameter; load all actor_{i}_*.jpg previews automatically
- Return [N, H, W, 3] batch tensor with all preview images for the actor
- Fix return type: must return (tensor,) tuple for ComfyUI merge_result_data
- Add SelectActorPreview node to load per-actor preview images as ComfyUI IMAGE tensors
- Fix SelectActorPreview return value: must return (tensor,) tuple, not bare tensor,
to prevent ComfyUI's merge_result_data from misinterpreting batch dimension as
output count (root cause of 1920-image bug)
- Change VideoActorExtractor third output from IMAGE tensor to INT actor_count
- Save top-5 preview JPGs per actor to {output_dir}/previews/ instead of
building a large preview tensor in memory
- Unify InsightFace model path to ComfyUI/models/video-actor-extract/buffalo_l/
- Add temporal gap penalty with grace gap to identity clustering
- Add temporal boost with Gaussian decay and spatial blocker check
- Add dual-signal track splitting (centroid displacement + bbox area ratio)
- Add spatial centroid proximity tiebreaker for identity merging
- Add debris mask filtering (min_mask_area=20000) in MaskTracker
- Clean up unused imports across modified files
- Add AGENTS.md project guidelines and test infrastructure
- Add ComfyUI Manager and git clone installation methods
- Replace hardcoded paths with ComfyUI/ relative path convention
- Add YOLOv8 model download commands (yolov8n.pt, yolov8n-seg.pt)
- Add InsightFace buffalo_l download via Python API with directory fix
- Include final model directory tree structure
- Update project structure to include nodes/select_preview.py
- Group preview frames per actor instead of flat list
- Pad each actor to exactly 5 frames with black frames
- Update all empty_preview shapes to match new 5D format
- Add debug print for preview tensor shape verification
- Add dual-signal track splitting (centroid displacement + bbox area ratio)
with dominant outlier detection to split mixed-identity tracks before
clustering, so each subtrack gets an independent merge decision
- Move track splitting from cluster_tracks() to actor_extractor.py to
preserve subtrack identity through the pipeline
- Add spatial centroid proximity tiebreaker: when face similarity scores
are within 0.1 margin, prefer merging with the spatially closer actor
- Add temporal gap penalty (gap < max_lost_frames) to respect tracker's
rejection decision, with grace gap (≤2 frames) for timing artifacts
- Add temporal boost (gap ≥ max_lost_frames, no blocker) for same-person
return detection using Gaussian decay confidence
- Add spatial blocker check: tracks in the temporal gap near the candidate
prevent incorrect cross-gap merges
- Add min_mask_area filtering (20000px) in MaskTracker to reject debris
masks from scene transitions
Key improvements:
1. Face detection: Try multiple crop sizes (0.6, 1.0, 1.5, 2.0x expansion) + full frame fallback
2. Frame sampling: 30 frames per track (up from 10), prioritize largest area frames
3. Clustering: Connected components instead of greedy, with temporal constraints
- Temporally adjacent tracks (gap <= 100 frames): normal threshold
- Non-adjacent tracks: require higher similarity (0.75) to merge
4. Debug logging: Print face detection counts and similarity scores
Test results on mmexport1776472655192.mp4 (1 adult + 2 children):
- 4 tracks detected by mask tracker
- Track 0: frames 0-100 (adult)
- Track 1: frames 97-196 (child 1)
- Track 2: frames 97-196 (child 2)
- Track 3: frames 197-288 (adult reappears)
- Similarity: Track 1 vs 2 = 0.982 (very high, likely same child or twins)
- Similarity: Track 1 vs 3 = 0.734, Track 2 vs 3 = 0.725 (adult-child similarity)
- Result: 2 actors (adult + child)
Note: Only 2 actors instead of expected 3 because Track 1 and Track 2 have
very high face similarity (0.982), suggesting they may be the same child or
twins. The face detection may not distinguish between the 2 children.
The renumbering step in identity.py was iterating over actor_map.items()
(old_actor_name -> new_index) instead of track_to_actor (track_id -> old_name),
causing the returned dict to have string keys instead of integer track IDs.
This led to KeyError when downstream code looked up tracks in long_actors.
- Replace ByteTrack mask assignment with centroid-distance MaskTracker
- Every detected person mask gets extracted, no more missed people
- Continuous segment output with interpolation for small gaps
- Accurate start_frame/end_frame/frame_count in JSON
- Replace bbox crop with YOLOv8-seg person segmentation
- Full-frame green screen: only person silhouette kept
- Top 5 preview images per actor by mask area (IMAGE output)
- Accurate time segmentation: only frames where person is visible
- No green gaps between segments
- Add seg_model_path parameter
- video-actor-extract.json: drag-and-drop workflow for ComfyUI
- Includes VideoActorExtractor + Save Text File nodes
- README with parameter tuning guide
- Saved to both examples/ and ComfyUI output/workflows/
- Rewrote __init__.py to use importlib for loading modules
- Avoids conflict between plugin's nodes/ dir and ComfyUI's nodes.py
- Tested in ComfyUI with 1080x1920 18.7s video
- Successfully detected 10 actors with green screen output
- actor_1 correctly shows 2 segments (appeared mid-video, left, returned)
- All actor_info.json and per-actor MP4s generated correctly
Cherokee