Compare commits

..
Author SHA1 Message Date
alisson-anjos acff10803e Fix Color Guide reference history and add global colors 2026-10-13 16:05:39 +00:00
alisson-anjos 71ded2ff5e Add Shot Planner language selector and optional tour 2026-10-13 15:57:23 +00:00
alisson-anjos b911443c71 Add optional Color Guide to Shot Planner 2026-10-13 15:49:54 +00:00
alisson-anjos 8aeb958ab6 Planner: stabilize VLM descriptions and default videos to 15s 2026-10-12 10:27:36 +00:00
alisson-anjosandClaude Opus 5.5 29cedad7be Shot loop: one-shot split mode, BFS Shot Seed, BFS Shot Noise, BFS Sigmas Preview (1.94.0)
- Planner: Mode "One shot (whole range)" (the range as a single shot, Max seconds / shot not applied) and
  "Range length, seconds" next to it, so a whole shot or pieces of at most X are set in the Video tab alone.
- BFS Shot Seed: one seed per shot from a base seed (same / increment / random per shot) with per-shot overrides.
- BFS Shot Noise (experimental, optional): starting noise for SamplerCustomAdvanced on H3's video + audio latent;
  every amount at 0 is exactly RandomNoise. shared_base (the coarse pattern from a base seed, the same across shots,
  resolutions and lengths), static_share, distribution, spectrum_tilt, pyramid, guide_structure, ref_tint.
- BFS Sigmas Preview: chart of a sigma schedule (steps on layout / shapes / detail), pass-through.

Tested: split logic, seed logic, noise statistics (unit variance, identity at 0) and the chart, in Python.
Not tested yet inside ComfyUI / the panel in a browser.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-11 23:47:53 +00:00
alisson-anjosandClaude Opus 5.5 187c9a1776 Find people / Cast without insightface: OpenCV Zoo YuNet + SFace (Apache-2.0, run by OpenCV, downloaded once into models/opencv_face); InsightFace only when installed; SAM 3.1 + colours as last resort
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 19:00:36 +00:00
alisson-anjosandClaude Opus 5.5 a04f3116cf {details} filled automatically by the render nodes too: the planner leaves it when it has no VLM, BFS Shot H3 Conditioning / Shot H3 Duet (new vlm input) describe the shot's references
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 12:48:30 +00:00
alisson-anjosandClaude Opus 5.5 84a953e8c5 source_blur (SAM 3.1, keeps the mouth): blur the source person in the guide / native ref video / duet panel so the model reads pose, motion and lips but has no face to copy - Shot Conditioning, Shot Duet, Duet, Duet Conditioning
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 12:10:27 +00:00
alisson-anjosandClaude Opus 5.5 0bf03a8fb7 Planner: chunked upload for big files (8 MB pieces to /bfs/upload/chunk, assembled on disk) - any size, past ComfyUI's 100 MB --max-upload-size
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:03:16 +00:00
alisson-anjosandClaude Opus 5.5 390f9e8714 Planner UI: ✕ clear all - removes every reference from the plan (global, shots, cast) and the recent list, after a confirm
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 07:50:53 +00:00
alisson-anjosandClaude Opus 5.5 515ab73c70 Planner UI: collapsible sections keep what the user clicked (re-renders snapped them shut); ✕ clear for the recent reference pictures
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 07:47:46 +00:00
alisson-anjosandClaude Opus 5.5 2e64d989f6 BFS Shot H3 Duet: continuity 'history' too (the previous end before frame 0, panel strip filled with the source under it); shared shot_history() pops it from the shot
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 02:17:41 +00:00
alisson-anjosandClaude Opus 5.5 b1e0fcbbb6 Shot Planner: continuity 'history' is the default for new plans and unset shots (cuts still skip it)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 02:14:20 +00:00
alisson-anjosandClaude Opus 5.5 0b5315c45a History / chain: load the previous result on the CPU, slice uint8 before converting, keep history as uint8 in the shot and drop it after conditioning (no idle RAM/VRAM)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 01:39:46 +00:00
alisson-anjosandClaude Opus 5.5 a03e0be342 Shot Planner: continuity 'history' - the last 17/34/51 frames of the previous result placed just before frame 0 on H3's timeline (sliding history, after Akatz's H3 Relay / Ethanfel's Context Loop), with every mask mode, crop and the duet canvas; cuts reset it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 01:37:36 +00:00
alisson-anjosandClaude Opus 5.5 78aa214584 Shot Planner: {details} described automatically before any SAM 3 work (it ran after the masks, which breaks the VLM: the button worked, the run did not); never leaves a hole in the sentence
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 23:03:47 +00:00
alisson-anjosandClaude Opus 5.5 05184747b2 Duet VLM prompt: place and look as lower-case noun phrases inside the sentences
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 21:32:31 +00:00
alisson-anjosandClaude Opus 5.5 596a1c2538 Duet prompts: TSC's newer strategy - the finished split screen as plain footage (no kept footage / generated / in sync), the pinned region named by place + one clause of what is seen ({panel_region}/{video_region}: top quarter, bottom three quarters...), motion beat by beat with times on the video clock, sounds of the new scene
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 21:18:07 +00:00
alisson-anjosandClaude Opus 5.5 4e4fc20fdd Duet: optional VLM prompt - writes it from the task, or fills the {...} fields of a hand-written template (TSC's); setting picture described as <Subject 2>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 21:08:35 +00:00
alisson-anjosandClaude Opus 5.5 458099a2aa Duet: size refs for the whole canvas (as TSC's Reference to Video at canvas size)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 21:01:00 +00:00
alisson-anjosandClaude Opus 5.5 ad99a9f91a Duet: panel_static (TV static over face/person, SAM 3) + setting picture input + BFS Setting Picture node
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 20:52:18 +00:00
alisson-anjosandClaude Opus 5.5 f763b51779 BFS H3 Duet: side_by_side output - the same comparison as BFS Shot Join (original | references | result, info on top, prompt below) (1.83.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 20:21:36 +00:00
alisson-anjosandClaude Opus 5.5 2b462b639a BFS H3 Duet: megapixels input removed again (the size is computed in the workflow instead) (1.82.1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 19:58:56 +00:00
alisson-anjosandClaude Opus 5.5 864a28b161 BFS H3 Duet / Duet Conditioning: megapixels - output size from the clip's aspect ratio at that area (0 = width / height) (1.82.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 19:54:02 +00:00
alisson-anjosandClaude Opus 5.5 e43cc14d9a Points: Track -> whole video - one selection tracked by SAM 3 through the whole video or range (chunked, forward and backward, re-found by text after cuts); shots without their own selection take their part (1.81.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 19:47:21 +00:00
alisson-anjosandClaude Opus 5.5 4fab22caa9 Shot Planner range: a player in the Video tab, start / end in frames (start here / end here take the player's exact frame; seconds of older plans still read), aligned layout (1.80.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 19:09:13 +00:00
alisson-anjosandClaude Opus 5.5 96f7e8a1b1 VLM descriptions: temperature and seed (0 = greedy as before), a dice button to try another description, the short preset capped at 30 words (1.79.1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 18:02:53 +00:00
alisson-anjosandClaude Opus 5.5 34ffcef172 Shot Planner: Start at / End at range - only that part of the video is split, analysed, generated and joined; timeline dims the rest; max total seconds counts from the start (1.79.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 17:42:42 +00:00
alisson-anjosandClaude Opus 5.5 9ae5fc9614 Fit the prompt: only for the short preset, the VLM is told to reply with the blank's words only, and an echo of the prompt sentence is cut; Checks warn when {details} is used with a paragraph preset (1.78.2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 15:14:58 +00:00
alisson-anjosandClaude Opus 5.5 7d5f60ce86 insightface and onnxruntime are optional: no pin in the requirements (the exact insightface pin and its own requirements conflicted with many environments); Find people / LTX Identity say what to install (1.78.1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 13:50:54 +00:00
alisson-anjosandClaude Opus 5.5 8e573275b8 Fit the prompt: descriptions ({details}, {target}) are written for the prompt sentence that holds them (blank in place, article matched); {target} is described automatically at run time for shots with a selection and no text (1.78.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 13:34:08 +00:00
alisson-anjosandClaude Opus 5.5 a5535dc77c Describe refs for any subject (person, creature, robot, animal, cartoon: says what it is first); new default preset 'short' - a noun phrase that fits '<Subject 1> is the {details} shown in <Picture 1>'; {target} describes animals/creatures too (1.77.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 11:35:11 +00:00
alisson-anjosandClaude Opus 5.5 7af3fa5618 pose look: clearer error (package vs model, with the download link) and no restart needed after adding the model (1.76.1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 16:07:09 +00:00
alisson-anjosandClaude Opus 5.5 39610ab1ef mask_guide_look: what the mask guide shows of the subject - grey blurred (default), colour, silhouette, edges, or pose (people, YOLO pose model from models/ultralytics, no new packages) (1.76.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 15:47:14 +00:00
alisson-anjosandClaude Opus 5.5 aa92d72990 Faster masks: separable dilation / blur (same result, ~7x faster on few CPU threads) and the generation mask computed once per shot (1.75.1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 15:35:14 +00:00
alisson-anjosandClaude Opus 5.5 f1388f0463 mask_guide (renamed from person_guide, works for any subject): masked region as an extra guide, instead of the full guide, or as a native reference video (<Video n>, with mask_ref_size); Mask only shots only; Frame + paste takes a per-shot 'new subject' text for the new outline (1.75.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 14:51:56 +00:00
alisson-anjosandClaude Opus 5.5 5c93d23c88 H3 Conditioning: experimental person_guide - a guide latent with only the masked person (rest grey), added next to the full guide or instead of it (1.74.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 14:45:55 +00:00
alisson-anjosandClaude Opus 5.5 7d9d7a9e26 Frame + paste mode: the whole frame is generated (pose follows the guide, the new person may be bigger) and BFS Shot Join pastes only the person (shot mask + the new outline found by SAM 3) onto the original frame; mask strength renamed mask opacity (1.73.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 14:22:16 +00:00
alisson-anjosandClaude Opus 5.5 1de7753be9 Mask strength per shot (mask modes): the masked rows start at strength x the noise level and keep part of the original, like an opacity (1.72.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 14:09:17 +00:00
alisson-anjosandClaude Opus 5.5 aa545097b7 External masks instead of SAM 3: a mask video per shot, the planner's optional mask input (whole video) or a mask video for the whole video in the panel; used by every mode, {target}, the setting picture and the comparison (1.71.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 14:01:13 +00:00
alisson-anjosandClaude Opus 5.5 1b04a822b3 Console logs for masks: H3 Conditioning prints each shot's mode, where its mask comes from, coverage (and empty frames) and how much of the latent is regenerated; SAM 3 runs print what, how many frames and how long (1.70.1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 13:45:07 +00:00
alisson-anjosandClaude Opus 5.5 61a9e732f2 Shot Planner refactor: tabbed panel, per-shot generation modes, Copy to other shots, checks (1.70.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 11:54:22 +00:00
alisson-anjosandClaude Opus 5.5 2022b48e07 Shot Join comparison: mask-only shots show their generation mask (as grown by expand) over the original column, not only cropped shots
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 11:50:44 +00:00
alisson-anjosandClaude Opus 5.5 736a7016e8 Shot Planner: wider node by default (1180 px, at least 1040 for older workflows), panel at least 900 px wide
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 11:32:25 +00:00
alisson-anjosandClaude Opus 5.5 88d0f0c159 Shot Planner: the panel scrolls inside a fixed area so switching tabs never resizes the node (Vue nodes mode sized it by content and could collapse its width); plan JSON widget hidden in Vue nodes mode too
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 11:22:08 +00:00
alisson-anjosandClaude Opus 5.5 1ced5f9798 Shot Planner: mask text updates live, Enter previews it, the old preview clears when the text changes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 11:09:02 +00:00
alisson-anjosandClaude Opus 5.5 718623ecb2 Shot Planner: reorganised panel - tabs (Video, Shots, People & masks, Prompts & refs, Run), views split into web/js/shotloop/, per-shot generation mode (Full frame / Mask only / Crop / Crop + mask) read by H3 Conditioning inpaint = per shot, Copy to other shots, grouped Checks, sideways shot strip with arrow keys; setting picture uses the shot's mask without crop; Auto split keeps {target}
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 10:40:07 +00:00
alisson-anjosandClaude Opus 5.5 907e923816 Shot H3 Conditioning: inpaint 'only the mask' also without crop (whole frame, no crop/uncrop); the mask's expand now grows the generation mask (1.69.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 09:30:48 +00:00
alisson-anjosandClaude Opus 5.5 52000210c3 Shot H3 Conditioning: inpaint 'only the mask (crop)' - latent starts from the cropped shot, the SAM 3 mask becomes the H3 generation mask; credit to Neko (Nekodificador) and AbleJones (1.68.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:44:33 +00:00
alisson-anjosandClaude Opus 5.5 3fc67735b8 Shot Join: stitch finishing for cropped shots - edge hardness, shot-wide colour match (ring around the subject for swaps), seamless edges; adapted from Nekodificador's NKD Inpaint Stitch (MIT) (1.67.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:39:34 +00:00
alisson-anjosandClaude Opus 5.5 ceaec97e63 Shot Join comparison: mask (red) and crop box (yellow) over the original column for cropped shots (comparison_mask) (1.66.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:26:58 +00:00
alisson-anjosandClaude Opus 5.5 0b14d45bf2 pyproject: description covers the current Shot Planner features (1.65.3)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:24:56 +00:00
alisson-anjosandClaude Opus 5.5 e770fb773b Shot Planner: clearer tooltips (mask text is enough, Mask text → all, Crop → all) (1.65.2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:23:08 +00:00
alisson-anjosandClaude Opus 5.5 6d91809e4c Shot Planner guide: step-by-step {target} selection, Save → all, Clear mask, examples and troubleshooting (1.65.1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:20:54 +00:00
alisson-anjosandClaude Opus 5.5 1b53a83860 Shot Planner: save points to every shot, Points → all, Clear mask per shot (1.65.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:18:49 +00:00
alisson-anjosandClaude Opus 5.5 1468279071 Shot Planner: {target} placeholder - pick the person (SAM 3 points / text), the VLM describes them, the prompt names who is replaced (1.64.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 08:16:18 +00:00
alisson-anjosandClaude Opus 5.5 90face345c Shot H3 Conditioning: setting_ref, the shot's frame with the person in TV static as an extra reference (TSC's tip), generation or source size (1.63.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 23:38:02 +00:00
alisson-anjosandClaude Opus 5.5 6015065898 Shot H3 Conditioning: optional duet prompt writer (task + instruction, VLM or template, prompt output); {layout} filled per duet mode; template without split-screen wording in shifted mode (1.62.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 23:26:57 +00:00
alisson-anjosandClaude Opus 5.5 f70b265d47 Duet prompt writer: describe the references and the clip in separate VLM questions (mixed, the VLM described the reference photo as the video) (1.61.1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 19:21:57 +00:00
alisson-anjosandClaude Opus 5.5 e8d2e68bde BFS H3 Duet Conditioning: duet conditioning for your own sampler with a VLM prompt writer (task + instruction); field-based prompt assembly; shifted mode no longer describes a split screen (1.61.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 18:30:36 +00:00
alisson-anjosandClaude Opus 5.5 371e31650c VLM descriptions: physical traits and clothing only (no pose, expression, camera); default max tokens 1024 (1.60.2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 17:08:32 +00:00
alisson-anjosandClaude Opus 5.5 75a0d65d36 Guides open as a formatted pop-up inside ComfyUI (Planner Guide button, Duet help) (1.60.1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 16:16:00 +00:00
alisson-anjosandClaude Opus 5.5 3541a830a6 H3 Duet: How to use window with drawn examples of every setting, advanced settings tucked away; node guides moved under the served web folder (help panel and Guide links work) (1.60.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 15:48:47 +00:00
alisson-anjosandClaude Opus 5.5 1531db27f6 Shot Planner: describe references with the VLM ({details}, presets incl. face attributes, editable), robust VLM generation, tooltips on every setting and a Guide button (1.59.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 15:42:29 +00:00
alisson-anjosandClaude Opus 5.5 4454bd617f Shot Planner: detect the video's audio, use it or a silent track; planner, join and duet always output valid audio (PyAV fallback) (1.58.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 14:52:23 +00:00
alisson-anjosandClaude Opus 5.5 15826c65cf Shot Planner: loading bar with live progress (analysis, cuts, faces, SAM, VLM), upload progress, slow work off the server loop, lighter video analysis (1.57.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 06:42:05 +00:00
alisson-anjosandClaude Opus 5.5 f5463db403 Shot Join: comparison built small and in batches (comparison_height, default 360); queue loop stores originals only for the comparison, small and uint8 (1.56.4)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 01:07:03 +00:00
alisson-anjosandClaude Opus 5.5 e30e8d4ef6 Duet panels keep the clip's aspect ratio by default (contain, no crop) (1.56.3)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 00:38:47 +00:00
alisson-anjosandClaude Opus 5.5 630dc7758a Shot Planner: mask / VLM preview works before any prompt has run (progress hook ids) (1.56.2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 00:35:14 +00:00
alisson-anjosandClaude Opus 5.5 43a31ed934 Shot Join: comparison frames on a multiple of 16 (x264 needs even sizes) (1.56.1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 00:10:58 +00:00
alisson-anjosandClaude Opus 5.5 c37667c70c Shot H3 Conditioning: duet option (canvas / shifted RoPE), the join cuts the panel off (1.56.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 00:07:59 +00:00
alisson-anjosandClaude Opus 5.5 16674de83f Shot Planner: SAM 3 mask & crop per shot with uncrop in the join, global mask settings, points editor, VLM suggestions (1.55.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 23:58:03 +00:00
alisson-anjosandClaude Opus 5.5 7017a7633d Shot Planner: per-shot continuity from the previous shot's result (reference or first frame) (1.54.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 23:31:44 +00:00
alisson-anjosandClaude Opus 5.5 dc2bf3d687 Shot H3 Conditioning: no angle brackets in the guide_mode options (the menu rendered them as HTML) (1.53.2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 23:15:52 +00:00
alisson-anjosandClaude Opus 5.5 78ce6afa91 Shot Planner: Delete on a selected shot removes its cut (1.53.1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 23:03:06 +00:00
alisson-anjosandClaude Opus 5.5 4ddac428c9 H3 Duet: dynamic references, task draft prompts, example and credits; Shot Join comparison output (1.53.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 22:16:20 +00:00
alisson-anjosandClaude Opus 5.5 10681dae03 H3 Duet: prompt template and a hint on how to refer to the side panel (1.52.1)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 21:55:26 +00:00
alisson-anjosandClaude Opus 5.5 002ec88fe0 H3 Duet: shifted RoPE option (video on its own grid, panel past the edge with a gap) (1.52.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 21:47:24 +00:00
alisson-anjosandClaude Opus 5.5 78ec6170c7 BFS Shot H3 Duet: render each shot as a duet, an aligned guide or both; H3 Duet panel optional (1.51.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 21:08:54 +00:00
alisson-anjosandClaude Opus 5.5 51a622c457 H3 Duet / Side Panel nodes (training-free pinned panel); cast on CPU with top faces per frame; Unpack shot output (1.50.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 21:03:00 +00:00
alisson-anjosandClaude Opus 5.5 fc7a026ab7 Shot Planner: face-tracked cast, link a reference per person, split/assign/skip shots by who is on screen (1.49.0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:20:48 +00:00
alisson-anjosandClaude Opus 5.5 24ac461e1f Shot Planner: preview player with shot start/end timecodes (1.48.3)
Play shot plays only the selected shot (stop or loop at its end), every shot card has
its own play button, and Play all plays the plan while skipping shots that will not run.
The panel shows start, end and current time and frame, a playhead follows on the
timeline, and clicking the timeline seeks. The source streams from the input folder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:12:38 +00:00
alisson-anjosandClaude Opus 5.5 e67b22a0ea Shot loop: BFS Shot Repack to put edited pieces back into a shot (1.48.2)
Unpack -> any processing (e.g. remove the reference background) -> Repack. Unconnected
inputs keep the shot's values; replacement guide frames are fitted to the shot's
generation size and length, and timing, cuts and audio are unchanged so the join still
works. Runs once per shot like the rest of the loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:10:32 +00:00
alisson-anjosandClaude Opus 5.5 ea129e03a1 Shot Planner: direct reference upload instead of the input-folder gallery (1.48.1)
The reference picker listed every image in the input folder with thumbnails, which froze
the node on large folders. Reference slots now upload straight from the computer, with
remove and "use for every shot" buttons; the last ten picked references are offered for
one-click reuse. The panel no longer downloads the image list.

The planner also outputs the references the shots use, without repeats (ref_image,
ref_image_2), so a plan that shares one reference returns a single image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 20:01:46 +00:00
alisson-anjosandClaude Opus 5.5 a207ca18bc Shorten the package description to fit the Comfy Registry limit
The 1.48.0 publish was rejected with "description is too long" (1053 chars);
the new text is under the limit and still lists every node family.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 19:42:50 +00:00
alisson-anjosandClaude Opus 5.5 bb87918e57 Shot Planner: content filters with YOLO person/face detection and timeline fill
- Filters per shot from a few sampled frames: needs a person, minimum person size,
  maximum people, needs a face, skip dark/fades, skip static shots, minimum frames.
  People and faces use YOLO from models/ultralytics (Impact Pack person/face models)
  with OpenCV fallbacks; stats are cached per shot.
- Per-shot override (filters decide / always run / always skip) and content badges
  on every shot card; "Analyse people & faces" previews the numbers.
- Planner outputs a timeline with every shot, including the ones that do not run.
  BFS Shot Join fills skipped shots with the original video (full duration and
  soundtrack) or removes them together with their audio.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 17:39:01 +00:00
alisson-anjosandClaude Opus 5.5 bf473e1dbd Shot Planner / Shot Loop: split long videos into model-sized shots (1.48.0)
A video model only follows a guide inside the clip length it was trained on, so long
sources are split into shots that fit and run once per shot.

- BFS Shot Planner: interactive Vue timeline (thumbnails, cut-detector curve, draggable
  boundaries, split/merge, per-shot reference gallery with upload, per-shot prompt).
  Splits at camera cuts via PySceneDetect (adaptive/content, built-in fallback), fixed
  length, or by hand; lengths snap to the model grid (H3 17n+5, LTX/Wan 8n+1, 4n+1).
  Outputs the shots as a ComfyUI list so everything downstream runs once per shot.
- BFS Shot Unpack: guide frames, references, prompt, length, audio of one shot.
- BFS Shot H3 Conditioning: native MiniMax H3 conditioning per shot (aligned guide,
  native reference video, or both).
- BFS Shot Join: concatenates in order, trims each shot to its true length, cross-fades
  soft joins with the real overlap frames, returns the soundtrack.
- Queue loop mode: one shot per run stored on disk, auto re-queue, nodes after the join
  blocked until the last shot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 17:32:32 +00:00
Codex f145527ea5 Add H3 reference RoPE options and caption-only identity conditioning 2026-09-30 20:30:01 +00:00
Codex bd23236bdd Add aligned downscaled MiniMax H3 guides and image/video targets 2026-09-30 17:26:56 +00:00
alisson-anjosandClaude Opus 5 5c8c41abbd LoRA Surgery: carry a recipe to the save node without copying JSON (1.45.0)
Getting a recipe from the interactive node into the save node meant copying the rules JSON
between two text widgets, which is a bad way to move state and easy to get wrong.

Two paths now. BFS LoRA Surgery gained a `rules` output that can be wired straight into the
save node's `rules` input, so the file is produced as part of the run. And the panel has a
"save as file" button with an optional name, which posts to a new /bfs/lora/save route and
reports the result inline, for when you are still exploring and do not want a second node in
the graph.

Both go through one write_surgery function, so the two paths cannot drift apart. That
function is the old node body lifted out, not a copy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-27 12:33:29 +00:00
alisson-anjosandClaude Opus 5 e72dd2569a LoRA Surgery: detect the block axis from the file, and list every layer (1.44.0)
The parser took the last numeric component of a key as the block index. That is wrong
whenever a name carries more than one number: transformer_blocks.12.attn1.to_q would be read
as block 1 of a family ending in attn, and a nested layout such as
down_blocks.0.attentions.1.transformer_blocks.0.ff.net.0.proj collapsed into a family called
ff.net. The block axis is now chosen per path template as the numeric slot that varies most
across the whole LoRA, which is information a single key does not have. Numbers outside that
slot stay in the module name, so attn1 and attn2 stay distinct rather than folding together.

The grouped tree is a summary, and a summary can hide things, so the panel now also has an
"every layer" view: the full module list with a filter box, each row showing its norm and
rank, and a + that writes a rule matching exactly that path. The structure endpoint returns
that list alongside the grouped one.

Checked against the body-swap recipe: same 136 of 192 modules kept, 56 dropped, 96 scaled as
before the change, and attn.to_out.0 is still treated as attention rather than as block 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-27 12:30:03 +00:00
alisson-anjosandClaude Opus 5 5073cf00ae LoRA Surgery: keep the panel's configuration across reloads, and stop implying Qwen names (1.43.2)
Two fixes.

The panel lost its configuration when a workflow was reopened. The Vue app mounts in
nodeCreated, which runs before ComfyUI restores widget values in configure, so it read an
empty widget and the first edit would then overwrite the stored recipe. The node now wraps
onConfigure and tells the panel to re-read once values have landed, and the rules widget is
explicitly marked serializable with a serializeValue, so the configuration travels with a
shared workflow rather than living only in the session.

The widget now stores {rules, ui} instead of a bare array, so the selected family, type,
block range and scale come back too, not just the rules. A bare array is still accepted, for
workflows saved by the previous version and because the Python side takes either shape;
verified both produce identical results.

The inline guide named gate_up as if it were universal. It now says up front that names
depend on the base model, gives the three architectures verified so far with their actual
family and module names and block counts, and frames the finding as the MLP input projection
whatever the model calls it, explicitly a starting point rather than a rule.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-27 12:24:49 +00:00
alisson-anjosandClaude Opus 5 ee4d2eecb8 LoRA Surgery save: bounded filenames and metadata that explains the file (1.43.1)
Filenames could exceed the OS limit: the slug was capped at 110 characters but the final
name is stem + slug, so a long source name pushed it past NAME_MAX (255 bytes on Linux, and
Windows caps the whole path near 260 without long paths). Names are now fitted to 160 bytes,
trimming both halves and appending a short hash of the full recipe when needed, so two
different recipes still never produce the same name. A path over 240 characters logs a
warning, since the folder above us is not ours to control.

Metadata now records what was applied rather than just that something was. The original
metadata is preserved untouched, and next to it:

  bfs_surgery_summary  one readable line, for viewers that know nothing about this tool
  bfs_surgery          tool, UTC timestamp, source filename and its sha256, the readable
                       recipe, the exact rules, and module counts (total/kept/dropped/scaled)
  ss_output_name       updated to the new filename

The sha256 matters more than the filename: it ties the result to the checkpoint it came from
even after either one is renamed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-27 12:15:30 +00:00
alisson-anjosandClaude Opus 5 ce11d082fd LoRA Surgery: a save node that names the file after the recipe (1.43.0)
BFS LoRA Surgery (save) applies the same rules and writes a .safetensors into models/loras,
for when a recipe is settled and you want it usable outside the workflow that produced it.

With no filename given, the name is derived from the rules themselves:
mylora__attn-x1.15__gate_up-b8_15-off__all-b16_23-off.safetensors. Names get sanitized, so a
type like *mlp* or a regex match still yields a valid file. The recipe also goes into the
safetensors metadata under bfs_surgery, next to the original metadata, so a shared file can
explain itself later.

Verified against the study output: saving the body-swap recipe produces the same 136 of 192
modules, and the effective dW matches the file produced by the standalone script to 3e-3
relative. The factors differ because the node scales only the up/B side while the script
split the scale across both, but the product is what the model sees.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-27 12:09:13 +00:00
alisson-anjosandClaude Opus 5 309daf6b38 LoRA Surgery: drag the block ruler instead of shift-clicking (1.42.3)
Shift-click was the wrong gesture inside a node: on the ComfyUI canvas it starts a text
selection, so picking a range fought the editor. Ranges are now painted by dragging across
the ruler, with pointerdown calling preventDefault and stopPropagation so the drag does not
reach the canvas and move the node.

Click-to-toggle still works, since a click is a one-cell drag, and the gesture adds or
removes depending on the cell you start from. Also added a text field that both shows and
accepts the selection (8-15, 0,4,7, 8-15,24-31) and shortcuts for all / first half / last
half / invert / none, which cover most of what the ruler is used for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-27 12:00:49 +00:00
alisson-anjosandClaude Opus 5 e79d590fd5 LoRA Surgery: controls on top, reference below, and an inline guide (1.42.2)
The panel opened with the layer tree, which pushed the controls off screen on a LoRA with
32 blocks: you had to scroll past the reference material to do anything. Order is now
header, guide, Selection, Rules, Layers.

The guide is a collapsed block that explains what the node is for, how to read the ||dW||
bars, where to look first (the MLP input projection), that later rules override earlier
ones, and the two things worth knowing before trusting a result: a variant can win every
metric and still have destroyed what you trained, and pruning after training is not the
same as training with those layers excluded.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-27 11:59:56 +00:00
alisson-anjosandClaude Opus 5 be0bf3e7ca Drop em dashes from the node's prose (1.42.1)
Plain punctuation reads better and does not carry the stylistic tell. No behaviour change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-27 11:49:10 +00:00
alisson-anjosandClaude Opus 5 816d942fb8 Release 1.42.0 — LoRA Surgery
Bumps the version so the registry workflow publishes, and keeps the vendored Vue out of
ComfyUI's extension scan: server.py globs **/*.js recursively under WEB_DIRECTORY and
imports every hit as an extension, which would have pulled in the Vue bundle as if it
were one. Renaming it to .mjs sidesteps that — mimetypes still resolves it to
text/javascript, so the module import is unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-27 11:37:26 +00:00
alisson-anjosandClaude Opus 5 e9a27c765c BFS LoRA Surgery: read any LoRA's layers and blocks, scale or drop them live
A defect a LoRA learned — smoothed skin, identity drifting at distance, a pose it
refuses to copy — usually lives in one module family and a range of blocks, not in the
whole adapter. This node makes that testable without retraining: pick a family, pick
blocks, set a scale, queue. Nothing is written to disk, so iterating is a slider away.

Model-agnostic by construction. The structure is discovered from key names rather than
hardcoded, covering the diffusers (lora_A/lora_B) and kohya (lora_down/lora_up + alpha)
conventions. Verified against three architectures with different naming: Qwen-Image-2.1
(transformer_blocks, 32), Krea 2 (blocks 28 + txtfusion) and MiniMax-H3 (blocks 50 +
token_refiner).

Scaling multiplies only the up/B factor, since (sB)A = s(BA) — exact under either alpha
convention, and negative scales stay usable.

The panel is a Vue app in a DOM widget, with Vue vendored so it works offline. It lists
each family's module types sorted by ||dW||, which reads as a map of where training
invested. Selections convert to a regex on demand for cases the tree cannot express.
Rules are ordered and later ones win, so the UI says so and lets you reorder.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-27 11:26:37 +00:00
alisson-anjos 12bd7a73b1 Shape the crop to the latent's aspect, not the latent to the crop (1.41.1)
1.41.0 fixed the wrong half of this. The deformation came from the crop and the
latent having different ASPECTS -- a mask around a face gives a portrait box,
sampled in a landscape latent it is stretched on the way in and squeezed back on
the way out -- and rebuilding the latent at the crop's size did remove the
mismatch, but by throwing away the resolution that cropping exists to buy. A
face 200 px tall in the source was then sampled at 200 px, which is the problem
the crop was there to solve.

The planner already takes an aspect_ratio and we were passing 0.0, letting the
box follow the subject. It now gets the connected latent's aspect, and
_static_box grows its short side to match when the planner is not in play. The
box still contains the subject and stays divisible by the grid.

So the latent's size is yours again -- it is the sampling resolution, choose it
for the quality you want -- and its shape no longer has to be guessed.

68 tests; the new ones pin that a landscape latent gives a landscape box, that
the box still contains the subject, and that a free aspect still follows it.
2026-09-03 17:08:17 +00:00
alisson-anjos dab4233e48 Rebuild the latent to the crop instead of asking for it (1.41.0)
The crop's size is derived from the mask, so an EmptyLTXVLatentVideo built
upstream cannot know it -- and a latent that disagreed was not an error
anywhere. The model sampled at the latent's shape and the paste-back squeezed
the result into the box: a face crop is portrait, sampled into a landscape
latent it came back visibly deformed, and under crop_mode zoomed the deformation
changed frame to frame. The only cure was to run once, read the size off the
debug string, rebuild the latent and run again, after every change to the mask.

The sampler now rebuilds the latent's video stream at the crop's shape when they
disagree, keeps the audio stream as it is, and reports it in debug. Nothing is
lost: at denoise 1.0 the initial latent contributes nothing, and the inpainting
path overwrites the video stream with the encoded guide regardless.

BFS Crop Size stays useful for building the latent at the right size up front
(skipping the rebuild) and for taking the per-frame boxes straight to BFS Paste
Back, but it is no longer the price of a correct result.

68 tests.
2026-09-03 16:52:09 +00:00
alisson-anjos c145b091d7 BFS Crop Size: the crop's size before anything is sampled (1.40.0)
The sampler derives its box from the mask, so the size to build the empty latent
at was only knowable after a run: sample a whole clip, read one number off the
debug string, rebuild the latent, sample again -- and repeat after every change
to the mask or the crop widgets.

Getting it wrong does not look like a sizing problem. A mask around a face in a
1920x1080 frame gives a portrait box, 512x672 say; sampled into a 768x512
landscape latent the crop is stretched horizontally, and the paste-back then
squeezes the result back into the portrait box. The face comes out deformed, and
under crop_mode: zoomed the deformation changes frame to frame.

The new node runs the same planner on the same mask and returns width, height,
the per-frame boxes and a note, so the latent can be built up front. The planner
is deterministic, so identical crop widgets give the sampler the identical box.

Also factored the per-frame box normalisation the sampler did inline into
_boxes_per_frame, which both nodes now share.

63 tests.
2026-09-03 16:50:39 +00:00
alisson-anjos b5d08490b0 Drop the overlap when concatenating chunks (1.39.2)
Chunked sampling built its chunks with an overlap for continuity and then
concatenated them whole, so the overlap was emitted twice. 97 frames at tile 72
overlap 16 gave 15 latent frames where the clip has 13: the output ran longer
than the input, and from the first seam on, frame i was no longer moment i.

Pasting back made it obvious rather than subtle -- crop_bboxes has one box per
source frame, so every frame after the seam went into a box belonging to a
different moment, and with crop_mode: zoomed into a box of a different size too.
That is the "crooked and out of sync" this was reported as.

Each chunk after the first now contributes only its tail, the part the previous
chunk did not already cover, so the concatenation is exactly the clip's latent
length. Single-chunk runs are untouched.
2026-09-03 15:45:58 +00:00
alisson-anjos 221d9382a0 Stop import comfy.nested_tensor from shadowing the module (1.39.1)
`import comfy.nested_tensor` inside execute() binds `comfy` as a LOCAL for the
entire function, so every other comfy.* use in it raises UnboundLocalError -- at
compile time Python decides the name is local, and the AV branch that would have
bound it does not run on a video-only latent.

It shipped in 1.37.0 with the chunk-offload line:

    got = got.to(comfy.model_management.intermediate_device())

which only runs when a clip is long enough to be sampled in more than one chunk.
That is why it survived the tests and the short runs and failed on the first
real long video:

    UnboundLocalError: cannot access local variable 'comfy'

Imported as `from comfy import nested_tensor as _nested` now, which binds
`_nested` and leaves the module-level `comfy` visible. A test asserts neither
node's execute() has `comfy` in co_varnames, which catches the whole class of it.
2026-09-03 14:44:49 +00:00
alisson-anjos c79629dc80 Confine the paste to the mask even with no crop (1.39.0)
_paste_back gave up when there was no crop box and returned the sampler's frame
whole. Nothing outside the mask *changed*, but the entire frame had been through
the VAE's encode/decode, so every pixel the edit never touched came back softer.
For a pointwise edit, or anything sharpness-adjacent, that is the opposite of
what the mask was for.

With no crop and a mask, the full frame is now composited into the ORIGINAL
frames through that mask, the way the crop path already did: outside the mask
the output is the source verbatim, never a round trip of it. The mask's own blur
supplies the soft edge -- there is no box border to feather here.

This makes crop_mode a resolution decision again. Before it, preserving the
untouched pixels required turning the crop on even when the region already had
all the resolution it needed.

52 tests; the new ones pin that outside is the source byte for byte, inside is
the result, no mask still composites nothing, and a result of another size is
brought back to the frame first.
2026-09-03 14:27:51 +00:00
alisson-anjos df32bdc0fe BFS Sampler: optional identity, and put the new widgets back at the end (1.38.0)
Three things.

`identity_image` moves from required to optional. The node was never head-swap
specific below the surface -- the reference slot is simply not packed when it is
empty, and everything else (crop, mask, inpainting, chunking, paste-back) is
indifferent to which LoRA is running. An instruction-edit, restyle or sharpening
IC-LoRA now runs on the guide alone. IMAGE is a link, not a widget, so this
costs no widget position.

Renamed to "BFS Sampler (crop · mask · loop)" and "BFS Paste Back". The
registration keys are deliberately unchanged: saved workflows reference those,
and only the display name is free to move.

And the fix for the reason this was noticed: 1.37.0 inserted auto_config and
identity_headroom BEFORE crop_mode. ComfyUI stores widgets_values positionally,
so every later value in every saved workflow shifted two slots -- crop_mode's
string landed in auto_config, crop_scale's float in identity_headroom, and the
graph failed validation with "input out of range" on crop_scale and
crop_divisible_by. Both widgets are appended at the end now, with a comment
saying why, and a test pins the historical order so the next addition cannot
repeat it. ltx_multiple_controls.py already carried that rule in a comment;
this file did not follow it.

48 tests.
2026-09-03 14:07:39 +00:00
alisson-anjos a2b499ce3e Stop the auto config from blowing the mask hole open (1.37.1)
Two compounding mistakes in 1.37.0's derivation, reported as a hole much larger
than the head with the new head generated floating inside it:

- the upward slack was scaled by the mask's HEIGHT (hh * delta / tilt). Height
  swings with how much neck the mask took, and a portrait-shaped identity image
  (tilt < 1) multiplied it further: a 300x400 head with a 0.6 reference asked
  for 136 px above. Every slack is now a fraction of the head's WIDTH, the
  stable dimension, capped at 25% sideways and 30% up.
- latent_mask_dilate was derived from mask_grow and forced to at least one
  cell, which added a whole 32 px on EVERY side -- including down, which the
  pixel grow deliberately holds at the base -- on top of a grow that already
  carried the slack. It is 0 now: _mask_to_latent reduces with MAX, so any cell
  the mask touches is fully editable already. The same head asked for 4 cells,
  128 px, before this.

Also undo the guider injections in reverse: with several chunks each injection
captured the state the previous one left, so undoing forwards restored chunk
1's specs onto the guider instead of clearing them.

Tests pin both: growth stays under 45% of the head's width across every
reference aspect and headroom, the latent dilation stays 0, and the crop box
always holds the grow plus the feather.
2026-09-01 14:29:22 +00:00
alisson-anjos 5dd563661f Name the Head Swap Sampler in the package description
The description still listed the pack as it was several nodes ago; the sampler
and LTX Multiple Controls are what people install it for now.
2026-09-01 12:42:11 +00:00
alisson-anjos f4af1cae69 Auto config from the mask, and give the guider back its options (1.37.0)
auto_config measures the subject in subject_mask -- width, height, travel, size
change, per-frame step -- and derives crop mode, crop scale, mask grow/blur,
paste feather and latent dilation from it. Every amount in this node is in
pixels, and a pixel only means something relative to how big the head is in
frame: the same 8 px is generous on a distant head and invisible on a close
one, and that mismatch is what leaves a seam.

The mask is the OLD head, so growing it by a fixed fraction only moves the
clip when the new head is wider or has more hair. The reference's proportions
are measured (identity_image's aspect against the mask's) and decide WHERE the
slack goes; its absolute size cannot be measured at all -- a cropped head
carries no scale -- so that part is identity_headroom, default 1.15. Growth is
anisotropic: up and sideways, never down into the neck and collar, which stay
the guide's own pixels. The margin between mask and crop border is a budget:
the grow takes what it needs, the feather gets half of what is left, floored --
a ramp wider than its share fades the head itself.

Two fixes for state that outlived a run:

- the transformer_options written into the guider are now undone in a finally.
  The guider belongs to another node and survives in the execution cache, so
  the reference latents stayed reachable (VRAM never freed) and a later run
  that did not overwrite the same keys sampled with the previous clip's specs.
- chunked clips move each sampled chunk to the offload device as it comes out,
  instead of holding all of them on the sampling device.

40 tests, including that each side grows by exactly its own amount (the pad is
mirrored: padding the top grows the mask downward) and that the feather never
reaches past the margin it has.
2026-09-01 12:40:30 +00:00
alisson-anjos 9cdd19ca2d Hard denoise mask, and dilation in latent cells (1.36.0)
The node was feeding one mask to two jobs with opposite needs. A soft edge
hides the seam when compositing, but in the denoise mask it means partial
denoising: the original latent, and with it the original identity, blends
back in exactly at the edge of the head. mask_hard_for_inpaint (default on)
binarises the mask for the sampler and leaves the soft one for the paste.

latent_mask_dilate grows the mask by whole latent cells after the reduction,
which is the scale that actually decides whether the head sits inside
editable blocks -- 32 px per cell, 8 video frames per latent frame. On a test
mask one cell went from 11% to 25% of the grid. latent_mask_dilate_frames
does the same along time. The latent_mask output reflects both.
2026-08-31 22:30:00 +00:00
alisson-anjos d6f3714e98 Recommend the external decode in the help page (1.35.1)
Internal decoding exists only because the paste-back needs pixels. For
anything past a single pass the graph's own VAE Decode (Tiled) is the better
tool -- tunable, visible, already there -- so the docs now say so and point
at decode: none plus the paste-back node.
2026-08-31 21:44:02 +00:00
alisson-anjos 2097d6f4b8 Add tiled and skippable VAE decode (1.35.0)
A second pass after a 2x upscaler leaves a latent the full decode cannot
hold: VRAM pins at 99%, the GPU oscillates between 0 and 100 as tensors
shuttle, and it reads as a hang. Graphs solve this with VAE Decode (Tiled);
the node had no way to.

decode: full | tiled | none, with the usual tile/overlap and temporal
controls. none returns the latent alone, for feeding your own tiled decode
downstream -- and then there is nothing to paste back, so compositing is
skipped too.
2026-08-31 21:43:30 +00:00
alisson-anjos 982ad94324 Make crop_bboxes a socket, not four widgets (1.34.2)
Without forceInput the BOUNDING_BOX input renders as editable x/y/width/height
fields, inviting a hand-typed box -- which would paste every frame into the
same fixed rectangle instead of following the planned path.
2026-08-31 21:20:52 +00:00
alisson-anjos 3d0160b0d5 Declare DESCRIPTION and the web folder so the help button shows (1.34.1)
The frontend builds node help as /extensions/<pack>/docs/<NodeID>.md, served
from the pack's registered web dir -- which ComfyUI only registers when the
folder exists at boot, and which the registry path reads from [tool.comfy]
web. Declared both, plus DESCRIPTION on each class, since the classic API
ignores docstrings.
2026-08-31 21:02:07 +00:00
alisson-anjos 1ba2131b04 Allow deferring the paste-back for a second pass (1.34.0)
With cropping on, the sampled latent is the crop's, which is exactly what a
refinement pass wants: the face has pixels there and the surrounding frame
never needed a second pass. But the node composited before returning, so
there was no way to get between the two.

paste_back: off now keeps images and latent in crop space, crop_bboxes comes
out alongside them, and the new BFS Head Swap Paste Back node composites at
the end with the same edge-aware feather and mask confinement.
2026-08-31 21:00:40 +00:00
alisson-anjos b440176096 Edge-aware feather and mask-confined paste-back (1.33.1)
Two behaviours the reference uncrop has and ours did not.

Sides of the crop that sit on the image edge are no longer feathered: there
is nothing outside to blend into, and fading there leaves a washed band along
the frame border.

paste_confine_to_mask (default on) pastes only inside the mask rather than
the whole crop rectangle, so whatever the model invented in the crop's
background never reaches the final frame.
2026-08-31 20:45:52 +00:00
alisson-anjos d96761df1f Seed the latent from the guide when inpainting (1.33.0)
Inpainting preserves the INITIAL latent outside the mask. The node was
handing it the empty latent, so everything outside the mask came back grey
instead of the source video -- the encode that a normal inpaint graph does
before Set Latent Noise Mask was simply missing.

The guide chunk is now VAE-encoded and becomes the sampling latent whenever
a mask is in play. On AV latents the audio stream from the connected latent
is kept and only the video stream is replaced.
2026-08-31 20:44:58 +00:00
alisson-anjos 83cd826963 Read the planner's boxes as dicts, and outline them per frame (1.32.2)
The vendored planner emits boxes as {"x","y","width","height"} dicts, not
(x,y,w,h) tuples, so tracked/zoomed died with KeyError: 0 on paste-back.
Both shapes are accepted now.

mask_over_source also draws the moving box, one per frame, so the planned
modes can be watched the same way the static one could.
2026-08-31 20:43:21 +00:00
alisson-anjos 7433c8457e Serve the help page from web/docs, and add mask_strength (1.32.1)
The frontend fetches node help from /extensions/<module>/docs/<NodeID>.md,
which is served out of WEB_DIRECTORY -- a docs/ folder at the pack root is
never reached. Moved to web/docs/.

mask_strength controls how completely the masked region is replaced. It is
NOT how you get expressions: those arrive through the aligned guide, which
carries the whole source performance regardless of the mask. Below 1.0 the
original latent blends back, keeping the original geometry and dragging the
original identity along with it -- an average of both faces. Documented as
such so it is not reached for by mistake.
2026-08-31 20:39:16 +00:00
alisson-anjos c19d2bb8a1 Vendor the crop planner so the node stands alone (1.32.0)
The sampler no longer reaches into a sibling pack at runtime: the crop path
planner and its helpers now live here as bfs_crop_planner.py and
bfs_subject_crop.py, copied from drozbay/MaskVidExperiments (d98cc89, v0.2.0).
Both packs are GPL-3.0, which is what makes the copy legal; attribution is in
NOTICE and in each file's header. Only the ComfyUI node classes were dropped
-- they stay in the original pack, where they belong.

tracked/zoomed therefore work out of the box now, and the per-frame paste-back
is ours rather than a call into their uncrop node.

Also adds the node's help page (docs/BFSHeadSwapMaskedSampler.md, the '?' on
the node): what each crop mode does, why the mask is native inpainting rather
than a LoRA feature, why latent_mask is the output to check first, how to size
the latent to the crop, and the order to switch things on when a run looks
wrong. scipy is now a declared dependency (the planner uses ndimage).
2026-08-31 20:34:39 +00:00
alisson-anjos 5203e8d30c Resize the sampled crop to the box before pasting it back (1.31.1)
The sampled size comes from the connected latent, and the zoomed crop mode
rescales crops by design, so the result is not guaranteed to match the box
it came from -- pasting then fails on a shape mismatch. Resize first, and
report the size the latent should be to avoid the resample entirely.
2026-08-31 20:29:43 +00:00
alisson-anjos 72cb9fe8fa Add inspection outputs: mask over source, cropped guide, crop and latent masks (1.31.0)
Debugging this pipeline blind is the slow way. Five outputs now say what the
node actually did:

mask_over_source  the mask painted on the ORIGINAL frames with the crop box
                  outlined -- is the mask where you think it is?
cropped_guide     exactly what the model was fed
crop_mask         the mask after grow/blur, in the crop's pixel space
latent_mask       the mask as the SAMPLER sees it: max-reduced to the latent
                  grid and upsampled back. One frame per latent frame, which
                  is the real resolution of the edit and where a thin mask
                  disappears entirely
debug             crop mode and box, mask ops, tiling
2026-08-31 20:08:19 +00:00
alisson-anjos 5d5703238b Inject transformer_options into the guider, and decode only the video stream (1.30.3)
CFGGuider captures model_patcher.model_options BY REFERENCE when it is
constructed (comfy/samplers.py:937), so swapping guider.model_patcher does
not propagate -- the guider keeps sampling with the dict it captured, and
the reference specs LTXMultipleControls wrote on the clone never reach the
forward. That is why the guide had no effect at all despite the node
reporting two active slots. The keys are now mutated into the guider's own
transformer_options in place, the way the Edit Anything looping sampler
already does it.

LTX-2.5 latents are AV, and the video VAE takes the video stream alone --
what LTXVSeparateAVLatent does in the stock graphs.
2026-08-31 20:06:23 +00:00
alisson-anjos 63dad57cc2 Take the latent from the graph instead of fabricating one (1.30.2)
LTX-2.5 latents are AV: video and audio nested together, which is why the
working graphs carry an LTXVSeparateAVLatent. Building a plain
[1,128,t,h,w] tensor hands the model a structure it does not expect, and
the guide stops reaching it -- the run returns the reference face and none
of the source clip.

The node now takes an optional latent (connect EmptyLTXVLatentVideo, sized
to the crop when cropping), slices it per chunk when it is video-only, and
refuses chunked sampling on a nested AV latent rather than corrupting it.
The old fabricated latent stays as a last resort, with a warning.
2026-08-31 20:00:28 +00:00
alisson-anjos 87355cb291 Swap the guider's model_patcher per chunk, and decode the tensor (1.30.1)
Two bugs from the first cut, both caught on the first real run.

_sample_chunk samples through guider.model_patcher, so passing the patched
clone as the model argument did nothing: the reference specs sat in
transformer_options that the forward never read, and guide + identity were
silently inert -- sampling ran to completion and ignored both. The pack
already had _set_guider_conds for exactly this; use it.

vae.decode takes the sample tensor, not a latent dict.
2026-08-31 19:55:09 +00:00
alisson-anjos 3e77a11908 Add the Head Swap masked sampler: stable crop, native mask inpainting, temporal loop (1.30.0)
The LoRA keeps doing only its job -- guide on source_id 1, identity on
source_id 2. The mask never reaches it: it becomes latent["noise_mask"],
which the guider consumes as denoise_mask, so only the masked region is
denoised and the original face stays visible to the model.

Masks are reduced to the latent grid with MAX rather than ComfyUI's
trilinear resize, which blurs a mask across frames and lets the original
content bleed through the edit.

Crop exists for the case a LoRA cannot fix: a face too small to carry
identity. tracked/zoomed borrow drozbay/MaskVidExperiments' planner when
that pack is installed -- its boxes hold still through mask noise and
occlusion, and a jittering crop reads as camera motion to a video model.
Without it, one static box around the subject's whole travel.

Helpers are unit-tested; the sampling path still needs a GPU run.
2026-08-31 19:38:25 +00:00
alisson-anjosandClaude Opus 5 0a25538692 Do the layout maths in float, not in the coordinate dtype (1.29.3)
ComfyUI's pixel coords are int64 -- latent corners multiplied by integer VAE scale factors --
while centring a reference on the target's height and taking a clip midpoint are half-pixel
operations. Adding a float shift into a Long tensor in place raises "result type Float can't
be cast to the desired output type Long", which is where the sidecar layout landed next.

_apply_tass_layout now promotes both coordinate tensors to float32 for the shift and rounds
back to the caller's dtype on the way out. Rounding matters: the caller casts the reference
coords to the target grid's dtype when concatenating, and a cast truncates where rounding is
what a half-pixel centre wants.

The earlier tests missed this because they built float tensors -- test_tass_layout now runs
every layout on int64 input and checks both the dtype and that the result stays within half
a pixel of the float placement.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HGYHCVjHe6acihEpCRyn1K
2026-08-23 17:01:25 +00:00
alisson-anjosandClaude Opus 5 268f00c1a7 Place the sidecar reference correctly on 2.3 as well (1.29.2)
1.29.1 stopped the crash but only got the placement right on 2.5. The temporal span the
trainer writes -- bounds = (t_min, t_max) of the target clip -- is collapsed to one number
per token by the model, and the two LTX generations do it differently:

  use_middle_indices_grid=True   (2.5, and any checkpoint whose metadata declares it)
                                 -> (start + end) / 2, i.e. the middle of the clip
  use_middle_indices_grid=False  (2.3 checkpoints that predate the flag)
                                 -> the start bound alone, i.e. frame zero

So the same bounds mean "centre of the clip" on 2.5 and "pinned to frame 0" on 2.3 -- one
LoRA, two placements, and the 2.3 one is exactly the single-instant conditioning sidecar
exists to avoid. The layout now takes the model's own flag (read off the transformer in
_process_input) and writes the midpoint into both bounds when the model reads the start,
which lands the reference in the same place under either rule. Legacy [B, 3, N] coordinates
get the midpoint for the same reason.

Verified by collapsing the coordinates the way generate_freqs does and measuring the
reference against the target in the same frame: 2.5 is bit-identical to the trainer, and
2.3 matches to within half a patch (the coordinate convention's own offset, which applies
to the target's tokens equally).

tests/test_tass_layout.py covers both shapes, both collapse rules, the batch-1 reference vs
batch-2 target that crashed, the margin, and that patch spans are translated rather than
squeezed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HGYHCVjHe6acihEpCRyn1K
2026-08-23 16:53:16 +00:00
alisson-anjosandClaude Opus 5 1fc84f4f37 Fix the sidecar layout for LTX-2.5 coordinates (1.29.1)
Two wrong assumptions in 1.29.0, both hit by the first real run:

  shape  -- LTX-2.5 hands out patch bounds, [B, 3, N, 2], not the one-corner-per-token
            [B, 3, N] the node was written against. The other layouts survived it by
            accident (their math is elementwise), but each was reducing over tokens only,
            so a patch's start and end bound were shifted by their own separate minima --
            which squeezes the span instead of translating it.
  batch  -- layouts run BEFORE the reference tokens are expanded to the CFG batch, so the
            target arrives [2, ...] while the reference is still [1, ...]. overlap/st_drc
            broadcast out of it; sidecar assigns in place and raised
            "The expanded size of the tensor (1) must match the existing size (2)".

_tass_axis_stats now reduces over every dimension except the T/H/W one -- tokens, the
bounds axis when present, and the batch -- which fixes both at once and keeps the older
[B, 3, N] shape working. Verified against ltx_core.conditioning.reference_layout: all four
layouts now agree with the trainer bit-for-bit on the 2.5 shape, in the exact batch-1
reference vs batch-2 target configuration that crashed.

Also passes the spec's sidecar_margin_pixels through instead of always using the default.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HGYHCVjHe6acihEpCRyn1K
2026-08-23 16:46:38 +00:00
Claude 1936d5dd5e Add the sidecar layout to the LTX conditioning nodes (1.29.0)
The trainer has had four reference layouts for a while — overlap, st_drc, strata
and sidecar — but the nodes only ever exposed three. Training a LoRA with the
reference in sidecar therefore had no matching inference path, which is how you
end up running a checkpoint in a layout it was never trained in.

sidecar places the reference beside the target on the W axis, vertically centred
on it, and stretches its T span to cover the target's whole clip, so an object
reference stays available for every frame instead of sitting at one instant. It
is the layout the replace_refedit recipe validated on 2.3 for exactly this role.

Mirrors ltx_core.conditioning.reference_layout, adapted to ComfyUI's [B, 3, N]
coordinate tensor instead of the trainer's [B, 3, N, 2] patch bounds, and takes
sidecar_margin_pixels like the trainer does.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 13:56:18 +00:00
Claude 8eb8a0546c Add deterministic Edit Anything prompt fixer (1.28.0)
Two formatting rules the VLM breaks consistently, and that no amount of prompt
work fixed — five rounds of rewriting plus temperature 0 still produced the same
violations:

  no reference  -> the marker sentence must not exist, and a swap is a plain
                   Replace. The model emits Add + marker + Remove instead, which
                   is the reference form applied where it does not belong.
  reference     -> the marker may only follow a sentence starting with Add. When
                   it lands on a Replace, that pair appears in zero training
                   samples.

Both are decidable in code: the caller already knows whether it attached a
reference image, so the node takes that boolean rather than asking the model to
infer it. Add+Remove is recomposed into Replace, and Replace+marker into
Add+Remove, preserving the object description the VLM did get right.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 09:51:52 +00:00
Claude 8ba0ea9620 Add nested-safe H3 latent saver (1.27.0)
The stock SaveLatent calls .contiguous() straight on samples["samples"], which
fails on H3 because the latent is a NestedTensor holding the video and audio
components. This unbinds them first and writes each as its own tensor, so an H3
latent can be taken out of the graph and decoded or inspected offline.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 02:04:42 +00:00
Claude 5331c16480 Load the VAE to GPU before direct decode (1.26.2)
The node read the device off the decoder parameters, but ComfyUI keeps the VAE on
CPU until something loads it — vae.decode() calls load_models_gpu first, and the
direct path skipped that. The result was a full ViT decode on CPU: minutes of
work, fans up, no output.

Now it calls comfy.model_management.load_models_gpu([vae.patcher]) and uses
vae.device / vae.vae_dtype like the native path, returning to the intermediate
device. Measured on a 768x1280 decode: 2.4s on cuda:0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 01:57:17 +00:00
Claude 16961e3905 Handle H3 nested latents in direct decode (1.26.1)
The H3 latent arrives as a NestedTensor holding the video and audio components,
so `z.dim()` raised AttributeError as soon as the node was fed a real generated
latent. The native nodes unbind and keep the first component (nodes.py:335); this
does the same, and re-adds the batch dimension that unbind drops.

Verified both paths produce identical output (MSE 0.00e+00) on the same latent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 01:37:45 +00:00
Claude eff6017e43 Add MiniMax-H3 direct single-frame decode node (1.26.0)
Decoding a DiT-generated H3 latent through the stock VAEDecode produces a visible
grid. The reference path does not use vae.decode() at all: both the checkpoint
card and the multimodalart/MiniMax-H3-images Space call the decoder directly.
This node is that recipe, with the four details its author flags as silently
corrupting if changed:

  1. vae.decoder(vae.post_quant_conv(z)) directly, never vae.decode(), whose
     temporal chunker cannot form a chunk from the few latent slices an image
     generation produces;
  2. decoded[:, :, -1] -- the last output slice, not index 0;
  3. latent denormalisation, z * latents_std + latents_mean;
  4. ImageNet pixel denormalisation, without which the frame comes out dark and
     over-contrasted like a tone-mapping bug.

Reconstruction check against a real image: 22.43 dB, block ratio 1.30 at the
32px patch boundary -- matching the raw decoder path measured separately, which
is what confirms the node reproduces the reference rather than approximating it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 01:30:07 +00:00
Claude b193b6b0b1 Add Sharpest Frame / Laplacian node (1.25.0)
Ports the sharpest-frame selection from scanpe_experiments/h3_image.py, which
takes it from IF-Edit (arXiv:2511.19435): decoded frames carry different amounts
of residual motion blur, so score by Laplacian variance and keep the crispest
instead of a fixed index.

Pairs with running MiniMax-H3 as an image generator: the VAE's minimum clip is 5
pixel frames (2 latents), decode yields frames of uneven sharpness, and the best
one is not always at the same index.

The node documents a limitation found while testing it, because it is easy to
misuse: Laplacian variance is a BLUR heuristic, not a quality metric -- artifacts
are high-frequency too. On a real H3 decode it picked the frame with the WORST
PSNR of its block (19.72 dB against 23.19 for its neighbour) because that frame
was the most artifacted. It is sound when the candidates are the same content
differing only in blur, which is the IF-Edit setting; it is meaningless for
choosing among frames from different moments.

Also relevant and not obvious: the "last frame of each decoded block is the valid
one" rule (decoded[:, :, -1] in the upstream example) held for the first block
here (34.49 dB against 26.80 for its neighbour) but inverted on the second.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 13:04:16 +00:00
Claude 0795e3fda2 Add MiniMax-H3 single-frame VAE loaders with tile control (1.24.0)
The published single-frame decoder (iamkaikai/MiniMax-H3-Single-Frame-VAE-500K)
cannot be loaded by the stock VAELoader: detection in comfy/sd.py requires an
encoder key, and the file is decoder-only in diffusers naming, so it falls back
to Stable Diffusion geometry (4 latent channels, 8x upscale) and decodes blocks.

Two nodes:
- MiniMaxH3SingleFrameVAELoader: combines the official H3 VAE with the author's
  decoder-only checkpoint, translating the naming in memory. Validated against
  the author's own diffusers path (load_decoder.py) on the same latent: 72.92 dB,
  i.e. fp16 noise.
- MiniMaxH3VAELoaderTiled: loads an already complete H3 VAE and exposes tile_size,
  which the stock loader does not.

Two key conversions are not derivable from names, and getting either wrong
produces wrong output with no load error. Both were settled by correlating
against the official VAE:
- to_qkv is interleaved per head (heads, 3, dim_head), not stacked [q;k;v]
  (see qkv.view(B, S, -1, 3*dim_head) in comfy/ldm/minimax/vae.py); corr 0.998.
- ff.w1 halves are swapped relative to diffusers, since ComfyUI reads gate first;
  correlation: straight -0.013, swapped +2.00.

tile_size defaults to 512 rather than the constructor's 256. Measured on a
1056x640 image: 512 gives 26.00 dB with a seam ratio of 0.91 (indistinguishable),
against 22.17/1.49 at 256 and 21.35/2.42 at 1024. The author trained 475k of
500k images at <=512px, so 512-wide tiles keep each piece inside the regime that
has training mass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 12:14:33 +00:00
alisson-anjos c285ead027 Let the slot tag scale with the adapter (1.23.1)
The learned slot tag was applied at fixed magnitude regardless of the LoRA
strength set on the loader. Tag and adapter are trained together at strength
1.0, so raising the LoRA alone leaves them out of proportion — the adapter
shouts while the signal that tells the references apart stays where it was.
This matters in practice: an undertrained adapter often needs 1.2-1.5 to take
effect at all, and at that point the tag is quietly under-weighted.

slot_strength exposes the same knob for the tag, defaulting to 1.0 so existing
graphs are unchanged.
2026-08-18 05:19:40 +00:00
alisson-anjos 3cec31ed42 Add multi-subject reference nodes with learned slot tags (1.23.0)
Three nodes for checkpoints trained with several references on distinct slots:

- LTX Reference Images collects up to six references into one ordered list. A
  plain IMAGE batch cannot carry references of different sizes, and subject
  references routinely differ — a portrait crop next to a landscape product
  shot — so the list keeps each at its own size and puts the slot order in one
  visible place.
- LTX Multi Reference Slots attaches them as separate reference blocks, each
  with its own source_id and, when the LoRA carries one, its trained slot
  embedding. The embedding is read straight from the LoRA file: ComfyUI's LoRA
  loader applies only LoRA weights and drops everything else, so without this
  the references are untagged and the checkpoint is sampled without a signal it
  was trained to use. Pointing it at a LoRA that has no slot embedding fails
  with an explanation rather than silently ignoring the setting.
- LTX Reference Tags builds the "<Image 1> is the woman." prompt prefix.
  Nothing in the model ties a tag to a slot — that correspondence exists only
  because the training captions used it consistently, so sampling with a
  different convention simply does not bind.

The identity node gains one hook: a spec may now carry a slot_vector, applied
to the raw patchified tokens before patchify_proj, which is exactly where the
trainer adds it. Specs without the key are untouched, so existing workflows
behave identically.
2026-08-18 04:57:42 +00:00
alisson-anjos 032014465d Bump BFS Nodes version to 1.22.1 2026-08-14 10:50:37 +00:00
alisson-anjos 3d4796684b Use standard zero reference offset by default 2026-08-14 10:48:37 +00:00
alisson-anjos 22d52d8e70 Add reference-only temporal offset controls 2026-08-14 10:41:06 +00:00
alisson-anjos 3fb1155e11 Add automatic LTX 2.5 identity transfer support 2026-08-11 21:00:08 +00:00
alisson-anjosandClaude Opus 5 f03f012fce Add LTX Multishot Prompt + Refs node for the shotplan strata LoRA
Builds a ShotPlan-format multishot prompt (global preamble + "Shot N: <modifier>
<size> <move> <action>" blocks) plus the ordered, gray-padded reference batch that
the multishot strata LoRA expects.

No new conditioning node is needed: LTX Identity Transfer already assigns
seg_value = (source_id + i) * phase_scale and strata_slot = i over the image batch,
so source_id=2 with layout="strata" over 4 refs reproduces the trained convention
(source_id 2/3/4/5 on slots 0/1/2/3). Batch order is shot order -- the LoRA was
trained with shuffle_reference_slots=false. Unused slots get a flat RGB-128 frame,
matching the dataset's blank.jpg placeholder.

The dropdown vocabularies are extracted from the 12,470 "Shot N:" headers of
Pensioner/shotplan train_meta_16fps.json (the span between "Shot N:" and the start
of the action sentence), covering 98% of size terms and 97% of move terms.
Modifiers (over-the-shoulder, low/high angle, POV) are a separate axis that
combines with a size rather than replacing it, as in the source captions.

Bump version to 1.21.1.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 10:03:55 +00:00
alisson-anjosandClaude Opus 5 f3bda037fc Release the model after a run: no more LTXAV leak warning and per-run slowdown
LTX Identity Overlap installed its four patches with types.MethodType(fn, ltxv) and captured the
originals as bound methods. Both put a strong reference to the model inside the model's own
__dict__, so every patched model sat in a reference cycle that only a full collection could break.
ComfyUI holds a weakref to detect exactly this: it reported "memory leak with model LTXAV", forced
a full gc, and repeated executions of the same workflow got progressively slower (#9).

The originals now come from the class (plain functions, called with self explicitly) and the
patches are installed as weakref-resolving callables with the same signature. Behaviour is
unchanged; the model is released by reference counting alone.

Verified against a stand-in LTXAV with cycle collection DISABLED, so only refcounting can free the
object: before this change the model was still alive after `del` (cycle present), after it the
weakref is dead, with the patched methods returning the same values in both cases.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-30 07:52:43 +00:00
alisson-anjos 169ed7eca5 Revert reference_face backward-compat alias, keep single reference_image
Two inputs doing the same thing is more confusing than a one-time
reconnect for anyone still on a pre-v1.14.0 workflow -- back to a single
required reference_image input. The actual RoPE crash fix (fused
rotation-matrix pe format) from the previous commit is unaffected and
stays in place.
2026-07-26 11:46:34 +00:00
alisson-anjos c79ae0e843 Fix RoPE phase rotation for ComfyUI's new fused rotation-matrix pe format
Recent ComfyUI (comfy/ldm/lightricks/model.py freqs_cis_matrix) changed
_prepare_positional_embeddings to return a single fused 2x2 rotation
matrix per (batch, token, head, freq) plus a split_mode bool, instead of
separate cos/sin tensors. _rotate_ref_block still assumed the old
(cos, sin, [flag]) tuple, so `sin` silently became the split_mode bool
and crashed with "'bool' object has no attribute 'dtype'" on the very
first sampling step after updating ComfyUI (reported by an external
user; reproduced with the same traceback on 0.28.0).

_rotate_ref_block now detects which format it received and, for the new
fused format, composes the phase rotation via matmul instead of the old
cos*a-sin*b trick -- 2D rotation matrices commute, so R(phase) @ R(existing)
gives the same result regardless of multiplication order. Verified against
the real (not mocked) freqs_cis_matrix from the currently installed
ComfyUI: output stays a valid rotation matrix (determinant 1), the
untouched token range is unaffected, and the legacy cos/sin path still
works unchanged for anyone on an older ComfyUI.
2026-07-26 11:26:43 +00:00
alisson-anjos f37fe43524 Fix LTXIdentityOverlapConditioning crash on pre-rename workflows
v1.14.0 renamed reference_face -> reference_image and dropped the
identity_projector/id_strength/arcface_mode ArcFace path, but old saved
workflows still send those exact input/kwarg names -- ComfyUI passes
whatever a saved prompt's "inputs" dict contains straight through as
kwargs, so loading one crashed with "unexpected keyword argument
'reference_face'" instead of just using the renamed input.

reference_image is now optional with reference_face kept as a deprecated
alias (resolved to reference_image when the new name isn't connected),
and **_legacy_kwargs absorbs the removed ArcFace-projector params so old
workflows load and run instead of hard-crashing.
2026-07-26 11:14:10 +00:00
alisson-anjos 73144a963a Update LTX Multiple Controls defaults for scail2v2_r128_v13 recipe
- guide/mask default to source_id=0 (no phase, matches v13's source_phase=false)
- identity default source_id=2 (matches v13's ref source_phase=true)
- mask_video is now expected as plain white-on-black binary (colored masks
  leaked hue into generated clothing during training, confirmed via v9-v12)
- add auto_mask_guide (on by default): paints guide_video black wherever
  mask_video is white before encoding, matching how v13 was trained -- this
  closes the reconstruction shortcut that let the model skip identity_image
  entirely and just copy guide's own person into the replacement region
2026-07-26 05:40:31 +00:00
alisson-anjos 4917711182 Add downscale_factor support to LTX Multiple Controls
Each slot (guide/mask/identity/identity_mask) now has its own
downscale_factor input: encodes that reference at target_size/N
instead of full resolution (cheaper VRAM for conditions trained at
reduced res, e.g. earlier scail2v2 recipes that downscaled guide/mask).

Ported the missing piece from ltx_core's VideoConditionByReferenceLatent
(positions[:,1:3,...] *= downscale_factor) into the shared process_input
patch in ltx_identity_overlap.py -- without it, an encoded-at-half-res
reference's raw pixel coordinates only spanned half the target's actual
extent instead of being re-stretched to overlap it, silently breaking
the position alignment the training recipe assumed.

Bump version to 1.19.0.
2026-07-24 16:20:42 +00:00
alisson-anjos f5e0b18f77 Add identity_mask slot to LTX Multiple Controls + LTX Color Mask node
LTX Multiple Controls now supports a 4th slot (identity_mask_image) for
scail2v2-style color-pointer masks, defaulting to inherit the identity
slot's own source_id/phase_scale (matches training convention). Also
updated guide/mask/identity default source_ids to match the scail2v2
3-phase recipe (1/2/3 instead of 0/0/2).

New LTX Color Mask node recolors plain white/binary SAM2-style masks
into the trained convention: 'silhouette' style (recolor, keep shape)
for the guide mask slot, or 'flat_marker' style (recolor + collapse to
a small dot at centroid, no shape) for the identity mask slot -- a
shaped identity mask re-injects a competing pose signal, the same bug
flatten_ref_masks.py fixes on the training-data side.

Bump version to 1.18.0.
2026-07-24 15:31:44 +00:00
alisson-anjos 4218591b47 Bump version to 1.17.0
New: LTX Multiple Controls node.
2026-07-22 07:10:53 +00:00
alisson-anjos e0c4c94661 Add LTX Multiple Controls node: guide+mask+identity in one call
Chaining multiple LTXIdentityOverlapConditioning nodes silently drops all
but the last one's reference -- each apply() overwrites transformer_options
["_id_ref_specs"] instead of merging with what the upstream node set. This
node builds every active slot's ref_specs in a single call so guide, mask,
and identity references all combine correctly. Each slot (all optional) has
its own source_id/phase_scale/layout/ref_resize_mode, defaulted (not
hardcoded) to match the trained recipe.
2026-07-22 07:08:41 +00:00
alisson-anjosandClaude Fable 5 7112d6e640 Add Entity Sheet 2x2 node (HuMoSet v7 semantic grid convention)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 06:40:26 +00:00
alisson-anjos c23e670237 fix: strata slot collision -- single-image refs span 1s, not a point
STRATA_SLOT_WIDTH=0.5 let adjacent numbered slots overlap: a reference
image is encoded at fps=1.0 (process_videos.py convention), so a single
1-latent-frame reference spans a full 1.0-second range on the T axis
after get_pixel_coords, not a point. Confirmed numerically: slot 0 at
width=0.5 occupied [target+0.5, target+1.5] and slot 1 occupied
[target+1.0, target+2.0] -- they collided in [target+1.0, target+1.5].
Bumped to 1.5s, which safely clears the 1.0s content span with margin.

Every checkpoint trained before this fix (phantom_stacked_r128/_p2)
learned under the collided geometry, so continuing training from one
now means adapting to the corrected spacing, not a free continuation.
2026-07-18 18:36:32 +00:00
alisson-anjos c14443b3a9 feat!: rename reference_face -> reference_image, drop ArcFace projector
BREAKING: reference_face renamed to reference_image (user confirmed:
not face-specific, references are arbitrary subjects -- toys, animals,
objects; accepted the input-name break since it's cosmetic and doesn't
silently change generation behavior like the earlier v1.13.0 mistake
did). Existing workflows will show this input disconnected on load and
need reconnecting once.

Also drops the identity_projector/id_strength/arcface_mode ArcFace path
entirely per explicit confirmation it isn't used -- kept the shared
_arcface_embed/_load_projector/_append_ctx_tokens/_find_ltxv helpers
intact since ltx_identity_multiangle.py still imports them.
2026-07-17 17:59:02 +00:00
alisson-anjos 4e3bb490a6 fix: restore backward compat, do stacked refs via batch not new inputs
v1.13.0 broke existing workflows: renamed the required reference_face
input to reference_image, and deleted the identity_projector/id_strength
/arcface_mode ArcFace inputs entirely. This repo has real users on the
current node -- reverting both: reference_face keeps its name, ArcFace
is back exactly as it was (still only used for the primary/1st reference).

Multiple stacked references (layout='strata') are now driven by feeding
a BATCH of images into the existing reference_face input (e.g. via an
Image Batch node) instead of a second fixed reference_face_2/source_id_2
pair of inputs -- a single image (the old, default case) behaves exactly
as before; N images each become their own reference block, source_id
auto-incrementing per index (1st keeps source_id as-is, 2nd gets +1, ...).
ref_preview/crop_overlay outputs now return one image per batch entry
instead of only the first, so all stacked refs are visible.

Also fixes a real bug in the reverted 1.13.0 approach the user hit while
testing (second reference silently not applying) -- the model patch now
tracks a list of (offset, length, source_phase) blocks end to end
(process_input concat + prepare_pe rotation), the same proven pattern
already used by ltx_identity_multiangle.py, instead of a single ref
latent/seg_value pair that only the last-installed spec could occupy.
2026-07-17 17:54:08 +00:00
alisson-anjos 299316f415 chore: bump version to 1.13.0 2026-07-17 17:40:39 +00:00
alisson-anjos c76e46ee6d feat: strata layout + stacked dual-reference, drop ArcFace projector
Reference-image conditioning is no longer face-specific: renamed
reference_face(_2) -> reference_image(_2) and removed the ArcFace
identity_projector/id_strength/arcface_mode inputs and code path (kept
the shared _arcface_embed/_append_ctx_tokens/_load_projector helpers --
still used by ltx_identity_multiangle.py).

Added 'strata' to the layout dropdown and an optional reference_image_2
input (source_id_2, default 3) so this node can drive checkpoints
trained on ltx_trainer's TASS strata layout: multiple SEPARATE
reference images sharing the overlap H/W range but placed in their own
numbered temporal slot (target_max_t + slot*STRATA_SLOT_WIDTH), instead
of the old single-reference-only overlap/st_drc path.

Generalizes the model patch from one ref block to a list of (offset,
length, source_phase) blocks -- process_input now shifts/concats N refs
in order, prepare_pe rotates each block's own RoPE phase independently
(pattern lifted from ltx_identity_multiangle.py's proven multi-block
timestep/grid_mask handling), and a new _forward capture point stashes
frame_rate so process_input can convert STRATA_SLOT_WIDTH (seconds, to
match the trainer) into this step's raw pixel/frame units.
2026-07-17 17:40:39 +00:00
alisson-anjos 61ce4ea3b3 chore: bump version to 1.12.1 2026-07-17 17:17:56 +00:00
alisson-anjos 4a1db299da fix: reference-CFG function signature triggered legacy 3-arg cfg path
set_model_sampler_cfg_function() detects the old (cond, uncond, cond_scale)
calling convention by counting ALL signature parameters, not just
positional ones -- the previous 3-param version (args + two defaulted
closure vars) tripped that check and got called with three raw tensors
instead of the args dict, crashing with "too many indices for tensor of
dimension 3". Fixed by taking a single `args` param and closing over
noref_to/ref_scale normally.
2026-07-17 17:17:30 +00:00
alisson-anjos c65d8d35d6 chore: bump version to 1.12.0 2026-07-17 17:08:36 +00:00
alisson-anjos 9d6a482f18 feat: add reference-CFG (ST-DRC) to LTX Identity Transfer
Optional reference_guidance_scale input (default 1.0 = off). When >1,
adds a third forward pass per denoising step with the reference tokens
dropped, isolating the reference's own contribution the same way CFG
isolates the text prompt: denoised += (scale-1)*(with_ref - without_ref).
Implemented via model.set_model_sampler_cfg_function(), reusing the
positive conditioning already passed to KSampler for the extra pass.

Mirrors the same reference-CFG mechanism added to ltx-trainer's
validation_runner.py so training-time validation and this node's
inference now share the same guidance formula.
2026-07-17 17:08:35 +00:00
alisson-anjosandClaude Sonnet 5 944c3bdff5 chore: bump version to 1.11.0
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-17 09:59:19 +00:00
alisson-anjosandClaude Sonnet 5 a434f88e45 feat: add layout dropdown (overlap/st_drc) to LTX Identity Transfer node
New optional input `layout` (default "overlap", the only behavior every
checkpoint so far was trained with -- old workflows keep working
unchanged). "st_drc" shifts the whole reference coordinate block past
the target's own extent on every axis (non-overlapping RoPE region),
mirroring ltx_trainer's training_strategies/tass.py apply_tass_layout
st_drc branch -- kept in sync manually since this node can't import the
trainer package, adapted to ComfyUI's own [B,3,N] single-corner
coordinate shape instead of the trainer's [B,3,N,2] patch-bounds shape
(the min/max-per-axis shift math is equivalent either way, verified
against a synthetic test matching apply_tass_layout's own logic).

Only usable with a checkpoint specifically trained with layout='st_drc'
-- using it against an 'overlap'-trained checkpoint will not work,
the model never learned that coordinate convention.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-17 09:59:18 +00:00
alisson-anjosandClaude Sonnet 5 a10dd8bd27 chore: bump version to 1.10.14
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-17 01:16:09 +00:00
alisson-anjosandClaude Sonnet 5 d2a25c7504 feat: ref_preview/crop_overlay outputs + optional crop_anchor for match_target
New IMAGE outputs (appended at the end of RETURN_TYPES, existing wiring
unaffected):
- ref_preview: exactly what gets encoded as the reference, after
  whatever resize/crop/letterbox ref_resize_mode applied -- lets you
  sanity-check the actual input to the model.
- crop_overlay: the ORIGINAL reference with a green rectangle around the
  region that survives match_target's crop (everything outside it gets
  discarded). For match_target_letterbox/native_resolution, which never
  crop, the box just outlines the full frame.

New optional input crop_anchor (center/top/bottom/left/right, default
"center"): controls which part of the reference match_target's crop
keeps, instead of always the exact center -- e.g. set "top" if your
sheet's face closeup sits in the top row so it survives a crop against
a differently-shaped output. Added under "optional", not "required", so
existing saved workflows keep working with zero changes (falls back to
the previous always-center-crop behavior). Verified byte-for-byte
equivalent to the old center-crop path on a smooth test image when left
at the default; the non-default anchors reuse the same crop math with a
different offset.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-17 01:16:09 +00:00
alisson-anjosandClaude Sonnet 5 2dc36ac48b chore: bump version to 1.10.13
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-17 00:55:05 +00:00
alisson-anjosandClaude Sonnet 5 0c24f592bd feat: add match_target_letterbox ref_resize_mode (no-crop option)
match_target resizes the reference to the output video's pixel size via
a center-crop (comfy.utils.common_upscale crop="center") then resize --
for aspect-mismatched refs (e.g. a landscape character sheet used with a
portrait output) this silently discards whatever isn't in the middle of
the reference, which can crop out the face closeup panel entirely.

Add match_target_letterbox: same target pixel size, but fits the WHOLE
reference inside it preserving its own aspect ratio (no crop), padding
the leftover space with white. Trades some wasted canvas space for
never losing reference content -- recommended over match_target when
ref and output aspect ratios differ, and over native_resolution when
native_resolution's landscape-composition bias causes an off-center/
cropped-looking result on the OUTPUT video itself.

Reported by a LoRA user: portrait output + landscape character sheet +
native_resolution produced an off-center-left result (model composing
as landscape then getting cropped to the portrait canvas); switching to
match_target "fixed" centering but lost face detail from the crop.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-17 00:55:05 +00:00
alisson-anjosandClaude Sonnet 5 22db16fc2b chore: bump version to 1.10.12
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-16 20:17:48 +00:00
alisson-anjosandClaude Sonnet 5 1f37355676 fix: fit_mode='fit' still underfilled the canvas on one axis
fit_mode='fit' only guaranteed the row-justified block filled canvas
width, never upscaling to also fill height when the natural block was
shorter than the canvas (e.g. 2 refs left a top/bottom bar). Add
fit_mode='cover': same row-justified layout, but when the natural block
underfills the canvas it scales up to fill height and center-crops the
resulting width overflow instead, and when it overfills height it crops
that overflow directly -- either way every pixel of the canvas is
covered, cropping only the minimum shared/symmetric amount.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-16 20:17:48 +00:00
alisson-anjos 3b7137a8a8 chore: bump version to 1.10.11 2026-07-14 11:35:11 +00:00
alisson-anjos d828823075 fix: fit_mode='fit' was letterboxing each panel independently (wasted space)
Replace per-panel independent letterboxing with a row-justified layout:
every row fills the canvas width exactly (each image scaled uniformly,
no distortion), rows are only shrunk together (never upscaled past
natural fill-width size) if they'd collectively overflow the canvas
height. Net effect: at most one shared pair of side/top-bottom bars
instead of padding scattered inside every individual panel.

Ported the identical algorithm into build_multiref_sheet.py (the
training-side script) so inference-time sheets match training-time
sheets pixel-for-pixel when fit_mode='fit' is used -- verified with a
byte-for-byte comparison test. fit_mode='crop' (default) is unchanged.
2026-07-14 11:35:00 +00:00
alisson-anjos 467cb2e8d3 chore: bump version to 1.10.10 2026-07-14 11:08:02 +00:00
alisson-anjos ca5b4767ba feat: Multi-Ref Sheet Builder — add fit_mode (crop/fit) toggle
crop (default, unchanged): zoom+center-crop to fill each panel, matches
the training data. fit: scale each reference to fit entirely inside its
panel with a single uniform scale factor (no distortion), letterboxed
with background padding for any leftover space -- keeps every pixel
of references with extreme aspect ratios instead of cropping them.
2026-07-14 11:08:02 +00:00
alisson-anjos cc110af228 chore: bump version to 1.10.9 2026-07-14 11:00:04 +00:00
alisson-anjos 2ab178f20c feat: Multi-Ref Sheet Builder — combine 1-5 ref images into the fixed 1536x1024 composite sheet
Same grid convention as the multi-ref LTX-2 LoRA training pipeline
(build_multiref_sheet.py): deterministic layout by ref count (1x1, 2x1,
3x1, 2x2, 3-top+2-bottom), 512x512 panels, centered onto a fixed
1536x1024 canvas regardless of how many refs are plugged in. Panel
order matches the image0/image1/... indexing used in training.
2026-07-14 10:59:34 +00:00
NRDX 2627e1bf7d Update pyproject.toml
Compatibility with IC LORA Guide node.
2026-07-10 13:41:11 +01:00
NRDX bbbf61b972 Merge pull request #7 from Bambushu/fix-icllora-guide-timestep
Fix: prepare_timestep mis-trims per-token timestep when IC-LoRA guides are active
2026-07-10 08:28:50 -03:00
Maikel SlompandClaude Fable 5 9fae36cffa fix: prepare_timestep mis-trims per-token timestep when IC-LoRA guides are active
With a guide node in the graph (LTXAddVideoICLoRAGuide etc.), the per-token
timestep legitimately spans the grid-mask domain (video + guide tokens) and is
indexed later via timestep[:, grid_mask]. The existing oversized heuristic
assumed oversize could only come from audio, trimmed the timestep to
video-only, and the subsequent grid_mask indexing crashed:

  IndexError: The shape of the mask [5616] at index 0 does not match the
  shape of the indexed tensor [2, 3726, 1] at index 1

Fix: when the incoming per-token timestep length equals grid_mask length,
skip the audio trim, append the reference zeros directly, and let the
existing grid_mask padding keep modulation and tokens in lockstep. The
audio-trim path is unchanged for graphs without guides.

Tested: LTX-2.3 22B distilled, Face-ID LoRA + union-control IC-LoRA pose
guide + frozen audio latent, 97-281 frame renders.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 19:03:52 +02:00
alisson-anjos 10c7153e67 chore: bump version to 1.10.7 2026-07-09 08:00:08 +00:00
alisson-anjos 35f67e3d7f fix: LTX Identity Transfer forced ref to output video resolution
ref_px was always resized via common_upscale(..., lat_w*w_sf, lat_h*h_sf)
i.e. the TARGET video's pixel size, regardless of what resolution the
reference was actually trained at. Fine for old single-face-crop recipes
(ref bucket == video bucket), but breaks checkpoints trained with a fixed,
independent ref resolution (e.g. the face+3views composite ref, always
2048x1024 no matter the output video size): the ref latent ends up with a
different token count than training, destroying fine facial detail.

Adds ref_resize_mode: "match_target" (old behavior, default) vs
"native_resolution" (encode the ref at its own size, rounded to 32px).
2026-07-09 08:00:04 +00:00
alisson-anjosandClaude Sonnet 4.6 8c3794a9ef chore: bump version to 1.10.6
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-07 20:42:17 +00:00
alisson-anjosandClaude Sonnet 4.6 912b907b33 fix: process_output double-trim when overlap+multiangle patches are chained
Same kwarg collision as prepare_timestep: overlap's process_output was
reading ref_len from kw["_id_ref_len"] (set by multiangle's process_input
to 768) and trimming 768 tokens, then multiangle's process_output trimmed
another 768 — leaving 6912-1536=5376 tokens and causing the einops
unpatchify crash (5376 != 6144).

Fix: use getattr(self, "_id_ref_len", 0) — instance attr only, which is
correctly set to 0 by overlap's own process_input when no single-ref is
active.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-07 20:42:07 +00:00
alisson-anjosandClaude Sonnet 4.6 d1ed08d0b9 chore: bump version to 1.10.5
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-07 19:44:09 +00:00
alisson-anjosandClaude Sonnet 4.6 79d96b5fb7 fix: timestep shape crash when overlap+multiangle patches chain after multiple runs
Two root causes fixed:

1. Kwarg collision (primary): overlap's prepare_timestep was reading
   ref_len from kw["_id_ref_len"] which multi-angle's process_input sets
   to 768 via additional_args. When overlap is outer (installed after
   multi-angle), it incorrectly extended the timestep even with no active
   single-ref, causing size mismatches on subsequent multi-angle runs.
   Fix: use getattr(self, "_id_ref_len", 0) — instance attr only, which
   is correctly set to 0 by overlap's own process_input when no ref found.

2. Combined audio+video timestep (secondary): with audio connected,
   LTXAV.process_timestep can produce a per-token v_timestep that covers
   both video and audio tokens (e.g. 24576 = 6144 video + 18432 audio).
   Both patches now trim to the first tgt_len tokens before extending,
   keeping modulation in lockstep with vx (video-only).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-07 19:18:30 +00:00
alisson-anjosandClaude Fable 5 880555c0c7 fix: compatibility with guide nodes (IC-LoRA Guide / Director Guide) — extend grid_mask for ref tokens
Guide nodes inject a grid_mask that filters x tokens in _process_input and indexes the
timestep in _prepare_timestep (timestep[:, grid_mask]). Our identity ref tokens are
appended AFTER that filter, so the mask (sized without them) crashed with
'shape of the mask [N] does not match the indexed tensor [B, N+ref, 1]'. Both nodes now
extend grid_mask with True for the appended ref tokens, keeping the AdaLN modulation in
lockstep with the token sequence. Reported by vlady (LTX Identity Transfer + Add Video
IC-LoRA Guide Advanced / LTX Director Guide).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 18:13:51 +00:00
alisson-anjosandClaude Fable 5 8c9614446c fix(multiangle): order/shape-robust timestep extension + nested transformer_options
The AdaLN modulation crashed on other ComfyUI builds (tensor 6912 vs 7680): timestep and
ref-token extension went out of lockstep because kwarg flattening / timestep granularity
differ across versions. Now: (1) measure-based timestep padding — detect per-token,
per-FRAME or already-extended from the actual length and pad by exactly what
process_input appended this forward; (2) read ref latents from flat kwargs OR nested
transformer_options; (3) unique _idma_* state reset every forward (no stale leakage, no
collision with the single-ref node); (4) _dbg tracing in every hook (debug_log=true).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 17:25:56 +00:00
alisson-anjosandClaude Fable 5 74590b3169 feat: LTX Identity Transfer (Multiple Angles) — multi-view refs with per-view source_id
Copy of LTX Identity Transfer generalized to N reference blocks: reference_face_front
(required, sid 2, drives ArcFace projector) + optional reference_back_head (sid 3),
reference_body_front (sid 4), reference_side_profile (sid 5). Empty optional inputs are
omitted, matching the multi-view training's view-dropout. Distinct patch flags so it
never collides with the single-ref node.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 16:55:32 +00:00
alisson-anjosandClaude Opus 4.8 2987893262 1.10.4: remove CAN/AdaLN from the nodes — it degrades the video
Empirically the AdaLN (CAN) modulation destroys the generation; the identity gain came from the
overlap + projector + LoRA, not the CAN. Reverted the Identity Transfer node to overlap + projector,
unregistered the standalone CAN node (file kept).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 00:31:01 +00:00
alisson-anjosandClaude Opus 4.8 40aa735e00 1.10.3: fix CAN device mismatch (move CAN + ArcFace to the model device before delta)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 00:18:02 +00:00
alisson-anjosandClaude Opus 4.8 55206e9361 1.10.2: explicit can_face_mode dropdown (auto_adjust/as_is) for the CAN reference
The CAN needs its own visible face-detection control; before it was implicit. Now a can_face_mode
dropdown, independent of the projector's arcface_mode.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 23:48:49 +00:00
alisson-anjosandClaude Opus 4.8 5e4ece1470 1.10.1: CAN always detects face (auto_adjust), ignoring projector's 'disable' mode
arcface_mode=disable was meant to turn OFF the weak projector, but it also skipped the CAN — and
the CAN REQUIRES the reference ArcFace (that's the identity it injects into the AdaLN). Now the CAN
forces auto_adjust when the projector mode is 'disable', so it still gets the face.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 23:22:59 +00:00
alisson-anjosandClaude Opus 4.8 78b0cf8be6 Bump to 1.10.0 (LTX Identity CAN / AdaLN node + integration into Identity Transfer)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 23:14:55 +00:00
alisson-anjosandClaude Opus 4.8 d3b6c071b6 CAN node: integrate into Identity Transfer node + use add_object_patch (no forward patch)
- The block-forward patch failed on LTXAV (v_context/ax kwargs). Instead add the CAN delta directly
  into scale_shift_table (row 0 shift_msa, row 2 gate_msa) via ComfyUI's reversible add_object_patch
  — mathematically identical ((sst+delta)+timestep), works on any LTX/LTXAV version, no forward
  rewrite. Confirmed AV row order: slice(0,2)=shift/scale_msa, slice(2,3)=gate_msa.
- Integrated CAN into LTX Identity Transfer (overlap) node: new can_weights + can_strength inputs,
  applied to the same model+reference (so one node does overlap + CAN). Standalone CAN node kept too.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 23:13:58 +00:00
alisson-anjosandClaude Opus 4.8 5ce4840242 Add LTX Identity CAN node: apply trained AdaLN identity modulation at inference
The ArcFace projector (appended text tokens) has ~0 impact (cross-attn can ignore them). CAN
injects identity into the self-attn AdaLN (shift+gate) of even blocks — a channel the model can't
ignore. Trained on top of the reference LoRA it lifts identity a lot (humo 0.52 -> 0.69) with no
first-frame leak. Node rebuilds CANModulation per even block from identity_adapters_*.safetensors
(can.<i>.*), extracts the reference ArcFace, and patches each even block's native forward with
shift_msa += dshift; gate_msa += tanh(dgate) — the exact training formula. Lazy imports; defensive
__init__. Weight-load + CAN forward validated headless; block patch untested in ComfyUI.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 22:31:46 +00:00
alisson-anjosandClaude Opus 4.8 6e7193c3ac Gemma-Vision node: use the CLIP's native multimodal tokenize (no gemma_path/AutoProcessor)
ComfyUI's native LTX Gemma-3 tokenizer already accepts image= (inserts <image_soft_token> +
carries pixels) and encode_from_tokens runs the multimodal Gemma + projection + connector. So the
node is just clip.tokenize(prompt, image=ref) -> encode_from_tokens_scheduled -> CONDITIONING.
Everything from the already-loaded CLIP; no separate Gemma path, no AutoProcessor, no _get_encoder.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 13:00:20 +00:00
alisson-anjosandClaude Opus 4.8 0e71b5d7b6 Bump to 1.9.0 + add transformers dep (Gemma-Vision node uses AutoProcessor)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 12:03:10 +00:00
alisson-anjosandClaude Opus 4.8 521c28a970 Add LTX Identity Gemma-Vision node (Caminho2): native-space identity conditioning
Feeds the reference image through the ALREADY-LOADED ComfyUI CLIP (LTX's own multimodal Gemma 3
vision) -> feature_extractor -> connector -> CONDITIONING. Identity is in Gemma's native space
(the reason foreign CLIP/DINO/ArcFace adapters fail on LTX); no reference latent (no copy-paste/
mask), no DiT surgery. Reuses the loaded CLIP (only loads a lightweight AutoProcessor for the
multimodal chat template). Mirrors ComfyUI-LTXVideo LTXVGemmaTextEncoderModel, run multimodally.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 11:59:23 +00:00
alisson-anjosandClaude Opus 4.8 96df648a6c LTX Identity Transfer: load projectors of any depth (4 or 16 tokens)
_load_projector now rebuilds the projector MLP from the state_dict (proj.N Linears),
so it accepts both the original 4-token/2-layer projector and the enhanced
16-token/3-layer one. Bump 1.8.0 -> 1.8.1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 14:54:59 +00:00
alisson-anjosandClaude Opus 4.8 ef64f26dfd Add LTX Identity Transfer node (IPT2V: overlap ref + source-phase RoPE)
- av_model model-patch: ref tokens at overlap positions + source_phase RoPE, sliced from output (not I2V)
- optional ArcFace projector (dropdown from loras, 'None' = overlap only), debug_log toggle, arcface auto-adjust
- bump 1.7.0 -> 1.8.0; deps: opencv-python, insightface==0.7.3, onnxruntime

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 17:08:14 +00:00
alisson-anjosandClaude Opus 4.8 13e663101e Add AMV Guide Builder node + bump to 1.7.0
BFSAmvGuideBuilder builds the keyframes+blank guide for the AMV av2av LoRA:
auto_scene / manual_indices / external_images, with audio-beat placement
(librosa beat_track, RMS-onset fallback) to sync cuts to the music. Adds
librosa to requirements.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 07:25:29 +00:00
alisson-anjosandClaude Opus 4.8 0273d3ad24 Bump version to 1.6.0 (Anime2Real Bernini node)
Triggers the Comfy registry publish workflow (paths: pyproject.toml).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-28 11:50:38 +00:00
65 changed files with 20194 additions and 14 deletions
+144
View File
@@ -0,0 +1,144 @@
# MiniMax-H3 downscaled latent guides
BFSNodes 1.46.0 adds two nodes for H3 upscale LoRAs trained in
[alisson-anjos/ai-toolkit](https://github.com/alisson-anjos/ai-toolkit/tree/minimax-h3-latent-guides)
with `target_crop_v2` / `target_grid_stride_v1` geometry:
- **MiniMax-H3 Downscaled Latent Guide (BFS)**: preprocess and encode an image
or video guide at a lower resolution, and return a model with compatible
spatial conditioning coordinates.
- **MiniMax-H3 Guide Target — Image / Video (BFS)**: make the joint target
latent at the intended output dimensions, with one frame for an image or
`17*n+5` frames for video. It does not resize the target canvas implicitly.
The native Add Guide node enlarges guides to the output canvas. This node
keeps the smaller encoded guide and selects every factor-th origin from the
**target patch grid**. Guide time stays aligned to the target. It modifies a
cloned model through its UNet wrapper; no global PackedLayout replacement is
installed, and other models/conditionings continue to use their native paths.
## Install / update
```bash
cd ComfyUI/custom_nodes/ComfyUI-BFSNodes
git pull --ff-only
```
Restart ComfyUI. Use a ComfyUI build with native H3 support; the layout tests
use the exact implementation in commit
[`8cfe5e1`](https://github.com/Comfy-Org/ComfyUI/blob/8cfe5e1ecb97512dea8deaac15e1228d7e6feeb1/comfy_extras/nodes_minimax_h3.py#L166).
Incompatible H3 layout/VAE APIs produce explicit errors.
## Image workflow
1. Load the **H3 Ref2V** model and the trained upscale LoRA with your normal
LoRA loader. Load the official complete H3 video VAE with its encoder and
the H3 Qwen3-VL text encoder (`CLIPLoader` type `minimax`).
2. Encode the caption without passing the guide image into the VLM. Start
with the training prompt:
`h3upscale, increase resolution and restore fine details while preserving the original content and visual style.`
3. Create a **Guide Target** with width `1024`, height `768`, frames `1`.
4. Connect the model after LoRA loading, positive conditioning, target latent,
VAE, and source image to **Downscaled Latent Guide**. Set
`downscale_factor=4`, `frame_idx=0`. Select the same `lora_name` to validate
training metadata; that selector does **not** load the LoRA a second time.
5. Connect **both returned MODEL and positive CONDITIONING** to the guider /
sampler. Use the returned model for sigma scheduling too. The sampler's
`latent_image` stays the target latent from step 3, not a guide latent.
6. Decode with the H3 VAE. The third output is the **actual reduced pixel
guide preview**; at this canvas it is `256 x 192`.
An [API-format image workflow template](examples/minimax_h3_upscale_image_api.json)
is included. Replace `YOUR_H3_UPSCALE_LORA.safetensors`,
`YOUR_INPUT_IMAGE.png`, and model filenames with files installed in your
ComfyUI. The template uses seed 42, Euler, 28 steps, full denoise, and BasicGuider.
It can be submitted as the `prompt` field of a ComfyUI `/prompt` API request.
It is an API graph, not a saved frontend canvas workflow.
## Video workflow
Create a target with `frames=73` (approximately 3 seconds at 24 fps) and supply
an IMAGE batch of **exactly 73 source video frames** resampled to 24 fps, in
chronological order. The video VAE produces 22 latent-time slices. Other valid
lengths include 5, 22, 39, 56, 90, 107 and 124; a one-frame batch is an image.
The node rejects invalid lengths instead of silently truncating or stretching
motion. It does not perform video decoding or FPS conversion itself.
Keep the same target canvas / guide factor as training. Increasing target
resolution or frame count increases target activation memory even though guide
spatial tokens are reduced. Start with a known-fitting canvas on the target GPU.
To save video, connect decoded frames to your video encoder/combiner at 24 fps.
An optional `audio` soundtrack plus `audio_vae` can add aligned guide audio.
Supply audio starting at the guide clip's first frame. It is cropped to the
clip and available target duration, and is never sent to the text encoder.
Omit it for visual-only inference. Native reference conditioning and native
same-resolution guides can be chained alongside this node; their row positions
are retained.
## Geometry and encoding
- Each target pixel axis must be a multiple of `32 * downscale_factor`.
- The source is fitted to the target canvas with the fork's centered bicubic
crop, then downscaled with Lanczos before VAE encoding. This also applies
when the supplied source is already small: inspect `guide_preview`.
- H3 VAE spatial compression is 16. At output `1024 x 768`, factor 4 yields
guide pixels `256 x 192`, guide latents `16 x 12`, and target latents `64 x 48`.
- The posterior is sampled with an isolated CPU generator (seed 42) and
rounded through FP16 **before** latent normalization, matching the fork's
image and video sampling paths. The native VAE mean-only encoding is not used.
- An image has one guide latent-time slice; 73 video frames have 22. Video
time coordinates use H3's `(1,4,4,4,4)` temporal pattern, not a uniform stride.
- The corrected packed layout keeps target/reference/audio coordinates and
reindexes condition positions, frozen/update masks and segment boundaries.
Downscaled guides receive the target spatial grid at the training stride;
coordinates from a separately normalized small grid are not multiplied.
## Validation and limits
CPU tests cover the audited native layout, factors 1/2/4, exact crop/resize,
posterior sampling, mixed/native guides, references, audio row preservation,
73-frame counts, metadata checks, clone isolation and wrapper composition.
Golden position fixtures come from the actual AI Toolkit packing implementation.
Run `python -m unittest discover -s tests -v`.
A full pretrained ComfyUI GPU sampling run has not been validated as part of
this change. Matching preprocessing and coordinates establishes the conditioning
contract; it does not establish the trained LoRA's visual quality. Compare
LoRA strength 0 and 1 on held-out inputs before evaluating upscale fidelity.
## Experimental reference layouts and source phase (1.47.0)
Optional inputs on Downscaled Guide expose `guide_rope_layout` (overlap/sidecar),
`reference_rope_layout` (native/overlap/sidecar), `reference_source_phase`,
`reference_phase_scale`, `reference_sidecar_margin`, and `source_id`. Defaults
preserve 1.46.0 behavior. H3 sidecar margins use normalized RoPE units, not pixels.
Select your LoRA to check these values against `minimax_h3_reference_rope` metadata.
Legacy checkpoints imply overlap/native/phase-off. Changing only inference geometry
on a legacy LoRA does not establish that the model can use it.
**MiniMax-H3 Identity Reference + RoPE (BFS)** adds a separate single-image latent
reference without passing it to the VLM. Supply the same target latent and full H3
VAE. Chain both model and conditioning through Guide and Identity Reference, then
connect both final outputs to the guider/sampler (and model to the scheduler).
Use guide `source_id=1`, identity `source_id=2` for training control channels 1 and 2.
The identity reference keeps its aspect and is resized down only to the trainer's
/32 reference-image bucket; the guide factor does not resize the portrait.
The reference preview shows the actual pixels used.
This node requires a checkpoint trained with a matching identity reference role
and `control_latent_only: true` for caption-only conditioning. The LoRA selector
checks that mode too. It does not load LoRA weights. Adding a portrait to a LoRA
trained only with a low-resolution guide is not a validated identity-preservation
recipe. Keep caption/source conventions consistent during training.
Source phase adds the phase `source_id * phase_scale * 10000**(-d/48)` to H3's
rotary angles. Text and targets remain source zero. The scoped ModelPatcher object
patch composes this phase only when this node's marked conditioning is active;
unmarked conditioning and other models keep their original rotary path.
For a first experiment use overlap plus phase and compare to the phase-off baseline.
Sidecar and phase have CPU geometry/rotary parity tests, including mixed image,
video and audio, but no full pretrained H3 quality validation yet. They distinguish
sources; they do not guarantee the same face. See the
[training configuration guide](https://github.com/alisson-anjos/ai-toolkit/blob/minimax-h3-latent-guides/H3_REFERENCE_ROPE.md).
+14
View File
@@ -0,0 +1,14 @@
Portions of this pack are vendored from third-party GPL-3.0 projects:
bfs_crop_planner.py, bfs_subject_crop.py
from drozbay/MaskVidExperiments (https://github.com/drozbay/MaskVidExperiments),
commit d98cc89 (v0.2.0), Copyright (c) drozbay, GPL-3.0.
Vendored so the head-swap sampler needs no runtime dependency on that pack;
behaviour unchanged, ComfyUI node classes dropped.
tests/fixtures/comfy_h3_layout_8cfe5e1.py
CPU-only excerpt of comfy/ldm/minimax/model.py from Comfy-Org/ComfyUI,
commit 8cfe5e1ecb97512dea8deaac15e1228d7e6feeb1,
Copyright ComfyUI contributors, GPL-3.0. Retained without model weights
to verify downscaled guide row counts and coordinates against native H3.
+73
View File
@@ -46,6 +46,11 @@ cd ComfyUI-BFSNodes
pip install -r requirements.txt
```
Optional: `pip install insightface` with `onnxruntime` or `onnxruntime-gpu`, only for LTX Identity. The Shot Planner's
*Find people* / Cast no longer needs it (1.93.0): it uses OpenCV's YuNet + SFace (Apache-2.0, downloaded once), and uses
InsightFace only when it is installed. They are not in the requirements because insightface pins packages
that conflict with many environments; without them everything else works and those features say what to install.
Restart ComfyUI after installation.
## LoRA Placement
@@ -107,6 +112,58 @@ These nodes target ComfyUI's LTXV/LTX-Video model implementation. The main apply
A plain LoRA loader alone is not enough for Edit Anything LoRAs that include role embedding and/or AdaLN reference conditioning. Use the standard LoRA loader first, then pass the result through **LTXV Edit Anything (Apply)**.
## Reference temporal offset
Nodes that inject clean appearance references expose a reference-only temporal offset in VAE
latent steps. The default is `0`, matching the standard frame-zero placement used by existing
LoRAs and preserving their expected behavior. Negative values are an experimental way to reduce
the overlap frame-0 artifact where the reference composition briefly appears at the start of the
generated video.
- `0`: standard frame-zero placement (default)
- `-1`: one latent step before frame zero (`-8` pixel frames with the standard LTX VAE)
- `-2`: two latent steps before frame zero (stronger separation, possibly weaker identity)
The offset is applied only to appearance/reference inputs. Guide videos, masks, identity masks,
and target video positions remain unchanged. In **LTX Multiple Controls**, the field is named
`identity_temporal_offset_latents` and affects only `identity_image`.
## Shot Planner / Shot Loop (1.48.0)
Split a long video into shots a video model can follow (at camera cuts via PySceneDetect, fixed
length, or by hand on a timeline), set a reference image and prompt per shot, run the rest of the
workflow once per shot, and join the results back with the original timing and soundtrack.
- **BFS Shot Planner** — interactive timeline (thumbnails, cut detector curve, draggable
boundaries, per-shot reference gallery and prompt). Outputs the shots as a ComfyUI list, so
every node downstream runs once per shot.
- **BFS Shot Unpack** — guide frames, references, prompt and length of one shot, for any model.
- **BFS Shot H3 Conditioning** — MiniMax H3 conditioning for one shot with the native nodes
(aligned guide and/or native reference video).
- **BFS Shot Join** — concatenates, trims to the true lengths, cross-fades soft joins, returns audio.
**Cast (1.49.0)**: *Find people* tracks faces through the video (InsightFace) and groups them
into people. Link each person to a reference and every shot uses the reference of the person
with the most screen time in it; optionally split where the main person changes and run only the
shots where a linked person appears.
Two run modes: *auto loop* (all shots in one run) and *queue loop* (one shot per run, stored on
disk, re-queued automatically; nodes after the join only run on the last shot). See
[`web/js/docs/BFSShotPlanner.md`](web/js/docs/BFSShotPlanner.md).
Shots can also be cropped to a **SAM 3 mask** (text or points per shot) and pasted back by the join, continue
from the previous shot's result, and get **VLM suggestions** (what to segment, a shot description for the
prompt, run/skip) from a Qwen3-VL connected to the planner.
## H3 Duet / Side Panel (1.50.0)
Training-free split-screen generation for MiniMax H3 (TSC's latent pin): a panel (the source clip
or a reference) is pinned beside the video with a noise mask, the video is generated in sync with
it, and the panel is cropped off before decoding. **BFS H3 Duet** does it all in one node, **BFS Shot H3 Duet** renders each shot of the Shot Loop that way (or with an aligned guide for body-swap LoRAs);
**BFS H3 Side Panel** + **BFS H3 Side Panel Crop** are the building blocks. Optional aligned latent
guide. 1.84.0: `panel_static` (TV static over the face / person of the pinned clip, SAM 3) and a `setting`
picture (**BFS Setting Picture**) for same-world swaps. See [`web/js/docs/BFSH3Duet.md`](web/js/docs/BFSH3Duet.md).
## Requirements
- ComfyUI
@@ -118,3 +175,19 @@ A plain LoRA loader alone is not enough for Edit Anything LoRAs that include rol
## License
This project follows the repository license. See [LICENSE](LICENSE).
## MiniMax-H3 downscaled latent guides (1.46.0)
**MiniMax-H3 Downscaled Latent Guide (BFS)** supports aligned lower-resolution
image/video guides for H3 LoRAs trained with `target_grid_stride_v1`. It encodes
the smaller guide and patches only the returned model's packed coordinates.
Connect both its **MODEL** and **positive CONDITIONING** outputs to sampling.
**MiniMax-H3 Guide Target — Image / Video (BFS)** creates an exact output canvas:
use one frame for images or 73 frames for approximately three seconds at 24 fps.
At output 1024 x 768 and factor 4, the guide preview is 256 x 192.
See [setup, workflow template, compatibility and validation](MINIMAX_H3_GUIDES.md).
BFSNodes **1.47.0** adds optional MiniMax-H3 overlap/sidecar source-phase RoPE and
**MiniMax-H3 Identity Reference + RoPE (BFS)**. See [the H3 guide](MINIMAX_H3_GUIDES.md#experimental-reference-layouts-and-source-phase-1470).
+152
View File
@@ -6,17 +6,169 @@ from .headswap_node import NODE_CLASS_MAPPINGS as HEADSWAP_NODE_CLASS_MAPPINGS
from .headswap_node import NODE_DISPLAY_NAME_MAPPINGS as HEADSWAP_NODE_DISPLAY_NAME_MAPPINGS
from .anime2real_node import NODE_CLASS_MAPPINGS as A2R_NODE_CLASS_MAPPINGS
from .anime2real_node import NODE_DISPLAY_NAME_MAPPINGS as A2R_NODE_DISPLAY_NAME_MAPPINGS
from .amv_guide_node import NODE_CLASS_MAPPINGS as AMV_NODE_CLASS_MAPPINGS
from .amv_guide_node import NODE_DISPLAY_NAME_MAPPINGS as AMV_NODE_DISPLAY_NAME_MAPPINGS
from .ltx_identity_overlap import NODE_CLASS_MAPPINGS as IDT_NODE_CLASS_MAPPINGS
from .ltx_identity_overlap import NODE_DISPLAY_NAME_MAPPINGS as IDT_NODE_DISPLAY_NAME_MAPPINGS
from .ltx_multiref_slots import NODE_CLASS_MAPPINGS as MRSLOT_NODE_CLASS_MAPPINGS
from .ltx_multiref_slots import NODE_DISPLAY_NAME_MAPPINGS as MRSLOT_NODE_DISPLAY_NAME_MAPPINGS
from .ltx_multiple_controls import NODE_CLASS_MAPPINGS as MC_NODE_CLASS_MAPPINGS
from .ltx_multiple_controls import NODE_DISPLAY_NAME_MAPPINGS as MC_NODE_DISPLAY_NAME_MAPPINGS
from .color_mask_node import NODE_CLASS_MAPPINGS as CM_NODE_CLASS_MAPPINGS
from .color_mask_node import NODE_DISPLAY_NAME_MAPPINGS as CM_NODE_DISPLAY_NAME_MAPPINGS
try:
from .minimax_h3_singleframe_vae import NODE_CLASS_MAPPINGS as MMVAE_NODE_CLASS_MAPPINGS
from .minimax_h3_singleframe_vae import NODE_DISPLAY_NAME_MAPPINGS as MMVAE_NODE_DISPLAY_NAME_MAPPINGS
except Exception as _e: # noqa
print(f"[BFSNodes] MiniMax-H3 VAE loaders not loaded: {_e!r}")
MMVAE_NODE_CLASS_MAPPINGS, MMVAE_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
try:
from .sharpest_frame import NODE_CLASS_MAPPINGS as SHARP_NODE_CLASS_MAPPINGS
from .sharpest_frame import NODE_DISPLAY_NAME_MAPPINGS as SHARP_NODE_DISPLAY_NAME_MAPPINGS
except Exception as _e: # noqa
print(f"[BFSNodes] Sharpest Frame node not loaded: {_e!r}")
SHARP_NODE_CLASS_MAPPINGS, SHARP_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
try:
from .minimax_h3_direct_decode import NODE_CLASS_MAPPINGS as MMDD_NODE_CLASS_MAPPINGS
from .minimax_h3_direct_decode import NODE_DISPLAY_NAME_MAPPINGS as MMDD_NODE_DISPLAY_NAME_MAPPINGS
except Exception as _e: # noqa
print(f"[BFSNodes] MiniMax-H3 direct decode node not loaded: {_e!r}")
MMDD_NODE_CLASS_MAPPINGS, MMDD_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
try:
from .save_h3_latent import NODE_CLASS_MAPPINGS as SVH3_NODE_CLASS_MAPPINGS
from .save_h3_latent import NODE_DISPLAY_NAME_MAPPINGS as SVH3_NODE_DISPLAY_NAME_MAPPINGS
except Exception as _e: # noqa
print(f"[BFSNodes] Save H3 Latent node not loaded: {_e!r}")
SVH3_NODE_CLASS_MAPPINGS, SVH3_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
try:
from .edit_prompt_fixer import NODE_CLASS_MAPPINGS as EPF_NODE_CLASS_MAPPINGS
from .edit_prompt_fixer import NODE_DISPLAY_NAME_MAPPINGS as EPF_NODE_DISPLAY_NAME_MAPPINGS
except Exception as _e: # noqa
print(f"[BFSNodes] Edit Anything Prompt Fixer not loaded: {_e!r}")
EPF_NODE_CLASS_MAPPINGS, EPF_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
try:
from .ltx_identity_multiangle import NODE_CLASS_MAPPINGS as MA_NODE_CLASS_MAPPINGS
from .ltx_identity_multiangle import NODE_DISPLAY_NAME_MAPPINGS as MA_NODE_DISPLAY_NAME_MAPPINGS
except Exception as _e: # noqa
print(f"[BFSNodes] LTX Identity Multiple Angles node not loaded: {_e!r}")
MA_NODE_CLASS_MAPPINGS, MA_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
try:
from .ltx_identity_gemma_vision import NODE_CLASS_MAPPINGS as GV_NODE_CLASS_MAPPINGS
from .ltx_identity_gemma_vision import NODE_DISPLAY_NAME_MAPPINGS as GV_NODE_DISPLAY_NAME_MAPPINGS
except Exception as _e: # noqa
print(f"[BFSNodes] LTX Identity Gemma-Vision node not loaded: {_e!r}")
GV_NODE_CLASS_MAPPINGS, GV_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
try:
from .multiref_sheet_node import NODE_CLASS_MAPPINGS as MRS_NODE_CLASS_MAPPINGS
from .multiref_sheet_node import NODE_DISPLAY_NAME_MAPPINGS as MRS_NODE_DISPLAY_NAME_MAPPINGS
except Exception as _e: # noqa
print(f"[BFSNodes] Multi-Ref Sheet Builder node not loaded: {_e!r}")
MRS_NODE_CLASS_MAPPINGS, MRS_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
try:
from .ltx_multishot_prompt import NODE_CLASS_MAPPINGS as MSP_NODE_CLASS_MAPPINGS
from .ltx_multishot_prompt import NODE_DISPLAY_NAME_MAPPINGS as MSP_NODE_DISPLAY_NAME_MAPPINGS
except Exception as _e: # noqa
print(f"[BFSNodes] LTX Multishot Prompt node not loaded: {_e!r}")
MSP_NODE_CLASS_MAPPINGS, MSP_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
# CAN / AdaLN node disabled: empirically the AdaLN modulation degrades the video (the identity
# gain came from the projector + LoRA, not the CAN). Kept the file but not registered.
CAN_NODE_CLASS_MAPPINGS, CAN_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
try:
from .bfs_h3_side_panel import NODE_CLASS_MAPPINGS as PANEL_NODE_CLASS_MAPPINGS
from .bfs_h3_side_panel import NODE_DISPLAY_NAME_MAPPINGS as PANEL_NODE_DISPLAY_NAME_MAPPINGS
except Exception as _e: # noqa
print(f"[BFSNodes] H3 side panel nodes unavailable: {_e}")
PANEL_NODE_CLASS_MAPPINGS, PANEL_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
try:
from .bfs_shot_loop import NODE_CLASS_MAPPINGS as SHOT_NODE_CLASS_MAPPINGS
from .bfs_shot_loop import NODE_DISPLAY_NAME_MAPPINGS as SHOT_NODE_DISPLAY_NAME_MAPPINGS
except Exception as _e: # noqa
print(f"[BFSNodes] Shot loop nodes not loaded: {_e!r}")
SHOT_NODE_CLASS_MAPPINGS, SHOT_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
try:
from .bfs_shot_noise import NODE_CLASS_MAPPINGS as SNOISE_NODE_CLASS_MAPPINGS
from .bfs_shot_noise import NODE_DISPLAY_NAME_MAPPINGS as SNOISE_NODE_DISPLAY_NAME_MAPPINGS
except Exception as _e: # noqa
print(f"[BFSNodes] BFS Shot Noise node not loaded: {_e!r}")
SNOISE_NODE_CLASS_MAPPINGS, SNOISE_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
try:
from .minimax_h3_downscaled_guide import NODE_CLASS_MAPPINGS as H3GUIDE_NODE_CLASS_MAPPINGS
from .minimax_h3_downscaled_guide import NODE_DISPLAY_NAME_MAPPINGS as H3GUIDE_NODE_DISPLAY_NAME_MAPPINGS
except Exception as _e: # noqa
print(f"[BFSNodes] MiniMax-H3 downscaled guide nodes not loaded: {_e!r}")
H3GUIDE_NODE_CLASS_MAPPINGS, H3GUIDE_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
try:
from .headswap_ltx_masked_sampler import NODE_CLASS_MAPPINGS as HSMS_NODE_CLASS_MAPPINGS
from .headswap_ltx_masked_sampler import NODE_DISPLAY_NAME_MAPPINGS as HSMS_NODE_DISPLAY_NAME_MAPPINGS
except Exception as _e: # noqa
print(f"[BFSNodes] BFS Sampler not loaded: {_e!r}")
HSMS_NODE_CLASS_MAPPINGS, HSMS_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
try:
from .bfs_lora_surgery import NODE_CLASS_MAPPINGS as LSURG_NODE_CLASS_MAPPINGS
from .bfs_lora_surgery import NODE_DISPLAY_NAME_MAPPINGS as LSURG_NODE_DISPLAY_NAME_MAPPINGS
except Exception as _e: # noqa
print(f"[BFSNodes] LoRA Surgery node not loaded: {_e!r}")
LSURG_NODE_CLASS_MAPPINGS, LSURG_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
NODE_CLASS_MAPPINGS = {
**H3GUIDE_NODE_CLASS_MAPPINGS,
**SHOT_NODE_CLASS_MAPPINGS,
**PANEL_NODE_CLASS_MAPPINGS,
**GV_NODE_CLASS_MAPPINGS,
**CAN_NODE_CLASS_MAPPINGS,
**BFS_NODE_CLASS_MAPPINGS,
**LTXV_EA_NODE_CLASS_MAPPINGS,
**HEADSWAP_NODE_CLASS_MAPPINGS,
**A2R_NODE_CLASS_MAPPINGS,
**AMV_NODE_CLASS_MAPPINGS,
**IDT_NODE_CLASS_MAPPINGS,
**MC_NODE_CLASS_MAPPINGS,
**MRSLOT_NODE_CLASS_MAPPINGS,
**MA_NODE_CLASS_MAPPINGS,
**MRS_NODE_CLASS_MAPPINGS,
**CM_NODE_CLASS_MAPPINGS,
**MSP_NODE_CLASS_MAPPINGS,
**MMVAE_NODE_CLASS_MAPPINGS,
**SHARP_NODE_CLASS_MAPPINGS,
**SNOISE_NODE_CLASS_MAPPINGS,
**MMDD_NODE_CLASS_MAPPINGS,
**SVH3_NODE_CLASS_MAPPINGS,
**EPF_NODE_CLASS_MAPPINGS,
**HSMS_NODE_CLASS_MAPPINGS,
**LSURG_NODE_CLASS_MAPPINGS,
}
NODE_DISPLAY_NAME_MAPPINGS = {
**H3GUIDE_NODE_DISPLAY_NAME_MAPPINGS,
**SHOT_NODE_DISPLAY_NAME_MAPPINGS,
**PANEL_NODE_DISPLAY_NAME_MAPPINGS,
**GV_NODE_DISPLAY_NAME_MAPPINGS,
**CAN_NODE_DISPLAY_NAME_MAPPINGS,
**BFS_NODE_DISPLAY_NAME_MAPPINGS,
**LTXV_EA_NODE_DISPLAY_NAME_MAPPINGS,
**HEADSWAP_NODE_DISPLAY_NAME_MAPPINGS,
**A2R_NODE_DISPLAY_NAME_MAPPINGS,
**AMV_NODE_DISPLAY_NAME_MAPPINGS,
**IDT_NODE_DISPLAY_NAME_MAPPINGS,
**MC_NODE_DISPLAY_NAME_MAPPINGS,
**MRSLOT_NODE_DISPLAY_NAME_MAPPINGS,
**MA_NODE_DISPLAY_NAME_MAPPINGS,
**MRS_NODE_DISPLAY_NAME_MAPPINGS,
**CM_NODE_DISPLAY_NAME_MAPPINGS,
**MSP_NODE_DISPLAY_NAME_MAPPINGS,
**MMVAE_NODE_DISPLAY_NAME_MAPPINGS,
**SHARP_NODE_DISPLAY_NAME_MAPPINGS,
**SNOISE_NODE_DISPLAY_NAME_MAPPINGS,
**MMDD_NODE_DISPLAY_NAME_MAPPINGS,
**SVH3_NODE_DISPLAY_NAME_MAPPINGS,
**EPF_NODE_DISPLAY_NAME_MAPPINGS,
**HSMS_NODE_DISPLAY_NAME_MAPPINGS,
**LSURG_NODE_DISPLAY_NAME_MAPPINGS,
}
WEB_DIRECTORY = "./web/js"
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS", "WEB_DIRECTORY"]
+154
View File
@@ -0,0 +1,154 @@
"""AMV Guide Builder — make the keyframes + blank guide video for the AMV av2av LoRA.
Modes (where the keyframe IMAGES come from):
- auto_scene: input video -> detect scene cuts -> keyframe at each cut
- manual_indices: input video + indices "0,16,34" -> those frames as keyframes
- external_images: a batch of images (chain Load Image -> Image Batch) -> placed on the timeline
Placement (WHERE keyframes go, for external_images):
- audio_beats: detect beats/onsets in the input `audio` and drop one image per beat (in order,
cycling the images if there are more beats than images). This syncs the cuts to
the music — exactly how AMVs are edited.
- even: evenly spaced across `length`
- indices: explicit `indices` string
Output is an IMAGE batch [length,H,W,C] (float 0..1), ready for VAE-encode / reference conditioning.
`hold` repeats each keyframe N frames so it survives the LTX 8x temporal VAE.
"""
import numpy as np
import torch
CATEGORY = "BFS/video"
def _parse_indices(s):
out = []
for tok in str(s).replace(";", ",").split(","):
tok = tok.strip()
if tok.isdigit():
out.append(int(tok))
return out
def _auto_cuts(video, threshold):
"""Content-diff scene cuts on [N,H,W,C]. Returns sorted cut indices (incl 0)."""
cuts = [0]
prev = video[0]
for i in range(1, video.shape[0]):
if (video[i] - prev).abs().mean().item() > threshold:
cuts.append(i)
prev = video[i]
return cuts
def _audio_to_mono(audio):
wf = audio["waveform"] # [B, C, samples]
sr = int(audio["sample_rate"])
y = wf[0].mean(0).detach().cpu().numpy().astype(np.float32)
return y, sr
def _beat_indices(audio, fps, length):
"""Beat (or onset) frame indices for the guide timeline. librosa if available, else RMS-onset."""
y, sr = _audio_to_mono(audio)
times = None
try:
import librosa
_, beats = librosa.beat.beat_track(y=y, sr=sr)
times = librosa.frames_to_time(beats, sr=sr)
if len(times) < 2: # fall back to onsets for sparse beats
on = librosa.onset.onset_detect(y=y, sr=sr, units="time")
if len(on) > len(times):
times = on
except Exception:
# fallback: RMS energy peaks
hop = max(1, sr // 100)
rms = np.array([np.sqrt(np.mean(y[i:i + hop] ** 2)) for i in range(0, len(y), hop)])
d = np.diff(rms, prepend=rms[:1])
thr = d.mean() + d.std()
peaks = np.where(d > thr)[0]
times = peaks * hop / sr
idx = sorted({int(round(t * fps)) for t in times if 0 <= t * fps < length})
if 0 not in idx:
idx = [0] + idx
return idx
class AmvGuideBuilder:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"mode": (["auto_scene", "manual_indices", "external_images"],),
"length": ("INT", {"default": 121, "min": 1, "max": 100000}),
"hold": ("INT", {"default": 4, "min": 1, "max": 64,
"tooltip": "Frames each keyframe is shown (>=4 survives LTX 8x temporal VAE)."}),
"fill": (["white", "black"],),
"placement": (["audio_beats", "even", "indices"],
{"tooltip": "external_images: how to position the images on the timeline."}),
"fps": ("FLOAT", {"default": 25.0, "min": 1.0, "max": 120.0,
"tooltip": "Used to convert audio beat times -> frame indices."}),
},
"optional": {
"video": ("IMAGE", {"tooltip": "Source frames (auto_scene / manual_indices)."}),
"images": ("IMAGE", {"tooltip": "Keyframe images batch (external_images)."}),
"audio": ("AUDIO", {"tooltip": "Drives placement=audio_beats (sync cuts to the music)."}),
"indices": ("STRING", {"default": "", "tooltip": "Comma list e.g. '0,16,34' (placement=indices / manual)."}),
"scene_threshold": ("FLOAT", {"default": 0.10, "min": 0.0, "max": 1.0, "step": 0.01}),
},
}
RETURN_TYPES = ("IMAGE", "STRING")
RETURN_NAMES = ("guide", "debug")
FUNCTION = "build"
CATEGORY = CATEGORY
DESCRIPTION = ("Builds the AMV guide (keyframes + blank). external_images can auto-place images "
"on the music's beats (placement=audio_beats). Feed `guide` into VAE/reference.")
def build(self, mode, length, hold, fill, placement, fps,
video=None, images=None, audio=None, indices="", scene_threshold=0.10):
idx = _parse_indices(indices)
keyframes = [] # (frame_index, image[H,W,C])
if mode == "external_images":
if images is None or images.shape[0] == 0:
raise ValueError("external_images mode needs an `images` batch.")
H, W, n = images.shape[1], images.shape[2], images.shape[0]
# decide placement positions
if placement == "audio_beats":
if audio is None:
raise ValueError("placement=audio_beats needs an `audio` input.")
pos = _beat_indices(audio, fps, length)
elif placement == "indices" and idx:
pos = idx
else: # even
pos = [round(k * (length - 1) / max(1, n - 1)) for k in range(n)] if n > 1 else [0]
# map one image per position (cycle images if fewer than positions)
for k, at in enumerate(pos):
keyframes.append((min(int(at), length - 1), images[k % n]))
else:
if video is None or video.shape[0] == 0:
raise ValueError(f"{mode} mode needs a `video`.")
H, W = video.shape[1], video.shape[2]
cuts = _auto_cuts(video, scene_threshold) if mode == "auto_scene" else (idx or [0])
for ci in cuts:
if 0 <= ci < video.shape[0]:
keyframes.append((min(ci, length - 1), video[ci]))
val = 1.0 if fill == "white" else 0.0
guide = torch.full((length, H, W, 3), val, dtype=torch.float32)
painted = []
for at, frame in keyframes:
f = frame[..., :3].to(torch.float32)
for h in range(hold):
if at + h < length:
guide[at + h] = f
painted.append(at)
dbg = (f"AMV guide | mode={mode} placement={placement if mode=='external_images' else '-'} | "
f"{W}x{H} {length}f hold={hold} fill={fill} | {len(keyframes)} keyframes @ {sorted(set(painted))}")
return (guide, dbg)
NODE_CLASS_MAPPINGS = {"BFSAmvGuideBuilder": AmvGuideBuilder}
NODE_DISPLAY_NAME_MAPPINGS = {"BFSAmvGuideBuilder": "AMV Guide Builder"}
+528
View File
@@ -0,0 +1,528 @@
"""Crop-path planner, vendored from drozbay/MaskVidExperiments.
Source: https://github.com/drozbay/MaskVidExperiments (commit d98cc89, v0.2.0)
Copyright (c) drozbay. Licensed GPL-3.0, the same licence this pack carries,
which is what makes vendoring it legal here.
Vendored rather than imported so the head-swap sampler has no runtime
dependency on that pack being installed. Behaviour is unchanged; only the
ComfyUI node classes were dropped (they stay in the original pack, where they
belong) and the planner import was repointed.
Why this code and not a crop of my own: naive per-frame crops around a mask
jitter in position and size, and a video model reads that jitter as camera
motion. These boxes are planned over the whole clip, so they hold still through
mask noise and occlusion and follow only sustained movement.
"""
"""Subject crop planner: shape as an optimization output.
Standalone numpy/scipy planner solved against binary masks; produces the
per-frame crop boxes only.
Structure per mode:
tracked -- one LP: scalar W, H + per-frame positions. Shape emerges.
zoomed -- outer sweep over constant aspect candidates, inner LP per
candidate: per-frame height (width slaved to aspect) +
positions. Rent is charged against the tightest
aspect-respecting padded box, never per axis.
combined -- closed form: padded union box (also the movement_cost -> inf
limit of the other two).
Dials (p):
crop_scale padding promise as a multiple of the subject
min_padding_allowed fraction of the promise enforced as hard floors
min_padding_allowed_window odd median window defining sustained motion
pad_deficit_tol shortfall above the floor costs 1/this per px-frame
pad_surplus_tol size beyond the padded need costs 1/this per px-frame
resize_cost cost per pixel of box size change (zoomed)
movement_cost cost per pixel of box movement
center_pull tie-break pull toward the subject center
end_tightening fraction of rent charged on the phantoms
end_tightening_window phantom frames held at each end
zoom_step >1 quantizes sizes to geometric levels (zoomed)
max_zoom_rate >1 caps per-frame size ratio change (zoomed), 0/1 off
aspect_ratio >0 fixes the shape and skips the sweep
seamless_loop periodic plan (positions and sizes wrap)
"""
import math
import numpy as np
from scipy import ndimage, sparse
from scipy.optimize import linprog
def _extent_tracks(binary):
"""Per-frame subject extents, NaN gaps interpolated and edge-held."""
n = binary.shape[0]
bx0 = np.full(n, np.nan)
bx1 = np.full(n, np.nan)
by0 = np.full(n, np.nan)
by1 = np.full(n, np.nan)
for i, frame in enumerate(binary):
ys, xs = frame.nonzero()
if len(xs):
bx0[i], bx1[i] = xs.min(), xs.max()
by0[i], by1[i] = ys.min(), ys.max()
known = ~np.isnan(bx0)
idx = np.arange(n)
out = [np.interp(idx, idx[known], t[known]) for t in (bx0, bx1, by0, by1)]
return out, known
def _median_tracks(tracks, window):
g = max(int(window), 1) | 1
return [ndimage.median_filter(t, size=g, mode="nearest") for t in tracks]
def _round_corridor(t, lo, hi):
ilo = np.ceil(lo - 1e-6)
ihi = np.floor(hi + 1e-6)
x = np.clip(np.round(t), ilo, ihi)
return np.where(ilo > ihi, np.round((lo + hi) / 2), x).astype(int)
class _LP:
"""Row-wise builder for sparse linprog problems."""
def __init__(self):
self.blocks = [] # (name, size)
self.off = {}
self.nv = 0
self.ri, self.ci, self.vv, self.rhs = [], [], [], []
self.c = []
self.lo, self.hi = [], []
def var(self, name, size, cost=0.0, lo=0.0, hi=None):
self.off[name] = self.nv
self.nv += size
cost = np.broadcast_to(np.asarray(cost, dtype=float), (size,))
lo = np.broadcast_to(np.asarray(lo, dtype=float), (size,))
self.c.extend(cost)
self.lo.extend(lo)
if hi is None:
self.hi.extend([None] * size)
else:
hi_arr = np.broadcast_to(np.asarray(hi, dtype=float), (size,))
self.hi.extend(hi_arr)
return self.off[name]
def row(self, cols, vals, b):
r = len(self.rhs)
self.ri.extend([r] * len(cols))
self.ci.extend(cols)
self.vv.extend(vals)
self.rhs.append(b)
def solve(self, eq=None):
A = sparse.csr_matrix((self.vv, (self.ri, self.ci)),
shape=(len(self.rhs), self.nv))
bounds = [(l, h) for l, h in zip(self.lo, self.hi)]
kw = {}
if eq is not None:
er, ec, ev, eb = eq
kw["A_eq"] = sparse.csr_matrix((ev, (er, ec)),
shape=(max(er) + 1, self.nv))
kw["b_eq"] = np.asarray(eb, dtype=float)
res = linprog(np.array(self.c), A_ub=A,
b_ub=np.array(self.rhs, dtype=float),
bounds=bounds, method="highs", **kw)
if not res.success:
raise RuntimeError(f"crop planner solve failed: {res.message}")
return res
def _prep(binary, img_w, img_h, p):
"""Shared preprocessing: extent tracks, phantoms, pads, floors."""
n = binary.shape[0]
(bx0, bx1, by0, by1), known = _extent_tracks(binary)
loop = bool(p.get("seamless_loop")) and n > 1
s0 = int(p.get("end_tightening_window", 0)) if not loop else 0
if s0:
hold = lambda a: np.concatenate(
[np.full(s0, a[0]), a, np.full(s0, a[-1])])
bx0, bx1, by0, by1 = hold(bx0), hold(bx1), hold(by0), hold(by1)
m = len(bx0)
scale = p["crop_scale"]
padx = (scale - 1) / 2 * (bx1 - bx0 + 1)
pady = (scale - 1) / 2 * (by1 - by0 + 1)
floor = float(p.get("min_padding_allowed", 0.0))
sx0, sx1, sy0, sy1 = _median_tracks([bx0, bx1, by0, by1],
p.get("min_padding_allowed_window", 16))
pad_fx = floor * (scale - 1) / 2 * (sx1 - sx0 + 1)
pad_fy = floor * (scale - 1) / 2 * (sy1 - sy0 + 1)
# floor datums, clamped to the image (a side pinned at the image edge
# is exempt from its promise)
flo_x = np.minimum(sx1 + 1 + pad_fx, img_w)
fhi_x = np.maximum(sx0 - pad_fx, 0)
flo_y = np.minimum(sy1 + 1 + pad_fy, img_h)
fhi_y = np.maximum(sy0 - pad_fy, 0)
# soft targets: the full promise, clamped to the image
lx = np.maximum(bx0 - padx, 0)
rx = np.minimum(bx1 + 1 + padx, img_w)
ly = np.maximum(by0 - pady, 0)
ry = np.minimum(by1 + 1 + pady, img_h)
rent_scale = np.ones(m)
if s0:
rent_scale[:s0] = p.get("end_tightening", 0.0)
rent_scale[m - s0:] = p.get("end_tightening", 0.0)
return {
"n": n, "m": m, "s0": s0, "loop": loop, "known": known,
"bx0": bx0, "bx1": bx1, "by0": by0, "by1": by1,
"padx": padx, "pady": pady, "floor": floor,
"flo_x": flo_x, "fhi_x": fhi_x, "flo_y": flo_y, "fhi_y": fhi_y,
"lx": lx, "rx": rx, "ly": ly, "ry": ry, "rent_scale": rent_scale,
}
def _tv_rows(lp, var, tv, m, loop):
nt = m if loop else m - 1
for i in range(nt):
a, b = lp.off[var] + i, lp.off[var] + (i + 1) % m
t = lp.off[tv] + i
lp.row([b, a, t], [1, -1, -1], 0)
lp.row([b, a, t], [-1, 1, -1], 0)
def _solve_tracked(pre, img_w, img_h, p):
"""Scalar-size LP: W, H are single variables, shape emerges from the
movement/rent balance. Manual aspect adds one equality."""
m, loop = pre["m"], pre["loop"]
nt = m if loop else m - 1
bx0, bx1, by0, by1 = pre["bx0"], pre["bx1"], pre["by0"], pre["by1"]
padx, pady = pre["padx"], pre["pady"]
soft = 1.0 / max(p.get("pad_deficit_tol", 16), 1e-6)
rent = pre["rent_scale"] / max(p.get("pad_surplus_tol", 16), 1e-6)
mc = p.get("movement_cost", 1.0)
eps = p.get("center_pull", 1e-4)
extw = bx1 - bx0 + 1
exth = by1 - by0 + 1
lp = _LP()
lp.var("W", 1, lo=max(extw.max(), 1), hi=img_w)
lp.var("H", 1, lo=max(exth.max(), 1), hi=img_h)
ub_x = np.maximum(bx0, 0)
ub_y = np.maximum(by0, 0)
if pre["floor"] > 0:
ub_x = np.minimum(ub_x, pre["fhi_x"])
ub_y = np.minimum(ub_y, pre["fhi_y"])
lp.var("x", m, lo=0.0, hi=ub_x)
lp.var("y", m, lo=0.0, hi=ub_y)
lp.var("tx", nt, cost=mc)
lp.var("ty", nt, cost=mc)
lp.var("vx", m, cost=soft)
lp.var("vy", m, cost=soft)
lp.var("rx", m, cost=rent)
lp.var("ry", m, cost=rent)
lp.var("ux", m, cost=eps)
lp.var("uy", m, cost=eps)
_tv_rows(lp, "x", "tx", m, loop)
_tv_rows(lp, "y", "ty", m, loop)
W, H = lp.off["W"], lp.off["H"]
for i in range(m):
x, y = lp.off["x"] + i, lp.off["y"] + i
vx, vy = lp.off["vx"] + i, lp.off["vy"] + i
rx, ry = lp.off["rx"] + i, lp.off["ry"] + i
ux, uy = lp.off["ux"] + i, lp.off["uy"] + i
lp.row([x, W], [-1, -1], -(bx1[i] + 1)) # containment right
lp.row([x, W], [1, 1], img_w) # image right
lp.row([y, H], [-1, -1], -(by1[i] + 1))
lp.row([y, H], [1, 1], img_h)
lp.row([x, vx], [1, -1], pre["lx"][i]) # soft pad, left
lp.row([x, W, vx], [-1, -1, -1], -pre["rx"][i]) # soft pad, right
lp.row([y, vy], [1, -1], pre["ly"][i])
lp.row([y, H, vy], [-1, -1, -1], -pre["ry"][i])
lp.row([W, rx], [1, -1], min(extw[i] + 2 * padx[i], img_w))
lp.row([H, ry], [1, -1], min(exth[i] + 2 * pady[i], img_h))
if pre["floor"] > 0:
lp.row([x, W], [-1, -1], -pre["flo_x"][i])
lp.row([y, H], [-1, -1], -pre["flo_y"][i])
cx = (bx0[i] + bx1[i] + 1) / 2
cy = (by0[i] + by1[i] + 1) / 2
lp.row([x, W, ux], [1, 0.5, -1], cx)
lp.row([x, W, ux], [-1, -0.5, -1], -cx)
lp.row([y, H, uy], [1, 0.5, -1], cy)
lp.row([y, H, uy], [-1, -0.5, -1], -cy)
eq = None
ar = p.get("aspect_ratio", 0.0)
if ar > 0:
eq = ([0, 0], [W, H], [1.0, -ar], [0.0])
res = lp.solve(eq=eq)
o = lp.off
return (res.x[W], res.x[H],
res.x[o["x"]:o["x"] + m], res.x[o["y"]:o["y"] + m])
def _yielded_floors(pre, ar, img_w, img_h):
"""Relax the floor datums per side wherever containment, the image, or
the aspect lock leaves no room."""
bx0, bx1, by0, by1 = pre["bx0"], pre["bx1"], pre["by0"], pre["by1"]
wmax = min(img_w, ar * img_h)
hmax = min(img_h, img_w / ar)
flo_x, fhi_x = pre["flo_x"].copy(), pre["fhi_x"].copy()
flo_y, fhi_y = pre["flo_y"].copy(), pre["fhi_y"].copy()
ex = np.maximum(flo_x - fhi_x - wmax, 0)
flo_x = np.maximum(flo_x - ex / 2, bx1 + 1)
fhi_x = np.minimum(fhi_x + ex / 2, np.maximum(bx0, 0))
flo_x = np.minimum(flo_x, np.maximum(bx0, 0) + wmax)
fhi_x = np.maximum(fhi_x, bx1 + 1 - wmax)
ey = np.maximum(flo_y - fhi_y - hmax, 0)
flo_y = np.maximum(flo_y - ey / 2, by1 + 1)
fhi_y = np.minimum(fhi_y + ey / 2, np.maximum(by0, 0))
flo_y = np.minimum(flo_y, np.maximum(by0, 0) + hmax)
fhi_y = np.maximum(fhi_y, by1 + 1 - hmax)
return flo_x, fhi_x, flo_y, fhi_y
def _solve_zoomed_at(pre, ar, img_w, img_h, p):
"""Inner LP for one aspect candidate. Width is ar*h throughout; rent
is charged on h beyond the tightest aspect-respecting padded need, so
width the ratio forces is never billed."""
m, loop = pre["m"], pre["loop"]
nt = m if loop else m - 1
bx0, bx1, by0, by1 = pre["bx0"], pre["bx1"], pre["by0"], pre["by1"]
padx, pady = pre["padx"], pre["pady"]
extw = bx1 - bx0 + 1
exth = by1 - by0 + 1
soft = 1.0 / max(p.get("pad_deficit_tol", 16), 1e-6)
# need is aspect-aware (width the ratio forces is never billed), but
# the excess is billed at its (1+ar) box magnitude so wide and narrow
# candidates pay the same for the same true slack
rent = (1.0 + ar) * pre["rent_scale"] / max(p.get("pad_surplus_tol", 16), 1e-6)
mc = p.get("movement_cost", 1.0)
sc = p.get("resize_cost", 2.0) * (1.0 + ar)
eps = p.get("center_pull", 1e-4)
hmax = min(img_h, img_w / ar)
hmin = np.maximum(np.maximum(exth, extw / ar), 1.0)
if hmin.max() > hmax + 1e-6:
return None # this aspect cannot contain the subject in-image
need_h = np.minimum(
np.maximum(exth + 2 * pady, (extw + 2 * padx) / ar), hmax)
floors = pre["floor"] > 0
if floors:
flo_x, fhi_x, flo_y, fhi_y = _yielded_floors(pre, ar, img_w, img_h)
lp = _LP()
ub_x = np.maximum(bx0, 0)
ub_y = np.maximum(by0, 0)
if floors:
ub_x = np.minimum(ub_x, fhi_x)
ub_y = np.minimum(ub_y, fhi_y)
lp.var("x", m, lo=0.0, hi=ub_x)
lp.var("y", m, lo=0.0, hi=ub_y)
lp.var("h", m, lo=hmin, hi=hmax)
lp.var("tx", nt, cost=mc)
lp.var("ty", nt, cost=mc)
lp.var("th", nt, cost=sc)
lp.var("vx", m, cost=soft)
lp.var("vy", m, cost=soft)
lp.var("r", m, cost=rent)
lp.var("ux", m, cost=eps)
lp.var("uy", m, cost=eps)
_tv_rows(lp, "x", "tx", m, loop)
_tv_rows(lp, "y", "ty", m, loop)
_tv_rows(lp, "h", "th", m, loop)
rate = p.get("max_zoom_rate", 0.0)
if rate and rate > 1.0:
for i in range(nt):
a, b = lp.off["h"] + i, lp.off["h"] + (i + 1) % m
lp.row([b, a], [1, -rate], 0)
lp.row([a, b], [1, -rate], 0)
for i in range(m):
x, y, h = lp.off["x"] + i, lp.off["y"] + i, lp.off["h"] + i
vx, vy = lp.off["vx"] + i, lp.off["vy"] + i
r = lp.off["r"] + i
ux, uy = lp.off["ux"] + i, lp.off["uy"] + i
lp.row([x, h], [-1, -ar], -(bx1[i] + 1)) # containment right
lp.row([x, h], [1, ar], img_w) # image right
lp.row([y, h], [-1, -1], -(by1[i] + 1))
lp.row([y, h], [1, 1], img_h)
lp.row([x, vx], [1, -1], pre["lx"][i])
lp.row([x, h, vx], [-1, -ar, -1], -pre["rx"][i])
lp.row([y, vy], [1, -1], pre["ly"][i])
lp.row([y, h, vy], [-1, -1, -1], -pre["ry"][i])
lp.row([h, r], [1, -1], need_h[i])
if floors:
lp.row([x, h], [-1, -ar], -flo_x[i])
lp.row([y, h], [-1, -1], -flo_y[i])
cx = (bx0[i] + bx1[i] + 1) / 2
cy = (by0[i] + by1[i] + 1) / 2
lp.row([x, h, ux], [1, 0.5 * ar, -1], cx)
lp.row([x, h, ux], [-1, -0.5 * ar, -1], -cx)
lp.row([y, h, uy], [1, 0.5, -1], cy)
lp.row([y, h, uy], [-1, -0.5, -1], -cy)
res = lp.solve()
o = lp.off
# the objective is the sweep score: floors carry the padding tier as
# hard rows, so candidates compare on how cheaply they serve the
# cell's own priced trade above them
viol = res.x[o["vx"]:o["vx"] + m].sum() + res.x[o["vy"]:o["vy"] + m].sum()
h_t = res.x[o["h"]:o["h"] + m]
# dead width -- width this ratio forces beyond the frame's padded
# horizontal need -- is billed in the score only, at the same rent as
# height slack. Inside the LP that width stays free (billing it there
# collapses manual wide aspects), but across candidates it
# must cost, or the sweep drifts to the widest feasible shape whenever
# the vertical need pins the height. Capped at need_h: excess above
# need_h already carries its width via the (1+ar) rent.
w_need = np.minimum(extw + 2 * padx, img_w)
dead_w = np.maximum(ar * np.minimum(h_t, need_h) - w_need, 0)
dead = ((pre["rent_scale"] * dead_w).sum()
/ max(p.get("pad_surplus_tol", 16), 1e-6))
return {
"ar": ar, "score": res.fun + dead, "viol": viol,
"x": res.x[o["x"]:o["x"] + m], "y": res.x[o["y"]:o["y"] + m],
"h": h_t,
}
def _aspect_candidates(pre, img_w, img_h, n_known):
"""Hull of the per-frame shape quantiles and the padded union aspect:
the winner lives between what the subject looks like and where it
goes."""
bx0, bx1, by0, by1 = pre["bx0"], pre["bx1"], pre["by0"], pre["by1"]
padx, pady = pre["padx"], pre["pady"]
frame_ar = (bx1 - bx0 + 1) / (by1 - by0 + 1)
qs = np.quantile(frame_ar, [0.05, 0.25, 0.5, 0.75, 0.95])
ux0 = (bx0 - padx).min()
ux1 = (bx1 + 1 + padx).max()
uy0 = (by0 - pady).min()
uy1 = (by1 + 1 + pady).max()
union_ar = max(ux1 - ux0, 1.0) / max(uy1 - uy0, 1.0)
lo = min(qs.min(), union_ar)
hi = max(qs.max(), union_ar)
grid = np.geomspace(lo, hi, 7) if hi > lo * 1.02 else np.array([lo])
cands = np.concatenate([grid, qs, [union_ar]])
# feasibility band: the box must contain every frame inside the image
extw = bx1 - bx0 + 1
exth = by1 - by0 + 1
band_lo = extw.max() / img_h
band_hi = img_w / exth.max()
cands = np.clip(cands, band_lo, band_hi)
cands = np.unique(np.round(cands, 3))
keep = [cands[0]]
for a in cands[1:]:
if a > keep[-1] * 1.02:
keep.append(a)
return keep
def _quantize_steps(v, step, upper):
base = v.min()
k = np.ceil(np.log(v / base) / np.log(step) - 1e-9)
return np.minimum(base * step ** k, upper)
def plan(binary, sel, p, divisible_by=16):
"""binary: (n, H, W) bool mask stack. Returns (boxes, info)."""
n, img_h, img_w = binary.shape
if not binary.any():
raise ValueError("all masks are empty, nothing to plan")
scale = p["crop_scale"]
if sel == "combined":
ys, xs = np.nonzero(binary.any(axis=0))
w0 = (xs.max() - xs.min() + 1) * scale
h0 = (ys.max() - ys.min() + 1) * scale
ar = p.get("aspect_ratio", 0.0)
if ar > 0:
if w0 < h0 * ar:
w0 = h0 * ar
else:
h0 = w0 / ar
w = min(math.ceil(w0 / divisible_by) * divisible_by,
(img_w // divisible_by) * divisible_by or img_w)
h = min(math.ceil(h0 / divisible_by) * divisible_by,
(img_h // divisible_by) * divisible_by or img_h)
x = min(max(0, round((xs.min() + xs.max() + 1) / 2 - w / 2)), img_w - w)
y = min(max(0, round((ys.min() + ys.max() + 1) / 2 - h / 2)), img_h - h)
box = {"x": int(x), "y": int(y), "width": int(w), "height": int(h)}
return [box] * n, {"aspect": w / h}
pre = _prep(binary, img_w, img_h, p)
m, s0 = pre["m"], pre["s0"]
bx0, bx1, by0, by1 = pre["bx0"], pre["bx1"], pre["by0"], pre["by1"]
if sel == "tracked":
Wf, Hf, xt, yt = _solve_tracked(pre, img_w, img_h, p)
w = min(math.ceil(Wf / divisible_by) * divisible_by,
(img_w // divisible_by) * divisible_by or img_w)
h = min(math.ceil(Hf / divisible_by) * divisible_by,
(img_h // divisible_by) * divisible_by or img_h)
lo_x = np.maximum(bx1 + 1 - w, 0)
hi_x = np.minimum(np.maximum(bx0, 0), img_w - w)
lo_y = np.maximum(by1 + 1 - h, 0)
hi_y = np.minimum(np.maximum(by0, 0), img_h - h)
if pre["floor"] > 0:
lo_x = np.maximum(lo_x, np.minimum(pre["flo_x"] - w, hi_x))
hi_x = np.minimum(hi_x, np.maximum(pre["fhi_x"], lo_x))
lo_y = np.maximum(lo_y, np.minimum(pre["flo_y"] - h, hi_y))
hi_y = np.minimum(hi_y, np.maximum(pre["fhi_y"], lo_y))
xi = _round_corridor(xt, lo_x, hi_x)[s0:s0 + n]
yi = _round_corridor(yt, lo_y, hi_y)[s0:s0 + n]
boxes = [{"x": int(xi[i]), "y": int(yi[i]),
"width": int(w), "height": int(h)} for i in range(n)]
return boxes, {"aspect": w / h, "W": Wf, "H": Hf}
# zoomed
ar_manual = p.get("aspect_ratio", 0.0)
if ar_manual > 0:
extw = bx1 - bx0 + 1
exth = by1 - by0 + 1
ar = min(max(ar_manual, extw.max() / img_h), img_w / exth.max())
best = _solve_zoomed_at(pre, ar, img_w, img_h, p)
if best is None:
raise RuntimeError("requested aspect_ratio cannot contain the subject")
tried = [best]
else:
tried = []
for ar in _aspect_candidates(pre, img_w, img_h, pre["known"].sum()):
sol = _solve_zoomed_at(pre, ar, img_w, img_h, p)
if sol is not None:
tried.append(sol)
if not tried:
raise RuntimeError("no feasible aspect candidate")
best = min(tried, key=lambda s: s["score"])
for ar in (best["ar"] * 0.93, best["ar"] * 1.075):
sol = _solve_zoomed_at(pre, ar, img_w, img_h, p)
if sol is not None and sol["score"] < best["score"]:
best = sol
ar = best["ar"]
h = best["h"]
if p.get("zoom_step", 1.0) > 1.0:
h = _quantize_steps(h, p["zoom_step"], min(img_h, img_w / ar))
w = np.minimum(h * ar, img_w)
bw = np.minimum(np.ceil(w - 1e-6), img_w).astype(int)
bh = np.minimum(np.ceil(h - 1e-6), img_h).astype(int)
lo_x = np.maximum(bx1 + 1 - bw, 0)
hi_x = np.minimum(np.maximum(bx0, 0), img_w - bw)
lo_y = np.maximum(by1 + 1 - bh, 0)
hi_y = np.minimum(np.maximum(by0, 0), img_h - bh)
if pre["floor"] > 0:
flo_x, fhi_x, flo_y, fhi_y = _yielded_floors(pre, ar, img_w, img_h)
lo_x = np.maximum(lo_x, np.minimum(flo_x - bw, hi_x))
hi_x = np.minimum(hi_x, np.maximum(fhi_x, lo_x))
lo_y = np.maximum(lo_y, np.minimum(flo_y - bh, hi_y))
hi_y = np.minimum(hi_y, np.maximum(fhi_y, lo_y))
xi = _round_corridor(best["x"], lo_x, hi_x)[s0:s0 + n]
yi = _round_corridor(best["y"], lo_y, hi_y)[s0:s0 + n]
bw, bh = bw[s0:s0 + n], bh[s0:s0 + n]
boxes = [{"x": int(min(max(xi[i], 0), img_w - bw[i])),
"y": int(min(max(yi[i], 0), img_h - bh[i])),
"width": int(bw[i]), "height": int(bh[i])} for i in range(n)]
return boxes, {"aspect": ar, "swept": ar_manual <= 0,
"candidates": len(tried)}
+107
View File
@@ -0,0 +1,107 @@
"""Sliding history for MiniMax H3: the end of the previous shot's result as clean frames placed just BEFORE the video
on H3's timeline, so a long continuous take is generated window by window with motion (and look) carried across.
Idea from Akatz's H3 Relay windowed edit and Ethanfel's ComfyUI-MiniMaxH3-Context-Loop (both GPL-3.0); this is an
independent implementation. A guide keyframe marked ``anchor = "history"`` is packed like any H3 guide, then its rows
are kept at the target's origin and everything else on the target timeline (the video, its audio, the other guides)
moves forward by the history's span. Text and reference rows are untouched.
The layout is patched in memory only (ComfyUI files are never edited), once, and only changes layouts that carry a
history keyframe; when ComfyUI's own H3 layout already understands history anchors the patch stands down.
"""
from __future__ import annotations
import logging
import torch
LOG = logging.getLogger("bfs.h3_history")
HISTORY_FRAMES = (17, 34, 51) # whole H3 VAE groups: 1+4+4+4+4 frames per 5 latent steps
_PATCH = "_bfs_history_patch"
def history_span(latent_t: int) -> float:
"""Timeline span of `latent_t` video latent steps (FRAME_RESCALE per pixel frame)."""
import comfy.ldm.minimax.model as mm
return float(sum(mm._video_t_spans(int(latent_t))))
def relocate(layout, keyframes) -> bool:
"""Moves the target timeline past the history keyframes. Returns whether anything changed."""
hist = [i for i, kf in enumerate(keyframes or ()) if kf.get("anchor") == "history"]
if not hist:
return False
# keyframe -> its packed segments: every guide with a video latent owns the next "cond" segment, every guide
# with audio the next "cond_audio" one, in keyframe order (the order PackedLayout packs them)
cond = [s for s in layout.segments if s[2] == "cond"]
cond_a = [s for s in layout.segments if s[2] == "cond_audio"]
own, ci, ai = {}, 0, 0
for i, kf in enumerate(keyframes):
segs = []
if kf.get("latent") is not None and ci < len(cond):
segs.append(cond[ci]); ci += 1
if kf.get("audio_latent") is not None and ai < len(cond_a):
segs.append(cond_a[ai]); ai += 1
own[i] = segs
span = sum(history_span(keyframes[i]["latent"].shape[2]) for i in hist if keyframes[i].get("latent") is not None)
if span <= 0:
return False
keep = torch.zeros(layout.position_ids.shape[0], dtype=torch.bool)
for i in hist:
for a, b, _ in own[i]:
keep[a:b] = True
vid = next(((a, b) for a, b, k in layout.segments if k == "video"), None)
if vid is not None and keep.any():
# another history implementation (e.g. H3 Relay's, or a future ComfyUI) already moved the target: skip
if float(layout.position_ids[vid[0]:vid[1], 0].min()) - float(layout.position_ids[keep, 0].min()) >= span - 1e-6:
return False
move = torch.zeros_like(keep)
for a, b, kind in layout.segments:
if kind in ("video", "audio", "cond", "cond_audio"):
move[a:b] = True
move &= ~keep
pos = layout.position_ids.clone()
pos[move, 0] += span
layout.position_ids = pos
LOG.info("H3 history: %d rows before the video, target timeline moved by %.2f", int(keep.sum()), span)
return True
def install() -> bool:
"""Wraps PackedLayout.__init__ once (process-local). False when ComfyUI is missing or already supports it."""
try:
import comfy.ldm.minimax.model as mm
except ImportError:
return False
cls = mm.PackedLayout
if getattr(cls.__init__, _PATCH, False):
return True
import inspect
if "history" in inspect.getsource(cls): # native history anchors: nothing to do
return False
orig = cls.__init__
def __init__(self, text_len, latent_t, latent_h, latent_w, audio_t, keyframes=None, refs=None):
orig(self, text_len, latent_t, latent_h, latent_w, audio_t, keyframes=keyframes, refs=refs)
relocate(self, keyframes)
setattr(__init__, _PATCH, True)
cls.__init__ = __init__
return True
def add_history(positive, vae, frames: torch.Tensor):
"""Appends `frames` ([N,H,W,3] at the video's size, N = 17k: the end of the previous result) as a history keyframe.
Returns (positive, frames used)."""
import node_helpers
n = (int(frames.shape[0]) // 17) * 17
if n < 17:
return positive, 0
install()
# encoded from CPU frames; the latent comes back on ComfyUI's intermediate device like any H3 guide, and the VAE
# is loaded / freed by ComfyUI's model management (nothing here keeps GPU memory)
with torch.no_grad():
lat = vae.encode(frames[-n:, ..., :3].float().cpu())
kfs = list(positive[0][1].get("minimax_keyframes", []))
kfs.append({"resolved_frame_index": 0, "latent": lat, "anchor": "history"})
return node_helpers.conditioning_set_values(positive, {"minimax_keyframes": kfs}), n
+1091
View File
File diff suppressed because it is too large Load Diff
+691
View File
@@ -0,0 +1,691 @@
"""BFS LoRA Surgery: read any LoRA's layers and blocks, then scale or drop them.
Model-agnostic: the structure is discovered from the key names, not hardcoded. Works with
the diffusers style (``lora_A``/``lora_B``), the kohya style (``lora_down``/``lora_up`` plus
``alpha``) and anything that follows either convention.
Why this exists: a defect a LoRA learned (smoothed skin, identity drifting at distance, a
pose it refuses to copy) usually lives in a specific module family and a specific range of
blocks. Scaling or dropping that group and regenerating tells you where it lives, in minutes,
without retraining. See the node's tooltip for the workflow.
Scaling a module by ``s`` multiplies only the up/B factor, since ``(sB)A = s(BA)``. That keeps
negative values usable and is exact regardless of the alpha convention.
"""
from __future__ import annotations
import hashlib
import json
import os
from datetime import datetime, timezone
import re
import struct
from typing import Any
import torch
import comfy.sd
import comfy.utils
import folder_paths
# ---------------------------------------------------------------------------- parsing
_SUFFIXES = (
".lora_down.weight", ".lora_up.weight",
".lora_A.weight", ".lora_B.weight",
".lora_A.default.weight", ".lora_B.default.weight",
".hada_w1_a", ".hada_w1_b", ".hada_w2_a", ".hada_w2_b",
".lokr_w1", ".lokr_w2", ".lokr_w1_a", ".lokr_w1_b", ".lokr_w2_a", ".lokr_w2_b",
".diff", ".diff_b", ".alpha", ".dora_scale",
)
_UP_SUFFIXES = (".lora_up.weight", ".lora_B.weight", ".lora_B.default.weight")
_BLOCK_RE = re.compile(r"(?<=[._])(\d+)(?=[._])")
def _strip_suffix(key: str) -> str | None:
for s in _SUFFIXES:
if key.endswith(s):
return key[: -len(s)]
return None
_NUM_RE = re.compile(r"\d+")
def _template(path: str) -> tuple[str, list[int], list[tuple[int, int]]]:
"""``a.12.attn1.q`` -> ("a.{}.attn{}.q", [12, 1], spans)."""
nums, spans, out, last = [], [], [], 0
for m in _NUM_RE.finditer(path):
out.append(path[last:m.start()])
out.append("{}")
nums.append(int(m.group()))
spans.append((m.start(), m.end()))
last = m.end()
out.append(path[last:])
return "".join(out), nums, spans
def choose_block_axis(paths: list[str]) -> dict[str, int]:
"""For each path template, decide which numeric slot is the block index.
Picking the last number breaks on names like ``transformer_blocks.12.attn1.to_q``, where
the last one is the ``1`` of ``attn1``, and on nested layouts such as
``down_blocks.0.attentions.1.transformer_blocks.0.ff.net.0.proj``. The block axis is the
slot that varies most across the whole LoRA, so decide it from the file, not from one key.
"""
seen: dict[str, list[set]] = {}
for path in paths:
tpl, nums, _ = _template(path)
slots = seen.setdefault(tpl, [set() for _ in nums])
for i, v in enumerate(nums):
if i < len(slots):
slots[i].add(v)
axis: dict[str, int] = {}
for tpl, slots in seen.items():
if not slots:
continue
counts = [len(x) for x in slots]
best = max(range(len(counts)), key=lambda i: (counts[i], -i))
axis[tpl] = best if counts[best] > 1 else -1
return axis
def _split_block(path: str, axis: dict[str, int] | None = None) -> tuple[str, int | None, str]:
"""Return (family, block_index, module_type).
``transformer_blocks.12.attn.to_q`` -> ("transformer_blocks", 12, "attn.to_q").
Numbers that are not the block axis stay in the name, so ``attn1`` and ``attn2`` remain
distinct module types instead of being folded together.
"""
tpl, nums, spans = _template(path)
if not nums:
return path, None, ""
idx = axis.get(tpl, -1) if axis is not None else (len(nums) - 1)
if idx < 0 or idx >= len(nums):
return path.rstrip("._"), None, ""
start, end = spans[idx]
family = path[:start].rstrip("._")
module = path[end:].lstrip("._")
return family, nums[idx], module
def read_structure(lora_path: str, with_norms: bool = True) -> dict[str, Any]:
"""Describe a LoRA: families, block indices, module types, ranks and ΔW norms.
Reads the safetensors header first (instant) and only loads tensors when norms are asked
for. Norms are what tell you where training actually invested.
"""
with open(lora_path, "rb") as fh:
n = struct.unpack("<Q", fh.read(8))[0]
header = json.loads(fh.read(n))
header.pop("__metadata__", None)
modules: dict[str, dict[str, Any]] = {}
for key, info in header.items():
base = _strip_suffix(key)
if base is None:
continue
entry = modules.setdefault(base, {"rank": None, "keys": []})
entry["keys"].append(key)
shape = info.get("shape") or []
if len(shape) == 2 and entry["rank"] is None:
entry["rank"] = int(min(shape))
norms: dict[str, float] = {}
if with_norms and modules:
norms = _module_norms(lora_path, modules)
axis = choose_block_axis(list(modules))
families: dict[str, dict[str, Any]] = {}
loose: list[dict[str, Any]] = []
all_modules: list[dict[str, Any]] = []
for base, entry in modules.items():
family, block, mtype = _split_block(base, axis)
all_modules.append({"path": base, "family": family, "block": block, "type": mtype,
"rank": entry["rank"], "norm": norms.get(base)})
if block is None:
loose.append({"path": base, "rank": entry["rank"], "norm": norms.get(base)})
continue
fam = families.setdefault(family, {"name": family, "blocks": set(), "types": {}})
fam["blocks"].add(block)
t = fam["types"].setdefault(mtype, {"name": mtype, "count": 0, "rank": entry["rank"],
"norm_sum": 0.0, "norm_n": 0, "blocks": []})
t["count"] += 1
t["blocks"].append(block)
if base in norms:
t["norm_sum"] += norms[base]
t["norm_n"] += 1
out_families = []
for fam in families.values():
types = []
for t in fam["types"].values():
types.append({
"name": t["name"], "count": t["count"], "rank": t["rank"],
"norm": (t["norm_sum"] / t["norm_n"]) if t["norm_n"] else None,
"blocks": sorted(t["blocks"]),
})
types.sort(key=lambda x: -(x["norm"] or 0))
out_families.append({
"name": fam["name"],
"blocks": sorted(fam["blocks"]),
"types": types,
})
out_families.sort(key=lambda f: -sum(t["count"] for t in f["types"]))
all_modules.sort(key=lambda m: (m["family"], m["block"] if m["block"] is not None else -1, m["type"]))
return {
"file": os.path.basename(lora_path),
"total_modules": len(modules),
"families": out_families,
"loose": sorted(loose, key=lambda x: x["path"])[:64],
"modules": all_modules,
"per_module_norms": norms,
}
def _module_norms(lora_path: str, modules: dict[str, dict[str, Any]]) -> dict[str, float]:
"""Exact ||ΔW||_F per module.
ΔW = B·A has rank <= r, so QR both factors and take the singular values of the small
r x r product. No need to materialize the full d x d matrix.
"""
try:
sd = comfy.utils.load_torch_file(lora_path, safe_load=True)
except Exception:
return {}
out: dict[str, float] = {}
for base in modules:
down = sd.get(base + ".lora_down.weight", sd.get(base + ".lora_A.weight"))
up = sd.get(base + ".lora_up.weight", sd.get(base + ".lora_B.weight"))
if down is None or up is None or down.ndim != 2 or up.ndim != 2:
continue
try:
a = down.float()
b = up.float()
scale = 1.0
alpha = sd.get(base + ".alpha")
if alpha is not None:
scale = float(alpha) / a.shape[0]
qb, rb = torch.linalg.qr(b)
qa, ra = torch.linalg.qr(a.T)
s = torch.linalg.svdvals(rb @ ra.T)
out[base] = float(s.pow(2).sum().sqrt()) * scale
except Exception:
continue
del sd
return out
# ---------------------------------------------------------------------------- rules
def _match_blocks(spec: str | None, block: int) -> bool:
"""``spec`` is a block selector: "" / "all" / "8-15" / "0,4,7" / "8-15,24-31"."""
if not spec or spec.strip().lower() in ("all", "*"):
return True
for part in spec.split(","):
part = part.strip()
if not part:
continue
if "-" in part:
lo, _, hi = part.partition("-")
try:
if int(lo) <= block <= int(hi):
return True
except ValueError:
continue
else:
try:
if int(part) == block:
return True
except ValueError:
continue
return False
def build_regex(family: str | None, mtype: str | None, blocks: str | None) -> str:
r"""Build the regex a selection stands for. This is what the UI's "to regex" button calls.
Selecting family ``transformer_blocks``, type ``img_mlp.gate_up`` and blocks ``8-15``
gives ``^.*transformer_blocks\.(8|9|10|11|12|13|14|15)\.img_mlp\.gate_up$``.
Leaving blocks empty matches every block.
"""
fam = re.escape(family) if family else r"[\\w.]+"
if blocks and blocks.strip().lower() not in ("all", "*", ""):
idx: list[int] = []
for part in blocks.split(","):
part = part.strip()
if not part:
continue
if "-" in part:
lo, _, hi = part.partition("-")
try:
idx.extend(range(int(lo), int(hi) + 1))
except ValueError:
continue
else:
try:
idx.append(int(part))
except ValueError:
continue
blk = "(" + "|".join(str(i) for i in sorted(set(idx))) + ")" if idx else r"\\d+"
else:
blk = r"\\d+"
if not mtype or mtype.strip() in ("*", ""):
mod = r".*"
elif mtype.endswith("*"):
mod = re.escape(mtype[:-1]) + r".*"
else:
mod = re.escape(mtype)
return rf"^.*{fam}\.{blk}\.{mod}$"
def _match_type(spec: str | None, mtype: str) -> bool:
if not spec or spec.strip() in ("*", ""):
return True
spec = spec.strip()
if spec.endswith("*"):
return mtype.startswith(spec[:-1])
return mtype == spec
_REGEX_CACHE: dict[str, Any] = {}
def _compiled(pattern: str):
rx = _REGEX_CACHE.get(pattern)
if rx is None:
try:
rx = re.compile(pattern)
except re.error:
rx = False # invalid pattern matches nothing, and never raises mid-run
_REGEX_CACHE[pattern] = rx
return rx
def resolve_scale(rules: list[dict[str, Any]], family: str, block: int | None, mtype: str,
full_path: str | None = None) -> float:
"""Last matching rule wins, which is what makes the UI predictable.
A rule matches either structurally (family / type / blocks) or by ``regex`` against the
module's full path. A rule carrying a regex ignores the structural fields.
"""
scale = 1.0
for rule in rules:
if not rule.get("enabled", True):
continue
m = rule.get("match", {})
pattern = m.get("regex")
if pattern:
rx = _compiled(pattern)
if not rx or full_path is None or not rx.search(full_path):
continue
try:
scale = float(rule.get("scale", 1.0))
except (TypeError, ValueError):
pass
continue
if m.get("family") and m["family"] != family:
continue
if not _match_type(m.get("type"), mtype):
continue
if block is not None and not _match_blocks(m.get("blocks"), block):
continue
if block is None and m.get("blocks"):
continue
try:
scale = float(rule.get("scale", 1.0))
except (TypeError, ValueError):
continue
return scale
def apply_rules(sd: dict[str, torch.Tensor], rules: list[dict[str, Any]]) -> tuple[dict, dict]:
"""Return (new state dict, stats). Scale 0 drops the module entirely."""
if not rules:
return sd, {"kept": None, "dropped": 0, "scaled": 0}
bases = {b for b in (_strip_suffix(k) for k in sd) if b}
axis = choose_block_axis(sorted(bases))
out: dict[str, torch.Tensor] = {}
dropped = scaled = kept = 0
seen: dict[str, float] = {}
for key, tensor in sd.items():
base = _strip_suffix(key)
if base is None:
out[key] = tensor
continue
if base not in seen:
family, block, mtype = _split_block(base, axis)
seen[base] = resolve_scale(rules, family, block, mtype, full_path=base)
s = seen[base]
if s == 0.0:
dropped += 1 if key.endswith(_UP_SUFFIXES) else 0
continue
if s != 1.0 and key.endswith(_UP_SUFFIXES):
# (sB)A == s(BA): exact, and negative values stay usable
out[key] = tensor.float().mul(s).to(tensor.dtype)
scaled += 1
else:
out[key] = tensor
if key.endswith(_UP_SUFFIXES):
kept += 1
return out, {"kept": kept, "dropped": dropped, "scaled": scaled}
# ---------------------------------------------------------------------------- node
class BFSLoraSurgery:
"""Load a LoRA, scale or drop parts of it, and apply the result to the model.
Nothing is written to disk: the edited LoRA lives only in this run, so you can move a
slider and re-queue. Use the panel to pick a module family, a block range and a scale,
or write a regex when you want something the tree cannot express.
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"model": ("MODEL",),
"lora_name": (folder_paths.get_filename_list("loras"),),
"strength_model": ("FLOAT", {"default": 1.0, "min": -20.0, "max": 20.0,
"step": 0.01, "tooltip": "Overall LoRA strength, as usual."}),
"rules": ("STRING", {"default": "[]", "multiline": True,
"tooltip": "JSON list of rules. The panel writes this for you."}),
},
"optional": {"clip": ("CLIP",)},
}
RETURN_TYPES = ("MODEL", "CLIP", "STRING", "STRING")
RETURN_NAMES = ("model", "clip", "report", "rules")
FUNCTION = "apply"
CATEGORY = "BFS/lora"
DESCRIPTION = ("Scale or drop a LoRA's module groups (attention, MLP, per block range) and "
"apply it live. Finds which layers carry a defect without retraining.")
def apply(self, model, lora_name, strength_model, rules, clip=None):
lora_path = folder_paths.get_full_path("loras", lora_name)
if lora_path is None:
raise ValueError(f"LoRA not found: {lora_name}")
sd = comfy.utils.load_torch_file(lora_path, safe_load=True)
parsed: list[dict[str, Any]] = []
if rules and rules.strip():
try:
loaded = json.loads(rules)
if isinstance(loaded, dict):
loaded = loaded.get("rules", [])
if isinstance(loaded, list):
parsed = loaded
except json.JSONDecodeError as exc:
raise ValueError(f"'rules' is not valid JSON: {exc}") from exc
edited, stats = apply_rules(sd, parsed)
total = sum(1 for k in sd if k.endswith(_UP_SUFFIXES))
kept = stats["kept"] if stats["kept"] is not None else total
report = (f"{lora_name}: {kept}/{total} modules kept, {stats['dropped']} dropped, "
f"{stats['scaled']} scaled, {len(parsed)} rule(s), strength {strength_model}")
print(f"[BFSNodes] LoRA surgery, {report}")
new_model, new_clip = comfy.sd.load_lora_for_models(
model, clip, edited, strength_model, strength_model if clip is not None else 0.0)
# `rules` comes back out so it can feed BFS LoRA Surgery (save) without copying JSON
return (new_model, new_clip if clip is not None else clip, report, rules)
def describe_rules(rules: list[dict[str, Any]]) -> str:
"""Short, readable slug for a rule list, used when no filename is given.
``gate_up-b8_15-off__all-b16_23-off__attn-x1.15``. A file named after what was done to it
beats one named v2_final_real.
"""
parts: list[str] = []
for rule in rules:
if not rule.get("enabled", True):
continue
m = rule.get("match", {})
try:
scale = float(rule.get("scale", 1.0))
except (TypeError, ValueError):
scale = 1.0
if m.get("regex"):
what = "re" + hashlib.sha1(m["regex"].encode()).hexdigest()[:6]
else:
core = (m.get("type") or "").replace("*", "").strip(".")
what = re.sub(r"[^A-Za-z0-9_-]", "", core.split(".")[-1]) or "all"
blocks = re.sub(r"[^0-9,_-]", "", (m.get("blocks") or "")).replace(",", "_").replace("-", "_")
tag = what if not blocks else f"{what}-b{blocks}"
parts.append(f"{tag}-{'off' if scale == 0 else f'x{scale:g}'}")
slug = "__".join(parts) if parts else "unchanged"
return re.sub(r"[^A-Za-z0-9._-]", "", slug)
# Linux caps a single filename at 255 bytes (NAME_MAX); Windows caps the whole path near 260
# unless long paths are enabled. Stay well under both, since the folder above us is not ours.
MAX_FILENAME = 160
def fit_filename(stem: str, slug: str, ext: str = ".safetensors", limit: int = MAX_FILENAME) -> str:
"""Build ``stem__slug.ext`` and shrink it to ``limit`` bytes without losing uniqueness.
When it does not fit, both halves are trimmed and a short hash of the full slug is
appended, so two different recipes never collapse onto the same name.
"""
name = f"{stem}__{slug}{ext}"
if len(name.encode("utf-8")) <= limit:
return name
digest = hashlib.sha1(f"{stem}__{slug}".encode()).hexdigest()[:8]
room = limit - len(ext) - len(digest) - 3 # "__" between halves, "-" before hash
keep_stem = min(len(stem), max(16, room // 2))
keep_slug = max(8, room - keep_stem)
return f"{stem[:keep_stem]}__{slug[:keep_slug]}-{digest}{ext}"
def summarize_rules(rules: list[dict[str, Any]]) -> str:
"""One sentence a human can read months later, without parsing JSON."""
drops, scales = [], []
for rule in rules:
if not rule.get("enabled", True):
continue
m = rule.get("match", {})
try:
scale = float(rule.get("scale", 1.0))
except (TypeError, ValueError):
continue
if m.get("regex"):
what = f"modules matching /{m['regex']}/"
else:
t = m.get("type") or "*"
what = "all modules" if t in ("*", "") else t
blocks = m.get("blocks")
if blocks and blocks.strip().lower() not in ("all", "*"):
what += f" in blocks {blocks}"
if scale == 0.0:
drops.append(what)
elif scale != 1.0:
scales.append(f"{what} x{scale:g}")
bits = []
if drops:
bits.append("dropped " + "; ".join(drops))
if scales:
bits.append("scaled " + "; ".join(scales))
return ", ".join(bits) if bits else "no changes"
def _file_digest(path: str) -> str:
"""sha256 of the source, so provenance survives a rename."""
h = hashlib.sha256()
try:
with open(path, "rb") as fh:
for chunk in iter(lambda: fh.read(4 << 20), b""):
h.update(chunk)
return h.hexdigest()
except Exception: # noqa: BLE001
return ""
class BFSLoraSurgerySave:
"""Write the edited LoRA to models/loras as a real file.
Same rules as BFS LoRA Surgery. Leave ``filename`` empty and the name is built from the
rules themselves, so the file says what was done to it. The recipe is also stored in the
safetensors metadata under ``bfs_surgery``, which survives being shared.
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"lora_name": (folder_paths.get_filename_list("loras"),),
"rules": ("STRING", {"default": "[]", "multiline": True,
"tooltip": "Same JSON the surgery panel writes."}),
"filename": ("STRING", {"default": "", "multiline": False,
"tooltip": "Leave empty to name it after the rules."}),
"subfolder": ("STRING", {"default": "surgery", "multiline": False}),
"save_dtype": (["keep", "float16", "bfloat16", "float32"],),
"overwrite": ("BOOLEAN", {"default": False}),
},
}
RETURN_TYPES = ("STRING",)
RETURN_NAMES = ("path",)
FUNCTION = "save"
OUTPUT_NODE = True
CATEGORY = "BFS/lora"
DESCRIPTION = "Apply the same surgery rules and write the result to models/loras."
def save(self, lora_name, rules, filename, subfolder, save_dtype, overwrite):
dest, msg = write_surgery(lora_name, rules, filename, subfolder, save_dtype, overwrite)
print(f"[BFSNodes] LoRA surgery save: {msg}")
return {"ui": {"text": [msg]}, "result": (dest,)}
def write_surgery(lora_name, rules, filename="", subfolder="surgery", save_dtype="keep",
overwrite=False) -> tuple[str, str]:
"""Apply the rules and write the file. Shared by the save node and the panel's button."""
src = folder_paths.get_full_path("loras", lora_name)
if src is None:
raise ValueError(f"LoRA not found: {lora_name}")
parsed: list[dict[str, Any]] = []
if rules and rules.strip():
loaded = json.loads(rules)
if isinstance(loaded, dict):
loaded = loaded.get("rules", [])
if isinstance(loaded, list):
parsed = loaded
sd = comfy.utils.load_torch_file(src, safe_load=True)
edited, stats = apply_rules(sd, parsed)
if save_dtype != "keep":
want = {"float16": torch.float16, "bfloat16": torch.bfloat16, "float32": torch.float32}[save_dtype]
edited = {k: (v.to(want) if v.is_floating_point() else v) for k, v in edited.items()}
stem = os.path.splitext(os.path.basename(src))[0]
typed = re.sub(r"[^A-Za-z0-9._ -]", "", filename).strip()
if typed:
if typed.endswith(".safetensors"):
typed = typed[: -len(".safetensors")]
name = fit_filename(typed, "", "").rstrip("_") + ".safetensors"
else:
name = fit_filename(stem, describe_rules(parsed))
root = folder_paths.get_folder_paths("loras")[0]
out_dir = os.path.join(root, subfolder.strip()) if subfolder.strip() else root
os.makedirs(out_dir, exist_ok=True)
dest = os.path.join(out_dir, name)
if len(dest) > 240:
print(f"[BFSNodes] warning: the path is {len(dest)} characters, which some systems reject")
if os.path.exists(dest) and not overwrite:
base, ext = os.path.splitext(dest)
n = 1
while os.path.exists(f"{base}_{n}{ext}"):
n += 1
dest = f"{base}_{n}{ext}"
meta = {}
try:
with open(src, "rb") as fh:
hn = struct.unpack("<Q", fh.read(8))[0]
meta = json.loads(fh.read(hn)).get("__metadata__", {}) or {}
except Exception: # noqa: BLE001 - metadata is a nicety, not a requirement
meta = {}
meta = {str(k): str(v) for k, v in meta.items()}
total_modules = sum(1 for k in sd if k.endswith(_UP_SUFFIXES))
kept_modules = stats["kept"] if stats["kept"] is not None else total_modules
summary = summarize_rules(parsed)
meta["bfs_surgery"] = json.dumps({
"tool": "ComfyUI-BFSNodes / BFS LoRA Surgery",
"created": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
"source_file": os.path.basename(src),
"source_sha256": _file_digest(src),
"recipe": describe_rules(parsed),
"summary": summary,
"rules": parsed,
"modules_total": total_modules,
"modules_kept": kept_modules,
"modules_dropped": stats["dropped"],
"modules_scaled": stats["scaled"],
"save_dtype": save_dtype,
}, ensure_ascii=False)
# a line any viewer shows without knowing about this tool
meta["bfs_surgery_summary"] = (f"{kept_modules}/{total_modules} modules kept: {summary} "
f"(from {os.path.basename(src)})")
meta["ss_output_name"] = os.path.splitext(name)[0]
from safetensors.torch import save_file
save_file({k: v.contiguous() for k, v in edited.items()}, dest, metadata=meta)
total = total_modules
kept = kept_modules
msg = f"saved {os.path.basename(dest)} ({kept}/{total} modules, {stats['dropped']} dropped)"
return dest, msg
NODE_CLASS_MAPPINGS = {"BFSLoraSurgery": BFSLoraSurgery, "BFSLoraSurgerySave": BFSLoraSurgerySave}
NODE_DISPLAY_NAME_MAPPINGS = {"BFSLoraSurgery": "BFS LoRA Surgery",
"BFSLoraSurgerySave": "BFS LoRA Surgery (save)"}
# ---------------------------------------------------------------------------- http api
try:
from aiohttp import web
from server import PromptServer
_STRUCT_CACHE: dict[tuple[str, float], dict] = {}
@PromptServer.instance.routes.get("/bfs/lora/structure")
async def _bfs_lora_structure(request):
name = request.query.get("name", "")
with_norms = request.query.get("norms", "1") not in ("0", "false", "no")
path = folder_paths.get_full_path("loras", name)
if path is None:
return web.json_response({"error": f"LoRA not found: {name}"}, status=404)
key = (path, os.path.getmtime(path))
cached = _STRUCT_CACHE.get(key)
if cached is None or (with_norms and not cached.get("per_module_norms")):
try:
cached = read_structure(path, with_norms=with_norms)
except Exception as exc: # noqa: BLE001 - surface the reason in the UI
return web.json_response({"error": f"{type(exc).__name__}: {exc}"}, status=500)
_STRUCT_CACHE[key] = cached
return web.json_response(cached)
@PromptServer.instance.routes.post("/bfs/lora/save")
async def _bfs_lora_save(request):
body = await request.json()
try:
dest, msg = write_surgery(
body.get("name", ""), json.dumps(body.get("rules", [])),
body.get("filename", ""), body.get("subfolder", "surgery"),
body.get("save_dtype", "keep"), bool(body.get("overwrite", False)))
except Exception as exc: # noqa: BLE001 - the panel shows the reason
return web.json_response({"error": f"{type(exc).__name__}: {exc}"}, status=500)
return web.json_response({"path": dest, "message": msg})
@PromptServer.instance.routes.post("/bfs/lora/regex")
async def _bfs_lora_regex(request):
body = await request.json()
return web.json_response({
"regex": build_regex(body.get("family"), body.get("type"), body.get("blocks"))})
except Exception as _exc: # noqa: BLE001 - the nodes still work without the panel
print(f"[BFSNodes] LoRA surgery HTTP routes not registered: {_exc!r}")
+4112
View File
File diff suppressed because it is too large Load Diff
+323
View File
@@ -0,0 +1,323 @@
"""BFS Shot Noise: the sampler's starting noise, with optional structure. Every option is off at 0, and with all of
them off the output is exactly ComfyUI's RandomNoise for the same seed.
What can be shaped (the video part of the latent only; the audio noise is always plain):
seed per shot same for every shot / increment / random per shot / overrides (as BFS Shot Seed)
shared_base the COARSE pattern of the noise (block averages on a fixed grid) comes from a base seed instead
of the shot's seed; the fine detail stays the shot's own. The coarse pattern decides layout and
who the subject becomes in the first steps, so shots of one video (or re-rolls of one clip)
agree on it. The grid is defined in blocks, not in latent cells, so the same base seed gives the
same coarse pattern at any resolution, and along the timeline it is one long tape: a shot reads
the part at its own position, whatever its length.
distribution gaussian (default) / laplacian / student-t / uniform, all at unit variance
spectrum_tilt 1/f^alpha spectrum: more (or less) energy in the large blobs ("fractal" noise)
pyramid coarser layers of noise added on top
static_share a part of the noise that is the same in every frame (temporally correlated noise)
guide_structure outside the mask, a little of the guide video's own latent is mixed into the noise, so the
background starts closer to the guide; inside the mask the noise stays free (no pull toward
the original person)
ref_tint inside the mask, the noise leans toward the reference picture's average latent (its colours)
All mixes keep the noise at unit variance (a*signal + sqrt(1-a^2)*noise, block means replaced by block means of the same
variance). The model was trained on plain noise: small values, and compare with 0."""
import hashlib
import math
import torch
import comfy.nested_tensor
import comfy.sample
try:
from .bfs_shot_loop import shot_seed, BFSShotSeed
from .bfs_h3_side_panel import latent_mask
except ImportError: # run as a script (tests)
from bfs_shot_loop import shot_seed, BFSShotSeed
from bfs_h3_side_panel import latent_mask
_pick_seed = shot_seed # BFSShotNoise.make has an input of the same name
def _gen(*key) -> torch.Generator:
return torch.Generator().manual_seed(int.from_bytes(hashlib.sha256("/".join(str(k) for k in key).encode()).digest()[:8], "big") >> 1)
def _edges(n: int, blocks: int) -> torch.Tensor:
"""Block id of each of n cells for `blocks` near-equal blocks (blocks is capped at n)."""
blocks = max(1, min(int(blocks), n))
return torch.clamp((torch.arange(n) * blocks) // n, max=blocks - 1)
def replace_coarse(x: torch.Tensor, base_seed: int, amount: float, grid: int = 6, time_block: int = 2, t0: int = 0) -> torch.Tensor:
"""x [C,T,h,w] unit noise. The mean of every (time block, y block, x block) cell is replaced (amount 1) or mixed
with a mean drawn from `base_seed`; the result is unit noise again. Blocks: `grid` along the short side, the long
side in proportion; `time_block` latent frames, counted from timeline position t0."""
if amount <= 0:
return x
C, T, h, w = x.shape
hb, wb = (grid, max(1, round(grid * w / h))) if h <= w else (max(1, round(grid * h / w)), grid)
yid, xid = _edges(h, hb), _edges(w, wb)
hb, wb = int(yid.max()) + 1, int(xid.max()) + 1
tid_abs = (t0 + torch.arange(T)) // max(1, int(time_block))
tids = torch.unique(tid_abs)
tloc = torch.searchsorted(tids, tid_abs)
nb = len(tids) * hb * wb
ids = ((tloc[:, None, None] * hb + yid[None, :, None]) * wb + xid[None, None, :]).reshape(-1)
flat = x.reshape(C, -1).double()
cnt = torch.zeros(nb, dtype=torch.float64).index_add_(0, ids, torch.ones_like(ids, dtype=torch.float64))
mean = torch.zeros(C, nb, dtype=torch.float64).index_add_(1, ids, flat) / cnt
z = torch.stack([torch.randn(C, hb, wb, generator=_gen("coarse", base_seed, int(t), hb, wb), dtype=torch.float64) for t in tids], 1).reshape(C, nb)
a = float(min(1.0, amount))
new = math.sqrt(1.0 - a * a) * mean + a * z / cnt.sqrt()
return (flat + (new - mean)[:, ids]).reshape(C, T, h, w).to(x.dtype)
DISTRIBUTIONS = ["gaussian", "laplacian", "student-t", "uniform"]
def draw(shape, kind: str, seed: int) -> torch.Tensor:
"""Unit-variance noise of another distribution (heavier or lighter tails than the gaussian the model was trained on)."""
g = _gen("dist", kind, seed)
if kind == "laplacian":
u = torch.rand(shape, generator=g, dtype=torch.float64) - 0.5
return (-torch.sign(u) * torch.log1p(-2 * u.abs().clamp(max=0.5 - 1e-12)) / math.sqrt(2.0)).float()
if kind == "student-t": # 5 degrees of freedom
n = torch.randn(shape, generator=g, dtype=torch.float64)
chi = torch.randn((5,) + tuple(shape), generator=g, dtype=torch.float64).pow(2).sum(0)
return (n / torch.sqrt(chi / 5.0) * math.sqrt(3.0 / 5.0)).float()
if kind == "uniform":
return ((torch.rand(shape, generator=g, dtype=torch.float64) - 0.5) * math.sqrt(12.0)).float()
return torch.randn(shape, generator=g)
def tilt_spectrum(x: torch.Tensor, alpha: float) -> torch.Tensor:
"""x [C,T,h,w]: power spectrum multiplied by 1/f^alpha over time and space (alpha > 0 = more energy in the large
blobs, 'fractal' noise; alpha < 0 = more in the fine grain), then back to zero mean and unit variance per channel."""
if alpha == 0:
return x
C, T, h, w = x.shape
f = torch.sqrt(torch.fft.fftfreq(T)[:, None, None] ** 2 + torch.fft.fftfreq(h)[None, :, None] ** 2 + torch.fft.rfftfreq(w)[None, None, :] ** 2)
f[0, 0, 0] = f[f > 0].min()
y = torch.fft.irfftn(torch.fft.rfftn(x.double(), dim=(1, 2, 3)) * f.pow(-alpha / 2.0), s=(T, h, w), dim=(1, 2, 3))
y = y - y.mean(dim=(1, 2, 3), keepdim=True)
return (y / y.std(dim=(1, 2, 3), keepdim=True).clamp_min(1e-8)).to(x.dtype)
def add_pyramid(x: torch.Tensor, amount: float, seed: int, levels: int = 4) -> torch.Tensor:
"""x [C,T,h,w] plus coarser copies of fresh noise (each level half the size, weight amount^level), unit variance."""
if amount <= 0:
return x
C, T, h, w = x.shape
y = x.double()
for i in range(1, levels + 1):
hh, ww = max(1, h >> i), max(1, w >> i)
if hh == 1 and ww == 1:
break
n = torch.randn(C, T, hh, ww, generator=_gen("pyramid", seed, i), dtype=torch.float64)
y = y + (amount ** i) * torch.nn.functional.interpolate(n, size=(h, w), mode="bilinear", align_corners=False)
y = y - y.mean(dim=(1, 2, 3), keepdim=True)
return (y / y.std(dim=(1, 2, 3), keepdim=True).clamp_min(1e-8)).to(x.dtype)
def mix(x: torch.Tensor, signal: torch.Tensor, weight: torch.Tensor) -> torch.Tensor:
weight = weight.clamp(0, 1)
return torch.sqrt(1.0 - weight ** 2) * x + weight * signal
class ShotNoise:
def __init__(self, seed, base_seed, opts, guide=None, tint=None, mask=None, t0=0):
self.seed, self.base_seed, self.o, self.guide, self.tint, self.mask, self.t0 = int(seed), int(base_seed), opts, guide, tint, mask, int(t0)
def generate_noise(self, input_latent):
samples = input_latent["samples"]
inds = input_latent["batch_index"] if "batch_index" in input_latent else None
noise = comfy.sample.prepare_noise(samples, self.seed, inds) # exactly RandomNoise
o = self.o
if not any(o[k] > 0 for k in ("shared_base", "static_share", "guide_structure", "ref_tint", "pyramid")) \
and o["spectrum_tilt"] == 0 and o["distribution"] == "gaussian":
return noise
nested = getattr(noise, "is_nested", False)
parts = list(noise.unbind()) if nested else [noise]
v = parts[0]
if v.ndim != 5:
return noise
out = []
for b in range(v.shape[0]):
x = v[b].float()
C, T, h, w = x.shape
if o["distribution"] != "gaussian":
x = draw((C, T, h, w), o["distribution"], self.seed + b)
x = add_pyramid(tilt_spectrum(x, o["spectrum_tilt"]), o["pyramid"], self.seed + b)
if o["static_share"] > 0:
s = torch.randn(C, 1, h, w, generator=_gen("static", self.base_seed, h, w))
x = mix(x, s.expand_as(x), torch.tensor(float(o["static_share"])))
x = replace_coarse(x, self.base_seed, o["shared_base"], o["base_grid"], o["time_block"], self.t0)
m = None
if self.mask is not None:
m = latent_mask(self.mask, T, h, w, grow=1)[0].to(x) # [1,T,h,w], 1 = the person
if o["guide_structure"] > 0 and self.guide is not None:
g = self.guide[min(b, self.guide.shape[0] - 1)].float().cpu()
if tuple(g.shape[1:]) != (T, h, w):
g = torch.nn.functional.interpolate(g[None], size=(T, h, w), mode="trilinear", align_corners=False)[0]
g = (g - g.mean(dim=(1, 2, 3), keepdim=True)) / g.std(dim=(1, 2, 3), keepdim=True).clamp_min(1e-6)
x = mix(x, g, float(o["guide_structure"]) * ((1.0 - m) if m is not None else torch.ones(1, T, h, w)))
if o["ref_tint"] > 0 and self.tint is not None:
x = mix(x, self.tint.view(C, 1, 1, 1).expand_as(x), float(o["ref_tint"]) * (m if m is not None else torch.ones(1, T, h, w)))
out.append(x.to(v.dtype))
parts[0] = torch.stack(out)
if nested:
return comfy.nested_tensor.NestedTensor(parts)
return parts[0]
class BFSShotNoise:
"""Starting noise for the sampler with optional structure (see the node description)."""
@classmethod
def INPUT_TYPES(cls):
f = lambda tip, mx=1.0: ("FLOAT", {"default": 0.0, "min": 0.0, "max": mx, "step": 0.01, "tooltip": tip})
return {
"required": {
"seed": ("INT", {"default": 0, "min": 0, "max": 0xffffffffffffffff, "control_after_generate": True,
"tooltip": "The seed (the base seed when a shot is connected)."}),
"shot_seed": (BFSShotSeed.MODES, {"default": BFSShotSeed.MODES[0], "tooltip":
"Only with a shot connected: the seed of each shot (same / counting up / unrelated but repeatable)."}),
"shared_base": f("0 = off (plain noise). The coarse pattern of the noise comes from base_seed instead of "
"the shot's seed: every shot, any length or size, agrees on it; the fine detail stays "
"per shot. 1 = fully shared. Try 0.5 - 1."),
"static_share": f("0 = off. A part of the noise that is identical in every frame (temporally "
"correlated noise). Try 0.1 - 0.3; too much freezes the motion.", 0.9),
"guide_structure": f("0 = off. Needs guide_frames + vae. Outside the mask the guide's latent is mixed "
"into the noise, so the background starts closer to the guide (without a mask: "
"everywhere, which also pulls toward the original person). Try 0.05 - 0.2.", 0.6),
"distribution": (DISTRIBUTIONS, {"default": "gaussian", "tooltip":
"gaussian = what the model was trained on. laplacian / student-t: more extreme values (stronger, "
"sparser kicks). uniform: none. All at unit variance. Like AdvancedNoise's types, but for H3's "
"video + audio latent (the audio noise always stays gaussian)."}),
"spectrum_tilt": ("FLOAT", {"default": 0.0, "min": -2.0, "max": 2.0, "step": 0.05, "tooltip":
"0 = off (white noise). > 0: more energy in the large blobs (fractal / pink noise, AdvancedNoise's "
"'fractal' alpha): the coarse layout, where the swap is decided, gets a harder push. < 0: more "
"fine grain. Try 0.2 - 0.6; large values burn colours."}),
"pyramid": f("0 = off. Adds coarser layers of noise (each half the size, weight = this value per "
"level): another way to put more energy in the large scales. Try 0.3 - 0.6.", 0.95),
"ref_tint": f("0 = off. Needs ref_image + vae. Inside the mask (everywhere without one) the noise "
"leans toward the reference's average latent, i.e. its colours. Try 0.05 - 0.15.", 0.6),
},
"optional": {
"shot": ("BFS_SHOT", {"tooltip": "From BFS Shot Planner / Unpack: per-shot seeds, and the shared base is "
"read at the shot's position on the timeline."}),
"base_seed": ("INT", {"default": -1, "min": -1, "max": 0xffffffffffffffff, "tooltip":
"Seed of the shared coarse pattern and of static_share. -1 = the seed above. Put here the seed of "
"a result that worked to keep its coarse noise while the detail changes with the seed."}),
"overrides": ("STRING", {"default": "", "tooltip": "Seeds for single shots: 'shot:seed, shot:seed' (shots count from 1)."}),
"base_grid": ("INT", {"default": 6, "min": 1, "max": 32, "advanced": True, "tooltip":
"Blocks of the coarse pattern along the short side (the long side in proportion). Fewer = only the "
"broadest layout is shared."}),
"time_block": ("INT", {"default": 2, "min": 1, "max": 32, "advanced": True, "tooltip": "Latent frames per block of the coarse pattern."}),
"vae": ("VAE", {"tooltip": "The video VAE, for guide_structure and ref_tint."}),
"guide_frames": ("IMAGE", {"tooltip": "The guide video frames (guide_structure)."}),
"ref_image": ("IMAGE", {"tooltip": "The reference picture (ref_tint)."}),
"mask": ("MASK", {"tooltip": "The person, one mask per frame (white = the person): guide_structure stays "
"outside it, ref_tint inside it."}),
},
}
RETURN_TYPES = ("NOISE", "INT")
RETURN_NAMES = ("noise", "seed")
OUTPUT_TOOLTIPS = ("For SamplerCustomAdvanced.", "The seed this shot used.")
FUNCTION = "make"
CATEGORY = "BFS/shot loop"
DESCRIPTION = ("Optional structured starting noise. With every amount at 0 it is exactly RandomNoise. shared_base: "
"the coarse noise pattern is the same for every shot / size / length (from base_seed), the detail "
"per shot. static_share: a part identical in all frames. guide_structure: the guide's latent mixed "
"in outside the mask. ref_tint: lean toward the reference's colours inside the mask. Experimental: "
"the model was trained on plain noise, so use small amounts and compare with 0.")
def make(self, seed, shot_seed, shared_base, static_share, guide_structure, ref_tint, distribution="gaussian", spectrum_tilt=0.0,
pyramid=0.0, shot=None, base_seed=-1, overrides="",
base_grid=6, time_block=2, vae=None, guide_frames=None, ref_image=None, mask=None):
index = int(shot.get("index", 0)) if shot else 0
s = _pick_seed(int(seed), index, shot_seed, overrides) if shot else int(seed)
base = int(seed) if int(base_seed) < 0 else int(base_seed)
guide = tint = None
if guide_structure > 0:
if vae is None or guide_frames is None:
raise ValueError("BFS Shot Noise: guide_structure needs vae and guide_frames")
guide = vae.encode(guide_frames[..., :3])
guide = guide.tensors[0] if getattr(guide, "is_nested", False) else guide
if ref_tint > 0:
if vae is None or ref_image is None:
raise ValueError("BFS Shot Noise: ref_tint needs vae and ref_image")
r = vae.encode(ref_image[:1, ..., :3])
r = (r.tensors[0] if getattr(r, "is_nested", False) else r).float().cpu()
tint = (r.mean(dim=(0, 2, 3, 4)) / r.std(dim=(0, 2, 3, 4)).clamp_min(1e-6)).clamp(-2, 2)
t0 = int(round(int(shot.get("start", 0)) / 4)) if shot else 0 # timeline position in latent frames
opts = {"shared_base": float(shared_base), "static_share": float(static_share), "guide_structure": float(guide_structure),
"ref_tint": float(ref_tint), "base_grid": int(base_grid), "time_block": int(time_block),
"distribution": distribution, "spectrum_tilt": float(spectrum_tilt), "pyramid": float(pyramid)}
return (ShotNoise(s, base, opts, guide, tint, mask, t0), s)
def sigmas_chart(sigmas: torch.Tensor, other: torch.Tensor | None = None, w: int = 640, h: int = 360):
"""(image [1,h,w,3], text): the noise level at the start of every step, with the zones where the steps are spent."""
import numpy as np
from PIL import Image, ImageDraw
s = [float(v) for v in sigmas.flatten().tolist()]
top = max(1e-6, s[0])
img = Image.new("RGB", (w, h), (24, 24, 28)); d = ImageDraw.Draw(img)
L, R, T, B = 52, w - 16, 34, h - 40
y = lambda v: B - (v / top) * (B - T)
zones = ((0.85, 1.01, (60, 40, 40), "layout / identity"), (0.5, 0.85, (48, 46, 36), "shapes"), (-0.01, 0.5, (36, 44, 52), "detail"))
for lo, hi, col, name in zones:
d.rectangle([L, y(min(hi, 1.0) * top), R, y(max(lo, 0.0) * top)], fill=col)
d.text((R - 6 - 6 * len(name), y(min(hi, 1.0) * top) + 3), name, fill=(150, 150, 150))
for v in (0.0, 0.25, 0.5, 0.75, 1.0):
d.line([L, y(v * top), R, y(v * top)], fill=(70, 70, 76)); d.text((8, y(v * top) - 6), f"{v * top:.2f}", fill=(170, 170, 170))
def line(vals, col, labels):
n = len(vals) - 1
pts = [(L + (R - L) * i / max(1, n), y(v)) for i, v in enumerate(vals)]
d.line(pts, fill=col, width=2)
for i, (px, py) in enumerate(pts):
d.ellipse([px - 3, py - 3, px + 3, py + 3], fill=col)
if labels and i < n:
d.text((px - 10, py - 16 if i % 2 == 0 else py + 6), f"{vals[i]:.2f}", fill=(235, 235, 235))
d.text((px - 3, B + 8), str(i + 1), fill=(170, 170, 170))
if other is not None:
line([float(v) for v in other.flatten().tolist()], (120, 120, 200), False)
line(s, (255, 190, 70), True)
steps = s[:-1]
cnt = [sum(lo * top < v <= hi * top or (hi > 1 and v >= lo * top) for v in steps) for lo, hi, _, _ in zones]
d.text((L, 10), f"{len(steps)} steps: {cnt[0]} on layout / identity (> {0.85 * top:.2f}), {cnt[1]} on shapes, {cnt[2]} on detail (< {0.5 * top:.2f})"
f" last jump {steps[-1]:.2f} -> {s[-1]:.2f}", fill=(235, 235, 235))
d.text((R - 24, B + 8), "step", fill=(170, 170, 170))
text = (f"{len(steps)} steps | layout/identity {cnt[0]}, shapes {cnt[1]}, detail {cnt[2]} | sigmas: " + ", ".join(f"{v:.3f}" for v in s))
return torch.from_numpy(np.asarray(img).astype("float32") / 255.0)[None], text
class BFSSigmasPreview:
"""Draws a sigma schedule: where the sampler's steps are spent (pass-through)."""
@classmethod
def INPUT_TYPES(cls):
return {"required": {"sigmas": ("SIGMAS", {"tooltip": "The schedule that goes to the sampler (after the scheduler and the shift)."})},
"optional": {"compare": ("SIGMAS", {"tooltip": "Another schedule drawn behind it, e.g. the same steps with another shift."})}}
RETURN_TYPES = ("SIGMAS", "IMAGE", "STRING")
RETURN_NAMES = ("sigmas", "chart", "text")
OUTPUT_TOOLTIPS = ("The same sigmas, untouched.", "The chart: connect a Preview Image.", "The values and the step count per zone.")
FUNCTION = "show"
CATEGORY = "BFS/shot loop"
DESCRIPTION = ("Shows a sigma schedule: the noise level at the start of each step and how many steps fall on layout / "
"identity (high noise), shapes and detail (low noise). With H3's video shift of 12 most steps sit at "
"the top; lowering the shift moves steps toward detail. Changes nothing: sigmas pass through.")
def show(self, sigmas, compare=None):
chart, text = sigmas_chart(sigmas, compare)
return (sigmas, chart, text)
NODE_CLASS_MAPPINGS = {"BFSShotNoise": BFSShotNoise, "BFSSigmasPreview": BFSSigmasPreview}
NODE_DISPLAY_NAME_MAPPINGS = {"BFSShotNoise": "BFS Shot Noise", "BFSSigmasPreview": "BFS Sigmas Preview"}
+86
View File
@@ -0,0 +1,86 @@
"""Optional blur of the source person in what the model SEES of the source (aligned guide, native reference video,
duet panel), keeping the mouth sharp: the model still reads the pose, the motion and the lips, but has no face to copy,
so the identity comes from the reference. JalenBrunson's trick for reliable ref2v swaps (Sapiens2 there); here the
masks come from SAM 3.1, already used by the planner. The kept area (inpaint) and the output are never blurred.
"""
from __future__ import annotations
import torch
SOURCE_BLUR = ["off", "face (keep mouth)", "head + hands (keep mouth)", "person (keep mouth)"]
_PARTS = {SOURCE_BLUR[1]: ("face",), SOURCE_BLUR[2]: ("head", "hand"), SOURCE_BLUR[3]: ("person",)}
_CACHE: dict = {}
def _sl():
try:
from . import bfs_shot_loop as SL
except ImportError:
import bfs_shot_loop as SL
return SL
def _small(frames: torch.Tensor, side: int = 640) -> torch.Tensor:
H, W = frames.shape[1:3]
s = side / max(H, W)
x = frames[..., :3].float()
if s >= 1:
return x
return torch.nn.functional.interpolate(x.movedim(-1, 1), size=(max(32, int(H * s) // 2 * 2), max(32, int(W * s) // 2 * 2)),
mode="bilinear", align_corners=False).movedim(1, -1)
def _grow(m: torch.Tensor, px: int) -> torch.Tensor:
if px <= 0:
return m
return torch.nn.functional.max_pool2d(m[:, None], 2 * px + 1, stride=1, padding=px)[:, 0]
def blur_mask(frames: torch.Tensor, mode: str, log: str = "") -> torch.Tensor | None:
"""[N,h,w] float mask (working size) of what to blur: the parts for `mode` minus the (slightly grown) mouth."""
parts = _PARTS.get(mode)
if not parts:
return None
SL = _sl()
small = _small(frames)
key = (mode, tuple(small.shape), SL._image_key(frames[:1]), SL._image_key(frames[-1:]),
SL._image_key(frames[frames.shape[0] // 2:frames.shape[0] // 2 + 1]))
if key in _CACHE:
return _CACHE[key]
m = torch.zeros(small.shape[:3])
for p in parts:
m = torch.maximum(m, (SL.segment_frames(small, dict(SL.DEFAULT_MASK, text=p, max_objects=8, threshold=0.4)) > 0.5).float())
mouth = (SL.segment_frames(small, dict(SL.DEFAULT_MASK, text="mouth", max_objects=8, threshold=0.3)) > 0.5).float()
keep = _grow(mouth, max(1, int(min(small.shape[1:3]) * 0.012)))
out = (_grow(m, max(1, int(min(small.shape[1:3]) * 0.015))) * (1 - keep)).clamp(0, 1)
if log:
print(f"{log}: source blur '{mode}': {float(m.mean()) * 100:.1f}% of the frame, mouth kept "
f"({'found' if float(mouth.sum()) > 0 else 'not found: the whole part is blurred'})", flush=True)
if len(_CACHE) > 8:
_CACHE.pop(next(iter(_CACHE)))
_CACHE[key] = out
return out
def blur_source(frames: torch.Tensor | None, mode: str, strength: float = 1.0, log: str = "") -> torch.Tensor | None:
"""The frames with the source person blurred per `mode` (mouth kept). `strength` 0-1 sets how coarse the blur is."""
if frames is None or not mode or mode == "off" or strength <= 0:
return frames
m = blur_mask(frames, mode, log)
if m is None or float(m.sum()) == 0:
return frames
x = frames[..., :3].float()
N, H, W = x.shape[:3]
# a heavy blur: down to ~1/12-1/40 of the size and back, then a soft pass; blocky detail never survives it
f = max(4, int(min(H, W) * (0.025 + 0.06 * float(strength))))
low = torch.nn.functional.interpolate(x.movedim(-1, 1), size=(max(2, H // f), max(2, W // f)), mode="area")
blur = torch.nn.functional.interpolate(low, size=(H, W), mode="bilinear", align_corners=False)
blur = torch.nn.functional.avg_pool2d(blur, 5, stride=1, padding=2).movedim(1, -1)
mm = torch.nn.functional.interpolate(m[:, None], size=(H, W), mode="bilinear", align_corners=False)
mm = torch.nn.functional.avg_pool2d(mm, 9, stride=1, padding=4)[:, 0, ..., None].clamp(0, 1)
if mm.shape[0] != N: # one mask per frame; a shorter list holds its last one
idx = [min(i, mm.shape[0] - 1) for i in range(N)]
mm = mm[idx]
out = x * (1 - mm) + blur * mm
del low, blur, mm
return out
+291
View File
@@ -0,0 +1,291 @@
"""Stable subject-crop planning, vendored from drozbay/MaskVidExperiments.
Source: https://github.com/drozbay/MaskVidExperiments (commit d98cc89, v0.2.0)
Copyright (c) drozbay. Licensed GPL-3.0, the same licence this pack carries,
which is what makes vendoring it legal here.
Vendored rather than imported so the head-swap sampler has no runtime
dependency on that pack being installed. Behaviour is unchanged; only the
ComfyUI node classes were dropped (they stay in the original pack, where they
belong) and the planner import was repointed.
Why this code and not a crop of my own: naive per-frame crops around a mask
jitter in position and size, and a video model reads that jitter as camera
motion. These boxes are planned over the whole clip, so they hold still through
mask noise and occlusion and follow only sustained movement.
"""
"""Subject-tracked batch crop/uncrop for video mask workflows. The crop
boxes come from planner.py, which plans position, size, and shape jointly
over the whole clip."""
import logging
import math
import numpy as np
import torch
from scipy import ndimage
import comfy.utils
from . import bfs_crop_planner as planner
CATEGORY = "MaskVidExperiments"
def _clean_mask_stack(masks, value_thresh, min_pixels, min_frames):
"""Binarize a mask batch and drop noise blobs.
Connected components are labelled in 3D (time + space). A blob survives if
its peak single-frame area clears min_pixels and it is either subject-sized
(peak >= 50% of the largest blob, so fast motion that splits the subject
into per-frame components doesn't delete it) or persists min_frames frames.
"""
binary = masks.detach().cpu().numpy() > value_thresh
if not binary.any():
return binary
labels, count = ndimage.label(binary, structure=np.ones((3, 3, 3), dtype=bool))
if count == 0:
return binary
span = np.zeros(count + 1, dtype=np.int64)
for label, location in enumerate(ndimage.find_objects(labels), start=1):
if location is not None:
span[label] = location[0].stop - location[0].start
peak = np.zeros(count + 1, dtype=np.int64)
for frame in labels:
peak = np.maximum(peak, np.bincount(frame.ravel(), minlength=count + 1))
span[0] = 0
peak[0] = 0
subject = peak >= 0.5 * peak.max() if peak.max() > 0 else np.zeros_like(peak, dtype=bool)
keep = (peak >= min_pixels) & (subject | (span >= min_frames))
keep[0] = False
return keep[labels]
def _open_reconstruct(masks, value_thresh, radius, min_frames):
"""Morphological opening by reconstruction on a mask batch.
Erodes each frame spatially so blobs thinner than ~2*radius vanish, then
instead of dilating back, keeps the original shape of every blob with a
surviving core, so the subject's silhouette (thin protrusions included) is
untouched. Blobs are 3D components (time + space): a core surviving in any
frame keeps the whole track. border_value=1 so subjects cut off by the
frame edge are not eroded from that side.
"""
binary = masks.detach().cpu().numpy() > value_thresh
if not binary.any():
return binary
spatial = np.zeros((3, 3, 3), dtype=bool)
spatial[1] = [[0, 1, 0], [1, 1, 1], [0, 1, 0]]
eroded = ndimage.binary_erosion(binary, structure=spatial, iterations=radius, border_value=1)
labels, count = ndimage.label(binary, structure=np.ones((3, 3, 3), dtype=bool))
if count == 0:
return binary
keep = np.zeros(count + 1, dtype=bool)
keep[np.unique(labels[eroded])] = True
if min_frames > 1:
for label, location in enumerate(ndimage.find_objects(labels), start=1):
if location is not None and location[0].stop - location[0].start < min_frames:
keep[label] = False
keep[0] = False
return keep[labels]
def _resize_image(img_hwc, w, h, method="lanczos"):
if img_hwc.shape[0] == h and img_hwc.shape[1] == w:
return img_hwc
return comfy.utils.common_upscale(
img_hwc.movedim(-1, 0).unsqueeze(0), w, h, method, "disabled"
).squeeze(0).movedim(0, -1)
def _resize_mask(mask_hw, w, h, method="bilinear"):
if mask_hw.shape[0] == h and mask_hw.shape[1] == w:
return mask_hw
return comfy.utils.common_upscale(
mask_hw[None, None], w, h, method, "disabled"
)[0, 0]
def _upscale_size(w, h, megapixels, divisible_by):
"""Size scaling a w x h crop to about abs(megapixels) pixels on the
divisible_by grid. A positive value is a floor: None when already at or
above it. A negative value is a target, reached by downscaling too."""
scale = math.sqrt(abs(megapixels) * 1024 * 1024 / (w * h))
if megapixels > 0 and scale <= 1.0:
return None
return (max(1, round(w * scale / divisible_by)) * divisible_by,
max(1, round(h * scale / divisible_by)) * divisible_by)
def _cell_params(mode):
"""Raw planner dials for the standard node's (padding, prefer) cell.
The advanced node feeds the same dials directly. guaranteed floors the
full promise against every raw frame (floor window 1): a true
guarantee, so mask noise must be cleaned upstream. firm floors 70%
against the sustained tracks; flexible has no floor. In tracked mode
the crop never rescales, so prefer differentiates through the oversize
rent instead: stillness holds a bigger box to move less. In zoomed
mode the rent is never scaled by prefer (cheapening it tips whole
clips into one constant maximum size, killing zoom-follow); prefer
differentiates through the resize cost."""
sel = mode["mode"]
pad_level = mode.get("padding", "firm")
prefer = mode.get("prefer", "stillness")
floor = {"guaranteed": 1.0, "firm": 0.7, "flexible": 0.0}[pad_level]
still = prefer == "stillness"
if sel == "tracked":
oversize = 32.0 if still else 8.0
else:
oversize = float(mode.get("pad_surplus_tol", 16))
return {
"crop_scale": mode["crop_scale"],
"min_padding_allowed": floor,
"min_padding_allowed_window": 1 if pad_level == "guaranteed" else 16,
"pad_deficit_tol": 16.0,
"pad_surplus_tol": oversize,
"resize_cost": 2.0 if still else 1.0,
"movement_cost": 1.0,
"center_pull": 1e-4,
"end_tightening": 0.0,
"end_tightening_window": 80 if floor > 0 else 0,
"zoom_step": mode.get("zoom_step", 1.0),
"max_zoom_rate": 0.0,
"aspect_ratio": mode["aspect_ratio"],
"seamless_loop": mode.get("seamless_loop", False),
}
def _debug_summary(binary, boxes, scale, sel, info, output_size):
"""Brief per-run report: what shape the planner chose and how well the
plan delivered the padding promise."""
n = len(boxes)
worst = []
img_h, img_w = binary.shape[1:]
for i in range(n):
ys, xs = binary[i].nonzero()
if len(xs) == 0:
worst.append(np.nan)
continue
b = boxes[i]
bx1 = b["x"] + b["width"] - 1
by1 = b["y"] + b["height"] - 1
wx = (scale - 1) / 2 * (xs.max() - xs.min() + 1)
wy = (scale - 1) / 2 * (ys.max() - ys.min() + 1)
fr = []
if b["x"] > 0: fr.append((xs.min() - b["x"]) / max(wx, 1e-6))
if bx1 < img_w - 1: fr.append((bx1 - xs.max()) / max(wx, 1e-6))
if b["y"] > 0: fr.append((ys.min() - b["y"]) / max(wy, 1e-6))
if by1 < img_h - 1: fr.append((by1 - ys.max()) / max(wy, 1e-6))
worst.append(min(fr) if fr else np.nan)
worst = np.array(worst)
v = worst[~np.isnan(worst)]
cx = np.array([b["x"] + b["width"] / 2 for b in boxes])
cy = np.array([b["y"] + b["height"] / 2 for b in boxes])
travel = np.hypot(np.diff(cx), np.diff(cy)).sum()
ar = info["aspect"]
if info.get("swept"):
shape = f"{ar:.2f} (chosen from {info['candidates']} candidates)"
elif sel == "zoomed":
shape = f"{ar:.2f} (manual)"
else:
shape = f"{ar:.2f}"
ws = [b["width"] for b in boxes]
hs = [b["height"] for b in boxes]
lines = [
f"mode: {sel}",
f"frames: {n} ({img_w}x{img_h} source)",
f"aspect_ratio: {shape}",
f"smallest_box: {min(ws)}x{min(hs)}",
f"largest_box: {max(ws)}x{max(hs)}",
]
if sel == "zoomed":
lines.append(f"zoom_ratio: {max(hs) / max(min(hs), 1):.2f} "
"(largest box / smallest box)")
lines.append(f"crop region resizing: {max(ws)} x {max(hs)} -> "
f"{output_size[0]} x {output_size[1]}")
lines.append(f"movement: {travel:.0f}px "
"(total distance the box center travels)")
if len(v) and scale > 1.0:
lines.append(f"padding_promised: {(scale - 1) / 2:.0%} of the "
f"subject's size on each side (crop_scale {scale:.2f})")
lines.append(f"padding_worst: {max(v.min(), 0):.0%} of promised")
lines.append(f"padding_typical: {max(np.median(v), 0):.0%} of promised")
return "\n".join(lines)
def _plan_and_crop(original_images, masks, sel, p, divisible_by,
mask_threshold, upscale_megapixels=0.0):
if original_images.shape[0] != masks.shape[0]:
raise ValueError(f"original_images ({original_images.shape[0]}) and masks ({masks.shape[0]}) must have the same frame count")
if original_images.shape[1:3] != masks.shape[1:3]:
raise ValueError(f"original_images ({original_images.shape[2]}x{original_images.shape[1]}) and masks ({masks.shape[2]}x{masks.shape[1]}) must have the same dimensions")
if 0.0 < p["crop_scale"] < 1.0:
raise ValueError("crop_scale must be 0 (full frame) or at least 1.0")
full_frame = p["crop_scale"] <= 0.0
binary = masks.detach().cpu().numpy() > mask_threshold
if full_frame:
img_h, img_w = masks.shape[1], masks.shape[2]
boxes = [{"x": 0, "y": 0, "width": img_w, "height": img_h}
for _ in range(masks.shape[0])]
info = {"aspect": img_w / img_h}
else:
if not binary.any():
raise ValueError("all masks are empty, nothing to crop")
boxes, info = planner.plan(binary, sel, p, divisible_by)
bw = max(b["width"] for b in boxes)
bh = max(b["height"] for b in boxes)
size = None
img_method, mask_method = "lanczos", "bilinear"
if full_frame:
gw = math.ceil(bw / divisible_by) * divisible_by
gh = math.ceil(bh / divisible_by) * divisible_by
if (gw, gh) != (bw, bh):
size = (gw, gh)
elif sel == "zoomed":
# Target sized to the largest planned box: that stretch is ~1:1
# and everything else upscales rather than losing detail.
th = math.ceil(bh / divisible_by) * divisible_by
size = (math.ceil(th * info["aspect"] / divisible_by) * divisible_by, th)
if upscale_megapixels != 0:
up = _upscale_size(*(size or (bw, bh)), upscale_megapixels, divisible_by)
if up is not None:
size = up
img_method, mask_method = "bicubic", "nearest-exact"
debug = _debug_summary(binary, boxes, p["crop_scale"],
f"{sel} (full frame)" if full_frame else sel,
info, size or (bw, bh))
if size is not None:
tw, th = size
cropped_images = torch.stack([
_resize_image(original_images[i, b["y"]:b["y"] + b["height"], b["x"]:b["x"] + b["width"], :], tw, th, img_method)
for i, b in enumerate(boxes)
])
cropped_masks = torch.stack([
_resize_mask(masks[i, b["y"]:b["y"] + b["height"], b["x"]:b["x"] + b["width"]], tw, th, mask_method)
for i, b in enumerate(boxes)
])
else:
cropped_images = torch.stack([
original_images[i, b["y"]:b["y"] + b["height"], b["x"]:b["x"] + b["width"], :]
for i, b in enumerate(boxes)
])
cropped_masks = torch.stack([
masks[i, b["y"]:b["y"] + b["height"], b["x"]:b["x"] + b["width"]]
for i, b in enumerate(boxes)
])
return (cropped_images, cropped_masks,
[[b] for b in boxes], debug)
+102
View File
@@ -0,0 +1,102 @@
"""Color Mask -- recolors a plain white/binary mask (e.g. straight out of SAM2) into the
colored-silhouette or flat-color-marker convention scail2v2 was trained on.
Two styles, matching the two different mask roles in the trained recipe:
- "silhouette": recolor white->color, keep the full shape. Use for the GUIDE mask slot
(LTX Multiple Controls' mask_video) -- its shape is what tells the model WHERE/WHAT POSE
to place the identity, so the shape must be preserved.
- "flat_marker": recolor AND collapse the whole mask down to a small solid-color dot at its
centroid, discarding the shape entirely. Use for the IDENTITY mask slot (identity_mask_
image) -- a full body-shaped silhouette there re-introduces a competing pose signal (this
is the exact bug flatten_ref_masks.py fixes on the training-data side: the reference's OWN
held pose bled into the output through a shaped identity mask). The color is the only
thing that matters there (it's what teaches "this color <-> this identity" correspondence
for multi-character replacement), not the shape.
"""
import numpy as np
import torch
_COLOR_PRESETS = {
"blue": (0, 0, 255),
"red": (255, 0, 0),
"green": (0, 255, 0),
"yellow": (255, 255, 0),
"cyan": (0, 255, 255),
"magenta": (255, 0, 255),
"custom": None,
}
class LTXColorMask:
@classmethod
def INPUT_TYPES(cls):
return {"required": {
"mask": ("IMAGE", {"tooltip": "Plain white/binary mask (e.g. from SAM2), one or more frames. "
"Any non-black pixel counts as 'inside' the mask, regardless of exact shade."}),
"color": (list(_COLOR_PRESETS.keys()), {"default": "blue",
"tooltip": "Preset color to recolor the mask to (matches scail2v2's per-person palette: "
"blue=person 0, red=person 1, x-sorted left-to-right). Pick 'custom' to use the "
"custom_r/g/b inputs instead."}),
"style": (["silhouette", "flat_marker"], {"default": "silhouette",
"tooltip": "silhouette: recolor, keep the full shape -- use for the GUIDE mask slot "
"(mask_video). flat_marker: recolor AND collapse to a small dot at the mask's "
"centroid -- use for the IDENTITY mask slot (identity_mask_image); a shaped "
"mask there re-injects a competing pose signal (the ref's own held pose)."}),
"marker_radius_frac": ("FLOAT", {"default": 0.06, "min": 0.01, "max": 0.3, "step": 0.01,
"tooltip": "flat_marker only: dot radius as a fraction of min(H, W). Matches the "
"training-side default (flatten_ref_masks.py)."}),
}, "optional": {
"custom_r": ("INT", {"default": 255, "min": 0, "max": 255}),
"custom_g": ("INT", {"default": 0, "min": 0, "max": 255}),
"custom_b": ("INT", {"default": 0, "min": 0, "max": 255}),
"background_black": ("BOOLEAN", {"default": True,
"tooltip": "On: output background is black (matches guide_mask's white-bg-colored-"
"shape convention actually being colored-on-black internally once VAE-"
"encoded -- for identity_mask specifically, training used a black bg). "
"Off: keep the mask's original background color where nothing was "
"recolored (rare; only if your source mask isn't black/white)."}),
}}
RETURN_TYPES = ("IMAGE",)
RETURN_NAMES = ("colored_mask",)
FUNCTION = "apply"
CATEGORY = "LTX/identity"
DESCRIPTION = ("Recolors a plain white/binary SAM2-style mask into scail2v2's trained convention. "
"'silhouette' for the guide mask slot, 'flat_marker' (small color dot, no shape) for the "
"identity mask slot -- see each input's tooltip for why the two differ.")
def apply(self, mask, color, style, marker_radius_frac,
custom_r=255, custom_g=0, custom_b=0, background_black=True):
if color == "custom":
rgb = (int(custom_r), int(custom_g), int(custom_b))
else:
rgb = _COLOR_PRESETS[color]
arr = (mask.clamp(0, 1).cpu().numpy() * 255).astype(np.uint8) # [N,H,W,C]
n, h, w = arr.shape[0], arr.shape[1], arr.shape[2]
color_arr = np.array(rgb, dtype=np.uint8)
out = np.zeros((n, h, w, 3), dtype=np.uint8)
for i in range(n):
frame = arr[i]
inside = frame[..., :3].sum(axis=-1) > 10 # non-black = inside the mask
if not inside.any():
continue
if style == "silhouette":
out[i][inside] = color_arr
if not background_black:
out[i][~inside] = frame[..., :3][~inside]
else: # flat_marker
ys, xs = np.where(inside)
cy, cx = int(ys.mean()), int(xs.mean())
radius = max(2, int(marker_radius_frac * min(h, w)))
yy, xx = np.ogrid[:h, :w]
dot = (yy - cy) ** 2 + (xx - cx) ** 2 <= radius ** 2
out[i][dot] = color_arr
out_t = torch.from_numpy(out.astype(np.float32) / 255.0)
return (out_t,)
NODE_CLASS_MAPPINGS = {"LTXColorMask": LTXColorMask}
NODE_DISPLAY_NAME_MAPPINGS = {"LTXColorMask": "LTX Color Mask"}
+88
View File
@@ -0,0 +1,88 @@
"""Corrige a saida do VLM para o formato que o LoRA foi treinado.
Duas regras que o VLM erra de forma consistente, por mais explicito que o prompt seja, e que
sao deterministicas -- quem chama ja sabe se anexou referencia:
sem referencia -> a frase-marcador nao pode existir, e "trocar" e um Replace simples.
O VLM tende a emitir Add + marcador + Remove, que e a forma da referencia.
com referencia -> o marcador so pode seguir uma frase que comeca com Add. Se vier num
Replace, vira Add + Remove.
"""
import re
MARCADOR = "The object to add is shown in the reference image."
TRIGGER = "edit_anything: "
def _frases(texto):
corpo = texto[len(TRIGGER):] if texto.startswith(TRIGGER) else texto
return [f.strip() for f in re.split(r"(?<=\.)\s+", corpo.strip()) if f.strip()]
def _juntar(frases):
return TRIGGER + " ".join(frases)
def corrigir(prompt, tem_referencia):
frases = _frases(prompt)
mudou = []
if not tem_referencia:
antes = len(frases)
frases = [f for f in frases if f.rstrip() != MARCADOR]
if len(frases) != antes:
mudou.append("marcador removido (sem referencia)")
# Add(X) ... Remove(Y) sem referencia = forma da referencia aplicada no lugar errado.
# Recompoe como Replace Y with X.
i_add = next((i for i, f in enumerate(frases) if f.startswith("Add ")), None)
i_rem = next((i for i, f in enumerate(frases) if f.startswith("Remove ")), None)
if i_add is not None and i_rem is not None:
novo = re.sub(r"^Add\s+", "", frases[i_add]).rstrip(".")
velho = re.sub(r"^Remove\s+", "", frases[i_rem]).rstrip(".")
# descarta localizacao redundante herdada do Add
novo = re.sub(r"\s+(where|parked|standing|placed)\b.*$", "", novo).strip()
frases = [f for k, f in enumerate(frases) if k not in (i_add, i_rem)]
frases.insert(min(i_add, i_rem), f"Replace {velho} with {novo}.")
mudou.append("Add+Remove recomposto como Replace")
else:
for i, f in enumerate(frases):
if f.rstrip() == MARCADOR and i > 0 and not frases[i - 1].startswith("Add "):
anterior = frases[i - 1]
m = re.match(r"^Replace\s+(.*?)\s+with\s+(.*)\.$", anterior)
if m:
velho, novo = m.group(1), m.group(2)
frases[i - 1] = f"Add {novo}."
frases.insert(i + 1, f"Remove {velho}.")
mudou.append("Replace com marcador virou Add + Remove")
break
return _juntar(frases), mudou
class EditAnythingPromptFixer:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"prompt": ("STRING", {"multiline": True, "forceInput": True}),
"has_reference": ("BOOLEAN", {"default": False, "tooltip": "Ligue quando uma imagem de referencia for anexada. Este e o mesmo booleano que decide anexar a imagem."}),
}
}
RETURN_TYPES = ("STRING",)
RETURN_NAMES = ("prompt",)
FUNCTION = "fix"
CATEGORY = "BFS/EditAnything"
def fix(self, prompt, has_reference):
saida, mudou = corrigir(prompt, has_reference)
if mudou:
print(f"[BFSNodes] prompt corrigido: {', '.join(mudou)}")
print(f"[BFSNodes] {saida}")
return (saida,)
NODE_CLASS_MAPPINGS = {"EditAnythingPromptFixer": EditAnythingPromptFixer}
NODE_DISPLAY_NAME_MAPPINGS = {"EditAnythingPromptFixer": "Edit Anything Prompt Fixer (BFS)"}
+181
View File
@@ -0,0 +1,181 @@
{
"1": {
"class_type": "UNETLoader",
"inputs": {
"unet_name": "minimax_h3_ref2va_pruned_int8_convrot.safetensors",
"weight_dtype": "default"
}
},
"2": {
"class_type": "LoraLoaderModelOnly",
"inputs": {
"model": [
"1",
0
],
"lora_name": "YOUR_H3_UPSCALE_LORA.safetensors",
"strength_model": 1.0
}
},
"3": {
"class_type": "CLIPLoader",
"inputs": {
"clip_name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors",
"type": "minimax",
"device": "default"
}
},
"4": {
"class_type": "CLIPTextEncode",
"inputs": {
"clip": [
"3",
0
],
"text": "h3upscale, increase resolution and restore fine details while preserving the original content and visual style."
}
},
"5": {
"class_type": "VAELoader",
"inputs": {
"vae_name": "minimax_h3_video_vae_fp16.safetensors"
}
},
"6": {
"class_type": "BFSMiniMaxH3GuideTarget",
"inputs": {
"width": 1024,
"height": 768,
"frames": 1
}
},
"7": {
"class_type": "LoadImage",
"inputs": {
"image": "YOUR_INPUT_IMAGE.png"
}
},
"8": {
"class_type": "BFSMiniMaxH3DownscaledGuide",
"inputs": {
"model": [
"2",
0
],
"positive": [
"4",
0
],
"vae": [
"5",
0
],
"latent": [
"6",
0
],
"image": [
"7",
0
],
"downscale_factor": 4,
"frame_idx": 0,
"lora_name": "YOUR_H3_UPSCALE_LORA.safetensors"
}
},
"9": {
"class_type": "BasicGuider",
"inputs": {
"model": [
"8",
0
],
"conditioning": [
"8",
1
]
}
},
"10": {
"class_type": "RandomNoise",
"inputs": {
"noise_seed": 42
}
},
"11": {
"class_type": "KSamplerSelect",
"inputs": {
"sampler_name": "euler"
}
},
"12": {
"class_type": "BasicScheduler",
"inputs": {
"model": [
"8",
0
],
"scheduler": "simple",
"steps": 28,
"denoise": 1.0
}
},
"13": {
"class_type": "SamplerCustomAdvanced",
"inputs": {
"noise": [
"10",
0
],
"guider": [
"9",
0
],
"sampler": [
"11",
0
],
"sigmas": [
"12",
0
],
"latent_image": [
"6",
0
]
}
},
"14": {
"class_type": "VAEDecode",
"inputs": {
"samples": [
"13",
0
],
"vae": [
"5",
0
]
}
},
"15": {
"class_type": "SaveImage",
"inputs": {
"images": [
"14",
0
],
"filename_prefix": "BFS_H3_Upscale"
}
},
"16": {
"class_type": "SaveImage",
"inputs": {
"images": [
"8",
2
],
"filename_prefix": "BFS_H3_EffectiveGuide"
}
}
}
File diff suppressed because it is too large Load Diff
+128
View File
@@ -0,0 +1,128 @@
"""LTX Identity CAN (AdaLN) — apply the trained Conditioned Adaptive Normalization at inference.
The ArcFace projector (appended text tokens) has ~0 impact (cross-attn can ignore appended tokens).
CAN instead modulates the self-attention AdaLN (shift + gate) of the even blocks with the reference
ArcFace — a channel the model can't ignore. Trained on top of the reference LoRA it lifts identity
a lot (humo 0.52 -> 0.69) with no first-frame leak.
Key trick: the CAN delta depends only on the (constant) reference ArcFace, and the block AdaLN is
`scale_shift_table[row] + timestep`. So instead of patching the (version-specific, AV) block
forward, we just add the delta into `scale_shift_table` (row 0 = shift_msa, row 2 = gate_msa) via
ComfyUI's reversible add_object_patch — mathematically identical, works on any LTX/LTXAV version.
"""
from __future__ import annotations
import numpy as np
import torch
from torch import nn
from safetensors.torch import load_file
class CANModulation(nn.Module):
"""Exact replica of ltx-core CANModulation: id_global[512] -> (dshift, dgate)[dim]."""
def __init__(self, id_dim: int, dim: int, hidden: int = 512):
super().__init__()
self.norm = nn.LayerNorm(id_dim)
self.mlp = nn.Sequential(nn.Linear(id_dim, hidden), nn.SiLU(), nn.Linear(hidden, 3 * dim))
self.dim = dim
def forward(self, id_global):
d = self.mlp(self.norm(id_global))
dshift, _dscale, dgate = d.chunk(3, dim=-1)
return dshift, dgate
def _find_diffusion_model(mp):
m = getattr(mp, "model", mp)
return getattr(m, "diffusion_model", m)
def apply_can_to_model(model_patcher, reference_image, can_weights_name, strength=1.0,
arcface_mode="auto_adjust"):
"""Add the trained CAN identity modulation to a ModelPatcher (in place, reversible).
Returns the number of blocks modulated."""
import folder_paths
from .ltx_identity_overlap import _arcface_embed
path = folder_paths.get_full_path("loras", can_weights_name)
sd = load_file(path)
idxs = sorted({int(k.split(".")[1]) for k in sd if k.startswith("can.")})
if not idxs:
print("[BFS CAN] no can.* weights in file — nothing to apply.")
return 0
face = _arcface_embed(reference_image, arcface_mode) if _arcface_takes_mode() else _arcface_embed(reference_image)
if face is None:
print("[BFS CAN] no face detected in reference — CAN skipped.")
return 0
id_global = torch.as_tensor(np.asarray(face), dtype=torch.float32).view(1, -1) # [1,512]
dm = _find_diffusion_model(model_patcher)
blocks = list(dm.transformer_blocks)
even_blocks = blocks[::2] # CAN was trained on even blocks (0,2,4,…) -> saved can.0,can.1,…
n = min(len(idxs), len(even_blocks))
applied = 0
for j in range(n):
# find this even block's index in the full list (for the dotted patch name)
blk = even_blocks[j]
block_index = j * 2
sst = blk.scale_shift_table
if sst.shape[0] < 3:
continue
dim = sst.shape[1]
dev = sst.device
can = CANModulation(id_dim=id_global.shape[1], dim=dim)
can.load_state_dict({k[len(f"can.{idxs[j]}."):]: v for k, v in sd.items()
if k.startswith(f"can.{idxs[j]}.")})
can = can.to(dev, torch.float32).eval()
with torch.no_grad():
dshift, dgate = can(id_global.to(dev, torch.float32)) # [1, dim], on the model device
new_sst = sst.detach().clone().float()
new_sst[0] = new_sst[0] + strength * dshift[0] # shift_msa
new_sst[2] = new_sst[2] + strength * torch.tanh(dgate)[0] # gate_msa
new_param = nn.Parameter(new_sst.to(sst.dtype), requires_grad=False)
name = f"diffusion_model.transformer_blocks.{block_index}.scale_shift_table"
model_patcher.add_object_patch(name, new_param)
applied += 1
print(f"[BFS CAN] applied CAN (AdaLN) to {applied} even blocks (strength {strength}).")
return applied
def _arcface_takes_mode():
import inspect
from .ltx_identity_overlap import _arcface_embed
try:
return "mode" in inspect.signature(_arcface_embed).parameters
except Exception:
return False
class LTXIdentityCAN:
@classmethod
def INPUT_TYPES(cls):
import folder_paths
adapters = ["None"] + [f for f in folder_paths.get_filename_list("loras") if "identity_adapters" in f]
return {
"required": {
"model": ("MODEL",),
"reference_image": ("IMAGE",),
"can_weights": (adapters,),
"strength": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 2.0, "step": 0.05}),
}
}
RETURN_TYPES = ("MODEL",)
FUNCTION = "apply"
CATEGORY = "BFS/LTX Identity"
def apply(self, model, reference_image, can_weights, strength):
if can_weights == "None":
return (model,)
m = model.clone()
apply_can_to_model(m, reference_image, can_weights, strength)
return (m,)
NODE_CLASS_MAPPINGS = {"LTXIdentityCAN": LTXIdentityCAN}
NODE_DISPLAY_NAME_MAPPINGS = {"LTXIdentityCAN": "LTX Identity CAN / AdaLN"}
+48
View File
@@ -0,0 +1,48 @@
"""LTX Identity — Gemma-Vision Conditioning (MagicMirror-Caminho2).
The identity breakthrough for LTX: the model conditions in Gemma's embedding space, so foreign
encoders (CLIP/DINO/ArcFace adapters) fail. Instead we feed the REFERENCE IMAGE through LTX's OWN
multimodal Gemma 3 (its text encoder is Gemma 3 with a vision tower), so the reference becomes
conditioning tokens IN THE NATIVE SPACE — the model reads the person like text. No reference
latent (no copy-paste/mask), no foreign encoder, no DiT surgery. Pair with a LoRA trained the
same way (reference image -> Gemma-vision conditioning + ArcFace loss).
Everything comes from the CLIP already loaded in the workflow: ComfyUI's native LTX Gemma-3
tokenizer already accepts an image (clip.tokenize(prompt, image=...) inserts <image_soft_token>
and carries the pixels), and clip.encode_from_tokens runs the multimodal Gemma + projection +
connector. So this node is just: tokenize(prompt, image) -> encode -> CONDITIONING.
"""
from __future__ import annotations
class LTXIdentityGemmaVision:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"clip": ("CLIP",),
"reference_image": ("IMAGE",),
"prompt": ("STRING", {"multiline": True, "default": "ref_t2v: a person in a room, medium shot."}),
}
}
RETURN_TYPES = ("CONDITIONING",)
RETURN_NAMES = ("conditioning",)
FUNCTION = "encode"
CATEGORY = "BFS/LTX Identity"
def encode(self, clip, reference_image, prompt):
# The LTX Gemma-3 tokenizer takes the ComfyUI IMAGE directly (movedim + <image_soft_token>),
# and encode runs the multimodal Gemma. Identity ends up in the native conditioning space.
try:
tokens = clip.tokenize(prompt, image=reference_image)
except TypeError:
# Older signatures may name it differently; fall back to positional image.
tokens = clip.tokenize(prompt, reference_image)
cond = clip.encode_from_tokens_scheduled(tokens)
print("[BFS Gemma-Vision] multimodal identity CONDITIONING ready (reference in native space).")
return (cond,)
NODE_CLASS_MAPPINGS = {"LTXIdentityGemmaVision": LTXIdentityGemmaVision}
NODE_DISPLAY_NAME_MAPPINGS = {"LTXIdentityGemmaVision": "LTX Identity Gemma-Vision (Caminho2)"}
+393
View File
@@ -0,0 +1,393 @@
"""LTX-2.3 Identity Transfer — MULTIPLE ANGLES (temporary tester).
A copy of "LTX Identity Transfer" (overlap + source_phase, ltx_identity_overlap.py) that accepts
several reference views, each injected as its OWN overlap segment tagged with its OWN source_id —
exactly the multi-view training convention:
reference_face_front -> source_id 2 (REQUIRED; also drives the ArcFace projector)
reference_back_head -> source_id 3 (optional — back of the head / nape, hair)
reference_body_front -> source_id 4 (optional — full-body front, build/proportions)
reference_side_profile -> source_id 5 (optional — side / profile of the face)
Empty optional inputs are simply omitted — matching the training view-dropout, so the model
degrades gracefully when you only provide the face. Non-frontal views feed only the overlap
latent (no ArcFace); the frontal face additionally drives the IdentityProjector tokens.
Load the matching multi-view LoRA on MODEL first. Reuses the exact ArcFace/projector/patch
machinery from ltx_identity_overlap; only the model patch is generalized to N reference blocks.
"""
import logging
import os
import types
import torch
from .ltx_identity_overlap import (
_append_ctx_tokens,
_arcface_embed,
_find_ltxv,
_load_projector,
)
from .reference_temporal_offset import (
DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
reference_temporal_offset_input,
reference_temporal_pixel_offset,
shift_reference_temporal_positions,
)
log = logging.getLogger("LTXIdentityMultiAngle")
# Fixed viewpoint -> source_id convention (must match the multi-view training config).
_SLOT_SOURCE_ID = {"face": 2.0, "back": 3.0, "body": 4.0, "side": 5.0}
_DEBUG = os.environ.get("LTX_IDOVERLAP_DEBUG", "0") == "1"
def _dbg(*a):
if _DEBUG:
print("[LTXIdMulti] " + " ".join(str(x) for x in a), flush=True)
def _rotate_block(pe, start, length, seg, theta=10000.0):
"""Rotate RoPE freqs of tokens [start:start+length] by phase = seg * theta^(-d/L).
Same math as ltx_identity_overlap._rotate_ref_freqs, but on an arbitrary token slice
(each reference block sits at its own offset after the target)."""
if length <= 0 or seg == 0.0:
return pe
cos, sin = pe[0], pe[1]
rest = tuple(pe[2:])
L = cos.shape[-1]
d = torch.arange(L, device=cos.device, dtype=torch.float32)
rate = theta ** (-d / float(L))
phase = seg * rate
pc = phase.cos().to(cos.dtype)
ps = phase.sin().to(sin.dtype)
idx = [slice(None)] * cos.dim()
idx[-2] = slice(start, start + length)
idx = tuple(idx)
c0, s0 = cos[idx], sin[idx]
cos = cos.clone(); sin = sin.clone()
cos[idx] = c0 * pc - s0 * ps
sin[idx] = s0 * pc + c0 * ps
_dbg("rotate block start", start, "len", length, "seg", seg)
return (cos, sin, *rest)
def _install_multi_patches(ltxv):
"""Idempotent multi-reference version of the overlap patch. Appends a LIST of reference
latents (each with its own source_phase segment) after the target tokens, and trims them
all before unpatchify. Uses distinct attribute/flag names so it never collides with the
single-ref LTXIdentityTransfer patch."""
if getattr(ltxv, "_id_multi_patched", False):
return
orig_process_input = ltxv._process_input
orig_prepare_ts = ltxv._prepare_timestep
orig_prepare_pe = ltxv._prepare_positional_embeddings
orig_process_output = ltxv._process_output
def process_input(self, x, keyframe_idxs, denoise_mask, **kw):
# Reset per-forward state FIRST so stale values from a previous run can never leak
# into this forward (e.g. if the ref latents stop reaching us after a Comfy update).
self._idma_ref_len = 0
self._idma_blocks = []
self._idma_target_len = None
self._idma_ppf = None
out = orig_process_input(x, keyframe_idxs, denoise_mask, **kw)
# Comfy versions differ on whether custom transformer_options keys arrive flattened
# in kwargs or nested under kwargs["transformer_options"] — accept both.
ref_lats = kw.get("_id_ref_latents")
if ref_lats is None:
ref_lats = (kw.get("transformer_options") or {}).get("_id_ref_latents")
if not ref_lats:
_dbg("process_input: NO ref latents in kwargs (flat or nested) -> vanilla forward. kw keys:", list(kw.keys()))
return out
from comfy.ldm.lightricks.model import latent_to_pixel_coords
xx, pix, add = out
is_av = isinstance(xx, (list, tuple))
vx = xx[0] if is_av else xx
vco = pix[0] if is_av else pix
target_len = vx.shape[1]
self._idma_target_len = target_len
segs = getattr(self, "_idma_segs", [2.0] * len(ref_lats))
temporal_offset_latents = getattr(
self,
"_idma_temporal_offset_latents",
0,
)
blocks = []
offset = target_len
for ref_lat, seg in zip(ref_lats, segs):
rt, rlc = self.patchifier.patchify(ref_lat.to(dtype=vx.dtype, device=vx.device))
rpc = latent_to_pixel_coords(latent_coords=rlc, scale_factors=self.vae_scale_factors,
causal_fix=self.causal_temporal_positioning)
rpc = shift_reference_temporal_positions(
rpc,
temporal_offset_latents,
self.vae_scale_factors[0],
)
rt = self.patchify_proj(rt)
if rt.shape[0] != vx.shape[0]:
rt = rt.expand(vx.shape[0], -1, -1)
if rpc.shape[0] != vco.shape[0]:
rpc = rpc.expand(vco.shape[0], *([-1] * (rpc.dim() - 1)))
rlen = rt.shape[1]
vx = torch.cat([vx, rt], dim=1)
vco = torch.cat([vco, rpc.to(vco)], dim=2)
blocks.append((offset, rlen, float(seg)))
offset += rlen
self._idma_ref_len = offset - target_len
self._idma_blocks = blocks
# each ref is one latent frame at the target grid -> its token count == tokens per frame
self._idma_ppf = blocks[0][1] if blocks else None
add = dict(add); add["_id_ref_len"] = self._idma_ref_len
_dbg("process_input OUT: blocks", blocks, "| target_len", target_len, "| total_ref", self._idma_ref_len)
if is_av:
xx = [vx, xx[1]]; pix = [vco, pix[1]]
else:
xx, pix = vx, vco
return xx, pix, add
def prepare_timestep(self, timestep, batch_size, hidden_dtype, **kw):
# MEASURE-based extension: only pad the timestep by exactly what process_input
# actually appended in THIS forward, and detect the timestep granularity from its
# real length (per-token / per-frame / already-extended) instead of assuming one.
# This keeps modulation and vx in lockstep across ComfyUI versions.
ref_len = getattr(self, "_idma_ref_len", 0)
tgt_len = getattr(self, "_idma_target_len", None)
if ref_len and tgt_len:
ppf = getattr(self, "_idma_ppf", None)
if timestep.dim() <= 1:
timestep = timestep.reshape(-1, 1).expand(batch_size, tgt_len).contiguous()
cur = timestep.shape[1]
# With audio connected, LTXAV may produce a combined video+audio per-token
# timestep (e.g. 24576 = video 6144 + audio 18432). Trim to video-only so
# the extension matches vx (which only has video + ref tokens).
if cur > tgt_len + ref_len:
_dbg("prepare_timestep: oversized", cur, "(tgt", tgt_len, "+ ref", ref_len,
") -> trimming to video-only", tgt_len)
timestep = timestep[:, :tgt_len]
cur = tgt_len
if cur == tgt_len: # per-token, target only -> append per-token zeros
z = torch.zeros(timestep.shape[0], ref_len, *timestep.shape[2:],
device=timestep.device, dtype=timestep.dtype)
timestep = torch.cat([timestep, z], dim=1)
_dbg("prepare_timestep: per-token extend", cur, "->", timestep.shape[1])
elif ppf and cur * ppf == tgt_len: # per-FRAME timestep -> append ref frames
n_ref_frames = max(1, ref_len // ppf)
z = torch.zeros(timestep.shape[0], n_ref_frames, *timestep.shape[2:],
device=timestep.device, dtype=timestep.dtype)
timestep = torch.cat([timestep, z], dim=1)
_dbg("prepare_timestep: per-frame extend", cur, "->", timestep.shape[1])
elif cur == tgt_len + ref_len: # already extended (double-patch guard)
_dbg("prepare_timestep: already extended", cur)
else:
_dbg("prepare_timestep: UNEXPECTED len", cur, "target", tgt_len, "ppf", ppf, "-> untouched")
# Guide nodes (IC-LoRA Guide / Director Guide, etc.) inject a grid_mask that
# FILTERS x tokens in _process_input (x = x[:, grid_mask]) and later indexes the
# timestep (timestep[:, grid_mask]). Our ref tokens are appended AFTER that
# filter, so extend the mask with True for them (False would drop our clean
# timesteps from the modulation and desync it from vx). Pad size is measured
# from the actual gap so per-token AND per-frame masks both work.
gm = kw.get("grid_mask")
if gm is not None and hasattr(gm, "shape") and timestep.dim() >= 2:
gap = timestep.shape[1] - gm.shape[-1]
if 0 < gap <= ref_len:
pad = torch.ones(*gm.shape[:-1], gap, dtype=gm.dtype, device=gm.device)
kw = dict(kw); kw["grid_mask"] = torch.cat([gm, pad], dim=-1)
_dbg("prepare_timestep: grid_mask extended by", gap)
return orig_prepare_ts(timestep, batch_size, hidden_dtype, **kw)
def prepare_pe(self, pixel_coords, frame_rate, x_dtype):
pe = orig_prepare_pe(pixel_coords, frame_rate, x_dtype)
blocks = getattr(self, "_idma_blocks", [])
if not blocks:
return pe
theta = getattr(self, "_id_rope_theta", 10000.0)
def rot(v_pe):
for (start, length, seg) in blocks:
v_pe = _rotate_block(v_pe, start, length, seg, theta)
return v_pe
if isinstance(pe, list) and len(pe) and isinstance(pe[0], (list, tuple)) and isinstance(pe[0][0], (list, tuple)):
v_pe, cross_v = pe[0][0], pe[0][1]
v_pe = rot(v_pe)
pe = [(v_pe, cross_v), pe[1]]
else:
pe = rot(pe)
return pe
def process_output(self, x, embedded_timestep, keyframe_idxs, **kw):
ref_len = getattr(self, "_idma_ref_len", 0)
if ref_len:
_dbg("process_output: trimming", ref_len, "ref tokens")
if ref_len:
import copy
from comfy.ldm.lightricks.av_model import CompressedTimestep
if isinstance(x, (list, tuple)):
x = [x[0][:, :x[0].shape[1] - ref_len], *x[1:]]
et_list = list(embedded_timestep) if isinstance(embedded_timestep, (list, tuple)) else [embedded_timestep]
v_et = et_list[0]
if isinstance(v_et, CompressedTimestep):
ppf = max(1, getattr(v_et, "patches_per_frame", 1) or 1)
n_ref_frames = max(1, ref_len // ppf)
v_et2 = copy.copy(v_et)
v_et2.data = v_et.data[:, : v_et.num_frames - n_ref_frames].contiguous()
v_et2.num_frames = v_et.num_frames - n_ref_frames
et_list[0] = v_et2
elif hasattr(v_et, "shape") and v_et.dim() >= 2 and v_et.shape[1] > 1:
et_list[0] = v_et[:, : v_et.shape[1] - ref_len]
embedded_timestep = et_list
else:
x = x[:, :x.shape[1] - ref_len]
if hasattr(embedded_timestep, "shape") and embedded_timestep.dim() >= 2 and embedded_timestep.shape[1] > 1:
embedded_timestep = embedded_timestep[:, : embedded_timestep.shape[1] - ref_len]
return orig_process_output(x, embedded_timestep, keyframe_idxs, **kw)
ltxv._process_input = types.MethodType(process_input, ltxv)
ltxv._prepare_timestep = types.MethodType(prepare_timestep, ltxv)
ltxv._prepare_positional_embeddings = types.MethodType(prepare_pe, ltxv)
ltxv._process_output = types.MethodType(process_output, ltxv)
ltxv._id_multi_patched = True
log.info("LTXIdentityMultiAngle patches installed on %s", type(ltxv).__name__)
class LTXIdentityMultiAngle:
@classmethod
def INPUT_TYPES(cls):
try:
import folder_paths
proj_choices = ["None"] + folder_paths.get_filename_list("loras")
except Exception:
proj_choices = ["None"]
return {
"required": {
"model": ("MODEL",),
"positive": ("CONDITIONING",),
"negative": ("CONDITIONING",),
"vae": ("VAE",),
"latent": ("LATENT",),
"reference_face_front": ("IMAGE", {
"tooltip": "REQUIRED — frontal face (source_id 2). Also drives the ArcFace projector."}),
"identity_projector": (proj_choices, {"default": "None",
"tooltip": "ArcFace projector .safetensors (from models/loras). 'None' = overlap only."}),
"phase_scale": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.1,
"tooltip": "Global RoPE phase scale (training used 1.0). Applied to every view's source_id."}),
"id_strength": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 50.0, "step": 0.5,
"tooltip": "Multiplies the ArcFace projector tokens (frontal face only)."}),
"arcface_mode": (["auto_adjust", "as_is", "disable"], {"default": "auto_adjust"}),
"debug_log": ("BOOLEAN", {"default": False}),
},
"optional": {
"reference_back_head": ("IMAGE", {
"tooltip": "OPTIONAL — back of the head / nape, hair from behind (source_id 3). Leave empty to skip."}),
"reference_body_front": ("IMAGE", {
"tooltip": "OPTIONAL — full-body front, build/proportions (source_id 4). Leave empty to skip."}),
"reference_side_profile": ("IMAGE", {
"tooltip": "OPTIONAL — side / profile of the face (source_id 5). Leave empty to skip."}),
# Appended last so saved workflows keep every historical widget index stable.
"reference_temporal_offset_latents": reference_temporal_offset_input(),
},
}
RETURN_TYPES = ("MODEL", "CONDITIONING", "CONDITIONING", "LATENT", "STRING")
RETURN_NAMES = ("model", "positive", "negative", "latent", "debug")
FUNCTION = "apply"
CATEGORY = "LTX/identity"
DESCRIPTION = ("Multi-angle identity transfer: frontal face (required) + optional back/body/side "
"reference views, each injected as its own overlap+source_phase segment (source_id "
"2/3/4/5), matching the multi-view training. Load the multi-view LoRA on MODEL first.")
def apply(self, model, positive, negative, vae, latent, reference_face_front,
reference_temporal_offset_latents=DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
identity_projector="None", phase_scale=1.0, id_strength=1.0,
arcface_mode="auto_adjust", debug_log=False,
reference_back_head=None, reference_body_front=None, reference_side_profile=None):
import comfy.utils
global _DEBUG
_DEBUG = bool(debug_log)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
m = model.clone()
ltxv = _find_ltxv(m)
_, w_sf, h_sf = vae.downscale_index_formula
_, _, _, lat_h, lat_w = latent["samples"].shape
px_w, px_h = lat_w * w_sf, lat_h * h_sf
def encode(img):
ref_px = comfy.utils.common_upscale(img.movedim(-1, 1), px_w, px_h, "bilinear", "center").movedim(1, -1)[:1, :, :, :3]
return vae.encode(ref_px)
# Build the provided views in a fixed order (face first). Empty optionals are skipped.
provided = [("face", reference_face_front)]
if reference_back_head is not None:
provided.append(("back", reference_back_head))
if reference_body_front is not None:
provided.append(("body", reference_body_front))
if reference_side_profile is not None:
provided.append(("side", reference_side_profile))
ref_lats, segs, used = [], [], []
for slot, img in provided:
ref_lats.append(encode(img))
segs.append(_SLOT_SOURCE_ID[slot] * float(phase_scale))
used.append(f"{slot}(sid={_SLOT_SOURCE_ID[slot]:.0f})")
_install_multi_patches(ltxv)
ltxv._idma_segs = segs
ltxv._idma_temporal_offset_latents = int(reference_temporal_offset_latents)
ltxv._id_rope_theta = 10000.0
m.model_options = dict(m.model_options)
to = dict(m.model_options.get("transformer_options", {}))
to["_id_ref_latents"] = ref_lats
m.model_options["transformer_options"] = to
# ArcFace projector tokens from the FRONTAL face only (optional).
use_projector = identity_projector not in (None, "", "None")
emb = _arcface_embed(reference_face_front, mode=arcface_mode) if use_projector else None
if not use_projector:
arc_status = "no projector (overlap only)"
elif arcface_mode == "disable":
arc_status = "disabled"
else:
arc_status = "OK" if emb is not None else "NO FACE -> skipped"
if emb is not None:
path = identity_projector
if not os.path.isabs(path):
try:
import folder_paths
resolved = folder_paths.get_full_path("loras", identity_projector)
if resolved:
path = resolved
except Exception:
pass
if not os.path.isabs(path) or not os.path.exists(path):
for b in ("models/loras", "models", "."):
if os.path.exists(os.path.join(b, identity_projector)):
path = os.path.join(b, identity_projector); break
projector = _load_projector(path, device)
emb = emb.to(device=device, dtype=torch.float32).unsqueeze(0)
with torch.no_grad():
id_tok = projector(emb) * float(id_strength)
unc = projector(torch.zeros(1, projector.in_dim, device=device))
positive = _append_ctx_tokens(positive, id_tok)
negative = _append_ctx_tokens(negative, unc)
dbg = (
"=== LTX Identity Transfer — MULTIPLE ANGLES ===\n"
f"reference views ({len(ref_lats)}): {', '.join(used)}\n"
f"ref latents: {[list(r.shape) for r in ref_lats]} | phase_scale={phase_scale}\n"
f"reference temporal offset: {int(reference_temporal_offset_latents)} latent step(s) "
f"= {reference_temporal_pixel_offset(reference_temporal_offset_latents, vae.downscale_index_formula[0])} "
"pixel frame(s) (all reference views only)\n"
f"arcface (frontal): {arc_status} (mode={arcface_mode}) | id_strength={id_strength}\n"
f"patches on {type(ltxv).__name__}: process_input/prepare_timestep/prepare_pe/process_output (multi-block)\n"
"Empty optional inputs are omitted (matches training view-dropout). Connect negative + CFG 3-5, no LightX2V."
)
log.info("\n" + dbg)
return (m, positive, negative, latent, dbg)
NODE_CLASS_MAPPINGS = {"LTXIdentityMultiAngle": LTXIdentityMultiAngle}
NODE_DISPLAY_NAME_MAPPINGS = {"LTXIdentityMultiAngle": "LTX Identity Transfer (Multiple Angles)"}
+912
View File
@@ -0,0 +1,912 @@
"""LTX-2.3/2.5 Identity OVERLAP — replica of ltx-trainer's overlap+source_phase reference.
Unlike the append_keyframe path (which makes the ref an I2V first-frame), this injects the
reference latent as SEPARATE tokens using the trainer's overlap/source-phase contract. The
reference-only temporal offset can move those positions before frame zero to reduce learned
frame-0 composition leakage; set it to 0 for the exact trainer overlap grid. The ref tokens
remain clean (timestep 0), attend to the target in self-attention, and are sliced off the
output (never rendered).
Model patch (installed idempotently on the LTX av_model), matching LTXBaseModel._forward:
1) _process_input : append ref video tokens (patchified ref latent) with overlap positions;
2) _prepare_positional_embeddings : rotate the ref tokens' RoPE freqs by source_phase;
3) _prepare_timestep : give the ref tokens clean (0) timestep;
4) _process_output : trim the ref tokens before unpatchify.
Plus the ArcFace IdentityProjector tokens on the text context (as trained).
Load the matching LoRA on MODEL first. Requires insightface + buffalo_l (ArcFace, CPU).
Set LTX_IDOVERLAP_DEBUG=1 to log shapes at each patch point for iteration.
"""
import functools
import logging
import os
import types
import weakref
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from safetensors.torch import load_file
from .reference_temporal_offset import (
DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
reference_temporal_offset_input,
reference_temporal_pixel_offset,
shift_reference_temporal_positions,
)
log = logging.getLogger("LTXIdentityOverlap")
_USE_GPU = os.environ.get("LTX_IDPROJ_ARCFACE_GPU", "0") == "1"
_FACE_APP = None
def _shape(x):
"""Readable shape/structure of tensors, lists, tuples, CompressedTimestep, etc."""
try:
if hasattr(x, "shape"):
return f"T{tuple(x.shape)}"
if isinstance(x, (list, tuple)):
return f"{type(x).__name__}[{', '.join(_shape(i) for i in x)}]"
return type(x).__name__
except Exception:
return "?"
# Per-step debug logging. Off by default; toggled by the node's `debug_log` input
# (env var LTX_IDOVERLAP_DEBUG=1 sets the initial value).
_DEBUG_ENABLED = os.environ.get("LTX_IDOVERLAP_DEBUG", "0") == "1"
def _dbg(*a):
if _DEBUG_ENABLED:
print("[LTXIdOverlap] " + " ".join(str(x) for x in a), flush=True)
# ---------------- ArcFace + projector (same as the other nodes) ----------------
def _get_face_app():
global _FACE_APP
if _FACE_APP is None:
try:
from insightface.app import FaceAnalysis
except ImportError as e:
raise RuntimeError("This node needs the optional insightface package: pip install insightface (with onnxruntime "
"or onnxruntime-gpu), then restart ComfyUI.") from e
providers = (["CUDAExecutionProvider", "CPUExecutionProvider"] if _USE_GPU else ["CPUExecutionProvider"])
app = FaceAnalysis(name="buffalo_l", providers=providers)
app.prepare(ctx_id=0 if _USE_GPU else -1, det_size=(640, 640))
_FACE_APP = app
return _FACE_APP
class IdentityProjector(nn.Module):
def __init__(self, in_dim=512, context_dim=4096, num_tokens=4, proj=None):
super().__init__()
self.in_dim = in_dim; self.context_dim = context_dim; self.num_tokens = num_tokens
# `proj` is built from the checkpoint (variable depth); fallback = original 2-linear MLP.
self.proj = proj if proj is not None else nn.Sequential(
nn.Linear(in_dim, 1024), nn.GELU(), nn.Linear(1024, num_tokens * context_dim))
self.norm = nn.LayerNorm(context_dim)
def forward(self, e):
return self.norm(self.proj(e).reshape(-1, self.num_tokens, self.context_dim))
def _load_projector(path, device):
"""Load an IdentityProjector of ANY depth by rebuilding proj.N Linears from the state_dict
(old = 2 linears proj.0/proj.2; new enhanced = 3 linears proj.0/proj.2/proj.4, 16 tokens)."""
sd = load_file(path)
context_dim = sd["norm.weight"].shape[0]
# collect Linear layers in order (proj.<idx>.weight), with GELU between them
idxs = sorted({int(k.split(".")[1]) for k in sd if k.startswith("proj.") and k.endswith(".weight")})
layers = []
for j, i in enumerate(idxs):
w = sd[f"proj.{i}.weight"]
layers.append(nn.Linear(w.shape[1], w.shape[0]))
if j < len(idxs) - 1:
layers.append(nn.GELU())
proj = nn.Sequential(*layers)
in_dim = sd["proj.0.weight"].shape[1]
num_tokens = sd[f"proj.{idxs[-1]}.weight"].shape[0] // context_dim
# Sequential indices must match the state_dict (Linear at even positions, GELU odd) — they do.
p = IdentityProjector(in_dim=in_dim, context_dim=context_dim, num_tokens=num_tokens, proj=proj)
p.load_state_dict(sd)
log.info("IdentityProjector loaded: %d tokens, %d Linear layers", num_tokens, len(idxs))
return p.to(device=device, dtype=torch.float32).eval()
def _arcface_embed(image_bhwc, mode="auto_adjust"):
"""Return the ArcFace embedding, or None if disabled / no face found.
mode: 'as_is' (detect on the image only), 'auto_adjust' (retry with border-pad
zoom-out + upscale when detection fails), 'disable' (skip ArcFace entirely).
"""
if mode == "disable":
return None
import cv2
app = _get_face_app()
img = np.ascontiguousarray((np.clip(image_bhwc[0].detach().cpu().numpy(), 0.0, 1.0) * 255.0).astype(np.uint8)[:, :, ::-1])
attempts = [img]
if mode == "auto_adjust":
h, w = img.shape[:2]
pad = int(0.4 * max(h, w)) # zoom-out: face may fill the frame
attempts.append(cv2.copyMakeBorder(img, pad, pad, pad, pad, cv2.BORDER_REPLICATE))
attempts.append(cv2.resize(img, None, fx=2.0, fy=2.0, interpolation=cv2.INTER_CUBIC)) # upscale small faces
attempts.append(cv2.resize(img, None, fx=0.5, fy=0.5, interpolation=cv2.INTER_AREA)) # downscale huge faces
for a in attempts:
faces = app.get(a)
if faces:
f = max(faces, key=lambda x: (x.bbox[2] - x.bbox[0]) * (x.bbox[3] - x.bbox[1]))
return torch.from_numpy(f.normed_embedding.astype(np.float32))
return None
def _append_ctx_tokens(conditioning, tokens):
out = []
for ce, cd in conditioning:
t = tokens.to(device=ce.device, dtype=ce.dtype)
if t.shape[0] != ce.shape[0]:
t = t.expand(ce.shape[0], -1, -1)
if t.shape[-1] < ce.shape[-1]:
t = F.pad(t, (0, ce.shape[-1] - t.shape[-1]))
elif t.shape[-1] > ce.shape[-1]:
t = t[..., :ce.shape[-1]]
nd = cd.copy()
am = nd.get("attention_mask")
if am is not None:
nd["attention_mask"] = torch.cat([am, torch.ones((*am.shape[:-1], t.shape[1]), device=am.device, dtype=am.dtype)], dim=-1)
out.append([torch.cat([ce, t], dim=1), nd])
return out
# ---------------- source_phase RoPE (port of ltx_core rope.apply_segment_phase) ----------------
def _rotate_ref_block(pe, start, length, seg_value, theta=10000.0):
"""Rotate RoPE freqs of tokens [start:start+length] by phase = seg_value * theta^(-d/L).
One reference block's own token range gets its own seg_value (source_id*phase_scale) --
callers loop this once per stacked reference so each keeps its own RoPE phase tag. With a
single reference (the common case), this is exactly the old "last ref_len tokens" rotation.
Two upstream pe formats, detected at runtime (ComfyUI changed this in 2026-07):
- fused matrix (current): pe = (rotation_matrix, split_mode_bool); rotation_matrix shape
[B, T, num_heads, L, 2, 2], one true 2x2 rotation matrix per (batch, token, head, freq).
We compose an additional phase rotation via matmul -- 2D rotation matrices commute, so
left/right composition order doesn't matter: R(seg*rate) @ R(existing) == R(existing+seg*rate).
- separate cos/sin (pre-2026-07 ComfyUI): pe = (cos, sin, [split_flag]), each [..., T, L].
"""
if length <= 0 or seg_value == 0.0:
return pe
first = pe[0]
if torch.is_tensor(first) and first.dim() >= 3 and first.shape[-2:] == (2, 2):
# fused rotation-matrix format: pe = (matrix[B,T,H,L,2,2], split_mode)
matrix = first
rest = tuple(pe[1:])
L = matrix.shape[-3]
d = torch.arange(L, device=matrix.device, dtype=torch.float32)
rate = theta ** (-d / float(L))
phase = seg_value * rate # [L]
cp = phase.cos().to(matrix.dtype)
sp = phase.sin().to(matrix.dtype)
r_phase = torch.stack(
(torch.stack((cp, -sp), dim=-1), torch.stack((sp, cp), dim=-1)), dim=-2
) # [L, 2, 2]
sub = matrix[:, start:start + length] # [B, length, H, L, 2, 2]
rotated = torch.matmul(r_phase, sub)
matrix = matrix.clone()
matrix[:, start:start + length] = rotated
_dbg("rotate ref block (matrix): L", L, "start", start, "length", length, "seg", seg_value)
return (matrix, *rest)
# legacy separate cos/sin format
cos, sin = pe[0], pe[1]
rest = tuple(pe[2:])
L = cos.shape[-1]
d = torch.arange(L, device=cos.device, dtype=torch.float32)
rate = theta ** (-d / float(L)) # (0,1], high-freq carries the tag
phase = (seg_value * rate) # [L]
pc = phase.cos().to(cos.dtype); ps = phase.sin().to(sin.dtype)
# index the token axis (=-2)
idx = [slice(None)] * cos.dim()
idx[-2] = slice(start, start + length)
idx = tuple(idx)
c0, s0 = cos[idx], sin[idx]
cos = cos.clone(); sin = sin.clone()
cos[idx] = c0 * pc - s0 * ps
sin[idx] = s0 * pc + c0 * ps
_dbg("rotate ref block: L", L, "start", start, "length", length, "seg", seg_value)
return (cos, sin, *rest)
# ---------------- model patches (idempotent, on the av_model instance) ----------------
def _find_ltxv(model):
m = getattr(model, "model", model)
m = getattr(m, "diffusion_model", m)
return m
def _detect_ltx_version(ltxv):
"""Detect the LTX generation from a model-native checkpoint parameter.
LTX-2.5 introduces ``keyframes_abs_pos_embedding``. ComfyUI detects the same
state-dict key when constructing the model and exposes both the feature flag
and the parameter on LTXBaseModel. Avoid filename-based detection because a
user can rename or quantize a checkpoint without changing its architecture.
"""
if bool(getattr(ltxv, "use_keyframes_abs_pos_embedding", False)):
return "2.5"
if getattr(ltxv, "keyframes_abs_pos_embedding", None) is not None:
return "2.5"
return "2.3/legacy"
def _letterbox_resize(ref_img, tgt_w, tgt_h, pad_value=1.0):
"""Resize `ref_img` ([B,H,W,C]) to fit ENTIRELY inside tgt_w x tgt_h, preserving its own
aspect ratio (no crop, no distortion), padding the leftover space with `pad_value`
(default white, matching the composite sheet's own white background). Unlike
common_upscale(..., crop="center") -- which center-crops to the target aspect ratio
BEFORE resizing, silently discarding whatever isn't in the middle of the source image --
this never discards any pixel of the reference."""
import comfy.utils
x = ref_img.movedim(-1, 1) # [B,C,H,W]
_, _, src_h, src_w = x.shape
scale = min(tgt_w / src_w, tgt_h / src_h)
new_w, new_h = max(1, round(src_w * scale)), max(1, round(src_h * scale))
resized = comfy.utils.common_upscale(x, new_w, new_h, "bilinear", "disabled")
pad_w, pad_h = tgt_w - new_w, tgt_h - new_h
left, top = pad_w // 2, pad_h // 2
right, bottom = pad_w - left, pad_h - top
padded = F.pad(resized, (left, right, top, bottom), mode="constant", value=pad_value)
return padded.movedim(1, -1)
def _anchored_crop_resize(ref_img, tgt_w, tgt_h, anchor="center"):
"""Like comfy.utils.common_upscale(..., crop="center") but with a configurable anchor
for WHICH part of the source survives the crop when the aspect ratio doesn't match,
instead of always the exact center (e.g. anchor="top" keeps the top of the sheet --
useful when the face closeup panel isn't centered in your layout). Returns the
cropped+resized image plus the crop box (x0, y0, crop_w, crop_h) in SOURCE pixel
coords, for drawing a preview overlay."""
import comfy.utils
x_img = ref_img.movedim(-1, 1) # [B,C,H,W]
_, _, old_h, old_w = x_img.shape
old_aspect = old_w / old_h
new_aspect = tgt_w / tgt_h
x0, y0, crop_w, crop_h = 0, 0, old_w, old_h
if old_aspect > new_aspect:
# source wider than target -- crop width
crop_w = max(1, round(old_w * (new_aspect / old_aspect)))
if anchor == "left":
x0 = 0
elif anchor == "right":
x0 = old_w - crop_w
else:
x0 = (old_w - crop_w) // 2
elif old_aspect < new_aspect:
# source taller than target -- crop height
crop_h = max(1, round(old_h * (old_aspect / new_aspect)))
if anchor == "top":
y0 = 0
elif anchor == "bottom":
y0 = old_h - crop_h
else:
y0 = (old_h - crop_h) // 2
cropped = x_img[:, :, y0:y0 + crop_h, x0:x0 + crop_w]
out = comfy.utils.common_upscale(cropped, tgt_w, tgt_h, "bilinear", "disabled")
return out.movedim(1, -1), (x0, y0, crop_w, crop_h)
def _draw_crop_overlay(ref_img, box):
"""Original reference with a green rectangle around the region that SURVIVES the crop
(everything outside the rectangle gets discarded). box = (x0, y0, w, h) in source pixel
coords, e.g. from _anchored_crop_resize. If box covers the whole image (no crop, as in
letterbox/native_resolution modes), the rectangle just outlines the full frame."""
from PIL import Image, ImageDraw
x0, y0, cw, ch = box
arr = (ref_img[0, :, :, :3].clamp(0, 1).cpu().numpy() * 255).astype(np.uint8)
pil = Image.fromarray(arr).convert("RGB")
draw = ImageDraw.Draw(pil)
w, h = pil.size
lw = max(2, min(w, h) // 150)
draw.rectangle([x0, y0, x0 + cw - 1, y0 + ch - 1], outline=(0, 255, 0), width=lw)
out = torch.from_numpy(np.array(pil).astype(np.float32) / 255.0).unsqueeze(0)
return out
# Seconds reserved per numbered strata slot -- MUST match ltx_trainer's
# training_strategies.tass.STRATA_SLOT_WIDTH for a strata-trained checkpoint's RoPE
# convention to line up at inference. Slot 0 (1st ref in the batch) lands at
# target_max_t + this value, slot 1 (2nd ref) at target_max_t + 2*this value, etc. --
# dynamic/target-relative, same as st_drc's own shift, so it stays correct for whatever
# video length is actually generated.
# CHANGED 2026-07-18 (0.5 -> 1.5): 0.5 caused a real slot0/slot1 collision -- a single-frame
# reference image (fps=1.0 convention) spans a full 1.0-SECOND range on the T axis, not a
# point, so width=0.5 let adjacent slots overlap. See tass.py's STRATA_SLOT_WIDTH docstring.
# Any checkpoint trained before this fix (e.g. phantom_stacked_r128/_p2) learned under the
# collided geometry; continuing training from one now adapts to the corrected spacing.
STRATA_SLOT_WIDTH = 1.5
def _tass_axis_stats(positions):
"""Per-axis (min, max), reduced over every dimension EXCEPT the T/H/W one.
Two coordinate shapes reach this node: older LTX builds hand out one corner per token,
[B, 3 (T/H/W), N], while LTX-2.5 hands out patch bounds, [B, 3, N, 2] -- the same shape
the trainer works in. Reducing over "everything but the axis" is correct for both, and
keeps a patch's two bounds shifted by the SAME amount instead of by their own separate
minima (which would squeeze the span).
The batch is reduced too. Layouts run before the reference tokens are expanded to the CFG
batch, so target_positions can be [2, ...] while the reference is still [1, ...]; cond and
uncond share one geometry, so a batch-wide min/max is the same number either way and
broadcasts into whichever side is batch-1.
"""
dims = (0,) + tuple(range(2, positions.dim()))
return positions.amin(dim=dims, keepdim=True), positions.amax(dim=dims, keepdim=True)
def _apply_tass_layout(reference_positions, target_positions, layout: str, strata_start: float | None = None,
sidecar_margin_pixels: float = 0.0, use_middle_indices_grid: bool = True):
"""Place reference pixel-coords in a non-overlapping TASS region -- mirrors
ltx_core.conditioning.reference_layout.apply_reference_layout (kept in sync manually since
this node can't import the trainer package). Handles both coordinate shapes ComfyUI emits;
see _tass_axis_stats.
layout='overlap' returns the input unchanged.
layout='st_drc' shifts every axis (T, H, W) past the target's own extent.
layout='strata' shifts ONLY the T axis to an absolute band start (`strata_start`, in the
same raw pixel/frame units as `reference_positions` -- caller converts from seconds using
the model's frame_rate), leaving H/W overlapping the target -- see STRATA_SLOT_WIDTH.
layout='sidecar' places the reference BESIDE the target on the W axis, vertically centred on
it, and stretches its T span to cover the target's whole clip -- so the reference is present
for every frame instead of sitting at one instant. This is the layout the replace_refedit
recipe validated for an object reference.
"""
if layout == "overlap":
return reference_positions
# ComfyUI's pixel coords are int64 (latent corners times integer VAE scale factors), while
# centring and midpoints are inherently half-pixel maths -- an in-place += of a float shift
# into a Long tensor just raises "result type Float can't be cast to the desired output
# type Long". Compute in float and round back on the way out, so callers keep the dtype
# they passed in (the caller casts to the target grid's dtype when concatenating, and a
# cast truncates where rounding is what we want).
coordinate_dtype = reference_positions.dtype
if not reference_positions.is_floating_point():
reference_positions = reference_positions.to(torch.float32)
if not target_positions.is_floating_point():
target_positions = target_positions.to(torch.float32)
def _restore(shifted):
if shifted.dtype == coordinate_dtype:
return shifted
return shifted.round().to(coordinate_dtype)
target_min, target_extent = _tass_axis_stats(target_positions)
reference_origin, reference_extent = _tass_axis_stats(reference_positions)
if layout == "st_drc":
return _restore(reference_positions + (target_extent - reference_origin))
if layout == "strata":
if strata_start is None:
raise ValueError("layout='strata' requires strata_start")
shifted = reference_positions.clone()
shifted[:, 0:1, ...] += strata_start - reference_origin[:, 0:1, ...]
return _restore(shifted)
if layout == "sidecar":
if reference_positions.shape[1] < 3:
raise ValueError(
f"layout='sidecar' expects at least 3 coordinate axes, got {reference_positions.shape[1]}"
)
shifted = reference_positions.clone()
# H: centraliza a referencia na altura do alvo
target_center_h = (target_min[:, 1:2, ...] + target_extent[:, 1:2, ...]) * 0.5
reference_center_h = (reference_origin[:, 1:2, ...] + reference_extent[:, 1:2, ...]) * 0.5
shifted[:, 1:2, ...] += target_center_h - reference_center_h
# W: desloca para a direita do alvo, mais a margem
shifted[:, 2:3, ...] += (
target_extent[:, 2:3, ...] + float(sidecar_margin_pixels) - reference_origin[:, 2:3, ...]
)
# T: a referencia cobre todo o intervalo do alvo. O trainer escreve o par
# (inicio, fim) = (t_min, t_max), e o modelo o colapsa num unico numero por token --
# de duas maneiras diferentes, e e por isso que aqui tem um `if`:
# use_middle_indices_grid=True (2.5, e qualquer 2.3 cujo metadata declare) -> (inicio+fim)/2
# use_middle_indices_grid=False (2.3 nos checkpoints antigos) -> so o inicio
# Escrever (t_min, t_max) nos dois casos poria a referencia no MEIO do clipe no 2.5 e no
# FRAME 0 no 2.3 -- placements diferentes para o mesmo LoRA. Escrever o ponto medio nos
# dois bounds da a mesma posicao efetiva sob qualquer uma das regras.
t_min = target_min[:, 0, 0, 0] if target_min.dim() == 4 else target_min[:, 0, 0]
t_max = target_extent[:, 0, 0, 0] if target_extent.dim() == 4 else target_extent[:, 0, 0]
if shifted.dim() == 4 and use_middle_indices_grid:
shifted[:, 0, :, 0] = t_min
shifted[:, 0, :, 1] = t_max
elif shifted.dim() == 4:
shifted[:, 0, :, :] = ((t_min + t_max) * 0.5).unsqueeze(-1)
else:
# Coordenadas legadas [B, 3, N]: um canto por token, lido direto pelo RoPE.
shifted[:, 0, :] = (t_min + t_max) * 0.5
return _restore(shifted)
raise ValueError(f"Unsupported TASS layout {layout!r}")
def _unbound(obj, name):
"""The plain function behind `obj.name`, so closures never pin the instance.
A bound method carries a strong reference to its instance; captured inside the patch closures
below it would keep the model alive through its own __dict__ (see _bind_weak).
"""
function = getattr(type(obj), name, None)
if function is not None:
return function
attribute = getattr(obj, name, None)
if attribute is None:
return None
# Dynamically assigned (already patched by another node): fall back to its underlying
# function when there is one, otherwise wrap the bound callable and accept the reference.
return getattr(attribute, "__func__", None) or (lambda _self, *a, **k: attribute(*a, **k))
def _bind_weak(function, instance):
"""Install-ready callable with a method's signature but no reference cycle.
`instance._x = types.MethodType(fn, instance)` makes instance -> __dict__ -> method ->
instance, a cycle only a full collection can break. ComfyUI holds a weakref to the model and,
finding it alive after a run, logs "memory leak with model LTXAV" and forces a full gc every
execution -- which is what makes repeated runs progressively slower (issue #9). Resolving the
instance from a weakref at call time keeps the patch working with none of that.
"""
reference = weakref.ref(instance)
@functools.wraps(function)
def patched(*args, **kwargs):
target = reference()
if target is None:
raise RuntimeError("LTXIdentityOverlap: the patched model no longer exists")
return function(target, *args, **kwargs)
return patched
def _install_patches(ltxv):
if getattr(ltxv, "_id_overlap_patched", False):
return
orig_process_input = _unbound(ltxv, "_process_input")
orig_prepare_ts = _unbound(ltxv, "_prepare_timestep")
orig_prepare_pe = _unbound(ltxv, "_prepare_positional_embeddings")
orig_process_output = _unbound(ltxv, "_process_output")
orig_forward_internal = _unbound(ltxv, "_forward")
if orig_forward_internal is not None:
def _forward_capture_fps(self, x, timestep, context, attention_mask, frame_rate=25,
transformer_options={}, keyframe_idxs=None, denoise_mask=None, **kwargs):
# _forward calls _process_input BEFORE _prepare_positional_embeddings (the only
# other place frame_rate normally reaches) -- process_input needs it EARLIER, to
# convert the seconds-scale STRATA_SLOT_WIDTH into this step's raw pixel/frame
# units. Stash here so it's always current, never a step behind.
self._id_frame_rate = float(frame_rate)
return orig_forward_internal(
self, x, timestep, context, attention_mask, frame_rate=frame_rate,
transformer_options=transformer_options, keyframe_idxs=keyframe_idxs,
denoise_mask=denoise_mask, **kwargs,
)
ltxv._forward = _bind_weak(_forward_capture_fps, ltxv)
def process_input(self, x, keyframe_idxs, denoise_mask, **kw):
# Reset per-forward state first so a stale value from a previous run can never leak
# into this forward (e.g. if ref specs stop arriving after a Comfy update).
self._id_ref_len = 0
self._id_blocks = []
out = orig_process_input(self, x, keyframe_idxs, denoise_mask, **kw)
ref_specs = kw.get("_id_ref_specs")
if ref_specs is None:
ref_specs = (kw.get("transformer_options") or {}).get("_id_ref_specs")
if not ref_specs:
return out
try:
from comfy.ldm.lightricks.model import latent_to_pixel_coords
xx, pix, add = out
is_av = isinstance(xx, (list, tuple))
vx = xx[0] if is_av else xx
vco = pix[0] if is_av else pix
target_len = vx.shape[1]
self._id_target_len = target_len
frame_rate = float(getattr(self, "_id_frame_rate", 25.0))
# Raw pixel/frame units (pre frame_rate-division -- that happens later inside
# _prepare_positional_embeddings) -- convert to seconds only for the strata math,
# then back, since STRATA_SLOT_WIDTH is calibrated in seconds on the trainer side.
target_max_t_raw = float(vco[:, 0, :].amax().item())
_dbg("process_input IN: is_av", is_av, "| vx", _shape(vx), "| vco", _shape(vco),
"| n_refs", len(ref_specs), "| frame_rate", frame_rate)
blocks = [] # (start, length, seg_value) per ref, in concatenation order
offset = target_len
for spec in ref_specs:
ref_lat = spec["latent"]
rt, rlc = self.patchifier.patchify(ref_lat.to(dtype=vx.dtype, device=vx.device))
rpc = latent_to_pixel_coords(latent_coords=rlc, scale_factors=self.vae_scale_factors,
causal_fix=self.causal_temporal_positioning)
# downscale_factor: this ref was VAE-encoded at target_size/downscale_factor,
# so its raw pixel-coords only span that smaller extent. Multiply the H/W axes
# back up by downscale_factor to re-stretch them across the target's actual
# coordinate range (fewer tokens, same spatial span) -- mirrors ltx_core's
# VideoConditionByReferenceLatent.apply_to (positions[:,1:3,...] *= downscale_factor).
dsf = spec.get("downscale_factor", 1)
if dsf != 1:
rpc = rpc.clone()
rpc[:, 1, ...] *= dsf
rpc[:, 2, ...] *= dsf
strata_start_raw = None
if spec["layout"] == "strata":
slot = int(spec["strata_slot"])
strata_start_sec = target_max_t_raw / frame_rate + (slot + 1) * STRATA_SLOT_WIDTH
strata_start_raw = strata_start_sec * frame_rate
rpc = _apply_tass_layout(
rpc, vco, spec["layout"], strata_start=strata_start_raw,
sidecar_margin_pixels=spec.get("sidecar_margin_pixels", 0.0),
use_middle_indices_grid=bool(getattr(self, "use_middle_indices_grid", False)),
)
# Appearance references may be placed before frame zero to reduce the learned
# overlap frame-0 copy/leak. Old/external specs without this key retain offset 0.
# The shift is applied AFTER layout placement so it remains explicit for every
# supported layout; guide/mask specs opt into 0 in LTX Multiple Controls.
rpc = shift_reference_temporal_positions(
rpc,
spec.get("temporal_offset_latents", 0),
self.vae_scale_factors[0],
)
# Learned per-slot tag, added in FEATURE space on the raw patchified tokens --
# i.e. before patchify_proj, exactly where the trainer adds it (the model's
# input projection is applied after). Specs without the key are unaffected.
slot_vector = spec.get("slot_vector")
if slot_vector is not None:
rt = rt + slot_vector.to(device=rt.device, dtype=rt.dtype)
rt = self.patchify_proj(rt)
if rt.shape[0] != vx.shape[0]:
rt = rt.expand(vx.shape[0], -1, -1)
if rpc.shape[0] != vco.shape[0]:
rpc = rpc.expand(vco.shape[0], *([-1] * (rpc.dim() - 1)))
rlen = rt.shape[1]
vx = torch.cat([vx, rt], dim=1)
vco = torch.cat([vco, rpc.to(vco)], dim=2)
blocks.append((offset, rlen, float(spec["seg_value"])))
offset += rlen
ref_len = offset - target_len
self._id_ref_len = ref_len
self._id_blocks = blocks
add = dict(add); add["_id_ref_len"] = ref_len
_dbg("process_input OUT: blocks", blocks, "| target_len", target_len,
"| vx", _shape(vx), "| vco", _shape(vco))
if is_av:
xx = [vx, xx[1]]; pix = [vco, pix[1]]
else:
xx, pix = vx, vco
return xx, pix, add
except Exception as e:
_dbg("ERROR process_input:", repr(e), "| out", _shape(out), "| n_refs", len(ref_specs) if ref_specs else 0)
raise
def prepare_timestep(self, timestep, batch_size, hidden_dtype, **kw):
# Give the ref tokens clean (0) timestep by editing the timestep INPUT before the
# model's per-frame compression/adaln (mirrors the audio-ref path in av_model).
# Use ONLY the instance attribute, not kw["_id_ref_len"] — when multi-angle patches
# are also installed, their process_input sets kw["_id_ref_len"] which we must NOT
# consume here (that would double-extend the timestep and cause shape mismatches).
ref_len = getattr(self, "_id_ref_len", 0)
if ref_len:
target_len = getattr(self, "_id_target_len", None)
if timestep.dim() <= 1 and target_len is not None:
timestep = timestep.view(-1, 1).expand(batch_size, target_len).contiguous()
if timestep.dim() >= 2 and target_len is not None:
cur = timestep.shape[1]
_gm0 = kw.get("grid_mask")
_full = _gm0.shape[-1] if (_gm0 is not None and hasattr(_gm0, "shape")) else None
if _full is not None and cur == _full:
# Per-token timestep spans the grid-mask domain (video + IC-LoRA guide
# tokens, indexed later via timestep[:, grid_mask]). Do NOT trim — append
# ref zeros and let the grid_mask pad below keep everything in lockstep.
ref_ts = torch.zeros(batch_size, ref_len, *timestep.shape[2:], device=timestep.device, dtype=timestep.dtype)
timestep = torch.cat([timestep, ref_ts], dim=1)
_dbg("prepare_timestep: grid-domain cur", cur, "-> appended ref", ref_len)
elif cur > target_len + ref_len:
_dbg("prepare_timestep: oversized", cur, "-> trimming to video-only", target_len)
timestep = timestep[:, :target_len]
cur = target_len
if (_full is None or timestep.shape[1] != _full + ref_len) and cur == target_len:
ref_ts = torch.zeros(batch_size, ref_len, *timestep.shape[2:], device=timestep.device, dtype=timestep.dtype)
timestep = torch.cat([timestep, ref_ts], dim=1)
_dbg("prepare_timestep: ref_len", ref_len, "| timestep ->", _shape(timestep), "| target_len", target_len)
else:
_dbg("prepare_timestep: skip (cur", cur, "!= target", target_len, ")")
# Guide nodes (IC-LoRA Guide / Director Guide, etc.) inject a grid_mask that
# FILTERS x tokens in _process_input (x = x[:, grid_mask]) and later indexes the
# timestep (timestep[:, grid_mask]). Our ref tokens are appended AFTER that
# filter, so extend the mask with True for them — keeps modulation and vx in
# lockstep (False would drop our clean timesteps and desync again).
gm = kw.get("grid_mask")
if gm is not None and hasattr(gm, "shape"):
gap = timestep.shape[1] - gm.shape[-1]
if 0 < gap <= ref_len:
pad = torch.ones(*gm.shape[:-1], gap, dtype=gm.dtype, device=gm.device)
kw = dict(kw); kw["grid_mask"] = torch.cat([gm, pad], dim=-1)
_dbg("prepare_timestep: grid_mask extended by", gap)
return orig_prepare_ts(self, timestep, batch_size, hidden_dtype, **kw)
def prepare_pe(self, pixel_coords, frame_rate, x_dtype):
pe = orig_prepare_pe(self, pixel_coords, frame_rate, x_dtype)
blocks = getattr(self, "_id_blocks", [])
theta = getattr(self, "_id_rope_theta", 10000.0)
if not blocks:
return pe
try:
_dbg("prepare_pe IN: pe struct", _shape(pe), "| blocks", blocks)
def rot(v_pe):
for start, length, seg in blocks:
v_pe = _rotate_ref_block(v_pe, start, length, seg, theta)
return v_pe
# av returns [(v_pe, av_cross_video), (a_pe, av_cross_audio)]; v_pe = (cos, sin, split).
if isinstance(pe, list) and len(pe) and isinstance(pe[0], (list, tuple)) and isinstance(pe[0][0], (list, tuple)):
v_pe, cross_v = pe[0][0], pe[0][1]
v_pe = rot(v_pe)
pe = [(v_pe, cross_v), pe[1]]
else:
pe = rot(pe)
return pe
except Exception as e:
_dbg("ERROR prepare_pe:", repr(e), "| pe", _shape(pe))
raise
def process_output(self, x, embedded_timestep, keyframe_idxs, **kw):
ref_len = getattr(self, "_id_ref_len", 0)
if ref_len:
try:
from comfy.ldm.lightricks.av_model import CompressedTimestep
_dbg("process_output IN: x", _shape(x), "| et", _shape(embedded_timestep), "| ref_len", ref_len)
# trim ref tokens from the video stream
if isinstance(x, (list, tuple)):
x = [x[0][:, :x[0].shape[1] - ref_len], *x[1:]]
import copy
et_list = list(embedded_timestep) if isinstance(embedded_timestep, (list, tuple)) else [embedded_timestep]
v_et = et_list[0]
if isinstance(v_et, CompressedTimestep):
# clone + edit slots directly (version-agnostic; some builds lack per_frame kwarg)
ppf = max(1, getattr(v_et, "patches_per_frame", 1) or 1)
n_ref_frames = max(1, ref_len // ppf)
v_et2 = copy.copy(v_et)
v_et2.data = v_et.data[:, : v_et.num_frames - n_ref_frames].contiguous()
v_et2.num_frames = v_et.num_frames - n_ref_frames
et_list[0] = v_et2
elif hasattr(v_et, "shape") and v_et.dim() >= 2 and v_et.shape[1] > 1:
et_list[0] = v_et[:, : v_et.shape[1] - ref_len]
embedded_timestep = et_list
else:
x = x[:, :x.shape[1] - ref_len]
if hasattr(embedded_timestep, "shape") and embedded_timestep.dim() >= 2 and embedded_timestep.shape[1] > 1:
embedded_timestep = embedded_timestep[:, : embedded_timestep.shape[1] - ref_len]
_dbg("process_output: trimmed -> x", _shape(x), "| et", _shape(embedded_timestep))
except Exception as e:
_dbg("ERROR process_output:", repr(e), "| x", _shape(x), "| et", _shape(embedded_timestep))
raise
return orig_process_output(self, x, embedded_timestep, keyframe_idxs, **kw)
ltxv._process_input = _bind_weak(process_input, ltxv)
ltxv._prepare_timestep = _bind_weak(prepare_timestep, ltxv)
ltxv._prepare_positional_embeddings = _bind_weak(prepare_pe, ltxv)
ltxv._process_output = _bind_weak(process_output, ltxv)
ltxv._id_overlap_patched = True
log.info("LTXIdentityOverlap patches installed on %s", type(ltxv).__name__)
class LTXIdentityOverlapConditioning:
@classmethod
def INPUT_TYPES(cls):
return {"required": {
"model": ("MODEL",),
"positive": ("CONDITIONING",),
"negative": ("CONDITIONING",),
"vae": ("VAE",),
"latent": ("LATENT",),
"reference_image": ("IMAGE", {
"tooltip": "Reference to copy into the generation -- any subject (object, animal, "
"character, person...), not just a face. Accepts a BATCH of N images (use "
"an Image Batch node to combine several) for checkpoints trained on "
"multiple STACKED references (layout='strata') -- each image in the batch "
"becomes its own reference block with source_id = source_id + its index "
"(0-based: 1st image keeps 'source_id' as-is, 2nd gets source_id+1, ...). "
"For the exact historical frame-0 placement, set the reference temporal "
"offset to 0."}),
"source_id": ("FLOAT", {"default": 2.0, "min": 0.0, "max": 8.0, "step": 1.0,
"tooltip": "source_phase segment id (training used 2). 0 = no phase."}),
"phase_scale": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.1}),
"ref_resize_mode": (["match_target", "match_target_letterbox", "native_resolution"], {"default": "match_target",
"tooltip": "match_target: resize ref to the OUTPUT video's pixel size via a CENTER-CROP then "
"resize (old single-face-crop recipes — ref resolution never mattered, but this "
"silently discards whatever isn't in the middle of the ref for mismatched aspect "
"ratios, e.g. a landscape character sheet used for a portrait output loses the "
"side panels/face closeup). match_target_letterbox: same target pixel size, but "
"fits the WHOLE ref inside it preserving aspect ratio (no crop, pads with white) "
"-- use this for a landscape composite sheet + portrait (or any mismatched-aspect) "
"output so no panel/face-detail gets cut off. native_resolution: encode the ref at "
"ITS OWN size (rounded to the nearest 32px), independent of the video size — "
"REQUIRED for checkpoints trained on a fixed ref resolution bucket that differs "
"from the video's own bucket (e.g. the composite face+3views ref, trained at "
"2048x1024 regardless of output video size) -- but for aspect-mismatched outputs "
"(e.g. portrait video) this can bias the model toward the ref's own (landscape) "
"composition, causing an off-center/cropped-looking result; try "
"match_target_letterbox first if you hit that."}),
"debug_log": ("BOOLEAN", {"default": False,
"tooltip": "Print per-step [LTXIdOverlap] shape logs to the console (for debugging)."}),
}, "optional": {
"crop_anchor": (["center", "top", "bottom", "left", "right"], {"default": "center",
"tooltip": "New in v1.10.13, optional -- old workflows without this input keep the "
"previous always-center-crop behavior. Only used by match_target (the "
"crop-then-resize mode). Which part of the reference survives the crop when its "
"aspect ratio doesn't match the output -- e.g. if your sheet's face closeup is "
"in the top row, set 'top' instead of the default center crop so it doesn't get "
"cut off. No effect on match_target_letterbox or native_resolution (neither "
"ever crops)."}),
"layout": (["overlap", "st_drc", "strata", "sidecar"], {"default": "overlap",
"tooltip": "The base positional layout used by the checkpoint. 'overlap' starts from the "
"target's own RoPE coordinate range and then applies the independent reference "
"temporal offset; use offset 0 for the exact historical overlap grid. 'st_drc' "
"shifts the WHOLE reference block past the target's coordinate extent on every "
"axis (non-overlapping region). 'strata' shifts ONLY the temporal axis to a slot "
"past the target's own length -- one slot per image in the reference_image batch "
"(1st image -> slot 0, 2nd -> slot 1, ...), leaving H/W overlapping the target; "
"this is the 'stacked references' convention (ltx_trainer TASS strata layout). "
"Only use whichever layout the loaded checkpoint was actually trained with -- "
"using the wrong one is not a quality hit, it's a coordinate convention the "
"model never learned, so identity transfer likely won't work at all."}),
"reference_guidance_scale": ("FLOAT", {"default": 1.0, "min": 1.0, "max": 10.0, "step": 0.1,
"tooltip": "ST-DRC-style reference-CFG (arxiv 2606.02441). 1.0 = off (identical to "
"before this input existed). >1.0 adds a THIRD forward pass per step with "
"the reference tokens dropped, and amplifies the reference's own "
"contribution: denoised += (scale-1)*(with_ref - without_ref) -- the same "
"way CFG amplifies the text prompt's. Costs one extra (cheaper, "
"ref-token-free) forward pass per step when enabled. Start around 2-4; "
"same units/convention as CFG scale on the sampler."}),
# Appended last so saved workflows keep every historical widget index stable.
"reference_temporal_offset_latents": reference_temporal_offset_input(),
}}
RETURN_TYPES = ("MODEL", "CONDITIONING", "CONDITIONING", "LATENT", "STRING", "IMAGE", "IMAGE")
RETURN_NAMES = ("model", "positive", "negative", "latent", "debug", "ref_preview", "crop_overlay")
FUNCTION = "apply"
CATEGORY = "LTX/identity"
DESCRIPTION = ("Overlap/st_drc/strata + source_phase reference conditioning via a model patch, with an "
"experimental reference-only temporal offset to reduce frame-0 composition leakage. "
"Reference is separate tokens (NOT I2V), any subject -- not just a face. Accepts a BATCH of "
"images for checkpoints trained on stacked references (layout='strata'). "
"Load LoRA on MODEL first. ref_preview/crop_overlay outputs show exactly what gets encoded "
"and, for match_target, what part of the reference survives the crop (green box) vs gets discarded.")
def apply(self, model, positive, negative, vae, latent, reference_image,
reference_temporal_offset_latents=DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
source_id=2.0, phase_scale=1.0,
ref_resize_mode="match_target", debug_log=False,
crop_anchor="center", layout="overlap", reference_guidance_scale=1.0):
import comfy.samplers
import comfy.utils
global _DEBUG_ENABLED
_DEBUG_ENABLED = bool(debug_log)
m = model.clone()
ltxv = _find_ltxv(m)
detected_ltx_version = _detect_ltx_version(ltxv)
_, w_sf, h_sf = vae.downscale_index_formula
n_refs = reference_image.shape[0]
def _encode_one(img1):
"""Resize (per ref_resize_mode/crop_anchor) + VAE-encode ONE reference image
([1,H,W,C] slice). Byte-identical to the pre-batch code path when n_refs == 1."""
if ref_resize_mode == "native_resolution":
# Encode at the ref image's OWN resolution (rounded to nearest 32px),
# independent of the output video size -- matches training when the ref used
# a fixed/own bucket.
_, src_h, src_w, _ = img1.shape
tgt_w = max(w_sf, round(src_w / w_sf) * w_sf)
tgt_h = max(h_sf, round(src_h / h_sf) * h_sf)
else:
# Legacy behavior: resize ref to match the target video's pixel size (correct
# for recipes where the ref used the SAME resolution bucket as the video, e.g.
# a small face crop -- resolution never mattered there).
_, _, _, lat_h, lat_w = latent["samples"].shape
tgt_w, tgt_h = lat_w * w_sf, lat_h * h_sf
_, src_h0, src_w0, _ = img1.shape
crop_box = (0, 0, src_w0, src_h0) # default: "nothing cropped" (letterbox/native modes)
if ref_resize_mode == "match_target_letterbox":
ref_px = _letterbox_resize(img1, tgt_w, tgt_h)[:1, :, :, :3]
elif ref_resize_mode == "match_target" and crop_anchor != "center":
# non-default anchor: use the configurable-anchor crop (new in v1.10.13)
ref_px, crop_box = _anchored_crop_resize(img1, tgt_w, tgt_h, anchor=crop_anchor)
ref_px = ref_px[:1, :, :, :3]
else:
# unchanged from before v1.10.13 -- exact original center-crop path, byte-identical
ref_px = comfy.utils.common_upscale(img1.movedim(-1, 1), tgt_w, tgt_h, "bilinear", "center").movedim(1, -1)[:1, :, :, :3]
if ref_resize_mode == "match_target":
_, crop_box = _anchored_crop_resize(img1, tgt_w, tgt_h, anchor="center") # for the preview overlay only
ref_lat = vae.encode(ref_px)
overlay = _draw_crop_overlay(img1[:1], crop_box)
return ref_lat, ref_px.clone(), overlay, crop_box, src_w0, src_h0
ref_specs, ref_previews, crop_overlays = [], [], []
for i in range(n_refs):
ref_lat_i, ref_px_i, overlay_i, crop_box, src_w0, src_h0 = _encode_one(reference_image[i:i + 1])
ref_specs.append({"latent": ref_lat_i, "seg_value": (float(source_id) + i) * float(phase_scale),
"layout": layout, "strata_slot": i,
"temporal_offset_latents": int(reference_temporal_offset_latents)})
ref_previews.append(ref_px_i)
crop_overlays.append(overlay_i)
ref_lat = ref_specs[0]["latent"] # for the debug string below (1st ref's shape)
ref_preview = torch.cat(ref_previews, dim=0)
crop_overlay = torch.cat(crop_overlays, dim=0)
_install_patches(ltxv)
ltxv._id_rope_theta = 10000.0
m.model_options = dict(m.model_options)
to = dict(m.model_options.get("transformer_options", {}))
to["_id_ref_specs"] = ref_specs
m.model_options["transformer_options"] = to
if reference_guidance_scale != 1.0:
# ST-DRC reference-CFG: a third forward pass per step with the reference tokens
# dropped isolates the reference's own contribution, the same way CFG isolates
# the text prompt's (arxiv 2606.02441). `args["input_cond"]` is the exact
# conditioning list KSampler passed for the positive/"cond" branch (same prompt,
# same batching) -- reuse it unchanged, only strip `_id_ref_specs` from the
# model_options used for THIS extra call so process_input sees no reference.
noref_to = dict(to)
noref_to.pop("_id_ref_specs", None)
ref_scale = float(reference_guidance_scale)
# NOTE: must take exactly ONE parameter -- set_model_sampler_cfg_function()
# inspects the signature's parameter COUNT (regardless of defaults) and treats
# anything with 3 params as the legacy (cond, uncond, cond_scale) calling
# convention, silently passing those three tensors/floats positionally instead
# of the `args` dict. Keep noref_to/ref_scale as plain closure vars, not defaults.
def _ref_cfg_function(args):
cond = args["cond"]
uncond = args["uncond"]
cond_scale = args["cond_scale"]
denoised = uncond + (cond - uncond) * cond_scale
noref_model_options = dict(args["model_options"])
noref_model_options["transformer_options"] = noref_to
(noref_pred,) = comfy.samplers.calc_cond_batch(
args["model"], [args["input_cond"]], args["input"], args["timestep"], noref_model_options,
)
noref_denoised = args["input"] - noref_pred
denoised = denoised + (ref_scale - 1.0) * (cond - noref_denoised)
return denoised
m.set_model_sampler_cfg_function(_ref_cfg_function, disable_cfg1_optimization=True)
seg_list = ", ".join(f"#{i}={s['seg_value']:g}" for i, s in enumerate(ref_specs))
temporal_pixel_offset = reference_temporal_pixel_offset(
reference_temporal_offset_latents,
vae.downscale_index_formula[0],
)
dbg = (
"=== LTX Identity OVERLAP (exact) ===\n"
f"detected model: LTX-{detected_ltx_version} "
f"(architecture detection via keyframes_abs_pos_embedding)\n"
f"references: {n_refs} (encoded at {ref_preview.shape[2]}x{ref_preview.shape[1]}px each, "
f"mode={ref_resize_mode}{f', crop_anchor={crop_anchor}' if ref_resize_mode == 'match_target' else ''}) "
f"-> {layout} tokens, source_phase seg per ref: {seg_list}\n"
f"reference temporal offset: {int(reference_temporal_offset_latents)} latent step(s) "
f"= {temporal_pixel_offset} pixel frame(s) (reference only)\n"
f"patches on {type(ltxv).__name__}: process_input/prepare_timestep/prepare_pe/process_output\n"
f"crop preview: kept region {crop_box[2]}x{crop_box[3]}px of the {src_w0}x{src_h0}px reference "
"-- see the ref_preview/crop_overlay IMAGE outputs to inspect what gets kept vs discarded.\n"
f"reference-CFG: {'off' if reference_guidance_scale == 1.0 else f'ON, scale={reference_guidance_scale} (+1 forward pass/step)'}\n"
"Set LTX_IDOVERLAP_DEBUG=1 for per-step shape logs. Connect negative + CFG 3-5, no LightX2V."
)
log.info("\n" + dbg)
# pass the latent through unchanged (the ref is injected inside the model, not here)
# so the graph can chain Empty -> this node -> sampler without branching.
return (m, positive, negative, latent, dbg, ref_preview, crop_overlay)
# Public node id + display name. Keep the old key as an alias so existing workflows load.
NODE_CLASS_MAPPINGS = {
"LTXIdentityTransfer": LTXIdentityOverlapConditioning,
"LTXIdentityOverlapConditioning": LTXIdentityOverlapConditioning,
}
NODE_DISPLAY_NAME_MAPPINGS = {
"LTXIdentityTransfer": "LTX Identity Transfer",
"LTXIdentityOverlapConditioning": "LTX Identity Transfer",
}
+308
View File
@@ -0,0 +1,308 @@
"""LTX Multiple Controls — single-node version of chaining several LTXIdentityOverlapConditioning
(LTX Identity Transfer) calls for a guide (motion/structure) + mask (replacement region) +
identity (appearance) recipe, e.g. character-replace-in-driving-video training.
WHY THIS NODE EXISTS: LTXIdentityOverlapConditioning stores its reference specs in
model.model_options["transformer_options"]["_id_ref_specs"], and each call OVERWRITES that key
with its own specs list rather than merging with whatever the upstream node already put there.
Chaining N of those nodes therefore only keeps the LAST node's reference active at inference —
the earlier ones (e.g. the guide) silently have ZERO effect despite the graph looking correct.
This node builds every slot's ref_specs in ONE apply() call so they all end up in the same list.
Each slot (guide/mask/identity/identity_mask) is fully independent: its own source_id,
phase_scale, layout, ref_resize_mode -- no slot is hardcoded to a fixed layout/phase, only
given sane defaults that match the CURRENT trained recipe (scail2v2_r128_v13): guide=0 and
mask=0 (both => no RoPE phase tag, same as source_phase=false at train time -- they must stay
positionally overlapped with the target frame-by-frame); identity=2 (=> tagged, same as
source_phase=true, since it has no positional correspondence with the target). identity_mask
is unused by v13 (ref_mask was dropped from the recipe entirely) -- leave it unconnected.
v13's key fix vs earlier recipes: `guide_video` must have the replacement region painted BLACK
(an inpainting hole) before the model sees it, otherwise it just copies guide's own person into
the masked region and never uses `identity_image` at all -- confirmed as the root cause of a
long stretch of "identity ignored" results earlier in training. This node does that
automatically (`auto_mask_guide`, on by default) using `mask_video`'s white region -- do NOT
also pre-mask guide upstream or the hole will double-apply harmlessly but wastefully.
mask_video is now a plain WHITE-silhouette-on-black binary mask (NOT colored) -- an earlier
per-identity color-coded mask design caused the model to leak the mask's own hue into the
generated clothing (confirmed via direct pixel inspection during training); white/black avoids
that since it carries no color information for the VAE to latch onto.
"""
import logging
import torch
from . import ltx_identity_overlap as _ido
from .ltx_identity_overlap import (
_anchored_crop_resize,
_draw_crop_overlay,
_find_ltxv,
_install_patches,
_letterbox_resize,
)
from .reference_temporal_offset import (
DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
reference_temporal_offset_input,
)
log = logging.getLogger("LTXMultipleControls")
_LAYOUT_CHOICES = ["overlap", "st_drc", "strata", "sidecar"]
_RESIZE_CHOICES = ["match_target", "match_target_letterbox", "native_resolution"]
def _encode_ref(vae, latent, img, ref_resize_mode, crop_anchor, w_sf, h_sf, downscale_factor=1):
"""Resize (per ref_resize_mode/crop_anchor) + VAE-encode ONE reference IMAGE batch
([N,H,W,C], typically a whole video's frames). Mirrors LTXIdentityOverlapConditioning's
own _encode_one so behavior is byte-identical per slot.
downscale_factor>1: encode at target_size/downscale_factor instead of full target size
(mirrors the trainer's reference_downscale_factor -- fewer tokens/cheaper VRAM for a
condition that doesn't need full detail, e.g. guide/mask). Snapped to a multiple of
w_sf/h_sf (must stay divisible by the VAE's own spatial downscale, same constraint as
training's "must be divisible by 32" check)."""
import comfy.utils
if ref_resize_mode == "native_resolution":
_, src_h, src_w, _ = img.shape
tgt_w = max(w_sf, round(src_w / w_sf) * w_sf)
tgt_h = max(h_sf, round(src_h / h_sf) * h_sf)
else:
_, _, _, lat_h, lat_w = latent["samples"].shape
tgt_w, tgt_h = lat_w * w_sf, lat_h * h_sf
if downscale_factor != 1:
tgt_w = max(w_sf, round(tgt_w / downscale_factor / w_sf) * w_sf)
tgt_h = max(h_sf, round(tgt_h / downscale_factor / h_sf) * h_sf)
_, src_h0, src_w0, _ = img.shape
crop_box = (0, 0, src_w0, src_h0)
if ref_resize_mode == "match_target_letterbox":
ref_px = _letterbox_resize(img, tgt_w, tgt_h)[:, :, :, :3]
elif ref_resize_mode == "match_target" and crop_anchor != "center":
ref_px, crop_box = _anchored_crop_resize(img, tgt_w, tgt_h, anchor=crop_anchor)
ref_px = ref_px[:, :, :, :3]
else:
ref_px = comfy.utils.common_upscale(img.movedim(-1, 1), tgt_w, tgt_h, "bilinear", "center").movedim(1, -1)[:, :, :, :3]
if ref_resize_mode == "match_target":
_, crop_box = _anchored_crop_resize(img[:1], tgt_w, tgt_h, anchor="center")
ref_lat = vae.encode(ref_px)
overlay = _draw_crop_overlay(img[:1], crop_box)
return ref_lat, ref_px[:1].clone(), overlay, crop_box, src_w0, src_h0
class LTXMultipleControls:
@classmethod
def INPUT_TYPES(cls):
return {"required": {
"model": ("MODEL",),
"positive": ("CONDITIONING",),
"negative": ("CONDITIONING",),
"vae": ("VAE",),
"latent": ("LATENT",),
}, "optional": {
"guide_video": ("IMAGE", {"tooltip": "Motion/structure driving video frames (IMAGE batch, e.g. from "
"GetVideoComponents). Leave unconnected to skip the guide slot. Gets auto-masked "
"black in the replacement region (see auto_mask_guide) unless disabled."}),
"guide_source_id": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 8.0, "step": 1.0,
"tooltip": "v13 recipe default: 0 (no RoPE phase tag at all -- must stay "
"source_phase=false so the guide's positions line up with the target "
"frame-by-frame). Non-zero only for older/experimental checkpoints."}),
"guide_phase_scale": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.1}),
"guide_layout": (_LAYOUT_CHOICES, {"default": "overlap"}),
"guide_ref_resize_mode": (_RESIZE_CHOICES, {"default": "match_target"}),
"guide_downscale_factor": ("INT", {"default": 1, "min": 1, "max": 8,
"tooltip": "Encode at target_size/N instead of full res (cheaper VRAM). Must match "
"whatever the checkpoint was trained with -- 1 for the no-downscale "
"recipes, 2 for the earlier downscaled-guide recipes. Wrong value = "
"positions the model never learned, not just a quality hit."}),
"mask_video": ("IMAGE", {"tooltip": "Per-frame replacement-region mask, same frame count/alignment "
"as guide_video -- v13 expects a plain WHITE-on-black binary silhouette (person "
"white, background black). Do NOT use a colored mask (see Color Mask node) -- "
"confirmed to leak hue into the generated clothing during training. Also drives "
"auto_mask_guide's black-out of guide_video. Leave unconnected to skip."}),
"mask_source_id": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 8.0, "step": 1.0,
"tooltip": "v13 recipe default: 0 (same non-tagged group as guide -- both need exact "
"positional correspondence with the target). Non-zero only for older/"
"experimental checkpoints."}),
"mask_phase_scale": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.1}),
"mask_layout": (_LAYOUT_CHOICES, {"default": "overlap"}),
"mask_ref_resize_mode": (_RESIZE_CHOICES, {"default": "match_target"}),
"mask_downscale_factor": ("INT", {"default": 1, "min": 1, "max": 8,
"tooltip": "Same as guide_downscale_factor -- match the checkpoint's training recipe."}),
"auto_mask_guide": ("BOOLEAN", {"default": True,
"tooltip": "v13 requirement: paints guide_video BLACK wherever mask_video is white, "
"before encoding -- an inpainting hole, so the model can't just copy "
"guide's own person into the replacement region and must pull identity "
"from identity_image instead (matches how the training data was built). "
"Turn off only if guide_video is already pre-masked upstream."}),
"mask_white_thresh": ("FLOAT", {"default": 0.5, "min": 0.0, "max": 1.0, "step": 0.01,
"tooltip": "auto_mask_guide only -- mask_video pixels with mean brightness >= this "
"(0-1 scale) count as foreground (person) and get blacked out in guide."}),
"identity_image": ("IMAGE", {"tooltip": "Appearance reference (face/character), no positional "
"correspondence with the target needed -- this is what fills auto_mask_guide's hole. "
"Leave unconnected to skip."}),
"identity_source_id": ("FLOAT", {"default": 2.0, "min": 0.0, "max": 8.0, "step": 1.0,
"tooltip": "v13 recipe default: 2. Nonzero = tagged with its own RoPE phase (matches "
"source_phase=true, since identity has no positional correspondence with "
"the target)."}),
"identity_phase_scale": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.1}),
"identity_layout": (_LAYOUT_CHOICES, {"default": "overlap"}),
"identity_ref_resize_mode": (_RESIZE_CHOICES, {"default": "native_resolution"}),
"identity_downscale_factor": ("INT", {"default": 1, "min": 1, "max": 8,
"tooltip": "Same as guide_downscale_factor -- match the checkpoint's training recipe."}),
"identity_mask_image": ("IMAGE", {"tooltip": "scail2-style color-pointer marker for the identity "
"slot (small flat color dot/blob, NOT a body silhouette -- a body-shaped mask here "
"would re-inject a competing pose signal, exactly the bug the flat-marker design "
"fixes on the training side). Same source_id/phase as identity_image by default "
"(they're trained as one group) -- override identity_mask_source_id if your "
"checkpoint used a different grouping. Leave unconnected to skip."}),
"identity_mask_source_id": ("FLOAT", {"default": -1.0, "min": -1.0, "max": 8.0, "step": 1.0,
"tooltip": "-1 = inherit identity_source_id/identity_phase_scale (matches training, "
"where ref+ref_mask share one group). Set >=0 to give it its own separate phase."}),
"identity_mask_phase_scale": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.1}),
"identity_mask_layout": (_LAYOUT_CHOICES, {"default": "overlap"}),
"identity_mask_ref_resize_mode": (_RESIZE_CHOICES, {"default": "native_resolution"}),
"identity_mask_downscale_factor": ("INT", {"default": -1, "min": -1, "max": 8,
"tooltip": "-1 = inherit identity_downscale_factor. Same meaning otherwise as "
"guide_downscale_factor -- match the checkpoint's training recipe."}),
"crop_anchor": (["center", "top", "bottom", "left", "right"], {"default": "center",
"tooltip": "Shared by all slots using ref_resize_mode=match_target with a mismatched "
"aspect ratio -- which part of the source survives the crop."}),
"reference_guidance_scale": ("FLOAT", {"default": 1.0, "min": 1.0, "max": 10.0, "step": 0.1,
"tooltip": "ST-DRC-style reference-CFG applied to the WHOLE combined reference set "
"(all active slots together). 1.0 = off."}),
"debug_log": ("BOOLEAN", {"default": False}),
# Reference-only despite being appended here for saved-workflow widget stability.
"identity_temporal_offset_latents": reference_temporal_offset_input(),
}}
RETURN_TYPES = ("MODEL", "CONDITIONING", "CONDITIONING", "LATENT", "STRING")
RETURN_NAMES = ("model", "positive", "negative", "latent", "debug")
FUNCTION = "apply"
CATEGORY = "LTX/identity"
DESCRIPTION = ("Guide + mask + identity + identity_mask reference conditioning in ONE node call, so all "
"active slots' RoPE reference specs actually combine (chaining LTXIdentityOverlapConditioning "
"nodes does NOT -- each call overwrites the previous one's specs). Every slot is optional and "
"its own source_id/phase_scale/layout/ref_resize_mode -- nothing is hardcoded, only defaulted "
"to match scail2v2_r128_v13 (guide=0, mask=0, both no-phase; identity=2, phased). guide_video "
"is auto-masked black in the replacement region (auto_mask_guide) to match how v13 was "
"trained -- this is the fix that made identity transfer actually work. identity_mask is "
"unused by v13 (leave unconnected); kept for older checkpoints.")
def apply(self, model, positive, negative, vae, latent,
guide_video=None, guide_source_id=0.0, guide_phase_scale=1.0,
guide_layout="overlap", guide_ref_resize_mode="match_target", guide_downscale_factor=1,
mask_video=None, mask_source_id=0.0, mask_phase_scale=1.0,
mask_layout="overlap", mask_ref_resize_mode="match_target", mask_downscale_factor=1,
auto_mask_guide=True, mask_white_thresh=0.5,
identity_image=None, identity_source_id=2.0, identity_phase_scale=1.0,
identity_layout="overlap", identity_ref_resize_mode="native_resolution", identity_downscale_factor=1,
identity_temporal_offset_latents=DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
identity_mask_image=None, identity_mask_source_id=-1.0, identity_mask_phase_scale=1.0,
identity_mask_layout="overlap", identity_mask_ref_resize_mode="native_resolution",
identity_mask_downscale_factor=-1,
crop_anchor="center", reference_guidance_scale=1.0, debug_log=False):
import comfy.samplers
import comfy.utils
_ido._DEBUG_ENABLED = bool(debug_log)
m = model.clone()
ltxv = _find_ltxv(m)
_, w_sf, h_sf = vae.downscale_index_formula
if guide_video is not None and mask_video is not None and auto_mask_guide:
mg = mask_video
if mg.shape[1:3] != guide_video.shape[1:3]:
mg = comfy.utils.common_upscale(
mg.movedim(-1, 1), guide_video.shape[2], guide_video.shape[1], "nearest-exact", "center",
).movedim(1, -1)
if mg.shape[0] != guide_video.shape[0]:
idx = torch.linspace(0, mg.shape[0] - 1, guide_video.shape[0]).round().long().clamp(max=mg.shape[0] - 1)
mg = mg[idx]
fg = mg.mean(dim=-1, keepdim=True) >= mask_white_thresh
guide_video = torch.where(fg, torch.zeros_like(guide_video), guide_video)
# -1 = inherit the identity slot's own grouping (matches training: ref+ref_mask share
# one source_id/phase_scale by default).
if identity_mask_source_id < 0:
identity_mask_source_id = identity_source_id
identity_mask_phase_scale = identity_phase_scale
if identity_mask_downscale_factor < 0:
identity_mask_downscale_factor = identity_downscale_factor
slots = [
("guide", guide_video, guide_source_id, guide_phase_scale, guide_layout, guide_ref_resize_mode, guide_downscale_factor, 0),
("mask", mask_video, mask_source_id, mask_phase_scale, mask_layout, mask_ref_resize_mode, mask_downscale_factor, 0),
("identity", identity_image, identity_source_id, identity_phase_scale, identity_layout,
identity_ref_resize_mode, identity_downscale_factor, identity_temporal_offset_latents),
("identity_mask", identity_mask_image, identity_mask_source_id, identity_mask_phase_scale,
identity_mask_layout, identity_mask_ref_resize_mode, identity_mask_downscale_factor, 0),
]
ref_specs = []
summary = []
for name, img, source_id, phase_scale, layout, resize_mode, downscale_factor, temporal_offset in slots:
if img is None:
continue
ref_lat, _px, _overlay, crop_box, src_w0, src_h0 = _encode_ref(
vae, latent, img, resize_mode, crop_anchor, w_sf, h_sf, downscale_factor=downscale_factor)
seg_value = float(source_id) * float(phase_scale)
ref_specs.append({"latent": ref_lat, "seg_value": seg_value, "layout": layout,
"strata_slot": len(ref_specs), "downscale_factor": downscale_factor,
"temporal_offset_latents": int(temporal_offset)})
dsf_note = f", downscale={downscale_factor}x" if downscale_factor != 1 else ""
offset_note = f", ref_t_offset={int(temporal_offset)} latent" if name == "identity" else ""
summary.append(
f"{name}: {img.shape[0]}f {src_w0}x{src_h0}px -> {layout}, seg={seg_value:g}, "
f"mode={resize_mode}{dsf_note}{offset_note}"
)
if not ref_specs:
log.warning("LTXMultipleControls: no slots connected -- passing through unchanged.")
return (m, positive, negative, latent, "LTX Multiple Controls: no active slots (nothing connected).")
_install_patches(ltxv)
ltxv._id_rope_theta = 10000.0
m.model_options = dict(m.model_options)
to = dict(m.model_options.get("transformer_options", {}))
to["_id_ref_specs"] = ref_specs
m.model_options["transformer_options"] = to
if reference_guidance_scale != 1.0:
noref_to = dict(to)
noref_to.pop("_id_ref_specs", None)
ref_scale = float(reference_guidance_scale)
def _ref_cfg_function(args):
cond = args["cond"]
uncond = args["uncond"]
cond_scale = args["cond_scale"]
denoised = uncond + (cond - uncond) * cond_scale
noref_model_options = dict(args["model_options"])
noref_model_options["transformer_options"] = noref_to
(noref_pred,) = comfy.samplers.calc_cond_batch(
args["model"], [args["input_cond"]], args["input"], args["timestep"], noref_model_options,
)
noref_denoised = args["input"] - noref_pred
denoised = denoised + (ref_scale - 1.0) * (cond - noref_denoised)
return denoised
m.set_model_sampler_cfg_function(_ref_cfg_function, disable_cfg1_optimization=True)
guide_masked = guide_video is not None and mask_video is not None and auto_mask_guide
dbg = (
"=== LTX Multiple Controls ===\n"
f"{len(ref_specs)} active slot(s):\n " + "\n ".join(summary) + "\n"
f"auto_mask_guide: {'applied' if guide_masked else 'off/skipped'}\n"
f"reference-CFG: {'off' if reference_guidance_scale == 1.0 else f'ON, scale={reference_guidance_scale}'}\n"
"Set LTX_IDOVERLAP_DEBUG=1 for per-step shape logs. Connect negative + CFG 3-5, no LightX2V."
)
log.info("\n" + dbg)
return (m, positive, negative, latent, dbg)
NODE_CLASS_MAPPINGS = {"LTXMultipleControls": LTXMultipleControls}
NODE_DISPLAY_NAME_MAPPINGS = {"LTXMultipleControls": "LTX Multiple Controls"}
+321
View File
@@ -0,0 +1,321 @@
"""LTX multi-subject references with learned slot tags.
Three nodes:
* **LTX Reference Images** — collects up to six reference images into one ordered list, so the
slot order is visible in one place and references of different sizes travel together.
* **LTX Multi Reference Slots** — up to six reference images, each becoming its own reference
block. Every block gets a distinct ``source_id`` (rotary phase) and, when the checkpoint
carries one, the trained per-slot embedding added in feature space.
* **LTX Reference Tags** — builds the ``<Image 1> is the ...`` prompt prefix that binds each tag
to its slot.
The prompt tags are the part people get wrong. Nothing in the model ties ``<Image 1>`` to slot 1:
that correspondence exists only because the training captions used it consistently. Sampling with
a different convention than training used simply does not bind, however well the slots are
separated — hence the second node, so the prompt is built the same way every time.
The slot embedding lives in the LoRA's own safetensors under
``diffusion_model.reference_slot_embedding.*`` with its hyperparameters in the file metadata, so
it is read straight from the LoRA file: ComfyUI's LoRA loader only applies LoRA weights and drops
everything else, which would silently sample the checkpoint without a signal it was trained on.
"""
import math
import torch
MAX_REFERENCES = 6
_PREFIX = "diffusion_model.reference_slot_embedding."
class _SlotEmbedding(torch.nn.Module):
"""Fourier features of the slot index through a small MLP. Mirrors the trainer's module —
same parameter names and shapes, so a checkpoint loads without remapping."""
def __init__(self, token_dim=128, num_frequencies=16, hidden_dim=256):
super().__init__()
self.register_buffer("frequencies", 2.0 ** torch.arange(num_frequencies, dtype=torch.float32))
self.net = torch.nn.Sequential(
torch.nn.Linear(1 + 2 * num_frequencies, hidden_dim),
torch.nn.SiLU(),
torch.nn.Linear(hidden_dim, token_dim),
)
def forward(self, slot_index):
index = torch.tensor([float(slot_index)], dtype=torch.float32, device=self.frequencies.device)
scaled = index * self.frequencies
features = torch.cat([index, torch.sin(scaled), torch.cos(scaled)], dim=-1)
return self.net(features.to(self.net[0].weight.dtype))
def _load_slot_embedding(lora_name):
"""Rebuild the slot embedding from a LoRA file, or None when it carries none."""
import folder_paths
from safetensors import safe_open
path = folder_paths.get_full_path("loras", lora_name)
if path is None:
raise ValueError(f"LoRA not found: {lora_name}")
with safe_open(path, framework="pt") as f:
keys = [k for k in f.keys() if k.startswith(_PREFIX)]
if not keys:
return None
metadata = f.metadata() or {}
state = {k[len(_PREFIX):]: f.get_tensor(k).float() for k in keys}
# Prefer the file's own metadata; fall back to the shapes when it is absent.
hidden_dim = int(metadata.get("reference_slot_embedding_hidden_dim", state["net.0.weight"].shape[0]))
num_frequencies = int(metadata.get("reference_slot_embedding_num_frequencies", 0)) or (
(state["net.0.weight"].shape[1] - 1) // 2
)
token_dim = int(metadata.get("reference_slot_embedding_dim", state["net.2.weight"].shape[0]))
module = _SlotEmbedding(token_dim=token_dim, num_frequencies=num_frequencies, hidden_dim=hidden_dim)
module.load_state_dict(state)
module.eval()
return module
class LTXReferenceImages:
"""Collect reference images into an ordered list. Images only — nothing else wired in.
A plain IMAGE batch cannot carry references of different sizes (torch stacking needs one
shape), and subject references routinely differ: a portrait crop next to a landscape product
shot. Emitting a list keeps each one at its own size, and keeps the slot order explicit and
visible on one node instead of spread across the conditioning node's inputs.
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {},
"optional": {
f"image_{i}": (
"IMAGE",
{"tooltip": f"Reference for slot {i} — the subject your prompt calls <Image {i}>. "
f"Leave unconnected to skip; connected slots renumber in order, so "
f"wiring 1 and 3 makes them <Image 1> and <Image 2>."},
)
for i in range(1, MAX_REFERENCES + 1)
},
}
RETURN_TYPES = ("LTX_REFS", "INT")
RETURN_NAMES = ("references", "count")
FUNCTION = "collect"
CATEGORY = "BFS/LTX"
DESCRIPTION = (
"Collects up to six reference images, in slot order, for LTX Multi Reference Slots. "
"Images of different sizes are fine — each keeps its own."
)
def collect(self, **images):
refs = [
images[f"image_{i}"]
for i in range(1, MAX_REFERENCES + 1)
if images.get(f"image_{i}") is not None
]
return (refs, len(refs))
class LTXReferenceTags:
"""Build the prompt prefix that binds each tag to its reference slot."""
@classmethod
def INPUT_TYPES(cls):
optional = {
f"subject_{i}": (
"STRING",
{"default": "", "multiline": False,
"tooltip": f"What reference image {i} is, as a short noun phrase — 'the woman', "
f"'the red sports car'. Becomes '<Image {i}> is <this>.'. Leave empty to skip."},
)
for i in range(1, MAX_REFERENCES + 1)
}
optional["scene"] = (
"STRING",
{"default": "", "multiline": True,
"tooltip": "The scene description, appended after the declarations. Refer back to the "
"subjects by tag here (\"<Image 1> hands <Image 2> a cup\") — a tag that only "
"ever appears in the declaration has to carry both jobs at once, which is a "
"harder association for the model to use."},
)
return {"required": {}, "optional": optional}
RETURN_TYPES = ("STRING",)
RETURN_NAMES = ("prompt",)
FUNCTION = "build"
CATEGORY = "BFS/LTX"
DESCRIPTION = (
"Assembles '<Image 1> is the woman. <Image 2> is the man. <scene>'. Order matters: tag N "
"must describe the image wired into reference_N on the slots node, and must match the "
"convention the checkpoint was trained with. Describe the subjects DISTINCTLY — two "
"references both called 'the woman' carry no information about which is which."
)
def build(self, scene="", **subjects):
parts = []
for i in range(1, MAX_REFERENCES + 1):
text = (subjects.get(f"subject_{i}") or "").strip().rstrip(".")
if text:
parts.append(f"<Image {i}> is {text}.")
scene = (scene or "").strip()
if scene:
parts.append(scene)
return (" ".join(parts),)
class LTXMultiReferenceSlots:
"""N reference images, each on its own slot: distinct rotary phase plus the trained tag."""
@classmethod
def INPUT_TYPES(cls):
import folder_paths
optional = {}
for i in range(1, MAX_REFERENCES + 1):
optional[f"reference_{i}"] = (
"IMAGE",
{"tooltip": f"Reference image for slot {i} — the subject your prompt calls "
f"<Image {i}>. Leave unconnected to skip; connected slots are numbered "
f"in order, so wiring 1 and 3 gives them source ids 1 and 2."},
)
optional["references"] = (
"LTX_REFS",
{"tooltip": "Output of LTX Reference Images. Use this OR the individual reference_N "
"slots below — the list wins if both are connected."},
)
optional["slot_embedding_lora"] = (
["none"] + folder_paths.get_filename_list("loras"),
{"default": "none",
"tooltip": "The LoRA file to read the trained slot embedding from. Point this at the "
"SAME LoRA you load for sampling. ComfyUI's LoRA loader applies only LoRA "
"weights and drops the slot embedding, so without this the references are "
"untagged and the checkpoint is sampled without a signal it was trained on. "
"'none' = phase separation only (older checkpoints)."},
)
optional["start_source_id"] = (
"INT",
{"default": 1, "min": 0, "max": 8,
"tooltip": "Source id of the first connected reference; the rest count up. The target "
"is always 0. Start at 1 unless the checkpoint was trained otherwise."},
)
optional["phase_scale"] = ("FLOAT", {"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.1})
optional["slot_strength"] = (
"FLOAT",
{"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.05,
"tooltip": "Scales the learned slot tag. The tag and the adapter were trained "
"together at strength 1.0, so raising the LoRA on the loader without "
"raising this leaves them out of proportion — the adapter shouts while "
"the tag that tells the references apart stays quiet. Match this to the "
"LoRA strength you set on the loader (an undertrained LoRA often wants "
"1.2-1.5, and the tag wants the same)."},
)
optional["layout"] = (
["overlap", "st_drc", "strata"],
{"default": "overlap",
"tooltip": "Where the references sit in the RoPE grid. Must match training."},
)
optional["ref_resize_mode"] = (
["match_target", "match_target_letterbox", "native_resolution"],
{"default": "match_target_letterbox",
"tooltip": "Letterbox is the safe default for subject references: it keeps the whole "
"image, where match_target centre-crops and can cut a subject in half when "
"the aspect ratios differ."},
)
optional["reference_temporal_offset_latents"] = ("INT", {"default": 0, "min": -8, "max": 8})
return {
"required": {
"model": ("MODEL",),
"positive": ("CONDITIONING",),
"negative": ("CONDITIONING",),
"vae": ("VAE",),
"latent": ("LATENT",),
},
"optional": optional,
}
RETURN_TYPES = ("MODEL", "CONDITIONING", "CONDITIONING", "LATENT", "STRING")
RETURN_NAMES = ("model", "positive", "negative", "latent", "info")
FUNCTION = "apply"
CATEGORY = "BFS/LTX"
DESCRIPTION = (
"Attaches up to six reference images as separate reference blocks, each with its own "
"source id and, when the LoRA carries one, its trained slot embedding. Pair with LTX "
"Reference Tags so the prompt names the slots the way training did."
)
def apply(self, model, positive, negative, vae, latent, references=None,
slot_embedding_lora="none", start_source_id=1, phase_scale=1.0, slot_strength=1.0,
layout="overlap", ref_resize_mode="match_target_letterbox",
reference_temporal_offset_latents=0, **slots):
from .ltx_identity_overlap import _find_ltxv, _install_patches
from .ltx_multiple_controls import _encode_ref
images = list(references) if references else [
slots[f"reference_{i}"]
for i in range(1, MAX_REFERENCES + 1)
if slots.get(f"reference_{i}") is not None
]
if not images:
return (model, positive, negative, latent, "no references connected — pass-through")
slot_module = None
if slot_embedding_lora != "none":
slot_module = _load_slot_embedding(slot_embedding_lora)
if slot_module is None:
raise ValueError(
f"{slot_embedding_lora} carries no slot embedding "
f"(no '{_PREFIX}*' keys). Set slot_embedding_lora to 'none' if this "
"checkpoint was trained with phase separation only."
)
m = model.clone()
ltxv = _find_ltxv(m)
w_sf, h_sf = ltxv.vae_scale_factors[2], ltxv.vae_scale_factors[1]
ref_specs = []
lines = []
for index, img in enumerate(images):
source_id = int(start_source_id) + index
ref_lat, _, _, _, _, _ = _encode_ref(
vae, latent, img, ref_resize_mode, "center", w_sf, h_sf
)
spec = {
"latent": ref_lat,
"seg_value": float(source_id) * float(phase_scale),
"layout": layout,
"strata_slot": index,
"temporal_offset_latents": int(reference_temporal_offset_latents),
}
if slot_module is not None:
with torch.no_grad():
spec["slot_vector"] = slot_module(source_id).detach() * float(slot_strength)
ref_specs.append(spec)
lines.append(f"<Image {index + 1}> -> source_id {source_id}")
_install_patches(ltxv)
ltxv._id_rope_theta = 10000.0
m.model_options = dict(m.model_options)
transformer_options = dict(m.model_options.get("transformer_options", {}))
transformer_options["_id_ref_specs"] = ref_specs
m.model_options["transformer_options"] = transformer_options
tag = (f"trained slot embedding x{slot_strength:g}" if slot_module is not None
else "phase only (no slot embedding)")
info = f"{len(images)} references, {layout}, {tag}\n" + "\n".join(lines)
return (m, positive, negative, latent, info)
NODE_CLASS_MAPPINGS = {
"LTXReferenceImages": LTXReferenceImages,
"LTXMultiReferenceSlots": LTXMultiReferenceSlots,
"LTXReferenceTags": LTXReferenceTags,
}
NODE_DISPLAY_NAME_MAPPINGS = {
"LTXReferenceImages": "LTX Reference Images",
"LTXMultiReferenceSlots": "LTX Multi Reference Slots",
"LTXReferenceTags": "LTX Reference Tags",
}
+200
View File
@@ -0,0 +1,200 @@
"""LTX-2.3 Multishot — prompt builder + ordered reference batch for the shotplan strata LoRA.
The multishot LoRA (`multishot_strata_r128_v1`, trained on Pensioner/shotplan keyframes) takes
ONE text prompt for the whole clip, in the ShotPlan caption format:
<global preamble: setting, palette, lighting, camera grammar, mood>
Shot 1: <shot size> <camera move> <what happens>
Shot 2: ...
plus ONE reference keyframe per shot, stacked as strata slots. This node assembles both from a
per-shot UI so you don't hand-write the format.
WIRING (no dedicated conditioning node is needed -- LTX Identity Transfer already does the right
thing for this checkpoint):
references (IMAGE batch) -> LTX Identity Transfer with layout="strata", source_id=2,
phase_scale=1.0
prompt (STRING) -> your LTX text encode -> positive
LTX Identity Transfer builds each ref's spec as
``seg_value = (source_id + i) * phase_scale`` and ``strata_slot = i`` over the image batch, so
source_id=2 over 4 images reproduces the trained convention exactly: source_id 2/3/4/5 on strata
slots 0/1/2/3. The batch ORDER IS THE SHOT ORDER -- the LoRA was trained with
`shuffle_reference_slots: false`, so slot N means shot N and nothing else.
Shots are COMPACTED, not positional: if you fill image_1 and image_3 and leave image_2 empty,
you get "Shot 1"/"Shot 2" on strata slots 0/1. Slot 0 must always be the opening shot -- the LoRA
never saw a gap in the middle of the slot sequence.
Unfilled shots are padded with a flat mid-gray frame, matching the `blank.jpg` (RGB 128)
placeholder the dataset builder used for samples with fewer than 4 shots -- the model was trained
to read that as "this slot carries no information".
The dropdown vocabularies are the terms actually present in the 17,525 shot headers of
`train_meta_16fps.json`, ordered by frequency (close-up 32%, medium shot 26%, medium close-up 20%
...; static 65%, pan 16%, tracking 9.5% ...). Terms outside these lists are out of distribution
for the LoRA.
Note the text encoder truncates at 1024 tokens and real ShotPlan captions already run long at 3-4
shots -- keep each action line to a sentence or two, or the last shot's description gets cut.
"""
import logging
import torch
log = logging.getLogger("LTXMultishotPrompt")
MAX_SHOTS = 4
# Vocabularies EXTRACTED from the 12,470 `Shot N:` headers of Pensioner/shotplan
# train_meta_16fps.json -- the header is the span between "Shot N:" and the start of the action
# sentence, and these lists cover 98% (size) / 97% (move) of them. Percentages are of all headers.
#
# Do NOT count these terms over the whole caption: the action text says things like "A close-up
# of the young man", which inflates close-up ~7x if the window isn't cut at the action boundary.
SHOT_SIZES = [
"Medium shot", # 37.4%
"Medium close-up", # 27.3%
"Wide shot", # 9.0%
"Extreme close-up", # 8.0%
"Medium long shot", # 7.4%
"Close-up", # 4.5%
"Long shot", # 1.7%
"Medium wide shot", # 1.1%
"Full shot", # 0.2%
"(auto)",
]
# Modifiers COMBINE with a shot size ("over-the-shoulder close-up static"); they are not
# alternatives to one. Emitted before the size, matching the most frequent header forms.
SHOT_MODIFIERS = [
"(none)",
"Over-the-shoulder", # 4.8%
"Low angle", # 3.1%
"High angle", # 2.9%
"POV", # 0.2%
"Overhead", # 0.1%
]
CAMERA_MOVES = [
"Static", # 55.1%
"Tracking", # 10.1%
"Pan", # 6.6%
"Static / locked-off", # 5.1%
"Pan right", # 4.2%
"Pan left", # 3.6%
"Handheld", # 3.3%
"Slow push-in", # 2.7%
"Tilt down", # 1.4%
"Dolly-in", # 1.0%
"Static with slight pan", # 0.8%
"Tracking / follow", # 0.8%
"Tilt up", # 0.7%
"(auto)",
]
def _gray_like(ref: torch.Tensor) -> torch.Tensor:
"""A single mid-gray frame shaped like `ref` ([1,H,W,C]) -- the dataset's blank.jpg is a
flat RGB 128, i.e. 0.5 in ComfyUI's 0-1 float IMAGE convention."""
return torch.full((1, ref.shape[1], ref.shape[2], ref.shape[3]), 0.5,
dtype=ref.dtype, device=ref.device)
def _resize_to(img: torch.Tensor, h: int, w: int) -> torch.Tensor:
"""Bilinear-resize an IMAGE [N,H,W,C] to h x w so every slot can be stacked into one batch."""
if img.shape[1] == h and img.shape[2] == w:
return img
import comfy.utils
return comfy.utils.common_upscale(img.movedim(-1, 1), w, h, "bilinear", "center").movedim(1, -1)
class LTXMultishotPrompt:
@classmethod
def INPUT_TYPES(cls):
required = {
"preamble": ("STRING", {
"multiline": True, "default": "",
"tooltip": "Describes the WHOLE sequence -- setting, colour palette, lighting, "
"overall camera grammar, mood. This is the paragraph that precedes "
"'Shot 1:' in the training captions. Leave empty to emit only the "
"shot blocks."}),
}
optional = {}
for i in range(1, MAX_SHOTS + 1):
optional[f"image_{i}"] = ("IMAGE", {
"tooltip": f"Reference keyframe for shot {i} (strata slot {i - 1}, source_id "
f"{i + 1}). Leave unconnected to end the sequence -- the slot is "
f"padded with the trained mid-gray blank."})
optional[f"shot_modifier_{i}"] = (SHOT_MODIFIERS, {"default": "(none)",
"tooltip": "Combines WITH the shot size (e.g. 'Over-the-shoulder' + 'Close-up' "
"-> 'Over-the-shoulder close-up'). Not a size on its own."})
optional[f"shot_size_{i}"] = (SHOT_SIZES, {"default": "Medium shot"})
optional[f"camera_move_{i}"] = (CAMERA_MOVES, {"default": "Static"})
optional[f"action_{i}"] = ("STRING", {
"multiline": True, "default": "",
"tooltip": f"What happens in shot {i} -- subject, wardrobe, action, framing "
f"details. One or two sentences; the encoder caps the whole prompt "
f"at 1024 tokens."})
return {"required": required, "optional": optional}
RETURN_TYPES = ("STRING", "IMAGE", "INT")
RETURN_NAMES = ("prompt", "references", "num_shots")
FUNCTION = "build"
CATEGORY = "BFS/video"
DESCRIPTION = ("Builds a ShotPlan-format multishot prompt plus the ordered, gray-padded "
"reference batch for the multishot strata LoRA. Feed `references` into LTX "
"Identity Transfer with layout='strata', source_id=2, phase_scale=1.0, and "
"`prompt` into your LTX text encode.")
def build(self, preamble, **kw):
shots, images = [], []
for i in range(1, MAX_SHOTS + 1):
img = kw.get(f"image_{i}")
action = (kw.get(f"action_{i}") or "").strip()
if img is None and not action:
continue
mod = kw.get(f"shot_modifier_{i}", "(none)")
size = kw.get(f"shot_size_{i}", "(auto)")
move = kw.get(f"camera_move_{i}", "(auto)")
# Dataset form: "<modifier> <size> <move>", e.g. "Over-the-shoulder close-up static".
# The size is lowercased after a modifier so the header reads as one phrase.
if mod and mod != "(none)" and size != "(auto)":
size = f"{mod} {size[0].lower()}{size[1:]}"
elif mod and mod != "(none)":
size = mod
header = " ".join(p for p in (size, move) if p and p != "(auto)")
body = " ".join(p for p in (header, action) if p)
shots.append(f"Shot {len(shots) + 1}: {body}".rstrip())
images.append(img)
if not shots:
raise ValueError(
"LTX Multishot Prompt: connect at least one image or write at least one action.")
parts = ([preamble.strip()] if preamble.strip() else []) + shots
prompt = " ".join(parts)
# Every slot must share one resolution to stack into a single IMAGE batch; the first
# connected reference sets it (that is the shot the model opens on).
first = next((im for im in images if im is not None), None)
if first is None:
raise ValueError(
"LTX Multishot Prompt: at least one image_N must be connected -- the LoRA is "
"conditioned on keyframes, an action-only prompt has nothing to anchor to.")
h, w = first.shape[1], first.shape[2]
batch = []
for im in images:
batch.append(_gray_like(first) if im is None else _resize_to(im[:1], h, w))
# Pad the unused slots: the LoRA always sees 4 strata slots, blank == "no information".
while len(batch) < MAX_SHOTS:
batch.append(_gray_like(first))
references = torch.cat(batch, dim=0)
log.info("[LTXMultishotPrompt] %d shot(s), refs %s, prompt %d chars",
len(shots), tuple(references.shape), len(prompt))
return (prompt, references, len(shots))
NODE_CLASS_MAPPINGS = {"LTXMultishotPrompt": LTXMultishotPrompt}
NODE_DISPLAY_NAME_MAPPINGS = {"LTXMultishotPrompt": "LTX Multishot Prompt + Refs"}
+34
View File
@@ -0,0 +1,34 @@
"""LTX Trim First Frames — drop the first N frames of an IMAGE batch.
Palliative for the overlap reference "frame-0 leak": with layout=overlap the reference
sits at the target's frame-0 RoPE positions, so the generated first frame tends to pull
toward the reference image before the sequence follows the prompt. Dropping the first
frame(s) removes that artifact from the final video.
"""
import torch
class LTXTrimFirstFrames:
@classmethod
def INPUT_TYPES(cls):
return {"required": {
"images": ("IMAGE",),
"trim": ("INT", {"default": 1, "min": 0, "max": 16,
"tooltip": "Number of leading frames to drop (the overlap frame-0 leak is usually 1 frame)."}),
}}
RETURN_TYPES = ("IMAGE",)
RETURN_NAMES = ("images",)
FUNCTION = "apply"
CATEGORY = "LTX/identity"
DESCRIPTION = "Drop the first N frames of a video (IMAGE batch) — removes the overlap reference frame-0 leak."
def apply(self, images, trim=1):
n = int(trim)
if n <= 0 or images.shape[0] <= n:
return (images,)
return (images[n:],)
NODE_CLASS_MAPPINGS = {"LTXTrimFirstFrames": LTXTrimFirstFrames}
NODE_DISPLAY_NAME_MAPPINGS = {"LTXTrimFirstFrames": "LTX Trim First Frames"}
+30 -11
View File
@@ -11,15 +11,21 @@ Usage in ComfyUI:
Load LoRA (standard) → LTXV Edit Anything Apply → KSampler
"""
import types
import logging
import os
import types
import folder_paths
import torch
import torch.nn.functional as F
import folder_paths
from safetensors.torch import load_file, save_file
from .reference_temporal_offset import (
DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
reference_temporal_offset_input,
reference_temporal_pixel_offset,
)
logger = logging.getLogger("ltxv.editanything")
# ── transformer_options keys (must not collide with other nodes) ──────────────
@@ -533,6 +539,8 @@ class LTXVEditAnythingApply:
"default": False,
"tooltip": "Print latent shapes and token counts to console.",
}),
# Reference-only; appended last to preserve saved-workflow widget indices.
"ref_temporal_offset_latents": reference_temporal_offset_input(),
},
}
@@ -544,6 +552,7 @@ class LTXVEditAnythingApply:
def apply(
self, model, positive, negative, vae, latent, ref_image, lora_name, resize_mode="pad_to_fit",
guide_frames=None, guide_strength=1.0, ref_strength=1.0,
ref_temporal_offset_latents=DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
role_strength=1.0, adaln_scale=1.0, enable_adaln=True, enable_role_embedding=True, debug=False,
):
import comfy.utils
@@ -626,9 +635,9 @@ class LTXVEditAnythingApply:
ref_px = canvas[:, :, :, :3]
ref_lat = vae.encode(ref_px) # [1, 128, 1, lat_h, lat_w]
# Temporal position for ref: t=-1 slot → -time_sf pixel frames (BEFORE frame 0)
# Training layout: [ref @ t=-8 | guide @ t=0..T | noisy_target @ t=0..T]
ref_pixel_pos = -time_sf # = -8 for LTX VAE (one latent frame before frame 0)
# Reference-only temporal placement. Default 0 matches the standard reference position;
# negative values can test [ref before t=0 | guide/target @ t=0..T]. Guide remains at frame 0.
ref_pixel_pos = reference_temporal_pixel_offset(ref_temporal_offset_latents, time_sf)
# Append ref first (entry[0] = appearance)
# causal_fix=True matches training validation_sampler.py:418 (get_pixel_coords causal_fix=True)
@@ -1157,8 +1166,8 @@ def _ea_install_ref_attn_modules(ltxv, context_dim, init_seed, init_from="attn2"
"attn1" — copy attn1 weights
"none" — random seeded init + zero out to_out (training only for "none" runs)
"""
from comfy.ldm.lightricks.model import CrossAttention
import comfy.ops
from comfy.ldm.lightricks.model import CrossAttention
if init_seed is not None and init_seed >= 0:
cpu_state = torch.random.get_rng_state()
@@ -1476,6 +1485,8 @@ class LTXVEditAnythingLoopingSampler:
"tooltip": "Strength multiplier for the ref_attn LoRA deltas.",
}),
"debug_ea": ("BOOLEAN", {"default": False}),
# Reference-only; appended last to preserve saved-workflow widget indices.
"ref_temporal_offset_latents": reference_temporal_offset_input(),
},
}
@@ -1520,7 +1531,7 @@ class LTXVEditAnythingLoopingSampler:
def _build_chunk_conds(
positive, negative, chunk_latent, chunk_mask,
ref_lat, guide_chunk_lat, scale_factors,
ref_strength, guide_strength, enable_ic_lora,
ref_strength, guide_strength, ref_temporal_offset_latents, enable_ic_lora,
):
"""Append ref + guide_chunk to chunk's positive/negative; return updated tuple."""
import comfy_extras.nodes_lt as nodes_lt
@@ -1530,10 +1541,11 @@ class LTXVEditAnythingLoopingSampler:
time_sf = scale_factors[0]
appended_latent_frames = 0
# entry[0] = ref @ t=-time_sf (appearance), training layout
# entry[0] = appearance reference at its own temporal offset. Guide remains at t=0.
if ref_lat is not None:
ref_pixel_pos = reference_temporal_pixel_offset(ref_temporal_offset_latents, time_sf)
positive, negative, chunk_latent, chunk_mask = nodes_lt.LTXVAddGuide.append_keyframe(
positive, negative, -time_sf, chunk_latent, chunk_mask,
positive, negative, ref_pixel_pos, chunk_latent, chunk_mask,
ref_lat, ref_strength, scale_factors, causal_fix=True,
)
positive, negative = nodes_lt._append_guide_attention_entry(
@@ -1593,8 +1605,8 @@ class LTXVEditAnythingLoopingSampler:
@staticmethod
def _sample_chunk(model, noise, sampler, sigmas, guider, latent, seed_offset=0):
import comfy.sample
import comfy.model_management
import comfy.sample
import comfy.utils
import latent_preview
@@ -1685,6 +1697,7 @@ class LTXVEditAnythingLoopingSampler:
ref_image=None, guide_frames=None,
ref_resize_mode="pad_to_fit",
ref_strength=1.0, guide_strength=1.0,
ref_temporal_offset_latents=DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
role_strength=1.0, adaln_scale=1.0,
enable_ic_lora=True, enable_role_embedding=True, enable_adaln=True,
reapply_per_chunk=True,
@@ -1884,6 +1897,11 @@ class LTXVEditAnythingLoopingSampler:
print(f" total latent frames : {total_frames}")
print(f" tile/overlap (lat) : {tile}/{overlap} step={step}")
print(f" ref_lat : {None if ref_lat is None else list(ref_lat.shape)}")
print(
" ref t offset: "
f"{int(ref_temporal_offset_latents)} latent = "
f"{reference_temporal_pixel_offset(ref_temporal_offset_latents, time_sf)} pixel frame(s)"
)
print(f" guide_lat : {None if guide_lat is None else list(guide_lat.shape)}")
print(f" role_weight present : {role_weight is not None}")
print(f" adaln_cond present : {adaln_cond is not None}")
@@ -1913,7 +1931,7 @@ class LTXVEditAnythingLoopingSampler:
chunk_pos, chunk_neg, chunk_latent, chunk_mask,
ref_lat if do_append else None,
guide_chunk_lat if do_append else None,
scale_factors, ref_strength, guide_strength,
scale_factors, ref_strength, guide_strength, ref_temporal_offset_latents,
enable_ic_lora=do_append,
)
@@ -1980,6 +1998,7 @@ class LTXVEditAnythingLoopingSampler:
"enable_visual_crossattn": bool(enable_visual_crossattn),
"reapply_per_chunk": bool(reapply_per_chunk),
"ref_strength": float(ref_strength),
"ref_temporal_offset_latents": int(ref_temporal_offset_latents),
"guide_strength": float(guide_strength),
"role_strength": float(role_strength),
"adaln_scale": float(adaln_scale),
+87
View File
@@ -0,0 +1,87 @@
"""Decode direto do single-frame VAE do MiniMax-H3, na receita do card.
Motivo de existir: gerando pelo DiT e decodificando com o `VAEDecode` padrao, a saida sai
quadriculada. O caminho de referencia (card do iamkaikai, e o Space
`multimodalart/MiniMax-H3-images`) NAO usa `vae.decode()` -- chama o decoder direto. Quatro
detalhes que corrompem a saida em silencio se mudarem, todos preservados aqui:
1. `vae.decoder(vae.post_quant_conv(z))` direto, nunca `vae.decode()`, cujo chunker temporal
nao consegue formar chunk com poucas fatias de latente;
2. `decoded[:, :, -1]` -- a ULTIMA fatia de saida, nao a de indice 0;
3. desnormalizacao do latente: `z * latents_std + latents_mean`;
4. desnormalizacao de pixel em ImageNet (mean .485/.456/.406, std .229/.224/.225) -- sem ela
a imagem sai escura e com contraste estourado, parecendo erro de tone mapping.
"""
import comfy.model_management
import torch
PIXEL_MEAN = (0.485, 0.456, 0.406)
PIXEL_STD = (0.229, 0.224, 0.225)
class MiniMaxH3DirectDecode:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"vae": ("VAE", {"tooltip": "VAE do H3 com o decoder single-frame (use o loader deste pacote)."}),
"samples": ("LATENT", {"tooltip": "Latente do H3. Funciona com o latente gerado pelo DiT, que e onde o VAEDecode padrao falha."}),
"latent_index": ("INT", {"default": 0, "min": -64, "max": 64, "tooltip": "Qual fatia temporal decodificar. 0 = primeira, -1 = ultima."}),
}
}
RETURN_TYPES = ("IMAGE",)
FUNCTION = "decode"
CATEGORY = "MiniMax-H3"
def decode(self, vae, samples, latent_index):
model = getattr(vae, "first_stage_model", None)
if model is None or not hasattr(model, "decoder"):
raise ValueError(
"este VAE nao e o do MiniMax-H3 (sem first_stage_model.decoder). "
"Carregue com o loader single-frame deste pacote."
)
z = samples["samples"]
# O latente do H3 chega aninhado (video + audio). Os nodes nativos desempacotam e ficam
# com o primeiro componente, que e o video -- ver VAEDecode em nodes.py:335.
if getattr(z, "is_nested", False):
z = z.unbind()[0]
if z.dim() == 4: # unbind entrega o item sem a dimensao de batch
z = z.unsqueeze(0)
if z.dim() == 4:
z = z.unsqueeze(0)
if z.dim() != 5:
raise ValueError(f"latente esperado [B, C, T, H, W]; veio {tuple(z.shape)}")
# Sem isto o decode roda onde os pesos estiverem -- e com o gerenciamento de memoria do
# ComfyUI o VAE fica na CPU ate ser usado, o que torna o decode lentissimo. O vae.decode()
# nativo chama load_models_gpu antes de rodar; aqui e o mesmo passo.
comfy.model_management.load_models_gpu([vae.patcher])
device = vae.device
dtype = vae.vae_dtype
i = latent_index if latent_index >= 0 else z.shape[2] + latent_index
i = max(0, min(i, z.shape[2] - 1))
z = z[:, :, i : i + 1].to(device=device, dtype=torch.float32)
# (3) desnormalizacao do latente
mean = model.latents_mean.view(1, -1, 1, 1, 1).to(device=device, dtype=torch.float32)
std = model.latents_std.view(1, -1, 1, 1, 1).to(device=device, dtype=torch.float32)
z = z * std + mean
# (1) decoder direto, sem vae.decode()
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.float16, enabled=(device.type == "cuda")):
decoded = model.decoder(model.post_quant_conv(z.to(dtype)))
# (2) ultima fatia de saida + (4) desnormalizacao ImageNet
pm = torch.tensor(PIXEL_MEAN, device=decoded.device).view(1, 3, 1, 1)
ps = torch.tensor(PIXEL_STD, device=decoded.device).view(1, 3, 1, 1)
frame = (decoded[:, :, -1].float() * ps + pm).clamp(0, 1)
print(f"[BFSNodes] H3 direct decode: latente {tuple(z.shape)} -> {tuple(frame.shape)} (fatia {i}, {device})")
saida = frame.permute(0, 2, 3, 1).to(comfy.model_management.intermediate_device())
return (saida,)
NODE_CLASS_MAPPINGS = {"MiniMaxH3DirectDecode": MiniMaxH3DirectDecode}
NODE_DISPLAY_NAME_MAPPINGS = {"MiniMaxH3DirectDecode": "MiniMax-H3 Single-Frame Decode / direct (BFS)"}
+422
View File
@@ -0,0 +1,422 @@
"""Aligned downscaled H3 guides matching ai-toolkit target_grid_stride_v1.
No global ComfyUI monkey patches: a cloned ModelPatcher replaces the packed
layout in its UNet wrapper only when our conditioning marker is present.
"""
from __future__ import annotations
import copy
import logging
import math
import json
try:
from .minimax_h3_reference_rope import validate_options, PHASE_VERSION, add_source_phase
from .minimax_h3_rope_layout import apply_rope_layout
except ImportError:
from minimax_h3_reference_rope import validate_options, PHASE_VERSION, add_source_phase
from minimax_h3_rope_layout import apply_rope_layout
import numpy as np
from PIL import Image, ImageOps
import torch
LOG = logging.getLogger(__name__)
MARKER = 'bfs_h3_aligned_guide'
WRAPPER_MARKER = 'bfs_h3_downscaled_guide_wrapper_v1'
SPATIAL_VERSION = 'target_crop_v2'
POSITION_VERSION = 'target_grid_stride_v1'
def prepare_guide_frames(images, width, height, factor):
if isinstance(factor, bool) or not isinstance(factor, int) or factor < 1:
raise ValueError('downscale_factor must be a positive integer')
if width <= 0 or height <= 0 or width % (32 * factor) or height % (32 * factor):
raise ValueError(f'Output dimensions must be positive multiples of {32 * factor}')
if images.ndim != 4 or images.shape[0] < 1 or images.shape[-1] < 3:
raise ValueError('image must be a nonempty IMAGE batch [frames, height, width, channels]')
result = []
for frame in images[..., :3].detach().float().cpu():
pixels = (frame.clamp(0, 1).numpy() * 255).round().astype(np.uint8)
source = Image.fromarray(pixels)
target = ImageOps.fit(source, (width, height), Image.Resampling.BICUBIC)
guide = target.resize((width // factor, height // factor), Image.Resampling.LANCZOS)
result.append(torch.from_numpy(np.asarray(guide).copy()).float() / 255)
return torch.stack(result)
def sample_posterior(moments, latent_mean, latent_std, seed=42, fp16_round=False):
mean, logvar = moments.float().chunk(2, dim=1)
generator = torch.Generator(device='cpu').manual_seed(seed)
noise = torch.randn(mean.shape, generator=generator, dtype=mean.dtype).to(mean.device)
z = mean + torch.exp(0.5 * logvar.clamp(-30, 20)) * noise
if fp16_round:
z = z.half().float()
shape = (1, -1, 1, 1, 1)
return (z - latent_mean.to(z).view(shape)) / latent_std.to(z).view(shape)
def encode_moments(stage, pixels, device, dtype):
# The toolkit normalizes pixels in FP32 before casting for the encoder.
# Native encode_temporal normalizes in the input dtype, so reproduce its
# chunk plan here rather than changing the shared VAE instance.
def encode_clip(clip):
normalized = stage._normalize_pixels(clip.float().to(device)).to(dtype)
return stage._adaptive_encode(normalized)
if pixels.shape[2] == 1:
return encode_clip(pixels)[:, :, -1:]
chunks = []
for start in range(0, pixels.shape[2], stage.clip_length):
clip = pixels[:, :, start:start + stage.clip_length]
missing = stage.clip_length - clip.shape[2]
if missing:
clip = torch.cat([clip, clip[:, :, -1:].repeat(1, 1, missing, 1, 1)], dim=2)
chunks.append(encode_clip(clip))
moments = torch.cat(chunks, dim=2)
if stage.token_drop > 0:
moments = moments[:, :, :-stage.token_drop]
return moments
@torch.no_grad()
def encode_guide(vae, frames):
import comfy.model_management
stage = getattr(vae, 'first_stage_model', None)
required = ('_adaptive_encode', '_normalize_pixels', 'clip_length', 'token_drop', 'latents_mean', 'latents_std')
if stage is None or any(not hasattr(stage, attr) for attr in required):
raise ValueError('Use a complete native MiniMax-H3 video VAE with its encoder')
comfy.model_management.load_models_gpu([vae.patcher])
x = frames.permute(3, 0, 1, 2).unsqueeze(0).float() * 2 - 1
moments = encode_moments(stage, x, vae.device, vae.vae_dtype)
# The fork's sampling path uses seed 42 and FP16 posterior rounding for
# both image keyframes and aligned guide videos.
latents = sample_posterior(moments, stage.latents_mean, stage.latents_std,
fp16_round=True)
return latents.cpu()
def downscaled_layout(native, keyframes, refs=()):
"""Select coarse condition rows from the native target grid, reindex all streams.
Native PackedLayout creates full-canvas keyframe rows even for small
latents. Keeping only stride-factor target patch origins fixes both row
counts and RoPE while preserving refs, audio, target rows and their clocks.
"""
required = ('signature', 'segments', 'position_ids', 'seq_len', 'img_pos', 'img_update',
'audio_pos', 'audio_update')
if any(not hasattr(native, name) for name in required):
raise RuntimeError('Unsupported ComfyUI H3 PackedLayout API; update ComfyUI')
_, _, height, width, _ = native.signature
if height % 2 or width % 2:
raise ValueError('H3 requires complete 2x2 spatial latent patches')
patch_h, patch_w = height // 2, width // 2
full_rows = patch_h * patch_w
video_guides = iter(kf for kf in keyframes if kf.get('latent') is not None)
keep_segments, new_segments = [], []
cursor = 0
for start, stop, kind in native.segments:
indices = torch.arange(start, stop)
if kind == 'cond':
kf = next(video_guides)
latent = kf['latent']
factor = kf.get(MARKER, {}).get('downscale_factor', 1)
if isinstance(factor, bool) or not isinstance(factor, int) or factor < 1:
raise ValueError('Invalid guide factor in conditioning')
if height % (2 * factor) or width % (2 * factor):
raise ValueError('Guide factor does not divide the target patch grid')
if latent.ndim != 5 or latent.shape[:2] != (1, 24):
raise ValueError('Guide latent must have shape [1, 24, time, height, width]')
if tuple(latent.shape[-2:]) != (height // factor, width // factor):
raise ValueError('Guide latent dimensions do not match the target and downscale factor')
vt = latent.shape[2]
if stop - start != vt * full_rows:
raise RuntimeError('Unsupported native H3 guide segment size')
grid = torch.arange(full_rows).reshape(patch_h, patch_w)[::factor, ::factor].flatten()
indices = (start + torch.arange(vt)[:, None] * full_rows + grid[None]).flatten()
keep_segments.append(indices)
new_segments.append((cursor, cursor + len(indices), kind))
cursor += len(indices)
keep = torch.cat(keep_segments)
remap = torch.full((native.seq_len,), -1, dtype=torch.long)
remap[keep] = torch.arange(cursor)
result = copy.copy(native)
result.seq_len = cursor
result.position_ids = native.position_ids[keep]
result.segments = new_segments
for modality in ('img', 'audio'):
positions = getattr(native, modality + '_pos')
selected = remap[positions] >= 0
setattr(result, modality + '_pos', remap[positions[selected]])
setattr(result, modality + '_update', getattr(native, modality + '_update')[selected])
# Metadata used only by this wrapper; native signatures remain intact.
result.bfs_h3_guide_layout = True
return apply_rope_layout(result, keyframes, refs)
def patch_guide_model(model, phase_enabled=False):
if model.model_options.get(WRAPPER_MARKER):
patched=model.clone()
if phase_enabled: install_phase_patch(patched)
return patched
patched = model.clone()
previous = patched.model_options.get('model_function_wrapper')
cache = [None, None]
def wrapper(model_function, args):
conds = args.get('c', {})
payload = conds.get('minimax_payload')
keyframes = (payload or {}).get('keyframes', [])
refs = (payload or {}).get('refs', [])
if any(MARKER in kf for kf in keyframes + refs):
native = payload.get('layout')
if native is None:
raise RuntimeError('H3 layout missing from payload; unsupported ComfyUI version')
if cache[0] is not native:
cache[:] = [native, downscaled_layout(native, keyframes, refs)]
copied_payload = dict(payload, layout=cache[1])
args = dict(args, c=dict(conds, minimax_payload=copied_payload))
else:
cache[:] = [None, None]
if previous is not None:
return previous(model_function, args)
return model_function(args['input'], args['timestep'], **args['c'])
patched.set_model_unet_function_wrapper(wrapper)
patched.model_options[WRAPPER_MARKER] = True
patched.model_options['bfs_h3_phase_getter'] = lambda: getattr(cache[1], 'bfs_source_phase_values', None)
if phase_enabled: install_phase_patch(patched)
return patched
def install_phase_patch(patched):
if patched.model_options.get('bfs_h3_source_phase_patch'): return
original=patched.get_model_object('diffusion_model.rope_freqs')
getter=patched.model_options['bfs_h3_phase_getter']
def rope(position_ids, device):
return add_source_phase(original(position_ids,device), getter())
patched.add_object_patch('diffusion_model.rope_freqs',rope)
patched.model_options['bfs_h3_source_phase_patch']=True
def rope_inputs():
return {'guide_rope_layout': (['overlap','sidecar'],),
'reference_rope_layout': (['native','overlap','sidecar'],),
'reference_source_phase': ('BOOLEAN',{'default':False}),
'reference_phase_scale': ('FLOAT',{'default':1.0,'min':0.0}),
'reference_sidecar_margin': ('FLOAT',{'default':0.0,'min':0.0}),
'source_id': ('INT',{'default':1,'min':1,'max':99})}
def rope_marker(options, source_id, role):
if isinstance(source_id,bool) or not isinstance(source_id,int) or source_id<1:
raise ValueError('source_id must match a positive training control channel')
return {'rope_layout': options['guide_rope_layout'] if role=='guide' else options['reference_rope_layout'],
'source_phase': options['reference_source_phase'], 'source_id': source_id,
'phase_scale': options['reference_phase_scale'], 'sidecar_margin': options['reference_sidecar_margin']}
def check_lora_metadata(lora_name, factor, options=None):
if not lora_name or lora_name == '(manual)':
return
import folder_paths
from safetensors import safe_open
path = folder_paths.get_full_path_or_raise('loras', lora_name)
with safe_open(path, framework='pt', device='cpu') as handle:
metadata = handle.metadata() or {}
for key, expected in (('minimax_h3_guide_spatial_version', SPATIAL_VERSION),
('minimax_h3_guide_position_version', POSITION_VERSION)):
if metadata.get(key) != expected:
raise ValueError(f'LoRA {key} must be {expected!r}; got {metadata.get(key)!r}')
if int(metadata.get('reference_downscale_factor', 0)) != factor:
raise ValueError('downscale_factor must match the LoRA training metadata')
stored=json.loads(metadata.get('minimax_h3_reference_rope','{}'))
expected=validate_options() if options is None else options
stored_options=validate_options(stored.get('guide_rope_layout','overlap'),stored.get('reference_rope_layout','native'),stored.get('reference_source_phase',False),stored.get('reference_phase_scale',1.0),stored.get('reference_sidecar_margin',0.0))
if stored_options != expected or stored.get('phase_version',PHASE_VERSION)!=PHASE_VERSION:
raise ValueError('Reference RoPE options must match LoRA training metadata')
if not any(metadata.get(key, '').lower()=='true' for key in ('guide_latent_only','control_latent_only')):
raise ValueError('This node expects a caption-only VLM with guide_latent_only=true')
class BFSMiniMaxH3DownscaledGuide:
@classmethod
def INPUT_TYPES(cls):
import folder_paths
return {'required': {
'model': ('MODEL',),
'positive': ('CONDITIONING',),
'vae': ('VAE',),
'latent': ('LATENT',),
'image': ('IMAGE', {'tooltip': 'Single image or video frames already resampled to 24 fps.'}),
'downscale_factor': ('INT', {'default': 4, 'min': 1, 'max': 8, 'step': 1}),
'frame_idx': ('INT', {'default': 0, 'min': -9999, 'max': 9999}),
}, 'optional': {
'audio': ('AUDIO', {'tooltip': 'Optional aligned guide soundtrack; never routed to the text encoder.'}),
'audio_vae': ('VAE',),
'lora_name': (['(manual)'] + folder_paths.get_filename_list('loras'),
{'tooltip': 'Validate metadata only. Load the same LoRA using your normal LoRA loader.'}),
**rope_inputs(),
}}
RETURN_TYPES = ('MODEL', 'CONDITIONING', 'IMAGE')
RETURN_NAMES = ('model', 'positive', 'guide_preview')
FUNCTION = 'apply'
CATEGORY = 'MiniMax-H3'
DESCRIPTION = 'Aligned lower-resolution VAE guides for H3 LoRAs trained with target_grid_stride_v1. Connect both returned MODEL and CONDITIONING to the sampler.'
@torch.no_grad()
def apply(self, model, positive, vae, latent, image, downscale_factor=4, frame_idx=0,
lora_name='(manual)', audio=None, audio_vae=None, guide_rope_layout='overlap', reference_rope_layout='native',
reference_source_phase=False, reference_phase_scale=1.0, reference_sidecar_margin=0.0, source_id=1):
from comfy.ldm.minimax import model as h3
samples = latent['samples']
streams = getattr(samples, 'tensors', None)
if not getattr(samples, 'is_nested', False) or streams is None or len(streams) != 2:
raise ValueError('Use a native MiniMax-H3 joint video/audio target latent')
video = streams[0]
if video.ndim != 5 or video.shape[:2] != (1, 24):
raise ValueError('Target video latent must have shape [1, 24, time, height, width]')
diffusion = model.get_model_object('diffusion_model')
if not isinstance(diffusion, h3.MiniMaxH3Model) or tuple(diffusion.patch_size) != (1, 2, 2):
raise ValueError('This node requires the native MiniMax-H3 model with 1x2x2 patches')
options=validate_options(guide_rope_layout,reference_rope_layout,reference_source_phase,reference_phase_scale,reference_sidecar_margin)
check_lora_metadata(lora_name, downscale_factor, options)
width, height = video.shape[-1] * 16, video.shape[-2] * 16
frame_count = sum(h3.FRAME_PER_TOKEN[k % 5] for k in range(video.shape[2]))
frames = image.shape[0]
if frames != 1 and (frames < 5 or frames % 17 != 5):
raise ValueError('Guide must be one image or exactly 17*n+5 video frames (5, 22, 39, 73, ...); trim/resample explicitly')
index = frame_idx if frame_idx >= 0 else frame_count + frame_idx
if index < 0 or index + frames > frame_count:
raise ValueError('Guide frames do not fit in the target timeline at frame_idx')
if not positive:
raise ValueError('positive conditioning must not be empty')
if audio is not None and audio_vae is None:
raise ValueError('audio_vae is required for a guide soundtrack')
preview = prepare_guide_frames(image, width, height, downscale_factor)
z = encode_guide(vae, preview)
expected_t = 1 if frames == 1 else (frames - 5) // 17 * 5 + 2
if tuple(z.shape) != (1, 24, expected_t, height // (16 * downscale_factor), width // (16 * downscale_factor)):
raise ValueError(f'Unexpected H3 VAE latent shape: {tuple(z.shape)}')
guide = {'resolved_frame_index': index, 'latent': z, MARKER: {
'downscale_factor': downscale_factor,
'spatial_version': SPATIAL_VERSION, 'position_version': POSITION_VERSION,
**rope_marker(options, source_id, 'guide'),
}}
if audio is not None:
if audio_vae is None:
raise ValueError('audio_vae is required for a guide soundtrack')
from comfy_extras.nodes_minimax_h3 import _encode_ref_audio
audio_z, _ = _encode_ref_audio(audio_vae, audio)
available = math.floor(streams[1].shape[-1] - h3.FRAME_RESCALE * index)
clip_audio_frames = round(frames / 24 * 40)
if available < 1:
raise ValueError('Guide soundtrack starts outside the target audio timeline')
guide['audio_latent'] = audio_z[..., :min(available, clip_audio_frames)].clone()
# Copy all conditioning entries, preserving existing guides/references.
updated = []
for embedding, values in positive:
values = dict(values)
values['minimax_keyframes'] = list(values.get('minimax_keyframes', [])) + [guide]
updated.append([embedding, values])
LOG.info('BFS H3 guide: %s frames, %sx%s pixels, factor %s, frame %s', frames,
preview.shape[2], preview.shape[1], downscale_factor, index)
return (patch_guide_model(model, reference_source_phase), updated, preview)
class BFSMiniMaxH3IdentityReference:
@classmethod
def INPUT_TYPES(cls):
import folder_paths
extra=rope_inputs()
extra['source_id']=('INT',{'default':2,'min':1,'max':99})
extra['downscale_factor']=('INT',{'default':4,'min':1,'max':8})
extra['lora_name']=(['(manual)']+folder_paths.get_filename_list('loras'),)
return {'required':{'model':('MODEL',),'positive':('CONDITIONING',),'vae':('VAE',),
'latent':('LATENT',),'image':('IMAGE',)},'optional':extra}
RETURN_TYPES=('MODEL','CONDITIONING','IMAGE')
RETURN_NAMES=('model','positive','reference_preview')
FUNCTION='apply'
CATEGORY='MiniMax-H3'
DESCRIPTION='Caption-only image identity reference. Match source_id to the training control channel. Requires an upscale checkpoint trained with the same reference role and RoPE options.'
@torch.no_grad()
def apply(self,model,positive,vae,latent,image,guide_rope_layout='overlap',reference_rope_layout='native',
reference_source_phase=False,reference_phase_scale=1.0,reference_sidecar_margin=0.0,
source_id=2,downscale_factor=4,lora_name='(manual)'):
from comfy.ldm.minimax import model as h3
diffusion=model.get_model_object('diffusion_model')
if not isinstance(diffusion,h3.MiniMaxH3Model) or tuple(diffusion.patch_size)!=(1,2,2):
raise ValueError('Identity reference requires native MiniMax-H3')
if image.ndim!=4 or image.shape[0]!=1:
raise ValueError('Identity reference must contain exactly one image')
if not positive:raise ValueError('positive conditioning must not be empty')
streams=getattr(latent.get('samples'),'tensors',None)
if streams is None or len(streams)!=2:raise ValueError('Use a joint H3 target latent')
video=streams[0]
if video.ndim!=5 or video.shape[:2]!=(1,24):raise ValueError('Target batch must be one H3 video')
options=validate_options(guide_rope_layout,reference_rope_layout,reference_source_phase,reference_phase_scale,reference_sidecar_margin)
check_lora_metadata(lora_name,downscale_factor,options)
if lora_name!='(manual)':
import folder_paths
from safetensors import safe_open
with safe_open(folder_paths.get_full_path_or_raise('loras',lora_name),framework='pt',device='cpu') as f:
if (f.metadata() or {}).get('control_latent_only','').lower()!='true':
raise ValueError('Caption-only identity node requires control_latent_only=true in training')
ih,iw=image.shape[1:3];target_h,target_w=video.shape[-2]*16,video.shape[-1]*16
scale=min(1.0,math.sqrt(target_h*target_w/(ih*iw)))
height=max(32,round(ih*scale/32)*32);width=max(32,round(iw*scale/32)*32)
pixels=(image[0,...,:3].detach().float().cpu().clamp(0,1).numpy()*255).round().astype(np.uint8)
ref=Image.fromarray(pixels).resize((width,height),Image.Resampling.LANCZOS)
preview=torch.from_numpy(np.asarray(ref).copy()).float().unsqueeze(0)/255
z=encode_guide(vae,preview)
block={'kind':'image','latent_h':height//16,'latent_w':width//16,'latent':z,
MARKER:rope_marker(options,source_id,'reference')}
updated=[]
for embedding,values in positive:
copied=dict(values);copied['minimax_refs']=list(values.get('minimax_refs',[]))+[block]
updated.append([embedding,copied])
return patch_guide_model(model,reference_source_phase),updated,preview
class BFSMiniMaxH3GuideTarget:
@classmethod
def INPUT_TYPES(cls):
return {'required': {
'width': ('INT', {'default': 1024, 'min': 32, 'max': 16384, 'step': 32}),
'height': ('INT', {'default': 768, 'min': 32, 'max': 16384, 'step': 32}),
'frames': ('INT', {'default': 73, 'min': 1, 'max': 3600, 'step': 1,
'tooltip': '1 for a true still image, or exactly 17*n+5 video frames.'}),
}}
RETURN_TYPES = ('LATENT',)
FUNCTION = 'create'
CATEGORY = 'MiniMax-H3'
def create(self, width, height, frames):
import comfy.model_management
from comfy.nested_tensor import NestedTensor
if width <= 0 or height <= 0 or width % 32 or height % 32:
raise ValueError('Target width/height must be positive multiples of 32')
if frames != 1 and (frames < 5 or frames % 17 != 5):
raise ValueError('Target frames must be 1 or 17*n+5')
vt = 1 if frames == 1 else (frames - 5) // 17 * 5 + 2
at = round(frames / 24 * 40)
device = comfy.model_management.intermediate_device()
video = torch.zeros(1, 24, vt, height // 16, width // 16, device=device)
audio = torch.zeros(1, 32, 2, at, device=device)
return ({'samples': NestedTensor((video, audio))},)
NODE_CLASS_MAPPINGS = {
'BFSMiniMaxH3DownscaledGuide': BFSMiniMaxH3DownscaledGuide,
'BFSMiniMaxH3GuideTarget': BFSMiniMaxH3GuideTarget,
'BFSMiniMaxH3IdentityReference': BFSMiniMaxH3IdentityReference,
}
NODE_DISPLAY_NAME_MAPPINGS = {
'BFSMiniMaxH3DownscaledGuide': 'MiniMax-H3 Downscaled Latent Guide (BFS)',
'BFSMiniMaxH3GuideTarget': 'MiniMax-H3 Guide Target — Image / Video (BFS)',
'BFSMiniMaxH3IdentityReference': 'MiniMax-H3 Identity Reference + RoPE (BFS)',
}
+68
View File
@@ -0,0 +1,68 @@
"""Experimental H3 reference geometry and source phase; shared with BFSNodes.
Coordinates use H3's normalized RoPE units, not LTX pixel patch bounds.
Source phase v1 composes theta**(-d/half_dim) with the original rotary angles.
"""
import math
import torch
PHASE_VERSION = 'h3_source_phase_v1'
def validate_options(guide_layout='overlap', reference_layout='native', source_phase=False, phase_scale=1.0, sidecar_margin=0.0):
if guide_layout not in ('overlap', 'sidecar'):
raise ValueError('guide_rope_layout must be overlap or sidecar')
if reference_layout not in ('native', 'overlap', 'sidecar'):
raise ValueError('reference_rope_layout must be native, overlap or sidecar')
if not isinstance(source_phase, bool):
raise ValueError('reference_source_phase must be boolean')
for name, value in [('reference_phase_scale', phase_scale), ('reference_sidecar_margin', sidecar_margin)]:
if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value) or value < 0:
raise ValueError(f'{name} must be finite and nonnegative')
return dict(guide_rope_layout=guide_layout, reference_rope_layout=reference_layout,
reference_source_phase=source_phase, reference_phase_scale=float(phase_scale),
reference_sidecar_margin=float(sidecar_margin))
def options_from_kwargs(kwargs):
return validate_options(kwargs.get('guide_rope_layout', 'overlap'), kwargs.get('reference_rope_layout', 'native'),
kwargs.get('reference_source_phase', False), kwargs.get('reference_phase_scale', 1.0),
kwargs.get('reference_sidecar_margin', 0.0))
def shift_reference(positions, target, layout, margin=0.0):
if layout in ('native', 'overlap'):
return positions
if layout != 'sidecar' or not math.isfinite(margin) or margin < 0:
raise ValueError('Invalid sidecar layout or margin')
result=positions.clone()
# Place beyond the final target patch origin, separated by one patch step.
widths=torch.unique(target[:,2],sorted=True)
step=float(widths[1]-widths[0]) if widths.numel()>1 else 1.0
result[:,2]+=target[:,2].max()+step+margin-positions[:,2].min()
result[:,1]+=(target[:,1].min()+target[:,1].max()-positions[:,1].min()-positions[:,1].max())/2
return result
def add_source_phase(angles, source_values, theta=10000.0):
"""Angles [..., S, 2*D], duplicated halves; zero sources stay bit-identical."""
if source_values is None:
return angles
if tuple(source_values.shape)!=tuple(angles.shape[:-1]):
raise ValueError('Source phase rows must match rotary rows')
if not torch.isfinite(source_values).all() or (source_values<0).any():
raise ValueError('Source phase values must be finite and nonnegative')
if not torch.count_nonzero(source_values):
return angles
half=angles.shape[-1]//2
rates=theta**(-torch.arange(half,device=angles.device,dtype=torch.float32)/half)
phase=source_values.to(device=angles.device,dtype=torch.float32)[...,None]*rates
return angles+torch.cat((phase,phase),dim=-1)
def control_source_channels(config, count):
"""Keep explicit channel numbers when earlier control paths are absent."""
channels=[i for i in (1,2,3) if getattr(config,f'control_path_{i}',None)]
if not channels:return list(range(1,count+1))
if len(channels)!=count:raise ValueError('Control channel/path counts do not match')
return channels
+60
View File
@@ -0,0 +1,60 @@
"""Apply H3 reference layouts to a cloned native Comfy layout."""
import torch
try:
from .minimax_h3_reference_rope import shift_reference, add_source_phase, validate_options
except ImportError:
from minimax_h3_reference_rope import shift_reference, add_source_phase, validate_options
MARKER='bfs_h3_aligned_guide'
def ref_span(ref):
if ref['kind']=='image': return 1.0
audio=float(ref.get('ref_audio_t',0))
if ref['kind']=='audio': return audio
vt=ref['latent_t']
return max(audio,sum((1,4,4,4,4)[i%5]*5/3 for i in range(vt)))
def apply_rope_layout(layout, keyframes, refs):
# Remove only experimental refs from the native cumulative target clock.
spans=[ref_span(r) for r in refs]
removed=sum(span for span,r in zip(spans,refs) if r.get(MARKER,{}).get('rope_layout','native')!='native')
positions=layout.position_ids.clone()
source=torch.zeros(layout.seq_len,dtype=torch.float32)
target_rows=next((slice(a,b) for a,b,k in layout.segments if k=='video'))
for a,b,kind in layout.segments:
if kind in ('video','audio','cond','cond_audio'):positions[a:b,0]-=removed
target=positions[target_rows]
guide_segments=iter((a,b,k) for a,b,k in layout.segments if k in ('cond','cond_audio'))
def apply_ranges(ranges, opts):
if opts.get('rope_layout','overlap')=='sidecar' and ranges:
indices=torch.cat([torch.arange(a,b) for a,b,_ in ranges])
positions[indices]=shift_reference(positions[indices],target,'sidecar',opts.get('sidecar_margin',0.0))
if opts.get('source_phase',False):
for a,b,_ in ranges:source[a:b]=opts['source_id']*opts.get('phase_scale',1.0)
for kf in keyframes:
ranges=[]
if kf.get('latent') is not None:ranges.append(next(guide_segments))
if kf.get('audio_latent') is not None:ranges.append(next(guide_segments))
opts=kf.get(MARKER,{})
apply_ranges(ranges, opts)
ref_segments=iter((a,b,k) for a,b,k in layout.segments if k in ('ref_img','ref_audio'))
cursor=float(layout.signature[0])
native_cursor=cursor
target_origin=float(target[0,0])
for ref,span in zip(refs,spans):
ranges=[]
if ref.get('ref_audio_t',0)>0:ranges.append(next(ref_segments))
if ref['kind'] in ('image','video','video_audio'):ranges.append(next(ref_segments))
opts=ref.get(MARKER,{})
name=opts.get('rope_layout','native')
origin=cursor if name=='native' else target_origin
for a,b,kind in ranges:
positions[a:b,0]+=origin-native_cursor
apply_ranges(ranges, opts)
native_cursor+=span
if name=='native':cursor+=span
layout.position_ids=positions
layout.bfs_source_phase_values=source if torch.count_nonzero(source) else None
return layout
+184
View File
@@ -0,0 +1,184 @@
"""Loader do decoder single-frame do MiniMax-H3 (iamkaikai/MiniMax-H3-Single-Frame-VAE-500K).
O checkpoint publicado e decoder-only e usa o naming do diffusers, entao o VAELoader padrao
do ComfyUI nao o carrega: a deteccao em comfy/sd.py exige tambem uma chave de encoder
(`encoder.down.5.block.0.conv1.weight`), e sem ela o arquivo cai numa arquitetura errada e a
saida vira blocos.
Este node resolve os dois lados: converte as chaves para o layout do ComfyUI e completa o que
falta (encoder, quant_conv, latents_mean/std, mask_token) a partir do VAE oficial do H3.
Duas transformacoes NAO sao dedutiveis pelo nome, e errar qualquer uma produz saida errada
sem nenhum erro de load. Ambas foram determinadas medindo correlacao contra o VAE oficial:
to_qkv e intercalado por cabeca -- (heads, 3, dim_head), nao [q;k;v] empilhado.
Confirmado em comfy/ldm/minimax/vae.py: `qkv.view(B, S, -1, 3 * dim_head)`.
ff.w1 tem as metades trocadas em relacao ao diffusers, porque o ComfyUI le
`gate, x = w1(x).chunk(2)` -- gate primeiro.
Validado contra o caminho de referencia do autor (diffusers + load_decoder.py dele) com o
mesmo latente: PSNR 72,92 dB entre as duas saidas, ou seja, ruido de fp16.
"""
import comfy.sd
import comfy.utils
import folder_paths
import torch
HEADS = 32
DIM_HEAD = 64
# Chaves que o comfy/sd.py usa para reconhecer o VAE do MiniMax-H3.
DETECT_KEYS = ("decoder.transformer_blocks.0.scale1", "encoder.down.5.block.0.conv1.weight")
def _convert_decoder_keys(decoder_sd):
"""Naming diffusers -> naming ComfyUI. Devolve (state_dict, n_blocos)."""
out = {}
blocks = set()
for key, tensor in decoder_sd.items():
if ".attn.to_q." in key or ".attn.to_k." in key or ".attn.to_v." in key:
blocks.add(int(key.split("transformer_blocks.")[1].split(".")[0]))
continue
new_key = (
key.replace(".ff.net.0.proj.", ".ff.w1.")
.replace(".ff.net.2.", ".ff.w2.")
.replace(".attn.to_out.0.", ".attn.to_out.")
.replace("decoder.proj_in.", "decoder.x_embedder.")
)
if ".ff.w1." in new_key: # ComfyUI le (gate, valor); diffusers grava (valor, gate)
half = tensor.shape[0] // 2
tensor = torch.cat([tensor[half:], tensor[:half]], dim=0)
out[new_key] = tensor
for b in sorted(blocks):
for suffix in ("weight", "bias"):
parts = [decoder_sd[f"decoder.transformer_blocks.{b}.attn.to_{n}.{suffix}"] for n in ("q", "k", "v")]
tail = parts[0].shape[1:]
stacked = torch.stack([p.view(HEADS, DIM_HEAD, *tail) for p in parts], dim=1) # [H, 3, DH, ...]
out[f"decoder.transformer_blocks.{b}.attn.to_qkv.{suffix}"] = stacked.reshape(HEADS * 3 * DIM_HEAD, *tail)
return out, len(blocks)
class MiniMaxH3SingleFrameVAELoader:
"""Monta um VAE do H3 com o decoder single-frame por cima do VAE oficial."""
@classmethod
def INPUT_TYPES(cls):
vaes = folder_paths.get_filename_list("vae")
return {
"required": {
"base_vae": (vaes, {"tooltip": "VAE oficial do MiniMax-H3 (fornece encoder, quant_conv e as estatisticas de latente)."}),
"single_frame_decoder": (vaes, {"tooltip": "Checkpoint decoder-only do autor (585 tensores, naming diffusers). Tambem aceita um ja convertido."}),
"tiling": ("BOOLEAN", {"default": True, "tooltip": "Tiling espacial. MANTENHA LIGADO acima de ~768px: o curriculo do autor parou em 1024 e a imagem inteira acima disso produz grade na fronteira de 32px (blocagem 2.9x em 1024, 3.5x em 1536, contra ~1.5x com tiling). Desligar so compensa em imagens pequenas, onde rende um pouco mais de nitidez."}),
"tile_size": ("INT", {"default": 512, "min": 128, "max": 2048, "step": 64, "tooltip": "So vale com tiling ligado. 512 e o otimo MEDIDO, nao um chute: numa imagem 1056x640, PSNR 26.00 dB com costura 0.91 (indistinguivel), contra 22.17/1.49 com 256 (o default do ComfyUI, que multiplica costuras) e 21.35/2.42 com 1024 (acima da resolucao treinada). O autor treinou 475k das 500k imagens em <=512px, entao o tile de 512 mantem cada pedaco na regiao com massa de treino."}),
"dtype": (["float16", "float32", "bfloat16"], {"default": "float16"}),
}
}
RETURN_TYPES = ("VAE",)
FUNCTION = "load"
CATEGORY = "MiniMax-H3"
def load(self, base_vae, single_frame_decoder, tiling, tile_size, dtype):
base_sd = comfy.utils.load_torch_file(folder_paths.get_full_path("vae", base_vae))
dec_sd = comfy.utils.load_torch_file(folder_paths.get_full_path("vae", single_frame_decoder))
missing_detect = [k for k in DETECT_KEYS if k not in base_sd]
if missing_detect:
raise ValueError(
f"'{base_vae}' nao parece o VAE oficial do MiniMax-H3 (faltam {missing_detect}). "
"Escolha o VAE completo, com encoder."
)
if any(".attn.to_q." in k for k in dec_sd):
converted, n_blocks = _convert_decoder_keys(dec_sd)
print(f"[BFSNodes] single-frame decoder convertido do naming diffusers ({n_blocks} blocos)")
else:
converted = dict(dec_sd) # ja esta no layout do ComfyUI
print("[BFSNodes] single-frame decoder ja estava no naming do ComfyUI")
sd = dict(base_sd)
replaced, unknown = 0, []
for key, tensor in converted.items():
if key in sd:
if tuple(sd[key].shape) != tuple(tensor.shape):
raise ValueError(f"shape divergente em '{key}': base {tuple(sd[key].shape)} vs decoder {tuple(tensor.shape)}")
sd[key] = tensor.to(sd[key].dtype)
replaced += 1
else:
unknown.append(key)
if unknown:
raise ValueError(f"chaves do decoder que nao existem no VAE base: {unknown[:5]} (total {len(unknown)})")
# Relatorio honesto: um checkpoint decoder-only substitui so decoder/post_quant_conv, mas um
# arquivo ja convertido carrega o VAE inteiro e sobrescreve tambem o encoder do base.
from collections import Counter
por_prefixo = Counter(k.split(".")[0] for k in converted)
do_base = sorted({k.split(".")[0] for k in sd} - set(por_prefixo))
print(f"[BFSNodes] {replaced} tensores substituidos: "
+ ", ".join(f"{p}={n}" for p, n in sorted(por_prefixo.items())))
print(f"[BFSNodes] preservado do base: {', '.join(do_base) if do_base else 'nada (o arquivo cobriu o VAE inteiro)'}")
if "encoder" in por_prefixo:
print("[BFSNodes] aviso: o arquivo tambem trouxe encoder, entao o encoder do base foi sobrescrito")
torch_dtype = {"float16": torch.float16, "float32": torch.float32, "bfloat16": torch.bfloat16}[dtype]
vae = comfy.sd.VAE(sd=sd, dtype=torch_dtype)
model = getattr(vae, "first_stage_model", None)
if model is not None and hasattr(model, "tiling"):
model.tiling = bool(tiling)
model.tile_size = int(tile_size)
print(f"[BFSNodes] tiling={'on' if tiling else 'off'} tile_size={tile_size}")
return (vae,)
class MiniMaxH3VAELoaderTiled:
"""Carrega um VAE do H3 ja completo (por exemplo um single-frame convertido) expondo o tiling.
Igual ao VAELoader nativo, com uma diferenca que muda o resultado: o construtor do
MiniMaxH3VideoVAE usa tile_size=256, e o loader nativo nao expoe esse campo. 256 e o pior
valor medido -- multiplica costuras. Aqui da para por 512.
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"vae_name": (folder_paths.get_filename_list("vae"), {"tooltip": "VAE completo do H3 (com encoder). Para o checkpoint decoder-only do autor, use o outro node."}),
"tiling": ("BOOLEAN", {"default": True, "tooltip": "Tiling espacial. MANTENHA LIGADO acima de ~768px: o curriculo do autor parou em 1024 e a imagem inteira acima disso produz grade na fronteira de 32px."}),
"tile_size": ("INT", {"default": 512, "min": 128, "max": 2048, "step": 64, "tooltip": "512 e o otimo MEDIDO: numa imagem 1056x640, PSNR 26.00 dB / costura 0.91, contra 22.17/1.49 com 256 (o default do ComfyUI) e 21.35/2.42 com 1024. O autor treinou 475k das 500k imagens em <=512px."}),
}
}
RETURN_TYPES = ("VAE",)
FUNCTION = "load"
CATEGORY = "MiniMax-H3"
def load(self, vae_name, tiling, tile_size):
path = folder_paths.get_full_path("vae", vae_name)
sd, metadata = comfy.utils.load_torch_file(path, return_metadata=True)
missing = [k for k in DETECT_KEYS if k not in sd]
if missing:
raise ValueError(
f"'{vae_name}' nao sera reconhecido como MiniMax-H3 pelo ComfyUI (faltam {missing}). "
"Um checkpoint decoder-only cai em geometria de Stable Diffusion e produz blocos -- "
"use o node 'MiniMax-H3 Single-Frame VAE Loader' para combina-lo com o VAE oficial."
)
vae = comfy.sd.VAE(sd=sd, metadata=metadata)
vae.throw_exception_if_invalid()
model = getattr(vae, "first_stage_model", None)
if model is not None and hasattr(model, "tiling"):
model.tiling = bool(tiling)
model.tile_size = int(tile_size)
print(f"[BFSNodes] {vae_name}: tiling={'on' if tiling else 'off'} tile_size={tile_size}")
return (vae,)
NODE_CLASS_MAPPINGS = {
"MiniMaxH3SingleFrameVAELoader": MiniMaxH3SingleFrameVAELoader,
"MiniMaxH3VAELoaderTiled": MiniMaxH3VAELoaderTiled,
}
NODE_DISPLAY_NAME_MAPPINGS = {
"MiniMaxH3SingleFrameVAELoader": "MiniMax-H3 Single-Frame VAE Loader (BFS)",
"MiniMaxH3VAELoaderTiled": "MiniMax-H3 VAE Loader / tile control (BFS)",
}
+285
View File
@@ -0,0 +1,285 @@
"""Multi-Ref Sheet Builder — combine 1-5 reference images into the single
composite reference sheet used by the multi-ref LTX-2 LoRA (source_id=2,
layout=overlap). Same grid convention as the training pipeline's
build_multiref_sheet.py: fixed 512x512 panels, deterministic grid by count
(1x1, 2x1, 3x1, 2x2, 3-top+2-bottom), centered with padding onto a fixed
1536x1024 canvas so every sheet is the same resolution regardless of how
many refs are plugged in.
Panel order = image index order (ref_image_1 -> image0, ref_image_2 ->
image1, ...), matching the training data's image0/image1/... convention.
"""
import torch
from PIL import Image
from .util import tensor_to_pil, pil_to_tensor
CATEGORY = "BFS/multiref"
PANEL_SIZE = 512
CANVAS_W, CANVAS_H = 1536, 1024
BG_COLOR = (255, 255, 255)
# row layout per ref count: list of ints = panels per row, top to bottom.
LAYOUTS = {
1: [1],
2: [2],
3: [3],
4: [2, 2],
5: [3, 2],
}
def _cover_resize_crop(img: Image.Image, size: int) -> Image.Image:
"""Resize+center-crop to exactly fill a size x size square (no stretch, crops excess)."""
img = img.convert("RGB")
w, h = img.size
scale = max(size / w, size / h)
nw, nh = round(w * scale), round(h * scale)
img = img.resize((nw, nh), Image.LANCZOS)
x0, y0 = (nw - size) // 2, (nh - size) // 2
return img.crop((x0, y0, x0 + size, y0 + size))
def _justified_compose(imgs, rows_counts, canvas_w, canvas_h, bg):
"""Row-justified layout (no cropping, no distortion) -- identical algorithm to
build_multiref_sheet.py's _justified_compose, kept in sync so inference-time
sheets match training-time sheets exactly. Each row is split into
`rows_counts[i]` images; every image in a row shares that row's height, and
its width = row_height * (image's own aspect ratio) -- one uniform scale
factor per image, so nothing is stretched.
Each row's natural height fills canvas_w at scale=1. If all rows fit within
canvas_h, they're drawn at natural size (leftover vertical space -> a single
top+bottom bar, whole block centered) -- never blown up past canvas_w. Only
if rows would collectively overflow canvas_h do all rows shrink by one
shared factor < 1, giving a single shared pair of side bars instead of
per-panel scattered padding.
"""
it = iter(imgs)
rows = []
for count in rows_counts:
row_imgs = [next(it) for _ in range(count)]
aspects = [im.width / im.height for im in row_imgs]
rows.append((row_imgs, aspects))
natural_heights = [canvas_w / sum(aspects) for _, aspects in rows]
scale = min(1.0, canvas_h / sum(natural_heights))
row_heights = [round(nat_h * scale) for nat_h in natural_heights]
drift_h = round(sum(natural_heights) * scale) - sum(row_heights)
if row_heights:
row_heights[-1] += drift_h
sheet = Image.new("RGB", (canvas_w, canvas_h), bg)
y = (canvas_h - sum(row_heights)) // 2
for (row_imgs, aspects), row_h in zip(rows, row_heights):
widths = [max(1, round(row_h * a)) for a in aspects]
row_w = sum(widths)
x = (canvas_w - row_w) // 2
for im, w_i in zip(row_imgs, widths):
resized = im.convert("RGB").resize((max(1, w_i), max(1, row_h)), Image.LANCZOS)
sheet.paste(resized, (x, y))
x += w_i
y += row_h
return sheet
def _cover_justified_compose(imgs, rows_counts, canvas_w, canvas_h, bg):
"""Row-justified layout, but COVER the canvas instead of contain -- fills both
width and height completely, cropping the minimum necessary (like
_cover_resize_crop, applied to the whole grid block instead of per-panel).
Identical algorithm to build_multiref_sheet.py's _cover_justified_compose,
kept in sync so inference-time sheets match training-time sheets exactly."""
it = iter(imgs)
rows = []
for count in rows_counts:
row_imgs = [next(it) for _ in range(count)]
aspects = [im.width / im.height for im in row_imgs]
rows.append((row_imgs, aspects))
natural_heights = [canvas_w / sum(aspects) for _, aspects in rows]
h1 = sum(natural_heights)
if h1 >= canvas_h:
# overfill: each row already fills canvas_w exactly at scale=1 (by construction
# of natural_heights) -- keep that, crop the excess height after assembly.
row_heights = [round(nh) for nh in natural_heights]
block_w = canvas_w
else:
# underfill: scale UP so total height == canvas_h; every row becomes wider
# than canvas_w by that same factor -- crop the excess width after assembly.
scale = canvas_h / h1
row_heights = [round(nh * scale) for nh in natural_heights]
block_w = max(canvas_w, round(canvas_w * scale))
block = Image.new("RGB", (block_w, sum(row_heights)), bg)
y = 0
for (row_imgs, aspects), row_h in zip(rows, row_heights):
widths = [max(1, round(row_h * a)) for a in aspects]
row_w = sum(widths)
x = (block_w - row_w) // 2
for im, w_i in zip(row_imgs, widths):
resized = im.convert("RGB").resize((max(1, w_i), max(1, row_h)), Image.LANCZOS)
block.paste(resized, (x, y))
x += w_i
y += row_h
bw, bh = block.size
x0 = max(0, (bw - canvas_w) // 2)
y0 = max(0, (bh - canvas_h) // 2)
return block.crop((x0, y0, x0 + canvas_w, y0 + canvas_h))
def compose_sheet(imgs, panel_size=PANEL_SIZE, canvas_w=CANVAS_W, canvas_h=CANVAS_H, bg=BG_COLOR, fit_mode="crop"):
"""fit_mode: 'crop' fills each fixed-size panel completely (crops excess, current
default, matches the training data); 'fit' uses a row-justified layout that keeps
every pixel of every reference (no cropping, no distortion) while maximizing
canvas coverage (can underfill on one axis); 'cover' row-justifies AND fills the
entire canvas on both axes, cropping the minimum shared/symmetric amount needed
-- no background bars."""
n = len(imgs)
if not 1 <= n <= 5:
raise ValueError(f"expected 1-5 reference images, got {n}")
rows = LAYOUTS[n]
if fit_mode == "cover":
return _cover_justified_compose(imgs, rows, canvas_w, canvas_h, bg)
if fit_mode == "fit":
return _justified_compose(imgs, rows, canvas_w, canvas_h, bg)
native_w = max(rows) * panel_size
native_h = len(rows) * panel_size
native = Image.new("RGB", (native_w, native_h), bg)
it = iter(imgs)
for row_idx, count in enumerate(rows):
row_w = count * panel_size
x_offset = (native_w - row_w) // 2 # center short rows (e.g. bottom row of a 5-ref sheet)
y = row_idx * panel_size
for col in range(count):
panel = _cover_resize_crop(next(it), panel_size)
x = x_offset + col * panel_size
native.paste(panel, (x, y))
sheet = Image.new("RGB", (canvas_w, canvas_h), bg)
px = (canvas_w - native_w) // 2
py = (canvas_h - native_h) // 2
sheet.paste(native, (px, py))
return sheet
class MultiRefSheetBuilder:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"fit_mode": (["crop", "fit", "cover"], {
"default": "crop",
"tooltip": "crop: zoom+center-crop to fill each panel completely (matches training data, "
"may cut off edges). fit: scale each reference down to fit entirely inside its "
"panel with no cropping (preserves every pixel, aspect ratio never distorted -- "
"one uniform scale factor for both axes -- leftover space padded with background). "
"cover: row-justified like fit, but fills the WHOLE 1536x1024 canvas on both axes "
"(no background bars) by cropping the minimum shared amount needed.",
}),
},
"optional": {
"ref_image_1": ("IMAGE", {"tooltip": "image0 in the compositional prompt (anchor)."}),
"ref_image_2": ("IMAGE", {"tooltip": "image1."}),
"ref_image_3": ("IMAGE", {"tooltip": "image2."}),
"ref_image_4": ("IMAGE", {"tooltip": "image3."}),
"ref_image_5": ("IMAGE", {"tooltip": "image4."}),
},
}
RETURN_TYPES = ("IMAGE", "INT", "STRING")
RETURN_NAMES = ("sheet", "n_refs", "debug")
FUNCTION = "build"
CATEGORY = CATEGORY
DESCRIPTION = ("Combines 1-5 plugged-in reference images into the fixed 1536x1024 composite "
"sheet the multi-ref LoRA was trained on. Leave slots empty for fewer refs; "
"an empty slot is simply skipped, not padded with blank content.")
def build(self, fit_mode="crop", ref_image_1=None, ref_image_2=None, ref_image_3=None,
ref_image_4=None, ref_image_5=None):
slots = [ref_image_1, ref_image_2, ref_image_3, ref_image_4, ref_image_5]
provided = [s for s in slots if s is not None]
if not provided:
raise ValueError("MultiRefSheetBuilder needs at least one ref_image_N input.")
pil_imgs = [tensor_to_pil(t[0] if t.dim() == 4 else t) for t in provided]
sheet = compose_sheet(pil_imgs, fit_mode=fit_mode)
sheet_t = pil_to_tensor(sheet).unsqueeze(0) # [1,H,W,C]
dbg = (f"MultiRefSheet | {len(provided)} refs -> {CANVAS_W}x{CANVAS_H} "
f"({'+'.join(str(r) for r in LAYOUTS[len(provided)])} grid, fit_mode={fit_mode})")
return (sheet_t, len(provided), dbg)
NODE_CLASS_MAPPINGS = {"BFSMultiRefSheetBuilder": MultiRefSheetBuilder}
NODE_DISPLAY_NAME_MAPPINGS = {"BFSMultiRefSheetBuilder": "Multi-Ref Sheet Builder"}
# ---------------------------------------------------------------------------
# Entity Sheet 2x2 (HuMoSet v7 convention) -- SEMANTIC panel positions, unlike
# MultiRefSheetBuilder's index-ordered justified layout. The humoset_sheets_v7
# LoRA was trained on sheets where each quadrant has a fixed MEANING:
# top-left = person identity top-right = clothing/garment
# bottom-left = object OR 2nd person bottom-right = scene/background
# Panels are 768x512, each ref letterboxed (fit, white padding) inside its
# quadrant; unused quadrants stay pure white (training used white = "no info").
# ---------------------------------------------------------------------------
V7_PANEL_W, V7_PANEL_H = 768, 512
def _v7_fit_panel(img: Image.Image) -> Image.Image:
img = img.convert("RGB")
s = min(V7_PANEL_W / img.width, V7_PANEL_H / img.height)
nw, nh = max(1, round(img.width * s)), max(1, round(img.height * s))
canvas = Image.new("RGB", (V7_PANEL_W, V7_PANEL_H), BG_COLOR)
canvas.paste(img.resize((nw, nh), Image.LANCZOS), ((V7_PANEL_W - nw) // 2, (V7_PANEL_H - nh) // 2))
return canvas
class EntitySheet2x2:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {},
"optional": {
"person": ("IMAGE", {"tooltip": "Person identity reference -> TOP-LEFT quadrant."}),
"clothing": ("IMAGE", {"tooltip": "Garment reference (isolated clothing item) -> TOP-RIGHT quadrant."}),
"object_or_person2": ("IMAGE", {"tooltip": "Held object OR a second person -> BOTTOM-LEFT quadrant."}),
"scene": ("IMAGE", {"tooltip": "Background/scene reference (no people) -> BOTTOM-RIGHT quadrant."}),
},
}
RETURN_TYPES = ("IMAGE", "INT", "STRING")
RETURN_NAMES = ("sheet", "n_refs", "debug")
FUNCTION = "build"
CATEGORY = CATEGORY
DESCRIPTION = ("Builds the FIXED 1536x1024 2x2 entity sheet the humoset_sheets_v7 multi-ref LoRA "
"was trained on. Quadrant meaning is fixed (person TL, clothing TR, object/person2 BL, "
"scene BR); leave inputs empty for white quadrants. Use with LTXIdentityOverlapConditioning: "
"layout=overlap, source_id=2, phase_scale=1.0, ref_resize_mode=native_resolution.")
def build(self, person=None, clothing=None, object_or_person2=None, scene=None):
entries = {(0, 0): person, (1, 0): clothing, (0, 1): object_or_person2, (1, 1): scene}
n = sum(1 for v in entries.values() if v is not None)
if n == 0:
raise ValueError("EntitySheet2x2 needs at least one input image.")
sheet = Image.new("RGB", (CANVAS_W, CANVAS_H), BG_COLOR)
for (cx, cy), t in entries.items():
if t is None:
continue
pil = tensor_to_pil(t[0] if t.dim() == 4 else t)
sheet.paste(_v7_fit_panel(pil), (cx * V7_PANEL_W, cy * V7_PANEL_H))
sheet_t = pil_to_tensor(sheet).unsqueeze(0)
dbg = f"EntitySheet2x2 | {n} refs -> {CANVAS_W}x{CANVAS_H} (v7 semantic grid)"
return (sheet_t, n, dbg)
NODE_CLASS_MAPPINGS["BFSEntitySheet2x2"] = EntitySheet2x2
NODE_DISPLAY_NAME_MAPPINGS["BFSEntitySheet2x2"] = "Entity Sheet 2x2 (HuMoSet v7)"
+9 -3
View File
@@ -1,13 +1,18 @@
[project]
name = "bfsnodes"
description = "Custom ComfyUI nodes including LTXV Edit Anything and Head-Swap Bernini conditioning helpers."
version = "1.5.0"
license = {file = "LICENSE"}
description = "Custom ComfyUI nodes for reference-guided video: BFS Sampler (guide-driven IC-LoRAs, auto crop, mask inpainting, temporal chunking), LTX Multiple Controls / Identity Transfer / Edit Anything, Head-Swap and Anime2Real conditioning, MiniMax-H3 latent guides, RoPE layouts and loaders, LoRA Surgery, and Shot Planner / Shot Loop (tabbed panel): split long videos into shots at camera cuts, face-tracked cast, SAM 3 mask per shot with full frame, paste, mask, crop and crop + mask generation modes, optional translucent Color Guide for one or two manually selected regions with cast identity color reuse, optional Qwen3-VL to suggest settings, describe references ({details}) and the person to replace ({target}), continuity between shots, H3 conditioning with an optional duet and prompt writer, join with the original timing, audio and a side-by-side comparison. H3 Duet: pin a source clip beside the video, training-free."
version = "1.97.0"
license = {file = "LICENSE"}
dependencies = [
"torch",
"numpy",
"Pillow",
"safetensors",
"librosa",
"opencv-python",
"scenedetect",
"transformers>=4.55.0",
"scipy",
]
# classifiers = [
# # For OS-independent nodes (works on all operating systems)
@@ -34,6 +39,7 @@ Documentation = "https://github.com/alisson-anjos/ComfyUI-BFSNodes/wiki"
"Bug Tracker" = "https://github.com/alisson-anjos/ComfyUI-BFSNodes/issues"
[tool.comfy]
web = "web"
PublisherId = "alisson-anjos"
DisplayName = "ComfyUI-BFSNodes"
Icon = ""
+70
View File
@@ -0,0 +1,70 @@
"""Shared temporal placement helpers for clean LTX reference tokens.
LTX VAEs normally compress eight pixel frames into one latent-time step. The
public node input is expressed in latent steps so ``-1`` consistently means
"one VAE temporal step before frame zero" without hard-coding a particular
VAE scale factor.
"""
from __future__ import annotations
import torch
DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS = 0
def reference_temporal_offset_input():
"""Return a fresh ComfyUI input declaration for reference-only placement."""
return (
"INT",
{
"default": DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
"min": -16,
"max": 16,
"step": 1,
"tooltip": (
"Temporal RoPE offset for appearance/reference tokens only, in VAE latent-time steps. "
"0 keeps the standard frame-0 placement used by existing LoRAs. "
"-1 places the reference one latent step before frame 0 (normally -8 pixel frames), "
"which can be tested to reduce overlap frame-0 reference leakage. "
"Guide, mask, and target positions are never changed."
),
},
)
def reference_temporal_pixel_offset(
offset_latents: int, temporal_scale_factor: int
) -> int:
"""Convert a latent-step offset to the raw pixel-frame units used by LTX RoPE."""
return int(offset_latents) * int(temporal_scale_factor)
def shift_reference_temporal_positions(
positions: torch.Tensor,
offset_latents: int,
temporal_scale_factor: int,
) -> torch.Tensor:
"""Shift only the temporal axis of a reference coordinate tensor.
The helper accepts both ComfyUI's ``[B, axes, tokens]`` coordinates and
trainer-style tensors with extra trailing dimensions. A zero offset is an
exact no-op and returns the original tensor for backward-compatible specs.
"""
pixel_offset = reference_temporal_pixel_offset(
offset_latents, temporal_scale_factor
)
if pixel_offset == 0:
return positions
if positions.ndim < 3 or positions.shape[1] < 1:
raise ValueError(
"Reference temporal positions must have shape [batch, axes, ...] "
f"with a temporal axis; got {tuple(positions.shape)}"
)
shifted = positions.clone()
shifted[:, 0, ...] += torch.as_tensor(
pixel_offset,
device=shifted.device,
dtype=shifted.dtype,
)
return shifted
+3
View File
@@ -2,3 +2,6 @@ torch
numpy
Pillow
safetensors
librosa
opencv-python
scenedetect
+55
View File
@@ -0,0 +1,55 @@
"""Salva um latente do MiniMax-H3 em disco.
O `SaveLatent` nativo quebra com o latente do H3: ele chega como NestedTensor (video + audio)
e o node chama `.contiguous()` direto. Aqui os componentes sao desempacotados e salvos
separadamente, para o latente poder ser inspecionado ou decodificado fora do grafo.
"""
import os
import folder_paths
import torch
from safetensors.torch import save_file
class SaveH3Latent:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"samples": ("LATENT",),
"filename_prefix": ("STRING", {"default": "h3_latent"}),
}
}
RETURN_TYPES = ()
FUNCTION = "save"
OUTPUT_NODE = True
CATEGORY = "BFS/debug"
def save(self, samples, filename_prefix):
z = samples["samples"]
tensores = {}
if getattr(z, "is_nested", False):
for i, parte in enumerate(z.unbind()):
t = parte if parte.dim() == 5 else parte.unsqueeze(0)
tensores[f"component_{i}"] = t.contiguous().cpu()
else:
t = z if z.dim() == 5 else z.unsqueeze(0)
tensores["component_0"] = t.contiguous().cpu()
pasta = folder_paths.get_output_directory()
n = 0
while os.path.exists(os.path.join(pasta, f"{filename_prefix}_{n:05d}.safetensors")):
n += 1
caminho = os.path.join(pasta, f"{filename_prefix}_{n:05d}.safetensors")
meta = {k: str(tuple(v.shape)) for k, v in tensores.items()}
save_file(tensores, caminho, metadata={"format": "pt", **meta})
print(f"[BFSNodes] latente salvo em {caminho}")
for k, v in tensores.items():
print(f"[BFSNodes] {k}: {tuple(v.shape)} {v.dtype}")
return {}
NODE_CLASS_MAPPINGS = {"SaveH3Latent": SaveH3Latent}
NODE_DISPLAY_NAME_MAPPINGS = {"SaveH3Latent": "Save H3 Latent / nested-safe (BFS)"}
+62
View File
@@ -0,0 +1,62 @@
"""Escolhe o frame mais nitido de um lote, por variancia do Laplaciano.
Vem do IF-Edit (arXiv:2511.19435): frames decodificados carregam quantidades diferentes de
borrao residual, entao pontuar e ficar com o mais crocante bate pegar um indice fixo.
Util em particular com o MiniMax-H3 usado como gerador de imagem: o clipe minimo do VAE
(5 frames = 2 latentes) sai do decode com nitidez desigual, e o melhor frame nao e sempre
o mesmo indice.
CUIDADO -- e uma heuristica de BORRAO, nao uma metrica de qualidade. Variancia do
Laplaciano mede energia de alta frequencia, e artefato tambem e alta frequencia. Medido
num decode do H3: entre 4 frames do mesmo bloco, o Laplaciano escolheu o de PIOR PSNR
(19.72 dB contra 23.19 do vizinho), porque ele era o mais artefatado. Use quando os
candidatos forem o MESMO conteudo diferindo so em borrao -- que e o caso do IF-Edit e do
clipe estatico gerado. Nao use para escolher entre frames de momentos diferentes.
"""
import torch
def _laplacian_variance(img: torch.Tensor) -> float:
"""img: [H, W, C] em [0,1]. Variancia do Laplaciano 4-vizinhos sobre a luminancia."""
weights = torch.tensor([0.299, 0.587, 0.114], device=img.device, dtype=torch.float32)
g = (img[..., :3].float() * weights).sum(-1)
lap = (
-4.0 * g[1:-1, 1:-1]
+ g[:-2, 1:-1] + g[2:, 1:-1] + g[1:-1, :-2] + g[1:-1, 2:]
)
return float(lap.var())
class SharpestFrame:
"""Recebe um lote de imagens e devolve a mais nitida, com o indice e as notas."""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"images": ("IMAGE", {"tooltip": "Lote de frames que sejam o MESMO conteudo diferindo em borrao (ex.: clipe estatico do H3 em modo imagem). Para frames de momentos diferentes isso nao faz sentido."}),
},
"optional": {
"skip_first": ("INT", {"default": 0, "min": 0, "max": 64, "tooltip": "Ignora os N primeiros frames antes de pontuar. No H3 os primeiros frames de cada bloco decodificado costumam ser preenchimento temporal."}),
},
}
RETURN_TYPES = ("IMAGE", "INT", "STRING")
RETURN_NAMES = ("image", "index", "scores")
FUNCTION = "pick"
CATEGORY = "BFS/image"
def pick(self, images, skip_first=0):
total = images.shape[0]
start = min(skip_first, max(total - 1, 0))
scores = [_laplacian_variance(images[i]) for i in range(start, total)]
best = start + max(range(len(scores)), key=scores.__getitem__)
texto = " | ".join(f"{start + i}:{s:.5f}{' <-' if start + i == best else ''}" for i, s in enumerate(scores))
print(f"[BFSNodes] SharpestFrame: {total} frames, escolhido {best} -> {texto}")
return (images[best:best + 1], best, texto)
NODE_CLASS_MAPPINGS = {"SharpestFrame": SharpestFrame}
NODE_DISPLAY_NAME_MAPPINGS = {"SharpestFrame": "Sharpest Frame / Laplacian (BFS)"}
+176
View File
@@ -0,0 +1,176 @@
"""CPU-only layout excerpt from ComfyUI 8cfe5e1.
Source: comfy/ldm/minimax/model.py, https://github.com/Comfy-Org/ComfyUI
Copyright ComfyUI contributors; GPL-3.0, same as this repository.
Original implementation retained to test the actual audited native layout API.
"""
import math
import torch
FRAME_PER_TOKEN = (1, 4, 4, 4, 4)
FRAME_RESCALE = 5.0 / 3.0
def _axis_from_sqrt_area(dim, patch, sqrt_area):
# linspace((1 - ratio) / 2, (1 + ratio) / 2, dim // patch, endpoint=False) * 32
ratio = dim / sqrt_area
n = dim // patch
return (torch.arange(n, dtype=torch.float64) * (ratio / n) + (1.0 - ratio) / 2.0) * 32.0
def _frame_grid(h, w):
# area-normalized (h, w) coordinates of one latent frame's 2x2-patch rows
area = math.sqrt(h * w)
hh, ww = torch.meshgrid(_axis_from_sqrt_area(h, 2, area), _axis_from_sqrt_area(w, 2, area), indexing="ij")
return torch.stack([hh.reshape(-1), ww.reshape(-1)], dim=-1), _axis_from_sqrt_area(w, 2, area)
def _video_t_spans(n):
return [FRAME_RESCALE * FRAME_PER_TOKEN[k % 5] for k in range(n)]
def _video_t_grid(n, origin):
# origin + exclusive cumsum
spans = torch.tensor(_video_t_spans(n), dtype=torch.float64)
return float(origin) + torch.cat([torch.zeros(1, dtype=torch.float64), spans[:-1].cumsum(0)])
def _ref_t_span(blk):
# time-axis span a reference block occupies ahead of the target streams
kind = blk["kind"]
if kind == "image":
return 1.0
if kind == "audio":
return float(blk["ref_audio_t"])
if kind in ("video", "video_audio"):
return max(float(blk["ref_audio_t"]), sum(_video_t_spans(blk["latent_t"])))
return 0.0
def _audio_grid(cursor, t, w_low, w_high):
# channel-major stereo rows: t advances per latent frame, w pinned to the grid extremes per stereo channel, h stays 0
g = torch.zeros(t * 2, 3, dtype=torch.float64)
g[:, 0] = (cursor + torch.arange(t, dtype=torch.float64)).repeat(2)
g[:t, 2] = w_low
g[t:, 2] = w_high
return g
def _video_grid(vt, frame, cursor):
g = torch.empty(vt, frame.shape[0], 3, dtype=torch.float64)
g[:, :, 0] = _video_t_grid(vt, cursor)[:, None]
g[:, :, 1:] = frame[None]
return g.reshape(-1, 3)
class PackedLayout:
"""Static packed-sequence structure for one shape/conditioning signature."""
def __init__(self, text_len, latent_t, latent_h, latent_w, audio_t, keyframes=None, refs=None):
frame, w_grid = _frame_grid(latent_h, latent_w)
frame_rows = frame.shape[0]
segments = [("text", text_len)] # (kind, n_rows)
g = torch.zeros(text_len, 3, dtype=torch.float64)
g[:, 0] = torch.arange(text_len, dtype=torch.float64)
pos = [g] # per segment: [n, 3] float64 (t, h, w)
img_pos, img_update = [], []
audio_pos, audio_update = [], []
row = text_len
target_audio_w = (float(w_grid[0]), float(w_grid[-1]))
# refs pack between text and the targets, so the target timeline starts after their spans
cursor = float(text_len)
for blk in refs or ():
cursor += _ref_t_span(blk)
if keyframes:
# fl2va: keyframe cond rows right after text, sharing the target spatial grid;
# anchors count from the target timeline origin, FRAME_RESCALE per pixel frame, 1.0 per audio latent frame
for kf in keyframes:
cond_t = cursor + FRAME_RESCALE * kf["resolved_frame_index"]
video_latent = kf.get("latent")
if video_latent is not None:
vt = video_latent.shape[2]
n = vt * frame_rows
segments.append(("cond", n))
pos.append(_video_grid(vt, frame, cond_t))
img_pos.append(torch.arange(row, row + n))
img_update.append(torch.zeros(n, dtype=torch.bool))
row += n
audio_latent = kf.get("audio_latent")
if audio_latent is not None:
rt = audio_latent.shape[-1]
segments.append(("cond_audio", rt * 2))
pos.append(_audio_grid(cond_t, rt, *target_audio_w))
audio_pos.append(torch.arange(row, row + rt * 2))
audio_update.append(torch.zeros(rt * 2, dtype=torch.bool))
row += rt * 2
if refs:
cursor = float(text_len)
for blk in refs:
kind = blk["kind"]
if kind == "image":
r_frame, _ = _frame_grid(blk["latent_h"], blk["latent_w"])
n = r_frame.shape[0]
g = torch.empty(n, 3, dtype=torch.float64)
g[:, 0] = cursor
g[:, 1:] = r_frame
segments.append(("ref_img", n))
pos.append(g)
img_pos.append(torch.arange(row, row + n))
img_update.append(torch.zeros(n, dtype=torch.bool))
row += n
cursor += 1.0
elif kind == "audio":
rt = blk["ref_audio_t"]
if rt > 0:
segments.append(("ref_audio", rt * 2))
pos.append(_audio_grid(cursor, rt, *target_audio_w))
audio_pos.append(torch.arange(row, row + rt * 2))
audio_update.append(torch.zeros(rt * 2, dtype=torch.bool))
row += rt * 2
cursor += float(rt)
elif kind in ("video", "video_audio"):
# the block's audio rows pack immediately before its video
# rows, both sharing the cursor origin
rt = blk["ref_audio_t"]
vt = blk["latent_t"]
r_frame, r_w_grid = _frame_grid(blk["latent_h"], blk["latent_w"])
if rt > 0:
segments.append(("ref_audio", rt * 2))
pos.append(_audio_grid(cursor, rt, float(r_w_grid[0]), float(r_w_grid[-1])))
audio_pos.append(torch.arange(row, row + rt * 2))
audio_update.append(torch.zeros(rt * 2, dtype=torch.bool))
row += rt * 2
n = vt * r_frame.shape[0]
segments.append(("ref_img", n))
pos.append(_video_grid(vt, r_frame, cursor))
img_pos.append(torch.arange(row, row + n))
img_update.append(torch.zeros(n, dtype=torch.bool))
row += n
cursor += max(float(rt), sum(_video_t_spans(vt)))
# target audio then target video, always the last two segments
segments.append(("audio", audio_t * 2))
pos.append(_audio_grid(cursor, audio_t, *target_audio_w))
audio_pos.append(torch.arange(row, row + audio_t * 2))
audio_update.append(torch.ones(audio_t * 2, dtype=torch.bool))
row += audio_t * 2
n_video = latent_t * frame_rows
segments.append(("video", n_video))
pos.append(_video_grid(latent_t, frame, cursor))
img_pos.append(torch.arange(row, row + n_video))
img_update.append(torch.ones(n_video, dtype=torch.bool))
row += n_video
self.seq_len = row
self.position_ids = torch.cat(pos) # [S, 3] float64
self.img_pos = torch.cat(img_pos)
self.img_update = torch.cat(img_update)
self.audio_pos = torch.cat(audio_pos)
self.audio_update = torch.cat(audio_update)
self.signature = (text_len, latent_t, latent_h, latent_w, audio_t)
# contiguous segment table (start, stop, kind)
# kinds: text / cond / cond_audio / ref_img / ref_audio / audio / video
# the packed sequence is uniform per segment in (modality tag, timestep class),
# except the text span (tag runs resolved at forward time from the presentation tags)
seg_abs = []
off = 0
for kind, n in segments:
seg_abs.append((off, off + n, kind))
off += n
self.segments = seg_abs
+974
View File
@@ -0,0 +1,974 @@
{
"source": "alisson-anjos/ai-toolkit e17e21c src/packing.py, target_grid_stride_v1, text=3 target=2x8x8 audio=2 guide=2x(8/f)x(8/f)",
"factors": {
"1": {
"positions": [
[
0.0,
0.0,
0.0
],
[
1.0,
0.0,
0.0
],
[
2.0,
0.0,
0.0
],
[
3.0,
0.0,
0.0
],
[
3.0,
0.0,
8.0
],
[
3.0,
0.0,
16.0
],
[
3.0,
0.0,
24.0
],
[
3.0,
8.0,
0.0
],
[
3.0,
8.0,
8.0
],
[
3.0,
8.0,
16.0
],
[
3.0,
8.0,
24.0
],
[
3.0,
16.0,
0.0
],
[
3.0,
16.0,
8.0
],
[
3.0,
16.0,
16.0
],
[
3.0,
16.0,
24.0
],
[
3.0,
24.0,
0.0
],
[
3.0,
24.0,
8.0
],
[
3.0,
24.0,
16.0
],
[
3.0,
24.0,
24.0
],
[
4.666666666666667,
0.0,
0.0
],
[
4.666666666666667,
0.0,
8.0
],
[
4.666666666666667,
0.0,
16.0
],
[
4.666666666666667,
0.0,
24.0
],
[
4.666666666666667,
8.0,
0.0
],
[
4.666666666666667,
8.0,
8.0
],
[
4.666666666666667,
8.0,
16.0
],
[
4.666666666666667,
8.0,
24.0
],
[
4.666666666666667,
16.0,
0.0
],
[
4.666666666666667,
16.0,
8.0
],
[
4.666666666666667,
16.0,
16.0
],
[
4.666666666666667,
16.0,
24.0
],
[
4.666666666666667,
24.0,
0.0
],
[
4.666666666666667,
24.0,
8.0
],
[
4.666666666666667,
24.0,
16.0
],
[
4.666666666666667,
24.0,
24.0
],
[
3.0,
0.0,
0.0
],
[
4.0,
0.0,
0.0
],
[
3.0,
0.0,
24.0
],
[
4.0,
0.0,
24.0
],
[
3.0,
0.0,
0.0
],
[
3.0,
0.0,
8.0
],
[
3.0,
0.0,
16.0
],
[
3.0,
0.0,
24.0
],
[
3.0,
8.0,
0.0
],
[
3.0,
8.0,
8.0
],
[
3.0,
8.0,
16.0
],
[
3.0,
8.0,
24.0
],
[
3.0,
16.0,
0.0
],
[
3.0,
16.0,
8.0
],
[
3.0,
16.0,
16.0
],
[
3.0,
16.0,
24.0
],
[
3.0,
24.0,
0.0
],
[
3.0,
24.0,
8.0
],
[
3.0,
24.0,
16.0
],
[
3.0,
24.0,
24.0
],
[
4.666666666666667,
0.0,
0.0
],
[
4.666666666666667,
0.0,
8.0
],
[
4.666666666666667,
0.0,
16.0
],
[
4.666666666666667,
0.0,
24.0
],
[
4.666666666666667,
8.0,
0.0
],
[
4.666666666666667,
8.0,
8.0
],
[
4.666666666666667,
8.0,
16.0
],
[
4.666666666666667,
8.0,
24.0
],
[
4.666666666666667,
16.0,
0.0
],
[
4.666666666666667,
16.0,
8.0
],
[
4.666666666666667,
16.0,
16.0
],
[
4.666666666666667,
16.0,
24.0
],
[
4.666666666666667,
24.0,
0.0
],
[
4.666666666666667,
24.0,
8.0
],
[
4.666666666666667,
24.0,
16.0
],
[
4.666666666666667,
24.0,
24.0
]
],
"video_indices": [
3,
4,
5,
6,
7,
8,
9,
10,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26,
27,
28,
29,
30,
31,
32,
33,
34,
39,
40,
41,
42,
43,
44,
45,
46,
47,
48,
49,
50,
51,
52,
53,
54,
55,
56,
57,
58,
59,
60,
61,
62,
63,
64,
65,
66,
67,
68,
69,
70
],
"audio_indices": [
35,
36,
37,
38
]
},
"2": {
"positions": [
[
0.0,
0.0,
0.0
],
[
1.0,
0.0,
0.0
],
[
2.0,
0.0,
0.0
],
[
3.0,
0.0,
0.0
],
[
3.0,
0.0,
16.0
],
[
3.0,
16.0,
0.0
],
[
3.0,
16.0,
16.0
],
[
4.666666666666667,
0.0,
0.0
],
[
4.666666666666667,
0.0,
16.0
],
[
4.666666666666667,
16.0,
0.0
],
[
4.666666666666667,
16.0,
16.0
],
[
3.0,
0.0,
0.0
],
[
4.0,
0.0,
0.0
],
[
3.0,
0.0,
24.0
],
[
4.0,
0.0,
24.0
],
[
3.0,
0.0,
0.0
],
[
3.0,
0.0,
8.0
],
[
3.0,
0.0,
16.0
],
[
3.0,
0.0,
24.0
],
[
3.0,
8.0,
0.0
],
[
3.0,
8.0,
8.0
],
[
3.0,
8.0,
16.0
],
[
3.0,
8.0,
24.0
],
[
3.0,
16.0,
0.0
],
[
3.0,
16.0,
8.0
],
[
3.0,
16.0,
16.0
],
[
3.0,
16.0,
24.0
],
[
3.0,
24.0,
0.0
],
[
3.0,
24.0,
8.0
],
[
3.0,
24.0,
16.0
],
[
3.0,
24.0,
24.0
],
[
4.666666666666667,
0.0,
0.0
],
[
4.666666666666667,
0.0,
8.0
],
[
4.666666666666667,
0.0,
16.0
],
[
4.666666666666667,
0.0,
24.0
],
[
4.666666666666667,
8.0,
0.0
],
[
4.666666666666667,
8.0,
8.0
],
[
4.666666666666667,
8.0,
16.0
],
[
4.666666666666667,
8.0,
24.0
],
[
4.666666666666667,
16.0,
0.0
],
[
4.666666666666667,
16.0,
8.0
],
[
4.666666666666667,
16.0,
16.0
],
[
4.666666666666667,
16.0,
24.0
],
[
4.666666666666667,
24.0,
0.0
],
[
4.666666666666667,
24.0,
8.0
],
[
4.666666666666667,
24.0,
16.0
],
[
4.666666666666667,
24.0,
24.0
]
],
"video_indices": [
3,
4,
5,
6,
7,
8,
9,
10,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26,
27,
28,
29,
30,
31,
32,
33,
34,
35,
36,
37,
38,
39,
40,
41,
42,
43,
44,
45,
46
],
"audio_indices": [
11,
12,
13,
14
]
},
"4": {
"positions": [
[
0.0,
0.0,
0.0
],
[
1.0,
0.0,
0.0
],
[
2.0,
0.0,
0.0
],
[
3.0,
0.0,
0.0
],
[
4.666666666666667,
0.0,
0.0
],
[
3.0,
0.0,
0.0
],
[
4.0,
0.0,
0.0
],
[
3.0,
0.0,
24.0
],
[
4.0,
0.0,
24.0
],
[
3.0,
0.0,
0.0
],
[
3.0,
0.0,
8.0
],
[
3.0,
0.0,
16.0
],
[
3.0,
0.0,
24.0
],
[
3.0,
8.0,
0.0
],
[
3.0,
8.0,
8.0
],
[
3.0,
8.0,
16.0
],
[
3.0,
8.0,
24.0
],
[
3.0,
16.0,
0.0
],
[
3.0,
16.0,
8.0
],
[
3.0,
16.0,
16.0
],
[
3.0,
16.0,
24.0
],
[
3.0,
24.0,
0.0
],
[
3.0,
24.0,
8.0
],
[
3.0,
24.0,
16.0
],
[
3.0,
24.0,
24.0
],
[
4.666666666666667,
0.0,
0.0
],
[
4.666666666666667,
0.0,
8.0
],
[
4.666666666666667,
0.0,
16.0
],
[
4.666666666666667,
0.0,
24.0
],
[
4.666666666666667,
8.0,
0.0
],
[
4.666666666666667,
8.0,
8.0
],
[
4.666666666666667,
8.0,
16.0
],
[
4.666666666666667,
8.0,
24.0
],
[
4.666666666666667,
16.0,
0.0
],
[
4.666666666666667,
16.0,
8.0
],
[
4.666666666666667,
16.0,
16.0
],
[
4.666666666666667,
16.0,
24.0
],
[
4.666666666666667,
24.0,
0.0
],
[
4.666666666666667,
24.0,
8.0
],
[
4.666666666666667,
24.0,
16.0
],
[
4.666666666666667,
24.0,
24.0
]
],
"video_indices": [
3,
4,
9,
10,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26,
27,
28,
29,
30,
31,
32,
33,
34,
35,
36,
37,
38,
39,
40
],
"audio_indices": [
5,
6,
7,
8
]
}
}
}
+48
View File
@@ -0,0 +1,48 @@
import importlib.util
import types
import unittest
from pathlib import Path
import torch
ROOT = Path(__file__).resolve().parents[1]
SPEC = importlib.util.spec_from_file_location("bfs_h3_history", ROOT / "bfs_h3_history.py")
HI = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(HI)
class Layout:
"""text 4 rows | guide cond 6 rows | history cond 4 rows | audio 3 | video 6, all at the target origin 4.0"""
def __init__(self):
self.segments = [(0, 4, "text"), (4, 10, "cond"), (10, 14, "cond"), (14, 17, "audio"), (17, 23, "video")]
p = torch.zeros(23, 3)
p[:4, 0] = torch.arange(4.)
p[4:, 0] = 4.0
self.position_ids = p
class HistoryTest(unittest.TestCase):
def setUp(self):
HI.history_span = lambda t: 10.0 * t
def test_target_moves_past_the_history_only(self):
lay = Layout()
kfs = [{"latent": torch.zeros(1, 24, 3, 2, 2)}, {"latent": torch.zeros(1, 24, 2, 2, 2), "anchor": "history"}]
self.assertTrue(HI.relocate(lay, kfs))
t = lay.position_ids[:, 0]
self.assertTrue(torch.equal(t[:4], torch.arange(4.))) # text untouched
self.assertTrue(bool((t[10:14] == 4.0).all())) # history stays at the origin
self.assertTrue(bool((t[4:10] == 24.0).all())) # the other guide moves with the video
self.assertTrue(bool((t[14:] == 24.0).all())) # audio and video move by the span (2 steps)
self.assertFalse(HI.relocate(lay, kfs)) # already moved: no double shift
def test_no_history_no_change(self):
lay = Layout()
before = lay.position_ids.clone()
self.assertFalse(HI.relocate(lay, [{"latent": torch.zeros(1, 24, 3, 2, 2)}]))
self.assertTrue(torch.equal(before, lay.position_ids))
if __name__ == "__main__":
unittest.main()
+163
View File
@@ -0,0 +1,163 @@
from __future__ import annotations
import importlib.util
import sys
import types
import unittest
from pathlib import Path
import torch
ROOT = Path(__file__).resolve().parents[1]
class _Nested:
is_nested = True
def __init__(self, tensors):
self.tensors = list(tensors)
def unbind(self):
return self.tensors
# stub the two comfy modules the node uses, so the tests run without ComfyUI
_comfy = types.ModuleType("comfy")
_nt = types.ModuleType("comfy.nested_tensor")
_nt.NestedTensor = _Nested
_ut = types.ModuleType("comfy.utils")
_ut.reshape_mask = lambda m, shape: torch.nn.functional.interpolate(m.reshape(1, 1, *m.shape[-3:]).float(), size=shape[2:])
_comfy.nested_tensor, _comfy.utils = _nt, _ut
for name, mod in (("comfy", _comfy), ("comfy.nested_tensor", _nt), ("comfy.utils", _ut)):
sys.modules.setdefault(name, mod)
SPEC = importlib.util.spec_from_file_location("bfs_h3_side_panel", ROOT / "bfs_h3_side_panel.py")
SP = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(SP)
class FakeVAE:
"""[N,H,W,3] -> [1,24,T,H/16,W/16] with the H3 frame grid; the mean colour in every channel."""
def encode(self, px):
n = px.shape[0]
t = 1 if n == 1 else ((n - 5) // 17) * 5 + 2
x = px.mean(-1)[None, None] # [1,1,N,H,W]
x = torch.nn.functional.interpolate(x, size=(t, px.shape[1] // 16, px.shape[2] // 16))
return x.repeat(1, 24, 1, 1, 1)
def decode(self, z):
t, h, w = z.shape[2:]
return torch.full((1, SP.frame_count_of(t) if t > 1 else 1, h * 16, w * 16, 3), float(z.mean()))
def _latent(w=576, h=1024, frames=22):
t = ((frames - 5) // 17) * 5 + 2
return {"samples": _Nested((torch.zeros(1, 24, t, h // 16, w // 16), torch.zeros(1, 32, 2, 37)))}
class SidePanelTest(unittest.TestCase):
def test_strip_sizes_snap_to_patches(self):
self.assertEqual(SP.strip_size(576, 1024, "left", 1.0, 0), (576, 1024, 576, 1024))
self.assertEqual(SP.strip_size(576, 1024, "top", 0.33, 32), (576, 384, 576, 352))
def test_mask_holds_only_the_strip(self):
info = {"position": "left", "h": 4, "w": 6, "strip_h": 0, "strip_w": 2}
m = SP.panel_mask(info, 3, "all frames", panel_noise=0.1)
self.assertEqual(tuple(m.shape), (1, 1, 3, 4, 8))
self.assertTrue(torch.allclose(m[..., :2], torch.tensor(0.1)))
self.assertTrue(bool((m[..., 2:] == 1).all()))
first = SP.panel_mask(info, 3, "first latent frame")
self.assertEqual(float(first[0, 0, 1, 0, 0]), 1.0)
def test_layout_text_follows_the_share(self):
half = SP.layout_text({"position": "left", "h": 64, "w": 36, "strip_h": 0, "strip_w": 36})
self.assertIn("the left half shows the original performance; the right half shows", half)
self.assertNotIn("kept", half)
narrow = SP.layout_text({"position": "left", "h": 64, "w": 72, "strip_h": 0, "strip_w": 36})
self.assertIn("the left third shows", narrow)
self.assertIn("the right two thirds show", narrow)
self.assertIn("a wider view of the same action", narrow)
top = {"position": "top", "h": 48, "w": 36, "strip_h": 16, "strip_w": 0}
txt = SP.fill_layout("{panel_region} shows a man dancing, and {video_region} shows <Subject 1>.", top)
self.assertEqual(txt, "the top quarter shows a man dancing, and the bottom three quarters show <Subject 1>.")
p = SP.build_prompt("character swap", "", 1, 0)
for w in ("kept footage", "kept exactly", "in sync", "generated"):
self.assertNotIn(w, SP.fill_layout(p, top))
def test_apply_and_crop_round_trip(self):
lat = _latent()
panel = torch.full((22, 512, 512, 3), 0.9)
guide = torch.full((22, 1024, 576, 3), 0.2)
pos, out, info, preview, text = SP.BFSH3SidePanel().apply(
[[torch.zeros(1), {}]], lat, FakeVAE(), panel, "left", 1.0, "contain", 0, "all frames", 0.0, guide, 0)
v = out["samples"].tensors[0]
self.assertEqual(tuple(v.shape), (1, 24, 7, 64, 72))
kf = pos[0][1]["minimax_keyframes"][0]
self.assertEqual(tuple(kf["latent"].shape[-2:]), (64, 72))
self.assertEqual(tuple(preview.shape), (1, 1024, 1152, 3))
cropped = SP.BFSH3SidePanelCrop().crop(info, latent=out, images=torch.zeros(5, 1024, 1152, 3))
self.assertEqual(tuple(cropped[0]["samples"].tensors[0].shape), (1, 24, 7, 64, 36))
self.assertNotIn("noise_mask", cropped[0])
self.assertEqual(tuple(cropped[1].shape), (5, 1024, 576, 3))
def test_existing_guides_are_moved_onto_the_canvas(self):
lat = _latent()
kf = {"resolved_frame_index": 0, "latent": torch.zeros(1, 24, 1, 64, 36)}
pos = SP.BFSH3SidePanel().apply([[torch.zeros(1), {"minimax_keyframes": [kf]}]], lat, FakeVAE(),
torch.ones(1, 64, 64, 3), "top", 0.5, "cover", 1, "all frames")[0]
self.assertEqual(tuple(pos[0][1]["minimax_keyframes"][0]["latent"].shape[-2:]), (64 + 34, 36))
class KeepMaskTest(unittest.TestCase):
def test_latent_mask_follows_the_h3_frame_grid(self):
m = torch.zeros(22, 64, 64)
m[5:9, 10:20, 10:20] = 1 # frames 5-8 = latent step 2 (1 + 4 frames before it)
lm = SP.latent_mask(m, 7, 4, 4, grow=0)
self.assertEqual(tuple(lm.shape), (1, 1, 7, 4, 4))
self.assertEqual(float(lm[0, 0, 2].max()), 1.0)
self.assertEqual(float(lm[0, 0, 1].max()), 0.0)
def test_apply_with_keep_mask_regenerates_only_the_mask(self):
lat = _latent()
panel = torch.full((22, 512, 512, 3), 0.9)
km = torch.zeros(22, 1024, 576); km[:, 300:700, 200:400] = 1
out = SP.BFSH3SidePanel().apply([[torch.zeros(1), {}]], lat, FakeVAE(), panel, "left", 1.0, "contain", 0,
"all frames", 0.0, None, 0, keep_video=torch.full((22, 1024, 576, 3), 0.3), keep_mask=km)[1]
m = out["noise_mask"].tensors[0]
self.assertEqual(float(m[0, 0, 3, 0, 40]), 0.0) # video area outside the mask: kept
self.assertEqual(float(m[0, 0, 3, 30, 36 + 18]), 1.0) # inside the mask: generated
self.assertEqual(float(m[0, 0, 3, 30, 5]), 0.0) # panel: pinned
def test_panel_static_covers_the_subject_every_frame(self):
seen = {}
def seg(imgs, spec):
seen["spec"] = spec
m = torch.zeros(imgs.shape[0], *imgs.shape[1:3]); m[:, 10:20, 10:20] = 1
return m
def tv(img, mask, grow=0.03, seed=0):
out = img.clone(); m = torch.nn.functional.interpolate(mask[None, None].float(), size=img.shape[1:3])[0, 0]
out[0][m > 0.5] = -1.0
return out
fake = types.SimpleNamespace(DEFAULT_MASK={"text": "", "threshold": 0.5}, segment_frames=seg, tv_static=tv,
add_setting=lambda text, k, swap: text + f" <Picture {k}> setting")
old, SP._shot_loop = SP._shot_loop, lambda: fake
try:
fr = torch.full((5, 64, 48, 3), 0.5)
self.assertIs(SP.static_over_panel(fr, "off"), fr)
out = SP.static_over_panel(fr, "face")
self.assertEqual(seen["spec"]["text"], "face")
self.assertEqual(tuple(out.shape), (5, 64, 48, 3))
self.assertTrue(bool((out[:, 15, 15] == -1).all()) and float(out[0, 40, 40, 0]) == 0.5)
st = SP.setting_from_clip(fr, 1.0, "dog")
self.assertEqual(tuple(st.shape), (1, 64, 48, 3)); self.assertEqual(seen["spec"]["text"], "dog")
self.assertTrue(SP.with_setting("x", 2, True).endswith("<Picture 2> setting"))
finally:
SP._shot_loop = old
if __name__ == "__main__":
unittest.main()
+900
View File
@@ -0,0 +1,900 @@
from __future__ import annotations
import importlib.util
import sys
import tempfile
import types
import unittest
from pathlib import Path
import torch
ROOT = Path(__file__).resolve().parents[1]
_TMP = tempfile.mkdtemp(prefix="bfs_shotloop_test_")
# The module only needs a few folder_paths calls; stub them so the tests run without ComfyUI.
_fp = types.ModuleType("folder_paths")
_fp.get_temp_directory = lambda: _TMP
_fp.get_input_directory = lambda: _TMP
_fp.get_annotated_filepath = lambda name: str(Path(_TMP) / name)
sys.modules.setdefault("folder_paths", _fp)
class _Blocker:
def __init__(self, message):
self.message = message
_ce = types.ModuleType("comfy_execution")
_gu = types.ModuleType("comfy_execution.graph_utils")
_gu.ExecutionBlocker = _Blocker
sys.modules.setdefault("comfy_execution", _ce)
sys.modules.setdefault("comfy_execution.graph_utils", _gu)
SPEC = importlib.util.spec_from_file_location("bfs_shot_loop", ROOT / "bfs_shot_loop.py")
assert SPEC is not None and SPEC.loader is not None
SL = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(SL)
H3 = "H3 (17n+5)"
def _shot(i, start, end, cut=False, gen=None, count=1, queue=False, run_id=""):
length = end - start
return {"index": i, "count": count, "start": start, "end": end, "length": length,
"gen_length": gen or SL.snap_up(length, H3), "fps": 24.0, "cut_before": cut,
"queue": queue, "run_id": run_id}
class GridTest(unittest.TestCase):
def test_h3_grid(self):
self.assertEqual([SL.snap_up(x, H3) for x in (1, 5, 6, 22, 23, 107, 108)], [5, 5, 22, 22, 39, 107, 124])
self.assertEqual([SL.snap_down(x, H3) for x in (4, 5, 21, 22, 106, 107, 110)], [5, 5, 5, 22, 90, 107, 107])
def test_other_grids(self):
self.assertEqual(SL.snap_up(50, "LTX / Wan (8n+1)"), 57)
self.assertEqual(SL.snap_down(50, "LTX / Wan (8n+1)"), 49)
self.assertEqual(SL.snap_up(6, "Wan (4n+1)"), 9)
self.assertEqual(SL.snap_up(37, "any"), 37)
def test_generation_size_is_on_the_multiple(self):
w, h = SL.generation_size(1280, 720, 0.15, 32)
self.assertEqual((w % 32, h % 32), (0, 0))
self.assertGreater(w, h)
class PlanTest(unittest.TestCase):
def test_fixed_mode_splits_evenly_under_the_limit(self):
segs = SL.plan_segments(300, [], "fixed", 107, 24)
self.assertEqual([s["end"] - s["start"] for s in segs], [100, 100, 100])
def test_shots_split_at_cuts_and_long_shots_are_divided(self):
segs = SL.plan_segments(300, [50, 80], "shots", 107, 10)
bounds = [(s["start"], s["end"]) for s in segs]
self.assertEqual(bounds[:2], [(0, 50), (50, 80)])
self.assertTrue(all(e - s <= 107 for s, e in bounds))
self.assertEqual(bounds[-1][1], 300)
self.assertTrue(segs[1]["cut_before"])
def test_short_shots_merge_into_a_neighbour_that_still_fits(self):
segs = SL.plan_segments(200, [10, 100], "shots", 107, 24)
self.assertEqual([(s["start"], s["end"]) for s in segs], [(0, 100), (100, 200)])
def test_limits_and_manual_bounds(self):
self.assertEqual(len(SL.plan_segments(1000, [], "fixed", 100, 10, max_parts=3)), 3)
self.assertEqual(SL.plan_segments(1000, [], "fixed", 100, 10, max_total=250)[-1]["end"], 250)
manual = SL.plan_segments(100, [], "manual", 107, 10, manual=[30, 70])
self.assertEqual([(s["start"], s["end"]) for s in manual], [(0, 30), (30, 70), (70, 100)])
def test_builtin_detector_finds_spikes_and_respects_the_gap(self):
score = [1.0] * 100
raw = [0.01] * 100
for i in (20, 22, 60):
score[i], raw[i] = 50.0, 0.5
self.assertEqual(SL.detect_cuts(score, raw, 0.5, 24.0), [20, 60])
class JoinTest(unittest.TestCase):
def _frames(self, n, value):
return torch.full((n, 8, 8, 3), float(value))
def test_trims_to_true_lengths_in_order(self):
shots = [_shot(1, 30, 50, cut=True, gen=22), _shot(0, 0, 30, gen=39)]
imgs = [self._frames(22, 0.8), self._frames(39, 0.2)]
video, audio, fps = SL.BFSShotJoin()._join_all(imgs, shots, [0])
self.assertEqual(video.shape[0], 50)
self.assertAlmostEqual(float(video[0, 0, 0, 0]), 0.2)
self.assertAlmostEqual(float(video[-1, 0, 0, 0]), 0.8)
self.assertEqual(fps, 24.0)
def test_crossfade_only_across_soft_joins(self):
imgs = [self._frames(39, 0.0), self._frames(39, 1.0)]
soft = SL.BFSShotJoin()._join_all(imgs, [_shot(0, 0, 30, gen=39), _shot(1, 30, 60, gen=39)], [4])[0]
hard = SL.BFSShotJoin()._join_all(imgs, [_shot(0, 0, 30, gen=39), _shot(1, 30, 60, cut=True, gen=39)], [4])[0]
self.assertGreater(float(soft[31, 0, 0, 0]), 0.0)
self.assertLess(float(soft[31, 0, 0, 0]), 1.0)
self.assertEqual(float(hard[31, 0, 0, 0]), 1.0)
def test_audio_is_trimmed_to_the_video(self):
audio = {"waveform": torch.zeros(1, 2, 48000 * 10), "sample_rate": 48000}
imgs = [self._frames(39, 0.5)]
_, a, _ = SL.BFSShotJoin()._join_all(imgs, [_shot(0, 0, 24, gen=39)], [0], [audio])
self.assertEqual(a["waveform"].shape[-1], 48000)
def test_queue_mode_blocks_until_every_shot_is_stored(self):
join = SL.BFSShotJoin()
shots = [_shot(0, 0, 20, count=2, queue=True, run_id="t1"), _shot(1, 20, 45, cut=True, count=2, queue=True, run_id="t1")]
first = join.join([self._frames(22, 0.1)], [shots[0]], [0])
self.assertIsInstance(first[0], _Blocker)
video, _, _, _ = join.join([self._frames(39, 0.9)], [shots[1]], [0])
self.assertEqual(video.shape[0], 45)
self.assertAlmostEqual(float(video[0, 0, 0, 0]), 0.1, places=2)
class FilterTest(unittest.TestCase):
STATS = {"persons": 1, "person_area": 0.2, "person_frames": 6, "faces": 1, "brightness": 0.4, "motion": 0.02, "sampled": 6}
def test_no_filter_keeps_everything(self):
self.assertEqual(SL.skip_reason(self.STATS, 50, dict(SL.DEFAULT_FILTERS)), "")
self.assertFalse(SL.filters_active(dict(SL.DEFAULT_FILTERS)))
def test_each_filter(self):
f = dict(SL.DEFAULT_FILTERS)
self.assertEqual(SL.skip_reason(dict(self.STATS, persons=0), 50, dict(f, person=True)), "no person")
self.assertIn("smaller", SL.skip_reason(dict(self.STATS, person_area=0.01), 50, dict(f, person=True, min_person_area=0.03)))
self.assertIn("more than", SL.skip_reason(dict(self.STATS, persons=5), 50, dict(f, max_persons=2)))
self.assertEqual(SL.skip_reason(dict(self.STATS, faces=0), 50, dict(f, face=True)), "no face")
self.assertEqual(SL.skip_reason(dict(self.STATS, brightness=0.02), 50, dict(f, skip_dark=True)), "dark / fade")
self.assertEqual(SL.skip_reason(dict(self.STATS, motion=0.0), 50, dict(f, skip_static=True)), "static")
self.assertIn("shorter", SL.skip_reason(self.STATS, 5, dict(f, min_frames=10)))
class TimelineJoinTest(unittest.TestCase):
def _tl(self, fill):
return {"path": "", "fps": 24.0, "width": 8, "height": 8, "fill": fill, "audio": None, "n": 60,
"src": list(range(60)),
"segs": [{"start": 0, "end": 20, "run": True, "cut_before": False, "run_index": 0},
{"start": 20, "end": 40, "run": False, "cut_before": True, "run_index": -1},
{"start": 40, "end": 60, "run": True, "cut_before": True, "run_index": 1}]}
def test_drop_removes_skipped_shots(self):
shots = [_shot(0, 0, 20, gen=22), _shot(1, 40, 60, cut=True, gen=22)]
imgs = [torch.zeros(22, 8, 8, 3), torch.ones(22, 8, 8, 3)]
video = SL.BFSShotJoin()._join_all(imgs, shots, [0], None, self._tl("drop"))[0]
self.assertEqual(video.shape[0], 40)
self.assertEqual(float(video[25, 0, 0, 0]), 1.0)
def test_original_fill_reads_the_source_frames(self):
shots = [_shot(0, 0, 20, gen=22), _shot(1, 40, 60, cut=True, gen=22)]
imgs = [torch.zeros(22, 8, 8, 3), torch.ones(22, 8, 8, 3)]
orig = SL._read_frames
SL._read_frames = lambda path, idx, size: [__import__("numpy").full((8, 8, 3), 128, "uint8") for _ in idx]
try:
video = SL.BFSShotJoin()._join_all(imgs, shots, [0], None, self._tl("original"))[0]
finally:
SL._read_frames = orig
self.assertEqual(video.shape[0], 60)
self.assertAlmostEqual(float(video[30, 0, 0, 0]), 128 / 255, places=3)
class CastTest(unittest.TestCase):
# person 0 on frames 0-47 (big face), person 1 on 48-95, both small on 96-119
CAST = {"step": 6, "people": [{"id": 0}, {"id": 1}], "samples":
[{"f": f, "faces": [{"pid": 0, "area": 0.1}]} for f in range(0, 48, 6)]
+ [{"f": f, "faces": [{"pid": 1, "area": 0.1}]} for f in range(48, 96, 6)]
+ [{"f": f, "faces": []} for f in range(96, 120, 6)]}
def test_shot_people_and_main(self):
self.assertEqual(SL.shot_people(self.CAST, 0, 60), {"people": [0, 1], "main": 0})
self.assertEqual(SL.shot_people(self.CAST, 96, 120), {"people": [], "main": -1})
def test_change_points_need_a_long_enough_run(self):
self.assertEqual(SL.person_change_points(self.CAST, 0, 120, 24), [48])
self.assertEqual(SL.person_change_points(self.CAST, 0, 60, 24), [])
def test_plan_splits_assigns_and_skips_by_person(self):
path = str(Path(_TMP) / "cast.mp4"); Path(path).write_bytes(b"x")
a = {"fps": 24.0, "n": 120, "score": [0.0] * 120, "raw": [0.0] * 120}
SL._CAST_CACHE[SL._cast_key(path, a)] = self.CAST
orig = SL.find_cuts
SL.find_cuts = lambda *args, **kw: ([], "test")
try:
plan = dict(SL.DEFAULT_PLAN, max_s=10, min_s=0.5, cast={"0": {"ref": "a.png"}}, cast_split=True, cast_only=True)
segs = SL.apply_filters(plan, a, path, SL.resolve_plan(plan, a, path))
finally:
SL.find_cuts = orig
self.assertEqual([(s["start"], s["end"]) for s in segs], [(0, 48), (48, 120)])
self.assertEqual(segs[0]["ref"], "a.png")
self.assertTrue(segs[0]["run"])
self.assertEqual(segs[1]["skip_reason"], "no linked person")
class ComparisonTest(unittest.TestCase):
def test_join_returns_a_labelled_side_by_side(self):
shots = [dict(_shot(0, 0, 20, gen=22), frames=torch.zeros(22, 32, 48, 3), ref=torch.ones(1, 40, 30, 3),
ref2=None, prompt="a test prompt"),
dict(_shot(1, 20, 40, cut=True, gen=22), frames=torch.zeros(22, 32, 48, 3), ref=None, ref2=None,
prompt="")]
imgs = [torch.full((22, 32, 48, 3), 0.5), torch.full((22, 32, 48, 3), 0.7)]
video, _, _, comp = SL.BFSShotJoin().join(imgs, shots, [0], comparison=[True], label=["steps 20"])
self.assertEqual(video.shape[0], 40)
self.assertEqual(comp.shape[0], 40)
self.assertEqual(comp.shape[2] % 16, 0)
self.assertEqual(comp.shape[1] % 16, 0)
self.assertGreaterEqual(comp.shape[2], 48 * 2 + 32)
self.assertGreater(comp.shape[1], 32)
off = SL.BFSShotJoin().join(imgs, shots, [0])[3]
self.assertEqual(off.shape[0], 1)
class ContinuityTest(unittest.TestCase):
def test_pick_frame_ignores_the_overlap(self):
frames = torch.arange(30, dtype=torch.float32).view(30, 1, 1, 1).expand(30, 2, 2, 3)
self.assertEqual(float(SL.pick_frame(frames, 20, "first")[0, 0, 0, 0]), 0.0)
self.assertEqual(float(SL.pick_frame(frames, 20, "middle")[0, 0, 0, 0]), 10.0)
self.assertEqual(float(SL.pick_frame(frames, 20, "last")[0, 0, 0, 0]), 19.0)
def test_chain_uses_the_previous_result_only_when_asked(self):
frames = torch.arange(30, dtype=torch.float32).view(30, 1, 1, 1).expand(30, 2, 2, 3)
SL.remember_result({"index": 0, "count": 2}, frames)
shot = {"index": 1, "count": 2, "chain": "reference", "chain_frame": "last", "prev_length": 20}
self.assertEqual(float(SL.chain_image(shot)[0, 0, 0, 0]), 19.0)
self.assertIsNone(SL.chain_image(dict(shot, chain="off")))
self.assertIsNone(SL.chain_image(dict(shot, index=0)))
self.assertIsNone(SL.chain_image(dict(shot, index=3, count=4))) # not the shot right after
stored = torch.ones(1, 2, 2, 3)
self.assertIs(SL.chain_image(dict(shot, chain_image=stored)), stored)
def test_plan_carries_the_per_shot_setting(self):
a = {"fps": 24.0, "n": 100, "score": [0.0] * 100, "raw": [0.0] * 100}
plan = dict(SL.DEFAULT_PLAN, mode="fixed", max_s=2, bounds=[50],
segs=[{}, {"chain": "first frame", "chain_frame": "middle"}])
segs = SL.resolve_plan(plan, a)
self.assertEqual((segs[0]["chain"], segs[1]["chain"], segs[1]["chain_frame"]), ("history", "first frame", "middle")) # history is the default
class MaskCropTest(unittest.TestCase):
def test_spec_takes_target_from_shot_and_settings_from_global(self):
spec = SL.mask_spec({"text": "person", "padding": 0.9}, {"padding": 0.3, "text": "ignored"})
self.assertEqual((spec["text"], spec["padding"], spec["fill_holes"]), ("person", 0.3, True))
def test_shape_fills_holes_and_holds_in_time(self):
m = torch.zeros(5, 9, 9, dtype=torch.uint8)
m[2, 2:7, 2:7] = 1
m[2, 4, 4] = 0 # a hole
out = SL.shape_mask(m, dict(SL.DEFAULT_MASK, temporal_expand=1))
self.assertEqual(int(out[2, 4, 4]), 1)
self.assertEqual(int(out[1, 3, 3]), 1) # held one frame back
self.assertEqual(int(out[0, 3, 3]), 0)
blocks = SL.shape_mask(m, dict(SL.DEFAULT_MASK, temporal_expand=0, blockify=4))
self.assertEqual(tuple(blocks.shape), (5, 9, 9))
def test_box_is_the_union_plus_padding(self):
m = torch.zeros(2, 10, 10, dtype=torch.uint8)
m[0, 2:4, 2:4] = 1
m[1, 6:8, 6:8] = 1
self.assertEqual(SL.crop_box(m, 0.0), [0.2, 0.2, 0.8, 0.8])
self.assertIsNone(SL.crop_box(torch.zeros(2, 10, 10), 0.1))
def test_uncrop_pastes_only_inside_the_mask(self):
full = torch.zeros(3, 20, 20, 3)
mask = torch.zeros(3, 20, 20)
mask[:, 5:15, 5:15] = 1
shot = {"full_frames": full, "crop": {"box": [0.25, 0.25, 0.75, 0.75], "mask": mask, "paste": "mask",
"expand": 0, "feather": 0}}
out = SL.uncrop(torch.ones(3, 8, 8, 3), shot)
self.assertEqual(tuple(out.shape), (3, 20, 20, 3))
self.assertEqual(float(out[0, 10, 10, 0]), 1.0)
self.assertEqual(float(out[0, 1, 1, 0]), 0.0)
class DuetPanelCropTest(unittest.TestCase):
def test_join_side_cuts_the_panel_off(self):
shot = {"width": 64, "height": 32, "panel": {"position": "left", "h": 2, "w": 4, "strip_h": 0, "strip_w": 4}}
canvas = torch.zeros(3, 32, 128, 3)
canvas[:, :, 64:] = 1.0
out = SL.crop_panel(canvas, shot)
self.assertEqual(tuple(out.shape), (3, 32, 64, 3))
self.assertEqual(float(out.min()), 1.0)
self.assertIs(SL.crop_panel(out, shot), out) # already the video size
self.assertIs(SL.crop_panel(canvas, {"width": 64}), canvas) # no panel
class SilentAudioTest(unittest.TestCase):
def test_join_without_audio_returns_a_silent_track(self):
imgs = [torch.full((39, 8, 8, 3), 0.5)]
video, a, fps = SL.BFSShotJoin()._join_all(imgs, [_shot(0, 0, 24, gen=39)], [0])
self.assertEqual(tuple(a["waveform"].shape[:2]), (1, 2))
self.assertEqual(a["waveform"].shape[-1], 44100) # 24 frames at 24 fps = 1 s
self.assertEqual(float(a["waveform"].abs().max()), 0.0)
self.assertFalse(SL._usable({"waveform": torch.zeros(1, 2, 0), "sample_rate": 44100}))
class RepackTest(unittest.TestCase):
def test_replaces_only_connected_pieces_and_fits_the_guide(self):
shot = dict(_shot(0, 0, 20, gen=22), width=8, height=8, frames=torch.zeros(22, 8, 8, 3),
ref=torch.zeros(1, 4, 4, 3), ref2=None, prompt="old", audio=None)
ref = torch.ones(1, 6, 6, 3)
out = SL.BFSShotRepack().repack(shot, ref_image=ref)[0]
self.assertTrue(torch.equal(out["ref"], ref))
self.assertEqual(out["prompt"], "old")
self.assertIs(out["frames"], shot["frames"])
out = SL.BFSShotRepack().repack(shot, guide_frames=torch.ones(30, 16, 16, 3), prompt="new")[0]
self.assertEqual(tuple(out["frames"].shape), (22, 8, 8, 3))
self.assertEqual(out["prompt"], "new")
self.assertEqual(out["length"], 20)
class ConditioningWriterTest(unittest.TestCase):
"""The optional prompt writer of BFS Shot H3 Conditioning (template path, no models)."""
def setUp(self):
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
import bfs_h3_side_panel as SP
self.SP, self.seen = SP, {}
seen = self.seen
class R2V:
@staticmethod
def execute(**kw):
seen["prompt"] = kw["prompt"]
return types.SimpleNamespace(args=("pos", "lat"))
mod = types.ModuleType("comfy_extras.nodes_minimax_h3")
mod.MiniMaxH3ReferenceToVideo, mod.MiniMaxH3AddGuide = R2V, R2V
self._old = {k: sys.modules.get(k) for k in ("comfy_extras", "comfy_extras.nodes_minimax_h3")}
sys.modules["comfy_extras"] = types.ModuleType("comfy_extras")
sys.modules["comfy_extras.nodes_minimax_h3"] = mod
self._apply = SP.BFSH3SidePanel.apply
SP.BFSH3SidePanel.apply = lambda self, pos, lat, *a, **k: (pos, lat, {"position": "left"}, None, None)
self._patch = SP.patch_model_rope
SP.patch_model_rope = lambda m, info, gap: "patched"
def tearDown(self):
self.SP.BFSH3SidePanel.apply, self.SP.patch_model_rope = self._apply, self._patch
for k, v in self._old.items():
if v is None:
sys.modules.pop(k, None)
else:
sys.modules[k] = v
def _shot(self):
return dict(prompt="my own {layout} prompt", width=448, height=800, gen_length=22,
frames=torch.zeros(22, 8, 8, 3), ref=torch.zeros(1, 4, 4, 3), ref2=None, audio=None)
def _run(self, **kw):
c = SL.BFSShotH3Conditioning()
return c.condition(self._shot(), "clip", "vae", "none", True, "none", "match", **kw)
def test_planner_prompt_is_default_and_layout_filled(self):
out = self._run()
self.assertEqual(out[3], "my own prompt")
out = self._run(duet="canvas")
self.assertIn("the left half shows the original performance", out[3])
self.assertEqual(self.seen["prompt"], out[3])
def test_task_template_follows_the_duet_mode(self):
out = self._run(duet="canvas", task="character swap")
self.assertIn("split screen", out[3])
self.assertIn("<Picture 1>", out[3])
out = self._run(duet="shifted RoPE", model="m", task="character swap")
self.assertNotIn("split screen", out[3])
self.assertEqual(out[2], "patched")
def test_setting_picture_is_the_last_reference_with_static(self):
seen = {}
orig = sys.modules["comfy_extras.nodes_minimax_h3"].MiniMaxH3ReferenceToVideo.execute
def capture(**kw):
seen.update(kw)
return orig(**kw)
sys.modules["comfy_extras.nodes_minimax_h3"].MiniMaxH3ReferenceToVideo.execute = staticmethod(capture)
shot = self._shot()
shot["frames"] = torch.full((22, 8, 8, 3), 0.5)
mask = torch.zeros(22, 8, 8)
mask[:, 2:4, 2:4] = 1
out = SL.BFSShotH3Conditioning().condition(shot, "clip", "vae", "none", True, "none", "match", duet="canvas",
task="character swap", setting_ref="on (generation size)",
setting_mask=mask)
refs = list(seen["ref_images"].values())
self.assertEqual(len(refs), 2)
pic = refs[-1][0]
self.assertTrue(((pic[3, 3] == 0) | (pic[3, 3] == 1)).all()) # static inside the (grown) mask
self.assertTrue(torch.allclose(pic[7, 7], torch.tensor(0.5))) # the place outside it
self.assertIn("<Picture 2> shows the setting", out[3])
self.assertEqual(SL.add_setting("a {setting} b", 3, True), "a <Picture 3> b")
def test_task_needs_duet(self):
with self.assertRaises(ValueError):
self._run(task="style", instruction="anime")
class TargetTest(unittest.TestCase):
def test_target_placeholder(self):
self.assertEqual(SL.fill_target("Replace {target} with <Subject 1>", "the man in a red coat"),
"Replace the man in a red coat with <Subject 1>")
self.assertEqual(SL.fill_target("Replace {target} with <Subject 1>", ""), "Replace the person with <Subject 1>")
self.assertEqual(SL.fill_target("no placeholder", "x"), "no placeholder")
class MaskOverlayTest(unittest.TestCase):
def test_overlay_marks_mask_and_box(self):
o = torch.zeros(3, 32, 32, 3)
m = torch.zeros(3, 32, 32); m[:, 8:16, 8:16] = 1
out = SL._mask_overlay(o, {"mask": m, "box": [0.1, 0.1, 0.9, 0.9]}, 3)
self.assertGreater(float(out[0, 12, 12, 0]), 0.3) # red inside the mask
self.assertEqual(float(out[0, 20, 20].sum()), 0.0) # untouched outside mask and box
self.assertGreater(float(out[0, 3, 16, 1]), 0.5) # yellow box edge
class StitchFinishTest(unittest.TestCase):
def _shot(self):
full = torch.full((4, 64, 64, 3), 0.5)
mask = torch.zeros(4, 64, 64); mask[:, 20:44, 24:40] = 1
return {"full_frames": full, "crop": {"box": [0.125, 0.125, 0.875, 0.875], "mask": mask, "paste": "mask",
"expand": 0, "feather": 2}}
def test_ring_colour_match_removes_drift_and_keeps_subject_contrast(self):
shot = self._shot()
gen = torch.full((4, 48, 48, 3), 0.62) # the model brightened the whole crop (+0.12)
gen[:, 12:36, 16:32] = 0.2 # a darker new subject
plain = SL.uncrop(gen, shot)
fixed = SL.uncrop(gen, shot, match_colors=1.0)
# around the subject the drift is gone, inside it the new subject stays darker than the scene
self.assertGreater(abs(float(plain[0, 21, 23].mean()) - 0.5), abs(float(fixed[0, 21, 23].mean()) - 0.5))
self.assertLess(float(fixed[0, 30, 30].mean()), 0.45)
def test_edge_hardness(self):
a = torch.tensor([[[0.1, 0.5, 0.9]]])
h = SL._alpha_hardness(a, 1.0)
self.assertEqual([round(float(x), 2) for x in h[0, 0]], [0.0, 0.5, 1.0])
self.assertTrue(torch.equal(SL._alpha_hardness(a, 0.0), a))
class InpaintInCropTest(unittest.TestCase):
def test_latent_starts_from_the_shot_and_only_the_mask_is_generated(self):
sys.path.insert(0, str(ROOT))
nested = sys.modules.get("comfy.nested_tensor")
if nested is None:
class _N:
is_nested = True
def __init__(self, t):
self.tensors = list(t)
comfy = sys.modules.setdefault("comfy", types.ModuleType("comfy"))
nested = types.ModuleType("comfy.nested_tensor"); nested.NestedTensor = _N
comfy.nested_tensor = nested
sys.modules["comfy.nested_tensor"] = nested
sys.modules.setdefault("comfy.utils", types.ModuleType("comfy.utils"))
class VAE:
def encode(self, px):
t = ((px.shape[0] - 5) // 17) * 5 + 2
x = px.mean(-1)[None, None]
return torch.nn.functional.interpolate(x, size=(t, px.shape[1] // 16, px.shape[2] // 16)).repeat(1, 24, 1, 1, 1)
lat = {"samples": nested.NestedTensor((torch.zeros(1, 24, 7, 8, 8), torch.zeros(1, 32, 2, 37)))}
frames = torch.full((22, 128, 128, 3), 0.4)
mask = torch.zeros(22, 128, 128); mask[:, 48:80, 48:80] = 1
pos, out = SL.inpaint_latent(lat, [], VAE(), frames, mask)
video = out["samples"].tensors[0]
self.assertAlmostEqual(float(video.mean()), 0.4, places=4) # starts from the shot's own frames
vm, am = out["noise_mask"].tensors
self.assertEqual(tuple(vm.shape), (1, 1, 7, 8, 8))
self.assertEqual(float(vm[0, 0, 3, 4, 4]), 1.0) # the person: generated
self.assertEqual(float(vm[0, 0, 3, 0, 0]), 0.0) # the rest of the crop: kept
self.assertEqual(float(vm[0, 0, 3, 2, 4]), 1.0) # grown by one latent cell
self.assertTrue(bool((am == 1).all())) # audio fully generated
_, out2 = SL.inpaint_latent(lat, [], VAE(), frames, mask, strength=0.8)
vm2 = out2["noise_mask"].tensors[0]
self.assertAlmostEqual(float(vm2[0, 0, 3, 4, 4]), 0.8, places=5) # partly regenerated inside the mask
self.assertEqual(float(vm2[0, 0, 3, 0, 0]), 0.0) # still kept outside
def test_generation_mask_with_and_without_crop(self):
m = torch.zeros(5, 64, 64); m[:, 20:40, 20:40] = 1
cropped = SL.generation_mask({"crop": {"crop_mask": m, "expand": 4}, "frames": torch.zeros(5, 64, 64, 3)})
self.assertEqual(float(cropped[0, 17, 30]), 1.0) # grown by expand
self.assertEqual(float(cropped[0, 10, 30]), 0.0)
with self.assertRaises(ValueError):
SL.generation_mask({"crop": None, "mask_src": None, "frames": torch.zeros(5, 64, 64, 3)})
orig = SL.shot_mask
SL.shot_mask = lambda *a: {"masks": torch.nn.functional.interpolate(m[:, None], size=(32, 32))[:, 0].to(torch.uint8)}
try:
full = SL.generation_mask({"crop": None, "frames": torch.zeros(5, 64, 64, 3),
"mask_src": {"path": "", "analysis": {}, "start": 0, "length": 5,
"spec": {"expand": 0}}})
finally:
SL.shot_mask = orig
self.assertEqual(tuple(full.shape), (5, 64, 64))
self.assertTrue(torch.equal(full, m))
def test_comparison_overlay_for_mask_only_shots(self):
m = torch.zeros(5, 32, 32); m[:, 8:24, 8:24] = 1
ms = {"path": __file__, "analysis": {"fps": 24.0}, "start": 0, "length": 5, "spec": dict(SL.DEFAULT_MASK, expand=0)}
shot = {"crop": None, "frames": torch.zeros(5, 64, 64, 3), "mask_src": ms, "inpaint": True}
orig = SL.shot_mask
SL.shot_mask = lambda *a: {"masks": m.to(torch.uint8)}
try:
ov = SL.overlay_of(shot)
self.assertEqual(tuple(ov["mask"].shape), (5, 64, 64))
self.assertIsNone(SL.overlay_of(dict(shot, inpaint=False))) # not used and not segmented yet: no SAM 3
self.assertIsNone(SL.overlay_of({"crop": None, "frames": shot["frames"]}))
finally:
SL.shot_mask = orig
o = SL._mask_overlay(torch.full((5, 32, 32, 3), 0.5), ov, 5)
self.assertGreater(float(o[0, 16, 16, 0]), float(o[0, 16, 16, 1])) # red inside the mask
self.assertAlmostEqual(float(o[0, 2, 2, 0]), 0.5) # untouched outside
class ExternalMaskTest(unittest.TestCase):
def _video(self, name, n=48, fps=24.0, w=64, h=36, box=(10, 5, 30, 25)):
import cv2
import numpy as np
path = str(Path(_TMP) / name)
vw = cv2.VideoWriter(path, cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
for i in range(n):
f = np.zeros((h, w, 3), np.uint8)
x0, y0, x1, y1 = box
f[y0:y1, x0 + i % 4:x1 + i % 4] = 255
vw.write(f)
vw.release()
return path
def setUp(self):
self.an = {"fps": 24.0, "n": 48, "n_src": 48, "fps_src": 24.0, "width": 64, "height": 36}
def test_precedence(self):
p = {"mask_video": "global.mp4"}
self.assertEqual(SL.plan_spec(p, {"video": "shot.mp4", "text": "man"})["ext"]["scope"], "shot")
self.assertNotIn("ext", SL.plan_spec(p, {"text": "man"})) # the shot's own SAM 3 text wins
self.assertEqual(SL.plan_spec(p, {})["ext"], {"file": "global.mp4", "scope": "video"})
SL._EXT_MASK["v.mp4"] = torch.ones(48, 36, 64)
try:
self.assertEqual(SL.plan_spec(p, {}, "v.mp4")["ext"], {"tensor": "v.mp4"}) # the node input beats the panel file
finally:
SL._EXT_MASK.pop("v.mp4")
self.assertFalse(SL.spec_has_mask(SL.plan_spec({}, {})))
def test_file_and_tensor_masks_replace_sam(self):
path = self._video("roto_whole.mp4")
self._video("roto_shot.mp4", n=12)
orig = SL.segment_frames
SL.segment_frames = lambda *a, **k: self.fail("SAM 3 must not run with an external mask")
try:
r = SL.shot_mask(path, self.an, 24, 12, SL.plan_spec({"mask_video": "roto_whole.mp4"}, {}))
self.assertEqual(r["masks"].shape[0], 12)
self.assertTrue(r["box"] is not None)
cov = float(r["masks"].float().mean())
self.assertTrue(0.1 < cov < 0.4, cov)
r2 = SL.shot_mask(path, self.an, 24, 12, SL.plan_spec({}, {"video": "roto_shot.mp4"}))
self.assertEqual(r2["masks"].shape[0], 12)
t = torch.zeros(48, 36, 64); t[:, 0:18, :] = 1 # top half
SL._EXT_MASK[path] = t
r3 = SL.shot_mask(path, self.an, 0, 10, SL.plan_spec({}, {}, path))
m = r3["masks"].float()
self.assertGreater(float(m[:, :8].mean()), 0.9)
self.assertLess(float(m[:, -8:].mean()), 0.1)
finally:
SL.segment_frames = orig
SL._EXT_MASK.pop(path, None)
class FramePasteTest(unittest.TestCase):
def test_paste_back_keeps_the_background_and_takes_the_new_outline(self):
base = torch.full((5, 64, 64, 3), 0.2) # the original frames
result = torch.full((5, 64, 64, 3), 0.9) # generated: everything changed
old = torch.zeros(5, 64, 64); old[:, 20:44, 20:36] = 1 # the original person
new = torch.zeros(5, 64, 64); new[:, 16:48, 18:44] = 1 # the new one is bigger
ms = {"path": "", "analysis": {}, "start": 0, "length": 5, "spec": dict(SL.DEFAULT_MASK, expand=0, feather=0)}
shot = {"index": 0, "frames": base, "mask_src": ms, "paste": True}
o1, o2 = SL.shot_mask, SL.segment_frames
SL.shot_mask = lambda *a: {"masks": old.to(torch.uint8)}
SL.segment_frames = lambda imgs, sp: torch.nn.functional.interpolate(new[:, None], size=imgs.shape[1:3])[:, 0]
try:
out = SL.paste_back(result, shot, True)
only_old = SL.paste_back(result, shot, False)
finally:
SL.shot_mask, SL.segment_frames = o1, o2
self.assertAlmostEqual(float(out[0, 2, 2, 0]), 0.2, places=4) # background: the original
self.assertAlmostEqual(float(out[0, 30, 28, 0]), 0.9, places=4) # the person: generated
self.assertAlmostEqual(float(out[0, 30, 41, 0]), 0.9, places=4) # the bigger new outline is pasted too
self.assertAlmostEqual(float(only_old[0, 30, 41, 0]), 0.2, places=4)
class MaskGuideTest(unittest.TestCase):
def test_masked_only_greys_everything_outside_the_mask(self):
fr = torch.full((4, 32, 32, 3), 0.9)
m = torch.zeros(4, 16, 16); m[:, 4:12, 4:12] = 1
g = SL.masked_only(fr, m)
self.assertEqual(tuple(g.shape), (4, 32, 32, 3))
self.assertAlmostEqual(float(g[0, 16, 16, 0]), 0.9) # the masked region
self.assertAlmostEqual(float(g[0, 2, 2, 0]), 0.5) # grey elsewhere
self.assertEqual(SL.MASK_GUIDES[0], "off")
fr2 = torch.rand(3, 64, 64, 3)
m2 = torch.zeros(3, 64, 64); m2[:, 10:50, 10:50] = 1
old_det = dict(SL._DET); SL._DET["pose"] = None # no YOLO pose model: a clear error
try:
with self.assertRaisesRegex(ValueError, "YOLO pose model"):
SL.masked_only(fr2, m2, look="pose (people)")
finally:
SL._DET.clear(); SL._DET.update(old_det)
for look in [x for x in SL.MASK_GUIDE_LOOKS if not x.startswith("pose")]:
g2 = SL.masked_only(fr2, m2, look=look)
self.assertEqual(tuple(g2.shape), (3, 64, 64, 3), look)
self.assertAlmostEqual(float(g2[0, 2, 2, 0]), 0.5, msg=look) # grey outside, every look
if look != "colour": # no colour inside
self.assertLess(float((g2[..., 0] - g2[..., 1]).abs().max()), 1e-5, look)
t = SL.add_mask_video("x subject_definitions: <Subject 1> is ...", 2)
self.assertIn("<Video 2> shows only the region", t)
self.assertEqual(SL.add_mask_video("follow {mask_video}", 1), "follow <Video 1>")
class ConditionFlowTest(unittest.TestCase):
"""BFS Shot H3 Conditioning with the native H3 nodes replaced by recorders."""
def setUp(self):
sys.path.insert(0, str(ROOT))
calls = self.calls = {"r2v": [], "guides": []}
class Out:
def __init__(self, *a):
self.args = a
class R2V:
@staticmethod
def execute(**kw):
calls["r2v"].append(kw)
N = sys.modules["comfy.nested_tensor"].NestedTensor
return Out([[torch.zeros(1), {}]], {"samples": N((torch.zeros(1, 24, 2, 4, 4), torch.zeros(1, 32, 2, 37)))})
class Guide:
@staticmethod
def execute(positive, latent, frame_idx, vae, image=None, **kw):
calls["guides"].append(image)
return Out(positive)
mod = types.ModuleType("comfy_extras.nodes_minimax_h3")
mod.MiniMaxH3ReferenceToVideo, mod.MiniMaxH3AddGuide = R2V, Guide
sys.modules.setdefault("comfy_extras", types.ModuleType("comfy_extras"))
self._old = sys.modules.get("comfy_extras.nodes_minimax_h3")
sys.modules["comfy_extras.nodes_minimax_h3"] = mod
if "comfy.nested_tensor" not in sys.modules:
class _N:
is_nested = True
def __init__(self, t):
self.tensors = list(t)
comfy = sys.modules.setdefault("comfy", types.ModuleType("comfy"))
nt = types.ModuleType("comfy.nested_tensor"); nt.NestedTensor = _N
comfy.nested_tensor = nt; sys.modules["comfy.nested_tensor"] = nt
def tearDown(self):
if self._old is not None:
sys.modules["comfy_extras.nodes_minimax_h3"] = self._old
else:
sys.modules.pop("comfy_extras.nodes_minimax_h3", None)
def _shot(self, **kw):
m = torch.zeros(22, 64, 64); m[:, 16:48, 16:40] = 1
shot = {"index": 0, "count": 1, "frames": torch.full((22, 64, 64, 3), 0.7), "ref": torch.zeros(1, 32, 32, 3),
"ref2": None, "prompt": "subject_definitions: <Subject 1> is ...", "width": 64, "height": 64,
"gen_length": 22, "audio": None, "crop": {"crop_mask": m, "expand": 0} if kw.get("crop") else None,
"inpaint": kw.get("inpaint", True)}
if not kw.get("crop"):
shot["mask_src"] = {"path": __file__, "analysis": {"fps": 24.0}, "start": 0, "length": 22, "spec": dict(SL.DEFAULT_MASK, expand=0)}
return shot, m
def run_cond(self, shot, m, **kw):
class VAE:
def encode(self, px):
return torch.zeros(1, 24, 2, px.shape[1] // 16, px.shape[2] // 16)
orig = SL.shot_mask
SL.shot_mask = lambda *a: {"masks": m.to(torch.uint8)}
try:
return SL.BFSShotH3Conditioning().condition(shot, None, VAE(), SL.BFSShotH3Conditioning.GUIDE_MODES[0], False,
"none", "match", **kw)
finally:
SL.shot_mask = orig
def test_reference_video_of_the_masked_region(self):
shot, m = self._shot()
self.run_cond(shot, m, inpaint=SL.INPAINT_MODES[0], mask_guide=SL.MASK_GUIDES[3], mask_ref_size="full",
mask_guide_look="colour")
kw = self.calls["r2v"][0]
v = kw["ref_videos"]["ref_video_1"]
self.assertAlmostEqual(float(v[0, 2, 2, 0]), 0.5) # grey outside the mask
self.assertAlmostEqual(float(v[0, 30, 30, 0]), 0.7) # the subject
self.assertIn("<Video 1> shows only the region", kw["prompt"])
self.assertEqual(len(self.calls["guides"]), 1) # the normal aligned guide is still there
self.calls["r2v"].clear()
self.run_cond(shot, m, inpaint=SL.INPAINT_MODES[0], mask_guide=SL.MASK_GUIDES[3], mask_ref_size="1/2")
self.assertEqual(tuple(self.calls["r2v"][0]["ref_videos"]["ref_video_1"].shape[1:3]), (32, 32))
def test_extra_and_instead_guides(self):
shot, m = self._shot()
self.run_cond(shot, m, inpaint=SL.INPAINT_MODES[0], mask_guide=SL.MASK_GUIDES[1])
self.assertEqual(len(self.calls["guides"]), 2)
self.calls["guides"].clear()
self.run_cond(shot, m, inpaint=SL.INPAINT_MODES[0], mask_guide=SL.MASK_GUIDES[2])
self.assertEqual(len(self.calls["guides"]), 1)
self.assertAlmostEqual(float(self.calls["guides"][0][0, 2, 2, 0]), 0.5) # it is the masked one
def test_ignored_outside_mask_only(self):
shot, m = self._shot(inpaint=False)
self.run_cond(shot, m, inpaint=SL.INPAINT_MODES[0], mask_guide=SL.MASK_GUIDES[3])
self.assertNotIn("ref_videos", self.calls["r2v"][0])
shot, m = self._shot(crop=True)
self.calls["r2v"].clear()
self.run_cond(shot, m, inpaint=SL.INPAINT_MODES[0], mask_guide=SL.MASK_GUIDES[3])
self.assertNotIn("ref_videos", self.calls["r2v"][0])
class DescribeShortTest(unittest.TestCase):
def test_short_details_fit_inside_the_prompt(self):
self.assertEqual(SL.tidy_details('"The green goblin in a torn leather vest."\nextra', {"describe_preset": "short"}),
"green goblin in a torn leather vest")
self.assertEqual(SL.tidy_details("A young woman.", {"describe_preset": "full body"}), "A young woman.")
self.assertIn("one English noun phrase", SL.describe_instruction({}))
def test_every_preset_removes_vlm_answer_labels(self):
for preset in SL.DESCRIBE_PRESETS:
with self.subTest(preset=preset):
self.assertEqual(SL.tidy_details("**Answer:** elderly woman with large glasses", {"describe_preset": preset}),
"elderly woman with large glasses")
def test_short_mode_honours_its_twenty_word_contract(self):
text = "Answer: The " + " ".join(f"word{i}" for i in range(30))
out = SL.tidy_details(text, {"describe_preset": "short"})
self.assertEqual(len(out.split()), 20)
self.assertFalse(out.lower().startswith(("answer", "the ")))
def test_long_modes_are_bounded_but_not_collapsed_to_short(self):
text = " ".join(f"word{i}" for i in range(170))
full = SL.tidy_details(text, {"describe_preset": "full body"})
face = SL.tidy_details(text, {"describe_preset": "head / face"})
self.assertEqual(len(full.split()), 150)
self.assertEqual(len(face.split()), 110)
self.assertTrue(full.endswith("."))
class FitPromptTest(unittest.TestCase):
P = ("body_swap: subject_definitions: <Subject 1> is the {details} shown in <Picture 1>, whose identity define the "
"character. The guide video provides the scene; only {target} is replaced: their face become <Subject 1>. "
"summary: Replace {target} in the guide video with <Subject 1>.")
def test_slot_is_the_sentence_with_a_blank(self):
d = SL.prompt_slot(self.P, "{details}")
self.assertIn("<Subject 1> is the ___ shown in <Picture 1>", d)
self.assertNotIn("{target}", d)
t = SL.prompt_slot(self.P, "{target}")
self.assertIn("only ___ is replaced", t)
self.assertEqual(SL.prompt_slot("no marker here", "{details}"), "")
def test_answer_matches_the_blank(self):
d = SL.prompt_slot(self.P, "{details}")
self.assertEqual(SL.tidy_slot("A golden retriever with a thick coat.", d), "golden retriever with a thick coat")
t = SL.prompt_slot(self.P, "{target}")
self.assertEqual(SL.tidy_slot("the young woman in a black corset.", t), "the young woman in a black corset")
self.assertIn("goes into the blank", SL.describe_instruction({"describe_preset": "short"}, d))
self.assertNotIn("goes into the blank", SL.describe_instruction({"describe_preset": "short", "fit_prompt": False}, d))
self.assertNotIn("goes into the blank", SL.describe_instruction({"describe_preset": "full body"}, d))
# the VLM echoing the sentence (the bug the user hit)
echo = ("humanoid robot is the ___ shown in <Picture 1>, whose identity define the character. The robot has a sleek "
"body with a metallic silver torso.")
self.assertEqual(SL.tidy_slot(echo, d), "humanoid robot")
self.assertEqual(SL.tidy_slot("sleek silver humanoid robot with black joints shown in <Picture 1>", d),
"sleek silver humanoid robot with black joints")
self.assertEqual(SL.tidy_slot("<Subject 1> is the green goblin in a vest", d), "green goblin in a vest")
def test_target_label_is_removed_even_without_a_prompt_slot(self):
old = SL.vlm_generate
SL.vlm_generate = lambda *args, **kwargs: "Answer: elderly woman with large tortoiseshell glasses"
try:
crop = __import__("numpy").zeros((8, 8, 3), dtype="uint8")
self.assertEqual(SL.describe_target(object(), crop),
"the elderly woman with large tortoiseshell glasses")
finally:
SL.vlm_generate = old
class RangeTest(unittest.TestCase):
def test_start_end_and_max_total_from_the_start(self):
self.assertEqual(SL.plan_range(1000, 24.0, 10, 20), (240, 480))
self.assertEqual(SL.plan_range(1000, 24.0, 10, 0), (240, 1000))
self.assertEqual(SL.plan_range(1000, 24.0, 10, 0, 5), (240, 360)) # max total counted from the start
self.assertEqual(SL.plan_range(1000, 24.0, 0, 0, 0, start_f=250, end_f=500), (250, 500)) # frames win
self.assertEqual(SL.plan_range(1000, 24.0, 0, 0, 0, start_f=250, end_f=0), (250, 1000))
segs = SL.plan_segments(480, [100, 300], "shots", 107, 10, start=240)
self.assertEqual(segs[0]["start"], 240)
self.assertEqual(segs[-1]["end"], 480)
self.assertTrue(all(240 <= s["start"] < s["end"] <= 480 for s in segs))
self.assertTrue(any(s["start"] == 300 for s in segs)) # the cut inside the range is kept
fixed = SL.plan_segments(480, [], "fixed", 107, 10, start=240)
self.assertEqual((fixed[0]["start"], fixed[-1]["end"]), (240, 480))
manual = SL.plan_segments(480, [], "manual", 107, 10, manual=[100, 350], start=240)
self.assertEqual([(s["start"], s["end"]) for s in manual], [(240, 350), (350, 480)])
class TrackModeTest(unittest.TestCase):
def test_track_is_used_by_shots_without_their_own_selection_and_sliced_per_shot(self):
p = {"track": {"points": [{"x": 0.5, "y": 0.5, "label": 1}], "frame": 120, "text": "woman"}, "fps": 24}
self.assertIn("track", SL.plan_spec(p, {}, "v.mp4")["ext"])
self.assertNotIn("ext", SL.plan_spec(p, {"text": "man"}, "v.mp4")) # the shot's own text wins
self.assertEqual(SL.plan_spec(p, {"video": "m.mp4"}, "v.mp4")["ext"]["scope"], "shot")
whole = torch.zeros(200, 36, 64, dtype=torch.uint8)
whole[100:150, 10:20, 10:20] = 1 # the subject only in frames 100-149
orig = SL.track_video
SL.track_video = lambda path, a, t: (whole, 50) # the track covers frames 50-249
try:
spec = SL.plan_spec(p, {}, "v.mp4")
an = {"n": 300, "fps": 24.0}
m = SL.ext_masks(spec["ext"], an, 140, 20, 64, 36) # shot 140-159
self.assertEqual(tuple(m.shape), (20, 36, 64))
self.assertFalse(bool(m[0].any())) # frame 140 = track index 90
self.assertTrue(bool(m[-1].any())) # frame 159 = index 109
finally:
SL.track_video = orig
class DuetPromptTest(unittest.TestCase):
def test_fields_are_found_and_the_vlm_fills_them(self):
tpl = ("subject_definitions:\n<Subject 1> is the woman from <Picture 1>: {skin, face}.\n<Subject 2> is the {room} "
"from {setting}.\n\nsummary:\nx {layout}")
self.assertEqual(SL.template_slots(tpl), ["skin, face", "room"])
asked = []
def gen(clip, q, images, max_tokens, temperature=0.0, seed=0):
asked.append(q)
if images is None:
return tpl.replace("{skin, face}", "pale skin, oval face").replace("{room}", "kitchen")
return '{"medium": "phone footage", "actions": "the person waves"}'
old = SL.vlm_generate, SL.vlm_describe
SL.vlm_generate, SL.vlm_describe = gen, lambda clip, ims, q, mt=320, temperature=0.0, seed=0: "the kitchen with white cabinets"
try:
fr = torch.rand(9, 32, 32, 3)
out = SL.duet_prompt(object(), tpl, fr, [torch.rand(1, 32, 32, 3)], "character swap", "", 512,
setting=torch.rand(1, 32, 32, 3))
self.assertIn("pale skin, oval face", out); self.assertIn("{setting}", out); self.assertIn("{layout}", out)
self.assertIn("the person waves", asked[-1])
self.assertEqual(SL.duet_prompt(None, tpl, fr, [], "character swap", ""), tpl.strip())
self.assertEqual(SL.duet_prompt(object(), "a full prompt", fr, [], "character swap", ""), "a full prompt")
w = SL.duet_prompt(object(), "", fr, [torch.rand(1, 32, 32, 3)], "character swap", "", 512,
setting=torch.rand(1, 32, 32, 3))
self.assertIn("<Subject 2> is the kitchen with white cabinets from {setting}.", w)
finally:
SL.vlm_generate, SL.vlm_describe = old
class ShotHistoryTest(unittest.TestCase):
def test_history_is_taken_out_of_the_shot(self):
h = torch.full((17, 8, 8, 3), 255, dtype=torch.uint8)
shot = {"chain": "history", "index": 1, "count": 2, "history": h, "history_src": h}
hist, src = SL.shot_history(shot)
self.assertEqual((tuple(hist.shape), float(hist.max()), "history" in shot), ((17, 8, 8, 3), 1.0, False))
self.assertEqual(SL.shot_history({"chain": "off", "index": 1}), (None, None))
def test_auto_loop_takes_the_previous_render(self):
SL._LAST_RESULT.clear()
SL._LAST_RESULT.update(index=0, count=2, frames=torch.rand(40, 8, 8, 3))
hist, _ = SL.shot_history({"chain": "history", "index": 1, "count": 2, "prev_length": 30, "chain_history": 34})
self.assertEqual(hist.shape[0], 17) # 30 real frames -> one whole group of 17
SL._LAST_RESULT.clear()
class FillDetailsTest(unittest.TestCase):
def test_render_node_fills_details_with_its_vlm(self):
old = SL.vlm_describe
SL.vlm_describe = lambda clip, ims, q, mt=320, temperature=0.0, seed=0: "The elderly man with white hair in a light blue t-shirt."
try:
out = SL.fill_details_now("<Subject 1> is the {details} shown in <Picture 1>.", object(), [torch.rand(1, 8, 8, 3)])
self.assertEqual(out, "<Subject 1> is the elderly man with white hair in a light blue t-shirt shown in <Picture 1>.")
self.assertIn("character", SL.fill_details_now("is the {details} shown", None, []))
self.assertEqual(SL.fill_details_now("no slot", None, []), "no slot")
finally:
SL.vlm_describe = old
if __name__ == "__main__":
unittest.main()
+483
View File
@@ -0,0 +1,483 @@
"""Auto config: every pixel amount is derived from the subject's own size.
The knobs the node exposes (mask_grow, mask_blur, uncrop_feather, the latent
dilation) are absolute pixel amounts, and a pixel only means something relative
to how big the head is in frame. These tests pin the derivation: same head, two
distances, and the amounts must scale with it.
"""
from __future__ import annotations
import importlib.util
import sys
import types
import unittest
from pathlib import Path
import torch
ROOT = Path(__file__).resolve().parents[1]
for name in ("comfy", "comfy.model_management", "comfy.utils"):
sys.modules.setdefault(name, types.ModuleType(name))
# the module reaches comfy.utils.common_upscale as an attribute, so the stub has
# to be attached to the package, not just registered in sys.modules
def _common_upscale(samples, width, height, upscale_method, crop):
return torch.nn.functional.interpolate(samples, size=(height, width), mode="bilinear")
sys.modules["comfy.utils"].common_upscale = _common_upscale
sys.modules["comfy"].utils = sys.modules["comfy.utils"]
sys.modules["comfy"].model_management = sys.modules["comfy.model_management"]
SPEC = importlib.util.spec_from_file_location(
"headswap_ltx_masked_sampler", ROOT / "headswap_ltx_masked_sampler.py")
assert SPEC is not None and SPEC.loader is not None
NODE = importlib.util.module_from_spec(SPEC)
sys.modules["headswap_ltx_masked_sampler"] = NODE
SPEC.loader.exec_module(NODE)
def masks(boxes, h=1080, w=1920):
m = torch.zeros(len(boxes), h, w)
for i, (x, y, bw, bh) in enumerate(boxes):
m[i, y:y + bh, x:x + bw] = 1.0
return m
def auto(boxes, cell=32, ref_aspect=None, headroom=1.15, **kw):
st = NODE._subject_stats(masks(boxes, **kw))
return NODE._auto_config(st, cell, ref_aspect, headroom)
def sides(a):
"""(up, down, left, right) growth."""
return a["mask_grow"]
class SubjectStatsTest(unittest.TestCase):
def test_median_ignores_a_broken_frame(self):
boxes = [(900, 300, 100, 130)] * 20
boxes[7] = (900, 300, 900, 900) # one blown-up mask frame
st = NODE._subject_stats(masks(boxes))
self.assertEqual(st["head_w"], 100.0)
def test_empty_mask_reports_nothing(self):
self.assertIsNone(NODE._subject_stats(torch.zeros(8, 64, 64)))
def test_frames_without_the_subject_do_not_break_travel(self):
boxes = [(100, 300, 100, 130), (0, 0, 0, 0), (900, 300, 100, 130)]
st = NODE._subject_stats(masks(boxes))
self.assertEqual(st["seen"], 2)
self.assertAlmostEqual(st["travel"], 800.0)
class AutoConfigTest(unittest.TestCase):
def test_amounts_scale_with_the_head(self):
near = auto([(700, 200, 600, 760)] * 12)
far = auto([(900, 300, 100, 130)] * 12)
self.assertGreater(sides(near)[2], sides(far)[2] * 4)
self.assertGreater(near["mask_blur"], far["mask_blur"] * 4)
self.assertGreater(near["uncrop_feather"], far["uncrop_feather"])
def test_feather_fits_inside_the_margin_it_has(self):
# a ramp wider than the gap between the mask and the crop border would
# fade the head itself -- this is the seam the auto mode exists to avoid
for w in (60, 100, 240, 600, 900):
a = auto([(400, 200, w, int(w * 1.3))] * 8)
margin = w * (a["crop_scale"] - 1.0) / 2.0
left = margin - sides(a)[2]
self.assertLessEqual(a["uncrop_feather"], max(4.0, 0.5 * left) + 1e-6,
f"feather spills past the margin at head width {w}")
def test_a_head_that_fills_the_frame_turns_cropping_off(self):
self.assertEqual(auto([(300, 100, 1150, 950)] * 8)["crop_mode"], "off")
def test_a_still_subject_gets_one_static_box(self):
self.assertEqual(auto([(900, 300, 120, 150)] * 8)["crop_mode"], "combined")
def test_a_travelling_subject_is_tracked(self):
boxes = [(200 + i * 18, 300, 120, 150) for i in range(40)]
self.assertEqual(auto(boxes)["crop_mode"], "tracked")
def test_a_subject_changing_size_is_zoomed(self):
boxes = [(900 - i * 3, 300, 100 + i * 4, 130 + i * 5) for i in range(40)]
self.assertEqual(auto(boxes)["crop_mode"], "zoomed")
def test_the_box_never_asks_for_more_than_the_frame(self):
a = auto([(100, 100, 1400, 900)] * 8)
self.assertLessEqual(1400 * a["crop_scale"], 1920 + 1e-6)
self.assertLessEqual(900 * a["crop_scale"], 1080 + 1e-6)
def test_the_reference_head_may_be_bigger_than_the_mask(self):
# the mask is the OLD head: a swap clipped at its edge is the seam
tight = auto([(900, 300, 120, 160)] * 8, headroom=1.0)
roomy = auto([(900, 300, 120, 160)] * 8, headroom=1.5)
self.assertGreater(sides(roomy)[0], sides(tight)[0])
self.assertGreater(sides(roomy)[2], sides(tight)[2])
def test_growth_never_goes_down_into_the_neck(self):
for hr in (1.0, 1.25, 1.5, 2.0):
up, down, left, right = sides(auto([(900, 300, 120, 160)] * 8, headroom=hr))
self.assertLess(down, up, "growing down eats the neck and collar")
self.assertLessEqual(down, left + 1)
def test_a_wider_reference_pushes_the_slack_sideways(self):
# below the caps, where the tilt still has room to act
wide = auto([(900, 300, 120, 160)] * 8, ref_aspect=1.4, headroom=1.15)
tall = auto([(900, 300, 120, 160)] * 8, ref_aspect=0.5, headroom=1.15)
self.assertGreater(sides(wide)[2], sides(tall)[2])
self.assertGreater(sides(tall)[0], sides(wide)[0])
def test_the_box_always_holds_the_grow_plus_the_ramp(self):
# the real invariant: whatever the mask grew by, the crop still has room
# for it AND for the feather, or the paste fades the head
for hr in (1.0, 1.15, 1.5, 2.0):
for ref in (0.5, 1.0, 1.6):
a = auto([(900, 300, 120, 160)] * 8, ref_aspect=ref, headroom=hr)
up, down, left, _ = sides(a)
margin_x = 120 * (a["crop_scale"] - 1.0) / 2.0
margin_y = 160 * (a["crop_scale"] - 1.0) / 2.0
self.assertLessEqual(left + a["uncrop_feather"], margin_x + 1)
self.assertLessEqual(max(up, down) + a["uncrop_feather"], margin_y + 1)
def test_latent_dilation_is_not_added_on_top_of_the_pixel_grow(self):
# _mask_to_latent reduces with MAX, so a cell the mask touches is already
# editable; dilating here too added a whole 32 px cell on every side and
# blew the hole up far past the head
for w in (40, 100, 600):
self.assertEqual(auto([(400, 200, w, w)] * 8)["latent_mask_dilate"], 0)
def test_growth_stays_a_sane_fraction_of_the_head(self):
# the failure this pins: slack scaled by the mask's HEIGHT produced a
# hole much larger than the head, with the new head floating inside it
for hw, hh in ((120, 160), (300, 400), (600, 900)):
for ref, hr in ((0.5, 1.15), (1.0, 1.15), (0.75, 1.5), (1.4, 2.0)):
up, down, left, _ = sides(auto([(300, 100, hw, hh)] * 8,
ref_aspect=ref, headroom=hr))
self.assertLessEqual(up, 0.45 * hw, f"up runs away at {hw}x{hh} ref {ref} hr {hr}")
self.assertLessEqual(left, 0.40 * hw, f"sides run away at {hw}x{hh} ref {ref} hr {hr}")
def test_a_fast_head_gets_slack_along_time(self):
still = auto([(800, 300, 100, 130)] * 20)
jumpy = auto([(800 + (60 if i % 2 else 0), 300, 100, 130) for i in range(20)])
self.assertEqual(still["latent_mask_dilate_frames"], 0)
self.assertEqual(jumpy["latent_mask_dilate_frames"], 1)
def test_the_note_says_what_it_chose(self):
note = auto([(900, 300, 120, 150)] * 8)["note"]
for token in ("head", "crop", "grow", "blur", "feather", "latent dilate",
"headroom", "reference aspect"):
self.assertIn(token, note)
class DirectionalGrowTest(unittest.TestCase):
def test_each_side_grows_by_exactly_its_own_amount(self):
m = torch.zeros(1, 40, 40)
m[0, 20, 20] = 1.0
out = NODE._grow_blur(m, (5, 1, 3, 2), 0)[0]
ys, xs = torch.where(out > 0.5)
self.assertEqual((int(ys.min()), int(ys.max())), (15, 21)) # up 5, down 1
self.assertEqual((int(xs.min()), int(xs.max())), (17, 22)) # left 3, right 2
def test_a_single_number_still_grows_every_side(self):
m = torch.zeros(1, 40, 40)
m[0, 20, 20] = 1.0
out = NODE._grow_blur(m, 4, 0)[0]
ys, xs = torch.where(out > 0.5)
self.assertEqual((int(ys.min()), int(ys.max())), (16, 24))
self.assertEqual((int(xs.min()), int(xs.max())), (16, 24))
def test_zero_growth_leaves_the_mask_alone(self):
m = torch.rand(2, 16, 16).round()
self.assertTrue(torch.equal(NODE._grow_blur(m, (0, 0, 0, 0), 0), m))
class GenericIcLoraTest(unittest.TestCase):
"""The node is not head-swap specific: a LoRA that works from the guide
alone must be able to run with the reference slot empty."""
def test_identity_image_is_optional(self):
types = NODE.BFSHeadSwapMaskedSampler.INPUT_TYPES()
self.assertNotIn("identity_image", types["required"])
self.assertIn("identity_image", types["optional"])
def test_execute_defaults_it_to_none(self):
import inspect
sig = inspect.signature(NODE.BFSHeadSwapMaskedSampler.execute)
self.assertIsNone(sig.parameters["identity_image"].default)
def test_the_guide_is_still_required(self):
self.assertIn("guide_video", NODE.BFSHeadSwapMaskedSampler.INPUT_TYPES()["required"])
def test_registration_keys_are_unchanged(self):
# saved workflows reference these, the display name is free to change
self.assertIn("BFSHeadSwapMaskedSampler", NODE.NODE_CLASS_MAPPINGS)
self.assertIn("BFSHeadSwapPasteBack", NODE.NODE_CLASS_MAPPINGS)
def test_the_display_name_no_longer_claims_head_swap(self):
for name in NODE.NODE_DISPLAY_NAME_MAPPINGS.values():
self.assertNotIn("Head Swap", name)
class PasteBackWithoutCropTest(unittest.TestCase):
"""Crop off, mask on: everything the edit did not touch must stay the
source's own pixels, not the VAE's round trip of them."""
def setUp(self):
self.node = NODE.BFSHeadSwapMaskedSampler()
self.original = torch.zeros(3, 8, 8, 3) # source frames
self.result = torch.ones(3, 8, 8, 3) # what the sampler returned
self.mask = torch.zeros(3, 8, 8)
self.mask[:, 2:5, 2:5] = 1.0
def test_outside_the_mask_is_the_source_verbatim(self):
out = self.node._paste_back(self.result, self.original, None, 16, self.mask)
self.assertTrue(torch.equal(out[:, 0, 0], self.original[:, 0, 0]))
self.assertTrue(torch.equal(out[:, 7, 7], self.original[:, 7, 7]))
def test_inside_the_mask_is_the_result(self):
out = self.node._paste_back(self.result, self.original, None, 16, self.mask)
self.assertTrue(torch.equal(out[:, 3, 3], self.result[:, 3, 3]))
def test_without_a_mask_nothing_is_composited(self):
out = self.node._paste_back(self.result, self.original, None, 16, None)
self.assertTrue(torch.equal(out, self.result))
def test_a_result_of_another_size_is_brought_back_to_the_frame(self):
big = torch.ones(3, 16, 16, 3)
out = self.node._paste_back(big, self.original, None, 16, self.mask)
self.assertEqual(out.shape, self.original.shape)
class ChunkOverlapTest(unittest.TestCase):
"""Chunks are sampled with an overlap for continuity. Concatenating them
whole made the clip longer than it is, so every frame after the first seam
landed on the wrong moment -- and, with per-frame crop boxes, in the wrong
box. 97 frames at tile 72 / overlap 16 produced 15 latent frames where the
clip has 13."""
@staticmethod
def run_chunks(lat_per_chunk, total):
produced, out = 0, []
for t in lat_per_chunk:
got, produced = NODE._keep_new_frames(torch.zeros(1, 4, t, 2, 2), produced, total)
out.append(got)
return torch.cat(out, dim=2)
def test_the_concatenation_is_exactly_the_clip(self):
self.assertEqual(self.run_chunks([9, 6], 13).shape[2], 13)
def test_it_keeps_each_chunk_s_tail_not_its_head(self):
a = torch.zeros(1, 4, 9, 2, 2)
b = torch.arange(6, dtype=torch.float32).view(1, 1, 6, 1, 1).expand(1, 4, 6, 2, 2).clone()
_, produced = NODE._keep_new_frames(a, 0, 13)
kept, _ = NODE._keep_new_frames(b, produced, 13)
self.assertEqual(kept.shape[2], 4)
self.assertEqual(kept[0, 0, 0, 0, 0].item(), 2.0) # frames 2..5, the new tail
def test_a_single_chunk_is_untouched(self):
got, produced = NODE._keep_new_frames(torch.zeros(1, 4, 13, 2, 2), 0, 13)
self.assertEqual(got.shape[2], 13)
self.assertEqual(produced, 13)
def test_it_never_returns_more_than_is_left(self):
got, _ = NODE._keep_new_frames(torch.zeros(1, 4, 9, 2, 2), 12, 13)
self.assertEqual(got.shape[2], 1)
class CropAspectTest(unittest.TestCase):
"""The crop is resized into the connected latent, so the box has to share
its aspect. Free-shaped boxes were stretched on the way in and squeezed on
the way back -- a portrait face crop sampled in a landscape latent came out
visibly deformed. The latent's SIZE is deliberately not touched: sampling a
small region at the model's resolution is why cropping exists."""
def setUp(self):
self.node = NODE.BFSHeadSwapMaskedSampler()
self.video = torch.zeros(6, 1080, 1920, 3)
self.mask = torch.zeros(6, 1080, 1920)
self.mask[:, 300:560, 900:1100] = 1.0 # 200x260, portrait
def box(self, aspect):
cropped, _, ctx, _ = self.node._crop(
self.video, self.mask, "off" if False else "combined", 1.2, 32, aspect)
return cropped.shape[2], cropped.shape[1], ctx
def test_a_free_box_follows_the_subject(self):
w, h, _ = self.box(0.0)
self.assertLess(w, h) # portrait, like the mask
def test_a_landscape_latent_gives_a_landscape_box(self):
w, h, _ = self.box(768 / 512)
self.assertAlmostEqual(w / h, 768 / 512, delta=0.25)
self.assertGreater(w, h)
def test_the_box_still_contains_the_subject(self):
w, h, ctx = self.box(768 / 512)
x0, y0, bw, bh = ctx[1]
self.assertLessEqual(x0, 900)
self.assertGreaterEqual(x0 + bw, 1100)
self.assertLessEqual(y0, 300)
self.assertGreaterEqual(y0 + bh, 560)
def test_the_box_stays_divisible(self):
w, h, _ = self.box(1.5)
self.assertEqual((w % 32, h % 32), (0, 0))
def test_static_box_honours_the_aspect_directly(self):
x0, y0, w, h = NODE._static_box(self.mask, self.video, 1.2, 32, aspect=2.0)
self.assertAlmostEqual(w / h, 2.0, delta=0.3)
class CropSizeTest(unittest.TestCase):
"""The size to build the empty latent at, known before sampling instead of
read off the debug string after a whole pass."""
def setUp(self):
self.video = torch.zeros(6, 256, 512, 3)
self.mask = torch.zeros(6, 256, 512)
self.mask[:, 60:160, 200:280] = 1.0 # 80x100, portrait
def size(self, mode="combined", scale=1.5, div=32):
w, h, boxes, info = NODE.BFSCropSize().execute(
self.video, self.mask, mode, scale, div)
return w, h, boxes, info
def test_it_reports_the_size_the_sampler_would_crop_to(self):
w, h, _, _ = self.size()
cropped, _, _, _ = NODE.BFSHeadSwapMaskedSampler()._crop(
self.video, self.mask, "combined", 1.5, 32)
self.assertEqual((w, h), (cropped.shape[2], cropped.shape[1]))
def test_the_size_is_divisible_by_the_grid(self):
for div in (8, 32, 64):
w, h, _, _ = self.size(div=div)
self.assertEqual((w % div, h % div), (0, 0))
def test_it_follows_the_subject_s_aspect_not_the_frame_s(self):
w, h, _, _ = self.size()
self.assertLess(w, h) # the mask is portrait
def test_one_box_per_frame(self):
_, _, boxes, _ = self.size()
self.assertEqual(len(boxes), self.video.shape[0])
def test_the_info_names_the_size(self):
w, h, _, info = self.size()
self.assertIn(f"{w}x{h}", info)
def test_it_is_registered(self):
self.assertIn("BFSCropSize", NODE.NODE_CLASS_MAPPINGS)
self.assertEqual(NODE.NODE_DISPLAY_NAME_MAPPINGS["BFSCropSize"], "BFS Crop Size")
class NoShadowedModulesTest(unittest.TestCase):
"""`import comfy.x` inside a function binds `comfy` as a LOCAL for the whole
function, so every earlier comfy.* use in it raises UnboundLocalError. It
shipped in 1.37.0 and only fired on multi-chunk clips, where the offload
line runs and the nested-latent import never had."""
def test_execute_does_not_shadow_the_comfy_module(self):
for cls in (NODE.BFSHeadSwapMaskedSampler, NODE.BFSHeadSwapPasteBack):
self.assertNotIn("comfy", cls.execute.__code__.co_varnames,
f"{cls.__name__}.execute binds `comfy` locally")
class WidgetOrderTest(unittest.TestCase):
"""ComfyUI stores widgets_values POSITIONALLY.
A widget added anywhere but the end shifts every later value in every saved
workflow — 1.37.0 inserted two before crop_mode and existing graphs failed
validation with "input out of range" on crop_scale and crop_divisible_by.
This pins the historical order; new widgets go after it, never inside it.
"""
HISTORICAL = [
"crop_mode", "crop_scale", "crop_divisible_by", "uncrop_feather",
"paste_back", "paste_confine_to_mask", "inpaint_with_mask", "mask_grow",
"mask_blur", "mask_hard_for_inpaint", "latent_mask_dilate",
"latent_mask_dilate_frames", "mask_strength", "decode",
"decode_tile_size", "decode_overlap", "decode_temporal_size",
"decode_temporal_overlap", "temporal_tile_size", "temporal_overlap",
"guide_source_id", "identity_source_id", "debug_log",
]
@staticmethod
def widgets():
opt = NODE.BFSHeadSwapMaskedSampler.INPUT_TYPES()["optional"]
kinds = ("BOOLEAN", "INT", "FLOAT", "STRING")
return [k for k, v in opt.items() if isinstance(v[0], list) or v[0] in kinds]
def test_the_historical_widgets_keep_their_positions(self):
self.assertEqual(self.widgets()[: len(self.HISTORICAL)], self.HISTORICAL)
def test_new_widgets_are_appended_after_them(self):
self.assertEqual(self.widgets()[len(self.HISTORICAL):],
["auto_config", "identity_headroom"])
class WiringTest(unittest.TestCase):
def test_the_widget_exists_and_defaults_to_off(self):
opt = NODE.BFSHeadSwapMaskedSampler.INPUT_TYPES()["optional"]
self.assertIn("auto_config", opt)
self.assertEqual(opt["auto_config"][1]["default"], False)
self.assertIn("identity_headroom", opt)
self.assertEqual(opt["identity_headroom"][1]["default"], 1.15)
def test_execute_accepts_it(self):
import inspect
sig = inspect.signature(NODE.BFSHeadSwapMaskedSampler.execute)
self.assertIn("auto_config", sig.parameters)
self.assertIs(sig.parameters["auto_config"].default, False)
self.assertEqual(sig.parameters["identity_headroom"].default, 1.15)
if __name__ == "__main__":
unittest.main()
class GuiderInjectionTest(unittest.TestCase):
"""The guider belongs to another node and survives between runs.
Whatever is written into its options has to come back out, or the reference
latents stay reachable (VRAM that is never freed) and the next run samples
with the previous clip's specs.
"""
class _Guider:
def __init__(self, options=None):
self.model_options = options if options is not None else {}
class _Patcher:
def __init__(self, opts):
self.model_options = {"transformer_options": opts}
def test_injection_is_undone(self):
g = self._Guider()
p = self._Patcher({"_id_ref_specs": ["latents"]})
inj = NODE._inject_transformer_options(g, p)
self.assertEqual(g.model_options["transformer_options"]["_id_ref_specs"], ["latents"])
inj.undo()
self.assertNotIn("_id_ref_specs", g.model_options["transformer_options"])
def test_undo_restores_a_pre_existing_value_instead_of_deleting_it(self):
g = self._Guider({"transformer_options": {"_id_ref_specs": "theirs"}})
p = self._Patcher({"_id_ref_specs": "ours"})
NODE._inject_transformer_options(g, p).undo()
self.assertEqual(g.model_options["transformer_options"]["_id_ref_specs"], "theirs")
def test_undo_is_idempotent(self):
g = self._Guider()
inj = NODE._inject_transformer_options(g, self._Patcher({"k": 1}))
inj.undo()
inj.undo()
self.assertEqual(g.model_options["transformer_options"], {})
def test_nothing_to_inject_still_returns_something_undoable(self):
NODE._inject_transformer_options(self._Guider(), self._Patcher({})).undo()
def test_the_loop_undoes_in_a_finally(self):
import inspect
src = inspect.getsource(NODE.BFSHeadSwapMaskedSampler.execute)
self.assertIn("finally:", src)
self.assertIn("inj.undo()", src)
+259
View File
@@ -0,0 +1,259 @@
"""CPU tests against the audited native ComfyUI layout and training fixtures."""
import copy
import importlib.util
import json
from pathlib import Path
import sys
from types import SimpleNamespace
import unittest
from unittest.mock import patch
import numpy as np
from PIL import Image, ImageOps
import torch
ROOT = Path(__file__).resolve().parents[1]
def load(name, path):
spec = importlib.util.spec_from_file_location(name, path)
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
spec.loader.exec_module(module)
return module
NODE = load('bfs_h3_guide_test', ROOT / 'minimax_h3_downscaled_guide.py')
CORE = load('bfs_h3_native_layout_test', ROOT / 'tests/fixtures/comfy_h3_layout_8cfe5e1.py')
def guide(factor, vt=2, h=8, w=8, index=0):
return {'resolved_frame_index': index, 'latent': torch.zeros(1, 24, vt, h//factor, w//factor),
NODE.MARKER: {'downscale_factor': factor}}
class GuideLayoutTest(unittest.TestCase):
def test_positions_match_training_for_factors_1_2_4(self):
golden = json.loads((ROOT/'tests/fixtures/h3_training_guide_positions.json').read_text())
for factor in (1, 2, 4):
with self.subTest(factor=factor):
keyframes = [guide(factor)]
native = CORE.PackedLayout(3, 2, 8, 8, 2, keyframes=keyframes)
layout = NODE.downscaled_layout(native, keyframes)
expected = golden['factors'][str(factor)]
torch.testing.assert_close(layout.position_ids, torch.tensor(expected['positions'], dtype=torch.float64), rtol=0, atol=1e-14)
self.assertEqual(layout.img_pos.tolist(), expected['video_indices'])
self.assertEqual(layout.audio_pos.tolist(), expected['audio_indices'])
self.assertEqual(layout.seq_len, len(expected['positions']))
self.assertEqual(layout.segments[-1][1], layout.seq_len)
self.assertEqual(len(layout.img_pos) + len(layout.audio_pos) + 3, layout.seq_len)
self.assertEqual(native.seq_len, 3+32+4+32)
def test_73_frame_guide_counts_and_frozen_rows(self):
keyframes = [guide(4, vt=22)]
native = CORE.PackedLayout(3, 22, 8, 8, 122, keyframes=keyframes)
layout = NODE.downscaled_layout(native, keyframes)
self.assertEqual(layout.segments[1], (3, 25, 'cond'))
self.assertFalse(layout.img_update[:22].any())
self.assertTrue(layout.img_update[22:].all())
self.assertEqual(layout.seq_len, 3+22+244+22*16)
original_target = native.position_ids[native.segments[-1][0]:]
torch.testing.assert_close(layout.position_ids[layout.segments[-1][0]:], original_target)
def test_multiple_guides_native_refs_and_audio_keep_their_coordinates(self):
keyframes = [guide(4, vt=1), guide(2, vt=2, index=1)]
keyframes[0]['audio_latent'] = torch.zeros(1,32,2,2)
refs = [{'kind':'image','latent_h':4,'latent_w':8}]
native = CORE.PackedLayout(4,7,8,8,40,keyframes=keyframes,refs=refs)
result = NODE.downscaled_layout(native,keyframes)
for orig, changed in zip(native.segments,result.segments):
if orig[2] != 'cond':
torch.testing.assert_close(native.position_ids[orig[0]:orig[1]],result.position_ids[changed[0]:changed[1]])
self.assertEqual(result.segments[-1][1],result.seq_len)
self.assertTrue((result.position_ids[result.segments[1][0],0] == 5).item())
def test_invalid_guide_shape_fails(self):
keyframes=[guide(4)]
keyframes[0]['latent']=torch.zeros(1,24,2,4,4)
native=CORE.PackedLayout(3,2,8,8,2,keyframes=keyframes)
with self.assertRaisesRegex(ValueError,'dimensions'):
NODE.downscaled_layout(native,keyframes)
def test_standard_guide_keeps_full_resolution(self):
keyframes=[guide(4),{'resolved_frame_index':0,'latent':torch.zeros(1,24,1,8,8)}]
native=CORE.PackedLayout(3,2,8,8,2,keyframes=keyframes)
result=NODE.downscaled_layout(native,keyframes)
self.assertEqual(result.segments[2][1]-result.segments[2][0],16)
class PreprocessingTest(unittest.TestCase):
def test_exact_training_crop_and_resize(self):
source=np.random.default_rng(7).integers(0,256,(80,120,3),dtype=np.uint8)
images=torch.from_numpy(source.copy()).float().unsqueeze(0)/255
result=NODE.prepare_guide_frames(images,128,128,4)
expected=ImageOps.fit(Image.fromarray(source),(128,128),Image.Resampling.BICUBIC).resize((32,32),Image.Resampling.LANCZOS)
self.assertEqual(tuple(result.shape),(1,32,32,3))
np.testing.assert_array_equal((result[0].numpy()*255).round().astype(np.uint8),np.array(expected))
def test_bad_canvas_and_factor(self):
image=torch.zeros(1,8,8,3)
for factor in (0,True,1.5):
with self.assertRaises(ValueError): NODE.prepare_guide_frames(image,128,128,factor)
with self.assertRaisesRegex(ValueError,'multiples'): NODE.prepare_guide_frames(image,160,128,4)
def test_posterior_seed_and_fp16_recipe_without_global_rng_changes(self):
moments=torch.zeros(1,48,1,2,2)
moments[:,:24]=.123456
mean=torch.full((24,),.25); std=torch.full((24,),1.5)
before=torch.random.get_rng_state()
result=NODE.sample_posterior(moments,mean,std,fp16_round=True)
generator=torch.Generator('cpu').manual_seed(42)
expected=(.123456+torch.randn((1,24,1,2,2),generator=generator)).half().float()
expected=(expected-.25)/1.5
torch.testing.assert_close(result,expected,rtol=0,atol=0)
self.assertTrue(torch.equal(before,torch.random.get_rng_state()))
torch.testing.assert_close(result,NODE.sample_posterior(moments,mean,std,fp16_round=True))
class EncoderMomentsTest(unittest.TestCase):
def stage(self):
calls = []
def normalize(x):
self.assertEqual(x.dtype, torch.float32)
return x + .123456
def adaptive(x):
calls.append(x.clone())
self.assertEqual(x.dtype, torch.float16)
return torch.zeros(1,48,5,2,2)
return SimpleNamespace(clip_length=17, token_drop=3,
_normalize_pixels=normalize, _adaptive_encode=adaptive), calls
def test_image_uses_last_posterior_slice(self):
stage,calls = self.stage()
result = NODE.encode_moments(stage,torch.zeros(1,3,1,32,32),'cpu',torch.float16)
self.assertEqual(tuple(result.shape),(1,48,1,2,2))
self.assertEqual(len(calls),1)
def test_video_chunk_padding_and_tail_drop_match_native_h3(self):
stage,calls = self.stage()
pixels = torch.arange(73,dtype=torch.float32).view(1,1,73,1,1).expand(1,3,73,2,2)
result = NODE.encode_moments(stage,pixels,'cpu',torch.float16)
self.assertEqual(result.shape[2],22)
self.assertEqual(len(calls),5)
self.assertEqual(calls[-1].shape[2],17)
torch.testing.assert_close(calls[-1][:,:,4],calls[-1][:,:,16])
class FakeModel:
def __init__(self, options=None): self.model_options=options or {}
def clone(self): return FakeModel(dict(self.model_options))
def set_model_unet_function_wrapper(self,fn): self.model_options['model_function_wrapper']=fn
class WrapperTest(unittest.TestCase):
def test_scoped_wrapper_composes_and_does_not_mutate_payload(self):
calls=[]
def previous(fn,args):
calls.append('previous')
return fn(args['input'],args['timestep'],**args['c'])
original=FakeModel({'model_function_wrapper':previous})
patched=NODE.patch_guide_model(original)
keyframes=[guide(4)]
native=CORE.PackedLayout(3,2,8,8,2,keyframes=keyframes)
payload={'keyframes':keyframes,'layout':native}
args={'input':'x','timestep':'t','c':{'minimax_payload':payload}}
def model_fn(x,t,**c): return c['minimax_payload']['layout']
result=patched.model_options['model_function_wrapper'](model_fn,args)
self.assertEqual(result.seq_len,41)
self.assertIs(payload['layout'],native)
self.assertIs(original.model_options['model_function_wrapper'],previous)
self.assertEqual(calls,['previous'])
again=NODE.patch_guide_model(patched)
self.assertIs(again.model_options['model_function_wrapper'],patched.model_options['model_function_wrapper'])
self.assertIs(patched.model_options['model_function_wrapper'](model_fn,args),result)
def test_other_models_and_conditioning_are_untouched(self):
model=NODE.patch_guide_model(FakeModel())
args={'input':1,'timestep':2,'c':{'other':3}}
result=model.model_options['model_function_wrapper'](lambda x,t,**c:(x,t,c),args)
self.assertEqual(result,(1,2,{'other':3}))
class NodeExecutionTest(unittest.TestCase):
def setUp(self):
import types
self.comfy=types.ModuleType('comfy')
ldm=types.ModuleType('comfy.ldm'); minimax=types.ModuleType('comfy.ldm.minimax')
core=types.ModuleType('comfy.ldm.minimax.model')
class Diffusion:
patch_size=(1,2,2)
core.MiniMaxH3Model=Diffusion
core.FRAME_PER_TOKEN=(1,4,4,4,4); core.FRAME_RESCALE=5/3
minimax.model=core; ldm.minimax=minimax; self.comfy.ldm=ldm
self.modules={'comfy':self.comfy,'comfy.ldm':ldm,'comfy.ldm.minimax':minimax,'comfy.ldm.minimax.model':core}
self.diffusion=Diffusion()
self.model=FakeModel()
self.model.get_model_object=lambda key:self.diffusion
def target(self, vt):
return {'samples':SimpleNamespace(is_nested=True,tensors=[torch.zeros(1,24,vt,8,8),torch.zeros(1,32,2,122)])}
def test_image_node_encodes_reduced_guide_and_keeps_inputs(self):
embedding=torch.zeros(1,3,4)
positive=[[embedding,{'minimax_token_tags':torch.ones(3)}]]
with patch.dict(sys.modules,self.modules), patch.object(NODE,'encode_guide',return_value=torch.zeros(1,24,1,2,2)) as encode:
model,cond,preview=NODE.BFSMiniMaxH3DownscaledGuide().apply(self.model,positive,None,self.target(1),torch.zeros(1,128,128,3))
self.assertEqual(tuple(preview.shape),(1,32,32,3))
self.assertEqual(tuple(encode.call_args[0][1].shape),(1,32,32,3))
self.assertNotIn('minimax_keyframes',positive[0][1])
self.assertIs(cond[0][0],embedding)
self.assertEqual(cond[0][1]['minimax_keyframes'][0][NODE.MARKER]['downscale_factor'],4)
self.assertTrue(model.model_options[NODE.WRAPPER_MARKER])
def test_video_node_validates_73_frames_without_truncation(self):
with patch.dict(sys.modules,self.modules), patch.object(NODE,'encode_guide',return_value=torch.zeros(1,24,22,2,2)):
_,cond,preview=NODE.BFSMiniMaxH3DownscaledGuide().apply(self.model,[[None,{}]],None,self.target(22),torch.zeros(73,32,32,3))
self.assertEqual(preview.shape[0],73)
self.assertEqual(cond[0][1]['minimax_keyframes'][0]['latent'].shape[2],22)
def test_invalid_video_or_temporal_window_fails_before_encoding(self):
for frames,index in [(3,0),(72,0),(73,1),(1,-74)]:
with patch.dict(sys.modules,self.modules), patch.object(NODE,'encode_guide') as encode:
with self.assertRaises(ValueError):
NODE.BFSMiniMaxH3DownscaledGuide().apply(self.model,[[None,{}]],None,self.target(22),torch.zeros(frames,32,32,3),frame_idx=index)
encode.assert_not_called()
def test_true_single_frame_and_video_targets(self):
import types
management=types.ModuleType('comfy.model_management')
management.intermediate_device=lambda:torch.device('cpu')
nested=types.ModuleType('comfy.nested_tensor')
nested.NestedTensor=lambda tensors:SimpleNamespace(tensors=tensors,is_nested=True)
self.comfy.model_management=management
modules=dict(self.modules,**{'comfy.model_management':management,'comfy.nested_tensor':nested})
with patch.dict(sys.modules,modules):
for frames,vt,at in [(1,1,2),(5,2,8),(73,22,122),(124,37,207)]:
result=NODE.BFSMiniMaxH3GuideTarget().create(128,128,frames)[0]['samples']
self.assertEqual(tuple(result.tensors[0].shape),(1,24,vt,8,8))
self.assertEqual(tuple(result.tensors[1].shape),(1,32,2,at))
with self.assertRaises(ValueError): NODE.BFSMiniMaxH3GuideTarget().create(128,128,72)
def test_metadata_factor_and_version_validation(self):
import tempfile, types
from safetensors.torch import save_file
with tempfile.TemporaryDirectory() as temp:
path=str(Path(temp)/'guide.safetensors')
save_file({'x':torch.zeros(1)},path,metadata={
'reference_downscale_factor':'4','guide_latent_only':'true',
'minimax_h3_guide_spatial_version':NODE.SPATIAL_VERSION,
'minimax_h3_guide_position_version':NODE.POSITION_VERSION})
folder=types.ModuleType('folder_paths'); folder.get_full_path_or_raise=lambda *args:path
with patch.dict(sys.modules,{'folder_paths':folder}):
NODE.check_lora_metadata('guide.safetensors',4)
with self.assertRaisesRegex(ValueError,'factor'): NODE.check_lora_metadata('guide.safetensors',2)
save_file({'x':torch.zeros(1)},path,metadata={})
with patch.dict(sys.modules,{'folder_paths':folder}):
with self.assertRaisesRegex(ValueError,'spatial_version'): NODE.check_lora_metadata('guide.safetensors',4)
if __name__ == "__main__":
unittest.main()
+71
View File
@@ -0,0 +1,71 @@
import json
import math
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
import torch
from safetensors.torch import save_file
from test_minimax_h3_downscaled_guide import NODE,CORE,FakeModel,NodeExecutionTest,guide
class PhaseModel(FakeModel):
def __init__(self,options=None,objects=None):
super().__init__(options);self.objects=objects or {}
def clone(self):return PhaseModel(dict(self.model_options),dict(self.objects))
def get_model_object(self,key):
return self.objects.get(key,self.original)
def add_object_patch(self,key,value):self.objects[key]=value
@staticmethod
def original(positions,device):
inv=10000**(-torch.arange(16)/16)
half=(positions.float().unsqueeze(-1)*inv).flatten(1)
return torch.cat((half,half),-1).to(device)
class SourcePhaseTest(unittest.TestCase):
def test_patch_is_scoped_composable_and_does_not_leak_to_unmarked_payload(self):
original=PhaseModel();patched=NODE.patch_guide_model(original,True)
kf=guide(4);kf[NODE.MARKER].update(source_phase=True,source_id=3,phase_scale=.7)
native=CORE.PackedLayout(3,2,8,8,2,keyframes=[kf])
def forward(x,t,**kwargs):
layout=kwargs['minimax_payload']['layout']
return patched.get_model_object('diffusion_model.rope_freqs')(layout.position_ids,'cpu'),layout
fn=patched.model_options['model_function_wrapper']
args={'input':0,'timestep':0,'c':{'minimax_payload':{'layout':native,'keyframes':[kf]}}}
angles,layout=fn(forward,args)
expected=NODE.add_source_phase(PhaseModel.original(layout.position_ids,'cpu'),layout.bfs_source_phase_values)
torch.testing.assert_close(angles,expected)
self.assertFalse(original.objects)
again=NODE.patch_guide_model(patched,True)
self.assertIs(again.objects['diffusion_model.rope_freqs'],patched.objects['diffusion_model.rope_freqs'])
blank=CORE.PackedLayout(3,2,8,8,2)
angles,_=fn(forward,{'input':0,'timestep':0,'c':{'minimax_payload':{'layout':blank}}})
torch.testing.assert_close(angles,PhaseModel.original(blank.position_ids,'cpu'))
def test_default_and_experimental_metadata_are_checked(self):
import types
with tempfile.TemporaryDirectory() as directory:
path=str(Path(directory)/'lora.safetensors')
meta={'minimax_h3_guide_spatial_version':NODE.SPATIAL_VERSION,'minimax_h3_guide_position_version':NODE.POSITION_VERSION,
'reference_downscale_factor':'4','guide_latent_only':'true'}
opts=NODE.validate_options('overlap','sidecar',True,.7,2)
meta['minimax_h3_reference_rope']=json.dumps(dict(opts,phase_version=NODE.PHASE_VERSION))
save_file({'w':torch.zeros(1)},path,metadata=meta)
folders=types.SimpleNamespace(get_full_path_or_raise=lambda *args:path)
with patch.dict('sys.modules',{'folder_paths':folders}):
NODE.check_lora_metadata('lora',4,opts)
with self.assertRaisesRegex(ValueError,'RoPE options'):NODE.check_lora_metadata('lora',4)
def test_identity_node_uses_own_aspect_and_training_image_bucket(self):
harness=NodeExecutionTest();harness.setUp()
positive=[[torch.zeros(1,3,4),{}]]
image=torch.zeros(1,192,64,3)
with patch.dict('sys.modules',harness.modules),patch.object(NODE,'encode_guide',return_value=torch.zeros(1,24,1,12,4)) as encoder:
model,conditioning,preview=NODE.BFSMiniMaxH3IdentityReference().apply(harness.model,positive,None,harness.target(2),image)
self.assertEqual(tuple(preview.shape),(1,192,64,3))
block=conditioning[0][1]['minimax_refs'][0]
self.assertEqual(block[NODE.MARKER]['source_id'],2)
self.assertEqual((block['latent_h'],block['latent_w']),(12,4))
self.assertNotIn('minimax_refs',positive[0][1])
self.assertEqual(encoder.call_count,1)
if __name__=='__main__':unittest.main()
+141
View File
@@ -0,0 +1,141 @@
from __future__ import annotations
import importlib.util
import sys
import types
import unittest
from pathlib import Path
import torch
ROOT = Path(__file__).resolve().parents[1]
MODULE_PATH = ROOT / "reference_temporal_offset.py"
SPEC = importlib.util.spec_from_file_location("reference_temporal_offset", MODULE_PATH)
assert SPEC is not None and SPEC.loader is not None
OFFSET = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(OFFSET)
class ReferenceTemporalOffsetTest(unittest.TestCase):
def test_default_matches_standard_frame_zero_placement(self):
kind, options = OFFSET.reference_temporal_offset_input()
self.assertEqual(kind, "INT")
self.assertEqual(options["default"], 0)
self.assertEqual(OFFSET.reference_temporal_pixel_offset(0, 8), 0)
self.assertEqual(OFFSET.reference_temporal_pixel_offset(-1, 8), -8)
def test_shift_changes_only_temporal_axis_without_mutating_input(self):
positions = torch.tensor(
[[[0.0, 8.0], [10.0, 20.0], [30.0, 40.0]]],
dtype=torch.float32,
)
shifted = OFFSET.shift_reference_temporal_positions(positions, -1, 8)
torch.testing.assert_close(positions[:, 0], torch.tensor([[0.0, 8.0]]))
torch.testing.assert_close(shifted[:, 0], torch.tensor([[-8.0, 0.0]]))
torch.testing.assert_close(shifted[:, 1:], positions[:, 1:])
self.assertIsNot(shifted, positions)
def test_zero_is_exact_noop_for_legacy_specs(self):
positions = torch.randn(1, 3, 4, 2)
shifted = OFFSET.shift_reference_temporal_positions(positions, 0, 8)
self.assertIs(shifted, positions)
def test_multiple_controls_exposes_only_identity_offset(self):
source = (ROOT / "ltx_multiple_controls.py").read_text(encoding="utf-8")
self.assertIn(
'"identity_temporal_offset_latents": reference_temporal_offset_input()',
source,
)
self.assertNotIn('"guide_temporal_offset_latents"', source)
self.assertNotIn('"mask_temporal_offset_latents"', source)
self.assertNotIn('"identity_mask_temporal_offset_latents"', source)
def test_edit_anything_offsets_reference_but_keeps_guide_at_zero(self):
package_name = "bfsnodes_offset_test"
package = types.ModuleType(package_name)
package.__path__ = [str(ROOT)]
sys.modules[package_name] = package
folder_paths = types.ModuleType("folder_paths")
folder_paths.get_filename_list = lambda _kind: []
sys.modules.setdefault("folder_paths", folder_paths)
comfy_extras = types.ModuleType("comfy_extras")
comfy_extras.__path__ = []
nodes_lt = types.ModuleType("comfy_extras.nodes_lt")
frame_indices = []
class AddGuide:
@staticmethod
def append_keyframe(
positive,
negative,
frame_idx,
latent_image,
noise_mask,
_guiding_latent,
_strength,
_scale_factors,
causal_fix,
):
self.assertTrue(causal_fix)
frame_indices.append(frame_idx)
return positive, negative, latent_image, noise_mask
nodes_lt.LTXVAddGuide = AddGuide
nodes_lt._append_guide_attention_entry = (
lambda positive, negative, *args, **kwargs: (
positive,
negative,
)
)
comfy_extras.nodes_lt = nodes_lt
sys.modules["comfy_extras"] = comfy_extras
sys.modules["comfy_extras.nodes_lt"] = nodes_lt
module_path = ROOT / "ltxv_editanything.py"
spec = importlib.util.spec_from_file_location(
f"{package_name}.ltxv_editanything",
module_path,
)
assert spec is not None and spec.loader is not None
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
target = torch.zeros(1, 128, 2, 1, 1)
mask = torch.ones(1, 1, 2, 1, 1)
reference = torch.zeros(1, 128, 1, 1, 1)
guide = torch.zeros(1, 128, 2, 1, 1)
module.LTXVEditAnythingLoopingSampler._build_chunk_conds(
"positive",
"negative",
target,
mask,
reference,
guide,
(8, 32, 32),
1.0,
1.0,
-1,
True,
)
self.assertEqual(frame_indices, [-8, 0])
def test_all_reference_injection_nodes_expose_the_control(self):
expected_fields = {
"ltx_identity_overlap.py": "reference_temporal_offset_latents",
"ltx_identity_multiangle.py": "reference_temporal_offset_latents",
"ltx_multiple_controls.py": "identity_temporal_offset_latents",
"ltxv_editanything.py": "ref_temporal_offset_latents",
}
for filename, field in expected_fields.items():
with self.subTest(filename=filename):
source = (ROOT / filename).read_text(encoding="utf-8")
self.assertIn(field, source)
if __name__ == "__main__":
unittest.main()
+141
View File
@@ -0,0 +1,141 @@
"""Reference-layout placement, on both LTX coordinate conventions.
The node runs against two shapes and two RoPE collapse rules, and the sidecar layout is the
one that is sensitive to all of them:
[B, 3, N, 2] patch bounds -- current builds, LTX-2.3 and LTX-2.5 alike
[B, 3, N] single corner -- legacy builds
use_middle_indices_grid -- True (2.5) collapses a token to (start+end)/2, False (2.3
checkpoints whose metadata omits the flag) reads the start
`_collapse` below mirrors comfy.ldm.lightricks.model.generate_freqs, so these tests compare
the position the model actually sees, not the raw tensor.
"""
from __future__ import annotations
import importlib
import sys
import types
import unittest
from pathlib import Path
import torch
ROOT = Path(__file__).resolve().parents[1]
_PKG = types.ModuleType("bfsnodes_under_test")
_PKG.__path__ = [str(ROOT)]
sys.modules.setdefault("bfsnodes_under_test", _PKG)
OVERLAP = importlib.import_module("bfsnodes_under_test.ltx_identity_overlap")
def _bounds(batch, tokens, t_span=576.0, height=384.0, width=704.0, seed=0):
"""Patch bounds [B, 3, N, 2] with a plausible per-patch extent."""
generator = torch.Generator().manual_seed(seed)
low = torch.stack([
torch.rand(tokens, generator=generator) * t_span,
torch.rand(tokens, generator=generator) * height,
torch.rand(tokens, generator=generator) * width,
])
high = low + torch.tensor([8.0, 32.0, 32.0]).unsqueeze(1)
return torch.stack([low, high], dim=-1).unsqueeze(0).expand(batch, -1, -1, -1).contiguous()
def _collapse(coords, use_middle):
"""The one number per token the RoPE actually uses -- see generate_freqs."""
if use_middle and coords.dim() == 4:
return (coords[..., 0] + coords[..., 1]) / 2.0
return coords[..., 0] if coords.dim() == 4 else coords
def _axis_centre(coords, axis):
return (coords[0, axis].min() + coords[0, axis].max()) / 2.0
class TassLayoutTest(unittest.TestCase):
def setUp(self):
# A batch-2 target (CFG) against a batch-1 reference: layouts run before the reference
# is expanded, which is what crashed the first sidecar run.
self.target = _bounds(2, 512, seed=1)
self.reference = _bounds(1, 64, t_span=8.0, height=128.0, width=128.0, seed=2)
def test_sidecar_places_reference_at_clip_centre_under_both_collapse_rules(self):
for use_middle in (True, False):
with self.subTest(use_middle=use_middle):
shifted = OVERLAP._apply_tass_layout(
self.reference, self.target, "sidecar", use_middle_indices_grid=use_middle
)
reference_seen = _collapse(shifted, use_middle)
target_seen = _collapse(self.target, use_middle)
# T: centred on the target's clip, not pinned to frame zero
self.assertAlmostEqual(
float(_axis_centre(reference_seen, 0)), float(_axis_centre(target_seen, 0)), delta=4.0
)
# H: centred on the target's height
self.assertAlmostEqual(
float(_axis_centre(reference_seen, 1)), float(_axis_centre(target_seen, 1)), delta=1e-3
)
# W: entirely beside the target, never overlapping it
self.assertGreaterEqual(
float(reference_seen[0, 2].min()), float(target_seen[0, 2].max()) - 1e-3
)
def test_sidecar_survives_legacy_single_corner_coordinates(self):
shifted = OVERLAP._apply_tass_layout(
self.reference[..., 0].contiguous(),
self.target[..., 0].contiguous(),
"sidecar",
use_middle_indices_grid=False,
)
self.assertEqual(shifted.shape, self.reference[..., 0].shape)
target = self.target[..., 0]
self.assertAlmostEqual(float(_axis_centre(shifted, 0)), float(_axis_centre(target, 0)), delta=4.0)
self.assertAlmostEqual(float(_axis_centre(shifted, 1)), float(_axis_centre(target, 1)), delta=1e-3)
self.assertGreaterEqual(float(shifted[0, 2].min()), float(target[0, 2].max()) - 1e-3)
def test_sidecar_margin_pushes_the_reference_further_right(self):
flush = OVERLAP._apply_tass_layout(self.reference, self.target, "sidecar")
spaced = OVERLAP._apply_tass_layout(
self.reference, self.target, "sidecar", sidecar_margin_pixels=64.0
)
self.assertAlmostEqual(float(spaced[0, 2].min() - flush[0, 2].min()), 64.0, delta=1e-3)
def test_patch_bounds_are_translated_not_squeezed(self):
span = (self.reference[..., 1] - self.reference[..., 0])
for layout, kwargs in (("st_drc", {}), ("strata", {"strata_start": 600.0}), ("sidecar", {})):
with self.subTest(layout=layout):
shifted = OVERLAP._apply_tass_layout(self.reference, self.target, layout, **kwargs)
moved = shifted[..., 1] - shifted[..., 0]
# sidecar deliberately rewrites the temporal span; H/W must keep their extent
axes = slice(1, 3) if layout == "sidecar" else slice(0, 3)
# atol covers float32 rounding on ~600px coordinates, not a real drift
torch.testing.assert_close(moved[:, axes], span[:, axes], atol=1e-3, rtol=0)
def test_integer_coordinates_keep_their_dtype(self):
"""ComfyUI hands out int64 pixel coords -- float shifts must not blow up on them."""
target = self.target.round().to(torch.int64)
reference = self.reference.round().to(torch.int64)
for layout, kwargs in (
("overlap", {}), ("st_drc", {}), ("strata", {"strata_start": 600.0}), ("sidecar", {})
):
with self.subTest(layout=layout):
shifted = OVERLAP._apply_tass_layout(reference, target, layout, **kwargs)
self.assertEqual(shifted.dtype, torch.int64)
float_result = OVERLAP._apply_tass_layout(
reference.float(), target.float(), layout, **kwargs
)
# rounded, never truncated: at most half a pixel from the exact placement
self.assertLessEqual(float((shifted.float() - float_result).abs().max()), 0.5)
def test_overlap_is_an_exact_noop(self):
self.assertIs(
OVERLAP._apply_tass_layout(self.reference, self.target, "overlap"), self.reference
)
def test_unknown_layout_is_rejected(self):
with self.assertRaises(ValueError):
OVERLAP._apply_tass_layout(self.reference, self.target, "nope")
if __name__ == "__main__":
unittest.main()
+171
View File
@@ -0,0 +1,171 @@
// "❓ How to use" for the H3 duet nodes: a window with drawn examples of every setting.
import { app } from "../../scripts/app.js";
import { showGuide } from "./bfs_md_view.js";
const NODES = ["BFSH3Duet", "BFSShotH3Duet", "BFSH3SidePanel", "BFSH3DuetConditioning"];
// ---- small SVG diagrams -------------------------------------------------------------------
const C = { video: "#3b6fd8", panel: "#d88a2b", grey: "#7a7a7a", ink: "#e8e8ee", dim: "#9a9aa8", bg: "#1b1b21" };
function canvas({ position = "left", size = 1.0, fit = "contain", gap = 0, w = 150, h = 96, label = "" }) {
// video area keeps 16:9-ish; the strip sits on one side
const horiz = position === "left" || position === "right";
const vw = horiz ? w / (1 + size + gap * 0.08) : w, vh = horiz ? h : h / (1 + size + gap * 0.08);
const sw = horiz ? vw * size : vw, sh = horiz ? vh : vh * size, g = gap * (horiz ? vw : vh) * 0.08;
const vx = position === "left" ? sw + g : 0, vy = position === "top" ? sh + g : 0;
const px = position === "right" ? vw + g : 0, py = position === "bottom" ? vh + g : 0;
// panel content (the source clip, 16:9) placed in the strip
const ar = vw / vh;
let cw = sw, ch = sh;
if (fit === "contain") { if (sw / sh > ar) cw = sh * ar; else ch = sw / ar; }
const cx = px + (sw - cw) / 2, cy = py + (sh - ch) / 2;
const W = horiz ? vw + sw + g : vw, H = horiz ? vh : vh + sh + g;
const clip = fit === "cover" && Math.abs(sw / sh - ar) > 1e-3;
// cover: the clip is scaled to fill the strip; what sticks out is cropped (dashed)
let ow = sw, oh = sh;
if (clip) { if (sw / sh > ar) oh = sw / ar; else ow = sh * ar; }
const ox = px + (sw - ow) / 2, oy = py + (sh - oh) / 2;
const pad = clip ? Math.max(0, (ow - sw) / 2, (oh - sh) / 2) + 2 : 2;
return `<svg viewBox="${-pad} ${-pad} ${W + 2 * pad} ${H + 16 + 2 * pad}" width="${W + 2 * pad}" height="${H + 16 + 2 * pad}">
<rect x="${px}" y="${py}" width="${sw}" height="${sh}" fill="${C.grey}" opacity=".45"/>
${clip ? `<rect x="${ox}" y="${oy}" width="${ow}" height="${oh}" fill="none" stroke="${C.panel}" stroke-dasharray="3 2"/>
<rect x="${px}" y="${py}" width="${sw}" height="${sh}" fill="${C.panel}"/>
<text x="${ox + 2}" y="${oy + 9}" fill="${C.dim}" font-size="8">cropped</text>` :
`<rect x="${cx}" y="${cy}" width="${cw}" height="${ch}" fill="${C.panel}"/>`}
<text x="${px + sw / 2}" y="${py + sh / 2 + 4}" fill="#fff" font-size="10" text-anchor="middle">panel</text>
${g ? `<rect x="${horiz ? (position === "left" ? sw : vw) : 0}" y="${horiz ? 0 : (position === "top" ? sh : vh)}" width="${horiz ? g : vw}" height="${horiz ? vh : g}" fill="${C.grey}"/>` : ""}
<rect x="${vx}" y="${vy}" width="${vw}" height="${vh}" fill="${C.video}"/>
<text x="${vx + vw / 2}" y="${vy + vh / 2 + 4}" fill="#fff" font-size="10" text-anchor="middle">video (output)</text>
<text x="${W / 2}" y="${H + 13}" fill="${C.dim}" font-size="10" text-anchor="middle">${label}</text></svg>`;
}
function rope(mode, gap) {
const n = 7, step = 18, y = 22;
let s = "", x = 6;
for (let i = 0; i < n; i++) { s += `<circle cx="${x}" cy="${y}" r="5" fill="${C.panel}"/>`; x += step; }
if (mode === "shifted") x += gap * step * 0.5;
const gx0 = x - step;
for (let i = 0; i < n; i++) { s += `<circle cx="${x}" cy="${y}" r="5" fill="${C.video}"/>`; x += step; }
const label = mode === "canvas" ? "canvas: one wide grid, panel and video side by side"
: `shifted: video keeps its own positions, panel further out${gap ? ` (gap ${gap})` : ""}`;
const W = Math.max(x + 4, 340);
return `<svg viewBox="0 0 ${W} 46" width="${W}" height="46">${s}
${mode === "shifted" && gap ? `<line x1="${gx0 + 8}" y1="${y}" x2="${gx0 + gap * step * 0.5 + step - 8}" y2="${y}" stroke="${C.dim}" stroke-dasharray="3 3"/>` : ""}
<text x="4" y="42" fill="${C.dim}" font-size="10">${label}</text></svg>`;
}
function hold(kind) {
const n = 7; let s = "";
for (let i = 0; i < n; i++) {
const held = kind === "all frames" || i === 0;
s += `<rect x="${4 + i * 20}" y="6" width="16" height="16" fill="${held ? C.panel : C.grey}" opacity="${held ? 1 : .45}"/>`;
}
return `<svg viewBox="0 0 260 40" width="260" height="40">${s}<text x="4" y="36" fill="${C.dim}" font-size="10">${kind}: panel held on the orange frames</text></svg>`;
}
const row = (...cells) => `<div class="bdh-row">${cells.map(c => `<div class="bdh-cell">${c}</div>`).join("")}</div>`;
// ---- content ------------------------------------------------------------------------------
function html(type) {
const shot = type === "BFSShotH3Duet";
return `
<h2>H3 Duet: how to use</h2>
<p>The <b>panel</b> (a clip or a picture) is pinned next to the video in one wide canvas. MiniMax H3 keeps the panel
exactly and generates the video beside it, <b>in sync</b>: the new video copies the panel's motion, camera, timing and
cuts. The panel is cut off before decoding, so only the video comes out. No LoRA is needed.</p>
${row(canvas({ label: "the canvas the model sees" }), `<svg width="40" height="96"><text x="8" y="52" fill="${C.ink}" font-size="22">→</text></svg>`,
canvas({ w: 75, size: 0.0001, label: "what you get" }).replace(">panel<", "><"))}
<h3>What goes in</h3>
<ul>
<li><b>panel</b>${shot ? " (the shot's own clip, set by the mode)" : ""}: usually the <b>source clip</b> you want to restyle or recast. A single picture also works but copies less.</li>
<li><b>ref_images</b> → <code>&lt;Picture 1&gt;</code>, <code>&lt;Picture 2&gt;</code>…: the new person, outfit or place.</li>
<li><b>guide</b> (optional): an aligned latent guide, for LoRAs trained on it. Usually leave it empty with a panel.</li>
</ul>
<h3>Prompt, task and instruction</h3>
<p><b>prompt</b> always wins. Leave it empty and the node writes a <b>draft</b> from <b>task</b> + <b>instruction</b>
(see the <code>prompt</code> output). The draft cannot see the video, so a prompt written for the clip is better.</p>
<table>
<tr><th>task</th><th>instruction: a few words of what is SEEN</th></tr>
<tr><td>character swap</td><td><i>(empty, or the look)</i> + the person's pictures as references</td></tr>
<tr><td>style</td><td><code>a 1990s anime cel style</code></td></tr>
<tr><td>setting</td><td><code>a sunny beach at sunset, waves behind her</code></td></tr>
<tr><td>appearance</td><td><code>an elderly woman with short grey hair</code></td></tr>
<tr><td>lighting / weather</td><td><code>night, lit by pink and blue neon signs</code></td></tr>
</table>
<p>Rules that matter (TSC, newer strategy): <b>H3 has no idea what is pinned and what is noised</b>, it reads one
description of one video. Describe the finished split screen as plain footage: never write "kept footage", "generated",
"preserved", "in sync with", "seamlessly". The pinned clip gets <b>no tag and no retention line</b>: name its region by where
it is plus what is seen there, in one clause ("the top quarter shows a man dancing in a small bedroom";
<code>{panel_region}</code> / <code>{video_region}</code> give the region names, <code>{layout}</code> the whole sentence).
Write the other region as its own scene in a real place. <b>Spell out the motion</b> beat by beat, with times for the big
moments on the video's clock. The soundscape is the sounds of the new scene. Cuts: "[Shot 2] At 00:03.708, both halves cut
together to…". No negations ("no hat" draws a hat). Connect a VLM to <code>vlm</code> to write the prompt or fill a
template's <code>{...}</code> fields. The full example is in the prompt tooltip.</p>
<h3>Layout</h3>
<p><b>position</b>: the side of the panel.</p>
${row(canvas({ position: "left", label: "left" }), canvas({ position: "right", label: "right" }),
canvas({ position: "top", w: 96, h: 120, label: "top" }), canvas({ position: "bottom", w: 96, h: 120, label: "bottom" }))}
<p><b>size</b>: the panel against the video (1.0 = two equal halves). Smaller is cheaper; keep faces readable.</p>
${row(canvas({ size: 1.0, label: "size 1.0" }), canvas({ size: 0.5, label: "size 0.5" }), canvas({ size: 0.33, label: "size 0.33" }))}
<p><b>fit</b> (advanced): <b>contain</b> keeps the whole clip smaller with grey around (nothing lost, default);
<b>cover</b> fills the panel and crops; stretch distorts. <b>gap</b> (advanced): a grey separator, in 32 px steps.</p>
${row(canvas({ size: 0.5, fit: "contain", label: "contain" }), canvas({ size: 0.5, fit: "cover", label: "cover" }), canvas({ size: 1.0, gap: 2, label: "gap 2" }))}
<h3>How strongly the panel is pinned</h3>
<p><b>panel_noise</b>: <b>0</b> copies the panel exactly (swaps, light changes). <b>0.1–0.2</b> gives the model room for big
changes (a style, a creature, a very different body).</p>
<p><b>hold</b> (advanced): pin the panel for the whole clip, or only at the start (the rest of the strip is free and cropped).</p>
${row(hold("all frames"), hold("first latent frame"))}
<h3>RoPE (advanced)</h3>
<p><b>rope_mode</b>: <b>canvas</b> = panel and video share one wide grid (the original duet). <b>shifted</b> = the video keeps
exactly the positions of a render without the panel, and the panel sits past its edge; <b>rope_gap</b> moves it further
out. Same quality in tests; keep the gap small at low resolution (a gap close to the video's width makes the model draw
its own split screen).</p>
${row(rope("canvas", 0), rope("shifted", 0), rope("shifted", 3))}
<h3>Defaults that work</h3>
<p>euler · beta · 20 steps · CFG 1 · size 1.0 · contain · panel_noise 0 · ${shot ? "mode duet" : "no guide"}.
${shot ? "Planner → this node → BFS Shot Join: every shot is pinned and generated in turn; the join cuts the panel off." :
"Outputs: <code>images</code> + <code>audio</code> (silent when there is no audio VAE) for Create Video; <code>canvas</code> to check the sync; <code>prompt</code> = what was sent."}</p>`;
}
function styles() {
if (document.getElementById("bfs-duet-help-css")) return;
const st = document.createElement("style"); st.id = "bfs-duet-help-css";
st.textContent = `
.bdh-back{position:fixed;inset:0;background:rgba(0,0,0,.7);z-index:10000;display:flex;align-items:center;justify-content:center}
.bdh{background:${C.bg};color:${C.ink};border:1px solid #3a3a45;border-radius:12px;width:min(860px,94vw);max-height:92vh;overflow:auto;
padding:18px 22px;font:13px/1.55 system-ui,sans-serif}
.bdh h2{margin:0 0 8px;font-size:18px}.bdh h3{margin:16px 0 6px;font-size:14px;color:#ffd27a}
.bdh code{background:#2a2a33;padding:1px 4px;border-radius:4px}
.bdh table{border-collapse:collapse;margin:6px 0}.bdh td,.bdh th{border:1px solid #33333d;padding:3px 8px;text-align:left}
.bdh-row{display:flex;gap:14px;flex-wrap:wrap;align-items:flex-end;margin:6px 0}
.bdh-cell{background:#141418;border-radius:8px;padding:6px}
.bdh-close{float:right;cursor:pointer;border:1px solid #444;background:#26262e;color:#ddd;border-radius:6px;padding:2px 10px}`;
document.head.appendChild(st);
}
function open(type) {
styles();
const back = document.createElement("div"); back.className = "bdh-back";
const guide = new URL("./docs/BFSH3Duet.md", import.meta.url).href;
back.innerHTML = `<div class="bdh"><button class="bdh-close">close</button>${html(type)}
<p style="margin-top:14px"><a href="#" class="bdh-guide" style="color:#8fb3ff">Full guide (credits, all settings, a complete example prompt)</a></p></div>`;
back.addEventListener("pointerdown", e => { if (e.target === back) back.remove(); });
back.querySelector(".bdh-close").addEventListener("click", () => back.remove());
back.querySelector(".bdh-guide").addEventListener("click", e => { e.preventDefault(); back.remove(); showGuide(guide); });
document.body.appendChild(back);
}
app.registerExtension({
name: "BFSNodes.DuetHelp",
async nodeCreated(node) {
if (!NODES.includes(node.comfyClass)) return;
const w = node.addWidget("button", "❓ How to use", null, () => open(node.comfyClass));
w.serialize = false;
},
});
+470
View File
@@ -0,0 +1,470 @@
/**
* BFS LoRA Surgery: a panel for reading a LoRA's layers and blocks and editing them live.
*
* The node keeps its state in a hidden `rules` STRING widget (a JSON list), so a workflow
* saves and reloads exactly what you set up. Everything visual lives in a Vue app mounted
* into a DOM widget.
*/
import { app } from "../../scripts/app.js";
import { api } from "../../scripts/api.js";
import { createApp, ref, computed, watch, h } from "./vendor/vue.esm-browser.prod.mjs";
// Starting points, not answers: which range works is per-LoRA, so ablate before trusting one.
const PRESETS = [
{ label: "MLP 50%", rules: [{ enabled: true, match: { type: "*mlp*" }, scale: 0.5 }] },
{ label: "MLP 35%", rules: [{ enabled: true, match: { type: "*mlp*" }, scale: 0.35 }] },
{ label: "Attention only", rules: [{ enabled: true, match: { type: "*mlp*" }, scale: 0 }] },
{ label: "MLP-in off, late half", rules: [{ enabled: true, match: { type: "*gate_up*" }, blocksHalf: true, scale: 0 }] },
];
function styles() {
if (document.getElementById("bfs-lora-surgery-css")) return;
const el = document.createElement("style");
el.id = "bfs-lora-surgery-css";
el.textContent = `
.bfsls{font:12px/1.45 var(--font-family,system-ui,sans-serif);color:var(--descrip-text,#c9c9cf);
background:var(--comfy-input-bg,#1b1b1f);border-radius:8px;padding:8px;height:100%;overflow:auto;box-sizing:border-box}
.bfsls h4{margin:0 0 6px;font-size:12px;letter-spacing:.02em;color:var(--input-text,#e6e6ea)}
.bfsls .muted{color:#8b8b93}
.bfsls .row{display:flex;gap:6px;align-items:center;flex-wrap:wrap}
.bfsls .fam{border:1px solid #33333c;border-radius:6px;margin-bottom:6px;overflow:hidden}
.bfsls .fam>summary{cursor:pointer;padding:6px 8px;background:#232329;list-style:none;display:flex;
justify-content:space-between;align-items:center;user-select:none}
.bfsls .fam>summary::-webkit-details-marker{display:none}
.bfsls .type{display:grid;grid-template-columns:1fr 78px 54px 30px;gap:6px;align-items:center;
padding:4px 8px;border-top:1px solid #2a2a31}
.bfsls .type:hover{background:#26262c}
.bfsls .name{font-family:ui-monospace,monospace;font-size:11px;color:#dcdce2;overflow:hidden;text-overflow:ellipsis}
.bfsls .bar{height:3px;border-radius:2px;background:#3a86ff;margin-top:3px;opacity:.85}
.bfsls input[type=range]{width:100%;accent-color:#3a86ff}
.bfsls input[type=text],.bfsls input[type=number]{background:#131317;border:1px solid #34343d;color:#e6e6ea;
border-radius:4px;padding:3px 6px;font-size:11px;width:100%;box-sizing:border-box;font-family:ui-monospace,monospace}
.bfsls button{background:#2b2b33;border:1px solid #3a3a45;color:#dcdce2;border-radius:5px;padding:3px 8px;
font-size:11px;cursor:pointer}
.bfsls button:hover{background:#35353f}
.bfsls button.pri{background:#2b5fd9;border-color:#3a6ee8;color:#fff}
.bfsls .rule{display:grid;grid-template-columns:18px 1fr 64px 18px 18px 20px;gap:5px;align-items:center;
padding:4px 6px;border:1px solid #33333c;border-radius:5px;margin-bottom:4px;background:#1f1f25}
.bfsls .rx{font-family:ui-monospace,monospace;font-size:10px;color:#9fd3a0;word-break:break-all}
.bfsls .blocks{display:flex;flex-wrap:wrap;gap:2px;margin:4px 0}
.bfsls .blk{width:19px;height:19px;user-select:none;touch-action:none;border-radius:3px;background:#2a2a32;border:1px solid #3a3a45;
font-size:9px;display:flex;align-items:center;justify-content:center;cursor:pointer;color:#a8a8b2}
.bfsls .blk.sel{background:#2b5fd9;border-color:#4a7ef0;color:#fff}
.bfsls .err{color:#ff8a8a}
.bfsls .mod{display:grid;grid-template-columns:1fr 62px 34px 26px;gap:6px;align-items:center;
padding:3px 6px;border-bottom:1px solid #26262c;font-size:11px}
.bfsls .mod:hover{background:#26262c}
.bfsls .modlist{max-height:260px;overflow:auto;border:1px solid #33333c;border-radius:6px}
.bfsls .tab{padding:3px 10px;border-radius:5px 5px 0 0;border:1px solid #33333c;border-bottom:none;
background:#1f1f25;cursor:pointer;font-size:11px}
.bfsls .tab.on{background:#2b2b33;color:#fff}
.bfsls .sec{border-top:1px solid #2c2c34;margin-top:8px;padding-top:6px}
.bfsls details.help{border:1px solid #33333c;border-radius:6px;background:#1c1c22;margin-bottom:8px}
.bfsls details.help>summary{cursor:pointer;padding:5px 8px;color:#9fb8ff;list-style:none;user-select:none}
.bfsls details.help>summary::-webkit-details-marker{display:none}
.bfsls .helpbody{padding:2px 10px 8px;color:#b4b4bd;font-size:11px;line-height:1.55}
.bfsls .helpbody code{background:#262630;padding:1px 4px;border-radius:3px;font-size:10px}
.bfsls .helpbody b{color:#dcdce2}
.bfsls .helpbody ol{margin:4px 0 6px 16px;padding:0}
.bfsls .helpbody li{margin:2px 0}
`;
document.head.appendChild(el);
}
function Panel(props, expose) {
const struct = ref(null);
const error = ref("");
const loading = ref(false);
const rules = ref([]);
const selFamily = ref("");
const selType = ref("*");
const selBlocks = ref(new Set());
const scale = ref(0);
const dragging = ref(false);
const view = ref("grouped"); // "grouped" | "all"
const filter = ref("");
const saveName = ref("");
const saving = ref(false);
const saved = ref("");
let dragAdds = true;
// Stored shape is {rules, ui}. A bare array is still accepted, both because older
// workflows carry one and because the Python side takes either.
function readRules() {
let parsed;
try { parsed = JSON.parse(props.getRules() || "[]"); } catch { parsed = []; }
if (Array.isArray(parsed)) { rules.value = parsed; return; }
rules.value = Array.isArray(parsed?.rules) ? parsed.rules : [];
const ui = parsed?.ui;
if (ui) {
if (ui.family) selFamily.value = ui.family;
if (typeof ui.type === "string") selType.value = ui.type;
if (typeof ui.scale === "number") scale.value = ui.scale;
if (typeof ui.blocks === "string" && ui.blocks) setBlocksFromText(ui.blocks);
}
}
function writeRules() {
props.setRules(JSON.stringify({
rules: rules.value,
ui: { family: selFamily.value, type: selType.value, blocks: blocksSpec(), scale: Number(scale.value) },
}));
}
async function load() {
const name = props.getLora();
if (!name) return;
loading.value = true; error.value = "";
try {
const r = await api.fetchApi(`/bfs/lora/structure?name=${encodeURIComponent(name)}`);
const data = await r.json();
if (data.error) { error.value = data.error; struct.value = null; }
else {
struct.value = data;
const known = (data.families || []).some(f => f.name === selFamily.value);
if (!known) selFamily.value = data.families?.[0]?.name || "";
}
} catch (e) { error.value = String(e); }
loading.value = false;
}
const family = computed(() => (struct.value?.families || []).find(f => f.name === selFamily.value)
|| struct.value?.families?.[0]);
const maxNorm = computed(() => Math.max(0.0001, ...((family.value?.types || []).map(t => t.norm || 0))));
function blocksSpec() {
const b = [...selBlocks.value].sort((x, y) => x - y);
if (!b.length) return "";
const out = []; let s = b[0], p = b[0];
for (let i = 1; i <= b.length; i++) {
if (b[i] === p + 1) { p = b[i]; continue; }
out.push(s === p ? `${s}` : `${s}-${p}`);
s = p = b[i];
}
return out.join(",");
}
// Drag across the ruler to pick a range. Click-to-toggle still works, since a click is
// just a drag of one cell. Shift is deliberately not used: on the ComfyUI canvas it starts
// a text selection instead.
function paint(i) {
if (dragAdds) selBlocks.value.add(i); else selBlocks.value.delete(i);
selBlocks.value = new Set(selBlocks.value);
}
function dragStart(i, ev) {
ev.preventDefault(); ev.stopPropagation();
dragging.value = true;
dragAdds = !selBlocks.value.has(i);
paint(i);
const stop = () => { dragging.value = false; window.removeEventListener("pointerup", stop); };
window.addEventListener("pointerup", stop);
}
function dragOver(i, ev) {
if (!dragging.value) return;
ev.preventDefault();
paint(i);
}
function setBlocksFromText(text) {
const next = new Set();
for (const part of String(text).split(",")) {
const t = part.trim();
if (!t) continue;
if (t.includes("-")) {
const [a, b] = t.split("-").map(Number);
if (Number.isFinite(a) && Number.isFinite(b)) for (let k = Math.min(a, b); k <= Math.max(a, b); k++) next.add(k);
} else if (Number.isFinite(Number(t))) next.add(Number(t));
}
selBlocks.value = next;
}
function selectAll() { selBlocks.value = new Set(family.value?.blocks || []); }
function selectHalf(second) {
const b = family.value?.blocks || [];
const mid = Math.floor(b.length / 2);
selBlocks.value = new Set(second ? b.slice(mid) : b.slice(0, mid));
}
function invert() {
const all = family.value?.blocks || [];
selBlocks.value = new Set(all.filter(i => !selBlocks.value.has(i)));
}
function addRule(asRegex) {
const fam = family.value?.name || "";
const spec = blocksSpec();
const match = asRegex
? { regex: buildRegex(fam, selType.value, spec) }
: { family: fam, type: selType.value, blocks: spec };
rules.value.push({ enabled: true, match, scale: Number(scale.value) });
writeRules();
}
function buildRegex(fam, type, spec) {
const esc = s => s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
let blk = "\\d+";
if (spec) {
const idx = [];
for (const part of spec.split(",")) {
if (part.includes("-")) { const [a, b] = part.split("-").map(Number); for (let k = a; k <= b; k++) idx.push(k); }
else if (part.trim() !== "") idx.push(Number(part));
}
if (idx.length) blk = "(" + [...new Set(idx)].sort((a, b) => a - b).join("|") + ")";
}
let mod = ".*";
if (type && type !== "*") mod = type.endsWith("*") ? esc(type.slice(0, -1)) + ".*" : esc(type);
return `^.*${fam ? esc(fam) : "[\\w.]+"}\\.${blk}\\.${mod}$`;
}
readRules();
load();
watch(() => props.getLora(), load);
// ComfyUI restores widget values after the node is created, so the panel has to be told
// to re-read once that happened. Without this, opening a saved workflow shows an empty
// panel and the first edit would overwrite the stored recipe.
if (expose) {
expose.reload = () => { readRules(); load(); };
}
const help = () => h("details", { class: "help" }, [
h("summary", {}, "What is this? (read once)"),
h("div", { class: "helpbody" }, [
h("p", {}, [
"A defect a LoRA learned (soft skin, identity drifting when the subject is far away, ",
"a pose it will not copy) usually lives in ", h("b", {}, "one module family and a range of blocks"),
", not in the whole adapter. This node lets you scale or drop that group and generate again, ",
"so you can find where it lives without retraining. Nothing is written to disk.",
]),
h("p", {}, [h("b", {}, "How to use it")]),
h("ol", {}, [
h("li", {}, ["Pick a LoRA. Layers below shows what that file contains, grouped by type, or switch to ", h("code", {}, "every layer"), " to see and filter each module individually."]),
h("li", {}, ["The bar next to each type is ", h("code", {}, "||dW||"),
", how large that group's update is. Big bars are where training invested."]),
h("li", {}, ["Click ", h("code", {}, "->"), " on a type to select it, then drag across the block ruler to pick a range."]),
h("li", {}, ["Set a scale and press ", h("code", {}, "add rule"), ". Scale 0 drops the group; 1 leaves it as trained."]),
h("li", {}, "Queue the prompt and compare against the unmodified LoRA at the same seed."),
]),
h("p", {}, [h("b", {}, "Names depend on the base model. "),
"This panel lists whatever your LoRA actually contains, so what you see here is not a fixed ",
"vocabulary. Qwen-Image-2.1 shows ", h("code", {}, "transformer_blocks"), " with ",
h("code", {}, "img_mlp.gate_up"), " and ", h("code", {}, "img_mlp.out"),
". Krea 2 shows ", h("code", {}, "blocks"), " with ", h("code", {}, "mlp.gate"), "/",
h("code", {}, "mlp.up"), "/", h("code", {}, "mlp.down"), ", plus separate ",
h("code", {}, "txtfusion"), " families. MiniMax-H3 shows ", h("code", {}, "mlp.fc1"), "/",
h("code", {}, "mlp.fc2"), " and a ", h("code", {}, "token_refiner"),
" family. Block counts differ too: 32, 28 and 50 in those three. Read the list below ",
"rather than assuming any name."]),
h("p", {}, [h("b", {}, "Where to look first. "),
"Across the models tested so far, the group that carries blur and lost detail is the ",
h("b", {}, "MLP input projection"), ", whatever your model calls it (",
h("code", {}, "gate_up"), ", ", h("code", {}, "mlp.gate"), "/", h("code", {}, "mlp.up"), ", ",
h("code", {}, "fc1"), ", ", h("code", {}, "w1"), "/", h("code", {}, "w3"),
"). The MLP output projection (", h("code", {}, "out"), ", ", h("code", {}, "down"), ", ",
h("code", {}, "fc2"),
") rarely matters, and attention is usually innocent. That is a starting point, not a rule: ",
"which group and which block range are right differs per LoRA, so ablate and measure."]),
h("p", {}, [h("b", {}, "Keeping the result. "),
"Once a recipe is settled, press ", h("code", {}, "save as file"),
" to write it into ", h("code", {}, "models/loras/surgery"),
" with the recipe in its name and metadata. In a workflow you can instead wire this node's ",
h("code", {}, "rules"), " output into ", h("code", {}, "BFS LoRA Surgery (save)"),
", so the file is produced as part of the run."]),
h("p", {}, [h("b", {}, "Order matters. "),
"Later rules override earlier ones where they overlap, so put boosts first and drops last, ",
"or a boost will undo a drop. Use the arrows to reorder."]),
h("p", {}, [h("b", {}, "One warning. "),
"A variant can win every metric and still have destroyed what you trained. Dropping the whole MLP ",
"path gave the sharpest skin in one real case and made the LoRA stop copying expression from the ",
"source image. Always check a hard case (a strong expression, an unusual angle) with your eyes."]),
h("p", {}, [h("b", {}, "Pruning is not the same as training without those layers. "),
"Excluding them during training may simply fail to converge: the layer is where the defect lodges, ",
"not where it comes from, which is usually the dataset."]),
]),
]);
return () => h("div", { class: "bfsls" }, [
h("div", { class: "row", style: "justify-content:space-between;margin-bottom:6px" }, [
h("h4", {}, struct.value ? `${struct.value.file}, ${struct.value.total_modules} modules`
: (loading.value ? "reading..." : "no LoRA loaded")),
h("button", { onClick: load }, "reload"),
]),
error.value ? h("div", { class: "err" }, error.value) : null,
help(),
h("div", { class: "sec" }, [h("h4", {}, "Selection")]),
h("div", { class: "row" }, [
h("span", { class: "muted" }, "type"),
h("input", { type: "text", value: selType.value, style: "flex:1",
onInput: e => { selType.value = e.target.value; } }),
]),
h("div", { class: "blocks", onPointerleave: () => { dragging.value = false; } },
(family.value?.blocks || []).map(i =>
h("div", { class: "blk" + (selBlocks.value.has(i) ? " sel" : ""),
title: "click, or drag across to select a range",
onPointerdown: e => dragStart(i, e),
onPointerenter: e => dragOver(i, e) }, String(i)))),
h("div", { class: "row", style: "margin-bottom:4px" }, [
h("span", { class: "muted" }, "blocks"),
h("input", { type: "text", value: blocksSpec(), placeholder: "all blocks", style: "flex:1",
title: "type a selection, e.g. 8-15 or 0,4,7 or 8-15,24-31",
onChange: e => setBlocksFromText(e.target.value) }),
]),
h("div", { class: "row", style: "margin-bottom:4px" }, [
h("button", { onClick: selectAll }, "all"),
h("button", { onClick: () => selectHalf(false) }, "first half"),
h("button", { onClick: () => selectHalf(true) }, "last half"),
h("button", { onClick: invert }, "invert"),
h("button", { onClick: () => { selBlocks.value = new Set(); } }, "none"),
]),
h("div", { class: "row" }, [
h("span", { class: "muted" }, "scale"),
h("input", { type: "range", min: 0, max: 2, step: 0.05, value: scale.value, style: "flex:1",
onInput: e => { scale.value = e.target.value; } }),
h("span", { style: "width:34px;text-align:right" }, Number(scale.value).toFixed(2)),
]),
h("div", { class: "row", style: "margin:6px 0" }, [
h("button", { class: "pri", onClick: () => addRule(false) }, "add rule"),
h("button", { onClick: () => addRule(true), title: "same selection, written as a regex" }, "add as regex"),
h("span", { class: "muted" }, blocksSpec() ? `blocks ${blocksSpec()}` : "all blocks"),
]),
h("div", { class: "row", style: "margin-bottom:6px" }, PRESETS.map(p =>
h("button", { onClick: () => { rules.value = JSON.parse(JSON.stringify(p.rules)); writeRules(); } }, p.label))),
h("div", { class: "sec" }, [h("h4", {}, `Rules (${rules.value.length}), later rules override earlier ones`)]),
rules.value.length
? h("div", { class: "row", style: "margin-bottom:5px" }, [
h("input", { type: "text", value: saveName.value, placeholder: "file name (blank = from the rules)",
style: "flex:1", onInput: e => { saveName.value = e.target.value; } }),
h("button", { disabled: saving.value, title: "write this as a .safetensors in models/loras/surgery",
onClick: async () => {
saving.value = true; saved.value = "";
try {
const r = await api.fetchApi("/bfs/lora/save", {
method: "POST", headers: { "Content-Type": "application/json" },
body: JSON.stringify({ name: props.getLora(), rules: rules.value,
filename: saveName.value }),
});
const d = await r.json();
saved.value = d.error ? `error: ${d.error}` : d.message;
} catch (e) { saved.value = String(e); }
saving.value = false;
} }, saving.value ? "saving..." : "save as file"),
])
: null,
saved.value ? h("div", { class: saved.value.startsWith("error") ? "err" : "muted",
style: "margin-bottom:5px" }, saved.value) : null,
rules.value.length > 1
? h("div", { class: "muted", style: "margin-bottom:4px" },
"Order matters: put boosts first and drops last, or a boost will undo a drop.")
: null,
...rules.value.map((r, i) => h("div", { class: "rule" }, [
h("input", { type: "checkbox", checked: r.enabled !== false,
onChange: e => { r.enabled = e.target.checked; writeRules(); } }),
r.match.regex
? h("div", { class: "rx", title: r.match.regex }, r.match.regex)
: h("div", { class: "name" }, `${r.match.type || "*"} @ ${r.match.blocks || "all"}`),
h("input", { type: "number", step: 0.05, value: r.scale,
onChange: e => { r.scale = Number(e.target.value); writeRules(); } }),
h("button", { title: "move up, later rules win", disabled: i === 0,
onClick: () => { const a = rules.value; [a[i - 1], a[i]] = [a[i], a[i - 1]]; writeRules(); } }, "↑"),
h("button", { title: "move down, later rules win", disabled: i === rules.value.length - 1,
onClick: () => { const a = rules.value; [a[i + 1], a[i]] = [a[i], a[i + 1]]; writeRules(); } }, "↓"),
h("button", { onClick: () => { rules.value.splice(i, 1); writeRules(); } }, "×"),
])),
rules.value.length
? h("div", { class: "row", style: "margin-top:4px" },
[h("button", { onClick: () => { rules.value = []; writeRules(); } }, "clear all")])
: h("div", { class: "muted" }, "No rules, the LoRA is applied unchanged."),
h("div", { class: "sec" }, [
h("div", { class: "row", style: "justify-content:space-between" }, [
h("h4", {}, `Layers (${struct.value?.total_modules ?? 0} modules)`),
h("div", { class: "row", style: "gap:2px" }, [
h("span", { class: "tab" + (view.value === "grouped" ? " on" : ""),
onClick: () => { view.value = "grouped"; } }, "grouped"),
h("span", { class: "tab" + (view.value === "all" ? " on" : ""),
onClick: () => { view.value = "all"; } }, "every layer"),
]),
]),
]),
view.value === "all"
? h("div", {}, [
h("input", { type: "text", placeholder: "filter, e.g. gate_up or blocks.1 or attn",
value: filter.value, style: "margin-bottom:4px",
onInput: e => { filter.value = e.target.value; } }),
(() => {
const all = struct.value?.modules || [];
const q = filter.value.trim().toLowerCase();
const hits = q ? all.filter(m => m.path.toLowerCase().includes(q)) : all;
const shown = hits.slice(0, 400);
return h("div", {}, [
h("div", { class: "muted", style: "margin-bottom:3px" },
`${hits.length} of ${all.length} modules${hits.length > shown.length ? `, showing first ${shown.length}` : ""}`),
h("div", { class: "modlist" }, shown.map(m => h("div", { class: "mod" }, [
h("div", { class: "name", title: m.path }, m.path),
h("span", { class: "muted" }, m.norm != null ? m.norm.toFixed(3) : ""),
h("span", { class: "muted" }, `r${m.rank ?? "?"}`),
h("button", { title: "add a rule for exactly this module",
onClick: () => {
const esc = m.path.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
rules.value.push({ enabled: true, match: { regex: `^${esc}$` }, scale: Number(scale.value) });
writeRules();
} }, "+"),
]))),
]);
})(),
])
: h("div", {}, (struct.value?.families || []).map(fam =>
h("details", { class: "fam", open: fam.name === selFamily.value }, [
h("summary", { onClick: () => { selFamily.value = fam.name; } }, [
h("span", { class: "name" }, fam.name || "(root)"),
h("span", { class: "muted" }, `${fam.blocks.length} blocks · ${fam.types.length} types`),
]),
...fam.types.map(t => h("div", { class: "type" }, [
h("div", {}, [
h("div", { class: "name", title: t.name }, t.name || "(none)"),
h("div", { class: "bar", style: `width:${Math.round(100 * (t.norm || 0) / maxNorm.value)}%` }),
]),
h("span", { class: "muted" }, t.norm != null ? `||dW|| ${t.norm.toFixed(2)}` : ""),
h("span", { class: "muted" }, `r${t.rank ?? "?"}`),
h("button", { title: "select this type", onClick: () => { selFamily.value = fam.name; selType.value = t.name; } }, "->"),
])),
]))),
]);
}
app.registerExtension({
name: "BFSNodes.LoraSurgery",
async nodeCreated(node) {
if (node.comfyClass !== "BFSLoraSurgery") return;
styles();
const rulesWidget = node.widgets?.find(w => w.name === "rules");
const loraWidget = node.widgets?.find(w => w.name === "lora_name");
if (rulesWidget) { rulesWidget.type = "hidden"; rulesWidget.computeSize = () => [0, -4]; }
const host = document.createElement("div");
host.style.cssText = "width:100%;height:100%;min-height:340px";
node.addDOMWidget("bfs_surgery_panel", "div", host, { serialize: false, hideOnZoom: false });
const panel = {};
createApp({
render: Panel({
getLora: () => loraWidget?.value,
getRules: () => rulesWidget?.value ?? "[]",
setRules: v => { if (rulesWidget) rulesWidget.value = v; node.graph?.setDirtyCanvas(true); },
}, panel),
}).mount(host);
// The widget holds the whole configuration, so it must survive save/load and travel
// with a shared workflow. Keep it serializable even though it is not drawn.
if (rulesWidget) {
rulesWidget.options = rulesWidget.options || {};
rulesWidget.options.serialize = true;
rulesWidget.serializeValue = () => rulesWidget.value;
}
const prevConfigure = node.onConfigure;
node.onConfigure = function (info) {
const r = prevConfigure?.apply(this, arguments);
// values land during configure; re-read on the next tick so the panel shows them
setTimeout(() => panel.reload?.(), 0);
return r;
};
node.size = [Math.max(node.size[0], 430), Math.max(node.size[1], 520)];
},
});
+85
View File
@@ -0,0 +1,85 @@
// Shows a node guide (Markdown in web/js/docs) as a formatted window inside ComfyUI.
// A small renderer for what the guides use: headings, paragraphs, lists, tables, code, bold / italic, links.
const esc = s => s.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
function inline(s) {
s = esc(s);
s = s.replace(/`([^`]+)`/g, "<code>$1</code>");
s = s.replace(/\*\*([^*]+)\*\*/g, "<b>$1</b>");
s = s.replace(/(^|[^*])\*([^*\s][^*]*)\*/g, "$1<i>$2</i>");
s = s.replace(/\[([^\]]+)\]\(([^)]+)\)/g, (m, t, u) => `<a href="${u}" target="_blank" rel="noopener">${t}</a>`);
return s;
}
export function renderMarkdown(md) {
const lines = md.replace(/\r/g, "").split("\n");
const out = [];
let i = 0;
while (i < lines.length) {
const l = lines[i];
if (/^```/.test(l)) { // code block
const buf = []; i++;
while (i < lines.length && !/^```/.test(lines[i])) buf.push(esc(lines[i++]));
i++; out.push(`<pre>${buf.join("\n")}</pre>`); continue;
}
const h = l.match(/^(#{1,4})\s+(.*)/);
if (h) { out.push(`<h${h[1].length}>${inline(h[2])}</h${h[1].length}>`); i++; continue; }
if (/^\s*\|/.test(l)) { // table
const rows = [];
while (i < lines.length && /^\s*\|/.test(lines[i])) rows.push(lines[i++]);
const cells = r => r.trim().replace(/^\||\|$/g, "").split(/(?<!\\)\|/).map(c => inline(c.trim().replace(/\\\|/g, "|")));
const body = rows.filter(r => !/^\s*\|[\s:|-]+\|\s*$/.test(r));
out.push("<table>" + body.map((r, k) => `<tr>${cells(r).map(c => k ? `<td>${c}</td>` : `<th>${c}</th>`).join("")}</tr>`).join("") + "</table>");
continue;
}
if (/^\s*([-*]|\d+\.)\s+/.test(l)) { // list (one level, continuation lines joined)
const ordered = /^\s*\d+\./.test(l); const items = [];
while (i < lines.length && (/^\s*([-*]|\d+\.)\s+/.test(lines[i]) || (/^\s{2,}\S/.test(lines[i]) && items.length))) {
const m = lines[i].match(/^\s*(?:[-*]|\d+\.)\s+(.*)/);
if (m) items.push(m[1]); else items[items.length - 1] += " " + lines[i].trim();
i++;
}
out.push(`<${ordered ? "ol" : "ul"}>${items.map(x => `<li>${inline(x)}</li>`).join("")}</${ordered ? "ol" : "ul"}>`);
continue;
}
if (!l.trim()) { i++; continue; }
const para = []; // paragraph
while (i < lines.length && lines[i].trim() && !/^(#{1,4}\s|```|\s*\||\s*([-*]|\d+\.)\s)/.test(lines[i])) para.push(lines[i++]);
out.push(`<p>${inline(para.join(" "))}</p>`);
}
return out.join("\n");
}
function styles() {
if (document.getElementById("bfs-md-css")) return;
const st = document.createElement("style"); st.id = "bfs-md-css";
st.textContent = `
.bmd-back{position:fixed;inset:0;background:rgba(0,0,0,.7);z-index:10000;display:flex;align-items:center;justify-content:center}
.bmd{background:#1b1b21;color:#e8e8ee;border:1px solid #3a3a45;border-radius:12px;width:min(900px,94vw);max-height:92vh;overflow:auto;
padding:18px 24px;font:13px/1.6 system-ui,sans-serif}
.bmd h1{font-size:20px;margin:4px 0 10px}.bmd h2{font-size:16px;margin:18px 0 6px;color:#ffd27a}.bmd h3{font-size:14px;margin:14px 0 4px}
.bmd code{background:#2a2a33;padding:1px 4px;border-radius:4px}
.bmd pre{background:#121217;border:1px solid #2c2c35;border-radius:8px;padding:10px;white-space:pre-wrap;font-size:12px}
.bmd table{border-collapse:collapse;margin:8px 0;width:100%}.bmd td,.bmd th{border:1px solid #33333d;padding:4px 8px;text-align:left;vertical-align:top}
.bmd th{background:#24242c}.bmd a{color:#8fb3ff}
.bmd-close{float:right;cursor:pointer;border:1px solid #444;background:#26262e;color:#ddd;border-radius:6px;padding:2px 10px}`;
document.head.appendChild(st);
}
export async function showGuide(url) {
styles();
const back = document.createElement("div"); back.className = "bmd-back";
back.innerHTML = `<div class="bmd"><button class="bmd-close">close</button><div class="bmd-body">Loading…</div></div>`;
back.addEventListener("pointerdown", e => { if (e.target === back) back.remove(); });
back.querySelector(".bmd-close").addEventListener("click", () => back.remove());
back.addEventListener("keydown", e => e.stopPropagation());
document.body.appendChild(back);
try {
const r = await fetch(url);
if (!r.ok) throw new Error(`HTTP ${r.status}`);
back.querySelector(".bmd-body").innerHTML = renderMarkdown(await r.text());
} catch (e) {
back.querySelector(".bmd-body").textContent = `Could not load the guide (${e.message}).`;
}
}
+706
View File
@@ -0,0 +1,706 @@
/**
* BFS Shot Planner: split a long video into model-sized shots on a timeline, give each shot its own
* reference image and prompt, and run the rest of the graph once per shot.
*
* All state lives in the hidden `plan` STRING widget (JSON) so a workflow saves, reloads and shares
* exactly what you set up. The panel is a Vue app mounted into a DOM widget; the server does the video
* work (probe, thumbnails, PySceneDetect cuts, split) through /bfs/shotloop/* routes.
*
* This file holds the state and the actions; the views (one per tab) live in ./shotloop/.
*/
import { app } from "../../scripts/app.js";
import { api } from "../../scripts/api.js";
import { showGuide } from "./bfs_md_view.js";
import { createApp, ref, reactive, computed, watch, nextTick, onMounted, onBeforeUnmount } from "./vendor/vue.esm-browser.prod.mjs";
import { h, t, uiLanguage, setUiLanguage, DEFAULTS, MASK_DEFAULTS, VLM_DEFAULTS, snapUp, snapDown, segKey, hasMask, pill } from "./shotloop/common.js";
import { styles } from "./shotloop/styles.js";
import { timelineCard } from "./shotloop/view_timeline.js";
import { videoTab } from "./shotloop/view_video.js";
import { shotsTab } from "./shotloop/view_shots.js";
import { peopleTab } from "./shotloop/view_people.js";
import { promptsTab } from "./shotloop/view_prompts.js";
import { runTab } from "./shotloop/view_run.js";
import { busyBar, pointsModal, tourModal } from "./shotloop/view_modal.js";
const viewUrl = name => {
if (!name) return "";
const i = name.lastIndexOf("/");
const sub = i >= 0 ? name.slice(0, i) : "", file = i >= 0 ? name.slice(i + 1) : name;
return api.apiURL(`/view?filename=${encodeURIComponent(file)}&subfolder=${encodeURIComponent(sub)}&type=input`);
};
const store = { // per-browser UI conveniences (never the plan itself)
get: (k, d) => { try { const v = localStorage.getItem(k); return v == null ? d : JSON.parse(v); } catch { return d; } },
set: (k, v) => { try { localStorage.setItem(k, JSON.stringify(v)); } catch { /* private mode */ } },
};
const DEFAULT_W = 1180, MIN_W = 1040, MIN_H = 1000; // node size: wide enough for the four-column settings
const TABS = [["video", "🎞 Video"], ["shots", "🎬 Shots"], ["people", "👥 People & masks"], ["prompts", "📝 Prompts & refs"], ["run", "▶ Run"]];
function Panel(io) {
const plan = reactive({ ...DEFAULTS });
const files = reactive({ videos: [], images: [] });
const an = ref(null); // analysis from the server
const cuts = ref([]);
const detectorUsed = ref("");
const size = reactive({ w: 0, h: 0 });
const busy = ref(""); const error = ref("");
// loading bar: what the server reports (stage, done/total) and how long it has been busy
const status = reactive({ stage: "", done: 0, total: 0 });
const busySince = ref(0); const now = ref(Date.now());
const upPct = ref(-1);
watch(busy, v => { if (v) { busySince.value = Date.now(); Object.assign(status, { stage: "", done: 0, total: 0 }); } else { busySince.value = 0; upPct.value = -1; } });
const sel = ref(0); const zoom = ref(1);
const rootEl = ref(null);
// keep the selected shot's card in view in the sideways strip
watch(sel, () => nextTick(() => rootEl.value?.querySelector(".shots .sc.sel")?.scrollIntoView({ block: "nearest", inline: "nearest" })));
const hover = ref(null); // {frame, x}
const prog = reactive({ done: 0, count: 0 });
const tlEl = ref(null);
const stats = ref([]); // per-shot content stats from /bfs/shotloop/filters
const people = ref([]); // cast from /bfs/shotloop/cast: [{id, thumb, share, first, last}]
const maskPrev = ref({}); // "start-end" -> {frames, box, coverage, empty} from /bfs/shotloop/mask
const showMasks = ref(true); // overlay the mask preview on the shot cards
const vlmSug = ref({}); // "start-end" -> {segment, shot, people, recommend, reason} from the VLM
const modal = reactive({ open: false, idx: -1, key: 0, points: [], text: "", prev: null, busy: "" });
const segPeople = ref({}); // "start-end" -> {people, main} from /bfs/shotloop/plan
const targetCrop = ref({}); // "start-end" -> data URL of the subject the description was made from
const tab = ref(store.get("bfs.shotloop.tab", "shots"));
watch(tab, v => store.set("bfs.shotloop.tab", v));
const tour = ref(0); // opened only by the user's Tour button; never shown automatically
// what "Copy to other shots" copies (remembered per browser)
const copyOpts = reactive(store.get("bfs.shotloop.copy", { ref: true, prompt: false, mask_text: false, points: false, crop: false,
target: false, chain: false, color_guide: false, scope: "all" }));
watch(copyOpts, v => store.set("bfs.shotloop.copy", { ...v }), { deep: true });
const load = () => {
let p = {};
try { p = JSON.parse(io.getPlan() || "{}"); } catch { p = {}; }
Object.assign(plan, DEFAULTS, p);
plan.filters = { ...DEFAULTS.filters, ...(p.filters || {}) };
plan.cast = { ...(p.cast || {}) };
plan.mask_cfg = { ...MASK_DEFAULTS, ...(p.mask_cfg || {}) };
plan.vlm_cfg = { ...VLM_DEFAULTS, ...(p.vlm_cfg || {}) };
plan.ref_details = { ...(p.ref_details || {}) };
if (!Array.isArray(plan.bounds)) plan.bounds = [];
if (!Array.isArray(plan.segs)) plan.segs = [];
};
const save = () => io.setPlan(JSON.stringify(plan));
const n = computed(() => (an.value ? an.value.n : 0)); // the whole video on the timeline
// the part the plan works on [lo, hi): start / end (0 = the end), then "max total seconds" counted from the start
const range = computed(() => {
const N = n.value; if (!N) return { lo: 0, hi: 0 };
// frames (older plans saved seconds)
const a = plan.start_f != null ? +plan.start_f : Math.round((+plan.start_s || 0) * plan.fps);
const b = plan.end_f != null ? +plan.end_f : (+plan.end_s > 0 ? Math.round(+plan.end_s * plan.fps) : 0);
const lo = Math.max(0, Math.min(N - 1, a));
let hi = b > 0 ? Math.max(lo + 1, Math.min(N, b)) : N;
if (+plan.max_total_s > 0) hi = Math.min(hi, lo + Math.max(1, Math.round(+plan.max_total_s * plan.fps)));
return { lo, hi };
});
const maxLen = computed(() => snapDown(Math.round(plan.max_s * plan.fps), plan.grid));
const segs = computed(() => {
const N = n.value; if (!N) return [];
const { lo, hi } = range.value;
const b = [lo, ...plan.bounds.filter(x => x > lo && x < hi), hi].sort((a, c) => a - c);
const cs = new Set(cuts.value);
let out = [];
for (let i = 0; i < b.length - 1; i++) {
if (b[i + 1] <= b[i]) continue;
const m = plan.segs[out.length] || {};
out.push({ start: b[i], end: b[i + 1], len: b[i + 1] - b[i], gen: snapUp(b[i + 1] - b[i], plan.grid),
cut: cs.has(b[i]), enabled: m.enabled !== false, ref: m.ref || "", ref2: m.ref2 || "", prompt: m.prompt || "",
force: m.force || "auto", chain: m.chain || "history", chainFrame: m.chain_frame || "first", chainHistory: m.chain_history || 17,
crop: !!m.crop, inpaint: !!m.inpaint, paste: !!m.paste, pasteText: m.paste_text || "", strength: m.strength ?? 1, mask: m.mask || {}, target: m.target || "",
extMask: !!plan.mask_video || !!(plan.track?.points?.length), trackMask: !!(plan.track?.points?.length) });
}
if (plan.max_parts > 0) out = out.slice(0, plan.max_parts);
return out;
});
const whoIn = s => segPeople.value[segKey(s)] || null;
const castOf = id => plan.cast[String(id)] || {};
const linked = id => !!castOf(id).ref && !castOf(id).ignore;
// reference a shot gets from its main person when it has none of its own
const castRef = (s, field) => {
if (!plan.cast_assign) return "";
const who = whoIn(s); if (!who || who.main < 0 || castOf(who.main).ignore) return "";
return castOf(who.main)[field] || "";
};
const personOf = id => people.value.find(p => p.id === id) || null;
const statFor = s => stats.value.find(x => x.start === s.start && x.end === s.end) || null;
const filtersOn = computed(() => { const f = plan.filters; return !!(f.person || f.face || f.skip_dark || f.skip_static || f.max_persons > 0 || f.min_frames > 0); });
const skipWhy = s => {
if (!s.enabled) return "disabled";
if (s.force === "run") return "";
if (s.force === "skip") return "skipped by hand";
const why = filtersOn.value ? (statFor(s)?.skip_reason || "") : "";
if (why) return why;
const who = whoIn(s);
if (plan.cast_only && who && !who.people.some(p => linked(p))) return "no linked person";
return "";
};
const active = computed(() => segs.value.filter(s => !skipWhy(s)));
const pxPerFrame = computed(() => {
const w = (tlEl.value?.clientWidth || 600) - 2;
return Math.max(0.2, (w / Math.max(1, n.value)) * zoom.value);
});
const tlWidth = computed(() => Math.max(1, Math.round(n.value * pxPerFrame.value)));
// how a shot is generated: crop (the planner crops, the join pastes back) x inpaint (only the mask is regenerated,
// read by BFS Shot H3 Conditioning with inpaint = per shot)
// [key, name, crop, inpaint, paste]; paste = the whole frame is generated and BFS Shot Join pastes only the person back
const MODES = [["frame", "Full frame", false, false, false], ["paste", "Frame + paste", false, false, true], ["mask", "Mask only", false, true, false],
["crop", "Crop", true, false, false], ["cropmask", "Crop + mask", true, true, false]];
const modeOf = s => (MODES.find(m => m[2] === !!s.crop && m[3] === !!s.inpaint && m[4] === (!!s.paste && !s.crop && !s.inpaint))
|| MODES[0])[0];
const modeName = s => MODES.find(m => m[0] === modeOf(s))[1];
const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return plan.segs[i]; };
// the reference a shot really uses: its own, its main person's, or the global one
const refOf = (s, k) => s[k] || castRef(s, k) || plan["global_" + k] || "";
const promptOf = s => s.prompt || plan.global_prompt || "";
const detailsOf = s => plan.ref_details?.[`${refOf(s, "ref")}|${refOf(s, "ref2")}`] || "";
// things worth fixing before a run, per shot (only shots that run)
const checks = computed(() => {
const out = [];
segs.value.forEach((s, i) => {
if (skipWhy(s)) return;
const add = (lvl, text) => out.push({ lvl, shot: i, text });
const p = promptOf(s);
if (s.len > maxLen.value) add(plan.mode === "single" ? "note" : "warn", plan.mode === "single"
? `one shot of ${s.len} frames (${(s.len / plan.fps).toFixed(1)} s), longer than Max seconds / shot: the model may not hold it`
: `longer than ${maxLen.value} frames: split it`);
if ((s.crop || s.inpaint || s.paste) && !hasMask(s)) add("warn", `${modeName(s)} needs a mask (text, points or a mask video; or the planner's mask input)`);
if (hasMask(s) && maskPrev.value[segKey(s)]?.empty) add("warn", "the mask preview found nothing");
if (/\{target\}/.test(p) && !s.target) add("warn", "the prompt uses {target} but the shot has no target description");
if (/\{details\}/.test(p) && !detailsOf(s) && !plan.vlm_cfg?.enabled) add("info", "the prompt uses {details} but its references have no description yet");
if (/\{details\}/.test(p) && (plan.vlm_cfg?.describe_preset || "short") !== "short")
add("warn", "{details} sits inside a sentence of the prompt: use the 'short' describe preset (Prompts & refs), the other presets write a paragraph");
if (!p) add("info", "no prompt (fine when the node's prompt input is connected)");
if (!refOf(s, "ref")) add("info", "no reference (fine when the node's ref_image input is connected)");
});
return out;
});
const checksFor = i => checks.value.filter(c => c.shot === i);
async function refreshFiles() {
try { const r = await api.fetchApi("/bfs/shotloop/files"); const j = await r.json(); files.videos = j.videos || []; } catch (e) { /* ignore */ }
}
async function analyze() {
an.value = null; error.value = "";
if (!plan.video) return;
busy.value = "Analysing video…";
try {
const r = await api.fetchApi(`/bfs/shotloop/analyze?video=${encodeURIComponent(plan.video)}&fps=${plan.fps}`);
const j = await r.json();
if (j.error) throw new Error(j.error);
an.value = j;
await autoSplit(plan.bounds.length === 0);
people.value = []; segPeople.value = {};
if (Object.keys(plan.cast).length || plan.cast_split || plan.cast_only) await findPeople(false); // restore the cast of a saved plan
} catch (e) { error.value = String(e.message || e); }
busy.value = "";
}
async function autoSplit(apply = true) {
if (!plan.video) return;
busy.value = "Detecting cuts…"; error.value = "";
try {
const body = { plan: { ...plan, bounds: apply ? [] : plan.bounds } };
const r = await api.fetchApi("/bfs/shotloop/plan", { method: "POST", body: JSON.stringify(body) });
const j = await r.json();
if (j.error) throw new Error(j.error);
cuts.value = j.cuts || []; detectorUsed.value = j.detector || ""; size.w = j.width; size.h = j.height;
segPeople.value = Object.fromEntries(j.segs.filter(s => s.main !== undefined && (s.people.length || people.value.length))
.map(s => [`${s.start}-${s.end}`, { people: s.people, main: s.main }]));
if (apply) {
plan.bounds = j.segs.slice(1).map(s => s.start);
const keep = plan.segs; plan.segs = j.segs.map((_, i) => ({ enabled: true, ref: keep[i]?.ref || "", ref2: keep[i]?.ref2 || "", prompt: keep[i]?.prompt || "",
chain: keep[i]?.chain || "history", chain_frame: keep[i]?.chain_frame || "first", chain_history: keep[i]?.chain_history || 17,
crop: !!keep[i]?.crop, inpaint: !!keep[i]?.inpaint, paste: !!keep[i]?.paste, paste_text: keep[i]?.paste_text || "", strength: keep[i]?.strength ?? 1, mask: keep[i]?.mask || {}, target: keep[i]?.target || "",
color_guide: !!keep[i]?.color_guide, color_guide_color: keep[i]?.color_guide_color || "", color_guide_opacity: keep[i]?.color_guide_opacity ?? 0.35,
color_guide_person: keep[i]?.color_guide_person ?? null, color_guide_text2: keep[i]?.color_guide_text2 || "",
color_guide_color2: keep[i]?.color_guide_color2 || "", color_guide_person2: keep[i]?.color_guide_person2 ?? null }));
sel.value = 0; save();
}
} catch (e) { error.value = String(e.message || e); }
busy.value = "";
}
async function analyzeContent() {
if (!plan.video) return;
busy.value = "Detecting people and faces…"; error.value = "";
try {
const r = await api.fetchApi("/bfs/shotloop/filters", { method: "POST", body: JSON.stringify({ plan: { ...plan } }) });
const j = await r.json(); if (j.error) throw new Error(j.error);
stats.value = j.segs.map(x => ({ ...x.stats, start: x.start, end: x.end, skip_reason: x.skip_reason }));
} catch (e) { error.value = String(e.message || e); }
busy.value = "";
}
async function findPeople(resplit = true) {
if (!plan.video) return;
busy.value = "Finding people by face…"; error.value = "";
try {
const r = await api.fetchApi("/bfs/shotloop/cast", { method: "POST", body: JSON.stringify({ plan: { ...plan } }) });
const j = await r.json(); if (j.error) throw new Error(j.error);
people.value = j.people || [];
} catch (e) { error.value = String(e.message || e); busy.value = ""; return; }
busy.value = "";
await autoSplit(resplit && !!plan.cast_split);
}
const setCast = (id, k, v) => {
plan.cast = { ...plan.cast, [String(id)]: { ...castOf(id), [k]: v } };
if (k === "ref" || k === "ref2") remember(v);
save();
};
async function describeTarget(i) {
const s = segs.value[i]; if (!s) return;
busy.value = "Selecting the person (SAM 3) and describing them (VLM)…"; busySince.value = Date.now(); error.value = "";
try {
const r = await api.fetchApi("/bfs/shotloop/target", { method: "POST", body: JSON.stringify({ plan: { ...plan }, index: i }) });
const j = await r.json(); if (j.error) throw new Error(j.error);
targetCrop.value = { ...targetCrop.value, [segKey(s)]: j.crop };
if (j.desc) setMeta(i, "target", j.desc);
else error.value = "The SAM 3 target cutout was created (shown below), but no VLM is loaded to describe it. Connect Qwen3-VL to the Shot Planner's vlm input and queue the workflow once; then click Describe target again. You can also type the target description yourself.";
} catch (e) { error.value = String(e.message || e); }
busy.value = "";
}
async function previewMask(i, spec, intoModal = false) {
const s = segs.value[i]; if (!s) return;
const label = "Segmenting with SAM 3… (the first time downloads/loads the model)";
if (intoModal) { modal.busy = label; busySince.value = Date.now(); Object.assign(status, { stage: "", done: 0, total: 0 }); } else busy.value = label;
error.value = "";
try {
const r = await api.fetchApi("/bfs/shotloop/mask", { method: "POST", body: JSON.stringify({ plan: { ...plan }, index: i, mask: spec || null }) });
const j = await r.json(); if (j.error) throw new Error(j.error);
if (intoModal) modal.prev = j; else maskPrev.value = { ...maskPrev.value, [segKey(s)]: j };
} catch (e) { error.value = String(e.message || e); }
if (intoModal) modal.busy = ""; else busy.value = "";
}
const setMask = (i, k, v) => {
const m = meta(i); if ((m.mask || {})[k] === v) return;
m.mask = { ...(m.mask || {}), [k]: v }; save();
const s = segs.value[i]; // the old preview no longer matches
if (s && maskPrev.value[segKey(s)]) { const mp = { ...maskPrev.value }; delete mp[segKey(s)]; maskPrev.value = mp; }
};
const setMaskCfg = (k, v) => { plan.mask_cfg = { ...plan.mask_cfg, [k]: v }; save(); maskPrev.value = {}; };
const openPoints = i => {
const s = segs.value[i]; if (!s) return;
Object.assign(modal, { open: true, idx: i, key: s.mask.key ?? Math.floor(s.len / 2), points: [...(s.mask.points || [])],
text: s.mask.text || "", prev: null, busy: "" });
};
const savePoints = () => {
const m = meta(modal.idx); m.mask = { ...(m.mask || {}), points: modal.points, key: modal.key, text: modal.text }; save();
if (modal.prev) maskPrev.value = { ...maskPrev.value, [segKey(segs.value[modal.idx])]: modal.prev };
modal.open = false;
};
const targets = (scope, from) => segs.value.map((_, i) => i).filter(i => i !== from && (scope === "all" || i > from));
// the same selection on other shots (a subject that stays in place): each shot gets the points on its frame at the
// same relative position as the key frame here
const pointsTo = (points, key, text, fromIdx, scope = "all") => {
const src = segs.value[fromIdx]; const rel = src ? key / Math.max(1, src.len - 1) : 0.5;
[fromIdx, ...targets(scope, fromIdx)].forEach(i => {
const x = segs.value[i], m = meta(i);
m.mask = { ...(m.mask || {}), points: points.map(p => ({ ...p })), key: Math.round(rel * Math.max(0, x.len - 1)), text: text ?? (m.mask || {}).text };
});
maskPrev.value = {}; save();
};
const savePointsAll = () => { pointsTo(modal.points, modal.key, modal.text, modal.idx); modal.open = false; };
// one selection tracked by SAM 3 through the whole video (or range): every shot without its own selection takes its
// part of that track, so the subject is found wherever it moves (copied points can fall on the background)
const saveTrack = () => {
const s0 = segs.value[modal.idx]; if (!s0 || !modal.points.length) return;
const own = segs.value.filter((x, i) => i !== modal.idx && (x.mask.text || (x.mask.points || []).length)).length;
if (own && !window.confirm(`${own} other shot(s) have their own mask text or points, which would win over the track. Clear them so every shot uses the track?`)) {
// keep them: those shots keep their own selection
} else {
segs.value.forEach((_, i) => { const m = meta(i); if (m.mask) { m.mask = { ...m.mask, points: [], text: "" }; } });
}
const m = meta(modal.idx); m.mask = { ...(m.mask || {}), points: [], text: "" };
plan.track = { points: modal.points.map(p => ({ ...p })), frame: s0.start + modal.key, text: modal.text || "",
threshold: plan.mask_cfg?.threshold ?? 0.5 };
maskPrev.value = {}; save(); modal.open = false;
};
const clearTrack = () => { plan.track = null; maskPrev.value = {}; save(); };
const clearMask = i => {
const m = meta(i); m.mask = {}; m.crop = false; m.inpaint = false; m.paste = false;
const k = segs.value[i] ? segKey(segs.value[i]) : null;
if (k) { const mp = { ...maskPrev.value }; delete mp[k]; maskPrev.value = mp; }
save();
};
// copy the chosen settings of shot `from` to the other shots (all of them, or the ones after it)
const COPY_FIELDS = { ref: "references", prompt: "prompt", mask_text: "mask text", points: "mask points", mask_video: "mask video", crop: "mode",
target: "{target}", chain: "continuity", color_guide: "Color Guide" };
const copyFrom = (from, fields = copyOpts, scope = copyOpts.scope) => {
const src = segs.value[from]; if (!src) return 0;
const to = targets(scope, from);
to.forEach(i => {
const m = meta(i), x = segs.value[i];
if (fields.ref) { m.ref = src.ref; m.ref2 = src.ref2; }
if (fields.prompt) m.prompt = src.prompt;
if (fields.mask_text) m.mask = { ...(m.mask || {}), text: src.mask.text || "" };
if (fields.mask_video) m.mask = { ...(m.mask || {}), video: src.mask.video || "" };
if (fields.crop && (hasMask(x) || src.mask.text || fields.mask_text)) { m.crop = src.crop; m.inpaint = src.inpaint; m.paste = src.paste; m.paste_text = src.pasteText; m.strength = src.strength; }
if (fields.target) m.target = src.target;
if (fields.color_guide) {
m.color_guide = src.color_guide; m.color_guide_color = src.color_guide_color;
m.color_guide_opacity = src.color_guide_opacity; m.color_guide_person = src.color_guide_person;
m.color_guide_text2 = src.color_guide_text2; m.color_guide_color2 = src.color_guide_color2;
m.color_guide_person2 = src.color_guide_person2;
}
if (fields.chain && i > 0) { m.chain = src.chain; m.chain_frame = src.chainFrame; m.chain_history = src.chainHistory; }
});
if (fields.points && (src.mask.points || []).length) pointsTo(src.mask.points, src.mask.key ?? Math.floor(src.len / 2), undefined, from, scope);
if (fields.mask_text || fields.points || fields.mask_video) maskPrev.value = {};
save();
return to.length;
};
const mergeSug = list => { const o = { ...vlmSug.value }; for (const x of list || []) o[`${x.start}-${x.end}`] = x; vlmSug.value = o; };
async function analyseVLM() {
busy.value = "Asking the VLM about every shot…"; error.value = "";
try {
const r = await api.fetchApi("/bfs/shotloop/vlm", { method: "POST", body: JSON.stringify({ plan: { ...plan } }) });
const j = await r.json(); if (j.error) throw new Error(j.error);
mergeSug(j.segs);
} catch (e) { error.value = String(e.message || e); }
busy.value = "";
}
// reference sets used by the plan (global, every shot's, every cast person's): one description each
const refSets = computed(() => {
const out = new Map();
const add = (a, b, where) => { if (!a && !b) return; const k = `${a || ""}|${b || ""}`; if (!out.has(k)) out.set(k, { a, b, where: [] }); out.get(k).where.push(where); };
add(plan.global_ref, plan.global_ref2, "global");
segs.value.forEach((x, i) => add(x.ref || plan.global_ref, x.ref2 || plan.global_ref2, `#${i + 1}`));
Object.entries(plan.cast || {}).forEach(([id, c]) => { if (!c.ignore) add(c.ref, c.ref2, `person ${id}`); });
return [...out.entries()].map(([key, v]) => ({ key, ...v }));
});
async function describeRefs() {
busy.value = "Describing the references with the VLM…"; error.value = "";
try {
const sets = refSets.value.map(x => [x.a || "", x.b || ""]);
const r = await api.fetchApi("/bfs/shotloop/describe", { method: "POST", body: JSON.stringify({ plan: { ...plan }, sets }) });
const j = await r.json(); if (j.error) throw new Error(j.error);
plan.ref_details = { ...plan.ref_details, ...j.texts }; save();
} catch (e) { error.value = String(e.message || e); }
busy.value = "";
}
const setDetail = (k, v) => { plan.ref_details = { ...plan.ref_details, [k]: v }; save(); };
const setVlmCfg = (k, v) => { plan.vlm_cfg = { ...plan.vlm_cfg, [k]: v }; save(); };
const applySug = (i, what) => {
const s = segs.value[i], g = s && vlmSug.value[segKey(s)]; if (!g) return;
if (what === "segment" && g.segment) setMask(i, "text", g.segment);
if (what === "skip") setMeta(i, "force", g.recommend === "skip" ? "skip" : "auto");
};
const applyAllSug = () => {
segs.value.forEach((s, i) => {
const g = vlmSug.value[segKey(s)]; if (!g) return;
const m = meta(i);
if (g.segment && !(m.mask?.text || (m.mask?.points || []).length)) m.mask = { ...(m.mask || {}), text: g.segment };
if (g.recommend === "skip" && (m.force || "auto") === "auto") m.force = "skip";
});
save();
};
const setCastOpt = (k, v) => { plan[k] = v; save(); if (k === "cast_split") autoSplit(true); };
const setFilter = (k, v) => { plan.filters = { ...plan.filters, [k]: v }; save(); if (stats.value.length) analyzeContent(); };
const setPlan = (k, v, after) => { plan[k] = v; save(); after && after(); };
// the range changed: split again inside it (the per-shot settings stay with the shot at the same position)
// start / end in timeline frames (end 0 = the end of the video); the old seconds fields are cleared
const setRange = (k, v) => { plan[k] = Math.max(0, Math.round(+v || 0)); plan.start_s = 0; plan.end_s = 0; save(); autoSplit(true); };
const setGlobalMaskVideo = v => { plan.mask_video = v; save(); maskPrev.value = {}; };
// XHR instead of fetch: big videos show how much has been sent
const postWithProgress = (url, body, onPct) => new Promise((resolve, reject) => {
const x = new XMLHttpRequest();
x.open("POST", api.apiURL(url));
if (api.user) x.setRequestHeader("Comfy-User", api.user);
x.upload.onprogress = e => { if (e.lengthComputable) onPct(e.loaded / e.total); };
x.onload = () => (x.status >= 200 && x.status < 300) ? resolve(JSON.parse(x.responseText || "{}")) : reject(new Error(`HTTP ${x.status}`));
x.onerror = () => reject(new Error("network error"));
x.send(body);
});
// big files go in 8 MB pieces: ComfyUI refuses any single request over --max-upload-size (100 MB by default)
const CHUNK = 8 * 1024 * 1024, CHUNKED_FROM = 32 * 1024 * 1024;
async function uploadChunked(file, onPct) {
const id = `${Date.now().toString(36)}_${Math.random().toString(36).slice(2, 12)}`;
for (let off = 0; off < file.size; off += CHUNK) {
const piece = file.slice(off, Math.min(file.size, off + CHUNK));
for (let tries = 0; ; tries++) { // a dropped piece is sent again (it is written at its offset)
try {
await postWithProgress(`/bfs/upload/chunk?id=${id}&offset=${off}`, piece,
f => onPct((off + f * piece.size) / file.size));
break;
} catch (e) { if (tries >= 3) throw e; await new Promise(r => setTimeout(r, 1000 * (tries + 1))); }
}
}
const r = await fetch(api.apiURL("/bfs/upload/finish"), { method: "POST", headers: { "Content-Type": "application/json" },
body: JSON.stringify({ id, name: file.name, size: file.size }) });
const j = await r.json();
if (!r.ok || j.error) throw new Error(j.error || `HTTP ${r.status}`);
return j.name;
}
async function upload(file, cb) {
const mb = (file.size / 1e6).toFixed(1);
busy.value = `Uploading ${file.name} (${mb} MB)…`; upPct.value = 0;
try {
let name;
if (file.size >= CHUNKED_FROM) name = await uploadChunked(file, f => { upPct.value = f; });
else {
const fd = new FormData(); fd.append("image", file); fd.append("type", "input"); fd.append("overwrite", "true");
const j = await postWithProgress("/upload/image", fd, f => { upPct.value = f; });
name = j.subfolder ? `${j.subfolder}/${j.name}` : j.name;
}
await refreshFiles();
cb(name);
} catch (e) { error.value = `Upload failed: ${e}`; }
busy.value = "";
}
const pickFile = (accept, cb) => {
const inp = document.createElement("input"); inp.type = "file"; inp.accept = accept;
inp.onchange = () => inp.files[0] && upload(inp.files[0], cb); inp.click();
};
async function progress(reset = false) {
if (!plan.video || plan.run !== "queue") return;
try {
const r = await api.fetchApi("/bfs/shotloop/progress", { method: "POST", body: JSON.stringify({ plan: { ...plan }, plan_raw: io.getPlan(), reset }) });
const j = await r.json(); if (!j.error) { prog.done = j.done; prog.count = j.count || active.value.length; }
} catch { /* ignore */ }
}
// ---- editing
const setVideo = v => {
plan.video = v; plan.bounds = []; plan.segs = [];
if (v) {
// A new source starts as one manageable clip. The user can widen the range or ask for an automatic split later;
// do not surprise them with a wall of detected shots on first load.
plan.mode = "single";
plan.start_f = 0;
plan.end_f = Math.round(15 * plan.fps);
plan.start_s = 0; plan.end_s = 0; plan.max_total_s = 0;
tab.value = "video";
}
save(); analyze();
};
const splitAt = f => {
const { lo, hi } = range.value; f = Math.round(f);
if (f <= lo || f >= hi || plan.bounds.includes(f)) return;
const i = segs.value.findIndex(s => f > s.start && f < s.end);
plan.bounds = [...plan.bounds, f].sort((a, c) => a - c);
plan.segs.splice(i + 1, 0, { ...(plan.segs[i] || {}) });
sel.value = i + 1; save();
};
const mergeNext = i => {
const s = segs.value[i]; if (!s || i >= segs.value.length - 1) return;
plan.bounds = plan.bounds.filter(b => b !== s.end); plan.segs.splice(i + 1, 1); save();
};
// Delete / Backspace on a selected shot: remove the cut at its start (merge into the previous shot;
// the first shot merges with the next one)
const removeCut = i => {
const S = segs.value, s = S[i]; if (!s || S.length < 2) return;
if (i === 0) { mergeNext(0); sel.value = 0; return; }
plan.bounds = plan.bounds.filter(b => b !== s.start); plan.segs.splice(i, 1);
sel.value = i - 1; save();
};
const select = i => { if (i >= 0 && i < segs.value.length) sel.value = i; };
const onKey = e => {
const t = e.target, tag = (t?.tagName || "").toLowerCase();
if (tag === "input" || tag === "textarea" || tag === "select" || t?.isContentEditable) return;
if (e.key === "ArrowLeft" || e.key === "ArrowRight") { // previous / next shot
e.preventDefault(); e.stopPropagation(); select(sel.value + (e.key === "ArrowLeft" ? -1 : 1)); return;
}
if (e.key !== "Delete" && e.key !== "Backspace") return;
e.preventDefault(); e.stopPropagation(); // keep ComfyUI from deleting the node
removeCut(sel.value);
};
const setMeta = (i, k, v) => { meta(i)[k] = v; save(); };
const setMode = (i, mode) => { const m = MODES.find(x => x[0] === mode); Object.assign(meta(i), { crop: m[2], inpaint: m[3], paste: m[4] }); save(); };
const drag = (k, ev) => {
ev.preventDefault(); ev.stopPropagation();
const box = tlEl.value.getBoundingClientRect();
const move = e => {
const x = e.clientX - box.left + tlEl.value.scrollLeft;
const f = Math.round(x / pxPerFrame.value);
const lo = (plan.bounds[k - 1] ?? range.value.lo) + 1, hi = (plan.bounds[k + 1] ?? range.value.hi) - 1;
plan.bounds[k] = Math.max(lo, Math.min(hi, f));
hover.value = { frame: plan.bounds[k], x };
};
const up = () => { window.removeEventListener("pointermove", move); window.removeEventListener("pointerup", up); save(); };
window.addEventListener("pointermove", move); window.addEventListener("pointerup", up);
};
const frameAt = e => { const box = tlEl.value.getBoundingClientRect(); const x = e.clientX - box.left + tlEl.value.scrollLeft; return { frame: Math.max(0, Math.min(n.value - 1, Math.round(x / pxPerFrame.value))), x }; };
const thumbFor = f => { const t = an.value?.thumbs; if (!t?.length) return ""; let b = t[0]; for (const x of t) { if (x.f <= f) b = x; else break; } return b.src; };
// ---- preview player
const vid = ref(null);
const play = reactive({ mode: "", idx: -1, frame: 0, loop: false });
const stopAt = () => { const s = segs.value[play.idx]; return s ? s.end / plan.fps : Infinity; };
const tick = () => {
const v = vid.value; if (!v) return;
play.frame = Math.floor(v.currentTime * plan.fps + 1e-3);
if (!play.mode) return;
if (play.mode === "all") {
const i = segs.value.findIndex(s => play.frame >= s.start && play.frame < s.end);
if (i >= 0 && skipWhy(segs.value[i])) { // jump over shots that will not run
const nxt = segs.value.slice(i + 1).find(s => !skipWhy(s));
if (nxt) v.currentTime = nxt.start / plan.fps + 1e-3; else { v.pause(); play.mode = ""; }
return;
}
if (i >= 0) play.idx = i;
if (play.frame >= n.value) { v.pause(); play.mode = ""; }
} else if (v.currentTime >= stopAt() - 1e-3) {
const s = segs.value[play.idx];
if (play.loop && s) v.currentTime = s.start / plan.fps + 1e-3; else { v.pause(); play.mode = ""; v.currentTime = stopAt() - 0.5 / plan.fps; }
}
};
let raf = 0;
const loop = () => { tick(); raf = requestAnimationFrame(loop); };
const playShot = i => {
const v = vid.value, s = segs.value[i]; if (!v || !s) return;
sel.value = i; play.idx = i; play.mode = "shot";
v.currentTime = s.start / plan.fps + 1e-3; v.play();
};
const playAll = () => {
const v = vid.value; if (!v) return;
play.mode = "all"; const first = segs.value.find(s => !skipWhy(s)) || segs.value[0];
play.idx = segs.value.indexOf(first); v.currentTime = (first?.start || 0) / plan.fps + 1e-3; v.play();
};
const stop = () => { vid.value?.pause(); play.mode = ""; };
const seek = f => { const v = vid.value; if (v) { v.currentTime = f / plan.fps + 1e-3; play.frame = f; } };
// ---- queue loop events
const onProg = e => { prog.done = e.detail.done; prog.count = e.detail.count; };
const onStatus = e => { if (busy.value || modal.busy) Object.assign(status, e.detail || {}); };
let clock = 0;
const onVlm = e => { if (e.detail?.video === plan.video) mergeSug(e.detail.segs); };
const onNext = e => { onProg(e); if (plan.run === "queue" && plan.auto_continue) setTimeout(() => app.queuePrompt(0, 1), 300); };
onMounted(() => {
load(); refreshFiles(); analyze(); progress(); raf = requestAnimationFrame(loop);
if (!plan.video) tab.value = "video";
api.addEventListener("bfs-shotloop-progress", onProg); api.addEventListener("bfs-shotloop-next", onNext);
api.addEventListener("bfs-shotloop-vlm", onVlm);
api.addEventListener("bfs-shotloop-status", onStatus);
clock = setInterval(() => { if (busy.value || modal.busy) now.value = Date.now(); }, 500);
});
onBeforeUnmount(() => { cancelAnimationFrame(raf); api.removeEventListener("bfs-shotloop-progress", onProg); api.removeEventListener("bfs-shotloop-next", onNext); api.removeEventListener("bfs-shotloop-vlm", onVlm); api.removeEventListener("bfs-shotloop-status", onStatus); clearInterval(clock); });
io.expose({ reload: () => { load(); analyze(); progress(); } });
// ---- references
// the last 10 references picked, remembered across workflows (per browser)
const RECENT_KEY = "bfs.shotloop.recentRefs";
const isColorValue = name => typeof name === "string" && /^#[0-9a-f]{6}$/i.test(name);
const savedRecent = store.get(RECENT_KEY, []);
const recent = ref((Array.isArray(savedRecent) ? savedRecent : []).filter(name => typeof name === "string" && !isColorValue(name)));
if (recent.value.length !== (Array.isArray(savedRecent) ? savedRecent.length : 0)) store.set(RECENT_KEY, recent.value);
const remember = name => {
if (!name || typeof name !== "string" || isColorValue(name)) return;
recent.value = [name, ...recent.value.filter(x => x !== name)].slice(0, 10);
store.set(RECENT_KEY, recent.value);
};
// target: shot index or "global"; field: ref | ref2 (global_ref | global_ref2 for "global")
const setRef = (target, field, name) => {
if (isColorValue(name)) return; // colors belong to Color Guide settings, never to image references
if (target === "global") plan[field] = name; else meta(target)[field] = name;
remember(name); save();
};
const uploadRef = (target, field) => pickFile("image/*", name => setRef(target, field, name));
const clearRecent = () => { recent.value = []; store.set(RECENT_KEY, []); };
// everything: the recent list AND every reference this plan uses (global, per shot, per person in the cast)
const clearAllRefs = () => {
if (!confirm("Remove every reference picture from this plan (global, every shot and every person) and forget the recent list?")) return;
clearRecent();
plan.global_ref = ""; plan.global_ref2 = "";
plan.segs.forEach(m => { if (m) { m.ref = ""; m.ref2 = ""; } });
Object.values(plan.cast || {}).forEach(cc => { if (cc) { cc.ref = ""; cc.ref2 = ""; } });
save();
};
const usedRefs = computed(() => {
const set = new Set([...recent.value, plan.global_ref, plan.global_ref2, ...plan.segs.flatMap(m => [m?.ref, m?.ref2])].filter(Boolean));
return [...set].slice(0, 10);
});
// everything the views need
const c = {
api, plan, files, an, cuts, detectorUsed, size, busy, error, status, busySince, now, upPct, sel, zoom, hover, prog, tlEl,
stats, people, maskPrev, showMasks, vlmSug, modal, segPeople, targetCrop, tab, copyOpts, COPY_FIELDS, vid, play, recent, clearRecent, clearAllRefs,
n, range, setRange, maxLen, segs, active, pxPerFrame, tlWidth, filtersOn, refSets, usedRefs, checks, tour,
MODES, modeOf, modeName, viewUrl, save, meta, setMeta, setMode, setPlan, whoIn, castOf, linked, castRef, personOf, statFor, skipWhy, refOf, promptOf, checksFor,
refreshFiles, analyze, autoSplit, analyzeContent, findPeople, setCast, describeTarget, previewMask, setMask, setMaskCfg,
openPoints, savePoints, savePointsAll, saveTrack, clearTrack, pointsTo, clearMask, copyFrom, analyseVLM, describeRefs, setDetail, setVlmCfg,
applySug, applyAllSug, setCastOpt, setGlobalMaskVideo, setFilter, pickFile, progress, setVideo, splitAt, mergeNext, removeCut, select, drag,
frameAt, thumbFor, playShot, playAll, stop, seek, setRef, uploadRef,
};
return () => {
const S = segs.value, N = n.value, fps = plan.fps;
const warns = checks.value.filter(x => x.lvl === "warn").length;
const guideUrl = new URL("./docs/BFSShotPlanner.md", import.meta.url).href;
const header = h("div", { class: "hdr" }, [
h("span", { class: "ttl" }, "🎬 Shot Planner"),
an.value ? pill(`${active.value.length}/${S.length} shots · ${((range.value.hi - range.value.lo) / fps).toFixed(1)}s`
+ (range.value.lo > 0 || range.value.hi < N ? " (range)" : ""), "ok") : null,
warns ? h("span", { class: "pill warn", style: "cursor:pointer", title: "open the checks (Run tab)", onClick: () => { tab.value = "run"; } }, `⚠ ${warns}`) : null,
h("span", { class: "grow" }),
h("label", { class: "langpick", title: t("Language") }, [
h("span", { class: "hint" }, "🌐"),
h("select", { value: uiLanguage.value, "aria-label": t("Language"), onChange: e => setUiLanguage(e.target.value) }, [
h("option", { value: "en" }, "English"), h("option", { value: "pt" }, "Português"),
h("option", { value: "es" }, "Español"), h("option", { value: "zh" }, "中文"),
]),
]),
pill(plan.run === "queue" ? "queue loop" : "auto loop"),
h("button", { title: t("Shot Planner tour"), onClick: () => { tour.value = 1; tab.value = "video"; } }, t("🧭 Tour")),
h("button", { title: "Open the full guide (every setting explained, with examples)", onClick: () => showGuide(guideUrl) }, "📖 Guide"),
]);
const counts = { shots: S.length || null, people: people.value.length || null };
const tabs = h("div", { class: "tabs" }, TABS.map(([k, label]) => h("button", {
class: ["tab", tab.value === k && "on"], disabled: k !== "video" && !an.value, onClick: () => { tab.value = k; } }, [
label, counts[k] ? h("span", { class: "n" }, counts[k]) : null, k === "run" && warns ? h("span", { class: "dot", title: `${warns} warning(s)` }) : null,
])));
const views = { video: videoTab, shots: shotsTab, people: peopleTab, prompts: promptsTab, run: runTab };
const cur = an.value || tab.value === "video" ? tab.value : "video";
return h("div", { class: "bsl root", ref: rootEl, tabindex: 0, onKeydown: onKey }, [
header, tabs,
busy.value ? busyBar(c, busy.value) : null,
error.value ? h("div", { class: "err" }, [h("span", { class: "grow" }, error.value), h("button", { class: "ghost", onClick: () => { error.value = ""; } }, "✕")]) : null,
an.value ? timelineCard(c) : null,
views[cur](c),
pointsModal(c),
tourModal({ tour, tab }),
]);
};
}
app.registerExtension({
name: "BFSNodes.ShotPlanner",
async nodeCreated(node) {
if (node.comfyClass !== "BFSShotPlanner") return;
styles();
const planWidget = node.widgets?.find(w => w.name === "plan");
if (planWidget) { // the JSON plan stays a widget (saved with the workflow) but is never shown, in both node renderers
planWidget.type = "hidden"; planWidget.hidden = true; planWidget.computeSize = () => [0, -4];
planWidget.options = { ...(planWidget.options || {}), hidden: true };
}
const host = document.createElement("div");
// the panel is absolutely positioned inside the host and scrolls by itself, so switching tabs never resizes the
// node (Vue nodes mode sizes widgets by their content); min-width keeps it from collapsing to its content width
host.style.cssText = "position:relative;width:100%;height:100%;min-width:900px;min-height:640px";
node.addDOMWidget("bfs_shot_planner", "div", host, { serialize: false, hideOnZoom: false });
const handle = {};
createApp({
setup: () => Panel({
getPlan: () => planWidget?.value ?? "{}",
setPlan: v => { if (planWidget) planWidget.value = v; node.graph?.setDirtyCanvas(true); },
expose: o => Object.assign(handle, o),
}),
}).mount(host);
if (planWidget) {
planWidget.options = planWidget.options || {};
planWidget.options.serialize = true;
planWidget.serializeValue = () => planWidget.value;
}
const prev = node.onConfigure;
node.onConfigure = function () {
const r = prev?.apply(this, arguments);
setTimeout(() => handle.reload?.(), 0);
if (this.size[0] < MIN_W) this.setSize([MIN_W, Math.max(this.size[1], MIN_H)]); // older workflows: too narrow
return r;
};
node.size = [Math.max(node.size[0], DEFAULT_W), Math.max(node.size[1], MIN_H)];
},
});
+182
View File
@@ -0,0 +1,182 @@
# BFS H3 Duet / Side Panel
Training-free split-screen ("duet") generation for MiniMax H3.
**Credits:** the initial idea for this H3 node came from TSC's latent-pin duet. BFS had already used the
same principle on LTX (a green side panel holding the reference) and implemented the virtual sidecar
approach (reference tokens placed beside the frame in RoPE). New here: the **shifted RoPE layout** (the
video keeps the RoPE positions of a render without the panel and the panel sits past its edge, with an
optional gap), dynamic references, task prompts and the shot-loop node.
A panel is pinned next to the video in the same canvas with a noise mask. H3 keeps the panel
exactly (it treats it like its own condition rows) and generates the video next to it, in sync.
The panel is cut off before decoding, so it never reaches the output.
## Nodes
| node | what it does |
|---|---|
| **BFS H3 Duet** | Everything in one node: native references + prompt, the pinned panel, an optional latent guide, sampling (euler / beta / CFG 1), decode and crop. Leave `panel` empty for a plain guided render. Outputs the generated video, its audio, the whole canvas (to check the sync) and the layout sentence for the prompt. |
| **BFS Shot H3 Conditioning** (duet option) | The shot loop's H3 conditioning can pin the shot's own clip as a duet panel too: *duet* = canvas or shifted RoPE (route the model through the node for the shift). BFS Shot Join cuts the panel off by itself, so the rest of the workflow (sampler, decode) stays as it is. Optional prompt writer: pick a *task* (character swap, style, setting, appearance, lighting / weather, custom) and an *instruction*; with a VLM on its *vlm* input it writes each shot's duet prompt from the shot and its references, otherwise a template for the task (*planner prompt*, the default, keeps the planner's prompt). The written text comes out on *prompt*. Optional *setting_ref* (TSC's tip): one more reference, the shot's middle frame with the person covered in TV static, so the model sees the place in full detail; it is the last `<Picture n>` (write `{setting}` for its tag, otherwise a sentence is added to subject_definitions). |
| **BFS Shot H3 Duet** | One shot of the shot loop (Planner -> this -> BFS Shot Join): *duet* pins the shot's own clip beside the video (no LoRA needed), *guide* puts the shot on the generated frames as a latent guide (for body-swap LoRAs), *duet + guide* does both. Write `{layout}` in the prompt to insert the split-screen sentence. |
| **BFS H3 Duet Conditioning (prompt writer)** | Conditioning only, for your own sampler: references + the pinned panel + the prompt. Write your prompt, or connect a VLM (`vlm`, a CLIPLoader with a Qwen3-VL text encoder) and it writes the prompt in the duet format from **task** + **instruction**, looking at the clip and the references (it fills a few fields; the node assembles the six sections). Outputs positive, latent (with the panel's mask), model (RoPE shift applied in shifted mode), panel_info and the prompt used. Chain: → BasicGuider / SamplerCustomAdvanced → **BFS H3 Side Panel Crop** (latent) → VAE Decode. |
| **BFS H3 Side Panel** | Only the canvas: takes the conditioning and AV latent from Reference to Video, adds the pinned strip (and the guide), and returns them for your own sampler. Guides already added with Add Guide for MiniMax H3 are moved onto the canvas. |
| **BFS H3 Side Panel Crop** | Removes the strip from the sampled latent (before VAE Decode) or from decoded images. |
## First time?
Click **❓ How to use** on the node: a window with drawings of every setting (panel side, size, contain/cover,
gap, hold, canvas vs shifted RoPE) and how to write the task, instruction and prompt. The node shows only the main
settings; *fit, gap, hold, rope_mode, rope_gap, ref_image_size, decode_canvas, guide_frame_idx* are under the
node's advanced settings, with defaults that work.
## What to pin
- **The source clip** (TSC's duet): the new character copies its motion, camera and cuts frame for
frame. Give the new character with `<Picture 1>` (and the place with `<Picture 2>`). The pinned clip
has no tag: the prompt names its region by where it is and what is seen there (`{layout}` / `{panel_region}`). See TSC's
prompting guide: never describe the source performer, restate the identity in every shot.
- **A reference image**: a static picture of the person next to the video.
## Tasks
Use it for any edit that keeps the source's motion, camera and timing. With an empty prompt box the node
writes the prompt from **task** + **instruction** (the `prompt` output shows what it sent):
| task | instruction example | panel_noise |
|---|---|---|
| character swap | (empty, or the new person's look) + the person's pictures as references | 0 |
| style | `a 1990s anime cel style` | 0.1-0.2 |
| setting | `a sunny beach at sunset, waves rolling behind her` | 0 |
| appearance | `an elderly woman with short grey hair` | 0.1 |
| lighting / weather | `night, lit by pink and blue neon signs` | 0 |
It does not change the motion or the camera: the panel makes the video copy them in sync.
The task prompt is a **draft**: it cannot see the video, so it is generic. In tests, a prompt written for the
clip (the example below) kept the room, framing and sync; the drafts got the change only partly (the style
barely changed, the setting and light lost the framing, the swap took the reference's backdrop). Start from
the `prompt` output and describe the clip.
References work like the official Reference to Video node: up to 9 images (`<Picture n>`), 3 videos
(`<Video n>`, each with its soundtrack) and 3 audios (`<Audio n>`), new slots appear as you connect them.
## Prompt
TSC's newer strategy (2026-10): **H3 has no idea what is pinned and what is noised.** It reads one description of
one video, so the prompt describes the finished split screen as plain footage. Never write "kept footage", "kept
exactly", "generated", "preserved", "in sync with", "seamlessly": they describe nothing on screen and crowd out the
action.
- **No tag and no retention line for the pinned clip.** It is not a reference input, only latents in the canvas. Name
its region by where it is plus what is seen there, in one clause: "the top quarter shows a man dancing in a small
bedroom". `{panel_region}` / `{video_region}` become the region names for the current side and size ("the top
quarter", "the bottom three quarters"), `{layout}` the whole split-screen sentence.
- **Write the other region as its own scene**, doing the same movements at the same moments, in a real place with
concrete details.
- **Spell out the motion** beat by beat, with times for the big moments (cuts, strikes, jumps, a line). The pinned
half alone does not carry the moves. Times are on the video's clock (the pinned clip starts at frame 0 here).
- **Soundscape** = the sounds of the new scene.
- Cuts: `[Shot 2] At MM:SS.mmm, both halves cut together to ...` with the new framing.
The VLM writer and the template filler follow these rules, and a scrub removes the old meta words from what they
return. Template (fill the `{...}`, or let the VLM fill them):
```
subject_definitions:
<Subject 1> is {name or plain description} whose appearance comes from <Picture 1>: {concrete look words}, wearing {outfit piece by piece}.
summary:
[reference generation] The target video is a split screen: {panel_region} shows {what is seen in the source, one clause}, and {video_region} shows <Subject 1> doing the same movements at the same moments, in {place}.
retention_analysis:
<Subject 1> (appears in [Shot 1]): fully_preserved - {key look words} are retained.
detailed_description:
The target video is in a {realistic / 3D CG / anime} style, as {the medium of the source clip}. A split screen divided by a thin straight {vertical / horizontal} line: {panel_region} shows {what is seen there, one clause}; {video_region} shows <Subject 1> performing the same movements at the same moments, in {place with concrete details}.
[Shot 1] {Camera distance, angle and movement as the source shows it}. <Subject 1>, the face from <Picture 1>, {what they do, one sentence per real action}. At {00:01.500} {big beat}.
overall_soundscape:
{The sounds of the new scene and when they happen.}
non_diegetic_music:
None.
```
Example (character swap, source clip pinned on the left half, one face picture and one full-body picture):
```
subject_definitions:
<Subject 1> is a young woman whose appearance comes from <Picture 1> and <Picture 2>: fair skin, a narrow oval face, grey-green eyes, light brown hair in a low ponytail, wearing a white long-sleeved top under a navy denim apron.
summary:
[reference generation] The target video is a split screen: the left half shows a man talking to a phone camera in a bedroom, and the right half shows <Subject 1> doing the same movements at the same moments, in a sunny kitchen with white tiles.
retention_analysis:
<Subject 1> (appears in [Shot 1], [Shot 2]): fully_preserved - her face, ponytail, white top and navy apron are retained.
detailed_description:
The target video is in a realistic style, as handheld vertical smartphone footage under soft daylight. A split screen divided by a thin straight vertical line: the left half shows a man talking to a phone camera in a bedroom; the right half shows <Subject 1> performing the same movements at the same moments, in a sunny kitchen with white tiles and a window behind her.
[Shot 1] A medium close-up at chest height, the phone steady at eye level. <Subject 1>, her face from <Picture 1> with grey-green eyes and soft pink lips, talks to the camera with small nods. At 00:01.800 she laughs and tilts her head back.
[Shot 2] At 00:03.708, both halves cut together to a closer framing. <Subject 1>, her face from <Picture 1>, the white sleeves and apron straps visible, raises one open hand beside her mouth and smiles.
overall_soundscape:
Her voice speaking to the camera, a kettle starting to hiss in the kitchen at 00:02.500.
non_diegetic_music:
None.
```
## Same-world swaps: static and setting picture (1.84.0)
TSC's two fixes for a swap in the same room with the same clothes, without a VLM:
- **`panel_static`** (`off` / `face` / `person`): SAM 3 tracks the face (or the whole person) of the pinned clip and
covers it in black-and-white TV static on every frame. The generated half can no longer copy the source face, so the
identity comes from the reference picture; motion, framing and camera still come from the panel. The static only
exists in the panel, which is cropped off, so it never shows in the output. `face` keeps the body (best when the
outfit should stay); `person` hides the whole performer (best when the build and outfit should change).
- **`setting`** + **BFS Setting Picture**: one frame of the clip with the person in static. Plugged into `setting`, it
becomes the last `<Picture n>` and the prompt gets *"<Picture n> shows the setting, the same place as the original performance
in full detail; the noise patch in it is where <Subject 1> stands."* (or write `{setting}` to place its tag yourself).
This stops the background and framing from being taken from the character picture.
## Prompt from a VLM (optional, 1.85.0)
Plug a Qwen3-VL CLIP (the same CLIPLoader as `clip` works) into `vlm`:
- **prompt empty**: the VLM writes it from the task + instruction (TSC's six sections; with a `setting` picture the
place becomes `<Subject 2> ... from <Picture n>`).
- **prompt with `{...}` fields** (TSC's template): the VLM fills only the fields, from the clip's frames, the
reference pictures and the setting picture; every other word stays. `{layout}`, `{setting}`, `{target}` and
`{details}` are filled by the node, not the VLM.
- **prompt without fields**: used as written (the VLM is not called).
Without `vlm`, a prompt that still has `{...}` fields logs a warning. The prompt actually used is the `prompt` output
and is printed under the comparison.
## Settings
| setting | effect |
|---|---|
| position / size | side of the strip and its size relative to the video (1.0 = two equal halves) |
| fit | contain (default: the whole clip, smaller, with grey around, nothing cropped), cover (fill and crop), stretch |
| gap | grey separator between panel and video, in 32 px patches |
| panel_noise | 0 pins the panel exactly; 0.05-0.15 loosens it when the result copies too much of it |
| hold | pin the panel for the whole clip, or only its first latent frame |
| guide | optional aligned latent guide in the video area (what the body-swap LoRAs use) |
| rope_mode | *canvas*: panel and video share one wide grid (TSC). *shifted*: the video keeps the RoPE positions of a render without the panel, and the panel sits past its edge |
| (prompt in shifted mode) | With *shifted*, `{layout}` becomes empty: do not describe a split screen there, or the model splits the video itself. *canvas* keeps the split-screen wording and follows the framing more closely (recommended). |
| rope_gap | *shifted* only: empty RoPE steps (2x2 patches) between video and panel; the panel moves away in position without any pixels in between. Keep it small against the video width: at 320 px wide (10 patches) a gap of 8 made the model draw its own split screen; 0 works everywhere |
## Notes from tests
- Source clip pinned, no LoRA, no guide (TSC's setup): the new person follows the clip's gestures,
timing and cuts, in the same room, and on-screen captions disappear.
- With a body-swap LoRA and a latent guide, the guide alone kept the room better than guide + panel:
the panel pulled the background toward the reference picture's backdrop. The shifted RoPE (gap 0 or 8)
did not change that, so it comes from the pinned panel itself, not from the canvas geometry.
- Duet with the shifted RoPE, gap 0 and gap 8: same quality and sync as the canvas layout. Use the panel without a
guide, or the guide without a panel, until a LoRA is trained for both.
+355
View File
@@ -0,0 +1,355 @@
# BFS Sampler (crop · mask · loop)
Runs a guide-driven LTX IC-LoRA and adds three things around it, each optional.
Connect only what you need — the node degrades to whatever is wired.
| what you connect | what you get |
|---|---|
| `guide_video` | the LoRA over the whole clip, one pass |
| `+ identity_image` | for a LoRA that takes a reference (head swap, identity transfer) |
| `+ subject_mask` | only the masked region is denoised |
| `+ crop_mode` | the edit runs inside a box around the subject, then is pasted back |
| `+ temporal_tile_size` | long clips sampled in chunks with overlap |
| `+ auto_config` | crop, mask and feather amounts measured off the subject itself |
`identity_image` is optional as of 1.38.0. A head-swap LoRA wants one; an
instruction-edit, restyle or sharpening LoRA works from the guide alone, and
with the slot empty it is simply not packed — the guide becomes the only
reference. Everything else in the node is indifferent to which kind it is.
The LoRA is never asked to understand a mask or a crop. It keeps doing its one
job: the guide rides on `source_id 1`, the identity on `source_id 2`. Everything
else here is plumbing around it.
## `auto_config`: let the mask set the amounts
Every amount in this node is in pixels — `mask_grow`, `mask_blur`,
`uncrop_feather`, and the latent dilation — and a pixel only means something
relative to how big the head is in the frame. The same `mask_grow: 8` is
generous slack on a head 90 px wide and invisible on one 900 px wide. That
mismatch is what leaves a visible seam when the crop goes back into the frame:
the amounts were tuned on one shot and the next shot has the subject at a
different distance.
Turn `auto_config` on and the node measures the subject in `subject_mask` — its
width, its height, how far it travels, how much its size changes, how fast it
moves between frames — and sets all of it from those numbers:
| knob | derived as | why |
|---|---|---|
| `crop_mode` | `off` when the head is ≥45% of the frame width; `zoomed` when its size changes >35%; `tracked` when it travels more than a quarter of the box; else `combined` | a crop buys resolution the head does not already have, and nothing else |
| `crop_scale` | 1.8, clamped so the box still fits in the frame | head plus neck and shoulders — a face-tight crop is a framing the LoRA never saw |
| `mask_grow` | 6% of the head's **width**, plus the capped headroom slack — **per side**: up ≫ sideways ≫ down | hair overflows upward and sideways; growing down only eats the neck and collar, which stay the guide's |
| `mask_blur` | 3% of the head's width | paste-back softness (the denoise mask is binarised anyway) |
| `uncrop_feather` | half the margin between mask and crop border, floored | **the seam fix**: a ramp wider than that margin fades the head itself |
| `latent_mask_dilate` | left at 0 | the pixel grow already carries the slack, and the pixel→latent reduction is a MAX, so any cell the mask touches is editable already — adding cells here stacks a whole 32 px on every side |
| `latent_mask_dilate_frames` | 1 when the head moves more than 15% of its width between frames | a fast head needs slack along time too |
### The mask is the old head, not the new one
This is the failure the mode is really for. `subject_mask` marks the head that
is being *replaced*. The reference head can be wider, or have more hair, or a
taller cut — and wherever it lands outside the mask, the swap is cut off at the
mask edge. Growing the mask by a fixed 6% does not fix that; it just moves the
clip.
Two things are done about it. The reference's **proportions** are measured —
`identity_image`'s aspect against the mask's own — and used to decide *where*
the slack goes: a relatively wider reference gets it at the sides, a taller one
above. The reference's **size** cannot be measured at all: a cropped head
carries no scale, and nothing in the graph says whether that crop is a 200 px
head or a 900 px one. That part is `identity_headroom`, default 1.15 — "the new
head may be up to 15% bigger than the old one". Raise it for big hair or a
visibly larger head, drop it to 1.0 when the two heads match.
Growth is never symmetric: down is held at the base 6% no matter what, because
the neck and collar below the jaw have to stay the guide's own pixels.
Every slack is a fraction of the head's **width** and capped there (25% sideways,
30% up). Width is the stable dimension of a head mask — height swings with how
much neck the mask happened to take — and scaling the upward slack by height,
as 1.37.0 did, produced holes far larger than the head, with the new head
generated floating inside the regenerated area. Fixed in 1.37.1.
It needs `subject_mask`; without one there is nothing to measure and the widget
values are kept. The widgets it overrides stay visible but are ignored, and the
`debug` output prints exactly what it chose:
```
auto: head 120x160px in 1920x1080 (6% of the width), seen in 73/73 frames;
reference aspect 1.00 vs mask 0.75 (tilt 1.33), headroom 1.15 -> crop tracked @ 1.8
(the subject travels 1296px, more than a quarter of the box); grow 19px sideways,
32px up, 7px down, blur 4px, feather 14px, latent dilate 1 cell(s), 0 frame(s)
```
Read that line before trusting it. It is a starting point measured from your own
shot, not a guarantee — a mask that leaks onto the shoulders makes the "head"
wider and every amount grows with it.
## The mask is native inpainting, not a LoRA feature
The mask becomes `latent["noise_mask"]`, which the guider consumes as
`denoise_mask` — ComfyUI's own inpainting path. Pixels outside the mask keep the
guide's own content, so **the original face stays visible to the model**. This is
deliberately different from a masked-hole recipe, where the region is painted out
before the model sees it.
When a mask is connected the node seeds the sampling latent with the
**VAE-encoded guide**, not with the empty latent you wired in — inpainting keeps
the initial latent outside the mask, and an empty one leaves grey where the video
should be. The connected latent still sets the size (and carries the audio
stream); its video content is replaced.
Masks are reduced to the latent grid with **max**, not with ComfyUI's trilinear
resize. Trilinear blurs a mask across frames and lets the original content bleed
through the edit. Max keeps a latent cell that any masked pixel touches fully
editable.
### Does the mask need to be semi-transparent so the LoRA can see the expression?
No — and it would cost you the swap. The expression does not travel under the
mask: the whole source clip, original face included, is in the conditioning as an
aligned reference on `source_id 1`, frame by frame. The model reads the
performance from there whatever the mask does. That is what the LoRA was trained
on: the new head follows the old head's acting.
Partial mask values do something else. In ComfyUI's inpaint path they blend the
original latent back in, so you keep the original geometry in pixels **and the
original identity with it** — the result drifts toward an average of both faces.
That is what `mask_strength` below 1.0 buys, and it is a deliberate choice, not a
way to get expressions. Soft edges are for hiding the seam; the middle of the
mask should stay at 1.0.
### Hard for the sampler, soft for the paste
`mask_blur` exists for compositing, where a soft edge hides the seam. In the
**denoise** mask a soft edge means *partial* denoising, which blends the original
latent — and the original identity — back in right at the edge of the head, so
the swap is weakest exactly where it is most visible. `mask_hard_for_inpaint`
(on by default) binarises the mask before it becomes the denoise mask, leaving
the soft version for the paste-back alone.
`latent_mask_dilate` grows the mask by whole **latent cells** after the
reduction. One cell is 32 px and one latent frame covers 8 video frames, so this
is far coarser than `mask_grow` in pixels — and it is what guarantees the head
sits inside editable blocks rather than clipping at a cell boundary. One cell
took a test mask from 11% to 25% of the grid; two took it to 44%. Watch
`latent_mask` and stop at the first value that covers the head.
**Watch the `latent_mask` output.** It has one frame per *latent* frame, not per
video frame, and that is the real resolution of the edit. A tight outline around
a face can nearly vanish at that scale — and then nothing changes, no matter how
good the LoRA is. If the mask looks thin there, raise `mask_grow`.
## Applying a LoRA to one region and putting it back
The shape of the thing: cut the region out as its own small clip, run the LoRA
on it, composite it back. What makes the result honest is that the composite
goes into the **original frames**, so every pixel outside the mask is the
source's own — it never went through the VAE at all.
```
subject_mask connected
inpaint_with_mask true # outside the mask the latent IS the guide, not noise
mask_hard_for_inpaint true # a soft DENOISE mask means partial denoising at the edge
mask_strength 1.0 # below this the original is blended back INSIDE the mask
paste_back true
paste_confine_to_mask true # only the mask reaches the frame
latent_mask_dilate 0 # one cell is 32 px of extra regenerated area, per side
mask_grow the least that fits the new content
mask_blur 3-5% of the subject's width — the paste's soft edge
```
Since 1.39.0 this works with `crop_mode: off` too. Before that, with no crop the
node returned the decoded frame whole: nothing *changed* outside the mask, but
everything had been through an encode/decode round trip and came back softer.
Now the mask composites against the source either way, and the crop goes back to
being purely a resolution decision.
**Two passes on the region.** Set `paste_back: false` and the images and latent
stay in the crop's own space, so you can upscale and refine there — where the
region actually has pixels — and composite at the end with **BFS Paste Back**,
feeding it the `crop_bboxes` output. That is the whole point of `crop_bboxes`
being a socket.
## Why crop, and which mode
At 512×288 a person filling a fifth of the frame leaves a face about 25 px tall.
No LoRA recovers identity from 25 px — it is missing pixels, not missing
training. Cropping the head region and sampling it full-frame gives that same
face 200–300 px, and the result is feathered back into untouched frames.
| `crop_mode` | behaviour |
|---|---|
| `off` | the whole frame is sampled. Use it first, to confirm the swap works at all. |
| `combined` | one static box around the subject's whole travel, held for the clip. No dependency. |
| `tracked` | constant-size box that stays still until the subject would leave it, then moves as little as possible. |
| `zoomed` | the box also follows the subject's size, planned over the whole clip, every crop resampled to one output size. |
`tracked` and `zoomed` use a planner vendored from
[drozbay/MaskVidExperiments](https://github.com/drozbay/MaskVidExperiments)
(GPL-3.0, the same licence this pack carries) — nothing extra to install. Those
boxes hold still through mask noise and occlusion, and **a jittering crop reads
to a video model as camera motion**, which is what a naive per-frame crop around
a mask produces.
`crop_scale` is the box as a multiple of the subject. At 1.5 the subject occupies
two thirds and the rest is margin. Keep neck and shoulders inside: a face-tight
crop is a framing the LoRA never saw in training, and it will show.
## Sizing, which is the one fiddly part
**Since 1.41.1 the shape is handled for you.** The crop is resized into the
connected latent, so the box has to share its *aspect* — the sampler now asks
the planner for exactly the latent's aspect ratio and says so in `debug`:
```
crop shaped to the latent's 1.500 aspect
```
Without it a mask around a face gives a portrait box, sampling it in a landscape
latent stretches it, and the paste-back squeezes the result back — the face comes
out deformed, and under `crop_mode: zoomed` by a different amount every frame.
The latent's **size** is deliberately left to you, because that is the sampling
resolution and it is the entire reason to crop: a face 200 px tall in the source
sampled at 768 px wide is a face with pixels to work with, and the paste-back
brings it home at the box's size. Choose the size for the quality you want; the
shape is no longer yours to get right.
**BFS Crop Size** is still there for when you want the number, or the per-frame
boxes, upstream of the sampler — building the latent at the right size skips the
rebuild, and the boxes feed BFS Paste Back directly:
```
subject_mask ──┬─→ BFS Crop Size ──→ width / height ──→ EmptyLTXVLatentVideo ──┐
│ │
└───────────────────────────────────────────→ BFS Sampler ←─────┘
```
Keep its three crop widgets identical to the sampler's — the planner is
deterministic, so equal inputs give the same box, and a mismatch means the
sampler silently resizes the crop into a latent that does not fit it.
That resize is worth naming, because it does not look like a sizing problem when
it happens. A mask around a face in a 1920x1080 frame gives a portrait box —
512x672, say. Sampling it into a 768x512 landscape latent stretches the crop
horizontally, and the paste-back then squeezes the result back into the portrait
box. The face comes out visibly deformed, and with `crop_mode: zoomed` the
amount of deformation changes from frame to frame.
### Doing it by hand
The connected `latent` sets the sampled size. When you crop, the box is smaller
than the frame, so the latent should match the box — the `debug` output prints
the size to use:
```
sample size 416x768 (connect EmptyLTXVLatentVideo at this size to avoid a resize)
```
If they disagree the node resamples the crop on the way back, which costs exactly
the sharpness the crop was there to buy. Run once, read `debug`, set
`EmptyLTXVLatentVideo`, run again.
**The latent must come from the graph.** LTX‑2.5 latents are AV — video and audio
nested together — and the node cannot fabricate that structure. Without one it
builds a plain video latent, warns, and the guide may not reach the model at all.
## Chunking
`temporal_tile_size` at 0 samples the clip in one pass. Set it to the length the
LoRA trained at (73 for the LTX head-swap recipe) for longer clips, and the node
slices the guide, the mask and the overlap together per chunk.
Chunked sampling with an AV latent is refused rather than silently corrupted —
slicing a nested latent in time is not a thing the node does yet.
## Decoding
`decode` chooses how the sampled latent becomes frames:
- **full** — one shot. Right for a single pass at the size the sampler ran at.
- **tiled** — for a latent too large to decode whole, e.g. after a 2x upscaler.
A full decode there pins VRAM at 99% and shuttles tensors until it looks like
a hang.
- **none** — skip it and return the latent alone.
**For anything beyond a single pass, prefer `none` and decode outside.** Your own
VAE Decode (Tiled) is tunable, visible in the graph, and reused by the rest of
the workflow. The only reason decoding lives in this node at all is the
paste-back, which needs pixels — so with `none` the compositing is skipped too,
and **BFS Head Swap Paste Back** does it after your decode.
## Second pass at higher resolution
With cropping on, `latent` is the **crop's** latent, not the frame's — which is
what you want to refine, because that is where the face has pixels. To chain a
second pass:
```
Head Swap Sampler (paste_back: off)
├─ latent → LTXV upscaler / second sampler → VAE Decode ─┐
└─ crop_bboxes ───────────────────────────────────────────────┤
guide_video (original frames) ────────────────────────────────────┤
▼
BFS Head Swap Paste Back → frames
```
With `paste_back: off` the node stops compositing, so `images` and `latent` stay
in the crop's own space. Refine there, then bring the result home with **BFS Head
Swap Paste Back**, which takes the crops, the original frames and `crop_bboxes`.
It does the same edge-aware feather and optional mask confinement the sampler
would have done.
Refining before compositing is the right order: upscaling the crop puts the
pixels where the identity is, and the frame around it never needed a second pass.
## Reading the outputs
| output | what it answers |
|---|---|
| `images` | the result, with the crop pasted back |
| `mask_over_source` | is the mask where I think it is? Red over the original frames, green box for the crop |
| `cropped_guide` | exactly what the model was fed. If this is wrong, nothing downstream can be right |
| `crop_mask` | the mask after grow/blur, in the crop's pixel space |
| `latent_mask` | the mask as the sampler sees it — where a thin mask disappears |
| `latent` | the sampled latent — the CROP's, when cropping is on |
| `crop_bboxes` | the boxes, for Head Swap Paste Back after a second pass |
| `debug` | crop mode and box, mask ops, tiling, the size to build the latent at |
## If a second run behaves differently from the first
The `guider` is an object another node builds, and ComfyUI keeps it in the
execution cache between runs. This node has to write the reference specs into
its options — the guider captured its own dict by reference when it was built,
so patching the model alone never reaches the forward — and those specs hold
the encoded reference **latents**.
Since 1.37.0 the write is undone in a `finally` when sampling ends, including
when it ends by raising. Before that the specs stayed on the cached guider:
the latents could not be freed while that guider lived, and a later run that
did not overwrite the same keys — a graph with no references, an aborted run,
another sampler sharing the same guider — sampled with the *previous* clip's
specs, whose shapes no longer fit. If you saw runs that worked once and then
started failing or ignoring the reference until you restarted ComfyUI, that
was it.
Chunked clips also used to keep every sampled chunk on the sampling device
until the whole clip was done; they now move to the offload device as they come
out, so VRAM no longer grows with clip length while the model is still loaded.
## Suggested order when something looks wrong
1. `crop_mode: off`, no mask, `temporal_tile_size: 0`. This is the plain swap; if
it fails, the problem is the recipe or the LoRA, not this node.
2. Add the mask. Check `latent_mask` before judging the result.
3. Add the crop. Check `mask_over_source` for the box, and set the latent to the
size `debug` reports.
4. Only then raise the frame count and turn on chunking.
Sampling settings follow the LoRA's recipe: CFG 3–5 with a real negative, and no
LightX2V (CFG 1.0 means no amplification, and the reference stops being
amplified).
+121
View File
@@ -0,0 +1,121 @@
# BFS LoRA Surgery
Read any LoRA's layers and blocks, then scale or drop them and apply the result live.
No retraining, no files written.
## Why
When a LoRA learns a defect (smoothed skin, identity drifting when the subject is far from
the camera, a pose it refuses to copy) that defect usually lives in one module family and a
range of blocks, not in the whole adapter. Scaling or dropping that group and regenerating
under a fixed seed tells you where it lives, in minutes.
In the cases tested so far the culprit was the MLP **input** projection, in a subset of
blocks. The MLP output projection barely mattered and attention was innocent. Treat that as
a starting point rather than a rule: which group and which block range are right differs per
LoRA, which is why the panel measures instead of assuming.
**Names depend on the base model**, so the panel lists whatever the file you loaded actually
contains:
| base | family | MLP input | MLP output | blocks |
|---|---|---|---|---|
| Qwen-Image-2.1 | `transformer_blocks` | `img_mlp.gate_up` | `img_mlp.out` | 32 |
| Krea 2 | `blocks`, plus `txtfusion.*` | `mlp.gate`, `mlp.up` | `mlp.down` | 28 |
| MiniMax-H3 | `blocks`, plus `token_refiner.blocks` | `mlp.fc1` | `mlp.fc2` | 50 |
Other bases will show other names again (`w1`/`w3`, `fc1`, `proj_in`). Read the list, do not
assume a name.
## How it works
The node discovers the structure from the key names, so it is not tied to any model. It
handles the diffusers convention (`lora_A` / `lora_B`) and the kohya one (`lora_down` /
`lora_up` plus `alpha`). Families like `transformer_blocks`, `double_blocks`, `token_refiner`
or `txtfusion` are detected automatically, along with the block indices and module types in
each.
The panel lists every family with its module types, sorted by how large the update is
(`‖ΔW‖`), and that bar is a quick read on where training actually invested.
Scaling multiplies only the up/B factor, since `(sB)A = s(BA)`. Exact, and negative scales
work.
## Using it
1. Pick a LoRA in `lora_name`. The panel reads it and lists the families.
2. Click a type's `→` to select it, then click blocks to select them (shift-click for a range).
3. Set a scale and press **add rule**, or **add as regex** if you want the same selection
written as a pattern you can hand-edit.
4. Queue the prompt. The edited LoRA is applied to the model in memory.
**Order matters.** Later rules override earlier ones for modules they both match, so put
boosts first and drops last. Use ↑/↓ to reorder.
`rules` is plain JSON and is saved with the workflow:
```json
[
{"enabled": true, "match": {"type": "attn.*"}, "scale": 1.15},
{"enabled": true, "match": {"type": "img_mlp.gate_up", "blocks": "8-15"}, "scale": 0},
{"enabled": true, "match": {"blocks": "16-23", "type": "*"}, "scale": 0}
]
```
A match is either structural (`family`, `type`, `blocks`) or a `regex` tested against the
module's full path. `type` accepts a trailing `*`. `blocks` accepts `8-15`, `0,4,7` or
`8-15,24-31`.
## Saving the result
Two ways, depending on whether you are exploring or building a pipeline.
**From the panel.** Once the rules look right, type a name (or leave it blank) and press
`save as file`. It writes straight into `models/loras/surgery` and reports the result inline.
Nothing else to wire.
**From the graph.** `BFS LoRA Surgery` has a `rules` output. Connect it to the `rules` input
of **BFS LoRA Surgery (save)** and the file is produced as part of the run, with no JSON
copied by hand. Both nodes read the same `lora_name`, so point them at the same file.
Either way the rules are applied identically, since both call the same code.
Leave `filename` empty and the name is built from the rules, so the file says what was done
to it:
```
mylora__attn-x1.15__gate_up-b8_15-off__all-b16_23-off.safetensors
```
Names are capped and sanitized, so a long source name or a 30-rule recipe still produces a
valid filename: when it would exceed the limit, both halves are trimmed and a short hash of
the full recipe is appended, so two different recipes never collapse onto the same name.
The file also carries its own history. Alongside the original metadata (which is preserved
untouched), it writes:
* `bfs_surgery_summary`, one line any metadata viewer will show:
`136/192 modules kept: dropped img_mlp.gate_up in blocks 8-15; all modules in blocks 16-23,
scaled attn.* x1.15 (from mylora.safetensors)`
* `bfs_surgery`, the full record as JSON: tool and timestamp, source filename **and its
sha256** so provenance survives a rename, the readable recipe, the exact rule list, and the
module counts (total, kept, dropped, scaled)
* `ss_output_name` updated to the new name
That means a file you share months from now can still explain what was done to it, and can be
traced back to the checkpoint it came from even if someone renamed either one.
`subfolder` defaults to `surgery` to keep these out of your main list, `save_dtype` can
downcast, and `overwrite` is off by default, so a repeat run gets `_1`, `_2` rather than
destroying the previous one.
## A warning
A variant can win every metric you are looking at and still have destroyed what you trained.
Removing the MLP path entirely gave the sharpest skin in one of the cases here, and made the
LoRA stop copying expression from the source image. Always check a deliberately hard case
(a strong expression, an unusual angle) with your eyes, not only the numbers.
Pruning after training is also not the same as training with those layers excluded. Excluding
them during training may simply fail to converge, because the layer is where the defect lodges,
not where it comes from. That is usually the dataset.
+530
View File
@@ -0,0 +1,530 @@
# BFS Shot Planner / Shot Loop
Turn a long video into shots a video model can actually follow, give every shot its own
reference image and prompt, run the rest of the workflow once per shot, and join the results
back into one video with the original timing and soundtrack.
## Why
A video model only follows a guide reliably inside the clip length it was trained on. An H3
body-swap LoRA trained on 107-frame clips keeps the scene, framing and motion at 4.5 s and
loses them at 10 s, where it starts summarising the clip instead of following it frame by
frame. Splitting the source into model-sized shots, at camera cuts when there are any, keeps
every shot inside that range.
## Nodes
| node | what it does |
|---|---|
| **BFS Shot Planner** | Pick a video, split it on a timeline, set a reference and prompt per shot. Outputs the shots as a ComfyUI **list**. |
| **BFS Shot Unpack** | Opens one shot into plain values: guide frames, reference, second reference, prompt, length, first frame, audio, size, index. Use it with any model. |
| **BFS Shot Repack** | Puts edited pieces back into a shot: Unpack's `shot` output goes to Repack's `shot`, the edited piece (e.g. the reference with its background removed) to its input. Unconnected inputs keep their values; timing, cuts and audio are unchanged. |
| **BFS Shot H3 Conditioning** | Ready-made MiniMax H3 conditioning for one shot, built with the native nodes (Reference to Video + Add Guide). Optional *duet*: the shot's clip pinned in a side panel (canvas or shifted RoPE; route the model through the node for the shift); the join cuts the panel off. Optional prompt writer: pick a *task* (character swap, style, setting, appearance, lighting / weather, custom) and an *instruction*; with a VLM on its *vlm* input it writes each shot's duet prompt from the shot and its references, otherwise a template for the task (*planner prompt*, the default, keeps the planner's prompt). The written text comes out on *prompt*. Optional *setting_ref* (TSC's tip): one more reference, the shot's middle frame with the person covered in TV static, so the model sees the place in full detail; it is the last `<Picture n>` (write `{setting}` for its tag, otherwise a sentence is added to subject_definitions). |
| **BFS Shot Join** | Concatenates the decoded shots in order, trims each to its true length, cross-fades soft joins and returns the soundtrack. With *comparison* on it also returns a side-by-side video (original shot \| references \| result) with the shot's info and your *label* on top and its prompt below, ready for Create Video. |
| **BFS Shot H3 Duet** | Renders one shot with H3: the shot's clip pinned beside the video (duet, no LoRA), the shot as an aligned guide (for body-swap LoRAs), or both. |
## The panel
Use the language selector in the top bar to switch the interface between English, Portuguese, Spanish and Simplified
Chinese. The choice is saved in this browser and does not alter the workflow plan or prompt text. **Tour** opens an
optional step-by-step introduction when clicked; it never appears automatically.
The planner's panel has five tabs, in the order you work. The timeline stays on top of every tab.
| tab | what is there |
|---|---|
| 🎞 **Video** | pick or upload the video, audio, how it is split into shots (cuts or fixed length), generation size |
| 🎬 **Shots** | one card per shot (scroll sideways, **← →** move between shots), the preview player and the selected shot's editor: reference & prompt, who is replaced (mask, generation mode, `{target}`), continuity, **Copy to other shots** |
| 👥 **People & masks** | the cast found by face (a reference per person) and the global mask settings |
| 📝 **Prompts & refs** | the global reference and prompt, the VLM and the reference descriptions (`{details}`) |
| ▶ **Run** | **Checks** (what to fix before running, with a button to each shot), auto / queue loop, filters, test limits |
Shot cards show badges for what each shot has: 🎯 mask, 🎭 / ✂ generation mode, 🧑 `{target}`, ⛓ continuity and ⚠ problems.
The header's ⚠ count opens the checks.
**Copy to other shots** (in the shot editor) copies the chosen parts of the selected shot to all shots, or to the
shots after it: references, prompt, mask text, mask points (each shot gets them on its frame at the same relative
position), generation mode, `{target}`, continuity. It replaces the old "→ all" buttons.
## How the loop works
ComfyUI runs any node that receives a list once per item and pairs several lists by index. The
planner's `shots` output is a list, so everything downstream of it (conditioning, sampler,
decode) runs once per shot, and shot *i* always gets guide *i*, reference *i* and prompt *i*.
Single values such as the model and VAE are shared by every run. The join takes the whole list
back (it is an `INPUT_IS_LIST` node) and puts the video together.
Two run modes:
- **Auto loop (one run)**: every shot in one queue run. Simple; holds all decoded shots in memory.
- **Queue loop (one shot per run)**: each run generates the next pending shot and stores it on
disk. The join blocks every node after it until the last shot, then outputs the full video,
so a Save Video node only fires once. With *Auto-queue the next shot* on, the panel re-queues
the workflow after each shot. *Reset loop* starts over. Changing the plan starts a new run.
## Guide frames and lengths
Each shot is generated at a length the model accepts (frame grid: H3 `17n+5`, LTX/Wan `8n+1`,
Wan `4n+1`, or any). When a shot is shorter than that length, the extra guide frames come from
the video that follows it, so the guide never repeats or stretches. The join cuts each result
back to the shot's own frames, which keeps the timing identical to the source. Those extra
frames are also a real overlap, which the join uses to cross-fade boundaries that are not
camera cuts.
## Splitting
- **Camera cuts** (default): cuts from **PySceneDetect** (`adaptive` or `content`; install
`scenedetect`), or the built-in detector when the package is missing. Shots shorter than the
minimum merge into a neighbour when the merge still fits; shots longer than the maximum split
into equal parts.
- **Fixed length**: equal parts no longer than the maximum.
- **One shot (whole range)**: the whole range (start / end frame, *Max total seconds*) as a single shot, whatever its
length; *Max seconds / shot* is not applied. For "N seconds in one piece" set *Range length, seconds* = N, right
next to the mode (it moves the end frame; *Max total seconds* in the Run tab does the same for quick tests).
For pieces of at most X seconds use *Fixed length* with *Max seconds / shot* = X.
- **By hand**: drag the white handles on the timeline to move a boundary, double-click the
shot bar to split, *⇥ Merge* to join, or select a shot and press **Delete** to remove the cut at its
start (it merges into the previous shot). Your edits are what runs.
**Range** (🎞 Video tab, with its own player): *Start frame* / *End frame* (timeline frames; end 0 = the end of the
video) limit the plan to a part of the video, before anything is split, analysed or generated. Play or scrub the player
(or click the timeline) and press *⇤ start here* / *⇥ end here* to take its exact frame. The
timeline dims what is outside; the join and the soundtrack cover only the range. *Max total seconds* counts from the
start of the range.
*Max seconds / shot* is converted to frames and snapped down to the grid (4.5 s at 24 fps =
107 frames for H3). *Max shots* and *Max total seconds* cap a long source.
## Audio
The source card shows whether the video has a usable audio track (rate, channels) or why not. *Use the
video's audio* or *silent track*. Without a usable track (none, unreadable, or silent chosen) the planner's
`audio`, BFS Shot Join's `audio` and the H3 Duet nodes return a **silent track of the right length** instead
of nothing, so Create Video / Save Video always work. Audio is read with ffmpeg, or with PyAV when the ffmpeg
binary is missing.
## Preview
*Play shot* plays only the selected shot, from its first to its last frame, and stops (or loops with
*loop shot*); every shot card has its own ▶. *Play all* plays the whole plan and jumps over shots
that will not run. The panel shows the shot's start, end and current time (and frame), a yellow
playhead follows on the timeline, and clicking the timeline seeks. The source video streams from
the input folder, so only what you play is loaded.
## Filters
Skip shots that should not run, decided per shot from a few sampled frames:
| filter | skips a shot when |
|---|---|
| Needs a person | no person is detected |
| Min person size | the largest person covers less than that fraction of the frame (wide shots) |
| Max people | more people than that (crowds) |
| Needs a face | no face is detected |
| Skip dark / fades | the mean brightness is below the level |
| Skip static | there is almost no motion (title cards, freeze frames) |
| Min frames | the shot is shorter than that |
People and faces use YOLO from `models/ultralytics` (`person_yolov8m-seg.pt`, `face_yolov8m.pt`, the
Impact Pack files) with OpenCV fallbacks. *Analyse people & faces* shows the numbers on every shot
card. Each shot can override the filters (always run / always skip).
Skipped shots do not run. Connect the planner's `timeline` output to BFS Shot Join and they are
filled with the original video, keeping the full duration and soundtrack, or removed (with the
matching audio) when *Skipped shots in the output* is set to remove them.
## Cast (people by face)
*Find people* samples the video every 0.5 s, detects faces and groups them into people by face
identity, on the CPU (it never touches the GPU; the result is cached per video). **No extra package is needed**: by
default it uses OpenCV Zoo's **YuNet** (detection) + **SFace** (recognition), Apache-2.0 models run by OpenCV itself and
downloaded once into `models/opencv_face` (0.2 + 37 MB). If **InsightFace** is installed (`buffalo_l` in
`models/insightface`) it is used instead (a bit more accurate; mind its license). Last resort, without either: SAM 3.1
faces grouped by colours (hair, skin, clothes). Each
person card shows the face, screen share and first/last appearance (click the face to seek).
Give a person a reference (and a second one) by clicking the slot, or pick one from the recent
references; *ignore* leaves that person out. Every shot card shows who is in it, with the main
person (most screen time) highlighted and linked people outlined in green.
| option | effect |
|---|---|
| Shots use their main person's reference | a shot without its own reference takes its main person's (shown with a dashed green outline) |
| Split where the main person changes | adds a boundary where the largest face switches to another person for at least a second, then re-splits |
| Only run shots with a linked person | shots where no linked, non-ignored person appears are skipped ("no linked person") |
A reference set on the shot itself always wins; the global reference is the last fallback.
## References and prompts
Every shot can have its own reference, second reference and prompt; shots without them use the
global ones. Click an empty reference slot to upload an image straight from your computer; a filled
slot has **✕** to remove it; **Copy to other shots** gives a shot's references to every shot. The last ten
references you picked are shown under the shot editor for one-click reuse (click = reference,
shift+click = second reference). Connected `ref_image`, `ref_image_2` and `prompt` inputs override
the panel's global values, so an edited first frame or a prompt from another node can drive the
defaults.
The planner also outputs `ref_image` and `ref_image_2`: the references the shots actually use,
without repeats, so a plan where every shot shares one reference returns a single image.
## Mask & crop (SAM 3)
Generate only part of the frame. In a shot's editor (*Who is replaced*), type what to segment in English
(`person in white`, `woman with red hair`, `red car`; up to 32 tokens, commas for several things) or click
**🎯 Points** to pick positive / negative points on any frame of the shot, then **👁 Preview** (red = mask,
yellow = crop box). Then pick how the shot is generated:
| mode | what is generated | crop / uncrop |
|---|---|---|
| **Full frame** | the whole frame; the mask only feeds `{target}` and the setting picture | no |
| **🧩 Frame + paste** | the whole frame (so the model follows the guide's pose freely and the new person may be bigger); BFS Shot Join then pastes only the person onto the original frame: the shot's mask plus the new subject's outline (SAM 3 on the result with the shot's *new subject* text, default `person`; `paste_new_outline`) | no crop; paste in the Join |
| **🎭 Mask only** | only the mask, on the whole frame; everything else stays exactly as it was | no |
| **✂ Crop** | a box around the mask (more pixels for a small person); BFS Shot Join pastes it back | yes |
| **✂🎭 Crop + mask** | only the mask, inside the crop: the most detail with the background kept | yes |
*Frame + paste* is the safest for a different body (e.g. a woman in profile replaced by a man facing the camera):
*Mask only* binds the new person to the old outline, *Frame + paste* does not. The two mask modes need **BFS Shot H3 Conditioning → inpaint = per shot (planner)** (the default; *only the mask*
forces it on every shot, *off* never). The crop is one box around the mask (the union over all its frames, so it
does not shake), at the source resolution, sent to the model at the generation size; **BFS Shot Join** pastes the
result back, feathered by the mask (or the whole box). **BFS Shot Unpack** also outputs the shot's mask (`mask`),
e.g. for an inpainting model. In **BFS Shot Join**'s comparison video, cropped shots show their mask (red) and crop
box (yellow) over the original column (`comparison_mask`, on by default).
**Mask opacity** (mask modes, per shot, slider under the modes): the value of the white inside the generation mask.
1 regenerates the masked area completely. A grey mask (e.g. 0.85) keeps part of the original there: H3 puts those
rows at *opacity* × the noise level, so they start from the original partly visible. 0.8-0.9 keeps pose, outline and lighting while still swapping; too low copies the original
person. With `guide_mode = aligned guide` the model also sees the whole original shot as its guide, so it follows
the motion either way. (Not applied with the duet panel.)
**🎨 Color Guide** is a separate optional visual cue. Turn it on for a shot with a mask and choose the marker color
for ref 1. The default colors for the global references are set below each image in **Prompts & refs**; a shot may
override them. The selected region is tinted in the source guide, while the image detail,
movement and expression remain visible underneath. This does not turn on `Mask only` or create an H3 generation
mask. The prompt must explain that the tint is temporary and that the matching reference replaces the tinted person.
For consistency across shots, **People & masks → Find people** groups sampled faces by identity (ArcFace through
InsightFace when installed; otherwise YuNet + SFace, or a weaker appearance fallback). Choose the matching `Person N`
in each shot to inherit that cast member's Color Guide color. This is a user-confirmed link: profile views, small faces
and occlusion can make identity grouping uncertain. Leave the shot on `manual identity` when unsure. The shot's mask
still selects the actual region to tint, and its color can be overridden for that shot.
For two people in one shot, open **+ Add a second person**, use **ref 2** for that selection, and choose a different
color (purple by default; duplicate colors are disabled). Enable *use_ref_2* on BFS Shot H3 Conditioning. The
second SAM 3 text selection is optional; preview it before queuing. Color opacity and ArcFace identity links are
under the optional settings. Use this prompt and replace the color names if you choose another palette:
```text
body_swap: subject_definitions: <Subject 1> is the character shown in <Picture 1>, whose identity, face, head, hair or fur, skin, body shape, proportions and clothing define the green-marked replacement. <Subject 2> is the character shown in <Picture 2>, whose identity, face, head, hair or fur, skin, body shape, proportions and clothing define the purple-marked replacement. In the guide video, the translucent green and purple tints are temporary editing markers, not character appearance. Replace the green-tinted person only with <Subject 1> and the purple-tinted person only with <Subject 2>. The guide video provides the scene and the complete performance of both people, including their body poses, gestures, head motion, gaze, speech and facial expressions.
summary: [video generation + references] Replace the green-tinted guide person with <Subject 1> from <Picture 1> and the purple-tinted guide person with <Subject 2> from <Picture 2>, preserving the scene, camera and complete motion and expression of both people.
detailed_description: [Shot 1] Replace the person marked by the translucent green tint with <Subject 1> from <Picture 1>, and the person marked by the translucent purple tint with <Subject 2> from <Picture 2>. Each character follows that person's exact position, timing, body pose, gestures, head motion, gaze, speech and facial expressions in the guide. Remove both temporary color markers completely. Keep the two reference identities distinct and stable in every frame. Preserve their interaction, the background, lighting, camera and all unselected content from the guide video. Prevent identity drift, clothing changes, texture crawling and flicker. The result is a real, photographic video.
```
The VLM's shot analysis suggests a SAM 3 text selection from the original frames; it does not assign guide colors or
decide that two detections are the same identity. **Describe target** only describes the first shot selection. For
two-color swaps, choose both selections and identities yourself, and use the explicit mapping above. The VLM prompt
writer for *duet* is a separate split-screen task; its generic character-swap draft assumes the supplied pictures
show one character, so use *planner prompt* for this two-character template. `Describe refs` also treats ref and ref 2
as views of one identity; write the two subject definitions explicitly when ref 2 is a different person.
**Mask guide** (experimental, *Mask only* shots, BFS Shot H3 Conditioning → `mask_guide`): the shot with only the
masked region visible, the rest grey, so the model looks at the subject's pose and outline on its own while the mask
limits where it generates. *+ extra guide* adds it as a second aligned guide next to the normal one (the model sees
both); *instead of the full guide* replaces the normal one; *+ reference video* gives it as a native reference video
(`<Video n>`, after the shot's own one when guide_mode uses it; write `{mask_video}` in the prompt where its tag goes, otherwise a sentence is added). `mask_guide_look` sets what it shows of the subject: *grey blurred* (default: volume, light and head direction, no colours or face, so the model follows the motion without copying the old subject), *colour* (as it is; can make the model copy the old subject), *silhouette* (flat shape), *edges* (outlines) or *pose (people)* (the skeleton of the people in the mask, OpenPose colours; needs ultralytics and a YOLO pose model in `models/ultralytics`, e.g. `pose/yolov8m-pose.pt`). `mask_ref_size` makes that reference video smaller (default 1/2, ~1/4 of the tokens): it only has to show the pose and outline. Compare with
*off*: the LoRAs were trained with one full guide, and it can also pull the old subject's look.
### Masks from a video (rotoscoping) instead of SAM 3
A mask made elsewhere (After Effects Roto Brush, DaVinci Resolve Magic Mask, another ComfyUI workflow…) can replace
SAM 3. It is a black and white video, **white = the subject**, and it works with every mode (Mask only, Crop,
Crop + mask), `{target}`, the setting picture and the comparison overlay. Three places, the most specific wins:
| where | covers | wins over |
|---|---|---|
| shot editor → **🎞 mask video** | only that shot (its first frame = the shot's first frame) | everything |
| the shot's own SAM 3 text / points | that shot | the two below |
| the planner's optional **`mask`** input (MASK or IMAGE batch, e.g. Load Video + Convert Image to Mask) | the whole video, one mask per source frame | the panel's mask video |
| People & masks → **🎞 Mask video (whole video)** | the whole video, matched by time | SAM 3 for shots without a mask |
Any size and frame rate: it is matched to the shot by time and resized. The console says which mask each shot used.
**Stitch finishing** (BFS Shot Join, cropped shots only; adapted from Neko (Nekodificador)'s *NKD Inpaint Stitch*, MIT, built on AbleJones's workflow and nodes):
| Option | What it does |
|---|---|
| `edge_hardness` (0-1) | hardens the soft edge of the paste; raise it when a faint ghost of the original person shows around the new one |
| `match_colors` (0-1) | corrects the colour / brightness drift of the generated patch (Reinhard in LAB), with statistics over the whole shot so it does not flicker |
| `match_region` | *around the subject (swap)*: measured on a ring of background around the mask, so a new person keeps their own colours. *inside the subject*: measured inside the mask, for retouching the same content |
| `seamless_edges` | Poisson blend (OpenCV) for stubborn seams; slower |
**Which mode.** A generation mask alone keeps the background exact but generates the person at the frame's own size,
so a small person gets few pixels; a crop alone gives the person more resolution, but the whole box is regenerated
and pasted back (that is what the finishing above is for). *Mask only* is the simplest when the person fills a good
part of the frame; *Crop + mask* is best for small people. In the mask modes the latent starts from the shot's own
frames and its SAM 3 mask (grown by *expand* plus one latent cell) becomes the H3 generation mask. Raise *expand*
when the new person is bigger than the old one (longer hair, wider body): the generated area cannot go past the
grown mask; their shadow or reflection outside the mask stays from the original.
Credit to **Neko (Nekodificador)** and **AbleJones**, whose workflow and nodes the crop + generation mask follows.
ComfyUI builds without native H3 generation masks need a per-row mask patch on the model
(e.g. ComfyUI-MiniMaxH3-PerRowMasking).
The **People & masks** tab holds the global mask settings, with defaults that work as they are: fill holes on,
temporal expand 2 frames (less flicker), expand 16 px, feather 12 px, padding 15 %, paste by mask, blockify off
(16 aligns the mask to H3's latent grid), threshold 0.5, 4 objects. *Show masks on shots* overlays the preview
on the shot cards. SAM 3 uses the official `sam3.1_multiplex_fp16.safetensors` in `models/checkpoints`,
downloaded from Comfy-Org/sam3.1 the first time.
## VLM suggestions
Connect a vision-language model to the planner's `vlm` input (CLIPLoader with `qwen3vl_4b` or `qwen3vl_8b`).
It looks at a few frames of every shot and suggests what to segment, a description of the shot (camera,
framing, action) and whether to run it. In the **Prompts & refs** tab: *Use the VLM when the workflow runs* applies the
suggestions at run time (mask text for shots without one; `{shot}` in a prompt becomes that shot's
description), *Analyse shots* asks from the panel (after one run with the VLM connected), and each shot's
editor shows its suggestion with buttons to apply it. The summary output lists them too.
## Describe references ({details})
The VLM can describe the references, for any task (not tied to swaps), and any subject: a person, a creature, a robot,
an animal or a cartoon character (it says what the subject is first). In the **Prompts & refs** tab pick an instruction
preset: *short* (the default: one noun phrase of at most ~20 words that fits inside a sentence, e.g. `green goblin with
long pointed ears in a torn brown leather vest`, for `<Subject 1> is the {details} shown in <Picture 1>`), *full body* (face, hair, skin, age, build, clothing piece by piece), *head / face*, *face attributes* (a
short comma-separated list), *outfit*, or *custom* (your own instruction). **📝 Describe refs** writes one
description per reference set: the global references, every shot's and every cast person's. They are saved in
the plan and editable. Write **`{details}`** anywhere in a prompt and every shot gets the description of its own
references; a set without a description is described by the VLM at run time (no click needed).
**Fit the prompt** (on by default, next to 📝 Describe refs): the VLM gets the sentence of the prompt that holds the
marker, with a blank in its place (e.g. `<Subject 1> is the ___ shown in <Picture 1>`), and writes the description for
it, so the description adapts to the prompt and not the other way round (no "the A goblin", no paragraph inside a
sentence). The panel buttons use the global prompt (or the shot's own); at run time the prompt the shot really uses.
The VLM buttons in the panel work after the workflow has run once with the VLM connected (ComfyUI only hands
models to nodes when they run). Tested with Qwen3-VL 8B; some smaller or modified models return empty answers
for some wordings: the planner retries with a reworded request and with sampling, but if a preset stays empty,
use the 8B or another preset.
## Settings reference
Every setting in the panel has a hover tooltip (ⓘ). The main ones:
| setting | what it does |
|---|---|
| Mode | Camera cuts (shots start at cuts, long ones split evenly), fixed length, or one shot (the whole range) |
| Detector / Sensitivity | PySceneDetect adaptive or content, or the built-in one; higher sensitivity finds more cuts |
| Frame grid | frame counts the model accepts (H3 17n+5, LTX/Wan 8n+1, Wan 4n+1) |
| Max / Min seconds per shot | longest shot sent to the model; shorter shots merge into a neighbour |
| Timeline fps | the frame rate the planner works on (generation and audio use it) |
| Max shots / Max total seconds | test on the first shots or seconds only |
| Megapixels / Size multiple | generation size at the source's aspect ratio, snapped to the multiple |
| Run | auto loop (all shots in one run) or queue loop (one shot per run, stored on disk) |
| Skipped shots in the output | keep the original video there or remove those shots |
| Mask: padding / expand / feather / temporal expand / blockify | crop context, paste-back growth and softness, flicker hold, square blocks |
| Mask: threshold / max objects / paste back | SAM 3 text matching, objects tracked, paste only the mask or the whole box |
| VLM: frames per shot / max tokens | how much the VLM sees and how long it may answer |
## One seed per shot: BFS Shot Seed (1.94.0)
A seed widget on the sampler gives every shot the SAME seed (the node runs once per shot with the same value).
**BFS Shot Seed** (shot in, seed out) makes it a choice: *same for every shot*, *increment per shot* (seed, seed+1 ...)
or *random per shot* (unrelated but repeatable seeds from the base seed). `overrides` pins single shots, e.g.
`3:12345, 5:777`, to keep the shots that came out well and re-roll only the others.
## Structured starting noise: BFS Shot Noise (1.94.0, experimental)
Optional. **BFS Shot Noise** gives the NOISE for SamplerCustomAdvanced; with every amount at 0 it is exactly RandomNoise.
| Input | What it does |
|---|---|
| `shot_seed` (+ `shot`, `overrides`) | the seed of each shot: same / increment / random per shot, as BFS Shot Seed |
| `shared_base` | the coarse pattern of the noise comes from `base_seed`: the same for every shot, at any resolution and any length (it is a tape along the timeline); the fine detail stays per shot. Put the seed of a result that worked in `base_seed` to keep its coarse noise |
| `distribution` | gaussian (default), laplacian, student-t or uniform, all at unit variance |
| `spectrum_tilt` | 1/f^alpha spectrum: > 0 puts more energy in the large blobs ("fractal" noise), < 0 in the fine grain |
| `pyramid` | coarser layers of noise added on top |
| `static_share` | a part of the noise identical in every frame |
| `guide_structure` (`vae`, `guide_frames`, `mask`) | outside the mask, the guide's latent is mixed into the noise |
| `ref_tint` (`vae`, `ref_image`, `mask`) | inside the mask, the noise leans toward the reference's average latent |
All mixes keep unit variance. The model was trained on plain noise: use small amounts and compare with 0.
## {details} filled automatically (1.92.0)
A `{details}` in a shot's prompt is filled without pressing 📝 Describe refs: the planner's VLM describes the shot's
references before any SAM 3 work; with no VLM on the planner, the slot is left for **BFS Shot H3 Conditioning** or
**BFS Shot H3 Duet**, whose `vlm` input (the same Qwen3-VL CLIPLoader) describes them (the *short* preset, fitted to
the sentence). Edited descriptions always win. With no VLM anywhere it becomes "character" and the log says so.
## Source blur (keep the mouth) (1.91.0)
`source_blur` on **BFS Shot H3 Conditioning**, **BFS Shot H3 Duet**, **BFS H3 Duet** and **BFS H3 Duet Conditioning**
blurs the source person in what the model *sees* of the source (aligned guide, native reference video, duet panel),
keeping the **mouth** sharp: the model still reads the pose, the motion and the lips but has no face to copy, so the
identity comes from the reference. Masks from SAM 3.1 (the part, minus a slightly grown "mouth" mask), cached per shot.
`face (keep mouth)`, `head + hands (keep mouth)` (also hides hair and hands), `person (keep mouth)` (the whole body:
only for big changes, the pose then comes from the blurred shape). The kept area of *Mask only* and the output are
never blurred. Trick shared by JalenBrunson (Sapiens2 there) for reliable ref2v swaps.
## Big videos (upload)
ComfyUI refuses any single request bigger than `--max-upload-size` (100 MB by default), which is why big videos fail
to upload in most nodes. The planner's ⬆ Upload (source video, mask videos, pictures) sends files of 32 MB or more in
8 MB pieces that are written straight to disk and assembled in the input folder, so any size works without changing
how ComfyUI is started (a dropped piece is sent again). For other nodes, start ComfyUI with a higher limit, e.g.
`--max-upload-size 4096`, or copy the file into `ComfyUI/input` and pick it from the list.
A newly selected or uploaded source starts as **one shot covering at most its first 15 seconds**. This is only the
initial plan: widen the range or choose *Camera cuts* / *Fixed length* and press *Auto split* when you want multiple
shots. Existing saved plans keep their own range and split mode.
## Continuity between shots
Each shot (after the first) can continue from the **previous shot's generated result**: in the shot editor,
*Continuity* = *previous shot as reference* adds a frame of it as one more `<Picture n>` after the shot's own
references (good across camera cuts, keeps the person consistent), and *previous shot as first frame* anchors it
at frame 0 of the shot (for continuous action without a cut). Pick which frame: the previous result's first,
middle or last. **Copy to other shots** with *continuity* ticked applies it to every shot after the first.
It needs the previous result to exist: it works in the **queue loop** (the planner reads the stored result) and in
the **auto loop with BFS Shot H3 Duet** (it renders shot by shot and keeps the last result). BFS Shot Unpack also
outputs it as `previous_result`. This is different from H3 Conditioning's *first_frame*, which anchors the shot's
own frame from the source video.
### History: a long take, window by window (1.87.0)
*Continuity* = **previous shot's end as history** carries the **last 17, 34 or 51 frames of the previous shot's
result** into the next shot as clean frames placed **just before frame 0 on H3's timeline**: the model sees them as
the immediate past of the video it is generating, so motion, camera, light and the new look continue across the join
(instead of one still frame). Idea from Akatz's [H3 Relay](https://github.com/akatz-ai/h3-relay) windowed edit and
Ethanfel's MiniMax H3 Context Loop; this is BFS's own implementation (a process-local patch of H3's packed layout
that moves the video, its audio and the other guides past the history; ComfyUI files are not edited).
- **On by default** (1.88.0) for every shot after the first; set *Continuity = off* on a shot to drop it. Plans saved
before keep what they had.
- **Same take only.** A shot that starts at a camera cut skips it (nothing to continue). For one long take, split it
with *Mode = fixed length* (or let *Camera cuts* split long shots) and set history on every shot after the first
with **Copy to other shots → continuity**. Shots still exist: they are the windows, and cuts reset the history.
- **Every mode:** full frame, *Frame + paste*, *Mask only* (the mask regenerates the target; the history is a
condition outside it), *Crop* (the history is cut with this shot's box) and the **duet canvas** (the source frames
under the history go into the panel strip). Not with duet *shifted RoPE*. The aligned guide still drives this shot's
own frames, so sync stays frame-exact.
- **BFS Shot H3 Conditioning and BFS Shot H3 Duet** both use it (duet canvas or aligned guide).
- **Queue loop:** shots run in order and each reads the stored result of the one before. Auto loop: taken from the
previous render when it matches.
- Video only for now (the previous audio is not carried). Costs one more 17/34/51-frame condition per shot.
## MiniMax H3 with an aligned guide (example)
```
BFS Shot Planner ── shots ──► BFS Shot H3 Conditioning (clip, vae, guide_mode = aligned guide)
└─ positive, latent ─► CFGGuider / sampler ─► VAE Decode ─┐
BFS Shot Join ◄── images ─────────────────────────────────────────────────────────────────┘
└─► Create Video (fps, audio) ─► Save Video
```
For other models, use **BFS Shot Unpack** and wire `guide_frames`, `ref_image`, `prompt` and
`length` into that model's own guide and reference nodes; the list mapping works the same way.
## Choosing who is replaced (`{target}`), step by step
With several people on screen, a prompt that says "replace the person" leaves the model to guess. `{target}`
puts a short description of the right person into the prompt of every shot ("the young woman with dark hair in a
pink top"), so the model knows WHICH one to swap. The selection is made with SAM 3, and the description by the
VLM, so it works for anyone or anything you can click on.
### What you need
- The planner's **`vlm`** input connected to a Qwen3-VL (CLIPLoader, type *stable_diffusion*, `qwen3vl_8b` recommended).
- The workflow run **once** with it connected. ComfyUI only hands models to nodes when they run; after that the
panel buttons can use it.
- Without a VLM it still works: you get the cut-out and type the description yourself.
### 1. Put `{target}` in the prompt
Write `{target}` wherever the prompt names the person being replaced: in the global prompt (all shots) or in a
shot's own prompt. For example, the swap caption the H3 LoRAs were trained with:
```
... only the face, body and clothing of {target} are replaced. summary: [video generation + reference] Replace
{target} in the guide video with <Subject 1>, keeping the scene, camera and the complete motion and facial
performance of the guide video. detailed_description: [Shot 1] <Subject 1> takes the exact place of {target}
and moves with the same timing, body pose, gestures, ...
```
`{target}` can be combined with `{details}` (the description of the references) and `{shot}` (the VLM's
description of the shot).
### 2. Select the person
Open a shot and click **🎯 Points…**. The modal shows one frame of the shot; the slider below picks the frame.
- **Click on the person's body** (torso), not only the face. A click on the face selects the head, and the
description then misses the clothes.
- **Right-click** (or shift+click) puts an *exclude* point, e.g. on a second person standing close.
- **👁 Segment** shows what SAM 3 picked (red). Add or remove points until only that person is red.
- Instead of points you can just type what to segment (`woman in pink`, `man with glasses`, `dog`): no clicking is
needed, SAM 3 finds and tracks it through the shot. Points are for when the text is ambiguous (two similar people).
Points win when both are set. **Copy to other shots** with *mask text* (and *mode*) ticked gives the text (and
the generation mode) to every shot.
Then save:
| Button | What it does |
|---|---|
| **Save (this shot)** | keeps the selection on this shot only |
| **Save → all shots** | applies the same points to every shot, each on its frame at the same relative position as the frame you clicked on (e.g. clicked in the middle of this shot → every shot uses its middle frame) |
**Save → all shots** is for a person who stays in the same place across shots (a sequential video, a fixed camera).
**🎯 Track → whole video** (in the points window) is for a person who moves around: the selection is tracked by SAM 3
through the whole video (or the range), forward and backward from the frame you clicked on, and every shot without
its own selection takes its part of that track. Also type a short text (e.g. `woman in pink top`): after a camera cut
the tracker uses it to find the subject again. The People & masks tab shows the track and removes it. The first run
tracks the whole range (a minute or two per few hundred frames); the result is cached until the selection changes.
### 3. Describe the person
Click **🧑 Describe target** in the shot's editor:
- SAM 3 cuts the person out of the selected frame, and a thumbnail of the cut-out appears next to the field.
- The VLM writes a short phrase into the **`{target}`** field.
- The field stays editable: correct it or write your own. Keep it short and visual (who, hair, main clothes with
colours) and start it with "the".
- **Copy to other shots** with *{target}* ticked copies the description to every shot. Use it when the same person is replaced throughout;
otherwise describe shot by shot.
### 4. Check and fix
- **👁 Preview** on a few shots shows the selection on several frames of each shot.
- In a shot where the selection picked the wrong thing, **✕ Clear** removes its points, text and mode. Select
again in that shot only (**Save (this shot)**) and describe again.
- **Copy to other shots** with *mask points* ticked copies that shot's points to every shot without opening the modal.
### What happens when the workflow runs
Every shot's prompt gets its own `{target}` text: the one you wrote or made with 🧑 Describe target; when the field is
empty but the shot has a selection (mask text, points or a mask video) and the VLM is connected, the subject is cut out
and described automatically (written for the prompt's `{target}` sentence). A shot with neither gets "the person". The selection is
only used to make the description: it does not crop or mask anything unless the shot's mode is *Mask only*, *Crop*
or *Crop + mask*.
### Examples
- **Two people, swap only one:** in the first shot, click on the woman's torso, put an exclude point on the man,
then **Save → all shots** and **🧑 Describe target** → "the young woman with long black hair in a white blouse".
**Copy to other shots** with *{target}* ticked.
- **The person changes between shots:** describe shot by shot, e.g. shot 1 "the man in the grey suit", shot 2
"the woman in the red dress". Each shot keeps its own text.
- **Animals:** it works the same way, e.g. "the small black dog in front".
### Troubleshooting
| Problem | Fix |
|---|---|
| "No VLM yet" | connect the `vlm` input and run the workflow once, or type the description |
| "Select the person first" | the shot has no points or mask text: use **🎯 Points…** |
| "SAM 3 found nothing" | click again on the person (on the torso), or try another frame with the slider |
| Only the face or the hair is described | the click was on the head: click the body |
| The other person is described | add an exclude point (right-click) on the other person, **👁 Segment**, save, describe again |
| The swap still hits the wrong person | make the description more specific (position: "on the left", clothing colours) |
+303
View File
@@ -0,0 +1,303 @@
// Shot Planner: constants and small helpers shared by the panel and its views (no side effects: ComfyUI loads every
// .js under web/js as an extension, so this module only exports).
import { h as vueH, reactive, ref } from "../vendor/vue.esm-browser.prod.mjs";
// UI language is a browser preference, not part of the workflow or generation prompts.
const LANG_KEY = "bfs.shotloop.language";
const savedLanguage = (() => { try { return localStorage.getItem(LANG_KEY) || "en"; } catch { return "en"; } })();
export const uiLanguage = ref(["en", "pt", "es", "zh"].includes(savedLanguage) ? savedLanguage : "en");
export function setUiLanguage(value) {
if (!["en", "pt", "es", "zh"].includes(value)) return;
uiLanguage.value = value;
try { localStorage.setItem(LANG_KEY, value); } catch { /* private mode */ }
}
const TEXT = {
"🎞 Video": ["🎞 Vídeo", "🎞 Vídeo", "🎞 视频"], "🎬 Shots": ["🎬 Shots", "🎬 Planos", "🎬 镜头"],
"👥 People & masks": ["👥 Pessoas e máscaras", "👥 Personas y máscaras", "👥 人物和蒙版"],
"📝 Prompts & refs": ["📝 Prompts e referências", "📝 Prompts y referencias", "📝 提示词和参考图"],
"▶ Run": ["▶ Executar", "▶ Ejecutar", "▶ 运行"], "📖 Guide": ["📖 Guia", "📖 Guía", "📖 指南"],
"🧭 Tour": ["🧭 Tour", "🧭 Recorrido", "🧭 引导"], "Language": ["Idioma", "Idioma", "语言"],
"Source video": ["Vídeo de origem", "Vídeo de origen", "源视频"], "Range": ["Trecho", "Rango", "范围"],
"Split into shots": ["Dividir em planos", "Dividir en planos", "拆分镜头"], "Generation size": ["Tamanho da geração", "Tamaño de generación", "生成尺寸"],
"Camera cuts": ["Cortes de câmera", "Cortes de cámara", "镜头切换"], "Fixed length": ["Duração fixa", "Duración fija", "固定时长"],
"One shot (whole range)": ["Um plano (trecho inteiro)", "Un plano (rango completo)", "单个镜头(整个范围)"],
"Mode": ["Modo", "Modo", "模式"], "Detector": ["Detector", "Detector", "检测器"], "Sensitivity": ["Sensibilidade", "Sensibilidad", "灵敏度"],
"Frame grid": ["Grade de frames", "Múltiplo de fotogramas", "帧步长"], "Max seconds / shot": ["Máx. segundos por plano", "Máx. segundos por plano", "每个镜头最长秒数"],
"Min seconds / shot": ["Mín. segundos por plano", "Mín. segundos por plano", "每个镜头最短秒数"], "Timeline fps": ["FPS da linha do tempo", "FPS de la línea de tiempo", "时间轴帧率"],
"Megapixels": ["Megapixels", "Megapíxeles", "百万像素"], "Size multiple": ["Múltiplo do tamanho", "Múltiplo del tamaño", "尺寸倍数"],
"✂ Auto split": ["✂ Dividir automaticamente", "✂ División automática", "✂ 自动拆分"], "↻ Re-analyse": ["↻ Analisar novamente", "↻ Volver a analizar", "↻ 重新分析"],
"Reference & prompt": ["Referência e prompt", "Referencia y prompt", "参考图和提示词"], "Who is replaced": ["Quem será substituído", "Quién se reemplaza", "替换对象"],
"Continuity": ["Continuidade", "Continuidad", "连续性"], "Copy to other shots": ["Copiar para outros planos", "Copiar a otros planos", "复制到其他镜头"],
"Global reference & prompt": ["Referência e prompt globais", "Referencia y prompt globales", "全局参考图和提示词"],
"Describe references ({details})": ["Descrever referências ({details})", "Describir referencias ({details})", "描述参考图({details})"],
"VLM": ["VLM", "VLM", "视觉语言模型"], "Use the VLM when the workflow runs": ["Usar o VLM ao executar o workflow", "Usar el VLM al ejecutar el flujo", "运行工作流时使用 VLM"],
"Fill the mask text": ["Preencher o texto da máscara", "Completar el texto de la máscara", "填写蒙版描述"],
"Fill {shot} in the prompt": ["Preencher {shot} no prompt", "Completar {shot} en el prompt", "在提示词中填入 {shot}"],
"Frames per shot": ["Frames por plano", "Fotogramas por plano", "每个镜头的帧数"], "Max tokens": ["Máx. tokens", "Máx. tokens", "最大 token 数"],
"🤖 Analyse shots": ["🤖 Analisar planos", "🤖 Analizar planos", "🤖 分析镜头"], "📝 Describe refs": ["📝 Descrever referências", "📝 Describir referencias", "📝 描述参考图"],
"🎨 Color Guide": ["🎨 Guia de cores", "🎨 Guía de color", "🎨 颜色引导"], "Ref 1 color": ["Cor da ref. 1", "Color de ref. 1", "参考图 1 的颜色"],
"Ref 2": ["Ref. 2", "Ref. 2", "参考图 2"], "+ Add a second person (optional)": ["+ Adicionar segunda pessoa (opcional)", "+ Añadir segunda persona (opcional)", "+ 添加第二个人(可选)"],
"Second person · Ref 2": ["Segunda pessoa · Ref. 2", "Segunda persona · Ref. 2", "第二个人 · 参考图 2"],
"Describe the second person for SAM 3": ["Descreva a segunda pessoa para o SAM 3", "Describe la segunda persona para SAM 3", "为 SAM 3 描述第二个人"],
"👁 Preview": ["👁 Prévia", "👁 Vista previa", "👁 预览"], "More options": ["Mais opções", "Más opciones", "更多选项"],
"Keep ref 2 color consistent across shots (optional)": ["Manter a cor da ref. 2 nos outros planos (opcional)", "Mantener el color de ref. 2 en otros planos (opcional)", "在其他镜头中保持参考图 2 的颜色(可选)"],
"Ref 1 identity": ["Identidade da ref. 1", "Identidad de ref. 1", "参考图 1 的身份"], "Ref 2 identity": ["Identidade da ref. 2", "Identidad de ref. 2", "参考图 2 的身份"],
"Choose manually": ["Escolher manualmente", "Elegir manualmente", "手动选择"], "↩ cast color": ["↩ cor do elenco", "↩ color del reparto", "↩ 使用角色颜色"],
"Opacity": ["Opacidade", "Opacidad", "不透明度"], "The default keeps movement and expression visible.": ["O padrão mantém o movimento e a expressão visíveis.", "El valor predeterminado mantiene visibles el movimiento y la expresión.", "默认值会保留动作和表情。"],
"Full frame": ["Quadro inteiro", "Fotograma completo", "整帧"], "🧩 Frame + paste": ["🧩 Quadro + colar", "🧩 Fotograma + pegar", "🧩 整帧生成并贴回"],
"🎭 Mask only": ["🎭 Somente máscara", "🎭 Solo máscara", "🎭 仅蒙版"], "✂ Crop": ["✂ Recorte", "✂ Recorte", "✂ 裁剪"], "✂🎭 Crop + mask": ["✂🎭 Recorte + máscara", "✂🎭 Recorte + máscara", "✂🎭 裁剪并蒙版"],
"People & masks": ["Pessoas e máscaras", "Personas y máscaras", "人物和蒙版"], "Cast": ["Elenco", "Reparto", "角色"],
"Mask & crop settings": ["Configurações de máscara e recorte", "Ajustes de máscara y recorte", "蒙版和裁剪设置"],
"Color Guide identity color": ["Cor de identidade do Guia de cores", "Color de identidad de la Guía de color", "颜色引导身份颜色"],
"No Color Guide color": ["Sem cor", "Sin color", "无颜色引导颜色"], "Green": ["Verde", "Verde", "绿色"], "Purple": ["Roxo", "Morado", "紫色"],
"Red": ["Vermelho", "Rojo", "红色"], "Blue": ["Azul", "Azul", "蓝色"], "Yellow": ["Amarelo", "Amarillo", "黄色"], "Magenta": ["Magenta", "Magenta", "品红"], "Cyan": ["Ciano", "Cian", "青色"],
"Checks": ["Verificações", "Comprobaciones", "检查"], "Run": ["Executar", "Ejecutar", "运行"], "Filters": ["Filtros", "Filtros", "筛选"],
"Auto loop (one run)": ["Loop automático (uma execução)", "Bucle automático (una ejecución)", "自动循环(一次运行)"],
"Queue loop (one shot per run)": ["Fila (um plano por execução)", "Cola (un plano por ejecución)", "队列循环(每次运行一个镜头)"],
"Skipped shots in the output": ["Planos ignorados na saída", "Planos omitidos en la salida", "输出中跳过的镜头"], "Keep original video": ["Manter vídeo original", "Mantener vídeo original", "保留原视频"],
"Remove them": ["Removê-los", "Quitarlos", "移除"], "Max shots (0 = all)": ["Máx. planos (0 = todos)", "Máx. planos (0 = todos)", "镜头上限(0 = 全部)"],
"Max total seconds (0 = all)": ["Máx. duração total (0 = tudo)", "Duración total máx. (0 = todo)", "总时长上限(0 = 全部)"],
"Auto-queue the next shot": ["Enfileirar próximo plano automaticamente", "Añadir automáticamente el siguiente plano a la cola", "自动将下一个镜头加入队列"],
"Needs a person": ["Precisa ter uma pessoa", "Debe haber una persona", "需要有人物"], "Needs a face": ["Precisa ter um rosto", "Debe haber un rostro", "需要有人脸"],
"Skip dark / fades": ["Ignorar cenas escuras / fades", "Omitir escenas oscuras / fundidos", "跳过暗场/淡入淡出"], "Skip static shots": ["Ignorar planos estáticos", "Omitir planos estáticos", "跳过静止镜头"],
"👤 Analyse people & faces": ["👤 Analisar pessoas e rostos", "👤 Analizar personas y rostros", "👤 分析人物和人脸"],
"Video": ["Vídeo", "Vídeo", "视频"], "Shots": ["Planos", "Planos", "镜头"], "People": ["Pessoas", "Personas", "人物"], "Prompts": ["Prompts", "Prompts", "提示词"],
"Back": ["Voltar", "Atrás", "上一步"], "Next": ["Avançar", "Siguiente", "下一步"], "Close tour": ["Fechar tour", "Cerrar recorrido", "关闭引导"],
"Shot Planner tour": ["Tour do Shot Planner", "Recorrido de Shot Planner", "Shot Planner 引导"], "Shot": ["Plano", "Plano", "镜头"],
"This tour is optional. Use Next to visit each main tab; close it whenever you like.": ["Este tour é opcional. Use Avançar para conhecer cada aba; feche quando quiser.", "Este recorrido es opcional. Pulsa Siguiente para visitar cada pestaña; puedes cerrarlo cuando quieras.", "此引导为可选。点击下一步浏览各个主选项卡,也可随时关闭。"],
"Choose a video, set the frame range, split at camera cuts, then set the generation size.": ["Escolha um vídeo e o trecho, divida nos cortes de câmera e defina o tamanho da geração.", "Elige un vídeo y un rango, divídelo en cortes de cámara y define el tamaño de generación.", "选择视频和范围,按镜头切换拆分,然后设置生成尺寸。"],
"Review each shot, assign its references and prompt, and choose how the result is applied.": ["Revise cada plano, associe referências e prompt e escolha como aplicar o resultado.", "Revisa cada plano, asigna referencias y prompt, y elige cómo aplicar el resultado.", "检查每个镜头,设置参考图和提示词,并选择结果的应用方式。"],
"Link detected people to references. Set masks here or per shot; ArcFace links are a guide and need your review.": ["Associe as pessoas detectadas às referências. Configure máscaras aqui ou por plano; revise os vínculos ArcFace.", "Vincula las personas detectadas con referencias. Configura máscaras aquí o por plano; revisa los enlaces ArcFace.", "将检测到的人物关联到参考图。可在此处或逐镜头设置蒙版;请检查 ArcFace 关联。"],
"Set shared references, prompts and optional VLM assistance. Written prompts are kept exactly as authored.": ["Defina referências, prompts e assistência opcional do VLM. Os prompts escritos são preservados como estão.", "Configura referencias, prompts y asistencia opcional del VLM. Los prompts escritos se conservan tal cual.", "设置共享参考图、提示词和可选的 VLM 辅助。提示词会保持原样。"],
"Check warnings, choose auto or queue loop, and set filters or quick-run limits.": ["Confira avisos, escolha loop automático ou fila e configure filtros e limites de teste.", "Revisa avisos, elige bucle automático o cola y configura filtros y límites de prueba.", "查看警告,选择自动循环或队列,并设置筛选条件和快速运行限制。"],
"Color Guide adds a translucent marker to a selected person. Ref 1 and Ref 2 must use different colors in the same shot.": ["O Guia de cores aplica uma cor translúcida à pessoa selecionada. Ref. 1 e Ref. 2 precisam de cores diferentes no mesmo plano.", "La Guía de color aplica una marca translúcida a la persona seleccionada. Ref. 1 y Ref. 2 deben tener colores distintos en el mismo plano.", "颜色引导会给选中的人物添加半透明颜色。同一镜头中的参考图 1 和参考图 2 必须使用不同颜色。"],
"🎬 Shot Planner": ["🎬 Planejador de planos", "🎬 Planificador de planos", "🎬 镜头规划器"],
"— choose a video from the input folder —": ["— escolha um vídeo da pasta de entrada —", "— elige un vídeo de la carpeta de entrada —", "— 从输入文件夹选择视频 —"],
"⬆ Upload": ["⬆ Enviar", "⬆ Subir", "⬆ 上传"], "Pick or upload a video to start": ["Escolha ou envie um vídeo para começar", "Elige o sube un vídeo para empezar", "选择或上传视频以开始"],
"It is split into shots the model can generate; every shot then gets its own reference, prompt and mask.": ["O vídeo será dividido em planos que o modelo consegue gerar; cada plano terá sua própria referência, prompt e máscara.", "El vídeo se divide en planos que el modelo puede generar; cada uno tendrá su referencia, prompt y máscara.", "视频会拆分为模型可生成的镜头,每个镜头都有自己的参考图、提示词和蒙版。"],
"work on a part of the video": ["trabalhe em um trecho do vídeo", "trabaja en una parte del vídeo", "处理视频的一部分"],
"Start frame": ["Frame inicial", "Fotograma inicial", "起始帧"], "End frame (0 = end)": ["Frame final (0 = fim)", "Fotograma final (0 = final)", "结束帧(0 = 末尾)"],
"⇤ start here": ["⇤ começar aqui", "⇤ empezar aquí", "⇤ 从此处开始"], "⇥ end here": ["⇥ terminar aqui", "⇥ terminar aquí", "⇥ 在此处结束"], "✕ whole video": ["✕ vídeo inteiro", "✕ vídeo completo", "✕ 整个视频"],
"Camera cuts or fixed length, on the model's frame grid": ["Cortes de câmera ou duração fixa, na grade de frames do modelo", "Cortes de cámara o duración fija, según la cuadrícula de fotogramas del modelo", "按镜头切换或固定时长,并符合模型帧步长"],
"re-analyses": ["reanalisar", "vuelve a analizar", "重新分析"], "ignores Max seconds / shot": ["ignora Máx. segundos por plano", "ignora Máx. segundos por plano", "忽略每个镜头的最长秒数"],
"cropped shots use it for the crop": ["planos recortados usam esse valor no recorte", "los planos recortados usan este valor", "裁剪镜头会使用此尺寸"],
"Shots use their main person's reference": ["Usar a referência da pessoa principal do plano", "Usar la referencia de la persona principal del plano", "镜头使用主要人物的参考图"],
"Split where the main person changes": ["Dividir quando a pessoa principal mudar", "Dividir cuando cambie la persona principal", "主要人物变化时拆分"],
"Only run shots with a linked person": ["Executar apenas planos com pessoa vinculada", "Ejecutar solo planos con una persona vinculada", "仅运行已关联人物的镜头"],
"a shot without its own reference takes the reference of the person with the most screen time in it": ["um plano sem referência própria usa a referência da pessoa que aparece por mais tempo nele", "un plano sin referencia propia usa la de la persona que aparece más tiempo", "没有单独参考图的镜头会使用出镜时间最长人物的参考图"],
"adds a boundary when the biggest face switches to someone else (re-splits the plan)": ["cria uma divisão quando o maior rosto muda para outra pessoa (refaz a divisão)", "añade un corte cuando el rostro más grande cambia de persona (vuelve a dividir)", "最大人脸切换到其他人时添加分界(重新拆分计划)"],
"shots where no linked person appears are skipped": ["planos sem pessoa vinculada são ignorados", "se omiten los planos sin una persona vinculada", "跳过没有关联人物的镜头"],
"Nobody found yet.": ["Nenhuma pessoa encontrada.", "Aún no se encontraron personas.", "尚未检测到人物。"], "↻ Re-analyse people": ["↻ Analisar pessoas novamente", "↻ Volver a analizar personas", "↻ 重新分析人物"], "👥 Find people": ["👥 Encontrar pessoas", "👥 Buscar personas", "👥 查找人物"],
"faces sampled every 0.5 s and grouped by identity (InsightFace, CPU)": ["rostos amostrados a cada 0,5 s e agrupados por identidade (InsightFace, CPU)", "rostros muestreados cada 0,5 s y agrupados por identidad (InsightFace, CPU)", "每 0.5 秒采样人脸并按身份分组(InsightFace,CPU)"],
"detects faces across the video and groups them into people": ["detecta rostos no vídeo e os agrupa por pessoa", "detecta rostros en el vídeo y los agrupa por persona", "检测视频中的人脸并按人物分组"],
"track faces, link each person to a reference, split and assign shots by who is on screen": ["rastreie rostos, associe cada pessoa a uma referência e divida os planos por quem aparece", "rastrea rostros, vincula cada persona a una referencia y divide los planos según quién aparece", "跟踪人脸、关联参考图,并按出镜人物拆分镜头"],
"Each shot picks what it marks (🎯 text or points, or 🎞 its own mask video) and how it is generated (": ["Cada plano define a seleção (🎯 texto, pontos ou 🎞 vídeo de máscara) e o modo de geração (", "Cada plano define la selección (🎯 texto, puntos o 🎞 vídeo de máscara) y el modo de generación (", "每个镜头设置选区(🎯 文本、点或 🎞 蒙版视频)和生成方式("],
"🎞 Mask video (whole video)": ["🎞 Vídeo de máscara (vídeo inteiro)", "🎞 Vídeo de máscara (vídeo completo)", "🎞 蒙版视频(整个视频)"],
"— none: SAM 3 per shot —": ["— nenhum: SAM 3 por plano —", "— ninguno: SAM 3 por plano —", "— 无:每个镜头使用 SAM 3 —"], "Fill holes": ["Preencher buracos", "Rellenar huecos", "填充孔洞"], "Invert": ["Inverter", "Invertir", "反转"],
"Show masks on shots": ["Mostrar máscaras nos planos", "Mostrar máscaras en los planos", "在镜头卡片上显示蒙版"], "Expand (px)": ["Expandir (px)", "Expandir (px)", "扩展(像素)"],
"Temporal expand (frames)": ["Expansão temporal (frames)", "Expansión temporal (fotogramas)", "时间扩展(帧)"], "Detection threshold": ["Limiar de detecção", "Umbral de detección", "检测阈值"], "Max objects": ["Máx. objetos", "Máx. objetos", "最大对象数"],
"Crop padding (%)": ["Margem do recorte (%)", "Margen del recorte (%)", "裁剪边距(%)"], "Feather (px)": ["Suavização da borda (px)", "Suavizado del borde (px)", "羽化(像素)"], "Paste back": ["Colar de volta", "Pegar de vuelta", "贴回方式"],
"only the mask (feathered)": ["somente a máscara (suavizada)", "solo la máscara (suavizada)", "仅蒙版(羽化边缘)"], "the whole box (feathered)": ["a caixa inteira (suavizada)", "todo el cuadro (suavizado)", "整个框(羽化边缘)"], "Blockify (px, 0 = off)": ["Blocos (px, 0 = desligado)", "Bloques (px, 0 = desactivado)", "块化(像素,0 = 关闭)"],
"Uses the official sam3.1_multiplex_fp16 checkpoint (models/checkpoints), downloaded from Comfy-Org/sam3.1 the first time.": ["Usa o checkpoint oficial sam3.1_multiplex_fp16 (models/checkpoints), baixado de Comfy-Org/sam3.1 na primeira utilização.", "Usa el checkpoint oficial sam3.1_multiplex_fp16 (models/checkpoints), descargado de Comfy-Org/sam3.1 la primera vez.", "使用官方 sam3.1_multiplex_fp16 检查点(models/checkpoints),首次运行时从 Comfy-Org/sam3.1 下载。"],
"SAM 3": ["SAM 3", "SAM 3", "SAM 3"], "track faces, link each person to a reference, split and assign shots by who is on screen": ["rastreie rostos, associe cada pessoa a uma referência e organize os planos por quem aparece", "rastrea rostros, vincula cada persona a una referencia y organiza los planos según quién aparece", "跟踪人脸、关联参考图,并按出镜人物分配镜头"],
"Global reference & prompt": ["Referência e prompt globais", "Referencia y prompt globales", "全局参考图和提示词"], "defaults for every shot": ["padrão para todos os planos", "valor predeterminado para cada plano", "所有镜头的默认设置"],
"who the shot becomes": ["em quem o plano será transformado", "en quién se transforma el plano", "镜头中的人物将替换为谁"], "Prompt for this shot (empty = global prompt). Placeholders: {target} {details} {shot} {setting}": ["Prompt deste plano (vazio = prompt global). Variáveis: {target} {details} {shot} {setting}", "Prompt de este plano (vacío = prompt global). Variables: {target} {details} {shot} {setting}", "此镜头的提示词(留空则使用全局提示词)。占位符:{target} {details} {shot} {setting}"],
"recent (click = ref, shift+click = ref 2):": ["recentes (clique = ref., Shift+clique = ref. 2):", "recientes (clic = ref., Mayús+clic = ref. 2):", "最近使用(点击 = 参考图,Shift+点击 = 参考图 2):"], "✕ clear": ["✕ limpar", "✕ borrar", "✕ 清除"], "✕ clear all": ["✕ limpar tudo", "✕ borrar todo", "✕ 全部清除"], "Use global": ["Usar global", "Usar global", "使用全局设置"],
"empty slots use the person's reference (Cast) or the global one (Prompts & refs tab)": ["campos vazios usam a referência da pessoa (Elenco) ou a global (aba Prompts e referências)", "los campos vacíos usan la referencia de la persona (Reparto) o la global (Prompts y referencias)", "空白项使用人物参考图(角色)或全局参考图(提示词和参考图选项卡)"],
"Full frame": ["Quadro inteiro", "Fotograma completo", "整帧"], "The whole frame is regenerated. The mask, if any, only feeds {target} / the setting picture.": ["O quadro inteiro é gerado novamente. A máscara, se houver, só fornece {target} / a imagem de cenário.", "Se vuelve a generar todo el fotograma. La máscara, si existe, solo aporta {target} / la imagen del escenario.", "重新生成整帧。蒙版(如有)仅用于提供 {target} / 场景参考图。"],
"Only the mask is regenerated, on the whole frame: the rest stays exactly as it was. No crop / uncrop.": ["Somente a área da máscara é regenerada; o restante permanece igual. Sem recorte.", "Solo se regenera el área de la máscara; el resto permanece igual. Sin recorte.", "仅重新生成蒙版区域,其余部分保持不变。不进行裁剪。"],
"A box around the mask is regenerated (more pixels for a small subject); BFS Shot Join pastes it back.": ["A caixa ao redor da máscara é regenerada (mais pixels para pessoas pequenas); o BFS Shot Join cola o resultado de volta.", "Se regenera el cuadro alrededor de la máscara (más píxeles para sujetos pequeños); BFS Shot Join lo pega de vuelta.", "重新生成蒙版周围的区域(为较小主体提供更多像素);BFS Shot Join 会将其贴回。"],
"Inside the crop, only the mask is regenerated: the most detail with the background kept.": ["Dentro do recorte, somente a máscara é regenerada: mais detalhes e fundo preservado.", "Dentro del recorte, solo se regenera la máscara: más detalle y fondo conservado.", "仅重新生成裁剪区域内的蒙版:细节更多,同时保留背景。"],
"The whole frame is generated (the pose follows the guide, the new subject may be bigger); BFS Shot Join pastes only the subject onto the original: background exact.": ["O quadro inteiro é gerado (a pose segue o guia e o novo sujeito pode ser maior); o BFS Shot Join cola apenas o sujeito no original e preserva o fundo.", "Se genera todo el fotograma (la pose sigue la guía y el sujeto puede ser más grande); BFS Shot Join pega solo al sujeto sobre el original y conserva el fondo.", "生成整帧(姿势遵循引导,主体可更大);BFS Shot Join 仅将主体贴回原画面,背景保持原样。"],
"🎯 Points…": ["🎯 Pontos…", "🎯 Puntos…", "🎯 点…"], "👁 Preview": ["👁 Prévia", "👁 Vista previa", "👁 预览"], "✕ Clear": ["✕ Limpar", "✕ Borrar", "✕ 清除"],
"🎞 mask video": ["🎞 vídeo de máscara", "🎞 vídeo de máscara", "🎞 蒙版视频"], "↳ global prompt": ["↳ prompt global", "↳ prompt global", "↳ 全局提示词"], "— no prompt —": ["— sem prompt —", "— sin prompt —", "— 无提示词 —"],
"▶ Play all": ["▶ Reproduzir todos", "▶ Reproducir todos", "▶ 播放全部"], "■ Stop": ["■ Parar", "■ Detener", "■ 停止"], "loop shot": ["repetir plano", "repetir plano", "循环播放镜头"],
"Click the timeline to seek. Play all skips shots that will not run.": ["Clique na linha do tempo para buscar. Reproduzir todos ignora planos que não serão executados.", "Haz clic en la línea de tiempo para buscar. Reproducir todos omite los planos que no se ejecutarán.", "点击时间轴进行定位。播放全部会跳过不运行的镜头。"],
"Reference & prompt": ["Referência e prompt", "Referencia y prompt", "参考图和提示词"], "Who is replaced": ["Quem será substituído", "Quién se reemplaza", "替换对象"], "Continuity": ["Continuidade", "Continuidad", "连续性"],
"Global reference & prompt": ["Referência e prompt globais", "Referencia y prompt globales", "全局参考图和提示词"], "Describe references ({details})": ["Descrever referências ({details})", "Describir referencias ({details})", "描述参考图({details})"],
"Custom": ["Personalizado", "Personalizado", "自定义"], "short": ["curto", "breve", "简短"], "full body": ["corpo inteiro", "cuerpo entero", "全身"], "head / face": ["cabeça / rosto", "cabeza / rostro", "头部/面部"], "face attributes": ["características do rosto", "rasgos faciales", "面部特征"], "outfit": ["roupa", "ropa", "服装"],
"Apply suggestions → all": ["Aplicar sugestões → todos", "Aplicar sugerencias → todos", "应用建议 → 全部"], "fit the prompt": ["ajustar ao prompt", "ajustar al prompt", "适配提示词"],
"Connect a Qwen3-VL (CLIPLoader) to the planner's vlm input. The VLM buttons work after the workflow has run once with it connected (ComfyUI only hands models to nodes when they run). Each shot's suggestion shows in its editor.": ["Conecte um Qwen3-VL (CLIPLoader) à entrada vlm do planner. Os botões do VLM funcionam depois que o workflow executar uma vez com ele conectado. A sugestão de cada plano aparece no editor.", "Conecta un Qwen3-VL (CLIPLoader) a la entrada vlm del planificador. Los botones del VLM funcionan después de ejecutar el flujo una vez con él conectado. La sugerencia de cada plano aparece en su editor.", "将 Qwen3-VL(CLIPLoader)连接到规划器的 vlm 输入。工作流连接后运行一次,VLM 按钮即可使用。每个镜头的建议会显示在其编辑器中。"],
"look at every shot and suggests the mask, a description and whether to run it": ["analisa cada plano e sugere máscara, descrição e se deve ser executado", "analiza cada plano y sugiere una máscara, una descripción y si debe ejecutarse", "分析每个镜头并建议蒙版、描述及是否运行"],
"🎯 Track → whole video": ["🎯 Rastrear → vídeo inteiro", "🎯 Seguir → vídeo completo", "🎯 跟踪 → 整个视频"], "Save (this shot)": ["Salvar (este plano)", "Guardar (este plano)", "保存(此镜头)"], "Save → all shots": ["Salvar → todos os planos", "Guardar → todos los planos", "保存 → 所有镜头"], "Cancel": ["Cancelar", "Cancelar", "取消"], "Clear points": ["Limpar pontos", "Borrar puntos", "清除点"], "👁 Segment": ["👁 Segmentar", "👁 Segmentar", "👁 分割"],
"Apply these points to EVERY shot, each on its frame at the same relative position (a subject that stays in place)": ["Aplicar estes pontos a TODOS os planos, na mesma posição relativa (para um sujeito que não muda de lugar)", "Aplicar estos puntos a TODOS los planos, en la misma posición relativa (si el sujeto no se mueve)", "将这些点应用到每个镜头的相同相对位置(适用于位置不变的主体)"],
"the range starts at the player's frame": ["o trecho começa no frame atual do player", "el rango empieza en el fotograma actual", "范围从播放器当前帧开始"], "the range ends after the player's frame": ["o trecho termina após o frame atual do player", "el rango termina después del fotograma actual", "范围在播放器当前帧之后结束"],
"Nothing to fix in the shots that run.": ["Nenhum problema nos planos que serão executados.", "No hay nada que corregir en los planos que se ejecutarán.", "将运行的镜头没有需要修复的问题。"], "all good": ["tudo certo", "todo bien", "一切正常"], "warning(s)": ["aviso(s)", "aviso(s)", "条警告"], "note(s)": ["observação(ões)", "nota(s)", "条备注"],
"quick tests": ["testes rápidos", "pruebas rápidas", "快速测试"], "from the start of the range": ["a partir do início do trecho", "desde el inicio del rango", "从范围起点开始"], "Each run generates one shot and stores it. Nodes after BFS Shot Join only run on the last shot, with the full video.": ["Cada execução gera e salva um plano. Os nodes após BFS Shot Join só executam no último plano, com o vídeo completo.", "Cada ejecución genera y guarda un plano. Los nodos posteriores a BFS Shot Join solo se ejecutan en el último plano, con el vídeo completo.", "每次运行生成并保存一个镜头。BFS Shot Join 后的节点仅在最后一个镜头运行,并接收完整视频。"],
"skipped shots do not run": ["planos ignorados não são executados", "los planos omitidos no se ejecutan", "跳过的镜头不会运行"], "only shots that run": ["somente planos executados", "solo los planos que se ejecutan", "仅统计将运行的镜头"],
"off": ["desligado", "desactivado", "关闭"], "on": ["ligado", "activado", "开启"], "run": ["executar", "ejecutar", "运行"], "skip": ["ignorar", "omitir", "跳过"], "filters decide": ["filtros decidem", "deciden los filtros", "由筛选条件决定"], "always run": ["sempre executar", "ejecutar siempre", "始终运行"], "always skip": ["sempre ignorar", "omitir siempre", "始终跳过"], "⏸ Disable": ["⏸ Desativar", "⏸ Desactivar", "⏸ 禁用"], "▶ Enable": ["▶ Ativar", "▶ Activar", "▶ 启用"], "✂ Split": ["✂ Dividir", "✂ Dividir", "✂ 拆分"], "⇥ Merge": ["⇥ Unir", "⇥ Unir", "⇥ 合并"],
"Advanced": ["Avançado", "Avanzado", "高级"], "English": ["Inglês", "Inglés", "英语"], "Português": ["Português", "Portugués", "葡萄牙语"], "Español": ["Espanhol", "Español", "西班牙语"], "中文": ["Chinês", "Chino", "中文"],
"Color Guide": ["Guia de cores", "Guía de color", "颜色引导"], "Video": ["Vídeo", "Vídeo", "视频"], "Shots": ["Planos", "Planos", "镜头"], "Prompts": ["Prompts", "Prompts", "提示词"],
"Guide color · Ref 1": ["Cor guia · Ref. 1", "Color guía · Ref. 1", "引导颜色 · 参考图 1"], "Guide color · Ref 2": ["Cor guia · Ref. 2", "Color guía · Ref. 2", "引导颜色 · 参考图 2"],
"These colors are defaults for the global references. Turn on Color Guide on a shot to apply them; shot-level colors can override them.": ["Estas cores são os padrões das referências globais. Ative o Guia de cores no plano para aplicá-las; cada plano pode substituí-las.", "Estos colores son los valores predeterminados de las referencias globales. Activa la Guía de color en un plano para aplicarlos; cada plano puede cambiarlos.", "这些颜色是全局参考图的默认值。启用镜头的颜色引导即可应用;镜头也可以单独覆盖。"],
"Audio": ["Áudio", "Audio", "音频"], "use the video's audio": ["usar o áudio do vídeo", "usar el audio del vídeo", "使用视频音频"], "silent track": ["faixa silenciosa", "pista silenciosa", "静音音轨"],
"auto": ["automático", "automático", "自动"], "fixed": ["fixo", "fijo", "固定"], "single": ["único", "único", "单个"], "adaptive": ["adaptativo", "adaptativo", "自适应"], "content": ["conteúdo", "contenido", "内容"], "builtin": ["integrado", "integrado", "内置"],
"PySceneDetect adaptive": ["PySceneDetect adaptativo", "PySceneDetect adaptativo", "PySceneDetect 自适应"], "PySceneDetect content": ["PySceneDetect por conteúdo", "PySceneDetect por contenido", "PySceneDetect 内容检测"], "Built-in": ["Integrado", "Integrado", "内置"],
"Auto loop": ["Loop automático", "Bucle automático", "自动循环"], "Queue loop": ["Loop em fila", "Bucle en cola", "队列循环"], "queue loop": ["loop em fila", "bucle en cola", "队列循环"], "auto loop": ["loop automático", "bucle automático", "自动循环"],
"no audio": ["sem áudio", "sin audio", "无音频"], "silent track": ["faixa silenciosa", "pista silenciosa", "静音音轨"], "timeline": ["linha do tempo", "línea de tiempo", "时间轴"], "generate at": ["gerar em", "generar a", "生成尺寸"],
"Max person size (0-1 of frame)": ["Tamanho mínimo da pessoa (0–1 do quadro)", "Tamaño mínimo de persona (0–1 del fotograma)", "人物最小尺寸(占画面 0–1)"], "Min person size (0-1 of frame)": ["Tamanho mínimo da pessoa (0–1 do quadro)", "Tamaño mínimo de persona (0–1 del fotograma)", "人物最小尺寸(占画面 0–1)"],
"Max people (0 = any)": ["Máx. pessoas (0 = qualquer)", "Máx. personas (0 = cualquiera)", "人物上限(0 = 不限)"], "Min frames (0 = off)": ["Mín. frames (0 = desligado)", "Mín. fotogramas (0 = desactivado)", "最少帧数(0 = 关闭)"], "Frames sampled / shot": ["Frames amostrados por plano", "Fotogramas muestreados por plano", "每个镜头采样帧数"], "Dark below (0-1)": ["Escuro abaixo de (0–1)", "Oscuro por debajo de (0–1)", "暗度低于(0–1)"], "Static below": ["Estático abaixo de", "Estático por debajo de", "静止度低于"],
"e.g. 0.03 skips wide shots": ["ex.: 0,03 ignora planos abertos", "p. ej., 0,03 omite planos amplios", "例如 0.03 会跳过远景镜头"], "skip crowds": ["ignorar multidões", "omitir multitudes", "跳过人群"], "re-analyses": ["reanalisar", "vuelve a analizar", "重新分析"],
"Describe target": ["Descrever alvo", "Describir objetivo", "描述目标"], "🧑 Describe target": ["🧑 Descrever alvo", "🧑 Describir objetivo", "🧑 描述目标"], "new subject": ["novo sujeito", "nuevo sujeto", "新主体"], "person (or dog, cat, car… what the result shows)": ["pessoa (ou cão, gato, carro… o que aparecerá no resultado)", "persona (o perro, gato, coche… lo que aparecerá en el resultado)", "人物(或狗、猫、汽车等结果中的主体)"],
"The second person (optional)": ["Segunda pessoa (opcional)", "Segunda persona (opcional)", "第二个人(可选)"], "Follow cast": ["Seguir identidade", "Seguir reparto", "跟随角色"], "manual identity": ["identidade manual", "identidad manual", "手动身份"],
"Guide video": ["Vídeo guia", "Vídeo guía", "引导视频"], "Reference": ["Referência", "Referencia", "参考图"], "Prompt": ["Prompt", "Prompt", "提示词"], "reference": ["referência", "referencia", "参考图"],
"on": ["ativado", "activado", "开启"], "off": ["desativado", "desactivado", "关闭"], "not analysed": ["não analisado", "sin analizar", "未分析"], "not analyzed": ["não analisado", "sin analizar", "未分析"],
"No shots yet": ["Nenhum plano ainda", "Aún no hay planos", "尚无镜头"], "No video": ["Nenhum vídeo", "Sin vídeo", "没有视频"], "No faces": ["Nenhum rosto", "Sin rostros", "没有人脸"], "no faces": ["sem rostos", "sin rostros", "没有人脸"],
"cut": ["corte", "corte", "切换"], "run": ["executar", "ejecutar", "运行"], "skip": ["ignorar", "omitir", "跳过"], "skip this shot": ["ignorar este plano", "omitir este plano", "跳过此镜头"],
"people": ["pessoas", "personas", "人物"], "person": ["pessoa", "persona", "人物"], "shots": ["planos", "planos", "镜头"], "shot(s)": ["plano(s)", "plano(s)", "个镜头"], "linked": ["vinculada", "vinculada", "已关联"], "main": ["principal", "principal", "主要"],
"No VLM yet": ["Nenhum VLM carregado", "Aún no hay un VLM", "尚未加载 VLM"], "Connect a Qwen3-VL": ["Conecte um Qwen3-VL", "Conecta un Qwen3-VL", "连接 Qwen3-VL"],
"↻ Status": ["↻ Status", "↻ Estado", "↻ 状态"], "⟲ Reset loop": ["⟲ Reiniciar loop", "⟲ Reiniciar bucle", "⟲ 重置循环"], "Queue loop": ["Loop em fila", "Bucle en cola", "队列循环"],
"Start here": ["Começar aqui", "Empezar aquí", "从此处开始"], "End here": ["Terminar aqui", "Terminar aquí", "在此处结束"], "reset": ["redefinir", "restablecer", "重置"], "all shots": ["todos os planos", "todos los planos", "所有镜头"], "this shot": ["este plano", "este plano", "此镜头"],
"Describe target": ["Descrever alvo", "Describir objetivo", "描述目标"], "Settings": ["Configurações", "Ajustes", "设置"], "mask text": ["texto da máscara", "texto de máscara", "蒙版描述"],
"The global reference and prompt are used when a shot has no values of its own.": ["A referência e o prompt globais são usados quando um plano não tem valores próprios.", "La referencia y el prompt globales se usan cuando un plano no tiene valores propios.", "镜头没有单独设置时会使用全局参考图和提示词。"],
"A VLM is optional": ["O VLM é opcional", "El VLM es opcional", "VLM 为可选项"], "Optional": ["Opcional", "Opcional", "可选"], "optional": ["opcional", "opcional", "可选"],
"person in white": ["pessoa de branco", "persona de blanco", "穿白衣的人"], "woman in pink top": ["mulher de blusa rosa", "mujer con blusa rosa", "穿粉色上衣的女人"],
"nothing found": ["nada encontrado", "no se encontró nada", "未找到内容"], "not found": ["não encontrado", "no encontrado", "未找到"], "Preview": ["Prévia", "Vista previa", "预览"],
"Mask only / Crop + mask": ["Somente máscara / Recorte + máscara", "Solo máscara / Recorte + máscara", "仅蒙版 / 裁剪并蒙版"], "per shot (planner)": ["por plano (planner)", "por plano (planificador)", "逐镜头(规划器)"],
"Person": ["Pessoa", "Persona", "人物"], "shot": ["plano", "plano", "镜头"], "shot(s)": ["plano(s)", "plano(s)", "个镜头"],
"mask covers": ["máscara cobre", "la máscara cubre", "蒙版覆盖"], "nothing to fix": ["nada a corrigir", "nada que corregir", "无需修复"],
"Camera cuts: shots start at the detected cuts, long ones are split evenly. Fixed length: equal parts no longer than the maximum.": ["Cortes de câmera: os planos começam nos cortes detectados; planos longos são divididos igualmente. Duração fixa: partes iguais limitadas pelo máximo.", "Cortes de cámara: los planos empiezan en los cortes detectados; los largos se dividen por igual. Duración fija: partes iguales con el máximo indicado.", "镜头切换:镜头从检测到的切点开始,较长镜头会均匀拆分。固定时长:拆分为不超过上限的等长片段。"],
"Cut detector. PySceneDetect adaptive works on most footage (fast camera moves don't trigger it); content is stricter on hard cuts; built-in needs no package.": ["Detector de cortes. O modo adaptativo do PySceneDetect funciona na maioria dos vídeos (movimentos rápidos de câmera não acionam cortes); o modo conteúdo é mais rigoroso; o integrado não exige pacote.", "Detector de cortes. El modo adaptativo de PySceneDetect funciona en la mayoría de vídeos (los movimientos rápidos no activan cortes); contenido es más estricto; el integrado no requiere paquetes.", "镜头切换检测器。PySceneDetect 自适应模式适用于大多数素材(快速运镜不会触发切点);内容模式对硬切更严格;内置模式无需额外软件包。"],
"How easily a change counts as a cut. Higher finds more cuts (and more false ones).": ["O quanto uma mudança é considerada corte. Um valor maior encontra mais cortes e também mais falsos positivos.", "Facilidad con la que un cambio cuenta como corte. Un valor mayor detecta más cortes y también más falsos positivos.", "变化被判定为切点的难易程度。数值越高,检测到的切点越多,误报也越多。"],
"Frame counts the model accepts. H3: 17n+5, LTX/Wan: 8n+1, Wan: 4n+1. Each shot is generated at the next valid length and trimmed back.": ["Contagens de frames aceitas pelo modelo. H3: 17n+5; LTX/Wan: 8n+1; Wan: 4n+1. Cada plano é gerado no próximo comprimento válido e depois aparado.", "Cantidad de fotogramas que acepta el modelo. H3: 17n+5; LTX/Wan: 8n+1; Wan: 4n+1. Cada plano se genera con la siguiente duración válida y luego se recorta.", "模型接受的帧数规则。H3:17n+5;LTX/Wan:8n+1;Wan:4n+1。每个镜头按下一个有效长度生成后再裁短。"],
"Longest shot sent to the model. Keep it within what the model / LoRA was trained on (H3 body swap: about 4.5 s).": ["Duração máxima enviada ao modelo. Mantenha dentro do que o modelo/LoRA aprendeu (body swap H3: cerca de 4,5 s).", "Duración máxima enviada al modelo. Respeta lo usado en el entrenamiento del modelo/LoRA (body swap H3: unos 4,5 s).", "发送给模型的镜头最长时长。应符合模型/LoRA 的训练长度(H3 换人约 4.5 秒)。"],
"Shorter shots merge into a neighbour when the merge still fits the maximum.": ["Planos mais curtos são unidos ao vizinho quando a junção ainda respeita o máximo.", "Los planos más cortos se unen al vecino si la unión respeta el máximo.", "较短镜头会在合并后仍未超过上限时与相邻镜头合并。"],
"The planner works on this frame rate (the source is resampled). Generated video and audio use it too.": ["O planner usa esta taxa de frames (o vídeo de origem é reamostrado). O vídeo e áudio gerados também a usam.", "El planificador usa esta frecuencia de fotogramas (el origen se remuestrea). El vídeo y el audio generados también la usan.", "规划器使用此帧率(源视频会重新采样)。生成的视频和音频也会使用此帧率。"],
"Only the first N shots (0 = all). Handy for quick tests.": ["Somente os primeiros N planos (0 = todos). Útil para testes rápidos.", "Solo los primeros N planos (0 = todos). Útil para pruebas rápidas.", "仅处理前 N 个镜头(0 = 全部)。适合快速测试。"],
"Only the first N seconds of the video (0 = all).": ["Somente os primeiros N segundos do vídeo (0 = todos).", "Solo los primeros N segundos del vídeo (0 = todos).", "仅处理视频前 N 秒(0 = 全部)。"],
"Generation size: the source aspect ratio at this pixel area (0.15 ≈ 512×288). Cropped shots use it for the crop.": ["Tamanho da geração: mantém a proporção do vídeo nessa área de pixels (0,15 ≈ 512×288). Planos recortados usam esse valor no recorte.", "Tamaño de generación: mantiene la proporción de origen con esta área de píxeles (0,15 ≈ 512×288). Los planos recortados usan este valor.", "生成尺寸:按此像素面积保持源视频宽高比(0.15 ≈ 512×288)。裁剪镜头使用此尺寸。"],
"Width and height snap to this multiple (32 for H3 and most video models).": ["Largura e altura são ajustadas a este múltiplo (32 para H3 e a maioria dos modelos de vídeo).", "El ancho y alto se ajustan a este múltiplo (32 para H3 y la mayoría de modelos de vídeo).", "宽度和高度会对齐到此倍数(H3 和多数视频模型使用 32)。"],
"Auto loop: every shot in one queue run. Queue loop: one shot per run, stored on disk, the join outputs the video after the last shot.": ["Loop automático: todos os planos em uma execução da fila. Loop em fila: um plano por execução, salvo em disco; o join gera o vídeo após o último plano.", "Bucle automático: todos los planos en una ejecución. Bucle en cola: un plano por ejecución, guardado en disco; el join genera el vídeo al terminar el último.", "自动循环:一次队列运行所有镜头。队列循环:每次运行一个镜头并保存到磁盘;最后一个镜头完成后由 Join 输出视频。"],
"Shots that do not run: keep the original video there, or remove them (with their audio).": ["Planos não executados: manter o vídeo original nesses trechos ou removê-los junto com o áudio.", "Planos que no se ejecutan: conserva el vídeo original en ese tramo o elimínalo junto con su audio.", "未运行的镜头:保留对应原视频片段,或连同音频一起移除。"],
"Context kept around the mask's box, as a % of the box. More context blends better; less gives the model more pixels on the subject.": ["Contexto mantido ao redor da caixa da máscara, como porcentagem da caixa. Mais contexto mistura melhor; menos dá mais pixels ao sujeito.", "Contexto alrededor del cuadro de la máscara, como porcentaje. Más contexto mezcla mejor; menos ofrece más píxeles al sujeto.", "蒙版边框周围保留的上下文比例。上下文越多越容易融合;越少则主体像素越多。"],
"Grow the mask by this many pixels: before pasting a cropped result back, and for the generation mask (inpaint 'only the mask'). Raise it when the new person is bigger than the old one (longer hair, wider body).": ["Expande a máscara por esta quantidade de pixels antes de colar o recorte e na máscara de geração. Aumente se a nova pessoa for maior (cabelo mais comprido, corpo mais largo).", "Amplía la máscara esta cantidad de píxeles antes de pegar el recorte y para la máscara de generación. Auméntala si la persona nueva es más grande.", "粘贴裁剪结果前以及生成蒙版时,将蒙版扩展指定像素。新人物更大时(头发更长、身体更宽)可调高。"],
"Soft edge of the paste-back, in pixels.": ["Suavização da borda ao colar de volta, em pixels.", "Suavizado del borde al pegar de vuelta, en píxeles.", "贴回时边缘的柔化像素数。"],
"Hold the mask this many frames before and after: removes flicker where SAM 3 drops a frame.": ["Estende a máscara por esta quantidade de frames antes e depois; reduz falhas quando o SAM 3 perde um frame.", "Mantiene la máscara estos fotogramas antes y después; reduce parpadeos cuando SAM 3 pierde uno.", "蒙版前后延续指定帧数,减少 SAM 3 漏掉某帧时的闪烁。"],
"Square blocks of this size (0 = off). 16 matches H3's latent grid, useful as an inpainting mask.": ["Blocos quadrados deste tamanho (0 = desligado). 16 corresponde à grade latente do H3 e pode ser útil como máscara de inpainting.", "Bloques cuadrados de este tamaño (0 = desactivado). 16 coincide con la cuadrícula latente de H3 y sirve para inpainting.", "使用此尺寸的方块(0 = 关闭)。16 与 H3 潜空间网格匹配,可用于修补蒙版。"],
"SAM 3 score needed to keep a text match. Lower finds more (and more wrong) objects.": ["Pontuação mínima do SAM 3 para manter uma correspondência por texto. Valores menores encontram mais objetos, inclusive incorretos.", "Puntuación mínima de SAM 3 para conservar una coincidencia de texto. Un valor menor encuentra más objetos, también erróneos.", "保留文本匹配所需的 SAM 3 分数。调低会找到更多对象,也会增加误检。"],
"How many matches of the text prompt are tracked together.": ["Quantas correspondências do texto serão rastreadas juntas.", "Cuántas coincidencias del texto se siguen a la vez.", "同时跟踪多少个符合文本的对象。"],
"Only the mask: the result replaces the masked pixels. The whole box: the full crop is pasted with feathered borders.": ["Somente máscara: o resultado substitui os pixels mascarados. Caixa inteira: todo o recorte é colado com bordas suaves.", "Solo máscara: el resultado sustituye los píxeles enmascarados. Cuadro completo: se pega todo el recorte con bordes suavizados.", "仅蒙版:结果替换蒙版像素。整个框:粘贴完整裁剪区域并柔化边缘。"],
"How many frames of each shot the VLM looks at.": ["Quantos frames de cada plano o VLM analisa.", "Cuántos fotogramas de cada plano analiza el VLM.", "VLM 分析每个镜头的帧数。"], "Longest VLM answer. Raise it for long descriptions.": ["Tamanho máximo da resposta do VLM. Aumente para descrições longas.", "Longitud máxima de respuesta del VLM. Auméntala para descripciones largas.", "VLM 回复的最大长度。长描述可调高。"],
};
export function t(value) {
const s = String(value ?? ""), lang = uiLanguage.value;
if (lang === "en") return s;
const entry = TEXT[s];
if (entry) return entry[{ pt: 0, es: 1, zh: 2 }[lang]];
let m;
if ((m = s.match(/^Person (\d+)$/))) return `${t("Person")} ${m[1]}`;
if ((m = s.match(/^Shot #(\d+)( · points)?$/))) return `${t("Shot")} #${m[1]}${m[2] ? ` · ${lang === "pt" ? "pontos" : lang === "es" ? "puntos" : "点"}` : ""}`;
if ((m = s.match(/^(\d+) people · (\d+) linked$/))) return lang === "pt" ? `${m[1]} pessoas · ${m[2]} vinculadas` : lang === "es" ? `${m[1]} personas · ${m[2]} vinculadas` : `${m[1]} 人物 · 已关联 ${m[2]} 人`;
if ((m = s.match(/^(\d+) shot\(s\) will run$/))) return lang === "pt" ? `${m[1]} plano(s) serão executados` : lang === "es" ? `se ejecutarán ${m[1]} plano(s)` : `将运行 ${m[1]} 个镜头`;
if ((m = s.match(/^(\d+) suggestions$/))) return lang === "pt" ? `${m[1]} sugestões` : lang === "es" ? `${m[1]} sugerencias` : `${m[1]} 条建议`;
if ((m = s.match(/^(\d+) shot\(s\)$/))) return lang === "pt" ? `${m[1]} plano(s)` : lang === "es" ? `${m[1]} plano(s)` : `${m[1]} 个镜头`;
if ((m = s.match(/^(\d+) warning\(s\) · (\d+) note\(s\)$/))) return lang === "pt" ? `${m[1]} aviso(s) · ${m[2]} observação(ões)` : lang === "es" ? `${m[1]} aviso(s) · ${m[2]} nota(s)` : `${m[1]} 条警告 · ${m[2]} 条备注`;
return s;
}
function localizeProps(props) {
if (!props || typeof props !== "object") return props;
let changed = false; const out = { ...props };
for (const k of ["title", "placeholder", "aria-label"]) if (typeof out[k] === "string") { const v = t(out[k]); changed ||= v !== out[k]; out[k] = v; }
return changed ? out : props;
}
function localizeChildren(value) {
if (typeof value === "string") return t(value);
if (Array.isArray(value)) return value.map(localizeChildren);
return value;
}
// All view modules use this h wrapper, so labels, placeholders, tooltips and section titles stay in sync.
export function h(type, props, children) {
const hasChildrenArg = arguments.length >= 3;
if (!hasChildrenArg && props?.__v_isVNode) return vueH(type, null, props);
const p = localizeProps(props);
if (hasChildrenArg) return vueH(type, p, localizeChildren(children));
if (typeof props === "string" || Array.isArray(props)) return vueH(type, localizeChildren(props));
return vueH(type, p);
}
export const GRIDS = { "H3 (17n+5)": [17, 5], "LTX / Wan (8n+1)": [8, 1], "Wan (4n+1)": [4, 1], "any": [1, 0] };
export const DEFAULTS = {
video: "", fps: 24, grid: "H3 (17n+5)", mode: "shots", max_s: 4.5, min_s: 1.0, sensitivity: 0.5,
max_parts: 0, max_total_s: 0, start_s: 0, end_s: 0, start_f: null, end_f: null, track: null, bounds: [], segs: [], global_ref: "", global_ref2: "", global_prompt: "", global_color_guide_ref1: "#00ff00", global_color_guide_ref2: "#9b30ff", mask_video: "",
megapixels: 0.15, multiple: 32, detector: "adaptive", run: "auto", auto_continue: true,
skip_fill: "original", audio_mode: "auto", ref_details: {}, mask_cfg: {}, vlm_cfg: {}, cast: {}, cast_assign: true, cast_split: false, cast_only: false,
filters: { person: false, min_person_area: 0, max_persons: 0, face: false, skip_dark: false, dark_level: 0.06,
skip_static: false, static_level: 0.004, min_frames: 0, samples: 6 },
};
// global mask settings (same defaults as the server's DEFAULT_MASK)
export const MASK_DEFAULTS = { threshold: 0.5, max_objects: 4, invert: false, fill_holes: true, temporal_expand: 2, blockify: 0,
padding: 0.15, expand: 16, feather: 12, paste: "mask" };
export const VLM_DEFAULTS = { enabled: false, frames: 3, max_tokens: 1024, auto_segment: true, auto_shot: true, instruction: "",
describe_preset: "short", describe_custom: "", fit_prompt: true, temperature: 0, seed: 0 };
// hover help for every setting (the 📖 Guide button opens the full documentation)
export const TIPS = {
"Mode": "Camera cuts: shots start at the detected cuts, long ones are split evenly. Fixed length: equal parts no longer than the maximum.",
"Detector": "Cut detector. PySceneDetect adaptive works on most footage (fast camera moves don't trigger it); content is stricter on hard cuts; built-in needs no package.",
"Sensitivity": "How easily a change counts as a cut. Higher finds more cuts (and more false ones).",
"Frame grid": "Frame counts the model accepts. H3: 17n+5, LTX/Wan: 8n+1, Wan: 4n+1. Each shot is generated at the next valid length and trimmed back.",
"Max seconds / shot": "Longest shot sent to the model. Keep it within what the model / LoRA was trained on (H3 body swap: about 4.5 s).",
"Min seconds / shot": "Shorter shots merge into a neighbour when the merge still fits the maximum.",
"Timeline fps": "The planner works on this frame rate (the source is resampled). Generated video and audio use it too.",
"Max shots": "Only the first N shots (0 = all). Handy for quick tests.",
"Max total seconds": "Only the first N seconds of the video (0 = all).",
"Megapixels": "Generation size: the source aspect ratio at this pixel area (0.15 ≈ 512×288). Cropped shots use it for the crop.",
"Size multiple": "Width and height snap to this multiple (32 for H3 and most video models).",
"Run": "Auto loop: every shot in one queue run. Queue loop: one shot per run, stored on disk, the join outputs the video after the last shot.",
"Skipped shots in the output": "Shots that do not run: keep the original video there, or remove them (with their audio).",
"Crop padding": "Context kept around the mask's box, as a % of the box. More context blends better; less gives the model more pixels on the subject.",
"Expand": "Grow the mask by this many pixels: before pasting a cropped result back, and for the generation mask (inpaint 'only the mask'). Raise it when the new person is bigger than the old one (longer hair, wider body).",
"Feather": "Soft edge of the paste-back, in pixels.",
"Temporal expand": "Hold the mask this many frames before and after: removes flicker where SAM 3 drops a frame.",
"Blockify": "Square blocks of this size (0 = off). 16 matches H3's latent grid, useful as an inpainting mask.",
"Detection threshold": "SAM 3 score needed to keep a text match. Lower finds more (and more wrong) objects.",
"Max objects": "How many matches of the text prompt are tracked together.",
"Paste back": "Only the mask: the result replaces the masked pixels. The whole box: the full crop is pasted with feathered borders.",
"Frames per shot": "How many frames of each shot the VLM looks at.",
"Max tokens": "Longest VLM answer. Raise it for long descriptions.",
};
export const snapUp = (n, grid) => {
const [s, o] = GRIDS[grid] || GRIDS["H3 (17n+5)"]; n = Math.max(1, n | 0);
if (s === 1) return n; if (n <= o) return o; return o + s * Math.ceil((n - o) / s);
};
export const snapDown = (n, grid) => {
const [s, o] = GRIDS[grid] || GRIDS["H3 (17n+5)"]; n = Math.max(1, n | 0);
if (s === 1) return n; if (n <= o) return o; return o + s * Math.floor((n - o) / s);
};
export const hue = i => `hsl(${(i * 47 + 200) % 360} 55% 46%)`;
export const fmtT = (f, fps) => { const s = f / fps; return `${Math.floor(s / 60)}:${(s % 60).toFixed(2).padStart(5, "0")}`; };
export const segKey = s => `${s.start}-${s.end}`;
// a shot has a mask: its own mask video, SAM 3 text or points, or the plan's mask video for the whole video
// (the planner's `mask` input is only known when the workflow runs)
export const hasMask = s => !!(s.mask?.video || s.mask?.text || (s.mask?.points || []).length || s.extMask);
export const maskSource = s => s.mask?.video ? "video" : (s.mask?.text || (s.mask?.points || []).length) ? "sam"
: s.trackMask ? "track" : s.extMask ? "global" : "";
// ---- small view helpers
export const pill = (text, kind = "", title = "") => h("span", { class: ["pill", kind], title }, text);
export const hint = (text, style = "") => h("span", { class: "hint", style }, text);
export const fld = (label, input, help) => {
const tip = Object.entries(TIPS).find(([k]) => String(label).startsWith(k))?.[1] || "";
return h("div", { class: "fld", title: tip }, [h("label", tip ? [label, h("span", { class: "qm" }, " ⓘ")] : label), input,
help ? h("div", { class: "hint" }, help) : null]);
};
export const check = (checked, onChange, label, title = "") =>
h("label", { class: "chk", title }, [h("input", { type: "checkbox", checked: !!checked, onChange: e => onChange(e.target.checked) }), label]);
export const select = (value, options, onChange, attrs = {}) => h("select", { value, onChange: e => onChange(e.target.value), ...attrs },
options.map(o => h("option", { value: Array.isArray(o) ? o[0] : o }, Array.isArray(o) ? o[1] : o)));
// a titled section inside a card; `open` makes it a collapsible <details>
// what the user opened or closed wins over a section's default: re-renders (a timer, a status line) no longer snap a
// section back shut right after it was clicked open
const OPEN = reactive({});
export const section = (title, body, { sub = "", right = null, open = null, cls = "" } = {}) => {
const head = [h("span", { class: "sect" }, title), sub ? hint(sub) : null, h("span", { class: "grow" }), right];
if (open === null) return h("div", { class: ["sec", cls] }, [h("div", { class: "sech" }, head), ...[body].flat()]);
const key = `${cls}|${typeof title === "string" ? title : ""}`;
const isOpen = key in OPEN ? OPEN[key] : !!open;
return h("details", { class: ["sec", cls], open: isOpen, onToggle: e => { OPEN[key] = e.target.open; } },
[h("summary", { class: "sech" }, [h("span", { class: "caret" }, "▸"), ...head]), ...[body].flat()]);
};
+207
View File
@@ -0,0 +1,207 @@
// Shot Planner styles (one <style> tag, added once).
const CSS = `
.bsl{--bg:#16161a;--card:#1d1d23;--card2:#22222a;--line:#2d2d37;--line2:#3a3a46;--txt:#c9c9cf;--txt2:#8c8c97;--hi:#ececf1;
--acc:#5b8cff;--acc2:#3a63e0;--ok:#7fe0a8;--warn:#ffc46b;--bad:#ff8f8f;
font:12px/1.45 var(--font-family,system-ui,sans-serif);color:var(--txt);background:var(--bg);border-radius:10px;
height:100%;overflow:auto;box-sizing:border-box;padding:10px;position:relative}
.bsl.root{position:absolute;inset:0;height:auto}
.bsl:focus{outline:none}
.bsl *{box-sizing:border-box}
.bsl .qm{color:#7d8bb0;cursor:help}
.bsl .grow{flex:1}
.bsl .row{display:flex;gap:6px;align-items:center;flex-wrap:wrap}
.bsl .col{display:flex;flex-direction:column;gap:6px}
.bsl .hint{font-size:10px;color:#7d7d88}
.bsl .chk{display:inline-flex;gap:5px;align-items:center;cursor:pointer;user-select:none}
.bsl .chk input{accent-color:var(--acc);margin:0}
/* header + step tabs */
.bsl .hdr{display:flex;align-items:center;gap:8px;margin-bottom:8px}
.bsl .ttl{font-weight:600;color:var(--hi);font-size:13px;letter-spacing:.01em}
.bsl .tabs{display:flex;gap:4px;margin-bottom:8px;background:#111115;border:1px solid var(--line);border-radius:9px;padding:3px;position:sticky;top:-10px;z-index:7}
.bsl .tab{flex:1;background:none;border:0;border-radius:7px;padding:6px 6px;color:var(--txt2);font-size:11px;font-weight:600;display:flex;gap:5px;align-items:center;justify-content:center}
.bsl .tab:hover{background:#22222a;color:var(--txt)}
.bsl .tab.on{background:var(--acc2);color:#fff}
.bsl .tab .n{font-size:9px;font-weight:600;opacity:.75;border:1px solid currentColor;border-radius:999px;padding:0 5px;line-height:14px}
.bsl .tab .dot{width:6px;height:6px;border-radius:50%;background:var(--warn)}
.bsl .tab:disabled{opacity:.35}
/* pills, inputs, buttons */
.bsl .pill{font-size:10px;padding:1px 7px;border-radius:999px;background:#25252d;color:#a9a9b4;border:1px solid #33333d;white-space:nowrap}
.bsl .pill.ok{background:#173527;color:var(--ok);border-color:#245c40}
.bsl .pill.warn{background:#3a2a12;color:var(--warn);border-color:#6a4a16}
.bsl .pill.bad{background:#3a1a1d;color:var(--bad);border-color:#6a2c33}
.bsl .pill.acc{background:#1d2a4d;color:#a9c1ff;border-color:#2f4a8f}
.bsl select,.bsl input[type=number],.bsl input[type=text],.bsl textarea{background:#101014;border:1px solid #33333d;color:#e6e6ea;
border-radius:6px;padding:4px 7px;font-size:11px;font-family:inherit;width:100%}
.bsl select:focus,.bsl input:focus,.bsl textarea:focus{border-color:var(--acc);outline:none}
.bsl textarea{resize:vertical;min-height:54px;font-family:ui-monospace,monospace}
.bsl input[type=range]{width:100%;accent-color:var(--acc)}
.bsl button{background:#26262e;border:1px solid #393945;color:#dcdce2;border-radius:6px;padding:4px 9px;font-size:11px;cursor:pointer;white-space:nowrap}
.bsl button:hover{background:#30303a}
.bsl button.pri{background:var(--acc2);border-color:#4b74f0;color:#fff}
.bsl button.pri:hover{background:#4672ee}
.bsl button.dng{background:#3a1f22;border-color:#6a2c33;color:#ffb3b3}
.bsl button.ghost{background:none;border-color:transparent;color:var(--txt2)}
.bsl button.ghost:hover{color:var(--txt);background:#26262e}
.bsl button:disabled{opacity:.45;cursor:default}
.bsl .seg-btns{display:inline-flex;border:1px solid #393945;border-radius:6px;overflow:hidden}
.bsl .seg-btns button{border:0;border-radius:0;border-right:1px solid #393945}
.bsl .seg-btns button:last-child{border-right:0}
.bsl .seg-btns button.on{background:var(--acc2);color:#fff}
/* cards and sections */
.bsl .card{background:var(--card);border:1px solid var(--line);border-radius:9px;padding:8px 10px;margin-bottom:8px}
.bsl .card h5{margin:0 0 6px;font-size:11px;font-weight:600;color:#b9b9c4;text-transform:uppercase;letter-spacing:.06em;display:flex;gap:6px;align-items:center}
.bsl .sec{border-top:1px solid var(--line);padding:8px 0 4px}
.bsl .sec:first-child{border-top:0;padding-top:2px}
.bsl .sech{display:flex;gap:6px;align-items:center;margin-bottom:6px;list-style:none;cursor:default}
.bsl summary.sech{cursor:pointer;user-select:none}
.bsl summary.sech::-webkit-details-marker{display:none}
.bsl .sect{font-size:11px;font-weight:600;color:#b9b9c4;text-transform:uppercase;letter-spacing:.06em;white-space:nowrap}
.bsl .caret{display:inline-block;transition:transform .15s;color:var(--txt2);font-size:10px}
.bsl details[open]>summary .caret{transform:rotate(90deg)}
.bsl details:not([open])>.sech{margin-bottom:0}
.bsl .grid{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:6px 8px}
.bsl .grid.g3{grid-template-columns:repeat(3,minmax(0,1fr))}
.bsl .fld label{display:block;font-size:10px;color:var(--txt2);margin-bottom:2px}
.bsl .err{color:var(--bad);background:#2a1517;border:1px solid #5a2228;border-radius:6px;padding:5px 8px;margin-bottom:8px;display:flex;gap:8px;align-items:center}
.bsl .note{background:#1a2034;border:1px solid #2b3557;border-radius:7px;padding:6px 8px;color:#b8c4e6;font-size:11px}
.bsl .note b{color:#dfe6ff}
.bsl .empty{padding:26px 10px;text-align:center;color:var(--txt2)}
.bsl .empty b{display:block;color:var(--hi);font-size:13px;margin-bottom:4px}
/* busy bar */
.bsl .busybar{position:sticky;top:0;z-index:8;margin:6px 0;padding:6px 8px;border-radius:8px;background:#22222b;border:1px solid #3d3d4a}
.bsl .busybar .brow{display:flex;gap:6px;align-items:center}
.bsl .busybar .spin{width:12px;height:12px;border-radius:50%;border:2px solid var(--acc);border-top-color:transparent;animation:bslspin .8s linear infinite;flex:none}
.bsl .busybar .bprog{height:4px;background:#33333d;border-radius:3px;margin-top:5px;overflow:hidden;position:relative}
.bsl .busybar .bprog>div{height:100%;background:var(--acc);transition:width .2s}
.bsl .busybar .bprog.ind>div{position:absolute;width:30%;animation:bslind 1.2s ease-in-out infinite}
@keyframes bslspin{to{transform:rotate(360deg)}}
@keyframes bslind{0%{left:-30%}100%{left:100%}}
/* timeline */
.bsl .tl{position:relative;overflow-x:auto;overflow-y:hidden;border-radius:7px;background:#111115;border:1px solid #2a2a33}
.bsl .tlin{position:relative}
.bsl .strip{display:flex;height:64px;overflow:hidden}
.bsl .strip img{height:64px;object-fit:cover;flex:none;opacity:.92}
.bsl .spark{display:block;height:26px;width:100%}
.bsl .segs{position:relative;height:30px;margin-top:2px;cursor:crosshair}
.bsl .seg{position:absolute;top:2px;bottom:2px;border-radius:5px;display:flex;align-items:center;justify-content:center;
font-size:10px;color:#fff;font-weight:600;overflow:hidden;cursor:pointer;border:2px solid transparent;user-select:none}
.bsl .seg.sel{border-color:#fff;box-shadow:0 0 0 2px #5b8cff66}
.bsl .seg.off{opacity:.35;background-image:repeating-linear-gradient(45deg,#0004 0 6px,#0000 6px 12px)}
.bsl .seg.long{outline:2px solid #ff5d5d;outline-offset:-2px}
.bsl .seg.playing{box-shadow:0 0 0 2px #ffd34d}
.bsl .hdl{position:absolute;top:-70px;bottom:0;width:9px;margin-left:-4px;cursor:ew-resize;z-index:3}
.bsl .hdl::after{content:"";position:absolute;left:3px;top:0;bottom:0;width:3px;background:#fff;opacity:.85;border-radius:2px;box-shadow:0 0 4px #000}
.bsl .hdl:hover::after{background:#ffd34d}
.bsl .rgrid{display:grid;grid-template-columns:auto 130px auto 130px auto;gap:6px 10px;align-items:start;justify-content:start}
.bsl .rgrid>button{margin-top:15px}
.bsl .rgrid .fld .hint{margin-top:2px}
.bsl .outr{position:absolute;top:0;bottom:0;background:repeating-linear-gradient(45deg,#000a 0 6px,#0007 6px 12px);z-index:2;pointer-events:none}
.bsl .cut{position:absolute;top:0;height:64px;width:2px;background:#ff4d6d;opacity:.9;pointer-events:none}
.bsl .ph{position:absolute;top:0;bottom:0;width:1px;background:#ffd34d;pointer-events:none;z-index:4}
.bsl .playhead{position:absolute;top:0;bottom:0;width:2px;background:#ffd34d;box-shadow:0 0 6px #ffd34d;pointer-events:none;z-index:5}
.bsl .tip{position:absolute;z-index:6;pointer-events:none;background:#0d0d10ee;border:1px solid #3a3a44;border-radius:6px;padding:3px;font-size:10px;color:#ddd}
.bsl .tip img{display:block;height:84px;border-radius:4px}
/* shot cards */
.bsl .shots{display:flex;gap:6px;overflow-x:auto;padding:2px 2px 6px;scroll-snap-type:x proximity}
.bsl .shots .sc{flex:0 0 158px;scroll-snap-align:start}
.bsl .sc{background:var(--card2);border:1px solid #30303a;border-radius:8px;padding:6px;cursor:pointer;position:relative}
.bsl .sc:hover{border-color:#4a4a58}
.bsl .sc.sel{border-color:var(--acc);box-shadow:0 0 0 1px var(--acc)}
.bsl .sc.skip{opacity:.62}
.bsl .sc .bar{height:3px;border-radius:2px;margin-bottom:5px}
.bsl .sc .t{font-size:10px;color:#9a9aa6}
.bsl .sc .p{font-size:10px;color:#c9c9d3;margin-top:3px;height:28px;overflow:hidden}
.bsl .sc .play{position:absolute;top:5px;right:5px;padding:1px 6px;font-size:10px}
.bsl .badges{display:flex;gap:3px;flex-wrap:wrap;margin-top:4px}
.bsl .bdg{font-size:9px;padding:0 5px;border-radius:4px;background:#2a2a33;color:#a9a9b4;line-height:15px}
.bsl .bdg.on{background:#1d2a4d;color:#a9c1ff}
.bsl .bdg.warn{background:#3a2a12;color:var(--warn)}
.bsl .thumbs{display:flex;gap:4px;margin-top:4px}
.bsl .rt{width:34px;height:34px;border-radius:5px;object-fit:cover;background:#2a2a33;border:1px solid #3a3a45}
.bsl .rt.ph2{display:flex;align-items:center;justify-content:center;color:#666;font-size:9px}
.bsl .who{display:flex;gap:3px;margin-top:3px;align-items:center}
.bsl .who img{width:22px;height:22px;border-radius:50%;object-fit:cover;border:1px solid #3a3a45;opacity:.7}
.bsl .who img.main{width:26px;height:26px;opacity:1;border-color:#fff}
.bsl .who img.lk{border-color:#8fd18f}
/* shot editor */
.bsl .edhd{display:flex;gap:6px;align-items:center;flex-wrap:wrap;margin-bottom:6px}
.bsl .edhd .big{font-size:14px;font-weight:700;color:var(--hi)}
.bsl .refbox{display:flex;gap:8px;align-items:center}
.bsl .refslot{width:78px;height:78px;border-radius:8px;border:1px dashed #44444f;background:#141418;display:flex;align-items:center;
justify-content:center;cursor:pointer;overflow:hidden;color:#6f6f7a;font-size:10px;text-align:center;flex:none;white-space:pre-line}
.bsl .refslot img{width:100%;height:100%;object-fit:cover}
.bsl .refslot:hover{border-color:var(--acc)}
.bsl .refslot.sm{width:56px;height:56px;font-size:10px}
.bsl .refslot.via{border-style:solid;border-color:#3b5b3b}
.bsl .rslot{display:flex;flex-direction:column;gap:3px;align-items:center}
.bsl .guidecolor{display:flex;flex-direction:column;align-items:center;gap:2px;max-width:112px}
.bsl .guidecolor select{width:100%;font-size:10px;padding:2px 4px}
.bsl .rbtns{display:flex;gap:3px;min-height:20px}
.bsl .rbtns button{padding:1px 6px;font-size:10px}
.bsl .recent{display:flex;gap:4px;flex-wrap:wrap;margin-top:6px;align-items:center}
.bsl .recent img{width:30px;height:30px;border-radius:5px;object-fit:cover;cursor:pointer;border:1px solid #3a3a45}
.bsl .recent img:hover{border-color:var(--acc)}
.bsl .recent.sm img{width:22px;height:22px}
.bsl .modes{display:grid;grid-template-columns:repeat(5,minmax(0,1fr));gap:6px;margin-bottom:6px}
.bsl .mode{border:1px solid var(--line2);border-radius:7px;padding:6px 8px;cursor:pointer;background:#18181d}
.bsl .mode:hover{border-color:#55556a}
.bsl .mode.on{border-color:var(--acc);background:#1a2034}
.bsl .mode b{display:block;color:var(--hi);font-size:11px}
.bsl .mode span{font-size:10px;color:var(--txt2)}
.bsl .mstrip{display:flex;gap:4px;margin-top:6px;align-items:center;flex-wrap:wrap}
.bsl .mstrip img{height:90px;border-radius:5px;border:1px solid #3a3a45}
.bsl .vsug{margin-top:6px;padding:6px 8px;border:1px dashed #4a4a58;border-radius:6px;display:flex;flex-direction:column;gap:3px}
.bsl .vsug button{margin-left:6px;padding:1px 6px}
.bsl .copy{display:flex;gap:10px;flex-wrap:wrap;align-items:center}
.bsl .player{display:flex;gap:10px;align-items:flex-start;flex-wrap:wrap}
.bsl .player video{max-height:220px;max-width:100%;border-radius:8px;background:#000;border:1px solid var(--line)}
.bsl .pinfo{flex:1;min-width:200px;display:flex;flex-direction:column;gap:6px}
.bsl .tc{font-family:ui-monospace,monospace;font-size:12px;color:#e6e6ea;background:#101014;border:1px solid var(--line);border-radius:6px;padding:6px 8px;line-height:1.6}
.bsl .tc b{color:#ffd34d}
/* cast, VLM, checks */
.bsl .cast{display:grid;grid-template-columns:repeat(auto-fill,minmax(150px,1fr));gap:8px}
.bsl .person{background:#141418;border:1px solid var(--line);border-radius:8px;padding:6px;display:flex;flex-direction:column;gap:4px;align-items:flex-start}
.bsl .person.ign{opacity:.45}
.bsl .person .face{width:64px;height:64px;border-radius:50%;object-fit:cover;cursor:pointer;border:2px solid #3a3a45}
.bsl .person .t{font-size:10px;color:#9a9aa6}
.bsl .dlist{display:flex;flex-direction:column;gap:6px;margin-top:6px}
.bsl .drow{display:flex;gap:8px;align-items:flex-start}
.bsl .dthumbs{display:flex;gap:3px}
.bsl .dthumbs img{width:44px;height:58px;object-fit:cover;border-radius:5px;border:1px solid #3a3a45}
.bsl .checks{display:flex;flex-direction:column;gap:3px}
.bsl .chip{padding:0 6px;font-size:10px;line-height:16px;border-radius:4px}
.bsl .ck{display:flex;gap:8px;align-items:flex-start;padding:4px 6px;border-radius:6px;background:#18181d;border:1px solid var(--line)}
.bsl .ck .lvl{font-size:12px}
.bsl .prog{height:7px;border-radius:4px;background:#26262e;overflow:hidden}
.bsl .prog>div{height:100%;background:linear-gradient(90deg,#3a63e0,#7fe0a8)}
.bsl .langpick{display:flex;gap:5px;align-items:center}
.bsl .langpick select{width:auto;min-width:100px;padding:4px 6px}
.bsl .tourback{position:fixed;inset:0;z-index:10001;background:rgba(0,0,0,.68);display:flex;align-items:center;justify-content:center;padding:16px}
.bsl .tourbox{width:min(480px,94vw);background:#1d1d23;border:1px solid #4a4a58;border-radius:12px;padding:16px;box-shadow:0 12px 48px #0009}
.bsl .tourbox h3{margin:18px 0 8px;color:var(--hi)}
.bsl .tourbox p{line-height:1.55}
/* points window (teleported to <body>) */
.bsl .mmodal{position:fixed;inset:0;background:rgba(0,0,0,.72);z-index:10000;display:flex;align-items:center;justify-content:center}
.bsl .mbox{background:#1d1d23;border:1px solid #3a3a45;border-radius:10px;padding:12px;width:min(960px,94vw);max-height:94vh;overflow:auto}
.bsl .mimg{position:relative;cursor:crosshair;user-select:none;line-height:0}
.bsl .mimg img{width:100%;border-radius:6px}
.bsl .pt{position:absolute;width:14px;height:14px;margin:-7px 0 0 -7px;border-radius:50%;border:2px solid #fff;cursor:pointer}
.bsl .pt.pos{background:#3ccf6b}.bsl .pt.neg{background:#e8455a}
`;
export function styles() {
if (document.getElementById("bfs-shotloop-css")) return;
const el = document.createElement("style");
el.id = "bfs-shotloop-css";
el.textContent = CSS;
document.head.appendChild(el);
}
+96
View File
@@ -0,0 +1,96 @@
// The busy bar and the points window (🎯 Points…: click on a frame to pick the subject for SAM 3).
import { Teleport } from "../vendor/vue.esm-browser.prod.mjs";
import { h, hint, t } from "./common.js";
export function busyBar(c, msg) {
const { busySince, now, upPct, status } = c;
const secs = busySince.value ? Math.max(0, Math.round((now.value - busySince.value) / 1000)) : 0;
const pct = upPct.value >= 0 ? upPct.value : (status.total > 0 ? Math.min(1, status.done / status.total) : -1);
const stage = status.stage && !msg.startsWith(status.stage) ? status.stage : "";
return h("div", { class: "busybar" }, [
h("div", { class: "brow" }, [h("span", { class: "spin" }), h("b", msg), stage ? hint(`· ${stage}`) : null,
h("span", { class: "grow" }), pct >= 0 ? h("span", `${Math.round(pct * 100)}%`) : null, hint(`${secs}s`)]),
h("div", { class: ["bprog", pct < 0 && "ind"] }, [h("div", { style: pct >= 0 ? `width:${(pct * 100).toFixed(1)}%` : "" })]),
]);
}
export function pointsModal(c) {
const { modal, segs, plan, api } = c;
const ms = segs.value[modal.idx];
if (!modal.open || !ms) return null;
return h(Teleport, { to: "body" }, h("div", { class: "bsl", style: "background:none;padding:0" }, h("div", { class: "mmodal",
onKeydown: e => e.stopPropagation(), onPointerdown: e => { if (e.target === e.currentTarget) modal.open = false; } }, [
h("div", { class: "mbox" }, [
h("div", { class: "row", style: "margin-bottom:6px" }, [h("span", { class: "sect" }, `Shot #${modal.idx + 1} · points`),
hint("click = keep (green) · right-click or shift+click = exclude (red) · click a point to remove it")]),
h("div", { class: "mimg", onContextmenu: e => e.preventDefault(), onPointerdown: e => {
const box = e.currentTarget.getBoundingClientRect();
const x = (e.clientX - box.left) / box.width, y = (e.clientY - box.top) / box.height;
modal.points = [...modal.points, { x: +x.toFixed(4), y: +y.toFixed(4), label: (e.button === 2 || e.shiftKey) ? 0 : 1 }];
modal.prev = null;
} }, [
h("img", { draggable: false, src: api.apiURL(`/bfs/shotloop/frame?video=${encodeURIComponent(plan.video)}&fps=${plan.fps}&f=${ms.start + modal.key}&w=900`) }),
...modal.points.map((p, k) => h("div", { class: ["pt", p.label ? "pos" : "neg"], style: `left:${p.x * 100}%;top:${p.y * 100}%`,
title: "click to remove", onPointerdown: e => { e.stopPropagation(); modal.points = modal.points.filter((_, j) => j !== k); modal.prev = null; } })),
]),
h("div", { class: "row", style: "margin-top:6px" }, [
hint(`frame ${ms.start + modal.key} (${modal.key + 1}/${ms.len})`),
h("input", { type: "range", min: 0, max: ms.len - 1, step: 1, value: modal.key, style: "flex:1",
onInput: e => { modal.key = parseInt(e.target.value); modal.prev = null; } }),
]),
h("div", { class: "row", style: "margin-top:6px" }, [
h("input", { type: "text", value: modal.text, style: "flex:1", placeholder: "text prompt (used when there are no points)",
onChange: e => { modal.text = e.target.value; } }),
h("button", { onClick: () => { modal.points = []; modal.prev = null; } }, "Clear points"),
h("button", { class: "pri", disabled: !!modal.busy || !(modal.points.length || modal.text),
onClick: () => c.previewMask(modal.idx, { points: modal.points, key: modal.key, text: modal.text }, true) }, "👁 Segment"),
]),
modal.busy ? busyBar(c, modal.busy) : null,
modal.prev ? h("div", { class: "mstrip" }, [...modal.prev.frames.map(f => h("img", { src: f.src, title: `frame ${f.f}` })),
hint(modal.prev.empty ? "nothing found" : `covers ${(modal.prev.coverage * 100).toFixed(1)}%`)]) : null,
h("div", { class: "row", style: "margin-top:8px;justify-content:flex-end" }, [
h("button", { onClick: () => { modal.open = false; } }, "Cancel"),
h("button", { title: "Apply these points to EVERY shot, each on its frame at the same relative position (a subject that stays in place)",
onClick: c.savePointsAll }, "Save → all shots"),
h("button", { disabled: !modal.points.length,
title: "Track this selection with SAM 3 through the whole video (or the range), forward and backward from this frame; every shot without its own selection takes its part of the track. Also type a short text (e.g. 'woman in pink top') so the subject is found again after camera cuts.",
onClick: c.saveTrack }, "🎯 Track → whole video"),
h("button", { class: "pri", onClick: c.savePoints }, "Save (this shot)"),
]),
]),
])));
}
const TOUR_STEPS = [
["video", "Video", "Choose a video, set the frame range, split at camera cuts, then set the generation size."],
["shots", "Shots", "Review each shot, assign its references and prompt, and choose how the result is applied."],
["people", "People", "Link detected people to references. Set masks here or per shot; ArcFace links are a guide and need your review."],
["prompts", "Prompts", "Set shared references, prompts and optional VLM assistance. Written prompts are kept exactly as authored."],
["shots", "Color Guide", "Color Guide adds a translucent marker to a selected person. Ref 1 and Ref 2 must use different colors in the same shot."],
["run", "Run", "Check warnings, choose auto or queue loop, and set filters or quick-run limits."],
];
export function tourModal(c) {
const { tour, tab } = c;
const step = tour.value - 1;
if (step < 0 || step >= TOUR_STEPS.length) return null;
const [targetTab, title, body] = TOUR_STEPS[step];
return h(Teleport, { to: "body" }, h("div", { class: "bsl", style: "background:none;padding:0" },
h("div", { class: "tourback", role: "dialog", "aria-modal": "true", "aria-label": t("Shot Planner tour"),
onKeydown: e => { if (e.key === "Escape") tour.value = 0; }, onPointerdown: e => { if (e.target === e.currentTarget) tour.value = 0; } }, [
h("div", { class: "tourbox" }, [
h("div", { class: "row" }, [h("b", t("Shot Planner tour")), h("span", { class: "grow" }), h("span", { class: "hint" }, `${step + 1} / ${TOUR_STEPS.length}`),
h("button", { class: "ghost", title: t("Close tour"), "aria-label": t("Close tour"), onClick: () => { tour.value = 0; } }, "✕")]),
h("h3", t(title)),
h("p", t(body)),
h("p", { class: "hint" }, t("This tour is optional. Use Next to visit each main tab; close it whenever you like.")),
h("div", { class: "row", style: "justify-content:flex-end;margin-top:12px" }, [
step > 0 ? h("button", { onClick: () => { tour.value--; tab.value = TOUR_STEPS[step - 1][0]; } }, t("Back")) : null,
h("button", { class: "pri", onClick: () => {
if (step === TOUR_STEPS.length - 1) tour.value = 0;
else { tour.value++; tab.value = TOUR_STEPS[step + 1][0]; }
} }, t(step === TOUR_STEPS.length - 1 ? "Close tour" : "Next")),
]),
]),
])));
}
+97
View File
@@ -0,0 +1,97 @@
// 👥 People & masks tab: the cast found by face (each person linked to a reference) and the global SAM 3 mask settings.
import { h, fmtT, fld, check, section, pill, hint } from "./common.js";
const CAST_COLORS = [["", "No Color Guide color"], ["#00ff00", "Green"], ["#9b30ff", "Purple"], ["#ff2020", "Red"], ["#168cff", "Blue"],
["#ffe600", "Yellow"], ["#ff00d4", "Magenta"], ["#00e5c0", "Cyan"]];
function castSection(c) {
const { plan, people, busy, usedRefs } = c;
const fps = plan.fps;
const castN = people.value.filter(p => c.linked(p.id)).length;
const opt = (k, label, title) => check(plan[k], v => c.setCastOpt(k, v), label, title);
return section("Cast", [
h("div", { class: "row", style: "gap:14px;margin-bottom:6px" }, [
opt("cast_assign", "Shots use their main person's reference", "a shot without its own reference takes the reference of the person with the most screen time in it"),
opt("cast_split", "Split where the main person changes", "adds a boundary when the biggest face switches to someone else (re-splits the plan)"),
opt("cast_only", "Only run shots with a linked person", "shots where no linked person appears are skipped"),
]),
people.value.length ? h("div", { class: "cast" }, people.value.map(p => {
const cc = c.castOf(p.id);
return h("div", { class: ["person", cc.ignore && "ign"] }, [
h("img", { class: "face", src: p.thumb, title: `first seen ${fmtT(p.first, fps)} (click to seek)`, onClick: () => c.seek(p.first) }),
h("div", { class: "t" }, [h("b", `Person ${p.id}`), ` · ${(p.share * 100).toFixed(1)}%`]),
h("div", { class: "t" }, `${fmtT(p.first, fps)} → ${fmtT(p.last, fps)}`),
h("div", { class: "row", style: "margin-top:4px;align-items:center" }, [
h("span", { class: "hint" }, "Color Guide identity color"),
h("select", { value: cc.guide_color || "", style: "width:auto", onChange: e => c.setCast(p.id, "guide_color", e.target.value) },
CAST_COLORS.map(([v, label]) => h("option", { value: v }, label))),
]),
h("div", { class: "refbox" }, ["ref", "ref2"].map(k => h("div", { class: "rslot" }, [
h("div", { class: "refslot sm", title: cc[k] ? cc[k] + " (click to replace)" : "click to upload",
onClick: () => c.pickFile("image/*", name => c.setCast(p.id, k, name)) }, cc[k] ? [h("img", { src: c.viewUrl(cc[k]) })] : [k === "ref" ? "⬆ ref" : "⬆ ref 2"]),
cc[k] ? h("div", { class: "rbtns" }, [h("button", { title: "remove", onClick: () => c.setCast(p.id, k, "") }, "✕")]) : null,
]))),
usedRefs.value.length ? h("div", { class: "recent sm" }, usedRefs.value.map(n => h("img", { src: c.viewUrl(n), title: `${n} (click = ref, shift+click = ref 2)`,
onClick: e => c.setCast(p.id, e.shiftKey ? "ref2" : "ref", n) }))) : null,
check(cc.ignore, v => c.setCast(p.id, "ignore", v), "ignore"),
]);
})) : h("div", { class: "hint" }, "Nobody found yet."),
h("div", { class: "row", style: "margin-top:8px" }, [
h("button", { class: "pri", disabled: !!busy.value, onClick: () => c.findPeople(true) }, people.value.length ? "↻ Re-analyse people" : "👥 Find people"),
hint(people.value.length ? "faces sampled every 0.5 s and grouped by identity (InsightFace, CPU)" : "detects faces across the video and groups them into people"),
]),
], { sub: "track faces, link each person to a reference, split and assign shots by who is on screen",
right: people.value.length ? pill(`${people.value.length} people · ${castN} linked`) : pill("not analysed") });
}
function maskSection(c) {
const { plan, segs, showMasks } = c;
const MC = plan.mask_cfg || {};
const mnum = (k, step, label, help, scale = 1) => fld(label, h("input", { type: "number", step, min: 0, value: +(MC[k] * scale).toFixed(3),
onChange: e => c.setMaskCfg(k, (parseFloat(e.target.value) || 0) / scale) }), help);
const S = segs.value;
const counts = { mask: 0, crop: 0, cropmask: 0, paste: 0 };
S.forEach(s => { const m = c.modeOf(s); if (m in counts) counts[m]++; });
const masked = S.filter(s => s.mask?.text || (s.mask?.points || []).length || s.mask?.video || s.extMask).length;
return section("Mask & crop settings", [
h("div", { class: "note", style: "margin-bottom:8px" }, [
"Each shot picks what it marks (🎯 text or points, or 🎞 its own mask video) and how it is generated (", h("b", "Full frame · Frame + paste · Mask only · Crop · Crop + mask"),
") in the Shots tab. Masks from a video (here, per shot, or the planner's ", h("b", "mask"), " input) replace SAM 3. These settings apply to every shot."]),
h("div", { class: "row", style: "margin-bottom:8px;align-items:center" }, [
h("span", { style: "white-space:nowrap", title: "A black/white video of the WHOLE source video (white = the subject), e.g. rotoscoped in another tool. Shots without their own mask (video, text or points) use it instead of SAM 3. The planner's optional mask input does the same from the graph and wins over this." }, "🎞 Mask video (whole video)"),
h("select", { value: plan.mask_video || "", style: "flex:1;min-width:180px", onChange: e => c.setGlobalMaskVideo(e.target.value) },
[h("option", { value: "" }, "— none: SAM 3 per shot —"), ...c.files.videos.map(v => h("option", { value: v }, v))]),
h("button", { onClick: () => c.pickFile("video/*", c.setGlobalMaskVideo) }, "⬆ Upload"),
plan.mask_video ? h("button", { onClick: () => c.setGlobalMaskVideo("") }, "✕") : null,
]),
plan.track?.points?.length ? h("div", { class: "note", style: "margin-bottom:8px;display:flex;gap:8px;align-items:center" }, [
h("span", { class: "grow" }, [h("b", "🎯 Whole-video track: "), `${plan.track.points.length} point(s) at frame ${plan.track.frame}`,
plan.track.text ? ` · re-found by "${plan.track.text}" after cuts` : " · no text: after a cut the subject may be lost (add one in 🎯 Points…)",
". Shots without their own selection use it."]),
h("button", { class: "ghost", onClick: c.clearTrack }, "✕ remove"),
]) : null,
h("div", { class: "row", style: "gap:14px;margin-bottom:6px" }, [
check(MC.fill_holes, v => c.setMaskCfg("fill_holes", v), "Fill holes", "Fill enclosed gaps so each region is solid"),
check(MC.invert, v => c.setMaskCfg("invert", v), "Invert", "Use everything except the segmented object"),
check(showMasks.value, v => { showMasks.value = v; }, "Show masks on shots", "Overlay the mask preview on the shot cards"),
]),
h("div", { class: "grid" }, [
mnum("expand", 1, "Expand (px)", "grow the mask (paste-back and generation mask)"),
mnum("temporal_expand", 1, "Temporal expand (frames)", "hold the mask a few frames: less flicker"),
mnum("threshold", 0.05, "Detection threshold", "SAM 3 score to keep an object"),
mnum("max_objects", 1, "Max objects", "how many matches of the text are tracked"),
mnum("padding", 1, "Crop padding (%)", "Crop modes: context around the mask's box", 100),
mnum("feather", 1, "Feather (px)", "Crop modes: soft edge of the paste"),
fld("Paste back", h("select", { value: MC.paste, onChange: e => c.setMaskCfg("paste", e.target.value) },
[h("option", { value: "mask" }, "only the mask (feathered)"), h("option", { value: "box" }, "the whole box (feathered)")]), "Crop modes"),
mnum("blockify", 1, "Blockify (px, 0 = off)", "square blocks; 16 matches H3's latent grid"),
]),
hint("Uses the official sam3.1_multiplex_fp16 checkpoint (models/checkpoints), downloaded from Comfy-Org/sam3.1 the first time.", "display:block;margin-top:6px"),
], { sub: "SAM 3",
right: h("span", { class: "row" }, [pill(`${masked} masked`), counts.mask ? pill(`${counts.mask} mask only`, "acc") : null, counts.paste ? pill(`${counts.paste} frame+paste`, "acc") : null,
counts.crop + counts.cropmask ? pill(`${counts.crop + counts.cropmask} cropped`, "warn") : null]) });
}
export function peopleTab(c) {
return h("div", { class: "card" }, [castSection(c), maskSection(c)]);
}
+98
View File
@@ -0,0 +1,98 @@
// 📝 Prompts & refs tab: the global reference / prompt, the VLM (shot suggestions) and the reference descriptions ({details}).
import { h, fld, check, section, pill, hint, select } from "./common.js";
const GUIDE_COLORS = [["#00ff00", "Green"], ["#9b30ff", "Purple"], ["#ff2020", "Red"], ["#168cff", "Blue"],
["#ffe600", "Yellow"], ["#ff00d4", "Magenta"], ["#00e5c0", "Cyan"]];
function globalSection(c) {
const { plan } = c;
const slot = (name, label, field) => h("div", { class: "rslot" }, [
h("div", { class: "refslot", title: name ? name + " (click to replace)" : "click to upload", onClick: () => c.uploadRef("global", field) },
name ? [h("img", { src: c.viewUrl(name) })] : [label]),
h("div", { class: "rbtns" }, [name ? h("button", { title: "remove", onClick: () => c.setRef("global", field, "") }, "✕") : null]),
h("label", { class: "guidecolor" }, [
h("span", { class: "hint" }, field === "global_ref" ? "Guide color · Ref 1" : "Guide color · Ref 2"),
h("select", { value: field === "global_ref" ? (plan.global_color_guide_ref1 || "#00ff00") : (plan.global_color_guide_ref2 || "#9b30ff"),
disabled: !name,
title: "Default Color Guide marker when this global reference is used",
onChange: e => c.setPlan(field === "global_ref" ? "global_color_guide_ref1" : "global_color_guide_ref2", e.target.value) },
GUIDE_COLORS.map(([v, text]) => h("option", { value: v }, text))),
]),
]);
return section("Global reference & prompt", [
h("div", { class: "row", style: "align-items:flex-start;gap:10px" }, [
h("div", { class: "refbox" }, [slot(plan.global_ref, "⬆ global\nreference", "global_ref"), slot(plan.global_ref2, "⬆ global\nref 2", "global_ref2")]),
h("div", { style: "flex:1;min-width:200px" }, [
h("textarea", { style: "min-height:78px", placeholder: "Global prompt (a connected `prompt` input overrides it)",
value: plan.global_prompt, onChange: e => c.setPlan("global_prompt", e.target.value) }),
]),
]),
h("div", { class: "hint", style: "margin-top:4px" }, [
"Used by shots without their own. Connected ref_image / ref_image_2 / prompt inputs on the node override these. Placeholders: ",
h("b", "{target}"), " who is replaced (per shot) · ", h("b", "{details}"), " description of the shot's references · ",
h("b", "{shot}"), " the VLM's description of the shot · ", h("b", "{setting}"), " the setting picture (H3 Conditioning)."]),
h("div", { class: "hint", style: "margin-top:4px" }, "These colors are defaults for the global references. Turn on Color Guide on a shot to apply them; shot-level colors can override them."),
], { sub: "defaults for every shot" });
}
function vlmSection(c) {
const { plan, vlmSug, busy, refSets } = c;
const VC = plan.vlm_cfg || {};
const opt = (k, label, title) => check(VC[k], v => c.setVlmCfg(k, v), label, title);
const sugN = Object.keys(vlmSug.value).length;
return [
section("VLM", [
h("div", { class: "row", style: "gap:14px;margin-bottom:6px" }, [
opt("enabled", "Use the VLM when the workflow runs", "At run time the planner asks the VLM about the shots it runs and applies the options below"),
opt("auto_segment", "Fill the mask text", "Shots without a mask text or points get the VLM's segment suggestion"),
opt("auto_shot", "Fill {shot} in the prompt", "Write {shot} in a prompt: it becomes the VLM's description of that shot (camera, framing, action)"),
]),
h("div", { class: "grid" }, [
fld("Frames per shot", h("input", { type: "number", min: 1, max: 8, step: 1, value: VC.frames, onChange: e => c.setVlmCfg("frames", parseInt(e.target.value) || 3) })),
fld("Max tokens", h("input", { type: "number", min: 64, max: 2048, step: 32, value: VC.max_tokens, onChange: e => c.setVlmCfg("max_tokens", parseInt(e.target.value) || 1024) })),
]),
h("textarea", { style: "margin-top:6px;min-height:44px", placeholder: "Extra instruction for the VLM (optional), e.g. 'segment the woman, not the man'",
value: VC.instruction || "", onChange: e => c.setVlmCfg("instruction", e.target.value) }),
h("div", { class: "row", style: "margin-top:6px" }, [
h("button", { class: "pri", disabled: !!busy.value, onClick: c.analyseVLM }, "🤖 Analyse shots"),
h("button", { disabled: !sugN, onClick: c.applyAllSug, title: "Mask text for shots without one, skip where the VLM says skip" }, "Apply suggestions → all"),
]),
hint("Connect a Qwen3-VL (CLIPLoader) to the planner's vlm input. The VLM buttons work after the workflow has run once with it connected (ComfyUI only hands models to nodes when they run). Each shot's suggestion shows in its editor.", "display:block;margin-top:4px"),
], { sub: "looks at every shot and suggests the mask, a description and whether to run it",
right: h("span", { class: "row" }, [VC.enabled ? pill("on", "ok") : pill("off"), sugN ? pill(`${sugN} suggestions`) : null]) }),
section("Describe references ({details})", [
h("div", { class: "row", style: "gap:8px" }, [
select(VC.describe_preset, ["short", "full body", "head / face", "face attributes", "outfit", "custom"], v => c.setVlmCfg("describe_preset", v), { style: "width:auto" }),
h("button", { class: "pri", disabled: !!busy.value || !refSets.value.length, onClick: c.describeRefs }, "📝 Describe refs"),
h("button", { disabled: !!busy.value || !refSets.value.length,
title: "Try another description: a new seed (and some temperature if it is 0), then describe again",
onClick: () => { if (!(+VC.temperature > 0)) c.setVlmCfg("temperature", 0.7); c.setVlmCfg("seed", (+VC.seed || 0) + 1); c.describeRefs(); } }, "🎲"),
fld("Temperature", h("input", { type: "number", min: 0, max: 1.5, step: 0.1, value: VC.temperature ?? 0, style: "width:70px",
title: "0 = always the same answer (greedy); 0.5-1 = varied answers, picked by the seed",
onChange: e => c.setVlmCfg("temperature", Math.max(0, parseFloat(e.target.value) || 0)) })),
fld("Seed", h("input", { type: "number", min: 0, step: 1, value: VC.seed ?? 0, style: "width:80px",
title: "Used when the temperature is above 0: another seed, another description",
onChange: e => c.setVlmCfg("seed", Math.max(0, parseInt(e.target.value) || 0)) })),
hint(`${refSets.value.length} reference set(s) · editable below`),
check(VC.fit_prompt !== false, v => c.setVlmCfg("fit_prompt", v), "fit the prompt",
"The VLM gets the global prompt's sentence that holds {details} (and the shot prompt's {target} sentence for 🧑 Describe target) with a blank in its place, and writes the description to fit it: the description adapts to the prompt, not the other way round."),
]),
VC.describe_preset === "custom" ? h("textarea", { style: "margin-top:6px;min-height:52px", value: VC.describe_custom || "",
placeholder: "Your instruction for the VLM, e.g. 'Describe the character's costume and props piece by piece…'",
onChange: e => c.setVlmCfg("describe_custom", e.target.value) }) : null,
refSets.value.length ? h("div", { class: "dlist" }, refSets.value.map(x => h("div", { class: "drow" }, [
h("div", { class: "dthumbs" }, [x.a ? h("img", { src: c.viewUrl(x.a) }) : null, x.b ? h("img", { src: c.viewUrl(x.b) }) : null]),
h("div", { style: "flex:1" }, [
hint(x.where.length > 4 ? `${x.where.slice(0, 4).join(", ")} +${x.where.length - 4}` : x.where.join(", ")),
h("textarea", { style: "min-height:44px", value: plan.ref_details?.[x.key] || "",
placeholder: "no description yet: Describe refs, write your own, or leave it to the VLM at run time",
onChange: e => c.setDetail(x.key, e.target.value) }),
]),
]))) : hint("Add a reference (global, per shot or per person) first."),
], { sub: "write {details} in any prompt: each shot gets the description of its own references" }),
];
}
export function promptsTab(c) {
return h("div", { class: "card" }, [globalSection(c), ...vlmSection(c)]);
}
+74
View File
@@ -0,0 +1,74 @@
// ▶ Run tab: checks before running, how the loop runs (auto / queue), which shots run (filters) and test limits.
import { h, fld, check, section, pill, hint, select } from "./common.js";
function checksSection(c) {
const { checks, tab } = c;
const list = checks.value;
const warns = list.filter(x => x.lvl === "warn").length;
// one row per kind of issue, with the shots it concerns
const groups = new Map();
for (const x of list) { const k = `${x.lvl}|${x.text}`; if (!groups.has(k)) groups.set(k, { ...x, shots: [] }); groups.get(k).shots.push(x.shot); }
const rows = [...groups.values()].sort((a, b) => (a.lvl === "warn" ? 0 : 1) - (b.lvl === "warn" ? 0 : 1));
const go = i => { c.select(i); tab.value = "shots"; };
return section("Checks", rows.length ? h("div", { class: "checks" }, rows.map(x => h("div", { class: "ck" }, [
h("span", { class: "lvl" }, x.lvl === "warn" ? "⚠" : "ℹ"),
h("span", { style: "flex:1;min-width:0" }, [h("div", x.text), h("div", { class: "row", style: "gap:3px;margin-top:3px" }, [
...x.shots.slice(0, 24).map(i => h("button", { class: "chip", title: "go to this shot", onClick: () => go(i) }, `#${i + 1}`)),
x.shots.length > 24 ? hint(`+${x.shots.length - 24} more`) : null])]),
pill(`${x.shots.length} shot(s)`, x.lvl === "warn" ? "warn" : ""),
]))) : h("div", { class: "hint" }, "✓ Nothing to fix in the shots that run."),
{ sub: "only shots that run", right: list.length ? pill(`${warns} warning(s) · ${list.length - warns} note(s)`, warns ? "warn" : "") : pill("all good", "ok") });
}
function loopSection(c) {
const { plan, prog, active } = c;
const num = (k, step, min) => h("input", { type: "number", step, min, value: plan[k], onChange: e => c.setPlan(k, parseFloat(e.target.value) || 0) });
return section("Run", [
h("div", { class: "grid" }, [
fld("Run", select(plan.run, [["auto", "Auto loop (one run)"], ["queue", "Queue loop (one shot per run)"]], v => c.setPlan("run", v, c.progress))),
fld("Skipped shots in the output", select(plan.skip_fill, [["original", "Keep original video"], ["drop", "Remove them"]], v => c.setPlan("skip_fill", v))),
fld("Max shots (0 = all)", num("max_parts", 1, 0), "quick tests"),
fld("Max total seconds (0 = all)", num("max_total_s", 0.5, 0), "from the start of the range"),
]),
plan.run === "queue" ? h("div", { style: "margin-top:8px" }, [
h("div", { class: "row", style: "margin-bottom:4px" }, [h("b", "Queue loop"), h("span", { class: "grow" }),
hint(`${prog.done}/${prog.count || active.value.length} shots done`)]),
h("div", { class: "prog" }, [h("div", { style: `width:${(100 * prog.done / Math.max(1, prog.count || active.value.length)).toFixed(1)}%` })]),
h("div", { class: "row", style: "margin-top:6px" }, [
check(plan.auto_continue, v => c.setPlan("auto_continue", v), "Auto-queue the next shot"),
h("button", { onClick: () => c.progress(false) }, "↻ Status"), h("button", { class: "dng", onClick: () => c.progress(true) }, "⟲ Reset loop"),
]),
hint("Each run generates one shot and stores it. Nodes after BFS Shot Join only run on the last shot, with the full video.", "display:block;margin-top:4px"),
]) : null,
], { sub: `${active.value.length} shot(s) will run` });
}
function filtersSection(c) {
const { plan, stats, segs, busy, filtersOn } = c;
const F = plan.filters;
const chk = (k, label) => check(F[k], v => c.setFilter(k, v), label);
const fnum = (k, step, label, help) => fld(label, h("input", { type: "number", step, min: 0, value: F[k], onChange: e => c.setFilter(k, parseFloat(e.target.value) || 0) }), help);
const skippedN = segs.value.filter(s => c.skipWhy(s)).length;
return section("Filters", [
h("div", { class: "row", style: "gap:14px;margin-bottom:6px" }, [
chk("person", "Needs a person"), chk("face", "Needs a face"), chk("skip_dark", "Skip dark / fades"), chk("skip_static", "Skip static shots"),
]),
h("div", { class: "grid" }, [
fnum("min_person_area", 0.01, "Min person size (0-1 of frame)", "e.g. 0.03 skips wide shots"),
fnum("max_persons", 1, "Max people (0 = any)", "skip crowds"),
fnum("min_frames", 1, "Min frames (0 = off)"),
fnum("samples", 1, "Frames sampled / shot"),
fnum("dark_level", 0.01, "Dark below (0-1)"),
fnum("static_level", 0.001, "Static below"),
]),
h("div", { class: "row", style: "margin-top:8px" }, [
h("button", { class: "pri", disabled: !!busy.value, onClick: c.analyzeContent }, "👤 Analyse people & faces"),
hint(stats.value.length ? `stats for ${stats.value.length} shots · YOLO person/face from models/ultralytics` : "runs the detectors on a few frames of every shot"),
]),
], { open: filtersOn.value || stats.value.length > 0, sub: "skipped shots do not run",
right: filtersOn.value ? pill(`${skippedN} skipped`, "warn") : pill("off") });
}
export function runTab(c) {
return h("div", { class: "card" }, [checksSection(c), loopSection(c), filtersSection(c)]);
}
+332
View File
@@ -0,0 +1,332 @@
// 🎬 Shots tab: one card per shot, then the selected shot's editor (preview, reference & prompt, who is replaced,
// continuity, copy to other shots).
import { h, hue, fmtT, segKey, hasMask, maskSource, pill, hint, check, section } from "./common.js";
import { h as rawH } from "../vendor/vue.esm-browser.prod.mjs";
const GUIDE_COLORS = [["#00ff00", "Green"], ["#9b30ff", "Purple"], ["#ff2020", "Red"], ["#168cff", "Blue"],
["#ffe600", "Yellow"], ["#ff00d4", "Magenta"], ["#00e5c0", "Cyan"]];
const bdg = (text, kind = "", title = "") => h("span", { class: ["bdg", kind], title }, text);
function shotCard(c, s, i) {
const { plan, sel, maskPrev, showMasks } = c;
const fps = plan.fps, why = c.skipWhy(s), who = c.whoIn(s), st = c.statFor(s), issues = c.checksFor(i);
const warnN = issues.filter(x => x.lvl === "warn").length;
const mp = maskPrev.value[segKey(s)];
return h("div", { class: ["sc", i === sel.value && "sel", why && "skip"], onClick: () => { sel.value = i; } }, [
h("button", { class: "play", title: "play this shot", onClick: e => { e.stopPropagation(); c.playShot(i); } }, "▶"),
h("div", { class: "bar", style: `background:${hue(i)}` }),
h("div", { class: "row" }, [h("b", `#${i + 1}`), s.cut ? pill("cut") : null,
why ? pill("skip", "warn", why) : pill("run", "ok"), s.force !== "auto" ? pill(s.force) : null]),
h("div", { class: "badges" }, [
{ video: bdg("🎞 mask video", "", s.mask.video), global: bdg("🎞 global mask", "", "the plan's mask video for the whole video"),
track: bdg("🎯 track", "", "its part of the whole-video SAM 3 track"),
sam: bdg("🎯 " + (s.mask.text ? s.mask.text.slice(0, 14) : `${(s.mask.points || []).length} pts`), "", s.mask.text || `${(s.mask.points || []).length} points`) }[maskSource(s)] || null,
s.crop || s.inpaint || s.paste ? bdg({ paste: "🧩 frame+paste", mask: "🎭 mask only", crop: "✂ crop", cropmask: "✂🎭 crop+mask" }[c.modeOf(s)] + (s.inpaint && s.strength < 1 ? ` ${Number(s.strength).toFixed(2)}` : ""), "on", c.modeName(s)) : null,
s.color_guide ? bdg("🎨 color guide", "on", `${s.color_guide_color || "cast color"} · opacity ${s.color_guide_opacity ?? 0.35}`) : null,
s.target ? bdg("🧑 target", "on", s.target) : null,
i > 0 && s.chain !== "off" ? bdg("⛓", "on", s.chain === "history" ? `continues the previous shot's last ${s.chainHistory} frames (history)` : `continues from the previous shot's ${s.chainFrame} frame as ${s.chain}`) : null,
warnN ? bdg(`⚠ ${warnN}`, "warn", issues.map(x => x.text).join("\n")) : null,
]),
who ? h("div", { class: "who" }, who.people.length ? who.people.map(id => h("img", {
src: c.personOf(id)?.thumb || "", title: `Person ${id}${id === who.main ? " (main)" : ""}${c.linked(id) ? " · linked" : ""}`,
class: [id === who.main && "main", c.linked(id) && "lk"] })) : [h("span", { class: "t" }, "no faces")]) : null,
st ? h("div", { class: "t", style: "margin-top:2px" },
`👤 ${st.persons} · ${(st.person_area * 100).toFixed(1)}% · 🙂 ${st.faces} · ☀ ${(st.brightness * 100).toFixed(0)}%`) : null,
why ? h("div", { class: "t", style: "color:#ffc46b" }, why) : null,
h("div", { class: "t" }, `${fmtT(s.start, fps)} → ${fmtT(s.end, fps)} · ${s.len}f → ${s.gen}f`),
h("div", { class: "thumbs" }, [
...["ref", "ref2"].map(k => {
const own = s[k], via = c.castRef(s, k), name = own || via || plan["global_" + k];
return name ? h("img", { class: "rt", src: c.viewUrl(name), title: own ? name : via ? `${name} (from the person)` : `${name} (global)`,
style: own ? "" : via ? "outline:1px dashed #8fd18f" : "opacity:.45" }) : h("div", { class: "rt ph2" }, k);
}),
h("img", { class: "rt", style: "width:56px", src: showMasks.value && mp?.frames?.length
? mp.frames[Math.floor(mp.frames.length / 2)].src : c.thumbFor(s.start + Math.floor(s.len / 2)) }),
]),
s.prompt ? rawH("div", { class: "p" }, s.prompt)
: h("div", { class: "p" }, plan.global_prompt ? "↳ global prompt" : "— no prompt —"),
]);
}
function player(c, cur) {
const { plan, segs, play, vid, sel, n } = c;
const S = segs.value, ps = S[play.idx] || cur, fps = plan.fps, N = n.value;
return h("div", { class: "player" }, [
h("video", { ref: vid, src: c.viewUrl(plan.video), preload: "metadata", muted: false, playsinline: true,
onPause: () => { if (play.mode) play.mode = ""; } }),
h("div", { class: "pinfo" }, [
h("div", { class: "row" }, [
h("button", { class: "pri", disabled: !cur, onClick: () => c.playShot(sel.value) }, `▶ Play #${sel.value + 1}`),
h("button", { onClick: c.playAll }, "▶ Play all"),
h("button", { onClick: c.stop }, "■ Stop"),
check(play.loop, v => { play.loop = v; }, "loop shot"),
play.mode ? pill(play.mode === "all" ? "playing all" : "playing shot", "warn") : null,
]),
ps ? h("div", { class: "tc" }, [
h("div", ["Shot ", h("b", `#${S.indexOf(ps) + 1}`), c.skipWhy(ps) ? ` (skipped: ${c.skipWhy(ps)})` : ""]),
h("div", ["start ", h("b", fmtT(ps.start, fps)), ` (frame ${ps.start})`]),
h("div", ["end ", h("b", fmtT(ps.end, fps)), ` (frame ${ps.end - 1}) · ${(ps.len / fps).toFixed(2)}s`]),
h("div", ["now ", h("b", fmtT(Math.min(N, play.frame), fps)), ` (frame ${Math.min(N, play.frame)})`]),
]) : null,
hint("Click the timeline to seek. Play all skips shots that will not run."),
]),
]);
}
function refSlot(c, name, label, target, field, via = "") {
return h("div", { class: "rslot" }, [
h("div", { class: ["refslot", !name && via && "via"], title: name ? name + " (click to replace)" : via ? `uses ${via}; click to upload this shot's own` : "click to upload",
onClick: () => c.uploadRef(target, field) },
name ? [h("img", { src: c.viewUrl(name) })] : via ? [h("img", { src: c.viewUrl(via), style: "opacity:.4" })] : [label]),
h("div", { class: "rbtns" }, [name ? h("button", { title: "remove (use the person's / global one)", onClick: () => c.setRef(target, field, "") }, "✕") : null]),
]);
}
function editor(c, cur) {
const { plan, sel, segs, maskPrev, targetCrop, vlmSug, busy, usedRefs, copyOpts, COPY_FIELDS } = c;
const i = sel.value, S = segs.value, k = segKey(cur), mp = maskPrev.value[k], g = vlmSug.value[k];
const issues = c.checksFor(i);
const guideColor1 = cur.color_guide_color || (cur.color_guide_person != null
? (c.castOf(cur.color_guide_person).guide_color || plan.global_color_guide_ref1 || "#00ff00")
: (plan.global_color_guide_ref1 || "#00ff00"));
const viaRef = f => cur[f] ? "" : (c.castRef(cur, f) || plan["global_" + f] || "");
const head = h("div", { class: "edhd" }, [
h("button", { class: "ghost", disabled: i === 0, title: "previous shot (←)", onClick: () => c.select(i - 1) }, "◀"),
h("span", { class: "big" }, `Shot #${i + 1}`),
h("button", { class: "ghost", disabled: i >= S.length - 1, title: "next shot (→)", onClick: () => c.select(i + 1) }, "▶"),
c.skipWhy(cur) ? pill(`skip · ${c.skipWhy(cur)}`, "warn") : pill("runs", "ok"),
hint(`frames ${cur.start}–${cur.end - 1} · ${cur.len} → generate ${cur.gen}`),
h("span", { class: "grow" }),
h("select", { value: cur.force, style: "width:auto", title: "Override the content filters for this shot",
onChange: e => c.setMeta(i, "force", e.target.value) },
[h("option", { value: "auto" }, "filters decide"), h("option", { value: "run" }, "always run"), h("option", { value: "skip" }, "always skip")]),
h("button", { onClick: () => c.setMeta(i, "enabled", !cur.enabled) }, cur.enabled ? "⏸ Disable" : "▶ Enable"),
h("button", { title: "split this shot in two", onClick: () => c.splitAt(cur.start + Math.floor(cur.len / 2)) }, "✂ Split"),
h("button", { disabled: i >= S.length - 1, title: "merge with the next shot", onClick: () => c.mergeNext(i) }, "⇥ Merge"),
]);
const warnsHere = issues.filter(x => x.lvl === "warn");
const problems = warnsHere.length ? h("div", { class: "checks", style: "margin-bottom:6px" }, warnsHere.map(x =>
h("div", { class: "ck" }, [h("span", { class: "lvl" }, x.lvl === "warn" ? "⚠" : "ℹ"), h("span", x.text)]))) : null;
const refPrompt = section("Reference & prompt", [
h("div", { class: "row", style: "align-items:flex-start;gap:10px" }, [
h("div", { class: "refbox" }, [
refSlot(c, cur.ref, "⬆ reference", i, "ref", viaRef("ref")),
refSlot(c, cur.ref2, "⬆ ref 2", i, "ref2", viaRef("ref2")),
]),
h("div", { style: "flex:1;min-width:200px" }, [
h("textarea", { placeholder: "Prompt for this shot (empty = global prompt). Placeholders: {target} {details} {shot} {setting}",
value: cur.prompt, onChange: e => c.setMeta(i, "prompt", e.target.value) }),
!cur.prompt && plan.global_prompt ? rawH("span", { class: "hint" }, `↳ uses the global prompt: ${plan.global_prompt.slice(0, 120)}${plan.global_prompt.length > 120 ? "…" : ""}`) : null,
]),
]),
usedRefs.value.length ? h("div", { class: "recent" }, [hint("recent (click = ref, shift+click = ref 2):"),
...usedRefs.value.map(n => h("img", { src: c.viewUrl(n), title: n, onClick: e => c.setRef(i, e.shiftKey ? "ref2" : "ref", n) })),
h("button", { class: "sm", title: "Forget the recent pictures (the ones this plan still uses stay)", onClick: () => c.clearRecent() }, "✕ clear"),
h("button", { class: "sm dng", title: "Remove EVERY reference from this plan (global, all shots, all people) and the recent list", onClick: () => c.clearAllRefs() }, "✕ clear all")]) : null,
h("div", { class: "row", style: "margin-top:6px" }, [
h("button", { class: "dng", title: "clear this shot's references and prompt: it uses its person's / the global ones",
onClick: () => { ["ref", "ref2", "prompt"].forEach(f => { c.meta(i)[f] = ""; }); c.save(); } }, "Use global"),
hint("empty slots use the person's reference (Cast) or the global one (Prompts & refs tab)"),
]),
], { sub: "who the shot becomes" });
const mode = (on, title, text, onClick) => h("div", { class: ["mode", on && "on"], onClick }, [h("b", title), h("span", text)]);
const who = section("Who is replaced", [
h("div", { class: "row", style: "align-items:center" }, [
h("input", { type: "text", value: cur.mask.text || "", style: "flex:1;min-width:160px",
placeholder: "mask: what to segment, in English (woman in pink top, black dog…) · Enter = preview",
title: "What to segment, in English: a short phrase (noun + one or two traits), e.g. 'woman in pink top'; commas for several things. Enough on its own - no points needed: SAM 3 finds and tracks it through the shot (every match, up to Max objects). Press Enter to preview. Points, when set, take priority.",
onInput: e => c.setMask(i, "text", e.target.value),
onKeydown: e => { if (e.key === "Enter") { e.preventDefault(); c.setMask(i, "text", e.target.value); if (e.target.value || (cur.mask.points || []).length) c.previewMask(i); } } }),
h("button", { title: "Pick positive / negative points on a frame of this shot (more precise than text when several people look alike)",
onClick: () => c.openPoints(i) }, (cur.mask.points || []).length ? `🎯 Points (${cur.mask.points.length})` : "🎯 Points…"),
h("button", { class: hasMask(cur) && !mp ? "pri" : "", disabled: !!busy.value || !hasMask(cur),
title: (cur.mask.points || []).length ? "Segment this shot (the points win over the text)" : "Segment this shot with the text and show a few frames",
onClick: () => c.previewMask(i) }, "👁 Preview"),
h("button", { class: "dng", disabled: !(cur.mask.text || (cur.mask.points || []).length || cur.mask.video || cur.crop || cur.inpaint), title: "Remove this shot's mask (points, text, mask video) and mode - for shots where it picked the wrong thing",
onClick: () => c.clearMask(i) }, "✕ Clear"),
]),
h("div", { class: "row", style: "margin-top:6px;align-items:center" }, [
h("span", { class: "hint", style: "white-space:nowrap", title: "A black/white video (white = the subject), e.g. rotoscoped in another tool. It replaces SAM 3 for this shot and covers only this shot: its first frame is the shot's first frame." }, "🎞 mask video"),
h("select", { value: cur.mask.video || "", style: "flex:1;min-width:160px",
onChange: e => c.setMask(i, "video", e.target.value) },
[h("option", { value: "" }, cur.extMask ? "— none: the global mask video (People & masks) —" : "— none: SAM 3 (text / points) —"),
...c.files.videos.map(v => h("option", { value: v }, v))]),
h("button", { title: "upload a mask video for this shot", onClick: () => c.pickFile("video/*", name => c.setMask(i, "video", name)) }, "⬆ Upload"),
cur.mask.video ? h("button", { title: "back to SAM 3 / the global mask", onClick: () => c.setMask(i, "video", "") }, "✕") : null,
]),
maskSource(cur) === "video" ? hint("This shot uses its mask video: the text and points are ignored.", "display:block;margin-top:2px")
: maskSource(cur) === "global" ? hint("No mask of its own: this shot uses the global mask video (People & masks tab).", "display:block;margin-top:2px")
: maskSource(cur) === "track" ? hint("No mask of its own: this shot uses its part of the whole-video track (People & masks tab).", "display:block;margin-top:2px") : null,
mp ? h("div", { class: "mstrip" }, [
...mp.frames.map(f => h("img", { src: f.src, title: `frame ${f.f}` })),
hint(mp.empty ? "nothing found: the shot runs without a mask" : `mask covers ${(mp.coverage * 100).toFixed(1)}%${cur.crop ? " · yellow = crop box" : ""}`),
]) : null,
h("div", { class: "modes", style: "margin-top:8px" }, [
["frame", "Full frame", "The whole frame is regenerated. The mask, if any, only feeds {target} / the setting picture."],
["paste", "🧩 Frame + paste", "The whole frame is generated (the pose follows the guide, the new subject may be bigger); BFS Shot Join pastes only the subject onto the original: background exact."],
["mask", "🎭 Mask only", "Only the mask is regenerated, on the whole frame: the rest stays exactly as it was. No crop / uncrop."],
["crop", "✂ Crop", "A box around the mask is regenerated (more pixels for a small subject); BFS Shot Join pastes it back."],
["cropmask", "✂🎭 Crop + mask", "Inside the crop, only the mask is regenerated: the most detail with the background kept."],
].map(([k, t, d]) => mode(c.modeOf(cur) === k, t, d, () => c.setMode(i, k)))),
h("div", { class: "row", style: "margin-top:8px;align-items:center;gap:8px" }, [
check(!!cur.color_guide, v => c.setMeta(i, "color_guide", v), "🎨 Color Guide",
"Tint the selected person in the source guide. This is a translucent visual cue; it does not enable H3 mask-only inpainting."),
cur.color_guide ? h("label", { class: "row", style: "gap:6px;align-items:center" }, [
h("span", { class: "hint" }, "Ref 1 color"),
h("select", { value: cur.color_guide_color ||
(cur.color_guide_person != null ? (c.castOf(cur.color_guide_person).guide_color || plan.global_color_guide_ref1 || "#00ff00")
: (plan.global_color_guide_ref1 || "#00ff00")),
onChange: e => c.setMeta(i, "color_guide_color", e.target.value) },
GUIDE_COLORS.map(([v, label]) => h("option", { value: v }, label))),
]) : null,
]),
cur.color_guide ? h("details", { class: "note", style: "margin:4px 0;padding:6px 8px", open: !!cur.color_guide_text2 }, [
h("summary", { style: "cursor:pointer" }, cur.color_guide_text2 ? "Second person · Ref 2" : "+ Add a second person (optional)"),
h("div", { class: "row", style: "margin-top:8px;align-items:center;gap:8px" }, [
h("span", { class: "hint", style: "white-space:nowrap" }, "Ref 2"),
h("input", { type: "text", value: cur.color_guide_text2 || "", style: "flex:1;min-width:160px",
placeholder: "Describe the second person for SAM 3",
onChange: e => c.setMeta(i, "color_guide_text2", e.target.value) }),
h("button", { disabled: !!busy.value || !cur.color_guide_text2,
onClick: () => c.previewMask(i, { text: cur.color_guide_text2, max_objects: 1 }) }, "👁 Preview"),
h("select", { value: cur.color_guide_color2 || (cur.color_guide_person2 != null
? (c.castOf(cur.color_guide_person2).guide_color || plan.global_color_guide_ref2 || "#9b30ff")
: (plan.global_color_guide_ref2 || "#9b30ff")),
title: "Choose the marker color for the person matched to ref 2",
onChange: e => c.setMeta(i, "color_guide_color2", e.target.value) },
GUIDE_COLORS.map(([v, label]) => h("option", { value: v, disabled: v === guideColor1 }, label))),
]),
h("details", { style: "margin-top:6px" }, [
h("summary", { class: "hint", style: "cursor:pointer" }, "Keep ref 2 color consistent across shots (optional)"),
h("div", { class: "row", style: "margin-top:6px;gap:8px" }, [
h("span", { class: "hint" }, "Ref 2 identity"),
h("select", { value: cur.color_guide_person2 == null ? "" : String(cur.color_guide_person2),
onChange: e => { c.setMeta(i, "color_guide_person2", e.target.value === "" ? null : Number(e.target.value)); c.setMeta(i, "color_guide_color2", ""); } },
[h("option", { value: "" }, "Choose manually"), ...c.people.value.map(p => h("option", { value: String(p.id) }, `Person ${p.id}`))]),
]),
]),
h("details", { style: "margin-top:6px" }, [
h("summary", { class: "hint", style: "cursor:pointer" }, `Advanced · opacity ${Number(cur.color_guide_opacity ?? 0.35).toFixed(2)}`),
h("div", { class: "row", style: "margin-top:6px;align-items:center" }, [
h("span", { class: "hint" }, "Opacity"),
h("input", { type: "range", min: 0.05, max: 0.8, step: 0.01, value: cur.color_guide_opacity ?? 0.35,
style: "flex:1;max-width:320px", onInput: e => { c.meta(i).color_guide_opacity = parseFloat(e.target.value); }, onChange: c.save }),
hint("The default keeps movement and expression visible."),
]),
]),
]) : null,
cur.color_guide ? h("details", { style: "margin:4px 0" }, [
h("summary", { class: "hint", style: "cursor:pointer" }, `More options · opacity ${Number(cur.color_guide_opacity ?? 0.35).toFixed(2)}`),
h("div", { class: "row", style: "margin-top:6px;gap:8px;align-items:center" }, [
h("span", { class: "hint" }, "Ref 1 identity"),
h("select", { value: cur.color_guide_person == null ? "" : String(cur.color_guide_person),
onChange: e => { c.setMeta(i, "color_guide_person", e.target.value === "" ? null : Number(e.target.value)); c.setMeta(i, "color_guide_color", ""); } },
[h("option", { value: "" }, "Choose manually"), ...c.people.value.map(p => h("option", { value: String(p.id) }, `Person ${p.id}`))]),
cur.color_guide_person != null ? h("button", { class: "ghost", title: "Use the cast member's assigned color",
onClick: () => c.setMeta(i, "color_guide_color", "") }, "↩ cast color") : null,
h("span", { class: "hint" }, "Opacity"),
h("input", { type: "range", min: 0.05, max: 0.8, step: 0.01, value: cur.color_guide_opacity ?? 0.35,
style: "flex:1;max-width:240px", onInput: e => { c.meta(i).color_guide_opacity = parseFloat(e.target.value); }, onChange: c.save }),
]),
]) : null,
cur.color_guide && !hasMask(cur) ? h("div", { class: "err" }, "Color Guide needs this shot's person selection: add mask text, points or a mask video, then preview it.") : null,
cur.color_guide && cur.crop && cur.color_guide_text2 ? h("div", { class: "err" }, "The second Color Guide selection uses the full frame; choose Full frame, Frame + paste or Mask only for this shot.") : null,
(cur.crop || cur.inpaint || cur.paste) && !hasMask(cur) ? h("div", { class: "err" }, `${c.modeName(cur)} needs a mask on this shot: a mask text, points or a mask video (or the planner's mask input).`) : null,
cur.inpaint ? h("div", { class: "row", style: "margin-bottom:6px;align-items:center",
title: "Opacity of the generation mask: the value of the white inside the mask (1 = fully white, the default). H3 reads a grey mask value as 'regenerate this much': at 0.85 the masked area starts at 85% of the noise and keeps ~15% of the original there. 0.8-0.9 keeps pose, outline and lighting while still swapping; too low copies the original person." }, [
h("span", { style: "white-space:nowrap" }, `Mask opacity ${Number(cur.strength).toFixed(2)}`),
h("input", { type: "range", min: 0.3, max: 1, step: 0.01, value: cur.strength, style: "flex:1;max-width:320px",
onInput: e => { c.meta(i).strength = parseFloat(e.target.value); }, onChange: c.save }),
h("button", { class: "ghost", disabled: cur.strength >= 1, onClick: () => c.setMeta(i, "strength", 1) }, "reset"),
hint(cur.strength >= 1 ? "regenerates the masked area completely" : `keeps ~${Math.round(100 - cur.strength * 100)}% of the original inside the mask`),
]) : null,
cur.inpaint ? h("div", { class: "note", style: "margin-bottom:6px" }, [h("b", "Mask only / Crop + mask: "),
"set BFS Shot H3 Conditioning → inpaint to ", h("b", "per shot (planner)"), " (the default). Raise Expand (People & masks tab) when the new subject is bigger than the old one."]) : null,
cur.paste ? h("div", { class: "row", style: "margin-bottom:6px;align-items:center",
title: "What the NEW subject is, in English, for SAM 3 on the result: BFS Shot Join also pastes its outline (when it is bigger than the old one). E.g. person, dog, cat, car. Empty = person." }, [
h("span", { style: "white-space:nowrap" }, "new subject"),
h("input", { type: "text", value: cur.pasteText || "", placeholder: "person (or dog, cat, car… what the result shows)", style: "flex:1;max-width:420px",
onChange: e => c.setMeta(i, "paste_text", e.target.value) }),
hint("its outline on the result is pasted too, so a bigger silhouette is not cut"),
]) : null,
h("div", { class: "row", style: "margin-top:4px;align-items:center" }, [
targetCrop.value[k] ? h("img", { src: targetCrop.value[k], style: "height:56px;border-radius:6px" }) : null,
h("span", { class: "hint", style: "white-space:nowrap" }, "{target}"),
h("input", { type: "text", value: cur.target, style: "flex:1;min-width:200px",
placeholder: "who is replaced, e.g. the young woman in a pink crop top (empty = the person)",
title: "Write {target} in the prompt: it becomes this description, so the model knows WHICH person to replace (useful with several people on screen).",
onChange: e => c.setMeta(i, "target", e.target.value) }),
h("button", { disabled: !!busy.value || !hasMask(cur),
title: "Uses this shot's selection (🎯 Points… on the person — click the BODY, not only the face, so the outfit is described — or the mask text): SAM 3 cuts the person out and the VLM describes them",
onClick: () => c.describeTarget(i) }, "🧑 Describe target"),
]),
g ? h("div", { class: "vsug" }, [
h("div", ["🤖 ", h("b", "segment: "), g.segment || "—", g.segment ? h("button", { onClick: () => c.applySug(i, "segment") }, "Use as mask") : null]),
h("div", [h("b", "shot: "), g.shot || "—", g.shot ? h("button", { title: "copy (the {shot} placeholder in the prompt gets it automatically at run time)",
onClick: () => navigator.clipboard?.writeText(g.shot) }, "Copy") : null]),
h("div", [h("b", "recommend: "), `${g.recommend}${g.people != null ? " · " + g.people + " people" : ""}${g.reason ? " · " + g.reason : ""}`,
g.recommend === "skip" ? h("button", { onClick: () => c.applySug(i, "skip") }, "Skip this shot") : null]),
g.raw ? hint("unparsed answer: " + g.raw.slice(0, 200)) : null,
]) : null,
], { sub: "SAM 3 mask (text or points), crop, {target}" });
const cont = section("Continuity", h("div", { class: "row" }, [
h("select", { value: cur.chain, style: "width:auto", disabled: i === 0,
title: "reference: the previous result's frame becomes one more <Picture n> after this shot's own references. first frame: it is anchored at frame 0 of this shot. history: the END of the previous result (17/34/51 frames) goes just before this shot on H3's timeline, so a long take continues its motion and look (same take only, not across a cut).",
onChange: e => c.setMeta(i, "chain", e.target.value) },
[h("option", { value: "off" }, "off"), h("option", { value: "reference" }, "previous shot as reference"),
h("option", { value: "first frame" }, "previous shot as first frame"),
h("option", { value: "history" }, "previous shot's end as history (long take)")]),
cur.chain === "history"
? h("select", { value: String(cur.chainHistory), style: "width:auto", disabled: i === 0,
title: "How much of the previous result's end goes before this shot (whole H3 VAE groups). More carries more motion, costs more memory.",
onChange: e => c.setMeta(i, "chain_history", Number(e.target.value)) },
[17, 34, 51].map(n => h("option", { value: String(n) }, `last ${n} frames (${(n / 24).toFixed(1)} s)`)))
: h("select", { value: cur.chainFrame, style: "width:auto", disabled: i === 0 || cur.chain === "off",
title: "Which frame of the previous shot's result", onChange: e => c.setMeta(i, "chain_frame", e.target.value) },
[h("option", { value: "first" }, "its first frame"), h("option", { value: "middle" }, "its middle frame"), h("option", { value: "last" }, "its last frame")]),
hint(i === 0 ? "the first shot uses only its references"
: cur.chain === "history" ? (cur.cut ? "⚠ a camera cut starts this shot: history is skipped (it only carries a continuing take)"
: "the end of the PREVIOUS shot's result continues into this one (queue loop: shots run in order). Works with every mask mode, crop and the duet canvas.")
: "uses a frame of the PREVIOUS shot's result (queue loop, or auto loop with BFS Shot H3 Duet)"),
]), { open: cur.chain !== "off", sub: cur.chain !== "off" ? cur.chain : "off" });
const copy = section("Copy to other shots", [
h("div", { class: "copy" }, Object.entries(COPY_FIELDS).map(([f, label]) =>
check(copyOpts[f], v => { copyOpts[f] = v; }, label, {
ref: "reference + ref 2", prompt: "this shot's prompt", mask_text: "the mask text (shots keep their own points)",
points: "the points, on each shot's frame at the same relative position (a subject that stays in place)",
mask_video: "the shot's mask video (it covers only its own shot: usually you want a different one per shot)",
crop: "the generation mode (Full frame / Frame + paste / Mask only / Crop / Crop + mask) and its mask strength (shots with a mask)", target: "the {target} description",
chain: "continuity (not on the first shot)" }[f]))),
h("div", { class: "row", style: "margin-top:6px" }, [
h("div", { class: "seg-btns" }, [["all", "all shots"], ["after", "shots after this"]].map(([v, l]) =>
h("button", { class: copyOpts.scope === v && "on", onClick: () => { copyOpts.scope = v; } }, l))),
h("button", { class: "pri", disabled: !Object.keys(COPY_FIELDS).some(f => copyOpts[f]) || S.length < 2,
onClick: e => { const n = c.copyFrom(i); const b = e.currentTarget; b.textContent = `✓ copied to ${n} shot(s)`; setTimeout(() => { b.textContent = "📋 Copy"; }, 1400); } }, "📋 Copy"),
hint("replaces the old “→ all” buttons"),
]),
], { open: false, sub: Object.entries(COPY_FIELDS).filter(([f]) => copyOpts[f]).map(([, l]) => l).join(", ") });
return h("div", { class: "card" }, [head, problems, refPrompt, who, cont, copy]);
}
export function shotsTab(c) {
const { segs, sel, showMasks } = c;
const S = segs.value, cur = S[sel.value];
if (!S.length) return h("div", { class: "card empty" }, [h("b", "No shots yet"), "Pick a video in the Video tab."]);
return h("div", [
h("div", { class: "card" }, [
h("div", { class: "row", style: "margin-bottom:6px" }, [h("span", { class: "sect" }, "Shots"),
hint(`${S.length} · scroll sideways · ← → to move`), h("span", { class: "grow" }),
check(showMasks.value, v => { showMasks.value = v; }, "show mask previews", "Overlay the mask preview on the shot cards")]),
h("div", { class: "shots" }, S.map((s, i) => shotCard(c, s, i))),
]),
cur ? h("div", { class: "card" }, [player(c, cur)]) : null,
cur ? editor(c, cur) : null,
]);
}
+52
View File
@@ -0,0 +1,52 @@
// Timeline: thumbnails, cut marks, cut score, shot bars (click = select, drag the white handles, double-click = split).
import { h, hue, fmtT, pill } from "./common.js";
export function timelineCard(c) {
const { plan, an, segs, sel, n, pxPerFrame, tlWidth, maxLen, cuts, hover, tlEl, zoom, play, vid, active } = c;
const S = segs.value, N = n.value, ppf = pxPerFrame.value, fps = plan.fps;
const thumbs = an.value?.thumbs || [];
const thumbW = Math.max(8, thumbs.length ? tlWidth.value / thumbs.length : 96);
const score = an.value?.score || [];
const maxS = Math.max(4, ...score.slice(0, N));
const sparkPts = score.slice(0, N).map((v, i) => `${(i * ppf).toFixed(1)},${(24 - Math.min(1, v / maxS) * 22).toFixed(1)}`).join(" ");
const tooLong = S.filter(s => s.len > maxLen.value).length;
const totalGen = active.value.reduce((a, s) => a + s.gen, 0);
return h("div", { class: "card" }, [
h("div", { class: "row", style: "margin-bottom:6px" }, [
h("span", { class: "sect" }, "Timeline"),
pill(`${S.length} shots`), pill(`${cuts.value.length} cuts`), pill(`generate ${totalGen} frames`),
tooLong ? pill(`${tooLong} too long (> ${maxLen.value} f)`, "warn") : null,
h("span", { class: "grow" }),
h("span", { class: "hint" }, "zoom"),
h("input", { type: "range", min: 1, max: 8, step: 0.5, value: zoom.value, style: "width:100px", onInput: e => { zoom.value = parseFloat(e.target.value); } }),
]),
h("div", { class: "tl", ref: tlEl,
onPointermove: e => { if (!e.buttons) hover.value = c.frameAt(e); }, onPointerleave: () => { hover.value = null; },
onClick: e => { if (e.target.classList.contains("hdl")) return; c.seek(c.frameAt(e).frame); } }, [
h("div", { class: "tlin", style: `width:${tlWidth.value}px` }, [
h("div", { class: "strip" }, thumbs.map(t => h("img", { src: t.src, style: `width:${thumbW}px` }))),
// outside the start / end range: dimmed, not planned
c.range.value.lo > 0 ? h("div", { class: "outr", style: `left:0;width:${c.range.value.lo * ppf}px`, title: "before the start: not planned" }) : null,
c.range.value.hi < N ? h("div", { class: "outr", style: `left:${c.range.value.hi * ppf}px;width:${(N - c.range.value.hi) * ppf}px`, title: "after the end: not planned" }) : null,
...cuts.value.filter(x => x < N).map(x => h("div", { class: "cut", style: `left:${x * ppf}px`, title: `cut @ ${x}` })),
h("svg", { class: "spark", viewBox: `0 0 ${tlWidth.value} 26`, preserveAspectRatio: "none" },
[h("polyline", { points: sparkPts, fill: "none", stroke: "#ff7a90", "stroke-width": 1, "vector-effect": "non-scaling-stroke" })]),
h("div", { class: "segs", onDblclick: e => c.splitAt(c.frameAt(e).frame) }, [
...S.map((s, i) => h("div", {
class: ["seg", i === sel.value && "sel", !!c.skipWhy(s) && "off", s.len > maxLen.value && "long", play.mode && play.idx === i && "playing"],
style: `left:${s.start * ppf}px;width:${Math.max(2, s.len * ppf - 1)}px;background:${hue(i)}`,
title: `#${i + 1} · frames ${s.start}-${s.end - 1} · ${s.len} → ${s.gen}${c.skipWhy(s) ? " · skip: " + c.skipWhy(s) : ""}`,
onClick: () => { sel.value = i; },
}, s.len * ppf > 26 ? `${i + 1}` : "")),
...plan.bounds.filter(b => b < N).map((b, k) => h("div", { class: "hdl", style: `left:${b * ppf}px`, title: `boundary @ ${b}`, onPointerdown: e => c.drag(k, e) })),
]),
hover.value ? h("div", { class: "ph", style: `left:${hover.value.x}px` }) : null,
vid.value && plan.video ? h("div", { class: "playhead", style: `left:${Math.min(N, play.frame) * ppf}px` }) : null,
]),
hover.value ? h("div", { class: "tip", style: `left:${Math.min(tlWidth.value - 160, Math.max(0, hover.value.x - (tlEl.value?.scrollLeft || 0) - 70))}px;top:2px` },
[h("img", { src: c.thumbFor(hover.value.frame) }), h("div", `frame ${hover.value.frame} · ${fmtT(hover.value.frame, fps)}`)]) : null,
]),
h("div", { class: "hint", style: "margin-top:4px" },
"Click a shot to select it (← → move between shots) · drag the white handles to move a boundary · double-click the shot bar to split · Delete removes the selected shot's cut"),
]);
}
+90
View File
@@ -0,0 +1,90 @@
// 🎞 Video tab: the source video, how it is split into shots, and the generation size.
import { h, GRIDS, fld, select, section, pill, hint, fmtT } from "./common.js";
export function videoTab(c) {
const { plan, an, files, size, busy, detectorUsed, maxLen } = c;
const fps = plan.fps;
const num = (k, step = 0.1, min = 0) => h("input", { type: "number", step, min, value: plan[k], onChange: e => c.setPlan(k, parseFloat(e.target.value) || 0) });
const sel_ = (k, opts, after) => select(plan[k], opts, v => c.setPlan(k, v, after));
const au = an.value?.audio;
const source = section("Source video", [
h("div", { class: "row" }, [
h("div", { style: "flex:1;min-width:180px" }, [h("select", { value: plan.video, onChange: e => c.setVideo(e.target.value) },
[h("option", { value: "" }, "— choose a video from the input folder —"), ...files.videos.map(v => h("option", { value: v }, v))])]),
h("button", { class: plan.video ? "" : "pri", onClick: () => c.pickFile("video/*", c.setVideo) }, "⬆ Upload"),
h("button", { onClick: c.refreshFiles, title: "refresh the input folder" }, "↻"),
]),
an.value ? h("div", { class: "row", style: "margin-top:6px" }, [
pill(`${an.value.width}×${an.value.height}`), pill(`${an.value.fps_src.toFixed(2)} fps`), pill(`${an.value.duration.toFixed(2)} s`),
pill(`timeline ${fps} fps · ${an.value.n} frames`), pill(`generate at ${size.w}×${size.h}`, "acc"),
au?.has_audio
? pill(`🔊 audio · ${au.sample_rate ? (au.sample_rate / 1000).toFixed(1) + " kHz" : "?"} · ${au.channels || "?"} ch`, "ok", au.codec || "")
: pill(`🔇 ${au?.reason || "no audio"} → silent track`, "warn", "The outputs carry a silent track of the right length, so Create Video works"),
h("select", { value: plan.audio_mode, style: "width:auto", disabled: !au?.has_audio, title: "Audio of the planner's and the join's outputs",
onChange: e => c.setPlan("audio_mode", e.target.value) },
[h("option", { value: "auto" }, "use the video's audio"), h("option", { value: "silent" }, "silent track")]),
]) : plan.video ? null : h("div", { class: "empty" }, [h("b", "Pick or upload a video to start"),
"It is split into shots the model can generate; every shot then gets its own reference, prompt and mask."]),
]);
const R = c.range.value, N = c.n.value;
const playF = Math.min(N - 1, Math.max(0, c.play.frame || 0));
// a player right here, so the range is picked by watching (scrub it, or click the timeline above)
const rng = section("Range", [
h("div", { class: "player" }, [
h("video", { ref: c.vid, src: c.viewUrl(plan.video), preload: "metadata", controls: true, playsinline: true,
style: "max-height:240px" }),
h("div", { class: "pinfo" }, [
h("div", { class: "tc" }, [
h("div", ["now ", h("b", fmtT(playF, fps)), ` (frame ${playF})`]),
h("div", ["start ", h("b", fmtT(R.lo, fps)), ` (frame ${R.lo})`]),
h("div", ["end ", h("b", fmtT(R.hi, fps)), ` (frame ${R.hi})`]),
]),
h("div", { class: "rgrid" }, [
h("button", { class: "pri", title: "the range starts at the player's frame", onClick: () => c.setRange("start_f", playF) }, "⇤ start here"),
fld("Start frame", h("input", { type: "number", step: 1, min: 0, max: N - 1, value: R.lo,
onChange: e => c.setRange("start_f", e.target.value) }), fmtT(R.lo, fps)),
h("button", { class: "pri", title: "the range ends after the player's frame", onClick: () => c.setRange("end_f", playF + 1) }, "⇥ end here"),
fld("End frame (0 = end)", h("input", { type: "number", step: 1, min: 0, max: N, value: R.hi >= N ? 0 : R.hi,
onChange: e => c.setRange("end_f", e.target.value) }), fmtT(R.hi, fps)),
(R.lo > 0 || R.hi < N) ? h("button", { class: "ghost", onClick: () => { plan.start_f = 0; c.setRange("end_f", 0); } }, "✕ whole video") : h("span"),
]),
hint(`Play or scrub the video (or click the timeline), then ⇤ / ⇥. ${R.hi - R.lo} frames (${((R.hi - R.lo) / fps).toFixed(2)} s) of ${N}: only this part is split, analysed, generated and joined.`),
]),
]),
], { sub: "work on a part of the video" });
const split = section("Split into shots", [
h("div", { class: "grid" }, [
fld("Mode", sel_("mode", [["shots", "Camera cuts"], ["fixed", "Fixed length"], ["single", "One shot (whole range)"]]),
plan.mode === "single" ? "ignores Max seconds / shot" : ""),
// the range in seconds, here next to the mode: "15 s in one shot" is set without leaving this tab
fld("Range length, seconds (0 = to the end)", h("input", { type: "number", step: 0.5, min: 0,
value: c.range.value.hi < c.n.value ? +((c.range.value.hi - c.range.value.lo) / plan.fps).toFixed(2) : 0,
onChange: e => { const v = parseFloat(e.target.value) || 0;
c.setRange("end_f", v > 0 ? c.range.value.lo + Math.round(v * plan.fps) : 0); } }),
`from the start frame · ${c.range.value.hi - c.range.value.lo} frames`),
fld("Detector", sel_("detector", [["adaptive", "PySceneDetect adaptive"], ["content", "PySceneDetect content"], ["builtin", "Built-in"]])),
fld(`Sensitivity ${Number(plan.sensitivity).toFixed(2)}`, h("input", { type: "range", min: 0, max: 1, step: 0.05, value: plan.sensitivity,
onInput: e => { plan.sensitivity = parseFloat(e.target.value); }, onChange: c.save })),
fld("Frame grid", sel_("grid", Object.keys(GRIDS))),
fld("Max seconds / shot", num("max_s", 0.1, 0.2), `= ${maxLen.value} frames (${(maxLen.value / fps).toFixed(2)} s)`),
fld("Min seconds / shot", num("min_s", 0.1, 0)),
fld("Timeline fps", num("fps", 1, 1), "re-analyses"),
]),
h("div", { class: "row", style: "margin-top:8px" }, [
h("button", { class: "pri", disabled: !plan.video || !!busy.value, onClick: () => c.autoSplit(true) }, "✂ Auto split"),
h("button", { disabled: !plan.video, onClick: c.analyze }, "↻ Re-analyse"),
detectorUsed.value ? hint(`detector: ${detectorUsed.value}`) : null,
hint("Auto split replaces the boundaries; per-shot settings stay with the shot at the same position."),
]),
], { sub: "camera cuts or fixed length, on the model's frame grid" });
const gen = section("Generation size", h("div", { class: "grid" }, [
fld("Megapixels", num("megapixels", 0.01, 0.02), size.w ? `${size.w}×${size.h}` : ""),
fld("Size multiple", num("multiple", 8, 8)),
]), { sub: "cropped shots use it for the crop" });
return h("div", { class: "card" }, [source, an.value ? rng : null, an.value ? split : null, an.value ? gen : null]);
}
File diff suppressed because one or more lines are too long