Compare commits
203
Commits
@@ -0,0 +1,144 @@
|
||||
# MiniMax-H3 downscaled latent guides
|
||||
|
||||
BFSNodes 1.46.0 adds two nodes for H3 upscale LoRAs trained in
|
||||
[alisson-anjos/ai-toolkit](https://github.com/alisson-anjos/ai-toolkit/tree/minimax-h3-latent-guides)
|
||||
with `target_crop_v2` / `target_grid_stride_v1` geometry:
|
||||
|
||||
- **MiniMax-H3 Downscaled Latent Guide (BFS)**: preprocess and encode an image
|
||||
or video guide at a lower resolution, and return a model with compatible
|
||||
spatial conditioning coordinates.
|
||||
- **MiniMax-H3 Guide Target — Image / Video (BFS)**: make the joint target
|
||||
latent at the intended output dimensions, with one frame for an image or
|
||||
`17*n+5` frames for video. It does not resize the target canvas implicitly.
|
||||
|
||||
The native Add Guide node enlarges guides to the output canvas. This node
|
||||
keeps the smaller encoded guide and selects every factor-th origin from the
|
||||
**target patch grid**. Guide time stays aligned to the target. It modifies a
|
||||
cloned model through its UNet wrapper; no global PackedLayout replacement is
|
||||
installed, and other models/conditionings continue to use their native paths.
|
||||
|
||||
## Install / update
|
||||
|
||||
```bash
|
||||
cd ComfyUI/custom_nodes/ComfyUI-BFSNodes
|
||||
git pull --ff-only
|
||||
```
|
||||
|
||||
Restart ComfyUI. Use a ComfyUI build with native H3 support; the layout tests
|
||||
use the exact implementation in commit
|
||||
[`8cfe5e1`](https://github.com/Comfy-Org/ComfyUI/blob/8cfe5e1ecb97512dea8deaac15e1228d7e6feeb1/comfy_extras/nodes_minimax_h3.py#L166).
|
||||
Incompatible H3 layout/VAE APIs produce explicit errors.
|
||||
|
||||
## Image workflow
|
||||
|
||||
1. Load the **H3 Ref2V** model and the trained upscale LoRA with your normal
|
||||
LoRA loader. Load the official complete H3 video VAE with its encoder and
|
||||
the H3 Qwen3-VL text encoder (`CLIPLoader` type `minimax`).
|
||||
2. Encode the caption without passing the guide image into the VLM. Start
|
||||
with the training prompt:
|
||||
`h3upscale, increase resolution and restore fine details while preserving the original content and visual style.`
|
||||
3. Create a **Guide Target** with width `1024`, height `768`, frames `1`.
|
||||
4. Connect the model after LoRA loading, positive conditioning, target latent,
|
||||
VAE, and source image to **Downscaled Latent Guide**. Set
|
||||
`downscale_factor=4`, `frame_idx=0`. Select the same `lora_name` to validate
|
||||
training metadata; that selector does **not** load the LoRA a second time.
|
||||
5. Connect **both returned MODEL and positive CONDITIONING** to the guider /
|
||||
sampler. Use the returned model for sigma scheduling too. The sampler's
|
||||
`latent_image` stays the target latent from step 3, not a guide latent.
|
||||
6. Decode with the H3 VAE. The third output is the **actual reduced pixel
|
||||
guide preview**; at this canvas it is `256 x 192`.
|
||||
|
||||
An [API-format image workflow template](examples/minimax_h3_upscale_image_api.json)
|
||||
is included. Replace `YOUR_H3_UPSCALE_LORA.safetensors`,
|
||||
`YOUR_INPUT_IMAGE.png`, and model filenames with files installed in your
|
||||
ComfyUI. The template uses seed 42, Euler, 28 steps, full denoise, and BasicGuider.
|
||||
It can be submitted as the `prompt` field of a ComfyUI `/prompt` API request.
|
||||
It is an API graph, not a saved frontend canvas workflow.
|
||||
|
||||
## Video workflow
|
||||
|
||||
Create a target with `frames=73` (approximately 3 seconds at 24 fps) and supply
|
||||
an IMAGE batch of **exactly 73 source video frames** resampled to 24 fps, in
|
||||
chronological order. The video VAE produces 22 latent-time slices. Other valid
|
||||
lengths include 5, 22, 39, 56, 90, 107 and 124; a one-frame batch is an image.
|
||||
The node rejects invalid lengths instead of silently truncating or stretching
|
||||
motion. It does not perform video decoding or FPS conversion itself.
|
||||
|
||||
Keep the same target canvas / guide factor as training. Increasing target
|
||||
resolution or frame count increases target activation memory even though guide
|
||||
spatial tokens are reduced. Start with a known-fitting canvas on the target GPU.
|
||||
To save video, connect decoded frames to your video encoder/combiner at 24 fps.
|
||||
|
||||
An optional `audio` soundtrack plus `audio_vae` can add aligned guide audio.
|
||||
Supply audio starting at the guide clip's first frame. It is cropped to the
|
||||
clip and available target duration, and is never sent to the text encoder.
|
||||
Omit it for visual-only inference. Native reference conditioning and native
|
||||
same-resolution guides can be chained alongside this node; their row positions
|
||||
are retained.
|
||||
|
||||
## Geometry and encoding
|
||||
|
||||
- Each target pixel axis must be a multiple of `32 * downscale_factor`.
|
||||
- The source is fitted to the target canvas with the fork's centered bicubic
|
||||
crop, then downscaled with Lanczos before VAE encoding. This also applies
|
||||
when the supplied source is already small: inspect `guide_preview`.
|
||||
- H3 VAE spatial compression is 16. At output `1024 x 768`, factor 4 yields
|
||||
guide pixels `256 x 192`, guide latents `16 x 12`, and target latents `64 x 48`.
|
||||
- The posterior is sampled with an isolated CPU generator (seed 42) and
|
||||
rounded through FP16 **before** latent normalization, matching the fork's
|
||||
image and video sampling paths. The native VAE mean-only encoding is not used.
|
||||
- An image has one guide latent-time slice; 73 video frames have 22. Video
|
||||
time coordinates use H3's `(1,4,4,4,4)` temporal pattern, not a uniform stride.
|
||||
- The corrected packed layout keeps target/reference/audio coordinates and
|
||||
reindexes condition positions, frozen/update masks and segment boundaries.
|
||||
Downscaled guides receive the target spatial grid at the training stride;
|
||||
coordinates from a separately normalized small grid are not multiplied.
|
||||
|
||||
## Validation and limits
|
||||
|
||||
CPU tests cover the audited native layout, factors 1/2/4, exact crop/resize,
|
||||
posterior sampling, mixed/native guides, references, audio row preservation,
|
||||
73-frame counts, metadata checks, clone isolation and wrapper composition.
|
||||
Golden position fixtures come from the actual AI Toolkit packing implementation.
|
||||
Run `python -m unittest discover -s tests -v`.
|
||||
|
||||
A full pretrained ComfyUI GPU sampling run has not been validated as part of
|
||||
this change. Matching preprocessing and coordinates establishes the conditioning
|
||||
contract; it does not establish the trained LoRA's visual quality. Compare
|
||||
LoRA strength 0 and 1 on held-out inputs before evaluating upscale fidelity.
|
||||
|
||||
## Experimental reference layouts and source phase (1.47.0)
|
||||
|
||||
Optional inputs on Downscaled Guide expose `guide_rope_layout` (overlap/sidecar),
|
||||
`reference_rope_layout` (native/overlap/sidecar), `reference_source_phase`,
|
||||
`reference_phase_scale`, `reference_sidecar_margin`, and `source_id`. Defaults
|
||||
preserve 1.46.0 behavior. H3 sidecar margins use normalized RoPE units, not pixels.
|
||||
Select your LoRA to check these values against `minimax_h3_reference_rope` metadata.
|
||||
Legacy checkpoints imply overlap/native/phase-off. Changing only inference geometry
|
||||
on a legacy LoRA does not establish that the model can use it.
|
||||
|
||||
**MiniMax-H3 Identity Reference + RoPE (BFS)** adds a separate single-image latent
|
||||
reference without passing it to the VLM. Supply the same target latent and full H3
|
||||
VAE. Chain both model and conditioning through Guide and Identity Reference, then
|
||||
connect both final outputs to the guider/sampler (and model to the scheduler).
|
||||
Use guide `source_id=1`, identity `source_id=2` for training control channels 1 and 2.
|
||||
The identity reference keeps its aspect and is resized down only to the trainer's
|
||||
/32 reference-image bucket; the guide factor does not resize the portrait.
|
||||
The reference preview shows the actual pixels used.
|
||||
|
||||
This node requires a checkpoint trained with a matching identity reference role
|
||||
and `control_latent_only: true` for caption-only conditioning. The LoRA selector
|
||||
checks that mode too. It does not load LoRA weights. Adding a portrait to a LoRA
|
||||
trained only with a low-resolution guide is not a validated identity-preservation
|
||||
recipe. Keep caption/source conventions consistent during training.
|
||||
|
||||
Source phase adds the phase `source_id * phase_scale * 10000**(-d/48)` to H3's
|
||||
rotary angles. Text and targets remain source zero. The scoped ModelPatcher object
|
||||
patch composes this phase only when this node's marked conditioning is active;
|
||||
unmarked conditioning and other models keep their original rotary path.
|
||||
|
||||
For a first experiment use overlap plus phase and compare to the phase-off baseline.
|
||||
Sidecar and phase have CPU geometry/rotary parity tests, including mixed image,
|
||||
video and audio, but no full pretrained H3 quality validation yet. They distinguish
|
||||
sources; they do not guarantee the same face. See the
|
||||
[training configuration guide](https://github.com/alisson-anjos/ai-toolkit/blob/minimax-h3-latent-guides/H3_REFERENCE_ROPE.md).
|
||||
@@ -0,0 +1,14 @@
|
||||
|
||||
Portions of this pack are vendored from third-party GPL-3.0 projects:
|
||||
|
||||
bfs_crop_planner.py, bfs_subject_crop.py
|
||||
from drozbay/MaskVidExperiments (https://github.com/drozbay/MaskVidExperiments),
|
||||
commit d98cc89 (v0.2.0), Copyright (c) drozbay, GPL-3.0.
|
||||
Vendored so the head-swap sampler needs no runtime dependency on that pack;
|
||||
behaviour unchanged, ComfyUI node classes dropped.
|
||||
|
||||
tests/fixtures/comfy_h3_layout_8cfe5e1.py
|
||||
CPU-only excerpt of comfy/ldm/minimax/model.py from Comfy-Org/ComfyUI,
|
||||
commit 8cfe5e1ecb97512dea8deaac15e1228d7e6feeb1,
|
||||
Copyright ComfyUI contributors, GPL-3.0. Retained without model weights
|
||||
to verify downscaled guide row counts and coordinates against native H3.
|
||||
@@ -46,6 +46,11 @@ cd ComfyUI-BFSNodes
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Optional: `pip install insightface` with `onnxruntime` or `onnxruntime-gpu`, only for LTX Identity. The Shot Planner's
|
||||
*Find people* / Cast no longer needs it (1.93.0): it uses OpenCV's YuNet + SFace (Apache-2.0, downloaded once), and uses
|
||||
InsightFace only when it is installed. They are not in the requirements because insightface pins packages
|
||||
that conflict with many environments; without them everything else works and those features say what to install.
|
||||
|
||||
Restart ComfyUI after installation.
|
||||
|
||||
## LoRA Placement
|
||||
@@ -107,6 +112,58 @@ These nodes target ComfyUI's LTXV/LTX-Video model implementation. The main apply
|
||||
|
||||
A plain LoRA loader alone is not enough for Edit Anything LoRAs that include role embedding and/or AdaLN reference conditioning. Use the standard LoRA loader first, then pass the result through **LTXV Edit Anything (Apply)**.
|
||||
|
||||
## Reference temporal offset
|
||||
|
||||
Nodes that inject clean appearance references expose a reference-only temporal offset in VAE
|
||||
latent steps. The default is `0`, matching the standard frame-zero placement used by existing
|
||||
LoRAs and preserving their expected behavior. Negative values are an experimental way to reduce
|
||||
the overlap frame-0 artifact where the reference composition briefly appears at the start of the
|
||||
generated video.
|
||||
|
||||
- `0`: standard frame-zero placement (default)
|
||||
- `-1`: one latent step before frame zero (`-8` pixel frames with the standard LTX VAE)
|
||||
- `-2`: two latent steps before frame zero (stronger separation, possibly weaker identity)
|
||||
|
||||
The offset is applied only to appearance/reference inputs. Guide videos, masks, identity masks,
|
||||
and target video positions remain unchanged. In **LTX Multiple Controls**, the field is named
|
||||
`identity_temporal_offset_latents` and affects only `identity_image`.
|
||||
|
||||
## Shot Planner / Shot Loop (1.48.0)
|
||||
|
||||
Split a long video into shots a video model can follow (at camera cuts via PySceneDetect, fixed
|
||||
length, or by hand on a timeline), set a reference image and prompt per shot, run the rest of the
|
||||
workflow once per shot, and join the results back with the original timing and soundtrack.
|
||||
|
||||
- **BFS Shot Planner** — interactive timeline (thumbnails, cut detector curve, draggable
|
||||
boundaries, per-shot reference gallery and prompt). Outputs the shots as a ComfyUI list, so
|
||||
every node downstream runs once per shot.
|
||||
- **BFS Shot Unpack** — guide frames, references, prompt and length of one shot, for any model.
|
||||
- **BFS Shot H3 Conditioning** — MiniMax H3 conditioning for one shot with the native nodes
|
||||
(aligned guide and/or native reference video).
|
||||
- **BFS Shot Join** — concatenates, trims to the true lengths, cross-fades soft joins, returns audio.
|
||||
|
||||
**Cast (1.49.0)**: *Find people* tracks faces through the video (InsightFace) and groups them
|
||||
into people. Link each person to a reference and every shot uses the reference of the person
|
||||
with the most screen time in it; optionally split where the main person changes and run only the
|
||||
shots where a linked person appears.
|
||||
|
||||
Two run modes: *auto loop* (all shots in one run) and *queue loop* (one shot per run, stored on
|
||||
disk, re-queued automatically; nodes after the join only run on the last shot). See
|
||||
[`web/js/docs/BFSShotPlanner.md`](web/js/docs/BFSShotPlanner.md).
|
||||
|
||||
Shots can also be cropped to a **SAM 3 mask** (text or points per shot) and pasted back by the join, continue
|
||||
from the previous shot's result, and get **VLM suggestions** (what to segment, a shot description for the
|
||||
prompt, run/skip) from a Qwen3-VL connected to the planner.
|
||||
|
||||
## H3 Duet / Side Panel (1.50.0)
|
||||
|
||||
Training-free split-screen generation for MiniMax H3 (TSC's latent pin): a panel (the source clip
|
||||
or a reference) is pinned beside the video with a noise mask, the video is generated in sync with
|
||||
it, and the panel is cropped off before decoding. **BFS H3 Duet** does it all in one node, **BFS Shot H3 Duet** renders each shot of the Shot Loop that way (or with an aligned guide for body-swap LoRAs);
|
||||
**BFS H3 Side Panel** + **BFS H3 Side Panel Crop** are the building blocks. Optional aligned latent
|
||||
guide. 1.84.0: `panel_static` (TV static over the face / person of the pinned clip, SAM 3) and a `setting`
|
||||
picture (**BFS Setting Picture**) for same-world swaps. See [`web/js/docs/BFSH3Duet.md`](web/js/docs/BFSH3Duet.md).
|
||||
|
||||
## Requirements
|
||||
|
||||
- ComfyUI
|
||||
@@ -118,3 +175,19 @@ A plain LoRA loader alone is not enough for Edit Anything LoRAs that include rol
|
||||
## License
|
||||
|
||||
This project follows the repository license. See [LICENSE](LICENSE).
|
||||
|
||||
## MiniMax-H3 downscaled latent guides (1.46.0)
|
||||
|
||||
**MiniMax-H3 Downscaled Latent Guide (BFS)** supports aligned lower-resolution
|
||||
image/video guides for H3 LoRAs trained with `target_grid_stride_v1`. It encodes
|
||||
the smaller guide and patches only the returned model's packed coordinates.
|
||||
Connect both its **MODEL** and **positive CONDITIONING** outputs to sampling.
|
||||
|
||||
**MiniMax-H3 Guide Target — Image / Video (BFS)** creates an exact output canvas:
|
||||
use one frame for images or 73 frames for approximately three seconds at 24 fps.
|
||||
At output 1024 x 768 and factor 4, the guide preview is 256 x 192.
|
||||
|
||||
See [setup, workflow template, compatibility and validation](MINIMAX_H3_GUIDES.md).
|
||||
|
||||
BFSNodes **1.47.0** adds optional MiniMax-H3 overlap/sidecar source-phase RoPE and
|
||||
**MiniMax-H3 Identity Reference + RoPE (BFS)**. See [the H3 guide](MINIMAX_H3_GUIDES.md#experimental-reference-layouts-and-source-phase-1470).
|
||||
|
||||
+152
@@ -6,17 +6,169 @@ from .headswap_node import NODE_CLASS_MAPPINGS as HEADSWAP_NODE_CLASS_MAPPINGS
|
||||
from .headswap_node import NODE_DISPLAY_NAME_MAPPINGS as HEADSWAP_NODE_DISPLAY_NAME_MAPPINGS
|
||||
from .anime2real_node import NODE_CLASS_MAPPINGS as A2R_NODE_CLASS_MAPPINGS
|
||||
from .anime2real_node import NODE_DISPLAY_NAME_MAPPINGS as A2R_NODE_DISPLAY_NAME_MAPPINGS
|
||||
from .amv_guide_node import NODE_CLASS_MAPPINGS as AMV_NODE_CLASS_MAPPINGS
|
||||
from .amv_guide_node import NODE_DISPLAY_NAME_MAPPINGS as AMV_NODE_DISPLAY_NAME_MAPPINGS
|
||||
from .ltx_identity_overlap import NODE_CLASS_MAPPINGS as IDT_NODE_CLASS_MAPPINGS
|
||||
from .ltx_identity_overlap import NODE_DISPLAY_NAME_MAPPINGS as IDT_NODE_DISPLAY_NAME_MAPPINGS
|
||||
from .ltx_multiref_slots import NODE_CLASS_MAPPINGS as MRSLOT_NODE_CLASS_MAPPINGS
|
||||
from .ltx_multiref_slots import NODE_DISPLAY_NAME_MAPPINGS as MRSLOT_NODE_DISPLAY_NAME_MAPPINGS
|
||||
from .ltx_multiple_controls import NODE_CLASS_MAPPINGS as MC_NODE_CLASS_MAPPINGS
|
||||
from .ltx_multiple_controls import NODE_DISPLAY_NAME_MAPPINGS as MC_NODE_DISPLAY_NAME_MAPPINGS
|
||||
from .color_mask_node import NODE_CLASS_MAPPINGS as CM_NODE_CLASS_MAPPINGS
|
||||
from .color_mask_node import NODE_DISPLAY_NAME_MAPPINGS as CM_NODE_DISPLAY_NAME_MAPPINGS
|
||||
try:
|
||||
from .minimax_h3_singleframe_vae import NODE_CLASS_MAPPINGS as MMVAE_NODE_CLASS_MAPPINGS
|
||||
from .minimax_h3_singleframe_vae import NODE_DISPLAY_NAME_MAPPINGS as MMVAE_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] MiniMax-H3 VAE loaders not loaded: {_e!r}")
|
||||
MMVAE_NODE_CLASS_MAPPINGS, MMVAE_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
try:
|
||||
from .sharpest_frame import NODE_CLASS_MAPPINGS as SHARP_NODE_CLASS_MAPPINGS
|
||||
from .sharpest_frame import NODE_DISPLAY_NAME_MAPPINGS as SHARP_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] Sharpest Frame node not loaded: {_e!r}")
|
||||
SHARP_NODE_CLASS_MAPPINGS, SHARP_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
try:
|
||||
from .minimax_h3_direct_decode import NODE_CLASS_MAPPINGS as MMDD_NODE_CLASS_MAPPINGS
|
||||
from .minimax_h3_direct_decode import NODE_DISPLAY_NAME_MAPPINGS as MMDD_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] MiniMax-H3 direct decode node not loaded: {_e!r}")
|
||||
MMDD_NODE_CLASS_MAPPINGS, MMDD_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
try:
|
||||
from .save_h3_latent import NODE_CLASS_MAPPINGS as SVH3_NODE_CLASS_MAPPINGS
|
||||
from .save_h3_latent import NODE_DISPLAY_NAME_MAPPINGS as SVH3_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] Save H3 Latent node not loaded: {_e!r}")
|
||||
SVH3_NODE_CLASS_MAPPINGS, SVH3_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
try:
|
||||
from .edit_prompt_fixer import NODE_CLASS_MAPPINGS as EPF_NODE_CLASS_MAPPINGS
|
||||
from .edit_prompt_fixer import NODE_DISPLAY_NAME_MAPPINGS as EPF_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] Edit Anything Prompt Fixer not loaded: {_e!r}")
|
||||
EPF_NODE_CLASS_MAPPINGS, EPF_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
try:
|
||||
from .ltx_identity_multiangle import NODE_CLASS_MAPPINGS as MA_NODE_CLASS_MAPPINGS
|
||||
from .ltx_identity_multiangle import NODE_DISPLAY_NAME_MAPPINGS as MA_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] LTX Identity Multiple Angles node not loaded: {_e!r}")
|
||||
MA_NODE_CLASS_MAPPINGS, MA_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
try:
|
||||
from .ltx_identity_gemma_vision import NODE_CLASS_MAPPINGS as GV_NODE_CLASS_MAPPINGS
|
||||
from .ltx_identity_gemma_vision import NODE_DISPLAY_NAME_MAPPINGS as GV_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] LTX Identity Gemma-Vision node not loaded: {_e!r}")
|
||||
GV_NODE_CLASS_MAPPINGS, GV_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
try:
|
||||
from .multiref_sheet_node import NODE_CLASS_MAPPINGS as MRS_NODE_CLASS_MAPPINGS
|
||||
from .multiref_sheet_node import NODE_DISPLAY_NAME_MAPPINGS as MRS_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] Multi-Ref Sheet Builder node not loaded: {_e!r}")
|
||||
MRS_NODE_CLASS_MAPPINGS, MRS_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
try:
|
||||
from .ltx_multishot_prompt import NODE_CLASS_MAPPINGS as MSP_NODE_CLASS_MAPPINGS
|
||||
from .ltx_multishot_prompt import NODE_DISPLAY_NAME_MAPPINGS as MSP_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] LTX Multishot Prompt node not loaded: {_e!r}")
|
||||
MSP_NODE_CLASS_MAPPINGS, MSP_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
# CAN / AdaLN node disabled: empirically the AdaLN modulation degrades the video (the identity
|
||||
# gain came from the projector + LoRA, not the CAN). Kept the file but not registered.
|
||||
CAN_NODE_CLASS_MAPPINGS, CAN_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
|
||||
try:
|
||||
from .bfs_h3_side_panel import NODE_CLASS_MAPPINGS as PANEL_NODE_CLASS_MAPPINGS
|
||||
from .bfs_h3_side_panel import NODE_DISPLAY_NAME_MAPPINGS as PANEL_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] H3 side panel nodes unavailable: {_e}")
|
||||
PANEL_NODE_CLASS_MAPPINGS, PANEL_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
try:
|
||||
from .bfs_shot_loop import NODE_CLASS_MAPPINGS as SHOT_NODE_CLASS_MAPPINGS
|
||||
from .bfs_shot_loop import NODE_DISPLAY_NAME_MAPPINGS as SHOT_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] Shot loop nodes not loaded: {_e!r}")
|
||||
SHOT_NODE_CLASS_MAPPINGS, SHOT_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
try:
|
||||
from .bfs_shot_noise import NODE_CLASS_MAPPINGS as SNOISE_NODE_CLASS_MAPPINGS
|
||||
from .bfs_shot_noise import NODE_DISPLAY_NAME_MAPPINGS as SNOISE_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] BFS Shot Noise node not loaded: {_e!r}")
|
||||
SNOISE_NODE_CLASS_MAPPINGS, SNOISE_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
try:
|
||||
from .minimax_h3_downscaled_guide import NODE_CLASS_MAPPINGS as H3GUIDE_NODE_CLASS_MAPPINGS
|
||||
from .minimax_h3_downscaled_guide import NODE_DISPLAY_NAME_MAPPINGS as H3GUIDE_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] MiniMax-H3 downscaled guide nodes not loaded: {_e!r}")
|
||||
H3GUIDE_NODE_CLASS_MAPPINGS, H3GUIDE_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
|
||||
try:
|
||||
from .headswap_ltx_masked_sampler import NODE_CLASS_MAPPINGS as HSMS_NODE_CLASS_MAPPINGS
|
||||
from .headswap_ltx_masked_sampler import NODE_DISPLAY_NAME_MAPPINGS as HSMS_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] BFS Sampler not loaded: {_e!r}")
|
||||
HSMS_NODE_CLASS_MAPPINGS, HSMS_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
|
||||
try:
|
||||
from .bfs_lora_surgery import NODE_CLASS_MAPPINGS as LSURG_NODE_CLASS_MAPPINGS
|
||||
from .bfs_lora_surgery import NODE_DISPLAY_NAME_MAPPINGS as LSURG_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] LoRA Surgery node not loaded: {_e!r}")
|
||||
LSURG_NODE_CLASS_MAPPINGS, LSURG_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
**H3GUIDE_NODE_CLASS_MAPPINGS,
|
||||
**SHOT_NODE_CLASS_MAPPINGS,
|
||||
**PANEL_NODE_CLASS_MAPPINGS,
|
||||
**GV_NODE_CLASS_MAPPINGS,
|
||||
**CAN_NODE_CLASS_MAPPINGS,
|
||||
**BFS_NODE_CLASS_MAPPINGS,
|
||||
**LTXV_EA_NODE_CLASS_MAPPINGS,
|
||||
**HEADSWAP_NODE_CLASS_MAPPINGS,
|
||||
**A2R_NODE_CLASS_MAPPINGS,
|
||||
**AMV_NODE_CLASS_MAPPINGS,
|
||||
**IDT_NODE_CLASS_MAPPINGS,
|
||||
**MC_NODE_CLASS_MAPPINGS,
|
||||
**MRSLOT_NODE_CLASS_MAPPINGS,
|
||||
**MA_NODE_CLASS_MAPPINGS,
|
||||
**MRS_NODE_CLASS_MAPPINGS,
|
||||
**CM_NODE_CLASS_MAPPINGS,
|
||||
**MSP_NODE_CLASS_MAPPINGS,
|
||||
**MMVAE_NODE_CLASS_MAPPINGS,
|
||||
**SHARP_NODE_CLASS_MAPPINGS,
|
||||
**SNOISE_NODE_CLASS_MAPPINGS,
|
||||
**MMDD_NODE_CLASS_MAPPINGS,
|
||||
**SVH3_NODE_CLASS_MAPPINGS,
|
||||
**EPF_NODE_CLASS_MAPPINGS,
|
||||
**HSMS_NODE_CLASS_MAPPINGS,
|
||||
**LSURG_NODE_CLASS_MAPPINGS,
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
**H3GUIDE_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**SHOT_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**PANEL_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**GV_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**CAN_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**BFS_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**LTXV_EA_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**HEADSWAP_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**A2R_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**AMV_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**IDT_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**MC_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**MRSLOT_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**MA_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**MRS_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**CM_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**MSP_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**MMVAE_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**SHARP_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**SNOISE_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**MMDD_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**SVH3_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**EPF_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**HSMS_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**LSURG_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
}
|
||||
|
||||
WEB_DIRECTORY = "./web/js"
|
||||
|
||||
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS", "WEB_DIRECTORY"]
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
"""AMV Guide Builder — make the keyframes + blank guide video for the AMV av2av LoRA.
|
||||
|
||||
Modes (where the keyframe IMAGES come from):
|
||||
- auto_scene: input video -> detect scene cuts -> keyframe at each cut
|
||||
- manual_indices: input video + indices "0,16,34" -> those frames as keyframes
|
||||
- external_images: a batch of images (chain Load Image -> Image Batch) -> placed on the timeline
|
||||
|
||||
Placement (WHERE keyframes go, for external_images):
|
||||
- audio_beats: detect beats/onsets in the input `audio` and drop one image per beat (in order,
|
||||
cycling the images if there are more beats than images). This syncs the cuts to
|
||||
the music — exactly how AMVs are edited.
|
||||
- even: evenly spaced across `length`
|
||||
- indices: explicit `indices` string
|
||||
|
||||
Output is an IMAGE batch [length,H,W,C] (float 0..1), ready for VAE-encode / reference conditioning.
|
||||
`hold` repeats each keyframe N frames so it survives the LTX 8x temporal VAE.
|
||||
"""
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
CATEGORY = "BFS/video"
|
||||
|
||||
|
||||
def _parse_indices(s):
|
||||
out = []
|
||||
for tok in str(s).replace(";", ",").split(","):
|
||||
tok = tok.strip()
|
||||
if tok.isdigit():
|
||||
out.append(int(tok))
|
||||
return out
|
||||
|
||||
|
||||
def _auto_cuts(video, threshold):
|
||||
"""Content-diff scene cuts on [N,H,W,C]. Returns sorted cut indices (incl 0)."""
|
||||
cuts = [0]
|
||||
prev = video[0]
|
||||
for i in range(1, video.shape[0]):
|
||||
if (video[i] - prev).abs().mean().item() > threshold:
|
||||
cuts.append(i)
|
||||
prev = video[i]
|
||||
return cuts
|
||||
|
||||
|
||||
def _audio_to_mono(audio):
|
||||
wf = audio["waveform"] # [B, C, samples]
|
||||
sr = int(audio["sample_rate"])
|
||||
y = wf[0].mean(0).detach().cpu().numpy().astype(np.float32)
|
||||
return y, sr
|
||||
|
||||
|
||||
def _beat_indices(audio, fps, length):
|
||||
"""Beat (or onset) frame indices for the guide timeline. librosa if available, else RMS-onset."""
|
||||
y, sr = _audio_to_mono(audio)
|
||||
times = None
|
||||
try:
|
||||
import librosa
|
||||
_, beats = librosa.beat.beat_track(y=y, sr=sr)
|
||||
times = librosa.frames_to_time(beats, sr=sr)
|
||||
if len(times) < 2: # fall back to onsets for sparse beats
|
||||
on = librosa.onset.onset_detect(y=y, sr=sr, units="time")
|
||||
if len(on) > len(times):
|
||||
times = on
|
||||
except Exception:
|
||||
# fallback: RMS energy peaks
|
||||
hop = max(1, sr // 100)
|
||||
rms = np.array([np.sqrt(np.mean(y[i:i + hop] ** 2)) for i in range(0, len(y), hop)])
|
||||
d = np.diff(rms, prepend=rms[:1])
|
||||
thr = d.mean() + d.std()
|
||||
peaks = np.where(d > thr)[0]
|
||||
times = peaks * hop / sr
|
||||
idx = sorted({int(round(t * fps)) for t in times if 0 <= t * fps < length})
|
||||
if 0 not in idx:
|
||||
idx = [0] + idx
|
||||
return idx
|
||||
|
||||
|
||||
class AmvGuideBuilder:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"mode": (["auto_scene", "manual_indices", "external_images"],),
|
||||
"length": ("INT", {"default": 121, "min": 1, "max": 100000}),
|
||||
"hold": ("INT", {"default": 4, "min": 1, "max": 64,
|
||||
"tooltip": "Frames each keyframe is shown (>=4 survives LTX 8x temporal VAE)."}),
|
||||
"fill": (["white", "black"],),
|
||||
"placement": (["audio_beats", "even", "indices"],
|
||||
{"tooltip": "external_images: how to position the images on the timeline."}),
|
||||
"fps": ("FLOAT", {"default": 25.0, "min": 1.0, "max": 120.0,
|
||||
"tooltip": "Used to convert audio beat times -> frame indices."}),
|
||||
},
|
||||
"optional": {
|
||||
"video": ("IMAGE", {"tooltip": "Source frames (auto_scene / manual_indices)."}),
|
||||
"images": ("IMAGE", {"tooltip": "Keyframe images batch (external_images)."}),
|
||||
"audio": ("AUDIO", {"tooltip": "Drives placement=audio_beats (sync cuts to the music)."}),
|
||||
"indices": ("STRING", {"default": "", "tooltip": "Comma list e.g. '0,16,34' (placement=indices / manual)."}),
|
||||
"scene_threshold": ("FLOAT", {"default": 0.10, "min": 0.0, "max": 1.0, "step": 0.01}),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "STRING")
|
||||
RETURN_NAMES = ("guide", "debug")
|
||||
FUNCTION = "build"
|
||||
CATEGORY = CATEGORY
|
||||
DESCRIPTION = ("Builds the AMV guide (keyframes + blank). external_images can auto-place images "
|
||||
"on the music's beats (placement=audio_beats). Feed `guide` into VAE/reference.")
|
||||
|
||||
def build(self, mode, length, hold, fill, placement, fps,
|
||||
video=None, images=None, audio=None, indices="", scene_threshold=0.10):
|
||||
idx = _parse_indices(indices)
|
||||
keyframes = [] # (frame_index, image[H,W,C])
|
||||
|
||||
if mode == "external_images":
|
||||
if images is None or images.shape[0] == 0:
|
||||
raise ValueError("external_images mode needs an `images` batch.")
|
||||
H, W, n = images.shape[1], images.shape[2], images.shape[0]
|
||||
# decide placement positions
|
||||
if placement == "audio_beats":
|
||||
if audio is None:
|
||||
raise ValueError("placement=audio_beats needs an `audio` input.")
|
||||
pos = _beat_indices(audio, fps, length)
|
||||
elif placement == "indices" and idx:
|
||||
pos = idx
|
||||
else: # even
|
||||
pos = [round(k * (length - 1) / max(1, n - 1)) for k in range(n)] if n > 1 else [0]
|
||||
# map one image per position (cycle images if fewer than positions)
|
||||
for k, at in enumerate(pos):
|
||||
keyframes.append((min(int(at), length - 1), images[k % n]))
|
||||
else:
|
||||
if video is None or video.shape[0] == 0:
|
||||
raise ValueError(f"{mode} mode needs a `video`.")
|
||||
H, W = video.shape[1], video.shape[2]
|
||||
cuts = _auto_cuts(video, scene_threshold) if mode == "auto_scene" else (idx or [0])
|
||||
for ci in cuts:
|
||||
if 0 <= ci < video.shape[0]:
|
||||
keyframes.append((min(ci, length - 1), video[ci]))
|
||||
|
||||
val = 1.0 if fill == "white" else 0.0
|
||||
guide = torch.full((length, H, W, 3), val, dtype=torch.float32)
|
||||
painted = []
|
||||
for at, frame in keyframes:
|
||||
f = frame[..., :3].to(torch.float32)
|
||||
for h in range(hold):
|
||||
if at + h < length:
|
||||
guide[at + h] = f
|
||||
painted.append(at)
|
||||
|
||||
dbg = (f"AMV guide | mode={mode} placement={placement if mode=='external_images' else '-'} | "
|
||||
f"{W}x{H} {length}f hold={hold} fill={fill} | {len(keyframes)} keyframes @ {sorted(set(painted))}")
|
||||
return (guide, dbg)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"BFSAmvGuideBuilder": AmvGuideBuilder}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"BFSAmvGuideBuilder": "AMV Guide Builder"}
|
||||
@@ -0,0 +1,528 @@
|
||||
"""Crop-path planner, vendored from drozbay/MaskVidExperiments.
|
||||
|
||||
Source: https://github.com/drozbay/MaskVidExperiments (commit d98cc89, v0.2.0)
|
||||
Copyright (c) drozbay. Licensed GPL-3.0, the same licence this pack carries,
|
||||
which is what makes vendoring it legal here.
|
||||
|
||||
Vendored rather than imported so the head-swap sampler has no runtime
|
||||
dependency on that pack being installed. Behaviour is unchanged; only the
|
||||
ComfyUI node classes were dropped (they stay in the original pack, where they
|
||||
belong) and the planner import was repointed.
|
||||
|
||||
Why this code and not a crop of my own: naive per-frame crops around a mask
|
||||
jitter in position and size, and a video model reads that jitter as camera
|
||||
motion. These boxes are planned over the whole clip, so they hold still through
|
||||
mask noise and occlusion and follow only sustained movement.
|
||||
"""
|
||||
|
||||
"""Subject crop planner: shape as an optimization output.
|
||||
|
||||
Standalone numpy/scipy planner solved against binary masks; produces the
|
||||
per-frame crop boxes only.
|
||||
|
||||
Structure per mode:
|
||||
tracked -- one LP: scalar W, H + per-frame positions. Shape emerges.
|
||||
zoomed -- outer sweep over constant aspect candidates, inner LP per
|
||||
candidate: per-frame height (width slaved to aspect) +
|
||||
positions. Rent is charged against the tightest
|
||||
aspect-respecting padded box, never per axis.
|
||||
combined -- closed form: padded union box (also the movement_cost -> inf
|
||||
limit of the other two).
|
||||
|
||||
Dials (p):
|
||||
crop_scale padding promise as a multiple of the subject
|
||||
min_padding_allowed fraction of the promise enforced as hard floors
|
||||
min_padding_allowed_window odd median window defining sustained motion
|
||||
pad_deficit_tol shortfall above the floor costs 1/this per px-frame
|
||||
pad_surplus_tol size beyond the padded need costs 1/this per px-frame
|
||||
resize_cost cost per pixel of box size change (zoomed)
|
||||
movement_cost cost per pixel of box movement
|
||||
center_pull tie-break pull toward the subject center
|
||||
end_tightening fraction of rent charged on the phantoms
|
||||
end_tightening_window phantom frames held at each end
|
||||
zoom_step >1 quantizes sizes to geometric levels (zoomed)
|
||||
max_zoom_rate >1 caps per-frame size ratio change (zoomed), 0/1 off
|
||||
aspect_ratio >0 fixes the shape and skips the sweep
|
||||
seamless_loop periodic plan (positions and sizes wrap)
|
||||
"""
|
||||
|
||||
import math
|
||||
|
||||
import numpy as np
|
||||
from scipy import ndimage, sparse
|
||||
from scipy.optimize import linprog
|
||||
|
||||
|
||||
def _extent_tracks(binary):
|
||||
"""Per-frame subject extents, NaN gaps interpolated and edge-held."""
|
||||
n = binary.shape[0]
|
||||
bx0 = np.full(n, np.nan)
|
||||
bx1 = np.full(n, np.nan)
|
||||
by0 = np.full(n, np.nan)
|
||||
by1 = np.full(n, np.nan)
|
||||
for i, frame in enumerate(binary):
|
||||
ys, xs = frame.nonzero()
|
||||
if len(xs):
|
||||
bx0[i], bx1[i] = xs.min(), xs.max()
|
||||
by0[i], by1[i] = ys.min(), ys.max()
|
||||
known = ~np.isnan(bx0)
|
||||
idx = np.arange(n)
|
||||
out = [np.interp(idx, idx[known], t[known]) for t in (bx0, bx1, by0, by1)]
|
||||
return out, known
|
||||
|
||||
|
||||
def _median_tracks(tracks, window):
|
||||
g = max(int(window), 1) | 1
|
||||
return [ndimage.median_filter(t, size=g, mode="nearest") for t in tracks]
|
||||
|
||||
|
||||
def _round_corridor(t, lo, hi):
|
||||
ilo = np.ceil(lo - 1e-6)
|
||||
ihi = np.floor(hi + 1e-6)
|
||||
x = np.clip(np.round(t), ilo, ihi)
|
||||
return np.where(ilo > ihi, np.round((lo + hi) / 2), x).astype(int)
|
||||
|
||||
|
||||
class _LP:
|
||||
"""Row-wise builder for sparse linprog problems."""
|
||||
|
||||
def __init__(self):
|
||||
self.blocks = [] # (name, size)
|
||||
self.off = {}
|
||||
self.nv = 0
|
||||
self.ri, self.ci, self.vv, self.rhs = [], [], [], []
|
||||
self.c = []
|
||||
self.lo, self.hi = [], []
|
||||
|
||||
def var(self, name, size, cost=0.0, lo=0.0, hi=None):
|
||||
self.off[name] = self.nv
|
||||
self.nv += size
|
||||
cost = np.broadcast_to(np.asarray(cost, dtype=float), (size,))
|
||||
lo = np.broadcast_to(np.asarray(lo, dtype=float), (size,))
|
||||
self.c.extend(cost)
|
||||
self.lo.extend(lo)
|
||||
if hi is None:
|
||||
self.hi.extend([None] * size)
|
||||
else:
|
||||
hi_arr = np.broadcast_to(np.asarray(hi, dtype=float), (size,))
|
||||
self.hi.extend(hi_arr)
|
||||
return self.off[name]
|
||||
|
||||
def row(self, cols, vals, b):
|
||||
r = len(self.rhs)
|
||||
self.ri.extend([r] * len(cols))
|
||||
self.ci.extend(cols)
|
||||
self.vv.extend(vals)
|
||||
self.rhs.append(b)
|
||||
|
||||
def solve(self, eq=None):
|
||||
A = sparse.csr_matrix((self.vv, (self.ri, self.ci)),
|
||||
shape=(len(self.rhs), self.nv))
|
||||
bounds = [(l, h) for l, h in zip(self.lo, self.hi)]
|
||||
kw = {}
|
||||
if eq is not None:
|
||||
er, ec, ev, eb = eq
|
||||
kw["A_eq"] = sparse.csr_matrix((ev, (er, ec)),
|
||||
shape=(max(er) + 1, self.nv))
|
||||
kw["b_eq"] = np.asarray(eb, dtype=float)
|
||||
res = linprog(np.array(self.c), A_ub=A,
|
||||
b_ub=np.array(self.rhs, dtype=float),
|
||||
bounds=bounds, method="highs", **kw)
|
||||
if not res.success:
|
||||
raise RuntimeError(f"crop planner solve failed: {res.message}")
|
||||
return res
|
||||
|
||||
|
||||
def _prep(binary, img_w, img_h, p):
|
||||
"""Shared preprocessing: extent tracks, phantoms, pads, floors."""
|
||||
n = binary.shape[0]
|
||||
(bx0, bx1, by0, by1), known = _extent_tracks(binary)
|
||||
loop = bool(p.get("seamless_loop")) and n > 1
|
||||
s0 = int(p.get("end_tightening_window", 0)) if not loop else 0
|
||||
if s0:
|
||||
hold = lambda a: np.concatenate(
|
||||
[np.full(s0, a[0]), a, np.full(s0, a[-1])])
|
||||
bx0, bx1, by0, by1 = hold(bx0), hold(bx1), hold(by0), hold(by1)
|
||||
m = len(bx0)
|
||||
scale = p["crop_scale"]
|
||||
padx = (scale - 1) / 2 * (bx1 - bx0 + 1)
|
||||
pady = (scale - 1) / 2 * (by1 - by0 + 1)
|
||||
|
||||
floor = float(p.get("min_padding_allowed", 0.0))
|
||||
sx0, sx1, sy0, sy1 = _median_tracks([bx0, bx1, by0, by1],
|
||||
p.get("min_padding_allowed_window", 16))
|
||||
pad_fx = floor * (scale - 1) / 2 * (sx1 - sx0 + 1)
|
||||
pad_fy = floor * (scale - 1) / 2 * (sy1 - sy0 + 1)
|
||||
# floor datums, clamped to the image (a side pinned at the image edge
|
||||
# is exempt from its promise)
|
||||
flo_x = np.minimum(sx1 + 1 + pad_fx, img_w)
|
||||
fhi_x = np.maximum(sx0 - pad_fx, 0)
|
||||
flo_y = np.minimum(sy1 + 1 + pad_fy, img_h)
|
||||
fhi_y = np.maximum(sy0 - pad_fy, 0)
|
||||
|
||||
# soft targets: the full promise, clamped to the image
|
||||
lx = np.maximum(bx0 - padx, 0)
|
||||
rx = np.minimum(bx1 + 1 + padx, img_w)
|
||||
ly = np.maximum(by0 - pady, 0)
|
||||
ry = np.minimum(by1 + 1 + pady, img_h)
|
||||
|
||||
rent_scale = np.ones(m)
|
||||
if s0:
|
||||
rent_scale[:s0] = p.get("end_tightening", 0.0)
|
||||
rent_scale[m - s0:] = p.get("end_tightening", 0.0)
|
||||
|
||||
return {
|
||||
"n": n, "m": m, "s0": s0, "loop": loop, "known": known,
|
||||
"bx0": bx0, "bx1": bx1, "by0": by0, "by1": by1,
|
||||
"padx": padx, "pady": pady, "floor": floor,
|
||||
"flo_x": flo_x, "fhi_x": fhi_x, "flo_y": flo_y, "fhi_y": fhi_y,
|
||||
"lx": lx, "rx": rx, "ly": ly, "ry": ry, "rent_scale": rent_scale,
|
||||
}
|
||||
|
||||
|
||||
def _tv_rows(lp, var, tv, m, loop):
|
||||
nt = m if loop else m - 1
|
||||
for i in range(nt):
|
||||
a, b = lp.off[var] + i, lp.off[var] + (i + 1) % m
|
||||
t = lp.off[tv] + i
|
||||
lp.row([b, a, t], [1, -1, -1], 0)
|
||||
lp.row([b, a, t], [-1, 1, -1], 0)
|
||||
|
||||
|
||||
def _solve_tracked(pre, img_w, img_h, p):
|
||||
"""Scalar-size LP: W, H are single variables, shape emerges from the
|
||||
movement/rent balance. Manual aspect adds one equality."""
|
||||
m, loop = pre["m"], pre["loop"]
|
||||
nt = m if loop else m - 1
|
||||
bx0, bx1, by0, by1 = pre["bx0"], pre["bx1"], pre["by0"], pre["by1"]
|
||||
padx, pady = pre["padx"], pre["pady"]
|
||||
soft = 1.0 / max(p.get("pad_deficit_tol", 16), 1e-6)
|
||||
rent = pre["rent_scale"] / max(p.get("pad_surplus_tol", 16), 1e-6)
|
||||
mc = p.get("movement_cost", 1.0)
|
||||
eps = p.get("center_pull", 1e-4)
|
||||
extw = bx1 - bx0 + 1
|
||||
exth = by1 - by0 + 1
|
||||
|
||||
lp = _LP()
|
||||
lp.var("W", 1, lo=max(extw.max(), 1), hi=img_w)
|
||||
lp.var("H", 1, lo=max(exth.max(), 1), hi=img_h)
|
||||
ub_x = np.maximum(bx0, 0)
|
||||
ub_y = np.maximum(by0, 0)
|
||||
if pre["floor"] > 0:
|
||||
ub_x = np.minimum(ub_x, pre["fhi_x"])
|
||||
ub_y = np.minimum(ub_y, pre["fhi_y"])
|
||||
lp.var("x", m, lo=0.0, hi=ub_x)
|
||||
lp.var("y", m, lo=0.0, hi=ub_y)
|
||||
lp.var("tx", nt, cost=mc)
|
||||
lp.var("ty", nt, cost=mc)
|
||||
lp.var("vx", m, cost=soft)
|
||||
lp.var("vy", m, cost=soft)
|
||||
lp.var("rx", m, cost=rent)
|
||||
lp.var("ry", m, cost=rent)
|
||||
lp.var("ux", m, cost=eps)
|
||||
lp.var("uy", m, cost=eps)
|
||||
|
||||
_tv_rows(lp, "x", "tx", m, loop)
|
||||
_tv_rows(lp, "y", "ty", m, loop)
|
||||
W, H = lp.off["W"], lp.off["H"]
|
||||
for i in range(m):
|
||||
x, y = lp.off["x"] + i, lp.off["y"] + i
|
||||
vx, vy = lp.off["vx"] + i, lp.off["vy"] + i
|
||||
rx, ry = lp.off["rx"] + i, lp.off["ry"] + i
|
||||
ux, uy = lp.off["ux"] + i, lp.off["uy"] + i
|
||||
lp.row([x, W], [-1, -1], -(bx1[i] + 1)) # containment right
|
||||
lp.row([x, W], [1, 1], img_w) # image right
|
||||
lp.row([y, H], [-1, -1], -(by1[i] + 1))
|
||||
lp.row([y, H], [1, 1], img_h)
|
||||
lp.row([x, vx], [1, -1], pre["lx"][i]) # soft pad, left
|
||||
lp.row([x, W, vx], [-1, -1, -1], -pre["rx"][i]) # soft pad, right
|
||||
lp.row([y, vy], [1, -1], pre["ly"][i])
|
||||
lp.row([y, H, vy], [-1, -1, -1], -pre["ry"][i])
|
||||
lp.row([W, rx], [1, -1], min(extw[i] + 2 * padx[i], img_w))
|
||||
lp.row([H, ry], [1, -1], min(exth[i] + 2 * pady[i], img_h))
|
||||
if pre["floor"] > 0:
|
||||
lp.row([x, W], [-1, -1], -pre["flo_x"][i])
|
||||
lp.row([y, H], [-1, -1], -pre["flo_y"][i])
|
||||
cx = (bx0[i] + bx1[i] + 1) / 2
|
||||
cy = (by0[i] + by1[i] + 1) / 2
|
||||
lp.row([x, W, ux], [1, 0.5, -1], cx)
|
||||
lp.row([x, W, ux], [-1, -0.5, -1], -cx)
|
||||
lp.row([y, H, uy], [1, 0.5, -1], cy)
|
||||
lp.row([y, H, uy], [-1, -0.5, -1], -cy)
|
||||
|
||||
eq = None
|
||||
ar = p.get("aspect_ratio", 0.0)
|
||||
if ar > 0:
|
||||
eq = ([0, 0], [W, H], [1.0, -ar], [0.0])
|
||||
res = lp.solve(eq=eq)
|
||||
o = lp.off
|
||||
return (res.x[W], res.x[H],
|
||||
res.x[o["x"]:o["x"] + m], res.x[o["y"]:o["y"] + m])
|
||||
|
||||
|
||||
def _yielded_floors(pre, ar, img_w, img_h):
|
||||
"""Relax the floor datums per side wherever containment, the image, or
|
||||
the aspect lock leaves no room."""
|
||||
bx0, bx1, by0, by1 = pre["bx0"], pre["bx1"], pre["by0"], pre["by1"]
|
||||
wmax = min(img_w, ar * img_h)
|
||||
hmax = min(img_h, img_w / ar)
|
||||
flo_x, fhi_x = pre["flo_x"].copy(), pre["fhi_x"].copy()
|
||||
flo_y, fhi_y = pre["flo_y"].copy(), pre["fhi_y"].copy()
|
||||
ex = np.maximum(flo_x - fhi_x - wmax, 0)
|
||||
flo_x = np.maximum(flo_x - ex / 2, bx1 + 1)
|
||||
fhi_x = np.minimum(fhi_x + ex / 2, np.maximum(bx0, 0))
|
||||
flo_x = np.minimum(flo_x, np.maximum(bx0, 0) + wmax)
|
||||
fhi_x = np.maximum(fhi_x, bx1 + 1 - wmax)
|
||||
ey = np.maximum(flo_y - fhi_y - hmax, 0)
|
||||
flo_y = np.maximum(flo_y - ey / 2, by1 + 1)
|
||||
fhi_y = np.minimum(fhi_y + ey / 2, np.maximum(by0, 0))
|
||||
flo_y = np.minimum(flo_y, np.maximum(by0, 0) + hmax)
|
||||
fhi_y = np.maximum(fhi_y, by1 + 1 - hmax)
|
||||
return flo_x, fhi_x, flo_y, fhi_y
|
||||
|
||||
|
||||
def _solve_zoomed_at(pre, ar, img_w, img_h, p):
|
||||
"""Inner LP for one aspect candidate. Width is ar*h throughout; rent
|
||||
is charged on h beyond the tightest aspect-respecting padded need, so
|
||||
width the ratio forces is never billed."""
|
||||
m, loop = pre["m"], pre["loop"]
|
||||
nt = m if loop else m - 1
|
||||
bx0, bx1, by0, by1 = pre["bx0"], pre["bx1"], pre["by0"], pre["by1"]
|
||||
padx, pady = pre["padx"], pre["pady"]
|
||||
extw = bx1 - bx0 + 1
|
||||
exth = by1 - by0 + 1
|
||||
soft = 1.0 / max(p.get("pad_deficit_tol", 16), 1e-6)
|
||||
# need is aspect-aware (width the ratio forces is never billed), but
|
||||
# the excess is billed at its (1+ar) box magnitude so wide and narrow
|
||||
# candidates pay the same for the same true slack
|
||||
rent = (1.0 + ar) * pre["rent_scale"] / max(p.get("pad_surplus_tol", 16), 1e-6)
|
||||
mc = p.get("movement_cost", 1.0)
|
||||
sc = p.get("resize_cost", 2.0) * (1.0 + ar)
|
||||
eps = p.get("center_pull", 1e-4)
|
||||
|
||||
hmax = min(img_h, img_w / ar)
|
||||
hmin = np.maximum(np.maximum(exth, extw / ar), 1.0)
|
||||
if hmin.max() > hmax + 1e-6:
|
||||
return None # this aspect cannot contain the subject in-image
|
||||
need_h = np.minimum(
|
||||
np.maximum(exth + 2 * pady, (extw + 2 * padx) / ar), hmax)
|
||||
|
||||
floors = pre["floor"] > 0
|
||||
if floors:
|
||||
flo_x, fhi_x, flo_y, fhi_y = _yielded_floors(pre, ar, img_w, img_h)
|
||||
|
||||
lp = _LP()
|
||||
ub_x = np.maximum(bx0, 0)
|
||||
ub_y = np.maximum(by0, 0)
|
||||
if floors:
|
||||
ub_x = np.minimum(ub_x, fhi_x)
|
||||
ub_y = np.minimum(ub_y, fhi_y)
|
||||
lp.var("x", m, lo=0.0, hi=ub_x)
|
||||
lp.var("y", m, lo=0.0, hi=ub_y)
|
||||
lp.var("h", m, lo=hmin, hi=hmax)
|
||||
lp.var("tx", nt, cost=mc)
|
||||
lp.var("ty", nt, cost=mc)
|
||||
lp.var("th", nt, cost=sc)
|
||||
lp.var("vx", m, cost=soft)
|
||||
lp.var("vy", m, cost=soft)
|
||||
lp.var("r", m, cost=rent)
|
||||
lp.var("ux", m, cost=eps)
|
||||
lp.var("uy", m, cost=eps)
|
||||
|
||||
_tv_rows(lp, "x", "tx", m, loop)
|
||||
_tv_rows(lp, "y", "ty", m, loop)
|
||||
_tv_rows(lp, "h", "th", m, loop)
|
||||
|
||||
rate = p.get("max_zoom_rate", 0.0)
|
||||
if rate and rate > 1.0:
|
||||
for i in range(nt):
|
||||
a, b = lp.off["h"] + i, lp.off["h"] + (i + 1) % m
|
||||
lp.row([b, a], [1, -rate], 0)
|
||||
lp.row([a, b], [1, -rate], 0)
|
||||
|
||||
for i in range(m):
|
||||
x, y, h = lp.off["x"] + i, lp.off["y"] + i, lp.off["h"] + i
|
||||
vx, vy = lp.off["vx"] + i, lp.off["vy"] + i
|
||||
r = lp.off["r"] + i
|
||||
ux, uy = lp.off["ux"] + i, lp.off["uy"] + i
|
||||
lp.row([x, h], [-1, -ar], -(bx1[i] + 1)) # containment right
|
||||
lp.row([x, h], [1, ar], img_w) # image right
|
||||
lp.row([y, h], [-1, -1], -(by1[i] + 1))
|
||||
lp.row([y, h], [1, 1], img_h)
|
||||
lp.row([x, vx], [1, -1], pre["lx"][i])
|
||||
lp.row([x, h, vx], [-1, -ar, -1], -pre["rx"][i])
|
||||
lp.row([y, vy], [1, -1], pre["ly"][i])
|
||||
lp.row([y, h, vy], [-1, -1, -1], -pre["ry"][i])
|
||||
lp.row([h, r], [1, -1], need_h[i])
|
||||
if floors:
|
||||
lp.row([x, h], [-1, -ar], -flo_x[i])
|
||||
lp.row([y, h], [-1, -1], -flo_y[i])
|
||||
cx = (bx0[i] + bx1[i] + 1) / 2
|
||||
cy = (by0[i] + by1[i] + 1) / 2
|
||||
lp.row([x, h, ux], [1, 0.5 * ar, -1], cx)
|
||||
lp.row([x, h, ux], [-1, -0.5 * ar, -1], -cx)
|
||||
lp.row([y, h, uy], [1, 0.5, -1], cy)
|
||||
lp.row([y, h, uy], [-1, -0.5, -1], -cy)
|
||||
|
||||
res = lp.solve()
|
||||
o = lp.off
|
||||
# the objective is the sweep score: floors carry the padding tier as
|
||||
# hard rows, so candidates compare on how cheaply they serve the
|
||||
# cell's own priced trade above them
|
||||
viol = res.x[o["vx"]:o["vx"] + m].sum() + res.x[o["vy"]:o["vy"] + m].sum()
|
||||
h_t = res.x[o["h"]:o["h"] + m]
|
||||
# dead width -- width this ratio forces beyond the frame's padded
|
||||
# horizontal need -- is billed in the score only, at the same rent as
|
||||
# height slack. Inside the LP that width stays free (billing it there
|
||||
# collapses manual wide aspects), but across candidates it
|
||||
# must cost, or the sweep drifts to the widest feasible shape whenever
|
||||
# the vertical need pins the height. Capped at need_h: excess above
|
||||
# need_h already carries its width via the (1+ar) rent.
|
||||
w_need = np.minimum(extw + 2 * padx, img_w)
|
||||
dead_w = np.maximum(ar * np.minimum(h_t, need_h) - w_need, 0)
|
||||
dead = ((pre["rent_scale"] * dead_w).sum()
|
||||
/ max(p.get("pad_surplus_tol", 16), 1e-6))
|
||||
return {
|
||||
"ar": ar, "score": res.fun + dead, "viol": viol,
|
||||
"x": res.x[o["x"]:o["x"] + m], "y": res.x[o["y"]:o["y"] + m],
|
||||
"h": h_t,
|
||||
}
|
||||
|
||||
|
||||
def _aspect_candidates(pre, img_w, img_h, n_known):
|
||||
"""Hull of the per-frame shape quantiles and the padded union aspect:
|
||||
the winner lives between what the subject looks like and where it
|
||||
goes."""
|
||||
bx0, bx1, by0, by1 = pre["bx0"], pre["bx1"], pre["by0"], pre["by1"]
|
||||
padx, pady = pre["padx"], pre["pady"]
|
||||
frame_ar = (bx1 - bx0 + 1) / (by1 - by0 + 1)
|
||||
qs = np.quantile(frame_ar, [0.05, 0.25, 0.5, 0.75, 0.95])
|
||||
ux0 = (bx0 - padx).min()
|
||||
ux1 = (bx1 + 1 + padx).max()
|
||||
uy0 = (by0 - pady).min()
|
||||
uy1 = (by1 + 1 + pady).max()
|
||||
union_ar = max(ux1 - ux0, 1.0) / max(uy1 - uy0, 1.0)
|
||||
lo = min(qs.min(), union_ar)
|
||||
hi = max(qs.max(), union_ar)
|
||||
grid = np.geomspace(lo, hi, 7) if hi > lo * 1.02 else np.array([lo])
|
||||
cands = np.concatenate([grid, qs, [union_ar]])
|
||||
# feasibility band: the box must contain every frame inside the image
|
||||
extw = bx1 - bx0 + 1
|
||||
exth = by1 - by0 + 1
|
||||
band_lo = extw.max() / img_h
|
||||
band_hi = img_w / exth.max()
|
||||
cands = np.clip(cands, band_lo, band_hi)
|
||||
cands = np.unique(np.round(cands, 3))
|
||||
keep = [cands[0]]
|
||||
for a in cands[1:]:
|
||||
if a > keep[-1] * 1.02:
|
||||
keep.append(a)
|
||||
return keep
|
||||
|
||||
|
||||
def _quantize_steps(v, step, upper):
|
||||
base = v.min()
|
||||
k = np.ceil(np.log(v / base) / np.log(step) - 1e-9)
|
||||
return np.minimum(base * step ** k, upper)
|
||||
|
||||
|
||||
def plan(binary, sel, p, divisible_by=16):
|
||||
"""binary: (n, H, W) bool mask stack. Returns (boxes, info)."""
|
||||
n, img_h, img_w = binary.shape
|
||||
if not binary.any():
|
||||
raise ValueError("all masks are empty, nothing to plan")
|
||||
scale = p["crop_scale"]
|
||||
|
||||
if sel == "combined":
|
||||
ys, xs = np.nonzero(binary.any(axis=0))
|
||||
w0 = (xs.max() - xs.min() + 1) * scale
|
||||
h0 = (ys.max() - ys.min() + 1) * scale
|
||||
ar = p.get("aspect_ratio", 0.0)
|
||||
if ar > 0:
|
||||
if w0 < h0 * ar:
|
||||
w0 = h0 * ar
|
||||
else:
|
||||
h0 = w0 / ar
|
||||
w = min(math.ceil(w0 / divisible_by) * divisible_by,
|
||||
(img_w // divisible_by) * divisible_by or img_w)
|
||||
h = min(math.ceil(h0 / divisible_by) * divisible_by,
|
||||
(img_h // divisible_by) * divisible_by or img_h)
|
||||
x = min(max(0, round((xs.min() + xs.max() + 1) / 2 - w / 2)), img_w - w)
|
||||
y = min(max(0, round((ys.min() + ys.max() + 1) / 2 - h / 2)), img_h - h)
|
||||
box = {"x": int(x), "y": int(y), "width": int(w), "height": int(h)}
|
||||
return [box] * n, {"aspect": w / h}
|
||||
|
||||
pre = _prep(binary, img_w, img_h, p)
|
||||
m, s0 = pre["m"], pre["s0"]
|
||||
bx0, bx1, by0, by1 = pre["bx0"], pre["bx1"], pre["by0"], pre["by1"]
|
||||
|
||||
if sel == "tracked":
|
||||
Wf, Hf, xt, yt = _solve_tracked(pre, img_w, img_h, p)
|
||||
w = min(math.ceil(Wf / divisible_by) * divisible_by,
|
||||
(img_w // divisible_by) * divisible_by or img_w)
|
||||
h = min(math.ceil(Hf / divisible_by) * divisible_by,
|
||||
(img_h // divisible_by) * divisible_by or img_h)
|
||||
lo_x = np.maximum(bx1 + 1 - w, 0)
|
||||
hi_x = np.minimum(np.maximum(bx0, 0), img_w - w)
|
||||
lo_y = np.maximum(by1 + 1 - h, 0)
|
||||
hi_y = np.minimum(np.maximum(by0, 0), img_h - h)
|
||||
if pre["floor"] > 0:
|
||||
lo_x = np.maximum(lo_x, np.minimum(pre["flo_x"] - w, hi_x))
|
||||
hi_x = np.minimum(hi_x, np.maximum(pre["fhi_x"], lo_x))
|
||||
lo_y = np.maximum(lo_y, np.minimum(pre["flo_y"] - h, hi_y))
|
||||
hi_y = np.minimum(hi_y, np.maximum(pre["fhi_y"], lo_y))
|
||||
xi = _round_corridor(xt, lo_x, hi_x)[s0:s0 + n]
|
||||
yi = _round_corridor(yt, lo_y, hi_y)[s0:s0 + n]
|
||||
boxes = [{"x": int(xi[i]), "y": int(yi[i]),
|
||||
"width": int(w), "height": int(h)} for i in range(n)]
|
||||
return boxes, {"aspect": w / h, "W": Wf, "H": Hf}
|
||||
|
||||
# zoomed
|
||||
ar_manual = p.get("aspect_ratio", 0.0)
|
||||
if ar_manual > 0:
|
||||
extw = bx1 - bx0 + 1
|
||||
exth = by1 - by0 + 1
|
||||
ar = min(max(ar_manual, extw.max() / img_h), img_w / exth.max())
|
||||
best = _solve_zoomed_at(pre, ar, img_w, img_h, p)
|
||||
if best is None:
|
||||
raise RuntimeError("requested aspect_ratio cannot contain the subject")
|
||||
tried = [best]
|
||||
else:
|
||||
tried = []
|
||||
for ar in _aspect_candidates(pre, img_w, img_h, pre["known"].sum()):
|
||||
sol = _solve_zoomed_at(pre, ar, img_w, img_h, p)
|
||||
if sol is not None:
|
||||
tried.append(sol)
|
||||
if not tried:
|
||||
raise RuntimeError("no feasible aspect candidate")
|
||||
best = min(tried, key=lambda s: s["score"])
|
||||
for ar in (best["ar"] * 0.93, best["ar"] * 1.075):
|
||||
sol = _solve_zoomed_at(pre, ar, img_w, img_h, p)
|
||||
if sol is not None and sol["score"] < best["score"]:
|
||||
best = sol
|
||||
|
||||
ar = best["ar"]
|
||||
h = best["h"]
|
||||
if p.get("zoom_step", 1.0) > 1.0:
|
||||
h = _quantize_steps(h, p["zoom_step"], min(img_h, img_w / ar))
|
||||
w = np.minimum(h * ar, img_w)
|
||||
bw = np.minimum(np.ceil(w - 1e-6), img_w).astype(int)
|
||||
bh = np.minimum(np.ceil(h - 1e-6), img_h).astype(int)
|
||||
lo_x = np.maximum(bx1 + 1 - bw, 0)
|
||||
hi_x = np.minimum(np.maximum(bx0, 0), img_w - bw)
|
||||
lo_y = np.maximum(by1 + 1 - bh, 0)
|
||||
hi_y = np.minimum(np.maximum(by0, 0), img_h - bh)
|
||||
if pre["floor"] > 0:
|
||||
flo_x, fhi_x, flo_y, fhi_y = _yielded_floors(pre, ar, img_w, img_h)
|
||||
lo_x = np.maximum(lo_x, np.minimum(flo_x - bw, hi_x))
|
||||
hi_x = np.minimum(hi_x, np.maximum(fhi_x, lo_x))
|
||||
lo_y = np.maximum(lo_y, np.minimum(flo_y - bh, hi_y))
|
||||
hi_y = np.minimum(hi_y, np.maximum(fhi_y, lo_y))
|
||||
xi = _round_corridor(best["x"], lo_x, hi_x)[s0:s0 + n]
|
||||
yi = _round_corridor(best["y"], lo_y, hi_y)[s0:s0 + n]
|
||||
bw, bh = bw[s0:s0 + n], bh[s0:s0 + n]
|
||||
boxes = [{"x": int(min(max(xi[i], 0), img_w - bw[i])),
|
||||
"y": int(min(max(yi[i], 0), img_h - bh[i])),
|
||||
"width": int(bw[i]), "height": int(bh[i])} for i in range(n)]
|
||||
return boxes, {"aspect": ar, "swept": ar_manual <= 0,
|
||||
"candidates": len(tried)}
|
||||
@@ -0,0 +1,107 @@
|
||||
"""Sliding history for MiniMax H3: the end of the previous shot's result as clean frames placed just BEFORE the video
|
||||
on H3's timeline, so a long continuous take is generated window by window with motion (and look) carried across.
|
||||
|
||||
Idea from Akatz's H3 Relay windowed edit and Ethanfel's ComfyUI-MiniMaxH3-Context-Loop (both GPL-3.0); this is an
|
||||
independent implementation. A guide keyframe marked ``anchor = "history"`` is packed like any H3 guide, then its rows
|
||||
are kept at the target's origin and everything else on the target timeline (the video, its audio, the other guides)
|
||||
moves forward by the history's span. Text and reference rows are untouched.
|
||||
|
||||
The layout is patched in memory only (ComfyUI files are never edited), once, and only changes layouts that carry a
|
||||
history keyframe; when ComfyUI's own H3 layout already understands history anchors the patch stands down.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
import torch
|
||||
|
||||
LOG = logging.getLogger("bfs.h3_history")
|
||||
HISTORY_FRAMES = (17, 34, 51) # whole H3 VAE groups: 1+4+4+4+4 frames per 5 latent steps
|
||||
_PATCH = "_bfs_history_patch"
|
||||
|
||||
|
||||
def history_span(latent_t: int) -> float:
|
||||
"""Timeline span of `latent_t` video latent steps (FRAME_RESCALE per pixel frame)."""
|
||||
import comfy.ldm.minimax.model as mm
|
||||
return float(sum(mm._video_t_spans(int(latent_t))))
|
||||
|
||||
|
||||
def relocate(layout, keyframes) -> bool:
|
||||
"""Moves the target timeline past the history keyframes. Returns whether anything changed."""
|
||||
hist = [i for i, kf in enumerate(keyframes or ()) if kf.get("anchor") == "history"]
|
||||
if not hist:
|
||||
return False
|
||||
# keyframe -> its packed segments: every guide with a video latent owns the next "cond" segment, every guide
|
||||
# with audio the next "cond_audio" one, in keyframe order (the order PackedLayout packs them)
|
||||
cond = [s for s in layout.segments if s[2] == "cond"]
|
||||
cond_a = [s for s in layout.segments if s[2] == "cond_audio"]
|
||||
own, ci, ai = {}, 0, 0
|
||||
for i, kf in enumerate(keyframes):
|
||||
segs = []
|
||||
if kf.get("latent") is not None and ci < len(cond):
|
||||
segs.append(cond[ci]); ci += 1
|
||||
if kf.get("audio_latent") is not None and ai < len(cond_a):
|
||||
segs.append(cond_a[ai]); ai += 1
|
||||
own[i] = segs
|
||||
span = sum(history_span(keyframes[i]["latent"].shape[2]) for i in hist if keyframes[i].get("latent") is not None)
|
||||
if span <= 0:
|
||||
return False
|
||||
keep = torch.zeros(layout.position_ids.shape[0], dtype=torch.bool)
|
||||
for i in hist:
|
||||
for a, b, _ in own[i]:
|
||||
keep[a:b] = True
|
||||
vid = next(((a, b) for a, b, k in layout.segments if k == "video"), None)
|
||||
if vid is not None and keep.any():
|
||||
# another history implementation (e.g. H3 Relay's, or a future ComfyUI) already moved the target: skip
|
||||
if float(layout.position_ids[vid[0]:vid[1], 0].min()) - float(layout.position_ids[keep, 0].min()) >= span - 1e-6:
|
||||
return False
|
||||
move = torch.zeros_like(keep)
|
||||
for a, b, kind in layout.segments:
|
||||
if kind in ("video", "audio", "cond", "cond_audio"):
|
||||
move[a:b] = True
|
||||
move &= ~keep
|
||||
pos = layout.position_ids.clone()
|
||||
pos[move, 0] += span
|
||||
layout.position_ids = pos
|
||||
LOG.info("H3 history: %d rows before the video, target timeline moved by %.2f", int(keep.sum()), span)
|
||||
return True
|
||||
|
||||
|
||||
def install() -> bool:
|
||||
"""Wraps PackedLayout.__init__ once (process-local). False when ComfyUI is missing or already supports it."""
|
||||
try:
|
||||
import comfy.ldm.minimax.model as mm
|
||||
except ImportError:
|
||||
return False
|
||||
cls = mm.PackedLayout
|
||||
if getattr(cls.__init__, _PATCH, False):
|
||||
return True
|
||||
import inspect
|
||||
if "history" in inspect.getsource(cls): # native history anchors: nothing to do
|
||||
return False
|
||||
orig = cls.__init__
|
||||
|
||||
def __init__(self, text_len, latent_t, latent_h, latent_w, audio_t, keyframes=None, refs=None):
|
||||
orig(self, text_len, latent_t, latent_h, latent_w, audio_t, keyframes=keyframes, refs=refs)
|
||||
relocate(self, keyframes)
|
||||
|
||||
setattr(__init__, _PATCH, True)
|
||||
cls.__init__ = __init__
|
||||
return True
|
||||
|
||||
|
||||
def add_history(positive, vae, frames: torch.Tensor):
|
||||
"""Appends `frames` ([N,H,W,3] at the video's size, N = 17k: the end of the previous result) as a history keyframe.
|
||||
Returns (positive, frames used)."""
|
||||
import node_helpers
|
||||
n = (int(frames.shape[0]) // 17) * 17
|
||||
if n < 17:
|
||||
return positive, 0
|
||||
install()
|
||||
# encoded from CPU frames; the latent comes back on ComfyUI's intermediate device like any H3 guide, and the VAE
|
||||
# is loaded / freed by ComfyUI's model management (nothing here keeps GPU memory)
|
||||
with torch.no_grad():
|
||||
lat = vae.encode(frames[-n:, ..., :3].float().cpu())
|
||||
kfs = list(positive[0][1].get("minimax_keyframes", []))
|
||||
kfs.append({"resolved_frame_index": 0, "latent": lat, "anchor": "history"})
|
||||
return node_helpers.conditioning_set_values(positive, {"minimax_keyframes": kfs}), n
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,691 @@
|
||||
"""BFS LoRA Surgery: read any LoRA's layers and blocks, then scale or drop them.
|
||||
|
||||
Model-agnostic: the structure is discovered from the key names, not hardcoded. Works with
|
||||
the diffusers style (``lora_A``/``lora_B``), the kohya style (``lora_down``/``lora_up`` plus
|
||||
``alpha``) and anything that follows either convention.
|
||||
|
||||
Why this exists: a defect a LoRA learned (smoothed skin, identity drifting at distance, a
|
||||
pose it refuses to copy) usually lives in a specific module family and a specific range of
|
||||
blocks. Scaling or dropping that group and regenerating tells you where it lives, in minutes,
|
||||
without retraining. See the node's tooltip for the workflow.
|
||||
|
||||
Scaling a module by ``s`` multiplies only the up/B factor, since ``(sB)A = s(BA)``. That keeps
|
||||
negative values usable and is exact regardless of the alpha convention.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from datetime import datetime, timezone
|
||||
import re
|
||||
import struct
|
||||
from typing import Any
|
||||
|
||||
import torch
|
||||
|
||||
import comfy.sd
|
||||
import comfy.utils
|
||||
import folder_paths
|
||||
|
||||
# ---------------------------------------------------------------------------- parsing
|
||||
|
||||
_SUFFIXES = (
|
||||
".lora_down.weight", ".lora_up.weight",
|
||||
".lora_A.weight", ".lora_B.weight",
|
||||
".lora_A.default.weight", ".lora_B.default.weight",
|
||||
".hada_w1_a", ".hada_w1_b", ".hada_w2_a", ".hada_w2_b",
|
||||
".lokr_w1", ".lokr_w2", ".lokr_w1_a", ".lokr_w1_b", ".lokr_w2_a", ".lokr_w2_b",
|
||||
".diff", ".diff_b", ".alpha", ".dora_scale",
|
||||
)
|
||||
_UP_SUFFIXES = (".lora_up.weight", ".lora_B.weight", ".lora_B.default.weight")
|
||||
_BLOCK_RE = re.compile(r"(?<=[._])(\d+)(?=[._])")
|
||||
|
||||
|
||||
def _strip_suffix(key: str) -> str | None:
|
||||
for s in _SUFFIXES:
|
||||
if key.endswith(s):
|
||||
return key[: -len(s)]
|
||||
return None
|
||||
|
||||
|
||||
_NUM_RE = re.compile(r"\d+")
|
||||
|
||||
|
||||
def _template(path: str) -> tuple[str, list[int], list[tuple[int, int]]]:
|
||||
"""``a.12.attn1.q`` -> ("a.{}.attn{}.q", [12, 1], spans)."""
|
||||
nums, spans, out, last = [], [], [], 0
|
||||
for m in _NUM_RE.finditer(path):
|
||||
out.append(path[last:m.start()])
|
||||
out.append("{}")
|
||||
nums.append(int(m.group()))
|
||||
spans.append((m.start(), m.end()))
|
||||
last = m.end()
|
||||
out.append(path[last:])
|
||||
return "".join(out), nums, spans
|
||||
|
||||
|
||||
def choose_block_axis(paths: list[str]) -> dict[str, int]:
|
||||
"""For each path template, decide which numeric slot is the block index.
|
||||
|
||||
Picking the last number breaks on names like ``transformer_blocks.12.attn1.to_q``, where
|
||||
the last one is the ``1`` of ``attn1``, and on nested layouts such as
|
||||
``down_blocks.0.attentions.1.transformer_blocks.0.ff.net.0.proj``. The block axis is the
|
||||
slot that varies most across the whole LoRA, so decide it from the file, not from one key.
|
||||
"""
|
||||
seen: dict[str, list[set]] = {}
|
||||
for path in paths:
|
||||
tpl, nums, _ = _template(path)
|
||||
slots = seen.setdefault(tpl, [set() for _ in nums])
|
||||
for i, v in enumerate(nums):
|
||||
if i < len(slots):
|
||||
slots[i].add(v)
|
||||
axis: dict[str, int] = {}
|
||||
for tpl, slots in seen.items():
|
||||
if not slots:
|
||||
continue
|
||||
counts = [len(x) for x in slots]
|
||||
best = max(range(len(counts)), key=lambda i: (counts[i], -i))
|
||||
axis[tpl] = best if counts[best] > 1 else -1
|
||||
return axis
|
||||
|
||||
|
||||
def _split_block(path: str, axis: dict[str, int] | None = None) -> tuple[str, int | None, str]:
|
||||
"""Return (family, block_index, module_type).
|
||||
|
||||
``transformer_blocks.12.attn.to_q`` -> ("transformer_blocks", 12, "attn.to_q").
|
||||
Numbers that are not the block axis stay in the name, so ``attn1`` and ``attn2`` remain
|
||||
distinct module types instead of being folded together.
|
||||
"""
|
||||
tpl, nums, spans = _template(path)
|
||||
if not nums:
|
||||
return path, None, ""
|
||||
idx = axis.get(tpl, -1) if axis is not None else (len(nums) - 1)
|
||||
if idx < 0 or idx >= len(nums):
|
||||
return path.rstrip("._"), None, ""
|
||||
start, end = spans[idx]
|
||||
family = path[:start].rstrip("._")
|
||||
module = path[end:].lstrip("._")
|
||||
return family, nums[idx], module
|
||||
|
||||
|
||||
def read_structure(lora_path: str, with_norms: bool = True) -> dict[str, Any]:
|
||||
"""Describe a LoRA: families, block indices, module types, ranks and ΔW norms.
|
||||
|
||||
Reads the safetensors header first (instant) and only loads tensors when norms are asked
|
||||
for. Norms are what tell you where training actually invested.
|
||||
"""
|
||||
with open(lora_path, "rb") as fh:
|
||||
n = struct.unpack("<Q", fh.read(8))[0]
|
||||
header = json.loads(fh.read(n))
|
||||
header.pop("__metadata__", None)
|
||||
|
||||
modules: dict[str, dict[str, Any]] = {}
|
||||
for key, info in header.items():
|
||||
base = _strip_suffix(key)
|
||||
if base is None:
|
||||
continue
|
||||
entry = modules.setdefault(base, {"rank": None, "keys": []})
|
||||
entry["keys"].append(key)
|
||||
shape = info.get("shape") or []
|
||||
if len(shape) == 2 and entry["rank"] is None:
|
||||
entry["rank"] = int(min(shape))
|
||||
|
||||
norms: dict[str, float] = {}
|
||||
if with_norms and modules:
|
||||
norms = _module_norms(lora_path, modules)
|
||||
axis = choose_block_axis(list(modules))
|
||||
|
||||
families: dict[str, dict[str, Any]] = {}
|
||||
loose: list[dict[str, Any]] = []
|
||||
all_modules: list[dict[str, Any]] = []
|
||||
for base, entry in modules.items():
|
||||
family, block, mtype = _split_block(base, axis)
|
||||
all_modules.append({"path": base, "family": family, "block": block, "type": mtype,
|
||||
"rank": entry["rank"], "norm": norms.get(base)})
|
||||
if block is None:
|
||||
loose.append({"path": base, "rank": entry["rank"], "norm": norms.get(base)})
|
||||
continue
|
||||
fam = families.setdefault(family, {"name": family, "blocks": set(), "types": {}})
|
||||
fam["blocks"].add(block)
|
||||
t = fam["types"].setdefault(mtype, {"name": mtype, "count": 0, "rank": entry["rank"],
|
||||
"norm_sum": 0.0, "norm_n": 0, "blocks": []})
|
||||
t["count"] += 1
|
||||
t["blocks"].append(block)
|
||||
if base in norms:
|
||||
t["norm_sum"] += norms[base]
|
||||
t["norm_n"] += 1
|
||||
|
||||
out_families = []
|
||||
for fam in families.values():
|
||||
types = []
|
||||
for t in fam["types"].values():
|
||||
types.append({
|
||||
"name": t["name"], "count": t["count"], "rank": t["rank"],
|
||||
"norm": (t["norm_sum"] / t["norm_n"]) if t["norm_n"] else None,
|
||||
"blocks": sorted(t["blocks"]),
|
||||
})
|
||||
types.sort(key=lambda x: -(x["norm"] or 0))
|
||||
out_families.append({
|
||||
"name": fam["name"],
|
||||
"blocks": sorted(fam["blocks"]),
|
||||
"types": types,
|
||||
})
|
||||
out_families.sort(key=lambda f: -sum(t["count"] for t in f["types"]))
|
||||
|
||||
all_modules.sort(key=lambda m: (m["family"], m["block"] if m["block"] is not None else -1, m["type"]))
|
||||
return {
|
||||
"file": os.path.basename(lora_path),
|
||||
"total_modules": len(modules),
|
||||
"families": out_families,
|
||||
"loose": sorted(loose, key=lambda x: x["path"])[:64],
|
||||
"modules": all_modules,
|
||||
"per_module_norms": norms,
|
||||
}
|
||||
|
||||
|
||||
def _module_norms(lora_path: str, modules: dict[str, dict[str, Any]]) -> dict[str, float]:
|
||||
"""Exact ||ΔW||_F per module.
|
||||
|
||||
ΔW = B·A has rank <= r, so QR both factors and take the singular values of the small
|
||||
r x r product. No need to materialize the full d x d matrix.
|
||||
"""
|
||||
try:
|
||||
sd = comfy.utils.load_torch_file(lora_path, safe_load=True)
|
||||
except Exception:
|
||||
return {}
|
||||
out: dict[str, float] = {}
|
||||
for base in modules:
|
||||
down = sd.get(base + ".lora_down.weight", sd.get(base + ".lora_A.weight"))
|
||||
up = sd.get(base + ".lora_up.weight", sd.get(base + ".lora_B.weight"))
|
||||
if down is None or up is None or down.ndim != 2 or up.ndim != 2:
|
||||
continue
|
||||
try:
|
||||
a = down.float()
|
||||
b = up.float()
|
||||
scale = 1.0
|
||||
alpha = sd.get(base + ".alpha")
|
||||
if alpha is not None:
|
||||
scale = float(alpha) / a.shape[0]
|
||||
qb, rb = torch.linalg.qr(b)
|
||||
qa, ra = torch.linalg.qr(a.T)
|
||||
s = torch.linalg.svdvals(rb @ ra.T)
|
||||
out[base] = float(s.pow(2).sum().sqrt()) * scale
|
||||
except Exception:
|
||||
continue
|
||||
del sd
|
||||
return out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------- rules
|
||||
|
||||
def _match_blocks(spec: str | None, block: int) -> bool:
|
||||
"""``spec`` is a block selector: "" / "all" / "8-15" / "0,4,7" / "8-15,24-31"."""
|
||||
if not spec or spec.strip().lower() in ("all", "*"):
|
||||
return True
|
||||
for part in spec.split(","):
|
||||
part = part.strip()
|
||||
if not part:
|
||||
continue
|
||||
if "-" in part:
|
||||
lo, _, hi = part.partition("-")
|
||||
try:
|
||||
if int(lo) <= block <= int(hi):
|
||||
return True
|
||||
except ValueError:
|
||||
continue
|
||||
else:
|
||||
try:
|
||||
if int(part) == block:
|
||||
return True
|
||||
except ValueError:
|
||||
continue
|
||||
return False
|
||||
|
||||
|
||||
def build_regex(family: str | None, mtype: str | None, blocks: str | None) -> str:
|
||||
r"""Build the regex a selection stands for. This is what the UI's "to regex" button calls.
|
||||
|
||||
Selecting family ``transformer_blocks``, type ``img_mlp.gate_up`` and blocks ``8-15``
|
||||
gives ``^.*transformer_blocks\.(8|9|10|11|12|13|14|15)\.img_mlp\.gate_up$``.
|
||||
Leaving blocks empty matches every block.
|
||||
"""
|
||||
fam = re.escape(family) if family else r"[\\w.]+"
|
||||
if blocks and blocks.strip().lower() not in ("all", "*", ""):
|
||||
idx: list[int] = []
|
||||
for part in blocks.split(","):
|
||||
part = part.strip()
|
||||
if not part:
|
||||
continue
|
||||
if "-" in part:
|
||||
lo, _, hi = part.partition("-")
|
||||
try:
|
||||
idx.extend(range(int(lo), int(hi) + 1))
|
||||
except ValueError:
|
||||
continue
|
||||
else:
|
||||
try:
|
||||
idx.append(int(part))
|
||||
except ValueError:
|
||||
continue
|
||||
blk = "(" + "|".join(str(i) for i in sorted(set(idx))) + ")" if idx else r"\\d+"
|
||||
else:
|
||||
blk = r"\\d+"
|
||||
if not mtype or mtype.strip() in ("*", ""):
|
||||
mod = r".*"
|
||||
elif mtype.endswith("*"):
|
||||
mod = re.escape(mtype[:-1]) + r".*"
|
||||
else:
|
||||
mod = re.escape(mtype)
|
||||
return rf"^.*{fam}\.{blk}\.{mod}$"
|
||||
|
||||
|
||||
def _match_type(spec: str | None, mtype: str) -> bool:
|
||||
if not spec or spec.strip() in ("*", ""):
|
||||
return True
|
||||
spec = spec.strip()
|
||||
if spec.endswith("*"):
|
||||
return mtype.startswith(spec[:-1])
|
||||
return mtype == spec
|
||||
|
||||
|
||||
_REGEX_CACHE: dict[str, Any] = {}
|
||||
|
||||
|
||||
def _compiled(pattern: str):
|
||||
rx = _REGEX_CACHE.get(pattern)
|
||||
if rx is None:
|
||||
try:
|
||||
rx = re.compile(pattern)
|
||||
except re.error:
|
||||
rx = False # invalid pattern matches nothing, and never raises mid-run
|
||||
_REGEX_CACHE[pattern] = rx
|
||||
return rx
|
||||
|
||||
|
||||
def resolve_scale(rules: list[dict[str, Any]], family: str, block: int | None, mtype: str,
|
||||
full_path: str | None = None) -> float:
|
||||
"""Last matching rule wins, which is what makes the UI predictable.
|
||||
|
||||
A rule matches either structurally (family / type / blocks) or by ``regex`` against the
|
||||
module's full path. A rule carrying a regex ignores the structural fields.
|
||||
"""
|
||||
scale = 1.0
|
||||
for rule in rules:
|
||||
if not rule.get("enabled", True):
|
||||
continue
|
||||
m = rule.get("match", {})
|
||||
pattern = m.get("regex")
|
||||
if pattern:
|
||||
rx = _compiled(pattern)
|
||||
if not rx or full_path is None or not rx.search(full_path):
|
||||
continue
|
||||
try:
|
||||
scale = float(rule.get("scale", 1.0))
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
continue
|
||||
if m.get("family") and m["family"] != family:
|
||||
continue
|
||||
if not _match_type(m.get("type"), mtype):
|
||||
continue
|
||||
if block is not None and not _match_blocks(m.get("blocks"), block):
|
||||
continue
|
||||
if block is None and m.get("blocks"):
|
||||
continue
|
||||
try:
|
||||
scale = float(rule.get("scale", 1.0))
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
return scale
|
||||
|
||||
|
||||
def apply_rules(sd: dict[str, torch.Tensor], rules: list[dict[str, Any]]) -> tuple[dict, dict]:
|
||||
"""Return (new state dict, stats). Scale 0 drops the module entirely."""
|
||||
if not rules:
|
||||
return sd, {"kept": None, "dropped": 0, "scaled": 0}
|
||||
bases = {b for b in (_strip_suffix(k) for k in sd) if b}
|
||||
axis = choose_block_axis(sorted(bases))
|
||||
out: dict[str, torch.Tensor] = {}
|
||||
dropped = scaled = kept = 0
|
||||
seen: dict[str, float] = {}
|
||||
for key, tensor in sd.items():
|
||||
base = _strip_suffix(key)
|
||||
if base is None:
|
||||
out[key] = tensor
|
||||
continue
|
||||
if base not in seen:
|
||||
family, block, mtype = _split_block(base, axis)
|
||||
seen[base] = resolve_scale(rules, family, block, mtype, full_path=base)
|
||||
s = seen[base]
|
||||
if s == 0.0:
|
||||
dropped += 1 if key.endswith(_UP_SUFFIXES) else 0
|
||||
continue
|
||||
if s != 1.0 and key.endswith(_UP_SUFFIXES):
|
||||
# (sB)A == s(BA): exact, and negative values stay usable
|
||||
out[key] = tensor.float().mul(s).to(tensor.dtype)
|
||||
scaled += 1
|
||||
else:
|
||||
out[key] = tensor
|
||||
if key.endswith(_UP_SUFFIXES):
|
||||
kept += 1
|
||||
return out, {"kept": kept, "dropped": dropped, "scaled": scaled}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------- node
|
||||
|
||||
class BFSLoraSurgery:
|
||||
"""Load a LoRA, scale or drop parts of it, and apply the result to the model.
|
||||
|
||||
Nothing is written to disk: the edited LoRA lives only in this run, so you can move a
|
||||
slider and re-queue. Use the panel to pick a module family, a block range and a scale,
|
||||
or write a regex when you want something the tree cannot express.
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"model": ("MODEL",),
|
||||
"lora_name": (folder_paths.get_filename_list("loras"),),
|
||||
"strength_model": ("FLOAT", {"default": 1.0, "min": -20.0, "max": 20.0,
|
||||
"step": 0.01, "tooltip": "Overall LoRA strength, as usual."}),
|
||||
"rules": ("STRING", {"default": "[]", "multiline": True,
|
||||
"tooltip": "JSON list of rules. The panel writes this for you."}),
|
||||
},
|
||||
"optional": {"clip": ("CLIP",)},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("MODEL", "CLIP", "STRING", "STRING")
|
||||
RETURN_NAMES = ("model", "clip", "report", "rules")
|
||||
FUNCTION = "apply"
|
||||
CATEGORY = "BFS/lora"
|
||||
DESCRIPTION = ("Scale or drop a LoRA's module groups (attention, MLP, per block range) and "
|
||||
"apply it live. Finds which layers carry a defect without retraining.")
|
||||
|
||||
def apply(self, model, lora_name, strength_model, rules, clip=None):
|
||||
lora_path = folder_paths.get_full_path("loras", lora_name)
|
||||
if lora_path is None:
|
||||
raise ValueError(f"LoRA not found: {lora_name}")
|
||||
sd = comfy.utils.load_torch_file(lora_path, safe_load=True)
|
||||
|
||||
parsed: list[dict[str, Any]] = []
|
||||
if rules and rules.strip():
|
||||
try:
|
||||
loaded = json.loads(rules)
|
||||
if isinstance(loaded, dict):
|
||||
loaded = loaded.get("rules", [])
|
||||
if isinstance(loaded, list):
|
||||
parsed = loaded
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ValueError(f"'rules' is not valid JSON: {exc}") from exc
|
||||
|
||||
edited, stats = apply_rules(sd, parsed)
|
||||
total = sum(1 for k in sd if k.endswith(_UP_SUFFIXES))
|
||||
kept = stats["kept"] if stats["kept"] is not None else total
|
||||
report = (f"{lora_name}: {kept}/{total} modules kept, {stats['dropped']} dropped, "
|
||||
f"{stats['scaled']} scaled, {len(parsed)} rule(s), strength {strength_model}")
|
||||
print(f"[BFSNodes] LoRA surgery, {report}")
|
||||
|
||||
new_model, new_clip = comfy.sd.load_lora_for_models(
|
||||
model, clip, edited, strength_model, strength_model if clip is not None else 0.0)
|
||||
# `rules` comes back out so it can feed BFS LoRA Surgery (save) without copying JSON
|
||||
return (new_model, new_clip if clip is not None else clip, report, rules)
|
||||
|
||||
|
||||
def describe_rules(rules: list[dict[str, Any]]) -> str:
|
||||
"""Short, readable slug for a rule list, used when no filename is given.
|
||||
|
||||
``gate_up-b8_15-off__all-b16_23-off__attn-x1.15``. A file named after what was done to it
|
||||
beats one named v2_final_real.
|
||||
"""
|
||||
parts: list[str] = []
|
||||
for rule in rules:
|
||||
if not rule.get("enabled", True):
|
||||
continue
|
||||
m = rule.get("match", {})
|
||||
try:
|
||||
scale = float(rule.get("scale", 1.0))
|
||||
except (TypeError, ValueError):
|
||||
scale = 1.0
|
||||
if m.get("regex"):
|
||||
what = "re" + hashlib.sha1(m["regex"].encode()).hexdigest()[:6]
|
||||
else:
|
||||
core = (m.get("type") or "").replace("*", "").strip(".")
|
||||
what = re.sub(r"[^A-Za-z0-9_-]", "", core.split(".")[-1]) or "all"
|
||||
blocks = re.sub(r"[^0-9,_-]", "", (m.get("blocks") or "")).replace(",", "_").replace("-", "_")
|
||||
tag = what if not blocks else f"{what}-b{blocks}"
|
||||
parts.append(f"{tag}-{'off' if scale == 0 else f'x{scale:g}'}")
|
||||
slug = "__".join(parts) if parts else "unchanged"
|
||||
return re.sub(r"[^A-Za-z0-9._-]", "", slug)
|
||||
|
||||
|
||||
# Linux caps a single filename at 255 bytes (NAME_MAX); Windows caps the whole path near 260
|
||||
# unless long paths are enabled. Stay well under both, since the folder above us is not ours.
|
||||
MAX_FILENAME = 160
|
||||
|
||||
|
||||
def fit_filename(stem: str, slug: str, ext: str = ".safetensors", limit: int = MAX_FILENAME) -> str:
|
||||
"""Build ``stem__slug.ext`` and shrink it to ``limit`` bytes without losing uniqueness.
|
||||
|
||||
When it does not fit, both halves are trimmed and a short hash of the full slug is
|
||||
appended, so two different recipes never collapse onto the same name.
|
||||
"""
|
||||
name = f"{stem}__{slug}{ext}"
|
||||
if len(name.encode("utf-8")) <= limit:
|
||||
return name
|
||||
digest = hashlib.sha1(f"{stem}__{slug}".encode()).hexdigest()[:8]
|
||||
room = limit - len(ext) - len(digest) - 3 # "__" between halves, "-" before hash
|
||||
keep_stem = min(len(stem), max(16, room // 2))
|
||||
keep_slug = max(8, room - keep_stem)
|
||||
return f"{stem[:keep_stem]}__{slug[:keep_slug]}-{digest}{ext}"
|
||||
|
||||
|
||||
def summarize_rules(rules: list[dict[str, Any]]) -> str:
|
||||
"""One sentence a human can read months later, without parsing JSON."""
|
||||
drops, scales = [], []
|
||||
for rule in rules:
|
||||
if not rule.get("enabled", True):
|
||||
continue
|
||||
m = rule.get("match", {})
|
||||
try:
|
||||
scale = float(rule.get("scale", 1.0))
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
if m.get("regex"):
|
||||
what = f"modules matching /{m['regex']}/"
|
||||
else:
|
||||
t = m.get("type") or "*"
|
||||
what = "all modules" if t in ("*", "") else t
|
||||
blocks = m.get("blocks")
|
||||
if blocks and blocks.strip().lower() not in ("all", "*"):
|
||||
what += f" in blocks {blocks}"
|
||||
if scale == 0.0:
|
||||
drops.append(what)
|
||||
elif scale != 1.0:
|
||||
scales.append(f"{what} x{scale:g}")
|
||||
bits = []
|
||||
if drops:
|
||||
bits.append("dropped " + "; ".join(drops))
|
||||
if scales:
|
||||
bits.append("scaled " + "; ".join(scales))
|
||||
return ", ".join(bits) if bits else "no changes"
|
||||
|
||||
|
||||
def _file_digest(path: str) -> str:
|
||||
"""sha256 of the source, so provenance survives a rename."""
|
||||
h = hashlib.sha256()
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
for chunk in iter(lambda: fh.read(4 << 20), b""):
|
||||
h.update(chunk)
|
||||
return h.hexdigest()
|
||||
except Exception: # noqa: BLE001
|
||||
return ""
|
||||
|
||||
|
||||
class BFSLoraSurgerySave:
|
||||
"""Write the edited LoRA to models/loras as a real file.
|
||||
|
||||
Same rules as BFS LoRA Surgery. Leave ``filename`` empty and the name is built from the
|
||||
rules themselves, so the file says what was done to it. The recipe is also stored in the
|
||||
safetensors metadata under ``bfs_surgery``, which survives being shared.
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"lora_name": (folder_paths.get_filename_list("loras"),),
|
||||
"rules": ("STRING", {"default": "[]", "multiline": True,
|
||||
"tooltip": "Same JSON the surgery panel writes."}),
|
||||
"filename": ("STRING", {"default": "", "multiline": False,
|
||||
"tooltip": "Leave empty to name it after the rules."}),
|
||||
"subfolder": ("STRING", {"default": "surgery", "multiline": False}),
|
||||
"save_dtype": (["keep", "float16", "bfloat16", "float32"],),
|
||||
"overwrite": ("BOOLEAN", {"default": False}),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("STRING",)
|
||||
RETURN_NAMES = ("path",)
|
||||
FUNCTION = "save"
|
||||
OUTPUT_NODE = True
|
||||
CATEGORY = "BFS/lora"
|
||||
DESCRIPTION = "Apply the same surgery rules and write the result to models/loras."
|
||||
|
||||
def save(self, lora_name, rules, filename, subfolder, save_dtype, overwrite):
|
||||
dest, msg = write_surgery(lora_name, rules, filename, subfolder, save_dtype, overwrite)
|
||||
print(f"[BFSNodes] LoRA surgery save: {msg}")
|
||||
return {"ui": {"text": [msg]}, "result": (dest,)}
|
||||
|
||||
|
||||
def write_surgery(lora_name, rules, filename="", subfolder="surgery", save_dtype="keep",
|
||||
overwrite=False) -> tuple[str, str]:
|
||||
"""Apply the rules and write the file. Shared by the save node and the panel's button."""
|
||||
src = folder_paths.get_full_path("loras", lora_name)
|
||||
if src is None:
|
||||
raise ValueError(f"LoRA not found: {lora_name}")
|
||||
parsed: list[dict[str, Any]] = []
|
||||
if rules and rules.strip():
|
||||
loaded = json.loads(rules)
|
||||
if isinstance(loaded, dict):
|
||||
loaded = loaded.get("rules", [])
|
||||
if isinstance(loaded, list):
|
||||
parsed = loaded
|
||||
|
||||
sd = comfy.utils.load_torch_file(src, safe_load=True)
|
||||
edited, stats = apply_rules(sd, parsed)
|
||||
|
||||
if save_dtype != "keep":
|
||||
want = {"float16": torch.float16, "bfloat16": torch.bfloat16, "float32": torch.float32}[save_dtype]
|
||||
edited = {k: (v.to(want) if v.is_floating_point() else v) for k, v in edited.items()}
|
||||
|
||||
stem = os.path.splitext(os.path.basename(src))[0]
|
||||
typed = re.sub(r"[^A-Za-z0-9._ -]", "", filename).strip()
|
||||
if typed:
|
||||
if typed.endswith(".safetensors"):
|
||||
typed = typed[: -len(".safetensors")]
|
||||
name = fit_filename(typed, "", "").rstrip("_") + ".safetensors"
|
||||
else:
|
||||
name = fit_filename(stem, describe_rules(parsed))
|
||||
root = folder_paths.get_folder_paths("loras")[0]
|
||||
out_dir = os.path.join(root, subfolder.strip()) if subfolder.strip() else root
|
||||
os.makedirs(out_dir, exist_ok=True)
|
||||
dest = os.path.join(out_dir, name)
|
||||
if len(dest) > 240:
|
||||
print(f"[BFSNodes] warning: the path is {len(dest)} characters, which some systems reject")
|
||||
if os.path.exists(dest) and not overwrite:
|
||||
base, ext = os.path.splitext(dest)
|
||||
n = 1
|
||||
while os.path.exists(f"{base}_{n}{ext}"):
|
||||
n += 1
|
||||
dest = f"{base}_{n}{ext}"
|
||||
|
||||
meta = {}
|
||||
try:
|
||||
with open(src, "rb") as fh:
|
||||
hn = struct.unpack("<Q", fh.read(8))[0]
|
||||
meta = json.loads(fh.read(hn)).get("__metadata__", {}) or {}
|
||||
except Exception: # noqa: BLE001 - metadata is a nicety, not a requirement
|
||||
meta = {}
|
||||
meta = {str(k): str(v) for k, v in meta.items()}
|
||||
total_modules = sum(1 for k in sd if k.endswith(_UP_SUFFIXES))
|
||||
kept_modules = stats["kept"] if stats["kept"] is not None else total_modules
|
||||
summary = summarize_rules(parsed)
|
||||
meta["bfs_surgery"] = json.dumps({
|
||||
"tool": "ComfyUI-BFSNodes / BFS LoRA Surgery",
|
||||
"created": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
||||
"source_file": os.path.basename(src),
|
||||
"source_sha256": _file_digest(src),
|
||||
"recipe": describe_rules(parsed),
|
||||
"summary": summary,
|
||||
"rules": parsed,
|
||||
"modules_total": total_modules,
|
||||
"modules_kept": kept_modules,
|
||||
"modules_dropped": stats["dropped"],
|
||||
"modules_scaled": stats["scaled"],
|
||||
"save_dtype": save_dtype,
|
||||
}, ensure_ascii=False)
|
||||
# a line any viewer shows without knowing about this tool
|
||||
meta["bfs_surgery_summary"] = (f"{kept_modules}/{total_modules} modules kept: {summary} "
|
||||
f"(from {os.path.basename(src)})")
|
||||
meta["ss_output_name"] = os.path.splitext(name)[0]
|
||||
|
||||
from safetensors.torch import save_file
|
||||
save_file({k: v.contiguous() for k, v in edited.items()}, dest, metadata=meta)
|
||||
total = total_modules
|
||||
kept = kept_modules
|
||||
msg = f"saved {os.path.basename(dest)} ({kept}/{total} modules, {stats['dropped']} dropped)"
|
||||
return dest, msg
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"BFSLoraSurgery": BFSLoraSurgery, "BFSLoraSurgerySave": BFSLoraSurgerySave}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"BFSLoraSurgery": "BFS LoRA Surgery",
|
||||
"BFSLoraSurgerySave": "BFS LoRA Surgery (save)"}
|
||||
|
||||
# ---------------------------------------------------------------------------- http api
|
||||
|
||||
try:
|
||||
from aiohttp import web
|
||||
from server import PromptServer
|
||||
|
||||
_STRUCT_CACHE: dict[tuple[str, float], dict] = {}
|
||||
|
||||
@PromptServer.instance.routes.get("/bfs/lora/structure")
|
||||
async def _bfs_lora_structure(request):
|
||||
name = request.query.get("name", "")
|
||||
with_norms = request.query.get("norms", "1") not in ("0", "false", "no")
|
||||
path = folder_paths.get_full_path("loras", name)
|
||||
if path is None:
|
||||
return web.json_response({"error": f"LoRA not found: {name}"}, status=404)
|
||||
key = (path, os.path.getmtime(path))
|
||||
cached = _STRUCT_CACHE.get(key)
|
||||
if cached is None or (with_norms and not cached.get("per_module_norms")):
|
||||
try:
|
||||
cached = read_structure(path, with_norms=with_norms)
|
||||
except Exception as exc: # noqa: BLE001 - surface the reason in the UI
|
||||
return web.json_response({"error": f"{type(exc).__name__}: {exc}"}, status=500)
|
||||
_STRUCT_CACHE[key] = cached
|
||||
return web.json_response(cached)
|
||||
|
||||
@PromptServer.instance.routes.post("/bfs/lora/save")
|
||||
async def _bfs_lora_save(request):
|
||||
body = await request.json()
|
||||
try:
|
||||
dest, msg = write_surgery(
|
||||
body.get("name", ""), json.dumps(body.get("rules", [])),
|
||||
body.get("filename", ""), body.get("subfolder", "surgery"),
|
||||
body.get("save_dtype", "keep"), bool(body.get("overwrite", False)))
|
||||
except Exception as exc: # noqa: BLE001 - the panel shows the reason
|
||||
return web.json_response({"error": f"{type(exc).__name__}: {exc}"}, status=500)
|
||||
return web.json_response({"path": dest, "message": msg})
|
||||
|
||||
@PromptServer.instance.routes.post("/bfs/lora/regex")
|
||||
async def _bfs_lora_regex(request):
|
||||
body = await request.json()
|
||||
return web.json_response({
|
||||
"regex": build_regex(body.get("family"), body.get("type"), body.get("blocks"))})
|
||||
except Exception as _exc: # noqa: BLE001 - the nodes still work without the panel
|
||||
print(f"[BFSNodes] LoRA surgery HTTP routes not registered: {_exc!r}")
|
||||
+4112
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,323 @@
|
||||
"""BFS Shot Noise: the sampler's starting noise, with optional structure. Every option is off at 0, and with all of
|
||||
them off the output is exactly ComfyUI's RandomNoise for the same seed.
|
||||
|
||||
What can be shaped (the video part of the latent only; the audio noise is always plain):
|
||||
seed per shot same for every shot / increment / random per shot / overrides (as BFS Shot Seed)
|
||||
shared_base the COARSE pattern of the noise (block averages on a fixed grid) comes from a base seed instead
|
||||
of the shot's seed; the fine detail stays the shot's own. The coarse pattern decides layout and
|
||||
who the subject becomes in the first steps, so shots of one video (or re-rolls of one clip)
|
||||
agree on it. The grid is defined in blocks, not in latent cells, so the same base seed gives the
|
||||
same coarse pattern at any resolution, and along the timeline it is one long tape: a shot reads
|
||||
the part at its own position, whatever its length.
|
||||
distribution gaussian (default) / laplacian / student-t / uniform, all at unit variance
|
||||
spectrum_tilt 1/f^alpha spectrum: more (or less) energy in the large blobs ("fractal" noise)
|
||||
pyramid coarser layers of noise added on top
|
||||
static_share a part of the noise that is the same in every frame (temporally correlated noise)
|
||||
guide_structure outside the mask, a little of the guide video's own latent is mixed into the noise, so the
|
||||
background starts closer to the guide; inside the mask the noise stays free (no pull toward
|
||||
the original person)
|
||||
ref_tint inside the mask, the noise leans toward the reference picture's average latent (its colours)
|
||||
|
||||
All mixes keep the noise at unit variance (a*signal + sqrt(1-a^2)*noise, block means replaced by block means of the same
|
||||
variance). The model was trained on plain noise: small values, and compare with 0."""
|
||||
import hashlib
|
||||
import math
|
||||
|
||||
import torch
|
||||
|
||||
import comfy.nested_tensor
|
||||
import comfy.sample
|
||||
|
||||
try:
|
||||
from .bfs_shot_loop import shot_seed, BFSShotSeed
|
||||
from .bfs_h3_side_panel import latent_mask
|
||||
except ImportError: # run as a script (tests)
|
||||
from bfs_shot_loop import shot_seed, BFSShotSeed
|
||||
from bfs_h3_side_panel import latent_mask
|
||||
|
||||
|
||||
_pick_seed = shot_seed # BFSShotNoise.make has an input of the same name
|
||||
|
||||
|
||||
def _gen(*key) -> torch.Generator:
|
||||
return torch.Generator().manual_seed(int.from_bytes(hashlib.sha256("/".join(str(k) for k in key).encode()).digest()[:8], "big") >> 1)
|
||||
|
||||
|
||||
def _edges(n: int, blocks: int) -> torch.Tensor:
|
||||
"""Block id of each of n cells for `blocks` near-equal blocks (blocks is capped at n)."""
|
||||
blocks = max(1, min(int(blocks), n))
|
||||
return torch.clamp((torch.arange(n) * blocks) // n, max=blocks - 1)
|
||||
|
||||
|
||||
def replace_coarse(x: torch.Tensor, base_seed: int, amount: float, grid: int = 6, time_block: int = 2, t0: int = 0) -> torch.Tensor:
|
||||
"""x [C,T,h,w] unit noise. The mean of every (time block, y block, x block) cell is replaced (amount 1) or mixed
|
||||
with a mean drawn from `base_seed`; the result is unit noise again. Blocks: `grid` along the short side, the long
|
||||
side in proportion; `time_block` latent frames, counted from timeline position t0."""
|
||||
if amount <= 0:
|
||||
return x
|
||||
C, T, h, w = x.shape
|
||||
hb, wb = (grid, max(1, round(grid * w / h))) if h <= w else (max(1, round(grid * h / w)), grid)
|
||||
yid, xid = _edges(h, hb), _edges(w, wb)
|
||||
hb, wb = int(yid.max()) + 1, int(xid.max()) + 1
|
||||
tid_abs = (t0 + torch.arange(T)) // max(1, int(time_block))
|
||||
tids = torch.unique(tid_abs)
|
||||
tloc = torch.searchsorted(tids, tid_abs)
|
||||
nb = len(tids) * hb * wb
|
||||
ids = ((tloc[:, None, None] * hb + yid[None, :, None]) * wb + xid[None, None, :]).reshape(-1)
|
||||
flat = x.reshape(C, -1).double()
|
||||
cnt = torch.zeros(nb, dtype=torch.float64).index_add_(0, ids, torch.ones_like(ids, dtype=torch.float64))
|
||||
mean = torch.zeros(C, nb, dtype=torch.float64).index_add_(1, ids, flat) / cnt
|
||||
z = torch.stack([torch.randn(C, hb, wb, generator=_gen("coarse", base_seed, int(t), hb, wb), dtype=torch.float64) for t in tids], 1).reshape(C, nb)
|
||||
a = float(min(1.0, amount))
|
||||
new = math.sqrt(1.0 - a * a) * mean + a * z / cnt.sqrt()
|
||||
return (flat + (new - mean)[:, ids]).reshape(C, T, h, w).to(x.dtype)
|
||||
|
||||
|
||||
DISTRIBUTIONS = ["gaussian", "laplacian", "student-t", "uniform"]
|
||||
|
||||
|
||||
def draw(shape, kind: str, seed: int) -> torch.Tensor:
|
||||
"""Unit-variance noise of another distribution (heavier or lighter tails than the gaussian the model was trained on)."""
|
||||
g = _gen("dist", kind, seed)
|
||||
if kind == "laplacian":
|
||||
u = torch.rand(shape, generator=g, dtype=torch.float64) - 0.5
|
||||
return (-torch.sign(u) * torch.log1p(-2 * u.abs().clamp(max=0.5 - 1e-12)) / math.sqrt(2.0)).float()
|
||||
if kind == "student-t": # 5 degrees of freedom
|
||||
n = torch.randn(shape, generator=g, dtype=torch.float64)
|
||||
chi = torch.randn((5,) + tuple(shape), generator=g, dtype=torch.float64).pow(2).sum(0)
|
||||
return (n / torch.sqrt(chi / 5.0) * math.sqrt(3.0 / 5.0)).float()
|
||||
if kind == "uniform":
|
||||
return ((torch.rand(shape, generator=g, dtype=torch.float64) - 0.5) * math.sqrt(12.0)).float()
|
||||
return torch.randn(shape, generator=g)
|
||||
|
||||
|
||||
def tilt_spectrum(x: torch.Tensor, alpha: float) -> torch.Tensor:
|
||||
"""x [C,T,h,w]: power spectrum multiplied by 1/f^alpha over time and space (alpha > 0 = more energy in the large
|
||||
blobs, 'fractal' noise; alpha < 0 = more in the fine grain), then back to zero mean and unit variance per channel."""
|
||||
if alpha == 0:
|
||||
return x
|
||||
C, T, h, w = x.shape
|
||||
f = torch.sqrt(torch.fft.fftfreq(T)[:, None, None] ** 2 + torch.fft.fftfreq(h)[None, :, None] ** 2 + torch.fft.rfftfreq(w)[None, None, :] ** 2)
|
||||
f[0, 0, 0] = f[f > 0].min()
|
||||
y = torch.fft.irfftn(torch.fft.rfftn(x.double(), dim=(1, 2, 3)) * f.pow(-alpha / 2.0), s=(T, h, w), dim=(1, 2, 3))
|
||||
y = y - y.mean(dim=(1, 2, 3), keepdim=True)
|
||||
return (y / y.std(dim=(1, 2, 3), keepdim=True).clamp_min(1e-8)).to(x.dtype)
|
||||
|
||||
|
||||
def add_pyramid(x: torch.Tensor, amount: float, seed: int, levels: int = 4) -> torch.Tensor:
|
||||
"""x [C,T,h,w] plus coarser copies of fresh noise (each level half the size, weight amount^level), unit variance."""
|
||||
if amount <= 0:
|
||||
return x
|
||||
C, T, h, w = x.shape
|
||||
y = x.double()
|
||||
for i in range(1, levels + 1):
|
||||
hh, ww = max(1, h >> i), max(1, w >> i)
|
||||
if hh == 1 and ww == 1:
|
||||
break
|
||||
n = torch.randn(C, T, hh, ww, generator=_gen("pyramid", seed, i), dtype=torch.float64)
|
||||
y = y + (amount ** i) * torch.nn.functional.interpolate(n, size=(h, w), mode="bilinear", align_corners=False)
|
||||
y = y - y.mean(dim=(1, 2, 3), keepdim=True)
|
||||
return (y / y.std(dim=(1, 2, 3), keepdim=True).clamp_min(1e-8)).to(x.dtype)
|
||||
|
||||
|
||||
def mix(x: torch.Tensor, signal: torch.Tensor, weight: torch.Tensor) -> torch.Tensor:
|
||||
weight = weight.clamp(0, 1)
|
||||
return torch.sqrt(1.0 - weight ** 2) * x + weight * signal
|
||||
|
||||
|
||||
class ShotNoise:
|
||||
def __init__(self, seed, base_seed, opts, guide=None, tint=None, mask=None, t0=0):
|
||||
self.seed, self.base_seed, self.o, self.guide, self.tint, self.mask, self.t0 = int(seed), int(base_seed), opts, guide, tint, mask, int(t0)
|
||||
|
||||
def generate_noise(self, input_latent):
|
||||
samples = input_latent["samples"]
|
||||
inds = input_latent["batch_index"] if "batch_index" in input_latent else None
|
||||
noise = comfy.sample.prepare_noise(samples, self.seed, inds) # exactly RandomNoise
|
||||
o = self.o
|
||||
if not any(o[k] > 0 for k in ("shared_base", "static_share", "guide_structure", "ref_tint", "pyramid")) \
|
||||
and o["spectrum_tilt"] == 0 and o["distribution"] == "gaussian":
|
||||
return noise
|
||||
nested = getattr(noise, "is_nested", False)
|
||||
parts = list(noise.unbind()) if nested else [noise]
|
||||
v = parts[0]
|
||||
if v.ndim != 5:
|
||||
return noise
|
||||
out = []
|
||||
for b in range(v.shape[0]):
|
||||
x = v[b].float()
|
||||
C, T, h, w = x.shape
|
||||
if o["distribution"] != "gaussian":
|
||||
x = draw((C, T, h, w), o["distribution"], self.seed + b)
|
||||
x = add_pyramid(tilt_spectrum(x, o["spectrum_tilt"]), o["pyramid"], self.seed + b)
|
||||
if o["static_share"] > 0:
|
||||
s = torch.randn(C, 1, h, w, generator=_gen("static", self.base_seed, h, w))
|
||||
x = mix(x, s.expand_as(x), torch.tensor(float(o["static_share"])))
|
||||
x = replace_coarse(x, self.base_seed, o["shared_base"], o["base_grid"], o["time_block"], self.t0)
|
||||
m = None
|
||||
if self.mask is not None:
|
||||
m = latent_mask(self.mask, T, h, w, grow=1)[0].to(x) # [1,T,h,w], 1 = the person
|
||||
if o["guide_structure"] > 0 and self.guide is not None:
|
||||
g = self.guide[min(b, self.guide.shape[0] - 1)].float().cpu()
|
||||
if tuple(g.shape[1:]) != (T, h, w):
|
||||
g = torch.nn.functional.interpolate(g[None], size=(T, h, w), mode="trilinear", align_corners=False)[0]
|
||||
g = (g - g.mean(dim=(1, 2, 3), keepdim=True)) / g.std(dim=(1, 2, 3), keepdim=True).clamp_min(1e-6)
|
||||
x = mix(x, g, float(o["guide_structure"]) * ((1.0 - m) if m is not None else torch.ones(1, T, h, w)))
|
||||
if o["ref_tint"] > 0 and self.tint is not None:
|
||||
x = mix(x, self.tint.view(C, 1, 1, 1).expand_as(x), float(o["ref_tint"]) * (m if m is not None else torch.ones(1, T, h, w)))
|
||||
out.append(x.to(v.dtype))
|
||||
parts[0] = torch.stack(out)
|
||||
if nested:
|
||||
return comfy.nested_tensor.NestedTensor(parts)
|
||||
return parts[0]
|
||||
|
||||
|
||||
class BFSShotNoise:
|
||||
"""Starting noise for the sampler with optional structure (see the node description)."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
f = lambda tip, mx=1.0: ("FLOAT", {"default": 0.0, "min": 0.0, "max": mx, "step": 0.01, "tooltip": tip})
|
||||
return {
|
||||
"required": {
|
||||
"seed": ("INT", {"default": 0, "min": 0, "max": 0xffffffffffffffff, "control_after_generate": True,
|
||||
"tooltip": "The seed (the base seed when a shot is connected)."}),
|
||||
"shot_seed": (BFSShotSeed.MODES, {"default": BFSShotSeed.MODES[0], "tooltip":
|
||||
"Only with a shot connected: the seed of each shot (same / counting up / unrelated but repeatable)."}),
|
||||
"shared_base": f("0 = off (plain noise). The coarse pattern of the noise comes from base_seed instead of "
|
||||
"the shot's seed: every shot, any length or size, agrees on it; the fine detail stays "
|
||||
"per shot. 1 = fully shared. Try 0.5 - 1."),
|
||||
"static_share": f("0 = off. A part of the noise that is identical in every frame (temporally "
|
||||
"correlated noise). Try 0.1 - 0.3; too much freezes the motion.", 0.9),
|
||||
"guide_structure": f("0 = off. Needs guide_frames + vae. Outside the mask the guide's latent is mixed "
|
||||
"into the noise, so the background starts closer to the guide (without a mask: "
|
||||
"everywhere, which also pulls toward the original person). Try 0.05 - 0.2.", 0.6),
|
||||
"distribution": (DISTRIBUTIONS, {"default": "gaussian", "tooltip":
|
||||
"gaussian = what the model was trained on. laplacian / student-t: more extreme values (stronger, "
|
||||
"sparser kicks). uniform: none. All at unit variance. Like AdvancedNoise's types, but for H3's "
|
||||
"video + audio latent (the audio noise always stays gaussian)."}),
|
||||
"spectrum_tilt": ("FLOAT", {"default": 0.0, "min": -2.0, "max": 2.0, "step": 0.05, "tooltip":
|
||||
"0 = off (white noise). > 0: more energy in the large blobs (fractal / pink noise, AdvancedNoise's "
|
||||
"'fractal' alpha): the coarse layout, where the swap is decided, gets a harder push. < 0: more "
|
||||
"fine grain. Try 0.2 - 0.6; large values burn colours."}),
|
||||
"pyramid": f("0 = off. Adds coarser layers of noise (each half the size, weight = this value per "
|
||||
"level): another way to put more energy in the large scales. Try 0.3 - 0.6.", 0.95),
|
||||
"ref_tint": f("0 = off. Needs ref_image + vae. Inside the mask (everywhere without one) the noise "
|
||||
"leans toward the reference's average latent, i.e. its colours. Try 0.05 - 0.15.", 0.6),
|
||||
},
|
||||
"optional": {
|
||||
"shot": ("BFS_SHOT", {"tooltip": "From BFS Shot Planner / Unpack: per-shot seeds, and the shared base is "
|
||||
"read at the shot's position on the timeline."}),
|
||||
"base_seed": ("INT", {"default": -1, "min": -1, "max": 0xffffffffffffffff, "tooltip":
|
||||
"Seed of the shared coarse pattern and of static_share. -1 = the seed above. Put here the seed of "
|
||||
"a result that worked to keep its coarse noise while the detail changes with the seed."}),
|
||||
"overrides": ("STRING", {"default": "", "tooltip": "Seeds for single shots: 'shot:seed, shot:seed' (shots count from 1)."}),
|
||||
"base_grid": ("INT", {"default": 6, "min": 1, "max": 32, "advanced": True, "tooltip":
|
||||
"Blocks of the coarse pattern along the short side (the long side in proportion). Fewer = only the "
|
||||
"broadest layout is shared."}),
|
||||
"time_block": ("INT", {"default": 2, "min": 1, "max": 32, "advanced": True, "tooltip": "Latent frames per block of the coarse pattern."}),
|
||||
"vae": ("VAE", {"tooltip": "The video VAE, for guide_structure and ref_tint."}),
|
||||
"guide_frames": ("IMAGE", {"tooltip": "The guide video frames (guide_structure)."}),
|
||||
"ref_image": ("IMAGE", {"tooltip": "The reference picture (ref_tint)."}),
|
||||
"mask": ("MASK", {"tooltip": "The person, one mask per frame (white = the person): guide_structure stays "
|
||||
"outside it, ref_tint inside it."}),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("NOISE", "INT")
|
||||
RETURN_NAMES = ("noise", "seed")
|
||||
OUTPUT_TOOLTIPS = ("For SamplerCustomAdvanced.", "The seed this shot used.")
|
||||
FUNCTION = "make"
|
||||
CATEGORY = "BFS/shot loop"
|
||||
DESCRIPTION = ("Optional structured starting noise. With every amount at 0 it is exactly RandomNoise. shared_base: "
|
||||
"the coarse noise pattern is the same for every shot / size / length (from base_seed), the detail "
|
||||
"per shot. static_share: a part identical in all frames. guide_structure: the guide's latent mixed "
|
||||
"in outside the mask. ref_tint: lean toward the reference's colours inside the mask. Experimental: "
|
||||
"the model was trained on plain noise, so use small amounts and compare with 0.")
|
||||
|
||||
def make(self, seed, shot_seed, shared_base, static_share, guide_structure, ref_tint, distribution="gaussian", spectrum_tilt=0.0,
|
||||
pyramid=0.0, shot=None, base_seed=-1, overrides="",
|
||||
base_grid=6, time_block=2, vae=None, guide_frames=None, ref_image=None, mask=None):
|
||||
index = int(shot.get("index", 0)) if shot else 0
|
||||
s = _pick_seed(int(seed), index, shot_seed, overrides) if shot else int(seed)
|
||||
base = int(seed) if int(base_seed) < 0 else int(base_seed)
|
||||
guide = tint = None
|
||||
if guide_structure > 0:
|
||||
if vae is None or guide_frames is None:
|
||||
raise ValueError("BFS Shot Noise: guide_structure needs vae and guide_frames")
|
||||
guide = vae.encode(guide_frames[..., :3])
|
||||
guide = guide.tensors[0] if getattr(guide, "is_nested", False) else guide
|
||||
if ref_tint > 0:
|
||||
if vae is None or ref_image is None:
|
||||
raise ValueError("BFS Shot Noise: ref_tint needs vae and ref_image")
|
||||
r = vae.encode(ref_image[:1, ..., :3])
|
||||
r = (r.tensors[0] if getattr(r, "is_nested", False) else r).float().cpu()
|
||||
tint = (r.mean(dim=(0, 2, 3, 4)) / r.std(dim=(0, 2, 3, 4)).clamp_min(1e-6)).clamp(-2, 2)
|
||||
t0 = int(round(int(shot.get("start", 0)) / 4)) if shot else 0 # timeline position in latent frames
|
||||
opts = {"shared_base": float(shared_base), "static_share": float(static_share), "guide_structure": float(guide_structure),
|
||||
"ref_tint": float(ref_tint), "base_grid": int(base_grid), "time_block": int(time_block),
|
||||
"distribution": distribution, "spectrum_tilt": float(spectrum_tilt), "pyramid": float(pyramid)}
|
||||
return (ShotNoise(s, base, opts, guide, tint, mask, t0), s)
|
||||
|
||||
|
||||
def sigmas_chart(sigmas: torch.Tensor, other: torch.Tensor | None = None, w: int = 640, h: int = 360):
|
||||
"""(image [1,h,w,3], text): the noise level at the start of every step, with the zones where the steps are spent."""
|
||||
import numpy as np
|
||||
from PIL import Image, ImageDraw
|
||||
s = [float(v) for v in sigmas.flatten().tolist()]
|
||||
top = max(1e-6, s[0])
|
||||
img = Image.new("RGB", (w, h), (24, 24, 28)); d = ImageDraw.Draw(img)
|
||||
L, R, T, B = 52, w - 16, 34, h - 40
|
||||
y = lambda v: B - (v / top) * (B - T)
|
||||
zones = ((0.85, 1.01, (60, 40, 40), "layout / identity"), (0.5, 0.85, (48, 46, 36), "shapes"), (-0.01, 0.5, (36, 44, 52), "detail"))
|
||||
for lo, hi, col, name in zones:
|
||||
d.rectangle([L, y(min(hi, 1.0) * top), R, y(max(lo, 0.0) * top)], fill=col)
|
||||
d.text((R - 6 - 6 * len(name), y(min(hi, 1.0) * top) + 3), name, fill=(150, 150, 150))
|
||||
for v in (0.0, 0.25, 0.5, 0.75, 1.0):
|
||||
d.line([L, y(v * top), R, y(v * top)], fill=(70, 70, 76)); d.text((8, y(v * top) - 6), f"{v * top:.2f}", fill=(170, 170, 170))
|
||||
|
||||
def line(vals, col, labels):
|
||||
n = len(vals) - 1
|
||||
pts = [(L + (R - L) * i / max(1, n), y(v)) for i, v in enumerate(vals)]
|
||||
d.line(pts, fill=col, width=2)
|
||||
for i, (px, py) in enumerate(pts):
|
||||
d.ellipse([px - 3, py - 3, px + 3, py + 3], fill=col)
|
||||
if labels and i < n:
|
||||
d.text((px - 10, py - 16 if i % 2 == 0 else py + 6), f"{vals[i]:.2f}", fill=(235, 235, 235))
|
||||
d.text((px - 3, B + 8), str(i + 1), fill=(170, 170, 170))
|
||||
|
||||
if other is not None:
|
||||
line([float(v) for v in other.flatten().tolist()], (120, 120, 200), False)
|
||||
line(s, (255, 190, 70), True)
|
||||
steps = s[:-1]
|
||||
cnt = [sum(lo * top < v <= hi * top or (hi > 1 and v >= lo * top) for v in steps) for lo, hi, _, _ in zones]
|
||||
d.text((L, 10), f"{len(steps)} steps: {cnt[0]} on layout / identity (> {0.85 * top:.2f}), {cnt[1]} on shapes, {cnt[2]} on detail (< {0.5 * top:.2f})"
|
||||
f" last jump {steps[-1]:.2f} -> {s[-1]:.2f}", fill=(235, 235, 235))
|
||||
d.text((R - 24, B + 8), "step", fill=(170, 170, 170))
|
||||
text = (f"{len(steps)} steps | layout/identity {cnt[0]}, shapes {cnt[1]}, detail {cnt[2]} | sigmas: " + ", ".join(f"{v:.3f}" for v in s))
|
||||
return torch.from_numpy(np.asarray(img).astype("float32") / 255.0)[None], text
|
||||
|
||||
|
||||
class BFSSigmasPreview:
|
||||
"""Draws a sigma schedule: where the sampler's steps are spent (pass-through)."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {"sigmas": ("SIGMAS", {"tooltip": "The schedule that goes to the sampler (after the scheduler and the shift)."})},
|
||||
"optional": {"compare": ("SIGMAS", {"tooltip": "Another schedule drawn behind it, e.g. the same steps with another shift."})}}
|
||||
|
||||
RETURN_TYPES = ("SIGMAS", "IMAGE", "STRING")
|
||||
RETURN_NAMES = ("sigmas", "chart", "text")
|
||||
OUTPUT_TOOLTIPS = ("The same sigmas, untouched.", "The chart: connect a Preview Image.", "The values and the step count per zone.")
|
||||
FUNCTION = "show"
|
||||
CATEGORY = "BFS/shot loop"
|
||||
DESCRIPTION = ("Shows a sigma schedule: the noise level at the start of each step and how many steps fall on layout / "
|
||||
"identity (high noise), shapes and detail (low noise). With H3's video shift of 12 most steps sit at "
|
||||
"the top; lowering the shift moves steps toward detail. Changes nothing: sigmas pass through.")
|
||||
|
||||
def show(self, sigmas, compare=None):
|
||||
chart, text = sigmas_chart(sigmas, compare)
|
||||
return (sigmas, chart, text)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"BFSShotNoise": BFSShotNoise, "BFSSigmasPreview": BFSSigmasPreview}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"BFSShotNoise": "BFS Shot Noise", "BFSSigmasPreview": "BFS Sigmas Preview"}
|
||||
@@ -0,0 +1,86 @@
|
||||
"""Optional blur of the source person in what the model SEES of the source (aligned guide, native reference video,
|
||||
duet panel), keeping the mouth sharp: the model still reads the pose, the motion and the lips, but has no face to copy,
|
||||
so the identity comes from the reference. JalenBrunson's trick for reliable ref2v swaps (Sapiens2 there); here the
|
||||
masks come from SAM 3.1, already used by the planner. The kept area (inpaint) and the output are never blurred.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import torch
|
||||
|
||||
SOURCE_BLUR = ["off", "face (keep mouth)", "head + hands (keep mouth)", "person (keep mouth)"]
|
||||
_PARTS = {SOURCE_BLUR[1]: ("face",), SOURCE_BLUR[2]: ("head", "hand"), SOURCE_BLUR[3]: ("person",)}
|
||||
_CACHE: dict = {}
|
||||
|
||||
|
||||
def _sl():
|
||||
try:
|
||||
from . import bfs_shot_loop as SL
|
||||
except ImportError:
|
||||
import bfs_shot_loop as SL
|
||||
return SL
|
||||
|
||||
|
||||
def _small(frames: torch.Tensor, side: int = 640) -> torch.Tensor:
|
||||
H, W = frames.shape[1:3]
|
||||
s = side / max(H, W)
|
||||
x = frames[..., :3].float()
|
||||
if s >= 1:
|
||||
return x
|
||||
return torch.nn.functional.interpolate(x.movedim(-1, 1), size=(max(32, int(H * s) // 2 * 2), max(32, int(W * s) // 2 * 2)),
|
||||
mode="bilinear", align_corners=False).movedim(1, -1)
|
||||
|
||||
|
||||
def _grow(m: torch.Tensor, px: int) -> torch.Tensor:
|
||||
if px <= 0:
|
||||
return m
|
||||
return torch.nn.functional.max_pool2d(m[:, None], 2 * px + 1, stride=1, padding=px)[:, 0]
|
||||
|
||||
|
||||
def blur_mask(frames: torch.Tensor, mode: str, log: str = "") -> torch.Tensor | None:
|
||||
"""[N,h,w] float mask (working size) of what to blur: the parts for `mode` minus the (slightly grown) mouth."""
|
||||
parts = _PARTS.get(mode)
|
||||
if not parts:
|
||||
return None
|
||||
SL = _sl()
|
||||
small = _small(frames)
|
||||
key = (mode, tuple(small.shape), SL._image_key(frames[:1]), SL._image_key(frames[-1:]),
|
||||
SL._image_key(frames[frames.shape[0] // 2:frames.shape[0] // 2 + 1]))
|
||||
if key in _CACHE:
|
||||
return _CACHE[key]
|
||||
m = torch.zeros(small.shape[:3])
|
||||
for p in parts:
|
||||
m = torch.maximum(m, (SL.segment_frames(small, dict(SL.DEFAULT_MASK, text=p, max_objects=8, threshold=0.4)) > 0.5).float())
|
||||
mouth = (SL.segment_frames(small, dict(SL.DEFAULT_MASK, text="mouth", max_objects=8, threshold=0.3)) > 0.5).float()
|
||||
keep = _grow(mouth, max(1, int(min(small.shape[1:3]) * 0.012)))
|
||||
out = (_grow(m, max(1, int(min(small.shape[1:3]) * 0.015))) * (1 - keep)).clamp(0, 1)
|
||||
if log:
|
||||
print(f"{log}: source blur '{mode}': {float(m.mean()) * 100:.1f}% of the frame, mouth kept "
|
||||
f"({'found' if float(mouth.sum()) > 0 else 'not found: the whole part is blurred'})", flush=True)
|
||||
if len(_CACHE) > 8:
|
||||
_CACHE.pop(next(iter(_CACHE)))
|
||||
_CACHE[key] = out
|
||||
return out
|
||||
|
||||
|
||||
def blur_source(frames: torch.Tensor | None, mode: str, strength: float = 1.0, log: str = "") -> torch.Tensor | None:
|
||||
"""The frames with the source person blurred per `mode` (mouth kept). `strength` 0-1 sets how coarse the blur is."""
|
||||
if frames is None or not mode or mode == "off" or strength <= 0:
|
||||
return frames
|
||||
m = blur_mask(frames, mode, log)
|
||||
if m is None or float(m.sum()) == 0:
|
||||
return frames
|
||||
x = frames[..., :3].float()
|
||||
N, H, W = x.shape[:3]
|
||||
# a heavy blur: down to ~1/12-1/40 of the size and back, then a soft pass; blocky detail never survives it
|
||||
f = max(4, int(min(H, W) * (0.025 + 0.06 * float(strength))))
|
||||
low = torch.nn.functional.interpolate(x.movedim(-1, 1), size=(max(2, H // f), max(2, W // f)), mode="area")
|
||||
blur = torch.nn.functional.interpolate(low, size=(H, W), mode="bilinear", align_corners=False)
|
||||
blur = torch.nn.functional.avg_pool2d(blur, 5, stride=1, padding=2).movedim(1, -1)
|
||||
mm = torch.nn.functional.interpolate(m[:, None], size=(H, W), mode="bilinear", align_corners=False)
|
||||
mm = torch.nn.functional.avg_pool2d(mm, 9, stride=1, padding=4)[:, 0, ..., None].clamp(0, 1)
|
||||
if mm.shape[0] != N: # one mask per frame; a shorter list holds its last one
|
||||
idx = [min(i, mm.shape[0] - 1) for i in range(N)]
|
||||
mm = mm[idx]
|
||||
out = x * (1 - mm) + blur * mm
|
||||
del low, blur, mm
|
||||
return out
|
||||
@@ -0,0 +1,291 @@
|
||||
"""Stable subject-crop planning, vendored from drozbay/MaskVidExperiments.
|
||||
|
||||
Source: https://github.com/drozbay/MaskVidExperiments (commit d98cc89, v0.2.0)
|
||||
Copyright (c) drozbay. Licensed GPL-3.0, the same licence this pack carries,
|
||||
which is what makes vendoring it legal here.
|
||||
|
||||
Vendored rather than imported so the head-swap sampler has no runtime
|
||||
dependency on that pack being installed. Behaviour is unchanged; only the
|
||||
ComfyUI node classes were dropped (they stay in the original pack, where they
|
||||
belong) and the planner import was repointed.
|
||||
|
||||
Why this code and not a crop of my own: naive per-frame crops around a mask
|
||||
jitter in position and size, and a video model reads that jitter as camera
|
||||
motion. These boxes are planned over the whole clip, so they hold still through
|
||||
mask noise and occlusion and follow only sustained movement.
|
||||
"""
|
||||
|
||||
"""Subject-tracked batch crop/uncrop for video mask workflows. The crop
|
||||
boxes come from planner.py, which plans position, size, and shape jointly
|
||||
over the whole clip."""
|
||||
|
||||
import logging
|
||||
import math
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from scipy import ndimage
|
||||
|
||||
import comfy.utils
|
||||
|
||||
from . import bfs_crop_planner as planner
|
||||
|
||||
CATEGORY = "MaskVidExperiments"
|
||||
|
||||
|
||||
def _clean_mask_stack(masks, value_thresh, min_pixels, min_frames):
|
||||
"""Binarize a mask batch and drop noise blobs.
|
||||
|
||||
Connected components are labelled in 3D (time + space). A blob survives if
|
||||
its peak single-frame area clears min_pixels and it is either subject-sized
|
||||
(peak >= 50% of the largest blob, so fast motion that splits the subject
|
||||
into per-frame components doesn't delete it) or persists min_frames frames.
|
||||
"""
|
||||
binary = masks.detach().cpu().numpy() > value_thresh
|
||||
if not binary.any():
|
||||
return binary
|
||||
|
||||
labels, count = ndimage.label(binary, structure=np.ones((3, 3, 3), dtype=bool))
|
||||
if count == 0:
|
||||
return binary
|
||||
|
||||
span = np.zeros(count + 1, dtype=np.int64)
|
||||
for label, location in enumerate(ndimage.find_objects(labels), start=1):
|
||||
if location is not None:
|
||||
span[label] = location[0].stop - location[0].start
|
||||
|
||||
peak = np.zeros(count + 1, dtype=np.int64)
|
||||
for frame in labels:
|
||||
peak = np.maximum(peak, np.bincount(frame.ravel(), minlength=count + 1))
|
||||
|
||||
span[0] = 0
|
||||
peak[0] = 0
|
||||
subject = peak >= 0.5 * peak.max() if peak.max() > 0 else np.zeros_like(peak, dtype=bool)
|
||||
keep = (peak >= min_pixels) & (subject | (span >= min_frames))
|
||||
keep[0] = False
|
||||
return keep[labels]
|
||||
|
||||
|
||||
def _open_reconstruct(masks, value_thresh, radius, min_frames):
|
||||
"""Morphological opening by reconstruction on a mask batch.
|
||||
|
||||
Erodes each frame spatially so blobs thinner than ~2*radius vanish, then
|
||||
instead of dilating back, keeps the original shape of every blob with a
|
||||
surviving core, so the subject's silhouette (thin protrusions included) is
|
||||
untouched. Blobs are 3D components (time + space): a core surviving in any
|
||||
frame keeps the whole track. border_value=1 so subjects cut off by the
|
||||
frame edge are not eroded from that side.
|
||||
"""
|
||||
binary = masks.detach().cpu().numpy() > value_thresh
|
||||
if not binary.any():
|
||||
return binary
|
||||
|
||||
spatial = np.zeros((3, 3, 3), dtype=bool)
|
||||
spatial[1] = [[0, 1, 0], [1, 1, 1], [0, 1, 0]]
|
||||
eroded = ndimage.binary_erosion(binary, structure=spatial, iterations=radius, border_value=1)
|
||||
|
||||
labels, count = ndimage.label(binary, structure=np.ones((3, 3, 3), dtype=bool))
|
||||
if count == 0:
|
||||
return binary
|
||||
|
||||
keep = np.zeros(count + 1, dtype=bool)
|
||||
keep[np.unique(labels[eroded])] = True
|
||||
if min_frames > 1:
|
||||
for label, location in enumerate(ndimage.find_objects(labels), start=1):
|
||||
if location is not None and location[0].stop - location[0].start < min_frames:
|
||||
keep[label] = False
|
||||
keep[0] = False
|
||||
return keep[labels]
|
||||
|
||||
|
||||
def _resize_image(img_hwc, w, h, method="lanczos"):
|
||||
if img_hwc.shape[0] == h and img_hwc.shape[1] == w:
|
||||
return img_hwc
|
||||
return comfy.utils.common_upscale(
|
||||
img_hwc.movedim(-1, 0).unsqueeze(0), w, h, method, "disabled"
|
||||
).squeeze(0).movedim(0, -1)
|
||||
|
||||
|
||||
def _resize_mask(mask_hw, w, h, method="bilinear"):
|
||||
if mask_hw.shape[0] == h and mask_hw.shape[1] == w:
|
||||
return mask_hw
|
||||
return comfy.utils.common_upscale(
|
||||
mask_hw[None, None], w, h, method, "disabled"
|
||||
)[0, 0]
|
||||
|
||||
|
||||
def _upscale_size(w, h, megapixels, divisible_by):
|
||||
"""Size scaling a w x h crop to about abs(megapixels) pixels on the
|
||||
divisible_by grid. A positive value is a floor: None when already at or
|
||||
above it. A negative value is a target, reached by downscaling too."""
|
||||
scale = math.sqrt(abs(megapixels) * 1024 * 1024 / (w * h))
|
||||
if megapixels > 0 and scale <= 1.0:
|
||||
return None
|
||||
return (max(1, round(w * scale / divisible_by)) * divisible_by,
|
||||
max(1, round(h * scale / divisible_by)) * divisible_by)
|
||||
|
||||
|
||||
def _cell_params(mode):
|
||||
"""Raw planner dials for the standard node's (padding, prefer) cell.
|
||||
The advanced node feeds the same dials directly. guaranteed floors the
|
||||
full promise against every raw frame (floor window 1): a true
|
||||
guarantee, so mask noise must be cleaned upstream. firm floors 70%
|
||||
against the sustained tracks; flexible has no floor. In tracked mode
|
||||
the crop never rescales, so prefer differentiates through the oversize
|
||||
rent instead: stillness holds a bigger box to move less. In zoomed
|
||||
mode the rent is never scaled by prefer (cheapening it tips whole
|
||||
clips into one constant maximum size, killing zoom-follow); prefer
|
||||
differentiates through the resize cost."""
|
||||
sel = mode["mode"]
|
||||
pad_level = mode.get("padding", "firm")
|
||||
prefer = mode.get("prefer", "stillness")
|
||||
floor = {"guaranteed": 1.0, "firm": 0.7, "flexible": 0.0}[pad_level]
|
||||
still = prefer == "stillness"
|
||||
if sel == "tracked":
|
||||
oversize = 32.0 if still else 8.0
|
||||
else:
|
||||
oversize = float(mode.get("pad_surplus_tol", 16))
|
||||
return {
|
||||
"crop_scale": mode["crop_scale"],
|
||||
"min_padding_allowed": floor,
|
||||
"min_padding_allowed_window": 1 if pad_level == "guaranteed" else 16,
|
||||
"pad_deficit_tol": 16.0,
|
||||
"pad_surplus_tol": oversize,
|
||||
"resize_cost": 2.0 if still else 1.0,
|
||||
"movement_cost": 1.0,
|
||||
"center_pull": 1e-4,
|
||||
"end_tightening": 0.0,
|
||||
"end_tightening_window": 80 if floor > 0 else 0,
|
||||
"zoom_step": mode.get("zoom_step", 1.0),
|
||||
"max_zoom_rate": 0.0,
|
||||
"aspect_ratio": mode["aspect_ratio"],
|
||||
"seamless_loop": mode.get("seamless_loop", False),
|
||||
}
|
||||
|
||||
|
||||
def _debug_summary(binary, boxes, scale, sel, info, output_size):
|
||||
"""Brief per-run report: what shape the planner chose and how well the
|
||||
plan delivered the padding promise."""
|
||||
n = len(boxes)
|
||||
worst = []
|
||||
img_h, img_w = binary.shape[1:]
|
||||
for i in range(n):
|
||||
ys, xs = binary[i].nonzero()
|
||||
if len(xs) == 0:
|
||||
worst.append(np.nan)
|
||||
continue
|
||||
b = boxes[i]
|
||||
bx1 = b["x"] + b["width"] - 1
|
||||
by1 = b["y"] + b["height"] - 1
|
||||
wx = (scale - 1) / 2 * (xs.max() - xs.min() + 1)
|
||||
wy = (scale - 1) / 2 * (ys.max() - ys.min() + 1)
|
||||
fr = []
|
||||
if b["x"] > 0: fr.append((xs.min() - b["x"]) / max(wx, 1e-6))
|
||||
if bx1 < img_w - 1: fr.append((bx1 - xs.max()) / max(wx, 1e-6))
|
||||
if b["y"] > 0: fr.append((ys.min() - b["y"]) / max(wy, 1e-6))
|
||||
if by1 < img_h - 1: fr.append((by1 - ys.max()) / max(wy, 1e-6))
|
||||
worst.append(min(fr) if fr else np.nan)
|
||||
worst = np.array(worst)
|
||||
v = worst[~np.isnan(worst)]
|
||||
cx = np.array([b["x"] + b["width"] / 2 for b in boxes])
|
||||
cy = np.array([b["y"] + b["height"] / 2 for b in boxes])
|
||||
travel = np.hypot(np.diff(cx), np.diff(cy)).sum()
|
||||
|
||||
ar = info["aspect"]
|
||||
if info.get("swept"):
|
||||
shape = f"{ar:.2f} (chosen from {info['candidates']} candidates)"
|
||||
elif sel == "zoomed":
|
||||
shape = f"{ar:.2f} (manual)"
|
||||
else:
|
||||
shape = f"{ar:.2f}"
|
||||
ws = [b["width"] for b in boxes]
|
||||
hs = [b["height"] for b in boxes]
|
||||
lines = [
|
||||
f"mode: {sel}",
|
||||
f"frames: {n} ({img_w}x{img_h} source)",
|
||||
f"aspect_ratio: {shape}",
|
||||
f"smallest_box: {min(ws)}x{min(hs)}",
|
||||
f"largest_box: {max(ws)}x{max(hs)}",
|
||||
]
|
||||
if sel == "zoomed":
|
||||
lines.append(f"zoom_ratio: {max(hs) / max(min(hs), 1):.2f} "
|
||||
"(largest box / smallest box)")
|
||||
lines.append(f"crop region resizing: {max(ws)} x {max(hs)} -> "
|
||||
f"{output_size[0]} x {output_size[1]}")
|
||||
lines.append(f"movement: {travel:.0f}px "
|
||||
"(total distance the box center travels)")
|
||||
if len(v) and scale > 1.0:
|
||||
lines.append(f"padding_promised: {(scale - 1) / 2:.0%} of the "
|
||||
f"subject's size on each side (crop_scale {scale:.2f})")
|
||||
lines.append(f"padding_worst: {max(v.min(), 0):.0%} of promised")
|
||||
lines.append(f"padding_typical: {max(np.median(v), 0):.0%} of promised")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _plan_and_crop(original_images, masks, sel, p, divisible_by,
|
||||
mask_threshold, upscale_megapixels=0.0):
|
||||
if original_images.shape[0] != masks.shape[0]:
|
||||
raise ValueError(f"original_images ({original_images.shape[0]}) and masks ({masks.shape[0]}) must have the same frame count")
|
||||
if original_images.shape[1:3] != masks.shape[1:3]:
|
||||
raise ValueError(f"original_images ({original_images.shape[2]}x{original_images.shape[1]}) and masks ({masks.shape[2]}x{masks.shape[1]}) must have the same dimensions")
|
||||
|
||||
if 0.0 < p["crop_scale"] < 1.0:
|
||||
raise ValueError("crop_scale must be 0 (full frame) or at least 1.0")
|
||||
full_frame = p["crop_scale"] <= 0.0
|
||||
|
||||
binary = masks.detach().cpu().numpy() > mask_threshold
|
||||
if full_frame:
|
||||
img_h, img_w = masks.shape[1], masks.shape[2]
|
||||
boxes = [{"x": 0, "y": 0, "width": img_w, "height": img_h}
|
||||
for _ in range(masks.shape[0])]
|
||||
info = {"aspect": img_w / img_h}
|
||||
else:
|
||||
if not binary.any():
|
||||
raise ValueError("all masks are empty, nothing to crop")
|
||||
boxes, info = planner.plan(binary, sel, p, divisible_by)
|
||||
|
||||
bw = max(b["width"] for b in boxes)
|
||||
bh = max(b["height"] for b in boxes)
|
||||
size = None
|
||||
img_method, mask_method = "lanczos", "bilinear"
|
||||
if full_frame:
|
||||
gw = math.ceil(bw / divisible_by) * divisible_by
|
||||
gh = math.ceil(bh / divisible_by) * divisible_by
|
||||
if (gw, gh) != (bw, bh):
|
||||
size = (gw, gh)
|
||||
elif sel == "zoomed":
|
||||
# Target sized to the largest planned box: that stretch is ~1:1
|
||||
# and everything else upscales rather than losing detail.
|
||||
th = math.ceil(bh / divisible_by) * divisible_by
|
||||
size = (math.ceil(th * info["aspect"] / divisible_by) * divisible_by, th)
|
||||
if upscale_megapixels != 0:
|
||||
up = _upscale_size(*(size or (bw, bh)), upscale_megapixels, divisible_by)
|
||||
if up is not None:
|
||||
size = up
|
||||
img_method, mask_method = "bicubic", "nearest-exact"
|
||||
|
||||
debug = _debug_summary(binary, boxes, p["crop_scale"],
|
||||
f"{sel} (full frame)" if full_frame else sel,
|
||||
info, size or (bw, bh))
|
||||
|
||||
if size is not None:
|
||||
tw, th = size
|
||||
cropped_images = torch.stack([
|
||||
_resize_image(original_images[i, b["y"]:b["y"] + b["height"], b["x"]:b["x"] + b["width"], :], tw, th, img_method)
|
||||
for i, b in enumerate(boxes)
|
||||
])
|
||||
cropped_masks = torch.stack([
|
||||
_resize_mask(masks[i, b["y"]:b["y"] + b["height"], b["x"]:b["x"] + b["width"]], tw, th, mask_method)
|
||||
for i, b in enumerate(boxes)
|
||||
])
|
||||
else:
|
||||
cropped_images = torch.stack([
|
||||
original_images[i, b["y"]:b["y"] + b["height"], b["x"]:b["x"] + b["width"], :]
|
||||
for i, b in enumerate(boxes)
|
||||
])
|
||||
cropped_masks = torch.stack([
|
||||
masks[i, b["y"]:b["y"] + b["height"], b["x"]:b["x"] + b["width"]]
|
||||
for i, b in enumerate(boxes)
|
||||
])
|
||||
return (cropped_images, cropped_masks,
|
||||
[[b] for b in boxes], debug)
|
||||
@@ -0,0 +1,102 @@
|
||||
"""Color Mask -- recolors a plain white/binary mask (e.g. straight out of SAM2) into the
|
||||
colored-silhouette or flat-color-marker convention scail2v2 was trained on.
|
||||
|
||||
Two styles, matching the two different mask roles in the trained recipe:
|
||||
- "silhouette": recolor white->color, keep the full shape. Use for the GUIDE mask slot
|
||||
(LTX Multiple Controls' mask_video) -- its shape is what tells the model WHERE/WHAT POSE
|
||||
to place the identity, so the shape must be preserved.
|
||||
- "flat_marker": recolor AND collapse the whole mask down to a small solid-color dot at its
|
||||
centroid, discarding the shape entirely. Use for the IDENTITY mask slot (identity_mask_
|
||||
image) -- a full body-shaped silhouette there re-introduces a competing pose signal (this
|
||||
is the exact bug flatten_ref_masks.py fixes on the training-data side: the reference's OWN
|
||||
held pose bled into the output through a shaped identity mask). The color is the only
|
||||
thing that matters there (it's what teaches "this color <-> this identity" correspondence
|
||||
for multi-character replacement), not the shape.
|
||||
"""
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
_COLOR_PRESETS = {
|
||||
"blue": (0, 0, 255),
|
||||
"red": (255, 0, 0),
|
||||
"green": (0, 255, 0),
|
||||
"yellow": (255, 255, 0),
|
||||
"cyan": (0, 255, 255),
|
||||
"magenta": (255, 0, 255),
|
||||
"custom": None,
|
||||
}
|
||||
|
||||
|
||||
class LTXColorMask:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {
|
||||
"mask": ("IMAGE", {"tooltip": "Plain white/binary mask (e.g. from SAM2), one or more frames. "
|
||||
"Any non-black pixel counts as 'inside' the mask, regardless of exact shade."}),
|
||||
"color": (list(_COLOR_PRESETS.keys()), {"default": "blue",
|
||||
"tooltip": "Preset color to recolor the mask to (matches scail2v2's per-person palette: "
|
||||
"blue=person 0, red=person 1, x-sorted left-to-right). Pick 'custom' to use the "
|
||||
"custom_r/g/b inputs instead."}),
|
||||
"style": (["silhouette", "flat_marker"], {"default": "silhouette",
|
||||
"tooltip": "silhouette: recolor, keep the full shape -- use for the GUIDE mask slot "
|
||||
"(mask_video). flat_marker: recolor AND collapse to a small dot at the mask's "
|
||||
"centroid -- use for the IDENTITY mask slot (identity_mask_image); a shaped "
|
||||
"mask there re-injects a competing pose signal (the ref's own held pose)."}),
|
||||
"marker_radius_frac": ("FLOAT", {"default": 0.06, "min": 0.01, "max": 0.3, "step": 0.01,
|
||||
"tooltip": "flat_marker only: dot radius as a fraction of min(H, W). Matches the "
|
||||
"training-side default (flatten_ref_masks.py)."}),
|
||||
}, "optional": {
|
||||
"custom_r": ("INT", {"default": 255, "min": 0, "max": 255}),
|
||||
"custom_g": ("INT", {"default": 0, "min": 0, "max": 255}),
|
||||
"custom_b": ("INT", {"default": 0, "min": 0, "max": 255}),
|
||||
"background_black": ("BOOLEAN", {"default": True,
|
||||
"tooltip": "On: output background is black (matches guide_mask's white-bg-colored-"
|
||||
"shape convention actually being colored-on-black internally once VAE-"
|
||||
"encoded -- for identity_mask specifically, training used a black bg). "
|
||||
"Off: keep the mask's original background color where nothing was "
|
||||
"recolored (rare; only if your source mask isn't black/white)."}),
|
||||
}}
|
||||
|
||||
RETURN_TYPES = ("IMAGE",)
|
||||
RETURN_NAMES = ("colored_mask",)
|
||||
FUNCTION = "apply"
|
||||
CATEGORY = "LTX/identity"
|
||||
DESCRIPTION = ("Recolors a plain white/binary SAM2-style mask into scail2v2's trained convention. "
|
||||
"'silhouette' for the guide mask slot, 'flat_marker' (small color dot, no shape) for the "
|
||||
"identity mask slot -- see each input's tooltip for why the two differ.")
|
||||
|
||||
def apply(self, mask, color, style, marker_radius_frac,
|
||||
custom_r=255, custom_g=0, custom_b=0, background_black=True):
|
||||
if color == "custom":
|
||||
rgb = (int(custom_r), int(custom_g), int(custom_b))
|
||||
else:
|
||||
rgb = _COLOR_PRESETS[color]
|
||||
|
||||
arr = (mask.clamp(0, 1).cpu().numpy() * 255).astype(np.uint8) # [N,H,W,C]
|
||||
n, h, w = arr.shape[0], arr.shape[1], arr.shape[2]
|
||||
color_arr = np.array(rgb, dtype=np.uint8)
|
||||
out = np.zeros((n, h, w, 3), dtype=np.uint8)
|
||||
|
||||
for i in range(n):
|
||||
frame = arr[i]
|
||||
inside = frame[..., :3].sum(axis=-1) > 10 # non-black = inside the mask
|
||||
if not inside.any():
|
||||
continue
|
||||
if style == "silhouette":
|
||||
out[i][inside] = color_arr
|
||||
if not background_black:
|
||||
out[i][~inside] = frame[..., :3][~inside]
|
||||
else: # flat_marker
|
||||
ys, xs = np.where(inside)
|
||||
cy, cx = int(ys.mean()), int(xs.mean())
|
||||
radius = max(2, int(marker_radius_frac * min(h, w)))
|
||||
yy, xx = np.ogrid[:h, :w]
|
||||
dot = (yy - cy) ** 2 + (xx - cx) ** 2 <= radius ** 2
|
||||
out[i][dot] = color_arr
|
||||
|
||||
out_t = torch.from_numpy(out.astype(np.float32) / 255.0)
|
||||
return (out_t,)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"LTXColorMask": LTXColorMask}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"LTXColorMask": "LTX Color Mask"}
|
||||
@@ -0,0 +1,88 @@
|
||||
"""Corrige a saida do VLM para o formato que o LoRA foi treinado.
|
||||
|
||||
Duas regras que o VLM erra de forma consistente, por mais explicito que o prompt seja, e que
|
||||
sao deterministicas -- quem chama ja sabe se anexou referencia:
|
||||
|
||||
sem referencia -> a frase-marcador nao pode existir, e "trocar" e um Replace simples.
|
||||
O VLM tende a emitir Add + marcador + Remove, que e a forma da referencia.
|
||||
com referencia -> o marcador so pode seguir uma frase que comeca com Add. Se vier num
|
||||
Replace, vira Add + Remove.
|
||||
"""
|
||||
|
||||
import re
|
||||
|
||||
MARCADOR = "The object to add is shown in the reference image."
|
||||
TRIGGER = "edit_anything: "
|
||||
|
||||
|
||||
def _frases(texto):
|
||||
corpo = texto[len(TRIGGER):] if texto.startswith(TRIGGER) else texto
|
||||
return [f.strip() for f in re.split(r"(?<=\.)\s+", corpo.strip()) if f.strip()]
|
||||
|
||||
|
||||
def _juntar(frases):
|
||||
return TRIGGER + " ".join(frases)
|
||||
|
||||
|
||||
def corrigir(prompt, tem_referencia):
|
||||
frases = _frases(prompt)
|
||||
mudou = []
|
||||
|
||||
if not tem_referencia:
|
||||
antes = len(frases)
|
||||
frases = [f for f in frases if f.rstrip() != MARCADOR]
|
||||
if len(frases) != antes:
|
||||
mudou.append("marcador removido (sem referencia)")
|
||||
|
||||
# Add(X) ... Remove(Y) sem referencia = forma da referencia aplicada no lugar errado.
|
||||
# Recompoe como Replace Y with X.
|
||||
i_add = next((i for i, f in enumerate(frases) if f.startswith("Add ")), None)
|
||||
i_rem = next((i for i, f in enumerate(frases) if f.startswith("Remove ")), None)
|
||||
if i_add is not None and i_rem is not None:
|
||||
novo = re.sub(r"^Add\s+", "", frases[i_add]).rstrip(".")
|
||||
velho = re.sub(r"^Remove\s+", "", frases[i_rem]).rstrip(".")
|
||||
# descarta localizacao redundante herdada do Add
|
||||
novo = re.sub(r"\s+(where|parked|standing|placed)\b.*$", "", novo).strip()
|
||||
frases = [f for k, f in enumerate(frases) if k not in (i_add, i_rem)]
|
||||
frases.insert(min(i_add, i_rem), f"Replace {velho} with {novo}.")
|
||||
mudou.append("Add+Remove recomposto como Replace")
|
||||
else:
|
||||
for i, f in enumerate(frases):
|
||||
if f.rstrip() == MARCADOR and i > 0 and not frases[i - 1].startswith("Add "):
|
||||
anterior = frases[i - 1]
|
||||
m = re.match(r"^Replace\s+(.*?)\s+with\s+(.*)\.$", anterior)
|
||||
if m:
|
||||
velho, novo = m.group(1), m.group(2)
|
||||
frases[i - 1] = f"Add {novo}."
|
||||
frases.insert(i + 1, f"Remove {velho}.")
|
||||
mudou.append("Replace com marcador virou Add + Remove")
|
||||
break
|
||||
|
||||
return _juntar(frases), mudou
|
||||
|
||||
|
||||
class EditAnythingPromptFixer:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"prompt": ("STRING", {"multiline": True, "forceInput": True}),
|
||||
"has_reference": ("BOOLEAN", {"default": False, "tooltip": "Ligue quando uma imagem de referencia for anexada. Este e o mesmo booleano que decide anexar a imagem."}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("STRING",)
|
||||
RETURN_NAMES = ("prompt",)
|
||||
FUNCTION = "fix"
|
||||
CATEGORY = "BFS/EditAnything"
|
||||
|
||||
def fix(self, prompt, has_reference):
|
||||
saida, mudou = corrigir(prompt, has_reference)
|
||||
if mudou:
|
||||
print(f"[BFSNodes] prompt corrigido: {', '.join(mudou)}")
|
||||
print(f"[BFSNodes] {saida}")
|
||||
return (saida,)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"EditAnythingPromptFixer": EditAnythingPromptFixer}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"EditAnythingPromptFixer": "Edit Anything Prompt Fixer (BFS)"}
|
||||
@@ -0,0 +1,181 @@
|
||||
{
|
||||
"1": {
|
||||
"class_type": "UNETLoader",
|
||||
"inputs": {
|
||||
"unet_name": "minimax_h3_ref2va_pruned_int8_convrot.safetensors",
|
||||
"weight_dtype": "default"
|
||||
}
|
||||
},
|
||||
"2": {
|
||||
"class_type": "LoraLoaderModelOnly",
|
||||
"inputs": {
|
||||
"model": [
|
||||
"1",
|
||||
0
|
||||
],
|
||||
"lora_name": "YOUR_H3_UPSCALE_LORA.safetensors",
|
||||
"strength_model": 1.0
|
||||
}
|
||||
},
|
||||
"3": {
|
||||
"class_type": "CLIPLoader",
|
||||
"inputs": {
|
||||
"clip_name": "qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors",
|
||||
"type": "minimax",
|
||||
"device": "default"
|
||||
}
|
||||
},
|
||||
"4": {
|
||||
"class_type": "CLIPTextEncode",
|
||||
"inputs": {
|
||||
"clip": [
|
||||
"3",
|
||||
0
|
||||
],
|
||||
"text": "h3upscale, increase resolution and restore fine details while preserving the original content and visual style."
|
||||
}
|
||||
},
|
||||
"5": {
|
||||
"class_type": "VAELoader",
|
||||
"inputs": {
|
||||
"vae_name": "minimax_h3_video_vae_fp16.safetensors"
|
||||
}
|
||||
},
|
||||
"6": {
|
||||
"class_type": "BFSMiniMaxH3GuideTarget",
|
||||
"inputs": {
|
||||
"width": 1024,
|
||||
"height": 768,
|
||||
"frames": 1
|
||||
}
|
||||
},
|
||||
"7": {
|
||||
"class_type": "LoadImage",
|
||||
"inputs": {
|
||||
"image": "YOUR_INPUT_IMAGE.png"
|
||||
}
|
||||
},
|
||||
"8": {
|
||||
"class_type": "BFSMiniMaxH3DownscaledGuide",
|
||||
"inputs": {
|
||||
"model": [
|
||||
"2",
|
||||
0
|
||||
],
|
||||
"positive": [
|
||||
"4",
|
||||
0
|
||||
],
|
||||
"vae": [
|
||||
"5",
|
||||
0
|
||||
],
|
||||
"latent": [
|
||||
"6",
|
||||
0
|
||||
],
|
||||
"image": [
|
||||
"7",
|
||||
0
|
||||
],
|
||||
"downscale_factor": 4,
|
||||
"frame_idx": 0,
|
||||
"lora_name": "YOUR_H3_UPSCALE_LORA.safetensors"
|
||||
}
|
||||
},
|
||||
"9": {
|
||||
"class_type": "BasicGuider",
|
||||
"inputs": {
|
||||
"model": [
|
||||
"8",
|
||||
0
|
||||
],
|
||||
"conditioning": [
|
||||
"8",
|
||||
1
|
||||
]
|
||||
}
|
||||
},
|
||||
"10": {
|
||||
"class_type": "RandomNoise",
|
||||
"inputs": {
|
||||
"noise_seed": 42
|
||||
}
|
||||
},
|
||||
"11": {
|
||||
"class_type": "KSamplerSelect",
|
||||
"inputs": {
|
||||
"sampler_name": "euler"
|
||||
}
|
||||
},
|
||||
"12": {
|
||||
"class_type": "BasicScheduler",
|
||||
"inputs": {
|
||||
"model": [
|
||||
"8",
|
||||
0
|
||||
],
|
||||
"scheduler": "simple",
|
||||
"steps": 28,
|
||||
"denoise": 1.0
|
||||
}
|
||||
},
|
||||
"13": {
|
||||
"class_type": "SamplerCustomAdvanced",
|
||||
"inputs": {
|
||||
"noise": [
|
||||
"10",
|
||||
0
|
||||
],
|
||||
"guider": [
|
||||
"9",
|
||||
0
|
||||
],
|
||||
"sampler": [
|
||||
"11",
|
||||
0
|
||||
],
|
||||
"sigmas": [
|
||||
"12",
|
||||
0
|
||||
],
|
||||
"latent_image": [
|
||||
"6",
|
||||
0
|
||||
]
|
||||
}
|
||||
},
|
||||
"14": {
|
||||
"class_type": "VAEDecode",
|
||||
"inputs": {
|
||||
"samples": [
|
||||
"13",
|
||||
0
|
||||
],
|
||||
"vae": [
|
||||
"5",
|
||||
0
|
||||
]
|
||||
}
|
||||
},
|
||||
"15": {
|
||||
"class_type": "SaveImage",
|
||||
"inputs": {
|
||||
"images": [
|
||||
"14",
|
||||
0
|
||||
],
|
||||
"filename_prefix": "BFS_H3_Upscale"
|
||||
}
|
||||
},
|
||||
"16": {
|
||||
"class_type": "SaveImage",
|
||||
"inputs": {
|
||||
"images": [
|
||||
"8",
|
||||
2
|
||||
],
|
||||
"filename_prefix": "BFS_H3_EffectiveGuide"
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,128 @@
|
||||
"""LTX Identity CAN (AdaLN) — apply the trained Conditioned Adaptive Normalization at inference.
|
||||
|
||||
The ArcFace projector (appended text tokens) has ~0 impact (cross-attn can ignore appended tokens).
|
||||
CAN instead modulates the self-attention AdaLN (shift + gate) of the even blocks with the reference
|
||||
ArcFace — a channel the model can't ignore. Trained on top of the reference LoRA it lifts identity
|
||||
a lot (humo 0.52 -> 0.69) with no first-frame leak.
|
||||
|
||||
Key trick: the CAN delta depends only on the (constant) reference ArcFace, and the block AdaLN is
|
||||
`scale_shift_table[row] + timestep`. So instead of patching the (version-specific, AV) block
|
||||
forward, we just add the delta into `scale_shift_table` (row 0 = shift_msa, row 2 = gate_msa) via
|
||||
ComfyUI's reversible add_object_patch — mathematically identical, works on any LTX/LTXAV version.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from torch import nn
|
||||
from safetensors.torch import load_file
|
||||
|
||||
|
||||
class CANModulation(nn.Module):
|
||||
"""Exact replica of ltx-core CANModulation: id_global[512] -> (dshift, dgate)[dim]."""
|
||||
|
||||
def __init__(self, id_dim: int, dim: int, hidden: int = 512):
|
||||
super().__init__()
|
||||
self.norm = nn.LayerNorm(id_dim)
|
||||
self.mlp = nn.Sequential(nn.Linear(id_dim, hidden), nn.SiLU(), nn.Linear(hidden, 3 * dim))
|
||||
self.dim = dim
|
||||
|
||||
def forward(self, id_global):
|
||||
d = self.mlp(self.norm(id_global))
|
||||
dshift, _dscale, dgate = d.chunk(3, dim=-1)
|
||||
return dshift, dgate
|
||||
|
||||
|
||||
def _find_diffusion_model(mp):
|
||||
m = getattr(mp, "model", mp)
|
||||
return getattr(m, "diffusion_model", m)
|
||||
|
||||
|
||||
def apply_can_to_model(model_patcher, reference_image, can_weights_name, strength=1.0,
|
||||
arcface_mode="auto_adjust"):
|
||||
"""Add the trained CAN identity modulation to a ModelPatcher (in place, reversible).
|
||||
Returns the number of blocks modulated."""
|
||||
import folder_paths
|
||||
from .ltx_identity_overlap import _arcface_embed
|
||||
|
||||
path = folder_paths.get_full_path("loras", can_weights_name)
|
||||
sd = load_file(path)
|
||||
idxs = sorted({int(k.split(".")[1]) for k in sd if k.startswith("can.")})
|
||||
if not idxs:
|
||||
print("[BFS CAN] no can.* weights in file — nothing to apply.")
|
||||
return 0
|
||||
|
||||
face = _arcface_embed(reference_image, arcface_mode) if _arcface_takes_mode() else _arcface_embed(reference_image)
|
||||
if face is None:
|
||||
print("[BFS CAN] no face detected in reference — CAN skipped.")
|
||||
return 0
|
||||
id_global = torch.as_tensor(np.asarray(face), dtype=torch.float32).view(1, -1) # [1,512]
|
||||
|
||||
dm = _find_diffusion_model(model_patcher)
|
||||
blocks = list(dm.transformer_blocks)
|
||||
even_blocks = blocks[::2] # CAN was trained on even blocks (0,2,4,…) -> saved can.0,can.1,…
|
||||
n = min(len(idxs), len(even_blocks))
|
||||
applied = 0
|
||||
for j in range(n):
|
||||
# find this even block's index in the full list (for the dotted patch name)
|
||||
blk = even_blocks[j]
|
||||
block_index = j * 2
|
||||
sst = blk.scale_shift_table
|
||||
if sst.shape[0] < 3:
|
||||
continue
|
||||
dim = sst.shape[1]
|
||||
dev = sst.device
|
||||
can = CANModulation(id_dim=id_global.shape[1], dim=dim)
|
||||
can.load_state_dict({k[len(f"can.{idxs[j]}."):]: v for k, v in sd.items()
|
||||
if k.startswith(f"can.{idxs[j]}.")})
|
||||
can = can.to(dev, torch.float32).eval()
|
||||
with torch.no_grad():
|
||||
dshift, dgate = can(id_global.to(dev, torch.float32)) # [1, dim], on the model device
|
||||
new_sst = sst.detach().clone().float()
|
||||
new_sst[0] = new_sst[0] + strength * dshift[0] # shift_msa
|
||||
new_sst[2] = new_sst[2] + strength * torch.tanh(dgate)[0] # gate_msa
|
||||
new_param = nn.Parameter(new_sst.to(sst.dtype), requires_grad=False)
|
||||
name = f"diffusion_model.transformer_blocks.{block_index}.scale_shift_table"
|
||||
model_patcher.add_object_patch(name, new_param)
|
||||
applied += 1
|
||||
print(f"[BFS CAN] applied CAN (AdaLN) to {applied} even blocks (strength {strength}).")
|
||||
return applied
|
||||
|
||||
|
||||
def _arcface_takes_mode():
|
||||
import inspect
|
||||
from .ltx_identity_overlap import _arcface_embed
|
||||
try:
|
||||
return "mode" in inspect.signature(_arcface_embed).parameters
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
class LTXIdentityCAN:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
import folder_paths
|
||||
adapters = ["None"] + [f for f in folder_paths.get_filename_list("loras") if "identity_adapters" in f]
|
||||
return {
|
||||
"required": {
|
||||
"model": ("MODEL",),
|
||||
"reference_image": ("IMAGE",),
|
||||
"can_weights": (adapters,),
|
||||
"strength": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 2.0, "step": 0.05}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("MODEL",)
|
||||
FUNCTION = "apply"
|
||||
CATEGORY = "BFS/LTX Identity"
|
||||
|
||||
def apply(self, model, reference_image, can_weights, strength):
|
||||
if can_weights == "None":
|
||||
return (model,)
|
||||
m = model.clone()
|
||||
apply_can_to_model(m, reference_image, can_weights, strength)
|
||||
return (m,)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"LTXIdentityCAN": LTXIdentityCAN}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"LTXIdentityCAN": "LTX Identity CAN / AdaLN"}
|
||||
@@ -0,0 +1,48 @@
|
||||
"""LTX Identity — Gemma-Vision Conditioning (MagicMirror-Caminho2).
|
||||
|
||||
The identity breakthrough for LTX: the model conditions in Gemma's embedding space, so foreign
|
||||
encoders (CLIP/DINO/ArcFace adapters) fail. Instead we feed the REFERENCE IMAGE through LTX's OWN
|
||||
multimodal Gemma 3 (its text encoder is Gemma 3 with a vision tower), so the reference becomes
|
||||
conditioning tokens IN THE NATIVE SPACE — the model reads the person like text. No reference
|
||||
latent (no copy-paste/mask), no foreign encoder, no DiT surgery. Pair with a LoRA trained the
|
||||
same way (reference image -> Gemma-vision conditioning + ArcFace loss).
|
||||
|
||||
Everything comes from the CLIP already loaded in the workflow: ComfyUI's native LTX Gemma-3
|
||||
tokenizer already accepts an image (clip.tokenize(prompt, image=...) inserts <image_soft_token>
|
||||
and carries the pixels), and clip.encode_from_tokens runs the multimodal Gemma + projection +
|
||||
connector. So this node is just: tokenize(prompt, image) -> encode -> CONDITIONING.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
class LTXIdentityGemmaVision:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"clip": ("CLIP",),
|
||||
"reference_image": ("IMAGE",),
|
||||
"prompt": ("STRING", {"multiline": True, "default": "ref_t2v: a person in a room, medium shot."}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("CONDITIONING",)
|
||||
RETURN_NAMES = ("conditioning",)
|
||||
FUNCTION = "encode"
|
||||
CATEGORY = "BFS/LTX Identity"
|
||||
|
||||
def encode(self, clip, reference_image, prompt):
|
||||
# The LTX Gemma-3 tokenizer takes the ComfyUI IMAGE directly (movedim + <image_soft_token>),
|
||||
# and encode runs the multimodal Gemma. Identity ends up in the native conditioning space.
|
||||
try:
|
||||
tokens = clip.tokenize(prompt, image=reference_image)
|
||||
except TypeError:
|
||||
# Older signatures may name it differently; fall back to positional image.
|
||||
tokens = clip.tokenize(prompt, reference_image)
|
||||
cond = clip.encode_from_tokens_scheduled(tokens)
|
||||
print("[BFS Gemma-Vision] multimodal identity CONDITIONING ready (reference in native space).")
|
||||
return (cond,)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"LTXIdentityGemmaVision": LTXIdentityGemmaVision}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"LTXIdentityGemmaVision": "LTX Identity Gemma-Vision (Caminho2)"}
|
||||
@@ -0,0 +1,393 @@
|
||||
"""LTX-2.3 Identity Transfer — MULTIPLE ANGLES (temporary tester).
|
||||
|
||||
A copy of "LTX Identity Transfer" (overlap + source_phase, ltx_identity_overlap.py) that accepts
|
||||
several reference views, each injected as its OWN overlap segment tagged with its OWN source_id —
|
||||
exactly the multi-view training convention:
|
||||
|
||||
reference_face_front -> source_id 2 (REQUIRED; also drives the ArcFace projector)
|
||||
reference_back_head -> source_id 3 (optional — back of the head / nape, hair)
|
||||
reference_body_front -> source_id 4 (optional — full-body front, build/proportions)
|
||||
reference_side_profile -> source_id 5 (optional — side / profile of the face)
|
||||
|
||||
Empty optional inputs are simply omitted — matching the training view-dropout, so the model
|
||||
degrades gracefully when you only provide the face. Non-frontal views feed only the overlap
|
||||
latent (no ArcFace); the frontal face additionally drives the IdentityProjector tokens.
|
||||
|
||||
Load the matching multi-view LoRA on MODEL first. Reuses the exact ArcFace/projector/patch
|
||||
machinery from ltx_identity_overlap; only the model patch is generalized to N reference blocks.
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import types
|
||||
|
||||
import torch
|
||||
|
||||
from .ltx_identity_overlap import (
|
||||
_append_ctx_tokens,
|
||||
_arcface_embed,
|
||||
_find_ltxv,
|
||||
_load_projector,
|
||||
)
|
||||
from .reference_temporal_offset import (
|
||||
DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
|
||||
reference_temporal_offset_input,
|
||||
reference_temporal_pixel_offset,
|
||||
shift_reference_temporal_positions,
|
||||
)
|
||||
|
||||
log = logging.getLogger("LTXIdentityMultiAngle")
|
||||
|
||||
# Fixed viewpoint -> source_id convention (must match the multi-view training config).
|
||||
_SLOT_SOURCE_ID = {"face": 2.0, "back": 3.0, "body": 4.0, "side": 5.0}
|
||||
_DEBUG = os.environ.get("LTX_IDOVERLAP_DEBUG", "0") == "1"
|
||||
|
||||
|
||||
def _dbg(*a):
|
||||
if _DEBUG:
|
||||
print("[LTXIdMulti] " + " ".join(str(x) for x in a), flush=True)
|
||||
|
||||
|
||||
def _rotate_block(pe, start, length, seg, theta=10000.0):
|
||||
"""Rotate RoPE freqs of tokens [start:start+length] by phase = seg * theta^(-d/L).
|
||||
Same math as ltx_identity_overlap._rotate_ref_freqs, but on an arbitrary token slice
|
||||
(each reference block sits at its own offset after the target)."""
|
||||
if length <= 0 or seg == 0.0:
|
||||
return pe
|
||||
cos, sin = pe[0], pe[1]
|
||||
rest = tuple(pe[2:])
|
||||
L = cos.shape[-1]
|
||||
d = torch.arange(L, device=cos.device, dtype=torch.float32)
|
||||
rate = theta ** (-d / float(L))
|
||||
phase = seg * rate
|
||||
pc = phase.cos().to(cos.dtype)
|
||||
ps = phase.sin().to(sin.dtype)
|
||||
idx = [slice(None)] * cos.dim()
|
||||
idx[-2] = slice(start, start + length)
|
||||
idx = tuple(idx)
|
||||
c0, s0 = cos[idx], sin[idx]
|
||||
cos = cos.clone(); sin = sin.clone()
|
||||
cos[idx] = c0 * pc - s0 * ps
|
||||
sin[idx] = s0 * pc + c0 * ps
|
||||
_dbg("rotate block start", start, "len", length, "seg", seg)
|
||||
return (cos, sin, *rest)
|
||||
|
||||
|
||||
def _install_multi_patches(ltxv):
|
||||
"""Idempotent multi-reference version of the overlap patch. Appends a LIST of reference
|
||||
latents (each with its own source_phase segment) after the target tokens, and trims them
|
||||
all before unpatchify. Uses distinct attribute/flag names so it never collides with the
|
||||
single-ref LTXIdentityTransfer patch."""
|
||||
if getattr(ltxv, "_id_multi_patched", False):
|
||||
return
|
||||
orig_process_input = ltxv._process_input
|
||||
orig_prepare_ts = ltxv._prepare_timestep
|
||||
orig_prepare_pe = ltxv._prepare_positional_embeddings
|
||||
orig_process_output = ltxv._process_output
|
||||
|
||||
def process_input(self, x, keyframe_idxs, denoise_mask, **kw):
|
||||
# Reset per-forward state FIRST so stale values from a previous run can never leak
|
||||
# into this forward (e.g. if the ref latents stop reaching us after a Comfy update).
|
||||
self._idma_ref_len = 0
|
||||
self._idma_blocks = []
|
||||
self._idma_target_len = None
|
||||
self._idma_ppf = None
|
||||
out = orig_process_input(x, keyframe_idxs, denoise_mask, **kw)
|
||||
# Comfy versions differ on whether custom transformer_options keys arrive flattened
|
||||
# in kwargs or nested under kwargs["transformer_options"] — accept both.
|
||||
ref_lats = kw.get("_id_ref_latents")
|
||||
if ref_lats is None:
|
||||
ref_lats = (kw.get("transformer_options") or {}).get("_id_ref_latents")
|
||||
if not ref_lats:
|
||||
_dbg("process_input: NO ref latents in kwargs (flat or nested) -> vanilla forward. kw keys:", list(kw.keys()))
|
||||
return out
|
||||
from comfy.ldm.lightricks.model import latent_to_pixel_coords
|
||||
xx, pix, add = out
|
||||
is_av = isinstance(xx, (list, tuple))
|
||||
vx = xx[0] if is_av else xx
|
||||
vco = pix[0] if is_av else pix
|
||||
target_len = vx.shape[1]
|
||||
self._idma_target_len = target_len
|
||||
segs = getattr(self, "_idma_segs", [2.0] * len(ref_lats))
|
||||
temporal_offset_latents = getattr(
|
||||
self,
|
||||
"_idma_temporal_offset_latents",
|
||||
0,
|
||||
)
|
||||
blocks = []
|
||||
offset = target_len
|
||||
for ref_lat, seg in zip(ref_lats, segs):
|
||||
rt, rlc = self.patchifier.patchify(ref_lat.to(dtype=vx.dtype, device=vx.device))
|
||||
rpc = latent_to_pixel_coords(latent_coords=rlc, scale_factors=self.vae_scale_factors,
|
||||
causal_fix=self.causal_temporal_positioning)
|
||||
rpc = shift_reference_temporal_positions(
|
||||
rpc,
|
||||
temporal_offset_latents,
|
||||
self.vae_scale_factors[0],
|
||||
)
|
||||
rt = self.patchify_proj(rt)
|
||||
if rt.shape[0] != vx.shape[0]:
|
||||
rt = rt.expand(vx.shape[0], -1, -1)
|
||||
if rpc.shape[0] != vco.shape[0]:
|
||||
rpc = rpc.expand(vco.shape[0], *([-1] * (rpc.dim() - 1)))
|
||||
rlen = rt.shape[1]
|
||||
vx = torch.cat([vx, rt], dim=1)
|
||||
vco = torch.cat([vco, rpc.to(vco)], dim=2)
|
||||
blocks.append((offset, rlen, float(seg)))
|
||||
offset += rlen
|
||||
self._idma_ref_len = offset - target_len
|
||||
self._idma_blocks = blocks
|
||||
# each ref is one latent frame at the target grid -> its token count == tokens per frame
|
||||
self._idma_ppf = blocks[0][1] if blocks else None
|
||||
add = dict(add); add["_id_ref_len"] = self._idma_ref_len
|
||||
_dbg("process_input OUT: blocks", blocks, "| target_len", target_len, "| total_ref", self._idma_ref_len)
|
||||
if is_av:
|
||||
xx = [vx, xx[1]]; pix = [vco, pix[1]]
|
||||
else:
|
||||
xx, pix = vx, vco
|
||||
return xx, pix, add
|
||||
|
||||
def prepare_timestep(self, timestep, batch_size, hidden_dtype, **kw):
|
||||
# MEASURE-based extension: only pad the timestep by exactly what process_input
|
||||
# actually appended in THIS forward, and detect the timestep granularity from its
|
||||
# real length (per-token / per-frame / already-extended) instead of assuming one.
|
||||
# This keeps modulation and vx in lockstep across ComfyUI versions.
|
||||
ref_len = getattr(self, "_idma_ref_len", 0)
|
||||
tgt_len = getattr(self, "_idma_target_len", None)
|
||||
if ref_len and tgt_len:
|
||||
ppf = getattr(self, "_idma_ppf", None)
|
||||
if timestep.dim() <= 1:
|
||||
timestep = timestep.reshape(-1, 1).expand(batch_size, tgt_len).contiguous()
|
||||
cur = timestep.shape[1]
|
||||
# With audio connected, LTXAV may produce a combined video+audio per-token
|
||||
# timestep (e.g. 24576 = video 6144 + audio 18432). Trim to video-only so
|
||||
# the extension matches vx (which only has video + ref tokens).
|
||||
if cur > tgt_len + ref_len:
|
||||
_dbg("prepare_timestep: oversized", cur, "(tgt", tgt_len, "+ ref", ref_len,
|
||||
") -> trimming to video-only", tgt_len)
|
||||
timestep = timestep[:, :tgt_len]
|
||||
cur = tgt_len
|
||||
if cur == tgt_len: # per-token, target only -> append per-token zeros
|
||||
z = torch.zeros(timestep.shape[0], ref_len, *timestep.shape[2:],
|
||||
device=timestep.device, dtype=timestep.dtype)
|
||||
timestep = torch.cat([timestep, z], dim=1)
|
||||
_dbg("prepare_timestep: per-token extend", cur, "->", timestep.shape[1])
|
||||
elif ppf and cur * ppf == tgt_len: # per-FRAME timestep -> append ref frames
|
||||
n_ref_frames = max(1, ref_len // ppf)
|
||||
z = torch.zeros(timestep.shape[0], n_ref_frames, *timestep.shape[2:],
|
||||
device=timestep.device, dtype=timestep.dtype)
|
||||
timestep = torch.cat([timestep, z], dim=1)
|
||||
_dbg("prepare_timestep: per-frame extend", cur, "->", timestep.shape[1])
|
||||
elif cur == tgt_len + ref_len: # already extended (double-patch guard)
|
||||
_dbg("prepare_timestep: already extended", cur)
|
||||
else:
|
||||
_dbg("prepare_timestep: UNEXPECTED len", cur, "target", tgt_len, "ppf", ppf, "-> untouched")
|
||||
# Guide nodes (IC-LoRA Guide / Director Guide, etc.) inject a grid_mask that
|
||||
# FILTERS x tokens in _process_input (x = x[:, grid_mask]) and later indexes the
|
||||
# timestep (timestep[:, grid_mask]). Our ref tokens are appended AFTER that
|
||||
# filter, so extend the mask with True for them (False would drop our clean
|
||||
# timesteps from the modulation and desync it from vx). Pad size is measured
|
||||
# from the actual gap so per-token AND per-frame masks both work.
|
||||
gm = kw.get("grid_mask")
|
||||
if gm is not None and hasattr(gm, "shape") and timestep.dim() >= 2:
|
||||
gap = timestep.shape[1] - gm.shape[-1]
|
||||
if 0 < gap <= ref_len:
|
||||
pad = torch.ones(*gm.shape[:-1], gap, dtype=gm.dtype, device=gm.device)
|
||||
kw = dict(kw); kw["grid_mask"] = torch.cat([gm, pad], dim=-1)
|
||||
_dbg("prepare_timestep: grid_mask extended by", gap)
|
||||
return orig_prepare_ts(timestep, batch_size, hidden_dtype, **kw)
|
||||
|
||||
def prepare_pe(self, pixel_coords, frame_rate, x_dtype):
|
||||
pe = orig_prepare_pe(pixel_coords, frame_rate, x_dtype)
|
||||
blocks = getattr(self, "_idma_blocks", [])
|
||||
if not blocks:
|
||||
return pe
|
||||
theta = getattr(self, "_id_rope_theta", 10000.0)
|
||||
|
||||
def rot(v_pe):
|
||||
for (start, length, seg) in blocks:
|
||||
v_pe = _rotate_block(v_pe, start, length, seg, theta)
|
||||
return v_pe
|
||||
|
||||
if isinstance(pe, list) and len(pe) and isinstance(pe[0], (list, tuple)) and isinstance(pe[0][0], (list, tuple)):
|
||||
v_pe, cross_v = pe[0][0], pe[0][1]
|
||||
v_pe = rot(v_pe)
|
||||
pe = [(v_pe, cross_v), pe[1]]
|
||||
else:
|
||||
pe = rot(pe)
|
||||
return pe
|
||||
|
||||
def process_output(self, x, embedded_timestep, keyframe_idxs, **kw):
|
||||
ref_len = getattr(self, "_idma_ref_len", 0)
|
||||
if ref_len:
|
||||
_dbg("process_output: trimming", ref_len, "ref tokens")
|
||||
if ref_len:
|
||||
import copy
|
||||
|
||||
from comfy.ldm.lightricks.av_model import CompressedTimestep
|
||||
if isinstance(x, (list, tuple)):
|
||||
x = [x[0][:, :x[0].shape[1] - ref_len], *x[1:]]
|
||||
et_list = list(embedded_timestep) if isinstance(embedded_timestep, (list, tuple)) else [embedded_timestep]
|
||||
v_et = et_list[0]
|
||||
if isinstance(v_et, CompressedTimestep):
|
||||
ppf = max(1, getattr(v_et, "patches_per_frame", 1) or 1)
|
||||
n_ref_frames = max(1, ref_len // ppf)
|
||||
v_et2 = copy.copy(v_et)
|
||||
v_et2.data = v_et.data[:, : v_et.num_frames - n_ref_frames].contiguous()
|
||||
v_et2.num_frames = v_et.num_frames - n_ref_frames
|
||||
et_list[0] = v_et2
|
||||
elif hasattr(v_et, "shape") and v_et.dim() >= 2 and v_et.shape[1] > 1:
|
||||
et_list[0] = v_et[:, : v_et.shape[1] - ref_len]
|
||||
embedded_timestep = et_list
|
||||
else:
|
||||
x = x[:, :x.shape[1] - ref_len]
|
||||
if hasattr(embedded_timestep, "shape") and embedded_timestep.dim() >= 2 and embedded_timestep.shape[1] > 1:
|
||||
embedded_timestep = embedded_timestep[:, : embedded_timestep.shape[1] - ref_len]
|
||||
return orig_process_output(x, embedded_timestep, keyframe_idxs, **kw)
|
||||
|
||||
ltxv._process_input = types.MethodType(process_input, ltxv)
|
||||
ltxv._prepare_timestep = types.MethodType(prepare_timestep, ltxv)
|
||||
ltxv._prepare_positional_embeddings = types.MethodType(prepare_pe, ltxv)
|
||||
ltxv._process_output = types.MethodType(process_output, ltxv)
|
||||
ltxv._id_multi_patched = True
|
||||
log.info("LTXIdentityMultiAngle patches installed on %s", type(ltxv).__name__)
|
||||
|
||||
|
||||
class LTXIdentityMultiAngle:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
try:
|
||||
import folder_paths
|
||||
proj_choices = ["None"] + folder_paths.get_filename_list("loras")
|
||||
except Exception:
|
||||
proj_choices = ["None"]
|
||||
return {
|
||||
"required": {
|
||||
"model": ("MODEL",),
|
||||
"positive": ("CONDITIONING",),
|
||||
"negative": ("CONDITIONING",),
|
||||
"vae": ("VAE",),
|
||||
"latent": ("LATENT",),
|
||||
"reference_face_front": ("IMAGE", {
|
||||
"tooltip": "REQUIRED — frontal face (source_id 2). Also drives the ArcFace projector."}),
|
||||
"identity_projector": (proj_choices, {"default": "None",
|
||||
"tooltip": "ArcFace projector .safetensors (from models/loras). 'None' = overlap only."}),
|
||||
"phase_scale": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.1,
|
||||
"tooltip": "Global RoPE phase scale (training used 1.0). Applied to every view's source_id."}),
|
||||
"id_strength": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 50.0, "step": 0.5,
|
||||
"tooltip": "Multiplies the ArcFace projector tokens (frontal face only)."}),
|
||||
"arcface_mode": (["auto_adjust", "as_is", "disable"], {"default": "auto_adjust"}),
|
||||
"debug_log": ("BOOLEAN", {"default": False}),
|
||||
},
|
||||
"optional": {
|
||||
"reference_back_head": ("IMAGE", {
|
||||
"tooltip": "OPTIONAL — back of the head / nape, hair from behind (source_id 3). Leave empty to skip."}),
|
||||
"reference_body_front": ("IMAGE", {
|
||||
"tooltip": "OPTIONAL — full-body front, build/proportions (source_id 4). Leave empty to skip."}),
|
||||
"reference_side_profile": ("IMAGE", {
|
||||
"tooltip": "OPTIONAL — side / profile of the face (source_id 5). Leave empty to skip."}),
|
||||
# Appended last so saved workflows keep every historical widget index stable.
|
||||
"reference_temporal_offset_latents": reference_temporal_offset_input(),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("MODEL", "CONDITIONING", "CONDITIONING", "LATENT", "STRING")
|
||||
RETURN_NAMES = ("model", "positive", "negative", "latent", "debug")
|
||||
FUNCTION = "apply"
|
||||
CATEGORY = "LTX/identity"
|
||||
DESCRIPTION = ("Multi-angle identity transfer: frontal face (required) + optional back/body/side "
|
||||
"reference views, each injected as its own overlap+source_phase segment (source_id "
|
||||
"2/3/4/5), matching the multi-view training. Load the multi-view LoRA on MODEL first.")
|
||||
|
||||
def apply(self, model, positive, negative, vae, latent, reference_face_front,
|
||||
reference_temporal_offset_latents=DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
|
||||
identity_projector="None", phase_scale=1.0, id_strength=1.0,
|
||||
arcface_mode="auto_adjust", debug_log=False,
|
||||
reference_back_head=None, reference_body_front=None, reference_side_profile=None):
|
||||
import comfy.utils
|
||||
|
||||
global _DEBUG
|
||||
_DEBUG = bool(debug_log)
|
||||
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
||||
m = model.clone()
|
||||
ltxv = _find_ltxv(m)
|
||||
|
||||
_, w_sf, h_sf = vae.downscale_index_formula
|
||||
_, _, _, lat_h, lat_w = latent["samples"].shape
|
||||
px_w, px_h = lat_w * w_sf, lat_h * h_sf
|
||||
|
||||
def encode(img):
|
||||
ref_px = comfy.utils.common_upscale(img.movedim(-1, 1), px_w, px_h, "bilinear", "center").movedim(1, -1)[:1, :, :, :3]
|
||||
return vae.encode(ref_px)
|
||||
|
||||
# Build the provided views in a fixed order (face first). Empty optionals are skipped.
|
||||
provided = [("face", reference_face_front)]
|
||||
if reference_back_head is not None:
|
||||
provided.append(("back", reference_back_head))
|
||||
if reference_body_front is not None:
|
||||
provided.append(("body", reference_body_front))
|
||||
if reference_side_profile is not None:
|
||||
provided.append(("side", reference_side_profile))
|
||||
|
||||
ref_lats, segs, used = [], [], []
|
||||
for slot, img in provided:
|
||||
ref_lats.append(encode(img))
|
||||
segs.append(_SLOT_SOURCE_ID[slot] * float(phase_scale))
|
||||
used.append(f"{slot}(sid={_SLOT_SOURCE_ID[slot]:.0f})")
|
||||
|
||||
_install_multi_patches(ltxv)
|
||||
ltxv._idma_segs = segs
|
||||
ltxv._idma_temporal_offset_latents = int(reference_temporal_offset_latents)
|
||||
ltxv._id_rope_theta = 10000.0
|
||||
m.model_options = dict(m.model_options)
|
||||
to = dict(m.model_options.get("transformer_options", {}))
|
||||
to["_id_ref_latents"] = ref_lats
|
||||
m.model_options["transformer_options"] = to
|
||||
|
||||
# ArcFace projector tokens from the FRONTAL face only (optional).
|
||||
use_projector = identity_projector not in (None, "", "None")
|
||||
emb = _arcface_embed(reference_face_front, mode=arcface_mode) if use_projector else None
|
||||
if not use_projector:
|
||||
arc_status = "no projector (overlap only)"
|
||||
elif arcface_mode == "disable":
|
||||
arc_status = "disabled"
|
||||
else:
|
||||
arc_status = "OK" if emb is not None else "NO FACE -> skipped"
|
||||
if emb is not None:
|
||||
path = identity_projector
|
||||
if not os.path.isabs(path):
|
||||
try:
|
||||
import folder_paths
|
||||
resolved = folder_paths.get_full_path("loras", identity_projector)
|
||||
if resolved:
|
||||
path = resolved
|
||||
except Exception:
|
||||
pass
|
||||
if not os.path.isabs(path) or not os.path.exists(path):
|
||||
for b in ("models/loras", "models", "."):
|
||||
if os.path.exists(os.path.join(b, identity_projector)):
|
||||
path = os.path.join(b, identity_projector); break
|
||||
projector = _load_projector(path, device)
|
||||
emb = emb.to(device=device, dtype=torch.float32).unsqueeze(0)
|
||||
with torch.no_grad():
|
||||
id_tok = projector(emb) * float(id_strength)
|
||||
unc = projector(torch.zeros(1, projector.in_dim, device=device))
|
||||
positive = _append_ctx_tokens(positive, id_tok)
|
||||
negative = _append_ctx_tokens(negative, unc)
|
||||
|
||||
dbg = (
|
||||
"=== LTX Identity Transfer — MULTIPLE ANGLES ===\n"
|
||||
f"reference views ({len(ref_lats)}): {', '.join(used)}\n"
|
||||
f"ref latents: {[list(r.shape) for r in ref_lats]} | phase_scale={phase_scale}\n"
|
||||
f"reference temporal offset: {int(reference_temporal_offset_latents)} latent step(s) "
|
||||
f"= {reference_temporal_pixel_offset(reference_temporal_offset_latents, vae.downscale_index_formula[0])} "
|
||||
"pixel frame(s) (all reference views only)\n"
|
||||
f"arcface (frontal): {arc_status} (mode={arcface_mode}) | id_strength={id_strength}\n"
|
||||
f"patches on {type(ltxv).__name__}: process_input/prepare_timestep/prepare_pe/process_output (multi-block)\n"
|
||||
"Empty optional inputs are omitted (matches training view-dropout). Connect negative + CFG 3-5, no LightX2V."
|
||||
)
|
||||
log.info("\n" + dbg)
|
||||
return (m, positive, negative, latent, dbg)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"LTXIdentityMultiAngle": LTXIdentityMultiAngle}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"LTXIdentityMultiAngle": "LTX Identity Transfer (Multiple Angles)"}
|
||||
@@ -0,0 +1,912 @@
|
||||
"""LTX-2.3/2.5 Identity OVERLAP — replica of ltx-trainer's overlap+source_phase reference.
|
||||
|
||||
Unlike the append_keyframe path (which makes the ref an I2V first-frame), this injects the
|
||||
reference latent as SEPARATE tokens using the trainer's overlap/source-phase contract. The
|
||||
reference-only temporal offset can move those positions before frame zero to reduce learned
|
||||
frame-0 composition leakage; set it to 0 for the exact trainer overlap grid. The ref tokens
|
||||
remain clean (timestep 0), attend to the target in self-attention, and are sliced off the
|
||||
output (never rendered).
|
||||
|
||||
Model patch (installed idempotently on the LTX av_model), matching LTXBaseModel._forward:
|
||||
1) _process_input : append ref video tokens (patchified ref latent) with overlap positions;
|
||||
2) _prepare_positional_embeddings : rotate the ref tokens' RoPE freqs by source_phase;
|
||||
3) _prepare_timestep : give the ref tokens clean (0) timestep;
|
||||
4) _process_output : trim the ref tokens before unpatchify.
|
||||
Plus the ArcFace IdentityProjector tokens on the text context (as trained).
|
||||
|
||||
Load the matching LoRA on MODEL first. Requires insightface + buffalo_l (ArcFace, CPU).
|
||||
Set LTX_IDOVERLAP_DEBUG=1 to log shapes at each patch point for iteration.
|
||||
"""
|
||||
import functools
|
||||
import logging
|
||||
import os
|
||||
import types
|
||||
import weakref
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import torch.nn.functional as F
|
||||
from safetensors.torch import load_file
|
||||
|
||||
from .reference_temporal_offset import (
|
||||
DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
|
||||
reference_temporal_offset_input,
|
||||
reference_temporal_pixel_offset,
|
||||
shift_reference_temporal_positions,
|
||||
)
|
||||
|
||||
log = logging.getLogger("LTXIdentityOverlap")
|
||||
_USE_GPU = os.environ.get("LTX_IDPROJ_ARCFACE_GPU", "0") == "1"
|
||||
_FACE_APP = None
|
||||
|
||||
|
||||
def _shape(x):
|
||||
"""Readable shape/structure of tensors, lists, tuples, CompressedTimestep, etc."""
|
||||
try:
|
||||
if hasattr(x, "shape"):
|
||||
return f"T{tuple(x.shape)}"
|
||||
if isinstance(x, (list, tuple)):
|
||||
return f"{type(x).__name__}[{', '.join(_shape(i) for i in x)}]"
|
||||
return type(x).__name__
|
||||
except Exception:
|
||||
return "?"
|
||||
|
||||
|
||||
# Per-step debug logging. Off by default; toggled by the node's `debug_log` input
|
||||
# (env var LTX_IDOVERLAP_DEBUG=1 sets the initial value).
|
||||
_DEBUG_ENABLED = os.environ.get("LTX_IDOVERLAP_DEBUG", "0") == "1"
|
||||
|
||||
|
||||
def _dbg(*a):
|
||||
if _DEBUG_ENABLED:
|
||||
print("[LTXIdOverlap] " + " ".join(str(x) for x in a), flush=True)
|
||||
|
||||
|
||||
# ---------------- ArcFace + projector (same as the other nodes) ----------------
|
||||
def _get_face_app():
|
||||
global _FACE_APP
|
||||
if _FACE_APP is None:
|
||||
try:
|
||||
from insightface.app import FaceAnalysis
|
||||
except ImportError as e:
|
||||
raise RuntimeError("This node needs the optional insightface package: pip install insightface (with onnxruntime "
|
||||
"or onnxruntime-gpu), then restart ComfyUI.") from e
|
||||
providers = (["CUDAExecutionProvider", "CPUExecutionProvider"] if _USE_GPU else ["CPUExecutionProvider"])
|
||||
app = FaceAnalysis(name="buffalo_l", providers=providers)
|
||||
app.prepare(ctx_id=0 if _USE_GPU else -1, det_size=(640, 640))
|
||||
_FACE_APP = app
|
||||
return _FACE_APP
|
||||
|
||||
|
||||
class IdentityProjector(nn.Module):
|
||||
def __init__(self, in_dim=512, context_dim=4096, num_tokens=4, proj=None):
|
||||
super().__init__()
|
||||
self.in_dim = in_dim; self.context_dim = context_dim; self.num_tokens = num_tokens
|
||||
# `proj` is built from the checkpoint (variable depth); fallback = original 2-linear MLP.
|
||||
self.proj = proj if proj is not None else nn.Sequential(
|
||||
nn.Linear(in_dim, 1024), nn.GELU(), nn.Linear(1024, num_tokens * context_dim))
|
||||
self.norm = nn.LayerNorm(context_dim)
|
||||
|
||||
def forward(self, e):
|
||||
return self.norm(self.proj(e).reshape(-1, self.num_tokens, self.context_dim))
|
||||
|
||||
|
||||
def _load_projector(path, device):
|
||||
"""Load an IdentityProjector of ANY depth by rebuilding proj.N Linears from the state_dict
|
||||
(old = 2 linears proj.0/proj.2; new enhanced = 3 linears proj.0/proj.2/proj.4, 16 tokens)."""
|
||||
sd = load_file(path)
|
||||
context_dim = sd["norm.weight"].shape[0]
|
||||
# collect Linear layers in order (proj.<idx>.weight), with GELU between them
|
||||
idxs = sorted({int(k.split(".")[1]) for k in sd if k.startswith("proj.") and k.endswith(".weight")})
|
||||
layers = []
|
||||
for j, i in enumerate(idxs):
|
||||
w = sd[f"proj.{i}.weight"]
|
||||
layers.append(nn.Linear(w.shape[1], w.shape[0]))
|
||||
if j < len(idxs) - 1:
|
||||
layers.append(nn.GELU())
|
||||
proj = nn.Sequential(*layers)
|
||||
in_dim = sd["proj.0.weight"].shape[1]
|
||||
num_tokens = sd[f"proj.{idxs[-1]}.weight"].shape[0] // context_dim
|
||||
# Sequential indices must match the state_dict (Linear at even positions, GELU odd) — they do.
|
||||
p = IdentityProjector(in_dim=in_dim, context_dim=context_dim, num_tokens=num_tokens, proj=proj)
|
||||
p.load_state_dict(sd)
|
||||
log.info("IdentityProjector loaded: %d tokens, %d Linear layers", num_tokens, len(idxs))
|
||||
return p.to(device=device, dtype=torch.float32).eval()
|
||||
|
||||
|
||||
def _arcface_embed(image_bhwc, mode="auto_adjust"):
|
||||
"""Return the ArcFace embedding, or None if disabled / no face found.
|
||||
mode: 'as_is' (detect on the image only), 'auto_adjust' (retry with border-pad
|
||||
zoom-out + upscale when detection fails), 'disable' (skip ArcFace entirely).
|
||||
"""
|
||||
if mode == "disable":
|
||||
return None
|
||||
import cv2
|
||||
app = _get_face_app()
|
||||
img = np.ascontiguousarray((np.clip(image_bhwc[0].detach().cpu().numpy(), 0.0, 1.0) * 255.0).astype(np.uint8)[:, :, ::-1])
|
||||
attempts = [img]
|
||||
if mode == "auto_adjust":
|
||||
h, w = img.shape[:2]
|
||||
pad = int(0.4 * max(h, w)) # zoom-out: face may fill the frame
|
||||
attempts.append(cv2.copyMakeBorder(img, pad, pad, pad, pad, cv2.BORDER_REPLICATE))
|
||||
attempts.append(cv2.resize(img, None, fx=2.0, fy=2.0, interpolation=cv2.INTER_CUBIC)) # upscale small faces
|
||||
attempts.append(cv2.resize(img, None, fx=0.5, fy=0.5, interpolation=cv2.INTER_AREA)) # downscale huge faces
|
||||
for a in attempts:
|
||||
faces = app.get(a)
|
||||
if faces:
|
||||
f = max(faces, key=lambda x: (x.bbox[2] - x.bbox[0]) * (x.bbox[3] - x.bbox[1]))
|
||||
return torch.from_numpy(f.normed_embedding.astype(np.float32))
|
||||
return None
|
||||
|
||||
|
||||
def _append_ctx_tokens(conditioning, tokens):
|
||||
out = []
|
||||
for ce, cd in conditioning:
|
||||
t = tokens.to(device=ce.device, dtype=ce.dtype)
|
||||
if t.shape[0] != ce.shape[0]:
|
||||
t = t.expand(ce.shape[0], -1, -1)
|
||||
if t.shape[-1] < ce.shape[-1]:
|
||||
t = F.pad(t, (0, ce.shape[-1] - t.shape[-1]))
|
||||
elif t.shape[-1] > ce.shape[-1]:
|
||||
t = t[..., :ce.shape[-1]]
|
||||
nd = cd.copy()
|
||||
am = nd.get("attention_mask")
|
||||
if am is not None:
|
||||
nd["attention_mask"] = torch.cat([am, torch.ones((*am.shape[:-1], t.shape[1]), device=am.device, dtype=am.dtype)], dim=-1)
|
||||
out.append([torch.cat([ce, t], dim=1), nd])
|
||||
return out
|
||||
|
||||
|
||||
# ---------------- source_phase RoPE (port of ltx_core rope.apply_segment_phase) ----------------
|
||||
def _rotate_ref_block(pe, start, length, seg_value, theta=10000.0):
|
||||
"""Rotate RoPE freqs of tokens [start:start+length] by phase = seg_value * theta^(-d/L).
|
||||
One reference block's own token range gets its own seg_value (source_id*phase_scale) --
|
||||
callers loop this once per stacked reference so each keeps its own RoPE phase tag. With a
|
||||
single reference (the common case), this is exactly the old "last ref_len tokens" rotation.
|
||||
|
||||
Two upstream pe formats, detected at runtime (ComfyUI changed this in 2026-07):
|
||||
- fused matrix (current): pe = (rotation_matrix, split_mode_bool); rotation_matrix shape
|
||||
[B, T, num_heads, L, 2, 2], one true 2x2 rotation matrix per (batch, token, head, freq).
|
||||
We compose an additional phase rotation via matmul -- 2D rotation matrices commute, so
|
||||
left/right composition order doesn't matter: R(seg*rate) @ R(existing) == R(existing+seg*rate).
|
||||
- separate cos/sin (pre-2026-07 ComfyUI): pe = (cos, sin, [split_flag]), each [..., T, L].
|
||||
"""
|
||||
if length <= 0 or seg_value == 0.0:
|
||||
return pe
|
||||
first = pe[0]
|
||||
if torch.is_tensor(first) and first.dim() >= 3 and first.shape[-2:] == (2, 2):
|
||||
# fused rotation-matrix format: pe = (matrix[B,T,H,L,2,2], split_mode)
|
||||
matrix = first
|
||||
rest = tuple(pe[1:])
|
||||
L = matrix.shape[-3]
|
||||
d = torch.arange(L, device=matrix.device, dtype=torch.float32)
|
||||
rate = theta ** (-d / float(L))
|
||||
phase = seg_value * rate # [L]
|
||||
cp = phase.cos().to(matrix.dtype)
|
||||
sp = phase.sin().to(matrix.dtype)
|
||||
r_phase = torch.stack(
|
||||
(torch.stack((cp, -sp), dim=-1), torch.stack((sp, cp), dim=-1)), dim=-2
|
||||
) # [L, 2, 2]
|
||||
sub = matrix[:, start:start + length] # [B, length, H, L, 2, 2]
|
||||
rotated = torch.matmul(r_phase, sub)
|
||||
matrix = matrix.clone()
|
||||
matrix[:, start:start + length] = rotated
|
||||
_dbg("rotate ref block (matrix): L", L, "start", start, "length", length, "seg", seg_value)
|
||||
return (matrix, *rest)
|
||||
|
||||
# legacy separate cos/sin format
|
||||
cos, sin = pe[0], pe[1]
|
||||
rest = tuple(pe[2:])
|
||||
L = cos.shape[-1]
|
||||
d = torch.arange(L, device=cos.device, dtype=torch.float32)
|
||||
rate = theta ** (-d / float(L)) # (0,1], high-freq carries the tag
|
||||
phase = (seg_value * rate) # [L]
|
||||
pc = phase.cos().to(cos.dtype); ps = phase.sin().to(sin.dtype)
|
||||
# index the token axis (=-2)
|
||||
idx = [slice(None)] * cos.dim()
|
||||
idx[-2] = slice(start, start + length)
|
||||
idx = tuple(idx)
|
||||
c0, s0 = cos[idx], sin[idx]
|
||||
cos = cos.clone(); sin = sin.clone()
|
||||
cos[idx] = c0 * pc - s0 * ps
|
||||
sin[idx] = s0 * pc + c0 * ps
|
||||
_dbg("rotate ref block: L", L, "start", start, "length", length, "seg", seg_value)
|
||||
return (cos, sin, *rest)
|
||||
|
||||
|
||||
# ---------------- model patches (idempotent, on the av_model instance) ----------------
|
||||
def _find_ltxv(model):
|
||||
m = getattr(model, "model", model)
|
||||
m = getattr(m, "diffusion_model", m)
|
||||
return m
|
||||
|
||||
|
||||
def _detect_ltx_version(ltxv):
|
||||
"""Detect the LTX generation from a model-native checkpoint parameter.
|
||||
|
||||
LTX-2.5 introduces ``keyframes_abs_pos_embedding``. ComfyUI detects the same
|
||||
state-dict key when constructing the model and exposes both the feature flag
|
||||
and the parameter on LTXBaseModel. Avoid filename-based detection because a
|
||||
user can rename or quantize a checkpoint without changing its architecture.
|
||||
"""
|
||||
if bool(getattr(ltxv, "use_keyframes_abs_pos_embedding", False)):
|
||||
return "2.5"
|
||||
if getattr(ltxv, "keyframes_abs_pos_embedding", None) is not None:
|
||||
return "2.5"
|
||||
return "2.3/legacy"
|
||||
|
||||
|
||||
def _letterbox_resize(ref_img, tgt_w, tgt_h, pad_value=1.0):
|
||||
"""Resize `ref_img` ([B,H,W,C]) to fit ENTIRELY inside tgt_w x tgt_h, preserving its own
|
||||
aspect ratio (no crop, no distortion), padding the leftover space with `pad_value`
|
||||
(default white, matching the composite sheet's own white background). Unlike
|
||||
common_upscale(..., crop="center") -- which center-crops to the target aspect ratio
|
||||
BEFORE resizing, silently discarding whatever isn't in the middle of the source image --
|
||||
this never discards any pixel of the reference."""
|
||||
import comfy.utils
|
||||
x = ref_img.movedim(-1, 1) # [B,C,H,W]
|
||||
_, _, src_h, src_w = x.shape
|
||||
scale = min(tgt_w / src_w, tgt_h / src_h)
|
||||
new_w, new_h = max(1, round(src_w * scale)), max(1, round(src_h * scale))
|
||||
resized = comfy.utils.common_upscale(x, new_w, new_h, "bilinear", "disabled")
|
||||
pad_w, pad_h = tgt_w - new_w, tgt_h - new_h
|
||||
left, top = pad_w // 2, pad_h // 2
|
||||
right, bottom = pad_w - left, pad_h - top
|
||||
padded = F.pad(resized, (left, right, top, bottom), mode="constant", value=pad_value)
|
||||
return padded.movedim(1, -1)
|
||||
|
||||
|
||||
def _anchored_crop_resize(ref_img, tgt_w, tgt_h, anchor="center"):
|
||||
"""Like comfy.utils.common_upscale(..., crop="center") but with a configurable anchor
|
||||
for WHICH part of the source survives the crop when the aspect ratio doesn't match,
|
||||
instead of always the exact center (e.g. anchor="top" keeps the top of the sheet --
|
||||
useful when the face closeup panel isn't centered in your layout). Returns the
|
||||
cropped+resized image plus the crop box (x0, y0, crop_w, crop_h) in SOURCE pixel
|
||||
coords, for drawing a preview overlay."""
|
||||
import comfy.utils
|
||||
x_img = ref_img.movedim(-1, 1) # [B,C,H,W]
|
||||
_, _, old_h, old_w = x_img.shape
|
||||
old_aspect = old_w / old_h
|
||||
new_aspect = tgt_w / tgt_h
|
||||
x0, y0, crop_w, crop_h = 0, 0, old_w, old_h
|
||||
if old_aspect > new_aspect:
|
||||
# source wider than target -- crop width
|
||||
crop_w = max(1, round(old_w * (new_aspect / old_aspect)))
|
||||
if anchor == "left":
|
||||
x0 = 0
|
||||
elif anchor == "right":
|
||||
x0 = old_w - crop_w
|
||||
else:
|
||||
x0 = (old_w - crop_w) // 2
|
||||
elif old_aspect < new_aspect:
|
||||
# source taller than target -- crop height
|
||||
crop_h = max(1, round(old_h * (old_aspect / new_aspect)))
|
||||
if anchor == "top":
|
||||
y0 = 0
|
||||
elif anchor == "bottom":
|
||||
y0 = old_h - crop_h
|
||||
else:
|
||||
y0 = (old_h - crop_h) // 2
|
||||
cropped = x_img[:, :, y0:y0 + crop_h, x0:x0 + crop_w]
|
||||
out = comfy.utils.common_upscale(cropped, tgt_w, tgt_h, "bilinear", "disabled")
|
||||
return out.movedim(1, -1), (x0, y0, crop_w, crop_h)
|
||||
|
||||
|
||||
def _draw_crop_overlay(ref_img, box):
|
||||
"""Original reference with a green rectangle around the region that SURVIVES the crop
|
||||
(everything outside the rectangle gets discarded). box = (x0, y0, w, h) in source pixel
|
||||
coords, e.g. from _anchored_crop_resize. If box covers the whole image (no crop, as in
|
||||
letterbox/native_resolution modes), the rectangle just outlines the full frame."""
|
||||
from PIL import Image, ImageDraw
|
||||
x0, y0, cw, ch = box
|
||||
arr = (ref_img[0, :, :, :3].clamp(0, 1).cpu().numpy() * 255).astype(np.uint8)
|
||||
pil = Image.fromarray(arr).convert("RGB")
|
||||
draw = ImageDraw.Draw(pil)
|
||||
w, h = pil.size
|
||||
lw = max(2, min(w, h) // 150)
|
||||
draw.rectangle([x0, y0, x0 + cw - 1, y0 + ch - 1], outline=(0, 255, 0), width=lw)
|
||||
out = torch.from_numpy(np.array(pil).astype(np.float32) / 255.0).unsqueeze(0)
|
||||
return out
|
||||
|
||||
|
||||
# Seconds reserved per numbered strata slot -- MUST match ltx_trainer's
|
||||
# training_strategies.tass.STRATA_SLOT_WIDTH for a strata-trained checkpoint's RoPE
|
||||
# convention to line up at inference. Slot 0 (1st ref in the batch) lands at
|
||||
# target_max_t + this value, slot 1 (2nd ref) at target_max_t + 2*this value, etc. --
|
||||
# dynamic/target-relative, same as st_drc's own shift, so it stays correct for whatever
|
||||
# video length is actually generated.
|
||||
# CHANGED 2026-07-18 (0.5 -> 1.5): 0.5 caused a real slot0/slot1 collision -- a single-frame
|
||||
# reference image (fps=1.0 convention) spans a full 1.0-SECOND range on the T axis, not a
|
||||
# point, so width=0.5 let adjacent slots overlap. See tass.py's STRATA_SLOT_WIDTH docstring.
|
||||
# Any checkpoint trained before this fix (e.g. phantom_stacked_r128/_p2) learned under the
|
||||
# collided geometry; continuing training from one now adapts to the corrected spacing.
|
||||
STRATA_SLOT_WIDTH = 1.5
|
||||
|
||||
|
||||
def _tass_axis_stats(positions):
|
||||
"""Per-axis (min, max), reduced over every dimension EXCEPT the T/H/W one.
|
||||
|
||||
Two coordinate shapes reach this node: older LTX builds hand out one corner per token,
|
||||
[B, 3 (T/H/W), N], while LTX-2.5 hands out patch bounds, [B, 3, N, 2] -- the same shape
|
||||
the trainer works in. Reducing over "everything but the axis" is correct for both, and
|
||||
keeps a patch's two bounds shifted by the SAME amount instead of by their own separate
|
||||
minima (which would squeeze the span).
|
||||
|
||||
The batch is reduced too. Layouts run before the reference tokens are expanded to the CFG
|
||||
batch, so target_positions can be [2, ...] while the reference is still [1, ...]; cond and
|
||||
uncond share one geometry, so a batch-wide min/max is the same number either way and
|
||||
broadcasts into whichever side is batch-1.
|
||||
"""
|
||||
dims = (0,) + tuple(range(2, positions.dim()))
|
||||
return positions.amin(dim=dims, keepdim=True), positions.amax(dim=dims, keepdim=True)
|
||||
|
||||
|
||||
def _apply_tass_layout(reference_positions, target_positions, layout: str, strata_start: float | None = None,
|
||||
sidecar_margin_pixels: float = 0.0, use_middle_indices_grid: bool = True):
|
||||
"""Place reference pixel-coords in a non-overlapping TASS region -- mirrors
|
||||
ltx_core.conditioning.reference_layout.apply_reference_layout (kept in sync manually since
|
||||
this node can't import the trainer package). Handles both coordinate shapes ComfyUI emits;
|
||||
see _tass_axis_stats.
|
||||
layout='overlap' returns the input unchanged.
|
||||
layout='st_drc' shifts every axis (T, H, W) past the target's own extent.
|
||||
layout='strata' shifts ONLY the T axis to an absolute band start (`strata_start`, in the
|
||||
same raw pixel/frame units as `reference_positions` -- caller converts from seconds using
|
||||
the model's frame_rate), leaving H/W overlapping the target -- see STRATA_SLOT_WIDTH.
|
||||
layout='sidecar' places the reference BESIDE the target on the W axis, vertically centred on
|
||||
it, and stretches its T span to cover the target's whole clip -- so the reference is present
|
||||
for every frame instead of sitting at one instant. This is the layout the replace_refedit
|
||||
recipe validated for an object reference.
|
||||
"""
|
||||
if layout == "overlap":
|
||||
return reference_positions
|
||||
# ComfyUI's pixel coords are int64 (latent corners times integer VAE scale factors), while
|
||||
# centring and midpoints are inherently half-pixel maths -- an in-place += of a float shift
|
||||
# into a Long tensor just raises "result type Float can't be cast to the desired output
|
||||
# type Long". Compute in float and round back on the way out, so callers keep the dtype
|
||||
# they passed in (the caller casts to the target grid's dtype when concatenating, and a
|
||||
# cast truncates where rounding is what we want).
|
||||
coordinate_dtype = reference_positions.dtype
|
||||
if not reference_positions.is_floating_point():
|
||||
reference_positions = reference_positions.to(torch.float32)
|
||||
if not target_positions.is_floating_point():
|
||||
target_positions = target_positions.to(torch.float32)
|
||||
|
||||
def _restore(shifted):
|
||||
if shifted.dtype == coordinate_dtype:
|
||||
return shifted
|
||||
return shifted.round().to(coordinate_dtype)
|
||||
|
||||
target_min, target_extent = _tass_axis_stats(target_positions)
|
||||
reference_origin, reference_extent = _tass_axis_stats(reference_positions)
|
||||
if layout == "st_drc":
|
||||
return _restore(reference_positions + (target_extent - reference_origin))
|
||||
if layout == "strata":
|
||||
if strata_start is None:
|
||||
raise ValueError("layout='strata' requires strata_start")
|
||||
shifted = reference_positions.clone()
|
||||
shifted[:, 0:1, ...] += strata_start - reference_origin[:, 0:1, ...]
|
||||
return _restore(shifted)
|
||||
if layout == "sidecar":
|
||||
if reference_positions.shape[1] < 3:
|
||||
raise ValueError(
|
||||
f"layout='sidecar' expects at least 3 coordinate axes, got {reference_positions.shape[1]}"
|
||||
)
|
||||
shifted = reference_positions.clone()
|
||||
# H: centraliza a referencia na altura do alvo
|
||||
target_center_h = (target_min[:, 1:2, ...] + target_extent[:, 1:2, ...]) * 0.5
|
||||
reference_center_h = (reference_origin[:, 1:2, ...] + reference_extent[:, 1:2, ...]) * 0.5
|
||||
shifted[:, 1:2, ...] += target_center_h - reference_center_h
|
||||
# W: desloca para a direita do alvo, mais a margem
|
||||
shifted[:, 2:3, ...] += (
|
||||
target_extent[:, 2:3, ...] + float(sidecar_margin_pixels) - reference_origin[:, 2:3, ...]
|
||||
)
|
||||
# T: a referencia cobre todo o intervalo do alvo. O trainer escreve o par
|
||||
# (inicio, fim) = (t_min, t_max), e o modelo o colapsa num unico numero por token --
|
||||
# de duas maneiras diferentes, e e por isso que aqui tem um `if`:
|
||||
# use_middle_indices_grid=True (2.5, e qualquer 2.3 cujo metadata declare) -> (inicio+fim)/2
|
||||
# use_middle_indices_grid=False (2.3 nos checkpoints antigos) -> so o inicio
|
||||
# Escrever (t_min, t_max) nos dois casos poria a referencia no MEIO do clipe no 2.5 e no
|
||||
# FRAME 0 no 2.3 -- placements diferentes para o mesmo LoRA. Escrever o ponto medio nos
|
||||
# dois bounds da a mesma posicao efetiva sob qualquer uma das regras.
|
||||
t_min = target_min[:, 0, 0, 0] if target_min.dim() == 4 else target_min[:, 0, 0]
|
||||
t_max = target_extent[:, 0, 0, 0] if target_extent.dim() == 4 else target_extent[:, 0, 0]
|
||||
if shifted.dim() == 4 and use_middle_indices_grid:
|
||||
shifted[:, 0, :, 0] = t_min
|
||||
shifted[:, 0, :, 1] = t_max
|
||||
elif shifted.dim() == 4:
|
||||
shifted[:, 0, :, :] = ((t_min + t_max) * 0.5).unsqueeze(-1)
|
||||
else:
|
||||
# Coordenadas legadas [B, 3, N]: um canto por token, lido direto pelo RoPE.
|
||||
shifted[:, 0, :] = (t_min + t_max) * 0.5
|
||||
return _restore(shifted)
|
||||
raise ValueError(f"Unsupported TASS layout {layout!r}")
|
||||
|
||||
|
||||
def _unbound(obj, name):
|
||||
"""The plain function behind `obj.name`, so closures never pin the instance.
|
||||
|
||||
A bound method carries a strong reference to its instance; captured inside the patch closures
|
||||
below it would keep the model alive through its own __dict__ (see _bind_weak).
|
||||
"""
|
||||
function = getattr(type(obj), name, None)
|
||||
if function is not None:
|
||||
return function
|
||||
attribute = getattr(obj, name, None)
|
||||
if attribute is None:
|
||||
return None
|
||||
# Dynamically assigned (already patched by another node): fall back to its underlying
|
||||
# function when there is one, otherwise wrap the bound callable and accept the reference.
|
||||
return getattr(attribute, "__func__", None) or (lambda _self, *a, **k: attribute(*a, **k))
|
||||
|
||||
|
||||
def _bind_weak(function, instance):
|
||||
"""Install-ready callable with a method's signature but no reference cycle.
|
||||
|
||||
`instance._x = types.MethodType(fn, instance)` makes instance -> __dict__ -> method ->
|
||||
instance, a cycle only a full collection can break. ComfyUI holds a weakref to the model and,
|
||||
finding it alive after a run, logs "memory leak with model LTXAV" and forces a full gc every
|
||||
execution -- which is what makes repeated runs progressively slower (issue #9). Resolving the
|
||||
instance from a weakref at call time keeps the patch working with none of that.
|
||||
"""
|
||||
reference = weakref.ref(instance)
|
||||
|
||||
@functools.wraps(function)
|
||||
def patched(*args, **kwargs):
|
||||
target = reference()
|
||||
if target is None:
|
||||
raise RuntimeError("LTXIdentityOverlap: the patched model no longer exists")
|
||||
return function(target, *args, **kwargs)
|
||||
|
||||
return patched
|
||||
|
||||
|
||||
def _install_patches(ltxv):
|
||||
if getattr(ltxv, "_id_overlap_patched", False):
|
||||
return
|
||||
orig_process_input = _unbound(ltxv, "_process_input")
|
||||
orig_prepare_ts = _unbound(ltxv, "_prepare_timestep")
|
||||
orig_prepare_pe = _unbound(ltxv, "_prepare_positional_embeddings")
|
||||
orig_process_output = _unbound(ltxv, "_process_output")
|
||||
orig_forward_internal = _unbound(ltxv, "_forward")
|
||||
|
||||
if orig_forward_internal is not None:
|
||||
def _forward_capture_fps(self, x, timestep, context, attention_mask, frame_rate=25,
|
||||
transformer_options={}, keyframe_idxs=None, denoise_mask=None, **kwargs):
|
||||
# _forward calls _process_input BEFORE _prepare_positional_embeddings (the only
|
||||
# other place frame_rate normally reaches) -- process_input needs it EARLIER, to
|
||||
# convert the seconds-scale STRATA_SLOT_WIDTH into this step's raw pixel/frame
|
||||
# units. Stash here so it's always current, never a step behind.
|
||||
self._id_frame_rate = float(frame_rate)
|
||||
return orig_forward_internal(
|
||||
self, x, timestep, context, attention_mask, frame_rate=frame_rate,
|
||||
transformer_options=transformer_options, keyframe_idxs=keyframe_idxs,
|
||||
denoise_mask=denoise_mask, **kwargs,
|
||||
)
|
||||
ltxv._forward = _bind_weak(_forward_capture_fps, ltxv)
|
||||
|
||||
def process_input(self, x, keyframe_idxs, denoise_mask, **kw):
|
||||
# Reset per-forward state first so a stale value from a previous run can never leak
|
||||
# into this forward (e.g. if ref specs stop arriving after a Comfy update).
|
||||
self._id_ref_len = 0
|
||||
self._id_blocks = []
|
||||
out = orig_process_input(self, x, keyframe_idxs, denoise_mask, **kw)
|
||||
ref_specs = kw.get("_id_ref_specs")
|
||||
if ref_specs is None:
|
||||
ref_specs = (kw.get("transformer_options") or {}).get("_id_ref_specs")
|
||||
if not ref_specs:
|
||||
return out
|
||||
try:
|
||||
from comfy.ldm.lightricks.model import latent_to_pixel_coords
|
||||
xx, pix, add = out
|
||||
is_av = isinstance(xx, (list, tuple))
|
||||
vx = xx[0] if is_av else xx
|
||||
vco = pix[0] if is_av else pix
|
||||
target_len = vx.shape[1]
|
||||
self._id_target_len = target_len
|
||||
frame_rate = float(getattr(self, "_id_frame_rate", 25.0))
|
||||
# Raw pixel/frame units (pre frame_rate-division -- that happens later inside
|
||||
# _prepare_positional_embeddings) -- convert to seconds only for the strata math,
|
||||
# then back, since STRATA_SLOT_WIDTH is calibrated in seconds on the trainer side.
|
||||
target_max_t_raw = float(vco[:, 0, :].amax().item())
|
||||
_dbg("process_input IN: is_av", is_av, "| vx", _shape(vx), "| vco", _shape(vco),
|
||||
"| n_refs", len(ref_specs), "| frame_rate", frame_rate)
|
||||
blocks = [] # (start, length, seg_value) per ref, in concatenation order
|
||||
offset = target_len
|
||||
for spec in ref_specs:
|
||||
ref_lat = spec["latent"]
|
||||
rt, rlc = self.patchifier.patchify(ref_lat.to(dtype=vx.dtype, device=vx.device))
|
||||
rpc = latent_to_pixel_coords(latent_coords=rlc, scale_factors=self.vae_scale_factors,
|
||||
causal_fix=self.causal_temporal_positioning)
|
||||
# downscale_factor: this ref was VAE-encoded at target_size/downscale_factor,
|
||||
# so its raw pixel-coords only span that smaller extent. Multiply the H/W axes
|
||||
# back up by downscale_factor to re-stretch them across the target's actual
|
||||
# coordinate range (fewer tokens, same spatial span) -- mirrors ltx_core's
|
||||
# VideoConditionByReferenceLatent.apply_to (positions[:,1:3,...] *= downscale_factor).
|
||||
dsf = spec.get("downscale_factor", 1)
|
||||
if dsf != 1:
|
||||
rpc = rpc.clone()
|
||||
rpc[:, 1, ...] *= dsf
|
||||
rpc[:, 2, ...] *= dsf
|
||||
strata_start_raw = None
|
||||
if spec["layout"] == "strata":
|
||||
slot = int(spec["strata_slot"])
|
||||
strata_start_sec = target_max_t_raw / frame_rate + (slot + 1) * STRATA_SLOT_WIDTH
|
||||
strata_start_raw = strata_start_sec * frame_rate
|
||||
rpc = _apply_tass_layout(
|
||||
rpc, vco, spec["layout"], strata_start=strata_start_raw,
|
||||
sidecar_margin_pixels=spec.get("sidecar_margin_pixels", 0.0),
|
||||
use_middle_indices_grid=bool(getattr(self, "use_middle_indices_grid", False)),
|
||||
)
|
||||
# Appearance references may be placed before frame zero to reduce the learned
|
||||
# overlap frame-0 copy/leak. Old/external specs without this key retain offset 0.
|
||||
# The shift is applied AFTER layout placement so it remains explicit for every
|
||||
# supported layout; guide/mask specs opt into 0 in LTX Multiple Controls.
|
||||
rpc = shift_reference_temporal_positions(
|
||||
rpc,
|
||||
spec.get("temporal_offset_latents", 0),
|
||||
self.vae_scale_factors[0],
|
||||
)
|
||||
# Learned per-slot tag, added in FEATURE space on the raw patchified tokens --
|
||||
# i.e. before patchify_proj, exactly where the trainer adds it (the model's
|
||||
# input projection is applied after). Specs without the key are unaffected.
|
||||
slot_vector = spec.get("slot_vector")
|
||||
if slot_vector is not None:
|
||||
rt = rt + slot_vector.to(device=rt.device, dtype=rt.dtype)
|
||||
rt = self.patchify_proj(rt)
|
||||
if rt.shape[0] != vx.shape[0]:
|
||||
rt = rt.expand(vx.shape[0], -1, -1)
|
||||
if rpc.shape[0] != vco.shape[0]:
|
||||
rpc = rpc.expand(vco.shape[0], *([-1] * (rpc.dim() - 1)))
|
||||
rlen = rt.shape[1]
|
||||
vx = torch.cat([vx, rt], dim=1)
|
||||
vco = torch.cat([vco, rpc.to(vco)], dim=2)
|
||||
blocks.append((offset, rlen, float(spec["seg_value"])))
|
||||
offset += rlen
|
||||
ref_len = offset - target_len
|
||||
self._id_ref_len = ref_len
|
||||
self._id_blocks = blocks
|
||||
add = dict(add); add["_id_ref_len"] = ref_len
|
||||
_dbg("process_input OUT: blocks", blocks, "| target_len", target_len,
|
||||
"| vx", _shape(vx), "| vco", _shape(vco))
|
||||
if is_av:
|
||||
xx = [vx, xx[1]]; pix = [vco, pix[1]]
|
||||
else:
|
||||
xx, pix = vx, vco
|
||||
return xx, pix, add
|
||||
except Exception as e:
|
||||
_dbg("ERROR process_input:", repr(e), "| out", _shape(out), "| n_refs", len(ref_specs) if ref_specs else 0)
|
||||
raise
|
||||
|
||||
def prepare_timestep(self, timestep, batch_size, hidden_dtype, **kw):
|
||||
# Give the ref tokens clean (0) timestep by editing the timestep INPUT before the
|
||||
# model's per-frame compression/adaln (mirrors the audio-ref path in av_model).
|
||||
# Use ONLY the instance attribute, not kw["_id_ref_len"] — when multi-angle patches
|
||||
# are also installed, their process_input sets kw["_id_ref_len"] which we must NOT
|
||||
# consume here (that would double-extend the timestep and cause shape mismatches).
|
||||
ref_len = getattr(self, "_id_ref_len", 0)
|
||||
if ref_len:
|
||||
target_len = getattr(self, "_id_target_len", None)
|
||||
if timestep.dim() <= 1 and target_len is not None:
|
||||
timestep = timestep.view(-1, 1).expand(batch_size, target_len).contiguous()
|
||||
if timestep.dim() >= 2 and target_len is not None:
|
||||
cur = timestep.shape[1]
|
||||
_gm0 = kw.get("grid_mask")
|
||||
_full = _gm0.shape[-1] if (_gm0 is not None and hasattr(_gm0, "shape")) else None
|
||||
if _full is not None and cur == _full:
|
||||
# Per-token timestep spans the grid-mask domain (video + IC-LoRA guide
|
||||
# tokens, indexed later via timestep[:, grid_mask]). Do NOT trim — append
|
||||
# ref zeros and let the grid_mask pad below keep everything in lockstep.
|
||||
ref_ts = torch.zeros(batch_size, ref_len, *timestep.shape[2:], device=timestep.device, dtype=timestep.dtype)
|
||||
timestep = torch.cat([timestep, ref_ts], dim=1)
|
||||
_dbg("prepare_timestep: grid-domain cur", cur, "-> appended ref", ref_len)
|
||||
elif cur > target_len + ref_len:
|
||||
_dbg("prepare_timestep: oversized", cur, "-> trimming to video-only", target_len)
|
||||
timestep = timestep[:, :target_len]
|
||||
cur = target_len
|
||||
if (_full is None or timestep.shape[1] != _full + ref_len) and cur == target_len:
|
||||
ref_ts = torch.zeros(batch_size, ref_len, *timestep.shape[2:], device=timestep.device, dtype=timestep.dtype)
|
||||
timestep = torch.cat([timestep, ref_ts], dim=1)
|
||||
_dbg("prepare_timestep: ref_len", ref_len, "| timestep ->", _shape(timestep), "| target_len", target_len)
|
||||
else:
|
||||
_dbg("prepare_timestep: skip (cur", cur, "!= target", target_len, ")")
|
||||
# Guide nodes (IC-LoRA Guide / Director Guide, etc.) inject a grid_mask that
|
||||
# FILTERS x tokens in _process_input (x = x[:, grid_mask]) and later indexes the
|
||||
# timestep (timestep[:, grid_mask]). Our ref tokens are appended AFTER that
|
||||
# filter, so extend the mask with True for them — keeps modulation and vx in
|
||||
# lockstep (False would drop our clean timesteps and desync again).
|
||||
gm = kw.get("grid_mask")
|
||||
if gm is not None and hasattr(gm, "shape"):
|
||||
gap = timestep.shape[1] - gm.shape[-1]
|
||||
if 0 < gap <= ref_len:
|
||||
pad = torch.ones(*gm.shape[:-1], gap, dtype=gm.dtype, device=gm.device)
|
||||
kw = dict(kw); kw["grid_mask"] = torch.cat([gm, pad], dim=-1)
|
||||
_dbg("prepare_timestep: grid_mask extended by", gap)
|
||||
return orig_prepare_ts(self, timestep, batch_size, hidden_dtype, **kw)
|
||||
|
||||
def prepare_pe(self, pixel_coords, frame_rate, x_dtype):
|
||||
pe = orig_prepare_pe(self, pixel_coords, frame_rate, x_dtype)
|
||||
blocks = getattr(self, "_id_blocks", [])
|
||||
theta = getattr(self, "_id_rope_theta", 10000.0)
|
||||
if not blocks:
|
||||
return pe
|
||||
try:
|
||||
_dbg("prepare_pe IN: pe struct", _shape(pe), "| blocks", blocks)
|
||||
|
||||
def rot(v_pe):
|
||||
for start, length, seg in blocks:
|
||||
v_pe = _rotate_ref_block(v_pe, start, length, seg, theta)
|
||||
return v_pe
|
||||
|
||||
# av returns [(v_pe, av_cross_video), (a_pe, av_cross_audio)]; v_pe = (cos, sin, split).
|
||||
if isinstance(pe, list) and len(pe) and isinstance(pe[0], (list, tuple)) and isinstance(pe[0][0], (list, tuple)):
|
||||
v_pe, cross_v = pe[0][0], pe[0][1]
|
||||
v_pe = rot(v_pe)
|
||||
pe = [(v_pe, cross_v), pe[1]]
|
||||
else:
|
||||
pe = rot(pe)
|
||||
return pe
|
||||
except Exception as e:
|
||||
_dbg("ERROR prepare_pe:", repr(e), "| pe", _shape(pe))
|
||||
raise
|
||||
|
||||
def process_output(self, x, embedded_timestep, keyframe_idxs, **kw):
|
||||
ref_len = getattr(self, "_id_ref_len", 0)
|
||||
if ref_len:
|
||||
try:
|
||||
from comfy.ldm.lightricks.av_model import CompressedTimestep
|
||||
_dbg("process_output IN: x", _shape(x), "| et", _shape(embedded_timestep), "| ref_len", ref_len)
|
||||
# trim ref tokens from the video stream
|
||||
if isinstance(x, (list, tuple)):
|
||||
x = [x[0][:, :x[0].shape[1] - ref_len], *x[1:]]
|
||||
import copy
|
||||
et_list = list(embedded_timestep) if isinstance(embedded_timestep, (list, tuple)) else [embedded_timestep]
|
||||
v_et = et_list[0]
|
||||
if isinstance(v_et, CompressedTimestep):
|
||||
# clone + edit slots directly (version-agnostic; some builds lack per_frame kwarg)
|
||||
ppf = max(1, getattr(v_et, "patches_per_frame", 1) or 1)
|
||||
n_ref_frames = max(1, ref_len // ppf)
|
||||
v_et2 = copy.copy(v_et)
|
||||
v_et2.data = v_et.data[:, : v_et.num_frames - n_ref_frames].contiguous()
|
||||
v_et2.num_frames = v_et.num_frames - n_ref_frames
|
||||
et_list[0] = v_et2
|
||||
elif hasattr(v_et, "shape") and v_et.dim() >= 2 and v_et.shape[1] > 1:
|
||||
et_list[0] = v_et[:, : v_et.shape[1] - ref_len]
|
||||
embedded_timestep = et_list
|
||||
else:
|
||||
x = x[:, :x.shape[1] - ref_len]
|
||||
if hasattr(embedded_timestep, "shape") and embedded_timestep.dim() >= 2 and embedded_timestep.shape[1] > 1:
|
||||
embedded_timestep = embedded_timestep[:, : embedded_timestep.shape[1] - ref_len]
|
||||
_dbg("process_output: trimmed -> x", _shape(x), "| et", _shape(embedded_timestep))
|
||||
except Exception as e:
|
||||
_dbg("ERROR process_output:", repr(e), "| x", _shape(x), "| et", _shape(embedded_timestep))
|
||||
raise
|
||||
return orig_process_output(self, x, embedded_timestep, keyframe_idxs, **kw)
|
||||
|
||||
ltxv._process_input = _bind_weak(process_input, ltxv)
|
||||
ltxv._prepare_timestep = _bind_weak(prepare_timestep, ltxv)
|
||||
ltxv._prepare_positional_embeddings = _bind_weak(prepare_pe, ltxv)
|
||||
ltxv._process_output = _bind_weak(process_output, ltxv)
|
||||
ltxv._id_overlap_patched = True
|
||||
log.info("LTXIdentityOverlap patches installed on %s", type(ltxv).__name__)
|
||||
|
||||
|
||||
class LTXIdentityOverlapConditioning:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {
|
||||
"model": ("MODEL",),
|
||||
"positive": ("CONDITIONING",),
|
||||
"negative": ("CONDITIONING",),
|
||||
"vae": ("VAE",),
|
||||
"latent": ("LATENT",),
|
||||
"reference_image": ("IMAGE", {
|
||||
"tooltip": "Reference to copy into the generation -- any subject (object, animal, "
|
||||
"character, person...), not just a face. Accepts a BATCH of N images (use "
|
||||
"an Image Batch node to combine several) for checkpoints trained on "
|
||||
"multiple STACKED references (layout='strata') -- each image in the batch "
|
||||
"becomes its own reference block with source_id = source_id + its index "
|
||||
"(0-based: 1st image keeps 'source_id' as-is, 2nd gets source_id+1, ...). "
|
||||
"For the exact historical frame-0 placement, set the reference temporal "
|
||||
"offset to 0."}),
|
||||
"source_id": ("FLOAT", {"default": 2.0, "min": 0.0, "max": 8.0, "step": 1.0,
|
||||
"tooltip": "source_phase segment id (training used 2). 0 = no phase."}),
|
||||
"phase_scale": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.1}),
|
||||
"ref_resize_mode": (["match_target", "match_target_letterbox", "native_resolution"], {"default": "match_target",
|
||||
"tooltip": "match_target: resize ref to the OUTPUT video's pixel size via a CENTER-CROP then "
|
||||
"resize (old single-face-crop recipes — ref resolution never mattered, but this "
|
||||
"silently discards whatever isn't in the middle of the ref for mismatched aspect "
|
||||
"ratios, e.g. a landscape character sheet used for a portrait output loses the "
|
||||
"side panels/face closeup). match_target_letterbox: same target pixel size, but "
|
||||
"fits the WHOLE ref inside it preserving aspect ratio (no crop, pads with white) "
|
||||
"-- use this for a landscape composite sheet + portrait (or any mismatched-aspect) "
|
||||
"output so no panel/face-detail gets cut off. native_resolution: encode the ref at "
|
||||
"ITS OWN size (rounded to the nearest 32px), independent of the video size — "
|
||||
"REQUIRED for checkpoints trained on a fixed ref resolution bucket that differs "
|
||||
"from the video's own bucket (e.g. the composite face+3views ref, trained at "
|
||||
"2048x1024 regardless of output video size) -- but for aspect-mismatched outputs "
|
||||
"(e.g. portrait video) this can bias the model toward the ref's own (landscape) "
|
||||
"composition, causing an off-center/cropped-looking result; try "
|
||||
"match_target_letterbox first if you hit that."}),
|
||||
"debug_log": ("BOOLEAN", {"default": False,
|
||||
"tooltip": "Print per-step [LTXIdOverlap] shape logs to the console (for debugging)."}),
|
||||
}, "optional": {
|
||||
"crop_anchor": (["center", "top", "bottom", "left", "right"], {"default": "center",
|
||||
"tooltip": "New in v1.10.13, optional -- old workflows without this input keep the "
|
||||
"previous always-center-crop behavior. Only used by match_target (the "
|
||||
"crop-then-resize mode). Which part of the reference survives the crop when its "
|
||||
"aspect ratio doesn't match the output -- e.g. if your sheet's face closeup is "
|
||||
"in the top row, set 'top' instead of the default center crop so it doesn't get "
|
||||
"cut off. No effect on match_target_letterbox or native_resolution (neither "
|
||||
"ever crops)."}),
|
||||
"layout": (["overlap", "st_drc", "strata", "sidecar"], {"default": "overlap",
|
||||
"tooltip": "The base positional layout used by the checkpoint. 'overlap' starts from the "
|
||||
"target's own RoPE coordinate range and then applies the independent reference "
|
||||
"temporal offset; use offset 0 for the exact historical overlap grid. 'st_drc' "
|
||||
"shifts the WHOLE reference block past the target's coordinate extent on every "
|
||||
"axis (non-overlapping region). 'strata' shifts ONLY the temporal axis to a slot "
|
||||
"past the target's own length -- one slot per image in the reference_image batch "
|
||||
"(1st image -> slot 0, 2nd -> slot 1, ...), leaving H/W overlapping the target; "
|
||||
"this is the 'stacked references' convention (ltx_trainer TASS strata layout). "
|
||||
"Only use whichever layout the loaded checkpoint was actually trained with -- "
|
||||
"using the wrong one is not a quality hit, it's a coordinate convention the "
|
||||
"model never learned, so identity transfer likely won't work at all."}),
|
||||
"reference_guidance_scale": ("FLOAT", {"default": 1.0, "min": 1.0, "max": 10.0, "step": 0.1,
|
||||
"tooltip": "ST-DRC-style reference-CFG (arxiv 2606.02441). 1.0 = off (identical to "
|
||||
"before this input existed). >1.0 adds a THIRD forward pass per step with "
|
||||
"the reference tokens dropped, and amplifies the reference's own "
|
||||
"contribution: denoised += (scale-1)*(with_ref - without_ref) -- the same "
|
||||
"way CFG amplifies the text prompt's. Costs one extra (cheaper, "
|
||||
"ref-token-free) forward pass per step when enabled. Start around 2-4; "
|
||||
"same units/convention as CFG scale on the sampler."}),
|
||||
# Appended last so saved workflows keep every historical widget index stable.
|
||||
"reference_temporal_offset_latents": reference_temporal_offset_input(),
|
||||
}}
|
||||
|
||||
RETURN_TYPES = ("MODEL", "CONDITIONING", "CONDITIONING", "LATENT", "STRING", "IMAGE", "IMAGE")
|
||||
RETURN_NAMES = ("model", "positive", "negative", "latent", "debug", "ref_preview", "crop_overlay")
|
||||
FUNCTION = "apply"
|
||||
CATEGORY = "LTX/identity"
|
||||
DESCRIPTION = ("Overlap/st_drc/strata + source_phase reference conditioning via a model patch, with an "
|
||||
"experimental reference-only temporal offset to reduce frame-0 composition leakage. "
|
||||
"Reference is separate tokens (NOT I2V), any subject -- not just a face. Accepts a BATCH of "
|
||||
"images for checkpoints trained on stacked references (layout='strata'). "
|
||||
"Load LoRA on MODEL first. ref_preview/crop_overlay outputs show exactly what gets encoded "
|
||||
"and, for match_target, what part of the reference survives the crop (green box) vs gets discarded.")
|
||||
|
||||
def apply(self, model, positive, negative, vae, latent, reference_image,
|
||||
reference_temporal_offset_latents=DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
|
||||
source_id=2.0, phase_scale=1.0,
|
||||
ref_resize_mode="match_target", debug_log=False,
|
||||
crop_anchor="center", layout="overlap", reference_guidance_scale=1.0):
|
||||
import comfy.samplers
|
||||
import comfy.utils
|
||||
|
||||
global _DEBUG_ENABLED
|
||||
_DEBUG_ENABLED = bool(debug_log)
|
||||
m = model.clone()
|
||||
ltxv = _find_ltxv(m)
|
||||
detected_ltx_version = _detect_ltx_version(ltxv)
|
||||
|
||||
_, w_sf, h_sf = vae.downscale_index_formula
|
||||
n_refs = reference_image.shape[0]
|
||||
|
||||
def _encode_one(img1):
|
||||
"""Resize (per ref_resize_mode/crop_anchor) + VAE-encode ONE reference image
|
||||
([1,H,W,C] slice). Byte-identical to the pre-batch code path when n_refs == 1."""
|
||||
if ref_resize_mode == "native_resolution":
|
||||
# Encode at the ref image's OWN resolution (rounded to nearest 32px),
|
||||
# independent of the output video size -- matches training when the ref used
|
||||
# a fixed/own bucket.
|
||||
_, src_h, src_w, _ = img1.shape
|
||||
tgt_w = max(w_sf, round(src_w / w_sf) * w_sf)
|
||||
tgt_h = max(h_sf, round(src_h / h_sf) * h_sf)
|
||||
else:
|
||||
# Legacy behavior: resize ref to match the target video's pixel size (correct
|
||||
# for recipes where the ref used the SAME resolution bucket as the video, e.g.
|
||||
# a small face crop -- resolution never mattered there).
|
||||
_, _, _, lat_h, lat_w = latent["samples"].shape
|
||||
tgt_w, tgt_h = lat_w * w_sf, lat_h * h_sf
|
||||
_, src_h0, src_w0, _ = img1.shape
|
||||
crop_box = (0, 0, src_w0, src_h0) # default: "nothing cropped" (letterbox/native modes)
|
||||
if ref_resize_mode == "match_target_letterbox":
|
||||
ref_px = _letterbox_resize(img1, tgt_w, tgt_h)[:1, :, :, :3]
|
||||
elif ref_resize_mode == "match_target" and crop_anchor != "center":
|
||||
# non-default anchor: use the configurable-anchor crop (new in v1.10.13)
|
||||
ref_px, crop_box = _anchored_crop_resize(img1, tgt_w, tgt_h, anchor=crop_anchor)
|
||||
ref_px = ref_px[:1, :, :, :3]
|
||||
else:
|
||||
# unchanged from before v1.10.13 -- exact original center-crop path, byte-identical
|
||||
ref_px = comfy.utils.common_upscale(img1.movedim(-1, 1), tgt_w, tgt_h, "bilinear", "center").movedim(1, -1)[:1, :, :, :3]
|
||||
if ref_resize_mode == "match_target":
|
||||
_, crop_box = _anchored_crop_resize(img1, tgt_w, tgt_h, anchor="center") # for the preview overlay only
|
||||
ref_lat = vae.encode(ref_px)
|
||||
overlay = _draw_crop_overlay(img1[:1], crop_box)
|
||||
return ref_lat, ref_px.clone(), overlay, crop_box, src_w0, src_h0
|
||||
|
||||
ref_specs, ref_previews, crop_overlays = [], [], []
|
||||
for i in range(n_refs):
|
||||
ref_lat_i, ref_px_i, overlay_i, crop_box, src_w0, src_h0 = _encode_one(reference_image[i:i + 1])
|
||||
ref_specs.append({"latent": ref_lat_i, "seg_value": (float(source_id) + i) * float(phase_scale),
|
||||
"layout": layout, "strata_slot": i,
|
||||
"temporal_offset_latents": int(reference_temporal_offset_latents)})
|
||||
ref_previews.append(ref_px_i)
|
||||
crop_overlays.append(overlay_i)
|
||||
ref_lat = ref_specs[0]["latent"] # for the debug string below (1st ref's shape)
|
||||
ref_preview = torch.cat(ref_previews, dim=0)
|
||||
crop_overlay = torch.cat(crop_overlays, dim=0)
|
||||
|
||||
_install_patches(ltxv)
|
||||
ltxv._id_rope_theta = 10000.0
|
||||
m.model_options = dict(m.model_options)
|
||||
to = dict(m.model_options.get("transformer_options", {}))
|
||||
to["_id_ref_specs"] = ref_specs
|
||||
m.model_options["transformer_options"] = to
|
||||
|
||||
if reference_guidance_scale != 1.0:
|
||||
# ST-DRC reference-CFG: a third forward pass per step with the reference tokens
|
||||
# dropped isolates the reference's own contribution, the same way CFG isolates
|
||||
# the text prompt's (arxiv 2606.02441). `args["input_cond"]` is the exact
|
||||
# conditioning list KSampler passed for the positive/"cond" branch (same prompt,
|
||||
# same batching) -- reuse it unchanged, only strip `_id_ref_specs` from the
|
||||
# model_options used for THIS extra call so process_input sees no reference.
|
||||
noref_to = dict(to)
|
||||
noref_to.pop("_id_ref_specs", None)
|
||||
ref_scale = float(reference_guidance_scale)
|
||||
|
||||
# NOTE: must take exactly ONE parameter -- set_model_sampler_cfg_function()
|
||||
# inspects the signature's parameter COUNT (regardless of defaults) and treats
|
||||
# anything with 3 params as the legacy (cond, uncond, cond_scale) calling
|
||||
# convention, silently passing those three tensors/floats positionally instead
|
||||
# of the `args` dict. Keep noref_to/ref_scale as plain closure vars, not defaults.
|
||||
def _ref_cfg_function(args):
|
||||
cond = args["cond"]
|
||||
uncond = args["uncond"]
|
||||
cond_scale = args["cond_scale"]
|
||||
denoised = uncond + (cond - uncond) * cond_scale
|
||||
noref_model_options = dict(args["model_options"])
|
||||
noref_model_options["transformer_options"] = noref_to
|
||||
(noref_pred,) = comfy.samplers.calc_cond_batch(
|
||||
args["model"], [args["input_cond"]], args["input"], args["timestep"], noref_model_options,
|
||||
)
|
||||
noref_denoised = args["input"] - noref_pred
|
||||
denoised = denoised + (ref_scale - 1.0) * (cond - noref_denoised)
|
||||
return denoised
|
||||
|
||||
m.set_model_sampler_cfg_function(_ref_cfg_function, disable_cfg1_optimization=True)
|
||||
|
||||
seg_list = ", ".join(f"#{i}={s['seg_value']:g}" for i, s in enumerate(ref_specs))
|
||||
temporal_pixel_offset = reference_temporal_pixel_offset(
|
||||
reference_temporal_offset_latents,
|
||||
vae.downscale_index_formula[0],
|
||||
)
|
||||
dbg = (
|
||||
"=== LTX Identity OVERLAP (exact) ===\n"
|
||||
f"detected model: LTX-{detected_ltx_version} "
|
||||
f"(architecture detection via keyframes_abs_pos_embedding)\n"
|
||||
f"references: {n_refs} (encoded at {ref_preview.shape[2]}x{ref_preview.shape[1]}px each, "
|
||||
f"mode={ref_resize_mode}{f', crop_anchor={crop_anchor}' if ref_resize_mode == 'match_target' else ''}) "
|
||||
f"-> {layout} tokens, source_phase seg per ref: {seg_list}\n"
|
||||
f"reference temporal offset: {int(reference_temporal_offset_latents)} latent step(s) "
|
||||
f"= {temporal_pixel_offset} pixel frame(s) (reference only)\n"
|
||||
f"patches on {type(ltxv).__name__}: process_input/prepare_timestep/prepare_pe/process_output\n"
|
||||
f"crop preview: kept region {crop_box[2]}x{crop_box[3]}px of the {src_w0}x{src_h0}px reference "
|
||||
"-- see the ref_preview/crop_overlay IMAGE outputs to inspect what gets kept vs discarded.\n"
|
||||
f"reference-CFG: {'off' if reference_guidance_scale == 1.0 else f'ON, scale={reference_guidance_scale} (+1 forward pass/step)'}\n"
|
||||
"Set LTX_IDOVERLAP_DEBUG=1 for per-step shape logs. Connect negative + CFG 3-5, no LightX2V."
|
||||
)
|
||||
log.info("\n" + dbg)
|
||||
# pass the latent through unchanged (the ref is injected inside the model, not here)
|
||||
# so the graph can chain Empty -> this node -> sampler without branching.
|
||||
return (m, positive, negative, latent, dbg, ref_preview, crop_overlay)
|
||||
|
||||
|
||||
# Public node id + display name. Keep the old key as an alias so existing workflows load.
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"LTXIdentityTransfer": LTXIdentityOverlapConditioning,
|
||||
"LTXIdentityOverlapConditioning": LTXIdentityOverlapConditioning,
|
||||
}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"LTXIdentityTransfer": "LTX Identity Transfer",
|
||||
"LTXIdentityOverlapConditioning": "LTX Identity Transfer",
|
||||
}
|
||||
@@ -0,0 +1,308 @@
|
||||
"""LTX Multiple Controls — single-node version of chaining several LTXIdentityOverlapConditioning
|
||||
(LTX Identity Transfer) calls for a guide (motion/structure) + mask (replacement region) +
|
||||
identity (appearance) recipe, e.g. character-replace-in-driving-video training.
|
||||
|
||||
WHY THIS NODE EXISTS: LTXIdentityOverlapConditioning stores its reference specs in
|
||||
model.model_options["transformer_options"]["_id_ref_specs"], and each call OVERWRITES that key
|
||||
with its own specs list rather than merging with whatever the upstream node already put there.
|
||||
Chaining N of those nodes therefore only keeps the LAST node's reference active at inference —
|
||||
the earlier ones (e.g. the guide) silently have ZERO effect despite the graph looking correct.
|
||||
This node builds every slot's ref_specs in ONE apply() call so they all end up in the same list.
|
||||
|
||||
Each slot (guide/mask/identity/identity_mask) is fully independent: its own source_id,
|
||||
phase_scale, layout, ref_resize_mode -- no slot is hardcoded to a fixed layout/phase, only
|
||||
given sane defaults that match the CURRENT trained recipe (scail2v2_r128_v13): guide=0 and
|
||||
mask=0 (both => no RoPE phase tag, same as source_phase=false at train time -- they must stay
|
||||
positionally overlapped with the target frame-by-frame); identity=2 (=> tagged, same as
|
||||
source_phase=true, since it has no positional correspondence with the target). identity_mask
|
||||
is unused by v13 (ref_mask was dropped from the recipe entirely) -- leave it unconnected.
|
||||
|
||||
v13's key fix vs earlier recipes: `guide_video` must have the replacement region painted BLACK
|
||||
(an inpainting hole) before the model sees it, otherwise it just copies guide's own person into
|
||||
the masked region and never uses `identity_image` at all -- confirmed as the root cause of a
|
||||
long stretch of "identity ignored" results earlier in training. This node does that
|
||||
automatically (`auto_mask_guide`, on by default) using `mask_video`'s white region -- do NOT
|
||||
also pre-mask guide upstream or the hole will double-apply harmlessly but wastefully.
|
||||
|
||||
mask_video is now a plain WHITE-silhouette-on-black binary mask (NOT colored) -- an earlier
|
||||
per-identity color-coded mask design caused the model to leak the mask's own hue into the
|
||||
generated clothing (confirmed via direct pixel inspection during training); white/black avoids
|
||||
that since it carries no color information for the VAE to latch onto.
|
||||
"""
|
||||
import logging
|
||||
|
||||
import torch
|
||||
|
||||
from . import ltx_identity_overlap as _ido
|
||||
from .ltx_identity_overlap import (
|
||||
_anchored_crop_resize,
|
||||
_draw_crop_overlay,
|
||||
_find_ltxv,
|
||||
_install_patches,
|
||||
_letterbox_resize,
|
||||
)
|
||||
from .reference_temporal_offset import (
|
||||
DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
|
||||
reference_temporal_offset_input,
|
||||
)
|
||||
|
||||
log = logging.getLogger("LTXMultipleControls")
|
||||
|
||||
_LAYOUT_CHOICES = ["overlap", "st_drc", "strata", "sidecar"]
|
||||
_RESIZE_CHOICES = ["match_target", "match_target_letterbox", "native_resolution"]
|
||||
|
||||
|
||||
def _encode_ref(vae, latent, img, ref_resize_mode, crop_anchor, w_sf, h_sf, downscale_factor=1):
|
||||
"""Resize (per ref_resize_mode/crop_anchor) + VAE-encode ONE reference IMAGE batch
|
||||
([N,H,W,C], typically a whole video's frames). Mirrors LTXIdentityOverlapConditioning's
|
||||
own _encode_one so behavior is byte-identical per slot.
|
||||
|
||||
downscale_factor>1: encode at target_size/downscale_factor instead of full target size
|
||||
(mirrors the trainer's reference_downscale_factor -- fewer tokens/cheaper VRAM for a
|
||||
condition that doesn't need full detail, e.g. guide/mask). Snapped to a multiple of
|
||||
w_sf/h_sf (must stay divisible by the VAE's own spatial downscale, same constraint as
|
||||
training's "must be divisible by 32" check)."""
|
||||
import comfy.utils
|
||||
|
||||
if ref_resize_mode == "native_resolution":
|
||||
_, src_h, src_w, _ = img.shape
|
||||
tgt_w = max(w_sf, round(src_w / w_sf) * w_sf)
|
||||
tgt_h = max(h_sf, round(src_h / h_sf) * h_sf)
|
||||
else:
|
||||
_, _, _, lat_h, lat_w = latent["samples"].shape
|
||||
tgt_w, tgt_h = lat_w * w_sf, lat_h * h_sf
|
||||
if downscale_factor != 1:
|
||||
tgt_w = max(w_sf, round(tgt_w / downscale_factor / w_sf) * w_sf)
|
||||
tgt_h = max(h_sf, round(tgt_h / downscale_factor / h_sf) * h_sf)
|
||||
_, src_h0, src_w0, _ = img.shape
|
||||
crop_box = (0, 0, src_w0, src_h0)
|
||||
if ref_resize_mode == "match_target_letterbox":
|
||||
ref_px = _letterbox_resize(img, tgt_w, tgt_h)[:, :, :, :3]
|
||||
elif ref_resize_mode == "match_target" and crop_anchor != "center":
|
||||
ref_px, crop_box = _anchored_crop_resize(img, tgt_w, tgt_h, anchor=crop_anchor)
|
||||
ref_px = ref_px[:, :, :, :3]
|
||||
else:
|
||||
ref_px = comfy.utils.common_upscale(img.movedim(-1, 1), tgt_w, tgt_h, "bilinear", "center").movedim(1, -1)[:, :, :, :3]
|
||||
if ref_resize_mode == "match_target":
|
||||
_, crop_box = _anchored_crop_resize(img[:1], tgt_w, tgt_h, anchor="center")
|
||||
ref_lat = vae.encode(ref_px)
|
||||
overlay = _draw_crop_overlay(img[:1], crop_box)
|
||||
return ref_lat, ref_px[:1].clone(), overlay, crop_box, src_w0, src_h0
|
||||
|
||||
|
||||
class LTXMultipleControls:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {
|
||||
"model": ("MODEL",),
|
||||
"positive": ("CONDITIONING",),
|
||||
"negative": ("CONDITIONING",),
|
||||
"vae": ("VAE",),
|
||||
"latent": ("LATENT",),
|
||||
}, "optional": {
|
||||
"guide_video": ("IMAGE", {"tooltip": "Motion/structure driving video frames (IMAGE batch, e.g. from "
|
||||
"GetVideoComponents). Leave unconnected to skip the guide slot. Gets auto-masked "
|
||||
"black in the replacement region (see auto_mask_guide) unless disabled."}),
|
||||
"guide_source_id": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 8.0, "step": 1.0,
|
||||
"tooltip": "v13 recipe default: 0 (no RoPE phase tag at all -- must stay "
|
||||
"source_phase=false so the guide's positions line up with the target "
|
||||
"frame-by-frame). Non-zero only for older/experimental checkpoints."}),
|
||||
"guide_phase_scale": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.1}),
|
||||
"guide_layout": (_LAYOUT_CHOICES, {"default": "overlap"}),
|
||||
"guide_ref_resize_mode": (_RESIZE_CHOICES, {"default": "match_target"}),
|
||||
"guide_downscale_factor": ("INT", {"default": 1, "min": 1, "max": 8,
|
||||
"tooltip": "Encode at target_size/N instead of full res (cheaper VRAM). Must match "
|
||||
"whatever the checkpoint was trained with -- 1 for the no-downscale "
|
||||
"recipes, 2 for the earlier downscaled-guide recipes. Wrong value = "
|
||||
"positions the model never learned, not just a quality hit."}),
|
||||
|
||||
"mask_video": ("IMAGE", {"tooltip": "Per-frame replacement-region mask, same frame count/alignment "
|
||||
"as guide_video -- v13 expects a plain WHITE-on-black binary silhouette (person "
|
||||
"white, background black). Do NOT use a colored mask (see Color Mask node) -- "
|
||||
"confirmed to leak hue into the generated clothing during training. Also drives "
|
||||
"auto_mask_guide's black-out of guide_video. Leave unconnected to skip."}),
|
||||
"mask_source_id": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 8.0, "step": 1.0,
|
||||
"tooltip": "v13 recipe default: 0 (same non-tagged group as guide -- both need exact "
|
||||
"positional correspondence with the target). Non-zero only for older/"
|
||||
"experimental checkpoints."}),
|
||||
"mask_phase_scale": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.1}),
|
||||
"mask_layout": (_LAYOUT_CHOICES, {"default": "overlap"}),
|
||||
"mask_ref_resize_mode": (_RESIZE_CHOICES, {"default": "match_target"}),
|
||||
"mask_downscale_factor": ("INT", {"default": 1, "min": 1, "max": 8,
|
||||
"tooltip": "Same as guide_downscale_factor -- match the checkpoint's training recipe."}),
|
||||
"auto_mask_guide": ("BOOLEAN", {"default": True,
|
||||
"tooltip": "v13 requirement: paints guide_video BLACK wherever mask_video is white, "
|
||||
"before encoding -- an inpainting hole, so the model can't just copy "
|
||||
"guide's own person into the replacement region and must pull identity "
|
||||
"from identity_image instead (matches how the training data was built). "
|
||||
"Turn off only if guide_video is already pre-masked upstream."}),
|
||||
"mask_white_thresh": ("FLOAT", {"default": 0.5, "min": 0.0, "max": 1.0, "step": 0.01,
|
||||
"tooltip": "auto_mask_guide only -- mask_video pixels with mean brightness >= this "
|
||||
"(0-1 scale) count as foreground (person) and get blacked out in guide."}),
|
||||
|
||||
"identity_image": ("IMAGE", {"tooltip": "Appearance reference (face/character), no positional "
|
||||
"correspondence with the target needed -- this is what fills auto_mask_guide's hole. "
|
||||
"Leave unconnected to skip."}),
|
||||
"identity_source_id": ("FLOAT", {"default": 2.0, "min": 0.0, "max": 8.0, "step": 1.0,
|
||||
"tooltip": "v13 recipe default: 2. Nonzero = tagged with its own RoPE phase (matches "
|
||||
"source_phase=true, since identity has no positional correspondence with "
|
||||
"the target)."}),
|
||||
"identity_phase_scale": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.1}),
|
||||
"identity_layout": (_LAYOUT_CHOICES, {"default": "overlap"}),
|
||||
"identity_ref_resize_mode": (_RESIZE_CHOICES, {"default": "native_resolution"}),
|
||||
"identity_downscale_factor": ("INT", {"default": 1, "min": 1, "max": 8,
|
||||
"tooltip": "Same as guide_downscale_factor -- match the checkpoint's training recipe."}),
|
||||
"identity_mask_image": ("IMAGE", {"tooltip": "scail2-style color-pointer marker for the identity "
|
||||
"slot (small flat color dot/blob, NOT a body silhouette -- a body-shaped mask here "
|
||||
"would re-inject a competing pose signal, exactly the bug the flat-marker design "
|
||||
"fixes on the training side). Same source_id/phase as identity_image by default "
|
||||
"(they're trained as one group) -- override identity_mask_source_id if your "
|
||||
"checkpoint used a different grouping. Leave unconnected to skip."}),
|
||||
"identity_mask_source_id": ("FLOAT", {"default": -1.0, "min": -1.0, "max": 8.0, "step": 1.0,
|
||||
"tooltip": "-1 = inherit identity_source_id/identity_phase_scale (matches training, "
|
||||
"where ref+ref_mask share one group). Set >=0 to give it its own separate phase."}),
|
||||
"identity_mask_phase_scale": ("FLOAT", {"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.1}),
|
||||
"identity_mask_layout": (_LAYOUT_CHOICES, {"default": "overlap"}),
|
||||
"identity_mask_ref_resize_mode": (_RESIZE_CHOICES, {"default": "native_resolution"}),
|
||||
"identity_mask_downscale_factor": ("INT", {"default": -1, "min": -1, "max": 8,
|
||||
"tooltip": "-1 = inherit identity_downscale_factor. Same meaning otherwise as "
|
||||
"guide_downscale_factor -- match the checkpoint's training recipe."}),
|
||||
|
||||
"crop_anchor": (["center", "top", "bottom", "left", "right"], {"default": "center",
|
||||
"tooltip": "Shared by all slots using ref_resize_mode=match_target with a mismatched "
|
||||
"aspect ratio -- which part of the source survives the crop."}),
|
||||
"reference_guidance_scale": ("FLOAT", {"default": 1.0, "min": 1.0, "max": 10.0, "step": 0.1,
|
||||
"tooltip": "ST-DRC-style reference-CFG applied to the WHOLE combined reference set "
|
||||
"(all active slots together). 1.0 = off."}),
|
||||
"debug_log": ("BOOLEAN", {"default": False}),
|
||||
# Reference-only despite being appended here for saved-workflow widget stability.
|
||||
"identity_temporal_offset_latents": reference_temporal_offset_input(),
|
||||
}}
|
||||
|
||||
RETURN_TYPES = ("MODEL", "CONDITIONING", "CONDITIONING", "LATENT", "STRING")
|
||||
RETURN_NAMES = ("model", "positive", "negative", "latent", "debug")
|
||||
FUNCTION = "apply"
|
||||
CATEGORY = "LTX/identity"
|
||||
DESCRIPTION = ("Guide + mask + identity + identity_mask reference conditioning in ONE node call, so all "
|
||||
"active slots' RoPE reference specs actually combine (chaining LTXIdentityOverlapConditioning "
|
||||
"nodes does NOT -- each call overwrites the previous one's specs). Every slot is optional and "
|
||||
"its own source_id/phase_scale/layout/ref_resize_mode -- nothing is hardcoded, only defaulted "
|
||||
"to match scail2v2_r128_v13 (guide=0, mask=0, both no-phase; identity=2, phased). guide_video "
|
||||
"is auto-masked black in the replacement region (auto_mask_guide) to match how v13 was "
|
||||
"trained -- this is the fix that made identity transfer actually work. identity_mask is "
|
||||
"unused by v13 (leave unconnected); kept for older checkpoints.")
|
||||
|
||||
def apply(self, model, positive, negative, vae, latent,
|
||||
guide_video=None, guide_source_id=0.0, guide_phase_scale=1.0,
|
||||
guide_layout="overlap", guide_ref_resize_mode="match_target", guide_downscale_factor=1,
|
||||
mask_video=None, mask_source_id=0.0, mask_phase_scale=1.0,
|
||||
mask_layout="overlap", mask_ref_resize_mode="match_target", mask_downscale_factor=1,
|
||||
auto_mask_guide=True, mask_white_thresh=0.5,
|
||||
identity_image=None, identity_source_id=2.0, identity_phase_scale=1.0,
|
||||
identity_layout="overlap", identity_ref_resize_mode="native_resolution", identity_downscale_factor=1,
|
||||
identity_temporal_offset_latents=DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
|
||||
identity_mask_image=None, identity_mask_source_id=-1.0, identity_mask_phase_scale=1.0,
|
||||
identity_mask_layout="overlap", identity_mask_ref_resize_mode="native_resolution",
|
||||
identity_mask_downscale_factor=-1,
|
||||
crop_anchor="center", reference_guidance_scale=1.0, debug_log=False):
|
||||
import comfy.samplers
|
||||
import comfy.utils
|
||||
|
||||
_ido._DEBUG_ENABLED = bool(debug_log)
|
||||
m = model.clone()
|
||||
ltxv = _find_ltxv(m)
|
||||
_, w_sf, h_sf = vae.downscale_index_formula
|
||||
|
||||
if guide_video is not None and mask_video is not None and auto_mask_guide:
|
||||
mg = mask_video
|
||||
if mg.shape[1:3] != guide_video.shape[1:3]:
|
||||
mg = comfy.utils.common_upscale(
|
||||
mg.movedim(-1, 1), guide_video.shape[2], guide_video.shape[1], "nearest-exact", "center",
|
||||
).movedim(1, -1)
|
||||
if mg.shape[0] != guide_video.shape[0]:
|
||||
idx = torch.linspace(0, mg.shape[0] - 1, guide_video.shape[0]).round().long().clamp(max=mg.shape[0] - 1)
|
||||
mg = mg[idx]
|
||||
fg = mg.mean(dim=-1, keepdim=True) >= mask_white_thresh
|
||||
guide_video = torch.where(fg, torch.zeros_like(guide_video), guide_video)
|
||||
|
||||
# -1 = inherit the identity slot's own grouping (matches training: ref+ref_mask share
|
||||
# one source_id/phase_scale by default).
|
||||
if identity_mask_source_id < 0:
|
||||
identity_mask_source_id = identity_source_id
|
||||
identity_mask_phase_scale = identity_phase_scale
|
||||
if identity_mask_downscale_factor < 0:
|
||||
identity_mask_downscale_factor = identity_downscale_factor
|
||||
|
||||
slots = [
|
||||
("guide", guide_video, guide_source_id, guide_phase_scale, guide_layout, guide_ref_resize_mode, guide_downscale_factor, 0),
|
||||
("mask", mask_video, mask_source_id, mask_phase_scale, mask_layout, mask_ref_resize_mode, mask_downscale_factor, 0),
|
||||
("identity", identity_image, identity_source_id, identity_phase_scale, identity_layout,
|
||||
identity_ref_resize_mode, identity_downscale_factor, identity_temporal_offset_latents),
|
||||
("identity_mask", identity_mask_image, identity_mask_source_id, identity_mask_phase_scale,
|
||||
identity_mask_layout, identity_mask_ref_resize_mode, identity_mask_downscale_factor, 0),
|
||||
]
|
||||
|
||||
ref_specs = []
|
||||
summary = []
|
||||
for name, img, source_id, phase_scale, layout, resize_mode, downscale_factor, temporal_offset in slots:
|
||||
if img is None:
|
||||
continue
|
||||
ref_lat, _px, _overlay, crop_box, src_w0, src_h0 = _encode_ref(
|
||||
vae, latent, img, resize_mode, crop_anchor, w_sf, h_sf, downscale_factor=downscale_factor)
|
||||
seg_value = float(source_id) * float(phase_scale)
|
||||
ref_specs.append({"latent": ref_lat, "seg_value": seg_value, "layout": layout,
|
||||
"strata_slot": len(ref_specs), "downscale_factor": downscale_factor,
|
||||
"temporal_offset_latents": int(temporal_offset)})
|
||||
dsf_note = f", downscale={downscale_factor}x" if downscale_factor != 1 else ""
|
||||
offset_note = f", ref_t_offset={int(temporal_offset)} latent" if name == "identity" else ""
|
||||
summary.append(
|
||||
f"{name}: {img.shape[0]}f {src_w0}x{src_h0}px -> {layout}, seg={seg_value:g}, "
|
||||
f"mode={resize_mode}{dsf_note}{offset_note}"
|
||||
)
|
||||
|
||||
if not ref_specs:
|
||||
log.warning("LTXMultipleControls: no slots connected -- passing through unchanged.")
|
||||
return (m, positive, negative, latent, "LTX Multiple Controls: no active slots (nothing connected).")
|
||||
|
||||
_install_patches(ltxv)
|
||||
ltxv._id_rope_theta = 10000.0
|
||||
m.model_options = dict(m.model_options)
|
||||
to = dict(m.model_options.get("transformer_options", {}))
|
||||
to["_id_ref_specs"] = ref_specs
|
||||
m.model_options["transformer_options"] = to
|
||||
|
||||
if reference_guidance_scale != 1.0:
|
||||
noref_to = dict(to)
|
||||
noref_to.pop("_id_ref_specs", None)
|
||||
ref_scale = float(reference_guidance_scale)
|
||||
|
||||
def _ref_cfg_function(args):
|
||||
cond = args["cond"]
|
||||
uncond = args["uncond"]
|
||||
cond_scale = args["cond_scale"]
|
||||
denoised = uncond + (cond - uncond) * cond_scale
|
||||
noref_model_options = dict(args["model_options"])
|
||||
noref_model_options["transformer_options"] = noref_to
|
||||
(noref_pred,) = comfy.samplers.calc_cond_batch(
|
||||
args["model"], [args["input_cond"]], args["input"], args["timestep"], noref_model_options,
|
||||
)
|
||||
noref_denoised = args["input"] - noref_pred
|
||||
denoised = denoised + (ref_scale - 1.0) * (cond - noref_denoised)
|
||||
return denoised
|
||||
|
||||
m.set_model_sampler_cfg_function(_ref_cfg_function, disable_cfg1_optimization=True)
|
||||
|
||||
guide_masked = guide_video is not None and mask_video is not None and auto_mask_guide
|
||||
dbg = (
|
||||
"=== LTX Multiple Controls ===\n"
|
||||
f"{len(ref_specs)} active slot(s):\n " + "\n ".join(summary) + "\n"
|
||||
f"auto_mask_guide: {'applied' if guide_masked else 'off/skipped'}\n"
|
||||
f"reference-CFG: {'off' if reference_guidance_scale == 1.0 else f'ON, scale={reference_guidance_scale}'}\n"
|
||||
"Set LTX_IDOVERLAP_DEBUG=1 for per-step shape logs. Connect negative + CFG 3-5, no LightX2V."
|
||||
)
|
||||
log.info("\n" + dbg)
|
||||
return (m, positive, negative, latent, dbg)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"LTXMultipleControls": LTXMultipleControls}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"LTXMultipleControls": "LTX Multiple Controls"}
|
||||
@@ -0,0 +1,321 @@
|
||||
"""LTX multi-subject references with learned slot tags.
|
||||
|
||||
Three nodes:
|
||||
|
||||
* **LTX Reference Images** — collects up to six reference images into one ordered list, so the
|
||||
slot order is visible in one place and references of different sizes travel together.
|
||||
* **LTX Multi Reference Slots** — up to six reference images, each becoming its own reference
|
||||
block. Every block gets a distinct ``source_id`` (rotary phase) and, when the checkpoint
|
||||
carries one, the trained per-slot embedding added in feature space.
|
||||
* **LTX Reference Tags** — builds the ``<Image 1> is the ...`` prompt prefix that binds each tag
|
||||
to its slot.
|
||||
|
||||
The prompt tags are the part people get wrong. Nothing in the model ties ``<Image 1>`` to slot 1:
|
||||
that correspondence exists only because the training captions used it consistently. Sampling with
|
||||
a different convention than training used simply does not bind, however well the slots are
|
||||
separated — hence the second node, so the prompt is built the same way every time.
|
||||
|
||||
The slot embedding lives in the LoRA's own safetensors under
|
||||
``diffusion_model.reference_slot_embedding.*`` with its hyperparameters in the file metadata, so
|
||||
it is read straight from the LoRA file: ComfyUI's LoRA loader only applies LoRA weights and drops
|
||||
everything else, which would silently sample the checkpoint without a signal it was trained on.
|
||||
"""
|
||||
|
||||
import math
|
||||
|
||||
import torch
|
||||
|
||||
MAX_REFERENCES = 6
|
||||
_PREFIX = "diffusion_model.reference_slot_embedding."
|
||||
|
||||
|
||||
class _SlotEmbedding(torch.nn.Module):
|
||||
"""Fourier features of the slot index through a small MLP. Mirrors the trainer's module —
|
||||
same parameter names and shapes, so a checkpoint loads without remapping."""
|
||||
|
||||
def __init__(self, token_dim=128, num_frequencies=16, hidden_dim=256):
|
||||
super().__init__()
|
||||
self.register_buffer("frequencies", 2.0 ** torch.arange(num_frequencies, dtype=torch.float32))
|
||||
self.net = torch.nn.Sequential(
|
||||
torch.nn.Linear(1 + 2 * num_frequencies, hidden_dim),
|
||||
torch.nn.SiLU(),
|
||||
torch.nn.Linear(hidden_dim, token_dim),
|
||||
)
|
||||
|
||||
def forward(self, slot_index):
|
||||
index = torch.tensor([float(slot_index)], dtype=torch.float32, device=self.frequencies.device)
|
||||
scaled = index * self.frequencies
|
||||
features = torch.cat([index, torch.sin(scaled), torch.cos(scaled)], dim=-1)
|
||||
return self.net(features.to(self.net[0].weight.dtype))
|
||||
|
||||
|
||||
def _load_slot_embedding(lora_name):
|
||||
"""Rebuild the slot embedding from a LoRA file, or None when it carries none."""
|
||||
import folder_paths
|
||||
from safetensors import safe_open
|
||||
|
||||
path = folder_paths.get_full_path("loras", lora_name)
|
||||
if path is None:
|
||||
raise ValueError(f"LoRA not found: {lora_name}")
|
||||
|
||||
with safe_open(path, framework="pt") as f:
|
||||
keys = [k for k in f.keys() if k.startswith(_PREFIX)]
|
||||
if not keys:
|
||||
return None
|
||||
metadata = f.metadata() or {}
|
||||
state = {k[len(_PREFIX):]: f.get_tensor(k).float() for k in keys}
|
||||
|
||||
# Prefer the file's own metadata; fall back to the shapes when it is absent.
|
||||
hidden_dim = int(metadata.get("reference_slot_embedding_hidden_dim", state["net.0.weight"].shape[0]))
|
||||
num_frequencies = int(metadata.get("reference_slot_embedding_num_frequencies", 0)) or (
|
||||
(state["net.0.weight"].shape[1] - 1) // 2
|
||||
)
|
||||
token_dim = int(metadata.get("reference_slot_embedding_dim", state["net.2.weight"].shape[0]))
|
||||
|
||||
module = _SlotEmbedding(token_dim=token_dim, num_frequencies=num_frequencies, hidden_dim=hidden_dim)
|
||||
module.load_state_dict(state)
|
||||
module.eval()
|
||||
return module
|
||||
|
||||
|
||||
class LTXReferenceImages:
|
||||
"""Collect reference images into an ordered list. Images only — nothing else wired in.
|
||||
|
||||
A plain IMAGE batch cannot carry references of different sizes (torch stacking needs one
|
||||
shape), and subject references routinely differ: a portrait crop next to a landscape product
|
||||
shot. Emitting a list keeps each one at its own size, and keeps the slot order explicit and
|
||||
visible on one node instead of spread across the conditioning node's inputs.
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {},
|
||||
"optional": {
|
||||
f"image_{i}": (
|
||||
"IMAGE",
|
||||
{"tooltip": f"Reference for slot {i} — the subject your prompt calls <Image {i}>. "
|
||||
f"Leave unconnected to skip; connected slots renumber in order, so "
|
||||
f"wiring 1 and 3 makes them <Image 1> and <Image 2>."},
|
||||
)
|
||||
for i in range(1, MAX_REFERENCES + 1)
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("LTX_REFS", "INT")
|
||||
RETURN_NAMES = ("references", "count")
|
||||
FUNCTION = "collect"
|
||||
CATEGORY = "BFS/LTX"
|
||||
DESCRIPTION = (
|
||||
"Collects up to six reference images, in slot order, for LTX Multi Reference Slots. "
|
||||
"Images of different sizes are fine — each keeps its own."
|
||||
)
|
||||
|
||||
def collect(self, **images):
|
||||
refs = [
|
||||
images[f"image_{i}"]
|
||||
for i in range(1, MAX_REFERENCES + 1)
|
||||
if images.get(f"image_{i}") is not None
|
||||
]
|
||||
return (refs, len(refs))
|
||||
|
||||
|
||||
class LTXReferenceTags:
|
||||
"""Build the prompt prefix that binds each tag to its reference slot."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
optional = {
|
||||
f"subject_{i}": (
|
||||
"STRING",
|
||||
{"default": "", "multiline": False,
|
||||
"tooltip": f"What reference image {i} is, as a short noun phrase — 'the woman', "
|
||||
f"'the red sports car'. Becomes '<Image {i}> is <this>.'. Leave empty to skip."},
|
||||
)
|
||||
for i in range(1, MAX_REFERENCES + 1)
|
||||
}
|
||||
optional["scene"] = (
|
||||
"STRING",
|
||||
{"default": "", "multiline": True,
|
||||
"tooltip": "The scene description, appended after the declarations. Refer back to the "
|
||||
"subjects by tag here (\"<Image 1> hands <Image 2> a cup\") — a tag that only "
|
||||
"ever appears in the declaration has to carry both jobs at once, which is a "
|
||||
"harder association for the model to use."},
|
||||
)
|
||||
return {"required": {}, "optional": optional}
|
||||
|
||||
RETURN_TYPES = ("STRING",)
|
||||
RETURN_NAMES = ("prompt",)
|
||||
FUNCTION = "build"
|
||||
CATEGORY = "BFS/LTX"
|
||||
DESCRIPTION = (
|
||||
"Assembles '<Image 1> is the woman. <Image 2> is the man. <scene>'. Order matters: tag N "
|
||||
"must describe the image wired into reference_N on the slots node, and must match the "
|
||||
"convention the checkpoint was trained with. Describe the subjects DISTINCTLY — two "
|
||||
"references both called 'the woman' carry no information about which is which."
|
||||
)
|
||||
|
||||
def build(self, scene="", **subjects):
|
||||
parts = []
|
||||
for i in range(1, MAX_REFERENCES + 1):
|
||||
text = (subjects.get(f"subject_{i}") or "").strip().rstrip(".")
|
||||
if text:
|
||||
parts.append(f"<Image {i}> is {text}.")
|
||||
scene = (scene or "").strip()
|
||||
if scene:
|
||||
parts.append(scene)
|
||||
return (" ".join(parts),)
|
||||
|
||||
|
||||
class LTXMultiReferenceSlots:
|
||||
"""N reference images, each on its own slot: distinct rotary phase plus the trained tag."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
import folder_paths
|
||||
|
||||
optional = {}
|
||||
for i in range(1, MAX_REFERENCES + 1):
|
||||
optional[f"reference_{i}"] = (
|
||||
"IMAGE",
|
||||
{"tooltip": f"Reference image for slot {i} — the subject your prompt calls "
|
||||
f"<Image {i}>. Leave unconnected to skip; connected slots are numbered "
|
||||
f"in order, so wiring 1 and 3 gives them source ids 1 and 2."},
|
||||
)
|
||||
optional["references"] = (
|
||||
"LTX_REFS",
|
||||
{"tooltip": "Output of LTX Reference Images. Use this OR the individual reference_N "
|
||||
"slots below — the list wins if both are connected."},
|
||||
)
|
||||
optional["slot_embedding_lora"] = (
|
||||
["none"] + folder_paths.get_filename_list("loras"),
|
||||
{"default": "none",
|
||||
"tooltip": "The LoRA file to read the trained slot embedding from. Point this at the "
|
||||
"SAME LoRA you load for sampling. ComfyUI's LoRA loader applies only LoRA "
|
||||
"weights and drops the slot embedding, so without this the references are "
|
||||
"untagged and the checkpoint is sampled without a signal it was trained on. "
|
||||
"'none' = phase separation only (older checkpoints)."},
|
||||
)
|
||||
optional["start_source_id"] = (
|
||||
"INT",
|
||||
{"default": 1, "min": 0, "max": 8,
|
||||
"tooltip": "Source id of the first connected reference; the rest count up. The target "
|
||||
"is always 0. Start at 1 unless the checkpoint was trained otherwise."},
|
||||
)
|
||||
optional["phase_scale"] = ("FLOAT", {"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.1})
|
||||
optional["slot_strength"] = (
|
||||
"FLOAT",
|
||||
{"default": 1.0, "min": 0.0, "max": 4.0, "step": 0.05,
|
||||
"tooltip": "Scales the learned slot tag. The tag and the adapter were trained "
|
||||
"together at strength 1.0, so raising the LoRA on the loader without "
|
||||
"raising this leaves them out of proportion — the adapter shouts while "
|
||||
"the tag that tells the references apart stays quiet. Match this to the "
|
||||
"LoRA strength you set on the loader (an undertrained LoRA often wants "
|
||||
"1.2-1.5, and the tag wants the same)."},
|
||||
)
|
||||
optional["layout"] = (
|
||||
["overlap", "st_drc", "strata"],
|
||||
{"default": "overlap",
|
||||
"tooltip": "Where the references sit in the RoPE grid. Must match training."},
|
||||
)
|
||||
optional["ref_resize_mode"] = (
|
||||
["match_target", "match_target_letterbox", "native_resolution"],
|
||||
{"default": "match_target_letterbox",
|
||||
"tooltip": "Letterbox is the safe default for subject references: it keeps the whole "
|
||||
"image, where match_target centre-crops and can cut a subject in half when "
|
||||
"the aspect ratios differ."},
|
||||
)
|
||||
optional["reference_temporal_offset_latents"] = ("INT", {"default": 0, "min": -8, "max": 8})
|
||||
return {
|
||||
"required": {
|
||||
"model": ("MODEL",),
|
||||
"positive": ("CONDITIONING",),
|
||||
"negative": ("CONDITIONING",),
|
||||
"vae": ("VAE",),
|
||||
"latent": ("LATENT",),
|
||||
},
|
||||
"optional": optional,
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("MODEL", "CONDITIONING", "CONDITIONING", "LATENT", "STRING")
|
||||
RETURN_NAMES = ("model", "positive", "negative", "latent", "info")
|
||||
FUNCTION = "apply"
|
||||
CATEGORY = "BFS/LTX"
|
||||
DESCRIPTION = (
|
||||
"Attaches up to six reference images as separate reference blocks, each with its own "
|
||||
"source id and, when the LoRA carries one, its trained slot embedding. Pair with LTX "
|
||||
"Reference Tags so the prompt names the slots the way training did."
|
||||
)
|
||||
|
||||
def apply(self, model, positive, negative, vae, latent, references=None,
|
||||
slot_embedding_lora="none", start_source_id=1, phase_scale=1.0, slot_strength=1.0,
|
||||
layout="overlap", ref_resize_mode="match_target_letterbox",
|
||||
reference_temporal_offset_latents=0, **slots):
|
||||
from .ltx_identity_overlap import _find_ltxv, _install_patches
|
||||
from .ltx_multiple_controls import _encode_ref
|
||||
|
||||
images = list(references) if references else [
|
||||
slots[f"reference_{i}"]
|
||||
for i in range(1, MAX_REFERENCES + 1)
|
||||
if slots.get(f"reference_{i}") is not None
|
||||
]
|
||||
if not images:
|
||||
return (model, positive, negative, latent, "no references connected — pass-through")
|
||||
|
||||
slot_module = None
|
||||
if slot_embedding_lora != "none":
|
||||
slot_module = _load_slot_embedding(slot_embedding_lora)
|
||||
if slot_module is None:
|
||||
raise ValueError(
|
||||
f"{slot_embedding_lora} carries no slot embedding "
|
||||
f"(no '{_PREFIX}*' keys). Set slot_embedding_lora to 'none' if this "
|
||||
"checkpoint was trained with phase separation only."
|
||||
)
|
||||
|
||||
m = model.clone()
|
||||
ltxv = _find_ltxv(m)
|
||||
w_sf, h_sf = ltxv.vae_scale_factors[2], ltxv.vae_scale_factors[1]
|
||||
|
||||
ref_specs = []
|
||||
lines = []
|
||||
for index, img in enumerate(images):
|
||||
source_id = int(start_source_id) + index
|
||||
ref_lat, _, _, _, _, _ = _encode_ref(
|
||||
vae, latent, img, ref_resize_mode, "center", w_sf, h_sf
|
||||
)
|
||||
spec = {
|
||||
"latent": ref_lat,
|
||||
"seg_value": float(source_id) * float(phase_scale),
|
||||
"layout": layout,
|
||||
"strata_slot": index,
|
||||
"temporal_offset_latents": int(reference_temporal_offset_latents),
|
||||
}
|
||||
if slot_module is not None:
|
||||
with torch.no_grad():
|
||||
spec["slot_vector"] = slot_module(source_id).detach() * float(slot_strength)
|
||||
ref_specs.append(spec)
|
||||
lines.append(f"<Image {index + 1}> -> source_id {source_id}")
|
||||
|
||||
_install_patches(ltxv)
|
||||
ltxv._id_rope_theta = 10000.0
|
||||
m.model_options = dict(m.model_options)
|
||||
transformer_options = dict(m.model_options.get("transformer_options", {}))
|
||||
transformer_options["_id_ref_specs"] = ref_specs
|
||||
m.model_options["transformer_options"] = transformer_options
|
||||
|
||||
tag = (f"trained slot embedding x{slot_strength:g}" if slot_module is not None
|
||||
else "phase only (no slot embedding)")
|
||||
info = f"{len(images)} references, {layout}, {tag}\n" + "\n".join(lines)
|
||||
return (m, positive, negative, latent, info)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"LTXReferenceImages": LTXReferenceImages,
|
||||
"LTXMultiReferenceSlots": LTXMultiReferenceSlots,
|
||||
"LTXReferenceTags": LTXReferenceTags,
|
||||
}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"LTXReferenceImages": "LTX Reference Images",
|
||||
"LTXMultiReferenceSlots": "LTX Multi Reference Slots",
|
||||
"LTXReferenceTags": "LTX Reference Tags",
|
||||
}
|
||||
@@ -0,0 +1,200 @@
|
||||
"""LTX-2.3 Multishot — prompt builder + ordered reference batch for the shotplan strata LoRA.
|
||||
|
||||
The multishot LoRA (`multishot_strata_r128_v1`, trained on Pensioner/shotplan keyframes) takes
|
||||
ONE text prompt for the whole clip, in the ShotPlan caption format:
|
||||
|
||||
<global preamble: setting, palette, lighting, camera grammar, mood>
|
||||
Shot 1: <shot size> <camera move> <what happens>
|
||||
Shot 2: ...
|
||||
|
||||
plus ONE reference keyframe per shot, stacked as strata slots. This node assembles both from a
|
||||
per-shot UI so you don't hand-write the format.
|
||||
|
||||
WIRING (no dedicated conditioning node is needed -- LTX Identity Transfer already does the right
|
||||
thing for this checkpoint):
|
||||
|
||||
references (IMAGE batch) -> LTX Identity Transfer with layout="strata", source_id=2,
|
||||
phase_scale=1.0
|
||||
prompt (STRING) -> your LTX text encode -> positive
|
||||
|
||||
LTX Identity Transfer builds each ref's spec as
|
||||
``seg_value = (source_id + i) * phase_scale`` and ``strata_slot = i`` over the image batch, so
|
||||
source_id=2 over 4 images reproduces the trained convention exactly: source_id 2/3/4/5 on strata
|
||||
slots 0/1/2/3. The batch ORDER IS THE SHOT ORDER -- the LoRA was trained with
|
||||
`shuffle_reference_slots: false`, so slot N means shot N and nothing else.
|
||||
|
||||
Shots are COMPACTED, not positional: if you fill image_1 and image_3 and leave image_2 empty,
|
||||
you get "Shot 1"/"Shot 2" on strata slots 0/1. Slot 0 must always be the opening shot -- the LoRA
|
||||
never saw a gap in the middle of the slot sequence.
|
||||
|
||||
Unfilled shots are padded with a flat mid-gray frame, matching the `blank.jpg` (RGB 128)
|
||||
placeholder the dataset builder used for samples with fewer than 4 shots -- the model was trained
|
||||
to read that as "this slot carries no information".
|
||||
|
||||
The dropdown vocabularies are the terms actually present in the 17,525 shot headers of
|
||||
`train_meta_16fps.json`, ordered by frequency (close-up 32%, medium shot 26%, medium close-up 20%
|
||||
...; static 65%, pan 16%, tracking 9.5% ...). Terms outside these lists are out of distribution
|
||||
for the LoRA.
|
||||
|
||||
Note the text encoder truncates at 1024 tokens and real ShotPlan captions already run long at 3-4
|
||||
shots -- keep each action line to a sentence or two, or the last shot's description gets cut.
|
||||
"""
|
||||
import logging
|
||||
|
||||
import torch
|
||||
|
||||
log = logging.getLogger("LTXMultishotPrompt")
|
||||
|
||||
MAX_SHOTS = 4
|
||||
|
||||
# Vocabularies EXTRACTED from the 12,470 `Shot N:` headers of Pensioner/shotplan
|
||||
# train_meta_16fps.json -- the header is the span between "Shot N:" and the start of the action
|
||||
# sentence, and these lists cover 98% (size) / 97% (move) of them. Percentages are of all headers.
|
||||
#
|
||||
# Do NOT count these terms over the whole caption: the action text says things like "A close-up
|
||||
# of the young man", which inflates close-up ~7x if the window isn't cut at the action boundary.
|
||||
SHOT_SIZES = [
|
||||
"Medium shot", # 37.4%
|
||||
"Medium close-up", # 27.3%
|
||||
"Wide shot", # 9.0%
|
||||
"Extreme close-up", # 8.0%
|
||||
"Medium long shot", # 7.4%
|
||||
"Close-up", # 4.5%
|
||||
"Long shot", # 1.7%
|
||||
"Medium wide shot", # 1.1%
|
||||
"Full shot", # 0.2%
|
||||
"(auto)",
|
||||
]
|
||||
# Modifiers COMBINE with a shot size ("over-the-shoulder close-up static"); they are not
|
||||
# alternatives to one. Emitted before the size, matching the most frequent header forms.
|
||||
SHOT_MODIFIERS = [
|
||||
"(none)",
|
||||
"Over-the-shoulder", # 4.8%
|
||||
"Low angle", # 3.1%
|
||||
"High angle", # 2.9%
|
||||
"POV", # 0.2%
|
||||
"Overhead", # 0.1%
|
||||
]
|
||||
CAMERA_MOVES = [
|
||||
"Static", # 55.1%
|
||||
"Tracking", # 10.1%
|
||||
"Pan", # 6.6%
|
||||
"Static / locked-off", # 5.1%
|
||||
"Pan right", # 4.2%
|
||||
"Pan left", # 3.6%
|
||||
"Handheld", # 3.3%
|
||||
"Slow push-in", # 2.7%
|
||||
"Tilt down", # 1.4%
|
||||
"Dolly-in", # 1.0%
|
||||
"Static with slight pan", # 0.8%
|
||||
"Tracking / follow", # 0.8%
|
||||
"Tilt up", # 0.7%
|
||||
"(auto)",
|
||||
]
|
||||
|
||||
|
||||
def _gray_like(ref: torch.Tensor) -> torch.Tensor:
|
||||
"""A single mid-gray frame shaped like `ref` ([1,H,W,C]) -- the dataset's blank.jpg is a
|
||||
flat RGB 128, i.e. 0.5 in ComfyUI's 0-1 float IMAGE convention."""
|
||||
return torch.full((1, ref.shape[1], ref.shape[2], ref.shape[3]), 0.5,
|
||||
dtype=ref.dtype, device=ref.device)
|
||||
|
||||
|
||||
def _resize_to(img: torch.Tensor, h: int, w: int) -> torch.Tensor:
|
||||
"""Bilinear-resize an IMAGE [N,H,W,C] to h x w so every slot can be stacked into one batch."""
|
||||
if img.shape[1] == h and img.shape[2] == w:
|
||||
return img
|
||||
import comfy.utils
|
||||
return comfy.utils.common_upscale(img.movedim(-1, 1), w, h, "bilinear", "center").movedim(1, -1)
|
||||
|
||||
|
||||
class LTXMultishotPrompt:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
required = {
|
||||
"preamble": ("STRING", {
|
||||
"multiline": True, "default": "",
|
||||
"tooltip": "Describes the WHOLE sequence -- setting, colour palette, lighting, "
|
||||
"overall camera grammar, mood. This is the paragraph that precedes "
|
||||
"'Shot 1:' in the training captions. Leave empty to emit only the "
|
||||
"shot blocks."}),
|
||||
}
|
||||
optional = {}
|
||||
for i in range(1, MAX_SHOTS + 1):
|
||||
optional[f"image_{i}"] = ("IMAGE", {
|
||||
"tooltip": f"Reference keyframe for shot {i} (strata slot {i - 1}, source_id "
|
||||
f"{i + 1}). Leave unconnected to end the sequence -- the slot is "
|
||||
f"padded with the trained mid-gray blank."})
|
||||
optional[f"shot_modifier_{i}"] = (SHOT_MODIFIERS, {"default": "(none)",
|
||||
"tooltip": "Combines WITH the shot size (e.g. 'Over-the-shoulder' + 'Close-up' "
|
||||
"-> 'Over-the-shoulder close-up'). Not a size on its own."})
|
||||
optional[f"shot_size_{i}"] = (SHOT_SIZES, {"default": "Medium shot"})
|
||||
optional[f"camera_move_{i}"] = (CAMERA_MOVES, {"default": "Static"})
|
||||
optional[f"action_{i}"] = ("STRING", {
|
||||
"multiline": True, "default": "",
|
||||
"tooltip": f"What happens in shot {i} -- subject, wardrobe, action, framing "
|
||||
f"details. One or two sentences; the encoder caps the whole prompt "
|
||||
f"at 1024 tokens."})
|
||||
return {"required": required, "optional": optional}
|
||||
|
||||
RETURN_TYPES = ("STRING", "IMAGE", "INT")
|
||||
RETURN_NAMES = ("prompt", "references", "num_shots")
|
||||
FUNCTION = "build"
|
||||
CATEGORY = "BFS/video"
|
||||
DESCRIPTION = ("Builds a ShotPlan-format multishot prompt plus the ordered, gray-padded "
|
||||
"reference batch for the multishot strata LoRA. Feed `references` into LTX "
|
||||
"Identity Transfer with layout='strata', source_id=2, phase_scale=1.0, and "
|
||||
"`prompt` into your LTX text encode.")
|
||||
|
||||
def build(self, preamble, **kw):
|
||||
shots, images = [], []
|
||||
for i in range(1, MAX_SHOTS + 1):
|
||||
img = kw.get(f"image_{i}")
|
||||
action = (kw.get(f"action_{i}") or "").strip()
|
||||
if img is None and not action:
|
||||
continue
|
||||
mod = kw.get(f"shot_modifier_{i}", "(none)")
|
||||
size = kw.get(f"shot_size_{i}", "(auto)")
|
||||
move = kw.get(f"camera_move_{i}", "(auto)")
|
||||
# Dataset form: "<modifier> <size> <move>", e.g. "Over-the-shoulder close-up static".
|
||||
# The size is lowercased after a modifier so the header reads as one phrase.
|
||||
if mod and mod != "(none)" and size != "(auto)":
|
||||
size = f"{mod} {size[0].lower()}{size[1:]}"
|
||||
elif mod and mod != "(none)":
|
||||
size = mod
|
||||
header = " ".join(p for p in (size, move) if p and p != "(auto)")
|
||||
body = " ".join(p for p in (header, action) if p)
|
||||
shots.append(f"Shot {len(shots) + 1}: {body}".rstrip())
|
||||
images.append(img)
|
||||
|
||||
if not shots:
|
||||
raise ValueError(
|
||||
"LTX Multishot Prompt: connect at least one image or write at least one action.")
|
||||
|
||||
parts = ([preamble.strip()] if preamble.strip() else []) + shots
|
||||
prompt = " ".join(parts)
|
||||
|
||||
# Every slot must share one resolution to stack into a single IMAGE batch; the first
|
||||
# connected reference sets it (that is the shot the model opens on).
|
||||
first = next((im for im in images if im is not None), None)
|
||||
if first is None:
|
||||
raise ValueError(
|
||||
"LTX Multishot Prompt: at least one image_N must be connected -- the LoRA is "
|
||||
"conditioned on keyframes, an action-only prompt has nothing to anchor to.")
|
||||
h, w = first.shape[1], first.shape[2]
|
||||
|
||||
batch = []
|
||||
for im in images:
|
||||
batch.append(_gray_like(first) if im is None else _resize_to(im[:1], h, w))
|
||||
# Pad the unused slots: the LoRA always sees 4 strata slots, blank == "no information".
|
||||
while len(batch) < MAX_SHOTS:
|
||||
batch.append(_gray_like(first))
|
||||
|
||||
references = torch.cat(batch, dim=0)
|
||||
log.info("[LTXMultishotPrompt] %d shot(s), refs %s, prompt %d chars",
|
||||
len(shots), tuple(references.shape), len(prompt))
|
||||
return (prompt, references, len(shots))
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"LTXMultishotPrompt": LTXMultishotPrompt}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"LTXMultishotPrompt": "LTX Multishot Prompt + Refs"}
|
||||
@@ -0,0 +1,34 @@
|
||||
"""LTX Trim First Frames — drop the first N frames of an IMAGE batch.
|
||||
|
||||
Palliative for the overlap reference "frame-0 leak": with layout=overlap the reference
|
||||
sits at the target's frame-0 RoPE positions, so the generated first frame tends to pull
|
||||
toward the reference image before the sequence follows the prompt. Dropping the first
|
||||
frame(s) removes that artifact from the final video.
|
||||
"""
|
||||
import torch
|
||||
|
||||
|
||||
class LTXTrimFirstFrames:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {"required": {
|
||||
"images": ("IMAGE",),
|
||||
"trim": ("INT", {"default": 1, "min": 0, "max": 16,
|
||||
"tooltip": "Number of leading frames to drop (the overlap frame-0 leak is usually 1 frame)."}),
|
||||
}}
|
||||
|
||||
RETURN_TYPES = ("IMAGE",)
|
||||
RETURN_NAMES = ("images",)
|
||||
FUNCTION = "apply"
|
||||
CATEGORY = "LTX/identity"
|
||||
DESCRIPTION = "Drop the first N frames of a video (IMAGE batch) — removes the overlap reference frame-0 leak."
|
||||
|
||||
def apply(self, images, trim=1):
|
||||
n = int(trim)
|
||||
if n <= 0 or images.shape[0] <= n:
|
||||
return (images,)
|
||||
return (images[n:],)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"LTXTrimFirstFrames": LTXTrimFirstFrames}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"LTXTrimFirstFrames": "LTX Trim First Frames"}
|
||||
+30
-11
@@ -11,15 +11,21 @@ Usage in ComfyUI:
|
||||
Load LoRA (standard) → LTXV Edit Anything Apply → KSampler
|
||||
"""
|
||||
|
||||
import types
|
||||
import logging
|
||||
import os
|
||||
import types
|
||||
|
||||
import folder_paths
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
import folder_paths
|
||||
from safetensors.torch import load_file, save_file
|
||||
|
||||
from .reference_temporal_offset import (
|
||||
DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
|
||||
reference_temporal_offset_input,
|
||||
reference_temporal_pixel_offset,
|
||||
)
|
||||
|
||||
logger = logging.getLogger("ltxv.editanything")
|
||||
|
||||
# ── transformer_options keys (must not collide with other nodes) ──────────────
|
||||
@@ -533,6 +539,8 @@ class LTXVEditAnythingApply:
|
||||
"default": False,
|
||||
"tooltip": "Print latent shapes and token counts to console.",
|
||||
}),
|
||||
# Reference-only; appended last to preserve saved-workflow widget indices.
|
||||
"ref_temporal_offset_latents": reference_temporal_offset_input(),
|
||||
},
|
||||
}
|
||||
|
||||
@@ -544,6 +552,7 @@ class LTXVEditAnythingApply:
|
||||
def apply(
|
||||
self, model, positive, negative, vae, latent, ref_image, lora_name, resize_mode="pad_to_fit",
|
||||
guide_frames=None, guide_strength=1.0, ref_strength=1.0,
|
||||
ref_temporal_offset_latents=DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
|
||||
role_strength=1.0, adaln_scale=1.0, enable_adaln=True, enable_role_embedding=True, debug=False,
|
||||
):
|
||||
import comfy.utils
|
||||
@@ -626,9 +635,9 @@ class LTXVEditAnythingApply:
|
||||
ref_px = canvas[:, :, :, :3]
|
||||
ref_lat = vae.encode(ref_px) # [1, 128, 1, lat_h, lat_w]
|
||||
|
||||
# Temporal position for ref: t=-1 slot → -time_sf pixel frames (BEFORE frame 0)
|
||||
# Training layout: [ref @ t=-8 | guide @ t=0..T | noisy_target @ t=0..T]
|
||||
ref_pixel_pos = -time_sf # = -8 for LTX VAE (one latent frame before frame 0)
|
||||
# Reference-only temporal placement. Default 0 matches the standard reference position;
|
||||
# negative values can test [ref before t=0 | guide/target @ t=0..T]. Guide remains at frame 0.
|
||||
ref_pixel_pos = reference_temporal_pixel_offset(ref_temporal_offset_latents, time_sf)
|
||||
|
||||
# Append ref first (entry[0] = appearance)
|
||||
# causal_fix=True matches training validation_sampler.py:418 (get_pixel_coords causal_fix=True)
|
||||
@@ -1157,8 +1166,8 @@ def _ea_install_ref_attn_modules(ltxv, context_dim, init_seed, init_from="attn2"
|
||||
"attn1" — copy attn1 weights
|
||||
"none" — random seeded init + zero out to_out (training only for "none" runs)
|
||||
"""
|
||||
from comfy.ldm.lightricks.model import CrossAttention
|
||||
import comfy.ops
|
||||
from comfy.ldm.lightricks.model import CrossAttention
|
||||
|
||||
if init_seed is not None and init_seed >= 0:
|
||||
cpu_state = torch.random.get_rng_state()
|
||||
@@ -1476,6 +1485,8 @@ class LTXVEditAnythingLoopingSampler:
|
||||
"tooltip": "Strength multiplier for the ref_attn LoRA deltas.",
|
||||
}),
|
||||
"debug_ea": ("BOOLEAN", {"default": False}),
|
||||
# Reference-only; appended last to preserve saved-workflow widget indices.
|
||||
"ref_temporal_offset_latents": reference_temporal_offset_input(),
|
||||
},
|
||||
}
|
||||
|
||||
@@ -1520,7 +1531,7 @@ class LTXVEditAnythingLoopingSampler:
|
||||
def _build_chunk_conds(
|
||||
positive, negative, chunk_latent, chunk_mask,
|
||||
ref_lat, guide_chunk_lat, scale_factors,
|
||||
ref_strength, guide_strength, enable_ic_lora,
|
||||
ref_strength, guide_strength, ref_temporal_offset_latents, enable_ic_lora,
|
||||
):
|
||||
"""Append ref + guide_chunk to chunk's positive/negative; return updated tuple."""
|
||||
import comfy_extras.nodes_lt as nodes_lt
|
||||
@@ -1530,10 +1541,11 @@ class LTXVEditAnythingLoopingSampler:
|
||||
time_sf = scale_factors[0]
|
||||
appended_latent_frames = 0
|
||||
|
||||
# entry[0] = ref @ t=-time_sf (appearance), training layout
|
||||
# entry[0] = appearance reference at its own temporal offset. Guide remains at t=0.
|
||||
if ref_lat is not None:
|
||||
ref_pixel_pos = reference_temporal_pixel_offset(ref_temporal_offset_latents, time_sf)
|
||||
positive, negative, chunk_latent, chunk_mask = nodes_lt.LTXVAddGuide.append_keyframe(
|
||||
positive, negative, -time_sf, chunk_latent, chunk_mask,
|
||||
positive, negative, ref_pixel_pos, chunk_latent, chunk_mask,
|
||||
ref_lat, ref_strength, scale_factors, causal_fix=True,
|
||||
)
|
||||
positive, negative = nodes_lt._append_guide_attention_entry(
|
||||
@@ -1593,8 +1605,8 @@ class LTXVEditAnythingLoopingSampler:
|
||||
|
||||
@staticmethod
|
||||
def _sample_chunk(model, noise, sampler, sigmas, guider, latent, seed_offset=0):
|
||||
import comfy.sample
|
||||
import comfy.model_management
|
||||
import comfy.sample
|
||||
import comfy.utils
|
||||
import latent_preview
|
||||
|
||||
@@ -1685,6 +1697,7 @@ class LTXVEditAnythingLoopingSampler:
|
||||
ref_image=None, guide_frames=None,
|
||||
ref_resize_mode="pad_to_fit",
|
||||
ref_strength=1.0, guide_strength=1.0,
|
||||
ref_temporal_offset_latents=DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
|
||||
role_strength=1.0, adaln_scale=1.0,
|
||||
enable_ic_lora=True, enable_role_embedding=True, enable_adaln=True,
|
||||
reapply_per_chunk=True,
|
||||
@@ -1884,6 +1897,11 @@ class LTXVEditAnythingLoopingSampler:
|
||||
print(f" total latent frames : {total_frames}")
|
||||
print(f" tile/overlap (lat) : {tile}/{overlap} step={step}")
|
||||
print(f" ref_lat : {None if ref_lat is None else list(ref_lat.shape)}")
|
||||
print(
|
||||
" ref t offset: "
|
||||
f"{int(ref_temporal_offset_latents)} latent = "
|
||||
f"{reference_temporal_pixel_offset(ref_temporal_offset_latents, time_sf)} pixel frame(s)"
|
||||
)
|
||||
print(f" guide_lat : {None if guide_lat is None else list(guide_lat.shape)}")
|
||||
print(f" role_weight present : {role_weight is not None}")
|
||||
print(f" adaln_cond present : {adaln_cond is not None}")
|
||||
@@ -1913,7 +1931,7 @@ class LTXVEditAnythingLoopingSampler:
|
||||
chunk_pos, chunk_neg, chunk_latent, chunk_mask,
|
||||
ref_lat if do_append else None,
|
||||
guide_chunk_lat if do_append else None,
|
||||
scale_factors, ref_strength, guide_strength,
|
||||
scale_factors, ref_strength, guide_strength, ref_temporal_offset_latents,
|
||||
enable_ic_lora=do_append,
|
||||
)
|
||||
|
||||
@@ -1980,6 +1998,7 @@ class LTXVEditAnythingLoopingSampler:
|
||||
"enable_visual_crossattn": bool(enable_visual_crossattn),
|
||||
"reapply_per_chunk": bool(reapply_per_chunk),
|
||||
"ref_strength": float(ref_strength),
|
||||
"ref_temporal_offset_latents": int(ref_temporal_offset_latents),
|
||||
"guide_strength": float(guide_strength),
|
||||
"role_strength": float(role_strength),
|
||||
"adaln_scale": float(adaln_scale),
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
"""Decode direto do single-frame VAE do MiniMax-H3, na receita do card.
|
||||
|
||||
Motivo de existir: gerando pelo DiT e decodificando com o `VAEDecode` padrao, a saida sai
|
||||
quadriculada. O caminho de referencia (card do iamkaikai, e o Space
|
||||
`multimodalart/MiniMax-H3-images`) NAO usa `vae.decode()` -- chama o decoder direto. Quatro
|
||||
detalhes que corrompem a saida em silencio se mudarem, todos preservados aqui:
|
||||
|
||||
1. `vae.decoder(vae.post_quant_conv(z))` direto, nunca `vae.decode()`, cujo chunker temporal
|
||||
nao consegue formar chunk com poucas fatias de latente;
|
||||
2. `decoded[:, :, -1]` -- a ULTIMA fatia de saida, nao a de indice 0;
|
||||
3. desnormalizacao do latente: `z * latents_std + latents_mean`;
|
||||
4. desnormalizacao de pixel em ImageNet (mean .485/.456/.406, std .229/.224/.225) -- sem ela
|
||||
a imagem sai escura e com contraste estourado, parecendo erro de tone mapping.
|
||||
"""
|
||||
|
||||
import comfy.model_management
|
||||
import torch
|
||||
|
||||
PIXEL_MEAN = (0.485, 0.456, 0.406)
|
||||
PIXEL_STD = (0.229, 0.224, 0.225)
|
||||
|
||||
|
||||
class MiniMaxH3DirectDecode:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"vae": ("VAE", {"tooltip": "VAE do H3 com o decoder single-frame (use o loader deste pacote)."}),
|
||||
"samples": ("LATENT", {"tooltip": "Latente do H3. Funciona com o latente gerado pelo DiT, que e onde o VAEDecode padrao falha."}),
|
||||
"latent_index": ("INT", {"default": 0, "min": -64, "max": 64, "tooltip": "Qual fatia temporal decodificar. 0 = primeira, -1 = ultima."}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("IMAGE",)
|
||||
FUNCTION = "decode"
|
||||
CATEGORY = "MiniMax-H3"
|
||||
|
||||
def decode(self, vae, samples, latent_index):
|
||||
model = getattr(vae, "first_stage_model", None)
|
||||
if model is None or not hasattr(model, "decoder"):
|
||||
raise ValueError(
|
||||
"este VAE nao e o do MiniMax-H3 (sem first_stage_model.decoder). "
|
||||
"Carregue com o loader single-frame deste pacote."
|
||||
)
|
||||
|
||||
z = samples["samples"]
|
||||
# O latente do H3 chega aninhado (video + audio). Os nodes nativos desempacotam e ficam
|
||||
# com o primeiro componente, que e o video -- ver VAEDecode em nodes.py:335.
|
||||
if getattr(z, "is_nested", False):
|
||||
z = z.unbind()[0]
|
||||
if z.dim() == 4: # unbind entrega o item sem a dimensao de batch
|
||||
z = z.unsqueeze(0)
|
||||
if z.dim() == 4:
|
||||
z = z.unsqueeze(0)
|
||||
if z.dim() != 5:
|
||||
raise ValueError(f"latente esperado [B, C, T, H, W]; veio {tuple(z.shape)}")
|
||||
|
||||
# Sem isto o decode roda onde os pesos estiverem -- e com o gerenciamento de memoria do
|
||||
# ComfyUI o VAE fica na CPU ate ser usado, o que torna o decode lentissimo. O vae.decode()
|
||||
# nativo chama load_models_gpu antes de rodar; aqui e o mesmo passo.
|
||||
comfy.model_management.load_models_gpu([vae.patcher])
|
||||
device = vae.device
|
||||
dtype = vae.vae_dtype
|
||||
i = latent_index if latent_index >= 0 else z.shape[2] + latent_index
|
||||
i = max(0, min(i, z.shape[2] - 1))
|
||||
z = z[:, :, i : i + 1].to(device=device, dtype=torch.float32)
|
||||
|
||||
# (3) desnormalizacao do latente
|
||||
mean = model.latents_mean.view(1, -1, 1, 1, 1).to(device=device, dtype=torch.float32)
|
||||
std = model.latents_std.view(1, -1, 1, 1, 1).to(device=device, dtype=torch.float32)
|
||||
z = z * std + mean
|
||||
|
||||
# (1) decoder direto, sem vae.decode()
|
||||
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.float16, enabled=(device.type == "cuda")):
|
||||
decoded = model.decoder(model.post_quant_conv(z.to(dtype)))
|
||||
|
||||
# (2) ultima fatia de saida + (4) desnormalizacao ImageNet
|
||||
pm = torch.tensor(PIXEL_MEAN, device=decoded.device).view(1, 3, 1, 1)
|
||||
ps = torch.tensor(PIXEL_STD, device=decoded.device).view(1, 3, 1, 1)
|
||||
frame = (decoded[:, :, -1].float() * ps + pm).clamp(0, 1)
|
||||
print(f"[BFSNodes] H3 direct decode: latente {tuple(z.shape)} -> {tuple(frame.shape)} (fatia {i}, {device})")
|
||||
saida = frame.permute(0, 2, 3, 1).to(comfy.model_management.intermediate_device())
|
||||
return (saida,)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"MiniMaxH3DirectDecode": MiniMaxH3DirectDecode}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"MiniMaxH3DirectDecode": "MiniMax-H3 Single-Frame Decode / direct (BFS)"}
|
||||
@@ -0,0 +1,422 @@
|
||||
"""Aligned downscaled H3 guides matching ai-toolkit target_grid_stride_v1.
|
||||
|
||||
No global ComfyUI monkey patches: a cloned ModelPatcher replaces the packed
|
||||
layout in its UNet wrapper only when our conditioning marker is present.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import logging
|
||||
import math
|
||||
import json
|
||||
|
||||
try:
|
||||
from .minimax_h3_reference_rope import validate_options, PHASE_VERSION, add_source_phase
|
||||
from .minimax_h3_rope_layout import apply_rope_layout
|
||||
except ImportError:
|
||||
from minimax_h3_reference_rope import validate_options, PHASE_VERSION, add_source_phase
|
||||
from minimax_h3_rope_layout import apply_rope_layout
|
||||
|
||||
import numpy as np
|
||||
from PIL import Image, ImageOps
|
||||
import torch
|
||||
|
||||
LOG = logging.getLogger(__name__)
|
||||
MARKER = 'bfs_h3_aligned_guide'
|
||||
WRAPPER_MARKER = 'bfs_h3_downscaled_guide_wrapper_v1'
|
||||
SPATIAL_VERSION = 'target_crop_v2'
|
||||
POSITION_VERSION = 'target_grid_stride_v1'
|
||||
|
||||
|
||||
def prepare_guide_frames(images, width, height, factor):
|
||||
if isinstance(factor, bool) or not isinstance(factor, int) or factor < 1:
|
||||
raise ValueError('downscale_factor must be a positive integer')
|
||||
if width <= 0 or height <= 0 or width % (32 * factor) or height % (32 * factor):
|
||||
raise ValueError(f'Output dimensions must be positive multiples of {32 * factor}')
|
||||
if images.ndim != 4 or images.shape[0] < 1 or images.shape[-1] < 3:
|
||||
raise ValueError('image must be a nonempty IMAGE batch [frames, height, width, channels]')
|
||||
result = []
|
||||
for frame in images[..., :3].detach().float().cpu():
|
||||
pixels = (frame.clamp(0, 1).numpy() * 255).round().astype(np.uint8)
|
||||
source = Image.fromarray(pixels)
|
||||
target = ImageOps.fit(source, (width, height), Image.Resampling.BICUBIC)
|
||||
guide = target.resize((width // factor, height // factor), Image.Resampling.LANCZOS)
|
||||
result.append(torch.from_numpy(np.asarray(guide).copy()).float() / 255)
|
||||
return torch.stack(result)
|
||||
|
||||
|
||||
def sample_posterior(moments, latent_mean, latent_std, seed=42, fp16_round=False):
|
||||
mean, logvar = moments.float().chunk(2, dim=1)
|
||||
generator = torch.Generator(device='cpu').manual_seed(seed)
|
||||
noise = torch.randn(mean.shape, generator=generator, dtype=mean.dtype).to(mean.device)
|
||||
z = mean + torch.exp(0.5 * logvar.clamp(-30, 20)) * noise
|
||||
if fp16_round:
|
||||
z = z.half().float()
|
||||
shape = (1, -1, 1, 1, 1)
|
||||
return (z - latent_mean.to(z).view(shape)) / latent_std.to(z).view(shape)
|
||||
|
||||
|
||||
def encode_moments(stage, pixels, device, dtype):
|
||||
# The toolkit normalizes pixels in FP32 before casting for the encoder.
|
||||
# Native encode_temporal normalizes in the input dtype, so reproduce its
|
||||
# chunk plan here rather than changing the shared VAE instance.
|
||||
def encode_clip(clip):
|
||||
normalized = stage._normalize_pixels(clip.float().to(device)).to(dtype)
|
||||
return stage._adaptive_encode(normalized)
|
||||
|
||||
if pixels.shape[2] == 1:
|
||||
return encode_clip(pixels)[:, :, -1:]
|
||||
chunks = []
|
||||
for start in range(0, pixels.shape[2], stage.clip_length):
|
||||
clip = pixels[:, :, start:start + stage.clip_length]
|
||||
missing = stage.clip_length - clip.shape[2]
|
||||
if missing:
|
||||
clip = torch.cat([clip, clip[:, :, -1:].repeat(1, 1, missing, 1, 1)], dim=2)
|
||||
chunks.append(encode_clip(clip))
|
||||
moments = torch.cat(chunks, dim=2)
|
||||
if stage.token_drop > 0:
|
||||
moments = moments[:, :, :-stage.token_drop]
|
||||
return moments
|
||||
|
||||
|
||||
@torch.no_grad()
|
||||
def encode_guide(vae, frames):
|
||||
import comfy.model_management
|
||||
stage = getattr(vae, 'first_stage_model', None)
|
||||
required = ('_adaptive_encode', '_normalize_pixels', 'clip_length', 'token_drop', 'latents_mean', 'latents_std')
|
||||
if stage is None or any(not hasattr(stage, attr) for attr in required):
|
||||
raise ValueError('Use a complete native MiniMax-H3 video VAE with its encoder')
|
||||
comfy.model_management.load_models_gpu([vae.patcher])
|
||||
x = frames.permute(3, 0, 1, 2).unsqueeze(0).float() * 2 - 1
|
||||
moments = encode_moments(stage, x, vae.device, vae.vae_dtype)
|
||||
# The fork's sampling path uses seed 42 and FP16 posterior rounding for
|
||||
# both image keyframes and aligned guide videos.
|
||||
latents = sample_posterior(moments, stage.latents_mean, stage.latents_std,
|
||||
fp16_round=True)
|
||||
return latents.cpu()
|
||||
|
||||
|
||||
def downscaled_layout(native, keyframes, refs=()):
|
||||
"""Select coarse condition rows from the native target grid, reindex all streams.
|
||||
|
||||
Native PackedLayout creates full-canvas keyframe rows even for small
|
||||
latents. Keeping only stride-factor target patch origins fixes both row
|
||||
counts and RoPE while preserving refs, audio, target rows and their clocks.
|
||||
"""
|
||||
required = ('signature', 'segments', 'position_ids', 'seq_len', 'img_pos', 'img_update',
|
||||
'audio_pos', 'audio_update')
|
||||
if any(not hasattr(native, name) for name in required):
|
||||
raise RuntimeError('Unsupported ComfyUI H3 PackedLayout API; update ComfyUI')
|
||||
_, _, height, width, _ = native.signature
|
||||
if height % 2 or width % 2:
|
||||
raise ValueError('H3 requires complete 2x2 spatial latent patches')
|
||||
patch_h, patch_w = height // 2, width // 2
|
||||
full_rows = patch_h * patch_w
|
||||
video_guides = iter(kf for kf in keyframes if kf.get('latent') is not None)
|
||||
keep_segments, new_segments = [], []
|
||||
cursor = 0
|
||||
for start, stop, kind in native.segments:
|
||||
indices = torch.arange(start, stop)
|
||||
if kind == 'cond':
|
||||
kf = next(video_guides)
|
||||
latent = kf['latent']
|
||||
factor = kf.get(MARKER, {}).get('downscale_factor', 1)
|
||||
if isinstance(factor, bool) or not isinstance(factor, int) or factor < 1:
|
||||
raise ValueError('Invalid guide factor in conditioning')
|
||||
if height % (2 * factor) or width % (2 * factor):
|
||||
raise ValueError('Guide factor does not divide the target patch grid')
|
||||
if latent.ndim != 5 or latent.shape[:2] != (1, 24):
|
||||
raise ValueError('Guide latent must have shape [1, 24, time, height, width]')
|
||||
if tuple(latent.shape[-2:]) != (height // factor, width // factor):
|
||||
raise ValueError('Guide latent dimensions do not match the target and downscale factor')
|
||||
vt = latent.shape[2]
|
||||
if stop - start != vt * full_rows:
|
||||
raise RuntimeError('Unsupported native H3 guide segment size')
|
||||
grid = torch.arange(full_rows).reshape(patch_h, patch_w)[::factor, ::factor].flatten()
|
||||
indices = (start + torch.arange(vt)[:, None] * full_rows + grid[None]).flatten()
|
||||
keep_segments.append(indices)
|
||||
new_segments.append((cursor, cursor + len(indices), kind))
|
||||
cursor += len(indices)
|
||||
keep = torch.cat(keep_segments)
|
||||
remap = torch.full((native.seq_len,), -1, dtype=torch.long)
|
||||
remap[keep] = torch.arange(cursor)
|
||||
result = copy.copy(native)
|
||||
result.seq_len = cursor
|
||||
result.position_ids = native.position_ids[keep]
|
||||
result.segments = new_segments
|
||||
for modality in ('img', 'audio'):
|
||||
positions = getattr(native, modality + '_pos')
|
||||
selected = remap[positions] >= 0
|
||||
setattr(result, modality + '_pos', remap[positions[selected]])
|
||||
setattr(result, modality + '_update', getattr(native, modality + '_update')[selected])
|
||||
# Metadata used only by this wrapper; native signatures remain intact.
|
||||
result.bfs_h3_guide_layout = True
|
||||
return apply_rope_layout(result, keyframes, refs)
|
||||
|
||||
|
||||
def patch_guide_model(model, phase_enabled=False):
|
||||
if model.model_options.get(WRAPPER_MARKER):
|
||||
patched=model.clone()
|
||||
if phase_enabled: install_phase_patch(patched)
|
||||
return patched
|
||||
patched = model.clone()
|
||||
previous = patched.model_options.get('model_function_wrapper')
|
||||
cache = [None, None]
|
||||
|
||||
def wrapper(model_function, args):
|
||||
conds = args.get('c', {})
|
||||
payload = conds.get('minimax_payload')
|
||||
keyframes = (payload or {}).get('keyframes', [])
|
||||
refs = (payload or {}).get('refs', [])
|
||||
if any(MARKER in kf for kf in keyframes + refs):
|
||||
native = payload.get('layout')
|
||||
if native is None:
|
||||
raise RuntimeError('H3 layout missing from payload; unsupported ComfyUI version')
|
||||
if cache[0] is not native:
|
||||
cache[:] = [native, downscaled_layout(native, keyframes, refs)]
|
||||
copied_payload = dict(payload, layout=cache[1])
|
||||
args = dict(args, c=dict(conds, minimax_payload=copied_payload))
|
||||
else:
|
||||
cache[:] = [None, None]
|
||||
if previous is not None:
|
||||
return previous(model_function, args)
|
||||
return model_function(args['input'], args['timestep'], **args['c'])
|
||||
|
||||
patched.set_model_unet_function_wrapper(wrapper)
|
||||
patched.model_options[WRAPPER_MARKER] = True
|
||||
patched.model_options['bfs_h3_phase_getter'] = lambda: getattr(cache[1], 'bfs_source_phase_values', None)
|
||||
if phase_enabled: install_phase_patch(patched)
|
||||
return patched
|
||||
|
||||
|
||||
def install_phase_patch(patched):
|
||||
if patched.model_options.get('bfs_h3_source_phase_patch'): return
|
||||
original=patched.get_model_object('diffusion_model.rope_freqs')
|
||||
getter=patched.model_options['bfs_h3_phase_getter']
|
||||
def rope(position_ids, device):
|
||||
return add_source_phase(original(position_ids,device), getter())
|
||||
patched.add_object_patch('diffusion_model.rope_freqs',rope)
|
||||
patched.model_options['bfs_h3_source_phase_patch']=True
|
||||
|
||||
|
||||
def rope_inputs():
|
||||
return {'guide_rope_layout': (['overlap','sidecar'],),
|
||||
'reference_rope_layout': (['native','overlap','sidecar'],),
|
||||
'reference_source_phase': ('BOOLEAN',{'default':False}),
|
||||
'reference_phase_scale': ('FLOAT',{'default':1.0,'min':0.0}),
|
||||
'reference_sidecar_margin': ('FLOAT',{'default':0.0,'min':0.0}),
|
||||
'source_id': ('INT',{'default':1,'min':1,'max':99})}
|
||||
|
||||
|
||||
def rope_marker(options, source_id, role):
|
||||
if isinstance(source_id,bool) or not isinstance(source_id,int) or source_id<1:
|
||||
raise ValueError('source_id must match a positive training control channel')
|
||||
return {'rope_layout': options['guide_rope_layout'] if role=='guide' else options['reference_rope_layout'],
|
||||
'source_phase': options['reference_source_phase'], 'source_id': source_id,
|
||||
'phase_scale': options['reference_phase_scale'], 'sidecar_margin': options['reference_sidecar_margin']}
|
||||
|
||||
|
||||
def check_lora_metadata(lora_name, factor, options=None):
|
||||
if not lora_name or lora_name == '(manual)':
|
||||
return
|
||||
import folder_paths
|
||||
from safetensors import safe_open
|
||||
path = folder_paths.get_full_path_or_raise('loras', lora_name)
|
||||
with safe_open(path, framework='pt', device='cpu') as handle:
|
||||
metadata = handle.metadata() or {}
|
||||
for key, expected in (('minimax_h3_guide_spatial_version', SPATIAL_VERSION),
|
||||
('minimax_h3_guide_position_version', POSITION_VERSION)):
|
||||
if metadata.get(key) != expected:
|
||||
raise ValueError(f'LoRA {key} must be {expected!r}; got {metadata.get(key)!r}')
|
||||
if int(metadata.get('reference_downscale_factor', 0)) != factor:
|
||||
raise ValueError('downscale_factor must match the LoRA training metadata')
|
||||
stored=json.loads(metadata.get('minimax_h3_reference_rope','{}'))
|
||||
expected=validate_options() if options is None else options
|
||||
stored_options=validate_options(stored.get('guide_rope_layout','overlap'),stored.get('reference_rope_layout','native'),stored.get('reference_source_phase',False),stored.get('reference_phase_scale',1.0),stored.get('reference_sidecar_margin',0.0))
|
||||
if stored_options != expected or stored.get('phase_version',PHASE_VERSION)!=PHASE_VERSION:
|
||||
raise ValueError('Reference RoPE options must match LoRA training metadata')
|
||||
if not any(metadata.get(key, '').lower()=='true' for key in ('guide_latent_only','control_latent_only')):
|
||||
raise ValueError('This node expects a caption-only VLM with guide_latent_only=true')
|
||||
|
||||
|
||||
class BFSMiniMaxH3DownscaledGuide:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
import folder_paths
|
||||
return {'required': {
|
||||
'model': ('MODEL',),
|
||||
'positive': ('CONDITIONING',),
|
||||
'vae': ('VAE',),
|
||||
'latent': ('LATENT',),
|
||||
'image': ('IMAGE', {'tooltip': 'Single image or video frames already resampled to 24 fps.'}),
|
||||
'downscale_factor': ('INT', {'default': 4, 'min': 1, 'max': 8, 'step': 1}),
|
||||
'frame_idx': ('INT', {'default': 0, 'min': -9999, 'max': 9999}),
|
||||
}, 'optional': {
|
||||
'audio': ('AUDIO', {'tooltip': 'Optional aligned guide soundtrack; never routed to the text encoder.'}),
|
||||
'audio_vae': ('VAE',),
|
||||
'lora_name': (['(manual)'] + folder_paths.get_filename_list('loras'),
|
||||
{'tooltip': 'Validate metadata only. Load the same LoRA using your normal LoRA loader.'}),
|
||||
**rope_inputs(),
|
||||
}}
|
||||
|
||||
RETURN_TYPES = ('MODEL', 'CONDITIONING', 'IMAGE')
|
||||
RETURN_NAMES = ('model', 'positive', 'guide_preview')
|
||||
FUNCTION = 'apply'
|
||||
CATEGORY = 'MiniMax-H3'
|
||||
DESCRIPTION = 'Aligned lower-resolution VAE guides for H3 LoRAs trained with target_grid_stride_v1. Connect both returned MODEL and CONDITIONING to the sampler.'
|
||||
|
||||
@torch.no_grad()
|
||||
def apply(self, model, positive, vae, latent, image, downscale_factor=4, frame_idx=0,
|
||||
lora_name='(manual)', audio=None, audio_vae=None, guide_rope_layout='overlap', reference_rope_layout='native',
|
||||
reference_source_phase=False, reference_phase_scale=1.0, reference_sidecar_margin=0.0, source_id=1):
|
||||
from comfy.ldm.minimax import model as h3
|
||||
samples = latent['samples']
|
||||
streams = getattr(samples, 'tensors', None)
|
||||
if not getattr(samples, 'is_nested', False) or streams is None or len(streams) != 2:
|
||||
raise ValueError('Use a native MiniMax-H3 joint video/audio target latent')
|
||||
video = streams[0]
|
||||
if video.ndim != 5 or video.shape[:2] != (1, 24):
|
||||
raise ValueError('Target video latent must have shape [1, 24, time, height, width]')
|
||||
diffusion = model.get_model_object('diffusion_model')
|
||||
if not isinstance(diffusion, h3.MiniMaxH3Model) or tuple(diffusion.patch_size) != (1, 2, 2):
|
||||
raise ValueError('This node requires the native MiniMax-H3 model with 1x2x2 patches')
|
||||
options=validate_options(guide_rope_layout,reference_rope_layout,reference_source_phase,reference_phase_scale,reference_sidecar_margin)
|
||||
check_lora_metadata(lora_name, downscale_factor, options)
|
||||
width, height = video.shape[-1] * 16, video.shape[-2] * 16
|
||||
frame_count = sum(h3.FRAME_PER_TOKEN[k % 5] for k in range(video.shape[2]))
|
||||
frames = image.shape[0]
|
||||
if frames != 1 and (frames < 5 or frames % 17 != 5):
|
||||
raise ValueError('Guide must be one image or exactly 17*n+5 video frames (5, 22, 39, 73, ...); trim/resample explicitly')
|
||||
index = frame_idx if frame_idx >= 0 else frame_count + frame_idx
|
||||
if index < 0 or index + frames > frame_count:
|
||||
raise ValueError('Guide frames do not fit in the target timeline at frame_idx')
|
||||
if not positive:
|
||||
raise ValueError('positive conditioning must not be empty')
|
||||
if audio is not None and audio_vae is None:
|
||||
raise ValueError('audio_vae is required for a guide soundtrack')
|
||||
preview = prepare_guide_frames(image, width, height, downscale_factor)
|
||||
z = encode_guide(vae, preview)
|
||||
expected_t = 1 if frames == 1 else (frames - 5) // 17 * 5 + 2
|
||||
if tuple(z.shape) != (1, 24, expected_t, height // (16 * downscale_factor), width // (16 * downscale_factor)):
|
||||
raise ValueError(f'Unexpected H3 VAE latent shape: {tuple(z.shape)}')
|
||||
guide = {'resolved_frame_index': index, 'latent': z, MARKER: {
|
||||
'downscale_factor': downscale_factor,
|
||||
'spatial_version': SPATIAL_VERSION, 'position_version': POSITION_VERSION,
|
||||
**rope_marker(options, source_id, 'guide'),
|
||||
}}
|
||||
if audio is not None:
|
||||
if audio_vae is None:
|
||||
raise ValueError('audio_vae is required for a guide soundtrack')
|
||||
from comfy_extras.nodes_minimax_h3 import _encode_ref_audio
|
||||
audio_z, _ = _encode_ref_audio(audio_vae, audio)
|
||||
available = math.floor(streams[1].shape[-1] - h3.FRAME_RESCALE * index)
|
||||
clip_audio_frames = round(frames / 24 * 40)
|
||||
if available < 1:
|
||||
raise ValueError('Guide soundtrack starts outside the target audio timeline')
|
||||
guide['audio_latent'] = audio_z[..., :min(available, clip_audio_frames)].clone()
|
||||
# Copy all conditioning entries, preserving existing guides/references.
|
||||
updated = []
|
||||
for embedding, values in positive:
|
||||
values = dict(values)
|
||||
values['minimax_keyframes'] = list(values.get('minimax_keyframes', [])) + [guide]
|
||||
updated.append([embedding, values])
|
||||
LOG.info('BFS H3 guide: %s frames, %sx%s pixels, factor %s, frame %s', frames,
|
||||
preview.shape[2], preview.shape[1], downscale_factor, index)
|
||||
return (patch_guide_model(model, reference_source_phase), updated, preview)
|
||||
|
||||
|
||||
class BFSMiniMaxH3IdentityReference:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
import folder_paths
|
||||
extra=rope_inputs()
|
||||
extra['source_id']=('INT',{'default':2,'min':1,'max':99})
|
||||
extra['downscale_factor']=('INT',{'default':4,'min':1,'max':8})
|
||||
extra['lora_name']=(['(manual)']+folder_paths.get_filename_list('loras'),)
|
||||
return {'required':{'model':('MODEL',),'positive':('CONDITIONING',),'vae':('VAE',),
|
||||
'latent':('LATENT',),'image':('IMAGE',)},'optional':extra}
|
||||
|
||||
RETURN_TYPES=('MODEL','CONDITIONING','IMAGE')
|
||||
RETURN_NAMES=('model','positive','reference_preview')
|
||||
FUNCTION='apply'
|
||||
CATEGORY='MiniMax-H3'
|
||||
DESCRIPTION='Caption-only image identity reference. Match source_id to the training control channel. Requires an upscale checkpoint trained with the same reference role and RoPE options.'
|
||||
|
||||
@torch.no_grad()
|
||||
def apply(self,model,positive,vae,latent,image,guide_rope_layout='overlap',reference_rope_layout='native',
|
||||
reference_source_phase=False,reference_phase_scale=1.0,reference_sidecar_margin=0.0,
|
||||
source_id=2,downscale_factor=4,lora_name='(manual)'):
|
||||
from comfy.ldm.minimax import model as h3
|
||||
diffusion=model.get_model_object('diffusion_model')
|
||||
if not isinstance(diffusion,h3.MiniMaxH3Model) or tuple(diffusion.patch_size)!=(1,2,2):
|
||||
raise ValueError('Identity reference requires native MiniMax-H3')
|
||||
if image.ndim!=4 or image.shape[0]!=1:
|
||||
raise ValueError('Identity reference must contain exactly one image')
|
||||
if not positive:raise ValueError('positive conditioning must not be empty')
|
||||
streams=getattr(latent.get('samples'),'tensors',None)
|
||||
if streams is None or len(streams)!=2:raise ValueError('Use a joint H3 target latent')
|
||||
video=streams[0]
|
||||
if video.ndim!=5 or video.shape[:2]!=(1,24):raise ValueError('Target batch must be one H3 video')
|
||||
options=validate_options(guide_rope_layout,reference_rope_layout,reference_source_phase,reference_phase_scale,reference_sidecar_margin)
|
||||
check_lora_metadata(lora_name,downscale_factor,options)
|
||||
if lora_name!='(manual)':
|
||||
import folder_paths
|
||||
from safetensors import safe_open
|
||||
with safe_open(folder_paths.get_full_path_or_raise('loras',lora_name),framework='pt',device='cpu') as f:
|
||||
if (f.metadata() or {}).get('control_latent_only','').lower()!='true':
|
||||
raise ValueError('Caption-only identity node requires control_latent_only=true in training')
|
||||
ih,iw=image.shape[1:3];target_h,target_w=video.shape[-2]*16,video.shape[-1]*16
|
||||
scale=min(1.0,math.sqrt(target_h*target_w/(ih*iw)))
|
||||
height=max(32,round(ih*scale/32)*32);width=max(32,round(iw*scale/32)*32)
|
||||
pixels=(image[0,...,:3].detach().float().cpu().clamp(0,1).numpy()*255).round().astype(np.uint8)
|
||||
ref=Image.fromarray(pixels).resize((width,height),Image.Resampling.LANCZOS)
|
||||
preview=torch.from_numpy(np.asarray(ref).copy()).float().unsqueeze(0)/255
|
||||
z=encode_guide(vae,preview)
|
||||
block={'kind':'image','latent_h':height//16,'latent_w':width//16,'latent':z,
|
||||
MARKER:rope_marker(options,source_id,'reference')}
|
||||
updated=[]
|
||||
for embedding,values in positive:
|
||||
copied=dict(values);copied['minimax_refs']=list(values.get('minimax_refs',[]))+[block]
|
||||
updated.append([embedding,copied])
|
||||
return patch_guide_model(model,reference_source_phase),updated,preview
|
||||
|
||||
|
||||
class BFSMiniMaxH3GuideTarget:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {'required': {
|
||||
'width': ('INT', {'default': 1024, 'min': 32, 'max': 16384, 'step': 32}),
|
||||
'height': ('INT', {'default': 768, 'min': 32, 'max': 16384, 'step': 32}),
|
||||
'frames': ('INT', {'default': 73, 'min': 1, 'max': 3600, 'step': 1,
|
||||
'tooltip': '1 for a true still image, or exactly 17*n+5 video frames.'}),
|
||||
}}
|
||||
|
||||
RETURN_TYPES = ('LATENT',)
|
||||
FUNCTION = 'create'
|
||||
CATEGORY = 'MiniMax-H3'
|
||||
|
||||
def create(self, width, height, frames):
|
||||
import comfy.model_management
|
||||
from comfy.nested_tensor import NestedTensor
|
||||
if width <= 0 or height <= 0 or width % 32 or height % 32:
|
||||
raise ValueError('Target width/height must be positive multiples of 32')
|
||||
if frames != 1 and (frames < 5 or frames % 17 != 5):
|
||||
raise ValueError('Target frames must be 1 or 17*n+5')
|
||||
vt = 1 if frames == 1 else (frames - 5) // 17 * 5 + 2
|
||||
at = round(frames / 24 * 40)
|
||||
device = comfy.model_management.intermediate_device()
|
||||
video = torch.zeros(1, 24, vt, height // 16, width // 16, device=device)
|
||||
audio = torch.zeros(1, 32, 2, at, device=device)
|
||||
return ({'samples': NestedTensor((video, audio))},)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
'BFSMiniMaxH3DownscaledGuide': BFSMiniMaxH3DownscaledGuide,
|
||||
'BFSMiniMaxH3GuideTarget': BFSMiniMaxH3GuideTarget,
|
||||
'BFSMiniMaxH3IdentityReference': BFSMiniMaxH3IdentityReference,
|
||||
}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
'BFSMiniMaxH3DownscaledGuide': 'MiniMax-H3 Downscaled Latent Guide (BFS)',
|
||||
'BFSMiniMaxH3GuideTarget': 'MiniMax-H3 Guide Target — Image / Video (BFS)',
|
||||
'BFSMiniMaxH3IdentityReference': 'MiniMax-H3 Identity Reference + RoPE (BFS)',
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
"""Experimental H3 reference geometry and source phase; shared with BFSNodes.
|
||||
|
||||
Coordinates use H3's normalized RoPE units, not LTX pixel patch bounds.
|
||||
Source phase v1 composes theta**(-d/half_dim) with the original rotary angles.
|
||||
"""
|
||||
import math
|
||||
import torch
|
||||
|
||||
PHASE_VERSION = 'h3_source_phase_v1'
|
||||
|
||||
|
||||
def validate_options(guide_layout='overlap', reference_layout='native', source_phase=False, phase_scale=1.0, sidecar_margin=0.0):
|
||||
if guide_layout not in ('overlap', 'sidecar'):
|
||||
raise ValueError('guide_rope_layout must be overlap or sidecar')
|
||||
if reference_layout not in ('native', 'overlap', 'sidecar'):
|
||||
raise ValueError('reference_rope_layout must be native, overlap or sidecar')
|
||||
if not isinstance(source_phase, bool):
|
||||
raise ValueError('reference_source_phase must be boolean')
|
||||
for name, value in [('reference_phase_scale', phase_scale), ('reference_sidecar_margin', sidecar_margin)]:
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value) or value < 0:
|
||||
raise ValueError(f'{name} must be finite and nonnegative')
|
||||
return dict(guide_rope_layout=guide_layout, reference_rope_layout=reference_layout,
|
||||
reference_source_phase=source_phase, reference_phase_scale=float(phase_scale),
|
||||
reference_sidecar_margin=float(sidecar_margin))
|
||||
|
||||
|
||||
def options_from_kwargs(kwargs):
|
||||
return validate_options(kwargs.get('guide_rope_layout', 'overlap'), kwargs.get('reference_rope_layout', 'native'),
|
||||
kwargs.get('reference_source_phase', False), kwargs.get('reference_phase_scale', 1.0),
|
||||
kwargs.get('reference_sidecar_margin', 0.0))
|
||||
|
||||
|
||||
def shift_reference(positions, target, layout, margin=0.0):
|
||||
if layout in ('native', 'overlap'):
|
||||
return positions
|
||||
if layout != 'sidecar' or not math.isfinite(margin) or margin < 0:
|
||||
raise ValueError('Invalid sidecar layout or margin')
|
||||
result=positions.clone()
|
||||
# Place beyond the final target patch origin, separated by one patch step.
|
||||
widths=torch.unique(target[:,2],sorted=True)
|
||||
step=float(widths[1]-widths[0]) if widths.numel()>1 else 1.0
|
||||
result[:,2]+=target[:,2].max()+step+margin-positions[:,2].min()
|
||||
result[:,1]+=(target[:,1].min()+target[:,1].max()-positions[:,1].min()-positions[:,1].max())/2
|
||||
return result
|
||||
|
||||
|
||||
def add_source_phase(angles, source_values, theta=10000.0):
|
||||
"""Angles [..., S, 2*D], duplicated halves; zero sources stay bit-identical."""
|
||||
if source_values is None:
|
||||
return angles
|
||||
if tuple(source_values.shape)!=tuple(angles.shape[:-1]):
|
||||
raise ValueError('Source phase rows must match rotary rows')
|
||||
if not torch.isfinite(source_values).all() or (source_values<0).any():
|
||||
raise ValueError('Source phase values must be finite and nonnegative')
|
||||
if not torch.count_nonzero(source_values):
|
||||
return angles
|
||||
half=angles.shape[-1]//2
|
||||
rates=theta**(-torch.arange(half,device=angles.device,dtype=torch.float32)/half)
|
||||
phase=source_values.to(device=angles.device,dtype=torch.float32)[...,None]*rates
|
||||
return angles+torch.cat((phase,phase),dim=-1)
|
||||
|
||||
|
||||
def control_source_channels(config, count):
|
||||
"""Keep explicit channel numbers when earlier control paths are absent."""
|
||||
channels=[i for i in (1,2,3) if getattr(config,f'control_path_{i}',None)]
|
||||
if not channels:return list(range(1,count+1))
|
||||
if len(channels)!=count:raise ValueError('Control channel/path counts do not match')
|
||||
return channels
|
||||
@@ -0,0 +1,60 @@
|
||||
"""Apply H3 reference layouts to a cloned native Comfy layout."""
|
||||
import torch
|
||||
try:
|
||||
from .minimax_h3_reference_rope import shift_reference, add_source_phase, validate_options
|
||||
except ImportError:
|
||||
from minimax_h3_reference_rope import shift_reference, add_source_phase, validate_options
|
||||
|
||||
MARKER='bfs_h3_aligned_guide'
|
||||
|
||||
|
||||
def ref_span(ref):
|
||||
if ref['kind']=='image': return 1.0
|
||||
audio=float(ref.get('ref_audio_t',0))
|
||||
if ref['kind']=='audio': return audio
|
||||
vt=ref['latent_t']
|
||||
return max(audio,sum((1,4,4,4,4)[i%5]*5/3 for i in range(vt)))
|
||||
|
||||
|
||||
def apply_rope_layout(layout, keyframes, refs):
|
||||
# Remove only experimental refs from the native cumulative target clock.
|
||||
spans=[ref_span(r) for r in refs]
|
||||
removed=sum(span for span,r in zip(spans,refs) if r.get(MARKER,{}).get('rope_layout','native')!='native')
|
||||
positions=layout.position_ids.clone()
|
||||
source=torch.zeros(layout.seq_len,dtype=torch.float32)
|
||||
target_rows=next((slice(a,b) for a,b,k in layout.segments if k=='video'))
|
||||
for a,b,kind in layout.segments:
|
||||
if kind in ('video','audio','cond','cond_audio'):positions[a:b,0]-=removed
|
||||
target=positions[target_rows]
|
||||
guide_segments=iter((a,b,k) for a,b,k in layout.segments if k in ('cond','cond_audio'))
|
||||
def apply_ranges(ranges, opts):
|
||||
if opts.get('rope_layout','overlap')=='sidecar' and ranges:
|
||||
indices=torch.cat([torch.arange(a,b) for a,b,_ in ranges])
|
||||
positions[indices]=shift_reference(positions[indices],target,'sidecar',opts.get('sidecar_margin',0.0))
|
||||
if opts.get('source_phase',False):
|
||||
for a,b,_ in ranges:source[a:b]=opts['source_id']*opts.get('phase_scale',1.0)
|
||||
for kf in keyframes:
|
||||
ranges=[]
|
||||
if kf.get('latent') is not None:ranges.append(next(guide_segments))
|
||||
if kf.get('audio_latent') is not None:ranges.append(next(guide_segments))
|
||||
opts=kf.get(MARKER,{})
|
||||
apply_ranges(ranges, opts)
|
||||
ref_segments=iter((a,b,k) for a,b,k in layout.segments if k in ('ref_img','ref_audio'))
|
||||
cursor=float(layout.signature[0])
|
||||
native_cursor=cursor
|
||||
target_origin=float(target[0,0])
|
||||
for ref,span in zip(refs,spans):
|
||||
ranges=[]
|
||||
if ref.get('ref_audio_t',0)>0:ranges.append(next(ref_segments))
|
||||
if ref['kind'] in ('image','video','video_audio'):ranges.append(next(ref_segments))
|
||||
opts=ref.get(MARKER,{})
|
||||
name=opts.get('rope_layout','native')
|
||||
origin=cursor if name=='native' else target_origin
|
||||
for a,b,kind in ranges:
|
||||
positions[a:b,0]+=origin-native_cursor
|
||||
apply_ranges(ranges, opts)
|
||||
native_cursor+=span
|
||||
if name=='native':cursor+=span
|
||||
layout.position_ids=positions
|
||||
layout.bfs_source_phase_values=source if torch.count_nonzero(source) else None
|
||||
return layout
|
||||
@@ -0,0 +1,184 @@
|
||||
"""Loader do decoder single-frame do MiniMax-H3 (iamkaikai/MiniMax-H3-Single-Frame-VAE-500K).
|
||||
|
||||
O checkpoint publicado e decoder-only e usa o naming do diffusers, entao o VAELoader padrao
|
||||
do ComfyUI nao o carrega: a deteccao em comfy/sd.py exige tambem uma chave de encoder
|
||||
(`encoder.down.5.block.0.conv1.weight`), e sem ela o arquivo cai numa arquitetura errada e a
|
||||
saida vira blocos.
|
||||
|
||||
Este node resolve os dois lados: converte as chaves para o layout do ComfyUI e completa o que
|
||||
falta (encoder, quant_conv, latents_mean/std, mask_token) a partir do VAE oficial do H3.
|
||||
|
||||
Duas transformacoes NAO sao dedutiveis pelo nome, e errar qualquer uma produz saida errada
|
||||
sem nenhum erro de load. Ambas foram determinadas medindo correlacao contra o VAE oficial:
|
||||
|
||||
to_qkv e intercalado por cabeca -- (heads, 3, dim_head), nao [q;k;v] empilhado.
|
||||
Confirmado em comfy/ldm/minimax/vae.py: `qkv.view(B, S, -1, 3 * dim_head)`.
|
||||
ff.w1 tem as metades trocadas em relacao ao diffusers, porque o ComfyUI le
|
||||
`gate, x = w1(x).chunk(2)` -- gate primeiro.
|
||||
|
||||
Validado contra o caminho de referencia do autor (diffusers + load_decoder.py dele) com o
|
||||
mesmo latente: PSNR 72,92 dB entre as duas saidas, ou seja, ruido de fp16.
|
||||
"""
|
||||
|
||||
import comfy.sd
|
||||
import comfy.utils
|
||||
import folder_paths
|
||||
import torch
|
||||
|
||||
HEADS = 32
|
||||
DIM_HEAD = 64
|
||||
# Chaves que o comfy/sd.py usa para reconhecer o VAE do MiniMax-H3.
|
||||
DETECT_KEYS = ("decoder.transformer_blocks.0.scale1", "encoder.down.5.block.0.conv1.weight")
|
||||
|
||||
|
||||
def _convert_decoder_keys(decoder_sd):
|
||||
"""Naming diffusers -> naming ComfyUI. Devolve (state_dict, n_blocos)."""
|
||||
out = {}
|
||||
blocks = set()
|
||||
for key, tensor in decoder_sd.items():
|
||||
if ".attn.to_q." in key or ".attn.to_k." in key or ".attn.to_v." in key:
|
||||
blocks.add(int(key.split("transformer_blocks.")[1].split(".")[0]))
|
||||
continue
|
||||
new_key = (
|
||||
key.replace(".ff.net.0.proj.", ".ff.w1.")
|
||||
.replace(".ff.net.2.", ".ff.w2.")
|
||||
.replace(".attn.to_out.0.", ".attn.to_out.")
|
||||
.replace("decoder.proj_in.", "decoder.x_embedder.")
|
||||
)
|
||||
if ".ff.w1." in new_key: # ComfyUI le (gate, valor); diffusers grava (valor, gate)
|
||||
half = tensor.shape[0] // 2
|
||||
tensor = torch.cat([tensor[half:], tensor[:half]], dim=0)
|
||||
out[new_key] = tensor
|
||||
|
||||
for b in sorted(blocks):
|
||||
for suffix in ("weight", "bias"):
|
||||
parts = [decoder_sd[f"decoder.transformer_blocks.{b}.attn.to_{n}.{suffix}"] for n in ("q", "k", "v")]
|
||||
tail = parts[0].shape[1:]
|
||||
stacked = torch.stack([p.view(HEADS, DIM_HEAD, *tail) for p in parts], dim=1) # [H, 3, DH, ...]
|
||||
out[f"decoder.transformer_blocks.{b}.attn.to_qkv.{suffix}"] = stacked.reshape(HEADS * 3 * DIM_HEAD, *tail)
|
||||
return out, len(blocks)
|
||||
|
||||
|
||||
class MiniMaxH3SingleFrameVAELoader:
|
||||
"""Monta um VAE do H3 com o decoder single-frame por cima do VAE oficial."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
vaes = folder_paths.get_filename_list("vae")
|
||||
return {
|
||||
"required": {
|
||||
"base_vae": (vaes, {"tooltip": "VAE oficial do MiniMax-H3 (fornece encoder, quant_conv e as estatisticas de latente)."}),
|
||||
"single_frame_decoder": (vaes, {"tooltip": "Checkpoint decoder-only do autor (585 tensores, naming diffusers). Tambem aceita um ja convertido."}),
|
||||
"tiling": ("BOOLEAN", {"default": True, "tooltip": "Tiling espacial. MANTENHA LIGADO acima de ~768px: o curriculo do autor parou em 1024 e a imagem inteira acima disso produz grade na fronteira de 32px (blocagem 2.9x em 1024, 3.5x em 1536, contra ~1.5x com tiling). Desligar so compensa em imagens pequenas, onde rende um pouco mais de nitidez."}),
|
||||
"tile_size": ("INT", {"default": 512, "min": 128, "max": 2048, "step": 64, "tooltip": "So vale com tiling ligado. 512 e o otimo MEDIDO, nao um chute: numa imagem 1056x640, PSNR 26.00 dB com costura 0.91 (indistinguivel), contra 22.17/1.49 com 256 (o default do ComfyUI, que multiplica costuras) e 21.35/2.42 com 1024 (acima da resolucao treinada). O autor treinou 475k das 500k imagens em <=512px, entao o tile de 512 mantem cada pedaco na regiao com massa de treino."}),
|
||||
"dtype": (["float16", "float32", "bfloat16"], {"default": "float16"}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("VAE",)
|
||||
FUNCTION = "load"
|
||||
CATEGORY = "MiniMax-H3"
|
||||
|
||||
def load(self, base_vae, single_frame_decoder, tiling, tile_size, dtype):
|
||||
base_sd = comfy.utils.load_torch_file(folder_paths.get_full_path("vae", base_vae))
|
||||
dec_sd = comfy.utils.load_torch_file(folder_paths.get_full_path("vae", single_frame_decoder))
|
||||
|
||||
missing_detect = [k for k in DETECT_KEYS if k not in base_sd]
|
||||
if missing_detect:
|
||||
raise ValueError(
|
||||
f"'{base_vae}' nao parece o VAE oficial do MiniMax-H3 (faltam {missing_detect}). "
|
||||
"Escolha o VAE completo, com encoder."
|
||||
)
|
||||
|
||||
if any(".attn.to_q." in k for k in dec_sd):
|
||||
converted, n_blocks = _convert_decoder_keys(dec_sd)
|
||||
print(f"[BFSNodes] single-frame decoder convertido do naming diffusers ({n_blocks} blocos)")
|
||||
else:
|
||||
converted = dict(dec_sd) # ja esta no layout do ComfyUI
|
||||
print("[BFSNodes] single-frame decoder ja estava no naming do ComfyUI")
|
||||
|
||||
sd = dict(base_sd)
|
||||
replaced, unknown = 0, []
|
||||
for key, tensor in converted.items():
|
||||
if key in sd:
|
||||
if tuple(sd[key].shape) != tuple(tensor.shape):
|
||||
raise ValueError(f"shape divergente em '{key}': base {tuple(sd[key].shape)} vs decoder {tuple(tensor.shape)}")
|
||||
sd[key] = tensor.to(sd[key].dtype)
|
||||
replaced += 1
|
||||
else:
|
||||
unknown.append(key)
|
||||
if unknown:
|
||||
raise ValueError(f"chaves do decoder que nao existem no VAE base: {unknown[:5]} (total {len(unknown)})")
|
||||
# Relatorio honesto: um checkpoint decoder-only substitui so decoder/post_quant_conv, mas um
|
||||
# arquivo ja convertido carrega o VAE inteiro e sobrescreve tambem o encoder do base.
|
||||
from collections import Counter
|
||||
por_prefixo = Counter(k.split(".")[0] for k in converted)
|
||||
do_base = sorted({k.split(".")[0] for k in sd} - set(por_prefixo))
|
||||
print(f"[BFSNodes] {replaced} tensores substituidos: "
|
||||
+ ", ".join(f"{p}={n}" for p, n in sorted(por_prefixo.items())))
|
||||
print(f"[BFSNodes] preservado do base: {', '.join(do_base) if do_base else 'nada (o arquivo cobriu o VAE inteiro)'}")
|
||||
if "encoder" in por_prefixo:
|
||||
print("[BFSNodes] aviso: o arquivo tambem trouxe encoder, entao o encoder do base foi sobrescrito")
|
||||
|
||||
torch_dtype = {"float16": torch.float16, "float32": torch.float32, "bfloat16": torch.bfloat16}[dtype]
|
||||
vae = comfy.sd.VAE(sd=sd, dtype=torch_dtype)
|
||||
|
||||
model = getattr(vae, "first_stage_model", None)
|
||||
if model is not None and hasattr(model, "tiling"):
|
||||
model.tiling = bool(tiling)
|
||||
model.tile_size = int(tile_size)
|
||||
print(f"[BFSNodes] tiling={'on' if tiling else 'off'} tile_size={tile_size}")
|
||||
return (vae,)
|
||||
|
||||
class MiniMaxH3VAELoaderTiled:
|
||||
"""Carrega um VAE do H3 ja completo (por exemplo um single-frame convertido) expondo o tiling.
|
||||
|
||||
Igual ao VAELoader nativo, com uma diferenca que muda o resultado: o construtor do
|
||||
MiniMaxH3VideoVAE usa tile_size=256, e o loader nativo nao expoe esse campo. 256 e o pior
|
||||
valor medido -- multiplica costuras. Aqui da para por 512.
|
||||
"""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"vae_name": (folder_paths.get_filename_list("vae"), {"tooltip": "VAE completo do H3 (com encoder). Para o checkpoint decoder-only do autor, use o outro node."}),
|
||||
"tiling": ("BOOLEAN", {"default": True, "tooltip": "Tiling espacial. MANTENHA LIGADO acima de ~768px: o curriculo do autor parou em 1024 e a imagem inteira acima disso produz grade na fronteira de 32px."}),
|
||||
"tile_size": ("INT", {"default": 512, "min": 128, "max": 2048, "step": 64, "tooltip": "512 e o otimo MEDIDO: numa imagem 1056x640, PSNR 26.00 dB / costura 0.91, contra 22.17/1.49 com 256 (o default do ComfyUI) e 21.35/2.42 com 1024. O autor treinou 475k das 500k imagens em <=512px."}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("VAE",)
|
||||
FUNCTION = "load"
|
||||
CATEGORY = "MiniMax-H3"
|
||||
|
||||
def load(self, vae_name, tiling, tile_size):
|
||||
path = folder_paths.get_full_path("vae", vae_name)
|
||||
sd, metadata = comfy.utils.load_torch_file(path, return_metadata=True)
|
||||
|
||||
missing = [k for k in DETECT_KEYS if k not in sd]
|
||||
if missing:
|
||||
raise ValueError(
|
||||
f"'{vae_name}' nao sera reconhecido como MiniMax-H3 pelo ComfyUI (faltam {missing}). "
|
||||
"Um checkpoint decoder-only cai em geometria de Stable Diffusion e produz blocos -- "
|
||||
"use o node 'MiniMax-H3 Single-Frame VAE Loader' para combina-lo com o VAE oficial."
|
||||
)
|
||||
|
||||
vae = comfy.sd.VAE(sd=sd, metadata=metadata)
|
||||
vae.throw_exception_if_invalid()
|
||||
model = getattr(vae, "first_stage_model", None)
|
||||
if model is not None and hasattr(model, "tiling"):
|
||||
model.tiling = bool(tiling)
|
||||
model.tile_size = int(tile_size)
|
||||
print(f"[BFSNodes] {vae_name}: tiling={'on' if tiling else 'off'} tile_size={tile_size}")
|
||||
return (vae,)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"MiniMaxH3SingleFrameVAELoader": MiniMaxH3SingleFrameVAELoader,
|
||||
"MiniMaxH3VAELoaderTiled": MiniMaxH3VAELoaderTiled,
|
||||
}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"MiniMaxH3SingleFrameVAELoader": "MiniMax-H3 Single-Frame VAE Loader (BFS)",
|
||||
"MiniMaxH3VAELoaderTiled": "MiniMax-H3 VAE Loader / tile control (BFS)",
|
||||
}
|
||||
@@ -0,0 +1,285 @@
|
||||
"""Multi-Ref Sheet Builder — combine 1-5 reference images into the single
|
||||
composite reference sheet used by the multi-ref LTX-2 LoRA (source_id=2,
|
||||
layout=overlap). Same grid convention as the training pipeline's
|
||||
build_multiref_sheet.py: fixed 512x512 panels, deterministic grid by count
|
||||
(1x1, 2x1, 3x1, 2x2, 3-top+2-bottom), centered with padding onto a fixed
|
||||
1536x1024 canvas so every sheet is the same resolution regardless of how
|
||||
many refs are plugged in.
|
||||
|
||||
Panel order = image index order (ref_image_1 -> image0, ref_image_2 ->
|
||||
image1, ...), matching the training data's image0/image1/... convention.
|
||||
"""
|
||||
import torch
|
||||
from PIL import Image
|
||||
|
||||
from .util import tensor_to_pil, pil_to_tensor
|
||||
|
||||
CATEGORY = "BFS/multiref"
|
||||
|
||||
PANEL_SIZE = 512
|
||||
CANVAS_W, CANVAS_H = 1536, 1024
|
||||
BG_COLOR = (255, 255, 255)
|
||||
|
||||
# row layout per ref count: list of ints = panels per row, top to bottom.
|
||||
LAYOUTS = {
|
||||
1: [1],
|
||||
2: [2],
|
||||
3: [3],
|
||||
4: [2, 2],
|
||||
5: [3, 2],
|
||||
}
|
||||
|
||||
|
||||
def _cover_resize_crop(img: Image.Image, size: int) -> Image.Image:
|
||||
"""Resize+center-crop to exactly fill a size x size square (no stretch, crops excess)."""
|
||||
img = img.convert("RGB")
|
||||
w, h = img.size
|
||||
scale = max(size / w, size / h)
|
||||
nw, nh = round(w * scale), round(h * scale)
|
||||
img = img.resize((nw, nh), Image.LANCZOS)
|
||||
x0, y0 = (nw - size) // 2, (nh - size) // 2
|
||||
return img.crop((x0, y0, x0 + size, y0 + size))
|
||||
|
||||
|
||||
def _justified_compose(imgs, rows_counts, canvas_w, canvas_h, bg):
|
||||
"""Row-justified layout (no cropping, no distortion) -- identical algorithm to
|
||||
build_multiref_sheet.py's _justified_compose, kept in sync so inference-time
|
||||
sheets match training-time sheets exactly. Each row is split into
|
||||
`rows_counts[i]` images; every image in a row shares that row's height, and
|
||||
its width = row_height * (image's own aspect ratio) -- one uniform scale
|
||||
factor per image, so nothing is stretched.
|
||||
|
||||
Each row's natural height fills canvas_w at scale=1. If all rows fit within
|
||||
canvas_h, they're drawn at natural size (leftover vertical space -> a single
|
||||
top+bottom bar, whole block centered) -- never blown up past canvas_w. Only
|
||||
if rows would collectively overflow canvas_h do all rows shrink by one
|
||||
shared factor < 1, giving a single shared pair of side bars instead of
|
||||
per-panel scattered padding.
|
||||
"""
|
||||
it = iter(imgs)
|
||||
rows = []
|
||||
for count in rows_counts:
|
||||
row_imgs = [next(it) for _ in range(count)]
|
||||
aspects = [im.width / im.height for im in row_imgs]
|
||||
rows.append((row_imgs, aspects))
|
||||
|
||||
natural_heights = [canvas_w / sum(aspects) for _, aspects in rows]
|
||||
scale = min(1.0, canvas_h / sum(natural_heights))
|
||||
|
||||
row_heights = [round(nat_h * scale) for nat_h in natural_heights]
|
||||
drift_h = round(sum(natural_heights) * scale) - sum(row_heights)
|
||||
if row_heights:
|
||||
row_heights[-1] += drift_h
|
||||
|
||||
sheet = Image.new("RGB", (canvas_w, canvas_h), bg)
|
||||
y = (canvas_h - sum(row_heights)) // 2
|
||||
for (row_imgs, aspects), row_h in zip(rows, row_heights):
|
||||
widths = [max(1, round(row_h * a)) for a in aspects]
|
||||
row_w = sum(widths)
|
||||
x = (canvas_w - row_w) // 2
|
||||
for im, w_i in zip(row_imgs, widths):
|
||||
resized = im.convert("RGB").resize((max(1, w_i), max(1, row_h)), Image.LANCZOS)
|
||||
sheet.paste(resized, (x, y))
|
||||
x += w_i
|
||||
y += row_h
|
||||
return sheet
|
||||
|
||||
|
||||
def _cover_justified_compose(imgs, rows_counts, canvas_w, canvas_h, bg):
|
||||
"""Row-justified layout, but COVER the canvas instead of contain -- fills both
|
||||
width and height completely, cropping the minimum necessary (like
|
||||
_cover_resize_crop, applied to the whole grid block instead of per-panel).
|
||||
Identical algorithm to build_multiref_sheet.py's _cover_justified_compose,
|
||||
kept in sync so inference-time sheets match training-time sheets exactly."""
|
||||
it = iter(imgs)
|
||||
rows = []
|
||||
for count in rows_counts:
|
||||
row_imgs = [next(it) for _ in range(count)]
|
||||
aspects = [im.width / im.height for im in row_imgs]
|
||||
rows.append((row_imgs, aspects))
|
||||
|
||||
natural_heights = [canvas_w / sum(aspects) for _, aspects in rows]
|
||||
h1 = sum(natural_heights)
|
||||
|
||||
if h1 >= canvas_h:
|
||||
# overfill: each row already fills canvas_w exactly at scale=1 (by construction
|
||||
# of natural_heights) -- keep that, crop the excess height after assembly.
|
||||
row_heights = [round(nh) for nh in natural_heights]
|
||||
block_w = canvas_w
|
||||
else:
|
||||
# underfill: scale UP so total height == canvas_h; every row becomes wider
|
||||
# than canvas_w by that same factor -- crop the excess width after assembly.
|
||||
scale = canvas_h / h1
|
||||
row_heights = [round(nh * scale) for nh in natural_heights]
|
||||
block_w = max(canvas_w, round(canvas_w * scale))
|
||||
|
||||
block = Image.new("RGB", (block_w, sum(row_heights)), bg)
|
||||
y = 0
|
||||
for (row_imgs, aspects), row_h in zip(rows, row_heights):
|
||||
widths = [max(1, round(row_h * a)) for a in aspects]
|
||||
row_w = sum(widths)
|
||||
x = (block_w - row_w) // 2
|
||||
for im, w_i in zip(row_imgs, widths):
|
||||
resized = im.convert("RGB").resize((max(1, w_i), max(1, row_h)), Image.LANCZOS)
|
||||
block.paste(resized, (x, y))
|
||||
x += w_i
|
||||
y += row_h
|
||||
|
||||
bw, bh = block.size
|
||||
x0 = max(0, (bw - canvas_w) // 2)
|
||||
y0 = max(0, (bh - canvas_h) // 2)
|
||||
return block.crop((x0, y0, x0 + canvas_w, y0 + canvas_h))
|
||||
|
||||
|
||||
def compose_sheet(imgs, panel_size=PANEL_SIZE, canvas_w=CANVAS_W, canvas_h=CANVAS_H, bg=BG_COLOR, fit_mode="crop"):
|
||||
"""fit_mode: 'crop' fills each fixed-size panel completely (crops excess, current
|
||||
default, matches the training data); 'fit' uses a row-justified layout that keeps
|
||||
every pixel of every reference (no cropping, no distortion) while maximizing
|
||||
canvas coverage (can underfill on one axis); 'cover' row-justifies AND fills the
|
||||
entire canvas on both axes, cropping the minimum shared/symmetric amount needed
|
||||
-- no background bars."""
|
||||
n = len(imgs)
|
||||
if not 1 <= n <= 5:
|
||||
raise ValueError(f"expected 1-5 reference images, got {n}")
|
||||
rows = LAYOUTS[n]
|
||||
|
||||
if fit_mode == "cover":
|
||||
return _cover_justified_compose(imgs, rows, canvas_w, canvas_h, bg)
|
||||
|
||||
if fit_mode == "fit":
|
||||
return _justified_compose(imgs, rows, canvas_w, canvas_h, bg)
|
||||
|
||||
native_w = max(rows) * panel_size
|
||||
native_h = len(rows) * panel_size
|
||||
native = Image.new("RGB", (native_w, native_h), bg)
|
||||
|
||||
it = iter(imgs)
|
||||
for row_idx, count in enumerate(rows):
|
||||
row_w = count * panel_size
|
||||
x_offset = (native_w - row_w) // 2 # center short rows (e.g. bottom row of a 5-ref sheet)
|
||||
y = row_idx * panel_size
|
||||
for col in range(count):
|
||||
panel = _cover_resize_crop(next(it), panel_size)
|
||||
x = x_offset + col * panel_size
|
||||
native.paste(panel, (x, y))
|
||||
|
||||
sheet = Image.new("RGB", (canvas_w, canvas_h), bg)
|
||||
px = (canvas_w - native_w) // 2
|
||||
py = (canvas_h - native_h) // 2
|
||||
sheet.paste(native, (px, py))
|
||||
return sheet
|
||||
|
||||
|
||||
class MultiRefSheetBuilder:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"fit_mode": (["crop", "fit", "cover"], {
|
||||
"default": "crop",
|
||||
"tooltip": "crop: zoom+center-crop to fill each panel completely (matches training data, "
|
||||
"may cut off edges). fit: scale each reference down to fit entirely inside its "
|
||||
"panel with no cropping (preserves every pixel, aspect ratio never distorted -- "
|
||||
"one uniform scale factor for both axes -- leftover space padded with background). "
|
||||
"cover: row-justified like fit, but fills the WHOLE 1536x1024 canvas on both axes "
|
||||
"(no background bars) by cropping the minimum shared amount needed.",
|
||||
}),
|
||||
},
|
||||
"optional": {
|
||||
"ref_image_1": ("IMAGE", {"tooltip": "image0 in the compositional prompt (anchor)."}),
|
||||
"ref_image_2": ("IMAGE", {"tooltip": "image1."}),
|
||||
"ref_image_3": ("IMAGE", {"tooltip": "image2."}),
|
||||
"ref_image_4": ("IMAGE", {"tooltip": "image3."}),
|
||||
"ref_image_5": ("IMAGE", {"tooltip": "image4."}),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "INT", "STRING")
|
||||
RETURN_NAMES = ("sheet", "n_refs", "debug")
|
||||
FUNCTION = "build"
|
||||
CATEGORY = CATEGORY
|
||||
DESCRIPTION = ("Combines 1-5 plugged-in reference images into the fixed 1536x1024 composite "
|
||||
"sheet the multi-ref LoRA was trained on. Leave slots empty for fewer refs; "
|
||||
"an empty slot is simply skipped, not padded with blank content.")
|
||||
|
||||
def build(self, fit_mode="crop", ref_image_1=None, ref_image_2=None, ref_image_3=None,
|
||||
ref_image_4=None, ref_image_5=None):
|
||||
slots = [ref_image_1, ref_image_2, ref_image_3, ref_image_4, ref_image_5]
|
||||
provided = [s for s in slots if s is not None]
|
||||
if not provided:
|
||||
raise ValueError("MultiRefSheetBuilder needs at least one ref_image_N input.")
|
||||
|
||||
pil_imgs = [tensor_to_pil(t[0] if t.dim() == 4 else t) for t in provided]
|
||||
sheet = compose_sheet(pil_imgs, fit_mode=fit_mode)
|
||||
sheet_t = pil_to_tensor(sheet).unsqueeze(0) # [1,H,W,C]
|
||||
|
||||
dbg = (f"MultiRefSheet | {len(provided)} refs -> {CANVAS_W}x{CANVAS_H} "
|
||||
f"({'+'.join(str(r) for r in LAYOUTS[len(provided)])} grid, fit_mode={fit_mode})")
|
||||
return (sheet_t, len(provided), dbg)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"BFSMultiRefSheetBuilder": MultiRefSheetBuilder}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"BFSMultiRefSheetBuilder": "Multi-Ref Sheet Builder"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entity Sheet 2x2 (HuMoSet v7 convention) -- SEMANTIC panel positions, unlike
|
||||
# MultiRefSheetBuilder's index-ordered justified layout. The humoset_sheets_v7
|
||||
# LoRA was trained on sheets where each quadrant has a fixed MEANING:
|
||||
# top-left = person identity top-right = clothing/garment
|
||||
# bottom-left = object OR 2nd person bottom-right = scene/background
|
||||
# Panels are 768x512, each ref letterboxed (fit, white padding) inside its
|
||||
# quadrant; unused quadrants stay pure white (training used white = "no info").
|
||||
# ---------------------------------------------------------------------------
|
||||
V7_PANEL_W, V7_PANEL_H = 768, 512
|
||||
|
||||
|
||||
def _v7_fit_panel(img: Image.Image) -> Image.Image:
|
||||
img = img.convert("RGB")
|
||||
s = min(V7_PANEL_W / img.width, V7_PANEL_H / img.height)
|
||||
nw, nh = max(1, round(img.width * s)), max(1, round(img.height * s))
|
||||
canvas = Image.new("RGB", (V7_PANEL_W, V7_PANEL_H), BG_COLOR)
|
||||
canvas.paste(img.resize((nw, nh), Image.LANCZOS), ((V7_PANEL_W - nw) // 2, (V7_PANEL_H - nh) // 2))
|
||||
return canvas
|
||||
|
||||
|
||||
class EntitySheet2x2:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {},
|
||||
"optional": {
|
||||
"person": ("IMAGE", {"tooltip": "Person identity reference -> TOP-LEFT quadrant."}),
|
||||
"clothing": ("IMAGE", {"tooltip": "Garment reference (isolated clothing item) -> TOP-RIGHT quadrant."}),
|
||||
"object_or_person2": ("IMAGE", {"tooltip": "Held object OR a second person -> BOTTOM-LEFT quadrant."}),
|
||||
"scene": ("IMAGE", {"tooltip": "Background/scene reference (no people) -> BOTTOM-RIGHT quadrant."}),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "INT", "STRING")
|
||||
RETURN_NAMES = ("sheet", "n_refs", "debug")
|
||||
FUNCTION = "build"
|
||||
CATEGORY = CATEGORY
|
||||
DESCRIPTION = ("Builds the FIXED 1536x1024 2x2 entity sheet the humoset_sheets_v7 multi-ref LoRA "
|
||||
"was trained on. Quadrant meaning is fixed (person TL, clothing TR, object/person2 BL, "
|
||||
"scene BR); leave inputs empty for white quadrants. Use with LTXIdentityOverlapConditioning: "
|
||||
"layout=overlap, source_id=2, phase_scale=1.0, ref_resize_mode=native_resolution.")
|
||||
|
||||
def build(self, person=None, clothing=None, object_or_person2=None, scene=None):
|
||||
entries = {(0, 0): person, (1, 0): clothing, (0, 1): object_or_person2, (1, 1): scene}
|
||||
n = sum(1 for v in entries.values() if v is not None)
|
||||
if n == 0:
|
||||
raise ValueError("EntitySheet2x2 needs at least one input image.")
|
||||
sheet = Image.new("RGB", (CANVAS_W, CANVAS_H), BG_COLOR)
|
||||
for (cx, cy), t in entries.items():
|
||||
if t is None:
|
||||
continue
|
||||
pil = tensor_to_pil(t[0] if t.dim() == 4 else t)
|
||||
sheet.paste(_v7_fit_panel(pil), (cx * V7_PANEL_W, cy * V7_PANEL_H))
|
||||
sheet_t = pil_to_tensor(sheet).unsqueeze(0)
|
||||
dbg = f"EntitySheet2x2 | {n} refs -> {CANVAS_W}x{CANVAS_H} (v7 semantic grid)"
|
||||
return (sheet_t, n, dbg)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS["BFSEntitySheet2x2"] = EntitySheet2x2
|
||||
NODE_DISPLAY_NAME_MAPPINGS["BFSEntitySheet2x2"] = "Entity Sheet 2x2 (HuMoSet v7)"
|
||||
+9
-3
@@ -1,13 +1,18 @@
|
||||
[project]
|
||||
name = "bfsnodes"
|
||||
description = "Custom ComfyUI nodes including LTXV Edit Anything and Head-Swap Bernini conditioning helpers."
|
||||
version = "1.5.0"
|
||||
license = {file = "LICENSE"}
|
||||
description = "Custom ComfyUI nodes for reference-guided video: BFS Sampler (guide-driven IC-LoRAs, auto crop, mask inpainting, temporal chunking), LTX Multiple Controls / Identity Transfer / Edit Anything, Head-Swap and Anime2Real conditioning, MiniMax-H3 latent guides, RoPE layouts and loaders, LoRA Surgery, and Shot Planner / Shot Loop (tabbed panel): split long videos into shots at camera cuts, face-tracked cast, SAM 3 mask per shot with full frame, paste, mask, crop and crop + mask generation modes, optional translucent Color Guide for one or two manually selected regions with cast identity color reuse, optional Qwen3-VL to suggest settings, describe references ({details}) and the person to replace ({target}), continuity between shots, H3 conditioning with an optional duet and prompt writer, join with the original timing, audio and a side-by-side comparison. H3 Duet: pin a source clip beside the video, training-free."
|
||||
version = "1.97.0"
|
||||
license = {file = "LICENSE"}
|
||||
dependencies = [
|
||||
"torch",
|
||||
"numpy",
|
||||
"Pillow",
|
||||
"safetensors",
|
||||
"librosa",
|
||||
"opencv-python",
|
||||
"scenedetect",
|
||||
"transformers>=4.55.0",
|
||||
"scipy",
|
||||
]
|
||||
# classifiers = [
|
||||
# # For OS-independent nodes (works on all operating systems)
|
||||
@@ -34,6 +39,7 @@ Documentation = "https://github.com/alisson-anjos/ComfyUI-BFSNodes/wiki"
|
||||
"Bug Tracker" = "https://github.com/alisson-anjos/ComfyUI-BFSNodes/issues"
|
||||
|
||||
[tool.comfy]
|
||||
web = "web"
|
||||
PublisherId = "alisson-anjos"
|
||||
DisplayName = "ComfyUI-BFSNodes"
|
||||
Icon = ""
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
"""Shared temporal placement helpers for clean LTX reference tokens.
|
||||
|
||||
LTX VAEs normally compress eight pixel frames into one latent-time step. The
|
||||
public node input is expressed in latent steps so ``-1`` consistently means
|
||||
"one VAE temporal step before frame zero" without hard-coding a particular
|
||||
VAE scale factor.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import torch
|
||||
|
||||
DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS = 0
|
||||
|
||||
|
||||
def reference_temporal_offset_input():
|
||||
"""Return a fresh ComfyUI input declaration for reference-only placement."""
|
||||
return (
|
||||
"INT",
|
||||
{
|
||||
"default": DEFAULT_REFERENCE_TEMPORAL_OFFSET_LATENTS,
|
||||
"min": -16,
|
||||
"max": 16,
|
||||
"step": 1,
|
||||
"tooltip": (
|
||||
"Temporal RoPE offset for appearance/reference tokens only, in VAE latent-time steps. "
|
||||
"0 keeps the standard frame-0 placement used by existing LoRAs. "
|
||||
"-1 places the reference one latent step before frame 0 (normally -8 pixel frames), "
|
||||
"which can be tested to reduce overlap frame-0 reference leakage. "
|
||||
"Guide, mask, and target positions are never changed."
|
||||
),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def reference_temporal_pixel_offset(
|
||||
offset_latents: int, temporal_scale_factor: int
|
||||
) -> int:
|
||||
"""Convert a latent-step offset to the raw pixel-frame units used by LTX RoPE."""
|
||||
return int(offset_latents) * int(temporal_scale_factor)
|
||||
|
||||
|
||||
def shift_reference_temporal_positions(
|
||||
positions: torch.Tensor,
|
||||
offset_latents: int,
|
||||
temporal_scale_factor: int,
|
||||
) -> torch.Tensor:
|
||||
"""Shift only the temporal axis of a reference coordinate tensor.
|
||||
|
||||
The helper accepts both ComfyUI's ``[B, axes, tokens]`` coordinates and
|
||||
trainer-style tensors with extra trailing dimensions. A zero offset is an
|
||||
exact no-op and returns the original tensor for backward-compatible specs.
|
||||
"""
|
||||
pixel_offset = reference_temporal_pixel_offset(
|
||||
offset_latents, temporal_scale_factor
|
||||
)
|
||||
if pixel_offset == 0:
|
||||
return positions
|
||||
if positions.ndim < 3 or positions.shape[1] < 1:
|
||||
raise ValueError(
|
||||
"Reference temporal positions must have shape [batch, axes, ...] "
|
||||
f"with a temporal axis; got {tuple(positions.shape)}"
|
||||
)
|
||||
shifted = positions.clone()
|
||||
shifted[:, 0, ...] += torch.as_tensor(
|
||||
pixel_offset,
|
||||
device=shifted.device,
|
||||
dtype=shifted.dtype,
|
||||
)
|
||||
return shifted
|
||||
@@ -2,3 +2,6 @@ torch
|
||||
numpy
|
||||
Pillow
|
||||
safetensors
|
||||
librosa
|
||||
opencv-python
|
||||
scenedetect
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
"""Salva um latente do MiniMax-H3 em disco.
|
||||
|
||||
O `SaveLatent` nativo quebra com o latente do H3: ele chega como NestedTensor (video + audio)
|
||||
e o node chama `.contiguous()` direto. Aqui os componentes sao desempacotados e salvos
|
||||
separadamente, para o latente poder ser inspecionado ou decodificado fora do grafo.
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
import folder_paths
|
||||
import torch
|
||||
from safetensors.torch import save_file
|
||||
|
||||
|
||||
class SaveH3Latent:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"samples": ("LATENT",),
|
||||
"filename_prefix": ("STRING", {"default": "h3_latent"}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ()
|
||||
FUNCTION = "save"
|
||||
OUTPUT_NODE = True
|
||||
CATEGORY = "BFS/debug"
|
||||
|
||||
def save(self, samples, filename_prefix):
|
||||
z = samples["samples"]
|
||||
tensores = {}
|
||||
if getattr(z, "is_nested", False):
|
||||
for i, parte in enumerate(z.unbind()):
|
||||
t = parte if parte.dim() == 5 else parte.unsqueeze(0)
|
||||
tensores[f"component_{i}"] = t.contiguous().cpu()
|
||||
else:
|
||||
t = z if z.dim() == 5 else z.unsqueeze(0)
|
||||
tensores["component_0"] = t.contiguous().cpu()
|
||||
|
||||
pasta = folder_paths.get_output_directory()
|
||||
n = 0
|
||||
while os.path.exists(os.path.join(pasta, f"{filename_prefix}_{n:05d}.safetensors")):
|
||||
n += 1
|
||||
caminho = os.path.join(pasta, f"{filename_prefix}_{n:05d}.safetensors")
|
||||
meta = {k: str(tuple(v.shape)) for k, v in tensores.items()}
|
||||
save_file(tensores, caminho, metadata={"format": "pt", **meta})
|
||||
print(f"[BFSNodes] latente salvo em {caminho}")
|
||||
for k, v in tensores.items():
|
||||
print(f"[BFSNodes] {k}: {tuple(v.shape)} {v.dtype}")
|
||||
return {}
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"SaveH3Latent": SaveH3Latent}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"SaveH3Latent": "Save H3 Latent / nested-safe (BFS)"}
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Escolhe o frame mais nitido de um lote, por variancia do Laplaciano.
|
||||
|
||||
Vem do IF-Edit (arXiv:2511.19435): frames decodificados carregam quantidades diferentes de
|
||||
borrao residual, entao pontuar e ficar com o mais crocante bate pegar um indice fixo.
|
||||
|
||||
Util em particular com o MiniMax-H3 usado como gerador de imagem: o clipe minimo do VAE
|
||||
(5 frames = 2 latentes) sai do decode com nitidez desigual, e o melhor frame nao e sempre
|
||||
o mesmo indice.
|
||||
|
||||
CUIDADO -- e uma heuristica de BORRAO, nao uma metrica de qualidade. Variancia do
|
||||
Laplaciano mede energia de alta frequencia, e artefato tambem e alta frequencia. Medido
|
||||
num decode do H3: entre 4 frames do mesmo bloco, o Laplaciano escolheu o de PIOR PSNR
|
||||
(19.72 dB contra 23.19 do vizinho), porque ele era o mais artefatado. Use quando os
|
||||
candidatos forem o MESMO conteudo diferindo so em borrao -- que e o caso do IF-Edit e do
|
||||
clipe estatico gerado. Nao use para escolher entre frames de momentos diferentes.
|
||||
"""
|
||||
|
||||
import torch
|
||||
|
||||
|
||||
def _laplacian_variance(img: torch.Tensor) -> float:
|
||||
"""img: [H, W, C] em [0,1]. Variancia do Laplaciano 4-vizinhos sobre a luminancia."""
|
||||
weights = torch.tensor([0.299, 0.587, 0.114], device=img.device, dtype=torch.float32)
|
||||
g = (img[..., :3].float() * weights).sum(-1)
|
||||
lap = (
|
||||
-4.0 * g[1:-1, 1:-1]
|
||||
+ g[:-2, 1:-1] + g[2:, 1:-1] + g[1:-1, :-2] + g[1:-1, 2:]
|
||||
)
|
||||
return float(lap.var())
|
||||
|
||||
|
||||
class SharpestFrame:
|
||||
"""Recebe um lote de imagens e devolve a mais nitida, com o indice e as notas."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"images": ("IMAGE", {"tooltip": "Lote de frames que sejam o MESMO conteudo diferindo em borrao (ex.: clipe estatico do H3 em modo imagem). Para frames de momentos diferentes isso nao faz sentido."}),
|
||||
},
|
||||
"optional": {
|
||||
"skip_first": ("INT", {"default": 0, "min": 0, "max": 64, "tooltip": "Ignora os N primeiros frames antes de pontuar. No H3 os primeiros frames de cada bloco decodificado costumam ser preenchimento temporal."}),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("IMAGE", "INT", "STRING")
|
||||
RETURN_NAMES = ("image", "index", "scores")
|
||||
FUNCTION = "pick"
|
||||
CATEGORY = "BFS/image"
|
||||
|
||||
def pick(self, images, skip_first=0):
|
||||
total = images.shape[0]
|
||||
start = min(skip_first, max(total - 1, 0))
|
||||
scores = [_laplacian_variance(images[i]) for i in range(start, total)]
|
||||
best = start + max(range(len(scores)), key=scores.__getitem__)
|
||||
texto = " | ".join(f"{start + i}:{s:.5f}{' <-' if start + i == best else ''}" for i, s in enumerate(scores))
|
||||
print(f"[BFSNodes] SharpestFrame: {total} frames, escolhido {best} -> {texto}")
|
||||
return (images[best:best + 1], best, texto)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"SharpestFrame": SharpestFrame}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"SharpestFrame": "Sharpest Frame / Laplacian (BFS)"}
|
||||
+176
@@ -0,0 +1,176 @@
|
||||
"""CPU-only layout excerpt from ComfyUI 8cfe5e1.
|
||||
Source: comfy/ldm/minimax/model.py, https://github.com/Comfy-Org/ComfyUI
|
||||
Copyright ComfyUI contributors; GPL-3.0, same as this repository.
|
||||
Original implementation retained to test the actual audited native layout API.
|
||||
"""
|
||||
import math
|
||||
import torch
|
||||
FRAME_PER_TOKEN = (1, 4, 4, 4, 4)
|
||||
FRAME_RESCALE = 5.0 / 3.0
|
||||
|
||||
def _axis_from_sqrt_area(dim, patch, sqrt_area):
|
||||
# linspace((1 - ratio) / 2, (1 + ratio) / 2, dim // patch, endpoint=False) * 32
|
||||
ratio = dim / sqrt_area
|
||||
n = dim // patch
|
||||
return (torch.arange(n, dtype=torch.float64) * (ratio / n) + (1.0 - ratio) / 2.0) * 32.0
|
||||
|
||||
def _frame_grid(h, w):
|
||||
# area-normalized (h, w) coordinates of one latent frame's 2x2-patch rows
|
||||
area = math.sqrt(h * w)
|
||||
hh, ww = torch.meshgrid(_axis_from_sqrt_area(h, 2, area), _axis_from_sqrt_area(w, 2, area), indexing="ij")
|
||||
return torch.stack([hh.reshape(-1), ww.reshape(-1)], dim=-1), _axis_from_sqrt_area(w, 2, area)
|
||||
|
||||
def _video_t_spans(n):
|
||||
return [FRAME_RESCALE * FRAME_PER_TOKEN[k % 5] for k in range(n)]
|
||||
|
||||
def _video_t_grid(n, origin):
|
||||
# origin + exclusive cumsum
|
||||
spans = torch.tensor(_video_t_spans(n), dtype=torch.float64)
|
||||
return float(origin) + torch.cat([torch.zeros(1, dtype=torch.float64), spans[:-1].cumsum(0)])
|
||||
|
||||
def _ref_t_span(blk):
|
||||
# time-axis span a reference block occupies ahead of the target streams
|
||||
kind = blk["kind"]
|
||||
if kind == "image":
|
||||
return 1.0
|
||||
if kind == "audio":
|
||||
return float(blk["ref_audio_t"])
|
||||
if kind in ("video", "video_audio"):
|
||||
return max(float(blk["ref_audio_t"]), sum(_video_t_spans(blk["latent_t"])))
|
||||
return 0.0
|
||||
|
||||
def _audio_grid(cursor, t, w_low, w_high):
|
||||
# channel-major stereo rows: t advances per latent frame, w pinned to the grid extremes per stereo channel, h stays 0
|
||||
g = torch.zeros(t * 2, 3, dtype=torch.float64)
|
||||
g[:, 0] = (cursor + torch.arange(t, dtype=torch.float64)).repeat(2)
|
||||
g[:t, 2] = w_low
|
||||
g[t:, 2] = w_high
|
||||
return g
|
||||
|
||||
def _video_grid(vt, frame, cursor):
|
||||
g = torch.empty(vt, frame.shape[0], 3, dtype=torch.float64)
|
||||
g[:, :, 0] = _video_t_grid(vt, cursor)[:, None]
|
||||
g[:, :, 1:] = frame[None]
|
||||
return g.reshape(-1, 3)
|
||||
|
||||
class PackedLayout:
|
||||
"""Static packed-sequence structure for one shape/conditioning signature."""
|
||||
|
||||
def __init__(self, text_len, latent_t, latent_h, latent_w, audio_t, keyframes=None, refs=None):
|
||||
frame, w_grid = _frame_grid(latent_h, latent_w)
|
||||
frame_rows = frame.shape[0]
|
||||
|
||||
segments = [("text", text_len)] # (kind, n_rows)
|
||||
g = torch.zeros(text_len, 3, dtype=torch.float64)
|
||||
g[:, 0] = torch.arange(text_len, dtype=torch.float64)
|
||||
pos = [g] # per segment: [n, 3] float64 (t, h, w)
|
||||
|
||||
img_pos, img_update = [], []
|
||||
audio_pos, audio_update = [], []
|
||||
row = text_len
|
||||
|
||||
target_audio_w = (float(w_grid[0]), float(w_grid[-1]))
|
||||
# refs pack between text and the targets, so the target timeline starts after their spans
|
||||
cursor = float(text_len)
|
||||
for blk in refs or ():
|
||||
cursor += _ref_t_span(blk)
|
||||
|
||||
if keyframes:
|
||||
# fl2va: keyframe cond rows right after text, sharing the target spatial grid;
|
||||
# anchors count from the target timeline origin, FRAME_RESCALE per pixel frame, 1.0 per audio latent frame
|
||||
for kf in keyframes:
|
||||
cond_t = cursor + FRAME_RESCALE * kf["resolved_frame_index"]
|
||||
video_latent = kf.get("latent")
|
||||
if video_latent is not None:
|
||||
vt = video_latent.shape[2]
|
||||
n = vt * frame_rows
|
||||
segments.append(("cond", n))
|
||||
pos.append(_video_grid(vt, frame, cond_t))
|
||||
img_pos.append(torch.arange(row, row + n))
|
||||
img_update.append(torch.zeros(n, dtype=torch.bool))
|
||||
row += n
|
||||
audio_latent = kf.get("audio_latent")
|
||||
if audio_latent is not None:
|
||||
rt = audio_latent.shape[-1]
|
||||
segments.append(("cond_audio", rt * 2))
|
||||
pos.append(_audio_grid(cond_t, rt, *target_audio_w))
|
||||
audio_pos.append(torch.arange(row, row + rt * 2))
|
||||
audio_update.append(torch.zeros(rt * 2, dtype=torch.bool))
|
||||
row += rt * 2
|
||||
|
||||
if refs:
|
||||
cursor = float(text_len)
|
||||
for blk in refs:
|
||||
kind = blk["kind"]
|
||||
if kind == "image":
|
||||
r_frame, _ = _frame_grid(blk["latent_h"], blk["latent_w"])
|
||||
n = r_frame.shape[0]
|
||||
g = torch.empty(n, 3, dtype=torch.float64)
|
||||
g[:, 0] = cursor
|
||||
g[:, 1:] = r_frame
|
||||
segments.append(("ref_img", n))
|
||||
pos.append(g)
|
||||
img_pos.append(torch.arange(row, row + n))
|
||||
img_update.append(torch.zeros(n, dtype=torch.bool))
|
||||
row += n
|
||||
cursor += 1.0
|
||||
elif kind == "audio":
|
||||
rt = blk["ref_audio_t"]
|
||||
if rt > 0:
|
||||
segments.append(("ref_audio", rt * 2))
|
||||
pos.append(_audio_grid(cursor, rt, *target_audio_w))
|
||||
audio_pos.append(torch.arange(row, row + rt * 2))
|
||||
audio_update.append(torch.zeros(rt * 2, dtype=torch.bool))
|
||||
row += rt * 2
|
||||
cursor += float(rt)
|
||||
elif kind in ("video", "video_audio"):
|
||||
# the block's audio rows pack immediately before its video
|
||||
# rows, both sharing the cursor origin
|
||||
rt = blk["ref_audio_t"]
|
||||
vt = blk["latent_t"]
|
||||
r_frame, r_w_grid = _frame_grid(blk["latent_h"], blk["latent_w"])
|
||||
if rt > 0:
|
||||
segments.append(("ref_audio", rt * 2))
|
||||
pos.append(_audio_grid(cursor, rt, float(r_w_grid[0]), float(r_w_grid[-1])))
|
||||
audio_pos.append(torch.arange(row, row + rt * 2))
|
||||
audio_update.append(torch.zeros(rt * 2, dtype=torch.bool))
|
||||
row += rt * 2
|
||||
n = vt * r_frame.shape[0]
|
||||
segments.append(("ref_img", n))
|
||||
pos.append(_video_grid(vt, r_frame, cursor))
|
||||
img_pos.append(torch.arange(row, row + n))
|
||||
img_update.append(torch.zeros(n, dtype=torch.bool))
|
||||
row += n
|
||||
cursor += max(float(rt), sum(_video_t_spans(vt)))
|
||||
|
||||
# target audio then target video, always the last two segments
|
||||
segments.append(("audio", audio_t * 2))
|
||||
pos.append(_audio_grid(cursor, audio_t, *target_audio_w))
|
||||
audio_pos.append(torch.arange(row, row + audio_t * 2))
|
||||
audio_update.append(torch.ones(audio_t * 2, dtype=torch.bool))
|
||||
row += audio_t * 2
|
||||
|
||||
n_video = latent_t * frame_rows
|
||||
segments.append(("video", n_video))
|
||||
pos.append(_video_grid(latent_t, frame, cursor))
|
||||
img_pos.append(torch.arange(row, row + n_video))
|
||||
img_update.append(torch.ones(n_video, dtype=torch.bool))
|
||||
row += n_video
|
||||
|
||||
self.seq_len = row
|
||||
self.position_ids = torch.cat(pos) # [S, 3] float64
|
||||
self.img_pos = torch.cat(img_pos)
|
||||
self.img_update = torch.cat(img_update)
|
||||
self.audio_pos = torch.cat(audio_pos)
|
||||
self.audio_update = torch.cat(audio_update)
|
||||
self.signature = (text_len, latent_t, latent_h, latent_w, audio_t)
|
||||
# contiguous segment table (start, stop, kind)
|
||||
# kinds: text / cond / cond_audio / ref_img / ref_audio / audio / video
|
||||
# the packed sequence is uniform per segment in (modality tag, timestep class),
|
||||
# except the text span (tag runs resolved at forward time from the presentation tags)
|
||||
seg_abs = []
|
||||
off = 0
|
||||
for kind, n in segments:
|
||||
seg_abs.append((off, off + n, kind))
|
||||
off += n
|
||||
self.segments = seg_abs
|
||||
+974
@@ -0,0 +1,974 @@
|
||||
{
|
||||
"source": "alisson-anjos/ai-toolkit e17e21c src/packing.py, target_grid_stride_v1, text=3 target=2x8x8 audio=2 guide=2x(8/f)x(8/f)",
|
||||
"factors": {
|
||||
"1": {
|
||||
"positions": [
|
||||
[
|
||||
0.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
1.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
2.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
8.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
8.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
8.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
8.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
24.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
24.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
24.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
24.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
8.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
8.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
8.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
8.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
24.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
24.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
24.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
24.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.0,
|
||||
0.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
8.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
8.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
8.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
8.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
24.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
24.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
24.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
24.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
8.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
8.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
8.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
8.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
24.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
24.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
24.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
24.0,
|
||||
24.0
|
||||
]
|
||||
],
|
||||
"video_indices": [
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
11,
|
||||
12,
|
||||
13,
|
||||
14,
|
||||
15,
|
||||
16,
|
||||
17,
|
||||
18,
|
||||
19,
|
||||
20,
|
||||
21,
|
||||
22,
|
||||
23,
|
||||
24,
|
||||
25,
|
||||
26,
|
||||
27,
|
||||
28,
|
||||
29,
|
||||
30,
|
||||
31,
|
||||
32,
|
||||
33,
|
||||
34,
|
||||
39,
|
||||
40,
|
||||
41,
|
||||
42,
|
||||
43,
|
||||
44,
|
||||
45,
|
||||
46,
|
||||
47,
|
||||
48,
|
||||
49,
|
||||
50,
|
||||
51,
|
||||
52,
|
||||
53,
|
||||
54,
|
||||
55,
|
||||
56,
|
||||
57,
|
||||
58,
|
||||
59,
|
||||
60,
|
||||
61,
|
||||
62,
|
||||
63,
|
||||
64,
|
||||
65,
|
||||
66,
|
||||
67,
|
||||
68,
|
||||
69,
|
||||
70
|
||||
],
|
||||
"audio_indices": [
|
||||
35,
|
||||
36,
|
||||
37,
|
||||
38
|
||||
]
|
||||
},
|
||||
"2": {
|
||||
"positions": [
|
||||
[
|
||||
0.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
1.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
2.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.0,
|
||||
0.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
8.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
8.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
8.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
8.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
24.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
24.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
24.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
24.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
8.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
8.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
8.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
8.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
24.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
24.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
24.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
24.0,
|
||||
24.0
|
||||
]
|
||||
],
|
||||
"video_indices": [
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8,
|
||||
9,
|
||||
10,
|
||||
15,
|
||||
16,
|
||||
17,
|
||||
18,
|
||||
19,
|
||||
20,
|
||||
21,
|
||||
22,
|
||||
23,
|
||||
24,
|
||||
25,
|
||||
26,
|
||||
27,
|
||||
28,
|
||||
29,
|
||||
30,
|
||||
31,
|
||||
32,
|
||||
33,
|
||||
34,
|
||||
35,
|
||||
36,
|
||||
37,
|
||||
38,
|
||||
39,
|
||||
40,
|
||||
41,
|
||||
42,
|
||||
43,
|
||||
44,
|
||||
45,
|
||||
46
|
||||
],
|
||||
"audio_indices": [
|
||||
11,
|
||||
12,
|
||||
13,
|
||||
14
|
||||
]
|
||||
},
|
||||
"4": {
|
||||
"positions": [
|
||||
[
|
||||
0.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
1.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
2.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.0,
|
||||
0.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
0.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
8.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
8.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
8.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
8.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
16.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
24.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
24.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
24.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
3.0,
|
||||
24.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
0.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
8.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
8.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
8.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
8.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
16.0,
|
||||
24.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
24.0,
|
||||
0.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
24.0,
|
||||
8.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
24.0,
|
||||
16.0
|
||||
],
|
||||
[
|
||||
4.666666666666667,
|
||||
24.0,
|
||||
24.0
|
||||
]
|
||||
],
|
||||
"video_indices": [
|
||||
3,
|
||||
4,
|
||||
9,
|
||||
10,
|
||||
11,
|
||||
12,
|
||||
13,
|
||||
14,
|
||||
15,
|
||||
16,
|
||||
17,
|
||||
18,
|
||||
19,
|
||||
20,
|
||||
21,
|
||||
22,
|
||||
23,
|
||||
24,
|
||||
25,
|
||||
26,
|
||||
27,
|
||||
28,
|
||||
29,
|
||||
30,
|
||||
31,
|
||||
32,
|
||||
33,
|
||||
34,
|
||||
35,
|
||||
36,
|
||||
37,
|
||||
38,
|
||||
39,
|
||||
40
|
||||
],
|
||||
"audio_indices": [
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
import importlib.util
|
||||
import types
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SPEC = importlib.util.spec_from_file_location("bfs_h3_history", ROOT / "bfs_h3_history.py")
|
||||
HI = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(HI)
|
||||
|
||||
|
||||
class Layout:
|
||||
"""text 4 rows | guide cond 6 rows | history cond 4 rows | audio 3 | video 6, all at the target origin 4.0"""
|
||||
|
||||
def __init__(self):
|
||||
self.segments = [(0, 4, "text"), (4, 10, "cond"), (10, 14, "cond"), (14, 17, "audio"), (17, 23, "video")]
|
||||
p = torch.zeros(23, 3)
|
||||
p[:4, 0] = torch.arange(4.)
|
||||
p[4:, 0] = 4.0
|
||||
self.position_ids = p
|
||||
|
||||
|
||||
class HistoryTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
HI.history_span = lambda t: 10.0 * t
|
||||
|
||||
def test_target_moves_past_the_history_only(self):
|
||||
lay = Layout()
|
||||
kfs = [{"latent": torch.zeros(1, 24, 3, 2, 2)}, {"latent": torch.zeros(1, 24, 2, 2, 2), "anchor": "history"}]
|
||||
self.assertTrue(HI.relocate(lay, kfs))
|
||||
t = lay.position_ids[:, 0]
|
||||
self.assertTrue(torch.equal(t[:4], torch.arange(4.))) # text untouched
|
||||
self.assertTrue(bool((t[10:14] == 4.0).all())) # history stays at the origin
|
||||
self.assertTrue(bool((t[4:10] == 24.0).all())) # the other guide moves with the video
|
||||
self.assertTrue(bool((t[14:] == 24.0).all())) # audio and video move by the span (2 steps)
|
||||
self.assertFalse(HI.relocate(lay, kfs)) # already moved: no double shift
|
||||
|
||||
def test_no_history_no_change(self):
|
||||
lay = Layout()
|
||||
before = lay.position_ids.clone()
|
||||
self.assertFalse(HI.relocate(lay, [{"latent": torch.zeros(1, 24, 3, 2, 2)}]))
|
||||
self.assertTrue(torch.equal(before, lay.position_ids))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,163 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
import types
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
class _Nested:
|
||||
is_nested = True
|
||||
|
||||
def __init__(self, tensors):
|
||||
self.tensors = list(tensors)
|
||||
|
||||
def unbind(self):
|
||||
return self.tensors
|
||||
|
||||
|
||||
# stub the two comfy modules the node uses, so the tests run without ComfyUI
|
||||
_comfy = types.ModuleType("comfy")
|
||||
_nt = types.ModuleType("comfy.nested_tensor")
|
||||
_nt.NestedTensor = _Nested
|
||||
_ut = types.ModuleType("comfy.utils")
|
||||
_ut.reshape_mask = lambda m, shape: torch.nn.functional.interpolate(m.reshape(1, 1, *m.shape[-3:]).float(), size=shape[2:])
|
||||
_comfy.nested_tensor, _comfy.utils = _nt, _ut
|
||||
for name, mod in (("comfy", _comfy), ("comfy.nested_tensor", _nt), ("comfy.utils", _ut)):
|
||||
sys.modules.setdefault(name, mod)
|
||||
|
||||
SPEC = importlib.util.spec_from_file_location("bfs_h3_side_panel", ROOT / "bfs_h3_side_panel.py")
|
||||
SP = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(SP)
|
||||
|
||||
|
||||
class FakeVAE:
|
||||
"""[N,H,W,3] -> [1,24,T,H/16,W/16] with the H3 frame grid; the mean colour in every channel."""
|
||||
|
||||
def encode(self, px):
|
||||
n = px.shape[0]
|
||||
t = 1 if n == 1 else ((n - 5) // 17) * 5 + 2
|
||||
x = px.mean(-1)[None, None] # [1,1,N,H,W]
|
||||
x = torch.nn.functional.interpolate(x, size=(t, px.shape[1] // 16, px.shape[2] // 16))
|
||||
return x.repeat(1, 24, 1, 1, 1)
|
||||
|
||||
def decode(self, z):
|
||||
t, h, w = z.shape[2:]
|
||||
return torch.full((1, SP.frame_count_of(t) if t > 1 else 1, h * 16, w * 16, 3), float(z.mean()))
|
||||
|
||||
|
||||
def _latent(w=576, h=1024, frames=22):
|
||||
t = ((frames - 5) // 17) * 5 + 2
|
||||
return {"samples": _Nested((torch.zeros(1, 24, t, h // 16, w // 16), torch.zeros(1, 32, 2, 37)))}
|
||||
|
||||
|
||||
class SidePanelTest(unittest.TestCase):
|
||||
def test_strip_sizes_snap_to_patches(self):
|
||||
self.assertEqual(SP.strip_size(576, 1024, "left", 1.0, 0), (576, 1024, 576, 1024))
|
||||
self.assertEqual(SP.strip_size(576, 1024, "top", 0.33, 32), (576, 384, 576, 352))
|
||||
|
||||
def test_mask_holds_only_the_strip(self):
|
||||
info = {"position": "left", "h": 4, "w": 6, "strip_h": 0, "strip_w": 2}
|
||||
m = SP.panel_mask(info, 3, "all frames", panel_noise=0.1)
|
||||
self.assertEqual(tuple(m.shape), (1, 1, 3, 4, 8))
|
||||
self.assertTrue(torch.allclose(m[..., :2], torch.tensor(0.1)))
|
||||
self.assertTrue(bool((m[..., 2:] == 1).all()))
|
||||
first = SP.panel_mask(info, 3, "first latent frame")
|
||||
self.assertEqual(float(first[0, 0, 1, 0, 0]), 1.0)
|
||||
|
||||
def test_layout_text_follows_the_share(self):
|
||||
half = SP.layout_text({"position": "left", "h": 64, "w": 36, "strip_h": 0, "strip_w": 36})
|
||||
self.assertIn("the left half shows the original performance; the right half shows", half)
|
||||
self.assertNotIn("kept", half)
|
||||
narrow = SP.layout_text({"position": "left", "h": 64, "w": 72, "strip_h": 0, "strip_w": 36})
|
||||
self.assertIn("the left third shows", narrow)
|
||||
self.assertIn("the right two thirds show", narrow)
|
||||
self.assertIn("a wider view of the same action", narrow)
|
||||
top = {"position": "top", "h": 48, "w": 36, "strip_h": 16, "strip_w": 0}
|
||||
txt = SP.fill_layout("{panel_region} shows a man dancing, and {video_region} shows <Subject 1>.", top)
|
||||
self.assertEqual(txt, "the top quarter shows a man dancing, and the bottom three quarters show <Subject 1>.")
|
||||
p = SP.build_prompt("character swap", "", 1, 0)
|
||||
for w in ("kept footage", "kept exactly", "in sync", "generated"):
|
||||
self.assertNotIn(w, SP.fill_layout(p, top))
|
||||
|
||||
def test_apply_and_crop_round_trip(self):
|
||||
lat = _latent()
|
||||
panel = torch.full((22, 512, 512, 3), 0.9)
|
||||
guide = torch.full((22, 1024, 576, 3), 0.2)
|
||||
pos, out, info, preview, text = SP.BFSH3SidePanel().apply(
|
||||
[[torch.zeros(1), {}]], lat, FakeVAE(), panel, "left", 1.0, "contain", 0, "all frames", 0.0, guide, 0)
|
||||
v = out["samples"].tensors[0]
|
||||
self.assertEqual(tuple(v.shape), (1, 24, 7, 64, 72))
|
||||
kf = pos[0][1]["minimax_keyframes"][0]
|
||||
self.assertEqual(tuple(kf["latent"].shape[-2:]), (64, 72))
|
||||
self.assertEqual(tuple(preview.shape), (1, 1024, 1152, 3))
|
||||
cropped = SP.BFSH3SidePanelCrop().crop(info, latent=out, images=torch.zeros(5, 1024, 1152, 3))
|
||||
self.assertEqual(tuple(cropped[0]["samples"].tensors[0].shape), (1, 24, 7, 64, 36))
|
||||
self.assertNotIn("noise_mask", cropped[0])
|
||||
self.assertEqual(tuple(cropped[1].shape), (5, 1024, 576, 3))
|
||||
|
||||
def test_existing_guides_are_moved_onto_the_canvas(self):
|
||||
lat = _latent()
|
||||
kf = {"resolved_frame_index": 0, "latent": torch.zeros(1, 24, 1, 64, 36)}
|
||||
pos = SP.BFSH3SidePanel().apply([[torch.zeros(1), {"minimax_keyframes": [kf]}]], lat, FakeVAE(),
|
||||
torch.ones(1, 64, 64, 3), "top", 0.5, "cover", 1, "all frames")[0]
|
||||
self.assertEqual(tuple(pos[0][1]["minimax_keyframes"][0]["latent"].shape[-2:]), (64 + 34, 36))
|
||||
|
||||
|
||||
class KeepMaskTest(unittest.TestCase):
|
||||
def test_latent_mask_follows_the_h3_frame_grid(self):
|
||||
m = torch.zeros(22, 64, 64)
|
||||
m[5:9, 10:20, 10:20] = 1 # frames 5-8 = latent step 2 (1 + 4 frames before it)
|
||||
lm = SP.latent_mask(m, 7, 4, 4, grow=0)
|
||||
self.assertEqual(tuple(lm.shape), (1, 1, 7, 4, 4))
|
||||
self.assertEqual(float(lm[0, 0, 2].max()), 1.0)
|
||||
self.assertEqual(float(lm[0, 0, 1].max()), 0.0)
|
||||
|
||||
def test_apply_with_keep_mask_regenerates_only_the_mask(self):
|
||||
lat = _latent()
|
||||
panel = torch.full((22, 512, 512, 3), 0.9)
|
||||
km = torch.zeros(22, 1024, 576); km[:, 300:700, 200:400] = 1
|
||||
out = SP.BFSH3SidePanel().apply([[torch.zeros(1), {}]], lat, FakeVAE(), panel, "left", 1.0, "contain", 0,
|
||||
"all frames", 0.0, None, 0, keep_video=torch.full((22, 1024, 576, 3), 0.3), keep_mask=km)[1]
|
||||
m = out["noise_mask"].tensors[0]
|
||||
self.assertEqual(float(m[0, 0, 3, 0, 40]), 0.0) # video area outside the mask: kept
|
||||
self.assertEqual(float(m[0, 0, 3, 30, 36 + 18]), 1.0) # inside the mask: generated
|
||||
self.assertEqual(float(m[0, 0, 3, 30, 5]), 0.0) # panel: pinned
|
||||
|
||||
def test_panel_static_covers_the_subject_every_frame(self):
|
||||
seen = {}
|
||||
|
||||
def seg(imgs, spec):
|
||||
seen["spec"] = spec
|
||||
m = torch.zeros(imgs.shape[0], *imgs.shape[1:3]); m[:, 10:20, 10:20] = 1
|
||||
return m
|
||||
|
||||
def tv(img, mask, grow=0.03, seed=0):
|
||||
out = img.clone(); m = torch.nn.functional.interpolate(mask[None, None].float(), size=img.shape[1:3])[0, 0]
|
||||
out[0][m > 0.5] = -1.0
|
||||
return out
|
||||
|
||||
fake = types.SimpleNamespace(DEFAULT_MASK={"text": "", "threshold": 0.5}, segment_frames=seg, tv_static=tv,
|
||||
add_setting=lambda text, k, swap: text + f" <Picture {k}> setting")
|
||||
old, SP._shot_loop = SP._shot_loop, lambda: fake
|
||||
try:
|
||||
fr = torch.full((5, 64, 48, 3), 0.5)
|
||||
self.assertIs(SP.static_over_panel(fr, "off"), fr)
|
||||
out = SP.static_over_panel(fr, "face")
|
||||
self.assertEqual(seen["spec"]["text"], "face")
|
||||
self.assertEqual(tuple(out.shape), (5, 64, 48, 3))
|
||||
self.assertTrue(bool((out[:, 15, 15] == -1).all()) and float(out[0, 40, 40, 0]) == 0.5)
|
||||
st = SP.setting_from_clip(fr, 1.0, "dog")
|
||||
self.assertEqual(tuple(st.shape), (1, 64, 48, 3)); self.assertEqual(seen["spec"]["text"], "dog")
|
||||
self.assertTrue(SP.with_setting("x", 2, True).endswith("<Picture 2> setting"))
|
||||
finally:
|
||||
SP._shot_loop = old
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,900 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
import tempfile
|
||||
import types
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
_TMP = tempfile.mkdtemp(prefix="bfs_shotloop_test_")
|
||||
|
||||
# The module only needs a few folder_paths calls; stub them so the tests run without ComfyUI.
|
||||
_fp = types.ModuleType("folder_paths")
|
||||
_fp.get_temp_directory = lambda: _TMP
|
||||
_fp.get_input_directory = lambda: _TMP
|
||||
_fp.get_annotated_filepath = lambda name: str(Path(_TMP) / name)
|
||||
sys.modules.setdefault("folder_paths", _fp)
|
||||
|
||||
|
||||
class _Blocker:
|
||||
def __init__(self, message):
|
||||
self.message = message
|
||||
|
||||
|
||||
_ce = types.ModuleType("comfy_execution")
|
||||
_gu = types.ModuleType("comfy_execution.graph_utils")
|
||||
_gu.ExecutionBlocker = _Blocker
|
||||
sys.modules.setdefault("comfy_execution", _ce)
|
||||
sys.modules.setdefault("comfy_execution.graph_utils", _gu)
|
||||
|
||||
SPEC = importlib.util.spec_from_file_location("bfs_shot_loop", ROOT / "bfs_shot_loop.py")
|
||||
assert SPEC is not None and SPEC.loader is not None
|
||||
SL = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(SL)
|
||||
H3 = "H3 (17n+5)"
|
||||
|
||||
|
||||
def _shot(i, start, end, cut=False, gen=None, count=1, queue=False, run_id=""):
|
||||
length = end - start
|
||||
return {"index": i, "count": count, "start": start, "end": end, "length": length,
|
||||
"gen_length": gen or SL.snap_up(length, H3), "fps": 24.0, "cut_before": cut,
|
||||
"queue": queue, "run_id": run_id}
|
||||
|
||||
|
||||
class GridTest(unittest.TestCase):
|
||||
def test_h3_grid(self):
|
||||
self.assertEqual([SL.snap_up(x, H3) for x in (1, 5, 6, 22, 23, 107, 108)], [5, 5, 22, 22, 39, 107, 124])
|
||||
self.assertEqual([SL.snap_down(x, H3) for x in (4, 5, 21, 22, 106, 107, 110)], [5, 5, 5, 22, 90, 107, 107])
|
||||
|
||||
def test_other_grids(self):
|
||||
self.assertEqual(SL.snap_up(50, "LTX / Wan (8n+1)"), 57)
|
||||
self.assertEqual(SL.snap_down(50, "LTX / Wan (8n+1)"), 49)
|
||||
self.assertEqual(SL.snap_up(6, "Wan (4n+1)"), 9)
|
||||
self.assertEqual(SL.snap_up(37, "any"), 37)
|
||||
|
||||
def test_generation_size_is_on_the_multiple(self):
|
||||
w, h = SL.generation_size(1280, 720, 0.15, 32)
|
||||
self.assertEqual((w % 32, h % 32), (0, 0))
|
||||
self.assertGreater(w, h)
|
||||
|
||||
|
||||
class PlanTest(unittest.TestCase):
|
||||
def test_fixed_mode_splits_evenly_under_the_limit(self):
|
||||
segs = SL.plan_segments(300, [], "fixed", 107, 24)
|
||||
self.assertEqual([s["end"] - s["start"] for s in segs], [100, 100, 100])
|
||||
|
||||
def test_shots_split_at_cuts_and_long_shots_are_divided(self):
|
||||
segs = SL.plan_segments(300, [50, 80], "shots", 107, 10)
|
||||
bounds = [(s["start"], s["end"]) for s in segs]
|
||||
self.assertEqual(bounds[:2], [(0, 50), (50, 80)])
|
||||
self.assertTrue(all(e - s <= 107 for s, e in bounds))
|
||||
self.assertEqual(bounds[-1][1], 300)
|
||||
self.assertTrue(segs[1]["cut_before"])
|
||||
|
||||
def test_short_shots_merge_into_a_neighbour_that_still_fits(self):
|
||||
segs = SL.plan_segments(200, [10, 100], "shots", 107, 24)
|
||||
self.assertEqual([(s["start"], s["end"]) for s in segs], [(0, 100), (100, 200)])
|
||||
|
||||
def test_limits_and_manual_bounds(self):
|
||||
self.assertEqual(len(SL.plan_segments(1000, [], "fixed", 100, 10, max_parts=3)), 3)
|
||||
self.assertEqual(SL.plan_segments(1000, [], "fixed", 100, 10, max_total=250)[-1]["end"], 250)
|
||||
manual = SL.plan_segments(100, [], "manual", 107, 10, manual=[30, 70])
|
||||
self.assertEqual([(s["start"], s["end"]) for s in manual], [(0, 30), (30, 70), (70, 100)])
|
||||
|
||||
def test_builtin_detector_finds_spikes_and_respects_the_gap(self):
|
||||
score = [1.0] * 100
|
||||
raw = [0.01] * 100
|
||||
for i in (20, 22, 60):
|
||||
score[i], raw[i] = 50.0, 0.5
|
||||
self.assertEqual(SL.detect_cuts(score, raw, 0.5, 24.0), [20, 60])
|
||||
|
||||
|
||||
class JoinTest(unittest.TestCase):
|
||||
def _frames(self, n, value):
|
||||
return torch.full((n, 8, 8, 3), float(value))
|
||||
|
||||
def test_trims_to_true_lengths_in_order(self):
|
||||
shots = [_shot(1, 30, 50, cut=True, gen=22), _shot(0, 0, 30, gen=39)]
|
||||
imgs = [self._frames(22, 0.8), self._frames(39, 0.2)]
|
||||
video, audio, fps = SL.BFSShotJoin()._join_all(imgs, shots, [0])
|
||||
self.assertEqual(video.shape[0], 50)
|
||||
self.assertAlmostEqual(float(video[0, 0, 0, 0]), 0.2)
|
||||
self.assertAlmostEqual(float(video[-1, 0, 0, 0]), 0.8)
|
||||
self.assertEqual(fps, 24.0)
|
||||
|
||||
def test_crossfade_only_across_soft_joins(self):
|
||||
imgs = [self._frames(39, 0.0), self._frames(39, 1.0)]
|
||||
soft = SL.BFSShotJoin()._join_all(imgs, [_shot(0, 0, 30, gen=39), _shot(1, 30, 60, gen=39)], [4])[0]
|
||||
hard = SL.BFSShotJoin()._join_all(imgs, [_shot(0, 0, 30, gen=39), _shot(1, 30, 60, cut=True, gen=39)], [4])[0]
|
||||
self.assertGreater(float(soft[31, 0, 0, 0]), 0.0)
|
||||
self.assertLess(float(soft[31, 0, 0, 0]), 1.0)
|
||||
self.assertEqual(float(hard[31, 0, 0, 0]), 1.0)
|
||||
|
||||
def test_audio_is_trimmed_to_the_video(self):
|
||||
audio = {"waveform": torch.zeros(1, 2, 48000 * 10), "sample_rate": 48000}
|
||||
imgs = [self._frames(39, 0.5)]
|
||||
_, a, _ = SL.BFSShotJoin()._join_all(imgs, [_shot(0, 0, 24, gen=39)], [0], [audio])
|
||||
self.assertEqual(a["waveform"].shape[-1], 48000)
|
||||
|
||||
def test_queue_mode_blocks_until_every_shot_is_stored(self):
|
||||
join = SL.BFSShotJoin()
|
||||
shots = [_shot(0, 0, 20, count=2, queue=True, run_id="t1"), _shot(1, 20, 45, cut=True, count=2, queue=True, run_id="t1")]
|
||||
first = join.join([self._frames(22, 0.1)], [shots[0]], [0])
|
||||
self.assertIsInstance(first[0], _Blocker)
|
||||
video, _, _, _ = join.join([self._frames(39, 0.9)], [shots[1]], [0])
|
||||
self.assertEqual(video.shape[0], 45)
|
||||
self.assertAlmostEqual(float(video[0, 0, 0, 0]), 0.1, places=2)
|
||||
|
||||
|
||||
class FilterTest(unittest.TestCase):
|
||||
STATS = {"persons": 1, "person_area": 0.2, "person_frames": 6, "faces": 1, "brightness": 0.4, "motion": 0.02, "sampled": 6}
|
||||
|
||||
def test_no_filter_keeps_everything(self):
|
||||
self.assertEqual(SL.skip_reason(self.STATS, 50, dict(SL.DEFAULT_FILTERS)), "")
|
||||
self.assertFalse(SL.filters_active(dict(SL.DEFAULT_FILTERS)))
|
||||
|
||||
def test_each_filter(self):
|
||||
f = dict(SL.DEFAULT_FILTERS)
|
||||
self.assertEqual(SL.skip_reason(dict(self.STATS, persons=0), 50, dict(f, person=True)), "no person")
|
||||
self.assertIn("smaller", SL.skip_reason(dict(self.STATS, person_area=0.01), 50, dict(f, person=True, min_person_area=0.03)))
|
||||
self.assertIn("more than", SL.skip_reason(dict(self.STATS, persons=5), 50, dict(f, max_persons=2)))
|
||||
self.assertEqual(SL.skip_reason(dict(self.STATS, faces=0), 50, dict(f, face=True)), "no face")
|
||||
self.assertEqual(SL.skip_reason(dict(self.STATS, brightness=0.02), 50, dict(f, skip_dark=True)), "dark / fade")
|
||||
self.assertEqual(SL.skip_reason(dict(self.STATS, motion=0.0), 50, dict(f, skip_static=True)), "static")
|
||||
self.assertIn("shorter", SL.skip_reason(self.STATS, 5, dict(f, min_frames=10)))
|
||||
|
||||
|
||||
class TimelineJoinTest(unittest.TestCase):
|
||||
def _tl(self, fill):
|
||||
return {"path": "", "fps": 24.0, "width": 8, "height": 8, "fill": fill, "audio": None, "n": 60,
|
||||
"src": list(range(60)),
|
||||
"segs": [{"start": 0, "end": 20, "run": True, "cut_before": False, "run_index": 0},
|
||||
{"start": 20, "end": 40, "run": False, "cut_before": True, "run_index": -1},
|
||||
{"start": 40, "end": 60, "run": True, "cut_before": True, "run_index": 1}]}
|
||||
|
||||
def test_drop_removes_skipped_shots(self):
|
||||
shots = [_shot(0, 0, 20, gen=22), _shot(1, 40, 60, cut=True, gen=22)]
|
||||
imgs = [torch.zeros(22, 8, 8, 3), torch.ones(22, 8, 8, 3)]
|
||||
video = SL.BFSShotJoin()._join_all(imgs, shots, [0], None, self._tl("drop"))[0]
|
||||
self.assertEqual(video.shape[0], 40)
|
||||
self.assertEqual(float(video[25, 0, 0, 0]), 1.0)
|
||||
|
||||
def test_original_fill_reads_the_source_frames(self):
|
||||
shots = [_shot(0, 0, 20, gen=22), _shot(1, 40, 60, cut=True, gen=22)]
|
||||
imgs = [torch.zeros(22, 8, 8, 3), torch.ones(22, 8, 8, 3)]
|
||||
orig = SL._read_frames
|
||||
SL._read_frames = lambda path, idx, size: [__import__("numpy").full((8, 8, 3), 128, "uint8") for _ in idx]
|
||||
try:
|
||||
video = SL.BFSShotJoin()._join_all(imgs, shots, [0], None, self._tl("original"))[0]
|
||||
finally:
|
||||
SL._read_frames = orig
|
||||
self.assertEqual(video.shape[0], 60)
|
||||
self.assertAlmostEqual(float(video[30, 0, 0, 0]), 128 / 255, places=3)
|
||||
|
||||
|
||||
class CastTest(unittest.TestCase):
|
||||
# person 0 on frames 0-47 (big face), person 1 on 48-95, both small on 96-119
|
||||
CAST = {"step": 6, "people": [{"id": 0}, {"id": 1}], "samples":
|
||||
[{"f": f, "faces": [{"pid": 0, "area": 0.1}]} for f in range(0, 48, 6)]
|
||||
+ [{"f": f, "faces": [{"pid": 1, "area": 0.1}]} for f in range(48, 96, 6)]
|
||||
+ [{"f": f, "faces": []} for f in range(96, 120, 6)]}
|
||||
|
||||
def test_shot_people_and_main(self):
|
||||
self.assertEqual(SL.shot_people(self.CAST, 0, 60), {"people": [0, 1], "main": 0})
|
||||
self.assertEqual(SL.shot_people(self.CAST, 96, 120), {"people": [], "main": -1})
|
||||
|
||||
def test_change_points_need_a_long_enough_run(self):
|
||||
self.assertEqual(SL.person_change_points(self.CAST, 0, 120, 24), [48])
|
||||
self.assertEqual(SL.person_change_points(self.CAST, 0, 60, 24), [])
|
||||
|
||||
def test_plan_splits_assigns_and_skips_by_person(self):
|
||||
path = str(Path(_TMP) / "cast.mp4"); Path(path).write_bytes(b"x")
|
||||
a = {"fps": 24.0, "n": 120, "score": [0.0] * 120, "raw": [0.0] * 120}
|
||||
SL._CAST_CACHE[SL._cast_key(path, a)] = self.CAST
|
||||
orig = SL.find_cuts
|
||||
SL.find_cuts = lambda *args, **kw: ([], "test")
|
||||
try:
|
||||
plan = dict(SL.DEFAULT_PLAN, max_s=10, min_s=0.5, cast={"0": {"ref": "a.png"}}, cast_split=True, cast_only=True)
|
||||
segs = SL.apply_filters(plan, a, path, SL.resolve_plan(plan, a, path))
|
||||
finally:
|
||||
SL.find_cuts = orig
|
||||
self.assertEqual([(s["start"], s["end"]) for s in segs], [(0, 48), (48, 120)])
|
||||
self.assertEqual(segs[0]["ref"], "a.png")
|
||||
self.assertTrue(segs[0]["run"])
|
||||
self.assertEqual(segs[1]["skip_reason"], "no linked person")
|
||||
|
||||
|
||||
class ComparisonTest(unittest.TestCase):
|
||||
def test_join_returns_a_labelled_side_by_side(self):
|
||||
shots = [dict(_shot(0, 0, 20, gen=22), frames=torch.zeros(22, 32, 48, 3), ref=torch.ones(1, 40, 30, 3),
|
||||
ref2=None, prompt="a test prompt"),
|
||||
dict(_shot(1, 20, 40, cut=True, gen=22), frames=torch.zeros(22, 32, 48, 3), ref=None, ref2=None,
|
||||
prompt="")]
|
||||
imgs = [torch.full((22, 32, 48, 3), 0.5), torch.full((22, 32, 48, 3), 0.7)]
|
||||
video, _, _, comp = SL.BFSShotJoin().join(imgs, shots, [0], comparison=[True], label=["steps 20"])
|
||||
self.assertEqual(video.shape[0], 40)
|
||||
self.assertEqual(comp.shape[0], 40)
|
||||
self.assertEqual(comp.shape[2] % 16, 0)
|
||||
self.assertEqual(comp.shape[1] % 16, 0)
|
||||
self.assertGreaterEqual(comp.shape[2], 48 * 2 + 32)
|
||||
self.assertGreater(comp.shape[1], 32)
|
||||
off = SL.BFSShotJoin().join(imgs, shots, [0])[3]
|
||||
self.assertEqual(off.shape[0], 1)
|
||||
|
||||
|
||||
class ContinuityTest(unittest.TestCase):
|
||||
def test_pick_frame_ignores_the_overlap(self):
|
||||
frames = torch.arange(30, dtype=torch.float32).view(30, 1, 1, 1).expand(30, 2, 2, 3)
|
||||
self.assertEqual(float(SL.pick_frame(frames, 20, "first")[0, 0, 0, 0]), 0.0)
|
||||
self.assertEqual(float(SL.pick_frame(frames, 20, "middle")[0, 0, 0, 0]), 10.0)
|
||||
self.assertEqual(float(SL.pick_frame(frames, 20, "last")[0, 0, 0, 0]), 19.0)
|
||||
|
||||
def test_chain_uses_the_previous_result_only_when_asked(self):
|
||||
frames = torch.arange(30, dtype=torch.float32).view(30, 1, 1, 1).expand(30, 2, 2, 3)
|
||||
SL.remember_result({"index": 0, "count": 2}, frames)
|
||||
shot = {"index": 1, "count": 2, "chain": "reference", "chain_frame": "last", "prev_length": 20}
|
||||
self.assertEqual(float(SL.chain_image(shot)[0, 0, 0, 0]), 19.0)
|
||||
self.assertIsNone(SL.chain_image(dict(shot, chain="off")))
|
||||
self.assertIsNone(SL.chain_image(dict(shot, index=0)))
|
||||
self.assertIsNone(SL.chain_image(dict(shot, index=3, count=4))) # not the shot right after
|
||||
stored = torch.ones(1, 2, 2, 3)
|
||||
self.assertIs(SL.chain_image(dict(shot, chain_image=stored)), stored)
|
||||
|
||||
def test_plan_carries_the_per_shot_setting(self):
|
||||
a = {"fps": 24.0, "n": 100, "score": [0.0] * 100, "raw": [0.0] * 100}
|
||||
plan = dict(SL.DEFAULT_PLAN, mode="fixed", max_s=2, bounds=[50],
|
||||
segs=[{}, {"chain": "first frame", "chain_frame": "middle"}])
|
||||
segs = SL.resolve_plan(plan, a)
|
||||
self.assertEqual((segs[0]["chain"], segs[1]["chain"], segs[1]["chain_frame"]), ("history", "first frame", "middle")) # history is the default
|
||||
|
||||
|
||||
class MaskCropTest(unittest.TestCase):
|
||||
def test_spec_takes_target_from_shot_and_settings_from_global(self):
|
||||
spec = SL.mask_spec({"text": "person", "padding": 0.9}, {"padding": 0.3, "text": "ignored"})
|
||||
self.assertEqual((spec["text"], spec["padding"], spec["fill_holes"]), ("person", 0.3, True))
|
||||
|
||||
def test_shape_fills_holes_and_holds_in_time(self):
|
||||
m = torch.zeros(5, 9, 9, dtype=torch.uint8)
|
||||
m[2, 2:7, 2:7] = 1
|
||||
m[2, 4, 4] = 0 # a hole
|
||||
out = SL.shape_mask(m, dict(SL.DEFAULT_MASK, temporal_expand=1))
|
||||
self.assertEqual(int(out[2, 4, 4]), 1)
|
||||
self.assertEqual(int(out[1, 3, 3]), 1) # held one frame back
|
||||
self.assertEqual(int(out[0, 3, 3]), 0)
|
||||
blocks = SL.shape_mask(m, dict(SL.DEFAULT_MASK, temporal_expand=0, blockify=4))
|
||||
self.assertEqual(tuple(blocks.shape), (5, 9, 9))
|
||||
|
||||
def test_box_is_the_union_plus_padding(self):
|
||||
m = torch.zeros(2, 10, 10, dtype=torch.uint8)
|
||||
m[0, 2:4, 2:4] = 1
|
||||
m[1, 6:8, 6:8] = 1
|
||||
self.assertEqual(SL.crop_box(m, 0.0), [0.2, 0.2, 0.8, 0.8])
|
||||
self.assertIsNone(SL.crop_box(torch.zeros(2, 10, 10), 0.1))
|
||||
|
||||
def test_uncrop_pastes_only_inside_the_mask(self):
|
||||
full = torch.zeros(3, 20, 20, 3)
|
||||
mask = torch.zeros(3, 20, 20)
|
||||
mask[:, 5:15, 5:15] = 1
|
||||
shot = {"full_frames": full, "crop": {"box": [0.25, 0.25, 0.75, 0.75], "mask": mask, "paste": "mask",
|
||||
"expand": 0, "feather": 0}}
|
||||
out = SL.uncrop(torch.ones(3, 8, 8, 3), shot)
|
||||
self.assertEqual(tuple(out.shape), (3, 20, 20, 3))
|
||||
self.assertEqual(float(out[0, 10, 10, 0]), 1.0)
|
||||
self.assertEqual(float(out[0, 1, 1, 0]), 0.0)
|
||||
|
||||
|
||||
class DuetPanelCropTest(unittest.TestCase):
|
||||
def test_join_side_cuts_the_panel_off(self):
|
||||
shot = {"width": 64, "height": 32, "panel": {"position": "left", "h": 2, "w": 4, "strip_h": 0, "strip_w": 4}}
|
||||
canvas = torch.zeros(3, 32, 128, 3)
|
||||
canvas[:, :, 64:] = 1.0
|
||||
out = SL.crop_panel(canvas, shot)
|
||||
self.assertEqual(tuple(out.shape), (3, 32, 64, 3))
|
||||
self.assertEqual(float(out.min()), 1.0)
|
||||
self.assertIs(SL.crop_panel(out, shot), out) # already the video size
|
||||
self.assertIs(SL.crop_panel(canvas, {"width": 64}), canvas) # no panel
|
||||
|
||||
|
||||
class SilentAudioTest(unittest.TestCase):
|
||||
def test_join_without_audio_returns_a_silent_track(self):
|
||||
imgs = [torch.full((39, 8, 8, 3), 0.5)]
|
||||
video, a, fps = SL.BFSShotJoin()._join_all(imgs, [_shot(0, 0, 24, gen=39)], [0])
|
||||
self.assertEqual(tuple(a["waveform"].shape[:2]), (1, 2))
|
||||
self.assertEqual(a["waveform"].shape[-1], 44100) # 24 frames at 24 fps = 1 s
|
||||
self.assertEqual(float(a["waveform"].abs().max()), 0.0)
|
||||
self.assertFalse(SL._usable({"waveform": torch.zeros(1, 2, 0), "sample_rate": 44100}))
|
||||
|
||||
|
||||
class RepackTest(unittest.TestCase):
|
||||
def test_replaces_only_connected_pieces_and_fits_the_guide(self):
|
||||
shot = dict(_shot(0, 0, 20, gen=22), width=8, height=8, frames=torch.zeros(22, 8, 8, 3),
|
||||
ref=torch.zeros(1, 4, 4, 3), ref2=None, prompt="old", audio=None)
|
||||
ref = torch.ones(1, 6, 6, 3)
|
||||
out = SL.BFSShotRepack().repack(shot, ref_image=ref)[0]
|
||||
self.assertTrue(torch.equal(out["ref"], ref))
|
||||
self.assertEqual(out["prompt"], "old")
|
||||
self.assertIs(out["frames"], shot["frames"])
|
||||
out = SL.BFSShotRepack().repack(shot, guide_frames=torch.ones(30, 16, 16, 3), prompt="new")[0]
|
||||
self.assertEqual(tuple(out["frames"].shape), (22, 8, 8, 3))
|
||||
self.assertEqual(out["prompt"], "new")
|
||||
self.assertEqual(out["length"], 20)
|
||||
|
||||
|
||||
class ConditioningWriterTest(unittest.TestCase):
|
||||
"""The optional prompt writer of BFS Shot H3 Conditioning (template path, no models)."""
|
||||
|
||||
def setUp(self):
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
import bfs_h3_side_panel as SP
|
||||
self.SP, self.seen = SP, {}
|
||||
seen = self.seen
|
||||
|
||||
class R2V:
|
||||
@staticmethod
|
||||
def execute(**kw):
|
||||
seen["prompt"] = kw["prompt"]
|
||||
return types.SimpleNamespace(args=("pos", "lat"))
|
||||
|
||||
mod = types.ModuleType("comfy_extras.nodes_minimax_h3")
|
||||
mod.MiniMaxH3ReferenceToVideo, mod.MiniMaxH3AddGuide = R2V, R2V
|
||||
self._old = {k: sys.modules.get(k) for k in ("comfy_extras", "comfy_extras.nodes_minimax_h3")}
|
||||
sys.modules["comfy_extras"] = types.ModuleType("comfy_extras")
|
||||
sys.modules["comfy_extras.nodes_minimax_h3"] = mod
|
||||
self._apply = SP.BFSH3SidePanel.apply
|
||||
SP.BFSH3SidePanel.apply = lambda self, pos, lat, *a, **k: (pos, lat, {"position": "left"}, None, None)
|
||||
self._patch = SP.patch_model_rope
|
||||
SP.patch_model_rope = lambda m, info, gap: "patched"
|
||||
|
||||
def tearDown(self):
|
||||
self.SP.BFSH3SidePanel.apply, self.SP.patch_model_rope = self._apply, self._patch
|
||||
for k, v in self._old.items():
|
||||
if v is None:
|
||||
sys.modules.pop(k, None)
|
||||
else:
|
||||
sys.modules[k] = v
|
||||
|
||||
def _shot(self):
|
||||
return dict(prompt="my own {layout} prompt", width=448, height=800, gen_length=22,
|
||||
frames=torch.zeros(22, 8, 8, 3), ref=torch.zeros(1, 4, 4, 3), ref2=None, audio=None)
|
||||
|
||||
def _run(self, **kw):
|
||||
c = SL.BFSShotH3Conditioning()
|
||||
return c.condition(self._shot(), "clip", "vae", "none", True, "none", "match", **kw)
|
||||
|
||||
def test_planner_prompt_is_default_and_layout_filled(self):
|
||||
out = self._run()
|
||||
self.assertEqual(out[3], "my own prompt")
|
||||
out = self._run(duet="canvas")
|
||||
self.assertIn("the left half shows the original performance", out[3])
|
||||
self.assertEqual(self.seen["prompt"], out[3])
|
||||
|
||||
def test_task_template_follows_the_duet_mode(self):
|
||||
out = self._run(duet="canvas", task="character swap")
|
||||
self.assertIn("split screen", out[3])
|
||||
self.assertIn("<Picture 1>", out[3])
|
||||
out = self._run(duet="shifted RoPE", model="m", task="character swap")
|
||||
self.assertNotIn("split screen", out[3])
|
||||
self.assertEqual(out[2], "patched")
|
||||
|
||||
def test_setting_picture_is_the_last_reference_with_static(self):
|
||||
seen = {}
|
||||
orig = sys.modules["comfy_extras.nodes_minimax_h3"].MiniMaxH3ReferenceToVideo.execute
|
||||
|
||||
def capture(**kw):
|
||||
seen.update(kw)
|
||||
return orig(**kw)
|
||||
sys.modules["comfy_extras.nodes_minimax_h3"].MiniMaxH3ReferenceToVideo.execute = staticmethod(capture)
|
||||
shot = self._shot()
|
||||
shot["frames"] = torch.full((22, 8, 8, 3), 0.5)
|
||||
mask = torch.zeros(22, 8, 8)
|
||||
mask[:, 2:4, 2:4] = 1
|
||||
out = SL.BFSShotH3Conditioning().condition(shot, "clip", "vae", "none", True, "none", "match", duet="canvas",
|
||||
task="character swap", setting_ref="on (generation size)",
|
||||
setting_mask=mask)
|
||||
refs = list(seen["ref_images"].values())
|
||||
self.assertEqual(len(refs), 2)
|
||||
pic = refs[-1][0]
|
||||
self.assertTrue(((pic[3, 3] == 0) | (pic[3, 3] == 1)).all()) # static inside the (grown) mask
|
||||
self.assertTrue(torch.allclose(pic[7, 7], torch.tensor(0.5))) # the place outside it
|
||||
self.assertIn("<Picture 2> shows the setting", out[3])
|
||||
self.assertEqual(SL.add_setting("a {setting} b", 3, True), "a <Picture 3> b")
|
||||
|
||||
def test_task_needs_duet(self):
|
||||
with self.assertRaises(ValueError):
|
||||
self._run(task="style", instruction="anime")
|
||||
|
||||
|
||||
class TargetTest(unittest.TestCase):
|
||||
def test_target_placeholder(self):
|
||||
self.assertEqual(SL.fill_target("Replace {target} with <Subject 1>", "the man in a red coat"),
|
||||
"Replace the man in a red coat with <Subject 1>")
|
||||
self.assertEqual(SL.fill_target("Replace {target} with <Subject 1>", ""), "Replace the person with <Subject 1>")
|
||||
self.assertEqual(SL.fill_target("no placeholder", "x"), "no placeholder")
|
||||
|
||||
|
||||
class MaskOverlayTest(unittest.TestCase):
|
||||
def test_overlay_marks_mask_and_box(self):
|
||||
o = torch.zeros(3, 32, 32, 3)
|
||||
m = torch.zeros(3, 32, 32); m[:, 8:16, 8:16] = 1
|
||||
out = SL._mask_overlay(o, {"mask": m, "box": [0.1, 0.1, 0.9, 0.9]}, 3)
|
||||
self.assertGreater(float(out[0, 12, 12, 0]), 0.3) # red inside the mask
|
||||
self.assertEqual(float(out[0, 20, 20].sum()), 0.0) # untouched outside mask and box
|
||||
self.assertGreater(float(out[0, 3, 16, 1]), 0.5) # yellow box edge
|
||||
|
||||
|
||||
class StitchFinishTest(unittest.TestCase):
|
||||
def _shot(self):
|
||||
full = torch.full((4, 64, 64, 3), 0.5)
|
||||
mask = torch.zeros(4, 64, 64); mask[:, 20:44, 24:40] = 1
|
||||
return {"full_frames": full, "crop": {"box": [0.125, 0.125, 0.875, 0.875], "mask": mask, "paste": "mask",
|
||||
"expand": 0, "feather": 2}}
|
||||
|
||||
def test_ring_colour_match_removes_drift_and_keeps_subject_contrast(self):
|
||||
shot = self._shot()
|
||||
gen = torch.full((4, 48, 48, 3), 0.62) # the model brightened the whole crop (+0.12)
|
||||
gen[:, 12:36, 16:32] = 0.2 # a darker new subject
|
||||
plain = SL.uncrop(gen, shot)
|
||||
fixed = SL.uncrop(gen, shot, match_colors=1.0)
|
||||
# around the subject the drift is gone, inside it the new subject stays darker than the scene
|
||||
self.assertGreater(abs(float(plain[0, 21, 23].mean()) - 0.5), abs(float(fixed[0, 21, 23].mean()) - 0.5))
|
||||
self.assertLess(float(fixed[0, 30, 30].mean()), 0.45)
|
||||
|
||||
def test_edge_hardness(self):
|
||||
a = torch.tensor([[[0.1, 0.5, 0.9]]])
|
||||
h = SL._alpha_hardness(a, 1.0)
|
||||
self.assertEqual([round(float(x), 2) for x in h[0, 0]], [0.0, 0.5, 1.0])
|
||||
self.assertTrue(torch.equal(SL._alpha_hardness(a, 0.0), a))
|
||||
|
||||
|
||||
class InpaintInCropTest(unittest.TestCase):
|
||||
def test_latent_starts_from_the_shot_and_only_the_mask_is_generated(self):
|
||||
sys.path.insert(0, str(ROOT))
|
||||
nested = sys.modules.get("comfy.nested_tensor")
|
||||
if nested is None:
|
||||
class _N:
|
||||
is_nested = True
|
||||
|
||||
def __init__(self, t):
|
||||
self.tensors = list(t)
|
||||
comfy = sys.modules.setdefault("comfy", types.ModuleType("comfy"))
|
||||
nested = types.ModuleType("comfy.nested_tensor"); nested.NestedTensor = _N
|
||||
comfy.nested_tensor = nested
|
||||
sys.modules["comfy.nested_tensor"] = nested
|
||||
sys.modules.setdefault("comfy.utils", types.ModuleType("comfy.utils"))
|
||||
|
||||
class VAE:
|
||||
def encode(self, px):
|
||||
t = ((px.shape[0] - 5) // 17) * 5 + 2
|
||||
x = px.mean(-1)[None, None]
|
||||
return torch.nn.functional.interpolate(x, size=(t, px.shape[1] // 16, px.shape[2] // 16)).repeat(1, 24, 1, 1, 1)
|
||||
|
||||
lat = {"samples": nested.NestedTensor((torch.zeros(1, 24, 7, 8, 8), torch.zeros(1, 32, 2, 37)))}
|
||||
frames = torch.full((22, 128, 128, 3), 0.4)
|
||||
mask = torch.zeros(22, 128, 128); mask[:, 48:80, 48:80] = 1
|
||||
pos, out = SL.inpaint_latent(lat, [], VAE(), frames, mask)
|
||||
video = out["samples"].tensors[0]
|
||||
self.assertAlmostEqual(float(video.mean()), 0.4, places=4) # starts from the shot's own frames
|
||||
vm, am = out["noise_mask"].tensors
|
||||
self.assertEqual(tuple(vm.shape), (1, 1, 7, 8, 8))
|
||||
self.assertEqual(float(vm[0, 0, 3, 4, 4]), 1.0) # the person: generated
|
||||
self.assertEqual(float(vm[0, 0, 3, 0, 0]), 0.0) # the rest of the crop: kept
|
||||
self.assertEqual(float(vm[0, 0, 3, 2, 4]), 1.0) # grown by one latent cell
|
||||
self.assertTrue(bool((am == 1).all())) # audio fully generated
|
||||
_, out2 = SL.inpaint_latent(lat, [], VAE(), frames, mask, strength=0.8)
|
||||
vm2 = out2["noise_mask"].tensors[0]
|
||||
self.assertAlmostEqual(float(vm2[0, 0, 3, 4, 4]), 0.8, places=5) # partly regenerated inside the mask
|
||||
self.assertEqual(float(vm2[0, 0, 3, 0, 0]), 0.0) # still kept outside
|
||||
|
||||
|
||||
def test_generation_mask_with_and_without_crop(self):
|
||||
m = torch.zeros(5, 64, 64); m[:, 20:40, 20:40] = 1
|
||||
cropped = SL.generation_mask({"crop": {"crop_mask": m, "expand": 4}, "frames": torch.zeros(5, 64, 64, 3)})
|
||||
self.assertEqual(float(cropped[0, 17, 30]), 1.0) # grown by expand
|
||||
self.assertEqual(float(cropped[0, 10, 30]), 0.0)
|
||||
with self.assertRaises(ValueError):
|
||||
SL.generation_mask({"crop": None, "mask_src": None, "frames": torch.zeros(5, 64, 64, 3)})
|
||||
orig = SL.shot_mask
|
||||
SL.shot_mask = lambda *a: {"masks": torch.nn.functional.interpolate(m[:, None], size=(32, 32))[:, 0].to(torch.uint8)}
|
||||
try:
|
||||
full = SL.generation_mask({"crop": None, "frames": torch.zeros(5, 64, 64, 3),
|
||||
"mask_src": {"path": "", "analysis": {}, "start": 0, "length": 5,
|
||||
"spec": {"expand": 0}}})
|
||||
finally:
|
||||
SL.shot_mask = orig
|
||||
self.assertEqual(tuple(full.shape), (5, 64, 64))
|
||||
self.assertTrue(torch.equal(full, m))
|
||||
|
||||
|
||||
def test_comparison_overlay_for_mask_only_shots(self):
|
||||
m = torch.zeros(5, 32, 32); m[:, 8:24, 8:24] = 1
|
||||
ms = {"path": __file__, "analysis": {"fps": 24.0}, "start": 0, "length": 5, "spec": dict(SL.DEFAULT_MASK, expand=0)}
|
||||
shot = {"crop": None, "frames": torch.zeros(5, 64, 64, 3), "mask_src": ms, "inpaint": True}
|
||||
orig = SL.shot_mask
|
||||
SL.shot_mask = lambda *a: {"masks": m.to(torch.uint8)}
|
||||
try:
|
||||
ov = SL.overlay_of(shot)
|
||||
self.assertEqual(tuple(ov["mask"].shape), (5, 64, 64))
|
||||
self.assertIsNone(SL.overlay_of(dict(shot, inpaint=False))) # not used and not segmented yet: no SAM 3
|
||||
self.assertIsNone(SL.overlay_of({"crop": None, "frames": shot["frames"]}))
|
||||
finally:
|
||||
SL.shot_mask = orig
|
||||
o = SL._mask_overlay(torch.full((5, 32, 32, 3), 0.5), ov, 5)
|
||||
self.assertGreater(float(o[0, 16, 16, 0]), float(o[0, 16, 16, 1])) # red inside the mask
|
||||
self.assertAlmostEqual(float(o[0, 2, 2, 0]), 0.5) # untouched outside
|
||||
|
||||
|
||||
class ExternalMaskTest(unittest.TestCase):
|
||||
def _video(self, name, n=48, fps=24.0, w=64, h=36, box=(10, 5, 30, 25)):
|
||||
import cv2
|
||||
import numpy as np
|
||||
path = str(Path(_TMP) / name)
|
||||
vw = cv2.VideoWriter(path, cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))
|
||||
for i in range(n):
|
||||
f = np.zeros((h, w, 3), np.uint8)
|
||||
x0, y0, x1, y1 = box
|
||||
f[y0:y1, x0 + i % 4:x1 + i % 4] = 255
|
||||
vw.write(f)
|
||||
vw.release()
|
||||
return path
|
||||
|
||||
def setUp(self):
|
||||
self.an = {"fps": 24.0, "n": 48, "n_src": 48, "fps_src": 24.0, "width": 64, "height": 36}
|
||||
|
||||
def test_precedence(self):
|
||||
p = {"mask_video": "global.mp4"}
|
||||
self.assertEqual(SL.plan_spec(p, {"video": "shot.mp4", "text": "man"})["ext"]["scope"], "shot")
|
||||
self.assertNotIn("ext", SL.plan_spec(p, {"text": "man"})) # the shot's own SAM 3 text wins
|
||||
self.assertEqual(SL.plan_spec(p, {})["ext"], {"file": "global.mp4", "scope": "video"})
|
||||
SL._EXT_MASK["v.mp4"] = torch.ones(48, 36, 64)
|
||||
try:
|
||||
self.assertEqual(SL.plan_spec(p, {}, "v.mp4")["ext"], {"tensor": "v.mp4"}) # the node input beats the panel file
|
||||
finally:
|
||||
SL._EXT_MASK.pop("v.mp4")
|
||||
self.assertFalse(SL.spec_has_mask(SL.plan_spec({}, {})))
|
||||
|
||||
def test_file_and_tensor_masks_replace_sam(self):
|
||||
path = self._video("roto_whole.mp4")
|
||||
self._video("roto_shot.mp4", n=12)
|
||||
orig = SL.segment_frames
|
||||
SL.segment_frames = lambda *a, **k: self.fail("SAM 3 must not run with an external mask")
|
||||
try:
|
||||
r = SL.shot_mask(path, self.an, 24, 12, SL.plan_spec({"mask_video": "roto_whole.mp4"}, {}))
|
||||
self.assertEqual(r["masks"].shape[0], 12)
|
||||
self.assertTrue(r["box"] is not None)
|
||||
cov = float(r["masks"].float().mean())
|
||||
self.assertTrue(0.1 < cov < 0.4, cov)
|
||||
r2 = SL.shot_mask(path, self.an, 24, 12, SL.plan_spec({}, {"video": "roto_shot.mp4"}))
|
||||
self.assertEqual(r2["masks"].shape[0], 12)
|
||||
t = torch.zeros(48, 36, 64); t[:, 0:18, :] = 1 # top half
|
||||
SL._EXT_MASK[path] = t
|
||||
r3 = SL.shot_mask(path, self.an, 0, 10, SL.plan_spec({}, {}, path))
|
||||
m = r3["masks"].float()
|
||||
self.assertGreater(float(m[:, :8].mean()), 0.9)
|
||||
self.assertLess(float(m[:, -8:].mean()), 0.1)
|
||||
finally:
|
||||
SL.segment_frames = orig
|
||||
SL._EXT_MASK.pop(path, None)
|
||||
|
||||
|
||||
class FramePasteTest(unittest.TestCase):
|
||||
def test_paste_back_keeps_the_background_and_takes_the_new_outline(self):
|
||||
base = torch.full((5, 64, 64, 3), 0.2) # the original frames
|
||||
result = torch.full((5, 64, 64, 3), 0.9) # generated: everything changed
|
||||
old = torch.zeros(5, 64, 64); old[:, 20:44, 20:36] = 1 # the original person
|
||||
new = torch.zeros(5, 64, 64); new[:, 16:48, 18:44] = 1 # the new one is bigger
|
||||
ms = {"path": "", "analysis": {}, "start": 0, "length": 5, "spec": dict(SL.DEFAULT_MASK, expand=0, feather=0)}
|
||||
shot = {"index": 0, "frames": base, "mask_src": ms, "paste": True}
|
||||
o1, o2 = SL.shot_mask, SL.segment_frames
|
||||
SL.shot_mask = lambda *a: {"masks": old.to(torch.uint8)}
|
||||
SL.segment_frames = lambda imgs, sp: torch.nn.functional.interpolate(new[:, None], size=imgs.shape[1:3])[:, 0]
|
||||
try:
|
||||
out = SL.paste_back(result, shot, True)
|
||||
only_old = SL.paste_back(result, shot, False)
|
||||
finally:
|
||||
SL.shot_mask, SL.segment_frames = o1, o2
|
||||
self.assertAlmostEqual(float(out[0, 2, 2, 0]), 0.2, places=4) # background: the original
|
||||
self.assertAlmostEqual(float(out[0, 30, 28, 0]), 0.9, places=4) # the person: generated
|
||||
self.assertAlmostEqual(float(out[0, 30, 41, 0]), 0.9, places=4) # the bigger new outline is pasted too
|
||||
self.assertAlmostEqual(float(only_old[0, 30, 41, 0]), 0.2, places=4)
|
||||
|
||||
|
||||
class MaskGuideTest(unittest.TestCase):
|
||||
def test_masked_only_greys_everything_outside_the_mask(self):
|
||||
fr = torch.full((4, 32, 32, 3), 0.9)
|
||||
m = torch.zeros(4, 16, 16); m[:, 4:12, 4:12] = 1
|
||||
g = SL.masked_only(fr, m)
|
||||
self.assertEqual(tuple(g.shape), (4, 32, 32, 3))
|
||||
self.assertAlmostEqual(float(g[0, 16, 16, 0]), 0.9) # the masked region
|
||||
self.assertAlmostEqual(float(g[0, 2, 2, 0]), 0.5) # grey elsewhere
|
||||
self.assertEqual(SL.MASK_GUIDES[0], "off")
|
||||
fr2 = torch.rand(3, 64, 64, 3)
|
||||
m2 = torch.zeros(3, 64, 64); m2[:, 10:50, 10:50] = 1
|
||||
old_det = dict(SL._DET); SL._DET["pose"] = None # no YOLO pose model: a clear error
|
||||
try:
|
||||
with self.assertRaisesRegex(ValueError, "YOLO pose model"):
|
||||
SL.masked_only(fr2, m2, look="pose (people)")
|
||||
finally:
|
||||
SL._DET.clear(); SL._DET.update(old_det)
|
||||
for look in [x for x in SL.MASK_GUIDE_LOOKS if not x.startswith("pose")]:
|
||||
g2 = SL.masked_only(fr2, m2, look=look)
|
||||
self.assertEqual(tuple(g2.shape), (3, 64, 64, 3), look)
|
||||
self.assertAlmostEqual(float(g2[0, 2, 2, 0]), 0.5, msg=look) # grey outside, every look
|
||||
if look != "colour": # no colour inside
|
||||
self.assertLess(float((g2[..., 0] - g2[..., 1]).abs().max()), 1e-5, look)
|
||||
t = SL.add_mask_video("x subject_definitions: <Subject 1> is ...", 2)
|
||||
self.assertIn("<Video 2> shows only the region", t)
|
||||
self.assertEqual(SL.add_mask_video("follow {mask_video}", 1), "follow <Video 1>")
|
||||
|
||||
|
||||
class ConditionFlowTest(unittest.TestCase):
|
||||
"""BFS Shot H3 Conditioning with the native H3 nodes replaced by recorders."""
|
||||
|
||||
def setUp(self):
|
||||
sys.path.insert(0, str(ROOT))
|
||||
calls = self.calls = {"r2v": [], "guides": []}
|
||||
|
||||
class Out:
|
||||
def __init__(self, *a):
|
||||
self.args = a
|
||||
|
||||
class R2V:
|
||||
@staticmethod
|
||||
def execute(**kw):
|
||||
calls["r2v"].append(kw)
|
||||
N = sys.modules["comfy.nested_tensor"].NestedTensor
|
||||
return Out([[torch.zeros(1), {}]], {"samples": N((torch.zeros(1, 24, 2, 4, 4), torch.zeros(1, 32, 2, 37)))})
|
||||
|
||||
class Guide:
|
||||
@staticmethod
|
||||
def execute(positive, latent, frame_idx, vae, image=None, **kw):
|
||||
calls["guides"].append(image)
|
||||
return Out(positive)
|
||||
|
||||
mod = types.ModuleType("comfy_extras.nodes_minimax_h3")
|
||||
mod.MiniMaxH3ReferenceToVideo, mod.MiniMaxH3AddGuide = R2V, Guide
|
||||
sys.modules.setdefault("comfy_extras", types.ModuleType("comfy_extras"))
|
||||
self._old = sys.modules.get("comfy_extras.nodes_minimax_h3")
|
||||
sys.modules["comfy_extras.nodes_minimax_h3"] = mod
|
||||
if "comfy.nested_tensor" not in sys.modules:
|
||||
class _N:
|
||||
is_nested = True
|
||||
|
||||
def __init__(self, t):
|
||||
self.tensors = list(t)
|
||||
comfy = sys.modules.setdefault("comfy", types.ModuleType("comfy"))
|
||||
nt = types.ModuleType("comfy.nested_tensor"); nt.NestedTensor = _N
|
||||
comfy.nested_tensor = nt; sys.modules["comfy.nested_tensor"] = nt
|
||||
|
||||
def tearDown(self):
|
||||
if self._old is not None:
|
||||
sys.modules["comfy_extras.nodes_minimax_h3"] = self._old
|
||||
else:
|
||||
sys.modules.pop("comfy_extras.nodes_minimax_h3", None)
|
||||
|
||||
def _shot(self, **kw):
|
||||
m = torch.zeros(22, 64, 64); m[:, 16:48, 16:40] = 1
|
||||
shot = {"index": 0, "count": 1, "frames": torch.full((22, 64, 64, 3), 0.7), "ref": torch.zeros(1, 32, 32, 3),
|
||||
"ref2": None, "prompt": "subject_definitions: <Subject 1> is ...", "width": 64, "height": 64,
|
||||
"gen_length": 22, "audio": None, "crop": {"crop_mask": m, "expand": 0} if kw.get("crop") else None,
|
||||
"inpaint": kw.get("inpaint", True)}
|
||||
if not kw.get("crop"):
|
||||
shot["mask_src"] = {"path": __file__, "analysis": {"fps": 24.0}, "start": 0, "length": 22, "spec": dict(SL.DEFAULT_MASK, expand=0)}
|
||||
return shot, m
|
||||
|
||||
def run_cond(self, shot, m, **kw):
|
||||
class VAE:
|
||||
def encode(self, px):
|
||||
return torch.zeros(1, 24, 2, px.shape[1] // 16, px.shape[2] // 16)
|
||||
orig = SL.shot_mask
|
||||
SL.shot_mask = lambda *a: {"masks": m.to(torch.uint8)}
|
||||
try:
|
||||
return SL.BFSShotH3Conditioning().condition(shot, None, VAE(), SL.BFSShotH3Conditioning.GUIDE_MODES[0], False,
|
||||
"none", "match", **kw)
|
||||
finally:
|
||||
SL.shot_mask = orig
|
||||
|
||||
def test_reference_video_of_the_masked_region(self):
|
||||
shot, m = self._shot()
|
||||
self.run_cond(shot, m, inpaint=SL.INPAINT_MODES[0], mask_guide=SL.MASK_GUIDES[3], mask_ref_size="full",
|
||||
mask_guide_look="colour")
|
||||
kw = self.calls["r2v"][0]
|
||||
v = kw["ref_videos"]["ref_video_1"]
|
||||
self.assertAlmostEqual(float(v[0, 2, 2, 0]), 0.5) # grey outside the mask
|
||||
self.assertAlmostEqual(float(v[0, 30, 30, 0]), 0.7) # the subject
|
||||
self.assertIn("<Video 1> shows only the region", kw["prompt"])
|
||||
self.assertEqual(len(self.calls["guides"]), 1) # the normal aligned guide is still there
|
||||
self.calls["r2v"].clear()
|
||||
self.run_cond(shot, m, inpaint=SL.INPAINT_MODES[0], mask_guide=SL.MASK_GUIDES[3], mask_ref_size="1/2")
|
||||
self.assertEqual(tuple(self.calls["r2v"][0]["ref_videos"]["ref_video_1"].shape[1:3]), (32, 32))
|
||||
|
||||
def test_extra_and_instead_guides(self):
|
||||
shot, m = self._shot()
|
||||
self.run_cond(shot, m, inpaint=SL.INPAINT_MODES[0], mask_guide=SL.MASK_GUIDES[1])
|
||||
self.assertEqual(len(self.calls["guides"]), 2)
|
||||
self.calls["guides"].clear()
|
||||
self.run_cond(shot, m, inpaint=SL.INPAINT_MODES[0], mask_guide=SL.MASK_GUIDES[2])
|
||||
self.assertEqual(len(self.calls["guides"]), 1)
|
||||
self.assertAlmostEqual(float(self.calls["guides"][0][0, 2, 2, 0]), 0.5) # it is the masked one
|
||||
|
||||
def test_ignored_outside_mask_only(self):
|
||||
shot, m = self._shot(inpaint=False)
|
||||
self.run_cond(shot, m, inpaint=SL.INPAINT_MODES[0], mask_guide=SL.MASK_GUIDES[3])
|
||||
self.assertNotIn("ref_videos", self.calls["r2v"][0])
|
||||
shot, m = self._shot(crop=True)
|
||||
self.calls["r2v"].clear()
|
||||
self.run_cond(shot, m, inpaint=SL.INPAINT_MODES[0], mask_guide=SL.MASK_GUIDES[3])
|
||||
self.assertNotIn("ref_videos", self.calls["r2v"][0])
|
||||
|
||||
|
||||
class DescribeShortTest(unittest.TestCase):
|
||||
def test_short_details_fit_inside_the_prompt(self):
|
||||
self.assertEqual(SL.tidy_details('"The green goblin in a torn leather vest."\nextra', {"describe_preset": "short"}),
|
||||
"green goblin in a torn leather vest")
|
||||
self.assertEqual(SL.tidy_details("A young woman.", {"describe_preset": "full body"}), "A young woman.")
|
||||
self.assertIn("one English noun phrase", SL.describe_instruction({}))
|
||||
|
||||
def test_every_preset_removes_vlm_answer_labels(self):
|
||||
for preset in SL.DESCRIBE_PRESETS:
|
||||
with self.subTest(preset=preset):
|
||||
self.assertEqual(SL.tidy_details("**Answer:** elderly woman with large glasses", {"describe_preset": preset}),
|
||||
"elderly woman with large glasses")
|
||||
|
||||
def test_short_mode_honours_its_twenty_word_contract(self):
|
||||
text = "Answer: The " + " ".join(f"word{i}" for i in range(30))
|
||||
out = SL.tidy_details(text, {"describe_preset": "short"})
|
||||
self.assertEqual(len(out.split()), 20)
|
||||
self.assertFalse(out.lower().startswith(("answer", "the ")))
|
||||
|
||||
def test_long_modes_are_bounded_but_not_collapsed_to_short(self):
|
||||
text = " ".join(f"word{i}" for i in range(170))
|
||||
full = SL.tidy_details(text, {"describe_preset": "full body"})
|
||||
face = SL.tidy_details(text, {"describe_preset": "head / face"})
|
||||
self.assertEqual(len(full.split()), 150)
|
||||
self.assertEqual(len(face.split()), 110)
|
||||
self.assertTrue(full.endswith("."))
|
||||
|
||||
|
||||
class FitPromptTest(unittest.TestCase):
|
||||
P = ("body_swap: subject_definitions: <Subject 1> is the {details} shown in <Picture 1>, whose identity define the "
|
||||
"character. The guide video provides the scene; only {target} is replaced: their face become <Subject 1>. "
|
||||
"summary: Replace {target} in the guide video with <Subject 1>.")
|
||||
|
||||
def test_slot_is_the_sentence_with_a_blank(self):
|
||||
d = SL.prompt_slot(self.P, "{details}")
|
||||
self.assertIn("<Subject 1> is the ___ shown in <Picture 1>", d)
|
||||
self.assertNotIn("{target}", d)
|
||||
t = SL.prompt_slot(self.P, "{target}")
|
||||
self.assertIn("only ___ is replaced", t)
|
||||
self.assertEqual(SL.prompt_slot("no marker here", "{details}"), "")
|
||||
|
||||
def test_answer_matches_the_blank(self):
|
||||
d = SL.prompt_slot(self.P, "{details}")
|
||||
self.assertEqual(SL.tidy_slot("A golden retriever with a thick coat.", d), "golden retriever with a thick coat")
|
||||
t = SL.prompt_slot(self.P, "{target}")
|
||||
self.assertEqual(SL.tidy_slot("the young woman in a black corset.", t), "the young woman in a black corset")
|
||||
self.assertIn("goes into the blank", SL.describe_instruction({"describe_preset": "short"}, d))
|
||||
self.assertNotIn("goes into the blank", SL.describe_instruction({"describe_preset": "short", "fit_prompt": False}, d))
|
||||
self.assertNotIn("goes into the blank", SL.describe_instruction({"describe_preset": "full body"}, d))
|
||||
# the VLM echoing the sentence (the bug the user hit)
|
||||
echo = ("humanoid robot is the ___ shown in <Picture 1>, whose identity define the character. The robot has a sleek "
|
||||
"body with a metallic silver torso.")
|
||||
self.assertEqual(SL.tidy_slot(echo, d), "humanoid robot")
|
||||
self.assertEqual(SL.tidy_slot("sleek silver humanoid robot with black joints shown in <Picture 1>", d),
|
||||
"sleek silver humanoid robot with black joints")
|
||||
self.assertEqual(SL.tidy_slot("<Subject 1> is the green goblin in a vest", d), "green goblin in a vest")
|
||||
|
||||
def test_target_label_is_removed_even_without_a_prompt_slot(self):
|
||||
old = SL.vlm_generate
|
||||
SL.vlm_generate = lambda *args, **kwargs: "Answer: elderly woman with large tortoiseshell glasses"
|
||||
try:
|
||||
crop = __import__("numpy").zeros((8, 8, 3), dtype="uint8")
|
||||
self.assertEqual(SL.describe_target(object(), crop),
|
||||
"the elderly woman with large tortoiseshell glasses")
|
||||
finally:
|
||||
SL.vlm_generate = old
|
||||
|
||||
|
||||
class RangeTest(unittest.TestCase):
|
||||
def test_start_end_and_max_total_from_the_start(self):
|
||||
self.assertEqual(SL.plan_range(1000, 24.0, 10, 20), (240, 480))
|
||||
self.assertEqual(SL.plan_range(1000, 24.0, 10, 0), (240, 1000))
|
||||
self.assertEqual(SL.plan_range(1000, 24.0, 10, 0, 5), (240, 360)) # max total counted from the start
|
||||
self.assertEqual(SL.plan_range(1000, 24.0, 0, 0, 0, start_f=250, end_f=500), (250, 500)) # frames win
|
||||
self.assertEqual(SL.plan_range(1000, 24.0, 0, 0, 0, start_f=250, end_f=0), (250, 1000))
|
||||
segs = SL.plan_segments(480, [100, 300], "shots", 107, 10, start=240)
|
||||
self.assertEqual(segs[0]["start"], 240)
|
||||
self.assertEqual(segs[-1]["end"], 480)
|
||||
self.assertTrue(all(240 <= s["start"] < s["end"] <= 480 for s in segs))
|
||||
self.assertTrue(any(s["start"] == 300 for s in segs)) # the cut inside the range is kept
|
||||
fixed = SL.plan_segments(480, [], "fixed", 107, 10, start=240)
|
||||
self.assertEqual((fixed[0]["start"], fixed[-1]["end"]), (240, 480))
|
||||
manual = SL.plan_segments(480, [], "manual", 107, 10, manual=[100, 350], start=240)
|
||||
self.assertEqual([(s["start"], s["end"]) for s in manual], [(240, 350), (350, 480)])
|
||||
|
||||
|
||||
class TrackModeTest(unittest.TestCase):
|
||||
def test_track_is_used_by_shots_without_their_own_selection_and_sliced_per_shot(self):
|
||||
p = {"track": {"points": [{"x": 0.5, "y": 0.5, "label": 1}], "frame": 120, "text": "woman"}, "fps": 24}
|
||||
self.assertIn("track", SL.plan_spec(p, {}, "v.mp4")["ext"])
|
||||
self.assertNotIn("ext", SL.plan_spec(p, {"text": "man"}, "v.mp4")) # the shot's own text wins
|
||||
self.assertEqual(SL.plan_spec(p, {"video": "m.mp4"}, "v.mp4")["ext"]["scope"], "shot")
|
||||
whole = torch.zeros(200, 36, 64, dtype=torch.uint8)
|
||||
whole[100:150, 10:20, 10:20] = 1 # the subject only in frames 100-149
|
||||
orig = SL.track_video
|
||||
SL.track_video = lambda path, a, t: (whole, 50) # the track covers frames 50-249
|
||||
try:
|
||||
spec = SL.plan_spec(p, {}, "v.mp4")
|
||||
an = {"n": 300, "fps": 24.0}
|
||||
m = SL.ext_masks(spec["ext"], an, 140, 20, 64, 36) # shot 140-159
|
||||
self.assertEqual(tuple(m.shape), (20, 36, 64))
|
||||
self.assertFalse(bool(m[0].any())) # frame 140 = track index 90
|
||||
self.assertTrue(bool(m[-1].any())) # frame 159 = index 109
|
||||
finally:
|
||||
SL.track_video = orig
|
||||
|
||||
|
||||
class DuetPromptTest(unittest.TestCase):
|
||||
def test_fields_are_found_and_the_vlm_fills_them(self):
|
||||
tpl = ("subject_definitions:\n<Subject 1> is the woman from <Picture 1>: {skin, face}.\n<Subject 2> is the {room} "
|
||||
"from {setting}.\n\nsummary:\nx {layout}")
|
||||
self.assertEqual(SL.template_slots(tpl), ["skin, face", "room"])
|
||||
asked = []
|
||||
|
||||
def gen(clip, q, images, max_tokens, temperature=0.0, seed=0):
|
||||
asked.append(q)
|
||||
if images is None:
|
||||
return tpl.replace("{skin, face}", "pale skin, oval face").replace("{room}", "kitchen")
|
||||
return '{"medium": "phone footage", "actions": "the person waves"}'
|
||||
|
||||
old = SL.vlm_generate, SL.vlm_describe
|
||||
SL.vlm_generate, SL.vlm_describe = gen, lambda clip, ims, q, mt=320, temperature=0.0, seed=0: "the kitchen with white cabinets"
|
||||
try:
|
||||
fr = torch.rand(9, 32, 32, 3)
|
||||
out = SL.duet_prompt(object(), tpl, fr, [torch.rand(1, 32, 32, 3)], "character swap", "", 512,
|
||||
setting=torch.rand(1, 32, 32, 3))
|
||||
self.assertIn("pale skin, oval face", out); self.assertIn("{setting}", out); self.assertIn("{layout}", out)
|
||||
self.assertIn("the person waves", asked[-1])
|
||||
self.assertEqual(SL.duet_prompt(None, tpl, fr, [], "character swap", ""), tpl.strip())
|
||||
self.assertEqual(SL.duet_prompt(object(), "a full prompt", fr, [], "character swap", ""), "a full prompt")
|
||||
w = SL.duet_prompt(object(), "", fr, [torch.rand(1, 32, 32, 3)], "character swap", "", 512,
|
||||
setting=torch.rand(1, 32, 32, 3))
|
||||
self.assertIn("<Subject 2> is the kitchen with white cabinets from {setting}.", w)
|
||||
finally:
|
||||
SL.vlm_generate, SL.vlm_describe = old
|
||||
|
||||
|
||||
class ShotHistoryTest(unittest.TestCase):
|
||||
def test_history_is_taken_out_of_the_shot(self):
|
||||
h = torch.full((17, 8, 8, 3), 255, dtype=torch.uint8)
|
||||
shot = {"chain": "history", "index": 1, "count": 2, "history": h, "history_src": h}
|
||||
hist, src = SL.shot_history(shot)
|
||||
self.assertEqual((tuple(hist.shape), float(hist.max()), "history" in shot), ((17, 8, 8, 3), 1.0, False))
|
||||
self.assertEqual(SL.shot_history({"chain": "off", "index": 1}), (None, None))
|
||||
|
||||
def test_auto_loop_takes_the_previous_render(self):
|
||||
SL._LAST_RESULT.clear()
|
||||
SL._LAST_RESULT.update(index=0, count=2, frames=torch.rand(40, 8, 8, 3))
|
||||
hist, _ = SL.shot_history({"chain": "history", "index": 1, "count": 2, "prev_length": 30, "chain_history": 34})
|
||||
self.assertEqual(hist.shape[0], 17) # 30 real frames -> one whole group of 17
|
||||
SL._LAST_RESULT.clear()
|
||||
|
||||
|
||||
class FillDetailsTest(unittest.TestCase):
|
||||
def test_render_node_fills_details_with_its_vlm(self):
|
||||
old = SL.vlm_describe
|
||||
SL.vlm_describe = lambda clip, ims, q, mt=320, temperature=0.0, seed=0: "The elderly man with white hair in a light blue t-shirt."
|
||||
try:
|
||||
out = SL.fill_details_now("<Subject 1> is the {details} shown in <Picture 1>.", object(), [torch.rand(1, 8, 8, 3)])
|
||||
self.assertEqual(out, "<Subject 1> is the elderly man with white hair in a light blue t-shirt shown in <Picture 1>.")
|
||||
self.assertIn("character", SL.fill_details_now("is the {details} shown", None, []))
|
||||
self.assertEqual(SL.fill_details_now("no slot", None, []), "no slot")
|
||||
finally:
|
||||
SL.vlm_describe = old
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,483 @@
|
||||
"""Auto config: every pixel amount is derived from the subject's own size.
|
||||
|
||||
The knobs the node exposes (mask_grow, mask_blur, uncrop_feather, the latent
|
||||
dilation) are absolute pixel amounts, and a pixel only means something relative
|
||||
to how big the head is in frame. These tests pin the derivation: same head, two
|
||||
distances, and the amounts must scale with it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
import types
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
for name in ("comfy", "comfy.model_management", "comfy.utils"):
|
||||
sys.modules.setdefault(name, types.ModuleType(name))
|
||||
|
||||
# the module reaches comfy.utils.common_upscale as an attribute, so the stub has
|
||||
# to be attached to the package, not just registered in sys.modules
|
||||
def _common_upscale(samples, width, height, upscale_method, crop):
|
||||
return torch.nn.functional.interpolate(samples, size=(height, width), mode="bilinear")
|
||||
|
||||
sys.modules["comfy.utils"].common_upscale = _common_upscale
|
||||
sys.modules["comfy"].utils = sys.modules["comfy.utils"]
|
||||
sys.modules["comfy"].model_management = sys.modules["comfy.model_management"]
|
||||
|
||||
SPEC = importlib.util.spec_from_file_location(
|
||||
"headswap_ltx_masked_sampler", ROOT / "headswap_ltx_masked_sampler.py")
|
||||
assert SPEC is not None and SPEC.loader is not None
|
||||
NODE = importlib.util.module_from_spec(SPEC)
|
||||
sys.modules["headswap_ltx_masked_sampler"] = NODE
|
||||
SPEC.loader.exec_module(NODE)
|
||||
|
||||
|
||||
def masks(boxes, h=1080, w=1920):
|
||||
m = torch.zeros(len(boxes), h, w)
|
||||
for i, (x, y, bw, bh) in enumerate(boxes):
|
||||
m[i, y:y + bh, x:x + bw] = 1.0
|
||||
return m
|
||||
|
||||
|
||||
def auto(boxes, cell=32, ref_aspect=None, headroom=1.15, **kw):
|
||||
st = NODE._subject_stats(masks(boxes, **kw))
|
||||
return NODE._auto_config(st, cell, ref_aspect, headroom)
|
||||
|
||||
|
||||
def sides(a):
|
||||
"""(up, down, left, right) growth."""
|
||||
return a["mask_grow"]
|
||||
|
||||
|
||||
class SubjectStatsTest(unittest.TestCase):
|
||||
def test_median_ignores_a_broken_frame(self):
|
||||
boxes = [(900, 300, 100, 130)] * 20
|
||||
boxes[7] = (900, 300, 900, 900) # one blown-up mask frame
|
||||
st = NODE._subject_stats(masks(boxes))
|
||||
self.assertEqual(st["head_w"], 100.0)
|
||||
|
||||
def test_empty_mask_reports_nothing(self):
|
||||
self.assertIsNone(NODE._subject_stats(torch.zeros(8, 64, 64)))
|
||||
|
||||
def test_frames_without_the_subject_do_not_break_travel(self):
|
||||
boxes = [(100, 300, 100, 130), (0, 0, 0, 0), (900, 300, 100, 130)]
|
||||
st = NODE._subject_stats(masks(boxes))
|
||||
self.assertEqual(st["seen"], 2)
|
||||
self.assertAlmostEqual(st["travel"], 800.0)
|
||||
|
||||
|
||||
class AutoConfigTest(unittest.TestCase):
|
||||
def test_amounts_scale_with_the_head(self):
|
||||
near = auto([(700, 200, 600, 760)] * 12)
|
||||
far = auto([(900, 300, 100, 130)] * 12)
|
||||
self.assertGreater(sides(near)[2], sides(far)[2] * 4)
|
||||
self.assertGreater(near["mask_blur"], far["mask_blur"] * 4)
|
||||
self.assertGreater(near["uncrop_feather"], far["uncrop_feather"])
|
||||
|
||||
def test_feather_fits_inside_the_margin_it_has(self):
|
||||
# a ramp wider than the gap between the mask and the crop border would
|
||||
# fade the head itself -- this is the seam the auto mode exists to avoid
|
||||
for w in (60, 100, 240, 600, 900):
|
||||
a = auto([(400, 200, w, int(w * 1.3))] * 8)
|
||||
margin = w * (a["crop_scale"] - 1.0) / 2.0
|
||||
left = margin - sides(a)[2]
|
||||
self.assertLessEqual(a["uncrop_feather"], max(4.0, 0.5 * left) + 1e-6,
|
||||
f"feather spills past the margin at head width {w}")
|
||||
|
||||
def test_a_head_that_fills_the_frame_turns_cropping_off(self):
|
||||
self.assertEqual(auto([(300, 100, 1150, 950)] * 8)["crop_mode"], "off")
|
||||
|
||||
def test_a_still_subject_gets_one_static_box(self):
|
||||
self.assertEqual(auto([(900, 300, 120, 150)] * 8)["crop_mode"], "combined")
|
||||
|
||||
def test_a_travelling_subject_is_tracked(self):
|
||||
boxes = [(200 + i * 18, 300, 120, 150) for i in range(40)]
|
||||
self.assertEqual(auto(boxes)["crop_mode"], "tracked")
|
||||
|
||||
def test_a_subject_changing_size_is_zoomed(self):
|
||||
boxes = [(900 - i * 3, 300, 100 + i * 4, 130 + i * 5) for i in range(40)]
|
||||
self.assertEqual(auto(boxes)["crop_mode"], "zoomed")
|
||||
|
||||
def test_the_box_never_asks_for_more_than_the_frame(self):
|
||||
a = auto([(100, 100, 1400, 900)] * 8)
|
||||
self.assertLessEqual(1400 * a["crop_scale"], 1920 + 1e-6)
|
||||
self.assertLessEqual(900 * a["crop_scale"], 1080 + 1e-6)
|
||||
|
||||
def test_the_reference_head_may_be_bigger_than_the_mask(self):
|
||||
# the mask is the OLD head: a swap clipped at its edge is the seam
|
||||
tight = auto([(900, 300, 120, 160)] * 8, headroom=1.0)
|
||||
roomy = auto([(900, 300, 120, 160)] * 8, headroom=1.5)
|
||||
self.assertGreater(sides(roomy)[0], sides(tight)[0])
|
||||
self.assertGreater(sides(roomy)[2], sides(tight)[2])
|
||||
|
||||
def test_growth_never_goes_down_into_the_neck(self):
|
||||
for hr in (1.0, 1.25, 1.5, 2.0):
|
||||
up, down, left, right = sides(auto([(900, 300, 120, 160)] * 8, headroom=hr))
|
||||
self.assertLess(down, up, "growing down eats the neck and collar")
|
||||
self.assertLessEqual(down, left + 1)
|
||||
|
||||
def test_a_wider_reference_pushes_the_slack_sideways(self):
|
||||
# below the caps, where the tilt still has room to act
|
||||
wide = auto([(900, 300, 120, 160)] * 8, ref_aspect=1.4, headroom=1.15)
|
||||
tall = auto([(900, 300, 120, 160)] * 8, ref_aspect=0.5, headroom=1.15)
|
||||
self.assertGreater(sides(wide)[2], sides(tall)[2])
|
||||
self.assertGreater(sides(tall)[0], sides(wide)[0])
|
||||
|
||||
def test_the_box_always_holds_the_grow_plus_the_ramp(self):
|
||||
# the real invariant: whatever the mask grew by, the crop still has room
|
||||
# for it AND for the feather, or the paste fades the head
|
||||
for hr in (1.0, 1.15, 1.5, 2.0):
|
||||
for ref in (0.5, 1.0, 1.6):
|
||||
a = auto([(900, 300, 120, 160)] * 8, ref_aspect=ref, headroom=hr)
|
||||
up, down, left, _ = sides(a)
|
||||
margin_x = 120 * (a["crop_scale"] - 1.0) / 2.0
|
||||
margin_y = 160 * (a["crop_scale"] - 1.0) / 2.0
|
||||
self.assertLessEqual(left + a["uncrop_feather"], margin_x + 1)
|
||||
self.assertLessEqual(max(up, down) + a["uncrop_feather"], margin_y + 1)
|
||||
|
||||
def test_latent_dilation_is_not_added_on_top_of_the_pixel_grow(self):
|
||||
# _mask_to_latent reduces with MAX, so a cell the mask touches is already
|
||||
# editable; dilating here too added a whole 32 px cell on every side and
|
||||
# blew the hole up far past the head
|
||||
for w in (40, 100, 600):
|
||||
self.assertEqual(auto([(400, 200, w, w)] * 8)["latent_mask_dilate"], 0)
|
||||
|
||||
def test_growth_stays_a_sane_fraction_of_the_head(self):
|
||||
# the failure this pins: slack scaled by the mask's HEIGHT produced a
|
||||
# hole much larger than the head, with the new head floating inside it
|
||||
for hw, hh in ((120, 160), (300, 400), (600, 900)):
|
||||
for ref, hr in ((0.5, 1.15), (1.0, 1.15), (0.75, 1.5), (1.4, 2.0)):
|
||||
up, down, left, _ = sides(auto([(300, 100, hw, hh)] * 8,
|
||||
ref_aspect=ref, headroom=hr))
|
||||
self.assertLessEqual(up, 0.45 * hw, f"up runs away at {hw}x{hh} ref {ref} hr {hr}")
|
||||
self.assertLessEqual(left, 0.40 * hw, f"sides run away at {hw}x{hh} ref {ref} hr {hr}")
|
||||
|
||||
def test_a_fast_head_gets_slack_along_time(self):
|
||||
still = auto([(800, 300, 100, 130)] * 20)
|
||||
jumpy = auto([(800 + (60 if i % 2 else 0), 300, 100, 130) for i in range(20)])
|
||||
self.assertEqual(still["latent_mask_dilate_frames"], 0)
|
||||
self.assertEqual(jumpy["latent_mask_dilate_frames"], 1)
|
||||
|
||||
def test_the_note_says_what_it_chose(self):
|
||||
note = auto([(900, 300, 120, 150)] * 8)["note"]
|
||||
for token in ("head", "crop", "grow", "blur", "feather", "latent dilate",
|
||||
"headroom", "reference aspect"):
|
||||
self.assertIn(token, note)
|
||||
|
||||
|
||||
class DirectionalGrowTest(unittest.TestCase):
|
||||
def test_each_side_grows_by_exactly_its_own_amount(self):
|
||||
m = torch.zeros(1, 40, 40)
|
||||
m[0, 20, 20] = 1.0
|
||||
out = NODE._grow_blur(m, (5, 1, 3, 2), 0)[0]
|
||||
ys, xs = torch.where(out > 0.5)
|
||||
self.assertEqual((int(ys.min()), int(ys.max())), (15, 21)) # up 5, down 1
|
||||
self.assertEqual((int(xs.min()), int(xs.max())), (17, 22)) # left 3, right 2
|
||||
|
||||
def test_a_single_number_still_grows_every_side(self):
|
||||
m = torch.zeros(1, 40, 40)
|
||||
m[0, 20, 20] = 1.0
|
||||
out = NODE._grow_blur(m, 4, 0)[0]
|
||||
ys, xs = torch.where(out > 0.5)
|
||||
self.assertEqual((int(ys.min()), int(ys.max())), (16, 24))
|
||||
self.assertEqual((int(xs.min()), int(xs.max())), (16, 24))
|
||||
|
||||
def test_zero_growth_leaves_the_mask_alone(self):
|
||||
m = torch.rand(2, 16, 16).round()
|
||||
self.assertTrue(torch.equal(NODE._grow_blur(m, (0, 0, 0, 0), 0), m))
|
||||
|
||||
|
||||
class GenericIcLoraTest(unittest.TestCase):
|
||||
"""The node is not head-swap specific: a LoRA that works from the guide
|
||||
alone must be able to run with the reference slot empty."""
|
||||
|
||||
def test_identity_image_is_optional(self):
|
||||
types = NODE.BFSHeadSwapMaskedSampler.INPUT_TYPES()
|
||||
self.assertNotIn("identity_image", types["required"])
|
||||
self.assertIn("identity_image", types["optional"])
|
||||
|
||||
def test_execute_defaults_it_to_none(self):
|
||||
import inspect
|
||||
sig = inspect.signature(NODE.BFSHeadSwapMaskedSampler.execute)
|
||||
self.assertIsNone(sig.parameters["identity_image"].default)
|
||||
|
||||
def test_the_guide_is_still_required(self):
|
||||
self.assertIn("guide_video", NODE.BFSHeadSwapMaskedSampler.INPUT_TYPES()["required"])
|
||||
|
||||
def test_registration_keys_are_unchanged(self):
|
||||
# saved workflows reference these, the display name is free to change
|
||||
self.assertIn("BFSHeadSwapMaskedSampler", NODE.NODE_CLASS_MAPPINGS)
|
||||
self.assertIn("BFSHeadSwapPasteBack", NODE.NODE_CLASS_MAPPINGS)
|
||||
|
||||
def test_the_display_name_no_longer_claims_head_swap(self):
|
||||
for name in NODE.NODE_DISPLAY_NAME_MAPPINGS.values():
|
||||
self.assertNotIn("Head Swap", name)
|
||||
|
||||
|
||||
class PasteBackWithoutCropTest(unittest.TestCase):
|
||||
"""Crop off, mask on: everything the edit did not touch must stay the
|
||||
source's own pixels, not the VAE's round trip of them."""
|
||||
|
||||
def setUp(self):
|
||||
self.node = NODE.BFSHeadSwapMaskedSampler()
|
||||
self.original = torch.zeros(3, 8, 8, 3) # source frames
|
||||
self.result = torch.ones(3, 8, 8, 3) # what the sampler returned
|
||||
self.mask = torch.zeros(3, 8, 8)
|
||||
self.mask[:, 2:5, 2:5] = 1.0
|
||||
|
||||
def test_outside_the_mask_is_the_source_verbatim(self):
|
||||
out = self.node._paste_back(self.result, self.original, None, 16, self.mask)
|
||||
self.assertTrue(torch.equal(out[:, 0, 0], self.original[:, 0, 0]))
|
||||
self.assertTrue(torch.equal(out[:, 7, 7], self.original[:, 7, 7]))
|
||||
|
||||
def test_inside_the_mask_is_the_result(self):
|
||||
out = self.node._paste_back(self.result, self.original, None, 16, self.mask)
|
||||
self.assertTrue(torch.equal(out[:, 3, 3], self.result[:, 3, 3]))
|
||||
|
||||
def test_without_a_mask_nothing_is_composited(self):
|
||||
out = self.node._paste_back(self.result, self.original, None, 16, None)
|
||||
self.assertTrue(torch.equal(out, self.result))
|
||||
|
||||
def test_a_result_of_another_size_is_brought_back_to_the_frame(self):
|
||||
big = torch.ones(3, 16, 16, 3)
|
||||
out = self.node._paste_back(big, self.original, None, 16, self.mask)
|
||||
self.assertEqual(out.shape, self.original.shape)
|
||||
|
||||
|
||||
class ChunkOverlapTest(unittest.TestCase):
|
||||
"""Chunks are sampled with an overlap for continuity. Concatenating them
|
||||
whole made the clip longer than it is, so every frame after the first seam
|
||||
landed on the wrong moment -- and, with per-frame crop boxes, in the wrong
|
||||
box. 97 frames at tile 72 / overlap 16 produced 15 latent frames where the
|
||||
clip has 13."""
|
||||
|
||||
@staticmethod
|
||||
def run_chunks(lat_per_chunk, total):
|
||||
produced, out = 0, []
|
||||
for t in lat_per_chunk:
|
||||
got, produced = NODE._keep_new_frames(torch.zeros(1, 4, t, 2, 2), produced, total)
|
||||
out.append(got)
|
||||
return torch.cat(out, dim=2)
|
||||
|
||||
def test_the_concatenation_is_exactly_the_clip(self):
|
||||
self.assertEqual(self.run_chunks([9, 6], 13).shape[2], 13)
|
||||
|
||||
def test_it_keeps_each_chunk_s_tail_not_its_head(self):
|
||||
a = torch.zeros(1, 4, 9, 2, 2)
|
||||
b = torch.arange(6, dtype=torch.float32).view(1, 1, 6, 1, 1).expand(1, 4, 6, 2, 2).clone()
|
||||
_, produced = NODE._keep_new_frames(a, 0, 13)
|
||||
kept, _ = NODE._keep_new_frames(b, produced, 13)
|
||||
self.assertEqual(kept.shape[2], 4)
|
||||
self.assertEqual(kept[0, 0, 0, 0, 0].item(), 2.0) # frames 2..5, the new tail
|
||||
|
||||
def test_a_single_chunk_is_untouched(self):
|
||||
got, produced = NODE._keep_new_frames(torch.zeros(1, 4, 13, 2, 2), 0, 13)
|
||||
self.assertEqual(got.shape[2], 13)
|
||||
self.assertEqual(produced, 13)
|
||||
|
||||
def test_it_never_returns_more_than_is_left(self):
|
||||
got, _ = NODE._keep_new_frames(torch.zeros(1, 4, 9, 2, 2), 12, 13)
|
||||
self.assertEqual(got.shape[2], 1)
|
||||
|
||||
|
||||
class CropAspectTest(unittest.TestCase):
|
||||
"""The crop is resized into the connected latent, so the box has to share
|
||||
its aspect. Free-shaped boxes were stretched on the way in and squeezed on
|
||||
the way back -- a portrait face crop sampled in a landscape latent came out
|
||||
visibly deformed. The latent's SIZE is deliberately not touched: sampling a
|
||||
small region at the model's resolution is why cropping exists."""
|
||||
|
||||
def setUp(self):
|
||||
self.node = NODE.BFSHeadSwapMaskedSampler()
|
||||
self.video = torch.zeros(6, 1080, 1920, 3)
|
||||
self.mask = torch.zeros(6, 1080, 1920)
|
||||
self.mask[:, 300:560, 900:1100] = 1.0 # 200x260, portrait
|
||||
|
||||
def box(self, aspect):
|
||||
cropped, _, ctx, _ = self.node._crop(
|
||||
self.video, self.mask, "off" if False else "combined", 1.2, 32, aspect)
|
||||
return cropped.shape[2], cropped.shape[1], ctx
|
||||
|
||||
def test_a_free_box_follows_the_subject(self):
|
||||
w, h, _ = self.box(0.0)
|
||||
self.assertLess(w, h) # portrait, like the mask
|
||||
|
||||
def test_a_landscape_latent_gives_a_landscape_box(self):
|
||||
w, h, _ = self.box(768 / 512)
|
||||
self.assertAlmostEqual(w / h, 768 / 512, delta=0.25)
|
||||
self.assertGreater(w, h)
|
||||
|
||||
def test_the_box_still_contains_the_subject(self):
|
||||
w, h, ctx = self.box(768 / 512)
|
||||
x0, y0, bw, bh = ctx[1]
|
||||
self.assertLessEqual(x0, 900)
|
||||
self.assertGreaterEqual(x0 + bw, 1100)
|
||||
self.assertLessEqual(y0, 300)
|
||||
self.assertGreaterEqual(y0 + bh, 560)
|
||||
|
||||
def test_the_box_stays_divisible(self):
|
||||
w, h, _ = self.box(1.5)
|
||||
self.assertEqual((w % 32, h % 32), (0, 0))
|
||||
|
||||
def test_static_box_honours_the_aspect_directly(self):
|
||||
x0, y0, w, h = NODE._static_box(self.mask, self.video, 1.2, 32, aspect=2.0)
|
||||
self.assertAlmostEqual(w / h, 2.0, delta=0.3)
|
||||
|
||||
|
||||
class CropSizeTest(unittest.TestCase):
|
||||
"""The size to build the empty latent at, known before sampling instead of
|
||||
read off the debug string after a whole pass."""
|
||||
|
||||
def setUp(self):
|
||||
self.video = torch.zeros(6, 256, 512, 3)
|
||||
self.mask = torch.zeros(6, 256, 512)
|
||||
self.mask[:, 60:160, 200:280] = 1.0 # 80x100, portrait
|
||||
|
||||
def size(self, mode="combined", scale=1.5, div=32):
|
||||
w, h, boxes, info = NODE.BFSCropSize().execute(
|
||||
self.video, self.mask, mode, scale, div)
|
||||
return w, h, boxes, info
|
||||
|
||||
def test_it_reports_the_size_the_sampler_would_crop_to(self):
|
||||
w, h, _, _ = self.size()
|
||||
cropped, _, _, _ = NODE.BFSHeadSwapMaskedSampler()._crop(
|
||||
self.video, self.mask, "combined", 1.5, 32)
|
||||
self.assertEqual((w, h), (cropped.shape[2], cropped.shape[1]))
|
||||
|
||||
def test_the_size_is_divisible_by_the_grid(self):
|
||||
for div in (8, 32, 64):
|
||||
w, h, _, _ = self.size(div=div)
|
||||
self.assertEqual((w % div, h % div), (0, 0))
|
||||
|
||||
def test_it_follows_the_subject_s_aspect_not_the_frame_s(self):
|
||||
w, h, _, _ = self.size()
|
||||
self.assertLess(w, h) # the mask is portrait
|
||||
|
||||
def test_one_box_per_frame(self):
|
||||
_, _, boxes, _ = self.size()
|
||||
self.assertEqual(len(boxes), self.video.shape[0])
|
||||
|
||||
def test_the_info_names_the_size(self):
|
||||
w, h, _, info = self.size()
|
||||
self.assertIn(f"{w}x{h}", info)
|
||||
|
||||
def test_it_is_registered(self):
|
||||
self.assertIn("BFSCropSize", NODE.NODE_CLASS_MAPPINGS)
|
||||
self.assertEqual(NODE.NODE_DISPLAY_NAME_MAPPINGS["BFSCropSize"], "BFS Crop Size")
|
||||
|
||||
|
||||
class NoShadowedModulesTest(unittest.TestCase):
|
||||
"""`import comfy.x` inside a function binds `comfy` as a LOCAL for the whole
|
||||
function, so every earlier comfy.* use in it raises UnboundLocalError. It
|
||||
shipped in 1.37.0 and only fired on multi-chunk clips, where the offload
|
||||
line runs and the nested-latent import never had."""
|
||||
|
||||
def test_execute_does_not_shadow_the_comfy_module(self):
|
||||
for cls in (NODE.BFSHeadSwapMaskedSampler, NODE.BFSHeadSwapPasteBack):
|
||||
self.assertNotIn("comfy", cls.execute.__code__.co_varnames,
|
||||
f"{cls.__name__}.execute binds `comfy` locally")
|
||||
|
||||
|
||||
class WidgetOrderTest(unittest.TestCase):
|
||||
"""ComfyUI stores widgets_values POSITIONALLY.
|
||||
|
||||
A widget added anywhere but the end shifts every later value in every saved
|
||||
workflow — 1.37.0 inserted two before crop_mode and existing graphs failed
|
||||
validation with "input out of range" on crop_scale and crop_divisible_by.
|
||||
This pins the historical order; new widgets go after it, never inside it.
|
||||
"""
|
||||
|
||||
HISTORICAL = [
|
||||
"crop_mode", "crop_scale", "crop_divisible_by", "uncrop_feather",
|
||||
"paste_back", "paste_confine_to_mask", "inpaint_with_mask", "mask_grow",
|
||||
"mask_blur", "mask_hard_for_inpaint", "latent_mask_dilate",
|
||||
"latent_mask_dilate_frames", "mask_strength", "decode",
|
||||
"decode_tile_size", "decode_overlap", "decode_temporal_size",
|
||||
"decode_temporal_overlap", "temporal_tile_size", "temporal_overlap",
|
||||
"guide_source_id", "identity_source_id", "debug_log",
|
||||
]
|
||||
|
||||
@staticmethod
|
||||
def widgets():
|
||||
opt = NODE.BFSHeadSwapMaskedSampler.INPUT_TYPES()["optional"]
|
||||
kinds = ("BOOLEAN", "INT", "FLOAT", "STRING")
|
||||
return [k for k, v in opt.items() if isinstance(v[0], list) or v[0] in kinds]
|
||||
|
||||
def test_the_historical_widgets_keep_their_positions(self):
|
||||
self.assertEqual(self.widgets()[: len(self.HISTORICAL)], self.HISTORICAL)
|
||||
|
||||
def test_new_widgets_are_appended_after_them(self):
|
||||
self.assertEqual(self.widgets()[len(self.HISTORICAL):],
|
||||
["auto_config", "identity_headroom"])
|
||||
|
||||
|
||||
class WiringTest(unittest.TestCase):
|
||||
def test_the_widget_exists_and_defaults_to_off(self):
|
||||
opt = NODE.BFSHeadSwapMaskedSampler.INPUT_TYPES()["optional"]
|
||||
self.assertIn("auto_config", opt)
|
||||
self.assertEqual(opt["auto_config"][1]["default"], False)
|
||||
self.assertIn("identity_headroom", opt)
|
||||
self.assertEqual(opt["identity_headroom"][1]["default"], 1.15)
|
||||
|
||||
def test_execute_accepts_it(self):
|
||||
import inspect
|
||||
sig = inspect.signature(NODE.BFSHeadSwapMaskedSampler.execute)
|
||||
self.assertIn("auto_config", sig.parameters)
|
||||
self.assertIs(sig.parameters["auto_config"].default, False)
|
||||
self.assertEqual(sig.parameters["identity_headroom"].default, 1.15)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
|
||||
class GuiderInjectionTest(unittest.TestCase):
|
||||
"""The guider belongs to another node and survives between runs.
|
||||
|
||||
Whatever is written into its options has to come back out, or the reference
|
||||
latents stay reachable (VRAM that is never freed) and the next run samples
|
||||
with the previous clip's specs.
|
||||
"""
|
||||
|
||||
class _Guider:
|
||||
def __init__(self, options=None):
|
||||
self.model_options = options if options is not None else {}
|
||||
|
||||
class _Patcher:
|
||||
def __init__(self, opts):
|
||||
self.model_options = {"transformer_options": opts}
|
||||
|
||||
def test_injection_is_undone(self):
|
||||
g = self._Guider()
|
||||
p = self._Patcher({"_id_ref_specs": ["latents"]})
|
||||
inj = NODE._inject_transformer_options(g, p)
|
||||
self.assertEqual(g.model_options["transformer_options"]["_id_ref_specs"], ["latents"])
|
||||
inj.undo()
|
||||
self.assertNotIn("_id_ref_specs", g.model_options["transformer_options"])
|
||||
|
||||
def test_undo_restores_a_pre_existing_value_instead_of_deleting_it(self):
|
||||
g = self._Guider({"transformer_options": {"_id_ref_specs": "theirs"}})
|
||||
p = self._Patcher({"_id_ref_specs": "ours"})
|
||||
NODE._inject_transformer_options(g, p).undo()
|
||||
self.assertEqual(g.model_options["transformer_options"]["_id_ref_specs"], "theirs")
|
||||
|
||||
def test_undo_is_idempotent(self):
|
||||
g = self._Guider()
|
||||
inj = NODE._inject_transformer_options(g, self._Patcher({"k": 1}))
|
||||
inj.undo()
|
||||
inj.undo()
|
||||
self.assertEqual(g.model_options["transformer_options"], {})
|
||||
|
||||
def test_nothing_to_inject_still_returns_something_undoable(self):
|
||||
NODE._inject_transformer_options(self._Guider(), self._Patcher({})).undo()
|
||||
|
||||
def test_the_loop_undoes_in_a_finally(self):
|
||||
import inspect
|
||||
src = inspect.getsource(NODE.BFSHeadSwapMaskedSampler.execute)
|
||||
self.assertIn("finally:", src)
|
||||
self.assertIn("inj.undo()", src)
|
||||
@@ -0,0 +1,259 @@
|
||||
"""CPU tests against the audited native ComfyUI layout and training fixtures."""
|
||||
import copy
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
import numpy as np
|
||||
from PIL import Image, ImageOps
|
||||
import torch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
def load(name, path):
|
||||
spec = importlib.util.spec_from_file_location(name, path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[name] = module
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
NODE = load('bfs_h3_guide_test', ROOT / 'minimax_h3_downscaled_guide.py')
|
||||
CORE = load('bfs_h3_native_layout_test', ROOT / 'tests/fixtures/comfy_h3_layout_8cfe5e1.py')
|
||||
|
||||
|
||||
def guide(factor, vt=2, h=8, w=8, index=0):
|
||||
return {'resolved_frame_index': index, 'latent': torch.zeros(1, 24, vt, h//factor, w//factor),
|
||||
NODE.MARKER: {'downscale_factor': factor}}
|
||||
|
||||
|
||||
class GuideLayoutTest(unittest.TestCase):
|
||||
def test_positions_match_training_for_factors_1_2_4(self):
|
||||
golden = json.loads((ROOT/'tests/fixtures/h3_training_guide_positions.json').read_text())
|
||||
for factor in (1, 2, 4):
|
||||
with self.subTest(factor=factor):
|
||||
keyframes = [guide(factor)]
|
||||
native = CORE.PackedLayout(3, 2, 8, 8, 2, keyframes=keyframes)
|
||||
layout = NODE.downscaled_layout(native, keyframes)
|
||||
expected = golden['factors'][str(factor)]
|
||||
torch.testing.assert_close(layout.position_ids, torch.tensor(expected['positions'], dtype=torch.float64), rtol=0, atol=1e-14)
|
||||
self.assertEqual(layout.img_pos.tolist(), expected['video_indices'])
|
||||
self.assertEqual(layout.audio_pos.tolist(), expected['audio_indices'])
|
||||
self.assertEqual(layout.seq_len, len(expected['positions']))
|
||||
self.assertEqual(layout.segments[-1][1], layout.seq_len)
|
||||
self.assertEqual(len(layout.img_pos) + len(layout.audio_pos) + 3, layout.seq_len)
|
||||
self.assertEqual(native.seq_len, 3+32+4+32)
|
||||
|
||||
def test_73_frame_guide_counts_and_frozen_rows(self):
|
||||
keyframes = [guide(4, vt=22)]
|
||||
native = CORE.PackedLayout(3, 22, 8, 8, 122, keyframes=keyframes)
|
||||
layout = NODE.downscaled_layout(native, keyframes)
|
||||
self.assertEqual(layout.segments[1], (3, 25, 'cond'))
|
||||
self.assertFalse(layout.img_update[:22].any())
|
||||
self.assertTrue(layout.img_update[22:].all())
|
||||
self.assertEqual(layout.seq_len, 3+22+244+22*16)
|
||||
original_target = native.position_ids[native.segments[-1][0]:]
|
||||
torch.testing.assert_close(layout.position_ids[layout.segments[-1][0]:], original_target)
|
||||
|
||||
def test_multiple_guides_native_refs_and_audio_keep_their_coordinates(self):
|
||||
keyframes = [guide(4, vt=1), guide(2, vt=2, index=1)]
|
||||
keyframes[0]['audio_latent'] = torch.zeros(1,32,2,2)
|
||||
refs = [{'kind':'image','latent_h':4,'latent_w':8}]
|
||||
native = CORE.PackedLayout(4,7,8,8,40,keyframes=keyframes,refs=refs)
|
||||
result = NODE.downscaled_layout(native,keyframes)
|
||||
for orig, changed in zip(native.segments,result.segments):
|
||||
if orig[2] != 'cond':
|
||||
torch.testing.assert_close(native.position_ids[orig[0]:orig[1]],result.position_ids[changed[0]:changed[1]])
|
||||
self.assertEqual(result.segments[-1][1],result.seq_len)
|
||||
self.assertTrue((result.position_ids[result.segments[1][0],0] == 5).item())
|
||||
|
||||
def test_invalid_guide_shape_fails(self):
|
||||
keyframes=[guide(4)]
|
||||
keyframes[0]['latent']=torch.zeros(1,24,2,4,4)
|
||||
native=CORE.PackedLayout(3,2,8,8,2,keyframes=keyframes)
|
||||
with self.assertRaisesRegex(ValueError,'dimensions'):
|
||||
NODE.downscaled_layout(native,keyframes)
|
||||
|
||||
def test_standard_guide_keeps_full_resolution(self):
|
||||
keyframes=[guide(4),{'resolved_frame_index':0,'latent':torch.zeros(1,24,1,8,8)}]
|
||||
native=CORE.PackedLayout(3,2,8,8,2,keyframes=keyframes)
|
||||
result=NODE.downscaled_layout(native,keyframes)
|
||||
self.assertEqual(result.segments[2][1]-result.segments[2][0],16)
|
||||
|
||||
|
||||
class PreprocessingTest(unittest.TestCase):
|
||||
def test_exact_training_crop_and_resize(self):
|
||||
source=np.random.default_rng(7).integers(0,256,(80,120,3),dtype=np.uint8)
|
||||
images=torch.from_numpy(source.copy()).float().unsqueeze(0)/255
|
||||
result=NODE.prepare_guide_frames(images,128,128,4)
|
||||
expected=ImageOps.fit(Image.fromarray(source),(128,128),Image.Resampling.BICUBIC).resize((32,32),Image.Resampling.LANCZOS)
|
||||
self.assertEqual(tuple(result.shape),(1,32,32,3))
|
||||
np.testing.assert_array_equal((result[0].numpy()*255).round().astype(np.uint8),np.array(expected))
|
||||
|
||||
def test_bad_canvas_and_factor(self):
|
||||
image=torch.zeros(1,8,8,3)
|
||||
for factor in (0,True,1.5):
|
||||
with self.assertRaises(ValueError): NODE.prepare_guide_frames(image,128,128,factor)
|
||||
with self.assertRaisesRegex(ValueError,'multiples'): NODE.prepare_guide_frames(image,160,128,4)
|
||||
|
||||
def test_posterior_seed_and_fp16_recipe_without_global_rng_changes(self):
|
||||
moments=torch.zeros(1,48,1,2,2)
|
||||
moments[:,:24]=.123456
|
||||
mean=torch.full((24,),.25); std=torch.full((24,),1.5)
|
||||
before=torch.random.get_rng_state()
|
||||
result=NODE.sample_posterior(moments,mean,std,fp16_round=True)
|
||||
generator=torch.Generator('cpu').manual_seed(42)
|
||||
expected=(.123456+torch.randn((1,24,1,2,2),generator=generator)).half().float()
|
||||
expected=(expected-.25)/1.5
|
||||
torch.testing.assert_close(result,expected,rtol=0,atol=0)
|
||||
self.assertTrue(torch.equal(before,torch.random.get_rng_state()))
|
||||
torch.testing.assert_close(result,NODE.sample_posterior(moments,mean,std,fp16_round=True))
|
||||
|
||||
|
||||
class EncoderMomentsTest(unittest.TestCase):
|
||||
def stage(self):
|
||||
calls = []
|
||||
def normalize(x):
|
||||
self.assertEqual(x.dtype, torch.float32)
|
||||
return x + .123456
|
||||
def adaptive(x):
|
||||
calls.append(x.clone())
|
||||
self.assertEqual(x.dtype, torch.float16)
|
||||
return torch.zeros(1,48,5,2,2)
|
||||
return SimpleNamespace(clip_length=17, token_drop=3,
|
||||
_normalize_pixels=normalize, _adaptive_encode=adaptive), calls
|
||||
|
||||
def test_image_uses_last_posterior_slice(self):
|
||||
stage,calls = self.stage()
|
||||
result = NODE.encode_moments(stage,torch.zeros(1,3,1,32,32),'cpu',torch.float16)
|
||||
self.assertEqual(tuple(result.shape),(1,48,1,2,2))
|
||||
self.assertEqual(len(calls),1)
|
||||
|
||||
def test_video_chunk_padding_and_tail_drop_match_native_h3(self):
|
||||
stage,calls = self.stage()
|
||||
pixels = torch.arange(73,dtype=torch.float32).view(1,1,73,1,1).expand(1,3,73,2,2)
|
||||
result = NODE.encode_moments(stage,pixels,'cpu',torch.float16)
|
||||
self.assertEqual(result.shape[2],22)
|
||||
self.assertEqual(len(calls),5)
|
||||
self.assertEqual(calls[-1].shape[2],17)
|
||||
torch.testing.assert_close(calls[-1][:,:,4],calls[-1][:,:,16])
|
||||
|
||||
|
||||
class FakeModel:
|
||||
def __init__(self, options=None): self.model_options=options or {}
|
||||
def clone(self): return FakeModel(dict(self.model_options))
|
||||
def set_model_unet_function_wrapper(self,fn): self.model_options['model_function_wrapper']=fn
|
||||
|
||||
|
||||
class WrapperTest(unittest.TestCase):
|
||||
def test_scoped_wrapper_composes_and_does_not_mutate_payload(self):
|
||||
calls=[]
|
||||
def previous(fn,args):
|
||||
calls.append('previous')
|
||||
return fn(args['input'],args['timestep'],**args['c'])
|
||||
original=FakeModel({'model_function_wrapper':previous})
|
||||
patched=NODE.patch_guide_model(original)
|
||||
keyframes=[guide(4)]
|
||||
native=CORE.PackedLayout(3,2,8,8,2,keyframes=keyframes)
|
||||
payload={'keyframes':keyframes,'layout':native}
|
||||
args={'input':'x','timestep':'t','c':{'minimax_payload':payload}}
|
||||
def model_fn(x,t,**c): return c['minimax_payload']['layout']
|
||||
result=patched.model_options['model_function_wrapper'](model_fn,args)
|
||||
self.assertEqual(result.seq_len,41)
|
||||
self.assertIs(payload['layout'],native)
|
||||
self.assertIs(original.model_options['model_function_wrapper'],previous)
|
||||
self.assertEqual(calls,['previous'])
|
||||
again=NODE.patch_guide_model(patched)
|
||||
self.assertIs(again.model_options['model_function_wrapper'],patched.model_options['model_function_wrapper'])
|
||||
self.assertIs(patched.model_options['model_function_wrapper'](model_fn,args),result)
|
||||
|
||||
def test_other_models_and_conditioning_are_untouched(self):
|
||||
model=NODE.patch_guide_model(FakeModel())
|
||||
args={'input':1,'timestep':2,'c':{'other':3}}
|
||||
result=model.model_options['model_function_wrapper'](lambda x,t,**c:(x,t,c),args)
|
||||
self.assertEqual(result,(1,2,{'other':3}))
|
||||
|
||||
|
||||
|
||||
class NodeExecutionTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
import types
|
||||
self.comfy=types.ModuleType('comfy')
|
||||
ldm=types.ModuleType('comfy.ldm'); minimax=types.ModuleType('comfy.ldm.minimax')
|
||||
core=types.ModuleType('comfy.ldm.minimax.model')
|
||||
class Diffusion:
|
||||
patch_size=(1,2,2)
|
||||
core.MiniMaxH3Model=Diffusion
|
||||
core.FRAME_PER_TOKEN=(1,4,4,4,4); core.FRAME_RESCALE=5/3
|
||||
minimax.model=core; ldm.minimax=minimax; self.comfy.ldm=ldm
|
||||
self.modules={'comfy':self.comfy,'comfy.ldm':ldm,'comfy.ldm.minimax':minimax,'comfy.ldm.minimax.model':core}
|
||||
self.diffusion=Diffusion()
|
||||
self.model=FakeModel()
|
||||
self.model.get_model_object=lambda key:self.diffusion
|
||||
|
||||
def target(self, vt):
|
||||
return {'samples':SimpleNamespace(is_nested=True,tensors=[torch.zeros(1,24,vt,8,8),torch.zeros(1,32,2,122)])}
|
||||
|
||||
def test_image_node_encodes_reduced_guide_and_keeps_inputs(self):
|
||||
embedding=torch.zeros(1,3,4)
|
||||
positive=[[embedding,{'minimax_token_tags':torch.ones(3)}]]
|
||||
with patch.dict(sys.modules,self.modules), patch.object(NODE,'encode_guide',return_value=torch.zeros(1,24,1,2,2)) as encode:
|
||||
model,cond,preview=NODE.BFSMiniMaxH3DownscaledGuide().apply(self.model,positive,None,self.target(1),torch.zeros(1,128,128,3))
|
||||
self.assertEqual(tuple(preview.shape),(1,32,32,3))
|
||||
self.assertEqual(tuple(encode.call_args[0][1].shape),(1,32,32,3))
|
||||
self.assertNotIn('minimax_keyframes',positive[0][1])
|
||||
self.assertIs(cond[0][0],embedding)
|
||||
self.assertEqual(cond[0][1]['minimax_keyframes'][0][NODE.MARKER]['downscale_factor'],4)
|
||||
self.assertTrue(model.model_options[NODE.WRAPPER_MARKER])
|
||||
|
||||
def test_video_node_validates_73_frames_without_truncation(self):
|
||||
with patch.dict(sys.modules,self.modules), patch.object(NODE,'encode_guide',return_value=torch.zeros(1,24,22,2,2)):
|
||||
_,cond,preview=NODE.BFSMiniMaxH3DownscaledGuide().apply(self.model,[[None,{}]],None,self.target(22),torch.zeros(73,32,32,3))
|
||||
self.assertEqual(preview.shape[0],73)
|
||||
self.assertEqual(cond[0][1]['minimax_keyframes'][0]['latent'].shape[2],22)
|
||||
|
||||
def test_invalid_video_or_temporal_window_fails_before_encoding(self):
|
||||
for frames,index in [(3,0),(72,0),(73,1),(1,-74)]:
|
||||
with patch.dict(sys.modules,self.modules), patch.object(NODE,'encode_guide') as encode:
|
||||
with self.assertRaises(ValueError):
|
||||
NODE.BFSMiniMaxH3DownscaledGuide().apply(self.model,[[None,{}]],None,self.target(22),torch.zeros(frames,32,32,3),frame_idx=index)
|
||||
encode.assert_not_called()
|
||||
|
||||
def test_true_single_frame_and_video_targets(self):
|
||||
import types
|
||||
management=types.ModuleType('comfy.model_management')
|
||||
management.intermediate_device=lambda:torch.device('cpu')
|
||||
nested=types.ModuleType('comfy.nested_tensor')
|
||||
nested.NestedTensor=lambda tensors:SimpleNamespace(tensors=tensors,is_nested=True)
|
||||
self.comfy.model_management=management
|
||||
modules=dict(self.modules,**{'comfy.model_management':management,'comfy.nested_tensor':nested})
|
||||
with patch.dict(sys.modules,modules):
|
||||
for frames,vt,at in [(1,1,2),(5,2,8),(73,22,122),(124,37,207)]:
|
||||
result=NODE.BFSMiniMaxH3GuideTarget().create(128,128,frames)[0]['samples']
|
||||
self.assertEqual(tuple(result.tensors[0].shape),(1,24,vt,8,8))
|
||||
self.assertEqual(tuple(result.tensors[1].shape),(1,32,2,at))
|
||||
with self.assertRaises(ValueError): NODE.BFSMiniMaxH3GuideTarget().create(128,128,72)
|
||||
|
||||
def test_metadata_factor_and_version_validation(self):
|
||||
import tempfile, types
|
||||
from safetensors.torch import save_file
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
path=str(Path(temp)/'guide.safetensors')
|
||||
save_file({'x':torch.zeros(1)},path,metadata={
|
||||
'reference_downscale_factor':'4','guide_latent_only':'true',
|
||||
'minimax_h3_guide_spatial_version':NODE.SPATIAL_VERSION,
|
||||
'minimax_h3_guide_position_version':NODE.POSITION_VERSION})
|
||||
folder=types.ModuleType('folder_paths'); folder.get_full_path_or_raise=lambda *args:path
|
||||
with patch.dict(sys.modules,{'folder_paths':folder}):
|
||||
NODE.check_lora_metadata('guide.safetensors',4)
|
||||
with self.assertRaisesRegex(ValueError,'factor'): NODE.check_lora_metadata('guide.safetensors',2)
|
||||
save_file({'x':torch.zeros(1)},path,metadata={})
|
||||
with patch.dict(sys.modules,{'folder_paths':folder}):
|
||||
with self.assertRaisesRegex(ValueError,'spatial_version'): NODE.check_lora_metadata('guide.safetensors',4)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,71 @@
|
||||
import json
|
||||
import math
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
import torch
|
||||
from safetensors.torch import save_file
|
||||
from test_minimax_h3_downscaled_guide import NODE,CORE,FakeModel,NodeExecutionTest,guide
|
||||
|
||||
class PhaseModel(FakeModel):
|
||||
def __init__(self,options=None,objects=None):
|
||||
super().__init__(options);self.objects=objects or {}
|
||||
def clone(self):return PhaseModel(dict(self.model_options),dict(self.objects))
|
||||
def get_model_object(self,key):
|
||||
return self.objects.get(key,self.original)
|
||||
def add_object_patch(self,key,value):self.objects[key]=value
|
||||
@staticmethod
|
||||
def original(positions,device):
|
||||
inv=10000**(-torch.arange(16)/16)
|
||||
half=(positions.float().unsqueeze(-1)*inv).flatten(1)
|
||||
return torch.cat((half,half),-1).to(device)
|
||||
|
||||
class SourcePhaseTest(unittest.TestCase):
|
||||
def test_patch_is_scoped_composable_and_does_not_leak_to_unmarked_payload(self):
|
||||
original=PhaseModel();patched=NODE.patch_guide_model(original,True)
|
||||
kf=guide(4);kf[NODE.MARKER].update(source_phase=True,source_id=3,phase_scale=.7)
|
||||
native=CORE.PackedLayout(3,2,8,8,2,keyframes=[kf])
|
||||
def forward(x,t,**kwargs):
|
||||
layout=kwargs['minimax_payload']['layout']
|
||||
return patched.get_model_object('diffusion_model.rope_freqs')(layout.position_ids,'cpu'),layout
|
||||
fn=patched.model_options['model_function_wrapper']
|
||||
args={'input':0,'timestep':0,'c':{'minimax_payload':{'layout':native,'keyframes':[kf]}}}
|
||||
angles,layout=fn(forward,args)
|
||||
expected=NODE.add_source_phase(PhaseModel.original(layout.position_ids,'cpu'),layout.bfs_source_phase_values)
|
||||
torch.testing.assert_close(angles,expected)
|
||||
self.assertFalse(original.objects)
|
||||
again=NODE.patch_guide_model(patched,True)
|
||||
self.assertIs(again.objects['diffusion_model.rope_freqs'],patched.objects['diffusion_model.rope_freqs'])
|
||||
blank=CORE.PackedLayout(3,2,8,8,2)
|
||||
angles,_=fn(forward,{'input':0,'timestep':0,'c':{'minimax_payload':{'layout':blank}}})
|
||||
torch.testing.assert_close(angles,PhaseModel.original(blank.position_ids,'cpu'))
|
||||
|
||||
def test_default_and_experimental_metadata_are_checked(self):
|
||||
import types
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path=str(Path(directory)/'lora.safetensors')
|
||||
meta={'minimax_h3_guide_spatial_version':NODE.SPATIAL_VERSION,'minimax_h3_guide_position_version':NODE.POSITION_VERSION,
|
||||
'reference_downscale_factor':'4','guide_latent_only':'true'}
|
||||
opts=NODE.validate_options('overlap','sidecar',True,.7,2)
|
||||
meta['minimax_h3_reference_rope']=json.dumps(dict(opts,phase_version=NODE.PHASE_VERSION))
|
||||
save_file({'w':torch.zeros(1)},path,metadata=meta)
|
||||
folders=types.SimpleNamespace(get_full_path_or_raise=lambda *args:path)
|
||||
with patch.dict('sys.modules',{'folder_paths':folders}):
|
||||
NODE.check_lora_metadata('lora',4,opts)
|
||||
with self.assertRaisesRegex(ValueError,'RoPE options'):NODE.check_lora_metadata('lora',4)
|
||||
|
||||
def test_identity_node_uses_own_aspect_and_training_image_bucket(self):
|
||||
harness=NodeExecutionTest();harness.setUp()
|
||||
positive=[[torch.zeros(1,3,4),{}]]
|
||||
image=torch.zeros(1,192,64,3)
|
||||
with patch.dict('sys.modules',harness.modules),patch.object(NODE,'encode_guide',return_value=torch.zeros(1,24,1,12,4)) as encoder:
|
||||
model,conditioning,preview=NODE.BFSMiniMaxH3IdentityReference().apply(harness.model,positive,None,harness.target(2),image)
|
||||
self.assertEqual(tuple(preview.shape),(1,192,64,3))
|
||||
block=conditioning[0][1]['minimax_refs'][0]
|
||||
self.assertEqual(block[NODE.MARKER]['source_id'],2)
|
||||
self.assertEqual((block['latent_h'],block['latent_w']),(12,4))
|
||||
self.assertNotIn('minimax_refs',positive[0][1])
|
||||
self.assertEqual(encoder.call_count,1)
|
||||
|
||||
if __name__=='__main__':unittest.main()
|
||||
@@ -0,0 +1,141 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
import types
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
MODULE_PATH = ROOT / "reference_temporal_offset.py"
|
||||
SPEC = importlib.util.spec_from_file_location("reference_temporal_offset", MODULE_PATH)
|
||||
assert SPEC is not None and SPEC.loader is not None
|
||||
OFFSET = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(OFFSET)
|
||||
|
||||
|
||||
class ReferenceTemporalOffsetTest(unittest.TestCase):
|
||||
def test_default_matches_standard_frame_zero_placement(self):
|
||||
kind, options = OFFSET.reference_temporal_offset_input()
|
||||
self.assertEqual(kind, "INT")
|
||||
self.assertEqual(options["default"], 0)
|
||||
self.assertEqual(OFFSET.reference_temporal_pixel_offset(0, 8), 0)
|
||||
self.assertEqual(OFFSET.reference_temporal_pixel_offset(-1, 8), -8)
|
||||
|
||||
def test_shift_changes_only_temporal_axis_without_mutating_input(self):
|
||||
positions = torch.tensor(
|
||||
[[[0.0, 8.0], [10.0, 20.0], [30.0, 40.0]]],
|
||||
dtype=torch.float32,
|
||||
)
|
||||
shifted = OFFSET.shift_reference_temporal_positions(positions, -1, 8)
|
||||
|
||||
torch.testing.assert_close(positions[:, 0], torch.tensor([[0.0, 8.0]]))
|
||||
torch.testing.assert_close(shifted[:, 0], torch.tensor([[-8.0, 0.0]]))
|
||||
torch.testing.assert_close(shifted[:, 1:], positions[:, 1:])
|
||||
self.assertIsNot(shifted, positions)
|
||||
|
||||
def test_zero_is_exact_noop_for_legacy_specs(self):
|
||||
positions = torch.randn(1, 3, 4, 2)
|
||||
shifted = OFFSET.shift_reference_temporal_positions(positions, 0, 8)
|
||||
self.assertIs(shifted, positions)
|
||||
|
||||
def test_multiple_controls_exposes_only_identity_offset(self):
|
||||
source = (ROOT / "ltx_multiple_controls.py").read_text(encoding="utf-8")
|
||||
self.assertIn(
|
||||
'"identity_temporal_offset_latents": reference_temporal_offset_input()',
|
||||
source,
|
||||
)
|
||||
self.assertNotIn('"guide_temporal_offset_latents"', source)
|
||||
self.assertNotIn('"mask_temporal_offset_latents"', source)
|
||||
self.assertNotIn('"identity_mask_temporal_offset_latents"', source)
|
||||
|
||||
def test_edit_anything_offsets_reference_but_keeps_guide_at_zero(self):
|
||||
package_name = "bfsnodes_offset_test"
|
||||
package = types.ModuleType(package_name)
|
||||
package.__path__ = [str(ROOT)]
|
||||
sys.modules[package_name] = package
|
||||
|
||||
folder_paths = types.ModuleType("folder_paths")
|
||||
folder_paths.get_filename_list = lambda _kind: []
|
||||
sys.modules.setdefault("folder_paths", folder_paths)
|
||||
|
||||
comfy_extras = types.ModuleType("comfy_extras")
|
||||
comfy_extras.__path__ = []
|
||||
nodes_lt = types.ModuleType("comfy_extras.nodes_lt")
|
||||
frame_indices = []
|
||||
|
||||
class AddGuide:
|
||||
@staticmethod
|
||||
def append_keyframe(
|
||||
positive,
|
||||
negative,
|
||||
frame_idx,
|
||||
latent_image,
|
||||
noise_mask,
|
||||
_guiding_latent,
|
||||
_strength,
|
||||
_scale_factors,
|
||||
causal_fix,
|
||||
):
|
||||
self.assertTrue(causal_fix)
|
||||
frame_indices.append(frame_idx)
|
||||
return positive, negative, latent_image, noise_mask
|
||||
|
||||
nodes_lt.LTXVAddGuide = AddGuide
|
||||
nodes_lt._append_guide_attention_entry = (
|
||||
lambda positive, negative, *args, **kwargs: (
|
||||
positive,
|
||||
negative,
|
||||
)
|
||||
)
|
||||
comfy_extras.nodes_lt = nodes_lt
|
||||
sys.modules["comfy_extras"] = comfy_extras
|
||||
sys.modules["comfy_extras.nodes_lt"] = nodes_lt
|
||||
|
||||
module_path = ROOT / "ltxv_editanything.py"
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
f"{package_name}.ltxv_editanything",
|
||||
module_path,
|
||||
)
|
||||
assert spec is not None and spec.loader is not None
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = module
|
||||
spec.loader.exec_module(module)
|
||||
|
||||
target = torch.zeros(1, 128, 2, 1, 1)
|
||||
mask = torch.ones(1, 1, 2, 1, 1)
|
||||
reference = torch.zeros(1, 128, 1, 1, 1)
|
||||
guide = torch.zeros(1, 128, 2, 1, 1)
|
||||
module.LTXVEditAnythingLoopingSampler._build_chunk_conds(
|
||||
"positive",
|
||||
"negative",
|
||||
target,
|
||||
mask,
|
||||
reference,
|
||||
guide,
|
||||
(8, 32, 32),
|
||||
1.0,
|
||||
1.0,
|
||||
-1,
|
||||
True,
|
||||
)
|
||||
|
||||
self.assertEqual(frame_indices, [-8, 0])
|
||||
|
||||
def test_all_reference_injection_nodes_expose_the_control(self):
|
||||
expected_fields = {
|
||||
"ltx_identity_overlap.py": "reference_temporal_offset_latents",
|
||||
"ltx_identity_multiangle.py": "reference_temporal_offset_latents",
|
||||
"ltx_multiple_controls.py": "identity_temporal_offset_latents",
|
||||
"ltxv_editanything.py": "ref_temporal_offset_latents",
|
||||
}
|
||||
for filename, field in expected_fields.items():
|
||||
with self.subTest(filename=filename):
|
||||
source = (ROOT / filename).read_text(encoding="utf-8")
|
||||
self.assertIn(field, source)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,141 @@
|
||||
"""Reference-layout placement, on both LTX coordinate conventions.
|
||||
|
||||
The node runs against two shapes and two RoPE collapse rules, and the sidecar layout is the
|
||||
one that is sensitive to all of them:
|
||||
|
||||
[B, 3, N, 2] patch bounds -- current builds, LTX-2.3 and LTX-2.5 alike
|
||||
[B, 3, N] single corner -- legacy builds
|
||||
use_middle_indices_grid -- True (2.5) collapses a token to (start+end)/2, False (2.3
|
||||
checkpoints whose metadata omits the flag) reads the start
|
||||
|
||||
`_collapse` below mirrors comfy.ldm.lightricks.model.generate_freqs, so these tests compare
|
||||
the position the model actually sees, not the raw tensor.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import sys
|
||||
import types
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
_PKG = types.ModuleType("bfsnodes_under_test")
|
||||
_PKG.__path__ = [str(ROOT)]
|
||||
sys.modules.setdefault("bfsnodes_under_test", _PKG)
|
||||
OVERLAP = importlib.import_module("bfsnodes_under_test.ltx_identity_overlap")
|
||||
|
||||
|
||||
def _bounds(batch, tokens, t_span=576.0, height=384.0, width=704.0, seed=0):
|
||||
"""Patch bounds [B, 3, N, 2] with a plausible per-patch extent."""
|
||||
generator = torch.Generator().manual_seed(seed)
|
||||
low = torch.stack([
|
||||
torch.rand(tokens, generator=generator) * t_span,
|
||||
torch.rand(tokens, generator=generator) * height,
|
||||
torch.rand(tokens, generator=generator) * width,
|
||||
])
|
||||
high = low + torch.tensor([8.0, 32.0, 32.0]).unsqueeze(1)
|
||||
return torch.stack([low, high], dim=-1).unsqueeze(0).expand(batch, -1, -1, -1).contiguous()
|
||||
|
||||
|
||||
def _collapse(coords, use_middle):
|
||||
"""The one number per token the RoPE actually uses -- see generate_freqs."""
|
||||
if use_middle and coords.dim() == 4:
|
||||
return (coords[..., 0] + coords[..., 1]) / 2.0
|
||||
return coords[..., 0] if coords.dim() == 4 else coords
|
||||
|
||||
|
||||
def _axis_centre(coords, axis):
|
||||
return (coords[0, axis].min() + coords[0, axis].max()) / 2.0
|
||||
|
||||
|
||||
class TassLayoutTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
# A batch-2 target (CFG) against a batch-1 reference: layouts run before the reference
|
||||
# is expanded, which is what crashed the first sidecar run.
|
||||
self.target = _bounds(2, 512, seed=1)
|
||||
self.reference = _bounds(1, 64, t_span=8.0, height=128.0, width=128.0, seed=2)
|
||||
|
||||
def test_sidecar_places_reference_at_clip_centre_under_both_collapse_rules(self):
|
||||
for use_middle in (True, False):
|
||||
with self.subTest(use_middle=use_middle):
|
||||
shifted = OVERLAP._apply_tass_layout(
|
||||
self.reference, self.target, "sidecar", use_middle_indices_grid=use_middle
|
||||
)
|
||||
reference_seen = _collapse(shifted, use_middle)
|
||||
target_seen = _collapse(self.target, use_middle)
|
||||
# T: centred on the target's clip, not pinned to frame zero
|
||||
self.assertAlmostEqual(
|
||||
float(_axis_centre(reference_seen, 0)), float(_axis_centre(target_seen, 0)), delta=4.0
|
||||
)
|
||||
# H: centred on the target's height
|
||||
self.assertAlmostEqual(
|
||||
float(_axis_centre(reference_seen, 1)), float(_axis_centre(target_seen, 1)), delta=1e-3
|
||||
)
|
||||
# W: entirely beside the target, never overlapping it
|
||||
self.assertGreaterEqual(
|
||||
float(reference_seen[0, 2].min()), float(target_seen[0, 2].max()) - 1e-3
|
||||
)
|
||||
|
||||
def test_sidecar_survives_legacy_single_corner_coordinates(self):
|
||||
shifted = OVERLAP._apply_tass_layout(
|
||||
self.reference[..., 0].contiguous(),
|
||||
self.target[..., 0].contiguous(),
|
||||
"sidecar",
|
||||
use_middle_indices_grid=False,
|
||||
)
|
||||
self.assertEqual(shifted.shape, self.reference[..., 0].shape)
|
||||
target = self.target[..., 0]
|
||||
self.assertAlmostEqual(float(_axis_centre(shifted, 0)), float(_axis_centre(target, 0)), delta=4.0)
|
||||
self.assertAlmostEqual(float(_axis_centre(shifted, 1)), float(_axis_centre(target, 1)), delta=1e-3)
|
||||
self.assertGreaterEqual(float(shifted[0, 2].min()), float(target[0, 2].max()) - 1e-3)
|
||||
|
||||
def test_sidecar_margin_pushes_the_reference_further_right(self):
|
||||
flush = OVERLAP._apply_tass_layout(self.reference, self.target, "sidecar")
|
||||
spaced = OVERLAP._apply_tass_layout(
|
||||
self.reference, self.target, "sidecar", sidecar_margin_pixels=64.0
|
||||
)
|
||||
self.assertAlmostEqual(float(spaced[0, 2].min() - flush[0, 2].min()), 64.0, delta=1e-3)
|
||||
|
||||
def test_patch_bounds_are_translated_not_squeezed(self):
|
||||
span = (self.reference[..., 1] - self.reference[..., 0])
|
||||
for layout, kwargs in (("st_drc", {}), ("strata", {"strata_start": 600.0}), ("sidecar", {})):
|
||||
with self.subTest(layout=layout):
|
||||
shifted = OVERLAP._apply_tass_layout(self.reference, self.target, layout, **kwargs)
|
||||
moved = shifted[..., 1] - shifted[..., 0]
|
||||
# sidecar deliberately rewrites the temporal span; H/W must keep their extent
|
||||
axes = slice(1, 3) if layout == "sidecar" else slice(0, 3)
|
||||
# atol covers float32 rounding on ~600px coordinates, not a real drift
|
||||
torch.testing.assert_close(moved[:, axes], span[:, axes], atol=1e-3, rtol=0)
|
||||
|
||||
def test_integer_coordinates_keep_their_dtype(self):
|
||||
"""ComfyUI hands out int64 pixel coords -- float shifts must not blow up on them."""
|
||||
target = self.target.round().to(torch.int64)
|
||||
reference = self.reference.round().to(torch.int64)
|
||||
for layout, kwargs in (
|
||||
("overlap", {}), ("st_drc", {}), ("strata", {"strata_start": 600.0}), ("sidecar", {})
|
||||
):
|
||||
with self.subTest(layout=layout):
|
||||
shifted = OVERLAP._apply_tass_layout(reference, target, layout, **kwargs)
|
||||
self.assertEqual(shifted.dtype, torch.int64)
|
||||
float_result = OVERLAP._apply_tass_layout(
|
||||
reference.float(), target.float(), layout, **kwargs
|
||||
)
|
||||
# rounded, never truncated: at most half a pixel from the exact placement
|
||||
self.assertLessEqual(float((shifted.float() - float_result).abs().max()), 0.5)
|
||||
|
||||
def test_overlap_is_an_exact_noop(self):
|
||||
self.assertIs(
|
||||
OVERLAP._apply_tass_layout(self.reference, self.target, "overlap"), self.reference
|
||||
)
|
||||
|
||||
def test_unknown_layout_is_rejected(self):
|
||||
with self.assertRaises(ValueError):
|
||||
OVERLAP._apply_tass_layout(self.reference, self.target, "nope")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,171 @@
|
||||
// "❓ How to use" for the H3 duet nodes: a window with drawn examples of every setting.
|
||||
import { app } from "../../scripts/app.js";
|
||||
import { showGuide } from "./bfs_md_view.js";
|
||||
|
||||
const NODES = ["BFSH3Duet", "BFSShotH3Duet", "BFSH3SidePanel", "BFSH3DuetConditioning"];
|
||||
|
||||
// ---- small SVG diagrams -------------------------------------------------------------------
|
||||
const C = { video: "#3b6fd8", panel: "#d88a2b", grey: "#7a7a7a", ink: "#e8e8ee", dim: "#9a9aa8", bg: "#1b1b21" };
|
||||
|
||||
function canvas({ position = "left", size = 1.0, fit = "contain", gap = 0, w = 150, h = 96, label = "" }) {
|
||||
// video area keeps 16:9-ish; the strip sits on one side
|
||||
const horiz = position === "left" || position === "right";
|
||||
const vw = horiz ? w / (1 + size + gap * 0.08) : w, vh = horiz ? h : h / (1 + size + gap * 0.08);
|
||||
const sw = horiz ? vw * size : vw, sh = horiz ? vh : vh * size, g = gap * (horiz ? vw : vh) * 0.08;
|
||||
const vx = position === "left" ? sw + g : 0, vy = position === "top" ? sh + g : 0;
|
||||
const px = position === "right" ? vw + g : 0, py = position === "bottom" ? vh + g : 0;
|
||||
// panel content (the source clip, 16:9) placed in the strip
|
||||
const ar = vw / vh;
|
||||
let cw = sw, ch = sh;
|
||||
if (fit === "contain") { if (sw / sh > ar) cw = sh * ar; else ch = sw / ar; }
|
||||
const cx = px + (sw - cw) / 2, cy = py + (sh - ch) / 2;
|
||||
const W = horiz ? vw + sw + g : vw, H = horiz ? vh : vh + sh + g;
|
||||
const clip = fit === "cover" && Math.abs(sw / sh - ar) > 1e-3;
|
||||
// cover: the clip is scaled to fill the strip; what sticks out is cropped (dashed)
|
||||
let ow = sw, oh = sh;
|
||||
if (clip) { if (sw / sh > ar) oh = sw / ar; else ow = sh * ar; }
|
||||
const ox = px + (sw - ow) / 2, oy = py + (sh - oh) / 2;
|
||||
const pad = clip ? Math.max(0, (ow - sw) / 2, (oh - sh) / 2) + 2 : 2;
|
||||
return `<svg viewBox="${-pad} ${-pad} ${W + 2 * pad} ${H + 16 + 2 * pad}" width="${W + 2 * pad}" height="${H + 16 + 2 * pad}">
|
||||
<rect x="${px}" y="${py}" width="${sw}" height="${sh}" fill="${C.grey}" opacity=".45"/>
|
||||
${clip ? `<rect x="${ox}" y="${oy}" width="${ow}" height="${oh}" fill="none" stroke="${C.panel}" stroke-dasharray="3 2"/>
|
||||
<rect x="${px}" y="${py}" width="${sw}" height="${sh}" fill="${C.panel}"/>
|
||||
<text x="${ox + 2}" y="${oy + 9}" fill="${C.dim}" font-size="8">cropped</text>` :
|
||||
`<rect x="${cx}" y="${cy}" width="${cw}" height="${ch}" fill="${C.panel}"/>`}
|
||||
<text x="${px + sw / 2}" y="${py + sh / 2 + 4}" fill="#fff" font-size="10" text-anchor="middle">panel</text>
|
||||
${g ? `<rect x="${horiz ? (position === "left" ? sw : vw) : 0}" y="${horiz ? 0 : (position === "top" ? sh : vh)}" width="${horiz ? g : vw}" height="${horiz ? vh : g}" fill="${C.grey}"/>` : ""}
|
||||
<rect x="${vx}" y="${vy}" width="${vw}" height="${vh}" fill="${C.video}"/>
|
||||
<text x="${vx + vw / 2}" y="${vy + vh / 2 + 4}" fill="#fff" font-size="10" text-anchor="middle">video (output)</text>
|
||||
<text x="${W / 2}" y="${H + 13}" fill="${C.dim}" font-size="10" text-anchor="middle">${label}</text></svg>`;
|
||||
}
|
||||
|
||||
function rope(mode, gap) {
|
||||
const n = 7, step = 18, y = 22;
|
||||
let s = "", x = 6;
|
||||
for (let i = 0; i < n; i++) { s += `<circle cx="${x}" cy="${y}" r="5" fill="${C.panel}"/>`; x += step; }
|
||||
if (mode === "shifted") x += gap * step * 0.5;
|
||||
const gx0 = x - step;
|
||||
for (let i = 0; i < n; i++) { s += `<circle cx="${x}" cy="${y}" r="5" fill="${C.video}"/>`; x += step; }
|
||||
const label = mode === "canvas" ? "canvas: one wide grid, panel and video side by side"
|
||||
: `shifted: video keeps its own positions, panel further out${gap ? ` (gap ${gap})` : ""}`;
|
||||
const W = Math.max(x + 4, 340);
|
||||
return `<svg viewBox="0 0 ${W} 46" width="${W}" height="46">${s}
|
||||
${mode === "shifted" && gap ? `<line x1="${gx0 + 8}" y1="${y}" x2="${gx0 + gap * step * 0.5 + step - 8}" y2="${y}" stroke="${C.dim}" stroke-dasharray="3 3"/>` : ""}
|
||||
<text x="4" y="42" fill="${C.dim}" font-size="10">${label}</text></svg>`;
|
||||
}
|
||||
|
||||
function hold(kind) {
|
||||
const n = 7; let s = "";
|
||||
for (let i = 0; i < n; i++) {
|
||||
const held = kind === "all frames" || i === 0;
|
||||
s += `<rect x="${4 + i * 20}" y="6" width="16" height="16" fill="${held ? C.panel : C.grey}" opacity="${held ? 1 : .45}"/>`;
|
||||
}
|
||||
return `<svg viewBox="0 0 260 40" width="260" height="40">${s}<text x="4" y="36" fill="${C.dim}" font-size="10">${kind}: panel held on the orange frames</text></svg>`;
|
||||
}
|
||||
|
||||
const row = (...cells) => `<div class="bdh-row">${cells.map(c => `<div class="bdh-cell">${c}</div>`).join("")}</div>`;
|
||||
|
||||
// ---- content ------------------------------------------------------------------------------
|
||||
function html(type) {
|
||||
const shot = type === "BFSShotH3Duet";
|
||||
return `
|
||||
<h2>H3 Duet: how to use</h2>
|
||||
<p>The <b>panel</b> (a clip or a picture) is pinned next to the video in one wide canvas. MiniMax H3 keeps the panel
|
||||
exactly and generates the video beside it, <b>in sync</b>: the new video copies the panel's motion, camera, timing and
|
||||
cuts. The panel is cut off before decoding, so only the video comes out. No LoRA is needed.</p>
|
||||
${row(canvas({ label: "the canvas the model sees" }), `<svg width="40" height="96"><text x="8" y="52" fill="${C.ink}" font-size="22">→</text></svg>`,
|
||||
canvas({ w: 75, size: 0.0001, label: "what you get" }).replace(">panel<", "><"))}
|
||||
|
||||
<h3>What goes in</h3>
|
||||
<ul>
|
||||
<li><b>panel</b>${shot ? " (the shot's own clip, set by the mode)" : ""}: usually the <b>source clip</b> you want to restyle or recast. A single picture also works but copies less.</li>
|
||||
<li><b>ref_images</b> → <code><Picture 1></code>, <code><Picture 2></code>…: the new person, outfit or place.</li>
|
||||
<li><b>guide</b> (optional): an aligned latent guide, for LoRAs trained on it. Usually leave it empty with a panel.</li>
|
||||
</ul>
|
||||
|
||||
<h3>Prompt, task and instruction</h3>
|
||||
<p><b>prompt</b> always wins. Leave it empty and the node writes a <b>draft</b> from <b>task</b> + <b>instruction</b>
|
||||
(see the <code>prompt</code> output). The draft cannot see the video, so a prompt written for the clip is better.</p>
|
||||
<table>
|
||||
<tr><th>task</th><th>instruction: a few words of what is SEEN</th></tr>
|
||||
<tr><td>character swap</td><td><i>(empty, or the look)</i> + the person's pictures as references</td></tr>
|
||||
<tr><td>style</td><td><code>a 1990s anime cel style</code></td></tr>
|
||||
<tr><td>setting</td><td><code>a sunny beach at sunset, waves behind her</code></td></tr>
|
||||
<tr><td>appearance</td><td><code>an elderly woman with short grey hair</code></td></tr>
|
||||
<tr><td>lighting / weather</td><td><code>night, lit by pink and blue neon signs</code></td></tr>
|
||||
</table>
|
||||
<p>Rules that matter (TSC, newer strategy): <b>H3 has no idea what is pinned and what is noised</b>, it reads one
|
||||
description of one video. Describe the finished split screen as plain footage: never write "kept footage", "generated",
|
||||
"preserved", "in sync with", "seamlessly". The pinned clip gets <b>no tag and no retention line</b>: name its region by where
|
||||
it is plus what is seen there, in one clause ("the top quarter shows a man dancing in a small bedroom";
|
||||
<code>{panel_region}</code> / <code>{video_region}</code> give the region names, <code>{layout}</code> the whole sentence).
|
||||
Write the other region as its own scene in a real place. <b>Spell out the motion</b> beat by beat, with times for the big
|
||||
moments on the video's clock. The soundscape is the sounds of the new scene. Cuts: "[Shot 2] At 00:03.708, both halves cut
|
||||
together to…". No negations ("no hat" draws a hat). Connect a VLM to <code>vlm</code> to write the prompt or fill a
|
||||
template's <code>{...}</code> fields. The full example is in the prompt tooltip.</p>
|
||||
|
||||
<h3>Layout</h3>
|
||||
<p><b>position</b>: the side of the panel.</p>
|
||||
${row(canvas({ position: "left", label: "left" }), canvas({ position: "right", label: "right" }),
|
||||
canvas({ position: "top", w: 96, h: 120, label: "top" }), canvas({ position: "bottom", w: 96, h: 120, label: "bottom" }))}
|
||||
<p><b>size</b>: the panel against the video (1.0 = two equal halves). Smaller is cheaper; keep faces readable.</p>
|
||||
${row(canvas({ size: 1.0, label: "size 1.0" }), canvas({ size: 0.5, label: "size 0.5" }), canvas({ size: 0.33, label: "size 0.33" }))}
|
||||
<p><b>fit</b> (advanced): <b>contain</b> keeps the whole clip smaller with grey around (nothing lost, default);
|
||||
<b>cover</b> fills the panel and crops; stretch distorts. <b>gap</b> (advanced): a grey separator, in 32 px steps.</p>
|
||||
${row(canvas({ size: 0.5, fit: "contain", label: "contain" }), canvas({ size: 0.5, fit: "cover", label: "cover" }), canvas({ size: 1.0, gap: 2, label: "gap 2" }))}
|
||||
|
||||
<h3>How strongly the panel is pinned</h3>
|
||||
<p><b>panel_noise</b>: <b>0</b> copies the panel exactly (swaps, light changes). <b>0.1–0.2</b> gives the model room for big
|
||||
changes (a style, a creature, a very different body).</p>
|
||||
<p><b>hold</b> (advanced): pin the panel for the whole clip, or only at the start (the rest of the strip is free and cropped).</p>
|
||||
${row(hold("all frames"), hold("first latent frame"))}
|
||||
|
||||
<h3>RoPE (advanced)</h3>
|
||||
<p><b>rope_mode</b>: <b>canvas</b> = panel and video share one wide grid (the original duet). <b>shifted</b> = the video keeps
|
||||
exactly the positions of a render without the panel, and the panel sits past its edge; <b>rope_gap</b> moves it further
|
||||
out. Same quality in tests; keep the gap small at low resolution (a gap close to the video's width makes the model draw
|
||||
its own split screen).</p>
|
||||
${row(rope("canvas", 0), rope("shifted", 0), rope("shifted", 3))}
|
||||
|
||||
<h3>Defaults that work</h3>
|
||||
<p>euler · beta · 20 steps · CFG 1 · size 1.0 · contain · panel_noise 0 · ${shot ? "mode duet" : "no guide"}.
|
||||
${shot ? "Planner → this node → BFS Shot Join: every shot is pinned and generated in turn; the join cuts the panel off." :
|
||||
"Outputs: <code>images</code> + <code>audio</code> (silent when there is no audio VAE) for Create Video; <code>canvas</code> to check the sync; <code>prompt</code> = what was sent."}</p>`;
|
||||
}
|
||||
|
||||
function styles() {
|
||||
if (document.getElementById("bfs-duet-help-css")) return;
|
||||
const st = document.createElement("style"); st.id = "bfs-duet-help-css";
|
||||
st.textContent = `
|
||||
.bdh-back{position:fixed;inset:0;background:rgba(0,0,0,.7);z-index:10000;display:flex;align-items:center;justify-content:center}
|
||||
.bdh{background:${C.bg};color:${C.ink};border:1px solid #3a3a45;border-radius:12px;width:min(860px,94vw);max-height:92vh;overflow:auto;
|
||||
padding:18px 22px;font:13px/1.55 system-ui,sans-serif}
|
||||
.bdh h2{margin:0 0 8px;font-size:18px}.bdh h3{margin:16px 0 6px;font-size:14px;color:#ffd27a}
|
||||
.bdh code{background:#2a2a33;padding:1px 4px;border-radius:4px}
|
||||
.bdh table{border-collapse:collapse;margin:6px 0}.bdh td,.bdh th{border:1px solid #33333d;padding:3px 8px;text-align:left}
|
||||
.bdh-row{display:flex;gap:14px;flex-wrap:wrap;align-items:flex-end;margin:6px 0}
|
||||
.bdh-cell{background:#141418;border-radius:8px;padding:6px}
|
||||
.bdh-close{float:right;cursor:pointer;border:1px solid #444;background:#26262e;color:#ddd;border-radius:6px;padding:2px 10px}`;
|
||||
document.head.appendChild(st);
|
||||
}
|
||||
|
||||
function open(type) {
|
||||
styles();
|
||||
const back = document.createElement("div"); back.className = "bdh-back";
|
||||
const guide = new URL("./docs/BFSH3Duet.md", import.meta.url).href;
|
||||
back.innerHTML = `<div class="bdh"><button class="bdh-close">close</button>${html(type)}
|
||||
<p style="margin-top:14px"><a href="#" class="bdh-guide" style="color:#8fb3ff">Full guide (credits, all settings, a complete example prompt)</a></p></div>`;
|
||||
back.addEventListener("pointerdown", e => { if (e.target === back) back.remove(); });
|
||||
back.querySelector(".bdh-close").addEventListener("click", () => back.remove());
|
||||
back.querySelector(".bdh-guide").addEventListener("click", e => { e.preventDefault(); back.remove(); showGuide(guide); });
|
||||
document.body.appendChild(back);
|
||||
}
|
||||
|
||||
app.registerExtension({
|
||||
name: "BFSNodes.DuetHelp",
|
||||
async nodeCreated(node) {
|
||||
if (!NODES.includes(node.comfyClass)) return;
|
||||
const w = node.addWidget("button", "❓ How to use", null, () => open(node.comfyClass));
|
||||
w.serialize = false;
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,470 @@
|
||||
/**
|
||||
* BFS LoRA Surgery: a panel for reading a LoRA's layers and blocks and editing them live.
|
||||
*
|
||||
* The node keeps its state in a hidden `rules` STRING widget (a JSON list), so a workflow
|
||||
* saves and reloads exactly what you set up. Everything visual lives in a Vue app mounted
|
||||
* into a DOM widget.
|
||||
*/
|
||||
import { app } from "../../scripts/app.js";
|
||||
import { api } from "../../scripts/api.js";
|
||||
import { createApp, ref, computed, watch, h } from "./vendor/vue.esm-browser.prod.mjs";
|
||||
|
||||
// Starting points, not answers: which range works is per-LoRA, so ablate before trusting one.
|
||||
const PRESETS = [
|
||||
{ label: "MLP 50%", rules: [{ enabled: true, match: { type: "*mlp*" }, scale: 0.5 }] },
|
||||
{ label: "MLP 35%", rules: [{ enabled: true, match: { type: "*mlp*" }, scale: 0.35 }] },
|
||||
{ label: "Attention only", rules: [{ enabled: true, match: { type: "*mlp*" }, scale: 0 }] },
|
||||
{ label: "MLP-in off, late half", rules: [{ enabled: true, match: { type: "*gate_up*" }, blocksHalf: true, scale: 0 }] },
|
||||
];
|
||||
|
||||
function styles() {
|
||||
if (document.getElementById("bfs-lora-surgery-css")) return;
|
||||
const el = document.createElement("style");
|
||||
el.id = "bfs-lora-surgery-css";
|
||||
el.textContent = `
|
||||
.bfsls{font:12px/1.45 var(--font-family,system-ui,sans-serif);color:var(--descrip-text,#c9c9cf);
|
||||
background:var(--comfy-input-bg,#1b1b1f);border-radius:8px;padding:8px;height:100%;overflow:auto;box-sizing:border-box}
|
||||
.bfsls h4{margin:0 0 6px;font-size:12px;letter-spacing:.02em;color:var(--input-text,#e6e6ea)}
|
||||
.bfsls .muted{color:#8b8b93}
|
||||
.bfsls .row{display:flex;gap:6px;align-items:center;flex-wrap:wrap}
|
||||
.bfsls .fam{border:1px solid #33333c;border-radius:6px;margin-bottom:6px;overflow:hidden}
|
||||
.bfsls .fam>summary{cursor:pointer;padding:6px 8px;background:#232329;list-style:none;display:flex;
|
||||
justify-content:space-between;align-items:center;user-select:none}
|
||||
.bfsls .fam>summary::-webkit-details-marker{display:none}
|
||||
.bfsls .type{display:grid;grid-template-columns:1fr 78px 54px 30px;gap:6px;align-items:center;
|
||||
padding:4px 8px;border-top:1px solid #2a2a31}
|
||||
.bfsls .type:hover{background:#26262c}
|
||||
.bfsls .name{font-family:ui-monospace,monospace;font-size:11px;color:#dcdce2;overflow:hidden;text-overflow:ellipsis}
|
||||
.bfsls .bar{height:3px;border-radius:2px;background:#3a86ff;margin-top:3px;opacity:.85}
|
||||
.bfsls input[type=range]{width:100%;accent-color:#3a86ff}
|
||||
.bfsls input[type=text],.bfsls input[type=number]{background:#131317;border:1px solid #34343d;color:#e6e6ea;
|
||||
border-radius:4px;padding:3px 6px;font-size:11px;width:100%;box-sizing:border-box;font-family:ui-monospace,monospace}
|
||||
.bfsls button{background:#2b2b33;border:1px solid #3a3a45;color:#dcdce2;border-radius:5px;padding:3px 8px;
|
||||
font-size:11px;cursor:pointer}
|
||||
.bfsls button:hover{background:#35353f}
|
||||
.bfsls button.pri{background:#2b5fd9;border-color:#3a6ee8;color:#fff}
|
||||
.bfsls .rule{display:grid;grid-template-columns:18px 1fr 64px 18px 18px 20px;gap:5px;align-items:center;
|
||||
padding:4px 6px;border:1px solid #33333c;border-radius:5px;margin-bottom:4px;background:#1f1f25}
|
||||
.bfsls .rx{font-family:ui-monospace,monospace;font-size:10px;color:#9fd3a0;word-break:break-all}
|
||||
.bfsls .blocks{display:flex;flex-wrap:wrap;gap:2px;margin:4px 0}
|
||||
.bfsls .blk{width:19px;height:19px;user-select:none;touch-action:none;border-radius:3px;background:#2a2a32;border:1px solid #3a3a45;
|
||||
font-size:9px;display:flex;align-items:center;justify-content:center;cursor:pointer;color:#a8a8b2}
|
||||
.bfsls .blk.sel{background:#2b5fd9;border-color:#4a7ef0;color:#fff}
|
||||
.bfsls .err{color:#ff8a8a}
|
||||
.bfsls .mod{display:grid;grid-template-columns:1fr 62px 34px 26px;gap:6px;align-items:center;
|
||||
padding:3px 6px;border-bottom:1px solid #26262c;font-size:11px}
|
||||
.bfsls .mod:hover{background:#26262c}
|
||||
.bfsls .modlist{max-height:260px;overflow:auto;border:1px solid #33333c;border-radius:6px}
|
||||
.bfsls .tab{padding:3px 10px;border-radius:5px 5px 0 0;border:1px solid #33333c;border-bottom:none;
|
||||
background:#1f1f25;cursor:pointer;font-size:11px}
|
||||
.bfsls .tab.on{background:#2b2b33;color:#fff}
|
||||
.bfsls .sec{border-top:1px solid #2c2c34;margin-top:8px;padding-top:6px}
|
||||
.bfsls details.help{border:1px solid #33333c;border-radius:6px;background:#1c1c22;margin-bottom:8px}
|
||||
.bfsls details.help>summary{cursor:pointer;padding:5px 8px;color:#9fb8ff;list-style:none;user-select:none}
|
||||
.bfsls details.help>summary::-webkit-details-marker{display:none}
|
||||
.bfsls .helpbody{padding:2px 10px 8px;color:#b4b4bd;font-size:11px;line-height:1.55}
|
||||
.bfsls .helpbody code{background:#262630;padding:1px 4px;border-radius:3px;font-size:10px}
|
||||
.bfsls .helpbody b{color:#dcdce2}
|
||||
.bfsls .helpbody ol{margin:4px 0 6px 16px;padding:0}
|
||||
.bfsls .helpbody li{margin:2px 0}
|
||||
`;
|
||||
document.head.appendChild(el);
|
||||
}
|
||||
|
||||
function Panel(props, expose) {
|
||||
const struct = ref(null);
|
||||
const error = ref("");
|
||||
const loading = ref(false);
|
||||
const rules = ref([]);
|
||||
const selFamily = ref("");
|
||||
const selType = ref("*");
|
||||
const selBlocks = ref(new Set());
|
||||
const scale = ref(0);
|
||||
const dragging = ref(false);
|
||||
const view = ref("grouped"); // "grouped" | "all"
|
||||
const filter = ref("");
|
||||
const saveName = ref("");
|
||||
const saving = ref(false);
|
||||
const saved = ref("");
|
||||
let dragAdds = true;
|
||||
|
||||
// Stored shape is {rules, ui}. A bare array is still accepted, both because older
|
||||
// workflows carry one and because the Python side takes either.
|
||||
function readRules() {
|
||||
let parsed;
|
||||
try { parsed = JSON.parse(props.getRules() || "[]"); } catch { parsed = []; }
|
||||
if (Array.isArray(parsed)) { rules.value = parsed; return; }
|
||||
rules.value = Array.isArray(parsed?.rules) ? parsed.rules : [];
|
||||
const ui = parsed?.ui;
|
||||
if (ui) {
|
||||
if (ui.family) selFamily.value = ui.family;
|
||||
if (typeof ui.type === "string") selType.value = ui.type;
|
||||
if (typeof ui.scale === "number") scale.value = ui.scale;
|
||||
if (typeof ui.blocks === "string" && ui.blocks) setBlocksFromText(ui.blocks);
|
||||
}
|
||||
}
|
||||
function writeRules() {
|
||||
props.setRules(JSON.stringify({
|
||||
rules: rules.value,
|
||||
ui: { family: selFamily.value, type: selType.value, blocks: blocksSpec(), scale: Number(scale.value) },
|
||||
}));
|
||||
}
|
||||
|
||||
async function load() {
|
||||
const name = props.getLora();
|
||||
if (!name) return;
|
||||
loading.value = true; error.value = "";
|
||||
try {
|
||||
const r = await api.fetchApi(`/bfs/lora/structure?name=${encodeURIComponent(name)}`);
|
||||
const data = await r.json();
|
||||
if (data.error) { error.value = data.error; struct.value = null; }
|
||||
else {
|
||||
struct.value = data;
|
||||
const known = (data.families || []).some(f => f.name === selFamily.value);
|
||||
if (!known) selFamily.value = data.families?.[0]?.name || "";
|
||||
}
|
||||
} catch (e) { error.value = String(e); }
|
||||
loading.value = false;
|
||||
}
|
||||
|
||||
const family = computed(() => (struct.value?.families || []).find(f => f.name === selFamily.value)
|
||||
|| struct.value?.families?.[0]);
|
||||
const maxNorm = computed(() => Math.max(0.0001, ...((family.value?.types || []).map(t => t.norm || 0))));
|
||||
|
||||
function blocksSpec() {
|
||||
const b = [...selBlocks.value].sort((x, y) => x - y);
|
||||
if (!b.length) return "";
|
||||
const out = []; let s = b[0], p = b[0];
|
||||
for (let i = 1; i <= b.length; i++) {
|
||||
if (b[i] === p + 1) { p = b[i]; continue; }
|
||||
out.push(s === p ? `${s}` : `${s}-${p}`);
|
||||
s = p = b[i];
|
||||
}
|
||||
return out.join(",");
|
||||
}
|
||||
// Drag across the ruler to pick a range. Click-to-toggle still works, since a click is
|
||||
// just a drag of one cell. Shift is deliberately not used: on the ComfyUI canvas it starts
|
||||
// a text selection instead.
|
||||
function paint(i) {
|
||||
if (dragAdds) selBlocks.value.add(i); else selBlocks.value.delete(i);
|
||||
selBlocks.value = new Set(selBlocks.value);
|
||||
}
|
||||
function dragStart(i, ev) {
|
||||
ev.preventDefault(); ev.stopPropagation();
|
||||
dragging.value = true;
|
||||
dragAdds = !selBlocks.value.has(i);
|
||||
paint(i);
|
||||
const stop = () => { dragging.value = false; window.removeEventListener("pointerup", stop); };
|
||||
window.addEventListener("pointerup", stop);
|
||||
}
|
||||
function dragOver(i, ev) {
|
||||
if (!dragging.value) return;
|
||||
ev.preventDefault();
|
||||
paint(i);
|
||||
}
|
||||
function setBlocksFromText(text) {
|
||||
const next = new Set();
|
||||
for (const part of String(text).split(",")) {
|
||||
const t = part.trim();
|
||||
if (!t) continue;
|
||||
if (t.includes("-")) {
|
||||
const [a, b] = t.split("-").map(Number);
|
||||
if (Number.isFinite(a) && Number.isFinite(b)) for (let k = Math.min(a, b); k <= Math.max(a, b); k++) next.add(k);
|
||||
} else if (Number.isFinite(Number(t))) next.add(Number(t));
|
||||
}
|
||||
selBlocks.value = next;
|
||||
}
|
||||
function selectAll() { selBlocks.value = new Set(family.value?.blocks || []); }
|
||||
function selectHalf(second) {
|
||||
const b = family.value?.blocks || [];
|
||||
const mid = Math.floor(b.length / 2);
|
||||
selBlocks.value = new Set(second ? b.slice(mid) : b.slice(0, mid));
|
||||
}
|
||||
function invert() {
|
||||
const all = family.value?.blocks || [];
|
||||
selBlocks.value = new Set(all.filter(i => !selBlocks.value.has(i)));
|
||||
}
|
||||
|
||||
function addRule(asRegex) {
|
||||
const fam = family.value?.name || "";
|
||||
const spec = blocksSpec();
|
||||
const match = asRegex
|
||||
? { regex: buildRegex(fam, selType.value, spec) }
|
||||
: { family: fam, type: selType.value, blocks: spec };
|
||||
rules.value.push({ enabled: true, match, scale: Number(scale.value) });
|
||||
writeRules();
|
||||
}
|
||||
function buildRegex(fam, type, spec) {
|
||||
const esc = s => s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
let blk = "\\d+";
|
||||
if (spec) {
|
||||
const idx = [];
|
||||
for (const part of spec.split(",")) {
|
||||
if (part.includes("-")) { const [a, b] = part.split("-").map(Number); for (let k = a; k <= b; k++) idx.push(k); }
|
||||
else if (part.trim() !== "") idx.push(Number(part));
|
||||
}
|
||||
if (idx.length) blk = "(" + [...new Set(idx)].sort((a, b) => a - b).join("|") + ")";
|
||||
}
|
||||
let mod = ".*";
|
||||
if (type && type !== "*") mod = type.endsWith("*") ? esc(type.slice(0, -1)) + ".*" : esc(type);
|
||||
return `^.*${fam ? esc(fam) : "[\\w.]+"}\\.${blk}\\.${mod}$`;
|
||||
}
|
||||
|
||||
readRules();
|
||||
load();
|
||||
watch(() => props.getLora(), load);
|
||||
// ComfyUI restores widget values after the node is created, so the panel has to be told
|
||||
// to re-read once that happened. Without this, opening a saved workflow shows an empty
|
||||
// panel and the first edit would overwrite the stored recipe.
|
||||
if (expose) {
|
||||
expose.reload = () => { readRules(); load(); };
|
||||
}
|
||||
|
||||
|
||||
const help = () => h("details", { class: "help" }, [
|
||||
h("summary", {}, "What is this? (read once)"),
|
||||
h("div", { class: "helpbody" }, [
|
||||
h("p", {}, [
|
||||
"A defect a LoRA learned (soft skin, identity drifting when the subject is far away, ",
|
||||
"a pose it will not copy) usually lives in ", h("b", {}, "one module family and a range of blocks"),
|
||||
", not in the whole adapter. This node lets you scale or drop that group and generate again, ",
|
||||
"so you can find where it lives without retraining. Nothing is written to disk.",
|
||||
]),
|
||||
h("p", {}, [h("b", {}, "How to use it")]),
|
||||
h("ol", {}, [
|
||||
h("li", {}, ["Pick a LoRA. Layers below shows what that file contains, grouped by type, or switch to ", h("code", {}, "every layer"), " to see and filter each module individually."]),
|
||||
h("li", {}, ["The bar next to each type is ", h("code", {}, "||dW||"),
|
||||
", how large that group's update is. Big bars are where training invested."]),
|
||||
h("li", {}, ["Click ", h("code", {}, "->"), " on a type to select it, then drag across the block ruler to pick a range."]),
|
||||
h("li", {}, ["Set a scale and press ", h("code", {}, "add rule"), ". Scale 0 drops the group; 1 leaves it as trained."]),
|
||||
h("li", {}, "Queue the prompt and compare against the unmodified LoRA at the same seed."),
|
||||
]),
|
||||
h("p", {}, [h("b", {}, "Names depend on the base model. "),
|
||||
"This panel lists whatever your LoRA actually contains, so what you see here is not a fixed ",
|
||||
"vocabulary. Qwen-Image-2.1 shows ", h("code", {}, "transformer_blocks"), " with ",
|
||||
h("code", {}, "img_mlp.gate_up"), " and ", h("code", {}, "img_mlp.out"),
|
||||
". Krea 2 shows ", h("code", {}, "blocks"), " with ", h("code", {}, "mlp.gate"), "/",
|
||||
h("code", {}, "mlp.up"), "/", h("code", {}, "mlp.down"), ", plus separate ",
|
||||
h("code", {}, "txtfusion"), " families. MiniMax-H3 shows ", h("code", {}, "mlp.fc1"), "/",
|
||||
h("code", {}, "mlp.fc2"), " and a ", h("code", {}, "token_refiner"),
|
||||
" family. Block counts differ too: 32, 28 and 50 in those three. Read the list below ",
|
||||
"rather than assuming any name."]),
|
||||
h("p", {}, [h("b", {}, "Where to look first. "),
|
||||
"Across the models tested so far, the group that carries blur and lost detail is the ",
|
||||
h("b", {}, "MLP input projection"), ", whatever your model calls it (",
|
||||
h("code", {}, "gate_up"), ", ", h("code", {}, "mlp.gate"), "/", h("code", {}, "mlp.up"), ", ",
|
||||
h("code", {}, "fc1"), ", ", h("code", {}, "w1"), "/", h("code", {}, "w3"),
|
||||
"). The MLP output projection (", h("code", {}, "out"), ", ", h("code", {}, "down"), ", ",
|
||||
h("code", {}, "fc2"),
|
||||
") rarely matters, and attention is usually innocent. That is a starting point, not a rule: ",
|
||||
"which group and which block range are right differs per LoRA, so ablate and measure."]),
|
||||
h("p", {}, [h("b", {}, "Keeping the result. "),
|
||||
"Once a recipe is settled, press ", h("code", {}, "save as file"),
|
||||
" to write it into ", h("code", {}, "models/loras/surgery"),
|
||||
" with the recipe in its name and metadata. In a workflow you can instead wire this node's ",
|
||||
h("code", {}, "rules"), " output into ", h("code", {}, "BFS LoRA Surgery (save)"),
|
||||
", so the file is produced as part of the run."]),
|
||||
h("p", {}, [h("b", {}, "Order matters. "),
|
||||
"Later rules override earlier ones where they overlap, so put boosts first and drops last, ",
|
||||
"or a boost will undo a drop. Use the arrows to reorder."]),
|
||||
h("p", {}, [h("b", {}, "One warning. "),
|
||||
"A variant can win every metric and still have destroyed what you trained. Dropping the whole MLP ",
|
||||
"path gave the sharpest skin in one real case and made the LoRA stop copying expression from the ",
|
||||
"source image. Always check a hard case (a strong expression, an unusual angle) with your eyes."]),
|
||||
h("p", {}, [h("b", {}, "Pruning is not the same as training without those layers. "),
|
||||
"Excluding them during training may simply fail to converge: the layer is where the defect lodges, ",
|
||||
"not where it comes from, which is usually the dataset."]),
|
||||
]),
|
||||
]);
|
||||
|
||||
return () => h("div", { class: "bfsls" }, [
|
||||
h("div", { class: "row", style: "justify-content:space-between;margin-bottom:6px" }, [
|
||||
h("h4", {}, struct.value ? `${struct.value.file}, ${struct.value.total_modules} modules`
|
||||
: (loading.value ? "reading..." : "no LoRA loaded")),
|
||||
h("button", { onClick: load }, "reload"),
|
||||
]),
|
||||
error.value ? h("div", { class: "err" }, error.value) : null,
|
||||
help(),
|
||||
|
||||
h("div", { class: "sec" }, [h("h4", {}, "Selection")]),
|
||||
h("div", { class: "row" }, [
|
||||
h("span", { class: "muted" }, "type"),
|
||||
h("input", { type: "text", value: selType.value, style: "flex:1",
|
||||
onInput: e => { selType.value = e.target.value; } }),
|
||||
]),
|
||||
h("div", { class: "blocks", onPointerleave: () => { dragging.value = false; } },
|
||||
(family.value?.blocks || []).map(i =>
|
||||
h("div", { class: "blk" + (selBlocks.value.has(i) ? " sel" : ""),
|
||||
title: "click, or drag across to select a range",
|
||||
onPointerdown: e => dragStart(i, e),
|
||||
onPointerenter: e => dragOver(i, e) }, String(i)))),
|
||||
h("div", { class: "row", style: "margin-bottom:4px" }, [
|
||||
h("span", { class: "muted" }, "blocks"),
|
||||
h("input", { type: "text", value: blocksSpec(), placeholder: "all blocks", style: "flex:1",
|
||||
title: "type a selection, e.g. 8-15 or 0,4,7 or 8-15,24-31",
|
||||
onChange: e => setBlocksFromText(e.target.value) }),
|
||||
]),
|
||||
h("div", { class: "row", style: "margin-bottom:4px" }, [
|
||||
h("button", { onClick: selectAll }, "all"),
|
||||
h("button", { onClick: () => selectHalf(false) }, "first half"),
|
||||
h("button", { onClick: () => selectHalf(true) }, "last half"),
|
||||
h("button", { onClick: invert }, "invert"),
|
||||
h("button", { onClick: () => { selBlocks.value = new Set(); } }, "none"),
|
||||
]),
|
||||
h("div", { class: "row" }, [
|
||||
h("span", { class: "muted" }, "scale"),
|
||||
h("input", { type: "range", min: 0, max: 2, step: 0.05, value: scale.value, style: "flex:1",
|
||||
onInput: e => { scale.value = e.target.value; } }),
|
||||
h("span", { style: "width:34px;text-align:right" }, Number(scale.value).toFixed(2)),
|
||||
]),
|
||||
h("div", { class: "row", style: "margin:6px 0" }, [
|
||||
h("button", { class: "pri", onClick: () => addRule(false) }, "add rule"),
|
||||
h("button", { onClick: () => addRule(true), title: "same selection, written as a regex" }, "add as regex"),
|
||||
h("span", { class: "muted" }, blocksSpec() ? `blocks ${blocksSpec()}` : "all blocks"),
|
||||
]),
|
||||
h("div", { class: "row", style: "margin-bottom:6px" }, PRESETS.map(p =>
|
||||
h("button", { onClick: () => { rules.value = JSON.parse(JSON.stringify(p.rules)); writeRules(); } }, p.label))),
|
||||
|
||||
h("div", { class: "sec" }, [h("h4", {}, `Rules (${rules.value.length}), later rules override earlier ones`)]),
|
||||
rules.value.length
|
||||
? h("div", { class: "row", style: "margin-bottom:5px" }, [
|
||||
h("input", { type: "text", value: saveName.value, placeholder: "file name (blank = from the rules)",
|
||||
style: "flex:1", onInput: e => { saveName.value = e.target.value; } }),
|
||||
h("button", { disabled: saving.value, title: "write this as a .safetensors in models/loras/surgery",
|
||||
onClick: async () => {
|
||||
saving.value = true; saved.value = "";
|
||||
try {
|
||||
const r = await api.fetchApi("/bfs/lora/save", {
|
||||
method: "POST", headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ name: props.getLora(), rules: rules.value,
|
||||
filename: saveName.value }),
|
||||
});
|
||||
const d = await r.json();
|
||||
saved.value = d.error ? `error: ${d.error}` : d.message;
|
||||
} catch (e) { saved.value = String(e); }
|
||||
saving.value = false;
|
||||
} }, saving.value ? "saving..." : "save as file"),
|
||||
])
|
||||
: null,
|
||||
saved.value ? h("div", { class: saved.value.startsWith("error") ? "err" : "muted",
|
||||
style: "margin-bottom:5px" }, saved.value) : null,
|
||||
rules.value.length > 1
|
||||
? h("div", { class: "muted", style: "margin-bottom:4px" },
|
||||
"Order matters: put boosts first and drops last, or a boost will undo a drop.")
|
||||
: null,
|
||||
...rules.value.map((r, i) => h("div", { class: "rule" }, [
|
||||
h("input", { type: "checkbox", checked: r.enabled !== false,
|
||||
onChange: e => { r.enabled = e.target.checked; writeRules(); } }),
|
||||
r.match.regex
|
||||
? h("div", { class: "rx", title: r.match.regex }, r.match.regex)
|
||||
: h("div", { class: "name" }, `${r.match.type || "*"} @ ${r.match.blocks || "all"}`),
|
||||
h("input", { type: "number", step: 0.05, value: r.scale,
|
||||
onChange: e => { r.scale = Number(e.target.value); writeRules(); } }),
|
||||
h("button", { title: "move up, later rules win", disabled: i === 0,
|
||||
onClick: () => { const a = rules.value; [a[i - 1], a[i]] = [a[i], a[i - 1]]; writeRules(); } }, "↑"),
|
||||
h("button", { title: "move down, later rules win", disabled: i === rules.value.length - 1,
|
||||
onClick: () => { const a = rules.value; [a[i + 1], a[i]] = [a[i], a[i + 1]]; writeRules(); } }, "↓"),
|
||||
h("button", { onClick: () => { rules.value.splice(i, 1); writeRules(); } }, "×"),
|
||||
])),
|
||||
rules.value.length
|
||||
? h("div", { class: "row", style: "margin-top:4px" },
|
||||
[h("button", { onClick: () => { rules.value = []; writeRules(); } }, "clear all")])
|
||||
: h("div", { class: "muted" }, "No rules, the LoRA is applied unchanged."),
|
||||
|
||||
h("div", { class: "sec" }, [
|
||||
h("div", { class: "row", style: "justify-content:space-between" }, [
|
||||
h("h4", {}, `Layers (${struct.value?.total_modules ?? 0} modules)`),
|
||||
h("div", { class: "row", style: "gap:2px" }, [
|
||||
h("span", { class: "tab" + (view.value === "grouped" ? " on" : ""),
|
||||
onClick: () => { view.value = "grouped"; } }, "grouped"),
|
||||
h("span", { class: "tab" + (view.value === "all" ? " on" : ""),
|
||||
onClick: () => { view.value = "all"; } }, "every layer"),
|
||||
]),
|
||||
]),
|
||||
]),
|
||||
view.value === "all"
|
||||
? h("div", {}, [
|
||||
h("input", { type: "text", placeholder: "filter, e.g. gate_up or blocks.1 or attn",
|
||||
value: filter.value, style: "margin-bottom:4px",
|
||||
onInput: e => { filter.value = e.target.value; } }),
|
||||
(() => {
|
||||
const all = struct.value?.modules || [];
|
||||
const q = filter.value.trim().toLowerCase();
|
||||
const hits = q ? all.filter(m => m.path.toLowerCase().includes(q)) : all;
|
||||
const shown = hits.slice(0, 400);
|
||||
return h("div", {}, [
|
||||
h("div", { class: "muted", style: "margin-bottom:3px" },
|
||||
`${hits.length} of ${all.length} modules${hits.length > shown.length ? `, showing first ${shown.length}` : ""}`),
|
||||
h("div", { class: "modlist" }, shown.map(m => h("div", { class: "mod" }, [
|
||||
h("div", { class: "name", title: m.path }, m.path),
|
||||
h("span", { class: "muted" }, m.norm != null ? m.norm.toFixed(3) : ""),
|
||||
h("span", { class: "muted" }, `r${m.rank ?? "?"}`),
|
||||
h("button", { title: "add a rule for exactly this module",
|
||||
onClick: () => {
|
||||
const esc = m.path.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
rules.value.push({ enabled: true, match: { regex: `^${esc}$` }, scale: Number(scale.value) });
|
||||
writeRules();
|
||||
} }, "+"),
|
||||
]))),
|
||||
]);
|
||||
})(),
|
||||
])
|
||||
: h("div", {}, (struct.value?.families || []).map(fam =>
|
||||
h("details", { class: "fam", open: fam.name === selFamily.value }, [
|
||||
h("summary", { onClick: () => { selFamily.value = fam.name; } }, [
|
||||
h("span", { class: "name" }, fam.name || "(root)"),
|
||||
h("span", { class: "muted" }, `${fam.blocks.length} blocks · ${fam.types.length} types`),
|
||||
]),
|
||||
...fam.types.map(t => h("div", { class: "type" }, [
|
||||
h("div", {}, [
|
||||
h("div", { class: "name", title: t.name }, t.name || "(none)"),
|
||||
h("div", { class: "bar", style: `width:${Math.round(100 * (t.norm || 0) / maxNorm.value)}%` }),
|
||||
]),
|
||||
h("span", { class: "muted" }, t.norm != null ? `||dW|| ${t.norm.toFixed(2)}` : ""),
|
||||
h("span", { class: "muted" }, `r${t.rank ?? "?"}`),
|
||||
h("button", { title: "select this type", onClick: () => { selFamily.value = fam.name; selType.value = t.name; } }, "->"),
|
||||
])),
|
||||
]))),
|
||||
]);
|
||||
}
|
||||
|
||||
app.registerExtension({
|
||||
name: "BFSNodes.LoraSurgery",
|
||||
async nodeCreated(node) {
|
||||
if (node.comfyClass !== "BFSLoraSurgery") return;
|
||||
styles();
|
||||
const rulesWidget = node.widgets?.find(w => w.name === "rules");
|
||||
const loraWidget = node.widgets?.find(w => w.name === "lora_name");
|
||||
if (rulesWidget) { rulesWidget.type = "hidden"; rulesWidget.computeSize = () => [0, -4]; }
|
||||
|
||||
const host = document.createElement("div");
|
||||
host.style.cssText = "width:100%;height:100%;min-height:340px";
|
||||
node.addDOMWidget("bfs_surgery_panel", "div", host, { serialize: false, hideOnZoom: false });
|
||||
|
||||
const panel = {};
|
||||
createApp({
|
||||
render: Panel({
|
||||
getLora: () => loraWidget?.value,
|
||||
getRules: () => rulesWidget?.value ?? "[]",
|
||||
setRules: v => { if (rulesWidget) rulesWidget.value = v; node.graph?.setDirtyCanvas(true); },
|
||||
}, panel),
|
||||
}).mount(host);
|
||||
|
||||
// The widget holds the whole configuration, so it must survive save/load and travel
|
||||
// with a shared workflow. Keep it serializable even though it is not drawn.
|
||||
if (rulesWidget) {
|
||||
rulesWidget.options = rulesWidget.options || {};
|
||||
rulesWidget.options.serialize = true;
|
||||
rulesWidget.serializeValue = () => rulesWidget.value;
|
||||
}
|
||||
const prevConfigure = node.onConfigure;
|
||||
node.onConfigure = function (info) {
|
||||
const r = prevConfigure?.apply(this, arguments);
|
||||
// values land during configure; re-read on the next tick so the panel shows them
|
||||
setTimeout(() => panel.reload?.(), 0);
|
||||
return r;
|
||||
};
|
||||
|
||||
node.size = [Math.max(node.size[0], 430), Math.max(node.size[1], 520)];
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,85 @@
|
||||
// Shows a node guide (Markdown in web/js/docs) as a formatted window inside ComfyUI.
|
||||
// A small renderer for what the guides use: headings, paragraphs, lists, tables, code, bold / italic, links.
|
||||
|
||||
const esc = s => s.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">");
|
||||
|
||||
function inline(s) {
|
||||
s = esc(s);
|
||||
s = s.replace(/`([^`]+)`/g, "<code>$1</code>");
|
||||
s = s.replace(/\*\*([^*]+)\*\*/g, "<b>$1</b>");
|
||||
s = s.replace(/(^|[^*])\*([^*\s][^*]*)\*/g, "$1<i>$2</i>");
|
||||
s = s.replace(/\[([^\]]+)\]\(([^)]+)\)/g, (m, t, u) => `<a href="${u}" target="_blank" rel="noopener">${t}</a>`);
|
||||
return s;
|
||||
}
|
||||
|
||||
export function renderMarkdown(md) {
|
||||
const lines = md.replace(/\r/g, "").split("\n");
|
||||
const out = [];
|
||||
let i = 0;
|
||||
while (i < lines.length) {
|
||||
const l = lines[i];
|
||||
if (/^```/.test(l)) { // code block
|
||||
const buf = []; i++;
|
||||
while (i < lines.length && !/^```/.test(lines[i])) buf.push(esc(lines[i++]));
|
||||
i++; out.push(`<pre>${buf.join("\n")}</pre>`); continue;
|
||||
}
|
||||
const h = l.match(/^(#{1,4})\s+(.*)/);
|
||||
if (h) { out.push(`<h${h[1].length}>${inline(h[2])}</h${h[1].length}>`); i++; continue; }
|
||||
if (/^\s*\|/.test(l)) { // table
|
||||
const rows = [];
|
||||
while (i < lines.length && /^\s*\|/.test(lines[i])) rows.push(lines[i++]);
|
||||
const cells = r => r.trim().replace(/^\||\|$/g, "").split(/(?<!\\)\|/).map(c => inline(c.trim().replace(/\\\|/g, "|")));
|
||||
const body = rows.filter(r => !/^\s*\|[\s:|-]+\|\s*$/.test(r));
|
||||
out.push("<table>" + body.map((r, k) => `<tr>${cells(r).map(c => k ? `<td>${c}</td>` : `<th>${c}</th>`).join("")}</tr>`).join("") + "</table>");
|
||||
continue;
|
||||
}
|
||||
if (/^\s*([-*]|\d+\.)\s+/.test(l)) { // list (one level, continuation lines joined)
|
||||
const ordered = /^\s*\d+\./.test(l); const items = [];
|
||||
while (i < lines.length && (/^\s*([-*]|\d+\.)\s+/.test(lines[i]) || (/^\s{2,}\S/.test(lines[i]) && items.length))) {
|
||||
const m = lines[i].match(/^\s*(?:[-*]|\d+\.)\s+(.*)/);
|
||||
if (m) items.push(m[1]); else items[items.length - 1] += " " + lines[i].trim();
|
||||
i++;
|
||||
}
|
||||
out.push(`<${ordered ? "ol" : "ul"}>${items.map(x => `<li>${inline(x)}</li>`).join("")}</${ordered ? "ol" : "ul"}>`);
|
||||
continue;
|
||||
}
|
||||
if (!l.trim()) { i++; continue; }
|
||||
const para = []; // paragraph
|
||||
while (i < lines.length && lines[i].trim() && !/^(#{1,4}\s|```|\s*\||\s*([-*]|\d+\.)\s)/.test(lines[i])) para.push(lines[i++]);
|
||||
out.push(`<p>${inline(para.join(" "))}</p>`);
|
||||
}
|
||||
return out.join("\n");
|
||||
}
|
||||
|
||||
function styles() {
|
||||
if (document.getElementById("bfs-md-css")) return;
|
||||
const st = document.createElement("style"); st.id = "bfs-md-css";
|
||||
st.textContent = `
|
||||
.bmd-back{position:fixed;inset:0;background:rgba(0,0,0,.7);z-index:10000;display:flex;align-items:center;justify-content:center}
|
||||
.bmd{background:#1b1b21;color:#e8e8ee;border:1px solid #3a3a45;border-radius:12px;width:min(900px,94vw);max-height:92vh;overflow:auto;
|
||||
padding:18px 24px;font:13px/1.6 system-ui,sans-serif}
|
||||
.bmd h1{font-size:20px;margin:4px 0 10px}.bmd h2{font-size:16px;margin:18px 0 6px;color:#ffd27a}.bmd h3{font-size:14px;margin:14px 0 4px}
|
||||
.bmd code{background:#2a2a33;padding:1px 4px;border-radius:4px}
|
||||
.bmd pre{background:#121217;border:1px solid #2c2c35;border-radius:8px;padding:10px;white-space:pre-wrap;font-size:12px}
|
||||
.bmd table{border-collapse:collapse;margin:8px 0;width:100%}.bmd td,.bmd th{border:1px solid #33333d;padding:4px 8px;text-align:left;vertical-align:top}
|
||||
.bmd th{background:#24242c}.bmd a{color:#8fb3ff}
|
||||
.bmd-close{float:right;cursor:pointer;border:1px solid #444;background:#26262e;color:#ddd;border-radius:6px;padding:2px 10px}`;
|
||||
document.head.appendChild(st);
|
||||
}
|
||||
|
||||
export async function showGuide(url) {
|
||||
styles();
|
||||
const back = document.createElement("div"); back.className = "bmd-back";
|
||||
back.innerHTML = `<div class="bmd"><button class="bmd-close">close</button><div class="bmd-body">Loading…</div></div>`;
|
||||
back.addEventListener("pointerdown", e => { if (e.target === back) back.remove(); });
|
||||
back.querySelector(".bmd-close").addEventListener("click", () => back.remove());
|
||||
back.addEventListener("keydown", e => e.stopPropagation());
|
||||
document.body.appendChild(back);
|
||||
try {
|
||||
const r = await fetch(url);
|
||||
if (!r.ok) throw new Error(`HTTP ${r.status}`);
|
||||
back.querySelector(".bmd-body").innerHTML = renderMarkdown(await r.text());
|
||||
} catch (e) {
|
||||
back.querySelector(".bmd-body").textContent = `Could not load the guide (${e.message}).`;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,706 @@
|
||||
/**
|
||||
* BFS Shot Planner: split a long video into model-sized shots on a timeline, give each shot its own
|
||||
* reference image and prompt, and run the rest of the graph once per shot.
|
||||
*
|
||||
* All state lives in the hidden `plan` STRING widget (JSON) so a workflow saves, reloads and shares
|
||||
* exactly what you set up. The panel is a Vue app mounted into a DOM widget; the server does the video
|
||||
* work (probe, thumbnails, PySceneDetect cuts, split) through /bfs/shotloop/* routes.
|
||||
*
|
||||
* This file holds the state and the actions; the views (one per tab) live in ./shotloop/.
|
||||
*/
|
||||
import { app } from "../../scripts/app.js";
|
||||
import { api } from "../../scripts/api.js";
|
||||
import { showGuide } from "./bfs_md_view.js";
|
||||
import { createApp, ref, reactive, computed, watch, nextTick, onMounted, onBeforeUnmount } from "./vendor/vue.esm-browser.prod.mjs";
|
||||
import { h, t, uiLanguage, setUiLanguage, DEFAULTS, MASK_DEFAULTS, VLM_DEFAULTS, snapUp, snapDown, segKey, hasMask, pill } from "./shotloop/common.js";
|
||||
import { styles } from "./shotloop/styles.js";
|
||||
import { timelineCard } from "./shotloop/view_timeline.js";
|
||||
import { videoTab } from "./shotloop/view_video.js";
|
||||
import { shotsTab } from "./shotloop/view_shots.js";
|
||||
import { peopleTab } from "./shotloop/view_people.js";
|
||||
import { promptsTab } from "./shotloop/view_prompts.js";
|
||||
import { runTab } from "./shotloop/view_run.js";
|
||||
import { busyBar, pointsModal, tourModal } from "./shotloop/view_modal.js";
|
||||
|
||||
const viewUrl = name => {
|
||||
if (!name) return "";
|
||||
const i = name.lastIndexOf("/");
|
||||
const sub = i >= 0 ? name.slice(0, i) : "", file = i >= 0 ? name.slice(i + 1) : name;
|
||||
return api.apiURL(`/view?filename=${encodeURIComponent(file)}&subfolder=${encodeURIComponent(sub)}&type=input`);
|
||||
};
|
||||
const store = { // per-browser UI conveniences (never the plan itself)
|
||||
get: (k, d) => { try { const v = localStorage.getItem(k); return v == null ? d : JSON.parse(v); } catch { return d; } },
|
||||
set: (k, v) => { try { localStorage.setItem(k, JSON.stringify(v)); } catch { /* private mode */ } },
|
||||
};
|
||||
const DEFAULT_W = 1180, MIN_W = 1040, MIN_H = 1000; // node size: wide enough for the four-column settings
|
||||
const TABS = [["video", "🎞 Video"], ["shots", "🎬 Shots"], ["people", "👥 People & masks"], ["prompts", "📝 Prompts & refs"], ["run", "▶ Run"]];
|
||||
|
||||
function Panel(io) {
|
||||
const plan = reactive({ ...DEFAULTS });
|
||||
const files = reactive({ videos: [], images: [] });
|
||||
const an = ref(null); // analysis from the server
|
||||
const cuts = ref([]);
|
||||
const detectorUsed = ref("");
|
||||
const size = reactive({ w: 0, h: 0 });
|
||||
const busy = ref(""); const error = ref("");
|
||||
// loading bar: what the server reports (stage, done/total) and how long it has been busy
|
||||
const status = reactive({ stage: "", done: 0, total: 0 });
|
||||
const busySince = ref(0); const now = ref(Date.now());
|
||||
const upPct = ref(-1);
|
||||
watch(busy, v => { if (v) { busySince.value = Date.now(); Object.assign(status, { stage: "", done: 0, total: 0 }); } else { busySince.value = 0; upPct.value = -1; } });
|
||||
const sel = ref(0); const zoom = ref(1);
|
||||
const rootEl = ref(null);
|
||||
// keep the selected shot's card in view in the sideways strip
|
||||
watch(sel, () => nextTick(() => rootEl.value?.querySelector(".shots .sc.sel")?.scrollIntoView({ block: "nearest", inline: "nearest" })));
|
||||
const hover = ref(null); // {frame, x}
|
||||
const prog = reactive({ done: 0, count: 0 });
|
||||
const tlEl = ref(null);
|
||||
const stats = ref([]); // per-shot content stats from /bfs/shotloop/filters
|
||||
const people = ref([]); // cast from /bfs/shotloop/cast: [{id, thumb, share, first, last}]
|
||||
const maskPrev = ref({}); // "start-end" -> {frames, box, coverage, empty} from /bfs/shotloop/mask
|
||||
const showMasks = ref(true); // overlay the mask preview on the shot cards
|
||||
const vlmSug = ref({}); // "start-end" -> {segment, shot, people, recommend, reason} from the VLM
|
||||
const modal = reactive({ open: false, idx: -1, key: 0, points: [], text: "", prev: null, busy: "" });
|
||||
const segPeople = ref({}); // "start-end" -> {people, main} from /bfs/shotloop/plan
|
||||
const targetCrop = ref({}); // "start-end" -> data URL of the subject the description was made from
|
||||
const tab = ref(store.get("bfs.shotloop.tab", "shots"));
|
||||
watch(tab, v => store.set("bfs.shotloop.tab", v));
|
||||
const tour = ref(0); // opened only by the user's Tour button; never shown automatically
|
||||
// what "Copy to other shots" copies (remembered per browser)
|
||||
const copyOpts = reactive(store.get("bfs.shotloop.copy", { ref: true, prompt: false, mask_text: false, points: false, crop: false,
|
||||
target: false, chain: false, color_guide: false, scope: "all" }));
|
||||
watch(copyOpts, v => store.set("bfs.shotloop.copy", { ...v }), { deep: true });
|
||||
|
||||
const load = () => {
|
||||
let p = {};
|
||||
try { p = JSON.parse(io.getPlan() || "{}"); } catch { p = {}; }
|
||||
Object.assign(plan, DEFAULTS, p);
|
||||
plan.filters = { ...DEFAULTS.filters, ...(p.filters || {}) };
|
||||
plan.cast = { ...(p.cast || {}) };
|
||||
plan.mask_cfg = { ...MASK_DEFAULTS, ...(p.mask_cfg || {}) };
|
||||
plan.vlm_cfg = { ...VLM_DEFAULTS, ...(p.vlm_cfg || {}) };
|
||||
plan.ref_details = { ...(p.ref_details || {}) };
|
||||
if (!Array.isArray(plan.bounds)) plan.bounds = [];
|
||||
if (!Array.isArray(plan.segs)) plan.segs = [];
|
||||
};
|
||||
const save = () => io.setPlan(JSON.stringify(plan));
|
||||
|
||||
const n = computed(() => (an.value ? an.value.n : 0)); // the whole video on the timeline
|
||||
// the part the plan works on [lo, hi): start / end (0 = the end), then "max total seconds" counted from the start
|
||||
const range = computed(() => {
|
||||
const N = n.value; if (!N) return { lo: 0, hi: 0 };
|
||||
// frames (older plans saved seconds)
|
||||
const a = plan.start_f != null ? +plan.start_f : Math.round((+plan.start_s || 0) * plan.fps);
|
||||
const b = plan.end_f != null ? +plan.end_f : (+plan.end_s > 0 ? Math.round(+plan.end_s * plan.fps) : 0);
|
||||
const lo = Math.max(0, Math.min(N - 1, a));
|
||||
let hi = b > 0 ? Math.max(lo + 1, Math.min(N, b)) : N;
|
||||
if (+plan.max_total_s > 0) hi = Math.min(hi, lo + Math.max(1, Math.round(+plan.max_total_s * plan.fps)));
|
||||
return { lo, hi };
|
||||
});
|
||||
const maxLen = computed(() => snapDown(Math.round(plan.max_s * plan.fps), plan.grid));
|
||||
const segs = computed(() => {
|
||||
const N = n.value; if (!N) return [];
|
||||
const { lo, hi } = range.value;
|
||||
const b = [lo, ...plan.bounds.filter(x => x > lo && x < hi), hi].sort((a, c) => a - c);
|
||||
const cs = new Set(cuts.value);
|
||||
let out = [];
|
||||
for (let i = 0; i < b.length - 1; i++) {
|
||||
if (b[i + 1] <= b[i]) continue;
|
||||
const m = plan.segs[out.length] || {};
|
||||
out.push({ start: b[i], end: b[i + 1], len: b[i + 1] - b[i], gen: snapUp(b[i + 1] - b[i], plan.grid),
|
||||
cut: cs.has(b[i]), enabled: m.enabled !== false, ref: m.ref || "", ref2: m.ref2 || "", prompt: m.prompt || "",
|
||||
force: m.force || "auto", chain: m.chain || "history", chainFrame: m.chain_frame || "first", chainHistory: m.chain_history || 17,
|
||||
crop: !!m.crop, inpaint: !!m.inpaint, paste: !!m.paste, pasteText: m.paste_text || "", strength: m.strength ?? 1, mask: m.mask || {}, target: m.target || "",
|
||||
extMask: !!plan.mask_video || !!(plan.track?.points?.length), trackMask: !!(plan.track?.points?.length) });
|
||||
}
|
||||
if (plan.max_parts > 0) out = out.slice(0, plan.max_parts);
|
||||
return out;
|
||||
});
|
||||
const whoIn = s => segPeople.value[segKey(s)] || null;
|
||||
const castOf = id => plan.cast[String(id)] || {};
|
||||
const linked = id => !!castOf(id).ref && !castOf(id).ignore;
|
||||
// reference a shot gets from its main person when it has none of its own
|
||||
const castRef = (s, field) => {
|
||||
if (!plan.cast_assign) return "";
|
||||
const who = whoIn(s); if (!who || who.main < 0 || castOf(who.main).ignore) return "";
|
||||
return castOf(who.main)[field] || "";
|
||||
};
|
||||
const personOf = id => people.value.find(p => p.id === id) || null;
|
||||
const statFor = s => stats.value.find(x => x.start === s.start && x.end === s.end) || null;
|
||||
const filtersOn = computed(() => { const f = plan.filters; return !!(f.person || f.face || f.skip_dark || f.skip_static || f.max_persons > 0 || f.min_frames > 0); });
|
||||
const skipWhy = s => {
|
||||
if (!s.enabled) return "disabled";
|
||||
if (s.force === "run") return "";
|
||||
if (s.force === "skip") return "skipped by hand";
|
||||
const why = filtersOn.value ? (statFor(s)?.skip_reason || "") : "";
|
||||
if (why) return why;
|
||||
const who = whoIn(s);
|
||||
if (plan.cast_only && who && !who.people.some(p => linked(p))) return "no linked person";
|
||||
return "";
|
||||
};
|
||||
const active = computed(() => segs.value.filter(s => !skipWhy(s)));
|
||||
const pxPerFrame = computed(() => {
|
||||
const w = (tlEl.value?.clientWidth || 600) - 2;
|
||||
return Math.max(0.2, (w / Math.max(1, n.value)) * zoom.value);
|
||||
});
|
||||
const tlWidth = computed(() => Math.max(1, Math.round(n.value * pxPerFrame.value)));
|
||||
|
||||
// how a shot is generated: crop (the planner crops, the join pastes back) x inpaint (only the mask is regenerated,
|
||||
// read by BFS Shot H3 Conditioning with inpaint = per shot)
|
||||
// [key, name, crop, inpaint, paste]; paste = the whole frame is generated and BFS Shot Join pastes only the person back
|
||||
const MODES = [["frame", "Full frame", false, false, false], ["paste", "Frame + paste", false, false, true], ["mask", "Mask only", false, true, false],
|
||||
["crop", "Crop", true, false, false], ["cropmask", "Crop + mask", true, true, false]];
|
||||
const modeOf = s => (MODES.find(m => m[2] === !!s.crop && m[3] === !!s.inpaint && m[4] === (!!s.paste && !s.crop && !s.inpaint))
|
||||
|| MODES[0])[0];
|
||||
const modeName = s => MODES.find(m => m[0] === modeOf(s))[1];
|
||||
|
||||
const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return plan.segs[i]; };
|
||||
// the reference a shot really uses: its own, its main person's, or the global one
|
||||
const refOf = (s, k) => s[k] || castRef(s, k) || plan["global_" + k] || "";
|
||||
const promptOf = s => s.prompt || plan.global_prompt || "";
|
||||
const detailsOf = s => plan.ref_details?.[`${refOf(s, "ref")}|${refOf(s, "ref2")}`] || "";
|
||||
|
||||
// things worth fixing before a run, per shot (only shots that run)
|
||||
const checks = computed(() => {
|
||||
const out = [];
|
||||
segs.value.forEach((s, i) => {
|
||||
if (skipWhy(s)) return;
|
||||
const add = (lvl, text) => out.push({ lvl, shot: i, text });
|
||||
const p = promptOf(s);
|
||||
if (s.len > maxLen.value) add(plan.mode === "single" ? "note" : "warn", plan.mode === "single"
|
||||
? `one shot of ${s.len} frames (${(s.len / plan.fps).toFixed(1)} s), longer than Max seconds / shot: the model may not hold it`
|
||||
: `longer than ${maxLen.value} frames: split it`);
|
||||
if ((s.crop || s.inpaint || s.paste) && !hasMask(s)) add("warn", `${modeName(s)} needs a mask (text, points or a mask video; or the planner's mask input)`);
|
||||
if (hasMask(s) && maskPrev.value[segKey(s)]?.empty) add("warn", "the mask preview found nothing");
|
||||
if (/\{target\}/.test(p) && !s.target) add("warn", "the prompt uses {target} but the shot has no target description");
|
||||
if (/\{details\}/.test(p) && !detailsOf(s) && !plan.vlm_cfg?.enabled) add("info", "the prompt uses {details} but its references have no description yet");
|
||||
if (/\{details\}/.test(p) && (plan.vlm_cfg?.describe_preset || "short") !== "short")
|
||||
add("warn", "{details} sits inside a sentence of the prompt: use the 'short' describe preset (Prompts & refs), the other presets write a paragraph");
|
||||
if (!p) add("info", "no prompt (fine when the node's prompt input is connected)");
|
||||
if (!refOf(s, "ref")) add("info", "no reference (fine when the node's ref_image input is connected)");
|
||||
});
|
||||
return out;
|
||||
});
|
||||
const checksFor = i => checks.value.filter(c => c.shot === i);
|
||||
|
||||
async function refreshFiles() {
|
||||
try { const r = await api.fetchApi("/bfs/shotloop/files"); const j = await r.json(); files.videos = j.videos || []; } catch (e) { /* ignore */ }
|
||||
}
|
||||
async function analyze() {
|
||||
an.value = null; error.value = "";
|
||||
if (!plan.video) return;
|
||||
busy.value = "Analysing video…";
|
||||
try {
|
||||
const r = await api.fetchApi(`/bfs/shotloop/analyze?video=${encodeURIComponent(plan.video)}&fps=${plan.fps}`);
|
||||
const j = await r.json();
|
||||
if (j.error) throw new Error(j.error);
|
||||
an.value = j;
|
||||
await autoSplit(plan.bounds.length === 0);
|
||||
people.value = []; segPeople.value = {};
|
||||
if (Object.keys(plan.cast).length || plan.cast_split || plan.cast_only) await findPeople(false); // restore the cast of a saved plan
|
||||
} catch (e) { error.value = String(e.message || e); }
|
||||
busy.value = "";
|
||||
}
|
||||
async function autoSplit(apply = true) {
|
||||
if (!plan.video) return;
|
||||
busy.value = "Detecting cuts…"; error.value = "";
|
||||
try {
|
||||
const body = { plan: { ...plan, bounds: apply ? [] : plan.bounds } };
|
||||
const r = await api.fetchApi("/bfs/shotloop/plan", { method: "POST", body: JSON.stringify(body) });
|
||||
const j = await r.json();
|
||||
if (j.error) throw new Error(j.error);
|
||||
cuts.value = j.cuts || []; detectorUsed.value = j.detector || ""; size.w = j.width; size.h = j.height;
|
||||
segPeople.value = Object.fromEntries(j.segs.filter(s => s.main !== undefined && (s.people.length || people.value.length))
|
||||
.map(s => [`${s.start}-${s.end}`, { people: s.people, main: s.main }]));
|
||||
if (apply) {
|
||||
plan.bounds = j.segs.slice(1).map(s => s.start);
|
||||
const keep = plan.segs; plan.segs = j.segs.map((_, i) => ({ enabled: true, ref: keep[i]?.ref || "", ref2: keep[i]?.ref2 || "", prompt: keep[i]?.prompt || "",
|
||||
chain: keep[i]?.chain || "history", chain_frame: keep[i]?.chain_frame || "first", chain_history: keep[i]?.chain_history || 17,
|
||||
crop: !!keep[i]?.crop, inpaint: !!keep[i]?.inpaint, paste: !!keep[i]?.paste, paste_text: keep[i]?.paste_text || "", strength: keep[i]?.strength ?? 1, mask: keep[i]?.mask || {}, target: keep[i]?.target || "",
|
||||
color_guide: !!keep[i]?.color_guide, color_guide_color: keep[i]?.color_guide_color || "", color_guide_opacity: keep[i]?.color_guide_opacity ?? 0.35,
|
||||
color_guide_person: keep[i]?.color_guide_person ?? null, color_guide_text2: keep[i]?.color_guide_text2 || "",
|
||||
color_guide_color2: keep[i]?.color_guide_color2 || "", color_guide_person2: keep[i]?.color_guide_person2 ?? null }));
|
||||
sel.value = 0; save();
|
||||
}
|
||||
} catch (e) { error.value = String(e.message || e); }
|
||||
busy.value = "";
|
||||
}
|
||||
async function analyzeContent() {
|
||||
if (!plan.video) return;
|
||||
busy.value = "Detecting people and faces…"; error.value = "";
|
||||
try {
|
||||
const r = await api.fetchApi("/bfs/shotloop/filters", { method: "POST", body: JSON.stringify({ plan: { ...plan } }) });
|
||||
const j = await r.json(); if (j.error) throw new Error(j.error);
|
||||
stats.value = j.segs.map(x => ({ ...x.stats, start: x.start, end: x.end, skip_reason: x.skip_reason }));
|
||||
} catch (e) { error.value = String(e.message || e); }
|
||||
busy.value = "";
|
||||
}
|
||||
async function findPeople(resplit = true) {
|
||||
if (!plan.video) return;
|
||||
busy.value = "Finding people by face…"; error.value = "";
|
||||
try {
|
||||
const r = await api.fetchApi("/bfs/shotloop/cast", { method: "POST", body: JSON.stringify({ plan: { ...plan } }) });
|
||||
const j = await r.json(); if (j.error) throw new Error(j.error);
|
||||
people.value = j.people || [];
|
||||
} catch (e) { error.value = String(e.message || e); busy.value = ""; return; }
|
||||
busy.value = "";
|
||||
await autoSplit(resplit && !!plan.cast_split);
|
||||
}
|
||||
const setCast = (id, k, v) => {
|
||||
plan.cast = { ...plan.cast, [String(id)]: { ...castOf(id), [k]: v } };
|
||||
if (k === "ref" || k === "ref2") remember(v);
|
||||
save();
|
||||
};
|
||||
async function describeTarget(i) {
|
||||
const s = segs.value[i]; if (!s) return;
|
||||
busy.value = "Selecting the person (SAM 3) and describing them (VLM)…"; busySince.value = Date.now(); error.value = "";
|
||||
try {
|
||||
const r = await api.fetchApi("/bfs/shotloop/target", { method: "POST", body: JSON.stringify({ plan: { ...plan }, index: i }) });
|
||||
const j = await r.json(); if (j.error) throw new Error(j.error);
|
||||
targetCrop.value = { ...targetCrop.value, [segKey(s)]: j.crop };
|
||||
if (j.desc) setMeta(i, "target", j.desc);
|
||||
else error.value = "The SAM 3 target cutout was created (shown below), but no VLM is loaded to describe it. Connect Qwen3-VL to the Shot Planner's vlm input and queue the workflow once; then click Describe target again. You can also type the target description yourself.";
|
||||
} catch (e) { error.value = String(e.message || e); }
|
||||
busy.value = "";
|
||||
}
|
||||
async function previewMask(i, spec, intoModal = false) {
|
||||
const s = segs.value[i]; if (!s) return;
|
||||
const label = "Segmenting with SAM 3… (the first time downloads/loads the model)";
|
||||
if (intoModal) { modal.busy = label; busySince.value = Date.now(); Object.assign(status, { stage: "", done: 0, total: 0 }); } else busy.value = label;
|
||||
error.value = "";
|
||||
try {
|
||||
const r = await api.fetchApi("/bfs/shotloop/mask", { method: "POST", body: JSON.stringify({ plan: { ...plan }, index: i, mask: spec || null }) });
|
||||
const j = await r.json(); if (j.error) throw new Error(j.error);
|
||||
if (intoModal) modal.prev = j; else maskPrev.value = { ...maskPrev.value, [segKey(s)]: j };
|
||||
} catch (e) { error.value = String(e.message || e); }
|
||||
if (intoModal) modal.busy = ""; else busy.value = "";
|
||||
}
|
||||
const setMask = (i, k, v) => {
|
||||
const m = meta(i); if ((m.mask || {})[k] === v) return;
|
||||
m.mask = { ...(m.mask || {}), [k]: v }; save();
|
||||
const s = segs.value[i]; // the old preview no longer matches
|
||||
if (s && maskPrev.value[segKey(s)]) { const mp = { ...maskPrev.value }; delete mp[segKey(s)]; maskPrev.value = mp; }
|
||||
};
|
||||
const setMaskCfg = (k, v) => { plan.mask_cfg = { ...plan.mask_cfg, [k]: v }; save(); maskPrev.value = {}; };
|
||||
const openPoints = i => {
|
||||
const s = segs.value[i]; if (!s) return;
|
||||
Object.assign(modal, { open: true, idx: i, key: s.mask.key ?? Math.floor(s.len / 2), points: [...(s.mask.points || [])],
|
||||
text: s.mask.text || "", prev: null, busy: "" });
|
||||
};
|
||||
const savePoints = () => {
|
||||
const m = meta(modal.idx); m.mask = { ...(m.mask || {}), points: modal.points, key: modal.key, text: modal.text }; save();
|
||||
if (modal.prev) maskPrev.value = { ...maskPrev.value, [segKey(segs.value[modal.idx])]: modal.prev };
|
||||
modal.open = false;
|
||||
};
|
||||
const targets = (scope, from) => segs.value.map((_, i) => i).filter(i => i !== from && (scope === "all" || i > from));
|
||||
// the same selection on other shots (a subject that stays in place): each shot gets the points on its frame at the
|
||||
// same relative position as the key frame here
|
||||
const pointsTo = (points, key, text, fromIdx, scope = "all") => {
|
||||
const src = segs.value[fromIdx]; const rel = src ? key / Math.max(1, src.len - 1) : 0.5;
|
||||
[fromIdx, ...targets(scope, fromIdx)].forEach(i => {
|
||||
const x = segs.value[i], m = meta(i);
|
||||
m.mask = { ...(m.mask || {}), points: points.map(p => ({ ...p })), key: Math.round(rel * Math.max(0, x.len - 1)), text: text ?? (m.mask || {}).text };
|
||||
});
|
||||
maskPrev.value = {}; save();
|
||||
};
|
||||
const savePointsAll = () => { pointsTo(modal.points, modal.key, modal.text, modal.idx); modal.open = false; };
|
||||
// one selection tracked by SAM 3 through the whole video (or range): every shot without its own selection takes its
|
||||
// part of that track, so the subject is found wherever it moves (copied points can fall on the background)
|
||||
const saveTrack = () => {
|
||||
const s0 = segs.value[modal.idx]; if (!s0 || !modal.points.length) return;
|
||||
const own = segs.value.filter((x, i) => i !== modal.idx && (x.mask.text || (x.mask.points || []).length)).length;
|
||||
if (own && !window.confirm(`${own} other shot(s) have their own mask text or points, which would win over the track. Clear them so every shot uses the track?`)) {
|
||||
// keep them: those shots keep their own selection
|
||||
} else {
|
||||
segs.value.forEach((_, i) => { const m = meta(i); if (m.mask) { m.mask = { ...m.mask, points: [], text: "" }; } });
|
||||
}
|
||||
const m = meta(modal.idx); m.mask = { ...(m.mask || {}), points: [], text: "" };
|
||||
plan.track = { points: modal.points.map(p => ({ ...p })), frame: s0.start + modal.key, text: modal.text || "",
|
||||
threshold: plan.mask_cfg?.threshold ?? 0.5 };
|
||||
maskPrev.value = {}; save(); modal.open = false;
|
||||
};
|
||||
const clearTrack = () => { plan.track = null; maskPrev.value = {}; save(); };
|
||||
const clearMask = i => {
|
||||
const m = meta(i); m.mask = {}; m.crop = false; m.inpaint = false; m.paste = false;
|
||||
const k = segs.value[i] ? segKey(segs.value[i]) : null;
|
||||
if (k) { const mp = { ...maskPrev.value }; delete mp[k]; maskPrev.value = mp; }
|
||||
save();
|
||||
};
|
||||
// copy the chosen settings of shot `from` to the other shots (all of them, or the ones after it)
|
||||
const COPY_FIELDS = { ref: "references", prompt: "prompt", mask_text: "mask text", points: "mask points", mask_video: "mask video", crop: "mode",
|
||||
target: "{target}", chain: "continuity", color_guide: "Color Guide" };
|
||||
const copyFrom = (from, fields = copyOpts, scope = copyOpts.scope) => {
|
||||
const src = segs.value[from]; if (!src) return 0;
|
||||
const to = targets(scope, from);
|
||||
to.forEach(i => {
|
||||
const m = meta(i), x = segs.value[i];
|
||||
if (fields.ref) { m.ref = src.ref; m.ref2 = src.ref2; }
|
||||
if (fields.prompt) m.prompt = src.prompt;
|
||||
if (fields.mask_text) m.mask = { ...(m.mask || {}), text: src.mask.text || "" };
|
||||
if (fields.mask_video) m.mask = { ...(m.mask || {}), video: src.mask.video || "" };
|
||||
if (fields.crop && (hasMask(x) || src.mask.text || fields.mask_text)) { m.crop = src.crop; m.inpaint = src.inpaint; m.paste = src.paste; m.paste_text = src.pasteText; m.strength = src.strength; }
|
||||
if (fields.target) m.target = src.target;
|
||||
if (fields.color_guide) {
|
||||
m.color_guide = src.color_guide; m.color_guide_color = src.color_guide_color;
|
||||
m.color_guide_opacity = src.color_guide_opacity; m.color_guide_person = src.color_guide_person;
|
||||
m.color_guide_text2 = src.color_guide_text2; m.color_guide_color2 = src.color_guide_color2;
|
||||
m.color_guide_person2 = src.color_guide_person2;
|
||||
}
|
||||
if (fields.chain && i > 0) { m.chain = src.chain; m.chain_frame = src.chainFrame; m.chain_history = src.chainHistory; }
|
||||
});
|
||||
if (fields.points && (src.mask.points || []).length) pointsTo(src.mask.points, src.mask.key ?? Math.floor(src.len / 2), undefined, from, scope);
|
||||
if (fields.mask_text || fields.points || fields.mask_video) maskPrev.value = {};
|
||||
save();
|
||||
return to.length;
|
||||
};
|
||||
const mergeSug = list => { const o = { ...vlmSug.value }; for (const x of list || []) o[`${x.start}-${x.end}`] = x; vlmSug.value = o; };
|
||||
async function analyseVLM() {
|
||||
busy.value = "Asking the VLM about every shot…"; error.value = "";
|
||||
try {
|
||||
const r = await api.fetchApi("/bfs/shotloop/vlm", { method: "POST", body: JSON.stringify({ plan: { ...plan } }) });
|
||||
const j = await r.json(); if (j.error) throw new Error(j.error);
|
||||
mergeSug(j.segs);
|
||||
} catch (e) { error.value = String(e.message || e); }
|
||||
busy.value = "";
|
||||
}
|
||||
// reference sets used by the plan (global, every shot's, every cast person's): one description each
|
||||
const refSets = computed(() => {
|
||||
const out = new Map();
|
||||
const add = (a, b, where) => { if (!a && !b) return; const k = `${a || ""}|${b || ""}`; if (!out.has(k)) out.set(k, { a, b, where: [] }); out.get(k).where.push(where); };
|
||||
add(plan.global_ref, plan.global_ref2, "global");
|
||||
segs.value.forEach((x, i) => add(x.ref || plan.global_ref, x.ref2 || plan.global_ref2, `#${i + 1}`));
|
||||
Object.entries(plan.cast || {}).forEach(([id, c]) => { if (!c.ignore) add(c.ref, c.ref2, `person ${id}`); });
|
||||
return [...out.entries()].map(([key, v]) => ({ key, ...v }));
|
||||
});
|
||||
async function describeRefs() {
|
||||
busy.value = "Describing the references with the VLM…"; error.value = "";
|
||||
try {
|
||||
const sets = refSets.value.map(x => [x.a || "", x.b || ""]);
|
||||
const r = await api.fetchApi("/bfs/shotloop/describe", { method: "POST", body: JSON.stringify({ plan: { ...plan }, sets }) });
|
||||
const j = await r.json(); if (j.error) throw new Error(j.error);
|
||||
plan.ref_details = { ...plan.ref_details, ...j.texts }; save();
|
||||
} catch (e) { error.value = String(e.message || e); }
|
||||
busy.value = "";
|
||||
}
|
||||
const setDetail = (k, v) => { plan.ref_details = { ...plan.ref_details, [k]: v }; save(); };
|
||||
const setVlmCfg = (k, v) => { plan.vlm_cfg = { ...plan.vlm_cfg, [k]: v }; save(); };
|
||||
const applySug = (i, what) => {
|
||||
const s = segs.value[i], g = s && vlmSug.value[segKey(s)]; if (!g) return;
|
||||
if (what === "segment" && g.segment) setMask(i, "text", g.segment);
|
||||
if (what === "skip") setMeta(i, "force", g.recommend === "skip" ? "skip" : "auto");
|
||||
};
|
||||
const applyAllSug = () => {
|
||||
segs.value.forEach((s, i) => {
|
||||
const g = vlmSug.value[segKey(s)]; if (!g) return;
|
||||
const m = meta(i);
|
||||
if (g.segment && !(m.mask?.text || (m.mask?.points || []).length)) m.mask = { ...(m.mask || {}), text: g.segment };
|
||||
if (g.recommend === "skip" && (m.force || "auto") === "auto") m.force = "skip";
|
||||
});
|
||||
save();
|
||||
};
|
||||
const setCastOpt = (k, v) => { plan[k] = v; save(); if (k === "cast_split") autoSplit(true); };
|
||||
const setFilter = (k, v) => { plan.filters = { ...plan.filters, [k]: v }; save(); if (stats.value.length) analyzeContent(); };
|
||||
const setPlan = (k, v, after) => { plan[k] = v; save(); after && after(); };
|
||||
// the range changed: split again inside it (the per-shot settings stay with the shot at the same position)
|
||||
// start / end in timeline frames (end 0 = the end of the video); the old seconds fields are cleared
|
||||
const setRange = (k, v) => { plan[k] = Math.max(0, Math.round(+v || 0)); plan.start_s = 0; plan.end_s = 0; save(); autoSplit(true); };
|
||||
const setGlobalMaskVideo = v => { plan.mask_video = v; save(); maskPrev.value = {}; };
|
||||
// XHR instead of fetch: big videos show how much has been sent
|
||||
const postWithProgress = (url, body, onPct) => new Promise((resolve, reject) => {
|
||||
const x = new XMLHttpRequest();
|
||||
x.open("POST", api.apiURL(url));
|
||||
if (api.user) x.setRequestHeader("Comfy-User", api.user);
|
||||
x.upload.onprogress = e => { if (e.lengthComputable) onPct(e.loaded / e.total); };
|
||||
x.onload = () => (x.status >= 200 && x.status < 300) ? resolve(JSON.parse(x.responseText || "{}")) : reject(new Error(`HTTP ${x.status}`));
|
||||
x.onerror = () => reject(new Error("network error"));
|
||||
x.send(body);
|
||||
});
|
||||
// big files go in 8 MB pieces: ComfyUI refuses any single request over --max-upload-size (100 MB by default)
|
||||
const CHUNK = 8 * 1024 * 1024, CHUNKED_FROM = 32 * 1024 * 1024;
|
||||
async function uploadChunked(file, onPct) {
|
||||
const id = `${Date.now().toString(36)}_${Math.random().toString(36).slice(2, 12)}`;
|
||||
for (let off = 0; off < file.size; off += CHUNK) {
|
||||
const piece = file.slice(off, Math.min(file.size, off + CHUNK));
|
||||
for (let tries = 0; ; tries++) { // a dropped piece is sent again (it is written at its offset)
|
||||
try {
|
||||
await postWithProgress(`/bfs/upload/chunk?id=${id}&offset=${off}`, piece,
|
||||
f => onPct((off + f * piece.size) / file.size));
|
||||
break;
|
||||
} catch (e) { if (tries >= 3) throw e; await new Promise(r => setTimeout(r, 1000 * (tries + 1))); }
|
||||
}
|
||||
}
|
||||
const r = await fetch(api.apiURL("/bfs/upload/finish"), { method: "POST", headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ id, name: file.name, size: file.size }) });
|
||||
const j = await r.json();
|
||||
if (!r.ok || j.error) throw new Error(j.error || `HTTP ${r.status}`);
|
||||
return j.name;
|
||||
}
|
||||
async function upload(file, cb) {
|
||||
const mb = (file.size / 1e6).toFixed(1);
|
||||
busy.value = `Uploading ${file.name} (${mb} MB)…`; upPct.value = 0;
|
||||
try {
|
||||
let name;
|
||||
if (file.size >= CHUNKED_FROM) name = await uploadChunked(file, f => { upPct.value = f; });
|
||||
else {
|
||||
const fd = new FormData(); fd.append("image", file); fd.append("type", "input"); fd.append("overwrite", "true");
|
||||
const j = await postWithProgress("/upload/image", fd, f => { upPct.value = f; });
|
||||
name = j.subfolder ? `${j.subfolder}/${j.name}` : j.name;
|
||||
}
|
||||
await refreshFiles();
|
||||
cb(name);
|
||||
} catch (e) { error.value = `Upload failed: ${e}`; }
|
||||
busy.value = "";
|
||||
}
|
||||
const pickFile = (accept, cb) => {
|
||||
const inp = document.createElement("input"); inp.type = "file"; inp.accept = accept;
|
||||
inp.onchange = () => inp.files[0] && upload(inp.files[0], cb); inp.click();
|
||||
};
|
||||
async function progress(reset = false) {
|
||||
if (!plan.video || plan.run !== "queue") return;
|
||||
try {
|
||||
const r = await api.fetchApi("/bfs/shotloop/progress", { method: "POST", body: JSON.stringify({ plan: { ...plan }, plan_raw: io.getPlan(), reset }) });
|
||||
const j = await r.json(); if (!j.error) { prog.done = j.done; prog.count = j.count || active.value.length; }
|
||||
} catch { /* ignore */ }
|
||||
}
|
||||
|
||||
// ---- editing
|
||||
const setVideo = v => {
|
||||
plan.video = v; plan.bounds = []; plan.segs = [];
|
||||
if (v) {
|
||||
// A new source starts as one manageable clip. The user can widen the range or ask for an automatic split later;
|
||||
// do not surprise them with a wall of detected shots on first load.
|
||||
plan.mode = "single";
|
||||
plan.start_f = 0;
|
||||
plan.end_f = Math.round(15 * plan.fps);
|
||||
plan.start_s = 0; plan.end_s = 0; plan.max_total_s = 0;
|
||||
tab.value = "video";
|
||||
}
|
||||
save(); analyze();
|
||||
};
|
||||
const splitAt = f => {
|
||||
const { lo, hi } = range.value; f = Math.round(f);
|
||||
if (f <= lo || f >= hi || plan.bounds.includes(f)) return;
|
||||
const i = segs.value.findIndex(s => f > s.start && f < s.end);
|
||||
plan.bounds = [...plan.bounds, f].sort((a, c) => a - c);
|
||||
plan.segs.splice(i + 1, 0, { ...(plan.segs[i] || {}) });
|
||||
sel.value = i + 1; save();
|
||||
};
|
||||
const mergeNext = i => {
|
||||
const s = segs.value[i]; if (!s || i >= segs.value.length - 1) return;
|
||||
plan.bounds = plan.bounds.filter(b => b !== s.end); plan.segs.splice(i + 1, 1); save();
|
||||
};
|
||||
// Delete / Backspace on a selected shot: remove the cut at its start (merge into the previous shot;
|
||||
// the first shot merges with the next one)
|
||||
const removeCut = i => {
|
||||
const S = segs.value, s = S[i]; if (!s || S.length < 2) return;
|
||||
if (i === 0) { mergeNext(0); sel.value = 0; return; }
|
||||
plan.bounds = plan.bounds.filter(b => b !== s.start); plan.segs.splice(i, 1);
|
||||
sel.value = i - 1; save();
|
||||
};
|
||||
const select = i => { if (i >= 0 && i < segs.value.length) sel.value = i; };
|
||||
const onKey = e => {
|
||||
const t = e.target, tag = (t?.tagName || "").toLowerCase();
|
||||
if (tag === "input" || tag === "textarea" || tag === "select" || t?.isContentEditable) return;
|
||||
if (e.key === "ArrowLeft" || e.key === "ArrowRight") { // previous / next shot
|
||||
e.preventDefault(); e.stopPropagation(); select(sel.value + (e.key === "ArrowLeft" ? -1 : 1)); return;
|
||||
}
|
||||
if (e.key !== "Delete" && e.key !== "Backspace") return;
|
||||
e.preventDefault(); e.stopPropagation(); // keep ComfyUI from deleting the node
|
||||
removeCut(sel.value);
|
||||
};
|
||||
const setMeta = (i, k, v) => { meta(i)[k] = v; save(); };
|
||||
const setMode = (i, mode) => { const m = MODES.find(x => x[0] === mode); Object.assign(meta(i), { crop: m[2], inpaint: m[3], paste: m[4] }); save(); };
|
||||
const drag = (k, ev) => {
|
||||
ev.preventDefault(); ev.stopPropagation();
|
||||
const box = tlEl.value.getBoundingClientRect();
|
||||
const move = e => {
|
||||
const x = e.clientX - box.left + tlEl.value.scrollLeft;
|
||||
const f = Math.round(x / pxPerFrame.value);
|
||||
const lo = (plan.bounds[k - 1] ?? range.value.lo) + 1, hi = (plan.bounds[k + 1] ?? range.value.hi) - 1;
|
||||
plan.bounds[k] = Math.max(lo, Math.min(hi, f));
|
||||
hover.value = { frame: plan.bounds[k], x };
|
||||
};
|
||||
const up = () => { window.removeEventListener("pointermove", move); window.removeEventListener("pointerup", up); save(); };
|
||||
window.addEventListener("pointermove", move); window.addEventListener("pointerup", up);
|
||||
};
|
||||
const frameAt = e => { const box = tlEl.value.getBoundingClientRect(); const x = e.clientX - box.left + tlEl.value.scrollLeft; return { frame: Math.max(0, Math.min(n.value - 1, Math.round(x / pxPerFrame.value))), x }; };
|
||||
const thumbFor = f => { const t = an.value?.thumbs; if (!t?.length) return ""; let b = t[0]; for (const x of t) { if (x.f <= f) b = x; else break; } return b.src; };
|
||||
|
||||
// ---- preview player
|
||||
const vid = ref(null);
|
||||
const play = reactive({ mode: "", idx: -1, frame: 0, loop: false });
|
||||
const stopAt = () => { const s = segs.value[play.idx]; return s ? s.end / plan.fps : Infinity; };
|
||||
const tick = () => {
|
||||
const v = vid.value; if (!v) return;
|
||||
play.frame = Math.floor(v.currentTime * plan.fps + 1e-3);
|
||||
if (!play.mode) return;
|
||||
if (play.mode === "all") {
|
||||
const i = segs.value.findIndex(s => play.frame >= s.start && play.frame < s.end);
|
||||
if (i >= 0 && skipWhy(segs.value[i])) { // jump over shots that will not run
|
||||
const nxt = segs.value.slice(i + 1).find(s => !skipWhy(s));
|
||||
if (nxt) v.currentTime = nxt.start / plan.fps + 1e-3; else { v.pause(); play.mode = ""; }
|
||||
return;
|
||||
}
|
||||
if (i >= 0) play.idx = i;
|
||||
if (play.frame >= n.value) { v.pause(); play.mode = ""; }
|
||||
} else if (v.currentTime >= stopAt() - 1e-3) {
|
||||
const s = segs.value[play.idx];
|
||||
if (play.loop && s) v.currentTime = s.start / plan.fps + 1e-3; else { v.pause(); play.mode = ""; v.currentTime = stopAt() - 0.5 / plan.fps; }
|
||||
}
|
||||
};
|
||||
let raf = 0;
|
||||
const loop = () => { tick(); raf = requestAnimationFrame(loop); };
|
||||
const playShot = i => {
|
||||
const v = vid.value, s = segs.value[i]; if (!v || !s) return;
|
||||
sel.value = i; play.idx = i; play.mode = "shot";
|
||||
v.currentTime = s.start / plan.fps + 1e-3; v.play();
|
||||
};
|
||||
const playAll = () => {
|
||||
const v = vid.value; if (!v) return;
|
||||
play.mode = "all"; const first = segs.value.find(s => !skipWhy(s)) || segs.value[0];
|
||||
play.idx = segs.value.indexOf(first); v.currentTime = (first?.start || 0) / plan.fps + 1e-3; v.play();
|
||||
};
|
||||
const stop = () => { vid.value?.pause(); play.mode = ""; };
|
||||
const seek = f => { const v = vid.value; if (v) { v.currentTime = f / plan.fps + 1e-3; play.frame = f; } };
|
||||
|
||||
// ---- queue loop events
|
||||
const onProg = e => { prog.done = e.detail.done; prog.count = e.detail.count; };
|
||||
const onStatus = e => { if (busy.value || modal.busy) Object.assign(status, e.detail || {}); };
|
||||
let clock = 0;
|
||||
const onVlm = e => { if (e.detail?.video === plan.video) mergeSug(e.detail.segs); };
|
||||
const onNext = e => { onProg(e); if (plan.run === "queue" && plan.auto_continue) setTimeout(() => app.queuePrompt(0, 1), 300); };
|
||||
onMounted(() => {
|
||||
load(); refreshFiles(); analyze(); progress(); raf = requestAnimationFrame(loop);
|
||||
if (!plan.video) tab.value = "video";
|
||||
api.addEventListener("bfs-shotloop-progress", onProg); api.addEventListener("bfs-shotloop-next", onNext);
|
||||
api.addEventListener("bfs-shotloop-vlm", onVlm);
|
||||
api.addEventListener("bfs-shotloop-status", onStatus);
|
||||
clock = setInterval(() => { if (busy.value || modal.busy) now.value = Date.now(); }, 500);
|
||||
});
|
||||
onBeforeUnmount(() => { cancelAnimationFrame(raf); api.removeEventListener("bfs-shotloop-progress", onProg); api.removeEventListener("bfs-shotloop-next", onNext); api.removeEventListener("bfs-shotloop-vlm", onVlm); api.removeEventListener("bfs-shotloop-status", onStatus); clearInterval(clock); });
|
||||
io.expose({ reload: () => { load(); analyze(); progress(); } });
|
||||
|
||||
// ---- references
|
||||
// the last 10 references picked, remembered across workflows (per browser)
|
||||
const RECENT_KEY = "bfs.shotloop.recentRefs";
|
||||
const isColorValue = name => typeof name === "string" && /^#[0-9a-f]{6}$/i.test(name);
|
||||
const savedRecent = store.get(RECENT_KEY, []);
|
||||
const recent = ref((Array.isArray(savedRecent) ? savedRecent : []).filter(name => typeof name === "string" && !isColorValue(name)));
|
||||
if (recent.value.length !== (Array.isArray(savedRecent) ? savedRecent.length : 0)) store.set(RECENT_KEY, recent.value);
|
||||
const remember = name => {
|
||||
if (!name || typeof name !== "string" || isColorValue(name)) return;
|
||||
recent.value = [name, ...recent.value.filter(x => x !== name)].slice(0, 10);
|
||||
store.set(RECENT_KEY, recent.value);
|
||||
};
|
||||
// target: shot index or "global"; field: ref | ref2 (global_ref | global_ref2 for "global")
|
||||
const setRef = (target, field, name) => {
|
||||
if (isColorValue(name)) return; // colors belong to Color Guide settings, never to image references
|
||||
if (target === "global") plan[field] = name; else meta(target)[field] = name;
|
||||
remember(name); save();
|
||||
};
|
||||
const uploadRef = (target, field) => pickFile("image/*", name => setRef(target, field, name));
|
||||
const clearRecent = () => { recent.value = []; store.set(RECENT_KEY, []); };
|
||||
// everything: the recent list AND every reference this plan uses (global, per shot, per person in the cast)
|
||||
const clearAllRefs = () => {
|
||||
if (!confirm("Remove every reference picture from this plan (global, every shot and every person) and forget the recent list?")) return;
|
||||
clearRecent();
|
||||
plan.global_ref = ""; plan.global_ref2 = "";
|
||||
plan.segs.forEach(m => { if (m) { m.ref = ""; m.ref2 = ""; } });
|
||||
Object.values(plan.cast || {}).forEach(cc => { if (cc) { cc.ref = ""; cc.ref2 = ""; } });
|
||||
save();
|
||||
};
|
||||
const usedRefs = computed(() => {
|
||||
const set = new Set([...recent.value, plan.global_ref, plan.global_ref2, ...plan.segs.flatMap(m => [m?.ref, m?.ref2])].filter(Boolean));
|
||||
return [...set].slice(0, 10);
|
||||
});
|
||||
|
||||
// everything the views need
|
||||
const c = {
|
||||
api, plan, files, an, cuts, detectorUsed, size, busy, error, status, busySince, now, upPct, sel, zoom, hover, prog, tlEl,
|
||||
stats, people, maskPrev, showMasks, vlmSug, modal, segPeople, targetCrop, tab, copyOpts, COPY_FIELDS, vid, play, recent, clearRecent, clearAllRefs,
|
||||
n, range, setRange, maxLen, segs, active, pxPerFrame, tlWidth, filtersOn, refSets, usedRefs, checks, tour,
|
||||
MODES, modeOf, modeName, viewUrl, save, meta, setMeta, setMode, setPlan, whoIn, castOf, linked, castRef, personOf, statFor, skipWhy, refOf, promptOf, checksFor,
|
||||
refreshFiles, analyze, autoSplit, analyzeContent, findPeople, setCast, describeTarget, previewMask, setMask, setMaskCfg,
|
||||
openPoints, savePoints, savePointsAll, saveTrack, clearTrack, pointsTo, clearMask, copyFrom, analyseVLM, describeRefs, setDetail, setVlmCfg,
|
||||
applySug, applyAllSug, setCastOpt, setGlobalMaskVideo, setFilter, pickFile, progress, setVideo, splitAt, mergeNext, removeCut, select, drag,
|
||||
frameAt, thumbFor, playShot, playAll, stop, seek, setRef, uploadRef,
|
||||
};
|
||||
|
||||
return () => {
|
||||
const S = segs.value, N = n.value, fps = plan.fps;
|
||||
const warns = checks.value.filter(x => x.lvl === "warn").length;
|
||||
const guideUrl = new URL("./docs/BFSShotPlanner.md", import.meta.url).href;
|
||||
const header = h("div", { class: "hdr" }, [
|
||||
h("span", { class: "ttl" }, "🎬 Shot Planner"),
|
||||
an.value ? pill(`${active.value.length}/${S.length} shots · ${((range.value.hi - range.value.lo) / fps).toFixed(1)}s`
|
||||
+ (range.value.lo > 0 || range.value.hi < N ? " (range)" : ""), "ok") : null,
|
||||
warns ? h("span", { class: "pill warn", style: "cursor:pointer", title: "open the checks (Run tab)", onClick: () => { tab.value = "run"; } }, `⚠ ${warns}`) : null,
|
||||
h("span", { class: "grow" }),
|
||||
h("label", { class: "langpick", title: t("Language") }, [
|
||||
h("span", { class: "hint" }, "🌐"),
|
||||
h("select", { value: uiLanguage.value, "aria-label": t("Language"), onChange: e => setUiLanguage(e.target.value) }, [
|
||||
h("option", { value: "en" }, "English"), h("option", { value: "pt" }, "Português"),
|
||||
h("option", { value: "es" }, "Español"), h("option", { value: "zh" }, "中文"),
|
||||
]),
|
||||
]),
|
||||
pill(plan.run === "queue" ? "queue loop" : "auto loop"),
|
||||
h("button", { title: t("Shot Planner tour"), onClick: () => { tour.value = 1; tab.value = "video"; } }, t("🧭 Tour")),
|
||||
h("button", { title: "Open the full guide (every setting explained, with examples)", onClick: () => showGuide(guideUrl) }, "📖 Guide"),
|
||||
]);
|
||||
const counts = { shots: S.length || null, people: people.value.length || null };
|
||||
const tabs = h("div", { class: "tabs" }, TABS.map(([k, label]) => h("button", {
|
||||
class: ["tab", tab.value === k && "on"], disabled: k !== "video" && !an.value, onClick: () => { tab.value = k; } }, [
|
||||
label, counts[k] ? h("span", { class: "n" }, counts[k]) : null, k === "run" && warns ? h("span", { class: "dot", title: `${warns} warning(s)` }) : null,
|
||||
])));
|
||||
const views = { video: videoTab, shots: shotsTab, people: peopleTab, prompts: promptsTab, run: runTab };
|
||||
const cur = an.value || tab.value === "video" ? tab.value : "video";
|
||||
return h("div", { class: "bsl root", ref: rootEl, tabindex: 0, onKeydown: onKey }, [
|
||||
header, tabs,
|
||||
busy.value ? busyBar(c, busy.value) : null,
|
||||
error.value ? h("div", { class: "err" }, [h("span", { class: "grow" }, error.value), h("button", { class: "ghost", onClick: () => { error.value = ""; } }, "✕")]) : null,
|
||||
an.value ? timelineCard(c) : null,
|
||||
views[cur](c),
|
||||
pointsModal(c),
|
||||
tourModal({ tour, tab }),
|
||||
]);
|
||||
};
|
||||
}
|
||||
|
||||
app.registerExtension({
|
||||
name: "BFSNodes.ShotPlanner",
|
||||
async nodeCreated(node) {
|
||||
if (node.comfyClass !== "BFSShotPlanner") return;
|
||||
styles();
|
||||
const planWidget = node.widgets?.find(w => w.name === "plan");
|
||||
if (planWidget) { // the JSON plan stays a widget (saved with the workflow) but is never shown, in both node renderers
|
||||
planWidget.type = "hidden"; planWidget.hidden = true; planWidget.computeSize = () => [0, -4];
|
||||
planWidget.options = { ...(planWidget.options || {}), hidden: true };
|
||||
}
|
||||
const host = document.createElement("div");
|
||||
// the panel is absolutely positioned inside the host and scrolls by itself, so switching tabs never resizes the
|
||||
// node (Vue nodes mode sizes widgets by their content); min-width keeps it from collapsing to its content width
|
||||
host.style.cssText = "position:relative;width:100%;height:100%;min-width:900px;min-height:640px";
|
||||
node.addDOMWidget("bfs_shot_planner", "div", host, { serialize: false, hideOnZoom: false });
|
||||
const handle = {};
|
||||
createApp({
|
||||
setup: () => Panel({
|
||||
getPlan: () => planWidget?.value ?? "{}",
|
||||
setPlan: v => { if (planWidget) planWidget.value = v; node.graph?.setDirtyCanvas(true); },
|
||||
expose: o => Object.assign(handle, o),
|
||||
}),
|
||||
}).mount(host);
|
||||
if (planWidget) {
|
||||
planWidget.options = planWidget.options || {};
|
||||
planWidget.options.serialize = true;
|
||||
planWidget.serializeValue = () => planWidget.value;
|
||||
}
|
||||
const prev = node.onConfigure;
|
||||
node.onConfigure = function () {
|
||||
const r = prev?.apply(this, arguments);
|
||||
setTimeout(() => handle.reload?.(), 0);
|
||||
if (this.size[0] < MIN_W) this.setSize([MIN_W, Math.max(this.size[1], MIN_H)]); // older workflows: too narrow
|
||||
return r;
|
||||
};
|
||||
node.size = [Math.max(node.size[0], DEFAULT_W), Math.max(node.size[1], MIN_H)];
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,182 @@
|
||||
# BFS H3 Duet / Side Panel
|
||||
|
||||
Training-free split-screen ("duet") generation for MiniMax H3.
|
||||
|
||||
**Credits:** the initial idea for this H3 node came from TSC's latent-pin duet. BFS had already used the
|
||||
same principle on LTX (a green side panel holding the reference) and implemented the virtual sidecar
|
||||
approach (reference tokens placed beside the frame in RoPE). New here: the **shifted RoPE layout** (the
|
||||
video keeps the RoPE positions of a render without the panel and the panel sits past its edge, with an
|
||||
optional gap), dynamic references, task prompts and the shot-loop node.
|
||||
A panel is pinned next to the video in the same canvas with a noise mask. H3 keeps the panel
|
||||
exactly (it treats it like its own condition rows) and generates the video next to it, in sync.
|
||||
The panel is cut off before decoding, so it never reaches the output.
|
||||
|
||||
## Nodes
|
||||
|
||||
| node | what it does |
|
||||
|---|---|
|
||||
| **BFS H3 Duet** | Everything in one node: native references + prompt, the pinned panel, an optional latent guide, sampling (euler / beta / CFG 1), decode and crop. Leave `panel` empty for a plain guided render. Outputs the generated video, its audio, the whole canvas (to check the sync) and the layout sentence for the prompt. |
|
||||
| **BFS Shot H3 Conditioning** (duet option) | The shot loop's H3 conditioning can pin the shot's own clip as a duet panel too: *duet* = canvas or shifted RoPE (route the model through the node for the shift). BFS Shot Join cuts the panel off by itself, so the rest of the workflow (sampler, decode) stays as it is. Optional prompt writer: pick a *task* (character swap, style, setting, appearance, lighting / weather, custom) and an *instruction*; with a VLM on its *vlm* input it writes each shot's duet prompt from the shot and its references, otherwise a template for the task (*planner prompt*, the default, keeps the planner's prompt). The written text comes out on *prompt*. Optional *setting_ref* (TSC's tip): one more reference, the shot's middle frame with the person covered in TV static, so the model sees the place in full detail; it is the last `<Picture n>` (write `{setting}` for its tag, otherwise a sentence is added to subject_definitions). |
|
||||
| **BFS Shot H3 Duet** | One shot of the shot loop (Planner -> this -> BFS Shot Join): *duet* pins the shot's own clip beside the video (no LoRA needed), *guide* puts the shot on the generated frames as a latent guide (for body-swap LoRAs), *duet + guide* does both. Write `{layout}` in the prompt to insert the split-screen sentence. |
|
||||
| **BFS H3 Duet Conditioning (prompt writer)** | Conditioning only, for your own sampler: references + the pinned panel + the prompt. Write your prompt, or connect a VLM (`vlm`, a CLIPLoader with a Qwen3-VL text encoder) and it writes the prompt in the duet format from **task** + **instruction**, looking at the clip and the references (it fills a few fields; the node assembles the six sections). Outputs positive, latent (with the panel's mask), model (RoPE shift applied in shifted mode), panel_info and the prompt used. Chain: → BasicGuider / SamplerCustomAdvanced → **BFS H3 Side Panel Crop** (latent) → VAE Decode. |
|
||||
| **BFS H3 Side Panel** | Only the canvas: takes the conditioning and AV latent from Reference to Video, adds the pinned strip (and the guide), and returns them for your own sampler. Guides already added with Add Guide for MiniMax H3 are moved onto the canvas. |
|
||||
| **BFS H3 Side Panel Crop** | Removes the strip from the sampled latent (before VAE Decode) or from decoded images. |
|
||||
|
||||
## First time?
|
||||
|
||||
Click **❓ How to use** on the node: a window with drawings of every setting (panel side, size, contain/cover,
|
||||
gap, hold, canvas vs shifted RoPE) and how to write the task, instruction and prompt. The node shows only the main
|
||||
settings; *fit, gap, hold, rope_mode, rope_gap, ref_image_size, decode_canvas, guide_frame_idx* are under the
|
||||
node's advanced settings, with defaults that work.
|
||||
|
||||
## What to pin
|
||||
|
||||
- **The source clip** (TSC's duet): the new character copies its motion, camera and cuts frame for
|
||||
frame. Give the new character with `<Picture 1>` (and the place with `<Picture 2>`). The pinned clip
|
||||
has no tag: the prompt names its region by where it is and what is seen there (`{layout}` / `{panel_region}`). See TSC's
|
||||
prompting guide: never describe the source performer, restate the identity in every shot.
|
||||
- **A reference image**: a static picture of the person next to the video.
|
||||
|
||||
## Tasks
|
||||
|
||||
Use it for any edit that keeps the source's motion, camera and timing. With an empty prompt box the node
|
||||
writes the prompt from **task** + **instruction** (the `prompt` output shows what it sent):
|
||||
|
||||
| task | instruction example | panel_noise |
|
||||
|---|---|---|
|
||||
| character swap | (empty, or the new person's look) + the person's pictures as references | 0 |
|
||||
| style | `a 1990s anime cel style` | 0.1-0.2 |
|
||||
| setting | `a sunny beach at sunset, waves rolling behind her` | 0 |
|
||||
| appearance | `an elderly woman with short grey hair` | 0.1 |
|
||||
| lighting / weather | `night, lit by pink and blue neon signs` | 0 |
|
||||
|
||||
It does not change the motion or the camera: the panel makes the video copy them in sync.
|
||||
|
||||
The task prompt is a **draft**: it cannot see the video, so it is generic. In tests, a prompt written for the
|
||||
clip (the example below) kept the room, framing and sync; the drafts got the change only partly (the style
|
||||
barely changed, the setting and light lost the framing, the swap took the reference's backdrop). Start from
|
||||
the `prompt` output and describe the clip.
|
||||
|
||||
References work like the official Reference to Video node: up to 9 images (`<Picture n>`), 3 videos
|
||||
(`<Video n>`, each with its soundtrack) and 3 audios (`<Audio n>`), new slots appear as you connect them.
|
||||
|
||||
## Prompt
|
||||
|
||||
TSC's newer strategy (2026-10): **H3 has no idea what is pinned and what is noised.** It reads one description of
|
||||
one video, so the prompt describes the finished split screen as plain footage. Never write "kept footage", "kept
|
||||
exactly", "generated", "preserved", "in sync with", "seamlessly": they describe nothing on screen and crowd out the
|
||||
action.
|
||||
|
||||
- **No tag and no retention line for the pinned clip.** It is not a reference input, only latents in the canvas. Name
|
||||
its region by where it is plus what is seen there, in one clause: "the top quarter shows a man dancing in a small
|
||||
bedroom". `{panel_region}` / `{video_region}` become the region names for the current side and size ("the top
|
||||
quarter", "the bottom three quarters"), `{layout}` the whole split-screen sentence.
|
||||
- **Write the other region as its own scene**, doing the same movements at the same moments, in a real place with
|
||||
concrete details.
|
||||
- **Spell out the motion** beat by beat, with times for the big moments (cuts, strikes, jumps, a line). The pinned
|
||||
half alone does not carry the moves. Times are on the video's clock (the pinned clip starts at frame 0 here).
|
||||
- **Soundscape** = the sounds of the new scene.
|
||||
- Cuts: `[Shot 2] At MM:SS.mmm, both halves cut together to ...` with the new framing.
|
||||
|
||||
The VLM writer and the template filler follow these rules, and a scrub removes the old meta words from what they
|
||||
return. Template (fill the `{...}`, or let the VLM fill them):
|
||||
|
||||
```
|
||||
subject_definitions:
|
||||
<Subject 1> is {name or plain description} whose appearance comes from <Picture 1>: {concrete look words}, wearing {outfit piece by piece}.
|
||||
|
||||
summary:
|
||||
[reference generation] The target video is a split screen: {panel_region} shows {what is seen in the source, one clause}, and {video_region} shows <Subject 1> doing the same movements at the same moments, in {place}.
|
||||
|
||||
retention_analysis:
|
||||
<Subject 1> (appears in [Shot 1]): fully_preserved - {key look words} are retained.
|
||||
|
||||
detailed_description:
|
||||
The target video is in a {realistic / 3D CG / anime} style, as {the medium of the source clip}. A split screen divided by a thin straight {vertical / horizontal} line: {panel_region} shows {what is seen there, one clause}; {video_region} shows <Subject 1> performing the same movements at the same moments, in {place with concrete details}.
|
||||
|
||||
[Shot 1] {Camera distance, angle and movement as the source shows it}. <Subject 1>, the face from <Picture 1>, {what they do, one sentence per real action}. At {00:01.500} {big beat}.
|
||||
|
||||
overall_soundscape:
|
||||
{The sounds of the new scene and when they happen.}
|
||||
|
||||
non_diegetic_music:
|
||||
None.
|
||||
```
|
||||
|
||||
Example (character swap, source clip pinned on the left half, one face picture and one full-body picture):
|
||||
|
||||
```
|
||||
subject_definitions:
|
||||
<Subject 1> is a young woman whose appearance comes from <Picture 1> and <Picture 2>: fair skin, a narrow oval face, grey-green eyes, light brown hair in a low ponytail, wearing a white long-sleeved top under a navy denim apron.
|
||||
|
||||
summary:
|
||||
[reference generation] The target video is a split screen: the left half shows a man talking to a phone camera in a bedroom, and the right half shows <Subject 1> doing the same movements at the same moments, in a sunny kitchen with white tiles.
|
||||
|
||||
retention_analysis:
|
||||
<Subject 1> (appears in [Shot 1], [Shot 2]): fully_preserved - her face, ponytail, white top and navy apron are retained.
|
||||
|
||||
detailed_description:
|
||||
The target video is in a realistic style, as handheld vertical smartphone footage under soft daylight. A split screen divided by a thin straight vertical line: the left half shows a man talking to a phone camera in a bedroom; the right half shows <Subject 1> performing the same movements at the same moments, in a sunny kitchen with white tiles and a window behind her.
|
||||
|
||||
[Shot 1] A medium close-up at chest height, the phone steady at eye level. <Subject 1>, her face from <Picture 1> with grey-green eyes and soft pink lips, talks to the camera with small nods. At 00:01.800 she laughs and tilts her head back.
|
||||
|
||||
[Shot 2] At 00:03.708, both halves cut together to a closer framing. <Subject 1>, her face from <Picture 1>, the white sleeves and apron straps visible, raises one open hand beside her mouth and smiles.
|
||||
|
||||
overall_soundscape:
|
||||
Her voice speaking to the camera, a kettle starting to hiss in the kitchen at 00:02.500.
|
||||
|
||||
non_diegetic_music:
|
||||
None.
|
||||
```
|
||||
|
||||
## Same-world swaps: static and setting picture (1.84.0)
|
||||
|
||||
TSC's two fixes for a swap in the same room with the same clothes, without a VLM:
|
||||
|
||||
- **`panel_static`** (`off` / `face` / `person`): SAM 3 tracks the face (or the whole person) of the pinned clip and
|
||||
covers it in black-and-white TV static on every frame. The generated half can no longer copy the source face, so the
|
||||
identity comes from the reference picture; motion, framing and camera still come from the panel. The static only
|
||||
exists in the panel, which is cropped off, so it never shows in the output. `face` keeps the body (best when the
|
||||
outfit should stay); `person` hides the whole performer (best when the build and outfit should change).
|
||||
- **`setting`** + **BFS Setting Picture**: one frame of the clip with the person in static. Plugged into `setting`, it
|
||||
becomes the last `<Picture n>` and the prompt gets *"<Picture n> shows the setting, the same place as the original performance
|
||||
in full detail; the noise patch in it is where <Subject 1> stands."* (or write `{setting}` to place its tag yourself).
|
||||
This stops the background and framing from being taken from the character picture.
|
||||
|
||||
## Prompt from a VLM (optional, 1.85.0)
|
||||
|
||||
Plug a Qwen3-VL CLIP (the same CLIPLoader as `clip` works) into `vlm`:
|
||||
|
||||
- **prompt empty**: the VLM writes it from the task + instruction (TSC's six sections; with a `setting` picture the
|
||||
place becomes `<Subject 2> ... from <Picture n>`).
|
||||
- **prompt with `{...}` fields** (TSC's template): the VLM fills only the fields, from the clip's frames, the
|
||||
reference pictures and the setting picture; every other word stays. `{layout}`, `{setting}`, `{target}` and
|
||||
`{details}` are filled by the node, not the VLM.
|
||||
- **prompt without fields**: used as written (the VLM is not called).
|
||||
|
||||
Without `vlm`, a prompt that still has `{...}` fields logs a warning. The prompt actually used is the `prompt` output
|
||||
and is printed under the comparison.
|
||||
|
||||
## Settings
|
||||
|
||||
| setting | effect |
|
||||
|---|---|
|
||||
| position / size | side of the strip and its size relative to the video (1.0 = two equal halves) |
|
||||
| fit | contain (default: the whole clip, smaller, with grey around, nothing cropped), cover (fill and crop), stretch |
|
||||
| gap | grey separator between panel and video, in 32 px patches |
|
||||
| panel_noise | 0 pins the panel exactly; 0.05-0.15 loosens it when the result copies too much of it |
|
||||
| hold | pin the panel for the whole clip, or only its first latent frame |
|
||||
| guide | optional aligned latent guide in the video area (what the body-swap LoRAs use) |
|
||||
| rope_mode | *canvas*: panel and video share one wide grid (TSC). *shifted*: the video keeps the RoPE positions of a render without the panel, and the panel sits past its edge |
|
||||
| (prompt in shifted mode) | With *shifted*, `{layout}` becomes empty: do not describe a split screen there, or the model splits the video itself. *canvas* keeps the split-screen wording and follows the framing more closely (recommended). |
|
||||
| rope_gap | *shifted* only: empty RoPE steps (2x2 patches) between video and panel; the panel moves away in position without any pixels in between. Keep it small against the video width: at 320 px wide (10 patches) a gap of 8 made the model draw its own split screen; 0 works everywhere |
|
||||
|
||||
## Notes from tests
|
||||
|
||||
- Source clip pinned, no LoRA, no guide (TSC's setup): the new person follows the clip's gestures,
|
||||
timing and cuts, in the same room, and on-screen captions disappear.
|
||||
- With a body-swap LoRA and a latent guide, the guide alone kept the room better than guide + panel:
|
||||
the panel pulled the background toward the reference picture's backdrop. The shifted RoPE (gap 0 or 8)
|
||||
did not change that, so it comes from the pinned panel itself, not from the canvas geometry.
|
||||
- Duet with the shifted RoPE, gap 0 and gap 8: same quality and sync as the canvas layout. Use the panel without a
|
||||
guide, or the guide without a panel, until a LoRA is trained for both.
|
||||
@@ -0,0 +1,355 @@
|
||||
# BFS Sampler (crop · mask · loop)
|
||||
|
||||
Runs a guide-driven LTX IC-LoRA and adds three things around it, each optional.
|
||||
Connect only what you need — the node degrades to whatever is wired.
|
||||
|
||||
| what you connect | what you get |
|
||||
|---|---|
|
||||
| `guide_video` | the LoRA over the whole clip, one pass |
|
||||
| `+ identity_image` | for a LoRA that takes a reference (head swap, identity transfer) |
|
||||
| `+ subject_mask` | only the masked region is denoised |
|
||||
| `+ crop_mode` | the edit runs inside a box around the subject, then is pasted back |
|
||||
| `+ temporal_tile_size` | long clips sampled in chunks with overlap |
|
||||
| `+ auto_config` | crop, mask and feather amounts measured off the subject itself |
|
||||
|
||||
`identity_image` is optional as of 1.38.0. A head-swap LoRA wants one; an
|
||||
instruction-edit, restyle or sharpening LoRA works from the guide alone, and
|
||||
with the slot empty it is simply not packed — the guide becomes the only
|
||||
reference. Everything else in the node is indifferent to which kind it is.
|
||||
|
||||
The LoRA is never asked to understand a mask or a crop. It keeps doing its one
|
||||
job: the guide rides on `source_id 1`, the identity on `source_id 2`. Everything
|
||||
else here is plumbing around it.
|
||||
|
||||
## `auto_config`: let the mask set the amounts
|
||||
|
||||
Every amount in this node is in pixels — `mask_grow`, `mask_blur`,
|
||||
`uncrop_feather`, and the latent dilation — and a pixel only means something
|
||||
relative to how big the head is in the frame. The same `mask_grow: 8` is
|
||||
generous slack on a head 90 px wide and invisible on one 900 px wide. That
|
||||
mismatch is what leaves a visible seam when the crop goes back into the frame:
|
||||
the amounts were tuned on one shot and the next shot has the subject at a
|
||||
different distance.
|
||||
|
||||
Turn `auto_config` on and the node measures the subject in `subject_mask` — its
|
||||
width, its height, how far it travels, how much its size changes, how fast it
|
||||
moves between frames — and sets all of it from those numbers:
|
||||
|
||||
| knob | derived as | why |
|
||||
|---|---|---|
|
||||
| `crop_mode` | `off` when the head is ≥45% of the frame width; `zoomed` when its size changes >35%; `tracked` when it travels more than a quarter of the box; else `combined` | a crop buys resolution the head does not already have, and nothing else |
|
||||
| `crop_scale` | 1.8, clamped so the box still fits in the frame | head plus neck and shoulders — a face-tight crop is a framing the LoRA never saw |
|
||||
| `mask_grow` | 6% of the head's **width**, plus the capped headroom slack — **per side**: up ≫ sideways ≫ down | hair overflows upward and sideways; growing down only eats the neck and collar, which stay the guide's |
|
||||
| `mask_blur` | 3% of the head's width | paste-back softness (the denoise mask is binarised anyway) |
|
||||
| `uncrop_feather` | half the margin between mask and crop border, floored | **the seam fix**: a ramp wider than that margin fades the head itself |
|
||||
| `latent_mask_dilate` | left at 0 | the pixel grow already carries the slack, and the pixel→latent reduction is a MAX, so any cell the mask touches is editable already — adding cells here stacks a whole 32 px on every side |
|
||||
| `latent_mask_dilate_frames` | 1 when the head moves more than 15% of its width between frames | a fast head needs slack along time too |
|
||||
|
||||
### The mask is the old head, not the new one
|
||||
|
||||
This is the failure the mode is really for. `subject_mask` marks the head that
|
||||
is being *replaced*. The reference head can be wider, or have more hair, or a
|
||||
taller cut — and wherever it lands outside the mask, the swap is cut off at the
|
||||
mask edge. Growing the mask by a fixed 6% does not fix that; it just moves the
|
||||
clip.
|
||||
|
||||
Two things are done about it. The reference's **proportions** are measured —
|
||||
`identity_image`'s aspect against the mask's own — and used to decide *where*
|
||||
the slack goes: a relatively wider reference gets it at the sides, a taller one
|
||||
above. The reference's **size** cannot be measured at all: a cropped head
|
||||
carries no scale, and nothing in the graph says whether that crop is a 200 px
|
||||
head or a 900 px one. That part is `identity_headroom`, default 1.15 — "the new
|
||||
head may be up to 15% bigger than the old one". Raise it for big hair or a
|
||||
visibly larger head, drop it to 1.0 when the two heads match.
|
||||
|
||||
Growth is never symmetric: down is held at the base 6% no matter what, because
|
||||
the neck and collar below the jaw have to stay the guide's own pixels.
|
||||
|
||||
Every slack is a fraction of the head's **width** and capped there (25% sideways,
|
||||
30% up). Width is the stable dimension of a head mask — height swings with how
|
||||
much neck the mask happened to take — and scaling the upward slack by height,
|
||||
as 1.37.0 did, produced holes far larger than the head, with the new head
|
||||
generated floating inside the regenerated area. Fixed in 1.37.1.
|
||||
|
||||
It needs `subject_mask`; without one there is nothing to measure and the widget
|
||||
values are kept. The widgets it overrides stay visible but are ignored, and the
|
||||
`debug` output prints exactly what it chose:
|
||||
|
||||
```
|
||||
auto: head 120x160px in 1920x1080 (6% of the width), seen in 73/73 frames;
|
||||
reference aspect 1.00 vs mask 0.75 (tilt 1.33), headroom 1.15 -> crop tracked @ 1.8
|
||||
(the subject travels 1296px, more than a quarter of the box); grow 19px sideways,
|
||||
32px up, 7px down, blur 4px, feather 14px, latent dilate 1 cell(s), 0 frame(s)
|
||||
```
|
||||
|
||||
Read that line before trusting it. It is a starting point measured from your own
|
||||
shot, not a guarantee — a mask that leaks onto the shoulders makes the "head"
|
||||
wider and every amount grows with it.
|
||||
|
||||
## The mask is native inpainting, not a LoRA feature
|
||||
|
||||
The mask becomes `latent["noise_mask"]`, which the guider consumes as
|
||||
`denoise_mask` — ComfyUI's own inpainting path. Pixels outside the mask keep the
|
||||
guide's own content, so **the original face stays visible to the model**. This is
|
||||
deliberately different from a masked-hole recipe, where the region is painted out
|
||||
before the model sees it.
|
||||
|
||||
When a mask is connected the node seeds the sampling latent with the
|
||||
**VAE-encoded guide**, not with the empty latent you wired in — inpainting keeps
|
||||
the initial latent outside the mask, and an empty one leaves grey where the video
|
||||
should be. The connected latent still sets the size (and carries the audio
|
||||
stream); its video content is replaced.
|
||||
|
||||
Masks are reduced to the latent grid with **max**, not with ComfyUI's trilinear
|
||||
resize. Trilinear blurs a mask across frames and lets the original content bleed
|
||||
through the edit. Max keeps a latent cell that any masked pixel touches fully
|
||||
editable.
|
||||
|
||||
### Does the mask need to be semi-transparent so the LoRA can see the expression?
|
||||
|
||||
No — and it would cost you the swap. The expression does not travel under the
|
||||
mask: the whole source clip, original face included, is in the conditioning as an
|
||||
aligned reference on `source_id 1`, frame by frame. The model reads the
|
||||
performance from there whatever the mask does. That is what the LoRA was trained
|
||||
on: the new head follows the old head's acting.
|
||||
|
||||
Partial mask values do something else. In ComfyUI's inpaint path they blend the
|
||||
original latent back in, so you keep the original geometry in pixels **and the
|
||||
original identity with it** — the result drifts toward an average of both faces.
|
||||
That is what `mask_strength` below 1.0 buys, and it is a deliberate choice, not a
|
||||
way to get expressions. Soft edges are for hiding the seam; the middle of the
|
||||
mask should stay at 1.0.
|
||||
|
||||
### Hard for the sampler, soft for the paste
|
||||
|
||||
`mask_blur` exists for compositing, where a soft edge hides the seam. In the
|
||||
**denoise** mask a soft edge means *partial* denoising, which blends the original
|
||||
latent — and the original identity — back in right at the edge of the head, so
|
||||
the swap is weakest exactly where it is most visible. `mask_hard_for_inpaint`
|
||||
(on by default) binarises the mask before it becomes the denoise mask, leaving
|
||||
the soft version for the paste-back alone.
|
||||
|
||||
`latent_mask_dilate` grows the mask by whole **latent cells** after the
|
||||
reduction. One cell is 32 px and one latent frame covers 8 video frames, so this
|
||||
is far coarser than `mask_grow` in pixels — and it is what guarantees the head
|
||||
sits inside editable blocks rather than clipping at a cell boundary. One cell
|
||||
took a test mask from 11% to 25% of the grid; two took it to 44%. Watch
|
||||
`latent_mask` and stop at the first value that covers the head.
|
||||
|
||||
**Watch the `latent_mask` output.** It has one frame per *latent* frame, not per
|
||||
video frame, and that is the real resolution of the edit. A tight outline around
|
||||
a face can nearly vanish at that scale — and then nothing changes, no matter how
|
||||
good the LoRA is. If the mask looks thin there, raise `mask_grow`.
|
||||
|
||||
## Applying a LoRA to one region and putting it back
|
||||
|
||||
The shape of the thing: cut the region out as its own small clip, run the LoRA
|
||||
on it, composite it back. What makes the result honest is that the composite
|
||||
goes into the **original frames**, so every pixel outside the mask is the
|
||||
source's own — it never went through the VAE at all.
|
||||
|
||||
```
|
||||
subject_mask connected
|
||||
inpaint_with_mask true # outside the mask the latent IS the guide, not noise
|
||||
mask_hard_for_inpaint true # a soft DENOISE mask means partial denoising at the edge
|
||||
mask_strength 1.0 # below this the original is blended back INSIDE the mask
|
||||
paste_back true
|
||||
paste_confine_to_mask true # only the mask reaches the frame
|
||||
latent_mask_dilate 0 # one cell is 32 px of extra regenerated area, per side
|
||||
mask_grow the least that fits the new content
|
||||
mask_blur 3-5% of the subject's width — the paste's soft edge
|
||||
```
|
||||
|
||||
Since 1.39.0 this works with `crop_mode: off` too. Before that, with no crop the
|
||||
node returned the decoded frame whole: nothing *changed* outside the mask, but
|
||||
everything had been through an encode/decode round trip and came back softer.
|
||||
Now the mask composites against the source either way, and the crop goes back to
|
||||
being purely a resolution decision.
|
||||
|
||||
**Two passes on the region.** Set `paste_back: false` and the images and latent
|
||||
stay in the crop's own space, so you can upscale and refine there — where the
|
||||
region actually has pixels — and composite at the end with **BFS Paste Back**,
|
||||
feeding it the `crop_bboxes` output. That is the whole point of `crop_bboxes`
|
||||
being a socket.
|
||||
|
||||
## Why crop, and which mode
|
||||
|
||||
At 512×288 a person filling a fifth of the frame leaves a face about 25 px tall.
|
||||
No LoRA recovers identity from 25 px — it is missing pixels, not missing
|
||||
training. Cropping the head region and sampling it full-frame gives that same
|
||||
face 200–300 px, and the result is feathered back into untouched frames.
|
||||
|
||||
| `crop_mode` | behaviour |
|
||||
|---|---|
|
||||
| `off` | the whole frame is sampled. Use it first, to confirm the swap works at all. |
|
||||
| `combined` | one static box around the subject's whole travel, held for the clip. No dependency. |
|
||||
| `tracked` | constant-size box that stays still until the subject would leave it, then moves as little as possible. |
|
||||
| `zoomed` | the box also follows the subject's size, planned over the whole clip, every crop resampled to one output size. |
|
||||
|
||||
`tracked` and `zoomed` use a planner vendored from
|
||||
[drozbay/MaskVidExperiments](https://github.com/drozbay/MaskVidExperiments)
|
||||
(GPL-3.0, the same licence this pack carries) — nothing extra to install. Those
|
||||
boxes hold still through mask noise and occlusion, and **a jittering crop reads
|
||||
to a video model as camera motion**, which is what a naive per-frame crop around
|
||||
a mask produces.
|
||||
|
||||
`crop_scale` is the box as a multiple of the subject. At 1.5 the subject occupies
|
||||
two thirds and the rest is margin. Keep neck and shoulders inside: a face-tight
|
||||
crop is a framing the LoRA never saw in training, and it will show.
|
||||
|
||||
## Sizing, which is the one fiddly part
|
||||
|
||||
**Since 1.41.1 the shape is handled for you.** The crop is resized into the
|
||||
connected latent, so the box has to share its *aspect* — the sampler now asks
|
||||
the planner for exactly the latent's aspect ratio and says so in `debug`:
|
||||
|
||||
```
|
||||
crop shaped to the latent's 1.500 aspect
|
||||
```
|
||||
|
||||
Without it a mask around a face gives a portrait box, sampling it in a landscape
|
||||
latent stretches it, and the paste-back squeezes the result back — the face comes
|
||||
out deformed, and under `crop_mode: zoomed` by a different amount every frame.
|
||||
|
||||
The latent's **size** is deliberately left to you, because that is the sampling
|
||||
resolution and it is the entire reason to crop: a face 200 px tall in the source
|
||||
sampled at 768 px wide is a face with pixels to work with, and the paste-back
|
||||
brings it home at the box's size. Choose the size for the quality you want; the
|
||||
shape is no longer yours to get right.
|
||||
|
||||
**BFS Crop Size** is still there for when you want the number, or the per-frame
|
||||
boxes, upstream of the sampler — building the latent at the right size skips the
|
||||
rebuild, and the boxes feed BFS Paste Back directly:
|
||||
|
||||
```
|
||||
subject_mask ──┬─→ BFS Crop Size ──→ width / height ──→ EmptyLTXVLatentVideo ──┐
|
||||
│ │
|
||||
└───────────────────────────────────────────→ BFS Sampler ←─────┘
|
||||
```
|
||||
|
||||
Keep its three crop widgets identical to the sampler's — the planner is
|
||||
deterministic, so equal inputs give the same box, and a mismatch means the
|
||||
sampler silently resizes the crop into a latent that does not fit it.
|
||||
|
||||
That resize is worth naming, because it does not look like a sizing problem when
|
||||
it happens. A mask around a face in a 1920x1080 frame gives a portrait box —
|
||||
512x672, say. Sampling it into a 768x512 landscape latent stretches the crop
|
||||
horizontally, and the paste-back then squeezes the result back into the portrait
|
||||
box. The face comes out visibly deformed, and with `crop_mode: zoomed` the
|
||||
amount of deformation changes from frame to frame.
|
||||
|
||||
### Doing it by hand
|
||||
|
||||
|
||||
|
||||
The connected `latent` sets the sampled size. When you crop, the box is smaller
|
||||
than the frame, so the latent should match the box — the `debug` output prints
|
||||
the size to use:
|
||||
|
||||
```
|
||||
sample size 416x768 (connect EmptyLTXVLatentVideo at this size to avoid a resize)
|
||||
```
|
||||
|
||||
If they disagree the node resamples the crop on the way back, which costs exactly
|
||||
the sharpness the crop was there to buy. Run once, read `debug`, set
|
||||
`EmptyLTXVLatentVideo`, run again.
|
||||
|
||||
**The latent must come from the graph.** LTX‑2.5 latents are AV — video and audio
|
||||
nested together — and the node cannot fabricate that structure. Without one it
|
||||
builds a plain video latent, warns, and the guide may not reach the model at all.
|
||||
|
||||
## Chunking
|
||||
|
||||
`temporal_tile_size` at 0 samples the clip in one pass. Set it to the length the
|
||||
LoRA trained at (73 for the LTX head-swap recipe) for longer clips, and the node
|
||||
slices the guide, the mask and the overlap together per chunk.
|
||||
|
||||
Chunked sampling with an AV latent is refused rather than silently corrupted —
|
||||
slicing a nested latent in time is not a thing the node does yet.
|
||||
|
||||
## Decoding
|
||||
|
||||
`decode` chooses how the sampled latent becomes frames:
|
||||
|
||||
- **full** — one shot. Right for a single pass at the size the sampler ran at.
|
||||
- **tiled** — for a latent too large to decode whole, e.g. after a 2x upscaler.
|
||||
A full decode there pins VRAM at 99% and shuttles tensors until it looks like
|
||||
a hang.
|
||||
- **none** — skip it and return the latent alone.
|
||||
|
||||
**For anything beyond a single pass, prefer `none` and decode outside.** Your own
|
||||
VAE Decode (Tiled) is tunable, visible in the graph, and reused by the rest of
|
||||
the workflow. The only reason decoding lives in this node at all is the
|
||||
paste-back, which needs pixels — so with `none` the compositing is skipped too,
|
||||
and **BFS Head Swap Paste Back** does it after your decode.
|
||||
|
||||
## Second pass at higher resolution
|
||||
|
||||
With cropping on, `latent` is the **crop's** latent, not the frame's — which is
|
||||
what you want to refine, because that is where the face has pixels. To chain a
|
||||
second pass:
|
||||
|
||||
```
|
||||
Head Swap Sampler (paste_back: off)
|
||||
├─ latent → LTXV upscaler / second sampler → VAE Decode ─┐
|
||||
└─ crop_bboxes ───────────────────────────────────────────────┤
|
||||
guide_video (original frames) ────────────────────────────────────┤
|
||||
▼
|
||||
BFS Head Swap Paste Back → frames
|
||||
```
|
||||
|
||||
With `paste_back: off` the node stops compositing, so `images` and `latent` stay
|
||||
in the crop's own space. Refine there, then bring the result home with **BFS Head
|
||||
Swap Paste Back**, which takes the crops, the original frames and `crop_bboxes`.
|
||||
It does the same edge-aware feather and optional mask confinement the sampler
|
||||
would have done.
|
||||
|
||||
Refining before compositing is the right order: upscaling the crop puts the
|
||||
pixels where the identity is, and the frame around it never needed a second pass.
|
||||
|
||||
## Reading the outputs
|
||||
|
||||
| output | what it answers |
|
||||
|---|---|
|
||||
| `images` | the result, with the crop pasted back |
|
||||
| `mask_over_source` | is the mask where I think it is? Red over the original frames, green box for the crop |
|
||||
| `cropped_guide` | exactly what the model was fed. If this is wrong, nothing downstream can be right |
|
||||
| `crop_mask` | the mask after grow/blur, in the crop's pixel space |
|
||||
| `latent_mask` | the mask as the sampler sees it — where a thin mask disappears |
|
||||
| `latent` | the sampled latent — the CROP's, when cropping is on |
|
||||
| `crop_bboxes` | the boxes, for Head Swap Paste Back after a second pass |
|
||||
| `debug` | crop mode and box, mask ops, tiling, the size to build the latent at |
|
||||
|
||||
## If a second run behaves differently from the first
|
||||
|
||||
The `guider` is an object another node builds, and ComfyUI keeps it in the
|
||||
execution cache between runs. This node has to write the reference specs into
|
||||
its options — the guider captured its own dict by reference when it was built,
|
||||
so patching the model alone never reaches the forward — and those specs hold
|
||||
the encoded reference **latents**.
|
||||
|
||||
Since 1.37.0 the write is undone in a `finally` when sampling ends, including
|
||||
when it ends by raising. Before that the specs stayed on the cached guider:
|
||||
the latents could not be freed while that guider lived, and a later run that
|
||||
did not overwrite the same keys — a graph with no references, an aborted run,
|
||||
another sampler sharing the same guider — sampled with the *previous* clip's
|
||||
specs, whose shapes no longer fit. If you saw runs that worked once and then
|
||||
started failing or ignoring the reference until you restarted ComfyUI, that
|
||||
was it.
|
||||
|
||||
Chunked clips also used to keep every sampled chunk on the sampling device
|
||||
until the whole clip was done; they now move to the offload device as they come
|
||||
out, so VRAM no longer grows with clip length while the model is still loaded.
|
||||
|
||||
## Suggested order when something looks wrong
|
||||
|
||||
1. `crop_mode: off`, no mask, `temporal_tile_size: 0`. This is the plain swap; if
|
||||
it fails, the problem is the recipe or the LoRA, not this node.
|
||||
2. Add the mask. Check `latent_mask` before judging the result.
|
||||
3. Add the crop. Check `mask_over_source` for the box, and set the latent to the
|
||||
size `debug` reports.
|
||||
4. Only then raise the frame count and turn on chunking.
|
||||
|
||||
Sampling settings follow the LoRA's recipe: CFG 3–5 with a real negative, and no
|
||||
LightX2V (CFG 1.0 means no amplification, and the reference stops being
|
||||
amplified).
|
||||
@@ -0,0 +1,121 @@
|
||||
# BFS LoRA Surgery
|
||||
|
||||
Read any LoRA's layers and blocks, then scale or drop them and apply the result live.
|
||||
No retraining, no files written.
|
||||
|
||||
## Why
|
||||
|
||||
When a LoRA learns a defect (smoothed skin, identity drifting when the subject is far from
|
||||
the camera, a pose it refuses to copy) that defect usually lives in one module family and a
|
||||
range of blocks, not in the whole adapter. Scaling or dropping that group and regenerating
|
||||
under a fixed seed tells you where it lives, in minutes.
|
||||
|
||||
In the cases tested so far the culprit was the MLP **input** projection, in a subset of
|
||||
blocks. The MLP output projection barely mattered and attention was innocent. Treat that as
|
||||
a starting point rather than a rule: which group and which block range are right differs per
|
||||
LoRA, which is why the panel measures instead of assuming.
|
||||
|
||||
**Names depend on the base model**, so the panel lists whatever the file you loaded actually
|
||||
contains:
|
||||
|
||||
| base | family | MLP input | MLP output | blocks |
|
||||
|---|---|---|---|---|
|
||||
| Qwen-Image-2.1 | `transformer_blocks` | `img_mlp.gate_up` | `img_mlp.out` | 32 |
|
||||
| Krea 2 | `blocks`, plus `txtfusion.*` | `mlp.gate`, `mlp.up` | `mlp.down` | 28 |
|
||||
| MiniMax-H3 | `blocks`, plus `token_refiner.blocks` | `mlp.fc1` | `mlp.fc2` | 50 |
|
||||
|
||||
Other bases will show other names again (`w1`/`w3`, `fc1`, `proj_in`). Read the list, do not
|
||||
assume a name.
|
||||
|
||||
## How it works
|
||||
|
||||
The node discovers the structure from the key names, so it is not tied to any model. It
|
||||
handles the diffusers convention (`lora_A` / `lora_B`) and the kohya one (`lora_down` /
|
||||
`lora_up` plus `alpha`). Families like `transformer_blocks`, `double_blocks`, `token_refiner`
|
||||
or `txtfusion` are detected automatically, along with the block indices and module types in
|
||||
each.
|
||||
|
||||
The panel lists every family with its module types, sorted by how large the update is
|
||||
(`‖ΔW‖`), and that bar is a quick read on where training actually invested.
|
||||
|
||||
Scaling multiplies only the up/B factor, since `(sB)A = s(BA)`. Exact, and negative scales
|
||||
work.
|
||||
|
||||
## Using it
|
||||
|
||||
1. Pick a LoRA in `lora_name`. The panel reads it and lists the families.
|
||||
2. Click a type's `→` to select it, then click blocks to select them (shift-click for a range).
|
||||
3. Set a scale and press **add rule**, or **add as regex** if you want the same selection
|
||||
written as a pattern you can hand-edit.
|
||||
4. Queue the prompt. The edited LoRA is applied to the model in memory.
|
||||
|
||||
**Order matters.** Later rules override earlier ones for modules they both match, so put
|
||||
boosts first and drops last. Use ↑/↓ to reorder.
|
||||
|
||||
`rules` is plain JSON and is saved with the workflow:
|
||||
|
||||
```json
|
||||
[
|
||||
{"enabled": true, "match": {"type": "attn.*"}, "scale": 1.15},
|
||||
{"enabled": true, "match": {"type": "img_mlp.gate_up", "blocks": "8-15"}, "scale": 0},
|
||||
{"enabled": true, "match": {"blocks": "16-23", "type": "*"}, "scale": 0}
|
||||
]
|
||||
```
|
||||
|
||||
A match is either structural (`family`, `type`, `blocks`) or a `regex` tested against the
|
||||
module's full path. `type` accepts a trailing `*`. `blocks` accepts `8-15`, `0,4,7` or
|
||||
`8-15,24-31`.
|
||||
|
||||
## Saving the result
|
||||
|
||||
Two ways, depending on whether you are exploring or building a pipeline.
|
||||
|
||||
**From the panel.** Once the rules look right, type a name (or leave it blank) and press
|
||||
`save as file`. It writes straight into `models/loras/surgery` and reports the result inline.
|
||||
Nothing else to wire.
|
||||
|
||||
**From the graph.** `BFS LoRA Surgery` has a `rules` output. Connect it to the `rules` input
|
||||
of **BFS LoRA Surgery (save)** and the file is produced as part of the run, with no JSON
|
||||
copied by hand. Both nodes read the same `lora_name`, so point them at the same file.
|
||||
|
||||
Either way the rules are applied identically, since both call the same code.
|
||||
|
||||
Leave `filename` empty and the name is built from the rules, so the file says what was done
|
||||
to it:
|
||||
|
||||
```
|
||||
mylora__attn-x1.15__gate_up-b8_15-off__all-b16_23-off.safetensors
|
||||
```
|
||||
|
||||
Names are capped and sanitized, so a long source name or a 30-rule recipe still produces a
|
||||
valid filename: when it would exceed the limit, both halves are trimmed and a short hash of
|
||||
the full recipe is appended, so two different recipes never collapse onto the same name.
|
||||
|
||||
The file also carries its own history. Alongside the original metadata (which is preserved
|
||||
untouched), it writes:
|
||||
|
||||
* `bfs_surgery_summary`, one line any metadata viewer will show:
|
||||
`136/192 modules kept: dropped img_mlp.gate_up in blocks 8-15; all modules in blocks 16-23,
|
||||
scaled attn.* x1.15 (from mylora.safetensors)`
|
||||
* `bfs_surgery`, the full record as JSON: tool and timestamp, source filename **and its
|
||||
sha256** so provenance survives a rename, the readable recipe, the exact rule list, and the
|
||||
module counts (total, kept, dropped, scaled)
|
||||
* `ss_output_name` updated to the new name
|
||||
|
||||
That means a file you share months from now can still explain what was done to it, and can be
|
||||
traced back to the checkpoint it came from even if someone renamed either one.
|
||||
|
||||
`subfolder` defaults to `surgery` to keep these out of your main list, `save_dtype` can
|
||||
downcast, and `overwrite` is off by default, so a repeat run gets `_1`, `_2` rather than
|
||||
destroying the previous one.
|
||||
|
||||
## A warning
|
||||
|
||||
A variant can win every metric you are looking at and still have destroyed what you trained.
|
||||
Removing the MLP path entirely gave the sharpest skin in one of the cases here, and made the
|
||||
LoRA stop copying expression from the source image. Always check a deliberately hard case
|
||||
(a strong expression, an unusual angle) with your eyes, not only the numbers.
|
||||
|
||||
Pruning after training is also not the same as training with those layers excluded. Excluding
|
||||
them during training may simply fail to converge, because the layer is where the defect lodges,
|
||||
not where it comes from. That is usually the dataset.
|
||||
@@ -0,0 +1,530 @@
|
||||
# BFS Shot Planner / Shot Loop
|
||||
|
||||
Turn a long video into shots a video model can actually follow, give every shot its own
|
||||
reference image and prompt, run the rest of the workflow once per shot, and join the results
|
||||
back into one video with the original timing and soundtrack.
|
||||
|
||||
## Why
|
||||
|
||||
A video model only follows a guide reliably inside the clip length it was trained on. An H3
|
||||
body-swap LoRA trained on 107-frame clips keeps the scene, framing and motion at 4.5 s and
|
||||
loses them at 10 s, where it starts summarising the clip instead of following it frame by
|
||||
frame. Splitting the source into model-sized shots, at camera cuts when there are any, keeps
|
||||
every shot inside that range.
|
||||
|
||||
## Nodes
|
||||
|
||||
| node | what it does |
|
||||
|---|---|
|
||||
| **BFS Shot Planner** | Pick a video, split it on a timeline, set a reference and prompt per shot. Outputs the shots as a ComfyUI **list**. |
|
||||
| **BFS Shot Unpack** | Opens one shot into plain values: guide frames, reference, second reference, prompt, length, first frame, audio, size, index. Use it with any model. |
|
||||
| **BFS Shot Repack** | Puts edited pieces back into a shot: Unpack's `shot` output goes to Repack's `shot`, the edited piece (e.g. the reference with its background removed) to its input. Unconnected inputs keep their values; timing, cuts and audio are unchanged. |
|
||||
| **BFS Shot H3 Conditioning** | Ready-made MiniMax H3 conditioning for one shot, built with the native nodes (Reference to Video + Add Guide). Optional *duet*: the shot's clip pinned in a side panel (canvas or shifted RoPE; route the model through the node for the shift); the join cuts the panel off. Optional prompt writer: pick a *task* (character swap, style, setting, appearance, lighting / weather, custom) and an *instruction*; with a VLM on its *vlm* input it writes each shot's duet prompt from the shot and its references, otherwise a template for the task (*planner prompt*, the default, keeps the planner's prompt). The written text comes out on *prompt*. Optional *setting_ref* (TSC's tip): one more reference, the shot's middle frame with the person covered in TV static, so the model sees the place in full detail; it is the last `<Picture n>` (write `{setting}` for its tag, otherwise a sentence is added to subject_definitions). |
|
||||
| **BFS Shot Join** | Concatenates the decoded shots in order, trims each to its true length, cross-fades soft joins and returns the soundtrack. With *comparison* on it also returns a side-by-side video (original shot \| references \| result) with the shot's info and your *label* on top and its prompt below, ready for Create Video. |
|
||||
| **BFS Shot H3 Duet** | Renders one shot with H3: the shot's clip pinned beside the video (duet, no LoRA), the shot as an aligned guide (for body-swap LoRAs), or both. |
|
||||
|
||||
## The panel
|
||||
|
||||
Use the language selector in the top bar to switch the interface between English, Portuguese, Spanish and Simplified
|
||||
Chinese. The choice is saved in this browser and does not alter the workflow plan or prompt text. **Tour** opens an
|
||||
optional step-by-step introduction when clicked; it never appears automatically.
|
||||
|
||||
The planner's panel has five tabs, in the order you work. The timeline stays on top of every tab.
|
||||
|
||||
| tab | what is there |
|
||||
|---|---|
|
||||
| 🎞 **Video** | pick or upload the video, audio, how it is split into shots (cuts or fixed length), generation size |
|
||||
| 🎬 **Shots** | one card per shot (scroll sideways, **← →** move between shots), the preview player and the selected shot's editor: reference & prompt, who is replaced (mask, generation mode, `{target}`), continuity, **Copy to other shots** |
|
||||
| 👥 **People & masks** | the cast found by face (a reference per person) and the global mask settings |
|
||||
| 📝 **Prompts & refs** | the global reference and prompt, the VLM and the reference descriptions (`{details}`) |
|
||||
| ▶ **Run** | **Checks** (what to fix before running, with a button to each shot), auto / queue loop, filters, test limits |
|
||||
|
||||
Shot cards show badges for what each shot has: 🎯 mask, 🎭 / ✂ generation mode, 🧑 `{target}`, ⛓ continuity and ⚠ problems.
|
||||
The header's ⚠ count opens the checks.
|
||||
|
||||
**Copy to other shots** (in the shot editor) copies the chosen parts of the selected shot to all shots, or to the
|
||||
shots after it: references, prompt, mask text, mask points (each shot gets them on its frame at the same relative
|
||||
position), generation mode, `{target}`, continuity. It replaces the old "→ all" buttons.
|
||||
|
||||
## How the loop works
|
||||
|
||||
ComfyUI runs any node that receives a list once per item and pairs several lists by index. The
|
||||
planner's `shots` output is a list, so everything downstream of it (conditioning, sampler,
|
||||
decode) runs once per shot, and shot *i* always gets guide *i*, reference *i* and prompt *i*.
|
||||
Single values such as the model and VAE are shared by every run. The join takes the whole list
|
||||
back (it is an `INPUT_IS_LIST` node) and puts the video together.
|
||||
|
||||
Two run modes:
|
||||
|
||||
- **Auto loop (one run)**: every shot in one queue run. Simple; holds all decoded shots in memory.
|
||||
- **Queue loop (one shot per run)**: each run generates the next pending shot and stores it on
|
||||
disk. The join blocks every node after it until the last shot, then outputs the full video,
|
||||
so a Save Video node only fires once. With *Auto-queue the next shot* on, the panel re-queues
|
||||
the workflow after each shot. *Reset loop* starts over. Changing the plan starts a new run.
|
||||
|
||||
## Guide frames and lengths
|
||||
|
||||
Each shot is generated at a length the model accepts (frame grid: H3 `17n+5`, LTX/Wan `8n+1`,
|
||||
Wan `4n+1`, or any). When a shot is shorter than that length, the extra guide frames come from
|
||||
the video that follows it, so the guide never repeats or stretches. The join cuts each result
|
||||
back to the shot's own frames, which keeps the timing identical to the source. Those extra
|
||||
frames are also a real overlap, which the join uses to cross-fade boundaries that are not
|
||||
camera cuts.
|
||||
|
||||
## Splitting
|
||||
|
||||
- **Camera cuts** (default): cuts from **PySceneDetect** (`adaptive` or `content`; install
|
||||
`scenedetect`), or the built-in detector when the package is missing. Shots shorter than the
|
||||
minimum merge into a neighbour when the merge still fits; shots longer than the maximum split
|
||||
into equal parts.
|
||||
- **Fixed length**: equal parts no longer than the maximum.
|
||||
- **One shot (whole range)**: the whole range (start / end frame, *Max total seconds*) as a single shot, whatever its
|
||||
length; *Max seconds / shot* is not applied. For "N seconds in one piece" set *Range length, seconds* = N, right
|
||||
next to the mode (it moves the end frame; *Max total seconds* in the Run tab does the same for quick tests).
|
||||
For pieces of at most X seconds use *Fixed length* with *Max seconds / shot* = X.
|
||||
- **By hand**: drag the white handles on the timeline to move a boundary, double-click the
|
||||
shot bar to split, *⇥ Merge* to join, or select a shot and press **Delete** to remove the cut at its
|
||||
start (it merges into the previous shot). Your edits are what runs.
|
||||
|
||||
**Range** (🎞 Video tab, with its own player): *Start frame* / *End frame* (timeline frames; end 0 = the end of the
|
||||
video) limit the plan to a part of the video, before anything is split, analysed or generated. Play or scrub the player
|
||||
(or click the timeline) and press *⇤ start here* / *⇥ end here* to take its exact frame. The
|
||||
timeline dims what is outside; the join and the soundtrack cover only the range. *Max total seconds* counts from the
|
||||
start of the range.
|
||||
|
||||
*Max seconds / shot* is converted to frames and snapped down to the grid (4.5 s at 24 fps =
|
||||
107 frames for H3). *Max shots* and *Max total seconds* cap a long source.
|
||||
|
||||
## Audio
|
||||
|
||||
The source card shows whether the video has a usable audio track (rate, channels) or why not. *Use the
|
||||
video's audio* or *silent track*. Without a usable track (none, unreadable, or silent chosen) the planner's
|
||||
`audio`, BFS Shot Join's `audio` and the H3 Duet nodes return a **silent track of the right length** instead
|
||||
of nothing, so Create Video / Save Video always work. Audio is read with ffmpeg, or with PyAV when the ffmpeg
|
||||
binary is missing.
|
||||
|
||||
## Preview
|
||||
|
||||
*Play shot* plays only the selected shot, from its first to its last frame, and stops (or loops with
|
||||
*loop shot*); every shot card has its own ▶. *Play all* plays the whole plan and jumps over shots
|
||||
that will not run. The panel shows the shot's start, end and current time (and frame), a yellow
|
||||
playhead follows on the timeline, and clicking the timeline seeks. The source video streams from
|
||||
the input folder, so only what you play is loaded.
|
||||
|
||||
## Filters
|
||||
|
||||
Skip shots that should not run, decided per shot from a few sampled frames:
|
||||
|
||||
| filter | skips a shot when |
|
||||
|---|---|
|
||||
| Needs a person | no person is detected |
|
||||
| Min person size | the largest person covers less than that fraction of the frame (wide shots) |
|
||||
| Max people | more people than that (crowds) |
|
||||
| Needs a face | no face is detected |
|
||||
| Skip dark / fades | the mean brightness is below the level |
|
||||
| Skip static | there is almost no motion (title cards, freeze frames) |
|
||||
| Min frames | the shot is shorter than that |
|
||||
|
||||
People and faces use YOLO from `models/ultralytics` (`person_yolov8m-seg.pt`, `face_yolov8m.pt`, the
|
||||
Impact Pack files) with OpenCV fallbacks. *Analyse people & faces* shows the numbers on every shot
|
||||
card. Each shot can override the filters (always run / always skip).
|
||||
|
||||
Skipped shots do not run. Connect the planner's `timeline` output to BFS Shot Join and they are
|
||||
filled with the original video, keeping the full duration and soundtrack, or removed (with the
|
||||
matching audio) when *Skipped shots in the output* is set to remove them.
|
||||
|
||||
## Cast (people by face)
|
||||
|
||||
*Find people* samples the video every 0.5 s, detects faces and groups them into people by face
|
||||
identity, on the CPU (it never touches the GPU; the result is cached per video). **No extra package is needed**: by
|
||||
default it uses OpenCV Zoo's **YuNet** (detection) + **SFace** (recognition), Apache-2.0 models run by OpenCV itself and
|
||||
downloaded once into `models/opencv_face` (0.2 + 37 MB). If **InsightFace** is installed (`buffalo_l` in
|
||||
`models/insightface`) it is used instead (a bit more accurate; mind its license). Last resort, without either: SAM 3.1
|
||||
faces grouped by colours (hair, skin, clothes). Each
|
||||
person card shows the face, screen share and first/last appearance (click the face to seek).
|
||||
Give a person a reference (and a second one) by clicking the slot, or pick one from the recent
|
||||
references; *ignore* leaves that person out. Every shot card shows who is in it, with the main
|
||||
person (most screen time) highlighted and linked people outlined in green.
|
||||
|
||||
| option | effect |
|
||||
|---|---|
|
||||
| Shots use their main person's reference | a shot without its own reference takes its main person's (shown with a dashed green outline) |
|
||||
| Split where the main person changes | adds a boundary where the largest face switches to another person for at least a second, then re-splits |
|
||||
| Only run shots with a linked person | shots where no linked, non-ignored person appears are skipped ("no linked person") |
|
||||
|
||||
A reference set on the shot itself always wins; the global reference is the last fallback.
|
||||
|
||||
## References and prompts
|
||||
|
||||
Every shot can have its own reference, second reference and prompt; shots without them use the
|
||||
global ones. Click an empty reference slot to upload an image straight from your computer; a filled
|
||||
slot has **✕** to remove it; **Copy to other shots** gives a shot's references to every shot. The last ten
|
||||
references you picked are shown under the shot editor for one-click reuse (click = reference,
|
||||
shift+click = second reference). Connected `ref_image`, `ref_image_2` and `prompt` inputs override
|
||||
the panel's global values, so an edited first frame or a prompt from another node can drive the
|
||||
defaults.
|
||||
|
||||
The planner also outputs `ref_image` and `ref_image_2`: the references the shots actually use,
|
||||
without repeats, so a plan where every shot shares one reference returns a single image.
|
||||
|
||||
## Mask & crop (SAM 3)
|
||||
|
||||
Generate only part of the frame. In a shot's editor (*Who is replaced*), type what to segment in English
|
||||
(`person in white`, `woman with red hair`, `red car`; up to 32 tokens, commas for several things) or click
|
||||
**🎯 Points** to pick positive / negative points on any frame of the shot, then **👁 Preview** (red = mask,
|
||||
yellow = crop box). Then pick how the shot is generated:
|
||||
|
||||
| mode | what is generated | crop / uncrop |
|
||||
|---|---|---|
|
||||
| **Full frame** | the whole frame; the mask only feeds `{target}` and the setting picture | no |
|
||||
| **🧩 Frame + paste** | the whole frame (so the model follows the guide's pose freely and the new person may be bigger); BFS Shot Join then pastes only the person onto the original frame: the shot's mask plus the new subject's outline (SAM 3 on the result with the shot's *new subject* text, default `person`; `paste_new_outline`) | no crop; paste in the Join |
|
||||
| **🎭 Mask only** | only the mask, on the whole frame; everything else stays exactly as it was | no |
|
||||
| **✂ Crop** | a box around the mask (more pixels for a small person); BFS Shot Join pastes it back | yes |
|
||||
| **✂🎭 Crop + mask** | only the mask, inside the crop: the most detail with the background kept | yes |
|
||||
|
||||
*Frame + paste* is the safest for a different body (e.g. a woman in profile replaced by a man facing the camera):
|
||||
*Mask only* binds the new person to the old outline, *Frame + paste* does not. The two mask modes need **BFS Shot H3 Conditioning → inpaint = per shot (planner)** (the default; *only the mask*
|
||||
forces it on every shot, *off* never). The crop is one box around the mask (the union over all its frames, so it
|
||||
does not shake), at the source resolution, sent to the model at the generation size; **BFS Shot Join** pastes the
|
||||
result back, feathered by the mask (or the whole box). **BFS Shot Unpack** also outputs the shot's mask (`mask`),
|
||||
e.g. for an inpainting model. In **BFS Shot Join**'s comparison video, cropped shots show their mask (red) and crop
|
||||
box (yellow) over the original column (`comparison_mask`, on by default).
|
||||
|
||||
**Mask opacity** (mask modes, per shot, slider under the modes): the value of the white inside the generation mask.
|
||||
1 regenerates the masked area completely. A grey mask (e.g. 0.85) keeps part of the original there: H3 puts those
|
||||
rows at *opacity* × the noise level, so they start from the original partly visible. 0.8-0.9 keeps pose, outline and lighting while still swapping; too low copies the original
|
||||
person. With `guide_mode = aligned guide` the model also sees the whole original shot as its guide, so it follows
|
||||
the motion either way. (Not applied with the duet panel.)
|
||||
|
||||
**🎨 Color Guide** is a separate optional visual cue. Turn it on for a shot with a mask and choose the marker color
|
||||
for ref 1. The default colors for the global references are set below each image in **Prompts & refs**; a shot may
|
||||
override them. The selected region is tinted in the source guide, while the image detail,
|
||||
movement and expression remain visible underneath. This does not turn on `Mask only` or create an H3 generation
|
||||
mask. The prompt must explain that the tint is temporary and that the matching reference replaces the tinted person.
|
||||
|
||||
For consistency across shots, **People & masks → Find people** groups sampled faces by identity (ArcFace through
|
||||
InsightFace when installed; otherwise YuNet + SFace, or a weaker appearance fallback). Choose the matching `Person N`
|
||||
in each shot to inherit that cast member's Color Guide color. This is a user-confirmed link: profile views, small faces
|
||||
and occlusion can make identity grouping uncertain. Leave the shot on `manual identity` when unsure. The shot's mask
|
||||
still selects the actual region to tint, and its color can be overridden for that shot.
|
||||
|
||||
For two people in one shot, open **+ Add a second person**, use **ref 2** for that selection, and choose a different
|
||||
color (purple by default; duplicate colors are disabled). Enable *use_ref_2* on BFS Shot H3 Conditioning. The
|
||||
second SAM 3 text selection is optional; preview it before queuing. Color opacity and ArcFace identity links are
|
||||
under the optional settings. Use this prompt and replace the color names if you choose another palette:
|
||||
|
||||
```text
|
||||
body_swap: subject_definitions: <Subject 1> is the character shown in <Picture 1>, whose identity, face, head, hair or fur, skin, body shape, proportions and clothing define the green-marked replacement. <Subject 2> is the character shown in <Picture 2>, whose identity, face, head, hair or fur, skin, body shape, proportions and clothing define the purple-marked replacement. In the guide video, the translucent green and purple tints are temporary editing markers, not character appearance. Replace the green-tinted person only with <Subject 1> and the purple-tinted person only with <Subject 2>. The guide video provides the scene and the complete performance of both people, including their body poses, gestures, head motion, gaze, speech and facial expressions.
|
||||
|
||||
summary: [video generation + references] Replace the green-tinted guide person with <Subject 1> from <Picture 1> and the purple-tinted guide person with <Subject 2> from <Picture 2>, preserving the scene, camera and complete motion and expression of both people.
|
||||
|
||||
detailed_description: [Shot 1] Replace the person marked by the translucent green tint with <Subject 1> from <Picture 1>, and the person marked by the translucent purple tint with <Subject 2> from <Picture 2>. Each character follows that person's exact position, timing, body pose, gestures, head motion, gaze, speech and facial expressions in the guide. Remove both temporary color markers completely. Keep the two reference identities distinct and stable in every frame. Preserve their interaction, the background, lighting, camera and all unselected content from the guide video. Prevent identity drift, clothing changes, texture crawling and flicker. The result is a real, photographic video.
|
||||
```
|
||||
|
||||
The VLM's shot analysis suggests a SAM 3 text selection from the original frames; it does not assign guide colors or
|
||||
decide that two detections are the same identity. **Describe target** only describes the first shot selection. For
|
||||
two-color swaps, choose both selections and identities yourself, and use the explicit mapping above. The VLM prompt
|
||||
writer for *duet* is a separate split-screen task; its generic character-swap draft assumes the supplied pictures
|
||||
show one character, so use *planner prompt* for this two-character template. `Describe refs` also treats ref and ref 2
|
||||
as views of one identity; write the two subject definitions explicitly when ref 2 is a different person.
|
||||
|
||||
**Mask guide** (experimental, *Mask only* shots, BFS Shot H3 Conditioning → `mask_guide`): the shot with only the
|
||||
masked region visible, the rest grey, so the model looks at the subject's pose and outline on its own while the mask
|
||||
limits where it generates. *+ extra guide* adds it as a second aligned guide next to the normal one (the model sees
|
||||
both); *instead of the full guide* replaces the normal one; *+ reference video* gives it as a native reference video
|
||||
(`<Video n>`, after the shot's own one when guide_mode uses it; write `{mask_video}` in the prompt where its tag goes, otherwise a sentence is added). `mask_guide_look` sets what it shows of the subject: *grey blurred* (default: volume, light and head direction, no colours or face, so the model follows the motion without copying the old subject), *colour* (as it is; can make the model copy the old subject), *silhouette* (flat shape), *edges* (outlines) or *pose (people)* (the skeleton of the people in the mask, OpenPose colours; needs ultralytics and a YOLO pose model in `models/ultralytics`, e.g. `pose/yolov8m-pose.pt`). `mask_ref_size` makes that reference video smaller (default 1/2, ~1/4 of the tokens): it only has to show the pose and outline. Compare with
|
||||
*off*: the LoRAs were trained with one full guide, and it can also pull the old subject's look.
|
||||
|
||||
### Masks from a video (rotoscoping) instead of SAM 3
|
||||
|
||||
A mask made elsewhere (After Effects Roto Brush, DaVinci Resolve Magic Mask, another ComfyUI workflow…) can replace
|
||||
SAM 3. It is a black and white video, **white = the subject**, and it works with every mode (Mask only, Crop,
|
||||
Crop + mask), `{target}`, the setting picture and the comparison overlay. Three places, the most specific wins:
|
||||
|
||||
| where | covers | wins over |
|
||||
|---|---|---|
|
||||
| shot editor → **🎞 mask video** | only that shot (its first frame = the shot's first frame) | everything |
|
||||
| the shot's own SAM 3 text / points | that shot | the two below |
|
||||
| the planner's optional **`mask`** input (MASK or IMAGE batch, e.g. Load Video + Convert Image to Mask) | the whole video, one mask per source frame | the panel's mask video |
|
||||
| People & masks → **🎞 Mask video (whole video)** | the whole video, matched by time | SAM 3 for shots without a mask |
|
||||
|
||||
Any size and frame rate: it is matched to the shot by time and resized. The console says which mask each shot used.
|
||||
|
||||
**Stitch finishing** (BFS Shot Join, cropped shots only; adapted from Neko (Nekodificador)'s *NKD Inpaint Stitch*, MIT, built on AbleJones's workflow and nodes):
|
||||
|
||||
| Option | What it does |
|
||||
|---|---|
|
||||
| `edge_hardness` (0-1) | hardens the soft edge of the paste; raise it when a faint ghost of the original person shows around the new one |
|
||||
| `match_colors` (0-1) | corrects the colour / brightness drift of the generated patch (Reinhard in LAB), with statistics over the whole shot so it does not flicker |
|
||||
| `match_region` | *around the subject (swap)*: measured on a ring of background around the mask, so a new person keeps their own colours. *inside the subject*: measured inside the mask, for retouching the same content |
|
||||
| `seamless_edges` | Poisson blend (OpenCV) for stubborn seams; slower |
|
||||
|
||||
**Which mode.** A generation mask alone keeps the background exact but generates the person at the frame's own size,
|
||||
so a small person gets few pixels; a crop alone gives the person more resolution, but the whole box is regenerated
|
||||
and pasted back (that is what the finishing above is for). *Mask only* is the simplest when the person fills a good
|
||||
part of the frame; *Crop + mask* is best for small people. In the mask modes the latent starts from the shot's own
|
||||
frames and its SAM 3 mask (grown by *expand* plus one latent cell) becomes the H3 generation mask. Raise *expand*
|
||||
when the new person is bigger than the old one (longer hair, wider body): the generated area cannot go past the
|
||||
grown mask; their shadow or reflection outside the mask stays from the original.
|
||||
|
||||
Credit to **Neko (Nekodificador)** and **AbleJones**, whose workflow and nodes the crop + generation mask follows.
|
||||
ComfyUI builds without native H3 generation masks need a per-row mask patch on the model
|
||||
(e.g. ComfyUI-MiniMaxH3-PerRowMasking).
|
||||
|
||||
The **People & masks** tab holds the global mask settings, with defaults that work as they are: fill holes on,
|
||||
temporal expand 2 frames (less flicker), expand 16 px, feather 12 px, padding 15 %, paste by mask, blockify off
|
||||
(16 aligns the mask to H3's latent grid), threshold 0.5, 4 objects. *Show masks on shots* overlays the preview
|
||||
on the shot cards. SAM 3 uses the official `sam3.1_multiplex_fp16.safetensors` in `models/checkpoints`,
|
||||
downloaded from Comfy-Org/sam3.1 the first time.
|
||||
|
||||
## VLM suggestions
|
||||
|
||||
Connect a vision-language model to the planner's `vlm` input (CLIPLoader with `qwen3vl_4b` or `qwen3vl_8b`).
|
||||
It looks at a few frames of every shot and suggests what to segment, a description of the shot (camera,
|
||||
framing, action) and whether to run it. In the **Prompts & refs** tab: *Use the VLM when the workflow runs* applies the
|
||||
suggestions at run time (mask text for shots without one; `{shot}` in a prompt becomes that shot's
|
||||
description), *Analyse shots* asks from the panel (after one run with the VLM connected), and each shot's
|
||||
editor shows its suggestion with buttons to apply it. The summary output lists them too.
|
||||
|
||||
## Describe references ({details})
|
||||
|
||||
The VLM can describe the references, for any task (not tied to swaps), and any subject: a person, a creature, a robot,
|
||||
an animal or a cartoon character (it says what the subject is first). In the **Prompts & refs** tab pick an instruction
|
||||
preset: *short* (the default: one noun phrase of at most ~20 words that fits inside a sentence, e.g. `green goblin with
|
||||
long pointed ears in a torn brown leather vest`, for `<Subject 1> is the {details} shown in <Picture 1>`), *full body* (face, hair, skin, age, build, clothing piece by piece), *head / face*, *face attributes* (a
|
||||
short comma-separated list), *outfit*, or *custom* (your own instruction). **📝 Describe refs** writes one
|
||||
description per reference set: the global references, every shot's and every cast person's. They are saved in
|
||||
the plan and editable. Write **`{details}`** anywhere in a prompt and every shot gets the description of its own
|
||||
references; a set without a description is described by the VLM at run time (no click needed).
|
||||
|
||||
**Fit the prompt** (on by default, next to 📝 Describe refs): the VLM gets the sentence of the prompt that holds the
|
||||
marker, with a blank in its place (e.g. `<Subject 1> is the ___ shown in <Picture 1>`), and writes the description for
|
||||
it, so the description adapts to the prompt and not the other way round (no "the A goblin", no paragraph inside a
|
||||
sentence). The panel buttons use the global prompt (or the shot's own); at run time the prompt the shot really uses.
|
||||
|
||||
The VLM buttons in the panel work after the workflow has run once with the VLM connected (ComfyUI only hands
|
||||
models to nodes when they run). Tested with Qwen3-VL 8B; some smaller or modified models return empty answers
|
||||
for some wordings: the planner retries with a reworded request and with sampling, but if a preset stays empty,
|
||||
use the 8B or another preset.
|
||||
|
||||
## Settings reference
|
||||
|
||||
Every setting in the panel has a hover tooltip (ⓘ). The main ones:
|
||||
|
||||
| setting | what it does |
|
||||
|---|---|
|
||||
| Mode | Camera cuts (shots start at cuts, long ones split evenly), fixed length, or one shot (the whole range) |
|
||||
| Detector / Sensitivity | PySceneDetect adaptive or content, or the built-in one; higher sensitivity finds more cuts |
|
||||
| Frame grid | frame counts the model accepts (H3 17n+5, LTX/Wan 8n+1, Wan 4n+1) |
|
||||
| Max / Min seconds per shot | longest shot sent to the model; shorter shots merge into a neighbour |
|
||||
| Timeline fps | the frame rate the planner works on (generation and audio use it) |
|
||||
| Max shots / Max total seconds | test on the first shots or seconds only |
|
||||
| Megapixels / Size multiple | generation size at the source's aspect ratio, snapped to the multiple |
|
||||
| Run | auto loop (all shots in one run) or queue loop (one shot per run, stored on disk) |
|
||||
| Skipped shots in the output | keep the original video there or remove those shots |
|
||||
| Mask: padding / expand / feather / temporal expand / blockify | crop context, paste-back growth and softness, flicker hold, square blocks |
|
||||
| Mask: threshold / max objects / paste back | SAM 3 text matching, objects tracked, paste only the mask or the whole box |
|
||||
| VLM: frames per shot / max tokens | how much the VLM sees and how long it may answer |
|
||||
|
||||
## One seed per shot: BFS Shot Seed (1.94.0)
|
||||
|
||||
A seed widget on the sampler gives every shot the SAME seed (the node runs once per shot with the same value).
|
||||
**BFS Shot Seed** (shot in, seed out) makes it a choice: *same for every shot*, *increment per shot* (seed, seed+1 ...)
|
||||
or *random per shot* (unrelated but repeatable seeds from the base seed). `overrides` pins single shots, e.g.
|
||||
`3:12345, 5:777`, to keep the shots that came out well and re-roll only the others.
|
||||
|
||||
## Structured starting noise: BFS Shot Noise (1.94.0, experimental)
|
||||
|
||||
Optional. **BFS Shot Noise** gives the NOISE for SamplerCustomAdvanced; with every amount at 0 it is exactly RandomNoise.
|
||||
|
||||
| Input | What it does |
|
||||
|---|---|
|
||||
| `shot_seed` (+ `shot`, `overrides`) | the seed of each shot: same / increment / random per shot, as BFS Shot Seed |
|
||||
| `shared_base` | the coarse pattern of the noise comes from `base_seed`: the same for every shot, at any resolution and any length (it is a tape along the timeline); the fine detail stays per shot. Put the seed of a result that worked in `base_seed` to keep its coarse noise |
|
||||
| `distribution` | gaussian (default), laplacian, student-t or uniform, all at unit variance |
|
||||
| `spectrum_tilt` | 1/f^alpha spectrum: > 0 puts more energy in the large blobs ("fractal" noise), < 0 in the fine grain |
|
||||
| `pyramid` | coarser layers of noise added on top |
|
||||
| `static_share` | a part of the noise identical in every frame |
|
||||
| `guide_structure` (`vae`, `guide_frames`, `mask`) | outside the mask, the guide's latent is mixed into the noise |
|
||||
| `ref_tint` (`vae`, `ref_image`, `mask`) | inside the mask, the noise leans toward the reference's average latent |
|
||||
|
||||
All mixes keep unit variance. The model was trained on plain noise: use small amounts and compare with 0.
|
||||
|
||||
## {details} filled automatically (1.92.0)
|
||||
|
||||
A `{details}` in a shot's prompt is filled without pressing 📝 Describe refs: the planner's VLM describes the shot's
|
||||
references before any SAM 3 work; with no VLM on the planner, the slot is left for **BFS Shot H3 Conditioning** or
|
||||
**BFS Shot H3 Duet**, whose `vlm` input (the same Qwen3-VL CLIPLoader) describes them (the *short* preset, fitted to
|
||||
the sentence). Edited descriptions always win. With no VLM anywhere it becomes "character" and the log says so.
|
||||
|
||||
## Source blur (keep the mouth) (1.91.0)
|
||||
|
||||
`source_blur` on **BFS Shot H3 Conditioning**, **BFS Shot H3 Duet**, **BFS H3 Duet** and **BFS H3 Duet Conditioning**
|
||||
blurs the source person in what the model *sees* of the source (aligned guide, native reference video, duet panel),
|
||||
keeping the **mouth** sharp: the model still reads the pose, the motion and the lips but has no face to copy, so the
|
||||
identity comes from the reference. Masks from SAM 3.1 (the part, minus a slightly grown "mouth" mask), cached per shot.
|
||||
`face (keep mouth)`, `head + hands (keep mouth)` (also hides hair and hands), `person (keep mouth)` (the whole body:
|
||||
only for big changes, the pose then comes from the blurred shape). The kept area of *Mask only* and the output are
|
||||
never blurred. Trick shared by JalenBrunson (Sapiens2 there) for reliable ref2v swaps.
|
||||
|
||||
## Big videos (upload)
|
||||
|
||||
ComfyUI refuses any single request bigger than `--max-upload-size` (100 MB by default), which is why big videos fail
|
||||
to upload in most nodes. The planner's ⬆ Upload (source video, mask videos, pictures) sends files of 32 MB or more in
|
||||
8 MB pieces that are written straight to disk and assembled in the input folder, so any size works without changing
|
||||
how ComfyUI is started (a dropped piece is sent again). For other nodes, start ComfyUI with a higher limit, e.g.
|
||||
`--max-upload-size 4096`, or copy the file into `ComfyUI/input` and pick it from the list.
|
||||
|
||||
A newly selected or uploaded source starts as **one shot covering at most its first 15 seconds**. This is only the
|
||||
initial plan: widen the range or choose *Camera cuts* / *Fixed length* and press *Auto split* when you want multiple
|
||||
shots. Existing saved plans keep their own range and split mode.
|
||||
|
||||
## Continuity between shots
|
||||
|
||||
Each shot (after the first) can continue from the **previous shot's generated result**: in the shot editor,
|
||||
*Continuity* = *previous shot as reference* adds a frame of it as one more `<Picture n>` after the shot's own
|
||||
references (good across camera cuts, keeps the person consistent), and *previous shot as first frame* anchors it
|
||||
at frame 0 of the shot (for continuous action without a cut). Pick which frame: the previous result's first,
|
||||
middle or last. **Copy to other shots** with *continuity* ticked applies it to every shot after the first.
|
||||
|
||||
It needs the previous result to exist: it works in the **queue loop** (the planner reads the stored result) and in
|
||||
the **auto loop with BFS Shot H3 Duet** (it renders shot by shot and keeps the last result). BFS Shot Unpack also
|
||||
outputs it as `previous_result`. This is different from H3 Conditioning's *first_frame*, which anchors the shot's
|
||||
own frame from the source video.
|
||||
|
||||
### History: a long take, window by window (1.87.0)
|
||||
|
||||
*Continuity* = **previous shot's end as history** carries the **last 17, 34 or 51 frames of the previous shot's
|
||||
result** into the next shot as clean frames placed **just before frame 0 on H3's timeline**: the model sees them as
|
||||
the immediate past of the video it is generating, so motion, camera, light and the new look continue across the join
|
||||
(instead of one still frame). Idea from Akatz's [H3 Relay](https://github.com/akatz-ai/h3-relay) windowed edit and
|
||||
Ethanfel's MiniMax H3 Context Loop; this is BFS's own implementation (a process-local patch of H3's packed layout
|
||||
that moves the video, its audio and the other guides past the history; ComfyUI files are not edited).
|
||||
|
||||
- **On by default** (1.88.0) for every shot after the first; set *Continuity = off* on a shot to drop it. Plans saved
|
||||
before keep what they had.
|
||||
- **Same take only.** A shot that starts at a camera cut skips it (nothing to continue). For one long take, split it
|
||||
with *Mode = fixed length* (or let *Camera cuts* split long shots) and set history on every shot after the first
|
||||
with **Copy to other shots → continuity**. Shots still exist: they are the windows, and cuts reset the history.
|
||||
- **Every mode:** full frame, *Frame + paste*, *Mask only* (the mask regenerates the target; the history is a
|
||||
condition outside it), *Crop* (the history is cut with this shot's box) and the **duet canvas** (the source frames
|
||||
under the history go into the panel strip). Not with duet *shifted RoPE*. The aligned guide still drives this shot's
|
||||
own frames, so sync stays frame-exact.
|
||||
- **BFS Shot H3 Conditioning and BFS Shot H3 Duet** both use it (duet canvas or aligned guide).
|
||||
- **Queue loop:** shots run in order and each reads the stored result of the one before. Auto loop: taken from the
|
||||
previous render when it matches.
|
||||
- Video only for now (the previous audio is not carried). Costs one more 17/34/51-frame condition per shot.
|
||||
|
||||
## MiniMax H3 with an aligned guide (example)
|
||||
|
||||
```
|
||||
BFS Shot Planner ── shots ──► BFS Shot H3 Conditioning (clip, vae, guide_mode = aligned guide)
|
||||
└─ positive, latent ─► CFGGuider / sampler ─► VAE Decode ─┐
|
||||
BFS Shot Join ◄── images ─────────────────────────────────────────────────────────────────┘
|
||||
└─► Create Video (fps, audio) ─► Save Video
|
||||
```
|
||||
|
||||
For other models, use **BFS Shot Unpack** and wire `guide_frames`, `ref_image`, `prompt` and
|
||||
`length` into that model's own guide and reference nodes; the list mapping works the same way.
|
||||
|
||||
## Choosing who is replaced (`{target}`), step by step
|
||||
|
||||
With several people on screen, a prompt that says "replace the person" leaves the model to guess. `{target}`
|
||||
puts a short description of the right person into the prompt of every shot ("the young woman with dark hair in a
|
||||
pink top"), so the model knows WHICH one to swap. The selection is made with SAM 3, and the description by the
|
||||
VLM, so it works for anyone or anything you can click on.
|
||||
|
||||
### What you need
|
||||
|
||||
- The planner's **`vlm`** input connected to a Qwen3-VL (CLIPLoader, type *stable_diffusion*, `qwen3vl_8b` recommended).
|
||||
- The workflow run **once** with it connected. ComfyUI only hands models to nodes when they run; after that the
|
||||
panel buttons can use it.
|
||||
- Without a VLM it still works: you get the cut-out and type the description yourself.
|
||||
|
||||
### 1. Put `{target}` in the prompt
|
||||
|
||||
Write `{target}` wherever the prompt names the person being replaced: in the global prompt (all shots) or in a
|
||||
shot's own prompt. For example, the swap caption the H3 LoRAs were trained with:
|
||||
|
||||
```
|
||||
... only the face, body and clothing of {target} are replaced. summary: [video generation + reference] Replace
|
||||
{target} in the guide video with <Subject 1>, keeping the scene, camera and the complete motion and facial
|
||||
performance of the guide video. detailed_description: [Shot 1] <Subject 1> takes the exact place of {target}
|
||||
and moves with the same timing, body pose, gestures, ...
|
||||
```
|
||||
|
||||
`{target}` can be combined with `{details}` (the description of the references) and `{shot}` (the VLM's
|
||||
description of the shot).
|
||||
|
||||
### 2. Select the person
|
||||
|
||||
Open a shot and click **🎯 Points…**. The modal shows one frame of the shot; the slider below picks the frame.
|
||||
|
||||
- **Click on the person's body** (torso), not only the face. A click on the face selects the head, and the
|
||||
description then misses the clothes.
|
||||
- **Right-click** (or shift+click) puts an *exclude* point, e.g. on a second person standing close.
|
||||
- **👁 Segment** shows what SAM 3 picked (red). Add or remove points until only that person is red.
|
||||
- Instead of points you can just type what to segment (`woman in pink`, `man with glasses`, `dog`): no clicking is
|
||||
needed, SAM 3 finds and tracks it through the shot. Points are for when the text is ambiguous (two similar people).
|
||||
Points win when both are set. **Copy to other shots** with *mask text* (and *mode*) ticked gives the text (and
|
||||
the generation mode) to every shot.
|
||||
|
||||
Then save:
|
||||
|
||||
| Button | What it does |
|
||||
|---|---|
|
||||
| **Save (this shot)** | keeps the selection on this shot only |
|
||||
| **Save → all shots** | applies the same points to every shot, each on its frame at the same relative position as the frame you clicked on (e.g. clicked in the middle of this shot → every shot uses its middle frame) |
|
||||
|
||||
**Save → all shots** is for a person who stays in the same place across shots (a sequential video, a fixed camera).
|
||||
|
||||
**🎯 Track → whole video** (in the points window) is for a person who moves around: the selection is tracked by SAM 3
|
||||
through the whole video (or the range), forward and backward from the frame you clicked on, and every shot without
|
||||
its own selection takes its part of that track. Also type a short text (e.g. `woman in pink top`): after a camera cut
|
||||
the tracker uses it to find the subject again. The People & masks tab shows the track and removes it. The first run
|
||||
tracks the whole range (a minute or two per few hundred frames); the result is cached until the selection changes.
|
||||
|
||||
### 3. Describe the person
|
||||
|
||||
Click **🧑 Describe target** in the shot's editor:
|
||||
- SAM 3 cuts the person out of the selected frame, and a thumbnail of the cut-out appears next to the field.
|
||||
- The VLM writes a short phrase into the **`{target}`** field.
|
||||
- The field stays editable: correct it or write your own. Keep it short and visual (who, hair, main clothes with
|
||||
colours) and start it with "the".
|
||||
- **Copy to other shots** with *{target}* ticked copies the description to every shot. Use it when the same person is replaced throughout;
|
||||
otherwise describe shot by shot.
|
||||
|
||||
### 4. Check and fix
|
||||
|
||||
- **👁 Preview** on a few shots shows the selection on several frames of each shot.
|
||||
- In a shot where the selection picked the wrong thing, **✕ Clear** removes its points, text and mode. Select
|
||||
again in that shot only (**Save (this shot)**) and describe again.
|
||||
- **Copy to other shots** with *mask points* ticked copies that shot's points to every shot without opening the modal.
|
||||
|
||||
### What happens when the workflow runs
|
||||
|
||||
Every shot's prompt gets its own `{target}` text: the one you wrote or made with 🧑 Describe target; when the field is
|
||||
empty but the shot has a selection (mask text, points or a mask video) and the VLM is connected, the subject is cut out
|
||||
and described automatically (written for the prompt's `{target}` sentence). A shot with neither gets "the person". The selection is
|
||||
only used to make the description: it does not crop or mask anything unless the shot's mode is *Mask only*, *Crop*
|
||||
or *Crop + mask*.
|
||||
|
||||
### Examples
|
||||
|
||||
- **Two people, swap only one:** in the first shot, click on the woman's torso, put an exclude point on the man,
|
||||
then **Save → all shots** and **🧑 Describe target** → "the young woman with long black hair in a white blouse".
|
||||
**Copy to other shots** with *{target}* ticked.
|
||||
- **The person changes between shots:** describe shot by shot, e.g. shot 1 "the man in the grey suit", shot 2
|
||||
"the woman in the red dress". Each shot keeps its own text.
|
||||
- **Animals:** it works the same way, e.g. "the small black dog in front".
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
| Problem | Fix |
|
||||
|---|---|
|
||||
| "No VLM yet" | connect the `vlm` input and run the workflow once, or type the description |
|
||||
| "Select the person first" | the shot has no points or mask text: use **🎯 Points…** |
|
||||
| "SAM 3 found nothing" | click again on the person (on the torso), or try another frame with the slider |
|
||||
| Only the face or the hair is described | the click was on the head: click the body |
|
||||
| The other person is described | add an exclude point (right-click) on the other person, **👁 Segment**, save, describe again |
|
||||
| The swap still hits the wrong person | make the description more specific (position: "on the left", clothing colours) |
|
||||
@@ -0,0 +1,303 @@
|
||||
// Shot Planner: constants and small helpers shared by the panel and its views (no side effects: ComfyUI loads every
|
||||
// .js under web/js as an extension, so this module only exports).
|
||||
import { h as vueH, reactive, ref } from "../vendor/vue.esm-browser.prod.mjs";
|
||||
|
||||
// UI language is a browser preference, not part of the workflow or generation prompts.
|
||||
const LANG_KEY = "bfs.shotloop.language";
|
||||
const savedLanguage = (() => { try { return localStorage.getItem(LANG_KEY) || "en"; } catch { return "en"; } })();
|
||||
export const uiLanguage = ref(["en", "pt", "es", "zh"].includes(savedLanguage) ? savedLanguage : "en");
|
||||
export function setUiLanguage(value) {
|
||||
if (!["en", "pt", "es", "zh"].includes(value)) return;
|
||||
uiLanguage.value = value;
|
||||
try { localStorage.setItem(LANG_KEY, value); } catch { /* private mode */ }
|
||||
}
|
||||
|
||||
const TEXT = {
|
||||
"🎞 Video": ["🎞 Vídeo", "🎞 Vídeo", "🎞 视频"], "🎬 Shots": ["🎬 Shots", "🎬 Planos", "🎬 镜头"],
|
||||
"👥 People & masks": ["👥 Pessoas e máscaras", "👥 Personas y máscaras", "👥 人物和蒙版"],
|
||||
"📝 Prompts & refs": ["📝 Prompts e referências", "📝 Prompts y referencias", "📝 提示词和参考图"],
|
||||
"▶ Run": ["▶ Executar", "▶ Ejecutar", "▶ 运行"], "📖 Guide": ["📖 Guia", "📖 Guía", "📖 指南"],
|
||||
"🧭 Tour": ["🧭 Tour", "🧭 Recorrido", "🧭 引导"], "Language": ["Idioma", "Idioma", "语言"],
|
||||
"Source video": ["Vídeo de origem", "Vídeo de origen", "源视频"], "Range": ["Trecho", "Rango", "范围"],
|
||||
"Split into shots": ["Dividir em planos", "Dividir en planos", "拆分镜头"], "Generation size": ["Tamanho da geração", "Tamaño de generación", "生成尺寸"],
|
||||
"Camera cuts": ["Cortes de câmera", "Cortes de cámara", "镜头切换"], "Fixed length": ["Duração fixa", "Duración fija", "固定时长"],
|
||||
"One shot (whole range)": ["Um plano (trecho inteiro)", "Un plano (rango completo)", "单个镜头(整个范围)"],
|
||||
"Mode": ["Modo", "Modo", "模式"], "Detector": ["Detector", "Detector", "检测器"], "Sensitivity": ["Sensibilidade", "Sensibilidad", "灵敏度"],
|
||||
"Frame grid": ["Grade de frames", "Múltiplo de fotogramas", "帧步长"], "Max seconds / shot": ["Máx. segundos por plano", "Máx. segundos por plano", "每个镜头最长秒数"],
|
||||
"Min seconds / shot": ["Mín. segundos por plano", "Mín. segundos por plano", "每个镜头最短秒数"], "Timeline fps": ["FPS da linha do tempo", "FPS de la línea de tiempo", "时间轴帧率"],
|
||||
"Megapixels": ["Megapixels", "Megapíxeles", "百万像素"], "Size multiple": ["Múltiplo do tamanho", "Múltiplo del tamaño", "尺寸倍数"],
|
||||
"✂ Auto split": ["✂ Dividir automaticamente", "✂ División automática", "✂ 自动拆分"], "↻ Re-analyse": ["↻ Analisar novamente", "↻ Volver a analizar", "↻ 重新分析"],
|
||||
"Reference & prompt": ["Referência e prompt", "Referencia y prompt", "参考图和提示词"], "Who is replaced": ["Quem será substituído", "Quién se reemplaza", "替换对象"],
|
||||
"Continuity": ["Continuidade", "Continuidad", "连续性"], "Copy to other shots": ["Copiar para outros planos", "Copiar a otros planos", "复制到其他镜头"],
|
||||
"Global reference & prompt": ["Referência e prompt globais", "Referencia y prompt globales", "全局参考图和提示词"],
|
||||
"Describe references ({details})": ["Descrever referências ({details})", "Describir referencias ({details})", "描述参考图({details})"],
|
||||
"VLM": ["VLM", "VLM", "视觉语言模型"], "Use the VLM when the workflow runs": ["Usar o VLM ao executar o workflow", "Usar el VLM al ejecutar el flujo", "运行工作流时使用 VLM"],
|
||||
"Fill the mask text": ["Preencher o texto da máscara", "Completar el texto de la máscara", "填写蒙版描述"],
|
||||
"Fill {shot} in the prompt": ["Preencher {shot} no prompt", "Completar {shot} en el prompt", "在提示词中填入 {shot}"],
|
||||
"Frames per shot": ["Frames por plano", "Fotogramas por plano", "每个镜头的帧数"], "Max tokens": ["Máx. tokens", "Máx. tokens", "最大 token 数"],
|
||||
"🤖 Analyse shots": ["🤖 Analisar planos", "🤖 Analizar planos", "🤖 分析镜头"], "📝 Describe refs": ["📝 Descrever referências", "📝 Describir referencias", "📝 描述参考图"],
|
||||
"🎨 Color Guide": ["🎨 Guia de cores", "🎨 Guía de color", "🎨 颜色引导"], "Ref 1 color": ["Cor da ref. 1", "Color de ref. 1", "参考图 1 的颜色"],
|
||||
"Ref 2": ["Ref. 2", "Ref. 2", "参考图 2"], "+ Add a second person (optional)": ["+ Adicionar segunda pessoa (opcional)", "+ Añadir segunda persona (opcional)", "+ 添加第二个人(可选)"],
|
||||
"Second person · Ref 2": ["Segunda pessoa · Ref. 2", "Segunda persona · Ref. 2", "第二个人 · 参考图 2"],
|
||||
"Describe the second person for SAM 3": ["Descreva a segunda pessoa para o SAM 3", "Describe la segunda persona para SAM 3", "为 SAM 3 描述第二个人"],
|
||||
"👁 Preview": ["👁 Prévia", "👁 Vista previa", "👁 预览"], "More options": ["Mais opções", "Más opciones", "更多选项"],
|
||||
"Keep ref 2 color consistent across shots (optional)": ["Manter a cor da ref. 2 nos outros planos (opcional)", "Mantener el color de ref. 2 en otros planos (opcional)", "在其他镜头中保持参考图 2 的颜色(可选)"],
|
||||
"Ref 1 identity": ["Identidade da ref. 1", "Identidad de ref. 1", "参考图 1 的身份"], "Ref 2 identity": ["Identidade da ref. 2", "Identidad de ref. 2", "参考图 2 的身份"],
|
||||
"Choose manually": ["Escolher manualmente", "Elegir manualmente", "手动选择"], "↩ cast color": ["↩ cor do elenco", "↩ color del reparto", "↩ 使用角色颜色"],
|
||||
"Opacity": ["Opacidade", "Opacidad", "不透明度"], "The default keeps movement and expression visible.": ["O padrão mantém o movimento e a expressão visíveis.", "El valor predeterminado mantiene visibles el movimiento y la expresión.", "默认值会保留动作和表情。"],
|
||||
"Full frame": ["Quadro inteiro", "Fotograma completo", "整帧"], "🧩 Frame + paste": ["🧩 Quadro + colar", "🧩 Fotograma + pegar", "🧩 整帧生成并贴回"],
|
||||
"🎭 Mask only": ["🎭 Somente máscara", "🎭 Solo máscara", "🎭 仅蒙版"], "✂ Crop": ["✂ Recorte", "✂ Recorte", "✂ 裁剪"], "✂🎭 Crop + mask": ["✂🎭 Recorte + máscara", "✂🎭 Recorte + máscara", "✂🎭 裁剪并蒙版"],
|
||||
"People & masks": ["Pessoas e máscaras", "Personas y máscaras", "人物和蒙版"], "Cast": ["Elenco", "Reparto", "角色"],
|
||||
"Mask & crop settings": ["Configurações de máscara e recorte", "Ajustes de máscara y recorte", "蒙版和裁剪设置"],
|
||||
"Color Guide identity color": ["Cor de identidade do Guia de cores", "Color de identidad de la Guía de color", "颜色引导身份颜色"],
|
||||
"No Color Guide color": ["Sem cor", "Sin color", "无颜色引导颜色"], "Green": ["Verde", "Verde", "绿色"], "Purple": ["Roxo", "Morado", "紫色"],
|
||||
"Red": ["Vermelho", "Rojo", "红色"], "Blue": ["Azul", "Azul", "蓝色"], "Yellow": ["Amarelo", "Amarillo", "黄色"], "Magenta": ["Magenta", "Magenta", "品红"], "Cyan": ["Ciano", "Cian", "青色"],
|
||||
"Checks": ["Verificações", "Comprobaciones", "检查"], "Run": ["Executar", "Ejecutar", "运行"], "Filters": ["Filtros", "Filtros", "筛选"],
|
||||
"Auto loop (one run)": ["Loop automático (uma execução)", "Bucle automático (una ejecución)", "自动循环(一次运行)"],
|
||||
"Queue loop (one shot per run)": ["Fila (um plano por execução)", "Cola (un plano por ejecución)", "队列循环(每次运行一个镜头)"],
|
||||
"Skipped shots in the output": ["Planos ignorados na saída", "Planos omitidos en la salida", "输出中跳过的镜头"], "Keep original video": ["Manter vídeo original", "Mantener vídeo original", "保留原视频"],
|
||||
"Remove them": ["Removê-los", "Quitarlos", "移除"], "Max shots (0 = all)": ["Máx. planos (0 = todos)", "Máx. planos (0 = todos)", "镜头上限(0 = 全部)"],
|
||||
"Max total seconds (0 = all)": ["Máx. duração total (0 = tudo)", "Duración total máx. (0 = todo)", "总时长上限(0 = 全部)"],
|
||||
"Auto-queue the next shot": ["Enfileirar próximo plano automaticamente", "Añadir automáticamente el siguiente plano a la cola", "自动将下一个镜头加入队列"],
|
||||
"Needs a person": ["Precisa ter uma pessoa", "Debe haber una persona", "需要有人物"], "Needs a face": ["Precisa ter um rosto", "Debe haber un rostro", "需要有人脸"],
|
||||
"Skip dark / fades": ["Ignorar cenas escuras / fades", "Omitir escenas oscuras / fundidos", "跳过暗场/淡入淡出"], "Skip static shots": ["Ignorar planos estáticos", "Omitir planos estáticos", "跳过静止镜头"],
|
||||
"👤 Analyse people & faces": ["👤 Analisar pessoas e rostos", "👤 Analizar personas y rostros", "👤 分析人物和人脸"],
|
||||
"Video": ["Vídeo", "Vídeo", "视频"], "Shots": ["Planos", "Planos", "镜头"], "People": ["Pessoas", "Personas", "人物"], "Prompts": ["Prompts", "Prompts", "提示词"],
|
||||
"Back": ["Voltar", "Atrás", "上一步"], "Next": ["Avançar", "Siguiente", "下一步"], "Close tour": ["Fechar tour", "Cerrar recorrido", "关闭引导"],
|
||||
"Shot Planner tour": ["Tour do Shot Planner", "Recorrido de Shot Planner", "Shot Planner 引导"], "Shot": ["Plano", "Plano", "镜头"],
|
||||
"This tour is optional. Use Next to visit each main tab; close it whenever you like.": ["Este tour é opcional. Use Avançar para conhecer cada aba; feche quando quiser.", "Este recorrido es opcional. Pulsa Siguiente para visitar cada pestaña; puedes cerrarlo cuando quieras.", "此引导为可选。点击下一步浏览各个主选项卡,也可随时关闭。"],
|
||||
"Choose a video, set the frame range, split at camera cuts, then set the generation size.": ["Escolha um vídeo e o trecho, divida nos cortes de câmera e defina o tamanho da geração.", "Elige un vídeo y un rango, divídelo en cortes de cámara y define el tamaño de generación.", "选择视频和范围,按镜头切换拆分,然后设置生成尺寸。"],
|
||||
"Review each shot, assign its references and prompt, and choose how the result is applied.": ["Revise cada plano, associe referências e prompt e escolha como aplicar o resultado.", "Revisa cada plano, asigna referencias y prompt, y elige cómo aplicar el resultado.", "检查每个镜头,设置参考图和提示词,并选择结果的应用方式。"],
|
||||
"Link detected people to references. Set masks here or per shot; ArcFace links are a guide and need your review.": ["Associe as pessoas detectadas às referências. Configure máscaras aqui ou por plano; revise os vínculos ArcFace.", "Vincula las personas detectadas con referencias. Configura máscaras aquí o por plano; revisa los enlaces ArcFace.", "将检测到的人物关联到参考图。可在此处或逐镜头设置蒙版;请检查 ArcFace 关联。"],
|
||||
"Set shared references, prompts and optional VLM assistance. Written prompts are kept exactly as authored.": ["Defina referências, prompts e assistência opcional do VLM. Os prompts escritos são preservados como estão.", "Configura referencias, prompts y asistencia opcional del VLM. Los prompts escritos se conservan tal cual.", "设置共享参考图、提示词和可选的 VLM 辅助。提示词会保持原样。"],
|
||||
"Check warnings, choose auto or queue loop, and set filters or quick-run limits.": ["Confira avisos, escolha loop automático ou fila e configure filtros e limites de teste.", "Revisa avisos, elige bucle automático o cola y configura filtros y límites de prueba.", "查看警告,选择自动循环或队列,并设置筛选条件和快速运行限制。"],
|
||||
"Color Guide adds a translucent marker to a selected person. Ref 1 and Ref 2 must use different colors in the same shot.": ["O Guia de cores aplica uma cor translúcida à pessoa selecionada. Ref. 1 e Ref. 2 precisam de cores diferentes no mesmo plano.", "La Guía de color aplica una marca translúcida a la persona seleccionada. Ref. 1 y Ref. 2 deben tener colores distintos en el mismo plano.", "颜色引导会给选中的人物添加半透明颜色。同一镜头中的参考图 1 和参考图 2 必须使用不同颜色。"],
|
||||
"🎬 Shot Planner": ["🎬 Planejador de planos", "🎬 Planificador de planos", "🎬 镜头规划器"],
|
||||
"— choose a video from the input folder —": ["— escolha um vídeo da pasta de entrada —", "— elige un vídeo de la carpeta de entrada —", "— 从输入文件夹选择视频 —"],
|
||||
"⬆ Upload": ["⬆ Enviar", "⬆ Subir", "⬆ 上传"], "Pick or upload a video to start": ["Escolha ou envie um vídeo para começar", "Elige o sube un vídeo para empezar", "选择或上传视频以开始"],
|
||||
"It is split into shots the model can generate; every shot then gets its own reference, prompt and mask.": ["O vídeo será dividido em planos que o modelo consegue gerar; cada plano terá sua própria referência, prompt e máscara.", "El vídeo se divide en planos que el modelo puede generar; cada uno tendrá su referencia, prompt y máscara.", "视频会拆分为模型可生成的镜头,每个镜头都有自己的参考图、提示词和蒙版。"],
|
||||
"work on a part of the video": ["trabalhe em um trecho do vídeo", "trabaja en una parte del vídeo", "处理视频的一部分"],
|
||||
"Start frame": ["Frame inicial", "Fotograma inicial", "起始帧"], "End frame (0 = end)": ["Frame final (0 = fim)", "Fotograma final (0 = final)", "结束帧(0 = 末尾)"],
|
||||
"⇤ start here": ["⇤ começar aqui", "⇤ empezar aquí", "⇤ 从此处开始"], "⇥ end here": ["⇥ terminar aqui", "⇥ terminar aquí", "⇥ 在此处结束"], "✕ whole video": ["✕ vídeo inteiro", "✕ vídeo completo", "✕ 整个视频"],
|
||||
"Camera cuts or fixed length, on the model's frame grid": ["Cortes de câmera ou duração fixa, na grade de frames do modelo", "Cortes de cámara o duración fija, según la cuadrícula de fotogramas del modelo", "按镜头切换或固定时长,并符合模型帧步长"],
|
||||
"re-analyses": ["reanalisar", "vuelve a analizar", "重新分析"], "ignores Max seconds / shot": ["ignora Máx. segundos por plano", "ignora Máx. segundos por plano", "忽略每个镜头的最长秒数"],
|
||||
"cropped shots use it for the crop": ["planos recortados usam esse valor no recorte", "los planos recortados usan este valor", "裁剪镜头会使用此尺寸"],
|
||||
"Shots use their main person's reference": ["Usar a referência da pessoa principal do plano", "Usar la referencia de la persona principal del plano", "镜头使用主要人物的参考图"],
|
||||
"Split where the main person changes": ["Dividir quando a pessoa principal mudar", "Dividir cuando cambie la persona principal", "主要人物变化时拆分"],
|
||||
"Only run shots with a linked person": ["Executar apenas planos com pessoa vinculada", "Ejecutar solo planos con una persona vinculada", "仅运行已关联人物的镜头"],
|
||||
"a shot without its own reference takes the reference of the person with the most screen time in it": ["um plano sem referência própria usa a referência da pessoa que aparece por mais tempo nele", "un plano sin referencia propia usa la de la persona que aparece más tiempo", "没有单独参考图的镜头会使用出镜时间最长人物的参考图"],
|
||||
"adds a boundary when the biggest face switches to someone else (re-splits the plan)": ["cria uma divisão quando o maior rosto muda para outra pessoa (refaz a divisão)", "añade un corte cuando el rostro más grande cambia de persona (vuelve a dividir)", "最大人脸切换到其他人时添加分界(重新拆分计划)"],
|
||||
"shots where no linked person appears are skipped": ["planos sem pessoa vinculada são ignorados", "se omiten los planos sin una persona vinculada", "跳过没有关联人物的镜头"],
|
||||
"Nobody found yet.": ["Nenhuma pessoa encontrada.", "Aún no se encontraron personas.", "尚未检测到人物。"], "↻ Re-analyse people": ["↻ Analisar pessoas novamente", "↻ Volver a analizar personas", "↻ 重新分析人物"], "👥 Find people": ["👥 Encontrar pessoas", "👥 Buscar personas", "👥 查找人物"],
|
||||
"faces sampled every 0.5 s and grouped by identity (InsightFace, CPU)": ["rostos amostrados a cada 0,5 s e agrupados por identidade (InsightFace, CPU)", "rostros muestreados cada 0,5 s y agrupados por identidad (InsightFace, CPU)", "每 0.5 秒采样人脸并按身份分组(InsightFace,CPU)"],
|
||||
"detects faces across the video and groups them into people": ["detecta rostos no vídeo e os agrupa por pessoa", "detecta rostros en el vídeo y los agrupa por persona", "检测视频中的人脸并按人物分组"],
|
||||
"track faces, link each person to a reference, split and assign shots by who is on screen": ["rastreie rostos, associe cada pessoa a uma referência e divida os planos por quem aparece", "rastrea rostros, vincula cada persona a una referencia y divide los planos según quién aparece", "跟踪人脸、关联参考图,并按出镜人物拆分镜头"],
|
||||
"Each shot picks what it marks (🎯 text or points, or 🎞 its own mask video) and how it is generated (": ["Cada plano define a seleção (🎯 texto, pontos ou 🎞 vídeo de máscara) e o modo de geração (", "Cada plano define la selección (🎯 texto, puntos o 🎞 vídeo de máscara) y el modo de generación (", "每个镜头设置选区(🎯 文本、点或 🎞 蒙版视频)和生成方式("],
|
||||
"🎞 Mask video (whole video)": ["🎞 Vídeo de máscara (vídeo inteiro)", "🎞 Vídeo de máscara (vídeo completo)", "🎞 蒙版视频(整个视频)"],
|
||||
"— none: SAM 3 per shot —": ["— nenhum: SAM 3 por plano —", "— ninguno: SAM 3 por plano —", "— 无:每个镜头使用 SAM 3 —"], "Fill holes": ["Preencher buracos", "Rellenar huecos", "填充孔洞"], "Invert": ["Inverter", "Invertir", "反转"],
|
||||
"Show masks on shots": ["Mostrar máscaras nos planos", "Mostrar máscaras en los planos", "在镜头卡片上显示蒙版"], "Expand (px)": ["Expandir (px)", "Expandir (px)", "扩展(像素)"],
|
||||
"Temporal expand (frames)": ["Expansão temporal (frames)", "Expansión temporal (fotogramas)", "时间扩展(帧)"], "Detection threshold": ["Limiar de detecção", "Umbral de detección", "检测阈值"], "Max objects": ["Máx. objetos", "Máx. objetos", "最大对象数"],
|
||||
"Crop padding (%)": ["Margem do recorte (%)", "Margen del recorte (%)", "裁剪边距(%)"], "Feather (px)": ["Suavização da borda (px)", "Suavizado del borde (px)", "羽化(像素)"], "Paste back": ["Colar de volta", "Pegar de vuelta", "贴回方式"],
|
||||
"only the mask (feathered)": ["somente a máscara (suavizada)", "solo la máscara (suavizada)", "仅蒙版(羽化边缘)"], "the whole box (feathered)": ["a caixa inteira (suavizada)", "todo el cuadro (suavizado)", "整个框(羽化边缘)"], "Blockify (px, 0 = off)": ["Blocos (px, 0 = desligado)", "Bloques (px, 0 = desactivado)", "块化(像素,0 = 关闭)"],
|
||||
"Uses the official sam3.1_multiplex_fp16 checkpoint (models/checkpoints), downloaded from Comfy-Org/sam3.1 the first time.": ["Usa o checkpoint oficial sam3.1_multiplex_fp16 (models/checkpoints), baixado de Comfy-Org/sam3.1 na primeira utilização.", "Usa el checkpoint oficial sam3.1_multiplex_fp16 (models/checkpoints), descargado de Comfy-Org/sam3.1 la primera vez.", "使用官方 sam3.1_multiplex_fp16 检查点(models/checkpoints),首次运行时从 Comfy-Org/sam3.1 下载。"],
|
||||
"SAM 3": ["SAM 3", "SAM 3", "SAM 3"], "track faces, link each person to a reference, split and assign shots by who is on screen": ["rastreie rostos, associe cada pessoa a uma referência e organize os planos por quem aparece", "rastrea rostros, vincula cada persona a una referencia y organiza los planos según quién aparece", "跟踪人脸、关联参考图,并按出镜人物分配镜头"],
|
||||
"Global reference & prompt": ["Referência e prompt globais", "Referencia y prompt globales", "全局参考图和提示词"], "defaults for every shot": ["padrão para todos os planos", "valor predeterminado para cada plano", "所有镜头的默认设置"],
|
||||
"who the shot becomes": ["em quem o plano será transformado", "en quién se transforma el plano", "镜头中的人物将替换为谁"], "Prompt for this shot (empty = global prompt). Placeholders: {target} {details} {shot} {setting}": ["Prompt deste plano (vazio = prompt global). Variáveis: {target} {details} {shot} {setting}", "Prompt de este plano (vacío = prompt global). Variables: {target} {details} {shot} {setting}", "此镜头的提示词(留空则使用全局提示词)。占位符:{target} {details} {shot} {setting}"],
|
||||
"recent (click = ref, shift+click = ref 2):": ["recentes (clique = ref., Shift+clique = ref. 2):", "recientes (clic = ref., Mayús+clic = ref. 2):", "最近使用(点击 = 参考图,Shift+点击 = 参考图 2):"], "✕ clear": ["✕ limpar", "✕ borrar", "✕ 清除"], "✕ clear all": ["✕ limpar tudo", "✕ borrar todo", "✕ 全部清除"], "Use global": ["Usar global", "Usar global", "使用全局设置"],
|
||||
"empty slots use the person's reference (Cast) or the global one (Prompts & refs tab)": ["campos vazios usam a referência da pessoa (Elenco) ou a global (aba Prompts e referências)", "los campos vacíos usan la referencia de la persona (Reparto) o la global (Prompts y referencias)", "空白项使用人物参考图(角色)或全局参考图(提示词和参考图选项卡)"],
|
||||
"Full frame": ["Quadro inteiro", "Fotograma completo", "整帧"], "The whole frame is regenerated. The mask, if any, only feeds {target} / the setting picture.": ["O quadro inteiro é gerado novamente. A máscara, se houver, só fornece {target} / a imagem de cenário.", "Se vuelve a generar todo el fotograma. La máscara, si existe, solo aporta {target} / la imagen del escenario.", "重新生成整帧。蒙版(如有)仅用于提供 {target} / 场景参考图。"],
|
||||
"Only the mask is regenerated, on the whole frame: the rest stays exactly as it was. No crop / uncrop.": ["Somente a área da máscara é regenerada; o restante permanece igual. Sem recorte.", "Solo se regenera el área de la máscara; el resto permanece igual. Sin recorte.", "仅重新生成蒙版区域,其余部分保持不变。不进行裁剪。"],
|
||||
"A box around the mask is regenerated (more pixels for a small subject); BFS Shot Join pastes it back.": ["A caixa ao redor da máscara é regenerada (mais pixels para pessoas pequenas); o BFS Shot Join cola o resultado de volta.", "Se regenera el cuadro alrededor de la máscara (más píxeles para sujetos pequeños); BFS Shot Join lo pega de vuelta.", "重新生成蒙版周围的区域(为较小主体提供更多像素);BFS Shot Join 会将其贴回。"],
|
||||
"Inside the crop, only the mask is regenerated: the most detail with the background kept.": ["Dentro do recorte, somente a máscara é regenerada: mais detalhes e fundo preservado.", "Dentro del recorte, solo se regenera la máscara: más detalle y fondo conservado.", "仅重新生成裁剪区域内的蒙版:细节更多,同时保留背景。"],
|
||||
"The whole frame is generated (the pose follows the guide, the new subject may be bigger); BFS Shot Join pastes only the subject onto the original: background exact.": ["O quadro inteiro é gerado (a pose segue o guia e o novo sujeito pode ser maior); o BFS Shot Join cola apenas o sujeito no original e preserva o fundo.", "Se genera todo el fotograma (la pose sigue la guía y el sujeto puede ser más grande); BFS Shot Join pega solo al sujeto sobre el original y conserva el fondo.", "生成整帧(姿势遵循引导,主体可更大);BFS Shot Join 仅将主体贴回原画面,背景保持原样。"],
|
||||
"🎯 Points…": ["🎯 Pontos…", "🎯 Puntos…", "🎯 点…"], "👁 Preview": ["👁 Prévia", "👁 Vista previa", "👁 预览"], "✕ Clear": ["✕ Limpar", "✕ Borrar", "✕ 清除"],
|
||||
"🎞 mask video": ["🎞 vídeo de máscara", "🎞 vídeo de máscara", "🎞 蒙版视频"], "↳ global prompt": ["↳ prompt global", "↳ prompt global", "↳ 全局提示词"], "— no prompt —": ["— sem prompt —", "— sin prompt —", "— 无提示词 —"],
|
||||
"▶ Play all": ["▶ Reproduzir todos", "▶ Reproducir todos", "▶ 播放全部"], "■ Stop": ["■ Parar", "■ Detener", "■ 停止"], "loop shot": ["repetir plano", "repetir plano", "循环播放镜头"],
|
||||
"Click the timeline to seek. Play all skips shots that will not run.": ["Clique na linha do tempo para buscar. Reproduzir todos ignora planos que não serão executados.", "Haz clic en la línea de tiempo para buscar. Reproducir todos omite los planos que no se ejecutarán.", "点击时间轴进行定位。播放全部会跳过不运行的镜头。"],
|
||||
"Reference & prompt": ["Referência e prompt", "Referencia y prompt", "参考图和提示词"], "Who is replaced": ["Quem será substituído", "Quién se reemplaza", "替换对象"], "Continuity": ["Continuidade", "Continuidad", "连续性"],
|
||||
"Global reference & prompt": ["Referência e prompt globais", "Referencia y prompt globales", "全局参考图和提示词"], "Describe references ({details})": ["Descrever referências ({details})", "Describir referencias ({details})", "描述参考图({details})"],
|
||||
"Custom": ["Personalizado", "Personalizado", "自定义"], "short": ["curto", "breve", "简短"], "full body": ["corpo inteiro", "cuerpo entero", "全身"], "head / face": ["cabeça / rosto", "cabeza / rostro", "头部/面部"], "face attributes": ["características do rosto", "rasgos faciales", "面部特征"], "outfit": ["roupa", "ropa", "服装"],
|
||||
"Apply suggestions → all": ["Aplicar sugestões → todos", "Aplicar sugerencias → todos", "应用建议 → 全部"], "fit the prompt": ["ajustar ao prompt", "ajustar al prompt", "适配提示词"],
|
||||
"Connect a Qwen3-VL (CLIPLoader) to the planner's vlm input. The VLM buttons work after the workflow has run once with it connected (ComfyUI only hands models to nodes when they run). Each shot's suggestion shows in its editor.": ["Conecte um Qwen3-VL (CLIPLoader) à entrada vlm do planner. Os botões do VLM funcionam depois que o workflow executar uma vez com ele conectado. A sugestão de cada plano aparece no editor.", "Conecta un Qwen3-VL (CLIPLoader) a la entrada vlm del planificador. Los botones del VLM funcionan después de ejecutar el flujo una vez con él conectado. La sugerencia de cada plano aparece en su editor.", "将 Qwen3-VL(CLIPLoader)连接到规划器的 vlm 输入。工作流连接后运行一次,VLM 按钮即可使用。每个镜头的建议会显示在其编辑器中。"],
|
||||
"look at every shot and suggests the mask, a description and whether to run it": ["analisa cada plano e sugere máscara, descrição e se deve ser executado", "analiza cada plano y sugiere una máscara, una descripción y si debe ejecutarse", "分析每个镜头并建议蒙版、描述及是否运行"],
|
||||
"🎯 Track → whole video": ["🎯 Rastrear → vídeo inteiro", "🎯 Seguir → vídeo completo", "🎯 跟踪 → 整个视频"], "Save (this shot)": ["Salvar (este plano)", "Guardar (este plano)", "保存(此镜头)"], "Save → all shots": ["Salvar → todos os planos", "Guardar → todos los planos", "保存 → 所有镜头"], "Cancel": ["Cancelar", "Cancelar", "取消"], "Clear points": ["Limpar pontos", "Borrar puntos", "清除点"], "👁 Segment": ["👁 Segmentar", "👁 Segmentar", "👁 分割"],
|
||||
"Apply these points to EVERY shot, each on its frame at the same relative position (a subject that stays in place)": ["Aplicar estes pontos a TODOS os planos, na mesma posição relativa (para um sujeito que não muda de lugar)", "Aplicar estos puntos a TODOS los planos, en la misma posición relativa (si el sujeto no se mueve)", "将这些点应用到每个镜头的相同相对位置(适用于位置不变的主体)"],
|
||||
"the range starts at the player's frame": ["o trecho começa no frame atual do player", "el rango empieza en el fotograma actual", "范围从播放器当前帧开始"], "the range ends after the player's frame": ["o trecho termina após o frame atual do player", "el rango termina después del fotograma actual", "范围在播放器当前帧之后结束"],
|
||||
"Nothing to fix in the shots that run.": ["Nenhum problema nos planos que serão executados.", "No hay nada que corregir en los planos que se ejecutarán.", "将运行的镜头没有需要修复的问题。"], "all good": ["tudo certo", "todo bien", "一切正常"], "warning(s)": ["aviso(s)", "aviso(s)", "条警告"], "note(s)": ["observação(ões)", "nota(s)", "条备注"],
|
||||
"quick tests": ["testes rápidos", "pruebas rápidas", "快速测试"], "from the start of the range": ["a partir do início do trecho", "desde el inicio del rango", "从范围起点开始"], "Each run generates one shot and stores it. Nodes after BFS Shot Join only run on the last shot, with the full video.": ["Cada execução gera e salva um plano. Os nodes após BFS Shot Join só executam no último plano, com o vídeo completo.", "Cada ejecución genera y guarda un plano. Los nodos posteriores a BFS Shot Join solo se ejecutan en el último plano, con el vídeo completo.", "每次运行生成并保存一个镜头。BFS Shot Join 后的节点仅在最后一个镜头运行,并接收完整视频。"],
|
||||
"skipped shots do not run": ["planos ignorados não são executados", "los planos omitidos no se ejecutan", "跳过的镜头不会运行"], "only shots that run": ["somente planos executados", "solo los planos que se ejecutan", "仅统计将运行的镜头"],
|
||||
"off": ["desligado", "desactivado", "关闭"], "on": ["ligado", "activado", "开启"], "run": ["executar", "ejecutar", "运行"], "skip": ["ignorar", "omitir", "跳过"], "filters decide": ["filtros decidem", "deciden los filtros", "由筛选条件决定"], "always run": ["sempre executar", "ejecutar siempre", "始终运行"], "always skip": ["sempre ignorar", "omitir siempre", "始终跳过"], "⏸ Disable": ["⏸ Desativar", "⏸ Desactivar", "⏸ 禁用"], "▶ Enable": ["▶ Ativar", "▶ Activar", "▶ 启用"], "✂ Split": ["✂ Dividir", "✂ Dividir", "✂ 拆分"], "⇥ Merge": ["⇥ Unir", "⇥ Unir", "⇥ 合并"],
|
||||
"Advanced": ["Avançado", "Avanzado", "高级"], "English": ["Inglês", "Inglés", "英语"], "Português": ["Português", "Portugués", "葡萄牙语"], "Español": ["Espanhol", "Español", "西班牙语"], "中文": ["Chinês", "Chino", "中文"],
|
||||
"Color Guide": ["Guia de cores", "Guía de color", "颜色引导"], "Video": ["Vídeo", "Vídeo", "视频"], "Shots": ["Planos", "Planos", "镜头"], "Prompts": ["Prompts", "Prompts", "提示词"],
|
||||
"Guide color · Ref 1": ["Cor guia · Ref. 1", "Color guía · Ref. 1", "引导颜色 · 参考图 1"], "Guide color · Ref 2": ["Cor guia · Ref. 2", "Color guía · Ref. 2", "引导颜色 · 参考图 2"],
|
||||
"These colors are defaults for the global references. Turn on Color Guide on a shot to apply them; shot-level colors can override them.": ["Estas cores são os padrões das referências globais. Ative o Guia de cores no plano para aplicá-las; cada plano pode substituí-las.", "Estos colores son los valores predeterminados de las referencias globales. Activa la Guía de color en un plano para aplicarlos; cada plano puede cambiarlos.", "这些颜色是全局参考图的默认值。启用镜头的颜色引导即可应用;镜头也可以单独覆盖。"],
|
||||
"Audio": ["Áudio", "Audio", "音频"], "use the video's audio": ["usar o áudio do vídeo", "usar el audio del vídeo", "使用视频音频"], "silent track": ["faixa silenciosa", "pista silenciosa", "静音音轨"],
|
||||
"auto": ["automático", "automático", "自动"], "fixed": ["fixo", "fijo", "固定"], "single": ["único", "único", "单个"], "adaptive": ["adaptativo", "adaptativo", "自适应"], "content": ["conteúdo", "contenido", "内容"], "builtin": ["integrado", "integrado", "内置"],
|
||||
"PySceneDetect adaptive": ["PySceneDetect adaptativo", "PySceneDetect adaptativo", "PySceneDetect 自适应"], "PySceneDetect content": ["PySceneDetect por conteúdo", "PySceneDetect por contenido", "PySceneDetect 内容检测"], "Built-in": ["Integrado", "Integrado", "内置"],
|
||||
"Auto loop": ["Loop automático", "Bucle automático", "自动循环"], "Queue loop": ["Loop em fila", "Bucle en cola", "队列循环"], "queue loop": ["loop em fila", "bucle en cola", "队列循环"], "auto loop": ["loop automático", "bucle automático", "自动循环"],
|
||||
"no audio": ["sem áudio", "sin audio", "无音频"], "silent track": ["faixa silenciosa", "pista silenciosa", "静音音轨"], "timeline": ["linha do tempo", "línea de tiempo", "时间轴"], "generate at": ["gerar em", "generar a", "生成尺寸"],
|
||||
"Max person size (0-1 of frame)": ["Tamanho mínimo da pessoa (0–1 do quadro)", "Tamaño mínimo de persona (0–1 del fotograma)", "人物最小尺寸(占画面 0–1)"], "Min person size (0-1 of frame)": ["Tamanho mínimo da pessoa (0–1 do quadro)", "Tamaño mínimo de persona (0–1 del fotograma)", "人物最小尺寸(占画面 0–1)"],
|
||||
"Max people (0 = any)": ["Máx. pessoas (0 = qualquer)", "Máx. personas (0 = cualquiera)", "人物上限(0 = 不限)"], "Min frames (0 = off)": ["Mín. frames (0 = desligado)", "Mín. fotogramas (0 = desactivado)", "最少帧数(0 = 关闭)"], "Frames sampled / shot": ["Frames amostrados por plano", "Fotogramas muestreados por plano", "每个镜头采样帧数"], "Dark below (0-1)": ["Escuro abaixo de (0–1)", "Oscuro por debajo de (0–1)", "暗度低于(0–1)"], "Static below": ["Estático abaixo de", "Estático por debajo de", "静止度低于"],
|
||||
"e.g. 0.03 skips wide shots": ["ex.: 0,03 ignora planos abertos", "p. ej., 0,03 omite planos amplios", "例如 0.03 会跳过远景镜头"], "skip crowds": ["ignorar multidões", "omitir multitudes", "跳过人群"], "re-analyses": ["reanalisar", "vuelve a analizar", "重新分析"],
|
||||
"Describe target": ["Descrever alvo", "Describir objetivo", "描述目标"], "🧑 Describe target": ["🧑 Descrever alvo", "🧑 Describir objetivo", "🧑 描述目标"], "new subject": ["novo sujeito", "nuevo sujeto", "新主体"], "person (or dog, cat, car… what the result shows)": ["pessoa (ou cão, gato, carro… o que aparecerá no resultado)", "persona (o perro, gato, coche… lo que aparecerá en el resultado)", "人物(或狗、猫、汽车等结果中的主体)"],
|
||||
"The second person (optional)": ["Segunda pessoa (opcional)", "Segunda persona (opcional)", "第二个人(可选)"], "Follow cast": ["Seguir identidade", "Seguir reparto", "跟随角色"], "manual identity": ["identidade manual", "identidad manual", "手动身份"],
|
||||
"Guide video": ["Vídeo guia", "Vídeo guía", "引导视频"], "Reference": ["Referência", "Referencia", "参考图"], "Prompt": ["Prompt", "Prompt", "提示词"], "reference": ["referência", "referencia", "参考图"],
|
||||
"on": ["ativado", "activado", "开启"], "off": ["desativado", "desactivado", "关闭"], "not analysed": ["não analisado", "sin analizar", "未分析"], "not analyzed": ["não analisado", "sin analizar", "未分析"],
|
||||
"No shots yet": ["Nenhum plano ainda", "Aún no hay planos", "尚无镜头"], "No video": ["Nenhum vídeo", "Sin vídeo", "没有视频"], "No faces": ["Nenhum rosto", "Sin rostros", "没有人脸"], "no faces": ["sem rostos", "sin rostros", "没有人脸"],
|
||||
"cut": ["corte", "corte", "切换"], "run": ["executar", "ejecutar", "运行"], "skip": ["ignorar", "omitir", "跳过"], "skip this shot": ["ignorar este plano", "omitir este plano", "跳过此镜头"],
|
||||
"people": ["pessoas", "personas", "人物"], "person": ["pessoa", "persona", "人物"], "shots": ["planos", "planos", "镜头"], "shot(s)": ["plano(s)", "plano(s)", "个镜头"], "linked": ["vinculada", "vinculada", "已关联"], "main": ["principal", "principal", "主要"],
|
||||
"No VLM yet": ["Nenhum VLM carregado", "Aún no hay un VLM", "尚未加载 VLM"], "Connect a Qwen3-VL": ["Conecte um Qwen3-VL", "Conecta un Qwen3-VL", "连接 Qwen3-VL"],
|
||||
"↻ Status": ["↻ Status", "↻ Estado", "↻ 状态"], "⟲ Reset loop": ["⟲ Reiniciar loop", "⟲ Reiniciar bucle", "⟲ 重置循环"], "Queue loop": ["Loop em fila", "Bucle en cola", "队列循环"],
|
||||
"Start here": ["Começar aqui", "Empezar aquí", "从此处开始"], "End here": ["Terminar aqui", "Terminar aquí", "在此处结束"], "reset": ["redefinir", "restablecer", "重置"], "all shots": ["todos os planos", "todos los planos", "所有镜头"], "this shot": ["este plano", "este plano", "此镜头"],
|
||||
"Describe target": ["Descrever alvo", "Describir objetivo", "描述目标"], "Settings": ["Configurações", "Ajustes", "设置"], "mask text": ["texto da máscara", "texto de máscara", "蒙版描述"],
|
||||
"The global reference and prompt are used when a shot has no values of its own.": ["A referência e o prompt globais são usados quando um plano não tem valores próprios.", "La referencia y el prompt globales se usan cuando un plano no tiene valores propios.", "镜头没有单独设置时会使用全局参考图和提示词。"],
|
||||
"A VLM is optional": ["O VLM é opcional", "El VLM es opcional", "VLM 为可选项"], "Optional": ["Opcional", "Opcional", "可选"], "optional": ["opcional", "opcional", "可选"],
|
||||
"person in white": ["pessoa de branco", "persona de blanco", "穿白衣的人"], "woman in pink top": ["mulher de blusa rosa", "mujer con blusa rosa", "穿粉色上衣的女人"],
|
||||
"nothing found": ["nada encontrado", "no se encontró nada", "未找到内容"], "not found": ["não encontrado", "no encontrado", "未找到"], "Preview": ["Prévia", "Vista previa", "预览"],
|
||||
"Mask only / Crop + mask": ["Somente máscara / Recorte + máscara", "Solo máscara / Recorte + máscara", "仅蒙版 / 裁剪并蒙版"], "per shot (planner)": ["por plano (planner)", "por plano (planificador)", "逐镜头(规划器)"],
|
||||
"Person": ["Pessoa", "Persona", "人物"], "shot": ["plano", "plano", "镜头"], "shot(s)": ["plano(s)", "plano(s)", "个镜头"],
|
||||
"mask covers": ["máscara cobre", "la máscara cubre", "蒙版覆盖"], "nothing to fix": ["nada a corrigir", "nada que corregir", "无需修复"],
|
||||
"Camera cuts: shots start at the detected cuts, long ones are split evenly. Fixed length: equal parts no longer than the maximum.": ["Cortes de câmera: os planos começam nos cortes detectados; planos longos são divididos igualmente. Duração fixa: partes iguais limitadas pelo máximo.", "Cortes de cámara: los planos empiezan en los cortes detectados; los largos se dividen por igual. Duración fija: partes iguales con el máximo indicado.", "镜头切换:镜头从检测到的切点开始,较长镜头会均匀拆分。固定时长:拆分为不超过上限的等长片段。"],
|
||||
"Cut detector. PySceneDetect adaptive works on most footage (fast camera moves don't trigger it); content is stricter on hard cuts; built-in needs no package.": ["Detector de cortes. O modo adaptativo do PySceneDetect funciona na maioria dos vídeos (movimentos rápidos de câmera não acionam cortes); o modo conteúdo é mais rigoroso; o integrado não exige pacote.", "Detector de cortes. El modo adaptativo de PySceneDetect funciona en la mayoría de vídeos (los movimientos rápidos no activan cortes); contenido es más estricto; el integrado no requiere paquetes.", "镜头切换检测器。PySceneDetect 自适应模式适用于大多数素材(快速运镜不会触发切点);内容模式对硬切更严格;内置模式无需额外软件包。"],
|
||||
"How easily a change counts as a cut. Higher finds more cuts (and more false ones).": ["O quanto uma mudança é considerada corte. Um valor maior encontra mais cortes e também mais falsos positivos.", "Facilidad con la que un cambio cuenta como corte. Un valor mayor detecta más cortes y también más falsos positivos.", "变化被判定为切点的难易程度。数值越高,检测到的切点越多,误报也越多。"],
|
||||
"Frame counts the model accepts. H3: 17n+5, LTX/Wan: 8n+1, Wan: 4n+1. Each shot is generated at the next valid length and trimmed back.": ["Contagens de frames aceitas pelo modelo. H3: 17n+5; LTX/Wan: 8n+1; Wan: 4n+1. Cada plano é gerado no próximo comprimento válido e depois aparado.", "Cantidad de fotogramas que acepta el modelo. H3: 17n+5; LTX/Wan: 8n+1; Wan: 4n+1. Cada plano se genera con la siguiente duración válida y luego se recorta.", "模型接受的帧数规则。H3:17n+5;LTX/Wan:8n+1;Wan:4n+1。每个镜头按下一个有效长度生成后再裁短。"],
|
||||
"Longest shot sent to the model. Keep it within what the model / LoRA was trained on (H3 body swap: about 4.5 s).": ["Duração máxima enviada ao modelo. Mantenha dentro do que o modelo/LoRA aprendeu (body swap H3: cerca de 4,5 s).", "Duración máxima enviada al modelo. Respeta lo usado en el entrenamiento del modelo/LoRA (body swap H3: unos 4,5 s).", "发送给模型的镜头最长时长。应符合模型/LoRA 的训练长度(H3 换人约 4.5 秒)。"],
|
||||
"Shorter shots merge into a neighbour when the merge still fits the maximum.": ["Planos mais curtos são unidos ao vizinho quando a junção ainda respeita o máximo.", "Los planos más cortos se unen al vecino si la unión respeta el máximo.", "较短镜头会在合并后仍未超过上限时与相邻镜头合并。"],
|
||||
"The planner works on this frame rate (the source is resampled). Generated video and audio use it too.": ["O planner usa esta taxa de frames (o vídeo de origem é reamostrado). O vídeo e áudio gerados também a usam.", "El planificador usa esta frecuencia de fotogramas (el origen se remuestrea). El vídeo y el audio generados también la usan.", "规划器使用此帧率(源视频会重新采样)。生成的视频和音频也会使用此帧率。"],
|
||||
"Only the first N shots (0 = all). Handy for quick tests.": ["Somente os primeiros N planos (0 = todos). Útil para testes rápidos.", "Solo los primeros N planos (0 = todos). Útil para pruebas rápidas.", "仅处理前 N 个镜头(0 = 全部)。适合快速测试。"],
|
||||
"Only the first N seconds of the video (0 = all).": ["Somente os primeiros N segundos do vídeo (0 = todos).", "Solo los primeros N segundos del vídeo (0 = todos).", "仅处理视频前 N 秒(0 = 全部)。"],
|
||||
"Generation size: the source aspect ratio at this pixel area (0.15 ≈ 512×288). Cropped shots use it for the crop.": ["Tamanho da geração: mantém a proporção do vídeo nessa área de pixels (0,15 ≈ 512×288). Planos recortados usam esse valor no recorte.", "Tamaño de generación: mantiene la proporción de origen con esta área de píxeles (0,15 ≈ 512×288). Los planos recortados usan este valor.", "生成尺寸:按此像素面积保持源视频宽高比(0.15 ≈ 512×288)。裁剪镜头使用此尺寸。"],
|
||||
"Width and height snap to this multiple (32 for H3 and most video models).": ["Largura e altura são ajustadas a este múltiplo (32 para H3 e a maioria dos modelos de vídeo).", "El ancho y alto se ajustan a este múltiplo (32 para H3 y la mayoría de modelos de vídeo).", "宽度和高度会对齐到此倍数(H3 和多数视频模型使用 32)。"],
|
||||
"Auto loop: every shot in one queue run. Queue loop: one shot per run, stored on disk, the join outputs the video after the last shot.": ["Loop automático: todos os planos em uma execução da fila. Loop em fila: um plano por execução, salvo em disco; o join gera o vídeo após o último plano.", "Bucle automático: todos los planos en una ejecución. Bucle en cola: un plano por ejecución, guardado en disco; el join genera el vídeo al terminar el último.", "自动循环:一次队列运行所有镜头。队列循环:每次运行一个镜头并保存到磁盘;最后一个镜头完成后由 Join 输出视频。"],
|
||||
"Shots that do not run: keep the original video there, or remove them (with their audio).": ["Planos não executados: manter o vídeo original nesses trechos ou removê-los junto com o áudio.", "Planos que no se ejecutan: conserva el vídeo original en ese tramo o elimínalo junto con su audio.", "未运行的镜头:保留对应原视频片段,或连同音频一起移除。"],
|
||||
"Context kept around the mask's box, as a % of the box. More context blends better; less gives the model more pixels on the subject.": ["Contexto mantido ao redor da caixa da máscara, como porcentagem da caixa. Mais contexto mistura melhor; menos dá mais pixels ao sujeito.", "Contexto alrededor del cuadro de la máscara, como porcentaje. Más contexto mezcla mejor; menos ofrece más píxeles al sujeto.", "蒙版边框周围保留的上下文比例。上下文越多越容易融合;越少则主体像素越多。"],
|
||||
"Grow the mask by this many pixels: before pasting a cropped result back, and for the generation mask (inpaint 'only the mask'). Raise it when the new person is bigger than the old one (longer hair, wider body).": ["Expande a máscara por esta quantidade de pixels antes de colar o recorte e na máscara de geração. Aumente se a nova pessoa for maior (cabelo mais comprido, corpo mais largo).", "Amplía la máscara esta cantidad de píxeles antes de pegar el recorte y para la máscara de generación. Auméntala si la persona nueva es más grande.", "粘贴裁剪结果前以及生成蒙版时,将蒙版扩展指定像素。新人物更大时(头发更长、身体更宽)可调高。"],
|
||||
"Soft edge of the paste-back, in pixels.": ["Suavização da borda ao colar de volta, em pixels.", "Suavizado del borde al pegar de vuelta, en píxeles.", "贴回时边缘的柔化像素数。"],
|
||||
"Hold the mask this many frames before and after: removes flicker where SAM 3 drops a frame.": ["Estende a máscara por esta quantidade de frames antes e depois; reduz falhas quando o SAM 3 perde um frame.", "Mantiene la máscara estos fotogramas antes y después; reduce parpadeos cuando SAM 3 pierde uno.", "蒙版前后延续指定帧数,减少 SAM 3 漏掉某帧时的闪烁。"],
|
||||
"Square blocks of this size (0 = off). 16 matches H3's latent grid, useful as an inpainting mask.": ["Blocos quadrados deste tamanho (0 = desligado). 16 corresponde à grade latente do H3 e pode ser útil como máscara de inpainting.", "Bloques cuadrados de este tamaño (0 = desactivado). 16 coincide con la cuadrícula latente de H3 y sirve para inpainting.", "使用此尺寸的方块(0 = 关闭)。16 与 H3 潜空间网格匹配,可用于修补蒙版。"],
|
||||
"SAM 3 score needed to keep a text match. Lower finds more (and more wrong) objects.": ["Pontuação mínima do SAM 3 para manter uma correspondência por texto. Valores menores encontram mais objetos, inclusive incorretos.", "Puntuación mínima de SAM 3 para conservar una coincidencia de texto. Un valor menor encuentra más objetos, también erróneos.", "保留文本匹配所需的 SAM 3 分数。调低会找到更多对象,也会增加误检。"],
|
||||
"How many matches of the text prompt are tracked together.": ["Quantas correspondências do texto serão rastreadas juntas.", "Cuántas coincidencias del texto se siguen a la vez.", "同时跟踪多少个符合文本的对象。"],
|
||||
"Only the mask: the result replaces the masked pixels. The whole box: the full crop is pasted with feathered borders.": ["Somente máscara: o resultado substitui os pixels mascarados. Caixa inteira: todo o recorte é colado com bordas suaves.", "Solo máscara: el resultado sustituye los píxeles enmascarados. Cuadro completo: se pega todo el recorte con bordes suavizados.", "仅蒙版:结果替换蒙版像素。整个框:粘贴完整裁剪区域并柔化边缘。"],
|
||||
"How many frames of each shot the VLM looks at.": ["Quantos frames de cada plano o VLM analisa.", "Cuántos fotogramas de cada plano analiza el VLM.", "VLM 分析每个镜头的帧数。"], "Longest VLM answer. Raise it for long descriptions.": ["Tamanho máximo da resposta do VLM. Aumente para descrições longas.", "Longitud máxima de respuesta del VLM. Auméntala para descripciones largas.", "VLM 回复的最大长度。长描述可调高。"],
|
||||
};
|
||||
export function t(value) {
|
||||
const s = String(value ?? ""), lang = uiLanguage.value;
|
||||
if (lang === "en") return s;
|
||||
const entry = TEXT[s];
|
||||
if (entry) return entry[{ pt: 0, es: 1, zh: 2 }[lang]];
|
||||
let m;
|
||||
if ((m = s.match(/^Person (\d+)$/))) return `${t("Person")} ${m[1]}`;
|
||||
if ((m = s.match(/^Shot #(\d+)( · points)?$/))) return `${t("Shot")} #${m[1]}${m[2] ? ` · ${lang === "pt" ? "pontos" : lang === "es" ? "puntos" : "点"}` : ""}`;
|
||||
if ((m = s.match(/^(\d+) people · (\d+) linked$/))) return lang === "pt" ? `${m[1]} pessoas · ${m[2]} vinculadas` : lang === "es" ? `${m[1]} personas · ${m[2]} vinculadas` : `${m[1]} 人物 · 已关联 ${m[2]} 人`;
|
||||
if ((m = s.match(/^(\d+) shot\(s\) will run$/))) return lang === "pt" ? `${m[1]} plano(s) serão executados` : lang === "es" ? `se ejecutarán ${m[1]} plano(s)` : `将运行 ${m[1]} 个镜头`;
|
||||
if ((m = s.match(/^(\d+) suggestions$/))) return lang === "pt" ? `${m[1]} sugestões` : lang === "es" ? `${m[1]} sugerencias` : `${m[1]} 条建议`;
|
||||
if ((m = s.match(/^(\d+) shot\(s\)$/))) return lang === "pt" ? `${m[1]} plano(s)` : lang === "es" ? `${m[1]} plano(s)` : `${m[1]} 个镜头`;
|
||||
if ((m = s.match(/^(\d+) warning\(s\) · (\d+) note\(s\)$/))) return lang === "pt" ? `${m[1]} aviso(s) · ${m[2]} observação(ões)` : lang === "es" ? `${m[1]} aviso(s) · ${m[2]} nota(s)` : `${m[1]} 条警告 · ${m[2]} 条备注`;
|
||||
return s;
|
||||
}
|
||||
function localizeProps(props) {
|
||||
if (!props || typeof props !== "object") return props;
|
||||
let changed = false; const out = { ...props };
|
||||
for (const k of ["title", "placeholder", "aria-label"]) if (typeof out[k] === "string") { const v = t(out[k]); changed ||= v !== out[k]; out[k] = v; }
|
||||
return changed ? out : props;
|
||||
}
|
||||
function localizeChildren(value) {
|
||||
if (typeof value === "string") return t(value);
|
||||
if (Array.isArray(value)) return value.map(localizeChildren);
|
||||
return value;
|
||||
}
|
||||
// All view modules use this h wrapper, so labels, placeholders, tooltips and section titles stay in sync.
|
||||
export function h(type, props, children) {
|
||||
const hasChildrenArg = arguments.length >= 3;
|
||||
if (!hasChildrenArg && props?.__v_isVNode) return vueH(type, null, props);
|
||||
const p = localizeProps(props);
|
||||
if (hasChildrenArg) return vueH(type, p, localizeChildren(children));
|
||||
if (typeof props === "string" || Array.isArray(props)) return vueH(type, localizeChildren(props));
|
||||
return vueH(type, p);
|
||||
}
|
||||
|
||||
export const GRIDS = { "H3 (17n+5)": [17, 5], "LTX / Wan (8n+1)": [8, 1], "Wan (4n+1)": [4, 1], "any": [1, 0] };
|
||||
export const DEFAULTS = {
|
||||
video: "", fps: 24, grid: "H3 (17n+5)", mode: "shots", max_s: 4.5, min_s: 1.0, sensitivity: 0.5,
|
||||
max_parts: 0, max_total_s: 0, start_s: 0, end_s: 0, start_f: null, end_f: null, track: null, bounds: [], segs: [], global_ref: "", global_ref2: "", global_prompt: "", global_color_guide_ref1: "#00ff00", global_color_guide_ref2: "#9b30ff", mask_video: "",
|
||||
megapixels: 0.15, multiple: 32, detector: "adaptive", run: "auto", auto_continue: true,
|
||||
skip_fill: "original", audio_mode: "auto", ref_details: {}, mask_cfg: {}, vlm_cfg: {}, cast: {}, cast_assign: true, cast_split: false, cast_only: false,
|
||||
filters: { person: false, min_person_area: 0, max_persons: 0, face: false, skip_dark: false, dark_level: 0.06,
|
||||
skip_static: false, static_level: 0.004, min_frames: 0, samples: 6 },
|
||||
};
|
||||
|
||||
// global mask settings (same defaults as the server's DEFAULT_MASK)
|
||||
export const MASK_DEFAULTS = { threshold: 0.5, max_objects: 4, invert: false, fill_holes: true, temporal_expand: 2, blockify: 0,
|
||||
padding: 0.15, expand: 16, feather: 12, paste: "mask" };
|
||||
|
||||
export const VLM_DEFAULTS = { enabled: false, frames: 3, max_tokens: 1024, auto_segment: true, auto_shot: true, instruction: "",
|
||||
describe_preset: "short", describe_custom: "", fit_prompt: true, temperature: 0, seed: 0 };
|
||||
|
||||
// hover help for every setting (the 📖 Guide button opens the full documentation)
|
||||
export const TIPS = {
|
||||
"Mode": "Camera cuts: shots start at the detected cuts, long ones are split evenly. Fixed length: equal parts no longer than the maximum.",
|
||||
"Detector": "Cut detector. PySceneDetect adaptive works on most footage (fast camera moves don't trigger it); content is stricter on hard cuts; built-in needs no package.",
|
||||
"Sensitivity": "How easily a change counts as a cut. Higher finds more cuts (and more false ones).",
|
||||
"Frame grid": "Frame counts the model accepts. H3: 17n+5, LTX/Wan: 8n+1, Wan: 4n+1. Each shot is generated at the next valid length and trimmed back.",
|
||||
"Max seconds / shot": "Longest shot sent to the model. Keep it within what the model / LoRA was trained on (H3 body swap: about 4.5 s).",
|
||||
"Min seconds / shot": "Shorter shots merge into a neighbour when the merge still fits the maximum.",
|
||||
"Timeline fps": "The planner works on this frame rate (the source is resampled). Generated video and audio use it too.",
|
||||
"Max shots": "Only the first N shots (0 = all). Handy for quick tests.",
|
||||
"Max total seconds": "Only the first N seconds of the video (0 = all).",
|
||||
"Megapixels": "Generation size: the source aspect ratio at this pixel area (0.15 ≈ 512×288). Cropped shots use it for the crop.",
|
||||
"Size multiple": "Width and height snap to this multiple (32 for H3 and most video models).",
|
||||
"Run": "Auto loop: every shot in one queue run. Queue loop: one shot per run, stored on disk, the join outputs the video after the last shot.",
|
||||
"Skipped shots in the output": "Shots that do not run: keep the original video there, or remove them (with their audio).",
|
||||
"Crop padding": "Context kept around the mask's box, as a % of the box. More context blends better; less gives the model more pixels on the subject.",
|
||||
"Expand": "Grow the mask by this many pixels: before pasting a cropped result back, and for the generation mask (inpaint 'only the mask'). Raise it when the new person is bigger than the old one (longer hair, wider body).",
|
||||
"Feather": "Soft edge of the paste-back, in pixels.",
|
||||
"Temporal expand": "Hold the mask this many frames before and after: removes flicker where SAM 3 drops a frame.",
|
||||
"Blockify": "Square blocks of this size (0 = off). 16 matches H3's latent grid, useful as an inpainting mask.",
|
||||
"Detection threshold": "SAM 3 score needed to keep a text match. Lower finds more (and more wrong) objects.",
|
||||
"Max objects": "How many matches of the text prompt are tracked together.",
|
||||
"Paste back": "Only the mask: the result replaces the masked pixels. The whole box: the full crop is pasted with feathered borders.",
|
||||
"Frames per shot": "How many frames of each shot the VLM looks at.",
|
||||
"Max tokens": "Longest VLM answer. Raise it for long descriptions.",
|
||||
};
|
||||
|
||||
export const snapUp = (n, grid) => {
|
||||
const [s, o] = GRIDS[grid] || GRIDS["H3 (17n+5)"]; n = Math.max(1, n | 0);
|
||||
if (s === 1) return n; if (n <= o) return o; return o + s * Math.ceil((n - o) / s);
|
||||
};
|
||||
export const snapDown = (n, grid) => {
|
||||
const [s, o] = GRIDS[grid] || GRIDS["H3 (17n+5)"]; n = Math.max(1, n | 0);
|
||||
if (s === 1) return n; if (n <= o) return o; return o + s * Math.floor((n - o) / s);
|
||||
};
|
||||
export const hue = i => `hsl(${(i * 47 + 200) % 360} 55% 46%)`;
|
||||
export const fmtT = (f, fps) => { const s = f / fps; return `${Math.floor(s / 60)}:${(s % 60).toFixed(2).padStart(5, "0")}`; };
|
||||
export const segKey = s => `${s.start}-${s.end}`;
|
||||
// a shot has a mask: its own mask video, SAM 3 text or points, or the plan's mask video for the whole video
|
||||
// (the planner's `mask` input is only known when the workflow runs)
|
||||
export const hasMask = s => !!(s.mask?.video || s.mask?.text || (s.mask?.points || []).length || s.extMask);
|
||||
export const maskSource = s => s.mask?.video ? "video" : (s.mask?.text || (s.mask?.points || []).length) ? "sam"
|
||||
: s.trackMask ? "track" : s.extMask ? "global" : "";
|
||||
|
||||
// ---- small view helpers
|
||||
export const pill = (text, kind = "", title = "") => h("span", { class: ["pill", kind], title }, text);
|
||||
export const hint = (text, style = "") => h("span", { class: "hint", style }, text);
|
||||
export const fld = (label, input, help) => {
|
||||
const tip = Object.entries(TIPS).find(([k]) => String(label).startsWith(k))?.[1] || "";
|
||||
return h("div", { class: "fld", title: tip }, [h("label", tip ? [label, h("span", { class: "qm" }, " ⓘ")] : label), input,
|
||||
help ? h("div", { class: "hint" }, help) : null]);
|
||||
};
|
||||
export const check = (checked, onChange, label, title = "") =>
|
||||
h("label", { class: "chk", title }, [h("input", { type: "checkbox", checked: !!checked, onChange: e => onChange(e.target.checked) }), label]);
|
||||
export const select = (value, options, onChange, attrs = {}) => h("select", { value, onChange: e => onChange(e.target.value), ...attrs },
|
||||
options.map(o => h("option", { value: Array.isArray(o) ? o[0] : o }, Array.isArray(o) ? o[1] : o)));
|
||||
// a titled section inside a card; `open` makes it a collapsible <details>
|
||||
// what the user opened or closed wins over a section's default: re-renders (a timer, a status line) no longer snap a
|
||||
// section back shut right after it was clicked open
|
||||
const OPEN = reactive({});
|
||||
export const section = (title, body, { sub = "", right = null, open = null, cls = "" } = {}) => {
|
||||
const head = [h("span", { class: "sect" }, title), sub ? hint(sub) : null, h("span", { class: "grow" }), right];
|
||||
if (open === null) return h("div", { class: ["sec", cls] }, [h("div", { class: "sech" }, head), ...[body].flat()]);
|
||||
const key = `${cls}|${typeof title === "string" ? title : ""}`;
|
||||
const isOpen = key in OPEN ? OPEN[key] : !!open;
|
||||
return h("details", { class: ["sec", cls], open: isOpen, onToggle: e => { OPEN[key] = e.target.open; } },
|
||||
[h("summary", { class: "sech" }, [h("span", { class: "caret" }, "▸"), ...head]), ...[body].flat()]);
|
||||
};
|
||||
@@ -0,0 +1,207 @@
|
||||
// Shot Planner styles (one <style> tag, added once).
|
||||
const CSS = `
|
||||
.bsl{--bg:#16161a;--card:#1d1d23;--card2:#22222a;--line:#2d2d37;--line2:#3a3a46;--txt:#c9c9cf;--txt2:#8c8c97;--hi:#ececf1;
|
||||
--acc:#5b8cff;--acc2:#3a63e0;--ok:#7fe0a8;--warn:#ffc46b;--bad:#ff8f8f;
|
||||
font:12px/1.45 var(--font-family,system-ui,sans-serif);color:var(--txt);background:var(--bg);border-radius:10px;
|
||||
height:100%;overflow:auto;box-sizing:border-box;padding:10px;position:relative}
|
||||
.bsl.root{position:absolute;inset:0;height:auto}
|
||||
.bsl:focus{outline:none}
|
||||
.bsl *{box-sizing:border-box}
|
||||
.bsl .qm{color:#7d8bb0;cursor:help}
|
||||
.bsl .grow{flex:1}
|
||||
.bsl .row{display:flex;gap:6px;align-items:center;flex-wrap:wrap}
|
||||
.bsl .col{display:flex;flex-direction:column;gap:6px}
|
||||
.bsl .hint{font-size:10px;color:#7d7d88}
|
||||
.bsl .chk{display:inline-flex;gap:5px;align-items:center;cursor:pointer;user-select:none}
|
||||
.bsl .chk input{accent-color:var(--acc);margin:0}
|
||||
|
||||
/* header + step tabs */
|
||||
.bsl .hdr{display:flex;align-items:center;gap:8px;margin-bottom:8px}
|
||||
.bsl .ttl{font-weight:600;color:var(--hi);font-size:13px;letter-spacing:.01em}
|
||||
.bsl .tabs{display:flex;gap:4px;margin-bottom:8px;background:#111115;border:1px solid var(--line);border-radius:9px;padding:3px;position:sticky;top:-10px;z-index:7}
|
||||
.bsl .tab{flex:1;background:none;border:0;border-radius:7px;padding:6px 6px;color:var(--txt2);font-size:11px;font-weight:600;display:flex;gap:5px;align-items:center;justify-content:center}
|
||||
.bsl .tab:hover{background:#22222a;color:var(--txt)}
|
||||
.bsl .tab.on{background:var(--acc2);color:#fff}
|
||||
.bsl .tab .n{font-size:9px;font-weight:600;opacity:.75;border:1px solid currentColor;border-radius:999px;padding:0 5px;line-height:14px}
|
||||
.bsl .tab .dot{width:6px;height:6px;border-radius:50%;background:var(--warn)}
|
||||
.bsl .tab:disabled{opacity:.35}
|
||||
|
||||
/* pills, inputs, buttons */
|
||||
.bsl .pill{font-size:10px;padding:1px 7px;border-radius:999px;background:#25252d;color:#a9a9b4;border:1px solid #33333d;white-space:nowrap}
|
||||
.bsl .pill.ok{background:#173527;color:var(--ok);border-color:#245c40}
|
||||
.bsl .pill.warn{background:#3a2a12;color:var(--warn);border-color:#6a4a16}
|
||||
.bsl .pill.bad{background:#3a1a1d;color:var(--bad);border-color:#6a2c33}
|
||||
.bsl .pill.acc{background:#1d2a4d;color:#a9c1ff;border-color:#2f4a8f}
|
||||
.bsl select,.bsl input[type=number],.bsl input[type=text],.bsl textarea{background:#101014;border:1px solid #33333d;color:#e6e6ea;
|
||||
border-radius:6px;padding:4px 7px;font-size:11px;font-family:inherit;width:100%}
|
||||
.bsl select:focus,.bsl input:focus,.bsl textarea:focus{border-color:var(--acc);outline:none}
|
||||
.bsl textarea{resize:vertical;min-height:54px;font-family:ui-monospace,monospace}
|
||||
.bsl input[type=range]{width:100%;accent-color:var(--acc)}
|
||||
.bsl button{background:#26262e;border:1px solid #393945;color:#dcdce2;border-radius:6px;padding:4px 9px;font-size:11px;cursor:pointer;white-space:nowrap}
|
||||
.bsl button:hover{background:#30303a}
|
||||
.bsl button.pri{background:var(--acc2);border-color:#4b74f0;color:#fff}
|
||||
.bsl button.pri:hover{background:#4672ee}
|
||||
.bsl button.dng{background:#3a1f22;border-color:#6a2c33;color:#ffb3b3}
|
||||
.bsl button.ghost{background:none;border-color:transparent;color:var(--txt2)}
|
||||
.bsl button.ghost:hover{color:var(--txt);background:#26262e}
|
||||
.bsl button:disabled{opacity:.45;cursor:default}
|
||||
.bsl .seg-btns{display:inline-flex;border:1px solid #393945;border-radius:6px;overflow:hidden}
|
||||
.bsl .seg-btns button{border:0;border-radius:0;border-right:1px solid #393945}
|
||||
.bsl .seg-btns button:last-child{border-right:0}
|
||||
.bsl .seg-btns button.on{background:var(--acc2);color:#fff}
|
||||
|
||||
/* cards and sections */
|
||||
.bsl .card{background:var(--card);border:1px solid var(--line);border-radius:9px;padding:8px 10px;margin-bottom:8px}
|
||||
.bsl .card h5{margin:0 0 6px;font-size:11px;font-weight:600;color:#b9b9c4;text-transform:uppercase;letter-spacing:.06em;display:flex;gap:6px;align-items:center}
|
||||
.bsl .sec{border-top:1px solid var(--line);padding:8px 0 4px}
|
||||
.bsl .sec:first-child{border-top:0;padding-top:2px}
|
||||
.bsl .sech{display:flex;gap:6px;align-items:center;margin-bottom:6px;list-style:none;cursor:default}
|
||||
.bsl summary.sech{cursor:pointer;user-select:none}
|
||||
.bsl summary.sech::-webkit-details-marker{display:none}
|
||||
.bsl .sect{font-size:11px;font-weight:600;color:#b9b9c4;text-transform:uppercase;letter-spacing:.06em;white-space:nowrap}
|
||||
.bsl .caret{display:inline-block;transition:transform .15s;color:var(--txt2);font-size:10px}
|
||||
.bsl details[open]>summary .caret{transform:rotate(90deg)}
|
||||
.bsl details:not([open])>.sech{margin-bottom:0}
|
||||
.bsl .grid{display:grid;grid-template-columns:repeat(4,minmax(0,1fr));gap:6px 8px}
|
||||
.bsl .grid.g3{grid-template-columns:repeat(3,minmax(0,1fr))}
|
||||
.bsl .fld label{display:block;font-size:10px;color:var(--txt2);margin-bottom:2px}
|
||||
.bsl .err{color:var(--bad);background:#2a1517;border:1px solid #5a2228;border-radius:6px;padding:5px 8px;margin-bottom:8px;display:flex;gap:8px;align-items:center}
|
||||
.bsl .note{background:#1a2034;border:1px solid #2b3557;border-radius:7px;padding:6px 8px;color:#b8c4e6;font-size:11px}
|
||||
.bsl .note b{color:#dfe6ff}
|
||||
.bsl .empty{padding:26px 10px;text-align:center;color:var(--txt2)}
|
||||
.bsl .empty b{display:block;color:var(--hi);font-size:13px;margin-bottom:4px}
|
||||
|
||||
/* busy bar */
|
||||
.bsl .busybar{position:sticky;top:0;z-index:8;margin:6px 0;padding:6px 8px;border-radius:8px;background:#22222b;border:1px solid #3d3d4a}
|
||||
.bsl .busybar .brow{display:flex;gap:6px;align-items:center}
|
||||
.bsl .busybar .spin{width:12px;height:12px;border-radius:50%;border:2px solid var(--acc);border-top-color:transparent;animation:bslspin .8s linear infinite;flex:none}
|
||||
.bsl .busybar .bprog{height:4px;background:#33333d;border-radius:3px;margin-top:5px;overflow:hidden;position:relative}
|
||||
.bsl .busybar .bprog>div{height:100%;background:var(--acc);transition:width .2s}
|
||||
.bsl .busybar .bprog.ind>div{position:absolute;width:30%;animation:bslind 1.2s ease-in-out infinite}
|
||||
@keyframes bslspin{to{transform:rotate(360deg)}}
|
||||
@keyframes bslind{0%{left:-30%}100%{left:100%}}
|
||||
|
||||
/* timeline */
|
||||
.bsl .tl{position:relative;overflow-x:auto;overflow-y:hidden;border-radius:7px;background:#111115;border:1px solid #2a2a33}
|
||||
.bsl .tlin{position:relative}
|
||||
.bsl .strip{display:flex;height:64px;overflow:hidden}
|
||||
.bsl .strip img{height:64px;object-fit:cover;flex:none;opacity:.92}
|
||||
.bsl .spark{display:block;height:26px;width:100%}
|
||||
.bsl .segs{position:relative;height:30px;margin-top:2px;cursor:crosshair}
|
||||
.bsl .seg{position:absolute;top:2px;bottom:2px;border-radius:5px;display:flex;align-items:center;justify-content:center;
|
||||
font-size:10px;color:#fff;font-weight:600;overflow:hidden;cursor:pointer;border:2px solid transparent;user-select:none}
|
||||
.bsl .seg.sel{border-color:#fff;box-shadow:0 0 0 2px #5b8cff66}
|
||||
.bsl .seg.off{opacity:.35;background-image:repeating-linear-gradient(45deg,#0004 0 6px,#0000 6px 12px)}
|
||||
.bsl .seg.long{outline:2px solid #ff5d5d;outline-offset:-2px}
|
||||
.bsl .seg.playing{box-shadow:0 0 0 2px #ffd34d}
|
||||
.bsl .hdl{position:absolute;top:-70px;bottom:0;width:9px;margin-left:-4px;cursor:ew-resize;z-index:3}
|
||||
.bsl .hdl::after{content:"";position:absolute;left:3px;top:0;bottom:0;width:3px;background:#fff;opacity:.85;border-radius:2px;box-shadow:0 0 4px #000}
|
||||
.bsl .hdl:hover::after{background:#ffd34d}
|
||||
.bsl .rgrid{display:grid;grid-template-columns:auto 130px auto 130px auto;gap:6px 10px;align-items:start;justify-content:start}
|
||||
.bsl .rgrid>button{margin-top:15px}
|
||||
.bsl .rgrid .fld .hint{margin-top:2px}
|
||||
.bsl .outr{position:absolute;top:0;bottom:0;background:repeating-linear-gradient(45deg,#000a 0 6px,#0007 6px 12px);z-index:2;pointer-events:none}
|
||||
.bsl .cut{position:absolute;top:0;height:64px;width:2px;background:#ff4d6d;opacity:.9;pointer-events:none}
|
||||
.bsl .ph{position:absolute;top:0;bottom:0;width:1px;background:#ffd34d;pointer-events:none;z-index:4}
|
||||
.bsl .playhead{position:absolute;top:0;bottom:0;width:2px;background:#ffd34d;box-shadow:0 0 6px #ffd34d;pointer-events:none;z-index:5}
|
||||
.bsl .tip{position:absolute;z-index:6;pointer-events:none;background:#0d0d10ee;border:1px solid #3a3a44;border-radius:6px;padding:3px;font-size:10px;color:#ddd}
|
||||
.bsl .tip img{display:block;height:84px;border-radius:4px}
|
||||
|
||||
/* shot cards */
|
||||
.bsl .shots{display:flex;gap:6px;overflow-x:auto;padding:2px 2px 6px;scroll-snap-type:x proximity}
|
||||
.bsl .shots .sc{flex:0 0 158px;scroll-snap-align:start}
|
||||
.bsl .sc{background:var(--card2);border:1px solid #30303a;border-radius:8px;padding:6px;cursor:pointer;position:relative}
|
||||
.bsl .sc:hover{border-color:#4a4a58}
|
||||
.bsl .sc.sel{border-color:var(--acc);box-shadow:0 0 0 1px var(--acc)}
|
||||
.bsl .sc.skip{opacity:.62}
|
||||
.bsl .sc .bar{height:3px;border-radius:2px;margin-bottom:5px}
|
||||
.bsl .sc .t{font-size:10px;color:#9a9aa6}
|
||||
.bsl .sc .p{font-size:10px;color:#c9c9d3;margin-top:3px;height:28px;overflow:hidden}
|
||||
.bsl .sc .play{position:absolute;top:5px;right:5px;padding:1px 6px;font-size:10px}
|
||||
.bsl .badges{display:flex;gap:3px;flex-wrap:wrap;margin-top:4px}
|
||||
.bsl .bdg{font-size:9px;padding:0 5px;border-radius:4px;background:#2a2a33;color:#a9a9b4;line-height:15px}
|
||||
.bsl .bdg.on{background:#1d2a4d;color:#a9c1ff}
|
||||
.bsl .bdg.warn{background:#3a2a12;color:var(--warn)}
|
||||
.bsl .thumbs{display:flex;gap:4px;margin-top:4px}
|
||||
.bsl .rt{width:34px;height:34px;border-radius:5px;object-fit:cover;background:#2a2a33;border:1px solid #3a3a45}
|
||||
.bsl .rt.ph2{display:flex;align-items:center;justify-content:center;color:#666;font-size:9px}
|
||||
.bsl .who{display:flex;gap:3px;margin-top:3px;align-items:center}
|
||||
.bsl .who img{width:22px;height:22px;border-radius:50%;object-fit:cover;border:1px solid #3a3a45;opacity:.7}
|
||||
.bsl .who img.main{width:26px;height:26px;opacity:1;border-color:#fff}
|
||||
.bsl .who img.lk{border-color:#8fd18f}
|
||||
|
||||
/* shot editor */
|
||||
.bsl .edhd{display:flex;gap:6px;align-items:center;flex-wrap:wrap;margin-bottom:6px}
|
||||
.bsl .edhd .big{font-size:14px;font-weight:700;color:var(--hi)}
|
||||
.bsl .refbox{display:flex;gap:8px;align-items:center}
|
||||
.bsl .refslot{width:78px;height:78px;border-radius:8px;border:1px dashed #44444f;background:#141418;display:flex;align-items:center;
|
||||
justify-content:center;cursor:pointer;overflow:hidden;color:#6f6f7a;font-size:10px;text-align:center;flex:none;white-space:pre-line}
|
||||
.bsl .refslot img{width:100%;height:100%;object-fit:cover}
|
||||
.bsl .refslot:hover{border-color:var(--acc)}
|
||||
.bsl .refslot.sm{width:56px;height:56px;font-size:10px}
|
||||
.bsl .refslot.via{border-style:solid;border-color:#3b5b3b}
|
||||
.bsl .rslot{display:flex;flex-direction:column;gap:3px;align-items:center}
|
||||
.bsl .guidecolor{display:flex;flex-direction:column;align-items:center;gap:2px;max-width:112px}
|
||||
.bsl .guidecolor select{width:100%;font-size:10px;padding:2px 4px}
|
||||
.bsl .rbtns{display:flex;gap:3px;min-height:20px}
|
||||
.bsl .rbtns button{padding:1px 6px;font-size:10px}
|
||||
.bsl .recent{display:flex;gap:4px;flex-wrap:wrap;margin-top:6px;align-items:center}
|
||||
.bsl .recent img{width:30px;height:30px;border-radius:5px;object-fit:cover;cursor:pointer;border:1px solid #3a3a45}
|
||||
.bsl .recent img:hover{border-color:var(--acc)}
|
||||
.bsl .recent.sm img{width:22px;height:22px}
|
||||
.bsl .modes{display:grid;grid-template-columns:repeat(5,minmax(0,1fr));gap:6px;margin-bottom:6px}
|
||||
.bsl .mode{border:1px solid var(--line2);border-radius:7px;padding:6px 8px;cursor:pointer;background:#18181d}
|
||||
.bsl .mode:hover{border-color:#55556a}
|
||||
.bsl .mode.on{border-color:var(--acc);background:#1a2034}
|
||||
.bsl .mode b{display:block;color:var(--hi);font-size:11px}
|
||||
.bsl .mode span{font-size:10px;color:var(--txt2)}
|
||||
.bsl .mstrip{display:flex;gap:4px;margin-top:6px;align-items:center;flex-wrap:wrap}
|
||||
.bsl .mstrip img{height:90px;border-radius:5px;border:1px solid #3a3a45}
|
||||
.bsl .vsug{margin-top:6px;padding:6px 8px;border:1px dashed #4a4a58;border-radius:6px;display:flex;flex-direction:column;gap:3px}
|
||||
.bsl .vsug button{margin-left:6px;padding:1px 6px}
|
||||
.bsl .copy{display:flex;gap:10px;flex-wrap:wrap;align-items:center}
|
||||
.bsl .player{display:flex;gap:10px;align-items:flex-start;flex-wrap:wrap}
|
||||
.bsl .player video{max-height:220px;max-width:100%;border-radius:8px;background:#000;border:1px solid var(--line)}
|
||||
.bsl .pinfo{flex:1;min-width:200px;display:flex;flex-direction:column;gap:6px}
|
||||
.bsl .tc{font-family:ui-monospace,monospace;font-size:12px;color:#e6e6ea;background:#101014;border:1px solid var(--line);border-radius:6px;padding:6px 8px;line-height:1.6}
|
||||
.bsl .tc b{color:#ffd34d}
|
||||
|
||||
/* cast, VLM, checks */
|
||||
.bsl .cast{display:grid;grid-template-columns:repeat(auto-fill,minmax(150px,1fr));gap:8px}
|
||||
.bsl .person{background:#141418;border:1px solid var(--line);border-radius:8px;padding:6px;display:flex;flex-direction:column;gap:4px;align-items:flex-start}
|
||||
.bsl .person.ign{opacity:.45}
|
||||
.bsl .person .face{width:64px;height:64px;border-radius:50%;object-fit:cover;cursor:pointer;border:2px solid #3a3a45}
|
||||
.bsl .person .t{font-size:10px;color:#9a9aa6}
|
||||
.bsl .dlist{display:flex;flex-direction:column;gap:6px;margin-top:6px}
|
||||
.bsl .drow{display:flex;gap:8px;align-items:flex-start}
|
||||
.bsl .dthumbs{display:flex;gap:3px}
|
||||
.bsl .dthumbs img{width:44px;height:58px;object-fit:cover;border-radius:5px;border:1px solid #3a3a45}
|
||||
.bsl .checks{display:flex;flex-direction:column;gap:3px}
|
||||
.bsl .chip{padding:0 6px;font-size:10px;line-height:16px;border-radius:4px}
|
||||
.bsl .ck{display:flex;gap:8px;align-items:flex-start;padding:4px 6px;border-radius:6px;background:#18181d;border:1px solid var(--line)}
|
||||
.bsl .ck .lvl{font-size:12px}
|
||||
.bsl .prog{height:7px;border-radius:4px;background:#26262e;overflow:hidden}
|
||||
.bsl .prog>div{height:100%;background:linear-gradient(90deg,#3a63e0,#7fe0a8)}
|
||||
.bsl .langpick{display:flex;gap:5px;align-items:center}
|
||||
.bsl .langpick select{width:auto;min-width:100px;padding:4px 6px}
|
||||
.bsl .tourback{position:fixed;inset:0;z-index:10001;background:rgba(0,0,0,.68);display:flex;align-items:center;justify-content:center;padding:16px}
|
||||
.bsl .tourbox{width:min(480px,94vw);background:#1d1d23;border:1px solid #4a4a58;border-radius:12px;padding:16px;box-shadow:0 12px 48px #0009}
|
||||
.bsl .tourbox h3{margin:18px 0 8px;color:var(--hi)}
|
||||
.bsl .tourbox p{line-height:1.55}
|
||||
|
||||
/* points window (teleported to <body>) */
|
||||
.bsl .mmodal{position:fixed;inset:0;background:rgba(0,0,0,.72);z-index:10000;display:flex;align-items:center;justify-content:center}
|
||||
.bsl .mbox{background:#1d1d23;border:1px solid #3a3a45;border-radius:10px;padding:12px;width:min(960px,94vw);max-height:94vh;overflow:auto}
|
||||
.bsl .mimg{position:relative;cursor:crosshair;user-select:none;line-height:0}
|
||||
.bsl .mimg img{width:100%;border-radius:6px}
|
||||
.bsl .pt{position:absolute;width:14px;height:14px;margin:-7px 0 0 -7px;border-radius:50%;border:2px solid #fff;cursor:pointer}
|
||||
.bsl .pt.pos{background:#3ccf6b}.bsl .pt.neg{background:#e8455a}
|
||||
`;
|
||||
|
||||
export function styles() {
|
||||
if (document.getElementById("bfs-shotloop-css")) return;
|
||||
const el = document.createElement("style");
|
||||
el.id = "bfs-shotloop-css";
|
||||
el.textContent = CSS;
|
||||
document.head.appendChild(el);
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
// The busy bar and the points window (🎯 Points…: click on a frame to pick the subject for SAM 3).
|
||||
import { Teleport } from "../vendor/vue.esm-browser.prod.mjs";
|
||||
import { h, hint, t } from "./common.js";
|
||||
|
||||
export function busyBar(c, msg) {
|
||||
const { busySince, now, upPct, status } = c;
|
||||
const secs = busySince.value ? Math.max(0, Math.round((now.value - busySince.value) / 1000)) : 0;
|
||||
const pct = upPct.value >= 0 ? upPct.value : (status.total > 0 ? Math.min(1, status.done / status.total) : -1);
|
||||
const stage = status.stage && !msg.startsWith(status.stage) ? status.stage : "";
|
||||
return h("div", { class: "busybar" }, [
|
||||
h("div", { class: "brow" }, [h("span", { class: "spin" }), h("b", msg), stage ? hint(`· ${stage}`) : null,
|
||||
h("span", { class: "grow" }), pct >= 0 ? h("span", `${Math.round(pct * 100)}%`) : null, hint(`${secs}s`)]),
|
||||
h("div", { class: ["bprog", pct < 0 && "ind"] }, [h("div", { style: pct >= 0 ? `width:${(pct * 100).toFixed(1)}%` : "" })]),
|
||||
]);
|
||||
}
|
||||
|
||||
export function pointsModal(c) {
|
||||
const { modal, segs, plan, api } = c;
|
||||
const ms = segs.value[modal.idx];
|
||||
if (!modal.open || !ms) return null;
|
||||
return h(Teleport, { to: "body" }, h("div", { class: "bsl", style: "background:none;padding:0" }, h("div", { class: "mmodal",
|
||||
onKeydown: e => e.stopPropagation(), onPointerdown: e => { if (e.target === e.currentTarget) modal.open = false; } }, [
|
||||
h("div", { class: "mbox" }, [
|
||||
h("div", { class: "row", style: "margin-bottom:6px" }, [h("span", { class: "sect" }, `Shot #${modal.idx + 1} · points`),
|
||||
hint("click = keep (green) · right-click or shift+click = exclude (red) · click a point to remove it")]),
|
||||
h("div", { class: "mimg", onContextmenu: e => e.preventDefault(), onPointerdown: e => {
|
||||
const box = e.currentTarget.getBoundingClientRect();
|
||||
const x = (e.clientX - box.left) / box.width, y = (e.clientY - box.top) / box.height;
|
||||
modal.points = [...modal.points, { x: +x.toFixed(4), y: +y.toFixed(4), label: (e.button === 2 || e.shiftKey) ? 0 : 1 }];
|
||||
modal.prev = null;
|
||||
} }, [
|
||||
h("img", { draggable: false, src: api.apiURL(`/bfs/shotloop/frame?video=${encodeURIComponent(plan.video)}&fps=${plan.fps}&f=${ms.start + modal.key}&w=900`) }),
|
||||
...modal.points.map((p, k) => h("div", { class: ["pt", p.label ? "pos" : "neg"], style: `left:${p.x * 100}%;top:${p.y * 100}%`,
|
||||
title: "click to remove", onPointerdown: e => { e.stopPropagation(); modal.points = modal.points.filter((_, j) => j !== k); modal.prev = null; } })),
|
||||
]),
|
||||
h("div", { class: "row", style: "margin-top:6px" }, [
|
||||
hint(`frame ${ms.start + modal.key} (${modal.key + 1}/${ms.len})`),
|
||||
h("input", { type: "range", min: 0, max: ms.len - 1, step: 1, value: modal.key, style: "flex:1",
|
||||
onInput: e => { modal.key = parseInt(e.target.value); modal.prev = null; } }),
|
||||
]),
|
||||
h("div", { class: "row", style: "margin-top:6px" }, [
|
||||
h("input", { type: "text", value: modal.text, style: "flex:1", placeholder: "text prompt (used when there are no points)",
|
||||
onChange: e => { modal.text = e.target.value; } }),
|
||||
h("button", { onClick: () => { modal.points = []; modal.prev = null; } }, "Clear points"),
|
||||
h("button", { class: "pri", disabled: !!modal.busy || !(modal.points.length || modal.text),
|
||||
onClick: () => c.previewMask(modal.idx, { points: modal.points, key: modal.key, text: modal.text }, true) }, "👁 Segment"),
|
||||
]),
|
||||
modal.busy ? busyBar(c, modal.busy) : null,
|
||||
modal.prev ? h("div", { class: "mstrip" }, [...modal.prev.frames.map(f => h("img", { src: f.src, title: `frame ${f.f}` })),
|
||||
hint(modal.prev.empty ? "nothing found" : `covers ${(modal.prev.coverage * 100).toFixed(1)}%`)]) : null,
|
||||
h("div", { class: "row", style: "margin-top:8px;justify-content:flex-end" }, [
|
||||
h("button", { onClick: () => { modal.open = false; } }, "Cancel"),
|
||||
h("button", { title: "Apply these points to EVERY shot, each on its frame at the same relative position (a subject that stays in place)",
|
||||
onClick: c.savePointsAll }, "Save → all shots"),
|
||||
h("button", { disabled: !modal.points.length,
|
||||
title: "Track this selection with SAM 3 through the whole video (or the range), forward and backward from this frame; every shot without its own selection takes its part of the track. Also type a short text (e.g. 'woman in pink top') so the subject is found again after camera cuts.",
|
||||
onClick: c.saveTrack }, "🎯 Track → whole video"),
|
||||
h("button", { class: "pri", onClick: c.savePoints }, "Save (this shot)"),
|
||||
]),
|
||||
]),
|
||||
])));
|
||||
}
|
||||
|
||||
const TOUR_STEPS = [
|
||||
["video", "Video", "Choose a video, set the frame range, split at camera cuts, then set the generation size."],
|
||||
["shots", "Shots", "Review each shot, assign its references and prompt, and choose how the result is applied."],
|
||||
["people", "People", "Link detected people to references. Set masks here or per shot; ArcFace links are a guide and need your review."],
|
||||
["prompts", "Prompts", "Set shared references, prompts and optional VLM assistance. Written prompts are kept exactly as authored."],
|
||||
["shots", "Color Guide", "Color Guide adds a translucent marker to a selected person. Ref 1 and Ref 2 must use different colors in the same shot."],
|
||||
["run", "Run", "Check warnings, choose auto or queue loop, and set filters or quick-run limits."],
|
||||
];
|
||||
|
||||
export function tourModal(c) {
|
||||
const { tour, tab } = c;
|
||||
const step = tour.value - 1;
|
||||
if (step < 0 || step >= TOUR_STEPS.length) return null;
|
||||
const [targetTab, title, body] = TOUR_STEPS[step];
|
||||
return h(Teleport, { to: "body" }, h("div", { class: "bsl", style: "background:none;padding:0" },
|
||||
h("div", { class: "tourback", role: "dialog", "aria-modal": "true", "aria-label": t("Shot Planner tour"),
|
||||
onKeydown: e => { if (e.key === "Escape") tour.value = 0; }, onPointerdown: e => { if (e.target === e.currentTarget) tour.value = 0; } }, [
|
||||
h("div", { class: "tourbox" }, [
|
||||
h("div", { class: "row" }, [h("b", t("Shot Planner tour")), h("span", { class: "grow" }), h("span", { class: "hint" }, `${step + 1} / ${TOUR_STEPS.length}`),
|
||||
h("button", { class: "ghost", title: t("Close tour"), "aria-label": t("Close tour"), onClick: () => { tour.value = 0; } }, "✕")]),
|
||||
h("h3", t(title)),
|
||||
h("p", t(body)),
|
||||
h("p", { class: "hint" }, t("This tour is optional. Use Next to visit each main tab; close it whenever you like.")),
|
||||
h("div", { class: "row", style: "justify-content:flex-end;margin-top:12px" }, [
|
||||
step > 0 ? h("button", { onClick: () => { tour.value--; tab.value = TOUR_STEPS[step - 1][0]; } }, t("Back")) : null,
|
||||
h("button", { class: "pri", onClick: () => {
|
||||
if (step === TOUR_STEPS.length - 1) tour.value = 0;
|
||||
else { tour.value++; tab.value = TOUR_STEPS[step + 1][0]; }
|
||||
} }, t(step === TOUR_STEPS.length - 1 ? "Close tour" : "Next")),
|
||||
]),
|
||||
]),
|
||||
])));
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
// 👥 People & masks tab: the cast found by face (each person linked to a reference) and the global SAM 3 mask settings.
|
||||
import { h, fmtT, fld, check, section, pill, hint } from "./common.js";
|
||||
|
||||
const CAST_COLORS = [["", "No Color Guide color"], ["#00ff00", "Green"], ["#9b30ff", "Purple"], ["#ff2020", "Red"], ["#168cff", "Blue"],
|
||||
["#ffe600", "Yellow"], ["#ff00d4", "Magenta"], ["#00e5c0", "Cyan"]];
|
||||
|
||||
function castSection(c) {
|
||||
const { plan, people, busy, usedRefs } = c;
|
||||
const fps = plan.fps;
|
||||
const castN = people.value.filter(p => c.linked(p.id)).length;
|
||||
const opt = (k, label, title) => check(plan[k], v => c.setCastOpt(k, v), label, title);
|
||||
return section("Cast", [
|
||||
h("div", { class: "row", style: "gap:14px;margin-bottom:6px" }, [
|
||||
opt("cast_assign", "Shots use their main person's reference", "a shot without its own reference takes the reference of the person with the most screen time in it"),
|
||||
opt("cast_split", "Split where the main person changes", "adds a boundary when the biggest face switches to someone else (re-splits the plan)"),
|
||||
opt("cast_only", "Only run shots with a linked person", "shots where no linked person appears are skipped"),
|
||||
]),
|
||||
people.value.length ? h("div", { class: "cast" }, people.value.map(p => {
|
||||
const cc = c.castOf(p.id);
|
||||
return h("div", { class: ["person", cc.ignore && "ign"] }, [
|
||||
h("img", { class: "face", src: p.thumb, title: `first seen ${fmtT(p.first, fps)} (click to seek)`, onClick: () => c.seek(p.first) }),
|
||||
h("div", { class: "t" }, [h("b", `Person ${p.id}`), ` · ${(p.share * 100).toFixed(1)}%`]),
|
||||
h("div", { class: "t" }, `${fmtT(p.first, fps)} → ${fmtT(p.last, fps)}`),
|
||||
h("div", { class: "row", style: "margin-top:4px;align-items:center" }, [
|
||||
h("span", { class: "hint" }, "Color Guide identity color"),
|
||||
h("select", { value: cc.guide_color || "", style: "width:auto", onChange: e => c.setCast(p.id, "guide_color", e.target.value) },
|
||||
CAST_COLORS.map(([v, label]) => h("option", { value: v }, label))),
|
||||
]),
|
||||
h("div", { class: "refbox" }, ["ref", "ref2"].map(k => h("div", { class: "rslot" }, [
|
||||
h("div", { class: "refslot sm", title: cc[k] ? cc[k] + " (click to replace)" : "click to upload",
|
||||
onClick: () => c.pickFile("image/*", name => c.setCast(p.id, k, name)) }, cc[k] ? [h("img", { src: c.viewUrl(cc[k]) })] : [k === "ref" ? "⬆ ref" : "⬆ ref 2"]),
|
||||
cc[k] ? h("div", { class: "rbtns" }, [h("button", { title: "remove", onClick: () => c.setCast(p.id, k, "") }, "✕")]) : null,
|
||||
]))),
|
||||
usedRefs.value.length ? h("div", { class: "recent sm" }, usedRefs.value.map(n => h("img", { src: c.viewUrl(n), title: `${n} (click = ref, shift+click = ref 2)`,
|
||||
onClick: e => c.setCast(p.id, e.shiftKey ? "ref2" : "ref", n) }))) : null,
|
||||
check(cc.ignore, v => c.setCast(p.id, "ignore", v), "ignore"),
|
||||
]);
|
||||
})) : h("div", { class: "hint" }, "Nobody found yet."),
|
||||
h("div", { class: "row", style: "margin-top:8px" }, [
|
||||
h("button", { class: "pri", disabled: !!busy.value, onClick: () => c.findPeople(true) }, people.value.length ? "↻ Re-analyse people" : "👥 Find people"),
|
||||
hint(people.value.length ? "faces sampled every 0.5 s and grouped by identity (InsightFace, CPU)" : "detects faces across the video and groups them into people"),
|
||||
]),
|
||||
], { sub: "track faces, link each person to a reference, split and assign shots by who is on screen",
|
||||
right: people.value.length ? pill(`${people.value.length} people · ${castN} linked`) : pill("not analysed") });
|
||||
}
|
||||
|
||||
function maskSection(c) {
|
||||
const { plan, segs, showMasks } = c;
|
||||
const MC = plan.mask_cfg || {};
|
||||
const mnum = (k, step, label, help, scale = 1) => fld(label, h("input", { type: "number", step, min: 0, value: +(MC[k] * scale).toFixed(3),
|
||||
onChange: e => c.setMaskCfg(k, (parseFloat(e.target.value) || 0) / scale) }), help);
|
||||
const S = segs.value;
|
||||
const counts = { mask: 0, crop: 0, cropmask: 0, paste: 0 };
|
||||
S.forEach(s => { const m = c.modeOf(s); if (m in counts) counts[m]++; });
|
||||
const masked = S.filter(s => s.mask?.text || (s.mask?.points || []).length || s.mask?.video || s.extMask).length;
|
||||
return section("Mask & crop settings", [
|
||||
h("div", { class: "note", style: "margin-bottom:8px" }, [
|
||||
"Each shot picks what it marks (🎯 text or points, or 🎞 its own mask video) and how it is generated (", h("b", "Full frame · Frame + paste · Mask only · Crop · Crop + mask"),
|
||||
") in the Shots tab. Masks from a video (here, per shot, or the planner's ", h("b", "mask"), " input) replace SAM 3. These settings apply to every shot."]),
|
||||
h("div", { class: "row", style: "margin-bottom:8px;align-items:center" }, [
|
||||
h("span", { style: "white-space:nowrap", title: "A black/white video of the WHOLE source video (white = the subject), e.g. rotoscoped in another tool. Shots without their own mask (video, text or points) use it instead of SAM 3. The planner's optional mask input does the same from the graph and wins over this." }, "🎞 Mask video (whole video)"),
|
||||
h("select", { value: plan.mask_video || "", style: "flex:1;min-width:180px", onChange: e => c.setGlobalMaskVideo(e.target.value) },
|
||||
[h("option", { value: "" }, "— none: SAM 3 per shot —"), ...c.files.videos.map(v => h("option", { value: v }, v))]),
|
||||
h("button", { onClick: () => c.pickFile("video/*", c.setGlobalMaskVideo) }, "⬆ Upload"),
|
||||
plan.mask_video ? h("button", { onClick: () => c.setGlobalMaskVideo("") }, "✕") : null,
|
||||
]),
|
||||
plan.track?.points?.length ? h("div", { class: "note", style: "margin-bottom:8px;display:flex;gap:8px;align-items:center" }, [
|
||||
h("span", { class: "grow" }, [h("b", "🎯 Whole-video track: "), `${plan.track.points.length} point(s) at frame ${plan.track.frame}`,
|
||||
plan.track.text ? ` · re-found by "${plan.track.text}" after cuts` : " · no text: after a cut the subject may be lost (add one in 🎯 Points…)",
|
||||
". Shots without their own selection use it."]),
|
||||
h("button", { class: "ghost", onClick: c.clearTrack }, "✕ remove"),
|
||||
]) : null,
|
||||
h("div", { class: "row", style: "gap:14px;margin-bottom:6px" }, [
|
||||
check(MC.fill_holes, v => c.setMaskCfg("fill_holes", v), "Fill holes", "Fill enclosed gaps so each region is solid"),
|
||||
check(MC.invert, v => c.setMaskCfg("invert", v), "Invert", "Use everything except the segmented object"),
|
||||
check(showMasks.value, v => { showMasks.value = v; }, "Show masks on shots", "Overlay the mask preview on the shot cards"),
|
||||
]),
|
||||
h("div", { class: "grid" }, [
|
||||
mnum("expand", 1, "Expand (px)", "grow the mask (paste-back and generation mask)"),
|
||||
mnum("temporal_expand", 1, "Temporal expand (frames)", "hold the mask a few frames: less flicker"),
|
||||
mnum("threshold", 0.05, "Detection threshold", "SAM 3 score to keep an object"),
|
||||
mnum("max_objects", 1, "Max objects", "how many matches of the text are tracked"),
|
||||
mnum("padding", 1, "Crop padding (%)", "Crop modes: context around the mask's box", 100),
|
||||
mnum("feather", 1, "Feather (px)", "Crop modes: soft edge of the paste"),
|
||||
fld("Paste back", h("select", { value: MC.paste, onChange: e => c.setMaskCfg("paste", e.target.value) },
|
||||
[h("option", { value: "mask" }, "only the mask (feathered)"), h("option", { value: "box" }, "the whole box (feathered)")]), "Crop modes"),
|
||||
mnum("blockify", 1, "Blockify (px, 0 = off)", "square blocks; 16 matches H3's latent grid"),
|
||||
]),
|
||||
hint("Uses the official sam3.1_multiplex_fp16 checkpoint (models/checkpoints), downloaded from Comfy-Org/sam3.1 the first time.", "display:block;margin-top:6px"),
|
||||
], { sub: "SAM 3",
|
||||
right: h("span", { class: "row" }, [pill(`${masked} masked`), counts.mask ? pill(`${counts.mask} mask only`, "acc") : null, counts.paste ? pill(`${counts.paste} frame+paste`, "acc") : null,
|
||||
counts.crop + counts.cropmask ? pill(`${counts.crop + counts.cropmask} cropped`, "warn") : null]) });
|
||||
}
|
||||
|
||||
export function peopleTab(c) {
|
||||
return h("div", { class: "card" }, [castSection(c), maskSection(c)]);
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
// 📝 Prompts & refs tab: the global reference / prompt, the VLM (shot suggestions) and the reference descriptions ({details}).
|
||||
import { h, fld, check, section, pill, hint, select } from "./common.js";
|
||||
|
||||
const GUIDE_COLORS = [["#00ff00", "Green"], ["#9b30ff", "Purple"], ["#ff2020", "Red"], ["#168cff", "Blue"],
|
||||
["#ffe600", "Yellow"], ["#ff00d4", "Magenta"], ["#00e5c0", "Cyan"]];
|
||||
|
||||
function globalSection(c) {
|
||||
const { plan } = c;
|
||||
const slot = (name, label, field) => h("div", { class: "rslot" }, [
|
||||
h("div", { class: "refslot", title: name ? name + " (click to replace)" : "click to upload", onClick: () => c.uploadRef("global", field) },
|
||||
name ? [h("img", { src: c.viewUrl(name) })] : [label]),
|
||||
h("div", { class: "rbtns" }, [name ? h("button", { title: "remove", onClick: () => c.setRef("global", field, "") }, "✕") : null]),
|
||||
h("label", { class: "guidecolor" }, [
|
||||
h("span", { class: "hint" }, field === "global_ref" ? "Guide color · Ref 1" : "Guide color · Ref 2"),
|
||||
h("select", { value: field === "global_ref" ? (plan.global_color_guide_ref1 || "#00ff00") : (plan.global_color_guide_ref2 || "#9b30ff"),
|
||||
disabled: !name,
|
||||
title: "Default Color Guide marker when this global reference is used",
|
||||
onChange: e => c.setPlan(field === "global_ref" ? "global_color_guide_ref1" : "global_color_guide_ref2", e.target.value) },
|
||||
GUIDE_COLORS.map(([v, text]) => h("option", { value: v }, text))),
|
||||
]),
|
||||
]);
|
||||
return section("Global reference & prompt", [
|
||||
h("div", { class: "row", style: "align-items:flex-start;gap:10px" }, [
|
||||
h("div", { class: "refbox" }, [slot(plan.global_ref, "⬆ global\nreference", "global_ref"), slot(plan.global_ref2, "⬆ global\nref 2", "global_ref2")]),
|
||||
h("div", { style: "flex:1;min-width:200px" }, [
|
||||
h("textarea", { style: "min-height:78px", placeholder: "Global prompt (a connected `prompt` input overrides it)",
|
||||
value: plan.global_prompt, onChange: e => c.setPlan("global_prompt", e.target.value) }),
|
||||
]),
|
||||
]),
|
||||
h("div", { class: "hint", style: "margin-top:4px" }, [
|
||||
"Used by shots without their own. Connected ref_image / ref_image_2 / prompt inputs on the node override these. Placeholders: ",
|
||||
h("b", "{target}"), " who is replaced (per shot) · ", h("b", "{details}"), " description of the shot's references · ",
|
||||
h("b", "{shot}"), " the VLM's description of the shot · ", h("b", "{setting}"), " the setting picture (H3 Conditioning)."]),
|
||||
h("div", { class: "hint", style: "margin-top:4px" }, "These colors are defaults for the global references. Turn on Color Guide on a shot to apply them; shot-level colors can override them."),
|
||||
], { sub: "defaults for every shot" });
|
||||
}
|
||||
|
||||
function vlmSection(c) {
|
||||
const { plan, vlmSug, busy, refSets } = c;
|
||||
const VC = plan.vlm_cfg || {};
|
||||
const opt = (k, label, title) => check(VC[k], v => c.setVlmCfg(k, v), label, title);
|
||||
const sugN = Object.keys(vlmSug.value).length;
|
||||
return [
|
||||
section("VLM", [
|
||||
h("div", { class: "row", style: "gap:14px;margin-bottom:6px" }, [
|
||||
opt("enabled", "Use the VLM when the workflow runs", "At run time the planner asks the VLM about the shots it runs and applies the options below"),
|
||||
opt("auto_segment", "Fill the mask text", "Shots without a mask text or points get the VLM's segment suggestion"),
|
||||
opt("auto_shot", "Fill {shot} in the prompt", "Write {shot} in a prompt: it becomes the VLM's description of that shot (camera, framing, action)"),
|
||||
]),
|
||||
h("div", { class: "grid" }, [
|
||||
fld("Frames per shot", h("input", { type: "number", min: 1, max: 8, step: 1, value: VC.frames, onChange: e => c.setVlmCfg("frames", parseInt(e.target.value) || 3) })),
|
||||
fld("Max tokens", h("input", { type: "number", min: 64, max: 2048, step: 32, value: VC.max_tokens, onChange: e => c.setVlmCfg("max_tokens", parseInt(e.target.value) || 1024) })),
|
||||
]),
|
||||
h("textarea", { style: "margin-top:6px;min-height:44px", placeholder: "Extra instruction for the VLM (optional), e.g. 'segment the woman, not the man'",
|
||||
value: VC.instruction || "", onChange: e => c.setVlmCfg("instruction", e.target.value) }),
|
||||
h("div", { class: "row", style: "margin-top:6px" }, [
|
||||
h("button", { class: "pri", disabled: !!busy.value, onClick: c.analyseVLM }, "🤖 Analyse shots"),
|
||||
h("button", { disabled: !sugN, onClick: c.applyAllSug, title: "Mask text for shots without one, skip where the VLM says skip" }, "Apply suggestions → all"),
|
||||
]),
|
||||
hint("Connect a Qwen3-VL (CLIPLoader) to the planner's vlm input. The VLM buttons work after the workflow has run once with it connected (ComfyUI only hands models to nodes when they run). Each shot's suggestion shows in its editor.", "display:block;margin-top:4px"),
|
||||
], { sub: "looks at every shot and suggests the mask, a description and whether to run it",
|
||||
right: h("span", { class: "row" }, [VC.enabled ? pill("on", "ok") : pill("off"), sugN ? pill(`${sugN} suggestions`) : null]) }),
|
||||
section("Describe references ({details})", [
|
||||
h("div", { class: "row", style: "gap:8px" }, [
|
||||
select(VC.describe_preset, ["short", "full body", "head / face", "face attributes", "outfit", "custom"], v => c.setVlmCfg("describe_preset", v), { style: "width:auto" }),
|
||||
h("button", { class: "pri", disabled: !!busy.value || !refSets.value.length, onClick: c.describeRefs }, "📝 Describe refs"),
|
||||
h("button", { disabled: !!busy.value || !refSets.value.length,
|
||||
title: "Try another description: a new seed (and some temperature if it is 0), then describe again",
|
||||
onClick: () => { if (!(+VC.temperature > 0)) c.setVlmCfg("temperature", 0.7); c.setVlmCfg("seed", (+VC.seed || 0) + 1); c.describeRefs(); } }, "🎲"),
|
||||
fld("Temperature", h("input", { type: "number", min: 0, max: 1.5, step: 0.1, value: VC.temperature ?? 0, style: "width:70px",
|
||||
title: "0 = always the same answer (greedy); 0.5-1 = varied answers, picked by the seed",
|
||||
onChange: e => c.setVlmCfg("temperature", Math.max(0, parseFloat(e.target.value) || 0)) })),
|
||||
fld("Seed", h("input", { type: "number", min: 0, step: 1, value: VC.seed ?? 0, style: "width:80px",
|
||||
title: "Used when the temperature is above 0: another seed, another description",
|
||||
onChange: e => c.setVlmCfg("seed", Math.max(0, parseInt(e.target.value) || 0)) })),
|
||||
hint(`${refSets.value.length} reference set(s) · editable below`),
|
||||
check(VC.fit_prompt !== false, v => c.setVlmCfg("fit_prompt", v), "fit the prompt",
|
||||
"The VLM gets the global prompt's sentence that holds {details} (and the shot prompt's {target} sentence for 🧑 Describe target) with a blank in its place, and writes the description to fit it: the description adapts to the prompt, not the other way round."),
|
||||
]),
|
||||
VC.describe_preset === "custom" ? h("textarea", { style: "margin-top:6px;min-height:52px", value: VC.describe_custom || "",
|
||||
placeholder: "Your instruction for the VLM, e.g. 'Describe the character's costume and props piece by piece…'",
|
||||
onChange: e => c.setVlmCfg("describe_custom", e.target.value) }) : null,
|
||||
refSets.value.length ? h("div", { class: "dlist" }, refSets.value.map(x => h("div", { class: "drow" }, [
|
||||
h("div", { class: "dthumbs" }, [x.a ? h("img", { src: c.viewUrl(x.a) }) : null, x.b ? h("img", { src: c.viewUrl(x.b) }) : null]),
|
||||
h("div", { style: "flex:1" }, [
|
||||
hint(x.where.length > 4 ? `${x.where.slice(0, 4).join(", ")} +${x.where.length - 4}` : x.where.join(", ")),
|
||||
h("textarea", { style: "min-height:44px", value: plan.ref_details?.[x.key] || "",
|
||||
placeholder: "no description yet: Describe refs, write your own, or leave it to the VLM at run time",
|
||||
onChange: e => c.setDetail(x.key, e.target.value) }),
|
||||
]),
|
||||
]))) : hint("Add a reference (global, per shot or per person) first."),
|
||||
], { sub: "write {details} in any prompt: each shot gets the description of its own references" }),
|
||||
];
|
||||
}
|
||||
|
||||
export function promptsTab(c) {
|
||||
return h("div", { class: "card" }, [globalSection(c), ...vlmSection(c)]);
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
// ▶ Run tab: checks before running, how the loop runs (auto / queue), which shots run (filters) and test limits.
|
||||
import { h, fld, check, section, pill, hint, select } from "./common.js";
|
||||
|
||||
function checksSection(c) {
|
||||
const { checks, tab } = c;
|
||||
const list = checks.value;
|
||||
const warns = list.filter(x => x.lvl === "warn").length;
|
||||
// one row per kind of issue, with the shots it concerns
|
||||
const groups = new Map();
|
||||
for (const x of list) { const k = `${x.lvl}|${x.text}`; if (!groups.has(k)) groups.set(k, { ...x, shots: [] }); groups.get(k).shots.push(x.shot); }
|
||||
const rows = [...groups.values()].sort((a, b) => (a.lvl === "warn" ? 0 : 1) - (b.lvl === "warn" ? 0 : 1));
|
||||
const go = i => { c.select(i); tab.value = "shots"; };
|
||||
return section("Checks", rows.length ? h("div", { class: "checks" }, rows.map(x => h("div", { class: "ck" }, [
|
||||
h("span", { class: "lvl" }, x.lvl === "warn" ? "⚠" : "ℹ"),
|
||||
h("span", { style: "flex:1;min-width:0" }, [h("div", x.text), h("div", { class: "row", style: "gap:3px;margin-top:3px" }, [
|
||||
...x.shots.slice(0, 24).map(i => h("button", { class: "chip", title: "go to this shot", onClick: () => go(i) }, `#${i + 1}`)),
|
||||
x.shots.length > 24 ? hint(`+${x.shots.length - 24} more`) : null])]),
|
||||
pill(`${x.shots.length} shot(s)`, x.lvl === "warn" ? "warn" : ""),
|
||||
]))) : h("div", { class: "hint" }, "✓ Nothing to fix in the shots that run."),
|
||||
{ sub: "only shots that run", right: list.length ? pill(`${warns} warning(s) · ${list.length - warns} note(s)`, warns ? "warn" : "") : pill("all good", "ok") });
|
||||
}
|
||||
|
||||
function loopSection(c) {
|
||||
const { plan, prog, active } = c;
|
||||
const num = (k, step, min) => h("input", { type: "number", step, min, value: plan[k], onChange: e => c.setPlan(k, parseFloat(e.target.value) || 0) });
|
||||
return section("Run", [
|
||||
h("div", { class: "grid" }, [
|
||||
fld("Run", select(plan.run, [["auto", "Auto loop (one run)"], ["queue", "Queue loop (one shot per run)"]], v => c.setPlan("run", v, c.progress))),
|
||||
fld("Skipped shots in the output", select(plan.skip_fill, [["original", "Keep original video"], ["drop", "Remove them"]], v => c.setPlan("skip_fill", v))),
|
||||
fld("Max shots (0 = all)", num("max_parts", 1, 0), "quick tests"),
|
||||
fld("Max total seconds (0 = all)", num("max_total_s", 0.5, 0), "from the start of the range"),
|
||||
]),
|
||||
plan.run === "queue" ? h("div", { style: "margin-top:8px" }, [
|
||||
h("div", { class: "row", style: "margin-bottom:4px" }, [h("b", "Queue loop"), h("span", { class: "grow" }),
|
||||
hint(`${prog.done}/${prog.count || active.value.length} shots done`)]),
|
||||
h("div", { class: "prog" }, [h("div", { style: `width:${(100 * prog.done / Math.max(1, prog.count || active.value.length)).toFixed(1)}%` })]),
|
||||
h("div", { class: "row", style: "margin-top:6px" }, [
|
||||
check(plan.auto_continue, v => c.setPlan("auto_continue", v), "Auto-queue the next shot"),
|
||||
h("button", { onClick: () => c.progress(false) }, "↻ Status"), h("button", { class: "dng", onClick: () => c.progress(true) }, "⟲ Reset loop"),
|
||||
]),
|
||||
hint("Each run generates one shot and stores it. Nodes after BFS Shot Join only run on the last shot, with the full video.", "display:block;margin-top:4px"),
|
||||
]) : null,
|
||||
], { sub: `${active.value.length} shot(s) will run` });
|
||||
}
|
||||
|
||||
function filtersSection(c) {
|
||||
const { plan, stats, segs, busy, filtersOn } = c;
|
||||
const F = plan.filters;
|
||||
const chk = (k, label) => check(F[k], v => c.setFilter(k, v), label);
|
||||
const fnum = (k, step, label, help) => fld(label, h("input", { type: "number", step, min: 0, value: F[k], onChange: e => c.setFilter(k, parseFloat(e.target.value) || 0) }), help);
|
||||
const skippedN = segs.value.filter(s => c.skipWhy(s)).length;
|
||||
return section("Filters", [
|
||||
h("div", { class: "row", style: "gap:14px;margin-bottom:6px" }, [
|
||||
chk("person", "Needs a person"), chk("face", "Needs a face"), chk("skip_dark", "Skip dark / fades"), chk("skip_static", "Skip static shots"),
|
||||
]),
|
||||
h("div", { class: "grid" }, [
|
||||
fnum("min_person_area", 0.01, "Min person size (0-1 of frame)", "e.g. 0.03 skips wide shots"),
|
||||
fnum("max_persons", 1, "Max people (0 = any)", "skip crowds"),
|
||||
fnum("min_frames", 1, "Min frames (0 = off)"),
|
||||
fnum("samples", 1, "Frames sampled / shot"),
|
||||
fnum("dark_level", 0.01, "Dark below (0-1)"),
|
||||
fnum("static_level", 0.001, "Static below"),
|
||||
]),
|
||||
h("div", { class: "row", style: "margin-top:8px" }, [
|
||||
h("button", { class: "pri", disabled: !!busy.value, onClick: c.analyzeContent }, "👤 Analyse people & faces"),
|
||||
hint(stats.value.length ? `stats for ${stats.value.length} shots · YOLO person/face from models/ultralytics` : "runs the detectors on a few frames of every shot"),
|
||||
]),
|
||||
], { open: filtersOn.value || stats.value.length > 0, sub: "skipped shots do not run",
|
||||
right: filtersOn.value ? pill(`${skippedN} skipped`, "warn") : pill("off") });
|
||||
}
|
||||
|
||||
export function runTab(c) {
|
||||
return h("div", { class: "card" }, [checksSection(c), loopSection(c), filtersSection(c)]);
|
||||
}
|
||||
@@ -0,0 +1,332 @@
|
||||
// 🎬 Shots tab: one card per shot, then the selected shot's editor (preview, reference & prompt, who is replaced,
|
||||
// continuity, copy to other shots).
|
||||
import { h, hue, fmtT, segKey, hasMask, maskSource, pill, hint, check, section } from "./common.js";
|
||||
import { h as rawH } from "../vendor/vue.esm-browser.prod.mjs";
|
||||
|
||||
const GUIDE_COLORS = [["#00ff00", "Green"], ["#9b30ff", "Purple"], ["#ff2020", "Red"], ["#168cff", "Blue"],
|
||||
["#ffe600", "Yellow"], ["#ff00d4", "Magenta"], ["#00e5c0", "Cyan"]];
|
||||
|
||||
const bdg = (text, kind = "", title = "") => h("span", { class: ["bdg", kind], title }, text);
|
||||
|
||||
function shotCard(c, s, i) {
|
||||
const { plan, sel, maskPrev, showMasks } = c;
|
||||
const fps = plan.fps, why = c.skipWhy(s), who = c.whoIn(s), st = c.statFor(s), issues = c.checksFor(i);
|
||||
const warnN = issues.filter(x => x.lvl === "warn").length;
|
||||
const mp = maskPrev.value[segKey(s)];
|
||||
return h("div", { class: ["sc", i === sel.value && "sel", why && "skip"], onClick: () => { sel.value = i; } }, [
|
||||
h("button", { class: "play", title: "play this shot", onClick: e => { e.stopPropagation(); c.playShot(i); } }, "▶"),
|
||||
h("div", { class: "bar", style: `background:${hue(i)}` }),
|
||||
h("div", { class: "row" }, [h("b", `#${i + 1}`), s.cut ? pill("cut") : null,
|
||||
why ? pill("skip", "warn", why) : pill("run", "ok"), s.force !== "auto" ? pill(s.force) : null]),
|
||||
h("div", { class: "badges" }, [
|
||||
{ video: bdg("🎞 mask video", "", s.mask.video), global: bdg("🎞 global mask", "", "the plan's mask video for the whole video"),
|
||||
track: bdg("🎯 track", "", "its part of the whole-video SAM 3 track"),
|
||||
sam: bdg("🎯 " + (s.mask.text ? s.mask.text.slice(0, 14) : `${(s.mask.points || []).length} pts`), "", s.mask.text || `${(s.mask.points || []).length} points`) }[maskSource(s)] || null,
|
||||
s.crop || s.inpaint || s.paste ? bdg({ paste: "🧩 frame+paste", mask: "🎭 mask only", crop: "✂ crop", cropmask: "✂🎭 crop+mask" }[c.modeOf(s)] + (s.inpaint && s.strength < 1 ? ` ${Number(s.strength).toFixed(2)}` : ""), "on", c.modeName(s)) : null,
|
||||
s.color_guide ? bdg("🎨 color guide", "on", `${s.color_guide_color || "cast color"} · opacity ${s.color_guide_opacity ?? 0.35}`) : null,
|
||||
s.target ? bdg("🧑 target", "on", s.target) : null,
|
||||
i > 0 && s.chain !== "off" ? bdg("⛓", "on", s.chain === "history" ? `continues the previous shot's last ${s.chainHistory} frames (history)` : `continues from the previous shot's ${s.chainFrame} frame as ${s.chain}`) : null,
|
||||
warnN ? bdg(`⚠ ${warnN}`, "warn", issues.map(x => x.text).join("\n")) : null,
|
||||
]),
|
||||
who ? h("div", { class: "who" }, who.people.length ? who.people.map(id => h("img", {
|
||||
src: c.personOf(id)?.thumb || "", title: `Person ${id}${id === who.main ? " (main)" : ""}${c.linked(id) ? " · linked" : ""}`,
|
||||
class: [id === who.main && "main", c.linked(id) && "lk"] })) : [h("span", { class: "t" }, "no faces")]) : null,
|
||||
st ? h("div", { class: "t", style: "margin-top:2px" },
|
||||
`👤 ${st.persons} · ${(st.person_area * 100).toFixed(1)}% · 🙂 ${st.faces} · ☀ ${(st.brightness * 100).toFixed(0)}%`) : null,
|
||||
why ? h("div", { class: "t", style: "color:#ffc46b" }, why) : null,
|
||||
h("div", { class: "t" }, `${fmtT(s.start, fps)} → ${fmtT(s.end, fps)} · ${s.len}f → ${s.gen}f`),
|
||||
h("div", { class: "thumbs" }, [
|
||||
...["ref", "ref2"].map(k => {
|
||||
const own = s[k], via = c.castRef(s, k), name = own || via || plan["global_" + k];
|
||||
return name ? h("img", { class: "rt", src: c.viewUrl(name), title: own ? name : via ? `${name} (from the person)` : `${name} (global)`,
|
||||
style: own ? "" : via ? "outline:1px dashed #8fd18f" : "opacity:.45" }) : h("div", { class: "rt ph2" }, k);
|
||||
}),
|
||||
h("img", { class: "rt", style: "width:56px", src: showMasks.value && mp?.frames?.length
|
||||
? mp.frames[Math.floor(mp.frames.length / 2)].src : c.thumbFor(s.start + Math.floor(s.len / 2)) }),
|
||||
]),
|
||||
s.prompt ? rawH("div", { class: "p" }, s.prompt)
|
||||
: h("div", { class: "p" }, plan.global_prompt ? "↳ global prompt" : "— no prompt —"),
|
||||
]);
|
||||
}
|
||||
|
||||
function player(c, cur) {
|
||||
const { plan, segs, play, vid, sel, n } = c;
|
||||
const S = segs.value, ps = S[play.idx] || cur, fps = plan.fps, N = n.value;
|
||||
return h("div", { class: "player" }, [
|
||||
h("video", { ref: vid, src: c.viewUrl(plan.video), preload: "metadata", muted: false, playsinline: true,
|
||||
onPause: () => { if (play.mode) play.mode = ""; } }),
|
||||
h("div", { class: "pinfo" }, [
|
||||
h("div", { class: "row" }, [
|
||||
h("button", { class: "pri", disabled: !cur, onClick: () => c.playShot(sel.value) }, `▶ Play #${sel.value + 1}`),
|
||||
h("button", { onClick: c.playAll }, "▶ Play all"),
|
||||
h("button", { onClick: c.stop }, "■ Stop"),
|
||||
check(play.loop, v => { play.loop = v; }, "loop shot"),
|
||||
play.mode ? pill(play.mode === "all" ? "playing all" : "playing shot", "warn") : null,
|
||||
]),
|
||||
ps ? h("div", { class: "tc" }, [
|
||||
h("div", ["Shot ", h("b", `#${S.indexOf(ps) + 1}`), c.skipWhy(ps) ? ` (skipped: ${c.skipWhy(ps)})` : ""]),
|
||||
h("div", ["start ", h("b", fmtT(ps.start, fps)), ` (frame ${ps.start})`]),
|
||||
h("div", ["end ", h("b", fmtT(ps.end, fps)), ` (frame ${ps.end - 1}) · ${(ps.len / fps).toFixed(2)}s`]),
|
||||
h("div", ["now ", h("b", fmtT(Math.min(N, play.frame), fps)), ` (frame ${Math.min(N, play.frame)})`]),
|
||||
]) : null,
|
||||
hint("Click the timeline to seek. Play all skips shots that will not run."),
|
||||
]),
|
||||
]);
|
||||
}
|
||||
|
||||
function refSlot(c, name, label, target, field, via = "") {
|
||||
return h("div", { class: "rslot" }, [
|
||||
h("div", { class: ["refslot", !name && via && "via"], title: name ? name + " (click to replace)" : via ? `uses ${via}; click to upload this shot's own` : "click to upload",
|
||||
onClick: () => c.uploadRef(target, field) },
|
||||
name ? [h("img", { src: c.viewUrl(name) })] : via ? [h("img", { src: c.viewUrl(via), style: "opacity:.4" })] : [label]),
|
||||
h("div", { class: "rbtns" }, [name ? h("button", { title: "remove (use the person's / global one)", onClick: () => c.setRef(target, field, "") }, "✕") : null]),
|
||||
]);
|
||||
}
|
||||
|
||||
function editor(c, cur) {
|
||||
const { plan, sel, segs, maskPrev, targetCrop, vlmSug, busy, usedRefs, copyOpts, COPY_FIELDS } = c;
|
||||
const i = sel.value, S = segs.value, k = segKey(cur), mp = maskPrev.value[k], g = vlmSug.value[k];
|
||||
const issues = c.checksFor(i);
|
||||
const guideColor1 = cur.color_guide_color || (cur.color_guide_person != null
|
||||
? (c.castOf(cur.color_guide_person).guide_color || plan.global_color_guide_ref1 || "#00ff00")
|
||||
: (plan.global_color_guide_ref1 || "#00ff00"));
|
||||
const viaRef = f => cur[f] ? "" : (c.castRef(cur, f) || plan["global_" + f] || "");
|
||||
|
||||
const head = h("div", { class: "edhd" }, [
|
||||
h("button", { class: "ghost", disabled: i === 0, title: "previous shot (←)", onClick: () => c.select(i - 1) }, "◀"),
|
||||
h("span", { class: "big" }, `Shot #${i + 1}`),
|
||||
h("button", { class: "ghost", disabled: i >= S.length - 1, title: "next shot (→)", onClick: () => c.select(i + 1) }, "▶"),
|
||||
c.skipWhy(cur) ? pill(`skip · ${c.skipWhy(cur)}`, "warn") : pill("runs", "ok"),
|
||||
hint(`frames ${cur.start}–${cur.end - 1} · ${cur.len} → generate ${cur.gen}`),
|
||||
h("span", { class: "grow" }),
|
||||
h("select", { value: cur.force, style: "width:auto", title: "Override the content filters for this shot",
|
||||
onChange: e => c.setMeta(i, "force", e.target.value) },
|
||||
[h("option", { value: "auto" }, "filters decide"), h("option", { value: "run" }, "always run"), h("option", { value: "skip" }, "always skip")]),
|
||||
h("button", { onClick: () => c.setMeta(i, "enabled", !cur.enabled) }, cur.enabled ? "⏸ Disable" : "▶ Enable"),
|
||||
h("button", { title: "split this shot in two", onClick: () => c.splitAt(cur.start + Math.floor(cur.len / 2)) }, "✂ Split"),
|
||||
h("button", { disabled: i >= S.length - 1, title: "merge with the next shot", onClick: () => c.mergeNext(i) }, "⇥ Merge"),
|
||||
]);
|
||||
const warnsHere = issues.filter(x => x.lvl === "warn");
|
||||
const problems = warnsHere.length ? h("div", { class: "checks", style: "margin-bottom:6px" }, warnsHere.map(x =>
|
||||
h("div", { class: "ck" }, [h("span", { class: "lvl" }, x.lvl === "warn" ? "⚠" : "ℹ"), h("span", x.text)]))) : null;
|
||||
|
||||
const refPrompt = section("Reference & prompt", [
|
||||
h("div", { class: "row", style: "align-items:flex-start;gap:10px" }, [
|
||||
h("div", { class: "refbox" }, [
|
||||
refSlot(c, cur.ref, "⬆ reference", i, "ref", viaRef("ref")),
|
||||
refSlot(c, cur.ref2, "⬆ ref 2", i, "ref2", viaRef("ref2")),
|
||||
]),
|
||||
h("div", { style: "flex:1;min-width:200px" }, [
|
||||
h("textarea", { placeholder: "Prompt for this shot (empty = global prompt). Placeholders: {target} {details} {shot} {setting}",
|
||||
value: cur.prompt, onChange: e => c.setMeta(i, "prompt", e.target.value) }),
|
||||
!cur.prompt && plan.global_prompt ? rawH("span", { class: "hint" }, `↳ uses the global prompt: ${plan.global_prompt.slice(0, 120)}${plan.global_prompt.length > 120 ? "…" : ""}`) : null,
|
||||
]),
|
||||
]),
|
||||
usedRefs.value.length ? h("div", { class: "recent" }, [hint("recent (click = ref, shift+click = ref 2):"),
|
||||
...usedRefs.value.map(n => h("img", { src: c.viewUrl(n), title: n, onClick: e => c.setRef(i, e.shiftKey ? "ref2" : "ref", n) })),
|
||||
h("button", { class: "sm", title: "Forget the recent pictures (the ones this plan still uses stay)", onClick: () => c.clearRecent() }, "✕ clear"),
|
||||
h("button", { class: "sm dng", title: "Remove EVERY reference from this plan (global, all shots, all people) and the recent list", onClick: () => c.clearAllRefs() }, "✕ clear all")]) : null,
|
||||
h("div", { class: "row", style: "margin-top:6px" }, [
|
||||
h("button", { class: "dng", title: "clear this shot's references and prompt: it uses its person's / the global ones",
|
||||
onClick: () => { ["ref", "ref2", "prompt"].forEach(f => { c.meta(i)[f] = ""; }); c.save(); } }, "Use global"),
|
||||
hint("empty slots use the person's reference (Cast) or the global one (Prompts & refs tab)"),
|
||||
]),
|
||||
], { sub: "who the shot becomes" });
|
||||
|
||||
const mode = (on, title, text, onClick) => h("div", { class: ["mode", on && "on"], onClick }, [h("b", title), h("span", text)]);
|
||||
const who = section("Who is replaced", [
|
||||
h("div", { class: "row", style: "align-items:center" }, [
|
||||
h("input", { type: "text", value: cur.mask.text || "", style: "flex:1;min-width:160px",
|
||||
placeholder: "mask: what to segment, in English (woman in pink top, black dog…) · Enter = preview",
|
||||
title: "What to segment, in English: a short phrase (noun + one or two traits), e.g. 'woman in pink top'; commas for several things. Enough on its own - no points needed: SAM 3 finds and tracks it through the shot (every match, up to Max objects). Press Enter to preview. Points, when set, take priority.",
|
||||
onInput: e => c.setMask(i, "text", e.target.value),
|
||||
onKeydown: e => { if (e.key === "Enter") { e.preventDefault(); c.setMask(i, "text", e.target.value); if (e.target.value || (cur.mask.points || []).length) c.previewMask(i); } } }),
|
||||
h("button", { title: "Pick positive / negative points on a frame of this shot (more precise than text when several people look alike)",
|
||||
onClick: () => c.openPoints(i) }, (cur.mask.points || []).length ? `🎯 Points (${cur.mask.points.length})` : "🎯 Points…"),
|
||||
h("button", { class: hasMask(cur) && !mp ? "pri" : "", disabled: !!busy.value || !hasMask(cur),
|
||||
title: (cur.mask.points || []).length ? "Segment this shot (the points win over the text)" : "Segment this shot with the text and show a few frames",
|
||||
onClick: () => c.previewMask(i) }, "👁 Preview"),
|
||||
h("button", { class: "dng", disabled: !(cur.mask.text || (cur.mask.points || []).length || cur.mask.video || cur.crop || cur.inpaint), title: "Remove this shot's mask (points, text, mask video) and mode - for shots where it picked the wrong thing",
|
||||
onClick: () => c.clearMask(i) }, "✕ Clear"),
|
||||
]),
|
||||
h("div", { class: "row", style: "margin-top:6px;align-items:center" }, [
|
||||
h("span", { class: "hint", style: "white-space:nowrap", title: "A black/white video (white = the subject), e.g. rotoscoped in another tool. It replaces SAM 3 for this shot and covers only this shot: its first frame is the shot's first frame." }, "🎞 mask video"),
|
||||
h("select", { value: cur.mask.video || "", style: "flex:1;min-width:160px",
|
||||
onChange: e => c.setMask(i, "video", e.target.value) },
|
||||
[h("option", { value: "" }, cur.extMask ? "— none: the global mask video (People & masks) —" : "— none: SAM 3 (text / points) —"),
|
||||
...c.files.videos.map(v => h("option", { value: v }, v))]),
|
||||
h("button", { title: "upload a mask video for this shot", onClick: () => c.pickFile("video/*", name => c.setMask(i, "video", name)) }, "⬆ Upload"),
|
||||
cur.mask.video ? h("button", { title: "back to SAM 3 / the global mask", onClick: () => c.setMask(i, "video", "") }, "✕") : null,
|
||||
]),
|
||||
maskSource(cur) === "video" ? hint("This shot uses its mask video: the text and points are ignored.", "display:block;margin-top:2px")
|
||||
: maskSource(cur) === "global" ? hint("No mask of its own: this shot uses the global mask video (People & masks tab).", "display:block;margin-top:2px")
|
||||
: maskSource(cur) === "track" ? hint("No mask of its own: this shot uses its part of the whole-video track (People & masks tab).", "display:block;margin-top:2px") : null,
|
||||
mp ? h("div", { class: "mstrip" }, [
|
||||
...mp.frames.map(f => h("img", { src: f.src, title: `frame ${f.f}` })),
|
||||
hint(mp.empty ? "nothing found: the shot runs without a mask" : `mask covers ${(mp.coverage * 100).toFixed(1)}%${cur.crop ? " · yellow = crop box" : ""}`),
|
||||
]) : null,
|
||||
h("div", { class: "modes", style: "margin-top:8px" }, [
|
||||
["frame", "Full frame", "The whole frame is regenerated. The mask, if any, only feeds {target} / the setting picture."],
|
||||
["paste", "🧩 Frame + paste", "The whole frame is generated (the pose follows the guide, the new subject may be bigger); BFS Shot Join pastes only the subject onto the original: background exact."],
|
||||
["mask", "🎭 Mask only", "Only the mask is regenerated, on the whole frame: the rest stays exactly as it was. No crop / uncrop."],
|
||||
["crop", "✂ Crop", "A box around the mask is regenerated (more pixels for a small subject); BFS Shot Join pastes it back."],
|
||||
["cropmask", "✂🎭 Crop + mask", "Inside the crop, only the mask is regenerated: the most detail with the background kept."],
|
||||
].map(([k, t, d]) => mode(c.modeOf(cur) === k, t, d, () => c.setMode(i, k)))),
|
||||
h("div", { class: "row", style: "margin-top:8px;align-items:center;gap:8px" }, [
|
||||
check(!!cur.color_guide, v => c.setMeta(i, "color_guide", v), "🎨 Color Guide",
|
||||
"Tint the selected person in the source guide. This is a translucent visual cue; it does not enable H3 mask-only inpainting."),
|
||||
cur.color_guide ? h("label", { class: "row", style: "gap:6px;align-items:center" }, [
|
||||
h("span", { class: "hint" }, "Ref 1 color"),
|
||||
h("select", { value: cur.color_guide_color ||
|
||||
(cur.color_guide_person != null ? (c.castOf(cur.color_guide_person).guide_color || plan.global_color_guide_ref1 || "#00ff00")
|
||||
: (plan.global_color_guide_ref1 || "#00ff00")),
|
||||
onChange: e => c.setMeta(i, "color_guide_color", e.target.value) },
|
||||
GUIDE_COLORS.map(([v, label]) => h("option", { value: v }, label))),
|
||||
]) : null,
|
||||
]),
|
||||
cur.color_guide ? h("details", { class: "note", style: "margin:4px 0;padding:6px 8px", open: !!cur.color_guide_text2 }, [
|
||||
h("summary", { style: "cursor:pointer" }, cur.color_guide_text2 ? "Second person · Ref 2" : "+ Add a second person (optional)"),
|
||||
h("div", { class: "row", style: "margin-top:8px;align-items:center;gap:8px" }, [
|
||||
h("span", { class: "hint", style: "white-space:nowrap" }, "Ref 2"),
|
||||
h("input", { type: "text", value: cur.color_guide_text2 || "", style: "flex:1;min-width:160px",
|
||||
placeholder: "Describe the second person for SAM 3",
|
||||
onChange: e => c.setMeta(i, "color_guide_text2", e.target.value) }),
|
||||
h("button", { disabled: !!busy.value || !cur.color_guide_text2,
|
||||
onClick: () => c.previewMask(i, { text: cur.color_guide_text2, max_objects: 1 }) }, "👁 Preview"),
|
||||
h("select", { value: cur.color_guide_color2 || (cur.color_guide_person2 != null
|
||||
? (c.castOf(cur.color_guide_person2).guide_color || plan.global_color_guide_ref2 || "#9b30ff")
|
||||
: (plan.global_color_guide_ref2 || "#9b30ff")),
|
||||
title: "Choose the marker color for the person matched to ref 2",
|
||||
onChange: e => c.setMeta(i, "color_guide_color2", e.target.value) },
|
||||
GUIDE_COLORS.map(([v, label]) => h("option", { value: v, disabled: v === guideColor1 }, label))),
|
||||
]),
|
||||
h("details", { style: "margin-top:6px" }, [
|
||||
h("summary", { class: "hint", style: "cursor:pointer" }, "Keep ref 2 color consistent across shots (optional)"),
|
||||
h("div", { class: "row", style: "margin-top:6px;gap:8px" }, [
|
||||
h("span", { class: "hint" }, "Ref 2 identity"),
|
||||
h("select", { value: cur.color_guide_person2 == null ? "" : String(cur.color_guide_person2),
|
||||
onChange: e => { c.setMeta(i, "color_guide_person2", e.target.value === "" ? null : Number(e.target.value)); c.setMeta(i, "color_guide_color2", ""); } },
|
||||
[h("option", { value: "" }, "Choose manually"), ...c.people.value.map(p => h("option", { value: String(p.id) }, `Person ${p.id}`))]),
|
||||
]),
|
||||
]),
|
||||
h("details", { style: "margin-top:6px" }, [
|
||||
h("summary", { class: "hint", style: "cursor:pointer" }, `Advanced · opacity ${Number(cur.color_guide_opacity ?? 0.35).toFixed(2)}`),
|
||||
h("div", { class: "row", style: "margin-top:6px;align-items:center" }, [
|
||||
h("span", { class: "hint" }, "Opacity"),
|
||||
h("input", { type: "range", min: 0.05, max: 0.8, step: 0.01, value: cur.color_guide_opacity ?? 0.35,
|
||||
style: "flex:1;max-width:320px", onInput: e => { c.meta(i).color_guide_opacity = parseFloat(e.target.value); }, onChange: c.save }),
|
||||
hint("The default keeps movement and expression visible."),
|
||||
]),
|
||||
]),
|
||||
]) : null,
|
||||
cur.color_guide ? h("details", { style: "margin:4px 0" }, [
|
||||
h("summary", { class: "hint", style: "cursor:pointer" }, `More options · opacity ${Number(cur.color_guide_opacity ?? 0.35).toFixed(2)}`),
|
||||
h("div", { class: "row", style: "margin-top:6px;gap:8px;align-items:center" }, [
|
||||
h("span", { class: "hint" }, "Ref 1 identity"),
|
||||
h("select", { value: cur.color_guide_person == null ? "" : String(cur.color_guide_person),
|
||||
onChange: e => { c.setMeta(i, "color_guide_person", e.target.value === "" ? null : Number(e.target.value)); c.setMeta(i, "color_guide_color", ""); } },
|
||||
[h("option", { value: "" }, "Choose manually"), ...c.people.value.map(p => h("option", { value: String(p.id) }, `Person ${p.id}`))]),
|
||||
cur.color_guide_person != null ? h("button", { class: "ghost", title: "Use the cast member's assigned color",
|
||||
onClick: () => c.setMeta(i, "color_guide_color", "") }, "↩ cast color") : null,
|
||||
h("span", { class: "hint" }, "Opacity"),
|
||||
h("input", { type: "range", min: 0.05, max: 0.8, step: 0.01, value: cur.color_guide_opacity ?? 0.35,
|
||||
style: "flex:1;max-width:240px", onInput: e => { c.meta(i).color_guide_opacity = parseFloat(e.target.value); }, onChange: c.save }),
|
||||
]),
|
||||
]) : null,
|
||||
cur.color_guide && !hasMask(cur) ? h("div", { class: "err" }, "Color Guide needs this shot's person selection: add mask text, points or a mask video, then preview it.") : null,
|
||||
cur.color_guide && cur.crop && cur.color_guide_text2 ? h("div", { class: "err" }, "The second Color Guide selection uses the full frame; choose Full frame, Frame + paste or Mask only for this shot.") : null,
|
||||
(cur.crop || cur.inpaint || cur.paste) && !hasMask(cur) ? h("div", { class: "err" }, `${c.modeName(cur)} needs a mask on this shot: a mask text, points or a mask video (or the planner's mask input).`) : null,
|
||||
cur.inpaint ? h("div", { class: "row", style: "margin-bottom:6px;align-items:center",
|
||||
title: "Opacity of the generation mask: the value of the white inside the mask (1 = fully white, the default). H3 reads a grey mask value as 'regenerate this much': at 0.85 the masked area starts at 85% of the noise and keeps ~15% of the original there. 0.8-0.9 keeps pose, outline and lighting while still swapping; too low copies the original person." }, [
|
||||
h("span", { style: "white-space:nowrap" }, `Mask opacity ${Number(cur.strength).toFixed(2)}`),
|
||||
h("input", { type: "range", min: 0.3, max: 1, step: 0.01, value: cur.strength, style: "flex:1;max-width:320px",
|
||||
onInput: e => { c.meta(i).strength = parseFloat(e.target.value); }, onChange: c.save }),
|
||||
h("button", { class: "ghost", disabled: cur.strength >= 1, onClick: () => c.setMeta(i, "strength", 1) }, "reset"),
|
||||
hint(cur.strength >= 1 ? "regenerates the masked area completely" : `keeps ~${Math.round(100 - cur.strength * 100)}% of the original inside the mask`),
|
||||
]) : null,
|
||||
cur.inpaint ? h("div", { class: "note", style: "margin-bottom:6px" }, [h("b", "Mask only / Crop + mask: "),
|
||||
"set BFS Shot H3 Conditioning → inpaint to ", h("b", "per shot (planner)"), " (the default). Raise Expand (People & masks tab) when the new subject is bigger than the old one."]) : null,
|
||||
cur.paste ? h("div", { class: "row", style: "margin-bottom:6px;align-items:center",
|
||||
title: "What the NEW subject is, in English, for SAM 3 on the result: BFS Shot Join also pastes its outline (when it is bigger than the old one). E.g. person, dog, cat, car. Empty = person." }, [
|
||||
h("span", { style: "white-space:nowrap" }, "new subject"),
|
||||
h("input", { type: "text", value: cur.pasteText || "", placeholder: "person (or dog, cat, car… what the result shows)", style: "flex:1;max-width:420px",
|
||||
onChange: e => c.setMeta(i, "paste_text", e.target.value) }),
|
||||
hint("its outline on the result is pasted too, so a bigger silhouette is not cut"),
|
||||
]) : null,
|
||||
h("div", { class: "row", style: "margin-top:4px;align-items:center" }, [
|
||||
targetCrop.value[k] ? h("img", { src: targetCrop.value[k], style: "height:56px;border-radius:6px" }) : null,
|
||||
h("span", { class: "hint", style: "white-space:nowrap" }, "{target}"),
|
||||
h("input", { type: "text", value: cur.target, style: "flex:1;min-width:200px",
|
||||
placeholder: "who is replaced, e.g. the young woman in a pink crop top (empty = the person)",
|
||||
title: "Write {target} in the prompt: it becomes this description, so the model knows WHICH person to replace (useful with several people on screen).",
|
||||
onChange: e => c.setMeta(i, "target", e.target.value) }),
|
||||
h("button", { disabled: !!busy.value || !hasMask(cur),
|
||||
title: "Uses this shot's selection (🎯 Points… on the person — click the BODY, not only the face, so the outfit is described — or the mask text): SAM 3 cuts the person out and the VLM describes them",
|
||||
onClick: () => c.describeTarget(i) }, "🧑 Describe target"),
|
||||
]),
|
||||
g ? h("div", { class: "vsug" }, [
|
||||
h("div", ["🤖 ", h("b", "segment: "), g.segment || "—", g.segment ? h("button", { onClick: () => c.applySug(i, "segment") }, "Use as mask") : null]),
|
||||
h("div", [h("b", "shot: "), g.shot || "—", g.shot ? h("button", { title: "copy (the {shot} placeholder in the prompt gets it automatically at run time)",
|
||||
onClick: () => navigator.clipboard?.writeText(g.shot) }, "Copy") : null]),
|
||||
h("div", [h("b", "recommend: "), `${g.recommend}${g.people != null ? " · " + g.people + " people" : ""}${g.reason ? " · " + g.reason : ""}`,
|
||||
g.recommend === "skip" ? h("button", { onClick: () => c.applySug(i, "skip") }, "Skip this shot") : null]),
|
||||
g.raw ? hint("unparsed answer: " + g.raw.slice(0, 200)) : null,
|
||||
]) : null,
|
||||
], { sub: "SAM 3 mask (text or points), crop, {target}" });
|
||||
|
||||
const cont = section("Continuity", h("div", { class: "row" }, [
|
||||
h("select", { value: cur.chain, style: "width:auto", disabled: i === 0,
|
||||
title: "reference: the previous result's frame becomes one more <Picture n> after this shot's own references. first frame: it is anchored at frame 0 of this shot. history: the END of the previous result (17/34/51 frames) goes just before this shot on H3's timeline, so a long take continues its motion and look (same take only, not across a cut).",
|
||||
onChange: e => c.setMeta(i, "chain", e.target.value) },
|
||||
[h("option", { value: "off" }, "off"), h("option", { value: "reference" }, "previous shot as reference"),
|
||||
h("option", { value: "first frame" }, "previous shot as first frame"),
|
||||
h("option", { value: "history" }, "previous shot's end as history (long take)")]),
|
||||
cur.chain === "history"
|
||||
? h("select", { value: String(cur.chainHistory), style: "width:auto", disabled: i === 0,
|
||||
title: "How much of the previous result's end goes before this shot (whole H3 VAE groups). More carries more motion, costs more memory.",
|
||||
onChange: e => c.setMeta(i, "chain_history", Number(e.target.value)) },
|
||||
[17, 34, 51].map(n => h("option", { value: String(n) }, `last ${n} frames (${(n / 24).toFixed(1)} s)`)))
|
||||
: h("select", { value: cur.chainFrame, style: "width:auto", disabled: i === 0 || cur.chain === "off",
|
||||
title: "Which frame of the previous shot's result", onChange: e => c.setMeta(i, "chain_frame", e.target.value) },
|
||||
[h("option", { value: "first" }, "its first frame"), h("option", { value: "middle" }, "its middle frame"), h("option", { value: "last" }, "its last frame")]),
|
||||
hint(i === 0 ? "the first shot uses only its references"
|
||||
: cur.chain === "history" ? (cur.cut ? "⚠ a camera cut starts this shot: history is skipped (it only carries a continuing take)"
|
||||
: "the end of the PREVIOUS shot's result continues into this one (queue loop: shots run in order). Works with every mask mode, crop and the duet canvas.")
|
||||
: "uses a frame of the PREVIOUS shot's result (queue loop, or auto loop with BFS Shot H3 Duet)"),
|
||||
]), { open: cur.chain !== "off", sub: cur.chain !== "off" ? cur.chain : "off" });
|
||||
|
||||
const copy = section("Copy to other shots", [
|
||||
h("div", { class: "copy" }, Object.entries(COPY_FIELDS).map(([f, label]) =>
|
||||
check(copyOpts[f], v => { copyOpts[f] = v; }, label, {
|
||||
ref: "reference + ref 2", prompt: "this shot's prompt", mask_text: "the mask text (shots keep their own points)",
|
||||
points: "the points, on each shot's frame at the same relative position (a subject that stays in place)",
|
||||
mask_video: "the shot's mask video (it covers only its own shot: usually you want a different one per shot)",
|
||||
crop: "the generation mode (Full frame / Frame + paste / Mask only / Crop / Crop + mask) and its mask strength (shots with a mask)", target: "the {target} description",
|
||||
chain: "continuity (not on the first shot)" }[f]))),
|
||||
h("div", { class: "row", style: "margin-top:6px" }, [
|
||||
h("div", { class: "seg-btns" }, [["all", "all shots"], ["after", "shots after this"]].map(([v, l]) =>
|
||||
h("button", { class: copyOpts.scope === v && "on", onClick: () => { copyOpts.scope = v; } }, l))),
|
||||
h("button", { class: "pri", disabled: !Object.keys(COPY_FIELDS).some(f => copyOpts[f]) || S.length < 2,
|
||||
onClick: e => { const n = c.copyFrom(i); const b = e.currentTarget; b.textContent = `✓ copied to ${n} shot(s)`; setTimeout(() => { b.textContent = "📋 Copy"; }, 1400); } }, "📋 Copy"),
|
||||
hint("replaces the old “→ all” buttons"),
|
||||
]),
|
||||
], { open: false, sub: Object.entries(COPY_FIELDS).filter(([f]) => copyOpts[f]).map(([, l]) => l).join(", ") });
|
||||
|
||||
return h("div", { class: "card" }, [head, problems, refPrompt, who, cont, copy]);
|
||||
}
|
||||
|
||||
export function shotsTab(c) {
|
||||
const { segs, sel, showMasks } = c;
|
||||
const S = segs.value, cur = S[sel.value];
|
||||
if (!S.length) return h("div", { class: "card empty" }, [h("b", "No shots yet"), "Pick a video in the Video tab."]);
|
||||
return h("div", [
|
||||
h("div", { class: "card" }, [
|
||||
h("div", { class: "row", style: "margin-bottom:6px" }, [h("span", { class: "sect" }, "Shots"),
|
||||
hint(`${S.length} · scroll sideways · ← → to move`), h("span", { class: "grow" }),
|
||||
check(showMasks.value, v => { showMasks.value = v; }, "show mask previews", "Overlay the mask preview on the shot cards")]),
|
||||
h("div", { class: "shots" }, S.map((s, i) => shotCard(c, s, i))),
|
||||
]),
|
||||
cur ? h("div", { class: "card" }, [player(c, cur)]) : null,
|
||||
cur ? editor(c, cur) : null,
|
||||
]);
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
// Timeline: thumbnails, cut marks, cut score, shot bars (click = select, drag the white handles, double-click = split).
|
||||
import { h, hue, fmtT, pill } from "./common.js";
|
||||
|
||||
export function timelineCard(c) {
|
||||
const { plan, an, segs, sel, n, pxPerFrame, tlWidth, maxLen, cuts, hover, tlEl, zoom, play, vid, active } = c;
|
||||
const S = segs.value, N = n.value, ppf = pxPerFrame.value, fps = plan.fps;
|
||||
const thumbs = an.value?.thumbs || [];
|
||||
const thumbW = Math.max(8, thumbs.length ? tlWidth.value / thumbs.length : 96);
|
||||
const score = an.value?.score || [];
|
||||
const maxS = Math.max(4, ...score.slice(0, N));
|
||||
const sparkPts = score.slice(0, N).map((v, i) => `${(i * ppf).toFixed(1)},${(24 - Math.min(1, v / maxS) * 22).toFixed(1)}`).join(" ");
|
||||
const tooLong = S.filter(s => s.len > maxLen.value).length;
|
||||
const totalGen = active.value.reduce((a, s) => a + s.gen, 0);
|
||||
return h("div", { class: "card" }, [
|
||||
h("div", { class: "row", style: "margin-bottom:6px" }, [
|
||||
h("span", { class: "sect" }, "Timeline"),
|
||||
pill(`${S.length} shots`), pill(`${cuts.value.length} cuts`), pill(`generate ${totalGen} frames`),
|
||||
tooLong ? pill(`${tooLong} too long (> ${maxLen.value} f)`, "warn") : null,
|
||||
h("span", { class: "grow" }),
|
||||
h("span", { class: "hint" }, "zoom"),
|
||||
h("input", { type: "range", min: 1, max: 8, step: 0.5, value: zoom.value, style: "width:100px", onInput: e => { zoom.value = parseFloat(e.target.value); } }),
|
||||
]),
|
||||
h("div", { class: "tl", ref: tlEl,
|
||||
onPointermove: e => { if (!e.buttons) hover.value = c.frameAt(e); }, onPointerleave: () => { hover.value = null; },
|
||||
onClick: e => { if (e.target.classList.contains("hdl")) return; c.seek(c.frameAt(e).frame); } }, [
|
||||
h("div", { class: "tlin", style: `width:${tlWidth.value}px` }, [
|
||||
h("div", { class: "strip" }, thumbs.map(t => h("img", { src: t.src, style: `width:${thumbW}px` }))),
|
||||
// outside the start / end range: dimmed, not planned
|
||||
c.range.value.lo > 0 ? h("div", { class: "outr", style: `left:0;width:${c.range.value.lo * ppf}px`, title: "before the start: not planned" }) : null,
|
||||
c.range.value.hi < N ? h("div", { class: "outr", style: `left:${c.range.value.hi * ppf}px;width:${(N - c.range.value.hi) * ppf}px`, title: "after the end: not planned" }) : null,
|
||||
...cuts.value.filter(x => x < N).map(x => h("div", { class: "cut", style: `left:${x * ppf}px`, title: `cut @ ${x}` })),
|
||||
h("svg", { class: "spark", viewBox: `0 0 ${tlWidth.value} 26`, preserveAspectRatio: "none" },
|
||||
[h("polyline", { points: sparkPts, fill: "none", stroke: "#ff7a90", "stroke-width": 1, "vector-effect": "non-scaling-stroke" })]),
|
||||
h("div", { class: "segs", onDblclick: e => c.splitAt(c.frameAt(e).frame) }, [
|
||||
...S.map((s, i) => h("div", {
|
||||
class: ["seg", i === sel.value && "sel", !!c.skipWhy(s) && "off", s.len > maxLen.value && "long", play.mode && play.idx === i && "playing"],
|
||||
style: `left:${s.start * ppf}px;width:${Math.max(2, s.len * ppf - 1)}px;background:${hue(i)}`,
|
||||
title: `#${i + 1} · frames ${s.start}-${s.end - 1} · ${s.len} → ${s.gen}${c.skipWhy(s) ? " · skip: " + c.skipWhy(s) : ""}`,
|
||||
onClick: () => { sel.value = i; },
|
||||
}, s.len * ppf > 26 ? `${i + 1}` : "")),
|
||||
...plan.bounds.filter(b => b < N).map((b, k) => h("div", { class: "hdl", style: `left:${b * ppf}px`, title: `boundary @ ${b}`, onPointerdown: e => c.drag(k, e) })),
|
||||
]),
|
||||
hover.value ? h("div", { class: "ph", style: `left:${hover.value.x}px` }) : null,
|
||||
vid.value && plan.video ? h("div", { class: "playhead", style: `left:${Math.min(N, play.frame) * ppf}px` }) : null,
|
||||
]),
|
||||
hover.value ? h("div", { class: "tip", style: `left:${Math.min(tlWidth.value - 160, Math.max(0, hover.value.x - (tlEl.value?.scrollLeft || 0) - 70))}px;top:2px` },
|
||||
[h("img", { src: c.thumbFor(hover.value.frame) }), h("div", `frame ${hover.value.frame} · ${fmtT(hover.value.frame, fps)}`)]) : null,
|
||||
]),
|
||||
h("div", { class: "hint", style: "margin-top:4px" },
|
||||
"Click a shot to select it (← → move between shots) · drag the white handles to move a boundary · double-click the shot bar to split · Delete removes the selected shot's cut"),
|
||||
]);
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
// 🎞 Video tab: the source video, how it is split into shots, and the generation size.
|
||||
import { h, GRIDS, fld, select, section, pill, hint, fmtT } from "./common.js";
|
||||
|
||||
export function videoTab(c) {
|
||||
const { plan, an, files, size, busy, detectorUsed, maxLen } = c;
|
||||
const fps = plan.fps;
|
||||
const num = (k, step = 0.1, min = 0) => h("input", { type: "number", step, min, value: plan[k], onChange: e => c.setPlan(k, parseFloat(e.target.value) || 0) });
|
||||
const sel_ = (k, opts, after) => select(plan[k], opts, v => c.setPlan(k, v, after));
|
||||
const au = an.value?.audio;
|
||||
|
||||
const source = section("Source video", [
|
||||
h("div", { class: "row" }, [
|
||||
h("div", { style: "flex:1;min-width:180px" }, [h("select", { value: plan.video, onChange: e => c.setVideo(e.target.value) },
|
||||
[h("option", { value: "" }, "— choose a video from the input folder —"), ...files.videos.map(v => h("option", { value: v }, v))])]),
|
||||
h("button", { class: plan.video ? "" : "pri", onClick: () => c.pickFile("video/*", c.setVideo) }, "⬆ Upload"),
|
||||
h("button", { onClick: c.refreshFiles, title: "refresh the input folder" }, "↻"),
|
||||
]),
|
||||
an.value ? h("div", { class: "row", style: "margin-top:6px" }, [
|
||||
pill(`${an.value.width}×${an.value.height}`), pill(`${an.value.fps_src.toFixed(2)} fps`), pill(`${an.value.duration.toFixed(2)} s`),
|
||||
pill(`timeline ${fps} fps · ${an.value.n} frames`), pill(`generate at ${size.w}×${size.h}`, "acc"),
|
||||
au?.has_audio
|
||||
? pill(`🔊 audio · ${au.sample_rate ? (au.sample_rate / 1000).toFixed(1) + " kHz" : "?"} · ${au.channels || "?"} ch`, "ok", au.codec || "")
|
||||
: pill(`🔇 ${au?.reason || "no audio"} → silent track`, "warn", "The outputs carry a silent track of the right length, so Create Video works"),
|
||||
h("select", { value: plan.audio_mode, style: "width:auto", disabled: !au?.has_audio, title: "Audio of the planner's and the join's outputs",
|
||||
onChange: e => c.setPlan("audio_mode", e.target.value) },
|
||||
[h("option", { value: "auto" }, "use the video's audio"), h("option", { value: "silent" }, "silent track")]),
|
||||
]) : plan.video ? null : h("div", { class: "empty" }, [h("b", "Pick or upload a video to start"),
|
||||
"It is split into shots the model can generate; every shot then gets its own reference, prompt and mask."]),
|
||||
]);
|
||||
|
||||
const R = c.range.value, N = c.n.value;
|
||||
const playF = Math.min(N - 1, Math.max(0, c.play.frame || 0));
|
||||
// a player right here, so the range is picked by watching (scrub it, or click the timeline above)
|
||||
const rng = section("Range", [
|
||||
h("div", { class: "player" }, [
|
||||
h("video", { ref: c.vid, src: c.viewUrl(plan.video), preload: "metadata", controls: true, playsinline: true,
|
||||
style: "max-height:240px" }),
|
||||
h("div", { class: "pinfo" }, [
|
||||
h("div", { class: "tc" }, [
|
||||
h("div", ["now ", h("b", fmtT(playF, fps)), ` (frame ${playF})`]),
|
||||
h("div", ["start ", h("b", fmtT(R.lo, fps)), ` (frame ${R.lo})`]),
|
||||
h("div", ["end ", h("b", fmtT(R.hi, fps)), ` (frame ${R.hi})`]),
|
||||
]),
|
||||
h("div", { class: "rgrid" }, [
|
||||
h("button", { class: "pri", title: "the range starts at the player's frame", onClick: () => c.setRange("start_f", playF) }, "⇤ start here"),
|
||||
fld("Start frame", h("input", { type: "number", step: 1, min: 0, max: N - 1, value: R.lo,
|
||||
onChange: e => c.setRange("start_f", e.target.value) }), fmtT(R.lo, fps)),
|
||||
h("button", { class: "pri", title: "the range ends after the player's frame", onClick: () => c.setRange("end_f", playF + 1) }, "⇥ end here"),
|
||||
fld("End frame (0 = end)", h("input", { type: "number", step: 1, min: 0, max: N, value: R.hi >= N ? 0 : R.hi,
|
||||
onChange: e => c.setRange("end_f", e.target.value) }), fmtT(R.hi, fps)),
|
||||
(R.lo > 0 || R.hi < N) ? h("button", { class: "ghost", onClick: () => { plan.start_f = 0; c.setRange("end_f", 0); } }, "✕ whole video") : h("span"),
|
||||
]),
|
||||
hint(`Play or scrub the video (or click the timeline), then ⇤ / ⇥. ${R.hi - R.lo} frames (${((R.hi - R.lo) / fps).toFixed(2)} s) of ${N}: only this part is split, analysed, generated and joined.`),
|
||||
]),
|
||||
]),
|
||||
], { sub: "work on a part of the video" });
|
||||
|
||||
const split = section("Split into shots", [
|
||||
h("div", { class: "grid" }, [
|
||||
fld("Mode", sel_("mode", [["shots", "Camera cuts"], ["fixed", "Fixed length"], ["single", "One shot (whole range)"]]),
|
||||
plan.mode === "single" ? "ignores Max seconds / shot" : ""),
|
||||
// the range in seconds, here next to the mode: "15 s in one shot" is set without leaving this tab
|
||||
fld("Range length, seconds (0 = to the end)", h("input", { type: "number", step: 0.5, min: 0,
|
||||
value: c.range.value.hi < c.n.value ? +((c.range.value.hi - c.range.value.lo) / plan.fps).toFixed(2) : 0,
|
||||
onChange: e => { const v = parseFloat(e.target.value) || 0;
|
||||
c.setRange("end_f", v > 0 ? c.range.value.lo + Math.round(v * plan.fps) : 0); } }),
|
||||
`from the start frame · ${c.range.value.hi - c.range.value.lo} frames`),
|
||||
fld("Detector", sel_("detector", [["adaptive", "PySceneDetect adaptive"], ["content", "PySceneDetect content"], ["builtin", "Built-in"]])),
|
||||
fld(`Sensitivity ${Number(plan.sensitivity).toFixed(2)}`, h("input", { type: "range", min: 0, max: 1, step: 0.05, value: plan.sensitivity,
|
||||
onInput: e => { plan.sensitivity = parseFloat(e.target.value); }, onChange: c.save })),
|
||||
fld("Frame grid", sel_("grid", Object.keys(GRIDS))),
|
||||
fld("Max seconds / shot", num("max_s", 0.1, 0.2), `= ${maxLen.value} frames (${(maxLen.value / fps).toFixed(2)} s)`),
|
||||
fld("Min seconds / shot", num("min_s", 0.1, 0)),
|
||||
fld("Timeline fps", num("fps", 1, 1), "re-analyses"),
|
||||
]),
|
||||
h("div", { class: "row", style: "margin-top:8px" }, [
|
||||
h("button", { class: "pri", disabled: !plan.video || !!busy.value, onClick: () => c.autoSplit(true) }, "✂ Auto split"),
|
||||
h("button", { disabled: !plan.video, onClick: c.analyze }, "↻ Re-analyse"),
|
||||
detectorUsed.value ? hint(`detector: ${detectorUsed.value}`) : null,
|
||||
hint("Auto split replaces the boundaries; per-shot settings stay with the shot at the same position."),
|
||||
]),
|
||||
], { sub: "camera cuts or fixed length, on the model's frame grid" });
|
||||
|
||||
const gen = section("Generation size", h("div", { class: "grid" }, [
|
||||
fld("Megapixels", num("megapixels", 0.01, 0.02), size.w ? `${size.w}×${size.h}` : ""),
|
||||
fld("Size multiple", num("multiple", 8, 8)),
|
||||
]), { sub: "cropped shots use it for the crop" });
|
||||
|
||||
return h("div", { class: "card" }, [source, an.value ? rng : null, an.value ? split : null, an.value ? gen : null]);
|
||||
}
|
||||
+9
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user