Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
605b6ed785 | ||
|
|
59f4bfd75d | ||
|
|
84caa6575a | ||
|
|
cc9e3f90be | ||
|
|
0577009f76 | ||
|
|
19599a1015 | ||
|
|
a2b847d6ce | ||
|
|
7626226b8e | ||
|
|
4af713f185 | ||
|
|
b9885ecfb9 | ||
|
|
0ce4c673f6 | ||
|
|
3f440788cf | ||
|
|
761df4b43d | ||
|
|
be21e891dd | ||
|
|
5e45c815fd | ||
|
|
64be9ced98 | ||
|
|
1e74b9c365 | ||
|
|
deb243cf2a | ||
|
|
4dbb29f68b | ||
|
|
a43183074f | ||
|
|
1ff7c1c9e1 | ||
|
|
5a4191f56c | ||
|
|
3ce2eb1ea0 | ||
|
|
27872290fe | ||
|
|
488cb77710 | ||
|
|
0cd1c04d17 | ||
|
|
857ba260f3 | ||
|
|
a93cea17a9 | ||
|
|
7b57eacb7b | ||
|
|
c37ce27ea8 | ||
|
|
0d831b0bb5 | ||
|
|
d94e1c2b19 | ||
|
|
6f335175d9 | ||
|
|
7da6f7427f | ||
|
|
084e1d22b5 | ||
|
|
55b120d14b | ||
|
|
526745455f | ||
|
|
78a806103e | ||
|
|
ce3ca59b17 |
@@ -0,0 +1,192 @@
|
||||
# Experimental H3 native temporal alignment
|
||||
|
||||
Matches ai-toolkit branch [`feat/h3-native-temporal-motion`](https://github.com/alisson-anjos/ai-toolkit/tree/feat/h3-native-temporal-motion), implementation `native_temporal_v1` (commit e0ecf3a). This mode retains a native video reference, aligns its temporal RoPE coordinates with the target, and softly penalizes target queries attending to distant guide times. It preserves native image-reference coordinates and native spatial geometry. It does not add learned weights or load a LoRA.
|
||||
|
||||
Existing Add Guide motion-transfer LoRAs were trained under a different conditioning contract, but the step-10500 example below shows useful transfer to this native temporal mode. This is an experimental observation, not a guarantee for every LoRA. CPU math/routing tests alone do not validate video quality.
|
||||
|
||||
## Video comparison: an Add Guide-trained LoRA with native temporal alignment
|
||||
|
||||
[**Play the synchronized side-by-side video**](https://github.com/alisson-anjos/ComfyUI-BFSNodes/raw/refs/heads/feat/h3-native-temporal/docs/media/h3_native_temporal_comparison.mp4)
|
||||
|
||||
All six columns play together in one 15-second video, starting at the beginning of each supplied export. From left to right:
|
||||
|
||||
1. Original guide.
|
||||
2. Step-10500 motion-transfer LoRA + native temporal alignment, with guide scale **1.0**.
|
||||
3. The same step-10500 LoRA + native temporal alignment, with guide scale **0.5**.
|
||||
4. The same step-10500 LoRA + guide latents / Add Guide.
|
||||
5. The same step-10500 LoRA + native reference, without the temporal patch.
|
||||
6. Base model + native reference, without a LoRA or temporal patch.
|
||||
|
||||
The temporal columns are labeled **NEW**. The footer includes the character and setting reference thumbnails extracted from the supplied exports. The user confirmed that all supplied runs used the **same seed**. At guide scale 0.5, character appearance remains consistent in the reviewed samples, but some source poses are reproduced less closely than at 1.0 (for example, the raised-arm gesture around 3.7 seconds). Other generation settings were not fully captured in the exports.
|
||||
|
||||
**Apparently, a LoRA trained with guide latents can also work with this new native temporal method.** In this example, the step-10500 Add Guide-trained LoRA follows the source poses and camera framing with the temporal patch, while maintaining the reference dog's white T-shirt and jeans through the sampled beginning, middle and end. Its Add Guide result also follows the performance. Native reference without the temporal patch preserves the character more successfully with this LoRA than without it, but reproduces the choreography less closely. The base native-reference result loses the reference T-shirt.
|
||||
|
||||
A plausible explanation is that the LoRA's learned appearance and motion-transfer behavior can be reused through shared model weights, while the runtime patch supplies temporal correspondence through native reference attention. This example does not isolate that mechanism experimentally or establish universal compatibility. The temporal mode still uses native video-reference latents; it does **not** inject Add Guide latents.
|
||||
|
||||
**The reported identity drift concerns the separate, fresh native-temporal LoRA at step 750, not the step-10500 LoRA shown here.** The 750 checkpoint is not included in this comparison. Comparing those checkpoints would also compare different training histories and training durations.
|
||||
|
||||
The requested background replacement is partial: source architecture and layout remain mixed with the new street reference. The seed is confirmed equal by the user, but these supplied exports do not include complete prompt and configuration metadata, so this is a qualitative demonstration rather than a controlled benchmark. Pixel difference and PSNR against the guide cannot rank replacement quality when character and background changes are intentional. The 30 fps comparison export does not establish the model's generation frame rate.
|
||||
|
||||
## Standalone native workflow
|
||||
|
||||
1. Load the native MiniMax H3 base and the native-temporal LoRA.
|
||||
2. Pass the guide as `<Video 1>` to **MiniMax H3 Reference to Video**, alongside any number of identity reference pictures supported by that native node. Picture and video numbering are independent.
|
||||
3. Place **BFS MiniMax H3 Native Temporal Alignment (Experimental)** on the MODEL path, after LoRA loading. Route its MODEL output to your guider/sampler. Keep Reference to Video's positive and AV latent outputs connected normally.
|
||||
4. Select `motion_video_index = 1` (or the actual `<Video N>` supplying motion).
|
||||
5. Use the same 24 fps frame window for guide and output. H3 may trim frame counts to its supported cadence; use a matching valid count such as 107.
|
||||
|
||||
Do not add Add Guide, first-frame guides, VSA, SLA, or unknown attention/block replacements. Model Sparse Attention with sol-attn is experimentally supported from 1.115.14; see its limitations below. The official Model Attention Backend node is supported from 1.115.13; see the Kitchen section below. The node raises a clear error for incompatible layouts and temporal lengths instead of silently stretching motion. Source video audio is supported by native packing but is not temporally biased by this patch. For CFG paths, every conditioning branch evaluated by the patched model must retain the native motion reference; an empty negative branch without references cannot provide its timeline. Training preview uses CFG 1.
|
||||
|
||||
## Planner
|
||||
|
||||
In **BFS Shot H3 Conditioning** (Advanced or Simple), choose:
|
||||
|
||||
```text
|
||||
guide_mode = native temporal alignment (experimental)
|
||||
```
|
||||
|
||||
Connect MODEL from your LoRA loader to the conditioning node, then connect the conditioning node's MODEL output to the guider. Advanced keeps its existing output order. Simple appends MODEL after positive/latent/prompt; old links keep their indices. Reload older nodes or recreate them if the new socket does not appear.
|
||||
|
||||
The shot is encoded as native `<Video 1>`, with identity pictures unchanged. The patch is applied separately to the returned model for each shot, and the current shot's packed temporal grid is used at sampling. Use 24 fps, first_frame none, mask_guide off and continuity off. Full-frame or ordinary native generation masks may still be used; masks do not create a temporal guide. Colored source overlays are retained if enabled, but color/mesh datasets have not yet been validated with this fresh experiment.
|
||||
|
||||
The Planner option uses the training defaults. To tune the policy manually, instead choose the ordinary `native reference video (Video 1)` conditioning mode and place the standalone node on the model path. Never install both patches on the same model path.
|
||||
|
||||
## Exact policy
|
||||
|
||||
Only target-video query to selected-motion-video key edges receive:
|
||||
|
||||
```text
|
||||
score -= min(strength * (max(abs(t_target - t_guide) - window_seconds, 0) / falloff_seconds)^2, max_penalty)
|
||||
```
|
||||
|
||||
Defaults: strength 2.0, window 0.20 s, falloff 0.25 s, maximum penalty 6.0. Set strength 0 to test temporal RoPE alignment alone. Bypass the node to test fully native baseline. Text, image references, target self-attention, reference audio, and token-refiner attention retain their normal behavior.
|
||||
|
||||
As of 1.115.11 the default backend is **flash partitions (no compile)**: it uses native CUDA FlashAttention with log-sum-exp outputs, positive partition merging and bounded frame chunks. No dense S×S bias tensor or torch.compile is used in this backend. CUDA FP16/BF16 and PyTorch's native FlashAttention-with-LSE primitive are required. The original compiled FlexAttention remains selectable for comparison. Start with model compilation/malloc graph optimizations disabled until validated on your local ComfyUI build.
|
||||
|
||||
Runtime confirmation:
|
||||
|
||||
```text
|
||||
[BFS H3 native temporal] native_temporal_v1: <Video 1>, ... motion rows; temporal RoPE aligned; soft window=0.2s strength=2.0 cap=6.0; images remain native; Add Guide OFF
|
||||
```
|
||||
|
||||
This log confirms the patch ran, not that visual motion or identity quality is correct.
|
||||
|
||||
## Validation
|
||||
|
||||
Tests compare an audited native ComfyUI layout with positions exported by the actual training implementation, including image references and reference audio. They verify original conditioning/layout immutability, temporal-length rejection, attention score math/output shapes, token-refiner fallback, per-model routing and incompatible patches. Planner tests verify native-video conditioning and returned MODEL routing. The user has since reported completed RTX 5090 model evaluations; the timings and limitations are documented in the illustrated explanation. Full visual-quality validation remains incomplete. The active training process is not restarted or modified by this source change.
|
||||
|
||||
|
||||
## Performance changes in 1.115.11
|
||||
|
||||
See the [illustrated Flash Partitions explanation](docs/H3_FLASH_PARTITIONS.md) for attention scope, derivation, log-sum-exp merging, measurements and limitations.
|
||||
|
||||
The temporal rule and RoPE policy are unchanged. For motion keys, the modified softmax weight is decomposed exactly as `exp(-cap) + (exp(-penalty) - exp(-cap))`. The first term is ordinary attention over all motion keys. The positive correction is zero for distant times and can be computed with short per-frame batches. Non-motion keys keep weight one. All partitions are recombined with their log-sum-exp normalizers. This is not a hard temporal window or sparse approximation. BF16 partition merging has normal floating-point rounding differences; the small CUDA comparison against SDPA measured relative output error 0.00452.
|
||||
|
||||
A synthetic long-sequence benchmark on RTX PRO 6000 Blackwell / PyTorch 2.12.0+cu130, 107126 tokens, 8 heads, head dimension 128 measured approximately 0.628 s for original FlexAttention, 0.422 s for Flash partitions and 0.327 s for unmodified native FlashAttention. Training was running concurrently, so these are indicative isolated-kernel timings, not whole-model inference results or RTX 5090 speed guarantees. Small sequences have additional partition-launch overhead and can favor FlexAttention. No full-model GPU generation is claimed here.
|
||||
|
||||
Standalone node: `attention_backend` selects the implementation. Planner conditioning: `native_temporal_backend` selects it. Neither setting loads a LoRA or changes its weights. Defaults select Flash partitions; choose `flex (original)` for an A/B comparison.
|
||||
|
||||
Planner `native_guide_scale` defaults to 1.0 for compatibility. Try 0.5 on VRAM-constrained GPUs: it reduces only the motion-reference spatial dimensions (about four times fewer guide tokens), keeping all timestamps/frames, output dimensions and identity images unchanged. This changes the visual detail supplied by the guide and must be evaluated for quality. In standalone workflows resize only the motion video before Reference to Video to get the same effect.
|
||||
|
||||
The console logs the first attention's input shape and completion time and the first two full model evaluation times. If attention is fast but a full model evaluation remains slow, investigate model loading/offload and the remaining blocks rather than assuming compilation. These timings synchronize CUDA at those diagnostic points only. The active training code and its backend are unchanged.
|
||||
|
||||
## One-pass native Kitchen kernels (1.115.16)
|
||||
|
||||
**From 1.115.17, BFS prepares the one-pass extension automatically on first use of Kitchen/sol-attn on supported Linux x86_64 SM120 systems with a driver supporting CUDA 13.** It downloads a fixed release into ComfyUI’s user cache, verifies embedded SHA256 hashes and ABI/architecture, and activates it in that execution. No pip install, package replacement or sampling-time compilation is required. Other systems retain the compatibility backends or can use an explicitly built manual overlay.
|
||||
|
||||
For an explicit installation or source build, use the [Kitchen fork's setup guide](https://github.com/alisson-anjos/comfy-kitchen/blob/feat/h3-native-temporal/H3_NATIVE_TEMPORAL_KERNELS.md). An [experimental Linux x86_64 SM120 prebuilt](https://github.com/alisson-anjos/comfy-kitchen/releases/tag/h3-native-temporal-kernel-v1) is available, including SHA256SUMS, so sampling need not compile anything. RTX 5090 shares SM120 with the tested RTX PRO 6000, but full video behavior/performance on the 5090 still requires validation. The installer adds a separate temporal module/library while preserving ordinary Kitchen's `_C` and operators. Source builds use `nvcc` explicitly; no per-shape torch.compile is used.
|
||||
|
||||
Keep the existing MODEL wiring and native temporal guide mode. Model Attention Backend's Kitchen choice selects native dense when available. Model Sparse Attention's sol-attn choice selects native Sol during its eligible sparse window; its dense fallback retains the temporal policy too. No new UI switch is required. Console confirmation:
|
||||
|
||||
```text
|
||||
Kitchen ONE-PASS dense kernel ACTIVE
|
||||
Kitchen ONE-PASS sol kernel ACTIVE
|
||||
```
|
||||
|
||||
Both kernels evaluate the same query-dependent bounded temporal penalty inside their native score/softmax work. Dense INT8 quantizes once and runs one attention launch, using exact constant-tile shortcuts plus per-score handling at mixed boundaries. Sol preprocesses/routes/consumes once with the penalty fused into its exact consumer. All conditioning keys remain exact for Sol, and its remaining video keys retain the usual sparse/pooled approximation. Sol still has its normal multi-kernel stages: "one pass" means one attention pipeline, not literally one CUDA launch total. The streaming QKV producer remains bypassed by the current BFS integration.
|
||||
|
||||
Synthetic isolated BF16 attention `[1,56,42836,128]`, 22 latent frames, 7,062 guide rows; RTX PRO 6000 Blackwell, PyTorch 2.12.0+cu130, Kitchen 0.2.35, concurrent training, one warmup plus one timed evaluation:
|
||||
|
||||
| Backend | Seconds |
|
||||
| --- | ---: |
|
||||
| Flash Partitions | 0.4232 |
|
||||
| Kitchen dense, earlier frame-wise path | 0.5413 |
|
||||
| **Kitchen dense, one-pass kernel** | **0.2463** |
|
||||
| sol-attn, earlier frame-wise path | 2.6752 |
|
||||
| **sol-attn, one-pass kernel** | **0.2163** |
|
||||
|
||||
These are not full-model/video or RTX 5090 timings. Full video speed can still be limited by offloading and other work. INT8 and Sol are approximate, so these measurements do not establish visual quality or dense-equivalent output.
|
||||
|
||||
GPU checks covered FP16/BF16, partial frames/tiles, nonuniform timestamps, zero strength/cap, the real BFS dense/Sol entry points and different policies on concurrent streams. Dense relative output norm error versus FP32 was approximately 1.2–1.6% in small cases; Sol with every key exact approximately 1.1–1.4%. Native one-pass Sol versus the same native pipeline repeated per frame differed by at most about 0.0022% in those initial cases. Each BFS bridge made one native API call. [Recorded diagnostics](docs/benchmarks/h3_onepass_temporal_kernels.json) and the Kitchen fork's source document the scope. Full ComfyUI generated-video validation remains pending.
|
||||
|
||||
The sections below retain the earlier compatibility implementations and measurements for comparison. The active training process/backend is unchanged.
|
||||
|
||||
## Automatic first-use setup (1.115.17)
|
||||
|
||||
Choose Kitchen or sol-attn normally and run the native temporal node. On supported Linux x86_64 / NVIDIA SM120 / CUDA-13-driver systems, it prepares the extension during node execution when the model's load device is known, before sampling. A lazy fallback can prepare it on the first attention call too. The ordinary Flash/Flex path does not request an installation.
|
||||
|
||||
The default cache is `ComfyUI user directory/bfs/h3_temporal_kernels/`; standalone use falls back to `~/.cache/bfsnodes/h3_temporal_kernels/`. Only the pinned Python API (~10 KB), SM120 library (~3.3 MB), LICENSE and NOTICE are downloaded. The implementation writes to its own cache, not `site-packages`, and leaves ordinary Kitchen's `_C` untouched. Existing compatible manual overlays are preferred.
|
||||
|
||||
Security/correctness properties: immutable source commit and embedded hashes; bounded download sizes; temporary downloads replaced atomically only after SHA256 validation; native ABI and GPU architecture checks before Python entry-point execution; thread/process installation locks; one setup attempt per device/cache location per process. An incomplete, incompatible or offline first installation logs its reason and retains the previous backend. A verified cache works in a new process without network access. No model, prompt or reference data is uploaded.
|
||||
|
||||
Expected console messages:
|
||||
|
||||
```text
|
||||
First-use setup: downloading the verified one-pass extension (~3.3 MB). No compilation or pip install.
|
||||
Verified one-pass extension ready: ...
|
||||
Kitchen ONE-PASS dense kernel ACTIVE
|
||||
# or: Kitchen ONE-PASS sol kernel ACTIVE
|
||||
```
|
||||
|
||||
The addon activates without a second restart during that first workflow. Restarting ComfyUI is still required after updating BFS's own Python code. `BFS_H3_TEMPORAL_AUTO_INSTALL=0` disables automatic setup; `BFS_H3_TEMPORAL_CACHE_DIR=/path/to/cache` chooses another cache location. For a failed setup, restart to retry or use the manual installer linked above. Unsupported systems do not trigger an automatic source build.
|
||||
|
||||
A real first-use download, ABI check, and dense/Sol GPU calls passed. A separate fresh process with network requests deliberately disabled reused the verified cache successfully. [Recorded first-use/offline checks](docs/benchmarks/h3_temporal_auto_setup.json).
|
||||
|
||||
## Comfy Kitchen and Model Attention Backend (1.115.13)
|
||||
|
||||
The official **Model Attention Backend** node is now recognized automatically. Select **comfy kitchen attention** there and route its MODEL through native temporal conditioning to the guider. Its official PyTorch selection is also accepted. Unrelated attention overrides remain incompatible. Model Sparse Attention with sol-attn is supported separately from 1.115.14, as documented below. The runtime MODEL wrapper recognizes a supported dense selection applied before or after the temporal patch; keep only one temporal patch on the path.
|
||||
|
||||
With Kitchen selected, the console reports `backend=comfy kitchen (experimental)`. You can also explicitly select this backend in the standalone node or Planner conditioning. Without an upstream Kitchen selection, the existing default remains Flash Partitions. If ComfyUI itself falls back from unavailable Kitchen to PyTorch, the temporal patch does not claim Kitchen was selected. A supported selection from Model Attention Backend takes precedence over the BFS backend selector.
|
||||
|
||||
This backend calls `comfy_kitchen.int8_attention(attn_mask=...)` directly. Each target latent frame's spatial queries share one broadcast key bias. Non-target queries use ordinary Kitchen attention, while target-to-guide scores receive the same bounded temporal penalty as the original rule. Native temporal RoPE alignment remains active. It uses no Add Guide injection, no partition log-sum-exp, and no full sequence-by-sequence bias allocation: bias storage is `latent_frames * packed_rows`. It retains full-key attention, not sparse attention. Multiple frame calls can repeat Kitchen quantization work, so compatibility does not imply acceleration.
|
||||
|
||||
Initial isolated tests on an RTX PRO 6000 Blackwell with PyTorch 2.12.0+cu130 and comfy-kitchen 0.2.35:
|
||||
|
||||
- CPU tests compare the frame-wise policy against the dense mathematical oracle, including zero strength/cap and unchanged non-target queries. Routing tests cover automatic detection of the official dense node.
|
||||
- A real CUDA BF16 test at `[1, 2, 1248, 128]` measured relative output norm error **0.01506** against the FP32 oracle, with finite outputs. INT8 quantization is approximate; this is not a visual quality measurement.
|
||||
- At `[1, 2, 42836, 128]`, 22 latent frames and 7,062 guide rows, one warmed evaluation measured **0.1043 s Kitchen** versus **0.01739 s Flash Partitions**. This initial Kitchen path was slower. The test used two heads rather than H3's full 56, with concurrent training; these are diagnostics, not end-to-end or RTX 5090 timings. No speed claim or default-backend change follows from this result.
|
||||
|
||||
The active training backend is unchanged. Full ComfyUI video output and local performance still require validation. Version 1.115.14 adds a separate experimental sol-attn compatibility path below; SLA and VSA remain unsupported.
|
||||
|
||||
## Model Sparse Attention: sol-attn compatibility (1.115.14)
|
||||
|
||||
Use **1.115.15 or newer**: 1.115.15 fixes recognition when ComfyUI loads `nodes_sparse_attention.py` under its extensionless absolute file path instead of a dotted module name. The regression test uses extracted official node factories under that loader name.
|
||||
|
||||
**This is an experimental compatibility implementation, not a speed optimization.** It automatically recognizes the official **Model Sparse Attention** node with `method = sol-attn`, keeps native temporal RoPE alignment and applies the temporal score penalty through Comfy Kitchen's `sol_attn(key_bias=...)`. The runtime banner identifies `backend=sol-attn frame bias (experimental)`; an `sol-attn ACTIVE` line confirms a sparse call actually ran. Dense windows can use the previously selected dense temporal backend, and the first attention log identifies that actual backend.
|
||||
|
||||
Suggested MODEL wiring:
|
||||
|
||||
```text
|
||||
LoRA loader -> Model Attention Backend (optional dense fallback)
|
||||
-> Model Sparse Attention (sol-attn)
|
||||
-> BFS native temporal node / Shot H3 Conditioning -> guider
|
||||
```
|
||||
|
||||
The official dense and sparse selections are detected, including owned H3 hooks when a later dense node hides the sparse override. Unrecognized replacements still raise an error. `tau`, `extra_tokens`, `start_percent`, `end_percent`, `dense_blocks` and `min_tokens` retain their sparse-node meanings. Outside eligible sparse calls, the bounded temporal rule still runs through the dense temporal backend. SLA and VSA are rejected.
|
||||
|
||||
ComfyUI's usual H3 sparse producer hooks would bypass BFS's temporal score modification. BFS therefore bypasses **only the hooks owned by that exact official sol-attn patch**, in a per-evaluation options copy, and performs attention through its temporal override. Other model clones and unrelated block hooks are not removed. This compatibility path does not retain the native sparse producer's streamed QKV memory optimization.
|
||||
|
||||
The public sol-attn kernel currently accepts equal-length Q/K/V and a **key-only** bias; it does not accept the query-dependent temporal bias in one pass. BFS runs one unbiased full-sequence pass for non-target queries, then one full-sequence pass per target latent frame with that frame's key bias, retaining only its target query rows. All conditioning keys are forced into sol-attn's exact branch, since its pooled branch does not honor `key_bias`. This exact-conditioning requirement takes precedence over `sink_conditioning = off`. Other video keys retain sol-attn's routing/pooled approximation. Bias storage is compact `latent_frames * packed_rows`, but sequence computation is repeated `latent_frames + 1` times when the temporal penalty is enabled. This is not equivalent to dense attention and can be substantially slower than Flash Partitions.
|
||||
|
||||
Measured diagnostics with PyTorch 2.12.0+cu130, comfy-kitchen 0.2.35, RTX PRO 6000 Blackwell, and concurrent Swap Anything training:
|
||||
|
||||
| Packed shape | Flash Partitions | sol-attn frame-wise compatibility |
|
||||
| --- | ---: | ---: |
|
||||
| `[1, 2, 42836, 128]` | 0.01742 s | 0.1604 s |
|
||||
| `[1, 56, 42836, 128]` | 0.4191 s | 2.6490 s |
|
||||
|
||||
Both tests used 22 latent frames, 7,062 guide rows, one warmup and one timed evaluation per backend. They measure isolated attention, not full video generation or an RTX 5090. **The initial full-head implementation was about 6.3 times slower; Flash Partitions remains the recommended default.** A fast one-pass implementation would require extending the sparse kernel with query-dependent temporal bias, not merely enabling the existing node.
|
||||
|
||||
CPU tests validate the temporal policy against the dense oracle, official-node recognition, owned-hook isolation and dense-window fallback. A CUDA random-input test at `[1, 2, 1248, 128]` produced finite output. Forcing all keys exact gave relative output norm error 0.01332 against FP32; the ordinary tau=1.3 sparse/pooled version gave 0.46068 on those random inputs. The latter is a substantial numerical difference, not evidence of visual quality or a universal model error. Real H3 video quality still requires evaluation. Diagnostics are recorded in [the initial sol-attn benchmark data](docs/benchmarks/h3_sol_temporal_initial.json).
|
||||
|
||||
Neither this feature nor its tests change the active training backend.
|
||||
@@ -142,3 +142,84 @@ Sidecar and phase have CPU geometry/rotary parity tests, including mixed image,
|
||||
video and audio, but no full pretrained H3 quality validation yet. They distinguish
|
||||
sources; they do not guarantee the same face. See the
|
||||
[training configuration guide](https://github.com/alisson-anjos/ai-toolkit/blob/minimax-h3-latent-guides/H3_REFERENCE_ROPE.md).
|
||||
|
||||
|
||||
## Video context in CLIP with aligned guide latents
|
||||
|
||||
Select **aligned guide + CLIP video context** in **BFS Shot H3 Conditioning**
|
||||
(or its Simple variant). Qwen receives only the CURRENT SHOT as `<Video 1>` at
|
||||
2 fps with real timestamps, using the same shot interval passed to Add Guide.
|
||||
It does not receive the full source video on every shot. The original shot frames
|
||||
provide visual context even when source blur modifies the aligned latent guide.
|
||||
Images retain their `<Picture 1>` through `<Picture N>` order and native image
|
||||
reference latents. Planner and the new standalone node allow up to 100 pictures
|
||||
(the Comfy autogrow capacity); available model context and memory can limit this.
|
||||
The original H3 node keeps its own unchanged input limits.
|
||||
|
||||
The shot itself enters the diffusion model only as aligned `minimax_keyframes`
|
||||
through Add Guide; the new mode does not create a native `minimax_refs` video block
|
||||
for it. Its Qwen embeddings still travel in `positive`. Explicitly enabling the
|
||||
separate native masked-region video reference adds that as `<Video 2>`.
|
||||
|
||||
For a native workflow, use **MiniMax H3 Reference to Video — Guide 0 CLIP Context
|
||||
(BFS)** in place of **MiniMax H3 Reference to Video**:
|
||||
|
||||
1. Connect your source guide to **guide_video_0**, and pictures to `ref_images`.
|
||||
`0` identifies the input's special role; its prompt token is `<Video 1>` because
|
||||
H3 numbers video tokens from 1. Only this guide is CLIP-only.
|
||||
2. Optional `ref_videos` remain ACTUAL native video references, receiving both Qwen
|
||||
encoding and native reference latents. With guide 0 connected they are
|
||||
`<Video 2>`, `<Video 3>`, `<Video 4>`; without it native numbering starts at 1.
|
||||
Up to 3 actual reference videos are available, as in the original node.
|
||||
3. Connect the returned `positive` and target `latent` to **MiniMax H3 Add Guide**,
|
||||
with the SAME guide frames, VAE and `frame_idx = 0`.
|
||||
4. Feed Add Guide's `positive` and the target `latent` to your sampler.
|
||||
|
||||
Video inputs use the native 24-fps convention, are trimmed to the requested target
|
||||
length, and round down to the native 17*k+5 frame grid. There is no separate 15s
|
||||
hard cap in this helper; practical context/memory and checkpoint training limits
|
||||
still apply. Native reference soundtracks remain paired to their `ref_video_N`
|
||||
input, not to guide 0. Supply guide audio through Add Guide.
|
||||
The helper resizes only the guide frames sampled for Qwen and never VAE-encodes
|
||||
that guide as a native video reference. Images, actual reference videos and audio
|
||||
retain the native implementation. Target latent starts empty; Add Guide supplies
|
||||
aligned motion. The BFS mode performs the context and aligned-guide steps
|
||||
internally; Shot Join only assembles/composites the generated outputs.
|
||||
|
||||
Example: `<Subject 1> comes from <Picture 3> and follows the blond performer in
|
||||
the yellow shirt in <Video 1>. <Subject 2> comes from <Picture 1> and <Picture 2>
|
||||
and follows the orange-haired performer in the white shirt in <Video 1>. Keep
|
||||
these assignments through crossings and occlusions.` A picture need not mean a
|
||||
separate person; multiple images can define one identity, and there can be N
|
||||
assigned subjects/reference assets within the available context.
|
||||
|
||||
Existing modes and defaults are unchanged. Qwen encoding cost and text context
|
||||
length increase. Checkpoints trained with `guide_latent_only` did not see the
|
||||
source video in Qwen: this is an experimental inference option, not a demonstrated
|
||||
identity-binding improvement. GPU and visual validation remain for the user's
|
||||
local workflow.
|
||||
|
||||
|
||||
### Guide sampling rate and console evidence
|
||||
|
||||
`clip_video_fps` is available on both BFS conditioning variants and the Guide 0
|
||||
node. Default is 2 fps; range is 0.5–24 fps, capped to the source frame rate
|
||||
without duplicating frames. It changes guide sampling in Qwen only, not the
|
||||
aligned guide, output fps, or native reference-video sampling. Try 4 or 8 fps
|
||||
against 2 fps with the same shot, prompt, references, seed and sampler settings.
|
||||
Native H3 timestamps are retained; higher sampling rates are an experiment, not
|
||||
a guarantee of stronger identity binding.
|
||||
|
||||
Console messages with `[BFS H3 CLIP context]` report the input/used frame count,
|
||||
requested/effective fps, sent RGB frames, dimensions and first/last timestamps.
|
||||
A second message confirms that the tokenizer actually emitted H3 video vision
|
||||
blocks; an incompatible CLIP fails explicitly rather than silently ignoring the
|
||||
video. The final message appears only after Qwen encoding returns, with elapsed
|
||||
time and conditioning shapes. Qwen can pad an odd sample count to pairs.
|
||||
These logs confirm the conditioning path ran, not visual quality.
|
||||
|
||||
Compare `aligned guide (Add Guide)` against `aligned guide + CLIP video context`
|
||||
with all other settings fixed to assess whether video context changes the result.
|
||||
Use a fresh sampler execution (Planner cached renders may otherwise return old
|
||||
results); the CLIP-context logs only appear when conditioning actually executes.
|
||||
Compare 2/4/8 fps separately to isolate the sampling-rate effect.
|
||||
|
||||
@@ -139,7 +139,10 @@ workflow once per shot, and join the results back with the original timing and s
|
||||
every node downstream runs once per shot.
|
||||
- **BFS Shot Unpack** — guide frames, references, prompt and length of one shot, for any model.
|
||||
- **BFS Shot H3 Conditioning** — MiniMax H3 conditioning for one shot with the native nodes
|
||||
(aligned guide and/or native reference video).
|
||||
(aligned guide, native reference video, or aligned guide + CLIP video context).
|
||||
- **MiniMax H3 Reference to Video — Guide 0 CLIP Context (BFS)** — lets Qwen see the video
|
||||
guide at 2 fps without native guide video-reference latents; use Add Guide separately for motion.
|
||||
See [H3 guide modes](MINIMAX_H3_GUIDES.md#video-context-in-clip-with-aligned-guide-latents).
|
||||
- **BFS Shot Join** — concatenates, trims to the true lengths, cross-fades soft joins, returns audio. With `comparison` on, its `guide` output is the original video frame for frame.
|
||||
- **BFS Video Compare** — one player for two videos (Join's `guide` and `images`): wipe ↔ / ↕, overlay or side by side, any of guide / target / difference on top, plus a per-pixel difference map and the difference per frame (click to jump).
|
||||
|
||||
@@ -192,3 +195,13 @@ See [setup, workflow template, compatibility and validation](MINIMAX_H3_GUIDES.m
|
||||
|
||||
BFSNodes **1.47.0** adds optional MiniMax-H3 overlap/sidecar source-phase RoPE and
|
||||
**MiniMax-H3 Identity Reference + RoPE (BFS)**. See [the H3 guide](MINIMAX_H3_GUIDES.md#experimental-reference-layouts-and-source-phase-1470).
|
||||
|
||||
Experimental native temporal motion-transfer inference (standalone node and Shot Conditioning mode): see [H3_NATIVE_TEMPORAL.md](H3_NATIVE_TEMPORAL.md). Requires the matching experimental LoRA; preserves native image references and applies temporal RoPE alignment plus soft temporal attention.
|
||||
|
||||
### H3 Flash Partitions
|
||||
|
||||
[Illustrated explanation: native temporal alignment with compile-free Flash Partitions](docs/H3_FLASH_PARTITIONS.md) covers the exact temporal rule, positive partition decomposition, log-sum-exp merging, measured performance, guide scaling and current backend limitations.
|
||||
|
||||
Experimental temporal mode automatically recognizes **Model Attention Backend** (Kitchen/PyTorch) and, from 1.115.14, **Model Sparse Attention → sol-attn**. Version 1.115.17 automatically prepares the verified extension on first use on supported Linux SM120 systems. Version 1.115.16+ can use the [one-pass Kitchen extension](H3_NATIVE_TEMPORAL.md#one-pass-native-kitchen-kernels-111516) for dense INT8 and sol-attn. The earlier compatibility path repeats frame-wise calls and was slower than Flash Partitions; see [setup, measurements and limitations](H3_NATIVE_TEMPORAL.md#model-sparse-attention-sol-attn-compatibility-111514). SLA/VSA remain unsupported.
|
||||
|
||||

|
||||
|
||||
@@ -92,6 +92,12 @@ try:
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] BFS Shot Noise node not loaded: {_e!r}")
|
||||
SNOISE_NODE_CLASS_MAPPINGS, SNOISE_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
try:
|
||||
from .minimax_h3_native_temporal import NODE_CLASS_MAPPINGS as H3TEMP_NODE_CLASS_MAPPINGS
|
||||
from .minimax_h3_native_temporal import NODE_DISPLAY_NAME_MAPPINGS as H3TEMP_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e:
|
||||
print(f"[BFSNodes] H3 native temporal nodes not loaded: {_e!r}")
|
||||
H3TEMP_NODE_CLASS_MAPPINGS, H3TEMP_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
try:
|
||||
from .minimax_h3_downscaled_guide import NODE_CLASS_MAPPINGS as H3GUIDE_NODE_CLASS_MAPPINGS
|
||||
from .minimax_h3_downscaled_guide import NODE_DISPLAY_NAME_MAPPINGS as H3GUIDE_NODE_DISPLAY_NAME_MAPPINGS
|
||||
@@ -119,8 +125,17 @@ except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] Video Compare node not loaded: {_e!r}")
|
||||
VCMP_NODE_CLASS_MAPPINGS, VCMP_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
|
||||
try:
|
||||
from .minimax_h3_clip_video import NODE_CLASS_MAPPINGS as H3CLIP_NODE_CLASS_MAPPINGS
|
||||
from .minimax_h3_clip_video import NODE_DISPLAY_NAME_MAPPINGS as H3CLIP_NODE_DISPLAY_NAME_MAPPINGS
|
||||
except Exception as _e: # noqa
|
||||
print(f"[BFSNodes] MiniMax H3 CLIP video context not loaded: {_e!r}")
|
||||
H3CLIP_NODE_CLASS_MAPPINGS, H3CLIP_NODE_DISPLAY_NAME_MAPPINGS = {}, {}
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
**H3CLIP_NODE_CLASS_MAPPINGS,
|
||||
**H3GUIDE_NODE_CLASS_MAPPINGS,
|
||||
**H3TEMP_NODE_CLASS_MAPPINGS,
|
||||
**SHOT_NODE_CLASS_MAPPINGS,
|
||||
**PANEL_NODE_CLASS_MAPPINGS,
|
||||
**GV_NODE_CLASS_MAPPINGS,
|
||||
@@ -149,7 +164,9 @@ NODE_CLASS_MAPPINGS = {
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
**H3CLIP_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**H3GUIDE_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**H3TEMP_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**SHOT_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**PANEL_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
**GV_NODE_DISPLAY_NAME_MAPPINGS,
|
||||
|
||||
@@ -713,7 +713,11 @@ if _io is not None:
|
||||
_io.Autogrow.Input("ref_audios", optional=True, template=_io.Autogrow.TemplatePrefix(
|
||||
input=_io.Audio.Input("ref_audio", tooltip="<Audio n> reference"), prefix="ref_audio_", min=0, max=3)),
|
||||
_io.Combo.Input("source_blur", options=_blur_modes(), default="off", optional=True, tooltip=
|
||||
"Blur the source person in what the model sees (the panel / the guide), keeping the mouth sharp (SAM 3.1): it reads the pose, motion and lips but has no face to copy. The output is never blurred. Works with or instead of panel_static."),
|
||||
BLUR_TT + " Works with or instead of panel_static."),
|
||||
_io.Combo.Input("source_blur_segmenter", options=_blur_lists().SEGMENTERS, default="SAM 3.1",
|
||||
optional=True, tooltip=SEGMENTER_TT),
|
||||
_io.Combo.Input("source_blur_effect", options=_blur_lists().EFFECTS, default="blur", optional=True,
|
||||
tooltip=EFFECT_TT),
|
||||
_io.Combo.Input("panel_static", options=PANEL_STATIC, default="off", optional=True, tooltip=
|
||||
"TSC's fix for swaps in the same world: cover the face (or the whole person) of the pinned clip in TV "
|
||||
"static, every frame (SAM 3), so the generated half cannot copy it and takes the identity from the "
|
||||
@@ -739,12 +743,13 @@ if _io is not None:
|
||||
panel_noise, hold, rope_mode, rope_gap, ref_image_size, steps, sampler_name, scheduler, seed,
|
||||
decode_canvas, audio_vae=None, panel=None, guide=None, guide_frame_idx=0, ref_images=None,
|
||||
ref_videos=None, ref_video_audios=None, ref_audios=None, panel_static="off", setting=None, vlm=None, source_blur="off",
|
||||
vlm_max_tokens=1024):
|
||||
vlm_max_tokens=1024, source_blur_segmenter="SAM 3.1", source_blur_effect="blur"):
|
||||
if panel is None and guide is None:
|
||||
raise ValueError("BFS H3 Duet needs a panel, a guide, or both")
|
||||
raw_panel = panel
|
||||
panel = static_over_panel(_blur(panel, source_blur, 1.0, "[BFS H3 Duet] panel"), panel_static)
|
||||
guide = _blur(guide, source_blur, 1.0, "[BFS H3 Duet] guide")
|
||||
panel = static_over_panel(_blur(panel, source_blur, 1.0, "[BFS H3 Duet] panel", source_blur_segmenter,
|
||||
source_blur_effect), panel_static)
|
||||
guide = _blur(guide, source_blur, 1.0, "[BFS H3 Duet] guide", source_blur_segmenter, source_blur_effect)
|
||||
if length <= 0:
|
||||
src = guide if guide is not None else panel
|
||||
length = src.shape[0] if src.shape[0] >= 5 else 124
|
||||
@@ -778,22 +783,32 @@ def _shot_loop():
|
||||
PANEL_STATIC = ["off", "face", "person"]
|
||||
|
||||
|
||||
def _blur_modes():
|
||||
def _blur_lists():
|
||||
try:
|
||||
from .bfs_source_blur import SOURCE_BLUR
|
||||
from . import bfs_source_blur as B
|
||||
except ImportError:
|
||||
from bfs_source_blur import SOURCE_BLUR
|
||||
return SOURCE_BLUR
|
||||
import bfs_source_blur as B
|
||||
return B
|
||||
|
||||
|
||||
def _blur(frames, mode, strength=1.0, log=""):
|
||||
def _blur_modes():
|
||||
return _blur_lists().SOURCE_BLUR
|
||||
|
||||
|
||||
BLUR_TT = ("Blur the source person in what the model sees (the panel / the guide): '<region> (<what stays sharp>)'. "
|
||||
"keep mouth: lips, teeth and tongue stay sharp; keep lips + tongue: the teeth are blurred too; mouth too: "
|
||||
"nothing stays sharp. masked area: the shot's own mask (elsewhere: the whole person). It reads the pose and "
|
||||
"motion but has no face to copy. The output is never blurred.")
|
||||
SEGMENTER_TT = ("What finds the face / head / hands / person and the mouth for source_blur. SAM 3.1: text prompts. "
|
||||
"Sapiens2: body-part classes in one pass, finer lips and teeth (downloads facebook/sapiens2-seg-0.4b, "
|
||||
"1.6 GB, to models/sapiens2 the first time). 'keep lips + tongue' always uses Sapiens2 for the lips.")
|
||||
EFFECT_TT = "blur: no detail left. invert colours: keeps the shapes and the expression but not the skin tone or colours."
|
||||
|
||||
|
||||
def _blur(frames, mode, strength=1.0, log="", segmenter="SAM 3.1", effect="blur", area=None):
|
||||
if frames is None or not mode or mode == "off":
|
||||
return frames
|
||||
try:
|
||||
from .bfs_source_blur import blur_source
|
||||
except ImportError:
|
||||
from bfs_source_blur import blur_source
|
||||
return blur_source(frames, mode, float(strength), log)
|
||||
return _blur_lists().blur_source(frames, mode, float(strength), log, segmenter, effect, area)
|
||||
|
||||
|
||||
def _small(frames: torch.Tensor, side: int = 640) -> torch.Tensor:
|
||||
@@ -909,7 +924,11 @@ if _io is not None:
|
||||
_io.Autogrow.Input("ref_images", optional=True, template=_io.Autogrow.TemplatePrefix(
|
||||
input=_io.Image.Input("ref_image", tooltip="<Picture n>, in order"), prefix="ref_image_", min=0, max=9)),
|
||||
_io.Combo.Input("source_blur", options=_blur_modes(), default="off", optional=True, tooltip=
|
||||
"Blur the source person in what the model sees (the panel / the guide), keeping the mouth sharp (SAM 3.1): it reads the pose, motion and lips but has no face to copy. The output is never blurred. Works with or instead of panel_static."),
|
||||
BLUR_TT + " Works with or instead of panel_static."),
|
||||
_io.Combo.Input("source_blur_segmenter", options=_blur_lists().SEGMENTERS, default="SAM 3.1",
|
||||
optional=True, tooltip=SEGMENTER_TT),
|
||||
_io.Combo.Input("source_blur_effect", options=_blur_lists().EFFECTS, default="blur", optional=True,
|
||||
tooltip=EFFECT_TT),
|
||||
_io.Combo.Input("panel_static", options=PANEL_STATIC, default="off", optional=True, tooltip=
|
||||
"TSC's fix for swaps in the same world: cover the face (or the whole person) of the pinned clip in TV "
|
||||
"static, every frame (SAM 3), so the generated half cannot copy it and takes the identity from the "
|
||||
@@ -927,12 +946,14 @@ if _io is not None:
|
||||
def execute(cls, clip, vae, panel, task, instruction, prompt, width, height, length, position, size, panel_noise,
|
||||
rope_mode, fit, gap, hold, rope_gap, ref_image_size, vlm_max_tokens, vlm=None, model=None,
|
||||
audio_vae=None, guide=None, ref_images=None, keep_mask=None, keep_video=None, panel_static="off",
|
||||
setting=None, source_blur="off"):
|
||||
setting=None, source_blur="off", source_blur_segmenter="SAM 3.1", source_blur_effect="blur"):
|
||||
from comfy_extras.nodes_minimax_h3 import MiniMaxH3ReferenceToVideo
|
||||
refs = [v for v in (ref_images or {}).values() if v is not None]
|
||||
raw_panel = panel
|
||||
panel = static_over_panel(_blur(panel, source_blur, 1.0, "[BFS H3 Duet Conditioning] panel"), panel_static)
|
||||
guide = _blur(guide, source_blur, 1.0, "[BFS H3 Duet Conditioning] guide")
|
||||
panel = static_over_panel(_blur(panel, source_blur, 1.0, "[BFS H3 Duet Conditioning] panel",
|
||||
source_blur_segmenter, source_blur_effect), panel_static)
|
||||
guide = _blur(guide, source_blur, 1.0, "[BFS H3 Duet Conditioning] guide", source_blur_segmenter,
|
||||
source_blur_effect)
|
||||
if length <= 0:
|
||||
length = panel.shape[0] if panel.shape[0] >= 5 else 124
|
||||
text = _shot_loop().duet_prompt(vlm, prompt, raw_panel, refs, task, instruction, int(vlm_max_tokens),
|
||||
@@ -996,8 +1017,10 @@ class BFSShotH3Duet:
|
||||
"optional": {"audio_vae": ("VAE",),
|
||||
"rope_mode": (ROPE_MODES, {"default": "canvas", "advanced": True}),
|
||||
"rope_gap": ("FLOAT", {"default": 0.0, "min": 0.0, "max": 256.0, "step": 1.0, "advanced": True}),
|
||||
"source_blur": (_blur_modes(), {"default": "off", "tooltip": "Blur the source person in what the model sees (the panel / the guide), keeping the mouth sharp (SAM 3.1): it reads the pose, motion and lips but has no face to copy. The output is never blurred."}),
|
||||
"source_blur": (_blur_modes(), {"default": "off", "tooltip": BLUR_TT}),
|
||||
"source_blur_strength": ("FLOAT", {"default": 1.0, "min": 0.1, "max": 1.0, "step": 0.05}),
|
||||
"source_blur_segmenter": (_blur_lists().SEGMENTERS, {"default": "SAM 3.1", "tooltip": SEGMENTER_TT}),
|
||||
"source_blur_effect": (_blur_lists().EFFECTS, {"default": "blur", "tooltip": EFFECT_TT}),
|
||||
"vlm": ("CLIP", {"tooltip": "Optional VLM (Qwen3-VL CLIPLoader): fills {details} in the shot's "
|
||||
"prompt from its references when the planner left it."})},
|
||||
}
|
||||
@@ -1013,12 +1036,10 @@ class BFSShotH3Duet:
|
||||
|
||||
def render(self, shot, model, clip, vae, mode, task, instruction, use_ref_2, position, size, fit, gap,
|
||||
panel_noise, ref_image_size, steps, sampler_name, scheduler, seed, decode_canvas, audio_vae=None,
|
||||
rope_mode="canvas", rope_gap=0.0, source_blur="off", source_blur_strength=1.0, vlm=None):
|
||||
imgs = {}
|
||||
if shot.get("ref") is not None:
|
||||
imgs["ref_image_1"] = shot["ref"]
|
||||
if use_ref_2 and shot.get("ref2") is not None:
|
||||
imgs[f"ref_image_{len(imgs) + 1}"] = shot["ref2"]
|
||||
rope_mode="canvas", rope_gap=0.0, source_blur="off", source_blur_strength=1.0, vlm=None,
|
||||
source_blur_segmenter="SAM 3.1", source_blur_effect="blur"):
|
||||
own = _shot_loop().shot_refs(shot)
|
||||
imgs = {f"ref_image_{k + 1}": r for k, r in enumerate(own[:_shot_loop().MAX_REFS if use_ref_2 else 1])}
|
||||
duet, guided = mode != self.MODES[1], mode != self.MODES[0]
|
||||
try:
|
||||
from .bfs_shot_loop import chain_image, remember_result, shot_history
|
||||
@@ -1028,12 +1049,15 @@ class BFSShotH3Duet:
|
||||
if prev is not None and shot.get("chain") == "reference":
|
||||
imgs[f"ref_image_{len(imgs) + 1}"] = prev
|
||||
text = _resolve_prompt(shot.get("prompt", ""), task, instruction, len(imgs), 0)
|
||||
if "{details}" in text:
|
||||
text = _shot_loop().fill_details_now(text, vlm, [shot.get("ref"), shot.get("ref2")],
|
||||
f"[BFS Shot H3 Duet] shot {int(shot.get('index', 0)) + 1}")
|
||||
if "{details" in text:
|
||||
text = _shot_loop().fill_details_now(text, vlm, own, f"[BFS Shot H3 Duet] shot {int(shot.get('index', 0)) + 1}",
|
||||
shot.get("ref_names"))
|
||||
hist, hsrc = shot_history(shot) if shot_history else (None, None)
|
||||
area = None
|
||||
if source_blur and source_blur.startswith("masked area") and (shot.get("mask_src") or (shot.get("crop") or {}).get("crop_mask") is not None):
|
||||
area = _shot_loop().generation_mask(shot)
|
||||
seen = _blur(shot["frames"], source_blur, source_blur_strength,
|
||||
f"[BFS Shot H3 Duet] shot {int(shot.get('index', 0)) + 1}")
|
||||
f"[BFS Shot H3 Duet] shot {int(shot.get('index', 0)) + 1}", source_blur_segmenter, source_blur_effect, area)
|
||||
out = h3_render(model, clip, vae, audio_vae, text, {"ref_images": imgs}, shot["width"], shot["height"],
|
||||
shot["gen_length"], steps, sampler_name, scheduler, seed + int(shot.get("index", 0)),
|
||||
panel=seen if duet else None, guide=seen if guided else None,
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
"""Sapiens2 body-part segmentation (Meta, facebook/sapiens2-seg-*): one class per pixel out of 29 (28 parts +
|
||||
background), lips, teeth and tongue separate from the face. Used by the source blur for parts SAM 3.1 can't
|
||||
find reliably from a text prompt (lips, tongue). It marks every person in the frame (no instances).
|
||||
|
||||
The weights (0.4B, 1.6 GB) go to models/sapiens2/sapiens2-seg-0.4b, downloaded from Hugging Face the first time.
|
||||
Sapiens2 License: https://github.com/facebookresearch/sapiens2/blob/main/LICENSE.md
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
import torch
|
||||
|
||||
REPO = "facebook/sapiens2-seg-0.4b"
|
||||
FILES = ("config.json", "preprocessor_config.json", "model.safetensors")
|
||||
CLASSES = ("Background", "Apparel", "Eyeglass", "Face_Neck", "Hair", "Left_Foot", "Left_Hand", "Left_Lower_Arm",
|
||||
"Left_Lower_Leg", "Left_Shoe", "Left_Sock", "Left_Upper_Arm", "Left_Upper_Leg", "Lower_Clothing",
|
||||
"Right_Foot", "Right_Hand", "Right_Lower_Arm", "Right_Lower_Leg", "Right_Shoe", "Right_Sock",
|
||||
"Right_Upper_Arm", "Right_Upper_Leg", "Torso", "Upper_Clothing", "Lower_Lip", "Upper_Lip", "Lower_Teeth",
|
||||
"Upper_Teeth", "Tongue")
|
||||
ID = {n: i for i, n in enumerate(CLASSES)}
|
||||
PIXELS = 1024 * 768 # the size it was trained at; other shapes keep this area and their aspect ratio
|
||||
_MEAN = (0.485, 0.456, 0.406)
|
||||
_STD = (0.229, 0.224, 0.225)
|
||||
_MODEL: dict = {}
|
||||
|
||||
|
||||
def model_dir() -> str:
|
||||
try:
|
||||
import folder_paths
|
||||
base = folder_paths.models_dir
|
||||
except Exception:
|
||||
base = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "..", "models")
|
||||
return os.path.join(base, "sapiens2", REPO.split("/")[-1])
|
||||
|
||||
|
||||
def _load():
|
||||
if "m" in _MODEL:
|
||||
return _MODEL["m"]
|
||||
d = model_dir()
|
||||
if not all(os.path.exists(os.path.join(d, f)) for f in FILES):
|
||||
print(f"[BFS Sapiens2] downloading {REPO} (1.6 GB) to {d}", flush=True)
|
||||
from huggingface_hub import hf_hub_download
|
||||
for f in FILES:
|
||||
hf_hub_download(REPO, f, local_dir=d)
|
||||
from transformers import Sapiens2ForSemanticSegmentation
|
||||
_MODEL["m"] = Sapiens2ForSemanticSegmentation.from_pretrained(d, dtype=torch.float32).eval()
|
||||
return _MODEL["m"]
|
||||
|
||||
|
||||
def _device():
|
||||
try:
|
||||
import comfy.model_management as mm
|
||||
return mm.get_torch_device()
|
||||
except Exception:
|
||||
return torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
||||
|
||||
|
||||
def input_size(h: int, w: int) -> tuple[int, int]:
|
||||
"""(H, W) with the trained area, the frame's aspect ratio, multiples of the 16 px patch."""
|
||||
s = (PIXELS / (h * w)) ** 0.5
|
||||
return max(16, round(h * s / 16) * 16), max(16, round(w * s / 16) * 16)
|
||||
|
||||
|
||||
@torch.no_grad()
|
||||
def segment(frames: torch.Tensor, out_size: tuple[int, int] | None = None, batch: int = 4) -> torch.Tensor:
|
||||
"""frames [N,H,W,3] 0-1 -> class ids [N,h,w] uint8 at `out_size` (default the frames' size)."""
|
||||
m = _load()
|
||||
dev = _device()
|
||||
half = dev.type == "cuda"
|
||||
m.to(dev, dtype=torch.float16 if half else torch.float32)
|
||||
N, H, W = frames.shape[:3]
|
||||
oh, ow = out_size or (H, W)
|
||||
ih, iw = input_size(H, W)
|
||||
mean = torch.tensor(_MEAN, device=dev).view(1, 3, 1, 1)
|
||||
std = torch.tensor(_STD, device=dev).view(1, 3, 1, 1)
|
||||
out = torch.empty(N, oh, ow, dtype=torch.uint8)
|
||||
try:
|
||||
for a in range(0, N, batch):
|
||||
x = frames[a:a + batch, ..., :3].to(dev).float().movedim(-1, 1)
|
||||
x = torch.nn.functional.interpolate(x, size=(ih, iw), mode="bilinear", align_corners=False, antialias=True)
|
||||
x = ((x - mean) / std).to(torch.float16 if half else torch.float32)
|
||||
lg = m(pixel_values=x).logits.float()
|
||||
lg = torch.nn.functional.interpolate(lg, size=(oh, ow), mode="bilinear", align_corners=False)
|
||||
out[a:a + batch] = lg.argmax(1).to(torch.uint8).cpu()
|
||||
del x, lg
|
||||
finally:
|
||||
m.to("cpu") # 0.4B: back to RAM, the VRAM is the video model's
|
||||
if half:
|
||||
torch.cuda.empty_cache()
|
||||
return out
|
||||
|
||||
|
||||
def parts(classes: torch.Tensor, names) -> torch.Tensor:
|
||||
"""[N,h,w] float mask of the pixels whose class is one of `names`."""
|
||||
ids = torch.tensor([ID[n] for n in names], dtype=classes.dtype)
|
||||
return torch.isin(classes, ids).float()
|
||||
@@ -1,17 +1,41 @@
|
||||
"""Optional blur of the source person in what the model SEES of the source (aligned guide, native reference video,
|
||||
duet panel), keeping the mouth sharp: the model still reads the pose, the motion and the lips, but has no face to copy,
|
||||
so the identity comes from the reference. JalenBrunson's trick for reliable ref2v swaps (Sapiens2 there); here the
|
||||
masks come from SAM 3.1, already used by the planner. The kept area (inpaint) and the output are never blurred.
|
||||
duet panel): the model still reads the pose, the motion and (if kept sharp) the lips, but has no face to copy, so the
|
||||
identity comes from the reference. JalenBrunson's trick for reliable ref2v swaps. The kept area (inpaint) and the
|
||||
output are never blurred.
|
||||
|
||||
Modes are "<region> (<what stays sharp>)":
|
||||
region face / head + hands / person: found by the segmenter, every person in the frame;
|
||||
masked area: the shot's own mask (the person being replaced, with its expand), no extra segmentation
|
||||
sharp keep mouth: lips, teeth and tongue; keep lips + tongue: the teeth are blurred too (always Sapiens2);
|
||||
mouth too: nothing stays sharp
|
||||
Segmenter: SAM 3.1 (text prompts, already used by the planner) or Sapiens2 (body-part classes, one pass, finer lips).
|
||||
Effect: blur, or invert the colours (keeps the shapes and the expression, loses skin tone and colours).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import torch
|
||||
|
||||
SOURCE_BLUR = ["off", "face (keep mouth)", "head + hands (keep mouth)", "person (keep mouth)"]
|
||||
_PARTS = {SOURCE_BLUR[1]: ("face",), SOURCE_BLUR[2]: ("head", "hand"), SOURCE_BLUR[3]: ("person",)}
|
||||
_REGIONS = ("face", "head + hands", "person", "masked area")
|
||||
_SHARP = ("keep mouth", "keep lips + tongue", "mouth too")
|
||||
SOURCE_BLUR = ["off"] + [f"{r} ({k})" for k in _SHARP for r in _REGIONS]
|
||||
SEGMENTERS = ["SAM 3.1", "Sapiens2"]
|
||||
EFFECTS = ["blur", "invert colours"]
|
||||
_SAM_PARTS = {"face": ("face",), "head + hands": ("head", "hand"), "person": ("person",)}
|
||||
_SAP_FACE = ("Face_Neck", "Eyeglass", "Lower_Lip", "Upper_Lip", "Lower_Teeth", "Upper_Teeth", "Tongue")
|
||||
_SAP_PARTS = {"face": _SAP_FACE, "head + hands": _SAP_FACE + ("Hair", "Left_Hand", "Right_Hand")}
|
||||
_SAP_SHARP = {"keep mouth": ("Lower_Lip", "Upper_Lip", "Lower_Teeth", "Upper_Teeth", "Tongue"),
|
||||
"keep lips + tongue": ("Lower_Lip", "Upper_Lip", "Tongue")}
|
||||
_CACHE: dict = {}
|
||||
|
||||
|
||||
def parse(mode: str) -> tuple[str, str] | None:
|
||||
"""'face (keep mouth)' -> ('face', 'keep mouth'); None for off / unknown."""
|
||||
if not mode or "(" not in mode:
|
||||
return None
|
||||
region, sharp = mode.rstrip(")").split(" (", 1)
|
||||
return (region, sharp) if region in _REGIONS and sharp in _SHARP else None
|
||||
|
||||
|
||||
def _sl():
|
||||
try:
|
||||
from . import bfs_shot_loop as SL
|
||||
@@ -20,6 +44,13 @@ def _sl():
|
||||
return SL
|
||||
|
||||
|
||||
def _sap():
|
||||
try:
|
||||
from . import bfs_sapiens as SP
|
||||
except ImportError:
|
||||
import bfs_sapiens as SP
|
||||
return SP
|
||||
|
||||
def _small(frames: torch.Tensor, side: int = 640) -> torch.Tensor:
|
||||
H, W = frames.shape[1:3]
|
||||
s = side / max(H, W)
|
||||
@@ -36,51 +67,93 @@ def _grow(m: torch.Tensor, px: int) -> torch.Tensor:
|
||||
return torch.nn.functional.max_pool2d(m[:, None], 2 * px + 1, stride=1, padding=px)[:, 0]
|
||||
|
||||
|
||||
def blur_mask(frames: torch.Tensor, mode: str, log: str = "") -> torch.Tensor | None:
|
||||
"""[N,h,w] float mask (working size) of what to blur: the parts for `mode` minus the (slightly grown) mouth."""
|
||||
parts = _PARTS.get(mode)
|
||||
if not parts:
|
||||
def blur_mask(frames: torch.Tensor, mode: str, log: str = "", segmenter: str = "SAM 3.1",
|
||||
area: torch.Tensor | None = None) -> torch.Tensor | None:
|
||||
"""[N,h,w] float mask (working size) of what to blur: the region for `mode` minus what stays sharp (grown a
|
||||
little). `area` [N,H,W]: the shot's own mask, for the 'masked area' region."""
|
||||
pr = parse(mode)
|
||||
if not pr:
|
||||
return None
|
||||
region, sharp = pr
|
||||
if region == "masked area" and area is None:
|
||||
if log:
|
||||
print(f"{log}: source blur 'masked area' needs a mask on the shot (SAM 3 text or points): blurring the "
|
||||
"whole person instead", flush=True)
|
||||
region = "person"
|
||||
SL = _sl()
|
||||
small = _small(frames)
|
||||
key = (mode, tuple(small.shape), SL._image_key(frames[:1]), SL._image_key(frames[-1:]),
|
||||
SL._image_key(frames[frames.shape[0] // 2:frames.shape[0] // 2 + 1]))
|
||||
h, w = small.shape[1:3]
|
||||
sap_region = region != "masked area" and segmenter == "Sapiens2"
|
||||
sap_sharp = sharp == "keep lips + tongue" or (sharp == "keep mouth" and segmenter == "Sapiens2")
|
||||
key = (region, sharp, segmenter, tuple(small.shape), SL._image_key(frames[:1]), SL._image_key(frames[-1:]),
|
||||
SL._image_key(frames[frames.shape[0] // 2:frames.shape[0] // 2 + 1]),
|
||||
None if area is None else (tuple(area.shape), float(area.float().sum())))
|
||||
if key in _CACHE:
|
||||
return _CACHE[key]
|
||||
m = torch.zeros(small.shape[:3])
|
||||
for p in parts:
|
||||
m = torch.maximum(m, (SL.segment_frames(small, dict(SL.DEFAULT_MASK, text=p, max_objects=8, threshold=0.4)) > 0.5).float())
|
||||
mouth = (SL.segment_frames(small, dict(SL.DEFAULT_MASK, text="mouth", max_objects=8, threshold=0.3)) > 0.5).float()
|
||||
keep = _grow(mouth, max(1, int(min(small.shape[1:3]) * 0.012)))
|
||||
out = (_grow(m, max(1, int(min(small.shape[1:3]) * 0.015))) * (1 - keep)).clamp(0, 1)
|
||||
cls = _sap().segment(small) if sap_region or sap_sharp else None
|
||||
if region == "masked area":
|
||||
m = torch.nn.functional.interpolate(area[:, None].float(), size=(h, w), mode="nearest")[:, 0].clamp(0, 1)
|
||||
m_grow = 0 # already grown by the mask's own expand
|
||||
elif sap_region:
|
||||
m = (cls > 0).float() if region == "person" else _sap().parts(cls, _SAP_PARTS[region])
|
||||
m_grow = max(1, int(min(h, w) * 0.01))
|
||||
else:
|
||||
m = torch.zeros(small.shape[:3])
|
||||
for p in _SAM_PARTS[region]:
|
||||
m = torch.maximum(m, (SL.segment_frames(small, dict(SL.DEFAULT_MASK, text=p, max_objects=8, threshold=0.4)) > 0.5).float())
|
||||
m_grow = max(1, int(min(h, w) * 0.015))
|
||||
if sharp == "mouth too":
|
||||
keep, found = torch.zeros_like(m), None
|
||||
elif sap_sharp:
|
||||
keep = _sap().parts(cls, _SAP_SHARP[sharp])
|
||||
keep = _grow(keep, max(1, int(min(h, w) * 0.005)))
|
||||
found = float(keep.sum()) > 0
|
||||
else:
|
||||
mouth = (SL.segment_frames(small, dict(SL.DEFAULT_MASK, text="mouth", max_objects=8, threshold=0.3)) > 0.5).float()
|
||||
keep = _grow(mouth, max(1, int(min(h, w) * 0.012)))
|
||||
found = float(mouth.sum()) > 0
|
||||
out = (_grow(m, m_grow) * (1 - keep)).clamp(0, 1)
|
||||
if log:
|
||||
print(f"{log}: source blur '{mode}': {float(m.mean()) * 100:.1f}% of the frame, mouth kept "
|
||||
f"({'found' if float(mouth.sum()) > 0 else 'not found: the whole part is blurred'})", flush=True)
|
||||
kept = "nothing kept sharp" if found is None else \
|
||||
f"{sharp[5:]} kept ({'found' if found else 'not found: the whole region is blurred'})"
|
||||
used = "shot mask" if region == "masked area" else "Sapiens2" if sap_region else "SAM 3.1"
|
||||
print(f"{log}: source blur '{region}' ({used}{' + Sapiens2 lips' if sap_sharp and not sap_region else ''}): "
|
||||
f"{float(m.mean()) * 100:.1f}% of the frame, {kept}", flush=True)
|
||||
if len(_CACHE) > 8:
|
||||
_CACHE.pop(next(iter(_CACHE)))
|
||||
_CACHE[key] = out
|
||||
return out
|
||||
|
||||
|
||||
def blur_source(frames: torch.Tensor | None, mode: str, strength: float = 1.0, log: str = "") -> torch.Tensor | None:
|
||||
"""The frames with the source person blurred per `mode` (mouth kept). `strength` 0-1 sets how coarse the blur is."""
|
||||
def _soft(m: torch.Tensor, H: int, W: int, N: int) -> torch.Tensor:
|
||||
"""The working-size mask at the frame size with a soft edge, one per frame ([N,H,W,1])."""
|
||||
mm = torch.nn.functional.interpolate(m[:, None], size=(H, W), mode="bilinear", align_corners=False)
|
||||
mm = torch.nn.functional.avg_pool2d(mm, 9, stride=1, padding=4)[:, 0, ..., None].clamp(0, 1)
|
||||
if mm.shape[0] != N: # one mask per frame; a shorter list holds its last one
|
||||
mm = mm[[min(i, mm.shape[0] - 1) for i in range(N)]]
|
||||
return mm
|
||||
|
||||
|
||||
def blur_source(frames: torch.Tensor | None, mode: str, strength: float = 1.0, log: str = "",
|
||||
segmenter: str = "SAM 3.1", effect: str = "blur", area: torch.Tensor | None = None) -> torch.Tensor | None:
|
||||
"""The frames with the source person blurred (or colour-inverted) per `mode`. `strength` 0-1 sets how coarse the
|
||||
blur is; `area`: the shot's mask for the 'masked area' modes."""
|
||||
if frames is None or not mode or mode == "off" or strength <= 0:
|
||||
return frames
|
||||
m = blur_mask(frames, mode, log)
|
||||
m = blur_mask(frames, mode, log, segmenter, area)
|
||||
if m is None or float(m.sum()) == 0:
|
||||
return frames
|
||||
x = frames[..., :3].float()
|
||||
N, H, W = x.shape[:3]
|
||||
if effect == "invert colours":
|
||||
mm = _soft(m, H, W, N)
|
||||
return x * (1 - mm) + (1 - x) * mm
|
||||
# a heavy blur: down to ~1/12-1/40 of the size and back, then a soft pass; blocky detail never survives it
|
||||
f = max(4, int(min(H, W) * (0.025 + 0.06 * float(strength))))
|
||||
low = torch.nn.functional.interpolate(x.movedim(-1, 1), size=(max(2, H // f), max(2, W // f)), mode="area")
|
||||
blur = torch.nn.functional.interpolate(low, size=(H, W), mode="bilinear", align_corners=False)
|
||||
blur = torch.nn.functional.avg_pool2d(blur, 5, stride=1, padding=2).movedim(1, -1)
|
||||
mm = torch.nn.functional.interpolate(m[:, None], size=(H, W), mode="bilinear", align_corners=False)
|
||||
mm = torch.nn.functional.avg_pool2d(mm, 9, stride=1, padding=4)[:, 0, ..., None].clamp(0, 1)
|
||||
if mm.shape[0] != N: # one mask per frame; a shorter list holds its last one
|
||||
idx = [min(i, mm.shape[0] - 1) for i in range(N)]
|
||||
mm = mm[idx]
|
||||
mm = _soft(m, H, W, N)
|
||||
out = x * (1 - mm) + blur * mm
|
||||
del low, blur, mm
|
||||
return out
|
||||
|
||||
@@ -7,8 +7,11 @@ gets three temporary MP4s (guide, target, difference) and the per-frame differen
|
||||
"""
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import uuid
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
|
||||
@@ -51,6 +54,54 @@ def psnr(mse: float) -> float:
|
||||
return float("inf") if mse <= 0 else 10 * math.log10(1.0 / mse)
|
||||
|
||||
|
||||
MAX_REFS = 12
|
||||
|
||||
|
||||
def ref_pictures(reference) -> list:
|
||||
"""The reference pictures, one [H,W,3] each: from a list of IMAGEs of any sizes (e.g. the planner's or BFS Shot
|
||||
Unpack's refs) and / or IMAGE batches."""
|
||||
out = []
|
||||
for item in reference if isinstance(reference, list) else [reference]:
|
||||
if item is not None:
|
||||
out += [item[i] for i in range(int(item.shape[0]))]
|
||||
return out[:MAX_REFS]
|
||||
|
||||
|
||||
def save_refs(reference, out_dir: str, height: int) -> list:
|
||||
"""The reference pictures as PNGs no taller than the player (no upscale) -> file names."""
|
||||
from PIL import Image
|
||||
names = []
|
||||
for i, pic in enumerate(ref_pictures(reference)):
|
||||
im = Image.fromarray((pic[..., :3].float().clamp(0, 1).cpu().numpy() * 255).round().astype(np.uint8))
|
||||
if im.height > height:
|
||||
im = im.resize((max(1, round(im.width * height / im.height)), height), Image.LANCZOS)
|
||||
names.append(f"ref_{i}.png")
|
||||
im.save(os.path.join(out_dir, names[-1]))
|
||||
return names
|
||||
|
||||
|
||||
def video_batches(value, name):
|
||||
parts = value if isinstance(value, list) else [value]
|
||||
if not parts or any(not isinstance(x, torch.Tensor) or x.ndim != 4 or
|
||||
x.shape[0] < 1 or x.shape[-1] < 3 or min(x.shape[1:3]) < 1 for x in parts):
|
||||
raise ValueError(f"{name} must contain nonempty IMAGE batches")
|
||||
return parts
|
||||
|
||||
|
||||
def video_chunk(parts, start, end, h, w):
|
||||
"""Read a bounded frame interval across shot batches without concatenating full videos."""
|
||||
result, pos = [], 0
|
||||
for part in parts:
|
||||
count = int(part.shape[0])
|
||||
a, b = max(0, start - pos), min(count, end - pos)
|
||||
if a < b:
|
||||
result.append(_scale(part[a:b, ..., :3].float(), h, w))
|
||||
pos += count
|
||||
if pos >= end:
|
||||
break
|
||||
return result[0] if len(result) == 1 else torch.cat(result, 0)
|
||||
|
||||
|
||||
class BFSVideoCompare:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
@@ -72,11 +123,17 @@ class BFSVideoCompare:
|
||||
"preview_height": ("INT", {"default": 720, "min": 0, "max": 4096, "step": 16, "tooltip":
|
||||
"Height of the player's videos (0 = the compared size). Only the preview: "
|
||||
"the difference is measured before it."}),
|
||||
"reference": ("IMAGE", {"tooltip": "Optional reference pictures (e.g. the people swapped in), shown in a "
|
||||
"strip below the videos, numbered as <Picture n>, and in what ⬇ save / ● rec "
|
||||
"download. Takes the refs list of BFS Shot Planner / Unpack (pictures of any "
|
||||
"size) or a batch; up to 12."}),
|
||||
"label_guide": ("STRING", {"default": "guide"}),
|
||||
"label_target": ("STRING", {"default": "target"}),
|
||||
},
|
||||
}
|
||||
|
||||
# IMAGE lists contain every shot, in order. Only scalar settings take their first item.
|
||||
INPUT_IS_LIST = True
|
||||
RETURN_TYPES = ("IMAGE", "STRING", "FLOAT")
|
||||
RETURN_NAMES = ("difference", "report", "mean_difference")
|
||||
OUTPUT_TOOLTIPS = ("The difference map, one frame per compared frame (black = equal, white = very different).",
|
||||
@@ -89,17 +146,30 @@ class BFSVideoCompare:
|
||||
"where they differ, pixel by pixel.")
|
||||
|
||||
def compare(self, guide, target, fps=24.0, audio=None, gain=2.0, size="guide", preview_height=720,
|
||||
label_guide="guide", label_target="target"):
|
||||
label_guide="guide", label_target="target", reference=None):
|
||||
one = lambda v: v[0] if isinstance(v, list) and v else v
|
||||
fps, audio, gain, size, preview_height, label_guide, label_target = map(
|
||||
one, (fps, audio, gain, size, preview_height, label_guide, label_target))
|
||||
guide, target = video_batches(guide, "guide"), video_batches(target, "target")
|
||||
ng, nt = sum(x.shape[0] for x in guide), sum(x.shape[0] for x in target)
|
||||
if len(guide) > 1 or len(target) > 1:
|
||||
if ng != nt or (len(guide) == len(target) and any(g.shape[0] != t.shape[0] for g, t in zip(guide, target))):
|
||||
raise ValueError("Guide/target shot lists have different timing. Connect BFS Shot Join's guide and images outputs from the SAME Join; Unpack guide frames can include continuity heads and generation padding.")
|
||||
notes = []
|
||||
n = min(guide.shape[0], target.shape[0])
|
||||
if guide.shape[0] != target.shape[0]:
|
||||
notes.append(f"lengths differ (guide {guide.shape[0]}, target {target.shape[0]} frames): "
|
||||
n = min(ng, nt)
|
||||
if ng != nt:
|
||||
notes.append(f"lengths differ (guide {ng}, target {nt} frames): "
|
||||
f"compared the first {n}")
|
||||
ref = guide if size == "guide" else target
|
||||
H, W = int(ref.shape[1]), int(ref.shape[2])
|
||||
for name, v in (("guide", guide), ("target", target)):
|
||||
if tuple(v.shape[1:3]) != (H, W):
|
||||
notes.append(f"{name} scaled from {v.shape[2]}x{v.shape[1]} to {W}x{H}")
|
||||
H, W = int(ref[0].shape[1]), int(ref[0].shape[2])
|
||||
for name, parts in (("guide", guide), ("target", target)):
|
||||
if len(parts) > 1:
|
||||
notes.append(f"{name}: compared all {len(parts)} shot batches in input order")
|
||||
for dims in sorted(set(tuple(v.shape[1:3]) for v in parts)):
|
||||
if dims != (H, W):
|
||||
notes.append(f"{name} scaled from {dims[1]}x{dims[0]} to {W}x{H}")
|
||||
if ng != nt:
|
||||
notes.append("Timing may be misaligned. For Planner results, connect guide and images from the SAME BFS Shot Join.")
|
||||
ph = min(H, int(preview_height)) if preview_height and preview_height > 0 else H
|
||||
PH, PW = max(16, ph // 2 * 2), max(16, int(round(W * ph / H)) // 2 * 2)
|
||||
|
||||
@@ -112,8 +182,8 @@ class BFSVideoCompare:
|
||||
try:
|
||||
for a in range(0, n, CHUNK):
|
||||
e = min(a + CHUNK, n) # the longer video stops where the shorter one ends
|
||||
g = _scale(guide[a:e, ..., :3].float(), H, W)
|
||||
t = _scale(target[a:e, ..., :3].float(), H, W)
|
||||
g = video_chunk(guide, a, e, H, W)
|
||||
t = video_chunk(target, a, e, H, W)
|
||||
m, e, s = difference(g, t, gain)
|
||||
maps.append(m)
|
||||
mae.append(e)
|
||||
@@ -136,11 +206,167 @@ class BFSVideoCompare:
|
||||
print(f"[BFS Video Compare] {report}", flush=True)
|
||||
sub = os.path.relpath(out_dir, folder_paths.get_temp_directory())
|
||||
files = {k: {"filename": f, "subfolder": sub, "type": "temp"} for k, f in names.items()}
|
||||
refs = [{"filename": f, "subfolder": sub, "type": "temp"} for f in save_refs(reference, out_dir, PH)]
|
||||
ui = {"bfs_compare": [{**files, "fps": float(fps), "frames": n, "width": PW, "height": PH,
|
||||
"labels": [label_guide, label_target], "mae": [round(float(x), 2) for x in mae],
|
||||
"report": report}]}
|
||||
"refs": refs,
|
||||
"report": report, "notes": notes}]}
|
||||
return {"ui": ui, "result": (diff_map, report, mean)}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------- save / record: the browser's file -> a shareable one
|
||||
|
||||
EXPORT_FORMATS = { # what the player's format list offers; "webm" never reaches the server (the browser's own file)
|
||||
"mp4": "MP4 (H.264)",
|
||||
"small": "MP4 small (under 10 MB)",
|
||||
"gif": "GIF (under 10 MB, no sound)",
|
||||
}
|
||||
LIMIT_BYTES = int(9.5 * 1024 * 1024) # under the 10 MB many chat apps accept for free
|
||||
_ID = re.compile(r"^[0-9a-f]{8,32}$")
|
||||
|
||||
|
||||
def _export_dir() -> str:
|
||||
d = os.path.join(folder_paths.get_temp_directory(), "bfs_compare", "exports")
|
||||
os.makedirs(d, exist_ok=True)
|
||||
return d
|
||||
|
||||
|
||||
def write_part(rid: str, offset: int, data: bytes) -> int:
|
||||
"""One piece of a recording uploaded by the player (pieces keep big recordings under the upload limit)."""
|
||||
if not _ID.match(rid or "") or offset < 0:
|
||||
raise ValueError("bad recording id / offset")
|
||||
path = os.path.join(_export_dir(), f"{rid}.rec")
|
||||
with open(path, "r+b" if os.path.exists(path) else "wb") as f:
|
||||
f.seek(offset)
|
||||
f.write(data)
|
||||
return len(data)
|
||||
|
||||
|
||||
def _run(cmd: list) -> None:
|
||||
r = subprocess.run(cmd, capture_output=True, text=True)
|
||||
if r.returncode != 0:
|
||||
raise RuntimeError(f"ffmpeg failed: {r.stderr.strip()[-600:]}")
|
||||
|
||||
|
||||
def encode(src: str, fmt: str, seconds: float, out_base: str) -> str:
|
||||
"""The recording `src` (WebM / fragmented MP4 from MediaRecorder, which many apps won't play or preview) as
|
||||
`fmt` -> the written path. Both MP4s are H.264 + AAC, yuv420p, faststart. mp4: CRF 18. small: the bitrate set so
|
||||
the file stays under 9.5 MB (smaller frame when it gets low). gif: palette per video, 15 fps, made smaller until
|
||||
it fits 9.5 MB."""
|
||||
if fmt not in EXPORT_FORMATS:
|
||||
raise ValueError(f"unknown format {fmt!r}")
|
||||
exe = VIO.ffmpeg_bin()
|
||||
if exe is None:
|
||||
return _encode_av(src, fmt, out_base)
|
||||
seconds = max(0.5, float(seconds or 0))
|
||||
even = "scale=trunc(iw/2)*2:trunc(ih/2)*2"
|
||||
if fmt == "gif":
|
||||
out = out_base + ".gif"
|
||||
for width, fps in ((720, 15), (540, 15), (480, 12), (360, 12), (270, 10)):
|
||||
vf = f"fps={fps},scale='min({width},iw)':-2:flags=lanczos,split[a][b];[a]palettegen=stats_mode=diff[p];[b][p]paletteuse=dither=bayer:bayer_scale=4"
|
||||
_run([exe, "-y", "-v", "error", "-i", src, "-vf", vf, "-loop", "0", out])
|
||||
if os.path.getsize(out) <= LIMIT_BYTES:
|
||||
break
|
||||
return out
|
||||
out = out_base + ".mp4"
|
||||
if fmt == "mp4":
|
||||
_run([exe, "-y", "-v", "error", "-i", src, "-vf", even, "-c:v", "libx264", "-preset", "medium", "-crf", "18",
|
||||
"-pix_fmt", "yuv420p", "-c:a", "aac", "-b:a", "192k", "-movflags", "+faststart", out])
|
||||
return out
|
||||
audio_k = 128
|
||||
for shrink in (1.0, 0.92, 0.8): # a VBR encode can overshoot a little: a second, lower try
|
||||
kbps = int(min(8000, max(250, LIMIT_BYTES * 8 / 1000 / seconds * shrink - audio_k)))
|
||||
vf = even if kbps >= 1500 else "scale=-2:'min(720,ih)':flags=lanczos," + even if kbps >= 700 else \
|
||||
"scale=-2:'min(480,ih)':flags=lanczos," + even
|
||||
_run([exe, "-y", "-v", "error", "-i", src, "-vf", vf, "-c:v", "libx264", "-preset", "medium",
|
||||
"-b:v", f"{kbps}k", "-maxrate", f"{int(kbps * 1.4)}k", "-bufsize", f"{kbps * 2}k", "-pix_fmt", "yuv420p",
|
||||
"-r", "30", "-c:a", "aac", "-b:a", f"{audio_k}k", "-ac", "2", "-movflags", "+faststart", out])
|
||||
if os.path.getsize(out) <= LIMIT_BYTES:
|
||||
break
|
||||
return out
|
||||
|
||||
|
||||
def _encode_av(src: str, fmt: str, out_base: str) -> str:
|
||||
"""No ffmpeg program: decode with PyAV and write an MP4 (no sound) or a GIF (PIL); 'small' at most 720p."""
|
||||
import av
|
||||
from PIL import Image
|
||||
frames, fps = [], 30.0
|
||||
with av.open(src) as c:
|
||||
st = c.streams.video[0]
|
||||
fps = float(st.average_rate or st.guessed_rate or 30)
|
||||
for f in c.decode(st):
|
||||
frames.append(f.to_ndarray(format="rgb24"))
|
||||
if not frames:
|
||||
raise RuntimeError("the recording has no frames")
|
||||
h, w = frames[0].shape[:2]
|
||||
if fmt == "gif":
|
||||
out = out_base + ".gif"
|
||||
step = max(1, round(fps / 12))
|
||||
for side in (540, 400, 300, 220):
|
||||
k = min(1.0, side / max(h, w))
|
||||
ims = [Image.fromarray(x).resize((max(2, int(w * k)), max(2, int(h * k))), Image.LANCZOS) for x in frames[::step]]
|
||||
ims[0].save(out, save_all=True, append_images=ims[1:], duration=int(1000 * step / fps), loop=0, optimize=True)
|
||||
if os.path.getsize(out) <= LIMIT_BYTES:
|
||||
break
|
||||
return out
|
||||
out = out_base + ".mp4"
|
||||
k = 1.0 if fmt == "mp4" else min(1.0, 720 / h)
|
||||
W2, H2 = int(w * k) // 2 * 2, int(h * k) // 2 * 2
|
||||
wr = VIO.Mp4Writer(out, W2, H2, fps, None, crf=18 if fmt == "mp4" else 27, who="BFS Video Compare")
|
||||
try:
|
||||
for a in range(0, len(frames), 16):
|
||||
x = torch.from_numpy(np.stack(frames[a:a + 16])).float() / 255
|
||||
wr.write(_scale(x, H2, W2))
|
||||
finally:
|
||||
wr.close()
|
||||
print("[BFS Video Compare] no ffmpeg program: saved without sound", flush=True)
|
||||
return out
|
||||
|
||||
|
||||
def finish_export(rid: str, fmt: str, seconds: float, name: str) -> dict:
|
||||
src = os.path.join(_export_dir(), f"{rid}.rec")
|
||||
if not _ID.match(rid or "") or not os.path.exists(src):
|
||||
raise ValueError("recording not found")
|
||||
stem = re.sub(r"[^\w.-]+", "_", os.path.splitext(name or "bfs_compare")[0])[:80] or "bfs_compare"
|
||||
try:
|
||||
out = encode(src, fmt, seconds, os.path.join(_export_dir(), f"{stem}_{rid[:6]}"))
|
||||
finally:
|
||||
os.remove(src)
|
||||
sub = os.path.relpath(os.path.dirname(out), folder_paths.get_temp_directory()).replace(os.sep, "/")
|
||||
mb = os.path.getsize(out) / 1024 / 1024
|
||||
print(f"[BFS Video Compare] saved {os.path.basename(out)} ({EXPORT_FORMATS[fmt]}, {mb:.1f} MB)", flush=True)
|
||||
return {"filename": os.path.basename(out), "subfolder": sub, "type": "temp", "mb": round(mb, 2)}
|
||||
|
||||
|
||||
try:
|
||||
import asyncio
|
||||
|
||||
from aiohttp import web
|
||||
from server import PromptServer
|
||||
|
||||
@PromptServer.instance.routes.post("/bfs/compare/part")
|
||||
async def _bfs_compare_part(request):
|
||||
try:
|
||||
data = await request.read()
|
||||
n = await asyncio.get_running_loop().run_in_executor(
|
||||
None, lambda: write_part(request.query.get("id", ""), int(request.query.get("offset", "-1")), data))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return web.json_response({"error": f"{type(exc).__name__}: {exc}"}, status=400)
|
||||
return web.json_response({"written": n})
|
||||
|
||||
@PromptServer.instance.routes.post("/bfs/compare/export")
|
||||
async def _bfs_compare_export(request):
|
||||
try:
|
||||
b = await request.json()
|
||||
r = await asyncio.get_running_loop().run_in_executor(
|
||||
None, lambda: finish_export(b.get("id", ""), b.get("format", ""), float(b.get("seconds") or 0), b.get("name", "")))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
return web.json_response({"error": f"{type(exc).__name__}: {exc}"}, status=400)
|
||||
return web.json_response(r)
|
||||
except Exception as _exc: # noqa: BLE001 - outside ComfyUI (tests)
|
||||
if "PromptServer" not in repr(_exc) and "server" not in repr(_exc):
|
||||
print(f"[BFS Video Compare] export routes not registered: {_exc!r}", flush=True)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {"BFSVideoCompare": BFSVideoCompare}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {"BFSVideoCompare": "BFS Video Compare"}
|
||||
|
||||
@@ -0,0 +1,166 @@
|
||||
# Flash Partitions for H3 native temporal alignment
|
||||
|
||||
Flash Partitions is an inference implementation of BFS's bounded temporal attention rule. It retains the native video reference and the same temporal RoPE alignment, while evaluating the biased attention through native FlashAttention calls and stable partition merging. It requires no new learned weights and no per-score `torch.compile` kernel.
|
||||
|
||||
**Scope:** this is an experimental backend, not a claim of universal acceleration or identity preservation. The equations are equivalent in real arithmetic; FP16/BF16 implementations are not bit-identical. Dense attention still has quadratic work in sequence length.
|
||||
|
||||
## 1. What receives the temporal bias?
|
||||
|
||||
H3 packs text, reference pictures, native motion-video latents, audio and output-video latents into one sequence. Only **output-video queries consulting the selected motion video's keys** receive our temporal penalty. Image-reference coordinates remain native, as do the guide's spatial coordinates. The guide's temporal RoPE origin is aligned to the output clock before attention.
|
||||
|
||||

|
||||
|
||||
The matrix above is illustrative, not a measured attention map or actual token-count distribution. Color shows the multiplier of an **unnormalized** softmax weight, not the final attention probability. All keys remain available. Gray edges retain their original score rule; their final normalized probabilities can still change because the softmax denominator changes.
|
||||
|
||||
For an affected query/key pair, define:
|
||||
|
||||
```text
|
||||
dt = abs(output_time - guide_time)
|
||||
penalty(dt) = min(strength * (max(dt - window, 0) / falloff)^2, cap)
|
||||
modified_score = dot(Q, K) / sqrt(head_dim) - penalty(dt)
|
||||
```
|
||||
|
||||
Defaults: strength 2.0, window 0.20 seconds, falloff 0.25 seconds, cap 6.0. Both videos must have matching temporal grids. This experiment uses 24 fps. This is a soft preference for corresponding times; it is not a proof of physical motion correspondence.
|
||||
|
||||
## 2. Why the original FlexAttention path can be slower
|
||||
|
||||
The original backend evaluates a score modifier inside compiled FlexAttention. Its rule includes time lookup, distance, a bounded quadratic penalty and edge selection. The expression is evaluated over attention score tiles across heads and blocks. It never creates a full S×S bias tensor, but the extra per-score work remains.
|
||||
|
||||
At 116,883 sequence rows and 56 heads, the conceptual full score space contains roughly 765 billion query/key/head combinations per attention block. Kernels tile that computation rather than storing all those scores simultaneously. A new compiled shape/backend can also require initialization or compilation; native FlashAttention avoids that particular custom-kernel compilation path.
|
||||
|
||||
The whole model additionally spends time on projections, MLPs, quantized weights, model management, preview decoding and potentially CPU/GPU offload. Large native guide sequences increase both compute and live tensor sizes. We have not measured each of those contributions in the user's full workflow, so a slow step must not automatically be labeled “compiling” or “offloading.” GPU utilization alone cannot distinguish them.
|
||||
|
||||
## 3. Decompose the same rule into positive contributions
|
||||
|
||||
Because `0 <= penalty <= cap`, the guide's exponential score factor can be written as:
|
||||
|
||||
```text
|
||||
exp(-penalty) = exp(-cap) + [exp(-penalty) - exp(-cap)]
|
||||
```
|
||||
|
||||
Both contributions are nonnegative. At distant times the penalty reaches its cap, so the bracketed local correction is exactly zero. The baseline contribution remains positive: distant guide keys are not removed.
|
||||
|
||||

|
||||
|
||||
With the defaults, the correction reaches zero at:
|
||||
|
||||
```text
|
||||
window + falloff * sqrt(cap / strength)
|
||||
= 0.20 + 0.25 * sqrt(6 / 2)
|
||||
≈ 0.633 seconds
|
||||
```
|
||||
|
||||
That threshold applies to the correction, not to the availability of guide keys.
|
||||
|
||||
| Time difference | Penalty | Required factor | Global baseline | Local correction |
|
||||
|---|---:|---:|---:|---:|
|
||||
| 0.00 s | 0.00 | 1.000000 | 0.002479 | 0.997521 |
|
||||
| 0.40 s | 1.28 | 0.278037 | 0.002479 | 0.275559 |
|
||||
| 0.80 s | 6.00 | 0.002479 | 0.002479 | 0.000000 |
|
||||
|
||||
The inference backend computes three types of contributions:
|
||||
|
||||
1. All non-motion keys, with ordinary weight 1. They can occupy multiple contiguous segments.
|
||||
2. All motion-video keys, with uniform factor `exp(-cap)`.
|
||||
3. Nearby motion-frame keys, with factor `exp(-penalty) - exp(-cap)`.
|
||||
|
||||
The last two contributions intentionally overlap in their keys. Their exponential weights add to the intended original weight, so this does not double-count the final distribution. All spatial tokens of a source frame have the same temporal correction for a given output frame. Frame batching exploits that repeated structure. It is unrelated to averaging generated RGB frames or merging outputs from separate video generations.
|
||||
|
||||

|
||||
|
||||
## 4. Why log-sum-exp is necessary
|
||||
|
||||
A normal attention call returns a normalized local result. Averaging those results would assign equal importance to partitions with very different softmax mass. We therefore retain each partition's natural-log normalizer, `L_i`.
|
||||
|
||||
For partition output `O_i` and nonnegative multiplier `w_i`:
|
||||
|
||||
```text
|
||||
adjusted_L_i = L_i + log(w_i)
|
||||
L = logsumexp(adjusted_L_1, adjusted_L_2, ...)
|
||||
O = sum_i exp(adjusted_L_i - L) * O_i
|
||||
```
|
||||
|
||||
Zero-weight corrections can be omitted. The code uses `logaddexp` for successive merges and `expm1` to calculate small positive correction weights stably. It does not subtract two large attention outputs.
|
||||
|
||||
Output-frame chunks default to eight latent frames. This bounds the temporary merge buffers. It does not crop the output into independently generated clips or restrict the keys to those eight frames. Queries outside the output video use ordinary full attention.
|
||||
|
||||
Setting strength or cap to zero removes the score penalty in this backend; the separate temporal RoPE alignment remains active. Bypassing the temporal node/mode removes both changes.
|
||||
|
||||
## 5. What was measured?
|
||||
|
||||
### Isolated synthetic attention
|
||||
|
||||
RTX PRO 6000 Blackwell, PyTorch 2.12.0+cu130, BF16, 107,126 tokens, 8 heads, head dimension 128:
|
||||
|
||||
| Backend | Approximate attention call time |
|
||||
|---|---:|
|
||||
| Unmodified native PyTorch FlashAttention | 0.327 s |
|
||||
| Flash Partitions | 0.422 s |
|
||||
| Original temporal FlexAttention | 0.628 s |
|
||||
|
||||
The training process was running concurrently. These short microbenchmarks indicate about 1.5× improvement over the original temporal backend for that tested shape; they are not a controlled full-model speed claim, do not use H3's full 56 heads and do not establish RTX 5090 performance. For a small 2,272-token shape, partition-launch/merge overhead made this backend slower than FlexAttention. There is no universally fastest choice demonstrated here.
|
||||
|
||||
A separate small CUDA comparison against SDPA with an explicit temporal bias measured relative output norm error 0.00452 in BF16. CPU tests exercise the decomposition against the original score rule at multiple strengths/caps and with frame-chunk boundaries. This validates numerical behavior, not identity or motion quality.
|
||||
|
||||
### User-reported full-workflow logs
|
||||
|
||||
RTX 5090, requested output 544×928 and 360 frames:
|
||||
|
||||
| Temporal backend | Guide scale | Guide rows | Packed rows | First complete model evaluation |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Flash Partitions | 1.0 | 52,751 | 116,883 | 179.18 s |
|
||||
| Flash Partitions | 0.5 | 11,984 | 70,020 | 54.45 s |
|
||||
|
||||
The second model evaluation at scale 0.5 was reported as 47.19 s. These are user logs, not a controlled benchmark with all settings and warm-up conditions audited. The original Flex sampler's first step was reported as 603.40 s, including initialization, so it is not directly comparable to a warmed-up attention call.
|
||||
|
||||
The separately reported first-attention times were 2.689 s at scale 1.0 and 4.113 s at scale 0.5. That first-call measurement is not representative of every block; do not multiply it by the number of blocks to estimate the full step. Whole-model evaluation timing is the more useful end-to-end observation here.
|
||||
|
||||
## 6. How to use it
|
||||
|
||||
See [native temporal setup](../H3_NATIVE_TEMPORAL.md) for MODEL routing and the conditioning contract.
|
||||
|
||||
- Standalone node: select `attention_backend = flash partitions (no compile)`.
|
||||
- Shot Conditioning: select `guide_mode = native temporal alignment (experimental)` and `native_temporal_backend = flash partitions (no compile)`.
|
||||
- Use only one temporal patch on the MODEL path. Keep Add Guide and conflicting attention/block replacements disabled.
|
||||
- `native_guide_scale = 1.0` preserves source detail. Try 0.5 for a lower-token motion reference; this keeps output dimensions, identity images and all guide frames/timestamps, but reduces guide spatial detail. H3 dimension rounding means the token reduction is not exactly fourfold.
|
||||
- For standalone conditioning, resize only the guide video before native Reference to Video to obtain that spatial reduction.
|
||||
|
||||
The selected video is still VAE-encoded as a native reference. “Add Guide OFF” does not mean “no guide-video latents.” Qwen also sees the native video presentation. Reducing Qwen sampling alone would not remove the native DiT reference tokens.
|
||||
|
||||
## 7. Backend requirements and limitations
|
||||
|
||||
The current compile-free path calls PyTorch's native CUDA FlashAttention primitive with log-sum-exp outputs. It requires CUDA FP16/BF16 and a compatible PyTorch build. It is inference-only, supports one packed self-attention sequence and requires contiguous video rows, fixed spatial token count per frame and matching guide/output temporal grids. It does not use the ComfyUI attention backend selector for the modified main attention.
|
||||
|
||||
Comfy Kitchen INT8 attention's audited public API returns the output tensor but no log-sum-exp normalizer. It is therefore not a drop-in replacement for these partition calls. Its support for masks does not by itself provide a fused temporal rule without constructing a bias representation. The separate [experimental Kitchen frame-wise backend](../H3_NATIVE_TEMPORAL.md#comfy-kitchen-and-model-attention-backend-111513), added in 1.115.13, avoids partition merging by using a broadcast key bias per target latent frame. It automatically honors the official Model Attention Backend selection. Its initial diagnostic was slower than Flash Partitions; it is not a drop-in partition kernel replacement.
|
||||
|
||||
Flash Partitions preserves the dense score policy and its quadratic work. More speed may require reducing native guide tokens, improving kernels/merges or choosing a measured alternative. Hard windows, sparse routing or independently generated temporal chunks would change behavior and need separate quality evaluation.
|
||||
|
||||
Alignment does not guarantee identity binding. The user has reported useful results from the existing 10500 LoRA plus this patch, and weaker identity retention from an early fresh-native checkpoint. Those are qualitative observations, not a matched-training comparison or evidence that the patch alone teaches swapping. No LoRA strength ramp, image-reference boost or identity loss is implemented by this backend.
|
||||
|
||||
## Native one-pass Kitchen extension
|
||||
|
||||
From 1.115.16, the optional [native Kitchen temporal extension](../H3_NATIVE_TEMPORAL.md#one-pass-native-kitchen-kernels-111516) fuses the temporal rule into dense INT8 and sol-attn kernels. It removes the per-frame attention loop. Initial full-head diagnostics were faster than Flash Partitions, while visual/end-to-end validation remains pending. From 1.115.17, first-use setup is automatic on supported Linux SM120 systems with a CUDA-13-capable driver; manual installation remains available elsewhere.
|
||||
|
||||
## Experimental sol-attn compatibility
|
||||
|
||||
Version 1.115.14 can recognize Model Sparse Attention with sol-attn and preserve the temporal score rule via repeated frame-wise calls. This initial compatibility path was slower than Flash Partitions; it is not the proposed fast one-pass sparse kernel. See [implementation limits and measurements](../H3_NATIVE_TEMPORAL.md#model-sparse-attention-sol-attn-compatibility-111514). SLA and VSA remain unsupported.
|
||||
|
||||
## Video example
|
||||
|
||||
See the [synchronized six-column video and interpretation](../H3_NATIVE_TEMPORAL.md#video-comparison-an-add-guide-trained-lora-with-native-temporal-alignment), including the step-10500 Add Guide-trained LoRA running with the new native temporal method. The separate early step-750 identity-drift report is distinguished there.
|
||||
|
||||
## Source and reproducible figures
|
||||
|
||||
- [Partition planning, native FlashAttention and stable merge](../minimax_h3_temporal_fast.py)
|
||||
- [MODEL wrapper, layout alignment and backend selection](../minimax_h3_native_temporal.py)
|
||||
- [Original temporal score policy / Flex backend](../minimax_h3_native_temporal_core.py)
|
||||
- [Numerical decomposition test](../tests/test_minimax_h3_temporal_fast.py)
|
||||
- [Figure source](illustrations/render_h3_flash_partitions.py), with exported SVGs beside the PNGs.
|
||||
|
||||
Figures can be regenerated on CPU with Python, NumPy and Matplotlib:
|
||||
|
||||
```bash
|
||||
python docs/illustrations/render_h3_flash_partitions.py
|
||||
```
|
||||
|
||||
The illustrations describe the mechanism; they are not generated attention measurements or model-output examples.
|
||||
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"probe": {
|
||||
"torch": "2.12.0+cu130",
|
||||
"kitchen": "0.2.35",
|
||||
"status": "passed",
|
||||
"relative_error": 0.01505568902939558,
|
||||
"first_call_seconds": 1.416990154015366,
|
||||
"shape": [
|
||||
1,
|
||||
2,
|
||||
1248,
|
||||
128
|
||||
],
|
||||
"bias_elements": 8736,
|
||||
"gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
|
||||
"peak_allocated_bytes": 29803008
|
||||
},
|
||||
"benchmark": {
|
||||
"torch": "2.12.0+cu130",
|
||||
"kitchen": "0.2.35",
|
||||
"status": "passed",
|
||||
"seconds": {
|
||||
"flash_partitions": 0.017392314039170742,
|
||||
"kitchen_frame_bias": 0.10426661401288584
|
||||
},
|
||||
"shape": [
|
||||
1,
|
||||
2,
|
||||
42836,
|
||||
128
|
||||
],
|
||||
"frames": 22,
|
||||
"guide_rows": 7062,
|
||||
"bias_elements": 942392,
|
||||
"gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
|
||||
"peak_allocated_bytes": 146629120,
|
||||
"conditions": "One warmup and one timed evaluation per backend; 2 heads, not full H3; concurrent Swap Anything training; not an end-to-end speed benchmark"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,137 @@
|
||||
{
|
||||
"verification": {
|
||||
"torch": "2.12.0+cu130",
|
||||
"gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
|
||||
"cases": [
|
||||
{
|
||||
"dtype": "torch.bfloat16",
|
||||
"shape": [
|
||||
1,
|
||||
2,
|
||||
53,
|
||||
128
|
||||
],
|
||||
"dense_vs_fp32": 0.012292885221540928,
|
||||
"sol_exact_vs_fp32": 0.011282283812761307,
|
||||
"sol_onepass_vs_framewise": 0.0,
|
||||
"finite": true
|
||||
},
|
||||
{
|
||||
"dtype": "torch.bfloat16",
|
||||
"shape": [
|
||||
1,
|
||||
2,
|
||||
1253,
|
||||
128
|
||||
],
|
||||
"dense_vs_fp32": 0.015596109442412853,
|
||||
"sol_exact_vs_fp32": 0.013514556922018528,
|
||||
"sol_onepass_vs_framewise": 2.1619540348183364e-05,
|
||||
"finite": true
|
||||
},
|
||||
{
|
||||
"dtype": "torch.float16",
|
||||
"shape": [
|
||||
1,
|
||||
2,
|
||||
53,
|
||||
128
|
||||
],
|
||||
"dense_vs_fp32": 0.013024155050516129,
|
||||
"sol_exact_vs_fp32": 0.011345107108354568,
|
||||
"sol_onepass_vs_framewise": 0.0,
|
||||
"finite": true
|
||||
},
|
||||
{
|
||||
"dtype": "torch.float16",
|
||||
"shape": [
|
||||
1,
|
||||
2,
|
||||
1253,
|
||||
128
|
||||
],
|
||||
"dense_vs_fp32": 0.015380656346678734,
|
||||
"sol_exact_vs_fp32": 0.013211732730269432,
|
||||
"sol_onepass_vs_framewise": 5.5366094784403685e-06,
|
||||
"finite": true
|
||||
}
|
||||
],
|
||||
"status": "passed"
|
||||
},
|
||||
"edge_and_bridge": {
|
||||
"cases": [
|
||||
{
|
||||
"strength": 10.0,
|
||||
"cap": 0.01,
|
||||
"window": 0.2,
|
||||
"dense_error": 0.015495287254452705,
|
||||
"sol_exact_error": 0.013426524586975574
|
||||
},
|
||||
{
|
||||
"strength": 0.1,
|
||||
"cap": 30.0,
|
||||
"window": 0.2,
|
||||
"dense_error": 0.015566659159958363,
|
||||
"sol_exact_error": 0.013471982441842556
|
||||
},
|
||||
{
|
||||
"strength": 0.0,
|
||||
"cap": 6.0,
|
||||
"window": 0.0,
|
||||
"dense_error": 0.015584140084683895,
|
||||
"sol_exact_error": 0.01359549630433321
|
||||
},
|
||||
{
|
||||
"strength": 2.0,
|
||||
"cap": 0.0,
|
||||
"window": 0.2,
|
||||
"dense_error": 0.015431488864123821,
|
||||
"sol_exact_error": 0.01330660656094551
|
||||
}
|
||||
],
|
||||
"bridge_native_calls": {
|
||||
"dense": 1,
|
||||
"sol": 1
|
||||
},
|
||||
"stream_isolation": "passed",
|
||||
"status": "passed"
|
||||
},
|
||||
"benchmark": {
|
||||
"torch": "2.12.0+cu130",
|
||||
"gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
|
||||
"conditions": "Concurrent training; isolated attention; one warmup + one timed evaluation per implementation; no end-to-end speed claim",
|
||||
"rows": [
|
||||
{
|
||||
"shape": [
|
||||
1,
|
||||
2,
|
||||
42836,
|
||||
128
|
||||
],
|
||||
"seconds": {
|
||||
"flash_partitions": 0.01711059600347653,
|
||||
"kitchen_framewise": 0.10465262399520725,
|
||||
"kitchen_onepass": 0.011890062014572322,
|
||||
"sol_framewise": 0.15987364604370669,
|
||||
"sol_onepass": 0.01116465002996847
|
||||
}
|
||||
},
|
||||
{
|
||||
"shape": [
|
||||
1,
|
||||
56,
|
||||
42836,
|
||||
128
|
||||
],
|
||||
"seconds": {
|
||||
"flash_partitions": 0.42317933501908556,
|
||||
"kitchen_framewise": 0.5413047629990615,
|
||||
"kitchen_onepass": 0.24634322000201792,
|
||||
"sol_framewise": 2.675203786988277,
|
||||
"sol_onepass": 0.21625870000571012
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": "passed"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
{
|
||||
"probe": {
|
||||
"torch": "2.12.0+cu130",
|
||||
"kitchen": "0.2.35",
|
||||
"status": "finite_sparse_output",
|
||||
"relative_error": 0.46068075299263,
|
||||
"forced_all_keys_exact_relative_error": 0.013319959864020348,
|
||||
"first_call_seconds": 1.1965116760111414,
|
||||
"shape": [
|
||||
1,
|
||||
2,
|
||||
1248,
|
||||
128
|
||||
],
|
||||
"bias_elements": 8736,
|
||||
"gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
|
||||
"peak_allocated_bytes": 29803008
|
||||
},
|
||||
"benchmark": {
|
||||
"torch": "2.12.0+cu130",
|
||||
"kitchen": "0.2.35",
|
||||
"status": "passed",
|
||||
"seconds": {
|
||||
"flash_partitions": 0.017417979950550944,
|
||||
"sol_frame_bias": 0.16037049802253023
|
||||
},
|
||||
"shape": [
|
||||
1,
|
||||
2,
|
||||
42836,
|
||||
128
|
||||
],
|
||||
"frames": 22,
|
||||
"guide_rows": 7062,
|
||||
"bias_elements": 942392,
|
||||
"gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
|
||||
"peak_allocated_bytes": 210820096,
|
||||
"conditions": "One warmup and one timed evaluation per backend; 2 heads, not full H3; concurrent Swap Anything training; not an end-to-end speed benchmark"
|
||||
},
|
||||
"benchmark_h56": {
|
||||
"torch": "2.12.0+cu130",
|
||||
"kitchen": "0.2.35",
|
||||
"status": "passed",
|
||||
"seconds": {
|
||||
"flash_partitions": 0.41910952096804976,
|
||||
"sol_frame_bias": 2.6489629900315776
|
||||
},
|
||||
"shape": [
|
||||
1,
|
||||
56,
|
||||
42836,
|
||||
128
|
||||
],
|
||||
"frames": 22,
|
||||
"guide_rows": 7062,
|
||||
"bias_elements": 942392,
|
||||
"gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
|
||||
"peak_allocated_bytes": 5760942592,
|
||||
"conditions": "One warmup and one timed evaluation per backend; 56 heads; isolated attention, not a full H3 model evaluation; concurrent Swap Anything training; not an end-to-end speed benchmark"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"first_use": {
|
||||
"status": "passed",
|
||||
"version": "h3_native_temporal_kernel_v1",
|
||||
"shape": [
|
||||
1,
|
||||
2,
|
||||
83,
|
||||
128
|
||||
],
|
||||
"dense_and_sol_finite": true
|
||||
},
|
||||
"fresh_process_offline_cache": {
|
||||
"status": "passed",
|
||||
"network_requests": 0,
|
||||
"api_version": "h3_native_temporal_kernel_v1",
|
||||
"api_file": "temporal_attention.py"
|
||||
},
|
||||
"platform": "Linux x86_64 / RTX PRO 6000 Blackwell SM120 / CUDA 13 driver",
|
||||
"automatic_component_release": "h3-native-temporal-kernel-v1",
|
||||
"source_commit": "9953b6caec7a5cb3e2486dd97a365f7af44eedad"
|
||||
}
|
||||
|
After Width: | Height: | Size: 142 KiB |
|
After Width: | Height: | Size: 74 KiB |
|
After Width: | Height: | Size: 103 KiB |
|
After Width: | Height: | Size: 76 KiB |
|
After Width: | Height: | Size: 118 KiB |
|
After Width: | Height: | Size: 93 KiB |
@@ -0,0 +1,67 @@
|
||||
"""Render explanatory figures; no model inference or GPU required."""
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
import matplotlib
|
||||
matplotlib.use('Agg')
|
||||
import matplotlib.pyplot as plt
|
||||
from matplotlib.patches import Rectangle, FancyBboxPatch
|
||||
|
||||
OUT=Path(__file__).parent
|
||||
plt.rcParams.update({'font.family':'DejaVu Sans','font.size':11,'savefig.facecolor':'white','svg.hashsalt':'h3-flash-partitions'})
|
||||
|
||||
def save(fig,name):
|
||||
for ext in ['png','svg']:
|
||||
path=OUT/(name+'.'+ext)
|
||||
fig.savefig(path,dpi=160,bbox_inches='tight',metadata={'Date':None} if ext=='svg' else None)
|
||||
if ext=='svg':path.write_text('\n'.join(line.rstrip() for line in path.read_text().splitlines())+'\n')
|
||||
plt.close(fig)
|
||||
|
||||
# Conceptual groups: equal spacing within each video, not actual H3 token counts.
|
||||
fig,ax=plt.subplots(figsize=(10,7))
|
||||
n=170;rgb=np.full((n,n,3),.91)
|
||||
t=np.linspace(0,4,60);dt=np.abs(t[:,None]-t[None,:])
|
||||
p=np.minimum(2*(np.maximum(dt-.2,0)/.25)**2,6);weights=np.exp(-p)
|
||||
rgb[110:170,40:100]=plt.get_cmap('YlOrRd')(weights)[...,:3]
|
||||
ax.imshow(rgb,interpolation='nearest',origin='upper')
|
||||
bounds=[0,20,40,100,110,170];names=['Text','Pictures','Motion video','Audio','Output video']
|
||||
centers=[(a+b-1)/2 for a,b in zip(bounds,bounds[1:])]
|
||||
ax.set_xticks(centers,names);ax.set_yticks(centers,names)
|
||||
ax.xaxis.tick_top();ax.xaxis.set_label_position('top')
|
||||
ax.set_xlabel('Keys: what the model can consult',labelpad=14)
|
||||
ax.set_ylabel('Queries: what is being updated')
|
||||
for b in bounds[1:-1]:ax.axhline(b-.5,color='white',lw=2);ax.axvline(b-.5,color='white',lw=2)
|
||||
ax.add_patch(Rectangle((39.5,109.5),60,60,fill=False,edgecolor='#111827',lw=2))
|
||||
ax.set_title('The temporal bias changes one family of attention edges',pad=55,weight='bold')
|
||||
fig.text(.12,.02,'Gray: unchanged score rule. Colored block: output queries consulting the motion video.\nNear-time keys retain weight 1; distant keys retain a small positive weight. No hard mask.',fontsize=10)
|
||||
sm=plt.cm.ScalarMappable(norm=plt.Normalize(0,1),cmap='YlOrRd');fig.colorbar(sm,ax=ax,shrink=.65,label='Unnormalized weight multiplier exp(-penalty)')
|
||||
save(fig,'h3_temporal_attention_scope')
|
||||
|
||||
fig,ax=plt.subplots(figsize=(11,5))
|
||||
x=np.linspace(0,1.1,600);penalty=np.minimum(2*(np.maximum(x-.2,0)/.25)**2,6)
|
||||
baseline=np.exp(-6);full=np.exp(-penalty);correction=full-baseline
|
||||
ax.plot(x,full,color='#172554',lw=3,label='Required weight: exp(-penalty)')
|
||||
ax.plot(x,np.full_like(x,baseline),color='#ea580c',lw=2,ls='--',label='Global guide contribution: exp(-6)')
|
||||
ax.fill_between(x,baseline,full,color='#22c55e',alpha=.35,label='Positive local correction: exp(-penalty) - exp(-6)')
|
||||
ax.axvline(.2,color='#64748b',ls=':');ax.axvline(.2+.25*np.sqrt(3),color='#64748b',ls=':')
|
||||
ax.text(.205,.91,'0.20 s\nfree window',fontsize=10)
|
||||
ax.text(.65,.5,'Beyond ~0.633 s:\ncorrection is zero,\nbaseline remains',fontsize=10)
|
||||
ax.set(xlabel='Absolute time difference between output and guide (seconds)',ylabel='Multiplier of unnormalized softmax weight',ylim=(-.03,1.08),xlim=(0,1.1))
|
||||
ax.set_title('Same temporal rule, decomposed into positive contributions',weight='bold')
|
||||
ax.legend(loc='upper right',fontsize=9);ax.grid(alpha=.18)
|
||||
fig.tight_layout();save(fig,'h3_temporal_weight_decomposition')
|
||||
|
||||
fig,ax=plt.subplots(figsize=(14,7));ax.set(xlim=(0,14),ylim=(0,7));ax.axis('off')
|
||||
def box(x,y,w,h,text,color):
|
||||
ax.add_patch(FancyBboxPatch((x,y),w,h,boxstyle='round,pad=0.12',facecolor=color,edgecolor='#475569',lw=1.2))
|
||||
ax.text(x+w/2,y+h/2,text,ha='center',va='center',fontsize=10)
|
||||
def arrow(a,b):ax.annotate('',xy=b,xytext=a,arrowprops=dict(arrowstyle='->',lw=1.6,color='#475569'))
|
||||
box(.2,2.8,2,1.2,'Output-video\nquery chunk','#dbeafe')
|
||||
parts=[(5.05,'Other keys\nweight = 1','#ede9fe'),(2.85,'All motion-video keys\nweight = exp(-cap)','#ffedd5'),(.65,'Nearby motion-frame keys\nweight = exp(-penalty) - exp(-cap)','#dcfce7')]
|
||||
for y,text,color in parts:
|
||||
box(3,y,4,1.3,text,color);arrow((2.3,3.4),(2.8,y+.65))
|
||||
box(7.7,y,2.3,1.3,'Native FlashAttention\noutput + logsumexp',color);arrow((7.15,y+.65),(7.5,y+.65));arrow((10.15,y+.65),(10.8,3.4))
|
||||
box(11,2.65,2.7,1.5,'Stable weighted merge\n\nSame biased softmax','#dbeafe')
|
||||
ax.text(7,6.7,'Flash Partitions: reuse dense kernels instead of compiling a per-score rule',ha='center',weight='bold',fontsize=13)
|
||||
ax.text(.2,.04,'Guide contributions overlap intentionally; their weights sum to the original weight.\nQueries outside the output video use ordinary attention. Spatial tokens are retained; corrections are batched by frame.',fontsize=10)
|
||||
save(fig,'h3_flash_partitions_flow')
|
||||
print('Rendered 3 PNG and 3 SVG figures')
|
||||
|
After Width: | Height: | Size: 59 KiB |
@@ -0,0 +1,159 @@
|
||||
"""H3 video context in Qwen embeddings without a native video-latent reference."""
|
||||
import math
|
||||
import time
|
||||
from dataclasses import replace
|
||||
|
||||
from comfy_extras.nodes_minimax_h3 import MiniMaxH3ReferenceToVideo
|
||||
from comfy_api.latest import io
|
||||
|
||||
|
||||
class _VideoContextClip:
|
||||
"""Request-local adapter; never changes the user's CLIP or cached conditioning."""
|
||||
|
||||
def __init__(self, clip, items):
|
||||
self._clip = clip
|
||||
self._items = items
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(self._clip, name)
|
||||
|
||||
def tokenize(self, text, **kwargs):
|
||||
items = list(kwargs.get("minimax_ref_items") or [])
|
||||
# Pictures keep their numbering. Context videos precede any explicit
|
||||
# native video/audio references, so the shot remains <Video 1>.
|
||||
index = 0
|
||||
while index < len(items) and items[index].get("type") == "image":
|
||||
index += 1
|
||||
kwargs["minimax_ref_items"] = items[:index] + self._items + items[index:]
|
||||
tokens = self._clip.tokenize(text, **kwargs)
|
||||
def count_video_blocks(value):
|
||||
if isinstance(value, dict):
|
||||
if value.get("minimax_video_block") is True:
|
||||
return 1
|
||||
return sum(count_video_blocks(v) for k, v in value.items() if k != "data")
|
||||
if isinstance(value, (list, tuple)):
|
||||
return sum(count_video_blocks(v) for v in value)
|
||||
return 0
|
||||
expected = sum((item["data"].shape[0] + 1) // 2 for item in self._items)
|
||||
actual = count_video_blocks(tokens)
|
||||
if actual < expected:
|
||||
raise ValueError("CLIP did not produce the H3 video vision blocks. Use the native MiniMax H3 CLIP loader.")
|
||||
print(f"[BFS H3 CLIP context] tokenizer confirmed {actual} video vision blocks "
|
||||
f"(guide requires {expected}); encoding next", flush=True)
|
||||
return tokens
|
||||
|
||||
def encode_from_tokens_scheduled(self, tokens, *args, **kwargs):
|
||||
start = time.perf_counter()
|
||||
result = self._clip.encode_from_tokens_scheduled(tokens, *args, **kwargs)
|
||||
shapes = [tuple(entry[0].shape) for entry in result]
|
||||
print(f"[BFS H3 CLIP context] Qwen encoding complete in {time.perf_counter() - start:.2f}s; "
|
||||
f"conditioning shapes={shapes}. Guide native video reference blocks: 0", flush=True)
|
||||
return result
|
||||
|
||||
|
||||
def clip_with_video_context(clip, videos, length, source_fps=24.0, context_fps=2.0):
|
||||
"""Native H3 24-fps frame convention, presented to Qwen at a selectable sampling rate.
|
||||
|
||||
Only selected frames are resized. No video VAE encoding or minimax_refs
|
||||
video block is created here. Real image reference blocks are unaffected.
|
||||
"""
|
||||
from comfy_extras.nodes_minimax_h3 import (
|
||||
CANVAS_MULTIPLE, _resize, adapt_canvas, temporal_shape,
|
||||
)
|
||||
|
||||
if not math.isfinite(source_fps) or source_fps <= 0:
|
||||
raise ValueError("CLIP video context needs a positive source frame rate")
|
||||
if not math.isfinite(context_fps) or not 0.5 <= context_fps <= 24:
|
||||
raise ValueError("clip_video_fps must be between 0.5 and 24")
|
||||
effective_fps = min(context_fps, source_fps)
|
||||
frame_count = temporal_shape(length)[0]
|
||||
items = []
|
||||
for frames in (videos or {}).values():
|
||||
if frames is None:
|
||||
continue
|
||||
if frames.ndim != 4 or frames.shape[-1] < 3:
|
||||
raise ValueError("CLIP video context needs IMAGE frames [frames, height, width, channels]")
|
||||
n = min(frames.shape[0], frame_count)
|
||||
if n < 5:
|
||||
raise ValueError("H3 CLIP video context needs at least 5 frames at the native 24 fps")
|
||||
n -= (n - 5) % 17
|
||||
height, width = frames.shape[1:3]
|
||||
cw, ch = adapt_canvas(width, height)
|
||||
if width * height < cw * ch:
|
||||
cw = max(CANVAS_MULTIPLE, round(width / CANVAS_MULTIPLE) * CANVAS_MULTIPLE)
|
||||
ch = max(CANVAS_MULTIPLE, round(height / CANVAS_MULTIPLE) * CANVAS_MULTIPLE)
|
||||
indices = sorted({round(i * source_fps / effective_fps)
|
||||
for i in range(math.ceil(n / source_fps * effective_fps))
|
||||
if round(i * source_fps / effective_fps) < n})
|
||||
selected = _resize(frames[indices], cw, ch, "disabled")
|
||||
times = [index / source_fps for index in indices]
|
||||
preview = times if len(times) <= 6 else times[:3] + times[-3:]
|
||||
print(f"[BFS H3 CLIP context] guide: input={frames.shape[0]} frames at {source_fps:g} fps; "
|
||||
f"used={n}; requested={context_fps:g} fps, effective={effective_fps:g} fps; "
|
||||
f"sent={len(indices)} RGB frames at {cw}x{ch}; "
|
||||
f"timestamps={[round(t, 3) for t in preview]} seconds (first/last); "
|
||||
"native guide video reference latents: OFF", flush=True)
|
||||
items.append({"type": "video", "data": selected,
|
||||
"timestamps": [index / source_fps for index in indices]})
|
||||
return _VideoContextClip(clip, items) if items else clip
|
||||
|
||||
|
||||
class BFSMiniMaxH3ReferenceToVideoGuideContext(MiniMaxH3ReferenceToVideo):
|
||||
@classmethod
|
||||
def define_schema(cls):
|
||||
schema = MiniMaxH3ReferenceToVideo.define_schema()
|
||||
# Keep native image/video/audio reference semantics. The separately
|
||||
# named guide is the only video excluded from native reference latents.
|
||||
inputs = list(schema.inputs)
|
||||
inputs.insert(1, io.Image.Input(
|
||||
"guide_video_0", optional=True,
|
||||
tooltip=("The motion guide / current shot, as native 24-fps frames. Qwen sees it at clip_video_fps "
|
||||
"as <Video 1>; no native guide video reference latents are added. Connect the "
|
||||
"same frames to Add Guide for motion. Other ref_videos start at <Video 2>."),
|
||||
))
|
||||
inputs.append(io.Float.Input(
|
||||
"clip_video_fps", default=2.0, min=0.5, max=24.0, step=0.5,
|
||||
tooltip=("Guide 0 sampling rate for Qwen only. Default/native presentation is 2 fps; "
|
||||
"try 4 or 8 fps. Higher values increase context and encoding cost, not output fps. "
|
||||
"Never duplicates frames above the source rate. Actual reference videos stay native 2 fps."),
|
||||
))
|
||||
for item in inputs:
|
||||
if item.id == "ref_images":
|
||||
item.template = io.Autogrow.TemplatePrefix(
|
||||
input=io.Image.Input("ref_image", tooltip="Reference image, in <Picture N> order"),
|
||||
prefix="ref_image_", min=0, max=100,
|
||||
)
|
||||
elif item.id == "ref_videos":
|
||||
item.tooltip = ("Actual native video references, separate from guide_video_0. With a guide "
|
||||
"connected, these are <Video 2>, <Video 3>, etc. They enter both CLIP and "
|
||||
"native reference latents, as in the original node. Up to 3 reference videos.")
|
||||
item.template.input.tooltip = item.tooltip
|
||||
return replace(
|
||||
schema, node_id="BFSMiniMaxH3ReferenceToVideoGuideContext",
|
||||
display_name="MiniMax H3 Reference to Video — Guide 0 CLIP Context (BFS)",
|
||||
description=("guide_video_0 enters Qwen/CLIP only as <Video 1>, without native guide video "
|
||||
"reference latents. Pictures and other reference videos retain native semantics. "
|
||||
"Use Add Guide separately for aligned motion. The target latent starts empty."),
|
||||
inputs=inputs,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def execute(cls, clip, prompt, width, height, length, ref_image_size="match", vae=None,
|
||||
audio_vae=None, guide_video_0=None, ref_images=None, ref_videos=None,
|
||||
ref_video_audios=None, ref_audios=None, clip_video_fps=2.0):
|
||||
if guide_video_0 is not None:
|
||||
clip = clip_with_video_context(clip, {"guide_video_0": guide_video_0}, length, context_fps=clip_video_fps)
|
||||
return MiniMaxH3ReferenceToVideo.execute(
|
||||
clip=clip, prompt=prompt, width=width, height=height, length=length,
|
||||
ref_image_size=ref_image_size, vae=vae, audio_vae=audio_vae,
|
||||
ref_images=ref_images, ref_videos=ref_videos,
|
||||
ref_video_audios=ref_video_audios, ref_audios=ref_audios,
|
||||
)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"BFSMiniMaxH3ReferenceToVideoGuideContext": BFSMiniMaxH3ReferenceToVideoGuideContext,
|
||||
}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"BFSMiniMaxH3ReferenceToVideoGuideContext": "MiniMax H3 Reference to Video — Guide 0 CLIP Context (BFS)",
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
"""Experimental native H3 temporal inference, matching ai-toolkit native_temporal_v1.
|
||||
|
||||
Per-model wrappers only: no global ComfyUI monkey patches or extra weights.
|
||||
"""
|
||||
import copy
|
||||
import inspect
|
||||
import math
|
||||
import time
|
||||
import torch
|
||||
try:
|
||||
from . import minimax_h3_native_temporal_core as core
|
||||
from . import minimax_h3_temporal_fast as fast
|
||||
from . import minimax_h3_temporal_kitchen as kitchen
|
||||
from . import minimax_h3_temporal_sol as sol
|
||||
from . import minimax_h3_sparse_compat as sparse_compat
|
||||
except ImportError:
|
||||
import minimax_h3_native_temporal_core as core
|
||||
import minimax_h3_temporal_fast as fast
|
||||
import minimax_h3_temporal_kitchen as kitchen
|
||||
import minimax_h3_temporal_sol as sol
|
||||
import minimax_h3_sparse_compat as sparse_compat
|
||||
|
||||
MARKER = 'bfs_h3_native_temporal_v1'
|
||||
SOL_BACKEND = 'sol-attn frame bias (experimental)'
|
||||
BACKENDS = ['flash partitions (no compile)', 'flex (original)', 'comfy kitchen (experimental)']
|
||||
|
||||
def dense_backend_selection(override):
|
||||
if override is None:
|
||||
return None
|
||||
# Recognize only ComfyUI's actual ModelPatcher setter wrapper and official
|
||||
# registered dense kernels, never arbitrary attention replacements by name.
|
||||
if getattr(override, '__module__', '') != 'comfy.model_patcher' or not getattr(override, '__qualname__', '').endswith('set_model_optimized_attention.<locals>.optimized_attention_override'):
|
||||
raise ValueError('Native temporal alignment conflicts with another attention replacement. Remove VSA/sparse or unknown attention patches.')
|
||||
try:
|
||||
from comfy.ldm.modules.attention import get_attention_function
|
||||
selected = inspect.getclosurevars(override).nonlocals['optimized_attention']
|
||||
except (ImportError, KeyError, TypeError) as exc:
|
||||
raise ValueError('Unable to identify the Model Attention Backend selection.') from exc
|
||||
for name in ('comfy_kitchen_int8', 'pytorch'):
|
||||
if selected is get_attention_function(name, None) and selected is not None:
|
||||
return name
|
||||
raise ValueError('Unsupported Model Attention Backend for native temporal alignment.')
|
||||
|
||||
|
||||
|
||||
def temporal_layout(native, refs, keyframes, video_index=1):
|
||||
if keyframes:
|
||||
raise ValueError('Native temporal mode cannot be combined with Add Guide, first-frame guides or history. Remove those guides.')
|
||||
required = ('position_ids', 'segments', 'seq_len', 'signature')
|
||||
if any(not hasattr(native, x) for x in required):
|
||||
raise RuntimeError('Unsupported H3 PackedLayout API; update ComfyUI.')
|
||||
result = copy.copy(native)
|
||||
result.position_ids = native.position_ids.clone()
|
||||
source = torch.zeros(native.seq_len, dtype=torch.bool)
|
||||
target = torch.zeros_like(source)
|
||||
ta, tb, _ = next(s for s in native.segments if s[2] == 'video')
|
||||
target[ta:tb] = True
|
||||
# Native packing is text, each reference's optional audio then video, targets.
|
||||
cursor = next(s[1] for s in native.segments if s[2] == 'text')
|
||||
videos = 0
|
||||
for ref in refs:
|
||||
kind = ref['kind']
|
||||
if kind == 'image':
|
||||
cursor += (ref['latent_h']//2)*(ref['latent_w']//2)
|
||||
elif kind == 'audio':
|
||||
cursor += 2*ref['ref_audio_t']
|
||||
elif kind in ('video', 'video_audio'):
|
||||
videos += 1
|
||||
cursor += 2*ref.get('ref_audio_t', 0)
|
||||
n = ref['latent_t']*(ref['latent_h']//2)*(ref['latent_w']//2)
|
||||
if videos == video_index:
|
||||
source[cursor:cursor+n] = True
|
||||
result.position_ids[cursor:cursor+n, 0] += native.position_ids[ta, 0] - native.position_ids[cursor, 0]
|
||||
cursor += n
|
||||
else:
|
||||
raise ValueError('Unknown native H3 reference kind: '+str(kind))
|
||||
if not source.any():
|
||||
raise ValueError(f'Native temporal mode requires <Video {video_index}> as a native video reference. Connect Reference to Video positive conditioning.')
|
||||
st = torch.unique_consecutive(result.position_ids[source, 0])
|
||||
tt = torch.unique_consecutive(result.position_ids[target, 0])
|
||||
if st.shape != tt.shape or not torch.allclose(st, tt, atol=1e-9, rtol=0):
|
||||
raise ValueError('Native temporal guide and output must have identical temporal grids. Use the same frame window, 24 fps and matching H3 frame count (e.g. 107). No automatic motion stretching is applied.')
|
||||
return result, source, target
|
||||
|
||||
|
||||
def attention_override(ctx, previous=None, backend=BACKENDS[1], plan=None, log_first=False, sparse_patch=None, dense_backend=None):
|
||||
first=[log_first]
|
||||
def override(original, q, k, v, heads, mask=None, skip_reshape=False, skip_output_reshape=False, **kwargs):
|
||||
# Token refiner and other attention paths retain their original backend.
|
||||
n = q.shape[-2] if skip_reshape else q.shape[1]
|
||||
if n != ctx.times.shape[1] or k.shape[-2 if skip_reshape else 1] != n:
|
||||
if previous:
|
||||
return previous(original, q, k, v, heads, mask=mask, skip_reshape=skip_reshape,
|
||||
skip_output_reshape=skip_output_reshape, **kwargs)
|
||||
return original(q, k, v, heads, mask=mask, skip_reshape=skip_reshape,
|
||||
skip_output_reshape=skip_output_reshape, **kwargs)
|
||||
if mask is not None:
|
||||
raise ValueError('Native temporal attention does not support another attention mask override.')
|
||||
if not skip_reshape:
|
||||
q, k, v = [x.view(x.shape[0], x.shape[1], heads, -1).transpose(1, 2) for x in (q,k,v)]
|
||||
chosen = backend
|
||||
reason = None
|
||||
if backend == SOL_BACKEND:
|
||||
options = kwargs.get('transformer_options') or {}
|
||||
reason = sparse_patch.dense_reason(options, n, options.get('block_index'))
|
||||
if reason is None:
|
||||
import comfy_kitchen as ck
|
||||
if q.shape[-1] != 128 or not ck.sol_attn_is_available(q.device):
|
||||
reason = 'sol-attn requires head dimension 128 and an available CUDA kernel'
|
||||
if reason is not None:
|
||||
sparse_patch.log_once(('temporal_dense', n, reason), f'temporal dense fallback: {reason}')
|
||||
chosen = dense_backend
|
||||
timing=first[0]
|
||||
if timing:
|
||||
if reason is not None:
|
||||
print(f'[BFS H3 native temporal] sol-attn dense fallback: {reason}',flush=True)
|
||||
display_backend=chosen
|
||||
if chosen in (SOL_BACKEND,BACKENDS[2]) and sol.native_kernel.api_for(q) is not None:
|
||||
display_backend='sol-attn one-pass (native Kitchen)' if chosen==SOL_BACKEND else 'comfy kitchen one-pass (native)'
|
||||
print(f'[BFS H3 native temporal] first attention starting: backend={display_backend}, Q={tuple(q.shape)}, dtype={q.dtype}',flush=True)
|
||||
if q.is_cuda: torch.cuda.synchronize(q.device)
|
||||
started=time.perf_counter()
|
||||
if chosen == SOL_BACKEND:
|
||||
if not getattr(plan, '_sol_active_logged', False):
|
||||
passes = 1 if sol.native_kernel.api_for(q) is not None else (plan.frames+1 if ctx.strength and ctx.cap else 1)
|
||||
print(f'[BFS H3 native temporal] sol-attn ACTIVE: tau={sparse_patch.tau}, exact conditioning keys, {passes} attention pipeline pass(es).',flush=True)
|
||||
plan._sol_active_logged = True
|
||||
_, sink_q = sparse_patch.sinks(options, n)
|
||||
out = sol.attention(q,k,v,plan,ctx,tau=sparse_patch.tau,
|
||||
token_aug=sparse_patch.extra_tokens,sink_q=sink_q)
|
||||
elif chosen == BACKENDS[2]:
|
||||
out = kitchen.attention(q,k,v,plan,ctx)
|
||||
elif chosen == BACKENDS[0]:
|
||||
out = fast.attention(q,k,v,plan)
|
||||
else:
|
||||
out = core.attention(q,k,v,ctx)
|
||||
if timing:
|
||||
if q.is_cuda: torch.cuda.synchronize(q.device)
|
||||
print(f'[BFS H3 native temporal] first attention finished in {time.perf_counter()-started:.3f}s',flush=True)
|
||||
first[0]=False
|
||||
return out if skip_output_reshape else out.transpose(1, 2).reshape(out.shape[0], -1, out.shape[1]*out.shape[3])
|
||||
return override
|
||||
|
||||
|
||||
def patch_temporal_model(model, video_index=1, strength=2.0, window=0.2, falloff=0.25, cap=6.0, backend=BACKENDS[0]):
|
||||
if backend not in BACKENDS: raise ValueError("Unknown native temporal attention backend")
|
||||
if backend==BACKENDS[0] and not hasattr(torch.ops.aten, '_scaled_dot_product_flash_attention'):
|
||||
raise RuntimeError("This PyTorch build lacks native FlashAttention with LSE. Update PyTorch or select flex (original).")
|
||||
cfg = core.options(dict(native_temporal_alignment=True, native_temporal_strength=strength,
|
||||
native_temporal_window_seconds=window, native_temporal_falloff_seconds=falloff,
|
||||
native_temporal_max_penalty=cap))
|
||||
if isinstance(video_index, bool) or not isinstance(video_index, int) or video_index < 1:
|
||||
raise ValueError('motion_video_index must be a positive integer')
|
||||
if model is None:
|
||||
raise ValueError('Native temporal mode needs the MODEL input. Route this node MODEL output to the sampler/guider.')
|
||||
if model.model_options.get(MARKER):
|
||||
raise ValueError('Native temporal alignment is already installed. Use only one native temporal node per model path.')
|
||||
net = model.get_model_object('diffusion_model')
|
||||
if net.__class__.__name__ != 'MiniMaxH3Model':
|
||||
raise ValueError('Native temporal alignment requires a native ComfyUI MiniMaxH3 model.')
|
||||
opts = model.model_options.get('transformer_options', {})
|
||||
dense_override, sparse_patch = sparse_compat.selection(opts)
|
||||
selected_dense_at_setup=dense_backend_selection(dense_override)
|
||||
sparse_compat.without_owned_blocks(opts, sparse_patch)
|
||||
if backend==BACKENDS[2] or selected_dense_at_setup=='comfy_kitchen_int8' or sparse_patch is not None:
|
||||
sol.native_kernel.prepare_model(model)
|
||||
patched = model.clone()
|
||||
previous = patched.model_options.get('model_function_wrapper')
|
||||
cache = [None, None]
|
||||
logged = [False]
|
||||
plan_cache = [None, None]
|
||||
calls = [0]
|
||||
|
||||
def wrapper(model_function, args):
|
||||
conds = args.get('c', {})
|
||||
payload = conds.get('minimax_payload') or {}
|
||||
native = payload.get('layout')
|
||||
if native is None:
|
||||
raise RuntimeError('Native H3 layout missing. Update ComfyUI and use native AV latent conditioning.')
|
||||
if cache[0] is not native:
|
||||
cache[:] = [native, temporal_layout(native, payload.get('refs', []), payload.get('keyframes'), video_index)]
|
||||
layout, source, target = cache[1]
|
||||
device = args['input'][0].device if isinstance(args['input'], (list, tuple)) else args['input'].device
|
||||
to = dict(conds.get('transformer_options', {}))
|
||||
incoming = to.get('optimized_attention_override')
|
||||
dense_override, sparse_patch = sparse_compat.selection(to)
|
||||
selected_dense = dense_backend_selection(dense_override)
|
||||
active_backend = (BACKENDS[2] if selected_dense == 'comfy_kitchen_int8' else
|
||||
BACKENDS[0] if selected_dense == 'pytorch' else backend)
|
||||
dense_backend = active_backend
|
||||
if sparse_patch is not None:
|
||||
active_backend = SOL_BACKEND
|
||||
to = sparse_compat.without_owned_blocks(to, sparse_patch)
|
||||
times = layout.position_ids[:, 0].float()[None].to(device)/40.0
|
||||
ctx = core.TemporalContext(times, source[None].to(device), target[None].to(device),
|
||||
torch.ones_like(source[None], device=device), strength, window, falloff, cap,
|
||||
torch.tensor([strength,window,falloff,cap],device=device,dtype=torch.float32))
|
||||
if plan_cache[0] is not layout:
|
||||
plan_cache[:] = [layout, fast.make_plan(layout.position_ids,source,target,strength,window,falloff,cap)]
|
||||
to['optimized_attention_override'] = attention_override(ctx, previous=incoming, backend=active_backend, plan=plan_cache[1], log_first=not logged[0], sparse_patch=sparse_patch, dense_backend=dense_backend)
|
||||
copied = dict(args, c=dict(conds, minimax_payload=dict(payload, layout=layout), transformer_options=to))
|
||||
if not logged[0]:
|
||||
print(f'[BFS H3 native temporal] {core.VERSION}: <Video {video_index}>, {int(source.sum())} motion rows; temporal RoPE aligned; soft window={window}s strength={strength} cap={cap}; images remain native; Add Guide OFF; backend={active_backend}; packed_rows={layout.seq_len}', flush=True)
|
||||
if sparse_patch is not None:
|
||||
print('[BFS H3 native temporal] sol-attn uses the native one-pass extension when installed; otherwise frame-wise compatibility repeats work. Watch for Kitchen ONE-PASS sol kernel ACTIVE.', flush=True)
|
||||
logged[0] = True
|
||||
calls[0]+=1
|
||||
started=time.perf_counter()
|
||||
result=previous(model_function, copied) if previous else model_function(copied['input'], copied['timestep'], **copied['c'])
|
||||
if calls[0]<=2:
|
||||
if device.type=='cuda':torch.cuda.synchronize(device)
|
||||
print(f'[BFS H3 native temporal] model evaluation {calls[0]} finished in {time.perf_counter()-started:.2f}s (includes weights/offload and all blocks)',flush=True)
|
||||
return result
|
||||
|
||||
patched.set_model_unet_function_wrapper(wrapper)
|
||||
patched.model_options[MARKER] = dict(version=core.VERSION, video_index=video_index, backend=backend, **cfg)
|
||||
return patched
|
||||
|
||||
|
||||
class BFSMiniMaxH3NativeTemporal:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {'required': {'model': ('MODEL',),
|
||||
'motion_video_index': ('INT', {'default':1, 'min':1, 'max':100,
|
||||
'tooltip':'Native <Video N> supplying motion. Picture numbering is independent.'}),
|
||||
'strength': ('FLOAT', {'default':2.0,'min':0.0,'max':20.0,'step':0.1}),
|
||||
'window_seconds': ('FLOAT', {'default':0.2,'min':0.0,'max':10.0,'step':0.01}),
|
||||
'falloff_seconds': ('FLOAT', {'default':0.25,'min':0.01,'max':10.0,'step':0.01}),
|
||||
'max_penalty': ('FLOAT', {'default':6.0,'min':0.0,'max':30.0,'step':0.1})},
|
||||
'optional': {'attention_backend': (BACKENDS, {'default': BACKENDS[0], 'tooltip':'Flash partitions is the default. An official Model Attention Backend Kitchen selection is detected automatically. Kitchen uses an optional one-pass temporal extension when installed, otherwise frame-wise compatibility. Flex is the original implementation.'})}}
|
||||
RETURN_TYPES = ('MODEL',)
|
||||
FUNCTION = 'apply'
|
||||
CATEGORY = 'BFS/MiniMax H3'
|
||||
DESCRIPTION = ('Experimental native_temporal_v1 inference for the matching motion-transfer LoRA. '
|
||||
'Keep the motion video in MiniMax H3 Reference to Video positive conditioning; route this MODEL '
|
||||
'to the guider. No Add Guide or VSA. Guide/output must share a 24fps temporal grid. '
|
||||
'Does not encode video or load the LoRA. Defaults match training.')
|
||||
|
||||
def apply(self, model, motion_video_index=1, strength=2.0, window_seconds=0.2, falloff_seconds=0.25, max_penalty=6.0, attention_backend=BACKENDS[0]):
|
||||
return (patch_temporal_model(model,motion_video_index,strength,window_seconds,falloff_seconds,max_penalty,attention_backend),)
|
||||
|
||||
NODE_CLASS_MAPPINGS = {'BFSMiniMaxH3NativeTemporal': BFSMiniMaxH3NativeTemporal}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {'BFSMiniMaxH3NativeTemporal': 'BFS MiniMax H3 Native Temporal Alignment (Experimental)'}
|
||||
@@ -0,0 +1,122 @@
|
||||
"""Native-reference temporal alignment v1. No new learned model weights.
|
||||
|
||||
Only source channel 1's VIDEO latent rows are moved onto the target clock.
|
||||
Image-reference spatial/temporal positions remain native. Target-to-motion
|
||||
attention receives a bounded, soft temporal penalty; all other edges stay dense.
|
||||
The CUDA implementation uses FlexAttention, never a quadratic dense bias tensor.
|
||||
"""
|
||||
from dataclasses import dataclass
|
||||
import math
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
|
||||
VERSION = 'native_temporal_v1'
|
||||
_FLEX = None
|
||||
|
||||
def options(kwargs):
|
||||
enabled = kwargs.get('native_temporal_alignment', False)
|
||||
if not isinstance(enabled, bool):raise ValueError('native_temporal_alignment must be boolean')
|
||||
result = dict(enabled=enabled, source_channel=kwargs.get('native_motion_source_channel',1),
|
||||
strength=kwargs.get('native_temporal_strength',2.0),
|
||||
window=kwargs.get('native_temporal_window_seconds',0.2),
|
||||
falloff=kwargs.get('native_temporal_falloff_seconds',0.25),
|
||||
cap=kwargs.get('native_temporal_max_penalty',6.0))
|
||||
if isinstance(result['source_channel'],bool) or not isinstance(result['source_channel'],int) or result['source_channel']<1:raise ValueError('Invalid native motion source channel')
|
||||
for name in ['strength','window','falloff','cap']:
|
||||
x=result[name]
|
||||
if isinstance(x,bool) or not isinstance(x,(int,float)) or not math.isfinite(x) or x<0 or (name=='falloff' and x==0):raise ValueError('Invalid native temporal '+name)
|
||||
if enabled and (kwargs.get('align_video_refs') or kwargs.get('align_image_refs') or kwargs.get('reference_source_phase') or kwargs.get('reference_rope_layout','native')!='native'):
|
||||
raise ValueError('Native temporal mode requires native references, no AddGuide alignment or source-phase patch')
|
||||
return result
|
||||
|
||||
def align_layout(layout, ref_blocks, source_ids, cfg):
|
||||
"""Return a sequence-row mask, aligning only the explicit motion video's clock.
|
||||
|
||||
Native video latents remain in ref_blocks and keep their own spatial grid.
|
||||
Reference audio is not temporally biased by this video-only experiment.
|
||||
"""
|
||||
mask=torch.zeros(layout.sequence_length,dtype=torch.bool)
|
||||
if not cfg['enabled'] or not ref_blocks:return mask
|
||||
if len(source_ids)!=len(ref_blocks):raise ValueError('Native temporal mode requires exact reference source IDs')
|
||||
cursor=int(layout.text_indices.numel());found=0
|
||||
# target rows are always the final packed rows; condition video indices precede them.
|
||||
target_start=int(layout.audio_indices[-1])+1 if layout.audio_indices.numel() else layout.sequence_length
|
||||
# For zero audio use the count of conditioning video rows to locate the target.
|
||||
target_idx=layout.video_indices[layout.num_condition_video_rows:]
|
||||
target_start=int(target_idx[0]);target_origin=layout.position_ids[target_start,0].clone()
|
||||
for block, source in zip(ref_blocks,source_ids):
|
||||
t,h,w=block[:3];a=block[3] if len(block)>3 else 0
|
||||
cursor+=int(a)*2
|
||||
rows=int(t)*(int(h)//2)*(int(w)//2)
|
||||
if source==cfg['source_channel']:
|
||||
if t<=1:raise ValueError('Motion channel must be a video, not an image')
|
||||
if len(block)>4 and block[4]:raise ValueError('Native motion video was accidentally converted to aligned guide')
|
||||
found+=1;mask[cursor:cursor+rows]=True
|
||||
layout.position_ids=layout.position_ids.clone()
|
||||
layout.position_ids[cursor:cursor+rows,0]+=target_origin-layout.position_ids[cursor,0].clone()
|
||||
# A common temporal extent is required; never silently stretch motion to fit.
|
||||
src_times=torch.unique_consecutive(layout.position_ids[cursor:cursor+rows,0])
|
||||
dst_times=torch.unique_consecutive(layout.position_ids[target_idx,0])
|
||||
if src_times.shape!=dst_times.shape or not torch.allclose(src_times,dst_times,atol=1e-9,rtol=0):raise ValueError('Native motion and target must use identical temporal grids; resample both to matching frame windows')
|
||||
cursor+=rows
|
||||
if found!=1:raise ValueError(f'Expected one motion video in channel {cfg["source_channel"]}, found {found}')
|
||||
assert not mask[target_idx].any()
|
||||
return mask
|
||||
|
||||
def pad_motion_masks(layouts, masks):
|
||||
max_text=max(int(x.text_indices.numel()) for x in layouts)
|
||||
total=max_text+layouts[0].sequence_length-int(layouts[0].text_indices.numel())
|
||||
result=torch.zeros(len(layouts),total,dtype=torch.bool)
|
||||
for i,(layout,mask) in enumerate(zip(layouts,masks)):
|
||||
n=int(layout.text_indices.numel());result[i,:n]=mask[:n];result[i,max_text:]=mask[n:]
|
||||
return result
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TemporalContext:
|
||||
times: torch.Tensor
|
||||
source: torch.Tensor
|
||||
target: torch.Tensor
|
||||
live: torch.Tensor
|
||||
strength: float
|
||||
window: float
|
||||
falloff: float
|
||||
cap: float
|
||||
policy: torch.Tensor | None = None
|
||||
|
||||
def context(position_ids, source_mask, tags, video_indices, target_grid, cfg):
|
||||
if not cfg['enabled'] or source_mask is None or not bool(source_mask.any()):return None
|
||||
if source_mask.shape!=tags.shape:raise ValueError('Motion mask must match packed token rows')
|
||||
target=torch.zeros_like(source_mask);n=math.prod(target_grid)
|
||||
target[:,video_indices[-n:]]=True
|
||||
return TemporalContext(position_ids[:,:,0].float()/40.0,source_mask,target,tags>=0,
|
||||
float(cfg['strength']),float(cfg['window']),float(cfg['falloff']),float(cfg['cap']),
|
||||
torch.tensor([cfg['strength'],cfg['window'],cfg['falloff'],cfg['cap']],device=position_ids.device,dtype=torch.float32))
|
||||
|
||||
def score_modifier(ctx):
|
||||
# Captured tensors are O(sequence length). No SxS matrix is materialized on CUDA.
|
||||
times,source,target,live=ctx.times,ctx.source,ctx.target,ctx.live
|
||||
policy=ctx.policy if ctx.policy is not None else torch.tensor([ctx.strength,ctx.window,ctx.falloff,ctx.cap],device=times.device,dtype=torch.float32)
|
||||
def modify(score,b,h,q,k):
|
||||
qi=q.clamp(max=times.shape[1]-1);ki=k.clamp(max=times.shape[1]-1)
|
||||
dt=(times[b,qi]-times[b,ki]).abs()
|
||||
penalty=torch.minimum(policy[0]*(torch.relu(dt-policy[1])/policy[2]).square(),policy[3])
|
||||
edited=torch.where(target[b,qi]&source[b,ki],score-penalty,score)
|
||||
return torch.where(live[b,ki],edited,float('-inf'))
|
||||
return modify
|
||||
|
||||
def dense_reference(q,k,v,ctx):
|
||||
"""Tiny test oracle only; forbidden at production sequence lengths."""
|
||||
if q.shape[-2]>256:raise ValueError('Dense temporal reference is only for tiny tests')
|
||||
delta=(ctx.times[:,:,None]-ctx.times[:,None,:]).abs()
|
||||
penalty=(ctx.strength*(F.relu(delta-ctx.window)/ctx.falloff).square()).clamp(max=ctx.cap)
|
||||
bias=torch.where(ctx.target[:,:,None]&ctx.source[:,None,:],-penalty,0.0)
|
||||
bias=bias.masked_fill(~ctx.live[:,None,:],float('-inf'))
|
||||
return F.scaled_dot_product_attention(q,k,v,attn_mask=bias[:,None].to(q.dtype))
|
||||
|
||||
def attention(q,k,v,ctx):
|
||||
global _FLEX
|
||||
if q.device.type!='cuda':return dense_reference(q,k,v,ctx)
|
||||
if _FLEX is None:
|
||||
from torch.nn.attention.flex_attention import flex_attention
|
||||
_FLEX=torch.compile(flex_attention,dynamic=True,fullgraph=True)
|
||||
return _FLEX(q,k,v,score_mod=score_modifier(ctx))
|
||||
@@ -0,0 +1,60 @@
|
||||
"""Recognize only the official ComfyUI sol-attn wrappers and owned block hooks."""
|
||||
import inspect
|
||||
|
||||
|
||||
def official_sparse_module(obj):
|
||||
name = getattr(obj, '__module__', '')
|
||||
# ComfyUI's file loader uses the extensionless absolute path as the
|
||||
# module name for .py extras, rather than a dotted import name.
|
||||
leaf = name.replace('\\', '/').rsplit('/', 1)[-1]
|
||||
return leaf == 'nodes_sparse_attention' or leaf.endswith('.nodes_sparse_attention')
|
||||
|
||||
|
||||
def unwrap_sol(override):
|
||||
if override is None or not official_sparse_module(override):
|
||||
return override, None
|
||||
if not getattr(override, '__qualname__', '').endswith('make_attention_override.<locals>.override'):
|
||||
raise ValueError('Unsupported sparse attention wrapper for native temporal alignment.')
|
||||
cells = inspect.getclosurevars(override).nonlocals
|
||||
patch = cells.get('patch')
|
||||
if patch is None or type(patch).__name__ != 'SparseAttnPatch' or not official_sparse_module(type(patch)):
|
||||
raise ValueError('Unable to identify the official sparse attention patch.')
|
||||
if patch.vsa or patch.topk_ratio != 0:
|
||||
raise ValueError('Native temporal alignment supports only sol-attn; SLA and VSA remain incompatible.')
|
||||
parent = cells.get('previous')
|
||||
if parent is not None and official_sparse_module(parent):
|
||||
raise ValueError('Use only one Model Sparse Attention node with native temporal alignment.')
|
||||
return parent, patch
|
||||
|
||||
|
||||
def without_owned_blocks(options, patch):
|
||||
blocks = options.get('patches_replace', {}).get('dit', {})
|
||||
if not blocks:
|
||||
return options
|
||||
if patch is None:
|
||||
raise ValueError('Remove VSA and other attention/block replacement patches before native temporal alignment.')
|
||||
for callback in blocks.values():
|
||||
if (not official_sparse_module(callback) or
|
||||
not getattr(callback, '__qualname__', '').endswith('make_h3_block_patch.<locals>.block_patch') or
|
||||
inspect.getclosurevars(callback).nonlocals.get('patch') is not patch):
|
||||
raise ValueError('Native temporal alignment conflicts with another attention/block replacement patch.')
|
||||
# A per-evaluation copy bypasses only this node's producer hooks. That native
|
||||
# producer has no temporal Q/K bias; leaving it installed would bypass BFS.
|
||||
return dict(options, patches_replace={**options['patches_replace'], 'dit': {}})
|
||||
|
||||
|
||||
def selection(options):
|
||||
"""Recover sparse state even when a later dense node replaced its override."""
|
||||
dense, patch = unwrap_sol(options.get('optimized_attention_override'))
|
||||
if patch is None:
|
||||
for callback in options.get('patches_replace', {}).get('dit', {}).values():
|
||||
if (official_sparse_module(callback) and
|
||||
getattr(callback, '__qualname__', '').endswith('make_h3_block_patch.<locals>.block_patch')):
|
||||
candidate = inspect.getclosurevars(callback).nonlocals.get('patch')
|
||||
if candidate is not None and type(candidate).__name__ == 'SparseAttnPatch' and official_sparse_module(type(candidate)):
|
||||
if candidate.vsa or candidate.topk_ratio != 0:
|
||||
raise ValueError('Native temporal alignment supports only sol-attn; SLA and VSA remain incompatible.')
|
||||
patch = candidate
|
||||
break
|
||||
without_owned_blocks(options, patch)
|
||||
return dense, patch
|
||||
@@ -0,0 +1,152 @@
|
||||
"""Pinned, node-local bootstrap for the optional native temporal extension.
|
||||
|
||||
Only fixed trusted-release bytes are loaded, after embedded SHA256 checks.
|
||||
No pip, site-packages writes, main Kitchen replacement, or sampling-time build.
|
||||
"""
|
||||
import ctypes
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
SOURCE_COMMIT = '9953b6caec7a5cb3e2486dd97a365f7af44eedad'
|
||||
RAW = f'https://raw.githubusercontent.com/alisson-anjos/comfy-kitchen/{SOURCE_COMMIT}'
|
||||
RELEASE = 'https://github.com/alisson-anjos/comfy-kitchen/releases/download/h3-native-temporal-kernel-v1'
|
||||
BINARY_HASH = 'ad87534bfb3ece2af84c201e326b2b405f0108f34efb36937c09967c14f0329d'
|
||||
ASSETS = (
|
||||
('temporal_attention.py', RAW+'/comfy_kitchen/temporal_attention.py', '0151e8b5b5886c9e8b4608cbc179e9dd6f934170cf45b8d27e14801f3c4828a1', 256*1024),
|
||||
('backends/cuda/native_temporal/libck_temporal_sm120.so', RELEASE+'/libck_temporal_sm120.so', BINARY_HASH, 16*1024*1024),
|
||||
('LICENSE', RAW+'/LICENSE', 'cd1c9ea1573c9047e9b5a3676ecc1c8f755e56f249ac1ebd8b5ceda7899037da', 64*1024),
|
||||
('NOTICE', RAW+'/NOTICE', '35c43127ad3acbd6bc94ff63c56684a7c26eedf07795e596c8105a0476dae2ea', 64*1024),
|
||||
)
|
||||
_LOCK = threading.Lock()
|
||||
_READY = {}
|
||||
_FAILED = set()
|
||||
|
||||
|
||||
def cache_root():
|
||||
explicit=os.environ.get('BFS_H3_TEMPORAL_CACHE_DIR')
|
||||
if explicit:
|
||||
base=Path(explicit).expanduser()
|
||||
else:
|
||||
try:
|
||||
import folder_paths
|
||||
base=Path(folder_paths.get_user_directory())/'bfs'/'h3_temporal_kernels'
|
||||
except (ImportError,AttributeError):
|
||||
base=Path.home()/'.cache'/'bfsnodes'/'h3_temporal_kernels'
|
||||
return base/('v1-linux-x86_64-sm120-'+BINARY_HASH[:12])
|
||||
|
||||
|
||||
def compatible(device):
|
||||
if os.environ.get('BFS_H3_TEMPORAL_AUTO_INSTALL','1').lower() in ('0','false','off'):
|
||||
return False,'automatic setup disabled'
|
||||
if platform.system()!='Linux' or platform.machine() not in ('x86_64','AMD64'):
|
||||
return False,'automatic prebuilt supports Linux x86_64 only'
|
||||
import torch
|
||||
if getattr(torch.version,'hip',None) or not torch.cuda.is_available() or torch.cuda.get_device_capability(device)!=(12,0):
|
||||
return False,'automatic prebuilt supports NVIDIA SM120 only'
|
||||
# Check the actual driver, not torch.version.cuda: the sidecar uses its own
|
||||
# static CUDA runtime and the same device/stream primary context.
|
||||
driver=ctypes.CDLL('libcuda.so.1');version=ctypes.c_int()
|
||||
if driver.cuDriverGetVersion(ctypes.byref(version))!=0 or version.value<13000:
|
||||
return False,'the prebuilt requires a driver supporting CUDA 13'
|
||||
import comfy_kitchen
|
||||
if not comfy_kitchen.int8_attention_is_available(device):
|
||||
return False,'Kitchen native INT8 attention unavailable'
|
||||
from comfy_kitchen.sage_attention import prequantize_int8_attention
|
||||
if not callable(prequantize_int8_attention):
|
||||
return False,'Kitchen prequantization API unavailable'
|
||||
return True,None
|
||||
|
||||
|
||||
def digest(path):
|
||||
h=hashlib.sha256()
|
||||
with path.open('rb') as f:
|
||||
for block in iter(lambda:f.read(1024*1024),b''):h.update(block)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def download(path,url,expected,limit):
|
||||
if path.is_file() and digest(path)==expected:
|
||||
return
|
||||
path.parent.mkdir(parents=True,exist_ok=True)
|
||||
fd,name=tempfile.mkstemp(prefix=path.name+'.',suffix='.partial',dir=path.parent)
|
||||
temp=Path(name)
|
||||
try:
|
||||
request=urllib.request.Request(url,headers={'User-Agent':'BFSNodes-H3-Temporal/1'})
|
||||
h=hashlib.sha256();total=0
|
||||
with os.fdopen(fd,'wb') as out,urllib.request.urlopen(request,timeout=20) as response:
|
||||
while True:
|
||||
block=response.read(256*1024)
|
||||
if not block:break
|
||||
total+=len(block)
|
||||
if total>limit:raise RuntimeError('Temporal asset exceeds the pinned size limit')
|
||||
h.update(block);out.write(block)
|
||||
if h.hexdigest()!=expected:
|
||||
raise RuntimeError('Temporal asset checksum mismatch; downloaded code was not loaded')
|
||||
os.replace(temp,path)
|
||||
finally:
|
||||
if temp.exists():temp.unlink()
|
||||
|
||||
|
||||
def provision(root):
|
||||
import fcntl
|
||||
root.mkdir(parents=True,exist_ok=True)
|
||||
with (root/'.install.lock').open('a+b') as lock:
|
||||
deadline=time.monotonic()+60
|
||||
while True:
|
||||
try:
|
||||
fcntl.flock(lock,fcntl.LOCK_EX|fcntl.LOCK_NB);break
|
||||
except BlockingIOError:
|
||||
if time.monotonic()>deadline:raise RuntimeError('Timed out waiting for temporal setup lock')
|
||||
time.sleep(.1)
|
||||
missing=any(not (root/rel).is_file() or digest(root/rel)!=sha for rel,_,sha,_ in ASSETS)
|
||||
if missing:
|
||||
print('[BFS H3 native temporal] First-use setup: downloading the verified one-pass extension (~3.3 MB). No compilation or pip install.',flush=True)
|
||||
for rel,url,sha,limit in ASSETS:
|
||||
download(root/rel,url,sha,limit)
|
||||
# Verify the native binary before executing the Python entry point too.
|
||||
binary=root/ASSETS[1][0];lib=ctypes.CDLL(str(binary))
|
||||
lib.ck_temporal_abi_version.restype=ctypes.c_int;lib.ck_temporal_arch.restype=ctypes.c_int
|
||||
if lib.ck_temporal_abi_version()!=1 or lib.ck_temporal_arch()!=120:
|
||||
raise RuntimeError('Temporal prebuilt ABI/architecture mismatch')
|
||||
name='_bfs_ck_temporal_v1_sm120_'+ASSETS[0][2][:12]+'_'+hashlib.sha256(str(root.resolve()).encode()).hexdigest()[:12]
|
||||
if name in sys.modules:
|
||||
return sys.modules[name]
|
||||
spec=importlib.util.spec_from_file_location(name,root/'temporal_attention.py')
|
||||
module=importlib.util.module_from_spec(spec);sys.modules[name]=module
|
||||
try:spec.loader.exec_module(module)
|
||||
except BaseException:
|
||||
sys.modules.pop(name,None);raise
|
||||
if getattr(module,'VERSION',None)!='h3_native_temporal_kernel_v1':
|
||||
sys.modules.pop(name,None);raise RuntimeError('Temporal API version mismatch')
|
||||
print(f'[BFS H3 native temporal] Verified one-pass extension ready: {root}',flush=True)
|
||||
return module
|
||||
|
||||
|
||||
def ensure(device):
|
||||
"""At most one automatic setup attempt per process/cache location."""
|
||||
key=(str(cache_root()),str(device))
|
||||
with _LOCK:
|
||||
if key in _READY:return _READY[key]
|
||||
if key in _FAILED:return None
|
||||
try:
|
||||
ok,reason=compatible(device)
|
||||
if not ok:
|
||||
_FAILED.add(key)
|
||||
print(f'[BFS H3 native temporal] Automatic native setup skipped: {reason}; retaining the existing backend.',flush=True)
|
||||
return None
|
||||
module=provision(Path(key[0]))
|
||||
if not module.is_available(device):raise RuntimeError('Verified native extension is unavailable for this device')
|
||||
_READY[key]=module
|
||||
return module
|
||||
except Exception as exc:
|
||||
_FAILED.add(key)
|
||||
print(f'[BFS H3 native temporal] Automatic native setup unavailable: {exc}. Retaining the existing backend. Restart ComfyUI to retry, or install the extension manually.',flush=True)
|
||||
return None
|
||||
@@ -0,0 +1,102 @@
|
||||
"""Exact bounded temporal bias through FlashAttention partitions; inference only.
|
||||
|
||||
The guide's weights are exp(-cap) globally plus positive local corrections
|
||||
exp(-penalty)-exp(-cap). Other keys have weight 1. Merge partitions by logsumexp.
|
||||
No compiled score modifier, dense score matrix, or subtraction of attention sums.
|
||||
"""
|
||||
from dataclasses import dataclass
|
||||
import math
|
||||
import torch
|
||||
|
||||
|
||||
@dataclass
|
||||
class Plan:
|
||||
source_start: int
|
||||
source_stop: int
|
||||
target_start: int
|
||||
target_stop: int
|
||||
source_frame_rows: int
|
||||
target_frame_rows: int
|
||||
frames: int
|
||||
cap: float
|
||||
corrections: list
|
||||
frame_times: tuple = ()
|
||||
|
||||
|
||||
def make_plan(positions, source, target, strength, window, falloff, cap):
|
||||
si=torch.where(source)[0];ti=torch.where(target)[0]
|
||||
sa,sb=int(si[0]),int(si[-1])+1;ta,tb=int(ti[0]),int(ti[-1])+1
|
||||
if si.numel()!=sb-sa or ti.numel()!=tb-ta:
|
||||
raise ValueError('Fast temporal attention requires contiguous source and target video rows')
|
||||
st,sc=torch.unique_consecutive(positions[sa:sb,0],return_counts=True)
|
||||
tt,tc=torch.unique_consecutive(positions[ta:tb,0],return_counts=True)
|
||||
if not torch.equal(sc,sc[:1].expand_as(sc)) or not torch.equal(tc,tc[:1].expand_as(tc)):
|
||||
raise ValueError('Fast temporal attention requires a fixed spatial grid per video')
|
||||
if st.shape!=tt.shape or not torch.allclose(st,tt,rtol=0,atol=1e-9):
|
||||
raise ValueError('Fast temporal attention requires matching temporal grids')
|
||||
frames=st.numel();corrections=[]
|
||||
if strength>0 and cap>0:
|
||||
times=st.double()/40
|
||||
for offset in range(1-frames,frames):
|
||||
lo=max(0,-offset);hi=min(frames,frames-offset)
|
||||
dt=(times[lo:hi]-times[lo+offset:hi+offset]).abs()
|
||||
penalty=(strength*(torch.relu(dt-window)/falloff).square()).clamp(max=cap)
|
||||
# Stable log(exp(-penalty)-exp(-cap)); zero correction becomes -inf.
|
||||
logs=-penalty+torch.log(-torch.expm1(penalty-cap))
|
||||
if torch.isfinite(logs).any():corrections.append((offset,lo,hi,logs.float()))
|
||||
return Plan(sa,sb,ta,tb,int(sc[0]),int(tc[0]),frames,cap if strength>0 else 0.,corrections,tuple((st.float()/40).tolist()))
|
||||
|
||||
|
||||
def flash_with_lse(q,k,v):
|
||||
# PyTorch's native FlashAttention primitive returns natural-log normalizers.
|
||||
if q.device.type!='cuda' or q.dtype not in (torch.float16,torch.bfloat16):
|
||||
raise RuntimeError('Fast temporal attention requires CUDA FP16/BF16. Choose flex backend for other configurations.')
|
||||
result=torch.ops.aten._scaled_dot_product_flash_attention.default(q,k,v,0.0,False,False)
|
||||
return result[0],result[1]
|
||||
|
||||
|
||||
def merge(a,la,b,lb):
|
||||
"""Stable positive weighted merge. Outputs retain model dtype; LSE stays FP32."""
|
||||
total=torch.logaddexp(la,lb)
|
||||
result=a.float()*torch.exp(la-total)[...,None]+b.float()*torch.exp(lb-total)[...,None]
|
||||
return result.to(a.dtype),total
|
||||
|
||||
|
||||
def attention(q,k,v,plan,flash=flash_with_lse,frame_chunk=8):
|
||||
if q.shape[0]!=1 or q.shape[2]!=k.shape[2] or q.shape!=v.shape:
|
||||
raise ValueError('Fast native temporal attention currently requires one packed self-attention sequence')
|
||||
if plan.cap==0:
|
||||
return flash(q,k,v)[0]
|
||||
out=torch.empty_like(q)
|
||||
sa,sb,ta,tb=plan.source_start,plan.source_stop,plan.target_start,plan.target_stop
|
||||
# Queries outside the target retain ordinary attention, including token refs/audio.
|
||||
for lo,hi in ((0,ta),(tb,q.shape[2])):
|
||||
if hi>lo:out[:,:,lo:hi]=flash(q[:,:,lo:hi],k,v)[0]
|
||||
heads,dim=q.shape[1],q.shape[-1]
|
||||
qframes=q[:,:,ta:tb].reshape(1,heads,plan.frames,plan.target_frame_rows,dim)[0].permute(1,0,2,3)
|
||||
kframes=k[:,:,sa:sb].reshape(1,heads,plan.frames,plan.source_frame_rows,dim)[0].permute(1,0,2,3)
|
||||
vframes=v[:,:,sa:sb].reshape(1,heads,plan.frames,plan.source_frame_rows,dim)[0].permute(1,0,2,3)
|
||||
if getattr(plan, '_correction_device', None) != q.device:
|
||||
plan._device_corrections=[(off,lo,hi,logs.to(device=q.device)) for off,lo,hi,logs in plan.corrections]
|
||||
plan._correction_device=q.device
|
||||
corrections=plan._device_corrections
|
||||
for first in range(0,plan.frames,frame_chunk):
|
||||
last=min(plan.frames,first+frame_chunk)
|
||||
start=ta+first*plan.target_frame_rows;stop=ta+last*plan.target_frame_rows
|
||||
query=q[:,:,start:stop]
|
||||
# Positive baseline: all motion keys downweighted by exp(-cap).
|
||||
acc,lse=flash(query,k[:,:,sa:sb],v[:,:,sa:sb]);lse=lse-plan.cap
|
||||
for lo,hi in ((0,sa),(sb,k.shape[2])):
|
||||
if hi>lo:
|
||||
part,pl=flash(query,k[:,:,lo:hi],v[:,:,lo:hi]);acc,lse=merge(acc,lse,part,pl)
|
||||
acc=acc.reshape(1,heads,last-first,plan.target_frame_rows,dim)[0].permute(1,0,2,3)
|
||||
lse=lse.reshape(1,heads,last-first,plan.target_frame_rows)[0].permute(1,0,2)
|
||||
for off,lo,hi,logs in corrections:
|
||||
a=max(first,lo);b=min(last,hi)
|
||||
if b<=a:continue
|
||||
part,pl=flash(qframes[a:b],kframes[a+off:b+off],vframes[a+off:b+off])
|
||||
pl=pl+logs[a-lo:b-lo,None,None]
|
||||
merged,ml=merge(acc[a-first:b-first],lse[a-first:b-first],part,pl)
|
||||
acc[a-first:b-first]=merged;lse[a-first:b-first]=ml
|
||||
out[:,:,start:stop]=acc.permute(1,0,2,3).reshape(1,heads,stop-start,dim)
|
||||
return out
|
||||
@@ -0,0 +1,57 @@
|
||||
"""Native Kitchen extension: existing installation or verified first-use cache."""
|
||||
import importlib
|
||||
try:
|
||||
from . import minimax_h3_temporal_bootstrap as bootstrap
|
||||
except ImportError:
|
||||
import minimax_h3_temporal_bootstrap as bootstrap
|
||||
|
||||
|
||||
def api_for(q):
|
||||
if q.device.type != 'cuda':
|
||||
return None
|
||||
try:
|
||||
api = importlib.import_module('comfy_kitchen.temporal_attention')
|
||||
except ImportError:
|
||||
return bootstrap.ensure(q.device)
|
||||
if getattr(api, 'VERSION', None) != 'h3_native_temporal_kernel_v1':
|
||||
raise RuntimeError('Unsupported Comfy Kitchen temporal extension version; update the temporal overlay.')
|
||||
return api if api.is_available(q.device) else bootstrap.ensure(q.device)
|
||||
|
||||
|
||||
def policy_for(api, plan, ctx):
|
||||
key=(api.VERSION,ctx.strength,ctx.window,ctx.falloff,ctx.cap)
|
||||
if getattr(plan,'_native_temporal_policy_key',None)!=key:
|
||||
times = getattr(plan, 'frame_times', ())
|
||||
if not times:
|
||||
times = tuple(ctx.times[0, plan.target_start:plan.target_stop:plan.target_frame_rows].detach().cpu().tolist())
|
||||
plan._native_temporal_policy = api.TemporalPolicy(
|
||||
plan.source_start, plan.source_stop, plan.target_start, plan.target_stop,
|
||||
plan.source_frame_rows, plan.target_frame_rows, tuple(times),
|
||||
ctx.strength, ctx.window, ctx.falloff, ctx.cap)
|
||||
plan._native_temporal_policy_key=key
|
||||
return plan._native_temporal_policy
|
||||
|
||||
|
||||
def run(kind, q, k, v, plan, ctx, **kwargs):
|
||||
api = api_for(q)
|
||||
if api is None:
|
||||
return None
|
||||
policy = policy_for(api, plan, ctx)
|
||||
if not getattr(plan, '_native_'+kind+'_logged', False):
|
||||
print(f'[BFS H3 native temporal] Kitchen ONE-PASS {kind} kernel ACTIVE; temporal bias fused; no per-frame attention loop.', flush=True)
|
||||
setattr(plan, '_native_'+kind+'_logged', True)
|
||||
dtype=q.dtype
|
||||
if str(dtype)=='torch.float32':
|
||||
import torch
|
||||
q,k,v=[t.to(torch.bfloat16) for t in (q,k,v)]
|
||||
fn = api.int8_attention if kind == 'dense' else api.sol_attn
|
||||
return fn(q,k,v,policy,**kwargs).to(dtype)
|
||||
|
||||
|
||||
def prepare_model(model):
|
||||
"""Prepare a selected native backend during node execution, before sampling."""
|
||||
device=getattr(model,'load_device',None)
|
||||
if getattr(device,'type',None)=='cuda':
|
||||
from types import SimpleNamespace
|
||||
return api_for(SimpleNamespace(device=device))
|
||||
return None
|
||||
@@ -0,0 +1,58 @@
|
||||
"""Prefer native one-pass Kitchen kernels, with a frame-wise fallback.
|
||||
|
||||
The fallback avoids partition LSE.
|
||||
|
||||
Each target frame shares a key bias across its spatial queries. The compact
|
||||
bias is O(frames * sequence), never an expanded sequence-by-sequence mask.
|
||||
INT8 quantization changes numerical results; GPU performance needs measurement.
|
||||
"""
|
||||
import inspect
|
||||
import torch
|
||||
try:
|
||||
from . import minimax_h3_temporal_kernel as native_kernel
|
||||
except ImportError:
|
||||
import minimax_h3_temporal_kernel as native_kernel
|
||||
|
||||
|
||||
def kitchen_function():
|
||||
try:
|
||||
import comfy_kitchen
|
||||
except ImportError as exc:
|
||||
raise RuntimeError('Comfy Kitchen temporal attention requires comfy-kitchen with masked INT8 attention.') from exc
|
||||
fn = getattr(comfy_kitchen, 'int8_attention', None)
|
||||
if fn is None or 'attn_mask' not in inspect.signature(fn).parameters:
|
||||
raise RuntimeError('Update comfy-kitchen: this temporal backend requires int8_attention(attn_mask=...).')
|
||||
return fn
|
||||
|
||||
|
||||
def attention(q, k, v, plan, ctx, kernel=None):
|
||||
if q.shape[0] != 1 or q.shape != k.shape or q.shape != v.shape:
|
||||
raise ValueError('Kitchen temporal attention requires one packed self-attention sequence')
|
||||
if kernel is None:
|
||||
native = native_kernel.run('dense',q,k,v,plan,ctx)
|
||||
if native is not None:
|
||||
return native
|
||||
kernel = kitchen_function() if kernel is None else kernel
|
||||
if ctx.strength == 0 or ctx.cap == 0:
|
||||
return kernel(q, k, v)
|
||||
sa, sb, ta, tb = plan.source_start, plan.source_stop, plan.target_start, plan.target_stop
|
||||
key = (q.device, q.shape[2], float(ctx.strength), float(ctx.window), float(ctx.falloff), float(ctx.cap))
|
||||
if getattr(plan, '_kitchen_key', None) != key:
|
||||
frame_times = ctx.times[0, ta:tb:plan.target_frame_rows]
|
||||
source_times = ctx.times[0, sa:sb]
|
||||
dt = (frame_times[:, None] - source_times[None, :]).abs()
|
||||
penalty = (ctx.strength * (torch.relu(dt-ctx.window)/ctx.falloff).square()).clamp(max=ctx.cap)
|
||||
biases = torch.zeros((plan.frames, q.shape[2]), device=q.device, dtype=torch.float32)
|
||||
biases[:, sa:sb] = -penalty
|
||||
plan._kitchen_biases, plan._kitchen_key = biases, key
|
||||
out = torch.empty_like(q)
|
||||
for lo, hi in ((0, ta), (tb, q.shape[2])):
|
||||
if hi > lo:
|
||||
out[:, :, lo:hi] = kernel(q[:, :, lo:hi], k, v)
|
||||
for frame in range(plan.frames):
|
||||
lo = ta + frame * plan.target_frame_rows
|
||||
hi = lo + plan.target_frame_rows
|
||||
# Broadcast only: Kitchen expands this key bias with zero query stride.
|
||||
mask = plan._kitchen_biases[frame].view(1, 1, 1, -1)
|
||||
out[:, :, lo:hi] = kernel(q[:, :, lo:hi], k, v, attn_mask=mask)
|
||||
return out
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Prefer the optional one-pass temporal Sol kernel; preserve legacy fallback.
|
||||
|
||||
Without that extension, the public kernel accepts equal-length Q/K/V and a key-only bias. Therefore
|
||||
this compatibility path runs one unbiased pass and one full-sequence pass per
|
||||
latent target frame, retaining only that frame's query rows. It is deliberately
|
||||
not advertised as a fast one-pass temporal sparse kernel.
|
||||
"""
|
||||
import inspect
|
||||
import torch
|
||||
try:
|
||||
from . import minimax_h3_temporal_kernel as native_kernel
|
||||
except ImportError:
|
||||
import minimax_h3_temporal_kernel as native_kernel
|
||||
|
||||
|
||||
def sol_function():
|
||||
try:
|
||||
import comfy_kitchen as ck
|
||||
except ImportError as exc:
|
||||
raise RuntimeError('Temporal sol-attn requires Comfy Kitchen.') from exc
|
||||
fn = getattr(ck, 'sol_attn', None)
|
||||
if fn is None or 'key_bias' not in inspect.signature(fn).parameters:
|
||||
raise RuntimeError('Update comfy-kitchen: temporal sol-attn requires sol_attn(key_bias=...).')
|
||||
return fn
|
||||
|
||||
|
||||
def attention(q, k, v, plan, ctx, *, tau, token_aug=0, sink_q=(0, 0), kernel=None):
|
||||
if q.shape[0] != 1 or q.shape != k.shape or q.shape != v.shape or q.shape[-1] != 128:
|
||||
raise ValueError('Temporal sol-attn requires one packed self-attention sequence with head dimension 128.')
|
||||
if plan.source_stop > plan.target_start:
|
||||
raise ValueError('Temporal sol-attn requires native motion references before the target video.')
|
||||
if kernel is None:
|
||||
native = native_kernel.run('sol',q,k,v,plan,ctx,tau=tau,token_aug=token_aug,sink_q=sink_q)
|
||||
if native is not None:
|
||||
return native
|
||||
fn = sol_function() if kernel is None else kernel
|
||||
# Sol's key bias is applied only in the exact branch: all conditioning keys,
|
||||
# including the selected motion video, must be forced exact regardless of
|
||||
# the ordinary sparse node's sink_conditioning setting.
|
||||
sink_blocks = [0, (plan.target_start+63)//64]
|
||||
common = dict(tau=float(tau), sink_blocks=sink_blocks, topk_ratio=0., tail=True, token_aug=int(token_aug))
|
||||
qs, ks, vs = [x.transpose(1, 2) for x in (q, k, v)]
|
||||
if kernel is None and q.dtype == torch.float32:
|
||||
qs, ks, vs = [x.to(torch.bfloat16) for x in (qs, ks, vs)]
|
||||
out = fn(qs, ks, vs, sink_q=list(sink_q), **common).transpose(1, 2).clone()
|
||||
if ctx.strength == 0 or ctx.cap == 0:
|
||||
return out.to(q.dtype)
|
||||
cache_key = (q.device, q.shape[2], float(ctx.strength), float(ctx.window), float(ctx.falloff), float(ctx.cap))
|
||||
if getattr(plan, '_sol_bias_key', None) != cache_key:
|
||||
ft = ctx.times[0, plan.target_start:plan.target_stop:plan.target_frame_rows]
|
||||
st = ctx.times[0, plan.source_start:plan.source_stop]
|
||||
penalty = (ctx.strength*(torch.relu((ft[:, None]-st[None, :]).abs()-ctx.window)/ctx.falloff).square()).clamp(max=ctx.cap)
|
||||
bias = torch.zeros((plan.frames, q.shape[2]), device=q.device, dtype=torch.float32)
|
||||
bias[:, plan.source_start:plan.source_stop] = -penalty
|
||||
plan._sol_bias, plan._sol_bias_key = bias, cache_key
|
||||
for frame in range(plan.frames):
|
||||
lo = plan.target_start+frame*plan.target_frame_rows
|
||||
hi = lo+plan.target_frame_rows
|
||||
# Outputs for other frames are discarded: their temporal bias differs.
|
||||
result = fn(qs, ks, vs, key_bias=plan._sol_bias[frame], sink_q=[0, 0], **common)
|
||||
out[:, :, lo:hi] = result[:, lo:hi].transpose(1, 2)
|
||||
return out.to(q.dtype)
|
||||
@@ -1,7 +1,7 @@
|
||||
[project]
|
||||
name = "bfsnodes"
|
||||
description = "Custom ComfyUI nodes for reference-guided video: BFS Sampler (guide-driven IC-LoRAs, auto crop, mask inpainting, temporal chunking), LTX Multiple Controls / Identity Transfer / Edit Anything, Head-Swap and Anime2Real conditioning, MiniMax-H3 latent guides, RoPE layouts and loaders, LoRA Surgery, and Shot Planner / Shot Loop (tabbed panel): split long videos into shots at camera cuts, face-tracked cast, SAM 3 mask per shot with full frame, paste, mask, crop and crop + mask generation modes, optional translucent Colored Mark overlay for manually selected regions, optional Qwen3-VL to suggest settings, describe references ({details}) and the person to replace ({target}), continuity between shots, H3 conditioning with an optional duet and prompt writer, join with the original timing, audio and a side-by-side comparison. H3 Duet: pin a source clip beside the video, training-free."
|
||||
version = "1.107.0"
|
||||
version = "1.115.18"
|
||||
license = {file = "LICENSE"}
|
||||
dependencies = [
|
||||
"torch",
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
# Extracted from official ComfyUI bef11fc nodes_sparse_attention.py for loader/closure compatibility tests.
|
||||
# SPDX-License-Identifier: GPL-3.0-only
|
||||
class SparseAttnPatch:
|
||||
"""Options plus runtime state for one patched model; an ON_CLEANUP callback
|
||||
resets the state when the sampling run ends."""
|
||||
|
||||
def __init__(self, tau, topk_ratio, vsa, sigma_start, sigma_end, min_tokens, dense_blocks, sink_conditioning, extra_tokens, verbose):
|
||||
self.tau = tau
|
||||
self.topk_ratio = topk_ratio
|
||||
self.extra_tokens = extra_tokens
|
||||
self.vsa = vsa
|
||||
self.sigma_start = sigma_start
|
||||
self.sigma_end = sigma_end
|
||||
self.min_tokens = min_tokens
|
||||
self.dense_blocks = dense_blocks
|
||||
self.sink_conditioning = sink_conditioning
|
||||
self.verbose = verbose
|
||||
self.installed = set()
|
||||
self.reset()
|
||||
|
||||
def reset(self):
|
||||
self.pooled = {}
|
||||
self.vsa_plans = {}
|
||||
self.vsa_rope = None
|
||||
self._logged = set()
|
||||
|
||||
def log_once(self, key, message):
|
||||
if self.verbose and key not in self._logged:
|
||||
self._logged.add(key)
|
||||
logging.info(f'BlockSparseAttention: {message}')
|
||||
|
||||
def dense_reason(self, transformer_options, tokens, block_index):
|
||||
"""Why this call stays dense regardless of its tensors, or None."""
|
||||
sigmas = transformer_options.get('sigmas')
|
||||
if sigmas is not None:
|
||||
sigma = float(sigmas[0])
|
||||
if sigma > self.sigma_start or sigma < self.sigma_end:
|
||||
return f'sigma {sigma:.3g} outside the start/end window'
|
||||
if tokens < self.min_tokens:
|
||||
return f'{tokens} tokens < min_tokens {self.min_tokens}'
|
||||
if self.dense_blocks:
|
||||
if block_index is None:
|
||||
self.log_once('no_block_index', 'this model does not report block indices; dense_blocks ignored')
|
||||
elif block_index in self.dense_blocks:
|
||||
return f'block {block_index} in dense_blocks'
|
||||
return None
|
||||
|
||||
def sinks(self, transformer_options, tokens):
|
||||
"""MiniMax-H3 conditioning rows as (exact-KV blocks, dense-query blocks):
|
||||
the packed prefix stays exact for every query, and optionally the
|
||||
target-audio query rows run dense."""
|
||||
layout = transformer_options.get('minimax_h3_layout')
|
||||
if self.sink_conditioning == 'off' or layout is None or layout.seq_len != tokens:
|
||||
return ((0, 0), (0, 0))
|
||||
video = next(((a, b) for a, b, kind in layout.segments if kind == 'video'), None)
|
||||
if video is None or video[0] <= 0:
|
||||
return ((0, 0), (0, 0))
|
||||
blocks = (0, (video[0] + BLOCK_SIZE - 1) // BLOCK_SIZE)
|
||||
if self.sink_conditioning != 'exact_kv_and_rows':
|
||||
return (blocks, (0, 0))
|
||||
audio = next(((a, b) for a, b, kind in layout.segments if kind == 'audio'), None)
|
||||
if audio is None:
|
||||
return (blocks, blocks)
|
||||
return (blocks, (audio[0] // BLOCK_SIZE, blocks[1]))
|
||||
|
||||
def vsa_plan(self, layout, device):
|
||||
"""Padded tile order: prefix segments in their own zero-padded 64-row
|
||||
tiles, video in 4x4x4 cubes. `src` maps padded row -> source row (-1 =
|
||||
pad), `inv` the reverse."""
|
||||
key = (tuple(layout.signature), tuple(layout.segments), str(device))
|
||||
plan = self.vsa_plans.get(key)
|
||||
if plan is not None:
|
||||
return plan
|
||||
_text_len, latent_t, latent_h, latent_w, _audio_t = layout.signature
|
||||
grid = (int(latent_t), int(latent_h) // 2, int(latent_w) // 2)
|
||||
tiles, n_prefix = ([], 0)
|
||||
for a, b, kind in layout.segments:
|
||||
n = b - a
|
||||
if kind != 'video':
|
||||
m = (n + BLOCK_SIZE - 1) // BLOCK_SIZE
|
||||
seg = torch.full((m * BLOCK_SIZE,), -1, dtype=torch.int64, device=device)
|
||||
seg[:n] = torch.arange(a, b, device=device)
|
||||
tiles.append(seg.view(m, BLOCK_SIZE))
|
||||
n_prefix += m
|
||||
continue
|
||||
if grid[0] * grid[1] * grid[2] != n:
|
||||
raise RuntimeError(f'VSA: video segment of {n} rows does not match the latent grid {grid}')
|
||||
ct, ch, cw = VSA_CUBE
|
||||
pt, ph, pw = ((g + c - 1) // c * c for g, c in zip(grid, VSA_CUBE))
|
||||
padded = torch.full((pt, ph, pw), -1, dtype=torch.int64, device=device)
|
||||
padded[:grid[0], :grid[1], :grid[2]] = torch.arange(a, b, device=device).view(*grid)
|
||||
cubes = padded.view(pt // ct, ct, ph // ch, ch, pw // cw, cw).permute(0, 2, 4, 1, 3, 5).reshape(-1, BLOCK_SIZE)
|
||||
order = torch.argsort((cubes < 0).to(torch.int8), dim=1, stable=True)
|
||||
tiles.append(torch.gather(cubes, 1, order))
|
||||
tiles = torch.cat(tiles)
|
||||
src = tiles.reshape(-1)
|
||||
live = src >= 0
|
||||
inv = torch.empty(layout.seq_len, dtype=torch.int64, device=device)
|
||||
inv[src[live]] = torch.nonzero(live).flatten()
|
||||
plan = {'n': int(src.numel()), 'n_prefix': n_prefix, 'src': src, 'inv': inv, 'block_len': (tiles >= 0).sum(1).to(torch.int32)}
|
||||
while len(self.vsa_plans) >= VSA_PLAN_CACHE:
|
||||
del self.vsa_plans[next(iter(self.vsa_plans))]
|
||||
self.vsa_plans[key] = plan
|
||||
return plan
|
||||
|
||||
def vsa_rope_freqs(self, rope_freqs, plan):
|
||||
hit = self.vsa_rope
|
||||
if hit is not None and hit[0]() is rope_freqs and (hit[1] is plan):
|
||||
return hit[2]
|
||||
padded = rope_freqs.new_zeros((1, plan['n']) + tuple(rope_freqs.shape[2:]))
|
||||
padded[0, plan['inv']] = rope_freqs[0]
|
||||
self.vsa_rope = (weakref.ref(rope_freqs), plan, padded)
|
||||
return padded
|
||||
|
||||
def make_attention_override(patch: SparseAttnPatch, previous):
|
||||
"""Attention override; declined calls run ``previous`` (the override that was
|
||||
on the hook before this one) or ``func``. Dense-only in VSA mode: a
|
||||
VSA-trained model must never see plain block-sparse attention."""
|
||||
|
||||
def override(func, q, k, v, heads, mask=None, attn_precision=None, skip_reshape=False, skip_output_reshape=False, **kwargs):
|
||||
transformer_options = kwargs.get('transformer_options') or {}
|
||||
|
||||
def dense():
|
||||
args = (q, k, v, heads)
|
||||
kw = dict(mask=mask, attn_precision=attn_precision, skip_reshape=skip_reshape, skip_output_reshape=skip_output_reshape, **kwargs)
|
||||
return func(*args, **kw) if previous is None else previous(func, *args, **kw)
|
||||
if mask is not None or patch.vsa:
|
||||
return dense()
|
||||
tokens = q.shape[2] if skip_reshape else q.shape[1]
|
||||
reason = patch.dense_reason(transformer_options, tokens, transformer_options.get('block_index'))
|
||||
if reason is not None:
|
||||
patch.log_once(('dense', tokens, reason), f'dense ({tokens} tokens): {reason}')
|
||||
return dense()
|
||||
if skip_reshape:
|
||||
b, _, _, dim_head = q.shape
|
||||
qs, ks, vs = (t.transpose(1, 2) for t in (q, k, v))
|
||||
else:
|
||||
b, _, dim_head = q.shape
|
||||
dim_head //= heads
|
||||
qs, ks, vs = (t.view(b, -1, heads, dim_head) for t in (q, k, v))
|
||||
reason = _ineligible(qs, ks, vs, dim_head)
|
||||
if reason is not None:
|
||||
patch.log_once(('ineligible', tuple(qs.shape), reason), f'dense {tuple(qs.shape)}: {reason}')
|
||||
return dense()
|
||||
sink, sink_q = patch.sinks(transformer_options, tokens)
|
||||
if q.dtype == torch.float32:
|
||||
qs, ks, vs = (t.to(torch.bfloat16) for t in (qs, ks, vs))
|
||||
out = ck.sol_attn(qs, ks, vs, tau=patch.tau, scale=kwargs.get('scale'), sink_blocks=list(sink), sink_q=list(sink_q), topk_ratio=patch.topk_ratio, token_aug=patch.extra_tokens).to(q.dtype)
|
||||
patch.log_once(('sparse', tuple(qs.shape)), f'sparse {tuple(qs.shape)}, sinks {sink}/{sink_q}')
|
||||
if skip_output_reshape:
|
||||
return out.transpose(1, 2)
|
||||
return out.reshape(b, -1, heads * dim_head)
|
||||
return override
|
||||
|
||||
def make_h3_block_patch(block, block_index, patch: SparseAttnPatch):
|
||||
"""Runs the block with its attention swapped for the sparse producer."""
|
||||
|
||||
def attention(h, rope_freqs=None, transformer_options={}):
|
||||
return h3_sparse_attention(block.attn, h, rope_freqs, transformer_options, patch, block_index)
|
||||
|
||||
def block_patch(args, extra):
|
||||
if h3_eligible(block.attn, args['img'], args['rope_freqs'], args['transformer_options'], patch, block_index):
|
||||
args = {**args, 'attention': attention}
|
||||
return extra['original_block'](args)
|
||||
return block_patch
|
||||
@@ -174,6 +174,43 @@ class JoinTest(unittest.TestCase):
|
||||
self.assertEqual(video.shape[0], 45)
|
||||
self.assertAlmostEqual(float(video[0, 0, 0, 0]), 0.1, places=2)
|
||||
|
||||
def test_run_again_with_every_shot_stored_generates_nothing(self):
|
||||
join = SL.BFSShotJoin()
|
||||
shots = [_shot(0, 0, 20, count=2, queue=True, run_id="t2"), _shot(1, 20, 45, cut=True, count=2, queue=True, run_id="t2")]
|
||||
join.join([self._frames(22, 0.1)], [shots[0]], [0])
|
||||
join.join([self._frames(39, 0.9)], [shots[1]], [0])
|
||||
again = dict(shots[1], assemble_only=True) # what the planner hands out when nothing is left to run
|
||||
self.assertEqual(join.check_lazy_status(images=None, shots=[again]), []) # the sampler is not pulled
|
||||
self.assertEqual(join.check_lazy_status(images=None, shots=[shots[1]]), ["images"])
|
||||
video = SL.BFSShotJoin().join(shots=[again], crossfade=[0])[0]
|
||||
self.assertEqual(video.shape[0], 45)
|
||||
self.assertAlmostEqual(float(video[0, 0, 0, 0]), 0.1, places=2)
|
||||
self.assertAlmostEqual(float(video[-1, 0, 0, 0]), 0.9, places=2)
|
||||
|
||||
def test_comparison_turned_on_later_reads_the_originals_from_the_source(self):
|
||||
import cv2
|
||||
import numpy as np
|
||||
src = os.path.join(_TMP, "cmp_src.mp4")
|
||||
vw = cv2.VideoWriter(src, cv2.VideoWriter_fourcc(*"mp4v"), 24, (32, 32))
|
||||
for _ in range(50):
|
||||
vw.write(np.full((32, 32, 3), 200, np.uint8))
|
||||
vw.release()
|
||||
tl = {"path": src, "fps": 24.0, "width": 32, "height": 32, "fill": "original", "audio": None,
|
||||
"segs": [{"start": 0, "end": 20, "run": True, "cut_before": False, "run_index": 0},
|
||||
{"start": 20, "end": 45, "run": True, "cut_before": True, "run_index": 1}],
|
||||
"src": list(range(50)), "n": 50}
|
||||
shots = [_shot(0, 0, 20, count=2, queue=True, run_id="t3"), _shot(1, 20, 45, cut=True, count=2, queue=True, run_id="t3")]
|
||||
join = SL.BFSShotJoin()
|
||||
frames = lambda n, v: torch.full((n, 32, 32, 3), float(v))
|
||||
join.join([frames(22, 0.1)], [shots[0]], [0], timeline=[tl]) # stored with the comparison off
|
||||
join.join([frames(39, 0.9)], [shots[1]], [0], timeline=[tl])
|
||||
out = SL.BFSShotJoin().join(shots=[dict(shots[1], assemble_only=True)], crossfade=[0], timeline=[tl],
|
||||
comparison=[True], comparison_height=[32])
|
||||
comp, guide = out[3], out[5]
|
||||
self.assertEqual(guide.shape[0], 45) # the guide output is there again
|
||||
self.assertAlmostEqual(float(guide[10].mean()), 200 / 255, places=1) # and it is the source video
|
||||
self.assertGreater((comp - 200 / 255).abs().lt(0.05).float().mean().item(), 0.05) # original column shown
|
||||
|
||||
|
||||
class FilterTest(unittest.TestCase):
|
||||
STATS = {"persons": 1, "person_area": 0.2, "person_frames": 6, "faces": 1, "brightness": 0.4, "motion": 0.02, "sampled": 6}
|
||||
@@ -385,29 +422,41 @@ class RepackTest(unittest.TestCase):
|
||||
shot = dict(_shot(0, 0, 20, gen=22), width=8, height=8, frames=torch.zeros(22, 8, 8, 3),
|
||||
ref=torch.zeros(1, 4, 4, 3), ref2=None, prompt="old", audio=None)
|
||||
ref = torch.ones(1, 6, 6, 3)
|
||||
out = SL.BFSShotRepack().repack(shot, ref_image=ref)[0]
|
||||
out = SL.BFSShotRepack().repack([shot], refs=[ref])[0][0]
|
||||
self.assertTrue(torch.equal(out["ref"], ref))
|
||||
self.assertEqual(out["prompt"], "old")
|
||||
self.assertIs(out["frames"], shot["frames"])
|
||||
out = SL.BFSShotRepack().repack(shot, guide_frames=torch.ones(30, 16, 16, 3), prompt="new")[0]
|
||||
out = SL.BFSShotRepack().repack([shot], guide_frames=[torch.ones(30, 16, 16, 3)], prompt=["new"])[0][0]
|
||||
self.assertEqual(tuple(out["frames"].shape), (22, 8, 8, 3))
|
||||
self.assertEqual(out["prompt"], "new")
|
||||
self.assertEqual(out["length"], 20)
|
||||
|
||||
def test_unpack_lists_the_pictures_and_repack_puts_them_back_per_shot(self):
|
||||
a = [torch.full((1, 4, 4, 3), k / 10) for k in range(3)]
|
||||
base = dict(_shot(0, 0, 20, gen=22), width=8, height=8, frames=torch.zeros(22, 8, 8, 3), prompt="",
|
||||
audio=None)
|
||||
s1, s2, s3 = dict(base, refs=a[:2]), dict(base, index=1, refs=[]), dict(base, index=2, refs=a[2:])
|
||||
flat = []
|
||||
for s in (s1, s2, s3): # what ComfyUI hands the next node: the lists of every Unpack call, in order
|
||||
out = SL.BFSShotUnpack().unpack(s)
|
||||
flat += out[1]
|
||||
self.assertEqual(out[2], len(s["refs"]))
|
||||
edited = [x + 1 for x in flat] # e.g. a background remover, once per picture
|
||||
s1n, s2n, s3n = SL.BFSShotRepack().repack([s1, s2, s3], refs=edited)[0]
|
||||
self.assertEqual([len(s1n["refs"]), len(s2n["refs"]), len(s3n["refs"])], [2, 0, 1])
|
||||
self.assertTrue(torch.equal(s3n["refs"][0], a[2] + 1) and torch.equal(s1n["ref2"], a[1] + 1))
|
||||
|
||||
class ConditioningWriterTest(unittest.TestCase):
|
||||
"""The optional prompt writer of BFS Shot H3 Conditioning (template path, no models)."""
|
||||
|
||||
class ConditioningPromptTest(unittest.TestCase):
|
||||
"""BFS Shot H3 Conditioning passes the planner's prompt and adds the setting picture (no models)."""
|
||||
|
||||
def setUp(self):
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
import bfs_h3_side_panel as SP
|
||||
self.SP, self.seen = SP, {}
|
||||
seen = self.seen
|
||||
self.seen = seen = {}
|
||||
|
||||
class R2V:
|
||||
@staticmethod
|
||||
def execute(**kw):
|
||||
seen["prompt"] = kw["prompt"]
|
||||
seen.update(kw)
|
||||
return types.SimpleNamespace(args=("pos", "lat"))
|
||||
|
||||
mod = types.ModuleType("comfy_extras.nodes_minimax_h3")
|
||||
@@ -415,13 +464,8 @@ class ConditioningWriterTest(unittest.TestCase):
|
||||
self._old = {k: sys.modules.get(k) for k in ("comfy_extras", "comfy_extras.nodes_minimax_h3")}
|
||||
sys.modules["comfy_extras"] = types.ModuleType("comfy_extras")
|
||||
sys.modules["comfy_extras.nodes_minimax_h3"] = mod
|
||||
self._apply = SP.BFSH3SidePanel.apply
|
||||
SP.BFSH3SidePanel.apply = lambda self, pos, lat, *a, **k: (pos, lat, {"position": "left"}, None, None)
|
||||
self._patch = SP.patch_model_rope
|
||||
SP.patch_model_rope = lambda m, info, gap: "patched"
|
||||
|
||||
def tearDown(self):
|
||||
self.SP.BFSH3SidePanel.apply, self.SP.patch_model_rope = self._apply, self._patch
|
||||
for k, v in self._old.items():
|
||||
if v is None:
|
||||
sys.modules.pop(k, None)
|
||||
@@ -429,44 +473,65 @@ class ConditioningWriterTest(unittest.TestCase):
|
||||
sys.modules[k] = v
|
||||
|
||||
def _shot(self):
|
||||
return dict(prompt="my own {layout} prompt", width=448, height=800, gen_length=22,
|
||||
frames=torch.zeros(22, 8, 8, 3), ref=torch.zeros(1, 4, 4, 3), ref2=None, audio=None)
|
||||
return dict(prompt="subject_definitions: <Subject 1> is the man in <Picture 1>.", width=448, height=800,
|
||||
gen_length=22, frames=torch.full((22, 8, 8, 3), 0.5), ref=torch.zeros(1, 4, 4, 3), ref2=None,
|
||||
audio=None)
|
||||
|
||||
def _run(self, **kw):
|
||||
c = SL.BFSShotH3Conditioning()
|
||||
return c.condition(self._shot(), "clip", "vae", "none", True, "none", "match", **kw)
|
||||
|
||||
def test_planner_prompt_is_default_and_layout_filled(self):
|
||||
out = self._run()
|
||||
self.assertEqual(out[3], "my own prompt")
|
||||
out = self._run(duet="canvas")
|
||||
self.assertIn("the left half shows the original performance", out[3])
|
||||
def test_the_planner_prompt_goes_as_it_is_and_the_model_passes_through(self):
|
||||
out = SL.BFSShotH3Conditioning().condition(self._shot(), "clip", "vae", "none", "none", "match", model="m")
|
||||
self.assertEqual(out[3], self._shot()["prompt"])
|
||||
self.assertEqual(self.seen["prompt"], out[3])
|
||||
self.assertEqual(out[2], "m")
|
||||
self.assertNotIn("duet", SL.BFSShotH3Conditioning.INPUT_TYPES()["optional"])
|
||||
|
||||
def test_task_template_follows_the_duet_mode(self):
|
||||
out = self._run(duet="canvas", task="character swap")
|
||||
self.assertIn("split screen", out[3])
|
||||
self.assertIn("<Picture 1>", out[3])
|
||||
out = self._run(duet="shifted RoPE", model="m", task="character swap")
|
||||
self.assertNotIn("split screen", out[3])
|
||||
def test_native_temporal_requires_model_before_encoding(self):
|
||||
shot = self._shot()
|
||||
with self.assertRaisesRegex(ValueError, "MODEL connected"):
|
||||
SL.BFSShotH3Conditioning().condition(shot, "clip", "vae", SL.BFSShotH3Conditioning.GUIDE_MODES[5], "none", "match")
|
||||
self.assertEqual(self.seen, {})
|
||||
|
||||
def test_native_temporal_passes_shot_as_video_and_returns_patched_model(self):
|
||||
import minimax_h3_native_temporal as temporal
|
||||
shot = self._shot(); shot["fps"] = 24
|
||||
with patch.object(temporal, "patch_temporal_model", return_value="patched"):
|
||||
out = SL.BFSShotH3Conditioning().condition(shot, "clip", "vae", SL.BFSShotH3Conditioning.GUIDE_MODES[5], "none", "match", model="model")
|
||||
self.assertEqual(out[2], "patched")
|
||||
self.assertIs(self.seen["ref_videos"]["ref_video_1"], shot["frames"])
|
||||
|
||||
def test_native_temporal_allows_dormant_first_shot_chain_modes(self):
|
||||
import minimax_h3_native_temporal as temporal
|
||||
for mode in ("head", "history", "reference", "first frame"):
|
||||
with self.subTest(mode=mode), patch.object(temporal, "patch_temporal_model", return_value="patched"):
|
||||
shot = self._shot()
|
||||
shot.update(fps=24, index=0, count=1, chain=mode, head=0, history=None, chain_image=None)
|
||||
out = SL.BFSShotH3Conditioning().condition(shot, "clip", "vae", SL.BFSShotH3Conditioning.GUIDE_MODES[5], "none", "match", model="model")
|
||||
self.assertEqual(out[2], "patched")
|
||||
|
||||
def test_native_temporal_rejects_real_continuity_inputs(self):
|
||||
cases = [dict(index=1, chain="head", head=5),
|
||||
dict(index=1, chain="history", history=torch.zeros(5, 8, 8, 3)),
|
||||
dict(index=1, chain="first frame", chain_image=torch.zeros(1, 8, 8, 3))]
|
||||
for fields in cases:
|
||||
with self.subTest(fields=list(fields)):
|
||||
shot = self._shot(); shot.update(fps=24, count=2, **fields)
|
||||
with self.assertRaisesRegex(ValueError, "active shot continuity"):
|
||||
SL.BFSShotH3Conditioning().condition(shot, "clip", "vae", SL.BFSShotH3Conditioning.GUIDE_MODES[5], "none", "match", model="model")
|
||||
|
||||
def test_native_guide_scale_keeps_output_and_frames(self):
|
||||
import minimax_h3_native_temporal as temporal
|
||||
shot = self._shot(); shot.update(fps=24, frames=torch.zeros(22,64,128,3))
|
||||
with patch.object(temporal, "patch_temporal_model", return_value="patched"):
|
||||
SL.BFSShotH3Conditioning().condition(shot,"clip","vae",SL.BFSShotH3Conditioning.GUIDE_MODES[5],"none","match",model="model",native_guide_scale=0.5)
|
||||
self.assertEqual(tuple(self.seen["ref_videos"]["ref_video_1"].shape),(22,32,64,3))
|
||||
self.assertEqual((self.seen["width"],self.seen["height"],self.seen["length"]),(448,800,22))
|
||||
self.assertEqual(tuple(shot["frames"].shape),(22,64,128,3))
|
||||
|
||||
def test_setting_picture_is_the_last_reference_with_static(self):
|
||||
seen = {}
|
||||
orig = sys.modules["comfy_extras.nodes_minimax_h3"].MiniMaxH3ReferenceToVideo.execute
|
||||
|
||||
def capture(**kw):
|
||||
seen.update(kw)
|
||||
return orig(**kw)
|
||||
sys.modules["comfy_extras.nodes_minimax_h3"].MiniMaxH3ReferenceToVideo.execute = staticmethod(capture)
|
||||
shot = self._shot()
|
||||
shot["frames"] = torch.full((22, 8, 8, 3), 0.5)
|
||||
mask = torch.zeros(22, 8, 8)
|
||||
mask[:, 2:4, 2:4] = 1
|
||||
out = SL.BFSShotH3Conditioning().condition(shot, "clip", "vae", "none", True, "none", "match", duet="canvas",
|
||||
task="character swap", setting_ref="on (generation size)",
|
||||
setting_mask=mask)
|
||||
refs = list(seen["ref_images"].values())
|
||||
out = SL.BFSShotH3Conditioning().condition(self._shot(), "clip", "vae", "none", "none", "match",
|
||||
setting_ref="on (generation size)", setting_mask=mask)
|
||||
refs = list(self.seen["ref_images"].values())
|
||||
self.assertEqual(len(refs), 2)
|
||||
pic = refs[-1][0]
|
||||
self.assertTrue(((pic[3, 3] == 0) | (pic[3, 3] == 1)).all()) # static inside the (grown) mask
|
||||
@@ -474,10 +539,6 @@ class ConditioningWriterTest(unittest.TestCase):
|
||||
self.assertIn("<Picture 2> shows the setting", out[3])
|
||||
self.assertEqual(SL.add_setting("a {setting} b", 3, True), "a <Picture 3> b")
|
||||
|
||||
def test_task_needs_duet(self):
|
||||
with self.assertRaises(ValueError):
|
||||
self._run(task="style", instruction="anime")
|
||||
|
||||
|
||||
class SimpleConditioningTest(unittest.TestCase):
|
||||
def test_exposes_only_the_common_controls(self):
|
||||
@@ -486,11 +547,11 @@ class SimpleConditioningTest(unittest.TestCase):
|
||||
self.assertIn("with_audio", names)
|
||||
self.assertIn("inpaint", names)
|
||||
self.assertNotIn("duet", names)
|
||||
self.assertNotIn("model", names)
|
||||
self.assertIn("model", names)
|
||||
self.assertNotIn("rope_gap", names)
|
||||
self.assertNotIn("task", names)
|
||||
|
||||
def test_forwards_safe_advanced_defaults_and_hides_model_output(self):
|
||||
def test_forwards_safe_advanced_defaults_and_appends_model_output(self):
|
||||
seen = {}
|
||||
original = SL.BFSShotH3Conditioning.condition
|
||||
|
||||
@@ -501,13 +562,13 @@ class SimpleConditioningTest(unittest.TestCase):
|
||||
SL.BFSShotH3Conditioning.condition = fake
|
||||
try:
|
||||
out = SL.BFSShotH3ConditioningSimple().condition(
|
||||
"shot", "clip", "vae", SL.BFSShotH3ConditioningSimple.GUIDE_MODES[0], True, "match",
|
||||
"shot", "clip", "vae", SL.BFSShotH3ConditioningSimple.GUIDE_MODES[0], "match",
|
||||
audio_vae="audio vae", with_audio=True)
|
||||
finally:
|
||||
SL.BFSShotH3Conditioning.condition = original
|
||||
|
||||
self.assertEqual(out, ("positive", "latent", "prompt"))
|
||||
self.assertEqual(seen["args"][5], "none")
|
||||
self.assertEqual(out, ("positive", "latent", "prompt", "unchanged model"))
|
||||
self.assertEqual(seen["args"][4], "none")
|
||||
self.assertTrue(seen["kwargs"]["with_audio"])
|
||||
self.assertEqual(seen["kwargs"]["inpaint"], SL.INPAINT_MODES[0])
|
||||
|
||||
@@ -796,7 +857,7 @@ class ConditionFlowTest(unittest.TestCase):
|
||||
orig = SL.shot_mask
|
||||
SL.shot_mask = lambda *a: {"masks": m.to(torch.uint8)}
|
||||
try:
|
||||
return SL.BFSShotH3Conditioning().condition(shot, None, VAE(), SL.BFSShotH3Conditioning.GUIDE_MODES[0], False,
|
||||
return SL.BFSShotH3Conditioning().condition(shot, None, VAE(), SL.BFSShotH3Conditioning.GUIDE_MODES[0],
|
||||
"none", "match", **kw)
|
||||
finally:
|
||||
SL.shot_mask = orig
|
||||
@@ -824,6 +885,41 @@ class ConditionFlowTest(unittest.TestCase):
|
||||
self.assertEqual(len(self.calls["guides"]), 1)
|
||||
self.assertAlmostEqual(float(self.calls["guides"][0][0, 2, 2, 0]), 0.5) # it is the masked one
|
||||
|
||||
def test_passes_every_reference_in_order(self):
|
||||
refs = [torch.full((1, 32, 32, 3), k / 12) for k in range(12)]
|
||||
shot, _ = self._shot(inpaint=False)
|
||||
shot.pop("mask_src")
|
||||
shot.update(refs=refs, ref=refs[0], ref2=refs[1], ref_names=[f"r{k}.png" for k in range(12)],
|
||||
prompt="<Picture 12> is the {details@Picture 12}.")
|
||||
out = SL.BFSShotH3Conditioning().condition(shot, None, None, SL.BFSShotH3Conditioning.GUIDE_MODES[3],
|
||||
"none", "match", inpaint="off")
|
||||
kw = self.calls["r2v"][0]
|
||||
self.assertEqual(list(kw["ref_images"]), [f"ref_image_{k}" for k in range(12)])
|
||||
self.assertIs(kw["ref_images"]["ref_image_11"], refs[11])
|
||||
self.assertEqual(out[3], "<Picture 12> is the character.") # no VLM and no saved description
|
||||
self.assertNotIn("use_ref_2", SL.BFSShotH3Conditioning.INPUT_TYPES()["required"])
|
||||
self.assertNotIn("use_ref_2", SL.BFSShotH3ConditioningSimple.INPUT_TYPES()["required"])
|
||||
|
||||
def test_every_marked_person_in_their_picture_color(self):
|
||||
refs = [torch.zeros(1, 32, 32, 3) for _ in range(3)]
|
||||
shot, m = self._shot(inpaint=False)
|
||||
marks = [{"picture": 1, "color": "#00ff00"}, {"picture": 3, "color": "#ff2020", "text": "the man", "points": [], "key": 0}]
|
||||
shot.update(refs=refs, color_guide=True, color_guide_marks=marks, color_guide_color="#00ff00", color_guide_opacity=0.5)
|
||||
second = torch.zeros(22, 64, 64); second[:, 0:10, 0:10] = 1
|
||||
orig = SL.shot_mask
|
||||
SL.shot_mask = lambda path, a, start, length, spec: {"masks": second if spec.get("text") == "the man" else m.to(torch.uint8)}
|
||||
try:
|
||||
SL.BFSShotH3Conditioning().condition(shot, None, None, SL.BFSShotH3Conditioning.GUIDE_MODES[0],
|
||||
"none", "match", inpaint="off")
|
||||
guide = self.calls["guides"][0]
|
||||
self.assertGreater(float(guide[0, 5, 5, 0]), 0.75) # the second person, in red
|
||||
self.assertGreater(float(guide[0, 30, 30, 1]), 0.75) # the selection, in green
|
||||
with self.assertRaisesRegex(ValueError, "Picture 3"):
|
||||
SL.BFSShotH3Conditioning().condition(dict(shot, refs=refs[:2]), None, None,
|
||||
SL.BFSShotH3Conditioning.GUIDE_MODES[0], "none", "match", inpaint="off")
|
||||
finally:
|
||||
SL.shot_mask = orig
|
||||
|
||||
def test_ignored_outside_mask_only(self):
|
||||
shot, m = self._shot(inpaint=False)
|
||||
self.run_cond(shot, m, inpaint=SL.INPAINT_MODES[0], mask_guide=SL.MASK_GUIDES[3])
|
||||
@@ -834,6 +930,217 @@ class ConditionFlowTest(unittest.TestCase):
|
||||
self.assertNotIn("ref_videos", self.calls["r2v"][0])
|
||||
|
||||
|
||||
class MultiRefTest(unittest.TestCase):
|
||||
"""Up to 12 reference pictures per shot (plans before 1.113 had ref / ref2)."""
|
||||
|
||||
def test_old_plans_become_a_list_and_keep_ref_and_ref2(self):
|
||||
p = SL._load_plan('{"global_ref": "a.png", "global_ref2": "b.png"}')
|
||||
self.assertEqual(p["global_refs"], ["a.png", "b.png"])
|
||||
p = SL._load_plan('{"global_refs": ["a.png", "", "c.png", ""], "global_ref": "stale.png"}')
|
||||
self.assertEqual(p["global_refs"], ["a.png", "", "c.png"])
|
||||
self.assertEqual((p["global_ref"], p["global_ref2"]), ("a.png", ""))
|
||||
self.assertEqual(len(SL.ref_list([f"{k}.png" for k in range(20)])), min(20, SL.MAX_REFS))
|
||||
|
||||
def test_a_shot_slot_falls_back_to_the_global_one_and_gaps_close(self):
|
||||
self.assertEqual(SL.shot_ref_names(["", "x.png"], ["a.png", "b.png", "c.png"]), ["a.png", "x.png", "c.png"])
|
||||
self.assertEqual(SL.shot_ref_names(["", "", "z.png"], ["a.png"]), ["a.png", "z.png"])
|
||||
|
||||
def test_signature_of_a_two_reference_shot_is_unchanged(self):
|
||||
seg = {"start": 0, "end": 10, "ref": "a.png", "ref2": "b.png", "prompt": ""}
|
||||
plan = dict(SL.DEFAULT_PLAN, global_ref="g.png")
|
||||
old = SL.shot_signature(plan, seg, 0, {})
|
||||
new = SL.shot_signature(dict(plan, global_refs=["g.png"]), dict(seg, refs=["a.png", "b.png"]), 0, {})
|
||||
self.assertEqual(old, new)
|
||||
three = SL.shot_signature(dict(plan, global_refs=["g.png"]), dict(seg, refs=["a.png", "b.png", "c.png"]), 0, {})
|
||||
self.assertNotEqual(old, three)
|
||||
|
||||
def test_picture_details(self):
|
||||
names, imgs = ["a.png", "b.png"], [torch.zeros(1, 4, 4, 3)] * 2
|
||||
text = "<Subject 1> is the {details@Picture 2} shown in <Picture 2>, {details@Picture1}."
|
||||
out = SL.fill_picture_details(text, names, imgs, {"b.png": "the tall man"})
|
||||
self.assertEqual(out, "<Subject 1> is the tall man shown in <Picture 2>, character.")
|
||||
kept = SL.fill_picture_details(text, names, imgs, {"b.png": "tall man"}, keep_missing=True)
|
||||
self.assertIn("{details@Picture1}", kept)
|
||||
self.assertEqual(SL.fill_picture_details("{details@Picture 5}", names, imgs, {}), "character")
|
||||
self.assertEqual(SL.prompt_slot(text, "{details@Picture 2}"),
|
||||
"<Subject 1> is the ___ shown in <Picture 2>, the subject of <Picture 1>.")
|
||||
self.assertEqual(SL.picture_marker(text, 1), "{details@Picture1}")
|
||||
self.assertEqual(SL.expand_two_ref_prompt(text), text) # the prompt already says what <Picture 2> is
|
||||
|
||||
def test_details_and_details2_are_pictures_1_and_2(self):
|
||||
self.assertEqual(SL.picture_markers("{details} and {details2}"), "{details@Picture 1} and {details@Picture 2}")
|
||||
old = {"ref_details": {"a.png|b.png": {"details": "goblin", "details2": "robot"}, "c.png|": "cat"},
|
||||
"picture_details": {"b.png": "tall robot"}}
|
||||
self.assertEqual(SL.saved_picture_details(old), {"a.png": "goblin", "b.png": "tall robot", "c.png": "cat"})
|
||||
|
||||
def test_a_picture_can_have_its_own_preset(self):
|
||||
cfg = SL.vlm_cfg({"describe_preset": "short"})
|
||||
plan = {"picture_cfg": {"a.png": {"preset": "custom", "custom": "Describe the outfit only."}}}
|
||||
self.assertEqual(SL.picture_cfg(cfg, plan, "a.png")["describe_preset"], "custom")
|
||||
self.assertIn("outfit only", SL.describe_instruction(SL.picture_cfg(cfg, plan, "a.png")))
|
||||
self.assertIs(SL.picture_cfg(cfg, plan, "b.png"), cfg)
|
||||
|
||||
def test_colored_mark_regions_follow_the_pictures(self):
|
||||
legacy = {"global_color_guide_ref1": "#168cff", "global_color_guide_ref2": "#9b30ff"}
|
||||
marks = SL.color_marks(legacy, {"color_guide_text2": "the man", "color_guide_key2": 4})
|
||||
self.assertEqual([(x["picture"], x["color"]) for x in marks], [(1, "#168cff"), (2, "#9b30ff")])
|
||||
self.assertEqual(marks[1]["text"], "the man")
|
||||
plan = {"picture_colors": ["#00ff00"]}
|
||||
marks = SL.color_marks(plan, {"color_guide_picture": 2, "color_guide_regions": [
|
||||
{"picture": 5, "text": "the dog"}, {"picture": 3}]}) # a region with neither text nor points is ignored
|
||||
self.assertEqual([(x["picture"], x["color"]) for x in marks], [(2, "#9b30ff"), (5, "#ffe600")])
|
||||
text = SL.color_guide_prompt("A scene.", marks)
|
||||
self.assertIn("marked in purple uses the character appearance from <Picture 2>", text)
|
||||
self.assertIn("marked in yellow uses the character appearance from <Picture 5>", text)
|
||||
one = SL.color_guide_prompt("", marks[:1])
|
||||
self.assertIn("the subject marked in purple uses the character appearance from <Picture 2>", one)
|
||||
|
||||
def test_color_contract_is_present_even_when_reference_appearance_mentions_the_color(self):
|
||||
seg = {"color_guide": True, "color_guide_marks": [{"picture": 3, "color": "#00ff00"}]}
|
||||
raw = "Replace the man with the green goblin from <Picture 3>. Render the scene as anime on a rooftop."
|
||||
text = SL.tidy_caption(raw, seg)
|
||||
self.assertIn("the subject marked in green uses the character appearance from <Picture 3>", text)
|
||||
self.assertIn("anime on a rooftop", text)
|
||||
self.assertNotIn("all unmarked content unchanged", text)
|
||||
self.assertEqual(SL.color_guide_prompt(text, seg["color_guide_marks"]), text)
|
||||
|
||||
def test_auto_prompt_asks_with_the_official_format(self):
|
||||
seen = {}
|
||||
|
||||
def fake(clip, prompt, images, max_tokens, *a, **k):
|
||||
seen.update(prompt=prompt, n=images.shape[0], tokens=max_tokens)
|
||||
return "```\n**subject_definitions:**\n<Subject 1> is the robot in <Picture 2>.\nsummary:\n[video editing] x```"
|
||||
old, SL.vlm_generate = SL.vlm_generate, fake
|
||||
try:
|
||||
seg = {"color_guide": True, "color_guide_marks": [{"picture": 2, "color": "#9b30ff"}], "mask": {"text": "the man"},
|
||||
"inpaint": True}
|
||||
frame, pic = torch.zeros(1, 8, 8, 3), torch.zeros(1, 4, 4, 3)
|
||||
out = SL.write_h3_prompt(None, [frame] * 3, [0, 0.5, 1.0], [pic, pic], ["", "a silver robot"],
|
||||
"swap the man for the robot", seg, "video", 1.5)
|
||||
finally:
|
||||
SL.vlm_generate = old
|
||||
self.assertEqual(seen["n"], 5) # 3 frames + 2 pictures
|
||||
for want in ("swap the man for the robot", "<Picture 2>: image 5, described as: a silver robot",
|
||||
"marked in purple follows the appearance from <Picture 2>", "Only the selected subject is generated again",
|
||||
"edited version of <Video 1>", "retention_analysis:", "non_diegetic_music:", "00:00.500",
|
||||
"Never write a timestamp inside a shot", "is replaced by <Subject N>"):
|
||||
self.assertIn(want, seen["prompt"])
|
||||
self.assertTrue(out.startswith("motion_transfer: subject_definitions:\n<Subject 1>")) # fences and bold removed
|
||||
self.assertIn("marked in purple", out) # the mapping the writer left out is added
|
||||
seg2 = dict(seg, prompt="")
|
||||
self.assertEqual(SL.auto_prompt_for("x", ["a.png"], seg2), SL.auto_prompt_for("x", ["a.png"], dict(seg2)))
|
||||
self.assertNotEqual(SL.auto_prompt_for("x", ["a.png"], seg2), SL.auto_prompt_for("y", ["a.png"], seg2))
|
||||
self.assertEqual(SL.auto_frame_count(24, 24.0), 3)
|
||||
self.assertEqual(SL.auto_frame_count(240, 24.0), 8)
|
||||
|
||||
def test_a_template_request_keeps_its_trigger_and_wording(self):
|
||||
seen = {}
|
||||
|
||||
def fake(clip, prompt, images, max_tokens, *a, **k):
|
||||
seen["prompt"] = prompt
|
||||
return "subject_definitions: <Subject 1> is the woman in Picture 1. detailed_description: [Shot 1] She walks."
|
||||
old, SL.vlm_generate = SL.vlm_generate, fake
|
||||
template = ("body_swap: subject_definitions: <Subject 1> is the {details@Picture 1} shown in <Picture 1>. "
|
||||
"summary: [video generation + reference] Replace the guide subject. detailed_description: [Shot 1] x")
|
||||
try:
|
||||
out = SL.write_h3_prompt(None, [torch.zeros(1, 8, 8, 3)], [0.0], [torch.zeros(1, 4, 4, 3)], [""], template,
|
||||
{}, "guide")
|
||||
finally:
|
||||
SL.vlm_generate = old
|
||||
self.assertIn("prompt TEMPLATE", seen["prompt"])
|
||||
self.assertIn("'body_swap'", seen["prompt"])
|
||||
self.assertIn("stands for the description of <Picture n>", seen["prompt"])
|
||||
self.assertTrue(out.startswith("body_swap: subject_definitions:")) # the trigger the writer dropped
|
||||
self.assertIn("the woman in <Picture 1>", out) # brackets put back
|
||||
self.assertEqual(SL.request_trigger("make it rain: now"), "")
|
||||
self.assertEqual(SL.request_trigger("head_swap: subject_definitions: x"), "head_swap")
|
||||
self.assertFalse(SL.is_template("replace the bottle with the can"))
|
||||
|
||||
def test_key_frames_cover_every_piece_and_the_cuts_reach_the_writer(self):
|
||||
self.assertEqual(SL.key_frames(0, 100, [], 4), [12, 38, 62, 88])
|
||||
frames = SL.key_frames(0, 300, [43, 86, 129, 172, 215, 258], 3) # a montage: 7 pieces, one frame each at least
|
||||
self.assertEqual(len(frames), 7)
|
||||
self.assertTrue(all(any(a <= f < b for f in frames) for a, b in zip([0, 43, 86, 129, 172, 215, 258], [43, 86, 129, 172, 215, 258, 300])))
|
||||
seen = {}
|
||||
|
||||
def fake(clip, prompt, images, max_tokens, *a, **k):
|
||||
seen["prompt"] = prompt
|
||||
return ("subject_definitions:\n<Subject 1> is x.\ndetailed_description:\n[Shot 1] At 00:00.000, a woman stands. "
|
||||
"At 00:01.792, she walks.")
|
||||
old, SL.vlm_generate = SL.vlm_generate, fake
|
||||
try:
|
||||
one = SL.write_h3_prompt(None, [torch.zeros(1, 8, 8, 3)], [0.0], [], [], "Troque a mulher pela <Picture 1>", {})
|
||||
two = SL.write_h3_prompt(None, [torch.zeros(1, 8, 8, 3)], [0.0], [], [], "swap", {}, cuts=[1.792],
|
||||
template="body_swap: subject_definitions: a. summary: b. detailed_description: [Shot 1] c")
|
||||
finally:
|
||||
SL.vlm_generate = old
|
||||
self.assertIn("[Shot 1] A woman stands. She walks.", one) # no timestamps for one continuous shot
|
||||
self.assertIn('"[Shot 2] At 00:01.792, the shot cuts to ..."', seen["prompt"])
|
||||
self.assertIn("The user's request: swap", seen["prompt"])
|
||||
self.assertIn("The prompt template:\nbody_swap:", seen["prompt"])
|
||||
self.assertTrue(two.startswith("body_swap: "))
|
||||
self.assertIn("[Shot 1] A woman stands. At 00:01.792", two)
|
||||
|
||||
def test_a_looping_answer_is_cleaned_and_the_labels_are_tied(self):
|
||||
loop = " The goblin looks at the cat, and the cat looks at the goblin. The goblin then looks up." * 6
|
||||
raw = ("subject_definitions:\n<Subject 1> is the gaunt, emaciated goblin with pale skin in <Picture 1>.\n"
|
||||
"<Subject 2> is the stout goblin with red skin in <Picture 2>.\nsummary:\n[video editing] x\n"
|
||||
"retention_analysis:\n<Subject 1>: fully_preserved - x\ndetailed_description:\n[Shot 1] A goblin types."
|
||||
+ loop + " The gaunt goblin")
|
||||
self.assertIn("sentences are repeated in a loop", SL.h3_prompt_problems(raw, 2))
|
||||
t = SL.tidy_h3_prompt(raw, {}, "", False, 2)
|
||||
self.assertEqual(t.count("The goblin then looks up."), 1)
|
||||
self.assertNotIn("The gaunt goblin\n", t) # the sentence cut in the middle
|
||||
self.assertIn("overall_soundscape:", t)
|
||||
self.assertIn("non_diegetic_music:\nN/A", t)
|
||||
named = SL.name_subjects(t)
|
||||
self.assertIn("[Shot 1] <Subject 1> is the gaunt. <Subject 2> is the stout goblin with red skin.", named)
|
||||
self.assertEqual(SL.h3_prompt_problems(named, 2), [])
|
||||
|
||||
def test_caption_style_asks_for_the_training_captions_shape(self):
|
||||
seen = {}
|
||||
|
||||
def fake(clip, prompt, images, max_tokens, *a, **k):
|
||||
seen.update(prompt=prompt, tokens=max_tokens)
|
||||
return ('"Replace the dog in the video with the gaunt goblin from Picture 1. Replace the cat in the video with '
|
||||
'the stout goblin from <Picture 2>. Keep the same background and location as in the video, and maintain '
|
||||
'the motion, camera angle, and timing."')
|
||||
old, SL.vlm_generate = SL.vlm_generate, fake
|
||||
try:
|
||||
out = SL.write_h3_prompt(None, [torch.zeros(1, 8, 8, 3)], [0.0], [torch.zeros(1, 4, 4, 3)] * 2, ["", ""],
|
||||
"Troque o cão pelo <Picture 1> e o gato pelo <Picture 2>", {}, style="captions")
|
||||
finally:
|
||||
SL.vlm_generate = old
|
||||
self.assertIn("short style of a video-edit request", seen["prompt"])
|
||||
self.assertNotIn("full-reference format", seen["prompt"])
|
||||
self.assertNotIn("anime", seen["prompt"].lower()) # no sample style may leak into a request with no style edit
|
||||
self.assertNotIn("Example request:", seen["prompt"])
|
||||
self.assertEqual(seen["tokens"], 512)
|
||||
self.assertTrue(out.startswith("motion_transfer: Replace the dog in the video with the gaunt goblin from <Picture 1>."))
|
||||
self.assertEqual(SL.caption_problems(out, 2), [])
|
||||
self.assertTrue(SL.caption_problems("subject_definitions: <Subject 1> is x", 1))
|
||||
|
||||
def test_a_written_prompt_is_part_of_the_shot_signature_only_with_auto_prompt_on(self):
|
||||
seg = {"start": 0, "end": 10, "ref": "", "ref2": "", "prompt": "make it rain"}
|
||||
plan = dict(SL.DEFAULT_PLAN, auto_prompts={"0-10": {"text": "A"}})
|
||||
base = SL.shot_signature(plan, seg, 0, {})
|
||||
self.assertEqual(base, SL.shot_signature(dict(SL.DEFAULT_PLAN), seg, 0, {}))
|
||||
on = dict(plan, auto_prompt={"enabled": True})
|
||||
self.assertNotEqual(SL.shot_signature(on, seg, 0, {}),
|
||||
SL.shot_signature(dict(on, auto_prompts={"0-10": {"text": "B"}}), seg, 0, {}))
|
||||
|
||||
def test_repack_replaces_the_first_two_and_keeps_the_rest(self):
|
||||
refs = [torch.zeros(1, 4, 4, 3) for _ in range(3)]
|
||||
new = torch.ones(1, 4, 4, 3)
|
||||
out = SL.BFSShotRepack().repack([dict(_shot(0, 0, 20), refs=list(refs), ref=refs[0], ref2=refs[1])],
|
||||
ref_image_2=[new])[0][0]
|
||||
self.assertTrue(torch.equal(out["refs"][1], new) and torch.equal(out["ref2"], new))
|
||||
self.assertIs(out["refs"][2], refs[2])
|
||||
|
||||
def test_the_planner_has_no_reference_inputs(self):
|
||||
self.assertNotIn("ref_image", SL.BFSShotPlanner.INPUT_TYPES()["optional"])
|
||||
|
||||
|
||||
class DescribeShortTest(unittest.TestCase):
|
||||
def test_short_details_fit_inside_the_prompt(self):
|
||||
self.assertEqual(SL.tidy_details('"The green goblin in a torn leather vest."\nextra', {"describe_preset": "short"}),
|
||||
@@ -1127,17 +1434,57 @@ class ShotStoreTest(unittest.TestCase):
|
||||
with patch.object(SL, "_next_output_path", lambda prefix: os.path.join(d, prefix + "_00001.mp4")):
|
||||
out = SL.BFSShotJoin().join(imgs, shots, [0], audio=[audio], save_to=[SL.SAVE_TO[1]],
|
||||
filename_prefix=["test"])
|
||||
res = out["result"]
|
||||
res = out
|
||||
self.assertEqual(res[0].shape[0], 45) # the output still carries the video (read back)
|
||||
self.assertAlmostEqual(float(res[0][30].mean()), 0.8, places=1)
|
||||
self.assertTrue(res[4].endswith("test_00001.mp4"))
|
||||
self.assertEqual(out["ui"]["images"][0]["filename"], "test_00001.mp4") # shown on the node
|
||||
self.assertIsInstance(out, tuple) # no preview on the node: plain outputs
|
||||
cap = cv2.VideoCapture(res[4]); n = 0
|
||||
while cap.read()[0]:
|
||||
n += 1
|
||||
self.assertEqual(n, 45)
|
||||
|
||||
|
||||
class RenderFingerprintTest(unittest.TestCase):
|
||||
"""Changing what renders the shots (LoRA, seed...) must make the stored shots stale; the join's settings must not."""
|
||||
|
||||
@staticmethod
|
||||
def graph():
|
||||
return {
|
||||
"1": {"class_type": "BFSShotPlanner", "inputs": {"plan": "{\"run\": \"auto\"}"}},
|
||||
"2": {"class_type": "BFSShotUnpack", "inputs": {"shot": ["1", 0]}},
|
||||
"3": {"class_type": "BFSShotH3ConditioningSimple", "inputs": {"shot": ["2", 10], "clip": ["4", 0], "guide_mode": "aligned guide"}},
|
||||
"4": {"class_type": "CLIPLoader", "inputs": {"clip_name": "qwen3vl.safetensors", "type": "minimax"}},
|
||||
"5": {"class_type": "UNETLoader", "inputs": {"unet_name": "h3.safetensors", "weight_dtype": "default"}},
|
||||
"6": {"class_type": "LoraLoaderModelOnly", "inputs": {"model": ["5", 0], "lora_name": "swap_v1.safetensors", "strength_model": 1.0}},
|
||||
"7": {"class_type": "KSampler", "inputs": {"model": ["6", 0], "positive": ["3", 0], "seed": 42, "steps": 8}},
|
||||
"8": {"class_type": "VAEDecode", "inputs": {"samples": ["7", 0], "vae": ["9", 0]}},
|
||||
"9": {"class_type": "VAELoader", "inputs": {"vae_name": "vae.safetensors"}},
|
||||
"10": {"class_type": "BFSShotJoin", "inputs": {"images": ["8", 0], "shots": ["2", 10], "crossfade": 4}},
|
||||
"11": {"class_type": "BFSVideoCompare", "inputs": {"guide": ["10", 5], "target": ["10", 0], "gain": 2.0}},
|
||||
}
|
||||
|
||||
def test_render_changes_count_and_join_settings_do_not(self):
|
||||
base = SL.render_fingerprint(self.graph(), "1")
|
||||
self.assertTrue(base)
|
||||
self.assertEqual(SL.render_fingerprint(self.graph(), "1"), base)
|
||||
for node, key, value, changes in (("6", "lora_name", "swap_v2.safetensors", True), ("6", "strength_model", 0.8, True),
|
||||
("7", "seed", 43, True), ("3", "guide_mode", "reference video", True),
|
||||
("10", "crossfade", 8, False), ("1", "plan", "{}", False), ("11", "gain", 4.0, False)):
|
||||
g = self.graph()
|
||||
g[node]["inputs"][key] = value
|
||||
self.assertEqual(SL.render_fingerprint(g, "1") != base, changes, f"{node}.{key}")
|
||||
g = self.graph()
|
||||
del g["10"]
|
||||
self.assertEqual(SL.render_fingerprint(g, "1"), "")
|
||||
|
||||
def test_the_fingerprint_is_part_of_the_shot_signature(self):
|
||||
seg = {"start": 0, "end": 20, "prompt": "x"}
|
||||
a = SL.shot_signature({}, seg, 0, {"render": "aaa"})
|
||||
self.assertNotEqual(a, SL.shot_signature({}, seg, 0, {"render": "bbb"}))
|
||||
self.assertEqual(SL.BFSShotPlanner.INPUT_TYPES()["hidden"], {"graph": "PROMPT", "node_id": "UNIQUE_ID"})
|
||||
|
||||
|
||||
class SecondPassSizeTest(unittest.TestCase):
|
||||
def _upscaler(self, w, h, mode, val, align=32):
|
||||
# Minimax H3 Latent Upscaler (3D), from a latent of w/16 x h/16 cells
|
||||
@@ -1170,7 +1517,7 @@ class JoinWithoutFfmpegTest(unittest.TestCase):
|
||||
patch.object(SL, "_ram_available", lambda: 1000), \
|
||||
patch.object(SL, "_next_output_path", lambda prefix: os.path.join(d, prefix + "_00001.mp4")):
|
||||
out = SL.BFSShotJoin().join(imgs, shots, [0], audio=[audio], comparison=[True])
|
||||
res = out["result"]
|
||||
res = out
|
||||
self.assertTrue(os.path.getsize(res[4]) > 0 and os.path.exists(res[4][:-4] + "_comparison.mp4"))
|
||||
self.assertEqual((res[0].shape[0], res[3].shape[0], res[5].shape[0]), (45, 45, 45))
|
||||
self.assertAlmostEqual(float(res[0][30].mean()), 0.8, delta=0.05)
|
||||
@@ -1186,7 +1533,7 @@ class JoinFileMetadataTest(unittest.TestCase):
|
||||
out = SL.BFSShotJoin().join([torch.rand(22, 32, 32, 3)], [_shot(0, 0, 20, gen=22)], [0], save_to=[SL.SAVE_TO[1]],
|
||||
filename_prefix=["m"], prompt=[pr], extra_pnginfo=[{"workflow": wf}])
|
||||
tags = json.loads(subprocess.run(["ffprobe", "-v", "error", "-show_entries", "format_tags", "-of", "json",
|
||||
out["result"][4]], capture_output=True).stdout)["format"]["tags"]
|
||||
out[4]], capture_output=True).stdout)["format"]["tags"]
|
||||
self.assertEqual((json.loads(tags["workflow"]), json.loads(tags["prompt"])), (wf, pr))
|
||||
|
||||
|
||||
@@ -1198,7 +1545,7 @@ class JoinRamGuardTest(unittest.TestCase):
|
||||
with patch.object(SL, "_ram_available", lambda: 1000), \
|
||||
patch.object(SL, "_next_output_path", lambda prefix: os.path.join(d, prefix + "_00001.mp4")):
|
||||
out = SL.BFSShotJoin().join(imgs, shots, [0], comparison=[True])
|
||||
res = out["result"]
|
||||
res = out
|
||||
self.assertTrue(os.path.exists(res[4]) and os.path.exists(res[4][:-4] + "_comparison.mp4"))
|
||||
self.assertEqual(res[0].shape[0], 45) # never an empty output node
|
||||
self.assertTrue(res[0].shape[1] < 32 and res[3].shape[0] == 45) # scaled down to the (tiny) free RAM
|
||||
@@ -1281,3 +1628,127 @@ class FillDetailsTest(unittest.TestCase):
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
|
||||
class RedoRecoveryTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.tmp.cleanup)
|
||||
self.directory = Path(self.tmp.name)
|
||||
self.plan = {"video": "test.mp4"}
|
||||
self.state = {"done": [0, 1], "count": 2, "sig": {"0": "a", "1": "b"},
|
||||
"gen": {"0": "old", "1": "next"}, "from": {"1": "old"},
|
||||
"cur": {"sigs": ["a", "b"], "prev": [False, True]}}
|
||||
for i in range(2):
|
||||
(self.directory / f"shot_{i:04d}.pt").write_bytes(b"saved take")
|
||||
for name, value in (("_input_path", lambda _: "video"), ("run_key", lambda *_: "test"),
|
||||
("run_dir", lambda *a, **kw: str(self.directory))):
|
||||
patcher = patch.object(SL, name, value)
|
||||
patcher.start()
|
||||
self.addCleanup(patcher.stop)
|
||||
SL.save_state("test", self.state)
|
||||
|
||||
def test_redo_can_be_cancelled_with_continuity_restored(self):
|
||||
self.assertEqual(SL.redo_shots(self.plan, [0])["ok"], [False, False])
|
||||
result = SL.redo_shots(self.plan, [0], cancel=True)
|
||||
self.assertEqual(result["ok"], [True, True])
|
||||
self.assertEqual(result["redo"], [])
|
||||
|
||||
def test_reset_preserves_files_and_state(self):
|
||||
SL.redo_shots(self.plan, [0], cascade=False)
|
||||
result = SL.cache_status(self.plan, reset=True)
|
||||
self.assertEqual(result["ok"], [True, True])
|
||||
self.assertEqual((self.directory / "shot_0000.pt").read_bytes(), b"saved take")
|
||||
self.assertNotIn("0", SL.run_state("test").get("keep_next", {}))
|
||||
self.assertEqual(SL.run_state("test")["gen"], self.state["gen"])
|
||||
|
||||
def test_repeated_selection_does_not_overwrite_undo(self):
|
||||
SL.redo_shots(self.plan, [0])
|
||||
SL.redo_shots(self.plan, [0])
|
||||
self.assertEqual(SL.redo_shots(self.plan, [0], cancel=True)["ok"], [True, True])
|
||||
|
||||
def test_cancel_does_not_make_changed_settings_valid(self):
|
||||
SL.redo_shots(self.plan, [0])
|
||||
st = SL.run_state("test")
|
||||
st["cur"]["sigs"][0] = "changed seed"
|
||||
SL.save_state("test", st)
|
||||
self.assertEqual(SL.redo_shots(self.plan, [0], cancel=True)["ok"], [False, False])
|
||||
|
||||
def test_cancel_does_not_restore_replaced_take(self):
|
||||
SL.redo_shots(self.plan, [0])
|
||||
st = SL.run_state("test")
|
||||
st["gen"]["0"] = "new"
|
||||
SL.save_state("test", st)
|
||||
self.assertEqual(SL.redo_shots(self.plan, [0], cancel=True)["ok"], [False, False])
|
||||
|
||||
def test_reset_without_selection_preserves_completed_run(self):
|
||||
self.assertEqual(SL.cache_status(self.plan, reset=True)["ok"], [True, True])
|
||||
|
||||
|
||||
class IsolatedReferenceBindingTests(unittest.TestCase):
|
||||
def test_each_reference_has_its_own_image_and_index(self):
|
||||
images = [torch.zeros(1, 4, 4, 3), torch.ones(1, 4, 4, 3)]
|
||||
calls = []
|
||||
def describe(clip, batch, instruction, tokens, **kwargs):
|
||||
self.assertEqual(len(batch), 1)
|
||||
calls.append((batch[0], instruction))
|
||||
return "human in dark suit" if batch[0] is images[0] else "goblin in loincloth"
|
||||
with patch.object(SL, "vlm_describe", side_effect=describe):
|
||||
result = SL.isolated_picture_descriptions(object(), images)
|
||||
self.assertEqual(result, ["human in dark suit", "goblin in loincloth"])
|
||||
self.assertIn("ONLY <Picture 1>", calls[0][1])
|
||||
self.assertIn("ONLY <Picture 2>", calls[1][1])
|
||||
self.assertIs(calls[0][0], images[0])
|
||||
self.assertIs(calls[1][0], images[1])
|
||||
|
||||
def test_final_writer_only_receives_separate_text_facts(self):
|
||||
captured = []
|
||||
def generate(clip, prompt, images, tokens, **kwargs):
|
||||
captured.append((prompt, images))
|
||||
return "Replace the woman with the human from <Picture 1> and the man with the goblin from <Picture 2>."
|
||||
with patch.object(SL, "vlm_generate", side_effect=generate):
|
||||
SL.write_h3_prompt(object(), [], [], [object(), object()],
|
||||
["human in dark suit", "pink goblin"], "Replace both fighters", {},
|
||||
style="captions", source_facts="A woman and a man are fighting.")
|
||||
self.assertIsNone(captured[0][1])
|
||||
self.assertIn("<Picture 1>: human in dark suit", captured[0][0])
|
||||
self.assertIn("<Picture 2>: pink goblin", captured[0][0])
|
||||
self.assertIn("A woman and a man are fighting", captured[0][0])
|
||||
|
||||
def test_browser_and_backend_use_same_prompt_revision(self):
|
||||
browser = (ROOT / "web/js/bfs_shot_loop.js").read_text()
|
||||
start = browser.index("const autoFor =")
|
||||
signature = browser[start:browser.index("const writtenOf", start)]
|
||||
self.assertIn('"' + SL.AUTO_PROMPT_REVISION + '"', signature)
|
||||
|
||||
def test_motion_trigger_is_added_once_and_can_be_disabled(self):
|
||||
self.assertEqual(SL.tidy_caption("Replace the actor.", trigger="motion_transfer"), "motion_transfer: Replace the actor.")
|
||||
self.assertEqual(SL.tidy_caption("motion_transfer: Replace the actor.", trigger="motion_transfer"), "motion_transfer: Replace the actor.")
|
||||
self.assertEqual(SL.tidy_caption("motion_transfer: Replace the actor.", trigger=""), "Replace the actor.")
|
||||
self.assertNotEqual(SL.auto_prompt_for("edit", [], {}, prompt_trigger=""), SL.auto_prompt_for("edit", [], {}, prompt_trigger="motion_transfer"))
|
||||
|
||||
|
||||
class PromptReferenceContractTest(unittest.TestCase):
|
||||
def test_short_caption_does_not_acquire_phantom_second_subject(self):
|
||||
text = "motion_transfer: Replace the dancer with the character from <Picture 1>. Use the requested setting."
|
||||
self.assertEqual(SL.expand_two_ref_prompt(text), text)
|
||||
|
||||
def test_legacy_expansion_requires_explicit_description_template(self):
|
||||
text = "subject_definitions: <Subject 1> is the {details} in <Picture 1>."
|
||||
self.assertIn("<Subject 2>", SL.expand_two_ref_prompt(text))
|
||||
finished = "subject_definitions: <Subject 1> is the fox in <Picture 1>."
|
||||
self.assertEqual(SL.expand_two_ref_prompt(finished), finished)
|
||||
|
||||
def test_invalid_picture_indices_are_reported_in_both_formats(self):
|
||||
text = "Replace the setting with <Picture 3>."
|
||||
self.assertTrue(SL.caption_problems(text, 2))
|
||||
self.assertTrue(SL.h3_prompt_problems(text, 2, template=True))
|
||||
self.assertEqual(SL.picture_index_problems(text, 3), [])
|
||||
|
||||
def test_writer_refuses_nonexistent_reference_after_retry(self):
|
||||
with patch.object(SL, "vlm_generate", return_value="Replace the setting with <Picture 3>.") as generator:
|
||||
with self.assertRaisesRegex(ValueError, "Picture 3.*does not exist"):
|
||||
SL.write_h3_prompt(object(), [], [], [object(), object()],
|
||||
["fox wearing blue pants", "city street"], "Use Picture 2 for the setting", {},
|
||||
style="captions", source_facts="A dancer is moving in a street.")
|
||||
self.assertEqual(generator.call_count, 2)
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
"""Source blur modes: '<region> (<what stays sharp>)', SAM 3.1 or Sapiens2, blur or inverted colours.
|
||||
The segmenters are replaced by fixed masks, so this checks which pixels end up blurred, not the models."""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
import types
|
||||
import unittest
|
||||
|
||||
import torch
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
sys.path.insert(0, ROOT)
|
||||
# only what the blur uses of the shot loop: the frame key and SAM 3 (a fixed 'mouth' / part mask here)
|
||||
_SL = types.ModuleType("bfs_shot_loop")
|
||||
_SL._image_key = lambda t: (tuple(t.shape), float(t.float().sum()))
|
||||
_SL.DEFAULT_MASK = {}
|
||||
_SL.SAM_CALLS = []
|
||||
|
||||
|
||||
def _sam(frames, spec):
|
||||
_SL.SAM_CALLS.append(spec["text"])
|
||||
m = torch.zeros(frames.shape[:3])
|
||||
if spec["text"] == "mouth":
|
||||
m[:, 22:25, 28:36] = 1
|
||||
else:
|
||||
m[:, 10:30, 20:44] = 1
|
||||
return m
|
||||
|
||||
|
||||
_SL.segment_frames = _sam
|
||||
sys.modules.setdefault("bfs_shot_loop", _SL)
|
||||
import bfs_sapiens as SP # noqa: E402
|
||||
import bfs_source_blur as B # noqa: E402
|
||||
|
||||
H, W = 48, 64
|
||||
FACE = (slice(10, 30), slice(20, 44))
|
||||
LIPS = (slice(22, 24), slice(28, 36))
|
||||
TEETH = (slice(24, 25), slice(28, 36))
|
||||
|
||||
|
||||
def fake_classes(frames, out_size=None, batch=4):
|
||||
c = torch.zeros(frames.shape[0], H, W, dtype=torch.uint8)
|
||||
c[:, 5:45, 10:54] = SP.ID["Upper_Clothing"]
|
||||
c[(slice(None),) + FACE] = SP.ID["Face_Neck"]
|
||||
c[(slice(None),) + LIPS] = SP.ID["Upper_Lip"]
|
||||
c[(slice(None),) + TEETH] = SP.ID["Upper_Teeth"]
|
||||
return c
|
||||
|
||||
|
||||
class SourceBlurTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
B._CACHE.clear()
|
||||
self._seg = SP.segment
|
||||
SP.segment = fake_classes
|
||||
self.frames = torch.rand(3, H, W, 3)
|
||||
|
||||
def tearDown(self):
|
||||
SP.segment = self._seg
|
||||
|
||||
def mask(self, mode, **kw):
|
||||
return B.blur_mask(self.frames, mode, segmenter=kw.pop("segmenter", "Sapiens2"), **kw)
|
||||
|
||||
def test_saved_workflows_keep_their_values(self):
|
||||
for old in ("off", "face (keep mouth)", "head + hands (keep mouth)", "person (keep mouth)"):
|
||||
self.assertIn(old, B.SOURCE_BLUR)
|
||||
self.assertEqual(B.parse("masked area (mouth too)"), ("masked area", "mouth too"))
|
||||
self.assertIsNone(B.parse("off"))
|
||||
|
||||
def test_keep_lips_and_tongue_blurs_the_teeth(self):
|
||||
m = self.mask("face (keep lips + tongue)")
|
||||
self.assertEqual(float(m[0, 15, 25]), 1.0) # face
|
||||
self.assertEqual(float(m[0, 23, 32]), 0.0) # lips stay sharp
|
||||
self.assertEqual(float(m[0, 24, 32]), 0.0) # teeth are inside the lips' small grow here...
|
||||
big = fake_classes(self.frames)
|
||||
big[:, 26:29, 28:36] = SP.ID["Lower_Teeth"] # ...so check teeth away from the lips
|
||||
SP.segment = lambda *a, **k: big
|
||||
B._CACHE.clear()
|
||||
m = self.mask("face (keep lips + tongue)")
|
||||
self.assertEqual(float(m[0, 28, 32]), 1.0)
|
||||
m = self.mask("face (keep mouth)")
|
||||
self.assertEqual(float(m[0, 28, 32]), 0.0) # keep mouth keeps the teeth too
|
||||
|
||||
def test_mouth_too_blurs_everything_in_the_region(self):
|
||||
m = self.mask("face (mouth too)")
|
||||
self.assertEqual(float(m[0, 23, 32]), 1.0)
|
||||
self.assertEqual(float(m[0, 2, 2]), 0.0) # background untouched
|
||||
|
||||
def test_masked_area_uses_the_shot_mask_without_segmenting(self):
|
||||
SP.segment = lambda *a, **k: self.fail("masked area + mouth too must not segment")
|
||||
area = torch.zeros(3, H, W)
|
||||
area[:, 0:10, 0:10] = 1
|
||||
m = self.mask("masked area (mouth too)", area=area)
|
||||
self.assertEqual(float(m[0, 5, 5]), 1.0)
|
||||
self.assertEqual(float(m[0, 20, 30]), 0.0)
|
||||
|
||||
def test_masked_area_without_a_mask_blurs_the_whole_person(self):
|
||||
m = self.mask("masked area (mouth too)")
|
||||
self.assertEqual(float(m[0, 40, 15]), 1.0) # clothing = person
|
||||
|
||||
def test_invert_colours_inside_the_mask_only(self):
|
||||
x = torch.full((3, H, W, 3), 0.2)
|
||||
out = B.blur_source(x, "face (mouth too)", 1.0, segmenter="Sapiens2", effect="invert colours")
|
||||
self.assertAlmostEqual(float(out[0, 18, 30, 0]), 0.8, places=4)
|
||||
self.assertAlmostEqual(float(out[0, 2, 2, 0]), 0.2, places=4)
|
||||
|
||||
def test_sam_keeps_the_old_behaviour(self):
|
||||
SP.segment = lambda *a, **k: self.fail("SAM 3.1 + keep mouth must not run Sapiens2")
|
||||
_SL.SAM_CALLS.clear()
|
||||
m = self.mask("face (keep mouth)", segmenter="SAM 3.1")
|
||||
self.assertEqual(_SL.SAM_CALLS, ["face", "mouth"])
|
||||
self.assertEqual(float(m[0, 15, 25]), 1.0)
|
||||
self.assertEqual(float(m[0, 23, 32]), 0.0)
|
||||
|
||||
|
||||
class SapiensSizeTest(unittest.TestCase):
|
||||
def test_input_keeps_the_area_and_the_aspect(self):
|
||||
for h, w in ((1080, 1920), (1920, 1080), (512, 512)):
|
||||
ih, iw = SP.input_size(h, w)
|
||||
self.assertEqual((ih % 16, iw % 16), (0, 0))
|
||||
self.assertAlmostEqual(iw / ih, w / h, delta=0.03) # never stretched to 1024x768
|
||||
self.assertAlmostEqual(ih * iw / SP.PIXELS, 1.0, delta=0.04)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -6,6 +6,7 @@ import sys
|
||||
import tempfile
|
||||
import types
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
@@ -65,6 +66,109 @@ class CompareNodeTest(unittest.TestCase):
|
||||
n += 1
|
||||
self.assertEqual(n, 18, k)
|
||||
|
||||
def test_all_shots_keep_guide_and_target_order_across_chunk_boundaries(self):
|
||||
class Writer:
|
||||
videos = {}
|
||||
def __init__(self, path, *args, **kwargs):
|
||||
self.frames = self.videos.setdefault(Path(path).name, [])
|
||||
def write(self, frames):
|
||||
self.frames.append(frames.clone())
|
||||
def close(self):
|
||||
pass
|
||||
# Scene 1 red, scene 2 green: 22 frames span the 16-frame encoder chunk.
|
||||
guide = [torch.zeros(10, 32, 48, 3), torch.zeros(12, 32, 48, 3)]
|
||||
guide[0][..., 0] = 1
|
||||
guide[1][..., 1] = 1
|
||||
target = [x.clone() for x in guide]
|
||||
with patch.object(VC.VIO, "Mp4Writer", Writer):
|
||||
out = VC.BFSVideoCompare().compare(guide, target, preview_height=32)
|
||||
self.assertEqual(out["ui"]["bfs_compare"][0]["frames"], 22)
|
||||
self.assertEqual(out["result"][2], 0)
|
||||
for name in ("guide.mp4", "target.mp4"):
|
||||
frames = torch.cat(Writer.videos[name])
|
||||
self.assertEqual(frames.shape[0], 22)
|
||||
self.assertEqual(frames[:10, 0, 0].tolist(), [[1., 0., 0.]] * 10)
|
||||
self.assertEqual(frames[10:, 0, 0].tolist(), [[0., 1., 0.]] * 12)
|
||||
|
||||
def test_joined_video_can_match_multiple_equal_timing_shots(self):
|
||||
g = [torch.zeros(3, 32, 48, 3), torch.ones(4, 32, 48, 3)]
|
||||
self.assertEqual(VC.video_chunk(g, 2, 6, 32, 48)[:, 0, 0, 0].tolist(), [0., 1., 1., 1.])
|
||||
out = VC.BFSVideoCompare().compare(g, torch.cat(g), preview_height=32)
|
||||
self.assertEqual(out["ui"]["bfs_compare"][0]["frames"], 7)
|
||||
self.assertEqual(out["result"][2], 0)
|
||||
|
||||
def test_misaligned_shot_lists_require_join_timing_instead_of_silent_truncation(self):
|
||||
g = [torch.zeros(5, 32, 48, 3), torch.zeros(7, 32, 48, 3)]
|
||||
t = [torch.zeros(6, 32, 48, 3), torch.zeros(6, 32, 48, 3)]
|
||||
with self.assertRaisesRegex(ValueError, "SAME Join"):
|
||||
VC.BFSVideoCompare().compare(g, t)
|
||||
with self.assertRaisesRegex(ValueError, "SAME Join"):
|
||||
VC.BFSVideoCompare().compare(g, torch.zeros(8, 32, 48, 3))
|
||||
with self.assertRaisesRegex(ValueError, "nonempty"):
|
||||
VC.BFSVideoCompare().compare([], t)
|
||||
|
||||
def test_reference_pictures_go_to_the_player_no_taller_than_it(self):
|
||||
from PIL import Image
|
||||
guide, target = torch.rand(4, 32, 48, 3), torch.rand(4, 32, 48, 3)
|
||||
ref = torch.rand(6, 80, 40, 3) # taller than the preview
|
||||
ui = VC.BFSVideoCompare().compare(guide, target, preview_height=32, reference=ref)["ui"]["bfs_compare"][0]
|
||||
self.assertEqual([r["filename"] for r in ui["refs"]], [f"ref_{k}.png" for k in range(6)])
|
||||
im = Image.open(os.path.join(_TMP, ui["refs"][0]["subfolder"], "ref_0.png"))
|
||||
self.assertEqual(im.size, (16, 32))
|
||||
# a refs list as the planner / Unpack give it: pictures of different sizes, as ComfyUI hands a list input
|
||||
refs = [torch.rand(1, 64, 32, 3), torch.rand(1, 20, 60, 3)] + [torch.rand(1, 8, 8, 3)] * 12
|
||||
ui = VC.BFSVideoCompare().compare([guide], [target], preview_height=[32], reference=refs)["ui"]["bfs_compare"][0]
|
||||
self.assertEqual(len(ui["refs"]), 12) # at most 12
|
||||
sizes = [Image.open(os.path.join(_TMP, r["subfolder"], r["filename"])).size for r in ui["refs"][:2]]
|
||||
self.assertEqual(sizes, [(16, 32), (60, 20)])
|
||||
none = VC.BFSVideoCompare().compare(guide, target, preview_height=32)["ui"]["bfs_compare"][0]
|
||||
self.assertEqual(none["refs"], [])
|
||||
|
||||
|
||||
@unittest.skipUnless(VC.VIO.ffmpeg_bin(), "needs an ffmpeg program")
|
||||
class ExportTest(unittest.TestCase):
|
||||
"""The player's recording (WebM from MediaRecorder) converted for sharing."""
|
||||
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
import subprocess
|
||||
cls.src = os.path.join(_TMP, "rec_src.webm")
|
||||
# 20 s of noise at 720p with sound: far above 10 MB at the browser's bitrate
|
||||
subprocess.run([VC.VIO.ffmpeg_bin(), "-y", "-v", "error", "-f", "lavfi", "-i", "nullsrc=s=1280x720:r=30:d=20,geq=random(1)*255:128:128",
|
||||
"-f", "lavfi", "-i", "sine=frequency=440:duration=20", "-c:v", "libvpx", "-b:v", "16M",
|
||||
"-deadline", "realtime", "-cpu-used", "8", "-c:a", "libopus", "-shortest", cls.src], check=True)
|
||||
|
||||
def probe(self, path):
|
||||
import json
|
||||
import subprocess
|
||||
r = subprocess.run([VC.VIO.ffmpeg_bin().replace("ffmpeg", "ffprobe"), "-v", "error", "-show_streams", "-of", "json", path],
|
||||
capture_output=True, text=True)
|
||||
return {st["codec_type"]: st for st in json.loads(r.stdout)["streams"]}
|
||||
|
||||
def test_small_mp4_is_h264_aac_faststart_under_the_limit(self):
|
||||
self.assertGreater(os.path.getsize(self.src), VC.LIMIT_BYTES)
|
||||
out = VC.encode(self.src, "small", 20.0, os.path.join(_TMP, "exp_small"))
|
||||
self.assertLessEqual(os.path.getsize(out), VC.LIMIT_BYTES)
|
||||
st = self.probe(out)
|
||||
self.assertEqual((st["video"]["codec_name"], st["video"]["pix_fmt"], st["audio"]["codec_name"]), ("h264", "yuv420p", "aac"))
|
||||
head = open(out, "rb").read(256 * 1024)
|
||||
self.assertLess(head.find(b"moov"), head.find(b"mdat") if b"mdat" in head else 1 << 30) # faststart: moov first
|
||||
|
||||
def test_gif_fits_the_limit(self):
|
||||
out = VC.encode(self.src, "gif", 20.0, os.path.join(_TMP, "exp_gif"))
|
||||
self.assertTrue(out.endswith(".gif"))
|
||||
self.assertLessEqual(os.path.getsize(out), VC.LIMIT_BYTES)
|
||||
|
||||
def test_bad_ids_and_formats_are_refused(self):
|
||||
with self.assertRaises(ValueError):
|
||||
VC.write_part("../../etc", 0, b"x")
|
||||
with self.assertRaises(ValueError):
|
||||
VC.encode(self.src, "avi", 1.0, os.path.join(_TMP, "x"))
|
||||
rid = "abcdef0123"
|
||||
VC.write_part(rid, 0, b"12345")
|
||||
VC.write_part(rid, 5, b"678")
|
||||
self.assertEqual(open(os.path.join(_TMP, "bfs_compare", "exports", rid + ".rec"), "rb").read(), b"12345678")
|
||||
|
||||
|
||||
class NoFfmpegTest(unittest.TestCase):
|
||||
"""Windows portable installs often have no ffmpeg program: everything must go through PyAV."""
|
||||
|
||||
@@ -0,0 +1,194 @@
|
||||
import copy
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from types import SimpleNamespace
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0,str(ROOT))
|
||||
import minimax_h3_native_temporal as node
|
||||
|
||||
def load(name,path):
|
||||
s=importlib.util.spec_from_file_location(name,path);m=importlib.util.module_from_spec(s);sys.modules[name]=m;s.loader.exec_module(m);return m
|
||||
CORE=load('native_temporal_comfy_fixture',ROOT/'tests/fixtures/comfy_h3_layout_8cfe5e1.py')
|
||||
|
||||
def example(vt=7):
|
||||
refs=[dict(kind='image',latent_h=4,latent_w=8),dict(kind='video_audio',latent_h=8,latent_w=4,latent_t=vt,ref_audio_t=3)]
|
||||
l=CORE.PackedLayout(3,7,8,8,2,refs=refs)
|
||||
return l,refs
|
||||
|
||||
def test_layout_matches_actual_training_coordinates():
|
||||
l,refs=example();before=l.position_ids.clone()
|
||||
aligned,src,tgt=node.temporal_layout(l,refs,None)
|
||||
golden=json.loads((ROOT/'tests/fixtures/h3_native_temporal_training_positions.json').read_text())
|
||||
torch.testing.assert_close(aligned.position_ids,torch.tensor(golden['position_ids'],dtype=torch.float64),rtol=0,atol=1e-12)
|
||||
assert src.tolist()==golden['source']
|
||||
assert torch.equal(l.position_ids,before)
|
||||
assert torch.equal(aligned.position_ids[~src],before[~src])
|
||||
assert torch.equal(aligned.position_ids[src,1:],before[src,1:])
|
||||
assert not (src&tgt).any()
|
||||
|
||||
def test_wrong_video_index_and_temporal_mismatch_rejected():
|
||||
l,r=example()
|
||||
with pytest.raises(ValueError,match='Video 2'):node.temporal_layout(l,r,None,2)
|
||||
l,r=example(6)
|
||||
with pytest.raises(ValueError,match='identical temporal'):node.temporal_layout(l,r,None)
|
||||
with pytest.raises(ValueError,match='Add Guide'):node.temporal_layout(l,r,[{}])
|
||||
|
||||
def context():
|
||||
l,r=example();a,s,t=node.temporal_layout(l,r,None)
|
||||
return node.core.TemporalContext(a.position_ids[:,0].float()[None]/40,s[None],t[None],torch.ones_like(s[None]),2,.2,.25,6)
|
||||
|
||||
def test_attention_adapter_matches_dense_score_policy_and_shapes():
|
||||
ctx=context();torch.manual_seed(12)
|
||||
q,k,v=[torch.randn(1,2,ctx.times.shape[1],8) for _ in range(3)]
|
||||
def forbidden(*args,**kwargs):raise AssertionError('wrong backend')
|
||||
out=node.attention_override(ctx)(forbidden,q,k,v,2,skip_reshape=True)
|
||||
expected=node.core.dense_reference(q,k,v,ctx).transpose(1,2).flatten(2)
|
||||
torch.testing.assert_close(out,expected)
|
||||
raw=[x.transpose(1,2).flatten(2) for x in (q,k,v)]
|
||||
torch.testing.assert_close(node.attention_override(ctx)(forbidden,*raw,2),expected)
|
||||
|
||||
def test_token_refiner_uses_existing_attention():
|
||||
ctx=context();q=torch.randn(1,2,3,8)
|
||||
def original(*args,**kwargs):return 'original'
|
||||
assert node.attention_override(ctx)(original,q,q,q,2,skip_reshape=True)=='original'
|
||||
|
||||
class Model:
|
||||
def __init__(self):self.model_options={}
|
||||
def clone(self):m=Model();m.model_options=copy.deepcopy(self.model_options);return m
|
||||
def get_model_object(self,key):return type('MiniMaxH3Model',(),{})()
|
||||
def set_model_unet_function_wrapper(self,w):self.model_options['model_function_wrapper']=w
|
||||
|
||||
def test_per_model_wrapper_clones_payload_and_applies_attention():
|
||||
m=Model();patched=node.patch_temporal_model(m);assert not m.model_options
|
||||
l,r=example();payload=dict(layout=l,refs=r)
|
||||
def fn(x,t,**c):
|
||||
assert c['minimax_payload']['layout'] is not l
|
||||
assert 'optimized_attention_override' in c['transformer_options']
|
||||
return 'ok'
|
||||
args=dict(input=torch.zeros(1),timestep=torch.ones(1),c=dict(minimax_payload=payload))
|
||||
assert patched.model_options['model_function_wrapper'](fn,args)=='ok'
|
||||
assert payload['layout'] is l and 'transformer_options' not in args['c']
|
||||
with pytest.raises(ValueError,match='already installed'):node.patch_temporal_model(patched)
|
||||
|
||||
def test_incompatible_attention_rejected():
|
||||
m=Model();m.model_options={'transformer_options':{'optimized_attention_override':object()}}
|
||||
with pytest.raises(ValueError,match='VSA'):node.patch_temporal_model(m)
|
||||
with pytest.raises(ValueError,match='MODEL input'):node.patch_temporal_model(None)
|
||||
|
||||
|
||||
def test_second_video_selection_preserves_first_video_and_images():
|
||||
refs=[dict(kind='image',latent_h=4,latent_w=8),
|
||||
dict(kind='video',latent_h=4,latent_w=4,latent_t=3,ref_audio_t=0),
|
||||
dict(kind='video',latent_h=8,latent_w=4,latent_t=7,ref_audio_t=0)]
|
||||
native=CORE.PackedLayout(3,7,8,8,2,refs=refs)
|
||||
aligned,source,target=node.temporal_layout(native,refs,None,2)
|
||||
assert int(source.sum())==56
|
||||
assert torch.equal(aligned.position_ids[~source],native.position_ids[~source])
|
||||
assert not source[3:23].any()
|
||||
|
||||
@pytest.mark.parametrize('install_after_temporal',[False,True])
|
||||
@pytest.mark.parametrize('dense_name,expected_backend',[('comfy_kitchen_int8',2),('pytorch',0)])
|
||||
def test_model_attention_backend_is_honored_without_manual_selection(monkeypatch,install_after_temporal,dense_name,expected_backend):
|
||||
from types import ModuleType
|
||||
mod=ModuleType('comfy.ldm.modules.attention')
|
||||
def dense(*args,**kwargs):return 'kitchen'
|
||||
mod.get_attention_function=lambda name,default=None:dense if name==dense_name else default
|
||||
monkeypatch.setitem(sys.modules,'comfy.ldm.modules.attention',mod)
|
||||
def setter(optimized_attention):
|
||||
def optimized_attention_override(_, *args, **kwargs):return optimized_attention(*args,**kwargs)
|
||||
optimized_attention_override.__module__='comfy.model_patcher'
|
||||
optimized_attention_override.__qualname__='ModelPatcher.set_model_optimized_attention.<locals>.optimized_attention_override'
|
||||
return optimized_attention_override
|
||||
incoming=setter(dense)
|
||||
m=Model()
|
||||
if not install_after_temporal:m.model_options={'transformer_options':{'optimized_attention_override':incoming}}
|
||||
patched=node.patch_temporal_model(m)
|
||||
selected=[]
|
||||
def capture(ctx,previous=None,backend=None,**kwargs):
|
||||
selected.append((backend,previous));return 'temporal attention'
|
||||
monkeypatch.setattr(node,'attention_override',capture)
|
||||
l,r=example()
|
||||
args=dict(input=torch.zeros(1),timestep=torch.ones(1),c=dict(minimax_payload=dict(layout=l,refs=r),transformer_options={'optimized_attention_override':incoming}))
|
||||
def fn(x,t,**c):
|
||||
assert c['transformer_options']['optimized_attention_override']=='temporal attention'
|
||||
return 'ok'
|
||||
assert patched.model_options['model_function_wrapper'](fn,args)=='ok'
|
||||
assert selected==[(node.BACKENDS[expected_backend],incoming)]
|
||||
|
||||
@pytest.mark.parametrize('hidden_by_later_dense_node',[False,True])
|
||||
def test_sol_node_owned_producer_is_bypassed_but_temporal_override_remains(monkeypatch,hidden_by_later_dense_node):
|
||||
Patch=type('SparseAttnPatch',(),{'__module__':'comfy_extras.nodes_sparse_attention'})
|
||||
patch=Patch();patch.vsa=False;patch.topk_ratio=0.
|
||||
previous=None
|
||||
def sparse_override(*args):return patch,previous,args
|
||||
sparse_override.__module__='comfy_extras.nodes_sparse_attention'
|
||||
sparse_override.__qualname__='make_attention_override.<locals>.override'
|
||||
def block_patch(*args):return patch,args
|
||||
block_patch.__module__=sparse_override.__module__
|
||||
block_patch.__qualname__='make_h3_block_patch.<locals>.block_patch'
|
||||
m=Model();m.model_options={'transformer_options':{'optimized_attention_override':None if hidden_by_later_dense_node else sparse_override,'patches_replace':{'dit':{'block':block_patch}}}}
|
||||
patched=node.patch_temporal_model(m)
|
||||
selected=[]
|
||||
def capture(ctx,previous=None,backend=None,sparse_patch=None,**kw):
|
||||
selected.append((backend,sparse_patch));return 'temporal sol hook'
|
||||
monkeypatch.setattr(node,'attention_override',capture)
|
||||
l,r=example();opts=patched.model_options['transformer_options']
|
||||
args=dict(input=torch.zeros(1),timestep=torch.ones(1),c=dict(minimax_payload=dict(layout=l,refs=r),transformer_options=opts))
|
||||
def fn(x,t,**c):
|
||||
assert c['transformer_options']['patches_replace']['dit']=={}
|
||||
assert c['transformer_options']['optimized_attention_override']=='temporal sol hook'
|
||||
return 'ok'
|
||||
assert patched.model_options['model_function_wrapper'](fn,args)=='ok'
|
||||
assert selected==[(node.SOL_BACKEND,patch)]
|
||||
assert m.model_options['transformer_options']['patches_replace']['dit']
|
||||
|
||||
|
||||
def test_sol_dense_window_preserves_temporal_policy(monkeypatch,capsys):
|
||||
l,r=example();aligned,src,tgt=node.temporal_layout(l,r,None)
|
||||
ctx=node.core.TemporalContext(aligned.position_ids[:,0].float()[None]/40,src[None],tgt[None],torch.ones(1,l.seq_len,dtype=torch.bool),2.,.2,.25,6.)
|
||||
plan=node.fast.make_plan(aligned.position_ids,src,tgt,2.,.2,.25,6.)
|
||||
patch=SimpleNamespace(dense_reason=lambda *a:'outside sigma window',log_once=lambda *a:None)
|
||||
q,k,v=[torch.randn(1,2,l.seq_len,128) for _ in range(3)]
|
||||
monkeypatch.setattr(node.fast,'attention',lambda q,k,v,p:node.core.dense_reference(q,k,v,ctx))
|
||||
expected=node.core.dense_reference(q,k,v,ctx)
|
||||
out=node.attention_override(ctx,backend=node.SOL_BACKEND,plan=plan,sparse_patch=patch,dense_backend=node.BACKENDS[0],log_first=True)(None,q,k,v,2,skip_reshape=True,skip_output_reshape=True)
|
||||
torch.testing.assert_close(out,expected)
|
||||
assert 'dense fallback: outside sigma window' in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_sol_sparse_window_runs_temporal_kernel_and_logs_activation(monkeypatch,capsys):
|
||||
from types import ModuleType
|
||||
ck=ModuleType('comfy_kitchen');ck.sol_attn_is_available=lambda device:True
|
||||
monkeypatch.setitem(sys.modules,'comfy_kitchen',ck)
|
||||
l,r=example();aligned,src,tgt=node.temporal_layout(l,r,None)
|
||||
ctx=node.core.TemporalContext(aligned.position_ids[:,0].float()[None]/40,src[None],tgt[None],torch.ones(1,l.seq_len,dtype=torch.bool),2.,.2,.25,6.)
|
||||
plan=node.fast.make_plan(aligned.position_ids,src,tgt,2.,.2,.25,6.)
|
||||
patch=SimpleNamespace(dense_reason=lambda *a:None,sinks=lambda *a:((0,0),(0,0)),tau=1.3,extra_tokens=256)
|
||||
actual_sol=node.sol.attention
|
||||
calls=[]
|
||||
def oracle(q,k,v,key_bias=None,**kw):
|
||||
calls.append(key_bias)
|
||||
mask=None if key_bias is None else key_bias.view(1,1,1,-1)
|
||||
return torch.nn.functional.scaled_dot_product_attention(q.transpose(1,2),k.transpose(1,2),v.transpose(1,2),attn_mask=mask).transpose(1,2)
|
||||
monkeypatch.setattr(node.sol,'attention',lambda *a,**kw:actual_sol(*a,**kw,kernel=oracle))
|
||||
q,k,v=[torch.randn(1,2,l.seq_len,128) for _ in range(3)]
|
||||
expected=node.core.dense_reference(q,k,v,ctx)
|
||||
out=node.attention_override(ctx,backend=node.SOL_BACKEND,plan=plan,sparse_patch=patch,dense_backend=node.BACKENDS[0])(None,q,k,v,2,skip_reshape=True,skip_output_reshape=True)
|
||||
torch.testing.assert_close(out,expected,atol=1e-6,rtol=1e-5)
|
||||
assert len(calls)==plan.frames+1
|
||||
assert 'sol-attn ACTIVE' in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_node_prepares_selected_kitchen_before_sampling_but_not_flash(monkeypatch):
|
||||
calls=[]
|
||||
monkeypatch.setattr(node.sol.native_kernel,'prepare_model',lambda model:calls.append(model))
|
||||
m=Model();m.load_device=torch.device('cuda:0')
|
||||
node.patch_temporal_model(m,backend=node.BACKENDS[0])
|
||||
assert calls==[]
|
||||
node.patch_temporal_model(m,backend=node.BACKENDS[2])
|
||||
assert calls==[m]
|
||||
@@ -0,0 +1,105 @@
|
||||
import concurrent.futures,hashlib,io,sys
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
import pytest
|
||||
sys.path.insert(0,str(Path(__file__).resolve().parents[1]))
|
||||
import minimax_h3_temporal_bootstrap as boot
|
||||
|
||||
|
||||
class NativeFunction:
|
||||
def __init__(self,result):self.result=result
|
||||
def __call__(self):return self.result
|
||||
|
||||
@pytest.fixture
|
||||
def setup(monkeypatch,tmp_path):
|
||||
monkeypatch.setenv('BFS_H3_TEMPORAL_CACHE_DIR',str(tmp_path))
|
||||
boot._READY.clear();boot._FAILED.clear()
|
||||
source=b"VERSION='h3_native_temporal_kernel_v1'\ndef is_available(device):return True\n"
|
||||
binary=b'fake verified native fixture'
|
||||
assets=[('temporal_attention.py','https://test.invalid/api',hashlib.sha256(source).hexdigest(),1024),('backends/cuda/native_temporal/libck_temporal_sm120.so','https://test.invalid/bin',hashlib.sha256(binary).hexdigest(),1024)]
|
||||
monkeypatch.setattr(boot,'ASSETS',tuple(assets))
|
||||
requests=[]
|
||||
def urlopen(request,timeout):
|
||||
requests.append(request.full_url)
|
||||
return io.BytesIO(source if request.full_url.endswith('/api') else binary)
|
||||
monkeypatch.setattr(boot.urllib.request,'urlopen',urlopen)
|
||||
loads=[]
|
||||
def cdll(path):
|
||||
loads.append(path)
|
||||
return SimpleNamespace(ck_temporal_abi_version=NativeFunction(1),ck_temporal_arch=NativeFunction(120))
|
||||
monkeypatch.setattr(boot.ctypes,'CDLL',cdll)
|
||||
monkeypatch.setattr(boot,'compatible',lambda d:(True,None))
|
||||
yield SimpleNamespace(root=boot.cache_root(),requests=requests,loads=loads,source=source,binary=binary)
|
||||
boot._READY.clear();boot._FAILED.clear()
|
||||
|
||||
|
||||
def test_first_use_installs_once_and_preserves_current_environment(setup):
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
|
||||
results=list(pool.map(boot.ensure,['cuda:0']*4))
|
||||
assert all(m is results[0] for m in results)
|
||||
assert len(setup.requests)==2 and len(setup.loads)==1
|
||||
assert (setup.root/'temporal_attention.py').read_bytes()==setup.source
|
||||
assert not list(setup.root.rglob('*.partial'))
|
||||
assert str(results[0].__file__).startswith(str(setup.root))
|
||||
|
||||
|
||||
def test_cached_bytes_work_without_network_after_restart(setup,monkeypatch):
|
||||
assert boot.ensure('cuda:0') is not None
|
||||
boot._READY.clear();boot._FAILED.clear()
|
||||
def offline(*a,**kw):raise AssertionError('valid cache must not access the network')
|
||||
monkeypatch.setattr(boot.urllib.request,'urlopen',offline)
|
||||
assert boot.ensure('cuda:0') is not None
|
||||
|
||||
|
||||
def test_corrupted_download_never_loads_or_executes_code(setup,monkeypatch):
|
||||
monkeypatch.setattr(boot.urllib.request,'urlopen',lambda *a,**kw:io.BytesIO(b'wrong bytes'))
|
||||
assert boot.ensure('cuda:0') is None
|
||||
assert setup.loads==[]
|
||||
assert not (setup.root/'temporal_attention.py').exists()
|
||||
assert not list(setup.root.rglob('*.partial'))
|
||||
|
||||
|
||||
def test_failure_is_not_retried_every_attention_block(setup,monkeypatch):
|
||||
calls=[]
|
||||
def broken(*a,**kw):calls.append(1);raise OSError('offline')
|
||||
monkeypatch.setattr(boot.urllib.request,'urlopen',broken)
|
||||
assert boot.ensure('cuda:0') is None
|
||||
assert boot.ensure('cuda:0') is None
|
||||
assert len(calls)==1
|
||||
|
||||
|
||||
def test_bad_existing_cache_is_replaced_only_after_verified_download(setup):
|
||||
setup.root.mkdir(parents=True)
|
||||
(setup.root/'temporal_attention.py').write_text('corrupt')
|
||||
assert boot.ensure('cuda:0') is not None
|
||||
assert (setup.root/'temporal_attention.py').read_bytes()==setup.source
|
||||
|
||||
|
||||
def test_size_limit_aborts_and_cleans_partial_file(tmp_path,monkeypatch):
|
||||
monkeypatch.setattr(boot.urllib.request,'urlopen',lambda *a,**kw:io.BytesIO(b'12345'))
|
||||
with pytest.raises(RuntimeError,match='size limit'):boot.download(tmp_path/'asset','https://test.invalid/asset','ignored',2)
|
||||
assert list(tmp_path.iterdir())==[]
|
||||
|
||||
|
||||
def test_unsupported_device_keeps_existing_backend_without_download(setup,monkeypatch):
|
||||
monkeypatch.setattr(boot,'compatible',lambda d:(False,'unsupported GPU'))
|
||||
assert boot.ensure('cuda:0') is None
|
||||
assert setup.requests==[]
|
||||
|
||||
|
||||
def test_device_failure_does_not_block_supported_device(setup,monkeypatch):
|
||||
monkeypatch.setattr(boot,'compatible',lambda d:(d=='cuda:1','unsupported GPU'))
|
||||
assert boot.ensure('cuda:0') is None
|
||||
assert boot.ensure('cuda:1') is not None
|
||||
|
||||
|
||||
def test_opt_out_is_checked_before_gpu_or_network(monkeypatch):
|
||||
monkeypatch.setenv('BFS_H3_TEMPORAL_AUTO_INSTALL','0')
|
||||
assert boot.compatible('cuda:0')==(False,'automatic setup disabled')
|
||||
|
||||
|
||||
def test_bad_binary_arch_is_rejected_before_python_api_execution(setup,monkeypatch):
|
||||
before=set(sys.modules)
|
||||
monkeypatch.setattr(boot.ctypes,'CDLL',lambda p:SimpleNamespace(ck_temporal_abi_version=NativeFunction(1),ck_temporal_arch=NativeFunction(89)))
|
||||
assert boot.ensure('cuda:0') is None
|
||||
assert not [n for n in set(sys.modules)-before if n.startswith('_bfs_ck_temporal')]
|
||||
@@ -0,0 +1,29 @@
|
||||
import sys
|
||||
from pathlib import Path
|
||||
import torch
|
||||
sys.path.insert(0,str(Path(__file__).resolve().parents[1]))
|
||||
import minimax_h3_temporal_fast as fast
|
||||
import minimax_h3_native_temporal_core as core
|
||||
|
||||
|
||||
def oracle(q,k,v):
|
||||
scores=q.double()@k.double().transpose(-1,-2)/q.shape[-1]**.5
|
||||
return (scores.softmax(-1)@v.double()).to(q.dtype),scores.logsumexp(-1).float()
|
||||
|
||||
|
||||
def test_exact_partition_matches_original_temporal_policy():
|
||||
torch.manual_seed(5)
|
||||
frames=7;sf=2;tf=3;n=4+frames*sf+2+frames*tf
|
||||
pos=torch.zeros(n,3,dtype=torch.float64)
|
||||
times=torch.tensor([0,1,5,9,13,17,18],dtype=torch.float64)*5/3
|
||||
pos[4:18,0]=times.repeat_interleave(sf)
|
||||
pos[20:,0]=times.repeat_interleave(tf)
|
||||
source=torch.zeros(n,dtype=torch.bool);source[4:18]=True
|
||||
target=torch.zeros(n,dtype=torch.bool);target[20:]=True
|
||||
q,k,v=[torch.randn(1,2,n,8) for _ in range(3)]
|
||||
for strength,cap in [(2.,6.),(0.,6.),(2.,0.),(10.,6.)]:
|
||||
plan=fast.make_plan(pos,source,target,strength,.2,.25,cap)
|
||||
ctx=core.TemporalContext(pos[:,0].float()[None]/40,source[None],target[None],torch.ones(1,n,dtype=torch.bool),strength,.2,.25,cap)
|
||||
actual=fast.attention(q,k,v,plan,flash=oracle,frame_chunk=3)
|
||||
expected=core.dense_reference(q,k,v,ctx)
|
||||
torch.testing.assert_close(actual,expected,atol=1e-6,rtol=1e-5)
|
||||
@@ -0,0 +1,55 @@
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from types import ModuleType,SimpleNamespace
|
||||
import torch
|
||||
import pytest
|
||||
sys.path.insert(0,str(Path(__file__).resolve().parents[1]))
|
||||
import minimax_h3_temporal_kernel as bridge
|
||||
|
||||
|
||||
def fake_api(monkeypatch):
|
||||
api=ModuleType('comfy_kitchen.temporal_attention');api.VERSION='h3_native_temporal_kernel_v1';api.is_available=lambda d:True
|
||||
records=[]
|
||||
api.TemporalPolicy=lambda *args:args
|
||||
def dense(q,k,v,p,**kw):records.append(('dense',p,kw));return torch.ones(1,dtype=torch.bfloat16)
|
||||
def sol(q,k,v,p,**kw):records.append(('sol',p,kw));return torch.ones(1,dtype=torch.bfloat16)
|
||||
api.int8_attention=dense;api.sol_attn=sol
|
||||
monkeypatch.setitem(sys.modules,'comfy_kitchen.temporal_attention',api)
|
||||
return api,records
|
||||
|
||||
|
||||
def test_bridge_uses_one_native_call_and_preserves_policy(monkeypatch):
|
||||
api,records=fake_api(monkeypatch)
|
||||
q=SimpleNamespace(device=torch.device('cuda'),dtype=torch.bfloat16)
|
||||
plan=SimpleNamespace(source_start=4,source_stop=10,target_start=12,target_stop=21,source_frame_rows=2,target_frame_rows=3,frame_times=(0.,.7,1.4))
|
||||
ctx=SimpleNamespace(strength=2.,window=.2,falloff=.25,cap=6.)
|
||||
bridge.run('dense',q,q,q,plan,ctx)
|
||||
bridge.run('sol',q,q,q,plan,ctx,tau=1.3,token_aug=256,sink_q=(0,1))
|
||||
assert len(records)==2 and [r[0] for r in records]==['dense','sol']
|
||||
assert records[0][1]==(4,10,12,21,2,3,(0.,.7,1.4),2.,.2,.25,6.)
|
||||
assert records[1][2]=={'tau':1.3,'token_aug':256,'sink_q':(0,1)}
|
||||
ctx.strength=3.
|
||||
assert bridge.policy_for(api,plan,ctx)[7]==3.
|
||||
|
||||
|
||||
def test_bridge_rejects_wrong_extension_version(monkeypatch):
|
||||
api,_=fake_api(monkeypatch);api.VERSION='unknown'
|
||||
q=SimpleNamespace(device=torch.device('cuda'))
|
||||
with pytest.raises(RuntimeError,match='version'):bridge.api_for(q)
|
||||
|
||||
|
||||
def test_cpu_and_missing_binary_keep_compatibility_path(monkeypatch):
|
||||
assert bridge.api_for(torch.zeros(1)) is None
|
||||
api,_=fake_api(monkeypatch);api.is_available=lambda d:False
|
||||
monkeypatch.setattr(bridge.bootstrap,'ensure',lambda d:None)
|
||||
q=SimpleNamespace(device=torch.device('cuda'))
|
||||
assert bridge.api_for(q) is None
|
||||
|
||||
|
||||
def test_prepare_model_runs_before_sampling_when_cuda_is_selected(monkeypatch):
|
||||
calls=[]
|
||||
monkeypatch.setattr(bridge,'api_for',lambda q:calls.append(q.device) or 'ready')
|
||||
assert bridge.prepare_model(SimpleNamespace(load_device=torch.device('cuda:0'))) == 'ready'
|
||||
assert calls==[torch.device('cuda:0')]
|
||||
assert bridge.prepare_model(SimpleNamespace(load_device=torch.device('cpu'))) is None
|
||||
assert bridge.prepare_model(SimpleNamespace()) is None
|
||||
@@ -0,0 +1,55 @@
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
import torch
|
||||
import pytest
|
||||
sys.path.insert(0,str(Path(__file__).resolve().parents[1]))
|
||||
import minimax_h3_temporal_kitchen as kitchen
|
||||
import minimax_h3_temporal_fast as fast
|
||||
import minimax_h3_native_temporal_core as core
|
||||
import minimax_h3_native_temporal as node
|
||||
|
||||
@pytest.mark.parametrize('strength,cap',[(2.,6.),(0.,6.),(2.,0.),(10.,6.)])
|
||||
def test_frame_bias_matches_full_temporal_rule(strength,cap):
|
||||
torch.manual_seed(42)
|
||||
frames,sf,tf=7,2,3
|
||||
n=4+frames*sf+2+frames*tf+3
|
||||
pos=torch.zeros(n,3,dtype=torch.float64)
|
||||
times=torch.tensor([0,1,5,9,13,17,18],dtype=torch.float64)*5/3
|
||||
pos[4:18,0]=times.repeat_interleave(sf);pos[20:41,0]=times.repeat_interleave(tf)
|
||||
source=torch.zeros(n,dtype=torch.bool);source[4:18]=True
|
||||
target=torch.zeros(n,dtype=torch.bool);target[20:41]=True
|
||||
plan=fast.make_plan(pos,source,target,strength,.2,.25,cap)
|
||||
ctx=core.TemporalContext(pos[:,0].float()[None]/40,source[None],target[None],torch.ones(1,n,dtype=torch.bool),strength,.2,.25,cap)
|
||||
q,k,v=[torch.randn(1,2,n,8) for _ in range(3)]
|
||||
calls=[]
|
||||
def oracle(q,k,v,attn_mask=None):
|
||||
if attn_mask is not None:
|
||||
assert attn_mask.shape==(1,1,1,n)
|
||||
calls.append(attn_mask)
|
||||
return torch.nn.functional.scaled_dot_product_attention(q,k,v,attn_mask=attn_mask)
|
||||
actual=kitchen.attention(q,k,v,plan,ctx,kernel=oracle)
|
||||
expected=core.dense_reference(q,k,v,ctx)
|
||||
torch.testing.assert_close(actual,expected,atol=1e-6,rtol=1e-5)
|
||||
torch.testing.assert_close(kitchen.attention(q,k,v,plan,ctx,kernel=oracle),actual)
|
||||
if strength and cap:
|
||||
assert len(calls)==2*frames
|
||||
assert plan._kitchen_biases.numel()==frames*n
|
||||
assert torch.all(plan._kitchen_biases[:,~source]==0)
|
||||
|
||||
|
||||
def test_detects_only_official_model_attention_backend(monkeypatch):
|
||||
mod=ModuleType('comfy.ldm.modules.attention')
|
||||
kernels={'pytorch':lambda:None,'comfy_kitchen_int8':lambda:None}
|
||||
mod.get_attention_function=lambda name,default=None:kernels.get(name,default)
|
||||
monkeypatch.setitem(sys.modules,'comfy.ldm.modules.attention',mod)
|
||||
def setter(optimized_attention):
|
||||
def optimized_attention_override(_, *args, **kwargs):
|
||||
return optimized_attention(*args,**kwargs)
|
||||
optimized_attention_override.__module__='comfy.model_patcher'
|
||||
optimized_attention_override.__qualname__='ModelPatcher.set_model_optimized_attention.<locals>.optimized_attention_override'
|
||||
return optimized_attention_override
|
||||
for key,fn in kernels.items():assert node.dense_backend_selection(setter(fn))==key
|
||||
assert node.dense_backend_selection(None) is None
|
||||
with pytest.raises(ValueError):node.dense_backend_selection(lambda *args:None)
|
||||
with pytest.raises(ValueError):node.dense_backend_selection(setter(lambda:None))
|
||||
@@ -0,0 +1,107 @@
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
import torch
|
||||
import pytest
|
||||
sys.path.insert(0,str(Path(__file__).resolve().parents[1]))
|
||||
import minimax_h3_temporal_sol as sol
|
||||
import minimax_h3_temporal_fast as fast
|
||||
import minimax_h3_native_temporal_core as core
|
||||
import minimax_h3_sparse_compat as compat
|
||||
|
||||
@pytest.mark.parametrize('strength,cap',[(2.,6.),(0.,6.),(2.,0.),(10.,6.)])
|
||||
def test_temporal_sol_bias_matches_dense_policy(strength,cap):
|
||||
torch.manual_seed(14);frames,sf,tf=7,2,3;n=44
|
||||
pos=torch.zeros(n,3,dtype=torch.float64)
|
||||
times=torch.tensor([0,1,5,9,13,17,18],dtype=torch.float64)*5/3
|
||||
pos[4:18,0]=times.repeat_interleave(sf);pos[20:41,0]=times.repeat_interleave(tf)
|
||||
source=torch.zeros(n,dtype=torch.bool);source[4:18]=True
|
||||
target=torch.zeros(n,dtype=torch.bool);target[20:41]=True
|
||||
plan=fast.make_plan(pos,source,target,strength,.2,.25,cap)
|
||||
ctx=core.TemporalContext(pos[:,0].float()[None]/40,source[None],target[None],torch.ones(1,n,dtype=torch.bool),strength,.2,.25,cap)
|
||||
q,k,v=[torch.randn(1,2,n,128) for _ in range(3)]
|
||||
calls=[]
|
||||
def oracle(q,k,v,key_bias=None,**kw):
|
||||
assert kw['sink_blocks'][1]*64>=plan.source_stop
|
||||
assert kw['topk_ratio']==0. and kw['tail'] is True
|
||||
if key_bias is not None:
|
||||
assert key_bias.shape==(n,)
|
||||
assert torch.all(key_bias[~source]==0)
|
||||
calls.append(key_bias)
|
||||
return torch.nn.functional.scaled_dot_product_attention(q.transpose(1,2),k.transpose(1,2),v.transpose(1,2),attn_mask=None if key_bias is None else key_bias.view(1,1,1,-1)).transpose(1,2)
|
||||
actual=sol.attention(q,k,v,plan,ctx,tau=1.3,kernel=oracle)
|
||||
expected=core.dense_reference(q,k,v,ctx)
|
||||
torch.testing.assert_close(actual,expected,atol=1e-6,rtol=1e-5)
|
||||
assert len(calls)==(frames+1 if strength and cap else 1)
|
||||
# Repeat uses the compact cached biases and does not change the policy.
|
||||
torch.testing.assert_close(sol.attention(q,k,v,plan,ctx,tau=1.3,kernel=oracle),actual)
|
||||
|
||||
|
||||
def make_sparse(vsa=False,topk=0.,module='comfy_extras.nodes_sparse_attention'):
|
||||
SparseAttnPatch=type('SparseAttnPatch',(),{'__module__':module})
|
||||
patch=SparseAttnPatch();patch.vsa=vsa;patch.topk_ratio=topk
|
||||
previous=None
|
||||
def override(*args):return patch,previous,args
|
||||
override.__module__=module
|
||||
override.__qualname__='make_attention_override.<locals>.override'
|
||||
def block_patch(*args):return patch,args
|
||||
block_patch.__module__=override.__module__
|
||||
block_patch.__qualname__='make_h3_block_patch.<locals>.block_patch'
|
||||
return patch,override,block_patch
|
||||
|
||||
|
||||
def test_only_sol_owned_hooks_are_bypassed_without_mutating_model():
|
||||
patch,override,block=make_sparse()
|
||||
options={'optimized_attention_override':override,'patches_replace':{'dit':{('double_block',0):block},'other':{'keep':1}}}
|
||||
dense,actual=compat.selection(options)
|
||||
assert dense is None and actual is patch
|
||||
copied=compat.without_owned_blocks(options,patch)
|
||||
assert copied['patches_replace']['dit']=={}
|
||||
assert copied['patches_replace']['other']=={'keep':1}
|
||||
assert options['patches_replace']['dit']
|
||||
# A dense selection applied after Sparse can hide the sparse closure; recover
|
||||
# the exact same state from the node's owned H3 block hook.
|
||||
hidden={**options,'optimized_attention_override':None}
|
||||
assert compat.selection(hidden)==(None,patch)
|
||||
unknown={**options,'patches_replace':{'dit':{'unknown':lambda:None}}}
|
||||
with pytest.raises(ValueError,match='another attention/block'):compat.selection(unknown)
|
||||
|
||||
@pytest.mark.parametrize('vsa,topk',[(True,0.),(False,.1)])
|
||||
def test_vsa_and_sla_still_rejected(vsa,topk):
|
||||
_,override,block=make_sparse(vsa,topk)
|
||||
with pytest.raises(ValueError,match='only sol-attn'):compat.selection({'optimized_attention_override':override})
|
||||
with pytest.raises(ValueError,match='only sol-attn'):compat.selection({'patches_replace':{'dit':{'block':block}}})
|
||||
|
||||
|
||||
@pytest.mark.parametrize('module',[
|
||||
'/home/alissonerdx/tools/ComfyUI/comfy_extras/nodes_sparse_attention',
|
||||
'C:\\ComfyUI\\comfy_extras\\nodes_sparse_attention',
|
||||
'nodes_sparse_attention',
|
||||
'comfy_extras.nodes_sparse_attention',
|
||||
])
|
||||
def test_comfy_file_loader_module_names_are_recognized(module):
|
||||
patch,override,block=make_sparse(module=module)
|
||||
options={'optimized_attention_override':override,'patches_replace':{'dit':{'block':block}}}
|
||||
assert compat.selection(options)==(None,patch)
|
||||
hidden={**options,'optimized_attention_override':None}
|
||||
assert compat.selection(hidden)==(None,patch)
|
||||
assert compat.without_owned_blocks(options,patch)['patches_replace']['dit']=={}
|
||||
|
||||
@pytest.mark.parametrize('module',['other_sparse_attention','nodes_sparse_attention_fake','/tmp/not_nodes_sparse_attention'])
|
||||
def test_similar_unrelated_module_names_are_rejected(module):
|
||||
_,override,block=make_sparse(module=module)
|
||||
with pytest.raises(ValueError):compat.selection({'optimized_attention_override':override,'patches_replace':{'dit':{'block':block}}})
|
||||
|
||||
|
||||
def test_actual_comfy_sparse_factories_under_file_loader_name():
|
||||
module='/home/alissonerdx/tools/ComfyUI/comfy_extras/nodes_sparse_attention'
|
||||
fixture=Path(__file__).parent/'fixtures/comfy_sparse_node_factories.py'
|
||||
namespace={'__name__':module}
|
||||
exec(compile(fixture.read_text(),str(fixture),'exec'),namespace)
|
||||
patch=namespace['SparseAttnPatch'](tau=1.3,topk_ratio=0.,vsa=False,sigma_start=1.,sigma_end=0.,min_tokens=12288,dense_blocks=set(),sink_conditioning='exact_kv_and_rows',extra_tokens=256,verbose=False)
|
||||
override=namespace['make_attention_override'](patch,None)
|
||||
block=namespace['make_h3_block_patch'](SimpleNamespace(),0,patch)
|
||||
options={'optimized_attention_override':override,'patches_replace':{'dit':{('double_block',0):block}}}
|
||||
assert compat.selection(options)==(None,patch)
|
||||
assert compat.without_owned_blocks(options,patch)['patches_replace']['dit']=={}
|
||||
assert options['patches_replace']['dit']
|
||||
@@ -11,8 +11,10 @@
|
||||
import { app } from "../../scripts/app.js";
|
||||
import { api } from "../../scripts/api.js";
|
||||
import { showGuide } from "./bfs_md_view.js";
|
||||
import { wheelToCanvas } from "./bfs_wheel.js";
|
||||
import { createApp, ref, reactive, computed, watch, nextTick, onMounted, onBeforeUnmount } from "./vendor/vue.esm-browser.prod.mjs";
|
||||
import { h, rawH, t, uiLanguage, setUiLanguage, DEFAULTS, MASK_DEFAULTS, VLM_DEFAULTS, snapUp, snapDown, segKey, hasMask, pill } from "./shotloop/common.js";
|
||||
import { h, rawH, t, uiLanguage, setUiLanguage, DEFAULTS, MASK_DEFAULTS, VLM_DEFAULTS, snapUp, snapDown, segKey, hasMask, pill,
|
||||
MAX_REFS, refList, shotRefNames, MARK_PALETTE, pictureColor, markRegions, AUTO_DEFAULTS } from "./shotloop/common.js";
|
||||
import { styles } from "./shotloop/styles.js";
|
||||
// history lengths H3's VAE encodes whole (17k+5); old plans saved 17/34/51, read as the next valid one up
|
||||
const historyLen = n => (n = Number(n) || 22) <= 5 ? 5 : 5 + Math.ceil((n - 5) / 17) * 17;
|
||||
@@ -79,6 +81,18 @@ function Panel(io) {
|
||||
target: false, chain: false, color_guide: false, scope: "all" }));
|
||||
watch(copyOpts, v => store.set("bfs.shotloop.copy", { ...v }), { deep: true });
|
||||
|
||||
// one description per picture: picture_details, plus the ones plans before 1.113 saved per reference set in
|
||||
// ref_details ("a|b" -> {details, details2}, or one text for a); same rule as bfs_shot_loop.saved_picture_details
|
||||
const pictureDetailsOf = p => {
|
||||
const out = {};
|
||||
Object.entries(p.ref_details || {}).forEach(([key, v]) => {
|
||||
const [a, b] = key.split("|");
|
||||
const pairs = v && typeof v === "object" ? [[a, v.details || v.legacy], [b, v.details2]] : [[a, v]];
|
||||
pairs.forEach(([name, text]) => { if (name && text && !(name in out)) out[name] = String(text); });
|
||||
});
|
||||
Object.entries(p.picture_details || {}).forEach(([name, text]) => { if (text) out[name] = text; });
|
||||
return out;
|
||||
};
|
||||
const load = () => {
|
||||
let p = {};
|
||||
try { p = JSON.parse(io.getPlan() || "{}"); } catch { p = {}; }
|
||||
@@ -88,14 +102,34 @@ function Panel(io) {
|
||||
plan.mask_cfg = { ...MASK_DEFAULTS, ...(p.mask_cfg || {}) };
|
||||
plan.vlm_cfg = { ...VLM_DEFAULTS, ...(p.vlm_cfg || {}) };
|
||||
plan.ref_details = { ...(p.ref_details || {}) };
|
||||
plan.picture_details = { ...pictureDetailsOf(p) };
|
||||
plan.picture_cfg = { ...(p.picture_cfg || {}) };
|
||||
plan.auto_prompt = { ...AUTO_DEFAULTS, ...(p.auto_prompt || {}) };
|
||||
plan.auto_prompts = { ...(p.auto_prompts || {}) };
|
||||
if (!Array.isArray(plan.bounds)) plan.bounds = [];
|
||||
if (!Array.isArray(plan.segs)) plan.segs = [];
|
||||
// plans before 1.113 had two references (ref / ref2): now a list of up to MAX_REFS pictures
|
||||
plan.global_refs = refList(p.global_refs, p.global_ref, p.global_ref2);
|
||||
plan.segs.forEach(m => { if (m) m.refs = refList(m.refs, m.ref, m.ref2); });
|
||||
// Colored Mark: a color per picture (was: two marker colors), regions tied to a picture (was: region 2)
|
||||
if (!Array.isArray(p.picture_colors)) plan.picture_colors = [p.global_color_guide_ref1 || MARK_PALETTE[0][0], p.global_color_guide_ref2 || MARK_PALETTE[1][0]];
|
||||
plan.segs.forEach(m => {
|
||||
if (!m) return;
|
||||
m.color_guide_regions = markRegions(m);
|
||||
["color_guide_text2", "color_guide_points2", "color_guide_key2", "color_guide_color", "color_guide_color2"].forEach(k => delete m[k]);
|
||||
});
|
||||
Object.values(plan.cast).forEach(person => { if (person && typeof person === "object") { delete person.ref; delete person.ref2; delete person.guide_color; } });
|
||||
plan.segs.forEach(meta => { if (meta) { delete meta.color_guide_person; delete meta.color_guide_person2; } });
|
||||
};
|
||||
// any change of the plan (a new video, a prompt, a setting...) is a new loop run on the server: show its progress
|
||||
let progTimer = 0;
|
||||
const save = () => { io.setPlan(JSON.stringify(plan)); clearTimeout(progTimer); progTimer = setTimeout(() => progress(), 400); };
|
||||
// ref / ref2 (and global_ref / global_ref2) keep pictures 1 and 2 for older readers of the plan
|
||||
const syncRefs = () => {
|
||||
plan.global_refs = refList(plan.global_refs); [plan.global_ref = "", plan.global_ref2 = ""] = plan.global_refs;
|
||||
plan.global_color_guide_ref1 = pictureColor(plan, 1); plan.global_color_guide_ref2 = pictureColor(plan, 2);
|
||||
plan.segs.forEach(m => { if (m) { m.refs = refList(m.refs); [m.ref = "", m.ref2 = ""] = m.refs; } });
|
||||
};
|
||||
const save = () => { syncRefs(); io.setPlan(JSON.stringify(plan)); clearTimeout(progTimer); progTimer = setTimeout(() => progress(), 400); };
|
||||
|
||||
const n = computed(() => (an.value ? an.value.n : 0)); // the whole video on the timeline
|
||||
// the part the plan works on [lo, hi): start / end (0 = the end), then "max total seconds" counted from the start
|
||||
@@ -120,15 +154,14 @@ function Panel(io) {
|
||||
if (b[i + 1] <= b[i]) continue;
|
||||
const m = plan.segs[out.length] || {};
|
||||
out.push({ start: b[i], end: b[i + 1], len: b[i + 1] - b[i], gen: snapUp(b[i + 1] - b[i], plan.grid),
|
||||
cut: cs.has(b[i]), enabled: m.enabled !== false, ref: m.ref || "", ref2: m.ref2 || "", prompt: m.prompt || "",
|
||||
cut: cs.has(b[i]), enabled: m.enabled !== false, refs: refList(m.refs, m.ref, m.ref2), prompt: m.prompt || "",
|
||||
force: m.force || "auto", chain: m.chain || "head", chainFrame: m.chain_frame || "first", chainHistory: historyLen(m.chain_history),
|
||||
color_guide: !!m.color_guide, color_guide_color: m.color_guide_color || "",
|
||||
color_guide_opacity: m.color_guide_opacity ?? 0.35,
|
||||
color_guide_text2: m.color_guide_text2 || "", color_guide_color2: m.color_guide_color2 || "",
|
||||
color_guide_points2: m.color_guide_points2 || [], color_guide_key2: m.color_guide_key2,
|
||||
color_guide: !!m.color_guide, color_guide_picture: +m.color_guide_picture || 1,
|
||||
color_guide_opacity: m.color_guide_opacity ?? 0.35, regions: markRegions(m),
|
||||
crop: !!m.crop, inpaint: !!m.inpaint, paste: !!m.paste, pasteText: m.paste_text || "", strength: m.strength ?? 1, mask: m.mask || {}, target: m.target || "",
|
||||
extMask: !!plan.mask_video || !!(plan.track?.points?.length), trackMask: !!(plan.track?.points?.length) });
|
||||
}
|
||||
out.forEach(s => { s.names = shotRefNames(s.refs, plan.global_refs); [s.ref = "", s.ref2 = ""] = s.refs; });
|
||||
if (plan.max_parts > 0) out = out.slice(0, plan.max_parts);
|
||||
return out;
|
||||
});
|
||||
@@ -172,12 +205,16 @@ const modeOf = s => (MODES.find(m => m[2] === !!s.crop && m[3] === !!s.inpaint &
|
||||
const modeName = s => MODES.find(m => m[0] === modeOf(s))[1];
|
||||
|
||||
const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return plan.segs[i]; };
|
||||
// the reference a shot uses: its own or the global one
|
||||
const refOf = (s, k) => s[k] || plan["global_" + k] || "";
|
||||
// the pictures a shot uses: "ref" = <Picture 1>, "ref2" = <Picture 2> (its own or the global ones)
|
||||
const refOf = (s, k) => s.names[k === "ref2" ? 1 : 0] || "";
|
||||
const promptOf = s => s.prompt || plan.global_prompt || "";
|
||||
const detailsOf = s => {
|
||||
const value = plan.ref_details?.[`${refOf(s, "ref")}|${refOf(s, "ref2")}`] || "";
|
||||
return typeof value === "object" ? Object.values(value).some(Boolean) : !!value;
|
||||
// the pictures a prompt describes ({details} = 1, {details2} = 2, {details@Picture n}) all have a description
|
||||
const detailsOf = (s, prompt) => {
|
||||
const want = new Set();
|
||||
if (prompt.includes("{details}")) want.add(1);
|
||||
if (prompt.includes("{details2}")) want.add(2);
|
||||
for (const m of prompt.matchAll(/\{details@Picture\s*(\d+)\}/g)) want.add(+m[1]);
|
||||
return [...want].every(k => !s.names[k - 1] || plan.picture_details?.[s.names[k - 1]]);
|
||||
};
|
||||
|
||||
// things worth fixing before a run, per shot (only shots that run)
|
||||
@@ -221,11 +258,11 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
if ((s.crop || s.inpaint || s.paste) && !hasMask(s)) add("warn", `${modeName(s)} needs a mask (text, points or a mask video; or the planner's mask input)`);
|
||||
if (hasMask(s) && maskPrev.value[segKey(s)]?.empty) add("warn", "the mask preview found nothing");
|
||||
if (/\{target\}/.test(p) && !s.target) add("warn", "the prompt uses {target} but the shot has no target description");
|
||||
if (/\{details2?\}/.test(p) && !detailsOf(s) && !plan.vlm_cfg?.enabled) add("info", "the prompt uses {details} but its references have no description yet");
|
||||
if (/\{details/.test(p) && !detailsOf(s, p) && !plan.vlm_cfg?.enabled) add("info", "the prompt uses {details} but its pictures have no description yet");
|
||||
if (/\{details2?\}/.test(p) && (plan.vlm_cfg?.describe_preset || "short") !== "short")
|
||||
add("warn", "{details} sits inside a sentence of the prompt: use the 'short' describe preset (Prompts & refs), the other presets write a paragraph");
|
||||
if (!p) add("info", "no prompt (fine when the node's prompt input is connected)");
|
||||
if (!refOf(s, "ref")) add("info", "no reference (fine when the node's ref_image input is connected)");
|
||||
if (!refOf(s, "ref")) add("info", "no reference picture");
|
||||
if (vram.max_tokens && shotTokens(s, i) > 1.3 * vram.max_tokens)
|
||||
add("warn", "much bigger than any shot that ran on this GPU (over 1.3×): it may run out of memory. Lower Megapixels or Max seconds / shot, use head 5, or set Between shots = free memory (Run tab)");
|
||||
if (i > 0 && plan.run === "auto" && (s.chain === "head" || s.chain === "history") && !s.cut)
|
||||
@@ -257,7 +294,9 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
if (!plan.video) return;
|
||||
busy.value = "Detecting cuts…"; error.value = "";
|
||||
try {
|
||||
const body = { plan: { ...plan, bounds: apply ? [] : plan.bounds } };
|
||||
// every detected boundary is kept (Max shots only picks how many shots run): asked for without the limit, the
|
||||
// last kept shot would otherwise have no end and run to the end of the range
|
||||
const body = { plan: { ...plan, bounds: apply ? [] : plan.bounds, max_parts: apply ? 0 : plan.max_parts } };
|
||||
const r = await api.fetchApi("/bfs/shotloop/plan", { method: "POST", body: JSON.stringify(body) });
|
||||
const j = await r.json();
|
||||
if (j.error) throw new Error(j.error);
|
||||
@@ -266,12 +305,11 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
.map(s => [`${s.start}-${s.end}`, { people: s.people, main: s.main }]));
|
||||
if (apply) {
|
||||
plan.bounds = j.segs.slice(1).map(s => s.start);
|
||||
const keep = plan.segs; plan.segs = j.segs.map((_, i) => ({ enabled: true, ref: keep[i]?.ref || "", ref2: keep[i]?.ref2 || "", prompt: keep[i]?.prompt || "",
|
||||
const keep = plan.segs; plan.segs = j.segs.map((_, i) => ({ enabled: true, refs: refList(keep[i]?.refs), prompt: keep[i]?.prompt || "",
|
||||
chain: keep[i]?.chain || "head", chain_frame: keep[i]?.chain_frame || "first", chain_history: historyLen(keep[i]?.chain_history),
|
||||
crop: !!keep[i]?.crop, inpaint: !!keep[i]?.inpaint, paste: !!keep[i]?.paste, paste_text: keep[i]?.paste_text || "", strength: keep[i]?.strength ?? 1, mask: keep[i]?.mask || {}, target: keep[i]?.target || "",
|
||||
color_guide: !!keep[i]?.color_guide, color_guide_color: keep[i]?.color_guide_color || "", color_guide_opacity: keep[i]?.color_guide_opacity ?? 0.35,
|
||||
color_guide_text2: keep[i]?.color_guide_text2 || "", color_guide_color2: keep[i]?.color_guide_color2 || "",
|
||||
color_guide_points2: keep[i]?.color_guide_points2 || [], color_guide_key2: keep[i]?.color_guide_key2 ?? 0 }));
|
||||
color_guide: !!keep[i]?.color_guide, color_guide_picture: keep[i]?.color_guide_picture || 1, color_guide_opacity: keep[i]?.color_guide_opacity ?? 0.35,
|
||||
color_guide_regions: (keep[i]?.color_guide_regions || []).map(r => ({ ...r })) }));
|
||||
sel.value = 0; save();
|
||||
}
|
||||
} catch (e) { error.value = String(e.message || e); }
|
||||
@@ -333,30 +371,54 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
if (s && maskPrev.value[segKey(s)]) { const mp = { ...maskPrev.value }; delete mp[segKey(s)]; maskPrev.value = mp; }
|
||||
};
|
||||
const setMaskCfg = (k, v) => { plan.mask_cfg = { ...plan.mask_cfg, [k]: v }; save(); maskPrev.value = {}; };
|
||||
// Colored Mark regions in the points window: 1 = the shot's own selection (its mask), 2.. = its extra regions
|
||||
const openPoints = (i, region = 1) => {
|
||||
const s = segs.value[i]; if (!s) return;
|
||||
const g1 = { key: s.mask.key ?? Math.floor(s.len / 2), points: [...(s.mask.points || [])], text: s.mask.text || "" };
|
||||
const g2 = { key: g1.key, points: [...(s.color_guide_points2 || [])], text: s.color_guide_text2 || "" };
|
||||
Object.assign(modal, { open: true, idx: i, region: s.color_guide ? region : 1, groups: { 1: g1, 2: g2 },
|
||||
key: (region === 2 && s.color_guide ? g2 : g1).key, points: [...((region === 2 && s.color_guide ? g2 : g1).points)],
|
||||
text: (region === 2 && s.color_guide ? g2 : g1).text, prev: null, busy: "" });
|
||||
const groups = { 1: g1 };
|
||||
if (s.color_guide) s.regions.forEach((r, j) => { groups[j + 2] = { key: r.key ?? g1.key, points: [...(r.points || [])], text: r.text || "" }; });
|
||||
const g = groups[region] || g1;
|
||||
Object.assign(modal, { open: true, idx: i, region: groups[region] ? region : 1, groups, key: g.key, points: [...g.points],
|
||||
text: g.text, prev: null, busy: "" });
|
||||
};
|
||||
const switchPointRegion = region => {
|
||||
const old = modal.groups[modal.region] || {};
|
||||
modal.groups[modal.region] = { key: modal.key, points: [...modal.points], text: modal.text };
|
||||
const next = modal.groups[region] || { key: modal.key, points: [], text: "" };
|
||||
modal.region = region; modal.points = [...next.points]; modal.text = next.text; modal.prev = null;
|
||||
};
|
||||
const commitPointGroups = i => {
|
||||
const m = meta(i), g1 = modal.groups[1] || {}, g2 = modal.groups[2] || {};
|
||||
m.mask = { ...(m.mask || {}), points: [...(modal.region === 1 ? modal.points : g1.points || [])],
|
||||
key: modal.region === 1 ? modal.key : g1.key, text: modal.region === 1 ? modal.text : g1.text };
|
||||
modal.groups[modal.region] = { key: modal.key, points: [...modal.points], text: modal.text };
|
||||
const m = meta(i), g1 = modal.groups[1];
|
||||
m.mask = { ...(m.mask || {}), points: [...g1.points], key: g1.key, text: g1.text };
|
||||
if (segs.value[i]?.color_guide) {
|
||||
m.color_guide_points2 = [...(modal.region === 2 ? modal.points : g2.points || [])];
|
||||
m.color_guide_key2 = modal.region === 2 ? modal.key : (g1.key ?? modal.key);
|
||||
m.color_guide_text2 = modal.region === 2 ? modal.text : g2.text || "";
|
||||
m.color_guide_regions = markRegions(m).map((r, j) => {
|
||||
const g = modal.groups[j + 2];
|
||||
return g ? { ...r, points: [...g.points], key: g.key, text: g.text } : r;
|
||||
});
|
||||
}
|
||||
};
|
||||
// the picture a region becomes (region 1 = the shot's own selection) and the extra regions
|
||||
const setMarkPicture = (i, region, k) => {
|
||||
const m = meta(i);
|
||||
if (region === 1) m.color_guide_picture = k;
|
||||
else m.color_guide_regions = markRegions(m).map((r, j) => (j === region - 2 ? { ...r, picture: k } : r));
|
||||
save();
|
||||
};
|
||||
const addMarkRegion = i => {
|
||||
const m = meta(i), used = new Set([+m.color_guide_picture || 1, ...markRegions(m).map(r => r.picture)]);
|
||||
const n = Math.max(2, segs.value[i]?.names.length || 0);
|
||||
let k = 1; while (used.has(k) && k < n) k++;
|
||||
m.color_guide_regions = [...markRegions(m), { picture: k, text: "", points: [], key: 0 }]; save();
|
||||
};
|
||||
const setMarkRegion = (i, j, field, v) => {
|
||||
const m = meta(i); m.color_guide_regions = markRegions(m).map((r, q) => (q === j ? { ...r, [field]: v } : r)); save();
|
||||
};
|
||||
const removeMarkRegion = i => j => { const m = meta(i); m.color_guide_regions = markRegions(m).filter((_, q) => q !== j); save(); };
|
||||
const setPictureColor = (k, color) => {
|
||||
const list = [...(plan.picture_colors || [])];
|
||||
while (list.length < k) list.push(pictureColor(plan, list.length + 1));
|
||||
list[k - 1] = color; plan.picture_colors = list; save();
|
||||
};
|
||||
const savePoints = () => {
|
||||
const i = modal.idx; commitPointGroups(i); save();
|
||||
if (modal.prev) maskPrev.value = { ...maskPrev.value, [segKey(segs.value[modal.idx])]: modal.prev };
|
||||
@@ -395,7 +457,7 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
};
|
||||
const clearTrack = () => { plan.track = null; maskPrev.value = {}; save(); };
|
||||
const clearMask = i => {
|
||||
const m = meta(i); m.mask = {}; m.color_guide_points2 = []; m.color_guide_key2 = 0; m.color_guide_text2 = "";
|
||||
const m = meta(i); m.mask = {}; m.color_guide_regions = [];
|
||||
m.crop = false; m.inpaint = false; m.paste = false;
|
||||
const k = segs.value[i] ? segKey(segs.value[i]) : null;
|
||||
if (k) { const mp = { ...maskPrev.value }; delete mp[k]; maskPrev.value = mp; }
|
||||
@@ -409,17 +471,16 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
const to = targets(scope, from);
|
||||
to.forEach(i => {
|
||||
const m = meta(i), x = segs.value[i];
|
||||
if (fields.ref) { m.ref = src.ref; m.ref2 = src.ref2; }
|
||||
if (fields.ref) m.refs = [...src.refs];
|
||||
if (fields.prompt) m.prompt = src.prompt;
|
||||
if (fields.mask_text) m.mask = { ...(m.mask || {}), text: src.mask.text || "" };
|
||||
if (fields.mask_video) m.mask = { ...(m.mask || {}), video: src.mask.video || "" };
|
||||
if (fields.crop && (hasMask(x) || src.mask.text || fields.mask_text)) { m.crop = src.crop; m.inpaint = src.inpaint; m.paste = src.paste; m.paste_text = src.pasteText; m.strength = src.strength; }
|
||||
if (fields.target) m.target = src.target;
|
||||
if (fields.color_guide) {
|
||||
m.color_guide = src.color_guide; m.color_guide_color = src.color_guide_color;
|
||||
m.color_guide = src.color_guide; m.color_guide_picture = src.color_guide_picture;
|
||||
m.color_guide_opacity = src.color_guide_opacity;
|
||||
m.color_guide_text2 = src.color_guide_text2; m.color_guide_color2 = src.color_guide_color2;
|
||||
m.color_guide_points2 = [...(src.color_guide_points2 || [])]; m.color_guide_key2 = src.color_guide_key2;
|
||||
m.color_guide_regions = src.regions.map(r => ({ ...r, points: [...(r.points || [])] }));
|
||||
}
|
||||
if (fields.chain && i > 0) { m.chain = src.chain; m.chain_frame = src.chainFrame; m.chain_history = src.chainHistory; }
|
||||
});
|
||||
@@ -439,24 +500,54 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
busy.value = "";
|
||||
}
|
||||
// reference sets used by the plan (global and per shot): one description each
|
||||
const refSets = computed(() => {
|
||||
// every picture the plan uses, once, with where it is <Picture n>: one description each ({details@Picture n})
|
||||
const pictures = computed(() => {
|
||||
const out = new Map();
|
||||
const add = (a, b, where) => { if (!a && !b) return; const k = `${a || ""}|${b || ""}`; if (!out.has(k)) out.set(k, { a, b, where: [] }); out.get(k).where.push(where); };
|
||||
add(plan.global_ref, plan.global_ref2, "global");
|
||||
segs.value.forEach((x, i) => add(x.ref || plan.global_ref, x.ref2 || plan.global_ref2, `#${i + 1}`));
|
||||
return [...out.entries()].map(([key, v]) => ({ key, ...v }));
|
||||
const add = (name, where) => { if (!name) return; if (!out.has(name)) out.set(name, []); out.get(name).push(where); };
|
||||
plan.global_refs.forEach((x, k) => add(x, `global ${k + 1}`));
|
||||
segs.value.forEach((x, i) => x.names.forEach((name, k) => { if (x.refs[k] === name) add(name, `#${i + 1}: ${k + 1}`); }));
|
||||
return [...out.entries()].map(([name, where]) => ({ name, where }));
|
||||
});
|
||||
async function describeRefs() {
|
||||
busy.value = "Describing the references with the VLM…"; error.value = "";
|
||||
// names: the pictures to describe (all of them by default)
|
||||
async function describePictures(names = pictures.value.map(x => x.name)) {
|
||||
busy.value = "Describing the pictures with the VLM…"; error.value = "";
|
||||
try {
|
||||
const sets = refSets.value.map(x => [x.a || "", x.b || ""]);
|
||||
const r = await api.fetchApi("/bfs/shotloop/describe", { method: "POST", body: JSON.stringify({ plan: { ...plan }, sets }) });
|
||||
const r = await api.fetchApi("/bfs/shotloop/describe", { method: "POST",
|
||||
body: JSON.stringify({ plan: { ...plan }, pictures: names }) });
|
||||
const j = await r.json(); if (j.error) throw new Error(j.error);
|
||||
plan.ref_details = { ...plan.ref_details, ...j.texts }; save();
|
||||
plan.picture_details = { ...plan.picture_details, ...j.pictures }; save();
|
||||
} catch (e) { error.value = String(e.message || e); }
|
||||
busy.value = "";
|
||||
}
|
||||
const setDetail = (k, v) => { plan.ref_details = { ...plan.ref_details, [k]: v }; save(); };
|
||||
const setPictureDetail = (name, v) => { plan.picture_details = { ...plan.picture_details, [name]: v }; save(); };
|
||||
// a picture's own describe preset ("" = the card's) and custom instruction
|
||||
// 🤖 Auto prompt: what a shot's written prompt was made from (same string as bfs_shot_loop.auto_prompt_for), so the
|
||||
// panel shows when the request, pictures, marks or mode changed after it was written
|
||||
const autoFor = s => {
|
||||
const marks = s.color_guide ? [[s.color_guide_picture, pictureColor(plan, s.color_guide_picture)],
|
||||
...s.regions.filter(r => String(r.text || "").trim() || (r.points || []).length).map(r => [r.picture, pictureColor(plan, r.picture)])] : [];
|
||||
return JSON.stringify([s.prompt || plan.global_prompt || "", s.names, marks, modeOf(s), s.mask?.text || "", s.target || "",
|
||||
plan.auto_prompt?.template || "", plan.auto_prompt?.style || "h3", plan.auto_prompt?.trigger ?? "motion_transfer", "motion-trigger-bindings-v7"]);
|
||||
};
|
||||
const writtenOf = s => plan.auto_prompts?.[segKey(s)] || null;
|
||||
const setAuto = (k, v) => { plan.auto_prompt = { ...plan.auto_prompt, [k]: v }; save(); };
|
||||
const setWritten = (s, text) => {
|
||||
const k = segKey(s), cur = plan.auto_prompts[k] || {};
|
||||
plan.auto_prompts = { ...plan.auto_prompts, [k]: { ...cur, text, for: cur.for ?? autoFor(s) } }; save();
|
||||
};
|
||||
// the VLM writes the prompts of these shots (all by default)
|
||||
async function writePrompts(indices = segs.value.map((_, i) => i)) {
|
||||
busy.value = `Writing ${indices.length} prompt(s) with the VLM…`; error.value = "";
|
||||
try {
|
||||
const r = await api.fetchApi("/bfs/shotloop/autoprompt", { method: "POST", body: JSON.stringify({ plan: { ...plan }, indices }) });
|
||||
const j = await r.json(); if (j.error) throw new Error(j.error);
|
||||
plan.auto_prompts = { ...plan.auto_prompts, ...j.prompts }; save();
|
||||
} catch (e) { error.value = String(e.message || e); }
|
||||
busy.value = "";
|
||||
}
|
||||
const setPictureCfg = (name, k, v) => {
|
||||
plan.picture_cfg = { ...plan.picture_cfg, [name]: { ...(plan.picture_cfg[name] || {}), [k]: v } }; save();
|
||||
};
|
||||
const setVlmCfg = (k, v) => { plan.vlm_cfg = { ...plan.vlm_cfg, [k]: v }; save(); };
|
||||
const applySug = (i, what) => {
|
||||
const s = segs.value[i], g = s && vlmSug.value[segKey(s)]; if (!g) return;
|
||||
@@ -529,6 +620,16 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
const inp = document.createElement("input"); inp.type = "file"; inp.accept = accept;
|
||||
inp.onchange = () => inp.files[0] && upload(inp.files[0], cb); inp.click();
|
||||
};
|
||||
// several files: uploaded one after the other, cb gets the names in order
|
||||
const uploadAll = async (files, cb) => {
|
||||
const names = [];
|
||||
for (const f of files) await upload(f, name => names.push(name));
|
||||
if (names.length) cb(names);
|
||||
};
|
||||
const pickFiles = (accept, cb) => {
|
||||
const inp = document.createElement("input"); inp.type = "file"; inp.accept = accept; inp.multiple = true;
|
||||
inp.onchange = () => uploadAll([...inp.files], cb); inp.click();
|
||||
};
|
||||
async function progress(reset = false) {
|
||||
if (!plan.video || plan.run === "auto") { prog.done = 0; prog.count = 0; Object.assign(cache, { ok: [], stored: [], mb: 0 }); return; }
|
||||
try {
|
||||
@@ -538,9 +639,9 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
} catch { /* ignore */ }
|
||||
}
|
||||
// ---- stored shots: redo some, or combine what is stored into an MP4 (low memory, on the server)
|
||||
async function redo(shots, cascade = true) {
|
||||
async function redo(shots, cascade = true, cancel = false) {
|
||||
try {
|
||||
const r = await api.fetchApi("/bfs/shotloop/redo", { method: "POST", body: JSON.stringify({ plan: { ...plan }, shots, cascade }) });
|
||||
const r = await api.fetchApi("/bfs/shotloop/redo", { method: "POST", body: JSON.stringify({ plan: { ...plan }, shots, cascade, cancel }) });
|
||||
const j = await r.json(); if (j.error) error.value = j.error; else { Object.assign(cache, j); prog.done = j.done; }
|
||||
} catch (e) { error.value = `${e}`; }
|
||||
}
|
||||
@@ -665,16 +766,24 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
const onStatus = e => { if (busy.value || modal.busy) Object.assign(status, e.detail || {}); };
|
||||
let clock = 0;
|
||||
const onVlm = e => { if (e.detail?.video === plan.video) mergeSug(e.detail.segs); };
|
||||
// prompts the planner's VLM wrote while running (shots that had none): kept in the plan, so they stay the same
|
||||
const onAutoPrompt = e => {
|
||||
const d = e.detail; if (d?.video !== plan.video) return;
|
||||
const tpl = typeof d.template === "string" && d.template !== (plan.auto_prompt.template || "");
|
||||
if (tpl) plan.auto_prompt = { ...plan.auto_prompt, template: d.template }; // the prompt input of the last run
|
||||
if (tpl || Object.keys(d.prompts || {}).length) { plan.auto_prompts = { ...plan.auto_prompts, ...(d.prompts || {}) }; save(); }
|
||||
};
|
||||
const onNext = e => { onProg(e); if (plan.run === "queue" && plan.auto_continue) setTimeout(() => app.queuePrompt(0, 1), 300); };
|
||||
onMounted(() => {
|
||||
load(); refreshFiles(); analyze(); progress(); refreshVram(); raf = requestAnimationFrame(loop);
|
||||
if (!plan.video) tab.value = "video";
|
||||
api.addEventListener("bfs-shotloop-progress", onProg); api.addEventListener("bfs-shotloop-next", onNext);
|
||||
api.addEventListener("bfs-shotloop-vlm", onVlm);
|
||||
api.addEventListener("bfs-shotloop-autoprompt", onAutoPrompt);
|
||||
api.addEventListener("bfs-shotloop-status", onStatus);
|
||||
clock = setInterval(() => { if (busy.value || modal.busy) now.value = Date.now(); }, 500);
|
||||
});
|
||||
onBeforeUnmount(() => { cancelAnimationFrame(raf); api.removeEventListener("bfs-shotloop-progress", onProg); api.removeEventListener("bfs-shotloop-next", onNext); api.removeEventListener("bfs-shotloop-vlm", onVlm); api.removeEventListener("bfs-shotloop-status", onStatus); clearInterval(clock); });
|
||||
onBeforeUnmount(() => { cancelAnimationFrame(raf); api.removeEventListener("bfs-shotloop-progress", onProg); api.removeEventListener("bfs-shotloop-next", onNext); api.removeEventListener("bfs-shotloop-vlm", onVlm); api.removeEventListener("bfs-shotloop-autoprompt", onAutoPrompt); api.removeEventListener("bfs-shotloop-status", onStatus); clearInterval(clock); });
|
||||
io.expose({ reload: () => { load(); analyze(); progress(); } });
|
||||
|
||||
// 🧠 the memory status at the top: the plan's biggest shot against the biggest one that ran on this GPU
|
||||
@@ -712,24 +821,46 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
recent.value = [name, ...recent.value.filter(x => x !== name)].slice(0, 10);
|
||||
store.set(RECENT_KEY, recent.value);
|
||||
};
|
||||
// target: shot index or "global"; field: ref | ref2 (global_ref | global_ref2 for "global")
|
||||
const setRef = (target, field, name) => {
|
||||
// target: shot index or "global"; k: the picture's slot (0 = <Picture 1>). "" removes it: a global picture is taken
|
||||
// out of the list (the next ones move up), a shot's own picture falls back to the global one at that slot
|
||||
const refsOf = target => (target === "global" ? plan.global_refs : (meta(target).refs ||= []));
|
||||
const setRef = (target, k, name) => {
|
||||
if (isColorValue(name)) return; // colors belong to Colored Mark settings, never to image references
|
||||
if (target === "global") plan[field] = name; else meta(target)[field] = name;
|
||||
const list = refsOf(target);
|
||||
if (name) { while (list.length < k) list.push(""); list[k] = name; }
|
||||
else if (target === "global" || k >= plan.global_refs.length) list.splice(k, 1);
|
||||
else list[k] = "";
|
||||
remember(name); save();
|
||||
};
|
||||
const uploadRef = (target, field) => pickFile("image/*", name => setRef(target, field, name));
|
||||
// pictures put in from slot k on (k = the list's end: added), never past 12
|
||||
const putRefs = (target, k, names) => {
|
||||
const list = refsOf(target);
|
||||
names.slice(0, Math.max(0, MAX_REFS - k)).forEach((name, j) => { while (list.length < k + j) list.push(""); list[k + j] = name; remember(name); });
|
||||
save();
|
||||
};
|
||||
const uploadRef = (target, k, many = false) => many
|
||||
? pickFiles("image/*", names => putRefs(target, k, names))
|
||||
: pickFile("image/*", name => setRef(target, k, name));
|
||||
// a drop on slot k replaces it (more files go on the next slots); k = null (the box): added at the end. A recent
|
||||
// thumbnail dragged in carries its name, image files are uploaded
|
||||
const dropRefs = (target, k, e) => {
|
||||
const at = k ?? refsOf(target).length;
|
||||
const name = e.dataTransfer?.getData("text/bfs-ref");
|
||||
if (name) { putRefs(target, at, [name]); return; }
|
||||
const files = [...(e.dataTransfer?.files || [])].filter(f => f.type.startsWith("image/"));
|
||||
if (files.length) uploadAll(files, names => putRefs(target, at, names));
|
||||
};
|
||||
const clearRecent = () => { recent.value = []; store.set(RECENT_KEY, []); };
|
||||
// everything: the recent list and every reference this plan uses (global and per shot)
|
||||
const clearAllRefs = () => {
|
||||
if (!confirm("Remove every reference picture from this plan (global, every shot) and forget the recent list?")) return;
|
||||
clearRecent();
|
||||
plan.global_ref = ""; plan.global_ref2 = "";
|
||||
plan.segs.forEach(m => { if (m) { m.ref = ""; m.ref2 = ""; } });
|
||||
plan.global_refs = [];
|
||||
plan.segs.forEach(m => { if (m) m.refs = []; });
|
||||
save();
|
||||
};
|
||||
const usedRefs = computed(() => {
|
||||
const set = new Set([...recent.value, plan.global_ref, plan.global_ref2, ...plan.segs.flatMap(m => [m?.ref, m?.ref2])].filter(Boolean));
|
||||
const set = new Set([...recent.value, ...plan.global_refs, ...plan.segs.flatMap(m => m?.refs || [])].filter(Boolean));
|
||||
return [...set].slice(0, 10);
|
||||
});
|
||||
|
||||
@@ -737,12 +868,14 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
const c = {
|
||||
api, plan, files, an, cuts, detectorUsed, size, busy, error, status, busySince, now, upPct, sel, zoom, hover, prog, tlEl,
|
||||
stats, people, maskPrev, showMasks, vlmSug, modal, segPeople, targetCrop, tab, copyOpts, COPY_FIELDS, vid, play, recent, clearRecent, clearAllRefs,
|
||||
n, range, setRange, maxLen, segs, allSegCount, active, pxPerFrame, tlWidth, filtersOn, refSets, usedRefs, checks, tour, memory, vram, refreshVram, cache, cacheOf, redo, combine,
|
||||
n, range, setRange, maxLen, segs, allSegCount, active, pxPerFrame, tlWidth, filtersOn, usedRefs, checks, tour, memory, vram, refreshVram, cache, cacheOf, redo, combine,
|
||||
MODES, modeOf, modeName, viewUrl, save, meta, setMeta, setMode, setPlan, whoIn, castOf, included, personOf, statFor, skipWhy, refOf, promptOf, checksFor,
|
||||
refreshFiles, analyze, autoSplit, analyzeContent, findPeople, setCast, describeTarget, previewMask, setMask, setMaskCfg,
|
||||
openPoints, switchPointRegion, savePoints, savePointsAll, saveTrack, clearTrack, pointsTo, clearMask, copyFrom, analyseVLM, describeRefs, setDetail, setVlmCfg,
|
||||
openPoints, switchPointRegion, savePoints, savePointsAll, saveTrack, clearTrack, pointsTo, clearMask, copyFrom, analyseVLM, setVlmCfg,
|
||||
applySug, applyAllSug, setCastOpt, setGlobalMaskVideo, setFilter, pickFile, progress, setVideo, splitAt, mergeNext, removeCut, select, drag,
|
||||
frameAt, thumbFor, playShot, playAll, stop, seek, setRef, uploadRef,
|
||||
frameAt, thumbFor, playShot, playAll, stop, seek, setRef, putRefs, uploadRef, dropRefs, pictures, describePictures, setPictureDetail, setPictureCfg,
|
||||
setMarkPicture, addMarkRegion, setMarkRegion, removeMarkRegion, setPictureColor,
|
||||
autoFor, writtenOf, setAuto, setWritten, writePrompts,
|
||||
};
|
||||
|
||||
return () => {
|
||||
@@ -786,8 +919,89 @@ const meta = i => { while (plan.segs.length <= i) plan.segs.push({}); return pla
|
||||
};
|
||||
}
|
||||
|
||||
// A workflow saved before 1.113 keeps the old slots of these nodes (ComfyUI restores a node's saved slots): the
|
||||
// planner's ref_image / ref_image_2 inputs are gone, the planner's and Unpack's ref_image / ref_image_2 outputs are now
|
||||
// refs / ref_count, and what fed Repack's ref_image now feeds its refs input.
|
||||
// Saved widget orders of the conditioning nodes (ComfyUI restores values by position): use_ref_2 (always second) left
|
||||
// in 1.114 and the Advanced node's duet widgets in 1.113, so older values are put back by name
|
||||
const OLD_ADVANCED_WIDGETS = ["guide_mode", "use_ref_2", "first_frame", "ref_image_size", "with_audio", "duet",
|
||||
"panel_position", "panel_size", "panel_noise", "rope_gap", "task", "instruction", "setting_ref", "inpaint", "mask_guide",
|
||||
"source_blur", "source_blur_strength", "source_blur_segmenter", "source_blur_effect", "mask_guide_look", "mask_ref_size"];
|
||||
const ADVANCED_1113 = ["guide_mode", "use_ref_2", "first_frame", "ref_image_size", "with_audio", "setting_ref", "inpaint",
|
||||
"mask_guide", "source_blur", "source_blur_strength", "source_blur_segmenter", "source_blur_effect", "mask_guide_look", "mask_ref_size"];
|
||||
const OLD_SIMPLE_WIDGETS = ["guide_mode", "use_ref_2", "ref_image_size", "with_audio", "inpaint", "source_blur",
|
||||
"source_blur_strength", "source_blur_segmenter", "source_blur_effect"];
|
||||
const NO_BLUR_PICKERS = n => n !== "source_blur_segmenter" && n !== "source_blur_effect"; // added later
|
||||
function oldWidgetNames(cls, v) {
|
||||
if (typeof v[1] !== "boolean") return null; // already without use_ref_2
|
||||
if (cls === "BFSShotH3ConditioningSimple") return v.length >= 9 ? OLD_SIMPLE_WIDGETS : OLD_SIMPLE_WIDGETS.filter(NO_BLUR_PICKERS);
|
||||
if (["off", "canvas", "shifted RoPE"].includes(v[5]) && ["left", "right", "top", "bottom"].includes(v[6]))
|
||||
return v.length >= 21 ? OLD_ADVANCED_WIDGETS : OLD_ADVANCED_WIDGETS.filter(NO_BLUR_PICKERS);
|
||||
return v.length >= 14 ? ADVANCED_1113 : ADVANCED_1113.filter(NO_BLUR_PICKERS);
|
||||
}
|
||||
function upgradeConditioningWidgets(node, info) {
|
||||
const v = info?.widgets_values;
|
||||
const names = Array.isArray(v) && oldWidgetNames(node.comfyClass, v);
|
||||
if (!names) return;
|
||||
names.forEach((name, k) => { const w = node.widgets?.find(x => x.name === name); if (w && k < v.length) w.value = v[k]; });
|
||||
// widgets the saved workflow did not have got a shifted value: back to their default (a list's first choice)
|
||||
node.widgets?.forEach(w => {
|
||||
const choices = w.options?.values;
|
||||
if (!names.includes(w.name) && Array.isArray(choices) && !choices.includes(w.value)) w.value = w.options.default ?? choices[0];
|
||||
});
|
||||
}
|
||||
|
||||
function upgradeLegacySlots(node, info) {
|
||||
const cls = node.comfyClass;
|
||||
if (cls === "BFSShotH3Conditioning" || cls === "BFSShotH3ConditioningSimple") {
|
||||
upgradeConditioningWidgets(node, info);
|
||||
// 1.115.9 appends MODEL routing to Simple; preserve all existing slot indices/links.
|
||||
if (cls === "BFSShotH3ConditioningSimple") {
|
||||
if (!node.inputs?.some(x => x.name === "model")) node.addInput("model", "MODEL");
|
||||
if (!node.outputs?.some(x => x.name === "model")) node.addOutput("model", "MODEL");
|
||||
}
|
||||
return;
|
||||
}
|
||||
// ref_image / ref_image_2 outputs at slots `at`, at+1 -> refs (list) / ref_count
|
||||
const listOutputs = at => {
|
||||
const a = node.outputs?.[at], b = node.outputs?.[at + 1];
|
||||
if (a?.name === "ref_image") Object.assign(a, { name: "refs", label: undefined, localized_name: "refs", shape: 6 });
|
||||
if (b?.name === "ref_image_2") { node.disconnectOutput(at + 1); Object.assign(b, { name: "ref_count", label: undefined, localized_name: "ref_count", type: "INT", shape: undefined }); }
|
||||
};
|
||||
if (cls === "BFSShotPlanner") {
|
||||
for (const name of ["ref_image", "ref_image_2"]) {
|
||||
const k = node.inputs?.findIndex(x => x.name === name) ?? -1;
|
||||
if (k >= 0) node.removeInput(k);
|
||||
}
|
||||
listOutputs(8);
|
||||
} else if (cls === "BFSShotUnpack") {
|
||||
listOutputs(1);
|
||||
} else if (cls === "BFSShotRepack") {
|
||||
// what fed ref_image now feeds refs; ref_image / ref_image_2 go away
|
||||
const at = name => node.inputs?.findIndex(x => x.name === name) ?? -1;
|
||||
for (const name of ["ref_image", "ref_image_2"]) {
|
||||
const k = at(name); if (k < 0) continue;
|
||||
const lid = node.inputs[k].link, l = lid == null ? null : (node.graph?.links?.get?.(lid) ?? node.graph?.links?.[lid]);
|
||||
const origin = l ? node.graph.getNodeById(l.origin_id) : null;
|
||||
node.removeInput(k);
|
||||
const to = at("refs");
|
||||
if (name === "ref_image" && origin && to >= 0 && node.inputs[to].link == null) origin.connect(l.origin_slot, node, to);
|
||||
}
|
||||
} else return;
|
||||
node.setDirtyCanvas?.(true, true);
|
||||
}
|
||||
|
||||
app.registerExtension({
|
||||
name: "BFSNodes.ShotPlanner",
|
||||
async beforeRegisterNodeDef(nodeType, nodeData) {
|
||||
if (!["BFSShotPlanner", "BFSShotUnpack", "BFSShotRepack", "BFSShotH3Conditioning", "BFSShotH3ConditioningSimple"].includes(nodeData.name)) return;
|
||||
const prev = nodeType.prototype.onConfigure;
|
||||
nodeType.prototype.onConfigure = function (info) {
|
||||
const r = prev?.apply(this, arguments);
|
||||
upgradeLegacySlots(this, info);
|
||||
return r;
|
||||
};
|
||||
},
|
||||
async nodeCreated(node) {
|
||||
if (node.comfyClass !== "BFSShotPlanner") return;
|
||||
styles();
|
||||
@@ -801,6 +1015,7 @@ app.registerExtension({
|
||||
// node (Vue nodes mode sizes widgets by their content); min-width keeps it from collapsing to its content width
|
||||
host.style.cssText = "position:relative;width:100%;height:100%;min-width:900px;min-height:640px";
|
||||
node.addDOMWidget("bfs_shot_planner", "div", host, { serialize: false, hideOnZoom: false });
|
||||
wheelToCanvas(host); // the wheel zooms the canvas over the panel too, except on something that scrolls
|
||||
const handle = {};
|
||||
createApp({
|
||||
setup: () => Panel({
|
||||
|
||||
@@ -4,12 +4,21 @@
|
||||
* Wipe (drag the divider), overlay (drag to fade) or side by side, with any of guide / target / difference on top
|
||||
* or below. The strip under the scrubber is the difference per frame, measured by the node on the real frames:
|
||||
* click it to jump to a frame. The node sends three temporary MP4s and that list when it runs.
|
||||
*
|
||||
* ⬇ save plays the video once from the start and downloads it as shown (same layout, divider, labels); ● rec records
|
||||
* whatever you do in the player (dragging the divider, pausing, scrubbing) until ■ stop. Both draw the two videos into
|
||||
* a canvas and record it in the browser, in real time. The browser's file (WebM, or a fragmented MP4 many apps won't
|
||||
* play) is then converted by ComfyUI to the chosen format: MP4 (H.264), a small MP4 (under 10 MB) or a GIF.
|
||||
*/
|
||||
import { app } from "../../scripts/app.js";
|
||||
import { api } from "../../scripts/api.js";
|
||||
import { wheelToCanvas } from "./bfs_wheel.js";
|
||||
|
||||
const SOURCES = ["guide", "target", "difference"];
|
||||
const MODES = [["wipe", "wipe ↔"], ["wipe-v", "wipe ↕"], ["overlay", "overlay"], ["side", "side by side"]];
|
||||
const FORMATS = [["mp4", "MP4"], ["small", "MP4 small (<10 MB)"], ["gif", "GIF"], ["webm", "browser file (WebM)"]];
|
||||
const FORMAT_KEY = "bfs.videoCompare.format";
|
||||
const PART = 8 * 1024 * 1024; // upload pieces, under ComfyUI's request limit
|
||||
|
||||
function styles() {
|
||||
if (document.getElementById("bfs-vcmp-css")) return;
|
||||
@@ -19,12 +28,16 @@ function styles() {
|
||||
.bfsvc{display:flex;flex-direction:column;gap:6px;height:100%;box-sizing:border-box;padding:6px;
|
||||
font:12px/1.4 var(--font-family,system-ui,sans-serif);color:var(--descrip-text,#c9c9cf);
|
||||
background:var(--comfy-input-bg,#1b1b1f);border-radius:8px}
|
||||
.bfsvc .stage{position:relative;flex:1;min-height:160px;background:#0e0e11;border-radius:6px;overflow:hidden;
|
||||
.bfsvc .stage{position:relative;flex:1 1 0;min-height:160px;background:#0e0e11;border-radius:6px;overflow:hidden;
|
||||
cursor:ew-resize;touch-action:none;user-select:none}
|
||||
.bfsvc .stage.v{cursor:ns-resize}.bfsvc .stage.o{cursor:col-resize}.bfsvc .stage.s{cursor:default;display:grid;
|
||||
grid-template-columns:1fr 1fr;gap:4px}
|
||||
.bfsvc .stage.v{cursor:ns-resize}.bfsvc .stage.o{cursor:col-resize}.bfsvc .stage.s{cursor:default}
|
||||
.bfsvc video{position:absolute;inset:0;width:100%;height:100%;object-fit:contain;pointer-events:none}
|
||||
.bfsvc .stage.s video{position:relative}
|
||||
.bfsvc .stage.s video{width:calc(50% - 2px)}.bfsvc .stage.s video.r{left:auto;right:0}
|
||||
.bfsvc .refs{flex:0 0 auto;height:84px;display:flex;gap:6px;justify-content:center;overflow-x:auto;padding:4px;
|
||||
background:#0e0e11;border-radius:6px}
|
||||
.bfsvc .refs figure{margin:0;height:100%;position:relative;flex:none}
|
||||
.bfsvc .refs img{height:100%;border-radius:4px;box-shadow:0 0 0 1px #000a;background:#000;display:block}
|
||||
.bfsvc .refs figcaption{position:absolute;left:3px;top:2px;font-size:10px;padding:0 4px;border-radius:3px;background:#000a;color:#fff}
|
||||
.bfsvc .line{position:absolute;background:#fff;box-shadow:0 0 0 1px #0008;pointer-events:none}
|
||||
.bfsvc .tag{position:absolute;top:6px;padding:1px 7px;border-radius:4px;background:#000a;color:#fff;font-size:11px;
|
||||
pointer-events:none}
|
||||
@@ -32,6 +45,9 @@ function styles() {
|
||||
.bfsvc select,.bfsvc button{background:#131317;border:1px solid #34343d;color:#e6e6ea;border-radius:4px;
|
||||
padding:2px 6px;font:inherit;cursor:pointer}
|
||||
.bfsvc button:hover,.bfsvc select:hover{border-color:#3a86ff}
|
||||
.bfsvc button.rec{border-color:#e0573a;color:#ff7a5c}
|
||||
.bfsvc button:disabled{opacity:.45;cursor:default}
|
||||
.bfsvc label{display:flex;gap:3px;align-items:center;cursor:pointer}
|
||||
.bfsvc input[type=range]{flex:1;accent-color:#3a86ff;min-width:80px}
|
||||
.bfsvc canvas{width:100%;height:26px;cursor:pointer;border-radius:3px;background:#131317;display:block}
|
||||
.bfsvc .info{font-family:ui-monospace,monospace;font-size:11px;white-space:nowrap}
|
||||
@@ -55,13 +71,15 @@ function el(tag, attrs = {}, kids = []) {
|
||||
}
|
||||
|
||||
function buildPlayer(host) {
|
||||
const st = { data: null, top: "target", bottom: "guide", mode: "wipe", pos: 0.5, playing: false, muted: true, frame: 0 };
|
||||
const st = { data: null, top: "target", bottom: "guide", mode: "wipe", pos: 0.5, playing: false, muted: true, frame: 0,
|
||||
labels: true, refs: true };
|
||||
const vTop = el("video", { playsinline: "", preload: "auto", loop: "" });
|
||||
const vBot = el("video", { playsinline: "", preload: "auto", loop: "" });
|
||||
vTop.muted = vBot.muted = true;
|
||||
const line = el("div", { class: "line" });
|
||||
const tagTop = el("div", { class: "tag" }), tagBot = el("div", { class: "tag" });
|
||||
const empty = el("div", { class: "empty" }, "Run the node to load the guide and the target.");
|
||||
const refBox = el("div", { class: "refs", style: "display:none" });
|
||||
const stage = el("div", { class: "stage", tabindex: "0" }, [vBot, vTop, line, tagTop, tagBot, empty]);
|
||||
const sel = (opts, val, on) => {
|
||||
const s = el("select", {}, opts.map(([v, l]) => { const o = el("option", { value: v }, l); if (v === val) o.selected = true; return o; }));
|
||||
@@ -83,6 +101,22 @@ function buildPlayer(host) {
|
||||
const btnNext = el("button", { title: "Next frame (→)", onclick: () => step(1) }, "|▶");
|
||||
const btnMute = el("button", { title: "Sound (from the target)", onclick: () => { st.muted = !st.muted; sound(); } }, "🔇");
|
||||
const btnFull = el("button", { title: "Full screen", onclick: () => (document.fullscreenElement ? document.exitFullscreen() : host.requestFullscreen?.())?.catch?.(() => {}) }, "⛶");
|
||||
const btnSave = el("button", { title: "Play the video once from the start and download it as shown", onclick: () => saveFull() }, "⬇ save");
|
||||
let format = "mp4";
|
||||
try { format = localStorage.getItem(FORMAT_KEY) || format; } catch { }
|
||||
if (!FORMATS.some(([v]) => v === format)) format = "mp4";
|
||||
const selFormat = sel(FORMATS, format, v => { format = v; try { localStorage.setItem(FORMAT_KEY, v); } catch { } });
|
||||
selFormat.title = "Format of what save / rec download. MP4: H.264 + AAC, plays and previews anywhere. MP4 small: "
|
||||
+ "the same, sized to stay under 10 MB. GIF: no sound, under 10 MB. browser file: the recording as it is, no conversion.";
|
||||
const btnRec = el("button", { title: "Record what you do in the player until stop, then download it", onclick: () => (rec.mr ? stopRec() : startRec()) }, "● rec");
|
||||
const chkLabels = el("input", { type: "checkbox" });
|
||||
chkLabels.checked = true;
|
||||
chkLabels.addEventListener("change", () => { st.labels = chkLabels.checked; });
|
||||
const chkRefs = el("input", { type: "checkbox" });
|
||||
chkRefs.checked = true;
|
||||
const showRefs = () => { refBox.style.display = st.refs && refBox.childElementCount ? "flex" : "none"; };
|
||||
chkRefs.addEventListener("change", () => { st.refs = chkRefs.checked; showRefs(); });
|
||||
const lblRefs = el("label", { title: "Show the reference pictures in a strip below the videos (also in what save / rec download)", style: "display:none" }, [chkRefs, "refs"]);
|
||||
const scrub = el("input", { type: "range", min: "0", max: "0", value: "0", step: "1" });
|
||||
scrub.addEventListener("input", () => seek(+scrub.value));
|
||||
const info = el("span", { class: "info" }, "");
|
||||
@@ -91,11 +125,15 @@ function buildPlayer(host) {
|
||||
const r = strip.getBoundingClientRect(), n = st.data?.frames || 0;
|
||||
if (n) seek(Math.min(n - 1, Math.max(0, Math.floor((e.clientX - r.left) / r.width * n))));
|
||||
});
|
||||
const notes = el("div", {style:"white-space:pre-wrap;font-size:12px;padding:6px;color:#e6b566"});
|
||||
host.append(el("div", { class: "bfsvc" }, [
|
||||
el("div", { class: "row" }, [el("span", {}, "top"), selTop, btnSwap, el("span", {}, "bottom"), selBot, selMode, btnFull]),
|
||||
stage,
|
||||
refBox,
|
||||
el("div", { class: "row" }, [btnPrev, btnPlay, btnNext, scrub, btnMute, info]),
|
||||
el("div", { class: "row" }, [btnSave, btnRec, selFormat, el("label", { title: "Draw the names on the saved / recorded video" }, [chkLabels, "labels"]), lblRefs]),
|
||||
strip,
|
||||
notes,
|
||||
]));
|
||||
|
||||
const fps = () => st.data?.fps || 24;
|
||||
@@ -118,6 +156,7 @@ function buildPlayer(host) {
|
||||
if (m === "wipe") line.style.cssText = `display:block;left:calc(${p * 100}% - 1px);top:0;bottom:0;width:2px`;
|
||||
if (m === "wipe-v") line.style.cssText = `display:block;top:calc(${p * 100}% - 1px);left:0;right:0;height:2px`;
|
||||
stage.prepend(vTop); stage.prepend(vBot); // under the divider and tags; side by side: bottom left, top right
|
||||
vTop.classList.toggle("r", m === "side");
|
||||
const name = s => s === "guide" ? (st.data?.labels?.[0] || "guide") : s === "target" ? (st.data?.labels?.[1] || "target") : "difference";
|
||||
tagTop.textContent = name(st.top) + (m === "overlay" ? ` ${Math.round(p * 100)}%` : "");
|
||||
tagBot.textContent = name(st.bottom);
|
||||
@@ -175,12 +214,17 @@ function buildPlayer(host) {
|
||||
st.playing = !st.playing;
|
||||
btnPlay.textContent = st.playing ? "⏸" : "▶";
|
||||
if (st.playing) { vTop.currentTime = vBot.currentTime; for (const v of videos()) v.play().catch(() => {}); tick(); }
|
||||
else for (const v of videos()) v.pause();
|
||||
else for (const v of videos()) { v.pause(); v.playbackRate = 1; }
|
||||
}
|
||||
|
||||
function tick() { // the bottom video is the clock; the top one follows it within half a frame
|
||||
if (!st.playing) return;
|
||||
if (Math.abs(vTop.currentTime - vBot.currentTime) > 0.5 / fps()) vTop.currentTime = vBot.currentTime;
|
||||
// small drift: nudge the top one's speed (seeking it every frame kept it seeking, so it stuttered and drew blank)
|
||||
const drift = vTop.currentTime - vBot.currentTime;
|
||||
if (!vTop.seeking) {
|
||||
if (Math.abs(drift) > 0.25) { vTop.currentTime = vBot.currentTime; vTop.playbackRate = 1; }
|
||||
else vTop.playbackRate = Math.abs(drift) > 0.5 / fps() ? Math.min(1.05, Math.max(0.95, 1 - 2 * drift)) : 1;
|
||||
}
|
||||
show(frameOf(vBot.currentTime));
|
||||
requestAnimationFrame(tick);
|
||||
}
|
||||
@@ -200,9 +244,221 @@ function buildPlayer(host) {
|
||||
layout();
|
||||
}
|
||||
|
||||
// --- save / record: the two videos drawn into a canvas the way the stage shows them, encoded by MediaRecorder
|
||||
const rec = { mr: null, chunks: [], raf: 0, t0: 0, full: false, onEnd: null, type: "", busy: "" };
|
||||
|
||||
function pickType(audio) { // name an audio codec only with a sound track: Firefox stalls on one it never gets
|
||||
const list = audio ? ["video/mp4;codecs=avc1.640028,mp4a.40.2", "video/mp4", "video/webm;codecs=vp9,opus", "video/webm;codecs=vp8,opus", "video/webm"]
|
||||
: ["video/mp4;codecs=avc1.640028", "video/mp4;codecs=avc1", "video/mp4", "video/webm;codecs=vp9", "video/webm;codecs=vp8", "video/webm"];
|
||||
return list.find(t => window.MediaRecorder?.isTypeSupported?.(t)) || "";
|
||||
}
|
||||
|
||||
function fit(aw, ah, x, y, w, h) { // the largest aw:ah box centred in (x, y, w, h)
|
||||
const k = Math.min(w / aw, h / ah), bw = aw * k, bh = ah * k;
|
||||
return [x + (w - bw) / 2, y + (h - bh) / 2, bw, bh];
|
||||
}
|
||||
|
||||
function label(ctx, text, x, y, size, right = false, bottom = false) {
|
||||
ctx.font = `${size}px system-ui,sans-serif`;
|
||||
const pad = size * 0.45, w = ctx.measureText(text).width + 2 * pad, h = size * 1.5;
|
||||
const lx = right ? x - w : x, ly = bottom ? y - h : y;
|
||||
ctx.fillStyle = "rgba(0,0,0,.65)"; ctx.fillRect(lx, ly, w, h);
|
||||
ctx.fillStyle = "#fff"; ctx.textBaseline = "middle"; ctx.fillText(text, lx + pad, ly + h / 2);
|
||||
}
|
||||
|
||||
const last = new Map(); // video -> its last whole frame, drawn while it seeks (a seeking video draws blank)
|
||||
function pic(v) {
|
||||
if (v.readyState < 2 || v.seeking) return last.get(v) || v;
|
||||
let c = last.get(v);
|
||||
if (!c) last.set(v, c = el("canvas"));
|
||||
if (c.width !== v.videoWidth || c.height !== v.videoHeight) { c.width = v.videoWidth; c.height = v.videoHeight; }
|
||||
c.getContext("2d").drawImage(v, 0, 0);
|
||||
return c;
|
||||
}
|
||||
|
||||
const refImages = () => (st.refs ? [...refBox.querySelectorAll("img")].filter(i => i.complete && i.naturalWidth) : []);
|
||||
// save / rec: the pictures in a strip below the videos, as in the player (22% of the video's height)
|
||||
const stripHeight = h => (refImages().length ? Math.round(h * 0.22) & ~1 : 0);
|
||||
function drawRefs(ctx, top) {
|
||||
const ims = refImages();
|
||||
if (!ims.length) return;
|
||||
const cw = ctx.canvas.width, sh = ctx.canvas.height - top, gap = Math.max(2, Math.round(sh * 0.05));
|
||||
let h = sh - 2 * gap;
|
||||
const widths = () => ims.map(im => Math.round(im.naturalWidth * h / im.naturalHeight));
|
||||
let total = widths().reduce((a, b) => a + b, 0) + gap * (ims.length - 1);
|
||||
if (total > cw - 2 * gap) { h = Math.floor(h * (cw - 2 * gap) / total); total = widths().reduce((a, b) => a + b, 0) + gap * (ims.length - 1); }
|
||||
ctx.fillStyle = "#0e0e11"; ctx.fillRect(0, top, cw, sh);
|
||||
let x = Math.round((cw - total) / 2);
|
||||
const y = top + Math.round((sh - h) / 2), fs = Math.max(10, Math.round(h * 0.12));
|
||||
ims.forEach((im, k) => {
|
||||
const w = Math.round(im.naturalWidth * h / im.naturalHeight);
|
||||
ctx.drawImage(im, x, y, w, h);
|
||||
if (st.labels) label(ctx, `Picture ${k + 1}`, x + 3, y + 3, fs);
|
||||
x += w + gap;
|
||||
});
|
||||
}
|
||||
|
||||
function compose(ctx) {
|
||||
const sh = ctx.canvas.height - (ctx.canvas.videoHeight || ctx.canvas.height);
|
||||
drawVideos(ctx, ctx.canvas.height - sh);
|
||||
drawRefs(ctx, ctx.canvas.height - sh);
|
||||
}
|
||||
|
||||
function drawVideos(ctx, area) {
|
||||
const cw = ctx.canvas.width, ch = area ?? ctx.canvas.height, m = st.mode, p = st.pos;
|
||||
const B = pic(vBot), T = pic(vTop);
|
||||
const aw = vBot.videoWidth || st.data.width, ah = vBot.videoHeight || st.data.height;
|
||||
const name = s => s === "guide" ? (st.data?.labels?.[0] || "guide") : s === "target" ? (st.data?.labels?.[1] || "target") : "difference";
|
||||
ctx.fillStyle = "#000"; ctx.fillRect(0, 0, cw, ch);
|
||||
if (m === "side") { // bottom left, top right, as on the stage
|
||||
const gap = Math.round(cw * 0.004), half = (cw - gap) / 2;
|
||||
const a = fit(aw, ah, 0, 0, half, ch), b = fit(aw, ah, half + gap, 0, half, ch);
|
||||
ctx.drawImage(B, ...a); ctx.drawImage(T, ...b);
|
||||
if (st.labels) {
|
||||
const fs = Math.max(12, Math.round(a[3] * 0.032)), o = fs * 0.5;
|
||||
label(ctx, name(st.bottom), a[0] + o, a[1] + o, fs); label(ctx, name(st.top), b[0] + o, b[1] + o, fs);
|
||||
}
|
||||
return;
|
||||
}
|
||||
const [x, y, w, h] = fit(aw, ah, 0, 0, cw, ch);
|
||||
ctx.drawImage(B, x, y, w, h);
|
||||
ctx.save();
|
||||
if (m === "wipe" || m === "wipe-v") { ctx.beginPath(); m === "wipe" ? ctx.rect(x, y, w * p, h) : ctx.rect(x, y, w, h * p); ctx.clip(); }
|
||||
if (m === "overlay") ctx.globalAlpha = p;
|
||||
ctx.drawImage(T, x, y, w, h);
|
||||
ctx.restore();
|
||||
const lw = Math.max(2, Math.round(h / 360));
|
||||
ctx.fillStyle = "#fff";
|
||||
if (m === "wipe") ctx.fillRect(x + w * p - lw / 2, y, lw, h);
|
||||
if (m === "wipe-v") ctx.fillRect(x, y + h * p - lw / 2, w, lw);
|
||||
if (st.labels) {
|
||||
const fs = Math.max(12, Math.round(h * 0.032)), o = fs * 0.5;
|
||||
label(ctx, name(st.top) + (m === "overlay" ? ` ${Math.round(p * 100)}%` : ""), x + o, y + o, fs);
|
||||
if (m === "wipe-v") label(ctx, name(st.bottom), x + o, y + h - o, fs, false, true);
|
||||
else label(ctx, name(st.bottom), x + w - o, y + o, fs, true);
|
||||
}
|
||||
}
|
||||
|
||||
function stem() {
|
||||
const d = st.data, clean = s => String(s || "").replace(/[^\w.-]+/g, "_").slice(0, 40);
|
||||
const stamp = new Date().toISOString().replace(/[:T]/g, "-").slice(0, 19);
|
||||
return `bfs_compare_${clean(d?.labels?.[0])}_vs_${clean(d?.labels?.[1])}_${st.mode}_${stamp}`;
|
||||
}
|
||||
|
||||
function download(href, name) {
|
||||
const a = el("a", { href, download: name });
|
||||
document.body.append(a); a.click(); a.remove();
|
||||
}
|
||||
|
||||
async function exportBlob(blob, ext, seconds) { // the browser's file -> ComfyUI converts it -> download
|
||||
const name = stem();
|
||||
if (format === "webm") {
|
||||
const href = URL.createObjectURL(blob);
|
||||
download(href, `${name}.${ext}`);
|
||||
setTimeout(() => URL.revokeObjectURL(href), 60000);
|
||||
return;
|
||||
}
|
||||
const id = [...crypto.getRandomValues(new Uint8Array(8))].map(b => b.toString(16).padStart(2, "0")).join("");
|
||||
rec.busy = "uploading"; buttons();
|
||||
try {
|
||||
for (let off = 0; off < blob.size; off += PART) {
|
||||
const r = await api.fetchApi(`/bfs/compare/part?id=${id}&offset=${off}`, { method: "POST", body: blob.slice(off, off + PART) });
|
||||
if (!r.ok) throw new Error((await r.json().catch(() => ({}))).error || `upload failed (${r.status})`);
|
||||
}
|
||||
rec.busy = "converting"; buttons();
|
||||
const r = await api.fetchApi("/bfs/compare/export", { method: "POST", headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ id, format, seconds, name }) });
|
||||
const f = await r.json();
|
||||
if (!r.ok || f.error) throw new Error(f.error || `conversion failed (${r.status})`);
|
||||
download(url(f), f.filename);
|
||||
} catch (e) {
|
||||
alert(`BFS Video Compare: could not convert the recording (${e.message}). Choose "browser file" to download it as recorded.`);
|
||||
} finally {
|
||||
rec.busy = ""; buttons();
|
||||
}
|
||||
}
|
||||
|
||||
function buttons() {
|
||||
if (rec.busy) { // uploading / converting the finished recording
|
||||
btnRec.classList.remove("rec");
|
||||
btnRec.disabled = btnSave.disabled = selFormat.disabled = true;
|
||||
btnSave.textContent = rec.busy === "uploading" ? "⬆ uploading…" : "⚙ converting…";
|
||||
btnRec.textContent = "● rec";
|
||||
return;
|
||||
}
|
||||
btnRec.disabled = selFormat.disabled = false;
|
||||
const busy = !!rec.mr;
|
||||
btnRec.classList.toggle("rec", busy);
|
||||
btnRec.textContent = busy ? (rec.full ? "■ stop" : `■ stop ${((performance.now() - rec.t0) / 1000).toFixed(0)}s`) : "● rec";
|
||||
btnSave.disabled = busy;
|
||||
btnSave.textContent = busy && rec.full ? `⬇ ${Math.round(100 * (vBot.currentTime / (vBot.duration || 1)))}%` : "⬇ save";
|
||||
}
|
||||
|
||||
function startRec(full = false) {
|
||||
if (!st.data || rec.mr || rec.busy) return false;
|
||||
if (!window.MediaRecorder || !HTMLCanvasElement.prototype.captureStream) { alert("This browser can't record a canvas (MediaRecorder)."); return false; }
|
||||
const aw = vBot.videoWidth || st.data.width, ah = vBot.videoHeight || st.data.height;
|
||||
const side = st.mode === "side", canvas = el("canvas");
|
||||
// the size of the layout it starts in (side by side is twice as wide); a later mode change is fitted into it
|
||||
canvas.width = (side ? 2 * aw + Math.round(2 * aw * 0.004) : aw) & ~1;
|
||||
canvas.videoHeight = ah & ~1; // the videos; the reference strip goes below them
|
||||
canvas.height = canvas.videoHeight + stripHeight(canvas.videoHeight);
|
||||
const ctx = canvas.getContext("2d");
|
||||
compose(ctx);
|
||||
const stream = canvas.captureStream(Math.max(30, Math.round(fps())));
|
||||
// the soundtrack is on the target. Not Firefox's mozCaptureStream: it takes the element's picture away from the canvas
|
||||
const vt = st.top === "target" ? vTop : st.bottom === "target" ? vBot : null;
|
||||
try { for (const t of vt?.captureStream?.()?.getAudioTracks?.() || []) stream.addTrack(t); } catch { }
|
||||
const type = pickType(stream.getAudioTracks().length > 0);
|
||||
if (!type) { for (const t of stream.getTracks()) t.stop(); alert("This browser can't record MP4 or WebM."); return false; }
|
||||
rec.type = type; rec.chunks = []; rec.full = full; rec.t0 = performance.now();
|
||||
rec.mr = new MediaRecorder(stream, { mimeType: type, videoBitsPerSecond: 16e6 });
|
||||
rec.mr.ondataavailable = e => { if (e.data?.size) rec.chunks.push(e.data); };
|
||||
rec.mr.onstop = () => {
|
||||
for (const t of stream.getTracks()) t.stop();
|
||||
const chunks = rec.chunks, seconds = (performance.now() - rec.t0) / 1000;
|
||||
rec.mr = null; rec.chunks = []; buttons();
|
||||
if (chunks.length) exportBlob(new Blob(chunks, { type: rec.type.split(";")[0] }), rec.type.includes("mp4") ? "mp4" : "webm", seconds);
|
||||
};
|
||||
rec.mr.start(1000);
|
||||
const frame = () => { if (!rec.mr) return; compose(ctx); buttons(); rec.raf = requestAnimationFrame(frame); };
|
||||
rec.raf = requestAnimationFrame(frame);
|
||||
buttons();
|
||||
return true;
|
||||
}
|
||||
|
||||
function stopRec() {
|
||||
if (!rec.mr) return;
|
||||
cancelAnimationFrame(rec.raf);
|
||||
if (rec.onEnd) { vBot.removeEventListener("ended", rec.onEnd); rec.onEnd = null; }
|
||||
if (rec.full) { for (const v of videos()) v.loop = true; if (st.playing) toggle(); }
|
||||
if (rec.mr.state !== "inactive") rec.mr.stop();
|
||||
}
|
||||
|
||||
async function saveFull() { // once from the first frame to the last, then stop and download
|
||||
if (!st.data || rec.mr || rec.busy) return;
|
||||
if (st.playing) toggle();
|
||||
for (const v of videos()) v.loop = false;
|
||||
const seeked = v => new Promise(res => { const t = setTimeout(res, 2000); v.addEventListener("seeked", () => { clearTimeout(t); res(); }, { once: true }); });
|
||||
const wait = Promise.all(videos().map(seeked));
|
||||
seek(0);
|
||||
await wait;
|
||||
if (!startRec(true)) { for (const v of videos()) v.loop = true; return; }
|
||||
rec.onEnd = () => { rec.onEnd = null; stopRec(); };
|
||||
vBot.addEventListener("ended", rec.onEnd, { once: true });
|
||||
toggle();
|
||||
}
|
||||
|
||||
return {
|
||||
setData(d) {
|
||||
stopRec();
|
||||
last.clear();
|
||||
refBox.replaceChildren(...(d.refs || []).map((f, k) => el("figure", { title: `Picture ${k + 1}` },
|
||||
[el("img", { src: url(f), crossorigin: "anonymous" }), el("figcaption", {}, `Picture ${k + 1}`)])));
|
||||
lblRefs.style.display = d.refs?.length ? "flex" : "none";
|
||||
showRefs();
|
||||
st.data = d;
|
||||
notes.textContent = (d.notes || []).join("\n");
|
||||
empty.style.display = "none";
|
||||
scrub.max = String(Math.max(0, d.frames - 1));
|
||||
refreshSelects();
|
||||
@@ -223,6 +479,7 @@ app.registerExtension({
|
||||
styles();
|
||||
const host = el("div", { style: "width:100%;height:100%;min-height:320px" });
|
||||
this._bfsCompare = buildPlayer(host);
|
||||
wheelToCanvas(host); // the wheel zooms the canvas over the player too
|
||||
this.addDOMWidget("bfs_video_compare", "div", host, { serialize: false, hideOnZoom: false, getMinHeight: () => 320 });
|
||||
this.setSize([Math.max(this.size[0], 560), Math.max(this.size[1], 620)]);
|
||||
new ResizeObserver(() => this._bfsCompare?.drawStrip()).observe(host);
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
/**
|
||||
* The mouse wheel over a node's HTML panel zooms / pans the canvas like anywhere else on the graph, unless the pointer
|
||||
* is on something that scrolls that way and still can (a list, a text box, a tall panel): that one scrolls, and once
|
||||
* it reaches its end the wheel goes back to the canvas. Ctrl / Cmd + wheel always goes to the canvas.
|
||||
*/
|
||||
import { app } from "../../scripts/app.js";
|
||||
|
||||
function scrolls(start, stop, dx, dy) {
|
||||
for (let n = start; n && n.nodeType === 1; n = n.parentElement) {
|
||||
if (n.tagName === "SELECT" && (n.size > 1 || n.multiple)) return true;
|
||||
const cs = getComputedStyle(n);
|
||||
if (dy && /(auto|scroll)/.test(cs.overflowY) && n.scrollHeight > n.clientHeight + 1
|
||||
&& (dy < 0 ? n.scrollTop > 0 : n.scrollTop + n.clientHeight < n.scrollHeight - 1)) return true;
|
||||
if (dx && /(auto|scroll)/.test(cs.overflowX) && n.scrollWidth > n.clientWidth + 1
|
||||
&& (dx < 0 ? n.scrollLeft > 0 : n.scrollLeft + n.clientWidth < n.scrollWidth - 1)) return true;
|
||||
if (n === stop) break;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
export function wheelToCanvas(el) {
|
||||
el.addEventListener("wheel", e => {
|
||||
const canvas = app.canvas?.canvas;
|
||||
if (!canvas || e.defaultPrevented || (!(e.ctrlKey || e.metaKey) && scrolls(e.target, el, e.deltaX, e.deltaY))) return;
|
||||
e.preventDefault();
|
||||
e.stopPropagation();
|
||||
canvas.dispatchEvent(new WheelEvent("wheel", {
|
||||
deltaX: e.deltaX, deltaY: e.deltaY, deltaZ: e.deltaZ, deltaMode: e.deltaMode, clientX: e.clientX, clientY: e.clientY,
|
||||
screenX: e.screenX, screenY: e.screenY, ctrlKey: e.ctrlKey, shiftKey: e.shiftKey, altKey: e.altKey, metaKey: e.metaKey,
|
||||
bubbles: true, cancelable: true,
|
||||
}));
|
||||
}, { passive: false });
|
||||
}
|
||||
@@ -17,10 +17,10 @@ every shot inside that range.
|
||||
| node | what it does |
|
||||
|---|---|
|
||||
| **BFS Shot Planner** | Pick a video, split it on a timeline, set a reference and prompt per shot. Outputs the shots as a ComfyUI **list**. |
|
||||
| **BFS Shot Unpack** | Opens one shot into plain values: guide frames, reference, second reference, prompt, length, first frame, audio, size, index. Use it with any model. |
|
||||
| **BFS Shot Unpack** | Opens one shot into plain values: guide frames, reference pictures (a list), prompt, length, first frame, audio, size, index. Use it with any model. |
|
||||
| **BFS Shot Repack** | Puts edited pieces back into a shot: Unpack's `shot` output goes to Repack's `shot`, the edited piece (e.g. the reference with its background removed) to its input. Unconnected inputs keep their values; timing, cuts and audio are unchanged. |
|
||||
| **BFS Shot H3 Conditioning** | The common MiniMax H3 path with the important controls only: aligned Add Guide and/or native video reference, references, optional audio conditioning, the Planner's inpaint mode and source-subject blur. |
|
||||
| **BFS Shot H3 Conditioning (Advanced)** | The full conditioning node, preserving the previous node and old workflows. Adds duet canvas, shifted RoPE, prompt writer, setting reference and experimental mask-guide controls. |
|
||||
| **BFS Shot H3 Conditioning (Advanced)** | The full conditioning node. Adds the setting reference, the first-frame anchor and the experimental mask-guide controls; its `model` input passes through to `model` (older workflows routed the model through it). The duet lives in **BFS Shot H3 Duet** (1.113 took it out of this node). |
|
||||
| **BFS Shot Join** | Concatenates the decoded shots in order, trims each to its true length, cross-fades soft joins and returns the soundtrack. With *comparison* on it also returns a side-by-side video (original shot \| references \| result) with the shot's info and your *label* on top and its prompt below, ready for Create Video. *save to = mp4 file* writes the video (and the comparison) straight to MP4 files instead, one shot in memory at a time (long videos); `video_path` gives the file. |
|
||||
| **BFS Shot H3 Duet** | Renders one shot with H3: the shot's clip pinned beside the video (duet, no LoRA), the shot as an aligned guide (for body-swap LoRAs), or both. |
|
||||
|
||||
@@ -61,14 +61,14 @@ Three run modes:
|
||||
stored on disk; then BFS Shot Join expands the part of the graph between the planner and itself (ComfyUI node
|
||||
expansion) for the next shot, so that shot is conditioned *after* the previous one exists and continuity
|
||||
(*head* / *history*) works. No browser re-queue (the tab can be closed), Save Video fires once with the full video,
|
||||
and after a failure a new Queue resumes from the stored shots (*Reset loop* starts over). Previews connected inside
|
||||
and after a failure a new Queue resumes from the stored shots (*Redo all* regenerates every shot). Previews connected inside
|
||||
the shot's part of the graph show every shot. Needs exactly one BFS Shot Planner before the join. Same idea as the
|
||||
recursive loop of Ethanfel's MiniMax H3 Context Loop.
|
||||
- **Queue loop (one shot per run)**: each run generates the next pending shot and stores it on
|
||||
disk. The join blocks every node after it until the last shot, then outputs the full video,
|
||||
so a Save Video node only fires once. With *Auto-queue the next shot* on (default), the panel re-queues
|
||||
the workflow after each shot, so one click still renders the whole video (keep the ComfyUI tab open: the panel
|
||||
does the re-queue). *Reset loop* starts over. Changing the plan starts a new run. It resumes from the stored
|
||||
does the re-queue). *Redo all* regenerates every shot. Changing the plan starts a new run. It resumes from the stored
|
||||
shots after a failure, and continuity *head* / *history* needs it: each shot reads the result of the one before.
|
||||
- **Auto loop (all shots at once)**: every shot in one queue run, each node runs for every shot before the next node.
|
||||
Holds all decoded shots in memory. Continuity *head* / *history* does not work here (all shots are conditioned
|
||||
@@ -133,8 +133,10 @@ card manages them.
|
||||
to the planner). The next Queue **runs only the shots that are not stored or whose settings changed**, plus the
|
||||
shots that **continue from a shot that was made again** (head, history, previous shot as reference / first
|
||||
frame), so every seam matches. Shot cards show 💾 (stored, up to date) or ⏳ (runs next time).
|
||||
- **Model, LoRA, seed, sampler:** the planner does not see them. After changing them, redo the shots (↻) or *Redo all*,
|
||||
or *Reset loop* (deletes the stored shots of this video).
|
||||
- **Model, LoRA, seed, sampler (1.112.0):** the planner reads them from the queued workflow (every node that feeds
|
||||
BFS Shot Join's `images`: loaders, LoRAs, sampler, seed, conditioning settings), so changing any of them runs every
|
||||
shot again. The join's own settings (crossfade, comparison) don't. A seed set to *randomize* changes on every Queue,
|
||||
so every Queue then runs every shot; fix the seed to keep the stored shots.
|
||||
- **↻ Redo** a shot (Stored shots card): it runs again on the next Queue, and the shots that continue from it run
|
||||
after it. Tick *↻ redoes only that shot* to keep the next shots as they are (the seam after it may then not match).
|
||||
- **🎬 Combine stored shots:** puts the stored shots together into an MP4 in the output folder **now, without
|
||||
@@ -236,16 +238,26 @@ Face identities are used for filtering and optional shot splitting. References a
|
||||
|
||||
## References and prompts
|
||||
|
||||
Every shot can have its own reference, second reference and prompt; shots without them use the
|
||||
global ones. Click an empty reference slot to upload an image straight from your computer; a filled
|
||||
slot has **✕** to remove it; **Copy to other shots** gives a shot's references to every shot. The last ten
|
||||
references you picked are shown under the shot editor for one-click reuse (click = reference,
|
||||
shift+click = second reference). Connected `ref_image`, `ref_image_2` and `prompt` inputs override
|
||||
the panel's global values, so an edited first frame or a prompt from another node can drive the
|
||||
defaults.
|
||||
Up to **12 reference pictures** (1.113.0): the global list starts with one slot and **+** adds more; they are
|
||||
`<Picture 1>` to `<Picture 12>`, in order. Click a slot to upload a picture (the **+** takes several files at once),
|
||||
or drop image files on a slot (replaces it) or anywhere on the box (adds them). A filled slot has **✕**: a global
|
||||
picture is taken out (the next ones move up), a shot's own picture falls back to the global one.
|
||||
|
||||
The planner also outputs `ref_image` and `ref_image_2`: the references the shots actually use,
|
||||
without repeats, so a plan where every shot shares one reference returns a single image.
|
||||
Every shot can have its own pictures and prompt. A shot's empty slot uses the global picture at the same position
|
||||
(shown faded), so a shot can change only `<Picture 2>` and keep the rest. **Copy to other shots** gives a shot's
|
||||
pictures to every shot. The last ten pictures you picked are shown under the shot editor: click = picture 1,
|
||||
shift+click = add it as the next picture, or drag it onto any slot.
|
||||
|
||||
Under every prompt (global and per shot) there is a chip per picture: **PictureN** inserts `<Picture N>` at the
|
||||
cursor, **📝** inserts `{details@Picture N}` (that picture's description, below).
|
||||
|
||||
The node no longer has `ref_image` / `ref_image_2` inputs (the pictures live in the panel); a workflow saved with
|
||||
them connected still runs, and the log says the images are ignored. A connected `prompt` input still overrides the
|
||||
global prompt. The planner outputs `refs` (every picture the shots use, without repeats, as a list) and `ref_count`
|
||||
where `ref_image` / `ref_image_2` were. **BFS Shot Unpack** outputs `refs`, every picture of the shot as a list (the next node runs once per picture,
|
||||
e.g. a background remover), and `ref_count`; **BFS Shot Repack**'s `refs` input takes that list back and gives each
|
||||
shot its own pictures (a workflow from before 1.113 that wired `ref_image` still works). **BFS Shot H3 Conditioning** passes every picture of the shot (the `use_ref_2` switch left in 1.114: the pictures are chosen in the planner); H3's own
|
||||
node stops at 9 and LoRAs trained like OpenJutsu saw up to 9, so the log notes a shot with more.
|
||||
|
||||
## Mask & crop (SAM 3)
|
||||
|
||||
@@ -274,11 +286,18 @@ box (yellow) over the original column (`comparison_mask`, on by default).
|
||||
1 regenerates the masked area completely. A grey mask (e.g. 0.85) keeps part of the original there: H3 puts those
|
||||
rows at *opacity* × the noise level, so they start from the original partly visible. 0.8-0.9 keeps pose, outline and lighting while still swapping; too low copies the original
|
||||
person. With `guide_mode = aligned guide` the model also sees the whole original shot as its guide, so it follows
|
||||
the motion either way. (Not applied with the duet panel.)
|
||||
the motion either way.
|
||||
|
||||
**🎨 Colored Mark** is an optional translucent overlay on the source video. Enable it before entering a SAM 3 description or opening **🎯 Points**; the selected color applies to that shot's marked region. This is only a visual marker: it does not create an inpainting mask or change **Full frame**, **Mask only** or crop behavior. The default colors are set under each global reference in **Prompts & refs**; choose a per-shot color when needed. The overlay keeps movement and facial expression visible.
|
||||
|
||||
To mark two regions in one shot, use the main description or points for **Ref 1**, then open **Mark a second region** and describe the second region for **Ref 2**. Give the two references different marker colors and enable *use_ref_2* on BFS Shot H3 Conditioning. The second region currently uses a text description.
|
||||
**🎨 Colored Mark** is an optional translucent overlay on the source video: each marked person is tinted with the
|
||||
color of the picture they become, and the prompt gets the mapping ("the person marked in red becomes the character
|
||||
from `<Picture 3>`"), for any model or LoRA. Every picture has its **Mark color** (under it in **Prompts & refs**:
|
||||
green, purple, red, blue, yellow, magenta, cyan, orange, white, brown, pink, olive by default); change it when it is
|
||||
too close to a color in the video. In a shot, turn on Colored Mark and pick which picture the shot's selection (its
|
||||
SAM 3 text or points) becomes; **+ Mark another person** adds a person with their own picture and text or 🎯 points
|
||||
(the points window has one point set per person). This is only a visual marker: it does not create an inpainting mask
|
||||
or change **Full frame**, **Mask only** or crop behavior. More than one marked person needs Crop off, and the
|
||||
shot must have every picture the marks use. The overlay keeps movement and facial
|
||||
expression visible. Plans before 1.113 (a second region for ref 2) open with that region as `<Picture 2>`.
|
||||
|
||||
```text
|
||||
body_swap: subject_definitions: <Subject 1> is the character shown in <Picture 1>, whose identity, face, head, hair or fur, skin, body shape, proportions and clothing define the green-marked replacement. <Subject 2> is the character shown in <Picture 2>, whose identity, face, head, hair or fur, skin, body shape, proportions and clothing define the purple-marked replacement. In the guide video, the translucent green and purple tints are temporary editing markers, not character appearance. Replace the green-tinted person only with <Subject 1> and the purple-tinted person only with <Subject 2>. The guide video provides the scene and the complete performance of both people, including their body poses, gestures, head motion, gaze, speech and facial expressions.
|
||||
@@ -290,10 +309,8 @@ detailed_description: [Shot 1] Replace the person marked by the translucent gree
|
||||
|
||||
The VLM's shot analysis suggests a SAM 3 text selection from the original frames; it does not assign guide colors or
|
||||
decide that two detections are the same identity. **Describe target** only describes the first shot selection. For
|
||||
two-color swaps, choose both selections and identities yourself, and use the explicit mapping above. The VLM prompt
|
||||
writer for *duet* is a separate split-screen task; its generic character-swap draft assumes the supplied pictures
|
||||
show one character, so use *planner prompt* for this two-character template. `Describe refs` also treats ref and ref 2
|
||||
as views of one identity; write the two subject definitions explicitly when ref 2 is a different person.
|
||||
two-color swaps, choose both selections and identities yourself, and use the explicit mapping above (or let
|
||||
🤖 Auto prompt write it: it is told which tinted person becomes which picture).
|
||||
|
||||
**Mask guide** (experimental, *Mask only* shots, BFS Shot H3 Conditioning → `mask_guide`): the shot with only the
|
||||
masked region visible, the rest grey, so the model looks at the subject's pose and outline on its own while the mask
|
||||
@@ -357,22 +374,96 @@ editor shows its suggestion with buttons to apply it. The summary output lists t
|
||||
|
||||
The VLM can describe the references, for any task (not tied to swaps), and any subject: a person, a creature, a robot,
|
||||
an animal or a cartoon character (it says what the subject is first). In the **Prompts & refs** tab pick an instruction
|
||||
preset: *short* (the default: one noun phrase of at most ~20 words that fits inside a sentence, e.g. `green goblin with
|
||||
preset: *object* (a noun phrase for a prop, product, vehicle...: what it is, shape, materials, colors, text), *short* (the default: one noun phrase of at most ~20 words that fits inside a sentence, e.g. `green goblin with
|
||||
long pointed ears in a torn brown leather vest`, for `<Subject 1> is the {details} shown in <Picture 1>`), *full body* (face, hair, skin, age, build, clothing piece by piece), *head / face*, *face attributes* (a
|
||||
short comma-separated list), *outfit*, or *custom* (your own instruction). **📝 Describe refs** writes one
|
||||
description per reference set: the global references and every shot's. They are saved in
|
||||
the plan and editable. Write **`{details}`** anywhere in a prompt and every shot gets the description of its own
|
||||
references; a set without a description is described by the VLM at run time (no click needed).
|
||||
description per picture (see below), saved in the plan and editable. Write **`{details}`** anywhere in a prompt and
|
||||
every shot gets the description of its own first picture; a picture without a description is described by the VLM
|
||||
at run time (no click needed).
|
||||
|
||||
**Image max side (VLM)** controls the longest side of reference and shot images prepared for VLM analysis (1024 px by
|
||||
default). Aspect ratio is preserved; the model's own image processor may resize them further. Increase it when fine
|
||||
reference details matter, or lower it to reduce analysis cost. This does not change the generated video size.
|
||||
|
||||
When a shot has two references, the planner expands a one-reference template at run time to include `<Subject 2>` and
|
||||
`{details2}` for `<Picture 2>`. The Describe refs panel provides a separate description field for each reference.
|
||||
`{details2}` for `<Picture 2>` (not when the prompt already names `<Picture 2>` or uses `{details@Picture n}`).
|
||||
Existing one-reference templates remain unchanged in shots with only one reference. You can also use `{details2}`
|
||||
explicitly in any template.
|
||||
|
||||
### 🤖 Auto prompt (1.113.0)
|
||||
|
||||
**Visual edits and color bindings (1.115.4):** the guide fixes the assigned performers' motion, action timing,
|
||||
head movement, gaze and expressions. Explicitly request the target style, background, lighting, weather, outfit
|
||||
or camera changes. For example: `Replace the man with the goblin from <Picture 1>, render the whole scene in anime
|
||||
style, and use the hall from <Picture 2> as the setting. Keep the guide performance and timing.` A setting can also
|
||||
be described in text. Use **full frame** generation for changes to the entire scene; masked generation only changes
|
||||
the generated region.
|
||||
|
||||
References can have different roles. In the example above, Picture 1 defines the character and Picture 2 defines
|
||||
the environment. To describe the latter, choose **scene / setting** in that picture's description preset. For several
|
||||
characters, bind colors only to their character references: `The subject marked in green uses <Picture 1>; the subject
|
||||
marked in purple uses <Picture 2>; <Picture 3> defines the setting.` Picture indices stay unchanged. A marking color
|
||||
identifies the same performer across position changes, occlusion and cuts; it is not a target skin/clothing color.
|
||||
Facial highlights are performance annotations. Prompt wording is generic and does not require a specific 2D/3D marking
|
||||
implementation. This update changes prompt writing, not the existing marking renderer or tracking behavior.
|
||||
|
||||
After updating, restart ComfyUI, hard-refresh the browser and click **Write all prompts** to regenerate previously
|
||||
written prompts with these rules. Existing written prompts are preserved until rewritten.
|
||||
|
||||
**1.115.5:** a setting replacement alone does not request a new rendering style. The writer preserves the source
|
||||
medium unless the actual request or an explicitly designated style reference changes it; instructional examples
|
||||
must not supply unrequested target attributes. For background replacement, use full-frame generation, disable
|
||||
paste/crop-only composition and inpainting outside a subject mask, set **first_frame** to **none**, and leave
|
||||
**setting_ref** off (that option adds the original source location). The full aligned guide still contains source
|
||||
scene geometry, so a correct prompt alone does not guarantee complete background replacement by a checkpoint.
|
||||
|
||||
In **Prompts & refs → 🤖 Auto prompt**, *Write the prompts with the VLM* turns the prompt boxes (global and per shot)
|
||||
into **requests** ("replace the bottle with the red can from Picture 2", "Picture 1 dances in the rain"), and the VLM
|
||||
writes each shot's MiniMax H3 prompt in the official full-reference format (MiniMaxAI/MiniMax-H3
|
||||
`docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md`): `subject_definitions`, `summary` with its task type, `retention_analysis`,
|
||||
`detailed_description` (`[Shot 1]`, the official camera vocabulary, sounds in time), `overall_soundscape` and
|
||||
`non_diegetic_music`. It looks at:
|
||||
|
||||
- **key frames of the shot** with their times (*Key frames*: auto = about 1.5 per second, 3 to 8);
|
||||
- **the shot's pictures** and their descriptions (Describe references);
|
||||
- **the request**, the shot's **Colored Marks** (which tinted person becomes which picture; the mapping is added if the
|
||||
writer leaves a color out), its selection / `{target}` and its mode (*Mask only*: only the subject is regenerated;
|
||||
*Crop*: the frames are a crop);
|
||||
- **how the source is given**: *guide video* (aligned guide, no tag) or `<Video 1>` (native reference video) — match the
|
||||
conditioning node's `guide_mode`.
|
||||
|
||||
**Template:** with Auto prompt on, the planner's `prompt` input (e.g. the template a LoRA was trained with,
|
||||
`body_swap: subject_definitions: ...`) is the **template** and the panel's text is the **request** applied to it
|
||||
("Troque a mulher no vídeo pela mulher da <Picture 1>"). The template is kept as it is: same sentences, same order, its
|
||||
trigger word first; the writer applies the request, adds the pictures' visible features and, after `[Shot 1]`, this
|
||||
shot's camera, place, light and actions. The panel's 🤖 Write uses the template of the last run (shown in the card).
|
||||
A request that is itself a whole prompt (H3 sections) is used as the template the same way. **Style** picks how a prompt without template is written: *H3
|
||||
official* (the six sections) or *OpenJutsu captions*, the short style of the OpenJutsu training captions ("Replace the
|
||||
dog in the video with the gaunt goblin … from <Picture 1>. Replace the cat … from <Picture 2>. Keep the same background
|
||||
and location as in the video, and maintain the motion, camera angle, and timing."). Without a template, a
|
||||
short request gets the full six-section prompt. Without a VLM, the template (else the request) is the prompt.
|
||||
|
||||
**Cuts:** the writer gets the shot's real camera cuts (the planner's cut detector) and writes `[Shot N] At MM:SS.mmm`
|
||||
exactly there, with at least one key frame in each piece between cuts; a shot with no cut gets no timestamp at all.
|
||||
Labels always keep their `< >`.
|
||||
|
||||
**🤖 Write all prompts** (or *only the missing*) writes them now; each shot shows its written prompt in the Shots tab,
|
||||
editable, with **🤖 Write / Write again**. A shot without one is written when the workflow runs (with the VLM on the
|
||||
planner, before any SAM 3 work) and the panel keeps it, so the next run uses the same text. When the request,
|
||||
pictures, marks, selection or mode change after a prompt was written, the shot says it is *older*; it is still used
|
||||
until you write it again. A written prompt is part of the stored shot's signature, so editing it re-renders the shot.
|
||||
Without a VLM and without a written prompt, the request is used as the prompt.
|
||||
|
||||
### One description per picture (1.113.0)
|
||||
|
||||
Descriptions are per picture: **📝 Describe refs** describes every picture the plan uses by itself, once per image.
|
||||
`{details}` is picture 1's description, `{details2}` picture 2's and **`{details@Picture n}`** (or
|
||||
`{details@Picture3}`) picture n's, so each shot gets the descriptions of *its* pictures. The dropdown next to the button
|
||||
sets what is described for every picture; a picture can have its own (e.g. *full body* for a character sheet, *head /
|
||||
face* for a close-up, *outfit*, or *custom* with its own instruction), and **🔄** describes only that picture again. A
|
||||
picture without a description is described at run time by the planner's VLM, else by the conditioning node's VLM, else
|
||||
it becomes "character". Descriptions saved by older plans (per reference set) are read as those pictures'.
|
||||
|
||||
**Fit the prompt** (on by default, next to 📝 Describe refs): the VLM gets the sentence of the prompt that holds the
|
||||
marker, with a blank in its place (e.g. `<Subject 1> is the ___ shown in <Picture 1>`), and writes the description for
|
||||
it, so the description adapts to the prompt and not the other way round (no "the A goblin", no paragraph inside a
|
||||
@@ -433,15 +524,37 @@ references before any SAM 3 work; with no VLM on the planner, the slot is left f
|
||||
**BFS Shot H3 Duet**, whose `vlm` input (the same Qwen3-VL CLIPLoader) describes them (the *short* preset, fitted to
|
||||
the sentence). Edited descriptions always win. With no VLM anywhere it becomes "character" and the log says so.
|
||||
|
||||
## Source blur (keep the mouth) (1.91.0)
|
||||
## Source blur (1.91.0, finer in 1.110.0)
|
||||
|
||||
`source_blur` on **BFS Shot H3 Conditioning**, **BFS Shot H3 Duet**, **BFS H3 Duet** and **BFS H3 Duet Conditioning**
|
||||
blurs the source person in what the model *sees* of the source (aligned guide, native reference video, duet panel),
|
||||
keeping the **mouth** sharp: the model still reads the pose, the motion and the lips but has no face to copy, so the
|
||||
identity comes from the reference. Masks from SAM 3.1 (the part, minus a slightly grown "mouth" mask), cached per shot.
|
||||
`face (keep mouth)`, `head + hands (keep mouth)` (also hides hair and hands), `person (keep mouth)` (the whole body:
|
||||
only for big changes, the pose then comes from the blurred shape). The kept area of *Mask only* and the output are
|
||||
never blurred. Trick shared by JalenBrunson (Sapiens2 there) for reliable ref2v swaps.
|
||||
`source_blur` on **BFS Shot H3 Conditioning** (and its Simple version), **BFS Shot H3 Duet**, **BFS H3 Duet** and
|
||||
**BFS H3 Duet Conditioning** blurs the source person in what the model *sees* of the source (aligned guide, native
|
||||
reference video, duet panel): the model still reads the pose and the motion but has no face to copy, so the identity
|
||||
comes from the reference. The kept area of *Mask only* and the output are never blurred. Trick shared by JalenBrunson
|
||||
(Sapiens2 there) for reliable ref2v swaps.
|
||||
|
||||
Modes read `<region> (<what stays sharp>)`:
|
||||
|
||||
| region | what is blurred |
|
||||
|---|---|
|
||||
| `face` | the face (and neck) |
|
||||
| `head + hands` | also the hair and the hands |
|
||||
| `person` | the whole body: only for big changes, the pose then comes from the blurred shape |
|
||||
| `masked area` | the shot's own mask (the person being replaced, with its expand); no extra segmentation. Without a mask on the shot (or in the plain Duet nodes) it blurs the whole person |
|
||||
|
||||
| sharp | |
|
||||
|---|---|
|
||||
| `keep mouth` | lips, teeth and tongue stay sharp (the lips carry a lot of the likeness and the lip sync) |
|
||||
| `keep lips + tongue` | the teeth are blurred too (always uses Sapiens2 for the lips) |
|
||||
| `mouth too` | nothing stays sharp |
|
||||
|
||||
`source_blur_segmenter`: **SAM 3.1** (text prompts, already used by the planner; the 1.91 behaviour) or **Sapiens2**
|
||||
(Meta's body-part segmentation, 28 classes with lips, teeth and tongue apart from the face; one pass for every part,
|
||||
finer around the mouth). Sapiens2 marks every person in the frame, like the SAM prompts here. Its weights
|
||||
(`facebook/sapiens2-seg-0.4b`, 1.6 GB) download to `models/sapiens2/` the first time; it runs on the GPU in half
|
||||
precision and goes back to RAM after each shot. Sapiens2 License.
|
||||
|
||||
`source_blur_effect`: **blur** (no detail left) or **invert colours** (keeps the shapes and the expression but not the
|
||||
skin tone or the colours; some users get better likeness with it).
|
||||
|
||||
## Big videos (upload)
|
||||
|
||||
@@ -529,7 +642,7 @@ BFS Shot Join ◄── images ────────────────
|
||||
└─► Create Video (fps, audio) ─► Save Video
|
||||
```
|
||||
|
||||
For other models, use **BFS Shot Unpack** and wire `guide_frames`, `ref_image`, `prompt` and
|
||||
For other models, use **BFS Shot Unpack** and wire `guide_frames`, `refs`, `prompt` and
|
||||
`length` into that model's own guide and reference nodes; the list mapping works the same way.
|
||||
|
||||
## Choosing who is replaced (`{target}`), step by step
|
||||
@@ -633,3 +746,19 @@ or *Crop + mask*.
|
||||
| Only the face or the hair is described | the click was on the head: click the body |
|
||||
| The other person is described | add an exclude point (right-click) on the other person, **👁 Segment**, save, describe again |
|
||||
| The swap still hits the wrong person | make the description more specific (position: "on the left", clothing colours) |
|
||||
|
||||
### Cancel a pending redo
|
||||
|
||||
After marking a stored shot for redo, click **Undo redo** to keep its previous take.
|
||||
**Clear redo selection** cancels all pending redo selections and preserves stored files and loop progress.
|
||||
Once a shot has been regenerated, its previous take cannot be restored through this control.
|
||||
**Refresh status** rereads the cache status; it cannot recover files deleted by older versions of Reset loop.
|
||||
Changing the seed, model or other render settings still invalidates stored shots even after cancelling redo.
|
||||
|
||||
### Experimental native temporal motion transfer
|
||||
|
||||
`BFS Shot H3 Conditioning` now has **native temporal alignment (experimental)** in `guide_mode`. Connect MODEL in and route its MODEL output to the guider. This uses the matching experimental LoRA, the shot as native `<Video 1>`, temporal RoPE alignment and soft temporal attention; it does not use Add Guide. Use 24 fps and disable continuity/first-frame/mask guides and VSA. Simple appends a MODEL output without moving existing outputs. The standalone **BFS MiniMax H3 Native Temporal Alignment (Experimental)** node provides the same patch for native workflows. See `H3_NATIVE_TEMPORAL.md` for the full contract and validation limits.
|
||||
|
||||
Native temporal performance (1.115.11): `native_temporal_backend` defaults to `flash partitions (no compile)`, preserving the same temporal penalty. `native_guide_scale` can reduce only guide resolution (try 0.5); output dimensions and frame count stay unchanged. The original Flex backend remains available for comparison.
|
||||
|
||||
Prompt bindings (1.115.12): completed short captions are no longer expanded into legacy two-subject templates. Auto-written prompts whose request/reference/mark signature is outdated are regenerated with a connected Planner VLM; otherwise use Write again before rendering. The writer rejects picture indices beyond the connected references after its bounded retry. Character outfits come from identity references, while requested setting changes override conflicting source scenery/lighting.
|
||||
|
||||
@@ -371,11 +371,11 @@ export const rawH = (...args) => vueH(...args);
|
||||
export const GRIDS = { "H3 (17n+5)": [17, 5], "LTX / Wan (8n+1)": [8, 1], "Wan (4n+1)": [4, 1], "any": [1, 0] };
|
||||
export const DEFAULTS = {
|
||||
video: "", fps: 24, grid: "H3 (17n+5)", mode: "shots", max_s: 4.5, min_s: 1.0, sensitivity: 0.5,
|
||||
max_parts: 0, max_total_s: 0, start_s: 0, end_s: 0, start_f: null, end_f: null, track: null, bounds: [], segs: [], global_ref: "", global_ref2: "", global_prompt: "", global_color_guide_ref1: "#00ff00", global_color_guide_ref2: "#9b30ff", mask_video: "",
|
||||
max_parts: 0, max_total_s: 0, start_s: 0, end_s: 0, start_f: null, end_f: null, track: null, bounds: [], segs: [], global_refs: [], global_ref: "", global_ref2: "", global_prompt: "", global_color_guide_ref1: "#00ff00", global_color_guide_ref2: "#9b30ff", mask_video: "",
|
||||
megapixels: 0.5, multiple: 32, generation_size_mode: "megapixels", custom_width: 1024, custom_height: 1024,
|
||||
second_pass: { enabled: false, mode: "scale", scale: 2, width: 1280, height: 704, megapixels: 1, align: 32 },
|
||||
detector: "adaptive", run: "recursive", between_shots: "keep", shot_cache: "output", shot_storage: "lossless", auto_continue: true,
|
||||
skip_fill: "original", audio_mode: "auto", ref_details: {}, mask_cfg: {}, vlm_cfg: {}, cast: {}, cast_split: false, cast_only: false,
|
||||
detector: "adaptive", run: "recursive", between_shots: "keep", shot_cache: "output", shot_storage: "lossless", unload_before_join: true, auto_continue: true,
|
||||
skip_fill: "original", audio_mode: "auto", ref_details: {}, picture_details: {}, auto_prompt: {}, auto_prompts: {}, mask_cfg: {}, vlm_cfg: {}, cast: {}, cast_split: false, cast_only: false,
|
||||
filters: { person: false, min_person_area: 0, max_persons: 0, face: false, skip_dark: false, dark_level: 0.06,
|
||||
skip_static: false, static_level: 0.004, min_frames: 0, samples: 6 },
|
||||
};
|
||||
@@ -470,3 +470,84 @@ export const secondPassSize = (w, h, sp) => {
|
||||
const al = Math.max(1, Math.round(+sp.align || 1)), r = x => Math.round(x / al) * al, c16 = x => Math.max(16, Math.round(x / 16) * 16);
|
||||
return { w: c16(r(tw)), h: c16(r(th)) };
|
||||
};
|
||||
|
||||
// ---- reference pictures (<Picture 1..12>)
|
||||
export const MAX_REFS = 100;
|
||||
// a reference list of the plan: `refs` (plans from 1.113 on), else the old ref / ref2 pair; empty slots at the end dropped
|
||||
export const refList = (refs, a = "", b = "") => {
|
||||
const out = (Array.isArray(refs) ? refs : [a, b]).map(x => (typeof x === "string" ? x : "")).slice(0, MAX_REFS);
|
||||
while (out.length && !out[out.length - 1]) out.pop();
|
||||
return out;
|
||||
};
|
||||
// the pictures a shot uses, in <Picture n> order: its own slot, else the global one at the same position; a slot with
|
||||
// neither is left out (same rule as bfs_shot_loop.shot_ref_names)
|
||||
export const shotRefNames = (own, glob) => {
|
||||
const out = [];
|
||||
for (let k = 0; k < Math.max(own.length, glob.length); k++) { const x = own[k] || glob[k]; if (x) out.push(x); }
|
||||
return out.slice(0, MAX_REFS);
|
||||
};
|
||||
// one slot per picture, then "+" (up to 12). Click: upload (the "+" takes several files at once); drop image files or a
|
||||
// recent thumbnail on a slot to replace it, anywhere else in the box to add them. `fallback`: the global pictures a
|
||||
// shot's empty slots use (shown faded). `extra(k, name)`: more controls under slot k.
|
||||
export function refBox(c, target, own, fallback = [], extra = () => null) {
|
||||
const n = Math.max(own.length, fallback.length);
|
||||
const drop = k => ({ onDragover: e => e.preventDefault(),
|
||||
onDrop: e => { e.preventDefault(); e.stopPropagation(); c.dropRefs(target, k, e); } });
|
||||
const slots = [];
|
||||
for (let k = 0; k < n; k++) {
|
||||
const name = own[k] || "", via = name ? "" : fallback[k] || "";
|
||||
slots.push(h("div", { class: "rslot" }, [
|
||||
h("div", { class: "refslot", ...drop(k),
|
||||
title: name ? `<Picture ${k + 1}>: ${name} (click to replace)` : via ? `<Picture ${k + 1}>: ${via} (global; click to use another picture in this shot)` : "click to upload",
|
||||
onClick: () => c.uploadRef(target, k) },
|
||||
name || via ? [h("img", { src: c.viewUrl(name || via), style: via ? "opacity:.45" : "" }), h("span", { class: "rnum" }, String(k + 1))]
|
||||
: [`⬆ ${k + 1}`]),
|
||||
h("div", { class: "rbtns" }, [name ? h("button", { title: target === "global" || k >= fallback.length ? "remove this picture"
|
||||
: "remove (use the global picture)", onClick: () => c.setRef(target, k, "") }, "✕") : null]),
|
||||
extra(k, name || via),
|
||||
]));
|
||||
}
|
||||
if (n < MAX_REFS) slots.push(h("div", { class: "rslot" }, [
|
||||
h("div", { class: "refslot add", ...drop(n), title: `add pictures (up to ${MAX_REFS}): click to upload one or more, or drop image files here`,
|
||||
onClick: () => c.uploadRef(target, n, true) }, "+"),
|
||||
h("div", { class: "rbtns" }, []),
|
||||
]));
|
||||
return h("div", { class: "refbox", ...drop(null) }, slots);
|
||||
}
|
||||
// chips under a prompt, one per picture: PictureN inserts <Picture N> at the cursor, 📝 inserts {details@Picture N}
|
||||
// (the description of that picture alone). The prompt box and its chips sit in one .pbox.
|
||||
export function promptChips(c, names, onSet) {
|
||||
if (!names.length) return null;
|
||||
const insert = (e, tag) => {
|
||||
e.preventDefault(); // mousedown: the prompt keeps its focus and cursor
|
||||
const ta = e.currentTarget.closest(".pbox")?.querySelector("textarea"); if (!ta) return;
|
||||
const v = ta.value, a = ta.selectionStart ?? v.length, b = ta.selectionEnd ?? v.length;
|
||||
const before = v.slice(0, a) + (a && !/\s$/.test(v.slice(0, a)) ? " " : "") + tag;
|
||||
onSet(before + v.slice(b));
|
||||
requestAnimationFrame(() => { ta.focus(); ta.setSelectionRange(before.length, before.length); });
|
||||
};
|
||||
return h("div", { class: "pchips" }, names.map((name, k) => h("span", { class: "pchip" }, [
|
||||
h("button", { class: "sm", title: `insert <Picture ${k + 1}> (${name})`, onMousedown: e => insert(e, `<Picture ${k + 1}>`) },
|
||||
[h("img", { src: c.viewUrl(name) }), `Picture${k + 1}`]),
|
||||
h("button", { class: "sm", title: `insert {details@Picture ${k + 1}}: the description of this picture (📝 Describe refs, or the VLM at run time)`,
|
||||
onMousedown: e => insert(e, `{details@Picture ${k + 1}}`) }, "📝"),
|
||||
])));
|
||||
}
|
||||
|
||||
// ---- Colored Mark: one color per picture (<Picture 1> first; the first seven are the colors of the first mark sets)
|
||||
export const MARK_PALETTE = [["#00ff00", "green"], ["#9b30ff", "purple"], ["#ff2020", "red"], ["#168cff", "blue"],
|
||||
["#ffe600", "yellow"], ["#ff00d4", "magenta"], ["#00e5c0", "cyan"], ["#ff8000", "orange"], ["#ffffff", "white"],
|
||||
["#8b4513", "brown"], ["#ff9ec4", "pink"], ["#808000", "olive"]];
|
||||
export const markName = color => (MARK_PALETTE.find(([v]) => v === String(color).toLowerCase()) || [0, color])[1];
|
||||
// the mark color of <Picture k> (same rule as bfs_shot_loop.picture_color)
|
||||
export const pictureColor = (plan, k) => {
|
||||
const c = Array.isArray(plan.picture_colors) ? plan.picture_colors[k - 1] : "";
|
||||
return /^#[0-9a-f]{6}$/i.test(c || "") ? c.toLowerCase() : MARK_PALETTE[(k - 1) % MARK_PALETTE.length][0];
|
||||
};
|
||||
// a shot's extra Colored Mark regions (the second person on): plans before 1.113 had one (text2 / points2 = picture 2)
|
||||
export const markRegions = m => Array.isArray(m.color_guide_regions) ? m.color_guide_regions
|
||||
: (m.color_guide_text2 || (m.color_guide_points2 || []).length)
|
||||
? [{ picture: 2, text: m.color_guide_text2 || "", points: m.color_guide_points2 || [], key: m.color_guide_key2 || 0 }] : [];
|
||||
|
||||
// ---- 🤖 Auto prompt: the VLM writes each shot's H3 prompt from the request in the prompt box
|
||||
export const AUTO_DEFAULTS = { enabled: false, frames: 0, source: "guide", style: "h3" };
|
||||
|
||||
@@ -140,6 +140,14 @@ const CSS = `
|
||||
.bsl .refslot:hover{border-color:var(--acc)}
|
||||
.bsl .refslot.sm{width:56px;height:56px;font-size:10px}
|
||||
.bsl .refslot.via{border-style:solid;border-color:#3b5b3b}
|
||||
.bsl .refslot{position:relative}
|
||||
.bsl .refslot .rnum{position:absolute;left:3px;top:2px;font-size:10px;padding:0 4px;border-radius:4px;background:#000a;color:#fff}
|
||||
.bsl .refslot.add{font-size:22px;color:#8a8a99}
|
||||
.bsl .refbox{flex-wrap:wrap;max-width:100%;align-items:flex-start}
|
||||
.bsl .pchips{display:flex;gap:4px;flex-wrap:wrap;margin-top:4px}
|
||||
.bsl .pchip{display:inline-flex;gap:1px}
|
||||
.bsl .pchip button{display:inline-flex;align-items:center;gap:3px;padding:1px 5px;font-size:10px}
|
||||
.bsl .pchip img{width:16px;height:16px;border-radius:3px;object-fit:cover}
|
||||
.bsl .rslot{display:flex;flex-direction:column;gap:3px;align-items:center}
|
||||
.bsl .guidecolor{display:flex;flex-direction:column;align-items:center;gap:2px;max-width:112px}
|
||||
.bsl .guidecolor select{width:100%;font-size:10px;padding:2px 4px}
|
||||
@@ -191,13 +199,17 @@ const CSS = `
|
||||
|
||||
/* points window (teleported to <body>) */
|
||||
.bsl .mmodal{position:fixed;inset:0;background:rgba(0,0,0,.72);z-index:10000;display:flex;align-items:center;justify-content:center}
|
||||
.bsl .mbox{background:#1d1d23;border:1px solid #3a3a45;border-radius:10px;padding:12px;width:min(960px,94vw);max-height:94vh;overflow:auto}
|
||||
.bsl .mimg{position:relative;cursor:crosshair;user-select:none;line-height:0}
|
||||
.bsl .mimg img{width:100%;border-radius:6px}
|
||||
.bsl .mbox{background:#1d1d23;border:1px solid #3a3a45;border-radius:10px;padding:12px;width:min(960px,94vw);height:min(900px,94vh);overflow:auto;box-sizing:border-box}
|
||||
/* the frame fits a fixed stage whatever the video's size or orientation; points are stored relative to the frame */
|
||||
.bsl .mstage{display:flex;justify-content:center;align-items:center;height:min(540px,56vh);background:#0d0d10;border-radius:6px}
|
||||
.bsl .mimg{position:relative;cursor:crosshair;user-select:none;line-height:0;display:inline-block}
|
||||
.bsl .mimg img{display:block;max-width:100%;max-height:min(540px,56vh);border-radius:6px}
|
||||
.bsl .mthumb{width:22px;height:22px;border-radius:4px;object-fit:cover;vertical-align:middle;border:2px solid var(--mark,#555)}
|
||||
.bsl .pt{position:absolute;width:14px;height:14px;margin:-7px 0 0 -7px;border-radius:50%;border:2px solid #fff;cursor:pointer}
|
||||
.bsl .pt.pos{background:#3ccf6b}.bsl .pt.neg{background:#e8455a}
|
||||
.bsl .pt.cg.pos{background:var(--point-color,#00ff00)}
|
||||
.bsl .pt.r2{box-shadow:0 0 0 2px #9b30ff}
|
||||
.bsl .pt.other{opacity:.5}
|
||||
.bsl .mreg{display:flex;gap:6px;align-items:center;flex-wrap:wrap;margin-top:6px}
|
||||
`;
|
||||
|
||||
export function styles() {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
// The busy bar and the points window (🎯 Points…: click on a frame to pick the subject for SAM 3).
|
||||
import { Teleport } from "../vendor/vue.esm-browser.prod.mjs";
|
||||
import { h, hint, t } from "./common.js";
|
||||
import { h, hint, t, pictureColor, markName } from "./common.js";
|
||||
|
||||
export function busyBar(c, msg) {
|
||||
const { busySince, now, upPct, status } = c;
|
||||
@@ -18,30 +18,34 @@ export function pointsModal(c) {
|
||||
const { modal, segs, plan, api, setMeta } = c;
|
||||
const ms = segs.value[modal.idx];
|
||||
if (!modal.open || !ms) return null;
|
||||
// Colored Mark: region 1 = the shot's own selection, 2.. = its extra regions; each one is a picture, in its color
|
||||
const regionPicture = r => (r === 1 ? ms.color_guide_picture : ms.regions[r - 2]?.picture) || 1;
|
||||
const regionColor = r => pictureColor(plan, regionPicture(r));
|
||||
const regionsN = ms.color_guide ? 1 + ms.regions.length : 1;
|
||||
const pointOverlays = ms.color_guide
|
||||
? [1, 2].flatMap(region => (region === modal.region ? modal.points : modal.groups[region]?.points || []).map((p, k) => h("div", {
|
||||
class: ["pt", "cg", region === 2 && "r2", p.label ? "pos" : "neg"],
|
||||
style: `left:${p.x * 100}%;top:${p.y * 100}%;--point-color:${region === 2 ? (ms.color_guide_color2 || plan.global_color_guide_ref2 || "#9b30ff") : (ms.color_guide_color || plan.global_color_guide_ref1 || "#00ff00")}`,
|
||||
title: `Region ${region}${region === modal.region ? " · click to remove" : ""}`,
|
||||
? Array.from({ length: regionsN }, (_, q) => q + 1).flatMap(region => (region === modal.region ? modal.points : modal.groups[region]?.points || []).map((p, k) => h("div", {
|
||||
class: ["pt", "cg", region !== modal.region && "other", p.label ? "pos" : "neg"],
|
||||
style: `left:${p.x * 100}%;top:${p.y * 100}%;--point-color:${regionColor(region)}`,
|
||||
title: `Region ${region} · Picture ${regionPicture(region)}${region === modal.region ? " · click to remove" : ""}`,
|
||||
onPointerdown: e => { if (region !== modal.region) return; e.stopPropagation(); modal.points = modal.points.filter((_, j) => j !== k); modal.prev = null; },
|
||||
})))
|
||||
: modal.points.map((p, k) => h("div", { class: ["pt", p.label ? "pos" : "neg"], style: `left:${p.x * 100}%;top:${p.y * 100}%`,
|
||||
title: "click to remove", onPointerdown: e => { e.stopPropagation(); modal.points = modal.points.filter((_, j) => j !== k); modal.prev = null; } }));
|
||||
const regionButton = r => {
|
||||
const k = regionPicture(r), name = ms.names[k - 1];
|
||||
return h("button", { class: modal.region === r ? "pri" : "", onClick: () => c.switchPointRegion(r),
|
||||
title: `Region ${r}: becomes <Picture ${k}> (${markName(regionColor(r))})` }, [
|
||||
name ? h("img", { class: "mthumb", src: c.viewUrl(name), style: `--mark:${regionColor(r)}` }) : null,
|
||||
` Picture ${k} · ${markName(regionColor(r))} (${(r === modal.region ? modal.points : modal.groups[r]?.points || []).length})`]);
|
||||
};
|
||||
return h(Teleport, { to: "body" }, h("div", { class: "bsl", style: "background:none;padding:0" }, h("div", { class: "mmodal",
|
||||
onKeydown: e => e.stopPropagation(), onPointerdown: e => { if (e.target === e.currentTarget) modal.open = false; } }, [
|
||||
h("div", { class: "mbox" }, [
|
||||
h("div", { class: "row", style: "margin-bottom:6px" }, [h("span", { class: "sect" }, `Shot #${modal.idx + 1} · points`),
|
||||
ms.color_guide ? h("label", { class: "row", style: "gap:6px;align-items:center" }, [h("span", { class: "hint" }, `Region ${modal.region} · color`),
|
||||
h("select", { value: modal.region === 2 ? (ms.color_guide_color2 || plan.global_color_guide_ref2 || "#9b30ff") : (ms.color_guide_color || plan.global_color_guide_ref1 || "#00ff00"), style: "width:auto", onChange: e => {
|
||||
setMeta(modal.idx, modal.region === 2 ? "color_guide_color2" : "color_guide_color", e.target.value);
|
||||
if (modal.prev) c.previewMask(modal.idx, { points: modal.points, key: modal.key, text: modal.text,
|
||||
preview_color: e.target.value }, true);
|
||||
} },
|
||||
[["#00ff00", "Green"], ["#9b30ff", "Purple"], ["#ff2020", "Red"], ["#168cff", "Blue"], ["#ffe600", "Yellow"], ["#ff00d4", "Magenta"], ["#00e5c0", "Cyan"]].map(([v, n]) => h("option", { value: v, disabled: ms.color_guide && v === (modal.region === 2 ? (ms.color_guide_color || "#00ff00") : (ms.color_guide_color2 || "#9b30ff")) }, n)))]) : null,
|
||||
ms.color_guide ? h("div", { class: "row" }, [hint("Colored Mark keeps two independent point sets; switch regions to edit each person. Both use the frame selected below."),
|
||||
h("button", { class: modal.region === 1 ? "pri" : "", onClick: () => c.switchPointRegion(1) }, `Region 1 (${modal.groups[1]?.points?.length || 0})`),
|
||||
h("button", { class: modal.region === 2 ? "pri" : "", onClick: () => c.switchPointRegion(2) }, `Region 2 (${modal.groups[2]?.points?.length || 0})`)]) : null,
|
||||
hint("click = keep (green) · right-click or shift+click = exclude (red) · click a point to remove it")]),
|
||||
ms.color_guide ? h("div", { class: "row" }, [hint("One point set per marked person; switch to edit each. All use the frame selected below."),
|
||||
...Array.from({ length: regionsN }, (_, q) => regionButton(q + 1))]) : null,
|
||||
hint("click = keep · right-click or shift+click = exclude · click a point to remove it")]),
|
||||
h("div", { class: "mstage" }, [
|
||||
h("div", { class: "mimg", onContextmenu: e => e.preventDefault(), onPointerdown: e => {
|
||||
const box = e.currentTarget.getBoundingClientRect();
|
||||
const x = (e.clientX - box.left) / box.width, y = (e.clientY - box.top) / box.height;
|
||||
@@ -50,7 +54,7 @@ export function pointsModal(c) {
|
||||
} }, [
|
||||
h("img", { draggable: false, src: api.apiURL(`/bfs/shotloop/frame?video=${encodeURIComponent(plan.video)}&fps=${plan.fps}&f=${ms.start + modal.key}&w=900`) }),
|
||||
...pointOverlays,
|
||||
]),
|
||||
])]),
|
||||
h("div", { class: "row", style: "margin-top:6px" }, [
|
||||
hint(`frame ${ms.start + modal.key} (${modal.key + 1}/${ms.len})`),
|
||||
h("input", { type: "range", min: 0, max: ms.len - 1, step: 1, value: modal.key, style: "flex:1",
|
||||
@@ -62,8 +66,7 @@ export function pointsModal(c) {
|
||||
h("button", { onClick: () => { modal.points = []; modal.prev = null; } }, "Clear points"),
|
||||
h("button", { class: "pri", disabled: !!modal.busy || !(modal.points.length || modal.text),
|
||||
onClick: () => c.previewMask(modal.idx, { points: modal.points, key: modal.key, text: modal.text,
|
||||
preview_color: modal.region === 2 ? (ms.color_guide_color2 || plan.global_color_guide_ref2 || "#9b30ff")
|
||||
: (ms.color_guide_color || plan.global_color_guide_ref1 || "#00ff00") }, true) }, "👁 Segment"),
|
||||
preview_color: ms.color_guide ? regionColor(modal.region) : undefined }, true) }, "👁 Segment"),
|
||||
]),
|
||||
modal.busy ? busyBar(c, modal.busy) : null,
|
||||
modal.prev ? h("div", { class: "mstrip" }, [...modal.prev.frames.map(f => h("img", { src: f.src, title: `frame ${f.f}` })),
|
||||
|
||||
@@ -1,42 +1,37 @@
|
||||
// 📝 Prompts & refs tab: the global reference / prompt, the VLM (shot suggestions) and the reference descriptions ({details}).
|
||||
import { h, fld, check, section, pill, hint, select } from "./common.js";
|
||||
import { h, fld, check, section, pill, hint, select, refBox, promptChips, MAX_REFS, MARK_PALETTE, pictureColor } from "./common.js";
|
||||
|
||||
const GUIDE_COLORS = [["#00ff00", "Green"], ["#9b30ff", "Purple"], ["#ff2020", "Red"], ["#168cff", "Blue"],
|
||||
["#ffe600", "Yellow"], ["#ff00d4", "Magenta"], ["#00e5c0", "Cyan"]];
|
||||
const PRESETS = ["short", "object", "full body", "head / face", "face attributes", "outfit", "scene / setting", "custom"];
|
||||
|
||||
function globalSection(c) {
|
||||
const { plan } = c;
|
||||
const slot = (name, label, field) => h("div", { class: "rslot" }, [
|
||||
h("div", { class: "refslot", title: name ? name + " (click to replace)" : "click to upload", onClick: () => c.uploadRef("global", field) },
|
||||
name ? [h("img", { src: c.viewUrl(name) })] : [label]),
|
||||
h("div", { class: "rbtns" }, [name ? h("button", { title: "remove", onClick: () => c.setRef("global", field, "") }, "✕") : null]),
|
||||
h("label", { class: "guidecolor" }, [
|
||||
h("span", { class: "hint" }, field === "global_ref" ? "Marker color · Ref 1" : "Marker color · Ref 2"),
|
||||
h("select", { value: field === "global_ref" ? (plan.global_color_guide_ref1 || "#00ff00") : (plan.global_color_guide_ref2 || "#9b30ff"),
|
||||
disabled: !name,
|
||||
title: "Default Colored Mark for this global reference",
|
||||
onChange: e => c.setPlan(field === "global_ref" ? "global_color_guide_ref1" : "global_color_guide_ref2", e.target.value) },
|
||||
GUIDE_COLORS.map(([v, text]) => h("option", { value: v }, text))),
|
||||
]),
|
||||
// Colored Mark: every picture has its mark color (a person marked in it becomes that picture)
|
||||
const marker = k => h("label", { class: "guidecolor" }, [
|
||||
h("span", { class: "hint" }, "Mark color"),
|
||||
h("select", { value: pictureColor(plan, k + 1), title: `Colored Mark color of <Picture ${k + 1}>`,
|
||||
onChange: e => c.setPictureColor(k + 1, e.target.value) },
|
||||
MARK_PALETTE.map(([v, name]) => h("option", { value: v }, name))),
|
||||
]);
|
||||
return section("Global reference & prompt", [
|
||||
h("div", { class: "row", style: "align-items:flex-start;gap:10px" }, [
|
||||
h("div", { class: "refbox" }, [slot(plan.global_ref, "⬆ global\nreference", "global_ref"), slot(plan.global_ref2, "⬆ global\nref 2", "global_ref2")]),
|
||||
h("div", { style: "flex:1;min-width:200px" }, [
|
||||
h("textarea", { style: "min-height:78px", placeholder: "Global prompt (a connected `prompt` input overrides it)",
|
||||
value: plan.global_prompt, onChange: e => c.setPlan("global_prompt", e.target.value) }),
|
||||
]),
|
||||
refBox(c, "global", plan.global_refs, [], marker),
|
||||
h("div", { class: "pbox", style: "margin-top:6px" }, [
|
||||
h("textarea", { style: "min-height:78px", placeholder: plan.auto_prompt?.enabled
|
||||
? "Request for every shot (🤖 Auto prompt writes the H3 prompt from it), e.g. 'replace the bottle with the red can from Picture 2'"
|
||||
: "Global prompt (a connected `prompt` input overrides it)",
|
||||
value: plan.global_prompt, onChange: e => c.setPlan("global_prompt", e.target.value) }),
|
||||
promptChips(c, plan.global_refs.filter(Boolean), v => c.setPlan("global_prompt", v)),
|
||||
]),
|
||||
h("div", { class: "hint", style: "margin-top:4px" }, [
|
||||
"Used by shots without their own. Connected ref_image / ref_image_2 / prompt inputs on the node override these. Placeholders: ",
|
||||
`Used by shots without their own (up to ${MAX_REFS} pictures, <Picture 1> first; click + or drop image files on the box). A connected prompt input on the node overrides the global prompt. Placeholders: `,
|
||||
h("b", "{target}"), " who is replaced (per shot) · ", h("b", "{details}"), " description of the shot's references · ",
|
||||
h("b", "{details@Picture n}"), " description of picture n alone · ",
|
||||
h("b", "{shot}"), " the VLM's description of the shot · ", h("b", "{setting}"), " the setting picture (H3 Conditioning)."]),
|
||||
h("div", { class: "hint", style: "margin-top:4px" }, "These are the default marker colors for the global references. Turn on Colored Mark on a shot to apply them; shot-level colors can override them."),
|
||||
h("div", { class: "hint", style: "margin-top:4px" }, "Mark color: the Colored Mark color of each picture. A person marked for <Picture n> in a shot is drawn in that color, and the prompt says which picture the color becomes. Change it when it is too close to a color in the video."),
|
||||
], { sub: "defaults for every shot" });
|
||||
}
|
||||
|
||||
function vlmSection(c) {
|
||||
const { plan, vlmSug, busy, refSets } = c;
|
||||
const { plan, vlmSug, busy, pictures } = c;
|
||||
const VC = plan.vlm_cfg || {};
|
||||
const opt = (k, label, title) => check(VC[k], v => c.setVlmCfg(k, v), label, title);
|
||||
const sugN = Object.keys(vlmSug.value).length;
|
||||
@@ -65,42 +60,81 @@ function vlmSection(c) {
|
||||
right: h("span", { class: "row" }, [VC.enabled ? pill("on", "ok") : pill("off"), sugN ? pill(`${sugN} suggestions`) : null]) }),
|
||||
section("Describe references ({details})", [
|
||||
h("div", { class: "row", style: "gap:8px" }, [
|
||||
select(VC.describe_preset, ["short", "full body", "head / face", "face attributes", "outfit", "custom"], v => c.setVlmCfg("describe_preset", v), { style: "width:auto" }),
|
||||
h("button", { class: "pri", disabled: !!busy.value || !refSets.value.length, onClick: c.describeRefs }, "📝 Describe refs"),
|
||||
h("button", { disabled: !!busy.value || !refSets.value.length,
|
||||
title: "Try another description: a new seed (and some temperature if it is 0), then describe again",
|
||||
onClick: () => { if (!(+VC.temperature > 0)) c.setVlmCfg("temperature", 0.7); c.setVlmCfg("seed", (+VC.seed || 0) + 1); c.describeRefs(); } }, "🎲"),
|
||||
fld("Describe", select(VC.describe_preset, PRESETS, v => c.setVlmCfg("describe_preset", v), { style: "width:auto" }),
|
||||
"what every picture gets, unless it has its own below"),
|
||||
h("button", { class: "pri", disabled: !!busy.value || !pictures.value.length, onClick: () => c.describePictures() }, "📝 Describe refs"),
|
||||
h("button", { disabled: !!busy.value || !pictures.value.length,
|
||||
title: "Try other descriptions: a new seed (and some temperature if it is 0), then describe again",
|
||||
onClick: () => { if (!(+VC.temperature > 0)) c.setVlmCfg("temperature", 0.7); c.setVlmCfg("seed", (+VC.seed || 0) + 1); c.describePictures(); } }, "🎲"),
|
||||
fld("Temperature", h("input", { type: "number", min: 0, max: 1.5, step: 0.1, value: VC.temperature ?? 0, style: "width:70px",
|
||||
title: "0 = always the same answer (greedy); 0.5-1 = varied answers, picked by the seed",
|
||||
onChange: e => c.setVlmCfg("temperature", Math.max(0, parseFloat(e.target.value) || 0)) })),
|
||||
fld("Seed", h("input", { type: "number", min: 0, step: 1, value: VC.seed ?? 0, style: "width:80px",
|
||||
title: "Used when the temperature is above 0: another seed, another description",
|
||||
onChange: e => c.setVlmCfg("seed", Math.max(0, parseInt(e.target.value) || 0)) })),
|
||||
hint(`${refSets.value.length} reference set(s) · editable below`),
|
||||
hint(`${pictures.value.length} picture(s) · editable below`),
|
||||
check(VC.fit_prompt !== false, v => c.setVlmCfg("fit_prompt", v), "fit the prompt",
|
||||
"The VLM gets the global prompt's sentence that holds {details} (and the shot prompt's {target} sentence for 🧑 Describe target) with a blank in its place, and writes the description to fit it: the description adapts to the prompt, not the other way round."),
|
||||
"The VLM gets the global prompt's sentence about that picture ({details}, {details2} or {details@Picture n}; and the shot prompt's {target} sentence for 🧑 Describe target) with a blank in its place, and writes the description to fit it: the description adapts to the prompt, not the other way round."),
|
||||
]),
|
||||
VC.describe_preset === "custom" ? h("textarea", { style: "margin-top:6px;min-height:52px", value: VC.describe_custom || "",
|
||||
placeholder: "Your instruction for the VLM, e.g. 'Describe the character's costume and props piece by piece…'",
|
||||
onChange: e => c.setVlmCfg("describe_custom", e.target.value) }) : null,
|
||||
refSets.value.length ? h("div", { class: "dlist" }, refSets.value.map(x => {
|
||||
const saved = plan.ref_details?.[x.key] || "";
|
||||
const input = (field, placeholder) => h("textarea", { style: "min-height:44px", value:
|
||||
typeof saved === "object" ? (saved[field] || "") : field === "details" ? saved : "",
|
||||
placeholder, onChange: e => c.setDetail(x.key, { ...(typeof saved === "object" ? saved : { details: saved }), [field]: e.target.value }) });
|
||||
pictures.value.length ? h("div", { class: "dlist" }, pictures.value.map(x => {
|
||||
const own = plan.picture_cfg?.[x.name] || {};
|
||||
return h("div", { class: "drow" }, [
|
||||
h("div", { class: "dthumbs" }, [x.a ? h("img", { src: c.viewUrl(x.a) }) : null, x.b ? h("img", { src: c.viewUrl(x.b) }) : null]),
|
||||
h("div", { class: "dthumbs" }, [h("img", { src: c.viewUrl(x.name) })]),
|
||||
h("div", { style: "flex:1" }, [
|
||||
hint(x.where.length > 4 ? `${x.where.slice(0, 4).join(", ")} +${x.where.length - 4}` : x.where.join(", ")),
|
||||
x.b ? h("div", { class: "grid" }, [input("details", "Ref 1 description ({details})"), input("details2", "Ref 2 description ({details2})")])
|
||||
: input("details", "no description yet: Describe refs, write your own, or leave it to the VLM at run time"),
|
||||
h("div", { class: "row", style: "gap:6px;margin-bottom:3px" }, [
|
||||
hint(`Picture n in ${x.where.length > 4 ? `${x.where.slice(0, 4).join(", ")} +${x.where.length - 4}` : x.where.join(", ")}`),
|
||||
h("span", { class: "grow" }),
|
||||
select(own.preset || "", [["", `default (${VC.describe_preset || "short"})`], ...PRESETS],
|
||||
v => c.setPictureCfg(x.name, "preset", v), { style: "width:auto", title: "What to describe in this picture" }),
|
||||
h("button", { class: "sm", disabled: !!busy.value, title: "Describe only this picture again",
|
||||
onClick: () => c.describePictures([x.name]) }, "🔄"),
|
||||
]),
|
||||
own.preset === "custom" ? h("textarea", { style: "min-height:36px;margin-bottom:3px", value: own.custom || "",
|
||||
placeholder: "Instruction for this picture, e.g. 'Describe only the outfit, piece by piece'",
|
||||
onChange: e => c.setPictureCfg(x.name, "custom", e.target.value) }) : null,
|
||||
h("textarea", { style: "min-height:44px", value: plan.picture_details?.[x.name] || "",
|
||||
placeholder: "no description yet: 📝 Describe refs, 🔄, write your own, or leave it to the VLM at run time",
|
||||
onChange: e => c.setPictureDetail(x.name, e.target.value) }),
|
||||
]),
|
||||
]);
|
||||
})) : hint("Add a reference (global, per shot or per shot) first."),
|
||||
], { sub: "write {details} in any prompt: each shot gets the description of its own references" }),
|
||||
})) : hint("Add a reference (global or per shot) first."),
|
||||
], { sub: "{details} = picture 1, {details2} = picture 2, {details@Picture n} = picture n: each shot gets the descriptions of its own pictures" }),
|
||||
];
|
||||
}
|
||||
|
||||
// 🤖 Auto prompt: the VLM writes each shot's prompt in H3's full-reference format from key frames of the shot, its
|
||||
// pictures and their descriptions, the request in the prompt box and the shot's marks and mode
|
||||
function autoSection(c) {
|
||||
const { plan, busy, segs } = c;
|
||||
const A = plan.auto_prompt || {};
|
||||
const S = segs.value, missing = S.filter(s => !c.writtenOf(s)?.text).length;
|
||||
const stale = S.filter(s => c.writtenOf(s)?.text && c.writtenOf(s).for !== c.autoFor(s)).length;
|
||||
return section("🤖 Auto prompt (H3)", [
|
||||
h("div", { class: "row", style: "gap:12px;align-items:center" }, [
|
||||
check(A.enabled, v => c.setAuto("enabled", v), "Write the prompts with the VLM",
|
||||
"On: the prompt boxes (global and per shot) hold your request ('change the bottle into a red can', 'Picture 1 dances in the rain'); the VLM writes each shot's MiniMax H3 prompt from key frames of the shot, the pictures and their descriptions, the request and the shot's Colored Marks, selection and mode, in H3's official full-reference format."),
|
||||
fld("Style", select(A.style || "h3", [["h3", "H3 official (six sections)"], ["captions", "Short edit request"]],
|
||||
v => c.setAuto("style", v), { style: "width:auto" }), "Choose prompt format. The guide performance stays fixed; request style, setting, lighting, outfit or camera changes explicitly. Identify setting pictures in the request; bind colors only to character references."),
|
||||
fld("Trigger", select(A.trigger ?? "motion_transfer", [["motion_transfer", "motion_transfer:"], ["body_swap", "body_swap:"], ["", "None (plain prompt)"]],
|
||||
v => c.setAuto("trigger", v), { style: "width:auto" }), "Match the LoRA caption prefix. Choose None for a model trained without a trigger."),
|
||||
fld("Key frames", select(String(A.frames || 0), [["0", "auto (by length)"], ...[3, 4, 5, 6, 8, 10, 12].map(n => [String(n), String(n)])],
|
||||
v => c.setAuto("frames", parseInt(v) || 0), { style: "width:auto" }), "auto: about 1.5 per second, 3 to 8"),
|
||||
fld("Source video in the prompt", select(A.source || "guide", [["guide", "guide video (aligned guide)"], ["video", "<Video 1> (native reference video)"]],
|
||||
v => c.setAuto("source", v), { style: "width:auto" }), "match the conditioning node's guide_mode"),
|
||||
h("button", { class: "pri", disabled: !!busy.value || !S.length, onClick: () => c.writePrompts() }, "🤖 Write all prompts"),
|
||||
h("button", { disabled: !!busy.value || !missing, onClick: () => c.writePrompts(S.map((s, i) => (c.writtenOf(s)?.text ? -1 : i)).filter(i => i >= 0)) },
|
||||
`only the missing (${missing})`),
|
||||
]),
|
||||
A.enabled && A.template ? h("div", { class: "hint", style: "display:block;margin-top:4px", title: A.template },
|
||||
`Template (the node's prompt input, last run): ${A.template.slice(0, 110)}${A.template.length > 110 ? "…" : ""} · the prompt boxes are the request applied to it`) : null,
|
||||
hint(A.enabled ? `${S.length - missing}/${S.length} shots have a written prompt${stale ? ` · ${stale} older than their request, pictures or marks` : ""} · shots without one are written when the workflow runs (with the VLM on the planner) · edit them in each shot (Shots tab)`
|
||||
: "Off: the prompt boxes are the prompts.", "display:block;margin-top:4px"),
|
||||
], { sub: "request + video + pictures -> MiniMax H3 prompt", right: A.enabled ? pill("on", "ok") : pill("off") });
|
||||
}
|
||||
|
||||
export function promptsTab(c) {
|
||||
return h("div", { class: "card" }, [globalSection(c), ...vlmSection(c)]);
|
||||
return h("div", { class: "card" }, [globalSection(c), autoSection(c), ...vlmSection(c)]);
|
||||
}
|
||||
|
||||
@@ -30,13 +30,17 @@ function loopSection(c) {
|
||||
["free", "Free memory: unload models (low VRAM)"]], v => c.setPlan("between_shots", v))) : null,
|
||||
fld("Skipped shots in the output", select(plan.skip_fill, [["original", "Keep original video"], ["drop", "Remove them"]], v => c.setPlan("skip_fill", v))),
|
||||
]),
|
||||
h("div", { class: "row", style: "margin-top:6px" }, [
|
||||
check(plan.unload_before_join !== false, v => c.setPlan("unload_before_join", v), "Unload models before the join",
|
||||
"Once every shot is rendered, unload the models so the join (and the nodes after it) get the VRAM and RAM. Turn it off if a model runs again after BFS Shot Join (e.g. an upscale pass on the joined video): it would load again."),
|
||||
]),
|
||||
plan.run !== "auto" ? h("div", { style: "margin-top:8px" }, [
|
||||
h("div", { class: "row", style: "margin-bottom:4px" }, [h("b", plan.run === "queue" ? "Queue loop" : "Shot by shot"), h("span", { class: "grow" }),
|
||||
hint(`${prog.done}/${prog.count || active.value.length} shots done`)]),
|
||||
h("div", { class: "prog" }, [h("div", { style: `width:${(100 * prog.done / Math.max(1, prog.count || active.value.length)).toFixed(1)}%` })]),
|
||||
h("div", { class: "row", style: "margin-top:6px" }, [
|
||||
plan.run === "queue" ? check(plan.auto_continue, v => c.setPlan("auto_continue", v), "Auto-queue the next shot") : null,
|
||||
h("button", { onClick: () => c.progress(false) }, "↻ Status"), h("button", { class: "dng", onClick: () => c.progress(true) }, "⟲ Reset loop"),
|
||||
h("button", { onClick: () => c.progress(false) }, "↻ Refresh status"), h("button", { title: "Cancel pending redo selections. Stored shots and loop progress are preserved.", onClick: () => c.progress(true) }, "Clear redo selection"),
|
||||
]),
|
||||
hint(plan.run === "queue"
|
||||
? "Each run generates one shot and stores it. Nodes after BFS Shot Join only run on the last shot, with the full video."
|
||||
@@ -87,7 +91,8 @@ function cacheSection(c) {
|
||||
const i = c.segs.value.indexOf(active.value[k]);
|
||||
return h("span", { class: ["pill", g ? "ok" : ""], title: g ? "stored and up to date" : "runs on the next Queue" }, [
|
||||
`#${(i >= 0 ? i : k) + 1} ${g ? "💾" : "⏳"}`,
|
||||
g ? h("button", { class: "chip", style: "margin-left:4px", title: "generate this shot again on the next Queue"
|
||||
(cache.redo || []).includes(k) ? h("button", { class: "chip", title: "Cancel redo and keep the stored take",
|
||||
onClick: () => c.redo([k], true, true) }, "Undo redo") : g ? h("button", { class: "chip", style: "margin-left:4px", title: "generate this shot again on the next Queue"
|
||||
+ (plan.redo_only ? " (only this shot)" : " (and the shots that continue from it)"),
|
||||
onClick: () => c.redo([k], !plan.redo_only) }, "↻") : null]);
|
||||
}) : [hint(n ? "Nothing stored yet for this video: Queue the workflow." : "")]),
|
||||
@@ -104,7 +109,7 @@ function cacheSection(c) {
|
||||
hint(`✓ ${cache.combined.shots}/${cache.combined.of} shots → `),
|
||||
h("a", { href: cache.combined.url, target: "_blank", style: "margin-left:4px" }, cache.combined.filename)]) : null,
|
||||
hint("Each shot remembers the settings it was made with: the next Queue runs only the shots whose settings changed "
|
||||
+ "(and the ones that continue from them). Model, LoRA, seed or sampler changes are not seen by the planner: use ↻ or Redo all. "
|
||||
+ "(and the ones that continue from them). Changing what renders them (model, LoRA, seed, sampler, conditioning) runs every shot again; the join's own settings do not. "
|
||||
+ (cache.folder ? `Folder: ${cache.folder}` : ""), "display:block;margin-top:6px"),
|
||||
], { sub: ok.length ? `${done}/${ok.length} stored` : "nothing stored yet" });
|
||||
}
|
||||
|
||||
@@ -1,11 +1,9 @@
|
||||
// 🎬 Shots tab: one card per shot, then the selected shot's editor (preview, reference & prompt, who is replaced,
|
||||
// continuity, copy to other shots).
|
||||
import { h, hue, fmtT, segKey, hasMask, maskSource, pill, hint, check, section } from "./common.js";
|
||||
import { h, hue, fmtT, segKey, hasMask, maskSource, pill, hint, check, section, refBox, promptChips, pictureColor, markName,
|
||||
MAX_REFS } from "./common.js";
|
||||
import { h as rawH } from "../vendor/vue.esm-browser.prod.mjs";
|
||||
|
||||
const GUIDE_COLORS = [["#00ff00", "Green"], ["#9b30ff", "Purple"], ["#ff2020", "Red"], ["#168cff", "Blue"],
|
||||
["#ffe600", "Yellow"], ["#ff00d4", "Magenta"], ["#00e5c0", "Cyan"]];
|
||||
|
||||
const bdg = (text, kind = "", title = "") => h("span", { class: ["bdg", kind], title }, text);
|
||||
|
||||
function shotCard(c, s, i) {
|
||||
@@ -23,7 +21,8 @@ function shotCard(c, s, i) {
|
||||
track: bdg("🎯 track", "", "its part of the whole-video SAM 3 track"),
|
||||
sam: bdg("🎯 " + (s.mask.text ? s.mask.text.slice(0, 14) : `${(s.mask.points || []).length} pts`), "", s.mask.text || `${(s.mask.points || []).length} points`) }[maskSource(s)] || null,
|
||||
s.crop || s.inpaint || s.paste ? bdg({ paste: "🧩 frame+paste", mask: "🎭 mask only", crop: "✂ crop", cropmask: "✂🎭 crop+mask" }[c.modeOf(s)] + (s.inpaint && s.strength < 1 ? ` ${Number(s.strength).toFixed(2)}` : ""), "on", c.modeName(s)) : null,
|
||||
s.color_guide ? bdg("🎨 Colored Mark", "on", `${s.color_guide_color || "green"} · opacity ${s.color_guide_opacity ?? 0.35}`) : null,
|
||||
s.color_guide ? bdg(`🎨 ${1 + s.regions.length}`, "on", [s.color_guide_picture, ...s.regions.map(r => r.picture)]
|
||||
.map(k => `Picture ${k} · ${markName(pictureColor(c.plan, k))}`).join("\n") + ` · opacity ${s.color_guide_opacity ?? 0.35}`) : null,
|
||||
s.target ? bdg("🧑 target", "on", s.target) : null,
|
||||
i > 0 && s.chain !== "off" ? bdg("⛓", "on", s.chain === "history" ? `continues the previous shot's last ${s.chainHistory} frames (history)` : s.chain === "head" ? `starts with the previous shot's last ${Math.min(s.chainHistory, 39)} frames, kept as they are (head)` : `continues from the previous shot's ${s.chainFrame} frame as ${s.chain}`) : null,
|
||||
{ ok: bdg("💾", "on", "stored and up to date: the next Queue does not generate it again (Run tab → Stored shots to redo it)"),
|
||||
@@ -38,11 +37,10 @@ function shotCard(c, s, i) {
|
||||
why ? h("div", { class: "t", style: "color:#ffc46b" }, why) : null,
|
||||
h("div", { class: "t" }, `${fmtT(s.start, fps)} → ${fmtT(s.end, fps)} · ${s.len}f → ${s.gen}f`),
|
||||
h("div", { class: "thumbs" }, [
|
||||
...["ref", "ref2"].map(k => {
|
||||
const own = s[k], name = own || plan["global_" + k];
|
||||
return name ? h("img", { class: "rt", src: c.viewUrl(name), title: own ? name : `${name} (global)`,
|
||||
style: own ? "" : "opacity:.45" }) : h("div", { class: "rt ph2" }, k);
|
||||
}),
|
||||
...(s.names.length ? s.names.slice(0, 3).map((name, k) => h("img", { class: "rt", src: c.viewUrl(name),
|
||||
title: s.refs.includes(name) ? name : `${name} (global)`, style: s.refs.includes(name) ? "" : "opacity:.45" }))
|
||||
: [h("div", { class: "rt ph2" }, "ref")]),
|
||||
s.names.length > 3 ? h("div", { class: "rt ph2", title: s.names.slice(3).join("\n") }, `+${s.names.length - 3}`) : null,
|
||||
h("img", { class: "rt", style: "width:56px", src: showMasks.value && mp?.frames?.length
|
||||
? mp.frames[Math.floor(mp.frames.length / 2)].src : c.thumbFor(s.start + Math.floor(s.len / 2)) }),
|
||||
]),
|
||||
@@ -76,19 +74,9 @@ function player(c, cur) {
|
||||
]);
|
||||
}
|
||||
|
||||
function refSlot(c, name, label, target, field) {
|
||||
return h("div", { class: "rslot" }, [
|
||||
h("div", { class: "refslot", title: name ? name + " (click to replace)" : "click to upload",
|
||||
onClick: () => c.uploadRef(target, field) },
|
||||
name ? [h("img", { src: c.viewUrl(name) })] : [label]),
|
||||
h("div", { class: "rbtns" }, [name ? h("button", { title: "remove (use the global reference)", onClick: () => c.setRef(target, field, "") }, "✕") : null]),
|
||||
]);
|
||||
}
|
||||
|
||||
function editor(c, cur) {
|
||||
const { plan, sel, segs, maskPrev, targetCrop, vlmSug, busy, usedRefs, copyOpts, COPY_FIELDS } = c;
|
||||
const i = sel.value, S = segs.value, k = segKey(cur), mp = maskPrev.value[k], g = vlmSug.value[k];
|
||||
const hasRef2 = !!(cur.ref2 || plan.global_ref2);
|
||||
const issues = c.checksFor(i);
|
||||
|
||||
const head = h("div", { class: "edhd" }, [
|
||||
@@ -110,53 +98,77 @@ function editor(c, cur) {
|
||||
h("div", { class: "ck" }, [h("span", { class: "lvl" }, x.lvl === "warn" ? "⚠" : "ℹ"), h("span", x.text)]))) : null;
|
||||
|
||||
const refPrompt = section("Reference & prompt", [
|
||||
h("div", { class: "row", style: "align-items:flex-start;gap:10px" }, [
|
||||
h("div", { class: "refbox" }, [
|
||||
refSlot(c, cur.ref, "⬆ reference", i, "ref"),
|
||||
refSlot(c, cur.ref2, "⬆ ref 2", i, "ref2"),
|
||||
]),
|
||||
h("div", { style: "flex:1;min-width:200px" }, [
|
||||
h("textarea", { placeholder: "Prompt for this shot (empty = global prompt). Placeholders: {target} {details} {shot} {setting}",
|
||||
value: cur.prompt, onChange: e => c.setMeta(i, "prompt", e.target.value) }),
|
||||
!cur.prompt && plan.global_prompt ? rawH("span", { class: "hint" }, `↳ uses the global prompt: ${plan.global_prompt.slice(0, 120)}${plan.global_prompt.length > 120 ? "…" : ""}`) : null,
|
||||
]),
|
||||
refBox(c, i, cur.refs, plan.global_refs),
|
||||
h("div", { class: "pbox", style: "margin-top:6px" }, [
|
||||
h("textarea", { placeholder: plan.auto_prompt?.enabled
|
||||
? "Request for this shot (empty = the global request); 🤖 Auto prompt writes the H3 prompt below from it"
|
||||
: "Prompt for this shot (empty = global prompt). Placeholders: {target} {details} {details@Picture n} {shot} {setting}",
|
||||
value: cur.prompt, onChange: e => c.setMeta(i, "prompt", e.target.value) }),
|
||||
promptChips(c, cur.names, v => c.setMeta(i, "prompt", v)),
|
||||
!cur.prompt && plan.global_prompt ? rawH("span", { class: "hint" }, `↳ uses the global prompt: ${plan.global_prompt.slice(0, 120)}${plan.global_prompt.length > 120 ? "…" : ""}`) : null,
|
||||
]),
|
||||
usedRefs.value.length ? h("div", { class: "recent" }, [hint("recent (click = ref, shift+click = ref 2):"),
|
||||
...usedRefs.value.map(n => h("img", { src: c.viewUrl(n), title: n, onClick: e => c.setRef(i, e.shiftKey ? "ref2" : "ref", n) })),
|
||||
plan.auto_prompt?.enabled ? (() => {
|
||||
const w = c.writtenOf(cur), old = w?.text && w.for !== c.autoFor(cur);
|
||||
return h("div", { class: "note", style: "margin-top:6px;padding:6px 8px" }, [
|
||||
h("div", { class: "row", style: "gap:6px;align-items:center;margin-bottom:4px" }, [
|
||||
h("b", "🤖 Written prompt"), old ? pill("older than the request / pictures / marks", "warn") : null, h("span", { class: "grow" }),
|
||||
h("button", { class: w?.text ? "" : "pri", disabled: !!busy.value, onClick: () => c.writePrompts([i]) }, w?.text ? "🤖 Write again" : "🤖 Write"),
|
||||
w?.text ? h("button", { class: "dng", title: "forget it: the next run writes a new one", onClick: () => c.setWritten(cur, "") }, "✕") : null,
|
||||
]),
|
||||
h("textarea", { style: "min-height:140px;font-size:11px", value: w?.text || "",
|
||||
placeholder: "not written yet: 🤖 Write, or the planner's VLM writes it when the workflow runs",
|
||||
onChange: e => c.setWritten(cur, e.target.value) }),
|
||||
]);
|
||||
})() : null,
|
||||
usedRefs.value.length ? h("div", { class: "recent" }, [hint("recent (click = picture 1, shift+click = add, or drag onto a slot):"),
|
||||
...usedRefs.value.map(n => h("img", { src: c.viewUrl(n), title: n, draggable: true,
|
||||
onDragstart: e => e.dataTransfer.setData("text/bfs-ref", n),
|
||||
onClick: e => (e.shiftKey ? c.putRefs(i, cur.refs.length, [n]) : c.setRef(i, 0, n)) })),
|
||||
h("button", { class: "sm", title: "Forget the recent pictures (the ones this plan still uses stay)", onClick: () => c.clearRecent() }, "✕ clear"),
|
||||
h("button", { class: "sm dng", title: "Remove EVERY reference from this plan (global and all shots) and the recent list", onClick: () => c.clearAllRefs() }, "✕ clear all")]) : null,
|
||||
h("div", { class: "row", style: "margin-top:6px" }, [
|
||||
h("button", { class: "dng", title: "clear this shot's references and prompt: it uses the global ones",
|
||||
onClick: () => { ["ref", "ref2", "prompt"].forEach(f => { c.meta(i)[f] = ""; }); c.save(); } }, "Use global"),
|
||||
onClick: () => { Object.assign(c.meta(i), { refs: [], prompt: "" }); c.save(); } }, "Use global"),
|
||||
hint("empty slots use the global references (Prompts & refs tab)"),
|
||||
]),
|
||||
], { sub: "who the shot becomes" });
|
||||
|
||||
const mode = (on, title, text, onClick) => h("div", { class: ["mode", on && "on"], onClick }, [h("b", title), h("span", text)]);
|
||||
// which picture a marked person becomes: "Picture 3 · red" with its thumbnail (the color is the picture's)
|
||||
const picturePick = (k, onPick) => {
|
||||
const n = Math.max(cur.names.length, k), color = pictureColor(plan, k);
|
||||
return h("span", { class: "row", style: "gap:4px;align-items:center" }, [
|
||||
cur.names[k - 1] ? h("img", { class: "mthumb", src: c.viewUrl(cur.names[k - 1]), style: `--mark:${color}`, title: cur.names[k - 1] })
|
||||
: h("span", { class: "mthumb", style: `--mark:${color};display:inline-block;background:${color}` }),
|
||||
h("select", { value: String(k), style: "width:auto", title: "The picture this person becomes; the mark is drawn in its color",
|
||||
onChange: e => onPick(+e.target.value) },
|
||||
Array.from({ length: Math.min(MAX_REFS, n) }, (_, q) => h("option", { value: String(q + 1) },
|
||||
`Picture ${q + 1} · ${markName(pictureColor(plan, q + 1))}${cur.names[q] ? "" : " (no picture)"}`))),
|
||||
]);
|
||||
};
|
||||
const who = section("Select region", [
|
||||
h("div", { class: "row", style: "margin-top:4px;align-items:center;gap:8px" }, [
|
||||
check(!!cur.color_guide, v => c.setMeta(i, "color_guide", v), "🎨 Colored Mark",
|
||||
"Color the selected region in the source video as a translucent visual marker. It does not create or enable an inpainting mask."),
|
||||
h("label", { class: "row", style: "gap:6px;align-items:center" }, [h("span", { class: "hint" }, "Region color · Ref 1"),
|
||||
h("select", { value: cur.color_guide_color || plan.global_color_guide_ref1 || "#00ff00", style: "width:auto", onChange: e => {
|
||||
c.setMeta(i, "color_guide_color", e.target.value);
|
||||
if (maskPrev.value[segKey(cur)]) c.previewMask(i);
|
||||
} },
|
||||
GUIDE_COLORS.map(([v, label]) => h("option", { value: v, disabled: v === (cur.color_guide_color2 || plan.global_color_guide_ref2 || "#9b30ff") }, label)))]),
|
||||
"Color each selected person in the source video with the color of the picture they become, as a translucent marker; the prompt gets 'the person marked in <color> becomes the character from <Picture n>'. It does not create or enable an inpainting mask."),
|
||||
cur.color_guide ? h("span", { class: "hint" }, "this selection becomes") : null,
|
||||
cur.color_guide ? picturePick(cur.color_guide_picture, k => { c.setMarkPicture(i, 1, k); if (maskPrev.value[segKey(cur)]) c.previewMask(i); }) : null,
|
||||
]),
|
||||
cur.color_guide ? h("details", { class: "note", style: "margin:4px 0;padding:6px 8px", open: !!cur.color_guide_text2 || hasRef2 }, [
|
||||
h("summary", { style: "cursor:pointer" }, cur.color_guide_text2 || (cur.color_guide_points2 || []).length ? "Mark a second region · Ref 2" : "+ Mark a second region (optional)"),
|
||||
h("div", { class: "row", style: "margin-top:8px;align-items:center;gap:8px" }, [
|
||||
h("span", { class: "hint", style: "white-space:nowrap" }, "Describe second region"),
|
||||
h("input", { type: "text", value: cur.color_guide_text2 || "", style: "flex:1;min-width:160px", placeholder: "e.g. the person on the left", onChange: e => c.setMeta(i, "color_guide_text2", e.target.value) }),
|
||||
h("button", { disabled: !!busy.value || !cur.color_guide_text2, onClick: () => c.previewMask(i, { text: cur.color_guide_text2, max_objects: 1 }) }, "👁 Preview"),
|
||||
h("button", { onClick: () => c.openPoints(i, 2) }, (cur.color_guide_points2 || []).length ? `🎯 Points 2 (${cur.color_guide_points2.length})` : "🎯 Mark with points"),
|
||||
h("select", { value: cur.color_guide_color2 || plan.global_color_guide_ref2 || "#9b30ff", onChange: e => {
|
||||
c.setMeta(i, "color_guide_color2", e.target.value);
|
||||
if ((cur.color_guide_points2 || []).length) c.previewMask(i, { points: cur.color_guide_points2, key: cur.color_guide_key2 || 0,
|
||||
text: cur.color_guide_text2 || "", preview_color: e.target.value });
|
||||
} },
|
||||
GUIDE_COLORS.map(([v, label]) => h("option", { value: v, disabled: v === (cur.color_guide_color || plan.global_color_guide_ref1 || "#00ff00") }, label))),
|
||||
cur.color_guide ? h("div", { class: "note", style: "margin:4px 0;padding:6px 8px" }, [
|
||||
...cur.regions.map((r, j) => h("div", { class: "mreg" }, [
|
||||
h("span", { class: "hint", style: "white-space:nowrap" }, `Person ${j + 2}`),
|
||||
picturePick(r.picture, k => c.setMarkRegion(i, j, "picture", k)),
|
||||
h("input", { type: "text", value: r.text || "", style: "flex:1;min-width:160px", placeholder: "who, in English: e.g. the man on the left",
|
||||
onChange: e => c.setMarkRegion(i, j, "text", e.target.value) }),
|
||||
h("button", { disabled: !!busy.value || !(r.text || (r.points || []).length),
|
||||
onClick: () => c.previewMask(i, { text: r.text || "", points: r.points || [], key: r.key || 0, max_objects: 1,
|
||||
preview_color: pictureColor(plan, r.picture) }) }, "👁 Preview"),
|
||||
h("button", { onClick: () => c.openPoints(i, j + 2) }, (r.points || []).length ? `🎯 Points (${r.points.length})` : "🎯 Points…"),
|
||||
h("button", { class: "dng", title: "remove this person's mark", onClick: () => c.removeMarkRegion(i)(j) }, "✕"),
|
||||
])),
|
||||
h("div", { class: "row", style: "margin-top:6px;align-items:center;gap:8px" }, [
|
||||
h("button", { disabled: 1 + cur.regions.length >= MAX_REFS, onClick: () => c.addMarkRegion(i) }, "+ Mark another person"),
|
||||
hint(cur.names.length > 1 + cur.regions.length ? "each person: a picture (its color) and a text or points"
|
||||
: "add more pictures (Reference & prompt) to give each person their own"),
|
||||
]),
|
||||
]) : null,
|
||||
cur.color_guide ? h("details", { style: "margin:4px 0" }, [h("summary", { class: "hint", style: "cursor:pointer" }, `Opacity · ${Number(cur.color_guide_opacity ?? 0.35).toFixed(2)}`),
|
||||
@@ -265,7 +277,7 @@ function editor(c, cur) {
|
||||
const copy = section("Copy to other shots", [
|
||||
h("div", { class: "copy" }, Object.entries(COPY_FIELDS).map(([f, label]) =>
|
||||
check(copyOpts[f], v => { copyOpts[f] = v; }, label, {
|
||||
ref: "reference + ref 2", prompt: "this shot's prompt", mask_text: "the mask text (shots keep their own points)",
|
||||
ref: "every reference picture", prompt: "this shot's prompt", mask_text: "the mask text (shots keep their own points)",
|
||||
points: "the points, on each shot's frame at the same relative position (a subject that stays in place)",
|
||||
mask_video: "the shot's mask video (it covers only its own shot: usually you want a different one per shot)",
|
||||
crop: "the generation mode (Full frame / Frame + paste / Mask only / Crop / Crop + mask) and its mask strength (shots with a mask)", target: "the {target} description",
|
||||
|
||||