feat: improve SCAIL-2 prefix conditioning

Changes since the previous commit:

- Add bg_image support for SCAIL-2 animation mode with an internal white mask and four-slot limits for user prefix frames/masks when bg_image is connected.

- Add single_frame_prefix_encoding so prefix frames can be encoded as full-resolution reference latents without expanding the output canvas, while keeping the legacy 37-frame canvas prefix path available.

- Add preserve_main_ref_background to optionally crop the main reference image background in animation mode using reference_image_mask.

- Route single-frame prefix masks through the reference-mask latent path, keep legacy prefix masks on the canvas/SAM mask path, and avoid duplicate prefix alpha cropping.

- Preserve SCAIL-2 prefix/transition context across context-window sampling and remove prepended predictions before overlap fusion.

- Add a local SCAIL-2 fast path in the sampler/model, including multi-reference prefix handling, RoPE/cache key updates, block-swap support, clip/text/time embedding handling, and dtype guards.

- Update README documentation for the new SCAIL-2 inputs, prefix modes, bg_image behavior, and transition trimming rules.
This commit is contained in:
wuwukasi
2026-06-13 00:35:43 +08:00
parent c312e1a519
commit 76173c0e66
4 changed files with 671 additions and 53 deletions
+12 -7
View File
@@ -72,12 +72,12 @@ Generates condition latents from source video, reference video, and/or reference
Provides SCAIL-2 ref / pose / mask conditioning through `WanAnimatePlus SCAIL_2 Embeds`.
- Encodes `ref_image`, `pose_images`, `pose_image_mask`, `prefix_mask`, and `reference_image_mask`
- Encodes `ref_image`, `pose_images`, `pose_image_mask`, `prefix_frames`, `prefix_mask`, `bg_image`, and `reference_image_mask`
- Aligns SCAIL-2 inputs to 32-pixel multiples before VAE encoding
- Supports animation and replacement modes
- Supports optional `prefix_frames` and `transition_video` hard-freeze conditioning
- Uses a single SCAIL-2 prefix layout: 37 front pixel frames when `prefix_frames` is connected, with transition frames placed at frames 17-36 when `transition_video` is also connected; transition-only uses 21 front pixel frames
- Writes `prefix_mask` into the prefix pixel-mask frames before SCAIL-2 mask latent encoding
- Supports single-frame prefix reference encoding and optional `transition_video` hard-freeze conditioning
- By default, `prefix_frames` are encoded as full-resolution reference latents and do not expand the output canvas; disable `single_frame_prefix_encoding` to use the legacy 37 front pixel-frame prefix layout
- In single-frame prefix mode, `prefix_mask` follows the same reference-mask path as `reference_image_mask`
- Supports context-window sampling; non-first windows can see prepended prefix/transition context without fusing those prepended predictions
## Installation
@@ -167,19 +167,24 @@ Creates SCAIL-2 conditioning for WanAnimatePlus sampling. Use this node with SCA
| `vae` | VAE model for encoding |
| `width` / `height` / `num_frames` | Target dimensions; width and height are aligned to multiples of 32 |
| `ref_image` | Reference image for SCAIL-2 conditioning |
| `bg_image` | Optional single background image for animation mode; prepended as the first `prefix_frames` item with an internal white mask and ignored in replacement mode |
| `pose_images` | Driving pose video/images, encoded at half resolution |
| `pose_image_mask` | Colored per-identity pose mask sequence |
| `prefix_mask` | Optional colored mask images matching `prefix_frames`; expanded as `1+4+4...` and written into the prefix mask frames before mask latent encoding |
| `reference_image_mask` | Colored reference mask image |
| `replacement_mode` | Enables SCAIL-2 replacement-mode RoPE and reference-mask compositing |
| `prefix_frames` | Optional frames to hard-freeze at the front of the latent sequence |
| `preserve_main_ref_background` | Animation mode only; keeps the main reference image background when enabled, or uses `reference_image_mask` as a black-background alpha crop when disabled. Ignored in replacement mode |
| `single_frame_prefix_encoding` | Encodes `prefix_frames` as individual full-resolution reference latents instead of expanding the canvas; enabled by default |
| `prefix_frames` | Optional prefix images. In default single-frame mode these become reference-stream latents; with single-frame mode disabled they hard-freeze the front canvas |
| `transition_video` | Optional transition frames to hard-freeze at the front of the latent sequence |
| `clip_embeds` | Optional CLIP vision features from `WanAnimatePlus ClipVisionEncode` |
| `force_offload` / `tiled_vae` | Memory controls for VAE encoding |
For short generations, context windows are optional. For long generations or low VRAM, context windows are recommended. In context-window mode, the sampler prepends protected prefix/transition latents for model context and removes those prepended predictions before overlap fusion.
For short generations, context windows are optional. For long generations or low VRAM, context windows are recommended. In context-window mode, single-frame prefix references remain visible through the SCAIL-2 reference stream. Legacy canvas prefixes and transition latents are prepended for model context and removed before overlap fusion.
For SCAIL-2, `prefix_frames` always expands the front canvas by 37 pixel frames. If only `transition_video` is connected, the front canvas expands by 21 pixel frames. The separate 45-frame outfit layout belongs to the regular AnimateEmbeds node and is not used by `WanAnimatePlus SCAIL_2 Embeds`.
For SCAIL-2, the default `single_frame_prefix_encoding` mode does not expand or trim the output for `prefix_frames`. If `transition_video` is connected, the front canvas expands by 21 pixel frames and those 21 frames are trimmed after decoding. With `single_frame_prefix_encoding` disabled, `prefix_frames` use the legacy 37 front pixel-frame canvas layout, with transition frames placed at frames 17-36 when `transition_video` is also connected.
When `bg_image` is connected in animation mode, it consumes the first prefix slot and is handled as the first prefix image. The node internally adds a white mask for that background image only; user-provided `prefix_frames` and `prefix_mask` are then limited to four images each. In replacement mode, `bg_image` is ignored.
## Project Structure
+193 -40
View File
@@ -1653,10 +1653,11 @@ class WanAnimatePlusSCAIL2Embeds:
"optional": {
"clip_embeds": ("WANVIDIMAGE_CLIPEMBEDS", {"tooltip": "Clip vision encoded image"}),
"ref_image": ("IMAGE", {"tooltip": "Reference image for SCAIL conditioning. If a sequence is connected, only the first frame is used."}),
"bg_image": ("IMAGE", {"tooltip": "Optional single background image for animation mode. Prepended before prefix_frames with an internal white mask and ignored in replacement mode."}),
"pose_images": ("IMAGE", {"tooltip": "Driving pose video. Encoded at half resolution for SCAIL."}),
"prefix_frames": ("IMAGE", {"tooltip": "Optional frames to hard-freeze at the beginning of the SCAIL-2 latent sequence."}),
"prefix_mask": ("IMAGE", {"tooltip": "Optional colored mask images matching prefix_frames. Expanded as 1+4+4... and written into the prefix mask frames before mask latent encoding. In replacement mode it also alpha-crops matching prefix frames to black background."}),
"transition_video": ("IMAGE", {"tooltip": "Optional transition frames to hard-freeze after prefix_frames at the beginning of the SCAIL-2 latent sequence."}),
"prefix_frames": ("IMAGE", {"tooltip": "Optional prefix images. In single-frame prefix mode these are encoded as reference latents; in legacy mode they hard-freeze the beginning of the canvas."}),
"prefix_mask": ("IMAGE", {"tooltip": "Optional colored mask images matching prefix_frames. In single-frame prefix mode this follows the reference-mask path; in legacy canvas-prefix mode it is expanded as 1+4+4... and written into the prefix mask frames."}),
"transition_video": ("IMAGE", {"tooltip": "Optional transition frames to hard-freeze at the beginning of the canvas. In legacy canvas-prefix mode, transition frames are placed after the prefix frames."}),
"pose_image_mask": ("IMAGE", {"tooltip": "SCAIL-2 colored per-identity driving pose mask. Background is normalized to black in animation mode and white in replacement mode."}),
"reference_image_mask": ("IMAGE", {"tooltip": "SCAIL-2 colored per-identity reference mask image. Background is normalized to white in animation mode and black in replacement mode."}),
"tiled_vae": ("BOOLEAN", {"default": False, "tooltip": "Use tiled VAE encoding for reduced memory use"}),
@@ -1669,7 +1670,9 @@ class WanAnimatePlusSCAIL2Embeds:
'hm-mvgd-hm',
'hm-mkl-hm',
], {"default": 'disabled', "tooltip": "Color match transition_video to ref_image."}),
"prefix_alpha_crop": ("BOOLEAN", {"default": False, "tooltip": "Off writes prefix_mask with white background and keeps prefix frames unchanged. On writes prefix_mask with black background and alpha-crops prefix_frames."}),
"prefix_alpha_crop": ("BOOLEAN", {"default": False, "tooltip": "Off keeps prefix masks as white-background reference masks in animation mode. On uses black-background masks and alpha-crops prefix_frames. Replacement mode always uses black-background reference masks."}),
"preserve_main_ref_background": ("BOOLEAN", {"default": True, "tooltip": "Animation mode only. Keep the main reference image background. When off, reference_image_mask is normalized to black background and used to alpha-crop ref_image. Ignored in replacement mode."}),
"single_frame_prefix_encoding": ("BOOLEAN", {"default": True, "tooltip": "Encode prefix images as individual full-resolution reference latents instead of expanding the canvas."}),
"by wuwukasi(bilibili)": ("BOOLEAN", {"default": True, "label_on": "ON", "label_off": "ON", "tooltip": "Follow wuwukasi on bilibili"}),
}
}
@@ -1775,6 +1778,20 @@ class WanAnimatePlusSCAIL2Embeds:
bg = mask.amin(dim=-1, keepdim=True) >= 0.95
return torch.where(bg, torch.zeros_like(mask), mask)
@staticmethod
def _alpha_crop_with_mask(images, masks):
if images is None or masks is None or images.shape[0] == 0 or masks.shape[0] == 0:
return images
count = min(images.shape[0], masks.shape[0])
crop_mask = masks[:count, :, :, :3]
if crop_mask.shape[1] != images.shape[1] or crop_mask.shape[2] != images.shape[2]:
crop_mask = WanAnimatePlusSCAIL2Embeds._resize_bhwc(crop_mask, images.shape[2], images.shape[1], mode="nearest-exact")
crop_mask = WanAnimatePlusSCAIL2Embeds._normalize_mask_background(crop_mask, white_background=False)
is_char = (crop_mask[..., :3].max(dim=-1, keepdim=True).values > 0.1).to(device=images.device, dtype=images.dtype)
out = images.clone()
out[:count] = out[:count] * is_char
return out
@staticmethod
def _color_match_frames(frames, ref_frame, method):
if method == "disabled" or ref_frame is None or frames is None or frames.shape[0] == 0:
@@ -1819,23 +1836,121 @@ class WanAnimatePlusSCAIL2Embeds:
padded = padded[:t_lat * 4]
return padded.view(t_lat, 28, h_lat, w_lat).movedim(0, 1).contiguous()
@staticmethod
def _empty_ref_mask_like(ref_latent):
return torch.zeros(
28, ref_latent.shape[1], ref_latent.shape[-2], ref_latent.shape[-1],
device=ref_latent.device, dtype=ref_latent.dtype,
)
def process(self, vae, width, height, num_frames, force_offload, pose_strength, ref_strength,
pose_start_percent, pose_end_percent, ref_start_percent, ref_end_percent, replacement_mode,
clip_embeds=None, ref_image=None, pose_images=None, prefix_frames=None, prefix_mask=None,
clip_embeds=None, ref_image=None, bg_image=None, pose_images=None, prefix_frames=None, prefix_mask=None,
transition_video=None, pose_image_mask=None, reference_image_mask=None, tiled_vae=False,
transition_colormatch='disabled', prefix_alpha_crop=False, **kwargs):
transition_colormatch='disabled', prefix_alpha_crop=False, preserve_main_ref_background=True,
single_frame_prefix_encoding=True, **kwargs):
W = (width // 32) * 32
H = (height // 32) * 32
num_frames = ((num_frames - 1) // 4) * 4 + 1
bg_prefix_mask_pixel_frames = 0
bg_prefix_mask_index = None
crop_main_ref_background = (not replacement_mode) and (not preserve_main_ref_background)
user_prefix_frames = prefix_frames
user_prefix_mask = prefix_mask if user_prefix_frames is not None else None
bg_prefix_active = bg_image is not None and not replacement_mode
if bg_image is not None and replacement_mode:
log.info("SCAIL-2 bg_image is ignored in replacement mode")
bg_prefix_image = None
bg_mask = None
user_prefix_count = 0
if bg_prefix_active:
if bg_image.shape[0] > 1:
log.warning("SCAIL-2 bg_image accepts one image; using the first frame")
bg_prefix_image = bg_image[:1, :, :, :3]
if user_prefix_frames is not None:
if user_prefix_frames.shape[0] > 4:
log.warning("SCAIL-2 bg_image uses one prefix slot; truncating prefix_frames to 4 images")
user_prefix_frames = user_prefix_frames[:4, :, :, :3]
user_prefix_count = user_prefix_frames.shape[0]
if bg_prefix_image.shape[1] != user_prefix_frames.shape[1] or bg_prefix_image.shape[2] != user_prefix_frames.shape[2]:
bg_prefix_image = self._resize_bhwc(bg_prefix_image, user_prefix_frames.shape[2], user_prefix_frames.shape[1])
bg_prefix_image = bg_prefix_image.to(device=user_prefix_frames.device, dtype=user_prefix_frames.dtype)
if user_prefix_mask is not None:
if user_prefix_mask.shape[0] > 4:
log.warning("SCAIL-2 bg_image uses one prefix-mask slot; truncating prefix_mask to 4 images")
user_prefix_mask = user_prefix_mask[:4, :, :, :3]
bg_mask = torch.ones(
1, user_prefix_mask.shape[1], user_prefix_mask.shape[2], 3,
device=user_prefix_mask.device, dtype=user_prefix_mask.dtype,
)
else:
bg_mask = torch.ones_like(bg_prefix_image[:, :, :, :3])
if user_prefix_frames is not None and user_prefix_mask is not None and (replacement_mode or prefix_alpha_crop):
crop_count = min(user_prefix_frames.shape[0], user_prefix_mask.shape[0], 5)
if crop_count > 0:
cropped_prefix_frames = user_prefix_frames.clone()
cropped_prefix_frames[:crop_count] = self._alpha_crop_with_mask(
cropped_prefix_frames[:crop_count],
user_prefix_mask[:crop_count],
)
user_prefix_frames = cropped_prefix_frames
prefix_frames = user_prefix_frames
prefix_mask_for_prefix = user_prefix_mask
if bg_prefix_active:
if single_frame_prefix_encoding:
if user_prefix_frames is not None:
prefix_frames = torch.cat([user_prefix_frames, bg_prefix_image], dim=0)
bg_prefix_mask_index = prefix_frames.shape[0] - 1
else:
prefix_frames = bg_prefix_image
bg_prefix_mask_index = 0
if user_prefix_mask is not None:
if user_prefix_mask.shape[0] < user_prefix_count:
empty_user_masks = torch.zeros(
user_prefix_count - user_prefix_mask.shape[0],
user_prefix_mask.shape[1], user_prefix_mask.shape[2], 3,
device=user_prefix_mask.device, dtype=user_prefix_mask.dtype,
)
user_prefix_mask = torch.cat([user_prefix_mask, empty_user_masks], dim=0)
elif user_prefix_mask.shape[0] > user_prefix_count:
user_prefix_mask = user_prefix_mask[:user_prefix_count]
prefix_mask_for_prefix = torch.cat([user_prefix_mask, bg_mask], dim=0)
elif user_prefix_count > 0:
empty_user_masks = torch.zeros(
user_prefix_count, bg_mask.shape[1], bg_mask.shape[2], 3,
device=bg_mask.device, dtype=bg_mask.dtype,
)
prefix_mask_for_prefix = torch.cat([empty_user_masks, bg_mask], dim=0)
else:
prefix_mask_for_prefix = bg_mask
else:
if user_prefix_frames is not None:
prefix_frames = torch.cat([bg_prefix_image, user_prefix_frames], dim=0)
else:
prefix_frames = bg_prefix_image
if user_prefix_mask is not None:
prefix_mask_for_prefix = torch.cat([bg_mask, user_prefix_mask], dim=0)
else:
prefix_mask_for_prefix = bg_mask
bg_prefix_mask_pixel_frames = 1
bg_prefix_mask_index = 0
canvas_expansion_px = 0
freeze_canvas = None
if prefix_frames is not None:
canvas_prefix_frames = prefix_frames if not single_frame_prefix_encoding else None
if canvas_prefix_frames is not None:
canvas_expansion_px = 37
elif transition_video is not None:
canvas_expansion_px = 21
transition_px_range = None
if transition_video is not None:
transition_px_range = (17, 37) if prefix_frames is not None else (0, 21)
transition_px_range = (17, 37) if canvas_prefix_frames is not None else (0, 21)
if canvas_expansion_px:
num_frames += canvas_expansion_px
@@ -1847,16 +1962,16 @@ class WanAnimatePlusSCAIL2Embeds:
pose_image_mask = torch.cat([_sample_reversed_prefix_frames(pose_image_mask, canvas_expansion_px), pose_image_mask], dim=0)
prefix_mask_pixels = prefix_mask_frame_mask = scail_sam_keep_mask = scail_transition_keep_mask = None
if prefix_mask is not None:
if prefix_mask_for_prefix is not None:
if prefix_frames is None:
log.warning("SCAIL-2 prefix_mask was provided without prefix_frames. Ignoring prefix_mask.")
else:
prefix_count = min(prefix_frames.shape[0], 5)
if prefix_mask.shape[0] > prefix_count:
log.warning(f"SCAIL-2 prefix_mask has {prefix_mask.shape[0]} images, but prefix_frames has {prefix_count}. Truncating prefix_mask.")
mask_count = min(prefix_mask.shape[0], prefix_count)
if prefix_mask_for_prefix.shape[0] > prefix_count:
log.warning(f"SCAIL-2 prefix_mask has {prefix_mask_for_prefix.shape[0]} images, but prefix_frames has {prefix_count}. Truncating prefix_mask.")
mask_count = min(prefix_mask_for_prefix.shape[0], prefix_count)
if mask_count > 0:
prefix_mask_pixels = self._build_prefix_pixels(prefix_mask[:mask_count])
prefix_mask_pixels = self._build_prefix_pixels(prefix_mask_for_prefix[:mask_count])
lat_h = H // vae.upsampling_factor
lat_w = W // vae.upsampling_factor
@@ -1875,17 +1990,62 @@ class WanAnimatePlusSCAIL2Embeds:
"ref_mask_flag": not replacement_mode,
}
ref_latents = []
ref_masks = []
ref_mask_condition_used = False
prefix_ref_count = 0
prefix_ref_black_background = replacement_mode or prefix_alpha_crop
if single_frame_prefix_encoding and prefix_frames is not None:
prefix_ref_images = prefix_frames[:5, :, :, :3]
prefix_ref_source_indices = list(range(prefix_ref_images.shape[0]))
prefix_ref_count = prefix_ref_images.shape[0]
prefix_ref_masks = [None] * prefix_ref_count
if prefix_mask_for_prefix is not None:
for out_idx, src_idx in enumerate(prefix_ref_source_indices):
if src_idx >= prefix_mask_for_prefix.shape[0]:
continue
mask = prefix_mask_for_prefix[src_idx:src_idx + 1, :, :, :3]
if src_idx != bg_prefix_mask_index:
mask = self._normalize_mask_background(mask, white_background=not prefix_ref_black_background)
prefix_ref_masks[out_idx] = mask
ref_mask_condition_used = any(mask is not None for mask in prefix_ref_masks)
for i in range(prefix_ref_images.shape[0]):
prefix_latent = self._encode_scail_20ch(vae, prefix_ref_images[i:i + 1], W, H, tiled_vae, scale=ref_strength).to(offload_device)
ref_latents.append(prefix_latent)
if i < len(prefix_ref_masks) and prefix_ref_masks[i] is not None:
ref_mask = self._resize_bhwc(prefix_ref_masks[i], W, H, mode="bicubic")
ref_masks.append(self._extract_mask_to_28ch(ref_mask).to(offload_device, vae.dtype))
else:
ref_masks.append(self._empty_ref_mask_like(prefix_latent))
log.info(f"SCAIL-2 prefix reference latents: {prefix_ref_count}")
if ref_image is not None:
ref_image = ref_image[:1, :, :, :3]
if replacement_mode and reference_image_mask is not None:
if (replacement_mode or crop_main_ref_background) and reference_image_mask is not None:
ref_mask = self._resize_bhwc(reference_image_mask[:1, :, :, :3], W, H, mode="nearest-exact")
ref_mask = self._normalize_mask_background(ref_mask, white_background=False)
is_char = (ref_mask[..., :3].max(dim=-1, keepdim=True).values > 0.1).to(ref_image.dtype)
ref_image = self._resize_bhwc(ref_image, W, H) * is_char
ref_image = self._resize_bhwc(ref_image, W, H)
is_char = is_char.to(device=ref_image.device, dtype=ref_image.dtype)
ref_image = ref_image * is_char
ref_latent = self._encode_scail_20ch(vae, ref_image, W, H, tiled_vae, scale=ref_strength).to(offload_device)
ref_latents.append(ref_latent)
if reference_image_mask is not None:
ref_mask_condition_used = True
ref_mask = self._fit_sequence(reference_image_mask[:, :, :, :3], 1)
ref_mask = self._normalize_mask_background(ref_mask, white_background=not (replacement_mode or crop_main_ref_background))
ref_mask = self._resize_bhwc(ref_mask, W, H, mode="bicubic")
ref_masks.append(self._extract_mask_to_28ch(ref_mask).to(offload_device, vae.dtype))
else:
ref_masks.append(self._empty_ref_mask_like(ref_latent))
log.info(f"SCAIL-2 reference latent shape: {ref_latent.shape}")
if ref_latents:
ref_latent = torch.cat(ref_latents, dim=1)
scail_embeds["ref_latent_pos"] = ref_latent
scail_embeds["ref_latent_neg"] = ref_latent
log.info(f"SCAIL-2 reference latent shape: {ref_latent.shape}")
scail_embeds["ref_prefix_latents"] = prefix_ref_count
log.info(f"SCAIL-2 combined reference latent shape: {ref_latent.shape}")
scail_freeze_latents = scail_freeze_mask = scail_condition_zero_mask = None
actual_prefix_px = 0
@@ -1895,19 +2055,9 @@ class WanAnimatePlusSCAIL2Embeds:
transition_match_ref = self._resize_bhwc(ref_image[:1, :, :, :3], W, H) if ref_image is not None else None
if transition_colormatch != "disabled" and transition_match_ref is None:
log.warning("SCAIL-2 transition_colormatch is enabled but ref_image is not connected. Skipping color match.")
if prefix_frames is not None:
prefix_pixels = self._build_prefix_pixels(prefix_frames)
if canvas_prefix_frames is not None:
prefix_pixels = self._build_prefix_pixels(canvas_prefix_frames)
actual_prefix_px = min(prefix_pixels.shape[0], 17)
if prefix_alpha_crop and prefix_mask_pixels is not None:
crop_len = min(actual_prefix_px, prefix_mask_pixels.shape[0])
if crop_len > 0:
prefix_alpha = prefix_mask_pixels[:crop_len, :, :, :3]
if prefix_alpha.shape[1] != prefix_pixels.shape[1] or prefix_alpha.shape[2] != prefix_pixels.shape[2]:
prefix_alpha = self._resize_bhwc(prefix_alpha, prefix_pixels.shape[2], prefix_pixels.shape[1], mode="nearest-exact")
prefix_alpha = self._normalize_mask_background(prefix_alpha, white_background=False)
is_char = (prefix_alpha[..., :3].max(dim=-1, keepdim=True).values > 0.1).to(prefix_pixels.dtype)
prefix_pixels = prefix_pixels.clone()
prefix_pixels[:crop_len] = prefix_pixels[:crop_len] * is_char
prefix_pixels = self._resize_bhwc(prefix_pixels[:actual_prefix_px], W, H)
freeze_canvas[:actual_prefix_px] = prefix_pixels.to(device, dtype=freeze_canvas.dtype)
freeze_frame_mask[:actual_prefix_px] = 1.0
@@ -1930,7 +2080,7 @@ class WanAnimatePlusSCAIL2Embeds:
scail_freeze_mask = scail_freeze_mask.to(offload_device)
scail_condition_zero_mask = (scail_freeze_mask > 0).any(dim=(1, 2))
log.info(f"SCAIL-2 freeze latents shape: {scail_freeze_latents.shape}")
if prefix_frames is not None and actual_prefix_px > 0:
if canvas_prefix_frames is not None and actual_prefix_px > 0:
prefix_frame_mask = torch.zeros(canvas_expansion_px, device=device, dtype=vae.dtype)
prefix_frame_mask[:actual_prefix_px] = 1.0
scail_prefix_prepend_latents = int(self._frame_mask_to_latent_mask(prefix_frame_mask, target_latents).sum().item())
@@ -1967,7 +2117,7 @@ class WanAnimatePlusSCAIL2Embeds:
if pose_image_mask is not None:
pose_image_mask = self._fit_sequence(pose_image_mask[:, :, :, :3], num_frames)
pose_image_mask = self._normalize_mask_background(pose_image_mask, white_background=replacement_mode)
elif prefix_mask_pixels is not None:
elif prefix_mask_pixels is not None and not single_frame_prefix_encoding:
pose_bg = 1.0 if replacement_mode else 0.0
pose_image_mask = torch.full(
(
@@ -1976,12 +2126,18 @@ class WanAnimatePlusSCAIL2Embeds:
pose_bg, device=prefix_mask_pixels.device, dtype=prefix_mask_pixels.dtype,
)
if pose_image_mask is not None and prefix_mask_pixels is not None:
if pose_image_mask is not None and prefix_mask_pixels is not None and not single_frame_prefix_encoding:
prefix_mask_len = min(prefix_mask_pixels.shape[0], pose_image_mask.shape[0])
prefix_mask_in = prefix_mask_pixels[:prefix_mask_len, :, :, :3]
if prefix_mask_in.shape[1] != pose_image_mask.shape[1] or prefix_mask_in.shape[2] != pose_image_mask.shape[2]:
prefix_mask_in = self._resize_bhwc(prefix_mask_in, pose_image_mask.shape[2], pose_image_mask.shape[1], mode="nearest-exact")
prefix_mask_in = self._normalize_mask_background(prefix_mask_in, white_background=not prefix_alpha_crop)
bg_keep = min(bg_prefix_mask_pixel_frames, prefix_mask_in.shape[0])
if prefix_mask_in.shape[0] > bg_keep:
user_prefix_mask_in = self._normalize_mask_background(prefix_mask_in[bg_keep:], white_background=not prefix_alpha_crop)
if bg_keep > 0:
prefix_mask_in = torch.cat([prefix_mask_in[:bg_keep], user_prefix_mask_in], dim=0)
else:
prefix_mask_in = user_prefix_mask_in
pose_image_mask = pose_image_mask.clone()
pose_image_mask[:prefix_mask_len] = prefix_mask_in.to(device=pose_image_mask.device, dtype=pose_image_mask.dtype)
prefix_mask_frame_mask = torch.zeros(num_frames, device=pose_image_mask.device, dtype=pose_image_mask.dtype)
@@ -2010,16 +2166,13 @@ class WanAnimatePlusSCAIL2Embeds:
scail_embeds["sam_latents"] = sam_latents
log.info(f"SCAIL-2 driving mask latents shape: {sam_latents.shape}")
if reference_image_mask is not None:
ref_mask = self._fit_sequence(reference_image_mask[:, :, :, :3], 1)
ref_mask = self._normalize_mask_background(ref_mask, white_background=not replacement_mode)
ref_mask = self._resize_bhwc(ref_mask, W, H, mode="bicubic")
ref_mask_1f = self._extract_mask_to_28ch(ref_mask).to(offload_device, vae.dtype)
if ref_masks and ref_mask_condition_used:
ref_mask_prefix = torch.cat(ref_masks, dim=1)
zeros = torch.zeros(
28, target_latents, ref_mask_1f.shape[-2], ref_mask_1f.shape[-1],
device=offload_device, dtype=ref_mask_1f.dtype,
28, target_latents, ref_mask_prefix.shape[-2], ref_mask_prefix.shape[-1],
device=offload_device, dtype=ref_mask_prefix.dtype,
)
scail_embeds["ref_mask_latents"] = torch.cat([ref_mask_1f, zeros], dim=1)
scail_embeds["ref_mask_latents"] = torch.cat([ref_mask_prefix, zeros], dim=1)
log.info(f"SCAIL-2 reference mask latents shape: {scail_embeds['ref_mask_latents'].shape}")
if force_offload:
+196 -1
View File
@@ -1130,6 +1130,15 @@ class WanVideoSampler:
tsr_k = experimental_args.get("tsr_k", 1.0)
tsr_sigma = experimental_args.get("tsr_sigma", 1.0)
scail2_fast_path_blocked_by_experimental_sampling = (
use_cfg_zero_star
or use_tangential
or raag_alpha > 0.0
or use_fresca
or use_tsr
or bool(transformer.video_attention_split_steps)
)
# Rotary positional embeddings (RoPE)
# RoPE base freq scaling as used with CineScale
@@ -1145,6 +1154,10 @@ class WanVideoSampler:
if context_latents is not None and "comfy" not in rope_function:
log.info("Bernini context requires comfy RoPE; overriding rope_function to comfy")
rope_function = "comfy"
scail_embeds_for_rope = image_embeds.get("scail_embeds", None)
if scail_embeds_for_rope is not None and "comfy" not in rope_function:
log.info("SCAIL-2 requires comfy RoPE; overriding rope_function to comfy")
rope_function = "comfy"
freqs = None
@@ -1328,6 +1341,7 @@ class WanVideoSampler:
log.info(f" {k}: {v.shape if isinstance(v, torch.Tensor) else v}")
scail_data = scail_embeds.copy()
scail_data = dict_to_device(scail_data, device, dtype)
scail2_fast_path_fallback_logged = False
# WanMove
@@ -1369,6 +1383,7 @@ class WanVideoSampler:
context_latents=None, context_roles=None, context_window_start=0, scail_context_prepend_latents=0,):
nonlocal transformer
nonlocal audio_cfg_scale
nonlocal scail2_fast_path_fallback_logged
autocast_enabled = ("fp8" in model["quantization"] and not transformer.patched_linear)
with torch.autocast(device_type=mm.get_autocast_device(device), dtype=dtype) if autocast_enabled else nullcontext():
@@ -1813,7 +1828,187 @@ class WanVideoSampler:
and base_params['num_memory_frames'] == 0
)
strict_plain_t2v_fast_path = context_latents is None and context_window is None and fast_path_no_extra_conditions
base_params["simple_t2v"] = context_latents is not None or strict_plain_t2v_fast_path
base_params["simple_t2v"] = (context_latents is not None and base_params['scail_input'] is None) or strict_plain_t2v_fast_path
scail2_fast_path = (
base_params['scail_input'] is not None
and guidance_mode == "cfg"
and context_latents is None
and base_params['y'] is None
and image_cond_neg is None
and attn_cond is None
and attn_cond_neg is None
and not reverse_time
and not enhance_enabled
and cache_args is None
and freeinit_args is None
and not scail2_fast_path_blocked_by_experimental_sampling
and slg_args is None
and getattr(transformer, "slg_blocks", None) is None
and not getattr(transformer, "is_longcat", False)
and getattr(transformer, "audio_model", None) is None
and getattr(transformer, "vace_layers", None) is None
and not batched_cfg
and not bidirectional_sampling
and loop_args is None
and qwenvl_embeds_pos is None
and qwenvl_embeds_neg is None
and control_latents is None
and vace_data is None
and unianim_data is None
and audio_proj is None
and control_camera_latents is None
and add_cond is None
and multitalk_audio_embeds is None
and fantasy_portrait_input is None
and mtv_motion_tokens is None
and s2v_audio_input is None
and s2v_ref_motion is None
and s2v_pose is None
and humo_image_cond is None
and humo_audio is None
and wananim_pose_latents is None
and wananim_face_pixels is None
and uni3c_data is None
and latent_model_input_ovi is None
and flashvsr_LQ_latent is None
and base_params['fun_ref'] is None
and base_params['fun_camera'] is None
and base_params['audio_proj'] is None
and base_params['uni3c_data'] is None
and base_params['controlnet'] is None
and base_params['add_cond'] is None
and not base_params['nag_params']
and base_params['nag_context'] is None
and base_params['multitalk_audio'] is None
and base_params['ref_target_masks'] is None
and base_params['inner_t'] is None
and base_params['standin_input'] is None
and base_params['fantasy_portrait_input'] is None
and base_params['phantom_ref'] is None
and base_params['mtv_motion_tokens'] is None
and base_params['mtv_motion_rotary_emb'] is None
and base_params['s2v_audio_input'] is None
and base_params['s2v_ref_latent'] is None
and base_params['s2v_ref_motion'] is None
and base_params['s2v_pose'] is None
and base_params['humo_audio'] is None
and base_params['wananim_pose_latents'] is None
and base_params['wananim_face_pixel_values'] is None
and base_params['lynx_embeds'] is None
and base_params['x_ovi'] is None
and base_params['flashvsr_LQ_latent'] is None
and base_params['longcat_num_cond_latents'] == 0
and base_params['longcat_num_ref_latents'] == 0
and base_params['longcat_avatar_options'] is None
and base_params['sdancer_input'] is None
and base_params['one_to_all_input'] is None
and base_params['dual_control_input'] is None
and base_params['rope_negative_offset'] == 0
and base_params['num_memory_frames'] == 0
and "comfy" in getattr(transformer, "rope_func", "")
)
if base_params['scail_input'] is not None and not scail2_fast_path:
if not scail2_fast_path_fallback_logged:
scail2_fast_path_fallback_logged = True
fallback_reasons = []
if guidance_mode != "cfg":
fallback_reasons.append(f"guidance_mode={guidance_mode!r}")
if context_latents is not None:
fallback_reasons.append("Bernini context_latents connected")
if base_params['y'] is not None:
fallback_reasons.append("image condition active")
if image_cond_neg is not None:
fallback_reasons.append("negative image condition active")
if attn_cond is not None or attn_cond_neg is not None:
fallback_reasons.append("attention condition active")
if reverse_time:
fallback_reasons.append("reverse_time enabled")
if enhance_enabled:
fallback_reasons.append("enhance enabled")
if cache_args is not None:
fallback_reasons.append("cache enabled")
if freeinit_args is not None:
fallback_reasons.append("FreeInit enabled")
if scail2_fast_path_blocked_by_experimental_sampling:
fallback_reasons.append("experimental sampling enabled")
if slg_args is not None or getattr(transformer, "slg_blocks", None) is not None:
fallback_reasons.append("SLG enabled")
if getattr(transformer, "is_longcat", False):
fallback_reasons.append("LongCat model active")
if getattr(transformer, "audio_model", None) is not None:
fallback_reasons.append("OVI audio model active")
if getattr(transformer, "vace_layers", None) is not None:
fallback_reasons.append("VACE layers active")
if batched_cfg:
fallback_reasons.append("batched CFG enabled")
if bidirectional_sampling:
fallback_reasons.append("bidirectional sampling enabled")
if loop_args is not None:
fallback_reasons.append("loop enabled")
if qwenvl_embeds_pos is not None or qwenvl_embeds_neg is not None:
fallback_reasons.append("QwenVL embeddings connected")
if control_latents is not None or control_camera_latents is not None:
fallback_reasons.append("control latents connected")
if vace_data is not None:
fallback_reasons.append("VACE input connected")
if unianim_data is not None:
fallback_reasons.append("UniAnimate input connected")
if audio_proj is not None:
fallback_reasons.append("audio projection connected")
if add_cond is not None:
fallback_reasons.append("additional condition connected")
if multitalk_audio_embeds is not None:
fallback_reasons.append("MultiTalk input connected")
if fantasy_portrait_input is not None:
fallback_reasons.append("FantasyPortrait input connected")
if mtv_motion_tokens is not None:
fallback_reasons.append("MTV motion tokens connected")
if s2v_audio_input is not None or s2v_ref_motion is not None or s2v_pose is not None:
fallback_reasons.append("S2V input connected")
if humo_image_cond is not None or humo_audio is not None:
fallback_reasons.append("HuMo input connected")
if wananim_pose_latents is not None or wananim_face_pixels is not None:
fallback_reasons.append("WanAnimate input connected")
if uni3c_data is not None:
fallback_reasons.append("Uni3C input connected")
if latent_model_input_ovi is not None:
fallback_reasons.append("OVI latent input connected")
if flashvsr_LQ_latent is not None:
fallback_reasons.append("FlashVSR input connected")
if base_params['fun_ref'] is not None or base_params['fun_camera'] is not None:
fallback_reasons.append("Fun reference/camera active")
if base_params['controlnet'] is not None:
fallback_reasons.append("ControlNet active")
if base_params['nag_params'] or base_params['nag_context'] is not None:
fallback_reasons.append("NAG active")
if base_params['ref_target_masks'] is not None:
fallback_reasons.append("reference target masks active")
if base_params['inner_t'] is not None:
fallback_reasons.append("EchoShot active")
if base_params['standin_input'] is not None:
fallback_reasons.append("Stand-In input connected")
if base_params['phantom_ref'] is not None:
fallback_reasons.append("Phantom reference connected")
if base_params['lynx_embeds'] is not None:
fallback_reasons.append("Lynx input connected")
if base_params['longcat_num_cond_latents'] != 0 or base_params['longcat_num_ref_latents'] != 0 or base_params['longcat_avatar_options'] is not None:
fallback_reasons.append("LongCat options active")
if base_params['sdancer_input'] is not None:
fallback_reasons.append("SteadyDancer input connected")
if base_params['one_to_all_input'] is not None:
fallback_reasons.append("One-to-All input connected")
if base_params['dual_control_input'] is not None:
fallback_reasons.append("Dual Control input connected")
if base_params['rope_negative_offset'] != 0 or base_params['num_memory_frames'] != 0:
fallback_reasons.append("StoryMem/memory RoPE active")
if "comfy" not in getattr(transformer, "rope_func", ""):
fallback_reasons.append(f"rope_func={getattr(transformer, 'rope_func', None)!r}")
if not fallback_reasons:
fallback_reasons.append("unsupported sampler/model option")
log.info("SCAIL-2 fast path disabled; using normal path. Reason(s): " + "; ".join(fallback_reasons))
base_params["simple_scail2"] = scail2_fast_path
if scail2_fast_path:
base_params["simple_t2v"] = False
batch_size = 1
+270 -5
View File
@@ -7,6 +7,7 @@
# - Added variable WanAnimate anchor count support for pose and face embedding offsets.
# - Added a streamlined simple T2V/Bernini fast path with block-swap prefetch support.
# - Added simple T2V text/time embedding caches, cache-key helpers, and guarded no-op device moves.
# - Added a streamlined SCAIL-2 CFG fast path for the local WanAnimatePlus sampler.
# Licensed under the Apache License, Version 2.0
import math
import torch
@@ -2388,7 +2389,10 @@ class WanModel(torch.nn.Module):
pose_img_ids = torch.zeros((pose_f_len_full, pose_h_len_full, pose_w_len_full, 3), device=device, dtype=dtype)
global_h_offset, global_w_offset = 0, 120 # global spatial offset to separate pose from main frames spatially (SCAIL uses 120 as offset)
pose_t_start = 1 if scail_ref_frame_shape is not None else t_start + freq_offset
scail_ref_t_len = 0
if scail_ref_frame_shape is not None:
scail_ref_t_len = ((scail_ref_frame_shape[-3] + (self.patch_size[0] // 2)) // self.patch_size[0])
pose_t_start = scail_ref_t_len if scail_ref_frame_shape is not None else t_start + freq_offset
pose_img_ids[:, :, :, 0] = pose_img_ids[:, :, :, 0] + torch.linspace(pose_t_start, pose_t_start + (pose_f_len_full - 1), steps=pose_f_len_full, device=device, dtype=dtype).reshape(-1, 1, 1)
pose_img_ids[:, :, :, 1] = pose_img_ids[:, :, :, 1] + torch.linspace(global_h_offset + freq_offset, global_h_offset + pose_h_len_full - 1, steps=pose_h_len_full, device=device, dtype=dtype).reshape(1, -1, 1)
pose_img_ids[:, :, :, 2] = pose_img_ids[:, :, :, 2] + torch.linspace(global_w_offset + freq_offset, global_w_offset + pose_w_len_full - 1, steps=pose_w_len_full, device=device, dtype=dtype).reshape(1, 1, -1)
@@ -2515,6 +2519,21 @@ class WanModel(torch.nn.Module):
List[Tensor]:
List of denoised video tensors with original input shapes [C_out, F, H / 8, W / 8]
"""
# Fast SCAIL-2 path: local SCAIL-2 CFG workflow without unrelated control branches.
if kwargs.get("simple_scail2", False):
return self._forward_scail2_fast(
x, t, context, seq_len, freqs, scail_input,
is_uncond=is_uncond,
current_step_percentage=current_step_percentage,
current_step=current_step,
last_step=last_step,
clip_fea=clip_fea,
control_lora_enabled=control_lora_enabled,
transformer_options=transformer_options,
ntk_alphas=ntk_alphas,
context_window_start=kwargs.get("context_window_start", 0),
)
# Fast T2V path: bypass all extra checks for simple text-to-video inference
if kwargs.get("simple_t2v", False):
return self._forward_t2v(x, t, context, seq_len, freqs,
@@ -2652,11 +2671,12 @@ class WanModel(torch.nn.Module):
if ref_latent is not None and scail_input['ref_start_percent'] <= current_step_percentage <= scail_input['ref_end_percent']:
scail_ref_mask_latents = scail_input.get("ref_mask_latents", None)
scail_ref_frame_shape = ref_latent.shape
scail_ref_frames = ref_latent.shape[1]
x = [torch.cat([v, u], dim=1) for v, u in zip([ref_latent], x)]
seq_len += math.ceil((ref_latent.shape[-1] * ref_latent.shape[-2]) / 4 * ref_latent.shape[-3])
F += 1
prefix_frames = 1
suffix_frames += 1
F += scail_ref_frames
prefix_frames += scail_ref_frames
suffix_frames += scail_ref_frames
#uni3c controlnet
if uni3c_data is not None:
@@ -3621,6 +3641,249 @@ class WanModel(torch.nn.Module):
x = [u[:, prefix_frames:suffix_frames, ...].float() for u in x]
return (x, x_ovi, pred_id) if pred_id is not None else (x, x_ovi, None)
def _forward_scail2_fast(self, x, t, context, seq_len, freqs, scail_input,
is_uncond=False, current_step_percentage=0.0, current_step=0, last_step=False,
clip_fea=None, control_lora_enabled=False, transformer_options={},
ntk_alphas=None, context_window_start=0):
"""Streamlined SCAIL-2 forward for the local CFG sampler path."""
device = self.main_device
ntk = ntk_alphas if ntk_alphas is not None else [1.0, 1.0, 1.0]
if self.lora_scheduling_enabled:
update_lora_step(self, current_step)
if freqs is not None and freqs.device != device:
freqs = freqs.to(device)
_, F, H, W = x[0].shape
suffix_frames = x[0].shape[1]
prefix_frames = 0
pose_frame_shape = None
scail_ref_frame_shape = None
scail_ref_mask_latents = None
scail_ref_mask_flag = None
if scail_input is not None:
scail_ref_mask_flag = scail_input.get("ref_mask_flag", None)
ref_latent = scail_input.get("ref_latent_pos", None) if not is_uncond else scail_input.get("ref_latent_neg", None)
if ref_latent is not None and scail_input["ref_start_percent"] <= current_step_percentage <= scail_input["ref_end_percent"]:
scail_ref_mask_latents = scail_input.get("ref_mask_latents", None)
scail_ref_frame_shape = ref_latent.shape
scail_ref_frames = ref_latent.shape[1]
x = [torch.cat([v, u], dim=1) for v, u in zip([ref_latent], x)]
seq_len += math.ceil((ref_latent.shape[-1] * ref_latent.shape[-2]) / 4 * ref_latent.shape[-3])
F += scail_ref_frames
prefix_frames += scail_ref_frames
suffix_frames += scail_ref_frames
patch_embedding = self.expanded_patch_embedding if control_lora_enabled else self.original_patch_embedding
_module_to_if_needed(patch_embedding, device)
x = [patch_embedding(u.unsqueeze(0).to(torch.float32)).to(x[0].dtype) for u in x]
if scail_ref_mask_latents is not None:
if not hasattr(self, "patch_embedding_mask"):
raise ValueError("SCAIL-2 mask latents were provided, but the model has no patch_embedding_mask module")
_module_to_if_needed(self.patch_embedding_mask, device)
scail_mask_x = self.patch_embedding_mask(scail_ref_mask_latents.unsqueeze(0).to(torch.float32)).to(x[0].dtype)
if scail_mask_x.shape[2:] != x[0].shape[2:]:
raise ValueError(f"SCAIL-2 ref_mask_latents shape {scail_mask_x.shape[2:]} does not match latent token grid {x[0].shape[2:]}")
x = [u + scail_mask_x for u in x]
grid_sizes = torch.stack([torch.tensor(u.shape[2:], device=device, dtype=torch.long) for u in x])
original_grid_sizes = grid_sizes.clone()
x = [u.flatten(2).transpose(1, 2) for u in x]
self.original_seq_len = x[0].shape[1]
if scail_input is not None:
scail_pose_latents = scail_input.get("pose_latent", None)
if scail_pose_latents is not None and scail_input["pose_start_percent"] <= current_step_percentage <= scail_input["pose_end_percent"]:
_module_to_if_needed(self.patch_embedding_pose, device)
scail_x = self.patch_embedding_pose(scail_pose_latents.unsqueeze(0).to(torch.float32)).to(x[0].dtype)
scail_sam_latents = scail_input.get("sam_latents", None)
if scail_sam_latents is not None:
if not hasattr(self, "patch_embedding_mask"):
raise ValueError("SCAIL-2 sam_latents were provided, but the model has no patch_embedding_mask module")
_module_to_if_needed(self.patch_embedding_mask, device)
scail_mask_x = self.patch_embedding_mask(scail_sam_latents.unsqueeze(0).to(torch.float32)).to(scail_x.dtype)
if scail_mask_x.shape[2:] != scail_x.shape[2:]:
raise ValueError(f"SCAIL-2 sam_latents shape {scail_mask_x.shape[2:]} does not match pose token grid {scail_x.shape[2:]}")
scail_x = scail_x + scail_mask_x
scail_x = scail_x.flatten(2).transpose(1, 2)
x = [torch.cat([u, scail_x], dim=1) for u in x]
seq_len += scail_x.shape[1]
pose_frame_shape = scail_pose_latents.shape
seq_lens = torch.tensor([u.size(1) for u in x], dtype=torch.int32)
assert seq_lens.max() <= seq_len, f"max seq len {seq_lens.max()} exceeds provided seq_len {seq_len}"
x = torch.cat([torch.cat([u, u.new_zeros(1, seq_len - u.size(1), u.size(2))], dim=1) for u in x])
x = x.to(self.base_dtype)
if freqs is None and "comfy" in self.rope_func:
cache_key = (
"scail2_fast",
F, H, W,
tuple(pose_frame_shape) if pose_frame_shape is not None else None,
tuple(scail_ref_frame_shape) if scail_ref_frame_shape is not None else None,
None if scail_ref_mask_flag is None else bool(scail_ref_mask_flag),
self.rope_embedder.k,
tuple(ntk),
context_window_start,
)
if self.cached_freqs is not None and hasattr(self, "cached_key") and self.cached_key == cache_key:
freqs = self.cached_freqs
else:
freqs = self.rope_encode_comfy(
F, H, W,
ntk_alphas=ntk,
pose_frame_shape=pose_frame_shape,
scail_ref_frame_shape=scail_ref_frame_shape,
ref_mask_flag=scail_ref_mask_flag,
context_window_start=context_window_start,
device=device,
dtype=x.dtype,
)
self.cached_freqs = freqs
self.cached_key = cache_key
if freqs is None:
raise ValueError("SCAIL-2 fast path requires Comfy RoPE or precomputed freqs")
if freqs.device != device:
freqs = freqs.to(device)
_module_to_if_needed(self.time_embedding, device)
time_embed_dtype = self.time_embedding[0].weight.dtype
if time_embed_dtype not in [torch.float16, torch.bfloat16, torch.float32]:
time_embed_dtype = self.base_dtype
time_projection_enabled = hasattr(self, "time_projection")
if time_projection_enabled:
_module_to_if_needed(self.time_projection, device)
cached_time = getattr(self, "_scail2_time_embed_cache", None)
if (cached_time is not None and cached_time[0] is t and cached_time[1] == device
and cached_time[2] == time_embed_dtype and cached_time[3] == time_projection_enabled):
e_raw, e = cached_time[4], cached_time[5]
else:
e_raw = self.time_embedding(sinusoidal_embedding_1d(self.freq_dim, t.flatten()).to(device=device, dtype=time_embed_dtype))
if time_projection_enabled:
e = self.time_projection(e_raw).unflatten(1, (6, self.dim))
else:
e = e_raw.reshape(1, 6, self.dim)
if e.dim() < 3 or e.shape[1] != 6:
e = e.reshape(1, 6, self.dim)
self._scail2_time_embed_cache = (t, device, time_embed_dtype, time_projection_enabled, e_raw, e)
clip_embed = None
if clip_fea is not None and hasattr(self, "img_emb"):
_module_to_if_needed(self.img_emb, device)
clip_embed = self.img_emb(clip_fea.to(device))
if self.offload_img_emb:
self.img_emb.to(self.offload_device, non_blocking=self.use_non_blocking)
if hasattr(self, "text_embedding") and isinstance(context, list) and context != []:
text_embed_dtype = self.text_embedding[0].weight.dtype
if text_embed_dtype not in [torch.float16, torch.bfloat16, torch.float32]:
text_embed_dtype = self.base_dtype
text_cache_key = _t2v_context_cache_key(context, text_embed_dtype, device, self.text_len)
text_cache = getattr(self, "_scail2_text_embed_cache", None)
if text_cache is None:
text_cache = {}
self._scail2_text_embed_cache = text_cache
cached_context = text_cache.get(text_cache_key)
if cached_context is None:
_module_to_if_needed(self.text_embedding, device)
padded_context = torch.stack([torch.cat([u, u.new_zeros(self.text_len - u.size(0), u.size(1))]) for u in context]).to(text_embed_dtype)
cached_context = self.text_embedding(padded_context.to(device))
if len(text_cache) >= 4:
text_cache.clear()
text_cache[text_cache_key] = cached_context
if self.offload_txt_emb:
self.text_embedding.to(self.offload_device, non_blocking=self.use_non_blocking)
context = cached_context
elif isinstance(context, list):
context = None
block_kwargs = dict(
e=e,
seq_lens=seq_lens,
grid_sizes=grid_sizes,
freqs=freqs,
context=context,
clip_embed=clip_embed,
current_step=torch.tensor(current_step),
last_step=torch.tensor(last_step, dtype=torch.bool),
chunked_self_attention=False,
seq_chunks=1,
num_latent_frames=F,
frame_tokens=x.shape[1] // F,
original_seq_len=self.original_seq_len,
transformer_options=transformer_options,
)
if torch.cuda.is_available():
cuda_stream = None
events = [torch.cuda.Event() for _ in self.blocks]
swap_start_idx = len(self.blocks) - self.blocks_to_swap if self.blocks_to_swap > 0 else len(self.blocks)
else:
cuda_stream = None
events = None
swap_start_idx = len(self.blocks)
attn_override_blocks = attention_mode = None
attention_mode_override_active = False
attention_mode_override = transformer_options.get("attention_mode_override", None)
if attention_mode_override is not None:
attn_override_blocks = attention_mode_override.get("blocks", range(len(self.blocks)))
if attention_mode_override["start_step"] <= current_step < attention_mode_override["end_step"]:
attention_mode_override_active = True
if attention_mode_override["verbose"]:
tqdm.write(f"Applying attention mode override: {attention_mode_override['mode']} at step {current_step} on blocks: {attn_override_blocks if attn_override_blocks is not None else 'all'}")
for b, block in enumerate(self.blocks):
mm.throw_exception_if_processing_interrupted()
if attention_mode_override_active and b in attn_override_blocks:
attention_mode = attention_mode_override["mode"]
else:
attention_mode = None
if self.prefetch_blocks > 0:
for prefetch_offset in range(1, self.prefetch_blocks + 1):
prefetch_idx = b + prefetch_offset
if prefetch_idx < len(self.blocks) and self.blocks_to_swap > 0 and prefetch_idx >= swap_start_idx:
context_mgr = torch.cuda.stream(cuda_stream) if torch.cuda.is_available() and cuda_stream is not None else nullcontext()
with context_mgr:
self.blocks[prefetch_idx].to(device, non_blocking=self.use_non_blocking)
if events is not None:
events[prefetch_idx].record(cuda_stream)
if self.block_swap_debug:
transfer_start = time.perf_counter()
if b >= swap_start_idx and self.blocks_to_swap > 0:
if self.prefetch_blocks > 0 and events is not None:
if not events[b].query():
events[b].synchronize()
block.to(device)
if self.block_swap_debug:
transfer_end = time.perf_counter()
transfer_time = transfer_end - transfer_start
compute_start = time.perf_counter()
x, _, _, _ = block(x, attention_mode_override=attention_mode, **block_kwargs)
if self.block_swap_debug:
compute_end = time.perf_counter()
compute_time = compute_end - compute_start
to_cpu_transfer_start = time.perf_counter()
if b >= swap_start_idx and self.blocks_to_swap > 0:
block.to(self.offload_device, non_blocking=self.use_non_blocking)
if self.block_swap_debug:
to_cpu_transfer_end = time.perf_counter()
to_cpu_transfer_time = to_cpu_transfer_end - to_cpu_transfer_start
log.info(f"Block {b}: transfer_time={transfer_time:.4f}s, compute_time={compute_time:.4f}s, to_cpu_transfer_time={to_cpu_transfer_time:.4f}s")
x = x[:, :self.original_seq_len]
x = self.head(x, e_raw.to(device), temp_length=F)
x = self.unpatchify(x, original_grid_sizes)
x = [u[:, prefix_frames:suffix_frames, ...].float() for u in x]
return x, None, None
def _forward_t2v(self, x, t, context, seq_len, freqs, context_latents=None,
context_frame_shapes=None, context_window_start=0, ntk_alphas=None):
"""Streamlined T2V forward — no block swap, no extra checks, ~15x faster."""
@@ -3675,6 +3938,8 @@ class WanModel(torch.nn.Module):
# Time embedding (must produce [batch, 6, dim] for block.get_mod compatibility)
_module_to_if_needed(self.time_embedding, device)
time_embed_dtype = self.time_embedding[0].weight.dtype
if time_embed_dtype not in [torch.float16, torch.bfloat16, torch.float32]:
time_embed_dtype = self.base_dtype
time_projection_enabled = hasattr(self, 'time_projection')
if time_projection_enabled:
_module_to_if_needed(self.time_projection, device)
@@ -3683,7 +3948,7 @@ class WanModel(torch.nn.Module):
and cached_time[2] == time_embed_dtype and cached_time[3] == time_projection_enabled):
e_raw, e = cached_time[4], cached_time[5]
else:
e_raw = self.time_embedding(sinusoidal_embedding_1d(self.freq_dim, t.flatten()).to(time_embed_dtype))
e_raw = self.time_embedding(sinusoidal_embedding_1d(self.freq_dim, t.flatten()).to(device=device, dtype=time_embed_dtype))
if time_projection_enabled:
e = self.time_projection(e_raw).unflatten(1, (6, self.dim)) # [batch, 6, dim]
else: