Compare commits
11
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
14d40b1d57 | ||
|
|
192f2c3cb3 | ||
|
|
62f4d6e441 | ||
|
|
09ffef9e0f | ||
|
|
d9702d937b | ||
|
|
7307f4ea40 | ||
|
|
746330b5bf | ||
|
|
91a286cdf6 | ||
|
|
07f8b8d2a9 | ||
|
|
d849f6c7d6 | ||
|
|
12ea0093e3 |
@@ -6,9 +6,16 @@
|
||||
|
||||
1. Clone this repo into `custom_nodes` folder.
|
||||
2. Download motion modules and put them under `comfyui-animatediff/models/`.
|
||||
* Original modules: [Google Drive](https://drive.google.com/drive/folders/1EqLC65eR1-W-sGD0Im7fkED6c8GkiNFI) | [HuggingFace](https://huggingface.co/guoyww/animatediff) | [CivitAI](https://civitai.com/models/108836) | [Baidu NetDisk](https://pan.baidu.com/s/18ZpcSM6poBqxWNHtnyMcxg?pwd=et8y)
|
||||
* Community modules: [manshoety/AD_Stabilized_Motion](https://huggingface.co/manshoety/AD_Stabilized_Motion) | [CiaraRowles/TemporalDiff](https://huggingface.co/CiaraRowles/TemporalDiff)
|
||||
* AnimateDiff v2 [mm_sd_v15_v2.ckpt](https://huggingface.co/guoyww/animatediff/blob/main/mm_sd_v15_v2.ckpt)
|
||||
|
||||
- Original modules: [Google Drive](https://drive.google.com/drive/folders/1EqLC65eR1-W-sGD0Im7fkED6c8GkiNFI) | [HuggingFace](https://huggingface.co/guoyww/animatediff) | [CivitAI](https://civitai.com/models/108836) | [Baidu NetDisk](https://pan.baidu.com/s/18ZpcSM6poBqxWNHtnyMcxg?pwd=et8y)
|
||||
- Community modules: [manshoety/AD_Stabilized_Motion](https://huggingface.co/manshoety/AD_Stabilized_Motion) | [CiaraRowles/TemporalDiff](https://huggingface.co/CiaraRowles/TemporalDiff)
|
||||
- AnimateDiff v2 [mm_sd_v15_v2.ckpt](https://huggingface.co/guoyww/animatediff/blob/main/mm_sd_v15_v2.ckpt)
|
||||
|
||||
## Update 2023/09/21
|
||||
|
||||
#### **Sliding Window** is now available!
|
||||
|
||||
Sliding window is trigged automatically when generating more than 16 frames. To adjust the trigger number and other options, use `SlidingWindowOptions` node. See the sample workflow bellow.
|
||||
|
||||
## Nodes
|
||||
|
||||
@@ -17,35 +24,137 @@
|
||||
<img width="370" alt="image" src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/9d756d01-ea45-4d1c-8e48-56f2725c7ca1">
|
||||
|
||||
#### AnimateDiffSampler
|
||||
|
||||
- Mostly the same with `KSampler`
|
||||
- Use `AnimateDiffLoader` to load the motion module
|
||||
- `motion_module`: use `AnimateDiffLoader` to load the motion module
|
||||
- `inject_method`: should left default
|
||||
- `frame_number`: animation length
|
||||
- `latent_image`: You can pass an `EmptyLatentImage`
|
||||
- `sliding_window_opts`: custom sliding window options
|
||||
|
||||
<img width="370" alt="image" src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/f22d6b36-ce36-44cc-80e8-dffe6f77b296">
|
||||
<img width="370" alt="image" src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/a352195d-f40c-494d-bd3d-30ee88174b88">
|
||||
|
||||
#### AnimateDiffCombine
|
||||
|
||||
- Combine GIF frames and produce the GIF image
|
||||
- `frame_rate`: number of frame per second
|
||||
- `loop_count`: use 0 for infinite loop
|
||||
- `save_image`: should GIF be saved to disk
|
||||
- `format`: supports `image/gif`, `image/webp` (better compression) or `video/webm` (need `ffmpeg` installed and available in PATH)
|
||||
- `format`: supports `image/gif`, `image/webp` (better compression), `video/webm`, `video/h264-mp4`, `video/h265-mp4`. To use video formats, you'll need [ffmpeg](https://ffmpeg.org/download.html) installed and available in **`PATH`**
|
||||
|
||||
<img width="370" alt="image" src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/381c5acc-06ef-43da-ada0-3dc76f37a3e4">
|
||||
|
||||
#### Example Workflow
|
||||
#### SlidingWindowOptions
|
||||
|
||||
<img width="1311" alt="image" src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/b7164539-bc58-4ef9-b178-d914e833805e">
|
||||
Custom sliding window options
|
||||
|
||||
- `context_length`: number of frame per _window_. Use **16** to get the best results. Reduce it if you have low VRAM.
|
||||
- `closed_loop`: try to make the GIF a closed loop. Will take longer to render.
|
||||
|
||||
Workflow file: https://github.com/ArtVentureX/comfyui-animatediff/blob/main/workflow.json
|
||||
<img width="370" alt="image" src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/6679a8dd-bf96-419f-8934-ea2b046dd23c">
|
||||
|
||||
## Samples
|
||||
#### LoadVideo
|
||||
|
||||

|
||||
Load GIF or video as images. Usefull to load a GIF as ControlNet input.
|
||||
|
||||

|
||||
- `frame_start`: Skip some begining frames and start at `frame_start`
|
||||
- `frame_limit`: Only take `frame_limit` frames
|
||||
|
||||
<img width="370" alt="image" src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/684176d5-6369-4a27-9f33-e721e0fe1876">
|
||||
|
||||
## Workflows
|
||||
|
||||
### Simple txt2gif
|
||||
|
||||
<img width="1280" alt="image" src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/b7164539-bc58-4ef9-b178-d914e833805e">
|
||||
|
||||
Workflow: [simple.json](https://github.com/ArtVentureX/comfyui-animatediff/blob/main/workflows/simple.json)
|
||||
|
||||
Samples:
|
||||
|
||||

|
||||
|
||||

|
||||
|
||||
### Long duration with sliding window
|
||||
|
||||
<img width="1280" alt="image" src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/0f8bfb87-83cb-4119-9777-e3948ec0cb5c">
|
||||
|
||||
Workflow: [sliding-window.json](https://github.com/ArtVentureX/comfyui-animatediff/blob/main/workflows/sliding-window.json)
|
||||
|
||||
Samples:
|
||||

|
||||
|
||||

|
||||
|
||||
### Latent upscale
|
||||
|
||||
Upscale latent output using `LatentUpscale` then do a 2nd pass with `AnimateDiffSampler`.
|
||||
|
||||
<img width="1280" alt="image" src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/987a1c5a-c1f8-4b24-8c62-f14496261d6c">
|
||||
|
||||
Workflow: [latent-upscale.json](https://github.com/ArtVentureX/comfyui-animatediff/blob/main/workflows/latent-upscale.json)
|
||||
|
||||
Samples:
|
||||

|
||||
|
||||
### Using with ControlNet
|
||||
|
||||
You will need following additional nodes:
|
||||
|
||||
- [Kosinkadink/ComfyUI-Advanced-ControlNet](https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet): Apply different weight for each latent in batch
|
||||
- [Fannovel16/comfyui_controlnet_aux](https://github.com/Fannovel16/comfyui_controlnet_aux): ControlNet preprocessors
|
||||
|
||||
#### Animate with starting and ending images
|
||||
|
||||
- Use `LatentKeyframe` and `TimestampKeyframe` from [ComfyUI-Advanced-ControlNet](https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet) to apply diffrent weights for each latent index.
|
||||
- Use 2 controlnet modules for two images with weights reverted.
|
||||
|
||||

|
||||
|
||||
Workflow: [cn-2images.json](https://github.com/ArtVentureX/comfyui-animatediff/blob/main/workflows/cn-2images.json)
|
||||
|
||||
Samples:
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td>
|
||||
<img src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/e73fc3cd-a590-40a9-8b33-11358b54f0cd">
|
||||
</td>
|
||||
<td>
|
||||
<img src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/96c2ee92-d457-4862-94d3-d675b7fa2d1f">
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>
|
||||
<img src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/46338853-1ae0-433e-925c-2a41e0382e68">
|
||||
</td>
|
||||
<td>
|
||||
<img src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/707e4ce3-3594-4ff5-9a5f-f9596eb2bcf4">
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
#### Using GIF as ControlNet input
|
||||
|
||||
Using a GIF (or video, or a list of images) as ControlNet input.
|
||||
|
||||

|
||||
|
||||
Workflow: [cn-vid2vid.json](https://github.com/ArtVentureX/comfyui-animatediff/blob/main/workflows/cn-vid2vid.json)
|
||||
|
||||
Samples:
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td>
|
||||
<img src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/bf926f52-da97-4fb4-b86a-8b26ef5fab04">
|
||||
</td>
|
||||
<td>
|
||||
<img src="https://github.com/ArtVentureX/comfyui-animatediff/assets/133728487/f6472c8c-9b92-47c2-8f28-638726f21be7">
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
## Known Issues
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,7 +1,16 @@
|
||||
import os
|
||||
import hashlib
|
||||
from typing import Dict
|
||||
|
||||
import folder_paths
|
||||
import comfy.model_management as model_management
|
||||
from comfy.utils import load_torch_file, calculate_parameters
|
||||
|
||||
from .logger import logger
|
||||
from .motion_module import MotionWrapper
|
||||
|
||||
|
||||
motion_modules: Dict[str, MotionWrapper] = {}
|
||||
|
||||
|
||||
folder_paths.folder_names_and_paths["AnimateDiff"] = (
|
||||
@@ -11,12 +20,6 @@ folder_paths.folder_names_and_paths["AnimateDiff"] = (
|
||||
],
|
||||
folder_paths.supported_pt_extensions,
|
||||
)
|
||||
folder_paths.folder_names_and_paths["video_formats"] = (
|
||||
[
|
||||
os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "video_formats"),
|
||||
],
|
||||
[".json"]
|
||||
)
|
||||
|
||||
|
||||
def get_available_models():
|
||||
@@ -31,3 +34,23 @@ def get_model_hash(file_path):
|
||||
with open(file_path, "rb") as f:
|
||||
bytes = f.read() # read entire file as bytes
|
||||
return hashlib.sha256(bytes).hexdigest()
|
||||
|
||||
|
||||
def load_motion_module(model_name: str):
|
||||
model_path = get_model_path(model_name)
|
||||
model_hash = get_model_hash(model_path)
|
||||
if model_hash not in motion_modules:
|
||||
logger.info(f"Loading motion module {model_name}")
|
||||
mm_state_dict = load_torch_file(model_path)
|
||||
motion_module = MotionWrapper.from_state_dict(mm_state_dict, model_name)
|
||||
|
||||
params = calculate_parameters(mm_state_dict, "")
|
||||
if model_management.should_use_fp16(model_params=params):
|
||||
logger.info(f"Converting motion module to fp16.")
|
||||
motion_module.half()
|
||||
offload_device = model_management.unet_offload_device()
|
||||
motion_module = motion_module.to(offload_device)
|
||||
|
||||
motion_modules[model_hash] = motion_module
|
||||
|
||||
return motion_modules[model_hash]
|
||||
|
||||
@@ -41,6 +41,7 @@ class MotionWrapper(nn.Module):
|
||||
self.down_blocks = nn.ModuleList([])
|
||||
self.up_blocks = nn.ModuleList([])
|
||||
self.mid_block = None
|
||||
self.encoding_max_len = encoding_max_len
|
||||
|
||||
for c in (320, 640, 1280, 1280):
|
||||
self.down_blocks.append(
|
||||
@@ -56,7 +57,7 @@ class MotionWrapper(nn.Module):
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_pretrained(cls, mm_state_dict: dict[str, Tensor], mm_type: str):
|
||||
def from_state_dict(cls, mm_state_dict: dict[str, Tensor], mm_type: str):
|
||||
encoding_max_len = get_encoding_max_len(mm_state_dict)
|
||||
is_v2 = has_mid_block(mm_state_dict)
|
||||
|
||||
|
||||
+181
-304
@@ -2,173 +2,23 @@ import os
|
||||
import json
|
||||
import torch
|
||||
import numpy as np
|
||||
from typing import Dict, List
|
||||
import hashlib
|
||||
from typing import List
|
||||
from torch import Tensor
|
||||
from torch.nn.functional import group_norm
|
||||
from PIL import Image
|
||||
from PIL import Image, ImageSequence
|
||||
from PIL.PngImagePlugin import PngInfo
|
||||
from einops import rearrange
|
||||
|
||||
import folder_paths
|
||||
import comfy.ldm.modules.diffusionmodules.openaimodel as openaimodel
|
||||
import comfy.model_management as model_management
|
||||
from comfy.model_base import BaseModel
|
||||
from comfy.ldm.modules.attention import SpatialTransformer
|
||||
from comfy.utils import load_torch_file, calculate_parameters
|
||||
from nodes import KSampler
|
||||
|
||||
from .logger import logger
|
||||
from .motion_module import MotionWrapper, VanillaTemporalModule
|
||||
from .model_utils import get_available_models, get_model_path, get_model_hash
|
||||
from .model_utils import get_available_models, load_motion_module
|
||||
from .utils import pil2tensor
|
||||
from .sampler import AnimateDiffSampler, AnimateDiffSlidingWindowOptions
|
||||
|
||||
|
||||
def forward_timestep_embed(
|
||||
ts, x, emb, context=None, transformer_options={}, output_shape=None
|
||||
):
|
||||
for layer in ts:
|
||||
if isinstance(layer, openaimodel.TimestepBlock):
|
||||
x = layer(x, emb)
|
||||
elif isinstance(layer, VanillaTemporalModule):
|
||||
x = layer(x, context)
|
||||
elif isinstance(layer, SpatialTransformer):
|
||||
x = layer(x, context, transformer_options)
|
||||
transformer_options["current_index"] += 1
|
||||
elif isinstance(layer, openaimodel.Upsample):
|
||||
x = layer(x, output_shape=output_shape)
|
||||
else:
|
||||
x = layer(x)
|
||||
return x
|
||||
SLIDING_CONTEXT_LENGTH = 16
|
||||
|
||||
|
||||
def groupnorm_mm_factory(video_length: int):
|
||||
def groupnorm_mm_forward(self, input: Tensor) -> Tensor:
|
||||
# axes_factor normalizes batch based on total conds and unconds passed in batch;
|
||||
# the conds and unconds per batch can change based on VRAM optimizations that may kick in
|
||||
axes_factor = input.size(0) // video_length
|
||||
|
||||
input = rearrange(input, "(b f) c h w -> b c f h w", b=axes_factor)
|
||||
input = group_norm(input, self.num_groups, self.weight, self.bias, self.eps)
|
||||
input = rearrange(input, "b c f h w -> (b f) c h w", b=axes_factor)
|
||||
return input
|
||||
|
||||
return groupnorm_mm_forward
|
||||
|
||||
|
||||
orig_forward_timestep_embed = openaimodel.forward_timestep_embed
|
||||
orig_maximum_batch_area = model_management.maximum_batch_area
|
||||
orig_groupnorm_forward = torch.nn.GroupNorm.forward
|
||||
openaimodel.forward_timestep_embed = forward_timestep_embed
|
||||
|
||||
motion_modules: Dict[str, MotionWrapper] = {}
|
||||
|
||||
|
||||
def load_motion_module(model_name: str):
|
||||
model_path = get_model_path(model_name)
|
||||
model_hash = get_model_hash(model_path)
|
||||
if model_hash not in motion_modules:
|
||||
logger.info(f"Loading motion module {model_name}")
|
||||
mm_state_dict = load_torch_file(model_path)
|
||||
motion_module = MotionWrapper.from_pretrained(mm_state_dict, model_name)
|
||||
|
||||
params = calculate_parameters(mm_state_dict, "")
|
||||
if model_management.should_use_fp16(model_params=params):
|
||||
logger.info(f"Converting motion module to fp16.")
|
||||
motion_module.half()
|
||||
offload_device = model_management.unet_offload_device()
|
||||
motion_module = motion_module.to(offload_device)
|
||||
|
||||
motion_modules[model_hash] = motion_module
|
||||
|
||||
return motion_modules[model_hash]
|
||||
|
||||
|
||||
def inject_motion_module_to_unet_legacy(unet, motion_module: MotionWrapper):
|
||||
for mm_idx, unet_idx in enumerate([1, 2, 4, 5, 7, 8, 10, 11]):
|
||||
mm_idx0, mm_idx1 = mm_idx // 2, mm_idx % 2
|
||||
unet.input_blocks[unet_idx].append(
|
||||
motion_module.down_blocks[mm_idx0].motion_modules[mm_idx1]
|
||||
)
|
||||
|
||||
for unet_idx in range(12):
|
||||
mm_idx0, mm_idx1 = unet_idx // 3, unet_idx % 3
|
||||
if unet_idx % 2 == 2:
|
||||
unet.output_blocks[unet_idx].insert(
|
||||
-1, motion_module.up_blocks[mm_idx0].motion_modules[mm_idx1]
|
||||
)
|
||||
else:
|
||||
unet.output_blocks[unet_idx].append(
|
||||
motion_module.up_blocks[mm_idx0].motion_modules[mm_idx1]
|
||||
)
|
||||
if motion_module.is_v2:
|
||||
unet.middle_block.insert(-1, motion_module.mid_block.motion_modules[0])
|
||||
|
||||
unet.motion_module = motion_module
|
||||
|
||||
|
||||
def eject_motion_module_from_unet_legacy(unet):
|
||||
for unet_idx in [1, 2, 4, 5, 7, 8, 10, 11]:
|
||||
unet.input_blocks[unet_idx].pop(-1)
|
||||
|
||||
for unet_idx in range(12):
|
||||
if unet_idx % 2 == 2:
|
||||
unet.output_blocks[unet_idx].pop(-2)
|
||||
else:
|
||||
unet.output_blocks[unet_idx].pop(-1)
|
||||
|
||||
if unet.motion_module.is_v2:
|
||||
unet.middle_block.pop(-2)
|
||||
|
||||
del unet.motion_module
|
||||
|
||||
|
||||
def inject_motion_module_to_unet(unet, motion_module: MotionWrapper):
|
||||
for mm_idx, unet_idx in enumerate([1, 2, 4, 5, 7, 8, 10, 11]):
|
||||
mm_idx0, mm_idx1 = mm_idx // 2, mm_idx % 2
|
||||
unet.input_blocks[unet_idx].append(
|
||||
motion_module.down_blocks[mm_idx0].motion_modules[mm_idx1]
|
||||
)
|
||||
|
||||
for unet_idx in range(12):
|
||||
mm_idx0, mm_idx1 = unet_idx // 3, unet_idx % 3
|
||||
if unet_idx % 3 == 2 and unet_idx != 11:
|
||||
unet.output_blocks[unet_idx].insert(
|
||||
-1, motion_module.up_blocks[mm_idx0].motion_modules[mm_idx1]
|
||||
)
|
||||
else:
|
||||
unet.output_blocks[unet_idx].append(
|
||||
motion_module.up_blocks[mm_idx0].motion_modules[mm_idx1]
|
||||
)
|
||||
if motion_module.is_v2:
|
||||
unet.middle_block.insert(-1, motion_module.mid_block.motion_modules[0])
|
||||
|
||||
unet.motion_module = motion_module
|
||||
|
||||
|
||||
def eject_motion_module_from_unet(unet):
|
||||
for unet_idx in [1, 2, 4, 5, 7, 8, 10, 11]:
|
||||
unet.input_blocks[unet_idx].pop(-1)
|
||||
|
||||
for unet_idx in range(12):
|
||||
if unet_idx % 3 == 2 and unet_idx != 11:
|
||||
unet.output_blocks[unet_idx].pop(-2)
|
||||
else:
|
||||
unet.output_blocks[unet_idx].pop(-1)
|
||||
|
||||
if unet.motion_module.is_v2:
|
||||
unet.middle_block.pop(-2)
|
||||
|
||||
del unet.motion_module
|
||||
|
||||
|
||||
injectors = {
|
||||
"legacy": inject_motion_module_to_unet_legacy,
|
||||
"default": inject_motion_module_to_unet,
|
||||
}
|
||||
|
||||
ejectors = {
|
||||
"legacy": eject_motion_module_from_unet_legacy,
|
||||
"default": eject_motion_module_from_unet,
|
||||
}
|
||||
video_formats_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "video_formats")
|
||||
video_formats = ["video/" + x[:-5] for x in os.listdir(video_formats_dir)]
|
||||
|
||||
|
||||
class AnimateDiffModuleLoader:
|
||||
@@ -193,134 +43,6 @@ class AnimateDiffModuleLoader:
|
||||
return (motion_module,)
|
||||
|
||||
|
||||
class AnimateDiffSampler(KSampler):
|
||||
@classmethod
|
||||
def INPUT_TYPES(s):
|
||||
inputs = {
|
||||
"required": {
|
||||
"motion_module": ("MOTION_MODULE",),
|
||||
"inject_method": (["default", "legacy"],),
|
||||
"frame_number": (
|
||||
"INT",
|
||||
{"default": 16, "min": 2, "max": 32, "step": 1},
|
||||
),
|
||||
}
|
||||
}
|
||||
inputs["required"].update(KSampler.INPUT_TYPES()["required"])
|
||||
return inputs
|
||||
|
||||
FUNCTION = "animatediff_sample"
|
||||
CATEGORY = "Animate Diff"
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.prev_beta = None
|
||||
self.prev_linear_start = None
|
||||
self.prev_linear_end = None
|
||||
|
||||
def override_beta_schedule(self, model: BaseModel):
|
||||
logger.info(f"Override beta schedule.")
|
||||
self.prev_beta = model.get_buffer("betas").cpu().clone()
|
||||
self.prev_linear_start = model.linear_start
|
||||
self.prev_linear_end = model.linear_end
|
||||
model.register_schedule(
|
||||
given_betas=None,
|
||||
beta_schedule="sqrt_linear",
|
||||
timesteps=1000,
|
||||
linear_start=0.00085,
|
||||
linear_end=0.012,
|
||||
cosine_s=8e-3,
|
||||
)
|
||||
|
||||
def restore_beta_schedule(self, model: BaseModel):
|
||||
logger.info(f"Restoring beta schedule.")
|
||||
model.register_schedule(
|
||||
given_betas=self.prev_beta,
|
||||
linear_start=self.prev_linear_start,
|
||||
linear_end=self.prev_linear_end,
|
||||
)
|
||||
self.prev_beta = None
|
||||
self.prev_linear_start = None
|
||||
self.prev_linear_end = None
|
||||
|
||||
def inject_motion_module(
|
||||
self, model, motion_module: MotionWrapper, inject_method: str, frame_number: int
|
||||
):
|
||||
model = model.clone()
|
||||
unet = model.model.diffusion_model
|
||||
|
||||
logger.info(f"Injecting motion module with method {inject_method}.")
|
||||
motion_module.set_video_length(frame_number)
|
||||
injectors[inject_method](unet, motion_module)
|
||||
self.override_beta_schedule(model.model)
|
||||
if not motion_module.is_v2:
|
||||
logger.info(f"Hacking GroupNorm.forward function.")
|
||||
torch.nn.GroupNorm.forward = groupnorm_mm_factory(frame_number)
|
||||
|
||||
return model
|
||||
|
||||
def eject_motion_module(self, model, inject_method):
|
||||
unet = model.model.diffusion_model
|
||||
|
||||
self.restore_beta_schedule(model.model)
|
||||
if not unet.motion_module.is_v2:
|
||||
logger.info(f"Restore GroupNorm.forward function.")
|
||||
torch.nn.GroupNorm.forward = orig_groupnorm_forward
|
||||
|
||||
logger.info(f"Ejecting motion module with method {inject_method}.")
|
||||
ejectors[inject_method](unet)
|
||||
|
||||
def animatediff_sample(
|
||||
self,
|
||||
motion_module,
|
||||
inject_method,
|
||||
frame_number,
|
||||
model,
|
||||
seed,
|
||||
steps,
|
||||
cfg,
|
||||
sampler_name,
|
||||
scheduler,
|
||||
positive,
|
||||
negative,
|
||||
latent_image,
|
||||
denoise=1.0,
|
||||
):
|
||||
model = self.inject_motion_module(
|
||||
model, motion_module, inject_method, frame_number
|
||||
)
|
||||
|
||||
init_frames = len(latent_image["samples"])
|
||||
samples = latent_image["samples"][:init_frames, :, :, :].clone().cpu()
|
||||
|
||||
if init_frames < frame_number:
|
||||
last_frame = samples[-1].unsqueeze(0)
|
||||
repeated_last_frames = last_frame.repeat(
|
||||
frame_number - init_frames, 1, 1, 1
|
||||
)
|
||||
samples = torch.cat((samples, repeated_last_frames), dim=0)
|
||||
|
||||
latent_image = {"samples": samples}
|
||||
|
||||
try:
|
||||
return super().sample(
|
||||
model,
|
||||
seed,
|
||||
steps,
|
||||
cfg,
|
||||
sampler_name,
|
||||
scheduler,
|
||||
positive,
|
||||
negative,
|
||||
latent_image,
|
||||
denoise=denoise,
|
||||
)
|
||||
except:
|
||||
raise
|
||||
finally:
|
||||
self.eject_motion_module(model, inject_method)
|
||||
|
||||
|
||||
class AnimateDiffCombine:
|
||||
@classmethod
|
||||
def INPUT_TYPES(s):
|
||||
@@ -332,11 +54,10 @@ class AnimateDiffCombine:
|
||||
{"default": 8, "min": 1, "max": 24, "step": 1},
|
||||
),
|
||||
"loop_count": ("INT", {"default": 0, "min": 0, "max": 100, "step": 1}),
|
||||
"save_image": ([True, False],),
|
||||
"save_image": ("BOOLEAN", {"default": True}),
|
||||
"filename_prefix": ("STRING", {"default": "animate_diff"}),
|
||||
"format": (["image/gif", "image/webp"] +
|
||||
["video/"+x[:-5] for x in folder_paths.get_filename_list("video_formats")],),
|
||||
"pingpong": ([False, True],),
|
||||
"format": (["image/gif", "image/webp"] + video_formats,),
|
||||
"pingpong": ("BOOLEAN", {"default": False}),
|
||||
},
|
||||
"hidden": {
|
||||
"prompt": "PROMPT",
|
||||
@@ -344,7 +65,7 @@ class AnimateDiffCombine:
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("GIF",)
|
||||
RETURN_TYPES = ()
|
||||
OUTPUT_NODE = True
|
||||
CATEGORY = "Animate Diff"
|
||||
FUNCTION = "generate_gif"
|
||||
@@ -369,11 +90,7 @@ class AnimateDiffCombine:
|
||||
frames.append(img)
|
||||
|
||||
# save image
|
||||
output_dir = (
|
||||
folder_paths.get_output_directory()
|
||||
if save_image
|
||||
else folder_paths.get_temp_directory()
|
||||
)
|
||||
output_dir = folder_paths.get_output_directory() if save_image else folder_paths.get_temp_directory()
|
||||
(
|
||||
full_output_folder,
|
||||
filename,
|
||||
@@ -422,17 +139,33 @@ class AnimateDiffCombine:
|
||||
ffmpeg_path = shutil.which("ffmpeg")
|
||||
if ffmpeg_path is None:
|
||||
raise ProcessLookupError("Could not find ffmpeg")
|
||||
video_format_path = folder_paths.get_full_path("video_formats", format_ext + ".json")
|
||||
with open(video_format_path, 'r') as stream:
|
||||
video_format_path = os.path.join(video_formats_dir, format_ext + ".json")
|
||||
with open(video_format_path, "r") as stream:
|
||||
video_format = json.load(stream)
|
||||
file = f"{filename}_{counter:05}_.{video_format['extension']}"
|
||||
file_path = os.path.join(full_output_folder, file)
|
||||
dimensions = f"{frames[0].width}x{frames[0].height}"
|
||||
args = [ffmpeg_path, "-v", "error", "-f", "rawvideo", "-pix_fmt", "rgb24",
|
||||
"-s", dimensions, "-r", str(frame_rate), "-i", "-"] \
|
||||
+ video_format['main_pass'] + [file_path]
|
||||
args = (
|
||||
[
|
||||
ffmpeg_path,
|
||||
"-v",
|
||||
"error",
|
||||
"-f",
|
||||
"rawvideo",
|
||||
"-pix_fmt",
|
||||
"rgb24",
|
||||
"-s",
|
||||
dimensions,
|
||||
"-r",
|
||||
str(frame_rate),
|
||||
"-i",
|
||||
"-",
|
||||
]
|
||||
+ video_format["main_pass"]
|
||||
+ [file_path]
|
||||
)
|
||||
|
||||
env=os.environ
|
||||
env = os.environ
|
||||
if "environment" in video_format:
|
||||
env.update(video_format["environment"])
|
||||
with subprocess.Popen(args, stdin=subprocess.PIPE, env=env) as proc:
|
||||
@@ -447,16 +180,160 @@ class AnimateDiffCombine:
|
||||
"format": format,
|
||||
}
|
||||
]
|
||||
return {"ui": {"gifs": previews}}
|
||||
return {"ui": {"videos": previews}}
|
||||
|
||||
|
||||
class LoadVideo:
|
||||
@classmethod
|
||||
def INPUT_TYPES(s):
|
||||
input_dir = os.path.join(folder_paths.get_input_directory(), "video")
|
||||
if not os.path.exists(input_dir):
|
||||
os.makedirs(input_dir, exist_ok=True)
|
||||
|
||||
files = [f"video/{f}" for f in os.listdir(input_dir) if os.path.isfile(os.path.join(input_dir, f))]
|
||||
|
||||
return {
|
||||
"required": {
|
||||
"video": (sorted(files), {"video_upload": True}),
|
||||
},
|
||||
"optional": {
|
||||
"frame_start": ("INT", {"default": 0, "min": 0, "max": 0xFFFFFFFF, "step": 1}),
|
||||
"frame_limit": ("INT", {"default": 16, "min": 1, "max": 10240, "step": 1}),
|
||||
},
|
||||
}
|
||||
|
||||
CATEGORY = "Animate Diff/Utils"
|
||||
RETURN_TYPES = ("IMAGE", "INT")
|
||||
RETURN_NAMES = ("frames", "frame_count")
|
||||
FUNCTION = "load"
|
||||
|
||||
def load_gif(self, gif_path: str, frame_start: int, frame_limit: int):
|
||||
image = Image.open(gif_path)
|
||||
frames = []
|
||||
|
||||
for i, frame in enumerate(ImageSequence.Iterator(image)):
|
||||
if i < frame_start:
|
||||
continue
|
||||
elif i >= frame_start + frame_limit:
|
||||
break
|
||||
else:
|
||||
frames.append(pil2tensor(frame.copy().convert("RGB")))
|
||||
|
||||
return frames
|
||||
|
||||
def load_video(self, video_path, frame_start: int, frame_limit: int):
|
||||
import cv2
|
||||
|
||||
video = cv2.VideoCapture(video_path)
|
||||
video.set(cv2.CAP_PROP_POS_FRAMES, frame_start)
|
||||
|
||||
frames = []
|
||||
for i in range(frame_limit):
|
||||
# Read the next frame
|
||||
ret, frame = video.read()
|
||||
if ret:
|
||||
# Convert the frame to RGB (OpenCV uses BGR)
|
||||
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
|
||||
# Convert the NumPy array to a PIL image and append to list
|
||||
frames.append(pil2tensor(Image.fromarray(frame)))
|
||||
else:
|
||||
break
|
||||
|
||||
video.release()
|
||||
|
||||
return frames
|
||||
|
||||
def load(self, video: str, frame_start=0, frame_limit=16):
|
||||
video_path = folder_paths.get_annotated_filepath(video)
|
||||
(_, ext) = os.path.splitext(video_path)
|
||||
|
||||
if ext.lower() in {".gif", ".webp"}:
|
||||
frames = self.load_gif(video_path, frame_start, frame_limit)
|
||||
elif ext.lower() in {".webp", ".mp4", ".mov", ".avi"}:
|
||||
frames = self.load_video(video_path, frame_start, frame_limit)
|
||||
else:
|
||||
raise ValueError(f"Unsupported video format: {ext}")
|
||||
|
||||
return (torch.cat(frames, dim=0),)
|
||||
|
||||
@classmethod
|
||||
def IS_CHANGED(s, image, *args, **kwargs):
|
||||
image_path = folder_paths.get_annotated_filepath(image)
|
||||
m = hashlib.sha256()
|
||||
with open(image_path, "rb") as f:
|
||||
m.update(f.read())
|
||||
return m.digest().hex()
|
||||
|
||||
@classmethod
|
||||
def VALIDATE_INPUTS(s, video, *args, **kwargs):
|
||||
if not folder_paths.exists_annotated_filepath(video):
|
||||
return "Invalid video file: {}".format(video)
|
||||
|
||||
return True
|
||||
|
||||
|
||||
class ImageSizeAndBatchSize:
|
||||
@classmethod
|
||||
def INPUT_TYPES(s):
|
||||
return {
|
||||
"required": {
|
||||
"image": ("IMAGE",),
|
||||
},
|
||||
}
|
||||
|
||||
CATEGORY = "Animate Diff/Utils"
|
||||
RETURN_TYPES = ("INT", "INT", "INT")
|
||||
RETURN_NAMES = ("width", "height", "batch_size")
|
||||
FUNCTION = "batch_size"
|
||||
|
||||
def batch_size(self, image: Tensor):
|
||||
(batch_size, height, width) = image.shape[0:3]
|
||||
return (width, height, batch_size)
|
||||
|
||||
|
||||
class ImageChunking:
|
||||
@classmethod
|
||||
def INPUT_TYPES(s):
|
||||
return {
|
||||
"required": {
|
||||
"images": ("IMAGE",),
|
||||
"chunk_size": ("INT", {"default": 16, "min": 1, "max": 1024, "step": 1}),
|
||||
"allow_remainder": ("BOOLEAN", {"default": True}),
|
||||
},
|
||||
}
|
||||
|
||||
CATEGORY = "Animate Diff/Utils"
|
||||
RETURN_TYPES = ("IMAGE",)
|
||||
OUTPUT_IS_LIST = (True,)
|
||||
FUNCTION = "chunk"
|
||||
|
||||
def chunk(self, images: Tensor, chunk_size: int, allow_remainder: bool):
|
||||
# Check if tensor is divisible into chunks of chunk_size
|
||||
if images.shape[0] % chunk_size != 0 and not allow_remainder:
|
||||
raise ValueError("Tensor's first dimension is not divisible by chunk size")
|
||||
|
||||
# Use torch.chunk to divide the tensor
|
||||
chunk_count = images.shape[0] // chunk_size + images.shape[0] % chunk_size
|
||||
|
||||
print("chunk_count", chunk_count)
|
||||
chunks = torch.chunk(images, chunk_count, dim=0)
|
||||
|
||||
return (list(chunks),)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"AnimateDiffModuleLoader": AnimateDiffModuleLoader,
|
||||
"AnimateDiffCombine": AnimateDiffCombine,
|
||||
"AnimateDiffSampler": AnimateDiffSampler,
|
||||
"AnimateDiffSlidingWindowOptions": AnimateDiffSlidingWindowOptions,
|
||||
"LoadVideo": LoadVideo,
|
||||
"ImageSizeAndBatchSize": ImageSizeAndBatchSize,
|
||||
}
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"AnimateDiffModuleLoader": "Animate Diff Module Loader",
|
||||
"AnimateDiffSampler": "Animate Diff Sampler",
|
||||
"AnimateDiffSlidingWindowOptions": "Sliding Window Options",
|
||||
"AnimateDiffCombine": "Animate Diff Combine",
|
||||
"LoadVideo": "Load Video",
|
||||
"ImageSizeAndBatchSize": "Get Image Size + Batch Size",
|
||||
}
|
||||
|
||||
@@ -0,0 +1,316 @@
|
||||
import torch
|
||||
from torch import Tensor
|
||||
from torch.nn.functional import group_norm
|
||||
from einops import rearrange
|
||||
|
||||
import comfy.ldm.modules.diffusionmodules.openaimodel as openaimodel
|
||||
import comfy.model_management as model_management
|
||||
from comfy.model_base import BaseModel
|
||||
from comfy.ldm.modules.attention import SpatialTransformer
|
||||
from nodes import KSampler
|
||||
|
||||
from .logger import logger
|
||||
from .motion_module import MotionWrapper, VanillaTemporalModule
|
||||
from .sliding_schedule import ContextSchedules
|
||||
from .sliding_context_sampling import SlidingContext, inject_sampling_function, eject_sampling_function
|
||||
|
||||
|
||||
SLIDING_CONTEXT_LENGTH = 16
|
||||
|
||||
|
||||
def forward_timestep_embed(ts, x, emb, context=None, transformer_options={}, output_shape=None):
|
||||
for layer in ts:
|
||||
if isinstance(layer, openaimodel.TimestepBlock):
|
||||
x = layer(x, emb)
|
||||
elif isinstance(layer, VanillaTemporalModule):
|
||||
x = layer(x, context)
|
||||
elif isinstance(layer, SpatialTransformer):
|
||||
x = layer(x, context, transformer_options)
|
||||
transformer_options["current_index"] += 1
|
||||
elif isinstance(layer, openaimodel.Upsample):
|
||||
x = layer(x, output_shape=output_shape)
|
||||
else:
|
||||
x = layer(x)
|
||||
return x
|
||||
|
||||
|
||||
def groupnorm_mm_factory(video_length: int):
|
||||
def groupnorm_mm_forward(self, input: Tensor) -> Tensor:
|
||||
# axes_factor normalizes batch based on total conds and unconds passed in batch;
|
||||
# the conds and unconds per batch can change based on VRAM optimizations that may kick in
|
||||
axes_factor = input.size(0) // video_length
|
||||
|
||||
input = rearrange(input, "(b f) c h w -> b c f h w", b=axes_factor)
|
||||
input = group_norm(input, self.num_groups, self.weight, self.bias, self.eps)
|
||||
input = rearrange(input, "b c f h w -> (b f) c h w", b=axes_factor)
|
||||
return input
|
||||
|
||||
return groupnorm_mm_forward
|
||||
|
||||
|
||||
orig_forward_timestep_embed = openaimodel.forward_timestep_embed
|
||||
orig_maximum_batch_area = model_management.maximum_batch_area
|
||||
orig_groupnorm_forward = torch.nn.GroupNorm.forward
|
||||
|
||||
|
||||
def inject_motion_module_to_unet_legacy(unet, motion_module: MotionWrapper):
|
||||
for mm_idx, unet_idx in enumerate([1, 2, 4, 5, 7, 8, 10, 11]):
|
||||
mm_idx0, mm_idx1 = mm_idx // 2, mm_idx % 2
|
||||
unet.input_blocks[unet_idx].append(motion_module.down_blocks[mm_idx0].motion_modules[mm_idx1])
|
||||
|
||||
for unet_idx in range(12):
|
||||
mm_idx0, mm_idx1 = unet_idx // 3, unet_idx % 3
|
||||
if unet_idx % 2 == 2:
|
||||
unet.output_blocks[unet_idx].insert(-1, motion_module.up_blocks[mm_idx0].motion_modules[mm_idx1])
|
||||
else:
|
||||
unet.output_blocks[unet_idx].append(motion_module.up_blocks[mm_idx0].motion_modules[mm_idx1])
|
||||
if motion_module.is_v2:
|
||||
unet.middle_block.insert(-1, motion_module.mid_block.motion_modules[0])
|
||||
|
||||
unet.motion_module = motion_module
|
||||
|
||||
|
||||
def eject_motion_module_from_unet_legacy(unet):
|
||||
for unet_idx in [1, 2, 4, 5, 7, 8, 10, 11]:
|
||||
unet.input_blocks[unet_idx].pop(-1)
|
||||
|
||||
for unet_idx in range(12):
|
||||
if unet_idx % 2 == 2:
|
||||
unet.output_blocks[unet_idx].pop(-2)
|
||||
else:
|
||||
unet.output_blocks[unet_idx].pop(-1)
|
||||
|
||||
if unet.motion_module.is_v2:
|
||||
unet.middle_block.pop(-2)
|
||||
|
||||
del unet.motion_module
|
||||
|
||||
|
||||
def inject_motion_module_to_unet(unet, motion_module: MotionWrapper):
|
||||
for mm_idx, unet_idx in enumerate([1, 2, 4, 5, 7, 8, 10, 11]):
|
||||
mm_idx0, mm_idx1 = mm_idx // 2, mm_idx % 2
|
||||
unet.input_blocks[unet_idx].append(motion_module.down_blocks[mm_idx0].motion_modules[mm_idx1])
|
||||
|
||||
for unet_idx in range(12):
|
||||
mm_idx0, mm_idx1 = unet_idx // 3, unet_idx % 3
|
||||
if unet_idx % 3 == 2 and unet_idx != 11:
|
||||
unet.output_blocks[unet_idx].insert(-1, motion_module.up_blocks[mm_idx0].motion_modules[mm_idx1])
|
||||
else:
|
||||
unet.output_blocks[unet_idx].append(motion_module.up_blocks[mm_idx0].motion_modules[mm_idx1])
|
||||
if motion_module.is_v2:
|
||||
unet.middle_block.insert(-1, motion_module.mid_block.motion_modules[0])
|
||||
|
||||
unet.motion_module = motion_module
|
||||
|
||||
|
||||
def eject_motion_module_from_unet(unet):
|
||||
for unet_idx in [1, 2, 4, 5, 7, 8, 10, 11]:
|
||||
unet.input_blocks[unet_idx].pop(-1)
|
||||
|
||||
for unet_idx in range(12):
|
||||
if unet_idx % 3 == 2 and unet_idx != 11:
|
||||
unet.output_blocks[unet_idx].pop(-2)
|
||||
else:
|
||||
unet.output_blocks[unet_idx].pop(-1)
|
||||
|
||||
if unet.motion_module.is_v2:
|
||||
unet.middle_block.pop(-2)
|
||||
|
||||
del unet.motion_module
|
||||
|
||||
|
||||
injectors = {
|
||||
"legacy": inject_motion_module_to_unet_legacy,
|
||||
"default": inject_motion_module_to_unet,
|
||||
}
|
||||
|
||||
ejectors = {
|
||||
"legacy": eject_motion_module_from_unet_legacy,
|
||||
"default": eject_motion_module_from_unet,
|
||||
}
|
||||
|
||||
|
||||
class AnimateDiffSlidingWindowOptions:
|
||||
@classmethod
|
||||
def INPUT_TYPES(s):
|
||||
return {
|
||||
"required": {
|
||||
"context_length": ("INT", {"default": SLIDING_CONTEXT_LENGTH, "min": 2, "max": 32}),
|
||||
"context_stride": ("INT", {"default": 1, "min": 1, "max": 32}),
|
||||
"context_overlap": ("INT", {"default": 4, "min": 0, "max": 32}),
|
||||
"context_schedule": (ContextSchedules.CONTEXT_SCHEDULE_LIST,),
|
||||
"closed_loop": ("BOOLEAN", {"default": False}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("SLIDING_WINDOW_OPTS",)
|
||||
FUNCTION = "init_options"
|
||||
CATEGORY = "Animate Diff"
|
||||
|
||||
def init_options(self, context_length, context_stride, context_overlap, context_schedule, closed_loop):
|
||||
ctx = SlidingContext(
|
||||
context_length=context_length,
|
||||
context_stride=context_stride,
|
||||
context_overlap=context_overlap,
|
||||
context_schedule=context_schedule,
|
||||
closed_loop=closed_loop,
|
||||
)
|
||||
|
||||
return (ctx,)
|
||||
|
||||
|
||||
class AnimateDiffSampler(KSampler):
|
||||
@classmethod
|
||||
def INPUT_TYPES(s):
|
||||
inputs = {
|
||||
"required": {
|
||||
"motion_module": ("MOTION_MODULE",),
|
||||
"inject_method": (["default", "legacy"],),
|
||||
"frame_number": (
|
||||
"INT",
|
||||
{"default": 16, "min": 2, "max": 10000, "step": 1},
|
||||
),
|
||||
}
|
||||
}
|
||||
inputs["required"].update(KSampler.INPUT_TYPES()["required"])
|
||||
inputs["optional"] = {"sliding_window_opts": ("SLIDING_WINDOW_OPTS",)}
|
||||
return inputs
|
||||
|
||||
FUNCTION = "animatediff_sample"
|
||||
CATEGORY = "Animate Diff"
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.prev_beta = None
|
||||
self.prev_linear_start = None
|
||||
self.prev_linear_end = None
|
||||
|
||||
def override_beta_schedule(self, model: BaseModel):
|
||||
self.prev_beta = model.get_buffer("betas").cpu().clone().detach()
|
||||
self.prev_linear_start = model.linear_start
|
||||
self.prev_linear_end = model.linear_end
|
||||
model.register_schedule(
|
||||
given_betas=None,
|
||||
beta_schedule="sqrt_linear",
|
||||
timesteps=1000,
|
||||
linear_start=0.00085,
|
||||
linear_end=0.012,
|
||||
cosine_s=8e-3,
|
||||
)
|
||||
|
||||
def restore_beta_schedule(self, model: BaseModel):
|
||||
model.register_schedule(
|
||||
given_betas=self.prev_beta,
|
||||
linear_start=self.prev_linear_start,
|
||||
linear_end=self.prev_linear_end,
|
||||
)
|
||||
self.prev_beta = None
|
||||
self.prev_linear_start = None
|
||||
self.prev_linear_end = None
|
||||
|
||||
def inject_motion_module(self, model, motion_module: MotionWrapper, inject_method: str, frame_number: int):
|
||||
model = model.clone()
|
||||
unet = model.model.diffusion_model
|
||||
|
||||
logger.info(f"Injecting motion module with method {inject_method}.")
|
||||
motion_module.set_video_length(frame_number)
|
||||
injectors[inject_method](unet, motion_module)
|
||||
self.override_beta_schedule(model.model)
|
||||
openaimodel.forward_timestep_embed = forward_timestep_embed
|
||||
if not motion_module.is_v2:
|
||||
logger.info(f"Hacking GroupNorm.forward function.")
|
||||
torch.nn.GroupNorm.forward = groupnorm_mm_factory(frame_number)
|
||||
|
||||
return model
|
||||
|
||||
def inject_sliding_sampler(self, video_length, sliding_window_opts: SlidingContext = None):
|
||||
ctx = sliding_window_opts.copy() if sliding_window_opts else SlidingContext()
|
||||
ctx.video_length = video_length
|
||||
|
||||
inject_sampling_function(ctx)
|
||||
|
||||
def eject_motion_module(self, model, inject_method):
|
||||
unet = model.model.diffusion_model
|
||||
|
||||
self.restore_beta_schedule(model.model)
|
||||
openaimodel.forward_timestep_embed = orig_forward_timestep_embed
|
||||
if not unet.motion_module.is_v2:
|
||||
logger.info(f"Restore GroupNorm.forward function.")
|
||||
torch.nn.GroupNorm.forward = orig_groupnorm_forward
|
||||
|
||||
logger.info(f"Ejecting motion module with method {inject_method}.")
|
||||
ejectors[inject_method](unet)
|
||||
|
||||
def eject_sliding_sampler(self):
|
||||
eject_sampling_function()
|
||||
|
||||
def animatediff_sample(
|
||||
self,
|
||||
motion_module,
|
||||
inject_method,
|
||||
frame_number,
|
||||
model,
|
||||
seed,
|
||||
steps,
|
||||
cfg,
|
||||
sampler_name,
|
||||
scheduler,
|
||||
positive,
|
||||
negative,
|
||||
latent_image,
|
||||
denoise=1.0,
|
||||
sliding_window_opts: SlidingContext = None,
|
||||
**kwargs,
|
||||
):
|
||||
# init latents
|
||||
samples = latent_image["samples"]
|
||||
init_frames = len(samples)
|
||||
if init_frames < frame_number:
|
||||
# TODO: apply different noise to each frame
|
||||
last_frame = samples[-1].clone().cpu().unsqueeze(0)
|
||||
repeated_last_frames = last_frame.repeat(frame_number - init_frames, 1, 1, 1)
|
||||
samples = torch.cat((samples, repeated_last_frames), dim=0)
|
||||
|
||||
latent_image = {"samples": samples}
|
||||
|
||||
# validate context_length
|
||||
context_length = sliding_window_opts.context_length if sliding_window_opts else SLIDING_CONTEXT_LENGTH
|
||||
is_sliding = frame_number > context_length
|
||||
video_length = context_length if is_sliding else frame_number
|
||||
|
||||
if video_length > motion_module.encoding_max_len:
|
||||
error = f'{"context_length" if is_sliding else "frame_number"} = {video_length}'
|
||||
raise ValueError(
|
||||
f"AnimateDiff model {motion_module.mm_type} has upper limit of {motion_module.encoding_max_len} frames, but received {error}."
|
||||
)
|
||||
|
||||
# inject motion module
|
||||
model = self.inject_motion_module(model, motion_module, inject_method, video_length)
|
||||
|
||||
# inject sliding sampler
|
||||
if is_sliding:
|
||||
self.inject_sliding_sampler(frame_number, sliding_window_opts=sliding_window_opts)
|
||||
|
||||
try:
|
||||
return super().sample(
|
||||
model,
|
||||
seed,
|
||||
steps,
|
||||
cfg,
|
||||
sampler_name,
|
||||
scheduler,
|
||||
positive,
|
||||
negative,
|
||||
latent_image,
|
||||
denoise=denoise,
|
||||
**kwargs,
|
||||
)
|
||||
except:
|
||||
raise
|
||||
finally:
|
||||
# eject motion module
|
||||
self.eject_motion_module(model, inject_method)
|
||||
|
||||
# eject sliding sampler
|
||||
if is_sliding:
|
||||
self.eject_sliding_sampler()
|
||||
@@ -0,0 +1,487 @@
|
||||
import torch
|
||||
from torch import Tensor
|
||||
import math
|
||||
|
||||
import comfy.utils
|
||||
import comfy.sample
|
||||
import comfy.samplers as comfy_samplers
|
||||
import comfy.model_management as model_management
|
||||
from comfy.controlnet import ControlBase
|
||||
from comfy.model_patcher import ModelPatcher
|
||||
|
||||
from .logger import logger
|
||||
from .sliding_schedule import get_context_scheduler, ContextSchedules
|
||||
|
||||
|
||||
orig_comfy_sample = comfy.sample.sample
|
||||
orig_sampling_function = comfy_samplers.sampling_function
|
||||
|
||||
|
||||
class SlidingContext:
|
||||
def __init__(
|
||||
self,
|
||||
context_length=16,
|
||||
context_stride=1,
|
||||
context_overlap=4,
|
||||
context_schedule=ContextSchedules.UNIFORM,
|
||||
closed_loop=False,
|
||||
video_length=0,
|
||||
current_step=0,
|
||||
total_steps=0,
|
||||
):
|
||||
self.context_length = context_length
|
||||
self.context_stride = context_stride
|
||||
self.context_overlap = context_overlap
|
||||
self.context_schedule = context_schedule
|
||||
self.closed_loop = closed_loop
|
||||
self.video_length = video_length
|
||||
self.current_step = current_step
|
||||
self.total_steps = total_steps
|
||||
|
||||
def copy(self):
|
||||
return SlidingContext(
|
||||
context_length=self.context_length,
|
||||
context_stride=self.context_stride,
|
||||
context_overlap=self.context_overlap,
|
||||
context_schedule=self.context_schedule,
|
||||
closed_loop=self.closed_loop,
|
||||
video_length=self.video_length,
|
||||
current_step=self.current_step,
|
||||
total_steps=self.total_steps,
|
||||
)
|
||||
|
||||
|
||||
def __sliding_sample_factory(ctx: SlidingContext):
|
||||
logger.info(f"Injecting sliding context sampling function.")
|
||||
logger.info(f"Video length: {ctx.video_length}")
|
||||
logger.info(f"Context length: {ctx.context_length}")
|
||||
logger.info(f"Context schedule: {ctx.context_schedule}")
|
||||
|
||||
context_scheduler = get_context_scheduler(ctx.context_schedule)
|
||||
|
||||
def sample(model: ModelPatcher, *args, **kwargs):
|
||||
orig_callback = kwargs.pop("callback", None)
|
||||
start_step = kwargs.get("start_step") or 0
|
||||
|
||||
# adjust progressbar to account for context frames
|
||||
def callback(step, x0, x, total_steps):
|
||||
if orig_callback:
|
||||
orig_callback(step, x0, x, total_steps)
|
||||
|
||||
ctx.current_step = start_step + step + 1
|
||||
|
||||
try:
|
||||
return orig_comfy_sample(model, *args, **kwargs, callback=callback)
|
||||
except RuntimeError as e:
|
||||
if str(e).startswith("CUDA error: invalid configuration argument"):
|
||||
raise RuntimeError(
|
||||
f"An xformers bug was encountered in AnimateDiff - to run your workflow, \
|
||||
disable xformers in ComfyUI using '--disable-xformers' startup argument."
|
||||
)
|
||||
raise
|
||||
|
||||
def sampling_function(
|
||||
model_function, x, timestep, uncond, cond, cond_scale, cond_concat=None, model_options={}, seed=None
|
||||
):
|
||||
def get_area_and_mult(cond, x_in, cond_concat_in, timestep_in):
|
||||
area = (x_in.shape[2], x_in.shape[3], 0, 0)
|
||||
strength = 1.0
|
||||
if "timestep_start" in cond[1]:
|
||||
timestep_start = cond[1]["timestep_start"]
|
||||
if timestep_in[0] > timestep_start:
|
||||
return None
|
||||
if "timestep_end" in cond[1]:
|
||||
timestep_end = cond[1]["timestep_end"]
|
||||
if timestep_in[0] < timestep_end:
|
||||
return None
|
||||
if "area" in cond[1]:
|
||||
area = cond[1]["area"]
|
||||
if "strength" in cond[1]:
|
||||
strength = cond[1]["strength"]
|
||||
|
||||
adm_cond = None
|
||||
if "adm_encoded" in cond[1]:
|
||||
adm_cond = cond[1]["adm_encoded"]
|
||||
|
||||
input_x = x_in[:, :, area[2] : area[0] + area[2], area[3] : area[1] + area[3]]
|
||||
if "mask" in cond[1]:
|
||||
# Scale the mask to the size of the input
|
||||
# The mask should have been resized as we began the sampling process
|
||||
mask_strength = 1.0
|
||||
if "mask_strength" in cond[1]:
|
||||
mask_strength = cond[1]["mask_strength"]
|
||||
mask = cond[1]["mask"]
|
||||
assert mask.shape[1] == x_in.shape[2]
|
||||
assert mask.shape[2] == x_in.shape[3]
|
||||
mask = mask[:, area[2] : area[0] + area[2], area[3] : area[1] + area[3]] * mask_strength
|
||||
mask = mask.unsqueeze(1).repeat(input_x.shape[0] // mask.shape[0], input_x.shape[1], 1, 1)
|
||||
else:
|
||||
mask = torch.ones_like(input_x)
|
||||
mult = mask * strength
|
||||
|
||||
if "mask" not in cond[1]:
|
||||
rr = 8
|
||||
if area[2] != 0:
|
||||
for t in range(rr):
|
||||
mult[:, :, t : 1 + t, :] *= (1.0 / rr) * (t + 1)
|
||||
if (area[0] + area[2]) < x_in.shape[2]:
|
||||
for t in range(rr):
|
||||
mult[:, :, area[0] - 1 - t : area[0] - t, :] *= (1.0 / rr) * (t + 1)
|
||||
if area[3] != 0:
|
||||
for t in range(rr):
|
||||
mult[:, :, :, t : 1 + t] *= (1.0 / rr) * (t + 1)
|
||||
if (area[1] + area[3]) < x_in.shape[3]:
|
||||
for t in range(rr):
|
||||
mult[:, :, :, area[1] - 1 - t : area[1] - t] *= (1.0 / rr) * (t + 1)
|
||||
|
||||
conditionning = {}
|
||||
conditionning["c_crossattn"] = cond[0]
|
||||
if cond_concat_in is not None and len(cond_concat_in) > 0:
|
||||
cropped = []
|
||||
for x in cond_concat_in:
|
||||
cr = x[:, :, area[2] : area[0] + area[2], area[3] : area[1] + area[3]]
|
||||
cropped.append(cr)
|
||||
conditionning["c_concat"] = torch.cat(cropped, dim=1)
|
||||
|
||||
if adm_cond is not None:
|
||||
conditionning["c_adm"] = adm_cond
|
||||
|
||||
control = None
|
||||
if "control" in cond[1]:
|
||||
control = cond[1]["control"]
|
||||
|
||||
patches = None
|
||||
if "gligen" in cond[1]:
|
||||
gligen = cond[1]["gligen"]
|
||||
patches = {}
|
||||
gligen_type = gligen[0]
|
||||
gligen_model = gligen[1]
|
||||
if gligen_type == "position":
|
||||
gligen_patch = gligen_model.model.set_position(input_x.shape, gligen[2], input_x.device)
|
||||
else:
|
||||
gligen_patch = gligen_model.model.set_empty(input_x.shape, input_x.device)
|
||||
|
||||
patches["middle_patch"] = [gligen_patch]
|
||||
|
||||
return (input_x, mult, conditionning, area, control, patches)
|
||||
|
||||
def cond_equal_size(c1, c2):
|
||||
if c1 is c2:
|
||||
return True
|
||||
if c1.keys() != c2.keys():
|
||||
return False
|
||||
if "c_crossattn" in c1:
|
||||
s1 = c1["c_crossattn"].shape
|
||||
s2 = c2["c_crossattn"].shape
|
||||
if s1 != s2:
|
||||
if s1[0] != s2[0] or s1[2] != s2[2]: # these 2 cases should not happen
|
||||
return False
|
||||
|
||||
mult_min = comfy_samplers.lcm(s1[1], s2[1])
|
||||
diff = mult_min // min(s1[1], s2[1])
|
||||
if (
|
||||
diff > 4
|
||||
): # arbitrary limit on the padding because it's probably going to impact performance negatively if it's too much
|
||||
return False
|
||||
if "c_concat" in c1:
|
||||
if c1["c_concat"].shape != c2["c_concat"].shape:
|
||||
return False
|
||||
if "c_adm" in c1:
|
||||
if c1["c_adm"].shape != c2["c_adm"].shape:
|
||||
return False
|
||||
return True
|
||||
|
||||
def can_concat_cond(c1, c2):
|
||||
if c1[0].shape != c2[0].shape:
|
||||
return False
|
||||
|
||||
# control
|
||||
if (c1[4] is None) != (c2[4] is None):
|
||||
return False
|
||||
if c1[4] is not None:
|
||||
if c1[4] is not c2[4]:
|
||||
return False
|
||||
|
||||
# patches
|
||||
if (c1[5] is None) != (c2[5] is None):
|
||||
return False
|
||||
if c1[5] is not None:
|
||||
if c1[5] is not c2[5]:
|
||||
return False
|
||||
|
||||
return cond_equal_size(c1[2], c2[2])
|
||||
|
||||
def cond_cat(c_list):
|
||||
c_crossattn = []
|
||||
c_concat = []
|
||||
c_adm = []
|
||||
crossattn_max_len = 0
|
||||
for x in c_list:
|
||||
if "c_crossattn" in x:
|
||||
c = x["c_crossattn"]
|
||||
if crossattn_max_len == 0:
|
||||
crossattn_max_len = c.shape[1]
|
||||
else:
|
||||
crossattn_max_len = comfy_samplers.lcm(crossattn_max_len, c.shape[1])
|
||||
c_crossattn.append(c)
|
||||
if "c_concat" in x:
|
||||
c_concat.append(x["c_concat"])
|
||||
if "c_adm" in x:
|
||||
c_adm.append(x["c_adm"])
|
||||
out = {}
|
||||
c_crossattn_out = []
|
||||
for c in c_crossattn:
|
||||
if c.shape[1] < crossattn_max_len:
|
||||
c = c.repeat(1, crossattn_max_len // c.shape[1], 1) # padding with repeat doesn't change result
|
||||
c_crossattn_out.append(c)
|
||||
|
||||
if len(c_crossattn_out) > 0:
|
||||
out["c_crossattn"] = torch.cat(c_crossattn_out)
|
||||
if len(c_concat) > 0:
|
||||
out["c_concat"] = torch.cat(c_concat)
|
||||
if len(c_adm) > 0:
|
||||
out["c_adm"] = torch.cat(c_adm)
|
||||
return out
|
||||
|
||||
def calc_cond_uncond_batch(
|
||||
model_function, cond, uncond, x_in, timestep, max_total_area, cond_concat_in, model_options
|
||||
):
|
||||
out_cond = torch.zeros_like(x_in)
|
||||
out_count = torch.ones_like(x_in) / 100000.0
|
||||
|
||||
out_uncond = torch.zeros_like(x_in)
|
||||
out_uncond_count = torch.ones_like(x_in) / 100000.0
|
||||
|
||||
COND = 0
|
||||
UNCOND = 1
|
||||
|
||||
to_run = []
|
||||
for x in cond:
|
||||
p = get_area_and_mult(x, x_in, cond_concat_in, timestep)
|
||||
if p is None:
|
||||
continue
|
||||
|
||||
to_run += [(p, COND)]
|
||||
if uncond is not None:
|
||||
for x in uncond:
|
||||
p = get_area_and_mult(x, x_in, cond_concat_in, timestep)
|
||||
if p is None:
|
||||
continue
|
||||
|
||||
to_run += [(p, UNCOND)]
|
||||
|
||||
while len(to_run) > 0:
|
||||
first = to_run[0]
|
||||
first_shape = first[0][0].shape
|
||||
to_batch_temp = []
|
||||
for x in range(len(to_run)):
|
||||
if can_concat_cond(to_run[x][0], first[0]):
|
||||
to_batch_temp += [x]
|
||||
|
||||
to_batch_temp.reverse()
|
||||
to_batch = to_batch_temp[:1]
|
||||
|
||||
for i in range(1, len(to_batch_temp) + 1):
|
||||
batch_amount = to_batch_temp[: len(to_batch_temp) // i]
|
||||
if len(batch_amount) * first_shape[0] * first_shape[2] * first_shape[3] < max_total_area:
|
||||
to_batch = batch_amount
|
||||
break
|
||||
|
||||
input_x = []
|
||||
mult = []
|
||||
c = []
|
||||
cond_or_uncond = []
|
||||
area = []
|
||||
control = None
|
||||
patches = None
|
||||
for x in to_batch:
|
||||
o = to_run.pop(x)
|
||||
p = o[0]
|
||||
input_x += [p[0]]
|
||||
mult += [p[1]]
|
||||
c += [p[2]]
|
||||
area += [p[3]]
|
||||
cond_or_uncond += [o[1]]
|
||||
control = p[4]
|
||||
patches = p[5]
|
||||
|
||||
batch_chunks = len(cond_or_uncond)
|
||||
input_x = torch.cat(input_x)
|
||||
c = cond_cat(c)
|
||||
timestep_ = torch.cat([timestep] * batch_chunks)
|
||||
|
||||
if control is not None:
|
||||
c["control"] = control.get_control(input_x, timestep_, c, len(cond_or_uncond))
|
||||
|
||||
transformer_options = {}
|
||||
if "transformer_options" in model_options:
|
||||
transformer_options = model_options["transformer_options"].copy()
|
||||
|
||||
if patches is not None:
|
||||
if "patches" in transformer_options:
|
||||
cur_patches = transformer_options["patches"].copy()
|
||||
for p in patches:
|
||||
if p in cur_patches:
|
||||
cur_patches[p] = cur_patches[p] + patches[p]
|
||||
else:
|
||||
cur_patches[p] = patches[p]
|
||||
else:
|
||||
transformer_options["patches"] = patches
|
||||
|
||||
transformer_options["cond_or_uncond"] = cond_or_uncond[:]
|
||||
c["transformer_options"] = transformer_options
|
||||
|
||||
if "model_function_wrapper" in model_options:
|
||||
output = model_options["model_function_wrapper"](
|
||||
model_function,
|
||||
{"input": input_x, "timestep": timestep_, "c": c, "cond_or_uncond": cond_or_uncond},
|
||||
).chunk(batch_chunks)
|
||||
else:
|
||||
output = model_function(input_x, timestep_, **c).chunk(batch_chunks)
|
||||
del input_x
|
||||
|
||||
for o in range(batch_chunks):
|
||||
if cond_or_uncond[o] == COND:
|
||||
out_cond[:, :, area[o][2] : area[o][0] + area[o][2], area[o][3] : area[o][1] + area[o][3]] += (
|
||||
output[o] * mult[o]
|
||||
)
|
||||
out_count[
|
||||
:, :, area[o][2] : area[o][0] + area[o][2], area[o][3] : area[o][1] + area[o][3]
|
||||
] += mult[o]
|
||||
else:
|
||||
out_uncond[
|
||||
:, :, area[o][2] : area[o][0] + area[o][2], area[o][3] : area[o][1] + area[o][3]
|
||||
] += (output[o] * mult[o])
|
||||
out_uncond_count[
|
||||
:, :, area[o][2] : area[o][0] + area[o][2], area[o][3] : area[o][1] + area[o][3]
|
||||
] += mult[o]
|
||||
del mult
|
||||
|
||||
out_cond /= out_count
|
||||
del out_count
|
||||
out_uncond /= out_uncond_count
|
||||
del out_uncond_count
|
||||
|
||||
return out_cond, out_uncond
|
||||
|
||||
# sliding_calc_cond_uncond_batch inspired by ashen's initial hack for 16-frame sliding context:
|
||||
# https://github.com/comfyanonymous/ComfyUI/compare/master...ashen-sensored:ComfyUI:master
|
||||
def sliding_calc_cond_uncond_batch(
|
||||
model_function, cond, uncond, x_in, timestep, max_total_area, cond_concat_in, model_options
|
||||
):
|
||||
# figure out how input is split
|
||||
axes_factor = x.size(0) // ctx.video_length
|
||||
|
||||
# prepare final cond, uncond, and out_count
|
||||
cond_final = torch.zeros_like(x)
|
||||
uncond_final = torch.zeros_like(x)
|
||||
out_count_final = torch.zeros((x.shape[0], 1, 1, 1), device=x.device)
|
||||
|
||||
def prepare_control_objects(control: ControlBase, full_idxs: list[int]):
|
||||
if control.previous_controlnet is not None:
|
||||
prepare_control_objects(control.previous_controlnet, full_idxs)
|
||||
control.sub_idxs = full_idxs
|
||||
control.full_latent_length = ctx.video_length
|
||||
control.context_length = ctx.context_length
|
||||
|
||||
def get_resized_cond(cond_in, full_idxs) -> list:
|
||||
# reuse or resize cond items to match context requirements
|
||||
resized_cond = []
|
||||
# cond object is a list containing a list - outer list is irrelevant, so just loop through it
|
||||
for actual_cond in cond_in:
|
||||
resized_actual_cond = []
|
||||
# now we are in the inner list - index 0 is tensor, index 1 is dictionary
|
||||
for cond_idx, cond_item in enumerate(actual_cond):
|
||||
if isinstance(cond_item, Tensor):
|
||||
# check that tensor is the expected length - x.size(0)
|
||||
if cond_item.size(0) == x.size(0):
|
||||
pass
|
||||
# if so, it's subsetting time - tell controls the expected indeces so they can handle them
|
||||
actual_cond_item = cond_item[full_idxs]
|
||||
resized_actual_cond.append(actual_cond_item)
|
||||
else:
|
||||
resized_actual_cond.append(cond_item)
|
||||
elif isinstance(cond_item, dict):
|
||||
# when in dictionary, look for control
|
||||
if "control" in cond_item:
|
||||
control_item = cond_item["control"]
|
||||
if hasattr(control_item, "sub_idxs"):
|
||||
prepare_control_objects(control_item, full_idxs)
|
||||
else:
|
||||
raise ValueError(
|
||||
f"Control type {type(control_item).__name__} may not support required features for sliding context window; use Control objects from Kosinkadink/Advanced-ControlNet nodes."
|
||||
)
|
||||
resized_actual_cond.append(cond_item)
|
||||
else:
|
||||
resized_actual_cond.append(cond_item)
|
||||
resized_cond.append(resized_actual_cond)
|
||||
return resized_cond
|
||||
|
||||
# perform calc_cond_uncond_batch per context window
|
||||
for ctx_idxs in context_scheduler(
|
||||
ctx.current_step,
|
||||
ctx.total_steps,
|
||||
ctx.video_length,
|
||||
ctx.context_length,
|
||||
ctx.context_stride,
|
||||
ctx.context_overlap,
|
||||
ctx.closed_loop,
|
||||
):
|
||||
# account for all portions of input frames
|
||||
full_idxs = []
|
||||
for n in range(axes_factor):
|
||||
for ind in ctx_idxs:
|
||||
full_idxs.append((ctx.video_length * n) + ind)
|
||||
# get subsections of x, timestep, cond, uncond, cond_concat
|
||||
sub_x = x[full_idxs]
|
||||
sub_timestep = timestep[full_idxs]
|
||||
sub_cond = get_resized_cond(cond, full_idxs) if cond is not None else None
|
||||
sub_uncond = get_resized_cond(uncond, full_idxs) if uncond is not None else None
|
||||
sub_cond_concat = get_resized_cond(cond_concat, full_idxs) if cond_concat is not None else None
|
||||
|
||||
sub_cond_out, sub_uncond_out = calc_cond_uncond_batch(
|
||||
model_function,
|
||||
sub_cond,
|
||||
sub_uncond,
|
||||
sub_x,
|
||||
sub_timestep,
|
||||
max_total_area,
|
||||
sub_cond_concat,
|
||||
model_options,
|
||||
)
|
||||
|
||||
cond_final[full_idxs] += sub_cond_out
|
||||
uncond_final[full_idxs] += sub_uncond_out
|
||||
out_count_final[full_idxs] += 1 # increment which indeces were used
|
||||
|
||||
# normalize cond and uncond via division by context usage counts
|
||||
cond_final /= out_count_final
|
||||
uncond_final /= out_count_final
|
||||
return cond_final, uncond_final
|
||||
|
||||
max_total_area = model_management.maximum_batch_area()
|
||||
if math.isclose(cond_scale, 1.0):
|
||||
uncond = None
|
||||
|
||||
cond, uncond = sliding_calc_cond_uncond_batch(
|
||||
model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options
|
||||
)
|
||||
|
||||
if "sampler_cfg_function" in model_options:
|
||||
args = {"cond": cond, "uncond": uncond, "cond_scale": cond_scale, "timestep": timestep}
|
||||
return model_options["sampler_cfg_function"](args)
|
||||
else:
|
||||
return uncond + (cond - uncond) * cond_scale
|
||||
|
||||
return (sample, sampling_function)
|
||||
|
||||
|
||||
def inject_sampling_function(ctx: SlidingContext):
|
||||
(sample, sampling_function) = __sliding_sample_factory(ctx)
|
||||
comfy.sample.sample = sample
|
||||
comfy_samplers.sampling_function = sampling_function
|
||||
|
||||
|
||||
def eject_sampling_function():
|
||||
comfy.sample.sample = orig_comfy_sample
|
||||
comfy_samplers.sampling_function = orig_sampling_function
|
||||
@@ -0,0 +1,155 @@
|
||||
# from https://github.com/neggles/animatediff-cli/blob/main/src/animatediff/pipelines/context.py
|
||||
from typing import Callable, Optional
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
class ContextSchedules:
|
||||
UNIFORM = "uniform"
|
||||
UNIFORM_CONSTANT = "uniform_constant"
|
||||
UNIFORM_V2 = "uniform v2"
|
||||
|
||||
CONTEXT_SCHEDULE_LIST = [UNIFORM, UNIFORM_V2]
|
||||
|
||||
|
||||
# Returns fraction that has denominator that is a power of 2
|
||||
def ordered_halving(val, print_final=False):
|
||||
# get binary value, padded with 0s for 64 bits
|
||||
bin_str = f"{val:064b}"
|
||||
# flip binary value, padding included
|
||||
bin_flip = bin_str[::-1]
|
||||
# convert binary to int
|
||||
as_int = int(bin_flip, 2)
|
||||
# divide by 1 << 64, equivalent to 2**64, or 18446744073709551616,
|
||||
# or b10000000000000000000000000000000000000000000000000000000000000000 (1 with 64 zero's)
|
||||
final = as_int / (1 << 64)
|
||||
if print_final:
|
||||
print(f"$$$$ final: {final}")
|
||||
return final
|
||||
|
||||
|
||||
# Generator that returns lists of latent indeces to diffuse on
|
||||
def uniform(
|
||||
step: int = ...,
|
||||
num_steps: Optional[int] = None,
|
||||
num_frames: int = ...,
|
||||
context_size: Optional[int] = None,
|
||||
context_stride: int = 3,
|
||||
context_overlap: int = 4,
|
||||
closed_loop: bool = True,
|
||||
print_final: bool = False,
|
||||
):
|
||||
if num_frames <= context_size:
|
||||
yield list(range(num_frames))
|
||||
return
|
||||
|
||||
context_stride = min(context_stride, int(np.ceil(np.log2(num_frames / context_size))) + 1)
|
||||
|
||||
for context_step in 1 << np.arange(context_stride):
|
||||
pad = int(round(num_frames * ordered_halving(step, print_final)))
|
||||
for j in range(
|
||||
int(ordered_halving(step) * context_step) + pad,
|
||||
num_frames + pad + (0 if closed_loop else -context_overlap),
|
||||
(context_size * context_step - context_overlap),
|
||||
):
|
||||
yield [e % num_frames for e in range(j, j + context_size * context_step, context_step)]
|
||||
|
||||
|
||||
def uniform_v2(
|
||||
step: int = ...,
|
||||
num_steps: Optional[int] = None,
|
||||
num_frames: int = ...,
|
||||
context_size: Optional[int] = None,
|
||||
context_stride: int = 3,
|
||||
context_overlap: int = 4,
|
||||
closed_loop: bool = True,
|
||||
print_final: bool = False,
|
||||
):
|
||||
if num_frames <= context_size:
|
||||
yield list(range(num_frames))
|
||||
return
|
||||
|
||||
context_stride = min(context_stride, int(np.ceil(np.log2(num_frames / context_size))) + 1)
|
||||
|
||||
pad = int(round(num_frames * ordered_halving(step, print_final)))
|
||||
for context_step in 1 << np.arange(context_stride):
|
||||
j_initial = int(ordered_halving(step) * context_step) + pad
|
||||
for j in range(
|
||||
j_initial,
|
||||
num_frames + pad - context_overlap,
|
||||
(context_size * context_step - context_overlap),
|
||||
):
|
||||
if context_size * context_step > num_frames:
|
||||
# On the final context_step,
|
||||
# ensure no frame appears in the window twice
|
||||
yield [e % num_frames for e in range(j, j + num_frames, context_step)]
|
||||
continue
|
||||
j = j % num_frames
|
||||
if j > (j + context_size * context_step) % num_frames and not closed_loop:
|
||||
yield [e for e in range(j, num_frames, context_step)]
|
||||
j_stop = (j + context_size * context_step) % num_frames
|
||||
# When ((num_frames % (context_size - context_overlap)+context_overlap) % context_size != 0,
|
||||
# This can cause 'superflous' runs where all frames in
|
||||
# a context window have already been processed during
|
||||
# the first context window of this stride and step.
|
||||
# While the following commented if should prevent this,
|
||||
# I believe leaving it in is more correct as it maintains
|
||||
# the total conditional passes per frame over a large total steps
|
||||
# if j_stop > context_overlap:
|
||||
yield [e for e in range(0, j_stop, context_step)]
|
||||
continue
|
||||
yield [e % num_frames for e in range(j, j + context_size * context_step, context_step)]
|
||||
|
||||
|
||||
def uniform_constant(
|
||||
step: int = ...,
|
||||
num_steps: Optional[int] = None,
|
||||
num_frames: int = ...,
|
||||
context_size: Optional[int] = None,
|
||||
context_stride: int = 3,
|
||||
context_overlap: int = 4,
|
||||
closed_loop: bool = True,
|
||||
print_final: bool = False,
|
||||
):
|
||||
if num_frames <= context_size:
|
||||
yield list(range(num_frames))
|
||||
return
|
||||
|
||||
context_stride = min(context_stride, int(np.ceil(np.log2(num_frames / context_size))) + 1)
|
||||
|
||||
# want to avoid loops that connect end to beginning
|
||||
|
||||
for context_step in 1 << np.arange(context_stride):
|
||||
pad = int(round(num_frames * ordered_halving(step, print_final)))
|
||||
for j in range(
|
||||
int(ordered_halving(step) * context_step) + pad,
|
||||
num_frames + pad + (0 if closed_loop else -context_overlap),
|
||||
(context_size * context_step - context_overlap),
|
||||
):
|
||||
skip_this_window = False
|
||||
prev_val = -1
|
||||
to_yield = []
|
||||
for e in range(j, j + context_size * context_step, context_step):
|
||||
e = e % num_frames
|
||||
# if not a closed loop and loops back on itself, should be skipped
|
||||
if not closed_loop and e < prev_val:
|
||||
skip_this_window = True
|
||||
break
|
||||
to_yield.append(e)
|
||||
prev_val = e
|
||||
if skip_this_window:
|
||||
continue
|
||||
# yield if not skipped
|
||||
yield to_yield
|
||||
|
||||
|
||||
def get_context_scheduler(name: str) -> Callable:
|
||||
match name:
|
||||
case ContextSchedules.UNIFORM:
|
||||
return uniform
|
||||
case ContextSchedules.UNIFORM_CONSTANT:
|
||||
return uniform_constant
|
||||
case ContextSchedules.UNIFORM_V2:
|
||||
return uniform_v2
|
||||
case _:
|
||||
raise ValueError(f"Unknown context_overlap policy {name}")
|
||||
@@ -0,0 +1,13 @@
|
||||
import torch
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
# Tensor to PIL
|
||||
def tensor2pil(image):
|
||||
return Image.fromarray(
|
||||
np.clip(255.0 * image.cpu().numpy().squeeze(), 0, 255).astype(np.uint8)
|
||||
)
|
||||
|
||||
# Convert PIL to Tensor
|
||||
def pil2tensor(image):
|
||||
return torch.from_numpy(np.array(image).astype(np.float32) / 255.0).unsqueeze(0)
|
||||
@@ -1,146 +0,0 @@
|
||||
import { app } from '../../../scripts/app.js'
|
||||
import { api } from '../../../scripts/api.js'
|
||||
|
||||
function offsetDOMWidget(
|
||||
widget,
|
||||
ctx,
|
||||
node,
|
||||
widgetWidth,
|
||||
widgetY,
|
||||
height
|
||||
) {
|
||||
const margin = 10
|
||||
const elRect = ctx.canvas.getBoundingClientRect()
|
||||
const transform = new DOMMatrix()
|
||||
.scaleSelf(
|
||||
elRect.width / ctx.canvas.width,
|
||||
elRect.height / ctx.canvas.height
|
||||
)
|
||||
.multiplySelf(ctx.getTransform())
|
||||
.translateSelf(0, widgetY + margin)
|
||||
|
||||
const scale = new DOMMatrix().scaleSelf(transform.a, transform.d)
|
||||
Object.assign(widget.inputEl.style, {
|
||||
transformOrigin: '0 0',
|
||||
transform: scale,
|
||||
left: `${transform.e}px`,
|
||||
top: `${transform.d + transform.f}px`,
|
||||
width: `${widgetWidth}px`,
|
||||
height: `${(height || widget.parent?.inputHeight || 32) - margin}px`,
|
||||
position: 'absolute',
|
||||
background: !node.color ? '' : node.color,
|
||||
color: !node.color ? '' : 'white',
|
||||
zIndex: 5, //app.graph._nodes.indexOf(node),
|
||||
})
|
||||
}
|
||||
|
||||
export const hasWidgets = (node) => {
|
||||
if (!node.widgets || !node.widgets?.[Symbol.iterator]) {
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
export const cleanupNode = (node) => {
|
||||
if (!hasWidgets(node)) {
|
||||
return
|
||||
}
|
||||
|
||||
for (const w of node.widgets) {
|
||||
if (w.canvas) {
|
||||
w.canvas.remove()
|
||||
}
|
||||
if (w.inputEl) {
|
||||
w.inputEl.remove()
|
||||
}
|
||||
// calls the widget remove callback
|
||||
w.onRemoved?.()
|
||||
}
|
||||
}
|
||||
|
||||
const CreatePreviewElement = (name, val, format) => {
|
||||
const [type] = format.split('/')
|
||||
|
||||
const w = {
|
||||
name,
|
||||
type,
|
||||
value: val,
|
||||
draw: function (ctx, node, widgetWidth, widgetY, height) {
|
||||
const [cw, ch] = this.computeSize(widgetWidth)
|
||||
offsetDOMWidget(this, ctx, node, widgetWidth, widgetY, ch)
|
||||
},
|
||||
computeSize: function (_) {
|
||||
const ratio = this.inputRatio || 1
|
||||
const width = Math.max(220, this.parent.size[0])
|
||||
return [width, (width / ratio + 10)]
|
||||
},
|
||||
onRemoved: function () {
|
||||
if (this.inputEl) {
|
||||
this.inputEl.remove()
|
||||
}
|
||||
},
|
||||
}
|
||||
|
||||
w.inputEl = document.createElement(type === 'video' ? 'video' : 'img')
|
||||
w.inputEl.src = w.value
|
||||
if (type === 'video') {
|
||||
w.inputEl.setAttribute('type', 'video/webm');
|
||||
w.inputEl.autoplay = true
|
||||
w.inputEl.loop = true
|
||||
w.inputEl.controls = false;
|
||||
}
|
||||
w.inputEl.onload = function () {
|
||||
w.inputRatio = w.inputEl.naturalWidth / w.inputEl.naturalHeight
|
||||
}
|
||||
document.body.appendChild(w.inputEl)
|
||||
return w
|
||||
}
|
||||
|
||||
const gif_preview = {
|
||||
name: 'AnimateDiff.gif_preview',
|
||||
async beforeRegisterNodeDef(nodeType, nodeData, app) {
|
||||
switch (nodeData.name) {
|
||||
case 'AnimateDiffCombine': {
|
||||
const onExecuted = nodeType.prototype.onExecuted
|
||||
nodeType.prototype.onExecuted = function (message) {
|
||||
const prefix = 'ad_gif_preview_'
|
||||
const r = onExecuted ? onExecuted.apply(this, message) : undefined
|
||||
|
||||
if (this.widgets) {
|
||||
const pos = this.widgets.findIndex((w) => w.name === `${prefix}_0`)
|
||||
if (pos !== -1) {
|
||||
for (let i = pos; i < this.widgets.length; i++) {
|
||||
this.widgets[i].onRemoved?.()
|
||||
}
|
||||
this.widgets.length = pos
|
||||
}
|
||||
if (message?.gifs) {
|
||||
message.gifs.forEach((params, i) => {
|
||||
const previewUrl = api.apiURL(
|
||||
'/view?' + new URLSearchParams(params).toString()
|
||||
)
|
||||
const w = this.addCustomWidget(
|
||||
CreatePreviewElement(`${prefix}_${i}`, previewUrl, params.format || 'image/gif')
|
||||
)
|
||||
w.parent = this
|
||||
})
|
||||
}
|
||||
|
||||
const onRemoved = this.onRemoved
|
||||
this.onRemoved = () => {
|
||||
cleanupNode(this)
|
||||
return onRemoved?.()
|
||||
}
|
||||
}
|
||||
|
||||
// keep width and update height
|
||||
this.setSize([this.size[0], this.computeSize([this.size[0], this.size[1]])[1]])
|
||||
return r
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
app.registerExtension(gif_preview)
|
||||
@@ -0,0 +1,162 @@
|
||||
import { app } from "../../../scripts/app.js";
|
||||
import { api } from "../../../scripts/api.js";
|
||||
|
||||
function offsetDOMWidget(widget, ctx, node, widgetWidth, widgetY, height) {
|
||||
const margin = 10;
|
||||
const elRect = ctx.canvas.getBoundingClientRect();
|
||||
const transform = new DOMMatrix()
|
||||
.scaleSelf(
|
||||
elRect.width / ctx.canvas.width,
|
||||
elRect.height / ctx.canvas.height
|
||||
)
|
||||
.multiplySelf(ctx.getTransform())
|
||||
.translateSelf(0, widgetY + margin);
|
||||
|
||||
const scale = new DOMMatrix().scaleSelf(transform.a, transform.d);
|
||||
Object.assign(widget.inputEl.style, {
|
||||
transformOrigin: "0 0",
|
||||
transform: scale,
|
||||
left: `${transform.e}px`,
|
||||
top: `${transform.d + transform.f}px`,
|
||||
width: `${widgetWidth}px`,
|
||||
height: `${(height || widget.parent?.inputHeight || 32) - margin}px`,
|
||||
position: "absolute",
|
||||
background: !node.color ? "" : node.color,
|
||||
color: !node.color ? "" : "white",
|
||||
zIndex: 5, //app.graph._nodes.indexOf(node),
|
||||
});
|
||||
}
|
||||
|
||||
export const hasWidgets = (node) => {
|
||||
if (!node.widgets || !node.widgets?.[Symbol.iterator]) {
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
};
|
||||
|
||||
export const cleanupNode = (node) => {
|
||||
if (!hasWidgets(node)) {
|
||||
return;
|
||||
}
|
||||
|
||||
for (const w of node.widgets) {
|
||||
if (w.canvas) {
|
||||
w.canvas.remove();
|
||||
}
|
||||
if (w.inputEl) {
|
||||
w.inputEl.remove();
|
||||
}
|
||||
// calls the widget remove callback
|
||||
w.onRemoved?.();
|
||||
}
|
||||
};
|
||||
|
||||
export const CreatePreviewElement = (name, val, format, callback) => {
|
||||
const [type] = format.split("/");
|
||||
|
||||
const w = {
|
||||
name,
|
||||
type,
|
||||
value: val,
|
||||
draw: function (ctx, node, widgetWidth, widgetY, height) {
|
||||
const [cw, ch] = this.computeSize(widgetWidth);
|
||||
offsetDOMWidget(this, ctx, node, widgetWidth, widgetY, ch);
|
||||
},
|
||||
computeSize: function (_) {
|
||||
const ratio = this.inputRatio || 1;
|
||||
const width = Math.max(220, this.parent.size[0]);
|
||||
return [width, width / ratio + 10];
|
||||
},
|
||||
onRemoved: function () {
|
||||
if (this.inputEl) {
|
||||
this.inputEl.remove();
|
||||
}
|
||||
},
|
||||
};
|
||||
|
||||
w.inputEl = document.createElement(type === "video" ? "video" : "img");
|
||||
w.inputEl.src = w.value;
|
||||
if (type === "video") {
|
||||
w.inputEl.setAttribute("type", "video/webm");
|
||||
w.inputEl.autoplay = true;
|
||||
w.inputEl.loop = true;
|
||||
w.inputEl.controls = false;
|
||||
}
|
||||
w.inputEl.onload = function () {
|
||||
w.inputRatio = w.inputEl.naturalWidth / w.inputEl.naturalHeight;
|
||||
callback?.();
|
||||
};
|
||||
document.body.appendChild(w.inputEl);
|
||||
return w;
|
||||
};
|
||||
|
||||
const videoPreview = {
|
||||
name: "AnimateDiff.VideoPreview",
|
||||
async beforeRegisterNodeDef(nodeType, nodeData, app) {
|
||||
const onExecuted = nodeType.prototype.onExecuted;
|
||||
nodeType.prototype.onExecuted = function (message) {
|
||||
const r = onExecuted ? onExecuted.apply(this, message) : undefined;
|
||||
|
||||
if (message?.videos) {
|
||||
this.videos = message.videos;
|
||||
}
|
||||
|
||||
return r;
|
||||
};
|
||||
|
||||
const onDrawBackground = nodeType.prototype.onDrawBackground;
|
||||
nodeType.prototype.onDrawBackground = function (ctx) {
|
||||
const r = onDrawBackground ? onDrawBackground.apply(this, arguments) : undefined;
|
||||
const node = this;
|
||||
const prefix = "ad_video_preview_";
|
||||
|
||||
if (node.videos_rendered === node.videos) {
|
||||
return r;
|
||||
}
|
||||
|
||||
if (node.widgets) {
|
||||
const pos = node.widgets.findIndex((w) => w.name === `${prefix}_0`);
|
||||
if (pos !== -1) {
|
||||
for (let i = pos; i < node.widgets.length; i++) {
|
||||
node.widgets[i].onRemoved?.();
|
||||
}
|
||||
node.widgets.length = pos;
|
||||
}
|
||||
}
|
||||
if (node.videos) {
|
||||
node.videos.forEach((params, i) => {
|
||||
const previewUrl = api.apiURL(
|
||||
"/view?" + new URLSearchParams(params).toString()
|
||||
);
|
||||
const w = node.addCustomWidget(
|
||||
CreatePreviewElement(
|
||||
`${prefix}_${i}`,
|
||||
previewUrl,
|
||||
params.format || "image/gif",
|
||||
node.computeSizeKeepWidth.bind(node)
|
||||
)
|
||||
);
|
||||
w.parent = node;
|
||||
});
|
||||
node.videos_rendered = node.videos;
|
||||
}
|
||||
|
||||
return r;
|
||||
};
|
||||
|
||||
const onRemoved = nodeType.prototype.onRemoved;
|
||||
nodeType.prototype.onRemoved = function () {
|
||||
cleanupNode(this);
|
||||
return onRemoved ? onRemoved.apply(this, arguments) : undefined;
|
||||
};
|
||||
|
||||
nodeType.prototype.computeSizeKeepWidth = function () {
|
||||
this.setSize([
|
||||
this.size[0],
|
||||
this.computeSize([this.size[0], this.size[1]])[1],
|
||||
]);
|
||||
};
|
||||
},
|
||||
};
|
||||
|
||||
app.registerExtension(videoPreview);
|
||||
@@ -0,0 +1,188 @@
|
||||
import { app } from "../../../scripts/app.js";
|
||||
import { api } from "../../../scripts/api.js";
|
||||
import { ComfyWidgets } from "../../../scripts/widgets.js";
|
||||
|
||||
const supportedVideoTypes = [
|
||||
"image/gif",
|
||||
"video/webm",
|
||||
"video/mp4",
|
||||
"video/mov",
|
||||
];
|
||||
|
||||
const VIDEOUPLOAD = (node, inputName, inputData, app) => {
|
||||
const previewWidget = "ad_video_preview";
|
||||
const videoWidget = node.widgets.find((w) => w.name === "video");
|
||||
let uploadWidget;
|
||||
|
||||
const showVideo = (name) => {
|
||||
let folder_separator = name.lastIndexOf("/");
|
||||
let subfolder = "";
|
||||
if (folder_separator > -1) {
|
||||
subfolder = name.substring(0, folder_separator);
|
||||
name = name.substring(folder_separator + 1);
|
||||
}
|
||||
const ext = name.substring(name.lastIndexOf(".") + 1);
|
||||
const format = supportedVideoTypes.find((t) => t.endsWith(ext));
|
||||
node.videos = [
|
||||
{
|
||||
filename: name,
|
||||
type: "input",
|
||||
subfolder: subfolder,
|
||||
format,
|
||||
},
|
||||
];
|
||||
};
|
||||
|
||||
var default_value = videoWidget.value;
|
||||
Object.defineProperty(videoWidget, "value", {
|
||||
set: function (value) {
|
||||
this._real_value = value;
|
||||
},
|
||||
|
||||
get: function () {
|
||||
let value = "";
|
||||
if (this._real_value) {
|
||||
value = this._real_value;
|
||||
} else {
|
||||
return default_value;
|
||||
}
|
||||
|
||||
if (value.filename) {
|
||||
let real_value = value;
|
||||
value = "";
|
||||
if (real_value.subfolder) {
|
||||
value = real_value.subfolder + "/";
|
||||
}
|
||||
|
||||
value += real_value.filename;
|
||||
|
||||
if (real_value.type && real_value.type !== "input")
|
||||
value += ` [${real_value.type}]`;
|
||||
}
|
||||
return value;
|
||||
},
|
||||
});
|
||||
|
||||
// Add our own callback to the combo widget to render an image when it changes
|
||||
const cb = node.callback;
|
||||
videoWidget.callback = function () {
|
||||
showVideo(videoWidget.value);
|
||||
if (cb) {
|
||||
return cb.apply(this, arguments);
|
||||
}
|
||||
};
|
||||
|
||||
// On load if we have a value then render the image
|
||||
// The value isnt set immediately so we need to wait a moment
|
||||
// No change callbacks seem to be fired on initial setting of the value
|
||||
requestAnimationFrame(() => {
|
||||
if (videoWidget.value) {
|
||||
showVideo(videoWidget.value);
|
||||
}
|
||||
});
|
||||
|
||||
async function uploadFile(file, updateNode, pasted = false) {
|
||||
try {
|
||||
// Wrap file in formdata so it includes filename
|
||||
const body = new FormData();
|
||||
body.append("image", file);
|
||||
body.append("subfolder", "video");
|
||||
const resp = await api.fetchApi("/upload/image", {
|
||||
method: "POST",
|
||||
body,
|
||||
});
|
||||
|
||||
if (resp.status === 200) {
|
||||
const data = await resp.json();
|
||||
// Add the file to the dropdown list and update the widget value
|
||||
let path = data.name;
|
||||
if (data.subfolder) path = data.subfolder + "/" + path;
|
||||
|
||||
if (!videoWidget.options.values.includes(path)) {
|
||||
videoWidget.options.values.push(path);
|
||||
}
|
||||
|
||||
if (updateNode) {
|
||||
showVideo(path);
|
||||
videoWidget.value = path;
|
||||
}
|
||||
} else {
|
||||
alert(resp.status + " - " + resp.statusText);
|
||||
}
|
||||
} catch (error) {
|
||||
alert(error);
|
||||
}
|
||||
}
|
||||
|
||||
const fileInput = document.createElement("input");
|
||||
Object.assign(fileInput, {
|
||||
type: "file",
|
||||
accept: supportedVideoTypes.join(","),
|
||||
style: "display: none",
|
||||
onchange: async () => {
|
||||
if (fileInput.files.length) {
|
||||
await uploadFile(fileInput.files[0], true);
|
||||
}
|
||||
},
|
||||
});
|
||||
document.body.append(fileInput);
|
||||
|
||||
// Create the button widget for selecting the files
|
||||
uploadWidget = node.addWidget(
|
||||
"button",
|
||||
"choose file to upload",
|
||||
"image",
|
||||
() => {
|
||||
fileInput.click();
|
||||
}
|
||||
);
|
||||
uploadWidget.serialize = false;
|
||||
|
||||
// Add handler to check if an image is being dragged over our node
|
||||
node.onDragOver = function (e) {
|
||||
if (e.dataTransfer && e.dataTransfer.items) {
|
||||
const image = [...e.dataTransfer.items].find((f) => f.kind === "file");
|
||||
return !!image;
|
||||
}
|
||||
|
||||
return false;
|
||||
};
|
||||
|
||||
// On drop upload files
|
||||
node.onDragDrop = function (e) {
|
||||
console.log("onDragDrop called");
|
||||
let handled = false;
|
||||
for (const file of e.dataTransfer.files) {
|
||||
if (file.type.startsWith("image/")) {
|
||||
uploadFile(file, !handled); // Dont await these, any order is fine, only update on first one
|
||||
handled = true;
|
||||
}
|
||||
}
|
||||
|
||||
return handled;
|
||||
};
|
||||
|
||||
node.pasteFile = function (file) {
|
||||
if (supportedVideoTypes.indexOf(file.type) > -1) {
|
||||
const is_pasted =
|
||||
file.name === "image.png" && file.lastModified - Date.now() < 2000;
|
||||
uploadFile(file, true, is_pasted);
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
};
|
||||
|
||||
return { widget: uploadWidget };
|
||||
};
|
||||
|
||||
ComfyWidgets["VIDEOUPLOAD"] = VIDEOUPLOAD;
|
||||
|
||||
// Adds an upload button to the nodes
|
||||
app.registerExtension({
|
||||
name: "AnimateDiff.UploadVideo",
|
||||
async beforeRegisterNodeDef(nodeType, nodeData, app) {
|
||||
if (nodeData?.input?.required?.video?.[1]?.video_upload === true) {
|
||||
nodeData.input.required.upload = ["VIDEOUPLOAD"];
|
||||
}
|
||||
},
|
||||
});
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,877 @@
|
||||
{
|
||||
"last_node_id": 106,
|
||||
"last_link_id": 189,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 16,
|
||||
"type": "AnimateDiffModuleLoader",
|
||||
"pos": [
|
||||
-280,
|
||||
140
|
||||
],
|
||||
"size": {
|
||||
"0": 310,
|
||||
"1": 60
|
||||
},
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "MOTION_MODULE",
|
||||
"type": "MOTION_MODULE",
|
||||
"links": [
|
||||
78
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "AnimateDiffModuleLoader"
|
||||
},
|
||||
"widgets_values": [
|
||||
"mm-Stabilized_mid.pth"
|
||||
],
|
||||
"color": "#571a1a",
|
||||
"bgcolor": "#6b2e2e"
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"type": "VAELoader",
|
||||
"pos": [
|
||||
-280,
|
||||
400
|
||||
],
|
||||
"size": {
|
||||
"0": 310,
|
||||
"1": 60
|
||||
},
|
||||
"flags": {},
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "VAE",
|
||||
"type": "VAE",
|
||||
"links": [
|
||||
82
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VAELoader"
|
||||
},
|
||||
"widgets_values": [
|
||||
"vae-ft-mse-840000-ema-pruned.safetensors"
|
||||
],
|
||||
"color": "#571a1a",
|
||||
"bgcolor": "#6b2e2e"
|
||||
},
|
||||
{
|
||||
"id": 45,
|
||||
"type": "AnimateDiffCombine",
|
||||
"pos": [
|
||||
1240,
|
||||
140
|
||||
],
|
||||
"size": [
|
||||
360,
|
||||
732
|
||||
],
|
||||
"flags": {},
|
||||
"order": 13,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"link": 172
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "GIF",
|
||||
"type": "GIF",
|
||||
"links": null,
|
||||
"shape": 3
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "AnimateDiffCombine"
|
||||
},
|
||||
"widgets_values": [
|
||||
8,
|
||||
0,
|
||||
true,
|
||||
"AnimateDiff",
|
||||
"image/gif",
|
||||
true,
|
||||
"/view?filename=AnimateDiff_00092_.gif&subfolder=&type=output&format=image%2Fgif"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "CheckpointLoaderSimple",
|
||||
"pos": [
|
||||
-280,
|
||||
250
|
||||
],
|
||||
"size": {
|
||||
"0": 310,
|
||||
"1": 100
|
||||
},
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "MODEL",
|
||||
"type": "MODEL",
|
||||
"links": [
|
||||
79
|
||||
],
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "CLIP",
|
||||
"type": "CLIP",
|
||||
"links": [
|
||||
3,
|
||||
5
|
||||
],
|
||||
"slot_index": 1
|
||||
},
|
||||
{
|
||||
"name": "VAE",
|
||||
"type": "VAE",
|
||||
"links": [],
|
||||
"slot_index": 2
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "CheckpointLoaderSimple"
|
||||
},
|
||||
"widgets_values": [
|
||||
"SDHK_v4.safetensors"
|
||||
],
|
||||
"color": "#571a1a",
|
||||
"bgcolor": "#6b2e2e"
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "CLIPTextEncode",
|
||||
"pos": [
|
||||
60,
|
||||
300
|
||||
],
|
||||
"size": [
|
||||
310,
|
||||
100
|
||||
],
|
||||
"flags": {},
|
||||
"order": 6,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "clip",
|
||||
"type": "CLIP",
|
||||
"link": 5
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "CONDITIONING",
|
||||
"type": "CONDITIONING",
|
||||
"links": [
|
||||
70
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "CLIPTextEncode"
|
||||
},
|
||||
"widgets_values": [
|
||||
"embedding:easynegative, embedding:badhandv4, nsfw"
|
||||
],
|
||||
"color": "#572e1a",
|
||||
"bgcolor": "#6b422e"
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "CLIPTextEncode",
|
||||
"pos": [
|
||||
60,
|
||||
140
|
||||
],
|
||||
"size": [
|
||||
310,
|
||||
110
|
||||
],
|
||||
"flags": {},
|
||||
"order": 5,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "clip",
|
||||
"type": "CLIP",
|
||||
"link": 3
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "CONDITIONING",
|
||||
"type": "CONDITIONING",
|
||||
"links": [
|
||||
69
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "CLIPTextEncode"
|
||||
},
|
||||
"widgets_values": [
|
||||
"(best quality, masterpiece), 1girl, short hair, blue eyes, dancing, city, cloudy"
|
||||
],
|
||||
"color": "#572e1a",
|
||||
"bgcolor": "#6b422e"
|
||||
},
|
||||
{
|
||||
"id": 41,
|
||||
"type": "AnimateDiffSampler",
|
||||
"pos": [
|
||||
900,
|
||||
140
|
||||
],
|
||||
"size": [
|
||||
310,
|
||||
330
|
||||
],
|
||||
"flags": {},
|
||||
"order": 11,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "motion_module",
|
||||
"type": "MOTION_MODULE",
|
||||
"link": 78,
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "model",
|
||||
"type": "MODEL",
|
||||
"link": 79,
|
||||
"slot_index": 1
|
||||
},
|
||||
{
|
||||
"name": "positive",
|
||||
"type": "CONDITIONING",
|
||||
"link": 176
|
||||
},
|
||||
{
|
||||
"name": "negative",
|
||||
"type": "CONDITIONING",
|
||||
"link": 180
|
||||
},
|
||||
{
|
||||
"name": "latent_image",
|
||||
"type": "LATENT",
|
||||
"link": 80
|
||||
},
|
||||
{
|
||||
"name": "frame_number",
|
||||
"type": "INT",
|
||||
"link": 185,
|
||||
"widget": {
|
||||
"name": "frame_number",
|
||||
"config": [
|
||||
"INT",
|
||||
{
|
||||
"default": 16,
|
||||
"min": 2,
|
||||
"max": 32,
|
||||
"step": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "LATENT",
|
||||
"type": "LATENT",
|
||||
"links": [
|
||||
81
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "AnimateDiffSampler"
|
||||
},
|
||||
"widgets_values": [
|
||||
"default",
|
||||
16,
|
||||
345029849956754,
|
||||
"fixed",
|
||||
20,
|
||||
8,
|
||||
"euler",
|
||||
"normal",
|
||||
1
|
||||
],
|
||||
"color": "#57571a",
|
||||
"bgcolor": "#6b6b2e"
|
||||
},
|
||||
{
|
||||
"id": 39,
|
||||
"type": "ControlNetApplyAdvanced",
|
||||
"pos": [
|
||||
471,
|
||||
275
|
||||
],
|
||||
"size": [
|
||||
300,
|
||||
170
|
||||
],
|
||||
"flags": {},
|
||||
"order": 9,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "positive",
|
||||
"type": "CONDITIONING",
|
||||
"link": 69
|
||||
},
|
||||
{
|
||||
"name": "negative",
|
||||
"type": "CONDITIONING",
|
||||
"link": 70
|
||||
},
|
||||
{
|
||||
"name": "control_net",
|
||||
"type": "CONTROL_NET",
|
||||
"link": 68
|
||||
},
|
||||
{
|
||||
"name": "image",
|
||||
"type": "IMAGE",
|
||||
"link": 181
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "positive",
|
||||
"type": "CONDITIONING",
|
||||
"links": [
|
||||
176
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "negative",
|
||||
"type": "CONDITIONING",
|
||||
"links": [
|
||||
180
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 1
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "ControlNetApplyAdvanced"
|
||||
},
|
||||
"widgets_values": [
|
||||
1,
|
||||
0,
|
||||
1
|
||||
],
|
||||
"color": "#43571a",
|
||||
"bgcolor": "#576b2e"
|
||||
},
|
||||
{
|
||||
"id": 44,
|
||||
"type": "VAEDecode",
|
||||
"pos": [
|
||||
1000,
|
||||
520
|
||||
],
|
||||
"size": {
|
||||
"0": 210,
|
||||
"1": 46
|
||||
},
|
||||
"flags": {},
|
||||
"order": 12,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "samples",
|
||||
"type": "LATENT",
|
||||
"link": 81
|
||||
},
|
||||
{
|
||||
"name": "vae",
|
||||
"type": "VAE",
|
||||
"link": 82
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
172,
|
||||
187
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VAEDecode"
|
||||
},
|
||||
"color": "#2e571a",
|
||||
"bgcolor": "#426b2e"
|
||||
},
|
||||
{
|
||||
"id": 103,
|
||||
"type": "LoadVideo",
|
||||
"pos": [
|
||||
-280,
|
||||
650
|
||||
],
|
||||
"size": [
|
||||
310,
|
||||
629
|
||||
],
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "frames",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
181,
|
||||
182,
|
||||
186
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "frame_count",
|
||||
"type": "INT",
|
||||
"links": null,
|
||||
"shape": 3
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LoadVideo"
|
||||
},
|
||||
"widgets_values": [
|
||||
"video/265043418-23291941-864d-495a-8ba8-d02e05756396.gif",
|
||||
"image",
|
||||
0,
|
||||
16,
|
||||
"/view?filename=265043418-23291941-864d-495a-8ba8-d02e05756396.gif&type=input&subfolder=video&format=image%2Fgif"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 20,
|
||||
"type": "EmptyLatentImage",
|
||||
"pos": [
|
||||
520,
|
||||
630
|
||||
],
|
||||
"size": [
|
||||
210,
|
||||
80
|
||||
],
|
||||
"flags": {},
|
||||
"order": 10,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "width",
|
||||
"type": "INT",
|
||||
"link": 189,
|
||||
"widget": {
|
||||
"name": "width",
|
||||
"config": [
|
||||
"INT",
|
||||
{
|
||||
"default": 512,
|
||||
"min": 64,
|
||||
"max": 8192,
|
||||
"step": 8
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "height",
|
||||
"type": "INT",
|
||||
"link": 188,
|
||||
"widget": {
|
||||
"name": "height",
|
||||
"config": [
|
||||
"INT",
|
||||
{
|
||||
"default": 512,
|
||||
"min": 64,
|
||||
"max": 8192,
|
||||
"step": 8
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "LATENT",
|
||||
"type": "LATENT",
|
||||
"links": [
|
||||
80
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "EmptyLatentImage"
|
||||
},
|
||||
"widgets_values": [
|
||||
512,
|
||||
512,
|
||||
1
|
||||
],
|
||||
"color": "#1a572e",
|
||||
"bgcolor": "#2e6b42"
|
||||
},
|
||||
{
|
||||
"id": 104,
|
||||
"type": "ImageSizeAndBatchSize",
|
||||
"pos": [
|
||||
300,
|
||||
630
|
||||
],
|
||||
"size": [
|
||||
190,
|
||||
80
|
||||
],
|
||||
"flags": {},
|
||||
"order": 7,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "image",
|
||||
"type": "IMAGE",
|
||||
"link": 182
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "width",
|
||||
"type": "INT",
|
||||
"links": [
|
||||
188
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "height",
|
||||
"type": "INT",
|
||||
"links": [
|
||||
189
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 1
|
||||
},
|
||||
{
|
||||
"name": "batch_size",
|
||||
"type": "INT",
|
||||
"links": [
|
||||
185
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 2
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "ImageSizeAndBatchSize"
|
||||
},
|
||||
"color": "#1a5757",
|
||||
"bgcolor": "#2e6b6b"
|
||||
},
|
||||
{
|
||||
"id": 36,
|
||||
"type": "ControlNetLoaderAdvanced",
|
||||
"pos": [
|
||||
-280,
|
||||
540
|
||||
],
|
||||
"size": [
|
||||
310,
|
||||
60
|
||||
],
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "timestep_keyframe",
|
||||
"type": "TIMESTEP_KEYFRAME",
|
||||
"link": null,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "CONTROL_NET",
|
||||
"type": "CONTROL_NET",
|
||||
"links": [
|
||||
68
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "ControlNetLoaderAdvanced"
|
||||
},
|
||||
"widgets_values": [
|
||||
"control_v11p_sd15_openpose.pth"
|
||||
],
|
||||
"color": "#571a1a",
|
||||
"bgcolor": "#6b2e2e"
|
||||
},
|
||||
{
|
||||
"id": 105,
|
||||
"type": "PreviewImage",
|
||||
"pos": [
|
||||
70,
|
||||
830
|
||||
],
|
||||
"size": [
|
||||
530,
|
||||
420
|
||||
],
|
||||
"flags": {},
|
||||
"order": 8,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"link": 186
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "PreviewImage"
|
||||
},
|
||||
"color": "#1a5757",
|
||||
"bgcolor": "#2e6b6b"
|
||||
},
|
||||
{
|
||||
"id": 106,
|
||||
"type": "PreviewImage",
|
||||
"pos": [
|
||||
670,
|
||||
830
|
||||
],
|
||||
"size": [
|
||||
530,
|
||||
420
|
||||
],
|
||||
"flags": {},
|
||||
"order": 14,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"link": 187
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "PreviewImage"
|
||||
},
|
||||
"color": "#1a5757",
|
||||
"bgcolor": "#2e6b6b"
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[
|
||||
3,
|
||||
4,
|
||||
1,
|
||||
6,
|
||||
0,
|
||||
"CLIP"
|
||||
],
|
||||
[
|
||||
5,
|
||||
4,
|
||||
1,
|
||||
7,
|
||||
0,
|
||||
"CLIP"
|
||||
],
|
||||
[
|
||||
68,
|
||||
36,
|
||||
0,
|
||||
39,
|
||||
2,
|
||||
"CONTROL_NET"
|
||||
],
|
||||
[
|
||||
69,
|
||||
6,
|
||||
0,
|
||||
39,
|
||||
0,
|
||||
"CONDITIONING"
|
||||
],
|
||||
[
|
||||
70,
|
||||
7,
|
||||
0,
|
||||
39,
|
||||
1,
|
||||
"CONDITIONING"
|
||||
],
|
||||
[
|
||||
78,
|
||||
16,
|
||||
0,
|
||||
41,
|
||||
0,
|
||||
"MOTION_MODULE"
|
||||
],
|
||||
[
|
||||
79,
|
||||
4,
|
||||
0,
|
||||
41,
|
||||
1,
|
||||
"MODEL"
|
||||
],
|
||||
[
|
||||
80,
|
||||
20,
|
||||
0,
|
||||
41,
|
||||
4,
|
||||
"LATENT"
|
||||
],
|
||||
[
|
||||
81,
|
||||
41,
|
||||
0,
|
||||
44,
|
||||
0,
|
||||
"LATENT"
|
||||
],
|
||||
[
|
||||
82,
|
||||
13,
|
||||
0,
|
||||
44,
|
||||
1,
|
||||
"VAE"
|
||||
],
|
||||
[
|
||||
172,
|
||||
44,
|
||||
0,
|
||||
45,
|
||||
0,
|
||||
"IMAGE"
|
||||
],
|
||||
[
|
||||
176,
|
||||
39,
|
||||
0,
|
||||
41,
|
||||
2,
|
||||
"CONDITIONING"
|
||||
],
|
||||
[
|
||||
180,
|
||||
39,
|
||||
1,
|
||||
41,
|
||||
3,
|
||||
"CONDITIONING"
|
||||
],
|
||||
[
|
||||
181,
|
||||
103,
|
||||
0,
|
||||
39,
|
||||
3,
|
||||
"IMAGE"
|
||||
],
|
||||
[
|
||||
182,
|
||||
103,
|
||||
0,
|
||||
104,
|
||||
0,
|
||||
"IMAGE"
|
||||
],
|
||||
[
|
||||
185,
|
||||
104,
|
||||
2,
|
||||
41,
|
||||
5,
|
||||
"INT"
|
||||
],
|
||||
[
|
||||
186,
|
||||
103,
|
||||
0,
|
||||
105,
|
||||
0,
|
||||
"IMAGE"
|
||||
],
|
||||
[
|
||||
187,
|
||||
44,
|
||||
0,
|
||||
106,
|
||||
0,
|
||||
"IMAGE"
|
||||
],
|
||||
[
|
||||
188,
|
||||
104,
|
||||
0,
|
||||
20,
|
||||
1,
|
||||
"INT"
|
||||
],
|
||||
[
|
||||
189,
|
||||
104,
|
||||
1,
|
||||
20,
|
||||
0,
|
||||
"INT"
|
||||
]
|
||||
],
|
||||
"groups": [],
|
||||
"config": {},
|
||||
"extra": {},
|
||||
"version": 0.4
|
||||
}
|
||||
@@ -0,0 +1,820 @@
|
||||
{
|
||||
"last_node_id": 28,
|
||||
"last_link_id": 56,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 20,
|
||||
"type": "EmptyLatentImage",
|
||||
"pos": [
|
||||
520,
|
||||
20
|
||||
],
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 106
|
||||
},
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "LATENT",
|
||||
"type": "LATENT",
|
||||
"links": [
|
||||
35
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "EmptyLatentImage"
|
||||
},
|
||||
"widgets_values": [
|
||||
512,
|
||||
512,
|
||||
1
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 25,
|
||||
"type": "Reroute",
|
||||
"pos": [
|
||||
440,
|
||||
611
|
||||
],
|
||||
"size": [
|
||||
75,
|
||||
26
|
||||
],
|
||||
"flags": {},
|
||||
"order": 9,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "",
|
||||
"type": "*",
|
||||
"link": 45
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "",
|
||||
"type": "LATENT",
|
||||
"links": [
|
||||
46
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"showOutputText": false,
|
||||
"horizontal": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 24,
|
||||
"type": "Reroute",
|
||||
"pos": [
|
||||
1224,
|
||||
604
|
||||
],
|
||||
"size": [
|
||||
75,
|
||||
26
|
||||
],
|
||||
"flags": {},
|
||||
"order": 7,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "",
|
||||
"type": "*",
|
||||
"link": 44
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "",
|
||||
"type": "LATENT",
|
||||
"links": [
|
||||
45
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"showOutputText": false,
|
||||
"horizontal": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 16,
|
||||
"type": "AnimateDiffModuleLoader",
|
||||
"pos": [
|
||||
27,
|
||||
345
|
||||
],
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 58
|
||||
},
|
||||
"flags": {},
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "MOTION_MODULE",
|
||||
"type": "MOTION_MODULE",
|
||||
"links": [
|
||||
24,
|
||||
48
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "AnimateDiffModuleLoader"
|
||||
},
|
||||
"widgets_values": [
|
||||
"mm-Stabilized_mid.pth"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "CheckpointLoaderSimple",
|
||||
"pos": [
|
||||
26,
|
||||
474
|
||||
],
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 98
|
||||
},
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "MODEL",
|
||||
"type": "MODEL",
|
||||
"links": [
|
||||
25,
|
||||
49
|
||||
],
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "CLIP",
|
||||
"type": "CLIP",
|
||||
"links": [
|
||||
3,
|
||||
5
|
||||
],
|
||||
"slot_index": 1
|
||||
},
|
||||
{
|
||||
"name": "VAE",
|
||||
"type": "VAE",
|
||||
"links": [],
|
||||
"slot_index": 2
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "CheckpointLoaderSimple"
|
||||
},
|
||||
"widgets_values": [
|
||||
"AnimeLike25D_v11.safetensors"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 22,
|
||||
"type": "LatentUpscaleBy",
|
||||
"pos": [
|
||||
571,
|
||||
712
|
||||
],
|
||||
"size": [
|
||||
275.35137939453125,
|
||||
82
|
||||
],
|
||||
"flags": {},
|
||||
"order": 11,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "samples",
|
||||
"type": "LATENT",
|
||||
"link": 46
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "LATENT",
|
||||
"type": "LATENT",
|
||||
"links": [
|
||||
47
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LatentUpscaleBy"
|
||||
},
|
||||
"widgets_values": [
|
||||
"nearest-exact",
|
||||
1.5
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "CLIPTextEncode",
|
||||
"pos": [
|
||||
415,
|
||||
186
|
||||
],
|
||||
"size": {
|
||||
"0": 422.84503173828125,
|
||||
"1": 164.31304931640625
|
||||
},
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "clip",
|
||||
"type": "CLIP",
|
||||
"link": 3
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "CONDITIONING",
|
||||
"type": "CONDITIONING",
|
||||
"links": [
|
||||
29,
|
||||
50
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "CLIPTextEncode"
|
||||
},
|
||||
"widgets_values": [
|
||||
"masterpiece, best quality, 1girl, solo, cherry blossoms, hanami, pink flower, white flower, spring season, wisteria, petals, flower, plum blossoms, outdoors, falling petals, white hair, black eyes"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "CLIPTextEncode",
|
||||
"pos": [
|
||||
413,
|
||||
389
|
||||
],
|
||||
"size": {
|
||||
"0": 425.27801513671875,
|
||||
"1": 180.6060791015625
|
||||
},
|
||||
"flags": {},
|
||||
"order": 5,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "clip",
|
||||
"type": "CLIP",
|
||||
"link": 5
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "CONDITIONING",
|
||||
"type": "CONDITIONING",
|
||||
"links": [
|
||||
30,
|
||||
51
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "CLIPTextEncode"
|
||||
},
|
||||
"widgets_values": [
|
||||
"embedding:easynegative, embedding:badhandv4, "
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 26,
|
||||
"type": "AnimateDiffSampler",
|
||||
"pos": [
|
||||
893,
|
||||
712
|
||||
],
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 330
|
||||
},
|
||||
"flags": {},
|
||||
"order": 12,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "motion_module",
|
||||
"type": "MOTION_MODULE",
|
||||
"link": 48,
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "model",
|
||||
"type": "MODEL",
|
||||
"link": 49,
|
||||
"slot_index": 1
|
||||
},
|
||||
{
|
||||
"name": "positive",
|
||||
"type": "CONDITIONING",
|
||||
"link": 50
|
||||
},
|
||||
{
|
||||
"name": "negative",
|
||||
"type": "CONDITIONING",
|
||||
"link": 51
|
||||
},
|
||||
{
|
||||
"name": "latent_image",
|
||||
"type": "LATENT",
|
||||
"link": 47
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "LATENT",
|
||||
"type": "LATENT",
|
||||
"links": [
|
||||
52
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "AnimateDiffSampler"
|
||||
},
|
||||
"widgets_values": [
|
||||
"default",
|
||||
16,
|
||||
345029849956687,
|
||||
"increment",
|
||||
20,
|
||||
8,
|
||||
"euler",
|
||||
"normal",
|
||||
0.4
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "VAEDecode",
|
||||
"pos": [
|
||||
1239,
|
||||
712
|
||||
],
|
||||
"size": {
|
||||
"0": 210,
|
||||
"1": 46
|
||||
},
|
||||
"flags": {},
|
||||
"order": 13,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "samples",
|
||||
"type": "LATENT",
|
||||
"link": 52
|
||||
},
|
||||
{
|
||||
"name": "vae",
|
||||
"type": "VAE",
|
||||
"link": 20
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
19
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VAEDecode"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"type": "AnimateDiffSampler",
|
||||
"pos": [
|
||||
882,
|
||||
192
|
||||
],
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 330
|
||||
},
|
||||
"flags": {},
|
||||
"order": 6,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "motion_module",
|
||||
"type": "MOTION_MODULE",
|
||||
"link": 24,
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "model",
|
||||
"type": "MODEL",
|
||||
"link": 25,
|
||||
"slot_index": 1
|
||||
},
|
||||
{
|
||||
"name": "positive",
|
||||
"type": "CONDITIONING",
|
||||
"link": 29
|
||||
},
|
||||
{
|
||||
"name": "negative",
|
||||
"type": "CONDITIONING",
|
||||
"link": 30
|
||||
},
|
||||
{
|
||||
"name": "latent_image",
|
||||
"type": "LATENT",
|
||||
"link": 35
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "LATENT",
|
||||
"type": "LATENT",
|
||||
"links": [
|
||||
44,
|
||||
53
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "AnimateDiffSampler"
|
||||
},
|
||||
"widgets_values": [
|
||||
"default",
|
||||
16,
|
||||
345029849956687,
|
||||
"increment",
|
||||
20,
|
||||
8,
|
||||
"euler",
|
||||
"normal",
|
||||
1
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"type": "VAELoader",
|
||||
"pos": [
|
||||
27,
|
||||
631
|
||||
],
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 58
|
||||
},
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "VAE",
|
||||
"type": "VAE",
|
||||
"links": [
|
||||
20,
|
||||
54
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VAELoader"
|
||||
},
|
||||
"widgets_values": [
|
||||
"klF8Anime2.ckpt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"type": "AnimateDiffCombine",
|
||||
"pos": [
|
||||
1504,
|
||||
481
|
||||
],
|
||||
"size": [
|
||||
325.7265625,
|
||||
517.7265625
|
||||
],
|
||||
"flags": {},
|
||||
"order": 14,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"link": 19
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "GIF",
|
||||
"type": "GIF",
|
||||
"links": null,
|
||||
"shape": 3
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "AnimateDiffCombine"
|
||||
},
|
||||
"widgets_values": [
|
||||
8,
|
||||
0,
|
||||
true,
|
||||
"AnimateDiff",
|
||||
"image/webp",
|
||||
false,
|
||||
"/view?filename=AnimateDiff_00033_.webp&subfolder=&type=output&format=image%2Fwebp"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 27,
|
||||
"type": "VAEDecode",
|
||||
"pos": [
|
||||
1258,
|
||||
187
|
||||
],
|
||||
"size": {
|
||||
"0": 210,
|
||||
"1": 46
|
||||
},
|
||||
"flags": {},
|
||||
"order": 8,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "samples",
|
||||
"type": "LATENT",
|
||||
"link": 53
|
||||
},
|
||||
{
|
||||
"name": "vae",
|
||||
"type": "VAE",
|
||||
"link": 54
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
56
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VAEDecode"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 28,
|
||||
"type": "AnimateDiffCombine",
|
||||
"pos": [
|
||||
1506,
|
||||
-60
|
||||
],
|
||||
"size": [
|
||||
321.19170831298857,
|
||||
513.0408935546875
|
||||
],
|
||||
"flags": {},
|
||||
"order": 10,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"link": 56
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "GIF",
|
||||
"type": "GIF",
|
||||
"links": null,
|
||||
"shape": 3
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "AnimateDiffCombine"
|
||||
},
|
||||
"widgets_values": [
|
||||
8,
|
||||
0,
|
||||
true,
|
||||
"AnimateDiff",
|
||||
"image/webp",
|
||||
false,
|
||||
"/view?filename=AnimateDiff_00032_.webp&subfolder=&type=output&format=image%2Fwebp"
|
||||
]
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[
|
||||
3,
|
||||
4,
|
||||
1,
|
||||
6,
|
||||
0,
|
||||
"CLIP"
|
||||
],
|
||||
[
|
||||
5,
|
||||
4,
|
||||
1,
|
||||
7,
|
||||
0,
|
||||
"CLIP"
|
||||
],
|
||||
[
|
||||
19,
|
||||
8,
|
||||
0,
|
||||
12,
|
||||
0,
|
||||
"IMAGE"
|
||||
],
|
||||
[
|
||||
20,
|
||||
13,
|
||||
0,
|
||||
8,
|
||||
1,
|
||||
"VAE"
|
||||
],
|
||||
[
|
||||
24,
|
||||
16,
|
||||
0,
|
||||
15,
|
||||
0,
|
||||
"MOTION_MODULE"
|
||||
],
|
||||
[
|
||||
25,
|
||||
4,
|
||||
0,
|
||||
15,
|
||||
1,
|
||||
"MODEL"
|
||||
],
|
||||
[
|
||||
29,
|
||||
6,
|
||||
0,
|
||||
15,
|
||||
2,
|
||||
"CONDITIONING"
|
||||
],
|
||||
[
|
||||
30,
|
||||
7,
|
||||
0,
|
||||
15,
|
||||
3,
|
||||
"CONDITIONING"
|
||||
],
|
||||
[
|
||||
35,
|
||||
20,
|
||||
0,
|
||||
15,
|
||||
4,
|
||||
"LATENT"
|
||||
],
|
||||
[
|
||||
44,
|
||||
15,
|
||||
0,
|
||||
24,
|
||||
0,
|
||||
"*"
|
||||
],
|
||||
[
|
||||
45,
|
||||
24,
|
||||
0,
|
||||
25,
|
||||
0,
|
||||
"*"
|
||||
],
|
||||
[
|
||||
46,
|
||||
25,
|
||||
0,
|
||||
22,
|
||||
0,
|
||||
"LATENT"
|
||||
],
|
||||
[
|
||||
47,
|
||||
22,
|
||||
0,
|
||||
26,
|
||||
4,
|
||||
"LATENT"
|
||||
],
|
||||
[
|
||||
48,
|
||||
16,
|
||||
0,
|
||||
26,
|
||||
0,
|
||||
"MOTION_MODULE"
|
||||
],
|
||||
[
|
||||
49,
|
||||
4,
|
||||
0,
|
||||
26,
|
||||
1,
|
||||
"MODEL"
|
||||
],
|
||||
[
|
||||
50,
|
||||
6,
|
||||
0,
|
||||
26,
|
||||
2,
|
||||
"CONDITIONING"
|
||||
],
|
||||
[
|
||||
51,
|
||||
7,
|
||||
0,
|
||||
26,
|
||||
3,
|
||||
"CONDITIONING"
|
||||
],
|
||||
[
|
||||
52,
|
||||
26,
|
||||
0,
|
||||
8,
|
||||
0,
|
||||
"LATENT"
|
||||
],
|
||||
[
|
||||
53,
|
||||
15,
|
||||
0,
|
||||
27,
|
||||
0,
|
||||
"LATENT"
|
||||
],
|
||||
[
|
||||
54,
|
||||
13,
|
||||
0,
|
||||
27,
|
||||
1,
|
||||
"VAE"
|
||||
],
|
||||
[
|
||||
56,
|
||||
27,
|
||||
0,
|
||||
28,
|
||||
0,
|
||||
"IMAGE"
|
||||
]
|
||||
],
|
||||
"groups": [],
|
||||
"config": {},
|
||||
"extra": {},
|
||||
"version": 0.4
|
||||
}
|
||||
@@ -0,0 +1,502 @@
|
||||
{
|
||||
"last_node_id": 21,
|
||||
"last_link_id": 36,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 6,
|
||||
"type": "CLIPTextEncode",
|
||||
"pos": [
|
||||
415,
|
||||
186
|
||||
],
|
||||
"size": {
|
||||
"0": 422.84503173828125,
|
||||
"1": 164.31304931640625
|
||||
},
|
||||
"flags": {},
|
||||
"order": 5,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "clip",
|
||||
"type": "CLIP",
|
||||
"link": 3
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "CONDITIONING",
|
||||
"type": "CONDITIONING",
|
||||
"links": [
|
||||
29
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "CLIPTextEncode"
|
||||
},
|
||||
"widgets_values": [
|
||||
"masterpiece, best quality, 1girl, solo, cherry blossoms, hanami, pink flower, white flower, spring season, wisteria, petals, flower, plum blossoms, outdoors, falling petals, white hair, black eyes"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "VAEDecode",
|
||||
"pos": [
|
||||
1253,
|
||||
191
|
||||
],
|
||||
"size": {
|
||||
"0": 210,
|
||||
"1": 46
|
||||
},
|
||||
"flags": {},
|
||||
"order": 8,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "samples",
|
||||
"type": "LATENT",
|
||||
"link": 28
|
||||
},
|
||||
{
|
||||
"name": "vae",
|
||||
"type": "VAE",
|
||||
"link": 20
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
19
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VAEDecode"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"type": "AnimateDiffCombine",
|
||||
"pos": [
|
||||
1254,
|
||||
290
|
||||
],
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 342
|
||||
},
|
||||
"flags": {},
|
||||
"order": 9,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"link": 19
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "AnimateDiffCombine"
|
||||
},
|
||||
"widgets_values": [
|
||||
8,
|
||||
0,
|
||||
false,
|
||||
"AnimateDiff",
|
||||
"image/gif",
|
||||
false
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 16,
|
||||
"type": "AnimateDiffModuleLoader",
|
||||
"pos": [
|
||||
27,
|
||||
345
|
||||
],
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 58
|
||||
},
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "MOTION_MODULE",
|
||||
"type": "MOTION_MODULE",
|
||||
"links": [
|
||||
24
|
||||
],
|
||||
"shape": 3
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "AnimateDiffModuleLoader"
|
||||
},
|
||||
"widgets_values": [
|
||||
"mm-Stabilized_mid.pth"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "CheckpointLoaderSimple",
|
||||
"pos": [
|
||||
26,
|
||||
474
|
||||
],
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 98
|
||||
},
|
||||
"flags": {},
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "MODEL",
|
||||
"type": "MODEL",
|
||||
"links": [
|
||||
25
|
||||
],
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "CLIP",
|
||||
"type": "CLIP",
|
||||
"links": [
|
||||
3,
|
||||
5
|
||||
],
|
||||
"slot_index": 1
|
||||
},
|
||||
{
|
||||
"name": "VAE",
|
||||
"type": "VAE",
|
||||
"links": [],
|
||||
"slot_index": 2
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "CheckpointLoaderSimple"
|
||||
},
|
||||
"widgets_values": [
|
||||
"AnimeLike25D_v11.safetensors"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"type": "VAELoader",
|
||||
"pos": [
|
||||
28,
|
||||
223
|
||||
],
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 58
|
||||
},
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "VAE",
|
||||
"type": "VAE",
|
||||
"links": [
|
||||
20
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VAELoader"
|
||||
},
|
||||
"widgets_values": [
|
||||
"klF8Anime2.ckpt"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "CLIPTextEncode",
|
||||
"pos": [
|
||||
413,
|
||||
389
|
||||
],
|
||||
"size": {
|
||||
"0": 425.27801513671875,
|
||||
"1": 180.6060791015625
|
||||
},
|
||||
"flags": {},
|
||||
"order": 6,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "clip",
|
||||
"type": "CLIP",
|
||||
"link": 5
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "CONDITIONING",
|
||||
"type": "CONDITIONING",
|
||||
"links": [
|
||||
30
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "CLIPTextEncode"
|
||||
},
|
||||
"widgets_values": [
|
||||
"embedding:easynegative, embedding:badhandv4, "
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 20,
|
||||
"type": "EmptyLatentImage",
|
||||
"pos": [
|
||||
522,
|
||||
621
|
||||
],
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 106
|
||||
},
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "LATENT",
|
||||
"type": "LATENT",
|
||||
"links": [
|
||||
35
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "EmptyLatentImage"
|
||||
},
|
||||
"widgets_values": [
|
||||
512,
|
||||
512,
|
||||
1
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 21,
|
||||
"type": "AnimateDiffSlidingWindowOptions",
|
||||
"pos": [
|
||||
517,
|
||||
-34
|
||||
],
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 154
|
||||
},
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "SLIDING_WINDOW_OPTS",
|
||||
"type": "SLIDING_WINDOW_OPTS",
|
||||
"links": [
|
||||
36
|
||||
],
|
||||
"shape": 3
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "AnimateDiffSlidingWindowOptions"
|
||||
},
|
||||
"widgets_values": [
|
||||
16,
|
||||
1,
|
||||
4,
|
||||
"uniform",
|
||||
true
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"type": "AnimateDiffSampler",
|
||||
"pos": [
|
||||
882,
|
||||
192
|
||||
],
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 350
|
||||
},
|
||||
"flags": {},
|
||||
"order": 7,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "motion_module",
|
||||
"type": "MOTION_MODULE",
|
||||
"link": 24,
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "model",
|
||||
"type": "MODEL",
|
||||
"link": 25,
|
||||
"slot_index": 1
|
||||
},
|
||||
{
|
||||
"name": "positive",
|
||||
"type": "CONDITIONING",
|
||||
"link": 29
|
||||
},
|
||||
{
|
||||
"name": "negative",
|
||||
"type": "CONDITIONING",
|
||||
"link": 30
|
||||
},
|
||||
{
|
||||
"name": "latent_image",
|
||||
"type": "LATENT",
|
||||
"link": 35
|
||||
},
|
||||
{
|
||||
"name": "sliding_window_opts",
|
||||
"type": "SLIDING_WINDOW_OPTS",
|
||||
"link": 36,
|
||||
"slot_index": 5
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "LATENT",
|
||||
"type": "LATENT",
|
||||
"links": [
|
||||
28
|
||||
],
|
||||
"shape": 3,
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "AnimateDiffSampler"
|
||||
},
|
||||
"widgets_values": [
|
||||
"default",
|
||||
40,
|
||||
345029849956677,
|
||||
"fixed",
|
||||
20,
|
||||
8,
|
||||
"euler",
|
||||
"normal",
|
||||
0.8
|
||||
]
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[
|
||||
3,
|
||||
4,
|
||||
1,
|
||||
6,
|
||||
0,
|
||||
"CLIP"
|
||||
],
|
||||
[
|
||||
5,
|
||||
4,
|
||||
1,
|
||||
7,
|
||||
0,
|
||||
"CLIP"
|
||||
],
|
||||
[
|
||||
19,
|
||||
8,
|
||||
0,
|
||||
12,
|
||||
0,
|
||||
"IMAGE"
|
||||
],
|
||||
[
|
||||
20,
|
||||
13,
|
||||
0,
|
||||
8,
|
||||
1,
|
||||
"VAE"
|
||||
],
|
||||
[
|
||||
24,
|
||||
16,
|
||||
0,
|
||||
15,
|
||||
0,
|
||||
"MOTION_MODULE"
|
||||
],
|
||||
[
|
||||
25,
|
||||
4,
|
||||
0,
|
||||
15,
|
||||
1,
|
||||
"MODEL"
|
||||
],
|
||||
[
|
||||
28,
|
||||
15,
|
||||
0,
|
||||
8,
|
||||
0,
|
||||
"LATENT"
|
||||
],
|
||||
[
|
||||
29,
|
||||
6,
|
||||
0,
|
||||
15,
|
||||
2,
|
||||
"CONDITIONING"
|
||||
],
|
||||
[
|
||||
30,
|
||||
7,
|
||||
0,
|
||||
15,
|
||||
3,
|
||||
"CONDITIONING"
|
||||
],
|
||||
[
|
||||
35,
|
||||
20,
|
||||
0,
|
||||
15,
|
||||
4,
|
||||
"LATENT"
|
||||
],
|
||||
[
|
||||
36,
|
||||
21,
|
||||
0,
|
||||
15,
|
||||
5,
|
||||
"SLIDING_WINDOW_OPTS"
|
||||
]
|
||||
],
|
||||
"groups": [],
|
||||
"config": {},
|
||||
"extra": {},
|
||||
"version": 0.4
|
||||
}
|
||||
Reference in New Issue
Block a user