init commit
This commit is contained in:
root
@@ -0,0 +1,109 @@
|
||||
## ComfyUI Nodes
|
||||
|
||||
After installation, you can find the three nodes provided by this repository in the *Add Node - Ruyi* menu, as shown in the image below:
|
||||
|
||||
<div align=center>
|
||||
<img src="https://github.com/user-attachments/assets/0760399c-c57f-465d-9685-ef910b60421c"></img>
|
||||
</div>
|
||||
|
||||
The following sections will introduce the functions and parameters of each node.
|
||||
|
||||
> Note: The new version of ComfyUI nodes is displayed in the NODE LIBRARY on the left side of the interface.
|
||||
|
||||
### Load Model
|
||||
|
||||
The Load Model node is used to load the model from disk. It also provides the functionality for **automatic model downloading** (auto_download parameter).
|
||||
|
||||
<div align=center>
|
||||
<img src="https://github.com/user-attachments/assets/0ec48688-30fa-40a1-9ad4-862549a70730"></img>
|
||||
</div>
|
||||
|
||||
- **model**: Select which model to use. Currently, Ruyi-Mini-7B is the only option.
|
||||
- **auto_download**: Whether to automatically download. Defaults to yes. If the model is detected as missing or incomplete, it will automatically download the model to the *ComfyUI/models/Ruyi* path.
|
||||
|
||||
### Load LoRA
|
||||
|
||||
The Load LoRA node is used to load LoRA models, which need to be placed in the *ComfyUI/models/loras* path.
|
||||
|
||||
<div align=center>
|
||||
<img src="https://github.com/user-attachments/assets/33f860ca-94c9-4686-afe4-7e00de15b0ce"></img>
|
||||
</div>
|
||||
|
||||
- **lora_name**: The LoRA to be loaded; it will automatically search and display all model files in the *ComfyUI/models/loras* path.
|
||||
- **strength_model**: The degree of influence of the LoRA, typically set between 1.0 and 1.4 for optimal results based on experience.
|
||||
|
||||
### Sampler for Image to Video
|
||||
|
||||
The Sampler for Image to Video node is used to generate videos based on input images. The starting frame image (start_img) is a required input, while the ending frame image (end_img) is optional. This node also supports **camera control** (camera_direction parameter) and **motion amplitude control** of the video subject (motion parameter).
|
||||
|
||||
<div align=center>
|
||||
<img src="https://github.com/user-attachments/assets/e5078e95-7100-4e9b-9baa-0d534b085e7a"></img>
|
||||
</div>
|
||||
|
||||
- **start_img**: The starting frame image.
|
||||
- **end_img**: The ending frame image, optional input.
|
||||
- **video_length**: The length of the video, which must be divisible by 8, with a maximum of 120 frames.
|
||||
- **base_resolution**: The video resolution, such as 512, indicates that the generated video will have pixel dimensions close to 512 x 512. The model will automatically select the closest output video aspect ratio based on the input image.
|
||||
- **seed**: A random number; different random numbers usually generate different videos. If the generated video does not meet requirements, this value can be adjusted to try other generation possibilities.
|
||||
- **control_after_generate**: The method of changing the random number after each generation.
|
||||
- **Fixed** indicates the seed is fixed.
|
||||
- **Increment** indicates the seed is increased by one each time.
|
||||
- **Decrement** indicates the seed is decreased by one each time.
|
||||
- **Randomize** indicates the seed is randomly set each time.
|
||||
- **steps**: The number of iterations for video generation. **More iterations require more time**. Typically, 25 iterations yield good results.
|
||||
- **cfg**: The guidance of instructions (such as input images). A higher value indicates better guidance, with values between 7 and 10 usually achieving better generation results.
|
||||
- **motion**: Controls the motion amplitude of the video subject.
|
||||
- **1** indicates minimal motion, which means the video subject is nearly static.
|
||||
- **2** indicates normal motion, which could be used in most case.
|
||||
- **3** indicates significant motion. The video subject is trying to move as much as possible.
|
||||
- **4** indicates a very large motion. Sometimes the video subject may move out of the camera frame.
|
||||
- **Auto** indicates the motion is automatically determined by the model.
|
||||
- **camera_direction**: Camera movement.
|
||||
- **Static** indicates a stationary camera.
|
||||
- **Left** indicates the camera moves left.
|
||||
- **Right** indicates the camera moves right.
|
||||
- **Up** indicates the camera moves up.
|
||||
- **Down** indicates the camera moves down.
|
||||
- **Auto** indicates automatic determination.
|
||||
- **GPU_memory_mode**: Determines how GPU memory is utilized.
|
||||
- **normal_mode** is the default mode, using more GPU memory and generating faster.
|
||||
- **low_memory_mode** is the low memory mode, which significantly reduces GPU memory usage but severely impacts generation speed.
|
||||
- **GPU_offload_steps**: Used to **optimize GPU memory usage** by moving some temporary variables from GPU memory to RAM, which **increases memory usage and decreases generation speed**.
|
||||
- **0** indicates no optimization.
|
||||
- **1 - 10**, where 1 has the least GPU memory usage and the slowest generation speed; 10 has the most GPU memory usage (less than the non-optimized case) and the fastest generation speed.
|
||||
- Generally, with 24GB of GPU memory, you can use 7 to generate a 512 resolution, 120 frame video. For more detailed data, please refer to the following.
|
||||
|
||||
## Workflow Example
|
||||
|
||||
This section presents an example workflow for generating videos from images. You can import the workflow using the *Load* button in the (bottom right) menu. Note that the new version of ComfyUI move the menu to the top left, and allows you to load workflows through the *Workflow - Open* option.
|
||||
|
||||
The workflows are located in the *[comfyui/workflows/](workflows/)* directory, while the assets can be found in the *[assets/](../assets/)* directory.
|
||||
|
||||
After importing the workflow, you **need to manually re-specify the input image for the LoadImage input node**. Since the workflow file can only record the names of input files, it does require manual configuration.
|
||||
|
||||
### Image to Video (Starting Frame)
|
||||
|
||||
The workflow corresponds to the *[workflow-ruyi-i2v-start-frame.json](workflows/workflow-ruyi-i2v-start-frame.json)* file. For users with larger GPU memory, you can also use *[workflow-ruyi-i2v-start-frame-80g.json](workflows/workflow-ruyi-i2v-start-frame-80g.json)* to enhance the generation speed.
|
||||
|
||||
<div align=center>
|
||||
<img style="width:80%" src="https://github.com/user-attachments/assets/4c0a58b8-ea04-4656-bf1f-b8665a3802a3"></img>
|
||||
</div>
|
||||
|
||||
### Image to Video (Starting and Ending Frames)
|
||||
|
||||
The workflow corresponds to the *[workflow-ruyi-i2v-start-end-frames.json](workflows/workflow-ruyi-i2v-start-end-frames.json)* file. For users with larger GPU memory, you can also use *[workflow-ruyi-i2v-start-end-frames-80g.json](workflows/workflow-ruyi-i2v-start-end-frames-80g.json)* to enhance the generation speed.
|
||||
|
||||
<div align=center>
|
||||
<img style="width:80%" src="https://github.com/user-attachments/assets/42b685a4-35ad-4dd8-afa2-79ded3e936cd"></img>
|
||||
</div>
|
||||
|
||||
## Frequently Asked Questions
|
||||
|
||||
### Model loading error: LoadModel: ConnectionError (MaxRetryError)
|
||||
|
||||
This is usually caused by network issues leading to a failure in downloading from huggingface_hub. If the network is functioning properly, simply rerunning the LoadModel node should resolve the issue.
|
||||
|
||||
### Video generation speed is slow and far below expectations
|
||||
|
||||
- First, check if the low_memory_mode in the Load Model node is enabled. This mode significantly reduces video generation speed.
|
||||
- Second, verify the version of PyTorch. PyTorch version 2.2 supports FlashAttention-2 ([link](https://pytorch.org/blog/pytorch2-2/)), which can greatly enhance computational efficiency. Installing the latest version of PyTorch can effectively improve generation speed.
|
||||
@@ -0,0 +1,111 @@
|
||||
## 节点功能
|
||||
|
||||
安装完成后，可在 ComfyUI 的 *Add Node - Ruyi* 菜单找到本仓库提供的 3 个节点，如下图所示：
|
||||
|
||||
<div align=center>
|
||||
<img src="https://github.com/user-attachments/assets/0760399c-c57f-465d-9685-ef910b60421c"></img>
|
||||
</div>
|
||||
|
||||
下文依次介绍各个节点的功能与参数。
|
||||
|
||||
> 注：新版 ComfyUI 节点在界面左侧 NODE LIBRARY 中显示。
|
||||
|
||||
### Load Model
|
||||
|
||||
用于加载模型，并提供**自动下载模型**（通过 auto_download 选项设置）的功能。
|
||||
|
||||
<div align=center>
|
||||
<img src="https://github.com/user-attachments/assets/0ec48688-30fa-40a1-9ad4-862549a70730"></img>
|
||||
</div>
|
||||
|
||||
- **model**: 选择使用哪个模型，目前只有 Ruyi-Mini-7B 一个选项。
|
||||
- **auto_download**: 是否自动下载，默认为 yes，检测到模型不存在（或不完整）时，将自动下载模型到 *ComfyUI/models/Ruyi* 路径。
|
||||
|
||||
### Load LoRA
|
||||
|
||||
用于加载 LoRA 模型，LoRA 模型需要放在 *ComfyUI/models/loras* 路径下。
|
||||
|
||||
<div align=center>
|
||||
<img src="https://github.com/user-attachments/assets/33f860ca-94c9-4686-afe4-7e00de15b0ce"></img>
|
||||
</div>
|
||||
|
||||
- **lora_name**: 需要加载的 LoRA，将自动搜索并显示 *ComfyUI/models/loras* 路径下所有模型文件。
|
||||
- **strength_model**: LoRA 的影响程度，根据经验通常设置在 1.0 ~ 1.4 效果较好。
|
||||
|
||||
### Sampler for Image to Video
|
||||
|
||||
用于根据输入图片生成视频，首帧图片（start_img）为必须输入，尾帧图片（end_img）为可选输入。同时，该节点支持**镜头控制**（camera_direction）与**视频主体的运动幅度控制**（motion）。
|
||||
|
||||
<div align=center>
|
||||
<img src="https://github.com/user-attachments/assets/e5078e95-7100-4e9b-9baa-0d534b085e7a"></img>
|
||||
</div>
|
||||
|
||||
- **start_img**: 首帧图片。
|
||||
- **end_img**: 尾帧图片，可选输入。
|
||||
- **video_length**: 视频长度，必须能**被 8 整除**，**最大支持 120 帧**。
|
||||
- **base_resolution**: 视频分辨率，**模型将根据输入图片的长宽比自动选择最接近的输出视频长宽**。
|
||||
- **512** 表示生成的视频像素数接近 512 * 512。
|
||||
- **768** 表示生成的视频像素数接近 768 * 768。
|
||||
- **seed**: 随机种子，用于控制随机数生成器产生随机数的序列。**不同的随机种子通常能生成不同的视频**，当生成的视频不符合需求时候，可调整此值以尝试其他的生成可能。
|
||||
- **control_after_generate**: 每次生成后随机种子的变化方式。
|
||||
- **Fixed** 表示随机种子固定不变。
|
||||
- **Increment** 表示随机种子每次增加一。
|
||||
- **Decrement** 表示随机种子每次减少一。
|
||||
- **Randomize** 表示随机种子运行后随机设置。
|
||||
- **steps**: 视频生成的迭代次数，**迭代次数越多，需要的时间越久**，通常 25 次能够得到较好的结果。
|
||||
- **cfg**: 指令（如输入图片）的遵循程度，数值越大遵循程度越好，取值 7 ～ 10 通常能取得较好的生成效果。
|
||||
- **motion**: 控制视频主体的运动幅度。
|
||||
- **1** 基本不运动，适用于静态场景。
|
||||
- **2** 正常运动幅度，适用于谈话、转头等常见场合。
|
||||
- **3** 运动幅度较大，可能出现转身、走动等情况。
|
||||
- **4** 运动幅度非常大，可能出现视频主体离开画面的情况。
|
||||
- **Auto** 表示模型自动判断运动幅度大小。
|
||||
- **camera_direction**: 镜头运动。
|
||||
- **Static** 表示静止镜头。
|
||||
- **Left** 表示镜头向左移动。
|
||||
- **Right** 表示镜头向右移动。
|
||||
- **Up** 表示镜头向上移动。
|
||||
- **Down** 表示镜头向下移动。
|
||||
- **Auto** 表示模型自动判断镜头运动方向。
|
||||
- **GPU_memory_mode**:
|
||||
- **normal_mode** 是默认模式，**使用显存较多，生成速度较快**。
|
||||
- **low_memory_mode** 是低显存模式，**能大幅降低显存用量，但严重影响生成速度**。
|
||||
- **GPU_offload_steps**: 用于**优化显存占用**，通过将部分临时变量从显存移动到内存而实现，会**增加内存的占用并降低生成速度**。
|
||||
- **0** 表示不优化。
|
||||
- **1 - 10**，1 显存占用最小，生成速度最慢；10 显存占用最多（少于不优化情况），生成速度最快。
|
||||
- 通常情况下，24G 显存可以使用 7 生成 512 分辨率，120 帧视频。更详细数据请参考下文。
|
||||
|
||||
## 工作流样例
|
||||
|
||||
本节展示了图生视频的工作流样例，可通过菜单中的 *Load* 按钮导入工作流。新版 ComfyUI 可通过左上方菜单的 *Workflow - Open* 加载工作流。
|
||||
|
||||
工作流位于 *[comfyui/workflows/](workflows/)* 目录中，素材位于 *[assets/](../assets/)* 目录中。
|
||||
|
||||
导入工作流后，**需要手动重新指定输入节点 LoadImage 的输入图片**，由于工作流文件只能记录输入文件的名字，所以目前只能手动设置。
|
||||
|
||||
### 图生视频（首帧）
|
||||
|
||||
工作流对应 *[workflow-ruyi-i2v-start-frame.json](workflows/workflow-ruyi-i2v-start-frame.json)* 文件。对于显存较大的用户，也可以使用 *[workflow-ruyi-i2v-start-frame-80g.json](workflows/workflow-ruyi-i2v-start-frame-80g.json)* 以提高生成速度。
|
||||
|
||||
<div align=center>
|
||||
<img style="width:80%" src="https://github.com/user-attachments/assets/4c0a58b8-ea04-4656-bf1f-b8665a3802a3"></img>
|
||||
</div>
|
||||
|
||||
### 图生视频（首尾帧）
|
||||
|
||||
工作流对应 *[workflow-ruyi-i2v-start-end-frames.json](workflows/workflow-ruyi-i2v-start-end-frames.json)* 文件。对于显存较大的用户，也可以使用 *[workflow-ruyi-i2v-start-end-frames-80g.json](workflows/workflow-ruyi-i2v-start-end-frames-80g.json)* 以提高生成速度。
|
||||
|
||||
<div align=center>
|
||||
<img style="width:80%" src="https://github.com/user-attachments/assets/42b685a4-35ad-4dd8-afa2-79ded3e936cd"></img>
|
||||
</div>
|
||||
|
||||
## 常见问题
|
||||
|
||||
### 模型加载错误，LoadModel: ConnectionError (MaxRetryError)
|
||||
|
||||
通常是网络问题导致 huggingface_hub 下载失败。网络正常的情况下，再次运行 LoadModel 节点即可解决。
|
||||
|
||||
### 视频生成速度慢、远低于预期
|
||||
|
||||
- 首先，请检查是否开启 Load Model 节点中 GPU_memory_mode 的 low_memory_mode，此模式会大幅降低视频生成速度。
|
||||
- 其次，请检查 PyTorch 版本。PyTorch 在 2.2 版本支持了 FlashAttention-2（[链接](https://pytorch.org/blog/pytorch2-2/)），能大幅提升计算效率。安装新版本的 PyTorch 能够有效提升生成速度。
|
||||
@@ -0,0 +1,426 @@
|
||||
"""Modified from https://github.com/kijai/ComfyUI-EasyAnimateWrapper/blob/main/nodes.py
|
||||
"""
|
||||
import gc
|
||||
import os
|
||||
|
||||
import comfy.model_management as mm
|
||||
import cv2
|
||||
import folder_paths
|
||||
import numpy as np
|
||||
import torch
|
||||
from comfy.utils import ProgressBar
|
||||
from diffusers import (DDIMScheduler, DPMSolverMultistepScheduler, PNDMScheduler,
|
||||
EulerAncestralDiscreteScheduler, EulerDiscreteScheduler)
|
||||
from einops import rearrange
|
||||
from omegaconf import OmegaConf
|
||||
from PIL import Image
|
||||
from transformers import CLIPImageProcessor, CLIPVisionModelWithProjection
|
||||
from safetensors.torch import load_file as load_safetensors
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
from ..ruyi.data.bucket_sampler import ASPECT_RATIO_512, get_closest_ratio
|
||||
from ..ruyi.models.autoencoder_magvit import AutoencoderKLMagvit
|
||||
from ..ruyi.models.transformer3d import HunyuanTransformer3DModel
|
||||
from ..ruyi.pipeline.pipeline_ruyi_inpaint import RuyiInpaintPipeline
|
||||
from ..ruyi.utils.lora_utils import merge_lora, unmerge_lora
|
||||
from ..ruyi.utils.utils import get_image_to_video_latent
|
||||
|
||||
# The directory of scripts
|
||||
script_directory = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
|
||||
class Ruyi_LoadModel:
|
||||
@classmethod
|
||||
def INPUT_TYPES(s):
|
||||
return {
|
||||
"required": {
|
||||
"model": (
|
||||
[
|
||||
"Ruyi-Mini-7B",
|
||||
],
|
||||
{ "default": "Ruyi-Mini-7B", }
|
||||
),
|
||||
"auto_download": (
|
||||
["yes", "no"],
|
||||
{
|
||||
"default": "yes"
|
||||
}
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("RUYI_MODEL",)
|
||||
RETURN_NAMES = ("ruyi_model",)
|
||||
FUNCTION = "load_model"
|
||||
CATEGORY = "Ruyi"
|
||||
|
||||
RUYI_MODEL_TYPE_DICT = {
|
||||
"Ruyi-Mini-7B": "Inpaint"
|
||||
}
|
||||
|
||||
def get_model_type(self, model_name):
|
||||
return self.RUYI_MODEL_TYPE_DICT.get(model_name, "Inpaint")
|
||||
|
||||
def try_setup_pipeline(self, model_path, weight_dtype, config):
|
||||
try:
|
||||
# Init processbar
|
||||
pbar = ProgressBar(5)
|
||||
|
||||
# Get Vae
|
||||
vae = AutoencoderKLMagvit.from_pretrained(
|
||||
model_path,
|
||||
subfolder="vae"
|
||||
).to(weight_dtype)
|
||||
# Update pbar
|
||||
pbar.update(1)
|
||||
|
||||
# Get Transformer
|
||||
transformer_additional_kwargs = OmegaConf.to_container(config['transformer_additional_kwargs'])
|
||||
transformer = HunyuanTransformer3DModel.from_pretrained_2d(
|
||||
model_path,
|
||||
subfolder="transformer",
|
||||
transformer_additional_kwargs=transformer_additional_kwargs
|
||||
).to(weight_dtype)
|
||||
# Update pbar
|
||||
pbar.update(1)
|
||||
|
||||
# Load Clip
|
||||
clip_image_encoder = CLIPVisionModelWithProjection.from_pretrained(
|
||||
model_path, subfolder="image_encoder"
|
||||
).to(weight_dtype)
|
||||
clip_image_processor = CLIPImageProcessor.from_pretrained(
|
||||
model_path, subfolder="image_encoder"
|
||||
)
|
||||
# Update pbar
|
||||
pbar.update(1)
|
||||
|
||||
# Load sampler and create pipeline
|
||||
Choosen_Scheduler = DDIMScheduler
|
||||
scheduler = Choosen_Scheduler.from_pretrained(
|
||||
model_path,
|
||||
subfolder="scheduler"
|
||||
)
|
||||
pipeline = RuyiInpaintPipeline.from_pretrained(
|
||||
model_path,
|
||||
vae=vae,
|
||||
transformer=transformer,
|
||||
scheduler=scheduler,
|
||||
torch_dtype=weight_dtype,
|
||||
clip_image_encoder=clip_image_encoder,
|
||||
clip_image_processor=clip_image_processor,
|
||||
)
|
||||
|
||||
# Load embeddings
|
||||
embeddings = load_safetensors(os.path.join(model_path, "embeddings.safetensors"))
|
||||
pipeline.embeddings = embeddings
|
||||
# Update pbar
|
||||
pbar.update(1)
|
||||
|
||||
return pipeline
|
||||
except Exception as e:
|
||||
print("[Ruyi] Setup pipeline failed:", e)
|
||||
return None
|
||||
|
||||
def load_model(self, model, auto_download):
|
||||
# Init weight_dtype and device
|
||||
device = mm.get_torch_device()
|
||||
offload_device = mm.unet_offload_device()
|
||||
|
||||
# Init model name and type
|
||||
model_name = model
|
||||
model_type = self.get_model_type(model_name)
|
||||
weight_dtype = torch.bfloat16
|
||||
|
||||
# Load config
|
||||
config_path = os.path.join(script_directory, "config", "default.yaml")
|
||||
config = OmegaConf.load(config_path)
|
||||
|
||||
# Init model
|
||||
repo_id = f"IamCreateAI/{model_name}"
|
||||
model_path = os.path.join(folder_paths.models_dir, "Ruyi", model_name)
|
||||
pipeline = self.try_setup_pipeline(model_path, weight_dtype, config)
|
||||
if pipeline is None and auto_download == "yes":
|
||||
mm.soft_empty_cache()
|
||||
gc.collect()
|
||||
|
||||
# Download the model
|
||||
snapshot_download(repo_id=repo_id, local_dir=model_path)
|
||||
|
||||
pipeline = self.try_setup_pipeline(model_path, weight_dtype, config)
|
||||
|
||||
if pipeline is None:
|
||||
message = (f"[ Load Model {model_name} Failed ]\n"
|
||||
f"Please download Ruyi model from huggingface repo '{repo_id}',\n"
|
||||
f"And put it into '{model_path}'.")
|
||||
if auto_download == "no":
|
||||
message += "\n\nOr just set auto_download to 'yes'."
|
||||
raise FileNotFoundError(message)
|
||||
|
||||
pipeline.enable_model_cpu_offload()
|
||||
ruyi_model = {
|
||||
'pipeline': pipeline,
|
||||
'dtype': weight_dtype,
|
||||
'model_path': model_path,
|
||||
'model_type': model_type,
|
||||
'loras': [],
|
||||
'strength_model': [],
|
||||
}
|
||||
return (ruyi_model,)
|
||||
|
||||
|
||||
class Ruyi_LoadLora:
|
||||
@classmethod
|
||||
def INPUT_TYPES(s):
|
||||
return {
|
||||
"required": {
|
||||
"ruyi_model": ("RUYI_MODEL",),
|
||||
"lora_name": (folder_paths.get_filename_list("loras"), {"default": None,}),
|
||||
"strength_model": ("FLOAT", {"default": 1.0, "min": -100.0, "max": 100.0, "step": 0.01}),
|
||||
}
|
||||
}
|
||||
RETURN_TYPES = ("RUYI_MODEL",)
|
||||
RETURN_NAMES = ("ruyi_model",)
|
||||
FUNCTION = "load_lora"
|
||||
CATEGORY = "Ruyi"
|
||||
|
||||
def load_lora(self, ruyi_model, lora_name, strength_model):
|
||||
if lora_name is not None:
|
||||
return (
|
||||
{
|
||||
'pipeline': ruyi_model["pipeline"],
|
||||
'dtype': ruyi_model["dtype"],
|
||||
'model_path': ruyi_model["model_path"],
|
||||
'model_type': ruyi_model["model_type"],
|
||||
'loras': ruyi_model.get("loras", []) + [folder_paths.get_full_path("loras", lora_name)],
|
||||
'strength_model': ruyi_model.get("strength_model", []) + [strength_model],
|
||||
},
|
||||
)
|
||||
else:
|
||||
return (ruyi_model,)
|
||||
|
||||
|
||||
class Ruyi_I2VSampler:
|
||||
@classmethod
|
||||
def INPUT_TYPES(s):
|
||||
return {
|
||||
"required": {
|
||||
"ruyi_model": (
|
||||
"RUYI_MODEL",
|
||||
),
|
||||
"video_length": (
|
||||
"INT", {"default": 72, "min": 8, "max": 120, "step": 8}
|
||||
),
|
||||
"base_resolution": (
|
||||
"INT", {"default": 512, "min": 384, "max": 1024, "step": 16}
|
||||
),
|
||||
"seed": (
|
||||
"INT", {"default": 42, "min": 0, "max": 0xffffffffffffffff}
|
||||
),
|
||||
"steps": (
|
||||
"INT", {"default": 25, "min": 1, "max": 200, "step": 1}
|
||||
),
|
||||
"cfg": (
|
||||
"FLOAT", {"default": 7.0, "min": 1.0, "max": 20.0, "step": 0.01}
|
||||
),
|
||||
"scheduler": (
|
||||
[
|
||||
"Euler",
|
||||
"Euler A",
|
||||
"DPM++",
|
||||
"PNDM",
|
||||
"DDIM",
|
||||
],
|
||||
{
|
||||
"default": 'DDIM'
|
||||
}
|
||||
),
|
||||
"motion": (
|
||||
[ "1", "2", "3", "4", "auto" ],
|
||||
{ "default": "2" }
|
||||
),
|
||||
"camera_direction": (
|
||||
[ "static", "left", "right", "up", "down", "auto" ],
|
||||
{ "default": "static" }
|
||||
),
|
||||
"GPU_memory_mode": (
|
||||
["normal_mode", "low_memory_mode"],
|
||||
{
|
||||
"default": "normal_mode",
|
||||
}
|
||||
),
|
||||
"GPU_offload_steps": (
|
||||
[ "0", "1", "5", "7", "10" ],
|
||||
{ "default": "0" }
|
||||
),
|
||||
"start_img": (
|
||||
"IMAGE",
|
||||
),
|
||||
},
|
||||
"optional":{
|
||||
"end_img": ("IMAGE",),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("IMAGE",)
|
||||
RETURN_NAMES =("images",)
|
||||
FUNCTION = "process"
|
||||
CATEGORY = "Ruyi"
|
||||
|
||||
def tensor2pil(self, image):
|
||||
return Image.fromarray(np.clip(255. * image.cpu().numpy(), 0, 255).astype(np.uint8))
|
||||
|
||||
def numpy2pil(self, image):
|
||||
return Image.fromarray(np.clip(255. * image, 0, 255).astype(np.uint8))
|
||||
|
||||
def to_pil(self, image):
|
||||
if isinstance(image, Image.Image):
|
||||
return image
|
||||
if isinstance(image, torch.Tensor):
|
||||
return self.tensor2pil(image)
|
||||
if isinstance(image, np.ndarray):
|
||||
return self.numpy2pil(image)
|
||||
raise ValueError(f"Cannot convert {type(image)} to PIL.Image")
|
||||
|
||||
def get_control_embeddings(self, pipeline, aspect_ratio, motion, camera_direction):
|
||||
# Default keys
|
||||
p_default_key = "p.default"
|
||||
n_default_key = "n.default"
|
||||
|
||||
# Load embeddings
|
||||
if motion == "auto":
|
||||
motion = "0"
|
||||
p_key = f"p.{aspect_ratio.replace(':', 'x')}movie{motion}{camera_direction}"
|
||||
embeddings = pipeline.embeddings
|
||||
|
||||
# Get embeddings
|
||||
positive_embeds = embeddings.get(f"{p_key}.emb1", embeddings[f"{p_default_key}.emb1"])
|
||||
positive_attention_mask = embeddings.get(f"{p_key}.mask1", embeddings[f"{p_default_key}.mask1"])
|
||||
positive_embeds_2 = embeddings.get(f"{p_key}.emb2", embeddings[f"{p_default_key}.emb2"])
|
||||
positive_attention_mask_2 = embeddings.get(f"{p_key}.mask2", embeddings[f"{p_default_key}.mask2"])
|
||||
|
||||
negative_embeds = embeddings[f"{n_default_key}.emb1"]
|
||||
negative_attention_mask = embeddings[f"{n_default_key}.mask1"]
|
||||
negative_embeds_2 = embeddings[f"{n_default_key}.emb2"]
|
||||
negative_attention_mask_2 = embeddings[f"{n_default_key}.mask2"]
|
||||
|
||||
return {
|
||||
"positive_embeds": positive_embeds,
|
||||
"positive_attention_mask": positive_attention_mask,
|
||||
"positive_embeds_2": positive_embeds_2,
|
||||
"positive_attention_mask_2": positive_attention_mask_2,
|
||||
|
||||
"negative_embeds": negative_embeds,
|
||||
"negative_attention_mask": negative_attention_mask,
|
||||
"negative_embeds_2": negative_embeds_2,
|
||||
"negative_attention_mask_2": negative_attention_mask_2,
|
||||
}
|
||||
|
||||
def get_scheduler(self, model_path, scheduler_name):
|
||||
if scheduler_name == "DPM++":
|
||||
noise_scheduler = DPMSolverMultistepScheduler.from_pretrained(model_path, subfolder='scheduler')
|
||||
elif scheduler_name == "Euler":
|
||||
noise_scheduler = EulerDiscreteScheduler.from_pretrained(model_path, subfolder='scheduler')
|
||||
elif scheduler_name == "Euler A":
|
||||
noise_scheduler = EulerAncestralDiscreteScheduler.from_pretrained(model_path, subfolder='scheduler')
|
||||
elif scheduler_name == "PNDM":
|
||||
noise_scheduler = PNDMScheduler.from_pretrained(model_path, subfolder='scheduler')
|
||||
elif scheduler_name == "DDIM":
|
||||
noise_scheduler = DDIMScheduler.from_pretrained(model_path, subfolder='scheduler')
|
||||
|
||||
return noise_scheduler
|
||||
|
||||
def process(
|
||||
self, ruyi_model, video_length, base_resolution, seed, steps, cfg, scheduler,
|
||||
motion, camera_direction, GPU_memory_mode, GPU_offload_steps,
|
||||
start_img, end_img=None
|
||||
):
|
||||
device = mm.get_torch_device()
|
||||
offload_device = mm.unet_offload_device()
|
||||
|
||||
mm.soft_empty_cache()
|
||||
gc.collect()
|
||||
|
||||
start_img = [self.to_pil(_start_img) for _start_img in start_img] if start_img is not None else None
|
||||
end_img = [self.to_pil(_end_img) for _end_img in end_img] if end_img is not None else None
|
||||
|
||||
# Count most suitable height and width
|
||||
aspect_ratio_sample_size = {key : [x / 512 * base_resolution for x in ASPECT_RATIO_512[key]] for key in ASPECT_RATIO_512.keys()}
|
||||
original_width, original_height = start_img[0].size if type(start_img) is list else Image.open(start_img).size
|
||||
closest_size, closest_ratio = get_closest_ratio(original_height, original_width, ratios=aspect_ratio_sample_size)
|
||||
height, width = [int(x / 16) * 16 for x in closest_size]
|
||||
aspect_ratio = "16:9" if width > height else "9:16"
|
||||
|
||||
# Get Pipeline
|
||||
pipeline = ruyi_model['pipeline']
|
||||
model_path = ruyi_model['model_path']
|
||||
|
||||
# Set GPU memory mode
|
||||
if GPU_memory_mode == "low_memory_mode" and pipeline.model_cpu_offload_flag:
|
||||
# Switch to low_memory_mode
|
||||
pipeline.enable_sequential_cpu_offload()
|
||||
elif GPU_memory_mode == "normal_mode" and not pipeline.model_cpu_offload_flag:
|
||||
# Switch to normal_mode
|
||||
pipeline.enable_model_cpu_offload()
|
||||
|
||||
# Set GPU offload steps
|
||||
pipeline.transformer.hidden_cache_size = int(GPU_offload_steps)
|
||||
|
||||
# Load Sampler
|
||||
pipeline.scheduler = self.get_scheduler(model_path, scheduler)
|
||||
|
||||
# Set random seed
|
||||
generator= torch.Generator(device).manual_seed(seed)
|
||||
|
||||
# Load control embeddings
|
||||
embeddings = self.get_control_embeddings(pipeline, aspect_ratio, motion, camera_direction)
|
||||
|
||||
# Inference
|
||||
with torch.no_grad():
|
||||
video_length = int(video_length // pipeline.vae.mini_batch_encoder * pipeline.vae.mini_batch_encoder) if video_length != 1 else 1
|
||||
input_video, input_video_mask, clip_image = get_image_to_video_latent(start_img, end_img, video_length=video_length, sample_size=(height, width))
|
||||
|
||||
for _lora_path, _lora_weight in zip(ruyi_model.get("loras", []), ruyi_model.get("strength_model", [])):
|
||||
pipeline = merge_lora(pipeline, _lora_path, _lora_weight)
|
||||
|
||||
sample = pipeline(
|
||||
prompt_embeds = embeddings["positive_embeds"],
|
||||
prompt_attention_mask = embeddings["positive_attention_mask"],
|
||||
prompt_embeds_2 = embeddings["positive_embeds_2"],
|
||||
prompt_attention_mask_2 = embeddings["positive_attention_mask_2"],
|
||||
|
||||
negative_prompt_embeds = embeddings["negative_embeds"],
|
||||
negative_prompt_attention_mask = embeddings["negative_attention_mask"],
|
||||
negative_prompt_embeds_2 = embeddings["negative_embeds_2"],
|
||||
negative_prompt_attention_mask_2 = embeddings["negative_attention_mask_2"],
|
||||
|
||||
video_length = video_length,
|
||||
height = height,
|
||||
width = width,
|
||||
generator = generator,
|
||||
guidance_scale = cfg,
|
||||
num_inference_steps = steps,
|
||||
|
||||
video = input_video,
|
||||
mask_video = input_video_mask,
|
||||
clip_image = clip_image,
|
||||
comfyui_progressbar = True,
|
||||
).videos
|
||||
videos = rearrange(sample, "b c t h w -> (b t) h w c")
|
||||
|
||||
for _lora_path, _lora_weight in zip(ruyi_model.get("loras", []), ruyi_model.get("strength_model", [])):
|
||||
pipeline = unmerge_lora(pipeline, _lora_path, _lora_weight)
|
||||
|
||||
return (videos,)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"Ruyi_LoadModel": Ruyi_LoadModel,
|
||||
"Ruyi_LoadLora": Ruyi_LoadLora,
|
||||
"Ruyi_I2VSampler": Ruyi_I2VSampler,
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"Ruyi_LoadModel": "Load Model",
|
||||
"Ruyi_LoadLora": "Load LoRA",
|
||||
"Ruyi_I2VSampler": "Sampler for Image to Video",
|
||||
}
|
||||
@@ -0,0 +1,293 @@
|
||||
{
|
||||
"last_node_id": 5,
|
||||
"last_link_id": 4,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "Ruyi_LoadModel",
|
||||
"pos": {
|
||||
"0": 295,
|
||||
"1": 238
|
||||
},
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 82
|
||||
},
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "ruyi_model",
|
||||
"type": "RUYI_MODEL",
|
||||
"links": [
|
||||
2
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "Ruyi_LoadModel"
|
||||
},
|
||||
"widgets_values": [
|
||||
"Ruyi-Mini-7B",
|
||||
"yes"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "Ruyi_I2VSampler",
|
||||
"pos": {
|
||||
"0": 753,
|
||||
"1": 326
|
||||
},
|
||||
"size": {
|
||||
"0": 327.5999755859375,
|
||||
"1": 338
|
||||
},
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "ruyi_model",
|
||||
"type": "RUYI_MODEL",
|
||||
"link": 2
|
||||
},
|
||||
{
|
||||
"name": "start_img",
|
||||
"type": "IMAGE",
|
||||
"link": 3
|
||||
},
|
||||
{
|
||||
"name": "end_img",
|
||||
"type": "IMAGE",
|
||||
"link": 4,
|
||||
"shape": 7
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
1
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "Ruyi_I2VSampler"
|
||||
},
|
||||
"widgets_values": [
|
||||
120,
|
||||
512,
|
||||
112876945129710,
|
||||
"randomize",
|
||||
25,
|
||||
7,
|
||||
"DDIM",
|
||||
"auto",
|
||||
"auto",
|
||||
"normal_mode",
|
||||
"0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "VHS_VideoCombine",
|
||||
"pos": {
|
||||
"0": 1183,
|
||||
"1": 89
|
||||
},
|
||||
"size": [
|
||||
317,
|
||||
758.0769230769231
|
||||
],
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"link": 1
|
||||
},
|
||||
{
|
||||
"name": "audio",
|
||||
"type": "AUDIO",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"name": "meta_batch",
|
||||
"type": "VHS_BatchManager",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"name": "vae",
|
||||
"type": "VAE",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "Filenames",
|
||||
"type": "VHS_FILENAMES",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VHS_VideoCombine"
|
||||
},
|
||||
"widgets_values": {
|
||||
"frame_rate": 24,
|
||||
"loop_count": 0,
|
||||
"filename_prefix": "Ruyi-I2V-StartEndFrames",
|
||||
"format": "video/h264-mp4",
|
||||
"pix_fmt": "yuv420p",
|
||||
"crf": 19,
|
||||
"save_metadata": true,
|
||||
"pingpong": false,
|
||||
"save_output": true,
|
||||
"videopreview": {
|
||||
"hidden": false,
|
||||
"paused": false,
|
||||
"params": {
|
||||
"filename": "Ruyi-I2V-StartEndFrames_00001.mp4",
|
||||
"subfolder": "",
|
||||
"type": "output",
|
||||
"format": "video/h264-mp4",
|
||||
"frame_rate": 24
|
||||
},
|
||||
"muted": false
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "LoadImage",
|
||||
"pos": {
|
||||
"0": 180,
|
||||
"1": 398
|
||||
},
|
||||
"size": {
|
||||
"0": 240.88999938964844,
|
||||
"1": 419.6499938964844
|
||||
},
|
||||
"flags": {},
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
3
|
||||
],
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "MASK",
|
||||
"type": "MASK",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LoadImage"
|
||||
},
|
||||
"widgets_values": [
|
||||
"girl_01.jpg",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "LoadImage",
|
||||
"pos": {
|
||||
"0": 453,
|
||||
"1": 398
|
||||
},
|
||||
"size": [
|
||||
242.0999999999999,
|
||||
421.6500000000001
|
||||
],
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
4
|
||||
],
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "MASK",
|
||||
"type": "MASK",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LoadImage"
|
||||
},
|
||||
"widgets_values": [
|
||||
"girl_02.jpg",
|
||||
"image"
|
||||
]
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[
|
||||
1,
|
||||
2,
|
||||
0,
|
||||
5,
|
||||
0,
|
||||
"IMAGE"
|
||||
],
|
||||
[
|
||||
2,
|
||||
1,
|
||||
0,
|
||||
2,
|
||||
0,
|
||||
"RUYI_MODEL"
|
||||
],
|
||||
[
|
||||
3,
|
||||
3,
|
||||
0,
|
||||
2,
|
||||
1,
|
||||
"IMAGE"
|
||||
],
|
||||
[
|
||||
4,
|
||||
4,
|
||||
0,
|
||||
2,
|
||||
2,
|
||||
"IMAGE"
|
||||
]
|
||||
],
|
||||
"groups": [],
|
||||
"config": {},
|
||||
"extra": {
|
||||
"ds": {
|
||||
"scale": 1,
|
||||
"offset": [
|
||||
-43.00999999999996,
|
||||
33.75000000000006
|
||||
]
|
||||
}
|
||||
},
|
||||
"version": 0.4
|
||||
}
|
||||
@@ -0,0 +1,293 @@
|
||||
{
|
||||
"last_node_id": 5,
|
||||
"last_link_id": 4,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "Ruyi_LoadModel",
|
||||
"pos": {
|
||||
"0": 295,
|
||||
"1": 238
|
||||
},
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 82
|
||||
},
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "ruyi_model",
|
||||
"type": "RUYI_MODEL",
|
||||
"links": [
|
||||
2
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "Ruyi_LoadModel"
|
||||
},
|
||||
"widgets_values": [
|
||||
"Ruyi-Mini-7B",
|
||||
"yes"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "VHS_VideoCombine",
|
||||
"pos": {
|
||||
"0": 1183,
|
||||
"1": 89
|
||||
},
|
||||
"size": [
|
||||
317,
|
||||
758.0769230769231
|
||||
],
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"link": 1
|
||||
},
|
||||
{
|
||||
"name": "audio",
|
||||
"type": "AUDIO",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"name": "meta_batch",
|
||||
"type": "VHS_BatchManager",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"name": "vae",
|
||||
"type": "VAE",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "Filenames",
|
||||
"type": "VHS_FILENAMES",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VHS_VideoCombine"
|
||||
},
|
||||
"widgets_values": {
|
||||
"frame_rate": 24,
|
||||
"loop_count": 0,
|
||||
"filename_prefix": "Ruyi-I2V-StartEndFrames",
|
||||
"format": "video/h264-mp4",
|
||||
"pix_fmt": "yuv420p",
|
||||
"crf": 19,
|
||||
"save_metadata": true,
|
||||
"pingpong": false,
|
||||
"save_output": true,
|
||||
"videopreview": {
|
||||
"hidden": false,
|
||||
"paused": false,
|
||||
"params": {
|
||||
"filename": "Ruyi-I2V-StartEndFrames_00001.mp4",
|
||||
"subfolder": "",
|
||||
"type": "output",
|
||||
"format": "video/h264-mp4",
|
||||
"frame_rate": 24
|
||||
},
|
||||
"muted": false
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "LoadImage",
|
||||
"pos": {
|
||||
"0": 180,
|
||||
"1": 398
|
||||
},
|
||||
"size": {
|
||||
"0": 240.88999938964844,
|
||||
"1": 419.6499938964844
|
||||
},
|
||||
"flags": {},
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
3
|
||||
],
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "MASK",
|
||||
"type": "MASK",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LoadImage"
|
||||
},
|
||||
"widgets_values": [
|
||||
"gir_01.jpg",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "LoadImage",
|
||||
"pos": {
|
||||
"0": 453,
|
||||
"1": 398
|
||||
},
|
||||
"size": [
|
||||
242.0999999999999,
|
||||
421.6500000000001
|
||||
],
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
4
|
||||
],
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "MASK",
|
||||
"type": "MASK",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LoadImage"
|
||||
},
|
||||
"widgets_values": [
|
||||
"girl_02.jpg",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "Ruyi_I2VSampler",
|
||||
"pos": {
|
||||
"0": 753,
|
||||
"1": 326
|
||||
},
|
||||
"size": {
|
||||
"0": 327.5999755859375,
|
||||
"1": 338
|
||||
},
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "ruyi_model",
|
||||
"type": "RUYI_MODEL",
|
||||
"link": 2
|
||||
},
|
||||
{
|
||||
"name": "start_img",
|
||||
"type": "IMAGE",
|
||||
"link": 3
|
||||
},
|
||||
{
|
||||
"name": "end_img",
|
||||
"type": "IMAGE",
|
||||
"link": 4,
|
||||
"shape": 7
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
1
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "Ruyi_I2VSampler"
|
||||
},
|
||||
"widgets_values": [
|
||||
120,
|
||||
512,
|
||||
112876945129710,
|
||||
"randomize",
|
||||
25,
|
||||
7,
|
||||
"DDIM",
|
||||
"auto",
|
||||
"auto",
|
||||
"normal_mode",
|
||||
"5"
|
||||
]
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[
|
||||
1,
|
||||
2,
|
||||
0,
|
||||
5,
|
||||
0,
|
||||
"IMAGE"
|
||||
],
|
||||
[
|
||||
2,
|
||||
1,
|
||||
0,
|
||||
2,
|
||||
0,
|
||||
"RUYI_MODEL"
|
||||
],
|
||||
[
|
||||
3,
|
||||
3,
|
||||
0,
|
||||
2,
|
||||
1,
|
||||
"IMAGE"
|
||||
],
|
||||
[
|
||||
4,
|
||||
4,
|
||||
0,
|
||||
2,
|
||||
2,
|
||||
"IMAGE"
|
||||
]
|
||||
],
|
||||
"groups": [],
|
||||
"config": {},
|
||||
"extra": {
|
||||
"ds": {
|
||||
"scale": 1,
|
||||
"offset": [
|
||||
-43.00999999999996,
|
||||
33.75000000000006
|
||||
]
|
||||
}
|
||||
},
|
||||
"version": 0.4
|
||||
}
|
||||
@@ -0,0 +1,247 @@
|
||||
{
|
||||
"last_node_id": 6,
|
||||
"last_link_id": 3,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 2,
|
||||
"type": "Ruyi_I2VSampler",
|
||||
"pos": {
|
||||
"0": 628,
|
||||
"1": 284
|
||||
},
|
||||
"size": {
|
||||
"0": 327.5999755859375,
|
||||
"1": 338
|
||||
},
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "ruyi_model",
|
||||
"type": "RUYI_MODEL",
|
||||
"link": 1
|
||||
},
|
||||
{
|
||||
"name": "start_img",
|
||||
"type": "IMAGE",
|
||||
"link": 2
|
||||
},
|
||||
{
|
||||
"name": "end_img",
|
||||
"type": "IMAGE",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
3
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "Ruyi_I2VSampler"
|
||||
},
|
||||
"widgets_values": [
|
||||
120,
|
||||
512,
|
||||
925247271358454,
|
||||
"randomize",
|
||||
25,
|
||||
7,
|
||||
"DDIM",
|
||||
"2",
|
||||
"static",
|
||||
"normal_mode",
|
||||
"0"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 1,
|
||||
"type": "Ruyi_LoadModel",
|
||||
"pos": {
|
||||
"0": 210,
|
||||
"1": 162
|
||||
},
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 82
|
||||
},
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "ruyi_model",
|
||||
"type": "RUYI_MODEL",
|
||||
"links": [
|
||||
1
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "Ruyi_LoadModel"
|
||||
},
|
||||
"widgets_values": [
|
||||
"Ruyi-Mini-7B",
|
||||
"yes"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "VHS_VideoCombine",
|
||||
"pos": {
|
||||
"0": 1045,
|
||||
"1": 133
|
||||
},
|
||||
"size": [
|
||||
404.73553466796875,
|
||||
601.8645528157551
|
||||
],
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"link": 3
|
||||
},
|
||||
{
|
||||
"name": "audio",
|
||||
"type": "AUDIO",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"name": "meta_batch",
|
||||
"type": "VHS_BatchManager",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"name": "vae",
|
||||
"type": "VAE",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "Filenames",
|
||||
"type": "VHS_FILENAMES",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VHS_VideoCombine"
|
||||
},
|
||||
"widgets_values": {
|
||||
"frame_rate": 24,
|
||||
"loop_count": 0,
|
||||
"filename_prefix": "Ruyi-I2V-StartFrame",
|
||||
"format": "video/h264-mp4",
|
||||
"pix_fmt": "yuv420p",
|
||||
"crf": 19,
|
||||
"save_metadata": true,
|
||||
"pingpong": false,
|
||||
"save_output": true,
|
||||
"videopreview": {
|
||||
"hidden": false,
|
||||
"paused": false,
|
||||
"params": {
|
||||
"filename": "Ruyi-I2V-StartFrame_00001.mp4",
|
||||
"subfolder": "",
|
||||
"type": "output",
|
||||
"format": "video/h264-mp4",
|
||||
"frame_rate": 24
|
||||
},
|
||||
"muted": false
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "LoadImage",
|
||||
"pos": {
|
||||
"0": 200,
|
||||
"1": 439
|
||||
},
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 314
|
||||
},
|
||||
"flags": {},
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
2
|
||||
],
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "MASK",
|
||||
"type": "MASK",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LoadImage"
|
||||
},
|
||||
"widgets_values": [
|
||||
"example_03.jpg",
|
||||
"image"
|
||||
]
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[
|
||||
1,
|
||||
1,
|
||||
0,
|
||||
2,
|
||||
0,
|
||||
"RUYI_MODEL"
|
||||
],
|
||||
[
|
||||
2,
|
||||
3,
|
||||
0,
|
||||
2,
|
||||
1,
|
||||
"IMAGE"
|
||||
],
|
||||
[
|
||||
3,
|
||||
2,
|
||||
0,
|
||||
4,
|
||||
0,
|
||||
"IMAGE"
|
||||
]
|
||||
],
|
||||
"groups": [],
|
||||
"config": {},
|
||||
"extra": {
|
||||
"ds": {
|
||||
"scale": 1,
|
||||
"offset": [
|
||||
0,
|
||||
0
|
||||
]
|
||||
}
|
||||
},
|
||||
"version": 0.4
|
||||
}
|
||||
@@ -0,0 +1,247 @@
|
||||
{
|
||||
"last_node_id": 6,
|
||||
"last_link_id": 3,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "Ruyi_LoadModel",
|
||||
"pos": {
|
||||
"0": 210,
|
||||
"1": 162
|
||||
},
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 82
|
||||
},
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "ruyi_model",
|
||||
"type": "RUYI_MODEL",
|
||||
"links": [
|
||||
1
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "Ruyi_LoadModel"
|
||||
},
|
||||
"widgets_values": [
|
||||
"Ruyi-Mini-7B",
|
||||
"yes"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "VHS_VideoCombine",
|
||||
"pos": {
|
||||
"0": 1045,
|
||||
"1": 133
|
||||
},
|
||||
"size": [
|
||||
404.73553466796875,
|
||||
601.8645528157551
|
||||
],
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"link": 3
|
||||
},
|
||||
{
|
||||
"name": "audio",
|
||||
"type": "AUDIO",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"name": "meta_batch",
|
||||
"type": "VHS_BatchManager",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
},
|
||||
{
|
||||
"name": "vae",
|
||||
"type": "VAE",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "Filenames",
|
||||
"type": "VHS_FILENAMES",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "VHS_VideoCombine"
|
||||
},
|
||||
"widgets_values": {
|
||||
"frame_rate": 24,
|
||||
"loop_count": 0,
|
||||
"filename_prefix": "Ruyi-I2V-StartFrame",
|
||||
"format": "video/h264-mp4",
|
||||
"pix_fmt": "yuv420p",
|
||||
"crf": 19,
|
||||
"save_metadata": true,
|
||||
"pingpong": false,
|
||||
"save_output": true,
|
||||
"videopreview": {
|
||||
"hidden": false,
|
||||
"paused": false,
|
||||
"params": {
|
||||
"filename": "Ruyi-I2V-StartFrame_00001.mp4",
|
||||
"subfolder": "",
|
||||
"type": "output",
|
||||
"format": "video/h264-mp4",
|
||||
"frame_rate": 24
|
||||
},
|
||||
"muted": false
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "LoadImage",
|
||||
"pos": {
|
||||
"0": 200,
|
||||
"1": 439
|
||||
},
|
||||
"size": {
|
||||
"0": 315,
|
||||
"1": 314
|
||||
},
|
||||
"flags": {},
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "IMAGE",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
2
|
||||
],
|
||||
"slot_index": 0
|
||||
},
|
||||
{
|
||||
"name": "MASK",
|
||||
"type": "MASK",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "LoadImage"
|
||||
},
|
||||
"widgets_values": [
|
||||
"example_03.jpg",
|
||||
"image"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "Ruyi_I2VSampler",
|
||||
"pos": {
|
||||
"0": 628,
|
||||
"1": 284
|
||||
},
|
||||
"size": {
|
||||
"0": 327.5999755859375,
|
||||
"1": 338
|
||||
},
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "ruyi_model",
|
||||
"type": "RUYI_MODEL",
|
||||
"link": 1
|
||||
},
|
||||
{
|
||||
"name": "start_img",
|
||||
"type": "IMAGE",
|
||||
"link": 2
|
||||
},
|
||||
{
|
||||
"name": "end_img",
|
||||
"type": "IMAGE",
|
||||
"link": null,
|
||||
"shape": 7
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "images",
|
||||
"type": "IMAGE",
|
||||
"links": [
|
||||
3
|
||||
],
|
||||
"slot_index": 0
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"Node name for S&R": "Ruyi_I2VSampler"
|
||||
},
|
||||
"widgets_values": [
|
||||
120,
|
||||
512,
|
||||
925247271358454,
|
||||
"randomize",
|
||||
25,
|
||||
7,
|
||||
"DDIM",
|
||||
"2",
|
||||
"static",
|
||||
"normal_mode",
|
||||
"5"
|
||||
]
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[
|
||||
1,
|
||||
1,
|
||||
0,
|
||||
2,
|
||||
0,
|
||||
"RUYI_MODEL"
|
||||
],
|
||||
[
|
||||
2,
|
||||
3,
|
||||
0,
|
||||
2,
|
||||
1,
|
||||
"IMAGE"
|
||||
],
|
||||
[
|
||||
3,
|
||||
2,
|
||||
0,
|
||||
4,
|
||||
0,
|
||||
"IMAGE"
|
||||
]
|
||||
],
|
||||
"groups": [],
|
||||
"config": {},
|
||||
"extra": {
|
||||
"ds": {
|
||||
"scale": 1,
|
||||
"offset": [
|
||||
0,
|
||||
0
|
||||
]
|
||||
}
|
||||
},
|
||||
"version": 0.4
|
||||
}
|
||||
Reference in New Issue
Block a user