5.1 KiB
5.1 KiB
Documentation
- Class name: Modelscopev2v
- Category: cspnodes/modelscope
- Output node: False
- Repo Ref: https://github.com/modelscope/modelscope
Modelscopev2v节点旨在实现视觉内容的领域间转换，利用先进模型对视觉数据进行高级理解和操作。它专注于实现复杂的视觉转换和增强，适用于需要深度理解和修改视觉信息的任务。
Input types
Required
- video_frames
- 指定需要转换的视频帧。这是定义将要进行转换的视觉内容的关键输入。
- Comfy dtype: IMAGE
- Python dtype: List[torch.Tensor]
- prompt
- 对所需转换或结果的文字描述，用于指导模型理解和操作视觉内容。
- Comfy dtype: STRING
- Python dtype: str
- negative_prompt
- 对不期望出现的元素或结果的文字描述，有助于引导模型避开某些转换方向。
- Comfy dtype: STRING
- Python dtype: str
- model_path
- 用于转换的模型路径，决定了转换过程的具体能力和特性。
- Comfy dtype: STRING
- Python dtype: str
- strength
- 控制转换的强度，允许微调视觉输出与提示的符合程度。
- Comfy dtype: FLOAT
- Python dtype: float
- num_inference_steps
- 确定模型执行转换的步骤数，影响输出的质量和细节。
- Comfy dtype: INT
- Python dtype: int
- guidance_scale
- 调整提示对转换的影响程度，平衡提示和原始内容之间的忠实度。
- Comfy dtype: FLOAT
- Python dtype: float
- seed
- 随机数生成的种子值，确保转换过程的可重复性。
- Comfy dtype: INT
- Python dtype: int
- enable_forward_chunking
- 启用或禁用前向分块，可能会提高处理大型视频帧的性能。
- Comfy dtype: BOOLEAN
- Python dtype: bool
- enable_vae_slicing
- 启用或禁用VAE切片，可能会影响模型的效率和输出质量。
- Comfy dtype: BOOLEAN
- Python dtype: bool
Output types
- image
- 转换后的视频帧，代表经过指定转换后的视觉内容。
- Comfy dtype: IMAGE
- Python dtype: List[torch.Tensor]
Usage tips
- Infra type:
GPU - Common nodes: unknown
Source code
class Modelscopev2v:
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"video_frames": ("IMAGE", {}),
"prompt": ("STRING", {}),
"negative_prompt": ("STRING", {"default": None}),
"model_path": ("STRING", {"default": "cerspense/zeroscope_v2_XL"}),
"strength": ("FLOAT", {"default": 0.70}),
"num_inference_steps": ("INT", {"default": 25}),
"guidance_scale": ("FLOAT", {"default": 8.50}),
"seed": ("INT", {"default": 42}),
"enable_forward_chunking": ("BOOLEAN", {"default": False}),
"enable_vae_slicing": ("BOOLEAN", {"default": True}),
}
}
RETURN_TYPES = ("IMAGE",)
FUNCTION = "transform_video_frames"
CATEGORY = "cspnodes/modelscope"
def transform_video_frames(self, video_frames, prompt, model_path, strength, num_inference_steps, guidance_scale, negative_prompt, seed, enable_forward_chunking, enable_vae_slicing):
# Set up the generator for deterministic results if seed is provided
generator = torch.Generator()
if seed is not None:
generator.manual_seed(seed)
# Initialize the diffusion pipeline with the specified model path
pipe = DiffusionPipeline.from_pretrained(model_path, torch_dtype=torch.float16)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()
# Apply memory optimizations based on the toggles
if enable_forward_chunking:
pipe.unet.enable_forward_chunking(chunk_size=1, dim=1)
if enable_vae_slicing:
pipe.enable_vae_slicing()
# Convert tensor to list of PIL Images
# Assuming video_frames is a float tensor with values in [0, 1]
video_frames_uint8 = (video_frames * 255).byte()
video = [Image.fromarray(frame.numpy(), 'RGB') for frame in video_frames_uint8]
# Generate new video frames
video_frames = pipe(prompt, video=video, strength=strength, num_inference_steps=num_inference_steps, guidance_scale=guidance_scale, negative_prompt=negative_prompt, generator=generator).frames
# Ensure video_frames is a PyTorch tensor
if not isinstance(video_frames, torch.Tensor):
video_frames = torch.tensor(video_frames, dtype=torch.float32)
# Normalize the tensor to have values between 0 and 1 if they are in the range 0-255
if video_frames.max() > 1.0:
video_frames = video_frames / 255.0
# The expected shape is (num_frames, height, width, channels)
video_frames = video_frames.squeeze(0).permute(0, 1, 2, 3)
# Convert the tensor to CPU and to uint8 if it's not already
video_frames = video_frames.to('cpu')
# return (video_frames_numpy,)
return (video_frames,)