132 lines
5.1 KiB
Markdown
132 lines
5.1 KiB
Markdown
|
|
# Documentation
|
|
- Class name: Modelscopev2v
|
|
- Category: cspnodes/modelscope
|
|
- Output node: False
|
|
- Repo Ref: https://github.com/modelscope/modelscope
|
|
|
|
Modelscopev2v节点旨在实现视觉内容的领域间转换，利用先进模型对视觉数据进行高级理解和操作。它专注于实现复杂的视觉转换和增强，适用于需要深度理解和修改视觉信息的任务。
|
|
|
|
# Input types
|
|
## Required
|
|
- video_frames
|
|
- 指定需要转换的视频帧。这是定义将要进行转换的视觉内容的关键输入。
|
|
- Comfy dtype: IMAGE
|
|
- Python dtype: List[torch.Tensor]
|
|
- prompt
|
|
- 对所需转换或结果的文字描述，用于指导模型理解和操作视觉内容。
|
|
- Comfy dtype: STRING
|
|
- Python dtype: str
|
|
- negative_prompt
|
|
- 对不期望出现的元素或结果的文字描述，有助于引导模型避开某些转换方向。
|
|
- Comfy dtype: STRING
|
|
- Python dtype: str
|
|
- model_path
|
|
- 用于转换的模型路径，决定了转换过程的具体能力和特性。
|
|
- Comfy dtype: STRING
|
|
- Python dtype: str
|
|
- strength
|
|
- 控制转换的强度，允许微调视觉输出与提示的符合程度。
|
|
- Comfy dtype: FLOAT
|
|
- Python dtype: float
|
|
- num_inference_steps
|
|
- 确定模型执行转换的步骤数，影响输出的质量和细节。
|
|
- Comfy dtype: INT
|
|
- Python dtype: int
|
|
- guidance_scale
|
|
- 调整提示对转换的影响程度，平衡提示和原始内容之间的忠实度。
|
|
- Comfy dtype: FLOAT
|
|
- Python dtype: float
|
|
- seed
|
|
- 随机数生成的种子值，确保转换过程的可重复性。
|
|
- Comfy dtype: INT
|
|
- Python dtype: int
|
|
- enable_forward_chunking
|
|
- 启用或禁用前向分块，可能会提高处理大型视频帧的性能。
|
|
- Comfy dtype: BOOLEAN
|
|
- Python dtype: bool
|
|
- enable_vae_slicing
|
|
- 启用或禁用VAE切片，可能会影响模型的效率和输出质量。
|
|
- Comfy dtype: BOOLEAN
|
|
- Python dtype: bool
|
|
|
|
# Output types
|
|
- image
|
|
- 转换后的视频帧，代表经过指定转换后的视觉内容。
|
|
- Comfy dtype: IMAGE
|
|
- Python dtype: List[torch.Tensor]
|
|
|
|
|
|
## Usage tips
|
|
- Infra type: `GPU`
|
|
- Common nodes: unknown
|
|
|
|
|
|
## Source code
|
|
```python
|
|
class Modelscopev2v:
|
|
@classmethod
|
|
def INPUT_TYPES(cls):
|
|
return {
|
|
"required": {
|
|
"video_frames": ("IMAGE", {}),
|
|
"prompt": ("STRING", {}),
|
|
"negative_prompt": ("STRING", {"default": None}),
|
|
"model_path": ("STRING", {"default": "cerspense/zeroscope_v2_XL"}),
|
|
"strength": ("FLOAT", {"default": 0.70}),
|
|
"num_inference_steps": ("INT", {"default": 25}),
|
|
"guidance_scale": ("FLOAT", {"default": 8.50}),
|
|
"seed": ("INT", {"default": 42}),
|
|
"enable_forward_chunking": ("BOOLEAN", {"default": False}),
|
|
"enable_vae_slicing": ("BOOLEAN", {"default": True}),
|
|
}
|
|
}
|
|
|
|
RETURN_TYPES = ("IMAGE",)
|
|
FUNCTION = "transform_video_frames"
|
|
CATEGORY = "cspnodes/modelscope"
|
|
|
|
def transform_video_frames(self, video_frames, prompt, model_path, strength, num_inference_steps, guidance_scale, negative_prompt, seed, enable_forward_chunking, enable_vae_slicing):
|
|
# Set up the generator for deterministic results if seed is provided
|
|
generator = torch.Generator()
|
|
if seed is not None:
|
|
generator.manual_seed(seed)
|
|
|
|
# Initialize the diffusion pipeline with the specified model path
|
|
pipe = DiffusionPipeline.from_pretrained(model_path, torch_dtype=torch.float16)
|
|
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
|
|
pipe.enable_model_cpu_offload()
|
|
|
|
# Apply memory optimizations based on the toggles
|
|
if enable_forward_chunking:
|
|
pipe.unet.enable_forward_chunking(chunk_size=1, dim=1)
|
|
if enable_vae_slicing:
|
|
pipe.enable_vae_slicing()
|
|
|
|
# Convert tensor to list of PIL Images
|
|
# Assuming video_frames is a float tensor with values in [0, 1]
|
|
video_frames_uint8 = (video_frames * 255).byte()
|
|
video = [Image.fromarray(frame.numpy(), 'RGB') for frame in video_frames_uint8]
|
|
|
|
# Generate new video frames
|
|
video_frames = pipe(prompt, video=video, strength=strength, num_inference_steps=num_inference_steps, guidance_scale=guidance_scale, negative_prompt=negative_prompt, generator=generator).frames
|
|
|
|
# Ensure video_frames is a PyTorch tensor
|
|
if not isinstance(video_frames, torch.Tensor):
|
|
video_frames = torch.tensor(video_frames, dtype=torch.float32)
|
|
|
|
# Normalize the tensor to have values between 0 and 1 if they are in the range 0-255
|
|
if video_frames.max() > 1.0:
|
|
video_frames = video_frames / 255.0
|
|
|
|
# The expected shape is (num_frames, height, width, channels)
|
|
video_frames = video_frames.squeeze(0).permute(0, 1, 2, 3)
|
|
|
|
# Convert the tensor to CPU and to uint8 if it's not already
|
|
video_frames = video_frames.to('cpu')
|
|
|
|
# return (video_frames_numpy,)
|
|
return (video_frames,)
|
|
|
|
```
|