diff --git a/Step-Video-TI2V/README.md b/Step-Video-TI2V/README.md index cc7edb5..05317df 100644 --- a/Step-Video-TI2V/README.md +++ b/Step-Video-TI2V/README.md @@ -1,5 +1,3 @@ -# Step-Video-Ti2V -

@@ -14,16 +12,186 @@ ## 🔥🔥🔥 News!! -* Feb 17, 2025: 👋 We release the inference code and model weights of Step-Video-Ti2V. [Download](https://huggingface.co/stepfun-ai/stepvideo-ti2v) -* Feb 17, 2025: 🎉 We have made our technical report available as open source. [Read](https://arxiv.org/abs/2502.10248) +* Mar 17, 2025: 👋 We release the inference code and model weights of Step-Video-Ti2V. [Download](https://huggingface.co/stepfun-ai/stepvideo-ti2v) +* Mar 17, 2025: 🎉 We have made our technical report available as open source. [Read](https://arxiv.org/abs/2502.10248) + + + +## Motion Control + + + + + + + +
+ +## Motion Amplitude Control + + + + + + + + + + + + +
Motion = 2Motion = 5Motion = 10
+ + + + + + + + + + + + +
Motion = 2Motion = 5Motion = 20
+ +🎯 Tips +The default motion_score = 5 is suitable for general use. If you need more stability, set motion_score = 2, though it may be less responsive to certain movements. For greater movement flexibility, you can use motion_score = 10 or motion_score = 20 to enable more intense actions. Feel free to customize the motion_score based on your creative needs to fit different use cases. + +## Camera Control + + + + + + + + + + + + +
镜头环绕镜头推进镜头拉远
+ + + + + + + + + + + + +
镜头固定镜头左移镜头右摇
+ +🎥 Camera Motion Tips +Fixed Camera +Pan (Up/Down/Left/Right) +Tilt (Up/Down/Left/Right) +Zoom In / Zoom Out +Dolly In / Dolly Out +Camera Rotation +Tracking Shot (Follow xxx) +Orbit Shot (Circle Around xxx) +Rack Focus (Focus Shift) +Camera Shake +High-Angle Dolly In / Dolly Out + +🎬 Advanced Movements +Dolly Zoom (Hitchcock Effect) +Grammy-Style Motion + + + + + + + + + + +
摄像机从下到中顺时针绕人物移动，镜头推远，1秒后慢镜头拍摄，人物从斜侧背对镜头，转过身正面面对镜头摄像机从下到中顺时针绕人物90度移动，镜头拉近，1秒后慢镜头拍摄，人物从背对镜头，身体逆时针旋转 180 度最终正对镜头
+ + +🔧 Motion Score Considerations +Certain camera movements, especially tracking shots, orbit shots, and complex motion effects, work significantly better with motion_score = 5 or 10 rather than motion_score = 2. Choosing the right setting can greatly enhance motion smoothness and accuracy. +## Visual Effects + + + + + + + + + + + + +
机器人变身特效机器人变身特效机器人变身特效
+ + + + + + + + + + + + +
毒液变身特效机器人变身特效毒液变身特效
+ + + + +## Table of Contents + +1. [Introduction](#1-introduction) +2. [Model Summary](#2-model-summary) +3. [Model Download](#3-model-download) +4. [Model Usage](#4-model-usage) +5. [Benchmark](#5-benchmark) +6. [Online Engine](#6-online-engine) +7. [Citation](#7-citation) +8. [Acknowledgement](#8-ackownledgement) + + + +## 1. Introduction +We present Step-Video-TI2V, a state-of-the-art text-driven image-to-video generation model with 30B parameters, capable of generating videos up to 102 frames +based on both text and image inputs. We build Step-Video-TI2V-Eval as a new benchmark for the text-driven image-to-video task and compare Step-Video-TI2V +with open-source and commercial TI2V engines using this dataset. Experimental results demonstrate the state-of-the-art performance of Step-Video-TI2V in the +image-to-video generation task. Both Step-Video-TI2V and Step-Video-TI2V-Eval are available. + +## 2. Model Summary +Step-Video-TI2V based on Step-Video-T2V. To incorporate the image condition as the first frame of the generated video, we encode it into latent representations using Step-Video- T2V’s Video-VAE and concatenate them along the channel dimension of the video latent. Additionally, we introduce a motion score condition, enabling users to control the dynamic level of the video generated from the image condition. Figure 1 shows an overview of our framework, highlighting these two modifications to the pre-trained T2V model. + +

+ +

+ +## 3. Model Download +| Models | 🤗Huggingface | 🤖Modelscope | +|:-------:|:-------:|:-------:| +| Step-Video-T2V | [download](https://huggingface.co/stepfun-ai/stepvideo-ti2v) | + + +## 4. Model Usage + +### 📜 4.1 Dependencies and Installation + ```bash git clone https://github.com/stepfun-ai/Step-Video-TI2V.git conda create -n stepvideo python=3.10 conda activate stepvideo -cd StepFun-StepVideo +cd Step-Video-TI2V pip install -e . ``` @@ -35,5 +203,59 @@ parallel=4 # or parallel=8 url='127.0.0.1' model_dir=where_you_download_dir -torchrun --nproc_per_node $parallel run_parallel --model_dir $model_dir --vae_url $url --caption_url $url --ulysses_degree $parallel --prompt "笑起来" --infer_steps 15 --cfg_scale 5.0 --time_shift 17.0 +torchrun --nproc_per_node $parallel run_parallel.py --model_dir $model_dir --vae_url $url --caption_url $url --ulysses_degree $parallel --prompt "笑起来" --first_image_path ./assets/demo.png --infer_steps 50 --cfg_scale 9.0 --time_shift 13.0 --motion_score 5.0 ``` + +We list some more useful configurations for easy usage: + +| Argument | Default | Description | +|:----------------------:|:---------:|:-----------------------------------------:| +| `--model_dir` | None | The model checkpoint for video generation | +| `--prompt` | “笑起来” | The text prompt for I2V generation | +| `--first_image_path` | ./assets/demo.png | The reference image path for I2V task. | +| `--infer_steps` | 50 | The number of steps for sampling | +| `--cfg_scale` | 9.0 | Embedded Classifier free guidance scale | +| `--time_shift` | 7.0 | Shift factor for flow matching schedulers. | +| `--motion_score` | 5.0 | Score to control the motion level of the video. | +| `--seed` | None | The random seed for generating video, if None, we init a random seed | +| `--use-cpu-offload` | False | Use CPU offload for the model load to save more memory, necessary for high-res video generation | +| `--save-path` | ./results | Path to save the generated video | + + +## 5. Benchmark +We first evaluated on VBench-I2V is a comprehensive benchmark suite that deconstructs “video generation +quality” into specific, hierarchical, and disentangled dimensions, each with tailored prompts and evaluation methods. We utilize the VBench-I2V benchmark to assess the performance of Step-Video- +TI2V alongside other TI2V models. + +

+ +

+ +We are releasing [Step-Video-Ti2V Eval](https://github.com/stepfun-ai/Step-Video-T2V/blob/main/benchmark/Step-Video-TI2V-Eval) , a new benchmark designed for the text-driven image-to-video +generation task. The dataset comprises 178 real-world and 120 anime-style prompt-image pairs, ensuring broad coverage of diverse user scenarios. To achieve comprehensive representation, we +developed a fine-grained schema for data collection in both categories. We compare Step-Video-TI2V with two recently released open-source TI2V models, OSTopA and +OSTopB, as well as two close-source commercial TI2V models, CSTopC and CSTopD. All these four models originate from China. Generally, Step-Video-TI2V achieves state-of-the-art performance on the total score across the +three evaluation dimensions in Step-Video-TI2V-Eval, either outperforming or matching leading open-source and closed-source commercial TI2V models. + +

+ +

+ + + +## 6. Online Engine +The online version of Step-Video-T2V is available on [跃问视频](https://yuewen.cn/videos), where you can also explore some impressive examples. + +## 7. Citation +``` +@misc{ma2025stepvideot2vtechnicalreportpractice, + title={Step-Video-Ti2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model}, + author={Guoqing Ma and Haoyang Huang and Kun Yan and Liangyu Chen and Nan Duan and Shengming Yin and Changyi Wan and Ranchen Ming and Xiaoniu Song and Xing Chen and Yu Zhou and Deshan Sun and Deyu Zhou and Jian Zhou and Kaijun Tan and Kang An and Mei Chen and Wei Ji and Qiling Wu and Wen Sun and Xin Han and Yanan Wei and Zheng Ge and Aojie Li and Bin Wang and Bizhu Huang and Bo Wang and Brian Li and Changxing Miao and Chen Xu and Chenfei Wu and Chenguang Yu and Dapeng Shi and Dingyuan Hu and Enle Liu and Gang Yu and Ge Yang and Guanzhe Huang and Gulin Yan and Haiyang Feng and Hao Nie and Haonan Jia and Hanpeng Hu and Hanqi Chen and Haolong Yan and Heng Wang and Hongcheng Guo and Huilin Xiong and Huixin Xiong and Jiahao Gong and Jianchang Wu and Jiaoren Wu and Jie Wu and Jie Yang and Jiashuai Liu and Jiashuo Li and Jingyang Zhang and Junjing Guo and Junzhe Lin and Kaixiang Li and Lei Liu and Lei Xia and Liang Zhao and Liguo Tan and Liwen Huang and Liying Shi and Ming Li and Mingliang Li and Muhua Cheng and Na Wang and Qiaohui Chen and Qinglin He and Qiuyan Liang and Quan Sun and Ran Sun and Rui Wang and Shaoliang Pang and Shiliang Yang and Sitong Liu and Siqi Liu and Shuli Gao and Tiancheng Cao and Tianyu Wang and Weipeng Ming and Wenqing He and Xu Zhao and Xuelin Zhang and Xianfang Zeng and Xiaojia Liu and Xuan Yang and Yaqi Dai and Yanbo Yu and Yang Li and Yineng Deng and Yingming Wang and Yilei Wang and Yuanwei Lu and Yu Chen and Yu Luo and Yuchu Luo and Yuhe Yin and Yuheng Feng and Yuxiang Yang and Zecheng Tang and Zekai Zhang and Zidong Yang and Binxing Jiao and Jiansheng Chen and Jing Li and Shuchang Zhou and Xiangyu Zhang and Xinhao Zhang and Yibo Zhu and Heung-Yeung Shum and Daxin Jiang}, + year={2025}, + eprint={2502.10248}, + archivePrefix={arXiv}, + primaryClass={cs.CV}, + url={https://arxiv.org/abs/2502.10248}, +} +``` + diff --git a/Step-Video-TI2V/assets/compare_1.png b/Step-Video-TI2V/assets/compare_1.png new file mode 100644 index 0000000..77a3c4c Binary files /dev/null and b/Step-Video-TI2V/assets/compare_1.png differ diff --git a/Step-Video-TI2V/assets/compare_2.png b/Step-Video-TI2V/assets/compare_2.png new file mode 100644 index 0000000..af83e4c Binary files /dev/null and b/Step-Video-TI2V/assets/compare_2.png differ diff --git a/Step-Video-TI2V/assets/logo.png b/Step-Video-TI2V/assets/logo.png new file mode 100644 index 0000000..0b97277 Binary files /dev/null and b/Step-Video-TI2V/assets/logo.png differ diff --git a/Step-Video-TI2V/assets/model.png b/Step-Video-TI2V/assets/model.png new file mode 100644 index 0000000..3e15677 Binary files /dev/null and b/Step-Video-TI2V/assets/model.png differ diff --git a/Step-Video-TI2V/run_parallel.py b/Step-Video-TI2V/run_parallel.py index 67f23ac..2c82d7e 100755 --- a/Step-Video-TI2V/run_parallel.py +++ b/Step-Video-TI2V/run_parallel.py @@ -18,10 +18,7 @@ if __name__ == "__main__": pipeline = StepVideoPipeline.from_pretrained(args.model_dir).to(dtype=torch.bfloat16, device="cpu") pipeline.transformer = pipeline.transformer.to(device) - pipeline.setup_api( - vae_url = args.vae_url, - caption_url = args.caption_url, - ) + pipeline.setup_pipeline(args) prompt = args.prompt @@ -36,7 +33,8 @@ if __name__ == "__main__": time_shift=args.time_shift, pos_magic=args.pos_magic, neg_magic=args.neg_magic, - output_file_name=args.output_file_name + output_file_name=args.output_file_name, + motion_score=args.motion_score ) dist.destroy_process_group() \ No newline at end of file diff --git a/Step-Video-TI2V/stepvideo/config.py b/Step-Video-TI2V/stepvideo/config.py index 01a1173..bf43713 100644 --- a/Step-Video-TI2V/stepvideo/config.py +++ b/Step-Video-TI2V/stepvideo/config.py @@ -156,14 +156,17 @@ def add_inference_args(parser: argparse.ArgumentParser): # Classifier-Free Guidance group.add_argument( - "--pos_magic", type=str, default="超高清、HDR 视频、环境光、杜比全景声、画面稳定、流畅动作、逼真的细节、专业级构图、超现实主义、自然、生动、超细节、清晰。", help="Positive magic prompt for sampling." + "--pos_magic", type=str, default="画面中的主体动作表现生动自然、画面流畅、生动细节、光线统一柔和、超真实动态捕捉、大师级运镜、整体不变形、超高清、画面稳定、逼真的细节、专业级构图、超细节、清晰。", help="Positive magic prompt for sampling." ) group.add_argument( - "--neg_magic", type=str, default="画面暗、低分辨率、不良手、文本、缺少手指、多余的手指、裁剪、低质量、颗粒状、签名、水印、用户名、模糊。", help="Negative magic prompt for sampling." + "--neg_magic", type=str, default="动画、模糊、变形、毁容、低质量、拼贴、粒状、标志、抽象、插图、计算机生成、扭曲、动作不流畅、面部有褶皱、表情僵硬、畸形手指", help="Negative magic prompt for sampling." ) group.add_argument( "--cfg_scale", type=float, default=9.0, help="Classifier free guidance scale." ) + group.add_argument( + "--motion_score", type=float, default=5, help="Score to control the motion level of the video." + ) return parser diff --git a/Step-Video-TI2V/stepvideo/diffusion/video_pipeline.py b/Step-Video-TI2V/stepvideo/diffusion/video_pipeline.py index 824e0bd..6516140 100755 --- a/Step-Video-TI2V/stepvideo/diffusion/video_pipeline.py +++ b/Step-Video-TI2V/stepvideo/diffusion/video_pipeline.py @@ -83,7 +83,7 @@ class StepVideoPipeline(DiffusionPipeline): vae_url: str = '127.0.0.1', caption_url: str = '127.0.0.1', save_path: str = './results', - name_suffix: str = 'comfyui', + name_suffix: str = '', ): super().__init__() @@ -99,7 +99,13 @@ class StepVideoPipeline(DiffusionPipeline): self.vae_url = vae_url self.caption_url = caption_url self.setup_api(self.vae_url, self.caption_url) - + + def setup_pipeline(self, args): + self.args = args + self.video_processor = VideoProcessor(self.args.save_path, self.args.name_suffix) + self.setup_api(args.vae_url, args.caption_url) + return self + def setup_api(self, vae_url, caption_url): self.vae_url = vae_url self.caption_url = caption_url diff --git a/node.py b/node.py index 7a7a018..ae77a34 100644 --- a/node.py +++ b/node.py @@ -32,24 +32,24 @@ class TI2V: "lazy": True }), "infer_steps": ("INT", { - "default": 15, + "default": 50, "min": 0, #Minimum value - "max": 50, #Maximum value + "max": 100, #Maximum value "step": 1, #Slider's step "display": "number", # Cosmetic only: display as "number" or "slider" # "lazy": True # Will only be evaluated if check_lazy_status requires it }), "cfg_scale": ("FLOAT", { - "default": 5.0, + "default": 9, "min": 0.0, - "max": 20.0, + "max": 50.0, "step": 0.1, # "round": 0.001, #The value representing the precision to round to, will be set to the step value by default. Can be set to False to disable rounding. "display": "number", # "lazy": True }), "time_shift": ("FLOAT", { - "default": 17.0, + "default": 13.0, "min": 0.0, "max": 50.0, "step": 0.1, @@ -60,11 +60,20 @@ class TI2V: "num_frames": ("INT", { "default": 102, "min": 0, #Minimum value - "max": 150, #Maximum value + "max": 204, #Maximum value "step": 1, #Slider's step "display": "number", # Cosmetic only: display as "number" or "slider" # "lazy": True # Will only be evaluated if check_lazy_status requires it }), + "motion_score": ("FLOAT", { + "default": 5, + "min": 0.0, + "max": 50.0, + "step": 0.1, + # "round": 0.001, #The value representing the precision to round to, will be set to the step value by default. Can be set to False to disable rounding. + "display": "number", + # "lazy": True + }), "text_prompt": ("STRING", { "multiline": True, "default": "笑起来", @@ -76,7 +85,7 @@ class TI2V: RETURN_TYPES = ("IMAGE",) FUNCTION = "ti2v" - def ti2v(self, image_input, remote_server_url, model_dir, infer_steps, cfg_scale, time_shift, num_frames, text_prompt): + def ti2v(self, image_input, remote_server_url, model_dir, infer_steps, cfg_scale, time_shift, num_frames, motion_score, text_prompt): script_dir = 'custom_nodes/ComfyUI-StepVideo/Step-Video-TI2V' os.makedirs(f'{script_dir}/results', exist_ok=True) @@ -85,7 +94,7 @@ class TI2V: parallel = 4 # - command = f'cd {script_dir} && torchrun --nproc_per_node {parallel} run_parallel.py --model_dir {model_dir} --vae_url {remote_server_url} --caption_url {remote_server_url} --ulysses_degree {parallel} --first_image_path results/{task_name}_img.png --prompt "{text_prompt}" --infer_steps {infer_steps} --cfg_scale {cfg_scale} --time_shift {time_shift} --num_frames {num_frames} --output_file_name {task_name}_vid' + command = f'cd {script_dir} && torchrun --nproc_per_node {parallel} run_parallel.py --model_dir {model_dir} --vae_url {remote_server_url} --caption_url {remote_server_url} --ulysses_degree {parallel} --first_image_path results/{task_name}_img.png --prompt "{text_prompt}" --infer_steps {infer_steps} --cfg_scale {cfg_scale} --time_shift {time_shift} --num_frames {num_frames} --output_file_name {task_name}_vid --motion_score {motion_score} --name_suffix comfyui' os.system(command) video_path = f'{script_dir}/results/{task_name}_vid-comfyui.mp4' diff --git a/workflow/TI2V.jpg b/workflow/TI2V.jpg index ec74bcc..3b42b79 100644 Binary files a/workflow/TI2V.jpg and b/workflow/TI2V.jpg differ