From cb524be214e7f564735cd5b17245848a3362aea5 Mon Sep 17 00:00:00 2001 From: Shenghai Yuan Date: Sun, 8 Mar 2026 12:41:05 +0800 Subject: [PATCH] updater diffusers example --- README.md | 118 ++++++++++++++++++++++++++++++++++--------- requirements_npu.txt | 12 ++--- 2 files changed, 101 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index 0876022..20c2c02 100644 --- a/README.md +++ b/README.md @@ -55,28 +55,6 @@ or you can click here [![Demo Video of Helios](https://github.com/user-attachments/assets/1d10da4a-aba9-4ac1-ab02-cd0dfce8d35b)](https://www.youtube.com/watch?v=vd_AgHtOUFQ) or you can click here to get the video. Some best prompts are [here](./example/prompt.txt). -
- -
💡 We also have other video generation projects that may interest you ✨.

- - -> [**Open-Sora Plan: Open-Source Large Video Generation Model**](https://arxiv.org/abs/2412.00131)
-> Bin Lin, Yunyang Ge and Xinhua Cheng etc.
-[![github](https://img.shields.io/badge/-Github-black?logo=github)](https://github.com/PKU-YuanGroup/Open-Sora-Plan) [![github](https://img.shields.io/github/stars/PKU-YuanGroup/Open-Sora-Plan.svg?style=social)](https://github.com/PKU-YuanGroup/Open-Sora-Plan) [![arXiv](https://img.shields.io/badge/Arxiv-2412.00131-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2412.00131)
-> -> [**OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation**](https://arxiv.org/abs/2505.20292)
-> Shenghai Yuan, Xianyi He and Yufan Deng etc.
-> [![github](https://img.shields.io/badge/-Github-black?logo=github)](https://github.com/PKU-YuanGroup/OpenS2V-Nexus) [![github](https://img.shields.io/github/stars/PKU-YuanGroup/OpenS2V-Nexus.svg?style=social)](https://github.com/PKU-YuanGroup/OpenS2V-Nexus) [![arXiv](https://img.shields.io/badge/Arxiv-2505.20292-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2505.20292)
-> -> [**ConsisID: Identity-Preserving Text-to-Video Generation by Frequency Decomposition**](https://arxiv.org/abs/2411.17440)
-> Shenghai Yuan, Jinfa Huang and Xianyi He etc.
-> [![github](https://img.shields.io/badge/-Github-black?logo=github)](https://github.com/PKU-YuanGroup/ConsisID/) [![github](https://img.shields.io/github/stars/PKU-YuanGroup/ConsisID.svg?style=social)](https://github.com/PKU-YuanGroup/ConsisID/) [![arXiv](https://img.shields.io/badge/Arxiv-2411.17440-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2411.17440)
-> -> [**MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators**](https://arxiv.org/abs/2404.05014)
-> Shenghai Yuan, Jinfa Huang and Yujun Shi etc.
-> [![github](https://img.shields.io/badge/-Github-black?logo=github)](https://github.com/PKU-YuanGroup/MagicTime) [![github](https://img.shields.io/github/stars/PKU-YuanGroup/MagicTime.svg?style=social)](https://github.com/PKU-YuanGroup/MagicTime) [![arXiv](https://img.shields.io/badge/Arxiv-2404.05014-b31b1b.svg?logo=arXiv)](https://arxiv.org/abs/2404.05014)
->

- ## 📣 Latest News!! @@ -233,7 +211,101 @@ Install diffusers from source: pip install git+https://github.com/huggingface/diffusers.git ``` -For example, let's take Helios-Distilled. +For example, let's take Helios-Distilled (**Standard Pipeline**). + +
+ Click to expand the code + + ```bash + import torch + from diffusers import AutoModel, HeliosPyramidPipeline + from diffusers.utils import export_to_video, load_video, load_image + + vae = AutoModel.from_pretrained("BestWishYsh/Helios-Distilled", subfolder="vae", torch_dtype=torch.float32) + + pipeline = HeliosPyramidPipeline.from_pretrained( + "BestWishYsh/Helios-Distilled", + vae=vae, + torch_dtype=torch.bfloat16 + ) + pipeline.to("cuda") + + negative_prompt = """ + Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, + low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, + misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards + """ + + # --- T2V --- + prompt = """ + A vibrant tropical fish swimming gracefully among colorful coral reefs in a clear, turquoise ocean. The fish has bright blue + and yellow scales with a small, distinctive orange spot on its side, its fins moving fluidly. The coral reefs are alive with + a variety of marine life, including small schools of colorful fish and sea turtles gliding by. The water is crystal clear, + allowing for a view of the sandy ocean floor below. The reef itself is adorned with a mix of hard and soft corals in shades + of red, orange, and green. The photo captures the fish from a slightly elevated angle, emphasizing its lively movements and + the vivid colors of its surroundings. A close-up shot with dynamic movement. + """ + + output = pipeline( + prompt=prompt, + negative_prompt=negative_prompt, + num_frames=240, + pyramid_num_inference_steps_list=[2, 2, 2], + guidance_scale=1.0, + is_amplify_first_chunk=True, + generator=torch.Generator("cuda").manual_seed(42), + ).frames[0] + export_to_video(output, "helios_distilled_t2v_output.mp4", fps=24) + + # --- I2V --- + i2v_prompt = """ + A towering emerald wave surges forward, its crest curling with raw power and energy. Sunlight glints off the translucent water, + illuminating the intricate textures and deep green hues within the wave’s body. A thick spray erupts from the breaking crest, + casting a misty veil that dances above the churning surface. As the perspective widens, the immense scale of the wave becomes + apparent, revealing the restless expanse of the ocean stretching beyond. The scene captures the ocean’s untamed beauty and + relentless force, with every droplet and ripple shimmering in the light. The dynamic motion and vivid colors evoke both awe and + respect for nature’s might. + """ + image_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/helios/wave.jpg" + + output = pipeline( + prompt=i2v_prompt, + negative_prompt=negative_prompt, + image=load_image(image_path).resize((640, 384)), + num_frames=240, + pyramid_num_inference_steps_list=[2, 2, 2], + guidance_scale=1.0, + is_amplify_first_chunk=True, + generator=torch.Generator("cuda").manual_seed(42), + ).frames[0] + export_to_video(output, "helios_distilled_i2v_output.mp4", fps=24) + + # --- V2V --- + v2v_prompt = """ + A bright yellow Lamborghini Huracn Tecnica speeds along a curving mountain road, surrounded by lush green trees + under a partly cloudy sky. The car's sleek design and vibrant color stand out against the natural backdrop, + emphasizing its dynamic movement. The road curves gently, with a guardrail visible on one side, adding depth to + the scene. The motion blur captures the sense of speed and energy, creating a thrilling and exhilarating atmosphere. + A front-facing shot from a slightly elevated angle, highlighting the car's aggressive stance and the surrounding greenery. + """ + video_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/helios/car.mp4" + + output = pipeline( + prompt=v2v_prompt, + negative_prompt=negative_prompt, + video=load_video(video_path), + num_frames=240, + pyramid_num_inference_steps_list=[2, 2, 2], + guidance_scale=1.0, + is_amplify_first_chunk=True, + generator=torch.Generator("cuda").manual_seed(42), + ).frames[0] + export_to_video(output, "helios_distilled_v2v_output.mp4", fps=24) + ``` + +
+ +For example, let's take Helios-Distilled (**Modular Pipeline**).
Click to expand the code diff --git a/requirements_npu.txt b/requirements_npu.txt index fa48a37..559be7d 100644 --- a/requirements_npu.txt +++ b/requirements_npu.txt @@ -1,6 +1,6 @@ -# Please refer to here for installation: https://github.com/Ascend/pytorch?tab=readme-ov-file#ascend-auxiliary-software -torch==2.7.1 -torchvision==0.22.1 -torchaudio==2.7.1 -triton==3.3.1 -torch_npu==2.7.1 # or torch_npu=2.7.1.post2 \ No newline at end of file +# Please refer to here for installation the latest version: https://github.com/Ascend/pytorch?tab=readme-ov-file#ascend-auxiliary-software +torch==2.9.0 +torchvision==0.24.0 +torchaudio==2.9.0 +torch_npu==2.9.0 +triton==3.5.1