updater diffusers example
This commit is contained in:
Shenghai Yuan
@@ -55,28 +55,6 @@ or you can click <a href="https://www.youtube.com/watch?v=vd_AgHtOUFQ">here</a>
|
||||
[](https://www.youtube.com/watch?v=vd_AgHtOUFQ)
|
||||
or you can click <a href="https://github.com/PKU-YuanGroup/Helios-Page/blob/main/videos/helios_features.mp4">here</a> to get the video. Some best prompts are [here](./example/prompt.txt).
|
||||
|
||||
<br>
|
||||
|
||||
<details open><summary>💡 We also have other video generation projects that may interest you ✨. </summary><p>
|
||||
<!-- may -->
|
||||
|
||||
> [**Open-Sora Plan: Open-Source Large Video Generation Model**](https://arxiv.org/abs/2412.00131) <br>
|
||||
> Bin Lin, Yunyang Ge and Xinhua Cheng etc. <br>
|
||||
[](https://github.com/PKU-YuanGroup/Open-Sora-Plan) [](https://github.com/PKU-YuanGroup/Open-Sora-Plan) [](https://arxiv.org/abs/2412.00131) <br>
|
||||
>
|
||||
> [**OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation**](https://arxiv.org/abs/2505.20292) <br>
|
||||
> Shenghai Yuan, Xianyi He and Yufan Deng etc. <br>
|
||||
> [](https://github.com/PKU-YuanGroup/OpenS2V-Nexus) [](https://github.com/PKU-YuanGroup/OpenS2V-Nexus) [](https://arxiv.org/abs/2505.20292) <br>
|
||||
>
|
||||
> [**ConsisID: Identity-Preserving Text-to-Video Generation by Frequency Decomposition**](https://arxiv.org/abs/2411.17440) <br>
|
||||
> Shenghai Yuan, Jinfa Huang and Xianyi He etc. <br>
|
||||
> [](https://github.com/PKU-YuanGroup/ConsisID/) [](https://github.com/PKU-YuanGroup/ConsisID/) [](https://arxiv.org/abs/2411.17440) <br>
|
||||
>
|
||||
> [**MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators**](https://arxiv.org/abs/2404.05014) <br>
|
||||
> Shenghai Yuan, Jinfa Huang and Yujun Shi etc. <br>
|
||||
> [](https://github.com/PKU-YuanGroup/MagicTime) [](https://github.com/PKU-YuanGroup/MagicTime) [](https://arxiv.org/abs/2404.05014) <br>
|
||||
> </p></details>
|
||||
|
||||
|
||||
## 📣 Latest News!!
|
||||
|
||||
@@ -233,7 +211,101 @@ Install diffusers from source:
|
||||
pip install git+https://github.com/huggingface/diffusers.git
|
||||
```
|
||||
|
||||
For example, let's take Helios-Distilled.
|
||||
For example, let's take Helios-Distilled (**Standard Pipeline**).
|
||||
|
||||
<details>
|
||||
<summary>Click to expand the code</summary>
|
||||
|
||||
```bash
|
||||
import torch
|
||||
from diffusers import AutoModel, HeliosPyramidPipeline
|
||||
from diffusers.utils import export_to_video, load_video, load_image
|
||||
|
||||
vae = AutoModel.from_pretrained("BestWishYsh/Helios-Distilled", subfolder="vae", torch_dtype=torch.float32)
|
||||
|
||||
pipeline = HeliosPyramidPipeline.from_pretrained(
|
||||
"BestWishYsh/Helios-Distilled",
|
||||
vae=vae,
|
||||
torch_dtype=torch.bfloat16
|
||||
)
|
||||
pipeline.to("cuda")
|
||||
|
||||
negative_prompt = """
|
||||
Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality,
|
||||
low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured,
|
||||
misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards
|
||||
"""
|
||||
|
||||
# --- T2V ---
|
||||
prompt = """
|
||||
A vibrant tropical fish swimming gracefully among colorful coral reefs in a clear, turquoise ocean. The fish has bright blue
|
||||
and yellow scales with a small, distinctive orange spot on its side, its fins moving fluidly. The coral reefs are alive with
|
||||
a variety of marine life, including small schools of colorful fish and sea turtles gliding by. The water is crystal clear,
|
||||
allowing for a view of the sandy ocean floor below. The reef itself is adorned with a mix of hard and soft corals in shades
|
||||
of red, orange, and green. The photo captures the fish from a slightly elevated angle, emphasizing its lively movements and
|
||||
the vivid colors of its surroundings. A close-up shot with dynamic movement.
|
||||
"""
|
||||
|
||||
output = pipeline(
|
||||
prompt=prompt,
|
||||
negative_prompt=negative_prompt,
|
||||
num_frames=240,
|
||||
pyramid_num_inference_steps_list=[2, 2, 2],
|
||||
guidance_scale=1.0,
|
||||
is_amplify_first_chunk=True,
|
||||
generator=torch.Generator("cuda").manual_seed(42),
|
||||
).frames[0]
|
||||
export_to_video(output, "helios_distilled_t2v_output.mp4", fps=24)
|
||||
|
||||
# --- I2V ---
|
||||
i2v_prompt = """
|
||||
A towering emerald wave surges forward, its crest curling with raw power and energy. Sunlight glints off the translucent water,
|
||||
illuminating the intricate textures and deep green hues within the wave’s body. A thick spray erupts from the breaking crest,
|
||||
casting a misty veil that dances above the churning surface. As the perspective widens, the immense scale of the wave becomes
|
||||
apparent, revealing the restless expanse of the ocean stretching beyond. The scene captures the ocean’s untamed beauty and
|
||||
relentless force, with every droplet and ripple shimmering in the light. The dynamic motion and vivid colors evoke both awe and
|
||||
respect for nature’s might.
|
||||
"""
|
||||
image_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/helios/wave.jpg"
|
||||
|
||||
output = pipeline(
|
||||
prompt=i2v_prompt,
|
||||
negative_prompt=negative_prompt,
|
||||
image=load_image(image_path).resize((640, 384)),
|
||||
num_frames=240,
|
||||
pyramid_num_inference_steps_list=[2, 2, 2],
|
||||
guidance_scale=1.0,
|
||||
is_amplify_first_chunk=True,
|
||||
generator=torch.Generator("cuda").manual_seed(42),
|
||||
).frames[0]
|
||||
export_to_video(output, "helios_distilled_i2v_output.mp4", fps=24)
|
||||
|
||||
# --- V2V ---
|
||||
v2v_prompt = """
|
||||
A bright yellow Lamborghini Huracn Tecnica speeds along a curving mountain road, surrounded by lush green trees
|
||||
under a partly cloudy sky. The car's sleek design and vibrant color stand out against the natural backdrop,
|
||||
emphasizing its dynamic movement. The road curves gently, with a guardrail visible on one side, adding depth to
|
||||
the scene. The motion blur captures the sense of speed and energy, creating a thrilling and exhilarating atmosphere.
|
||||
A front-facing shot from a slightly elevated angle, highlighting the car's aggressive stance and the surrounding greenery.
|
||||
"""
|
||||
video_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/helios/car.mp4"
|
||||
|
||||
output = pipeline(
|
||||
prompt=v2v_prompt,
|
||||
negative_prompt=negative_prompt,
|
||||
video=load_video(video_path),
|
||||
num_frames=240,
|
||||
pyramid_num_inference_steps_list=[2, 2, 2],
|
||||
guidance_scale=1.0,
|
||||
is_amplify_first_chunk=True,
|
||||
generator=torch.Generator("cuda").manual_seed(42),
|
||||
).frames[0]
|
||||
export_to_video(output, "helios_distilled_v2v_output.mp4", fps=24)
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
For example, let's take Helios-Distilled (**Modular Pipeline**).
|
||||
|
||||
<details>
|
||||
<summary>Click to expand the code</summary>
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Please refer to here for installation: https://github.com/Ascend/pytorch?tab=readme-ov-file#ascend-auxiliary-software
|
||||
torch==2.7.1
|
||||
torchvision==0.22.1
|
||||
torchaudio==2.7.1
|
||||
triton==3.3.1
|
||||
torch_npu==2.7.1 # or torch_npu=2.7.1.post2
|
||||
# Please refer to here for installation the latest version: https://github.com/Ascend/pytorch?tab=readme-ov-file#ascend-auxiliary-software
|
||||
torch==2.9.0
|
||||
torchvision==0.24.0
|
||||
torchaudio==2.9.0
|
||||
torch_npu==2.9.0
|
||||
triton==3.5.1
|
||||
|
||||
Reference in New Issue
Block a user