diff --git a/docs/README.md b/docs/README.md index 0bcab3afc..1234fb599 100644 --- a/docs/README.md +++ b/docs/README.md @@ -6,7 +6,7 @@ This directory contains the FastVideo documentation built with MkDocs. ```bash # Install dependencies -pip install -r docs/requirements-mkdocs.txt +pip install -r requirements-mkdocs.txt # Serve docs with live reload (recommended for development) mkdocs serve diff --git a/docs/contributing/overview.md b/docs/contributing/overview.md index 9c71ccd86..f6547c2b0 100644 --- a/docs/contributing/overview.md +++ b/docs/contributing/overview.md @@ -6,7 +6,7 @@ Thank you for your interest in contributing to FastVideo. We want to make the pr Our community is open to everyone and welcomes any contributions no matter how large or small. # Developer Environment: -Do make sure you have CUDA 12.4 installed and supported. FastVideo currently only support Linux and CUDA GPUs, but we hope to support other platforms in the future. +Do make sure you have CUDA 12.4 installed and supported. FastVideo currently only supports Linux and CUDA GPUs, but we hope to support other platforms in the future. We recommend using a fresh Python 3.10 Conda environment to develop FastVideo: diff --git a/docs/contributing/profiling.md b/docs/contributing/profiling.md index d2dacb65f..077951686 100644 --- a/docs/contributing/profiling.md +++ b/docs/contributing/profiling.md @@ -1,7 +1,7 @@ # Profiling FastVideo !!! warning - Profiling is only intended for FastVideo developers and maintainers to understand the proportion of time spent in different parts of the codebase. **FastVideo end-users should never turn on profiling** as it will significantly slow down the inference. + Profiling is only intended for FastVideo developers and maintainers to understand the proportion of time spent in different parts of the codebase. **FastVideo end-users should never turn on profiling** as it will significantly slow down inference. ## Profiling with PyTorch @@ -49,5 +49,5 @@ Traces can be visualized using . ### Best Practices - Keep the profiled step count small; traces can be large and slow down job shutdown while the profiler flushes data. -- After profiling, clean up trace directories to avoid filling disks. +- After profiling, clean up trace directories to avoid filling disk storage. - When adding new regions, register them in `fastvideo.profiler` and wrap the corresponding code block with `with self.profiler_controller.region("your_region"):` or the `@profile_region` decorator. diff --git a/docs/contributing/testing.md b/docs/contributing/testing.md index 0a3414529..f611ea688 100644 --- a/docs/contributing/testing.md +++ b/docs/contributing/testing.md @@ -74,9 +74,11 @@ To add a new SSIM test, follow these steps: generator.generate_video(prompt, ...) # Compare with Reference - ssim_values = compute_video_ssim_torchvision(reference_path, generated_path, use_ms_ssim=True) - assert ssim_values[0] >= 0.98 # Threshold - ``` + ssim_values = compute_video_ssim_torchvision( + reference_path, generated_path, use_ms_ssim=True + ) + assert ssim_values[0] >= 0.98 # Threshold + ``` 4. **Reference Videos**: * When running the test for the first time (or when updating the reference), the test will fail because the reference video is missing. The generated video will be saved in `fastvideo/tests/ssim/generated_videos`. diff --git a/docs/distillation/dmd.md b/docs/distillation/dmd.md index b190d613d..f166f5074 100644 --- a/docs/distillation/dmd.md +++ b/docs/distillation/dmd.md @@ -1,6 +1,6 @@ # 🎯 Distillation -We introduce a new finetuning strategy - **Sparse-distill**, which jointly integrates **[DMD](https://arxiv.org/abs/2405.14867)** and **[VSA](https://arxiv.org/abs/2505.13389)** in a single training process. This approach combines the benefits of both distillation to shorten diffusion steps and sparse attention to reduce attention computations, enabling much faster video generation. +We introduce a new finetuning strategy - **Sparse-distill**, which jointly integrates **[DMD](https://arxiv.org/abs/2405.14867)** and **[VSA](https://arxiv.org/abs/2505.13389)** in a single training process. This approach combines the benefits of both distillation to shorten diffusion steps and sparse attention to reduce attention computation, enabling much faster video generation. ## 📊 Model Overview diff --git a/docs/getting_started/quick_start.md b/docs/getting_started/quick_start.md index f6db8620f..a2fd39168 100644 --- a/docs/getting_started/quick_start.md +++ b/docs/getting_started/quick_start.md @@ -7,6 +7,11 @@ Get up and running with FastVideo in minutes! First, install FastVideo: ```bash +# Create and activate a new conda environment +conda create -n fastvideo python=3.12 +conda activate fastvideo + +# Install FastVideo pip install fastvideo ``` @@ -15,36 +20,53 @@ pip install fastvideo ### Text-to-Video Generation ```python -from fastvideo import FastVideoPipeline +from fastvideo import VideoGenerator -# Initialize the pipeline -pipe = FastVideoPipeline.from_pretrained("wan2.1-t2v-1.3B") +def main(): + # Create a video generator with a pre-trained model + generator = VideoGenerator.from_pretrained( + "Wan-AI/Wan2.1-T2V-1.3B-Diffusers", + num_gpus=1, # Adjust based on your hardware + ) -# Generate a video -prompt = "A cat playing with a ball of yarn" -video = pipe(prompt, num_frames=16, height=512, width=512) + # Define a prompt for your video + prompt = "A curious raccoon peers through a vibrant field of yellow sunflowers, its eyes wide with interest." -# Save the video -video.save("output.mp4") + # Generate the video + video = generator.generate_video( + prompt, + return_frames=True, # Also return frames from this call (defaults to False) + output_path="my_videos/", # Controls where videos are saved + save_video=True + ) + +if __name__ == '__main__': + main() ``` ### Image-to-Video Generation ```python -from fastvideo import FastVideoPipeline -from PIL import Image +from fastvideo import VideoGenerator, SamplingParam -# Load an image -image = Image.open("input.jpg") +def main(): + # Create the generator + model_name = "Wan-AI/Wan2.1-I2V-14B-480P-Diffusers" + generator = VideoGenerator.from_pretrained(model_name, num_gpus=1) -# Initialize the pipeline -pipe = FastVideoPipeline.from_pretrained("wan2.1-i2v-14B-480p") + # Set up parameters with an initial image + sampling_param = SamplingParam.from_pretrained(model_name) + sampling_param.image_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg" + sampling_param.num_frames = 107 -# Generate a video from the image -video = pipe(image, num_frames=16, height=480, width=480) + # Generate video based on the image + prompt = "A photograph coming to life with gentle movement" + generator.generate_video(prompt, sampling_param=sampling_param, + output_path="my_videos/", + save_video=True) -# Save the video -video.save("output.mp4") +if __name__ == '__main__': + main() ``` ## Next Steps diff --git a/docs/sliding_tile_attention/installation.md b/docs/sliding_tile_attention/installation.md index 0ed54f3db..25d183ab2 100644 --- a/docs/sliding_tile_attention/installation.md +++ b/docs/sliding_tile_attention/installation.md @@ -7,7 +7,7 @@ pip install st_attn ``` # Building from Source -We test our code on Pytorch 2.5.0 and CUDA>=12.4. Currently we only have implementation on H100. +We test our code on Pytorch 2.5.0 and CUDA>=12.4. Currently, we only have an implementation for H100s. First, install C++20 for ThunderKittens: ```bash diff --git a/docs/training/data_preprocess.md b/docs/training/data_preprocess.md index dd6c34990..f5755b375 100644 --- a/docs/training/data_preprocess.md +++ b/docs/training/data_preprocess.md @@ -30,7 +30,7 @@ path_to_your_dataset_folder/ └── prompt.txt ``` -To geranate the `videos2caption.json` and `merge.txt`, run +To generate the `videos2caption.json` and `merge.txt`, run ``` python python scripts/dataset_preparation/prepare_json_file.py --data_folder mini_i2v_dataset/crush-smol_raw/ --output your_output_folder diff --git a/docs/video_sparse_attention/installation.md b/docs/video_sparse_attention/installation.md index 4da426b57..6ca95bd6d 100644 --- a/docs/video_sparse_attention/installation.md +++ b/docs/video_sparse_attention/installation.md @@ -7,9 +7,9 @@ pip install vsa ``` # Building from Source -We support H100 (via ThunderKittens) and any other GPU (via Triton) for VSA. +We support H100s (via ThunderKittens) and any other GPU (via Triton) for VSA. -First, install C++20 for ThunderKittens (if using H100): +First, install C++20 for ThunderKittens (if using an H100): ```bash sudo apt update