From 0cf001986a800977f45ec725e417c7833aba62ce Mon Sep 17 00:00:00 2001 From: William Lin Date: Sun, 11 May 2025 16:19:47 -0700 Subject: [PATCH] [Docs] More docs update (#394) --- .pre-commit-config.yaml | 3 +- docs/source/getting_started/installation.md | 86 ++++++---- docs/source/getting_started/v1_api.md | 71 ++++++++- docs/source/index.md | 11 +- .../add_pipeline.md | 2 +- .../source/inference/inference_quick_start.md | 141 +---------------- docs/source/inference/optimizations.md | 147 ++++++++++++++++++ docs/source/inference/support_matrix.md | 92 +++++++++-- examples/inference/basic/README.md | 2 +- examples/inference/gradio/README.md | 16 +- examples/inference/optimizations/README.md | 125 +-------------- 11 files changed, 363 insertions(+), 333 deletions(-) rename docs/source/{contributing => inference}/add_pipeline.md (99%) create mode 100644 docs/source/inference/optimizations.md diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index f50aba44b..ad2ba3a75 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -22,7 +22,8 @@ exclude: | examples/.*| .github/workflows/fastvideo-publish.yml| .github/workflows/sta-publish.yml| - .github/workflows/build-image-template.yml + .github/workflows/build-image-template.yml| + docs/source/inference/support_matrix.md ) repos: - repo: https://github.com/google/yapf diff --git a/docs/source/getting_started/installation.md b/docs/source/getting_started/installation.md index 9b20831d1..e9a204ccf 100644 --- a/docs/source/getting_started/installation.md +++ b/docs/source/getting_started/installation.md @@ -11,11 +11,52 @@ FastVideo currently only supports Linux and NVIDIA CUDA GPUs. - **CUDA 12.4** - **At least 1 NVIDIA GPU** -## Quick start -We recommend using an environment manager such as [Conda](#miniconda-recommended) +## Set up using Python +### Create a new Python environment + +#### Conda +You can create a new python environment using [Conda](https://docs.conda.io/projects/conda/en/stable/user-guide/getting-started.html) +##### 1. Install Miniconda (if not already installed) + +```bash +wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh +bash Miniconda3-latest-Linux-x86_64.sh +source ~/.bashrc +``` + +##### 2. Create and activate a Conda environment for FastVideo + +```bash +# (Recommended) Create a new conda environment. +conda create -n fastvideo python=3.12 -y +conda activate fastvideo +``` + +:::{note} +[PyTorch has deprecated the conda release channel](https://github.com/pytorch/pytorch/issues/138506). If you use `conda`, please only use it to create Python environment rather than installing packages. +::: + +#### uv + +:::{tip} +We highly recommend using `uv` to install FastVideo. In our experience, `uv` speeds up installation by at least 3x. +::: + +Or you can create a new Python environment using [uv](https://docs.astral.sh/uv/), a very fast Python environment manager. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment using the following command: + +```console +# (Recommended) Create a new uv environment. Use `--seed` to install `pip` and `setuptools` in the environment. +uv venv --python 3.12 --seed +source .venv/bin/activate +``` + +### Installation ```bash pip install fastvideo + +# or if you are using uv +uv pip install fastvideo ``` Also optionally install flash-attn: @@ -24,30 +65,9 @@ Also optionally install flash-attn: pip install flash-attn==2.7.4.post1 --no-build-isolation ``` -[Click here to install from source](#installation-from-source) +### Installation from Source -## Environment Setup - -We highly recommend using a `conda` or `virtualenv` environment: -### Miniconda (Recommended) -#### 1. Install Miniconda (if not already installed) - -```bash -wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -bash Miniconda3-latest-Linux-x86_64.sh -source ~/.bashrc -``` - -#### 2. Create and activate a Conda environment for FastVideo - -```bash -conda create -n fastvideo python=3.12 -y -conda activate fastvideo -``` - -## Installation from Source - -### 1. Clone the FastVideo repository +#### 1. Clone the FastVideo repository ```bash git clone https://github.com/hao-ai-lab/FastVideo.git && cd FastVideo @@ -59,23 +79,23 @@ Basic installation: ```bash pip install -e . + +# or if you are using uv +uv pip install -e . ``` -## Optional Dependencies +### Optional Dependencies -### Flash Attention +#### Flash Attention ```bash pip install flash-attn==2.7.4.post1 --no-build-isolation ``` -### Sliding Tile Attention (STA) (Requires CUDA 12.4 and H100) - -To try Sliding Tile Attention (optional), please follow the instructions in [csrc/sliding_tile_attention/README.md](#sta-installation) to install STA. - -## Docker Images +## Set up using Docker We also have prebuilt docker images with FastVideo dependencies pre-installed: [Docker Images](#docker) + ## Development Environment Setup If you're planning to contribute to FastVideo please see the following page: @@ -84,7 +104,7 @@ If you're planning to contribute to FastVideo please see the following page: ## Hardware Requirements ### For Basic Inference -- NVIDIA GPU with CUDA 12.4support +- NVIDIA GPU with CUDA 12.4 support ### For Lora Finetuning - 40GB GPU memory each for 2 GPUs with lora diff --git a/docs/source/getting_started/v1_api.md b/docs/source/getting_started/v1_api.md index 15d39a99f..cba870c25 100644 --- a/docs/source/getting_started/v1_api.md +++ b/docs/source/getting_started/v1_api.md @@ -10,9 +10,74 @@ This class will be the primary Python API for generating videos and images. fastvideo.VideoGenerator ``` -VideoGenerator.from_pretrained() should be the primary way of creating a new video generator. +`````{py:class} VideoGenerator(fastvideo_args: fastvideo.v1.fastvideo_args.FastVideoArgs, executor_class: type[fastvideo.v1.worker.executor.Executor], log_stats: bool) +:canonical: fastvideo.v1.entrypoints.video_generator.VideoGenerator -````{py:method} from_pretrained(model_path: str, device: typing.optional[str] = none, torch_dtype: typing.optional[torch.dtype] = none, pipeline_config: typing.optional[typing.union[str | fastvideo.v1.configs.pipelines.pipelineconfig]] = none, **kwargs) -> v1.entrypoints.video_generator.videogenerator -:canonical: v1.entrypoints.video_generator.videogenerator.from_pretrained +```{autodoc2-docstring} fastvideo.v1.entrypoints.video_generator.VideoGenerator +:parser: docs.source.autodoc2_docstring_parser +``` + +`VideoGenerator.from_pretrained()` should be the primary way of creating a new video generator. + +````{py:method} from_pretrained(model_path: str, device: typing.Optional[str] = None, torch_dtype: typing.Optional[torch.dtype] = None, pipeline_config: typing.Optional[typing.Union[str | fastvideo.v1.configs.pipelines.PipelineConfig]] = None, **kwargs) -> fastvideo.v1.entrypoints.video_generator.VideoGenerator +:canonical: fastvideo.v1.entrypoints.video_generator.VideoGenerator.from_pretrained :classmethod: + +```{autodoc2-docstring} fastvideo.v1.entrypoints.video_generator.VideoGenerator.from_pretrained +:parser: docs.source.autodoc2_docstring_parser +``` + + +## Configuring FastVideo + +The follow two classes `PipelineConfig` and `SamplingParam` are used to configure initialization and sampling parameters, respectively. + +### PipelineConfig +```{autodoc2-summary} + fastvideo.PipelineConfig +``` + +`````{py:class} PipelineConfig +:canonical: fastvideo.v1.configs.pipelines.base.PipelineConfig + +```{autodoc2-docstring} fastvideo.v1.configs.pipelines.base.PipelineConfig +:parser: docs.source.autodoc2_docstring_parser +``` + +````{py:method} from_pretrained(model_path: str) -> fastvideo.v1.configs.pipelines.base.PipelineConfig +:canonical: fastvideo.v1.configs.pipelines.base.PipelineConfig.from_pretrained +:classmethod: + +```{autodoc2-docstring} fastvideo.v1.configs.pipelines.base.PipelineConfig.from_pretrained +:parser: docs.source.autodoc2_docstring_parser +``` + + +````{py:method} dump_to_json(file_path: str) +:canonical: fastvideo.v1.configs.pipelines.base.PipelineConfig.dump_to_json + +```{autodoc2-docstring} fastvideo.v1.configs.pipelines.base.PipelineConfig.dump_to_json +:parser: docs.source.autodoc2_docstring_parser +``` + + +### SamplingParam + +```{autodoc2-summary} + fastvideo.SamplingParam +``` + +`````{py:class} SamplingParam +:canonical: fastvideo.v1.configs.sample.base.SamplingParam + +```{autodoc2-docstring} fastvideo.v1.configs.sample.base.SamplingParam +:parser: docs.source.autodoc2_docstring_parser +``` + +````{py:method} from_pretrained(model_path: str) -> fastvideo.v1.configs.sample.base.SamplingParam +:canonical: fastvideo.v1.configs.sample.base.SamplingParam.from_pretrained +:classmethod: + +```{autodoc2-docstring} fastvideo.v1.configs.sample.base.SamplingParam.from_pretrained +:parser: docs.source.autodoc2_docstring_parser ``` diff --git a/docs/source/index.md b/docs/source/index.md index f1d342966..69ba5895b 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -51,7 +51,7 @@ Dev in progress and highly experimental. :maxdepth: 1 getting_started/installation -getting_started/v1_api + ::: :::{toctree} @@ -60,10 +60,10 @@ getting_started/v1_api inference/inference_quick_start inference/configuration +inference/optimizations inference/support_matrix -inference/examples/basic -inference/examples/gradio -inference/examples/optimizations +inference/examples/examples_inference_index +inference/add_pipeline inference/v0_inference ::: @@ -98,14 +98,13 @@ design/overview contributing/overview contributing/developer_env/index -contributing/add_pipeline ::: :::{toctree} :caption: API Reference :maxdepth: 2 -api/summary + api/fastvideo/fastvideo ::: diff --git a/docs/source/contributing/add_pipeline.md b/docs/source/inference/add_pipeline.md similarity index 99% rename from docs/source/contributing/add_pipeline.md rename to docs/source/inference/add_pipeline.md index 919f7c223..3acd384e3 100644 --- a/docs/source/contributing/add_pipeline.md +++ b/docs/source/inference/add_pipeline.md @@ -1,6 +1,6 @@ (add-pipeline)= -# 🏗️ Adding a New Diffusion Pipeline +# 🏗️ Adding a New Pipeline This guide explains how to implement a custom diffusion pipeline in FastVideo, leveraging the framework's modular architecture for high-performance video generation. diff --git a/docs/source/inference/inference_quick_start.md b/docs/source/inference/inference_quick_start.md index 25ed61887..f9d800bba 100644 --- a/docs/source/inference/inference_quick_start.md +++ b/docs/source/inference/inference_quick_start.md @@ -3,15 +3,10 @@ This page contains step-by-step instructions to get you quickly started with video generation using FastVideo. ## Table of Contents -- [Software Requirements](#software-requirements) -- [Installation](#installation) - [Generating Your First Video](#generating-your-first-video) - [Customizing Generation](#customizing-generation) - [Available Models](#available-models) -- [Multi-GPU Setup](#multi-gpu-setup) - [Image-to-Video Generation](#image-to-video-generation) -- [Output Formats and Options](#output-formats-and-options) -- [Performance Optimization](#performance-optimization) - [Troubleshooting](#troubleshooting) - [Advanced Configuration](#advanced-configuration) - [Next Steps](#next-steps) @@ -27,7 +22,7 @@ We recommend using an environment manager such as `Conda` to create a clean envi ```bash # Create and activate a new conda environment -conda create -n fastvideo python=3.10 +conda create -n fastvideo python=3.12 conda activate fastvideo # Install FastVideo @@ -72,72 +67,10 @@ python example.py The generated video will be saved in the current directory under `my_videos/`. -## Customizing Generation - -You can customize various parameters when generating videos using the `SamplingParam` class: - -```python -from fastvideo import VideoGenerator, SamplingParam - -def main(): - # Create the generator - generator = VideoGenerator.from_pretrained( - "Wan-AI/Wan2.1-T2V-1.3B-Diffusers", - num_gpus=1, - ) - - # Create and customize sampling parameters - sampling_param = SamplingParam.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers") - - # How many frames to generate - sampling_param.num_frames = 45 - - # Video resolution (width, height) - sampling_param.width = 1024 - sampling_param.height = 576 - - # How many steps we denoise the video (higher = better quality, slower generation) - sampling_param.num_inference_steps = 30 - - # How strongly the video conforms to the prompt (higher = more faithful to prompt) - sampling_param.guidance_scale = 7.5 - - # Random seed for reproducibility - sampling_param.seed = 42 # Optional, leave unset for random results - - # Generate video with custom parameters - prompt = "A beautiful sunset over a calm ocean, with gentle waves." - video = generator.generate_video( - prompt, - sampling_param=sampling_param, - output_path="my_videos/", # Controls where videos are saved - return_frames=True, # Also return frames from this call (defaults to False) - save_video=True - ) - - # If return_frames=True, video contains the generated frames as a NumPy array - print(f"Generated {len(video)} frames") - -if __name__ == '__main__': - main() -``` - ## Available Models Please see the [support matrix](#support-matrix) for the list of supported models and their available optimizations. -## Multi-GPU Setup - -FastVideo automatically distributes the generation process when multiple GPUs are specified: - -```python -# Will use 4 GPUs in parallel for faster generation -generator = VideoGenerator.from_pretrained( - "Wan-AI/Wan2.1-T2V-1.3B-Diffusers", - num_gpus=4, -) -``` - ## Image-to-Video Generation You can generate a video starting from an initial image: @@ -159,53 +92,6 @@ prompt = "A photograph coming to life with gentle movement" video = generator.generate_video(prompt, sampling_param=sampling_param) ``` -## Output Formats and Options - -FastVideo provides several options for saving and manipulating the generated videos: - -```python -from fastvideo import VideoGenerator, SamplingParam, VideoFormat - -# Create the generator -generator = VideoGenerator.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers") - -# Generate with custom output options -video = generator.generate_video( - "Drone footage of a tropical rainforest", - output_path="videos/rainforest/", # Custom save location - filename="rainforest_flyover", # Custom filename (no extension needed) - format=VideoFormat.MP4, # Output format: MP4, GIF, or WEBM - fps=30, # Frames per second for the output video - loop=False, # Whether to loop the video (for GIF) - save_frames=True, # Also save individual frames as images -) -``` - -## Performance Optimization - -Optimize FastVideo for your hardware: - -```python -from fastvideo import VideoGenerator, PipelineConfig - -# Create an optimized configuration -config = PipelineConfig.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers") - -# Memory optimization -config.enable_vae_slicing = True # Process VAE in slices to reduce memory -config.enable_model_cpu_offload = True # Offload models to CPU when not in use - -# Speed optimization -config.vae_config.precision = "fp16" # Use half precision for VAE -config.enable_xformers = True # Use xformers for attention computation - -# Create generator with optimized config -generator = VideoGenerator.from_pretrained( - "Wan-AI/Wan2.1-T2V-1.3B-Diffusers", - pipeline_config=config, -) -``` - ## Troubleshooting Common issues and their solutions: @@ -232,30 +118,7 @@ If the generated video doesn't match your prompt: ## Advanced Configuration -For advanced customization, use the `PipelineConfig` class: - -```python -from fastvideo import VideoGenerator, PipelineConfig - -# Load the default configuration for a model -config = PipelineConfig.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers") - -# Modify configuration settings -config.vae_config.scale_factor = 16 -config.vae_config.precision = "fp16" -config.scheduler_config.beta_schedule = "linear" -config.attention_config.attention_type = "xformers" - -# Create generator with custom config -generator = VideoGenerator.from_pretrained( - "Wan-AI/Wan2.1-T2V-1.3B-Diffusers", - pipeline_config=config, -) - -# Generate video -prompt = "A futuristic cityscape with flying cars and neon signs." -video = generator.generate_video(prompt) -``` +## Optimizations ## Next Steps diff --git a/docs/source/inference/optimizations.md b/docs/source/inference/optimizations.md new file mode 100644 index 000000000..402add3fc --- /dev/null +++ b/docs/source/inference/optimizations.md @@ -0,0 +1,147 @@ +# Optimizations + +This page describes the various options for speeding up generation times in FastVideo. + +## Table of Contents +- Optimized Attention Backends + - [Flash Attention](#optimizations-flash) + - [Sliding Tile Attention](#optimizations-sta) + - [Sage Attention](#optimizations-sage) + +- Caching Techniques + - [TeaCache](#optimizations-teacache) + +(optimizations-backends)= +## Attention Backends + +### Available Backends +- Torch SDPA: `FASTVIDEO_ATTENTION_BACKEND=TORCH_SDPA` +- Flash Attention 2 and 3: `FASTVIDEO_ATTENTION_BACKEND=FLASH_ATTN` +- Sliding Tile Attention: `FASTVIDEO_ATTENTION_BACKEND=SLIDING_TILE_ATTN` +- Sage Attention: `FASTVIDEO_ATTENTION_BACKEND=SAGE_ATTN` + +### Configuring Backends + +There are two ways to configure the attention backend in FastVideo. + +#### 1. In Python +In python, set the `FASTVIDEO_ATTENTION_BACKEND` environment variable before instantiating `VideoGenerator` like this: + +```python +os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "SLIDING_TILE_ATTN" +``` + +#### 2. In CLI +You can also set the environment variable on the command line: + +```bash +FASTVIDEO_ATTENTION_BACKEND=SAGE_ATTN python example.py +``` + +(optimizations-flash)= +### Flash Attention + +**`FLASH_ATTN`** + +We recommend always installing [Flash Attention 2](https://github.com/Dao-AILab/flash-attention): + +```bash +pip install flash-attn==2.7.4.post1 --no-build-isolation +``` + +And if using a Hopper+ GPU (ie H100), installing [Flash Attention 3](https://github.com/Dao-AILab/flash-attention?tab=readme-ov-file#flashattention-3-beta-release) by compiling it from source (takes about 10 minutes for me): + +```bash +git clone https://github.com/Dao-AILab/flash-attention.git && cd flash-attention + +cd hopper +pip install ninja +python setup.py install +``` + +:::{note} +FastVideo will automatically detect and use `FA3` if it is installed when using `FLASH_ATTN` backend. +::: + +(optimizations-sta)= +### Sliding Tile Attention +**`SLIDING_TILE_ATTN`** + +```bash +pip install st_attn==0.0.4 +``` + +Please see [this page](#sta-installation) for more installation instructions. + +(optimizations-sage)= +### Sage Attention +**`SAGE_ATTN`** + +To use [SageAttention](https://github.com/thu-ml/SageAttention) 2.1.1, please compile from source: + +```bash +git clone https://github.com/thu-ml/SageAttention.git +cd sageattention +python setup.py install # or pip install -e . +``` + +(optimizations-teacache)= +## Teacache +TeaCache is an optimization technique supported in FastVideo that can significantly speed up video generation by skipping redundant calculations across diffusion steps. This guide explains how to enable and configure TeaCache for optimal performance in FastVideo. + +### What is TeaCache? + +See the official [TeaCache](https://github.com/ali-vilab/TeaCache) repo and their paper for more details. + +### How to Enable TeaCache + +Enabling TeaCache is straightforward - simply add the `enable_teacache=True` parameter to your `generate_video()` call: + +```python +# ... previous code +generator.generate_video( + prompt="Your prompt here", + sampling_param=params, + enable_teacache=True +) +# more code ... +``` + +### Complete Example + +At the bottom is a complete example of using TeaCache for faster video generation. You can run it using the following command: + +```bash +python examples/inference/optimizations/teacache_example.py +``` + +### Advanced Configuration + +While TeaCache works well with default settings, you can fine-tune its behavior by adjusting the threshold value: + +1. Lower threshold values (e.g., 0.1) will result in more skipped calculations and faster generation with slightly more potential for quality degradation +2. Higher threshold values (e.g., 0.15-0.23) will skip fewer calculations but maintain quality closer to the original + +Note that the optimal threshold depends on your specific model and content. + +## Benchmarking different optimizations + +To benchmark the performance improvement, try generating the same video with and without TeaCache enabled and compare the generation times: + +```python +# Without TeaCache +start_time = time.perf_counter() +generator.generate_video(prompt="Your prompt", enable_teacache=False) +standard_time = time.perf_counter() - start_time + +# With TeaCache +start_time = time.perf_counter() +generator.generate_video(prompt="Your prompt", enable_teacache=True) +teacache_time = time.perf_counter() - start_time + +print(f"Standard generation: {standard_time:.2f} seconds") +print(f"TeaCache generation: {teacache_time:.2f} seconds") +print(f"Speedup: {standard_time/teacache_time:.2f}x") +``` + +Note: If you want to benchmark different attention backends, you'll need to reinstantiate `VideoGenerator`. diff --git a/docs/source/inference/support_matrix.md b/docs/source/inference/support_matrix.md index c14798c95..6624b12e1 100644 --- a/docs/source/inference/support_matrix.md +++ b/docs/source/inference/support_matrix.md @@ -1,15 +1,87 @@ (support-matrix)= -# Support Matrix +# Compatibility Matrix +The table below shows every supported model and optimizations supported for them. -| Model Name | HuggingFace Model ID | Resolutions | TeaCache| Sliding Tile Attn | Sage Attn | -|------------|----------------------|-------------|-------------|--------------------------|--| -| HunyuanVideo | hunyuanvideo-community/HunyuanVideo | | | ✅ | ✅ | -| FastHunyuan | FastVideo/FastHunyuan-diffusers | | | ✅| ✅| -| Wan T2V 1.4B | Wan-AI/Wan2.1-T2V-1.3B-Diffusers | 480P | ✅ | ✅ | ✅ | -| Wan T2V 14B | Wan-AI/Wan2.1-T2V-1.3B-Diffusers | 480P, 720P | ✅| ✅| ✅| -| Wan I2V 480P | Wan-AI/Wan2.1-I2V-14B-480P-Diffusers | 480P | ✅| ✅| ✅| -| Wan T2V 720P | Wan-AI/Wan2.1-T2V-14B-Diffusers | 720P | ✅| ✅| ✅| -| StepVideo T2V | Coming soon! | 768px768px204f, 544px992px204f, 544px992px136f | | | | +The symbols used have the following meanings: + +- ✅ = Full compatibility +- ❌ = No compatibility + +## Models x Optimization +The `HuggingFace Model ID` can be directly pass to `from_pretrained()` methods and FastVideo will use the optimal default parameters when initializing and generating videos. + +:::{raw} html + +::: + +:::{list-table} +:header-rows: 1 +:stub-columns: 3 +:widths: auto +:class: vertical-table-header + +- * Model Name + * HuggingFace Model ID + * Resolutions + * TeaCache + * Sliding Tile Attn + * Sage Attn +- * HunyuanVideo + * `hunyuanvideo-community/HunyuanVideo` + * 720px1280p
544px960p + * ❌ + * ✅ + * ✅ +- * FastHunyuan + * `FastVideo/FastHunyuan-diffusers` + * 720px1280p
544px960p + * ❌ + * ✅ + * ✅ +- * Wan T2V 1.4B + * `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` + * 480P + * ✅ + * ✅* + * ✅ +- * Wan T2V 14B + * `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` + * 480P, 720P + * ✅ + * ✅* + * ✅ +- * Wan I2V 480P + * `Wan-AI/Wan2.1-I2V-14B-480P-Diffusers` + * 480P + * ✅ + * ✅* + * ✅ +- * Wan T2V 720P + * `Wan-AI/Wan2.1-T2V-14B-Diffusers` + * 720P + * ✅ + * ✅* + * ✅ +- * StepVideo T2V + * Coming soon! + * 768px768px204f
544px992px204f
544px992px136f + * + * + * +::: + +**Note**: there are some known quality issues with Wan2.1 + Sliding Tile Attn. We are working on fixing this issue. ## Special requirements diff --git a/examples/inference/basic/README.md b/examples/inference/basic/README.md index 68047d232..9257dbd57 100644 --- a/examples/inference/basic/README.md +++ b/examples/inference/basic/README.md @@ -15,7 +15,7 @@ The first script in this example shows the most basic usage of FastVideo. If you # if you have not cloned the directory: git clone https://github.com/hao-ai-lab/FastVideo.git && cd FastVideo -python fastvideo/v1/examples/inference/basic/basic.py +python examples/inference/basic/basic.py ``` ## Basic Walkthrough diff --git a/examples/inference/gradio/README.md b/examples/inference/gradio/README.md index 6add451d2..c7ff9f5fd 100644 --- a/examples/inference/gradio/README.md +++ b/examples/inference/gradio/README.md @@ -13,26 +13,12 @@ The demo uses the FastVideo framework to generate videos based on text prompts. --- -## Requirements - -- Linux-based OS -- Python 3.10-3.12 -- NVIDIA GPU -- Cuda 12.4 -- FastVideo - -## Installation - -```bash -pip install fastvideo -``` - ## Usage Run the demo with: ```bash -python fastvideo/v1/examples/inference/gradio/gradio_demo.py +python examples/inference/gradio/gradio_demo.py ``` This will start a web server at `http://0.0.0.0:7860` where you can access the interface. diff --git a/examples/inference/optimizations/README.md b/examples/inference/optimizations/README.md index 7b5090ebc..66ea69318 100644 --- a/examples/inference/optimizations/README.md +++ b/examples/inference/optimizations/README.md @@ -1,132 +1,9 @@ -# Optimizing Generation +# Optimization Examples -This page describes the various options for speeding up generation times in FastVideo. - -## Table of Contents -- Optimized Attention Backends - - [Flash Attention](#optimizations-flash) - - [Sliding Tile Attention](#optimizations-sta) - - [Sage Attention](#optimizations-sage) - -- Caching Techniques - - [TeaCache](#optimizations-teacache) - - -(optimizations-backends)= -## Attention Backends - -### Available Backends -- Torch SDPA: `FASTVIDEO_ATTENTION_BACKEND=TORCH_SDPA` -- Flash Attention 2 and 3: `FASTVIDEO_ATTENTION_BACKEND=FLASH_ATTN` -- Sliding Tile Attention: `FASTVIDEO_ATTENTION_BACKEND=SLIDING_TILE_ATTN` -- Sage Attention: `FASTVIDEO_ATTENTION_BACKEND=SAGE_ATTN` - -### How to configure - -`attention_example.py` shows how to set `FASTVIDEO_ATTENTION_BACKEND` env var to change attention backends. To run this example: ```bash python examples/inference/optimizations/attention_example.py ``` -In python, set the `FASTVIDEO_ATTENTION_BACKEND` before instantiating `VideoGenerator` like this: - -```python -os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "SLIDING_TILE_ATTN" -``` - -You can also set the env var when running any of the other example like this: -```bash -FASTVIDEO_ATTENTION_BACKEND=SAGE_ATTN python example.py -``` - -(optimizations-flash)= -### Flash Attention - -**`FLASH_ATTN`** - -We recommend always installing [Flash Attention 2](https://github.com/Dao-AILab/flash-attention): -```bash -``` - -And if using a Hopper+ GPU (ie H100), installing [Flash Attention 3](https://github.com/Dao-AILab/flash-attention?tab=readme-ov-file#flashattention-3-beta-release) by compiling it from source (takes about 10 minutes for me): - -```bash -git clone https://github.com/Dao-AILab/flash-attention.git && cd flash-attention - -cd hopper -pip install ninja -python setup.py install -``` - -FastVideo will automatically detect and use FA3 if it is installed when using or falling back to `FLASH_ATTN` backend. - -(optimizations-sta)= -### Sliding Tile Attention - - -(optimizations-sage)= -### Sage Attention - - - -(optimizations-teacache)= -## Teacache -TeaCache is an optimization technique supported in FastVideo that can significantly speed up video generation by skipping redundant calculations across diffusion steps. This guide explains how to enable and configure TeaCache for optimal performance in FastVideo. - -### What is TeaCache? - -See the official [TeaCache](https://github.com/ali-vilab/TeaCache) repo and their paper for more details. - - -### How to Enable TeaCache - -Enabling TeaCache is straightforward - simply add the `enable_teacache=True` parameter to your `generate_video()` call: - -```python -# ... previous code -generator.generate_video( - prompt="Your prompt here", - sampling_param=params, - enable_teacache=True -) -# more code ... -``` - -### Complete Example - -At the bottom is a complete example of using TeaCache for faster video generation. You can run it using the following command: - ```bash python examples/inference/optimizations/teacache_example.py ``` - -### Advanced Configuration - -While TeaCache works well with default settings, you can fine-tune its behavior by adjusting the threshold value: - -1. Lower threshold values (e.g., 0.1) will result in more skipped calculations and faster generation with slightly more potential for quality degradation -2. Higher threshold values (e.g., 0.15-0.23) will skip fewer calculations but maintain quality closer to the original - -Note that the optimal threshold depends on your specific model and content. - -## Benchmarking different optimizations - -To benchmark the performance improvement, try generating the same video with and without TeaCache enabled and compare the generation times: - -```python -# Without TeaCache -start_time = time.perf_counter() -generator.generate_video(prompt="Your prompt", enable_teacache=False) -standard_time = time.perf_counter() - start_time - -# With TeaCache -start_time = time.perf_counter() -generator.generate_video(prompt="Your prompt", enable_teacache=True) -teacache_time = time.perf_counter() - start_time - -print(f"Standard generation: {standard_time:.2f} seconds") -print(f"TeaCache generation: {teacache_time:.2f} seconds") -print(f"Speedup: {standard_time/teacache_time:.2f}x") -``` - -Note: If you want to benchmark different attention backends, you'll need to reinstantiate `VideoGenerator`.