diff --git a/README.md b/README.md index 6a1ae9fa0..4607b5de8 100644 --- a/README.md +++ b/README.md @@ -19,11 +19,11 @@ With FastVideo's optimizations, you can achieve more than 3x inference improveme FastVideo has the following features: - State-of-the-art performance optimizations for inference - - [Sliding Tile Attention](https://hao-ai-lab.github.io/blogs/sta/). - - TeaCache - - Sage Attention + - [Sliding Tile Attention](https://arxiv.org/pdf/2502.04507) + - [TeaCache](https://arxiv.org/pdf/2411.19108) + - [Sage Attention](https://arxiv.org/abs/2410.02367) - Cutting edge models - - Wan2.1 T2V, I2v + - Wan2.1 T2V, I2V - HunyuanVideo - FastHunyuan: consistency distilled video diffusion models for 8x inference speedup. - StepVideo T2V diff --git a/assets/perf.png b/assets/perf.png index 7ee6cf6de..8c2ace189 100644 Binary files a/assets/perf.png and b/assets/perf.png differ diff --git a/docs/source/_static/images/perf.png b/docs/source/_static/images/perf.png index 7ee6cf6de..8c2ace189 100644 Binary files a/docs/source/_static/images/perf.png and b/docs/source/_static/images/perf.png differ diff --git a/docs/source/index.md b/docs/source/index.md index c2b4ad2f5..d992e5d50 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -32,11 +32,11 @@ With FastVideo's optimizations, you can achieve more than 3x inference improveme FastVideo has the following features: - State-of-the-art performance optimizations for inference - - [Sliding Tile Attention](https://hao-ai-lab.github.io/blogs/sta/). - - TeaCache - - Sage Attention + - [Sliding Tile Attention](https://arxiv.org/pdf/2502.04507) + - [TeaCache](https://arxiv.org/pdf/2411.19108) + - [Sage Attention](https://arxiv.org/abs/2410.02367) - Cutting edge models - - Wan2.1 T2V, I2v + - Wan2.1 T2V, I2V - HunyuanVideo - FastHunyuan: consistency distilled video diffusion models for 8x inference speedup. - StepVideo T2V diff --git a/docs/source/inference/configuration.md b/docs/source/inference/configuration.md new file mode 100644 index 000000000..581fb151e --- /dev/null +++ b/docs/source/inference/configuration.md @@ -0,0 +1,77 @@ +(inference-configuration)= +# Configuration + +## Multi-GPU Setup + +FastVideo automatically distributes the generation process when multiple GPUs are specified: + +```python +# Will use 4 GPUs in parallel for faster generation +generator = VideoGenerator.from_pretrained( + "Wan-AI/Wan2.1-T2V-1.3B-Diffusers", + num_gpus=4, +) +``` + +## Customizing Generation + +- `PipelineConfig`: Initialization time parameters +- `SamplingParam`: Generation time parameters + +You can customize various parameters when generating videos using the `PipelineConfig` and `SamplingParam` class: + +```python +from fastvideo import VideoGenerator, SamplingParam, PipelineConfig + +def main(): + model_name = "Wan-AI/Wan2.1-T2V-1.3B-Diffusers" + config = PipelineConfig.from_pretrained(model_name) + config.vae_precision = "fp16" + config.use_cpu_offload = True + + # Create the generator + generator = VideoGenerator.from_pretrained( + model_name, + num_gpus=1, + pipeline_config=config + ) + + # Create and customize sampling parameters + sampling_param = SamplingParam.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers") + + # How many frames to generate + sampling_param.num_frames = 45 + + # Video resolution (width, height) + sampling_param.width = 1024 + sampling_param.height = 576 + + # How many steps we denoise the video (higher = better quality, slower generation) + sampling_param.num_inference_steps = 30 + + # How strongly the video conforms to the prompt (higher = more faithful to prompt) + sampling_param.guidance_scale = 7.5 + + # Random seed for reproducibility + sampling_param.seed = 42 # Optional, leave unset for random results + + # Generate video with custom parameters + prompt = "A beautiful sunset over a calm ocean, with gentle waves." + video = generator.generate_video( + prompt, + sampling_param=sampling_param, + output_path="my_videos/", # Controls where videos are saved + return_frames=True, # Also return frames from this call (defaults to False) + save_video=True + ) + + # If return_frames=True, video contains the generated frames as a NumPy array + print(f"Generated {len(video)} frames") + +if __name__ == '__main__': + main() +``` + +## Performance Optimization + +For configuring optimizations, please see our [optimizations guide](#inference-optimizations) diff --git a/docs/source/inference/inference_quick_start.md b/docs/source/inference/inference_quick_start.md index f9d800bba..ede36dc72 100644 --- a/docs/source/inference/inference_quick_start.md +++ b/docs/source/inference/inference_quick_start.md @@ -2,19 +2,11 @@ This page contains step-by-step instructions to get you quickly started with video generation using FastVideo. -## Table of Contents -- [Generating Your First Video](#generating-your-first-video) -- [Customizing Generation](#customizing-generation) -- [Available Models](#available-models) -- [Image-to-Video Generation](#image-to-video-generation) -- [Troubleshooting](#troubleshooting) -- [Advanced Configuration](#advanced-configuration) -- [Next Steps](#next-steps) - -## Software Requirements +## Requirements - **OS**: Linux (Tested on Ubuntu 22.04+) - **Python**: 3.10-3.12 - **CUDA**: 12.4 +- **GPU**: At least one NVIDIA GPU ## Installation @@ -78,18 +70,25 @@ You can generate a video starting from an initial image: ```python from fastvideo import VideoGenerator, SamplingParam -# Create the generator -generator = VideoGenerator.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers") +def main(): + # Create the generator + model_name = "Wan-AI/Wan2.1-I2V-14B-480P-Diffusers" + generator = VideoGenerator.from_pretrained(model_name, num_gpus=1) -# Set up parameters with an initial image -sampling_param = SamplingParam.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers") -sampling_param.image_path = "path/to/your/image.jpg" -sampling_param.num_frames = 24 -sampling_param.image_strength = 0.8 # How much to preserve the original image (0-1) + # Set up parameters with an initial image + sampling_param = SamplingParam.from_pretrained(model_name) + sampling_param.image_path = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg" + sampling_param.num_frames = 107 + sampling_param.image_strength = 0.8 # How much to preserve the original image (0-1) -# Generate video based on the image -prompt = "A photograph coming to life with gentle movement" -video = generator.generate_video(prompt, sampling_param=sampling_param) + # Generate video based on the image + prompt = "A photograph coming to life with gentle movement" + generator.generate_video(prompt, sampling_param=sampling_param, + output_path="my_videos/", + save_video=True) + +if __name__ == '__main__': + main() ``` ## Troubleshooting @@ -100,7 +99,7 @@ Common issues and their solutions: If you encounter CUDA out of memory errors: - Reduce `num_frames` or video resolution - Enable memory optimization with `enable_model_cpu_offload` -- Try a smaller model or use quantized versions +- Try a smaller model or use distilled versions - Use `num_gpus` > 1 if multiple GPUs are available ### Slow Generation @@ -116,14 +115,10 @@ If the generated video doesn't match your prompt: - Experiment with different random seeds - Try a different model -## Advanced Configuration - -## Optimizations - ## Next Steps -- Explore the [API Reference](../api/index.md) for detailed documentation -- Learn about [Advanced Inference Options](../inference/overview_back.md) -- See [Examples](../examples/index.md) for more usage scenarios -- Check out the [Model Training](../training/overview.md) guide to fine-tune models -- Join our [Community Discord](https://discord.gg/fastvideo) for support and sharing +- Learn about [Advanced Inference Configurations](#inference-configuration) +- Learn about using [Optimizations](#inference-optimizations) +- See [Examples](../examples/examples_inference_index.md) for more usage scenarios +- Join our [Community Discord](https://discord.gg/JA7cksDz86). +- Join our [Community Slack](https://join.slack.com/t/fastvideo/shared_invite/zt-2zf6ru791-sRwI9lPIUJQq1mIeB_yjJg). diff --git a/docs/source/inference/optimizations.md b/docs/source/inference/optimizations.md index 402add3fc..7cff763d1 100644 --- a/docs/source/inference/optimizations.md +++ b/docs/source/inference/optimizations.md @@ -1,3 +1,4 @@ +(inference-optimizations)= # Optimizations This page describes the various options for speeding up generation times in FastVideo.