nodes/hao-ai-lab-FastVideo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
4b0970cbbf90dcf38743370d85beb2f3d7a9c36f
hao-ai-lab-FastVideo/docs/source/inference/inference_quick_start.md
T

8.2 KiB
Raw Blame History

Inference Quick Start

This page contains step-by-step instructions to get you quickly started with video generation using FastVideo.

Table of Contents

Software Requirements

  • OS: Linux (Tested on Ubuntu 22.04+)
  • Python: 3.10-3.12
  • CUDA: 12.4

Installation

We recommend using an environment manager such as Conda to create a clean environment:

# Create and activate a new conda environment
conda create -n fastvideo python=3.10
conda activate fastvideo

# Install FastVideo
pip install fastvideo

For advanced installation options, see the Installation Guide.

Generating Your First Video

Here's a minimal example to generate a video using the default settings. Create a file called example.py with the following code:

from fastvideo import VideoGenerator

def main():
    # Create a video generator with a pre-trained model
    generator = VideoGenerator.from_pretrained(
        "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
        num_gpus=1,  # Adjust based on your hardware
    )

    # Define a prompt for your video
    prompt = "A curious raccoon peers through a vibrant field of yellow sunflowers, its eyes wide with interest."

    # Generate the video
    video = generator.generate_video(
        prompt,
        return_frames=True,  # Also return frames from this call (defaults to False)
        output_path="my_videos/",  # Controls where videos are saved
        save_video=True
    )

if __name__ == '__main__':
    main()

Run the script with:

python example.py

The generated video will be saved in the current directory under my_videos/.

Customizing Generation

You can customize various parameters when generating videos using the SamplingParam class:

from fastvideo import VideoGenerator, SamplingParam

def main():
    # Create the generator
    generator = VideoGenerator.from_pretrained(
        "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
        num_gpus=1,
    )

    # Create and customize sampling parameters
    sampling_param = SamplingParam.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers")
    
    # How many frames to generate
    sampling_param.num_frames = 45
    
    # Video resolution (width, height)
    sampling_param.width = 1024
    sampling_param.height = 576
    
    # How many steps we denoise the video (higher = better quality, slower generation)
    sampling_param.num_inference_steps = 30
    
    # How strongly the video conforms to the prompt (higher = more faithful to prompt)
    sampling_param.guidance_scale = 7.5
    
    # Random seed for reproducibility
    sampling_param.seed = 42  # Optional, leave unset for random results

    # Generate video with custom parameters
    prompt = "A beautiful sunset over a calm ocean, with gentle waves."
    video = generator.generate_video(
        prompt, 
        sampling_param=sampling_param, 
        output_path="my_videos/",  # Controls where videos are saved
        return_frames=True,  # Also return frames from this call (defaults to False)
        save_video=True
    )

    # If return_frames=True, video contains the generated frames as a NumPy array
    print(f"Generated {len(video)} frames")

if __name__ == '__main__':
    main()

Available Models

Please see the support matrix for the list of supported models and their available optimizations.

Multi-GPU Setup

FastVideo automatically distributes the generation process when multiple GPUs are specified:

# Will use 4 GPUs in parallel for faster generation
generator = VideoGenerator.from_pretrained(
    "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
    num_gpus=4,
)

Image-to-Video Generation

You can generate a video starting from an initial image:

from fastvideo import VideoGenerator, SamplingParam

# Create the generator
generator = VideoGenerator.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers")

# Set up parameters with an initial image
sampling_param = SamplingParam.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers")
sampling_param.image_path = "path/to/your/image.jpg"
sampling_param.num_frames = 24
sampling_param.image_strength = 0.8  # How much to preserve the original image (0-1)

# Generate video based on the image
prompt = "A photograph coming to life with gentle movement"
video = generator.generate_video(prompt, sampling_param=sampling_param)

Output Formats and Options

FastVideo provides several options for saving and manipulating the generated videos:

from fastvideo import VideoGenerator, SamplingParam, VideoFormat

# Create the generator
generator = VideoGenerator.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers")

# Generate with custom output options
video = generator.generate_video(
    "Drone footage of a tropical rainforest",
    output_path="videos/rainforest/",  # Custom save location
    filename="rainforest_flyover",  # Custom filename (no extension needed)
    format=VideoFormat.MP4,  # Output format: MP4, GIF, or WEBM
    fps=30,  # Frames per second for the output video
    loop=False,  # Whether to loop the video (for GIF)
    save_frames=True,  # Also save individual frames as images
)

Performance Optimization

Optimize FastVideo for your hardware:

from fastvideo import VideoGenerator, PipelineConfig

# Create an optimized configuration
config = PipelineConfig.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers")

# Memory optimization
config.enable_vae_slicing = True  # Process VAE in slices to reduce memory
config.enable_model_cpu_offload = True  # Offload models to CPU when not in use

# Speed optimization
config.vae_config.precision = "fp16"  # Use half precision for VAE
config.enable_xformers = True  # Use xformers for attention computation

# Create generator with optimized config
generator = VideoGenerator.from_pretrained(
    "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
    pipeline_config=config,
)

Troubleshooting

Common issues and their solutions:

Out of Memory Errors

If you encounter CUDA out of memory errors:

  • Reduce num_frames or video resolution
  • Enable memory optimization with enable_model_cpu_offload
  • Try a smaller model or use quantized versions
  • Use num_gpus > 1 if multiple GPUs are available

Slow Generation

To speed up generation:

  • Reduce num_inference_steps (20-30 is usually sufficient)
  • Use half precision (fp16) for the VAE
  • Use multiple GPUs if available

Unexpected Results

If the generated video doesn't match your prompt:

  • Try increasing guidance_scale (7.0-9.0 works well)
  • Make your prompt more detailed and specific
  • Experiment with different random seeds
  • Try a different model

Advanced Configuration

For advanced customization, use the PipelineConfig class:

from fastvideo import VideoGenerator, PipelineConfig

# Load the default configuration for a model
config = PipelineConfig.from_pretrained("Wan-AI/Wan2.1-T2V-1.3B-Diffusers")

# Modify configuration settings
config.vae_config.scale_factor = 16
config.vae_config.precision = "fp16"
config.scheduler_config.beta_schedule = "linear"
config.attention_config.attention_type = "xformers"

# Create generator with custom config
generator = VideoGenerator.from_pretrained(
    "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
    pipeline_config=config,
)

# Generate video
prompt = "A futuristic cityscape with flying cars and neon signs."
video = generator.generate_video(prompt)

Next Steps

Reference in New Issue View Git Blame Copy Permalink