hao-ai-lab-FastVideo/docs/EVAL_README.md

Evaluate the Generated Video_Quality

You can easily calculate the following video quality metrics, which supports the batch-wise process.

• CLIP-SCORE: It uses the pretrained CLIP model to measure the cosine similarity between two modalities.
• FVD: Frechét Video Distance
• SSIM: structural similarity index measure
• LPIPS: learned perceptual image patch similarity
• PSNR: peak-signal-to-noise ratio

Requrirement

Environemnt

• install Pytorch (torch>=1.7.1)
• install CLIP

      pip install git+https://github.com/openai/CLIP.git
  

• install clip-cose from PyPi

      pip install clip-score
  

• Other package

      pip install lpips
      pip install scipy (scipy==1.7.3/1.9.3, if you use 1.11.3, **you will calculate a WRONG FVD VALUE!!!**)
      pip install numpy
      pip install pillow
      pip install torchvision>=0.8.2
      pip install ftfy
      pip install regex
      pip install tqdm
  

Pretrain model

• FVD Before you cacluate FVD, you should first download the FVD pre-trained model. You can manually download any of the following and put it into FVD folder.
  • i3d_torchscript.pt from here
  • i3d_pretrained_400.pt from here

Other Notices

1. Make sure the pixel value of videos should be in [0, 1].
2. We average SSIM when images have 3 channels, ssim is the only metric extremely sensitive to gray being compared to b/w.
3. Because the i3d model downsamples in the time dimension, frames_num should > 10 when calculating FVD, so FVD calculation begins from 10-th frame, like upper example.
4. For grayscale videos, we multiply to 3 channels
5. data input specifications for clip_score

• Image Files:All images should be stored in a single directory. The image files can be in either .png or .jpg format.

• Text Files: All text data should be contained in plain text files in a separate directory. These text files should have the extension .txt.

Note: The number of files in the image directory should be exactly equal to the number of files in the text directory. Additionally, the files in the image directory and text directory should be paired by file name. For instance, if there is a cat.png in the image directory, there should be a corresponding cat.txt in the text directory.

Directory Structure Example:

  ├── path/to/image
  │   ├── cat.png
  │   ├── dog.png
  │   └── bird.jpg
  └── path/to/text
      ├── cat.txt
      ├── dog.txt
      └── bird.txt

6. data input specifications for fvd, psnr, ssim, lpips

Directory Structure Example:

  ├── path/to/generated_image
  │   ├── cat.mp4
  │   ├── dog.mp4
  │   └── bird.mp4
  └── path/to/real_image
      ├── cat.mp4
      ├── dog.mp4
      └── bird.mp4

Usage

# clip_score cross modality
python eval_clip_score.py \
    --real_path path/to/image \
    --generated_path path/to/text \
    --batch-size 50 \
    --device "cuda"

# clip_score within the same modality
python eval_clip_score.py \
    --real_path path/to/textA \
    --generated_path path/to/textB \
    --real_flag txt \
    --generated_flag txt \
    --batch-size 50 \
    --device "cuda"

python eval_clip_score.py \
    --real_path path/to/imageA \
    --generated_path path/to/imageB \
    --real_flag img \
    --generated_flag img \
    --batch-size 50 \
    --device "cuda"


# fvd, psnr, ssim, lpips, you need to set the frame_num, resolution etc...

python eval_common_metric.py \
    --real_video_dir path/to/imageA\
    --generated_video_dir path/to/imageB \
    --batch_size 10 \
    --crop_size 64 \
    --num_frames 20 \
    --device 'cuda' \
    --metric 'fvd' \
    --fvd_method 'styleganv'

python eval_common_metric.py \
    --real_video_dir path/to/imageA\
    --generated_video_dir path/to/imageB \
    --batch_size 10 \
    --num_frames 20 \
    --crop_size 64 \
    --device 'cuda' \
    --metric 'psnr'


python eval_common_metric.py \
    --real_video_dir path/to/imageA\
    --generated_video_dir path/to/imageB \
    --batch_size 10 \
    --num_frames 20 \
    --crop_size 64 \
    --device 'cuda' \
    --metric 'ssim'

python eval_common_metric.py \
    --real_video_dir path/to/imageA\
    --generated_video_dir path/to/imageB \
    --batch_size 10 \
    --num_frames 20 \
    --crop_size 64 \
    --device 'cuda' \
    --metric 'lpips'

Acknowledgement

The evaluation codebase refers to clip-score and common_metrics.