From 2dbd3bd700147fc2eebdc0f8648a3ef3e85dab51 Mon Sep 17 00:00:00 2001 From: lb203 <62638829+LinB203@users.noreply.github.com> Date: Wed, 27 Mar 2024 23:10:14 +0800 Subject: [PATCH] Update README.md Former-commit-id: 90d6d298eaee6255098b342b31444e7a347fc50e --- README.md | 41 ++++++++--------------------------------- 1 file changed, 8 insertions(+), 33 deletions(-) diff --git a/README.md b/README.md index 21306fece..be32657bd 100644 --- a/README.md +++ b/README.md @@ -197,47 +197,22 @@ Refer to [Data.md](docs/Data.md) ### Evaluation Refer to the document [EVAL.md](docs/EVAL.md). -### Video-VQVAE (VideoGPT) - -#### Training - -To train VQVAE, run the script: - -``` -scripts/videogpt/train_videogpt.sh -``` - -You can modify the training parameters within the script. For training parameters, please refer to [transformers.TrainingArguments](https://huggingface.co/docs/transformers/v4.38.2/en/main_classes/trainer#transformers.TrainingArguments). Other parameters are explained as follows: - -##### VQ-VAE Specific Settings - -* `--embedding_dim`: number of dimensions for codebooks embeddings -* `--n_codes 2048`: number of codes in the codebook -* `--n_hiddens 240`: number of hidden features in the residual blocks -* `--n_res_layers 4`: number of residual blocks -* `--downsample "4,4,4"`: T H W downsampling stride of the encoder - -##### Dataset Settings -* `--data_path `: path to an `hdf5` file or a folder containing `train` and `test` folders with subdirectories of videos -* `--resolution 128`: spatial resolution to train on -* `--sequence_length 16`: temporal resolution, or video clip length +### Causal Video VAE #### Reconstructing ```Python -python examples/rec_video.py --video-path "assets/origin_video_0.mp4" --rec-path "rec_video_0.mp4" --num-frames 500 --sample-rate 1 -``` -```Python -python examples/rec_video.py --video-path "assets/origin_video_1.mp4" --rec-path "rec_video_1.mp4" --resolution 196 --num-frames 600 --sample-rate 1 +python examples/rec_video_vae.py --rec-path test_video.mp4 --video-path video.mp4 --resolution 512 --num-frames 1440 --sample-rate 1 --sample-fps 24 - +-device cuda --ckpt ``` -We present four reconstructed videos in this demonstration, arranged from left to right as follows: +We present our reconstructed video in this demonstration as follows: -| **3s 596x336** | **10s 256x256** | **18s 196x196** | **24s 168x96** | -| --- | --- | --- | --- | -| | | | | +https://github.com/qqingzheng/Open-Sora-Plan/assets/88202804/0dbf8f8f-efba-414e-ad42-1f839839f9cf -#### Others +For more details, please refer to: [CausalVideoVAE Report](docs/CausalVideoVAE.md). + +### VideoGPT VQVAE Please refer to the document [VQVAE](docs/VQVAE.md).