-
-This project aims to create a simple and scalable repo, to reproduce [Sora](https://openai.com/sora) (OpenAI, but we prefer to call it "ClosedAI" ). We wish the open-source community can contribute to this project. Pull requests are welcome! The current code supports complete training and inference using the Huawei Ascend AI computing system. Models trained on Huawei Ascend can also output video quality comparable to industry standards.
-
-本项目希望通过开源社区的力量复现Sora，由北大-兔展AIGC联合实验室共同发起，当前版本离目标差距仍然较大，仍需持续完善和快速迭代，欢迎Pull request！目前代码同时支持使用国产AI计算系统（华为昇腾）进行完整的训练和推理。基于昇腾训练出的模型，也可输出持平业界的视频质量。
-
-
If you like our project, please give us a star ⭐ on GitHub for latest update.
-
-
-# 📣 News
-* **[2024.08.13]** 🎉 We are launching Open-Sora Plan v1.2.0 **I2V** model, which based on Open-Sora Plan v1.2.0. The current version supports image-to-video generation and transition generation (the starting and ending frames conditions for video generation). Checking out the Image-to-Video section in this [report](https://github.com/PKU-YuanGroup/Open-Sora-Plan/blob/main/docs/Report-v1.2.0.md#training-image-to-video-diffusion-model).
-* **[2024.07.24]** 🔥🔥🔥 v1.2.0 is here! Utilizing a 3D full attention architecture instead of 2+1D. We released a true 3D video diffusion model trained on 4s 720p. Checking out our latest [report](docs/Report-v1.2.0.md).
-* **[2024.05.27]** 🎉 We are launching Open-Sora Plan v1.1.0, which significantly improves video quality and length, and is fully open source! Please check out our latest [report](docs/Report-v1.1.0.md). Thanks to [ShareGPT4Video's](https://sharegpt4video.github.io/) capability to annotate long videos.
-* **[2024.04.09]** 🤝 Excited to share our latest exploration on metamorphic time-lapse video generation: [MagicTime](https://github.com/PKU-YuanGroup/MagicTime), which learns real-world physics knowledge from time-lapse videos.
-* **[2024.04.07]** 🎉🎉🎉 Today, we are thrilled to present Open-Sora-Plan v1.0.0, which significantly enhances video generation quality and text control capabilities. See our [report](docs/Report-v1.0.0.md). Thanks to HUAWEI NPU for supporting us.
-* **[2024.03.27]** 🚀🚀🚀 We release the report of [VideoCausalVAE](docs/CausalVideoVAE.md), which supports both images and videos. We present our reconstructed video in this demonstration as follows. The text-to-video model is on the way.
-* **[2024.03.01]** 🤗 We launched a plan to reproduce Sora, called Open-Sora Plan! Welcome to **watch** 👀 this repository for the latest updates.
-
-# 😍 Gallery
-
-93×1280×720 Text-to-Video Generation. The video quality has been compressed for playback on GitHub.
-
-
-
-
-
-
-
-
-# 😮 Highlights
-
-Open-Sora Plan shows excellent performance in video generation.
-
-### 🔥 High performance CausalVideoVAE, but with fewer training cost
-- High compression ratio with excellent performance, capable of **compressing videos by 256 times (4×8×8)**. Causal convolution supports simultaneous inference of images and videos but only need **1 node to train**.
-
-### 🚀 Video Diffusion Model based on 3D attention, joint learning of spatiotemporal features.
-- With a **3D full attention architecture** instead of a 2+1D model, 3D attention can better capture joint spatial and temporal features.
-
-
-
-
-
-# 🤗 Demo
-
-### Gradio Web UI
-
-Highly recommend trying out our web demo by the following command.
-
-```bash
-python -m opensora.serve.gradio_web_server --model_path "path/to/model" --ae_path "path/to/causalvideovae"
-```
-
-### ComfyUI
-
-Coming soon...
-
-# 🐳 Resource
-
-| Version | Architecture | Diffusion Model | CausalVideoVAE | Data|
-|:---|:---|:---|:---|:---|
-| v1.2.0 | 3D | [93x720p](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/93x720p), [29x720p](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/29x720p)[1], [93x480p](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/93x480p)[1,2], [29x480p](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/29x480p), [1x480p](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/1x480p), [93x480p_i2v](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/93x480p_i2v) | [Anysize](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/vae)| [Annotations](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.2.0) |
-| v1.1.0 | 2+1D | [221x512x512](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.1.0/tree/main/221x512x512), [65x512x512](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.1.0/tree/main/65x512x512) |[Anysize](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.1.0/tree/main/vae) |[Data and Annotations](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.1.0)|
-| v1.0.0 | 2+1D | [65x512x512](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.0.0/tree/main/65x512x512), [65x256x256](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.0.0/tree/main/65x256x256), [17x256x256](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.0.0/tree/main/17x256x256) | [Anysize](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.0.0/tree/main/vae) | [Data and Annotations](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.0.0)|
-
-> [1] Please note that the weights for v1.2.0 29×720p and 93×480p were trained on Panda70M and have not undergone final high-quality data fine-tuning, so they may produce watermarks.
-
-> [2] We fine-tuned 3.5k steps from 93×720p to get 93×480p for community research use.
-
-> [!Warning]
->
->
->
-> 🚨 For version 1.2.0, we no longer support 2+1D models.
->
->
-
-# ⚙️ Requirements and Installation
-
-1. Clone this repository and navigate to Open-Sora-Plan folder
-```
-git clone https://github.com/PKU-YuanGroup/Open-Sora-Plan
-cd Open-Sora-Plan
-```
-2. Install required packages
-We recommend the requirements as follows.
-* Python >= 3.8
-* Pytorch >= 2.1.0
-* CUDA Version >= 11.7
-```
-conda create -n opensora python=3.8 -y
-conda activate opensora
-pip install -e .
-```
-3. Install additional packages for training cases
-```
-pip install -e ".[train]"
-```
-4. Install optional requirements such as static type checking:
-```
-pip install -e '.[dev]'
-```
-
-# 🗝️ Training & Validating
-
-## 🗜️ CausalVideoVAE
-
-### Data prepare
-The organization of the training data is easy. We only need to put all the videos recursively in a directory. This makes the training more convenient when using multiple datasets.
-``` shell
-Training Dataset
-|——sub_dataset1
- |——sub_sub_dataset1
- |——video1.mp4
- |——video2.mp4
- ......
- |——sub_sub_dataset2
- |——video3.mp4
- |——video4.mp4
- ......
-|——sub_dataset2
- |——video5.mp4
- |——video6.mp4
- ......
-|——video7.mp4
-|——video8.mp4
-```
-### Training
-``` shell
-bash scripts/causalvae/train.sh
-```
-We introduce the important args for training.
-| Argparse | Usage |
-|:---|:---|
-|_Training size_||
-|`--num_frames`|The number of using frames for training videos|
-|`--resolution`|The resolution of the input to the VAE|
-|`--batch_size`|The local batch size in each GPU|
-|`--sample_rate`|The frame interval of when loading training videos|
-|_Data processing_||
-|`--video_path`|/path/to/dataset|
-|_Load weights_||
-|`--model_config`|/path/to/config.json The model config of VAE. If you want to train from scratch use this parameter.|
-|`--pretrained_model_name_or_path`|A directory containing a model checkpoint and its config. Using this parameter will only load its weight but not load the state of the optimizer|
-|`--resume_from_checkpoint`|/path/to/checkpoint It will resume the training process from the checkpoint including the weight and the optimizer.|
-### Inference
-``` shell
-bash scripts/causalvae/rec_video.sh
-```
-We introduce the important args for inference.
-| Argparse | Usage |
-|:---|:---|
-|_Ouoput video size_||
-|`--num_frames`|The number of frames of generated videos|
-|`--height`|The resolution of generated videos|
-|`--width`|The resolution of generated videos|
-|_Data processing_||
-|`--video_path`|The path to the original video|
-|`--rec_path`|The path to the generated video|
-|_Load weights_||
-|`--ae_path`|/path/to/model_dir. A directory containing the checkpoint of VAE is used for inference and its model config.json|
-|_Other_||
-|`--enable_tilintg`|Use tiling to deal with videos of high resolution and long duration|
-|`--save_memory`|Save memory to inference but lightly influence quality|
-
-
-### Evaluation
-
-
-For evaluation, you should save the original video clips by using `--output_origin`.
-``` shell
-bash scripts/causalvae/prepare_eval.sh
-```
-We introduce the important args for inference.
-| Argparse | Usage |
-|:---|:---|
-|_Ouoput video size_||
-|`--num_frames`|The number of frames of generated videos|
-|`--resolution`|The resolution of generated videos|
-|_Data processing_||
-|`--real_video_dir`|The directory of the original videos.|
-|`--generated_video_dir`|The directory of the generated videos.|
-|_Load weights_||
-|`--ckpt`|/path/to/model_dir. A directory containing the checkpoint of VAE is used for inference and its model config.|
-|_Other_||
-|`--enable_tilintg`|Use tiling to deal with videos of high resolution and long time.|
-|`--output_origin`|Output the original video clips, fed into the VAE.|
-
-
-Then, we begin to eval. We introduce the important args in the script for evaluation.
-``` shell
-bash scripts/causalvae/eval.sh
-```
-| Argparse | Usage |
-|:---|:---|
-|`--metric`|The metric, such as psnr, ssim, lpips|
-|`--real_video_dir`|The directory of the original videos.|
-|`--generated_video_dir`|The directory of the generated videos.|
-
-## 📜 Text-to-Video
-
-### Data prepare
-We use a `data.txt` file to specify all the training data. Each line in the file consists of `DATA_ROOT` and `DATA_JSON`. The example of `data.txt` is as follows.
-```
-/path/to/data_root_1,/path/to/data_json_1.json
-/path/to/data_root_2,/path/to/data_json_2.json
-...
-```
-Then, we introduce the format of the annotation json file. The absolute data path is the concatenation of `DATA_ROOT` and the `"path"` field in the annotation json file.
-#### For image
-The format of image annotation file is as follows.
-```
-[
- {
- "path": "00168/001680102.jpg",
- "cap": [
- "xxxxx."
- ],
- "resolution": {
- "height": 512,
- "width": 683
- }
- },
- ...
-]
-```
-
-#### For video
-The format of video annotation file is as follows. More details refer to [HF dataset](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.2.0).
-```
-[
- {
- "path": "panda70m_part_5565/qLqjjDhhD5Q/qLqjjDhhD5Q_segment_0.mp4",
- "cap": [
- "A man and a woman are sitting down on a news anchor talking to each other."
- ],
- "resolution": {
- "height": 720,
- "width": 1280
- },
- "fps": 29.97002997002997,
- "duration": 11.444767
- },
- ...
-]
-```
-
-### Training
+## Training
```
bash scripts/text_condition/gpu/train_t2v.sh
```
-We introduce some key parameters in order to customize your training process.
-| Argparse | Usage |
-|:---|:---|
-|_Training size_||
-|`--num_frames 61`|To train videos of different durations, e.g, 29, 61, 93, 125...|
-|`--max_height 640`|To train videos of different resolutions|
-|`--max_width 480`|To train videos of different resolutions|
-|_Data processing_||
-|`--data /path/to/data.txt`|Specify your training data.|
-|`--speed_factor 1.25`|To accelerate 1.25x videos. |
-|`--drop_short_ratio 1.0`|Do not want to train on videos of dynamic durations, discard all video data with frame counts not equal to `--num_frames`|
-|`--group_frame`|If you want to train with videos of dynamic durations, we highly recommend specifying `--group_frame` as well. It improves computational efficiency during training.|
-|_Multi-stage transfer learning_||
-|`--interpolation_scale_h 1.0`|When training a base model, such as 240p (`--max_height 240`, `--interpolation_scale_h 1.0`) , and you want to initialize higher resolution models like 480p (height 480) from 240p's weights, you need to adjust `--max_height 480`, `--interpolation_scale_h 2.0`, and set `--pretrained` to your 240p weights path (path/to/240p/xxx.safetensors).|
-|`--interpolation_scale_w 1.0`|Same as `--interpolation_scale_h 1.0`|
-|_Load weights_||
-|`--pretrained`|This is typically used for loading pretrained weights across stages, such as using 240p weights to initialize 480p training. Or when switching datasets and you do not want the previous optimizer state.|
-|`--resume_from_checkpoint`|It will resume the training process from the latest checkpoint in `--output_dir`. Typically, we set `--resume_from_checkpoint="latest"`, which is useful in cases of unexpected interruptions during training.|
-|_Sequence Parallelism_||
-|`--sp_size 8 --train_sp_batch_size 2`|It means running a batch size of 2 across 8 GPUs (8 GPUs on the same node).|
-
-> [!Warning]
->
->
-> 🚨 We have two ways to load weights: `--pretrained` and `--resume_from_checkpoint`. The latter will override the former.
->
->
-
-### Inference
-
-We provide multiple inference scripts to support various requirements. We recommend configuration `--guidance_scale 7.5 --num_sampling_steps 100 --sample_method EulerAncestralDiscrete` for sampling.
-
-**Inference on 93×720p**, we report speed on H100.
-
-| Size | 1 GPU | 8 GPUs (sp) |
-|---|---|---|
-|29×720p|420s/100step|80s/100step|
-|93×720p|3400s/100step|450s/100step|
-
-#### 🖥️ 1 GPU
-If you only have one GPU, it will perform inference on each sample sequentially, one at a time.
-```
-bash scripts/text_condition/gpu/sample_t2v.sh
-```
-
-#### 🖥️🖥️ Multi-GPUs
-If you want to batch infer a large number of samples, each GPU will infer one sample.
-```
-bash scripts/text_condition/gpu/sample_t2v_ddp.sh
-```
-
-#### 🖥️🖥️ Multi-GPUs & Sequence Parallelism
-If you want to quickly infer one sample, it will utilize all GPUs simultaneously to infer that sample.
-```
-bash scripts/text_condition/gpu/sample_t2v_sp.sh
-```
-
-## 🖼️ Image-to-Video
-
-### Data prepare
-Same as Text-to-Video.
-
-### Training
-```
-bash scripts/text_condition/gpu/train_inpaint.sh
-```
-
-In addition to the parameters shared with the Text-to-Video mode, there are some unique parameters specific to the Image-to-Video mode that you need to be aware of.
-
-| Argparse | Usage |
-| :---------------- | :----------------------------------------------------------- |
-| _Training size_ | |
-| `--use_vae_preprocessed_mask` | Whether to use VAE (Variational Autoencoder) to encode the mask in order to achieve frame-level mask alignment. |
-| _Data processing_ | |
-| `--i2v_ratio 0.5` | The proportion of training data allocated to executing the Image-to-Video task. |
-| `--transition_ratio 0.4 ` | The proportion of training data allocated to executing the transition task. |
-| ` --v2v_ratio 0.1 ` | The proportion of training data allocated to executing the video continuation task. |
-| `--default_text_ratio 0.5` | When training with CFG (Classifier-Free Guidance) enabled, a portion of the text is replaced with default text, while another portion is set to an empty string. |
-| _Load weights_ | |
-| `--pretrained_transformer_model_path ` | This parameter functions the same as the `--pretrained` parameter. |
-
-### Inference
-
-In the current version, we have only open-sourced the 480p version of the Image-to-Video (I2V) model. We recommend configuration `--guidance_scale 7.5 --num_sampling_steps 100 --sample_method PNDM` for sampling. Please note that due to the addition of frame-controllable fine-tuning, using the other samplers may not yield satisfactory results.
-
-**Inference on 93×480p**, we report speed on H100.
-
-| Size | 1 GPU | 8 GPUs (sp) |
-| ------- | ------------ | ----------- |
-| 93×480p | 427s/100step | 81s/100step |
-
-Before inference, you need to create two text files: one named `prompt.txt` and another named `conditional_images_path.txt`. Each line of text in `prompt.txt` should correspond to the paths on each line in `conditional_images_path.txt`.
-
-For example, if the content of `prompt.txt` is:
-
-```
-this is a prompt of i2v task.
-this is a prompt of transition task.
-```
-
-Then the content of conditional_images_path should be:
-
-```
-/path/to/image_0.png
-/path/to/image_1_0.png,/path/to/image_1_1.png
-```
-
-This means we will execute a image-to-video task using `/path/to/image_0.png` and "this is a prompt of i2v task." For the transition task, we'll use `/path/to/image_1_0.png` and `/path/to/image_1_1.png` (note that these two paths are separated by a comma without any spaces) along with "this is a prompt of transition task."
-
-After creating the files, make sure to specify their paths in the `sample_inpaint.sh` script.
-
-#### 🖥️ 1 GPU
-
-If you only have one GPU, it will perform inference on each sample sequentially, one at a time.
-```
-bash scripts/text_condition/gpu/sample_inpaint.sh
-```
-#### 🖥️🖥️ Multi-GPUs
-If you want to batch infer a large number of samples, each GPU will infer one sample.
-```
-bash scripts/text_condition/gpu/sample_inpaint_ddp.sh
-```
-
-#### 🖥️🖥️ Multi-GPUs & Sequence Parallelism
-If you want to quickly infer one sample, it will utilize all GPUs simultaneously to infer that sample.
-```
-bash scripts/text_condition/gpu/sample_inpaint_sp.sh
-```
-
-# 💡 How to Contribute
-We greatly appreciate your contributions to the Open-Sora Plan open-source community and helping us make it even better than it is now!
-
-For more details, please refer to the [Contribution Guidelines](docs/Contribution_Guidelines.md)
-
-
-
-
-# 👍 Acknowledgement
-* [Latte](https://github.com/Vchitect/Latte): It is an wonderful 2+1D video generated model.
-* [PixArt-alpha](https://github.com/PixArt-alpha/PixArt-alpha): Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis.
-* [ShareGPT4Video](https://github.com/InternLM/InternLM-XComposer/tree/main/projects/ShareGPT4Video): Improving Video Understanding and Generation with Better Captions.
-* [VideoGPT](https://github.com/wilson1yan/VideoGPT): Video Generation using VQ-VAE and Transformers.
-* [DiT](https://github.com/facebookresearch/DiT): Scalable Diffusion Models with Transformers.
-* [FiT](https://github.com/whlzy/FiT): Flexible Vision Transformer for Diffusion Model.
-* [Positional Interpolation](https://arxiv.org/abs/2306.15595): Extending Context Window of Large Language Models via Positional Interpolation.
-
-
-# 🔒 License
-* See [LICENSE](LICENSE) for details.
-
-
-
-
-# ✏️ Citing
-
-## BibTeX
-
-```bibtex
-@software{pku_yuan_lab_and_tuzhan_ai_etc_2024_10948109,
- author = {PKU-Yuan Lab and Tuzhan AI etc.},
- title = {Open-Sora-Plan},
- month = apr,
- year = 2024,
- publisher = {GitHub},
- doi = {10.5281/zenodo.10948109},
- url = {https://doi.org/10.5281/zenodo.10948109}
-}
-```
-## Latest DOI
-
-[](https://zenodo.org/records/10948109)
-
-# 🤝 Community contributors
-
-
-
-
+## TODO
+- [ ] Add Mochi
+- [ ] Add Mochi VAE
+- [ ] Create dummy debug data.
\ No newline at end of file
diff --git a/docker/LICENSE b/docker/LICENSE
deleted file mode 100644
index c4bc32c80..000000000
--- a/docker/LICENSE
+++ /dev/null
@@ -1,21 +0,0 @@
-MIT License
-
-Copyright (c) 2024 SimonLee
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
diff --git a/docker/README.md b/docker/README.md
deleted file mode 100644
index 461c24dde..000000000
--- a/docker/README.md
+++ /dev/null
@@ -1,87 +0,0 @@
-# Docker4ML
-
-Useful docker scripts for ML development.
-[https://github.com/SimonLeeGit/Docker4ML](https://github.com/SimonLeeGit/Docker4ML)
-
-## Build Docker Image
-
-```bash
-bash docker_build.sh
-```
-
-
-
-## Run Docker Container as Development Environment
-
-```bash
-bash docker_run.sh
-```
-
-
-
-## Custom Docker Config
-
-### Config [setup_env.sh](./setup_env.sh)
-
-You can modify this file to custom your settings.
-
-```bash
-TAG=ml:dev
-BASE_TAG=nvcr.io/nvidia/pytorch:23.12-py3
-```
-
-#### TAG
-
-Your built docker image tag, you can set it as what you what.
-
-#### BASE_TAG
-
-The base docker image tag for your built docker image, here we use nvidia pytorch images.
-You can check it from [https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch/tags](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch/tags)
-
-Also, you can use other docker image as base, such as: [ubuntu](https://hub.docker.com/_/ubuntu/tags)
-
-### USER_NAME
-
-Your user name used in docker container.
-
-### USER_PASSWD
-
-Your user password used in docker container.
-
-### Config [requirements.txt](./requirements.txt)
-
-You can add your default installed python libraries here.
-
-```txt
-transformers==4.27.1
-```
-
-By default, it has some libs installed, you can check it from [https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel-24-01.html](https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel-24-01.html)
-
-### Config [packages.txt](./packages.txt)
-
-You can add your default apt-get installed packages here.
-
-```txt
-wget
-curl
-git
-```
-
-### Config [ports.txt](./ports.txt)
-
-You can add some ports enabled for docker container here.
-
-```txt
--p 6006:6006
--p 8080:8080
-```
-
-### Config [postinstallscript.sh](./postinstallscript.sh)
-
-You can add your custom script to run when build docker image.
-
-## Q&A
-
-If you have any use problems, please contact to .
diff --git a/docker/build_docker.png b/docker/build_docker.png
deleted file mode 100755
index a5dbf2c14..000000000
Binary files a/docker/build_docker.png and /dev/null differ
diff --git a/docker/docker_build.sh b/docker/docker_build.sh
deleted file mode 100644
index 6e82a157f..000000000
--- a/docker/docker_build.sh
+++ /dev/null
@@ -1,8 +0,0 @@
-#!/usr/bin/env bash
-
-WORK_DIR=$(dirname "$(readlink -f "$0")")
-cd $WORK_DIR
-
-source setup_env.sh
-
-docker build -t $TAG --build-arg BASE_TAG=$BASE_TAG --build-arg USER_NAME=$USER_NAME --build-arg USER_PASSWD=$USER_PASSWD . -f dockerfile.base
diff --git a/docker/docker_run.sh b/docker/docker_run.sh
deleted file mode 100644
index 5fa4b407d..000000000
--- a/docker/docker_run.sh
+++ /dev/null
@@ -1,45 +0,0 @@
-#!/usr/bin/env bash
-
-WORK_DIR=$(dirname "$(readlink -f "$0")")
-source $WORK_DIR/setup_env.sh
-
-RUNNING_IDS="$(docker ps --filter ancestor=$TAG --format "{{.ID}}")"
-
-if [ -n "$RUNNING_IDS" ]; then
- # Initialize an array to hold the container IDs
- declare -a container_ids=($RUNNING_IDS)
-
- # Get the first container ID using array indexing
- ID=${container_ids[0]}
-
- # Print the first container ID
- echo ' '
- echo "The running container ID is: $ID, enter it!"
-else
- echo ' '
- echo "Not found running containers, run it!"
-
- # Run a new docker container instance
- ID=$(docker run \
- --rm \
- --gpus all \
- -itd \
- --ipc=host \
- --ulimit memlock=-1 \
- --ulimit stack=67108864 \
- -e DISPLAY=$DISPLAY \
- -v /tmp/.X11-unix/:/tmp/.X11-unix/ \
- -v $PWD:/home/$USER_NAME/workspace \
- -w /home/$USER_NAME/workspace \
- $(cat $WORK_DIR/ports.txt) \
- $TAG)
-fi
-
-docker logs $ID
-
-echo ' '
-echo ' '
-echo '========================================='
-echo ' '
-
-docker exec -it $ID bash
diff --git a/docker/dockerfile.base b/docker/dockerfile.base
deleted file mode 100644
index b83488d48..000000000
--- a/docker/dockerfile.base
+++ /dev/null
@@ -1,24 +0,0 @@
-ARG BASE_TAG
-FROM ${BASE_TAG}
-ARG USER_NAME=myuser
-ARG USER_PASSWD=111111
-ARG DEBIAN_FRONTEND=noninteractive
-
-# Pre-install packages, pip install requirements and run post install script.
-COPY packages.txt .
-COPY requirements.txt .
-COPY postinstallscript.sh .
-RUN apt-get update && apt-get install -y sudo $(cat packages.txt)
-RUN pip install --no-cache-dir -r requirements.txt
-RUN bash postinstallscript.sh
-
-# Create a new user and group using the username argument
-RUN groupadd -r ${USER_NAME} && useradd -r -m -g${USER_NAME} ${USER_NAME}
-RUN echo "${USER_NAME}:${USER_PASSWD}" | chpasswd
-RUN usermod -aG sudo ${USER_NAME}
-USER ${USER_NAME}
-ENV USER=${USER_NAME}
-WORKDIR /home/${USER_NAME}/workspace
-
-# Set the prompt to highlight the username
-RUN echo "export PS1='\[\033[01;32m\]\u\[\033[00m\]@\[\033[01;34m\]\h\[\033[00m\]:\[\033[01;36m\]\w\[\033[00m\]\$'" >> /home/${USER_NAME}/.bashrc
diff --git a/docker/packages.txt b/docker/packages.txt
deleted file mode 100644
index 1ec90dd5b..000000000
--- a/docker/packages.txt
+++ /dev/null
@@ -1,3 +0,0 @@
-wget
-curl
-git
\ No newline at end of file
diff --git a/docker/ports.txt b/docker/ports.txt
deleted file mode 100644
index 2fd9c4567..000000000
--- a/docker/ports.txt
+++ /dev/null
@@ -1 +0,0 @@
--p 6006:6006
\ No newline at end of file
diff --git a/docker/postinstallscript.sh b/docker/postinstallscript.sh
deleted file mode 100644
index 457f47c17..000000000
--- a/docker/postinstallscript.sh
+++ /dev/null
@@ -1,3 +0,0 @@
-#!/usr/bin/env bash
-# this script will run when build docker image.
-
diff --git a/docker/requirements.txt b/docker/requirements.txt
deleted file mode 100644
index d2c490c3b..000000000
--- a/docker/requirements.txt
+++ /dev/null
@@ -1,40 +0,0 @@
-setuptools>=61.0
-torch==2.0.1
-torchvision==0.15.2
-transformers==4.32.0
-albumentations==1.4.0
-av==11.0.0
-decord==0.6.0
-einops==0.3.0
-fastapi==0.110.0
-accelerate==0.21.0
-gdown==5.1.0
-h5py==3.10.0
-idna==3.6
-imageio==2.34.0
-matplotlib==3.7.5
-numpy==1.24.4
-omegaconf==2.1.1
-opencv-python==4.9.0.80
-opencv-python-headless==4.9.0.80
-pandas==2.0.3
-pillow==10.2.0
-pydub==0.25.1
-pytorch-lightning==1.4.2
-pytorchvideo==0.1.5
-PyYAML==6.0.1
-regex==2023.12.25
-requests==2.31.0
-scikit-learn==1.3.2
-scipy==1.10.1
-six==1.16.0
-tensorboard==2.14.0
-test-tube==0.7.5
-timm==0.9.16
-torchdiffeq==0.2.3
-torchmetrics==0.5.0
-tqdm==4.66.2
-urllib3==2.2.1
-uvicorn==0.27.1
-diffusers==0.24.0
-scikit-video==1.1.11
diff --git a/docker/run_docker.png b/docker/run_docker.png
deleted file mode 100755
index dbce9b9c4..000000000
Binary files a/docker/run_docker.png and /dev/null differ
diff --git a/docker/setup_env.sh b/docker/setup_env.sh
deleted file mode 100644
index e7b1ecc77..000000000
--- a/docker/setup_env.sh
+++ /dev/null
@@ -1,11 +0,0 @@
-# Docker tag for new build image
-TAG=open_sora_plan:dev
-
-# Base docker image tag used by docker build
-BASE_TAG=nvcr.io/nvidia/pytorch:23.05-py3
-
-# User name used in docker container
-USER_NAME=developer
-
-# User password used in docker container
-USER_PASSWD=666666
\ No newline at end of file
diff --git a/docs/Contribution_Guidelines.md b/docs/Contribution_Guidelines.md
deleted file mode 100644
index d7f163a3d..000000000
--- a/docs/Contribution_Guidelines.md
+++ /dev/null
@@ -1,87 +0,0 @@
-# Contributing to the Open-Sora Plan Community
-
-The Open-Sora Plan open-source community is a collaborative initiative driven by the community, emphasizing a commitment to being free and void of exploitation. Organized spontaneously by community members, we invite you to contribute to the Open-Sora Plan open-source community and help elevate it to new heights!
-
-## Submitting a Pull Request (PR)
-
-As a contributor, before submitting your request, kindly follow these guidelines:
-
-1. Start by checking the [Open-Sora Plan GitHub](https://github.com/PKU-YuanGroup/Open-Sora-Plan/pulls) to see if there are any open or closed pull requests related to your intended submission. Avoid duplicating existing work.
-
-2. [Fork](https://github.com/PKU-YuanGroup/Open-Sora-Plan/fork) the [open-sora plan](https://github.com/PKU-YuanGroup/Open-Sora-Plan) repository and download your forked repository to your local machine.
-
- ```bash
- git clone [your-forked-repository-url]
- ```
-
-3. Add the original Open-Sora Plan repository as a remote to sync with the latest updates:
-
- ```bash
- git remote add upstream https://github.com/PKU-YuanGroup/Open-Sora-Plan
- ```
-
-4. Sync the code from the main repository to your local machine, and then push it back to your forked remote repository.
-
- ```
- # Pull the latest code from the upstream branch
- git fetch upstream
-
- # Switch to the main branch
- git checkout main
-
- # Merge the updates from the upstream branch into main, synchronizing the local main branch with the upstream
- git merge upstream/main
-
- # Additionally, sync the local main branch to the remote branch of your forked repository
- git push origin main
- ```
-
-
- > Note: Sync the code from the main repository before each submission.
-
-5. Create a branch in your forked repository for your changes, ensuring the branch name is meaningful.
-
- ```bash
- git checkout -b my-docs-branch main
- ```
-
-6. While making modifications and committing changes, adhere to our [Commit Message Format](#Commit-Message-Format).
-
- ```bash
- git commit -m "[docs]: xxxx"
- ```
-
-7. Push your changes to your GitHub repository.
-
- ```bash
- git push origin my-docs-branch
- ```
-
-8. Submit a pull request to `Open-Sora-Plan:main` on the GitHub repository page.
-
-## Commit Message Format
-
-Commit messages must include both `` and `` sections.
-
-```bash
-[]:
- │ │
- │ └─⫸ Briefly describe your changes, without ending with a period.
- │
- └─⫸ Commit Type: |docs|feat|fix|refactor|
-```
-
-### Type
-
-* **docs**: Modify or add documents.
-* **feat**: Introduce a new feature.
-* **fix**: Fix a bug.
-* **refactor**: Restructure code, excluding new features or bug fixes.
-
-### Summary
-
-Describe modifications in English, without ending with a period.
-
-> e.g., git commit -m "[docs]: add a contributing.md file"
-
-This guideline is borrowed by [minisora](https://github.com/mini-sora/minisora). We sincerely appreciate MiniSora authors for their awesome templates.
diff --git a/docs/Report-v1.0.0-cn.md b/docs/Report-v1.0.0-cn.md
deleted file mode 100644
index 3ddc23e0b..000000000
--- a/docs/Report-v1.0.0-cn.md
+++ /dev/null
@@ -1,135 +0,0 @@
-# 技术报告 v1.0.0
-
-在2024年3月，我们推出了Open-Sora-Plan，一个旨在复现OpenAI [Sora](https://openai.com/sora)的开源计划。它作为一个基础的开源框架，能够训练视频生成模型包括无条件视频生成，类别引导视频生成，文生视频。
-
-**今天，我们兴奋地展示Open-Sora-Plan v1.0.0，极大地改进视频生成质量、文本控制能力。**
-
-相比于之前的视频生成模型，Open-Sora-Plan v1.0.0 有以下的改进：
-
-1. **CausalVideoVAE高效的训练与推理**。 我们用4×8×8的对视频进行时间和空间的压缩。
-2. **图片视频联合训练提升视觉质量**。 CasualVideoVAE 将首帧看作图片，天然支持同时编码图片和视频。这允许扩散模型提取更多时空细节来改善质量。
-
-
-### Open-Source Release
-我们开源了Open-Sora-Plan去促进视频生成社区的进一步发展。公开代码、数据、模型。
-- 在线演示：Hugging Face [](https://huggingface.co/spaces/LanguageBind/Open-Sora-Plan-v1.0.0)， [](https://replicate.com/camenduru/open-sora-plan-512x512) 和 [](https://colab.research.google.com/github/camenduru/Open-Sora-Plan-jupyter/blob/main/Open_Sora_Plan_jupyter.ipynb), 感谢[@camenduru](https://github.com/camenduru)大力支持我们的工作！🤝
-- 代码：所有训练脚本和采样代码。
-- 模型：包括扩散模型和CausalVideoVAE [这里](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.0.0)。
-- 数据：所有原视频和对应描述 [这里](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.0.0)。
-
-## 效果
-
-Open-Sora-Plan v1.0.0支持图片视频联合训练。我们在此展示视频和图片的重建以及生成：
-
-720×1280**视频重建**。 因为github的限制，原视频放在: [1](https://streamable.com/gqojal), [2](https://streamable.com/6nu3j8).
-
-https://github.com/PKU-YuanGroup/Open-Sora-Plan/assets/88202804/c100bb02-2420-48a3-9d7b-4608a41f14aa
-
-https://github.com/PKU-YuanGroup/Open-Sora-Plan/assets/88202804/8aa8f587-d9f1-4e8b-8a82-d3bf9ba91d68
-
-1536×1024**图片重建**
-
-
-
-65×1024×1024**文生视频**
-
-https://github.com/PKU-YuanGroup/Open-Sora-Plan/assets/62638829/2641a8aa-66ac-4cda-8279-86b2e6a6e011
-
-65×512×512**文生视频**
-
-https://github.com/PKU-YuanGroup/Open-Sora-Plan/assets/62638829/37e3107e-56b3-4b09-8920-fa1d8d144b9e
-
-
-512×512**文生视频**
-
-
-
-## 详细技术报告
-
-### CausalVideoVAE
-
-#### 模型结构
-
-
-
-因果VAE架构继承了[Stable-Diffusion Image VAE](https://github.com/CompVis/stable-diffusion/tree/main)。 为了保证图片VAE的预训练权重可以无缝运用到视频VAE中，模型结构采取如下设计:
-
-1. **CausalConv3D**: 将Conv2D 转变成CausalConv3D可以实现图片和视频的联合训练. CausalConv3D 对第一帧进行特殊处理，因为它无法访问后续帧。对于更多细节，请参考https://github.com/PKU-YuanGroup/Open-Sora-Plan/pull/145
-
-2. **初始化**：将Conv2D扩展到Conv3D常用的[方法](https://github.com/hassony2/inflated_convnets_pytorch/blob/master/src/inflate.py#L5)有两种：平均初始化和中心初始化。 但我们采用了特定的初始化方法（尾部初始化）。 这种初始化方法确保模型无需任何训练就能够直接重建图像，甚至视频。
-
-#### 训练细节
-
-
-
-我们展示了 17×256×256 下两种不同初始化方法的损失曲线。黄色曲线代表使用尾部初始化的损失，而蓝色曲线对应中心初始化的损失。 如图所示，尾部初始化在损失曲线上表现出更好的性能。 此外，我们发现中心初始化会导致错误累积，导致长时间内崩溃。
-
-#### 推理技巧
-尽管训练Diffusion中VAE始终是冻住的，我们仍然无法负担CasualVideoVAE的花销。在我们的实验中, 80G的显存只能够在半精度下推理一个256×512×512或32×1024×1024的视频 ，这限制了我们扩展到更长更高清的视频。因此我们采用tile convolution，能够以几乎恒定的内存推理任意时长或任意分辨率的视频。
-
-### 数据构建
-我们定义高质量的视频数据集包括两个核心法则：(1) 没有与内容无关的水印。(2) 高质量的文本注释。
-
-**对于法则1**，我们从开源网站（CC0协议）爬取了大约40k videos：1234个来自[mixkit](https://mixkit.co/)，7408个来自[pexels](https://www.pexels.com/)，31616个来自[pixabay](https://pixabay.com/)。我们根据[Panda70M](https://github.com/snap-research/Panda-70M/blob/main/splitting/README.md)提供的场景变换剪切script将这些视频切成大约434k video clips。事实上，根据我们的剪切结果，从这些网上上爬取的99%的视频都是单一的场景。另外，我们发现爬取的数据中超过60%为风景相关视频。更多细节可以在[这](https://github.com/PKU-YuanGroup/Open-Sora-Dataset)找到。
-
-**对于法则2**，很难有大量的高质量的文本注释能够从网上直接爬取。因此我们用成熟的图片标注模型来获取高质量的稠密描述。我们对2个多模态大模型进行消融实验：[ShareGPT4V-Captioner-7B](https://github.com/InternLM/InternLM-XComposer/blob/main/projects/ShareGPT4V/README.md) 和 [LLaVA-1.6-34B](https://github.com/haotian-liu/LLaVA)。前者是专门用来制作文本注释的模型，而后者是一个通用的多模态大模型。经过我们的消融实验，他们在caption的表现差不多。然而他们的推理速度在A800上差距很大：40s/it of batch size of 12 for ShareGPT4V-Captioner-7B，15s/it of batch size of 1 for ShareGPT4V-Captioner-7B。我们开源所有的[文本注释和原视频](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.0.0)。
-
-| 模型名字 | 平均长度 | 最大值 | 标准差 |
-|---|---|---|---|
-| ShareGPT4V-Captioner-7B | 170.0827524529121 | 467 | 53.689967539537776 |
-| LLaVA-1.6-34B | 141.75851073472666 | 472 | 48.52492072346965 |
-
-### 训练扩散模型
-与之前的工作类似，我们采用多阶段的级联的训练方法，总共消耗了2048个A800 GPU 小时。我们发现联合图片训练能够显著加速模型的收敛并且增强视觉观感，这与[Latte](https://github.com/Vchitect/Latte)一致。以下是我们的训练花销。
-
-| 名字 | Stage 1 | Stage 2 | Stage 3 | Stage 4 |
-|---|---|---|---|---|
-| 训练视频尺寸 | 17×256×256 | 65×256×256 | 65×512×512 | 65×1024×1024 |
-| 计算资源 (#A800 GPU x #小时) | 32 × 40 | 32 × 18 | 32 × 6 | 训练中 |
-| 权重 | [HF](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.0.0/tree/main/17x256x256) | [HF](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.0.0/tree/main/65x256x256) | [HF](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.0.0/tree/main/65x512x512) | 训练中 |
-| 日志 | [wandb](https://api.wandb.ai/links/linbin/p6n3evym) | [wandb](https://api.wandb.ai/links/linbin/t2g53sew) | [wandb](https://api.wandb.ai/links/linbin/uomr0xzb) | 训练中 |
-| 训练数据 | ~40k videos | ~40k videos | ~40k videos | ~40k videos |
-
-## 下版本预览
-### CausalVideoVAE
-目前我们发布的CausalVideoVAE v1.0.0版本存在2个主要的缺陷：**运动模糊**以及**网格效应**。我们对CasualVideoVAE做了一系列的改进使它推理成本更低且性能更强大，我们暂时叫它为预览版本，将在下个版本发布。
-
-**1分钟720×1280视频重建**。 受限于GitHub，我们将原视频放在这：[原视频](https://streamable.com/u4onbb)，[重建视频](https://streamable.com/qt8ncc)。
-
-https://github.com/PKU-YuanGroup/Open-Sora-Plan/assets/88202804/cdcfa9a3-4de0-42d4-94c0-0669710e407b
-
-我们从kinetic 400的验证集中随机选取100个样本进行评估，结果表如下所示：
-
-| | SSIM↑ | LPIPS↓ | PSNR↑ | FLOLPIPS↓ |
-|---|---|---|---|---|
-| v1.0.0 | 0.829 | 0.106 | 27.171 | 0.119 |
-| Preview | 0.877 | 0.064 | 29.695 | 0.070 |
-
-#### 运动模糊
-
-| **v1.0.0** | **预览版本** |
-| --- | --- |
-|  |  |
-
-#### 网格效应
-
-| **v1.0.0** | **预览版本** |
-| --- | --- |
-|  |  |
-
-### 数据构建
-
-**数据源**：正如上文提到，我们的数据集中超过60%为风景视频。这意味着我们的开域视频生成能力有限。然而当前的大规模开源数据集大多从YouTube爬取，尽管视频的数量多，但我们担忧视频本身的质量是否达标。因此，我们将继续收集高质量的数据集，同时也欢迎开源社区的推荐。
-
-**Caption生成流程**：当我们训练时长增加时，我们不得不考虑更有效的视频caption生成方法，而不是多模态图片大模型。我们正在开发一个新的视频注释生成管线，它能够很好的支持长视频，敬请期待。
-
-### 训练扩散模型
-尽管目前v1.0.0展现了可喜的结果，但我们仍然离Sora有一段距离。在接下来的工作中，我们主要围绕这三个方面:
-
-1. **动态分辨率与时长的训练**: 我们的目标是开发出能够以不同分辨率和持续时间训练模型的技术，使训练过程更加灵活、适应性更强。
-
-2. **更长的视频生成**: 我们将探索扩展模型生成能力的方法，使其能够制作更长的视频，超越目前的限制。
-
-3. **更多条件控制**: 我们力求增强模型的条件控制能力，为用户提供更多的选项和对生成视频的控制能力。
-
-另外，通过仔细观察生成的视频，我们发现存在一些不符合常理的斑点或异常的流动，这是由于CasualVideoVAE的性能不足导致的 如上面提到。在未来的实验中，我们将使用更强的VAE，重新训练一个扩散模型。
diff --git a/docs/Report-v1.0.0.md b/docs/Report-v1.0.0.md
deleted file mode 100644
index 8b8419976..000000000
--- a/docs/Report-v1.0.0.md
+++ /dev/null
@@ -1,136 +0,0 @@
-# Report v1.0.0
-
-In March 2024, we launched a plan called Open-Sora-Plan, which aims to reproduce the OpenAI [Sora](https://openai.com/sora) through an open-source framework. As a foundational open-source framework, it enables training of video generation models, including Unconditioned Video Generation, Class Video Generation, and Text-to-Video Generation.
-
-**Today, we are thrilled to present Open-Sora-Plan v1.0.0, which significantly enhances video generation quality and text control capabilities.**
-
-Compared with previous video generation model, Open-Sora-Plan v1.0.0 has several improvements:
-
-1. **Efficient training and inference with CausalVideoVAE**. We apply a spatial-temporal compression to the videos by 4×8×8.
-2. **Joint image-video training for better quality**. Our CausalVideoVAE considers the first frame as an image, allowing for the simultaneous encoding of both images and videos in a natural manner. This allows the diffusion model to grasp more spatial-visual details to improve visual quality.
-
-### Open-Source Release
-We open-source the Open-Sora-Plan to facilitate future development of Video Generation in the community. Code, data, model are made publicly available.
-- Demo: Hugging Face demo [](https://huggingface.co/spaces/LanguageBind/Open-Sora-Plan-v1.0.0). 🤝 Enjoying the [](https://replicate.com/camenduru/open-sora-plan-512x512) and [](https://colab.research.google.com/github/camenduru/Open-Sora-Plan-jupyter/blob/main/Open_Sora_Plan_jupyter.ipynb), created by [@camenduru](https://github.com/camenduru), who generously supports our research!
-- Code: All training scripts and sample scripts.
-- Model: Both Diffusion Model and CausalVideoVAE [here](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.0.0).
-- Data: Both raw videos and captions [here](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.0.0).
-
-## Gallery
-
-Open-Sora-Plan v1.0.0 supports joint training of images and videos. Here, we present the capabilities of Video/Image Reconstruction and Generation:
-
-### CausalVideoVAE Reconstruction
-
-**Video Reconstruction** with 720×1280. Since github can't upload large video, we put it here: [1](https://streamable.com/gqojal), [2](https://streamable.com/6nu3j8).
-
-https://github.com/PKU-YuanGroup/Open-Sora-Plan/assets/88202804/c100bb02-2420-48a3-9d7b-4608a41f14aa
-
-https://github.com/PKU-YuanGroup/Open-Sora-Plan/assets/88202804/8aa8f587-d9f1-4e8b-8a82-d3bf9ba91d68
-
-**Image Reconstruction** in 1536×1024.
-
-
-
-**Text-to-Video Generation** with 65×1024×1024
-
-https://github.com/PKU-YuanGroup/Open-Sora-Plan/assets/62638829/2641a8aa-66ac-4cda-8279-86b2e6a6e011
-
-**Text-to-Video Generation** with 65×512×512
-
-https://github.com/PKU-YuanGroup/Open-Sora-Plan/assets/62638829/37e3107e-56b3-4b09-8920-fa1d8d144b9e
-
-
-**Text-to-Image Generation** with 512×512
-
-
-
-## Detailed Technical Report
-
-### CausalVideoVAE
-
-#### Model Structure
-
-
-
-The CausalVideoVAE architecture inherits from the [Stable-Diffusion Image VAE](https://github.com/CompVis/stable-diffusion/tree/main). To ensure that the pretrained weights of the Image VAE can be seamlessly applied to the Video VAE, the model structure has been designed as follows:
-
-1. **CausalConv3D**: Converting Conv2D to CausalConv3D enables joint training of image and video data. CausalConv3D applies a special treatment to the first frame, as it does not have access to subsequent frames. For more specific details, please refer to https://github.com/PKU-YuanGroup/Open-Sora-Plan/pull/145
-
-2. **Initialization**: There are two common [methods](https://github.com/hassony2/inflated_convnets_pytorch/blob/master/src/inflate.py#L5) to expand Conv2D to Conv3D: average initialization and center initialization. But we employ a specific initialization method (tail initialization). This initialization method ensures that without any training, the model is capable of directly reconstructing images, and even videos.
-
-#### Training Details
-
-
-
-We present the loss curves for two distinct initialization methods under 17×256×256. The yellow curve represents the loss using tail init, while the blue curve corresponds to the loss from center initialization. As shown in the graph, tail initialization demonstrates better performance on the loss curve. Additionally, we found that center initialization leads to error accumulation, causing the collapse over extended durations.
-
-#### Inference Tricks
-Despite the VAE in Diffusion training being frozen, we still find it challenging to afford the cost of the CausalVideoVAE. In our case, with 80GB of GPU memory, we can only infer a video of either 256×512×512 or 32×1024×1024 resolution using half-precision, which limits our ability to scale up to longer and higher-resolution videos. Therefore, we adopt tile convolution, which allows us to infer videos of arbitrary duration or resolution with nearly constant memory usage.
-
-### Data Construction
-We define a high-quality video dataset based on two core principles: (1) No content-unrelated watermarks. (2) High-quality and dense captions.
-
-**For principles 1**, we crawled approximately 40,000 videos from open-source websites under the CC0 license. Specifically, we obtained 1,234 videos from [mixkit](https://mixkit.co/), 7,408 videos from [pexels](https://www.pexels.com/), and 31,616 videos from [pixabay](https://pixabay.com/). These videos adhere to the principle of having no content-unrelated watermarks. According to the scene transformation and clipping script provided by [Panda70M](https://github.com/snap-research/Panda-70M/blob/main/splitting/README.md), we have divided these videos into approximately 434,000 video clips. In fact, based on our clipping results, 99% of the videos obtained from these online sources are found to contain single scenes. Additionally, we have observed that over 60% of the crawled data comprises landscape videos. More details can be found [here](https://github.com/PKU-YuanGroup/Open-Sora-Dataset).
-
-**For principles 2**, it is challenging to directly crawl a large quantity of high-quality dense captions from the internet. Therefore, we utilize a mature Image-captioner model to obtain high-quality dense captions. We conducted ablation experiments on two multimodal large models: [ShareGPT4V-Captioner-7B](https://github.com/InternLM/InternLM-XComposer/blob/main/projects/ShareGPT4V/README.md) and [LLaVA-1.6-34B](https://github.com/haotian-liu/LLaVA). The former is specifically designed for caption generation, while the latter is a general-purpose multimodal large model. After conducting our ablation experiments, we found that they are comparable in performance. However, there is a significant difference in their inference speed on the A800 GPU: 40s/it of batch size of 12 for ShareGPT4V-Captioner-7B, 15s/it of batch size of 1 for LLaVA-1.6-34B. We open-source all annotations [here](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.0.0). We show some statistics here, and we set the maximum length of the model to 300, which covers almost 99% of the samples.
-
-| Name | Avg length | Max | Std |
-|---|---|---|---|
-| ShareGPT4V-Captioner-7B | 170.0827524529121 | 467 | 53.689967539537776 |
-| LLaVA-1.6-34B | 141.75851073472666 | 472 | 48.52492072346965 |
-
-### Training Diffusion Model
-Similar to previous work, we employ a multi-stage cascaded training approach, which consumes a total of 2,528 A800 GPU hours. We found that joint training with images significantly accelerates model convergence and enhances visual perception, aligning with the findings of [Latte](https://github.com/Vchitect/Latte). Below is our training card:
-
-| Name | Stage 1 | Stage 2 | Stage 3 | Stage 4 |
-|---|---|---|---|---|
-| Training Video Size | 17×256×256 | 65×256×256 | 65×512×512 | 65×1024×1024 |
-| Compute (#A800 GPU x #Hours) | 32 × 40 | 32 × 22 | 32 × 17 | Under training |
-| Checkpoint | [HF](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.0.0/tree/main/17x256x256) | [HF](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.0.0/tree/main/65x256x256) | [HF](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.0.0/tree/main/65x512x512) | Under training |
-| Log | [wandb](https://api.wandb.ai/links/linbin/p6n3evym) | [wandb](https://api.wandb.ai/links/linbin/t2g53sew) | [wandb](https://api.wandb.ai/links/linbin/uomr0xzb) | Under training |
-| Training Data | ~40k videos | ~40k videos | ~40k videos | ~40k videos |
-
-## Next Release Preview
-### CausalVideoVAE
-Currently, the released version of CausalVideoVAE (v1.0.0) has two main drawbacks: **motion blurring** and **gridding effect**. We have made a series of improvements to CausalVideoVAE to reduce its inference cost and enhance its performance. We are currently referring to this enhanced version as the "preview version," which will be released in the next update. Preview reconstruction is as follows:
-
-**1 min Video Reconstruction with 720×1280**. Since github can't put too big video, we put it here: [origin video](https://streamable.com/u4onbb), [reconstruction video](https://streamable.com/qt8ncc).
-
-https://github.com/PKU-YuanGroup/Open-Sora-Plan/assets/88202804/cdcfa9a3-4de0-42d4-94c0-0669710e407b
-
-We randomly selected 100 samples from the validation set of Kinetics-400 for evaluation, and the results are presented in the following table:
-
-| | SSIM↑ | LPIPS↓ | PSNR↑ | FLOLPIPS↓ |
-|---|---|---|---|---|
-| v1.0.0 | 0.829 | 0.106 | 27.171 | 0.119 |
-| Preview | 0.877 | 0.064 | 29.695 | 0.070 |
-
-#### Motion Blurring
-
-| **v1.0.0** | **Preview** |
-| --- | --- |
-|  |  |
-
-#### Gridding effect
-
-| **v1.0.0** | **Preview** |
-| --- | --- |
-|  |  |
-
-### Data Construction
-
-**Data source**. As mentioned earlier, over 60% of our dataset consists of landscape videos. This implies that our ability to generate videos in other domains is limited. However, most of the current large-scale open-source datasets are primarily obtained through web scraping from platforms like YouTube. While these datasets provide a vast quantity of videos, we have concerns about the quality of the videos themselves. Therefore, we will continue to collect high-quality datasets and also welcome recommendations from the open-source community. We are launching an Open-Sora-Dataset project, check out the details at [Open-Sora-Dataset](https://github.com/PKU-YuanGroup/Open-Sora-Dataset)
-
-**Caption Generation Pipeline**. As the video duration increases, we need to consider more efficient methods for video caption generation instead of relying solely on large multimodal image models. We are currently developing a new video caption generation pipeline that provides robust support for long videos. We are excited to share more details with you in the near future. Stay tuned!
-
-### Training Diffusion Model
-Although v1.0.0 has shown promising results, we acknowledge that we still have a ways to go to reach the level of Sora. In our upcoming work, we will primarily focus on three aspects:
-
-1. **Training support for dynamic resolution and duration**: We aim to develop techniques that enable training models with varying resolutions and durations, allowing for more flexible and adaptable training processes.
-
-2. **Support for longer video generation**: We will explore methods to extend the generation capabilities of our models, enabling them to produce longer videos beyond the current limitations.
-
-3. **Enhanced conditional control**: We seek to enhance the conditional control capabilities of our models, providing users with more options and control over the generated videos.
-
-Furthermore, through careful observation of the generated videos, we have noticed the presence of some non-physiological speckles or abnormal flow. This can be attributed to the limited performance of CausalVideoVAE, as mentioned earlier. In future experiments, we plan to retrain a diffusion model using a more powerful version of CausalVideoVAE to address these issues.
diff --git a/docs/Report-v1.1.0.md b/docs/Report-v1.1.0.md
deleted file mode 100644
index ca3ec2ba5..000000000
--- a/docs/Report-v1.1.0.md
+++ /dev/null
@@ -1,256 +0,0 @@
-# Report v1.1.0
-
-In April 2024, we launched Open-Sora-Plan v1.0.0, featuring a simple and efficient design along with remarkable performance in text-to-video generation. It has already been adopted as a foundational model in numerous research projects, including its data and model.
-
-**Today, we are excited to present Open-Sora-Plan v1.1.0, which significantly improves video generation quality and duration.**
-
-Compared to the previous version, Open-Sora-Plan v1.1.0, the improvements include:
-
-1. **Better compressed visual representations**. We optimized the CausalVideoVAE architecture, which now has stronger performance and higher inference efficiency.
-2. **Generate higher quality, longer videos**. We used higher quality visual data and captions by [ShareGPT4Video](https://sharegpt4video.github.io/), enabling the model to better understand the workings of the world.
-
-Along with performance improvements, Open-Sora-Plan v1.1.0 maintains the minimalist design and data efficiency of v1.0.0. Remarkably, we found that v1.1.0 exhibits similar performance to the Sora base model, indicating that our version's evolution aligns with the scaling law demonstrated by Sora.
-
-### Open-Source Release
-We open-source the Open-Sora-Plan to facilitate future development of Video Generation in the community. Code, data, model will be made publicly available.
-- Demo: Hugging Face demo [here](https://huggingface.co/spaces/LanguageBind/Open-Sora-Plan-v1.1.0).
-- Code: All training scripts and sample scripts.
-- Model: Both Diffusion Model and CasualVideoVAE [here](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.1.0).
-- Data: Both raw videos and captions [here](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.1.0).
-
-## Gallery
-
-
-### 221×512×512 Text-to-Video Generation
-
-| 221×512×512 (9.2s) | 221×512×512 (9.2s) | 221×512×512 (9.2s) | 221×512×512 (9.2s) |
-| --- | --- | --- | --- |
-| | | | |
-| This close-up shot of a Victoria crowned pigeon showcases its striking blue plumage ... | a cat wearing sunglasses and working as a lifeguard at pool. | Photorealistic closeup video of two pirate ships battling each other as they sail ... | A movie trailer featuring the adventures ofthe 30 year old spacemanwearing a redwool ... |
-| | | | |
-| A snowy forest landscape with a dirt road running through it. The road is flanked by ... | Drone shot along the Hawaii jungle coastline, sunny day. Kayaks in the water. | Alpacas wearing knit wool sweaters, graffiti background, sunglasses. | The camera rotates around a large stack of vintage televisions all showing different ... |
-| | | | |
-| A drone camera circles around a beautiful historic church built on a rocky outcropping ... | Aerial view of Santorini during the blue hour, showcasing the stunning architecture ... | A robot dog explores the surface of Mars, kicking up red dust as it investigates ... | An aerial shot of a lighthouse standing tall on a rocky cliff, its beacon cutting ... |
-| | | | |
-| 3D animation of a small, round, fluffy creature with big, expressive eyes explores ... | A corgi vlogging itself in tropical Maui. | A single drop of liquid metal falls from a floating orb, landing on a mirror-like ... | The video presents an abstract composition centered around a hexagonal shape adorned ... |
-
-### 65×512×512 Text-to-Video Generation
-
-| 65×512×512 (2.7s) | 65×512×512 (2.7s) | 65×512×512 (2.7s) | 65×512×512 (2.7s) |
-| --- | --- | --- | --- |
-| | | | |
-| Extreme close-up of chicken and green pepper kebabs grilling on a barbeque with flames. | 3D animation of a small, round, fluffy creature with big, expressive eyes explores a ... | A corgi vlogging itself in tropical Maui. | In a studio, there is a painting depicting a ship sailing through the rough sea. |
-| | | | |
-| A robot dog trots down a deserted alley at night, its metallic paws clinking softly ... | A solitary spider weaves its web in a quiet corner. The web shimmers and glows with ... | A lone surfer rides a massive wave, skillfully maneuvering through the surf. The water ... | A solitary cheetah sprints across the savannah, its powerful muscles propelling it ... |
-| | | | |
-| A solitary astronaut plants a flag on an alien planet covered in crystal formations ... | At dawn's first light, a spaceship slowly exits the edge of the galaxy against a ...| A dapper puppy in a miniature suit, basking in the afternoon sun, adjusting his tie ... | A wise old elephant painting abstract art with its trunk, each stroke a burst of color ... |
-| | | | |
-| In an ornate, historical hall, a massive tidal wave peaks and begins to crash. Two ... | A Shiba Inu dog wearing a beret and black turtleneck. | A painting of a boat on water comes to life, with waves crashing and the boat becoming ... | Many spotted jellyfish pulsating under water. Their bodies are transparent and glowing ... |
-| | | | |
-| An animated hedgehog with distinctive spiky hair and large eyes is seen exploring a ... | An animated rabbit in a playful pink snowboarding outfit is carving its way down a ... | A person clad in a space suit with a helmet and equipped with a chest light and arm ... | |
-
-### 65×512×512 Video Editing
-
-| generated 65×512×512 (2.7s) | edited 65×512×512 (2.7s) |
-| --- | --- |
-| | |
-| | |
-| | |
-
-### 512×512 Text-to-Image Generation
-
-
-
-## Detailed Technical Report
-
-### CasualVideoVAE
-
-#### Model Structure
-
-As the number of frames increases, the encoder overhead of CausalVideoVAE gradually rises. When training with 257 frames, 80GB of VRAM is insufficient for the VAE to encode the video. Therefore, we reduced the number of CausalConv3D layers, retaining only the last two stages of CausalConv3D in the encoder. This change significantly lowers the overhead while maintaining nearly the same performance. Note that we only modified the encoder; the decoder still retains all CausalConv3D layers, as training the Diffusion Model does not require the decoder.
-
-
-
-We compare the computational overhead of the two versions by testing the forward inference of the encoder on the H100.
-
-| Version | 129×256×256 | | 257×256×256 | | 513×256×256 | |
-|---|---|---|---|---|---|---|
-| | Peak Mem. | Speed | Peak Mem. | Speed |Peak Mem. | Speed |
-| v1.0.0 | 22G | 2.9 it/s | OOM | - | OOM | - |
-| v1.1.0 | 18G | 4.9 it/s | 34G | 2.5 it/s | 61G | 1.2 it/s |
-
-
-#### Temporal Module
-
-
-
-In v1.0.0, our temporal module had only TemporalAvgPool. TemporalAvgPool leads to the loss of high-frequency information in the video, such as details and edges. To address this issue, we improved this module in v1.1.0. As shown in the figure below, we introduced convolution and added learnable weights, allowing different branches to decouple different features. When we omit CausalConv3D, the video is reconstructed very blurry. Similarly, when we omit TemporalAvgPool, the video becomes very sharp.
-
-| | SSIM↑ | LPIPS↓ | PSNR↑ |
-|---|---|---|---|
-| Base | 0.850 | 0.091 | 28.047 |
-| + Frames | 0.868 | 0.070 | 28.829 |
-| + Reset mixed factor | 0.873 | 0.070 | 29.140 |
-
-
-
-
-
-#### Training Details
-
-Similar to v1.0.0, we initialized from the Latent Diffusion's VAE and used tail initialization. For CausalVideoVAE, we trained for 100k steps in the first stage with a video shape of 9×256×256. Subsequently, we increased the frame count from 9 to 25 and found that this significantly improved the model's performance. It is important to clarify that we enabled the mixed factor during both the first and second stages, with a value of a (sigmoid(mixed factor)) reaching 0.88 at the end of training, indicating the model's tendency to retain low-frequency information. In the third stage, we reinitialized the mixed factor to 0.5 (sigmoid(0.5) = 0.6225), which further enhanced the model's capabilities.
-
-#### Loss Function
-
-We found that using GAN loss helps retain high-frequency information and alleviates grid artifacts. Additionally, we observed that switching from 2D GAN to 3D GAN provides further improvements.
-
-| GAN Loss/Step | SSIM↑ | LPIPS↓ | PSNR↑ |
-|---|---|---|---|
-| 2D/80k | 0.879 | 0.068 | 29.480 |
-| 3D/80k | 0.882 | 0.067 | 29.890 |
-
-#### Inference Tricks
-Therefore, we introduced a method called **temporal rollback tiled convolution**, a tiling approach specifically designed for CausalVideoVAE. Specifically, all windows except the first one discard the first frame because the first frame in a window is treated as an image, while the remaining frames should be treated as video frames.
-
-
-
-We tested the speed on the H100 with a window size of 65×256×256.
-
-| Version | 129×256×256 | | 257×256×256 | | 513×256×256 | |
-|---|---|---|---|---|---|---|
-| | Peak Mem. | Speed | Peak Mem. | Speed |Peak Mem. | Speed |
-| 4×8×8 | 10G | 1.3 s/it | 10G | 2.6 s/it | 10G | 5.3 s/it |
-
-### Data Construction
-Since Open-Sora-Plan supports joint training of images and videos, our data collection is divided into two parts: images and videos. Images do not need to originate from videos; they are independent datasets. We spent approximately 32×240 H100 hours generating image and video captions, and all of this is **open source**!
-
-#### Image-Text Collection Pipeline
-We obtained 11 million image-text pairs from [Pixart-Alpha](https://huggingface.co/datasets/PixArt-alpha/SAM-LLaVA-Captions10M), with captions generated by [LLaVA](https://github.com/haotian-liu/LLaVA). Additionally, we utilized the high-quality OCR dataset [Anytext-3M](https://github.com/tyxsspa/AnyText), which pairs each image with corresponding OCR characters. However, these captions were insufficient to describe the entire image, so we used [InternVL-1.5](https://github.com/OpenGVLab/InternVL) for supplementary descriptions. Since T5 only supports English, we filtered for English data, which constitutes about half of the complete dataset. Furthermore, we selected high-quality images from [Laion-5B](https://laion.ai/blog/laion-5b/) to enhance human-like generation quality. The selection criteria included high resolution, high aesthetic scores, and watermark-free images containing people.
-
-Here, we are open-sourcing the prompt used for InternVL-1.5:
-```
-# for anytext-3m
-Combine this rough caption: "{}", analyze the image in a comprehensive and detailed manner. "{}" can be recognized in the image.
-# for human-160k
-Analyze the image in a comprehensive and detailed manner.
-```
-
-| Name | Image Source | Text Captioner | Num pair |
-|---|---|---|---|
-| SAM-11M | [SAM](https://ai.meta.com/datasets/segment-anything/) | [LLaVA](https://github.com/haotian-liu/LLaVA) | 11,185,255 |
-| Anytext-3M-en | [Anytext](https://github.com/tyxsspa/AnyText) | [InternVL-1.5](https://github.com/OpenGVLab/InternVL) | 1,886,137 |
-| Human-160k | [Laion](https://laion.ai/blog/laion-5b/) | [InternVL-1.5](https://github.com/OpenGVLab/InternVL) | 162,094 |
-
-
-#### Video-Text Collection Pipeline
-In v1.0.0, we sampled one frame from each video to generate captions. However, as video length increased, a single frame could not adequately describe the entire video's content or temporal movements. Therefore, we used a video captioner to generate captions for the entire video clip. Specifically, we used [ShareGPT4Video](https://sharegpt4video.github.io/), which effectively covers temporal information and describes the entire video content. The v1.1.0 video dataset comprises approximately 3k hours, compared to only 300 hours in v1.0.0. As before, we have open-sourced all text annotations and videos (both under the CC0 license), which can be found [here](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.1.0/tree/main).
-
-
-
-| Name | Hours | Num frames | Num pair |
-|---|---|---|---|
-| [Mixkit](https://mixkit.co/) | 42.0h | 65 | 54,735 |
-| | | 513 | 1,997 |
-| [Pixabay](https://pixabay.com/) | 353.3h | 65 | 601,513 |
-| | | 513 | 51,483 |
-| [Pexel](https://www.pexels.com/) | 2561.9h | 65 | 3,832,666 |
-| | | 513 | 271,782 |
-
-### Training Diffusion Model
-Similar to our previous work, we employed a multi-stage cascaded training method. Below is our training card:
-
-#### Stage 1
-
-Surprisingly, we initially believed that the performance of the diffusion model would improve with longer training. However, by observing the [logs](https://api.wandb.ai/links/linbin/o76j03j4), we found that videos generated at 50k steps were of higher quality than those at 70-100k steps. In fact, extensive sampling revealed that checkpoints at 40-60k steps outperformed those at 80-100k steps. Quantitatively, 50k steps correspond to approximately 2 epochs of training. It is currently unclear whether this is due to overfitting from a small dataset or the limited capacity of the 2+1D model.
-
-#### Stage 2
-
-In the second stage, we used Huawei Ascend computing power for training. This stage's training and inference were fully supported by Huawei. We conducted sequence parallel training and inference on a large-scale cluster, distributing one sample across eight ranks. Models trained on Huawei Ascend can also be loaded into GPUs and generate videos of the same quality.
-
-
-#### Stage 3
-
-In the third stage, we further increased the frame count to 513 frames, approximately 21 seconds at 24 FPS. However, this stage presents several challenges, such as ensuring temporal consistency in the 2+1D model over long durations and whether the current amount of data is sufficient. We are still training the model for this stage and continuously monitoring its progress.
-
-| Name | Stage 1 | Stage 2 | Stage 3 |
-|---|---|---|---|
-| Training Video Size | 65×512×512 | 221×512×512 | 513×512×512 |
-| Compute (#Num x #Hours) | 80 H100 × 72 | 512 Ascend × 72 | Under Training |
-| Checkpoint | [HF](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.1.0/tree/main/65x512x512) | [HF](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.1.0/tree/main/221x512x512) | Under Training |
-| Log | [wandb](https://api.wandb.ai/links/linbin/o76j03j4) | - | - |
-| Training Data | ~3k hours videos + 13M images | | |
-
-### Video Editing
-
-The recently proposed [ReVideo](https://mc-e.github.io/project/ReVideo/) achieves accurate video editing by modifying the first frame and applying motion control within the edited area. Although it achieves excellent video editing performance, the editing length is limited by the base model [SVD](https://github.com/Stability-AI/generative-models). Open-Sora, as a fundamental model for long-video generation, can compensate for this issue. Currently, we are collaborating with the ReVideo team to use Open-Sora as the base model for long video editing. Some preliminary results are shown [here]().
-
-The initial version still needs improvement in several aspects. In the future, we will continue to explore integration with ReVideo to develop improved long-video editing models.
-
-## Failed Case and Discussion
-
-Despite the promising results of v1.1.0, there remains a gap between our model and Sora. Here, we present some failure cases and discuss them.
-
-### CasualVideoVAE
-
-Despite the significant performance improvement of VAE in v1.1.0 over v1.0.0, we still encounter failures in challenging cases, such as sand dunes and leaves. The video on the left shows the reconstructed video downsampled by a factor of 4 in time, while the video on the right is downsampled by a factor of 2. Both exhibit jitter when reconstructing fine-grained features. This indicates that reducing temporal downsampling alone cannot fully resolve the jitter issue.
-
-https://github.com/PKU-YuanGroup/Open-Sora-Plan/assets/62638829/1a87d6d8-4bf1-4b4e-83bb-84870c5c3a11
-
-https://github.com/PKU-YuanGroup/Open-Sora-Plan/assets/62638829/1a87d6d8-4bf1-4b4e-83bb-84870c5c3a11
-
-### Diffusion Model
-
-#### Semantic distortion
-
-On the left is a video generated by v1.1.0 showing a puppy in the snow. In this video, the puppy's head exhibits semantic distortion, indicating that the model struggles to correctly identify which head belongs to which dog. On the right is a video generated by Sora's [base model](https://openai.com/index/video-generation-models-as-world-simulators/). We observe that Sora's early base model also experienced semantic distortion issues. This suggests that we may achieve better results by scaling up the model and increasing the amount of training data.
-
-Prompt：A litter of golden retriever puppies playing in the snow.Their heads pop out of the snow, covered in.
-
-| Our | Sora Base×1 | Sora Base×4 | Sora Base×32 |
-|---|---|---|---|
-| | | | |
-
-#### Limited dynamics
-
-The primary difference between videos and images lies in their dynamic nature, where objects undergo a series of changes across consecutive frames. However, the videos generated by v1.1.0 still contain many instances of limited dynamics. Upon reviewing a large number of training videos, we found that while web-crawled videos have high visual quality, they are often filled with meaningless close-up shots. These close-ups typically show minimal movement or are even static. On the left, we present a generated video of a bird, while on the right is a training video we found, which is almost static. There are many similar videos in the dataset from stock footage sites.
-
-Prompt：This close-up shot of a Victoria crowned pigeon showcases its striking blue plumage and red chest. Its crest is made of delicate, lacy feathers, while its eye is a striking red color. The bird's head is tilted slightly to the side,giving the impression of it looking regal and majestic. The background is blurred,drawing attention to the bird's striking appearance.
-
-
-| Our | Raw video |
-|---|---|
-| | |
-
-#### Negative prompt
-
-We found that using negative prompts can significantly improve video quality, even though we did not explicitly tag the training data with different labels. On the left is a video sampled using a negative prompt, while on the right is a video generated without a negative prompt. This suggests that we may need to incorporate more prior knowledge into the training data. For example, when a video has a watermark, we should note "watermark" in the corresponding caption. When a video's bitrate is too low, we should add more tags to distinguish it from high-quality videos, such as "low quality" or "blurry." We believe that explicitly injecting these priors can help the model differentiate between the vast amounts of pretraining data (low quality) and the smaller amounts of fine-tuning data (high quality), thereby generating higher quality videos.
-
-Prompt：A litter of golden retriever puppies playing in the snow.Their heads pop out of the snow, covered in.
-Negative Prompt：distorted, discontinuous, ugly, blurry, low resolution, motionless, static, low quality
-
-
-| With Negative Prompt | Without Negative Prompt |
-|---|---|
-| | |
-
-## Future Work
-
-In our future work, we will focus on two main areas: (1) data scaling and (2) model design. Once we have a robust baseline model, we will extend it to handle variable durations and conditional control models.
-
-### Data Scaling
-
-#### Data source
-
-As mentioned earlier, our dataset is entirely sourced from stock footage websites. Although these videos are of high quality, many consist of close-up shots of specific areas, resulting in slow motion in the videos. We believe this is one of the main reasons for the limited dynamics observed. Therefore, we will continue to collect datasets from diverse sources to address this issue.
-
-#### Data volume
-
-In v1.1.0, our dataset comprises only ~3k hours of video. We are actively collecting more data and anticipate that the video dataset for the next version will reach ~100k hours. We welcome recommendations from the open-source community for additional datasets.
-
-### Model Design
-
-#### CasualVideoVAE
-In our internal testing, even without downsampling in time, we found that it is not possible to completely resolve the jitter issue in reconstructing fine-grained features. Therefore, we need to reconsider how to mitigate video jitter to the greatest extent possible while simultaneously supporting both images and videos. We will introduce a more powerful CasualVideoVAE in the next version.
-
-#### Diffusion Model
-In v1.1.0, we found that 2+1D models can generate higher-quality videos in short durations. However, for long videos, they tend to exhibit discontinuities and inconsistencies. Therefore, we will explore more possibilities in model architecture to address this issue.
diff --git a/docs/Report-v1.2.0.md b/docs/Report-v1.2.0.md
deleted file mode 100644
index 8a122211a..000000000
--- a/docs/Report-v1.2.0.md
+++ /dev/null
@@ -1,193 +0,0 @@
-# Report v1.2.0
-
-In May 2024, we launched Open-Sora-Plan v1.1.0, featuring a 2+1D model architecture that could be quickly utilized for exploratory training in text-to-video generation tasks. However, when handling dense visual tokens, the 2+1D architecture could not simultaneously process spatial and temporal dimensions. Therefore, we transitioned to **a 3D full attention architecture**, which better captures the joint spatial-temporal features. Although this version is experimental, it advances video generation architecture to a new realm, leading us to release it as v1.2.0.
-
-Compared to previous video generation models, Open-Sora-Plan v1.2.0 offers the following improvements:
-
-1. **Better compressed visual representations**. We optimized the structure of CausalVideoVAE, which now delivers enhanced performance and higher inference efficiency.
-2. **Better video generation architecture**. Instead of 2+1D, we use a diffusion model with a 3D full attention architecture, which provides a better understanding of the world.
-
-
-### Open-Source Release
-We open-source the Open-Sora-Plan to facilitate future development of Video Generation in the community. Code, data, model are made publicly available.
-- Code: All training scripts and sample scripts.
-- Model: Both Diffusion Model and CasualVideoVAE [here](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0).
-- Data: Filtered data [here](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.2.0).
-
-## Gallery
-
-93×1280×720 Text-to-Video Generation. The video quality has been compressed for playback on GitHub.
-
-
-
-
-
-
-
-## Detailed Technical Report
-
-### CausalVideoVAE
-
-#### Model Structure
-
-The VAE in version 1.2.0 maintains the overall architecture of the previous version but merges the temporal and spatial downsampling layers. In version 1.1.0, we performed spatial downsampling (stride=1,2,2) followed by temporal downsampling (stride=2,1,1). In version 1.2.0, we conduct both spatial and temporal downsampling simultaneously (stride=2,2,2) and perform spatial-temporal upsampling in the decoder (interpolate_factor=2,2,2).
-
-Due to the absence of additional convolutions during downsampling and upsampling, this method more seamlessly inherits the weights from the SD2.1 VAE, leading to improved initialization of our VAE.
-
-
-
-
-#### Training Details
-
-
-As with v1.1.0, we initialize from the [SD2.1 VAE](https://huggingface.co/stabilityai/sd-vae-ft-mse) using tail initialization. We perform the first phase of training on the Kinetic400 video dataset, then use the EMA weights from this phase to initialize the second phase, which is fine-tuned on high-quality data (collected in v1.1.0). All training is conducted on 25-frame 256×256 videos using **one A100 node**.
-
-
-| Training stage | Dataset | Training steps |
-|---|---|---|
-| 1 | K400 | 200,000 |
-| 2 | collected in v1.1.0 | 450,000 |
-
-
-#### Evaluation
-
-We evaluated our VAE on the validation sets of two video datasets: [Webvid](https://github.com/m-bain/webvid) and [Panda70m](https://github.com/snap-research/Panda-70M/), and compared it with our [v1.1.0](https://github.com/PKU-YuanGroup/Open-Sora-Plan/blob/main/docs/Report-v1.1.0.md), [SD2.1 VAE](https://huggingface.co/stabilityai/sd-vae-ft-mse), [CV-VAE](https://github.com/AILab-CVC/CV-VAE), and [Open-Sora's VAE](https://github.com/hpcaitech/Open-Sora). The Webvid validation set contains 5k videos, while the Panda70m validation set has 6k videos. The videos were resized to 256 pixels on the short side, center-cropped to 256x256, and then 33 consecutive frames were extracted. We used PSNR, SSIM, and LPIPS metrics, and measured the encoding speed on an A100 GPU. The specific results are as follows:
-
-
-**WebVid**
-
-| Model | Compress Ratio |PNSR↑ | SSIM↑ |LPIPS↓ |
-|---|---|---|---|---|
-| SD2-1 VAE | 1x8x8 | 30.19 | 0.8379 | 0.0568 |
-| SVD VAE | 1x8x8 |31.15 |0.8686 | **0.0547** |
-| CV-VAE | 4x8x8 | 30.76 | 0.8566 | 0.0803 |
-| Open-Sora VAE | 4x8x8 | 31.12 | 0.8569 | 0.1003 |
-| Open-Sora Plan v1.1 | 4x8x8 | 30.26 | 0.8597 |0.0551 |
-| Open-Sora Plan v1.2 | 4x8x8| **31.16** | **0.8694** | 0.0586 |
-
-**Panda70M**
-
-| Model | Compress Ratio| PNSR↑ | SSIM↑ |LPIPS↓ |
-|---|---|---|---|---|
-| SD2-1 VAE | 1x8x8 |30.40 | 0.8894 | 0.0396 |
-| SVD VAE | 1x8x8 |31.00 | **0.9058** | **0.0379** |
-| CV-VAE | 4x8x8| 29.57 | 0.8795 | 0.0673 |
-| Open-Sora VAE | 4x8x8 | **31.06** | 0.8969 | 0.0666 |
-| Open-Sora Plan v1.1 | 4x8x8 | 29.16 | 0.8844 | 0.0481 |
-| Open-Sora Plan v1.2 | 4x8x8| 30.49 |0.8970 |0.0454|
-
-**Encode Time on A100**
-
-|Input Size| CV-VAE | Open-Sora | Open-Sora Plan v1.1 | Open-Sora Plan v1.2 |
-|---|---|---|---|---|
-| 33x256x256 | 0.186 | 0.147 |0.104 | **0.102** |
-| 81x256x256 | 0.465 | 0.357 |0.243 | **0.242** |
-
-### Training Text-to-Video Diffusion Model
-
-#### Model Structure
-
-The most significant change is that we **replaced all 2+1D Transformer blocks with 3D full attention blocks**. Each video is first processed by a patch embedding layer, which downsamples the spatial dimensions by a factor of 2. The video is then flattened into a one-dimensional sequence across the frame, width, and height dimensions. We replaced [T5-XXL](https://huggingface.co/DeepFloyd/t5-v1_1-xxl) with [mT5-XXL](https://huggingface.co/google/mt5-xxl) to enhance multilingual adaptation. Additionally, we incorporated RoPE.
-
-
-### Sequence Parallelism
-
-Due to the high computational complexity of 3D full attention, we must allocate a video across 2 GPUs for parallel processing when training with long-duration and high-resolution videos. We can control the number of GPUs used for a video sample by adjusting the batch size on a node. For example, with `sp_size=8` and `train_sp_batch_size=4`, 2 GPUs are used for a single sample. **We support sequence parallelism for both training and inference**.
-
-**Training on 93×720p**, we report speed on H100.
-
-| GPU （sp_size） | batch size | Enable sp | Train_sp_batch_size | Speed | Step per day |
-|---|---|---|---|---|---|
-|8|8|×|-|100s/step|~850|
-|8|-|√|4|53s/step|~1600|
-|8|-|√|2|27s/step|~3200|
-
-**Inference on 93×720p**, we report speed on H100.
-
-| Size | 1 GPU | 8 GPUs |
-|---|---|---|
-|29×720p|420s/100step|80s/100step|
-|93×720p|3400s/100step|450s/100step|
-
-#### Dynamic training
-
-Deep neural networks are typically trained using batched inputs. For efficient hardware processing, batch shapes are fixed, leading to a fixed data size. This requires either cropping or padding images to a uniform size, both of which have drawbacks: cropping degrades performance, while padding is inefficient and results in significant information loss. Generally, there are three methods for training with arbitrary token counts: Patch n' Pack, bucket, and pad-mask.
-
-
-
-**Patch n' Pack** ([NaViT](https://arxiv.org/abs/2307.06304)): bypasses the fixed sequence length limitation by combining tokens from multiple samples into a new sample. This approach allows variable-resolution images while maintaining aspect ratios by packaging multiple samples together, thereby reducing training time and enhancing performance and flexibility. However, this method involves significant code modifications and requires re-adaptation when exploring different model architectures in fields with unstable model designs.
-
-
-**Bucket** ([Pixart-alpha](https://arxiv.org/abs/2310.00426), [Open-Sora](https://github.com/hpcaitech/Open-Sora)): This method packages data of different resolutions into buckets, sampling batches from each bucket to ensure same resolution within each batch. It requires minimal code modifications to the model, mainly adjusting the data sampling strategy.
-
-**Pad-mask** ([FiT](https://arxiv.org/abs/2402.12376), our v1.0/v1.1): This method sets a maximum resolution and pads all data to this resolution, generating a corresponding mask. Although the approach is straightforward, it is computationally inefficient.
-
-We believe that current video generation models are still in an exploratory phase. Extensive modifications to model code during this period can incur unnecessary development costs. The pad-mask method, while straightforward, is computationally inefficient and can waste resources in video, which involves dense computations. Ultimately, we chose the bucket strategy, which requires no modifications to the model code. Next, we will explain how our bucket strategy supports arbitrary lengths and resolutions. For simplicity, we will use video duration as an example:
-
-
-
-
-We define a megabatch as the total data processed in a single step across all GPUs. A megabatch can be divided into multiple batches, with each batch corresponding to the data processed by a single GPU.
-
-**Sort by frame**: The first step is to count the number of frames in all video data and sort them. This step aims to group similar data together, with sorting being one method to achieve this.
-
-**Group megabatch**: Next, all data is divided into groups, each forming a megabatch. Since all data is pre-sorted, most videos within a megabatch have the same number of frames. However, there will always be boundary cases, such as having both 61-frame and 1-frame videos in a single megabatch.
-
-**Re-organize megabatch**: We re-organize these special megabatches, which actually constitute a small proportion. We randomly replace the minority data in the megabatch with the majority data, thus re-organizing it into a megabatch with same frame counts.
-
-**Shuffle megabatch**: To ensure data randomness, we shuffle both within each megabatch and between different megabatches.
-
-When supporting dynamic resolutions, we simply replace each sample's frame sequence with (frame × height × width). This method ensures that the data dimension processed by each GPU in every step is the same, preventing situations where GPU1 waits for GPU0 to finish processing a longer video. Moreover, it is entirely decoupled from the model code, serving as a plug-and-play video sampling strategy.
-
-
-#### Training stage
-
-Similar to previous work, we use a multi-stage training approach. With the 3D DiT architecture, all parameters can be transferred from images to videos without loss. To explore training costs, all parameters of the diffusion model are trained from scratch. Therefore, we first train an text-to-image model, using the training strategy from [Pixart-alpha](https://arxiv.org/abs/2310.00426).
-
-The video model is initialized with weights from a 480p image model. We first train 480p videos with 29 frames. Next, we adapt the weights to 720p resolution, training on approximately 6 million higher-quality (HQ) samples from Panda70M, filtered for aesthetic quality and motion. Finally, we refine the model with a more higher-quality (HQ) subset of 1 million samples. After that, we use a filtered data (collected in v1.1.0) for fine-tuning 93-frame 720p videos. Below is our training card. We release the annotation file [here](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/anno_json).
-
-| Name | Stage 1 | Stage 2 | Stage 3 | Stage 4 |Stage 5 |
-|---|---|---|---|---|---|
-| Training Video Size | 1×320×240 | 1×640×480 | 29×640×480 | 29×1280×720 | 93×1280×720 |
-| Training Step| 146k | 200k | 30k | 21k | 3k |
-| Compute (#Num x #Hours) | 32 Ascend × 81 | 32 Ascend × 142 | 128 Ascend × 38 | 256 H100 × 64 | 256 H100 × 84 |
-| Checkpoint | - | [HF](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/1x480p) | [HF](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/29x480p) | [HF](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/29x720p) | [HF](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/93x720p) |
-| Log | - | - | [wandb](https://api.wandb.ai/links/1471742727-Huawei/trdu2kba) | [wandb](https://api.wandb.ai/links/linbin/vvxvcd7s) | [wandb](https://api.wandb.ai/links/linbin/easg3qkl)
-| Training Data | [10M SAM](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.2.0/blob/main/anno_json/sam_image_11185255_resolution.json) | 5M internal image data | [6M HQ Panda70M](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.2.0/blob/main/anno_json/Panda70M_HQ6M.json) | [6M HQ Panda70M](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.2.0/blob/main/anno_json/Panda70M_HQ6M.json) | [1M HQ Panda70M](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.2.0/blob/main/anno_json/Panda70M_HQ1M.json) and [100k HQ data](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/anno_json) (collected in v1.1.0) |
-
-Additionally, we fine-tuned 3.5k steps from the final 93×720p to get [93×480p](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/93x480p) for community research use.
-
-### Training Image-to-Video Diffusion Model
-
-#### Model Structure
-
-
-
-To reuse the weights of the Text-to-Video model, our Image-to-Video model is inspired by the Stable Diffusion Inpainting Model and adopts a strategy based on frame-level inpainting. By incorporating three types of information—original noise, masked video, and mask—under different control frame conditions, our model can generate coherent videos while ensuring flexibility in its usage.
-
-Compared to the denoiser structure of the Text-to-Video model, the Inpainting Model's denoiser has only changed the number of channels in the `conv in` layer. To ensure the model has a good prior knowledge, we introduce the masked video and mask information through zero initialization. We believe this is due to the 2+1D structure's lack of ability to establish long-range information dependencies, and relying solely on attention in the temporal dimension makes it difficult to capture information changes under frame control. In Text-to-Video tasks, this phenomenon is not as evident because all frames share the same text prompt embedding. However, in Image-to-Video tasks, simply concatenating images in the channel dimension does not ensure the model can accurately capture changes between frames. This is because the model cannot directly replicate image information from the channels to reduce the loss, and the 2+1D structure's interaction solely on the temporal axis fails to allow the model to discern which information from the control frames can be utilized, especially there are significant differences between frames. Therefore, without a shared image-semantic information, the control frame information might not be effectively conveyed to each frame.
-
-##### About Semantic Adapter
-
-In previous models based on the Unet 2+1D architecture, it is necessary to input the control frames into the CLIP model to obtain semantic embeddings. These semantic embeddings are then injected into the denoiser through cross-attention. The structure that extracts CLIP embeddings and injects them into the denoiser is commonly referred to as a semantic adapter.
-
-In the 2+1D architecture, the semantic adapter is commonly present. Additionally, papers like [DynamiCrafter](https://arxiv.org/abs/2310.12190) have pointed out that incorporating the semantic adapter helps maintain stability in the generated videos. We believe this is because the 2+1D structure lacks the ability to establish long-range information dependencies, and relying solely on attention in the temporal dimension makes it difficult to capture information changes under frame control. In the Text-to-Video task, this phenomenon is not as evident because all frames share the same text prompt embedding. However, in the Image-to-Video task, without shared semantic information, it may lead to the inability to effectively transfer control frame information to each individual frame.
-
-
-
-
-
-
-
-
-We conducted a simple comparison of the performance of using the Inpainting Model under the 2+1D structure (Open-Sora Plan v1.1, left in the figure) versus the 3D structure (Open-Sora Plan v1.2, right in the figure). With the same number of optimization steps, the probability of unstable visual performance in the 2+1D structure was significantly higher than in the 3D structure. Even at convergence, the 2+1D structure's visual stability was still inferior to that of the 3D structure, and it was even worse than the early training stages of the 3D structure.
-
-## Future Work and Discussion
-
-#### CausalVideoVAE
-We observed that high-frequency motion information in videos tends to exhibit jitter, and increasing training duration and data volume does not significantly alleviate this issue. In videos, compressing the duration while maintaining the original latent dimension can lead to significant information loss. A more robust VAE will be released in the next version.
-
-#### Diffusion Model
-We replaced T5 with mT5 to enhance multilingual capabilities, but this capability is limited as our training data is currently only in English. The multilingual ability primarily comes from the mT5 mapping space. We will explore additional text encoders and expand the data in the next steps.
-
-Our model performs well in generating character consistency, likely due to panda70m being a character-centric dataset. However, it still shows poor performance in text consistency and object generalization. We suspect this may be due to the limited amount of data the model has seen, as evidenced by the non-convergence of the loss in the final stage. **We hope to collaborate with the open-source community to optimize the 3D DiT architecture.**
diff --git a/examples/conditional_images_path.txt b/examples/conditional_images_path.txt
deleted file mode 100644
index e8159a9aa..000000000
--- a/examples/conditional_images_path.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-/path/to/image_0.png
-/path/to/image_1_0.png,/path/to/image_1_1.png
diff --git a/examples/get_latents_std.py b/examples/get_latents_std.py
deleted file mode 100644
index 5f9937024..000000000
--- a/examples/get_latents_std.py
+++ /dev/null
@@ -1,38 +0,0 @@
-import torch
-from torch.utils.data import DataLoader, Subset
-import sys
-sys.path.append(".")
-from opensora.models.ae.videobase import CausalVAEModel, CausalVAEDataset
-
-num_workers = 4
-batch_size = 12
-
-torch.manual_seed(0)
-torch.set_grad_enabled(False)
-
-device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
-
-pretrained_model_name_or_path = 'results/causalvae/checkpoint-26000'
-data_path = '/remote-home1/dataset/UCF-101'
-video_num_frames = 17
-resolution = 128
-sample_rate = 10
-
-vae = CausalVAEModel.load_from_checkpoint(pretrained_model_name_or_path)
-vae.to(device)
-
-dataset = CausalVAEDataset(data_path, sequence_length=video_num_frames, resolution=resolution, sample_rate=sample_rate)
-subset_indices = list(range(1000))
-subset_dataset = Subset(dataset, subset_indices)
-loader = DataLoader(subset_dataset, batch_size=8, pin_memory=True)
-
-all_latents = []
-for video_data in loader:
- video_data = video_data['video'].to(device)
- latents = vae.encode(video_data).sample()
- all_latents.append(video_data.cpu())
-
-all_latents_tensor = torch.cat(all_latents)
-std = all_latents_tensor.std().item()
-normalizer = 1 / std
-print(f'{normalizer = }')
\ No newline at end of file
diff --git a/examples/prompt_list_0.txt b/examples/prompt_list_0.txt
deleted file mode 100644
index 92fb551c6..000000000
--- a/examples/prompt_list_0.txt
+++ /dev/null
@@ -1,20 +0,0 @@
-A young man at his 20s is sitting on a piece of cloud in the sky, reading a book.
-An extreme close-up of an gray-haired man with a beard in his 60s, he is deep in thought pondering the history of the universe as he sits at a cafe in Paris, his eyes focus on people offscreen as they walk as he sits mostly motionless, he is dressed in a wool coat suit coat with a button-down shirt, he wears a brown beret and glasses and has a very professorial appearance, and the end he offers a subtle closed-mouth smile as if he found the answer to the mystery of life, the lighting is very cinematic with the golden light and the Parisian streets and city in the background, depth of field, cinematic 35mm film.
-A close-up of a woman’s face, illuminated by the soft light of dawn, her expression serene and content as she wakes up in a cozy bedroom.
-An intense close-up of a detective’s face, lit by a single desk lamp, his eyes scanning a wall covered in photos and notes, deep in thought.
-Audience members in a theater are captured in a series of medium shots, with a young man and woman in formal attire centrally positioned and illuminated by a spotlight effect.
-A soaring drone footage captures the majestic beauty of a coastal cliff, its red and yellow stratified rock faces rich in color and against the vibrant turquoise of the sea. Seabirds can be seen taking flight around the cliff's precipices. As the drone slowly moves from different angles, the changing sunlight casts shifting shadows that highlight the rugged textures of the cliff and the surrounding calm sea. The water gently laps at the rock base and the greenery that clings to the top of the cliff, and the scene gives a sense of peaceful isolation at the fringes of the ocean. The video captures the essence of pristine natural beauty untouched by human structures.
-a realistic 3d rendering of a female character with curly blonde hair and blue eyes. she is wearing a black tank top and has a neutral expression while facing the camera directly. the background is a plain blue sky, and the scene is devoid of any other objects or text. the character is detailed, with realistic textures and lighting, suitable for a video game or high-quality animation. there is no movement or additional action in the video. the focus is entirely on the character's appearance and realistic rendering.
-A panda strumming a guitar under a bamboo grove, its paws gently plucking the strings as a group of mesmerized rabbits watch, the music blending with the rustle of bamboo leaves. HD.
-A close-up of a woman with a vintage hairstyle and bright red lipstick, gazing seductively into the camera, the background blurred to keep the focus solely on her.
-In the jungle, a hidden temple stands guarded by statues of lions, their eyes glowing with emerald light, protecting secrets untold for millennia. 8K.
-A close-up of an old man’s weathered face, with deep wrinkles and a thick white mustache, looking out to sea, the wind gently blowing through his hair.
-An intense close-up of a soldier’s face, covered in dirt and sweat, his eyes filled with determination as he surveys the battlefield.
-A river that flows uphill, defying gravity as it returns lost treasures from the sea to the mountain top, each item telling a story of a voyage gone by. HD.
-A close-up of a man’s face, lit only by the glow of his computer screen, his eyes wide and unblinking as he discovers something shocking online.
-On a deserted island, palm trees sway to summon a rainstorm, their leaves conducting the wind like maestros, orchestrating a symphony of thunder and lightning. High Resolution.
-An extreme close-up of a middle-aged man’s face, with a five o’clock shadow, staring pensively into the distance as rain softly taps against the window beside him, his thoughts deep and contemplative.
-A close-up of a man’s face, his expression one of deep concentration as he works on a complex task.
-Drone view of waves crashing against the rugged cliffs along Big Sur's garay point beach.The crashing blue waters create white-tipped waves,while the golden light of the setting sun illuminates the rocky shore. A small island with a lighthouse sits in the distance, and green
-shrubbery covers the cliffs edge. The steep drop from the road down to the beach is adramatic feat, with the cliff's edges jutting out over the sea. This is a view that captures the raw beauty of the coast and the rugged landscape of the Pacific Coast Highway.
-a close-up shot of a woman standing in a dimly lit room. she is wearing a traditional chinese outfit, which includes a red and gold dress with intricate designs and a matching headpiece. the woman has her hair styled in an updo, adorned with a gold accessory. her makeup is done in a way that accentuates her features, with red lipstick and dark eyeshadow. she is looking directly at the camera with a neutral expression. the room has a rustic feel, with wooden beams and a stone wall visible in the background. the lighting in the room is soft and warm, creating a contrast with the woman's vibrant attire. there are no texts or other objects in the video. the style of the video is a portrait, focusing on the woman and her attire.
\ No newline at end of file
diff --git a/examples/rec_image.py b/examples/rec_image.py
deleted file mode 100644
index ad869eb04..000000000
--- a/examples/rec_image.py
+++ /dev/null
@@ -1,67 +0,0 @@
-import sys
-from PIL import Image
-import torch
-from torchvision.transforms import ToTensor, Compose, Resize, Normalize, Lambda
-from torch.nn import functional as F
-import argparse
-import numpy as np
-from opensora.models import CausalVAEModelWrapper
-
-def preprocess(video_data: torch.Tensor, short_size: int = 128) -> torch.Tensor:
- transform = Compose(
- [
- ToTensor(),
- Lambda(lambda x: 2. * x - 1.),
- Resize(size=short_size),
- ]
- )
- outputs = transform(video_data)
- outputs = outputs.unsqueeze(0).unsqueeze(2)
- return outputs
-
-def main(args: argparse.Namespace):
- image_path = args.image_path
- short_size = args.short_size
- device = args.device
- kwarg = {}
-
- # vae = getae_wrapper(args.ae)(args.model_path, subfolder="vae", cache_dir='cache_dir', **kwarg).to(device)
- vae = CausalVAEModelWrapper(args.ae_path, **kwarg).to(device)
- if args.enable_tiling:
- vae.vae.enable_tiling()
- vae.vae.tile_overlap_factor = args.tile_overlap_factor
- vae.eval()
- vae = vae.to(device)
- vae = vae.half()
-
- with torch.no_grad():
- x_vae = preprocess(Image.open(image_path), short_size)
- x_vae = x_vae.to(device, dtype=torch.float16) # b c t h w
- latents = vae.encode(x_vae)
- latents = latents.to(torch.float16)
- image_recon = vae.decode(latents) # b t c h w
- x = image_recon[0, 0, :, :, :]
- x = x.squeeze()
- x = x.detach().cpu().numpy()
- x = np.clip(x, -1, 1)
- x = (x + 1) / 2
- x = (255*x).astype(np.uint8)
- x = x.transpose(1,2,0)
- image = Image.fromarray(x)
- image.save(args.rec_path)
-
-
-if __name__ == '__main__':
- parser = argparse.ArgumentParser()
- parser.add_argument('--image_path', type=str, default='')
- parser.add_argument('--rec_path', type=str, default='')
- parser.add_argument('--ae', type=str, default='')
- parser.add_argument('--ae_path', type=str, default='')
- parser.add_argument('--model_path', type=str, default='results/pretrained')
- parser.add_argument('--short_size', type=int, default=336)
- parser.add_argument('--device', type=str, default='cuda')
- parser.add_argument('--tile_overlap_factor', type=float, default=0.25)
- parser.add_argument('--enable_tiling', action='store_true')
-
- args = parser.parse_args()
- main(args)
diff --git a/examples/rec_video.py b/examples/rec_video.py
deleted file mode 100644
index 06ee2b0d2..000000000
--- a/examples/rec_video.py
+++ /dev/null
@@ -1,131 +0,0 @@
-import math
-import random
-import argparse
-from typing import Optional
-
-import cv2
-import numpy as np
-import numpy.typing as npt
-import torch
-from PIL import Image
-from decord import VideoReader, cpu
-from torch.nn import functional as F
-from pytorchvideo.transforms import ShortSideScale
-from torchvision.transforms import Lambda, Compose
-from opensora.models import CausalVAEModelWrapper
-from opensora.dataset.transform import ToTensorVideo, CenterCropResizeVideo
-
-
-def array_to_video(image_array: npt.NDArray, fps: float = 30.0, output_file: str = 'output_video.mp4') -> None:
- height, width, channels = image_array[0].shape
- fourcc = cv2.VideoWriter_fourcc(*'mp4v')
- video_writer = cv2.VideoWriter(output_file, fourcc, float(fps), (width, height))
-
- for image in image_array:
- image_rgb = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
- video_writer.write(image_rgb)
-
- video_writer.release()
-
-
-def custom_to_video(x: torch.Tensor, fps: float = 2.0, output_file: str = 'output_video.mp4') -> None:
- x = x.detach().cpu()
- x = torch.clamp(x, -1, 1)
- x = (x + 1) / 2
- x = x.permute(0, 2, 3, 1).numpy()
- x = (255 * x).astype(np.uint8)
- array_to_video(x, fps=fps, output_file=output_file)
- return
-
-
-def read_video(video_path: str, num_frames: int, sample_rate: int) -> torch.Tensor:
- decord_vr = VideoReader(video_path, ctx=cpu(0))
- total_frames = len(decord_vr)
- sample_frames_len = sample_rate * num_frames
-
- if total_frames > sample_frames_len:
- s = random.randint(0, total_frames - sample_frames_len - 1)
- s = 0
- e = s + sample_frames_len
- num_frames = num_frames
- else:
- s = 0
- e = total_frames
- num_frames = int(total_frames / sample_frames_len * num_frames)
- print(f'sample_frames_len {sample_frames_len}, only can sample {num_frames * sample_rate}', video_path,
- total_frames)
-
- frame_id_list = np.linspace(s, e - 1, num_frames, dtype=int)
- video_data = decord_vr.get_batch(frame_id_list).asnumpy()
- video_data = torch.from_numpy(video_data)
- video_data = video_data.permute(3, 0, 1, 2) # (T, H, W, C) -> (C, T, H, W)
- return video_data
-
-
-def preprocess(video_data: torch.Tensor, height: int = 128, width: int = 128) -> torch.Tensor:
- transform = Compose(
- [
- ToTensorVideo(),
- CenterCropResizeVideo((height, width)),
- Lambda(lambda x: 2. * x - 1.)
- ]
- )
-
- video_outputs = transform(video_data)
- video_outputs = torch.unsqueeze(video_outputs, 0)
-
- return video_outputs
-
-
-def main(args: argparse.Namespace):
- device = args.device
- kwarg = {}
- # vae = getae_wrapper(args.ae)(args.model_path, subfolder="vae", cache_dir='cache_dir', **kwarg).to(device)
- vae = CausalVAEModelWrapper(args.ae_path, **kwarg).to(device)
- if args.enable_tiling:
- vae.vae.enable_tiling()
- vae.vae.tile_overlap_factor = args.tile_overlap_factor
- vae.vae.tile_overlap_factor = args.tile_overlap_factor
- if args.save_memory:
- vae.vae.tile_sample_min_size = 256
- vae.vae.tile_latent_min_size = 32
- vae.vae.tile_sample_min_size_t = 29
- vae.vae.tile_latent_min_size_t = 8
- vae.eval()
- vae = vae.to(device)
- vae = vae.half()
-
- with torch.no_grad():
- x_vae = preprocess(read_video(args.video_path, args.num_frames, args.sample_rate), args.height,
- args.width)
- print(x_vae.shape)
- x_vae = x_vae.to(device, dtype=torch.float16) # b c t h w
- latents = vae.encode(x_vae)
- print(latents.shape)
- latents = latents.to(torch.float16)
- video_recon = vae.decode(latents) # b t c h w
- print(video_recon.shape)
-
- custom_to_video(video_recon[0], fps=args.fps, output_file=args.rec_path)
-
-
-if __name__ == '__main__':
- parser = argparse.ArgumentParser()
- parser.add_argument('--video_path', type=str, default='')
- parser.add_argument('--rec_path', type=str, default='')
- parser.add_argument('--ae', type=str, default='')
- parser.add_argument('--ae_path', type=str, default='')
- parser.add_argument('--model_path', type=str, default='results/pretrained')
- parser.add_argument('--fps', type=int, default=30)
- parser.add_argument('--height', type=int, default=336)
- parser.add_argument('--width', type=int, default=336)
- parser.add_argument('--num_frames', type=int, default=100)
- parser.add_argument('--sample_rate', type=int, default=1)
- parser.add_argument('--device', type=str, default="cuda")
- parser.add_argument('--tile_overlap_factor', type=float, default=0.25)
- parser.add_argument('--tile_sample_min_size', type=int, default=256)
- parser.add_argument('--enable_tiling', action='store_true')
- parser.add_argument('--save_memory', action='store_true')
-
- args = parser.parse_args()
- main(args)
diff --git a/pyproject.toml b/pyproject.toml
index 431e0c32e..d66684424 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -13,7 +13,6 @@ classifiers = [
"License :: OSI Approved :: Apache Software License",
]
dependencies = [
- "torch==2.1.0", "torchvision==0.16.0",
"transformers==4.40.1", "accelerate==0.29.3", "tokenizers==0.19.1", "diffusers==0.28.0",
"albumentations==1.4.0", "av==11.0.0", "decord==0.6.0", "einops==0.7.0", "fastapi==0.110.0",
"gdown==5.1.0", "h5py==3.10.0", "idna==3.6", 'imageio==2.34.0', "matplotlib==3.7.5", "numpy==1.24.4",
diff --git a/tools/get_img_info.py b/tools/get_img_info.py
deleted file mode 100644
index 0617f02b1..000000000
--- a/tools/get_img_info.py
+++ /dev/null
@@ -1,51 +0,0 @@
-
-import cv2
-from tqdm import tqdm
-from glob import glob
-import json
-import os
-from multiprocessing import Pool
-import argparse
-
-def get_image_size(image_path):
- """
- Given an image path, return its width and height.
- """
- try:
- image = cv2.imread(image_path)
- height, width = image.shape[:2]
- return image_path, height, width
- except Exception as e:
- return image_path, None, None
-
-def process_image_paths(image_paths):
- items = []
- for image_path, height, width in image_paths:
- path = image_path.replace(image_root if image_root.endswith('/') else image_root + '/', '')
- item = dict(path=path, resolution=dict(height=height, width=width))
- items.append(item)
- return items
-
-if __name__ == '__main__':
- parser = argparse.ArgumentParser()
- parser.add_argument("--image_root", type=str, required=True)
- parser.add_argument("--save_root", type=str, required=True)
- parser.add_argument("--save_name", type=str, required=True)
- args = parser.parse_args()
- # image_root = '/storage/dataset/xx/xx_v1'
- # save_root = '/storage/dataset/image'
- image_root = args.image_root
- save_root = args.save_root
- os.makedirs(save_root, exist_ok=True)
- save_name = '{}_{}_resolution.json'
- all_paths = glob(os.path.join(image_root, '**', '*.jpg'), recursive=True)
-
- num_processes = os.cpu_count() # Use the number of CPU cores
- num_processes = 128 # Use the number of CPU cores
- with Pool(num_processes) as pool:
- results = list(tqdm(pool.imap(get_image_size, all_paths), total=len(all_paths)))
-
- items = process_image_paths(results)
-
- with open(os.path.join(save_root, save_name.format(args.save_name, len(items))), 'w') as f:
- json.dump(items, f, indent=2)
diff --git a/tools/merge_imginfo_to_anno.py b/tools/merge_imginfo_to_anno.py
deleted file mode 100644
index 32d3a1220..000000000
--- a/tools/merge_imginfo_to_anno.py
+++ /dev/null
@@ -1,42 +0,0 @@
-
-from tqdm import tqdm
-from glob import glob
-import json
-import os
-
-
-
-if __name__ == '__main__':
-
- parser = argparse.ArgumentParser()
- parser.add_argument("--anno_path", type=str, required=True)
- parser.add_argument("--img_info_path", type=str, required=True)
- parser.add_argument("--save_root", type=str, required=True)
- parser.add_argument("--save_name", type=str, required=True)
- args = parser.parse_args()
-
- # anno_path = '/storage/anno_jsons/sam_1940032.json'
- # img_info_path = '/storage/dataset/image/sam_2151074_resolution.json'
- # save_root = '/storage/anno_jsons'
- # save_name = 'sam_{}_resolution.json'
-
- anno_path = args.anno_path
- img_info_path = args.img_info_path
- save_root = args.save_root
- save_name = '{}_{}_resolution.json'
-
-
- with open(anno_path, 'r') as f:
- anno = json.load(f)
- with open(img_info_path, 'r') as f:
- img_info = json.load(f)
- img_info = {i['path']: i['resolution'] for i in img_info}
-
- items = []
- cnt = 0
- for i in tqdm(anno):
- resolution = img_info[i['path']]
- i['resolution'] = resolution
- items.append(i)
- with open(os.path.join(save_root, save_name.format(args.save_name, len(items))), 'w') as f:
- json.dump(items, f, indent=2)
\ No newline at end of file
-
-
-
-#### Training Details
-
-
-As with v1.1.0, we initialize from the [SD2.1 VAE](https://huggingface.co/stabilityai/sd-vae-ft-mse) using tail initialization. We perform the first phase of training on the Kinetic400 video dataset, then use the EMA weights from this phase to initialize the second phase, which is fine-tuned on high-quality data (collected in v1.1.0). All training is conducted on 25-frame 256×256 videos using **one A100 node**.
-
-
-| Training stage | Dataset | Training steps |
-|---|---|---|
-| 1 | K400 | 200,000 |
-| 2 | collected in v1.1.0 | 450,000 |
-
-
-#### Evaluation
-
-We evaluated our VAE on the validation sets of two video datasets: [Webvid](https://github.com/m-bain/webvid) and [Panda70m](https://github.com/snap-research/Panda-70M/), and compared it with our [v1.1.0](https://github.com/PKU-YuanGroup/Open-Sora-Plan/blob/main/docs/Report-v1.1.0.md), [SD2.1 VAE](https://huggingface.co/stabilityai/sd-vae-ft-mse), [CV-VAE](https://github.com/AILab-CVC/CV-VAE), and [Open-Sora's VAE](https://github.com/hpcaitech/Open-Sora). The Webvid validation set contains 5k videos, while the Panda70m validation set has 6k videos. The videos were resized to 256 pixels on the short side, center-cropped to 256x256, and then 33 consecutive frames were extracted. We used PSNR, SSIM, and LPIPS metrics, and measured the encoding speed on an A100 GPU. The specific results are as follows:
-
-
-**WebVid**
-
-| Model | Compress Ratio |PNSR↑ | SSIM↑ |LPIPS↓ |
-|---|---|---|---|---|
-| SD2-1 VAE | 1x8x8 | 30.19 | 0.8379 | 0.0568 |
-| SVD VAE | 1x8x8 |31.15 |0.8686 | **0.0547** |
-| CV-VAE | 4x8x8 | 30.76 | 0.8566 | 0.0803 |
-| Open-Sora VAE | 4x8x8 | 31.12 | 0.8569 | 0.1003 |
-| Open-Sora Plan v1.1 | 4x8x8 | 30.26 | 0.8597 |0.0551 |
-| Open-Sora Plan v1.2 | 4x8x8| **31.16** | **0.8694** | 0.0586 |
-
-**Panda70M**
-
-| Model | Compress Ratio| PNSR↑ | SSIM↑ |LPIPS↓ |
-|---|---|---|---|---|
-| SD2-1 VAE | 1x8x8 |30.40 | 0.8894 | 0.0396 |
-| SVD VAE | 1x8x8 |31.00 | **0.9058** | **0.0379** |
-| CV-VAE | 4x8x8| 29.57 | 0.8795 | 0.0673 |
-| Open-Sora VAE | 4x8x8 | **31.06** | 0.8969 | 0.0666 |
-| Open-Sora Plan v1.1 | 4x8x8 | 29.16 | 0.8844 | 0.0481 |
-| Open-Sora Plan v1.2 | 4x8x8| 30.49 |0.8970 |0.0454|
-
-**Encode Time on A100**
-
-|Input Size| CV-VAE | Open-Sora | Open-Sora Plan v1.1 | Open-Sora Plan v1.2 |
-|---|---|---|---|---|
-| 33x256x256 | 0.186 | 0.147 |0.104 | **0.102** |
-| 81x256x256 | 0.465 | 0.357 |0.243 | **0.242** |
-
-### Training Text-to-Video Diffusion Model
-
-#### Model Structure
-
-The most significant change is that we **replaced all 2+1D Transformer blocks with 3D full attention blocks**. Each video is first processed by a patch embedding layer, which downsamples the spatial dimensions by a factor of 2. The video is then flattened into a one-dimensional sequence across the frame, width, and height dimensions. We replaced [T5-XXL](https://huggingface.co/DeepFloyd/t5-v1_1-xxl) with [mT5-XXL](https://huggingface.co/google/mt5-xxl) to enhance multilingual adaptation. Additionally, we incorporated RoPE.
-
-
-### Sequence Parallelism
-
-Due to the high computational complexity of 3D full attention, we must allocate a video across 2 GPUs for parallel processing when training with long-duration and high-resolution videos. We can control the number of GPUs used for a video sample by adjusting the batch size on a node. For example, with `sp_size=8` and `train_sp_batch_size=4`, 2 GPUs are used for a single sample. **We support sequence parallelism for both training and inference**.
-
-**Training on 93×720p**, we report speed on H100.
-
-| GPU （sp_size） | batch size | Enable sp | Train_sp_batch_size | Speed | Step per day |
-|---|---|---|---|---|---|
-|8|8|×|-|100s/step|~850|
-|8|-|√|4|53s/step|~1600|
-|8|-|√|2|27s/step|~3200|
-
-**Inference on 93×720p**, we report speed on H100.
-
-| Size | 1 GPU | 8 GPUs |
-|---|---|---|
-|29×720p|420s/100step|80s/100step|
-|93×720p|3400s/100step|450s/100step|
-
-#### Dynamic training
-
-Deep neural networks are typically trained using batched inputs. For efficient hardware processing, batch shapes are fixed, leading to a fixed data size. This requires either cropping or padding images to a uniform size, both of which have drawbacks: cropping degrades performance, while padding is inefficient and results in significant information loss. Generally, there are three methods for training with arbitrary token counts: Patch n' Pack, bucket, and pad-mask.
-
-
-
-**Patch n' Pack** ([NaViT](https://arxiv.org/abs/2307.06304)): bypasses the fixed sequence length limitation by combining tokens from multiple samples into a new sample. This approach allows variable-resolution images while maintaining aspect ratios by packaging multiple samples together, thereby reducing training time and enhancing performance and flexibility. However, this method involves significant code modifications and requires re-adaptation when exploring different model architectures in fields with unstable model designs.
-
-
-**Bucket** ([Pixart-alpha](https://arxiv.org/abs/2310.00426), [Open-Sora](https://github.com/hpcaitech/Open-Sora)): This method packages data of different resolutions into buckets, sampling batches from each bucket to ensure same resolution within each batch. It requires minimal code modifications to the model, mainly adjusting the data sampling strategy.
-
-**Pad-mask** ([FiT](https://arxiv.org/abs/2402.12376), our v1.0/v1.1): This method sets a maximum resolution and pads all data to this resolution, generating a corresponding mask. Although the approach is straightforward, it is computationally inefficient.
-
-We believe that current video generation models are still in an exploratory phase. Extensive modifications to model code during this period can incur unnecessary development costs. The pad-mask method, while straightforward, is computationally inefficient and can waste resources in video, which involves dense computations. Ultimately, we chose the bucket strategy, which requires no modifications to the model code. Next, we will explain how our bucket strategy supports arbitrary lengths and resolutions. For simplicity, we will use video duration as an example:
-
-
-
-
-We define a megabatch as the total data processed in a single step across all GPUs. A megabatch can be divided into multiple batches, with each batch corresponding to the data processed by a single GPU.
-
-**Sort by frame**: The first step is to count the number of frames in all video data and sort them. This step aims to group similar data together, with sorting being one method to achieve this.
-
-**Group megabatch**: Next, all data is divided into groups, each forming a megabatch. Since all data is pre-sorted, most videos within a megabatch have the same number of frames. However, there will always be boundary cases, such as having both 61-frame and 1-frame videos in a single megabatch.
-
-**Re-organize megabatch**: We re-organize these special megabatches, which actually constitute a small proportion. We randomly replace the minority data in the megabatch with the majority data, thus re-organizing it into a megabatch with same frame counts.
-
-**Shuffle megabatch**: To ensure data randomness, we shuffle both within each megabatch and between different megabatches.
-
-When supporting dynamic resolutions, we simply replace each sample's frame sequence with (frame × height × width). This method ensures that the data dimension processed by each GPU in every step is the same, preventing situations where GPU1 waits for GPU0 to finish processing a longer video. Moreover, it is entirely decoupled from the model code, serving as a plug-and-play video sampling strategy.
-
-
-#### Training stage
-
-Similar to previous work, we use a multi-stage training approach. With the 3D DiT architecture, all parameters can be transferred from images to videos without loss. To explore training costs, all parameters of the diffusion model are trained from scratch. Therefore, we first train an text-to-image model, using the training strategy from [Pixart-alpha](https://arxiv.org/abs/2310.00426).
-
-The video model is initialized with weights from a 480p image model. We first train 480p videos with 29 frames. Next, we adapt the weights to 720p resolution, training on approximately 6 million higher-quality (HQ) samples from Panda70M, filtered for aesthetic quality and motion. Finally, we refine the model with a more higher-quality (HQ) subset of 1 million samples. After that, we use a filtered data (collected in v1.1.0) for fine-tuning 93-frame 720p videos. Below is our training card. We release the annotation file [here](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/anno_json).
-
-| Name | Stage 1 | Stage 2 | Stage 3 | Stage 4 |Stage 5 |
-|---|---|---|---|---|---|
-| Training Video Size | 1×320×240 | 1×640×480 | 29×640×480 | 29×1280×720 | 93×1280×720 |
-| Training Step| 146k | 200k | 30k | 21k | 3k |
-| Compute (#Num x #Hours) | 32 Ascend × 81 | 32 Ascend × 142 | 128 Ascend × 38 | 256 H100 × 64 | 256 H100 × 84 |
-| Checkpoint | - | [HF](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/1x480p) | [HF](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/29x480p) | [HF](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/29x720p) | [HF](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/93x720p) |
-| Log | - | - | [wandb](https://api.wandb.ai/links/1471742727-Huawei/trdu2kba) | [wandb](https://api.wandb.ai/links/linbin/vvxvcd7s) | [wandb](https://api.wandb.ai/links/linbin/easg3qkl)
-| Training Data | [10M SAM](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.2.0/blob/main/anno_json/sam_image_11185255_resolution.json) | 5M internal image data | [6M HQ Panda70M](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.2.0/blob/main/anno_json/Panda70M_HQ6M.json) | [6M HQ Panda70M](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.2.0/blob/main/anno_json/Panda70M_HQ6M.json) | [1M HQ Panda70M](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.2.0/blob/main/anno_json/Panda70M_HQ1M.json) and [100k HQ data](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/anno_json) (collected in v1.1.0) |
-
-Additionally, we fine-tuned 3.5k steps from the final 93×720p to get [93×480p](https://huggingface.co/LanguageBind/Open-Sora-Plan-v1.2.0/tree/main/93x480p) for community research use.
-
-### Training Image-to-Video Diffusion Model
-
-#### Model Structure
-
-
-
-To reuse the weights of the Text-to-Video model, our Image-to-Video model is inspired by the Stable Diffusion Inpainting Model and adopts a strategy based on frame-level inpainting. By incorporating three types of information—original noise, masked video, and mask—under different control frame conditions, our model can generate coherent videos while ensuring flexibility in its usage.
-
-Compared to the denoiser structure of the Text-to-Video model, the Inpainting Model's denoiser has only changed the number of channels in the `conv in` layer. To ensure the model has a good prior knowledge, we introduce the masked video and mask information through zero initialization. We believe this is due to the 2+1D structure's lack of ability to establish long-range information dependencies, and relying solely on attention in the temporal dimension makes it difficult to capture information changes under frame control. In Text-to-Video tasks, this phenomenon is not as evident because all frames share the same text prompt embedding. However, in Image-to-Video tasks, simply concatenating images in the channel dimension does not ensure the model can accurately capture changes between frames. This is because the model cannot directly replicate image information from the channels to reduce the loss, and the 2+1D structure's interaction solely on the temporal axis fails to allow the model to discern which information from the control frames can be utilized, especially there are significant differences between frames. Therefore, without a shared image-semantic information, the control frame information might not be effectively conveyed to each frame.
-
-##### About Semantic Adapter
-
-In previous models based on the Unet 2+1D architecture, it is necessary to input the control frames into the CLIP model to obtain semantic embeddings. These semantic embeddings are then injected into the denoiser through cross-attention. The structure that extracts CLIP embeddings and injects them into the denoiser is commonly referred to as a semantic adapter.
-
-In the 2+1D architecture, the semantic adapter is commonly present. Additionally, papers like [DynamiCrafter](https://arxiv.org/abs/2310.12190) have pointed out that incorporating the semantic adapter helps maintain stability in the generated videos. We believe this is because the 2+1D structure lacks the ability to establish long-range information dependencies, and relying solely on attention in the temporal dimension makes it difficult to capture information changes under frame control. In the Text-to-Video task, this phenomenon is not as evident because all frames share the same text prompt embedding. However, in the Image-to-Video task, without shared semantic information, it may lead to the inability to effectively transfer control frame information to each individual frame.
-
-