Update README.md

Former-commit-id: be7ecc3d56f7289052fc66d535038e7111f7ac84

README.md

@@ -34,16 +34,41 @@ Project stages:
 
 ## Todo
 
-- [x] support variable aspect ratios, resolutions, durations training on DiT
-- [x] dynamic mask input
-- [x] add class-conditioning on embeddings
+### Setup the codebase and train a unconditional model on landscape dataset.
+- [x] Setup repo-structure
+- [x] Add Video-VQGAN model, which is borrowed from [VideoGPT](https://github.com/wilson1yan/VideoGPT)
+- [x] Support variable aspect ratios, resolutions, durations training on [Di](https://github.com/facebookresearch/DiT)
+- [x] Support Dynamic mask input inspired [FiT](https://github.com/whlzy/FiT)
+- [x] Add class-conditioning on embeddings
+- [ ] Incorporating [Latte](https://github.com/Vchitect/Latte) as main codebase
+- [ ] Add VAE model, which is borrowed from [Stable Diffusion](https://github.com/CompVis/latent-diffusion)
+- [ ] Joint dynamic mask input with VAE
+- [ ] Make the codebase ready for the cluster training. Add SLURM scripts.
+- [ ] Add sampling script
+- [ ] Incorporating [SiT](https://github.com/willisma/SiT)
 
-- [ ] sampling script
-- [ ] add positional interpolation
-- [ ] fine-tune Video-VQVAE on higher resolution
-- [ ] incorporating SiT
-- [ ] incorporating more conditions
-- [ ] training with more data and more GPU
+### Train models that boost resolution and duration.
+- [ ] Add [PI](https://arxiv.org/abs/2306.15595) to support out-of-domain size.
+- [ ] Add frame interpolation model
+
+### Conduct text2video experiments on landscape dataset.
+- [ ] Finish data loading, pre-processing utils.
+- [ ] Add CLIP and T5 support.
+- [ ] Add text2image training script.
+- [ ] Add prompt captioner.
+
+### Train the 1080p model on video2text dataset.
+- [ ] Finish data loading, pre-processing utils.
+- [ ] Support memory friendly training
+  - [ ] Add flash-attention2 from pytorch
+  - [ ] Add xformers
+  - [ ] Add accelerate to automatically manage training, e.g. mixed precision training.
+  - [ ] Add gradient checkpoint
+  - [ ] Train using the deepspeed engine.
+
+### Control model with more condition.
+- [ ] Load pretrained weight from [PixArt-α](https://github.com/PixArt-alpha/PixArt-alpha)
+- [ ] Incorporating [ControlNet](https://github.com/lllyasviel/ControlNet).
 
 ## Repo structure
 ```