Peiyuan Zhang
5680039dbe
Update dependencies; Setup code for debugging Delete unused files and code Remove all npu code Update torchvision imports Update EMA model and t2v_debug.sh script Delete npu related stuff and remove inpaint module Remove compress kv Fix warning with dataset handling and model loading Update PyTorch index URLs and video length tolerance range Remove UDIT and inpaint fix typo for dataset download include pretrained open-sora Add pretrained model for OpenSoraT2V-ROPE-L Update max height and width for video processing
Fast Video
This is currently based on Open-Sora-1.2.0: https://github.com/PKU-YuanGroup/Open-Sora-Plan/tree/294993ca78bf65dec1c3b6fb25541432c545eda9
Envrironment
Change the index-url cuda version according to your system.
conda create -n fastvideo python=3.10.12
conda activate fastvideo
pip3 install torch==2.5.0 torchvision==0.2.0 --index-url https://download.pytorch.org/whl/cu121
pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu121
pip install ninja
pip install flash-attn --no-build-isolation
pip install -e .
pip install -e ".[train]"
Prepare Data & Models
We've prepared some debug data to facilitate development. To make sure the training pipeline is correct, train on the debug data and make sure the model overfit on it (feed it the same text prompt and see if the output video is the same as the training data)
python scripts/download_hf.py --repo_id=Stealths-Video/dummyVid --local_dir=data/dummyVid --repo_type=dataset
python scripts/download_hf.py --repo_id=LanguageBind/Open-Sora-Plan-v1.2.0 --file_name vae/checkpoint.ckpt --local_dir=data/Open-Sora-Plan-v1.2.0 --repo_type=model
python scripts/download_hf.py --repo_id=LanguageBind/Open-Sora-Plan-v1.2.0 --file_name 29x480p/config.json --local_dir=data/Open-Sora-Plan-v1.2.0 --repo_type=model
python scripts/download_hf.py --repo_id=LanguageBind/Open-Sora-Plan-v1.2.0 --file_name 29x480p/diffusion_pytorch_model.safetensors --local_dir=data/Open-Sora-Plan-v1.2.0 --repo_type=model
python scripts/download_hf.py --repo_id=LanguageBind/Open-Sora-Plan-v1.2.0 --file_name vae/config.json --local_dir=data/Open-Sora-Plan-v1.2.0 --repo_type=model
Debug Training
bash t2v_debug_single.sh
TODO
- Delete all npu related stuff.
- Remove inpaint.
- Create dummy debug data.
- Add Mochi
- Add Mochi VAE