72 lines
2.8 KiB
Markdown
72 lines
2.8 KiB
Markdown
(v0-finetune)=
|
|
# 🧠 Finetune
|
|
## ⚡ Full Finetune
|
|
Ensure your data is prepared and preprocessed in the format specified in [data_preprocess.md](#v0-data-preprocess). For convenience, we also provide a mochi preprocessed Black Myth Wukong data that can be downloaded directly:
|
|
|
|
```bash
|
|
python scripts/huggingface/download_hf.py --repo_id=FastVideo/Mochi-Black-Myth --local_dir=data/Mochi-Black-Myth --repo_type=dataset
|
|
```
|
|
|
|
Download the original model weights as specified in [Distill Section](#v0-distill):
|
|
|
|
Then you can run the finetune with:
|
|
|
|
```
|
|
bash scripts/finetune/finetune_mochi.sh # for mochi
|
|
```
|
|
|
|
**Note that for finetuning, we did not tune the hyperparameters in the provided script.**
|
|
## ⚡ Lora Finetune
|
|
|
|
Hunyuan supports Lora fine-tuning of videos up to 720p. Demos and prompts of Black-Myth-Wukong can be found in [here](https://huggingface.co/FastVideo/Hunyuan-Black-Myth-Wukong-lora-weight). You can download the Lora weight through:
|
|
|
|
```bash
|
|
python scripts/huggingface/download_hf.py --repo_id=FastVideo/Hunyuan-Black-Myth-Wukong-lora-weight --local_dir=data/Hunyuan-Black-Myth-Wukong-lora-weight --repo_type=model
|
|
```
|
|
|
|
### Minimum Hardware Requirement
|
|
- 40 GB GPU memory each for 2 GPUs with lora.
|
|
- 30 GB GPU memory each for 2 GPUs with CPU offload and lora.
|
|
|
|
Currently, both Mochi and Hunyuan models support Lora finetuning through diffusers. To generate personalized videos from your own dataset, you'll need to follow three main steps: dataset preparation, finetuning, and inference.
|
|
|
|
### Dataset Preparation
|
|
We provide scripts to better help you get started to train on your own characters!
|
|
You can run this to organize your dataset to get the videos2caption.json before preprocess. Specify your video folder and corresponding caption folder (caption files should be .txt files and have the same name with its video):
|
|
|
|
```
|
|
python scripts/dataset_preparation/prepare_json_file.py --video_dir data/input_videos/ --prompt_dir data/captions/ --output_path data/output_folder/videos2caption.json --verbose
|
|
```
|
|
|
|
Also, we provide script to resize your videos:
|
|
|
|
```
|
|
python scripts/data_preprocess/resize_videos.py
|
|
```
|
|
|
|
### Finetuning
|
|
After basic dataset preparation and preprocess, you can start to finetune your model using Lora:
|
|
|
|
```
|
|
bash scripts/finetune/finetune_hunyuan_hf_lora.sh
|
|
```
|
|
|
|
### Inference
|
|
For inference with Lora checkpoint, you can run the following scripts with additional parameter `--lora_checkpoint_dir`:
|
|
|
|
```
|
|
bash scripts/inference/inference_hunyuan_hf.sh
|
|
```
|
|
|
|
**We also provide scripts for Mochi in the same directory.**
|
|
|
|
### Finetune with Both Image and Video
|
|
Our codebase support finetuning with both image and video.
|
|
|
|
```bash
|
|
bash scripts/finetune/finetune_hunyuan.sh
|
|
bash scripts/finetune/finetune_mochi_lora_mix.sh
|
|
```
|
|
|
|
For Image-Video Mixture Fine-tuning, make sure to enable the `--group_frame` option in your script.
|