nodes/hao-ai-lab-FastVideo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

update readme

Browse Source
This commit is contained in:
rlsu9
2024-12-14 20:33:12 +00:00
parent 61ab8221ef
commit bd0ef67632
1 changed files with 7 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+7 -8
View File
@@ -1,3 +1,4 @@
### prepare the src data
```
python scripts/download_hf.py --repo_id=FastVideo/Distill-30K-Data-Src --local_dir=data/Distill-30K-Data-Src --repo_type=dataset
cd data/Distill-30K-Data-Src
@@ -8,12 +9,10 @@ mv ephemeral/hao.zhang/codefolder/FastVideo-OSP/data/Distill-30K-Data-Src/* .
rm -r ephemeral
rm Distill-30K-Data-Src.tar.gz
cd ../..
# Now the src data is stored in folder data/Distill-30K-Data-Src, adjust the path in `data/Distill-30K-Src/merge.txt` correspondingly
# Adjust the `SHARD_IDX=0` in `./scripts/preprocess_hunyuan_data.sh` for propressing data on multiple nodes
```
### Now the src data is stored in folder data/Distill-30K-Data-Src, adjust the path in `data/Distill-30K-Src/merge.txt` correspondingly
### Adjust the `SHARD_NUM` (total shards num) and `SHARD_IDX` (idx of the shard to preprocess) correspoindingly in `./scripts/preprocess_hunyuan_data.sh` for each GPU node to preprocess different shards
```
bash ./scripts/preprocess_hunyuan_data.sh
# The preprocessed data will be stored in `data/HD-Hunyuan-30K-Distill-Data_Shard*`
```
```
### The preprocessed data will be stored in `data/HD-Hunyuan-30K-Distill-Data_Shard*`