From bd0ef67632ac2533f18ed6086fadbcdb85d07ca2 Mon Sep 17 00:00:00 2001 From: rlsu9 Date: Sat, 14 Dec 2024 20:33:12 +0000 Subject: [PATCH] update readme --- README.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 23dbbb87b..d69af2908 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,4 @@ +### prepare the src data ``` python scripts/download_hf.py --repo_id=FastVideo/Distill-30K-Data-Src --local_dir=data/Distill-30K-Data-Src --repo_type=dataset cd data/Distill-30K-Data-Src @@ -8,12 +9,10 @@ mv ephemeral/hao.zhang/codefolder/FastVideo-OSP/data/Distill-30K-Data-Src/* . rm -r ephemeral rm Distill-30K-Data-Src.tar.gz cd ../.. -# Now the src data is stored in folder data/Distill-30K-Data-Src, adjust the path in `data/Distill-30K-Src/merge.txt` correspondingly -# Adjust the `SHARD_IDX=0` in `./scripts/preprocess_hunyuan_data.sh` for propressing data on multiple nodes - - +``` +### Now the src data is stored in folder data/Distill-30K-Data-Src, adjust the path in `data/Distill-30K-Src/merge.txt` correspondingly +### Adjust the `SHARD_NUM` (total shards num) and `SHARD_IDX` (idx of the shard to preprocess) correspoindingly in `./scripts/preprocess_hunyuan_data.sh` for each GPU node to preprocess different shards +``` bash ./scripts/preprocess_hunyuan_data.sh - -# The preprocessed data will be stored in `data/HD-Hunyuan-30K-Distill-Data_Shard*` - -``` \ No newline at end of file +``` +### The preprocessed data will be stored in `data/HD-Hunyuan-30K-Distill-Data_Shard*`