JerryZhou54 42a2f272d5 Preprocessing Stage: 1. Save to parquets periodically 2. Allow resuming from the middle 3. Doesn't support multi-gpu for now. Data Loader Stage: 1. Allow multi-gpu dataloader 2. Doesn't drop files even if number of parquet files is not divisible by num_gpus 3. Able to resume training

2025-05-23 00:57:39 +00:00

..

preprocess_hunyuan_data.sh

update release readme (#81)

2024-12-15 22:24:13 +08:00

preprocess_mochi_data.sh

Cleanup

2024-12-15 17:03:29 -08:00

preprocess_wan_data.sh

Preprocessing Stage: 1. Save to parquets periodically 2. Allow resuming from the middle 3. Doesn't support multi-gpu for now. Data Loader Stage: 1. Allow multi-gpu dataloader 2. Doesn't drop files even if number of parquet files is not divisible by num_gpus 3. Able to resume training

2025-05-23 00:57:39 +00:00