4 people 191d1be3b4 Will/training (#425) ...

Co-authored-by: Kevin Lin <42618777+kevin314@users.noreply.github.com>
Co-authored-by: William Lin <SolitaryThinker@users.noreply.github.com>
Co-authored-by: Will Lin <wlsaidhi@gmail.com>

2025-05-23 02:06:54 -04:00

..

preprocess_hunyuan_data.sh

update release readme (#81)

2024-12-15 22:24:13 +08:00

preprocess_mochi_data.sh

Cleanup

2024-12-15 17:03:29 -08:00

preprocess_wan_data_v1.sh

Will/training (#425)

2025-05-23 02:06:54 -04:00

preprocess_wan_data.sh

Preprocessing Stage: 1. Save to parquets periodically 2. Allow resuming from the middle 3. Doesn't support multi-gpu for now. Data Loader Stage: 1. Allow multi-gpu dataloader 2. Doesn't drop files even if number of parquet files is not divisible by num_gpus 3. Able to resume training

2025-05-23 00:57:39 +00:00