## ENใ€Training video generation model ### iใ€Base on webvid dataset If using the webvid dataset for training, you need to download the webvid dataset firstly. You need to arrange the webvid dataset in this format. ``` ๐Ÿ“ฆ project/ โ”œโ”€โ”€ ๐Ÿ“‚ datasets/ โ”‚ โ”œโ”€โ”€ ๐Ÿ“‚ webvid/ โ”‚ โ”œโ”€โ”€ ๐Ÿ“‚ videos/ โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“„ 00000001.mp4 โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“„ 00000002.mp4 โ”‚ โ”‚ โ””โ”€โ”€ ๐Ÿ“„ ..... โ”‚ โ””โ”€โ”€ ๐Ÿ“„ csv_of_webvid.csv ``` Then๏ผŒset scripts/train_t2v.sh. ``` export DATASET_NAME="datasets/webvid/videos/" export DATASET_META_NAME="datasets/webvid/csv_of_webvid.csv" ... train_data_format="webvid" ``` Then, we run scripts/train_t2v.sh. ```sh sh scripts/train_t2v.sh ``` ### iiใ€Base on internal dataset If using the internal dataset for training, you need to format the dataset firstly. You need to arrange the dataset in this format. ``` ๐Ÿ“ฆ project/ โ”œโ”€โ”€ ๐Ÿ“‚ datasets/ โ”‚ โ”œโ”€โ”€ ๐Ÿ“‚ internal_datasets/ โ”‚ โ”œโ”€โ”€ ๐Ÿ“‚ videos/ โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“„ 00000001.mp4 โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“„ 00000002.mp4 โ”‚ โ”‚ โ””โ”€โ”€ ๐Ÿ“„ ..... โ”‚ โ””โ”€โ”€ ๐Ÿ“„ json_of_internal_datasets.json ``` The json_of_internal_datasets.json is a standard JSON file, as shown in below: ```json [ { "file_path": "videos/00000001.mp4", "text": "A group of young men in suits and sunglasses are walking down a city street.", "type": "video" }, { "file_path": "videos/00000002.mp4", "text": "A notepad with a drawing of a woman on it.", "type": "video" } ..... ] ``` The file_path in the json needs to be set as relative path. Then, set scripts/train_t2v.sh. ``` export DATASET_NAME="datasets/internal_datasets/" export DATASET_META_NAME="datasets/internal_datasets/json_of_internal_datasets.json" ... train_data_format="normal" ``` Then, we run scripts/train_t2v.sh. ```sh sh scripts/train_t2v.sh ``` ## CNใ€่ฎญ็ปƒ่ง†้ข‘็”Ÿๆˆๆจกๅž‹ ### iใ€ๅŸบไบŽwebvidๆ•ฐๆฎ้›† ๅฆ‚ๆžœไฝฟ็”จwebvidๆ•ฐๆฎ้›†่ฟ›่กŒ่ฎญ็ปƒ๏ผŒๅˆ™้œ€่ฆ้ฆ–ๅ…ˆไธ‹่ฝฝwebvid็š„ๆ•ฐๆฎ้›†ใ€‚ ๆ‚จ้œ€่ฆไปฅ่ฟ™็งๆ ผๅผๆŽ’ๅˆ—webvidๆ•ฐๆฎ้›†ใ€‚ ``` ๐Ÿ“ฆ project/ โ”œโ”€โ”€ ๐Ÿ“‚ datasets/ โ”‚ โ”œโ”€โ”€ ๐Ÿ“‚ webvid/ โ”‚ โ”œโ”€โ”€ ๐Ÿ“‚ videos/ โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“„ 00000001.mp4 โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“„ 00000002.mp4 โ”‚ โ”‚ โ””โ”€โ”€ ๐Ÿ“„ ..... โ”‚ โ””โ”€โ”€ ๐Ÿ“„ csv_of_webvid.csv ``` ็„ถๅŽ๏ผŒ่ฟ›ๅ…ฅscripts/train_t2v.sh่ฟ›่กŒ่ฎพ็ฝฎใ€‚ ``` export DATASET_NAME="datasets/webvid/videos/" export DATASET_META_NAME="datasets/webvid/csv_of_webvid.csv" ... train_data_format="webvid" ``` ๆœ€ๅŽ่ฟ่กŒscripts/train_t2v.shใ€‚ ```sh sh scripts/train_t2v.sh ``` ### iiใ€ๅŸบไบŽ่‡ชๅปบๆ•ฐๆฎ้›† ๅฆ‚ๆžœไฝฟ็”จๅ†…้ƒจๆ•ฐๆฎ้›†่ฟ›่กŒ่ฎญ็ปƒ๏ผŒๅˆ™้œ€่ฆ้ฆ–ๅ…ˆๆ ผๅผๅŒ–ๆ•ฐๆฎ้›†ใ€‚ ๆ‚จ้œ€่ฆไปฅ่ฟ™็งๆ ผๅผๆŽ’ๅˆ—ๆ•ฐๆฎ้›†ใ€‚ ``` ๐Ÿ“ฆ project/ โ”œโ”€โ”€ ๐Ÿ“‚ datasets/ โ”‚ โ”œโ”€โ”€ ๐Ÿ“‚ internal_datasets/ โ”‚ โ”œโ”€โ”€ ๐Ÿ“‚ videos/ โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“„ 00000001.mp4 โ”‚ โ”‚ โ”œโ”€โ”€ ๐Ÿ“„ 00000002.mp4 โ”‚ โ”‚ โ””โ”€โ”€ ๐Ÿ“„ ..... โ”‚ โ””โ”€โ”€ ๐Ÿ“„ json_of_internal_datasets.json ``` json_of_internal_datasets.jsonๆ˜ฏไธ€ไธชๆ ‡ๅ‡†็š„jsonๆ–‡ไปถ๏ผŒๅฆ‚ไธ‹ๆ‰€็คบ๏ผš ```json [ { "file_path": "videos/00000001.mp4", "text": "A group of young men in suits and sunglasses are walking down a city street.", "type": "video" }, { "file_path": "videos/00000002.mp4", "text": "A notepad with a drawing of a woman on it.", "type": "video" } ..... ] ``` jsonไธญ็š„file_path้œ€่ฆ่ฎพ็ฝฎไธบ็›ธๅฏน่ทฏๅพ„ใ€‚ ็„ถๅŽ๏ผŒ่ฟ›ๅ…ฅscripts/train_t2v.sh่ฟ›่กŒ่ฎพ็ฝฎใ€‚ ``` export DATASET_NAME="datasets/internal_datasets/" export DATASET_META_NAME="datasets/internal_datasets/json_of_internal_datasets.json" ... train_data_format="normal" ``` ๆœ€ๅŽ่ฟ่กŒscripts/train_t2v.shใ€‚ ```sh sh scripts/train_t2v.sh ```