nodes/aigc-apps-VideoX-Fun
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix Bug in Self Forcing in Multi-Gpus Infernece && Update Training Code and Docs && Update Reward Models (#498)

Browse Source
This commit is contained in:
Bubbliiiing
2026-07-14 10:30:54 +08:00
committed by GitHub
parent 1fd9ed9208
commit 403f1f7b78
183 changed files with 5524 additions and 1832 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/cogvideox_fun/README_TRAIN.md
+7 -6
View File
@@ -45,9 +45,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -137,8 +138,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**Absolute Path**:
@@ -147,7 +148,7 @@ If your data uses absolute paths, configure in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -223,8 +224,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/CogVideoX-Fun-2b-InP` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size per GPU | 1 |
| `--image_sample_size` | Maximum image training resolution | 512 |
| `--video_sample_size` | Maximum video training resolution | 512 |
scripts/cogvideox_fun/README_TRAIN_CONTROL.md
+7 -6
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -145,8 +146,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Controls-Demo --local_dir ./datas
If your data uses relative paths, configure in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Controls-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Controls-Demo/metadata_add_width_height.json"
```
**Absolute Path**:
@@ -155,7 +156,7 @@ If your data uses absolute paths, configure in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -231,8 +232,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/CogVideoX-Fun-V1.1-2b-Control` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Controls-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Controls-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size per GPU | 4 |
| `--image_sample_size` | Maximum image training resolution | 512 |
| `--video_sample_size` | Maximum video training resolution | 512 |
scripts/cogvideox_fun/README_TRAIN_CONTROL_zh-CN.md
+7 -6
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -145,8 +146,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Controls-Demo --local_dir ./datas
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Controls-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Controls-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -155,7 +156,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -231,8 +232,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/CogVideoX-Fun-V1.1-2b-Control` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Controls-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Controls-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 4 |
| `--image_sample_size` | 图像最大训练分辨率 | 512 |
| `--video_sample_size` | 视频最大训练分辨率 | 512 |
scripts/cogvideox_fun/README_TRAIN_LORA.md
+7 -6
View File
@@ -45,9 +45,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -137,8 +138,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**Absolute Path**:
@@ -147,7 +148,7 @@ If your data uses absolute paths, configure in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, relative paths are recommended. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, absolute paths are recommended.
@@ -225,8 +226,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----------|-------------|---------------|
| `--pretrained_model_name_or_path` | Pre-trained model path | `models/Diffusion_Transformer/CogVideoX-Fun-2b-InP` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size | 1 |
| `--image_sample_size` | Maximum training resolution for images | 512 |
| `--video_sample_size` | Maximum training resolution for videos | 512 |
scripts/cogvideox_fun/README_TRAIN_LORA_zh-CN.md
+7 -6
View File
@@ -45,9 +45,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -137,8 +138,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -147,7 +148,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -225,8 +226,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/CogVideoX-Fun-2b-InP` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 图像最大训练分辨率 | 512 |
| `--video_sample_size` | 视频最大训练分辨率 | 512 |
scripts/cogvideox_fun/README_TRAIN_zh-CN.md
+7 -6
View File
@@ -45,9 +45,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -137,8 +138,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -147,7 +148,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -223,8 +224,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/CogVideoX-Fun-2b-InP` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 图像最大训练分辨率 | 512 |
| `--video_sample_size` | 视频最大训练分辨率 | 512 |
scripts/cogvideox_fun/train_reward_lora.py
+4 -1
View File
@@ -38,7 +38,10 @@ from accelerate import Accelerator
from accelerate.logging import get_logger
from accelerate.state import AcceleratorState
from accelerate.utils import ProjectConfiguration, set_seed
from decord import VideoReader
try:
from decord import VideoReader
except ImportError:
from videox_fun.data.utils import AVVideoReader as VideoReader
from diffusers import CogVideoXDPMScheduler, DDIMScheduler
from diffusers.optimization import get_scheduler
from diffusers.utils import check_min_version, deprecate, is_wandb_available
scripts/ernie_image/README_TRAIN.md
+11 -10
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -139,7 +140,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -208,8 +209,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Path to pretrained model | `models/Diffusion_Transformer/ERNIE-Image` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution, auto bucketing | 1328 |
| `--gradient_accumulation_steps` | Gradient accumulation steps (equivalent to larger batch) | 1 |
@@ -267,8 +268,8 @@ You can configure validation parameters to periodically generate test images dur
```sh
export MODEL_NAME="models/Diffusion_Transformer/ERNIE-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -306,8 +307,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/ERNIE-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/ernie_image/README_TRAIN_zh-CN.md
+11 -10
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -139,7 +140,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -208,8 +209,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/ERNIE-Image` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 最大训练分辨率,代码会自动分桶 | 1328 |
| `--gradient_accumulation_steps` | 梯度累积步数（等效增大 batch） | 1 |
@@ -267,8 +268,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/ERNIE-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -306,8 +307,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/ERNIE-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/fantasytalking/README_TRAIN.md
+7 -6
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Option 3: Using Docker**
@@ -149,8 +150,8 @@ The `metadata.json` for FantasyTalking is slightly different from the normal JSO
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -159,7 +160,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/path/to/your/metadata.json"
export DATASET_META_NAME="/path/to/your/metadata_add_width_height.json"
```
> 💡 **Tip**: If your dataset is small and stored locally, use relative paths. If your dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -249,8 +250,8 @@ Here is a detailed explanation of the key parameters in the training script:
| `config_path` | Model configuration file path | `config/wan2.1/wan_civitai.yaml` |
| `pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/Wan2.1-I2V-14B-720P` |
| `pretrained_audio_model_name_or_path` | Audio encoder path | `None` (automatically uses $MODEL_NAME/audio_encoder) |
| `train_data_dir` | Training dataset directory | `datasets/internal_datasets/` |
| `train_data_meta` | Training dataset metadata file | `datasets/internal_datasets/metadata.json` |
| `train_data_dir` | Training dataset directory | `datasets/X-Fun-Videos-Audios-Demo/` |
| `train_data_meta` | Training dataset metadata file | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `video_sample_size` | Video sample size (maximum resolution) | `512` |
| `token_sample_size` | Token sample size | `512` |
| `video_sample_stride` | Video sample stride | `1` |
scripts/fantasytalking/README_TRAIN_zh-CN.md
+7 -6
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式三：使用 Docker**
@@ -149,8 +150,8 @@ FantasyTalking 的 `metadata.json` 与 VideoX-Fun 的普通 JSON 格式略有不
如果你的数据使用的是相对路径，训练脚本中请这样配置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -159,7 +160,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/path/to/your/metadata.json"
export DATASET_META_NAME="/path/to/your/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存放在本地，请使用相对路径。如果数据集存放在外部存储（如 NAS、OSS）或多机共享，请使用绝对路径。
@@ -249,8 +250,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| `config_path` | 模型配置文件路径 | `config/wan2.1/wan_civitai.yaml` |
| `pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Wan2.1-I2V-14B-720P` |
| `pretrained_audio_model_name_or_path` | 音频编码器路径 | `None`（自动使用 $MODEL_NAME/audio_encoder） |
| `train_data_dir` | 训练数据集目录 | `datasets/internal_datasets/` |
| `train_data_meta` | 训练数据集元数据文件 | `datasets/internal_datasets/metadata.json` |
| `train_data_dir` | 训练数据集目录 | `datasets/X-Fun-Videos-Audios-Demo/` |
| `train_data_meta` | 训练数据集元数据文件 | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `video_sample_size` | 视频采样尺寸（最大分辨率） | `512` |
| `token_sample_size` | Token 采样尺寸 | `512` |
| `video_sample_stride` | 视频采样步幅 | `1` |
scripts/flashhead/README_TRAIN_S2V.md
+7 -6
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -154,8 +155,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -164,7 +165,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -247,8 +248,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| `--config_path` | Model configuration file path | `config/wan2.1/wan_civitai.yaml` |
| `--pretrained_model_name_or_path` | Path to pretrained model | `models/Diffusion_Transformer/SoulX-FlashHead-1_3B` |
| `--audio_encoder_path` | Audio encoder path (**FlashHead-S2V specific**) | `models/Diffusion_Transformer/wav2vec2-base-960h` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Samples per batch | 1 |
| `--video_sample_size` | Maximum video resolution for training | 512 |
| `--token_sample_size` | Token length sampling size | 512 |
scripts/flashhead/README_TRAIN_S2V_zh-CN.md
+7 -6
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式三：使用 Docker**
@@ -147,8 +148,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
如果你的数据使用的是相对路径，训练脚本中请这样配置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -157,7 +158,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存放在本地，请使用相对路径。如果数据集存放在外部存储（如 NAS、OSS）或多机共享，请使用绝对路径。
@@ -240,8 +241,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| `--config_path` | 模型配置文件路径 | `config/wan2.1/wan_civitai.yaml` |
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/SoulX-FlashHead-1_3B` |
| `--audio_encoder_path` | 音频编码器路径（**FlashHead-S2V 特有**） | `models/Diffusion_Transformer/wav2vec2-base-960h` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批训练的样本数 | 1 |
| `--video_sample_size` | 视频最大训练分辨率 | 512 |
| `--token_sample_size` | Token 长度采样大小 | 512 |
scripts/flux/README_TRAIN.md
+11 -10
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -139,7 +140,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -208,8 +209,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/FLUX.1-dev` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size per device | 1 |
| `--image_sample_size` | Maximum training resolution (auto bucketing) | 1024 |
| `--gradient_accumulation_steps` | Gradient accumulation steps (effective batch size increase) | 1 |
@@ -266,8 +267,8 @@ You can configure validation parameters to periodically generate test images dur
```sh
export MODEL_NAME="models/Diffusion_Transformer/FLUX.1-dev"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -305,8 +306,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/FLUX.1-dev"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/flux/README_TRAIN_LORA.md
+11 -10
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -139,7 +140,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -209,8 +210,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/FLUX.1-dev` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size per device | 1 |
| `--image_sample_size` | Maximum training resolution (auto bucketing) | 1024 |
| `--gradient_accumulation_steps` | Gradient accumulation steps (effective batch size increase) | 1 |
@@ -267,8 +268,8 @@ You can configure validation parameters to periodically generate test images dur
```sh
export MODEL_NAME="models/Diffusion_Transformer/FLUX.1-dev"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -307,8 +308,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/FLUX.1-dev"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/flux/README_TRAIN_LORA_zh-CN.md
+11 -10
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
如果你的数据使用相对路径，训练脚本中这样配置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**使用绝对路径**：
@@ -139,7 +140,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，使用相对路径。如果数据集存储在外部存储（如 NAS、OSS）或多机共享，使用绝对路径。
@@ -209,8 +210,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/FLUX.1-dev` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每张卡的批次大小 | 1 |
| `--image_sample_size` | 最大训练分辨率（自动分桶） | 1024 |
| `--gradient_accumulation_steps` | 梯度累积步数（等效增大 batch size） | 1 |
@@ -267,8 +268,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/FLUX.1-dev"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -307,8 +308,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/FLUX.1-dev"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/flux/README_TRAIN_zh-CN.md
+11 -10
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
如果你的数据使用相对路径，训练脚本中这样配置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**使用绝对路径**：
@@ -139,7 +140,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，使用相对路径。如果数据集存储在外部存储（如 NAS、OSS）或多机共享，使用绝对路径。
@@ -208,8 +209,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/FLUX.1-dev` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每张卡的批次大小 | 1 |
| `--image_sample_size` | 最大训练分辨率（自动分桶） | 1024 |
| `--gradient_accumulation_steps` | 梯度累积步数（等效增大 batch size） | 1 |
@@ -266,8 +267,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/FLUX.1-dev"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -305,8 +306,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/FLUX.1-dev"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/flux2/README_TRAIN.md
+11 -10
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -139,7 +140,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -208,8 +209,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/FLUX.2-dev` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size per device | 1 |
| `--image_sample_size` | Maximum training resolution (auto bucketing) | 1328 |
| `--gradient_accumulation_steps` | Gradient accumulation steps (effective batch size increase) | 1 |
@@ -266,8 +267,8 @@ You can configure validation parameters to periodically generate test images dur
```sh
export MODEL_NAME="models/Diffusion_Transformer/FLUX.2-dev"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -305,8 +306,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/FLUX.2-dev"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/flux2/README_TRAIN_LORA.md
+11 -10
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -139,7 +140,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -209,8 +210,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/FLUX.2-dev` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size per device | 1 |
| `--image_sample_size` | Maximum training resolution (auto bucketing) | 1328 |
| `--gradient_accumulation_steps` | Gradient accumulation steps (effective batch size increase) | 1 |
@@ -266,8 +267,8 @@ You can configure validation parameters to periodically generate test images dur
```sh
export MODEL_NAME="models/Diffusion_Transformer/FLUX.2-dev"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -306,8 +307,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/FLUX.2-dev"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/flux2/README_TRAIN_LORA_zh-CN.md
+11 -10
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -139,7 +140,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -209,8 +210,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/FLUX.2-dev` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 最大训练分辨率，代码会自动分桶 | 1328 |
| `--gradient_accumulation_steps` | 梯度累积步数（等效增大 batch） | 1 |
@@ -266,8 +267,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/FLUX.2-dev"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -306,8 +307,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/FLUX.2-dev"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/flux2/README_TRAIN_zh-CN.md
+11 -10
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -139,7 +140,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -208,8 +209,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/FLUX.2-dev` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 最大训练分辨率，代码会自动分桶 | 1328 |
| `--gradient_accumulation_steps` | 梯度累积步数（等效增大 batch） | 1 |
@@ -266,8 +267,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/FLUX.2-dev"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -305,8 +306,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/FLUX.2-dev"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/flux2_fun/README_TRAIN.md
+13 -32
View File
@@ -45,9 +45,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Option 3: Using Docker**
@@ -144,45 +145,25 @@ It is recommended to use tools like [DWPose](https://github.com/IDEA-Research/DW
### 2.4 Relative vs Absolute Path Usage
**Option 1: Using Relative Paths (Recommended)**
**Relative Paths**:
When data paths are not fixed or need to be trained on different machines, relative paths are recommended.
If your data uses relative paths, set in the training script:
Configure relative paths in `metadata.json`, then specify the dataset root directory via `--train_data_dir` in the training script:
```json
[
{
"file_path": "train/image001.jpg",
"control_file_path": "control/image001.jpg",
"text": "A group of young men in suits and sunglasses are walking down a city street.",
"width": 1024,
"height": 1024,
"type": "image"
}
]
```bash
export DATASET_NAME="datasets/X-Fun-Images-Controls-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Controls-Demo/metadata.json"
```
The training process will automatically search for files corresponding to the relative paths under `--train_data_dir`.
**Absolute Paths**:
**Option 2: Using Absolute Paths**
If your data uses absolute paths, set in the training script:
If the dataset path is fixed, you can directly configure absolute paths in `metadata.json`:
```json
[
{
"file_path": "/mnt/data/images/image001.jpg",
"control_file_path": "/mnt/data/controls/image001.jpg",
"text": "A group of young men in suits and sunglasses.",
"width": 1024,
"height": 1024,
"type": "image"
}
]
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
```
When using absolute paths, the `--train_data_dir` parameter serves only as a default path, and the absolute paths in the JSON will take priority.
> 💡 **Recommendation**: If the dataset is small and stored locally, relative paths are recommended; if the dataset is stored on external storage (such as NAS, OSS) or shared across multiple machines, absolute paths are recommended.
---
scripts/flux2_fun/README_TRAIN_zh-CN.md
+13 -32
View File
@@ -45,9 +45,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -144,45 +145,25 @@ Control 模式的 metadata.json 与普通 FLUX.2 的 json 略有不同，需要
### 2.4 相对路径与绝对路径使用方案
**方案 1：使用相对路径（推荐）**
**相对路径**：
当数据路径不固定，或需要在不同机器上训练时，推荐使用相对路径。
如果数据的路径为相对路径，则在训练脚本中设置：
在 `metadata.json` 中配置相对路径，然后在训练脚本中通过 `--train_data_dir` 指定数据集根目录：
```json
[
{
"file_path": "train/image001.jpg",
"control_file_path": "control/image001.jpg",
"text": "A group of young men in suits and sunglasses are walking down a city street.",
"width": 1024,
"height": 1024,
"type": "image"
}
]
```bash
export DATASET_NAME="datasets/X-Fun-Images-Controls-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Controls-Demo/metadata.json"
```
训练时会自动在 `--train_data_dir` 下寻找相对路径对应的文件。
**绝对路径**：
**方案 2：使用绝对路径**
如果数据的路径为绝对路径，则在训练脚本中设置：
如果数据集路径固定，可以直接在 `metadata.json` 中配置绝对路径：
```json
[
{
"file_path": "/mnt/data/images/image001.jpg",
"control_file_path": "/mnt/data/controls/image001.jpg",
"text": "A group of young men in suits and sunglasses.",
"width": 1024,
"height": 1024,
"type": "image"
}
]
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
```
使用绝对路径时，`--train_data_dir` 参数仅作为默认路径，实际会优先使用 json 中的绝对路径。
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
---
scripts/get_rewards_1video.py
+4 -1
View File
@@ -77,7 +77,10 @@ import sys
import numpy as np
import torch
import torchvision.transforms as transforms
from decord import VideoReader
try:
from decord import VideoReader
except ImportError:
from videox_fun.data.utils import AVVideoReader as VideoReader
from einops import rearrange
project_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
scripts/hunyuanvideo/README_TRAIN.md
+7 -6
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**Absolute Path**:
@@ -146,7 +147,7 @@ If your data uses absolute paths, configure in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, relative paths are recommended. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, absolute paths are recommended.
@@ -224,8 +225,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/HunyuanVideo` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Number of samples per batch | 16 |
| `--image_sample_size` | Maximum training resolution for images | 512 |
| `--video_sample_size` | Maximum training resolution for videos | 512 |
scripts/hunyuanvideo/README_TRAIN_LORA.md
+7 -6
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Option 3: Using Docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, set in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**Absolute paths**:
@@ -146,7 +147,7 @@ If your data uses absolute paths, set in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, relative paths are recommended. If the dataset is stored on external storage (such as NAS, OSS) or shared across multiple machines, absolute paths are recommended.
@@ -224,8 +225,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----------|-------------|---------------|
| `--pretrained_model_name_or_path` | Pre-trained model path | `models/Diffusion_Transformer/HunyuanVideo` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Number of samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution for images | 640 |
| `--video_sample_size` | Maximum training resolution for videos | 640 |
scripts/hunyuanvideo/README_TRAIN_LORA_zh-CN.md
+7 -6
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -146,7 +147,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -224,8 +225,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/HunyuanVideo` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 图像最大训练分辨率 | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
scripts/hunyuanvideo/README_TRAIN_zh-CN.md
+7 -6
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -146,7 +147,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -224,8 +225,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/HunyuanVideo` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 16 |
| `--image_sample_size` | 图像最大训练分辨率 | 512 |
| `--video_sample_size` | 视频最大训练分辨率 | 512 |
scripts/infinitetalk/README_TRAIN_S2V.md
+7 -6
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -154,8 +155,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -164,7 +165,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -253,8 +254,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| `--config_path` | Model configuration file path | `config/wan2.1/wan_civitai.yaml` |
| `--pretrained_model_name_or_path` | Path to pretrained model | `models/Diffusion_Transformer/Wan2.1-I2V-14B-480P` |
| `--audio_encoder_path` | Audio encoder path (**InfiniteTalk-S2V specific**) | `models/Diffusion_Transformer/chinese-wav2vec2-base/` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Samples per batch | 1 |
| `--video_sample_size` | Maximum video resolution for training | 640 |
| `--token_sample_size` | Token length sampling size | 640 |
scripts/infinitetalk/README_TRAIN_S2V_zh-CN.md
+7 -6
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式三：使用 Docker**
@@ -147,8 +148,8 @@ InfiniteTalk 的 `metadata.json` 与 VideoX-Fun 的普通 JSON 格式略有不
如果你的数据使用的是相对路径，训练脚本中请这样配置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -157,7 +158,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存放在本地，请使用相对路径。如果数据集存放在外部存储（如 NAS、OSS）或多机共享，请使用绝对路径。
@@ -248,8 +249,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| `--config_path` | 模型配置文件路径 | `config/wan2.1/wan_civitai.yaml` |
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Wan2.1-I2V-14B-480P` |
| `--audio_encoder_path` | 音频编码器路径（**InfiniteTalk 特有**） | `models/Diffusion_Transformer/chinese-wav2vec2-base/` |
| `--train_data_dir` | 训练数据集目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据集元数据文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据集目录 | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | 训练数据集元数据文件 | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--video_sample_size` | 视频采样尺寸（最大分辨率） | `640` |
| `--token_sample_size` | Token 采样尺寸 | `640` |
| `--video_sample_stride` | 视频采样步幅 | `1` |
scripts/lens/README_TRAIN.md
+11 -10
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -139,7 +140,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -208,8 +209,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Path to pretrained model | `models/Diffusion_Transformer/Lens` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution, auto bucketing | 1328 |
| `--gradient_accumulation_steps` | Gradient accumulation steps (equivalent to larger batch) | 1 |
@@ -267,8 +268,8 @@ You can configure validation parameters to periodically generate test images dur
```sh
export MODEL_NAME="models/Diffusion_Transformer/Lens"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -306,8 +307,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/Lens"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/lens/README_TRAIN_LORA.md
+11 -10
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -139,7 +140,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -209,8 +210,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Path to pretrained model | `models/Diffusion_Transformer/Lens` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution, auto bucketing | 1328 |
| `--gradient_accumulation_steps` | Gradient accumulation steps (equivalent to larger batch) | 1 |
@@ -269,8 +270,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/Lens"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -309,8 +310,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/Lens"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/lens/README_TRAIN_LORA_zh-CN.md
+11 -10
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -139,7 +140,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -209,8 +210,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Lens` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 最大训练分辨率，代码会自动分桶 | 1328 |
| `--gradient_accumulation_steps` | 梯度累积步数（等效增大 batch） | 1 |
@@ -270,8 +271,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/Lens"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -310,8 +311,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/Lens"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/lens/README_TRAIN_zh-CN.md
+11 -10
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -139,7 +140,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -208,8 +209,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Lens` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 最大训练分辨率,代码会自动分桶 | 1328 |
| `--gradient_accumulation_steps` | 梯度累积步数（等效增大 batch） | 1 |
@@ -267,8 +268,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/Lens"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -306,8 +307,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/Lens"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/longcatvideo/README_TRAIN.md
+7 -6
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**Absolute Path**:
@@ -146,7 +147,7 @@ If your data uses absolute paths, configure in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -224,8 +225,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/LongCat-Video` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size per GPU | 1 |
| `--image_sample_size` | Maximum image training resolution | 640 |
| `--video_sample_size` | Maximum video training resolution | 640 |
scripts/longcatvideo/README_TRAIN_AVATAR.md
+7 -6
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -154,8 +155,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -164,7 +165,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -247,8 +248,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
|-----|------|-------|
| `--pretrained_model_name_or_path` | Path to pretrained base model | `models/Diffusion_Transformer/LongCat-Video` |
| `--pretrained_avatar_model_name_or_path` | Path to pretrained avatar model | `models/Diffusion_Transformer/LongCat-Video-Avatar` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Samples per batch | 1 |
| `--video_sample_size` | Maximum video resolution for training | 640 |
| `--token_sample_size` | Token length sampling size | 640 |
scripts/longcatvideo/README_TRAIN_AVATAR_LORA.md
+7 -6
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -154,8 +155,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -164,7 +165,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -256,8 +257,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
|-----|------|-------|
| `--pretrained_model_name_or_path` | Path to pretrained base model | `models/Diffusion_Transformer/LongCat-Video` |
| `--pretrained_avatar_model_name_or_path` | Path to pretrained avatar model | `models/Diffusion_Transformer/LongCat-Video-Avatar` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution, auto bucketing | 640 |
| `--video_sample_size` | Maximum video resolution for training | 640 |
scripts/longcatvideo/README_TRAIN_AVATAR_LORA_zh-CN.md
+7 -6
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式三：使用 Docker**
@@ -154,8 +155,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
如果你的数据使用的是相对路径，训练脚本中请这样配置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -164,7 +165,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存放在本地，请使用相对路径。如果数据集存放在外部存储（如 NAS、OSS）或多机共享，请使用绝对路径。
@@ -256,8 +257,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练基础模型路径 | `models/Diffusion_Transformer/LongCat-Video` |
| `--pretrained_avatar_model_name_or_path` | 预训练 Avatar 模型路径 | `models/Diffusion_Transformer/LongCat-Video-Avatar` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批训练的样本数 | 1 |
| `--image_sample_size` | 最大训练分辨率，自动 bucket | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
scripts/longcatvideo/README_TRAIN_AVATAR_zh-CN.md
+7 -6
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式三：使用 Docker**
@@ -147,8 +148,8 @@ LongCat-Video-Avatar 的 `metadata.json` 与 VideoX-Fun 的普通 JSON 格式略
如果你的数据使用的是相对路径，训练脚本中请这样配置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -157,7 +158,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存放在本地，请使用相对路径。如果数据集存放在外部存储（如 NAS、OSS）或多机共享，请使用绝对路径。
@@ -242,8 +243,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
|------|------|--------|
| `--pretrained_model_name_or_path` | 预训练基础模型路径 | `models/Diffusion_Transformer/LongCat-Video` |
| `--pretrained_avatar_model_name_or_path` | 预训练 Avatar 模型路径 | `models/Diffusion_Transformer/LongCat-Video-Avatar` |
| `--train_data_dir` | 训练数据集目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据集元数据文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据集目录 | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | 训练数据集元数据文件 | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--video_sample_size` | 视频采样尺寸（最大分辨率） | `640` |
| `--token_sample_size` | Token 采样尺寸 | `640` |
| `--video_sample_stride` | 视频采样步幅 | `2` |
scripts/longcatvideo/README_TRAIN_LORA.md
+7 -6
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**Absolute Path**:
@@ -146,7 +147,7 @@ If your data uses absolute paths, configure in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, relative paths are recommended. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, absolute paths are recommended.
@@ -225,8 +226,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----------|-------------|---------------|
| `--pretrained_model_name_or_path` | Pre-trained model path | `models/Diffusion_Transformer/LongCat-Video` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size | 1 |
| `--image_sample_size` | Maximum training resolution for images | 640 |
| `--video_sample_size` | Maximum training resolution for videos | 640 |
scripts/longcatvideo/README_TRAIN_LORA_zh-CN.md
+7 -6
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -146,7 +147,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -225,8 +226,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/LongCat-Video` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 图像最大训练分辨率 | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
scripts/longcatvideo/README_TRAIN_zh-CN.md
+7 -6
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -146,7 +147,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -224,8 +225,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/LongCat-Video` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 图像最大训练分辨率 | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
scripts/longcatvideo/train.py
-5
View File
@@ -985,11 +985,6 @@ def main():
if args.gradient_checkpointing:
transformer3d.enable_gradient_checkpointing()
elif args.selective_ac > 0:
from videox_fun.models.wan_transformer3d import WanAttentionBlock
from videox_fun.utils.ac_handle import apply_checkpointing, partial
apply_selective_ac = partial(apply_checkpointing, block=WanAttentionBlock)
apply_selective_ac(transformer3d, p=args.selective_ac)
# Enable TF32 for faster training on Ampere GPUs,
# cf https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices
scripts/longcatvideo/train_avatar.py
-5
View File
@@ -1027,11 +1027,6 @@ def main():
if args.gradient_checkpointing:
transformer3d.enable_gradient_checkpointing()
elif args.selective_ac > 0:
from videox_fun.models.wan_transformer3d import WanAttentionBlock
from videox_fun.utils.ac_handle import apply_checkpointing, partial
apply_selective_ac = partial(apply_checkpointing, block=WanAttentionBlock)
apply_selective_ac(transformer3d, p=args.selective_ac)
# Enable TF32 for faster training on Ampere GPUs,
# cf https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices
scripts/ltx2.3/README_TRAIN.md
+11 -10
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -154,8 +155,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -164,7 +165,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -241,8 +242,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Path to pretrained model | `models/Diffusion_Transformer/LTX-2.3-Diffusers` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution, auto bucketing | 640 |
| `--video_sample_size` | Maximum video resolution for training | 640 |
@@ -307,8 +308,8 @@ You can configure validation parameters to periodically generate test videos dur
```sh
export MODEL_NAME="models/Diffusion_Transformer/LTX-2.3-Diffusers"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -354,8 +355,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/LTX-2.3-Diffusers"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/ltx2.3/README_TRAIN_LORA.md
+11 -10
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -154,8 +155,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -164,7 +165,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -251,8 +252,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Path to pretrained model | `models/Diffusion_Transformer/LTX-2.3-Diffusers` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution, auto bucketing | 640 |
| `--video_sample_size` | Maximum video resolution for training | 640 |
@@ -314,8 +315,8 @@ You can configure validation parameters to periodically generate test videos dur
```sh
export MODEL_NAME="models/Diffusion_Transformer/LTX-2.3-Diffusers"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -362,8 +363,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/LTX-2.3-Diffusers"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/ltx2.3/README_TRAIN_LORA_zh-CN.md
+11 -10
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式三：使用 Docker**
@@ -154,8 +155,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
如果你的数据使用的是相对路径，训练脚本中请这样配置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -164,7 +165,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存放在本地，请使用相对路径。如果数据集存放在外部存储（如 NAS、OSS）或多机共享，请使用绝对路径。
@@ -251,8 +252,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/LTX-2` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批训练的样本数 | 1 |
| `--image_sample_size` | 最大训练分辨率，自动 bucket | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
@@ -314,8 +315,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/LTX-2"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -362,8 +363,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/LTX-2"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/ltx2.3/README_TRAIN_zh-CN.md
+11 -10
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式三：使用 Docker**
@@ -154,8 +155,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
如果你的数据使用的是相对路径，训练脚本中请这样配置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -164,7 +165,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存放在本地，请使用相对路径。如果数据集存放在外部存储（如 NAS、OSS）或多机共享，请使用绝对路径。
@@ -241,8 +242,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/LTX-2.3-Diffusers` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批训练的样本数 | 1 |
| `--image_sample_size` | 最大训练分辨率，自动 bucket | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
@@ -307,8 +308,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/LTX-2.3-Diffusers"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -354,8 +355,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/LTX-2.3-Diffusers"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/ltx2/README_TRAIN.md
+11 -10
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -154,8 +155,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -164,7 +165,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -244,8 +245,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Path to pretrained model | `models/Diffusion_Transformer/LTX-2` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution, auto bucketing | 640 |
| `--video_sample_size` | Maximum video resolution for training | 640 |
@@ -310,8 +311,8 @@ You can configure validation parameters to periodically generate test videos dur
```sh
export MODEL_NAME="models/Diffusion_Transformer/LTX-2"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -357,8 +358,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/LTX-2"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/ltx2/README_TRAIN_LORA.md
+11 -10
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -154,8 +155,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -164,7 +165,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -254,8 +255,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Path to pretrained model | `models/Diffusion_Transformer/LTX-2` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution, auto bucketing | 640 |
| `--video_sample_size` | Maximum video resolution for training | 640 |
@@ -317,8 +318,8 @@ You can configure validation parameters to periodically generate test videos dur
```sh
export MODEL_NAME="models/Diffusion_Transformer/LTX-2"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -365,8 +366,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/LTX-2"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/ltx2/README_TRAIN_LORA_zh-CN.md
+11 -10
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式三：使用 Docker**
@@ -154,8 +155,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
如果你的数据使用的是相对路径，训练脚本中请这样配置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -164,7 +165,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存放在本地，请使用相对路径。如果数据集存放在外部存储（如 NAS、OSS）或多机共享，请使用绝对路径。
@@ -254,8 +255,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/LTX-2` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批训练的样本数 | 1 |
| `--image_sample_size` | 最大训练分辨率，自动 bucket | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
@@ -317,8 +318,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/LTX-2"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -365,8 +366,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/LTX-2"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/ltx2/README_TRAIN_zh-CN.md
+11 -10
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式三：使用 Docker**
@@ -154,8 +155,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
如果你的数据使用的是相对路径，训练脚本中请这样配置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -164,7 +165,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存放在本地，请使用相对路径。如果数据集存放在外部存储（如 NAS、OSS）或多机共享，请使用绝对路径。
@@ -244,8 +245,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/LTX-2` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批训练的样本数 | 1 |
| `--image_sample_size` | 最大训练分辨率，自动 bucket | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
@@ -310,8 +311,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/LTX-2"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -357,8 +358,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/LTX-2"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/ltx2/train_upsampler.py
+12 -10
View File
@@ -124,7 +124,10 @@ def log_validation(vae, latent_upsampler, args, accelerator, weight_dtype, globa
logger.info("No validation_paths provided, skipping validation.")
return
from decord import VideoReader
try:
from decord import VideoReader
except ImportError:
from videox_fun.data.utils import AVVideoReader as VideoReader
for i, video_path in enumerate(args.validation_paths):
if not os.path.exists(video_path):
@@ -159,8 +162,8 @@ def log_validation(vae, latent_upsampler, args, accelerator, weight_dtype, globa
# Encode high-res
gt_latents = vae.encode(pixel_values)[0].mode()
# Create low-res input
scale = args.spatial_scale
# Create low-res input (scale is fixed by upsampler architecture)
scale = float(accelerator.unwrap_model(latent_upsampler).config.rational_spatial_scale)
low_h, low_w = int(h_target / scale), int(w_target / scale)
# Downsample spatially: flatten batch and frames, interpolate, unflatten
b, c, f, h, w = pixel_values.shape
@@ -488,12 +491,6 @@ def parse_args():
default=None,
help=("If you want to load the weight from other vaes, input its path."),
)
parser.add_argument(
"--spatial_scale",
type=float,
default=2.0,
help="Spatial upsampling scale factor (must match model config).",
)
parser.add_argument(
'--trainable_modules',
nargs='+',
@@ -664,6 +661,11 @@ def main():
m, u = latent_upsampler.load_state_dict(state_dict, strict=False)
print(f"Upsampler missing keys: {len(m)}, unexpected keys: {len(u)}")
# Spatial scale is fixed by the upsampler architecture; read it directly from the model config.
spatial_scale = float(latent_upsampler.config.rational_spatial_scale)
if accelerator.is_main_process:
logger.info(f"Using spatial_scale={spatial_scale} from latent_upsampler.config.rational_spatial_scale.")
# Set trainable parameters
latent_upsampler.requires_grad_(False)
latent_upsampler.train()
@@ -857,7 +859,7 @@ def main():
batch_video_length = 1
# Compute low-res target size (aligned to spatial_compression_ratio)
scale = args.spatial_scale
scale = spatial_scale
spatial_ratio = vae.config.spatial_compression_ratio
closest_size_list = list(map(lambda x: int(x), closest_size))
low_h = int(closest_size_list[0] / scale / spatial_ratio) * spatial_ratio
scripts/mova/README_TRAIN.md
+13 -20
View File
@@ -47,20 +47,13 @@ pip install Pillow einops safetensors timm tomesd librosa transformers accelerat
### 2.1 Quick Test Dataset
For testing purposes, you can use the demo dataset from Hugging Face:
We provide a test dataset containing several video-audio training samples.
```bash
# Create dataset directory
mkdir -p datasets
# Download demo dataset from Hugging Face
cd datasets
git clone https://huggingface.co/datasets/modelscope/X-Fun-Videos-Audios-Demo
cd ..
# Download demo dataset
modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./datasets/X-Fun-Videos-Audios-Demo
```
This dataset contains 17 video-audio pairs for quick testing.
### 2.2 Dataset Structure
```
@@ -133,8 +126,8 @@ This dataset contains 17 video-audio pairs for quick testing.
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -143,7 +136,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Tip**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -169,7 +162,7 @@ FSDP training is recommended as it can significantly save VRAM.
```bash
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi-node environments without RDMA
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -219,8 +212,8 @@ accelerate launch --mixed_precision="bf16" scripts/mova/train.py \
| Parameter | Description | Example Value |
|-----------|-------------|---------------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/MOVA-360p` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Number of samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution for auto bucket | 480 |
| `--video_sample_size` | Maximum video training resolution | 480 |
@@ -269,8 +262,8 @@ Edit the script to load your trained checkpoint and generate test videos.
```bash
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -320,8 +313,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```bash
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/mova/README_TRAIN_LORA.md
+14 -13
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Option 3: Using Docker**
@@ -154,8 +155,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -164,7 +165,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Tip**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -192,7 +193,7 @@ The difference between DeepSpeed-Zero-2 and FSDP is whether model weights are sh
```bash
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi-node environments without RDMA
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -254,8 +255,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----------|-------------|---------------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/MOVA-360p` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Number of samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution for auto bucket | 360 |
| `--video_sample_size` | Maximum video training resolution | 360 |
@@ -337,8 +338,8 @@ You can configure validation parameters to regularly generate test videos during
```sh
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -394,8 +395,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -451,7 +452,7 @@ Assuming 2 machines, each with 8 GPUs:
```bash
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
export MASTER_ADDR="192.168.1.100" # Master machine IP
export MASTER_PORT=10086
export WORLD_SIZE=2 # Total number of machines
@@ -504,7 +505,7 @@ accelerate launch --mixed_precision="bf16" --main_process_ip=$MASTER_ADDR --main
```bash
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
export MASTER_ADDR="192.168.1.100" # Same as Master
export MASTER_PORT=10086
export WORLD_SIZE=2
scripts/mova/README_TRAIN_LORA_zh-CN.md
+14 -13
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式三：使用 Docker**
@@ -154,8 +155,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
如果你的数据使用的是相对路径，训练脚本中请这样配置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -164,7 +165,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存放在本地，请使用相对路径。如果数据集存放在外部存储（如 NAS、OSS）或多机共享，请使用绝对路径。
@@ -192,7 +193,7 @@ DeepSpeed-Zero-2 与 FSDP 的区别在于模型权重是否分片。**如果多
```bash
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -254,8 +255,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/MOVA-360p` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批训练的样本数 | 1 |
| `--image_sample_size` | 最大训练分辨率，自动 bucket | 360 |
| `--video_sample_size` | 视频最大训练分辨率 | 360 |
@@ -338,8 +339,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -395,8 +396,8 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```sh
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -452,7 +453,7 @@ accelerate launch --mixed_precision="bf16" scripts/mova/train_lora.py \
```bash
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
export MASTER_ADDR="192.168.1.100" # Master 机器 IP
export MASTER_PORT=10086
export WORLD_SIZE=2 # 总机器数
@@ -505,7 +506,7 @@ accelerate launch --mixed_precision="bf16" --main_process_ip=$MASTER_ADDR --main
```bash
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
export MASTER_ADDR="192.168.1.100" # 与 Master 相同
export MASTER_PORT=10086
export WORLD_SIZE=2
scripts/mova/README_TRAIN_zh-CN.md
+14 -13
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式三：使用 Docker**
@@ -152,8 +153,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./dataset
如果你的数据使用的是相对路径，训练脚本中请这样配置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -162,7 +163,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存放在本地，请使用相对路径。如果数据集存放在外部存储（如 NAS、OSS）或多机共享，请使用绝对路径。
@@ -190,7 +191,7 @@ DeepSpeed-Zero-2 与 FSDP 的区别在于模型权重是否分片。**如果多
```bash
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -240,8 +241,8 @@ accelerate launch --mixed_precision="bf16" scripts/mova/train.py \
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/MOVA-360p` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Audios-Demo/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批训练的样本数 | 1 |
| `--image_sample_size` | 最大训练分辨率，自动 bucket | 480 |
| `--video_sample_size` | 视频最大训练分辨率 | 480 |
@@ -313,8 +314,8 @@ accelerate launch --mixed_precision="bf16" scripts/mova/train.py \
```sh
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -363,8 +364,8 @@ accelerate launch --mixed_precision="bf16" scripts/mova/train.py \
```sh
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 和 NCCL_P2P_DISABLE=1 用于无 RDMA 的多机环境
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -422,7 +423,7 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
```bash
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
export MASTER_ADDR="192.168.1.100" # Master 机器 IP
export MASTER_PORT=10086
export WORLD_SIZE=2 # 总机器数
@@ -474,7 +475,7 @@ accelerate launch --mixed_precision="bf16" --main_process_ip=$MASTER_ADDR --main
```bash
export MODEL_NAME="models/Diffusion_Transformer/MOVA-360p"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
export MASTER_ADDR="192.168.1.100" # 与 Master 相同
export MASTER_PORT=10086
export WORLD_SIZE=2
scripts/qwenimage/README_TRAIN.md
+13 -12
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -139,7 +140,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -208,8 +209,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Path to pretrained model | `models/Diffusion_Transformer/Qwen-Image` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution, auto bucketing | 1328 |
| `--gradient_accumulation_steps` | Gradient accumulation steps (equivalent to larger batch) | 1 |
@@ -270,8 +271,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/Qwen-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -320,8 +321,8 @@ python scripts/zero_to_bf16.py output_dir/checkpoint-{our-num-steps} output_dir/
Training shell command:
```sh
export MODEL_NAME="models/Diffusion_Transformer/Qwen-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -359,8 +360,8 @@ accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag
```sh
export MODEL_NAME="models/Diffusion_Transformer/Qwen-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/qwenimage/README_TRAIN_EDIT.md
+7 -6
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Option 3: Using Docker**
@@ -128,8 +129,8 @@ The metadata.json for Edit model is different from the normal version, requiring
If your data uses relative paths, set in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Edit-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Edit-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -138,7 +139,7 @@ If your data uses absolute paths, set in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, relative paths are recommended; if the dataset is stored on external storage (such as NAS, OSS) or shared across multiple machines, absolute paths are recommended.
@@ -216,8 +217,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Pre-trained model path | `models/Diffusion_Transformer/Qwen-Image-Edit` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Images-Edit-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Images-Edit-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Number of samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution, code will automatically bucket | 1328 |
| `--gradient_accumulation_steps` | Gradient accumulation steps (equivalent to increasing batch) | 1 |
scripts/qwenimage/README_TRAIN_EDIT_zh-CN.md
+7 -6
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -128,8 +129,8 @@ Edit 模型的 metadata.json 与普通版本不同，需要添加 `source_file_p
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Edit-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Edit-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -138,7 +139,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -216,8 +217,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Qwen-Image-Edit` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Images-Edit-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Images-Edit-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 最大训练分辨率，代码会自动分桶 | 1328 |
| `--gradient_accumulation_steps` | 梯度累积步数（等效增大 batch） | 1 |
scripts/qwenimage/README_TRAIN_LORA.md
+13 -12
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure the training script as follows:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -139,7 +140,7 @@ If your data uses absolute paths, configure the training script as follows:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -209,8 +210,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Path to pretrained model | `models/Diffusion_Transformer/Qwen-Image` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution, auto bucketing | 1328 |
| `--gradient_accumulation_steps` | Gradient accumulation steps (equivalent to larger batch) | 1 |
@@ -269,8 +270,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/Qwen-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -320,8 +321,8 @@ python scripts/zero_to_bf16.py output_dir/checkpoint-{our-num-steps} output_dir/
Training shell command:
```sh
export MODEL_NAME="models/Diffusion_Transformer/Qwen-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -360,8 +361,8 @@ accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag
```sh
export MODEL_NAME="models/Diffusion_Transformer/Qwen-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/qwenimage/README_TRAIN_LORA_zh-CN.md
+13 -12
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -139,7 +140,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -209,8 +210,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Qwen-Image` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 最大训练分辨率，代码会自动分桶 | 1328 |
| `--gradient_accumulation_steps` | 梯度累积步数（等效增大 batch） | 1 |
@@ -270,8 +271,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/Qwen-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -321,8 +322,8 @@ python scripts/zero_to_bf16.py output_dir/checkpoint-{our-num-steps} output_dir/
执行命令为：
```sh
export MODEL_NAME="models/Diffusion_Transformer/Qwen-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -361,8 +362,8 @@ accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag
```sh
export MODEL_NAME="models/Diffusion_Transformer/Qwen-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/qwenimage/README_TRAIN_zh-CN.md
+13 -12
View File
@@ -41,9 +41,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -129,8 +130,8 @@ modelscope download --dataset PAI/X-Fun-Images-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -139,7 +140,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -208,8 +209,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Qwen-Image` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Images-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Images-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 最大训练分辨率，代码会自动分桶 | 1328 |
| `--gradient_accumulation_steps` | 梯度累积步数（等效增大 batch） | 1 |
@@ -271,8 +272,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```sh
export MODEL_NAME="models/Diffusion_Transformer/Qwen-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -321,8 +322,8 @@ python scripts/zero_to_bf16.py output_dir/checkpoint-{our-num-steps} output_dir/
执行命令为：
```sh
export MODEL_NAME="models/Diffusion_Transformer/Qwen-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -360,8 +361,8 @@ accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag
```sh
export MODEL_NAME="models/Diffusion_Transformer/Qwen-Image"
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Images-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
scripts/qwenimage_fun/README_TRAIN.md
+13 -32
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Option 3: Using Docker**
@@ -142,45 +143,25 @@ It is recommended to use tools like [DWPose](https://github.com/IDEA-Research/DW
### 2.4 Relative vs Absolute Path Usage
**Option 1: Using Relative Paths (Recommended)**
**Relative Paths**:
When data paths are not fixed or you need to train on different machines, relative paths are recommended.
If your data uses relative paths, set in the training script:
Configure relative paths in `metadata.json`, then specify the dataset root directory via `--train_data_dir` in the training script:
```json
[
{
"file_path": "train/image001.jpg",
"control_file_path": "control/image001.jpg",
"text": "A group of young men in suits and sunglasses are walking down a city street.",
"width": 1024,
"height": 1024,
"type": "image"
}
]
```bash
export DATASET_NAME="datasets/X-Fun-Images-Controls-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Controls-Demo/metadata.json"
```
During training, the script will automatically search for files corresponding to relative paths under `--train_data_dir`.
**Absolute Paths**:
**Option 2: Using Absolute Paths**
If your data uses absolute paths, set in the training script:
If the dataset path is fixed, you can directly configure absolute paths in `metadata.json`:
```json
[
{
"file_path": "/mnt/data/images/image001.jpg",
"control_file_path": "/mnt/data/controls/image001.jpg",
"text": "A group of young men in suits and sunglasses.",
"width": 1024,
"height": 1024,
"type": "image"
}
]
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
```
When using absolute paths, the `--train_data_dir` parameter serves only as a default path, and the absolute paths in the JSON will take priority.
> 💡 **Recommendation**: If the dataset is small and stored locally, relative paths are recommended; if the dataset is stored on external storage (such as NAS, OSS) or shared across multiple machines, absolute paths are recommended.
---
scripts/qwenimage_fun/README_TRAIN_zh-CN.md
+13 -32
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3:使用docker**
@@ -142,45 +143,25 @@ Control 模式的 metadata.json 与普通 Qwen-Image 的 json 略有不同，需
### 2.4 相对路径与绝对路径使用方案
**方案 1:使用相对路径(推荐)**
**相对路径**：
当数据路径不固定,或需要在不同机器上训练时,推荐使用相对路径。
如果数据的路径为相对路径，则在训练脚本中设置：
在 `metadata.json` 中配置相对路径,然后在训练脚本中通过 `--train_data_dir` 指定数据集根目录:
```json
[
{
"file_path": "train/image001.jpg",
"control_file_path": "control/image001.jpg",
"text": "A group of young men in suits and sunglasses are walking down a city street.",
"width": 1024,
"height": 1024,
"type": "image"
}
]
```bash
export DATASET_NAME="datasets/X-Fun-Images-Controls-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Controls-Demo/metadata.json"
```
训练时会自动在 `--train_data_dir` 下寻找相对路径对应的文件。
**绝对路径**：
**方案 2:使用绝对路径**
如果数据的路径为绝对路径，则在训练脚本中设置：
如果数据集路径固定,可以直接在 `metadata.json` 中配置绝对路径:
```json
[
{
"file_path": "/mnt/data/images/image001.jpg",
"control_file_path": "/mnt/data/controls/image001.jpg",
"text": "A group of young men in suits and sunglasses.",
"width": 1024,
"height": 1024,
"type": "image"
}
]
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
```
使用绝对路径时,`--train_data_dir` 参数仅作为默认路径,实际会优先使用 json 中的绝对路径。
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
---
scripts/qwenimage_instantx/README_TRAIN.md
+13 -32
View File
@@ -45,9 +45,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Option 3: Using Docker**
@@ -144,45 +145,25 @@ It is recommended to use tools like [DWPose](https://github.com/IDEA-Research/DW
### 2.4 Relative vs Absolute Path Usage
**Option 1: Using Relative Paths (Recommended)**
**Relative Paths**:
When data paths are not fixed or need to be trained on different machines, relative paths are recommended.
If your data uses relative paths, set in the training script:
Configure relative paths in `metadata.json`, then specify the dataset root directory via `--train_data_dir` in the training script:
```json
[
{
"file_path": "train/image001.jpg",
"control_file_path": "control/image001.jpg",
"text": "A group of young men in suits and sunglasses are walking down a city street.",
"width": 1024,
"height": 1024,
"type": "image"
}
]
```bash
export DATASET_NAME="datasets/X-Fun-Images-Controls-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Controls-Demo/metadata.json"
```
The training process will automatically search for files corresponding to the relative paths under `--train_data_dir`.
**Absolute Paths**:
**Option 2: Using Absolute Paths**
If your data uses absolute paths, set in the training script:
If the dataset path is fixed, you can directly configure absolute paths in `metadata.json`:
```json
[
{
"file_path": "/mnt/data/images/image001.jpg",
"control_file_path": "/mnt/data/controls/image001.jpg",
"text": "A group of young men in suits and sunglasses.",
"width": 1024,
"height": 1024,
"type": "image"
}
]
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
```
When using absolute paths, the `--train_data_dir` parameter serves only as a default path, and the absolute paths in the JSON will take priority.
> 💡 **Recommendation**: If the dataset is small and stored locally, relative paths are recommended; if the dataset is stored on external storage (such as NAS, OSS) or shared across multiple machines, absolute paths are recommended.
---
scripts/qwenimage_instantx/README_TRAIN_zh-CN.md
+13 -32
View File
@@ -45,9 +45,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3:使用docker**
@@ -144,45 +145,25 @@ Control 模式的 metadata.json 与普通 Qwen-Image 的 json 略有不同，需
### 2.4 相对路径与绝对路径使用方案
**方案 1:使用相对路径(推荐)**
**相对路径**：
当数据路径不固定,或需要在不同机器上训练时,推荐使用相对路径。
如果数据的路径为相对路径，则在训练脚本中设置：
在 `metadata.json` 中配置相对路径,然后在训练脚本中通过 `--train_data_dir` 指定数据集根目录:
```json
[
{
"file_path": "train/image001.jpg",
"control_file_path": "control/image001.jpg",
"text": "A group of young men in suits and sunglasses are walking down a city street.",
"width": 1024,
"height": 1024,
"type": "image"
}
]
```bash
export DATASET_NAME="datasets/X-Fun-Images-Controls-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Images-Controls-Demo/metadata.json"
```
训练时会自动在 `--train_data_dir` 下寻找相对路径对应的文件。
**绝对路径**：
**方案 2:使用绝对路径**
如果数据的路径为绝对路径，则在训练脚本中设置：
如果数据集路径固定,可以直接在 `metadata.json` 中配置绝对路径:
```json
[
{
"file_path": "/mnt/data/images/image001.jpg",
"control_file_path": "/mnt/data/controls/image001.jpg",
"text": "A group of young men in suits and sunglasses.",
"width": 1024,
"height": 1024,
"type": "image"
}
]
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
```
使用绝对路径时,`--train_data_dir` 参数仅作为默认路径,实际会优先使用 json 中的绝对路径。
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
---
scripts/turbodiffusion/README_TRAIN_DISTILL.md
+13 -12
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**Absolute Path**:
@@ -146,7 +147,7 @@ If your data uses absolute paths, configure in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -199,7 +200,7 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -227,8 +228,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
|-----|------|-------|
| `--config_path` | Config file path | `config/wan2.1/wan_civitai.yaml` |
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/Wan2.1-T2V-1.3B/` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size per GPU | 1 |
| `--image_sample_size` | Maximum image training resolution | 640 |
| `--video_sample_size` | Maximum video training resolution | 640 |
@@ -240,7 +241,7 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| `--num_train_epochs` | Number of training epochs | 100 |
| `--checkpointing_steps` | Save checkpoint every N steps | 50 |
| `--learning_rate` | Initial learning rate (generator) | 2e-06 |
| `--learning_rate_critic` | Initial learning rate (critic) | 2e-07 |
| `--learning_rate_critic` | Initial learning rate (critic) | 2e-06 |
| `--lr_scheduler` | Learning rate scheduler | `constant_with_warmup` |
| `--lr_warmup_steps` | Learning rate warmup steps | 100 |
| `--seed` | Random seed | 42 |
@@ -376,7 +377,7 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -434,7 +435,7 @@ accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -484,7 +485,7 @@ accelerate launch --mixed_precision="bf16" scripts/turbodiffusion/train_distill.
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -544,7 +545,7 @@ accelerate launch --mixed_precision="bf16" --main_process_ip=$MASTER_ADDR --main
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
scripts/turbodiffusion/README_TRAIN_DISTILL_zh-CN.md
+13 -12
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方法 3: 使用 Docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
如果您的数据使用相对路径，请在训练脚本中配置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -146,7 +147,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，请使用相对路径。如果数据集存储在外部存储（如 NAS、OSS）或在多台机器间共享，请使用绝对路径。
@@ -199,7 +200,7 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -227,8 +228,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
|-----|------|-------|
| `--config_path` | 配置文件路径 | `config/wan2.1/wan_civitai.yaml` |
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Wan2.1-T2V-1.3B/` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | 训练数据元数据文件 | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每个 GPU 的批次大小 | 1 |
| `--image_sample_size` | 最大图像训练分辨率 | 640 |
| `--video_sample_size` | 最大视频训练分辨率 | 640 |
@@ -240,7 +241,7 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| `--num_train_epochs` | 训练轮数 | 100 |
| `--checkpointing_steps` | 每 N 步保存检查点 | 50 |
| `--learning_rate` | 初始学习率（生成器） | 2e-06 |
| `--learning_rate_critic` | 初始学习率（判别器） | 2e-07 |
| `--learning_rate_critic` | 初始学习率（判别器） | 2e-06 |
| `--lr_scheduler` | 学习率调度器 | `constant_with_warmup` |
| `--lr_warmup_steps` | 学习率预热步数 | 100 |
| `--seed` | 随机种子 | 42 |
@@ -376,7 +377,7 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -434,7 +435,7 @@ accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -484,7 +485,7 @@ accelerate launch --mixed_precision="bf16" scripts/turbodiffusion/train_distill.
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -544,7 +545,7 @@ accelerate launch --mixed_precision="bf16" --main_process_ip=$MASTER_ADDR --main
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
scripts/turbodiffusion/train_distill.py
+74 -6
View File
@@ -730,6 +730,11 @@ def parse_args():
default=[1000, 750, 500, 250],
help="The denoising step list.",
)
parser.add_argument(
"--randomize_step_indices",
action="store_true",
help="whether to use randomize timesteps indices in training.",
)
args = parser.parse_args()
env_local_rank = int(os.environ.get("LOCAL_RANK", -1))
@@ -857,6 +862,7 @@ def main():
weight_dtype = torch.bfloat16
args.mixed_precision = accelerator.mixed_precision
args.denoising_step_indices_list = [int(i) for i in args.denoising_step_indices_list]
# Load scheduler, tokenizer and models.
noise_scheduler = FlowMatchEulerDiscreteScheduler(
**filter_kwargs(FlowMatchEulerDiscreteScheduler, OmegaConf.to_container(config['scheduler_kwargs']))
@@ -1107,8 +1113,8 @@ def main():
fake_trainable_params = list(filter(lambda p: p.requires_grad, fake_score_transformer3d.parameters()))
fake_trainable_params_optim = [
{'params': [], 'lr': args.learning_rate},
{'params': [], 'lr': args.learning_rate / 2},
{'params': [], 'lr': args.learning_rate_critic},
{'params': [], 'lr': args.learning_rate_critic / 2},
]
in_already = []
for name, param in fake_score_transformer3d.named_parameters():
@@ -1121,7 +1127,7 @@ def main():
high_lr_flag = True
fake_trainable_params_optim[0]['params'].append(param)
if accelerator.is_main_process:
print(f"Set {name} to lr : {args.learning_rate}")
print(f"Set {name} to lr : {args.learning_rate_critic}")
break
if high_lr_flag:
continue
@@ -1130,7 +1136,7 @@ def main():
in_already.append(name)
fake_trainable_params_optim[1]['params'].append(param)
if accelerator.is_main_process:
print(f"Set {name} to lr : {args.learning_rate / 2}")
print(f"Set {name} to lr : {args.learning_rate_critic / 2}")
break
if args.use_came:
@@ -1501,7 +1507,7 @@ def main():
)
fake_score_lr_scheduler = get_scheduler(
args.lr_scheduler,
optimizer=optimizer,
optimizer=critic_optimizer,
num_warmup_steps=args.lr_warmup_steps * accelerator.num_processes,
num_training_steps=args.max_train_steps * accelerator.num_processes,
)
@@ -1625,7 +1631,55 @@ def main():
vae_stream_2 = None
idx_sampling = DiscreteSampling(args.train_sampling_steps, uniform_sampling=args.uniform_sampling)
denoising_step_list = noise_scheduler.timesteps[args.train_sampling_steps - torch.tensor(args.denoising_step_indices_list)]
def randomize_denoising_step_indices(
denoising_step_indices_list,
train_sampling_steps,
torch_rng,
accelerator,
jitter_ratio=0.3,
):
indices = list(denoising_step_indices_list)
n = len(indices)
if n <= 2:
low = indices[1]
high = indices[0] - 1
random_tail = torch.randint(low, high + 1, (1,)).item()
result = torch.tensor([indices[0], random_tail])
else:
result = [0] * n
result[0] = indices[0]
result[-1] = indices[-1]
for i in range(1, n - 1):
gap_upper = indices[i - 1] - indices[i]
gap_lower = indices[i] - indices[i + 1]
max_jitter = int(min(gap_upper, gap_lower) * jitter_ratio)
if max_jitter > 0:
jitter = torch.randint(
-max_jitter, max_jitter + 1, (1,)
).item()
else:
jitter = 0
result[i] = indices[i] + jitter
for i in range(1, n):
if result[i] >= result[i - 1]:
result[i] = result[i - 1] - 1
result = [max(1, min(train_sampling_steps, x)) for x in result]
result = torch.tensor(result)
if dist.is_initialized():
result = result.to(accelerator.device)
dist.broadcast(result, src=0)
result = result.cpu()
return result
for epoch in range(first_epoch, args.num_train_epochs):
train_dmd_loss = 0.0
@@ -1946,6 +2000,20 @@ def main():
x0_pred = xt - sigma_t * flow_pred
return x0_pred.to(original_dtype)
# Create discrete denoising steps (per-step, with optional randomization)
if getattr(args, 'randomize_step_indices', False):
random_indices = randomize_denoising_step_indices(
args.denoising_step_indices_list,
args.train_sampling_steps,
torch_rng,
accelerator,
jitter_ratio=getattr(args, 'index_jitter_ratio', 0.30),
)
else:
random_indices = torch.tensor(args.denoising_step_indices_list)
denoising_step_list = noise_scheduler.timesteps[args.train_sampling_steps - random_indices]
# --- Main Training Logic ---
bsz, channel, num_frames, height, width = target_shape
if step % args.gen_update_interval == 0:
scripts/turbodiffusion/train_distill.sh
+1 -1
View File
@@ -24,7 +24,7 @@ accelerate launch --mixed_precision="bf16" scripts/turbodiffusion/train_distill.
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
scripts/wan2.1/README_TRAIN.md
+7 -6
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Option 3: Using Docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**Absolute Path**:
@@ -146,7 +147,7 @@ If your data uses absolute paths, configure in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -228,8 +229,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----------|-------------|---------------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/Wan2.1-T2V-1.3B` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size | 1 |
| `--image_sample_size` | Maximum training resolution for images | 640 |
| `--video_sample_size` | Maximum training resolution for videos | 640 |
scripts/wan2.1/README_TRAIN_DISTILL.md
+13 -12
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -146,7 +147,7 @@ If your data uses absolute paths, configure in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (such as NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -202,7 +203,7 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -229,8 +230,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/Wan2.1-T2V-1.3B/` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size per GPU | 1 |
| `--image_sample_size` | Maximum image training resolution | 640 |
| `--video_sample_size` | Maximum video training resolution | 640 |
@@ -242,7 +243,7 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| `--num_train_epochs` | Number of training epochs | 100 |
| `--checkpointing_steps` | Save checkpoint every N steps | 50 |
| `--learning_rate` | Initial learning rate (generator) | 2e-06 |
| `--learning_rate_critic` | Initial learning rate (critic) | 2e-07 |
| `--learning_rate_critic` | Initial learning rate (critic) | 2e-06 |
| `--lr_scheduler` | Learning rate scheduler | `constant_with_warmup` |
| `--lr_warmup_steps` | Learning rate warmup steps | 100 |
| `--seed` | Random seed | 42 |
@@ -360,7 +361,7 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -418,7 +419,7 @@ accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -468,7 +469,7 @@ accelerate launch --mixed_precision="bf16" scripts/wan2.1/train_distill.py \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -528,7 +529,7 @@ accelerate launch --mixed_precision="bf16" --main_process_ip=$MASTER_ADDR --main
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
scripts/wan2.1/README_TRAIN_DISTILL_LORA.md
+19 -18
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**Absolute Path**:
@@ -146,7 +147,7 @@ If your data uses absolute paths, configure in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -179,7 +180,7 @@ The difference between DeepSpeed-Zero-2 and FSDP in this repository is whether m
```bash
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-T2V-1.3B/"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -202,7 +203,7 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-05 \
--learning_rate_critic=1e-06 \
--learning_rate_critic=1e-05 \
--seed=42 \
--output_dir="output_dir_wan2.1_distill_lora" \
--gradient_checkpointing \
@@ -248,8 +249,8 @@ In addition to distillation training, LoRA training adds the following specific
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/Wan2.1-T2V-1.3B/` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Number of samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution for images | 640 |
| `--video_sample_size` | Maximum training resolution for videos | 640 |
@@ -261,7 +262,7 @@ In addition to distillation training, LoRA training adds the following specific
| `--num_train_epochs` | Number of training epochs | 100 |
| `--checkpointing_steps` | Save checkpoint every N steps | 50 |
| `--learning_rate` | Initial learning rate (generator) | 1e-05 |
| `--learning_rate_critic` | Initial learning rate (discriminator) | 1e-06 |
| `--learning_rate_critic` | Initial learning rate (discriminator) | 1e-05 |
| `--seed` | Random seed | 42 |
| `--output_dir` | Output directory | `output_dir_wan2.1_distill_lora` |
| `--gradient_checkpointing` | Activation recomputation | - |
@@ -353,7 +354,7 @@ If VRAM is insufficient when using multiple GPUs with DeepSpeed-Zero-2, you can
```bash
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-T2V-1.3B/"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -376,7 +377,7 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-05 \
--learning_rate_critic=1e-06 \
--learning_rate_critic=1e-05 \
--seed=42 \
--output_dir="output_dir_wan2.1_distill_lora" \
--gradient_checkpointing \
@@ -414,7 +415,7 @@ The training shell command is as follows:
```bash
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-T2V-1.3B/"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -437,7 +438,7 @@ accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-05 \
--learning_rate_critic=1e-06 \
--learning_rate_critic=1e-05 \
--seed=42 \
--output_dir="output_dir_wan2.1_distill_lora" \
--gradient_checkpointing \
@@ -461,7 +462,7 @@ accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag
```bash
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-T2V-1.3B/"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -484,7 +485,7 @@ accelerate launch --mixed_precision="bf16" scripts/wan2.1/train_distill_lora.py
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-05 \
--learning_rate_critic=1e-06 \
--learning_rate_critic=1e-05 \
--seed=42 \
--output_dir="output_dir_wan2.1_distill_lora" \
--gradient_checkpointing \
@@ -517,7 +518,7 @@ Assuming 2 machines, each with 8 GPUs:
```bash
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-T2V-1.3B/"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
export MASTER_ADDR="192.168.1.100" # Master machine IP
export MASTER_PORT=10086
export WORLD_SIZE=2 # Total number of machines
@@ -545,7 +546,7 @@ accelerate launch --mixed_precision="bf16" --main_process_ip=$MASTER_ADDR --main
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-05 \
--learning_rate_critic=1e-06 \
--learning_rate_critic=1e-05 \
--seed=42 \
--output_dir="output_dir_wan2.1_distill_lora" \
--gradient_checkpointing \
@@ -570,7 +571,7 @@ accelerate launch --mixed_precision="bf16" --main_process_ip=$MASTER_ADDR --main
```bash
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-T2V-1.3B/"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
export MASTER_ADDR="192.168.1.100" # Same as Master
export MASTER_PORT=10086
export WORLD_SIZE=2
scripts/wan2.1/README_TRAIN_DISTILL_LORA_zh-CN.md
+19 -18
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3:使用 docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径,则在训练脚本中设置:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**绝对路径**:
@@ -146,7 +147,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**:如果数据集较小且存储在本地,推荐使用相对路径;如果数据集存储在外部存储(如 NAS、OSS)或多个机器共享存储,推荐使用绝对路径。
@@ -179,7 +180,7 @@ modelscope download --model Wan-AI/Wan2.1-T2V-1.3B --local_dir models/Diffusion_
```bash
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-T2V-1.3B/"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -202,7 +203,7 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-05 \
--learning_rate_critic=1e-06 \
--learning_rate_critic=1e-05 \
--seed=42 \
--output_dir="output_dir_wan2.1_distill_lora" \
--gradient_checkpointing \
@@ -248,8 +249,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Wan2.1-T2V-1.3B/` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 图像最大训练分辨率 | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
@@ -261,7 +262,7 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| `--num_train_epochs` | 训练 epoch 数 | 100 |
| `--checkpointing_steps` | 每 N 步保存 checkpoint | 50 |
| `--learning_rate` | 初始学习率(生成器) | 1e-05 |
| `--learning_rate_critic` | 初始学习率(判别器) | 1e-06 |
| `--learning_rate_critic` | 初始学习率(判别器) | 1e-05 |
| `--seed` | 随机种子 | 42 |
| `--output_dir` | 输出目录 | `output_dir_wan2.1_distill_lora` |
| `--gradient_checkpointing` | 激活重计算 | - |
@@ -353,7 +354,7 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
```bash
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-T2V-1.3B/"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -376,7 +377,7 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-05 \
--learning_rate_critic=1e-06 \
--learning_rate_critic=1e-05 \
--seed=42 \
--output_dir="output_dir_wan2.1_distill_lora" \
--gradient_checkpointing \
@@ -414,7 +415,7 @@ python scripts/zero_to_bf16.py output_dir/checkpoint-{your-num-steps} output_dir
```bash
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-T2V-1.3B/"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -437,7 +438,7 @@ accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-05 \
--learning_rate_critic=1e-06 \
--learning_rate_critic=1e-05 \
--seed=42 \
--output_dir="output_dir_wan2.1_distill_lora" \
--gradient_checkpointing \
@@ -461,7 +462,7 @@ accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag
```bash
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-T2V-1.3B/"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 are used in multi nodes without RDMA.
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
@@ -484,7 +485,7 @@ accelerate launch --mixed_precision="bf16" scripts/wan2.1/train_distill_lora.py
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-05 \
--learning_rate_critic=1e-06 \
--learning_rate_critic=1e-05 \
--seed=42 \
--output_dir="output_dir_wan2.1_distill_lora" \
--gradient_checkpointing \
@@ -517,7 +518,7 @@ accelerate launch --mixed_precision="bf16" scripts/wan2.1/train_distill_lora.py
```bash
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-T2V-1.3B/"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
export MASTER_ADDR="192.168.1.100" # Master 机器 IP
export MASTER_PORT=10086
export WORLD_SIZE=2 # 机器总数
@@ -545,7 +546,7 @@ accelerate launch --mixed_precision="bf16" --main_process_ip=$MASTER_ADDR --main
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-05 \
--learning_rate_critic=1e-06 \
--learning_rate_critic=1e-05 \
--seed=42 \
--output_dir="output_dir_wan2.1_distill_lora" \
--gradient_checkpointing \
@@ -570,7 +571,7 @@ accelerate launch --mixed_precision="bf16" --main_process_ip=$MASTER_ADDR --main
```bash
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-T2V-1.3B/"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata.json"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
export MASTER_ADDR="192.168.1.100" # 与 Master 相同
export MASTER_PORT=10086
export WORLD_SIZE=2
scripts/wan2.1/README_TRAIN_DISTILL_zh-CN.md
+13 -12
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -146,7 +147,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -202,7 +203,7 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -229,8 +230,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Wan2.1-T2V-1.3B/` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 图像最大训练分辨率 | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
@@ -242,7 +243,7 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| `--num_train_epochs` | 训练 epoch 数 | 100 |
| `--checkpointing_steps` | 每 N 步保存 checkpoint | 50 |
| `--learning_rate` | 初始学习率（生成器） | 2e-06 |
| `--learning_rate_critic` | 初始学习率（判别器） | 2e-07 |
| `--learning_rate_critic` | 初始学习率（判别器） | 2e-06 |
| `--lr_scheduler` | 学习率调度器 | `constant_with_warmup` |
| `--lr_warmup_steps` | 学习率预热步数 | 100 |
| `--seed` | 随机种子 | 42 |
@@ -360,7 +361,7 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -418,7 +419,7 @@ accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -468,7 +469,7 @@ accelerate launch --mixed_precision="bf16" scripts/wan2.1/train_distill.py \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -528,7 +529,7 @@ accelerate launch --mixed_precision="bf16" --main_process_ip=$MASTER_ADDR --main
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
scripts/wan2.1/README_TRAIN_LORA.md
+7 -6
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, set in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**Absolute Path**:
@@ -146,7 +147,7 @@ If your data uses absolute paths, set in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Suggestion**: If the dataset is small and stored locally, relative paths are recommended. If the dataset is stored on external storage (such as NAS, OSS) or shared across multiple machines, absolute paths are recommended.
@@ -230,8 +231,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
|-----------|-------------|---------------|
| `--config_path` | Configuration file path | `config/wan2.1/wan_civitai.yaml` |
| `--pretrained_model_name_or_path` | Pre-trained model path | `models/Diffusion_Transformer/Wan2.1-T2V-1.3B` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Number of samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution for images | 640 |
| `--video_sample_size` | Maximum training resolution for videos | 640 |
scripts/wan2.1/README_TRAIN_LORA_zh-CN.md
+7 -6
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -146,7 +147,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -230,8 +231,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
|-----|------|-------|
| `--config_path` | 配置文件路径 | `config/wan2.1/wan_civitai.yaml` |
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Wan2.1-T2V-1.3B` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 图像最大训练分辨率 | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
scripts/wan2.1/README_TRAIN_zh-CN.md
+7 -6
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -146,7 +147,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -228,8 +229,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Wan2.1-T2V-1.3B` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 图像最大训练分辨率 | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
scripts/wan2.1/train.py
-5
View File
@@ -1047,11 +1047,6 @@ def main():
if args.gradient_checkpointing:
transformer3d.enable_gradient_checkpointing()
elif args.selective_ac > 0:
from videox_fun.models.wan_transformer3d import WanAttentionBlock
from videox_fun.utils.ac_handle import apply_checkpointing, partial
apply_selective_ac = partial(apply_checkpointing, block=WanAttentionBlock)
apply_selective_ac(transformer3d, p=args.selective_ac)
# Enable TF32 for faster training on Ampere GPUs,
# cf https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices
scripts/wan2.1/train_distill.py
+74 -6
View File
@@ -729,6 +729,11 @@ def parse_args():
default=[1000, 750, 500, 250],
help="The denoising step list.",
)
parser.add_argument(
"--randomize_step_indices",
action="store_true",
help="whether to use randomize timesteps indices in training.",
)
args = parser.parse_args()
env_local_rank = int(os.environ.get("LOCAL_RANK", -1))
@@ -856,6 +861,7 @@ def main():
weight_dtype = torch.bfloat16
args.mixed_precision = accelerator.mixed_precision
args.denoising_step_indices_list = [int(i) for i in args.denoising_step_indices_list]
# Load scheduler, tokenizer and models.
noise_scheduler = FlowMatchEulerDiscreteScheduler(
**filter_kwargs(FlowMatchEulerDiscreteScheduler, OmegaConf.to_container(config['scheduler_kwargs']))
@@ -1106,8 +1112,8 @@ def main():
fake_trainable_params = list(filter(lambda p: p.requires_grad, fake_score_transformer3d.parameters()))
fake_trainable_params_optim = [
{'params': [], 'lr': args.learning_rate},
{'params': [], 'lr': args.learning_rate / 2},
{'params': [], 'lr': args.learning_rate_critic},
{'params': [], 'lr': args.learning_rate_critic / 2},
]
in_already = []
for name, param in fake_score_transformer3d.named_parameters():
@@ -1120,7 +1126,7 @@ def main():
high_lr_flag = True
fake_trainable_params_optim[0]['params'].append(param)
if accelerator.is_main_process:
print(f"Set {name} to lr : {args.learning_rate}")
print(f"Set {name} to lr : {args.learning_rate_critic}")
break
if high_lr_flag:
continue
@@ -1129,7 +1135,7 @@ def main():
in_already.append(name)
fake_trainable_params_optim[1]['params'].append(param)
if accelerator.is_main_process:
print(f"Set {name} to lr : {args.learning_rate / 2}")
print(f"Set {name} to lr : {args.learning_rate_critic / 2}")
break
if args.use_came:
@@ -1500,7 +1506,7 @@ def main():
)
fake_score_lr_scheduler = get_scheduler(
args.lr_scheduler,
optimizer=optimizer,
optimizer=critic_optimizer,
num_warmup_steps=args.lr_warmup_steps * accelerator.num_processes,
num_training_steps=args.max_train_steps * accelerator.num_processes,
)
@@ -1623,7 +1629,55 @@ def main():
vae_stream_2 = None
idx_sampling = DiscreteSampling(args.train_sampling_steps, uniform_sampling=args.uniform_sampling)
denoising_step_list = noise_scheduler.timesteps[args.train_sampling_steps - torch.tensor(args.denoising_step_indices_list)]
def randomize_denoising_step_indices(
denoising_step_indices_list,
train_sampling_steps,
torch_rng,
accelerator,
jitter_ratio=0.3,
):
indices = list(denoising_step_indices_list)
n = len(indices)
if n <= 2:
low = indices[1]
high = indices[0] - 1
random_tail = torch.randint(low, high + 1, (1,)).item()
result = torch.tensor([indices[0], random_tail])
else:
result = [0] * n
result[0] = indices[0]
result[-1] = indices[-1]
for i in range(1, n - 1):
gap_upper = indices[i - 1] - indices[i]
gap_lower = indices[i] - indices[i + 1]
max_jitter = int(min(gap_upper, gap_lower) * jitter_ratio)
if max_jitter > 0:
jitter = torch.randint(
-max_jitter, max_jitter + 1, (1,)
).item()
else:
jitter = 0
result[i] = indices[i] + jitter
for i in range(1, n):
if result[i] >= result[i - 1]:
result[i] = result[i - 1] - 1
result = [max(1, min(train_sampling_steps, x)) for x in result]
result = torch.tensor(result)
if dist.is_initialized():
result = result.to(accelerator.device)
dist.broadcast(result, src=0)
result = result.cpu()
return result
for epoch in range(first_epoch, args.num_train_epochs):
train_dmd_loss = 0.0
@@ -1944,6 +1998,20 @@ def main():
x0_pred = xt - sigma_t * flow_pred
return x0_pred.to(original_dtype)
# Create discrete denoising steps (per-step, with optional randomization)
if getattr(args, 'randomize_step_indices', False):
random_indices = randomize_denoising_step_indices(
args.denoising_step_indices_list,
args.train_sampling_steps,
torch_rng,
accelerator,
jitter_ratio=getattr(args, 'index_jitter_ratio', 0.30),
)
else:
random_indices = torch.tensor(args.denoising_step_indices_list)
denoising_step_list = noise_scheduler.timesteps[args.train_sampling_steps - random_indices]
# --- Main Training Logic ---
bsz, channel, num_frames, height, width = target_shape
if step % args.gen_update_interval == 0:
scripts/wan2.1/train_distill.sh
+1 -1
View File
@@ -23,7 +23,7 @@ accelerate launch --mixed_precision="bf16" scripts/wan2.1/train_distill.py \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=2e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
scripts/wan2.1/train_distill_lora.py
+71 -3
View File
@@ -744,6 +744,11 @@ def parse_args():
default=[1000, 750, 500, 250],
help="The denoising step list.",
)
parser.add_argument(
"--randomize_step_indices",
action="store_true",
help="whether to use randomize timesteps indices in training.",
)
parser.add_argument(
"--lora_skip_name",
type=str,
@@ -883,6 +888,7 @@ def main():
weight_dtype = torch.bfloat16
args.mixed_precision = accelerator.mixed_precision
args.denoising_step_indices_list = [int(i) for i in args.denoising_step_indices_list]
# Load scheduler, tokenizer and models.
noise_scheduler = FlowMatchEulerDiscreteScheduler(
**filter_kwargs(FlowMatchEulerDiscreteScheduler, OmegaConf.to_container(config['scheduler_kwargs']))
@@ -1151,7 +1157,7 @@ def main():
logging.info("Add fake_score_network parameters")
fake_trainable_params = list(filter(lambda p: p.requires_grad, fake_score_network.parameters()))
fake_trainable_params_optim = fake_score_network.prepare_optimizer_params(args.learning_rate / 2, args.learning_rate, args.learning_rate)
fake_trainable_params_optim = fake_score_network.prepare_optimizer_params(args.learning_rate_critic / 2, args.learning_rate_critic, args.learning_rate_critic)
if args.use_came:
optimizer = optimizer_cls(
@@ -1521,7 +1527,7 @@ def main():
)
fake_score_lr_scheduler = get_scheduler(
args.lr_scheduler,
optimizer=optimizer,
optimizer=critic_optimizer,
num_warmup_steps=args.lr_warmup_steps * accelerator.num_processes,
num_training_steps=args.max_train_steps * accelerator.num_processes,
)
@@ -1664,7 +1670,55 @@ def main():
vae_stream_2 = None
idx_sampling = DiscreteSampling(args.train_sampling_steps, uniform_sampling=args.uniform_sampling)
denoising_step_list = noise_scheduler.timesteps[args.train_sampling_steps - torch.tensor(args.denoising_step_indices_list)]
def randomize_denoising_step_indices(
denoising_step_indices_list,
train_sampling_steps,
torch_rng,
accelerator,
jitter_ratio=0.3,
):
indices = list(denoising_step_indices_list)
n = len(indices)
if n <= 2:
low = indices[1]
high = indices[0] - 1
random_tail = torch.randint(low, high + 1, (1,)).item()
result = torch.tensor([indices[0], random_tail])
else:
result = [0] * n
result[0] = indices[0]
result[-1] = indices[-1]
for i in range(1, n - 1):
gap_upper = indices[i - 1] - indices[i]
gap_lower = indices[i] - indices[i + 1]
max_jitter = int(min(gap_upper, gap_lower) * jitter_ratio)
if max_jitter > 0:
jitter = torch.randint(
-max_jitter, max_jitter + 1, (1,)
).item()
else:
jitter = 0
result[i] = indices[i] + jitter
for i in range(1, n):
if result[i] >= result[i - 1]:
result[i] = result[i - 1] - 1
result = [max(1, min(train_sampling_steps, x)) for x in result]
result = torch.tensor(result)
if dist.is_initialized():
result = result.to(accelerator.device)
dist.broadcast(result, src=0)
result = result.cpu()
return result
for epoch in range(first_epoch, args.num_train_epochs):
train_dmd_loss = 0.0
@@ -1985,6 +2039,20 @@ def main():
x0_pred = xt - sigma_t * flow_pred
return x0_pred.to(original_dtype)
# Create discrete denoising steps (per-step, with optional randomization)
if getattr(args, 'randomize_step_indices', False):
random_indices = randomize_denoising_step_indices(
args.denoising_step_indices_list,
args.train_sampling_steps,
torch_rng,
accelerator,
jitter_ratio=getattr(args, 'index_jitter_ratio', 0.30),
)
else:
random_indices = torch.tensor(args.denoising_step_indices_list)
denoising_step_list = noise_scheduler.timesteps[args.train_sampling_steps - random_indices]
# --- Main Training Logic ---
bsz, channel, num_frames, height, width = target_shape
if step % args.gen_update_interval == 0:
scripts/wan2.1/train_distill_lora.sh
+1 -1
View File
@@ -23,7 +23,7 @@ accelerate launch --mixed_precision="bf16" scripts/wan2.1/train_distill_lora.py
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=1e-05 \
--learning_rate_critic=1e-06 \
--learning_rate_critic=1e-05 \
--seed=42 \
--output_dir="output_dir_wan2.1_distill_lora" \
--gradient_checkpointing \
scripts/wan2.1/train_reward_lora.py
+4 -1
View File
@@ -38,7 +38,10 @@ from accelerate import Accelerator
from accelerate.logging import get_logger
from accelerate.state import AcceleratorState
from accelerate.utils import ProjectConfiguration, set_seed
from decord import VideoReader
try:
from decord import VideoReader
except ImportError:
from videox_fun.data.utils import AVVideoReader as VideoReader
from diffusers import DDIMScheduler, FlowMatchEulerDiscreteScheduler
from diffusers.optimization import get_scheduler
from diffusers.utils import check_min_version, is_wandb_available
scripts/wan2.1_fun/README_TRAIN.md
+7 -6
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Option 3: Using Docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**Absolute Path**:
@@ -146,7 +147,7 @@ If your data uses absolute paths, configure in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -228,8 +229,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----------|-------------|---------------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/Wan2.1-Fun-V1.1-1.3B-InP` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size | 1 |
| `--image_sample_size` | Maximum training resolution for images | 640 |
| `--video_sample_size` | Maximum training resolution for videos | 640 |
scripts/wan2.1_fun/README_TRAIN_CONTROL.md
+7 -6
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -145,8 +146,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Controls-Demo --local_dir ./datas
If your data uses relative paths, configure in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata_control.json"
export DATASET_NAME="datasets/X-Fun-Videos-Controls-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Controls-Demo/metadata_add_width_height.json"
```
**Absolute Path**:
@@ -155,7 +156,7 @@ If your data uses absolute paths, configure in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata_control.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -239,8 +240,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
|-----|------|-------|
| `--config_path` | Model config file path | `config/wan2.1/wan_civitai.yaml` |
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/Wan2.1-Fun-V1.1-14B-Control` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata_control.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Controls-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Controls-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size per GPU | 1 |
| `--image_sample_size` | Maximum image training resolution | 640 |
| `--video_sample_size` | Maximum video training resolution | 640 |
scripts/wan2.1_fun/README_TRAIN_CONTROL_LORA.md
+7 -6
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -145,8 +146,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Controls-Demo --local_dir ./datas
If your data uses relative paths, configure in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata_control.json"
export DATASET_NAME="datasets/X-Fun-Videos-Controls-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Controls-Demo/metadata_add_width_height.json"
```
**Absolute Path**:
@@ -155,7 +156,7 @@ If your data uses absolute paths, configure in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata_control.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -240,8 +241,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
|-----|------|-------|
| `--config_path` | Model config file path | `config/wan2.1/wan_civitai.yaml` |
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/Wan2.1-Fun-V1.1-14B-Control` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata_control.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Controls-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Controls-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size per GPU | 1 |
| `--image_sample_size` | Maximum image training resolution | 640 |
| `--video_sample_size` | Maximum video training resolution | 640 |
scripts/wan2.1_fun/README_TRAIN_CONTROL_LORA_zh-CN.md
+7 -6
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -145,8 +146,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Controls-Demo --local_dir ./datas
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata_control.json"
export DATASET_NAME="datasets/X-Fun-Videos-Controls-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Controls-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -155,7 +156,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata_control.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata_control.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -240,8 +241,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
|-----|------|-------|
| `--config_path` | 配置文件路径 | `config/wan2.1/wan_civitai.yaml` |
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Wan2.1-Fun-V1.1-14B-Control` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata_control.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Controls-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Controls-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 图像最大训练分辨率 | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
scripts/wan2.1_fun/README_TRAIN_CONTROL_zh-CN.md
+7 -6
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -145,8 +146,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Controls-Demo --local_dir ./datas
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata_control.json"
export DATASET_NAME="datasets/X-Fun-Videos-Controls-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Controls-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -155,7 +156,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata_control.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata_control.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -239,8 +240,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
|-----|------|-------|
| `--config_path` | 模型配置文件路径 | `config/wan2.1/wan_civitai.yaml` |
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Wan2.1-Fun-V1.1-14B-Control` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata_control.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Controls-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Controls-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 图像最大训练分辨率 | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
scripts/wan2.1_fun/README_TRAIN_LORA.md
+7 -6
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, set in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**Absolute Path**:
@@ -146,7 +147,7 @@ If your data uses absolute paths, set in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Suggestion**: If the dataset is small and stored locally, relative paths are recommended. If the dataset is stored on external storage (such as NAS, OSS) or shared across multiple machines, absolute paths are recommended.
@@ -230,8 +231,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
|-----------|-------------|---------------|
| `--config_path` | Configuration file path | `config/wan2.1/wan_civitai.yaml` |
| `--pretrained_model_name_or_path` | Pre-trained model path | `models/Diffusion_Transformer/Wan2.1-Fun-V1.1-1.3B-InP` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Number of samples per batch | 1 |
| `--image_sample_size` | Maximum training resolution for images | 640 |
| `--video_sample_size` | Maximum training resolution for videos | 640 |
scripts/wan2.1_fun/README_TRAIN_LORA_zh-CN.md
+7 -6
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -146,7 +147,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -230,8 +231,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
|-----|------|-------|
| `--config_path` | 配置文件路径 | `config/wan2.1/wan_civitai.yaml` |
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Wan2.1-Fun-V1.1-1.3B-InP` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 图像最大训练分辨率 | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
scripts/wan2.1_fun/README_TRAIN_zh-CN.md
+7 -6
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -136,8 +137,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -146,7 +147,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -228,8 +229,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Wan2.1-Fun-V1.1-1.3B-InP` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 图像最大训练分辨率 | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
scripts/wan2.1_fun/train_reward_lora.py
+4 -1
View File
@@ -39,7 +39,10 @@ from accelerate import Accelerator
from accelerate.logging import get_logger
from accelerate.state import AcceleratorState
from accelerate.utils import ProjectConfiguration, set_seed
from decord import VideoReader
try:
from decord import VideoReader
except ImportError:
from videox_fun.data.utils import AVVideoReader as VideoReader
from diffusers import DDIMScheduler, FlowMatchEulerDiscreteScheduler
from diffusers.optimization import get_scheduler
from diffusers.utils import check_min_version, is_wandb_available
scripts/wan2.1_self_forcing/README_TRAIN_DISTILL.md
+13 -12
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
@@ -135,8 +136,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
If your data uses relative paths, configure in the training script:
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**Absolute Paths**:
@@ -145,7 +146,7 @@ If your data uses absolute paths, configure in the training script:
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **Recommendation**: If the dataset is small and stored locally, use relative paths. If the dataset is stored on external storage (such as NAS, OSS) or shared across multiple machines, use absolute paths.
@@ -203,7 +204,7 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=4e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -233,8 +234,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| Parameter | Description | Example Value |
|-----|------|-------|
| `--pretrained_model_name_or_path` | Pretrained model path | `models/Diffusion_Transformer/Wan2.1-T2V-1.3B/` |
| `--train_data_dir` | Training data directory | `datasets/internal_datasets/` |
| `--train_data_meta` | Training data metadata file | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | Training data directory | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | Training data metadata file | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | Batch size per GPU | 1 |
| `--image_sample_size` | Maximum image training resolution | 640 |
| `--video_sample_size` | Maximum video training resolution | 640 |
@@ -246,7 +247,7 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| `--num_train_epochs` | Number of training epochs | 100 |
| `--checkpointing_steps` | Save checkpoint every N steps | 50 |
| `--learning_rate` | Initial learning rate (generator) | 2e-06 |
| `--learning_rate_critic` | Initial learning rate (critic) | 2e-07 |
| `--learning_rate_critic` | Initial learning rate (critic) | 2e-06 |
| `--lr_scheduler` | Learning rate scheduler | `constant_with_warmup` |
| `--lr_warmup_steps` | Learning rate warmup steps | 100 |
| `--seed` | Random seed | 42 |
@@ -366,7 +367,7 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=4e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -428,7 +429,7 @@ accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=4e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -482,7 +483,7 @@ accelerate launch --mixed_precision="bf16" scripts/wan2.1_self_forcing/train_dis
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=4e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -546,7 +547,7 @@ accelerate launch --mixed_precision="bf16" --main_process_ip=$MASTER_ADDR --main
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=4e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
scripts/wan2.1_self_forcing/README_TRAIN_DISTILL_zh-CN.md
+13 -12
View File
@@ -43,9 +43,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用docker**
@@ -135,8 +136,8 @@ modelscope download --dataset PAI/X-Fun-Videos-Demo --local_dir ./datasets/X-Fun
如果数据的路径为相对路径，则在训练脚本中设置：
```bash
export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
export DATASET_NAME="datasets/X-Fun-Videos-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Demo/metadata_add_width_height.json"
```
**绝对路径**：
@@ -145,7 +146,7 @@ export DATASET_META_NAME="datasets/internal_datasets/metadata.json"
```bash
export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/metadata.json"
export DATASET_META_NAME="/mnt/data/metadata_add_width_height.json"
```
> 💡 **建议**：如果数据集较小且存储在本地，推荐使用相对路径；如果数据集存储在外部存储（如 NAS、OSS）或多个机器共享存储，推荐使用绝对路径。
@@ -204,7 +205,7 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=4e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -234,8 +235,8 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| 参数 | 说明 | 示例值 |
|-----|------|-------|
| `--pretrained_model_name_or_path` | 预训练模型路径 | `models/Diffusion_Transformer/Wan2.1-T2V-1.3B/` |
| `--train_data_dir` | 训练数据目录 | `datasets/internal_datasets/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/internal_datasets/metadata.json` |
| `--train_data_dir` | 训练数据目录 | `datasets/X-Fun-Videos-Demo/` |
| `--train_data_meta` | 训练数据元文件 | `datasets/X-Fun-Videos-Demo/metadata_add_width_height.json` |
| `--train_batch_size` | 每批次样本数 | 1 |
| `--image_sample_size` | 图像最大训练分辨率 | 640 |
| `--video_sample_size` | 视频最大训练分辨率 | 640 |
@@ -247,7 +248,7 @@ accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_con
| `--num_train_epochs` | 训练 epoch 数 | 100 |
| `--checkpointing_steps` | 每 N 步保存 checkpoint | 50 |
| `--learning_rate` | 初始学习率（生成器） | 2e-06 |
| `--learning_rate_critic` | 初始学习率（判别器） | 2e-07 |
| `--learning_rate_critic` | 初始学习率（判别器） | 2e-06 |
| `--lr_scheduler` | 学习率调度器 | `constant_with_warmup` |
| `--lr_warmup_steps` | 学习率预热步数 | 100 |
| `--seed` | 随机种子 | 42 |
@@ -367,7 +368,7 @@ accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TR
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=4e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -429,7 +430,7 @@ accelerate launch --zero_stage 3 --zero3_save_16bit_model true --zero3_init_flag
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=4e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -483,7 +484,7 @@ accelerate launch --mixed_precision="bf16" scripts/wan2.1_self_forcing/train_dis
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=4e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
@@ -547,7 +548,7 @@ accelerate launch --mixed_precision="bf16" --main_process_ip=$MASTER_ADDR --main
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=4e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
scripts/wan2.1_self_forcing/README_TRAIN_ODE.md
+2 -1
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**Method 3: Using Docker**
scripts/wan2.1_self_forcing/README_TRAIN_ODE_zh-CN.md
+2 -1
View File
@@ -44,9 +44,10 @@ pip install -r requirements.txt
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl deepspeed==0.17.0 numpy==1.26.4
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
```
**方式 3：使用 docker**
scripts/wan2.1_self_forcing/train_distill.py
+303 -30
View File
@@ -117,6 +117,54 @@ def initialize_crossattn_cache_for_training(batch_size, text_len, num_layers, nu
return crossattn_cache
def slice_last_n_latent_frames(tensor, n):
"""Slice last n frames from [B, C, F, H, W] tensor."""
if tensor.shape[2] <= n:
return tensor
return tensor[:, :, -n:]
def reencode_boundary_latent(vae, pred_latents, weight_dtype, score_num_frames=21):
"""
Re-encode the boundary frame to get a clean latent for the score window.
Follows Self-Forcing reference: decode all frames before the score window, take last pixel frame, re-encode.
Input: pred_latents [B, C, F, H, W] (all generated latent frames)
Output: boundary_latent [B, C, 1, H, W]
"""
with torch.no_grad():
# Decode all frames except the last (score_num_frames - 1) to pixels
tail_len = score_num_frames - 1
latent_to_decode = pred_latents[:, :, :-tail_len]
# VAE expects [B, C, F, H, W], decode returns [B, C, F, H, W] pixels
pixels = vae.decode(latent_to_decode.to(vae.dtype)).sample # [B, C, F, H, W]
# Take the last frame
frame = pixels[:, :, -1:, :, :] # [B, C, 1, H, W]
# Re-encode the last frame to get clean boundary latent
boundary_latent = vae.encode(frame)[0].sample().to(weight_dtype) # [B, C, 1, H, W]
return boundary_latent
def slice_for_score(pred, vae, weight_dtype, score_num_frames=21, independent_first_frame=False):
"""
Slice the last `score_num_frames` latent frames for score computation.
If pred has more than score_num_frames, re-encode boundary frame for clean context.
Returns: (pred_for_score, score_num_frames, need_gradient_mask)
"""
num_frames = pred.shape[2]
if num_frames <= score_num_frames:
return pred, num_frames, False
# Re-encode boundary for cleaner score input
try:
boundary_latent = reencode_boundary_latent(vae, pred, weight_dtype, score_num_frames=score_num_frames)
pred_for_score = torch.cat([boundary_latent, pred[:, :, -(score_num_frames - 1):]], dim=2)
except Exception:
# Fallback: simple slice without boundary re-encoding
pred_for_score = pred[:, :, -score_num_frames:]
return pred_for_score, score_num_frames, True
def filter_kwargs(cls, kwargs):
import inspect
sig = inspect.signature(cls.__init__)
@@ -740,6 +788,11 @@ def parse_args():
default=[1000, 750, 500, 250],
help="The denoising step list.",
)
parser.add_argument(
"--randomize_step_indices",
action="store_true",
help="whether to use randomize timesteps indices in training.",
)
parser.add_argument(
"--num_frame_per_block",
type=int,
@@ -756,6 +809,22 @@ def parse_args():
action="store_true",
help="Use KV cache block-by-block training (matches original Self-Forcing)"
)
parser.add_argument(
"--score_num_frames",
type=int,
default=21,
help="Number of latent frames for score computation window (default: 21, matching base model). "
"fake_score/real_score always receive this many frames."
)
parser.add_argument(
"--min_length_prob_bias",
type=float,
default=0.0,
help="Probability bias for sampling the minimum length (score_num_frames). "
"0.0 = uniform sampling (default), 0.5 = 50%% prob for min length, "
"remaining prob distributed equally among longer lengths. "
"Use this to increase 21-frame training ratio."
)
parser.add_argument(
"--context_noise",
type=int,
@@ -900,6 +969,7 @@ def main():
weight_dtype = torch.bfloat16
args.mixed_precision = accelerator.mixed_precision
args.denoising_step_indices_list = [int(i) for i in args.denoising_step_indices_list]
# Load scheduler, tokenizer and models.
noise_scheduler = FlowMatchEulerDiscreteScheduler(
**filter_kwargs(FlowMatchEulerDiscreteScheduler, OmegaConf.to_container(config['scheduler_kwargs']))
@@ -1173,8 +1243,8 @@ def main():
fake_trainable_params = list(filter(lambda p: p.requires_grad, fake_score_transformer3d.parameters()))
fake_trainable_params_optim = [
{'params': [], 'lr': args.learning_rate},
{'params': [], 'lr': args.learning_rate / 2},
{'params': [], 'lr': args.learning_rate_critic},
{'params': [], 'lr': args.learning_rate_critic / 2},
]
in_already = []
for name, param in fake_score_transformer3d.named_parameters():
@@ -1187,7 +1257,7 @@ def main():
high_lr_flag = True
fake_trainable_params_optim[0]['params'].append(param)
if accelerator.is_main_process:
print(f"Set {name} to lr : {args.learning_rate}")
print(f"Set {name} to lr : {args.learning_rate_critic}")
break
if high_lr_flag:
continue
@@ -1196,7 +1266,7 @@ def main():
in_already.append(name)
fake_trainable_params_optim[1]['params'].append(param)
if accelerator.is_main_process:
print(f"Set {name} to lr : {args.learning_rate / 2}")
print(f"Set {name} to lr : {args.learning_rate_critic / 2}")
break
if args.use_came:
@@ -1461,7 +1531,7 @@ def main():
new_examples['encoder_hidden_states'] = prompt_embeds
neg_txt = [
"色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走" for text in batch['text']
"色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走" for text in new_examples['text']
]
neg_prompt_ids = tokenizer(
neg_txt,
@@ -1519,7 +1589,7 @@ def main():
new_examples['encoder_hidden_states'] = prompt_embeds
neg_txt = [
"色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走" for text in batch['text']
"色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走" for text in new_examples['text']
]
neg_prompt_ids = tokenizer(
neg_txt,
@@ -1578,7 +1648,7 @@ def main():
)
fake_score_lr_scheduler = get_scheduler(
args.lr_scheduler,
optimizer=optimizer,
optimizer=critic_optimizer,
num_warmup_steps=args.lr_warmup_steps * accelerator.num_processes,
num_training_steps=args.max_train_steps * accelerator.num_processes,
)
@@ -1701,7 +1771,55 @@ def main():
vae_stream_2 = None
idx_sampling = DiscreteSampling(args.train_sampling_steps, uniform_sampling=args.uniform_sampling)
denoising_step_list = noise_scheduler.timesteps[args.train_sampling_steps - torch.tensor(args.denoising_step_indices_list)]
def randomize_denoising_step_indices(
denoising_step_indices_list,
train_sampling_steps,
torch_rng,
accelerator,
jitter_ratio=0.3,
):
indices = list(denoising_step_indices_list)
n = len(indices)
if n <= 2:
low = indices[1]
high = indices[0] - 1
random_tail = torch.randint(low, high + 1, (1,)).item()
result = torch.tensor([indices[0], random_tail])
else:
result = [0] * n
result[0] = indices[0]
result[-1] = indices[-1]
for i in range(1, n - 1):
gap_upper = indices[i - 1] - indices[i]
gap_lower = indices[i] - indices[i + 1]
max_jitter = int(min(gap_upper, gap_lower) * jitter_ratio)
if max_jitter > 0:
jitter = torch.randint(
-max_jitter, max_jitter + 1, (1,)
).item()
else:
jitter = 0
result[i] = indices[i] + jitter
for i in range(1, n):
if result[i] >= result[i - 1]:
result[i] = result[i - 1] - 1
result = [max(1, min(train_sampling_steps, x)) for x in result]
result = torch.tensor(result)
if dist.is_initialized():
result = result.to(accelerator.device)
dist.broadcast(result, src=0)
result = result.cpu()
return result
for epoch in range(first_epoch, args.num_train_epochs):
train_dmd_loss = 0.0
@@ -1908,10 +2026,25 @@ def main():
local_sample_size = aspect_ratio_sample_size[aspect_ratio_key]
local_sample_size = [int(x / 16) * 16 for x in local_sample_size]
# Compute latent frame count
latent_num_frames = int((num_frames - 1) // vae.temporal_compression_ratio + 1)
# Align latent_num_frames to num_frame_per_block for KV cache training
if args.use_kv_cache_training:
if args.independent_first_frame:
# latent_frames - 1 must be divisible by num_frame_per_block
k = latent_num_frames - 1
k = (k // args.num_frame_per_block) * args.num_frame_per_block
latent_num_frames = k + 1
else:
# latent_frames must be divisible by num_frame_per_block
latent_num_frames = (latent_num_frames // args.num_frame_per_block) * args.num_frame_per_block
latent_num_frames = max(latent_num_frames, args.num_frame_per_block)
target_shape = (
len(text),
vae.latent_channels,
int((num_frames - 1) // vae.temporal_compression_ratio + 1),
latent_num_frames,
int(local_sample_size[0] // vae.spatial_compression_ratio),
int(local_sample_size[1] // vae.spatial_compression_ratio),
)
@@ -2032,6 +2165,20 @@ def main():
x0_pred = xt - sigma_t * flow_pred
return x0_pred.to(original_dtype)
# Create discrete denoising steps (per-step, with optional randomization)
if getattr(args, 'randomize_step_indices', False):
random_indices = randomize_denoising_step_indices(
args.denoising_step_indices_list,
args.train_sampling_steps,
torch_rng,
accelerator,
jitter_ratio=getattr(args, 'index_jitter_ratio', 0.30),
)
else:
random_indices = torch.tensor(args.denoising_step_indices_list)
denoising_step_list = noise_scheduler.timesteps[args.train_sampling_steps - random_indices]
# --- Main Training Logic ---
bsz, channel, num_frames, height, width = target_shape
if step % args.gen_update_interval == 0:
@@ -2040,17 +2187,47 @@ def main():
patch_h, patch_w = accelerator.unwrap_model(generator_transformer3d).config.patch_size[1:]
frame_seq_length = (target_shape[3] * target_shape[4]) // (patch_h * patch_w)
# Determine block structure
# Determine block structure with variable-length support
if not args.independent_first_frame:
assert num_frames % args.num_frame_per_block == 0
num_blocks = num_frames // args.num_frame_per_block
max_num_blocks = num_frames // args.num_frame_per_block
assert args.score_num_frames % args.num_frame_per_block == 0
min_num_blocks = args.score_num_frames // args.num_frame_per_block
else:
assert (num_frames - 1) % args.num_frame_per_block == 0
num_blocks = (num_frames - 1) // args.num_frame_per_block
all_num_frames = [args.num_frame_per_block] * num_blocks
max_num_blocks = (num_frames - 1) // args.num_frame_per_block
if args.score_num_frames > 1:
assert (args.score_num_frames - 1) % args.num_frame_per_block == 0
min_num_blocks = (args.score_num_frames - 1) // args.num_frame_per_block
else:
min_num_blocks = 0
# Random sample number of blocks (Self-Forcing variable-length training)
if args.min_length_prob_bias > 0.0 and max_num_blocks > min_num_blocks:
# Weighted sampling: give min_num_blocks a higher probability
num_options = max_num_blocks - min_num_blocks + 1
bias = min(args.min_length_prob_bias, 0.99)
remaining_prob = (1.0 - bias) / (num_options - 1)
probs = [remaining_prob] * num_options
probs[0] = bias # min_num_blocks gets the bias
probs_tensor = torch.tensor(probs, device=accelerator.device)
block_indices = torch.multinomial(probs_tensor, 1, generator=torch_rng)
num_generated_blocks = (min_num_blocks + block_indices).item()
else:
num_generated_blocks = torch.randint(
min_num_blocks, max_num_blocks + 1, (1,),
generator=torch_rng, device=accelerator.device
).item()
if dist.is_initialized():
_sync = torch.tensor([num_generated_blocks], device=accelerator.device)
dist.broadcast(_sync, src=0)
num_generated_blocks = _sync.item()
all_num_frames = [args.num_frame_per_block] * num_generated_blocks
if args.independent_first_frame:
all_num_frames = [1] + all_num_frames
num_generated_frames = sum(all_num_frames)
# Initialize KV cache
num_layers = generator_transformer3d.config.num_layers
@@ -2098,6 +2275,10 @@ def main():
num_denoising_steps = len(denoising_step_list)
final_step_index = generate_and_sync_list(num_denoising_steps, device=accelerator.device)[0]
# Only blocks in the last score_num_frames get gradient at exit step
# (matches Self-Forcing: start_gradient_frame_index = num_output_frames - 21)
start_gradient_frame_index = num_generated_frames - args.score_num_frames
for block_idx, current_num_frames in enumerate(all_num_frames):
# Extract noise for current block
start_idx = current_start_frame - num_input_frames
@@ -2113,7 +2294,11 @@ def main():
dtype=torch.int64
)
context_manager = torch.no_grad() if not is_final_step else contextlib.nullcontext()
# Only enable gradient for final step AND block within score window
if not is_final_step or current_start_frame < start_gradient_frame_index:
context_manager = torch.no_grad()
else:
context_manager = contextlib.nullcontext()
with context_manager:
# Convert noisy_input to list format
@@ -2194,13 +2379,54 @@ def main():
current_start_frame += current_num_frames
# Final output
generator_pred = output_pred
seq_len = frame_seq_length * num_frames # For fake/real score computation
# Final output — slice generated frames (may be < num_frames for variable-length)
generator_pred_full = output_pred[:, :, :num_generated_frames]
# Gradient mask: first block gets no gradient when generating > min frames
# (matches Self-Forcing reference: model/base.py L182-L190)
min_num_frames_score = args.score_num_frames
need_gradient_mask = (num_generated_frames != min_num_frames_score)
gradient_mask = None
if need_gradient_mask:
gradient_mask = torch.ones_like(generator_pred_full, dtype=torch.bool)
if args.independent_first_frame:
gradient_mask[:, :, :1] = False
else:
gradient_mask[:, :, :args.num_frame_per_block] = False
# Slice for score computation: last score_num_frames frames
if num_generated_frames > args.score_num_frames:
# Re-encode boundary for cleaner score input
generator_pred_for_score, score_num_frames, _ = slice_for_score(
generator_pred_full, vae, weight_dtype,
score_num_frames=args.score_num_frames,
independent_first_frame=args.independent_first_frame,
)
else:
generator_pred_for_score = generator_pred_full
score_num_frames = num_generated_frames
# Compute score_mask for DMD loss (matches Self-Forcing: dmd.py L199-204)
score_mask = None
if gradient_mask is not None:
mask_offset = num_generated_frames - score_num_frames
score_mask = gradient_mask[:, :, mask_offset:mask_offset + score_num_frames]
# generator_pred = the sliced version for DMD loss
generator_pred = generator_pred_for_score
seq_len = frame_seq_length * score_num_frames # Score always on fixed window
else:
# === Block mask training (flex attention, no KV cache) ===
# Block mask training: use flex attention to process entire video at once
# Note: for long videos, use KV cache mode instead
score_mask = None # Block mask mode: no gradient mask needed
if num_frames > args.score_num_frames:
raise ValueError(
f"Block mask mode does not support variable-length training "
f"(video produces {num_frames} latent frames > score_num_frames={args.score_num_frames}). "
f"Use --use_kv_cache_training for long video training."
)
patch_h_bm, patch_w_bm = accelerator.unwrap_model(generator_transformer3d).config.patch_size[1:]
frame_seqlen_bm = (height * width) // (patch_h_bm * patch_w_bm)
@@ -2379,11 +2605,20 @@ def main():
fake_to_real_grad = fake_to_real_grad / normalizer
fake_to_real_grad = torch.nan_to_num(fake_to_real_grad)
dmd_loss = 0.5 * F.mse_loss(
generator_pred.double(),
(generator_pred.double() - fake_to_real_grad.double()).detach(),
reduction="mean"
)
# Apply gradient mask: only compute loss on unmasked elements
# (matches Self-Forcing dmd.py: F.mse_loss(x[mask], target[mask]))
if score_mask is not None:
dmd_loss = 0.5 * F.mse_loss(
generator_pred.double()[score_mask],
(generator_pred.double() - fake_to_real_grad.double()).detach()[score_mask],
reduction="mean"
)
else:
dmd_loss = 0.5 * F.mse_loss(
generator_pred.double(),
(generator_pred.double() - fake_to_real_grad.double()).detach(),
reduction="mean"
)
avg_dmd_loss = accelerator.gather(dmd_loss.repeat(args.train_batch_size)).mean()
train_dmd_loss += avg_dmd_loss.item() / args.gradient_accumulation_steps
@@ -2414,15 +2649,41 @@ def main():
# Calculate frame_seq_length
frame_seq_length = (target_shape[3] * target_shape[4]) // (patch_h * patch_w)
# Determine block structure
# Determine block structure (variable-length, mirrors generator branch)
if not args.independent_first_frame:
num_blocks = num_frames // args.num_frame_per_block
max_num_blocks_critic = num_frames // args.num_frame_per_block
min_num_blocks_critic = args.score_num_frames // args.num_frame_per_block
else:
num_blocks = (num_frames - 1) // args.num_frame_per_block
all_num_frames = [args.num_frame_per_block] * num_blocks
max_num_blocks_critic = (num_frames - 1) // args.num_frame_per_block
if args.score_num_frames > 1:
min_num_blocks_critic = (args.score_num_frames - 1) // args.num_frame_per_block
else:
min_num_blocks_critic = 0
# Random sample number of blocks (mirrors generator's variable-length training)
if args.min_length_prob_bias > 0.0 and max_num_blocks_critic > min_num_blocks_critic:
num_options = max_num_blocks_critic - min_num_blocks_critic + 1
bias = min(args.min_length_prob_bias, 0.99)
remaining_prob = (1.0 - bias) / (num_options - 1)
probs = [remaining_prob] * num_options
probs[0] = bias # min_num_blocks_critic gets the bias
probs_tensor = torch.tensor(probs, device=accelerator.device)
block_indices = torch.multinomial(probs_tensor, 1, generator=torch_rng)
num_generated_blocks_critic = (min_num_blocks_critic + block_indices).item()
else:
num_generated_blocks_critic = torch.randint(
min_num_blocks_critic, max_num_blocks_critic + 1, (1,),
generator=torch_rng, device=accelerator.device
).item()
if dist.is_initialized():
_sync = torch.tensor([num_generated_blocks_critic], device=accelerator.device)
dist.broadcast(_sync, src=0)
num_generated_blocks_critic = _sync.item()
all_num_frames = [args.num_frame_per_block] * num_generated_blocks_critic
if args.independent_first_frame:
all_num_frames = [1] + all_num_frames
num_generated_frames_critic = sum(all_num_frames)
# Initialize KV cache
num_layers = generator_transformer3d.config.num_layers
@@ -2556,8 +2817,20 @@ def main():
current_start_frame += current_num_frames
fake_score_denoised_pred = output_pred
seq_len = frame_seq_length * num_frames
fake_score_denoised_pred_full = output_pred[:, :, :num_generated_frames_critic]
# Slice for critic score: last score_num_frames frames
if num_generated_frames_critic > args.score_num_frames:
fake_score_denoised_pred, critic_score_num_frames, _ = slice_for_score(
fake_score_denoised_pred_full, vae, weight_dtype,
score_num_frames=args.score_num_frames,
independent_first_frame=args.independent_first_frame,
)
else:
fake_score_denoised_pred = fake_score_denoised_pred_full
critic_score_num_frames = num_generated_frames_critic
seq_len = frame_seq_length * critic_score_num_frames
else:
with torch.no_grad():
scripts/wan2.1_self_forcing/train_distill.sh
+1 -1
View File
@@ -24,7 +24,7 @@ accelerate launch --mixed_precision="bf16" scripts/wan2.1_self_forcing/train_dis
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-06 \
--learning_rate_critic=4e-07 \
--learning_rate_critic=2e-06 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \

Some files were not shown because too many files have changed in this diff Show More