27 KiB
Executable File
FantasyTalking-S2V Full Parameter Training Guide
This document provides a complete workflow for full parameter training of FantasyTalking-S2V (an audio-driven digital human video generation model), including environment configuration, data preparation, distributed training, and inference testing.
Note
: FantasyTalking is an audio-driven digital human video generation model that requires both a reference image and an audio file to generate talking videos. Training data needs to include videos, audios, and reference images.
Table of Contents
- 1. Environment Configuration
- 2. Data Preparation
- 3. Full Parameter Training
- 4. Inference Testing
- 5. Additional Resources
1. Environment Configuration
Option 1: Using requirements.txt
pip install -r requirements.txt
Option 2: Manual Installation
pip install Pillow einops safetensors timm tomesd librosa "torch>=2.1.2" torchdiffeq torchsde decord datasets numpy scikit-image
pip install omegaconf SentencePiece imageio[ffmpeg] imageio[pyav] tensorboard beautifulsoup4 ftfy func_timeout onnxruntime
pip install "peft>=0.17.0" "accelerate>=0.25.0" "gradio>=3.41.2" "diffusers>=0.30.1" "transformers>=4.46.2"
pip install yunchang xfuser modelscope openpyxl
pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y
pip install opencv-python-headless
pip install deepspeed==0.17.0 numpy==1.26.4
Option 3: Using Docker
When using Docker, first ensure that GPU drivers and CUDA environment are properly installed on your machine, then execute the following commands:
# Pull the image
docker pull mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:cogvideox_fun
# Run the container
docker run -it -p 7860:7860 --network host --gpus all --security-opt seccomp:unconfined --shm-size 200g mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:cogvideox_fun
2. Data Preparation
2.1 Quick Test Dataset
We provide a test dataset containing several audio-video training samples.
# Download the official demo dataset
modelscope download --dataset PAI/X-Fun-Videos-Audios-Demo --local_dir ./datasets/X-Fun-Videos-Audios-Demo
2.2 Dataset Structure
📦 datasets/
├── 📂 my_dataset/
│ ├── 📂 train/
│ │ ├── 📄 video001.mp4
│ │ ├── 📄 video002.mp4
│ │ └── 📄 ...
│ ├── 📂 wav/
│ │ ├── 📄 audio001.wav
│ │ ├── 📄 audio002.wav
│ │ └── 📄 ...
│ └── 📄 metadata.json
2.3 metadata.json Format
The metadata.json for FantasyTalking is slightly different from the normal JSON format in VideoX-Fun. You need to add an audio_path field.
Relative Path Format (Example):
[
{
"file_path": "train/00000001.mp4",
"audio_path": "wav/00000001.wav",
"text": "A girl talking by the sea.",
"type": "video",
"width": 512,
"height": 512
},
{
"file_path": "train/00000002.mp4",
"audio_path": "wav/00000002.wav",
"text": "A man talking in a room.",
"type": "video",
"width": 512,
"height": 512
}
]
Absolute Path Format:
[
{
"file_path": "/path/to/your/dataset/train/00000001.mp4",
"audio_path": "/path/to/your/dataset/wav/00000001.wav",
"text": "A girl talking by the sea.",
"type": "video",
"width": 512,
"height": 512
}
]
Field Description:
file_path: Relative or absolute path to the video fileaudio_path: Relative or absolute path to the audio file (Required field for FantasyTalking)- Audio files are typically in
.wavformat - Path should correspond to
file_path, e.g.,train/00000001.mp4corresponds towav/00000001.wav
- Audio files are typically in
text: Text description (prompt) for the video (optional)type: Data type, fixed as"video"width/height: Video dimensions (Recommended to provide for bucket training; if not provided, they will be automatically read during training, which may slow down training when data is stored on slow systems like OSS)- You can use
scripts/process_json_add_width_and_height.pyto add width and height fields to JSON files that don't have them, supporting both images and videos - Usage:
python scripts/process_json_add_width_and_height.py --input_file datasets/X-Fun-Videos-Audios-Demo/metadata.json --output_file datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json
- You can use
2.4 Relative vs Absolute Paths
Relative Paths:
If your data uses relative paths, configure the training script as follows:
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
Absolute Paths:
If your data uses absolute paths, configure the training script as follows:
export DATASET_NAME=""
export DATASET_META_NAME="/path/to/your/metadata_add_width_height.json"
💡 Tip: If your dataset is small and stored locally, use relative paths. If your dataset is stored on external storage (e.g., NAS, OSS) or shared across multiple machines, use absolute paths.
3. Full Parameter Training
3.1 Download Pretrained Models
Before training, you need to download the following pretrained models:
# Create model directories
mkdir -p models/Diffusion_Transformer
mkdir -p models/Personalized_Model
# Download Wan2.1-I2V-14B-720P model
modelscope download --model Wan-AI/Wan2.1-I2V-14B-720P --local_dir models/Diffusion_Transformer/Wan2.1-I2V-14B-720P
# Download wav2vec2 audio encoder
modelscope download --model AI-ModelScope/wav2vec2-base-960h --local_dir models/Diffusion_Transformer/wav2vec2-base-960h
# Download FantasyTalking pretrained weights
modelscope download --model amap_cvlab/FantasyTalking --local_dir models/Personalized_Model/FantasyTalking/
3.2 Quick Start (DeepSpeed-Zero-2)
If you have downloaded the data according to 2.1 Quick Test Dataset and the weights according to 3.1 Download Pretrained Models, you can directly copy and run the quick start command.
DeepSpeed-Zero-2 or FSDP is recommended for training. Here we use DeepSpeed-Zero-2 as an example.
The difference between DeepSpeed-Zero-2 and FSDP is whether model weights are sharded. If you run out of VRAM with DeepSpeed-Zero-2 on multiple GPUs, you can switch to FSDP.
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-I2V-14B-720P"
export MODEL_NAME_AUDIO="models/Diffusion_Transformer/wav2vec2-base-960h"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
# Uncomment the following two lines for multi-node training without RDMA
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --use_deepspeed --deepspeed_config_file config/zero_stage2_config.json --deepspeed_multinode_launcher standard scripts/fantasytalking/train.py \
--config_path="config/wan2.1/wan_civitai.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--pretrained_audio_model_name_or_path=$MODEL_NAME_AUDIO \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--video_sample_size=512 \
--token_sample_size=512 \
--video_sample_stride=1 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-05 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
--output_dir="output_dir_fantasytalking" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--low_vram \
--transformer_path="models/Personalized_Model/FantasyTalking/fantasytalking_model.ckpt" \
--trainable_modules "processor." "proj_model."
3.3 Common Training Parameters
Here is a detailed explanation of the key parameters in the training script:
| Parameter | Description | Default |
|---|---|---|
config_path |
Model configuration file path | config/wan2.1/wan_civitai.yaml |
pretrained_model_name_or_path |
Pretrained model path | models/Diffusion_Transformer/Wan2.1-I2V-14B-720P |
pretrained_audio_model_name_or_path |
Audio encoder path | None (automatically uses $MODEL_NAME/audio_encoder) |
train_data_dir |
Training dataset directory | datasets/X-Fun-Videos-Audios-Demo/ |
train_data_meta |
Training dataset metadata file | datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json |
video_sample_size |
Video sample size (maximum resolution) | 512 |
token_sample_size |
Token sample size | 512 |
video_sample_stride |
Video sample stride | 1 |
video_sample_n_frames |
Video sample frame count | 81 |
train_batch_size |
Training batch size | 1 |
gradient_accumulation_steps |
Gradient accumulation steps | 1 |
dataloader_num_workers |
Data loader worker threads | 8 |
num_train_epochs |
Number of training epochs | 100 |
checkpointing_steps |
Steps to save checkpoint | 50 |
learning_rate |
Learning rate | 2e-05 |
lr_scheduler |
Learning rate scheduler | constant_with_warmup |
lr_warmup_steps |
Learning rate warmup steps | 100 |
seed |
Random seed | 42 |
output_dir |
Output directory | output_dir_fantasytalking |
gradient_checkpointing |
Enable gradient checkpointing to save memory | True |
mixed_precision |
Mixed precision training: bf16 or fp16 |
bf16 |
adam_weight_decay |
Adam weight decay | 3e-2 |
adam_epsilon |
Adam epsilon | 1e-10 |
vae_mini_batch |
VAE mini-batch size | 1 |
max_grad_norm |
Maximum gradient norm | 0.05 |
transformer_path |
Pretrained Transformer weights path | models/FantasyTalking/fantasytalking_model.ckpt |
trainable_modules |
List of trainable modules | "processor." "proj_model." |
Advanced Parameters Explanation:
The following parameters in the training script might be confusing, here's a detailed explanation:
-
enable_bucket: Enable bucket training. When enabled, the model does not crop videos at the center, but instead groups videos into different buckets based on resolution for training. This allows the model to better adapt to videos of different resolutions. -
random_frame_crop: Random cropping on video frames to simulate videos with different frame counts. This helps the model better generalize to videos of varying lengths. -
random_hw_adapt: Enable automatic height and width scaling. When enabled, training video dimensions will be set to:- Maximum:
video_sample_size - Minimum:
512
Example: With
random_hw_adaptenabled,video_sample_n_frames=81,video_sample_size=768, the training input video resolutions can be512x512x81or768x768x81. - Maximum:
-
training_with_video_token_length: Train the model based on token length. When enabled, training video dimensions will be set to:- Maximum:
video_sample_size - Minimum:
256
Example: With
training_with_video_token_lengthenabled,video_sample_n_frames=81,token_sample_size=512,video_sample_size=768, the training input video resolutions can be256x256x81,512x512x81, or768x768x37.Token Length Calculation:
- For a 512x512 resolution video with 81 frames, the token length is approximately 21,952
- We need to set
token_sample_size = 512- At 512x512 resolution, the number of video frames is 81 (≈ 512 * 512 * 81 / 512 / 512)
- At 768x768 resolution, the number of video frames is 37 (≈ 512 * 512 * 81 / 768 / 768)
- At 1024x1024 resolution, the number of video frames is 16 (≈ 512 * 512 * 81 / 1024 / 1024)
- These resolutions combined with their corresponding lengths allow the model to generate videos of different sizes.
- Maximum:
-
resume_from_checkpoint: Resume training from a previous checkpoint. Use a path or"latest"to automatically select the last available checkpoint. -
low_vram: Enable low VRAM mode to reduce memory usage through memory optimization. -
uniform_sampling: Use uniform sampling strategy for timestep sampling.
3.4 Training Validation
You can configure validation parameters to periodically generate test videos during training, allowing you to monitor training progress and model quality.
Validation Parameters:
| Parameter | Description | Recommended Value |
|---|---|---|
--validation_steps |
Execute validation every N steps | 100 |
--validation_epochs |
Execute validation every N epochs | 500 |
--validation_image_paths |
List of reference image paths for validation, multiple paths separated by spaces | Multiple space-separated image paths |
--validation_audio_paths |
List of audio paths for validation, multiple paths separated by spaces | Multiple space-separated audio paths |
--validation_prompts |
List of prompts for validation, multiple prompts separated by spaces | Multiple space-separated prompts |
Example:
--validation_image_paths="asset/8.png" \
--validation_audio_paths="asset/talk.wav" \
--validation_prompts="A girl talking by the sea." \
--validation_steps=100 \
--validation_epochs=500
Notes:
- The number of
validation_image_paths,validation_audio_paths, andvalidation_promptsmust be consistent - When both
validation_stepsandvalidation_epochsare set, validation is triggered when either condition is met
3.5 Training with FSDP
If VRAM is insufficient when using multiple GPUs with DeepSpeed-Zero-2, you can switch to FSDP.
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-I2V-14B-720P"
export MODEL_NAME_AUDIO="models/Diffusion_Transformer/wav2vec2-base-960h"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
NCCL_DEBUG=INFO
accelerate launch --mixed_precision="bf16" --use_fsdp --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP --fsdp_transformer_layer_cls_to_wrap=AudioAttentionBlock --fsdp_sharding_strategy "FULL_SHARD" --fsdp_state_dict_type=SHARDED_STATE_DICT --fsdp_backward_prefetch "BACKWARD_PRE" --fsdp_cpu_ram_efficient_loading False scripts/fantasytalking/train.py \
--config_path="config/wan2.1/wan_civitai.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--pretrained_audio_model_name_or_path=$MODEL_NAME_AUDIO \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--video_sample_size=512 \
--token_sample_size=512 \
--video_sample_stride=1 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-05 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
--output_dir="output_dir_fantasytalking" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--low_vram \
--transformer_path="models/Personalized_Model/FantasyTalking/fantasytalking_model.ckpt" \
--trainable_modules "processor." "proj_model."
FSDP Key Parameters:
| Parameter | Description |
|---|---|
--use_fsdp |
Enable FSDP |
--fsdp_auto_wrap_policy |
Auto wrap policy: TRANSFORMER_BASED_WRAP |
--fsdp_transformer_layer_cls_to_wrap |
Transformer layer class name to wrap: AudioAttentionBlock |
--fsdp_sharding_strategy |
Sharding strategy: FULL_SHARD |
--fsdp_state_dict_type |
State dict type: SHARDED_STATE_DICT |
--fsdp_backward_prefetch |
Backward prefetch: BACKWARD_PRE |
--fsdp_cpu_ram_efficient_loading |
CPU RAM efficient loading: False |
3.6 Training without DeepSpeed or FSDP
This approach is not recommended as it lacks VRAM-saving backends and may easily cause out-of-memory errors. This is provided for reference only.
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-I2V-14B-720P"
export MODEL_NAME_AUDIO="models/Diffusion_Transformer/wav2vec2-base-960h"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
NCCL_DEBUG=INFO
accelerate launch --mixed_precision="bf16" scripts/fantasytalking/train.py \
--config_path="config/wan2.1/wan_civitai.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--pretrained_audio_model_name_or_path=$MODEL_NAME_AUDIO \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--video_sample_size=512 \
--token_sample_size=512 \
--video_sample_stride=1 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-05 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
--output_dir="output_dir_fantasytalking" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--low_vram \
--transformer_path="models/Personalized_Model/FantasyTalking/fantasytalking_model.ckpt" \
--trainable_modules "processor." "proj_model."
3.7 Multi-Node Distributed Training
This approach is not recommended as it lacks VRAM-saving backends and may easily cause out-of-memory errors. This is provided for reference only.
3.7.1 Environment Configuration
Assuming 2 machines with 8 GPUs each:
Machine 0 (Master):
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-I2V-14B-720P"
export MODEL_NAME_AUDIO="models/Diffusion_Transformer/wav2vec2-base-960h"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
export MASTER_ADDR="192.168.1.100" # Master machine IP
export MASTER_PORT=10086
export WORLD_SIZE=2 # Total number of machines
export NUM_PROCESS=16 # Total processes = machines × 8
export RANK=0 # Current machine rank (0 or 1)
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 for multi-node environment without RDMA
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
accelerate launch --mixed_precision="bf16" --main_process_ip=$MASTER_ADDR --main_process_port=$MASTER_PORT --num_machines=$WORLD_SIZE --num_processes=$NUM_PROCESS --machine_rank=$RANK --use_deepspeed --deepspeed_config_file config/zero_stage2_config.json --deepspeed_multinode_launcher standard scripts/fantasytalking/train.py \
--config_path="config/wan2.1/wan_civitai.yaml" \
--pretrained_model_name_or_path=$MODEL_NAME \
--pretrained_audio_model_name_or_path=$MODEL_NAME_AUDIO \
--train_data_dir=$DATASET_NAME \
--train_data_meta=$DATASET_META_NAME \
--video_sample_size=512 \
--token_sample_size=512 \
--video_sample_stride=1 \
--video_sample_n_frames=81 \
--train_batch_size=1 \
--gradient_accumulation_steps=1 \
--dataloader_num_workers=8 \
--num_train_epochs=100 \
--checkpointing_steps=50 \
--learning_rate=2e-05 \
--lr_scheduler="constant_with_warmup" \
--lr_warmup_steps=100 \
--seed=42 \
--output_dir="output_dir_fantasytalking" \
--gradient_checkpointing \
--mixed_precision="bf16" \
--adam_weight_decay=3e-2 \
--adam_epsilon=1e-10 \
--vae_mini_batch=1 \
--max_grad_norm=0.05 \
--random_hw_adapt \
--training_with_video_token_length \
--enable_bucket \
--uniform_sampling \
--low_vram \
--transformer_path="models/Personalized_Model/FantasyTalking/fantasytalking_model.ckpt" \
--trainable_modules "processor." "proj_model."
Machine 1 (Worker):
export MODEL_NAME="models/Diffusion_Transformer/Wan2.1-I2V-14B-720P"
export MODEL_NAME_AUDIO="models/Diffusion_Transformer/wav2vec2-base-960h"
export DATASET_NAME="datasets/X-Fun-Videos-Audios-Demo/"
export DATASET_META_NAME="datasets/X-Fun-Videos-Audios-Demo/metadata_add_width_height.json"
export MASTER_ADDR="192.168.1.100" # Same as Master
export MASTER_PORT=10086
export WORLD_SIZE=2
export NUM_PROCESS=16
export RANK=1 # Note this is 1
# NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1 for multi-node environment without RDMA
# export NCCL_IB_DISABLE=1
# export NCCL_P2P_DISABLE=1
NCCL_DEBUG=INFO
# Use the same accelerate launch command as Machine 0
3.7.2 Multi-Node Training Notes
-
Network Requirements:
- RDMA/InfiniBand recommended (high performance)
- Without RDMA, add environment variables:
export NCCL_IB_DISABLE=1 export NCCL_P2P_DISABLE=1
-
Data Synchronization: All machines must be able to access the same data paths (NFS/shared storage)
4. Inference Testing
After training is complete, you can use the inference script to test the generated model.
4.1 Inference Parameters
Main parameters in the inference script examples/fantasytalking/predict_s2v.py:
| Parameter | Description | Default |
|---|---|---|
GPU_memory_mode |
GPU memory mode: model_full_load, model_full_load_and_qfloat8, model_cpu_offload, model_cpu_offload_and_qfloat8, sequential_cpu_offload |
sequential_cpu_offload |
ulysses_degree |
Ulysses parallelism degree for multi-GPU inference | 1 |
ring_degree |
Ring parallelism degree for multi-GPU inference | 1 |
fsdp_dit |
Use FSDP for Transformer during multi-GPU inference to save memory | False |
compile_dit |
Compile Transformer for faster inference (fixed resolution only) | False |
config_path |
Model configuration file path | config/wan2.1/wan_civitai.yaml |
model_name |
Model path | models/Diffusion_Transformer/Wan2.1-I2V-14B-720P |
model_name_audio |
Audio encoder path | models/Diffusion_Transformer/wav2vec2-base-960h |
sampler_name |
Sampler type: Flow, Flow_Unipc, Flow_DPM++ |
Flow |
shift |
Sampler shift parameter | 5.0 |
transformer_path |
Trained Transformer weights path | models/Personalized_Model/FantasyTalking/fantasytalking_model.ckpt |
vae_path |
Trained VAE weights path | None |
lora_path |
LoRA weights path | None |
sample_size |
Generated video resolution [height, width] |
[832, 480] |
video_length |
Generated video frame count | 81 |
fps |
Frames per second | 23 |
weight_dtype |
Model weight dtype, use torch.float16 for GPUs without bf16 |
torch.bfloat16 |
validation_image_start |
Reference image path | "asset/8.png" |
audio_path |
Input audio path | "asset/talk.wav" |
prompt |
Generation prompt | "A girl talking by the sea." |
negative_prompt |
Negative prompt | See code |
guidance_scale |
Prompt guidance strength | 4.5 |
audio_guide_scale |
Audio guidance strength | 4.0 |
seed |
Random seed for reproducibility | 43 |
num_inference_steps |
Inference steps | 40 |
lora_weight |
LoRA weight strength | 0.55 |
save_path |
Generated video save path | samples/fantasy-talking-videos-speech2v |
TeaCache Acceleration Configuration:
| Parameter | Description | Default |
|---|---|---|
enable_teacache |
Enable TeaCache acceleration | True |
teacache_threshold |
TeaCache threshold (recommended 0.05~0.30) | 0.10 |
num_skip_start_steps |
Initial steps to skip TeaCache | 5 |
teacache_offload |
Offload TeaCache tensors to CPU to save memory | False |
GPU Memory Mode Description:
| Mode | Description | Memory Usage |
|---|---|---|
model_full_load |
Load entire model to GPU | Highest |
model_full_load_and_qfloat8 |
Full load + FP8 quantization | High |
model_cpu_offload |
Offload model to CPU after use | Medium |
model_cpu_offload_and_qfloat8 |
CPU offload + FP8 quantization | Medium-Low |
sequential_cpu_offload |
Offload each layer to CPU after use (slowest) | Lowest |
4.2 Single GPU Inference
Run single GPU inference:
python examples/fantasytalking/predict_s2v.py
Edit examples/fantasytalking/predict_s2v.py according to your needs. For first-time inference, focus on modifying the following parameters. For other parameters, see the inference parameters description above.
# Choose based on your GPU memory
GPU_memory_mode = "model_full_load"
# Model configuration file path
config_path = "config/wan2.1/wan_civitai.yaml"
# Your actual model path
model_name = "models/Diffusion_Transformer/Wan2.1-I2V-14B-720P"
# Audio encoder path
model_name_audio = "models/Diffusion_Transformer/wav2vec2-base-960h"
# Trained weights path, e.g., "output_dir_fantasytalking/checkpoint-xxx/diffusion_pytorch_model.safetensors"
transformer_path = "models/Personalized_Model/FantasyTalking/fantasytalking_model.ckpt"
# Reference image path
validation_image_start = "asset/8.png"
# Input audio path
audio_path = "asset/talk.wav"
# Generation prompt
prompt = "A girl talking by the sea."
# ...
4.3 Multi-GPU Parallel Inference
Use Case: High-resolution generation, accelerated inference
Install Parallel Inference Dependencies
pip install xfuser==0.4.2 yunchang==0.6.2
Configure Parallel Strategy
Edit examples/fantasytalking/predict_s2v.py:
# Ensure ulysses_degree × ring_degree = number of GPUs used
# For example, using 2 GPUs:
ulysses_degree = 2 # Head dimension parallelism
ring_degree = 1 # Sequence dimension parallelism
Configuration Principles:
ulysses_degreemust be divisible by the model's head countring_degreesplits along the sequence dimension and affects communication overhead. Try to avoid using it if heads are divisible.
Configuration Examples:
| GPU Count | ulysses_degree | ring_degree | Description |
|---|---|---|---|
| 1 | 1 | 1 | Single GPU |
| 4 | 4 | 1 | Head parallelism |
| 8 | 2 | 4 | Hybrid parallelism |
| 8 | 8 | 1 | Head parallelism |
Run Multi-GPU Inference
torchrun --nproc-per-node=2 examples/fantasytalking/predict_s2v.py
5. Additional Resources
- Official GitHub: https://github.com/aigc-apps/VideoX-Fun