228 lines
8.7 KiB
Markdown
228 lines
8.7 KiB
Markdown
# Fine-tuning
|
||
|
||
This guide covers finetuning video diffusion models with FastVideo, including full finetuning and LoRA.
|
||
|
||
## Training arguments
|
||
|
||
FastVideo training scripts use several argument groups:
|
||
|
||
### Loop and model arguments
|
||
|
||
| Argument | Description |
|
||
|----------|-------------|
|
||
| `--max_train_steps` | Total training steps |
|
||
| `--train_batch_size` | Batch size per GPU |
|
||
| `--gradient_accumulation_steps` | Steps to accumulate before optimizer update |
|
||
| `--num_latent_t` | Temporal latent dimension (reduce to save memory) |
|
||
| `--num_height` / `--num_width` | Video resolution |
|
||
| `--num_frames` | Number of frames per video |
|
||
| `--output_dir` | Directory for checkpoints |
|
||
|
||
### Parallelism Arguments
|
||
|
||
| Argument | Description |
|
||
|----------|-------------|
|
||
| `--num_gpus` | Total number of GPUs |
|
||
| `--sp_size` | Sequence parallel size (increase to reduce memory per GPU) |
|
||
| `--tp_size` | Tensor parallel size |
|
||
| `--hsdp_replicate_dim` | HSDP replication dimension |
|
||
| `--hsdp_shard_dim` | HSDP sharding dimension |
|
||
|
||
### Optimizer Arguments
|
||
|
||
| Argument | Description |
|
||
|----------|-------------|
|
||
| `--learning_rate` | Base learning rate |
|
||
| `--mixed_precision` | Precision mode (`bf16` recommended) |
|
||
| `--weight_decay` | Weight decay for regularization |
|
||
| `--max_grad_norm` | Gradient clipping threshold |
|
||
|
||
### Validation Arguments
|
||
|
||
| Argument | Description |
|
||
|----------|-------------|
|
||
| `--log_validation` | Enable validation logging |
|
||
| `--validation_dataset_file` | JSON file with validation prompts |
|
||
| `--validation_steps` | Run validation every N steps |
|
||
| `--validation_sampling_steps` | Inference steps for validation |
|
||
| `--validation_guidance_scale` | CFG scale for validation |
|
||
|
||
## Runner Entry Point
|
||
|
||
All training launchers go through the generic runner, which resolves the pipeline class from two
|
||
required flags:
|
||
|
||
| Argument | Description |
|
||
|----------|-------------|
|
||
| `--pipeline-class` | Pipeline class to run, e.g. `WanTrainingPipeline` |
|
||
| `--pipeline-module` | Module that defines the class, e.g. `fastvideo.training.wan_training_pipeline` |
|
||
|
||
```bash
|
||
torchrun --nnodes 1 --nproc_per_node 4 \
|
||
fastvideo/training/runner.py \
|
||
--pipeline-class WanTrainingPipeline \
|
||
--pipeline-module fastvideo.training.wan_training_pipeline \
|
||
--pretrained_model_name_or_path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
|
||
--data_path data/crush-smol_processed_t2v/combined_parquet_dataset
|
||
```
|
||
|
||
Running a pipeline module directly (e.g. `fastvideo/training/wan_training_pipeline.py`) still works
|
||
but prints a deprecation banner.
|
||
|
||
## Full Finetuning
|
||
|
||
Full finetuning updates all model weights. This provides the best quality but requires more GPU memory.
|
||
|
||
```bash
|
||
# Example: Wan2.1 T2V 1.3B full finetune (4 GPUs)
|
||
bash examples/training/finetune/wan_t2v_1.3B/crush_smol/finetune_t2v.sh
|
||
```
|
||
|
||
**Typical settings:**
|
||
|
||
- Learning rate: `1e-5` to `5e-5`
|
||
- Gradient checkpointing: `--enable_gradient_checkpointing_type "full"`
|
||
- Memory scaling: Increase `--sp_size` or reduce `--num_latent_t` to fit in memory
|
||
|
||
## Attention Quantization-Aware Training
|
||
|
||
Attn-QAT fine-tunes a model while simulating low-bit attention in the forward
|
||
and backward passes. The modular trainer can select the backend per model role,
|
||
so a later DMD2 stage can keep fake quantization on the student while the
|
||
teacher and critic use Flash Attention.
|
||
|
||
The ready-to-run Wan2.1 MixKit workflow includes supervised fine-tuning,
|
||
checkpoint export, and three-step DMD2 distillation:
|
||
|
||
**→ [Follow the Attn-QAT training guide](attn_qat.md)**
|
||
|
||
## LoRA Finetuning
|
||
|
||
LoRA (Low-Rank Adaptation) trains lightweight adapters while keeping the base model frozen. This significantly reduces memory usage and training time.
|
||
|
||
### LoRA-Specific Arguments
|
||
|
||
| Argument | Description |
|
||
|----------|-------------|
|
||
| `--lora_training True` | Enable LoRA mode |
|
||
| `--lora_rank` | Rank of LoRA adapters (16, 32, 64, 128) |
|
||
|
||
### Learning Rate for LoRA
|
||
|
||
**Important:** LoRA typically requires a **10–20× higher learning rate** than full finetuning because only the low-rank adapters are being trained while the base model is frozen.
|
||
|
||
| Training Mode | Recommended Learning Rate |
|
||
|---------------|---------------------------|
|
||
| Full finetune | `1e-5` to `5e-5` |
|
||
| LoRA | `1e-4` to `2e-4` |
|
||
|
||
### Example LoRA Training
|
||
|
||
```bash
|
||
# Example: Wan2.1 T2V 1.3B LoRA finetune (1 GPU)
|
||
bash examples/training/finetune/wan_t2v_1.3B/crush_smol/finetune_t2v_lora.sh
|
||
```
|
||
|
||
Key differences from full finetune:
|
||
|
||
- Add `--lora_training True --lora_rank 32`
|
||
- Use higher learning rate (10–20× full finetune)
|
||
- Can run on fewer GPUs (even single GPU)
|
||
- Outputs adapter weights instead of full model
|
||
|
||
## LoRA Extraction and Merging
|
||
|
||
FastVideo provides generic runtime adapter extraction and retains a legacy merger for its previously supported adapter layouts.
|
||
|
||
### Extract LoRA Adapter
|
||
|
||
Extract a LoRA adapter by comparing a finetuned model to its base:
|
||
|
||
```bash
|
||
python scripts/lora_extraction/extract_lora.py \
|
||
--base Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
|
||
--ft path/to/your/finetuned_model \
|
||
--out adapter_r32.safetensors \
|
||
--rank 32 \
|
||
--exact-tensor-pattern '^condition_embedder\.' \
|
||
--exact-tensor-pattern '^proj_out\.weight$'
|
||
```
|
||
|
||
| Argument | Description |
|
||
|----------|-------------|
|
||
| `--base` | Base model (Hugging Face ID or local path) |
|
||
| `--ft` | Fine-tuned model (Hugging Face ID or local path) |
|
||
| `--out` | Output adapter file (`.safetensors`) |
|
||
| `--rank` | Requested LoRA rank (for example, 16, 32, 64, or 128) |
|
||
| `--full-rank` | Extract full-rank adapter (optional) |
|
||
| `--min-delta` | Omit tensors whose maximum absolute FP32 delta is at or below the threshold (default: `1e-8`) |
|
||
| `--load-mode` | `auto` (indexed, then pipeline fallback), `indexed`, or `pipeline` |
|
||
| `--device` | SVD device, such as `cpu` or `cuda:0` |
|
||
| `--svd-method` | Exact or randomized SVD |
|
||
| `--factor-dtype` | Storage dtype for the low-rank factors |
|
||
| `--dense-dtype` | Storage dtype for exact `.diff`/`.diff_b`/`.diff_param` payloads (default: `float32`) |
|
||
| `--exact-tensor-pattern` | Repeatable regex for matrices the target runtime cannot load as LoRA factors |
|
||
|
||
For large checkpoints, indexed loading streams one transformer tensor pair at a time and downloads only `transformer/*`. The extractor also preserves changed norms, biases, and standalone parameters as exact deltas, and fine-tuned-only parameters as `.set_weight` or `.set_param`. Matrix selection is runtime-agnostic, so full-finetune extraction must keep runtime-unsupported matrices exact, as the Wan example does. See the [LoRA utilities](../utilities/lora.md) for the GPU/randomized-SVD command, resume options, and accuracy controls.
|
||
|
||
Mixed low-rank/dense adapters from the generic extractor must be supplied when constructing FastVideo through
|
||
`ComponentConfig(lora_path=...)`; their dense payload cannot be swapped later with `set_lora_adapter`. The legacy
|
||
offline merger below is not part of this extraction workflow.
|
||
|
||
### Legacy Merge LoRA Adapter
|
||
|
||
The command below documents the pre-existing merger for adapters it already supports. Do not pass a mixed adapter from
|
||
the generic extractor to it: the legacy merger does not apply exact dense or replacement payloads.
|
||
|
||
```bash
|
||
python scripts/lora_extraction/merge_lora.py \
|
||
--base Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
|
||
--adapter legacy_factor_only_adapter.safetensors \
|
||
--ft path/to/your/finetuned_model \
|
||
--output merged_model
|
||
```
|
||
|
||
| Argument | Description |
|
||
|----------|-------------|
|
||
| `--base` | Base model path |
|
||
| `--adapter` | LoRA adapter file |
|
||
| `--ft` | Finetuned model (for config reference) |
|
||
| `--output` | Output directory for merged model |
|
||
|
||
### Validate Merged Model
|
||
|
||
Compare the merged model against the original finetuned model:
|
||
|
||
```bash
|
||
python scripts/lora_extraction/lora_inference_comparison.py \
|
||
--base merged_model \
|
||
--ft path/to/your/finetuned_model \
|
||
--adapter NONE \
|
||
--output-dir results \
|
||
--prompt "A cat sitting on a windowsill" \
|
||
--compute-ssim \
|
||
--compute-lpips
|
||
```
|
||
|
||
## Training Examples
|
||
|
||
Ready-to-run training scripts are available for multiple models:
|
||
|
||
**→ [Browse all training examples](examples/examples_training_index.md)**
|
||
|
||
| Model | Type | Example |
|
||
|-------|------|---------|
|
||
| Wan2.1 T2V 1.3B | T2V | `examples/training/finetune/wan_t2v_1.3B/crush_smol/` |
|
||
| Wan2.1 I2V 14B | I2V | `examples/training/finetune/wan_i2v_14B_480p/crush_smol/` |
|
||
| Wan2.1-Fun 1.3B InP | I2V | `examples/training/finetune/Wan2.1-Fun-1.3B-InP/crush_smol/` |
|
||
| Wan2.1 VSA | T2V/I2V | `examples/training/finetune/Wan2.1-VSA/Wan-Syn-Data/` |
|
||
| Wan2.1 T2V 1.3B Attn-QAT | QAT SFT + DMD2 | `examples/train/scenario/qad_wan2_1_mixkit/` |
|
||
|
||
Each example includes:
|
||
|
||
- a README pointing at the matching download script under `examples/datasets/`
|
||
- `preprocess_*.sh` — run preprocessing
|
||
- `finetune_*.sh` — full finetune launcher
|
||
- `finetune_*_lora.sh` — LoRA finetune launcher
|
||
- `validation.json` — validation prompts
|