80 lines
2.1 KiB
Markdown
80 lines
2.1 KiB
Markdown
# Optimization Examples
|
|
|
|
## Wan 2.1 QAT Attention 14B Inference
|
|
|
|
Use these files for Wan 2.1 14B inference with the `ATTN_QAT_INFER` backend:
|
|
|
|
- `examples/inference/optimizations/download_14B_qat.sh`
|
|
- `examples/inference/optimizations/attn_qat_inference_example.py`
|
|
|
|
### 1. Download the 14B QAT checkpoint
|
|
|
|
The helper script downloads the QAT safetensors from
|
|
`FastVideo/14B_qat_400` into `checkpoints/14B_qat_400` by default.
|
|
|
|
Prerequisites:
|
|
|
|
- `huggingface_hub` installed, for example: `uv pip install huggingface_hub`
|
|
- access to the model repo if it is private or gated: `huggingface-cli login`
|
|
|
|
Run:
|
|
|
|
```bash
|
|
bash examples/inference/optimizations/download_14B_qat.sh
|
|
```
|
|
|
|
To download into a custom directory, pass it as the first argument:
|
|
|
|
```bash
|
|
bash examples/inference/optimizations/download_14B_qat.sh /path/to/14B_qat_400
|
|
```
|
|
|
|
### 2. Edit the inference example for Wan 2.1 14B
|
|
|
|
Open `examples/inference/optimizations/attn_qat_inference_example.py` and
|
|
update these two values:
|
|
|
|
1. Change the base model from `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` to
|
|
`Wan-AI/Wan2.1-T2V-14B-Diffusers`.
|
|
2. Replace the placeholder
|
|
`init_weights_from_safetensors="safetensors_path"` with the directory that
|
|
contains the downloaded `.safetensors` files.
|
|
|
|
Example:
|
|
|
|
```python
|
|
generator = VideoGenerator.from_pretrained(
|
|
"Wan-AI/Wan2.1-T2V-14B-Diffusers",
|
|
num_gpus=1,
|
|
use_fsdp_inference=True,
|
|
dit_cpu_offload=False,
|
|
vae_cpu_offload=False,
|
|
text_encoder_cpu_offload=True,
|
|
pin_cpu_memory=False,
|
|
init_weights_from_safetensors="checkpoints/14B_qat_400",
|
|
)
|
|
```
|
|
|
|
The script already sets:
|
|
|
|
```python
|
|
os.environ["FASTVIDEO_ATTENTION_BACKEND"] = "ATTN_QAT_INFER"
|
|
```
|
|
|
|
### 3. Run the example
|
|
|
|
```bash
|
|
python examples/inference/optimizations/attn_qat_inference_example.py
|
|
```
|
|
|
|
The generated videos are written to `video_samples/` by default.
|
|
|
|
### Notes
|
|
|
|
- `ATTN_QAT_INFER` requires the in-repo `fastvideo-kernel` build to expose the
|
|
`attn_qat_infer` package.
|
|
- If you have not built the kernel yet, run `cd fastvideo-kernel && ./build.sh`
|
|
first.
|
|
- If you keep the example on the `1.3B` base model while loading the 14B QAT
|
|
weights, the model/config will not match.
|