# 🧱 Data Preprocessing

To save GPU memory during training, FastVideo precomputes text embeddings and VAE latents. This eliminates the need to load the text encoder and VAE during training.

## Quick Start

Download the sample dataset and run preprocessing:

```bash
# Download the crush-smol dataset
python scripts/huggingface/download_hf.py \
    --repo_id "wlsaidhi/crush-smol-merged" \
    --local_dir "data/crush-smol" \
    --repo_type "dataset"

# Run preprocessing
bash examples/training/finetune/wan_t2v_1.3B/crush_smol/preprocess_wan_data_t2v_new.sh
```

## Preprocessing Pipeline

The preprocessing pipeline supports multiple dataset formats and video loaders. It reads a `PreprocessRunConfig`
YAML file (`fastvideo/api/training_schema.py`): the model and workload type at the top level, and the preprocessing
settings in the `preprocess` section (the fields of `PreprocessConfig` in `fastvideo/configs/configs.py`):

```yaml
# preprocess_t2v.yaml
model_path: Wan-AI/Wan2.1-T2V-1.3B-Diffusers
pipeline:
  workload_type: t2v
preprocess:
  video_loader_type: torchvision
  dataset_type: merged
  preprocess_video_batch_size: 2
  dataloader_num_workers: 0
  max_height: 480
  max_width: 832
  num_frames: 77
  train_fps: 16
  samples_per_file: 8
  flush_frequency: 8
  video_length_tolerance_range: 5
```

Pass the file with `--config`. Each dotted override after it sets one field, for example the values that come from
shell variables:

```bash
GPU_NUM=2
DATASET_PATH="data/crush-smol/"
OUTPUT_DIR="data/crush-smol_processed_t2v/"

torchrun --nproc_per_node=$GPU_NUM \
    -m fastvideo.pipelines.preprocess.v1_preprocessing_new \
    --config preprocess_t2v.yaml \
    --preprocess.dataset_path "$DATASET_PATH" \
    --preprocess.dataset_output_dir "$OUTPUT_DIR"
```

### Key Parameters

| Parameter                             | Description                                                 |
| ------------------------------------- | ----------------------------------------------------------- |
| `pipeline.workload_type`              | Task type: `t2v` (text-to-video) or `i2v` (image-to-video)  |
| `preprocess.dataset_type`             | Input format: `hf` (HuggingFace) or `merged` (local folder) |
| `preprocess.dataset_path`             | Path to dataset (HF repo ID or local folder)                |
| `preprocess.dataset_output_dir`       | Output directory for Parquet files                          |
| `preprocess.video_loader_type`        | Video decoder: `torchcodec` or `torchvision`                |
| `preprocess.max_height` / `max_width` | Target resolution for videos                                |
| `preprocess.num_frames`               | Number of frames to extract per video                       |
| `preprocess.train_fps`                | Target FPS for frame extraction                             |

## Dataset Formats

### Merged Dataset (Local Folder)

Structure your dataset as follows:

```
your_dataset/
├── videos/
│   ├── video_001.mp4
│   ├── video_002.mp4
│   └── ...
└── videos2caption.json
```

The `videos2caption.json` maps video filenames to captions:

```json
[
  {"path": "video_001.mp4", "cap": "A cat playing with yarn..."},
  {"path": "video_002.mp4", "cap": "Ocean waves at sunset..."}
]
```

### HuggingFace Dataset

Use `--preprocess.dataset_type hf` and point `--preprocess.dataset_path` to a HuggingFace dataset with `video` and `caption` columns.

## Creating Your Own Dataset

If you have raw videos and captions in separate files, generate the `videos2caption.json`:

```bash
python scripts/dataset_preparation/prepare_json_file.py \
    --data_folder path/to/your_raw_data/ \
    --output path/to/output_folder
```

Your raw data folder should contain:

```
your_raw_data/
├── videos/
│   ├── 0.mp4
│   ├── 1.mp4
│   └── ...
├── videos.txt    # list of video filenames
└── prompt.txt    # corresponding captions (one per line)
```

## Output Format

Preprocessing outputs Parquet files in the `combined_parquet_dataset/` subdirectory containing:

- `vae_latent_bytes` — VAE-encoded video latent
- `text_embedding_bytes` — text encoder output
- `clip_feature_bytes` — CLIP image features (I2V only)
- `first_frame_latent_bytes` — first frame latent (I2V only)
- Metadata: shapes, dtypes, and sample identifiers

## Examples

See ready-to-run preprocessing scripts in the training examples:

- **T2V**: `examples/training/finetune/wan_t2v_1.3B/crush_smol/preprocess_wan_data_t2v_new.sh`
- **I2V**: `examples/training/finetune/wan_i2v_14B_480p/crush_smol/preprocess_wan_data_i2v_new.sh`

**→ [Browse all training examples](examples/examples_training_index.md)**