hao-ai-lab-FastVideo/fastvideo/eval/README.md

# `fastvideo.eval`

In-process evaluation suite for video generations. Includes pixel
metrics (SSIM, PSNR, LPIPS), Fréchet Video Distance (FVD), optical-flow
comparisons, the full VBench suite, Physics-IQ, audio metrics, an
absolute VLM scorer (`videoscore2`), and a pairwise VLM judge
(`judge.third_person_separation`), plus VQeval long-video failure scoring —
all behind a single registry-driven API.

## Install

| Use case | Install |
|---|---|
| Default (common, optical_flow, vbench, physics_iq, videoscore2) | `uv pip install -e .[eval]` |
| Just VBench (11 of 16 by default; +4 with detectron2) | `uv pip install -e .[eval-vbench]` |
| VQeval long-video composite | `uv pip install -e .[eval-vqeval]` |
| Just Physics-IQ (covered by `[eval]`) | `uv pip install -e .[eval-physics-iq]` |
| Audio metrics (CLAP, FAD, KL, WER, AudioBox, DeSync, ImageBind) | `uv pip install -e .[eval-audio]` |
| Everything: `[eval]` + `[eval-audio]` + `[eval-vqeval]` + `vbench.scene` | `uv pip install -e .[eval-full]` |
| Optional faster video decode (x86_64 only; opt-in) | `uv pip install -e .[eval-fast-decode]` |

`[eval-audio]` covers every `audio.*` metric. ImageBind
(`facebookresearch/ImageBind`, CC BY-NC-SA 4.0) is git-sourced via
`[tool.uv.sources]` rather than vendored.

`[eval-fast-decode]` is opt-in. It pulls `decord`, which is faster than
the default PyAV decoder but has no aarch64 wheels and is effectively
unmaintained upstream. The default `[eval]` install uses PyAV (via
`av`); decord only matters if you've measured decode as the bottleneck
on x86_64. `audio.imagebind_score` declares a hard `decord` dependency
and will skip without it.

`audio.desync` and `audio.wer (glm_asr)` import vendored upstream from
`fastvideo/third_party/eval/synchformer/` (MIT) and
`fastvideo/third_party/eval/glmasr/` (Apache-2.0). Both trees keep
their upstream `LICENSE` files alongside.

### `audio.*` metric input contracts

Every audio metric reads from these sample-dict keys (extra keys are
ignored):

| Metric | Per-sample? | Required keys |
|---|---|---|
| `audio.clap_score` | yes | `audio` (path), `text_prompt` (str) |
| `audio.audiobox_aesthetics` | yes | `audio` (path) |
| `audio.kl_divergence` | yes | `audio` (path), `reference_audio` (path) |
| `audio.frechet_distance` | **set-vs-set** | `audio` (path), `reference_audio` (path) — accumulated across ≥2 samples; `corpus["audio.frechet_distance"]` carries the score |
| `audio.wer` | yes | `audio` (path), `reference_text` (str) or `text_prompt` (str) |
| `audio.desync` | yes | `video` (decoded tensor or path), `audio` (path) |
| `audio.imagebind_score` | yes | `video_path` (str) **and** `audio` (path) — needs the path, not the pool-decoded tensor, because ImageBind's preprocessing decodes its own clips |

`audio.frechet_distance` is the only set-vs-set metric. The kwargs
form (`ev.evaluate(audio=...)`) raises with a clear message because a
single sample cannot produce a corpus result; use
`ev.evaluate(samples=[...])`.

### Reference repos for audio

The audio set ports its math 1:1 from `hkchengrex/av-benchmark` (the
V2A literature's de-facto eval harness — used by MMAudio, FoleyCrafter,
V2A-Mapper). Per-metric upstream:

| Metric | Upstream |
|---|---|
| `audio.frechet_distance` (PaSST-FAD) | `av_bench/metrics/fad.py::compute_fd` over `hear21passt` 768-d embeds |
| `audio.kl_divergence` | `av_bench/metrics/kl.py::compute_kl` over PaSST 527-d logits |
| `audio.clap_score` | HF `transformers.ClapModel` (`laion/clap-htsat-fused` — closest HF mirror of `630k-audioset-fusion-best`) |
| `audio.audiobox_aesthetics` | `facebookresearch/audiobox-aesthetics` (PQ as primary score, CE/CU/PC in details) |
| `audio.wer` | MagiHuman-style: NFKC + CJK char-level via `jiwer`, GLM-ASR or Whisper backbone |
| `audio.desync` | `av_bench/synchformer/` (vendored under `third_party/eval/synchformer/`); checkpoint from `hkchengrex/MMAudio/releases/v0.1/synchformer_state_dict.pth` |
| `audio.imagebind_score` | `facebookresearch/ImageBind` (`imagebind_huge` pretrained) |
| Plus `vbench.{color, multiple_objects, object_class, spatial_relationship}` (GRiT) | `uv pip install -e .[eval-vbench]` then `uv pip install --no-build-isolation 'git+https://github.com/facebookresearch/detectron2.git'` |

To use VBench, also pull the upstream submodule:

```bash
git submodule update --init --recursive  # fetches vbench + kernel deps
```

The submodule is a clean upstream pin. Compat with current
transformers/numpy/timm versions is applied at import time in
`fastvideo/eval/metrics/vbench/__init__.py` via attribute-level
monkey-patches; the submodule files are unchanged.

### VQeval long-video scoring

`vqeval.composite` wraps the six VQeval dimensions behind one metric so
CLIP, DINOv2, pyiqa, and optical-flow state are shared. The composite score is
0–100 (higher is better); individual dimension scores and raw metrics are
returned under `MetricResult.details["dimensions"]`.

Pull the pinned source and install the optional dependencies:

```bash
git submodule update --init fastvideo/third_party/eval/vqeval
uv pip install -e '.[eval-vqeval]'
```

MediaPipe is intentionally not part of `[eval-vqeval]`: VQeval's
anatomical-error detector wants the legacy `mp.solutions` API, which only
exists in `mediapipe` 0.10.x, and every 0.10.x pins `protobuf<5` —
unsatisfiable next to FastVideo's `protobuf>=5.28.3`. Upstream reports zero
anatomical errors without MediaPipe (its import degrades gracefully), and
`spatial_quality` falls back to pyiqa's BRISQUE when `cv2.quality`
(opencv-contrib) is absent. An environment that can relax the protobuf pin
can install `mediapipe<1` manually to enable the anatomical checks.

Frame rate is required because VQeval scores every frame through five seconds
and samples longer videos at approximately 2 fps. Text alignment is included
only when `--text-prompt` is provided.

```bash
fastvideo eval run \
    --videos outputs/*.mp4 \
    --metrics vqeval.composite \
    --fps 16 \
    --text-prompt "A dog running through a forest" \
    --output vqeval.json
```

The upstream source is pinned as a submodule rather than copied into FastVideo.
The LVSA root license and VQeval package metadata say Apache-2.0, while the
VQeval README says MIT; keeping an unmodified gitlink preserves the exact
upstream provenance while that documentation inconsistency remains.

## Public API

```python
from fastvideo.eval import (
    create_evaluator,        # build a reusable Evaluator
    evaluate,                # one-shot helper
    Evaluator,               # the class itself
    samples_from, as_video,  # path-style inputs → canonical samples list
    BaseMetric, MetricResult,
    register, list_metrics, get_metric,
    ensure_checkpoint, get_cache_dir,
)

# Single-sample form (per-sample metrics):
ev = create_evaluator(metrics=["common.ssim", "vbench.aesthetic_quality"],
                      device="cuda")
scores = ev.evaluate(video=tensor, reference=ref, fps=8.0)

# Many samples from two directories (paired metrics + corpus metrics):
results = ev.evaluate(samples=samples_from(video="gen/", reference="ref/"))
# results[i]["common.ssim"]              → per-pair SSIM
# results.corpus["common.fvd"]           → corpus FVD (if FVD is registered)
```

`samples_from` turns path-style inputs into the canonical samples list.
Cardinality determines the shape: equal `|gen| == |ref|` zips into paired
samples; unequal cardinality role-tags the unmatched references as
trailing set-style samples so corpus metrics like FVD see the full
reference set while paired metrics like LPIPS only score the first N
pairs. Per-sample attachments (`text_prompt(s)`, `fps`,
`auxiliary_info`, `extras=`) attach by kwarg; `extract_audio=True` pulls
audio tracks off video sources for audio metrics.

Samples whose `video` is a path (or a `Video` without frames) are decoded in
the worker that picks them up. When the caller did not attach `fps`, the pool
probes the container's frame rate and attaches it as `sample["fps"]` so
fps-aware metrics see real timing; an explicit `fps` (for example `--fps` on
`fastvideo eval run`) always wins and skips the probe. Scores recorded before
this probe existed can therefore shift unless `--fps` is passed.

`evaluate` also accepts a pre-loaded `(T, C, H, W)` tensor or a path
string under `video` / `reference`. Paths are decoded inside the worker
that picks up the sample, so peak memory stays bounded by `num_gpus`
when scoring large batches. Use `evaluate(samples=..., metrics=[...])`
to run a subset of the Evaluator's registered metrics on this batch
(useful for scoring different corpora with different metric subsets in
successive calls without burning model loads).

### Training-time validation metrics

The modular trainer can score validation videos during training through
`callbacks.validation.metrics`. Each distributed rank that writes local
validation videos also runs its local metrics on `cuda:<local_rank>`;
rank 0 only merges scalar summaries and logs artifacts.

```yaml
callbacks:
  validation:
    pipeline_target: fastvideo.pipelines.basic.matrixgame2.matrixgame2_causal_dmd_pipeline.MatrixGame2CausalDMDPipeline
    dataset_file: examples/distill/MatrixGame2.0/validation.json
    sampling_steps: [4]
    metrics:
      enabled: true
      names:
        - vbench.imaging_quality
        - vbench.aesthetic_quality
        - optical_flow.synthetic_optical_flow
        - common.fvd
      skip_missing_deps: true
      strict: false
      unload_after_validation: true

      # `synthetic_optical_flow` also needs actions in the validation
      # manifest (`action_path`) plus a repo-local calibration file.
      calibration_path: assets/eval/worldmodel_synthetic_flow_calibration.json
```

Metric summaries are written under
`<output_dir>/eval/step_<step>/inference_steps_<n>_rank_<rank>.json` and
scalar means are logged with the `metrics/validation/...` prefix.
Install `fastvideo[eval]` for optical-flow dependencies such as
`ptlflow`; VBench metrics use the pinned submodule under
`fastvideo/third_party/eval/vbench`. FVD uses `ref_video` entries from
the validation manifest when present, or the standard
`FASTVIDEO_FVD_REF_FEATURES` / eval-cache reference feature path.

### CLI

```bash
fastvideo eval list                              # list registered metrics
fastvideo eval list --group vbench               # filter by group
fastvideo eval run --videos clip.mp4 \
    --metrics vbench.aesthetic_quality \
    --output scores.json
fastvideo eval run --videos generated/*.mp4 \
    --reference reference/ \
    --metrics common.ssim,common.lpips
```

### Generate-then-score example

`examples/inference/eval/eval_ltx2_vbench.py` runs an LTX2 prompt
through `VideoGenerator` and scores the resulting mp4 with
`vbench.aesthetic_quality` and `vbench.subject_consistency`. Use it as
a template for end-to-end "generate then score" pipelines.

## Layout

```
fastvideo/
├── eval/
│   ├── api.py, evaluator.py, registry.py, models.py, ...
│   ├── io/                        # video loading helpers
│   ├── datasets/                  # prompt corpora (vbench, physics_iq)
│   └── metrics/
│       ├── base.py                # BaseMetric + @register contract
│       ├── common/                # SSIM, PSNR, LPIPS, FVD
│       ├── optical_flow/          # gt_optical_flow, synthetic_optical_flow
│       ├── audio/                 # clap_score, audiobox_aesthetics, kl_divergence,
│       │                          # frechet_distance, wer, desync, imagebind_score
│       ├── videoscore2/           # VideoScore-2 (Qwen2.5-VL)
│       ├── judge/                 # pairwise VLM judges (third_person_separation)
│       ├── physics_iq/            # PhysicsIQ + sub-metrics
│       ├── vbench/                # adapter: sys.path bootstrap + shims
│       │   ├── __init__.py
│       │   └── <16 sub-metric pkgs>
│       └── vqeval/                 # shared six-dimension composite adapter
└── third_party/
    └── eval/
        ├── vbench/                # git submodule (Vchitect/VBench)
        ├── vqeval/                 # git submodule (JiusiServe/LVSA)
        ├── synchformer/           # vendored (MIT), used by audio.desync
        └── glmasr/                # vendored (Apache-2.0), used by audio.wer (glm_asr)
```

### Prompt datasets

```python
from fastvideo.eval.datasets import get_dataset, list_datasets

list_datasets()                    # ['physics_iq', 'vbench']

ds = get_dataset("physics_iq", limit=4)   # auto-fetches assets on first miss
for row in ds:
    # row contains 'prompt', 'reference', 'reference_take2', and
    # metric-specific aux fields. Drop straight into Evaluator.evaluate(**row).
    ...
```

The Physics-IQ manifest CSV is vendored at
`fastvideo/eval/metrics/physics_iq/_vendored/descriptions.csv`.
Per-scenario videos, masks, and switch-frames auto-fetch on first use
into `${FASTVIDEO_EVAL_CACHE}/datasets/physics_iq/`. For air-gapped
runs, pass `auto_download=False` or `dataset_root=` a pre-downloaded
copy. Set `FASTVIDEO_PHYSICS_IQ_BUCKET_URL` to redirect the fetch to
an internal mirror.

## Adding a new metric

The full porting guide is at
[`docs/contributing/eval-metrics.md`](../../docs/contributing/eval-metrics.md).
Summary below.

### Native metric (no submodule)

```python
# fastvideo/eval/metrics/common/<your_metric>/metric.py
from fastvideo.eval.metrics.base import BaseMetric
from fastvideo.eval.registry import register
from fastvideo.eval.types import MetricResult


@register("common.your_metric")
class YourMetric(BaseMetric):
    name = "common.your_metric"
    requires_reference = True
    needs_gpu = False
    dependencies: list[str] = []  # e.g. ["pyiqa"] if relevant

    def compute(self, sample) -> MetricResult:
        ...
```

The metric is auto-discovered by `fastvideo/eval/metrics/__init__.py`,
which walks all non-underscore subdirectories and imports their
`metric` module.

### Wrapping upstream code

Three patterns coexist depending on how the upstream ships and what
licence it's under. All three keep upstream files on disk unmodified;
behavioural patches live as runtime shims in the consuming code.

1. **Git submodule** — large research packages pinned to a SHA, accessed
   via `sys.path` bootstrap. See `fastvideo/eval/metrics/vbench/` (with
   `fastvideo/third_party/eval/vbench/`).
2. **Vendored under `third_party/eval/<name>/`** — small/surgical upstream
   trees with permissive licences (MIT, Apache-2.0). See
   `fastvideo/third_party/eval/synchformer/` and `.../glmasr/`.
3. **Git-source via `[tool.uv.sources]`** — license-restricted upstream
   that cannot be redistributed in the FastVideo source tree. See
   ImageBind (CC BY-NC-SA 4.0) in `pyproject.toml`.

## Caches

Eval cache root: `${FASTVIDEO_CACHE_ROOT}/eval/`, default
`~/.cache/fastvideo/eval/`. Override with `FASTVIDEO_EVAL_CACHE`.

```
${FASTVIDEO_CACHE_ROOT}/eval/
├── models/      # URL-fetched checkpoints (LAION head, GRiT, Synchformer, ImageBind)
├── torch/       # redirected TORCH_HOME (DINO via torch.hub, lpips)
├── clip/        # passed as download_root= to clip.load callsites
└── datasets/    # auto-fetched dataset assets, one subdir per benchmark
                 # (e.g. datasets/physics_iq/{split-videos,switch-frames,...})
```

HF-hosted models stay in HF's default cache
(`~/.cache/huggingface/hub/`) so they dedupe with other ML projects on
the same host.

### Convention for new metrics

If your metric wraps a third-party loader that has its own cache
directory, route it through `get_cache_dir()` so users get one knob
to redirect everything.

```python
# CLIP: pass download_root explicitly
import clip
from fastvideo.eval.models import get_cache_dir
model, _ = clip.load("ViT-B/32", device=device,
                     download_root=str(get_cache_dir() / "clip"))

# torch.hub is already redirected by fastvideo.eval.__init__ via
# TORCH_HOME; no per-callsite work needed.

# transformers / huggingface_hub: leave alone. HF's default cache is
# shared with other tools.
```

For libraries that do not honour any env var or kwarg (pyiqa, funasr),
their cache lands in the library's own dir. Document the exception in
the metric's docstring if it matters.

## `common.fvd` — Fréchet Video Distance

Set-vs-set metric. Computes the Fréchet distance between Gaussian
moments of I3D features (Kinetics-400) over the generated set and a
reference set. Lower is better; standard protocol uses 2048 videos
and a warning fires below 256.

```python
from fastvideo.eval import create_evaluator, samples_from

ev = create_evaluator(metrics=["common.fvd"], device="cuda")
result = ev.evaluate(samples=samples_from(
    video="gen/", reference="ref/",
)).corpus["common.fvd"]
```

`samples_from` zips the directories into paired samples when their
cardinalities match (FVD pulls features from both `sample["video"]` and
`sample["reference"]`); when `|ref| > |gen|`, the unmatched references
become role-tagged trailing samples so FVD still sees the full
reference corpus while paired per-sample metrics (LPIPS / PSNR / SSIM)
running alongside only score the matched pairs.

Reference features are cached to
``${FASTVIDEO_EVAL_CACHE}/fvd/real_features_{extractor}.pt`` the first
time reference features are streamed; subsequent runs load the cache
automatically (omit `reference=` to score new generations against the
cached set). Override with `$FASTVIDEO_FVD_REF_FEATURES`, the
`cache_path=` constructor kwarg, or `cache_mode={"off","read","read_write"}`
to control read/write behavior. The example script
`examples/inference/eval/eval_fvd.py` demonstrates the full
two-directory workflow.

## `judge.third_person_separation` — pairwise VLM judge

A **preference** metric (a judge, not an absolute score), and the suite's first
remote-API one. For each pair the judge (Gemini) sees the shared first frame and
two rollouts — a candidate and a reference model under the same control signal —
and picks the one that better separates the third-person CHARACTER (foreground)
from the BACKGROUND. The corpus score is the candidate's win-rate, excluding
ties. Set-vs-set, motion-first; it reads native mp4s, so samples carry path
strings, not decoded tensors.

```bash
uv pip install -e .[eval-judge]      # opt-in: needs network + an API key
export GEMINI_API_KEY=...            # or GOOGLE_API_KEY, or ~/.gemini_token
```

```python
from fastvideo.eval import create_evaluator

ev = create_evaluator(metrics=["judge.third_person_separation"], device="cpu")
result = ev.evaluate(samples=[
    {"video_path": "cand/000.mp4", "reference_path": "base/000.mp4",
     "image_path": "frames/000.png", "text_prompt": "W: moves forward", "action": "W"},
    # ... more pairs ...
]).corpus["judge.third_person_separation"]
result.score    # candidate win-rate excl. ties; result.details has the breakdown
```

Only `video_path`/`reference_path` are required; `image_path`/`text_prompt`/
`action` are optional. Verdicts are cached under `${FASTVIDEO_EVAL_CACHE}/eval/judge/`.
The judge separates best when the control yields genuine parallax (e.g.
translation); rigid whole-frame motion (e.g. pure camera rotation) is harder. To
sweep several baselines into a table, see
`examples/inference/eval/eval_third_person_separation.py`.

## Exact V2A corpus evaluation

`fastvideo eval v2a` is a dedicated corpus-level entry point for
video-to-audio benchmarks. Unlike `fastvideo eval run`, which composes native
per-sample and set metrics through the registry, the V2A command can launch an
upstream benchmark as one protocol-preserving unit.

The `av-benchmark` backend calls the official `av_bench.extract` and
`av_bench.evaluate` functions. Its dependency stack is intentionally not part
of any standard FastVideo extra. Point the command at a Python interpreter from
an isolated av-benchmark environment:

```bash
fastvideo eval v2a \
  --backend av-benchmark \
  --audio-dir outputs/v2a/audio \
  --gt-cache /path/to/official/ground-truth-cache \
  --prediction-cache outputs/v2a/av_benchmark_cache \
  --output outputs/v2a/av_benchmark_results.json \
  --python-executable /path/to/av-benchmark/.venv/bin/python \
  --audio-length 8 \
  --batch-size 32 \
  --num-workers 0
```

The backend is model-independent: generated filenames only need to match the
keys in the selected official cache. It can therefore evaluate MMAudio or any
other V2A model. A complete prediction cache is reused automatically;
`--recompute` forces extraction again.

Some official dataset caches sanitize filenames. For such a cache,
`--align-prediction-keys` atomically remaps only prediction keys having a
unique match after leading underscores are ignored. The source audio is never
renamed, ambiguous keys remain unmatched, and alignment counts are serialized
in the result JSON.

The official VGGSound cache reports:

```text
FD-VGG, FD-PANN, FD-PASST
KL-PANNS-softmax, KL-PASST-softmax
ISC-PANNS-mean/std, ISC-PASST-mean/std
IB-Score, DeSync
```

This command runs in a subprocess by design. It does not import av-benchmark,
alter CUDA/PyTorch packages, or register metrics in ordinary FastVideo
generation and evaluation processes.

## Out of scope (follow-up PRs)

- **MIND** metrics. Depend on a separate `vipe` upstream submodule.
- **VBench-2.0**. Sibling vbench2 package; needs its own port.