In-process evaluation suite for video generations. Includes pixel metrics (SSIM, PSNR, LPIPS), Fréchet Video Distance (FVD), optical-flow comparisons, the full VBench suite, Physics-IQ, audio metrics, an absolute VLM scorer (videoscore2), and a pairwise VLM judge (judge.third_person_separation) — all behind a single registry-driven API.

Install

Use case	Install
Default (common, optical_flow, vbench, physics_iq, videoscore2)	`uv pip install -e .[eval]`
Just VBench (11 of 16 by default; +4 with detectron2)	`uv pip install -e .[eval-vbench]`
Just Physics-IQ (covered by `[eval]`)	`uv pip install -e .[eval-physics-iq]`
Audio metrics (CLAP, FAD, KL, WER, AudioBox, DeSync, ImageBind)	`uv pip install -e .[eval-audio]`
Everything: `[eval]` + `[eval-audio]` + `vbench.scene` (AVoCaDO)	`uv pip install -e .[eval-full]`
Optional faster video decode (x86_64 only; opt-in)	`uv pip install -e .[eval-fast-decode]`

[eval-audio] covers every audio.* metric. ImageBind (facebookresearch/ImageBind, CC BY-NC-SA 4.0) is git-sourced via [tool.uv.sources] rather than vendored.

[eval-fast-decode] is opt-in. It pulls decord, which is faster than the default PyAV decoder but has no aarch64 wheels and is effectively unmaintained upstream. The default [eval] install uses PyAV (via av); decord only matters if you've measured decode as the bottleneck on x86_64. audio.imagebind_score declares a hard decord dependency and will skip without it.

audio.desync and audio.wer (glm_asr) import vendored upstream from fastvideo/third_party/eval/synchformer/ (MIT) and fastvideo/third_party/eval/glmasr/ (Apache-2.0). Both trees keep their upstream LICENSE files alongside.

`audio.*` metric input contracts

Every audio metric reads from these sample-dict keys (extra keys are ignored):

Metric	Per-sample?	Required keys
`audio.clap_score`	yes	`audio` (path), `text_prompt` (str)
`audio.audiobox_aesthetics`	yes	`audio` (path)
`audio.kl_divergence`	yes	`audio` (path), `reference_audio` (path)
`audio.frechet_distance`	set-vs-set	`audio` (path), `reference_audio` (path) — accumulated across ≥2 samples; `corpus["audio.frechet_distance"]` carries the score
`audio.wer`	yes	`audio` (path), `reference_text` (str) or `text_prompt` (str)
`audio.desync`	yes	`video` (decoded tensor or path), `audio` (path)
`audio.imagebind_score`	yes	`video_path` (str) and `audio` (path) — needs the path, not the pool-decoded tensor, because ImageBind's preprocessing decodes its own clips

audio.frechet_distance is the only set-vs-set metric. The kwargs form (ev.evaluate(audio=...)) raises with a clear message because a single sample cannot produce a corpus result; use ev.evaluate(samples=[...]).

Reference repos for audio

The audio set ports its math 1:1 from hkchengrex/av-benchmark (the V2A literature's de-facto eval harness — used by MMAudio, FoleyCrafter, V2A-Mapper). Per-metric upstream:

Metric	Upstream
`audio.frechet_distance` (PaSST-FAD)	`av_bench/metrics/fad.py::compute_fd` over `hear21passt` 768-d embeds
`audio.kl_divergence`	`av_bench/metrics/kl.py::compute_kl` over PaSST 527-d logits
`audio.clap_score`	HF `transformers.ClapModel` (`laion/clap-htsat-fused` — closest HF mirror of `630k-audioset-fusion-best`)
`audio.audiobox_aesthetics`	`facebookresearch/audiobox-aesthetics` (PQ as primary score, CE/CU/PC in details)
`audio.wer`	MagiHuman-style: NFKC + CJK char-level via `jiwer`, GLM-ASR or Whisper backbone
`audio.desync`	`av_bench/synchformer/` (vendored under `third_party/eval/synchformer/`); checkpoint from `hkchengrex/MMAudio/releases/v0.1/synchformer_state_dict.pth`
`audio.imagebind_score`	`facebookresearch/ImageBind` (`imagebind_huge` pretrained)
Plus `vbench.{color, multiple_objects, object_class, spatial_relationship}` (GRiT)	`uv pip install -e .[eval-vbench]` then `uv pip install --no-build-isolation 'git+https://github.com/facebookresearch/detectron2.git'`

To use VBench, also pull the upstream submodule:

git submodule update --init --recursive  # fetches vbench + kernel deps

The submodule is a clean upstream pin. Compat with current transformers/numpy/timm versions is applied at import time in fastvideo/eval/metrics/vbench/__init__.py via attribute-level monkey-patches; the submodule files are unchanged.

Public API

from fastvideo.eval import (
    create_evaluator,        # build a reusable Evaluator
    evaluate,                # one-shot helper
    Evaluator,               # the class itself
    samples_from, as_video,  # path-style inputs → canonical samples list
    BaseMetric, MetricResult,
    register, list_metrics, get_metric,
    ensure_checkpoint, get_cache_dir,
)

# Single-sample form (per-sample metrics):
ev = create_evaluator(metrics=["common.ssim", "vbench.aesthetic_quality"],
                      device="cuda")
scores = ev.evaluate(video=tensor, reference=ref, fps=8.0)

# Many samples from two directories (paired metrics + corpus metrics):
results = ev.evaluate(samples=samples_from(video="gen/", reference="ref/"))
# results[i]["common.ssim"]              → per-pair SSIM
# results.corpus["common.fvd"]           → corpus FVD (if FVD is registered)

samples_from turns path-style inputs into the canonical samples list. Cardinality determines the shape: equal |gen| == |ref| zips into paired samples; unequal cardinality role-tags the unmatched references as trailing set-style samples so corpus metrics like FVD see the full reference set while paired metrics like LPIPS only score the first N pairs. Per-sample attachments (text_prompt(s), fps, auxiliary_info, extras=) attach by kwarg; extract_audio=True pulls audio tracks off video sources for audio metrics.

evaluate also accepts a pre-loaded (T, C, H, W) tensor or a path string under video / reference. Paths are decoded inside the worker that picks up the sample, so peak memory stays bounded by num_gpus when scoring large batches. Use evaluate(samples=..., metrics=[...]) to run a subset of the Evaluator's registered metrics on this batch (useful for scoring different corpora with different metric subsets in successive calls without burning model loads).

Training-time validation metrics

The modular trainer can score validation videos during training through callbacks.validation.metrics. Each distributed rank that writes local validation videos also runs its local metrics on cuda:<local_rank>; rank 0 only merges scalar summaries and logs artifacts.

callbacks:
  validation:
    pipeline_target: fastvideo.pipelines.basic.matrixgame2.matrixgame2_causal_dmd_pipeline.MatrixGame2CausalDMDPipeline
    dataset_file: examples/distill/MatrixGame2.0/validation.json
    sampling_steps: [4]
    metrics:
      enabled: true
      names:
        - vbench.imaging_quality
        - vbench.aesthetic_quality
        - optical_flow.synthetic_optical_flow
        - common.fvd
      skip_missing_deps: true
      strict: false
      unload_after_validation: true

      # `synthetic_optical_flow` also needs actions in the validation
      # manifest (`action_path`) plus a repo-local calibration file.
      calibration_path: assets/eval/worldmodel_synthetic_flow_calibration.json

Metric summaries are written under <output_dir>/eval/step_<step>/inference_steps_<n>_rank_<rank>.json and scalar means are logged with the metrics/validation/... prefix. Install fastvideo[eval] for optical-flow dependencies such as ptlflow; VBench metrics use the pinned submodule under fastvideo/third_party/eval/vbench. FVD uses ref_video entries from the validation manifest when present, or the standard FASTVIDEO_FVD_REF_FEATURES / eval-cache reference feature path.

CLI

fastvideo eval list                              # list registered metrics
fastvideo eval list --group vbench               # filter by group
fastvideo eval run --videos clip.mp4 \
    --metrics vbench.aesthetic_quality \
    --output scores.json
fastvideo eval run --videos generated/*.mp4 \
    --reference reference/ \
    --metrics common.ssim,common.lpips

Generate-then-score example

examples/inference/eval/eval_ltx2_vbench.py runs an LTX2 prompt through VideoGenerator and scores the resulting mp4 with vbench.aesthetic_quality and vbench.subject_consistency. Use it as a template for end-to-end "generate then score" pipelines.

Layout

fastvideo/
├── eval/
│   ├── api.py, evaluator.py, registry.py, models.py, ...
│   ├── io/                        # video loading helpers
│   ├── datasets/                  # prompt corpora (vbench, physics_iq)
│   └── metrics/
│       ├── base.py                # BaseMetric + @register contract
│       ├── common/                # SSIM, PSNR, LPIPS, FVD
│       ├── optical_flow/          # gt_optical_flow, synthetic_optical_flow
│       ├── audio/                 # clap_score, audiobox_aesthetics, kl_divergence,
│       │                          # frechet_distance, wer, desync, imagebind_score
│       ├── videoscore2/           # VideoScore-2 (Qwen2.5-VL)
│       ├── judge/                 # pairwise VLM judges (third_person_separation)
│       ├── physics_iq/            # PhysicsIQ + sub-metrics
│       └── vbench/                # adapter: sys.path bootstrap + shims
│           ├── __init__.py
│           └── <16 sub-metric pkgs>
└── third_party/
    └── eval/
        ├── vbench/                # git submodule (Vchitect/VBench)
        ├── synchformer/           # vendored (MIT), used by audio.desync
        └── glmasr/                # vendored (Apache-2.0), used by audio.wer (glm_asr)

Prompt datasets

from fastvideo.eval.datasets import get_dataset, list_datasets

list_datasets()                    # ['physics_iq', 'vbench']

ds = get_dataset("physics_iq", limit=4)   # auto-fetches assets on first miss
for row in ds:
    # row contains 'prompt', 'reference', 'reference_take2', and
    # metric-specific aux fields. Drop straight into Evaluator.evaluate(**row).
    ...

The Physics-IQ manifest CSV is vendored at fastvideo/eval/metrics/physics_iq/_vendored/descriptions.csv. Per-scenario videos, masks, and switch-frames auto-fetch on first use into ${FASTVIDEO_EVAL_CACHE}/datasets/physics_iq/. For air-gapped runs, pass auto_download=False or dataset_root= a pre-downloaded copy. Set FASTVIDEO_PHYSICS_IQ_BUCKET_URL to redirect the fetch to an internal mirror.

Adding a new metric

The full porting guide is at docs/contributing/eval-metrics.md. Summary below.

Native metric (no submodule)

# fastvideo/eval/metrics/common/<your_metric>/metric.py
from fastvideo.eval.metrics.base import BaseMetric
from fastvideo.eval.registry import register
from fastvideo.eval.types import MetricResult


@register("common.your_metric")
class YourMetric(BaseMetric):
    name = "common.your_metric"
    requires_reference = True
    needs_gpu = False
    dependencies: list[str] = []  # e.g. ["pyiqa"] if relevant

    def compute(self, sample) -> MetricResult:
        ...

The metric is auto-discovered by fastvideo/eval/metrics/__init__.py, which walks all non-underscore subdirectories and imports their metric module.

Wrapping upstream code

Three patterns coexist depending on how the upstream ships and what licence it's under. All three keep upstream files on disk unmodified; behavioural patches live as runtime shims in the consuming code.

Git submodule — large research packages pinned to a SHA, accessed via sys.path bootstrap. See fastvideo/eval/metrics/vbench/ (with fastvideo/third_party/eval/vbench/).
Vendored under third_party/eval/<name>/ — small/surgical upstream trees with permissive licences (MIT, Apache-2.0). See fastvideo/third_party/eval/synchformer/ and .../glmasr/.
Git-source via [tool.uv.sources] — license-restricted upstream that cannot be redistributed in the FastVideo source tree. See ImageBind (CC BY-NC-SA 4.0) in pyproject.toml.

Caches

Eval cache root: ${FASTVIDEO_CACHE_ROOT}/eval/, default ~/.cache/fastvideo/eval/. Override with FASTVIDEO_EVAL_CACHE.

${FASTVIDEO_CACHE_ROOT}/eval/
├── models/      # URL-fetched checkpoints (LAION head, GRiT, Synchformer, ImageBind)
├── torch/       # redirected TORCH_HOME (DINO via torch.hub, lpips)
├── clip/        # passed as download_root= to clip.load callsites
└── datasets/    # auto-fetched dataset assets, one subdir per benchmark
                 # (e.g. datasets/physics_iq/{split-videos,switch-frames,...})

HF-hosted models stay in HF's default cache (~/.cache/huggingface/hub/) so they dedupe with other ML projects on the same host.

Convention for new metrics

If your metric wraps a third-party loader that has its own cache directory, route it through get_cache_dir() so users get one knob to redirect everything.

# CLIP: pass download_root explicitly
import clip
from fastvideo.eval.models import get_cache_dir
model, _ = clip.load("ViT-B/32", device=device,
                     download_root=str(get_cache_dir() / "clip"))

# torch.hub is already redirected by fastvideo.eval.__init__ via
# TORCH_HOME; no per-callsite work needed.

# transformers / huggingface_hub: leave alone. HF's default cache is
# shared with other tools.

For libraries that do not honour any env var or kwarg (pyiqa, funasr), their cache lands in the library's own dir. Document the exception in the metric's docstring if it matters.

`common.fvd` — Fréchet Video Distance

Set-vs-set metric. Computes the Fréchet distance between Gaussian moments of I3D features (Kinetics-400) over the generated set and a reference set. Lower is better; standard protocol uses 2048 videos and a warning fires below 256.

from fastvideo.eval import create_evaluator, samples_from

ev = create_evaluator(metrics=["common.fvd"], device="cuda")
result = ev.evaluate(samples=samples_from(
    video="gen/", reference="ref/",
)).corpus["common.fvd"]

samples_from zips the directories into paired samples when their cardinalities match (FVD pulls features from both sample["video"] and sample["reference"]); when |ref| > |gen|, the unmatched references become role-tagged trailing samples so FVD still sees the full reference corpus while paired per-sample metrics (LPIPS / PSNR / SSIM) running alongside only score the matched pairs.

Reference features are cached to ${FASTVIDEO_EVAL_CACHE}/fvd/real_features_{extractor}.pt the first time reference features are streamed; subsequent runs load the cache automatically (omit reference= to score new generations against the cached set). Override with $FASTVIDEO_FVD_REF_FEATURES, the cache_path= constructor kwarg, or cache_mode={"off","read","read_write"} to control read/write behavior. The example script examples/inference/eval/eval_fvd.py demonstrates the full two-directory workflow.

`judge.third_person_separation` — pairwise VLM judge

A preference metric (a judge, not an absolute score), and the suite's first remote-API one. For each pair the judge (Gemini) sees the shared first frame and two rollouts — a candidate and a reference model under the same control signal — and picks the one that better separates the third-person CHARACTER (foreground) from the BACKGROUND. The corpus score is the candidate's win-rate, excluding ties. Set-vs-set, motion-first; it reads native mp4s, so samples carry path strings, not decoded tensors.

uv pip install -e .[eval-judge]      # opt-in: needs network + an API key
export GEMINI_API_KEY=...            # or GOOGLE_API_KEY, or ~/.gemini_token

from fastvideo.eval import create_evaluator

ev = create_evaluator(metrics=["judge.third_person_separation"], device="cpu")
result = ev.evaluate(samples=[
    {"video_path": "cand/000.mp4", "reference_path": "base/000.mp4",
     "image_path": "frames/000.png", "text_prompt": "W: moves forward", "action": "W"},
    # ... more pairs ...
]).corpus["judge.third_person_separation"]
result.score    # candidate win-rate excl. ties; result.details has the breakdown

Only video_path/reference_path are required; image_path/text_prompt/ action are optional. Verdicts are cached under ${FASTVIDEO_EVAL_CACHE}/eval/judge/. The judge separates best when the control yields genuine parallax (e.g. translation); rigid whole-frame motion (e.g. pure camera rotation) is harder. To sweep several baselines into a table, see examples/inference/eval/eval_third_person_separation.py.

Out of scope (follow-up PRs)

MIND metrics. Depend on a separate vipe upstream submodule.
VBench-2.0. Sibling vbench2 package; needs its own port.

README.md

fastvideo.eval