shaoxiongduan
f6275f8005
Adds an in-process evaluation suite covering native (SSIM/PSNR/LPIPS/ optical_flow), audio, physics_iq, vbench (16 sub-metrics), and videoscore2. Public API: create_evaluator/evaluate, BaseMetric + @register, ensure_checkpoint, get_cache_dir. CLI: fastvideo eval list/run. VBench upstream pinned as a git submodule at fastvideo/third_party/eval/vbench (Vchitect/VBench@45e79ec); modern-dep compat is achieved via runtime shims in vbench/__init__.py rather than on-disk patches. CLIP/torch.hub caches are routed through ${FASTVIDEO_EVAL_CACHE}; HF cache stays at the system default. ensure_checkpoint delegates to fastvideo.utils.get_lock + huggingface_hub primitives. Evaluator supports release_cuda_memory(), unload(), reload() for training-time eval that frees GPU between calls. Verified parity against upstream vbench (5/8 metrics bit-exact, others within 1% drift driven by transformers/torch version skew) and against upstream VideoScore2 (regex anchored on the actual model's output format, soft-score formula matches upstream's argmax*max_prob/total). Includes docs/contributing/eval-metrics.md as the porting guide. Out of scope (deferred follow-ups): MIND, VBench-2.0, FVD as a registered metric, training-time EvalCallback.
34 lines
987 B
Python
34 lines
987 B
Python
from __future__ import annotations
|
|
|
|
import torch
|
|
|
|
from fastvideo.eval.evaluator import create_evaluator
|
|
from fastvideo.eval.types import MetricResult
|
|
|
|
|
|
def evaluate(
|
|
generated: torch.Tensor,
|
|
reference: torch.Tensor | None = None,
|
|
metrics: list[str] | str = "all",
|
|
device: str = "cuda",
|
|
**kwargs,
|
|
) -> dict[str, MetricResult] | list[dict[str, MetricResult]]:
|
|
"""One-shot evaluation. For repeated use, prefer :func:`create_evaluator`.
|
|
|
|
Parameters
|
|
----------
|
|
generated : Tensor
|
|
Generated video ``(T, C, H, W)`` or batch ``(B, T, C, H, W)``.
|
|
reference : Tensor | None
|
|
Reference video (same shapes as *generated*).
|
|
metrics : list[str] | str
|
|
Metric names, or ``"all"``.
|
|
device : str
|
|
PyTorch device string.
|
|
"""
|
|
ev = create_evaluator(metrics=metrics, device=device)
|
|
kw: dict = {"video": generated, **kwargs}
|
|
if reference is not None:
|
|
kw["reference"] = reference
|
|
return ev.evaluate(**kw)
|