shaoxiongduan
f6275f8005
Adds an in-process evaluation suite covering native (SSIM/PSNR/LPIPS/ optical_flow), audio, physics_iq, vbench (16 sub-metrics), and videoscore2. Public API: create_evaluator/evaluate, BaseMetric + @register, ensure_checkpoint, get_cache_dir. CLI: fastvideo eval list/run. VBench upstream pinned as a git submodule at fastvideo/third_party/eval/vbench (Vchitect/VBench@45e79ec); modern-dep compat is achieved via runtime shims in vbench/__init__.py rather than on-disk patches. CLIP/torch.hub caches are routed through ${FASTVIDEO_EVAL_CACHE}; HF cache stays at the system default. ensure_checkpoint delegates to fastvideo.utils.get_lock + huggingface_hub primitives. Evaluator supports release_cuda_memory(), unload(), reload() for training-time eval that frees GPU between calls. Verified parity against upstream vbench (5/8 metrics bit-exact, others within 1% drift driven by transformers/torch version skew) and against upstream VideoScore2 (regex anchored on the actual model's output format, soft-score formula matches upstream's argmax*max_prob/total). Includes docs/contributing/eval-metrics.md as the porting guide. Out of scope (deferred follow-ups): MIND, VBench-2.0, FVD as a registered metric, training-time EvalCallback.
17 lines
431 B
Python
17 lines
431 B
Python
from __future__ import annotations
|
|
|
|
from dataclasses import dataclass, field
|
|
from typing import Any
|
|
|
|
|
|
@dataclass
|
|
class MetricResult:
|
|
"""Standard result container returned by all metrics.
|
|
|
|
``score`` is ``None`` when the metric was skipped (e.g. missing
|
|
required input). Check ``details["skipped"]`` for the reason.
|
|
"""
|
|
name: str
|
|
score: float | None
|
|
details: dict[str, Any] = field(default_factory=dict)
|