shaoxiongduan
de3cd6aab2
``Evaluator.evaluate`` and the one-shot ``evaluate()`` helper now accept ``video`` / ``reference`` as either a pre-loaded ``(T, C, H, W)`` tensor or a path-like (``str`` / ``Path``). Paths are decoded inside the worker thread that picks the sample up — see ``EvalWorker._resolve_video_input``. Why this matters for batch eval: the dispatcher in ``Evaluator.evaluate(samples=[...])`` submits every sample to the ``ThreadPoolExecutor`` upfront. With pre-loaded tensors that meant every video in the batch was resident in CPU RAM at once (≈ 3 GB per video at 1088×1920×121); a full benchmark of hundreds of clips would OOM before scoring started. With paths, the queued futures hold cheap strings; only ``num_gpus`` videos are decoded concurrently. Peak resident memory becomes ``O(num_gpus)`` instead of ``O(len(samples))``. No dispatcher rewrite needed — the change is ~10 lines at the worker boundary. * ``EvalWorker.evaluate`` now normalizes ``sample["video"]`` and ``sample["reference"]`` through ``_resolve_video_input`` (path → ``load_video`` decode; ``(1, T, C, H, W)`` → squeeze; tensor → passthrough). Metrics keep the existing contract: by the time ``compute()`` runs, ``sample["video"]`` is always a 4-D tensor. * ``score_folder.py`` and ``bench_vbench.py`` simplified to pass paths directly. ``bench_physics_iq.py`` already did. * ``fastvideo.eval.api.evaluate`` signature widened to ``Tensor | str | Path``; the worker handles the decode either way. Tests: new ``test_evaluator_paths.py`` covers the kwargs form, the samples-list form, mixing paths and tensors in the same batch, parity between path-form and tensor-form scores, and the missing-path exception path. End-to-end ``score_folder.py`` smoke run confirmed.
37 lines
1.1 KiB
Python
37 lines
1.1 KiB
Python
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
|
|
import torch
|
|
|
|
from fastvideo.eval.evaluator import create_evaluator
|
|
from fastvideo.eval.types import MetricResult
|
|
|
|
|
|
def evaluate(
|
|
generated: torch.Tensor | str | Path,
|
|
reference: torch.Tensor | str | Path | None = None,
|
|
metrics: list[str] | str = "all",
|
|
device: str = "cuda",
|
|
**kwargs,
|
|
) -> dict[str, MetricResult] | list[dict[str, MetricResult]]:
|
|
"""One-shot evaluation. For repeated use, prefer :func:`create_evaluator`.
|
|
|
|
Parameters
|
|
----------
|
|
generated : Tensor | str | Path
|
|
Generated video. Either a pre-loaded ``(T, C, H, W)`` tensor or a
|
|
path to an mp4/avi/etc. — paths are decoded by the worker.
|
|
reference : Tensor | str | Path | None
|
|
Reference video (same accepted shapes as *generated*).
|
|
metrics : list[str] | str
|
|
Metric names, or ``"all"``.
|
|
device : str
|
|
PyTorch device string.
|
|
"""
|
|
ev = create_evaluator(metrics=metrics, device=device)
|
|
kw: dict = {"video": generated, **kwargs}
|
|
if reference is not None:
|
|
kw["reference"] = reference
|
|
return ev.evaluate(**kw)
|