Two small fixes in ``fastvideo/entrypoints/cli/eval.py``:
* ``_expand_paths`` deduplicated through ``[x for x in out if not (x in
seen or seen.add(x))]`` — a known idiom that ruff flags because
``set.add`` returns ``None`` (B023/func-returns-value). Replace with
an explicit loop so the side effect doesn't ride on the boolean
expression.
* ``_cmd_run`` constructed ``metrics_arg`` through an
``if/else``-block where SIM108 (project policy) prefers a ternary.
Fold to a single conditional expression.
Pure mechanical cleanups; ``ruff check`` is now clean on the file and
the eval CLI smoke test (``score_video.py`` against a self-paired
mp4 → ``common.psnr=100``) still passes.
Pure formatting pass — no behavior changes. Most edits are one of:
* yapf splitting / re-flowing kwarg-heavy callsites (argparse setup,
``MetricResult(...)`` construction).
* ruff auto-fixes around imports (collapsing ``Iterable`` to
``collections.abc``, removing the ``typing.Tuple`` shim where
``tuple[...]`` works, dropping unused imports).
* ``zip(..., strict=False)`` added to every two-iterable zip call.
* ``getattr(np, "trapz")`` → ``np.trapz`` where the fallback can read
the attribute directly (the ``trapezoid`` lookup still uses
``getattr`` because the name itself is the moving target).
Verified the test suite still passes after the pass.
``fastvideo eval run --output scores.json`` would crash with
``TypeError: Object of type float32 is not JSON serializable`` whenever
the chosen metric set landed numpy or torch values in
``MetricResult.details``. The optical-flow metrics in particular
populate ``per_frame_metrics`` with ``np.float64`` scalars, and pretty
much any metric is one ``np.percentile`` call away from the same crash.
Pass a ``default=`` callback to ``json.dumps`` that walks the unknown
leaves and coerces:
* ``np.integer`` / ``np.floating`` / ``np.bool_`` → native Python scalars
* ``np.ndarray`` → ``.tolist()``
* ``torch.Tensor`` → detached CPU ``.tolist()``
* ``pathlib.Path`` → ``str``
Anything else still raises ``TypeError`` — the goal is to handle the
known metric outputs cleanly, not to silently coerce arbitrary objects.
Adds an in-process evaluation suite covering native (SSIM/PSNR/LPIPS/
optical_flow), audio, physics_iq, vbench (16 sub-metrics), and
videoscore2. Public API: create_evaluator/evaluate, BaseMetric +
@register, ensure_checkpoint, get_cache_dir. CLI: fastvideo eval
list/run.
VBench upstream pinned as a git submodule at
fastvideo/third_party/eval/vbench (Vchitect/VBench@45e79ec); modern-dep
compat is achieved via runtime shims in vbench/__init__.py rather than
on-disk patches. CLIP/torch.hub caches are routed through
${FASTVIDEO_EVAL_CACHE}; HF cache stays at the system default.
ensure_checkpoint delegates to fastvideo.utils.get_lock + huggingface_hub
primitives. Evaluator supports release_cuda_memory(), unload(),
reload() for training-time eval that frees GPU between calls.
Verified parity against upstream vbench (5/8 metrics bit-exact, others
within 1% drift driven by transformers/torch version skew) and against
upstream VideoScore2 (regex anchored on the actual model's output
format, soft-score formula matches upstream's argmax*max_prob/total).
Includes docs/contributing/eval-metrics.md as the porting guide.
Out of scope (deferred follow-ups): MIND, VBench-2.0, FVD as a
registered metric, training-time EvalCallback.
shaoxiongduan