Pure formatting pass — no behavior changes. Most edits are one of:
* yapf splitting / re-flowing kwarg-heavy callsites (argparse setup,
``MetricResult(...)`` construction).
* ruff auto-fixes around imports (collapsing ``Iterable`` to
``collections.abc``, removing the ``typing.Tuple`` shim where
``tuple[...]`` works, dropping unused imports).
* ``zip(..., strict=False)`` added to every two-iterable zip call.
* ``getattr(np, "trapz")`` → ``np.trapz`` where the fallback can read
the attribute directly (the ``trapezoid`` lookup still uses
``getattr`` because the name itself is the moving target).
Verified the test suite still passes after the pass.
Update the layout snippets in both ``fastvideo/eval/README.md`` and
``docs/contributing/eval-metrics.md`` to show the new
``optical_flow/`` group sibling to ``common/``, with the two
sub-metrics (``gt_optical_flow``, ``synthetic_optical_flow``) listed
explicitly.
Several scaffolds that pre-date the runner refactor still pointed at
removed APIs:
* ``EvalResult`` (and its ``Evaluator.evaluate_dataset`` docstring)
exposed a class that is never returned anywhere; drop it from the
public API.
* ``Evaluator``'s module docstring still referenced the removed
``EvalRunner`` layer; trim to the current Evaluator → EvalWorker
shape.
* ``WM_EVAL_CACHE`` survived as a fallback env var from the wm-eval
port; remove and keep only ``FASTVIDEO_EVAL_CACHE``.
* The contributor docs documented ``batch_unit`` and
``trial_forward``/``Evaluator.calibrate()`` which were dropped from
``BaseMetric`` in earlier refactors; update to the current contract.
Adds an in-process evaluation suite covering native (SSIM/PSNR/LPIPS/
optical_flow), audio, physics_iq, vbench (16 sub-metrics), and
videoscore2. Public API: create_evaluator/evaluate, BaseMetric +
@register, ensure_checkpoint, get_cache_dir. CLI: fastvideo eval
list/run.
VBench upstream pinned as a git submodule at
fastvideo/third_party/eval/vbench (Vchitect/VBench@45e79ec); modern-dep
compat is achieved via runtime shims in vbench/__init__.py rather than
on-disk patches. CLIP/torch.hub caches are routed through
${FASTVIDEO_EVAL_CACHE}; HF cache stays at the system default.
ensure_checkpoint delegates to fastvideo.utils.get_lock + huggingface_hub
primitives. Evaluator supports release_cuda_memory(), unload(),
reload() for training-time eval that frees GPU between calls.
Verified parity against upstream vbench (5/8 metrics bit-exact, others
within 1% drift driven by transformers/torch version skew) and against
upstream VideoScore2 (regex anchored on the actual model's output
format, soft-score formula matches upstream's argmax*max_prob/total).
Includes docs/contributing/eval-metrics.md as the porting guide.
Out of scope (deferred follow-ups): MIND, VBench-2.0, FVD as a
registered metric, training-time EvalCallback.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: William Lin <SolitaryThinker@users.noreply.github.com>
shaoxiongduan