CI runs `pre-commit run --all-files` which surfaces every latent issue
across the eval suite, not just the ones in changed files. Before this
commit, that surfaced 9 ruff errors and 47 mypy errors that had built
up over earlier PRs (last touched in `[style] eval: ruff/yapf pass` and
related). Cleaning them in one sweep so subsequent eval PRs land green.
Ruff
- yapf reformat: 3 files (entrypoints/cli/eval.py, optical_flow/_shared.py,
physics_iq/utils.py).
- B024: drop ABC inheritance from PromptDataset (it has no abstract
methods; subclasses just populate self._rows).
- B027: BaseMetric.setup is intentionally an optional no-op override
(metrics with no eager state inherit it). Document and noqa instead
of forcing every native metric to declare an empty override.
- SIM105 / SIM115: contextlib.suppress for ipc_collect, mkstemp for
the temp video file in vbench.scene.
- UP038: isinstance(x, (A, B)) -> isinstance(x, A | B).
Mypy
- Optional-narrowing pattern across 12 metric files: each metric class
initialised self._model (and sometimes _processor / _tokenizer /
_head) to None and reassigned in setup(), but mypy narrows the type
to None and flags every later attribute access. Annotate the slots
as Any. Same fix already applied to physics_iq + human_action +
videoscore2 in earlier commits; this extends it to the rest of the
suite.
- Untyped helpers: annotate _safe_amt_forward (motion_smoothness),
_patch_detectron2_registries (_grit_helper), _Loader (vbench/__init__).
- _grit_helper: function-attribute writes (`func._patched_idempotent`)
type: ignore'd at the assignment site.
- imaging_quality: rename `all_scores` rebinding to `chunks` / `per_frame`
so mypy doesn't carry the list[Any] type into the cat'd tensor.
- worker._resolve_video_input: add Any -> Any annotation.
No runtime behaviour changes. 33/33 eval pytest still pass.
Two small fixes in ``fastvideo/entrypoints/cli/eval.py``:
* ``_expand_paths`` deduplicated through ``[x for x in out if not (x in
seen or seen.add(x))]`` — a known idiom that ruff flags because
``set.add`` returns ``None`` (B023/func-returns-value). Replace with
an explicit loop so the side effect doesn't ride on the boolean
expression.
* ``_cmd_run`` constructed ``metrics_arg`` through an
``if/else``-block where SIM108 (project policy) prefers a ternary.
Fold to a single conditional expression.
Pure mechanical cleanups; ``ruff check`` is now clean on the file and
the eval CLI smoke test (``score_video.py`` against a self-paired
mp4 → ``common.psnr=100``) still passes.
Pure formatting pass — no behavior changes. Most edits are one of:
* yapf splitting / re-flowing kwarg-heavy callsites (argparse setup,
``MetricResult(...)`` construction).
* ruff auto-fixes around imports (collapsing ``Iterable`` to
``collections.abc``, removing the ``typing.Tuple`` shim where
``tuple[...]`` works, dropping unused imports).
* ``zip(..., strict=False)`` added to every two-iterable zip call.
* ``getattr(np, "trapz")`` → ``np.trapz`` where the fallback can read
the attribute directly (the ``trapezoid`` lookup still uses
``getattr`` because the name itself is the moving target).
Verified the test suite still passes after the pass.
``fastvideo eval run --output scores.json`` would crash with
``TypeError: Object of type float32 is not JSON serializable`` whenever
the chosen metric set landed numpy or torch values in
``MetricResult.details``. The optical-flow metrics in particular
populate ``per_frame_metrics`` with ``np.float64`` scalars, and pretty
much any metric is one ``np.percentile`` call away from the same crash.
Pass a ``default=`` callback to ``json.dumps`` that walks the unknown
leaves and coerces:
* ``np.integer`` / ``np.floating`` / ``np.bool_`` → native Python scalars
* ``np.ndarray`` → ``.tolist()``
* ``torch.Tensor`` → detached CPU ``.tolist()``
* ``pathlib.Path`` → ``str``
Anything else still raises ``TypeError`` — the goal is to handle the
known metric outputs cleanly, not to silently coerce arbitrary objects.
Adds an in-process evaluation suite covering native (SSIM/PSNR/LPIPS/
optical_flow), audio, physics_iq, vbench (16 sub-metrics), and
videoscore2. Public API: create_evaluator/evaluate, BaseMetric +
@register, ensure_checkpoint, get_cache_dir. CLI: fastvideo eval
list/run.
VBench upstream pinned as a git submodule at
fastvideo/third_party/eval/vbench (Vchitect/VBench@45e79ec); modern-dep
compat is achieved via runtime shims in vbench/__init__.py rather than
on-disk patches. CLIP/torch.hub caches are routed through
${FASTVIDEO_EVAL_CACHE}; HF cache stays at the system default.
ensure_checkpoint delegates to fastvideo.utils.get_lock + huggingface_hub
primitives. Evaluator supports release_cuda_memory(), unload(),
reload() for training-time eval that frees GPU between calls.
Verified parity against upstream vbench (5/8 metrics bit-exact, others
within 1% drift driven by transformers/torch version skew) and against
upstream VideoScore2 (regex anchored on the actual model's output
format, soft-score formula matches upstream's argmax*max_prob/total).
Includes docs/contributing/eval-metrics.md as the porting guide.
Out of scope (deferred follow-ups): MIND, VBench-2.0, FVD as a
registered metric, training-time EvalCallback.
shaoxiongduan