[cleanup] eval: PR-readiness audit — empty inits, fix stale docs, drop unused deps
Three loose ends from earlier hygiene passes that the final PR review
caught:
* Empty out 27 sub-metric ``__init__.py`` files that still re-exported
the metric class. The earlier ``54f26bea`` commit only cleaned 2 of
them; the rest still carried ``from .metric import FooMetric # noqa:
F401`` lines copied from the original wm-eval port. Auto-discovery
doesn't need them, and ``get_metric("group.name")`` is the canonical
access pattern. Now uniformly empty across all sub-metric inits.
* Fix stale layout references in ``fastvideo/eval/README.md`` and
``.agents/workflows/evaluation-development.md`` that still described
the abandoned ``<bench>/external/upstream/`` per-metric layout. The
actual layout is the flat ``fastvideo/third_party/eval/<bench>/`` we
switched to during the port. Also drops the stale ``_third_party``
example from ``fastvideo/eval/metrics/__init__.py``'s docstring (the
underscore-skip rule is unchanged; the example was just wrong).
* Drop ``open_clip_torch`` and ``torchmetrics`` from the
``[eval]`` extra. Neither is imported anywhere under
``fastvideo/eval/`` (or in the vbench submodule we vendor); they were
carried over from the wm-eval port's heavier dep graph.
Verified: 32 metrics still register, 33/4 tests pass/skip on 1 GPU.
This commit is contained in:
shaoxiongduan
@@ -64,8 +64,9 @@ via `@register("<group>.<name>")` and are auto-discovered when
|
||||
(`fastvideo/eval/metrics/common/`, `audio/`, `vlm/`).
|
||||
- **Metrics that wrap upstream research code**: follow the vbench
|
||||
pattern in `fastvideo/eval/metrics/vbench/`:
|
||||
- upstream is a git submodule under `<metric>/external/upstream/`,
|
||||
pinned to a SHA in repo-root `.gitmodules`;
|
||||
- upstream is a git submodule under
|
||||
`fastvideo/third_party/eval/<bench>/`, pinned to a SHA in repo-root
|
||||
`.gitmodules`;
|
||||
- the metric package's `__init__.py` does the `sys.path` insert AND
|
||||
installs runtime compat shims (attribute-level monkey-patches) for
|
||||
any modern-dep drift. **Do not** modify upstream files on disk and
|
||||
|
||||
@@ -133,11 +133,14 @@ module).
|
||||
|
||||
Pattern: see `fastvideo/eval/metrics/vbench/`. The contract is:
|
||||
|
||||
1. `<bench>/external/upstream/` is a git submodule pinned to a SHA
|
||||
(registered in repo-root `.gitmodules`).
|
||||
2. `<bench>/__init__.py` inserts that path on `sys.path` and installs
|
||||
any compat shims (attribute-level monkey-patches) needed for modern
|
||||
torch/transformers/numpy. **Do not modify upstream files on disk.**
|
||||
1. The upstream code lives as a git submodule under
|
||||
`fastvideo/third_party/eval/<bench>/`, pinned to a SHA (registered
|
||||
in repo-root `.gitmodules`).
|
||||
2. The metric package's `__init__.py`
|
||||
(`fastvideo/eval/metrics/<bench>/__init__.py`) inserts that submodule
|
||||
path on `sys.path` and installs any compat shims (attribute-level
|
||||
monkey-patches) needed for modern torch/transformers/numpy.
|
||||
**Do not modify upstream files on disk.**
|
||||
3. Per-sub-metric `metric.py` files use `@register("<bench>.<name>")`.
|
||||
|
||||
There is no per-benchmark setup script. Patches live as Python in
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
"""Auto-discover and register all built-in metrics (recursive).
|
||||
|
||||
Manually walks the metrics package tree and imports any leaf-package's
|
||||
`metric` module so @register decorators fire. Skips any path component
|
||||
starting with `_` (e.g. _vendor_*, _third_party) to avoid pulling in
|
||||
vendored model code that may have side-effecting imports.
|
||||
Walks the metrics package tree and imports any leaf-package's
|
||||
``metric`` module so ``@register`` decorators fire. Path components
|
||||
starting with ``_`` are skipped (used for shared helpers like
|
||||
``optical_flow/_shared.py`` and `vbench/_grit_helper.py`).
|
||||
"""
|
||||
|
||||
import importlib
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import VBAudioBoxAestheticsMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import VBFrechetDistanceAudioMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import VBKLDivergenceAudioMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import VBWERMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import LPIPSMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import PSNRMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import SSIMMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from fastvideo.eval.metrics.physics_iq.mse.metric import PhysicsIQMSEMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from fastvideo.eval.metrics.physics_iq.spatial_iou.metric import SpatialIoUMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from fastvideo.eval.metrics.physics_iq.spatiotemporal_iou.metric import SpatiotemporalIoUMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from fastvideo.eval.metrics.physics_iq.weighted_spatial_iou.metric import WeightedSpatialIoUMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import AppearanceStyleMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import BackgroundConsistencyMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import ColorMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import DynamicDegreeMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import HumanActionMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import ImagingQualityMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import MotionSmoothnessMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import MultipleObjectsMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import ObjectClassMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import OverallConsistencyMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import SceneMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import SpatialRelationshipMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import SubjectConsistencyMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import TemporalFlickeringMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import TemporalStyleMetric # noqa: F401
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
from .metric import VideoScore2Metric # noqa: F401
|
||||
|
||||
@@ -134,11 +134,9 @@ test = [
|
||||
eval = [
|
||||
"decord",
|
||||
"openai-clip",
|
||||
"open_clip_torch",
|
||||
"pyiqa",
|
||||
"ptlflow",
|
||||
"lpips",
|
||||
"torchmetrics",
|
||||
"easydict",
|
||||
"qwen-vl-utils",
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user