shaoxiongduan
d07a7691fe
Match FastVideo's existing depth: VideoGenerator is the top-level inference object with no Runner above it; loops live in scripts. Eval should mirror that — Evaluator is the top-level scoring object, EvalWorker × N is the layer below, and end-to-end pipelines (prompts → generate → score) are scripts, not classes. Removed: - fastvideo/eval/runner.py (EvalRunner + classmethod constructors). - EvalRunner export from fastvideo.eval. Added: - fastvideo/eval/io/paths.py with sanitize_prompt, default_filename, glob_videos, build_eval_kwargs as free functions. The reusable bits of the runner survive; the class wrapping them does not. Rewritten: - scripts/eval/run_vbench_e2e.py is now top-to-bottom procedural: parse_args → dataset → optional generate() loop → optional score() loop. Reads in one pass; no classmethod indirection to chase. Parity verified on fs-mbz-gpu-085 against runs/vbench_smoke/videos/ (2 videos × 3 metrics, dimensions=subject_consistency): - 1-GPU run (results_v4.json) is byte-identical to v3 baseline. - 2-GPU run (results_v4_2gpu.json) is byte-identical to v3 baseline (order-preserving fan-out via Evaluator.evaluate(samples=[...])).
47 lines
1.5 KiB
Python
47 lines
1.5 KiB
Python
from fastvideo.eval.models import ensure_checkpoint, get_cache_dir
|
|
|
|
|
|
def _redirect_third_party_caches() -> None:
|
|
"""Point libraries that respect env vars at the eval cache root.
|
|
|
|
Run before metric modules import torch.hub (and friends) so the
|
|
redirect actually takes effect. We intentionally leave ``HF_HOME``
|
|
alone — HF's default cache (``~/.cache/huggingface/hub``) is widely
|
|
shared with other ML projects, and isolating it for eval would force
|
|
users to re-download already-cached transformers weights.
|
|
|
|
Other libraries with non-standard caches (CLIP, pyiqa) don't honour
|
|
env vars at all; their callsites in metric.py files pass
|
|
``download_root=str(get_cache_dir() / "<library>")`` directly.
|
|
"""
|
|
import os
|
|
root = get_cache_dir()
|
|
os.environ.setdefault("TORCH_HOME", str(root / "torch"))
|
|
|
|
|
|
_redirect_third_party_caches()
|
|
|
|
|
|
from fastvideo.eval.types import EvalResult, MetricResult # noqa: E402
|
|
from fastvideo.eval.metrics.base import BaseMetric # noqa: E402
|
|
from fastvideo.eval.registry import register, list_metrics, get_metric # noqa: E402
|
|
from fastvideo.eval.api import evaluate # noqa: E402
|
|
from fastvideo.eval.evaluator import Evaluator, create_evaluator # noqa: E402
|
|
|
|
# Trigger metric auto-discovery
|
|
import fastvideo.eval.metrics # noqa: F401, E402
|
|
|
|
__all__ = [
|
|
"evaluate",
|
|
"Evaluator",
|
|
"create_evaluator",
|
|
"MetricResult",
|
|
"EvalResult",
|
|
"BaseMetric",
|
|
"register",
|
|
"list_metrics",
|
|
"get_metric",
|
|
"ensure_checkpoint",
|
|
"get_cache_dir",
|
|
]
|