shaoxiongduan
960485519f
Continues the eval refactor. Datasets now yield plain dicts (matches
ValidationDataset / VideoGenerator / Evaluator's kwargs-flowing-through
pattern). End-to-end orchestration moves into EvalRunner so neither the
dataset nor the Evaluator owns videos_dir / fps / filename / generator
state.
Layering:
EvalRunner dataset / generator / file conventions / manifest
└── Evaluator thin dispatcher (single evaluate(), 1 or N samples)
└── EvalWorker × N single-GPU metric replicas
Dataset (fastvideo/eval/datasets/):
- PromptDataset is just an Iterable[dict]. _rows holds dicts with
prompt / n_samples / dimensions / auxiliary_info / ... — unused
fields stay absent rather than living as Optional[None] on a
dataclass. BasePromptDataset alias kept for back-compat.
- BenchmarkSample dataclass deleted. Sample TypedDict documents the
recognized keys without forcing the schema.
- VBench flattens its nested {dim: {key: val}} aux into {key: val} at
load time. Aux-using metrics already read flat keys; no metric edits.
Runner (fastvideo/eval/runner.py, new):
- Three named constructors: from_dataset, from_videos, from_samples.
- generate() drives the generator over the dataset (writes manifest);
score() loads videos and dispatches via Evaluator.evaluate(list).
run() = generate then score.
- Multi-GPU lives in the underlying Evaluator; the runner just hands
num_gpus through. No double-orchestration.
- File naming convention (filename_fn) and eval-kwargs assembly
(eval_kwargs_fn) are runner-side hooks, not dataset overrides.
Script (scripts/eval/run_vbench_e2e.py):
- Rewritten against EvalRunner; ~95 LOC, was 149.
Includes prior scaffolding from dataloader (datasets/registry.py,
the test file, the script) carried over so the refactor lands as one
coherent state.
54 lines
1.8 KiB
Python
54 lines
1.8 KiB
Python
"""Prompt-corpus datasets for end-to-end benchmark evaluation.
|
|
|
|
Public API mirrors :mod:`fastvideo.eval` (metrics side):
|
|
|
|
from fastvideo.eval.datasets import (
|
|
PromptDataset, Sample,
|
|
register_dataset, get_dataset, list_datasets,
|
|
)
|
|
|
|
A dataset is an iterable of plain dicts (one per sample). Built-in
|
|
datasets self-register at import time. To add one, drop a module into
|
|
this package that subclasses :class:`PromptDataset` and decorates with
|
|
``@register_dataset("name")`` — auto-discovery picks it up.
|
|
"""
|
|
from fastvideo.eval.datasets.base import (BasePromptDataset, PromptDataset,
|
|
Sample)
|
|
from fastvideo.eval.datasets.registry import (get_dataset, list_datasets,
|
|
register_dataset)
|
|
|
|
|
|
def _autodiscover() -> None:
|
|
"""Import every non-underscore .py module / subpackage in this package
|
|
so the ``@register_dataset`` decorators fire."""
|
|
import importlib
|
|
import os
|
|
|
|
for entry in os.listdir(os.path.dirname(__file__)):
|
|
if entry.startswith("_") or entry.startswith("."):
|
|
continue
|
|
if entry in {"base.py", "registry.py"}:
|
|
continue
|
|
if entry.endswith(".py"):
|
|
importlib.import_module(f"{__name__}.{entry[:-3]}")
|
|
elif os.path.isdir(os.path.join(os.path.dirname(__file__), entry)) \
|
|
and os.path.exists(os.path.join(
|
|
os.path.dirname(__file__), entry, "__init__.py")):
|
|
importlib.import_module(f"{__name__}.{entry}")
|
|
|
|
|
|
_autodiscover()
|
|
|
|
# Re-export the canonical class for typed imports.
|
|
from fastvideo.eval.datasets.vbench import VBenchPromptDataset # noqa: E402
|
|
|
|
__all__ = [
|
|
"PromptDataset",
|
|
"BasePromptDataset",
|
|
"Sample",
|
|
"register_dataset",
|
|
"get_dataset",
|
|
"list_datasets",
|
|
"VBenchPromptDataset",
|
|
]
|