shaoxiongduan
960485519f
Continues the eval refactor. Datasets now yield plain dicts (matches
ValidationDataset / VideoGenerator / Evaluator's kwargs-flowing-through
pattern). End-to-end orchestration moves into EvalRunner so neither the
dataset nor the Evaluator owns videos_dir / fps / filename / generator
state.
Layering:
EvalRunner dataset / generator / file conventions / manifest
└── Evaluator thin dispatcher (single evaluate(), 1 or N samples)
└── EvalWorker × N single-GPU metric replicas
Dataset (fastvideo/eval/datasets/):
- PromptDataset is just an Iterable[dict]. _rows holds dicts with
prompt / n_samples / dimensions / auxiliary_info / ... — unused
fields stay absent rather than living as Optional[None] on a
dataclass. BasePromptDataset alias kept for back-compat.
- BenchmarkSample dataclass deleted. Sample TypedDict documents the
recognized keys without forcing the schema.
- VBench flattens its nested {dim: {key: val}} aux into {key: val} at
load time. Aux-using metrics already read flat keys; no metric edits.
Runner (fastvideo/eval/runner.py, new):
- Three named constructors: from_dataset, from_videos, from_samples.
- generate() drives the generator over the dataset (writes manifest);
score() loads videos and dispatches via Evaluator.evaluate(list).
run() = generate then score.
- Multi-GPU lives in the underlying Evaluator; the runner just hands
num_gpus through. No double-orchestration.
- File naming convention (filename_fn) and eval-kwargs assembly
(eval_kwargs_fn) are runner-side hooks, not dataset overrides.
Script (scripts/eval/run_vbench_e2e.py):
- Rewritten against EvalRunner; ~95 LOC, was 149.
Includes prior scaffolding from dataloader (datasets/registry.py,
the test file, the script) carried over so the refactor lands as one
coherent state.
82 lines
2.6 KiB
Python
82 lines
2.6 KiB
Python
"""Prompt-corpus datasets.
|
|
|
|
A :class:`PromptDataset` is an iterable of *sample dicts* describing the
|
|
prompts and conditions for a benchmark. Each sample is a plain dict —
|
|
no dataclass, no schema enforcement — that flows directly into both
|
|
generation (``VideoGenerator.generate_video(**sample)``) and scoring
|
|
(``Evaluator.evaluate(**eval_kwargs)``). The runner picks well-known
|
|
keys (``prompt``, ``n_samples``, ``dimensions``, ``auxiliary_info``,
|
|
...) and passes the rest through.
|
|
|
|
This matches the surrounding FastVideo style:
|
|
|
|
* :class:`fastvideo.dataset.validation_dataset.ValidationDataset` yields dicts.
|
|
* :meth:`fastvideo.VideoGenerator.generate_video` consumes ``**kwargs``.
|
|
* :meth:`fastvideo.eval.Evaluator.evaluate` consumes ``**kwargs``.
|
|
|
|
To add a new benchmark:
|
|
|
|
1. Subclass :class:`PromptDataset`, populate ``self._rows`` with dicts in
|
|
``__init__``.
|
|
2. Decorate with ``@register_dataset("my_bench")``.
|
|
|
|
Convention for ``auxiliary_info``: a *flat* dict of metric-keyed values
|
|
(e.g. ``{"color": "red"}``). Benchmarks with nested aux schemas (VBench's
|
|
``{dim: {key: val}}``) flatten at load time so every consumer sees the
|
|
same shape.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
from abc import ABC, abstractmethod
|
|
from typing import Iterator, TypedDict
|
|
|
|
|
|
class Sample(TypedDict, total=False):
|
|
"""Documented schema for a row yielded by :class:`PromptDataset`.
|
|
|
|
Only ``prompt`` is required. Extra keys beyond these are forwarded to
|
|
the runner's eval-kwargs builder verbatim, so action-conditioned or
|
|
audio-bearing benchmarks can add their own fields without changing
|
|
the base class.
|
|
"""
|
|
prompt: str
|
|
n_samples: int
|
|
dimensions: list[str]
|
|
auxiliary_info: dict
|
|
image_path: str
|
|
reference_video: str
|
|
|
|
|
|
class PromptDataset(ABC):
|
|
"""Iterable corpus of sample dicts."""
|
|
|
|
name: str = ""
|
|
description: str = ""
|
|
supports_dimensions: bool = False
|
|
requires_reference_image: bool = False
|
|
requires_reference_video: bool = False
|
|
|
|
def __init__(self) -> None:
|
|
self._rows: list[dict] = []
|
|
|
|
def __iter__(self) -> Iterator[dict]:
|
|
return iter(self._rows)
|
|
|
|
def __len__(self) -> int:
|
|
return len(self._rows)
|
|
|
|
def __getitem__(self, i: int) -> dict:
|
|
return self._rows[i]
|
|
|
|
def by_dimension(self) -> dict[str, list[dict]]:
|
|
"""Group samples by dimension. A multi-dim sample appears under each."""
|
|
out: dict[str, list[dict]] = {}
|
|
for s in self._rows:
|
|
for d in s.get("dimensions", ()):
|
|
out.setdefault(d, []).append(s)
|
|
return out
|
|
|
|
|
|
# Back-compat alias for callers still importing the old class name.
|
|
BasePromptDataset = PromptDataset
|