Three loose ends from earlier hygiene passes that the final PR review
caught:
* Empty out 27 sub-metric ``__init__.py`` files that still re-exported
the metric class. The earlier ``54f26bea`` commit only cleaned 2 of
them; the rest still carried ``from .metric import FooMetric # noqa:
F401`` lines copied from the original wm-eval port. Auto-discovery
doesn't need them, and ``get_metric("group.name")`` is the canonical
access pattern. Now uniformly empty across all sub-metric inits.
* Fix stale layout references in ``fastvideo/eval/README.md`` and
``.agents/workflows/evaluation-development.md`` that still described
the abandoned ``<bench>/external/upstream/`` per-metric layout. The
actual layout is the flat ``fastvideo/third_party/eval/<bench>/`` we
switched to during the port. Also drops the stale ``_third_party``
example from ``fastvideo/eval/metrics/__init__.py``'s docstring (the
underscore-skip rule is unchanged; the example was just wrong).
* Drop ``open_clip_torch`` and ``torchmetrics`` from the
``[eval]`` extra. Neither is imported anywhere under
``fastvideo/eval/`` (or in the vbench submodule we vendor); they were
carried over from the wm-eval port's heavier dep graph.
Verified: 32 metrics still register, 33/4 tests pass/skip on 1 GPU.
``Evaluator.evaluate`` and the one-shot ``evaluate()`` helper now accept
``video`` / ``reference`` as either a pre-loaded ``(T, C, H, W)`` tensor
or a path-like (``str`` / ``Path``). Paths are decoded inside the
worker thread that picks the sample up — see
``EvalWorker._resolve_video_input``.
Why this matters for batch eval: the dispatcher in
``Evaluator.evaluate(samples=[...])`` submits every sample to the
``ThreadPoolExecutor`` upfront. With pre-loaded tensors that meant
every video in the batch was resident in CPU RAM at once (≈ 3 GB per
video at 1088×1920×121); a full benchmark of hundreds of clips would
OOM before scoring started.
With paths, the queued futures hold cheap strings; only ``num_gpus``
videos are decoded concurrently. Peak resident memory becomes
``O(num_gpus)`` instead of ``O(len(samples))``. No dispatcher rewrite
needed — the change is ~10 lines at the worker boundary.
* ``EvalWorker.evaluate`` now normalizes ``sample["video"]`` and
``sample["reference"]`` through ``_resolve_video_input`` (path →
``load_video`` decode; ``(1, T, C, H, W)`` → squeeze; tensor →
passthrough). Metrics keep the existing contract: by the time
``compute()`` runs, ``sample["video"]`` is always a 4-D tensor.
* ``score_folder.py`` and ``bench_vbench.py`` simplified to pass paths
directly. ``bench_physics_iq.py`` already did.
* ``fastvideo.eval.api.evaluate`` signature widened to ``Tensor | str |
Path``; the worker handles the decode either way.
Tests: new ``test_evaluator_paths.py`` covers the kwargs form, the
samples-list form, mixing paths and tensors in the same batch, parity
between path-form and tensor-form scores, and the missing-path
exception path. End-to-end ``score_folder.py`` smoke run confirmed.
Two small fixes in ``fastvideo/entrypoints/cli/eval.py``:
* ``_expand_paths`` deduplicated through ``[x for x in out if not (x in
seen or seen.add(x))]`` — a known idiom that ruff flags because
``set.add`` returns ``None`` (B023/func-returns-value). Replace with
an explicit loop so the side effect doesn't ride on the boolean
expression.
* ``_cmd_run`` constructed ``metrics_arg`` through an
``if/else``-block where SIM108 (project policy) prefers a ternary.
Fold to a single conditional expression.
Pure mechanical cleanups; ``ruff check`` is now clean on the file and
the eval CLI smoke test (``score_video.py`` against a self-paired
mp4 → ``common.psnr=100``) still passes.
Pure formatting pass — no behavior changes. Most edits are one of:
* yapf splitting / re-flowing kwarg-heavy callsites (argparse setup,
``MetricResult(...)`` construction).
* ruff auto-fixes around imports (collapsing ``Iterable`` to
``collections.abc``, removing the ``typing.Tuple`` shim where
``tuple[...]`` works, dropping unused imports).
* ``zip(..., strict=False)`` added to every two-iterable zip call.
* ``getattr(np, "trapz")`` → ``np.trapz`` where the fallback can read
the attribute directly (the ``trapezoid`` lookup still uses
``getattr`` because the name itself is the moving target).
Verified the test suite still passes after the pass.
The two end-to-end runners called ``VideoGenerator.generate_video``
without specifying generation dimensions, leaving the model to fall
back on whatever default sampling shape it carries internally — which
isn't necessarily what the user wants for a benchmark run.
Mirror the basic ``examples/inference/basic/basic_ltx2.py`` script:
default to ``121 x 1088 x 1920`` (LTX2's intended sampling shape) and
expose ``--num-frames`` / ``--height`` / ``--width`` so users can
downscale for smoke runs without editing the script.
Verified end-to-end on a borrowed H200: ``bench_vbench.py --limit 1
--num-frames 49 --height 480 --width 768`` generates an mp4 in ~97s
and scores it with ``vbench.aesthetic_quality`` (auto-downloads
``Davids048/LTX2-Base-Diffusers`` weights through ``from_pretrained``).
Add four small example scripts under ``examples/inference/eval/``,
each driving the public eval API in a different real-world shape:
* ``score_video.py`` — score one mp4 on one GPU. Smallest possible
use of ``create_evaluator`` + ``Evaluator.evaluate``. Optional
``--reference``, ``--text-prompt``, ``--fps`` for paired / prompt-
aware / fps-aware metrics.
* ``score_folder.py`` — score every mp4 in a directory across ``--num-gpus``
replicas via ``Evaluator.evaluate(samples=[...])``. Pair each
generated video with a same-name reference under ``--reference-dir``
if you want paired metrics.
* ``bench_vbench.py`` — full VBench end-to-end: ``get_dataset("vbench",
dimensions=...)`` → ``VideoGenerator`` (LTX2 by default) → score
with the matching ``vbench.*`` sub-metrics → print per-metric
averages. ``--skip-generation`` re-uses existing mp4s under
``--videos-dir`` so you can iterate on metric selection without
re-paying generation cost.
* ``bench_physics_iq.py`` — full Physics-IQ end-to-end:
``get_dataset("physics_iq", dataset_root=...)`` → ``VideoGenerator``
→ score with the composite ``physics_iq`` metric → aggregate via the
upstream's ``aggregate_components`` recipe.
The pre-existing ``basic_ltx2_eval.py`` and ``eval_ltx2_vbench.py`` are
left in place — they target the single-prompt generate-and-score case
and complement (rather than overlap with) the new full-dataset runners.
Verified end-to-end on a borrowed GPU node:
* ``score_video.py``: PSNR=100, SSIM=1.0 on a self-paired mp4 (sanity).
* ``score_folder.py``: produces well-formed scores.json.
Add four end-to-end test modules under ``fastvideo/tests/eval/``,
all driving the public API only — no reaches into ``EvalWorker``,
``BaseMetric``, or the ``_REGISTRY`` dict. Each test mirrors a real
caller flow.
* ``test_registry.py`` — ``list_metrics`` invariants, group-prefix
resolution (verified against the no-model ``physics_iq`` group so
the test stays cheap), unknown-metric error path.
* ``test_evaluator_single.py`` — one-shot ``evaluate(...)``,
long-lived ``Evaluator`` with both kwargs and ``samples=[...]``
shapes, deterministic scoring, the legacy ``(1, T, C, H, W)``
back-compat unwrap. Runs on CPU using ``common.psnr`` /
``common.ssim`` so it's free in CI.
* ``test_evaluator_multi_gpu.py`` — auto-skips when fewer than 2 CUDA
devices visible. Pins the round-robin contract by computing a
single-GPU baseline and asserting bit-equivalent scores under
multi-GPU dispatch with the same input list, plus the kwargs-form
→ worker-0 invariant and ``release_cuda_memory`` no-crash check.
* ``test_evaluator_with_dataset.py`` — full ``get_dataset("vbench")
→ Evaluator.evaluate(**row)`` flow. Synthesizes random video tensors
per row (no diffusion model needed) and verifies that extra dataset
keys (``prompt``, ``n_samples``, ``dimensions``, ``auxiliary_info``)
flow through unused metrics without breaking them.
29 tests total: 25 pass on a single GPU, all 29 pass on 2 GPUs.
``fastvideo eval run --output scores.json`` would crash with
``TypeError: Object of type float32 is not JSON serializable`` whenever
the chosen metric set landed numpy or torch values in
``MetricResult.details``. The optical-flow metrics in particular
populate ``per_frame_metrics`` with ``np.float64`` scalars, and pretty
much any metric is one ``np.percentile`` call away from the same crash.
Pass a ``default=`` callback to ``json.dumps`` that walks the unknown
leaves and coerces:
* ``np.integer`` / ``np.floating`` / ``np.bool_`` → native Python scalars
* ``np.ndarray`` → ``.tolist()``
* ``torch.Tensor`` → detached CPU ``.tolist()``
* ``pathlib.Path`` → ``str``
Anything else still raises ``TypeError`` — the goal is to handle the
known metric outputs cleanly, not to silently coerce arbitrary objects.
The Physics-IQ metric package was carrying a ~200-line dataset loader
(``PhysicsIQDataLoader`` + ``PhysicsIQScenario`` + an FPS-conversion
helper + manifest-walking constants) inside its metric directory, with
``__init__.py`` re-exporting the loader as a public name. Two distinct
concerns were mixed: dataset traversal (a user-of-the-metric concern)
and metric-pipeline configuration (an internal concern).
Split them:
* New ``fastvideo/eval/datasets/physics_iq.py`` registers a
``PhysicsIQPromptDataset(PromptDataset)`` under
``@register_dataset("physics_iq")``. It walks ``descriptions.csv``,
resolves per-take video and real-mask paths, FPS-converts the source
release on cache miss, and yields one sample dict per take-1
scenario shaped to drop straight into ``Evaluator.evaluate(**row)``.
The previously-public ``PhysicsIQScenario`` dataclass moves with it.
* Metric defaults (``DEFAULT_TARGET_FPS=30``,
``DEFAULT_DURATION_SECONDS=5``) move into
``metrics/physics_iq/utils.py``, where they were already used as
default kwargs.
* ``metrics/physics_iq/models.py`` is deleted; ``__init__.py`` is
emptied to match the rest of the metric directories.
Users now do ``get_dataset("physics_iq", dataset_root=...)`` to walk
the corpus instead of reaching for ``PhysicsIQDataLoader`` directly.
The old import path is dropped without a back-compat shim — the eval
suite hasn't shipped yet, so there's no API contract to honor.
The eval-suite README (``fastvideo/eval/README.md``) gets a short
``Prompt datasets`` section showing the ``get_dataset`` workflow. The
project root README is unchanged.
Two sub-metric ``__init__.py`` files re-exported their metric class
(``VBClapScoreMetric``, ``AestheticQualityMetric``) while every other
sub-metric leaves ``__init__.py`` empty. Auto-discovery imports each
``metric.py`` module by full path, so the re-export was redundant —
and the inconsistency obscured the contract for new contributors
(adding a metric should not require touching ``__init__.py``).
Standardize on empty sub-metric ``__init__.py``; users instantiate
metrics through ``get_metric("group.name")`` rather than reaching for
the class object directly.
The ``physics_iq`` group's ``__init__.py`` also re-exports a dataset
loader and scenario dataclass; that one's left alone in this commit
pending a separate move of those helpers into ``fastvideo/eval/datasets/``.
The ``EvalWorker`` always invokes metrics on a single video and only
ever reads ``result[0]`` from the returned list, so every metric had a
dead ``for b in range(B)`` loop. Tighten the contract:
* ``BaseMetric.compute(sample) -> MetricResult`` — return one result,
not a one-element list.
* ``BaseMetric._skip(sample, reason) -> MetricResult`` likewise.
* Sample-side: ``video`` and ``reference`` are ``(T, C, H, W)`` —
no leading batch dim. The worker still unwraps a ``(1, T, C, H, W)``
caller for back-compat, so existing user code keeps working.
* ``Evaluator.metric_names`` now reads through a public
``EvalWorker.metric_names`` property instead of poking the private
``_metrics`` dict.
All 28 metrics — common.{ssim,psnr,lpips}, optical_flow.*, audio.*,
vbench.* (16), physics_iq.* (5 incl. composite), videoscore2 — drop
their ``for b in range(B)`` loop and return a single ``MetricResult``.
List-shaped optional inputs (``text_prompt``, ``audio``,
``auxiliary_info``, ``actions``) are still accepted: each metric
unwraps a single-element list before use, so callers can keep the
existing ``[prompt]`` convention or pass a scalar — both work.
Update the layout snippets in both ``fastvideo/eval/README.md`` and
``docs/contributing/eval-metrics.md`` to show the new
``optical_flow/`` group sibling to ``common/``, with the two
sub-metrics (``gt_optical_flow``, ``synthetic_optical_flow``) listed
explicitly.
Move ``common.optical_flow`` to a dedicated ``optical_flow`` group so
flow-based comparisons can register additional reference modes without
piling into ``common``. Two metrics live under the new group:
* ``optical_flow.gt_optical_flow`` — the existing video-vs-video
comparison, ported verbatim minus a thin shared-helper extraction.
* ``optical_flow.synthetic_optical_flow`` — new metric that takes a
per-frame action stream + a ``ThirdPersonCalibration`` JSON and
predicts the reference flow analytically (Longuet-Higgins + off-pivot
translation, no depth) instead of extracting it from a GT video.
Both metrics share ``optical_flow/_shared.py`` for ptlflow loading,
per-frame metric computation, and temporal aggregation, so scores are
directly comparable across the two reference modes.
The third-person predictor is vendored at
``optical_flow/synthetic_optical_flow/_thirdperson.py`` to keep the
metric self-contained; the calibration *fitter* (``calibrate.py`` etc.)
is intentionally not part of the eval suite — it's an offline fitting
tool.
Several scaffolds that pre-date the runner refactor still pointed at
removed APIs:
* ``EvalResult`` (and its ``Evaluator.evaluate_dataset`` docstring)
exposed a class that is never returned anywhere; drop it from the
public API.
* ``Evaluator``'s module docstring still referenced the removed
``EvalRunner`` layer; trim to the current Evaluator → EvalWorker
shape.
* ``WM_EVAL_CACHE`` survived as a fallback env var from the wm-eval
port; remove and keep only ``FASTVIDEO_EVAL_CACHE``.
* The contributor docs documented ``batch_unit`` and
``trial_forward``/``Evaluator.calibrate()`` which were dropped from
``BaseMetric`` in earlier refactors; update to the current contract.
The metric resolved the Kinetics-400 label file via a wm-eval-era
``_third_party/umt/kinetics_400_categories.txt`` path that no longer
exists after the vbench-as-submodule port. ``os.path.exists`` was
silently False, leaving the label dict empty; every prediction then
mapped to the empty string and every score returned 0.0.
Resolve the path through ``vbench.third_party.umt.__file__`` instead,
which always points at the pinned upstream submodule.
Two changes that together let motion_smoothness score 1088×1920×121
videos on shared GPUs:
1. _get_scale() now queries torch.cuda.mem_get_info() free memory on
every call instead of caching total_memory at setup() time.
Adapts to whatever's actually available — other metric replicas
already loaded, residual generator allocations, another process
sharing the GPU. Upstream cached total_memory once which on a
shared/loaded GPU lets AMT attempt 30+ GB correlation reshapes.
2. _safe_amt_forward() wraps the model call with two-axis OOM retry:
- Halve batch until size 1 (per-pair memory dominates).
- Halve scale_factor until 1/16 (AMT's internal feature-map
resolution and therefore correlation volume size).
- Bottom out at scale=1/16, batch=1; if still OOM, the resolution
is genuinely impossible at this headroom and we re-raise.
The second change is needed because upstream's autoscale formula in
_get_scale mis-extrapolates at high resolution: it scales linearly
in pixel count, but AMT's correlation volume grows quadratically.
Rather than rewriting upstream's formula, the retry path makes the
metric robust to whatever the formula picks.
Verified on fs-mbz-gpu-085 (shared with another job holding 45 GB):
Before: motion_smoothness OOM at 31.75 GB allocation on 1088×1920×121.
After: motion_smoothness=0.9897 across all 8 metrics, no OOM, all
other scores (aesthetic_quality=0.5430, subject_consistency=
0.8629, etc.) unchanged.
Parity against runs/vbench_smoke/results_v3.json byte-identical.
Two new entrypoints showcasing the post-refactor eval surface:
- examples/inference/eval/basic_ltx2_eval.py — generate one LTX2
video using the same parameters as basic_ltx2.py (same prompt,
model, 1088×1920×121), then score it with the prompt-aware VBench
subset. Builds Evaluator directly, calls evaluate(**kwargs) per
the single-sample API; no runner, no argparse.
- scripts/eval/score_folder.py — bulk-score every video in a folder
with prompt-free VBench metrics. Optional --prompts-json maps
filenames to prompts to enable prompt-aware metrics. Uses
Evaluator.evaluate(samples=[...]) for multi-GPU fan-out.
Tested on fs-mbz-gpu-085:
- score_folder.py: 3 duplicate mp4s × 4 metrics → 3 identical score
sets, summary aggregated, scores.json written. Clean run.
- basic_ltx2_eval.py: generation succeeded; scoring path verified
for 7 of 8 metrics on the LTX2 output (aesthetic_quality,
subject_consistency, background_consistency, imaging_quality,
temporal_flickering, dynamic_degree, overall_consistency).
vbench.motion_smoothness OOMs at 1088×1920 on a shared GPU because
VBench's AMT memory autoscale reads total_memory rather than
mem_get_info() free memory, so it underestimates the required
resolution scale-down when another process holds 45 GB. Documented
in the script docstring; not refactor-related (same behavior at
HEAD pre-refactor). Drop motion_smoothness from METRICS or run on
a dedicated GPU to score it.
Added a torch.cuda.empty_cache() between generator.shutdown() and
evaluator construction to free residual generation memory.
Completes the d1bc1413 cleanup pass on the five files that were
skipped because dataloader had pending modifications. With those
modifications now landed (commits 7cdbcc94 + c2a22560), the dead
code in optical_flow + the four aux-using vbench metrics
(color/multiple_objects/object_class/spatial_relationship) can go.
Mechanical removal of batch_unit class attrs and trial_forward
method overrides — neither is read anywhere since calibrate() was
deleted in 502f061d.
Parity verified on fs-mbz-gpu-085: byte-identical to v3 baseline.
Replace the single mean-EPE port with mhuo's complete validation set
(see fastvideo/training/ptlflow_validation.py in mhuo's tree):
Per-frame: mf_epe, mf_angle_err, mf_cosine, mf_mag_ratio,
pixel_epe_mean/max, px_angle_rmse, grid_epe_mean/max,
fl_all, foe_dist, flow_kl_2d
Aggregated over time: <name>_mean / _std / _max / _auc, plus
divergence_onset_frame / divergence_threshold
Added supporting math: least-squares Focus-of-Expansion estimation,
2D KL divergence over (angle, log-magnitude) histogram, trapezoid
integral with numpy 2.x compat shim.
Headline ``score`` is pixel_epe_mean_mean (lower is better);
everything else lives in MetricResult.details so downstream
consumers can pick whichever scalar they care about.
Pairs gen and ref videos across all B inputs and batches the model
through them together (chunk_size=16) for GPU efficiency.
Authored by dataloader.
The four aux-using vbench metrics (color, multiple_objects,
object_class, spatial_relationship) used to bail at the top of
compute() if auxiliary_info was None, then unconditionally indexed
aux[b]["<key>"] for every row. This crashed with KeyError on rows
that had aux but lacked the metric's specific key — common once
the dataset yields heterogeneous rows under "vbench" / multi-dim
runs (each row carries flat aux populated only for its own
dimensions).
Switch to per-row skip: each row checks for its own required key
and emits MetricResult(score=None, details={"skipped": "..."}) if
absent. Lets a single evaluator.evaluate(samples=[...]) call score
heterogeneous-aux rows in one pass.
Also handle two structural sub-cases:
- multiple_objects requires "<a> and <b>" in aux["object"]; rows
without the separator (single-object data leaking in) skip.
- object_class is the inverse: rows whose "object" contains " and "
belong to multiple_objects' territory and skip here.
- spatial_relationship's nested {object_a/object_b/relationship}
sub-dict is read defensively; missing keys skip with a reason.
Pairs with the flat-aux-at-load behavior in VBenchPromptDataset
(commit 06735161). Without these per-row skips, dimensions="all"
runs crash on the first row that lacks the active metric's key.
Authored by dataloader.
Two small additions paired with the in-progress eval orchestration:
- types.py: EvalResult dataclass with summary/per_video fields plus
from_raw / save / print helpers. Used by scripts and any future
callers that aggregate per-sample MetricResults into a corpus-level
summary (e.g. scripts/eval/run_vbench_e2e.py).
- .gitignore: exclude fastvideo/eval/_assets/ which holds calibration
data, downloaded videos, and run dumps used by the synthetic-flow
evaluator that we don't want to track.
Authored by dataloader.
The flatten loop in VBenchPromptDataset strips exactly one level of
upstream's {dim_name: ...} wrapper. For most VBench dims this leaves
a flat scalar dict ({"color": "red"}, {"object": "person"}). For
spatial_relationship, upstream double-wraps, so one level of unwrap
leaves {"spatial_relationship": {object_a, object_b, relationship}} —
which is exactly what SpatialRelationshipMetric reads.
That last case looks accidental; it isn't. Documenting all four shapes
inline so a future reader doesn't "simplify" the wrapping away and
silently break spatial_relationship scoring.
Empirically verified output for all four aux dims matches the
documented shapes.
The Evaluator's calibrate() is gone, so batch_unit class attrs and
trial_forward() method overrides have no callers — pure cruft from the
old auto-calibration design. Mechanical removal across 15 metric files.
Internal time-dim chunking (the part of batching that actually does
work) lives on as metric-owned _chunk_size constants set in __init__,
unaffected by this change.
Five files (color/multiple_objects/object_class/spatial_relationship/
optical_flow metrics) skipped — they have dataloader's in-flight
modifications uncommitted; left for that branch's cleanup pass.
Parity verified on fs-mbz-gpu-085: results byte-identical to v3
baseline.
Match FastVideo's existing depth: VideoGenerator is the top-level
inference object with no Runner above it; loops live in scripts.
Eval should mirror that — Evaluator is the top-level scoring object,
EvalWorker × N is the layer below, and end-to-end pipelines (prompts
→ generate → score) are scripts, not classes.
Removed:
- fastvideo/eval/runner.py (EvalRunner + classmethod constructors).
- EvalRunner export from fastvideo.eval.
Added:
- fastvideo/eval/io/paths.py with sanitize_prompt, default_filename,
glob_videos, build_eval_kwargs as free functions. The reusable bits
of the runner survive; the class wrapping them does not.
Rewritten:
- scripts/eval/run_vbench_e2e.py is now top-to-bottom procedural:
parse_args → dataset → optional generate() loop → optional score()
loop. Reads in one pass; no classmethod indirection to chase.
Parity verified on fs-mbz-gpu-085 against runs/vbench_smoke/videos/
(2 videos × 3 metrics, dimensions=subject_consistency):
- 1-GPU run (results_v4.json) is byte-identical to v3 baseline.
- 2-GPU run (results_v4_2gpu.json) is byte-identical to v3 baseline
(order-preserving fan-out via Evaluator.evaluate(samples=[...])).
Continues the eval refactor. Datasets now yield plain dicts (matches
ValidationDataset / VideoGenerator / Evaluator's kwargs-flowing-through
pattern). End-to-end orchestration moves into EvalRunner so neither the
dataset nor the Evaluator owns videos_dir / fps / filename / generator
state.
Layering:
EvalRunner dataset / generator / file conventions / manifest
└── Evaluator thin dispatcher (single evaluate(), 1 or N samples)
└── EvalWorker × N single-GPU metric replicas
Dataset (fastvideo/eval/datasets/):
- PromptDataset is just an Iterable[dict]. _rows holds dicts with
prompt / n_samples / dimensions / auxiliary_info / ... — unused
fields stay absent rather than living as Optional[None] on a
dataclass. BasePromptDataset alias kept for back-compat.
- BenchmarkSample dataclass deleted. Sample TypedDict documents the
recognized keys without forcing the schema.
- VBench flattens its nested {dim: {key: val}} aux into {key: val} at
load time. Aux-using metrics already read flat keys; no metric edits.
Runner (fastvideo/eval/runner.py, new):
- Three named constructors: from_dataset, from_videos, from_samples.
- generate() drives the generator over the dataset (writes manifest);
score() loads videos and dispatches via Evaluator.evaluate(list).
run() = generate then score.
- Multi-GPU lives in the underlying Evaluator; the runner just hands
num_gpus through. No double-orchestration.
- File naming convention (filename_fn) and eval-kwargs assembly
(eval_kwargs_fn) are runner-side hooks, not dataset overrides.
Script (scripts/eval/run_vbench_e2e.py):
- Rewritten against EvalRunner; ~95 LOC, was 149.
Includes prior scaffolding from dataloader (datasets/registry.py,
the test file, the script) carried over so the refactor lands as one
coherent state.
Mirrors FastVideo's VideoGenerator → Worker layering, in-process:
Evaluator (user-facing dispatcher)
└── EvalWorker × N (single-GPU, owns metric replicas)
EvalWorker (new) holds metric replicas on one device and scores one
sample at a time. Evaluator builds num_gpus workers eagerly in __init__
(every metric loaded on every replica) and exposes a single evaluate()
method that handles both shapes:
ev.evaluate(video=..., ...) → one sample, runs on worker 0
ev.evaluate([s1, s2, ...]) → fan out across workers
Removed (dead code or moved to runner in a follow-up):
- Evaluator.calibrate, _compute_chunked, _evaluate_multi_gpu,
_gpu_metrics, evaluate_dataset, B>1 input path
- BaseMetric.batch_unit and trial_forward (kept _chunk_size as a plain
default class attr; metrics use it for internal time-dim chunking)
- memory.is_batch_too_large, slice_sample (clear_cache stays)
Per-metric batch_unit/trial_forward overrides remain as harmless
class-attr cruft pending a mechanical removal pass.
Net: evaluator.py 447 → 135 LOC; metrics/base.py 92 → 66; memory.py
40 → 14; +78 LOC for worker.py.
Adds an in-process evaluation suite covering native (SSIM/PSNR/LPIPS/
optical_flow), audio, physics_iq, vbench (16 sub-metrics), and
videoscore2. Public API: create_evaluator/evaluate, BaseMetric +
@register, ensure_checkpoint, get_cache_dir. CLI: fastvideo eval
list/run.
VBench upstream pinned as a git submodule at
fastvideo/third_party/eval/vbench (Vchitect/VBench@45e79ec); modern-dep
compat is achieved via runtime shims in vbench/__init__.py rather than
on-disk patches. CLIP/torch.hub caches are routed through
${FASTVIDEO_EVAL_CACHE}; HF cache stays at the system default.
ensure_checkpoint delegates to fastvideo.utils.get_lock + huggingface_hub
primitives. Evaluator supports release_cuda_memory(), unload(),
reload() for training-time eval that frees GPU between calls.
Verified parity against upstream vbench (5/8 metrics bit-exact, others
within 1% drift driven by transformers/torch version skew) and against
upstream VideoScore2 (regex anchored on the actual model's output
format, soft-score formula matches upstream's argmax*max_prob/total).
Includes docs/contributing/eval-metrics.md as the porting guide.
Out of scope (deferred follow-ups): MIND, VBench-2.0, FVD as a
registered metric, training-time EvalCallback.
shaoxiongduan