``Evaluator.evaluate`` and the one-shot ``evaluate()`` helper now accept
``video`` / ``reference`` as either a pre-loaded ``(T, C, H, W)`` tensor
or a path-like (``str`` / ``Path``). Paths are decoded inside the
worker thread that picks the sample up — see
``EvalWorker._resolve_video_input``.
Why this matters for batch eval: the dispatcher in
``Evaluator.evaluate(samples=[...])`` submits every sample to the
``ThreadPoolExecutor`` upfront. With pre-loaded tensors that meant
every video in the batch was resident in CPU RAM at once (≈ 3 GB per
video at 1088×1920×121); a full benchmark of hundreds of clips would
OOM before scoring started.
With paths, the queued futures hold cheap strings; only ``num_gpus``
videos are decoded concurrently. Peak resident memory becomes
``O(num_gpus)`` instead of ``O(len(samples))``. No dispatcher rewrite
needed — the change is ~10 lines at the worker boundary.
* ``EvalWorker.evaluate`` now normalizes ``sample["video"]`` and
``sample["reference"]`` through ``_resolve_video_input`` (path →
``load_video`` decode; ``(1, T, C, H, W)`` → squeeze; tensor →
passthrough). Metrics keep the existing contract: by the time
``compute()`` runs, ``sample["video"]`` is always a 4-D tensor.
* ``score_folder.py`` and ``bench_vbench.py`` simplified to pass paths
directly. ``bench_physics_iq.py`` already did.
* ``fastvideo.eval.api.evaluate`` signature widened to ``Tensor | str |
Path``; the worker handles the decode either way.
Tests: new ``test_evaluator_paths.py`` covers the kwargs form, the
samples-list form, mixing paths and tensors in the same batch, parity
between path-form and tensor-form scores, and the missing-path
exception path. End-to-end ``score_folder.py`` smoke run confirmed.
The two end-to-end runners called ``VideoGenerator.generate_video``
without specifying generation dimensions, leaving the model to fall
back on whatever default sampling shape it carries internally — which
isn't necessarily what the user wants for a benchmark run.
Mirror the basic ``examples/inference/basic/basic_ltx2.py`` script:
default to ``121 x 1088 x 1920`` (LTX2's intended sampling shape) and
expose ``--num-frames`` / ``--height`` / ``--width`` so users can
downscale for smoke runs without editing the script.
Verified end-to-end on a borrowed H200: ``bench_vbench.py --limit 1
--num-frames 49 --height 480 --width 768`` generates an mp4 in ~97s
and scores it with ``vbench.aesthetic_quality`` (auto-downloads
``Davids048/LTX2-Base-Diffusers`` weights through ``from_pretrained``).
Add four small example scripts under ``examples/inference/eval/``,
each driving the public eval API in a different real-world shape:
* ``score_video.py`` — score one mp4 on one GPU. Smallest possible
use of ``create_evaluator`` + ``Evaluator.evaluate``. Optional
``--reference``, ``--text-prompt``, ``--fps`` for paired / prompt-
aware / fps-aware metrics.
* ``score_folder.py`` — score every mp4 in a directory across ``--num-gpus``
replicas via ``Evaluator.evaluate(samples=[...])``. Pair each
generated video with a same-name reference under ``--reference-dir``
if you want paired metrics.
* ``bench_vbench.py`` — full VBench end-to-end: ``get_dataset("vbench",
dimensions=...)`` → ``VideoGenerator`` (LTX2 by default) → score
with the matching ``vbench.*`` sub-metrics → print per-metric
averages. ``--skip-generation`` re-uses existing mp4s under
``--videos-dir`` so you can iterate on metric selection without
re-paying generation cost.
* ``bench_physics_iq.py`` — full Physics-IQ end-to-end:
``get_dataset("physics_iq", dataset_root=...)`` → ``VideoGenerator``
→ score with the composite ``physics_iq`` metric → aggregate via the
upstream's ``aggregate_components`` recipe.
The pre-existing ``basic_ltx2_eval.py`` and ``eval_ltx2_vbench.py`` are
left in place — they target the single-prompt generate-and-score case
and complement (rather than overlap with) the new full-dataset runners.
Verified end-to-end on a borrowed GPU node:
* ``score_video.py``: PSNR=100, SSIM=1.0 on a self-paired mp4 (sanity).
* ``score_folder.py``: produces well-formed scores.json.
Two new entrypoints showcasing the post-refactor eval surface:
- examples/inference/eval/basic_ltx2_eval.py — generate one LTX2
video using the same parameters as basic_ltx2.py (same prompt,
model, 1088×1920×121), then score it with the prompt-aware VBench
subset. Builds Evaluator directly, calls evaluate(**kwargs) per
the single-sample API; no runner, no argparse.
- scripts/eval/score_folder.py — bulk-score every video in a folder
with prompt-free VBench metrics. Optional --prompts-json maps
filenames to prompts to enable prompt-aware metrics. Uses
Evaluator.evaluate(samples=[...]) for multi-GPU fan-out.
Tested on fs-mbz-gpu-085:
- score_folder.py: 3 duplicate mp4s × 4 metrics → 3 identical score
sets, summary aggregated, scores.json written. Clean run.
- basic_ltx2_eval.py: generation succeeded; scoring path verified
for 7 of 8 metrics on the LTX2 output (aesthetic_quality,
subject_consistency, background_consistency, imaging_quality,
temporal_flickering, dynamic_degree, overall_consistency).
vbench.motion_smoothness OOMs at 1088×1920 on a shared GPU because
VBench's AMT memory autoscale reads total_memory rather than
mem_get_info() free memory, so it underestimates the required
resolution scale-down when another process holds 45 GB. Documented
in the script docstring; not refactor-related (same behavior at
HEAD pre-refactor). Drop motion_smoothness from METRICS or run on
a dedicated GPU to score it.
Added a torch.cuda.empty_cache() between generator.shutdown() and
evaluator construction to free residual generation memory.
Adds an in-process evaluation suite covering native (SSIM/PSNR/LPIPS/
optical_flow), audio, physics_iq, vbench (16 sub-metrics), and
videoscore2. Public API: create_evaluator/evaluate, BaseMetric +
@register, ensure_checkpoint, get_cache_dir. CLI: fastvideo eval
list/run.
VBench upstream pinned as a git submodule at
fastvideo/third_party/eval/vbench (Vchitect/VBench@45e79ec); modern-dep
compat is achieved via runtime shims in vbench/__init__.py rather than
on-disk patches. CLIP/torch.hub caches are routed through
${FASTVIDEO_EVAL_CACHE}; HF cache stays at the system default.
ensure_checkpoint delegates to fastvideo.utils.get_lock + huggingface_hub
primitives. Evaluator supports release_cuda_memory(), unload(),
reload() for training-time eval that frees GPU between calls.
Verified parity against upstream vbench (5/8 metrics bit-exact, others
within 1% drift driven by transformers/torch version skew) and against
upstream VideoScore2 (regex anchored on the actual model's output
format, soft-score formula matches upstream's argmax*max_prob/total).
Includes docs/contributing/eval-metrics.md as the porting guide.
Out of scope (deferred follow-ups): MIND, VBench-2.0, FVD as a
registered metric, training-time EvalCallback.
shaoxiongduan