Voice pass over the three eval-related docs to bring them in line with
the project's existing voice (see docs/contributing/pull_requests.md,
docs/contributing/testing.md, docs/getting_started/installation/gpu.md):
* fastvideo/eval/README.md
* docs/contributing/eval-metrics.md
* .agents/workflows/evaluation-development.md
Concretely: cut em-dashes from ~30 across the three files to 1 (left
in a table cell where it reads naturally), removed "no X, no Y"
negation patterns, dropped buzzy phrasing ("first-class", "drop a
file", "out of the box"), and demoted bold imperatives ("Do not X")
to plain prose where the surrounding context already conveys the
instruction.
Two stale path references in evaluation-development.md fixed while
the file was open:
* "Update .agents/skills/evaluate-video-quality.md" pointed at a flat
file; skills in this repo are directories. Corrected to
.agents/skills/evaluate-video-quality/SKILL.md.
* "Check the evaluation_registry.md" referenced a bare filename that
does not exist; aligned to .agents/memory/evaluation-registry/README.md
to match the other references in the same doc.
No code changes; tests still 33/4 on a 1-GPU borrow.
`get_dataset("physics_iq")` now works with no kwargs. The manifest CSV is
vendored under fastvideo/eval/metrics/physics_iq/_vendored/; per-scenario
videos, masks, and switch-frames auto-fetch on first use from the public
DeepMind bucket (https://storage.googleapis.com/physics-iq-benchmark) into
${FASTVIDEO_EVAL_CACHE}/datasets/physics_iq/, sibling to the existing
models/torch/clip/ subdirs.
Why
- Old default dataset_root was /root/physics-IQ-benchmark, which doesn't
exist on shared hosts and isn't documented anywhere.
- Examples crashed with FileNotFoundError before any user-facing message
pointed at where to get the data.
- The official upstream download script needs gcloud SDK; the bucket is
also reachable over plain HTTPS (verified) so no SDK dependency is
needed.
Behavior
- dataset_root= is now an opt-in override (mirroring vbench's
full_info_path=); defaults to get_cache_dir() / "datasets" /
"physics_iq".
- auto_download=True by default; flip to False for air-gapped runs.
- FASTVIDEO_PHYSICS_IQ_BUCKET_URL env var redirects to internal mirrors.
- Atomic .part -> rename for safe concurrent SLURM-rank fetches.
Bench script fix
- bench_physics_iq.py: --limit was applied as a post-construction
list slice, after the dataset module had already eagerly resolved
every scenario's on-disk paths. With auto-download that means we'd
pull all 198 scenarios on every smoke run. Pass limit=args.limit
through to get_dataset() so partial-data smoke runs only fetch what
they need. --dataset-root is now optional (defaults to the cache
path).
Convention for future vendored files
- _vendored/ subdirs hold upstream-provenance content. They're
auto-skipped by metric discovery (the leading _) and by codespell
(single */_vendored/* glob in [tool.codespell].skip), so future
vendored files require no further config.
- docs/contributing/eval-metrics.md updated to point at this convention
for new metrics that ship paired-reference datasets.
Tested
- pytest fastvideo/tests/eval/ -> 33 passed (1-GPU subset); 4 multi-GPU
tests skipped on 1-GPU borrow as expected.
- Smoke: get_dataset("physics_iq", limit=1) auto-fetches the 5 expected
assets; physics_iq.{mse,spatial_iou} on take-1 self-pair returns
0.0 / 1.0.
Pure formatting pass — no behavior changes. Most edits are one of:
* yapf splitting / re-flowing kwarg-heavy callsites (argparse setup,
``MetricResult(...)`` construction).
* ruff auto-fixes around imports (collapsing ``Iterable`` to
``collections.abc``, removing the ``typing.Tuple`` shim where
``tuple[...]`` works, dropping unused imports).
* ``zip(..., strict=False)`` added to every two-iterable zip call.
* ``getattr(np, "trapz")`` → ``np.trapz`` where the fallback can read
the attribute directly (the ``trapezoid`` lookup still uses
``getattr`` because the name itself is the moving target).
Verified the test suite still passes after the pass.
Update the layout snippets in both ``fastvideo/eval/README.md`` and
``docs/contributing/eval-metrics.md`` to show the new
``optical_flow/`` group sibling to ``common/``, with the two
sub-metrics (``gt_optical_flow``, ``synthetic_optical_flow``) listed
explicitly.
Several scaffolds that pre-date the runner refactor still pointed at
removed APIs:
* ``EvalResult`` (and its ``Evaluator.evaluate_dataset`` docstring)
exposed a class that is never returned anywhere; drop it from the
public API.
* ``Evaluator``'s module docstring still referenced the removed
``EvalRunner`` layer; trim to the current Evaluator → EvalWorker
shape.
* ``WM_EVAL_CACHE`` survived as a fallback env var from the wm-eval
port; remove and keep only ``FASTVIDEO_EVAL_CACHE``.
* The contributor docs documented ``batch_unit`` and
``trial_forward``/``Evaluator.calibrate()`` which were dropped from
``BaseMetric`` in earlier refactors; update to the current contract.
Adds an in-process evaluation suite covering native (SSIM/PSNR/LPIPS/
optical_flow), audio, physics_iq, vbench (16 sub-metrics), and
videoscore2. Public API: create_evaluator/evaluate, BaseMetric +
@register, ensure_checkpoint, get_cache_dir. CLI: fastvideo eval
list/run.
VBench upstream pinned as a git submodule at
fastvideo/third_party/eval/vbench (Vchitect/VBench@45e79ec); modern-dep
compat is achieved via runtime shims in vbench/__init__.py rather than
on-disk patches. CLIP/torch.hub caches are routed through
${FASTVIDEO_EVAL_CACHE}; HF cache stays at the system default.
ensure_checkpoint delegates to fastvideo.utils.get_lock + huggingface_hub
primitives. Evaluator supports release_cuda_memory(), unload(),
reload() for training-time eval that frees GPU between calls.
Verified parity against upstream vbench (5/8 metrics bit-exact, others
within 1% drift driven by transformers/torch version skew) and against
upstream VideoScore2 (regex anchored on the actual model's output
format, soft-score formula matches upstream's argmax*max_prob/total).
Includes docs/contributing/eval-metrics.md as the porting guide.
Out of scope (deferred follow-ups): MIND, VBench-2.0, FVD as a
registered metric, training-time EvalCallback.
shaoxiongduan