`get_dataset("physics_iq")` now works with no kwargs. The manifest CSV is
vendored under fastvideo/eval/metrics/physics_iq/_vendored/; per-scenario
videos, masks, and switch-frames auto-fetch on first use from the public
DeepMind bucket (https://storage.googleapis.com/physics-iq-benchmark) into
${FASTVIDEO_EVAL_CACHE}/datasets/physics_iq/, sibling to the existing
models/torch/clip/ subdirs.
Why
- Old default dataset_root was /root/physics-IQ-benchmark, which doesn't
exist on shared hosts and isn't documented anywhere.
- Examples crashed with FileNotFoundError before any user-facing message
pointed at where to get the data.
- The official upstream download script needs gcloud SDK; the bucket is
also reachable over plain HTTPS (verified) so no SDK dependency is
needed.
Behavior
- dataset_root= is now an opt-in override (mirroring vbench's
full_info_path=); defaults to get_cache_dir() / "datasets" /
"physics_iq".
- auto_download=True by default; flip to False for air-gapped runs.
- FASTVIDEO_PHYSICS_IQ_BUCKET_URL env var redirects to internal mirrors.
- Atomic .part -> rename for safe concurrent SLURM-rank fetches.
Bench script fix
- bench_physics_iq.py: --limit was applied as a post-construction
list slice, after the dataset module had already eagerly resolved
every scenario's on-disk paths. With auto-download that means we'd
pull all 198 scenarios on every smoke run. Pass limit=args.limit
through to get_dataset() so partial-data smoke runs only fetch what
they need. --dataset-root is now optional (defaults to the cache
path).
Convention for future vendored files
- _vendored/ subdirs hold upstream-provenance content. They're
auto-skipped by metric discovery (the leading _) and by codespell
(single */_vendored/* glob in [tool.codespell].skip), so future
vendored files require no further config.
- docs/contributing/eval-metrics.md updated to point at this convention
for new metrics that ship paired-reference datasets.
Tested
- pytest fastvideo/tests/eval/ -> 33 passed (1-GPU subset); 4 multi-GPU
tests skipped on 1-GPU borrow as expected.
- Smoke: get_dataset("physics_iq", limit=1) auto-fetches the 5 expected
assets; physics_iq.{mse,spatial_iou} on take-1 self-pair returns
0.0 / 1.0.
shaoxiongduan
419e1c68f1