The metric resolved the Kinetics-400 label file via a wm-eval-era
``_third_party/umt/kinetics_400_categories.txt`` path that no longer
exists after the vbench-as-submodule port. ``os.path.exists`` was
silently False, leaving the label dict empty; every prediction then
mapped to the empty string and every score returned 0.0.
Resolve the path through ``vbench.third_party.umt.__file__`` instead,
which always points at the pinned upstream submodule.
Two changes that together let motion_smoothness score 1088×1920×121
videos on shared GPUs:
1. _get_scale() now queries torch.cuda.mem_get_info() free memory on
every call instead of caching total_memory at setup() time.
Adapts to whatever's actually available — other metric replicas
already loaded, residual generator allocations, another process
sharing the GPU. Upstream cached total_memory once which on a
shared/loaded GPU lets AMT attempt 30+ GB correlation reshapes.
2. _safe_amt_forward() wraps the model call with two-axis OOM retry:
- Halve batch until size 1 (per-pair memory dominates).
- Halve scale_factor until 1/16 (AMT's internal feature-map
resolution and therefore correlation volume size).
- Bottom out at scale=1/16, batch=1; if still OOM, the resolution
is genuinely impossible at this headroom and we re-raise.
The second change is needed because upstream's autoscale formula in
_get_scale mis-extrapolates at high resolution: it scales linearly
in pixel count, but AMT's correlation volume grows quadratically.
Rather than rewriting upstream's formula, the retry path makes the
metric robust to whatever the formula picks.
Verified on fs-mbz-gpu-085 (shared with another job holding 45 GB):
Before: motion_smoothness OOM at 31.75 GB allocation on 1088×1920×121.
After: motion_smoothness=0.9897 across all 8 metrics, no OOM, all
other scores (aesthetic_quality=0.5430, subject_consistency=
0.8629, etc.) unchanged.
Parity against runs/vbench_smoke/results_v3.json byte-identical.
Two new entrypoints showcasing the post-refactor eval surface:
- examples/inference/eval/basic_ltx2_eval.py — generate one LTX2
video using the same parameters as basic_ltx2.py (same prompt,
model, 1088×1920×121), then score it with the prompt-aware VBench
subset. Builds Evaluator directly, calls evaluate(**kwargs) per
the single-sample API; no runner, no argparse.
- scripts/eval/score_folder.py — bulk-score every video in a folder
with prompt-free VBench metrics. Optional --prompts-json maps
filenames to prompts to enable prompt-aware metrics. Uses
Evaluator.evaluate(samples=[...]) for multi-GPU fan-out.
Tested on fs-mbz-gpu-085:
- score_folder.py: 3 duplicate mp4s × 4 metrics → 3 identical score
sets, summary aggregated, scores.json written. Clean run.
- basic_ltx2_eval.py: generation succeeded; scoring path verified
for 7 of 8 metrics on the LTX2 output (aesthetic_quality,
subject_consistency, background_consistency, imaging_quality,
temporal_flickering, dynamic_degree, overall_consistency).
vbench.motion_smoothness OOMs at 1088×1920 on a shared GPU because
VBench's AMT memory autoscale reads total_memory rather than
mem_get_info() free memory, so it underestimates the required
resolution scale-down when another process holds 45 GB. Documented
in the script docstring; not refactor-related (same behavior at
HEAD pre-refactor). Drop motion_smoothness from METRICS or run on
a dedicated GPU to score it.
Added a torch.cuda.empty_cache() between generator.shutdown() and
evaluator construction to free residual generation memory.
Completes the d1bc1413 cleanup pass on the five files that were
skipped because dataloader had pending modifications. With those
modifications now landed (commits 7cdbcc94 + c2a22560), the dead
code in optical_flow + the four aux-using vbench metrics
(color/multiple_objects/object_class/spatial_relationship) can go.
Mechanical removal of batch_unit class attrs and trial_forward
method overrides — neither is read anywhere since calibrate() was
deleted in 502f061d.
Parity verified on fs-mbz-gpu-085: byte-identical to v3 baseline.
Replace the single mean-EPE port with mhuo's complete validation set
(see fastvideo/training/ptlflow_validation.py in mhuo's tree):
Per-frame: mf_epe, mf_angle_err, mf_cosine, mf_mag_ratio,
pixel_epe_mean/max, px_angle_rmse, grid_epe_mean/max,
fl_all, foe_dist, flow_kl_2d
Aggregated over time: <name>_mean / _std / _max / _auc, plus
divergence_onset_frame / divergence_threshold
Added supporting math: least-squares Focus-of-Expansion estimation,
2D KL divergence over (angle, log-magnitude) histogram, trapezoid
integral with numpy 2.x compat shim.
Headline ``score`` is pixel_epe_mean_mean (lower is better);
everything else lives in MetricResult.details so downstream
consumers can pick whichever scalar they care about.
Pairs gen and ref videos across all B inputs and batches the model
through them together (chunk_size=16) for GPU efficiency.
Authored by dataloader.
The four aux-using vbench metrics (color, multiple_objects,
object_class, spatial_relationship) used to bail at the top of
compute() if auxiliary_info was None, then unconditionally indexed
aux[b]["<key>"] for every row. This crashed with KeyError on rows
that had aux but lacked the metric's specific key — common once
the dataset yields heterogeneous rows under "vbench" / multi-dim
runs (each row carries flat aux populated only for its own
dimensions).
Switch to per-row skip: each row checks for its own required key
and emits MetricResult(score=None, details={"skipped": "..."}) if
absent. Lets a single evaluator.evaluate(samples=[...]) call score
heterogeneous-aux rows in one pass.
Also handle two structural sub-cases:
- multiple_objects requires "<a> and <b>" in aux["object"]; rows
without the separator (single-object data leaking in) skip.
- object_class is the inverse: rows whose "object" contains " and "
belong to multiple_objects' territory and skip here.
- spatial_relationship's nested {object_a/object_b/relationship}
sub-dict is read defensively; missing keys skip with a reason.
Pairs with the flat-aux-at-load behavior in VBenchPromptDataset
(commit 06735161). Without these per-row skips, dimensions="all"
runs crash on the first row that lacks the active metric's key.
Authored by dataloader.
Two small additions paired with the in-progress eval orchestration:
- types.py: EvalResult dataclass with summary/per_video fields plus
from_raw / save / print helpers. Used by scripts and any future
callers that aggregate per-sample MetricResults into a corpus-level
summary (e.g. scripts/eval/run_vbench_e2e.py).
- .gitignore: exclude fastvideo/eval/_assets/ which holds calibration
data, downloaded videos, and run dumps used by the synthetic-flow
evaluator that we don't want to track.
Authored by dataloader.
The flatten loop in VBenchPromptDataset strips exactly one level of
upstream's {dim_name: ...} wrapper. For most VBench dims this leaves
a flat scalar dict ({"color": "red"}, {"object": "person"}). For
spatial_relationship, upstream double-wraps, so one level of unwrap
leaves {"spatial_relationship": {object_a, object_b, relationship}} —
which is exactly what SpatialRelationshipMetric reads.
That last case looks accidental; it isn't. Documenting all four shapes
inline so a future reader doesn't "simplify" the wrapping away and
silently break spatial_relationship scoring.
Empirically verified output for all four aux dims matches the
documented shapes.
The Evaluator's calibrate() is gone, so batch_unit class attrs and
trial_forward() method overrides have no callers — pure cruft from the
old auto-calibration design. Mechanical removal across 15 metric files.
Internal time-dim chunking (the part of batching that actually does
work) lives on as metric-owned _chunk_size constants set in __init__,
unaffected by this change.
Five files (color/multiple_objects/object_class/spatial_relationship/
optical_flow metrics) skipped — they have dataloader's in-flight
modifications uncommitted; left for that branch's cleanup pass.
Parity verified on fs-mbz-gpu-085: results byte-identical to v3
baseline.
Match FastVideo's existing depth: VideoGenerator is the top-level
inference object with no Runner above it; loops live in scripts.
Eval should mirror that — Evaluator is the top-level scoring object,
EvalWorker × N is the layer below, and end-to-end pipelines (prompts
→ generate → score) are scripts, not classes.
Removed:
- fastvideo/eval/runner.py (EvalRunner + classmethod constructors).
- EvalRunner export from fastvideo.eval.
Added:
- fastvideo/eval/io/paths.py with sanitize_prompt, default_filename,
glob_videos, build_eval_kwargs as free functions. The reusable bits
of the runner survive; the class wrapping them does not.
Rewritten:
- scripts/eval/run_vbench_e2e.py is now top-to-bottom procedural:
parse_args → dataset → optional generate() loop → optional score()
loop. Reads in one pass; no classmethod indirection to chase.
Parity verified on fs-mbz-gpu-085 against runs/vbench_smoke/videos/
(2 videos × 3 metrics, dimensions=subject_consistency):
- 1-GPU run (results_v4.json) is byte-identical to v3 baseline.
- 2-GPU run (results_v4_2gpu.json) is byte-identical to v3 baseline
(order-preserving fan-out via Evaluator.evaluate(samples=[...])).
Continues the eval refactor. Datasets now yield plain dicts (matches
ValidationDataset / VideoGenerator / Evaluator's kwargs-flowing-through
pattern). End-to-end orchestration moves into EvalRunner so neither the
dataset nor the Evaluator owns videos_dir / fps / filename / generator
state.
Layering:
EvalRunner dataset / generator / file conventions / manifest
└── Evaluator thin dispatcher (single evaluate(), 1 or N samples)
└── EvalWorker × N single-GPU metric replicas
Dataset (fastvideo/eval/datasets/):
- PromptDataset is just an Iterable[dict]. _rows holds dicts with
prompt / n_samples / dimensions / auxiliary_info / ... — unused
fields stay absent rather than living as Optional[None] on a
dataclass. BasePromptDataset alias kept for back-compat.
- BenchmarkSample dataclass deleted. Sample TypedDict documents the
recognized keys without forcing the schema.
- VBench flattens its nested {dim: {key: val}} aux into {key: val} at
load time. Aux-using metrics already read flat keys; no metric edits.
Runner (fastvideo/eval/runner.py, new):
- Three named constructors: from_dataset, from_videos, from_samples.
- generate() drives the generator over the dataset (writes manifest);
score() loads videos and dispatches via Evaluator.evaluate(list).
run() = generate then score.
- Multi-GPU lives in the underlying Evaluator; the runner just hands
num_gpus through. No double-orchestration.
- File naming convention (filename_fn) and eval-kwargs assembly
(eval_kwargs_fn) are runner-side hooks, not dataset overrides.
Script (scripts/eval/run_vbench_e2e.py):
- Rewritten against EvalRunner; ~95 LOC, was 149.
Includes prior scaffolding from dataloader (datasets/registry.py,
the test file, the script) carried over so the refactor lands as one
coherent state.
Mirrors FastVideo's VideoGenerator → Worker layering, in-process:
Evaluator (user-facing dispatcher)
└── EvalWorker × N (single-GPU, owns metric replicas)
EvalWorker (new) holds metric replicas on one device and scores one
sample at a time. Evaluator builds num_gpus workers eagerly in __init__
(every metric loaded on every replica) and exposes a single evaluate()
method that handles both shapes:
ev.evaluate(video=..., ...) → one sample, runs on worker 0
ev.evaluate([s1, s2, ...]) → fan out across workers
Removed (dead code or moved to runner in a follow-up):
- Evaluator.calibrate, _compute_chunked, _evaluate_multi_gpu,
_gpu_metrics, evaluate_dataset, B>1 input path
- BaseMetric.batch_unit and trial_forward (kept _chunk_size as a plain
default class attr; metrics use it for internal time-dim chunking)
- memory.is_batch_too_large, slice_sample (clear_cache stays)
Per-metric batch_unit/trial_forward overrides remain as harmless
class-attr cruft pending a mechanical removal pass.
Net: evaluator.py 447 → 135 LOC; metrics/base.py 92 → 66; memory.py
40 → 14; +78 LOC for worker.py.
Adds an in-process evaluation suite covering native (SSIM/PSNR/LPIPS/
optical_flow), audio, physics_iq, vbench (16 sub-metrics), and
videoscore2. Public API: create_evaluator/evaluate, BaseMetric +
@register, ensure_checkpoint, get_cache_dir. CLI: fastvideo eval
list/run.
VBench upstream pinned as a git submodule at
fastvideo/third_party/eval/vbench (Vchitect/VBench@45e79ec); modern-dep
compat is achieved via runtime shims in vbench/__init__.py rather than
on-disk patches. CLIP/torch.hub caches are routed through
${FASTVIDEO_EVAL_CACHE}; HF cache stays at the system default.
ensure_checkpoint delegates to fastvideo.utils.get_lock + huggingface_hub
primitives. Evaluator supports release_cuda_memory(), unload(),
reload() for training-time eval that frees GPU between calls.
Verified parity against upstream vbench (5/8 metrics bit-exact, others
within 1% drift driven by transformers/torch version skew) and against
upstream VideoScore2 (regex anchored on the actual model's output
format, soft-score formula matches upstream's argmax*max_prob/total).
Includes docs/contributing/eval-metrics.md as the porting guide.
Out of scope (deferred follow-ups): MIND, VBench-2.0, FVD as a
registered metric, training-time EvalCallback.
shaoxiongduan