Two new entrypoints showcasing the post-refactor eval surface:
- examples/inference/eval/basic_ltx2_eval.py — generate one LTX2
video using the same parameters as basic_ltx2.py (same prompt,
model, 1088×1920×121), then score it with the prompt-aware VBench
subset. Builds Evaluator directly, calls evaluate(**kwargs) per
the single-sample API; no runner, no argparse.
- scripts/eval/score_folder.py — bulk-score every video in a folder
with prompt-free VBench metrics. Optional --prompts-json maps
filenames to prompts to enable prompt-aware metrics. Uses
Evaluator.evaluate(samples=[...]) for multi-GPU fan-out.
Tested on fs-mbz-gpu-085:
- score_folder.py: 3 duplicate mp4s × 4 metrics → 3 identical score
sets, summary aggregated, scores.json written. Clean run.
- basic_ltx2_eval.py: generation succeeded; scoring path verified
for 7 of 8 metrics on the LTX2 output (aesthetic_quality,
subject_consistency, background_consistency, imaging_quality,
temporal_flickering, dynamic_degree, overall_consistency).
vbench.motion_smoothness OOMs at 1088×1920 on a shared GPU because
VBench's AMT memory autoscale reads total_memory rather than
mem_get_info() free memory, so it underestimates the required
resolution scale-down when another process holds 45 GB. Documented
in the script docstring; not refactor-related (same behavior at
HEAD pre-refactor). Drop motion_smoothness from METRICS or run on
a dedicated GPU to score it.
Added a torch.cuda.empty_cache() between generator.shutdown() and
evaluator construction to free residual generation memory.
Match FastVideo's existing depth: VideoGenerator is the top-level
inference object with no Runner above it; loops live in scripts.
Eval should mirror that — Evaluator is the top-level scoring object,
EvalWorker × N is the layer below, and end-to-end pipelines (prompts
→ generate → score) are scripts, not classes.
Removed:
- fastvideo/eval/runner.py (EvalRunner + classmethod constructors).
- EvalRunner export from fastvideo.eval.
Added:
- fastvideo/eval/io/paths.py with sanitize_prompt, default_filename,
glob_videos, build_eval_kwargs as free functions. The reusable bits
of the runner survive; the class wrapping them does not.
Rewritten:
- scripts/eval/run_vbench_e2e.py is now top-to-bottom procedural:
parse_args → dataset → optional generate() loop → optional score()
loop. Reads in one pass; no classmethod indirection to chase.
Parity verified on fs-mbz-gpu-085 against runs/vbench_smoke/videos/
(2 videos × 3 metrics, dimensions=subject_consistency):
- 1-GPU run (results_v4.json) is byte-identical to v3 baseline.
- 2-GPU run (results_v4_2gpu.json) is byte-identical to v3 baseline
(order-preserving fan-out via Evaluator.evaluate(samples=[...])).
Continues the eval refactor. Datasets now yield plain dicts (matches
ValidationDataset / VideoGenerator / Evaluator's kwargs-flowing-through
pattern). End-to-end orchestration moves into EvalRunner so neither the
dataset nor the Evaluator owns videos_dir / fps / filename / generator
state.
Layering:
EvalRunner dataset / generator / file conventions / manifest
└── Evaluator thin dispatcher (single evaluate(), 1 or N samples)
└── EvalWorker × N single-GPU metric replicas
Dataset (fastvideo/eval/datasets/):
- PromptDataset is just an Iterable[dict]. _rows holds dicts with
prompt / n_samples / dimensions / auxiliary_info / ... — unused
fields stay absent rather than living as Optional[None] on a
dataclass. BasePromptDataset alias kept for back-compat.
- BenchmarkSample dataclass deleted. Sample TypedDict documents the
recognized keys without forcing the schema.
- VBench flattens its nested {dim: {key: val}} aux into {key: val} at
load time. Aux-using metrics already read flat keys; no metric edits.
Runner (fastvideo/eval/runner.py, new):
- Three named constructors: from_dataset, from_videos, from_samples.
- generate() drives the generator over the dataset (writes manifest);
score() loads videos and dispatches via Evaluator.evaluate(list).
run() = generate then score.
- Multi-GPU lives in the underlying Evaluator; the runner just hands
num_gpus through. No double-orchestration.
- File naming convention (filename_fn) and eval-kwargs assembly
(eval_kwargs_fn) are runner-side hooks, not dataset overrides.
Script (scripts/eval/run_vbench_e2e.py):
- Rewritten against EvalRunner; ~95 LOC, was 149.
Includes prior scaffolding from dataloader (datasets/registry.py,
the test file, the script) carried over so the refactor lands as one
coherent state.
shaoxiongduan