Two new entrypoints showcasing the post-refactor eval surface:
- examples/inference/eval/basic_ltx2_eval.py — generate one LTX2
video using the same parameters as basic_ltx2.py (same prompt,
model, 1088×1920×121), then score it with the prompt-aware VBench
subset. Builds Evaluator directly, calls evaluate(**kwargs) per
the single-sample API; no runner, no argparse.
- scripts/eval/score_folder.py — bulk-score every video in a folder
with prompt-free VBench metrics. Optional --prompts-json maps
filenames to prompts to enable prompt-aware metrics. Uses
Evaluator.evaluate(samples=[...]) for multi-GPU fan-out.
Tested on fs-mbz-gpu-085:
- score_folder.py: 3 duplicate mp4s × 4 metrics → 3 identical score
sets, summary aggregated, scores.json written. Clean run.
- basic_ltx2_eval.py: generation succeeded; scoring path verified
for 7 of 8 metrics on the LTX2 output (aesthetic_quality,
subject_consistency, background_consistency, imaging_quality,
temporal_flickering, dynamic_degree, overall_consistency).
vbench.motion_smoothness OOMs at 1088×1920 on a shared GPU because
VBench's AMT memory autoscale reads total_memory rather than
mem_get_info() free memory, so it underestimates the required
resolution scale-down when another process holds 45 GB. Documented
in the script docstring; not refactor-related (same behavior at
HEAD pre-refactor). Drop motion_smoothness from METRICS or run on
a dedicated GPU to score it.
Added a torch.cuda.empty_cache() between generator.shutdown() and
evaluator construction to free residual generation memory.
shaoxiongduan
5cc337fdb6