94 lines
3.0 KiB
Markdown
94 lines
3.0 KiB
Markdown
---
|
|
description: How to develop, validate, and register a new evaluation metric
|
|
---
|
|
|
|
# Evaluation Development SOP
|
|
|
|
Standard procedure for adding new video quality evaluation metrics to
|
|
the FastVideo agent toolkit.
|
|
|
|
## When to use
|
|
|
|
- You need a metric that does not exist in
|
|
`.agents/memory/evaluation-registry/README.md`.
|
|
- An existing metric needs significant changes to its methodology.
|
|
- You are exploring a new evaluation approach.
|
|
|
|
## Steps
|
|
|
|
### 1. Research
|
|
|
|
- Search `.agents/memory/related-work/` for existing evaluation
|
|
approaches.
|
|
- Check `.agents/memory/evaluation-registry/README.md` for current
|
|
metrics and their limitations.
|
|
- Review literature: FVD, CLIP-Score, human preference, etc.
|
|
|
|
### 2. Prototype
|
|
|
|
- Write a standalone script in `.agents/exploration/<metric-name>.md`.
|
|
- Keep it simple: one script, minimal dependencies.
|
|
- Test on a few known-good and known-bad video samples.
|
|
|
|
### 3. Validate
|
|
|
|
- **Known-good test**: metric should score high on reference-quality
|
|
videos.
|
|
- **Known-bad test**: metric should score low on degraded or unrelated
|
|
videos.
|
|
- **Sensitivity test**: small quality differences should produce
|
|
meaningful score differences.
|
|
- Document thresholds and their justification.
|
|
|
|
### 4. Register
|
|
|
|
Update `.agents/memory/evaluation-registry/README.md`:
|
|
|
|
- Add the metric with status `Active`.
|
|
- Document location, thresholds, and trust level.
|
|
|
|
### 5. Integrate
|
|
|
|
Update `.agents/skills/evaluate-video-quality/SKILL.md`:
|
|
|
|
- Add the new metric as a section.
|
|
- Include code examples and interpretation guide.
|
|
|
|
### 6. Document
|
|
|
|
- Move the exploration log content into the skill.
|
|
- Clean up the exploration file or mark it as `promoted`.
|
|
- If anything went wrong during development, create a lesson.
|
|
|
|
## Where the metrics live
|
|
|
|
The eval suite is `fastvideo/eval/`. New metrics register themselves
|
|
via `@register("<group>.<name>")` and are auto-discovered when
|
|
`fastvideo.eval.metrics` is imported.
|
|
|
|
- **Native metrics** (SSIM, PSNR, LPIPS, optical flow, VLM): add a
|
|
file under the appropriate group dir
|
|
(`fastvideo/eval/metrics/common/`, `optical_flow/`, `videoscore2/`,
|
|
`physics_iq/`).
|
|
- **Metrics that wrap upstream research code**: follow the vbench
|
|
pattern in `fastvideo/eval/metrics/vbench/`. The contract is:
|
|
- Upstream lives as a git submodule under
|
|
`fastvideo/third_party/eval/<bench>/`, pinned to a SHA in repo-root
|
|
`.gitmodules`.
|
|
- The metric package's `__init__.py` inserts the submodule path on
|
|
`sys.path` and installs runtime compat shims (attribute-level
|
|
monkey-patches) for any modern-dep drift. Do not modify upstream
|
|
files on disk, and do not ship a `setup.sh`.
|
|
- See `fastvideo/eval/README.md` for the worked vbench example.
|
|
- Full porting guide:
|
|
[`docs/contributing/eval-metrics.md`](../../docs/contributing/eval-metrics.md).
|
|
|
|
## Out of scope of the initial eval port
|
|
|
|
The following land in follow-up PRs:
|
|
|
|
- **MIND** metrics (depends on a separate `vipe` submodule).
|
|
- **VBench-2.0** sibling package.
|
|
- Native conversion of **FVD** under `fastvideo/eval/metrics/fvd/`.
|
|
- The training-time `EvalCallback`.
|