# Performance Benchmarks FastVideo's performance benchmark suite measures end-to-end inference latency, throughput, peak GPU memory, and component-level pipeline timings for representative pipeline configurations. It tracks those metrics over time against a rolling baseline stored on the Hugging Face Hub. It serves three audiences: * **CI** — gates pull requests against a per-GPU static threshold and a rolling-median regression check. * **Maintainers** — surfaces regressions in a Markdown summary on every performance build and a long-form Plotly dashboard. * **Local developers** — lets you run the same benchmark on your own machine, then compare against the historical baseline for the same model and GPU. ## Quick start (local) ```bash # Run all benchmarks; writes raw perf_*.json under # fastvideo/tests/performance/results/ pytest fastvideo/tests/performance/ -vs # Optional: compare against the rolling HF baseline. # PERF_REPORTS_DIR defaults to /root/data/perf_reports for Modal/CI, so # override it when running outside the container. PERF_REPORTS_DIR=/tmp/fastvideo_perf_reports \ python fastvideo/tests/performance/compare_baseline.py # Optional: explicitly upload a passing local/manual run. HF_TOKEN=hf_... \ PERF_RUN_SOURCE=local \ PERF_UPLOAD_POLICY=pass \ PERF_REPORTS_DIR=/tmp/fastvideo_perf_reports \ python fastvideo/tests/performance/compare_baseline.py # Optional: build the Plotly dashboard locally. PERF_REPORTS_DIR=/tmp/fastvideo_perf_reports \ python fastvideo/tests/performance/dashboard.py ``` The pytest run never uploads anything. `compare_baseline.py` uploads only when `PERF_UPLOAD_POLICY` is set. Local uploads are explicit opt-in and require HF credentials. PR/direct performance runs upload passing records for dashboard visibility, while scheduled-main runs upload both pass and fail records. The report directory default is container-oriented; set `PERF_REPORTS_DIR` to a writable local path when generating dashboards or when you want local Markdown/normalized-result artifacts from the comparator. `compare_baseline.py` reads every `perf_*.json` currently present in `fastvideo/tests/performance/results/`; remove stale result files if you only want to compare the latest local run. ## Local live dashboard For an app-style local dashboard backed by the same HF performance-tracking records, see `performance_dashboard/README.md`. The dashboard provides a FastAPI API plus a React UI and can be exposed with `ngrok` after building the frontend. ## Architecture ``` .buildkite/performance-benchmarks/tests/*.json └── per-benchmark configs: model, gen kwargs, per-GPU thresholds fastvideo/tests/performance/ ├── test_inference_performance.py │ └── pytest test that runs each config, writes perf_*.json │ with latency, memory, throughput, and component timings ├── compare_baseline.py │ └── normalizes raw results, compares against HF rolling baseline, │ writes Markdown summary + (optionally) uploads new records ├── dashboard.py │ └── builds time-series Plotly HTML from HF history fastvideo/performance/ ├── hf_store.py # shared HF I/O + DataFrame helpers └── metric_policy.py # shared rolling-baseline threshold policy ``` The HF dataset (`FastVideo/performance-tracking` by default) holds one normalized JSON per `(model_id, gpu_type, run)` tuple. The rolling baseline is the median of the last 5 successful, baseline-eligible records for that model+GPU. PR and local records are visible in the dashboard but are not baseline eligible. ## Planned Coverage The current rollout tracks a small set of representative inference workloads. Broader coverage is planned for additional models, GPU types, attention backends, workload shapes, and inference recipes. As that coverage lands, the performance tracking system will also add environment-specific considerations so comparisons remain meaningful across hardware, runtime, attention backend, and recipe changes instead of treating all records for a model as equivalent. ## Metrics Each benchmark records six metrics. The rolling-baseline comparator also has a per-metric policy with direction, percent threshold, absolute threshold, and a `gated` flag. | Metric | Raw key | Normalized key | Direction | Default rolling policy | |---|---|---|---|---| | End-to-end generation latency | `avg_generation_time_s` | `latency` | Lower is better | 8% and 0.5 s | | Video throughput | `throughput_fps` | `throughput` | Higher is better | 8% and 0.05 FPS | | Peak GPU memory | `max_peak_memory_mb` | `memory` | Lower is better | 5% and 256 MB | | Text encoder time | `text_encoder_time_s` | `text_encoder_time_s` | Lower is better | 5% and 0.25 s | | DiT denoising time | `dit_time_s` | `dit_time_s` | Lower is better | 5% and 0.25 s | | VAE decode time | `vae_decode_time_s` | `vae_decode_time_s` | Lower is better | 5% and 0.25 s | `test_inference_performance.py` temporarily sets `FASTVIDEO_STAGE_LOGGING=1` while it runs so pipeline stage execution times are available in `generate_video(...).logging_info`. Stage logs use pipeline-unique keys such as `prompt_encoding_stage` so duplicate stage classes do not collide. For `PipelineStage` entries, shared component stage bases emit a stable `component_metric`: text encoding stages map to `text_encoder_time_s`, denoising stages and subclasses map to `dit_time_s`, and decoding stages map to `vae_decode_time_s`. The extractor falls back to known `stage_class` names for older logs that do not include `component_metric` or that used the class name as the stage key. Generator-side timings such as `PostDecodeFrameProcessStage`, `VideoSaveStage`, and `AudioMuxStage` are intentionally ignored. If a pipeline does not report one of the mapped stages, that component metric is stored as `null` and is skipped by the static threshold and rolling baseline checks. ## The two gates There are **two independent regression gates** — they protect against different failure modes and are not redundant. ### Static thresholds (per-GPU) Defined in `.buildkite/performance-benchmarks/tests/.json` under `thresholds`. Example: ```json "thresholds": { "L40S": { "max_generation_time_s": 34.0, "max_peak_memory_mb": 11000.0, "max_text_encoder_time_s": 5.0, "max_dit_time_s": 10.0, "max_vae_decode_time_s": 10.0 }, "default": { "max_generation_time_s": 120.0, "max_peak_memory_mb": 30000.0 } } ``` Selection: `_get_thresholds(cfg)` matches the current GPU name (substring match) against keys; falls back to `default` if no GPU matches. `max_generation_time_s` and `max_peak_memory_mb` are required for every selected threshold block. Component limits are optional: if `max_text_encoder_time_s`, `max_dit_time_s`, or `max_vae_decode_time_s` is absent, the pytest static-threshold gate skips that component. These are **fail-safes** — they catch order-of-magnitude regressions, unrealistic memory growth, and optionally large component-specific slowdowns even when the rolling baseline is empty. They are hand-set with generous headroom and almost never need touching. ### Rolling baseline (per `(model_id, gpu_type)`) `compare_baseline.py` loads the last 5 successful, baseline-eligible records for the same `(model_id, gpu_type)` from the HF dataset, computes the median for each available metric, and evaluates the current run with the metric's rolling regression policy. For latency, memory, and component times, higher values are regressions. For throughput, lower values are regressions. A metric exceeds its rolling threshold when both of these are true: ```text percent_delta > threshold_percent absolute_delta > threshold_absolute ``` Gated metrics fail CI when that threshold crossing happens. Set `gated: false` for metrics that should remain visible in reports and the dashboard without failing CI. Dashboard/API payloads expose `threshold_exceeded` separately from `regressed`, where `regressed` means a gated CI failure. Missing or `null` metrics are skipped. This is the **drift detector** — it catches sub-threshold regressions that slowly add up. Only scheduled-main successful records are baseline eligible. Local and pull-request runs can upload dashboard-visible records, but they do not update future gating baselines. When the baseline shifts for a legitimate reason (torch upgrade, kernel change, etc.) and CI starts failing, use the [`reseed-performance-baseline`](https://github.com/hao-ai-lab/FastVideo/blob/main/.agents/skills/reseed-performance-baseline/SKILL.md) agent skill to advance the rolling median. ## Schemas ### Benchmark config (`.buildkite/performance-benchmarks/tests/*.json`) Benchmark configs without `config_schema_version` are treated as legacy v1 configs and remain loadable. New or migrated configs should use `config_schema_version: 2` and include explicit comparable identity fields: ```jsonc { "benchmark_id": "wan-t2v-1.3b-2gpu", "config_schema_version": 2, "workload_id": "wan-t2v-1.3b", "variant_id": "canonical", "benchmark_version": 1 } ``` `benchmark_id` is still required in this phase because raw artifact names, generated-video directories, normalized record paths, and the current rolling baseline comparator still depend on it. The v2 identity fields are config metadata that make the measured workload explicit: | Field | Purpose | |---|---| | `workload_id` | Stable benchmark family, such as `wan-t2v-1.3b`. | | `variant_id` | Intentional recipe family, such as `canonical`. | | `benchmark_version` | Version of the measurement protocol and comparison policy. | If a config declares `config_schema_version: 2`, loading fails clearly when any required v2 identity field is missing. If v2 identity or metadata fields are added without `config_schema_version: 2`, loading also fails so partial migrations do not silently run as v1 configs. Optional v2 metadata fields reserved for follow-up work, such as `recipe`, `metric_threshold_policy`, and `quality_metadata`, must be JSON objects when present. Recipe fingerprinting, hardware/software profile IDs, exact-identity comparison, metric-specific threshold policy behavior, promoted baselines, and dashboard regrouping are separate follow-up changes. Until those land, rolling baseline comparison remains keyed by `(model_id, gpu_type)`. ### Raw record (`results/perf_*.json`) Written by `test_inference_performance.py`. One file per benchmark run. ```jsonc { "benchmark_id": "wan-t2v-1.3b-2gpu", "config_schema_version": 2, "workload_id": "wan-t2v-1.3b", "variant_id": "canonical", "benchmark_version": 1, "model_short_name": "Wan2.1-T2V-1.3B-Diffusers", "device": "NVIDIA L40S", "num_gpus": 2, "num_warmup_runs": 1, "num_measurement_runs": 3, "avg_generation_time_s": 28.4, "individual_times_s": [28.5, 28.3, 28.4], "throughput_fps": 1.58, "max_peak_memory_mb": 10840.0, "individual_peak_memories_mb": [10840.0, 10822.0, 10833.0], "thresholds": { "max_generation_time_s": 34.0, "max_peak_memory_mb": 11000.0, "max_text_encoder_time_s": 5.0, "max_dit_time_s": 10.0, "max_vae_decode_time_s": 10.0 }, "regression_thresholds": { "latency": { "threshold_percent": 0.10, "threshold_absolute": 1.0, "gated": true } }, "commit": "", "pr_number": "1234", "timestamp": "2026-05-08T22:00:00+00:00", "text_encoder_time_s": 2.141, "dit_time_s": 8.437, "vae_decode_time_s": 3.208 } ``` ### Normalized record (HF dataset, also dumped as `normalized_perf_*.json`) Written by `compare_baseline.py:_normalize_record`. One file per benchmark result, used as the rolling-baseline source of truth. ```jsonc { "model_id": "wan-t2v-1.3b-2gpu", "timestamp": "2026-05-08T22:00:00+00:00", "commit_sha": "", "gpu_type": "NVIDIA L40S", "latency": 28.4, "throughput": 1.58, "memory": 10840.0, "text_encoder_time_s": 2.141, "dit_time_s": 8.437, "vae_decode_time_s": 3.208, "regression_thresholds": { "latency": { "threshold_percent": 0.08, "threshold_absolute": 0.5, "gated": true } }, "success": true } ``` ### Compatibility with legacy records Older records in the HF dataset may not have component timing fields. The comparator ignores missing or `null` metrics when computing a median, and the dashboard lists skipped plots for metric series that have no non-null values. Records missing both `run_source` and `baseline_eligible` are treated as legacy successful main/full-suite uploads and remain eligible for rolling baselines. ## Environment variable reference | Variable | Default | Used by | Purpose | |---|---|---|---| | `PERFORMANCE_TRACKING_ROOT` | `/tmp/perf-tracking` | `compare_baseline.py`, `dashboard.py` | Local directory the HF dataset is synced to. | | `PERF_REPORTS_DIR` | `/root/data/perf_reports` | `compare_baseline.py`, `dashboard.py` | Where the Markdown summary and Plotly HTML get written for Buildkite to pick up. | | `HF_REPO_ID` | `FastVideo/performance-tracking` | `fastvideo/performance/hf_store.py` | HF dataset repo holding rolling-baseline records. | | `HF_API_KEY`, `HUGGINGFACE_HUB_TOKEN`, `HF_TOKEN` | unset | `fastvideo/performance/hf_store.py` | Required for upload or private dataset reads. | | `PERF_RUN_SOURCE` | inferred | `compare_baseline.py` | Source metadata for uploaded records: `pr`, `local`, `scheduled_main`, or `unknown`. | | `PERF_UPLOAD_POLICY` | `never` | `compare_baseline.py` | Upload policy: `never`, `pass`, or `always`. | | `PERF_PYTEST_RC` | unset | `compare_baseline.py` | Static-threshold pytest exit code, used so scheduled-main failures can be uploaded with `success=false`. | | `TEST_SCOPE` | unset | `compare_baseline.py` | CI context used to infer scheduled-main runs together with `BUILDKITE_BRANCH=main`. | | `BUILDKITE_BRANCH`, `BUILDKITE_COMMIT`, `BUILDKITE_PULL_REQUEST` | unset | `compare_baseline.py`, `test_inference_performance.py` | CI metadata stamped into records. | | `DASHBOARD_DAYS` | `30` | `dashboard.py` | Lookback window for the Plotly trend pages. | | `PERFORMANCE_TRACKING_SYNC_REUSE_TTL_SECONDS` | `3600` | `fastvideo/performance/hf_store.py` | Freshness window for reusing an existing HF sync when requested by dashboard consumers. | | `FASTVIDEO_STAGE_LOGGING` | set by the pytest test | `test_inference_performance.py` | Enables pipeline stage timing capture for component metrics during benchmark runs. | ## CI integration The performance step can run on demand with `/test performance` and as part of the Full Suite (see [CI/CD Architecture](ci_architecture.md)). The Modal entry point is `fastvideo/tests/modal/pr_test.py:run_performance_tests` and the Buildkite artifact upload is in `.buildkite/scripts/pr_test.sh:upload_performance_artifacts`. Each performance build runs pytest first. If that fixed-threshold phase fails, PR/direct runs skip `compare_baseline.py` because they only upload passing records. Scheduled-main runs still execute `compare_baseline.py` with `PERF_PYTEST_RC` set so the failed canonical attempt is visible in normalized JSON and dashboard history. The dashboard runs best-effort for observability. When the rolling-baseline phase runs, it emits: * **Markdown summary** — appended to `$GITHUB_STEP_SUMMARY` when that variable is set, and written as `perf__.md` for Buildkite upload. Contains a per-benchmark row with current vs. baseline values for latency, throughput, memory, text encoder time, DiT time, and VAE decode time. * **Plotly dashboard** — `dashboard__.html` showing time-series for each metric grouped by `(model_id, gpu_type)`. * **Normalized records** — `normalized_perf_*.json`, one per benchmark. Useful as input to the [`reseed-performance-baseline`](https://github.com/hao-ai-lab/FastVideo/blob/main/.agents/skills/reseed-performance-baseline/SKILL.md) skill. ## Adding a new benchmark 1. Drop a new JSON config into `.buildkite/performance-benchmarks/tests/.json`. New configs should use v2 identity fields: ```json { "benchmark_id": "", "config_schema_version": 2, "workload_id": "", "variant_id": "canonical", "benchmark_version": 1, "model": { "model_path": "...", "model_short_name": "..." }, "init_kwargs": { "num_gpus": 1, ... }, "generation_kwargs": { "num_frames": 45, ... }, "test_prompts": ["..."], "run_config": { "required_gpus": 1, "num_warmup_runs": 1, "num_measurement_runs": 3 }, "thresholds": { "L40S": { "max_generation_time_s": 34.0, "max_peak_memory_mb": 11000.0, "max_text_encoder_time_s": 5.0, "max_dit_time_s": 10.0, "max_vae_decode_time_s": 10.0 }, "default": { "max_generation_time_s": 120.0, "max_peak_memory_mb": 30000.0 } }, "regression_thresholds": { "latency": { "threshold_percent": 0.10, "threshold_absolute": 1.0, "gated": true } } } ``` Legacy v1 configs without `config_schema_version` still load, but should not gain v2 identity or metadata fields until they are migrated to `config_schema_version: 2`. 2. The pytest test auto-discovers all configs — no test code needed. CI picks it up on the next `/test performance` run. 3. The first persisted main-branch run with no HF history initializes the baseline (passes automatically). Subsequent runs compare against it. Local and pull-request runs with no HF history also pass, but they do not seed the shared baseline. 4. If the benchmark targets a GPU not currently in `thresholds`, either add that GPU as a key or rely on the `default` block. Note that `default` is intended for slower fallback GPUs, so its values should be relaxed relative to the fastest entry. 5. Add component thresholds only when the stage timing is stable enough to be a useful fixed gate. The rolling baseline will still track component times when static component thresholds are omitted. 6. Omit `regression_thresholds` to use the default rolling-baseline policy, or include only benchmark-specific deviations. Tune these independently from the fixed thresholds when a metric is noisy or should be informational. The fixed `thresholds` block is an absolute pytest ceiling. The `regression_thresholds` block controls rolling-baseline comparisons against recent scheduled-main records. ## Troubleshooting **"No baseline for ... Initializing"** — first run for this `(model_id, gpu_type)`. Run will pass and (if persisting) seed the first record. **Persistent failure right after a torch / kernel / image upgrade** — genuine regression *or* baseline drift. Compare the failing normalized record with recent successful records in the HF dataset. If the shift is expected and reviewed, use the `reseed-performance-baseline` skill. **Dashboard reports skipped metric plots** — the loaded records do not have non-null values for that metric. This is expected for older records or for pipelines that did not report a mapped component stage. **Component timing is `null`** — the generated result did not include a mapped stage in `logging_info.stages`. Check that the pipeline emits stage logging and that the stage emits `component_metric` or is covered by the legacy `STAGE_METRIC_MAP` fallback in `test_inference_performance.py`.