hao-ai-lab-FastVideo/docs/contributing/performance_benchmarks.md

Performance Benchmarks

FastVideo's performance benchmark suite measures end-to-end inference latency, throughput, peak GPU memory, and component-level pipeline timings for representative pipeline configurations. It tracks those metrics over time against a rolling baseline stored on the Hugging Face Hub.

It serves three audiences:

• CI — gates pull requests against a per-GPU static threshold and a rolling-median regression check.
• Maintainers — surfaces regressions in a Markdown summary on every performance build and a long-form Plotly dashboard.
• Local developers — lets you run the same benchmark on your own machine, then compare against the historical baseline for the same comparable identity.

Quick start (local)

# Run all measured benchmarks; writes raw perf_*.json under
# fastvideo/tests/performance/results/
pytest fastvideo/tests/performance/test_inference_performance.py -vs

pytest tests/local_tests/performance/ -vs

# Optional: compare against the rolling HF baseline.
# PERF_REPORTS_DIR defaults to /root/data/perf_reports in a CI container, so
# override it for a local run.
PERF_REPORTS_DIR=/tmp/fastvideo_perf_reports \
python fastvideo/tests/performance/compare_baseline.py

# Optional: explicitly upload a passing local/manual run.
HF_TOKEN=hf_... \
PERF_RUN_SOURCE=local \
PERF_UPLOAD_POLICY=pass \
PERF_REPORTS_DIR=/tmp/fastvideo_perf_reports \
python fastvideo/tests/performance/compare_baseline.py

# Optional: build the Plotly dashboard locally.
PERF_REPORTS_DIR=/tmp/fastvideo_perf_reports \
python fastvideo/tests/performance/dashboard.py

The pytest run never uploads anything. compare_baseline.py uploads only when PERF_UPLOAD_POLICY is set. Local uploads are explicit opt-in and require HF credentials. PR performance runs read baselines strictly and retain comparison reports as Buildkite artifacts without uploading records. Scheduled-main runs upload both pass and fail records using an operator-provisioned CI credential. The report directory default is container-oriented; set PERF_REPORTS_DIR to a writable local path when generating dashboards or when you want local Markdown/normalized-result artifacts from the comparator. compare_baseline.py reads every perf_*.json currently present in fastvideo/tests/performance/results/; remove stale result files if you only want to compare the latest local run.

Local live dashboard

For an app-style local dashboard backed by the same HF performance-tracking records, see performance_dashboard/README.md. The dashboard provides a FastAPI API plus a React UI and can be exposed with ngrok after building the frontend.

DGX Spark (GB10) local benchmarking

The NVIDIA DGX Spark (GB10) is not available on Modal, so its coverage is local/manual rather than automated CI. The GB10 benchmark wan-t2v-1.3b-1gpu-gb10 is gated to the GB10 via run_config.gpu_types (matched as substrings of the CUDA device name), so the shared H100/L40S performance lanes discover it and skip it, while a GB10 owner runs it locally.

Run just the GB10 benchmark on a DGX Spark:

pytest 'fastvideo/tests/performance/test_inference_performance.py::test_inference_performance[wan-t2v-1.3b-1gpu-gb10]' -vs

To check run-to-run stability (latency, peak memory, throughput), run it a few times from a clean results directory, then normalize:

rm -f fastvideo/tests/performance/results/perf_*.json
for i in 1 2 3 4 5; do
  pytest 'fastvideo/tests/performance/test_inference_performance.py::test_inference_performance[wan-t2v-1.3b-1gpu-gb10]' -vs
done
PERF_REPORTS_DIR=/tmp/fastvideo_perf_reports \
  python fastvideo/tests/performance/compare_baseline.py

compare_baseline.py reports CALIBRATION_NEEDED until a baseline exists for the GB10 identity, and writes one normalized_perf_*.json per run. Reference figures on a GB10 (torch 2.12.0+cu130, transformers 5.14.0): generation ~39.3 s, peak ~8.4 GB, throughput ~1.15 fps, stable to ~0.4% across five runs.

Seeding the GB10 baseline (follow-up)

Seeding a baseline-eligible record for the GB10 identity is intentionally not done from a local run: seed_baseline.py accepts only scheduled_main full-suite source artifacts, so ordinary local/manual uploads stay baseline_eligible=false (dashboard-visible, but they do not move the rolling baseline). Establishing the GB10 baseline requires either:

• a scheduled-main performance run on a GB10 CI runner once one is available, or
• a carefully scoped, maintainer-approved manual calibration path that preserves the existing exact-identity, batch-consistency, provenance, and explicit-approval safeguards — it must not make arbitrary local uploads baseline-eligible.

The reviewed five-run GB10 artifacts are the stability evidence for that first baseline. Tracked in #1632.

Architecture

.buildkite/performance-benchmarks/tests/*.json
    └── per-benchmark configs: model, gen kwargs, per-GPU thresholds

fastvideo/tests/performance/
    ├── test_inference_performance.py
    │       └── pytest test that runs each config, writes perf_*.json
    │           with latency, memory, throughput, and component timings
    ├── compare_baseline.py
    │       └── normalizes raw results, compares against HF rolling baseline,
    │           writes Markdown summary + (optionally) uploads new records
    ├── dashboard.py
    │       └── builds time-series Plotly HTML from HF history

tests/local_tests/performance/
    └── local-only behavior tests for config, policy, identity, dashboards,
        and worker-log capture

fastvideo/performance/
    ├── hf_store.py               # shared HF I/O + DataFrame helpers
    └── metric_policy.py          # shared rolling-baseline threshold policy

The HF dataset (FastVideo/performance-tracking by default) holds one normalized JSON per run. For v2 records, the rolling baseline is the median of the last 5 successful, baseline-eligible records in the same comparison cohort: workload_id, variant_id, benchmark_version, recipe_fingerprint, hardware_profile_id, and software_profile_id. PR and local records are visible in the dashboard but are not baseline eligible.

Planned Coverage

The current rollout tracks a small set of representative inference workloads. Broader coverage is planned for additional models, GPU types, attention backends, workload shapes, and inference recipes. As that coverage lands, the performance tracking system will also add environment-specific considerations so comparisons remain meaningful across hardware, runtime, attention backend, and recipe changes instead of treating all records for a model as equivalent.

Metrics

Each benchmark records six metrics. The rolling-baseline comparator also has a per-metric policy with direction, percent threshold, absolute threshold, and a gated flag.

Metric	Raw key	Normalized key	Direction	Default rolling policy
End-to-end generation latency	avg_generation_time_s	latency	Lower is better	8% and 0.5 s
Video throughput	throughput_fps	throughput	Higher is better	8% and 0.05 FPS
Peak GPU memory	max_peak_memory_mb	memory	Lower is better	5% and 256 MB
Text encoder time	text_encoder_time_s	text_encoder_time_s	Lower is better	5% and 0.25 s
DiT denoising time	dit_time_s	dit_time_s	Lower is better	5% and 0.25 s
VAE decode time	vae_decode_time_s	vae_decode_time_s	Lower is better	5% and 0.25 s

test_inference_performance.py temporarily sets FASTVIDEO_STAGE_LOGGING=1 while it runs so pipeline stage execution times are available in generate_video(...).logging_info. Stage logs use pipeline-unique keys such as prompt_encoding_stage so duplicate stage classes do not collide. For PipelineStage entries, shared component stage bases emit a stable component_metric: text encoding stages map to text_encoder_time_s, denoising stages and subclasses map to dit_time_s, and decoding stages map to vae_decode_time_s. The extractor falls back to known stage_class names for older logs that do not include component_metric or that used the class name as the stage key. Generator-side timings such as PostDecodeFrameProcessStage, VideoSaveStage, and AudioMuxStage are intentionally ignored. If a pipeline does not report one of the mapped stages, that component metric is stored as null and is skipped by the static threshold and rolling baseline checks.

The two gates

There are two independent regression gates — they protect against different failure modes and are not redundant.

A config can additionally restrict which hardware it runs on via run_config.gpu_types (substring match against the CUDA device name); a non-matching device skips the benchmark entirely. That is a run gate, not a regression gate.

Static thresholds (per-GPU)

Defined in .buildkite/performance-benchmarks/tests/<benchmark>.json under thresholds. Example:

"thresholds": {
  "L40S": {
    "max_generation_time_s": 34.0,
    "max_peak_memory_mb": 11000.0,
    "max_text_encoder_time_s": 5.0,
    "max_dit_time_s": 10.0,
    "max_vae_decode_time_s": 10.0
  },
  "default": { "max_generation_time_s": 120.0, "max_peak_memory_mb": 30000.0 }
}

Selection: _get_thresholds(cfg) matches the current GPU name (substring match) against keys; falls back to default if no GPU matches.

max_generation_time_s and max_peak_memory_mb are required for every selected threshold block. Component limits are optional: if max_text_encoder_time_s, max_dit_time_s, or max_vae_decode_time_s is absent, the pytest static-threshold gate skips that component.

These are fail-safes — they catch order-of-magnitude regressions, unrealistic memory growth, and optionally large component-specific slowdowns even when the rolling baseline is empty. They are hand-set with generous headroom and almost never need touching.

Rolling baseline (per comparison cohort)

compare_baseline.py loads the last 5 successful, baseline-eligible records for the same comparison cohort from the HF dataset, computes the median for each available metric, and evaluates the current run with the metric's rolling regression policy. For v2 records, that cohort is workload_id, variant_id, benchmark_version, recipe_fingerprint, hardware_profile_id, and software_profile_id. For latency, memory, and component times, higher values are regressions. For throughput, lower values are regressions.

A metric exceeds its rolling threshold when both of these are true:

percent_delta > threshold_percent
absolute_delta > threshold_absolute

Gated metrics fail CI when that threshold crossing happens. Set gated: false for metrics that should remain visible in reports and the dashboard without failing CI. Dashboard/API payloads expose threshold_exceeded separately from regressed, where regressed means a gated CI failure. Missing or null metrics are skipped.

This is the drift detector — it catches sub-threshold regressions that slowly add up. Only scheduled-main successful records are baseline eligible. Local uploads are explicit opt-in and do not update future gating baselines. PR runs retain their normalized records and reports as Buildkite artifacts.

Comparator summaries and normalized artifacts include an explicit comparison_status:

Status	Meaning	CI behavior
PASS	Comparable baseline exists and no gated metric regressed. Legacy records with no baseline also keep the historical initialization behavior.	Passes
REGRESSION	The record exceeds one of its static thresholds or at least one gated metric regressed against a comparable baseline.	Fails
CALIBRATION_NEEDED	A v2 record has no exact comparable baseline.	Passes, may upload when PERF_UPLOAD_POLICY=pass, never seeds a baseline
RECIPE_MISMATCH	The same workload, variant, and benchmark version has trusted successful records under another recipe fingerprint, including records from other hardware or software profiles.	Fails
INFRA_ERROR	The comparator cannot safely classify the record, such as a v2 record missing required identity fields.	Fails

QUALITY_BLOCKED is reserved for a future variant-promotion workflow and is not emitted by normal rolling-baseline comparison.

For RECIPE_MISMATCH, trusted records are scheduled-main records or records already marked baseline_eligible. PR and local calibration uploads remain visible but do not authorize future CI-gating recipe mismatch failures.

When the baseline shifts for a legitimate reason (torch upgrade, kernel change, etc.) and CI starts failing, use the reseed-performance-baseline agent skill to advance the rolling median. To approve the first baseline for a new v2 exact identity, follow that skill with a reviewed scheduled-main full-suite CALIBRATION_NEEDED normalized artifact. Its prepare step uses fastvideo/tests/performance/seed_baseline.py; after a separate human gate, the skill rechecks the current HF revision and conditionally uploads the whole seed batch in one commit.

Schemas

Benchmark config (.buildkite/performance-benchmarks/tests/*.json)

Benchmark configs without config_schema_version are treated as legacy v1 configs and remain loadable. New or migrated configs should use config_schema_version: 2 and include explicit comparable identity fields:

{
  "benchmark_id": "wan-t2v-1.3b-2gpu",
  "config_schema_version": 2,
  "workload_id": "wan-t2v",
  "variant_id": "1.3b-sp2",
  "benchmark_version": 3
}

benchmark_id is still required because raw artifact names, generated-video directories, normalized record paths, and legacy storage directories depend on it. The v2 comparator does not use it as part of the comparison cohort. The v2 identity fields make the measured workload explicit:

Field	Purpose
workload_id	Stable benchmark family, such as wan-t2v.
variant_id	Intentional recipe family, including model size and parallelism config, such as 1.3b-sp2.
benchmark_version	Version of the measurement protocol and comparison policy.

If a config declares config_schema_version: 2, loading fails clearly when any required v2 identity field is missing. If v2 identity or metadata fields are added without config_schema_version: 2, loading also fails so partial migrations do not silently run as v1 configs. Optional v2 quality_metadata and the v1/v2 regression_thresholds policy must be JSON objects when present. (recipe is emitted by the harness and is not config-declarable.)

run_config.gpu_types is an optional list of non-empty strings. Each entry is substring-matched against the CUDA device name; if no entry matches, the benchmark is skipped on that device, so a hardware-specific config does not run on shared lanes. Discovery fails when an entry has no matching thresholds key or when thresholds has no default block, which prevents a gated config from silently falling back to default.

V2 records compare only within their exact identity cohort. A record that opens a new cohort is marked baseline_status: "initialized_new_cohort" and comparison_status: "CALIBRATION_NEEDED"; it remains ineligible until a reviewed scheduled-main artifact is seeded explicitly. Legacy v1 configs still run and are normalized for reporting, but their records skip rolling-baseline comparison entirely (baseline_status: "skipped_missing_identity", never baseline eligible); only static thresholds gate them. Metric-specific threshold policies are active. QUALITY_BLOCKED remains reserved for future variant promotion policy.

The shipped Wan benchmark uses benchmark_version: 3 because recipe schema 2 changed the recipe fingerprint by removing the legacy benchmark_id display name. This intentionally opens a new comparison cohort: after deployment, a reviewed scheduled-main full-suite CALIBRATION_NEEDED artifact must be seeded once before rolling regression gating resumes for that exact identity. Static thresholds remain active during calibration.

Raw record (results/perf_*.json)

Written by test_inference_performance.py. One file per benchmark run.

{
  "benchmark_id": "wan-t2v-1.3b-2gpu",
  "result_schema_version": 2,
  "workload_id": "wan-t2v",
  "variant_id": "1.3b-sp2",
  "benchmark_version": 3,
  "model_short_name": "Wan2.1-T2V-1.3B-Diffusers",
  "device": "NVIDIA L40S",
  "num_gpus": 2,
  "num_warmup_runs": 1,
  "num_measurement_runs": 3,
  "avg_generation_time_s": 28.4,
  "individual_times_s": [28.5, 28.3, 28.4],
  "throughput_fps": 1.58,
  "max_peak_memory_mb": 10840.0,
  "individual_peak_memories_mb": [10840.0, 10822.0, 10833.0],
  "thresholds": {
    "max_generation_time_s": 34.0,
    "max_peak_memory_mb": 11000.0,
    "max_text_encoder_time_s": 5.0,
    "max_dit_time_s": 10.0,
    "max_vae_decode_time_s": 10.0
  },
  "worker_log_path": "<container-local path to the captured worker log, or null>",
  "regression_thresholds": {
    "latency": {
      "threshold_percent": 0.10,
      "threshold_absolute": 1.0,
      "gated": true
    }
  },
  "commit": "<full sha>",
  "run_source": "pr",
  "branch": "feature/perf-change",
  "pr_number": "1234",
  "test_scope": "direct",
  "build_url": "https://buildkite.example/build",
  "build_id": "<buildkite-build-id>",
  "job_id": "<buildkite-job-id>",
  "timestamp": "2026-05-08T22:00:00+00:00",
  "quality_metadata": { "quality_status": "canonical" },
  "text_encoder_time_s": 2.141,
  "dit_time_s": 8.437,
  "vae_decode_time_s": 3.208,
  "recipe": {
    "recipe_schema_version": 2,
    "benchmark": {
      "benchmark_id": "wan-t2v-1.3b-2gpu",
      "workload_id": "wan-t2v",
      "variant_id": "1.3b-sp2",
      "benchmark_version": 3
    },
    "model": { "model_path": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers" },
    "init_kwargs": { "num_gpus": 2, "sp_size": 2, "tp_size": 1 },
    "generation_kwargs": { "height": 480, "width": 832, "num_frames": 45 },
    "inputs": { "prompt_count": 1, "prompt_sha256": ["<measured-prompt-sha256>"] },
    "attention": { "requested_backend": "FLASH_ATTN", "resolved_backend": "FLASH_ATTN" }
  },
  "recipe_fingerprint": "<sha256>",
  "hardware_profile": {
    "device_type": "cuda",
    "gpu_count": 2,
    "gpus": [{ "name": "NVIDIA L40S", "memory_gb": 48, "compute_capability": "8.9" }],
    "interconnect": "none_or_partial"
  },
  "hardware_profile_id": "hw-<sha256-prefix>",
  "software_profile": {
    "python": "3.12",
    "pytorch": "2.12",
    "cuda": "13.0",
    "attention_backend": "FLASH_ATTN",
    "flash_attention_4_enabled": true,
    "container_image_version": "py3.12-cuda13.0.0",
    "packages": {
      "fastvideo_kernel": "0.3.2",
      "flashinfer": "0.2.11",
      "nvidia_cutlass_dsl": "4.5.0",
      "triton": "3.4.1"
    }
  },
  "software_profile_id": "sw-<sha256-prefix>",
  "environment_metadata": {
    "env": {
      "IMAGE_VERSION": "py3.12-cuda13.0.0",
      "FASTVIDEO_CONTAINER_IMAGE_REF": "ghcr.io/hao-ai-lab/fastvideo/fastvideo-dev:py3.12-cuda13.0.0@sha256:<digest>"
    }
  },
  "environment_fingerprint": "env-<sha256-prefix>"
}

Normalized record (HF dataset, also dumped as normalized_perf_*.json)

Written by compare_baseline.py:_normalize_record. One file per benchmark result, used as the rolling-baseline source of truth.

{
  "model_id": "wan-t2v-1.3b-2gpu",
  "result_schema_version": 2,
  "workload_id": "wan-t2v",
  "variant_id": "1.3b-sp2",
  "benchmark_version": 3,
  "timestamp": "2026-05-08T22:00:00+00:00",
  "commit_sha": "<full sha>",
  "gpu_type": "NVIDIA L40S",
  "latency": 28.4,
  "throughput": 1.58,
  "memory": 10840.0,
  "text_encoder_time_s": 2.141,
  "dit_time_s": 8.437,
  "vae_decode_time_s": 3.208,
  "regression_thresholds": {
    "latency": {
      "threshold_percent": 0.08,
      "threshold_absolute": 0.5,
      "gated": true
    }
  },
  "recipe_fingerprint": "<sha256>",
  "hardware_profile_id": "hw-<sha256-prefix>",
  "software_profile_id": "sw-<sha256-prefix>",
  "environment_fingerprint": "env-<sha256-prefix>",
  "run_source": "pr",
  "branch": "feature/perf-change",
  "pr_number": "1234",
  "test_scope": "direct",
  "build_url": "https://buildkite.example/build",
  "build_id": "<buildkite-build-id>",
  "job_id": "<buildkite-job-id>",
  "quality_metadata": { "quality_status": "canonical" },
  "baseline_status": "compared",
  "comparison_status": "PASS",
  "comparison_status_reason": "Comparable baseline found with no gated regressions",
  "baseline_eligible": false,
  "success": true
}

Compatibility with legacy records

Older records in the HF dataset may not have result_schema_version, component timing fields, or v2 identity/profile fields. Records without result_schema_version are treated as v1. The comparator ignores missing or null metrics when computing a median, and the dashboard lists skipped plots for metric series that have no non-null values. Records missing both run_source and baseline_eligible are treated as legacy successful main/full-suite uploads and remain eligible for rolling baselines. Current perf_*.json artifacts that lack the v2 comparison identity are normalized for reporting but skip rolling-baseline comparison and are not marked baseline eligible. Dashboard reports keep historical v1 records readable by grouping them under the explicit Legacy v1 label by (model_id, gpu_type). Complete v2 cohorts remain separate and use the six-field comparison identity. Records that claim or partially use v2 identity without all required fields are labeled Invalid v2 and remain scoped by their display metadata and available identity fields. Stored comparison status, comparison reason, and baseline status are shown when present; old records remain readable when those fields are absent.

New v2 records compare only against the same workload_id, variant_id, benchmark_version, recipe_fingerprint, hardware_profile_id, and software_profile_id cohort, independent of the legacy model_id directory and gpu_type display string. Historical v1 records remain readable for reporting, but current legacy artifacts do not perform a (model_id, gpu_type) rolling comparison or seed new rolling baselines. environment_metadata and environment_fingerprint are audit data and are not part of the comparison key. The recipe prompt digests describe the prompts actually measured by the benchmark run; extra configured prompts are ignored unless the benchmark runner executes them. Software profile package cohorts keep exact versions for relevant attention/kernel packages, including FastVideo kernels, FlashAttention, FlashInfer, Cutlass DSL, SageAttention, Triton, and xFormers when installed.

Environment variable reference

Variable	Default	Used by	Purpose
PERFORMANCE_TRACKING_ROOT	/tmp/perf-tracking	compare_baseline.py, dashboard.py	Local directory the HF dataset is synced to.
PERF_REPORTS_DIR	/root/data/perf_reports	compare_baseline.py, dashboard.py	Where the Markdown summary and Plotly HTML get written for Buildkite to pick up.
HF_REPO_ID	FastVideo/performance-tracking	fastvideo/performance/hf_store.py	HF dataset repo holding rolling-baseline records.
HF_API_KEY, HUGGINGFACE_HUB_TOKEN, HF_TOKEN	unset	fastvideo/performance/hf_store.py	Required for upload or private dataset reads.
PERF_RUN_SOURCE	inferred	compare_baseline.py, test_inference_performance.py	Source metadata for uploaded records: pr, local, scheduled_main, or unknown.
PERF_UPLOAD_POLICY	never	compare_baseline.py	Upload policy: never, pass, or always.
PERF_PYTEST_RC	unset	compare_baseline.py	Performance pytest exit code. Measured static-threshold failures are attributed per record; otherwise a nonzero code reports an infrastructure error.
TEST_SCOPE	unset	compare_baseline.py	CI context used to infer scheduled-main runs together with BUILDKITE_BRANCH=main.
BUILDKITE_BRANCH, BUILDKITE_COMMIT, BUILDKITE_PULL_REQUEST	unset	compare_baseline.py, test_inference_performance.py	CI metadata stamped into records.
DASHBOARD_DAYS	30	dashboard.py	Lookback window for the Plotly trend pages.
PERFORMANCE_TRACKING_SYNC_REUSE_TTL_SECONDS	3600	fastvideo/performance/hf_store.py	Freshness window for reusing an existing HF sync when requested by dashboard consumers.
FASTVIDEO_ATTENTION_BACKEND	auto	test_inference_performance.py	Requested attention backend included in software_profile_id.
FASTVIDEO_FA4	0	test_inference_performance.py	FlashAttention-4 toggle included in software_profile_id.
FASTVIDEO_PERFORMANCE_PROFILE_VERSION	unset	test_inference_performance.py	Optional explicit software cohort/profile version included in software_profile_id.
IMAGE_VERSION	unset	test_inference_performance.py	CI container image/profile version included in software_profile_id when available.
FASTVIDEO_CONTAINER_IMAGE_REF	unset	Slurm runner, test_inference_performance.py	Pinned CI container image digest recorded in environment_metadata and software_profile_id.
FASTVIDEO_STAGE_LOGGING	set by the pytest test	test_inference_performance.py	Enables pipeline stage timing capture for component metrics during benchmark runs.

CI integration

The performance step can run on demand with /test performance, through a merge gate when performance tests or benchmark policy changed, and as part of an explicit /test full run (see CI/CD Architecture). The weekly fastvideo-performance-lane schedule runs the same Slurm payload. The active entry point is .buildkite/scripts/lanes/performance.sh; the trusted host dispatcher relays its allowlisted reports to Buildkite after the isolated container exits.

Each performance build runs pytest first. PR and direct runs only continue to compare_baseline.py when that fixed-threshold phase passes; if pytest fails, Markdown summaries and normalized JSON artifacts are not emitted. Scheduled main runs set PERF_UPLOAD_POLICY=always, so they still run compare_baseline.py (with PERF_PYTEST_RC set) after pytest fails. Each raw record is checked against its own static thresholds: a measured breach reports REGRESSION, while unaffected records retain their rolling-baseline status. A nonzero pytest exit with no attributable static-threshold breach reports INFRA_ERROR. Failed records have success=false and are excluded from future rolling baselines. The dashboard still runs best-effort for observability.

Each benchmark run also captures worker-process logs into results/worker_logs/worker_<benchmark_id>_<ts>.log (raw records point to it via worker_log_path; the field is null/absent in older and HF-synced records). On a hard regression, a 200-line tail of that log is printed in the failure output — by the pytest assertion on PR runs and by compare_baseline.py on scheduled runs. The performance lane (.buildkite/scripts/lanes/performance.sh) also mirrors each captured log into $PERF_REPORTS_DIR as worker_<benchmark_id>_<ts>.md next to the raw JSON results, so the log is retrievable as a build artifact even when the comparison phase never runs. The mirror uses an allowlisted extension because the trusted host relays only .md, .html, .json, and .csv files from that directory; the .log files themselves are never uploaded. Coverage caveat: only the fastvideo logger is captured (no torch/NCCL or raw stderr output), and ranks > 0 suppress logger.info by default, so the file contains rank-0 INFO plus WARNING/ERROR from all ranks. When the rolling-baseline phase runs, it emits:

• Markdown summary — appended to $GITHUB_STEP_SUMMARY when that variable is set, and written as perf_<sha>_<ts>.md for Buildkite upload. Contains a per-benchmark row with current vs. baseline values for latency, throughput, memory, text encoder time, DiT time, and VAE decode time.
• Plotly dashboard — dashboard_<sha>_<ts>.html showing time-series for each metric grouped by comparison cohort. Chart titles distinguish Legacy v1 and incomplete Invalid v2 records, and point hover data includes stored comparison and baseline status metadata.
• Normalized records — normalized_perf_*.json, one per benchmark. Useful as input to the reseed-performance-baseline skill.

Adding a new benchmark

1. Drop a new JSON config into .buildkite/performance-benchmarks/tests/<name>.json. New configs should use v2 identity fields:

   {
     "benchmark_id": "<unique-id>",
     "config_schema_version": 2,
     "workload_id": "<stable-workload-id>",
     "variant_id": "<variant, e.g. 1.3b-sp2>",
     "benchmark_version": 1,
     "model": { "model_path": "...", "model_short_name": "..." },
     "init_kwargs": { "num_gpus": 1, ... },
     "generation_kwargs": { "num_frames": 45, ... },
     "test_prompts": ["..."],
     "run_config": { "required_gpus": 1,
                     "num_warmup_runs": 1, "num_measurement_runs": 3,
                     "gpu_types": ["<device-name-substring>"] },
     "thresholds": {
       "L40S": {
         "max_generation_time_s": 34.0,
         "max_peak_memory_mb": 11000.0,
         "max_text_encoder_time_s": 5.0,
         "max_dit_time_s": 10.0,
         "max_vae_decode_time_s": 10.0
       },
       "default": { "max_generation_time_s": 120.0, "max_peak_memory_mb": 30000.0 }
     },
     "regression_thresholds": {
       "latency": { "threshold_percent": 0.10, "threshold_absolute": 1.0, "gated": true }
     }
   }
   
   

   Legacy v1 configs without config_schema_version still load, but should not gain v2 identity or metadata fields until they are migrated to config_schema_version: 2. For v2 configs, workload_id, variant_id, and benchmark_version are part of the comparison key; benchmark runs fail if any of these identity fields are missing.

2. The pytest test auto-discovers all configs — no test code needed. CI picks it up on the next /test performance run.

3. Legacy benchmarks are gated only by their static thresholds. Their current records skip rolling-baseline comparison and are never baseline eligible. V2 benchmarks with no exact comparable baseline report CALIBRATION_NEEDED; the record remains visible but does not become baseline eligible until a comparable scheduled-main full-suite run is reviewed and seeded through the prepare, review, and conditional-upload steps in the reseed-performance-baseline skill.

4. If the benchmark targets a GPU not currently in thresholds, either add that GPU as a key or rely on the default block. Note that default is intended for slower fallback GPUs, so its values should be relaxed relative to the fastest entry.

5. Add component thresholds only when the stage timing is stable enough to be a useful fixed gate. The rolling baseline will still track component times when static component thresholds are omitted.

6. Omit regression_thresholds to use the default rolling-baseline policy, or include only benchmark-specific deviations. Tune these independently from the fixed thresholds when a metric is noisy or should be informational. The fixed thresholds block is an absolute pytest ceiling. The regression_thresholds block controls rolling-baseline comparisons against recent scheduled-main records.

Troubleshooting

CALIBRATION_NEEDED / "No baseline found for exact comparable identity" — the v2 run passes, but its normalized record remains baseline_eligible=false. Review a successful scheduled-main full-suite normalized artifact, then follow the reseed-performance-baseline skill. The utility only prepares a digest-protected manifest; the separately confirmed upload rechecks remote state and commits the batch atomically.

Persistent failure right after a torch / kernel / image upgrade — genuine regression or baseline drift. Compare the failing normalized record with recent successful records in the HF dataset. If the shift is expected and reviewed, use the reseed-performance-baseline skill.

Dashboard reports skipped metric plots — the loaded records do not have non-null values for that metric. This is expected for older records or for pipelines that did not report a mapped component stage.

Component timing is null — the generated result did not include a mapped stage in logging_info.stages. Check that the pipeline emits stage logging and that the stage emits component_metric or is covered by the legacy STAGE_METRIC_MAP fallback in test_inference_performance.py.