Commit Graph
12 Commits
Author SHA1 Message Date
SolitaryThinker da9cbe3da9 [ci]: skip perf gating on packed hosts via host CPU profile guard
The wan-t2v perf lane intermittently runs ~1.7x slow on packed Modal
hosts (TE pinned at 3.595s vs the 2.03s healthy cluster, DiT +58-80%,
GPU clocks at full boost): host CPU contention that cpu=8/16
reservations cannot isolate. 9 of the last ~11 full suites redded only
on this lane. A perf gate must never fail a PR on a measurement taken
on a degraded host (review r30, option a).

The harness now runs a ~1.5s host CPU probe (single-thread pure-Python
loop, gated axis; 8-thread torch matmul, informational) once per pytest
process and stores host_cpu_score on every record. The comparator emits
HOST_BELOW_PROFILE when the score is under the calibrated floor
(PERF_HOST_CPU_MIN_SCORE, default 0.75): gating is skipped loudly, CI
passes, and the record can never seed a baseline. Records without a
score gate normally.

Calibrated against five fresh perf-lane containers (L40S:2 cpu=8):
healthy hosts score 0.97-1.17, packed-host projections 0.54-0.73;
sampled data and recalibration procedure documented in host_probe.py.

Stacked on maint/perf-comparator-statuses (f46c9e61).
2026-07-13 16:25:26 -07:00
SolitaryThinker 04bfddb7fa [ci]: add exact-identity perf comparator statuses
Comparator verdicts per issue #1532: PASS / REGRESSION /
CALIBRATION_NEEDED / RECIPE_MISMATCH / INFRA_ERROR (QUALITY_BLOCKED
reserved for the promoted-baseline workflow). REGRESSION and
RECIPE_MISMATCH fail CI; CALIBRATION_NEEDED is loud but no longer
auto-seeds a passing baseline.
2026-07-13 16:24:52 -07:00
Mac LeeandSolitaryThinker afb4f7d3c5 [ci]: emit v2 performance result schema (#1551)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2026-07-09 06:11:15 +00:00
Mac LeeandSolitaryThinker 1ee11e08dc [ci]: add performance fingerprint cohorts (#1546)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2026-07-07 07:24:00 +00:00
Mac Lee 30b523edd6 [ci] Normalize performance stage component metrics (#1475) (#1550) 2026-07-05 14:05:26 -07:00
Mac Lee 9cd53fe5f8 [ci] Add metric-specific performance thresholds (#1545) 2026-07-05 12:04:33 -07:00
Mac Lee c53e85b767 [ci] Add v2 performance benchmark config identity fields (#1544) 2026-07-05 06:18:15 -07:00
Satyam Srivastava 8658f774f3 [docs] Restructure contributing CI/CD and testing docs (#1501) 2026-06-27 11:46:18 -07:00
Satyam Srivastava 3d8cc4f0a0 [bugfix] Fix performance component timing extraction (#1473) 2026-06-22 13:05:35 -07:00
Satyam Srivastava 0557f7a7d9 [ci] Add performance dashboard metadata and visualizations (#1470) 2026-06-19 14:27:01 -07:00
Satyam Srivastava b80148819c [ci] Add performance dashboard visualisation scripts (#1469) 2026-06-16 23:29:29 -07:00
Satyam SrivastavaandSolitaryThinker 2dd5760291 [docs] Document performance benchmark workflow (#1376)
Co-authored-by: SolitaryThinker <wlsaidhi@gmail.com>
2026-05-21 20:25:58 -07:00