docs(testing): server lifetime, stderr attachments and the final measured CI times

TESTING.md describes the stdin-EOF shutdown and the server-stderr
attachment, states that the npm cache key names the job's image tag
(now tested), quotes run 39725176320 at cbc8a61 (8m38s) and explains the
20-minute browser job limit. The workflow timing comment quotes the same
run.
This commit is contained in:
Vito Sansevero
2026-10-21 22:37:49 -07:00
parent ed809f7fa4
commit 14cfb8aedf
2 changed files with 18 additions and 9 deletions
+4 -3
View File
@@ -22,9 +22,10 @@ concurrency:
# no npm cache, where npm ci stalled for 5m in six of eight jobs): coverage shards 11m19s/10m47s/10m49s
# (tests 5m20s/4m42s/4m46s), merge 5m52s (report and check 4s), firefox shards 7m53s/9m27s (tests
# 4m41s/3m29s), webkit shards 12m25s+ (cancelled at the old 12-minute limit after its 267 tests passed in
# 6m29s) and 4m53s (tests 3m08s). With the npm cache hit (run 39722172529, npm ci 0-2s): coverage shards
# 7m56s/5m10s/4m55s (tests 6m42s/4m09s/3m45s), merge 47s, firefox 8m10s/5m28s (tests 7m14s/4m34s),
# webkit 6m31s/4m30s (tests 5m32s/3m29s); a cache miss adds up to 5m to npm ci, which the limits absorb.
# 6m29s) and 4m53s (tests 3m08s). With the npm cache hit (run 39725176320 at cbc8a61, npm ci 0-2s):
# coverage shards 7m34s/6m04s/5m22s (tests 6m38s/4m53s/4m04s), merge 54s, firefox 7m57s/5m21s (tests
# 7m08s/4m34s), webkit 8m16s/5m03s (tests 7m26s/3m58s); a cache miss adds up to 5m to npm ci, which
# the 20-minute limits absorb.
jobs:
frontend-coverage:
name: frontend coverage (shard ${{ matrix.shard }}/3)
+14 -6
View File
@@ -45,7 +45,11 @@ which binds a port the OS picks and prints it as `PM_TEST_SERVER_PORT=<n>`, with
its own temp root and database (`tests/browser/fixtures.mjs`), so
parallel workers and concurrent runs never share state. `PM_TEST_WORKERS` sets the
worker count (default 1 under `CI`, otherwise a quarter of the CPUs, at most 8).
Test servers are never reused across runs.
Test servers are never reused across runs. The launcher holds each server's
stdin open and the server (`--exit-on-stdin-eof`) shuts down cleanly, removing its
temp root, when that pipe closes, so a worker killed without teardown leaves no
server behind. A test that fails gets the stderr its server wrote during the test
attached to the report as `server-stderr` (an aiohttp traceback behind a 500).
The coverage run can be split the way CI splits it, and each shard keeps the raw
V8 files it writes under `coverage/js/raw`:
@@ -164,13 +168,17 @@ screenshots/traces and `browser-traces-<browser>-shard-N` the other browsers'
traces. Open traces with Playwright's `show-trace` tool.
Every `npm ci` first restores npm's download cache (`~/.npm`), keyed on the vendor
lockfile and the pinned image. Without it, `npm ci` stalled for about 300 s
lockfile and the job's pinned image tag (`tests/vendor/ci-workflow.test.mjs` checks
the key names the same tag as the job's `container.image`). Without it, `npm ci` stalled for about 300 s
(npm's fetch timeout) in most of the eight concurrent jobs; with a hit it takes
0 to 2 s. A lockfile change misses the cache once, so expect one slow run after it.
Measured on PR #274 (run 39722172529, cache hit): the whole workflow took 8m54s,
against about 33 minutes with one worker and no shards. Jobs: coverage shards
7m56s/5m10s/4m55s (tests 6m42s/4m09s/3m45s), merge 47s, firefox shards
8m10s/5m28s (tests 7m14s/4m34s), webkit shards 6m31s/4m30s (tests 5m32s/3m29s).
Measured on PR #274 (run 39725176320 at cbc8a61, cache hit): the whole workflow
took 8m38s, against about 33 minutes with one worker and no shards. Jobs: coverage
shards 7m34s/6m04s/5m22s (tests 6m38s/4m53s/4m04s), merge 54s, firefox shards
7m57s/5m21s (tests 7m08s/4m34s), webkit shards 8m16s/5m03s (tests 7m26s/3m58s).
Browser jobs keep a 20-minute limit: the slowest job is about 8.5 minutes, a cache
miss adds up to 5 to `npm ci`, and runner speed varies (the first sharded run was
cancelled at a 12-minute limit).
Playwright shards by test count, whole files at a time, so shard 1 carries the
slow concept-comparison specs (`*-concept.spec.mjs`, alphabetically first) and
runs longer; the remedy, if it starts to matter, is more shards, not reordering.