From 14cfb8aedf78411d9a13bedba573226a2deeb7f2 Mon Sep 17 00:00:00 2001 From: Vito Sansevero Date: Wed, 21 Oct 2026 22:37:49 -0700 Subject: [PATCH] docs(testing): server lifetime, stderr attachments and the final measured CI times TESTING.md describes the stdin-EOF shutdown and the server-stderr attachment, states that the npm cache key names the job's image tag (now tested), quotes run 39725176320 at cbc8a61 (8m38s) and explains the 20-minute browser job limit. The workflow timing comment quotes the same run. --- .github/workflows/viewerjs-check.yml | 7 ++++--- TESTING.md | 20 ++++++++++++++------ 2 files changed, 18 insertions(+), 9 deletions(-) diff --git a/.github/workflows/viewerjs-check.yml b/.github/workflows/viewerjs-check.yml index 0432084..7c57cd2 100644 --- a/.github/workflows/viewerjs-check.yml +++ b/.github/workflows/viewerjs-check.yml @@ -22,9 +22,10 @@ concurrency: # no npm cache, where npm ci stalled for 5m in six of eight jobs): coverage shards 11m19s/10m47s/10m49s # (tests 5m20s/4m42s/4m46s), merge 5m52s (report and check 4s), firefox shards 7m53s/9m27s (tests # 4m41s/3m29s), webkit shards 12m25s+ (cancelled at the old 12-minute limit after its 267 tests passed in -# 6m29s) and 4m53s (tests 3m08s). With the npm cache hit (run 39722172529, npm ci 0-2s): coverage shards -# 7m56s/5m10s/4m55s (tests 6m42s/4m09s/3m45s), merge 47s, firefox 8m10s/5m28s (tests 7m14s/4m34s), -# webkit 6m31s/4m30s (tests 5m32s/3m29s); a cache miss adds up to 5m to npm ci, which the limits absorb. +# 6m29s) and 4m53s (tests 3m08s). With the npm cache hit (run 39725176320 at cbc8a61, npm ci 0-2s): +# coverage shards 7m34s/6m04s/5m22s (tests 6m38s/4m53s/4m04s), merge 54s, firefox 7m57s/5m21s (tests +# 7m08s/4m34s), webkit 8m16s/5m03s (tests 7m26s/3m58s); a cache miss adds up to 5m to npm ci, which +# the 20-minute limits absorb. jobs: frontend-coverage: name: frontend coverage (shard ${{ matrix.shard }}/3) diff --git a/TESTING.md b/TESTING.md index 0c2901a..76031db 100644 --- a/TESTING.md +++ b/TESTING.md @@ -45,7 +45,11 @@ which binds a port the OS picks and prints it as `PM_TEST_SERVER_PORT=`, with its own temp root and database (`tests/browser/fixtures.mjs`), so parallel workers and concurrent runs never share state. `PM_TEST_WORKERS` sets the worker count (default 1 under `CI`, otherwise a quarter of the CPUs, at most 8). -Test servers are never reused across runs. +Test servers are never reused across runs. The launcher holds each server's +stdin open and the server (`--exit-on-stdin-eof`) shuts down cleanly, removing its +temp root, when that pipe closes, so a worker killed without teardown leaves no +server behind. A test that fails gets the stderr its server wrote during the test +attached to the report as `server-stderr` (an aiohttp traceback behind a 500). The coverage run can be split the way CI splits it, and each shard keeps the raw V8 files it writes under `coverage/js/raw`: @@ -164,13 +168,17 @@ screenshots/traces and `browser-traces--shard-N` the other browsers' traces. Open traces with Playwright's `show-trace` tool. Every `npm ci` first restores npm's download cache (`~/.npm`), keyed on the vendor -lockfile and the pinned image. Without it, `npm ci` stalled for about 300 s +lockfile and the job's pinned image tag (`tests/vendor/ci-workflow.test.mjs` checks +the key names the same tag as the job's `container.image`). Without it, `npm ci` stalled for about 300 s (npm's fetch timeout) in most of the eight concurrent jobs; with a hit it takes 0 to 2 s. A lockfile change misses the cache once, so expect one slow run after it. -Measured on PR #274 (run 39722172529, cache hit): the whole workflow took 8m54s, -against about 33 minutes with one worker and no shards. Jobs: coverage shards -7m56s/5m10s/4m55s (tests 6m42s/4m09s/3m45s), merge 47s, firefox shards -8m10s/5m28s (tests 7m14s/4m34s), webkit shards 6m31s/4m30s (tests 5m32s/3m29s). +Measured on PR #274 (run 39725176320 at cbc8a61, cache hit): the whole workflow +took 8m38s, against about 33 minutes with one worker and no shards. Jobs: coverage +shards 7m34s/6m04s/5m22s (tests 6m38s/4m53s/4m04s), merge 54s, firefox shards +7m57s/5m21s (tests 7m08s/4m34s), webkit shards 8m16s/5m03s (tests 7m26s/3m58s). +Browser jobs keep a 20-minute limit: the slowest job is about 8.5 minutes, a cache +miss adds up to 5 to `npm ci`, and runner speed varies (the first sharded run was +cancelled at a 12-minute limit). Playwright shards by test count, whole files at a time, so shard 1 carries the slow concept-comparison specs (`*-concept.spec.mjs`, alphabetically first) and runs longer; the remedy, if it starts to matter, is more shards, not reordering.