test(browser): stable under parallel workers
Document how to run the browser suites in parallel and sharded, with the
measured times, and what keeps specs stable beside each other: wait for
a rendering frame, a response or a visible state, never for time.
With b665aa6 the full workflows suite passed on chromium, touch, firefox
and webkit with 8 workers, each test 3 times (6132 passed, 0 failed,
24 minutes), and once at CI size (docker --cpus=4, 3 workers: 2044
passed, 25.5 minutes).
The npm download cache removed the npm ci stall: 0 to 2 s in every job
of run 39722172529, against about 300 s in most jobs before. The
workflow's timing comment now records that run; the whole workflow took
8m54s against about 33 minutes before sharding. Shard imbalance (shard 1
carries the concept-comparison specs) is documented, not tuned.
This commit is contained in:
@@ -22,7 +22,9 @@ concurrency:
|
|||||||
# no npm cache, where npm ci stalled for 5m in six of eight jobs): coverage shards 11m19s/10m47s/10m49s
|
# no npm cache, where npm ci stalled for 5m in six of eight jobs): coverage shards 11m19s/10m47s/10m49s
|
||||||
# (tests 5m20s/4m42s/4m46s), merge 5m52s (report and check 4s), firefox shards 7m53s/9m27s (tests
|
# (tests 5m20s/4m42s/4m46s), merge 5m52s (report and check 4s), firefox shards 7m53s/9m27s (tests
|
||||||
# 4m41s/3m29s), webkit shards 12m25s+ (cancelled at the old 12-minute limit after its 267 tests passed in
|
# 4m41s/3m29s), webkit shards 12m25s+ (cancelled at the old 12-minute limit after its 267 tests passed in
|
||||||
# 6m29s) and 4m53s (tests 3m08s).
|
# 6m29s) and 4m53s (tests 3m08s). With the npm cache hit (run 39722172529, npm ci 0-2s): coverage shards
|
||||||
|
# 7m56s/5m10s/4m55s (tests 6m42s/4m09s/3m45s), merge 47s, firefox 8m10s/5m28s (tests 7m14s/4m34s),
|
||||||
|
# webkit 6m31s/4m30s (tests 5m32s/3m29s); a cache miss adds up to 5m to npm ci, which the limits absorb.
|
||||||
jobs:
|
jobs:
|
||||||
frontend-coverage:
|
frontend-coverage:
|
||||||
name: frontend coverage (shard ${{ matrix.shard }}/3)
|
name: frontend coverage (shard ${{ matrix.shard }}/3)
|
||||||
|
|||||||
+40
-1
@@ -45,7 +45,34 @@ which binds a port the OS picks and prints it as `PM_TEST_SERVER_PORT=<n>`, with
|
|||||||
its own temp root and database (`tests/browser/fixtures.mjs`), so
|
its own temp root and database (`tests/browser/fixtures.mjs`), so
|
||||||
parallel workers and concurrent runs never share state. `PM_TEST_WORKERS` sets the
|
parallel workers and concurrent runs never share state. `PM_TEST_WORKERS` sets the
|
||||||
worker count (default 1 under `CI`, otherwise a quarter of the CPUs, at most 8).
|
worker count (default 1 under `CI`, otherwise a quarter of the CPUs, at most 8).
|
||||||
Test servers are never reused across runs. On unsupported Linux
|
Test servers are never reused across runs.
|
||||||
|
|
||||||
|
The coverage run can be split the way CI splits it, and each shard keeps the raw
|
||||||
|
V8 files it writes under `coverage/js/raw`:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
PM_TEST_WORKERS=8 npm run --prefix tools/vendor test:coverage # whole run, one report
|
||||||
|
npm run --prefix tools/vendor test:coverage:shard -- 1/3 # node units, viewer suite, workflows 1/3
|
||||||
|
npm run --prefix tools/vendor test:coverage:shard -- 2/3 # workflows 2/3 (likewise 3/3)
|
||||||
|
npm run --prefix tools/vendor coverage:merge # report and baseline check of every raw file
|
||||||
|
npm run --prefix tools/vendor test:workflows -- --project webkit --shard 2/2
|
||||||
|
```
|
||||||
|
|
||||||
|
`coverage:merge` refuses an empty raw directory; it does not count shards, so run
|
||||||
|
every shard first (CI's merge job checks all three arrived). Measured on a 32-CPU
|
||||||
|
machine in the pinned image (2026-10-21): `test:coverage` takes about 5 minutes
|
||||||
|
with 8 workers and 24 minutes with 1; all four projects, 8 workers, each test 3
|
||||||
|
times (6132 tests), 24 minutes. A CI-sized run (`docker run --cpus=4`,
|
||||||
|
`PM_TEST_WORKERS=3`, all four projects once, 2044 tests) takes 25.5 minutes.
|
||||||
|
|
||||||
|
Specs must pass beside each other, not only alone. Wait for the page's own
|
||||||
|
signal, never for time: a rendering frame (`requestAnimationFrame`) for layout
|
||||||
|
work queued by a render or a scroll, a response or a visible state for data. A
|
||||||
|
timer that "usually" covers a frame is a race on a loaded runner. Library
|
||||||
|
virtualization specs settle with `libraryObservation.settled()` (two consecutive
|
||||||
|
frames with no Library frame queued). To look for such races, repeat a spec under
|
||||||
|
load (`--workers 8 --repeat-each 10`, and in the image with `--cpus=1` or `2`).
|
||||||
|
On unsupported Linux
|
||||||
distributions, browser system dependencies may need manual installation. For a
|
distributions, browser system dependencies may need manual installation. For a
|
||||||
quick browser run, use `test:workflows -- --project chromium`.
|
quick browser run, use `test:workflows -- --project chromium`.
|
||||||
|
|
||||||
@@ -136,6 +163,18 @@ JSON summary; `frontend-traces-shard-N` hold the Chromium/touch failure
|
|||||||
screenshots/traces and `browser-traces-<browser>-shard-N` the other browsers'
|
screenshots/traces and `browser-traces-<browser>-shard-N` the other browsers'
|
||||||
traces. Open traces with Playwright's `show-trace` tool.
|
traces. Open traces with Playwright's `show-trace` tool.
|
||||||
|
|
||||||
|
Every `npm ci` first restores npm's download cache (`~/.npm`), keyed on the vendor
|
||||||
|
lockfile and the pinned image. Without it, `npm ci` stalled for about 300 s
|
||||||
|
(npm's fetch timeout) in most of the eight concurrent jobs; with a hit it takes
|
||||||
|
0 to 2 s. A lockfile change misses the cache once, so expect one slow run after it.
|
||||||
|
Measured on PR #274 (run 39722172529, cache hit): the whole workflow took 8m54s,
|
||||||
|
against about 33 minutes with one worker and no shards. Jobs: coverage shards
|
||||||
|
7m56s/5m10s/4m55s (tests 6m42s/4m09s/3m45s), merge 47s, firefox shards
|
||||||
|
8m10s/5m28s (tests 7m14s/4m34s), webkit shards 6m31s/4m30s (tests 5m32s/3m29s).
|
||||||
|
Playwright shards by test count, whole files at a time, so shard 1 carries the
|
||||||
|
slow concept-comparison specs (`*-concept.spec.mjs`, alphabetically first) and
|
||||||
|
runs longer; the remedy, if it starts to matter, is more shards, not reordering.
|
||||||
|
|
||||||
The real-host suite covers both Classic and Nodes 2.0 startup, switching both
|
The real-host suite covers both Classic and Nodes 2.0 startup, switching both
|
||||||
ways, actual database prompt selection, execution-text serialization, workflow
|
ways, actual database prompt selection, execution-text serialization, workflow
|
||||||
reload, widget/style cleanup, node schemas and standalone-page opening. It does
|
reload, widget/style cleanup, node schemas and standalone-page opening. It does
|
||||||
|
|||||||
Reference in New Issue
Block a user