From cbc8a61b1da6aa10bb878d22155ee9d473ce1e90 Mon Sep 17 00:00:00 2001 From: Vito Sansevero Date: Wed, 21 Oct 2026 21:55:21 -0700 Subject: [PATCH] test(browser): stable under parallel workers Document how to run the browser suites in parallel and sharded, with the measured times, and what keeps specs stable beside each other: wait for a rendering frame, a response or a visible state, never for time. With b665aa6 the full workflows suite passed on chromium, touch, firefox and webkit with 8 workers, each test 3 times (6132 passed, 0 failed, 24 minutes), and once at CI size (docker --cpus=4, 3 workers: 2044 passed, 25.5 minutes). The npm download cache removed the npm ci stall: 0 to 2 s in every job of run 39722172529, against about 300 s in most jobs before. The workflow's timing comment now records that run; the whole workflow took 8m54s against about 33 minutes before sharding. Shard imbalance (shard 1 carries the concept-comparison specs) is documented, not tuned. --- .github/workflows/viewerjs-check.yml | 4 ++- TESTING.md | 41 +++++++++++++++++++++++++++- 2 files changed, 43 insertions(+), 2 deletions(-) diff --git a/.github/workflows/viewerjs-check.yml b/.github/workflows/viewerjs-check.yml index ecbb413..0432084 100644 --- a/.github/workflows/viewerjs-check.yml +++ b/.github/workflows/viewerjs-check.yml @@ -22,7 +22,9 @@ concurrency: # no npm cache, where npm ci stalled for 5m in six of eight jobs): coverage shards 11m19s/10m47s/10m49s # (tests 5m20s/4m42s/4m46s), merge 5m52s (report and check 4s), firefox shards 7m53s/9m27s (tests # 4m41s/3m29s), webkit shards 12m25s+ (cancelled at the old 12-minute limit after its 267 tests passed in -# 6m29s) and 4m53s (tests 3m08s). +# 6m29s) and 4m53s (tests 3m08s). With the npm cache hit (run 39722172529, npm ci 0-2s): coverage shards +# 7m56s/5m10s/4m55s (tests 6m42s/4m09s/3m45s), merge 47s, firefox 8m10s/5m28s (tests 7m14s/4m34s), +# webkit 6m31s/4m30s (tests 5m32s/3m29s); a cache miss adds up to 5m to npm ci, which the limits absorb. jobs: frontend-coverage: name: frontend coverage (shard ${{ matrix.shard }}/3) diff --git a/TESTING.md b/TESTING.md index 3d84a81..0c2901a 100644 --- a/TESTING.md +++ b/TESTING.md @@ -45,7 +45,34 @@ which binds a port the OS picks and prints it as `PM_TEST_SERVER_PORT=`, with its own temp root and database (`tests/browser/fixtures.mjs`), so parallel workers and concurrent runs never share state. `PM_TEST_WORKERS` sets the worker count (default 1 under `CI`, otherwise a quarter of the CPUs, at most 8). -Test servers are never reused across runs. On unsupported Linux +Test servers are never reused across runs. + +The coverage run can be split the way CI splits it, and each shard keeps the raw +V8 files it writes under `coverage/js/raw`: + +```sh +PM_TEST_WORKERS=8 npm run --prefix tools/vendor test:coverage # whole run, one report +npm run --prefix tools/vendor test:coverage:shard -- 1/3 # node units, viewer suite, workflows 1/3 +npm run --prefix tools/vendor test:coverage:shard -- 2/3 # workflows 2/3 (likewise 3/3) +npm run --prefix tools/vendor coverage:merge # report and baseline check of every raw file +npm run --prefix tools/vendor test:workflows -- --project webkit --shard 2/2 +``` + +`coverage:merge` refuses an empty raw directory; it does not count shards, so run +every shard first (CI's merge job checks all three arrived). Measured on a 32-CPU +machine in the pinned image (2026-10-21): `test:coverage` takes about 5 minutes +with 8 workers and 24 minutes with 1; all four projects, 8 workers, each test 3 +times (6132 tests), 24 minutes. A CI-sized run (`docker run --cpus=4`, +`PM_TEST_WORKERS=3`, all four projects once, 2044 tests) takes 25.5 minutes. + +Specs must pass beside each other, not only alone. Wait for the page's own +signal, never for time: a rendering frame (`requestAnimationFrame`) for layout +work queued by a render or a scroll, a response or a visible state for data. A +timer that "usually" covers a frame is a race on a loaded runner. Library +virtualization specs settle with `libraryObservation.settled()` (two consecutive +frames with no Library frame queued). To look for such races, repeat a spec under +load (`--workers 8 --repeat-each 10`, and in the image with `--cpus=1` or `2`). +On unsupported Linux distributions, browser system dependencies may need manual installation. For a quick browser run, use `test:workflows -- --project chromium`. @@ -136,6 +163,18 @@ JSON summary; `frontend-traces-shard-N` hold the Chromium/touch failure screenshots/traces and `browser-traces--shard-N` the other browsers' traces. Open traces with Playwright's `show-trace` tool. +Every `npm ci` first restores npm's download cache (`~/.npm`), keyed on the vendor +lockfile and the pinned image. Without it, `npm ci` stalled for about 300 s +(npm's fetch timeout) in most of the eight concurrent jobs; with a hit it takes +0 to 2 s. A lockfile change misses the cache once, so expect one slow run after it. +Measured on PR #274 (run 39722172529, cache hit): the whole workflow took 8m54s, +against about 33 minutes with one worker and no shards. Jobs: coverage shards +7m56s/5m10s/4m55s (tests 6m42s/4m09s/3m45s), merge 47s, firefox shards +8m10s/5m28s (tests 7m14s/4m34s), webkit shards 6m31s/4m30s (tests 5m32s/3m29s). +Playwright shards by test count, whole files at a time, so shard 1 carries the +slow concept-comparison specs (`*-concept.spec.mjs`, alphabetically first) and +runs longer; the remedy, if it starts to matter, is more shards, not reordering. + The real-host suite covers both Classic and Nodes 2.0 startup, switching both ways, actual database prompt selection, execution-text serialization, workflow reload, widget/style cleanup, node schemas and standalone-page opening. It does