ci: run frontend coverage in four shards (#565)
* ci: run frontend coverage in four shards Coverage shards reached their 18-minute step limit three times on 2026-10-31 with every test passing. Split the run into four shards with the limits unchanged; the merge job now expects four raw coverage artifacts, and the vendor checks move to shard 4. Closes #563 * test(tags): pin the 390px window check to the first page's last row Reaching the last row loads the next page (D183), and last() then followed row 200, far outside the window, so the WebKit check failed in 4-5 of 20 runs on 2 CPUs. Pinned to row 100, it passed 90 of 90 across WebKit, Chromium and Firefox. Closes #566
This commit is contained in:
Vito
@@ -41,9 +41,13 @@ concurrency:
|
||||
# 7.3m-12.2m), because --shard splits by test count and the slow concept specs sort into shard 1. The page
|
||||
# workflows now run in three shards per browser with the limits unchanged; the per-shard times are
|
||||
# re-measured after three green main runs.
|
||||
# #563: coverage shards reached their 18-minute step limit three times on 2026-10-31 with no failing
|
||||
# test (PR #560 shard 2/3: 814 passed in 17.9m; queue run 41206495451 shard 1/3: 827 passed; PR #564
|
||||
# shard 2/3 twice), against recent green shards of 11m-17m. Coverage now runs in four shards with the
|
||||
# limits unchanged; the per-shard times are re-measured after three green main runs.
|
||||
jobs:
|
||||
frontend-coverage:
|
||||
name: frontend coverage (shard ${{ matrix.shard }}/3)
|
||||
name: frontend coverage (shard ${{ matrix.shard }}/4)
|
||||
runs-on: ubuntu-latest
|
||||
# Layout comparisons depend on the fonts each browser falls back to, so browser jobs run in the
|
||||
# same pinned Playwright image as local runs (tag = @playwright/test in tools/vendor/package-lock.json;
|
||||
@@ -59,7 +63,7 @@ jobs:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
shard: [1, 2, 3]
|
||||
shard: [1, 2, 3, 4]
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
# The image's own Python 3.12 and Node 24: setup-python/setup-node install the runner's toolcache
|
||||
@@ -88,13 +92,13 @@ jobs:
|
||||
- run: npm ci --prefix tools/vendor --ignore-scripts
|
||||
# Shard 1 also runs the node units and the viewer suite, so the vendor pins and the tooling tests
|
||||
# run on the lightest shard instead.
|
||||
- if: matrix.shard == 3
|
||||
- if: matrix.shard == 4
|
||||
run: npm run --prefix tools/vendor check
|
||||
- if: matrix.shard == 3
|
||||
- if: matrix.shard == 4
|
||||
run: npm run --prefix tools/vendor test
|
||||
- name: Page workflows with coverage, shard ${{ matrix.shard }}/3
|
||||
- name: Page workflows with coverage, shard ${{ matrix.shard }}/4
|
||||
timeout-minutes: 18
|
||||
run: npm run --prefix tools/vendor test:coverage:shard -- ${{ matrix.shard }}/3
|
||||
run: npm run --prefix tools/vendor test:coverage:shard -- ${{ matrix.shard }}/4
|
||||
- name: Upload raw coverage
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
@@ -136,13 +140,13 @@ jobs:
|
||||
with:
|
||||
pattern: coverage-raw-*
|
||||
path: ${{ runner.temp }}/coverage-raw
|
||||
# coverage:merge only checks that raw files exist; merging two of three shards would report
|
||||
# coverage:merge only checks that raw files exist; merging three of four shards would report
|
||||
# partial coverage, so a missing or empty shard artifact fails here.
|
||||
- name: Verify every shard's raw coverage arrived
|
||||
shell: bash
|
||||
env:
|
||||
STAGED_RAW: ${{ runner.temp }}/coverage-raw
|
||||
EXPECTED_SHARDS: 3
|
||||
EXPECTED_SHARDS: 4
|
||||
run: |
|
||||
found=0
|
||||
for shard in $(seq 1 "$EXPECTED_SHARDS"); do
|
||||
|
||||
+7
-7
@@ -74,14 +74,14 @@ V8 files it writes under `coverage/js/raw`:
|
||||
|
||||
```sh
|
||||
PM_TEST_WORKERS=8 npm run --prefix tools/vendor test:coverage # whole run, one report
|
||||
npm run --prefix tools/vendor test:coverage:shard -- 1/3 # node units, viewer suite, workflows 1/3
|
||||
npm run --prefix tools/vendor test:coverage:shard -- 2/3 # workflows 2/3 (likewise 3/3)
|
||||
npm run --prefix tools/vendor test:coverage:shard -- 1/4 # node units, viewer suite, workflows 1/4
|
||||
npm run --prefix tools/vendor test:coverage:shard -- 2/4 # workflows 2/4 (likewise 3/4, 4/4)
|
||||
npm run --prefix tools/vendor coverage:merge # report and baseline check of every raw file
|
||||
npm run --prefix tools/vendor test:workflows -- --project webkit --shard 2/2
|
||||
```
|
||||
|
||||
`coverage:merge` refuses an empty raw directory; it does not count shards, so run
|
||||
every shard first (CI's merge job checks all three arrived). Measured on a 32-CPU
|
||||
every shard first (CI's merge job checks all four arrived). Measured on a 32-CPU
|
||||
machine in the pinned image (2026-10-21): `test:coverage` takes about 5 minutes
|
||||
with 8 workers and 24 minutes with 1; all four projects, 8 workers, each test 3
|
||||
times (6132 tests), 24 minutes. A CI-sized run (`docker run --cpus=4`,
|
||||
@@ -189,14 +189,14 @@ check, so run it before pushing a new or deleted frontend file instead of
|
||||
waiting for the `frontend coverage` jobs.
|
||||
|
||||
The `viewerjs` check runs on every PR, including backend-only changes. It is an
|
||||
aggregate job. It requires the three `frontend coverage (shard N/3)` jobs and
|
||||
aggregate job. It requires the four `frontend coverage (shard N/4)` jobs and
|
||||
`frontend coverage (merge)` to succeed on every event. The four
|
||||
`page workflows (firefox|webkit, shard N/2)` jobs run only in the merge queue,
|
||||
on `main` and on `workflow_dispatch` (see "Pull requests and the merge queue").
|
||||
`viewerjs` requires them to be skipped on a pull request and to succeed
|
||||
everywhere else. Any other failed, cancelled or skipped job fails it. Every browser job runs with `PM_TEST_WORKERS: 3` (the public
|
||||
runners have 4 vCPUs). Each coverage shard runs `test:coverage:shard -- N/3`
|
||||
(shard 1 also runs the node units and the viewer suite, shard 3 the vendor pins
|
||||
runners have 4 vCPUs). Each coverage shard runs `test:coverage:shard -- N/4`
|
||||
(shard 1 also runs the node units and the viewer suite, shard 4 the vendor pins
|
||||
and tooling tests) and uploads its raw V8 files as `coverage-raw-N` (kept 3
|
||||
days). The merge job downloads them, fails unless all three arrived with raw
|
||||
files and without clashing names, then runs `coverage:merge` (report plus
|
||||
@@ -664,7 +664,7 @@ what runs:
|
||||
| `py3.13 on ubuntu-latest`, `py3.14 on ubuntu-latest` | not created | run |
|
||||
| `real ComfyUI compatibility` | skipped | runs |
|
||||
| `coverage gate (85%)`, `black and flake8`, `frontend modules`, `bandit (high severity)` | run | run |
|
||||
| `frontend coverage (shard N/3)`, `frontend coverage (merge)` (vendor pins and tooling tests on shard 3) | run | run |
|
||||
| `frontend coverage (shard N/4)`, `frontend coverage (merge)` (vendor pins and tooling tests on shard 4) | run | run |
|
||||
| `page workflows (firefox\|webkit, shard N/2)` | skipped | run |
|
||||
| `all tests`, `viewerjs` (gates) | run | run |
|
||||
| `perf budget` | skipped | push and dispatch only (not in the queue) |
|
||||
|
||||
@@ -156,8 +156,9 @@ test('at 390px the window fits the screen, actions stay reachable and the page s
|
||||
expect(frame.bottom-frame.top).toBeLessThan(844);
|
||||
expect(frame.bottom-frame.top).toBeGreaterThanOrEqual(280);
|
||||
expect(await page.evaluate(()=>document.documentElement.scrollWidth)).toBeLessThanOrEqual(390);
|
||||
// The last row's actions can be scrolled to inside the window and fit its width.
|
||||
const last=rows(page).last().getByRole('button',{name:'Rename / merge',exact:true});
|
||||
// The first page's last row's actions can be scrolled to inside the window and fit its width.
|
||||
// Pinned to row 100: reaching it loads the next page (D183), and last() would then follow row 200.
|
||||
const last=rows(page).nth(99).getByRole('button',{name:'Rename / merge',exact:true});
|
||||
await last.scrollIntoViewIfNeeded();await expect(last).toBeInViewport();
|
||||
const [button]=await boxes(last);const [after]=await boxes(tagWindow(page));
|
||||
expect(button.right).toBeLessThanOrEqual(after.right);
|
||||
|
||||
Vendored
+4
-4
@@ -129,11 +129,11 @@ test('Firefox/WebKit page workflows run only outside pull requests; coverage run
|
||||
}
|
||||
});
|
||||
|
||||
test('coverage runs as three shards that each upload their raw coverage', () => {
|
||||
test('coverage runs as four shards that each upload their raw coverage', () => {
|
||||
const coverage = allJobs.get('frontend-coverage');
|
||||
assert.deepEqual(matrixList(coverage, 'shard'), ['1', '2', '3']);
|
||||
assert.match(coverage, /^ {4}name: frontend coverage \(shard \$\{\{ matrix\.shard \}\}\/3\)$/m);
|
||||
assert.match(coverage, /npm run --prefix tools\/vendor test:coverage:shard -- \$\{\{ matrix\.shard \}\}\/3/);
|
||||
assert.deepEqual(matrixList(coverage, 'shard'), ['1', '2', '3', '4']);
|
||||
assert.match(coverage, /^ {4}name: frontend coverage \(shard \$\{\{ matrix\.shard \}\}\/4\)$/m);
|
||||
assert.match(coverage, /npm run --prefix tools\/vendor test:coverage:shard -- \$\{\{ matrix\.shard \}\}\/4/);
|
||||
const upload = steps(coverage).find(step => /name: coverage-raw-\$\{\{ matrix\.shard \}\}/.test(step));
|
||||
assert.ok(upload, 'each shard uploads coverage-raw-N');
|
||||
assert.match(upload, /^ {10}path: coverage\/js\/raw\/?$/m);
|
||||
|
||||
Reference in New Issue
Block a user