Vito
3ebd5e4166
* ci(browser): stop a hung browser run before the job limit and name it A browser job killed at its 20-minute limit keeps no log and skips its upload steps (merge-queue run 40723857746, firefox shard 2/2), so nothing showed which test hung. - Each browser test step gets timeout-minutes 18, below the job's 20, so the step fails and the always() uploads still run. - Under CI, Playwright's globalTimeout stops the run first: PM_TEST_GLOBAL_TIMEOUT_MINUTES 17 for page workflows, 15 for coverage workflows, a fixed 3 for the viewer suite. Locally there is no limit unless the variable is set. Each test keeps the 30 s default. - running-tests-reporter.mjs prints the tests still running when the run ends; Playwright's own reporters cannot, since those tests never end. - A blob report goes to tools/vendor/test-results/blob/, inside the uploaded traces path. - ci-workflow.test.mjs pins globalTimeout < step limit < job limit and the always() uploads; no existing limit is raised. Closes #425 * ci(browser): rely on the step limit, report the running tests on SIGTERM Maintainer ruling on #425: no Playwright globalTimeout. The 18-minute step limit under the unchanged 20-minute job limit stays, as do the always() uploads and the CI blob report. - Drop globalTimeout and PM_TEST_GLOBAL_TIMEOUT_MINUTES with its helper and tests. - A step that reaches its limit gets SIGTERM, which ended Playwright with no report and no blob. The running-tests reporter now catches the first SIGTERM, prints the tests still running and sends Playwright its own SIGINT, so the run ends interrupted and the blob report is written. A second SIGTERM kills as usual. - Test paths are relative to the suite's rootDir instead of the cwd, so `npm run --prefix` no longer prints ../../ prefixes. - The workflow comment and TESTING.md say the guarantee assumes an npm cache hit and that coverage shard 1 is tight in its 18-minute step. * ci(browser): forward SIGTERM and SIGINT from frontend-coverage.mjs to its child In the coverage job npm signals only frontend-coverage.mjs. It blocked in spawnSync with no listener, so Playwright never saw a step timeout. It now runs each step's child with async spawn, in its own process group, and forwards SIGTERM and SIGINT to the running child. Step order, exit codes and the "Command failed (N)" message are unchanged. Locally, in the pinned image, `timeout -s TERM` on test:coverage printed the still-running tests and wrote both blob reports. TESTING.md records that a SIGTERM to npm alone stops at npm's `sh -c`, and that delivery through `docker exec` in CI is not yet observed. Refs #425 * ci(browser): load the running-tests reporter by absolute path A config imported from another directory (the workflow-artifacts test copies it to a temp root) resolved './running-tests-reporter.mjs' against that root under CI and failed to load it. Resolve it from this module instead.
260 lines
13 KiB
YAML
260 lines
13 KiB
YAML
name: Viewer.js vendor and compatibility
|
|
|
|
on:
|
|
merge_group:
|
|
pull_request:
|
|
branches: [main]
|
|
push:
|
|
branches: [main]
|
|
workflow_dispatch:
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
# Superseded PR pushes cancel their older run. Every main push gets its own
|
|
# group (run_id), so no main commit's run is ever cancelled or dropped as pending.
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.ref || github.run_id }}
|
|
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
|
|
|
# Timeouts are the measured job time plus at least 50%, and never under 20 minutes for a browser job,
|
|
# because runner speed varies. Before sharding (main 2cecfcf, one worker): frontend coverage 32m10s,
|
|
# page workflows firefox 18m59s, webkit 19m26s. First sharded run (PR #274 run 39712909047, 3 workers,
|
|
# no npm cache, where npm ci stalled for 5m in six of eight jobs): coverage shards 11m19s/10m47s/10m49s
|
|
# (tests 5m20s/4m42s/4m46s), merge 5m52s (report and check 4s), firefox shards 7m53s/9m27s (tests
|
|
# 4m41s/3m29s), webkit shards 12m25s+ (cancelled at the old 12-minute limit after its 267 tests passed in
|
|
# 6m29s) and 4m53s (tests 3m08s). With the npm cache hit (run 39725176320 at cbc8a61, npm ci 0-2s):
|
|
# coverage shards 7m34s/6m04s/5m22s (tests 6m38s/4m53s/4m04s), merge 54s, firefox 7m57s/5m21s (tests
|
|
# 7m08s/4m34s), webkit 8m16s/5m03s (tests 7m26s/3m58s); a cache miss adds up to 5m to npm ci, which
|
|
# the 20-minute limits absorb.
|
|
# Hangs (#425): a job killed at its limit keeps no log and skips its upload steps, so each browser
|
|
# test step has its own timeout-minutes (18) below the job's 20. At the step limit, SIGTERM makes
|
|
# tools/vendor/running-tests-reporter.mjs print the tests still running and hand over to Playwright's
|
|
# interrupt, which writes the blob report; the always() uploads then run.
|
|
# Measured on main (runs 40725289094, 40719375442, 40700439399): setup before the test step <= 52s,
|
|
# uploads <= 35s; test steps up to 14m42s (webkit shard 1/2) and 13m05s for coverage (shard 1/3:
|
|
# 12m53s, node units and the viewer suite before 11.8m of workflows), so coverage is the tight one
|
|
# in its 18 minutes. This holds only when the npm cache hits: a cold cache adds up to 5m to npm ci,
|
|
# and then the job limit can still end the job before the step limit does.
|
|
jobs:
|
|
frontend-coverage:
|
|
name: frontend coverage (shard ${{ matrix.shard }}/3)
|
|
runs-on: ubuntu-latest
|
|
# Layout comparisons depend on the fonts each browser falls back to, so browser jobs run in the
|
|
# same pinned Playwright image as local runs (tag = @playwright/test in tools/vendor/package-lock.json;
|
|
# tests/vendor/browser-image.test.mjs checks it). Browsers and fonts come from the image.
|
|
container:
|
|
image: mcr.microsoft.com/playwright:v1.64.1-noble
|
|
options: --user 1001
|
|
timeout-minutes: 20
|
|
# Public-repo ubuntu-latest runners have 4 vCPUs: three Playwright workers, each with its own test
|
|
# server, leave one CPU for the servers and the runner.
|
|
env:
|
|
PM_TEST_WORKERS: 3
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
shard: [1, 2, 3]
|
|
steps:
|
|
- uses: actions/checkout@v7
|
|
# The image's own Python 3.12 and Node 24: setup-python/setup-node install the runner's toolcache
|
|
# builds, whose entry points name host paths that do not exist in the container. The image has no
|
|
# ensurepip and the job has no root, so a checksum-pinned pip wheel bootstraps a user-space venv.
|
|
- name: Python environment
|
|
env:
|
|
PIP_WHEEL: https://files.pythonhosted.org/packages/f3/6e/1736e5b4ae2b778ef2f81c47d797de9f891d4d8acb047a24ca37a60294dd/pip-26.2.1-py3-none-any.whl
|
|
PIP_SHA256: 71138adf1f4ca900cdb7d289c21b7494329f2332b6d85f0e1c42108c0384ed3e
|
|
run: |
|
|
venv="$RUNNER_TEMP/venv" wheel="$RUNNER_TEMP/pip-26.2.1-py3-none-any.whl"
|
|
python3 -m venv --without-pip "$venv"
|
|
curl -fsSL --retry 3 -o "$wheel" "$PIP_WHEEL"
|
|
echo "$PIP_SHA256 $wheel" | sha256sum -c -
|
|
"$venv/bin/python" "$wheel/pip" install --no-index --no-cache-dir "$wheel"
|
|
"$venv/bin/python" -m pip install --no-cache-dir -r requirements-dev.txt
|
|
echo "$venv/bin" >> "$GITHUB_PATH"
|
|
- run: python --version && node --version && node --version | grep -q '^v24\.'
|
|
# npm's download cache, keyed on the lockfile; inside the container ~ is the job's HOME (/github/home),
|
|
# which npm also uses. node_modules itself is still built fresh by npm ci.
|
|
- name: Cache npm downloads
|
|
uses: actions/cache@v6
|
|
with:
|
|
path: ~/.npm
|
|
key: npm-${{ runner.os }}-playwright-v1.64.1-${{ hashFiles('tools/vendor/package-lock.json') }}
|
|
- run: npm ci --prefix tools/vendor --ignore-scripts
|
|
# Shard 1 also runs the node units and the viewer suite, so the vendor pins and the tooling tests
|
|
# run on the lightest shard instead.
|
|
- if: matrix.shard == 3
|
|
run: npm run --prefix tools/vendor check
|
|
- if: matrix.shard == 3
|
|
run: npm run --prefix tools/vendor test
|
|
- name: Page workflows with coverage, shard ${{ matrix.shard }}/3
|
|
timeout-minutes: 18
|
|
run: npm run --prefix tools/vendor test:coverage:shard -- ${{ matrix.shard }}/3
|
|
- name: Upload raw coverage
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: coverage-raw-${{ matrix.shard }}
|
|
path: coverage/js/raw/
|
|
if-no-files-found: error
|
|
retention-days: 3
|
|
# Traces and the blob report (test-results/blob/) of a failed, timed-out or cancelled step.
|
|
- name: Upload Chromium and touch traces
|
|
if: always()
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: frontend-traces-shard-${{ matrix.shard }}
|
|
path: tools/vendor/test-results/
|
|
retention-days: 14
|
|
|
|
frontend-coverage-merge:
|
|
name: frontend coverage (merge)
|
|
needs: frontend-coverage
|
|
runs-on: ubuntu-latest
|
|
# Same image as the shards: the raw files name sources by the container's workspace path.
|
|
container:
|
|
image: mcr.microsoft.com/playwright:v1.64.1-noble
|
|
options: --user 1001
|
|
timeout-minutes: 10
|
|
steps:
|
|
- uses: actions/checkout@v7
|
|
- run: node --version && node --version | grep -q '^v24\.'
|
|
# npm's download cache, keyed on the lockfile; inside the container ~ is the job's HOME (/github/home),
|
|
# which npm also uses. node_modules itself is still built fresh by npm ci.
|
|
- name: Cache npm downloads
|
|
uses: actions/cache@v6
|
|
with:
|
|
path: ~/.npm
|
|
key: npm-${{ runner.os }}-playwright-v1.64.1-${{ hashFiles('tools/vendor/package-lock.json') }}
|
|
- run: npm ci --prefix tools/vendor --ignore-scripts
|
|
# Each artifact lands in its own coverage-raw-N directory, so the next step can count them.
|
|
- uses: actions/download-artifact@v8
|
|
with:
|
|
pattern: coverage-raw-*
|
|
path: ${{ runner.temp }}/coverage-raw
|
|
# coverage:merge only checks that raw files exist; merging two of three shards would report
|
|
# partial coverage, so a missing or empty shard artifact fails here.
|
|
- name: Verify every shard's raw coverage arrived
|
|
shell: bash
|
|
env:
|
|
STAGED_RAW: ${{ runner.temp }}/coverage-raw
|
|
EXPECTED_SHARDS: 3
|
|
run: |
|
|
found=0
|
|
for shard in $(seq 1 "$EXPECTED_SHARDS"); do
|
|
directory="$STAGED_RAW/coverage-raw-$shard"
|
|
count=0
|
|
if [ -d "$directory" ]; then count=$(find "$directory" -maxdepth 1 -type f -name '*.json' | wc -l); fi
|
|
echo "coverage-raw-$shard: $count raw files"
|
|
if [ "$count" -gt 0 ]; then found=$((found + 1)); fi
|
|
done
|
|
echo "shard artifacts with raw coverage: $found of $EXPECTED_SHARDS"
|
|
if [ "$found" -ne "$EXPECTED_SHARDS" ]; then
|
|
echo "::error::Expected raw coverage from $EXPECTED_SHARDS shards, got $found of $EXPECTED_SHARDS"
|
|
exit 1
|
|
fi
|
|
staged=$(find "$STAGED_RAW" -mindepth 2 -maxdepth 2 -type f -name '*.json' | wc -l)
|
|
mkdir -p coverage/js/raw
|
|
# cp -n never overwrites; a clash shows up as fewer merged files than staged ones.
|
|
find "$STAGED_RAW" -mindepth 2 -maxdepth 2 -type f -name '*.json' -exec cp -n -t coverage/js/raw {} + || true
|
|
merged=$(find coverage/js/raw -maxdepth 1 -type f -name '*.json' | wc -l)
|
|
if [ "$merged" -ne "$staged" ]; then
|
|
echo "::error::Shards wrote clashing raw file names: $staged staged, $merged merged"
|
|
exit 1
|
|
fi
|
|
echo "merged $merged raw files"
|
|
- name: Frontend coverage report and baseline check
|
|
run: npm run --prefix tools/vendor coverage:merge
|
|
- name: Upload frontend coverage report
|
|
if: always()
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: frontend-test-reports
|
|
path: |
|
|
coverage/js/
|
|
!coverage/js/raw/**
|
|
retention-days: 14
|
|
|
|
browser-workflows:
|
|
name: page workflows (${{ matrix.browser }}, shard ${{ matrix.shard }}/2)
|
|
# Merge queue, main and manual runs only (#420); pull requests run the Chromium/touch coverage shards.
|
|
if: github.event_name != 'pull_request'
|
|
runs-on: ubuntu-latest
|
|
# Layout comparisons depend on the fonts each browser falls back to, so browser jobs run in the
|
|
# same pinned Playwright image as local runs (tag = @playwright/test in tools/vendor/package-lock.json;
|
|
# tests/vendor/browser-image.test.mjs checks it). Browsers and fonts come from the image.
|
|
container:
|
|
image: mcr.microsoft.com/playwright:v1.64.1-noble
|
|
options: --user 1001
|
|
timeout-minutes: 20
|
|
# Public-repo ubuntu-latest runners have 4 vCPUs: three Playwright workers, each with its own test
|
|
# server, leave one CPU for the servers and the runner.
|
|
env:
|
|
PM_TEST_WORKERS: 3
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
browser: [firefox, webkit]
|
|
shard: [1, 2]
|
|
steps:
|
|
- uses: actions/checkout@v7
|
|
# The image's own Python 3.12 and Node 24: setup-python/setup-node install the runner's toolcache
|
|
# builds, whose entry points name host paths that do not exist in the container. The image has no
|
|
# ensurepip and the job has no root, so a checksum-pinned pip wheel bootstraps a user-space venv.
|
|
- name: Python environment
|
|
env:
|
|
PIP_WHEEL: https://files.pythonhosted.org/packages/f3/6e/1736e5b4ae2b778ef2f81c47d797de9f891d4d8acb047a24ca37a60294dd/pip-26.2.1-py3-none-any.whl
|
|
PIP_SHA256: 71138adf1f4ca900cdb7d289c21b7494329f2332b6d85f0e1c42108c0384ed3e
|
|
run: |
|
|
venv="$RUNNER_TEMP/venv" wheel="$RUNNER_TEMP/pip-26.2.1-py3-none-any.whl"
|
|
python3 -m venv --without-pip "$venv"
|
|
curl -fsSL --retry 3 -o "$wheel" "$PIP_WHEEL"
|
|
echo "$PIP_SHA256 $wheel" | sha256sum -c -
|
|
"$venv/bin/python" "$wheel/pip" install --no-index --no-cache-dir "$wheel"
|
|
"$venv/bin/python" -m pip install --no-cache-dir -r requirements-dev.txt
|
|
echo "$venv/bin" >> "$GITHUB_PATH"
|
|
- run: python --version && node --version && node --version | grep -q '^v24\.'
|
|
# npm's download cache, keyed on the lockfile; inside the container ~ is the job's HOME (/github/home),
|
|
# which npm also uses. node_modules itself is still built fresh by npm ci.
|
|
- name: Cache npm downloads
|
|
uses: actions/cache@v6
|
|
with:
|
|
path: ~/.npm
|
|
key: npm-${{ runner.os }}-playwright-v1.64.1-${{ hashFiles('tools/vendor/package-lock.json') }}
|
|
- run: npm ci --prefix tools/vendor --ignore-scripts
|
|
- name: Page workflows in ${{ matrix.browser }}, shard ${{ matrix.shard }}/2
|
|
timeout-minutes: 18
|
|
run: npm run --prefix tools/vendor test:workflows -- --project ${{ matrix.browser }} --shard ${{ matrix.shard }}/2
|
|
# Traces and the blob report (test-results/blob/) of a failed, timed-out or cancelled step.
|
|
- name: Upload browser traces
|
|
if: always()
|
|
uses: actions/upload-artifact@v7
|
|
with:
|
|
name: browser-traces-${{ matrix.browser }}-shard-${{ matrix.shard }}
|
|
path: tools/vendor/test-results/
|
|
retention-days: 14
|
|
|
|
# Branch protection requires the exact context "viewerjs". This job keeps that
|
|
# name and fails unless every coverage shard and the coverage merge succeeded and
|
|
# the browser shards succeeded, or, on a pull request only, were skipped (#420).
|
|
# Any failure, cancellation or other skip fails it.
|
|
viewerjs:
|
|
needs: [frontend-coverage, frontend-coverage-merge, browser-workflows]
|
|
if: always()
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 5
|
|
steps:
|
|
- name: Require every frontend job to succeed
|
|
env:
|
|
EVENT_NAME: ${{ github.event_name }}
|
|
COVERAGE_RESULT: ${{ needs.frontend-coverage.result }}
|
|
MERGE_RESULT: ${{ needs.frontend-coverage-merge.result }}
|
|
WORKFLOWS_RESULT: ${{ needs.browser-workflows.result }}
|
|
run: |
|
|
queue_only=success
|
|
if [ "$EVENT_NAME" = pull_request ]; then queue_only=skipped; fi
|
|
echo "event: $EVENT_NAME (browser-workflows must be $queue_only)"
|
|
echo "frontend-coverage: $COVERAGE_RESULT"
|
|
echo "frontend-coverage-merge: $MERGE_RESULT"
|
|
echo "browser-workflows: $WORKFLOWS_RESULT"
|
|
test "$COVERAGE_RESULT" = success && test "$MERGE_RESULT" = success && test "$WORKFLOWS_RESULT" = "$queue_only"
|