[ci]: add transient pytest reruns for modal tests

This commit is contained in:
Mac Lee
2026-07-16 20:13:10 -07:00
committed by SolitaryThinker
parent e180bc832a
commit 9287c744b0
8 changed files with 200 additions and 7 deletions
+15 -1
View File
@@ -125,6 +125,18 @@ it), while a GitHub outage or a >25 min wait lets it run anyway (fail open).
See [Performance Benchmarks](performance_benchmarks.md) for the performance
lane's thresholds, rolling baseline, artifacts, and reseeding process.
### Modal Pytest Reruns
Modal pytest lanes install `pytest-rerunfailures` through the `test` extra.
`fastvideo/tests/modal/pr_test.py` applies the shared retry policy through
`PYTEST_ADDOPTS`, and `fastvideo/tests/modal/ssim_test.py` appends the same
arguments to each SSIM subprocess.
The policy reruns an individual pytest failure twice with a short delay, but
only when the failure text matches the transient infrastructure regex in
`fastvideo/tests/modal/pytest_retry.py`. Plain assertion and numerical parity
failures are not matched by that policy and should fail without rerun.
## Slash Commands
Slash commands are handled by `.github/workflows/ci-slash-commands.yml`.
@@ -239,7 +251,9 @@ All Buildkite test jobs go through `.buildkite/scripts/pr_test.sh`, which:
3. Passes Buildkite metadata into the Modal container.
4. Runs the selected test command from `fastvideo/tests/modal/pr_test.py` or
`fastvideo/tests/modal/ssim_test.py`.
5. Uploads performance artifacts for `TEST_TYPE=performance`.
5. Applies the shared transient-only pytest rerun policy to Modal pytest
commands.
6. Uploads performance artifacts for `TEST_TYPE=performance`.
If you add a new CI test category:
+6
View File
@@ -219,6 +219,12 @@ instances. The main files are:
For exact tier membership, path filters, slash commands, and aggregate statuses,
see [CI/CD Architecture](ci_architecture.md).
Modal pytest commands use `pytest-rerunfailures` with the shared transient-only
policy in `fastvideo/tests/modal/pytest_retry.py`. The retry layer reruns
individual pytest failures for infrastructure-style errors such as connection,
Modal worker, NCCL, or CUDA initialization failures. Deterministic assertion
and numerical parity failures should still fail immediately.
### Adding A New CI Test Category
If a new test does not fit an existing lane: