89 lines
4.6 KiB
Markdown
89 lines
4.6 KiB
Markdown
# Testing and merge confidence
|
|
|
|
Routine changes should merge after the required checks pass and the diff is
|
|
reviewed. Add a regression test for each bug; prove it fails before the fix.
|
|
|
|
## What runs
|
|
|
|
| Layer | Real components | Replaced boundaries |
|
|
| --- | --- | --- |
|
|
| Python tests | Domain logic, SQLite, API handlers, filesystem checks | ComfyUI host and ML runtimes where needed |
|
|
| JavaScript tests | Frontend modules and controller methods | DOM/network fixtures for focused edge cases |
|
|
| Viewer checks | Production Viewer.js and frontend methods in Chromium desktop/touch | Controlled images and page fixtures |
|
|
| Page workflows | Production HTML, startup, click handlers, HTTP routes, SQLite, PNG files, thumbnails | ComfyUI host services and the tagging model |
|
|
|
|
Page workflows cover save/edit/reload, duplicate-edit rejection, gallery image
|
|
identity, Tags selection and viewer cleanup, delayed responses, auto-tag review
|
|
and persistence, and infinite scrolling without replacing existing cards.
|
|
Data safety workflows download and upload actual SQLite backups, verify restored
|
|
prompts/tags/image links, reject corrupt uploads without changing live data, and
|
|
exercise cancellation and network-failure recovery for restores and deletion.
|
|
Duplicate cleanup verifies only the selected file disappears; prompt deletion
|
|
verifies the other prompt and original image files survive.
|
|
They run in Chromium, Firefox, WebKit, and Chromium touch emulation. No GPU,
|
|
model downloads, running ComfyUI instance, or personal configuration is required.
|
|
The server binds to localhost and creates disposable directories and a database;
|
|
its reset endpoint exists only in the test harness.
|
|
|
|
## Run locally
|
|
|
|
Use a Python environment containing `requirements-dev.txt` and Node.js 24:
|
|
|
|
```sh
|
|
python -m unittest discover -s tests -q
|
|
npm ci --prefix tools/vendor --ignore-scripts
|
|
npx --prefix tools/vendor playwright install --with-deps chromium firefox webkit
|
|
npm run --prefix tools/vendor test
|
|
npm run --prefix tools/vendor test:coverage
|
|
npm run --prefix tools/vendor test:workflows -- --project firefox --project webkit
|
|
```
|
|
|
|
`PM_TEST_PYTHON` selects the Python executable; `PM_TEST_PORT` selects the local
|
|
port (default 8765). Set these environment variables when Python is outside PATH
|
|
or the port is occupied. Test servers are never reused. On unsupported Linux
|
|
distributions, browser system dependencies may need manual installation; CI uses
|
|
Ubuntu. For a quick browser run, use `test:workflows -- --project chromium`.
|
|
|
|
## Coverage and failures
|
|
|
|
Python CI enforces 85% combined statement/branch coverage. Frontend coverage merges
|
|
Node V8 unit coverage with Chromium page-workflow coverage through c8. Every
|
|
`web/js/*.js` file appears, including untouched files at zero coverage. The
|
|
controller tests provide V8 filenames so their executed code is counted too.
|
|
|
|
`tools/vendor/frontend-coverage-baseline.json` holds a floor for every file and
|
|
metric. CI rejects regressions, missing measurements, and new files without a
|
|
reviewed baseline. These floors reflect current coverage, with two percentage
|
|
points of allowance for runtime/timing differences; they do not claim 80% coverage
|
|
of the large controllers. New files need at least 80% line coverage when added via
|
|
the baseline updater. Raise floors with `npm run --prefix tools/vendor
|
|
coverage:update`; it reruns the suites and cannot lower an existing floor.
|
|
Review baseline changes like code changes.
|
|
|
|
The `viewerjs` check runs on every PR, including backend-only changes. GitHub
|
|
artifacts named `frontend-test-reports` contain HTML coverage, a JSON summary,
|
|
and failure screenshots/traces. Open traces with Playwright's `show-trace` tool.
|
|
|
|
Live checks remain useful for ComfyUI host compatibility, real inference/GPU or
|
|
model dependency changes, and visual design changes. The automated fixtures do
|
|
not certify those environments or every existing product feature.
|
|
|
|
## Required merge checks
|
|
|
|
`.github/branch-protection.json` records the required checks: all nine Python/OS
|
|
combinations, Python coverage, formatting, frontend modules, and `viewerjs`.
|
|
The branch must be current with main, and administrators follow the same checks.
|
|
It does not require an additional human approval.
|
|
|
|
A repository administrator applies the configuration after the new workflow is
|
|
passing, then verifies it:
|
|
|
|
```sh
|
|
gh api --method PUT repos/ComfyAssets/ComfyUI_PromptManager/branches/main/protection \
|
|
--input .github/branch-protection.json
|
|
gh api repos/ComfyAssets/ComfyUI_PromptManager/branches/main/protection/required_status_checks
|
|
```
|
|
|
|
Update the manifest whenever required job names change. Keep the frontend check
|
|
free of path filters so a required check is never silently skipped.
|