100 lines
2.7 KiB
Markdown
100 lines
2.7 KiB
Markdown
# Runtime Validation
|
|
|
|
This repo has two validation tiers:
|
|
|
|
- `fast`
|
|
- `runtime`
|
|
|
|
Use `uv` for repo commands.
|
|
|
|
## Local Runtime Paths
|
|
|
|
Pinned ComfyUI checkout:
|
|
|
|
- `/home/peyton/code/ComfyUI`
|
|
|
|
The runtime harness uses `COMFYUI_PATH` to point generated scripts and export logic at the target ComfyUI checkout.
|
|
For full E2E runtime execution, the harness prefers the ComfyUI checkout's own Python interpreter at `/home/peyton/code/ComfyUI/.venv/bin/python` when available.
|
|
|
|
## Fixtures
|
|
|
|
Committed runtime fixtures live in:
|
|
|
|
- [tests/fixtures/runtime](/home/peyton/code/ComfyUI-to-Python-Extension/tests/fixtures/runtime)
|
|
|
|
Current fixtures:
|
|
|
|
- `upscale-model-loader`
|
|
- `text-to-image`
|
|
- `unsafe-kwargs`
|
|
- `subgraph-identifiers`
|
|
|
|
Runtime-tier fixtures are based on official core ComfyUI workflow shapes:
|
|
|
|
- the Image Upscale workflow
|
|
- the Text to Image workflow
|
|
|
|
Sources:
|
|
|
|
- https://docs.comfy.org/tutorials/basic/upscale
|
|
- https://docs.comfy.org/tutorials/basic/text-to-image
|
|
|
|
## Commands
|
|
|
|
### Fast
|
|
|
|
Run exporter-only validation without a real ComfyUI runtime:
|
|
|
|
```bash
|
|
uv run python tests/runtime/run_runtime_validation.py --tier fast --fixture all
|
|
```
|
|
|
|
### Runtime
|
|
|
|
Run runtime-capable validation against the pinned baseline checkout. This now performs full E2E execution by default:
|
|
|
|
```bash
|
|
uv run python tests/runtime/run_runtime_validation.py --tier runtime --fixture upscale-model-loader
|
|
```
|
|
|
|
Run both official runtime fixtures:
|
|
|
|
```bash
|
|
uv run python tests/runtime/run_runtime_validation.py --tier runtime --fixture all
|
|
```
|
|
|
|
### Model Provisioning
|
|
|
|
If a runtime fixture is missing required models, the harness will report `model provisioning failure`.
|
|
|
|
To print the required download commands without executing them:
|
|
|
|
```bash
|
|
uv run python tests/runtime/run_runtime_validation.py --tier runtime --fixture all --print-download-plan
|
|
```
|
|
|
|
Model downloads should be approval-gated and should target the existing `/home/peyton/code/ComfyUI/models/...` directories.
|
|
|
|
## Failure Classification
|
|
|
|
The runtime harness classifies failures as:
|
|
|
|
- `repo regression`
|
|
- `model provisioning failure`
|
|
- `fixture bug`
|
|
- `environment/setup failure`
|
|
|
|
This classification matters more than raw pass/fail. A failed run without classification is not actionable enough for maintenance work.
|
|
|
|
## Notes
|
|
|
|
- `unsafe-kwargs` and `subgraph-identifiers` are committed fast-tier regression fixtures. They are not runtime-tier fixtures.
|
|
- `upscale-model-loader` and `text-to-image` are the current runtime-tier fixtures and are based on official ComfyUI tutorials.
|
|
- Runtime-tier success now means:
|
|
- generated Python parses
|
|
- generated script executes
|
|
- a new output file is created
|
|
- output dimensions meet expectations
|
|
- output metadata contains expected markers
|
|
- `Save As Script` route validation is still phase 2 work.
|