137 lines
4.3 KiB
Markdown
137 lines
4.3 KiB
Markdown
# Runtime Validation Environment Spec
|
|
|
|
## Purpose
|
|
|
|
Define the environment, fixture layout, and execution contract for end-to-end runtime validation of `ComfyUI-to-Python-Extension`.
|
|
|
|
## Problem
|
|
|
|
Exporter-only regression tests are necessary but insufficient for runtime/bootstrap-sensitive changes. Local verification becomes unreliable when tests depend on an arbitrary ComfyUI installation or on environment details that are not controlled. The repo needs a repeatable runtime validation harness.
|
|
|
|
## Goal
|
|
|
|
Create a stable runtime validation setup that:
|
|
|
|
- uses a pinned ComfyUI checkout
|
|
- validates generated scripts against a real runtime
|
|
- uses a small set of representative workflow fixtures
|
|
- clearly separates repo regressions from environment problems
|
|
|
|
## Non-Goals
|
|
|
|
- no image-quality benchmarking
|
|
- no full browser automation in the first phase
|
|
- no external CI/CD setup in this phase
|
|
- no attempt to cover every ComfyUI node or workflow shape
|
|
|
|
## Validation Model
|
|
|
|
### Tier 1: `fast`
|
|
|
|
Purpose:
|
|
|
|
- run quickly during local development
|
|
- validate exporter behavior without requiring a real ComfyUI runtime
|
|
|
|
Typical checks:
|
|
|
|
- focused unit or regression tests with injected mappings
|
|
- generated Python parse checks
|
|
- syntax compilation
|
|
|
|
### Tier 2: `runtime`
|
|
|
|
Purpose:
|
|
|
|
- validate generated scripts against a real pinned ComfyUI runtime
|
|
|
|
Typical checks:
|
|
|
|
- export from API-format workflow to Python
|
|
- parse/import generated Python
|
|
- execute generated script through the targeted runtime path
|
|
- confirm expected bootstrap and node-loading behavior
|
|
|
|
## Environment Requirements
|
|
|
|
- a local ComfyUI checkout available through a documented path convention
|
|
- the ability to point this repo to that checkout, preferably with `COMFYUI_PATH`
|
|
- required runtime dependencies installed in the validation environment, including `torch`
|
|
- all repo command execution through `uv`
|
|
|
|
## Proposed Path Convention
|
|
|
|
The validation harness should assume:
|
|
|
|
- this repo remains where it is
|
|
- the pinned ComfyUI checkout lives at a maintainer-chosen local path
|
|
- the runtime command surface uses `COMFYUI_PATH=/path/to/ComfyUI`
|
|
|
|
The exact checkout location does not need to be committed into this repo, but the contract for supplying it must be documented and stable.
|
|
|
|
## Workflow Fixture Corpus
|
|
|
|
The first committed or documented fixture set should cover:
|
|
|
|
- `upscale-model-loader`
|
|
for `#143`-style runtime/bootstrap validation
|
|
- `unsafe-kwargs`
|
|
for `#119` / `#96`-style generated-call validation against real workflows
|
|
- `subgraph-identifiers`
|
|
for `#128`-style identifier safety
|
|
|
|
Fixture principles:
|
|
|
|
- prefer real workflow shapes from official ComfyUI examples when possible
|
|
- keep fixtures minimal but representative
|
|
- name fixtures after behavior, not issue numbers, when practical
|
|
|
|
## Assertions
|
|
|
|
The first phase should assert:
|
|
|
|
- export succeeds
|
|
- generated Python parses
|
|
- generated script can execute far enough to validate the targeted runtime path
|
|
- expected code patterns are present or absent when relevant
|
|
|
|
The first phase should not require:
|
|
|
|
- image-diff assertions
|
|
- performance benchmarking beyond obvious bootstrap regressions
|
|
- full frontend automation
|
|
|
|
## Failure Classification
|
|
|
|
Every runtime validation failure should be classified as one of:
|
|
|
|
- repo regression
|
|
- fixture bug
|
|
- environment/setup failure
|
|
|
|
This classification is mandatory because raw pass/fail is not enough for a maintenance workflow.
|
|
|
|
## Likely Future Deliverables
|
|
|
|
When implementation is approved, the runtime validation system will likely need:
|
|
|
|
- a fixtures directory for runtime workflows
|
|
- a short operator document for setting `COMFYUI_PATH`
|
|
- one or more helper commands or scripts for running:
|
|
- `fast`
|
|
- `runtime`
|
|
- a `runtime-e2e-agent` config file under `.codex/agents/`
|
|
|
|
## Acceptance Criteria
|
|
|
|
- runtime validation is defined around a pinned ComfyUI target
|
|
- the initial workflow corpus covers the main known runtime-sensitive issue classes
|
|
- validation tiers are clearly separated
|
|
- pass/fail reporting includes failure classification
|
|
- the setup is small enough to maintain manually inside Codex/chat workflows
|
|
|
|
## Open Questions
|
|
|
|
- should the workflow corpus be committed into this repo or referenced from a local external fixture path?
|
|
- should the first implementation cover only CLI/runtime execution, or also a later `Save As Script` route check?
|