pydn-ComfyUI-to-Python-Extension/docs/specs/runtime-validation-implementation-spec.md

# Runtime Validation Implementation Spec

## Document Status

- Date: March 28, 2026
- Scope: implementation spec only
- Approval status: approved for spec creation, not yet approved for implementation

## Problem

The repo now has a runtime validation strategy, but not an implementation contract. Runtime-sensitive changes still rely too heavily on ad hoc local setups and narrow regression tests. The repo needs a small, repeatable runtime validation harness that agents and maintainers can run consistently.

## Intent

Implement a phase-based runtime validation harness and add a project-scoped `runtime-e2e-agent` that can operate against it.

## Locked Decisions

- Baseline pinned ComfyUI checkout: `/home/peyton/code/ComfyUI`
- Runtime workflow fixtures will be committed in this repo
- Implementation covers both CLI/runtime and `Save As Script`, but in phases
- Phase 1: CLI/runtime first
- Phase 2: `Save As Script` route validation

## Goals

- create a repeatable runtime validation harness
- keep the harness small enough to maintain manually
- make the workflow corpus reviewable and committed in-repo
- support pinned-runtime validation with a stable command surface
- give the future `runtime-e2e-agent` a stable command surface

## Non-Goals

- no full browser automation in phase 1
- no image-quality regression testing
- no external CI/CD setup in this phase
- no vendoring of the ComfyUI repo into this repo
- no attempt to validate every possible custom-node workflow

## Phase Plan

### Phase 1: CLI / Runtime Harness

Phase 1 must provide:

- in-repo runtime workflow fixtures
- a command path that exports workflow JSON to Python
- a command path that validates generated Python parses
- a command path that executes generated Python against a pinned ComfyUI runtime
- clear pass/fail classification

This phase should be enough to validate:

- runtime/bootstrap regressions like `#143`
- generated-code validity issues when they must be checked against a real runtime
- runtime/bootstrap changes that affect node-loading or script execution

### Phase 2: `Save As Script` Route Validation

Phase 2 must build on the phase 1 harness and add:

- route-level validation of the `Save As Script` request path
- coverage for server-side export behavior in [__init__.py](/home/peyton/code/ComfyUI-to-Python-Extension/__init__.py)
- coverage for the exported payload shape expected by the frontend flow

Phase 2 should not start with full browser automation unless route-level checks prove insufficient.

## Directory Layout

The first implementation should add:

- `tests/fixtures/runtime/`
- `tests/fixtures/runtime/upscale-model-loader.json`
- `tests/fixtures/runtime/unsafe-kwargs.json`
- `tests/fixtures/runtime/subgraph-identifiers.json`
- `tests/runtime/`
- a focused runtime validation runner or helper module inside `tests/runtime/`
- `.codex/agents/runtime-e2e-agent.toml`

Optional supporting docs:

- `docs/runtime-validation.md`

## Fixture Rules

- prefer real ComfyUI workflow shapes over synthetic-only fixtures
- keep fixtures as small as possible while still reproducing the target behavior
- name fixtures after behavior, not issue numbers, when practical
- commit fixtures to the repo so they are reviewable and stable

## Initial Fixture Set

### `upscale-model-loader.json`

Purpose:

- validate the `#143`/`#148` runtime/bootstrap path

Assertions:

- export succeeds
- generated Python parses
- generated script reaches the relevant loader and upscale path without the old missing-node failure

### `unsafe-kwargs.json`

Purpose:

- validate mixed safe and unsafe keys in a real workflow shape

Assertions:

- export succeeds
- generated Python parses
- generated code remains safe when real workflow data contains symbol-heavy or invalid keys

### `subgraph-identifiers.json`

Purpose:

- validate that subgraph-expanded identifiers do not break generated Python in real workflow shapes

Assertions:

- export succeeds
- generated Python parses
- generated variable names are safe enough for execution

## Command Contract

The harness should expose two stable command categories.

### `fast`

Purpose:

- exporter-only validation without a real ComfyUI runtime

Examples:

- `uv run python -m unittest tests.test_upscale_model_loader_export -v`
- additional focused exporter tests as they are added

### `runtime`

Purpose:

- run runtime validation against the pinned baseline ComfyUI checkout

Environment:

- `COMFYUI_PATH=/home/peyton/code/ComfyUI`

Behavior:

- export fixture workflow to Python
- parse or compile generated Python
- execute generated Python with the pinned runtime available

## Runner Shape

The implementation should prefer one small runner over many scattered commands.

Recommended shape:

- a small Python entry point that:
  - selects a tier
  - selects a fixture or fixture subset
  - sets or reads `COMFYUI_PATH`
  - runs export
  - validates generated Python
  - optionally executes the generated script
  - reports failure classification

The runner must stay small and repo-local. It should not introduce a large new framework.

## Failure Classification Contract

Every runtime validation result must classify failure as one of:

- repo regression
- fixture bug
- environment/setup failure

This classification is mandatory in both manual runs and future agent reports.

## Runtime-E2E Agent Addition

A future implementation step should add:

- `.codex/agents/runtime-e2e-agent.toml`

Its instructions should align with:

- [runtime-e2e-agent-spec.md](/home/peyton/code/ComfyUI-to-Python-Extension/docs/specs/runtime-e2e-agent-spec.md)
- [runtime-validation-environment-spec.md](/home/peyton/code/ComfyUI-to-Python-Extension/docs/specs/runtime-validation-environment-spec.md)

The agent should:

- run the `runtime` tier
- report concise results
- avoid long logs
- return `pass`, `pass with risk`, or `fail`

## Validation Expectations

Phase 1 success means:

- the pinned runtime can validate at least the three initial fixtures
- the runner can distinguish export failures from runtime execution failures
- the runtime harness can be invoked consistently by maintainers and agents

Phase 2 success means:

- route-level `Save As Script` validation exists
- frontend/server export behavior can be checked without relying on a full browser automation stack

## Risks

- fixture maintenance can drift if fixtures are too complex
- local ComfyUI checkout state can become stale if not periodically refreshed intentionally
- runtime validation may still fail for environment reasons if `torch` or ComfyUI dependencies are missing

## Acceptance Criteria

- the implementation plan is explicit enough to build without inventing workflow shape later
- fixture names and locations are defined
- command tiers are defined
- phase split is defined
- failure classification is mandatory
- the `runtime-e2e-agent` rollout target is defined

## Next Step

If this implementation spec is approved, the next step is to produce a concrete implementation plan and then request explicit approval before creating:

- runtime fixture files
- the runner/helper module
- the `.codex/agents/runtime-e2e-agent.toml` file
- the short operator doc