pydn-ComfyUI-to-Python-Extension/docs/specs/runtime-validation-environment-spec.md

Runtime Validation Environment Spec

Purpose

Define the environment, fixture layout, and execution contract for end-to-end runtime validation of ComfyUI-to-Python-Extension.

Problem

Exporter-only regression tests are necessary but insufficient for runtime/bootstrap-sensitive changes. Local verification becomes unreliable when tests depend on an arbitrary ComfyUI installation or on environment details that are not controlled. The repo needs a repeatable runtime validation harness.

Goal

Create a stable runtime validation setup that:

• uses a pinned ComfyUI checkout
• validates generated scripts against a real runtime
• uses a small set of representative workflow fixtures
• clearly separates repo regressions from environment problems

Non-Goals

• no image-quality benchmarking
• no full browser automation in the first phase
• no external CI/CD setup in this phase
• no attempt to cover every ComfyUI node or workflow shape

Validation Model

Tier 1: fast

Purpose:

• run quickly during local development
• validate exporter behavior without requiring a real ComfyUI runtime

Typical checks:

• focused unit or regression tests with injected mappings
• generated Python parse checks
• syntax compilation

Tier 2: runtime

Purpose:

• validate generated scripts against a real pinned ComfyUI runtime

Typical checks:

• export from API-format workflow to Python
• parse/import generated Python
• execute generated script through the targeted runtime path
• confirm expected bootstrap and node-loading behavior

Environment Requirements

• a local ComfyUI checkout available through a documented path convention
• the ability to point this repo to that checkout, preferably with COMFYUI_PATH
• required runtime dependencies installed in the validation environment, including torch
• all repo command execution through uv

Proposed Path Convention

The validation harness should assume:

• this repo remains where it is
• the pinned ComfyUI checkout lives at a maintainer-chosen local path
• the runtime command surface uses COMFYUI_PATH=/path/to/ComfyUI

The exact checkout location does not need to be committed into this repo, but the contract for supplying it must be documented and stable.

Workflow Fixture Corpus

The first committed or documented fixture set should cover:

• upscale-model-loader for #143-style runtime/bootstrap validation
• unsafe-kwargs for #119 / #96-style generated-call validation against real workflows
• subgraph-identifiers for #128-style identifier safety

Fixture principles:

• prefer real workflow shapes from official ComfyUI examples when possible
• keep fixtures minimal but representative
• name fixtures after behavior, not issue numbers, when practical

Assertions

The first phase should assert:

• export succeeds
• generated Python parses
• generated script can execute far enough to validate the targeted runtime path
• expected code patterns are present or absent when relevant

The first phase should not require:

• image-diff assertions
• performance benchmarking beyond obvious bootstrap regressions
• full frontend automation

Failure Classification

Every runtime validation failure should be classified as one of:

• repo regression
• fixture bug
• environment/setup failure

This classification is mandatory because raw pass/fail is not enough for a maintenance workflow.

Likely Future Deliverables

When implementation is approved, the runtime validation system will likely need:

• a fixtures directory for runtime workflows
• a short operator document for setting COMFYUI_PATH
• one or more helper commands or scripts for running:
  • fast
  • runtime
• a runtime-e2e-agent config file under .codex/agents/

Acceptance Criteria

• runtime validation is defined around a pinned ComfyUI target
• the initial workflow corpus covers the main known runtime-sensitive issue classes
• validation tiers are clearly separated
• pass/fail reporting includes failure classification
• the setup is small enough to maintain manually inside Codex/chat workflows

Open Questions

• should the workflow corpus be committed into this repo or referenced from a local external fixture path?
• should the first implementation cover only CLI/runtime execution, or also a later Save As Script route check?