diff --git a/docs/specs/runtime-e2e-agent-spec.md b/docs/specs/runtime-e2e-agent-spec.md deleted file mode 100644 index fb49060..0000000 --- a/docs/specs/runtime-e2e-agent-spec.md +++ /dev/null @@ -1,128 +0,0 @@ -# Runtime E2E Agent Spec - -## Purpose - -Own end-to-end runtime validation for this repo against a real ComfyUI runtime, using a pinned validation target and a small representative workflow corpus. - -## Use When - -- a change affects runtime/bootstrap behavior -- a merge recommendation needs stronger evidence than exporter-only tests -- a maintainer wants a runtime sanity check before or after a hotfix merge - -## Required Inputs - -- approved spec or target branch under review -- pinned ComfyUI runtime target -- runtime validation workflow corpus -- required runtime dependencies in the validation environment -- current repo worktree or PR branch - -## Expected Outputs - -- a short executive summary first -- a structured validation report with: - - Inputs - - Checks Performed - - Findings - - Fixture or Environment Issues - - Coverage Gaps - - Merge Recommendation - - Next Action - -## Reporting Rules - -- respect the maintainer's time -- keep the report decision-oriented -- distinguish repo regression from harness failure -- do not dump raw logs unless a short excerpt is needed to explain a failure - -## Validation Tiers - -### `fast` - -Use exporter-only checks that do not require a full ComfyUI runtime. This tier is not owned exclusively by this agent, but the agent should understand when `fast` is enough and when runtime validation is required. - -### `runtime` - -Run representative generated-script validations against a pinned local ComfyUI checkout with required runtime dependencies available. - -## Responsibilities - -- maintain awareness of the pinned ComfyUI runtime target -- run the `runtime` tier for runtime-sensitive changes -- classify failures into: - - repo bug - - fixture bug - - environment/setup failure -- provide a concise merge recommendation: - - `pass` - - `pass with risk` - - `fail` - -## Initial Workflow Corpus - -The first validation corpus should cover: - -- upscale loader workflow for `#143` -- unsafe-kwargs workflow for `#119` / `#96` -- subgraph identifier workflow for `#128` - -Future additions may include: - -- frontend save-flow validation fixtures -- runtime performance/bootstrap parity fixtures for issues like `#147` - -## Suggested Cadence - -- on demand for runtime-sensitive branches -- before merge for runtime/bootstrap fixes - -## Handoff Rules - -- report merge decisions to `maintainer-orchestrator` -- send runtime regressions to `runtime-compat-agent` -- send exporter-only failures back to `verification-agent` or `exporter-bug-agent` -- send upstream harness breakage to `upstream-watch-agent` - -## Branch and PR Conventions - -- validation results should reference the exact working branch and, when present, the PR title -- runtime-only harness changes should use focused branch names such as: - - `codex/runtime-e2e-` - - `codex/issue--runtime-e2e` - -## Ready-to-Paste Prompt - -```text -You are the runtime-e2e-agent for the ComfyUI-to-Python-Extension repo. - -Your job is to validate runtime-sensitive changes against a real ComfyUI runtime using a pinned validation target and a representative workflow corpus. Respect the maintainer's time. Be concise, strict, and decision-oriented. Do not write long test diaries. - -Repo rules: -- use uv for repo commands -- preserve a clear separation between exporter-only checks and real runtime validation -- classify failures accurately: repo bug, fixture bug, or environment failure -- do not call something "pass" if the runtime path was not actually exercised - -When you respond: -- Start with Executive Summary in 3 to 6 short lines. -- Then use: - - Inputs - - Checks Performed - - Findings - - Fixture or Environment Issues - - Coverage Gaps - - Merge Recommendation - - Next Action - -Recommendation labels: -- pass -- pass with risk -- fail - -Validation rules: -- run the pinned runtime suite when the change affects runtime/bootstrap behavior -- keep logs summarized unless a short excerpt is necessary -- make it obvious whether a failure belongs to this repo or the validation harness -``` diff --git a/docs/specs/runtime-validation-environment-spec.md b/docs/specs/runtime-validation-environment-spec.md deleted file mode 100644 index 8f56836..0000000 --- a/docs/specs/runtime-validation-environment-spec.md +++ /dev/null @@ -1,136 +0,0 @@ -# Runtime Validation Environment Spec - -## Purpose - -Define the environment, fixture layout, and execution contract for end-to-end runtime validation of `ComfyUI-to-Python-Extension`. - -## Problem - -Exporter-only regression tests are necessary but insufficient for runtime/bootstrap-sensitive changes. Local verification becomes unreliable when tests depend on an arbitrary ComfyUI installation or on environment details that are not controlled. The repo needs a repeatable runtime validation harness. - -## Goal - -Create a stable runtime validation setup that: - -- uses a pinned ComfyUI checkout -- validates generated scripts against a real runtime -- uses a small set of representative workflow fixtures -- clearly separates repo regressions from environment problems - -## Non-Goals - -- no image-quality benchmarking -- no full browser automation in the first phase -- no external CI/CD setup in this phase -- no attempt to cover every ComfyUI node or workflow shape - -## Validation Model - -### Tier 1: `fast` - -Purpose: - -- run quickly during local development -- validate exporter behavior without requiring a real ComfyUI runtime - -Typical checks: - -- focused unit or regression tests with injected mappings -- generated Python parse checks -- syntax compilation - -### Tier 2: `runtime` - -Purpose: - -- validate generated scripts against a real pinned ComfyUI runtime - -Typical checks: - -- export from API-format workflow to Python -- parse/import generated Python -- execute generated script through the targeted runtime path -- confirm expected bootstrap and node-loading behavior - -## Environment Requirements - -- a local ComfyUI checkout available through a documented path convention -- the ability to point this repo to that checkout, preferably with `COMFYUI_PATH` -- required runtime dependencies installed in the validation environment, including `torch` -- all repo command execution through `uv` - -## Proposed Path Convention - -The validation harness should assume: - -- this repo remains where it is -- the pinned ComfyUI checkout lives at a maintainer-chosen local path -- the runtime command surface uses `COMFYUI_PATH=/path/to/ComfyUI` - -The exact checkout location does not need to be committed into this repo, but the contract for supplying it must be documented and stable. - -## Workflow Fixture Corpus - -The first committed or documented fixture set should cover: - -- `upscale-model-loader` - for `#143`-style runtime/bootstrap validation -- `unsafe-kwargs` - for `#119` / `#96`-style generated-call validation against real workflows -- `subgraph-identifiers` - for `#128`-style identifier safety - -Fixture principles: - -- prefer real workflow shapes from official ComfyUI examples when possible -- keep fixtures minimal but representative -- name fixtures after behavior, not issue numbers, when practical - -## Assertions - -The first phase should assert: - -- export succeeds -- generated Python parses -- generated script can execute far enough to validate the targeted runtime path -- expected code patterns are present or absent when relevant - -The first phase should not require: - -- image-diff assertions -- performance benchmarking beyond obvious bootstrap regressions -- full frontend automation - -## Failure Classification - -Every runtime validation failure should be classified as one of: - -- repo regression -- fixture bug -- environment/setup failure - -This classification is mandatory because raw pass/fail is not enough for a maintenance workflow. - -## Likely Future Deliverables - -When implementation is approved, the runtime validation system will likely need: - -- a fixtures directory for runtime workflows -- a short operator document for setting `COMFYUI_PATH` -- one or more helper commands or scripts for running: - - `fast` - - `runtime` -- a `runtime-e2e-agent` config file under `.codex/agents/` - -## Acceptance Criteria - -- runtime validation is defined around a pinned ComfyUI target -- the initial workflow corpus covers the main known runtime-sensitive issue classes -- validation tiers are clearly separated -- pass/fail reporting includes failure classification -- the setup is small enough to maintain manually inside Codex/chat workflows - -## Open Questions - -- should the workflow corpus be committed into this repo or referenced from a local external fixture path? -- should the first implementation cover only CLI/runtime execution, or also a later `Save As Script` route check? diff --git a/docs/specs/runtime-validation-implementation-spec.md b/docs/specs/runtime-validation-implementation-spec.md deleted file mode 100644 index deed264..0000000 --- a/docs/specs/runtime-validation-implementation-spec.md +++ /dev/null @@ -1,241 +0,0 @@ -# Runtime Validation Implementation Spec - -## Document Status - -- Date: March 28, 2026 -- Scope: implementation spec only -- Approval status: approved for spec creation, not yet approved for implementation - -## Problem - -The repo now has a runtime validation strategy, but not an implementation contract. Runtime-sensitive changes still rely too heavily on ad hoc local setups and narrow regression tests. The repo needs a small, repeatable runtime validation harness that agents and maintainers can run consistently. - -## Intent - -Implement a phase-based runtime validation harness and add a project-scoped `runtime-e2e-agent` that can operate against it. - -## Locked Decisions - -- Baseline pinned ComfyUI checkout: `/home/peyton/code/ComfyUI` -- Runtime workflow fixtures will be committed in this repo -- Implementation covers both CLI/runtime and `Save As Script`, but in phases -- Phase 1: CLI/runtime first -- Phase 2: `Save As Script` route validation - -## Goals - -- create a repeatable runtime validation harness -- keep the harness small enough to maintain manually -- make the workflow corpus reviewable and committed in-repo -- support pinned-runtime validation with a stable command surface -- give the future `runtime-e2e-agent` a stable command surface - -## Non-Goals - -- no full browser automation in phase 1 -- no image-quality regression testing -- no external CI/CD setup in this phase -- no vendoring of the ComfyUI repo into this repo -- no attempt to validate every possible custom-node workflow - -## Phase Plan - -### Phase 1: CLI / Runtime Harness - -Phase 1 must provide: - -- in-repo runtime workflow fixtures -- a command path that exports workflow JSON to Python -- a command path that validates generated Python parses -- a command path that executes generated Python against a pinned ComfyUI runtime -- clear pass/fail classification - -This phase should be enough to validate: - -- runtime/bootstrap regressions like `#143` -- generated-code validity issues when they must be checked against a real runtime -- runtime/bootstrap changes that affect node-loading or script execution - -### Phase 2: `Save As Script` Route Validation - -Phase 2 must build on the phase 1 harness and add: - -- route-level validation of the `Save As Script` request path -- coverage for server-side export behavior in [__init__.py](/home/peyton/code/ComfyUI-to-Python-Extension/__init__.py) -- coverage for the exported payload shape expected by the frontend flow - -Phase 2 should not start with full browser automation unless route-level checks prove insufficient. - -## Directory Layout - -The first implementation should add: - -- `tests/fixtures/runtime/` -- `tests/fixtures/runtime/upscale-model-loader.json` -- `tests/fixtures/runtime/unsafe-kwargs.json` -- `tests/fixtures/runtime/subgraph-identifiers.json` -- `tests/runtime/` -- a focused runtime validation runner or helper module inside `tests/runtime/` -- `.codex/agents/runtime-e2e-agent.toml` - -Optional supporting docs: - -- `docs/runtime-validation.md` - -## Fixture Rules - -- prefer real ComfyUI workflow shapes over synthetic-only fixtures -- keep fixtures as small as possible while still reproducing the target behavior -- name fixtures after behavior, not issue numbers, when practical -- commit fixtures to the repo so they are reviewable and stable - -## Initial Fixture Set - -### `upscale-model-loader.json` - -Purpose: - -- validate the `#143`/`#148` runtime/bootstrap path - -Assertions: - -- export succeeds -- generated Python parses -- generated script reaches the relevant loader and upscale path without the old missing-node failure - -### `unsafe-kwargs.json` - -Purpose: - -- validate mixed safe and unsafe keys in a real workflow shape - -Assertions: - -- export succeeds -- generated Python parses -- generated code remains safe when real workflow data contains symbol-heavy or invalid keys - -### `subgraph-identifiers.json` - -Purpose: - -- validate that subgraph-expanded identifiers do not break generated Python in real workflow shapes - -Assertions: - -- export succeeds -- generated Python parses -- generated variable names are safe enough for execution - -## Command Contract - -The harness should expose two stable command categories. - -### `fast` - -Purpose: - -- exporter-only validation without a real ComfyUI runtime - -Examples: - -- `uv run python -m unittest tests.test_upscale_model_loader_export -v` -- additional focused exporter tests as they are added - -### `runtime` - -Purpose: - -- run runtime validation against the pinned baseline ComfyUI checkout - -Environment: - -- `COMFYUI_PATH=/home/peyton/code/ComfyUI` - -Behavior: - -- export fixture workflow to Python -- parse or compile generated Python -- execute generated Python with the pinned runtime available - -## Runner Shape - -The implementation should prefer one small runner over many scattered commands. - -Recommended shape: - -- a small Python entry point that: - - selects a tier - - selects a fixture or fixture subset - - sets or reads `COMFYUI_PATH` - - runs export - - validates generated Python - - optionally executes the generated script - - reports failure classification - -The runner must stay small and repo-local. It should not introduce a large new framework. - -## Failure Classification Contract - -Every runtime validation result must classify failure as one of: - -- repo regression -- fixture bug -- environment/setup failure - -This classification is mandatory in both manual runs and future agent reports. - -## Runtime-E2E Agent Addition - -A future implementation step should add: - -- `.codex/agents/runtime-e2e-agent.toml` - -Its instructions should align with: - -- [runtime-e2e-agent-spec.md](/home/peyton/code/ComfyUI-to-Python-Extension/docs/specs/runtime-e2e-agent-spec.md) -- [runtime-validation-environment-spec.md](/home/peyton/code/ComfyUI-to-Python-Extension/docs/specs/runtime-validation-environment-spec.md) - -The agent should: - -- run the `runtime` tier -- report concise results -- avoid long logs -- return `pass`, `pass with risk`, or `fail` - -## Validation Expectations - -Phase 1 success means: - -- the pinned runtime can validate at least the three initial fixtures -- the runner can distinguish export failures from runtime execution failures -- the runtime harness can be invoked consistently by maintainers and agents - -Phase 2 success means: - -- route-level `Save As Script` validation exists -- frontend/server export behavior can be checked without relying on a full browser automation stack - -## Risks - -- fixture maintenance can drift if fixtures are too complex -- local ComfyUI checkout state can become stale if not periodically refreshed intentionally -- runtime validation may still fail for environment reasons if `torch` or ComfyUI dependencies are missing - -## Acceptance Criteria - -- the implementation plan is explicit enough to build without inventing workflow shape later -- fixture names and locations are defined -- command tiers are defined -- phase split is defined -- failure classification is mandatory -- the `runtime-e2e-agent` rollout target is defined - -## Next Step - -If this implementation spec is approved, the next step is to produce a concrete implementation plan and then request explicit approval before creating: - -- runtime fixture files -- the runner/helper module -- the `.codex/agents/runtime-e2e-agent.toml` file -- the short operator doc