nodes/pydn-ComfyUI-to-Python-Extension
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Remove tracked runtime spec docs from branch

Browse Source
This commit is contained in:
Peyton DeNiro
2026-03-28 16:12:05 -05:00
parent b0109aa90f
commit d57b2167ca
3 changed files with 0 additions and 505 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/specs/runtime-e2e-agent-spec.md
-128
View File
@@ -1,128 +0,0 @@
# Runtime E2E Agent Spec
## Purpose
Own end-to-end runtime validation for this repo against a real ComfyUI runtime, using a pinned validation target and a small representative workflow corpus.
## Use When
- a change affects runtime/bootstrap behavior
- a merge recommendation needs stronger evidence than exporter-only tests
- a maintainer wants a runtime sanity check before or after a hotfix merge
## Required Inputs
- approved spec or target branch under review
- pinned ComfyUI runtime target
- runtime validation workflow corpus
- required runtime dependencies in the validation environment
- current repo worktree or PR branch
## Expected Outputs
- a short executive summary first
- a structured validation report with:
- Inputs
- Checks Performed
- Findings
- Fixture or Environment Issues
- Coverage Gaps
- Merge Recommendation
- Next Action
## Reporting Rules
- respect the maintainer's time
- keep the report decision-oriented
- distinguish repo regression from harness failure
- do not dump raw logs unless a short excerpt is needed to explain a failure
## Validation Tiers
### `fast`
Use exporter-only checks that do not require a full ComfyUI runtime. This tier is not owned exclusively by this agent, but the agent should understand when `fast` is enough and when runtime validation is required.
### `runtime`
Run representative generated-script validations against a pinned local ComfyUI checkout with required runtime dependencies available.
## Responsibilities
- maintain awareness of the pinned ComfyUI runtime target
- run the `runtime` tier for runtime-sensitive changes
- classify failures into:
- repo bug
- fixture bug
- environment/setup failure
- provide a concise merge recommendation:
- `pass`
- `pass with risk`
- `fail`
## Initial Workflow Corpus
The first validation corpus should cover:
- upscale loader workflow for `#143`
- unsafe-kwargs workflow for `#119` / `#96`
- subgraph identifier workflow for `#128`
Future additions may include:
- frontend save-flow validation fixtures
- runtime performance/bootstrap parity fixtures for issues like `#147`
## Suggested Cadence
- on demand for runtime-sensitive branches
- before merge for runtime/bootstrap fixes
## Handoff Rules
- report merge decisions to `maintainer-orchestrator`
- send runtime regressions to `runtime-compat-agent`
- send exporter-only failures back to `verification-agent` or `exporter-bug-agent`
- send upstream harness breakage to `upstream-watch-agent`
## Branch and PR Conventions
- validation results should reference the exact working branch and, when present, the PR title
- runtime-only harness changes should use focused branch names such as:
- `codex/runtime-e2e-<short-slug>`
- `codex/issue-<number>-runtime-e2e`
## Ready-to-Paste Prompt
```text
You are the runtime-e2e-agent for the ComfyUI-to-Python-Extension repo.
Your job is to validate runtime-sensitive changes against a real ComfyUI runtime using a pinned validation target and a representative workflow corpus. Respect the maintainer's time. Be concise, strict, and decision-oriented. Do not write long test diaries.
Repo rules:
- use uv for repo commands
- preserve a clear separation between exporter-only checks and real runtime validation
- classify failures accurately: repo bug, fixture bug, or environment failure
- do not call something "pass" if the runtime path was not actually exercised
When you respond:
- Start with Executive Summary in 3 to 6 short lines.
- Then use:
- Inputs
- Checks Performed
- Findings
- Fixture or Environment Issues
- Coverage Gaps
- Merge Recommendation
- Next Action
Recommendation labels:
- pass
- pass with risk
- fail
Validation rules:
- run the pinned runtime suite when the change affects runtime/bootstrap behavior
- keep logs summarized unless a short excerpt is necessary
- make it obvious whether a failure belongs to this repo or the validation harness
```
docs/specs/runtime-validation-environment-spec.md
-136
View File
@@ -1,136 +0,0 @@
# Runtime Validation Environment Spec
## Purpose
Define the environment, fixture layout, and execution contract for end-to-end runtime validation of `ComfyUI-to-Python-Extension`.
## Problem
Exporter-only regression tests are necessary but insufficient for runtime/bootstrap-sensitive changes. Local verification becomes unreliable when tests depend on an arbitrary ComfyUI installation or on environment details that are not controlled. The repo needs a repeatable runtime validation harness.
## Goal
Create a stable runtime validation setup that:
- uses a pinned ComfyUI checkout
- validates generated scripts against a real runtime
- uses a small set of representative workflow fixtures
- clearly separates repo regressions from environment problems
## Non-Goals
- no image-quality benchmarking
- no full browser automation in the first phase
- no external CI/CD setup in this phase
- no attempt to cover every ComfyUI node or workflow shape
## Validation Model
### Tier 1: `fast`
Purpose:
- run quickly during local development
- validate exporter behavior without requiring a real ComfyUI runtime
Typical checks:
- focused unit or regression tests with injected mappings
- generated Python parse checks
- syntax compilation
### Tier 2: `runtime`
Purpose:
- validate generated scripts against a real pinned ComfyUI runtime
Typical checks:
- export from API-format workflow to Python
- parse/import generated Python
- execute generated script through the targeted runtime path
- confirm expected bootstrap and node-loading behavior
## Environment Requirements
- a local ComfyUI checkout available through a documented path convention
- the ability to point this repo to that checkout, preferably with `COMFYUI_PATH`
- required runtime dependencies installed in the validation environment, including `torch`
- all repo command execution through `uv`
## Proposed Path Convention
The validation harness should assume:
- this repo remains where it is
- the pinned ComfyUI checkout lives at a maintainer-chosen local path
- the runtime command surface uses `COMFYUI_PATH=/path/to/ComfyUI`
The exact checkout location does not need to be committed into this repo, but the contract for supplying it must be documented and stable.
## Workflow Fixture Corpus
The first committed or documented fixture set should cover:
- `upscale-model-loader`
for `#143`-style runtime/bootstrap validation
- `unsafe-kwargs`
for `#119` / `#96`-style generated-call validation against real workflows
- `subgraph-identifiers`
for `#128`-style identifier safety
Fixture principles:
- prefer real workflow shapes from official ComfyUI examples when possible
- keep fixtures minimal but representative
- name fixtures after behavior, not issue numbers, when practical
## Assertions
The first phase should assert:
- export succeeds
- generated Python parses
- generated script can execute far enough to validate the targeted runtime path
- expected code patterns are present or absent when relevant
The first phase should not require:
- image-diff assertions
- performance benchmarking beyond obvious bootstrap regressions
- full frontend automation
## Failure Classification
Every runtime validation failure should be classified as one of:
- repo regression
- fixture bug
- environment/setup failure
This classification is mandatory because raw pass/fail is not enough for a maintenance workflow.
## Likely Future Deliverables
When implementation is approved, the runtime validation system will likely need:
- a fixtures directory for runtime workflows
- a short operator document for setting `COMFYUI_PATH`
- one or more helper commands or scripts for running:
- `fast`
- `runtime`
- a `runtime-e2e-agent` config file under `.codex/agents/`
## Acceptance Criteria
- runtime validation is defined around a pinned ComfyUI target
- the initial workflow corpus covers the main known runtime-sensitive issue classes
- validation tiers are clearly separated
- pass/fail reporting includes failure classification
- the setup is small enough to maintain manually inside Codex/chat workflows
## Open Questions
- should the workflow corpus be committed into this repo or referenced from a local external fixture path?
- should the first implementation cover only CLI/runtime execution, or also a later `Save As Script` route check?
docs/specs/runtime-validation-implementation-spec.md
-241
View File
@@ -1,241 +0,0 @@
# Runtime Validation Implementation Spec
## Document Status
- Date: March 28, 2026
- Scope: implementation spec only
- Approval status: approved for spec creation, not yet approved for implementation
## Problem
The repo now has a runtime validation strategy, but not an implementation contract. Runtime-sensitive changes still rely too heavily on ad hoc local setups and narrow regression tests. The repo needs a small, repeatable runtime validation harness that agents and maintainers can run consistently.
## Intent
Implement a phase-based runtime validation harness and add a project-scoped `runtime-e2e-agent` that can operate against it.
## Locked Decisions
- Baseline pinned ComfyUI checkout: `/home/peyton/code/ComfyUI`
- Runtime workflow fixtures will be committed in this repo
- Implementation covers both CLI/runtime and `Save As Script`, but in phases
- Phase 1: CLI/runtime first
- Phase 2: `Save As Script` route validation
## Goals
- create a repeatable runtime validation harness
- keep the harness small enough to maintain manually
- make the workflow corpus reviewable and committed in-repo
- support pinned-runtime validation with a stable command surface
- give the future `runtime-e2e-agent` a stable command surface
## Non-Goals
- no full browser automation in phase 1
- no image-quality regression testing
- no external CI/CD setup in this phase
- no vendoring of the ComfyUI repo into this repo
- no attempt to validate every possible custom-node workflow
## Phase Plan
### Phase 1: CLI / Runtime Harness
Phase 1 must provide:
- in-repo runtime workflow fixtures
- a command path that exports workflow JSON to Python
- a command path that validates generated Python parses
- a command path that executes generated Python against a pinned ComfyUI runtime
- clear pass/fail classification
This phase should be enough to validate:
- runtime/bootstrap regressions like `#143`
- generated-code validity issues when they must be checked against a real runtime
- runtime/bootstrap changes that affect node-loading or script execution
### Phase 2: `Save As Script` Route Validation
Phase 2 must build on the phase 1 harness and add:
- route-level validation of the `Save As Script` request path
- coverage for server-side export behavior in [__init__.py](/home/peyton/code/ComfyUI-to-Python-Extension/__init__.py)
- coverage for the exported payload shape expected by the frontend flow
Phase 2 should not start with full browser automation unless route-level checks prove insufficient.
## Directory Layout
The first implementation should add:
- `tests/fixtures/runtime/`
- `tests/fixtures/runtime/upscale-model-loader.json`
- `tests/fixtures/runtime/unsafe-kwargs.json`
- `tests/fixtures/runtime/subgraph-identifiers.json`
- `tests/runtime/`
- a focused runtime validation runner or helper module inside `tests/runtime/`
- `.codex/agents/runtime-e2e-agent.toml`
Optional supporting docs:
- `docs/runtime-validation.md`
## Fixture Rules
- prefer real ComfyUI workflow shapes over synthetic-only fixtures
- keep fixtures as small as possible while still reproducing the target behavior
- name fixtures after behavior, not issue numbers, when practical
- commit fixtures to the repo so they are reviewable and stable
## Initial Fixture Set
### `upscale-model-loader.json`
Purpose:
- validate the `#143`/`#148` runtime/bootstrap path
Assertions:
- export succeeds
- generated Python parses
- generated script reaches the relevant loader and upscale path without the old missing-node failure
### `unsafe-kwargs.json`
Purpose:
- validate mixed safe and unsafe keys in a real workflow shape
Assertions:
- export succeeds
- generated Python parses
- generated code remains safe when real workflow data contains symbol-heavy or invalid keys
### `subgraph-identifiers.json`
Purpose:
- validate that subgraph-expanded identifiers do not break generated Python in real workflow shapes
Assertions:
- export succeeds
- generated Python parses
- generated variable names are safe enough for execution
## Command Contract
The harness should expose two stable command categories.
### `fast`
Purpose:
- exporter-only validation without a real ComfyUI runtime
Examples:
- `uv run python -m unittest tests.test_upscale_model_loader_export -v`
- additional focused exporter tests as they are added
### `runtime`
Purpose:
- run runtime validation against the pinned baseline ComfyUI checkout
Environment:
- `COMFYUI_PATH=/home/peyton/code/ComfyUI`
Behavior:
- export fixture workflow to Python
- parse or compile generated Python
- execute generated Python with the pinned runtime available
## Runner Shape
The implementation should prefer one small runner over many scattered commands.
Recommended shape:
- a small Python entry point that:
- selects a tier
- selects a fixture or fixture subset
- sets or reads `COMFYUI_PATH`
- runs export
- validates generated Python
- optionally executes the generated script
- reports failure classification
The runner must stay small and repo-local. It should not introduce a large new framework.
## Failure Classification Contract
Every runtime validation result must classify failure as one of:
- repo regression
- fixture bug
- environment/setup failure
This classification is mandatory in both manual runs and future agent reports.
## Runtime-E2E Agent Addition
A future implementation step should add:
- `.codex/agents/runtime-e2e-agent.toml`
Its instructions should align with:
- [runtime-e2e-agent-spec.md](/home/peyton/code/ComfyUI-to-Python-Extension/docs/specs/runtime-e2e-agent-spec.md)
- [runtime-validation-environment-spec.md](/home/peyton/code/ComfyUI-to-Python-Extension/docs/specs/runtime-validation-environment-spec.md)
The agent should:
- run the `runtime` tier
- report concise results
- avoid long logs
- return `pass`, `pass with risk`, or `fail`
## Validation Expectations
Phase 1 success means:
- the pinned runtime can validate at least the three initial fixtures
- the runner can distinguish export failures from runtime execution failures
- the runtime harness can be invoked consistently by maintainers and agents
Phase 2 success means:
- route-level `Save As Script` validation exists
- frontend/server export behavior can be checked without relying on a full browser automation stack
## Risks
- fixture maintenance can drift if fixtures are too complex
- local ComfyUI checkout state can become stale if not periodically refreshed intentionally
- runtime validation may still fail for environment reasons if `torch` or ComfyUI dependencies are missing
## Acceptance Criteria
- the implementation plan is explicit enough to build without inventing workflow shape later
- fixture names and locations are defined
- command tiers are defined
- phase split is defined
- failure classification is mandatory
- the `runtime-e2e-agent` rollout target is defined
## Next Step
If this implementation spec is approved, the next step is to produce a concrete implementation plan and then request explicit approval before creating:
- runtime fixture files
- the runner/helper module
- the `.codex/agents/runtime-e2e-agent.toml` file
- the short operator doc