pydn-ComfyUI-to-Python-Extension/docs/runtime-validation.md

Runtime Validation

This repo has two validation tiers:

• fast
• runtime

Use uv for repo commands.

Local Runtime Paths

Pinned ComfyUI checkout:

• /home/peyton/code/ComfyUI

The runtime harness uses COMFYUI_PATH to point generated scripts and export logic at the target ComfyUI checkout. For full E2E runtime execution, the harness prefers the ComfyUI checkout's own Python interpreter at /home/peyton/code/ComfyUI/.venv/bin/python when available.

Fixtures

Committed runtime fixtures live in:

• tests/fixtures/runtime

Current fixtures:

• upscale-model-loader
• text-to-image
• unsafe-kwargs
• subgraph-identifiers

Runtime-tier fixtures are based on official core ComfyUI workflow shapes:

• the Image Upscale workflow
• the Text to Image workflow

Sources:

• https://docs.comfy.org/tutorials/basic/upscale
• https://docs.comfy.org/tutorials/basic/text-to-image

Commands

Fast

Run exporter-only validation without a real ComfyUI runtime:

uv run python tests/runtime/run_runtime_validation.py --tier fast --fixture all

Runtime

Run runtime-capable validation against the pinned baseline checkout. This now performs full E2E execution by default:

uv run python tests/runtime/run_runtime_validation.py --tier runtime --fixture upscale-model-loader

Run both official runtime fixtures:

uv run python tests/runtime/run_runtime_validation.py --tier runtime --fixture all

Model Provisioning

If a runtime fixture is missing required models, the harness will report model provisioning failure.

To print the required download commands without executing them:

uv run python tests/runtime/run_runtime_validation.py --tier runtime --fixture all --print-download-plan

Model downloads should be approval-gated and should target the existing /home/peyton/code/ComfyUI/models/... directories.

Failure Classification

The runtime harness classifies failures as:

• repo regression
• model provisioning failure
• fixture bug
• environment/setup failure

This classification matters more than raw pass/fail. A failed run without classification is not actionable enough for maintenance work.

Notes

• unsafe-kwargs and subgraph-identifiers are committed fast-tier regression fixtures. They are not runtime-tier fixtures.
• upscale-model-loader and text-to-image are the current runtime-tier fixtures and are based on official ComfyUI tutorials.
• Runtime-tier success now means:
  • generated Python parses
  • generated script executes
  • a new output file is created
  • output dimensions meet expectations
  • output metadata contains expected markers
• Save As Script route validation is still phase 2 work.