ComfyAssets-ComfyUI_PromptManager/TESTING.md

Testing and merge confidence

Routine changes should merge after the required checks pass and the diff is reviewed. Add a regression test for each bug; prove it fails before the fix.

What runs

Layer	Real components	Replaced boundaries
Python tests	Domain logic, SQLite, API handlers, filesystem checks	ComfyUI host and ML runtimes where needed
JavaScript tests	Frontend modules and controller methods	DOM/network fixtures for focused edge cases
Viewer checks	Production Viewer.js and frontend methods in Chromium desktop/touch	Controlled images and page fixtures
Real ComfyUI	Pinned CPU host, actual extension, node definitions, widgets, routes, SQLite and real execution lifecycle	Stock UI cases isolate host writes; the execution suite queues deterministic test-only nodes without models
Page workflows	Production HTML, startup, click handlers, HTTP routes, SQLite, PNG files, thumbnails	ComfyUI host services and the tagging model

Page workflows cover save/edit/reload, duplicate-edit rejection, gallery image identity, Tags selection and viewer cleanup, delayed responses, auto-tag review and persistence, and infinite scrolling without replacing existing cards. Data safety workflows download and upload actual SQLite backups, verify restored prompts/tags/image links, reject corrupt uploads without changing live data, and exercise cancellation and network-failure recovery for restores and deletion. Duplicate cleanup verifies only the selected file disappears; prompt deletion verifies the other prompt and original image files survive. They run in Chromium, Firefox, WebKit, and Chromium touch emulation. No GPU, model downloads, running ComfyUI instance, or personal configuration is required. The server binds to localhost and creates disposable directories and a database; its reset endpoint exists only in the test harness.

Run locally

Use a Python environment containing requirements-dev.txt and Node.js 24:

python -m unittest discover -s tests -q
npm ci --prefix tools/vendor --ignore-scripts
npx --prefix tools/vendor playwright install --with-deps chromium firefox webkit
npm run --prefix tools/vendor test
npm run --prefix tools/vendor test:coverage
npm run --prefix tools/vendor test:workflows -- --project firefox --project webkit

PM_TEST_PYTHON selects the Python executable; PM_TEST_PORT selects the local port (default 8765). Set these environment variables when Python is outside PATH or the port is occupied. Test servers are never reused. On unsupported Linux distributions, browser system dependencies may need manual installation. For a quick browser run, use test:workflows -- --project chromium.

CI runs the browser jobs inside the pinned image mcr.microsoft.com/playwright:v<@playwright/test version>-noble (tests/vendor/browser-image.test.mjs keeps the tag in step with the lockfile). Concept comparisons measure geometry, and geometry follows the fonts a browser falls back to: the concept names Inter, which neither the image nor a stock runner ships. On a plain ubuntu-latest runner the fallback is DejaVu Sans, whose wider metrics make the concept's own Tags table overflow a 390px viewport, so the 390px comparisons are only meaningful in the pinned image. To reproduce CI, run the suites in that image.

Managed real ComfyUI

The shared engine comes from an exact commit of ComfyAssets Toolkit, recorded in tools/vendor/package.json and its integrity-checked lockfile. The local adapter sets PromptManager's database/configuration and monitoring policy; source discovery, coverage mapping/gates and process lifecycle are shared code. KikoTools remains the reference and is unchanged by this integration.

Use a dedicated disposable ComfyUI checkout at the revision in tools/vendor/comfyui-runtime.json. With a separate Python environment, install that checkout's requirements, this repo's requirements-dev.txt, and the exact CPU torch/torchvision pins from the JSON (using PyTorch's CPU wheel index). Install Chromium using the Playwright command above, then run:

PM_COMFY_ROOT=/path/to/disposable/ComfyUI \
PM_TEST_PYTHON=/path/to/cpu-env/bin/python \
npm run --prefix tools/vendor test:comfyui

CI performs this setup automatically. PM_COMFY_URL can select another http://127.0.0.1:PORT/ address (default port 8189). This is an owned managed server: a running host is never reused. Do not point it at your live checkout. The launcher verifies the upstream SHA and clean tracked files, snapshots only tracked PromptManager files, and creates temporary user/input/output/database/ configuration/log directories. Personal untracked files are excluded. It runs CPU/offline with only PromptManager enabled and removes owned state on shutdown. The upstream checkout must also be disposable: ComfyUI may initialize local runtime files there. Browser host-settings writes are intercepted; actual PromptManager routes use the disposable host and database.

real-comfyui-reports retains startup logs, version attachments, screenshots and traces for 14 days. Update upstream/frontend/CPU pins together after reviewing and running both startup projects; never replace the SHA with a moving branch. Update the toolkit via an exact reviewed commit archive and commit both npm manifest and lockfile. No toolkit or npm installation is needed by end users.

Coverage and failures

Python CI enforces 85% combined statement/branch coverage. Frontend coverage merges Node V8 unit coverage with Chromium page-workflow coverage through c8. Every web/js/**/*.js and web/js/**/*.mjs file appears, including untouched files at zero coverage. The controller tests provide V8 filenames so their executed code is counted too.

tools/vendor/frontend-coverage-baseline.json holds a floor for every file and metric. CI rejects regressions, missing measurements, and new files without a reviewed baseline. These floors reflect current coverage, with two percentage points of allowance for runtime/timing differences; they do not claim 80% coverage of the large controllers. New files need at least 80% line coverage when added via the baseline updater. Raise floors with npm run --prefix tools/vendor coverage:update; it reruns the suites and cannot lower an existing floor. Review baseline changes like code changes.

The viewerjs check runs on every PR, including backend-only changes. It is an aggregate job that passes only when the frontend coverage (chromium and touch) job and both page workflows (firefox) and page workflows (webkit) jobs succeed; the browsers run on separate runners in parallel. GitHub artifacts named frontend-test-reports contain HTML coverage, a JSON summary, and the Chromium/touch failure screenshots/traces; browser-traces-firefox and browser-traces-webkit hold the other browsers' traces. Open traces with Playwright's show-trace tool.

The real-host suite covers both Classic and Nodes 2.0 startup, switching both ways, actual database prompt selection, execution-text serialization, workflow reload, widget/style cleanup, node schemas and standalone-page opening. It does not certify every existing feature, arbitrary host versions, GPU/model changes, cross-pack installation or visual design changes. Those still need targeted validation.

Performance budgets

A spec performance budget gets two tests:

• A structural test that always runs, on every OS/Python job and under the coverage tracer. It asserts counts, never time: SELECTs per request, the EXPLAIN QUERY PLAN of the hot query, cache builds on cold and warm requests, and calls of the sort key per cursor page. Example: test_filter_work_is_bounded in tests/test_api_workspace_images.py.
• A wall-clock test that runs only with PROMPTMANAGER_PERF_BUDGET=1 and without a tracer (sys.gettrace() or a sys.monitoring tool). It skips otherwise. It warms up once, then measures three rounds and asserts the best round's p95; the failure message lists every timing. Example: test_filter_budget (Images filters, spec D12: p95 < 150 ms at 50,000 files and 20,000 links).

The perf budget job in .github/workflows/test.yml is the reference measurement: ubuntu-latest, one Python, no coverage, only the wall-clock tests. It runs on every push to main and on workflow_dispatch, never on pull requests, because shared-runner timing must not block a merge. Dispatch it on a branch to measure before merging. Shared Windows and macOS runners do not run them; the structural tests cover portability. Run it locally with PROMPTMANAGER_PERF_BUDGET=1 python -m unittest tests.test_api_workspace_images.TestWorkspaceImages.test_filter_budget and quote the printed p95 in the PR body.

A red perf budget run on main means the code got slower: fix the code (or change the budget in the spec with a reason) in a follow-up PR. Never rerun it to green. It is not a required check, and it cannot be one while it skips pull requests (.github/branch-protection.json).

Windows portability preflight

Before changing paths, SQLite, streams, restore or cleanup:

• Follow the boundary's existing native canonical/identity contract; Path.resolve() alone does not fold Windows case. Keep lexical aliases distinct from resolved targets. Cover mixed case, drive and UNC spellings, plus foreign-namespace denial with an allowed native control. See Python's normcase documentation.
• Close every owned database/model connection, worker and stream handle before unlink or TemporaryDirectory.cleanup(). Await worker/stream retirement. A SQLite with conn manages transactions, not connection closure, and addCleanup runs after teardown; use explicit finally cleanup when teardown removes files first.
• Windows can reject unlink of an open file; Unix can remove its directory entry while the handle remains usable. Preserve the shared behavior assertions and exercise the actual native branch, then prove handle release and successful cleanup. Do not hide failures with blanket skips or retries. See Python's remove/unlink semantics.
• Concurrent SQLite tests must not depend on how fast the host commits. Managed connections use WAL with synchronous=NORMAL (see configure_connection) so commits do not hold the write lock across a flush. Make real overlap deterministic (for example hold the first writer open with FirstWriterHold) and give peers that must all succeed the ordinary busy timeout; cross-thread handoffs wait on events with generous ceilings, never a short wall clock (#242).
• Run native Windows CI early for these changes and include a Windows impact and evidence note in the PR. All twelve OS/Python combinations and every applicable exact-head check must pass before merge. Portable ntpath/PureWindowsPath controls do not establish native filesystem, locking or cleanup success.

Required merge checks

.github/branch-protection.json records the required checks: all twelve Python/OS combinations, Python coverage, formatting, frontend modules, viewerjs, real ComfyUI compatibility, and bandit (high severity). The branch must be current with main, and administrators follow the same checks. It does not require an additional human approval.

A repository administrator applies the configuration after the new workflow is passing, then verifies it:

gh api --method PUT repos/ComfyAssets/ComfyUI_PromptManager/branches/main/protection \
  --input .github/branch-protection.json
gh api repos/ComfyAssets/ComfyUI_PromptManager/branches/main/protection/required_status_checks

Update the manifest whenever required job names change. Keep the frontend check free of path filters so a required check is never silently skipped.

Unified workspace

The opt-in /prompt_manager/workspace page opens the native Images workspace. Overview retains real /prompt_manager/stats counts. Both use the shipped toolkit. Existing admin, gallery and metadata URLs remain available while replacement workspaces are developed. Browser smoke tests in tests/browser/workspace.spec.mjs cover navigation/history, keyboard skip links, mobile layout, theme persistence/fallback, loading/retry and stale responses. The Images cases in tests/browser/workspace-images.spec.mjs additionally cover server-side filters, cursor append/retry, Grid/List/Masonry, bounded DOM, keyboard navigation, restored browsing context, Viewer.js cleanup and actual video playback. tests/test_api_workspace_images.py checks signed cursors, source identities, legacy compatibility and header-only dimensions. Frontend stream/layout units cover cancellation, malformed responses and bounded visibility lookup. These run automatically in the existing four-browser-project workflow and frontend coverage job. tests/comfyui/workspace.spec.mjs verifies the page against the managed host with both renderer startup configurations.

The shared inline and collection inspector reads the current rooted file's supported embedded records, exact generation facts and recorded or inferred prompt evidence. tests/browser/full-inspector.spec.mjs covers original/metadata/workflow downloads, response limits, source changes, nested sending, matched dark/light and mobile geometry, and the loaded-collection viewer. viewer matches the concept shell and toolbar order measures the concept's #viewer-dialog shell, header copy, toolbar order and frame geometry in the concept and in production at 1600 and 390 pixels; its #249 D34 mapping keeps production's Download link, the appended position and reason lines, coarse 40px targets and the toolbar inside .viewer-left (Viewer.js's own .viewer-toolbar rule lifts the concept's across the dialog). The viewer never fetches another page or preloads all originals. Its explicit slideshow waits 3.5 seconds after image readiness, pauses at native video without advancing on ended, and resumes on a manually selected image only while prior intent remains. Pause/Stop, inspector reading/copy/download/send, hidden document, media failure, close, query/route disposal and pagehide cancel that intent; returning visibility or closing the sender does not resume it. Stop leaves native video playback alone. Video uses only the existing normalized gallerySettings autoplay/mute/loop values, read without writes; denied/malformed storage falls back to autoplay off, mute on and loop on. Rejected autoplay keeps native controls usable.

The inspector does not establish historical file identity, infer run membership, edit image tags, export all EXIF, or load/execute a workflow. Saved vocabulary is labelled by its recipe or positive-prompt scope. Limited raw evidence is omitted atomically, with JSON exports unavailable and the rooted original still downloadable. The inspection API uses one bounded read with per-record budgets: role records (prompt, parameters, the pair execution record) share 2 MiB and exceeding it fails the read; the workflow record has its own 4 MiB and past it only the workflow becomes record format limited (no raw text, graph limited, roles and facts unaffected); other text chunks are skipped unread; byte-identical duplicates are accepted within a bounded read budget, while differing duplicates (and any duplicate role record over that budget) still fail. Other read bounds are unchanged: 8 MiB non-PNG header, 10,000 PNG chunks and a three-second ffprobe. Resolution under INSPECTION_PROFILE bounds work, not graph size: 2,000 visits per candidate chain, 20,000 in total, 64 link/JSON-container levels, 100,000 resolved characters, and a two 20,000-node ceilings: one over the top-level entries of the API prompt, and one over the workflow graph's UI nodes plus subgraph-definition nodes (past it the workflow graph and its facts become limited). The serialized response stays under 8 MiB by omitting, in order, the workflow graph, then the prompt graph, then all raw records atomically. Retry does not raise these limits. Ingestion (scan, live monitor, admin import) and queue-time run counting keep the default profile, including its 500-node gate. Only INSPECTION_PROFILE reads the node semantics table web/js/inspection-semantics.json (at most 1 MiB and eight nesting levels, loaded on first use). With it, SamplerCustomAdvanced follows its guider link (through Reroutes) to CFGGuider (positive and negative) or BasicGuider (positive only; the negative is absent, "Guider has no negative input"). Any other guider, such as DualCFGGuider, is unsupported-node. Conditioning is traced through the table's pass-throughs (FluxGuidance, ReferenceLatent, the ConditioningSet* nodes, ControlNet apply nodes, InpaintModelConditioning and WanImageToVideo by output slot). ConditioningZeroOut is an explicit empty text with source zeroed-conditioning. easy ifElse with a literal boolean follows that branch only; with a linked boolean both branches are traced, equal texts resolve and different texts make the sampler ambiguous with one candidate per choice (at most 32, else work-limit). ConditioningCombine/Concat/Average resolve only when every part has the same text, otherwise unresolved with composite naming the node. Results that depend on a forced choice are never memoized. A FaceDetailer (detailer-stage) is a sampler stage: the samplers reached through its image ancestry are compared with it. Equal texts resolve with the detailer as the sampler; different texts are ambiguous with multistage-differs and one candidate per stage; an unreadable stage makes the chain unresolved. Unanchored, a sampler whose output only feeds detailers (or previews) is not a candidate of its own. A sampler's latent ancestry is walked through every link input of non-sampler nodes within the same visit and depth budgets: with no earlier sampler it is a single stage whatever the origin (VAEEncode, EmptyLatentBatch, EmptyFlux2LatentImage, EmptyHunyuanLatentVideo, SetLatentNoiseMask, InpaintModelConditioning or WanImageToVideo slot 2). An earlier sampler (img2img from a decode, hires fix) stays unresolved "Multistage sampler ancestry is ambiguous" until stage texts are compared; an unlisted node with sampler-shaped inputs (positive, negative, guider, latent_image, a pipe, video embeds, custom-sampler parts, or a model link together with conditioning, neg_conditioning, samples or latent) is unsupported-node naming it, here and in a detailer's image ancestry; a latent that is not a link or reaches its own sampler is malformed; running out of budget is work-limit or depth-limit, never single stage. Each sampler's walk runs once per resolution; selector branch runs share it and other candidates are charged its visit cost. Text encoders: single encoders (the Qwen image encoders, TextEncodeQwenImage21 slot 0 prompt and slot 1 negative_prompt, KikoTextEncode) read one string input unchanged. parts-equal encoders (CLIPTextEncodeSDXL, Flux, SD3) resolve when every part is identical or exactly one part is non-empty (exact "", whitespace is text); otherwise composite. String sources reproduce the pinned KJNodes code from literal settings only: StringConstant is its string; StringConstantMultiline with a literal true strip_newlines is replace("\n", "").strip(); JoinStringMulti keeps string_1 (even empty), appends string_2..string_N (N = inputcount) after the literal delimiter unless missing or exactly "", and ignores later inputs. A linked setting is unsupported-node; a list output (return_list true or linked) and PromptSearchList are runtime-value ("Text is decided at run time"), never one list item. Unverified text nodes (LoRAfromText, RegexReplace, XYZPrompt, ImpactWildcardProcessor) stay unsupported-node. Every intermediate text is length-checked and charged to the scanned-character budget. If the table cannot be loaded, graph inspection is unresolved ("Inspection node semantics are unavailable") and ingestion is unaffected, because it never imports the table.

tools/inspection_corpus.py --root DIR --out JSONL [--workers N] [--baseline JSONL] [--summary TXT] [--code TREE] measures a media folder read-only against the #240 invariants H1 to H8. It runs the inspection route pipeline and the workspace JS facts of --code (default: its own tree), opens media only for reading, follows no symlinks, imports no database, and refuses --out/--summary inside the root or a code tree. Its rows hold statuses, sizes, timings and SHA-256 digests of resolved role texts, never file names or prompt text unless --with-paths is given; digests of short prompts can still be guessed, so the JSONL stays local and only the aggregate summary goes into a PR body. Run it on the base commit first and pass that JSONL as --baseline for H1 and H8 (2x the baseline p99, never below 50 ms). It exits 1 when an invariant fails, including the harness line: a tool exception or facts that were not computed (no Node.js) fail the run. Unreadable or unsupported files are inspector outcomes, not harness errors. Never point it at a live install's folders from an automated job. Installing the Python inspection/source-scope changes requires a maintainer ComfyUI restart. Automated tests use disposable hosts and never restart the running personal ComfyUI.

Run the focused page checks with:

npm run --prefix tools/vendor test:workflows -- workspace.spec.mjs workspace-images.spec.mjs
node --test tests/js/workspace-*.test.js

Use the disposable fixture and managed-host prerequisites above. A maintainer live check needs a ComfyUI restart after adding the Python route, then opening /prompt_manager/workspace; the mockup preview server is a separate design tool.

Concept shell and comparison method (#249)

CSS base and drift check. web/js/workspace/concept.css is a byte-identical copy of the approved concept's docs/concepts/workspace/style.css. Never edit the copy; it is only rewritten from the concept with npm run --prefix tools/vendor sync:concept. npm run --prefix tools/vendor check runs sync-concept-css.mjs --check, which fails on a missing copy or names the first differing byte. workspace.html loads, in order, the toolkit controls, Viewer.js, concept.css, the page stylesheets that their own PRs retire, and the production overlay workspace.css last. tests/vendor/concept-css.test.mjs asserts that order, the drift check, and that no other page or canvas script loads workspace CSS.

Overlay decision-marker lint. lintOverlay in tools/vendor/sync-concept-css.mjs fails any workspace.css rule whose selector is also a concept selector unless the rule carries a /* D<n> */ comment, before its selector or inside its block. The D-number must be one of this spec's (#249): a comment that cites any other issue, such as D5 (#245), is that issue's ruling and is not a marker, so write #245 decision 5 for those. The lint test fails if workspace.css is missing. The match is by exact selector, so a narrower selector over a concept rule (for example .ca-tools dialog.prompt-sender) passes the lint. Such rules, and the page-stylesheet leftovers such as library.css .row-actions, name the PR that retires them, by convention.

The lint covers workspace.css only. The page stylesheets that load before it (images.css, library.css, inspector.css) are not linted yet, and images.css still has unmarked concept-selector overrides (.prompt-text, .prompt-actions, .ca-tools .prompt-actions button; its .image-grid rule moved to the overlay as #249 D3 in C3). PRs C and D retire those sheets: each one runs lintOverlay over every page sheet it keeps, and adds that sheet to the real-file lint test.

Comparison method. Production must equal the concept except for the recorded deviations in the #249 design (spec §8, D-numbers). It is never compared against blessed production screenshots.

• The fixture is tests/browser/workspace_concept_fixture.py, reset through POST /__test__/concept/reset. It holds the concept's six prompts and pairs, and 24 PNGs with real links, runs and the concept's inspector facts. The sidebar counts are therefore the concept's own: 24, 6 and 15.
• The shared inventory is tests/browser/concept-inventory.mjs. One self-contained in-page measurer runs the same concept selectors in the immutable concept (/__test__/canonical/workspace) and in production. An absent element becomes null, so the diff names its path (for example chrome.breadcrumbs) instead of failing the measurement.
• Each page's map<Page>Concept(reference) applies only the recorded deviations, each commented with its D-number. diffInventory must then report nothing.
• Sizes, offsets and heights that follow from a removed element are derived from the concept's own measurements, never copied from production.
• mutate(page, {css, remove}) appends a stylesheet and detaches elements, and returns an undo. Each comparison has a mutation check proving it fails. For the shell, a .sidebar nav a { min-height: 30px } rule must fail at chrome.navs.0.rows.0.minHeight and .height, and removing .breadcrumbs must fail at chrome.breadcrumbs.
• Paired concept/production captures are attached as test artifacts. There are no PNG baselines.

tests/browser/shell-concept.spec.mjs compares the shell: the sidebar, topbar, page heading and footer, at 1600x1100 and 390x844, in dark and light, under normal and reduced motion, on Images, Prompt library and Tags. The heading is compared in full on Tags; on Images and Library only its frame and h1 are compared, because their copy and actions belong to their own PRs. The spec also compares the hover state of the first non-current nav row, the Theme select and the ComfyUI link, and the keyboard rings of the first nav link and heading button.

Recorded shell deviations:

• D4: a "Theme" select with Dark, Light and System. System follows the ComfyUI palette, then the OS.
• D5: no Activity button or Activity row until an activity center exists.
• D28: no SAVED VIEWS or COLLECTIONS, and no Design review. The bottom nav is System plus the production Overview hub (its own outline home glyph, not the concept's activity pulse, and no count). The connection reads "Your library, on your server" with a neutral muted dot.
• D29: no prototype copy. There is no "Interactive concept" badge and no "What changes & why" button. The footer reads "ComfyAssets · Local tools. Your library.", and the title has no " concept" suffix.
• D31: at max-width: 950px the overlay caps the one-column shell at minmax(0, 1fr), so extreme zoom cannot widen it to the sidebar's min-content.
• #245 D2: no "Auto-tag images" on the Tags heading until the image-annotation slice.
• Brand name: not a deviation. The brand link's accessible name matches the concept's, mark included, because production's .brand-mark is not aria-hidden either. The inventory compares brand.name and brand.mark.hidden.

npm run --prefix tools/vendor check
npm test --prefix tools/vendor
npm run --prefix tools/vendor test:workflows -- shell-concept.spec.mjs \
  --project chromium --project touch --project firefox --project webkit

tests/browser/images-concept.spec.mjs compares the Images page (#249 C7): heading, toolbar, filter row, caption, grid, the first six cards, the continuation and the panel box, in every layout, at both sizes, themes and motion settings. The concept is compared once its own Load more has loaded all 24 fixture images (it loads 12 per batch, production 40). Behaviors: hover zoom, the 2nd card opens the viewer titled "