test(history): verify run output evidence on real ComfyUI

This commit is contained in:
Vito Sansevero
2026-10-17 21:49:00 -07:00
parent 8b3b2737b2
commit 48ab5ffda9
9 changed files with 426 additions and 30 deletions
+38 -8
View File
@@ -200,8 +200,8 @@ loading a workflow or queueing generation.
## Durable execution ledger
`npm run --prefix tools/vendor test:executions` runs 26 real execution cases
(13 each in Classic and Nodes 2.0, retaining the original ten ledger cases),
`npm run --prefix tools/vendor test:executions` runs 42 real execution cases
(21 each in Classic and Nodes 2.0, retaining all 26 prior execution cases),
alongside the existing 78 stock host cases and six pack compatibility cases. Run
these suites serially because they share report cleanup and owned host resources. Use the same pinned disposable `PM_COMFY_ROOT`,
`PM_TEST_PYTHON` and Playwright browser installation as `test:comfyui`:
@@ -217,8 +217,9 @@ This suite reserves `http://127.0.0.1:8196/` and never attaches to an existing
server. Each renderer worker launches a CPU/offline host with independent
configuration, database, user and output directories. The shared launcher installs
tracked snapshots of PromptManager and a tiny temporary Git-tracked fixture pack.
Only that owned test pack provides cached output, deliberate failure, an
interruptible queue gate, deterministic CLIP/model/latent inputs and control routes.
Only that owned test pack provides cached text, deliberate failure, an
interruptible queue gate, deterministic CLIP/model/latent and CPU tensor inputs,
and control routes.
It replaces external sampler inference and trigger injection only inside the
disposable host; real registered PromptManager classes, queue, executor and cache
remain active. Its cache control resets the real executor on its worker to exercise
@@ -236,8 +237,24 @@ mode, exact UUID job/session filters and a 256-row limit; absent ledger schema i
explicit, while missing/corrupt databases and other read errors fail the suite.
Sensitive diagnostic and extra-data sentinels must never appear in lifecycle
rows. Exact prompt fields are deliberately persisted only in context snapshots.
The same read-only helper calls the internal `get_execution_contexts` reader for
requested run IDs and inspects their snapshot identities and current usage counts.
The same read-only helper calls the internal `get_execution_contexts` and
`get_execution_outputs` readers for requested run IDs and inspects their snapshot
identities and current usage counts. Its `outputs` map is keyed by immutable run
ID, alongside the existing `rows`, `captures`, `usage` and `snapshotIds` results.
`execution-outputs.spec.mjs` runs actual stock SaveImage and PreviewImage nodes
from tiny CPU tensors. It compares each attempt's persisted descriptors with that
attempt's final host history, including A,A,B batches, multiple output nodes,
cached locators, reused public UUIDs, PNG metadata disabled, graphs without PM
nodes and text-only outputs. The pinned SaveImage returns an IMAGE socket; linking
it to the owned failure/gate node orders the partial-run tests explicitly. A held
downstream gate proves completed image work does not finalize the run early.
Other cases verify real output capture across startup in each renderer, switching
both ways and workflow reload. PM usage counts retain their existing per-job
behavior. References identify reported locators only: file identity is unverified,
new generation and context membership are unknown, and temp references are
transient. The stock suite excludes both execution files; discovery retains its
78 cases and the six pack cases unchanged.
For a meaningful historical failure, create a separate disposable detached
checkout at `fdba6fd0126a8a40cbd6f3e50ed4c5de8991efd4` and pass its path to
@@ -285,8 +302,21 @@ Fault injection for restore fencing, inherited async capability revocation, queu
finalization, epoch and generation revocation, provider-local cache entries, eviction/reset, malformed or
oversized payloads and missing host interfaces lives in the focused Python
`test_execution*.py` suites. These use disposable storage and mocked host interfaces;
they are distinct from the pinned real executor/cache checks above. No output-file
associations or chronological timeline are introduced. Deploying the Python changes
they are distinct from the pinned real executor/cache checks above. Run-owned
output descriptors do not create gallery image associations or a chronological
timeline. Deploying the Python changes
requires a ComfyUI restart. After restart, queue three new jobs A,A,B with identical
positive fields and two negative recipes; existing counters should advance as usual,
and internal context inspection should find three attempts and two combinations.
For a live output check after that manual restart:
- Queue SaveImage and PreviewImage with a batch of two, then submit an unchanged
cached repeat and a changed tensor/input under new jobs.
- Inspect each run's internal output capture against its final ComfyUI history;
cached references may repeat, while the attempts remain distinct.
- Switch Classic to Nodes 2.0 and back, reload the workflow and queue again.
- Verify prompt/pair counters and gallery links retain their existing behavior;
output references alone do not prove file bytes or prompt membership.
Automated checks use only managed disposable hosts. They never restart or upgrade
the personal ComfyUI server.
+38 -16
View File
@@ -1,9 +1,9 @@
# Durable Prompt History
Status: the durable job ledger is implemented as the first slice of
[#209](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/209). Prompt
contexts, output associations and the chronological timeline remain pending;
#209 stays open.
Status: the durable job ledger, immutable prompt contexts and run-owned output
references are implemented for
[#209](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/209). Gallery
image associations and the chronological timeline remain pending; #209 stays open.
## Recorded jobs
@@ -21,7 +21,7 @@ outcomes. Missing or conflicting terminal evidence remains unknown; a process
crash can leave an observed row without an invented completion time. Bounded
safe diagnostics retain node ID/type and exception type, never graph text,
arbitrary extra data, credentials, exception messages, traceback, inputs or
output payloads. A bounded background writer keeps storage failures from
arbitrary output payloads. A bounded background writer keeps storage failures from
changing host execution and reports capture gaps rather than inventing rows.
Existing databases begin with zero execution rows. Migration and backup restore
@@ -47,16 +47,38 @@ aggregate uses. It does not expose the ledger as chronological dots or repeat-ru
branches. `PromptTracker` identities and PNG recipe evidence remain separate
from durable host job identity.
## Immutable contexts and output references
Prompt contexts retain bounded raw/effective positive, negative and composition
snapshots with nullable recipe navigation hints and explicit evidence provenance.
Unknown text remains SQL NULL; a known empty negative remains empty text. Cached
evidence belongs to its exact cache provider and attempt, rather than inheriting
unrelated recent prompts. Deletion and restore can remove navigation hints without
rewriting historical snapshots or rebinding them to current library rows.
Schema v3 adds `execution_output_capture` and `execution_run_outputs` without
backfilling older runs or modifying gallery image links. At final queue completion,
the writer attaches bounded output evidence from that attempt's final host history
to its immutable run ID. The internal `get_execution_outputs(run_id)` reader
returns state, reason, format version, source and ordered references. A complete
capture with `none_reported` means the final host history reported no images;
`not_captured`, partial, unavailable and conflicting evidence remain explicit.
Failed and interrupted jobs can still have complete evidence for images reported
before termination. Running jobs do not gain a final capture merely because an
image node finished.
Each reference records output node ID, image ordinal, folder type, subfolder,
filename, path style and an opaque configured-root key. A cached repeat can retain
the same locators under a new run. Locators do not verify file bytes, continued
existence, a newly generated file or membership in any prompt context: the reader
reports file identity as `unverified`, generation and context membership as
`unknown`, and temp outputs as transient. Capture never reads, resolves or serves
the files and adds no public route. PNG metadata is not required. Same-path file
replacement and existing gallery provenance retain their separate semantics.
## Remaining history work
Design bounded execution contexts with nullable recipe references, immutable
raw/effective positive, negative and composition snapshots and explicit
inferred/recorded/unknown provenance. Cached nodes must not inherit unrelated
recent prompt evidence. Define deletion and restore behavior before connecting
saved recipes to historical snapshots.
Add exact output references and a deduplicated run/image relationship without
claiming cached outputs were newly created. Preserve existing image provenance
and same-path replacement protection. Only after those semantics are tested can
the timeline honestly show prompt-level execution chronology, batched outputs
and repeat-run branches.
Define a deduplicated run/gallery-image relationship that preserves existing image
provenance and same-path replacement protection. Add a chronological timeline only
after its navigation and evidence semantics can honestly represent prompt-level
execution chronology, batched outputs and repeat-run branches.
+263
View File
@@ -0,0 +1,263 @@
import { randomUUID } from 'node:crypto';
import { expect } from '../../tools/vendor/node_modules/@playwright/test/index.mjs';
import { test, readExecutionRuns } from './execution-fixture.mjs';
async function resetCache({ execution }) {
expect((await execution.request.post('/pm_execution_fixture/cache', { data: { mode: 'RAM_PRESSURE' } })).ok()).toBe(true);
}
test.beforeEach(resetCache);
test.afterAll(resetCache);
function images(value, batch = 1) {
return {
'1': { class_type: 'PMHistoryImages', inputs: { value, batch } },
'2': { class_type: 'SaveImage', inputs: { images: ['1', 0], filename_prefix: 'output-evidence/nested/image' } },
'3': { class_type: 'PreviewImage', inputs: { images: ['1', 0] } },
};
}
async function submit(execution, prompt, id = randomUUID()) {
const response = await execution.request.post('/prompt', { data: { prompt, prompt_id: id } });
expect(response.ok()).toBe(true);
expect((await response.json()).prompt_id).toBe(id);
return id;
}
async function persisted(execution, id, outcomes) {
let result;
await expect.poll(async () => {
result = await readExecutionRuns(execution.databasePath, id);
return { outcomes: result.rows.map(row => row.outcome),
states: result.rows.map(row => result.outputs[row.run_id].state) };
}, { timeout: 15000, message: 'Exact attempts and final output evidence reach SQLite' }).toEqual({
outcomes, states: outcomes.map(() => 'complete'),
});
return result;
}
async function finalHistory(execution, id) {
const response = await execution.request.get(`/history/${id}`);
expect(response.ok()).toBe(true);
const item = (await response.json())[id];
expect(item.prompt[1]).toBe(id);
return item;
}
function expectHistory(capture, history) {
const reported = Object.entries(history.outputs).flatMap(([node, output]) =>
(output.images || []).map((image, ordinal) => ({ output_node_id: node, image_ordinal: ordinal,
folder_type: image.type, subfolder: image.subfolder, filename: image.filename })));
reported.sort((a, b) => a.output_node_id.localeCompare(b.output_node_id) || a.image_ordinal - b.image_ordinal);
expect(capture).toMatchObject({ state: 'complete', reason: reported.length ? 'reported' : 'none_reported',
source: 'host_final_history', format_version: 1 });
expect(capture.references.map(({ output_node_id, image_ordinal, folder_type, subfolder, filename }) =>
({ output_node_id, image_ordinal, folder_type, subfolder, filename }))).toEqual(reported);
for (const reference of capture.references) {
expect(reference).toMatchObject({ path_style: 'posix', file_identity: 'unverified',
newly_generated: 'unknown', context_membership: 'unknown', transient: reference.folder_type === 'temp' });
expect(reference.root_key).toMatch(/^[a-f0-9]{64}$/);
}
}
async function gate(execution, action) {
expect((await execution.request.post(`/pm_execution_fixture/gate/${action}`)).ok()).toBe(true);
}
async function entered(execution) {
await expect.poll(async () => (await (await execution.request.get('/pm_execution_fixture/gate')).json()).entered,
{ timeout: 10000 }).toBe(true);
}
function downstream(value, mode) {
const graph = images(value);
delete graph['3'];
graph['4'] = { class_type: 'PMHistoryImageOutput', inputs: { images: ['2', 0], mode } };
return graph;
}
test('headless stock SaveImage and PreviewImage retain A,A,B batches under a reused public UUID', async ({ execution }) => {
const id = randomUUID();
const graph = images(31, 2);
let result;
for (const [index, value] of [31, 31, 73].entries()) {
graph['1'].inputs.value = value;
await submit(execution, graph, id);
result = await persisted(execution, id, Array(index + 1).fill('succeeded'));
const item = await finalHistory(execution, id);
const row = result.rows[index];
expectHistory(result.outputs[row.run_id], item);
expect(result.outputs[row.run_id].references).toHaveLength(4);
expect(result.captures[row.run_id].contexts).toEqual([]);
expect(result.usage[row.run_id]).toEqual([]);
if (index === 1) expect(item.status.messages.find(([kind]) => kind === 'execution_cached')[1].nodes).toEqual(expect.arrayContaining(['1', '2', '3']));
}
const attempts = result.rows;
expect(new Set(attempts.map(row => row.run_id)).size).toBe(3);
expect(new Set(attempts.map(row => row.queue_item_id)).size).toBe(3);
const captures = attempts.map(row => result.outputs[row.run_id]);
expect(captures.map(capture => capture.state)).toEqual(['complete', 'complete', 'complete']);
expect(captures[1].references).toEqual(captures[0].references);
expect(captures[2].references).not.toEqual(captures[0].references);
expect(captures[0].references[0].context_membership).toBe('unknown');
expect(captures[0].references.filter(reference => reference.transient)).toHaveLength(2);
});
test('stock image evidence survives disabled PNG metadata', async ({ execution }) => {
const enabled = images(44);
delete enabled['3'];
const enabledId = await submit(execution, enabled);
const enabledResult = await persisted(execution, enabledId, ['succeeded']);
const enabledReference = enabledResult.outputs[enabledResult.rows[0].run_id].references[0];
const withMetadata = await execution.request.get('/view', { params: {
filename: enabledReference.filename, subfolder: enabledReference.subfolder, type: enabledReference.folder_type,
} });
expect(withMetadata.ok()).toBe(true);
expect((await withMetadata.body()).includes(Buffer.from('tEXtprompt'))).toBe(true);
expect((await execution.request.post('/pm_execution_fixture/metadata', { data: { disabled: true } })).ok()).toBe(true);
try {
const graph = images(45);
delete graph['3'];
const id = await submit(execution, graph);
const result = await persisted(execution, id, ['succeeded']);
const capture = result.outputs[result.rows[0].run_id];
expectHistory(capture, await finalHistory(execution, id));
expect(capture.references).toHaveLength(1);
const { filename, subfolder, folder_type: type } = capture.references[0];
const response = await execution.request.get('/view', { params: { filename, subfolder, type } });
expect(response.ok()).toBe(true);
const png = await response.body();
expect(png.subarray(1, 4).toString()).toBe('PNG');
expect(png.includes(Buffer.from('tEXtprompt'))).toBe(false);
} finally {
expect((await execution.request.post('/pm_execution_fixture/metadata', { data: { disabled: false } })).ok()).toBe(true);
}
});
test('text-only output reports a complete empty image set', async ({ execution }) => {
const id = await submit(execution, { '1': { class_type: 'PMHistoryCached', inputs: { value: 'text has no image locator' } } });
const result = await persisted(execution, id, ['succeeded']);
const capture = result.outputs[result.rows[0].run_id];
expectHistory(capture, await finalHistory(execution, id));
expect(capture.references).toEqual([]);
expect(capture.reason).toBe('none_reported');
});
test('batched outputs preserve PM usage while context membership stays unknown', async ({ execution }) => {
const graph = images(57, 2);
graph['10'] = { class_type: 'PMHistoryInputs', inputs: {} };
graph['20'] = { class_type: 'PromptManagerPair', inputs: { clip: ['10', 0], text: `output usage ${randomUUID()}`,
negative_text: `negative ${randomUUID()}`, prepend_text: '', append_text: '' } };
graph['30'] = { class_type: 'KSampler', inputs: { model: ['10', 1], latent_image: ['10', 2],
positive: ['20', 0], negative: ['20', 1], seed: 1, steps: 1, cfg: 1, sampler_name: 'euler', scheduler: 'normal', denoise: 1 } };
graph['1'].inputs.latent = ['30', 0];
const id = await submit(execution, graph);
const first = await persisted(execution, id, ['succeeded']);
const runId = first.rows[0].run_id;
expectHistory(first.outputs[runId], await finalHistory(execution, id));
expect(first.outputs[runId].references).toHaveLength(4);
expect(first.captures[runId].contexts).toHaveLength(1);
expect(first.usage[runId]).toEqual([{ role: 'positive', run_count: 1 }, { role: 'negative', run_count: 0 }, { role: 'pair', run_count: 1 }]);
await submit(execution, graph, id);
const second = await persisted(execution, id, ['succeeded', 'succeeded']);
expect(second.outputs[second.rows[1].run_id].references).toEqual(first.outputs[runId].references);
expect(second.usage[runId]).toEqual([{ role: 'positive', run_count: 2 }, { role: 'negative', run_count: 0 }, { role: 'pair', run_count: 2 }]);
});
test('failed work after stock SaveImage preserves exactly the final reported partial-run outputs', async ({ execution }) => {
const id = await submit(execution, downstream(81, 'failure'));
const result = await persisted(execution, id, ['failed']);
const item = await finalHistory(execution, id);
expect(item.status.messages.some(([kind]) => kind === 'execution_error')).toBe(true);
expectHistory(result.outputs[result.rows[0].run_id], item);
expect(result.outputs[result.rows[0].run_id].references).toHaveLength(1);
});
test('interruption after stock SaveImage preserves exactly the final reported outputs', async ({ execution }) => {
await gate(execution, 'hold');
try {
const id = await submit(execution, downstream(93, 'gate'));
await entered(execution);
expect((await execution.request.post('/interrupt', { data: {} })).ok()).toBe(true);
const result = await persisted(execution, id, ['interrupted']);
const item = await finalHistory(execution, id);
expect(item.status.messages.some(([kind]) => kind === 'execution_interrupted')).toBe(true);
expectHistory(result.outputs[result.rows[0].run_id], item);
expect(result.outputs[result.rows[0].run_id].references).toHaveLength(1);
} finally { await gate(execution, 'release'); }
});
test('completed image before pending gated work is not finalized early', async ({ execution }) => {
await gate(execution, 'hold');
try {
const id = await submit(execution, downstream(107, 'gate'));
await entered(execution);
let pending;
await expect.poll(async () => {
pending = await readExecutionRuns(execution.databasePath, id);
return pending.rows.map(row => row.outcome);
}, { timeout: 10000 }).toEqual(['observed']);
expect(pending.outputs[pending.rows[0].run_id]).toMatchObject({ state: 'not_captured', references: [] });
expect((await (await execution.request.get(`/history/${id}`)).json())[id]).toBeUndefined();
await gate(execution, 'release');
const result = await persisted(execution, id, ['succeeded']);
expectHistory(result.outputs[result.rows[0].run_id], await finalHistory(execution, id));
expect(result.outputs[result.rows[0].run_id].references).toHaveLength(1);
} finally { await gate(execution, 'release'); }
});
test('stock output capture survives both renderer switches and workflow reload', async ({ execution, page, context }, testInfo) => {
await context.route('**/*', async route => {
const path = new URL(route.request().url()).pathname.replace(/^\/api\//, '/');
if (path === '/settings' && route.request().method() === 'GET') {
const response = await route.fetch();
return route.fulfill({ json: { ...await response.json(),
'Comfy.VueNodes.Enabled': testInfo.project.metadata.initialRenderer,
'Comfy.TutorialCompleted': true, 'Comfy.Workflow.Persist': false,
'Comfy.Notification.ShowVersionUpdates': false } });
}
return route.continue();
});
const errors = [];
page.on('pageerror', error => errors.push(error.message));
await page.goto('/', { waitUntil: 'domcontentloaded' });
await expect(page.getByRole('status', { name: 'Loading ComfyUI' })).toBeHidden({ timeout: 30000 });
await page.waitForFunction(async () => {
const { app } = await import('/scripts/app.js');
return !!app.graph && !!globalThis.LiteGraph?.registered_node_types?.PMHistoryImages;
});
await expect.poll(() => page.evaluate(() => LiteGraph.vueNodesMode)).toBe(testInfo.project.metadata.initialRenderer);
await page.evaluate(async () => {
const { app } = await import('/scripts/app.js');
app.graph.clear();
const source = LiteGraph.createNode('PMHistoryImages');
const saved = LiteGraph.createNode('SaveImage');
const preview = LiteGraph.createNode('PreviewImage');
for (const node of [source, saved, preview]) app.graph.add(node);
source.widgets.find(widget => widget.name === 'value').value = 119;
source.widgets.find(widget => widget.name === 'batch').value = 1;
saved.widgets.find(widget => widget.name === 'filename_prefix').value = 'renderer-evidence/image';
source.connect(0, saved, 0);
source.connect(0, preview, 0);
});
const captures = [];
for (const renderer of [testInfo.project.metadata.initialRenderer, !testInfo.project.metadata.initialRenderer, testInfo.project.metadata.initialRenderer]) {
await page.evaluate(async renderer => {
const { app } = await import('/scripts/app.js');
await app.ui.settings.setSettingValue('Comfy.VueNodes.Enabled', renderer);
await app.loadGraphData(app.graph.serialize());
}, renderer);
await expect.poll(() => page.evaluate(() => LiteGraph.vueNodesMode)).toBe(renderer);
const graph = await page.evaluate(async () => (await (await import('/scripts/app.js')).app.graphToPrompt()).output);
expect(Object.values(graph).map(node => node.class_type).sort()).toEqual(['PMHistoryImages', 'PreviewImage', 'SaveImage']);
const id = await submit(execution, graph);
const result = await persisted(execution, id, ['succeeded']);
const capture = result.outputs[result.rows[0].run_id];
expectHistory(capture, await finalHistory(execution, id));
expect(capture.references).toHaveLength(2);
captures.push(capture);
}
expect(captures[1].references).toEqual(captures[0].references);
expect(captures[2].references).toEqual(captures[0].references);
expect(errors).toEqual([]);
});
+57
View File
@@ -5,6 +5,7 @@ import time
import nodes
import execution
import torch
from aiohttp import web
import comfy.model_management
@@ -152,6 +153,62 @@ NODE_CLASS_MAPPINGS.update(
{"PMHistoryList": PMHistoryList, "PMHistoryTransform": PMHistoryTransform}
)
class PMHistoryImages:
@classmethod
def INPUT_TYPES(cls) -> dict:
return {
"required": {
"value": ("INT", {"default": 0, "min": 0, "max": 255}),
"batch": ("INT", {"default": 1, "min": 1, "max": 2}),
},
"optional": {"latent": ("LATENT",)},
}
RETURN_TYPES = ("IMAGE",)
FUNCTION = "execute"
CATEGORY = "testing/execution-history"
def execute(self, value: int, batch: int, latent: dict | None = None) -> tuple:
return (torch.full((batch, 8, 8, 3), value / 255.0, device="cpu"),)
class PMHistoryImageOutput:
@classmethod
def INPUT_TYPES(cls) -> dict:
return {
"required": {
"images": ("IMAGE",),
"mode": (["failure", "gate"],),
}
}
RETURN_TYPES = ()
FUNCTION = "execute"
CATEGORY = "testing/execution-history"
OUTPUT_NODE = True
def execute(self, images: torch.Tensor, mode: str) -> tuple:
if mode == "failure":
raise ValueError("owned failure after real image output")
PMHistoryGate().execute("owned gate after real image output")
return ()
NODE_CLASS_MAPPINGS.update(
{"PMHistoryImages": PMHistoryImages, "PMHistoryImageOutput": PMHistoryImageOutput}
)
@PromptServer.instance.routes.post("/pm_execution_fixture/metadata")
async def metadata(request: web.Request) -> web.Response:
disabled = (await request.json())["disabled"]
if type(disabled) is not bool:
raise web.HTTPBadRequest()
nodes.args.disable_metadata = disabled
return web.json_response({"disabled": disabled})
_INJECTION_ORIGINALS = {}
+8 -1
View File
@@ -11,9 +11,12 @@ from uuid import UUID
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from database.execution_contexts import ExecutionContextOperations # noqa: E402
from database.execution_history import ExecutionHistoryOperations # noqa: E402
from database.execution_outputs import ExecutionOutputOperations # noqa: E402
class _ReadOnlyHistory(ExecutionContextOperations, ExecutionHistoryOperations):
class _ReadOnlyHistory(
ExecutionContextOperations, ExecutionHistoryOperations, ExecutionOutputOperations
):
def __init__(self, connection: sqlite3.Connection) -> None:
self.model = SimpleNamespace(get_connection=lambda: connection)
@@ -79,6 +82,10 @@ def read_runs(database: str, job_id: str, session_id: str | None = None) -> dict
"captures": captures,
"usage": usage,
"snapshotIds": snapshot_ids,
"outputs": {
row["run_id"]: reader.get_execution_outputs(row["run_id"])
for row in rows
},
}
+5 -2
View File
@@ -24,12 +24,15 @@ test('execution script discovers only the new matrix and preserves all 78 stock
const stock = list('tools/vendor/comfyui.config.mjs');
assert.match(stock, /Total: 78 tests in 7 files/);
assert.doesNotMatch(stock, /execution-history/);
assert.doesNotMatch(stock, /execution-outputs/);
const executions = list('tools/vendor/playwright.execution.config.mjs');
assert.match(executions, /\[classic-start\]/);
assert.match(executions, /\[nodes2-start\]/);
assert.match(executions, /Total: 26 tests in 1 file/);
assert.match(executions, /Total: 42 tests in 2 files/);
for (const project of ['classic-start', 'nodes2-start']) {
assert.equal(executions.split('\n').filter(line => line.includes(`[${project}]`)).length, 13);
const cases = executions.split('\n').filter(line => line.includes(`[${project}]`));
assert.equal(cases.filter(line => line.includes('execution-history.spec.mjs')).length, 13);
assert.equal(cases.filter(line => line.includes('execution-outputs.spec.mjs')).length, 8);
}
assert.doesNotMatch(executions, /prompt-manager.spec/);
});
+15 -1
View File
@@ -97,6 +97,7 @@ test('read-only ledger helper distinguishes absent schema from missing, corrupt
from database.operations import PromptDatabase
from utils.execution_history_events import RunStart, make_finish
from utils.execution_contexts import ContextBundle, PromptSnapshot, RecipeContext, TextEvidence
from utils.execution_outputs import OutputBundle, OutputReference
from types import SimpleNamespace
db = PromptDatabase(sys.argv[1])
@@ -123,6 +124,10 @@ try:
db.write_execution_contexts(sys.argv[5], epoch, ContextBundle(
(RecipeContext('4', snapshot(' positive '), snapshot('negative'), 'inferred'),),
'complete', 'captured'))
db.write_execution_outputs_for_finish(make_finish(sys.argv[3], 1, epoch,
SimpleNamespace(messages=[('execution_success', {'prompt_id': sys.argv[2]})]), 200),
OutputBundle((OutputReference('9', 0, 'temp', 'nested', 'preview.png', 'posix', 'a' * 64),),
'complete', 'reported'))
print(epoch)
finally:
db.close_all()
@@ -138,6 +143,12 @@ finally:
node_id: null, node_type: null, exception_type: null,
}]);
assert.deepEqual(Object.keys(result.captures), [runId]);
assert.deepEqual(result.outputs, { [runId]: {
state: 'complete', reason: 'reported', format_version: 1, source: 'host_final_history',
references: [{ output_node_id: '9', image_ordinal: 0, folder_type: 'temp',
subfolder: 'nested', filename: 'preview.png', path_style: 'posix', root_key: 'a'.repeat(64),
file_identity: 'unverified', newly_generated: 'unknown', context_membership: 'unknown', transient: true }],
} });
const capture = result.captures[runId];
assert.equal(capture.contexts.length, 1);
const contextId = capture.contexts[0].context_id;
@@ -170,8 +181,11 @@ finally:
});
assert.deepEqual(unfiltered.usage[uncapturedId], []);
assert.deepEqual(unfiltered.snapshotIds[uncapturedId], []);
assert.deepEqual(unfiltered.outputs[uncapturedId], {
state: 'not_captured', reason: null, format_version: null, source: null, references: [],
});
assert.deepEqual(await readExecutionRuns(database, job, '60000000-0000-4000-8000-000000000001'), {
schemaAvailable: true, rows: [], captures: {}, usage: {}, snapshotIds: {},
schemaAvailable: true, rows: [], captures: {}, usage: {}, snapshotIds: {}, outputs: {},
});
assert.deepEqual(await readFile(database), before);
await assert.rejects(readExecutionRuns(database, "' OR 1=1 --"));
+1 -1
View File
@@ -14,7 +14,7 @@ const servers = [{
export default defineConfig({
testDir: '../../tests/comfyui',
testIgnore: 'execution-history.spec.mjs',
testIgnore: ['execution-history.spec.mjs', 'execution-outputs.spec.mjs'],
outputDir: './test-results/comfyui',
timeout: 60000,
workers: 1,
+1 -1
View File
@@ -6,7 +6,7 @@ if (host.href !== 'http://127.0.0.1:8196/') throw new Error('Execution tests req
export default defineConfig({
testDir: '../../tests/comfyui',
testMatch: 'execution-history.spec.mjs',
testMatch: ['execution-history.spec.mjs', 'execution-outputs.spec.mjs'],
outputDir: './test-results/executions/cases',
timeout: 60000,
workers: 1,