diff --git a/TESTING.md b/TESTING.md index 94c9296..67c1ca2 100644 --- a/TESTING.md +++ b/TESTING.md @@ -200,8 +200,8 @@ loading a workflow or queueing generation. ## Durable execution ledger -`npm run --prefix tools/vendor test:executions` runs 26 real execution cases -(13 each in Classic and Nodes 2.0, retaining the original ten ledger cases), +`npm run --prefix tools/vendor test:executions` runs 42 real execution cases +(21 each in Classic and Nodes 2.0, retaining all 26 prior execution cases), alongside the existing 78 stock host cases and six pack compatibility cases. Run these suites serially because they share report cleanup and owned host resources. Use the same pinned disposable `PM_COMFY_ROOT`, `PM_TEST_PYTHON` and Playwright browser installation as `test:comfyui`: @@ -217,8 +217,9 @@ This suite reserves `http://127.0.0.1:8196/` and never attaches to an existing server. Each renderer worker launches a CPU/offline host with independent configuration, database, user and output directories. The shared launcher installs tracked snapshots of PromptManager and a tiny temporary Git-tracked fixture pack. -Only that owned test pack provides cached output, deliberate failure, an -interruptible queue gate, deterministic CLIP/model/latent inputs and control routes. +Only that owned test pack provides cached text, deliberate failure, an +interruptible queue gate, deterministic CLIP/model/latent and CPU tensor inputs, +and control routes. It replaces external sampler inference and trigger injection only inside the disposable host; real registered PromptManager classes, queue, executor and cache remain active. Its cache control resets the real executor on its worker to exercise @@ -236,8 +237,24 @@ mode, exact UUID job/session filters and a 256-row limit; absent ledger schema i explicit, while missing/corrupt databases and other read errors fail the suite. Sensitive diagnostic and extra-data sentinels must never appear in lifecycle rows. Exact prompt fields are deliberately persisted only in context snapshots. -The same read-only helper calls the internal `get_execution_contexts` reader for -requested run IDs and inspects their snapshot identities and current usage counts. +The same read-only helper calls the internal `get_execution_contexts` and +`get_execution_outputs` readers for requested run IDs and inspects their snapshot +identities and current usage counts. Its `outputs` map is keyed by immutable run +ID, alongside the existing `rows`, `captures`, `usage` and `snapshotIds` results. + +`execution-outputs.spec.mjs` runs actual stock SaveImage and PreviewImage nodes +from tiny CPU tensors. It compares each attempt's persisted descriptors with that +attempt's final host history, including A,A,B batches, multiple output nodes, +cached locators, reused public UUIDs, PNG metadata disabled, graphs without PM +nodes and text-only outputs. The pinned SaveImage returns an IMAGE socket; linking +it to the owned failure/gate node orders the partial-run tests explicitly. A held +downstream gate proves completed image work does not finalize the run early. +Other cases verify real output capture across startup in each renderer, switching +both ways and workflow reload. PM usage counts retain their existing per-job +behavior. References identify reported locators only: file identity is unverified, +new generation and context membership are unknown, and temp references are +transient. The stock suite excludes both execution files; discovery retains its +78 cases and the six pack cases unchanged. For a meaningful historical failure, create a separate disposable detached checkout at `fdba6fd0126a8a40cbd6f3e50ed4c5de8991efd4` and pass its path to @@ -285,8 +302,21 @@ Fault injection for restore fencing, inherited async capability revocation, queu finalization, epoch and generation revocation, provider-local cache entries, eviction/reset, malformed or oversized payloads and missing host interfaces lives in the focused Python `test_execution*.py` suites. These use disposable storage and mocked host interfaces; -they are distinct from the pinned real executor/cache checks above. No output-file -associations or chronological timeline are introduced. Deploying the Python changes +they are distinct from the pinned real executor/cache checks above. Run-owned +output descriptors do not create gallery image associations or a chronological +timeline. Deploying the Python changes requires a ComfyUI restart. After restart, queue three new jobs A,A,B with identical positive fields and two negative recipes; existing counters should advance as usual, and internal context inspection should find three attempts and two combinations. +For a live output check after that manual restart: + +- Queue SaveImage and PreviewImage with a batch of two, then submit an unchanged + cached repeat and a changed tensor/input under new jobs. +- Inspect each run's internal output capture against its final ComfyUI history; + cached references may repeat, while the attempts remain distinct. +- Switch Classic to Nodes 2.0 and back, reload the workflow and queue again. +- Verify prompt/pair counters and gallery links retain their existing behavior; + output references alone do not prove file bytes or prompt membership. + +Automated checks use only managed disposable hosts. They never restart or upgrade +the personal ComfyUI server. diff --git a/docs/prompt-history-storage.md b/docs/prompt-history-storage.md index 2e5440c..d1e826e 100644 --- a/docs/prompt-history-storage.md +++ b/docs/prompt-history-storage.md @@ -1,9 +1,9 @@ # Durable Prompt History -Status: the durable job ledger is implemented as the first slice of -[#209](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/209). Prompt -contexts, output associations and the chronological timeline remain pending; -#209 stays open. +Status: the durable job ledger, immutable prompt contexts and run-owned output +references are implemented for +[#209](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/209). Gallery +image associations and the chronological timeline remain pending; #209 stays open. ## Recorded jobs @@ -21,7 +21,7 @@ outcomes. Missing or conflicting terminal evidence remains unknown; a process crash can leave an observed row without an invented completion time. Bounded safe diagnostics retain node ID/type and exception type, never graph text, arbitrary extra data, credentials, exception messages, traceback, inputs or -output payloads. A bounded background writer keeps storage failures from +arbitrary output payloads. A bounded background writer keeps storage failures from changing host execution and reports capture gaps rather than inventing rows. Existing databases begin with zero execution rows. Migration and backup restore @@ -47,16 +47,38 @@ aggregate uses. It does not expose the ledger as chronological dots or repeat-ru branches. `PromptTracker` identities and PNG recipe evidence remain separate from durable host job identity. +## Immutable contexts and output references + +Prompt contexts retain bounded raw/effective positive, negative and composition +snapshots with nullable recipe navigation hints and explicit evidence provenance. +Unknown text remains SQL NULL; a known empty negative remains empty text. Cached +evidence belongs to its exact cache provider and attempt, rather than inheriting +unrelated recent prompts. Deletion and restore can remove navigation hints without +rewriting historical snapshots or rebinding them to current library rows. + +Schema v3 adds `execution_output_capture` and `execution_run_outputs` without +backfilling older runs or modifying gallery image links. At final queue completion, +the writer attaches bounded output evidence from that attempt's final host history +to its immutable run ID. The internal `get_execution_outputs(run_id)` reader +returns state, reason, format version, source and ordered references. A complete +capture with `none_reported` means the final host history reported no images; +`not_captured`, partial, unavailable and conflicting evidence remain explicit. +Failed and interrupted jobs can still have complete evidence for images reported +before termination. Running jobs do not gain a final capture merely because an +image node finished. + +Each reference records output node ID, image ordinal, folder type, subfolder, +filename, path style and an opaque configured-root key. A cached repeat can retain +the same locators under a new run. Locators do not verify file bytes, continued +existence, a newly generated file or membership in any prompt context: the reader +reports file identity as `unverified`, generation and context membership as +`unknown`, and temp outputs as transient. Capture never reads, resolves or serves +the files and adds no public route. PNG metadata is not required. Same-path file +replacement and existing gallery provenance retain their separate semantics. + ## Remaining history work -Design bounded execution contexts with nullable recipe references, immutable -raw/effective positive, negative and composition snapshots and explicit -inferred/recorded/unknown provenance. Cached nodes must not inherit unrelated -recent prompt evidence. Define deletion and restore behavior before connecting -saved recipes to historical snapshots. - -Add exact output references and a deduplicated run/image relationship without -claiming cached outputs were newly created. Preserve existing image provenance -and same-path replacement protection. Only after those semantics are tested can -the timeline honestly show prompt-level execution chronology, batched outputs -and repeat-run branches. +Define a deduplicated run/gallery-image relationship that preserves existing image +provenance and same-path replacement protection. Add a chronological timeline only +after its navigation and evidence semantics can honestly represent prompt-level +execution chronology, batched outputs and repeat-run branches. diff --git a/tests/comfyui/execution-outputs.spec.mjs b/tests/comfyui/execution-outputs.spec.mjs new file mode 100644 index 0000000..ad98a57 --- /dev/null +++ b/tests/comfyui/execution-outputs.spec.mjs @@ -0,0 +1,263 @@ +import { randomUUID } from 'node:crypto'; +import { expect } from '../../tools/vendor/node_modules/@playwright/test/index.mjs'; +import { test, readExecutionRuns } from './execution-fixture.mjs'; + +async function resetCache({ execution }) { + expect((await execution.request.post('/pm_execution_fixture/cache', { data: { mode: 'RAM_PRESSURE' } })).ok()).toBe(true); +} + +test.beforeEach(resetCache); +test.afterAll(resetCache); + +function images(value, batch = 1) { + return { + '1': { class_type: 'PMHistoryImages', inputs: { value, batch } }, + '2': { class_type: 'SaveImage', inputs: { images: ['1', 0], filename_prefix: 'output-evidence/nested/image' } }, + '3': { class_type: 'PreviewImage', inputs: { images: ['1', 0] } }, + }; +} + +async function submit(execution, prompt, id = randomUUID()) { + const response = await execution.request.post('/prompt', { data: { prompt, prompt_id: id } }); + expect(response.ok()).toBe(true); + expect((await response.json()).prompt_id).toBe(id); + return id; +} + +async function persisted(execution, id, outcomes) { + let result; + await expect.poll(async () => { + result = await readExecutionRuns(execution.databasePath, id); + return { outcomes: result.rows.map(row => row.outcome), + states: result.rows.map(row => result.outputs[row.run_id].state) }; + }, { timeout: 15000, message: 'Exact attempts and final output evidence reach SQLite' }).toEqual({ + outcomes, states: outcomes.map(() => 'complete'), + }); + return result; +} + +async function finalHistory(execution, id) { + const response = await execution.request.get(`/history/${id}`); + expect(response.ok()).toBe(true); + const item = (await response.json())[id]; + expect(item.prompt[1]).toBe(id); + return item; +} + +function expectHistory(capture, history) { + const reported = Object.entries(history.outputs).flatMap(([node, output]) => + (output.images || []).map((image, ordinal) => ({ output_node_id: node, image_ordinal: ordinal, + folder_type: image.type, subfolder: image.subfolder, filename: image.filename }))); + reported.sort((a, b) => a.output_node_id.localeCompare(b.output_node_id) || a.image_ordinal - b.image_ordinal); + expect(capture).toMatchObject({ state: 'complete', reason: reported.length ? 'reported' : 'none_reported', + source: 'host_final_history', format_version: 1 }); + expect(capture.references.map(({ output_node_id, image_ordinal, folder_type, subfolder, filename }) => + ({ output_node_id, image_ordinal, folder_type, subfolder, filename }))).toEqual(reported); + for (const reference of capture.references) { + expect(reference).toMatchObject({ path_style: 'posix', file_identity: 'unverified', + newly_generated: 'unknown', context_membership: 'unknown', transient: reference.folder_type === 'temp' }); + expect(reference.root_key).toMatch(/^[a-f0-9]{64}$/); + } +} + +async function gate(execution, action) { + expect((await execution.request.post(`/pm_execution_fixture/gate/${action}`)).ok()).toBe(true); +} + +async function entered(execution) { + await expect.poll(async () => (await (await execution.request.get('/pm_execution_fixture/gate')).json()).entered, + { timeout: 10000 }).toBe(true); +} + +function downstream(value, mode) { + const graph = images(value); + delete graph['3']; + graph['4'] = { class_type: 'PMHistoryImageOutput', inputs: { images: ['2', 0], mode } }; + return graph; +} + +test('headless stock SaveImage and PreviewImage retain A,A,B batches under a reused public UUID', async ({ execution }) => { + const id = randomUUID(); + const graph = images(31, 2); + let result; + for (const [index, value] of [31, 31, 73].entries()) { + graph['1'].inputs.value = value; + await submit(execution, graph, id); + result = await persisted(execution, id, Array(index + 1).fill('succeeded')); + const item = await finalHistory(execution, id); + const row = result.rows[index]; + expectHistory(result.outputs[row.run_id], item); + expect(result.outputs[row.run_id].references).toHaveLength(4); + expect(result.captures[row.run_id].contexts).toEqual([]); + expect(result.usage[row.run_id]).toEqual([]); + if (index === 1) expect(item.status.messages.find(([kind]) => kind === 'execution_cached')[1].nodes).toEqual(expect.arrayContaining(['1', '2', '3'])); + } + const attempts = result.rows; + expect(new Set(attempts.map(row => row.run_id)).size).toBe(3); + expect(new Set(attempts.map(row => row.queue_item_id)).size).toBe(3); + const captures = attempts.map(row => result.outputs[row.run_id]); + expect(captures.map(capture => capture.state)).toEqual(['complete', 'complete', 'complete']); + expect(captures[1].references).toEqual(captures[0].references); + expect(captures[2].references).not.toEqual(captures[0].references); + expect(captures[0].references[0].context_membership).toBe('unknown'); + expect(captures[0].references.filter(reference => reference.transient)).toHaveLength(2); +}); + +test('stock image evidence survives disabled PNG metadata', async ({ execution }) => { + const enabled = images(44); + delete enabled['3']; + const enabledId = await submit(execution, enabled); + const enabledResult = await persisted(execution, enabledId, ['succeeded']); + const enabledReference = enabledResult.outputs[enabledResult.rows[0].run_id].references[0]; + const withMetadata = await execution.request.get('/view', { params: { + filename: enabledReference.filename, subfolder: enabledReference.subfolder, type: enabledReference.folder_type, + } }); + expect(withMetadata.ok()).toBe(true); + expect((await withMetadata.body()).includes(Buffer.from('tEXtprompt'))).toBe(true); + expect((await execution.request.post('/pm_execution_fixture/metadata', { data: { disabled: true } })).ok()).toBe(true); + try { + const graph = images(45); + delete graph['3']; + const id = await submit(execution, graph); + const result = await persisted(execution, id, ['succeeded']); + const capture = result.outputs[result.rows[0].run_id]; + expectHistory(capture, await finalHistory(execution, id)); + expect(capture.references).toHaveLength(1); + const { filename, subfolder, folder_type: type } = capture.references[0]; + const response = await execution.request.get('/view', { params: { filename, subfolder, type } }); + expect(response.ok()).toBe(true); + const png = await response.body(); + expect(png.subarray(1, 4).toString()).toBe('PNG'); + expect(png.includes(Buffer.from('tEXtprompt'))).toBe(false); + } finally { + expect((await execution.request.post('/pm_execution_fixture/metadata', { data: { disabled: false } })).ok()).toBe(true); + } +}); + +test('text-only output reports a complete empty image set', async ({ execution }) => { + const id = await submit(execution, { '1': { class_type: 'PMHistoryCached', inputs: { value: 'text has no image locator' } } }); + const result = await persisted(execution, id, ['succeeded']); + const capture = result.outputs[result.rows[0].run_id]; + expectHistory(capture, await finalHistory(execution, id)); + expect(capture.references).toEqual([]); + expect(capture.reason).toBe('none_reported'); +}); + +test('batched outputs preserve PM usage while context membership stays unknown', async ({ execution }) => { + const graph = images(57, 2); + graph['10'] = { class_type: 'PMHistoryInputs', inputs: {} }; + graph['20'] = { class_type: 'PromptManagerPair', inputs: { clip: ['10', 0], text: `output usage ${randomUUID()}`, + negative_text: `negative ${randomUUID()}`, prepend_text: '', append_text: '' } }; + graph['30'] = { class_type: 'KSampler', inputs: { model: ['10', 1], latent_image: ['10', 2], + positive: ['20', 0], negative: ['20', 1], seed: 1, steps: 1, cfg: 1, sampler_name: 'euler', scheduler: 'normal', denoise: 1 } }; + graph['1'].inputs.latent = ['30', 0]; + const id = await submit(execution, graph); + const first = await persisted(execution, id, ['succeeded']); + const runId = first.rows[0].run_id; + expectHistory(first.outputs[runId], await finalHistory(execution, id)); + expect(first.outputs[runId].references).toHaveLength(4); + expect(first.captures[runId].contexts).toHaveLength(1); + expect(first.usage[runId]).toEqual([{ role: 'positive', run_count: 1 }, { role: 'negative', run_count: 0 }, { role: 'pair', run_count: 1 }]); + await submit(execution, graph, id); + const second = await persisted(execution, id, ['succeeded', 'succeeded']); + expect(second.outputs[second.rows[1].run_id].references).toEqual(first.outputs[runId].references); + expect(second.usage[runId]).toEqual([{ role: 'positive', run_count: 2 }, { role: 'negative', run_count: 0 }, { role: 'pair', run_count: 2 }]); +}); + +test('failed work after stock SaveImage preserves exactly the final reported partial-run outputs', async ({ execution }) => { + const id = await submit(execution, downstream(81, 'failure')); + const result = await persisted(execution, id, ['failed']); + const item = await finalHistory(execution, id); + expect(item.status.messages.some(([kind]) => kind === 'execution_error')).toBe(true); + expectHistory(result.outputs[result.rows[0].run_id], item); + expect(result.outputs[result.rows[0].run_id].references).toHaveLength(1); +}); + +test('interruption after stock SaveImage preserves exactly the final reported outputs', async ({ execution }) => { + await gate(execution, 'hold'); + try { + const id = await submit(execution, downstream(93, 'gate')); + await entered(execution); + expect((await execution.request.post('/interrupt', { data: {} })).ok()).toBe(true); + const result = await persisted(execution, id, ['interrupted']); + const item = await finalHistory(execution, id); + expect(item.status.messages.some(([kind]) => kind === 'execution_interrupted')).toBe(true); + expectHistory(result.outputs[result.rows[0].run_id], item); + expect(result.outputs[result.rows[0].run_id].references).toHaveLength(1); + } finally { await gate(execution, 'release'); } +}); + +test('completed image before pending gated work is not finalized early', async ({ execution }) => { + await gate(execution, 'hold'); + try { + const id = await submit(execution, downstream(107, 'gate')); + await entered(execution); + let pending; + await expect.poll(async () => { + pending = await readExecutionRuns(execution.databasePath, id); + return pending.rows.map(row => row.outcome); + }, { timeout: 10000 }).toEqual(['observed']); + expect(pending.outputs[pending.rows[0].run_id]).toMatchObject({ state: 'not_captured', references: [] }); + expect((await (await execution.request.get(`/history/${id}`)).json())[id]).toBeUndefined(); + await gate(execution, 'release'); + const result = await persisted(execution, id, ['succeeded']); + expectHistory(result.outputs[result.rows[0].run_id], await finalHistory(execution, id)); + expect(result.outputs[result.rows[0].run_id].references).toHaveLength(1); + } finally { await gate(execution, 'release'); } +}); + +test('stock output capture survives both renderer switches and workflow reload', async ({ execution, page, context }, testInfo) => { + await context.route('**/*', async route => { + const path = new URL(route.request().url()).pathname.replace(/^\/api\//, '/'); + if (path === '/settings' && route.request().method() === 'GET') { + const response = await route.fetch(); + return route.fulfill({ json: { ...await response.json(), + 'Comfy.VueNodes.Enabled': testInfo.project.metadata.initialRenderer, + 'Comfy.TutorialCompleted': true, 'Comfy.Workflow.Persist': false, + 'Comfy.Notification.ShowVersionUpdates': false } }); + } + return route.continue(); + }); + const errors = []; + page.on('pageerror', error => errors.push(error.message)); + await page.goto('/', { waitUntil: 'domcontentloaded' }); + await expect(page.getByRole('status', { name: 'Loading ComfyUI' })).toBeHidden({ timeout: 30000 }); + await page.waitForFunction(async () => { + const { app } = await import('/scripts/app.js'); + return !!app.graph && !!globalThis.LiteGraph?.registered_node_types?.PMHistoryImages; + }); + await expect.poll(() => page.evaluate(() => LiteGraph.vueNodesMode)).toBe(testInfo.project.metadata.initialRenderer); + await page.evaluate(async () => { + const { app } = await import('/scripts/app.js'); + app.graph.clear(); + const source = LiteGraph.createNode('PMHistoryImages'); + const saved = LiteGraph.createNode('SaveImage'); + const preview = LiteGraph.createNode('PreviewImage'); + for (const node of [source, saved, preview]) app.graph.add(node); + source.widgets.find(widget => widget.name === 'value').value = 119; + source.widgets.find(widget => widget.name === 'batch').value = 1; + saved.widgets.find(widget => widget.name === 'filename_prefix').value = 'renderer-evidence/image'; + source.connect(0, saved, 0); + source.connect(0, preview, 0); + }); + const captures = []; + for (const renderer of [testInfo.project.metadata.initialRenderer, !testInfo.project.metadata.initialRenderer, testInfo.project.metadata.initialRenderer]) { + await page.evaluate(async renderer => { + const { app } = await import('/scripts/app.js'); + await app.ui.settings.setSettingValue('Comfy.VueNodes.Enabled', renderer); + await app.loadGraphData(app.graph.serialize()); + }, renderer); + await expect.poll(() => page.evaluate(() => LiteGraph.vueNodesMode)).toBe(renderer); + const graph = await page.evaluate(async () => (await (await import('/scripts/app.js')).app.graphToPrompt()).output); + expect(Object.values(graph).map(node => node.class_type).sort()).toEqual(['PMHistoryImages', 'PreviewImage', 'SaveImage']); + const id = await submit(execution, graph); + const result = await persisted(execution, id, ['succeeded']); + const capture = result.outputs[result.rows[0].run_id]; + expectHistory(capture, await finalHistory(execution, id)); + expect(capture.references).toHaveLength(2); + captures.push(capture); + } + expect(captures[1].references).toEqual(captures[0].references); + expect(captures[2].references).toEqual(captures[0].references); + expect(errors).toEqual([]); +}); diff --git a/tests/comfyui/fixtures/ledger_nodes.py b/tests/comfyui/fixtures/ledger_nodes.py index e5d0588..31196f5 100644 --- a/tests/comfyui/fixtures/ledger_nodes.py +++ b/tests/comfyui/fixtures/ledger_nodes.py @@ -5,6 +5,7 @@ import time import nodes import execution +import torch from aiohttp import web import comfy.model_management @@ -152,6 +153,62 @@ NODE_CLASS_MAPPINGS.update( {"PMHistoryList": PMHistoryList, "PMHistoryTransform": PMHistoryTransform} ) + +class PMHistoryImages: + @classmethod + def INPUT_TYPES(cls) -> dict: + return { + "required": { + "value": ("INT", {"default": 0, "min": 0, "max": 255}), + "batch": ("INT", {"default": 1, "min": 1, "max": 2}), + }, + "optional": {"latent": ("LATENT",)}, + } + + RETURN_TYPES = ("IMAGE",) + FUNCTION = "execute" + CATEGORY = "testing/execution-history" + + def execute(self, value: int, batch: int, latent: dict | None = None) -> tuple: + return (torch.full((batch, 8, 8, 3), value / 255.0, device="cpu"),) + + +class PMHistoryImageOutput: + @classmethod + def INPUT_TYPES(cls) -> dict: + return { + "required": { + "images": ("IMAGE",), + "mode": (["failure", "gate"],), + } + } + + RETURN_TYPES = () + FUNCTION = "execute" + CATEGORY = "testing/execution-history" + OUTPUT_NODE = True + + def execute(self, images: torch.Tensor, mode: str) -> tuple: + if mode == "failure": + raise ValueError("owned failure after real image output") + PMHistoryGate().execute("owned gate after real image output") + return () + + +NODE_CLASS_MAPPINGS.update( + {"PMHistoryImages": PMHistoryImages, "PMHistoryImageOutput": PMHistoryImageOutput} +) + + +@PromptServer.instance.routes.post("/pm_execution_fixture/metadata") +async def metadata(request: web.Request) -> web.Response: + disabled = (await request.json())["disabled"] + if type(disabled) is not bool: + raise web.HTTPBadRequest() + nodes.args.disable_metadata = disabled + return web.json_response({"disabled": disabled}) + + _INJECTION_ORIGINALS = {} diff --git a/tests/comfyui/read_execution_history.py b/tests/comfyui/read_execution_history.py index 9974bf2..ccbc106 100644 --- a/tests/comfyui/read_execution_history.py +++ b/tests/comfyui/read_execution_history.py @@ -11,9 +11,12 @@ from uuid import UUID sys.path.insert(0, str(Path(__file__).resolve().parents[2])) from database.execution_contexts import ExecutionContextOperations # noqa: E402 from database.execution_history import ExecutionHistoryOperations # noqa: E402 +from database.execution_outputs import ExecutionOutputOperations # noqa: E402 -class _ReadOnlyHistory(ExecutionContextOperations, ExecutionHistoryOperations): +class _ReadOnlyHistory( + ExecutionContextOperations, ExecutionHistoryOperations, ExecutionOutputOperations +): def __init__(self, connection: sqlite3.Connection) -> None: self.model = SimpleNamespace(get_connection=lambda: connection) @@ -79,6 +82,10 @@ def read_runs(database: str, job_id: str, session_id: str | None = None) -> dict "captures": captures, "usage": usage, "snapshotIds": snapshot_ids, + "outputs": { + row["run_id"]: reader.get_execution_outputs(row["run_id"]) + for row in rows + }, } diff --git a/tests/vendor/comfyui-config.test.mjs b/tests/vendor/comfyui-config.test.mjs index ec2d3c8..8e87063 100644 --- a/tests/vendor/comfyui-config.test.mjs +++ b/tests/vendor/comfyui-config.test.mjs @@ -24,12 +24,15 @@ test('execution script discovers only the new matrix and preserves all 78 stock const stock = list('tools/vendor/comfyui.config.mjs'); assert.match(stock, /Total: 78 tests in 7 files/); assert.doesNotMatch(stock, /execution-history/); + assert.doesNotMatch(stock, /execution-outputs/); const executions = list('tools/vendor/playwright.execution.config.mjs'); assert.match(executions, /\[classic-start\]/); assert.match(executions, /\[nodes2-start\]/); - assert.match(executions, /Total: 26 tests in 1 file/); + assert.match(executions, /Total: 42 tests in 2 files/); for (const project of ['classic-start', 'nodes2-start']) { - assert.equal(executions.split('\n').filter(line => line.includes(`[${project}]`)).length, 13); + const cases = executions.split('\n').filter(line => line.includes(`[${project}]`)); + assert.equal(cases.filter(line => line.includes('execution-history.spec.mjs')).length, 13); + assert.equal(cases.filter(line => line.includes('execution-outputs.spec.mjs')).length, 8); } assert.doesNotMatch(executions, /prompt-manager.spec/); }); diff --git a/tests/vendor/comfyui-host.test.mjs b/tests/vendor/comfyui-host.test.mjs index 78e11d5..a39a289 100644 --- a/tests/vendor/comfyui-host.test.mjs +++ b/tests/vendor/comfyui-host.test.mjs @@ -97,6 +97,7 @@ test('read-only ledger helper distinguishes absent schema from missing, corrupt from database.operations import PromptDatabase from utils.execution_history_events import RunStart, make_finish from utils.execution_contexts import ContextBundle, PromptSnapshot, RecipeContext, TextEvidence +from utils.execution_outputs import OutputBundle, OutputReference from types import SimpleNamespace db = PromptDatabase(sys.argv[1]) @@ -123,6 +124,10 @@ try: db.write_execution_contexts(sys.argv[5], epoch, ContextBundle( (RecipeContext('4', snapshot(' positive '), snapshot('negative'), 'inferred'),), 'complete', 'captured')) + db.write_execution_outputs_for_finish(make_finish(sys.argv[3], 1, epoch, + SimpleNamespace(messages=[('execution_success', {'prompt_id': sys.argv[2]})]), 200), + OutputBundle((OutputReference('9', 0, 'temp', 'nested', 'preview.png', 'posix', 'a' * 64),), + 'complete', 'reported')) print(epoch) finally: db.close_all() @@ -138,6 +143,12 @@ finally: node_id: null, node_type: null, exception_type: null, }]); assert.deepEqual(Object.keys(result.captures), [runId]); + assert.deepEqual(result.outputs, { [runId]: { + state: 'complete', reason: 'reported', format_version: 1, source: 'host_final_history', + references: [{ output_node_id: '9', image_ordinal: 0, folder_type: 'temp', + subfolder: 'nested', filename: 'preview.png', path_style: 'posix', root_key: 'a'.repeat(64), + file_identity: 'unverified', newly_generated: 'unknown', context_membership: 'unknown', transient: true }], + } }); const capture = result.captures[runId]; assert.equal(capture.contexts.length, 1); const contextId = capture.contexts[0].context_id; @@ -170,8 +181,11 @@ finally: }); assert.deepEqual(unfiltered.usage[uncapturedId], []); assert.deepEqual(unfiltered.snapshotIds[uncapturedId], []); + assert.deepEqual(unfiltered.outputs[uncapturedId], { + state: 'not_captured', reason: null, format_version: null, source: null, references: [], + }); assert.deepEqual(await readExecutionRuns(database, job, '60000000-0000-4000-8000-000000000001'), { - schemaAvailable: true, rows: [], captures: {}, usage: {}, snapshotIds: {}, + schemaAvailable: true, rows: [], captures: {}, usage: {}, snapshotIds: {}, outputs: {}, }); assert.deepEqual(await readFile(database), before); await assert.rejects(readExecutionRuns(database, "' OR 1=1 --")); diff --git a/tools/vendor/comfyui.config.mjs b/tools/vendor/comfyui.config.mjs index 1d2f206..6d7dc76 100644 --- a/tools/vendor/comfyui.config.mjs +++ b/tools/vendor/comfyui.config.mjs @@ -14,7 +14,7 @@ const servers = [{ export default defineConfig({ testDir: '../../tests/comfyui', - testIgnore: 'execution-history.spec.mjs', + testIgnore: ['execution-history.spec.mjs', 'execution-outputs.spec.mjs'], outputDir: './test-results/comfyui', timeout: 60000, workers: 1, diff --git a/tools/vendor/playwright.execution.config.mjs b/tools/vendor/playwright.execution.config.mjs index c57349b..0de5792 100644 --- a/tools/vendor/playwright.execution.config.mjs +++ b/tools/vendor/playwright.execution.config.mjs @@ -6,7 +6,7 @@ if (host.href !== 'http://127.0.0.1:8196/') throw new Error('Execution tests req export default defineConfig({ testDir: '../../tests/comfyui', - testMatch: 'execution-history.spec.mjs', + testMatch: ['execution-history.spec.mjs', 'execution-outputs.spec.mjs'], outputDir: './test-results/executions/cases', timeout: 60000, workers: 1,