test(history): verify run output evidence on real ComfyUI
This commit is contained in:
+38
-8
@@ -200,8 +200,8 @@ loading a workflow or queueing generation.
|
||||
|
||||
## Durable execution ledger
|
||||
|
||||
`npm run --prefix tools/vendor test:executions` runs 26 real execution cases
|
||||
(13 each in Classic and Nodes 2.0, retaining the original ten ledger cases),
|
||||
`npm run --prefix tools/vendor test:executions` runs 42 real execution cases
|
||||
(21 each in Classic and Nodes 2.0, retaining all 26 prior execution cases),
|
||||
alongside the existing 78 stock host cases and six pack compatibility cases. Run
|
||||
these suites serially because they share report cleanup and owned host resources. Use the same pinned disposable `PM_COMFY_ROOT`,
|
||||
`PM_TEST_PYTHON` and Playwright browser installation as `test:comfyui`:
|
||||
@@ -217,8 +217,9 @@ This suite reserves `http://127.0.0.1:8196/` and never attaches to an existing
|
||||
server. Each renderer worker launches a CPU/offline host with independent
|
||||
configuration, database, user and output directories. The shared launcher installs
|
||||
tracked snapshots of PromptManager and a tiny temporary Git-tracked fixture pack.
|
||||
Only that owned test pack provides cached output, deliberate failure, an
|
||||
interruptible queue gate, deterministic CLIP/model/latent inputs and control routes.
|
||||
Only that owned test pack provides cached text, deliberate failure, an
|
||||
interruptible queue gate, deterministic CLIP/model/latent and CPU tensor inputs,
|
||||
and control routes.
|
||||
It replaces external sampler inference and trigger injection only inside the
|
||||
disposable host; real registered PromptManager classes, queue, executor and cache
|
||||
remain active. Its cache control resets the real executor on its worker to exercise
|
||||
@@ -236,8 +237,24 @@ mode, exact UUID job/session filters and a 256-row limit; absent ledger schema i
|
||||
explicit, while missing/corrupt databases and other read errors fail the suite.
|
||||
Sensitive diagnostic and extra-data sentinels must never appear in lifecycle
|
||||
rows. Exact prompt fields are deliberately persisted only in context snapshots.
|
||||
The same read-only helper calls the internal `get_execution_contexts` reader for
|
||||
requested run IDs and inspects their snapshot identities and current usage counts.
|
||||
The same read-only helper calls the internal `get_execution_contexts` and
|
||||
`get_execution_outputs` readers for requested run IDs and inspects their snapshot
|
||||
identities and current usage counts. Its `outputs` map is keyed by immutable run
|
||||
ID, alongside the existing `rows`, `captures`, `usage` and `snapshotIds` results.
|
||||
|
||||
`execution-outputs.spec.mjs` runs actual stock SaveImage and PreviewImage nodes
|
||||
from tiny CPU tensors. It compares each attempt's persisted descriptors with that
|
||||
attempt's final host history, including A,A,B batches, multiple output nodes,
|
||||
cached locators, reused public UUIDs, PNG metadata disabled, graphs without PM
|
||||
nodes and text-only outputs. The pinned SaveImage returns an IMAGE socket; linking
|
||||
it to the owned failure/gate node orders the partial-run tests explicitly. A held
|
||||
downstream gate proves completed image work does not finalize the run early.
|
||||
Other cases verify real output capture across startup in each renderer, switching
|
||||
both ways and workflow reload. PM usage counts retain their existing per-job
|
||||
behavior. References identify reported locators only: file identity is unverified,
|
||||
new generation and context membership are unknown, and temp references are
|
||||
transient. The stock suite excludes both execution files; discovery retains its
|
||||
78 cases and the six pack cases unchanged.
|
||||
|
||||
For a meaningful historical failure, create a separate disposable detached
|
||||
checkout at `fdba6fd0126a8a40cbd6f3e50ed4c5de8991efd4` and pass its path to
|
||||
@@ -285,8 +302,21 @@ Fault injection for restore fencing, inherited async capability revocation, queu
|
||||
finalization, epoch and generation revocation, provider-local cache entries, eviction/reset, malformed or
|
||||
oversized payloads and missing host interfaces lives in the focused Python
|
||||
`test_execution*.py` suites. These use disposable storage and mocked host interfaces;
|
||||
they are distinct from the pinned real executor/cache checks above. No output-file
|
||||
associations or chronological timeline are introduced. Deploying the Python changes
|
||||
they are distinct from the pinned real executor/cache checks above. Run-owned
|
||||
output descriptors do not create gallery image associations or a chronological
|
||||
timeline. Deploying the Python changes
|
||||
requires a ComfyUI restart. After restart, queue three new jobs A,A,B with identical
|
||||
positive fields and two negative recipes; existing counters should advance as usual,
|
||||
and internal context inspection should find three attempts and two combinations.
|
||||
For a live output check after that manual restart:
|
||||
|
||||
- Queue SaveImage and PreviewImage with a batch of two, then submit an unchanged
|
||||
cached repeat and a changed tensor/input under new jobs.
|
||||
- Inspect each run's internal output capture against its final ComfyUI history;
|
||||
cached references may repeat, while the attempts remain distinct.
|
||||
- Switch Classic to Nodes 2.0 and back, reload the workflow and queue again.
|
||||
- Verify prompt/pair counters and gallery links retain their existing behavior;
|
||||
output references alone do not prove file bytes or prompt membership.
|
||||
|
||||
Automated checks use only managed disposable hosts. They never restart or upgrade
|
||||
the personal ComfyUI server.
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Durable Prompt History
|
||||
|
||||
Status: the durable job ledger is implemented as the first slice of
|
||||
[#209](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/209). Prompt
|
||||
contexts, output associations and the chronological timeline remain pending;
|
||||
#209 stays open.
|
||||
Status: the durable job ledger, immutable prompt contexts and run-owned output
|
||||
references are implemented for
|
||||
[#209](https://github.com/ComfyAssets/ComfyUI_PromptManager/issues/209). Gallery
|
||||
image associations and the chronological timeline remain pending; #209 stays open.
|
||||
|
||||
## Recorded jobs
|
||||
|
||||
@@ -21,7 +21,7 @@ outcomes. Missing or conflicting terminal evidence remains unknown; a process
|
||||
crash can leave an observed row without an invented completion time. Bounded
|
||||
safe diagnostics retain node ID/type and exception type, never graph text,
|
||||
arbitrary extra data, credentials, exception messages, traceback, inputs or
|
||||
output payloads. A bounded background writer keeps storage failures from
|
||||
arbitrary output payloads. A bounded background writer keeps storage failures from
|
||||
changing host execution and reports capture gaps rather than inventing rows.
|
||||
|
||||
Existing databases begin with zero execution rows. Migration and backup restore
|
||||
@@ -47,16 +47,38 @@ aggregate uses. It does not expose the ledger as chronological dots or repeat-ru
|
||||
branches. `PromptTracker` identities and PNG recipe evidence remain separate
|
||||
from durable host job identity.
|
||||
|
||||
## Immutable contexts and output references
|
||||
|
||||
Prompt contexts retain bounded raw/effective positive, negative and composition
|
||||
snapshots with nullable recipe navigation hints and explicit evidence provenance.
|
||||
Unknown text remains SQL NULL; a known empty negative remains empty text. Cached
|
||||
evidence belongs to its exact cache provider and attempt, rather than inheriting
|
||||
unrelated recent prompts. Deletion and restore can remove navigation hints without
|
||||
rewriting historical snapshots or rebinding them to current library rows.
|
||||
|
||||
Schema v3 adds `execution_output_capture` and `execution_run_outputs` without
|
||||
backfilling older runs or modifying gallery image links. At final queue completion,
|
||||
the writer attaches bounded output evidence from that attempt's final host history
|
||||
to its immutable run ID. The internal `get_execution_outputs(run_id)` reader
|
||||
returns state, reason, format version, source and ordered references. A complete
|
||||
capture with `none_reported` means the final host history reported no images;
|
||||
`not_captured`, partial, unavailable and conflicting evidence remain explicit.
|
||||
Failed and interrupted jobs can still have complete evidence for images reported
|
||||
before termination. Running jobs do not gain a final capture merely because an
|
||||
image node finished.
|
||||
|
||||
Each reference records output node ID, image ordinal, folder type, subfolder,
|
||||
filename, path style and an opaque configured-root key. A cached repeat can retain
|
||||
the same locators under a new run. Locators do not verify file bytes, continued
|
||||
existence, a newly generated file or membership in any prompt context: the reader
|
||||
reports file identity as `unverified`, generation and context membership as
|
||||
`unknown`, and temp outputs as transient. Capture never reads, resolves or serves
|
||||
the files and adds no public route. PNG metadata is not required. Same-path file
|
||||
replacement and existing gallery provenance retain their separate semantics.
|
||||
|
||||
## Remaining history work
|
||||
|
||||
Design bounded execution contexts with nullable recipe references, immutable
|
||||
raw/effective positive, negative and composition snapshots and explicit
|
||||
inferred/recorded/unknown provenance. Cached nodes must not inherit unrelated
|
||||
recent prompt evidence. Define deletion and restore behavior before connecting
|
||||
saved recipes to historical snapshots.
|
||||
|
||||
Add exact output references and a deduplicated run/image relationship without
|
||||
claiming cached outputs were newly created. Preserve existing image provenance
|
||||
and same-path replacement protection. Only after those semantics are tested can
|
||||
the timeline honestly show prompt-level execution chronology, batched outputs
|
||||
and repeat-run branches.
|
||||
Define a deduplicated run/gallery-image relationship that preserves existing image
|
||||
provenance and same-path replacement protection. Add a chronological timeline only
|
||||
after its navigation and evidence semantics can honestly represent prompt-level
|
||||
execution chronology, batched outputs and repeat-run branches.
|
||||
|
||||
@@ -0,0 +1,263 @@
|
||||
import { randomUUID } from 'node:crypto';
|
||||
import { expect } from '../../tools/vendor/node_modules/@playwright/test/index.mjs';
|
||||
import { test, readExecutionRuns } from './execution-fixture.mjs';
|
||||
|
||||
async function resetCache({ execution }) {
|
||||
expect((await execution.request.post('/pm_execution_fixture/cache', { data: { mode: 'RAM_PRESSURE' } })).ok()).toBe(true);
|
||||
}
|
||||
|
||||
test.beforeEach(resetCache);
|
||||
test.afterAll(resetCache);
|
||||
|
||||
function images(value, batch = 1) {
|
||||
return {
|
||||
'1': { class_type: 'PMHistoryImages', inputs: { value, batch } },
|
||||
'2': { class_type: 'SaveImage', inputs: { images: ['1', 0], filename_prefix: 'output-evidence/nested/image' } },
|
||||
'3': { class_type: 'PreviewImage', inputs: { images: ['1', 0] } },
|
||||
};
|
||||
}
|
||||
|
||||
async function submit(execution, prompt, id = randomUUID()) {
|
||||
const response = await execution.request.post('/prompt', { data: { prompt, prompt_id: id } });
|
||||
expect(response.ok()).toBe(true);
|
||||
expect((await response.json()).prompt_id).toBe(id);
|
||||
return id;
|
||||
}
|
||||
|
||||
async function persisted(execution, id, outcomes) {
|
||||
let result;
|
||||
await expect.poll(async () => {
|
||||
result = await readExecutionRuns(execution.databasePath, id);
|
||||
return { outcomes: result.rows.map(row => row.outcome),
|
||||
states: result.rows.map(row => result.outputs[row.run_id].state) };
|
||||
}, { timeout: 15000, message: 'Exact attempts and final output evidence reach SQLite' }).toEqual({
|
||||
outcomes, states: outcomes.map(() => 'complete'),
|
||||
});
|
||||
return result;
|
||||
}
|
||||
|
||||
async function finalHistory(execution, id) {
|
||||
const response = await execution.request.get(`/history/${id}`);
|
||||
expect(response.ok()).toBe(true);
|
||||
const item = (await response.json())[id];
|
||||
expect(item.prompt[1]).toBe(id);
|
||||
return item;
|
||||
}
|
||||
|
||||
function expectHistory(capture, history) {
|
||||
const reported = Object.entries(history.outputs).flatMap(([node, output]) =>
|
||||
(output.images || []).map((image, ordinal) => ({ output_node_id: node, image_ordinal: ordinal,
|
||||
folder_type: image.type, subfolder: image.subfolder, filename: image.filename })));
|
||||
reported.sort((a, b) => a.output_node_id.localeCompare(b.output_node_id) || a.image_ordinal - b.image_ordinal);
|
||||
expect(capture).toMatchObject({ state: 'complete', reason: reported.length ? 'reported' : 'none_reported',
|
||||
source: 'host_final_history', format_version: 1 });
|
||||
expect(capture.references.map(({ output_node_id, image_ordinal, folder_type, subfolder, filename }) =>
|
||||
({ output_node_id, image_ordinal, folder_type, subfolder, filename }))).toEqual(reported);
|
||||
for (const reference of capture.references) {
|
||||
expect(reference).toMatchObject({ path_style: 'posix', file_identity: 'unverified',
|
||||
newly_generated: 'unknown', context_membership: 'unknown', transient: reference.folder_type === 'temp' });
|
||||
expect(reference.root_key).toMatch(/^[a-f0-9]{64}$/);
|
||||
}
|
||||
}
|
||||
|
||||
async function gate(execution, action) {
|
||||
expect((await execution.request.post(`/pm_execution_fixture/gate/${action}`)).ok()).toBe(true);
|
||||
}
|
||||
|
||||
async function entered(execution) {
|
||||
await expect.poll(async () => (await (await execution.request.get('/pm_execution_fixture/gate')).json()).entered,
|
||||
{ timeout: 10000 }).toBe(true);
|
||||
}
|
||||
|
||||
function downstream(value, mode) {
|
||||
const graph = images(value);
|
||||
delete graph['3'];
|
||||
graph['4'] = { class_type: 'PMHistoryImageOutput', inputs: { images: ['2', 0], mode } };
|
||||
return graph;
|
||||
}
|
||||
|
||||
test('headless stock SaveImage and PreviewImage retain A,A,B batches under a reused public UUID', async ({ execution }) => {
|
||||
const id = randomUUID();
|
||||
const graph = images(31, 2);
|
||||
let result;
|
||||
for (const [index, value] of [31, 31, 73].entries()) {
|
||||
graph['1'].inputs.value = value;
|
||||
await submit(execution, graph, id);
|
||||
result = await persisted(execution, id, Array(index + 1).fill('succeeded'));
|
||||
const item = await finalHistory(execution, id);
|
||||
const row = result.rows[index];
|
||||
expectHistory(result.outputs[row.run_id], item);
|
||||
expect(result.outputs[row.run_id].references).toHaveLength(4);
|
||||
expect(result.captures[row.run_id].contexts).toEqual([]);
|
||||
expect(result.usage[row.run_id]).toEqual([]);
|
||||
if (index === 1) expect(item.status.messages.find(([kind]) => kind === 'execution_cached')[1].nodes).toEqual(expect.arrayContaining(['1', '2', '3']));
|
||||
}
|
||||
const attempts = result.rows;
|
||||
expect(new Set(attempts.map(row => row.run_id)).size).toBe(3);
|
||||
expect(new Set(attempts.map(row => row.queue_item_id)).size).toBe(3);
|
||||
const captures = attempts.map(row => result.outputs[row.run_id]);
|
||||
expect(captures.map(capture => capture.state)).toEqual(['complete', 'complete', 'complete']);
|
||||
expect(captures[1].references).toEqual(captures[0].references);
|
||||
expect(captures[2].references).not.toEqual(captures[0].references);
|
||||
expect(captures[0].references[0].context_membership).toBe('unknown');
|
||||
expect(captures[0].references.filter(reference => reference.transient)).toHaveLength(2);
|
||||
});
|
||||
|
||||
test('stock image evidence survives disabled PNG metadata', async ({ execution }) => {
|
||||
const enabled = images(44);
|
||||
delete enabled['3'];
|
||||
const enabledId = await submit(execution, enabled);
|
||||
const enabledResult = await persisted(execution, enabledId, ['succeeded']);
|
||||
const enabledReference = enabledResult.outputs[enabledResult.rows[0].run_id].references[0];
|
||||
const withMetadata = await execution.request.get('/view', { params: {
|
||||
filename: enabledReference.filename, subfolder: enabledReference.subfolder, type: enabledReference.folder_type,
|
||||
} });
|
||||
expect(withMetadata.ok()).toBe(true);
|
||||
expect((await withMetadata.body()).includes(Buffer.from('tEXtprompt'))).toBe(true);
|
||||
expect((await execution.request.post('/pm_execution_fixture/metadata', { data: { disabled: true } })).ok()).toBe(true);
|
||||
try {
|
||||
const graph = images(45);
|
||||
delete graph['3'];
|
||||
const id = await submit(execution, graph);
|
||||
const result = await persisted(execution, id, ['succeeded']);
|
||||
const capture = result.outputs[result.rows[0].run_id];
|
||||
expectHistory(capture, await finalHistory(execution, id));
|
||||
expect(capture.references).toHaveLength(1);
|
||||
const { filename, subfolder, folder_type: type } = capture.references[0];
|
||||
const response = await execution.request.get('/view', { params: { filename, subfolder, type } });
|
||||
expect(response.ok()).toBe(true);
|
||||
const png = await response.body();
|
||||
expect(png.subarray(1, 4).toString()).toBe('PNG');
|
||||
expect(png.includes(Buffer.from('tEXtprompt'))).toBe(false);
|
||||
} finally {
|
||||
expect((await execution.request.post('/pm_execution_fixture/metadata', { data: { disabled: false } })).ok()).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test('text-only output reports a complete empty image set', async ({ execution }) => {
|
||||
const id = await submit(execution, { '1': { class_type: 'PMHistoryCached', inputs: { value: 'text has no image locator' } } });
|
||||
const result = await persisted(execution, id, ['succeeded']);
|
||||
const capture = result.outputs[result.rows[0].run_id];
|
||||
expectHistory(capture, await finalHistory(execution, id));
|
||||
expect(capture.references).toEqual([]);
|
||||
expect(capture.reason).toBe('none_reported');
|
||||
});
|
||||
|
||||
test('batched outputs preserve PM usage while context membership stays unknown', async ({ execution }) => {
|
||||
const graph = images(57, 2);
|
||||
graph['10'] = { class_type: 'PMHistoryInputs', inputs: {} };
|
||||
graph['20'] = { class_type: 'PromptManagerPair', inputs: { clip: ['10', 0], text: `output usage ${randomUUID()}`,
|
||||
negative_text: `negative ${randomUUID()}`, prepend_text: '', append_text: '' } };
|
||||
graph['30'] = { class_type: 'KSampler', inputs: { model: ['10', 1], latent_image: ['10', 2],
|
||||
positive: ['20', 0], negative: ['20', 1], seed: 1, steps: 1, cfg: 1, sampler_name: 'euler', scheduler: 'normal', denoise: 1 } };
|
||||
graph['1'].inputs.latent = ['30', 0];
|
||||
const id = await submit(execution, graph);
|
||||
const first = await persisted(execution, id, ['succeeded']);
|
||||
const runId = first.rows[0].run_id;
|
||||
expectHistory(first.outputs[runId], await finalHistory(execution, id));
|
||||
expect(first.outputs[runId].references).toHaveLength(4);
|
||||
expect(first.captures[runId].contexts).toHaveLength(1);
|
||||
expect(first.usage[runId]).toEqual([{ role: 'positive', run_count: 1 }, { role: 'negative', run_count: 0 }, { role: 'pair', run_count: 1 }]);
|
||||
await submit(execution, graph, id);
|
||||
const second = await persisted(execution, id, ['succeeded', 'succeeded']);
|
||||
expect(second.outputs[second.rows[1].run_id].references).toEqual(first.outputs[runId].references);
|
||||
expect(second.usage[runId]).toEqual([{ role: 'positive', run_count: 2 }, { role: 'negative', run_count: 0 }, { role: 'pair', run_count: 2 }]);
|
||||
});
|
||||
|
||||
test('failed work after stock SaveImage preserves exactly the final reported partial-run outputs', async ({ execution }) => {
|
||||
const id = await submit(execution, downstream(81, 'failure'));
|
||||
const result = await persisted(execution, id, ['failed']);
|
||||
const item = await finalHistory(execution, id);
|
||||
expect(item.status.messages.some(([kind]) => kind === 'execution_error')).toBe(true);
|
||||
expectHistory(result.outputs[result.rows[0].run_id], item);
|
||||
expect(result.outputs[result.rows[0].run_id].references).toHaveLength(1);
|
||||
});
|
||||
|
||||
test('interruption after stock SaveImage preserves exactly the final reported outputs', async ({ execution }) => {
|
||||
await gate(execution, 'hold');
|
||||
try {
|
||||
const id = await submit(execution, downstream(93, 'gate'));
|
||||
await entered(execution);
|
||||
expect((await execution.request.post('/interrupt', { data: {} })).ok()).toBe(true);
|
||||
const result = await persisted(execution, id, ['interrupted']);
|
||||
const item = await finalHistory(execution, id);
|
||||
expect(item.status.messages.some(([kind]) => kind === 'execution_interrupted')).toBe(true);
|
||||
expectHistory(result.outputs[result.rows[0].run_id], item);
|
||||
expect(result.outputs[result.rows[0].run_id].references).toHaveLength(1);
|
||||
} finally { await gate(execution, 'release'); }
|
||||
});
|
||||
|
||||
test('completed image before pending gated work is not finalized early', async ({ execution }) => {
|
||||
await gate(execution, 'hold');
|
||||
try {
|
||||
const id = await submit(execution, downstream(107, 'gate'));
|
||||
await entered(execution);
|
||||
let pending;
|
||||
await expect.poll(async () => {
|
||||
pending = await readExecutionRuns(execution.databasePath, id);
|
||||
return pending.rows.map(row => row.outcome);
|
||||
}, { timeout: 10000 }).toEqual(['observed']);
|
||||
expect(pending.outputs[pending.rows[0].run_id]).toMatchObject({ state: 'not_captured', references: [] });
|
||||
expect((await (await execution.request.get(`/history/${id}`)).json())[id]).toBeUndefined();
|
||||
await gate(execution, 'release');
|
||||
const result = await persisted(execution, id, ['succeeded']);
|
||||
expectHistory(result.outputs[result.rows[0].run_id], await finalHistory(execution, id));
|
||||
expect(result.outputs[result.rows[0].run_id].references).toHaveLength(1);
|
||||
} finally { await gate(execution, 'release'); }
|
||||
});
|
||||
|
||||
test('stock output capture survives both renderer switches and workflow reload', async ({ execution, page, context }, testInfo) => {
|
||||
await context.route('**/*', async route => {
|
||||
const path = new URL(route.request().url()).pathname.replace(/^\/api\//, '/');
|
||||
if (path === '/settings' && route.request().method() === 'GET') {
|
||||
const response = await route.fetch();
|
||||
return route.fulfill({ json: { ...await response.json(),
|
||||
'Comfy.VueNodes.Enabled': testInfo.project.metadata.initialRenderer,
|
||||
'Comfy.TutorialCompleted': true, 'Comfy.Workflow.Persist': false,
|
||||
'Comfy.Notification.ShowVersionUpdates': false } });
|
||||
}
|
||||
return route.continue();
|
||||
});
|
||||
const errors = [];
|
||||
page.on('pageerror', error => errors.push(error.message));
|
||||
await page.goto('/', { waitUntil: 'domcontentloaded' });
|
||||
await expect(page.getByRole('status', { name: 'Loading ComfyUI' })).toBeHidden({ timeout: 30000 });
|
||||
await page.waitForFunction(async () => {
|
||||
const { app } = await import('/scripts/app.js');
|
||||
return !!app.graph && !!globalThis.LiteGraph?.registered_node_types?.PMHistoryImages;
|
||||
});
|
||||
await expect.poll(() => page.evaluate(() => LiteGraph.vueNodesMode)).toBe(testInfo.project.metadata.initialRenderer);
|
||||
await page.evaluate(async () => {
|
||||
const { app } = await import('/scripts/app.js');
|
||||
app.graph.clear();
|
||||
const source = LiteGraph.createNode('PMHistoryImages');
|
||||
const saved = LiteGraph.createNode('SaveImage');
|
||||
const preview = LiteGraph.createNode('PreviewImage');
|
||||
for (const node of [source, saved, preview]) app.graph.add(node);
|
||||
source.widgets.find(widget => widget.name === 'value').value = 119;
|
||||
source.widgets.find(widget => widget.name === 'batch').value = 1;
|
||||
saved.widgets.find(widget => widget.name === 'filename_prefix').value = 'renderer-evidence/image';
|
||||
source.connect(0, saved, 0);
|
||||
source.connect(0, preview, 0);
|
||||
});
|
||||
const captures = [];
|
||||
for (const renderer of [testInfo.project.metadata.initialRenderer, !testInfo.project.metadata.initialRenderer, testInfo.project.metadata.initialRenderer]) {
|
||||
await page.evaluate(async renderer => {
|
||||
const { app } = await import('/scripts/app.js');
|
||||
await app.ui.settings.setSettingValue('Comfy.VueNodes.Enabled', renderer);
|
||||
await app.loadGraphData(app.graph.serialize());
|
||||
}, renderer);
|
||||
await expect.poll(() => page.evaluate(() => LiteGraph.vueNodesMode)).toBe(renderer);
|
||||
const graph = await page.evaluate(async () => (await (await import('/scripts/app.js')).app.graphToPrompt()).output);
|
||||
expect(Object.values(graph).map(node => node.class_type).sort()).toEqual(['PMHistoryImages', 'PreviewImage', 'SaveImage']);
|
||||
const id = await submit(execution, graph);
|
||||
const result = await persisted(execution, id, ['succeeded']);
|
||||
const capture = result.outputs[result.rows[0].run_id];
|
||||
expectHistory(capture, await finalHistory(execution, id));
|
||||
expect(capture.references).toHaveLength(2);
|
||||
captures.push(capture);
|
||||
}
|
||||
expect(captures[1].references).toEqual(captures[0].references);
|
||||
expect(captures[2].references).toEqual(captures[0].references);
|
||||
expect(errors).toEqual([]);
|
||||
});
|
||||
@@ -5,6 +5,7 @@ import time
|
||||
|
||||
import nodes
|
||||
import execution
|
||||
import torch
|
||||
|
||||
from aiohttp import web
|
||||
import comfy.model_management
|
||||
@@ -152,6 +153,62 @@ NODE_CLASS_MAPPINGS.update(
|
||||
{"PMHistoryList": PMHistoryList, "PMHistoryTransform": PMHistoryTransform}
|
||||
)
|
||||
|
||||
|
||||
class PMHistoryImages:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls) -> dict:
|
||||
return {
|
||||
"required": {
|
||||
"value": ("INT", {"default": 0, "min": 0, "max": 255}),
|
||||
"batch": ("INT", {"default": 1, "min": 1, "max": 2}),
|
||||
},
|
||||
"optional": {"latent": ("LATENT",)},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("IMAGE",)
|
||||
FUNCTION = "execute"
|
||||
CATEGORY = "testing/execution-history"
|
||||
|
||||
def execute(self, value: int, batch: int, latent: dict | None = None) -> tuple:
|
||||
return (torch.full((batch, 8, 8, 3), value / 255.0, device="cpu"),)
|
||||
|
||||
|
||||
class PMHistoryImageOutput:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls) -> dict:
|
||||
return {
|
||||
"required": {
|
||||
"images": ("IMAGE",),
|
||||
"mode": (["failure", "gate"],),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ()
|
||||
FUNCTION = "execute"
|
||||
CATEGORY = "testing/execution-history"
|
||||
OUTPUT_NODE = True
|
||||
|
||||
def execute(self, images: torch.Tensor, mode: str) -> tuple:
|
||||
if mode == "failure":
|
||||
raise ValueError("owned failure after real image output")
|
||||
PMHistoryGate().execute("owned gate after real image output")
|
||||
return ()
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS.update(
|
||||
{"PMHistoryImages": PMHistoryImages, "PMHistoryImageOutput": PMHistoryImageOutput}
|
||||
)
|
||||
|
||||
|
||||
@PromptServer.instance.routes.post("/pm_execution_fixture/metadata")
|
||||
async def metadata(request: web.Request) -> web.Response:
|
||||
disabled = (await request.json())["disabled"]
|
||||
if type(disabled) is not bool:
|
||||
raise web.HTTPBadRequest()
|
||||
nodes.args.disable_metadata = disabled
|
||||
return web.json_response({"disabled": disabled})
|
||||
|
||||
|
||||
_INJECTION_ORIGINALS = {}
|
||||
|
||||
|
||||
|
||||
@@ -11,9 +11,12 @@ from uuid import UUID
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
from database.execution_contexts import ExecutionContextOperations # noqa: E402
|
||||
from database.execution_history import ExecutionHistoryOperations # noqa: E402
|
||||
from database.execution_outputs import ExecutionOutputOperations # noqa: E402
|
||||
|
||||
|
||||
class _ReadOnlyHistory(ExecutionContextOperations, ExecutionHistoryOperations):
|
||||
class _ReadOnlyHistory(
|
||||
ExecutionContextOperations, ExecutionHistoryOperations, ExecutionOutputOperations
|
||||
):
|
||||
def __init__(self, connection: sqlite3.Connection) -> None:
|
||||
self.model = SimpleNamespace(get_connection=lambda: connection)
|
||||
|
||||
@@ -79,6 +82,10 @@ def read_runs(database: str, job_id: str, session_id: str | None = None) -> dict
|
||||
"captures": captures,
|
||||
"usage": usage,
|
||||
"snapshotIds": snapshot_ids,
|
||||
"outputs": {
|
||||
row["run_id"]: reader.get_execution_outputs(row["run_id"])
|
||||
for row in rows
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
|
||||
Vendored
+5
-2
@@ -24,12 +24,15 @@ test('execution script discovers only the new matrix and preserves all 78 stock
|
||||
const stock = list('tools/vendor/comfyui.config.mjs');
|
||||
assert.match(stock, /Total: 78 tests in 7 files/);
|
||||
assert.doesNotMatch(stock, /execution-history/);
|
||||
assert.doesNotMatch(stock, /execution-outputs/);
|
||||
const executions = list('tools/vendor/playwright.execution.config.mjs');
|
||||
assert.match(executions, /\[classic-start\]/);
|
||||
assert.match(executions, /\[nodes2-start\]/);
|
||||
assert.match(executions, /Total: 26 tests in 1 file/);
|
||||
assert.match(executions, /Total: 42 tests in 2 files/);
|
||||
for (const project of ['classic-start', 'nodes2-start']) {
|
||||
assert.equal(executions.split('\n').filter(line => line.includes(`[${project}]`)).length, 13);
|
||||
const cases = executions.split('\n').filter(line => line.includes(`[${project}]`));
|
||||
assert.equal(cases.filter(line => line.includes('execution-history.spec.mjs')).length, 13);
|
||||
assert.equal(cases.filter(line => line.includes('execution-outputs.spec.mjs')).length, 8);
|
||||
}
|
||||
assert.doesNotMatch(executions, /prompt-manager.spec/);
|
||||
});
|
||||
|
||||
Vendored
+15
-1
@@ -97,6 +97,7 @@ test('read-only ledger helper distinguishes absent schema from missing, corrupt
|
||||
from database.operations import PromptDatabase
|
||||
from utils.execution_history_events import RunStart, make_finish
|
||||
from utils.execution_contexts import ContextBundle, PromptSnapshot, RecipeContext, TextEvidence
|
||||
from utils.execution_outputs import OutputBundle, OutputReference
|
||||
from types import SimpleNamespace
|
||||
|
||||
db = PromptDatabase(sys.argv[1])
|
||||
@@ -123,6 +124,10 @@ try:
|
||||
db.write_execution_contexts(sys.argv[5], epoch, ContextBundle(
|
||||
(RecipeContext('4', snapshot(' positive '), snapshot('negative'), 'inferred'),),
|
||||
'complete', 'captured'))
|
||||
db.write_execution_outputs_for_finish(make_finish(sys.argv[3], 1, epoch,
|
||||
SimpleNamespace(messages=[('execution_success', {'prompt_id': sys.argv[2]})]), 200),
|
||||
OutputBundle((OutputReference('9', 0, 'temp', 'nested', 'preview.png', 'posix', 'a' * 64),),
|
||||
'complete', 'reported'))
|
||||
print(epoch)
|
||||
finally:
|
||||
db.close_all()
|
||||
@@ -138,6 +143,12 @@ finally:
|
||||
node_id: null, node_type: null, exception_type: null,
|
||||
}]);
|
||||
assert.deepEqual(Object.keys(result.captures), [runId]);
|
||||
assert.deepEqual(result.outputs, { [runId]: {
|
||||
state: 'complete', reason: 'reported', format_version: 1, source: 'host_final_history',
|
||||
references: [{ output_node_id: '9', image_ordinal: 0, folder_type: 'temp',
|
||||
subfolder: 'nested', filename: 'preview.png', path_style: 'posix', root_key: 'a'.repeat(64),
|
||||
file_identity: 'unverified', newly_generated: 'unknown', context_membership: 'unknown', transient: true }],
|
||||
} });
|
||||
const capture = result.captures[runId];
|
||||
assert.equal(capture.contexts.length, 1);
|
||||
const contextId = capture.contexts[0].context_id;
|
||||
@@ -170,8 +181,11 @@ finally:
|
||||
});
|
||||
assert.deepEqual(unfiltered.usage[uncapturedId], []);
|
||||
assert.deepEqual(unfiltered.snapshotIds[uncapturedId], []);
|
||||
assert.deepEqual(unfiltered.outputs[uncapturedId], {
|
||||
state: 'not_captured', reason: null, format_version: null, source: null, references: [],
|
||||
});
|
||||
assert.deepEqual(await readExecutionRuns(database, job, '60000000-0000-4000-8000-000000000001'), {
|
||||
schemaAvailable: true, rows: [], captures: {}, usage: {}, snapshotIds: {},
|
||||
schemaAvailable: true, rows: [], captures: {}, usage: {}, snapshotIds: {}, outputs: {},
|
||||
});
|
||||
assert.deepEqual(await readFile(database), before);
|
||||
await assert.rejects(readExecutionRuns(database, "' OR 1=1 --"));
|
||||
|
||||
Vendored
+1
-1
@@ -14,7 +14,7 @@ const servers = [{
|
||||
|
||||
export default defineConfig({
|
||||
testDir: '../../tests/comfyui',
|
||||
testIgnore: 'execution-history.spec.mjs',
|
||||
testIgnore: ['execution-history.spec.mjs', 'execution-outputs.spec.mjs'],
|
||||
outputDir: './test-results/comfyui',
|
||||
timeout: 60000,
|
||||
workers: 1,
|
||||
|
||||
+1
-1
@@ -6,7 +6,7 @@ if (host.href !== 'http://127.0.0.1:8196/') throw new Error('Execution tests req
|
||||
|
||||
export default defineConfig({
|
||||
testDir: '../../tests/comfyui',
|
||||
testMatch: 'execution-history.spec.mjs',
|
||||
testMatch: ['execution-history.spec.mjs', 'execution-outputs.spec.mjs'],
|
||||
outputDir: './test-results/executions/cases',
|
||||
timeout: 60000,
|
||||
workers: 1,
|
||||
|
||||
Reference in New Issue
Block a user