nodes/ComfyAssets-ComfyUI_PromptManager
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
bc14f29db7a676710d8efe062fe4be040f2d1e86
ComfyAssets-ComfyUI_PromptM…/tests/browser/pair-node.spec.mjs
T
Vito f95f66d76a ci(browser): run page workflows with parallel workers and CI shards (#274) 
* test(browser): one test server per Playwright worker

The page workflows started one global webServer and ran with one worker.
Each worker now owns its own tests/browser/server.py on an OS-chosen port,
with its own temp root and database, through a shared `test` in
tests/browser/fixtures.mjs. baseURL, and so page, request and
browser.newContext, point at that worker's server. A server that dies
fails the running test with its stderr tail and the next test gets a
fresh one. The worker count comes from PM_TEST_WORKERS, else 1 under CI,
else a quarter of the CPUs up to 8.

Chromium workflows in the pinned image: 510 tests in 606 s with one
worker on main, 511 (one new isolation spec) in 119 s with eight.

* test(browser): test servers bind an OS-chosen port themselves

tests/browser/server.py accepts --port 0: it binds 127.0.0.1:0 before
building the app, holds that socket for its whole life and prints
PM_TEST_SERVER_PORT=<n> to stdout. startBrowserServer defaults to port 0
and reads the announced port, so no port is reserved and released before
the server binds and two launchers can no longer race for one. The
"address already in use" retry and free-port.mjs are gone; --port N still
works and is checked against the announcement.

* ci(coverage): shard and merge modes for the frontend coverage run

frontend-coverage.mjs now plans its steps from the command line.
No arguments still runs everything, then reports and checks.
--shard I/N runs one slice of the chromium+touch workflows into
coverage/js/raw without wiping it or reporting; shard 1 also runs the
node units and the viewer suite. --merge-only reports and checks over raw
coverage the shards wrote, with --update-baseline as before, and refuses
an empty or missing raw dir. npm scripts: test:coverage:shard and
coverage:merge.

* ci(viewerjs): run browser suites with workers and shards

Frontend coverage runs as three shards (PM_TEST_WORKERS=3 each) that upload
their raw V8 files as coverage-raw-N; a merge job downloads them, fails unless
all three arrived with raw files and no clashing names, then runs
coverage:merge. Firefox and WebKit page workflows run as two shards each with
three workers. The required viewerjs context keeps its name and now needs the
shards, the merge and the browser shards, failing unless every one succeeded.

* ci(viewerjs): cache npm downloads and size timeouts from the first sharded run

The first sharded run lost a webkit shard to its 12-minute limit after all
267 tests passed, and npm ci stalled for 5 minutes in six of eight jobs. Each
npm ci now restores npm's download cache keyed on the vendor lockfile, and
browser job timeouts are the measured times plus at least 50%, never under 20
minutes. The workflow comment records the measurements.

* test(browser): wait for rendering frames, not timers or luck, under load

Two Library workflows failed in CI's webkit shard 2/2 (run 39714840809)
under parallel load:

- "library infinite loading ..." counted cards right after the last
  page hid "Load more". That render queues the window update for the
  next frame, so a slow frame left the 120 cards of the all-rows
  layout in place (received 120, expected < 80). It now waits for the
  next frame, which runs after every callback the page queued.
- "row measurement preserves anchor ..." settled with a 60 ms timer. A
  scroll event is delivered in a rendering frame, so on a starved
  renderer the timer ended before the scroll's frame and no card was in
  view (null anchor). settleLibrary now waits for two consecutive
  native frames with no Library frame queued, checked inside the frame
  callback.

Before/after: a scratch copy that delays every frame by 120 ms (as a
starved runner) failed the old helpers 19 of 40 runs, at the CI lines,
and passed the new ones 40 of 40 on chromium and webkit. Throttled to
1 CPU with 4 workers, the original spec failed 1 of 42 for each test.

The Tags viewer workflow closed the viewer as soon as its image was
visible, before ViewerJS rendered the title, so CI's coverage never ran
the page's title callback (tags-page.js functions 26/46 = 56.52%, one
function above the floor of 56; locally 27/46). The test now asserts the
title names the shown file, which always runs the callback.

* test(browser): stable under parallel workers

Document how to run the browser suites in parallel and sharded, with the
measured times, and what keeps specs stable beside each other: wait for
a rendering frame, a response or a visible state, never for time.

With b665aa6 the full workflows suite passed on chromium, touch, firefox
and webkit with 8 workers, each test 3 times (6132 passed, 0 failed,
24 minutes), and once at CI size (docker --cpus=4, 3 workers: 2044
passed, 25.5 minutes).

The npm download cache removed the npm ci stall: 0 to 2 s in every job
of run 39722172529, against about 300 s in most jobs before. The
workflow's timing comment now records that run; the whole workflow took
8m54s against about 33 minutes before sharding. Shard imbalance (shard 1
carries the concept-comparison specs) is documented, not tuned.

* test(browser): server stderr on failures, no orphan servers, waits on state

Final review fixes for the per-worker test servers (#274):

- A failed test gets the stderr its server wrote during that test
  attached as `server-stderr`, so an aiohttp traceback behind a 500
  reaches the CI report. The launcher keeps the newest 256 KB with a
  stream position per test (stderrMark/stderrSince), and the fixture
  attaches it only when the test did not end as expected.
- The port announcement counts only as a complete line; a chunk ending
  mid-number was read as a shorter port.
- server.py --exit-on-stdin-eof shuts down gracefully (database closed,
  temp root removed) when stdin closes. The launcher holds that pipe,
  so a SIGKILLed worker leaves no server behind, on every OS.
- settled() gives up after 600 frames with the observation stats
  instead of hanging to the test timeout.
- The Classic text-sync test waits for the page's 50 ms timer to
  install serializeValue instead of sleeping 60 ms. With that timer run
  100 ms late (a scratch copy), the old wait failed 20/20 and the poll
  passed 20/20; the spec passed 240/240 at 8 workers x10 and 240/240
  at 1 CPU with 4 workers.
- The workflow test checks each npm cache key names its job's pinned
  image tag.

Every new unit test failed before its change (missing export, the old
regex timing out on the split announcement, an unknown server flag, a
hung settle, a mismatched cache key).

* docs(testing): server lifetime, stderr attachments and the final measured CI times

TESTING.md describes the stdin-EOF shutdown and the server-stderr
attachment, states that the npm cache key names the job's image tag
(now tested), quotes run 39725176320 at cbc8a61 (8m38s) and explains the
20-minute browser job limit. The workflow timing comment quotes the same
run.

* fix(browser): server shutdown and its tests on Windows, Python < 3.13 and CI

Three CI failures on ed809f7:

- Windows: server.py exited 1 on stdin EOF because create_app works with
  the temp root as its cwd, and Windows cannot remove a process's cwd.
  main() now restores the original cwd before the root is removed.
- Python 3.10 to 3.12: the lifetime test's cleanup called communicate()
  after closing stdin, which flushes the closed pipe (ValueError); 3.13
  tolerates it. It now waits and closes the pipes itself, and the exit
  assertion shows the server's stderr. Under 3.12: errors=2 before, OK
  after.
- Tooling (CI container): the SIGKILLed-worker tests treated an exited
  server as alive, because the container's PID 1 does not reap orphans
  and kill(pid, 0) succeeds on a zombie. Reproduced locally with docker
  without --init (2 failed); a zombie now counts as exited (2 passed).
2026-10-21 23:10:57 -07:00

187 lines
17 KiB
JavaScript
Raw Blame History

import {readFileSync} from 'node:fs';
import {test,expect} from './fixtures.mjs';
import {startCoverage,saveCoverage} from '../../tools/vendor/browser-coverage.mjs';
const pair={id:4,positive_text:'Library positive',negative_text:'Library negative',category:'Portrait',tags:['studio']};
test.beforeEach(async({page,browserName})=>{
await startCoverage(page,browserName);
page.pairErrors=[];page.on('pageerror',error=>page.pairErrors.push(error.message));
await page.route('**/scripts/app.js',route=>route.fulfill({contentType:'text/javascript',body:'export const app=window.fixtureApp;'}));
await page.route('**/scripts/api.js',route=>route.fulfill({contentType:'text/javascript',body:'export const api={fileURL:path=>path,queuePrompt:async(...args)=>args};'}));
await page.route('**/extensions/promptmanager/prompt_manager.js',route=>route.fulfill({contentType:'text/javascript',body:readFileSync(new URL('../../web/comfy/prompt_manager.js',import.meta.url),'utf8')}));
await page.route('**/prompt_manager/pairs?**',route=>route.fulfill({json:{success:true,pairs:[pair],total:1}}));
await page.goto('/prompt_manager/test');
await page.evaluate(async()=>{
document.body.replaceChildren();
const settings=new EventTarget();settings.getSettingValue=()=> 'dark';
window.fixtureApp={registerExtension(extension){this.extension=extension;},ui:{settings},graph:{_nodes:[],setDirtyCanvas(){}},canvas:{},graphToPrompt:async()=>({})};
await import('/extensions/promptmanager/prompt_manager.js');
class Node {
constructor(){this.id=1;this.type='PromptManagerPair';this.widgets=['text','negative_text','category','tags','search_text'].map(name=>({name,type:'text',value:'',options:{},inputEl:document.createElement('textarea')}));this.inputs=['clip','prepend_text','append_text'].map(name=>({name,link:null}));this.properties={};this.size=[1100,700];}
addWidget(type,name,value,callback,options){const widget={type,name,value,callback,options};this.widgets.push(widget);return widget;}
addDOMWidget(name,type,element){const host=document.createElement('div');host.style.cssText='height:700px;width:1100px;max-width:100%';host.className='fixture-dom-widget';host.append(element);document.body.append(host);return {element};}
setSize(size){this.size=size;}setDirtyCanvas(){}
serialize(){return {widgets_values:this.widgets.map(widget=>widget.value),properties:{...this.properties}};}
configure(data){this.widgets.forEach((widget,index)=>widget.value=data.widgets_values[index]);this.properties={...data.properties};}
}
await fixtureApp.extension.beforeRegisterNodeDef(Node,{name:'PromptManagerPair'},fixtureApp);
window.node=new Node();node.onNodeCreated?.();fixtureApp.graph._nodes=[node];
});
});
test.afterEach(async({page,browserName})=>{await saveCoverage(page,browserName);expect(page.pairErrors).toEqual([]);});
async function selectPair(page){await page.getByRole('button',{name:'Recent',exact:true}).click();await page.locator('.pm-editor-result').click();}
async function draft(page,positive='Draft positive',negative='Draft negative'){await page.getByRole('textbox',{name:'Positive prompt',exact:true}).fill(positive);await page.getByRole('textbox',{name:'Negative prompt',exact:true}).fill(negative);}
test('paired editor previews both texts before explicitly replacing both',async({page})=>{
await expect(page.locator('.pm-pair')).toBeVisible();
await draft(page);
await selectPair(page);
await expect(page.locator('.pm-editor-selection')).toContainText('Library positive');
await expect(page.locator('.pm-editor-selection')).toContainText('Library negative');
await expect(page.getByRole('textbox',{name:'Positive prompt',exact:true})).toHaveValue('Draft positive');
await expect(page.getByRole('textbox',{name:'Negative prompt',exact:true})).toHaveValue('Draft negative');
await page.getByRole('switch',{name:'Load metadata'}).uncheck();
await page.getByRole('button',{name:'Use pair',exact:true}).click();
await expect(page.getByRole('textbox',{name:'Positive prompt',exact:true})).toHaveValue(pair.positive_text);
await expect(page.getByRole('textbox',{name:'Negative prompt',exact:true})).toHaveValue(pair.negative_text);
expect(await page.evaluate(()=>Object.fromEntries(node.widgets.map(w=>[w.name,w.value])))).toMatchObject({text:pair.positive_text,negative_text:pair.negative_text,category:'',tags:''});
await page.evaluate(()=>{const saved=node.serialize();node.configure(saved);});
await expect(page.getByRole('textbox',{name:'Negative prompt',exact:true})).toHaveValue(pair.negative_text);
});
test('paired editor sizes both editors equally and composes positive text only',async({page},testInfo)=>{
await draft(page);
const positive=page.getByRole('textbox',{name:'Positive prompt',exact:true});const negative=page.getByRole('textbox',{name:'Negative prompt',exact:true});
const [p,n]=await Promise.all([positive.boundingBox(),negative.boundingBox()]);expect(Math.abs(p.height-n.height)).toBeLessThan(1);expect(Math.abs(p.width-n.width)).toBeLessThan(1);
await page.getByText('Composition · prepend / append',{exact:true}).click();
await page.getByRole('textbox',{name:'Prepend',exact:true}).fill('trigger');
await page.getByRole('textbox',{name:'Append',exact:true}).fill('detail');
await page.getByText('Composed text preview',{exact:true}).click();
await expect(page.locator('[data-composed-preview]')).toHaveText('trigger Draft positive detail');
await expect(negative).toHaveValue('Draft negative');
await page.screenshot({path:testInfo.outputPath('pair-editor.png')});
});
for(const field of ['text','negative_text']) test(`paired loading is atomic when ${field} is connected`,async({page})=>{
await draft(page);
await selectPair(page);
await page.evaluate(field=>{node.inputs.push({name:field,link:8});node.onConnectionsChange();},field);
await expect(page.getByRole('button',{name:'Use pair',exact:true})).toBeDisabled();
await page.getByRole('button',{name:'Use pair',exact:true}).evaluate(button=>button.dispatchEvent(new Event('click',{bubbles:true})));
expect(await page.evaluate(()=>Object.fromEntries(node.widgets.map(w=>[w.name,w.value])))).toMatchObject({text:'Draft positive',negative_text:'Draft negative'});
await page.evaluate(()=>{node.inputs=node.inputs.filter(input=>input.link!==8);node.onConnectionsChange();});
await expect(page.getByRole('button',{name:'Use pair',exact:true})).toBeEnabled();
});
test('save pair accepts empty negatives, recovers after errors and never clears the draft',async({page})=>{
let attempts=0;const bodies=[];
await page.route('**/prompt_manager/pairs',route=>{bodies.push(route.request().postDataJSON());return ++attempts===1?route.fulfill({status:503,json:{success:false,error:'Temporary save error'}}):route.fulfill({json:{success:true,pair:{...pair,negative_text:''}}});});
await draft(page,'Save this','');
await page.getByRole('button',{name:'Save pair',exact:true}).click();
await expect(page.getByRole('alert')).toContainText('Temporary save error');
await expect(page.getByRole('textbox',{name:'Positive prompt',exact:true})).toHaveValue('Save this');
await page.getByRole('button',{name:'Save pair',exact:true}).click();
await expect(page.getByRole('status').filter({hasText:'Pair saved'})).toBeVisible();
expect(bodies).toEqual([{positive_text:'Save this',negative_text:'',category:'',tags:[]},{positive_text:'Save this',negative_text:'',category:'',tags:[]}]);
});
test('pair filters use repeated tags and removal cleans up',async({page})=>{
const urls=[];await page.route('**/prompt_manager/pairs?**',route=>{urls.push(new URL(route.request().url()));return route.fulfill({json:{success:true,pairs:[pair]}});});
await page.getByRole('searchbox',{name:'Search saved prompts'}).fill('portrait tag:studio');
await page.getByLabel('Filter category',{exact:true}).fill('Portrait');
await page.getByLabel('Filter tags',{exact:true}).fill('film, studio, ');
await page.getByRole('button',{name:'Search',exact:true}).click();await expect(page.locator('.pm-editor-result')).toBeVisible();
expect(urls[0].searchParams.get('category')).toBe('Portrait');expect(urls[0].searchParams.getAll('tag')).toEqual(['studio','film']);
await page.evaluate(()=>node.onRemoved());await expect(page.locator('.ca-completions')).toHaveCount(0);
});
test('validating and saving prevents duplicate submissions while preserving pair metadata and drafts',async({page})=>{
let release;let requests=0;
await page.route('**/prompt_manager/pairs',async route=>{requests++;await new Promise(resolve=>release=resolve);await route.fulfill({json:{success:true,pair:{...pair,category:'Original category',tags:['original-tag']}}});});
await page.getByRole('button',{name:'Save pair',exact:true}).click();
await expect(page.getByRole('alert')).toContainText('positive');expect(requests).toBe(0);
await draft(page);
await page.getByRole('button',{name:'Save pair',exact:true}).click();
await expect(page.getByRole('button',{name:'Save pair',exact:true})).toBeDisabled();
await page.getByRole('button',{name:'Save pair',exact:true}).evaluate(button=>button.dispatchEvent(new Event('click',{bubbles:true})));
expect(requests).toBe(1);release();
await expect(page.getByRole('status').filter({hasText:'Pair saved'})).toBeVisible();
expect(await page.evaluate(()=>Object.fromEntries(node.widgets.map(w=>[w.name,w.value])))).toMatchObject({text:'Draft positive',negative_text:'Draft negative',category:'',tags:''});
});
test('a superseded library response never replaces the latest pair selection',async({page})=>{
let release;let started=false;
await page.route('**/prompt_manager/pairs?**',async route=>{
if(new URL(route.request().url()).searchParams.get('text')==='old'){started=true;await new Promise(resolve=>release=resolve);await route.fulfill({json:{success:true,pairs:[{...pair,positive_text:'Stale positive'}]}}).catch(()=>{});}
else await route.fulfill({json:{success:true,pairs:[pair]}});
});
const search=page.getByRole('searchbox',{name:'Search saved prompts'});
await search.fill('old');await page.getByRole('button',{name:'Search',exact:true}).click();await expect.poll(()=>started).toBe(true);
await search.fill('new');await page.getByRole('button',{name:'Search',exact:true}).click();
await expect(page.locator('.pm-editor-result')).toContainText('Library positive');release();
await expect(page.locator('.pm-editor-result')).not.toContainText('Stale positive');
});
test('queue hook synchronizes both editors, preserves links and forwards branch execution arguments',async({page})=>{
await draft(page);
const args=await page.evaluate(async()=>{
await fixtureApp.extension.setup();
const {api}=await import('/scripts/api.js');
const output={1:{inputs:{text:'stale',negative_text:'stale'}}};
const first=await api.queuePrompt(7,{output,workflow:{name:'workflow'}},['branch'],{preview:'latent'});
node.inputs.push({name:'negative_text',link:17});
const linked={1:{inputs:{text:'old',negative_text:['17',0]}}};
const second=await api.queuePrompt(8,{output:linked,workflow:{}},['another']);
return {first,second};
});
expect(args.first).toEqual([7,{output:{1:{inputs:{text:'Draft positive',negative_text:'Draft negative'}}},workflow:{name:'workflow'}},['branch'],{preview:'latent'}]);
expect(args.second[1].output[1].inputs).toEqual({text:'Draft positive',negative_text:['17',0]});
});
test('paired editors stack at narrow widths while keeping equal editing areas',async({page})=>{
await page.locator('.fixture-dom-widget').evaluate(host=>host.style.width='560px');
const [p,n]=await Promise.all(['Positive prompt','Negative prompt'].map(name=>page.getByRole('textbox',{name,exact:true}).boundingBox()));expect(n.y).toBeGreaterThanOrEqual(p.y+p.height);expect(Math.abs(p.width-n.width)).toBeLessThan(1);expect(Math.abs(p.height-n.height)).toBeLessThan(1);
});
test('removal aborts an in-flight pair save and restores host-owned textarea references',async({page})=>{
await draft(page);
const state=await page.evaluate(async()=>{
const previous=window.fetch;let signal;
window.fetch=(url,options)=>{if(String(url).endsWith('/pairs')){signal=options.signal;return new Promise((resolve,reject)=>signal.addEventListener('abort',()=>reject(new DOMException('Aborted','AbortError')),{once:true}));}return previous(url,options);};
const visible=node.widgets.find(w=>w.name==='negative_text').inputEl;
[...document.querySelectorAll('button')].find(button=>button.textContent==='Save pair').click();
node.onRemoved();
return {aborted:signal.aborted,restored:node.widgets.find(w=>w.name==='negative_text').inputEl!==visible};
});
expect(state).toEqual({aborted:true,restored:true});
});
test('actual pair saves coexist with resetting the disposable browser fixture',async({page,request})=>{
await draft(page,'Actual browser pair','Actual negative');await page.getByRole('button',{name:'Save pair',exact:true}).click();await expect(page.getByRole('status').filter({hasText:'Pair saved'})).toBeVisible();
const response=await request.get('/prompt_manager/pairs?text=Actual%20browser%20pair');expect(response.ok()).toBe(true);expect((await response.json()).pairs).toEqual(expect.arrayContaining([expect.objectContaining({positive_text:'Actual browser pair',negative_text:'Actual negative'})]));
const reset=await request.post('/__test__/reset');expect(reset.ok()).toBe(true);
const after=await request.get('/prompt_manager/pairs?text=Actual%20browser%20pair');expect((await after.json()).pairs).toHaveLength(0);
});
for(const field of ['category','tags'])test(`manual pair save waits for execution when ${field} comes from an upstream link`,async({page})=>{
let requests=0;await page.route('**/prompt_manager/pairs',route=>{requests++;return route.fulfill({json:{success:true,pair}});});await draft(page);
await page.evaluate(field=>{node.inputs.push({name:field,link:19});node.onConnectionsChange();},field);
const save=page.getByRole('button',{name:'Save pair',exact:true});await expect(save).toBeDisabled();await expect(page.getByRole('status').filter({hasText:'Run the node to save connected values.'})).toBeVisible();
await save.evaluate(button=>button.dispatchEvent(new Event('click',{bubbles:true})));expect(requests).toBe(0);
await page.evaluate(()=>{node.inputs=node.inputs.filter(input=>input.link!==19);node.onConnectionsChange();});await expect(save).toBeEnabled();await save.click();await expect.poll(()=>requests).toBe(1);
});
for(const field of ['positive','negative','category','tags'])test(`save completion identifies an earlier draft after editing ${field}`,async({page})=>{
let release;await page.route('**/prompt_manager/pairs',async route=>{await new Promise(resolve=>release=resolve);await route.fulfill({json:{success:true,pair}});});await draft(page);
await page.getByText('Metadata · category / tags',{exact:true}).click();await page.getByRole('button',{name:'Save pair',exact:true}).click();await expect.poll(()=>Boolean(release)).toBe(true);
const input=field==='positive'?page.getByRole('textbox',{name:'Positive prompt',exact:true}):field==='negative'?page.getByRole('textbox',{name:'Negative prompt',exact:true}):page.getByRole('combobox',{name:field==='category'?'Category':'Tags',exact:true});
await input.fill(`Changed ${field}`);release();await expect(page.getByRole('status').filter({hasText:'Earlier draft saved. Current changes are not saved yet.'})).toBeVisible();await expect(input).toHaveValue(`Changed ${field}`);
});
test('pair library rows distinguish negative variants and explicitly show empty negatives as text',async({page})=>{
const longNegative='Variant one '+ 'x'.repeat(200);
await page.route('**/prompt_manager/pairs?**',route=>route.fulfill({json:{success:true,pairs:[{...pair,negative_text:longNegative},{...pair,id:5,negative_text:'Variant two'},{...pair,id:6,negative_text:''},{...pair,id:7,negative_text:'<img src=x onerror=alert(1)>'}]}}));
await page.getByRole('button',{name:'Recent',exact:true}).click();const rows=page.locator('.pm-editor-result');await expect(rows).toHaveCount(4);await expect(rows.nth(0)).toContainText('Negative: Variant one');await expect(rows.nth(1)).toContainText('Negative: Variant two');await expect(rows.nth(2)).toContainText('Negative: (empty)');await expect(rows.nth(3)).toContainText('Negative: <img src=x onerror=alert(1)>');await expect(rows.locator('img')).toHaveCount(0);await expect(rows.nth(0)).not.toContainText('x'.repeat(180));await expect(rows.nth(0).locator('span').nth(1)).toHaveAttribute('title',longNegative);
});
Reference in New Issue View Git Blame Copy Permalink