From 6aabea97de95a0737f9e024fc44b7c5f40880bdb Mon Sep 17 00:00:00 2001 From: Jason Hoku Date: Wed, 4 Nov 2026 00:06:55 -0800 Subject: [PATCH] docs: pairwise review Co-Authored-By: Claude Fable 5.1 --- ProjectStructure.md | 14 +++++++++++++- README.md | 28 +++++++++++++++++++++++++++- 2 files changed, 40 insertions(+), 2 deletions(-) diff --git a/ProjectStructure.md b/ProjectStructure.md index cabe861..41e13c6 100644 --- a/ProjectStructure.md +++ b/ProjectStructure.md @@ -74,6 +74,8 @@ ComfyUI-Ultimate-Sampler-Grid/ │ ├── logic_init.js # Bootstrapper (Load JSON, Init State) (104 lines) │ ├── logic_state.js # State Store (Redux-style: filters, favorites) (104 lines) │ ├── logic_pipeline.js # Data Processing (Filter -> Sort -> Layout) (510 lines) +│ ├── logic_ratings.js # Pairwise review pure logic: axis normalisers, Bradley-Terry, pairing, scoreboard (no DOM, node-testable) +│ ├── logic_compare.js # Pairwise review DOM glue: compare overlay, vote queue, scoreboard modal, rating getters │ ├── logic_virtual.js # Virtual DOM / Infinite Scroller Engine (925 lines) │ ├── logic_ui.js # UI Renderer (DOM creation, Modal handling) (1780 lines) │ ├── logic_upscale_modal.js # Dashboard upscale modal (inline config, presets, async upscale) (380 lines) @@ -201,6 +203,8 @@ All routes registered via `PromptServer.instance.routes`. | POST | `/config_tester/delete_session` | 142 | Delete session folder `{session_name}` | | GET | `/config_tester/list_sessions` | 174 | List all sessions with metadata (name, count, thumbnail, mtime) | | POST | `/config_tester/save_changes` | 234 | Delta save — only changed items `{session_name, changed_items}` | +| POST | `/config_tester/save_comparisons` | - | Pairwise-review votes `{session_name, append, remove}`; only touches manifest `comparisons` | +| POST | `/config_tester/clear_comparisons` | - | Delete all votes of a session `{session_name}` | | POST | `/config_tester/save_manifest` | 308 | Full merge save `{session_name, manifest}` | | POST | `/config_tester/get_session_html` | 390 | Generate dashboard HTML on-demand `{session_name, node_id}` | | POST | `/config_tester/export_favorites` | 435 | Export favorited images with options (metadata, workflow, organize) | @@ -273,7 +277,7 @@ All routes registered via `PromptServer.instance.routes`. * **`html_generator.py`** (57 lines) * `get_html_template(title, manifest_data, node_id)` — reads all `/resources/` files, inlines into `template.html` - * **JS Load Order:** `logic_state.js` → `logic_utils.js` → `logic_ui.js` → `logic_virtual.js` → `logic_pipeline.js` → `logic_events.js` → `logic_init.js` + * **JS Load Order:** `logic_state.js` → `logic_utils.js` → `logic_ratings.js` → `logic_ui.js` → `logic_upscale_modal.js` → `logic_virtual.js` → `logic_pipeline.js` → `logic_compare.js` → `logic_events.js` → `logic_init.js` (all files share ONE inlined script: top-level names must be unique across them) * Replaces placeholders: `__CSS_CONTENT__`, `__JS_CONTENT__`, `__TITLE__`, `__NODE_ID__` * Zero external HTTP requests — everything inlined @@ -441,6 +445,12 @@ The Dashboard is a standalone single-page app rendered inside an iframe. All fil ### **Key Data Shapes** +**Manifest `comparisons`** (top level, sibling of `items`/`meta`; written ONLY by `manifest_utils.update_manifest_comparisons`, disk always wins in every other writer via `preserve_disk_comparisons`): +```json +"comparisons": [{"a": 180000000000001, "b": 180000000000002, "winner": "a", "ts": 1762000000000, "mode": "matched"}] +``` +`mode` is `matched|seed|one_axis|random`; `blind: false` is present only for votes cast unblinded. Dedupe key is `ts:a:b`. + **Config Array State** (`node.state.config_arrays[n]` in JS, parsed from `lora_config` widget in Python): ```json { @@ -864,4 +874,6 @@ __init__.py | `logic_utils.js` | `loadSession()`, `exportFavorites()`, `scanDirectory()`, `triggerGen()`, `toggleFullscreen()`, `deleteNonFavorites()` | | `logic_virtual.js` | `renderDOM()`, `updateVisibleItems()`, `calculateVisibleRange()`, `autoFitZoom()`, `goToImage()` | | `logic_pipeline.js` | `updateDataPipeline()`, `executePipeline()`, `processNewData()`, `incrementalFilter()` | +| `logic_ratings.js` | `computeBradleyTerry()`, `buildPairSource()`, `nextPair()`, `axisDiff()`, `computeScoreboard()` (pure; tests in `tests/js/ratings.test.js`, run `node --test tests/js/*.test.js`) | +| `logic_compare.js` | `openCompareMode()`, `cmpVoteSide()`, `cmpUndo()`, `openScoreboard()`, `getItemRating()`, `compareRatings()`, `onComparisonsReplaced()` | | `logic_events.js` | Message handler for `progress_update`, `update_data`, `distribution_status` | diff --git a/README.md b/README.md index eb9193c..4698f3f 100644 --- a/README.md +++ b/README.md @@ -1371,6 +1371,19 @@ ComfyUI/output/benchmarks/model-data/ --- +## Pairwise Review (blind A/B compare + scoreboard) + +The star is an absolute signal that says little about which *setting* wins. Compare mode records fast two-up votes and rolls them up per setting. + +* **Start:** the two-rectangles button in the dashboard header, the `C` key, or the cog panel section **PAIRWISE REVIEW**. Votes go through the current filter and never include rejected items. +* **Pairing modes:** *Matched* (same prompt and seed, different settings), *Same settings, different seed* (noise-floor check), *One axis at a time* (exactly one of model, LoRA, sampler, scheduler, CFG, steps, denoise, resolution, prompt, upscale differs; optional axis focus) and *Random* (same media type). Pairs prefer items with few votes and never mix images with videos. When every unvoted pair is used up the queue repeats voted pairs and shows a badge. +* **Blind by default:** only "A" and "B" are shown, placement is randomised per pair. After a vote a short reveal (off / 1 s / 2 s) outlines the winner and shows only the settings that differ. Any key or tap during the reveal advances without voting. Tap or click an image to vote for it. Panes stack on portrait phones. +* **Ratings:** votes feed a Bradley-Terry model (ghost-player prior, 1500 = average). Sort the grid by **Rating**, or enable the **Rating** field in Label Mode to see `★rating · votes` on voted cards. +* **Scoreboard:** per axis, win % (ties count half) with 95% Wilson intervals and the mean rating of the items with that value. Only votes where exactly one axis differs are attributed (a hidden difference such as the VAE blocks attribution; the footer lists the most common extra differences). Rows with fewer than 3 pairs read "not enough data". Seed is a noise-floor block: if seeds alone "win" as much as settings do, the other numbers are noise. Copy JSON / Copy CSV export the board. +* **Storage:** votes live in the session `manifest.json` as a top-level `comparisons` list `[{a, b, winner, ts, mode, blind?}]` (`a`/`b` are item ids, `a` is the left/top item, `winner` is `a`, `b` or `tie`). Old manifests without the field work unchanged. Votes for items that were deleted stay on disk and are ignored. Every manifest writer (generation saves, the remote VAE worker, the legacy `save_manifest` route) keeps the on-disk `comparisons` untouched. A directory re-scan (`scan_directory`) regenerates item ids, so votes of a scanned session are not carried over. +* **Routes:** `POST /config_tester/save_comparisons` `{session_name, append: [...], remove: ["ts:a:b", ...]}` and `POST /config_tester/clear_comparisons` `{session_name}`. They only touch `comparisons`. +* **Caveat:** win % across mixed opponents is not a controlled effect size; pair choice is biased by the filter and the low-vote preference. Use it to rank candidates, then confirm with a matched one-axis run. + ## Keyboard Shortcuts ### Dashboard @@ -1382,7 +1395,20 @@ ComfyUI/output/benchmarks/model-data/ | `+/-` | Zoom in/out | | `0` | Reset zoom to 1:1 | | `F` | Auto-fit first row to viewport width | -| `Escape` | Close active modal/popup (priority: Revision > Filters > Settings > Fullscreen) | +| `Escape` | Close active modal/popup (priority: Revision > Scoreboard > Analytics > Settings > Filters) | +| `C` | Open Compare mode (blind pairwise review, see below) | + +### Dashboard: Compare mode (blind pairwise review) +| Shortcut | Action | +|----------|--------| +| `A` / `Left` | A wins | +| `D` / `Right` | B wins | +| `S` / `Down` | Tie | +| `W` / `Up` | Skip (not saved, pair is excluded for this compare session) | +| `Z` / `Backspace` / `Ctrl+Z` | Undo the last vote (multi-level, this compare session) | +| `I` | Unblind: show what differs before voting (the vote is stored with `blind: false`) | +| `B` | Open the scoreboard | +| `Escape` | Close the scoreboard, then compare mode | ### Config Builder | Shortcut | Action |