diff --git a/ProjectStructure.md b/ProjectStructure.md index 9395766..464b16c 100644 --- a/ProjectStructure.md +++ b/ProjectStructure.md @@ -76,6 +76,8 @@ ComfyUI-Ultimate-Sampler-Grid/ │ ├── logic_standalone.js # Standalone-page mode: own websocket, run-API GENERATE NEW, openStandalone() (loaded last) │ ├── logic_state.js # State Store (Redux-style: filters, favorites) (104 lines) │ ├── logic_pipeline.js # Data Processing (Filter -> Sort -> Layout) (510 lines) +│ ├── logic_ratings.js # Pairwise review pure logic: axis normalisers, Bradley-Terry, pairing, scoreboard (no DOM, node-testable) +│ ├── logic_compare.js # Pairwise review DOM glue: compare overlay, vote queue, scoreboard modal, rating getters │ ├── logic_virtual.js # Virtual DOM / Infinite Scroller Engine (925 lines) │ ├── logic_ui.js # UI Renderer (DOM creation, Modal handling) (1780 lines) │ ├── logic_upscale_modal.js # Dashboard upscale modal (inline config, presets, async upscale) (380 lines) @@ -204,6 +206,8 @@ All routes registered via `PromptServer.instance.routes`. | POST | `/config_tester/delete_session` | 142 | Delete session folder `{session_name}` | | GET | `/config_tester/list_sessions` | 174 | List all sessions with metadata (name, count, thumbnail, mtime) | | POST | `/config_tester/save_changes` | 234 | Delta save — only changed items `{session_name, changed_items}` | +| POST | `/config_tester/save_comparisons` | - | Pairwise-review votes `{session_name, append, remove}`; only touches manifest `comparisons` | +| POST | `/config_tester/clear_comparisons` | - | Delete all votes of a session `{session_name}` | | POST | `/config_tester/save_manifest` | 308 | Full merge save `{session_name, manifest}` | | POST | `/config_tester/get_session_html` | 390 | Generate dashboard HTML on-demand `{session_name, node_id}` | | GET | `/config_tester/dashboard` | - | Dashboard SPA as a top-level page, optional `?session=` (standalone mode, `USCG_STANDALONE`, own websocket via `resources/logic_standalone.js`) | @@ -279,7 +283,7 @@ All routes registered via `PromptServer.instance.routes`. * **`html_generator.py`** (57 lines) * `get_html_template(title, manifest_data, node_id)` — reads all `/resources/` files, inlines into `template.html` - * **JS Load Order:** `logic_state.js` → `logic_utils.js` → `logic_ui.js` → `logic_virtual.js` → `logic_pipeline.js` → `logic_events.js` → `logic_init.js` → `logic_standalone.js` + * **JS Load Order:** `logic_state.js` → `logic_utils.js` → `logic_ratings.js` → `logic_ui.js` → `logic_upscale_modal.js` → `logic_virtual.js` → `logic_pipeline.js` → `logic_compare.js` → `logic_events.js` → `logic_init.js` → `logic_standalone.js` (all files share ONE inlined script: top-level names must be unique across them) * Replaces placeholders: `__CSS_CONTENT__`, `__JS_CONTENT__`, `__TITLE__`, `__NODE_ID__` * Zero external HTTP requests — everything inlined @@ -448,6 +452,12 @@ The Dashboard is a standalone single-page app rendered inside an iframe. All fil ### **Key Data Shapes** +**Manifest `comparisons`** (top level, sibling of `items`/`meta`; written ONLY by `manifest_utils.update_manifest_comparisons`, disk always wins in every other writer via `preserve_disk_comparisons`): +```json +"comparisons": [{"a": 180000000000001, "b": 180000000000002, "winner": "a", "ts": 1762000000000, "mode": "matched"}] +``` +`mode` is `matched|seed|one_axis|random`; `blind: false` is present only for votes cast unblinded. Dedupe key is `ts:a:b`. + **Config Array State** (`node.state.config_arrays[n]` in JS, parsed from `lora_config` widget in Python): ```json { @@ -871,4 +881,6 @@ __init__.py | `logic_utils.js` | `loadSession()`, `exportFavorites()`, `scanDirectory()`, `triggerGen()`, `toggleFullscreen()`, `deleteNonFavorites()` | | `logic_virtual.js` | `renderDOM()`, `updateVisibleItems()`, `calculateVisibleRange()`, `autoFitZoom()`, `goToImage()` | | `logic_pipeline.js` | `updateDataPipeline()`, `executePipeline()`, `processNewData()`, `incrementalFilter()` | +| `logic_ratings.js` | `computeBradleyTerry()`, `buildPairSource()`, `nextPair()`, `axisDiff()`, `computeScoreboard()` (pure; tests in `tests/js/ratings.test.js`, run `node --test tests/js/*.test.js`) | +| `logic_compare.js` | `openCompareMode()`, `cmpVoteSide()`, `cmpUndo()`, `openScoreboard()`, `getItemRating()`, `compareRatings()`, `onComparisonsReplaced()` | | `logic_events.js` | Message handler for `progress_update`, `update_data`, `distribution_status` | diff --git a/README.md b/README.md index a7f0642..7225398 100644 --- a/README.md +++ b/README.md @@ -1434,6 +1434,19 @@ ComfyUI/output/benchmarks/model-data/ --- +## Pairwise Review (blind A/B compare + scoreboard) + +The star is an absolute signal that says little about which *setting* wins. Compare mode records fast two-up votes and rolls them up per setting. + +* **Start:** the two-rectangles button in the dashboard header, the `C` key, or the cog panel section **PAIRWISE REVIEW**. Votes go through the current filter and never include rejected items. +* **Pairing modes:** *Matched* (same prompt and seed, different settings), *Same settings, different seed* (noise-floor check), *One axis at a time* (exactly one of model, LoRA, sampler, scheduler, CFG, steps, denoise, resolution, prompt, upscale differs; optional axis focus) and *Random* (same media type). Pairs prefer items with few votes and never mix images with videos. When every unvoted pair is used up the queue repeats voted pairs and shows a badge. +* **Blind by default:** only "A" and "B" are shown, placement is randomised per pair. After a vote a short reveal (off / 1 s / 2 s) outlines the winner and shows only the settings that differ. Any key or tap during the reveal advances without voting. Tap or click an image to vote for it. Panes stack on portrait phones. +* **Ratings:** votes feed a Bradley-Terry model (ghost-player prior, 1500 = average). Sort the grid by **Rating**, or enable the **Rating** field in Label Mode to see `★rating · votes` on voted cards. +* **Scoreboard:** per axis, win % (ties count half) with 95% Wilson intervals and the mean rating of the items with that value. Only votes where exactly one axis differs are attributed (a hidden difference such as the VAE blocks attribution; the footer lists the most common extra differences). Rows with fewer than 3 pairs read "not enough data". Seed is a noise-floor block: if seeds alone "win" as much as settings do, the other numbers are noise. Copy JSON / Copy CSV export the board. +* **Storage:** votes live in the session `manifest.json` as a top-level `comparisons` list `[{a, b, winner, ts, mode, blind?}]` (`a`/`b` are item ids, `a` is the left/top item, `winner` is `a`, `b` or `tie`). Old manifests without the field work unchanged. Votes for items that were deleted stay on disk and are ignored. Every manifest writer (generation saves, the remote VAE worker, the legacy `save_manifest` route) keeps the on-disk `comparisons` untouched. A directory re-scan (`scan_directory`) regenerates item ids, so votes of a scanned session are not carried over. +* **Routes:** `POST /config_tester/save_comparisons` `{session_name, append: [...], remove: ["ts:a:b", ...]}` and `POST /config_tester/clear_comparisons` `{session_name}`. They only touch `comparisons`. +* **Caveat:** win % across mixed opponents is not a controlled effect size; pair choice is biased by the filter and the low-vote preference. Use it to rank candidates, then confirm with a matched one-axis run. + ## Keyboard Shortcuts ### Dashboard @@ -1445,7 +1458,20 @@ ComfyUI/output/benchmarks/model-data/ | `+/-` | Zoom in/out | | `0` | Reset zoom to 1:1 | | `F` | Auto-fit first row to viewport width | -| `Escape` | Close active modal/popup (priority: Revision > Filters > Settings > Fullscreen) | +| `Escape` | Close active modal/popup (priority: Revision > Scoreboard > Analytics > Settings > Filters) | +| `C` | Open Compare mode (blind pairwise review, see below) | + +### Dashboard: Compare mode (blind pairwise review) +| Shortcut | Action | +|----------|--------| +| `A` / `Left` | A wins | +| `D` / `Right` | B wins | +| `S` / `Down` | Tie | +| `W` / `Up` | Skip (not saved, pair is excluded for this compare session) | +| `Z` / `Backspace` / `Ctrl+Z` | Undo the last vote (multi-level, this compare session) | +| `I` | Unblind: show what differs before voting (the vote is stored with `blind: false`) | +| `B` | Open the scoreboard | +| `Escape` | Close the scoreboard, then compare mode | ### Config Builder | Shortcut | Action | diff --git a/__init__.py b/__init__.py index 65d8f54..8f2f824 100644 --- a/__init__.py +++ b/__init__.py @@ -416,6 +416,81 @@ async def save_changes(request): traceback.print_exc() return web.Response(status=500, text=str(e)) +# ============================================================================= +# API: PAIRWISE REVIEW VOTES (manifest "comparisons") +# ============================================================================= + +def _comparisons_manifest_path(session_name): + """Resolve a sanitized session name to its manifest path; returns (path, error_response).""" + if session_name: + session_name = re.sub(r'[^\w\-]', '', str(session_name)) + if not session_name: + return None, web.Response(status=400, text="Missing session_name") + base_dir = os.path.join(folder_paths.get_output_directory(), "benchmarks", session_name) + if not _is_path_within(base_dir, _get_benchmarks_base()): + return None, web.Response(status=403, text="Forbidden: path outside benchmarks directory") + manifest_path = os.path.join(base_dir, "manifest.json") + if not os.path.exists(manifest_path): + return None, web.Response(status=404, text=f"Session '{session_name}' not found") + return manifest_path, None + + +@server.PromptServer.instance.routes.post("/config_tester/save_comparisons") +async def save_comparisons(request): + """ + Append / remove blind-compare votes in manifest["comparisons"]. + Body: {session_name, append: [{a,b,winner,ts,mode,blind?}], remove: ["ts:a:b", ...]} + Only the comparisons list is touched; items/meta are written back as read. + """ + try: + from .manifest_utils import update_manifest_comparisons + data = await request.json() + append = data.get("append") or [] + remove = data.get("remove") or [] + if not isinstance(append, list) or not isinstance(remove, list): + return web.Response(status=400, text="append and remove must be lists") + if (not append and not remove) or len(append) > 5000 or len(remove) > 5000: + return web.Response(status=400, text="Nothing to save or too many votes") + manifest_path, err = _comparisons_manifest_path(data.get("session_name")) + if err: + return err + try: + result = await asyncio.to_thread( + update_manifest_comparisons, manifest_path, append, + [str(k) for k in remove]) + except FileNotFoundError: + return web.Response(status=404, text="Session manifest not found") + except ValueError: # includes json.JSONDecodeError (manifest mid-write): client retries + return web.Response(status=409, text="Manifest unreadable, retry") + print(f"[ConfigTester] ⚖️ Votes: +{result['added']} -{result['removed']} = {result['total']} in {data.get('session_name')}") + return web.json_response(result) + except Exception as e: + print(f"[ConfigTester] Error saving comparisons: {e}") + return web.Response(status=500, text=str(e)) + + +@server.PromptServer.instance.routes.post("/config_tester/clear_comparisons") +async def clear_comparisons(request): + """Delete every vote of a session (body: {session_name}).""" + try: + from .manifest_utils import update_manifest_comparisons + data = await request.json() + manifest_path, err = _comparisons_manifest_path(data.get("session_name")) + if err: + return err + try: + result = await asyncio.to_thread(update_manifest_comparisons, manifest_path, (), (), True) + except FileNotFoundError: + return web.Response(status=404, text="Session manifest not found") + except ValueError: + return web.Response(status=409, text="Manifest unreadable, retry") + print(f"[ConfigTester] ⚖️ Votes cleared ({result['removed']}) in {data.get('session_name')}") + return web.json_response(result) + except Exception as e: + print(f"[ConfigTester] Error clearing comparisons: {e}") + return web.Response(status=500, text=str(e)) + + # ============================================================================= # API: SAVE MANIFEST (Legacy - Full Save) # ============================================================================= @@ -485,6 +560,10 @@ async def save_manifest(request): if "meta" in server_manifest: manifest_data["meta"] = server_manifest["meta"] + # Preserve pairwise-review votes (disk wins; the dashboard copy can be stale) + if "comparisons" in server_manifest: + manifest_data["comparisons"] = server_manifest["comparisons"] + # Save merged manifest os.makedirs(base_dir, exist_ok=True) with open(manifest_path, "w") as f: diff --git a/html_generator.py b/html_generator.py index bae1d4f..2cbc147 100644 --- a/html_generator.py +++ b/html_generator.py @@ -21,10 +21,12 @@ def get_html_template(title, manifest_data, node_id, standalone=False): js_files = [ "logic_state.js", # Global variables "logic_utils.js", # API & Helper functions + "logic_ratings.js", # Pairwise review: pure logic (normalisers, Bradley-Terry, pairing, scoreboard) "logic_ui.js", # DOM Creation, Modals, JSON Bars "logic_upscale_modal.js", # Upscale Modal (preset management, async upscale) "logic_virtual.js", # The Virtual Scroll Engine & Viewport "logic_pipeline.js", # Sorting & Filtering Logic + "logic_compare.js", # Pairwise review: compare overlay, votes, scoreboard modal "logic_events.js", # Event Listeners (Message, Keyboard) "logic_init.js", # Initialization "logic_standalone.js" # Standalone-page mode (own websocket, run API); must stay LAST diff --git a/image_generation.py b/image_generation.py index f7ea9f5..d8e8a50 100644 --- a/image_generation.py +++ b/image_generation.py @@ -1084,6 +1084,10 @@ def flush_batch_with_vae(pending_batch, vae, img_dir, existing_data, session_nam try: with open(manifest_path, "r") as f: disk_manifest = json.load(f) + + # Pairwise-review votes live on disk (disk wins), never drop them on this save + from .manifest_utils import preserve_disk_comparisons + preserve_disk_comparisons(disk_manifest, existing_data) # Create a lookup map for items currently in memory memory_items_map = { diff --git a/manifest_utils.py b/manifest_utils.py index afe8a99..4121c67 100644 --- a/manifest_utils.py +++ b/manifest_utils.py @@ -6,12 +6,136 @@ Handles saving and merging manifest.json with user modifications import json import os import threading +import time # Thread-safe lock for manifest file writes. # Required when multiple distributed workers submit results concurrently. _manifest_lock = threading.Lock() +# --------------------------------------------------------------------------- +# Pairwise review votes (manifest top-level "comparisons" list) +# --------------------------------------------------------------------------- +_WINNERS = ("a", "b", "tie") + + +def comparison_key(c): + """Dedupe key of a vote: "ts:a:b" (string).""" + return f"{c.get('ts')}:{c.get('a')}:{c.get('b')}" + + +def validate_comparison(c): + """Return a cleaned vote dict, or None when the entry is not a valid vote.""" + if not isinstance(c, dict): + return None + a, b = c.get("a"), c.get("b") + if type(a) is not int or type(b) is not int or a == b: + return None + if c.get("winner") not in _WINNERS: + return None + ts = c.get("ts") + if type(ts) not in (int, float) or ts != ts or ts < 0: + return None + mode = c.get("mode") + mode = mode[:32] if isinstance(mode, str) and mode else "matched" + out = {"a": a, "b": b, "winner": c["winner"], "ts": ts, "mode": mode} + if isinstance(c.get("blind"), bool): + out["blind"] = c["blind"] + return out + + +def merge_comparison_lists(existing, append, remove_keys): + """ + Merge votes: keep existing order, drop entries whose key is in remove_keys, + append valid new entries whose key is not present yet. + Invalid existing entries are kept untouched. Returns (merged, added, removed). + """ + remove = set(remove_keys or ()) + merged = [] + seen = set() + removed = 0 + for c in (existing or []): + if isinstance(c, dict) and validate_comparison(c) is not None: + k = comparison_key(c) + if k in remove: + removed += 1 + continue + seen.add(k) + merged.append(c) + added = 0 + for c in (append or []): + v = validate_comparison(c) + if v is None: + continue + k = comparison_key(v) + if k in seen or k in remove: + continue + seen.add(k) + merged.append(v) + added += 1 + return merged, added, removed + + +def preserve_disk_comparisons(disk_manifest, target_manifest): + """ + Copy the on-disk votes onto target_manifest (disk wins). Only the compare + routes create or remove votes, so a stale in-memory copy must never + resurrect undone/cleared votes or drop new ones. + """ + try: + if (isinstance(disk_manifest, dict) and "comparisons" in disk_manifest + and isinstance(target_manifest, dict)): + target_manifest["comparisons"] = disk_manifest["comparisons"] + except Exception: + pass + + +def _atomic_write_json(path, data): + tmp = path + ".tmp" + with open(tmp, "w") as f: + json.dump(data, f, indent=4) + last = None + for _ in range(5): + try: + os.replace(tmp, path) + return + except PermissionError as e: # Windows: a reader has the file open + last = e + time.sleep(0.04) + # Fall back to a plain in-place write (same discipline as save_manifest) + with open(path, "w") as f: + json.dump(data, f, indent=4) + try: + os.remove(tmp) + except OSError: + pass + + +def update_manifest_comparisons(manifest_path, append=(), remove_keys=(), clear=False): + """ + Locked read-modify-write of ONLY the "comparisons" list. + Raises FileNotFoundError when the manifest is missing and ValueError/JSON + errors when it is unreadable (nothing is written in that case). + """ + with _manifest_lock: + with open(manifest_path, "r") as f: + data = json.load(f) + if not isinstance(data, dict): + raise ValueError("manifest is not a JSON object") + if clear: + old = data.get("comparisons") + removed = len(old) if isinstance(old, list) else 0 + data["comparisons"], added = [], 0 + else: + existing = data.get("comparisons") + if not isinstance(existing, list): + existing = [] + data["comparisons"], added, removed = merge_comparison_lists(existing, append, remove_keys) + _atomic_write_json(manifest_path, data) + return {"comparisons": data["comparisons"], "added": added, + "removed": removed, "total": len(data["comparisons"])} + + def merge_manifest_user_changes(manifest_path, existing_data): """ Reload manifest and merge user changes (favorites, rejected, notes) to preserve them. @@ -24,6 +148,9 @@ def merge_manifest_user_changes(manifest_path, existing_data): try: with open(manifest_path, "r") as f: current_manifest = json.load(f) + + # Votes are owned by the compare routes: the disk copy always wins + preserve_disk_comparisons(current_manifest, existing_data) # Create lookup dict of current items by ID from DISK current_items_dict = { diff --git a/remote_vae.py b/remote_vae.py index 3cf7204..21ff6c0 100644 --- a/remote_vae.py +++ b/remote_vae.py @@ -231,6 +231,10 @@ class RemoteVAEDecodeWorker: try: with open(self.manifest_path, "r") as f: disk_manifest = json.load(f) + + # Pairwise-review votes live on disk (disk wins), never drop them on this save + from .manifest_utils import preserve_disk_comparisons + preserve_disk_comparisons(disk_manifest, self.existing_data) # Create a lookup map for items currently in memory # This allows us to update self.existing_data in-place diff --git a/resources/logic_compare.js b/resources/logic_compare.js new file mode 100644 index 0000000..dadc2ca --- /dev/null +++ b/resources/logic_compare.js @@ -0,0 +1,824 @@ +/** + * PAIRWISE REVIEW - DOM + STATE GLUE + * Compare overlay, vote/undo/reveal flow, persistence queue, scoreboard modal, + * settings bindings, and the rating getters used by the grid (sort + label badge). + * Pure logic lives in logic_ratings.js (no DOM there). Everything here is prefixed + * cmp / _cmp because all logic_*.js files share one inlined