diff --git a/AI-CONTEXT.md b/AI-CONTEXT.md index 79d456d..66d3a38 100644 --- a/AI-CONTEXT.md +++ b/AI-CONTEXT.md @@ -40,7 +40,7 @@ Read `ProjectStructure.md` in this directory for the full reference (file struct | `manifest_utils.py` | 316 | Load/save/merge manifest. `save_manifest()` reloads from disk first to preserve concurrent user edits (favorites, votes, metrics) | | `metadata_packer.py` | 561 | PNG metadata embedding for CivitAI. `pack_metadata_into_image()` accepts `workflow_data` kwarg | | `run_api.py` | 196 | Pure helpers behind `POST /config_tester/run`, `GET /config_tester/run_status` and `GET /config_tester/manifest` (`load_manifest_for_fetch`) | -| `image_metrics.py`, `metrics_runner.py` | 668, 167 | Optional per-image metrics at save time; background job that backfills them for old sessions | +| `image_metrics.py`, `metrics_runner.py` | 668, 167 | Optional per-image metrics at save time (incl. `sobel`); background job that backfills them (and `fill_missing` keys) for old sessions | | `workflow_sweep.py`, `workflow_sweep_runner.py`, `workflow_sweep_collect.py` | 1106, 1027, 281 | Workflow Sweep: pure expansion core, in-process queue feeder (never blocks the executor), `USCGSweepCollect` / `USCGSweepCollectVideo` output nodes | | `web/conf_builder/conf-builder-main.js` | 779 | Node registration. Default state and its migration blocks define ALL fields including upscaling/cooldown | | `web/conf_builder/conf-builder-utilities.js` | 866 | Model-list cache (`LS_CACHE_VERSION`), `getIterationCount()`, LoRA string helpers. No config converter: the preview calls `/configbuilder/preview` | @@ -50,6 +50,7 @@ Read `ProjectStructure.md` in this directory for the full reference (file struct | `resources/logic_ui.js` | 2907 | Dashboard DOM: cards, modals, `buildComfyNodesWorkflow()`, analytics | | `resources/logic_virtual.js` | 1092 | Virtual scroller: renders only visible cards from thousands | | `resources/logic_ratings.js`, `resources/logic_compare.js` | 538, 824 | Pairwise review: pure Bradley-Terry / pairing / scoreboard logic, and the compare overlay that uses it | +| `resources/logic_analytics.js`, `resources/logic_analytics_panel.js` | new | Settings Stats panel: `computeAnalytics()` (pure) groups items per setting axis (fav/reject rate, time, metrics, rating); the panel is docked, live (hooked into `updateJSONs`), has Copy Markdown | | `resources/logic_wfparams.js`, `resources/logic_dragdrop.js` | 81, small | Pure helpers (node-testable): Workflow Sweep `wf_params` keys/matching/labels; Revise-image drag payload + `[output]` path for Load Image | | `resources/logic_standalone.js` | 170 | Standalone-page mode (own websocket, run-API GENERATE NEW); loaded last | @@ -166,4 +167,4 @@ Images saved as WebP at `output/benchmarks/{session}/images/img_{id}.webp` **Modify dashboard UI:** 1. Edit files in `resources/` (HTML in `template.html`, JS in `logic_*.js`, CSS in `report.css`) 2. Remember: changes require HTML regeneration to take effect -3. JS load order is `html_generator.js_files`: logic_state.js → logic_utils.js → logic_ratings.js → logic_wfparams.js → logic_dragdrop.js → logic_ui.js → logic_upscale_modal.js → logic_virtual.js → logic_pipeline.js → logic_compare.js → logic_events.js → logic_init.js → logic_standalone.js (last). All of them share ONE inlined script scope, so top-level names must be unique +3. JS load order is `html_generator.js_files`: logic_state.js → logic_utils.js → logic_ratings.js → logic_wfparams.js → logic_analytics.js → logic_dragdrop.js → logic_ui.js → logic_upscale_modal.js → logic_virtual.js → logic_pipeline.js → logic_compare.js → logic_analytics_panel.js → logic_events.js → logic_init.js → logic_standalone.js (last). All of them share ONE inlined script scope, so top-level names must be unique diff --git a/ProjectStructure.md b/ProjectStructure.md index f44eb28..6245b37 100644 --- a/ProjectStructure.md +++ b/ProjectStructure.md @@ -40,8 +40,8 @@ ComfyUI-Ultimate-Sampler-Grid/ │ ├── generation_orchestrator.py # The "Conductor" (Loop Logic, job plan, preview pass, 3892 lines) ├── image_generation.py # KSampler & VAE Wrappers (1321 lines) -├── image_metrics.py # Optional per-image metrics (sharpness/entropy/...), run context, scorer-pack adapters (ADAPTERS registry: node ids/signatures pinned to pack shas, loader defaults + `prefer`, `scale`, `max_batch`, `vram_mb`; `adapter_status()` reports `problem`) (668 lines) -├── metrics_runner.py # Background job: compute metrics for an existing session (CPU, manifest lock) (167 lines) +├── image_metrics.py # Optional per-image metrics (sharpness/entropy/sobel/...), run context, scorer-pack adapters (ADAPTERS registry: node ids/signatures pinned to pack shas, loader defaults + `prefer`, `scale`, `max_batch`, `vram_mb`; `adapter_status()` reports `problem`) (668 lines) +├── metrics_runner.py # Background job: compute metrics for an existing session; `fill_missing` adds only missing keys (CPU, manifest lock) (182 lines) ├── model_loader.py # Checkpoint/LoRA/VAE/GGUF Loading Logic, latent geometry, CLIP type (907 lines) ├── model_cache.py # 3-Tier Model Caching System (935 lines) ├── remote_vae.py # Remote VAE decode worker + facade for the RemoteVAE companion (314 lines) @@ -87,6 +87,7 @@ ComfyUI-Ultimate-Sampler-Grid/ │ ├── logic_state.js # State Store (Redux-style: filters, favorites) (149 lines) │ ├── logic_utils.js # Helpers (Export, Scan, Session loading) (1073 lines) │ ├── logic_ratings.js # Pairwise review pure logic: axis normalisers, Bradley-Terry, pairing, scoreboard (no DOM, node-testable) (538 lines) +│ ├── logic_analytics.js # Settings stats pure logic: computeAnalytics (one pass, per-axis groups), sortAnalyticsRows, analyticsToMarkdown (node-testable) │ ├── logic_wfparams.js # Workflow Sweep wf_params helpers (keys, numeric detection, match, short labels); Quick/Logic Filter + label overlay use them (pure, node-testable) │ ├── logic_dragdrop.js # Revise-modal image drag/send helpers (issue #22): uscgImageRef, asset-info + uri-list + DownloadURL carriers, annotated `[output]` path (no DOM, node-testable) │ ├── logic_ui.js # UI Renderer (DOM creation, Modal handling) (2907 lines) @@ -94,6 +95,7 @@ ComfyUI-Ultimate-Sampler-Grid/ │ ├── logic_virtual.js # Virtual DOM / Infinite Scroller Engine (1092 lines) │ ├── logic_pipeline.js # Data Processing (Filter -> Sort -> Layout) (831 lines) │ ├── logic_compare.js # Pairwise review DOM glue: compare overlay, vote queue, scoreboard modal, rating getters (824 lines) +│ ├── logic_analytics_panel.js # Docked Settings Stats panel (DOM): scope, sortable tables, Copy Markdown, live refresh hooks │ ├── logic_events.js # Interaction Handler (Hotkeys, API calls) (101 lines) │ ├── logic_init.js # Bootstrapper (Load JSON, Init State) (112 lines) │ └── logic_standalone.js # Standalone-page mode: own websocket, run-API GENERATE NEW, openStandalone() (loaded last, 170 lines) @@ -254,7 +256,7 @@ All routes registered via `PromptServer.instance.routes`. One row per route this | POST | `/config_tester/upscale_images` | `__init__.py` | `{session_name, image_ids, upscale_config: {pipelines: [...]}, all_favorited?}` | Start an async dashboard upscale job; returns `{job_id, total_images}` | | GET | `/config_tester/upscale_status` | `__init__.py` | `?job_id=` | Progress of an upscale job | | POST | `/config_tester/cancel_upscale` | `__init__.py` | `{job_id}` | Cancel an upscale job; returns `{cancelled}` | -| POST | `/config_tester/compute_metrics` | `__init__.py` | `{session_name, overwrite?}` | Start the background metrics job for a session; returns `{job_id, total_images}` | +| POST | `/config_tester/compute_metrics` | `__init__.py` | `{session_name, overwrite?, fill_missing?}` | Start the background metrics job for a session (`fill_missing`: only the metric keys an item lacks, disk values win); returns `{job_id, total_images}` | | GET | `/config_tester/metrics_status` | `__init__.py` | `?job_id=` | Metrics job progress | | POST | `/config_tester/cancel_metrics` | `__init__.py` | `{job_id}` | Cancel a metrics job; returns `{cancelled}` | | POST | `/config_tester/sweep/parse` | `__init__.py` | `{api_json}` | Workflow Sweep: describe an API-format workflow (nodes, outputs, `image_sources`, `video_sources`, `default_output`, suggested bindings) | @@ -332,7 +334,7 @@ All routes registered via `PromptServer.instance.routes`. One row per route this * **`html_generator.py`** (57 lines) * `get_html_template(title, manifest_data, node_id, standalone=False, inline_manifest=False)` — reads all `/resources/` files, inlines them into `template.html`. By default the page is a **lazy shell**: `fullManifest` is empty and flagged `__uscg_lazy__`, `manifest_data` is ignored, and the SPA fetches `GET /config_tester/manifest?session=` itself (`inline_manifest=True` restores the old inlined data; used by tests) - * **JS Load Order:** `logic_state.js` → `logic_utils.js` → `logic_ratings.js` → `logic_wfparams.js` → `logic_dragdrop.js` → `logic_ui.js` → `logic_upscale_modal.js` → `logic_virtual.js` → `logic_pipeline.js` → `logic_compare.js` → `logic_events.js` → `logic_init.js` → `logic_standalone.js` (all files share ONE inlined script: top-level names must be unique across them) + * **JS Load Order:** `logic_state.js` → `logic_utils.js` → `logic_ratings.js` → `logic_wfparams.js` → `logic_analytics.js` → `logic_dragdrop.js` → `logic_ui.js` → `logic_upscale_modal.js` → `logic_virtual.js` → `logic_pipeline.js` → `logic_compare.js` → `logic_analytics_panel.js` → `logic_events.js` → `logic_init.js` → `logic_standalone.js` (all files share ONE inlined script: top-level names must be unique across them) * Replaces placeholders: `__CSS_CONTENT__`, `__JS_CONTENT__`, `__TITLE__`, `__NODE_ID__` * Zero external HTTP requests for code (JS/CSS inlined); the manifest comes from `/config_tester/manifest` @@ -899,7 +901,7 @@ __init__.py 35. **Dynamic-VRAM re-prepare**: on ComfyUI 0.4x (comfy_aimdo) `load_models_gpu` re-runs `ModelPatcherDynamic.load()` on every encode; the repeated "prepared for dynamic VRAM loading" log line is the symptom. TE batching (`te_batch_size`) self-checks batch 1 against a single encode and halves on OOM; non-batchable prompts keep the per-prompt path. -36. **`html_generator.js_files` order is load-bearing** (one shared script scope): `logic_ratings.js` (pure) before `logic_ui.js`, `logic_compare.js` after `logic_pipeline.js`, `logic_standalone.js` LAST (top-level-page overrides). CI assembles the page and syntax-checks it. +36. **`html_generator.js_files` order is load-bearing** (one shared script scope): `logic_ratings.js` (pure) before `logic_ui.js`, `logic_analytics.js` after `logic_ratings.js` + `logic_wfparams.js`, `logic_compare.js` after `logic_pipeline.js`, `logic_analytics_panel.js` after `logic_compare.js` (uses `cmpEl`/`cmpCopyText`/`cmpRatings`) and `pipelineFilterCtx`/`pipelineMatches` from `logic_pipeline.js`, `logic_standalone.js` LAST (top-level-page overrides). CI assembles the page and syntax-checks it. 37. **Workflow Sweep video outputs.** `workflow_output` carries optional `media: "video"` and (VHS frames) `frames_from: <VHS node id>`; both are emitted only for video, and `fingerprint()` / `template_hash()` add `m`/`f` to the hashed payload only for video, so image fingerprints stay byte-identical (pinned by golden values in `tests/test_workflow_sweep_video.py`; changing the image payload would orphan every old session's resume). VHS frames are converted IN THE GRAPH by a spliced core `CreateVideo` (`build_job_prompt`), never in the collector, so the executor handles cache/hidden inputs. `USCGSweepCollectVideo` saves with `VideoInput.save_to` (SaveVideo itself needs executor-populated hidden inputs). `manifest_utils.append_items(remove_ids=...)` deletes a removed video's `<stem>.preview.png` along with the mp4. No new top-level Builder field, so the rule-5 checklist does not apply. @@ -941,7 +943,7 @@ __init__.py | `distribution.py` | `is_distribution_available()`, `create_manager()`, `notify_workers_to_start()`, `stop_all_workers()`, `get_master_url()` (facade; the manager and worker live in the companion) | | | `run_api.py` | `build_run_prompt()`, `normalize_configs_payload()`, `resolve_saved_config()`, `summarize_history()`, `RunTracker` | | | `image_metrics.py` | `compute_metrics_numpy()`, `compute_metrics_torch()`, `begin_run()`, `end_run()`, `finalize()`, `adapter_status()` | | -| `metrics_runner.py` | `start_metrics_job()`, `get_metrics_status()`, `cancel_metrics_job()` | | +| `metrics_runner.py` | `start_metrics_job(session, overwrite, keys, fill_missing)`, `get_metrics_status()`, `cancel_metrics_job()` | | | `workflow_sweep.py` | `parse_api_workflow()`, `suggest_bindings()`, `parse_for_ui()` (pure; no torch or server imports) | | | `workflow_sweep_runner.py` | `dry_run()`, `api_start()`, `start_sweep()`, `get_status()`, `pause()`, `resume()`, `cancel()`, `ComfyBridge` | | @@ -959,6 +961,9 @@ __init__.py | `logic_utils.js` | `loadSession()`, `exportFavorites()`, `scanDirectory()`, `triggerGen()`, `toggleFullscreen()`, `deleteNonFavorites()` | | `logic_virtual.js` | `renderDOM()`, `updateVisibleItems()`, `calculateVisibleRange()`, `autoFitZoom()`, `goToImage()` | | `logic_pipeline.js` | `updateDataPipeline()`, `executePipeline()`, `processNewData()`, `incrementalFilter()` | +| `logic_analytics.js` | `computeAnalytics()`, `sortAnalyticsRows()`, `analyticsToMarkdown()` (pure; tests in `tests/js/analytics.test.js`) | +| `logic_analytics_panel.js` | `openAnalyticsPanel()`, `toggleAnalyticsPanel()`, `closeAnalyticsPanel()`, `renderAnalyticsPanel()`, `analyticsPanelOnDataChanged()` (live refresh hook from `updateJSONs`/`cmpAfterRatingsChanged`), `analyticsPanelMarkdown()` | +| `logic_pipeline.js` (filters) | `pipelineFilterCtx()`, `pipelineMatches(d, ctx, ignoreVisibility)`: the filter callback of `executePipeline`, shared with the Stats panel | | `logic_wfparams.js` | `wfParamKeys()`, `wfParamValue/Number()`, `wfParamMatches()`, `wfShortLabel()`, `wfDisplay()` (pure; tests in `tests/js/wfparams.test.js`) | | `logic_ratings.js` | `computeBradleyTerry()`, `buildPairSource()`, `nextPair()`, `axisDiff()`, `computeScoreboard()` (pure; tests in `tests/js/ratings.test.js`, run `node --test tests/js/*.test.js`) | | `logic_dragdrop.js` | `uscgImageRef()`, `uscgImageAssetInfo()`, `uscgAnnotatedPath()`, `uscgFillDragData()`, `USCG_DRAG_UPLOAD` (pure; tests in `tests/js/dragdrop.test.js`). Used by `logic_ui.js` (`dragstart` on `#m-img`, `sendModalToLoadImage()`). Parent bridge: iframe posts `uscg_send_to_load_image` {url, filename, subfolder, imageType, upload, node_id}, `web/dashboard.js` fills the selected/only/new LoadImage and answers `uscg_send_to_load_image_result` {ok, message} | diff --git a/README.md b/README.md index e0e5091..b4928b4 100644 --- a/README.md +++ b/README.md @@ -119,6 +119,7 @@ If you'd rather build the graph yourself: - [JSON Bars](#json-bars-bottom--horizontal-layout) - [Revision Modal](#revision-modal) - [Pairwise Review](#pairwise-review-blind-ab-compare--scoreboard) + - [Stats Panel](#stats-panel-per-setting-analytics) - [Standalone Dashboard](#standalone-dashboard-phone--second-monitor) - [Example Workflows](#-example-workflows) - [Quick Quality Test](#quick-quality-test-40-images) @@ -208,7 +209,7 @@ If you'd rather build the graph yourself: * **Seed Behavior Settings:** Per-config seed control — `fixed` uses the node seed, `randomize` generates a new seed per image, `full_run_seed` overrides the node seed entirely. * **Configurable Label Mode:** Toggle overlay labels on dashboard cards showing model, sampler, scheduler, steps, CFG, and other parameters directly on each image. * **GPU Cooldown:** Automatic pause between generations to prevent thermal throttling. Configurable interval, duration, and optional VRAM clearing. -* **Image Metrics (optional):** Run Settings checkbox that stores sharpness, entropy, colorfulness, contrast, luma and saturation for every saved image in the manifest (see [Image Metrics](#image-metrics-optional)). +* **Image Metrics (optional):** Run Settings checkbox that stores sharpness, entropy, colorfulness, contrast, luma, saturation and Sobel edges for every saved image in the manifest (see [Image Metrics](#image-metrics-optional)). * **Workflow Sweep:** sweep any ComfyUI workflow exported as API JSON. Bind any input as an axis, dry run it, then pause, resume or cancel; every image lands in a normal USCG session (see [Workflow Sweep](#workflow-sweep-sweep-any-comfyui-workflow)). * **Job Order (Minimize reloads):** reorders the whole job list so each model and LoRA stack loads as few times as possible (see [Job Order & Preview Pass](#job-order--preview-pass-run-settings)). * **Preview Pass (generator side):** renders the whole grid cheaply first; reject the losers and only the survivors render at full quality (see [Job Order & Preview Pass](#job-order--preview-pass-run-settings)). @@ -882,12 +883,14 @@ Run Settings has a **Compute Image Metrics** checkbox (default OFF). When it is | `contrast` | Standard deviation of luma (RMS contrast) | 0-0.5 | | `luma` | Mean brightness (Rec.601 luma) | 0-1 | | `saturation` | Mean HSV saturation | 0-1 | +| `sobel` | Mean Sobel gradient magnitude of luma (0-1 scale; kernels /4, so a full black-to-white step reads 1) | 0 = flat; most generations land well under 0.2 (measure your own); max ~1.41 | Notes: +* `sobel` complements `sharpness`: it responds to real edges and ignores pixel-level noise (a 1-pixel checkerboard reads 0 Sobel but a huge sharpness), so a high Sobel with a moderate sharpness means structure, not grain. It is absent for images smaller than 3 px on a side. * Metrics are measured on an "analysis image": the long side is block-averaged down to at most 1024 px (by the smallest integer factor), so a 2x upscale stays comparable with its 1024 base image and the cost is capped. The constant is `ANALYSIS_MAX_SIDE` in `image_metrics.py`; changing it would make old sessions incomparable. * They measure image **signal, not taste**: noise raises sharpness and entropy, and a dark moody image is not a worse one. Use them to rank and filter, then look. * Upscales are measured again (a derived image never inherits its source's numbers). Dashboard-triggered upscales measure only when the source image had metrics. -* Items from remote Distributed workers are measured too: the worker computes the built-in metrics on its decoded image, and the master scores the optional scorer packs for every worker image in one pass after collection (so workers do not need the scorer pack installed). An older companion/worker sends no metrics and the master measures the saved file instead. +* Items from remote Distributed workers are measured too: the worker computes the built-in metrics on its decoded image, and the master scores the optional scorer packs for every worker image in one pass after collection (so workers do not need the scorer pack installed). An older companion/worker sends no metrics and the master measures the saved file instead; workers on an older USCG omit Sobel, and the master measures it from the saved file. * Optional scorer packs plug in through `nodes.NODE_CLASS_MAPPINGS` without any import, and their scores are stored next to the built-ins. The Builder lists each pack with `installed`, `installed, but <problem>` (for example "no model file in models/aesthetic") or `not installed (install ...)` (text only, nothing is downloaded); the checkbox is only clickable when the pack is usable. See **Optional scorer packs** below. **Optional scorer packs** @@ -900,7 +903,7 @@ Notes: USCG passes **one image per call** (the node only scores image 1 of a batch) and rescales the node's integer x100 output. The node reloads CLIP from disk on every call, so expect several seconds per image (measured 7.6 s per image on an RTX 3070 8 GB next to an SDXL 1024x1024 run, about 0.9 GB extra VRAM; the SDXL run itself took 10-16 s per image). USCG asks ComfyUI's memory manager to free VRAM before each call. On a card this small a model may be evicted and reloaded, which is slower but correct. The node hardcodes `cuda`, so CPU/MPS/AMD hosts fail and the adapter switches itself off for the run with one warning (the built-in metrics are unaffected). A per-adapter timing line (`[GridTester] 📏 metric adapter 'aesthetic': N images, mean X ms`) is printed when the run ends. * **ImageReward** has a registry entry checked against the source of `ZaneA/ComfyUI-ImageReward@5e9936d` (prompt comes from the item's positive prompt, score is the FLOAT output, roughly -2 to +2 and relative to the prompt, so only compare scores of the same prompt), but it ships **disabled**, and a wave-3 feasibility check (11/4/26) confirmed why: the only PyPI release the pack pins, `image-reward==1.4`, fails to import under transformers 5 (`find_pruneable_heads_and_indices` and `PreTrainedModel.get_head_mask` were removed), so it needs a hand patch of `site-packages/ImageReward/models/BLIP/med.py` on every transformers-5 host; it also pins `timm==0.6.13`, builds `fairscale`, downloads 1.8 GB on first load and may trip `torch.load(weights_only=True)`. On a transformers-4 host that has the pack working, enabling it is a one-word change in `image_metrics.ADAPTERS` (set `enabled` to True); note the score uses only the first ~35 BERT tokens of the prompt, so compare within one prompt only. * **In the Dashboard:** once a session has metrics, the Sort popup gains one option per metric (descending, items without the metric sort last, in both the primary and secondary lists), the Logic Filter field list gains a **Metrics** group, and a **Metric range** row adds `>= min` / `<= max` filters in one click (placeholders show the observed range). The label panel has a **Metrics** checkbox that prints `Shp:812 Ent:7.41 Col:41.2 ...` on each card. Logic filters skip items that lack the field, so images without metrics (older items, videos) stay visible; Quick Filter is categorical and does not list metrics, use the Logic Filter for ranges. Sessions without metrics show none of this. -* **Older sessions:** the cog panel's **IMAGE METRICS** section has **Compute metrics for this session**, a background job (CPU only, no VRAM, about 50 ms per image) that measures every image without metrics, skips videos, shows progress and can be cancelled by clicking the button again. It rewrites only the `metrics` field under the manifest lock, so favorites and notes you change meanwhile survive. Do not run it on a session that is still generating. +* **Older sessions:** the cog panel's **IMAGE METRICS** section has **Compute metrics for this session**, a background job (CPU only, no VRAM, about 50 ms per image) that measures every image that is missing any metric (for example Sobel on sessions measured before it existed; values already stored are never changed), skips videos, shows progress and can be cancelled by clicking the button again. It rewrites only the `metrics` field under the manifest lock, so favorites and notes you change meanwhile survive. Do not run it on a session that is still generating. * The setting is session-level (like GPU Cooldown), so it is NOT a per-config field: it lives in `_session_settings.metrics`, not in `_EXPECTED_CONFIG_KEYS` or `expand_configs`. ### Workflow Sweep (sweep any ComfyUI workflow) @@ -1012,6 +1015,17 @@ The star is an absolute signal that says little about which *setting* wins. Comp * **Manifest fields:** every upscaled item carries `base_width` / `base_height` (the size of the generated image the upscale started from; compare's Resolution axis uses them, older manifests fall back to an estimate from `upscale_ratio`). Scanned folders get `source_relpath` and **stable ids** (derived from the relative path, in `[2^52, 2^53)`, so they never collide with generated ids), so re-scanning a folder keeps favorites, notes, metrics and votes. * **Caveat:** win % across mixed opponents is not a controlled effect size; pair choice is biased by the filter and the low-vote preference. Use it to rank candidates, then confirm with a matched one-axis run. +### Stats Panel (per-setting analytics) + +The **bar-chart button** in the header bar (or **SETTINGS STATS (PER AXIS)** in the cog panel's Manifest Analytics section) opens a docked panel in the top-right corner (bottom of the screen on phones). It is not a modal: it stays open while you favorite, reject, filter, vote and while new images arrive, and refreshes itself (at most 4 times per second). + +* **One table per setting:** model, LoRA, sampler, scheduler, steps, CFG, denoise, resolution, attention mode, upscale, pass (preview or full), prompt, plus every Workflow Sweep parameter (`wf_params`; parameters with more than 64 distinct values, such as seeds, are skipped). Axes where every item has the same value are folded into a "Same for every item" line. +* **Columns:** n, Fav % and Rej % (share of that value's items), Avg s (mean generation time), one column per metric the session has (mean over the items that have it) and Rating (mean pairwise rating, only after you voted). Click a header to sort; the choice is remembered. +* **Small samples:** rows with fewer than 3 items are dimmed and always sorted below the rest; the best Fav % row (3 or more items) is highlighted. +* **Scope:** *Current filter* applies the Quick/Button/Logic/Search filters but not the Favorites / Non-Favorited / Rejected visibility toggles, so rejects and favorites are still counted when the grid hides them (otherwise Reject % would always read 0). *Whole session* counts everything except superseded previews. +* **Copy Markdown** copies every table for the current scope and sort, ready to paste into an issue or chat. +* Press `Escape` (after the Filters popup) or the X to close; the panel can also be collapsed to its title bar. + ### Standalone Dashboard (phone / second monitor) The dashboard is also served as a normal web page, with the same live updates as the node: @@ -1566,7 +1580,7 @@ Bot pattern: `POST /run`, post `dashboard_url` to the channel, poll `run_status` | GET | `/config_tester/manifest` | A session's manifest as compact JSON `?session=<name>` (gzip, ETag / 304); what the Dashboard fetches | | POST | `/config_tester/save_comparisons` | Append / remove Pairwise Review votes `{session_name, append, remove}` | | POST | `/config_tester/clear_comparisons` | Delete every vote of a session `{session_name}` | -| POST | `/config_tester/compute_metrics` | Start the background metrics job for a session `{session_name, overwrite?}`; returns `{job_id, total_images}` | +| POST | `/config_tester/compute_metrics` | Start the background metrics job for a session `{session_name, overwrite?, fill_missing?}`; returns `{job_id, total_images}` | | GET | `/config_tester/metrics_status` | Progress of a metrics job, `?job_id=` | | POST | `/config_tester/cancel_metrics` | Cancel a metrics job `{job_id}` | | POST | `/config_tester/sweep/parse` | Workflow Sweep: describe an API-format workflow `{api_json}` | @@ -1625,7 +1639,7 @@ ComfyUI/output/benchmarks/model-data/ | `+/-` | Zoom in/out | | `0` | Reset zoom to 1:1 | | `F` | Auto-fit first row to viewport width | -| `Escape` | Close active modal/popup (priority: Revision > Scoreboard > Analytics > Settings > Filters) | +| `Escape` | Close active modal/popup (priority: Revision > Scoreboard > Analytics > Settings > Filters > Stats panel) | | `C` | Open Compare mode (blind pairwise review, see [Pairwise Review](#pairwise-review-blind-ab-compare--scoreboard)) | ### Dashboard: Compare mode (blind pairwise review) @@ -1832,10 +1846,12 @@ After modifying backend Python files, restart ComfyUI. After modifying `web/` JS * 📐 **Latent input size:** a connected latent's image size was always cells x 8; it is now cells x the loaded model's latent ratio (16 for Flux.2/MOSS-Image/Krea 2). ### Update 11/4/26: Standalone dashboard, pairwise review, image metrics, preview pass, workflow sweeps +* 📊 **Stats panel:** header button (and cog entry) opens a live, docked panel with favorites rate, reject rate, mean time, mean metrics and mean rating per sampler, scheduler, model, LoRA, steps, CFG, resolution, pass and every workflow-sweep parameter, with a Copy Markdown button (see [Stats Panel](#stats-panel-per-setting-analytics)). +* 🔲 **Sobel edge metric:** new built-in image metric `sobel` (mean Sobel gradient of luma, 0-1) next to Sharpness; "Compute metrics for this session" now also fills in metrics that older items are missing, without changing stored values. * 🧩 **Regional prompts (`COUPLE`):** `base COUPLE left | right` (or `COUPLE(rows)`, 2-4 regions, per-region `:w` weight) gives each part of the image its own prompt using native masked conditioning. Triggers, model prefix/suffix and HiRes "Append to end" text go into the base; works with HiRes, tiled HiRes and Florence2 crops. The conditioning cache on disk now keeps every entry (AND prompts included), so older cache files are re-encoded once. See [Prompt Combinators & Regional Prompts (COUPLE)](#prompt-combinators--regional-prompts-couple). * 📱 **Standalone Dashboard + Run API:** open the dashboard as a normal web page at `/config_tester/dashboard?session=<name>` (phone, second monitor) with live updates over its own websocket. `POST /config_tester/run` and `GET /config_tester/run_status` start and track headless runs (Discord bot, curl) without any workflow JSON. * ⚖️ **Pairwise Review:** blind A/B compare mode (`C` key) with matched, same-settings, one-axis and random pairing, Bradley-Terry ratings (Sort by Rating) and a per-setting scoreboard with 95% intervals. Votes are stored in the session manifest under `comparisons`. -* 📏 **Image Metrics (optional):** a Run Settings checkbox stores sharpness, entropy, colorfulness, contrast, luma and saturation for every saved image. Sort, filter and label by them in the Dashboard, or compute them for older sessions from the cog panel. +* 📏 **Image Metrics (optional):** a Run Settings checkbox stores sharpness, entropy, colorfulness, contrast, luma, saturation and Sobel edges for every saved image. Sort, filter and label by them in the Dashboard, or compute them for older sessions from the cog panel. * 🧭 **Job Order & Preview Pass:** `Minimize model/LoRA reloads` reorders the whole job list so each model and LoRA stack loads as few times as possible. Preview Pass renders the whole grid cheaply first; reject the losers and only the survivors render at full quality (PREVIEW badge, Pass filter, "Full pass for survivors" button). * 🔀 **Workflow Sweep:** sweep any ComfyUI workflow exported as API JSON. Bind any input as an axis (prompts, LoRAs, resolution, seeds, any widget), dry run it, then pause, resume or cancel; every image lands in a normal USCG session. * ⚡ **Faster prompt encoding:** the text encoder now loads once per pre-encode pass instead of once per prompt, and the new `te_batch_size` setting encodes several prompts per pass for MOSS-Image, Krea 2 and Qwen-Image (off by default, self-checked, OOM-safe). diff --git a/Roadmap.md b/Roadmap.md index fd49f7a..e35d578 100644 --- a/Roadmap.md +++ b/Roadmap.md @@ -150,10 +150,10 @@ Compute objective quality metrics during generation and store in manifest: - **Sobel edge quality** (detail preservation) Add as sortable/filterable fields in Dashboard. Enables data-driven parameter selection instead of purely visual comparison. Prior art: SamplerSchedulerMetricsTester (proves the concept, only 9 stars but validates demand). -DONE (11/4/26): built-in metrics (sharpness, entropy, colorfulness, contrast, luma, saturation), Dashboard sort/filter/labels, backfill job for old sessions. Still open: Sobel edge quality, scorer packs. +DONE (11/4/26): Sobel edge quality (sobel), backfill fills missing keys. Built-in metrics (sharpness, entropy, colorfulness, contrast, luma, saturation, sobel), Dashboard sort/filter/labels, backfill job for old sessions. Still open: scorer packs. #### **26. Analytics Panel in Dashboard** (transforms tool from "look at pictures" to "data-driven decisions") -* **Milestone 1:** Aggregate stats bar — avg generation time by sampler/scheduler, favorites rate by model/LoRA +* **Milestone 1:** Aggregate stats bar — avg generation time by sampler/scheduler, favorites rate by model/LoRA. DONE (11/4/26) Milestone 1: Stats panel (computeAnalytics in resources/logic_analytics.js is the data source for M2). * **Milestone 2:** Canvas-based charts — bar charts, scatter plots (steps vs duration, CFG vs quality metrics) * **Milestone 3:** Export — downloadable reports with charts + top images Pure client-side using canvas API from existing `activeData`. Prior art: W&B dashboards. diff --git a/__init__.py b/__init__.py index e647d92..eab2b30 100644 --- a/__init__.py +++ b/__init__.py @@ -1452,7 +1452,8 @@ async def compute_metrics(request): if not _is_path_within(base_dir, _get_benchmarks_base()): return web.json_response({"error": "Forbidden: path outside benchmarks directory"}, status=403) from .metrics_runner import start_metrics_job, get_metrics_status - job_id, error = start_metrics_job(session_name, overwrite=bool(data.get("overwrite", False))) + job_id, error = start_metrics_job(session_name, overwrite=bool(data.get("overwrite", False)), + fill_missing=bool(data.get("fill_missing", False))) if error: return web.json_response({"error": error}, status=400) return web.json_response({"job_id": job_id, "total_images": get_metrics_status(job_id)["total"]}) diff --git a/generation_orchestrator.py b/generation_orchestrator.py index f167c1e..048a046 100644 --- a/generation_orchestrator.py +++ b/generation_orchestrator.py @@ -3324,7 +3324,9 @@ def _distributed_metrics_pass(existing_data, pre_ids, images_dir, image_path_for have = item.get("metrics") or {} need_builtins = bool(spec.get("builtins")) and not have need_adapters = any(a not in have for a in adapter_names) - if not (need_builtins or need_adapters): + # A worker on an older USCG omits builtins it does not know (e.g. sobel): fill just those from the file + missing_b = [k for k in (spec.get("builtins") or []) if have and k not in have] + if not (need_builtins or need_adapters or missing_b): continue path = image_path_for(item, images_dir) if path is None: @@ -3339,7 +3341,13 @@ def _distributed_metrics_pass(existing_data, pre_ids, images_dir, image_path_for image_metrics.attach(item, pil) out["builtins_from_file"] += 1 else: - image_metrics.attach_adapters(item, pil) + if missing_b: + extra = image_metrics.compute_for_pil(pil, missing_b) + cur = item.setdefault("metrics", {}) + cur.update({k: v for k, v in extra.items() if k not in cur}) + out["builtins_from_file"] += 1 + if need_adapters: + image_metrics.attach_adapters(item, pil) if need_adapters: out["adapters"] += 1 return out diff --git a/html_generator.py b/html_generator.py index a30b418..62a78d5 100644 --- a/html_generator.py +++ b/html_generator.py @@ -34,12 +34,14 @@ def get_html_template(title, manifest_data, node_id, standalone=False, inline_ma "logic_utils.js", # API & Helper functions "logic_ratings.js", # Pairwise review: pure logic (normalisers, Bradley-Terry, pairing, scoreboard) "logic_wfparams.js", # Workflow Sweep: wf_params helpers (pure; must load before logic_ui/logic_pipeline) + "logic_analytics.js", # Settings stats: computeAnalytics/analyticsToMarkdown (pure; after ratings + wfparams) "logic_dragdrop.js", # Revise-modal image drag/send helpers (issue #22; pure, node-testable) "logic_ui.js", # DOM Creation, Modals, JSON Bars "logic_upscale_modal.js", # Upscale Modal (preset management, async upscale) "logic_virtual.js", # The Virtual Scroll Engine & Viewport "logic_pipeline.js", # Sorting & Filtering Logic "logic_compare.js", # Pairwise review: compare overlay, votes, scoreboard modal + "logic_analytics_panel.js", # Settings stats panel (DOM; uses cmpEl/cmpCopyText/cmpRatings from logic_compare) "logic_events.js", # Event Listeners (Message, Keyboard) "logic_init.js", # Initialization "logic_standalone.js" # Standalone-page mode (own websocket, run API); must stay LAST diff --git a/image_metrics.py b/image_metrics.py index 75f514d..a33215f 100644 --- a/image_metrics.py +++ b/image_metrics.py @@ -16,6 +16,7 @@ upscale stays comparable with its base and the cost is capped. contrast std of luma (RMS contrast, 0..0.5) luma mean of Y = 0.299 R + 0.587 G + 0.114 B saturation mean HSV saturation + sobel mean Sobel gradient magnitude of luma (0-1 scale, kernels /4: a full step edge = 1) These measure signal, not taste: noise raises sharpness and entropy. ANALYSIS_MAX_SIDE is a versioned constant; changing it breaks comparisons @@ -28,8 +29,9 @@ import time import numpy as np -BUILTIN_METRICS = ("sharpness", "entropy", "colorfulness", "contrast", "luma", "saturation") +BUILTIN_METRICS = ("sharpness", "entropy", "colorfulness", "contrast", "luma", "saturation", "sobel") ANALYSIS_MAX_SIDE = 1024 +_SOBEL_SCALE = 0.25 _LUMA_W = (0.299, 0.587, 0.114) @@ -183,6 +185,11 @@ def compute_metrics_numpy(arr, keys=BUILTIN_METRICS): mn = a.min(axis=-1) s = np.where(mx >= 1e-6, (mx - mn) / np.maximum(mx, np.float32(1e-6)), np.float32(0.0)) out["saturation"] = float(s.mean(dtype=np.float64)) + if "sobel" in keys and y.shape[0] >= 3 and y.shape[1] >= 3: + two = np.float32(2.0) + gx = (y[:-2, 2:] + two * y[1:-1, 2:] + y[2:, 2:]) - (y[:-2, :-2] + two * y[1:-1, :-2] + y[2:, :-2]) + gy = (y[2:, :-2] + two * y[2:, 1:-1] + y[2:, 2:]) - (y[:-2, :-2] + two * y[:-2, 1:-1] + y[:-2, 2:]) + out["sobel"] = float((np.sqrt(gx * gx + gy * gy) * np.float32(_SOBEL_SCALE)).mean(dtype=np.float64)) return {k: _round4(v) for k, v in out.items() if math.isfinite(v)} @@ -278,6 +285,12 @@ def compute_metrics_torch(images, keys=BUILTIN_METRICS): s = torch.where(mx >= 1e-6, (mx - mn) / mx.clamp_min(1e-6), torch.zeros_like(mx)) cols.append(s.reshape(B, -1).mean(dim=1)) names.append("saturation") + if "sobel" in keys and H >= 3 and W >= 3: + kx = torch.tensor([[-1.0, 0.0, 1.0], [-2.0, 0.0, 2.0], [-1.0, 0.0, 1.0]], device=y.device, dtype=y.dtype) + k = (torch.stack([kx, kx.t()]) * _SOBEL_SCALE)[:, None] # [2,1,3,3] + gr = torch.nn.functional.conv2d(y[:, None], k) # [B,2,H-2,W-2] + cols.append(gr.pow(2).sum(dim=1).sqrt().reshape(B, -1).mean(dim=1)) + names.append("sobel") if not cols: return [{} for _ in range(B)] table = torch.stack([c.float() for c in cols], dim=1).cpu().tolist() # the one sync diff --git a/metrics_runner.py b/metrics_runner.py index 5fad9b5..9c029be 100644 --- a/metrics_runner.py +++ b/metrics_runner.py @@ -48,7 +48,7 @@ def _image_path_for(item, images_dir): return path -def start_metrics_job(session_name, overwrite=False, keys=None): +def start_metrics_job(session_name, overwrite=False, keys=None, fill_missing=False): """Start the job. Returns (job_id, error). session_name must already be sanitized by the caller.""" import folder_paths benchmarks_dir = os.path.realpath(os.path.join(folder_paths.get_output_directory(), "benchmarks")) @@ -67,8 +67,16 @@ def start_metrics_job(session_name, overwrite=False, keys=None): with manifest_utils._manifest_lock: with open(manifest_path, "r", encoding="utf-8") as f: items = json.load(f).get("items", []) - targets = [it["id"] for it in items if "id" in it and (overwrite or not it.get("metrics")) - and _image_path_for(it, images_dir)] + if overwrite: + targets = [it["id"] for it in items if "id" in it and _image_path_for(it, images_dir)] + elif fill_missing: + # Opt-in: items that have SOME metrics but lack a requested key (e.g. Sobel on older sessions) + targets = [it["id"] for it in items if "id" in it + and any(k not in (it.get("metrics") or {}) for k in spec["builtins"]) + and _image_path_for(it, images_dir)] + else: + targets = [it["id"] for it in items if "id" in it and not it.get("metrics") + and _image_path_for(it, images_dir)] if not targets: return None, "Nothing to compute (every image already has metrics, or there are no images)" for jid, j in list(_jobs.items()): @@ -80,7 +88,7 @@ def start_metrics_job(session_name, overwrite=False, keys=None): "failed": 0, "error": None, "cancel": threading.Event()} _jobs[job_id] = job threading.Thread(target=_run_metrics_thread, - args=(job, targets, spec["builtins"], overwrite, manifest_path, images_dir, session_name), + args=(job, targets, spec["builtins"], overwrite, manifest_path, images_dir, session_name, fill_missing), daemon=True).start() return job_id, None @@ -100,8 +108,11 @@ def cancel_metrics_job(job_id): return False -def _flush(manifest_path, results, overwrite): - """Merge `results` ({id: metrics}) into the manifest on disk under the lock; atomic replace.""" +def _flush(manifest_path, results, overwrite, merge=False): + """Merge `results` ({id: metrics}) into the manifest on disk under the lock; atomic replace. + + merge=True (fill_missing): add only keys the item on disk does not have (disk wins; adapter keys survive). + """ if not results: return with manifest_utils._manifest_lock: @@ -109,7 +120,10 @@ def _flush(manifest_path, results, overwrite): data = json.load(f) for it in data.get("items", []): m = results.get(it.get("id")) - if m and (overwrite or not it.get("metrics")): + if m and merge: + cur = it.get("metrics") or {} + it["metrics"] = {**cur, **{k: v for k, v in m.items() if k not in cur}} + elif m and (overwrite or not it.get("metrics")): it["metrics"] = m tmp = manifest_path + ".metrics.tmp" with open(tmp, "w", encoding="utf-8") as f: @@ -128,7 +142,7 @@ def _send_progress(job, session_name): pass -def _run_metrics_thread(job, target_ids, keys, overwrite, manifest_path, images_dir, session_name): +def _run_metrics_thread(job, target_ids, keys, overwrite, manifest_path, images_dir, session_name, fill_missing=False): pending = {} try: with manifest_utils._manifest_lock: @@ -144,7 +158,8 @@ def _run_metrics_thread(job, target_ids, keys, overwrite, manifest_path, images_ raise ValueError("no image file") with Image.open(path) as im: arr = np.asarray(im.convert("RGB")) - m = image_metrics.compute_metrics_numpy(arr, keys) + need = [k for k in keys if k not in (it.get("metrics") or {})] if fill_missing else keys + m = image_metrics.compute_metrics_numpy(arr, need) if need else {} if m: pending[tid] = m else: @@ -154,10 +169,10 @@ def _run_metrics_thread(job, target_ids, keys, overwrite, manifest_path, images_ print(f"[GridTester] ⚠️ metrics skipped for item {tid}: {e}") job["completed"] += 1 if len(pending) >= FLUSH_EVERY: - _flush(manifest_path, pending, overwrite) + _flush(manifest_path, pending, overwrite, fill_missing) pending = {} _send_progress(job, session_name) - _flush(manifest_path, pending, overwrite) + _flush(manifest_path, pending, overwrite, fill_missing) job["status"] = "cancelled" if job["cancel"].is_set() else "complete" print(f"[GridTester] 📏 metrics job {job['status']}: {job['completed']}/{job['total']} images, {job['failed']} skipped") except Exception as e: diff --git a/resources/logic_analytics.js b/resources/logic_analytics.js new file mode 100644 index 0000000..92452a5 --- /dev/null +++ b/resources/logic_analytics.js @@ -0,0 +1,269 @@ +/** + * SETTINGS STATS (ANALYTICS) - PURE LOGIC (no DOM access) + * Groups the items by every setting axis (model, LoRA, sampler, ... plus workflow-sweep params) and + * reduces each group to favorite/reject rate, mean duration, mean metrics and mean pairwise rating. + * The dashboard inlines every logic_*.js into ONE <script>, so top-level names here are prefixed + * (AN_ / an). Node tests load this file through the module.exports guard at the bottom + * (tests/js/analytics.test.js). The result shape is the data source for the later chart milestones. + * + * Needs normalizeAxis/fmtNum (logic_ratings.js) and wfShortLabel (logic_wfparams.js). + */ + +var AN_AXES = ['model', 'lora', 'sampler', 'scheduler', 'steps', 'cfg', 'denoise', 'resolution', 'attention_mode', 'upscale', 'pass', 'prompt']; +var AN_AXIS_LABELS = { + model: 'Model', lora: 'LoRA', sampler: 'Sampler', scheduler: 'Scheduler', steps: 'Steps', cfg: 'CFG', + denoise: 'Denoise', resolution: 'Resolution', attention_mode: 'Attention', upscale: 'Upscale', pass: 'Pass', prompt: 'Prompt' +}; +var AN_MAX_WF_DISTINCT = 64; // wf_params axes with more values than this are skipped (seeds, ...) +var AN_MIN_N = 3; +var AN_DEPS = (typeof module !== 'undefined' && module.exports && typeof require === 'function') + ? { normalizeAxis: require('./logic_ratings.js').normalizeAxis, fmtNum: require('./logic_ratings.js').fmtNum, wfShortLabel: require('./logic_wfparams.js').wfShortLabel } + : null; + +function anDep(name) { return AN_DEPS ? AN_DEPS[name] : (typeof window !== 'undefined' ? window[name] : undefined); } + +/** {key, label} of item d on one axis. */ +function anAxisValue(axis, d, meta) { + if (axis === 'attention_mode') { + var v = d.attention_mode; + var s = Array.isArray(v) ? v.join('+') : String(v || 'default'); + return { key: s, label: s }; + } + if (axis === 'pass') { + var p = d.preview ? 'Preview' : 'Full'; + return { key: p, label: p }; + } + return anDep('normalizeAxis')(axis, d, meta); +} + +function anNewAcc(key, label) { + return { key: key, label: label, n: 0, fav: 0, rej: 0, durSum: 0, durN: 0, mSum: {}, mN: {}, rSum: 0, rN: 0 }; +} + +function anAccAdd(acc, fav, rej, dur, metrics, mkeys, rating) { + acc.n++; + if (fav) acc.fav++; + if (rej) acc.rej++; + if (dur !== null) { acc.durSum += dur; acc.durN++; } + for (var i = 0; i < mkeys.length; i++) { + var k = mkeys[i]; + acc.mSum[k] = (acc.mSum[k] || 0) + metrics[k]; + acc.mN[k] = (acc.mN[k] || 0) + 1; + } + if (rating !== null) { acc.rSum += rating; acc.rN++; } +} + +function anFinishRow(acc, metricKeys, minN) { + var metrics = {}; + for (var i = 0; i < metricKeys.length; i++) { + var k = metricKeys[i]; + metrics[k] = acc.mN[k] ? { mean: acc.mSum[k] / acc.mN[k], n: acc.mN[k] } : null; + } + return { + key: acc.key, label: acc.label, n: acc.n, fav: acc.fav, rej: acc.rej, + favRate: acc.n ? acc.fav / acc.n : 0, rejRate: acc.n ? acc.rej / acc.n : 0, + durMean: acc.durN ? acc.durSum / acc.durN : null, durN: acc.durN, + metrics: metrics, ratingMean: acc.rN ? acc.rSum / acc.rN : null, ratingN: acc.rN, + enough: acc.n >= minN + }; +} + +function anDefaultMetricOrder() { + return (typeof METRIC_BUILTIN_ORDER !== 'undefined' && Array.isArray(METRIC_BUILTIN_ORDER)) ? METRIC_BUILTIN_ORDER : []; +} + +/** + * One pass over `items`. opts = { meta, ratings: Map|null, metricOrder, minN, axes }. + * Returns { total, favorited, rejected, metricKeys, hasRatings, overall, axes:[{id,label,kind,n,distinct,constant,rows}], skippedAxes }. + */ +function computeAnalytics(items, opts) { + opts = opts || {}; + var meta = opts.meta; + var ratings = opts.ratings || null; + var order = opts.metricOrder || anDefaultMetricOrder(); + var minN = opts.minN === undefined ? AN_MIN_N : opts.minN; + var axisIds = opts.axes || AN_AXES; + var fmtNum = anDep('fmtNum'); + var list = items || []; + + var axisMaps = new Map(); // axis id -> Map(key -> acc) + for (var a = 0; a < axisIds.length; a++) axisMaps.set(axisIds[a], new Map()); + var wfMaps = new Map(); // 'wf:<key>' -> Map(key -> acc) + var overall = anNewAcc('(all)', '(all)'); + var metricSeen = new Set(); + var total = 0, favorited = 0, rejected = 0, hasRatings = false; + + for (var i = 0; i < list.length; i++) { + var d = list[i]; + if (!d) continue; + var fav = !!d.favorited, rej = !!d.rejected; + var dn = +d.duration; + var dur = (isFinite(dn) && dn > 0) ? dn : null; + var metrics = d.metrics && typeof d.metrics === 'object' ? d.metrics : null; + var mkeys = []; + if (metrics) { + for (var mk in metrics) { + var mv = metrics[mk]; + if (typeof mv === 'number' && isFinite(mv)) { mkeys.push(mk); metricSeen.add(mk); } + } + } + var rating = null; + if (ratings) { + var rr = ratings.get(d.id); + if (rr && rr.votes > 0 && isFinite(+rr.rating)) { rating = +rr.rating; hasRatings = true; } + } + total++; + if (fav) favorited++; + if (rej) rejected++; + anAccAdd(overall, fav, rej, dur, metrics, mkeys, rating); + + for (var ai = 0; ai < axisIds.length; ai++) { + var id = axisIds[ai]; + var av = anAxisValue(id, d, meta); + var m = axisMaps.get(id); + var acc = m.get(av.key); + if (!acc) { acc = anNewAcc(av.key, av.label); m.set(av.key, acc); } + anAccAdd(acc, fav, rej, dur, metrics, mkeys, rating); + } + + var wf = d.wf_params; + if (wf && typeof wf === 'object') { + for (var wk in wf) { + var wv = wf[wk]; + if (wv === null || wv === undefined) continue; + var wid = 'wf:' + wk; + var wm = wfMaps.get(wid); + if (!wm) { wm = new Map(); wfMaps.set(wid, wm); } + var vk = typeof wv === 'number' ? fmtNum(wv) : String(wv); + var wacc = wm.get(vk); + if (!wacc) { wacc = anNewAcc(vk, vk); wm.set(vk, wacc); } + anAccAdd(wacc, fav, rej, dur, metrics, mkeys, rating); + } + } + } + + var metricKeys = order.filter(function (k) { return metricSeen.has(k); }) + .concat(Array.from(metricSeen).filter(function (k) { return order.indexOf(k) < 0; }).sort()); + + function build(id, label, kind, map) { + var rows = []; + map.forEach(function (acc) { rows.push(anFinishRow(acc, metricKeys, minN)); }); + return { id: id, label: label, kind: kind, n: total, distinct: rows.length, constant: rows.length < 2, rows: sortAnalyticsRows(rows, 'favRate', -1) }; + } + + var axes = []; + for (var b = 0; b < axisIds.length; b++) { + var bm = axisMaps.get(axisIds[b]); + if (bm.size) axes.push(build(axisIds[b], AN_AXIS_LABELS[axisIds[b]] || axisIds[b], 'builtin', bm)); + } + var wfIds = Array.from(wfMaps.keys()).sort(); + var wfKeys = wfIds.map(function (x) { return x.slice(3); }); + var skippedAxes = []; + var shortLabel = anDep('wfShortLabel'); + for (var w = 0; w < wfIds.length; w++) { + var wmap = wfMaps.get(wfIds[w]); + if (wmap.size > AN_MAX_WF_DISTINCT) { skippedAxes.push({ id: wfIds[w], distinct: wmap.size }); continue; } + var shortName = shortLabel ? shortLabel(wfKeys[w], wfKeys) : wfKeys[w]; + axes.push(build(wfIds[w], 'Workflow: ' + shortName, 'wf', wmap)); + } + + return { + total: total, favorited: favorited, rejected: rejected, metricKeys: metricKeys, hasRatings: hasRatings, + overall: anFinishRow(overall, metricKeys, minN), axes: axes, skippedAxes: skippedAxes + }; +} + +function anSortValue(row, col) { + if (col === 'label') return row.label; + if (col === 'n') return row.n; + if (col === 'favRate') return row.favRate; + if (col === 'rejRate') return row.rejRate; + if (col === 'durMean') return row.durMean; + if (col === 'rating') return row.ratingMean; + if (typeof col === 'string' && col.indexOf('metric:') === 0) { + var m = row.metrics && row.metrics[col.slice(7)]; + return m ? m.mean : null; + } + return row.n; +} + +/** New sorted array. Rows with enough samples always come first; nulls last; ties: n desc then label asc. */ +function sortAnalyticsRows(rows, col, dir) { + var sign = dir < 0 ? -1 : 1; + function lab(a, b) { return String(a.label).localeCompare(String(b.label), undefined, { numeric: true }); } + return rows.slice().sort(function (a, b) { + if (a.enough !== b.enough) return a.enough ? -1 : 1; + var va = anSortValue(a, col), vb = anSortValue(b, col); + var na = va === null || va === undefined || (typeof va === 'number' && !isFinite(va)); + var nb = vb === null || vb === undefined || (typeof vb === 'number' && !isFinite(vb)); + if (na !== nb) return na ? 1 : -1; + if (!na) { + var c = typeof va === 'string' ? String(va).localeCompare(String(vb), undefined, { numeric: true }) : (va < vb ? -1 : (va > vb ? 1 : 0)); + if (c) return c * sign; + } + if (b.n !== a.n) return b.n - a.n; + return lab(a, b); + }); +} + +/** Cell text. kind: pct | dur | metric | rating. null -> blank (markdown, default) or the en dash (panel, dash=true). */ +function anFmt(v, kind, dash) { + if (v === null || v === undefined || (typeof v === 'number' && !isFinite(v))) return dash ? '\u2013' : ''; + switch (kind) { + case 'pct': return (v * 100).toFixed(1); + case 'dur': return v.toFixed(2); + case 'metric': return String(parseFloat(v.toPrecision(3))); + case 'rating': return String(Math.round(v)); + } + return String(v); +} + +function anMetricLabel(k) { + if (typeof metricLabel === 'function') return metricLabel(k); + return String(k).replace(/[_-]+/g, ' ').replace(/\b\w/g, function (c) { return c.toUpperCase(); }); +} + +function anEsc(s) { return String(s).replace(/\|/g, '\\|').replace(/[\r\n]+/g, ' '); } + +/** Markdown report of a computeAnalytics result. opts = { title, scopeLabel, sortCol, sortDir }. */ +function analyticsToMarkdown(res, opts) { + opts = opts || {}; + var col = opts.sortCol || 'favRate'; + var dir = opts.sortDir === undefined ? -1 : opts.sortDir; + var head = '## USCG stats: ' + anEsc(opts.title || 'session') + ' (' + (opts.scopeLabel ? anEsc(opts.scopeLabel) + ', ' : '') + res.total + ' items)'; + var out = [head, + "_Fav % / Rej % = share of the group's items; Avg s = mean duration; metric columns = mean over items that have it; rows with n < " + AN_MIN_N + ' are marked *._']; + var constants = []; + for (var a = 0; a < res.axes.length; a++) { + var ax = res.axes[a]; + if (ax.constant) { if (ax.rows.length) constants.push(ax.label + ' = ' + anEsc(ax.rows[0].label)); continue; } + var cols = ['Value', 'n', 'Fav %', 'Rej %', 'Avg s'].concat(res.metricKeys.map(anMetricLabel)); + var seps = ['---', '---:', '---:', '---:', '---:'].concat(res.metricKeys.map(function () { return '---:'; })); + if (res.hasRatings) { cols.push('Rating'); seps.push('---:'); } + out.push('', '### ' + anEsc(ax.label), '| ' + cols.join(' | ') + ' |', '|' + seps.join('|') + '|'); + var rows = sortAnalyticsRows(ax.rows, col, dir); + for (var r = 0; r < rows.length; r++) { + var row = rows[r]; + var cells = [anEsc(row.label), String(row.n) + (row.enough ? '' : '*'), anFmt(row.favRate, 'pct'), anFmt(row.rejRate, 'pct'), anFmt(row.durMean, 'dur')]; + for (var k = 0; k < res.metricKeys.length; k++) { + var mm = row.metrics[res.metricKeys[k]]; + cells.push(mm ? anFmt(mm.mean, 'metric') : ''); + } + if (res.hasRatings) cells.push(row.ratingMean === null ? '' : anFmt(row.ratingMean, 'rating') + ' (' + row.ratingN + ')'); + out.push('| ' + cells.join(' | ') + ' |'); + } + } + if (constants.length) out.push('', 'Same for every item: ' + constants.join(', ')); + if (res.skippedAxes && res.skippedAxes.length) { + out.push('', 'Skipped (too many distinct values): ' + res.skippedAxes.map(function (s) { return anEsc(s.id) + ' (' + s.distinct + ')'; }).join(', ')); + } + return out.join('\n') + '\n'; +} + +if (typeof module !== 'undefined' && module.exports) { + module.exports = { + AN_AXES: AN_AXES, AN_MAX_WF_DISTINCT: AN_MAX_WF_DISTINCT, AN_MIN_N: AN_MIN_N, + computeAnalytics: computeAnalytics, sortAnalyticsRows: sortAnalyticsRows, + analyticsToMarkdown: analyticsToMarkdown, anAxisValue: anAxisValue, anFmt: anFmt + }; +} diff --git a/resources/logic_analytics_panel.js b/resources/logic_analytics_panel.js new file mode 100644 index 0000000..5e7d2c7 --- /dev/null +++ b/resources/logic_analytics_panel.js @@ -0,0 +1,303 @@ +/** + * SETTINGS STATS PANEL (DOM) + * A docked, non-modal panel (header bar button / cog entry) that shows, per setting axis, how often each value + * was favorited / rejected, its mean duration, mean metrics and mean pairwise rating. The numbers come from + * computeAnalytics() (logic_analytics.js). Loaded after logic_compare.js: uses cmpEl, cmpCopyText, cmpRatings. + * Every node is built with textContent (manifest strings never reach innerHTML). + * + * Refresh hooks: analyticsPanelOnDataChanged() is called from updateJSONs() (every pipeline run / favorite / + * reject / new item) and cmpAfterRatingsChanged() (votes); it coalesces renders to once per 250 ms. + */ + +var _anPanel = { open: false, collapsed: false, dirty: false, raf: 0, lastRender: 0, scope: 'filter', sortCol: 'favRate', sortDir: -1, openAxes: null }; +var AN_STORE_KEY = 'ultimate_grid_stats'; +var AN_ROW_CAP = 50; +var AN_RENDER_GAP_MS = 250; +var AN_DEFAULT_OPEN = ['model', 'lora', 'sampler', 'scheduler', 'steps', 'cfg']; + +function anLoadSettings() { + try { + var s = JSON.parse(localStorage.getItem(AN_STORE_KEY) || 'null'); + if (!s || typeof s !== 'object') return; + if (s.scope === 'all' || s.scope === 'filter') _anPanel.scope = s.scope; + if (typeof s.collapsed === 'boolean') _anPanel.collapsed = s.collapsed; + if (typeof s.sortCol === 'string' && s.sortCol.length < 80) _anPanel.sortCol = s.sortCol; + if (s.sortDir === 1 || s.sortDir === -1) _anPanel.sortDir = s.sortDir; + } catch (e) { /* storage unavailable */ } +} + +function anSaveSettings() { + try { + localStorage.setItem(AN_STORE_KEY, JSON.stringify({ + scope: _anPanel.scope, collapsed: _anPanel.collapsed, sortCol: _anPanel.sortCol, sortDir: _anPanel.sortDir + })); + } catch (e) { /* storage unavailable */ } +} + +/** Items the panel counts. 'filter' = the content filters, ignoring the favorites/non-fav/rejected visibility toggles. */ +function anScopeItems() { + var src = activeData || []; + if (_anPanel.scope === 'all') { + return src.filter(function (d) { return !(d.preview && isSupersededPreview(d)); }); + } + var ctx = pipelineFilterCtx(); + return src.filter(function (d) { return pipelineMatches(d, ctx, true); }); +} + +function anCompute() { + return computeAnalytics(anScopeItems(), { + meta: meta, + ratings: typeof cmpRatings === 'function' ? cmpRatings() : null + }); +} + +function anSessionTitle() { + var inp = document.getElementById('session-input'); + return (inp && inp.value) || (meta && meta.session_name) || 'session'; +} + +function anScopeLabel() { + return _anPanel.scope === 'all' ? 'whole session' : 'current filter'; +} + +/** Markdown for the current scope and sort (Copy Markdown button; also used by the e2e test). */ +function analyticsPanelMarkdown() { + return analyticsToMarkdown(anCompute(), { + title: anSessionTitle(), scopeLabel: anScopeLabel(), sortCol: _anPanel.sortCol, sortDir: _anPanel.sortDir + }); +} + +function anPanelEl() { return document.getElementById('analytics-panel'); } + +function anCreatePanel() { + var panel = cmpEl('div', 'ap-panel'); + panel.id = 'analytics-panel'; + + var head = cmpEl('div', 'ap-head'); + head.appendChild(cmpEl('span', 'ap-title', 'STATS')); + + var scope = document.createElement('select'); + scope.id = 'analytics-scope'; + scope.title = 'Which items are counted'; + [['filter', 'Current filter (all favs/rejects counted)'], ['all', 'Whole session']].forEach(function (o) { + var opt = document.createElement('option'); + opt.value = o[0]; + opt.textContent = o[1]; + scope.appendChild(opt); + }); + scope.value = _anPanel.scope; + scope.addEventListener('change', function () { + _anPanel.scope = scope.value === 'all' ? 'all' : 'filter'; + anSaveSettings(); + renderAnalyticsPanel(); + }); + head.appendChild(scope); + + var copy = cmpEl('button', '', 'Copy Markdown'); + copy.id = 'analytics-copy-md'; + copy.title = 'Copy every table as Markdown'; + copy.addEventListener('click', function () { cmpCopyText(analyticsPanelMarkdown(), copy); }); + head.appendChild(copy); + + var col = cmpEl('button', '', _anPanel.collapsed ? '▸' : '▾'); + col.id = 'analytics-collapse-btn'; + col.title = 'Collapse / expand'; + col.addEventListener('click', function () { + _anPanel.collapsed = !_anPanel.collapsed; + anSaveSettings(); + col.textContent = _anPanel.collapsed ? '▸' : '▾'; + panel.classList.toggle('ap-collapsed', _anPanel.collapsed); + if (!_anPanel.collapsed && _anPanel.dirty) renderAnalyticsPanel(); + }); + head.appendChild(col); + + var close = cmpEl('button', '', '✕'); + close.id = 'analytics-close-btn'; + close.title = 'Close (Esc)'; + close.addEventListener('click', function () { closeAnalyticsPanel(); }); + head.appendChild(close); + panel.appendChild(head); + + var summary = cmpEl('div', 'analytics-summary'); + summary.id = 'analytics-summary'; + panel.appendChild(summary); + panel.appendChild(cmpEl('div', 'ap-body')); + panel.appendChild(cmpEl('div', 'ap-foot')); + panel.classList.toggle('ap-collapsed', _anPanel.collapsed); + document.body.appendChild(panel); + return panel; +} + +function openAnalyticsPanel() { + if (!_anPanel.loaded) { anLoadSettings(); _anPanel.loaded = true; } + var panel = anPanelEl(); + if (!panel) panel = anCreatePanel(); + panel.style.display = 'flex'; + _anPanel.open = true; + var cog = document.getElementById('cog-menu-dropdown'); + if (cog && cog.style.display !== 'none' && cog.style.display !== '' && typeof toggleCogMenu === 'function') toggleCogMenu(); + renderAnalyticsPanel(); +} + +function toggleAnalyticsPanel() { + if (_anPanel.open) closeAnalyticsPanel(); else openAnalyticsPanel(); +} + +/** Returns true when it closed something (Escape priority chain). */ +function closeAnalyticsPanel() { + if (!_anPanel.open) return false; + var panel = anPanelEl(); + if (panel) panel.style.display = 'none'; + _anPanel.open = false; + if (_anPanel.raf) { clearTimeout(_anPanel.raf); _anPanel.raf = 0; } + return true; +} + +function anSortArrow(col) { + return _anPanel.sortCol === col ? (_anPanel.sortDir < 0 ? ' ▼' : ' ▲') : ''; +} + +function anSetSort(col) { + if (_anPanel.sortCol === col) _anPanel.sortDir = -_anPanel.sortDir; + else { _anPanel.sortCol = col; _anPanel.sortDir = col === 'label' ? 1 : -1; } + anSaveSettings(); + renderAnalyticsPanel(); +} + +function anHeaderCell(text, col, numeric) { + var th = cmpEl('th', 'analytics-th ap-sortable', text + anSortArrow(col)); + th.dataset.col = col; + if (numeric) th.style.textAlign = 'right'; + th.addEventListener('click', function () { anSetSort(col); }); + return th; +} + +function anAxisTable(ax, res) { + var wrap = cmpEl('div', 'ap-table-wrap'); + var table = cmpEl('table', 'analytics-table'); + var thead = document.createElement('thead'); + var hr = document.createElement('tr'); + hr.appendChild(anHeaderCell('Value', 'label', false)); + hr.appendChild(anHeaderCell('n', 'n', true)); + hr.appendChild(anHeaderCell('Fav %', 'favRate', true)); + hr.appendChild(anHeaderCell('Rej %', 'rejRate', true)); + hr.appendChild(anHeaderCell('Avg s', 'durMean', true)); + res.metricKeys.forEach(function (k) { hr.appendChild(anHeaderCell(metricLabel(k), 'metric:' + k, true)); }); + if (res.hasRatings) hr.appendChild(anHeaderCell('Rating', 'rating', true)); + thead.appendChild(hr); + table.appendChild(thead); + + var tbody = document.createElement('tbody'); + var rows = sortAnalyticsRows(ax.rows, _anPanel.sortCol, _anPanel.sortDir); + var bestKey = (ax.rows[0] && ax.rows[0].enough && ax.rows[0].favRate > 0) ? ax.rows[0].key : null; + rows.slice(0, AN_ROW_CAP).forEach(function (r) { + var tr = cmpEl('tr', 'analytics-row'); + if (!r.enough) tr.classList.add('sb-row-dim'); + if (bestKey !== null && r.key === bestKey) tr.classList.add('sb-row-best'); + var v = cmpEl('td', 'sb-val', r.label); + v.title = r.key; + tr.appendChild(v); + var cells = [String(r.n), anFmt(r.favRate, 'pct', true), anFmt(r.rejRate, 'pct', true), anFmt(r.durMean, 'dur', true)]; + res.metricKeys.forEach(function (k) { cells.push(r.metrics[k] ? anFmt(r.metrics[k].mean, 'metric', true) : '–'); }); + if (res.hasRatings) cells.push(r.ratingMean === null ? '–' : anFmt(r.ratingMean, 'rating', true) + ' (' + r.ratingN + ')'); + cells.forEach(function (c) { tr.appendChild(cmpEl('td', 'sb-num', c)); }); + tbody.appendChild(tr); + }); + table.appendChild(tbody); + wrap.appendChild(table); + if (rows.length > AN_ROW_CAP) { + var more = cmpEl('div', 'ap-more', '+' + (rows.length - AN_ROW_CAP) + ' more (included in Copy Markdown)'); + var holder = document.createElement('div'); + holder.appendChild(wrap); + holder.appendChild(more); + return holder; + } + return wrap; +} + +function anIsAxisOpen(ax) { + var st = _anPanel.openAxes; + if (st && Object.prototype.hasOwnProperty.call(st, ax.id)) return !!st[ax.id]; + return ax.kind === 'wf' || AN_DEFAULT_OPEN.indexOf(ax.id) >= 0; +} + +function renderAnalyticsPanel() { + var panel = anPanelEl(); + if (!panel || !_anPanel.open) return; + _anPanel.dirty = false; + _anPanel.lastRender = Date.now(); + var res = anCompute(); + + var body = panel.querySelector('.ap-body'); + var summary = document.getElementById('analytics-summary'); + var foot = panel.querySelector('.ap-foot'); + var scrollTop = body.scrollTop; + // Remember which axes the user opened or closed (survives the rebuild) + body.querySelectorAll('details[data-axis]').forEach(function (d) { + if (!_anPanel.openAxes) _anPanel.openAxes = {}; + _anPanel.openAxes[d.dataset.axis] = d.open; + }); + + var scopeSel = document.getElementById('analytics-scope'); + if (scopeSel && scopeSel.value !== _anPanel.scope) scopeSel.value = _anPanel.scope; + + // Summary line + var o = res.overall; + var metricsOn = 0; + res.metricKeys.forEach(function (k) { if (o.metrics[k] && o.metrics[k].n > metricsOn) metricsOn = o.metrics[k].n; }); + var favPct = res.total ? (res.favorited / res.total * 100).toFixed(0) : '0'; + var bits = [res.total + ' items', res.favorited + ' favorited (' + favPct + '%)', res.rejected + ' rejected']; + if (o.durMean !== null) bits.push('avg ' + o.durMean.toFixed(2) + ' s'); + bits.push('metrics on ' + metricsOn); + bits.push('ratings on ' + o.ratingN); + summary.textContent = bits.join(' · '); + + // Axis tables + while (body.firstChild) body.removeChild(body.firstChild); + var constants = []; + res.axes.forEach(function (ax) { + if (ax.constant) { if (ax.rows.length) constants.push(ax.label + ' = ' + ax.rows[0].label); return; } + var det = cmpEl('details', 'sb-axis'); + det.dataset.axis = ax.id; + det.open = anIsAxisOpen(ax); + var sum = document.createElement('summary'); + sum.textContent = ax.label; + sum.appendChild(cmpEl('span', 'sb-meta', ' · ' + ax.distinct + ' values')); + det.appendChild(sum); + det.appendChild(anAxisTable(ax, res)); + det.addEventListener('toggle', function () { + if (!_anPanel.openAxes) _anPanel.openAxes = {}; + _anPanel.openAxes[ax.id] = det.open; + }); + body.appendChild(det); + }); + if (!res.axes.length || res.total === 0) body.appendChild(cmpEl('div', 'ap-more', 'No items in this scope.')); + body.scrollTop = scrollTop; + + // Footer + while (foot.firstChild) foot.removeChild(foot.firstChild); + if (constants.length) foot.appendChild(cmpEl('div', '', 'Same for every item: ' + constants.join(', '))); + if (res.skippedAxes.length) { + foot.appendChild(cmpEl('div', '', 'Skipped (too many distinct values): ' + + res.skippedAxes.map(function (s) { return s.id.slice(3) + ' (' + s.distinct + ')'; }).join(', '))); + } + if (_anPanel.scope === 'filter' && typeof topFilters !== 'undefined' && !topFilters.showRejected) { + foot.appendChild(cmpEl('div', '', 'Rejected items are counted even though the grid hides them.')); + } + foot.appendChild(cmpEl('div', '', 'Rows with n < ' + AN_MIN_N + ' are dimmed and sorted last.')); +} + +/** Hook target (updateJSONs, cmpAfterRatingsChanged): coalesced live refresh. */ +function analyticsPanelOnDataChanged() { + if (!_anPanel.open) return; + if (_anPanel.collapsed) { _anPanel.dirty = true; return; } + if (_anPanel.raf) return; + var wait = Math.max(0, AN_RENDER_GAP_MS - (Date.now() - _anPanel.lastRender)); + var run = function () { + _anPanel.raf = 0; + if (_anPanel.open && !_anPanel.collapsed) renderAnalyticsPanel(); + }; + _anPanel.raf = setTimeout(function () { + if (typeof requestAnimationFrame === 'function') requestAnimationFrame(run); else run(); + }, wait); +} diff --git a/resources/logic_compare.js b/resources/logic_compare.js index 1875c09..ec1246b 100644 --- a/resources/logic_compare.js +++ b/resources/logic_compare.js @@ -295,6 +295,7 @@ async function clearComparisons() { } function cmpAfterRatingsChanged() { + if (typeof analyticsPanelOnDataChanged === 'function') analyticsPanelOnDataChanged(); if (typeof refreshLabelOverlays === 'function') refreshLabelOverlays(); if ((currentSort === 'rating' || currentSecondarySort === 'rating') && typeof updateDataPipeline === 'function') { updateDataPipeline(); diff --git a/resources/logic_pipeline.js b/resources/logic_pipeline.js index 76aa21a..18ab87d 100644 --- a/resources/logic_pipeline.js +++ b/resources/logic_pipeline.js @@ -549,21 +549,8 @@ function updateDataPipeline() { // Abort controller for canceling in-progress async pipelines let pipelineAbortId = 0; -function executePipeline() { - const startTime = performance.now(); - refreshSupersededFps(); - if (typeof populateMetricFilterUI === 'function') populateMetricFilterUI(); - if (typeof populateWfParamFilterUI === 'function') populateWfParamFilterUI(); - const currentFilterKey = getFilterKey(); - const filtersChanged = currentFilterKey !== lastFilterKey; - - // Always filter from activeData to prevent data loss from self-filtering - console.log(`[Pipeline] ${filtersChanged ? 'Filters changed, full reprocess' : 'Re-applying filters'}`); - if (filtersChanged) { - lastFilterKey = currentFilterKey; - } - - // Check for button filters once to avoid repeated Set.size lookups +/** Filter context for pipelineMatches(): computed once per pass. */ +function pipelineFilterCtx() { const hasButtonFilters = filters.sampler.size > 0 || filters.scheduler.size > 0 || filters.denoise.size > 0 || @@ -580,65 +567,92 @@ function executePipeline() { (filters.passType && filters.passType.size > 0); const hasSearchFilters = typeof matchesSearchFilters === 'function' && searchFilters.length > 0; + return { hasButtonFilters: hasButtonFilters, hasSearchFilters: hasSearchFilters }; +} - processedData = activeData.filter(d => { +/** + * Does item d pass the content filters (superseded previews, button, search, logic and quick filters)? + * ignoreVisibility = true skips the Favorites / Non-Favorited / Rejected visibility toggles (the Stats panel + * counts favorites and rejects even while the grid hides them). + */ +function pipelineMatches(d, ctx, ignoreVisibility) { + if (!ignoreVisibility) { // Apply top-level filters (Favorites/Non-Favorited/Rejected visibility) if (d.favorited && !topFilters.showFavorites) return false; if (!d.favorited && !d.rejected && !topFilters.showNonFavorited) return false; if (d.rejected && !topFilters.showRejected) return false; - if (d.preview && isSupersededPreview(d)) return false; + } + if (d.preview && isSupersededPreview(d)) return false; - // Apply button filters (skip entirely if none are active) - if (hasButtonFilters) { - if (filters.sampler.size > 0 && !filters.sampler.has(d.sampler)) return false; - if (filters.scheduler.size > 0 && !filters.scheduler.has(d.scheduler)) return false; - if (filters.denoise.size > 0 && !filters.denoise.has(d.denoise)) return false; - if (filters.lora.size > 0 && !filters.lora.has(d.lora)) return false; - if (filters.model.size > 0 && !filters.model.has(d.model || meta.model || "Default")) return false; - if (filters.positive.size > 0) { - const st = document.getElementById('toggle-triggers')?.checked !== false; - const posValue = st ? (d.positive || meta.positive || "") : (d.config_positive || d.positive || meta.positive || ""); - if (!filters.positive.has(posValue)) return false; - } - if (filters.negative.size > 0) { - const st = document.getElementById('toggle-triggers')?.checked !== false; - const negValue = st ? (d.negative || meta.negative || "") : (d.config_negative || d.negative || meta.negative || ""); - if (!filters.negative.has(negValue)) return false; - } - if (filters.size.size > 0 && !filters.size.has(`${d.width}x${d.height}`)) return false; - if (filters.seed.size > 0 && !filters.seed.has(d.seed)) return false; - if (filters.steps.size > 0 && !filters.steps.has(String(d.steps))) return false; - if (filters.cfg.size > 0 && !filters.cfg.has(String(d.cfg))) return false; - if (filters.upscaleMethod.size > 0) { - let upVal; - if (!d.upscaled) { upVal = 'No Upscale'; } - else { - const mode = d.upscale_mode || ''; - const model = d.upscale_model; - const shortModel = model ? String(model).replace(/\\/g, '/').split('/').pop().replace(/\.[^.]+$/, '') : ''; - upVal = shortModel ? `${mode} + ${shortModel}` : mode || 'Upscaled'; - } - if (!filters.upscaleMethod.has(upVal)) return false; - } - if (filters.mediaType.size > 0) { - const mt = d.media_type || 'image'; - if (!filters.mediaType.has(mt)) return false; - } - if (filters.passType && filters.passType.size > 0) { - if (!filters.passType.has(d.preview ? 'Preview' : 'Full')) return false; - } + // Apply button filters (skip entirely if none are active) + if (ctx.hasButtonFilters) { + if (filters.sampler.size > 0 && !filters.sampler.has(d.sampler)) return false; + if (filters.scheduler.size > 0 && !filters.scheduler.has(d.scheduler)) return false; + if (filters.denoise.size > 0 && !filters.denoise.has(d.denoise)) return false; + if (filters.lora.size > 0 && !filters.lora.has(d.lora)) return false; + if (filters.model.size > 0 && !filters.model.has(d.model || meta.model || "Default")) return false; + if (filters.positive.size > 0) { + const st = document.getElementById('toggle-triggers')?.checked !== false; + const posValue = st ? (d.positive || meta.positive || "") : (d.config_positive || d.positive || meta.positive || ""); + if (!filters.positive.has(posValue)) return false; } + if (filters.negative.size > 0) { + const st = document.getElementById('toggle-triggers')?.checked !== false; + const negValue = st ? (d.negative || meta.negative || "") : (d.config_negative || d.negative || meta.negative || ""); + if (!filters.negative.has(negValue)) return false; + } + if (filters.size.size > 0 && !filters.size.has(`${d.width}x${d.height}`)) return false; + if (filters.seed.size > 0 && !filters.seed.has(d.seed)) return false; + if (filters.steps.size > 0 && !filters.steps.has(String(d.steps))) return false; + if (filters.cfg.size > 0 && !filters.cfg.has(String(d.cfg))) return false; + if (filters.upscaleMethod.size > 0) { + let upVal; + if (!d.upscaled) { upVal = 'No Upscale'; } + else { + const mode = d.upscale_mode || ''; + const model = d.upscale_model; + const shortModel = model ? String(model).replace(/\\/g, '/').split('/').pop().replace(/\.[^.]+$/, '') : ''; + upVal = shortModel ? `${mode} + ${shortModel}` : mode || 'Upscaled'; + } + if (!filters.upscaleMethod.has(upVal)) return false; + } + if (filters.mediaType.size > 0) { + const mt = d.media_type || 'image'; + if (!filters.mediaType.has(mt)) return false; + } + if (filters.passType && filters.passType.size > 0) { + if (!filters.passType.has(d.preview ? 'Preview' : 'Full')) return false; + } + } - if (hasSearchFilters && !matchesSearchFilters(d)) return false; + if (ctx.hasSearchFilters && !matchesSearchFilters(d)) return false; - // Apply logic filters (numeric comparisons) - if (!matchesLogicFilters(d)) return false; + // Apply logic filters (numeric comparisons) + if (!matchesLogicFilters(d)) return false; - // Apply quick filters (faceted chip AND logic) - if (!matchesQuickFilters(d)) return false; + // Apply quick filters (faceted chip AND logic) + if (!matchesQuickFilters(d)) return false; - return true; - }); + return true; +} + +function executePipeline() { + const startTime = performance.now(); + refreshSupersededFps(); + if (typeof populateMetricFilterUI === 'function') populateMetricFilterUI(); + if (typeof populateWfParamFilterUI === 'function') populateWfParamFilterUI(); + const currentFilterKey = getFilterKey(); + const filtersChanged = currentFilterKey !== lastFilterKey; + + // Always filter from activeData to prevent data loss from self-filtering + console.log(`[Pipeline] ${filtersChanged ? 'Filters changed, full reprocess' : 'Re-applying filters'}`); + if (filtersChanged) { + lastFilterKey = currentFilterKey; + } + + const fctx = pipelineFilterCtx(); + + processedData = activeData.filter(d => pipelineMatches(d, fctx, false)); // Apply sorting // Fast path: for oldest/newest with no secondary sort, activeData is stored newest-first diff --git a/resources/logic_state.js b/resources/logic_state.js index 6236e0c..fac518d 100644 --- a/resources/logic_state.js +++ b/resources/logic_state.js @@ -65,14 +65,14 @@ const sortOptionsList = [ // Optional per-image metrics (item.metrics = {sharpness, entropy, ...}); see image_metrics.py. // Built-ins come first in this order, unknown keys (scorer packs) follow alphabetically. -const METRIC_BUILTIN_ORDER = ['sharpness', 'entropy', 'colorfulness', 'contrast', 'luma', 'saturation']; +const METRIC_BUILTIN_ORDER = ['sharpness', 'entropy', 'colorfulness', 'contrast', 'luma', 'saturation', 'sobel']; const METRIC_LABELS = { sharpness: 'Sharpness', entropy: 'Entropy', colorfulness: 'Colorful', - contrast: 'Contrast', luma: 'Luma', saturation: 'Saturation', aesthetic: 'Aesthetic' + contrast: 'Contrast', luma: 'Luma', saturation: 'Saturation', sobel: 'Sobel', aesthetic: 'Aesthetic' }; const METRIC_SHORT = { sharpness: 'Shp', entropy: 'Ent', colorfulness: 'Col', - contrast: 'Ctr', luma: 'Lum', saturation: 'Sat', aesthetic: 'Aes' + contrast: 'Ctr', luma: 'Lum', saturation: 'Sat', sobel: 'Sob', aesthetic: 'Aes' }; // --- SEARCH FILTER STATE --- diff --git a/resources/logic_ui.js b/resources/logic_ui.js index 1fe0595..48015d3 100644 --- a/resources/logic_ui.js +++ b/resources/logic_ui.js @@ -583,7 +583,11 @@ document.addEventListener('keydown', (e) => { const filtersPopup = document.getElementById('filters-popup'); if (filtersPopup && filtersPopup.style.display !== 'none') { toggleFiltersPopup(); + return; } + + // Close the docked Stats panel + if (typeof closeAnalyticsPanel === 'function' && closeAnalyticsPanel()) return; } }); @@ -1808,6 +1812,7 @@ function updateJSONs(visible) { // Update button labels with unique counts updateJSONButtonLabels(goodUniqueCount, favUniqueCount, rejUniqueCount); + if (typeof analyticsPanelOnDataChanged === 'function') analyticsPanelOnDataChanged(); }, 100); // Reduced from 300ms since we have separate debounce above } @@ -3075,7 +3080,7 @@ async function startSessionMetrics() { try { const resp = await fetch('/config_tester/compute_metrics', { method: 'POST', headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify({ session_name: sessionName }) + body: JSON.stringify({ session_name: sessionName, fill_missing: true }) }); const result = await resp.json(); if (result.error) { if (out) out.textContent = result.error; return; } diff --git a/resources/report.css b/resources/report.css index 15d0f8c..18ef620 100644 --- a/resources/report.css +++ b/resources/report.css @@ -2110,3 +2110,65 @@ body { background: rgba(0, 0, 0, 0.6); color: #fff; font-size: 26px; cursor: pointer; pointer-events: auto; z-index: 3; } + +/* ============================================================ + ANALYTICS PANEL (docked, non-modal Settings Stats) + ============================================================ */ + +.ap-panel { + position: fixed; + top: 44px; + right: 10px; + width: min(680px, 96vw); + max-height: 75vh; + z-index: 450; + display: flex; + flex-direction: column; + background: #111; + border: 1px solid #333; + border-radius: 8px; + box-shadow: 0 8px 28px rgba(0, 0, 0, 0.6); +} + +.ap-head { + display: flex; + align-items: center; + gap: 8px; + padding: 8px 12px; + border-bottom: 2px solid #3e8ed0; + flex-shrink: 0; +} + +.ap-title { + font-size: 12px; + font-weight: 700; + letter-spacing: 1px; + color: #3e8ed0; + margin-right: auto; +} + +.ap-head select, .ap-head button { + background: #1a1a1a; + color: #ccc; + border: 1px solid #333; + border-radius: 4px; + font-size: 10px; + padding: 4px 8px; + cursor: pointer; +} + +.ap-head button:hover { background: #262626; color: #fff; } +.ap-panel.ap-collapsed .ap-body, +.ap-panel.ap-collapsed #analytics-summary, +.ap-panel.ap-collapsed .ap-foot { display: none; } +.ap-body { overflow-y: auto; flex: 1; min-height: 0; } +.ap-table-wrap { overflow-x: auto; } +th.analytics-th.ap-sortable { cursor: pointer; user-select: none; white-space: nowrap; } +th.analytics-th.ap-sortable:hover { color: #aaa; } +.ap-foot { font-size: 10px; color: #777; padding: 6px 12px; border-top: 1px solid #222; flex-shrink: 0; } +.ap-foot div { margin-top: 2px; } +.ap-more { font-size: 10px; color: #777; padding: 4px 12px; } + +@media (max-width: 800px) { + .ap-panel { right: 2vw; top: auto; bottom: 8px; max-height: 60vh; } +} diff --git a/resources/template.html b/resources/template.html index 8afff8e..d1e0d1d 100644 --- a/resources/template.html +++ b/resources/template.html @@ -53,6 +53,10 @@ </svg> </button> + <button class="fs-btn" id="analytics-open-btn" onclick="toggleAnalyticsPanel()" title="Stats per setting (favorites rate, time, metrics)"> + <svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><line x1="6" y1="20" x2="6" y2="12"></line><line x1="12" y1="20" x2="12" y2="6"></line><line x1="18" y1="20" x2="18" y2="9"></line></svg> + </button> + <button class="fs-btn" onclick="loadSession()" title="Reload Session (R)"> <svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"> <polyline points="23 4 23 10 17 10"></polyline> @@ -256,12 +260,12 @@ <div class="cog-panel-section-title">IMAGE METRICS</div> <button class="session-action-btn" id="compute-metrics-btn" style="background: #2b5f9e; width: 100%; font-size: 11px; padding: 10px;" - onclick="startSessionMetrics()" title="Measure sharpness, entropy, colorfulness, contrast, luma and saturation for every image that has no metrics yet (CPU only)"> + onclick="startSessionMetrics()" title="Measure sharpness, entropy, colorfulness, contrast, luma, saturation and Sobel edges for every image that is missing any of them (CPU only)"> COMPUTE METRICS FOR THIS SESSION </button> <div id="compute-metrics-progress" style="display: none; margin-top: 6px; font-size: 10px; color: #9ab;"></div> <div style="font-size: 9px; color: #666; margin-top: 6px;"> - Adds sortable/filterable metrics to older sessions. Skips videos and images that already have metrics. Do not run it while this session is still generating. + Fills in metrics missing from older items (for example Sobel on sessions measured before it existed). Never changes values already stored. Skips videos. Do not run it while this session is still generating. </div> </div> @@ -430,8 +434,13 @@ <div class="cog-panel-section"> <div class="cog-panel-section-title">MANIFEST ANALYTICS</div> <div style="font-size: 9px; color: #666; margin-bottom: 8px;"> - Analyze favorited items in the current session manifest. + Per-setting favorites rate, time and metrics (live), or usage counts of favorited items. </div> + <button class="session-action-btn" + style="background: #3e8ed0; width: 100%; font-size: 11px; padding: 10px; margin-bottom: 6px;" + onclick="openAnalyticsPanel()" title="Favorites rate, rejects, time and metrics per setting value (live panel)"> + SETTINGS STATS (PER AXIS) + </button> <button class="session-action-btn" style="background: #8b5cf6; width: 100%; font-size: 11px; padding: 10px; margin-bottom: 6px;" onclick="runManifestAnalysis('lora')" title="Count LoRA usage across favorited items"> diff --git a/tests/e2e/test_analytics_panel.py b/tests/e2e/test_analytics_panel.py new file mode 100644 index 0000000..8326fa4 --- /dev/null +++ b/tests/e2e/test_analytics_panel.py @@ -0,0 +1,91 @@ +"""Browser test of the docked Settings Stats panel (resources/logic_analytics*.js inlined by html_generator). + +Needs `pip install -r tests/e2e/requirements.txt` and `python -m playwright install chromium`. +With USCG_REQUIRE_E2E=1 (CI) a missing playwright is a failure instead of a skip. +""" +import os + +import pytest + +if os.environ.get("USCG_REQUIRE_E2E") == "1": + import playwright.sync_api # noqa: F401 +else: + pytest.importorskip("playwright.sync_api") + +from playwright.sync_api import expect # noqa: E402 + +pytestmark = pytest.mark.e2e + + +def _axis(page, axis): + return page.locator(f'#analytics-panel details[data-axis="{axis}"]') + + +def _row_cells(row): + return [c.strip() for c in row.locator("td").all_inner_texts()] + + +def test_panel_groups_and_metrics(dashboard): + page = dashboard.page + dashboard.ev("activeData.forEach((d, i) => { d.metrics = {sharpness: 100 + i, sobel: 0.01 * (i % 5)};" + " d.favorited = d.sampler === 'euler'; }); updateDataPipeline();") + page.locator("#analytics-open-btn").click() + expect(page.locator("#analytics-panel")).to_be_visible() + first = _axis(page, "sampler").locator("tbody tr").first + cells = _row_cells(first) + assert cells[0] == "euler" and cells[1] == "8" and cells[2] == "100.0" + head = _axis(page, "sampler").locator("thead").inner_text().lower() # CSS upper-cases the header + assert "sobel" in head and "sharpness" in head + assert _axis(page, "model").locator("tbody tr").count() == 2 + + +def test_panel_live_updates(dashboard): + page = dashboard.page + page.locator("#analytics-open-btn").click() + expect(page.locator("#analytics-summary")).to_contain_text("24 items") + page.locator("#card-1000 .favorite-btn").click() + expect(page.locator("#analytics-summary")).to_contain_text("1 favorited") + page.locator("#card-1005 .reject-btn").dispatch_event("click") # the docked panel may cover the card + dpm = _axis(page, "sampler").locator("tbody tr", has_text="dpmpp_2m") + expect(dpm.locator("td").nth(3)).to_have_text("12.5") # counted although the grid hides it + expect(dashboard.visible_cards).to_have_count(23) + item = dashboard.make_item(24, "sdxl_base.safetensors", "euler", "normal", 9.0, 40, "None") + page.evaluate("item => window.postMessage({type:'update_data', data:{new_items:[item]}}, '*')", item) + expect(page.locator("#analytics-summary")).to_contain_text("25 items") + + +def test_panel_scope_markdown_escape(dashboard): + page = dashboard.page + page.locator("#analytics-open-btn").click() + dashboard.ev("filters.sampler.clear(); filters.sampler.add('euler'); updateDataPipeline();") + expect(page.locator("#analytics-summary")).to_contain_text("8 items") + page.select_option("#analytics-scope", "all") + expect(page.locator("#analytics-summary")).to_contain_text("24 items") + md = dashboard.ev("analyticsPanelMarkdown()") + assert md.startswith("## USCG stats") + assert "| euler |" in md + page.locator("#analytics-copy-md").click() + expect(page.locator("#analytics-copy-md")).to_have_text("Copied") + page.locator("#analytics-collapse-btn").click() + expect(page.locator("#analytics-panel .ap-body")).to_be_hidden() + expect(page.locator("#analytics-panel .ap-head")).to_be_visible() + page.locator("#analytics-collapse-btn").click() + expect(page.locator("#analytics-panel .ap-body")).to_be_visible() + page.keyboard.press("Escape") + expect(page.locator("#analytics-panel")).to_be_hidden() + + +def test_panel_cog_entry_and_manifest_text_is_not_html(dashboard): + page = dashboard.page + dashboard.ev("activeData[0].sampler = '<img src=x onerror=window.__pwned=1>'; updateDataPipeline();") + dashboard.ev("openAnalyticsPanel()") + expect(page.locator("#analytics-panel")).to_be_visible() + assert dashboard.ev("window.__pwned") is None + assert page.locator("#analytics-panel img").count() == 0 + + +def test_panel_fits_a_phone(phone_dashboard): + d = phone_dashboard() + d.ev("openAnalyticsPanel()") + box = d.page.locator("#analytics-panel").bounding_box() + assert box["width"] <= 412 and box["x"] >= 0 diff --git a/tests/js/analytics.test.js b/tests/js/analytics.test.js new file mode 100644 index 0000000..c7746ec --- /dev/null +++ b/tests/js/analytics.test.js @@ -0,0 +1,176 @@ +'use strict'; +const test = require('node:test'); +const assert = require('node:assert'); +const A = require('../../resources/logic_analytics.js'); + +let nextId = 1; +function item(o) { + return Object.assign({ + id: nextId++, file: '/view?x', model: 'm.safetensors', lora: 'None', sampler: 'euler', scheduler: 'simple', + cfg: 7, steps: 20, denoise: 1, seed: 1, width: 1024, height: 1024, + config_positive: 'a cat', positive: 'a cat', negative: '', duration: 1, favorited: false + }, o); +} +function many(n, o) { const out = []; for (let i = 0; i < n; i++) out.push(item(typeof o === 'function' ? o(i) : o)); return out; } +const axis = (res, id) => res.axes.find(a => a.id === id); +const row = (res, id, key) => axis(res, id).rows.find(r => r.key === key); + +test('grouping by sampler with fav and reject rates', () => { + const items = [ + ...many(3, i => ({ sampler: 'euler', favorited: i < 2 })), + ...many(3, i => ({ sampler: 'dpmpp', rejected: i === 0 })) + ]; + const res = A.computeAnalytics(items, {}); + const s = axis(res, 'sampler'); + assert.deepStrictEqual(s.rows.map(r => r.key), ['euler', 'dpmpp']); + assert.ok(Math.abs(s.rows[0].favRate - 2 / 3) < 1e-9); + assert.ok(Math.abs(s.rows[1].rejRate - 1 / 3) < 1e-9); + assert.strictEqual(res.overall.n, 6); + assert.strictEqual(res.total, 6); + assert.strictEqual(res.favorited, 2); + assert.strictEqual(res.rejected, 1); +}); + +test('duration mean skips missing, zero and NaN', () => { + const items = [item({ duration: 2 }), item({ duration: 4 }), item({ duration: 0 }), item({ duration: NaN }), item({ duration: undefined })]; + const res = A.computeAnalytics(items, {}); + assert.strictEqual(res.overall.durMean, 3); + assert.strictEqual(res.overall.durN, 2); + const none = A.computeAnalytics([item({ duration: 0 })], {}); + assert.strictEqual(none.overall.durMean, null); +}); + +test('metric means count only items that have the metric; key order', () => { + const items = [ + item({ metrics: { sobel: 0.2, sharpness: 10, aesthetic: 5 } }), + item({ metrics: { sharpness: 30 } }), + item({}) + ]; + const res = A.computeAnalytics(items, { metricOrder: ['sharpness', 'sobel'] }); + assert.deepStrictEqual(res.metricKeys, ['sharpness', 'sobel', 'aesthetic']); + assert.deepStrictEqual(res.overall.metrics.sharpness, { mean: 20, n: 2 }); + assert.deepStrictEqual(res.overall.metrics.sobel, { mean: 0.2, n: 1 }); +}); + +test('ratings map feeds ratingMean/ratingN; hasRatings false without votes', () => { + const a = item({}), b = item({}), c = item({}); + const ratings = new Map([[a.id, { rating: 1500, votes: 4 }], [b.id, { rating: 1600, votes: 2 }]]); + const res = A.computeAnalytics([a, b, c], { ratings }); + assert.strictEqual(res.hasRatings, true); + assert.strictEqual(res.overall.ratingMean, 1550); + assert.strictEqual(res.overall.ratingN, 2); + assert.strictEqual(A.computeAnalytics([a], { ratings: new Map() }).hasRatings, false); + assert.strictEqual(A.computeAnalytics([a], { ratings: null }).hasRatings, false); + assert.strictEqual(A.computeAnalytics([a], { ratings: new Map([[a.id, { rating: 1500, votes: 0 }]]) }).hasRatings, false); +}); + +test('normalizeAxis reuse: lora stack order, model path separators, upscale base resolution', () => { + const items = [ + item({ lora: 'b.safetensors:1 + a.safetensors:0.8' }), + item({ lora: 'a.safetensors:0.8 + b.safetensors:1' }), + item({ model: 'sub\\m.safetensors' }), + item({ model: 'sub/m.safetensors' }), + item({ width: 2048, height: 2048, base_width: 1024, base_height: 1024, upscaled: true }) + ]; + const res = A.computeAnalytics(items, {}); + const lora = axis(res, 'lora'); + assert.strictEqual(lora.rows.find(r => r.n === 2 && r.key !== 'None').n, 2); + assert.strictEqual(axis(res, 'model').rows.find(r => r.key === 'sub/m.safetensors').n, 2); + const reso = axis(res, 'resolution'); + assert.strictEqual(reso.rows.length, 1); + assert.strictEqual(reso.rows[0].n, 5); +}); + +test('attention_mode and pass axes', () => { + const items = [item({}), item({ attention_mode: 'sage' }), item({ preview: true })]; + const res = A.computeAnalytics(items, {}); + assert.deepStrictEqual(axis(res, 'attention_mode').rows.map(r => r.key).sort(), ['default', 'sage']); + assert.strictEqual(row(res, 'attention_mode', 'default').n, 2); + assert.deepStrictEqual(axis(res, 'pass').rows.map(r => r.key).sort(), ['Full', 'Preview']); +}); + +test('workflow param axes, constants and the distinct cap', () => { + const items = many(70, i => ({ wf_params: { 'KSampler.cfg': i % 2 ? 4 : 5, 'Node.fixed': 'x', 'Node.seed': i } })); + const res = A.computeAnalytics(items, {}); + const cfg = axis(res, 'wf:KSampler.cfg'); + assert.ok(cfg && cfg.kind === 'wf' && cfg.rows.length === 2 && !cfg.constant); + assert.strictEqual(axis(res, 'wf:Node.fixed').constant, true); + assert.strictEqual(axis(res, 'wf:Node.seed'), undefined); + assert.deepStrictEqual(res.skippedAxes, [{ id: 'wf:Node.seed', distinct: 70 }]); +}); + +test('constant builtin axes are flagged', () => { + const res = A.computeAnalytics(many(4, i => ({ sampler: i % 2 ? 'a' : 'b' })), {}); + assert.strictEqual(axis(res, 'model').constant, true); + assert.strictEqual(axis(res, 'sampler').constant, false); +}); + +test('sortAnalyticsRows: enough first, metric desc nulls last, label asc', () => { + const mk = (label, n, favRate, sharp) => ({ key: label, label, n, favRate, rejRate: 0, durMean: null, ratingMean: null, enough: n >= 3, + metrics: { sharpness: sharp === null ? null : { mean: sharp, n } } }); + const rows = [mk('small', 1, 1, 999), mk('big', 4, 0.75, 10), mk('mid', 3, 0.5, 30), mk('nomet', 3, 0.9, null)]; + assert.deepStrictEqual(A.sortAnalyticsRows(rows, 'favRate', -1).map(r => r.label), ['nomet', 'big', 'mid', 'small']); + assert.deepStrictEqual(A.sortAnalyticsRows(rows, 'metric:sharpness', -1).map(r => r.label), ['mid', 'big', 'nomet', 'small']); + assert.deepStrictEqual(A.sortAnalyticsRows(rows, 'metric:sharpness', 1).map(r => r.label), ['big', 'mid', 'nomet', 'small']); + assert.deepStrictEqual(A.sortAnalyticsRows(rows, 'label', 1).map(r => r.label), ['big', 'mid', 'nomet', 'small']); + assert.strictEqual(rows[0].label, 'small'); // input not mutated +}); + +test('analyticsToMarkdown: exact output for a small input', () => { + const items = [ + item({ sampler: 'euler', favorited: true, duration: 2, metrics: { sharpness: 100.123 } }), + item({ sampler: 'euler', duration: 4, metrics: { sharpness: 200 } }), + item({ sampler: 'dpm|pp', rejected: true, duration: 3 }) + ]; + const res = A.computeAnalytics(items, { metricOrder: ['sharpness'] }); + const md = A.analyticsToMarkdown(res, { title: 'demo', scopeLabel: 'whole session' }); + const expected = [ + '## USCG stats: demo (whole session, 3 items)', + "_Fav % / Rej % = share of the group's items; Avg s = mean duration; metric columns = mean over items that have it; rows with n < 3 are marked *._", + '', + '### Sampler', + '| Value | n | Fav % | Rej % | Avg s | Sharpness |', + '|---|---:|---:|---:|---:|---:|', + '| euler | 2* | 50.0 | 0.0 | 3.00 | 150 |', + '| dpm\\|pp | 1* | 0.0 | 100.0 | 3.00 | |', + '', + 'Same for every item: Model = m, LoRA = None, Scheduler = simple, Steps = 20, CFG = 7, Denoise = 1, Resolution = 1024x1024, Attention = default, Upscale = none, Pass = Full, Prompt = a cat', + '' + ].join('\n'); + assert.strictEqual(md, expected); + assert.ok(!/Rating/.test(md)); + const withR = A.computeAnalytics(items, { ratings: new Map([[items[0].id, { rating: 1523.4, votes: 5 }]]) }); + const md2 = A.analyticsToMarkdown(withR, {}); + assert.ok(md2.includes('| Rating |')); + assert.ok(md2.includes('1523 (1)')); +}); + +test('one pass: favorited read once per item, and large input is fast', () => { + const reads = new Map(); + const items = many(200, {}).map(d => new Proxy(d, { + get(t, k) { if (k === 'favorited') reads.set(t.id, (reads.get(t.id) || 0) + 1); return t[k]; } + })); + A.computeAnalytics(items, {}); + assert.strictEqual(reads.size, 200); + for (const v of reads.values()) assert.strictEqual(v, 1); + + const big = many(5000, i => ({ + sampler: ['euler', 'dpmpp', 'lms'][i % 3], cfg: 3 + (i % 5), + metrics: { sharpness: i, entropy: 5, colorfulness: 20, contrast: 0.2, luma: 0.5, saturation: 0.4, sobel: 0.05 }, + wf_params: { 'A.x': i % 7, 'B.y': i % 4 } + })); + const t0 = Date.now(); + const res = A.computeAnalytics(big, {}); + assert.ok(Date.now() - t0 < 500, 'took ' + (Date.now() - t0) + ' ms'); + assert.strictEqual(res.total, 5000); +}); + +test('empty input does not throw', () => { + for (const input of [[], null, undefined]) { + const res = A.computeAnalytics(input, {}); + assert.strictEqual(res.total, 0); + assert.deepStrictEqual(res.axes, []); + assert.strictEqual(res.overall.durMean, null); + assert.ok(A.analyticsToMarkdown(res, {}).startsWith('## USCG stats')); + } +}); diff --git a/tests/test_analytics_assembly.py b/tests/test_analytics_assembly.py new file mode 100644 index 0000000..eb0396e --- /dev/null +++ b/tests/test_analytics_assembly.py @@ -0,0 +1,35 @@ +"""The Stats panel files are wired into the assembled dashboard page in the right order.""" +import importlib.util +import os +import re + +_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + + +def _real_html_generator(): + spec = importlib.util.spec_from_file_location("uscg_real_html_generator_an", os.path.join(_ROOT, "html_generator.py")) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def test_page_has_panel_logic_in_order_and_header_button(): + page = _real_html_generator().get_html_template("ci", {"items": []}, "1") + assert 'id="analytics-open-btn"' in page + a = page.index("function computeAnalytics(") + b = page.index("function openAnalyticsPanel(") + assert a < b + assert page.index("function pipelineMatches(") < b + + +def test_js_load_order(): + src = open(os.path.join(_ROOT, "html_generator.py"), encoding="utf-8").read() + m = re.search(r"js_files\s*=\s*\[(.*?)\n\s*\]", src, re.S) + names = re.findall(r'"(logic_[a-z_]+\.js)"', m.group(1)) + assert names.index("logic_ratings.js") < names.index("logic_analytics.js") + assert names.index("logic_wfparams.js") < names.index("logic_analytics.js") + assert names.index("logic_analytics.js") < names.index("logic_ui.js") + assert names.index("logic_analytics_panel.js") > names.index("logic_compare.js") + assert names.index("logic_analytics_panel.js") < names.index("logic_standalone.js") + for n in names: + assert os.path.isfile(os.path.join(_ROOT, "resources", n)) diff --git a/tests/test_dist_metrics_partial.py b/tests/test_dist_metrics_partial.py new file mode 100644 index 0000000..281d3ba --- /dev/null +++ b/tests/test_dist_metrics_partial.py @@ -0,0 +1,36 @@ +"""_distributed_metrics_pass: a worker that omitted a builtin (older USCG, no sobel) gets just that key from the file.""" +import importlib +import os +import sys + +import numpy as np +import pytest +from PIL import Image + +_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, _ROOT) +_PKG = os.path.basename(_ROOT) + +import generation_orchestrator as go # preloaded by tests/conftest.py + +im = importlib.import_module(f"{_PKG}.image_metrics") + + +@pytest.fixture(autouse=True) +def _clean(): + im._stack.clear() + yield + im._stack.clear() + + +def test_missing_builtin_filled_from_file_without_touching_existing(tmp_path, monkeypatch): + arr = np.zeros((16, 16, 3), np.uint8) + arr[:, 8:] = 255 + Image.fromarray(arr).save(tmp_path / "w.png") + monkeypatch.setattr(im, "current_spec", lambda: {"builtins": ["sharpness", "sobel"], "adapters": []}) + items = [{"id": 5, "filename": "w.png", "metrics": {"sharpness": 2.0}}] + path_for = lambda item, d: str(tmp_path / item["filename"]) + out = go._distributed_metrics_pass({"items": items}, set(), str(tmp_path), path_for) + assert out == {"adapters": 0, "builtins_from_file": 1} + assert items[0]["metrics"]["sharpness"] == 2.0 + assert items[0]["metrics"]["sobel"] > 0 diff --git a/tests/test_image_metrics_sobel.py b/tests/test_image_metrics_sobel.py new file mode 100644 index 0000000..e9b5cd0 --- /dev/null +++ b/tests/test_image_metrics_sobel.py @@ -0,0 +1,88 @@ +"""Sobel edge metric (item 25): exact values on synthetic images, numpy reference and torch parity.""" +import importlib +import os + +import numpy as np +import pytest + +_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +_PKG = os.path.basename(_ROOT) +im = importlib.import_module(f"{_PKG}.image_metrics") + + +def _gray(plane): + return np.repeat(np.asarray(plane, dtype=np.float32)[..., None], 3, axis=-1) + + +def _step(lo=0.0, hi=1.0, h=64, w=64, col=32): + p = np.full((h, w), lo, dtype=np.float32) + p[:, col:] = hi + return _gray(p) + + +def _blur5(a): + k = np.ones(5) / 5 + out = a.copy() + for ax in (0, 1): + out = np.apply_along_axis(lambda v: np.convolve(np.pad(v, 2, mode="edge"), k, mode="valid"), ax, out) + return out + + +def test_step_edge_vertical_and_horizontal(): + a = _step() + assert im.compute_metrics_numpy(a, ["sobel"])["sobel"] == pytest.approx(2 / 62, abs=1e-4) + t = np.transpose(a, (1, 0, 2)).copy() + assert im.compute_metrics_numpy(t, ["sobel"])["sobel"] == pytest.approx(2 / 62, abs=1e-4) + + +def test_half_contrast_step(): + got = im.compute_metrics_numpy(_step(0.25, 0.75), ["sobel"])["sobel"] + assert got == pytest.approx(1 / 62, abs=1e-4) + + +def test_flat_is_zero_and_checkerboard_is_zero_but_sharp(): + flat = _gray(np.full((32, 32), 0.5)) + assert im.compute_metrics_numpy(flat, ["sobel"])["sobel"] == 0 + yy, xx = np.indices((32, 32)) + chk = _gray(((yy + xx) % 2).astype(np.float32)) + m = im.compute_metrics_numpy(chk, ["sobel", "sharpness"]) + assert m["sobel"] == 0 + assert m["sharpness"] > 1e6 + + +def test_noise_beats_blur_and_uint8_equals_float(): + rng = np.random.default_rng(0) + a = rng.random((64, 64, 3)).astype(np.float32) + assert im.compute_metrics_numpy(a, ["sobel"])["sobel"] > im.compute_metrics_numpy(_blur5(a), ["sobel"])["sobel"] + u8 = (a * 255).round().astype(np.uint8) + assert im.compute_metrics_numpy(u8, ["sobel"]) == im.compute_metrics_numpy(u8.astype(np.float32) / 255.0, ["sobel"]) + + +def test_too_small_has_no_sobel_key(): + assert "sobel" not in im.compute_metrics_numpy(np.zeros((2, 8, 3), dtype=np.float32), ["sobel", "luma"]) + assert "sobel" in im.compute_metrics_numpy(np.zeros((3, 3, 3), dtype=np.float32), ["sobel"]) + + +def test_keys_and_spec_order(): + a = _step() + assert set(im.compute_metrics_numpy(a, ["sobel"])) == {"sobel"} + assert im.normalize_spec({"enabled": True, "builtins": ["sobel", "luma"]})["builtins"] == ["luma", "sobel"] + + +def test_downscale_invariance(): + big = _step(h=64, w=2048, col=1024) + small = _step(h=32, w=1024, col=512) + assert im.compute_metrics_numpy(big, ["sobel"]) == im.compute_metrics_numpy(small, ["sobel"]) + + +def test_torch_parity(): + torch = pytest.importorskip("torch") + if not hasattr(torch, "conv2d"): + pytest.skip("torch is stubbed") + rng = np.random.default_rng(5) + cases = [_step()] + [rng.random(s).astype(np.float32) for s in ((96, 128, 3), (1500, 1100, 3))] + for a in cases: + got = im.compute_metrics_torch(torch.from_numpy(a)[None], ["sobel"])[0] + ref = im.compute_metrics_numpy(a, ["sobel"]) + assert set(got) == set(ref) == {"sobel"} + assert got["sobel"] == pytest.approx(ref["sobel"], rel=1e-4, abs=1e-4) diff --git a/tests/test_metrics_config.py b/tests/test_metrics_config.py index aa432d6..cac9b31 100644 --- a/tests/test_metrics_config.py +++ b/tests/test_metrics_config.py @@ -65,6 +65,6 @@ def test_metrics_info_route_shape(monkeypatch): monkeypatch.setattr(cbn.web, "json_response", lambda data, **kw: captured.setdefault("data", data), raising=False) asyncio.run(cbn.get_metrics_info(None)) data = captured["data"] - assert data["builtins"] == ["sharpness", "entropy", "colorfulness", "contrast", "luma", "saturation"] + assert data["builtins"] == ["sharpness", "entropy", "colorfulness", "contrast", "luma", "saturation", "sobel"] assert {a["name"] for a in data["adapters"]} >= {"aesthetic", "imagereward"} assert all({"name", "label", "installed", "enabled", "pack_hint"} <= set(a) for a in data["adapters"]) diff --git a/tests/test_metrics_runner_fill_missing.py b/tests/test_metrics_runner_fill_missing.py new file mode 100644 index 0000000..1151cc8 --- /dev/null +++ b/tests/test_metrics_runner_fill_missing.py @@ -0,0 +1,101 @@ +"""metrics_runner fill_missing: backfill only the metric keys an item lacks (item 25, Sobel on older sessions).""" +import importlib +import importlib.util +import json +import os +import sys +import time + +import numpy as np +import pytest +from PIL import Image + +_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +_PKG = os.path.basename(_ROOT) +mr = importlib.import_module(f"{_PKG}.metrics_runner") +_mu_spec = importlib.util.spec_from_file_location(f"{_PKG}.manifest_utils_real_fm", os.path.join(_ROOT, "manifest_utils.py")) +_mu = importlib.util.module_from_spec(_mu_spec) +_mu.__package__ = _PKG +_mu_spec.loader.exec_module(_mu) +im = importlib.import_module(f"{_PKG}.image_metrics") + +OLD_SIX = ("sharpness", "entropy", "colorfulness", "contrast", "luma", "saturation") + + +@pytest.fixture +def session(tmp_path, monkeypatch): + out = tmp_path / "out" + sdir = out / "benchmarks" / "s1" + (sdir / "images").mkdir(parents=True) + monkeypatch.setattr(sys.modules["folder_paths"], "get_output_directory", lambda: str(out), raising=False) + for n, v in (("a.webp", 10), ("b.webp", 200), ("c.png", 90)): + Image.fromarray(np.full((30, 40, 3), v, np.uint8)).save(sdir / "images" / n) + old = {k: 1.5 for k in OLD_SIX} + old["aesthetic"] = 6.5 + items = [ + {"id": 1, "file": "/view?filename=a.webp&type=output&subfolder=benchmarks/s1/images"}, + {"id": 2, "file": "/view?filename=b.webp&type=output&subfolder=benchmarks/s1/images", "metrics": {"luma": 9.0}}, + {"id": 3, "file": "/view?filename=c.png&type=output&subfolder=benchmarks/s1/images", "metrics": old}, + ] + mp = sdir / "manifest.json" + mp.write_text(json.dumps({"session_name": "s1", "meta": {}, "items": items}), encoding="utf-8") + monkeypatch.setattr(mr, "manifest_utils", _mu) + mr._jobs.clear() + return mp + + +def _wait(job_id): + for _ in range(200): + st = mr.get_metrics_status(job_id) + if st["status"] != "running": + return st + time.sleep(0.02) + raise AssertionError("job did not finish") + + +def _items(mp): + return {i["id"]: i for i in json.loads(mp.read_text(encoding="utf-8"))["items"]} + + +def test_fill_missing_keeps_existing_and_adds_the_rest(session): + job_id, err = mr.start_metrics_job("s1", fill_missing=True) + assert err is None + st = _wait(job_id) + assert st["status"] == "complete" and st["total"] == 3 + it = _items(session) + assert it[2]["metrics"]["luma"] == 9.0 + assert set(it[2]["metrics"]) == set(im.BUILTIN_METRICS) + assert set(it[1]["metrics"]) == set(im.BUILTIN_METRICS) + + +def test_fill_missing_adds_only_sobel_to_old_items(session): + job_id, _ = mr.start_metrics_job("s1", fill_missing=True) + _wait(job_id) + m = _items(session)[3]["metrics"] + assert {k: m[k] for k in OLD_SIX} == {k: 1.5 for k in OLD_SIX} + assert m["aesthetic"] == 6.5 and "sobel" in m + + +def test_second_fill_missing_run_has_nothing_to_do(session): + job_id, _ = mr.start_metrics_job("s1", fill_missing=True) + _wait(job_id) + jid, err = mr.start_metrics_job("s1", fill_missing=True) + assert jid is None and err.startswith("Nothing to compute") + + +def test_default_mode_still_skips_items_with_metrics(session): + job_id, _ = mr.start_metrics_job("s1") + assert _wait(job_id)["total"] == 1 + + +def test_flush_merge_never_overwrites_disk_keys_and_keeps_edits(session): + data = json.loads(session.read_text(encoding="utf-8")) + for i in data["items"]: + if i["id"] == 3: + i["favorited"] = True + i["metrics"]["luma"] = 0.42 + session.write_text(json.dumps(data), encoding="utf-8") + mr._flush(str(session), {3: {"luma": 0.99, "sobel": 0.1}}, False, merge=True) + m = _items(session)[3] + assert m["favorited"] is True + assert m["metrics"]["luma"] == 0.42 and m["metrics"]["sobel"] == 0.1 and m["metrics"]["aesthetic"] == 6.5 diff --git a/web/conf_builder/conf-builder-config-management.js b/web/conf_builder/conf-builder-config-management.js index c746e96..376d62e 100644 --- a/web/conf_builder/conf-builder-config-management.js +++ b/web/conf_builder/conf-builder-config-management.js @@ -6983,7 +6983,7 @@ export function renderRunSettingsSection(node, container) { if (!node.state.metrics || typeof node.state.metrics !== "object") node.state.metrics = { enabled: false, builtins: {}, adapters: {} }; if (!node.state.metrics.builtins) node.state.metrics.builtins = {}; if (!node.state.metrics.adapters) node.state.metrics.adapters = {}; - ["sharpness", "entropy", "colorfulness", "contrast", "luma", "saturation"].forEach(k => { + ["sharpness", "entropy", "colorfulness", "contrast", "luma", "saturation", "sobel"].forEach(k => { if (node.state.metrics.builtins[k] === undefined) node.state.metrics.builtins[k] = true; }); @@ -7253,6 +7253,7 @@ export function renderRunSettingsSection(node, container) { ["contrast", "Contrast", "RMS contrast: standard deviation of luma (0-0.5)."], ["luma", "Luma", "Mean brightness (0-1)."], ["saturation", "Saturation", "Mean HSV saturation (0-1)."], + ["sobel", "Sobel edges", "Mean Sobel gradient magnitude of luma (0-1; a full black-to-white edge reads 1). Higher = more and stronger edges; ignores pixel-level noise better than Sharpness."], ]; const wrap = document.createElement("div"); wrap.style.cssText = "margin-top: 10px;"; @@ -7263,7 +7264,7 @@ export function renderRunSettingsSection(node, container) { label.style.cssText = "font-size: 12px; color: #ccc; white-space: nowrap;"; const help = document.createElement("span"); help.textContent = "?"; - help.title = "Stores sharpness, entropy, colorfulness, contrast, luma and saturation for every saved image (base images and upscales; not videos) in manifest.json, computed on the decoded image in about 1-5 ms each. Sort and filter by them in the Dashboard. They measure image signal, not taste: noise raises sharpness and entropy."; + help.title = "Stores sharpness, entropy, colorfulness, contrast, luma, saturation and Sobel edges for every saved image (base images and upscales; not videos) in manifest.json, computed on the decoded image in about 1-5 ms each. Sort and filter by them in the Dashboard. They measure image signal, not taste: noise raises sharpness and entropy."; help.style.cssText = "display: inline-flex; align-items: center; justify-content: center; width: 16px; height: 16px; border-radius: 50%; background: #2a2a2a; color: #8af; font-size: 10px; font-weight: bold; cursor: help; user-select: none;"; const cb = document.createElement("input"); cb.type = "checkbox"; diff --git a/web/conf_builder/conf-builder-main.js b/web/conf_builder/conf-builder-main.js index a4d4769..37089cf 100644 --- a/web/conf_builder/conf-builder-main.js +++ b/web/conf_builder/conf-builder-main.js @@ -217,7 +217,7 @@ app.registerExtension({ clear_vram: false }, image_format: "webp", - metrics: { enabled: false, builtins: { sharpness: true, entropy: true, colorfulness: true, contrast: true, luma: true, saturation: true }, adapters: {} }, + metrics: { enabled: false, builtins: { sharpness: true, entropy: true, colorfulness: true, contrast: true, luma: true, saturation: true, sobel: true }, adapters: {} }, overwrite_existing: false, flush_batch_every: 1, lora_triggerwords_mode: "None", @@ -351,7 +351,7 @@ app.registerExtension({ if (!this.state.metrics || typeof this.state.metrics !== "object") this.state.metrics = { enabled: false, builtins: {}, adapters: {} }; if (!this.state.metrics.builtins) this.state.metrics.builtins = {}; if (!this.state.metrics.adapters) this.state.metrics.adapters = {}; - ["sharpness", "entropy", "colorfulness", "contrast", "luma", "saturation"].forEach(k => { + ["sharpness", "entropy", "colorfulness", "contrast", "luma", "saturation", "sobel"].forEach(k => { if (this.state.metrics.builtins[k] === undefined) this.state.metrics.builtins[k] = true; });