chore(phase6,bench): record quantization tradeoff matrix

Phase 6 reference numbers for SD1.5 1x512x512 SPLIT_EINSUM, captured against commit 0bbd8d8 on M2 Pro 32GB (macOS 26.1, coremltools 9.0, numpy 1.26.4, torch 2.7.1, Python 3.12.11) — same toolchain as Phase 5 baseline ef2a18c-rebased 1e5791d. Headline: - size shrinks 1641 MB -> 822 / 617 / 412 MB (1/2, 1/2.7, 1/4) - fwd-pass median drops 197 ms -> 187 / 183 / 180 ms (5-9% faster) - noise_pred PSNR vs unquantized: 53.5 / 40.2 / 27.5 dB The fwd-ms improvement is mostly weight-load bandwidth (smaller LUT reads). PSNR is on the raw UNet output at a fixed seed; final-image PSNR after 20 sampler steps is comfortably higher.
2026-05-25 01:31:30 +02:00
parent 0bbd8d8e0d
commit 3adb216763
2 changed files with 61 additions and 0 deletions
@@ -0,0 +1,49 @@
+{
+  "git_sha": "0bbd8d8",
+  "compute_unit": "CPU_AND_NE",
+  "repeats": 20,
+  "input_seed": 0,
+  "ckpt_stem": "v1-5-pruned-emaonly",
+  "rows": [
+    {
+      "nbits": "none",
+      "path": "/Users/aszczepanski/dev/ComfyUI/models/unet/v1-5-pruned-emaonly_1x512x512_se_unet.mlmodelc",
+      "size_bytes": 1720796055,
+      "load_time_s": 0.1997,
+      "fwd_ms_median": 196.651,
+      "fwd_ms_min": 196.588,
+      "psnr_db_vs_none": null,
+      "size_ratio_vs_none": 1.0
+    },
+    {
+      "nbits": "8",
+      "path": "/Users/aszczepanski/dev/ComfyUI/models/unet/v1-5-pruned-emaonly_1x512x512_se_q8_unet.mlmodelc",
+      "size_bytes": 861946375,
+      "load_time_s": 0.1475,
+      "fwd_ms_median": 185.223,
+      "fwd_ms_min": 183.373,
+      "psnr_db_vs_none": 53.54,
+      "size_ratio_vs_none": 0.501
+    },
+    {
+      "nbits": "6",
+      "path": "/Users/aszczepanski/dev/ComfyUI/models/unet/v1-5-pruned-emaonly_1x512x512_se_q6_unet.mlmodelc",
+      "size_bytes": 647068433,
+      "load_time_s": 0.1457,
+      "fwd_ms_median": 181.091,
+      "fwd_ms_min": 180.977,
+      "psnr_db_vs_none": 40.2,
+      "size_ratio_vs_none": 0.376
+    },
+    {
+      "nbits": "4",
+      "path": "/Users/aszczepanski/dev/ComfyUI/models/unet/v1-5-pruned-emaonly_1x512x512_se_q4_unet.mlmodelc",
+      "size_bytes": 432280915,
+      "load_time_s": 0.1427,
+      "fwd_ms_median": 178.953,
+      "fwd_ms_min": 178.753,
+      "psnr_db_vs_none": 27.47,
+      "size_ratio_vs_none": 0.251
+    }
+  ]
+}
@@ -0,0 +1,12 @@
+# Quantization tradeoff matrix — `0bbd8d8`
+
+- compute unit: CPU_AND_NE
+- repeats per variant: 20
+- ckpt: v1-5-pruned-emaonly, 1x512x512, attn=se
+
+| nbits | size (MB) | size vs none | load (s) | fwd median (ms) | fwd min (ms) | PSNR vs none (dB) | error |
+|---|---|---|---|---|---|---|---|
+| none | 1641.1 | 1.0 | 0.1997 | 196.651 | 196.588 | - |  |
+| 8 | 822.0 | 0.501 | 0.1475 | 185.223 | 183.373 | 53.54 |  |
+| 6 | 617.1 | 0.376 | 0.1457 | 181.091 | 180.977 | 40.2 |  |
+| 4 | 412.3 | 0.251 | 0.1427 | 178.953 | 178.753 | 27.47 |  |