Phase 6 of the modernization plan: add weight palettization to the
Core ML converter as an opt-in knob, so the SD1.5 / SDXL UNet can
ship at 1/2, 1/2.7 or 1/4 of its current size with ANE-friendly
inference.
CoreMLConverter (and the LCM converter) gains a `quantize_nbits`
dropdown: `none` (default — identical to pre-Phase-6 behavior and
filenames, so existing cached .mlpackages still resolve) / `8` / `6` /
`4`. The value is encoded as `_q<bits>` after the attn suffix, so the
unquantized model and the three palettized variants coexist on disk
under distinct cache keys.
Implementation
- core/naming.compose_out_name: accepts `quantize_nbits`, validates
against {none, 8, 6, 4}, appends `_q<bits>` (none = empty).
- converter.convert_unet: after ct.convert + before .save, runs
coremltools.optimize.coreml.palettize_weights with
OpPalettizerConfig(mode="kmeans", nbits=...) when the value is not
"none". Adds a `Palettization took Xs` log line.
- converter.convert / nodes.CoreMLConverter.convert: pipe the new arg
through; the ComfyUI node exposes it as a dropdown with default
"none" so existing workflows are unchanged at load time.
- bench/scripts/convert_sd15.py: QUANT_NBITS env knob; uses the
pure compose_out_name (replaces the inline string formatter).
Test infra
- tests/unit/test_characterization_out_name.py: 6 new tests pinning
the `_q<bits>` suffix contract, the "none" passthrough (backward
compat), the cn + lora + quant combination, and the invalid-value
ValueError. Total Tier 0 now at 94.
- Makefile gains `bench-quant` (runs the matrix script) and
`convert-quant` (converts q8, q6, q4 sequentially).
- bench/scripts/quant_matrix.py (new): loads each variant, runs
REPEATS forward passes with a fixed seed, then computes the
noise_pred PSNR of each quantized variant against the unquantized
baseline. Writes bench/results/quant_matrix_<sha>.{json,md}.
README
- New "Quantization (Phase 6, opt-in)" section: tradeoff table
measured on M2 Pro SD1.5 1x512x512 SPLIT_EINSUM (sizes 1641/822/
617/412 MB; fwd 197/187/183/180 ms; PSNR 53.5 / 40.2 / 27.5 dB),
plus per-chip/RAM recommendations.
Default-path safety
- "none" produces the same out_name as Phase 5 -> existing
v1-5-pruned-emaonly_1x512x512_se_unet.mlmodelc is still picked up
unchanged; the m2 golden image test continues to anchor.
Phase 5 reference numbers, captured against commit 1e5791d on macOS
26.1 with the bumped toolchain (torch 2.7.1, coremltools 9.0, numpy
1.26.4, Python 3.12.11).
- bench/env/baseline-1e5791d.txt: full uv-pip freeze + ComfyUI sha +
macOS + resolved ml-stable-diffusion git metadata.
- bench/env/pytest-unit-1e5791d.txt: pytest -m unit 88/88 passing
(Tier-0 purity gate confirms no comfy/coreml leak).
- bench/results/1e5791d.{json,md}: SD1.5 1x512x512 SPLIT_EINSUM UNet
forward latency on the Apple Neural Engine and CPU+GPU. Held within
noise of the Phase 1 baseline (ef2a18c.json) — bump is
performance-neutral.
Phase 5 of the modernization plan: the intentional tooling upgrade
against the Phase 1 baseline. numpy 2 stays out of scope (decoupled —
see docs/deps.md).
Pyproject pins
- requires-python: ">=3.11,<3.12" -> ">=3.12,<3.13"
- torch: ==2.0.1 -> >=2.7,<2.8 (latest the coremltools 9 PyTorch
frontend has been tested against)
- coremltools: ==8.2 -> >=9,<10
- numpy: <1.25 -> >=1.24,<2 (held below 2 — coremltools+numpy2 has
known SD UNet trace bugs in `_cast` and `view`; none of our modules
need numpy 2)
- ml-stable-diffusion SHA: unchanged at e5d960c4 (upstream main has
the same restrictive pins; no working alternative)
uv overrides
- override-dependencies relaxes the four hard pins ml-stable-diffusion
ships in setup.py: numpy<1.24, diffusers==0.30.2, transformers==4.44.2,
huggingface-hub==0.24.6. The .unet / .coreml_model symbols we
actually import (see docs/deps.md) are stable across the bumped
versions.
[dependency-groups] comfy
- New group with ComfyUI's runtime deps (einops, torchvision, torchsde,
comfyui-frontend-package, spandrel, ...). Replaces the Phase 1 / 4
`uv pip install -r ComfyUI/requirements.txt` dance that floated torch
to the latest version and broke the coremltools ceiling. `uv sync
--group comfy` is the new contract; the Makefile already invokes the
project venv directly.
Tier 2 golden re-anchored
- The toolchain bump is performance-neutral on SD1.5 (NE fwd median
delta +0.2%, GPU +0.6% — within run-to-run noise) but bit-changes
the Core ML UNet output (different MIL graph + kernel selection).
The Phase 2 golden PNG hashes to a different SHA256 now and lands
at ~29 dB PSNR against itself. Visually identical, just numerically
different.
- tests/m2/goldens/sd15_seed42.{png,sha256} re-captured against the
bumped toolchain.
- tests/m2/test_golden_image.py: GOLDEN_PSNR_MIN_DB lowered from 40
to 25 (typical post-toolchain-bump tolerance). Header docstring
updated to explain when to raise it back for refactor PRs.
docs/deps.md (new)
- ml-stable-diffusion compatibility decision (override vs vendor vs
fork), why numpy 2 was punted, Tier 2 PSNR threshold reasoning,
bench diff table, and explicit rollback instructions.
Local verification
- pytest -m unit -> 88/88 passed in 1.87s
- pytest -m smoke -> 1/1 passed in 2.59s
- pytest -m m2 -> 1/1 passed (after re-anchor)
- bench/run.py -> SD1.5 NE 197 ms / GPU 272 ms, perf-neutral vs
Phase 1 baseline (ef2a18c.json)
Phase 4 of the modernization plan: institutionalize the 3-tier strategy
so future changes are guarded automatically, and pin down the
self-hosted M2 path the maintainer's hardware needs.
Tier dispatch
- Makefile targets test-unit / test-smoke / test-m2 / bench (plus
ci-tier0 / ci-tier1 wrappers that echo env first). check-macos-arm
fails fast on non-Apple-Silicon hosts.
Tier 1 smoke
- tests/smoke/test_synthetic_unet.py: builds a TinyUNet (conv-in,
time/text projections, conv-out), traces it, ct.convert to
mlprogram + fp16 CPU_ONLY, loads back via CoreMLModel and asserts
expected_inputs + named output. Runs in ~2s; auto-skips on
non-Apple-Silicon. Catches coremltools / ml-stable-diffusion API
drift without needing a real SD checkpoint or the ANE.
GitHub Actions
- .github/workflows/tier0.yml: ubuntu-latest on every push/PR, ~10
min budget, minimal-deps install (torch==2.0.1, numpy<1.25, pytest)
-> pytest -m unit.
- .github/workflows/tier1.yml: macos-14 (M1) on push/PR; opt-in via
run-tier1 label on labeled PRs to spare external-doc PRs.
- .github/workflows/tier2.yml: self-hosted [macOS, ARM64, coreml] on
PR label run-m2 / nightly cron / workflow_dispatch. Starts ComfyUI
with --cpu-vae, runs pytest -m m2 + bench/run.py, uploads bench
results.
Integration coverage moved
- Removed tests/integration/test_basic_conversion_1_5.py: it required
an MPS reference image (broken on macOS 26 + torch 2.0.1, see
Phase 1 Gate) and a checkpoint the maintainer doesn't have on disk
(dreamshaper_8). The same coverage now lives in
tests/m2/test_golden_image.py: deterministic numerical pass/fail
(SHA256 + PSNR fallback) against a stored golden, Core ML pipeline
only. No more human eyeballing.
Docs
- docs/ci-m2.md: one-time runner registration steps, COMFY_DIR
persistence, baseline model pre-conversion, trigger semantics, what
to do when the runner is offline, and the migration note from
integration -> m2 golden.
Sanity check
- Temporarily set convert_to="BREAKAGE_CANARY_NOT_A_REAL_FORMAT" in
the smoke test; Tier 1 surfaced
NotImplementedError: Backend converter BREAKAGE_CANARY_NOT_A_REAL_FORMAT not implemented
immediately. Reverted.
Local verification
- make test-unit -> 88/88 passed in 2.09s
- make test-smoke -> 1/1 passed in 1.99s
Phase 3 of the modernization plan: move the framework-free math out of
the comfy-coupled modules so Tier-0 tests can run on plain Linux without
ComfyUI, coremltools, or python_coreml_stable_diffusion.
New pure-core package (no comfy / coreml / mps imports):
- coreml_suite.core.latents: chunk_batch, merge_chunks
- coreml_suite.core.controlnet: expand_inputs, no_control,
extract_residual_kwargs, chunk_control
- coreml_suite.core.inputs: CoreMLInputs (chunks + coreml_kwargs)
- coreml_suite.core.sdxl: is_sdxl / is_sdxl_base / is_sdxl_refiner,
build_sdxl_time_ids (base len 6, refiner len 5), build_sdxl_text_embeds,
sdxl_model_function_wrapper
- coreml_suite.core.naming: compose_out_name, lora_names_from_params
Thin adapters keep the public import paths:
- coreml_suite.latents / coreml_suite.controlnet: re-export from core
- coreml_suite.models: CoreMLModelWrapper, CoreMLModelWrapperLCM,
add_sdxl_model_options (now uses the pure builders from core.sdxl),
get_latent_image, get_model_patcher remain framework-coupled
- coreml_suite.nodes: CoreMLConverter.convert now delegates the out_name
composition to core.naming.compose_out_name
Test infra:
- tests/unit/* re-pointed at coreml_suite.core.*
- test_chunks.py dropped `from comfy.model_management import ...` and
the dead `model_config` fixture (Phase 1 left it broken; Phase 3
removes it entirely)
- test_characterization_sdxl_options now targets the pure builders
directly via inspect.getclosurevars on the wrapper closure
- test_characterization_out_name now calls compose_out_name without the
heavy CoreMLConverter monkey-patching that Phase 2 needed
- tests/unit/test_tier0_purity.py: new gate that fails if comfy /
coremltools / etc leak into sys.modules during a pure `-m unit` run
(skipped in mixed runs where m2 / integration legitimately import them)
- tests/__init__.py + top-level conftest.py + pyproject addopts
`--import-mode=importlib --confcutdir=tests` together stop pytest from
importing the repo-root `__init__.py` (the ComfyUI custom-node entry
pulls in comfy)
- tests/conftest.py adds tier-aware collect_ignore so `-m unit` skips
tests/m2 + tests/integration at collection time
Verification:
- `pytest -m unit tests/` → 88 passed in ~2s; deterministic across runs
- Tier-0 purity gate confirms no comfy/coreml/etc in sys.modules
- m2 golden image (Phase 2 anchor) still hashes identical → refactor
produced bit-for-bit unchanged output
- `git diff main -- __init__.py coreml_suite/nodes.py` shows zero churn
to NODE_CLASS_MAPPINGS keys or INPUT_TYPES field names (public
workflow contract intact)
Phase 2 of the modernization plan: lock the current behavior of the pure
math so the Phase 3 refactor cannot silently change it.
Unit characterization tests (Tier 0, 64 new):
- test_characterization_latents.py: chunk_batch / merge_chunks padding,
truncation, and identity contracts.
- test_characterization_controlnet.py: expand_inputs / no_control /
extract_residual_kwargs / chunk_control shape, dtype, and zero-fill
behavior, including the [None]*target contract.
- test_characterization_inputs.py: CoreMLInputs.chunks / coreml_kwargs
for SD1.5, SDXL base (time_ids len 6), SDXL refiner (time_ids len 5),
and LCM (timestep_cond).
- test_characterization_sdxl_options.py: add_sdxl_model_options time_ids
/ text_embeds assembly via a SimpleNamespace fake ModelPatcher and
inspect.getclosurevars on the returned model_function_wrapper.
- test_characterization_out_name.py: CoreMLConverter out_name encoding
for attn_impl suffix, batch/size, ControlNet, LoRA (sorted), SDXL.
M2 [Tier 2] golden image anchor (1 new):
- test_golden_image.py: posts the SD1.5+CoreML workflow to a local
ComfyUI server (auto-skips if unreachable), asserts SHA256 of the
generated PNG against tests/m2/goldens/sd15_seed42.sha256; falls back
to PSNR >= 40 dB if the hash drifts.
Test infra:
- pyproject.toml [tool.pytest.ini_options]: unit / m2 / smoke markers,
testpaths=tests; rootdir is now this package (was ComfyUI's pytest.ini).
- tests/conftest.py: bootstraps sys.path for comfy imports, auto-marks
tests by directory, and ignores the maintainer's WIP scaffolds
(test_experiments / test_unet_conversion / standalone_test) so they
don't break collection.
All 85 collected tests pass; two consecutive runs produced identical
results (run1: 3.55s, run2: 3.40s).
Phase 1 reference numbers, captured against commit ef2a18c on macOS 26.1
with the pinned toolchain (torch 2.0.1, coremltools 8.2, numpy 1.23.5,
python_coreml_stable_diffusion@e5d960c4).
- bench/env/baseline-ef2a18c.txt: full pip freeze + ComfyUI sha + macOS +
resolved ml-stable-diffusion git metadata.
- bench/env/pytest-unit-ef2a18c.txt: pytest tests/unit (test_chunks +
test_controlnet) 20/20 passing.
- bench/results/ef2a18c.{json,md}: SD1.5 1x512x512 SPLIT_EINSUM UNet
forward latency on the Apple Neural Engine and CPU+GPU. NE median 197 ms
/ GPU median 270 ms; a second run reproduced both within 0.3% (noise).
- bench/results/smoke/ef2a18c/: end-to-end Core ML image (E2E-1.5-CoreML
workflow, seed=42) saved by smoke_image.py. The MPS reference branch of
the original workflow is omitted because torch 2.0.1's MPS backend on
macOS 26.1 trips a BFloat16 conversion error in VAEDecode and an
mps.add element-type mismatch in KSampler — both go away with newer
torch and are tracked for the Phase 5 toolchain bump. The server was
started with --cpu-vae to route the VAE through CPU; this is a runtime
flag, not a pin change.
Phase 1 of the modernization plan: freeze the currently-working environment
so later refactors have a measured reference point.
- Pin python-coreml-stable-diffusion to commit e5d960c4 (the one already
installed in the maintainer's apple_env), plus torch==2.0.1, coremltools==8.2
and numpy<1.25 to match the only env that loads ComfyUI successfully
(Comfy's checkpoint-safe-loading branch in utils.py is gated on torch>=2.4,
so newer torch + numpy 1.23 breaks at import).
- Mirror the same pins in requirements.txt and commit uv.lock for
reproducible installs.
- Add requires-comfyui pinning ComfyUI to ab541335 (the validated commit).
- Fix tests/unit/test_chunks.py fixture: get_model_config() now takes a
ModelVersion argument; pass ModelVersion.SD15 (the previously-broken test
was the only Phase 1 production-code change required).
- Add the Phase 1 baseline harness: bench/run.py (direct Core ML UNet
latency, deterministic), bench/scripts/convert_sd15.py (one-command
conversion bypassing the node graph), bench/scripts/smoke_image.py (POSTs
the existing e2e workflow to a local ComfyUI server and saves the Core ML
image), bench/env/capture.sh (env snapshot), bench/prompts.json (fixed
prompt set).
- Ignore apple_env/, comfy_env/, and bench/scripts/*.log.
Tests: 20/20 unit pass (test_chunks + test_controlnet).
- Added permissions for issue writing
- Updated action version to v1 for publish-node-action
- Added condition to run job only for 'aszc-dev' repository owner
aszc-dev