nodes/aszc-dev-ComfyUI-CoreMLSuite
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat!: modernize toolchain and replace apple/ml-stable-diffusion with native diffusers conversion (#58)

Browse Source 
* feat(deps): support ComfyUI's numpy 2 toolchain; make conversion optional

The runtime package now installs and runs under numpy 2 / coremltools 9 /
torch 2.7 — matching current ComfyUI — without apple/ml-stable-diffusion.

- Drop the heavy converter stack (ml-stable-diffusion, diffusers, peft,
  omegaconf, overrides, transformers) from runtime dependencies; require
  numpy>=2.
- Vendor the runtime pieces: a slim CoreMLModel wrapper around coremltools
  and the attention-implementation constants.
- Lazy-import the converters; the Convert nodes raise a clear error when the
  legacy conversion dependencies are absent. Loading and sampling existing
  Core ML models no longer needs them.
- CI: Tier 0 tracks the numpy 2 / torch 2.7 toolchain; drop the Tier 2
  golden-image lane (it converts at runtime, which now requires the legacy
  stack) and its fixtures.

* feat(conversion): replace apple/ml-stable-diffusion with native diffusers path

Reimplement Core ML UNet conversion on top of diffusers instead of the
apple/ml-stable-diffusion git dependency, so the full suite (including
conversion) installs through ComfyUI Manager without extras on the NumPy 2
toolchain.

- Add coreml_suite/conversion package: split-einsum attention processors,
  a conv2d output-shape helper, Transformer2D trace patches, and a UNet
  input-adapter wrapper preserving the historical Core ML I/O contract.
- Drop python_coreml_stable_diffusion and overrides; route SD15, SDXL,
  SDXL refiner, and LCM conversion through diffusers UNet2DConditionModel.
- Declare diffusers, peft, omegaconf, and transformers as runtime deps.
- Add characterization tests asserting split-einsum matches reference
  attention math; extend the synthetic-UNet smoke test for the wrapper.
- Bump to 1.1.0 and set requires-comfyui to a semver constraint (>=0.3.27)
  so the Comfy Registry publish succeeds.

* refactor(conversion)!: native diffusers context layout; drop legacy fallbacks

Address PR review feedback:

- Drop the legacy converter ImportError fallbacks and LEGACY_CONVERTER_MODULES
  guards in nodes.py and lcm/nodes.py. Conversion dependencies are mandatory in
  pyproject, so the indirection is dead code.
- Tier 0 CI resolves its toolchain from pyproject via uv (uv sync + uv run)
  instead of hand-pinned pip installs, removing duplicated version maintenance.
- Document the conversion lineage: credit apple/ml-stable-diffusion as the
  origin, note the implementation has diverged to a native diffusers path, and
  state the intent to iterate independently. Fix stale README links that pointed
  users to apple/ml-stable-diffusion for conversion.
- Drop the unused `sources` argument from CoreMLModel.

BREAKING CHANGE: the converted Core ML UNet now takes encoder_hidden_states in
the native diffusers layout (batch, tokens, hidden) instead of
(batch, hidden, 1, tokens). This removes the boundary transposes in
CoreMLUNetWrapper and CoreMLInputs. Core ML models converted with earlier
versions are incompatible and must be re-converted. Bump to 2.0.0.

* test: widen split-einsum allclose tolerance for cross-platform float drift

The split-einsum attention reorders float32 reductions relative to the
reference, so equality holds only up to rounding. The default allclose atol
(1e-8) is too tight on Linux x86 BLAS and failed Tier 0 CI; use atol=1e-6 to
match the existing chunked-path characterization test.

* ci: run macOS smoke tier on the self-hosted Apple Silicon runner

GitHub-hosted macOS carries a 10x minute multiplier and exhausts the included
Actions minutes too quickly. Move the Tier 1 smoke job onto the self-hosted
Apple Silicon runner ([self-hosted, macOS, ARM64, coreml]) so macOS coverage no
longer consumes hosted minutes. Tier 0 stays on hosted ubuntu (1x).

* ci: fix uv setup for both tiers

astral-sh/setup-uv@v3 was retagged and its old commit garbage-collected, so
codeload 404s when Actions resolves the stale SHA. Bump Tier 0 (ubuntu) to
setup-uv@v7, and drop the action entirely from Tier 1 since the self-hosted
runner already provides uv.

* test(ci): restore golden-image correctness gate on the self-hosted runner

The Tier 2 end-to-end correctness check (real SD1.5 -> Core ML -> image,
gated on SHA/PSNR vs a golden) was dropped during the modernization. With the
breaking 3D-context change, the synthetic smoke and shape/attention
characterization tests no longer cover real-model conversion correctness.

Restore tier2.yml (on the [self-hosted, macOS, ARM64, coreml] runner shared
with Tier 1), the golden-image test, and the e2e workflow. Adapt the pinned
ComfyUI resolution to the requires-comfyui semver tag (vX.Y.Z) instead of a
commit SHA, and re-register the m2 marker. The golden is intentionally not
committed: the first self-hosted run regenerates it under the new 3D contract
and fails for review, per the test's documented bootstrap.

* test(ci): add golden image for SD1.5 seed 42 under the 3D-context contract

Generated by the first Tier 2 self-hosted run after the native diffusers
conversion change. The decoded image is a coherent SD1.5 generation, confirming
the (batch, tokens, hidden) Core ML contract produces correct output
end-to-end. Subsequent runs gate on this golden (SHA-strict, PSNR fallback).

* test(ci): force fresh conversion in Tier 2; drop stale golden

The converter skips conversion when a same-named model already exists, keyed on
conversion parameters but not the conversion code/toolchain. The self-hosted
runner held a pre-existing v1-5 .mlmodelc (4D-context, old toolchain), so the
Tier 2 runs reused it (~8-30s) instead of converting — the gate validated a
stale model, not the new native diffusers path.

Purge the cached Core ML UNets before running so every Tier 2 run does a real
convert -> compile -> sample. Drop the golden generated from the stale cache;
the next run regenerates it from a genuine 3D-contract conversion and fails for
review.

* test(ci): add golden image from a genuine 3D-contract conversion

Regenerated by a Tier 2 run with the model cache purged, so the converter
actually ran (62s, not a cache hit). The fresh model exposes the new 3D
encoder_hidden_states input [1, 77, 768], and its decoded SD1.5 seed-42 image
is byte-identical to the prior baseline — confirming the native diffusers
conversion is behavior-preserving end to end.
This commit is contained in:
aszc
2026-05-26 15:46:09 +02:00
committed by GitHub
parent 02b6e8ece3
commit 65a2de2fab
32 changed files with 1032 additions and 793 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pyproject.toml
+8 -46
View File
@@ -1,50 +1,35 @@
[project]
name = "comfyui-coremlsuite"
description = "This extension contains a set of custom nodes for ComfyUI that allow you to use Core ML models in your ComfyUI workflows."
version = "1.0.1"
version = "2.0.0"
license = "MIT"
requires-python = ">=3.12,<3.13"
packages = [{ include = "coreml_suite" }]
dependencies = [
# Python 3.12, coremltools 9, torch 2.7.
# numpy stays in the 1.24..1.x range — none of our modules need
# numpy 2, and coremltools + ml-stable-diffusion's SD UNet trace
# hit hard bugs under numpy 2 (`_cast` int(ndarray) strictness and
# `view` mixed-Var shape lists).
# torch 2.7 is the latest version coremltools 9's PyTorch frontend
# has been tested against.
"python-coreml-stable-diffusion @ git+https://github.com/apple/ml-stable-diffusion.git@e5d960c41a6a4ab200b8db379194127607b1c590",
"torch>=2.7,<2.8",
"coremltools>=9,<10",
"numpy>=1.24,<2",
"overrides",
"diffusers>=0.22",
"peft>=0.6.2",
"numpy>=2,<3",
"diffusers>=0.30",
"peft>=0.13",
"omegaconf>=2.3",
"transformers>=4.44",
]
[project.urls]
Repository = "https://github.com/aszc-dev/ComfyUI-CoreMLSuite"
# Used by Comfy Registry https://comfyregistry.org
[tool.comfy]
PublisherId = "aszc-dev"
DisplayName = "ComfyUI-CoreMLSuite"
Icon = ""
# Pinned to the ComfyUI commit this toolchain was validated against.
requires-comfyui = "==ab5413351eee61f3d7f10c74e75286df0058bb18"
requires-comfyui = ">=0.3.27"
[dependency-groups]
dev = [
"pillow>=12.2.0",
"psutil>=7.2.2",
"pytest>=9.0.3",
]
# ComfyUI runtime deps that aren't part of our package's runtime contract
# but are needed to spin up the ComfyUI server for Tier 2 integration tests.
# Kept in a uv group so `uv sync --group comfy` brings them in without
# polluting the published metadata (and without re-bumping our torch pin
# via `uv pip install -r ComfyUI/requirements.txt`, which would float to
# the latest torch and break the coremltools 9 compatibility ceiling).
comfy = [
"comfyui-frontend-package==1.14.6",
"torchvision",
@@ -61,34 +46,11 @@ comfy = [
"sentencepiece",
]
[tool.uv]
# ml-stable-diffusion's setup.py hard-pins numpy<1.24, diffusers==0.30.2
# and transformers==4.44.2, which blocks the modern torch / coremltools
# combo on Python 3.12. Override the four blocking pins; the .unet /
# .coreml_model symbols we actually import are stable across the bumped
# versions.
override-dependencies = [
"numpy>=1.24,<2",
"diffusers>=0.30",
"transformers>=4.44",
"huggingface-hub>=0.24",
]
[tool.pytest.ini_options]
# Tier markers gate which environment a test needs.
# - unit: framework-free pure-logic tests (Tier 0; run without ComfyUI on
# Linux).
# - m2: needs an Apple Silicon Mac with the Neural Engine (Tier 2),
# typically a self-hosted runner or local M-series box.
# - smoke: lightweight checks that need Apple Silicon + coremltools but no
# ANE/real model (Tier 1).
markers = [
"unit: framework-free unit test (Tier 0)",
"m2: requires Apple Silicon + Neural Engine (Tier 2)",
"smoke: macOS-ARM smoke test on a synthetic micro-model (Tier 1)",
"m2: requires Apple Silicon + Neural Engine (Tier 2)",
]
testpaths = ["tests"]
# importlib mode keeps pytest from importing the repo-root __init__.py
# (which is the ComfyUI custom-node entry and pulls in comfy + nodes).
# Without this Tier-0 leaks the entire ComfyUI runtime on collection.
addopts = ["--import-mode=importlib", "--confcutdir=tests"]