CthomasdesignandClaude Opus 4.8
1db2179abe
Groundwork for image-grounded conditioning (rung B). Vendors mlx-vlm's pure-MLX Qwen3-VL vision tower into krea2_engine/vision/ (self-contained: inlined SDPA helper, local VisionConfig) and adds torch-free image preprocessing (smart-resize + CLIP normalize + patch flatten). Both validated against the PyTorch reference (tools/test_vision_tower.py): - tower vs transformers Qwen3VLVisionModel: merged embeds + all 3 deepstack features match to max|Δ|~2e-4, cosine 1.0 (bf16 rounding). - preprocessing vs transformers image processor: max|Δ|~2e-7, grid exact. Loads the visual.* weights already present in the Krea-2-Turbo text encoder (315 tensors, 0 missing). Not yet wired into any node — the grounded-encode integration (M-RoPE, deepstack injection, 12-layer tap) is the next step. mlx-vlm is MIT; attribution added to NOTICE. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
89 lines
4.8 KiB
Plaintext
89 lines
4.8 KiB
Plaintext
Krea 2 is licensed under the Krea 2 Community License Agreement.
|
|
For more information, visit https://krea.ai/krea-2-licensing.
|
|
|
|
------------------------------------------------------------------------
|
|
This repository contains a MODIFIED DERIVATIVE of the Krea Model
|
|
(krea/Krea-2-Turbo). Modifications by avlp12 (ALISVOLATPROPRIIS):
|
|
|
|
- Re-implementation of the Krea 2 Turbo transformer, Qwen3-VL-4B text
|
|
encoder, and flow-matching sampler for Apple MLX.
|
|
- 8-bit (group-size 64) quantization of the transformer's 28
|
|
attention + SwiGLU blocks (sensitivity-graded; all other paths bf16).
|
|
|
|
This is an independent, unofficial port. It is NOT an official Krea
|
|
product and is NOT endorsed by or affiliated with Krea.ai, Inc.
|
|
|
|
A full copy of the Krea 2 Community License Agreement is included in the
|
|
LICENSE file in this repository. Your use of these weights is governed by
|
|
that agreement (including, e.g., the < $1,000,000 USD annual-revenue
|
|
threshold for Commercial Use and the content-filtering requirements).
|
|
------------------------------------------------------------------------
|
|
|
|
------------------------------------------------------------------------
|
|
FURTHER MODIFICATIONS in the krea2-mlx-studio fork, by Casey Thomas
|
|
(github.com/Cthomasdesign):
|
|
|
|
- LoRA support: a runtime low-rank adapter applied to the (quantized)
|
|
transformer (krea2/lora.py), Krea2Pipeline.set_loras(), CLI flags,
|
|
and web-UI controls. The key-mapper handles both the official Krea /
|
|
diffusers and the ai-toolkit / PEFT LoRA naming conventions.
|
|
|
|
This fork is independent and unofficial. It is NOT endorsed by or
|
|
affiliated with avlp12 (ALISVOLATPROPRIIS) or Krea.ai, Inc.
|
|
------------------------------------------------------------------------
|
|
|
|
------------------------------------------------------------------------
|
|
FURTHER MODIFICATIONS in this repository (ComfyUI-Krea2-MLX), by Casey
|
|
Thomas (github.com/Cthomasdesign):
|
|
|
|
- The `krea2` engine (transformer, text encoder, sampler, LoRA, safety
|
|
filter) is VENDORED into this repository as `krea2_engine/`, so this
|
|
node package has no external git dependency on krea2-mlx-studio.
|
|
The unused HF-Transformers-based text encoder variant
|
|
(text_encoder_hf.py) and the auto-download-by-precision helpers
|
|
(BUILDS/resolve_weights) were dropped as dead code not used by the
|
|
ComfyUI nodes, which always load an explicit local transformer file
|
|
chosen via the model loader node.
|
|
- Performance modifications to the vendored engine (v0.4.0):
|
|
native grouped-query attention in the fused SDPA kernels (no K/V
|
|
head repetition); step-invariant conditioning (text fusion, RoPE
|
|
tables, attention masks) hoisted out of the denoising loop; padded
|
|
text tokens trimmed before the DiT; dynamic-length text encoding
|
|
(pad to longest-in-batch with preserved absolute RoPE positions);
|
|
and a per-prompt embedding cache in the pipeline. Mathematically
|
|
faithful to the reference computation (validated); note that the
|
|
sequence-length changes alter bf16 kernel rounding order, so fixed
|
|
seeds render equal-quality but not bit-identical images vs earlier
|
|
versions. Set KREA2_EXACT_LEGACY=1 for the pre-0.4 exact behavior.
|
|
- ComfyUI node wrappers (nodes.py): model loading from
|
|
ComfyUI/models/krea2, LoRA stacking from ComfyUI/models/loras,
|
|
generation with progress/cancel support, and memory unloading.
|
|
- Image-conditioning (v0.5.0+): img2img and in-context edit paths,
|
|
and (v0.7.0+) image-grounded text conditioning via the Qwen3-VL
|
|
vision tower.
|
|
|
|
------------------------------------------------------------------------
|
|
THIRD-PARTY CODE (krea2_engine/vision/): the Qwen3-VL vision tower
|
|
implementation is vendored from mlx-vlm (github.com/Blaizzy/mlx-vlm),
|
|
Copyright © 2025 Prince Canuma, MIT License:
|
|
|
|
MIT License. Permission is hereby granted, free of charge, to any
|
|
person obtaining a copy of this software and associated documentation
|
|
files (the "Software"), to deal in the Software without restriction,
|
|
including without limitation the rights to use, copy, modify, merge,
|
|
publish, distribute, sublicense, and/or sell copies of the Software,
|
|
and to permit persons to whom the Software is furnished to do so,
|
|
subject to including the above copyright notice and this permission
|
|
notice in all copies or substantial portions of the Software. THE
|
|
SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND.
|
|
|
|
The vendored files (krea2_engine/vision/vision.py, config.py) are
|
|
adapted only to be self-contained (inlined a small SDPA helper, local
|
|
config import). The vision weights themselves are part of the Qwen3-VL
|
|
text encoder shipped with krea/Krea-2-Turbo.
|
|
------------------------------------------------------------------------
|
|
|
|
This integration is independent and unofficial. It is NOT endorsed by or
|
|
affiliated with avlp12 (ALISVOLATPROPRIIS) or Krea.ai, Inc.
|
|
------------------------------------------------------------------------
|