CthomasdesignandClaude Opus 4.8
1db2179abe
Groundwork for image-grounded conditioning (rung B). Vendors mlx-vlm's pure-MLX Qwen3-VL vision tower into krea2_engine/vision/ (self-contained: inlined SDPA helper, local VisionConfig) and adds torch-free image preprocessing (smart-resize + CLIP normalize + patch flatten). Both validated against the PyTorch reference (tools/test_vision_tower.py): - tower vs transformers Qwen3VLVisionModel: merged embeds + all 3 deepstack features match to max|Δ|~2e-4, cosine 1.0 (bf16 rounding). - preprocessing vs transformers image processor: max|Δ|~2e-7, grid exact. Loads the visual.* weights already present in the Krea-2-Turbo text encoder (315 tensors, 0 missing). Not yet wired into any node — the grounded-encode integration (M-RoPE, deepstack injection, 12-layer tap) is the next step. mlx-vlm is MIT; attribution added to NOTICE. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
38 lines
1.1 KiB
Python
38 lines
1.1 KiB
Python
"""VisionConfig for the vendored Qwen3-VL vision tower (from mlx-vlm, MIT — see ../../NOTICE)."""
|
|
|
|
import inspect
|
|
from dataclasses import dataclass, field
|
|
|
|
|
|
class BaseModelConfig:
|
|
@classmethod
|
|
def from_dict(cls, params):
|
|
if not params:
|
|
return cls()
|
|
return cls(**{k: v for k, v in params.items()
|
|
if k in inspect.signature(cls).parameters})
|
|
|
|
|
|
@dataclass
|
|
class VisionConfig(BaseModelConfig):
|
|
model_type: str = "qwen3_vl"
|
|
depth: int = 32
|
|
hidden_size: int = 1280
|
|
intermediate_size: int = 3420
|
|
out_hidden_size: int = 1536
|
|
num_heads: int = 16
|
|
image_size: int = 384
|
|
patch_size: int = 14
|
|
vocab_size: int = 32000
|
|
mlp_ratio: float = 4.0
|
|
in_channels: int = 3
|
|
layer_norm_eps: float = 1e-6
|
|
spatial_patch_size: int = 14
|
|
spatial_merge_size: int = 2
|
|
tokens_per_second: int = 2
|
|
temporal_patch_size: int = 2
|
|
num_position_embeddings: int = 2304
|
|
window_size: int = 112
|
|
fullatt_block_indexes: list[int] = field(default_factory=lambda: [7, 15, 23, 31])
|
|
deepstack_visual_indexes: list[int] = field(default_factory=list)
|