aadebuger-ComfyUI-MiMoASR/README.md

# ComfyUI-MiMoASR

ComfyUI custom nodes for [Xiaomi **MiMo-V2.5-ASR**](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-ASR)
— an 8B end-to-end ASR model with native support for Mandarin-English
code-switching, Chinese dialects (Wu, Cantonese, Hokkien, Sichuanese,
Henan, Northeastern Mandarin), song lyrics, knowledge-heavy speech, and
noisy / multi-speaker audio. Punctuation is generated natively.

| Node | Purpose |
|---|---|
| 🎯 **MiMo ASR Loader** | Loads `MiMo-V2.5-ASR` (8B) + `MiMo-Audio-Tokenizer`. Cached. |
| 🎯 **MiMo ASR Transcribe** | Runs `asr_sft` on an `AUDIO` input. Outputs `STRING`. |

---

## ⚠️ Read this before installing

This model is fussier than typical ComfyUI custom nodes. **Three** things
will burn your afternoon if you skip them. The node will install
successfully but fail at run time with confusing errors.

1. **`transformers` must be `==4.49.0`**. Newer versions changed the
   tied-weights API and MiMo's modeling code relies on the old one.
   This may conflict with other custom nodes that need `transformers >=4.57`
   (Sonic, Qwen3TTS, recent VibeVoice builds). Either accept that or run
   MiMo in a dedicated ComfyUI instance.
2. **`flash-attn` is required, not optional.** The audio tokenizer
   encoder calls `flash_attn_varlen_func` with **no SDPA fallback**.
   Plan to build from source unless your env exactly matches the
   upstream wheel (`cu12 + torch 2.6 + Python 3.12`).
3. **You need a local clone of the upstream repo on disk.**
   `MimoAudio` is not pip-installable. The loader injects it into
   `sys.path` at runtime.

Quick gut check — verify each row for your env. If any are off, jump
straight to [INSTALL_NOTES.md](./INSTALL_NOTES.md):

| | Required | Upstream pin | How to check |
|---|---|---|---|
| Python | 3.10+ | 3.12 | `python --version` |
| `torch` | any recent | 2.6.0 | `python -c "import torch; print(torch.__version__)"` |
| `transformers` | **4.49.0** | 4.49.0 | `python -c "import transformers; print(transformers.__version__)"` |
| `flash-attn` | **any working build** | 2.7.4.post1 | `python -c "from flash_attn import flash_attn_varlen_func"` |
| CUDA toolkit | ≥12.0 (for flash-attn build) | 12.x | `nvcc --version` |
| VRAM | 24 GB | — | `nvidia-smi` |

---

## Install (the proven order)

> **TL;DR** — if you're on Linux with a CUDA toolkit installed and uv-managed
> ComfyUI venv, you can run the one-shot script and skip to Step 6:
> ```bash
> COMFYUI_DIR=/path/to/ComfyUI ./scripts/install_server.sh
> ```
> It does steps 2–5 below idempotently. Otherwise, follow each step manually.

Each step has a verification at the end. Don't skip the verifications.

### 1. Clone this node pack

```bash
cd ComfyUI/custom_nodes
git clone https://github.com/aadebuger/ComfyUI-MiMoASR.git
```

### 2. Clone the upstream repo

```bash
# Pick any path with disk space; the loader will find it.
git clone https://github.com/XiaomiMiMo/MiMo-V2.5-ASR.git \
    /opt/repos/MiMo-V2.5-ASR

# Verify:
ls /opt/repos/MiMo-V2.5-ASR/src/mimo_audio/   # must list .py files
```

If `src/mimo_audio/` is missing, try `git submodule update --init --recursive`
inside the cloned repo.

### 3. Download the model weights

```bash
# Redirect HF cache if your root disk is small. MiMo-V2.5-ASR is ~16-32 GB
# on disk and HF stages a second copy before symlinking.
export HF_HOME=/big_disk/hf_cache
mkdir -p "$HF_HOME"

pip install -U huggingface-hub

hf download XiaomiMiMo/MiMo-V2.5-ASR \
    --local-dir /big_disk/models/MiMo-V2.5-ASR

hf download XiaomiMiMo/MiMo-Audio-Tokenizer \
    --local-dir /big_disk/models/MiMo-Audio-Tokenizer

# Verify:
ls -lh /big_disk/models/MiMo-V2.5-ASR/*.safetensors
ls -lh /big_disk/models/MiMo-Audio-Tokenizer/
```

### 4. Install Python deps (in ComfyUI's venv)

```bash
cd ComfyUI && source .venv/bin/activate

# Snapshot current state — you'll want this if the transformers downgrade
# breaks something else.
uv pip freeze > ~/pip_freeze_before_mimo_$(date +%Y%m%d).txt

# Pinned transformers + matching tokenizers
uv pip install "transformers==4.49.0" "tokenizers>=0.21,<0.22" --force-reinstall

# MiMo's runtime deps (from the upstream requirements.txt)
uv pip install "accelerate>=1.9.0" "librosa>=0.11.0" "scipy>=1.16.1" "zhon==2.1.1"

# Audio I/O for this node pack
uv pip install torchaudio

# CLI deps (skip if you only use the UI)
uv pip install requests websocket-client

# Verify:
python -c "import transformers; assert transformers.__version__ == '4.49.0'; print('OK')"
```

### 5. Install `flash-attn`

**Option A — your env matches the upstream wheel** (cu12 + torch 2.6 + py 3.12):

```bash
uv pip install flash-attn==2.7.4.post1
```

**Option B — build from source** (any newer torch / Python combo).
Budget 20–40 minutes on a 3090 host.

```bash
uv pip install ninja packaging

# Adjust to your toolchain
export CUDA_HOME=/usr/local/cuda-12.6
export TORCH_CUDA_ARCH_LIST="8.6"   # 8.6=3090, 8.9=4090, 9.0=H100
export MAX_JOBS=4                    # tune to RAM; lower if you OOM

# Background-build and tail the log so you can detach
nohup uv pip install flash-attn --no-build-isolation \
    > ~/flash_attn_build.log 2>&1 &
tail -f ~/flash_attn_build.log
```

Verify:

```bash
python -c "from flash_attn import flash_attn_varlen_func; print(flash_attn_varlen_func)"
# Must print: <function flash_attn_varlen_func at 0x...>
# NOT: None
```

If `flash_attn_varlen_func` is `None`, the install silently failed —
check `~/flash_attn_build.log` for compile errors. See INSTALL_NOTES.md
for the usual suspects (missing `nvcc`, OOM, mismatched
`TORCH_CUDA_ARCH_LIST`).

### 6. Launch ComfyUI with the env vars

```bash
export HF_HOME=/big_disk/hf_cache
export MIMO_ASR_REPO=/opt/repos/MiMo-V2.5-ASR

cd ComfyUI && source .venv/bin/activate
python main.py --listen 0.0.0.0 --port 8188
```

### 7. Verify everything is wired up

```bash
curl -s http://SERVER:8188/object_info | python3 -c "
import json, sys
d = json.load(sys.stdin)
for n in ('MiMoASRLoader', 'MiMoASRTranscribe', 'LoadAudio', 'PreviewAny'):
    print(f'  {n}: {\"✓\" if n in d else \"✗\"}')"
```

All four should show `✓`. If `PreviewAny` is missing, install
[rgthree-comfy](https://github.com/rgthree/rgthree-comfy) or swap it out
for `ShowText|pysssss` in your workflow.

---

## Usage

### Minimal workflow

```
LoadAudio  ──┐
             ├──►  🎯 MiMo ASR Transcribe  ──►  STRING
MiMo ASR ────┘
Loader
```

**Loader fields** (absolute paths, please):

- `model_path` — `/big_disk/models/MiMo-V2.5-ASR`
- `tokenizer_path` — `/big_disk/models/MiMo-Audio-Tokenizer`
- `repo_path` — leave blank if `MIMO_ASR_REPO` is set

**Transcribe fields**:

- `language` — `Auto` (recommended for code-switched speech),
  `Chinese`, or `English`

The first run loads the 8B model (30–60 s). Subsequent runs in the same
ComfyUI session are instant — the loader has a process-wide cache.

An API-format example workflow is at
[`example_workflows/mimo_asr_basic.json`](./example_workflows/mimo_asr_basic.json).

### CLI

A single-file CLI that uploads an audio file, drives the workflow over
ComfyUI's HTTP + WebSocket API, and prints the transcript:

```bash
uv run python cli/mimo_asr_cli.py speech.wav \
    --ip SERVER \
    --model-path /big_disk/models/MiMo-V2.5-ASR \
    --tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer \
    --repo-path /opt/repos/MiMo-V2.5-ASR
```

See [`cli/README.md`](./cli/README.md) for all flags and a wrapper-alias
recipe to avoid retyping the paths every time.

---

## VRAM

| | |
|---|---|
| Load (bf16) | ~17 GB |
| Inference peak | ~20–22 GB |
| Minimum GPU | 24 GB (3090, 4090, A100-40G) |
| Speed (30 s audio, 3090) | ~2–4 s after first load |

A single RTX 3090 is enough. Dual-GPU machines can pin MiMo to one card
via `CUDA_VISIBLE_DEVICES=1` before launching ComfyUI.

---

## Troubleshooting (the five most common)

For the full table see [INSTALL_NOTES.md](./INSTALL_NOTES.md).

| Symptom | Cause | Fix |
|---|---|---|
| `RuntimeError: MiMo ASR repo path is empty` | Step 2 missed or env var not set | Clone the repo, set `MIMO_ASR_REPO` |
| `RuntimeError: Did not find src/mimo_audio/` | Empty / partial clone | `git submodule update --init --recursive` inside the clone |
| `RuntimeError: ... model path does not exist` | Step 3 missed, or relative path in node field | Download the weights; use absolute paths |
| `AttributeError: 'MiMoAudioTokenizer' has no attribute 'all_tied_weights_keys'` | `transformers` too new | Downgrade to `4.49.0` |
| `TypeError: 'NoneType' object is not callable` (inside `flash_attn_varlen_func`) | `flash-attn` not installed | Build per Step 5 |

---

## Links

- Hugging Face: <https://huggingface.co/XiaomiMiMo/MiMo-V2.5-ASR>
- Audio tokenizer: <https://huggingface.co/XiaomiMiMo/MiMo-Audio-Tokenizer>
- Upstream repo: <https://github.com/XiaomiMiMo/MiMo-V2.5-ASR>
- Blog: <https://mimo.xiaomi.com/mimo-v2-5-asr>
- Live demo: <https://huggingface.co/spaces/XiaomiMiMo/MiMo-V2.5-ASR>

---

## License

Apache-2.0 (this node pack). MiMo-V2.5-ASR model weights are MIT-licensed
by Xiaomi; upstream `MimoAudio` source is Apache-2.0.

## Citation

```bibtex
@misc{coreteam2026mimov25asr,
      title  = {MiMo-V2.5-ASR: Robust Speech Recognition Across Languages,
                Dialects, and Complex Acoustic Scenarios},
      author = {LLM-Core-Team Xiaomi},
      year   = {2026},
      url    = {https://github.com/XiaomiMiMo/MiMo-V2.5-ASR}
}
```