Initial commit: ComfyUI-MiMoASR

This commit is contained in:
hua zhuang
2026-05-21 18:48:03 +08:00
commit a80d2b1910
17 changed files with 1919 additions and 0 deletions
+38
View File
@@ -0,0 +1,38 @@
# Python
__pycache__/
*.py[cod]
*.so
*.egg-info/
.eggs/
build/
dist/
# Virtualenvs
.venv/
venv/
env/
# Editor
.vscode/
.idea/
*.swp
# OS
.DS_Store
Thumbs.db
# Test / scratch
/out/
/tmp/
*.tar.gz
*.zip
# Models — never commit weights
/models/
*.safetensors
*.bin
*.pt
*.ckpt
# CLI temp
mimoasr_*.wav
+71
View File
@@ -0,0 +1,71 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/).
## [Unreleased]
## [0.1.2] - 2026-05-21
### Added
- `scripts/install_server.sh` — idempotent one-shot install for Linux
servers. Clones the upstream repo, downloads both checkpoints, pins
`transformers` to 4.49.0, installs MiMo's runtime deps, and builds
`flash-attn` from source if needed. Configurable via env vars
(`COMFYUI_DIR`, `MODELS_DIR`, `REPOS_DIR`, `GPU_ARCH`, `MAX_JOBS`).
### Changed
- **Documentation overhaul** based on real-world install pain.
- README now front-loads the three things that will break your install
(`transformers` pin, `flash-attn` requirement, repo clone) before
any step-by-step instructions.
- Each install step gets a copy-pasteable verification command.
- Added a "compatibility matrix" gut-check table.
- **INSTALL_NOTES.md** rewritten as an exhaustive error → fix reference,
covering every error message we observed in real installs:
- `MiMo ASR repo path is empty`
- `Did not find src/mimo_audio/`
- `model path does not exist`
- `MimoAudio.__init__() got an unexpected keyword argument 'tokenizer_path'`
- `AttributeError: ... no attribute 'all_tied_weights_keys'`
- `TypeError: 'NoneType' object is not callable` (flash-attn missing)
- `flash-attn` build failures (missing nvcc, OOM, wrong arch)
- Documented the three install strategies for handling the
`transformers==4.49.0` requirement when it conflicts with other
custom nodes (accept, dedicated ComfyUI instance, roll back).
- Detailed `flash-attn` source-build guide with the exact env vars
needed for an RTX 3090 host (`TORCH_CUDA_ARCH_LIST="8.6"`, `MAX_JOBS=4`).
### Notes
- No code changes since v0.1.1. This release is documentation-only.
## [0.1.1] - 2026-05-21
### Fixed
- `MiMoASRLoader` now calls `MimoAudio(model_path, tokenizer_path)` with
positional args. The HF model card shows kwargs, but the actual
upstream `__init__` signature doesn't accept `tokenizer_path=` as a
keyword — `run_mimo_asr.py` confirms positional usage. Symptom was
`TypeError: MimoAudio.__init__() got an unexpected keyword argument 'tokenizer_path'`.
## [0.1.0] - 2026-05-21
### Added
- `MiMoASRLoader` node — loads `MiMo-V2.5-ASR` + `MiMo-Audio-Tokenizer`
with process-wide caching keyed by `(model_path, tokenizer_path, repo_path)`.
- `MiMoASRTranscribe` node — runs `MimoAudio.asr_sft` on a ComfyUI
`AUDIO` input, writing it to a tempfile first; supports `Auto` /
`Chinese` / `English` language tags.
- `cli/mimo_asr_cli.py` — single-file CLI that uploads an audio file,
builds the API-format workflow, runs it over WebSocket, and prints +
optionally saves the transcript.
- Example workflow JSON at `example_workflows/mimo_asr_basic.json`.
- `INSTALL_NOTES.md` with the compatibility matrix and the
`transformers` / `flash-attn` workarounds.
[Unreleased]: https://github.com/aadebuger/ComfyUI-MiMoASR/compare/v0.1.2...HEAD
[0.1.2]: https://github.com/aadebuger/ComfyUI-MiMoASR/compare/v0.1.1...v0.1.2
[0.1.1]: https://github.com/aadebuger/ComfyUI-MiMoASR/compare/v0.1.0...v0.1.1
[0.1.0]: https://github.com/aadebuger/ComfyUI-MiMoASR/releases/tag/v0.1.0
+41
View File
@@ -0,0 +1,41 @@
# Contributing
Thanks for your interest! Small contributions are very welcome.
## Filing issues
Useful info to include:
- ComfyUI commit (`git -C ComfyUI rev-parse HEAD`)
- `torch.__version__`, `transformers.__version__`, GPU model
- The exact `MIMO_ASR_REPO` you're pointing at + its commit hash
- Full traceback (not just the final line)
## Pull requests
1. Fork + branch from `main`.
2. Keep PRs scoped — one feature or fix per PR.
3. Match the existing style (the loader / transcribe nodes show the
intended pattern: lazy heavy imports, clear runtime errors, a
`_MODEL_CACHE` for anything expensive).
4. If you add a new node, add it to **both** `__init__.py` mapping dicts
and update the README's node table.
5. Bump `CHANGELOG.md` under `[Unreleased]`.
## Local sanity check
```bash
# Lint
python -m py_compile nodes/*.py cli/*.py
# Optional: install into a ComfyUI checkout and reload
ln -s "$(pwd)" /path/to/ComfyUI/custom_nodes/ComfyUI-MiMoASR
```
## Scope
This pack is intentionally minimal — just the two nodes needed to use
MiMo-V2.5-ASR from ComfyUI. Anything that belongs upstream (model code,
sampler changes, etc.) should go to
[XiaomiMiMo/MiMo-V2.5-ASR](https://github.com/XiaomiMiMo/MiMo-V2.5-ASR)
instead.
+287
View File
@@ -0,0 +1,287 @@
# Install notes
The exhaustive companion to the README. Every error message we hit
during real-world install, with the fix and why it happens.
If you're hitting an issue and just want it fixed, jump to
**[Error → Fix table](#error--fix-table)**.
---
## Why this model is annoying to install
Three structural issues, all in the upstream repo:
1. **`MimoAudio` is not pip-installable.** The repo lays the code out as
`src/mimo_audio/mimo_audio.py` and uses imports of the form
`from src.mimo_audio.foo import bar`. That only resolves if the repo
*root* is on `sys.path`. The loader handles this via either the
`repo_path` node field or the `MIMO_ASR_REPO` env var.
2. **`MimoAudio.__init__` takes positional args**, not keywords. The HF
model card shows `MimoAudio(model_path="...", tokenizer_path="...")`
but the actual signature doesn't accept those kwargs.
`run_mimo_asr.py` calls it positionally. v0.1.1+ of this node pack
uses positional args to match.
3. **Hard dependencies on specific `transformers` and `flash-attn`
versions.** The audio tokenizer encoder calls
`flash_attn_varlen_func` with no SDPA fallback, and the modeling code
uses the pre-`all_tied_weights_keys` API from `transformers <=4.49`.
Both must be addressed before the model can run.
---
## Compatibility matrix
| Component | Upstream pin | What you probably have | What you need |
|---|---|---|---|
| Python | 3.12 | 3.10–3.13 | 3.10+ (3.13 works) |
| `torch` | 2.6.0 | 2.4–2.11 | any recent, don't downgrade |
| `transformers` | **4.49.0** | 4.50–5.x | **must downgrade to 4.49.0** |
| `tokenizers` | implicit | varies | `>=0.21,<0.22` |
| `flash-attn` | 2.7.4.post1 | none | any working build |
| CUDA toolkit | ≥12.0 | 12.4–12.6 | matches `nvcc --version` |
---
## Disk space (HF cache)
`MiMo-V2.5-ASR` is 8B params. The safetensors are stored in F32 on the
Hub (~32 GB), and HF stages a full copy in `HF_HOME` before symlinking
to your `--local-dir`. If your root partition is small:
```bash
# Add to ~/.bashrc so every shell sees it
export HF_HOME=/big_disk/hf_cache
mkdir -p "$HF_HOME"
```
Symptom of not doing this: `OSError: [Errno 28] No space left on device`
half-way through the download.
---
## `transformers` downgrade — what to expect
```bash
uv pip freeze > ~/pip_freeze_before_mimo_$(date +%Y%m%d).txt
uv pip install "transformers==4.49.0" "tokenizers>=0.21,<0.22" --force-reinstall
```
This will almost certainly affect other ComfyUI custom nodes.
| Custom node | Risk | Notes |
|---|---|---|
| Pure SDXL / Florence2 workflows | Low | Uses `diffusers`, mostly unaffected. |
| ComfyUI-VibeVoiceASR | Medium | Qwen2 backbone; may need adjustment. |
| ComfyUI-PaddleOCRVL | High | PaddleOCR-VL expects newer `transformers`. |
| Sonic / Qwen3TTS-class nodes | Very high | Often pinned to `transformers >=4.57`. |
| Anything with HF auto-class registration in 5.x | Very high | API renames. |
**Strategy A: accept the conflict.** Use one ComfyUI install for MiMo
ASR, restart with the other version when you need the other nodes.
**Strategy B: dedicated ComfyUI for MiMo.** Recommended if you use the
high-risk nodes daily.
```bash
# Clone the venv (cheap because Python deps share .so files via inodes)
cd /mnt/modelall
cp -r ComfyUI ComfyUI-MiMo
cd ComfyUI-MiMo && source .venv/bin/activate
uv pip install "transformers==4.49.0" "tokenizers>=0.21,<0.22" --force-reinstall
# Launch on a different port
python main.py --listen 0.0.0.0 --port 8189
```
Then point the CLI at the dedicated instance: `--ip SERVER --port 8189`.
**Strategy C: roll back if the downgrade breaks too much.**
```bash
uv pip install -r ~/pip_freeze_before_mimo_*.txt --force-reinstall
```
---
## Building `flash-attn` from source
The precompiled `flash-attn==2.7.4.post1` wheel only matches one exact
combo: **`cu12 + torch 2.6 + Python 3.12 + cxx11abi=False`**. Anything
else needs a source build.
### Prerequisites
```bash
# Tooling
uv pip install ninja packaging
# CUDA toolkit must be present and findable.
# Default Ubuntu location is /usr/local/cuda-XX.X — adjust to yours.
ls /usr/local/cuda-12.6/bin/nvcc # must exist
```
If `nvcc` is missing, install the matching toolkit:
```bash
sudo apt install cuda-toolkit-12-6 # adjust to your CUDA version
```
### Build
```bash
cd ComfyUI && source .venv/bin/activate
export CUDA_HOME=/usr/local/cuda-12.6
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
# GPU compute capability — set this to avoid building for archs you don't have
# 3090 = 8.6, A100 = 8.0, 4090 = 8.9, H100 = 9.0
export TORCH_CUDA_ARCH_LIST="8.6"
# Parallel jobs. Each nvcc job uses ~2 GB RAM. Tune to your hardware.
export MAX_JOBS=4
# Build (will take 20-40 min on a 3090 host)
nohup uv pip install flash-attn --no-build-isolation \
> ~/flash_attn_build.log 2>&1 &
tail -f ~/flash_attn_build.log
```
When it finishes, the log ends with something like
`Successfully installed flash-attn-X.Y.Z`. Verify:
```bash
python -c "from flash_attn import flash_attn_varlen_func; print(flash_attn_varlen_func)"
# Must print: <function flash_attn_varlen_func at 0x...>
```
### Common build failures
| Log message | Fix |
|---|---|
| `nvcc: command not found` | `export PATH=$CUDA_HOME/bin:$PATH` |
| `cannot find -lcuda` | `export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH` |
| `g++: fatal error: cuda_runtime.h: No such file` | Install matching `cuda-toolkit-XX-X` |
| `nvcc fatal: Unsupported gpu architecture 'compute_XX'` | Wrong `TORCH_CUDA_ARCH_LIST`; query yours with `python -c "import torch; print(torch.cuda.get_device_capability(0))"` (returns e.g. `(8, 6)` → use `"8.6"`) |
| `g++: internal compiler error: Killed (program cc1plus)` | Out of RAM. Reduce `MAX_JOBS` |
| Build hangs at "Building wheel" with no progress | Patience — first nvcc compile step has no output for ~5 min |
If the build finishes but
`python -c "from flash_attn import flash_attn_varlen_func"` returns
`None` instead of a function, the install silently failed. Re-check the
log.
---
## sys.path injection — how it actually works
`MimoAudio` imports look like:
```python
from src.mimo_audio.mimo_audio import MimoAudio
```
So the **parent** of `src/` must be on `sys.path` — i.e., the repo root,
the directory containing `run_mimo_asr.py`. Not `src/mimo_audio/`
itself. The loader does:
```python
sys.path.insert(0, str(repo_root))
```
Two ways to feed `repo_root` to the loader:
- **Per-node**: fill in the `repo_path` field on the `MiMoASRLoader` node.
- **Globally**: set `MIMO_ASR_REPO=/path/to/repo` before launching ComfyUI.
Recommended — your workflows then load with `repo_path` blank.
Symptoms of getting this wrong:
- `repo_path` blank and env unset → `RuntimeError: MiMo ASR repo path is empty.`
- Path points at `src/mimo_audio/` instead of the repo root →
`RuntimeError: Did not find src/mimo_audio/ under <path>.`
- Path points at a partial clone (no `src/`) → same error. Fix with
`git submodule update --init --recursive`.
---
## Error → Fix table
Reference card. Each row is a real error message you might see, with the
cause and the fix.
### Install / configuration errors
| Symptom | Cause | Fix |
|---|---|---|
| `RuntimeError: MiMo ASR repo path is empty.` | Neither `repo_path` node field nor `MIMO_ASR_REPO` env var is set. | Clone the upstream repo; set the env var or the node field. |
| `RuntimeError: Did not find src/mimo_audio/ under <path>.` | Wrong path, or empty / partial clone. | Verify `ls <path>/src/mimo_audio/`. Run `git submodule update --init --recursive` inside the clone if needed. |
| `RuntimeError: MiMo-V2.5-ASR model path does not exist: ...` | Step 3 skipped, or relative path used. | Download via `hf download`; pass an absolute path to the node. |
| `RuntimeError: MiMo-Audio-Tokenizer path does not exist: ...` | Same as above for the tokenizer. | `hf download XiaomiMiMo/MiMo-Audio-Tokenizer --local-dir ...` |
| `ModuleNotFoundError: No module named 'src'` (at first load) | `sys.path` injection happened but the path is wrong. | Verify `<repo_path>/src/mimo_audio/` exists. |
### Python dep errors
| Symptom | Cause | Fix |
|---|---|---|
| `AttributeError: 'MiMoAudioTokenizer' object has no attribute 'all_tied_weights_keys'. Did you mean: '_tied_weights_keys'?` | `transformers` >= 4.50 changed the tied-weights API. MiMo's modeling code uses the old name. | Downgrade: `uv pip install "transformers==4.49.0" "tokenizers>=0.21,<0.22" --force-reinstall` |
| `TypeError: MimoAudio.__init__() got an unexpected keyword argument 'tokenizer_path'` | You're on v0.1.0 of this node pack; it calls `MimoAudio(model_path=..., tokenizer_path=...)`. The actual signature is positional. | Upgrade to v0.1.1+ of this node pack. |
| `ImportError: cannot import name 'XXX' from 'transformers'` | Some other custom node forced `transformers` to a newer or older version after MiMo was working. | Re-run the step 4 install. Consider Strategy B (dedicated ComfyUI). |
### Runtime errors
| Symptom | Cause | Fix |
|---|---|---|
| `TypeError: 'NoneType' object is not callable` deep in `mimo_audio_tokenizer/modeling_audio_tokenizer.py` at `flash_attn_varlen_func(...)` | `flash-attn` not installed; upstream binds `flash_attn_varlen_func = None` on `ImportError` and uses it anyway. | Install per Step 5 of the README. There is no SDPA fallback. |
| `RuntimeError: CUDA out of memory` during load | Other models still resident in VRAM. | Restart ComfyUI, or use ComfyUI-Manager's "Free model and node cache". |
| `RuntimeError: FlashAttention only supports fp16 and bf16 dtype` | Loader's dtype detection failed; the model is in fp32. | Open an issue with your `torch.__version__` and the full traceback. |
| Empty transcript output (no error) | `PreviewAny` node missing from your server, but the workflow expects it. | Install rgthree-comfy, or use `ShowText|pysssss`. The CLI has a fallback that reads the asr node's output directly. |
| First run takes 30–60 s | Normal — loading 8B weights from disk. | Subsequent runs in the same ComfyUI session are instant thanks to `_MODEL_CACHE`. |
---
## End-to-end smoke test
After install, run this from the client side (your dev machine, not the
server):
```bash
# 0. ComfyUI reachable
curl -s -o /dev/null -w "HTTP %{http_code}\n" http://SERVER:8188/
# 1. Nodes registered
curl -s http://SERVER:8188/object_info | python3 -c "
import json, sys
d = json.load(sys.stdin)
for n in ('MiMoASRLoader', 'MiMoASRTranscribe', 'LoadAudio', 'PreviewAny'):
print(f' {n}: {\"OK\" if n in d else \"MISSING\"}')"
# 2. Dump a workflow without submitting (verifies argparse / paths)
uv run python cli/mimo_asr_cli.py /path/to/test.wav --ip SERVER \
--model-path /big_disk/models/MiMo-V2.5-ASR \
--tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer \
--repo-path /opt/repos/MiMo-V2.5-ASR \
--dump-workflow /tmp/wf.json --no-run
# 3. Inspect — load /tmp/wf.json in ComfyUI's UI via "Load" to verify connections
# 4. Real run
uv run python cli/mimo_asr_cli.py /path/to/test.wav --ip SERVER \
--model-path /big_disk/models/MiMo-V2.5-ASR \
--tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer \
--repo-path /opt/repos/MiMo-V2.5-ASR
```
On a 3090, expect:
- First run: 30–60 s (model load) + 2–4 s (inference)
- Subsequent runs in same session: 2–4 s
- VRAM after load: ~17 GB
- VRAM at peak: ~20–22 GB
If your numbers are wildly off, file an issue with `nvidia-smi` output
and your `torch` / `transformers` / `flash-attn` versions.
+201
View File
@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for describing the origin of the Work and
reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Support. While redistributing the Work or
Derivative Works thereof, You may choose to offer, and charge a fee
for, acceptance of support, warranty, indemnity, or other liability
obligations and/or rights consistent with this License. However,
in accepting such obligations, You may act only on Your own behalf
and on Your sole responsibility, not on behalf of any other
Contributor, and only if You agree to indemnify, defend, and hold
each Contributor harmless for any liability incurred by, or claims
asserted against, such Contributor by reason of your accepting any
such warranty or support.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2026 aadebuger
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied. See the License for the specific language governing
permissions and limitations under the License.
+290
View File
@@ -0,0 +1,290 @@
# ComfyUI-MiMoASR
ComfyUI custom nodes for [Xiaomi **MiMo-V2.5-ASR**](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-ASR)
— an 8B end-to-end ASR model with native support for Mandarin-English
code-switching, Chinese dialects (Wu, Cantonese, Hokkien, Sichuanese,
Henan, Northeastern Mandarin), song lyrics, knowledge-heavy speech, and
noisy / multi-speaker audio. Punctuation is generated natively.
| Node | Purpose |
|---|---|
| 🎯 **MiMo ASR Loader** | Loads `MiMo-V2.5-ASR` (8B) + `MiMo-Audio-Tokenizer`. Cached. |
| 🎯 **MiMo ASR Transcribe** | Runs `asr_sft` on an `AUDIO` input. Outputs `STRING`. |
---
## ⚠️ Read this before installing
This model is fussier than typical ComfyUI custom nodes. **Three** things
will burn your afternoon if you skip them. The node will install
successfully but fail at run time with confusing errors.
1. **`transformers` must be `==4.49.0`**. Newer versions changed the
tied-weights API and MiMo's modeling code relies on the old one.
This may conflict with other custom nodes that need `transformers >=4.57`
(Sonic, Qwen3TTS, recent VibeVoice builds). Either accept that or run
MiMo in a dedicated ComfyUI instance.
2. **`flash-attn` is required, not optional.** The audio tokenizer
encoder calls `flash_attn_varlen_func` with **no SDPA fallback**.
Plan to build from source unless your env exactly matches the
upstream wheel (`cu12 + torch 2.6 + Python 3.12`).
3. **You need a local clone of the upstream repo on disk.**
`MimoAudio` is not pip-installable. The loader injects it into
`sys.path` at runtime.
Quick gut check — verify each row for your env. If any are off, jump
straight to [INSTALL_NOTES.md](./INSTALL_NOTES.md):
| | Required | Upstream pin | How to check |
|---|---|---|---|
| Python | 3.10+ | 3.12 | `python --version` |
| `torch` | any recent | 2.6.0 | `python -c "import torch; print(torch.__version__)"` |
| `transformers` | **4.49.0** | 4.49.0 | `python -c "import transformers; print(transformers.__version__)"` |
| `flash-attn` | **any working build** | 2.7.4.post1 | `python -c "from flash_attn import flash_attn_varlen_func"` |
| CUDA toolkit | ≥12.0 (for flash-attn build) | 12.x | `nvcc --version` |
| VRAM | 24 GB | — | `nvidia-smi` |
---
## Install (the proven order)
> **TL;DR** — if you're on Linux with a CUDA toolkit installed and uv-managed
> ComfyUI venv, you can run the one-shot script and skip to Step 6:
> ```bash
> COMFYUI_DIR=/path/to/ComfyUI ./scripts/install_server.sh
> ```
> It does steps 2–5 below idempotently. Otherwise, follow each step manually.
Each step has a verification at the end. Don't skip the verifications.
### 1. Clone this node pack
```bash
cd ComfyUI/custom_nodes
git clone https://github.com/aadebuger/ComfyUI-MiMoASR.git
```
### 2. Clone the upstream repo
```bash
# Pick any path with disk space; the loader will find it.
git clone https://github.com/XiaomiMiMo/MiMo-V2.5-ASR.git \
/opt/repos/MiMo-V2.5-ASR
# Verify:
ls /opt/repos/MiMo-V2.5-ASR/src/mimo_audio/ # must list .py files
```
If `src/mimo_audio/` is missing, try `git submodule update --init --recursive`
inside the cloned repo.
### 3. Download the model weights
```bash
# Redirect HF cache if your root disk is small. MiMo-V2.5-ASR is ~16-32 GB
# on disk and HF stages a second copy before symlinking.
export HF_HOME=/big_disk/hf_cache
mkdir -p "$HF_HOME"
pip install -U huggingface-hub
hf download XiaomiMiMo/MiMo-V2.5-ASR \
--local-dir /big_disk/models/MiMo-V2.5-ASR
hf download XiaomiMiMo/MiMo-Audio-Tokenizer \
--local-dir /big_disk/models/MiMo-Audio-Tokenizer
# Verify:
ls -lh /big_disk/models/MiMo-V2.5-ASR/*.safetensors
ls -lh /big_disk/models/MiMo-Audio-Tokenizer/
```
### 4. Install Python deps (in ComfyUI's venv)
```bash
cd ComfyUI && source .venv/bin/activate
# Snapshot current state — you'll want this if the transformers downgrade
# breaks something else.
uv pip freeze > ~/pip_freeze_before_mimo_$(date +%Y%m%d).txt
# Pinned transformers + matching tokenizers
uv pip install "transformers==4.49.0" "tokenizers>=0.21,<0.22" --force-reinstall
# MiMo's runtime deps (from the upstream requirements.txt)
uv pip install "accelerate>=1.9.0" "librosa>=0.11.0" "scipy>=1.16.1" "zhon==2.1.1"
# Audio I/O for this node pack
uv pip install torchaudio
# CLI deps (skip if you only use the UI)
uv pip install requests websocket-client
# Verify:
python -c "import transformers; assert transformers.__version__ == '4.49.0'; print('OK')"
```
### 5. Install `flash-attn`
**Option A — your env matches the upstream wheel** (cu12 + torch 2.6 + py 3.12):
```bash
uv pip install flash-attn==2.7.4.post1
```
**Option B — build from source** (any newer torch / Python combo).
Budget 20–40 minutes on a 3090 host.
```bash
uv pip install ninja packaging
# Adjust to your toolchain
export CUDA_HOME=/usr/local/cuda-12.6
export TORCH_CUDA_ARCH_LIST="8.6" # 8.6=3090, 8.9=4090, 9.0=H100
export MAX_JOBS=4 # tune to RAM; lower if you OOM
# Background-build and tail the log so you can detach
nohup uv pip install flash-attn --no-build-isolation \
> ~/flash_attn_build.log 2>&1 &
tail -f ~/flash_attn_build.log
```
Verify:
```bash
python -c "from flash_attn import flash_attn_varlen_func; print(flash_attn_varlen_func)"
# Must print: <function flash_attn_varlen_func at 0x...>
# NOT: None
```
If `flash_attn_varlen_func` is `None`, the install silently failed —
check `~/flash_attn_build.log` for compile errors. See INSTALL_NOTES.md
for the usual suspects (missing `nvcc`, OOM, mismatched
`TORCH_CUDA_ARCH_LIST`).
### 6. Launch ComfyUI with the env vars
```bash
export HF_HOME=/big_disk/hf_cache
export MIMO_ASR_REPO=/opt/repos/MiMo-V2.5-ASR
cd ComfyUI && source .venv/bin/activate
python main.py --listen 0.0.0.0 --port 8188
```
### 7. Verify everything is wired up
```bash
curl -s http://SERVER:8188/object_info | python3 -c "
import json, sys
d = json.load(sys.stdin)
for n in ('MiMoASRLoader', 'MiMoASRTranscribe', 'LoadAudio', 'PreviewAny'):
print(f' {n}: {\"✓\" if n in d else \"✗\"}')"
```
All four should show `✓`. If `PreviewAny` is missing, install
[rgthree-comfy](https://github.com/rgthree/rgthree-comfy) or swap it out
for `ShowText|pysssss` in your workflow.
---
## Usage
### Minimal workflow
```
LoadAudio ──┐
├──► 🎯 MiMo ASR Transcribe ──► STRING
MiMo ASR ────┘
Loader
```
**Loader fields** (absolute paths, please):
- `model_path` — `/big_disk/models/MiMo-V2.5-ASR`
- `tokenizer_path` — `/big_disk/models/MiMo-Audio-Tokenizer`
- `repo_path` — leave blank if `MIMO_ASR_REPO` is set
**Transcribe fields**:
- `language` — `Auto` (recommended for code-switched speech),
`Chinese`, or `English`
The first run loads the 8B model (30–60 s). Subsequent runs in the same
ComfyUI session are instant — the loader has a process-wide cache.
An API-format example workflow is at
[`example_workflows/mimo_asr_basic.json`](./example_workflows/mimo_asr_basic.json).
### CLI
A single-file CLI that uploads an audio file, drives the workflow over
ComfyUI's HTTP + WebSocket API, and prints the transcript:
```bash
uv run python cli/mimo_asr_cli.py speech.wav \
--ip SERVER \
--model-path /big_disk/models/MiMo-V2.5-ASR \
--tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer \
--repo-path /opt/repos/MiMo-V2.5-ASR
```
See [`cli/README.md`](./cli/README.md) for all flags and a wrapper-alias
recipe to avoid retyping the paths every time.
---
## VRAM
| | |
|---|---|
| Load (bf16) | ~17 GB |
| Inference peak | ~20–22 GB |
| Minimum GPU | 24 GB (3090, 4090, A100-40G) |
| Speed (30 s audio, 3090) | ~2–4 s after first load |
A single RTX 3090 is enough. Dual-GPU machines can pin MiMo to one card
via `CUDA_VISIBLE_DEVICES=1` before launching ComfyUI.
---
## Troubleshooting (the five most common)
For the full table see [INSTALL_NOTES.md](./INSTALL_NOTES.md).
| Symptom | Cause | Fix |
|---|---|---|
| `RuntimeError: MiMo ASR repo path is empty` | Step 2 missed or env var not set | Clone the repo, set `MIMO_ASR_REPO` |
| `RuntimeError: Did not find src/mimo_audio/` | Empty / partial clone | `git submodule update --init --recursive` inside the clone |
| `RuntimeError: ... model path does not exist` | Step 3 missed, or relative path in node field | Download the weights; use absolute paths |
| `AttributeError: 'MiMoAudioTokenizer' has no attribute 'all_tied_weights_keys'` | `transformers` too new | Downgrade to `4.49.0` |
| `TypeError: 'NoneType' object is not callable` (inside `flash_attn_varlen_func`) | `flash-attn` not installed | Build per Step 5 |
---
## Links
- Hugging Face: <https://huggingface.co/XiaomiMiMo/MiMo-V2.5-ASR>
- Audio tokenizer: <https://huggingface.co/XiaomiMiMo/MiMo-Audio-Tokenizer>
- Upstream repo: <https://github.com/XiaomiMiMo/MiMo-V2.5-ASR>
- Blog: <https://mimo.xiaomi.com/mimo-v2-5-asr>
- Live demo: <https://huggingface.co/spaces/XiaomiMiMo/MiMo-V2.5-ASR>
---
## License
Apache-2.0 (this node pack). MiMo-V2.5-ASR model weights are MIT-licensed
by Xiaomi; upstream `MimoAudio` source is Apache-2.0.
## Citation
```bibtex
@misc{coreteam2026mimov25asr,
title = {MiMo-V2.5-ASR: Robust Speech Recognition Across Languages,
Dialects, and Complex Acoustic Scenarios},
author = {LLM-Core-Team Xiaomi},
year = {2026},
url = {https://github.com/XiaomiMiMo/MiMo-V2.5-ASR}
}
```
+18
View File
@@ -0,0 +1,18 @@
"""ComfyUI-MiMoASR — Xiaomi MiMo-V2.5-ASR nodes for ComfyUI."""
from .nodes.loader import MiMoASRLoader
from .nodes.transcribe import MiMoASRTranscribe
NODE_CLASS_MAPPINGS = {
"MiMoASRLoader": MiMoASRLoader,
"MiMoASRTranscribe": MiMoASRTranscribe,
}
NODE_DISPLAY_NAME_MAPPINGS = {
"MiMoASRLoader": "🎯 MiMo ASR Loader",
"MiMoASRTranscribe": "🎯 MiMo ASR Transcribe",
}
WEB_DIRECTORY = None
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
+66
View File
@@ -0,0 +1,66 @@
# ComfyUI-MiMoASR CLI
Drives the `MiMoASRLoader` + `MiMoASRTranscribe` nodes over ComfyUI's HTTP +
WebSocket API. Same `ComfyClient` template used across the other
`ComfyUI-*` packages.
## Quick start
```bash
# uv handles dependencies via the PEP 723 inline script metadata
uv run python mimo_asr_cli.py /path/to/speech.wav --ip 10.8.0.12
# Explicit Chinese tag (useful for accented or noisy speech)
uv run python mimo_asr_cli.py speech.wav --language Chinese --ip 10.8.0.12
# Pass model + repo paths if your ComfyUI server can't see the defaults
uv run python mimo_asr_cli.py speech.wav \
--model-path /mnt/models/MiMo-V2.5-ASR \
--tokenizer-path /mnt/models/MiMo-Audio-Tokenizer \
--repo-path /mnt/repos/MiMo-V2.5-ASR \
--ip 10.8.0.12
# Save transcript to file
uv run python mimo_asr_cli.py speech.wav --out transcript.txt --ip 10.8.0.12
```
## Flags
| Flag | Default | Notes |
|---|---|---|
| `audio` (positional) | — | Local audio file. Uploaded to ComfyUI's `input/`. |
| `--ip` | `127.0.0.1` | ComfyUI host. |
| `--port` | `8188` | ComfyUI port. |
| `--server` | — | Full `host:port`, overrides `--ip`/`--port`. |
| `--out` | — | Write transcript to file. |
| `--model-path` | `./models/MiMo-V2.5-ASR` | Relative to ComfyUI cwd, or absolute. |
| `--tokenizer-path` | `./models/MiMo-Audio-Tokenizer` | Same. |
| `--repo-path` | (env `MIMO_ASR_REPO`) | Local clone of [XiaomiMiMo/MiMo-V2.5-ASR](https://github.com/XiaomiMiMo/MiMo-V2.5-ASR). |
| `--language` | `Auto` | `Auto` / `Chinese` / `English`. |
| `--dump-workflow PATH` | — | Save the API-format JSON for inspection. |
| `--no-run` | off | Combine with `--dump-workflow` to skip submission. |
| `--timeout` | `600` | Seconds. |
## Server-side checklist
```bash
# 1. ComfyUI is up
curl -s -o /dev/null -w "HTTP %{http_code}\n" http://SERVER:8188/
# 2. Nodes loaded
curl -s http://SERVER:8188/object_info | python3 -c "
import json, sys
d = json.load(sys.stdin)
for n in ('MiMoASRLoader', 'MiMoASRTranscribe', 'LoadAudio', 'PreviewAny'):
print(f' {n}: {\"✓\" if n in d else \"✗\"}')"
```
`PreviewAny` ships with `rgthree-comfy` (and several others). If it's not
installed, swap the `preview` node in `build_workflow()` for whatever your
environment provides — `ShowText|pysssss` is a drop-in replacement.
## Audio format notes
`/upload/image` accepts any file (the endpoint's name is misleading). Audio
gets stored under ComfyUI's `input/` folder, and `LoadAudio` handles
decoding via torchaudio — so wav / mp3 / flac / m4a / ogg all work.
+333
View File
@@ -0,0 +1,333 @@
#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "requests",
# "websocket-client",
# ]
# ///
"""mimo_asr_cli.py — drive ComfyUI-MiMoASR from the command line.
Examples
--------
# 本地音频 → 自动语言检测
uv run python cli/mimo_asr_cli.py audio.wav --ip 10.8.0.12
# 显式指定中文
uv run python cli/mimo_asr_cli.py speech.flac --language Chinese --ip 10.8.0.12
# 自定义模型路径(默认是 ./models/MiMo-V2.5-ASR + ./models/MiMo-Audio-Tokenizer
# 相对于 ComfyUI 启动目录)
uv run python cli/mimo_asr_cli.py speech.wav \
--model-path /mnt/models/MiMo-V2.5-ASR \
--tokenizer-path /mnt/models/MiMo-Audio-Tokenizer \
--repo-path /mnt/repos/MiMo-V2.5-ASR \
--ip 10.8.0.12
# 调试:dump 生成的 workflow 不实际执行
uv run python cli/mimo_asr_cli.py speech.wav --dump-workflow /tmp/wf.json --no-run
"""
from __future__ import annotations
import argparse
import json
import mimetypes
import re
import sys
import time
import urllib.parse
import uuid
from pathlib import Path
import requests
import websocket # websocket-client
LANGUAGES = ["Auto", "Chinese", "English"]
# ---------------------------------------------------------------------------
# ComfyClient — minimal HTTP + WS wrapper around a ComfyUI server
# ---------------------------------------------------------------------------
class ComfyClient:
def __init__(self, server: str = "127.0.0.1:8188") -> None:
self.server = server
self.http = f"http://{server}"
self.ws = f"ws://{server}/ws"
self.client_id = str(uuid.uuid4())
def upload_image(self, file_path: Path, *, overwrite: bool = True) -> str:
"""Upload any file via POST /upload/image. Works for audio too."""
mime, _ = mimetypes.guess_type(str(file_path))
with open(file_path, "rb") as fh:
files = {"image": (file_path.name, fh, mime or "application/octet-stream")}
data = {"overwrite": "true" if overwrite else "false"}
r = requests.post(f"{self.http}/upload/image", files=files, data=data, timeout=60)
r.raise_for_status()
payload = r.json()
# ComfyUI returns {"name": "<server-side filename>", ...}
return payload["name"]
def object_info(self) -> dict:
r = requests.get(f"{self.http}/object_info", timeout=15)
r.raise_for_status()
return r.json()
def submit(self, workflow: dict) -> str:
r = requests.post(
f"{self.http}/prompt",
json={"prompt": workflow, "client_id": self.client_id},
timeout=30,
)
r.raise_for_status()
payload = r.json()
if "prompt_id" not in payload:
raise RuntimeError(f"Submit failed: {payload}")
return payload["prompt_id"]
def history(self, prompt_id: str) -> dict:
r = requests.get(f"{self.http}/history/{prompt_id}", timeout=15)
r.raise_for_status()
data = r.json()
return data.get(prompt_id, {})
def view(self, filename: str, subfolder: str, type_: str) -> bytes:
params = urllib.parse.urlencode(
{"filename": filename, "subfolder": subfolder, "type": type_}
)
r = requests.get(f"{self.http}/view?{params}", timeout=60)
r.raise_for_status()
return r.content
def run(self, workflow: dict, *, timeout_s: float = 600.0) -> dict:
"""Submit a workflow and stream progress until completion."""
prompt_id = self.submit(workflow)
print(f" prompt_id = {prompt_id}")
ws = websocket.create_connection(f"{self.ws}?clientId={self.client_id}", timeout=30)
ws.settimeout(timeout_s)
deadline = time.time() + timeout_s
last_node = None
try:
while True:
if time.time() > deadline:
raise TimeoutError(f"Workflow exceeded {timeout_s}s")
msg = ws.recv()
if isinstance(msg, (bytes, bytearray)):
continue
try:
payload = json.loads(msg)
except json.JSONDecodeError:
continue
t = payload.get("type")
d = payload.get("data", {})
if d.get("prompt_id") and d.get("prompt_id") != prompt_id:
continue
if t == "progress":
cur, tot = d.get("value", 0), d.get("max", 1)
sys.stdout.write(f"\r ▸ {cur:>4}/{tot}")
sys.stdout.flush()
elif t == "executing":
node = d.get("node")
if node is None:
# execution finished for this prompt
sys.stdout.write("\n")
break
if node != last_node:
sys.stdout.write(f"\r ▸ node {str(node):<20}\n")
sys.stdout.flush()
last_node = node
elif t == "execution_error":
sys.stdout.write("\n")
raise RuntimeError(
f"Workflow error in node {d.get('node_id')}: "
f"{d.get('exception_message')}"
)
finally:
try:
ws.close()
except Exception: # noqa: BLE001
pass
return self.history(prompt_id)
# ---------------------------------------------------------------------------
# Workflow construction
# ---------------------------------------------------------------------------
def build_workflow(
*,
audio_filename: str,
model_path: str,
tokenizer_path: str,
repo_path: str,
language: str,
) -> dict:
"""Return the API-format workflow dict."""
return {
"loader": {
"class_type": "MiMoASRLoader",
"inputs": {
"model_path": model_path,
"tokenizer_path": tokenizer_path,
"repo_path": repo_path,
"force_reload": False,
},
},
"load_audio": {
"class_type": "LoadAudio",
"inputs": {
"audio": audio_filename,
},
},
"asr": {
"class_type": "MiMoASRTranscribe",
"inputs": {
"model": ["loader", 0],
"audio": ["load_audio", 0],
"language": language,
},
},
# Capture STRING output so we can read it back from /history.
"preview": {
"class_type": "PreviewAny",
"inputs": {
"source": ["asr", 0],
},
},
}
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def extract_text_from_history(history: dict, node_id: str = "preview") -> str:
"""Pull the text payload PreviewAny stored under outputs[node_id]."""
outs = history.get("outputs", {})
if node_id not in outs:
# Fall back to the asr node itself if PreviewAny was missing.
for nid, node_out in outs.items():
if "text" in node_out:
node_id = nid
break
else:
return ""
node_out = outs[node_id]
if "text" in node_out:
v = node_out["text"]
return "\n".join(str(x) for x in v) if isinstance(v, list) else str(v)
parts: list[str] = []
for v in node_out.values():
if isinstance(v, list):
parts.extend(str(x) for x in v)
return "\n".join(parts)
_SAFE_NAME_RE = re.compile(r"[^A-Za-z0-9._-]+")
def sanitize_filename(name: str) -> str:
sanitized = _SAFE_NAME_RE.sub("_", name).strip("._-")
return sanitized or "audio"
# ---------------------------------------------------------------------------
# main
# ---------------------------------------------------------------------------
def main() -> int:
ap = argparse.ArgumentParser(description="ComfyUI-MiMoASR CLI")
ap.add_argument("audio", type=Path, help="Local audio file to transcribe (wav/mp3/flac/m4a/...)")
# Server.
ap.add_argument("--ip", default="127.0.0.1")
ap.add_argument("--port", type=int, default=8188)
ap.add_argument("--server", default=None, help="Full host:port, overrides --ip/--port")
# Output.
ap.add_argument("--out", type=Path, default=None,
help="Write transcript to file (default: stdout only).")
# MiMo paths.
ap.add_argument("--model-path", default="./models/MiMo-V2.5-ASR",
help="Path to MiMo-V2.5-ASR weights (relative to ComfyUI cwd or absolute).")
ap.add_argument("--tokenizer-path", default="./models/MiMo-Audio-Tokenizer",
help="Path to MiMo-Audio-Tokenizer.")
ap.add_argument("--repo-path", default="",
help="Path to local clone of XiaomiMiMo/MiMo-V2.5-ASR. "
"Leave blank to use the MIMO_ASR_REPO env var on the server.")
# Generation.
ap.add_argument("--language", choices=LANGUAGES, default="Auto")
# Debug.
ap.add_argument("--dump-workflow", type=Path, default=None,
help="Save the generated workflow JSON for inspection.")
ap.add_argument("--no-run", action="store_true",
help="Skip submission (use with --dump-workflow to just produce the JSON).")
ap.add_argument("--timeout", type=float, default=600.0)
args = ap.parse_args()
server = args.server or f"{args.ip}:{args.port}"
if not args.audio.is_file():
print(f"Error: audio file not found: {args.audio}", file=sys.stderr)
return 2
client = ComfyClient(server=server)
# 1) Upload audio.
if not args.no_run:
print(f"▸ Uploading {args.audio.name} to {server} ...")
uploaded_name = client.upload_image(args.audio)
print(f" uploaded as: {uploaded_name}")
else:
uploaded_name = sanitize_filename(args.audio.name)
# 2) Build workflow.
workflow = build_workflow(
audio_filename=uploaded_name,
model_path=args.model_path,
tokenizer_path=args.tokenizer_path,
repo_path=args.repo_path,
language=args.language,
)
if args.dump_workflow:
args.dump_workflow.parent.mkdir(parents=True, exist_ok=True)
args.dump_workflow.write_text(
json.dumps(workflow, indent=2, ensure_ascii=False), encoding="utf-8"
)
print(f"▸ Workflow JSON written to {args.dump_workflow}")
if args.no_run:
print("▸ --no-run set, exiting without submission.")
return 0
# 3) Run.
print(f"▸ Submitting workflow (language={args.language!r}) ...")
t0 = time.time()
history = client.run(workflow, timeout_s=args.timeout)
elapsed = time.time() - t0
print(f"▸ Workflow finished in {elapsed:.2f}s")
# 4) Extract text.
text = extract_text_from_history(history, node_id="preview")
if not text:
print("Warning: no text output found in workflow history.", file=sys.stderr)
print(json.dumps(history.get("outputs", {}), indent=2, ensure_ascii=False)[:2000],
file=sys.stderr)
return 1
print("\n----- Transcript -----")
print(text)
print("----------------------")
if args.out:
args.out.parent.mkdir(parents=True, exist_ok=True)
args.out.write_text(text + "\n", encoding="utf-8")
print(f"▸ Saved to {args.out}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+31
View File
@@ -0,0 +1,31 @@
{
"loader": {
"class_type": "MiMoASRLoader",
"inputs": {
"model_path": "./models/MiMo-V2.5-ASR",
"tokenizer_path": "./models/MiMo-Audio-Tokenizer",
"repo_path": "",
"force_reload": false
}
},
"load_audio": {
"class_type": "LoadAudio",
"inputs": {
"audio": "sample.wav"
}
},
"asr": {
"class_type": "MiMoASRTranscribe",
"inputs": {
"model": ["loader", 0],
"audio": ["load_audio", 0],
"language": "Auto"
}
},
"preview": {
"class_type": "PreviewAny",
"inputs": {
"source": ["asr", 0]
}
}
}
View File
+18
View File
@@ -0,0 +1,18 @@
"""Shared types and constants for ComfyUI-MiMoASR."""
# ComfyUI custom type name for the MiMo model bundle returned by the loader.
MIMO_ASR_MODEL = "MIMO_ASR_MODEL"
# Language tags accepted by MimoAudio.asr_sft(audio_tag=...).
LANGUAGE_TAGS = {
"Auto": "",
"Chinese": "<chinese>",
"English": "<english>",
}
# Node category shown in the ComfyUI search bar.
CATEGORY = "🎯 MiMo ASR"
# Env var the loader falls back to when `repo_path` is left blank in the node UI.
# Should point at a local clone of https://github.com/XiaomiMiMo/MiMo-V2.5-ASR
ENV_REPO_PATH = "MIMO_ASR_REPO"
+155
View File
@@ -0,0 +1,155 @@
"""MiMoASRLoader — load the MiMo-V2.5-ASR model + audio tokenizer."""
from __future__ import annotations
import os
import sys
from pathlib import Path
from ._types import CATEGORY, ENV_REPO_PATH, MIMO_ASR_MODEL
# Process-wide cache so reruns of the same workflow don't reload the 8B model.
_MODEL_CACHE: dict[str, dict] = {}
def _resolve_repo_path(repo_path: str) -> str:
"""Return the resolved repo path, or raise with a useful error."""
candidate = repo_path.strip() or os.environ.get(ENV_REPO_PATH, "").strip()
if not candidate:
raise RuntimeError(
"MiMo ASR repo path is empty. Either:\n"
" 1. Set the `repo_path` field on the MiMoASRLoader node, OR\n"
f" 2. Set the {ENV_REPO_PATH} env var before launching ComfyUI.\n\n"
"The path must point to a local clone of:\n"
" https://github.com/XiaomiMiMo/MiMo-V2.5-ASR\n"
"i.e. the directory that contains the `src/mimo_audio/` folder."
)
repo = Path(candidate).expanduser().resolve()
src_dir = repo / "src" / "mimo_audio"
if not src_dir.is_dir():
raise RuntimeError(
f"Did not find `src/mimo_audio/` under {repo}.\n"
"Make sure repo_path points at the root of the MiMo-V2.5-ASR repo "
"(the folder containing `run_mimo_asr.py` and the `src/` directory)."
)
repo_str = str(repo)
if repo_str not in sys.path:
sys.path.insert(0, repo_str)
return repo_str
class MiMoASRLoader:
"""Load Xiaomi MiMo-V2.5-ASR (8B) and its audio tokenizer.
The MimoAudio class is not pip-installable, so this node sources it
from a local clone of the upstream repo. Loading takes ~30-60 s on a
3090 and consumes ~16 GB of VRAM — the result is cached by
(model_path, tokenizer_path, repo_path) so subsequent nodes reuse it.
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"model_path": (
"STRING",
{
"default": "./models/MiMo-V2.5-ASR",
"multiline": False,
"tooltip": "Local path to the MiMo-V2.5-ASR weights "
"(downloaded from XiaomiMiMo/MiMo-V2.5-ASR).",
},
),
"tokenizer_path": (
"STRING",
{
"default": "./models/MiMo-Audio-Tokenizer",
"multiline": False,
"tooltip": "Local path to the MiMo-Audio-Tokenizer "
"(downloaded from XiaomiMiMo/MiMo-Audio-Tokenizer).",
},
),
"repo_path": (
"STRING",
{
"default": "",
"multiline": False,
"tooltip": f"Path to a clone of XiaomiMiMo/MiMo-V2.5-ASR "
f"(the folder containing src/mimo_audio/). "
f"Leave blank to read from the {ENV_REPO_PATH} env var.",
},
),
},
"optional": {
"force_reload": (
"BOOLEAN",
{"default": False, "tooltip": "Bypass the in-memory model cache."},
),
},
}
RETURN_TYPES = (MIMO_ASR_MODEL,)
RETURN_NAMES = ("model",)
FUNCTION = "load"
CATEGORY = CATEGORY
def load(
self,
model_path: str,
tokenizer_path: str,
repo_path: str,
force_reload: bool = False,
):
model_path = str(Path(model_path).expanduser())
tokenizer_path = str(Path(tokenizer_path).expanduser())
repo = _resolve_repo_path(repo_path)
cache_key = f"{model_path}|{tokenizer_path}|{repo}"
if not force_reload and cache_key in _MODEL_CACHE:
return (_MODEL_CACHE[cache_key],)
# Lazy import — only after sys.path is patched and only when loading
# is actually requested. Avoids importing torch/transformers at
# ComfyUI startup.
from src.mimo_audio.mimo_audio import MimoAudio # type: ignore
if not Path(model_path).is_dir():
raise RuntimeError(
f"MiMo-V2.5-ASR model path does not exist: {model_path}\n"
"Download with:\n"
" hf download XiaomiMiMo/MiMo-V2.5-ASR "
f"--local-dir {model_path}"
)
if not Path(tokenizer_path).is_dir():
raise RuntimeError(
f"MiMo-Audio-Tokenizer path does not exist: {tokenizer_path}\n"
"Download with:\n"
" hf download XiaomiMiMo/MiMo-Audio-Tokenizer "
f"--local-dir {tokenizer_path}"
)
print(f"[MiMoASRLoader] Loading MiMo-V2.5-ASR from {model_path}")
print(f"[MiMoASRLoader] Loading audio tokenizer from {tokenizer_path}")
# Upstream's run_mimo_asr.py uses positional args. The HF README shows
# kwargs that don't match the actual __init__ signature — call by
# position to stay compatible with both old and new upstream commits.
model = MimoAudio(model_path, tokenizer_path)
bundle = {
"model": model,
"model_path": model_path,
"tokenizer_path": tokenizer_path,
"repo_path": repo,
}
_MODEL_CACHE[cache_key] = bundle
print("[MiMoASRLoader] Model ready.")
return (bundle,)
@classmethod
def IS_CHANGED(cls, model_path, tokenizer_path, repo_path, force_reload=False):
# When force_reload is on, return a sentinel that always differs.
if force_reload:
return float("NaN")
return f"{model_path}|{tokenizer_path}|{repo_path}"
+123
View File
@@ -0,0 +1,123 @@
"""MiMoASRTranscribe — run MiMo-V2.5-ASR on an audio input."""
from __future__ import annotations
import os
import tempfile
from pathlib import Path
from ._types import CATEGORY, LANGUAGE_TAGS, MIMO_ASR_MODEL
def _audio_dict_to_wav(audio: dict, out_path: Path) -> None:
"""Persist a ComfyUI AUDIO dict to a wav file on disk.
AUDIO is `{"waveform": Tensor[B, C, N], "sample_rate": int}`.
We squeeze the batch axis and pick channel 0 if multi-channel — the
upstream MimoAudio pipeline expects a mono 16k-style waveform but
handles its own resampling internally.
"""
import torch # noqa: F401 (kept local — heavy import)
import torchaudio
waveform = audio["waveform"]
sample_rate = int(audio["sample_rate"])
if waveform.ndim == 3:
# [B, C, N] → [C, N] (take first item in batch)
waveform = waveform[0]
elif waveform.ndim == 1:
waveform = waveform.unsqueeze(0)
# MimoAudio expects 16 kHz mono internally; downmix to mono here and
# leave resampling to the model's own preprocessing.
if waveform.size(0) > 1:
waveform = waveform.mean(dim=0, keepdim=True)
# torchaudio.save needs a float CPU tensor.
waveform = waveform.detach().to("cpu").float()
torchaudio.save(str(out_path), waveform, sample_rate)
class MiMoASRTranscribe:
"""Transcribe an AUDIO input using a loaded MiMo-V2.5-ASR model.
Supports automatic language detection (recommended for code-switched
speech) or explicit Chinese / English language tags. Punctuation is
produced natively by the model — no post-processing needed.
"""
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"model": (MIMO_ASR_MODEL,),
"audio": ("AUDIO",),
"language": (
list(LANGUAGE_TAGS.keys()),
{
"default": "Auto",
"tooltip": "Auto = automatic language detection "
"(recommended for code-switched speech).",
},
),
},
"optional": {
"keep_temp_file": (
"BOOLEAN",
{
"default": False,
"tooltip": "If True, keep the intermediate wav file "
"(useful for debugging).",
},
),
},
}
RETURN_TYPES = ("STRING",)
RETURN_NAMES = ("text",)
FUNCTION = "transcribe"
CATEGORY = CATEGORY
OUTPUT_NODE = False
def transcribe(
self,
model: dict,
audio: dict,
language: str = "Auto",
keep_temp_file: bool = False,
):
if not isinstance(model, dict) or "model" not in model:
raise RuntimeError(
"MiMoASRTranscribe got an invalid model bundle. "
"Connect the `model` output of MiMoASRLoader."
)
if not isinstance(audio, dict) or "waveform" not in audio:
raise RuntimeError(
"MiMoASRTranscribe expected a ComfyUI AUDIO dict on `audio` "
"(connect a LoadAudio node)."
)
audio_tag = LANGUAGE_TAGS.get(language, "")
mimo = model["model"]
# MimoAudio.asr_sft takes a file path, so dump the waveform to a tempfile.
fd, tmp_path = tempfile.mkstemp(prefix="mimoasr_", suffix=".wav")
os.close(fd)
tmp = Path(tmp_path)
try:
_audio_dict_to_wav(audio, tmp)
print(f"[MiMoASRTranscribe] Transcribing {tmp} (lang={language!r})")
text = mimo.asr_sft(str(tmp), audio_tag=audio_tag)
if text is None:
text = ""
text = str(text).strip()
print(f"[MiMoASRTranscribe] {len(text)} chars produced.")
return (text,)
finally:
if not keep_temp_file:
try:
tmp.unlink(missing_ok=True)
except Exception: # noqa: BLE001
pass
+44
View File
@@ -0,0 +1,44 @@
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "comfyui-mimoasr"
version = "0.1.2"
description = "ComfyUI custom nodes for Xiaomi MiMo-V2.5-ASR speech recognition"
readme = "README.md"
license = { text = "Apache-2.0" }
requires-python = ">=3.10"
authors = [
{ name = "aadebuger" },
]
keywords = ["comfyui", "asr", "speech-recognition", "mimo", "xiaomi", "whisper"]
classifiers = [
"Development Status :: 3 - Alpha",
"Intended Audience :: Developers",
"License :: OSI Approved :: Apache Software License",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Topic :: Multimedia :: Sound/Audio :: Speech",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
]
dependencies = [
"torchaudio",
]
[project.optional-dependencies]
cli = [
"requests",
"websocket-client",
]
[project.urls]
Homepage = "https://github.com/aadebuger/ComfyUI-MiMoASR"
Issues = "https://github.com/aadebuger/ComfyUI-MiMoASR/issues"
Upstream = "https://github.com/XiaomiMiMo/MiMo-V2.5-ASR"
HuggingFace = "https://huggingface.co/XiaomiMiMo/MiMo-V2.5-ASR"
[tool.setuptools]
packages = ["nodes"]
+9
View File
@@ -0,0 +1,9 @@
# Runtime deps for the ComfyUI node pack itself.
# (Heavy deps — torch, transformers, accelerate, librosa, flash-attn —
# must come from the upstream MiMo-V2.5-ASR repo's requirements.txt,
# installed into your ComfyUI venv. See INSTALL_NOTES.md.)
torchaudio
# CLI deps (only needed for cli/mimo_asr_cli.py)
requests
websocket-client
+194
View File
@@ -0,0 +1,194 @@
#!/usr/bin/env bash
# install_server.sh — one-shot server install for ComfyUI-MiMoASR.
#
# Run this on the ComfyUI server (the machine with the GPU), AFTER
# you've cloned this node pack into ComfyUI/custom_nodes/.
#
# What it does:
# 1. Clones the upstream MiMo-V2.5-ASR repo
# 2. Downloads the two model checkpoints
# 3. Pins transformers to 4.49.0 in your ComfyUI venv
# 4. Installs MiMo's other runtime deps
# 5. Builds flash-attn from source (only if not already importable)
#
# What it does NOT do:
# - Launch ComfyUI (that's your call — `python main.py --listen 0.0.0.0`)
# - Modify your shell rc files
# - Touch other custom nodes
#
# Configure via env vars before running:
# COMFYUI_DIR — where your ComfyUI lives (must have .venv/)
# MODELS_DIR — where to put MiMo weights (default: $COMFYUI_DIR/models/mimo)
# REPOS_DIR — where to clone upstream repo (default: $COMFYUI_DIR/repos)
# HF_HOME — Hugging Face cache (default: $MODELS_DIR/_hf_cache)
# CUDA_HOME — required for flash-attn build (default: /usr/local/cuda)
# GPU_ARCH — TORCH_CUDA_ARCH_LIST (default: auto-detected via torch)
# MAX_JOBS — parallel compile jobs for flash-attn (default: 4)
# SKIP_DOWNLOAD — set to 1 to skip the model download step
# SKIP_FLASH_ATTN — set to 1 to skip the flash-attn build step
set -euo pipefail
# ---------- Configuration ----------------------------------------------------
COMFYUI_DIR="${COMFYUI_DIR:-/mnt/modelall/ComfyUI}"
MODELS_DIR="${MODELS_DIR:-$COMFYUI_DIR/models/mimo}"
REPOS_DIR="${REPOS_DIR:-$COMFYUI_DIR/repos}"
HF_HOME="${HF_HOME:-$MODELS_DIR/_hf_cache}"
CUDA_HOME="${CUDA_HOME:-/usr/local/cuda}"
MAX_JOBS="${MAX_JOBS:-4}"
export HF_HOME
# ---------- Helpers ----------------------------------------------------------
log() { printf '\033[1;36m▸\033[0m %s\n' "$*"; }
ok() { printf '\033[1;32m✓\033[0m %s\n' "$*"; }
warn() { printf '\033[1;33m!\033[0m %s\n' "$*"; }
fatal() { printf '\033[1;31m✗\033[0m %s\n' "$*" >&2; exit 1; }
# ---------- Pre-flight -------------------------------------------------------
log "Configuration:"
echo " COMFYUI_DIR = $COMFYUI_DIR"
echo " MODELS_DIR = $MODELS_DIR"
echo " REPOS_DIR = $REPOS_DIR"
echo " HF_HOME = $HF_HOME"
echo " CUDA_HOME = $CUDA_HOME"
echo " MAX_JOBS = $MAX_JOBS"
echo
[ -d "$COMFYUI_DIR" ] || fatal "ComfyUI dir not found: $COMFYUI_DIR (set COMFYUI_DIR)"
[ -d "$COMFYUI_DIR/.venv" ] || fatal "No .venv in $COMFYUI_DIR — this script expects a uv-managed venv"
mkdir -p "$MODELS_DIR" "$REPOS_DIR" "$HF_HOME"
# Activate venv
# shellcheck disable=SC1091
source "$COMFYUI_DIR/.venv/bin/activate"
ok "Activated venv: $(which python)"
# ---------- 1. Clone upstream repo -------------------------------------------
log "Step 1/5: Clone XiaomiMiMo/MiMo-V2.5-ASR"
if [ -d "$REPOS_DIR/MiMo-V2.5-ASR/.git" ]; then
ok "Already cloned at $REPOS_DIR/MiMo-V2.5-ASR"
else
git clone https://github.com/XiaomiMiMo/MiMo-V2.5-ASR.git "$REPOS_DIR/MiMo-V2.5-ASR"
fi
if [ ! -d "$REPOS_DIR/MiMo-V2.5-ASR/src/mimo_audio" ]; then
warn "src/mimo_audio/ missing — running git submodule init"
(cd "$REPOS_DIR/MiMo-V2.5-ASR" && git submodule update --init --recursive)
fi
[ -d "$REPOS_DIR/MiMo-V2.5-ASR/src/mimo_audio" ] \
|| fatal "Still no src/mimo_audio/ — repo structure may have changed upstream"
ok "Upstream repo OK"
# ---------- 2. Download models -----------------------------------------------
log "Step 2/5: Download model weights"
if [ "${SKIP_DOWNLOAD:-0}" = "1" ]; then
warn "SKIP_DOWNLOAD=1, skipping"
else
uv pip install -U huggingface-hub --quiet
if [ ! -f "$MODELS_DIR/MiMo-V2.5-ASR/config.json" ]; then
log " Downloading MiMo-V2.5-ASR (~16-32 GB)"
hf download XiaomiMiMo/MiMo-V2.5-ASR --local-dir "$MODELS_DIR/MiMo-V2.5-ASR"
else
ok "MiMo-V2.5-ASR already present"
fi
if [ ! -f "$MODELS_DIR/MiMo-Audio-Tokenizer/config.json" ]; then
log " Downloading MiMo-Audio-Tokenizer"
hf download XiaomiMiMo/MiMo-Audio-Tokenizer --local-dir "$MODELS_DIR/MiMo-Audio-Tokenizer"
else
ok "MiMo-Audio-Tokenizer already present"
fi
fi
# ---------- 3. Pin transformers ----------------------------------------------
log "Step 3/5: Snapshot freeze + pin transformers==4.49.0"
FREEZE_FILE="$HOME/pip_freeze_before_mimo_$(date +%Y%m%d_%H%M%S).txt"
uv pip freeze > "$FREEZE_FILE"
ok "Snapshot saved to $FREEZE_FILE (use this to roll back if something breaks)"
current_transformers=$(python -c "import transformers; print(transformers.__version__)" 2>/dev/null || echo "none")
if [ "$current_transformers" = "4.49.0" ]; then
ok "transformers already at 4.49.0"
else
warn "Current transformers: $current_transformers — downgrading to 4.49.0"
uv pip install "transformers==4.49.0" "tokenizers>=0.21,<0.22" --force-reinstall
fi
# ---------- 4. Other deps ----------------------------------------------------
log "Step 4/5: Install MiMo runtime deps + audio I/O"
uv pip install "accelerate>=1.9.0" "librosa>=0.11.0" "scipy>=1.16.1" "zhon==2.1.1"
uv pip install torchaudio requests websocket-client
ok "Deps installed"
# ---------- 5. flash-attn ----------------------------------------------------
log "Step 5/5: flash-attn"
if [ "${SKIP_FLASH_ATTN:-0}" = "1" ]; then
warn "SKIP_FLASH_ATTN=1, skipping"
elif python -c "from flash_attn import flash_attn_varlen_func; assert flash_attn_varlen_func is not None" 2>/dev/null; then
ok "flash-attn already installed and working"
else
warn "flash-attn not importable — building from source (20-40 min)"
[ -x "$CUDA_HOME/bin/nvcc" ] || fatal "nvcc not found at $CUDA_HOME/bin/nvcc — set CUDA_HOME"
# Auto-detect GPU arch if not provided
if [ -z "${GPU_ARCH:-}" ]; then
GPU_ARCH=$(python -c "
import torch
major, minor = torch.cuda.get_device_capability(0)
print(f'{major}.{minor}')" 2>/dev/null) || GPU_ARCH="8.6"
log " Auto-detected GPU arch: $GPU_ARCH (override with GPU_ARCH=...)"
fi
uv pip install ninja packaging
export CUDA_HOME
export PATH="$CUDA_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib64:${LD_LIBRARY_PATH:-}"
export TORCH_CUDA_ARCH_LIST="$GPU_ARCH"
export MAX_JOBS
BUILD_LOG="$HOME/flash_attn_build_$(date +%Y%m%d_%H%M%S).log"
log " Building (this WILL take a long time). Log: $BUILD_LOG"
log " Tip: detach from screen/tmux now and run \`tail -f $BUILD_LOG\` in another shell"
if uv pip install flash-attn --no-build-isolation 2>&1 | tee "$BUILD_LOG"; then
if python -c "from flash_attn import flash_attn_varlen_func; assert flash_attn_varlen_func is not None"; then
ok "flash-attn built and importable"
else
fatal "flash-attn install reported success but import returned None. Check $BUILD_LOG"
fi
else
fatal "flash-attn build failed. See $BUILD_LOG"
fi
fi
# ---------- Done -------------------------------------------------------------
cat <<EOF
$(ok "Install complete.")
Next steps:
1. Launch ComfyUI with the env vars set:
export HF_HOME=$HF_HOME
export MIMO_ASR_REPO=$REPOS_DIR/MiMo-V2.5-ASR
cd $COMFYUI_DIR && source .venv/bin/activate
python main.py --listen 0.0.0.0 --port 8188
2. In the MiMoASRLoader node, set:
model_path = $MODELS_DIR/MiMo-V2.5-ASR
tokenizer_path = $MODELS_DIR/MiMo-Audio-Tokenizer
repo_path = (blank — uses MIMO_ASR_REPO env var)
3. From the client, smoke-test the CLI:
uv run python cli/mimo_asr_cli.py speech.wav \\
--ip SERVER_IP \\
--model-path $MODELS_DIR/MiMo-V2.5-ASR \\
--tokenizer-path $MODELS_DIR/MiMo-Audio-Tokenizer \\
--repo-path $REPOS_DIR/MiMo-V2.5-ASR
EOF