304 lines
10 KiB
Markdown
304 lines
10 KiB
Markdown
# Install Notes — 实战安装记录
|
||
|
||
Real-world install log for the video-to-text pipeline. 这是一份"踩坑笔记"，按照实际部署顺序记录可能的问题和解法。Read this before opening an issue.
|
||
|
||
## 架构 / Architecture
|
||
|
||
ComfyUI-VideoToText 是个**包装层**，本身**不包含 ASR 模型**。完整链路：
|
||
|
||
```
|
||
your_video.mp4
|
||
│
|
||
├─► [本包] VTT_LoadVideo
|
||
│ │
|
||
│ └─► [本包] VTT_ExtractAudio (ffmpeg)
|
||
│ │
|
||
│ └─► [本包] VTT_AudioFromPath (torchaudio)
|
||
│ │
|
||
│ ├──────────────────┐
|
||
│ ▼ ▼
|
||
└────► [MiMoASR] MiMoASRLoader + MiMoASRTranscribe
|
||
│
|
||
└─► [本包] VTT_BuildTranscriptJSON
|
||
│
|
||
└─► [本包] VTT_SaveTranscript
|
||
│
|
||
▼
|
||
ComfyUI/output/<file>.json
|
||
```
|
||
|
||
依赖的两个外部组件：
|
||
|
||
| 节点 | 来源 | 用途 |
|
||
| --- | --- | --- |
|
||
| `MiMoASRLoader`, `MiMoASRTranscribe` | [aadebuger/ComfyUI-MiMoASR](https://github.com/aadebuger/ComfyUI-MiMoASR) | ASR 核心 |
|
||
| `PreviewAny` | [rgthree/rgthree-comfy](https://github.com/rgthree/rgthree-comfy) | CLI 抓 STRING 输出用 |
|
||
| `LoadAudio` | ComfyUI 内置 | 节点 UI 端测试用 |
|
||
|
||
## 测试环境 / Tested Environment
|
||
|
||
| Item | Value |
|
||
| --- | --- |
|
||
| OS | Ubuntu 22.04 |
|
||
| GPU | 2× RTX 3090 (24 GB each) |
|
||
| CUDA toolkit | 12.6 (`/usr/local/cuda-12.6`) |
|
||
| Python | 3.13 |
|
||
| PyTorch | 2.11.0+cu126 |
|
||
| ComfyUI venv | `uv`-managed at `/mnt/modelall/ComfyUI/.venv` |
|
||
| HF cache | `/mnt/modelall/hf_cache` (system disk only had 10 GB → must redirect) |
|
||
| MiMo 上游 repo | `/opt/repos/MiMo-V2.5-ASR` (set via `MIMO_ASR_REPO` env var) |
|
||
| MiMo 模型 | `/big_disk/models/MiMo-V2.5-ASR` + `/big_disk/models/MiMo-Audio-Tokenizer` |
|
||
|
||
---
|
||
|
||
## 安装步骤 (Step-by-Step)
|
||
|
||
### Step 1 — `ffmpeg`
|
||
|
||
```bash
|
||
$ which ffmpeg || sudo apt install ffmpeg
|
||
$ ffmpeg -version
|
||
```
|
||
|
||
`VTT_ExtractAudio` 走 `subprocess` 直调 `ffmpeg` —— 没装就报：
|
||
|
||
```
|
||
RuntimeError: ffmpeg not found on PATH. Install with: sudo apt install ffmpeg
|
||
```
|
||
|
||
如果 ComfyUI 在容器里跑，必须装在容器里，不是宿主机。
|
||
|
||
### Step 2 — `torchaudio`
|
||
|
||
```bash
|
||
$ cd /mnt/modelall/ComfyUI && source .venv/bin/activate
|
||
$ uv pip install torchaudio
|
||
$ python -c "import torchaudio; print(torchaudio.__version__)"
|
||
```
|
||
|
||
`torchaudio` 在 `AudioFromPath.load()` 里 **lazy import** ——
|
||
所以没装上时 ComfyUI 启动**不会报错**，只在你跑工作流时才看到：
|
||
|
||
```
|
||
RuntimeError: torchaudio is required. Install with: uv pip install torchaudio
|
||
```
|
||
|
||
### Step 3 — MiMoASR 装好（关键三个坑）
|
||
|
||
按 [MiMoASR 自己的 README](https://github.com/aadebuger/ComfyUI-MiMoASR#install-the-proven-order) 走。三个**必踩**的坑：
|
||
|
||
#### 3.1 `transformers` 必须降到 `4.49.0`
|
||
|
||
```bash
|
||
$ uv pip freeze > ~/pip_freeze_before_mimo_$(date +%Y%m%d).txt # 先备份！
|
||
$ uv pip install "transformers==4.49.0" "tokenizers>=0.21,<0.22" --force-reinstall
|
||
$ python -c "import transformers; assert transformers.__version__ == '4.49.0'; print('OK')"
|
||
```
|
||
|
||
> ⚠️ 这次降级会**破坏** `Sonic`、`Qwen3TTS`、新版 `VibeVoice` 等需要 `transformers>=4.57` 的节点。
|
||
> 没办法同居 —— 要么牺牲那些节点，要么开一个独立的 ComfyUI 实例专门跑 MiMo。
|
||
|
||
#### 3.2 `flash-attn` 必装，没 SDPA 回退
|
||
|
||
```bash
|
||
# 看 torch 版本决定走 wheel 还是源码
|
||
$ python -c "import torch; print(torch.__version__)"
|
||
```
|
||
|
||
**A. 你的 torch 是 `2.6.0 + cu12 + Python 3.12`** → 直接装 wheel：
|
||
|
||
```bash
|
||
$ uv pip install flash-attn==2.7.4.post1
|
||
```
|
||
|
||
**B. 你的 torch 是 2.11（实战环境）** → 从源码编译，约 20–40 分钟：
|
||
|
||
```bash
|
||
$ uv pip install ninja packaging
|
||
$ export CUDA_HOME=/usr/local/cuda-12.6
|
||
$ export TORCH_CUDA_ARCH_LIST="8.6" # 3090=8.6 4090=8.9 H100=9.0
|
||
$ export MAX_JOBS=4 # 64GB RAM 可以开 8，OOM 就降到 2
|
||
$ nohup uv pip install flash-attn --no-build-isolation \
|
||
> ~/flash_attn_build.log 2>&1 &
|
||
$ tail -f ~/flash_attn_build.log
|
||
```
|
||
|
||
**验证（必做）**：
|
||
|
||
```bash
|
||
$ python -c "from flash_attn import flash_attn_varlen_func; print(flash_attn_varlen_func)"
|
||
# 期望：<function flash_attn_varlen_func at 0x...>
|
||
# 如果是 None 或 ImportError → 编译失败，看 ~/flash_attn_build.log
|
||
```
|
||
|
||
如果这里漏了验证，后面会以 `TypeError: 'NoneType' object is not callable` 神秘报错。
|
||
|
||
#### 3.3 必须把 MiMo 上游 repo 克隆到磁盘
|
||
|
||
`MimoAudio` 不能 pip 装，loader 会 `sys.path.insert`：
|
||
|
||
```bash
|
||
$ mkdir -p /opt/repos && cd /opt/repos
|
||
$ git clone https://github.com/XiaomiMiMo/MiMo-V2.5-ASR.git
|
||
$ ls /opt/repos/MiMo-V2.5-ASR/src/mimo_audio/
|
||
# 必须能列出 .py 文件。如果是空目录：
|
||
$ cd /opt/repos/MiMo-V2.5-ASR && git submodule update --init --recursive
|
||
```
|
||
|
||
### Step 4 — 下载 MiMo 权重（~24 GB）
|
||
|
||
```bash
|
||
# HF cache 必须重定向到大盘
|
||
$ export HF_HOME=/mnt/modelall/hf_cache
|
||
$ mkdir -p $HF_HOME
|
||
|
||
$ uv pip install -U huggingface-hub
|
||
|
||
# 主模型 ~16 GB
|
||
$ hf download XiaomiMiMo/MiMo-V2.5-ASR \
|
||
--local-dir /mnt/modelall/models/MiMo-V2.5-ASR
|
||
|
||
# Audio tokenizer ~1-2 GB
|
||
$ hf download XiaomiMiMo/MiMo-Audio-Tokenizer \
|
||
--local-dir /mnt/modelall/models/MiMo-Audio-Tokenizer
|
||
|
||
# 验证
|
||
$ ls -lh /mnt/modelall/models/MiMo-V2.5-ASR/*.safetensors
|
||
$ ls -lh /mnt/modelall/models/MiMo-Audio-Tokenizer/
|
||
```
|
||
|
||
国内下载慢可加 HF 镜像：
|
||
|
||
```bash
|
||
$ export HF_ENDPOINT=https://hf-mirror.com
|
||
$ hf download XiaomiMiMo/MiMo-V2.5-ASR --local-dir /mnt/modelall/models/MiMo-V2.5-ASR
|
||
```
|
||
|
||
### Step 5 — 装本包
|
||
|
||
```bash
|
||
$ cd /mnt/modelall/ComfyUI/custom_nodes
|
||
$ git clone https://github.com/aadebuger/ComfyUI-VideoToText.git
|
||
|
||
# OR 从 tarball 装：
|
||
$ cd /mnt/modelall/ComfyUI/custom_nodes
|
||
$ tar xzf /tmp/ComfyUI-VideoToText-v0.2.0.tar.gz
|
||
|
||
# 验证目录结构（关键！不能多嵌套一层）
|
||
$ ls ComfyUI-VideoToText/
|
||
# 必须直接看到 __init__.py / nodes/ / cli/ ...
|
||
|
||
# 如果是 ComfyUI-VideoToText/ComfyUI-VideoToText/__init__.py（嵌套了），扁平化：
|
||
$ cd ComfyUI-VideoToText
|
||
$ mv ComfyUI-VideoToText/* .
|
||
$ mv ComfyUI-VideoToText/.* . 2>/dev/null
|
||
$ rmdir ComfyUI-VideoToText
|
||
```
|
||
|
||
### Step 6 — 装 `rgthree-comfy`（如果还没装）
|
||
|
||
CLI 用 `PreviewAny` 抓 STRING 输出，必须有：
|
||
|
||
```bash
|
||
$ cd /mnt/modelall/ComfyUI/custom_nodes
|
||
$ git clone https://github.com/rgthree/rgthree-comfy.git
|
||
```
|
||
|
||
### Step 7 — 启动 ComfyUI（三个 env var）
|
||
|
||
```bash
|
||
$ pkill -9 -f main.py; sleep 3
|
||
$ cd /mnt/modelall/ComfyUI && source .venv/bin/activate
|
||
|
||
$ export HF_HOME=/mnt/modelall/hf_cache
|
||
$ export MIMO_ASR_REPO=/opt/repos/MiMo-V2.5-ASR
|
||
# 可选：把 MiMo 绑到 GPU 1，让 GPU 0 留给图像模型
|
||
# $ export CUDA_VISIBLE_DEVICES=1
|
||
|
||
$ nohup python main.py --listen 0.0.0.0 --port 8188 > ~/comfy_start.log 2>&1 &
|
||
$ sleep 30
|
||
|
||
# 看启动日志
|
||
$ grep -iE "IMPORT FAILED|VideoToText|MiMo|error" ~/comfy_start.log | head -20
|
||
```
|
||
|
||
### Step 8 — 验证 9 个节点都在
|
||
|
||
```bash
|
||
$ curl -s http://SERVER:8188/object_info | python3 -c "
|
||
import json, sys
|
||
d = json.load(sys.stdin)
|
||
for n in ('VTT_LoadVideo','VTT_ExtractAudio','VTT_AudioFromPath',
|
||
'VTT_BuildTranscriptJSON','VTT_SaveTranscript',
|
||
'MiMoASRLoader','MiMoASRTranscribe','PreviewAny','LoadAudio'):
|
||
print(f' {n}: {\"OK\" if n in d else \"MISSING\"}')"
|
||
```
|
||
|
||
9 个全 OK 才算 ready。
|
||
|
||
### Step 9 — 跑一个 CLI 测试
|
||
|
||
```bash
|
||
$ cd /mnt/modelall/ComfyUI/custom_nodes/ComfyUI-VideoToText/cli
|
||
$ uv pip install --system websocket-client requests
|
||
|
||
$ uv run python video_to_text_cli.py /path/to/test.mp4 \
|
||
--ip 10.8.0.12 \
|
||
--model-path /mnt/modelall/models/MiMo-V2.5-ASR \
|
||
--tokenizer-path /mnt/modelall/models/MiMo-Audio-Tokenizer \
|
||
--repo-path /opt/repos/MiMo-V2.5-ASR \
|
||
--language Auto
|
||
```
|
||
|
||
第一次跑会卡 30–60s（MiMo 8B 模型进 VRAM），之后秒级。
|
||
|
||
---
|
||
|
||
## 常见失败模式 / Common Failures
|
||
|
||
| Symptom | Cause | Fix |
|
||
| --- | --- | --- |
|
||
| `VTT_*: MISSING` 在 `/object_info` | 包没装上 / 嵌套了 / ComfyUI 没重启 | 走完 Step 5 + 重启 |
|
||
| `IMPORT FAILED ComfyUI-VideoToText` 在 startup log | Python import 错 | 看 traceback；通常是 `No module named 'folder_paths'`（包不在 `custom_nodes/` 里）|
|
||
| `MiMoASRLoader: MISSING` | MiMoASR 装失败 | 看 MiMoASR 的 INSTALL_NOTES |
|
||
| `AttributeError: ... 'all_tied_weights_keys'` | `transformers` 太新 | 降到 `4.49.0`，见 Step 3.1 |
|
||
| `TypeError: 'NoneType' object is not callable` 在 `flash_attn_varlen_func` | `flash-attn` 编译静默失败 | 重新编，看 `~/flash_attn_build.log`，见 Step 3.2 |
|
||
| `RuntimeError: MiMo ASR repo path is empty` | `MIMO_ASR_REPO` 没设 | 启动前 `$ export MIMO_ASR_REPO=...` |
|
||
| `RuntimeError: ffmpeg not found on PATH` | ffmpeg 没装 | `$ sudo apt install ffmpeg` |
|
||
| `RuntimeError: torchaudio is required` | torchaudio 没装 | `$ uv pip install torchaudio` |
|
||
| `model path does not exist` | 工作流里填了相对路径 | 改成绝对路径 |
|
||
| CUDA OOM 在 transcribe | GPU 0 被图像模型占了 | `$ export CUDA_VISIBLE_DEVICES=1` 再启动 |
|
||
| 启动后再次验证还是 MISSING | 重启没生效 | `$ ps aux \| grep main.py` 确认 PID + `$ stat -c '%y' ~/comfy_start.log` 看时间戳 |
|
||
|
||
---
|
||
|
||
## VRAM 预算
|
||
|
||
| | VRAM |
|
||
| --- | --- |
|
||
| MiMo-V2.5-ASR load (bf16) | ~17 GB |
|
||
| MiMo 推理峰值 | ~20–22 GB |
|
||
| 本包节点 | 可忽略（ffmpeg + torchaudio buffer） |
|
||
|
||
24 GB GPU 够 MiMo + 轻度 ComfyUI 共用。如果你还在同一张卡跑 SDXL / Z-Image-Turbo，**强烈建议** `$ export CUDA_VISIBLE_DEVICES=1` 把 MiMo 隔离到另一张卡。
|
||
|
||
---
|
||
|
||
## 清理 / Cleanup
|
||
|
||
```bash
|
||
# 音频缓存（VTT_ExtractAudio 写到 $TMPDIR/comfyui_vtt/）
|
||
$ rm -rf /tmp/comfyui_vtt
|
||
|
||
# ComfyUI output（含已保存的 transcript JSON）
|
||
$ ls /mnt/modelall/ComfyUI/output/transcript_*.json
|
||
|
||
# MiMo 模型（占 ~24GB，慎删）
|
||
# $ rm -rf /mnt/modelall/models/MiMo-V2.5-ASR
|
||
# $ rm -rf /mnt/modelall/models/MiMo-Audio-Tokenizer
|
||
|
||
# 卸载本包
|
||
$ rm -rf /mnt/modelall/ComfyUI/custom_nodes/ComfyUI-VideoToText
|
||
# 重启 ComfyUI
|
||
```
|