# ComfyUI-VideoToText

> Video → audio → JSON transcript pipeline for ComfyUI. Pairs with [**ComfyUI-MiMoASR**](https://github.com/aadebuger/ComfyUI-MiMoASR) (Xiaomi MiMo-V2.5-ASR) for the ASR step.

This pack provides the **non-ASR** pieces of a video-to-text workflow: video loading, audio extraction (`ffmpeg`), `AUDIO` dict bridging, JSON packaging, and disk persistence. The ASR itself is delegated to MiMoASR, which gives you strong Mandarin / English code-switching, dialect support (Wu, Cantonese, Hokkien, Sichuanese, Henan, Northeastern Mandarin), and noisy-audio robustness.

| Node | Purpose |
| --- | --- |
| 🎙️ **Load Video (ASR)** | Pick a video from `ComfyUI/input/` or pass an absolute path. |
| 🎙️ **Extract Audio (ffmpeg)** | Demux audio track to a 16 kHz mono WAV. On-disk cache keyed by mtime + params. |
| 🎙️ **Audio From Path** | Bridge: WAV file → ComfyUI `AUDIO` dict. Optional resample / mono. Also reports duration. |
| 🎙️ **Build Transcript JSON** | Wrap ASR text + source metadata into a structured v2.0 JSON. |
| 🎙️ **Save Transcript JSON** | Validate + write JSON to `ComfyUI/output/` with timestamped filename. |

---

## ⚠️ Read this before installing

This pack does **not** ship an ASR model. Three things must be true before you can run a workflow:

1. **[ComfyUI-MiMoASR](https://github.com/aadebuger/ComfyUI-MiMoASR) installed and working** — including its `transformers==4.49.0` downgrade and `flash-attn` build. Verify those nodes register before you even start with this pack.
2. **`ffmpeg` on your server's `PATH`** — `$ which ffmpeg` must return a path.
3. **`torchaudio` in ComfyUI's venv** — installed in Step 2 below.

Quick gut check:

| | Required | How to check |
| --- | --- | --- |
| `ffmpeg` | any recent | `$ ffmpeg -version` |
| `torchaudio` | any | `$ python -c "import torchaudio; print(torchaudio.__version__)"` |
| ComfyUI-MiMoASR | installed | `$ curl -s http://SERVER:8188/object_info \| grep -o MiMoASRLoader` |
| `PreviewAny` (from rgthree-comfy) | installed | `$ curl -s http://SERVER:8188/object_info \| grep -o PreviewAny` |

---

## Install (the proven order)

Each step has a verification at the end. Don't skip the verifications.

### 1. Install MiMoASR first

This pack depends on `MiMoASRLoader` and `MiMoASRTranscribe` from a separate repo. Follow [its full install guide](https://github.com/aadebuger/ComfyUI-MiMoASR#install-the-proven-order) before continuing here. **Stop and verify**:

```bash
$ curl -s http://SERVER:8188/object_info | python3 -c "
import json, sys
d = json.load(sys.stdin)
for n in ('MiMoASRLoader', 'MiMoASRTranscribe'):
    print(f'  {n}: {\"OK\" if n in d else \"MISSING\"}')"
```

Both must print `OK`. If `MISSING`, fix MiMoASR before continuing.

### 2. Clone this node pack

```bash
$ cd ComfyUI/custom_nodes
$ git clone https://github.com/aadebuger/ComfyUI-VideoToText.git
```

Or, if installing from a tarball release:

```bash
$ cd ComfyUI/custom_nodes
$ tar xzf /path/to/ComfyUI-VideoToText-v0.2.0.tar.gz
```

Verify:

```bash
$ ls ComfyUI/custom_nodes/ComfyUI-VideoToText/__init__.py
# Must exist. If it's nested inside ComfyUI-VideoToText/ComfyUI-VideoToText/, you have a double-wrap — flatten it.
```

### 3. Install Python deps (in ComfyUI's venv)

```bash
$ cd ComfyUI && source .venv/bin/activate
$ uv pip install torchaudio
```

Verify:

```bash
$ python -c "import torchaudio; print('torchaudio', torchaudio.__version__)"
# Must print a version string. Errors here → re-run install.
```

### 4. Install `ffmpeg` on the server (if missing)

```bash
$ which ffmpeg || sudo apt install ffmpeg
$ ffmpeg -version
```

### 5. Restart ComfyUI

The same launch incantation MiMoASR needs — both `HF_HOME` and `MIMO_ASR_REPO` env vars must be set:

```bash
$ pkill -9 -f main.py; sleep 3
$ cd ComfyUI && source .venv/bin/activate
$ export HF_HOME=/big_disk/hf_cache
$ export MIMO_ASR_REPO=/opt/repos/MiMo-V2.5-ASR
$ nohup python main.py --listen 0.0.0.0 --port 8188 > ~/comfy_start.log 2>&1 &
$ sleep 30
$ grep -iE "IMPORT FAILED|VideoToText|MiMo" ~/comfy_start.log | head -20
```

### 6. Verify all 9 required nodes are registered

```bash
$ curl -s http://SERVER:8188/object_info | python3 -c "
import json, sys
d = json.load(sys.stdin)
for n in ('VTT_LoadVideo','VTT_ExtractAudio','VTT_AudioFromPath',
          'VTT_BuildTranscriptJSON','VTT_SaveTranscript',
          'MiMoASRLoader','MiMoASRTranscribe','PreviewAny','LoadAudio'):
    print(f'  {n}: {\"OK\" if n in d else \"MISSING\"}')"
```

All 9 should show `OK`. If `PreviewAny` is missing, install [rgthree-comfy](https://github.com/rgthree/rgthree-comfy).

---

## Usage

### Minimal workflow

```
VTT_LoadVideo  →  VTT_ExtractAudio  →  VTT_AudioFromPath  ─┐
                                                            ├──►  MiMoASRTranscribe  →  text
                                       MiMoASRLoader ───────┘            │
                                                                         ▼
                                                          VTT_BuildTranscriptJSON  →  STRING
                                                                         │
                                                                         ▼
                                                                VTT_SaveTranscript  →  ComfyUI/output/*.json
```

**Required fields**:

- **MiMoASRLoader** (absolute paths!):
  - `model_path` = `/big_disk/models/MiMo-V2.5-ASR`
  - `tokenizer_path` = `/big_disk/models/MiMo-Audio-Tokenizer`
  - `repo_path` = leave blank if `MIMO_ASR_REPO` env var is set
- **MiMoASRTranscribe**:
  - `language` = `Auto` (code-switched), `Chinese`, or `English`

The first run loads the 8B MiMo model into VRAM (30–60s). Subsequent runs in the same ComfyUI session are instant — MiMo's loader has a process-wide cache.

An API-format example workflow ships at [`example_workflows/video_to_json.json`](example_workflows/video_to_json.json). Edit the three MiMo paths to match your server before loading it in the UI.

### CLI

A single-file CLI uploads a local video, drives the workflow over ComfyUI's HTTP + WebSocket API, and dumps the JSON transcript locally:

```bash
$ uv pip install --system websocket-client requests

$ uv run python cli/video_to_text_cli.py /path/to/lecture.mp4 \
    --ip 10.8.0.12 \
    --model-path /big_disk/models/MiMo-V2.5-ASR \
    --tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer \
    --repo-path /opt/repos/MiMo-V2.5-ASR \
    --language Auto
```

A wrapper alias to skip retyping paths every time:

```bash
# ~/.bashrc
alias v2t='uv run python /path/to/ComfyUI-VideoToText/cli/video_to_text_cli.py \
    --ip 10.8.0.12 \
    --model-path /big_disk/models/MiMo-V2.5-ASR \
    --tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer \
    --repo-path /opt/repos/MiMo-V2.5-ASR'

# Then:
$ v2t my_video.mp4 --language Chinese
$ v2t lecture.mp4 --language Auto --out ./transcripts
```

See [`cli/README.md`](cli/README.md) for all flags.

---

## JSON schema v2.0

```json
{
  "schema_version": "2.0",
  "source": {
    "audio_path": "/tmp/comfyui_vtt/lecture_abc123.wav",
    "duration_sec": 1834.512,
    "video_path": "/path/to/lecture.mp4"
  },
  "model": {
    "name": "MiMo-V2.5-ASR",
    "tokenizer": "MiMo-Audio-Tokenizer",
    "language": "Auto"
  },
  "text": "你好，今天我们聊一下 ESP32 嵌入式开发……"
}
```

### Trade-off vs faster-whisper

MiMo-V2.5-ASR returns a **single plain-text transcript** — no word/segment timestamps, no confidence scores. Schema v2.0 reflects this: top-level `text` only, no `segments[]`. If you need time-aligned subtitles, run a different ASR backend (or wait for the planned v0.3.0 hybrid mode that pairs MiMo for text quality with faster-whisper for time alignment).

---

## Tips

- **Pure-language audio**: setting `language=Chinese` or `English` is slightly more accurate than `Auto`. Use `Auto` only for code-switched content.
- **GPU pinning**: on dual-GPU rigs, `$ export CUDA_VISIBLE_DEVICES=1` before launching ComfyUI keeps MiMo from fighting image models on `cuda:0`.
- **Audio cache**: `VTT_ExtractAudio` writes to `$TMPDIR/comfyui_vtt/` keyed by source mtime + sample rate + mono flag. Re-runs on the same video skip `ffmpeg` entirely. Nuke with `$ rm -rf /tmp/comfyui_vtt`.
- **Long audio**: MiMo handles multi-hour lectures in a single call. No manual chunking needed at typical podcast / lecture lengths. For 6h+ recordings, consider splitting at silence with `ffmpeg`.

---

## Troubleshooting

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| `VTT_*` nodes show `MISSING` in `/object_info` | Pack not loaded; ComfyUI didn't restart; or `IMPORT FAILED` | Check `~/comfy_start.log` for `IMPORT FAILED ComfyUI-VideoToText` traceback |
| `MiMoASRLoader: MISSING` | MiMoASR not installed | See [its README](https://github.com/aadebuger/ComfyUI-MiMoASR) |
| `PreviewAny: MISSING` | rgthree-comfy not installed | `$ git clone https://github.com/rgthree/rgthree-comfy.git` into `custom_nodes/` |
| `RuntimeError: ffmpeg not found on PATH` | ffmpeg missing on the server | `$ sudo apt install ffmpeg` |
| `RuntimeError: torchaudio is required` | torchaudio missing in ComfyUI's venv | `$ uv pip install torchaudio` in the right venv |
| `RuntimeError: MiMo ASR repo path is empty` | `MIMO_ASR_REPO` env var not set | Set it before launching ComfyUI, OR pass `repo_path` in the workflow |
| MiMoASRLoader: "model path does not exist" | Relative path in the workflow | Always use absolute paths |
| `IMPORT FAILED ComfyUI-VideoToText` with `No module named 'folder_paths'` | Pack at wrong location | Must be in `ComfyUI/custom_nodes/`, not elsewhere |
| CUDA OOM on transcribe | GPU 0 held by image models | `$ export CUDA_VISIBLE_DEVICES=1` before launching ComfyUI |

See [`INSTALL_NOTES.md`](INSTALL_NOTES.md) for the full debug log of real-world install gotchas.

---

## Compatibility

| | Required | Tested with |
| --- | --- | --- |
| Python | 3.10+ | 3.13 |
| PyTorch | matches ComfyUI's | 2.11.0+cu126 |
| `torchaudio` | any | matches torch |
| `transformers` | **4.49.0** (MiMo pin) | 4.49.0 |
| CUDA | ≥12.0 | 12.6 |
| GPU VRAM | 24 GB (MiMo needs it) | 2× RTX 3090 |
| OS | Linux | Ubuntu 22.04 |

> ⚠️ The `transformers==4.49.0` pin (required by MiMo) will break `Sonic`, `Qwen3TTS`, and recent `VibeVoice` builds — they want `>=4.57`. If you need both, run MiMo + VideoToText on a dedicated ComfyUI instance.

---

## Links

- 🤗 MiMo-V2.5-ASR weights: <https://huggingface.co/XiaomiMiMo/MiMo-V2.5-ASR>
- 🤗 MiMo-Audio-Tokenizer: <https://huggingface.co/XiaomiMiMo/MiMo-Audio-Tokenizer>
- MiMo upstream repo: <https://github.com/XiaomiMiMo/MiMo-V2.5-ASR>
- MiMo ComfyUI nodes: <https://github.com/aadebuger/ComfyUI-MiMoASR>
- Blog: <https://mimo.xiaomi.com/mimo-v2-5-asr>

---

## License

MIT — see [LICENSE](LICENSE). MiMoASR upstream is Apache-2.0; MiMo-V2.5-ASR weights are MIT (Xiaomi).