269 lines
11 KiB
Markdown
269 lines
11 KiB
Markdown
# ComfyUI-VideoToText
|
||
|
||
> Video → audio → JSON transcript pipeline for ComfyUI. Pairs with [**ComfyUI-MiMoASR**](https://github.com/aadebuger/ComfyUI-MiMoASR) (Xiaomi MiMo-V2.5-ASR) for the ASR step.
|
||
|
||
This pack provides the **non-ASR** pieces of a video-to-text workflow: video loading, audio extraction (`ffmpeg`), `AUDIO` dict bridging, JSON packaging, and disk persistence. The ASR itself is delegated to MiMoASR, which gives you strong Mandarin / English code-switching, dialect support (Wu, Cantonese, Hokkien, Sichuanese, Henan, Northeastern Mandarin), and noisy-audio robustness.
|
||
|
||
| Node | Purpose |
|
||
| --- | --- |
|
||
| 🎙️ **Load Video (ASR)** | Pick a video from `ComfyUI/input/` or pass an absolute path. |
|
||
| 🎙️ **Extract Audio (ffmpeg)** | Demux audio track to a 16 kHz mono WAV. On-disk cache keyed by mtime + params. |
|
||
| 🎙️ **Audio From Path** | Bridge: WAV file → ComfyUI `AUDIO` dict. Optional resample / mono. Also reports duration. |
|
||
| 🎙️ **Build Transcript JSON** | Wrap ASR text + source metadata into a structured v2.0 JSON. |
|
||
| 🎙️ **Save Transcript JSON** | Validate + write JSON to `ComfyUI/output/` with timestamped filename. |
|
||
|
||
---
|
||
|
||
## ⚠️ Read this before installing
|
||
|
||
This pack does **not** ship an ASR model. Three things must be true before you can run a workflow:
|
||
|
||
1. **[ComfyUI-MiMoASR](https://github.com/aadebuger/ComfyUI-MiMoASR) installed and working** — including its `transformers==4.49.0` downgrade and `flash-attn` build. Verify those nodes register before you even start with this pack.
|
||
2. **`ffmpeg` on your server's `PATH`** — `$ which ffmpeg` must return a path.
|
||
3. **`torchaudio` in ComfyUI's venv** — installed in Step 2 below.
|
||
|
||
Quick gut check:
|
||
|
||
| | Required | How to check |
|
||
| --- | --- | --- |
|
||
| `ffmpeg` | any recent | `$ ffmpeg -version` |
|
||
| `torchaudio` | any | `$ python -c "import torchaudio; print(torchaudio.__version__)"` |
|
||
| ComfyUI-MiMoASR | installed | `$ curl -s http://SERVER:8188/object_info \| grep -o MiMoASRLoader` |
|
||
| `PreviewAny` (from rgthree-comfy) | installed | `$ curl -s http://SERVER:8188/object_info \| grep -o PreviewAny` |
|
||
|
||
---
|
||
|
||
## Install (the proven order)
|
||
|
||
Each step has a verification at the end. Don't skip the verifications.
|
||
|
||
### 1. Install MiMoASR first
|
||
|
||
This pack depends on `MiMoASRLoader` and `MiMoASRTranscribe` from a separate repo. Follow [its full install guide](https://github.com/aadebuger/ComfyUI-MiMoASR#install-the-proven-order) before continuing here. **Stop and verify**:
|
||
|
||
```bash
|
||
$ curl -s http://SERVER:8188/object_info | python3 -c "
|
||
import json, sys
|
||
d = json.load(sys.stdin)
|
||
for n in ('MiMoASRLoader', 'MiMoASRTranscribe'):
|
||
print(f' {n}: {\"OK\" if n in d else \"MISSING\"}')"
|
||
```
|
||
|
||
Both must print `OK`. If `MISSING`, fix MiMoASR before continuing.
|
||
|
||
### 2. Clone this node pack
|
||
|
||
```bash
|
||
$ cd ComfyUI/custom_nodes
|
||
$ git clone https://github.com/aadebuger/ComfyUI-VideoToText.git
|
||
```
|
||
|
||
Or, if installing from a tarball release:
|
||
|
||
```bash
|
||
$ cd ComfyUI/custom_nodes
|
||
$ tar xzf /path/to/ComfyUI-VideoToText-v0.2.0.tar.gz
|
||
```
|
||
|
||
Verify:
|
||
|
||
```bash
|
||
$ ls ComfyUI/custom_nodes/ComfyUI-VideoToText/__init__.py
|
||
# Must exist. If it's nested inside ComfyUI-VideoToText/ComfyUI-VideoToText/, you have a double-wrap — flatten it.
|
||
```
|
||
|
||
### 3. Install Python deps (in ComfyUI's venv)
|
||
|
||
```bash
|
||
$ cd ComfyUI && source .venv/bin/activate
|
||
$ uv pip install torchaudio
|
||
```
|
||
|
||
Verify:
|
||
|
||
```bash
|
||
$ python -c "import torchaudio; print('torchaudio', torchaudio.__version__)"
|
||
# Must print a version string. Errors here → re-run install.
|
||
```
|
||
|
||
### 4. Install `ffmpeg` on the server (if missing)
|
||
|
||
```bash
|
||
$ which ffmpeg || sudo apt install ffmpeg
|
||
$ ffmpeg -version
|
||
```
|
||
|
||
### 5. Restart ComfyUI
|
||
|
||
The same launch incantation MiMoASR needs — both `HF_HOME` and `MIMO_ASR_REPO` env vars must be set:
|
||
|
||
```bash
|
||
$ pkill -9 -f main.py; sleep 3
|
||
$ cd ComfyUI && source .venv/bin/activate
|
||
$ export HF_HOME=/big_disk/hf_cache
|
||
$ export MIMO_ASR_REPO=/opt/repos/MiMo-V2.5-ASR
|
||
$ nohup python main.py --listen 0.0.0.0 --port 8188 > ~/comfy_start.log 2>&1 &
|
||
$ sleep 30
|
||
$ grep -iE "IMPORT FAILED|VideoToText|MiMo" ~/comfy_start.log | head -20
|
||
```
|
||
|
||
### 6. Verify all 9 required nodes are registered
|
||
|
||
```bash
|
||
$ curl -s http://SERVER:8188/object_info | python3 -c "
|
||
import json, sys
|
||
d = json.load(sys.stdin)
|
||
for n in ('VTT_LoadVideo','VTT_ExtractAudio','VTT_AudioFromPath',
|
||
'VTT_BuildTranscriptJSON','VTT_SaveTranscript',
|
||
'MiMoASRLoader','MiMoASRTranscribe','PreviewAny','LoadAudio'):
|
||
print(f' {n}: {\"OK\" if n in d else \"MISSING\"}')"
|
||
```
|
||
|
||
All 9 should show `OK`. If `PreviewAny` is missing, install [rgthree-comfy](https://github.com/rgthree/rgthree-comfy).
|
||
|
||
---
|
||
|
||
## Usage
|
||
|
||
### Minimal workflow
|
||
|
||
```
|
||
VTT_LoadVideo → VTT_ExtractAudio → VTT_AudioFromPath ─┐
|
||
├──► MiMoASRTranscribe → text
|
||
MiMoASRLoader ───────┘ │
|
||
▼
|
||
VTT_BuildTranscriptJSON → STRING
|
||
│
|
||
▼
|
||
VTT_SaveTranscript → ComfyUI/output/*.json
|
||
```
|
||
|
||
**Required fields**:
|
||
|
||
- **MiMoASRLoader** (absolute paths!):
|
||
- `model_path` = `/big_disk/models/MiMo-V2.5-ASR`
|
||
- `tokenizer_path` = `/big_disk/models/MiMo-Audio-Tokenizer`
|
||
- `repo_path` = leave blank if `MIMO_ASR_REPO` env var is set
|
||
- **MiMoASRTranscribe**:
|
||
- `language` = `Auto` (code-switched), `Chinese`, or `English`
|
||
|
||
The first run loads the 8B MiMo model into VRAM (30–60s). Subsequent runs in the same ComfyUI session are instant — MiMo's loader has a process-wide cache.
|
||
|
||
An API-format example workflow ships at [`example_workflows/video_to_json.json`](example_workflows/video_to_json.json). Edit the three MiMo paths to match your server before loading it in the UI.
|
||
|
||
### CLI
|
||
|
||
A single-file CLI uploads a local video, drives the workflow over ComfyUI's HTTP + WebSocket API, and dumps the JSON transcript locally:
|
||
|
||
```bash
|
||
$ uv pip install --system websocket-client requests
|
||
|
||
$ uv run python cli/video_to_text_cli.py /path/to/lecture.mp4 \
|
||
--ip 10.8.0.12 \
|
||
--model-path /big_disk/models/MiMo-V2.5-ASR \
|
||
--tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer \
|
||
--repo-path /opt/repos/MiMo-V2.5-ASR \
|
||
--language Auto
|
||
```
|
||
|
||
A wrapper alias to skip retyping paths every time:
|
||
|
||
```bash
|
||
# ~/.bashrc
|
||
alias v2t='uv run python /path/to/ComfyUI-VideoToText/cli/video_to_text_cli.py \
|
||
--ip 10.8.0.12 \
|
||
--model-path /big_disk/models/MiMo-V2.5-ASR \
|
||
--tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer \
|
||
--repo-path /opt/repos/MiMo-V2.5-ASR'
|
||
|
||
# Then:
|
||
$ v2t my_video.mp4 --language Chinese
|
||
$ v2t lecture.mp4 --language Auto --out ./transcripts
|
||
```
|
||
|
||
See [`cli/README.md`](cli/README.md) for all flags.
|
||
|
||
---
|
||
|
||
## JSON schema v2.0
|
||
|
||
```json
|
||
{
|
||
"schema_version": "2.0",
|
||
"source": {
|
||
"audio_path": "/tmp/comfyui_vtt/lecture_abc123.wav",
|
||
"duration_sec": 1834.512,
|
||
"video_path": "/path/to/lecture.mp4"
|
||
},
|
||
"model": {
|
||
"name": "MiMo-V2.5-ASR",
|
||
"tokenizer": "MiMo-Audio-Tokenizer",
|
||
"language": "Auto"
|
||
},
|
||
"text": "你好，今天我们聊一下 ESP32 嵌入式开发……"
|
||
}
|
||
```
|
||
|
||
### Trade-off vs faster-whisper
|
||
|
||
MiMo-V2.5-ASR returns a **single plain-text transcript** — no word/segment timestamps, no confidence scores. Schema v2.0 reflects this: top-level `text` only, no `segments[]`. If you need time-aligned subtitles, run a different ASR backend (or wait for the planned v0.3.0 hybrid mode that pairs MiMo for text quality with faster-whisper for time alignment).
|
||
|
||
---
|
||
|
||
## Tips
|
||
|
||
- **Pure-language audio**: setting `language=Chinese` or `English` is slightly more accurate than `Auto`. Use `Auto` only for code-switched content.
|
||
- **GPU pinning**: on dual-GPU rigs, `$ export CUDA_VISIBLE_DEVICES=1` before launching ComfyUI keeps MiMo from fighting image models on `cuda:0`.
|
||
- **Audio cache**: `VTT_ExtractAudio` writes to `$TMPDIR/comfyui_vtt/` keyed by source mtime + sample rate + mono flag. Re-runs on the same video skip `ffmpeg` entirely. Nuke with `$ rm -rf /tmp/comfyui_vtt`.
|
||
- **Long audio**: MiMo handles multi-hour lectures in a single call. No manual chunking needed at typical podcast / lecture lengths. For 6h+ recordings, consider splitting at silence with `ffmpeg`.
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
| Symptom | Likely cause | Fix |
|
||
| --- | --- | --- |
|
||
| `VTT_*` nodes show `MISSING` in `/object_info` | Pack not loaded; ComfyUI didn't restart; or `IMPORT FAILED` | Check `~/comfy_start.log` for `IMPORT FAILED ComfyUI-VideoToText` traceback |
|
||
| `MiMoASRLoader: MISSING` | MiMoASR not installed | See [its README](https://github.com/aadebuger/ComfyUI-MiMoASR) |
|
||
| `PreviewAny: MISSING` | rgthree-comfy not installed | `$ git clone https://github.com/rgthree/rgthree-comfy.git` into `custom_nodes/` |
|
||
| `RuntimeError: ffmpeg not found on PATH` | ffmpeg missing on the server | `$ sudo apt install ffmpeg` |
|
||
| `RuntimeError: torchaudio is required` | torchaudio missing in ComfyUI's venv | `$ uv pip install torchaudio` in the right venv |
|
||
| `RuntimeError: MiMo ASR repo path is empty` | `MIMO_ASR_REPO` env var not set | Set it before launching ComfyUI, OR pass `repo_path` in the workflow |
|
||
| MiMoASRLoader: "model path does not exist" | Relative path in the workflow | Always use absolute paths |
|
||
| `IMPORT FAILED ComfyUI-VideoToText` with `No module named 'folder_paths'` | Pack at wrong location | Must be in `ComfyUI/custom_nodes/`, not elsewhere |
|
||
| CUDA OOM on transcribe | GPU 0 held by image models | `$ export CUDA_VISIBLE_DEVICES=1` before launching ComfyUI |
|
||
|
||
See [`INSTALL_NOTES.md`](INSTALL_NOTES.md) for the full debug log of real-world install gotchas.
|
||
|
||
---
|
||
|
||
## Compatibility
|
||
|
||
| | Required | Tested with |
|
||
| --- | --- | --- |
|
||
| Python | 3.10+ | 3.13 |
|
||
| PyTorch | matches ComfyUI's | 2.11.0+cu126 |
|
||
| `torchaudio` | any | matches torch |
|
||
| `transformers` | **4.49.0** (MiMo pin) | 4.49.0 |
|
||
| CUDA | ≥12.0 | 12.6 |
|
||
| GPU VRAM | 24 GB (MiMo needs it) | 2× RTX 3090 |
|
||
| OS | Linux | Ubuntu 22.04 |
|
||
|
||
> ⚠️ The `transformers==4.49.0` pin (required by MiMo) will break `Sonic`, `Qwen3TTS`, and recent `VibeVoice` builds — they want `>=4.57`. If you need both, run MiMo + VideoToText on a dedicated ComfyUI instance.
|
||
|
||
---
|
||
|
||
## Links
|
||
|
||
- 🤗 MiMo-V2.5-ASR weights: <https://huggingface.co/XiaomiMiMo/MiMo-V2.5-ASR>
|
||
- 🤗 MiMo-Audio-Tokenizer: <https://huggingface.co/XiaomiMiMo/MiMo-Audio-Tokenizer>
|
||
- MiMo upstream repo: <https://github.com/XiaomiMiMo/MiMo-V2.5-ASR>
|
||
- MiMo ComfyUI nodes: <https://github.com/aadebuger/ComfyUI-MiMoASR>
|
||
- Blog: <https://mimo.xiaomi.com/mimo-v2-5-asr>
|
||
|
||
---
|
||
|
||
## License
|
||
|
||
MIT — see [LICENSE](LICENSE). MiMoASR upstream is Apache-2.0; MiMo-V2.5-ASR weights are MIT (Xiaomi).
|