commit 7e0e1bf4fb8525fd9a0e332b5debe78ac5a9029f Author: hua zhuang <30758919@qq.com> Date: Fri May 22 09:33:52 2026 +0800 Initial commit: ComfyUI-VideoToText diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..460bd48 --- /dev/null +++ b/.gitignore @@ -0,0 +1,17 @@ +__pycache__/ +*.py[cod] +*$py.class +.venv/ +venv/ +*.egg-info/ +dist/ +build/ +out/ +*.tar.gz +*.zip +.DS_Store +.idea/ +.vscode/ +.lock +.pytest_cache/ +.ipynb_checkpoints/ diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..511e2d7 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,42 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [0.2.1] - 2026-05-22 + +### Added +- `scripts/install_server.sh` — one-shot idempotent server-side install (ffmpeg check + torchaudio install + smoke test). +- `README.zh-CN.md` — Chinese quick-start guide (shortest working path). +- `README.md` rewritten with explicit shell `$` prompts, ⚠️ pre-install warning block, step-by-step `Install (the proven order)` with per-step verification, and a Tested-with compatibility table — matching the ComfyUI-MiMoASR documentation style. +- `INSTALL_NOTES.md` reorganized as a 9-step debug log with the three MiMo install pitfalls called out (transformers downgrade, flash-attn build, upstream repo clone). + +### Changed +- `README.md` now uses 9 distinct verification steps. Each install step ends with a `Verify:` block. +- Troubleshooting tables now distinguish errors that surface at ComfyUI startup vs at workflow runtime (lazy-imports). + +## [0.2.0] - 2026-05-21 + +### Changed +- **BREAKING**: dropped the bundled `FasterWhisperTranscribe` node. ASR is now handled externally by [ComfyUI-MiMoASR](https://github.com/aadebuger/ComfyUI-MiMoASR) (`MiMoASRLoader` + `MiMoASRTranscribe`). +- **BREAKING**: JSON schema bumped to `2.0`. `segments[]`, per-segment `confidence`, `avg_logprob`, `no_speech_prob`, `compression_ratio`, and word-level timestamps are removed because MiMo's `asr_sft` does not expose them. Top-level `text` field now carries the full transcript. +- Dropped `faster-whisper` from `requirements.txt`. Added `torchaudio`. + +### Added +- `VTT_AudioFromPath` — bridge node: loads an arbitrary WAV/audio file (from absolute path) into ComfyUI's `AUDIO` dict so it can be fed into `MiMoASRTranscribe`. Optional `target_sample_rate` and `force_mono`. +- `VTT_BuildTranscriptJSON` — pack ASR text + source metadata into the v2.0 JSON schema. +- Example workflow now chains `VTT_*` nodes with `MiMoASRLoader` + `MiMoASRTranscribe`. + +### Removed +- `nodes/whisper_transcribe.py` and all references to faster-whisper. + +## [0.1.0] - 2026-05-21 + +### Added +- Four nodes: `VTT_LoadVideo`, `VTT_ExtractAudio`, `VTT_FasterWhisperTranscribe`, `VTT_SaveTranscript`. +- faster-whisper backend with per-segment confidence and word-level timestamps. +- Multi-GPU device selection; `_MODEL_CACHE` for warm model instances. +- ffmpeg-based audio extraction with on-disk caching. +- CLI client (`cli/video_to_text_cli.py`) using `PreviewAny` to pull outputs over WebSocket. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..427154a --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,44 @@ +# Contributing + +Thanks for your interest in improving ComfyUI-VideoToText. + +## Dev setup + +```bash +# In your ComfyUI install: +cd ComfyUI/custom_nodes +git clone https://github.com/aadebuger/ComfyUI-VideoToText.git +cd ComfyUI-VideoToText + +# Install deps into ComfyUI's venv +source ../../.venv/bin/activate # adjust to your ComfyUI venv path +uv pip install -r requirements.txt +``` + +Then restart ComfyUI and look for the `🎙️ VideoToText` category in the node search. + +## Reporting bugs + +Please include: + +- ComfyUI version (`git rev-parse HEAD` inside the ComfyUI repo) +- PyTorch + CUDA versions (`python -c "import torch; print(torch.__version__, torch.version.cuda)"`) +- `faster-whisper` version (`pip show faster-whisper`) +- `ffmpeg -version` output +- The minimal workflow JSON that reproduces the issue (use the CLI's `--dump-workflow` flag) +- Full stderr / ComfyUI console traceback + +## Pull requests + +- Keep nodes single-purpose. New node ideas: SRT/VTT exporter, transcript diff, translation, summarization. +- Don't reach into ComfyUI internals beyond `folder_paths`. +- Lazy-import heavy deps (`faster_whisper`, `torch`) inside method bodies, not at module top. +- Preserve the API contract for existing nodes — add new optional fields, don't rename required ones. +- Run the example workflow before opening the PR. + +## Style + +- Type hints encouraged but not enforced. +- f-strings, not `.format()`. +- 4-space indent, no tabs. +- One node per file under `nodes/`. diff --git a/INSTALL_NOTES.md b/INSTALL_NOTES.md new file mode 100644 index 0000000..0534562 --- /dev/null +++ b/INSTALL_NOTES.md @@ -0,0 +1,303 @@ +# Install Notes — 实战安装记录 + +Real-world install log for the video-to-text pipeline. 这是一份"踩坑笔记",按照实际部署顺序记录可能的问题和解法。Read this before opening an issue. + +## 架构 / Architecture + +ComfyUI-VideoToText 是个**包装层**,本身**不包含 ASR 模型**。完整链路: + +``` +your_video.mp4 + │ + ├─► [本包] VTT_LoadVideo + │ │ + │ └─► [本包] VTT_ExtractAudio (ffmpeg) + │ │ + │ └─► [本包] VTT_AudioFromPath (torchaudio) + │ │ + │ ├──────────────────┐ + │ ▼ ▼ + └────► [MiMoASR] MiMoASRLoader + MiMoASRTranscribe + │ + └─► [本包] VTT_BuildTranscriptJSON + │ + └─► [本包] VTT_SaveTranscript + │ + ▼ + ComfyUI/output/.json +``` + +依赖的两个外部组件: + +| 节点 | 来源 | 用途 | +| --- | --- | --- | +| `MiMoASRLoader`, `MiMoASRTranscribe` | [aadebuger/ComfyUI-MiMoASR](https://github.com/aadebuger/ComfyUI-MiMoASR) | ASR 核心 | +| `PreviewAny` | [rgthree/rgthree-comfy](https://github.com/rgthree/rgthree-comfy) | CLI 抓 STRING 输出用 | +| `LoadAudio` | ComfyUI 内置 | 节点 UI 端测试用 | + +## 测试环境 / Tested Environment + +| Item | Value | +| --- | --- | +| OS | Ubuntu 22.04 | +| GPU | 2× RTX 3090 (24 GB each) | +| CUDA toolkit | 12.6 (`/usr/local/cuda-12.6`) | +| Python | 3.13 | +| PyTorch | 2.11.0+cu126 | +| ComfyUI venv | `uv`-managed at `/mnt/modelall/ComfyUI/.venv` | +| HF cache | `/mnt/modelall/hf_cache` (system disk only had 10 GB → must redirect) | +| MiMo 上游 repo | `/opt/repos/MiMo-V2.5-ASR` (set via `MIMO_ASR_REPO` env var) | +| MiMo 模型 | `/big_disk/models/MiMo-V2.5-ASR` + `/big_disk/models/MiMo-Audio-Tokenizer` | + +--- + +## 安装步骤 (Step-by-Step) + +### Step 1 — `ffmpeg` + +```bash +$ which ffmpeg || sudo apt install ffmpeg +$ ffmpeg -version +``` + +`VTT_ExtractAudio` 走 `subprocess` 直调 `ffmpeg` —— 没装就报: + +``` +RuntimeError: ffmpeg not found on PATH. Install with: sudo apt install ffmpeg +``` + +如果 ComfyUI 在容器里跑,必须装在容器里,不是宿主机。 + +### Step 2 — `torchaudio` + +```bash +$ cd /mnt/modelall/ComfyUI && source .venv/bin/activate +$ uv pip install torchaudio +$ python -c "import torchaudio; print(torchaudio.__version__)" +``` + +`torchaudio` 在 `AudioFromPath.load()` 里 **lazy import** —— +所以没装上时 ComfyUI 启动**不会报错**,只在你跑工作流时才看到: + +``` +RuntimeError: torchaudio is required. Install with: uv pip install torchaudio +``` + +### Step 3 — MiMoASR 装好(关键三个坑) + +按 [MiMoASR 自己的 README](https://github.com/aadebuger/ComfyUI-MiMoASR#install-the-proven-order) 走。三个**必踩**的坑: + +#### 3.1 `transformers` 必须降到 `4.49.0` + +```bash +$ uv pip freeze > ~/pip_freeze_before_mimo_$(date +%Y%m%d).txt # 先备份! +$ uv pip install "transformers==4.49.0" "tokenizers>=0.21,<0.22" --force-reinstall +$ python -c "import transformers; assert transformers.__version__ == '4.49.0'; print('OK')" +``` + +> ⚠️ 这次降级会**破坏** `Sonic`、`Qwen3TTS`、新版 `VibeVoice` 等需要 `transformers>=4.57` 的节点。 +> 没办法同居 —— 要么牺牲那些节点,要么开一个独立的 ComfyUI 实例专门跑 MiMo。 + +#### 3.2 `flash-attn` 必装,没 SDPA 回退 + +```bash +# 看 torch 版本决定走 wheel 还是源码 +$ python -c "import torch; print(torch.__version__)" +``` + +**A. 你的 torch 是 `2.6.0 + cu12 + Python 3.12`** → 直接装 wheel: + +```bash +$ uv pip install flash-attn==2.7.4.post1 +``` + +**B. 你的 torch 是 2.11(实战环境)** → 从源码编译,约 20–40 分钟: + +```bash +$ uv pip install ninja packaging +$ export CUDA_HOME=/usr/local/cuda-12.6 +$ export TORCH_CUDA_ARCH_LIST="8.6" # 3090=8.6 4090=8.9 H100=9.0 +$ export MAX_JOBS=4 # 64GB RAM 可以开 8,OOM 就降到 2 +$ nohup uv pip install flash-attn --no-build-isolation \ + > ~/flash_attn_build.log 2>&1 & +$ tail -f ~/flash_attn_build.log +``` + +**验证(必做)**: + +```bash +$ python -c "from flash_attn import flash_attn_varlen_func; print(flash_attn_varlen_func)" +# 期望: +# 如果是 None 或 ImportError → 编译失败,看 ~/flash_attn_build.log +``` + +如果这里漏了验证,后面会以 `TypeError: 'NoneType' object is not callable` 神秘报错。 + +#### 3.3 必须把 MiMo 上游 repo 克隆到磁盘 + +`MimoAudio` 不能 pip 装,loader 会 `sys.path.insert`: + +```bash +$ mkdir -p /opt/repos && cd /opt/repos +$ git clone https://github.com/XiaomiMiMo/MiMo-V2.5-ASR.git +$ ls /opt/repos/MiMo-V2.5-ASR/src/mimo_audio/ +# 必须能列出 .py 文件。如果是空目录: +$ cd /opt/repos/MiMo-V2.5-ASR && git submodule update --init --recursive +``` + +### Step 4 — 下载 MiMo 权重(~24 GB) + +```bash +# HF cache 必须重定向到大盘 +$ export HF_HOME=/mnt/modelall/hf_cache +$ mkdir -p $HF_HOME + +$ uv pip install -U huggingface-hub + +# 主模型 ~16 GB +$ hf download XiaomiMiMo/MiMo-V2.5-ASR \ + --local-dir /mnt/modelall/models/MiMo-V2.5-ASR + +# Audio tokenizer ~1-2 GB +$ hf download XiaomiMiMo/MiMo-Audio-Tokenizer \ + --local-dir /mnt/modelall/models/MiMo-Audio-Tokenizer + +# 验证 +$ ls -lh /mnt/modelall/models/MiMo-V2.5-ASR/*.safetensors +$ ls -lh /mnt/modelall/models/MiMo-Audio-Tokenizer/ +``` + +国内下载慢可加 HF 镜像: + +```bash +$ export HF_ENDPOINT=https://hf-mirror.com +$ hf download XiaomiMiMo/MiMo-V2.5-ASR --local-dir /mnt/modelall/models/MiMo-V2.5-ASR +``` + +### Step 5 — 装本包 + +```bash +$ cd /mnt/modelall/ComfyUI/custom_nodes +$ git clone https://github.com/aadebuger/ComfyUI-VideoToText.git + +# OR 从 tarball 装: +$ cd /mnt/modelall/ComfyUI/custom_nodes +$ tar xzf /tmp/ComfyUI-VideoToText-v0.2.0.tar.gz + +# 验证目录结构(关键!不能多嵌套一层) +$ ls ComfyUI-VideoToText/ +# 必须直接看到 __init__.py / nodes/ / cli/ ... + +# 如果是 ComfyUI-VideoToText/ComfyUI-VideoToText/__init__.py(嵌套了),扁平化: +$ cd ComfyUI-VideoToText +$ mv ComfyUI-VideoToText/* . +$ mv ComfyUI-VideoToText/.* . 2>/dev/null +$ rmdir ComfyUI-VideoToText +``` + +### Step 6 — 装 `rgthree-comfy`(如果还没装) + +CLI 用 `PreviewAny` 抓 STRING 输出,必须有: + +```bash +$ cd /mnt/modelall/ComfyUI/custom_nodes +$ git clone https://github.com/rgthree/rgthree-comfy.git +``` + +### Step 7 — 启动 ComfyUI(三个 env var) + +```bash +$ pkill -9 -f main.py; sleep 3 +$ cd /mnt/modelall/ComfyUI && source .venv/bin/activate + +$ export HF_HOME=/mnt/modelall/hf_cache +$ export MIMO_ASR_REPO=/opt/repos/MiMo-V2.5-ASR +# 可选:把 MiMo 绑到 GPU 1,让 GPU 0 留给图像模型 +# $ export CUDA_VISIBLE_DEVICES=1 + +$ nohup python main.py --listen 0.0.0.0 --port 8188 > ~/comfy_start.log 2>&1 & +$ sleep 30 + +# 看启动日志 +$ grep -iE "IMPORT FAILED|VideoToText|MiMo|error" ~/comfy_start.log | head -20 +``` + +### Step 8 — 验证 9 个节点都在 + +```bash +$ curl -s http://SERVER:8188/object_info | python3 -c " +import json, sys +d = json.load(sys.stdin) +for n in ('VTT_LoadVideo','VTT_ExtractAudio','VTT_AudioFromPath', + 'VTT_BuildTranscriptJSON','VTT_SaveTranscript', + 'MiMoASRLoader','MiMoASRTranscribe','PreviewAny','LoadAudio'): + print(f' {n}: {\"OK\" if n in d else \"MISSING\"}')" +``` + +9 个全 OK 才算 ready。 + +### Step 9 — 跑一个 CLI 测试 + +```bash +$ cd /mnt/modelall/ComfyUI/custom_nodes/ComfyUI-VideoToText/cli +$ uv pip install --system websocket-client requests + +$ uv run python video_to_text_cli.py /path/to/test.mp4 \ + --ip 10.8.0.12 \ + --model-path /mnt/modelall/models/MiMo-V2.5-ASR \ + --tokenizer-path /mnt/modelall/models/MiMo-Audio-Tokenizer \ + --repo-path /opt/repos/MiMo-V2.5-ASR \ + --language Auto +``` + +第一次跑会卡 30–60s(MiMo 8B 模型进 VRAM),之后秒级。 + +--- + +## 常见失败模式 / Common Failures + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `VTT_*: MISSING` 在 `/object_info` | 包没装上 / 嵌套了 / ComfyUI 没重启 | 走完 Step 5 + 重启 | +| `IMPORT FAILED ComfyUI-VideoToText` 在 startup log | Python import 错 | 看 traceback;通常是 `No module named 'folder_paths'`(包不在 `custom_nodes/` 里)| +| `MiMoASRLoader: MISSING` | MiMoASR 装失败 | 看 MiMoASR 的 INSTALL_NOTES | +| `AttributeError: ... 'all_tied_weights_keys'` | `transformers` 太新 | 降到 `4.49.0`,见 Step 3.1 | +| `TypeError: 'NoneType' object is not callable` 在 `flash_attn_varlen_func` | `flash-attn` 编译静默失败 | 重新编,看 `~/flash_attn_build.log`,见 Step 3.2 | +| `RuntimeError: MiMo ASR repo path is empty` | `MIMO_ASR_REPO` 没设 | 启动前 `$ export MIMO_ASR_REPO=...` | +| `RuntimeError: ffmpeg not found on PATH` | ffmpeg 没装 | `$ sudo apt install ffmpeg` | +| `RuntimeError: torchaudio is required` | torchaudio 没装 | `$ uv pip install torchaudio` | +| `model path does not exist` | 工作流里填了相对路径 | 改成绝对路径 | +| CUDA OOM 在 transcribe | GPU 0 被图像模型占了 | `$ export CUDA_VISIBLE_DEVICES=1` 再启动 | +| 启动后再次验证还是 MISSING | 重启没生效 | `$ ps aux \| grep main.py` 确认 PID + `$ stat -c '%y' ~/comfy_start.log` 看时间戳 | + +--- + +## VRAM 预算 + +| | VRAM | +| --- | --- | +| MiMo-V2.5-ASR load (bf16) | ~17 GB | +| MiMo 推理峰值 | ~20–22 GB | +| 本包节点 | 可忽略(ffmpeg + torchaudio buffer) | + +24 GB GPU 够 MiMo + 轻度 ComfyUI 共用。如果你还在同一张卡跑 SDXL / Z-Image-Turbo,**强烈建议** `$ export CUDA_VISIBLE_DEVICES=1` 把 MiMo 隔离到另一张卡。 + +--- + +## 清理 / Cleanup + +```bash +# 音频缓存(VTT_ExtractAudio 写到 $TMPDIR/comfyui_vtt/) +$ rm -rf /tmp/comfyui_vtt + +# ComfyUI output(含已保存的 transcript JSON) +$ ls /mnt/modelall/ComfyUI/output/transcript_*.json + +# MiMo 模型(占 ~24GB,慎删) +# $ rm -rf /mnt/modelall/models/MiMo-V2.5-ASR +# $ rm -rf /mnt/modelall/models/MiMo-Audio-Tokenizer + +# 卸载本包 +$ rm -rf /mnt/modelall/ComfyUI/custom_nodes/ComfyUI-VideoToText +# 重启 ComfyUI +``` diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..be1c276 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 aadebuger + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 index 0000000..d80974c --- /dev/null +++ b/README.md @@ -0,0 +1,268 @@ +# ComfyUI-VideoToText + +> Video → audio → JSON transcript pipeline for ComfyUI. Pairs with [**ComfyUI-MiMoASR**](https://github.com/aadebuger/ComfyUI-MiMoASR) (Xiaomi MiMo-V2.5-ASR) for the ASR step. + +This pack provides the **non-ASR** pieces of a video-to-text workflow: video loading, audio extraction (`ffmpeg`), `AUDIO` dict bridging, JSON packaging, and disk persistence. The ASR itself is delegated to MiMoASR, which gives you strong Mandarin / English code-switching, dialect support (Wu, Cantonese, Hokkien, Sichuanese, Henan, Northeastern Mandarin), and noisy-audio robustness. + +| Node | Purpose | +| --- | --- | +| 🎙️ **Load Video (ASR)** | Pick a video from `ComfyUI/input/` or pass an absolute path. | +| 🎙️ **Extract Audio (ffmpeg)** | Demux audio track to a 16 kHz mono WAV. On-disk cache keyed by mtime + params. | +| 🎙️ **Audio From Path** | Bridge: WAV file → ComfyUI `AUDIO` dict. Optional resample / mono. Also reports duration. | +| 🎙️ **Build Transcript JSON** | Wrap ASR text + source metadata into a structured v2.0 JSON. | +| 🎙️ **Save Transcript JSON** | Validate + write JSON to `ComfyUI/output/` with timestamped filename. | + +--- + +## ⚠️ Read this before installing + +This pack does **not** ship an ASR model. Three things must be true before you can run a workflow: + +1. **[ComfyUI-MiMoASR](https://github.com/aadebuger/ComfyUI-MiMoASR) installed and working** — including its `transformers==4.49.0` downgrade and `flash-attn` build. Verify those nodes register before you even start with this pack. +2. **`ffmpeg` on your server's `PATH`** — `$ which ffmpeg` must return a path. +3. **`torchaudio` in ComfyUI's venv** — installed in Step 2 below. + +Quick gut check: + +| | Required | How to check | +| --- | --- | --- | +| `ffmpeg` | any recent | `$ ffmpeg -version` | +| `torchaudio` | any | `$ python -c "import torchaudio; print(torchaudio.__version__)"` | +| ComfyUI-MiMoASR | installed | `$ curl -s http://SERVER:8188/object_info \| grep -o MiMoASRLoader` | +| `PreviewAny` (from rgthree-comfy) | installed | `$ curl -s http://SERVER:8188/object_info \| grep -o PreviewAny` | + +--- + +## Install (the proven order) + +Each step has a verification at the end. Don't skip the verifications. + +### 1. Install MiMoASR first + +This pack depends on `MiMoASRLoader` and `MiMoASRTranscribe` from a separate repo. Follow [its full install guide](https://github.com/aadebuger/ComfyUI-MiMoASR#install-the-proven-order) before continuing here. **Stop and verify**: + +```bash +$ curl -s http://SERVER:8188/object_info | python3 -c " +import json, sys +d = json.load(sys.stdin) +for n in ('MiMoASRLoader', 'MiMoASRTranscribe'): + print(f' {n}: {\"OK\" if n in d else \"MISSING\"}')" +``` + +Both must print `OK`. If `MISSING`, fix MiMoASR before continuing. + +### 2. Clone this node pack + +```bash +$ cd ComfyUI/custom_nodes +$ git clone https://github.com/aadebuger/ComfyUI-VideoToText.git +``` + +Or, if installing from a tarball release: + +```bash +$ cd ComfyUI/custom_nodes +$ tar xzf /path/to/ComfyUI-VideoToText-v0.2.0.tar.gz +``` + +Verify: + +```bash +$ ls ComfyUI/custom_nodes/ComfyUI-VideoToText/__init__.py +# Must exist. If it's nested inside ComfyUI-VideoToText/ComfyUI-VideoToText/, you have a double-wrap — flatten it. +``` + +### 3. Install Python deps (in ComfyUI's venv) + +```bash +$ cd ComfyUI && source .venv/bin/activate +$ uv pip install torchaudio +``` + +Verify: + +```bash +$ python -c "import torchaudio; print('torchaudio', torchaudio.__version__)" +# Must print a version string. Errors here → re-run install. +``` + +### 4. Install `ffmpeg` on the server (if missing) + +```bash +$ which ffmpeg || sudo apt install ffmpeg +$ ffmpeg -version +``` + +### 5. Restart ComfyUI + +The same launch incantation MiMoASR needs — both `HF_HOME` and `MIMO_ASR_REPO` env vars must be set: + +```bash +$ pkill -9 -f main.py; sleep 3 +$ cd ComfyUI && source .venv/bin/activate +$ export HF_HOME=/big_disk/hf_cache +$ export MIMO_ASR_REPO=/opt/repos/MiMo-V2.5-ASR +$ nohup python main.py --listen 0.0.0.0 --port 8188 > ~/comfy_start.log 2>&1 & +$ sleep 30 +$ grep -iE "IMPORT FAILED|VideoToText|MiMo" ~/comfy_start.log | head -20 +``` + +### 6. Verify all 9 required nodes are registered + +```bash +$ curl -s http://SERVER:8188/object_info | python3 -c " +import json, sys +d = json.load(sys.stdin) +for n in ('VTT_LoadVideo','VTT_ExtractAudio','VTT_AudioFromPath', + 'VTT_BuildTranscriptJSON','VTT_SaveTranscript', + 'MiMoASRLoader','MiMoASRTranscribe','PreviewAny','LoadAudio'): + print(f' {n}: {\"OK\" if n in d else \"MISSING\"}')" +``` + +All 9 should show `OK`. If `PreviewAny` is missing, install [rgthree-comfy](https://github.com/rgthree/rgthree-comfy). + +--- + +## Usage + +### Minimal workflow + +``` +VTT_LoadVideo → VTT_ExtractAudio → VTT_AudioFromPath ─┐ + ├──► MiMoASRTranscribe → text + MiMoASRLoader ───────┘ │ + ▼ + VTT_BuildTranscriptJSON → STRING + │ + ▼ + VTT_SaveTranscript → ComfyUI/output/*.json +``` + +**Required fields**: + +- **MiMoASRLoader** (absolute paths!): + - `model_path` = `/big_disk/models/MiMo-V2.5-ASR` + - `tokenizer_path` = `/big_disk/models/MiMo-Audio-Tokenizer` + - `repo_path` = leave blank if `MIMO_ASR_REPO` env var is set +- **MiMoASRTranscribe**: + - `language` = `Auto` (code-switched), `Chinese`, or `English` + +The first run loads the 8B MiMo model into VRAM (30–60s). Subsequent runs in the same ComfyUI session are instant — MiMo's loader has a process-wide cache. + +An API-format example workflow ships at [`example_workflows/video_to_json.json`](example_workflows/video_to_json.json). Edit the three MiMo paths to match your server before loading it in the UI. + +### CLI + +A single-file CLI uploads a local video, drives the workflow over ComfyUI's HTTP + WebSocket API, and dumps the JSON transcript locally: + +```bash +$ uv pip install --system websocket-client requests + +$ uv run python cli/video_to_text_cli.py /path/to/lecture.mp4 \ + --ip 10.8.0.12 \ + --model-path /big_disk/models/MiMo-V2.5-ASR \ + --tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer \ + --repo-path /opt/repos/MiMo-V2.5-ASR \ + --language Auto +``` + +A wrapper alias to skip retyping paths every time: + +```bash +# ~/.bashrc +alias v2t='uv run python /path/to/ComfyUI-VideoToText/cli/video_to_text_cli.py \ + --ip 10.8.0.12 \ + --model-path /big_disk/models/MiMo-V2.5-ASR \ + --tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer \ + --repo-path /opt/repos/MiMo-V2.5-ASR' + +# Then: +$ v2t my_video.mp4 --language Chinese +$ v2t lecture.mp4 --language Auto --out ./transcripts +``` + +See [`cli/README.md`](cli/README.md) for all flags. + +--- + +## JSON schema v2.0 + +```json +{ + "schema_version": "2.0", + "source": { + "audio_path": "/tmp/comfyui_vtt/lecture_abc123.wav", + "duration_sec": 1834.512, + "video_path": "/path/to/lecture.mp4" + }, + "model": { + "name": "MiMo-V2.5-ASR", + "tokenizer": "MiMo-Audio-Tokenizer", + "language": "Auto" + }, + "text": "你好,今天我们聊一下 ESP32 嵌入式开发……" +} +``` + +### Trade-off vs faster-whisper + +MiMo-V2.5-ASR returns a **single plain-text transcript** — no word/segment timestamps, no confidence scores. Schema v2.0 reflects this: top-level `text` only, no `segments[]`. If you need time-aligned subtitles, run a different ASR backend (or wait for the planned v0.3.0 hybrid mode that pairs MiMo for text quality with faster-whisper for time alignment). + +--- + +## Tips + +- **Pure-language audio**: setting `language=Chinese` or `English` is slightly more accurate than `Auto`. Use `Auto` only for code-switched content. +- **GPU pinning**: on dual-GPU rigs, `$ export CUDA_VISIBLE_DEVICES=1` before launching ComfyUI keeps MiMo from fighting image models on `cuda:0`. +- **Audio cache**: `VTT_ExtractAudio` writes to `$TMPDIR/comfyui_vtt/` keyed by source mtime + sample rate + mono flag. Re-runs on the same video skip `ffmpeg` entirely. Nuke with `$ rm -rf /tmp/comfyui_vtt`. +- **Long audio**: MiMo handles multi-hour lectures in a single call. No manual chunking needed at typical podcast / lecture lengths. For 6h+ recordings, consider splitting at silence with `ffmpeg`. + +--- + +## Troubleshooting + +| Symptom | Likely cause | Fix | +| --- | --- | --- | +| `VTT_*` nodes show `MISSING` in `/object_info` | Pack not loaded; ComfyUI didn't restart; or `IMPORT FAILED` | Check `~/comfy_start.log` for `IMPORT FAILED ComfyUI-VideoToText` traceback | +| `MiMoASRLoader: MISSING` | MiMoASR not installed | See [its README](https://github.com/aadebuger/ComfyUI-MiMoASR) | +| `PreviewAny: MISSING` | rgthree-comfy not installed | `$ git clone https://github.com/rgthree/rgthree-comfy.git` into `custom_nodes/` | +| `RuntimeError: ffmpeg not found on PATH` | ffmpeg missing on the server | `$ sudo apt install ffmpeg` | +| `RuntimeError: torchaudio is required` | torchaudio missing in ComfyUI's venv | `$ uv pip install torchaudio` in the right venv | +| `RuntimeError: MiMo ASR repo path is empty` | `MIMO_ASR_REPO` env var not set | Set it before launching ComfyUI, OR pass `repo_path` in the workflow | +| MiMoASRLoader: "model path does not exist" | Relative path in the workflow | Always use absolute paths | +| `IMPORT FAILED ComfyUI-VideoToText` with `No module named 'folder_paths'` | Pack at wrong location | Must be in `ComfyUI/custom_nodes/`, not elsewhere | +| CUDA OOM on transcribe | GPU 0 held by image models | `$ export CUDA_VISIBLE_DEVICES=1` before launching ComfyUI | + +See [`INSTALL_NOTES.md`](INSTALL_NOTES.md) for the full debug log of real-world install gotchas. + +--- + +## Compatibility + +| | Required | Tested with | +| --- | --- | --- | +| Python | 3.10+ | 3.13 | +| PyTorch | matches ComfyUI's | 2.11.0+cu126 | +| `torchaudio` | any | matches torch | +| `transformers` | **4.49.0** (MiMo pin) | 4.49.0 | +| CUDA | ≥12.0 | 12.6 | +| GPU VRAM | 24 GB (MiMo needs it) | 2× RTX 3090 | +| OS | Linux | Ubuntu 22.04 | + +> ⚠️ The `transformers==4.49.0` pin (required by MiMo) will break `Sonic`, `Qwen3TTS`, and recent `VibeVoice` builds — they want `>=4.57`. If you need both, run MiMo + VideoToText on a dedicated ComfyUI instance. + +--- + +## Links + +- 🤗 MiMo-V2.5-ASR weights: +- 🤗 MiMo-Audio-Tokenizer: +- MiMo upstream repo: +- MiMo ComfyUI nodes: +- Blog: + +--- + +## License + +MIT — see [LICENSE](LICENSE). MiMoASR upstream is Apache-2.0; MiMo-V2.5-ASR weights are MIT (Xiaomi). diff --git a/README.zh-CN.md b/README.zh-CN.md new file mode 100644 index 0000000..08399f8 --- /dev/null +++ b/README.zh-CN.md @@ -0,0 +1,167 @@ +# ComfyUI-VideoToText 中文快速入门 + +> 视频转文字 ComfyUI 节点包,配合 [ComfyUI-MiMoASR](https://github.com/aadebuger/ComfyUI-MiMoASR)(小米 MiMo-V2.5-ASR)使用。 + +完整英文文档请看 [README.md](README.md),实战安装记录在 [INSTALL_NOTES.md](INSTALL_NOTES.md)。本文档给的是**最短可工作路径**。 + +## 这个包做什么 + +提供视频→音频→JSON 转录管线的"包装层"。ASR 本身由 MiMoASR 完成(强中英 code-switching、方言、抗噪),本包负责前后处理: + +| 节点 | 作用 | +| --- | --- | +| 🎙️ Load Video (ASR) | 从 `ComfyUI/input` 选视频 | +| 🎙️ Extract Audio (ffmpeg) | 抽音轨成 16kHz WAV,带 mtime 缓存 | +| 🎙️ Audio From Path | WAV 路径 → ComfyUI AUDIO dict(喂给 MiMoASR) | +| 🎙️ Build Transcript JSON | 文本 + 元数据 → v2.0 JSON | +| 🎙️ Save Transcript JSON | 保存到 `ComfyUI/output/` | + +## 装节点(在 ComfyUI 服务器上) + +```bash +# 1) 装 ffmpeg(如果还没装) +$ sudo apt install ffmpeg + +# 2) clone 本包 +$ cd /mnt/modelall/ComfyUI/custom_nodes +$ git clone https://github.com/aadebuger/ComfyUI-VideoToText.git + +# 或者从 tarball 装: +$ tar xzf /tmp/ComfyUI-VideoToText-v0.2.0.tar.gz + +# 验证目录结构 +$ ls ComfyUI-VideoToText/__init__.py # 必须存在,不能多嵌套一层 + +# 3) 装唯一的依赖 +$ cd /mnt/modelall/ComfyUI && source .venv/bin/activate +$ uv pip install torchaudio + +# 4) 装 MiMoASR(如果还没装,详见它自己的 README) +$ cd /mnt/modelall/ComfyUI/custom_nodes +$ git clone https://github.com/aadebuger/ComfyUI-MiMoASR.git +$ cd ComfyUI-MiMoASR && ./scripts/install_server.sh + +# 5) 装 PreviewAny(CLI 用,从 rgthree-comfy) +$ cd /mnt/modelall/ComfyUI/custom_nodes +$ git clone https://github.com/rgthree/rgthree-comfy.git +``` + +## 装模型(MiMo 权重,~24 GB) + +```bash +# HF cache 必须重定向到大盘 +$ export HF_HOME=/mnt/modelall/hf_cache +$ mkdir -p $HF_HOME + +$ uv pip install -U huggingface-hub + +# 主模型 ~16 GB +$ hf download XiaomiMiMo/MiMo-V2.5-ASR \ + --local-dir /mnt/modelall/models/MiMo-V2.5-ASR + +# Audio tokenizer ~1-2 GB +$ hf download XiaomiMiMo/MiMo-Audio-Tokenizer \ + --local-dir /mnt/modelall/models/MiMo-Audio-Tokenizer + +# 国内下载慢的话加镜像: +# $ export HF_ENDPOINT=https://hf-mirror.com +``` + +还要 clone MiMo 上游 repo(loader 注入 sys.path 用): + +```bash +$ mkdir -p /opt/repos && cd /opt/repos +$ git clone https://github.com/XiaomiMiMo/MiMo-V2.5-ASR.git +``` + +## 启动 ComfyUI(关键三个 env var) + +```bash +$ pkill -9 -f main.py; sleep 3 +$ cd /mnt/modelall/ComfyUI && source .venv/bin/activate + +$ export HF_HOME=/mnt/modelall/hf_cache +$ export MIMO_ASR_REPO=/opt/repos/MiMo-V2.5-ASR +# 可选:把 MiMo 绑到 GPU 1 +# $ export CUDA_VISIBLE_DEVICES=1 + +$ nohup python main.py --listen 0.0.0.0 --port 8188 > ~/comfy_start.log 2>&1 & +$ sleep 30 +$ grep -iE "IMPORT FAILED|VideoToText" ~/comfy_start.log +``` + +## 验证(在 Mac 上跑) + +```bash +$ curl -s http://10.8.0.12:8188/object_info | python3 -c " +import json, sys +d = json.load(sys.stdin) +for n in ('VTT_LoadVideo','VTT_ExtractAudio','VTT_AudioFromPath', + 'VTT_BuildTranscriptJSON','VTT_SaveTranscript', + 'MiMoASRLoader','MiMoASRTranscribe','PreviewAny','LoadAudio'): + print(f' {n}: {\"OK\" if n in d else \"MISSING\"}')" +``` + +9 个全 OK 就 ready。 + +## 跑一个视频(在 Mac 上跑 CLI) + +```bash +$ cd ~/ComfyUI-VideoToText/cli +$ uv pip install --system websocket-client requests + +$ uv run python video_to_text_cli.py /path/to/test.mp4 \ + --ip 10.8.0.12 \ + --model-path /mnt/modelall/models/MiMo-V2.5-ASR \ + --tokenizer-path /mnt/modelall/models/MiMo-Audio-Tokenizer \ + --language Auto +``` + +第一次跑会卡 30–60 秒(MiMo 8B 模型进 VRAM),之后秒级。 + +可设个 alias 省事: + +```bash +# ~/.zshrc +alias v2t='uv run python ~/ComfyUI-VideoToText/cli/video_to_text_cli.py \ + --ip 10.8.0.12 \ + --model-path /mnt/modelall/models/MiMo-V2.5-ASR \ + --tokenizer-path /mnt/modelall/models/MiMo-Audio-Tokenizer' + +# 用: +$ v2t my_video.mp4 --language Chinese +``` + +## 输出长这样 + +```json +{ + "schema_version": "2.0", + "source": { + "audio_path": "/tmp/comfyui_vtt/foo.wav", + "duration_sec": 123.45, + "video_path": "/path/to/foo.mp4" + }, + "model": { + "name": "MiMo-V2.5-ASR", + "tokenizer": "MiMo-Audio-Tokenizer", + "language": "Auto" + }, + "text": "你好,今天我们聊一下 ESP32 嵌入式开发……" +} +``` + +## 常见问题速查 + +| 现象 | 原因 | 解决 | +| --- | --- | --- | +| `VTT_*` 节点 MISSING | 包没装 / 嵌套了 / 没重启 | 重新装 + 重启 ComfyUI | +| `MiMoASRLoader: MISSING` | MiMoASR 没装好 | 看 MiMoASR README | +| `transformers` 太新报错 | MiMo 要求 `==4.49.0` | `$ uv pip install transformers==4.49.0 --force-reinstall` | +| flash-attn `NoneType` 错误 | 编译静默失败 | 重编,看 `~/flash_attn_build.log` | +| `MIMO_ASR_REPO is empty` | 没设环境变量 | 启动前 `$ export MIMO_ASR_REPO=/opt/repos/MiMo-V2.5-ASR` | +| `ffmpeg not found` | 服务器没装 ffmpeg | `$ sudo apt install ffmpeg` | +| `torchaudio is required` | 漏装依赖 | `$ uv pip install torchaudio` | +| CUDA OOM | GPU 0 被占 | `$ export CUDA_VISIBLE_DEVICES=1` | + +详细排错见 [INSTALL_NOTES.md](INSTALL_NOTES.md)。 diff --git a/__init__.py b/__init__.py new file mode 100644 index 0000000..b4bdf44 --- /dev/null +++ b/__init__.py @@ -0,0 +1,32 @@ +"""ComfyUI-VideoToText: Video → Audio → MiMo-V2.5-ASR → JSON pipeline helpers. + +This pack provides the *non-ASR* pieces of a video-to-text pipeline. The ASR itself +is handled by ComfyUI-MiMoASR (https://github.com/aadebuger/ComfyUI-MiMoASR), which +must be installed separately. +""" + +from .nodes.load_video import LoadVideoForASR +from .nodes.extract_audio import ExtractAudio +from .nodes.audio_from_path import AudioFromPath +from .nodes.build_json import BuildTranscriptJSON +from .nodes.save_transcript import SaveTranscript + + +NODE_CLASS_MAPPINGS = { + "VTT_LoadVideo": LoadVideoForASR, + "VTT_ExtractAudio": ExtractAudio, + "VTT_AudioFromPath": AudioFromPath, + "VTT_BuildTranscriptJSON": BuildTranscriptJSON, + "VTT_SaveTranscript": SaveTranscript, +} + +NODE_DISPLAY_NAME_MAPPINGS = { + "VTT_LoadVideo": "🎙️ Load Video (ASR)", + "VTT_ExtractAudio": "🎙️ Extract Audio (ffmpeg)", + "VTT_AudioFromPath": "🎙️ Audio From Path", + "VTT_BuildTranscriptJSON": "🎙️ Build Transcript JSON", + "VTT_SaveTranscript": "🎙️ Save Transcript JSON", +} + +WEB_DIRECTORY = None +__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"] diff --git a/cli/README.md b/cli/README.md new file mode 100644 index 0000000..51a7c53 --- /dev/null +++ b/cli/README.md @@ -0,0 +1,99 @@ +# video_to_text_cli.py + +Submit a video to a remote ComfyUI server running `ComfyUI-VideoToText` + `ComfyUI-MiMoASR` and dump the JSON transcript locally. + +## Install + +```bash +uv pip install --system websocket-client requests +``` + +## Basic usage + +```bash +# Upload a local file and transcribe with Auto language +uv run python video_to_text_cli.py /path/to/lecture.mp4 \ + --ip 10.8.0.12 \ + --model-path /big_disk/models/MiMo-V2.5-ASR \ + --tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer + +# Pure Chinese content +uv run python video_to_text_cli.py speech_zh.mp4 \ + --ip 10.8.0.12 \ + --model-path /big_disk/models/MiMo-V2.5-ASR \ + --tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer \ + --language Chinese + +# Reference a file already in ComfyUI/input (no re-upload) +uv run python video_to_text_cli.py already_uploaded.mp4 \ + --ip 10.8.0.12 \ + --model-path /big_disk/models/MiMo-V2.5-ASR \ + --tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer + +# Dump workflow without running — feed into UI for debugging +uv run python video_to_text_cli.py video.mp4 \ + --model-path /big_disk/models/MiMo-V2.5-ASR \ + --tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer \ + --dump-workflow /tmp/wf.json +``` + +## Skip retyping paths + +Add an alias to your shell: + +```bash +# ~/.bashrc +alias v2t='uv run python /path/to/ComfyUI-VideoToText/cli/video_to_text_cli.py \ + --ip 10.8.0.12 \ + --model-path /big_disk/models/MiMo-V2.5-ASR \ + --tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer \ + --repo-path /opt/repos/MiMo-V2.5-ASR' + +# Then: +v2t my_video.mp4 --language Auto +v2t lecture.mp4 --language Chinese --out ./transcripts +``` + +## All flags + +| Flag | Default | Purpose | +|---|---|---| +| `video` (positional) | — | Local path OR filename already in `ComfyUI/input`. | +| `--ip` | `127.0.0.1` | ComfyUI host. | +| `--port` | `8188` | ComfyUI port. | +| `--server` | — | Full `host:port` (overrides `--ip` / `--port`). | +| `--model-path` | **required** | Absolute path to `MiMo-V2.5-ASR` on the server. | +| `--tokenizer-path` | **required** | Absolute path to `MiMo-Audio-Tokenizer` on the server. | +| `--repo-path` | `""` | Absolute path to MiMo source repo. Empty if `MIMO_ASR_REPO` is set on the server. | +| `--force-reload` | off | Drop the MiMo loader cache and re-instantiate. | +| `--language` | `Auto` | `Auto` / `Chinese` / `English`. | +| `--sample-rate` | `16000` | ffmpeg output rate before MiMo. | +| `--save-prefix` | `transcript` | Filename prefix on the server side. | +| `--out` | `./out` | Local directory to save the JSON. | +| `--dump-workflow` | — | Path to dump generated workflow JSON. | +| `--timeout` | `1800` | WebSocket wait timeout (sec). | + +## How it works + +1. Upload the local video to `ComfyUI/input/` via `POST /upload/image`. +2. Build the 9-node API workflow: + - `VTT_LoadVideo` → `VTT_ExtractAudio` → `VTT_AudioFromPath` + - `MiMoASRLoader` + above audio → `MiMoASRTranscribe` → STRING + - `VTT_BuildTranscriptJSON` → `VTT_SaveTranscript` + - 2 × `PreviewAny` to surface outputs over WebSocket +3. Submit via `POST /prompt`, stream `executing` / `progress` events over WS. +4. Pull `transcript_json` from `history.outputs.preview_json.text` and write to `<--out>/.json`. + +The `save` node also persists a timestamped copy to the server's `ComfyUI/output/`. + +## Troubleshooting + +**`Connection refused`** — ComfyUI not running, or wrong `--ip`. `curl http://:8188/`. + +**`Node error: ... MiMo ASR repo path is empty`** — Either set `MIMO_ASR_REPO` on the server, or pass `--repo-path` with an absolute path. + +**`Node error: ... model path does not exist`** — Check `--model-path` and `--tokenizer-path` are absolute paths that exist on the **server** (not your local machine). + +**`No transcript output`** — script dumps the first 2KB of `/history` to stderr. Most common: MiMoASR not installed, or `flash-attn` install silently failed. + +**First run takes 30–60s with no progress output** — that's MiMo loading its 8B model into VRAM. Subsequent runs in the same ComfyUI session hit the cache. diff --git a/cli/video_to_text_cli.py b/cli/video_to_text_cli.py new file mode 100644 index 0000000..ce06617 --- /dev/null +++ b/cli/video_to_text_cli.py @@ -0,0 +1,337 @@ +#!/usr/bin/env python3 +""" +video_to_text_cli.py - Submit a video to ComfyUI-VideoToText + ComfyUI-MiMoASR and dump the JSON transcript. + +Pipeline: VTT_LoadVideo -> VTT_ExtractAudio -> VTT_AudioFromPath + -> MiMoASRLoader + MiMoASRTranscribe + -> VTT_BuildTranscriptJSON -> VTT_SaveTranscript + +Usage: + uv run python video_to_text_cli.py /path/to/video.mp4 \ + --ip 10.8.0.12 \ + --model-path /big_disk/models/MiMo-V2.5-ASR \ + --tokenizer-path /big_disk/models/MiMo-Audio-Tokenizer \ + --repo-path /opt/repos/MiMo-V2.5-ASR \ + --language Auto + +Dependencies: + uv pip install --system websocket-client requests +""" +from __future__ import annotations + +import argparse +import json +import sys +import time +import uuid +from pathlib import Path +from urllib.parse import urlparse + +import requests +import websocket + + +# ---------------------------------------------------------------------- +# ComfyClient - same shape as the rest of the aadebuger CLI fleet. +# ---------------------------------------------------------------------- +class ComfyClient: + def __init__(self, server: str = "127.0.0.1:8188"): + self.server = server + self.client_id = str(uuid.uuid4()) + self.base = f"http://{server}" + + def upload_file(self, file_path: Path, *, overwrite: bool = True) -> str: + """POST /upload/image - endpoint is named 'image' but accepts any file.""" + with open(file_path, "rb") as f: + files = {"image": (file_path.name, f, "application/octet-stream")} + data = {"overwrite": "true" if overwrite else "false"} + r = requests.post(f"{self.base}/upload/image", files=files, data=data, timeout=600) + r.raise_for_status() + return r.json().get("name", file_path.name) + + def submit(self, workflow: dict) -> str: + payload = {"prompt": workflow, "client_id": self.client_id} + r = requests.post(f"{self.base}/prompt", json=payload, timeout=30) + r.raise_for_status() + return r.json()["prompt_id"] + + def history(self, prompt_id: str) -> dict: + r = requests.get(f"{self.base}/history/{prompt_id}", timeout=30) + r.raise_for_status() + return r.json().get(prompt_id, {}) + + def view(self, filename: str, subfolder: str = "", type_: str = "output") -> bytes: + params = {"filename": filename, "subfolder": subfolder, "type": type_} + r = requests.get(f"{self.base}/view", params=params, timeout=300) + r.raise_for_status() + return r.content + + def run(self, workflow: dict, *, timeout_s: float = 1800) -> dict: + ws = websocket.WebSocket() + ws.connect(f"ws://{self.server}/ws?clientId={self.client_id}") + prompt_id = self.submit(workflow) + start = time.time() + last_node = None + try: + while True: + if time.time() - start > timeout_s: + raise TimeoutError(f"Timed out after {timeout_s:.0f}s") + msg = ws.recv() + if not isinstance(msg, str): + continue + m = json.loads(msg) + t = m.get("type") + d = m.get("data", {}) + if t == "executing": + if d.get("prompt_id") != prompt_id: + continue + nid = d.get("node") + if nid is None: + print() + break + if nid != last_node: + print(f"\r > node {str(nid):<28}", end="", flush=True) + last_node = nid + elif t == "progress": + if d.get("prompt_id") != prompt_id: + continue + cur, tot = d.get("value", 0), d.get("max", 1) + print(f"\r > {str(last_node or '?'):<14} {cur:>4}/{tot}", end="", flush=True) + elif t == "execution_error": + raise RuntimeError(f"Node error: {json.dumps(d, ensure_ascii=False)[:600]}") + finally: + ws.close() + return self.history(prompt_id) + + +# ---------------------------------------------------------------------- +# build_workflow - API format; semantic node ids. +# ---------------------------------------------------------------------- +def build_workflow( + *, + video_filename: str, + model_path: str, + tokenizer_path: str, + repo_path: str, + language: str, + sample_rate: int, + save_prefix: str, + force_reload: bool = False, +) -> dict: + return { + "loader_video": { + "class_type": "VTT_LoadVideo", + "inputs": { + "video": video_filename, + "video_path_override": "", + }, + }, + "extract_audio": { + "class_type": "VTT_ExtractAudio", + "inputs": { + "video_path": ["loader_video", 0], + "sample_rate": sample_rate, + "mono": True, + }, + }, + "audio_in": { + "class_type": "VTT_AudioFromPath", + "inputs": { + "audio_path": ["extract_audio", 0], + "target_sample_rate": 0, + "force_mono": False, + }, + }, + "mimo_loader": { + "class_type": "MiMoASRLoader", + "inputs": { + "model_path": model_path, + "tokenizer_path": tokenizer_path, + "repo_path": repo_path, + "force_reload": force_reload, + }, + }, + "mimo_asr": { + "class_type": "MiMoASRTranscribe", + "inputs": { + "model": ["mimo_loader", 0], + "audio": ["audio_in", 0], + "language": language, + }, + }, + "build_json": { + "class_type": "VTT_BuildTranscriptJSON", + "inputs": { + "text": ["mimo_asr", 0], + "audio_path": ["extract_audio", 0], + "duration_sec": ["audio_in", 1], + "model_name": "MiMo-V2.5-ASR", + "tokenizer_name": "MiMo-Audio-Tokenizer", + "language": language, + "source_video_path": ["loader_video", 0], + }, + }, + "save": { + "class_type": "VTT_SaveTranscript", + "inputs": { + "transcript_json": ["build_json", 0], + "filename_prefix": save_prefix, + "include_timestamp": True, + "subfolder": "", + }, + }, + "preview_text": { + "class_type": "PreviewAny", + "inputs": {"source": ["mimo_asr", 0]}, + }, + "preview_json": { + "class_type": "PreviewAny", + "inputs": {"source": ["build_json", 0]}, + }, + } + + +# ---------------------------------------------------------------------- +# extract_text_from_history - pull STRING output from PreviewAny. +# ---------------------------------------------------------------------- +def extract_text_from_history(history: dict, node_id: str) -> str: + outs = history.get("outputs", {}) + if node_id not in outs: + return "" + no = outs[node_id] + if "text" in no: + v = no["text"] + return "\n".join(str(x) for x in v) if isinstance(v, list) else str(v) + parts = [] + for v in no.values(): + if isinstance(v, list): + parts.extend(str(x) for x in v) + return "\n".join(parts) + + +def is_url(s: str) -> bool: + try: + u = urlparse(s) + return u.scheme in {"http", "https"} and bool(u.netloc) + except Exception: + return False + + +# ---------------------------------------------------------------------- +# main +# ---------------------------------------------------------------------- +def main() -> int: + ap = argparse.ArgumentParser( + description="Submit a video to ComfyUI-VideoToText + ComfyUI-MiMoASR and dump the JSON transcript." + ) + ap.add_argument( + "video", + help="Local video path (will be uploaded) OR filename already in ComfyUI/input.", + ) + + # Server + ap.add_argument("--ip", default="127.0.0.1") + ap.add_argument("--port", type=int, default=8188) + ap.add_argument("--server", default=None, help="Full host:port (overrides --ip/--port).") + + # MiMo paths (required on first run; can hardcode in a wrapper script) + ap.add_argument( + "--model-path", required=True, + help="Absolute path to MiMo-V2.5-ASR on the server (e.g. /big_disk/models/MiMo-V2.5-ASR)." + ) + ap.add_argument( + "--tokenizer-path", required=True, + help="Absolute path to MiMo-Audio-Tokenizer on the server." + ) + ap.add_argument( + "--repo-path", default="", + help="Absolute path to MiMo-V2.5-ASR source repo. Leave blank if MIMO_ASR_REPO is set." + ) + ap.add_argument("--force-reload", action="store_true", + help="Reload the model even if cached.") + + # Inference + ap.add_argument("--language", default="Auto", + choices=["Auto", "Chinese", "English"]) + ap.add_argument("--sample-rate", type=int, default=16000) + ap.add_argument("--save-prefix", default="transcript") + + # Output / debug + ap.add_argument("--out", type=Path, default=Path("./out")) + ap.add_argument("--dump-workflow", type=Path, default=None, + help="Dump generated workflow JSON for debugging.") + ap.add_argument("--timeout", type=float, default=1800) + + args = ap.parse_args() + + server = args.server or f"{args.ip}:{args.port}" + client = ComfyClient(server=server) + + # Resolve input + if is_url(args.video): + print("[!] URL input not supported. Download first or upload to ComfyUI/input.", + file=sys.stderr) + return 2 + + src = Path(args.video).expanduser() + if src.exists() and src.is_file(): + size_mb = src.stat().st_size / 1e6 + print(f"[+] Uploading {src.name} ({size_mb:.1f} MB) -> {server} ...") + uploaded_name = client.upload_file(src) + print(f" -> ComfyUI/input/{uploaded_name}") + else: + uploaded_name = args.video + print(f"[+] Using existing ComfyUI/input/{uploaded_name}") + + workflow = build_workflow( + video_filename=uploaded_name, + model_path=args.model_path, + tokenizer_path=args.tokenizer_path, + repo_path=args.repo_path, + language=args.language, + sample_rate=args.sample_rate, + save_prefix=args.save_prefix, + force_reload=args.force_reload, + ) + + if args.dump_workflow: + args.dump_workflow.parent.mkdir(parents=True, exist_ok=True) + args.dump_workflow.write_text( + json.dumps(workflow, ensure_ascii=False, indent=2), encoding="utf-8" + ) + print(f"[+] Dumped workflow -> {args.dump_workflow}") + + print(f"[+] Submitting (model=MiMo-V2.5-ASR, language={args.language}) ...") + history = client.run(workflow, timeout_s=args.timeout) + + transcript = extract_text_from_history(history, "preview_json") + text_only = extract_text_from_history(history, "preview_text") + + if not transcript: + print("[!] No transcript output. Dumping head of history:", file=sys.stderr) + print(json.dumps(history, ensure_ascii=False, indent=2)[:2000], file=sys.stderr) + return 1 + + args.out.mkdir(parents=True, exist_ok=True) + stem = src.stem if src.exists() else Path(args.video).stem + out_path = args.out / f"{stem}.json" + out_path.write_text(transcript, encoding="utf-8") + + # Summary + try: + data = json.loads(transcript) + dur = data.get("source", {}).get("duration_sec", 0.0) + chars = len(data.get("text", "")) + print(f"[v] duration {dur:.1f}s | {chars} chars | language={args.language}") + except Exception: + pass + + print(f"[v] Saved -> {out_path}") + if text_only: + preview = text_only[:200].replace("\n", " ") + print(f"[v] Preview: {preview}{'...' if len(text_only) > 200 else ''}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/example_workflows/video_to_json.json b/example_workflows/video_to_json.json new file mode 100644 index 0000000..3a2ec91 --- /dev/null +++ b/example_workflows/video_to_json.json @@ -0,0 +1,71 @@ +{ + "loader_video": { + "class_type": "VTT_LoadVideo", + "inputs": { + "video": "example.mp4", + "video_path_override": "" + } + }, + "extract_audio": { + "class_type": "VTT_ExtractAudio", + "inputs": { + "video_path": ["loader_video", 0], + "sample_rate": 16000, + "mono": true + } + }, + "audio_in": { + "class_type": "VTT_AudioFromPath", + "inputs": { + "audio_path": ["extract_audio", 0], + "target_sample_rate": 0, + "force_mono": false + } + }, + "mimo_loader": { + "class_type": "MiMoASRLoader", + "inputs": { + "model_path": "/big_disk/models/MiMo-V2.5-ASR", + "tokenizer_path": "/big_disk/models/MiMo-Audio-Tokenizer", + "repo_path": "", + "force_reload": false + } + }, + "mimo_asr": { + "class_type": "MiMoASRTranscribe", + "inputs": { + "model": ["mimo_loader", 0], + "audio": ["audio_in", 0], + "language": "Auto" + } + }, + "build_json": { + "class_type": "VTT_BuildTranscriptJSON", + "inputs": { + "text": ["mimo_asr", 0], + "audio_path": ["extract_audio", 0], + "duration_sec": ["audio_in", 1], + "model_name": "MiMo-V2.5-ASR", + "tokenizer_name": "MiMo-Audio-Tokenizer", + "language": "Auto", + "source_video_path": ["loader_video", 0] + } + }, + "save": { + "class_type": "VTT_SaveTranscript", + "inputs": { + "transcript_json": ["build_json", 0], + "filename_prefix": "transcript", + "include_timestamp": true, + "subfolder": "" + } + }, + "preview_text": { + "class_type": "PreviewAny", + "inputs": {"source": ["mimo_asr", 0]} + }, + "preview_json": { + "class_type": "PreviewAny", + "inputs": {"source": ["build_json", 0]} + } +} diff --git a/nodes/__init__.py b/nodes/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/nodes/_types.py b/nodes/_types.py new file mode 100644 index 0000000..875cb75 --- /dev/null +++ b/nodes/_types.py @@ -0,0 +1,15 @@ +"""Shared type constants for ComfyUI-VideoToText.""" + +CATEGORY = "🎙️ VideoToText" + +VIDEO_EXTS = { + ".mp4", ".mov", ".mkv", ".avi", ".webm", + ".flv", ".m4v", ".wmv", ".ts", ".mpg", ".mpeg", +} + +# MiMo-V2.5-ASR language modes +MIMO_LANGUAGES = ["Auto", "Chinese", "English"] + +# Default model component identifiers — surfaced in the JSON metadata. +DEFAULT_MODEL_NAME = "MiMo-V2.5-ASR" +DEFAULT_TOKENIZER_NAME = "MiMo-Audio-Tokenizer" diff --git a/nodes/audio_from_path.py b/nodes/audio_from_path.py new file mode 100644 index 0000000..fb795c8 --- /dev/null +++ b/nodes/audio_from_path.py @@ -0,0 +1,84 @@ +"""AudioFromPath - load an arbitrary audio file (from absolute path) into ComfyUI's AUDIO dict. + +ComfyUI's built-in LoadAudio only reads from ComfyUI/input/. Our ExtractAudio writes to +$TMPDIR/comfyui_vtt/, so we need this bridge to feed MiMoASRTranscribe's AUDIO input. + +ComfyUI AUDIO format: {"waveform": torch.Tensor[B, C, T], "sample_rate": int} +""" + +from pathlib import Path + +from ._types import CATEGORY + + +class AudioFromPath: + """Load any audio file path → ComfyUI AUDIO dict. Also reports duration in seconds.""" + + @classmethod + def INPUT_TYPES(cls): + return { + "required": { + "audio_path": ("STRING", {"forceInput": True}), + }, + "optional": { + "target_sample_rate": ( + "INT", + { + "default": 0, + "min": 0, + "max": 48000, + "step": 1000, + "tooltip": "Resample to this rate if non-zero. 0 = keep source rate.", + }, + ), + "force_mono": ("BOOLEAN", {"default": False}), + }, + } + + RETURN_TYPES = ("AUDIO", "FLOAT", "INT") + RETURN_NAMES = ("audio", "duration_sec", "sample_rate") + FUNCTION = "load" + CATEGORY = CATEGORY + + def load(self, audio_path, target_sample_rate=0, force_mono=False): + if not Path(audio_path).exists(): + raise FileNotFoundError(f"Audio not found: {audio_path}") + + # lazy import - torchaudio is heavy and not needed at module-load time + try: + import torch + import torchaudio + except ImportError as e: + raise RuntimeError( + "torchaudio is required. Install with: uv pip install torchaudio" + ) from e + + waveform, sample_rate = torchaudio.load(audio_path) + + # waveform shape from torchaudio.load: (channels, samples) + if force_mono and waveform.shape[0] > 1: + waveform = waveform.mean(dim=0, keepdim=True) + + if target_sample_rate and target_sample_rate != sample_rate: + resampler = torchaudio.transforms.Resample( + orig_freq=sample_rate, new_freq=target_sample_rate + ) + waveform = resampler(waveform) + sample_rate = target_sample_rate + + # ComfyUI AUDIO expects (batch, channels, samples) + if waveform.dim() == 2: + waveform = waveform.unsqueeze(0) + + duration = float(waveform.shape[-1]) / float(sample_rate) + audio_dict = {"waveform": waveform, "sample_rate": int(sample_rate)} + + return (audio_dict, duration, int(sample_rate)) + + @classmethod + def IS_CHANGED(cls, audio_path, target_sample_rate=0, force_mono=False): + try: + mtime = Path(audio_path).stat().st_mtime + except Exception: + mtime = 0 + return f"{audio_path}|{mtime}|{target_sample_rate}|{force_mono}" diff --git a/nodes/build_json.py b/nodes/build_json.py new file mode 100644 index 0000000..9cdf946 --- /dev/null +++ b/nodes/build_json.py @@ -0,0 +1,77 @@ +"""BuildTranscriptJSON - wrap MiMo's plain-text transcript + metadata into JSON schema v2.0. + +MiMo-V2.5-ASR returns a single string. This node packages it with source / model metadata +so the downstream SaveTranscript can write a structured file consistent with v0.1.0 outputs. + +Note: schema v2.0 drops `segments[]`, `confidence`, and word timestamps because MiMo's +`asr_sft` does not expose those. See the top-level README for the trade-off. +""" + +import json + +from ._types import ( + CATEGORY, + MIMO_LANGUAGES, + DEFAULT_MODEL_NAME, + DEFAULT_TOKENIZER_NAME, +) + + +class BuildTranscriptJSON: + """Pack ASR text + metadata into the v2.0 JSON schema (STRING output).""" + + @classmethod + def INPUT_TYPES(cls): + return { + "required": { + "text": ("STRING", {"forceInput": True}), + }, + "optional": { + "audio_path": ("STRING", {"default": ""}), + "duration_sec": ( + "FLOAT", + {"default": 0.0, "min": 0.0, "step": 0.001}, + ), + "model_name": ("STRING", {"default": DEFAULT_MODEL_NAME}), + "tokenizer_name": ("STRING", {"default": DEFAULT_TOKENIZER_NAME}), + "language": (MIMO_LANGUAGES, {"default": "Auto"}), + "source_video_path": ("STRING", {"default": ""}), + }, + } + + RETURN_TYPES = ("STRING",) + RETURN_NAMES = ("transcript_json",) + FUNCTION = "build" + CATEGORY = CATEGORY + + def build( + self, + text, + audio_path="", + duration_sec=0.0, + model_name=DEFAULT_MODEL_NAME, + tokenizer_name=DEFAULT_TOKENIZER_NAME, + language="Auto", + source_video_path="", + ): + clean_text = (text or "").strip() + + source = { + "audio_path": audio_path or None, + "duration_sec": round(float(duration_sec or 0.0), 3), + } + if source_video_path: + source["video_path"] = source_video_path + + result = { + "schema_version": "2.0", + "source": source, + "model": { + "name": model_name, + "tokenizer": tokenizer_name, + "language": language, + }, + "text": clean_text, + } + + return (json.dumps(result, ensure_ascii=False, indent=2),) diff --git a/nodes/extract_audio.py b/nodes/extract_audio.py new file mode 100644 index 0000000..3c6845f --- /dev/null +++ b/nodes/extract_audio.py @@ -0,0 +1,101 @@ +"""ExtractAudio node - demux audio track from a video into a temporary WAV via ffmpeg.""" + +import hashlib +import os +import shutil +import subprocess +import tempfile +from pathlib import Path + +from ._types import CATEGORY + + +class ExtractAudio: + """Extract mono/stereo PCM WAV from a video file using ffmpeg. + + Output sample rate defaults to 16 kHz mono — the format faster-whisper expects. + """ + + @classmethod + def INPUT_TYPES(cls): + return { + "required": { + "video_path": ("STRING", {"forceInput": True}), + "sample_rate": ("INT", {"default": 16000, "min": 8000, "max": 48000, "step": 1000}), + "mono": ("BOOLEAN", {"default": True}), + }, + "optional": { + "ffmpeg_extra_args": ( + "STRING", + {"default": "", "tooltip": "Extra args passed to ffmpeg before the output path."}, + ), + }, + } + + RETURN_TYPES = ("STRING",) + RETURN_NAMES = ("audio_path",) + FUNCTION = "extract" + CATEGORY = CATEGORY + + def _check_ffmpeg(self): + if shutil.which("ffmpeg") is None: + raise RuntimeError( + "ffmpeg not found on PATH. Install with: sudo apt install ffmpeg" + ) + + def extract(self, video_path, sample_rate, mono, ffmpeg_extra_args=""): + self._check_ffmpeg() + + src = Path(video_path) + if not src.exists() or not src.is_file(): + raise FileNotFoundError(f"Video not found: {src}") + + # Cache key: stable hash of source path + mtime + params + try: + mtime = src.stat().st_mtime + except Exception: + mtime = 0 + key = hashlib.sha1( + f"{src.resolve()}|{mtime}|{sample_rate}|{mono}|{ffmpeg_extra_args}".encode() + ).hexdigest()[:12] + + tmpdir = Path(tempfile.gettempdir()) / "comfyui_vtt" + tmpdir.mkdir(parents=True, exist_ok=True) + dst = tmpdir / f"{src.stem}_{key}.wav" + + # Reuse cached extraction if it's already there. + if dst.exists() and dst.stat().st_size > 0: + return (str(dst),) + + cmd = [ + "ffmpeg", "-y", + "-i", str(src), + "-vn", # no video + "-acodec", "pcm_s16le", + "-ar", str(sample_rate), + "-ac", "1" if mono else "2", + ] + if ffmpeg_extra_args.strip(): + cmd.extend(ffmpeg_extra_args.strip().split()) + cmd.append(str(dst)) + + result = subprocess.run(cmd, capture_output=True, text=True) + if result.returncode != 0: + raise RuntimeError( + f"ffmpeg failed (returncode={result.returncode}):\n" + f"CMD: {' '.join(cmd)}\n" + f"STDERR:\n{result.stderr[-2000:]}" + ) + + if not dst.exists() or dst.stat().st_size == 0: + raise RuntimeError(f"ffmpeg produced empty output: {dst}") + + return (str(dst),) + + @classmethod + def IS_CHANGED(cls, video_path, sample_rate, mono, ffmpeg_extra_args=""): + try: + mtime = Path(video_path).stat().st_mtime + except Exception: + mtime = 0 + return f"{video_path}|{mtime}|{sample_rate}|{mono}|{ffmpeg_extra_args}" diff --git a/nodes/load_video.py b/nodes/load_video.py new file mode 100644 index 0000000..9dfea5e --- /dev/null +++ b/nodes/load_video.py @@ -0,0 +1,68 @@ +"""LoadVideo node - resolve a video file path from ComfyUI/input or absolute path.""" + +import os +from pathlib import Path + +import folder_paths + +from ._types import CATEGORY, VIDEO_EXTS + + +class LoadVideoForASR: + """Pick a video from ComfyUI/input (or pass an absolute path override) and emit its path as STRING.""" + + @classmethod + def INPUT_TYPES(cls): + input_dir = folder_paths.get_input_directory() + files = [] + if os.path.isdir(input_dir): + for f in sorted(os.listdir(input_dir)): + ext = os.path.splitext(f)[1].lower() + if ext in VIDEO_EXTS: + files.append(f) + + # Always include the dropdown widget; fall back to a placeholder if no files. + video_widget = (files if files else [""],) + + return { + "required": { + "video": video_widget, + }, + "optional": { + "video_path_override": ( + "STRING", + { + "default": "", + "multiline": False, + "tooltip": "Absolute path. If non-empty, overrides the dropdown.", + }, + ), + }, + } + + RETURN_TYPES = ("STRING",) + RETURN_NAMES = ("video_path",) + FUNCTION = "load" + CATEGORY = CATEGORY + + def load(self, video, video_path_override=""): + if video_path_override.strip(): + p = Path(video_path_override).expanduser().resolve() + else: + p = Path(folder_paths.get_input_directory()) / video + + if not p.exists() or not p.is_file(): + raise FileNotFoundError(f"Video not found: {p}") + + return (str(p),) + + @classmethod + def IS_CHANGED(cls, video, video_path_override=""): + if video_path_override.strip(): + target = Path(video_path_override).expanduser() + else: + target = Path(folder_paths.get_input_directory()) / video + try: + return f"{target}|{target.stat().st_mtime}" + except Exception: + return str(target) diff --git a/nodes/save_transcript.py b/nodes/save_transcript.py new file mode 100644 index 0000000..40155f1 --- /dev/null +++ b/nodes/save_transcript.py @@ -0,0 +1,73 @@ +"""SaveTranscript node - persist transcript JSON to ComfyUI/output.""" + +import json +import re +from datetime import datetime +from pathlib import Path + +import folder_paths + +from ._types import CATEGORY + + +_SAFE_NAME = re.compile(r"[^A-Za-z0-9_.\-]+") + + +def _sanitize(name: str) -> str: + name = _SAFE_NAME.sub("_", name).strip("_") + return name or "transcript" + + +class SaveTranscript: + """Validate and write transcript JSON to ComfyUI/output/[_timestamp].json.""" + + @classmethod + def INPUT_TYPES(cls): + return { + "required": { + "transcript_json": ("STRING", {"forceInput": True}), + "filename_prefix": ("STRING", {"default": "transcript"}), + }, + "optional": { + "include_timestamp": ("BOOLEAN", {"default": True}), + "subfolder": ("STRING", {"default": ""}), + }, + } + + RETURN_TYPES = ("STRING",) + RETURN_NAMES = ("file_path",) + FUNCTION = "save" + CATEGORY = CATEGORY + OUTPUT_NODE = True + + def save(self, transcript_json, filename_prefix, include_timestamp=True, subfolder=""): + # Validate JSON before writing — fail fast on garbage input. + try: + data = json.loads(transcript_json) + except json.JSONDecodeError as e: + raise ValueError(f"transcript_json is not valid JSON: {e}") from e + + out_root = Path(folder_paths.get_output_directory()) + out_dir = out_root / _sanitize(subfolder) if subfolder.strip() else out_root + out_dir.mkdir(parents=True, exist_ok=True) + + prefix = _sanitize(filename_prefix) + if include_timestamp: + ts = datetime.now().strftime("%Y%m%d_%H%M%S") + name = f"{prefix}_{ts}.json" + else: + name = f"{prefix}.json" + + dst = out_dir / name + dst.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8") + + # OUTPUT_NODE convention: return UI dict so the file shows up in the ComfyUI history panel. + rel_sub = str(out_dir.relative_to(out_root)) if out_dir != out_root else "" + ui = { + "ui": { + "text": [str(dst)], + "files": [{"filename": name, "subfolder": rel_sub, "type": "output"}], + }, + "result": (str(dst),), + } + return ui diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..514f22a --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,39 @@ +[build-system] +requires = ["setuptools>=68", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "comfyui-videototext" +version = "0.2.1" +description = "ComfyUI nodes for video -> audio -> JSON transcript pipeline. Designed to pair with ComfyUI-MiMoASR for the ASR step." +readme = "README.md" +license = { text = "MIT" } +authors = [{ name = "aadebuger" }] +requires-python = ">=3.10" +dependencies = [ + "torchaudio", +] +keywords = [ + "comfyui", + "asr", + "mimo", + "speech-to-text", + "transcription", + "video", +] +classifiers = [ + "License :: OSI Approved :: MIT License", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Topic :: Multimedia :: Sound/Audio :: Speech", +] + +[project.urls] +Repository = "https://github.com/aadebuger/ComfyUI-VideoToText" +Issues = "https://github.com/aadebuger/ComfyUI-VideoToText/issues" + +[tool.setuptools] +packages = ["nodes"] diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..d220393 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,3 @@ +# ComfyUI-VideoToText itself only needs torchaudio for the AudioFromPath bridge. +# torch comes from ComfyUI's own install. +torchaudio diff --git a/scripts/install_server.sh b/scripts/install_server.sh new file mode 100755 index 0000000..b7bdaa4 --- /dev/null +++ b/scripts/install_server.sh @@ -0,0 +1,103 @@ +#!/bin/bash +# +# install_server.sh — idempotent server-side install for ComfyUI-VideoToText. +# +# Usage: +# COMFYUI_DIR=/mnt/modelall/ComfyUI ./scripts/install_server.sh +# +# What it does: +# 1. Verifies ffmpeg is on PATH +# 2. Installs torchaudio into ComfyUI's venv (uv-managed) +# 3. Smoke-tests the imports +# +# What it does NOT do: +# - Install MiMoASR (use that pack's own install_server.sh) +# - Install rgthree-comfy (clone it manually if needed) +# - Download model weights +# - Restart ComfyUI (you do this after; see README Step 7) +# +set -euo pipefail + +# ----- helpers ----- +log() { printf '\033[1;34m[+]\033[0m %s\n' "$*"; } +warn() { printf '\033[1;33m[!]\033[0m %s\n' "$*"; } +err() { printf '\033[1;31m[x]\033[0m %s\n' "$*" >&2; } + +# ----- 1. ffmpeg ----- +log "Checking ffmpeg ..." +if ! command -v ffmpeg >/dev/null 2>&1; then + err "ffmpeg not on PATH. Install with: sudo apt install ffmpeg" + exit 1 +fi +ffmpeg -version | head -1 + +# ----- 2. ComfyUI venv ----- +: "${COMFYUI_DIR:?Please set COMFYUI_DIR (e.g. /mnt/modelall/ComfyUI)}" +if [[ ! -d "$COMFYUI_DIR/.venv" ]]; then + err "ComfyUI venv not found: $COMFYUI_DIR/.venv" + exit 1 +fi + +log "Activating ComfyUI venv ..." +# shellcheck disable=SC1091 +source "$COMFYUI_DIR/.venv/bin/activate" + +# Pick uv if available; fall back to pip +if command -v uv >/dev/null 2>&1; then + PIP="uv pip" +else + PIP="pip" +fi + +# ----- 3. torchaudio ----- +log "Installing torchaudio ..." +$PIP install torchaudio + +log "Smoke-test imports ..." +python - <<'PY' +import sys +ok = True +try: + import torchaudio + print(f" torchaudio {torchaudio.__version__} OK") +except Exception as e: + print(f" torchaudio FAILED: {e}"); ok = False + +try: + import folder_paths # ComfyUI core module + print(f" folder_paths OK ({folder_paths.__file__})") +except Exception as e: + print(f" folder_paths (ComfyUI core): cannot import outside ComfyUI runtime — that's normal, skipping") + +sys.exit(0 if ok else 1) +PY + +# ----- 4. final advice ----- +cat <<'NEXT' + +[+] torchaudio installed. + +Next steps (manual): + 1. If not yet installed: clone ComfyUI-MiMoASR and rgthree-comfy + $ cd "$COMFYUI_DIR/custom_nodes" + $ git clone https://github.com/aadebuger/ComfyUI-MiMoASR.git + $ git clone https://github.com/rgthree/rgthree-comfy.git + + 2. Download MiMo weights (~24 GB): + $ export HF_HOME=/mnt/modelall/hf_cache + $ hf download XiaomiMiMo/MiMo-V2.5-ASR --local-dir /mnt/modelall/models/MiMo-V2.5-ASR + $ hf download XiaomiMiMo/MiMo-Audio-Tokenizer --local-dir /mnt/modelall/models/MiMo-Audio-Tokenizer + + 3. Clone MiMo upstream repo: + $ git clone https://github.com/XiaomiMiMo/MiMo-V2.5-ASR.git /opt/repos/MiMo-V2.5-ASR + + 4. Restart ComfyUI with required env vars: + $ pkill -9 -f main.py; sleep 3 + $ export HF_HOME=/mnt/modelall/hf_cache + $ export MIMO_ASR_REPO=/opt/repos/MiMo-V2.5-ASR + $ cd "$COMFYUI_DIR" && source .venv/bin/activate + $ nohup python main.py --listen 0.0.0.0 --port 8188 > ~/comfy_start.log 2>&1 & + + 5. Verify 9 nodes registered (see README Step 8). + +NEXT