Compare commits

...
Author SHA1 Message Date
diodiogod 73a282406e Add Audio8 TTS engine integration 2026-07-31 07:49:24 -03:00
36 changed files with 3237 additions and 170 deletions
+1
View File
@@ -44,6 +44,7 @@ The project code is MIT. Model weights carry their own licenses:
Echo-TTS CC-BY-NC-SA-4.0 No
Fish Audio S2 Pro Fish Audio Research License No
Dots TTS Apache-2.0 Yes
Audio8 TTS Apache-2.0 Yes
DramaBox LTX-2 Community License Conditional
OmniVoice Apache-2.0 Yes
MOSS-TTS Apache-2.0 Yes
+7 -5
View File
@@ -17,7 +17,7 @@
**Key architectural rules:**
- Chunking happens in the **processor**, not the adapter (`generate_single()` on adapter = raw single call)
- Runtime routing happens through `ModelLoadConfig.runtime_mode` + `runtime_profile`, not ad-hoc subprocess calls
- Shared runtime workers are currently used for fragile engine families such as VibeVoice, Qwen3-TTS / ASR, Granite forced alignment, and Higgs Audio 2. Engines that support the modern stack run natively in the main Transformers 5 environment.
- Shared runtime workers are currently used for fragile engine families such as VibeVoice, Qwen3-TTS / ASR, Granite forced alignment, Higgs Audio 2, and Audio8 TTS. Engines that support the modern stack run natively in the main Transformers 5 environment.
- YAML (`docs/Dev reports/tts_audio_suite_engines.yaml`) is source of truth for engine doc tables → run `python3 scripts/generate_engine_tables.py --readme` to regenerate
- Auxiliary YAML (`docs/Dev reports/tts_audio_suite_aux_models.yaml`) is source of truth for helper/post-process model docs → run `python3 scripts/generate_aux_model_docs.py`
- All models download to `ComfyUI/models/TTS/<model-name>/`
@@ -25,7 +25,7 @@
## Engines
19 engines follow the pattern above:
20 engines follow the pattern above:
| Engine | Adapter | Processor | SRT Processor | Engine Node |
|--------|---------|-----------|---------------|-------------|
@@ -44,13 +44,14 @@
| Echo-TTS | `echo_tts_adapter.py` | `nodes/echo_tts/echo_tts_processor.py` | `echo_tts_srt_processor.py` | `echo_tts_engine_node.py` |
| Fish Audio S2 Pro | `fish_audio_s2_adapter.py` | `nodes/fish_audio_s2/fish_audio_s2_processor.py` | `fish_audio_s2_srt_processor.py` | `fish_audio_s2_engine_node.py` |
| Dots TTS | `dots_tts_adapter.py` | `nodes/dots_tts/dots_tts_processor.py` | `dots_tts_srt_processor.py` | `dots_tts_engine_node.py` |
| Audio8 TTS | `audio8_tts_adapter.py` | `nodes/audio8_tts/audio8_tts_processor.py` | `audio8_tts_srt_processor.py` | `audio8_tts_engine_node.py` |
| DramaBox | `dramabox_adapter.py` | `nodes/dramabox/dramabox_processor.py` | `dramabox_srt_processor.py` | `dramabox_engine_node.py` |
| OmniVoice | `omnivoice_adapter.py` | `nodes/omnivoice/omnivoice_processor.py` | `omnivoice_srt_processor.py` | `omnivoice_engine_node.py` |
| MOSS-SoundEffect v2 | `moss_soundeffect_v2_adapter.py` | — | — | `moss_soundeffect_v2_engine_node.py` |
| RVC | — | `engines/rvc/` | — | `rvc_engine_node.py` |
**Engine implementations live in:**
- `engines/chatterbox/`, `engines/chatterbox_official_23lang/`, `engines/f5tts/`, `engines/higgs_audio/`, `engines/higgs_audio_v3/`, `engines/vibevoice_engine/`, `engines/step_audio_editx/`, `engines/cosyvoice/`, `engines/qwen3_tts/`, `engines/qwen3_asr/`, `engines/moss_tts/`, `engines/moss_soundeffect_v2/`, `engines/granite_asr/`, `engines/echo_tts/`, `engines/fish_audio_s2/`, `engines/dots_tts/`, `engines/dramabox/`, `engines/omnivoice/`, `engines/rvc/`
- `engines/chatterbox/`, `engines/chatterbox_official_23lang/`, `engines/f5tts/`, `engines/higgs_audio/`, `engines/higgs_audio_v3/`, `engines/vibevoice_engine/`, `engines/step_audio_editx/`, `engines/cosyvoice/`, `engines/qwen3_tts/`, `engines/qwen3_asr/`, `engines/moss_tts/`, `engines/moss_soundeffect_v2/`, `engines/granite_asr/`, `engines/echo_tts/`, `engines/fish_audio_s2/`, `engines/dots_tts/`, `engines/audio8_tts/`, `engines/dramabox/`, `engines/omnivoice/`, `engines/rvc/`
## Documentation Files
@@ -145,13 +146,14 @@
- `launcher.py` - runtime bootstrap, venv creation, Windows toolchain env setup
- `session.py`, `protocol.py` - JSONL worker transport and message protocol
- `bootstrap.py` - shared runtime bootstrap helpers
- `vibevoice_proxy.py`, `qwen3_tts_proxy.py`, `qwen3_asr_proxy.py`, `higgs_audio_proxy.py` - parent-process proxies
- `workers/` - worker subprocess entrypoints for VibeVoice, Qwen3-TTS, Qwen3-ASR/aligner, Higgs Audio
- `vibevoice_proxy.py`, `qwen3_tts_proxy.py`, `qwen3_asr_proxy.py`, `higgs_audio_proxy.py`, `audio8_tts_proxy.py` - parent-process proxies
- `workers/` - worker subprocess entrypoints for VibeVoice, Qwen3-TTS, Qwen3-ASR/aligner, Higgs Audio, and Audio8 TTS
- Current shared legacy T4 runtime profile is reused by:
- VibeVoice / Kugel
- Qwen3-TTS
- Qwen3-ASR and Granite's optional Qwen forced aligner
- Higgs Audio 2
- Audio8 TTS
### Audio (`utils/audio/`)
- `processing.py` - Tensor manipulation, normalization, format conversion
+3 -1
View File
@@ -23,7 +23,7 @@ Subtitle workflows are still a core focus: the suite can transcribe to SRT, rebu
<!-- ENGINE_COMPARISON_START -->
## Quick Engine Comparison — 19 Engines
## Quick Engine Comparison — 20 Engines
| Engine | Languages | Size | Key Features |
|--------|-----------|------|--------------|
@@ -41,6 +41,7 @@ Subtitle workflows are still a core focus: the suite can transcribe to SRT, rebu
| **Echo-TTS** | 🇺🇸 | ~5.3GB + ~1.8GB | Diffusion-based (~30s best), Force Speaker KV (speaker drift control) |
| **Fish Audio S2 Pro** | 🌐 80+ languages | ~10.3GB / ~8.0GB | Free-form sub-word emotion/prosody tags, Native multi-speaker and multi-turn dialogue with dynamic speaker references |
| **Dots TTS** | 🇺🇸​🇨🇳​🇩🇪​🇪🇸​🇫🇷​🇮🇹 +13 | ~6GB | Official auto language detect / language control, SOAR and MeanFlow distilled variants |
| **Audio8 TTS** | 🌐 11 recommended languages | ~2.39 GiB | Reference-free TTS + zero-shot/cross-lingual cloning, 44.1kHz output with sampling or greedy decoding |
| **DramaBox** | 🇺🇸 | ~16.4GB | Expressive scene prompting and stage directions, Native and SRT-aware duration targeting |
| **OmniVoice** | 🌐 600+ languages | ~3.7GB | Inline non-verbal tags and pronunciation overrides, Reference-free voice design |
| **MOSS-TTS** | 🇺🇸​🇨🇳​🇩🇪​🇪🇸​🇫🇷​🇮🇹 +18 | ~8.5GB tokenizer + ~6.1GB/17GB/18GB model | Reference-free voice design with MOSS-VoiceGenerator, Native 1-5 speaker TTSD dialogue |
@@ -1527,6 +1528,7 @@ For offline/manual setup:
| Granite ASR | `ComfyUI/models/TTS/granite_asr/` | ✅ | Granite ASR models; plus adds native diarization/timestamps, optional Qwen forced aligner reused lazily for timestamps/SRT fallback |
| Echo-TTS | `ComfyUI/models/TTS/echo-tts-base/` | ✅ | ~7.1GB total (base + dac); CC-BY-NC-SA |
| Dots TTS | `ComfyUI/models/TTS/dots_tts/` | ✅ | Official base / soar / mf checkpoints with tokenizer, vocoder, speaker encoder |
| Audio8 TTS | `ComfyUI/models/TTS/audio8_tts/Audio8-TTS-Preview-0.6b/` | ✅ | ~2.39 GiB official 0.6B Preview checkpoint; shared Transformers 4.57.3 runtime required because main 5.10.2 collapses voice cloning |
| DramaBox | `ComfyUI/models/TTS/dramabox/DramaBox/` | ✅ | ~16.4GB download; fast mode roughly 24GB VRAM; experimental FP8 peaks on RTX 4090: staged ~15.1GB allocated, sequential ~11.7GB allocated / ~12.4GB reserved; conditional LTX-2 Community License |
| Fish Audio S2 Pro | `ComfyUI/models/TTS/fish_audio_s2_pro/` | ✅ | Official BF16 or optional community FP8 checkpoint; the official checkpoint can be quantized on load with BNB INT8/NF4; main T5 environment with process teardown for Clear VRAM; Fish Audio Research License |
| OmniVoice | `ComfyUI/models/TTS/omnivoice/` | ✅ | Official OmniVoice model. Voice cloning in this suite requires explicit reference text. |
+95 -2
View File
@@ -1339,6 +1339,77 @@ engines:
speed_performance: { supported: "partial", notes: "Moderate; mf variant is faster" }
reference_free_tts: { supported: true, notes: "(default speaker)" }
- id: audio8_tts
name: Audio8 TTS
models: "Audio8 TTS Preview 0.6B"
size: "~2.39 GiB"
license: "Apache-2.0"
commercial: true
language_summary_full: "11 recommended languages (including Cantonese)"
language_summary_compact: "🌐 11 recommended languages"
capabilities:
tts: true
srt: true
vc: false
asr: false
training: false
runtime_isolation:
default_mode: "shared_runtime"
main_environment: false
shared_runtime: true
dedicated_runtime: false
runtime_profile: "vibevoice_transformers4_shared"
notes: "Required for feature-complete Audio8 inference: main Transformers 5.10.2 collapsed the official voice-clone demo to 0.325s, while the shared Transformers 4.57.3 runtime generated 10.495s for the published 10.54s demo"
readme_key_features:
- "Reference-free TTS + zero-shot/cross-lingual cloning"
- "44.1kHz output with sampling or greedy decoding"
special_features:
- "44.1kHz reference-free TTS and zero-shot/cross-lingual voice cloning"
- "Exact reference transcript required when voice cloning"
- "Greedy or sampled decoding with temperature, top-p, top-k, seed, and token-limit controls"
- "Suite text chunking, character switching, and SRT timing/assembly"
- "No explicit language selector; multilingual behavior is text-driven"
- "No native emotion/style/speed controls, streaming, timestamps, dialogue/multi-speaker mode, or presets"
- "Not compatible with Voice Designer: the official model has no instruction-conditioned voice-design mode"
- "Unified TTS/SRT integration only; no special node or suite training integration"
model_sources:
- component: "Audio8 TTS Preview 0.6B"
source_name: "Audio8/Audio8-TTS-Preview-0.6b"
source_url: "https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6b"
size: "~2.39 GiB"
auto_download: true
notes: "Official 0.6B Preview checkpoint with bundled 44.1kHz codec; suite uses the shared Transformers 4.57.3 runtime because voice cloning is not compatible with main Transformers 5.10.2"
languages:
en: { supported: true, flag: "🇺🇸", notes: "" }
zh: { supported: true, flag: "🇨🇳", notes: "(Chinese; upstream separately recommends Cantonese, folded into this matrix row)" }
de: { supported: true, flag: "🇩🇪", notes: "" }
es: { supported: true, flag: "🇪🇸", notes: "" }
fr: { supported: true, flag: "🇫🇷", notes: "" }
it: { supported: true, flag: "🇮🇹", notes: "" }
ja: { supported: true, flag: "🇯🇵", notes: "" }
ko: { supported: true, flag: "🇰🇷", notes: "" }
pl: { supported: true, flag: "🇵🇱", notes: "" }
nl: { supported: true, flag: "🇳🇱", notes: "" }
features:
voice_cloning: { supported: true, notes: "Zero-shot and cross-lingual; reference audio must be paired with its exact transcript" }
reference_transcript: { requirement: conditional }
native_multi_speaker: { supported: false, notes: "No native dialogue or multi-speaker interface; suite character switching generates speakers as separate segments" }
voice_conversion: { supported: false, notes: "No separate voice-conversion mode" }
asr_transcribe: { supported: false, notes: "" }
emotion_control: { supported: false, notes: "No native emotion, style, or speed controls" }
native_long_form: { supported: false, notes: "(uses suite text chunking and SRT assembly)" }
community_finetunes: { supported: false, notes: "No community model or upstream SFT/training integration in the suite" }
vram_efficient: { supported: "partial", notes: "~1.75 GiB model-plus-codec parameter memory at fp16/bf16 or ~3.5 GiB at float32/CPU, before KV cache, activations, and framework overhead; shared runtime reuses the main environment's PyTorch" }
speed_performance: { supported: "partial", notes: "Live shared-runtime validation: 6.36s reference-free audio in 39.3s including load; 9.99s cloned audio in 36.7s warm; exact cache reuse in 56ms" }
reference_free_tts: { supported: true, notes: "Reference audio is optional" }
- id: dramabox
name: DramaBox
models: "DramaBox 3.3B"
@@ -1903,8 +1974,9 @@ table_notes:
reference_transcript: >-
Conditional means the transcript is required only for the specific mode:
CosyVoice3 zero-shot, Qwen3-TTS full Base cloning, or MOSS-TTSD cloned-speaker
dialogue. Higgs Audio 2, Higgs Audio v3, and Dots TTS accept matching text
when provided but do not require it.
dialogue, and Audio8 TTS zero-shot/cross-lingual cloning. Higgs Audio 2,
Higgs Audio v3, and Dots TTS accept matching text when provided but do not
require it.
language_support:
- "**CosyVoice3 Chinese**: Includes 18+ dialects (Cantonese, Sichuan, Dongbei, Shanghai, etc.)"
- "**Higgs Audio 2**: Trained on EN, ZH (Mandarin), KO, DE, ES (English majority) - 10M hours AudioVerse dataset"
@@ -1912,6 +1984,7 @@ table_notes:
- "**IndexTTS-2**: Trained on 55K+ hours - ZH, EN, JA primary."
- "**MOSS-TTS**: Official list includes ZH, EN, DE, ES, FR, JA, IT, HU, KO, RU, FA, AR, PL, PT, CS, DA, SV, EL, TR; HU/FA/CS are not separate columns in this matrix."
- "**OmniVoice**: Official model supports 600+ languages; this matrix only shows the suite's comparison subset."
- "**Audio8 TTS**: The Preview recommends Cantonese, Chinese, Dutch, English, French, German, Italian, Japanese, Korean, Polish, and Spanish. It has no explicit language selector; language behavior is text-driven. Cantonese is folded into the Chinese row in this matrix."
- "**RVC**: Language-agnostic voice conversion/post-processing; it is marked supported for every language row."
- "Qwen3-TTS: Supports pt-BR only with instruction (voice design or harcoded voices). Base can't do instructions, so it will always ouput pt-PT."
@@ -1980,6 +2053,10 @@ readme_model_download_table:
primary_model_path: "ComfyUI/models/TTS/dots_tts/"
auto_download: "✅"
notes: "Official base / soar / mf checkpoints with tokenizer, vocoder, speaker encoder"
- engine: "Audio8 TTS"
primary_model_path: "ComfyUI/models/TTS/audio8_tts/Audio8-TTS-Preview-0.6b/"
auto_download: "✅"
notes: "~2.39 GiB official 0.6B Preview checkpoint; shared Transformers 4.57.3 runtime required because main 5.10.2 collapses voice cloning"
- engine: "DramaBox"
primary_model_path: "ComfyUI/models/TTS/dramabox/DramaBox/"
auto_download: "✅"
@@ -2423,6 +2500,22 @@ model_layouts_markdown: |
- Native sample rate is 48kHz.
- Main-environment support works on Transformers 5; on Windows, `normalize_text` falls back to no-op if `WeTextProcessing` is unavailable.
## Audio8 TTS
```text
ComfyUI/models/TTS/audio8_tts/
└── Audio8-TTS-Preview-0.6b/
└── (13 required official runtime files, including codec weights)
```
Notes:
- The official `Audio8/Audio8-TTS-Preview-0.6b` checkpoint is about 2.39 GiB and includes the 44.1kHz neural codec.
- The 0.6B model plus codec use about 1.75 GiB for parameters at fp16/bf16 or about 3.5 GiB at float32/CPU, before KV cache, activations, and framework overhead.
- The suite uses the existing `vibevoice_transformers4_shared` runtime with Transformers 4.57.3. Main Transformers 5.10.2 is not compatible with Audio8 voice cloning: the official demo collapsed to 0.325s, while the shared runtime generated 10.495s for the published 10.54s demo.
- Live shared-runtime validation produced 6.36s of reference-free audio in 39.3s including model load, 9.99s of cloned audio in 36.7s warm, and exact cache reuse in 56ms.
- The engine has unified TTS and SRT support only: no Voice Designer support, special node, training integration, native streaming/timestamps/dialogue, or built-in presets.
## OmniVoice
```text
+1
View File
@@ -18,6 +18,7 @@
| **Echo-TTS** | Main | echo-tts-base + fish-s1-dac-min | ~5.3GB + ~1.8GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | CC-BY-NC-SA-4.0 | Diffusion-based (~30s best), Force Speaker KV (speaker drift control) | 1 |
| **Fish Audio S2 Pro** | Main | S2 Pro 4B / FP8 | ~10.3GB / ~8.0GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Fish Audio Research License | Free-form sub-word emotion/prosody tags, Native multi-speaker and multi-turn dialogue with dynamic speaker references, Zero-shot voice cloning, Optional per-segment custom character switching, Configurable 4K-32K native context with reduced KV-cache VRAM, Optional community FP8 weight-only checkpoint with BF16 activations, Optional on-the-fly BitsAndBytes INT8/NF4 for the official checkpoint | 80+ languages |
| **Dots TTS** | Main | dots.tts-base, dots.tts-soar, dots.tts-mf | ~6GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 | Official auto language detect / language control, SOAR and MeanFlow distilled variants | 19 |
| **Audio8 TTS** | Shared | Audio8 TTS Preview 0.6B | ~2.39 GiB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 | 44.1kHz reference-free TTS and zero-shot/cross-lingual voice cloning, Exact reference transcript required when voice cloning, Greedy or sampled decoding with temperature, top-p, top-k, seed, and token-limit controls, Suite text chunking, character switching, and SRT timing/assembly, No explicit language selector; multilingual behavior is text-driven, No native emotion/style/speed controls, streaming, timestamps, dialogue/multi-speaker mode, or presets, Not compatible with Voice Designer: the official model has no instruction-conditioned voice-design mode, Unified TTS/SRT integration only; no special node or suite training integration | 11 recommended languages (including Cantonese) |
| **DramaBox** | Main | DramaBox 3.3B | ~16.4GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | LTX-2 Community License | Expressive scene prompting and stage directions, Native and SRT-aware duration targeting, Official duration-aware long-form chunking with scene-prefix preservation, Optional 10-second zero-shot voice reference, CFG negative prompt with per-segment switching, Explicit generation/reference durations, CFG rescale control, and optional Perth watermark, Experimental staged and sequential strategies for lowering peak VRAM; official LTX FP8-cast storage, Optional official torch.compile path | 1 |
| **OmniVoice** | Main | OmniVoice | ~3.7GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 | Inline non-verbal tags and pronunciation overrides, Reference-free voice design, 600+ language support, Upstream long-form chunk orchestration | 600+ |
| **MOSS-TTS** | Main | Local 1.7B, Delay 8B v1.5/1.0, VoiceGenerator 1.7B, SoundEffect 8B v1, TTSD 8B | ~8.5GB tokenizer + ~6.1GB/17GB/18GB model | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | Apache-2.0 | Reference-free voice design with MOSS-VoiceGenerator, Native 1-5 speaker TTSD dialogue, 31-language generation with MOSS-TTS-v1.5, Prompt-only sound-effect generation with MOSS-SoundEffect v1, Long-form generation (TTSD/Delay), Duration token hint, Local/Delay/TTSD variants, Initial integrated LoRA training workflow (Delay 8B) | 24 |
+18 -18
View File
@@ -2,22 +2,22 @@
## Feature Comparison Matrix
| Feature | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | DramaBox | OmniVoice | MOSS-TTS | MOSS-SoundEffect v2 | RVC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| **TTS** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| **SRT** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| **Voice Conversion** | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| **ASR (Transcribe)** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **Sound Effects** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ |
| **Training** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
| **Voice Cloning** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (Base model) | ❌ | ✅ | ✅ | ✅ Reference audio plus exact transcript | ✅ | ✅ Optional reference audio; configurable 3-30 second window from the beginning of the source, default 10 seconds | ✅ | ✅ | ❌ | ⚠️ (needs training) |
| **Reference Transcript†** | **Required** | Not used | Not used | Not used | Optional | Optional | Not used | Conditional | Conditional | N/A | **Required** | Not used | **Required** | Optional | Not used | **Required** | Conditional | Not used | N/A |
| **Native Multi-Speaker** | ❌ | ❌ | ❌ | ✅ (Base only, Kugel uses fallback) | ✅ | ❌ | ❌ | ❌ | ❌ | ⚠️ (Plus variant speaker attribution / diarization) | ❌ | ❌ | ✅ Native dialogue is default; optional custom mode generates each [Character] segment independently as local speaker 0 | ❌ | ❌ | ❌ | ✅ (TTSD v1.0; 1-5 speakers) | ❌ | ❌ |
| **Emotion Control** | ❌ | ❌ | ⚠️ (v2 tags - doesn't work) | ❌ | ⚠️ (via prompt) | ✅ (native inline tags) | ✅ (8 emotions) | ⚠️ (via instruct) | ⚠️ (via instruct) | ❌ | ✅ (14 emotions) | ❌ | ✅ Free-form inline natural-language tags | ❌ | ✅ Natural-language scene prompt and stage directions | ⚠️ (voice-design instruct + inline non-verbal tags) | ✅ (MOSS-VoiceGenerator instruction-conditioned voice design) | ❌ | ❌ |
| **Native Long-form** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Configurable 4K-32K native context; suite text chunking is bypassed | ❌ | ✅ Official duration-aware quote-group chunking; ~37s target / 45s cap | ✅ (uses upstream audio_chunk_duration / audio_chunk_threshold orchestration; bypasses suite char-based chunk splitting) | ✅ (TTSD/Delay long-form; use chunk orchestration for very long inputs) | ❌ | N/A |
| **Community Finetunes** | ✅ | ✅ | ✅ | ✅ KugelAudio, Hindi | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (LoRA adapter inference supported; initial integrated LoRA training support added for MOSS-TTS Delay 8B) | ❌ | ✅ |
| **VRAM Efficient** | ✅ | ✅ | ✅ | ⚠️ (5-18GB) | ⚠️ (9GB) | ⚠️ (~8-10GB) | ⚠️ (9-12GB) | ✅ (5.4GB) | ✅ (3-6GB) | ✅ (~4.6GB) | ⚠️ (7GB) | ⚠️ (~7GB total) | ⚠️ 8K context measured at ~15.2GB BF16, ~11.2GB FP8, ~11.2GB BNB INT8, or ~8.9GB BNB NF4; BF16 codec and activations; BNB is a load-time option for the official checkpoint | ⚠️ (main env works; 2B-class model is not lightweight) | ✅ Fast mode is ~24GB; experimental FP8 peaks on RTX 4090: staged ~15.1GB allocated, sequential ~11.7GB allocated / ~12.4GB reserved | ⚠️ (main env works; 2B-class model is not lightweight) | ⚠️ (Local 1.7B smaller; tokenizer is large) | ⚠️ (runs in the main ComfyUI environment but remains GPU-heavy) | ✅ |
| **Speed/Performance** | ✅ Very Fast | ✅ Fast | ✅ Fast | ⚠️ | ⚠️ | ⚠️ CUDA recommended | ⚠️ | ✅ Fast | ⚠️ | ⚠️ Moderate | ⚠️ | ✅ Fast (diffusion, realtime-capable) | ✅ Main-environment subprocess with reliable teardown; local compile measurements: ~40 it/s BF16, ~11.8 it/s NF4 at ~8.9GB VRAM, and ~3.7 it/s INT8 at ~11.2GB VRAM; quality comparison pending | ⚠️ Moderate; mf variant is faster | ⚠️ Fast favors repeated generation; staged reloads temporary components; sequential also transfers the transformer; optional block-level torch.compile accelerates repeat denoising | ✅ Fast; upstream reports sub-realtime RTF | ✅ Fast with CUDA/FlashAttention | ⚠️ (100 diffusion steps by default) | ✅ Fast |
| **No Narrator Required** | ❌ | ✅ (default speaker) | ✅ (default speaker) | ✅ (zero-shot / default speaker) | ✅ (basic TTS if no narrator/reference is provided) | ✅ (zero-shot) | ❌ | ✅ (cross-lingual or instruct mode) | ✅ (Base default voice or CustomVoice presets) | N/A | ❌ | ❌ | ✅ Reference audio is optional | ✅ (default speaker) | ✅ Voice reference is optional | ✅ (native default voice; instruct also works without narrator) | ✅ (direct TTS and prompt-only generation) | ❌ | N/A |
| Feature | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | Audio8 TTS | DramaBox | OmniVoice | MOSS-TTS | MOSS-SoundEffect v2 | RVC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| **TTS** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| **SRT** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
| **Voice Conversion** | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| **ASR (Transcribe)** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **Sound Effects** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ |
| **Training** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
| **Voice Cloning** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (Base model) | ❌ | ✅ | ✅ | ✅ Reference audio plus exact transcript | ✅ | ✅ Zero-shot and cross-lingual; reference audio must be paired with its exact transcript | ✅ Optional reference audio; configurable 3-30 second window from the beginning of the source, default 10 seconds | ✅ | ✅ | ❌ | ⚠️ (needs training) |
| **Reference Transcript†** | **Required** | Not used | Not used | Not used | Optional | Optional | Not used | Conditional | Conditional | N/A | **Required** | Not used | **Required** | Optional | Conditional | Not used | **Required** | Conditional | Not used | N/A |
| **Native Multi-Speaker** | ❌ | ❌ | ❌ | ✅ (Base only, Kugel uses fallback) | ✅ | ❌ | ❌ | ❌ | ❌ | ⚠️ (Plus variant speaker attribution / diarization) | ❌ | ❌ | ✅ Native dialogue is default; optional custom mode generates each [Character] segment independently as local speaker 0 | ❌ | ❌ | ❌ | ❌ | ✅ (TTSD v1.0; 1-5 speakers) | ❌ | ❌ |
| **Emotion Control** | ❌ | ❌ | ⚠️ (v2 tags - doesn't work) | ❌ | ⚠️ (via prompt) | ✅ (native inline tags) | ✅ (8 emotions) | ⚠️ (via instruct) | ⚠️ (via instruct) | ❌ | ✅ (14 emotions) | ❌ | ✅ Free-form inline natural-language tags | ❌ | ❌ | ✅ Natural-language scene prompt and stage directions | ⚠️ (voice-design instruct + inline non-verbal tags) | ✅ (MOSS-VoiceGenerator instruction-conditioned voice design) | ❌ | ❌ |
| **Native Long-form** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Configurable 4K-32K native context; suite text chunking is bypassed | ❌ | ❌ | ✅ Official duration-aware quote-group chunking; ~37s target / 45s cap | ✅ (uses upstream audio_chunk_duration / audio_chunk_threshold orchestration; bypasses suite char-based chunk splitting) | ✅ (TTSD/Delay long-form; use chunk orchestration for very long inputs) | ❌ | N/A |
| **Community Finetunes** | ✅ | ✅ | ✅ | ✅ KugelAudio, Hindi | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (LoRA adapter inference supported; initial integrated LoRA training support added for MOSS-TTS Delay 8B) | ❌ | ✅ |
| **VRAM Efficient** | ✅ | ✅ | ✅ | ⚠️ (5-18GB) | ⚠️ (9GB) | ⚠️ (~8-10GB) | ⚠️ (9-12GB) | ✅ (5.4GB) | ✅ (3-6GB) | ✅ (~4.6GB) | ⚠️ (7GB) | ⚠️ (~7GB total) | ⚠️ 8K context measured at ~15.2GB BF16, ~11.2GB FP8, ~11.2GB BNB INT8, or ~8.9GB BNB NF4; BF16 codec and activations; BNB is a load-time option for the official checkpoint | ⚠️ (main env works; 2B-class model is not lightweight) | ⚠️ ~1.75 GiB model-plus-codec parameter memory at fp16/bf16 or ~3.5 GiB at float32/CPU, before KV cache, activations, and framework overhead; shared runtime reuses the main environment's PyTorch | ✅ Fast mode is ~24GB; experimental FP8 peaks on RTX 4090: staged ~15.1GB allocated, sequential ~11.7GB allocated / ~12.4GB reserved | ⚠️ (main env works; 2B-class model is not lightweight) | ⚠️ (Local 1.7B smaller; tokenizer is large) | ⚠️ (runs in the main ComfyUI environment but remains GPU-heavy) | ✅ |
| **Speed/Performance** | ✅ Very Fast | ✅ Fast | ✅ Fast | ⚠️ | ⚠️ | ⚠️ CUDA recommended | ⚠️ | ✅ Fast | ⚠️ | ⚠️ Moderate | ⚠️ | ✅ Fast (diffusion, realtime-capable) | ✅ Main-environment subprocess with reliable teardown; local compile measurements: ~40 it/s BF16, ~11.8 it/s NF4 at ~8.9GB VRAM, and ~3.7 it/s INT8 at ~11.2GB VRAM; quality comparison pending | ⚠️ Moderate; mf variant is faster | ⚠️ Live shared-runtime validation: 6.36s reference-free audio in 39.3s including load; 9.99s cloned audio in 36.7s warm; exact cache reuse in 56ms | ⚠️ Fast favors repeated generation; staged reloads temporary components; sequential also transfers the transformer; optional block-level torch.compile accelerates repeat denoising | ✅ Fast; upstream reports sub-realtime RTF | ✅ Fast with CUDA/FlashAttention | ⚠️ (100 diffusion steps by default) | ✅ Fast |
| **No Narrator Required** | ❌ | ✅ (default speaker) | ✅ (default speaker) | ✅ (zero-shot / default speaker) | ✅ (basic TTS if no narrator/reference is provided) | ✅ (zero-shot) | ❌ | ✅ (cross-lingual or instruct mode) | ✅ (Base default voice or CustomVoice presets) | N/A | ❌ | ❌ | ✅ Reference audio is optional | ✅ (default speaker) | ✅ Reference audio is optional | ✅ Voice reference is optional | ✅ (native default voice; instruct also works without narrator) | ✅ (direct TTS and prompt-only generation) | ❌ | N/A |
† **Reference Transcript:** Conditional means the transcript is required only for the specific mode: CosyVoice3 zero-shot, Qwen3-TTS full Base cloning, or MOSS-TTSD cloned-speaker dialogue. Higgs Audio 2, Higgs Audio v3, and Dots TTS accept matching text when provided but do not require it.
† **Reference Transcript:** Conditional means the transcript is required only for the specific mode: CosyVoice3 zero-shot, Qwen3-TTS full Base cloning, or MOSS-TTSD cloned-speaker dialogue, and Audio8 TTS zero-shot/cross-lingual cloning. Higgs Audio 2, Higgs Audio v3, and Dots TTS accept matching text when provided but do not require it.
+105 -104
View File
@@ -2,110 +2,110 @@
## Language Support by Engine
| Language | Code | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | DramaBox | OmniVoice | MOSS-TTS | MOSS-SoundEffect v2 | RVC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 🇺🇸 **English** | EN | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ Tier 1 | ✅ | ✅ Official model is English-only | ✅ | ✅ | ✅ Officially demonstrated prompt language | ✅ |
| 🇨🇳 **Chinese** | ZH | ❌ | ❌ | ✅ | ✅ | ✅ (Mandarin) | ✅ | ✅ | ✅ + 18 dialects | ✅ | ❌ | ✅ (Mandarin + Sichuanese, Cantonese) | ❌ | ✅ Tier 1 | ✅ (Mandarin; official also exposes YUE separately outside this matrix) | ❌ | ✅ | ✅ | ✅ Officially demonstrated prompt language | ✅ |
| 🇩🇪 **German** | DE | ✅ | ✅ (×3) | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇪🇸 **Spanish** | ES | ✅ | ❌ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇫🇷 **French** | FR | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇮🇹 **Italian** | IT | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇯🇵 **Japanese** | JA | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ✅ ? | ✅ | ✅ | ✅ (4.1-2b supports Japanese; plus variant does not) | ✅ | ❌ | ✅ Tier 1 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇰🇷 **Korean** | KO | ❌ | ✅ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇷🇺 **Russian** | RU | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇧🇷 **Portuguese** | PT | ✅ (BR) | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ (EU/BR*) | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ (Official PT tag is generic; upstream does not expose separate PT-BR/PT-PT tags and it may lean more European Portuguese than Brazilian Portuguese) | ❌ | ✅ (generic PT; official language space is much broader than this matrix) | ✅ | ❌ | ✅ |
| 🇵🇱 **Polish** | PL | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇮🇳 **Hindi** | HI | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇪🇬 **Arabic** | AR | ❌ | ❌ | ✅ (Egyptian) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇹🇷 **Turkish** | TR | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇹🇭 **Thai** | TH | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇳🇴 **Norwegian** | NO | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
| 🇻🇳 **Vietnamese** | VI | ❌ | ❌ | ✅ (Viterbox) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇦🇲 **Armenian** | HY | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
| 🇬🇪 **Georgian** | KA | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
| 🇩🇰 **Danish** | DA | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇫🇮 **Finnish** | FI | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇬🇷 **Greek** | EL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇮🇱 **Hebrew** | HE | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇲🇾 **Malay** | MS | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇳🇱 **Dutch** | NL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇸🇪 **Swedish** | SV | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇰🇪 **Swahili** | SW | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇿🇦 **Afrikaans** | AF | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇱 **Albanian** | SQ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Assamese** | AS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Asturian** | AST | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇿 **Azerbaijani** | AZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇺 **Bashkir** | BA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Basque** | EU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇾 **Belarusian** | BE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇩 **Bengali** | BN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇦 **Bosnian** | BS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇬 **Bulgarian** | BG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Catalan** | CA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇭 **Cebuano** | CEB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇶 **Central Kurdish** | CKB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇭🇷 **Croatian** | HR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇿 **Czech** | CS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇺 **Eastern Mari** | MHR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🌍 **Esperanto** | EO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇪 **Estonian** | ET | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Galician** | GL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Gujarati** | GU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇭🇹 **Haitian Creole** | HT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇬 **Hausa** | HA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇭🇺 **Hungarian** | HU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇩 **Indonesian** | ID | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇩 **Javanese** | JV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Kannada** | KN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇿 **Kazakh** | KK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇼 **Kinyarwanda** | RW | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇬 **Kyrgyz** | KY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇱🇻 **Latvian** | LV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇩 **Lingala** | LN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇱🇹 **Lithuanian** | LT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇪 **Luo** | LUO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇰 **Macedonian** | MK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Malayalam** | ML | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇹 **Maltese** | MT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇿 **Māori** | MI | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Marathi** | MR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇳 **Mongolian** | MN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇵 **Nepali** | NE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇫🇷 **Occitan** | OC | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇷 **Persian** | FA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇴 **Romanian** | RO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇦 **Sepedi** | NSO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇸 **Serbian** | SR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇼 **Shona** | SN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇸🇰 **Slovak** | SK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇸🇮 **Slovene** | SL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇭 **Tagalog** | TL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇹🇯 **Tajik** | TG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Tamil** | TA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Telugu** | TE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇺🇦 **Ukrainian** | UK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇰 **Urdu** | UR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇳 **Uyghur** | UG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇺🇿 **Uzbek** | UZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇦 **Xhosa** | XH | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇦 **Zulu** | ZU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇼 **Chichewa/Nyanja** | NY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Eastern Punjabi** | PA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇺🇬 **Ganda** | LG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇸 **Icelandic** | IS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇪 **Irish** | GA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇩🇿 **Kabyle** | KAB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇻 **Kabuverdianu** | KEA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇪 **Kamba** | KAM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇻🇦 **Latin** | LA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇱🇺 **Luxembourgish** | LB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇹 **Oromo** | OM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇫 **Pashto** | PS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇰 **Sindhi** | SD | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇸🇴 **Somali** | SO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇴 **Umbundu** | UMB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🏴 **Welsh** | CY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| Language | Code | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | Audio8 TTS | DramaBox | OmniVoice | MOSS-TTS | MOSS-SoundEffect v2 | RVC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 🇺🇸 **English** | EN | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ Tier 1 | ✅ | ✅ | ✅ Official model is English-only | ✅ | ✅ | ✅ Officially demonstrated prompt language | ✅ |
| 🇨🇳 **Chinese** | ZH | ❌ | ❌ | ✅ | ✅ | ✅ (Mandarin) | ✅ | ✅ | ✅ + 18 dialects | ✅ | ❌ | ✅ (Mandarin + Sichuanese, Cantonese) | ❌ | ✅ Tier 1 | ✅ (Mandarin; official also exposes YUE separately outside this matrix) | ✅ (Chinese; upstream separately recommends Cantonese, folded into this matrix row) | ❌ | ✅ | ✅ | ✅ Officially demonstrated prompt language | ✅ |
| 🇩🇪 **German** | DE | ✅ | ✅ (×3) | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇪🇸 **Spanish** | ES | ✅ | ❌ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇫🇷 **French** | FR | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇮🇹 **Italian** | IT | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇯🇵 **Japanese** | JA | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ✅ ? | ✅ | ✅ | ✅ (4.1-2b supports Japanese; plus variant does not) | ✅ | ❌ | ✅ Tier 1 | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇰🇷 **Korean** | KO | ❌ | ✅ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ Tier 2 | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇷🇺 **Russian** | RU | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇧🇷 **Portuguese** | PT | ✅ (BR) | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ (EU/BR*) | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ (Official PT tag is generic; upstream does not expose separate PT-BR/PT-PT tags and it may lean more European Portuguese than Brazilian Portuguese) | ❌ | ❌ | ✅ (generic PT; official language space is much broader than this matrix) | ✅ | ❌ | ✅ |
| 🇵🇱 **Polish** | PL | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇮🇳 **Hindi** | HI | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇪🇬 **Arabic** | AR | ❌ | ❌ | ✅ (Egyptian) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇹🇷 **Turkish** | TR | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇹🇭 **Thai** | TH | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇳🇴 **Norwegian** | NO | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
| 🇻🇳 **Vietnamese** | VI | ❌ | ❌ | ✅ (Viterbox) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇦🇲 **Armenian** | HY | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
| 🇬🇪 **Georgian** | KA | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
| 🇩🇰 **Danish** | DA | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇫🇮 **Finnish** | FI | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇬🇷 **Greek** | EL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇮🇱 **Hebrew** | HE | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇲🇾 **Malay** | MS | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇳🇱 **Dutch** | NL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇸🇪 **Swedish** | SV | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇰🇪 **Swahili** | SW | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
| 🇿🇦 **Afrikaans** | AF | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇱 **Albanian** | SQ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Assamese** | AS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Asturian** | AST | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇿 **Azerbaijani** | AZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇺 **Bashkir** | BA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Basque** | EU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇾 **Belarusian** | BE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇩 **Bengali** | BN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇦 **Bosnian** | BS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇬 **Bulgarian** | BG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Catalan** | CA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇭 **Cebuano** | CEB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇶 **Central Kurdish** | CKB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇭🇷 **Croatian** | HR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇿 **Czech** | CS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇺 **Eastern Mari** | MHR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🌍 **Esperanto** | EO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇪 **Estonian** | ET | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Galician** | GL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Gujarati** | GU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇭🇹 **Haitian Creole** | HT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇬 **Hausa** | HA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇭🇺 **Hungarian** | HU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇩 **Indonesian** | ID | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇩 **Javanese** | JV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Kannada** | KN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇿 **Kazakh** | KK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇼 **Kinyarwanda** | RW | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇬 **Kyrgyz** | KY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇱🇻 **Latvian** | LV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇩 **Lingala** | LN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇱🇹 **Lithuanian** | LT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇪 **Luo** | LUO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇰 **Macedonian** | MK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Malayalam** | ML | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇹 **Maltese** | MT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇿 **Māori** | MI | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Marathi** | MR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇳 **Mongolian** | MN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇵 **Nepali** | NE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇫🇷 **Occitan** | OC | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇷 **Persian** | FA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇴 **Romanian** | RO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇦 **Sepedi** | NSO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇸 **Serbian** | SR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇼 **Shona** | SN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇸🇰 **Slovak** | SK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇸🇮 **Slovene** | SL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇭 **Tagalog** | TL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇹🇯 **Tajik** | TG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Tamil** | TA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Telugu** | TE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇺🇦 **Ukrainian** | UK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇰 **Urdu** | UR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇳 **Uyghur** | UG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇺🇿 **Uzbek** | UZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇦 **Xhosa** | XH | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇦 **Zulu** | ZU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇼 **Chichewa/Nyanja** | NY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Eastern Punjabi** | PA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇺🇬 **Ganda** | LG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇸 **Icelandic** | IS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇪 **Irish** | GA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇩🇿 **Kabyle** | KAB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇻 **Kabuverdianu** | KEA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇪 **Kamba** | KAM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇻🇦 **Latin** | LA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇱🇺 **Luxembourgish** | LB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇹 **Oromo** | OM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇫 **Pashto** | PS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇰 **Sindhi** | SD | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇸🇴 **Somali** | SO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇴 **Umbundu** | UMB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🏴 **Welsh** | CY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
**Notes:**
@@ -115,5 +115,6 @@
- **IndexTTS-2**: Trained on 55K+ hours - ZH, EN, JA primary.
- **MOSS-TTS**: Official list includes ZH, EN, DE, ES, FR, JA, IT, HU, KO, RU, FA, AR, PL, PT, CS, DA, SV, EL, TR; HU/FA/CS are not separate columns in this matrix.
- **OmniVoice**: Official model supports 600+ languages; this matrix only shows the suite's comparison subset.
- **Audio8 TTS**: The Preview recommends Cantonese, Chinese, Dutch, English, French, German, Italian, Japanese, Korean, Polish, and Spanish. It has no explicit language selector; language behavior is text-driven. Cantonese is folded into the Chinese row in this matrix.
- **RVC**: Language-agnostic voice conversion/post-processing; it is marked supported for every language row.
- Qwen3-TTS: Supports pt-BR only with instruction (voice design or harcoded voices). Base can't do instructions, so it will always ouput pt-PT.
+6
View File
@@ -129,6 +129,12 @@ Use this as the canonical list of model repositories/links for offline setup.
| dots.tts-soar | [rednote-hilab/dots.tts-soar](https://huggingface.co/rednote-hilab/dots.tts-soar) | ~6GB | ✅ | Official SOAR checkpoint for higher-quality zero-shot cloning |
| dots.tts-mf | [rednote-hilab/dots.tts-mf](https://huggingface.co/rednote-hilab/dots.tts-mf) | ~6GB | ✅ | Official MeanFlow-distilled checkpoint for faster inference |
## Audio8 TTS
| Component | Source | Size | Auto-Download | Notes |
|---|---|---|---|---|
| Audio8 TTS Preview 0.6B | [Audio8/Audio8-TTS-Preview-0.6b](https://huggingface.co/Audio8/Audio8-TTS-Preview-0.6b) | ~2.39 GiB | ✅ | Official 0.6B Preview checkpoint with bundled 44.1kHz codec; suite uses the shared Transformers 4.57.3 runtime because voice cloning is not compatible with main Transformers 5.10.2 |
## DramaBox
| Component | Source | Size | Auto-Download | Notes |
+16
View File
@@ -427,6 +427,22 @@ Notes:
- Native sample rate is 48kHz.
- Main-environment support works on Transformers 5; on Windows, `normalize_text` falls back to no-op if `WeTextProcessing` is unavailable.
## Audio8 TTS
```text
ComfyUI/models/TTS/audio8_tts/
└── Audio8-TTS-Preview-0.6b/
└── (13 required official runtime files, including codec weights)
```
Notes:
- The official `Audio8/Audio8-TTS-Preview-0.6b` checkpoint is about 2.39 GiB and includes the 44.1kHz neural codec.
- The 0.6B model plus codec use about 1.75 GiB for parameters at fp16/bf16 or about 3.5 GiB at float32/CPU, before KV cache, activations, and framework overhead.
- The suite uses the existing `vibevoice_transformers4_shared` runtime with Transformers 4.57.3. Main Transformers 5.10.2 is not compatible with Audio8 voice cloning: the official demo collapsed to 0.325s, while the shared runtime generated 10.495s for the published 10.54s demo.
- Live shared-runtime validation produced 6.36s of reference-free audio in 39.3s including model load, 9.99s of cloned audio in 36.7s warm, and exact cache reuse in 56ms.
- The engine has unified TTS and SRT support only: no Voice Designer support, special node, training integration, native streaming/timestamps/dialogue, or built-in presets.
## OmniVoice
```text
@@ -466,7 +466,7 @@ Also to test, requirements and dependencies need to be added.
- [ ] Pause tags work with `[pause:1.5s]`
- [ ] Caching works (same input = cached output)
- [ ] Model auto-download works
- [ ] VRAM management works (model unloads)
- [ ] VRAM management works (unload, switch to another isolated engine, then switch back with cache disabled)
- [ ] Different parameter combinations work
- [ ] Engine prints a standard `Settings:` summary with the active generation/load parameters
- [ ] **Interrupt handling works** - User can stop SRT generation and it stops within ~1 segment
@@ -4,7 +4,7 @@
### Import Errors
- **Missing import**: Always add `import folder_paths` when using `folder_paths.get_temp_directory()` in node files
- **Bundled code imports**: For complex bundled packages with internal cross-imports, add `sys.path.insert(0, impl_dir)` at top of main files instead of converting all imports
- **Bundled code imports**: Never permanently prepend a bundled implementation directory to `sys.path` in the main ComfyUI process. Generic files such as `utils.py` can shadow suite packages and break later isolated workers. Prefer package-relative or vendor-namespaced imports; if upstream imports make path injection unavoidable, confine it to the isolated worker and load suite protocol modules before adding the vendor path.
### Audio Utility Functions
- **Temp file creation**: Use `AudioProcessingUtils.save_audio_to_temp_file()` not `save_audio()` (doesn't exist)
+15 -2
View File
@@ -49,6 +49,19 @@ except Exception as e:
def __init__(self, *args, **kwargs):
raise ImportError(f"Dots TTS adapter not available: {e}")
try:
from .audio8_tts_adapter import Audio8TTSEngineAdapter
AUDIO8_TTS_ADAPTER_AVAILABLE = True
except Exception as exc:
AUDIO8_TTS_ADAPTER_AVAILABLE = False
_AUDIO8_TTS_ADAPTER_ERROR = str(exc)
class Audio8TTSEngineAdapter:
def __init__(self, *args, **kwargs):
raise ImportError(
f"Audio8 TTS adapter not available: {_AUDIO8_TTS_ADAPTER_ERROR}"
)
try:
from .dramabox_adapter import DramaBoxEngineAdapter
DRAMABOX_ADAPTER_AVAILABLE = True
@@ -96,10 +109,10 @@ except Exception as e:
__all__ = [
'ChatterBoxEngineAdapter', 'F5TTSEngineAdapter', 'CosyVoiceAdapter', 'EchoTTSEngineAdapter',
'DotsTTSEngineAdapter', 'DramaBoxEngineAdapter', 'OmniVoiceEngineAdapter',
'DotsTTSEngineAdapter', 'Audio8TTSEngineAdapter', 'DramaBoxEngineAdapter', 'OmniVoiceEngineAdapter',
'MossTTSEngineAdapter', 'HiggsAudioV3EngineAdapter',
'CHATTERBOX_ADAPTER_AVAILABLE', 'F5TTS_ADAPTER_AVAILABLE', 'COSYVOICE_ADAPTER_AVAILABLE',
'ECHO_TTS_ADAPTER_AVAILABLE', 'DOTS_TTS_ADAPTER_AVAILABLE',
'ECHO_TTS_ADAPTER_AVAILABLE', 'DOTS_TTS_ADAPTER_AVAILABLE', 'AUDIO8_TTS_ADAPTER_AVAILABLE',
'DRAMABOX_ADAPTER_AVAILABLE', 'OMNIVOICE_ADAPTER_AVAILABLE',
'MOSS_TTS_ADAPTER_AVAILABLE', 'HIGGS_AUDIO_V3_ADAPTER_AVAILABLE',
'MossSoundEffectV2Adapter'
+277
View File
@@ -0,0 +1,277 @@
"""Adapter between suite processors and the official Audio8 TTS engine."""
from __future__ import annotations
import os
from typing import Any, Dict, Optional, Tuple
import torch
from engines.audio8_tts.downloader import Audio8TTSDownloader
from utils.audio.audio_hash import generate_stable_audio_component
from utils.audio.cache import get_audio_cache
from utils.device import resolve_torch_device
from utils.models.factory_config import ModelLoadConfig, RUNTIME_MODE_SHARED
from utils.voice.character_logging import resolved_character_label
from utils.voice.reference import effective_voice_audio
class Audio8TTSEngineAdapter:
"""Translate unified TTS calls into Audio8's official inference API."""
SAMPLE_RATE = 44100
def __init__(self, config: Optional[Dict[str, Any]] = None):
self.config = dict(config or {})
self.audio_cache = get_audio_cache()
self.downloader = Audio8TTSDownloader()
self._last_config: Optional[ModelLoadConfig] = None
self._load_signature: Optional[Tuple[Any, ...]] = None
def update_config(self, new_config: Dict[str, Any]):
self.config = dict(new_config or {})
def _model_selection(self) -> str:
return str(
self.config.get(
"model_variant",
self.config.get(
"model_name",
Audio8TTSDownloader.MODEL_NAME,
),
)
or Audio8TTSDownloader.MODEL_NAME
)
def _build_load_signature(self) -> Tuple[Any, ...]:
return (
self._model_selection(),
resolve_torch_device(self.config.get("device", "auto")),
self.config.get("dtype", "auto"),
)
def load_model(
self,
model_variant: str,
device: str = "auto",
dtype: str = "auto",
):
"""Resolve the organized model path and load through the unified factory."""
from utils.models.unified_model_interface import unified_model_interface
model_path = self.downloader.resolve_model_path(model_variant)
config = ModelLoadConfig(
engine_name="audio8_tts",
model_type="tts",
model_name=model_variant,
model_path=model_path,
device=device,
additional_params={"dtype": dtype},
runtime_mode=RUNTIME_MODE_SHARED,
runtime_profile="vibevoice_transformers4_shared",
)
self._last_config = config
return unified_model_interface.load_model(config)
def _ensure_model_loaded(self):
signature = self._build_load_signature()
if signature != self._load_signature or self._last_config is None:
self.load_model(
model_variant=self._model_selection(),
device=self.config.get("device", "auto"),
dtype=self.config.get("dtype", "auto"),
)
self._load_signature = signature
def _get_engine(self):
if self._last_config is None:
self._ensure_model_loaded()
from utils.models.unified_model_interface import unified_model_interface
return unified_model_interface.load_model(self._last_config)
@staticmethod
def _mono_waveform(waveform: Any) -> torch.Tensor:
audio = torch.as_tensor(waveform, dtype=torch.float32).detach().cpu()
if audio.ndim == 3:
if audio.shape[0] != 1:
raise ValueError(
"Audio8 TTS reference audio must contain one batch item"
)
audio = audio[0]
if audio.ndim == 2:
audio = audio.mean(dim=0)
if audio.ndim != 1 or audio.numel() == 0:
raise ValueError(
"Audio8 TTS reference audio must be non-empty mono or "
"channels-first audio"
)
return audio.contiguous()
def _in_memory_reference(
self,
waveform: Any,
sample_rate: Any,
) -> Tuple[Dict[str, Any], str]:
mono = self._mono_waveform(waveform)
sample_rate = int(sample_rate)
comfy_audio = {
"waveform": mono.unsqueeze(0),
"sample_rate": sample_rate,
}
processor_audio = {
"array": mono,
"sampling_rate": sample_rate,
}
return (
processor_audio,
generate_stable_audio_component(reference_audio=comfy_audio),
)
def _extract_voice_reference(
self,
voice_ref: Optional[Dict[str, Any]],
) -> Tuple[Any, str, str]:
if not isinstance(voice_ref, dict):
return None, "", "default_voice"
reference_text = str(
voice_ref.get("reference_text")
or voice_ref.get("prompt_text")
or voice_ref.get("text")
or ""
).strip()
reference_audio = effective_voice_audio(voice_ref)
if reference_audio is None:
return None, reference_text, "default_voice"
if isinstance(reference_audio, (str, os.PathLike)):
path = os.fspath(reference_audio)
component = generate_stable_audio_component(audio_file_path=path)
return path, reference_text, component
if isinstance(reference_audio, dict):
waveform = reference_audio.get(
"waveform",
reference_audio.get("array"),
)
sample_rate = reference_audio.get(
"sample_rate",
reference_audio.get("sampling_rate"),
)
if waveform is None or sample_rate is None:
raise ValueError(
"Audio8 TTS in-memory reference requires waveform and sample_rate"
)
normalized, component = self._in_memory_reference(
waveform,
sample_rate,
)
return normalized, reference_text, component
if isinstance(reference_audio, (tuple, list)) and len(reference_audio) == 2:
normalized, component = self._in_memory_reference(
reference_audio[0],
reference_audio[1],
)
return normalized, reference_text, component
if torch.is_tensor(reference_audio):
normalized, component = self._in_memory_reference(
reference_audio,
voice_ref.get("sample_rate", self.SAMPLE_RATE),
)
return normalized, reference_text, component
raise TypeError(
f"Unsupported Audio8 TTS reference type: {type(reference_audio)}"
)
def generate_single(
self,
text: str,
voice_ref: Optional[Dict[str, Any]],
seed: int = 0,
enable_audio_cache: bool = True,
character_name: Optional[str] = None,
) -> torch.Tensor:
"""Generate one raw utterance; chunking remains processor-owned."""
text = str(text or "").strip()
if not text:
return torch.zeros(1, 0, dtype=torch.float32)
reference_audio, reference_text, audio_component = (
self._extract_voice_reference(voice_ref)
)
if reference_audio is not None and not reference_text:
raise ValueError(
"Audio8 TTS voice cloning requires the exact transcript of "
"the reference audio. Add reference text to Character Voices "
"or the narrator voice."
)
params = {
"model_variant": self._model_selection(),
"max_new_tokens": int(self.config.get("max_new_tokens", 1024)),
"retry_max_new_tokens": int(self.config.get("retry_max_new_tokens", 2000)),
"temperature": float(self.config.get("temperature", 0.8)),
"top_p": float(self.config.get("top_p", 0.95)),
"top_k": int(self.config.get("top_k", 50)),
"do_sample": bool(self.config.get("do_sample", True)),
"dtype": self.config.get("dtype", "auto"),
"device": resolve_torch_device(self.config.get("device", "auto")),
"seed": int(seed),
}
cache_key = None
if enable_audio_cache:
cache_key = self.audio_cache.generate_cache_key(
"audio8_tts",
text=text,
audio_component=audio_component,
reference_text=reference_text,
character=character_name or "narrator",
**params,
)
cached = self.audio_cache.get_cached_audio(cache_key)
if cached:
display_name = resolved_character_label(
character_name or "narrator",
voice_ref,
)
print(
"💾 Using cached Audio8 TTS audio for "
f"'{display_name}': '{text[:30]}...'"
)
return cached[0]
self._ensure_model_loaded()
audio = self._get_engine().generate(
text=text,
reference_audio=reference_audio,
reference_text=reference_text or None,
max_new_tokens=params["max_new_tokens"],
retry_max_new_tokens=params["retry_max_new_tokens"],
temperature=params["temperature"],
top_p=params["top_p"],
top_k=params["top_k"],
do_sample=params["do_sample"],
seed=params["seed"],
)
if not isinstance(audio, torch.Tensor):
audio = torch.as_tensor(audio, dtype=torch.float32)
audio = audio.detach().float().cpu()
if audio.ndim == 1:
audio = audio.unsqueeze(0)
if audio.ndim != 2:
raise RuntimeError(
f"Audio8 TTS returned invalid audio shape: {tuple(audio.shape)}"
)
if cache_key:
self.audio_cache.cache_audio(
cache_key,
audio,
audio.shape[-1] / self.SAMPLE_RATE,
)
return audio
+6
View File
@@ -0,0 +1,6 @@
"""Audio8 TTS engine integration."""
from .audio8_tts_engine import Audio8TTSEngine
from .downloader import Audio8TTSDownloader
__all__ = ["Audio8TTSEngine", "Audio8TTSDownloader"]
+375
View File
@@ -0,0 +1,375 @@
"""Wrapper for the official Audio8 TTS remote-code model."""
from __future__ import annotations
import os
import random
import warnings
from typing import Any, Dict, Optional, Tuple
import torch
from engines.audio8_tts.downloader import Audio8TTSDownloader
from engines.audio8_tts.progress import build_audio8_stopping_criteria
from utils.device import resolve_torch_device
class Audio8TTSEngine:
"""ComfyUI-friendly wrapper around the official Audio8 model."""
SAMPLE_RATE = 44100
def __init__(
self,
model_name: str = Audio8TTSDownloader.MODEL_NAME,
device: str = "auto",
dtype: str = "auto",
model_dir: Optional[str] = None,
):
self.model_name = model_name
self.device = resolve_torch_device(device)
self.dtype_name = str(dtype or "auto").lower()
self.dtype = self._resolve_dtype(self.dtype_name, self.device)
self.model_dir = model_dir or Audio8TTSDownloader().resolve_model_path(
model_name
)
self._processor = None
self._model = None
self._codec = None
@staticmethod
def _resolve_dtype(dtype: str, device: str) -> torch.dtype:
normalized = str(dtype or "auto").lower()
valid = {
"bfloat16": torch.bfloat16,
"float16": torch.float16,
"float32": torch.float32,
}
if normalized not in {"auto", *valid}:
raise ValueError(
"Audio8 TTS dtype must be auto, bfloat16, float16, or float32"
)
if str(device).startswith("cpu"):
return torch.float32
if normalized in valid:
return valid[normalized]
if str(device).startswith("cuda") and torch.cuda.is_available():
major, _minor = torch.cuda.get_device_capability(torch.device(device))
return torch.bfloat16 if major >= 8 else torch.float16
if str(device).startswith("xpu"):
return torch.bfloat16
return torch.float16
def _ensure_model_loaded(self):
"""Load the processor, model, and bundled codec strictly from local files."""
if (
self._processor is not None
and self._model is not None
and self._codec is not None
):
return self
if not os.path.isdir(self.model_dir):
raise FileNotFoundError(
f"Audio8 TTS local model directory does not exist: {self.model_dir}"
)
try:
from transformers import AutoModel, AutoProcessor
except ImportError as exc:
raise ImportError(
"Audio8 TTS requires the suite's shared Transformers 4 runtime"
) from exc
print(
f"📦 Loading Audio8 TTS from {self.model_dir} "
f"on {self.device} ({self.dtype})"
)
with warnings.catch_warnings():
warnings.filterwarnings(
"ignore",
message=(
r"`torch\.nn\.utils\.weight_norm` is deprecated in favor of "
r"`torch\.nn\.utils\.parametrizations\.weight_norm`\."
),
category=FutureWarning,
)
self._processor = AutoProcessor.from_pretrained(
self.model_dir,
trust_remote_code=True,
local_files_only=True,
)
self._model = AutoModel.from_pretrained(
self.model_dir,
trust_remote_code=True,
local_files_only=True,
dtype=self.dtype,
)
self._model = self._model.eval().to(self.device)
# The official model stores the codec outside nn.Module registration.
# Hold and move it explicitly so model offload/reload cannot strand it.
self._codec = self._model.load_codec(
device=self.device,
dtype=self.dtype,
)
self._codec.eval()
sample_rate = int(
getattr(self._model.config, "codec_sample_rate", self.SAMPLE_RATE)
)
if sample_rate != self.SAMPLE_RATE:
raise RuntimeError(
f"Audio8 TTS checkpoint reports unexpected sample rate "
f"{sample_rate}; expected {self.SAMPLE_RATE}"
)
print("✅ Audio8 TTS model and codec loaded")
return self
# Compatibility alias for callers using the shorter established name.
_ensure_loaded = _ensure_model_loaded
def _clear_kv_caches(self) -> None:
"""Drop every static slow/fast attention cache before device movement."""
if self._model is None:
return
for collection_name in ("layers", "fast_layers"):
for layer in getattr(self._model, collection_name, ()) or ():
attention = getattr(layer, "attention", None)
if attention is not None and hasattr(attention, "kv_cache"):
attention.kv_cache = None
def to(self, device):
"""Move all weights for ComfyUI Clear VRAM and reload handling."""
target = resolve_torch_device(
str(device) if not isinstance(device, str) else device
)
self._clear_kv_caches()
self.device = target
if self._model is not None:
self._model = self._model.to(target).eval()
if self._codec is not None:
codec_dtype = torch.float32 if str(target).startswith("cpu") else self.dtype
self._codec = self._codec.to(
device=target,
dtype=codec_dtype,
).eval()
return self
def parameters(self):
"""Expose all weight tensors to ComfyUI memory accounting."""
if self._model is not None:
yield from self._model.parameters()
if self._codec is not None:
yield from self._codec.parameters()
@staticmethod
def _mono_audio(waveform: Any) -> torch.Tensor:
audio = torch.as_tensor(waveform, dtype=torch.float32).detach().cpu()
if audio.ndim == 3:
if audio.shape[0] != 1:
raise ValueError(
"Audio8 TTS reference audio must contain one batch item"
)
audio = audio[0]
if audio.ndim == 2:
audio = audio.mean(dim=0)
if audio.ndim != 1 or audio.numel() == 0:
raise ValueError(
"Audio8 TTS reference audio must be non-empty mono or "
"channels-first audio"
)
return audio.contiguous()
def _normalize_reference_audio(
self,
reference_audio: Any,
) -> Tuple[Any, Optional[int]]:
if isinstance(reference_audio, (str, os.PathLike)):
return os.fspath(reference_audio), None
if isinstance(reference_audio, dict):
waveform = reference_audio.get(
"waveform",
reference_audio.get("array"),
)
sample_rate = reference_audio.get(
"sample_rate",
reference_audio.get("sampling_rate"),
)
if waveform is None or sample_rate is None:
raise ValueError(
"Audio8 TTS in-memory reference audio requires waveform "
"and sample_rate"
)
return {
"array": self._mono_audio(waveform),
"sampling_rate": int(sample_rate),
}, int(sample_rate)
if isinstance(reference_audio, (tuple, list)) and len(reference_audio) == 2:
waveform, sample_rate = reference_audio
if not isinstance(sample_rate, (int, float)):
raise ValueError(
"Audio8 TTS reference tuple must be (waveform, sample_rate)"
)
return {
"array": self._mono_audio(waveform),
"sampling_rate": int(sample_rate),
}, int(sample_rate)
raise TypeError(
f"Unsupported Audio8 TTS reference audio type: {type(reference_audio)}"
)
def _prepare_inputs(
self,
text: str,
reference_audio: Any,
reference_text: Optional[str],
) -> Dict[str, torch.Tensor]:
processor_kwargs: Dict[str, Any] = {
"text": [text],
"return_tensors": "pt",
}
if reference_audio is not None:
normalized_audio, sample_rate = self._normalize_reference_audio(
reference_audio
)
processor_kwargs.update(
reference_audio=[normalized_audio],
reference_text=[reference_text],
)
if sample_rate is not None:
processor_kwargs["sampling_rate"] = [sample_rate]
inputs = self._processor(**processor_kwargs)
return {name: value.to(self.device) for name, value in inputs.items()}
def _make_generator(self, seed: int):
seed = int(seed)
random.seed(seed)
torch.manual_seed(seed)
if torch.cuda.is_available():
torch.cuda.manual_seed_all(seed)
try:
return torch.Generator(device=torch.device(self.device)).manual_seed(seed)
except Exception:
return None
def _generate_once(
self,
inputs: Dict[str, torch.Tensor],
*,
max_new_tokens: int,
temperature: float,
top_p: float,
top_k: int,
do_sample: bool,
generator,
):
criteria, tracker = build_audio8_stopping_criteria(max_new_tokens)
try:
output = self._model.generate(
**inputs,
max_new_tokens=max_new_tokens,
temperature=temperature,
top_p=top_p,
top_k=top_k,
do_sample=do_sample,
generator=generator,
stopping_criteria=criteria,
return_dict_in_generate=True,
)
except BaseException:
tracker.abort()
raise
tracker.close()
return output
def generate(
self,
text: str,
reference_audio: Any = None,
reference_text: Optional[str] = None,
*,
max_new_tokens: int = 1024,
retry_max_new_tokens: int = 2000,
temperature: float = 0.8,
top_p: float = 0.95,
top_k: int = 50,
do_sample: bool = True,
seed: int = 42,
) -> torch.Tensor:
"""Generate one utterance as a CPU float tensor ``[channels, samples]``."""
text = str(text or "").strip()
if not text:
return torch.zeros(1, 0, dtype=torch.float32)
reference_text = str(reference_text or "").strip()
if reference_audio is not None and not reference_text:
raise ValueError(
"Audio8 TTS voice cloning requires the exact transcript of "
"the reference audio"
)
if reference_audio is None and reference_text:
raise ValueError(
"Audio8 TTS reference_text requires matching reference audio"
)
max_new_tokens = int(max_new_tokens)
retry_max_new_tokens = int(retry_max_new_tokens)
temperature = float(temperature)
top_p = float(top_p)
top_k = int(top_k)
if max_new_tokens < 1:
raise ValueError("Audio8 TTS max_new_tokens must be positive")
if retry_max_new_tokens < max_new_tokens:
raise ValueError(
"Audio8 TTS retry_max_new_tokens must be >= max_new_tokens"
)
if temperature <= 0:
raise ValueError("Audio8 TTS temperature must be positive")
if not 0 < top_p <= 1:
raise ValueError("Audio8 TTS top_p must be in (0, 1]")
if top_k < 1:
raise ValueError("Audio8 TTS top_k must be at least 1")
self._ensure_model_loaded()
inputs = self._prepare_inputs(text, reference_audio, reference_text)
generator = self._make_generator(seed)
output = self._generate_once(
inputs,
max_new_tokens=max_new_tokens,
temperature=temperature,
top_p=top_p,
top_k=top_k,
do_sample=bool(do_sample),
generator=generator,
)
finished = bool(output.finished[0].item())
if not finished and retry_max_new_tokens > max_new_tokens:
print(
"⚠️ Audio8 TTS reached max_new_tokens without EOS; "
f"retrying with {retry_max_new_tokens}"
)
output = self._generate_once(
inputs,
max_new_tokens=retry_max_new_tokens,
temperature=temperature,
top_p=top_p,
top_k=top_k,
do_sample=bool(do_sample),
generator=generator,
)
finished = bool(output.finished[0].item())
if not finished:
print(
"⚠️ Audio8 TTS generation ended without EOS; returning the "
"valid decoded frames"
)
waveforms, waveform_lengths = self._model.decode_audio(output.codes)
waveform_length = int(waveform_lengths[0].item())
waveform = waveforms[0, :waveform_length].detach().float().cpu()
return waveform.unsqueeze(0).contiguous()
+180
View File
@@ -0,0 +1,180 @@
"""Organized model download and discovery for official Audio8 TTS."""
from __future__ import annotations
import os
from typing import List, Optional
import folder_paths
from utils.downloads.unified_downloader import unified_downloader
from utils.models.extra_paths import (
get_all_tts_model_paths,
get_preferred_download_path,
)
class Audio8TTSDownloader:
"""Resolve Audio8 checkpoints without using the Hugging Face model cache."""
MODEL_NAME = "Audio8-TTS-Preview-0.6b"
REPO_ID = "Audio8/Audio8-TTS-Preview-0.6b"
REVISION = "1b17c91db5f4dccb6914aa4aa5cb0e56661a6c17"
# Complete runtime snapshot at the pinned revision. Non-runtime model-card
# assets are deliberately excluded.
REQUIRED_FILES = [
"codec.pth",
"config.json",
"configuration_arktts.py",
"generation_config.json",
"model.safetensors",
"modeling_arktts.py",
"modeling_arktts_codec.py",
"preprocessor_config.json",
"processing_arktts.py",
"processor_config.json",
"special_tokens_map.json",
"tokenizer.json",
"tokenizer_config.json",
]
def __init__(self, base_path: Optional[str] = None):
if base_path is None:
try:
self.base_path = get_preferred_download_path(
model_type="TTS",
engine_name="audio8_tts",
)
except Exception:
self.base_path = os.path.join(
folder_paths.models_dir,
"TTS",
"audio8_tts",
)
else:
self.base_path = os.path.abspath(os.fspath(base_path))
os.makedirs(self.base_path, exist_ok=True)
def get_available_models(self) -> List[str]:
"""Return the canonical model plus complete local installations."""
models = [self.MODEL_NAME]
canonical_path = os.path.normcase(
os.path.abspath(os.path.join(self.base_path, self.MODEL_NAME))
)
for base_path in get_all_tts_model_paths("TTS"):
for folder_name in ("audio8_tts", "Audio8_TTS", ""):
root = (
os.path.join(base_path, folder_name) if folder_name else base_path
)
if not os.path.isdir(root):
continue
for item in sorted(os.listdir(root)):
candidate = os.path.join(root, item)
if os.path.normcase(os.path.abspath(candidate)) == canonical_path:
continue
local_name = f"local:{item}"
if local_name not in models and self._is_model_complete(candidate):
models.append(local_name)
return models
def resolve_model_path(
self,
model_identifier: str = MODEL_NAME,
) -> str:
"""Resolve an absolute path, ``local:`` name, or canonical model name."""
model_identifier = str(model_identifier or self.MODEL_NAME).strip()
if os.path.isabs(model_identifier) or os.path.isdir(model_identifier):
candidate = os.path.abspath(model_identifier)
if self._is_model_complete(candidate, verbose=True):
return candidate
raise FileNotFoundError(
f"Audio8 TTS model path is missing required files: {candidate}"
)
if model_identifier.startswith("local:"):
local_name = model_identifier[6:].strip()
if not local_name:
raise ValueError("Audio8 TTS local model name must not be empty")
for base_path in get_all_tts_model_paths("TTS"):
for folder_name in ("audio8_tts", "Audio8_TTS", ""):
candidate = (
os.path.join(base_path, folder_name, local_name)
if folder_name
else os.path.join(base_path, local_name)
)
if self._is_model_complete(candidate):
print(f"📁 Using local Audio8 TTS model: {candidate}")
return candidate
raise FileNotFoundError(
f"Local Audio8 TTS model not found or incomplete: {local_name}"
)
if model_identifier != self.MODEL_NAME:
raise ValueError(f"Unknown Audio8 TTS model: {model_identifier}")
return self.get_model_path()
def get_model_path(self, model_name: str = MODEL_NAME) -> str:
"""Return the organized canonical model path, downloading if needed."""
if model_name != self.MODEL_NAME:
raise ValueError(f"Unknown Audio8 TTS model: {model_name}")
model_dir = os.path.join(self.base_path, self.MODEL_NAME)
if not self._is_model_complete(model_dir):
return self.download_model(model_dir)
return model_dir
def download_model(
self,
model_dir: Optional[str] = None,
force: bool = False,
) -> str:
"""Download the pinned official snapshot into the organized model folder."""
model_dir = model_dir or os.path.join(self.base_path, self.MODEL_NAME)
print("\n" + "=" * 60)
print("📦 Audio8 TTS Model Download")
print("=" * 60)
print(f"Repository: {self.REPO_ID}")
print(f"Revision: {self.REVISION}")
print(f"Target: {model_dir}")
print("=" * 60 + "\n")
unified_downloader.download_huggingface_snapshot(
repo_id=self.REPO_ID,
target_dir=model_dir,
revision=self.REVISION,
allow_patterns=self.REQUIRED_FILES,
required_files=self.REQUIRED_FILES,
force_download=bool(force),
description=self.MODEL_NAME,
)
if not self._is_model_complete(model_dir, verbose=True):
raise RuntimeError(
f"Downloaded Audio8 TTS model is incomplete: {model_dir}"
)
print(f"✅ Audio8 TTS model ready: {model_dir}")
return model_dir
def _is_model_complete(
self,
model_dir: str,
*,
verbose: bool = False,
) -> bool:
if not os.path.isdir(model_dir):
return False
missing = [
rel_path
for rel_path in self.REQUIRED_FILES
if not os.path.isfile(os.path.join(model_dir, rel_path))
or os.path.getsize(os.path.join(model_dir, rel_path)) <= 0
]
if missing and verbose:
print(
f"❌ Audio8 TTS model incomplete. Missing or empty "
f"{len(missing)} file(s):"
)
for rel_path in missing:
print(f" - {rel_path}")
return not missing
+116
View File
@@ -0,0 +1,116 @@
"""Native generation progress and interruption support for Audio8 TTS."""
from __future__ import annotations
import time
from typing import Any, Optional
import torch
from transformers import StoppingCriteria, StoppingCriteriaList
def _make_comfy_progress_bar(total_steps: int) -> Optional[Any]:
try:
from comfy.utils import ProgressBar
return ProgressBar(total_steps)
except Exception:
return None
class Audio8TTSStoppingCriteria(StoppingCriteria):
"""Track native codec-frame generation and honor ComfyUI interruption."""
def __init__(self, max_steps: int, progress_bar: Optional[Any] = None):
self.max_steps = max(1, int(max_steps))
self.progress_bar = progress_bar
self.steps = 0
self.start_time = time.monotonic()
self.last_print_time = self.start_time
self.last_print_step = 0
self.rendered = False
@staticmethod
def _check_interrupted() -> None:
try:
import comfy.model_management as model_management
except Exception:
return
checker = getattr(
model_management,
"throw_exception_if_processing_interrupted",
None,
)
if callable(checker):
checker()
elif getattr(model_management, "interrupt_processing", False):
raise InterruptedError("Audio8 TTS generation interrupted by user")
def __call__(self, input_ids, scores, **kwargs):
del scores, kwargs
self._check_interrupted()
self.steps = min(self.steps + 1, self.max_steps)
if self.progress_bar is not None:
try:
self.progress_bar.update(1)
except Exception:
pass
now = time.monotonic()
if (
self.steps == 1
or self.steps >= self.max_steps
or now - self.last_print_time >= 0.5
):
delta_time = max(now - self.last_print_time, 1e-6)
delta_steps = self.steps - self.last_print_step
rate = delta_steps / delta_time
elapsed = now - self.start_time
width = 12
filled = min(
width,
int(width * self.steps / self.max_steps),
)
bar = "█" * filled + "░" * (width - filled)
print(
f"\r Audio8: [{bar}] {self.steps}/{self.max_steps} | "
f"{rate:.1f} frames/s | {elapsed:.1f}s ",
end="",
flush=True,
)
self.last_print_time = now
self.last_print_step = self.steps
self.rendered = True
# The official model handles EOS itself. Interruption raises above.
batch_size = int(input_ids.shape[0]) if input_ids.ndim else 1
return torch.zeros(
batch_size,
dtype=torch.bool,
device=input_ids.device,
)
def close(self) -> None:
elapsed = max(time.monotonic() - self.start_time, 1e-6)
average = self.steps / elapsed
if self.rendered:
print(
f"\r Audio8 complete: {self.steps} frames in "
f"{elapsed:.1f}s ({average:.1f} frames/s)" + " " * 20
)
def abort(self) -> None:
if self.rendered:
print()
def build_audio8_stopping_criteria(
max_steps: int,
) -> tuple[StoppingCriteriaList, Audio8TTSStoppingCriteria]:
"""Create the native stopping-criteria list and its progress tracker."""
tracker = Audio8TTSStoppingCriteria(
max_steps=max_steps,
progress_bar=_make_comfy_progress_bar(max(1, int(max_steps))),
)
return StoppingCriteriaList([tracker]), tracker
+12
View File
@@ -179,6 +179,14 @@ except Exception as e:
print(f"❌ Dots TTS Engine failed: {e}")
DOTS_TTS_ENGINE_AVAILABLE = False
try:
audio8_tts_engine_module = load_node_module("audio8_tts_engine_node", "engines/audio8_tts_engine_node.py")
Audio8TTSEngineNode = audio8_tts_engine_module.Audio8TTSEngineNode
AUDIO8_TTS_ENGINE_AVAILABLE = True
except Exception as e:
print(f"❌ Audio8 TTS Engine failed: {e}")
AUDIO8_TTS_ENGINE_AVAILABLE = False
try:
dramabox_engine_module = load_node_module("dramabox_engine_node", "engines/dramabox_engine_node.py")
DramaBoxEngineNode = dramabox_engine_module.DramaBoxEngineNode
@@ -670,6 +678,10 @@ if DOTS_TTS_ENGINE_AVAILABLE:
NODE_CLASS_MAPPINGS["DotsTTSEngineNode"] = DotsTTSEngineNode
NODE_DISPLAY_NAME_MAPPINGS["DotsTTSEngineNode"] = "⚙️ Dots TTS Engine"
if AUDIO8_TTS_ENGINE_AVAILABLE:
NODE_CLASS_MAPPINGS["Audio8TTSEngineNode"] = Audio8TTSEngineNode
NODE_DISPLAY_NAME_MAPPINGS["Audio8TTSEngineNode"] = "⚙️ Audio8 TTS Engine"
if DRAMABOX_ENGINE_AVAILABLE:
NODE_CLASS_MAPPINGS["DramaBoxEngineNode"] = DramaBoxEngineNode
NODE_DISPLAY_NAME_MAPPINGS["DramaBoxEngineNode"] = "⚙️ DramaBox Engine"
+1
View File
@@ -0,0 +1 @@
"""Audio8 TTS text and SRT processors."""
+403
View File
@@ -0,0 +1,403 @@
"""Suite orchestration for Audio8 TTS text generation."""
import os
import re
import sys
from typing import Any, Dict, List, Tuple, Union
import comfy.model_management as model_management
import torch
current_dir = os.path.dirname(__file__)
nodes_dir = os.path.dirname(current_dir)
project_root = os.path.dirname(nodes_dir)
if project_root not in sys.path:
sys.path.insert(0, project_root)
from utils.audio.chunk_timing import ChunkTimingHelper
from utils.audio.edit_post_processor import (
process_segments as apply_edit_post_processing,
)
from utils.text.character_parser import character_parser
from utils.text.pause_processor import PauseTagProcessor
from utils.text.segment_parameters import ParameterValidator, apply_segment_parameters
from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment
from utils.voice.character_logging import (
format_resolved_character_block,
resolved_character_label,
)
from utils.voice.discovery import (
get_available_characters,
get_character_mapping,
voice_discovery,
)
from utils.voice.reference import effective_voice_audio
class Audio8TTSProcessor:
"""Handle suite-level tags, voices, chunking, and Audio8 generation."""
SAMPLE_RATE = 44100
def __init__(self, adapter, engine_config: Dict[str, Any]):
self.adapter = adapter
self.config = engine_config.copy() if engine_config else {}
def update_config(self, new_config: Dict[str, Any]):
self.config = new_config.copy() if new_config else {}
self.adapter.update_config(self.config)
@staticmethod
def _check_interrupt(context: str = "") -> None:
if not model_management.interrupt_processing:
return
suffix = f" {context}" if context else ""
raise InterruptedError(f"Audio8 TTS generation interrupted{suffix}")
def _setup_character_parser(self, text: str) -> None:
character_tags = re.findall(r"\[([^\]]+)\]", text or "")
characters_from_tags = [
tag.split("|")[0].strip()
for tag in character_tags
if not tag.lower().startswith("pause:")
]
all_available = set(get_available_characters() or [])
for alias, target in voice_discovery.get_character_aliases().items():
all_available.add(alias.lower())
all_available.add(target.lower())
all_available.update(
character.lower() for character in characters_from_tags if character
)
all_available.add("narrator")
character_parser.set_available_characters(list(all_available))
character_parser.reset_session_cache()
@staticmethod
def _as_voice_reference(value: Any) -> Dict[str, Any]:
if isinstance(value, dict):
return value.copy()
if value is None:
return {}
return {"audio": value}
def _resolve_voice(
self,
character: str,
voice_mapping: Dict[str, Any],
discovered_mapping: Dict[str, Tuple[Any, Any]],
) -> Dict[str, Any]:
if character in voice_mapping:
mapped_voice = self._as_voice_reference(voice_mapping[character])
if effective_voice_audio(mapped_voice) is not None or mapped_voice.get(
"reference_text"
):
return mapped_voice
if character != "narrator":
audio_path, reference_text = discovered_mapping.get(character, (None, None))
if audio_path:
return {
"audio_path": audio_path,
"reference_text": reference_text or "",
}
print(
f"⚠️ Audio8 TTS: No voice found for '{character}'; "
"using narrator/no-reference fallback"
)
return self._as_voice_reference(voice_mapping.get("narrator"))
@staticmethod
def _voice_log_note(voice_ref: Dict[str, Any]) -> str:
reference_text = ""
if isinstance(voice_ref, dict):
reference_text = str(voice_ref.get("reference_text") or "").strip()
has_audio = (
isinstance(voice_ref, dict) and effective_voice_audio(voice_ref) is not None
)
if not has_audio and not reference_text:
return " [no reference voice]"
if has_audio and reference_text:
return f" [reference transcript: {len(reference_text)} chars]"
if has_audio:
return " [reference audio has no transcript; adapter will validate]"
return " [reference transcript has no audio; adapter will validate]"
@staticmethod
def _format_parameter_log(
filtered_params: Dict[str, Any],
current_config: Dict[str, Any],
current_seed: int,
) -> str:
if not filtered_params:
return ""
values = []
for key in ("seed", "temperature", "top_p", "top_k", "max_new_tokens"):
if key not in filtered_params:
continue
value = current_seed if key == "seed" else current_config.get(key)
values.append(f"{key}={value}")
return ", ".join(values)
@staticmethod
def _validate_segment_config(
config: Dict[str, Any],
filtered_params: Dict[str, Any],
) -> Dict[str, Any]:
"""Apply Audio8-specific bounds and keep retry budgets consistent."""
if "top_p" in filtered_params and float(config["top_p"]) <= 0:
raise ValueError("Audio8 segment top_p must be greater than 0")
if "max_new_tokens" in filtered_params:
max_new_tokens = int(config["max_new_tokens"])
if max_new_tokens > 2048:
raise ValueError("Audio8 segment max_new_tokens must be at most 2048")
config["retry_max_new_tokens"] = max(
max_new_tokens,
int(config.get("retry_max_new_tokens", 2000)),
)
return config
def _log_generation(
self,
character: str,
text: str,
voice_ref: Dict[str, Any],
config: Dict[str, Any],
chunk_count: int,
parameter_log: str,
show_text_logging: bool,
) -> None:
display_name = resolved_character_label(character, voice_ref)
print(
f"🎭 Audio8 TTS - Generating for '{display_name}'"
f"{self._voice_log_note(voice_ref)}"
)
print(
" Settings: "
f"mode={'Sampling' if config.get('do_sample', True) else 'Greedy'}, "
f"temperature={config.get('temperature', 0.8)}, "
f"top_p={config.get('top_p', 0.95)}, "
f"top_k={config.get('top_k', 50)}, "
f"max_new_tokens={config.get('max_new_tokens', 1024)}"
)
if parameter_log:
print(f"🎛️ Audio8 TTS segment params: {parameter_log}")
if show_text_logging:
print(format_resolved_character_block(character, text, voice_ref))
if chunk_count > 1:
print(
f"📝 Audio8 TTS: Chunking '{display_name}' into "
f"{chunk_count} suite chunks"
)
@staticmethod
def _normalize_audio(audio: Any) -> torch.Tensor:
if not isinstance(audio, torch.Tensor):
audio = torch.as_tensor(audio, dtype=torch.float32)
audio = audio.detach().to(device="cpu", dtype=torch.float32)
if audio.dim() == 1:
audio = audio.unsqueeze(0)
elif audio.dim() == 3 and audio.shape[0] == 1:
audio = audio.squeeze(0)
if audio.dim() != 2:
raise ValueError(
"Audio8 adapter returned an invalid waveform shape; "
"expected [channels, samples]"
)
return audio
def process_text(
self,
text: str,
voice_mapping: Dict[str, Any],
seed: int,
enable_chunking: bool = True,
max_chars_per_chunk: int = 400,
chunk_combination_method: str = "auto",
silence_between_chunks_ms: int = 100,
enable_audio_cache: bool = True,
apply_edit_postprocessing: bool = True,
show_text_logging: bool = True,
**_unused,
) -> List[Dict[str, Any]]:
del chunk_combination_method, silence_between_chunks_ms
self._check_interrupt()
self._setup_character_parser(text)
base_config = self.config.copy()
segment_objects = character_parser.parse_text_segments(text)
if not segment_objects:
segment_objects = character_parser.parse_text_segments(
"narrator " + (text or "")
)
characters = list(
{segment.character for segment in segment_objects if segment.character}
)
discovered_mapping = get_character_mapping(characters, engine_type="audio_only")
segment_records: List[Dict[str, Any]] = []
try:
for segment_index, segment in enumerate(segment_objects):
character = segment.character or "narrator"
self._check_interrupt(
f"before segment {segment_index + 1}/{len(segment_objects)} "
f"for '{character}'"
)
segment_text = (segment.text or "").strip()
if not segment_text:
continue
segment_params = segment.parameters or {}
filtered_params = ParameterValidator.filter_parameters_for_engine(
segment_params, "audio8_tts"
)
current_config = (
apply_segment_parameters(base_config, filtered_params, "audio8_tts")
if filtered_params
else base_config.copy()
)
current_config = self._validate_segment_config(
current_config, filtered_params
)
current_seed = int(current_config.get("seed", seed))
parameter_log = self._format_parameter_log(
filtered_params, current_config, current_seed
)
self.adapter.update_config(current_config)
voice_ref = self._resolve_voice(
character, voice_mapping, discovered_mapping
)
seed_offset = 0
def generate_chunks(text_content: str, edit_tags: list) -> None:
nonlocal seed_offset
clean_content = (text_content or "").strip()
if not clean_content:
return
if enable_chunking:
from utils.text.chunking import ImprovedChatterBoxChunker
max_chars = ImprovedChatterBoxChunker.validate_chunking_params(
max_chars_per_chunk
)
chunks = ImprovedChatterBoxChunker.split_into_chunks(
clean_content, max_chars=max_chars
)
else:
chunks = [clean_content]
chunks = [chunk.strip() for chunk in chunks if chunk.strip()]
self._log_generation(
character=character,
text=clean_content,
voice_ref=voice_ref,
config=current_config,
chunk_count=len(chunks),
parameter_log=parameter_log,
show_text_logging=show_text_logging,
)
for chunk_index, chunk in enumerate(chunks):
self._check_interrupt(
f"before chunk {chunk_index + 1}/{len(chunks)} "
f"for '{character}'"
)
chunk_seed = current_seed + seed_offset
seed_offset += 1
audio = self.adapter.generate_single(
text=chunk,
voice_ref=voice_ref,
seed=chunk_seed,
enable_audio_cache=enable_audio_cache,
character_name=character,
)
segment_records.append(
{
"waveform": self._normalize_audio(audio),
"sample_rate": self.SAMPLE_RATE,
"text": chunk,
"edit_tags": (edit_tags if chunk_index == 0 else []),
}
)
if PauseTagProcessor.has_pause_tags(segment_text):
pause_segments, _ = PauseTagProcessor.parse_pause_tags(segment_text)
for fragment_type, fragment_content in pause_segments:
self._check_interrupt(
f"while processing segment {segment_index + 1}"
)
if fragment_type == "text":
clean_text, edit_tags = get_edit_tags_for_segment(
fragment_content
)
generate_chunks(clean_text, edit_tags)
elif fragment_type == "pause":
silence = PauseTagProcessor.create_silence_segment(
fragment_content,
self.SAMPLE_RATE,
torch.device("cpu"),
torch.float32,
)
if silence.dim() == 1:
silence = silence.unsqueeze(0)
segment_records.append(
{
"waveform": silence.cpu(),
"sample_rate": self.SAMPLE_RATE,
"text": f"[pause:{fragment_content}s]",
"edit_tags": [],
}
)
else:
clean_text, edit_tags = get_edit_tags_for_segment(segment_text)
generate_chunks(clean_text, edit_tags)
finally:
self.adapter.update_config(base_config)
if (
apply_edit_postprocessing
and segment_records
and any(record.get("edit_tags") for record in segment_records)
):
self._check_interrupt("before edit post-processing")
segment_records = apply_edit_post_processing(
segment_records, engine_config=base_config
)
for record in segment_records:
record["waveform"] = self._normalize_audio(record["waveform"])
return segment_records
def combine_audio_segments(
self,
segments: List[Dict[str, Any]],
method: str = "auto",
silence_ms: int = 100,
original_text: str = "",
return_info: bool = False,
) -> Union[torch.Tensor, Tuple[torch.Tensor, Dict[str, Any]]]:
self._check_interrupt("before audio assembly")
if not segments:
empty = torch.zeros(0, dtype=torch.float32)
return (empty, {}) if return_info else empty
text_chunks = [segment.get("text", "") for segment in segments]
combined_audio, chunk_info = ChunkTimingHelper.combine_audio_with_timing(
audio_segments=[segment["waveform"] for segment in segments],
combination_method=method,
silence_ms=silence_ms,
crossfade_duration=0.1,
sample_rate=self.SAMPLE_RATE,
text_length=len(" ".join(text_chunks)),
original_text=original_text,
text_chunks=text_chunks,
)
return (combined_audio, chunk_info) if return_info else combined_audio
@@ -0,0 +1,222 @@
"""SRT orchestration for Audio8 TTS."""
import importlib.util
import os
import comfy.model_management as model_management
import torch
from engines.adapters.audio8_tts_adapter import Audio8TTSEngineAdapter
from utils.system.import_manager import import_manager
from utils.timing.assembly import AudioAssemblyEngine
from utils.timing.engine import TimingEngine
from utils.timing.overlap_detection import SRTOverlapHandler
from utils.timing.reporting import SRTReportGenerator
class Audio8TTSSRTProcessor:
"""Generate Audio8 speech per subtitle and apply shared timing modes."""
SAMPLE_RATE = 44100
def __init__(self, node_instance, config):
self.node_instance = node_instance
self.config = dict(config or {})
self.adapter = Audio8TTSEngineAdapter(self.config)
self._processor = None
success, modules, message = import_manager.import_srt_modules()
if not success:
raise ImportError(f"Audio8 TTS SRT unavailable: {message}")
self.SRTParser = modules["SRTParser"]
@property
def processor(self):
if self._processor is None:
processor_path = os.path.join(
os.path.dirname(__file__), "audio8_tts_processor.py"
)
spec = importlib.util.spec_from_file_location(
"audio8_tts_processor_module", processor_path
)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
self._processor = module.Audio8TTSProcessor(self.adapter, self.config)
return self._processor
def update_config(self, config):
self.config = dict(config or {})
self.adapter.update_config(self.config)
if self._processor is not None:
self._processor.update_config(self.config)
@staticmethod
def _check_interrupt(index=None, total=None):
if not model_management.interrupt_processing:
return
location = f" at subtitle {index + 1}/{total}" if index is not None else ""
raise InterruptedError(f"Audio8 TTS SRT generation interrupted{location}")
def process_srt_content(
self,
srt_content,
voice_mapping,
seed,
timing_mode,
timing_params,
enable_audio_cache=True,
):
self._check_interrupt()
subtitles = self.SRTParser().parse_srt_content(srt_content, allow_overlaps=True)
has_overlaps = SRTOverlapHandler.detect_overlaps(subtitles)
active_mode, switched = SRTOverlapHandler.handle_smart_natural_fallback(
timing_mode, has_overlaps, "Audio8 TTS SRT"
)
audio_segments = []
adjustments = []
for index, subtitle in enumerate(subtitles):
self._check_interrupt(index, len(subtitles))
text = (subtitle.text or "").strip()
if text:
records = self.processor.process_text(
text=text,
voice_mapping=voice_mapping,
seed=seed + index,
enable_chunking=False,
enable_audio_cache=enable_audio_cache,
apply_edit_postprocessing=True,
)
audio, _ = self.processor.combine_audio_segments(
records,
method="auto",
silence_ms=0,
original_text=text,
return_info=True,
)
if audio.dim() == 1:
audio = audio.unsqueeze(0)
else:
audio = torch.zeros(1, int(subtitle.duration * self.SAMPLE_RATE))
audio = audio.detach().cpu().float()
audio_segments.append(audio)
natural_duration = audio.shape[-1] / self.SAMPLE_RATE
target_duration = subtitle.duration
stretch_factor = (
target_duration / natural_duration if natural_duration else 1.0
)
adjustments.append(
{
"index": index,
"segment_index": index,
"sequence": subtitle.sequence,
"natural_duration": natural_duration,
"target_start": subtitle.start_time,
"target_end": subtitle.end_time,
"target_duration": target_duration,
"start_time": subtitle.start_time,
"end_time": subtitle.end_time,
"stretch_factor": stretch_factor,
"needs_stretching": (abs(stretch_factor - 1.0) > 0.05),
"stretch_type": (
"compress"
if stretch_factor < 1.0
else "expand"
if stretch_factor > 1.0
else "none"
),
"adjustment": natural_duration - target_duration,
"adjusted_start": subtitle.start_time,
"adjusted_end": subtitle.end_time,
"adjusted_duration": natural_duration,
}
)
self._check_interrupt()
final_audio, replacement, stretch_method = self._assemble(
audio_segments, subtitles, active_mode, timing_params or {}
)
self._check_interrupt()
if replacement is not None:
adjustments = replacement
reporter = SRTReportGenerator()
timing_report = reporter.generate_timing_report(
subtitles,
adjustments,
active_mode,
has_overlaps,
switched,
timing_mode if switched else None,
stretch_method,
)
adjusted_srt = reporter.generate_adjusted_srt_string(
subtitles, adjustments, active_mode
)
final_audio = final_audio.detach().cpu().float()
if final_audio.dim() == 1:
final_audio = final_audio.unsqueeze(0).unsqueeze(0)
elif final_audio.dim() == 2:
final_audio = final_audio.unsqueeze(0)
duration = final_audio.shape[-1] / self.SAMPLE_RATE
generation_info = (
f"Generated {duration:.1f}s Audio8 TTS SRT audio from "
f"{len(subtitles)} subtitles using {active_mode} mode"
)
return (
{"waveform": final_audio, "sample_rate": self.SAMPLE_RATE},
generation_info,
timing_report,
adjusted_srt,
)
def _assemble(self, audio_segments, subtitles, mode, params):
if mode == "stretch_to_fit":
from engines.chatterbox.audio_timing import TimedAudioAssembler
assembler = TimedAudioAssembler(self.SAMPLE_RATE)
audio, method = assembler.assemble_timed_audio(
audio_segments,
[(subtitle.start_time, subtitle.end_time) for subtitle in subtitles],
fade_duration=params.get("fade_for_StretchToFit", 0.01),
)
return audio, None, method
assembler = AudioAssemblyEngine(self.SAMPLE_RATE)
if mode == "pad_with_silence":
audio = assembler.assemble_with_overlaps(
audio_segments, subtitles, torch.device("cpu")
)
return audio, None, None
timing = TimingEngine(self.SAMPLE_RATE)
if mode == "concatenate":
replacements = timing.calculate_concatenation_adjustments(
audio_segments, subtitles
)
audio = assembler.assemble_concatenation(
audio_segments,
params.get("fade_for_StretchToFit", 0.01),
)
return audio, replacements, None
if mode != "smart_natural":
raise ValueError(f"Unsupported Audio8 SRT timing mode: {mode}")
replacements, processed = timing.calculate_smart_timing_adjustments(
audio_segments,
subtitles,
params.get("timing_tolerance", 2.0),
params.get("max_stretch_ratio", 1.0),
params.get("min_stretch_ratio", 0.5),
torch.device("cpu"),
)
audio = assembler.assemble_smart_natural(
audio_segments,
processed,
replacements,
subtitles,
torch.device("cpu"),
)
return audio, replacements, None
+230
View File
@@ -0,0 +1,230 @@
"""Audio8 TTS Preview engine configuration node."""
import importlib.util
import os
import sys
current_dir = os.path.dirname(__file__)
nodes_dir = os.path.dirname(current_dir)
project_root = os.path.dirname(nodes_dir)
if project_root not in sys.path:
sys.path.insert(0, project_root)
base_node_path = os.path.join(nodes_dir, "base", "base_node.py")
base_spec = importlib.util.spec_from_file_location("base_node_module", base_node_path)
base_module = importlib.util.module_from_spec(base_spec)
sys.modules["base_node_module"] = base_module
base_spec.loader.exec_module(base_module)
BaseTTSNode = base_module.BaseTTSNode
class Audio8TTSEngineNode(BaseTTSNode):
"""Configure the official Audio8 TTS Preview checkpoint."""
DEFAULT_MODEL = "Audio8-TTS-Preview-0.6b"
@classmethod
def NAME(cls):
return "⚙️ Audio8 TTS Engine"
@classmethod
def _get_model_options(cls):
try:
from engines.audio8_tts.downloader import Audio8TTSDownloader
options = Audio8TTSDownloader().get_available_models()
return options or [cls.DEFAULT_MODEL]
except Exception:
return [cls.DEFAULT_MODEL]
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"model_variant": (
cls._get_model_options(),
{
"default": cls.DEFAULT_MODEL,
"tooltip": (
"Official 0.6B Audio8 TTS Preview checkpoint. "
"The canonical option downloads from "
"Audio8/Audio8-TTS-Preview-0.6b; local: options are "
"detected model folders.\n\n"
"Preview scope: multilingual speech and zero-shot "
"voice cloning, with 11 recommended languages. The "
"model infers language from text and has no language "
"selection control.\n\n"
"Voice cloning requires reference audio and its exact "
"matching transcript. Generation without a reference "
"is supported. Audio8 has no instruction-conditioned "
"voice-design mode. The suite runs it in the existing "
"shared Transformers 4 runtime for correct cloning. "
"Code and weights: Apache License 2.0."
),
},
),
"device": (
["auto", "cuda", "cpu"],
{
"default": "auto",
"tooltip": (
"Execution device. CUDA is recommended. CPU inference "
"is supported but slow and runs in float32."
),
},
),
"temperature": (
"FLOAT",
{
"default": 0.8,
"min": 0.1,
"max": 2.0,
"step": 0.05,
"tooltip": (
"Official sampling temperature. Used only in Sampling "
"mode; lower values are more conservative."
),
},
),
"top_p": (
"FLOAT",
{
"default": 0.95,
"min": 0.01,
"max": 1.0,
"step": 0.01,
"tooltip": (
"Official nucleus-sampling cutoff. Used only in "
"Sampling mode."
),
},
),
"top_k": (
"INT",
{
"default": 50,
"min": 1,
"max": 100,
"step": 1,
"tooltip": (
"Official top-k sampling limit. Used only in Sampling mode."
),
},
),
"max_new_tokens": (
"INT",
{
"default": 1024,
"min": 64,
"max": 2048,
"step": 64,
"tooltip": (
"Maximum acoustic frames for the first generation "
"attempt. Audio8 emits about 21.5 frames per second. "
"Larger budgets take more time and context memory."
),
},
),
},
"optional": {
"retry_max_new_tokens": (
"INT",
{
"default": 2048,
"min": 64,
"max": 2048,
"step": 8,
"tooltip": (
"Second-attempt frame budget when the first attempt "
"does not emit EOS. Must be at least max_new_tokens. "
"Set it equal to max_new_tokens to disable the larger "
"retry."
),
},
),
"dtype": (
["auto", "bfloat16", "float16", "float32"],
{
"default": "auto",
"tooltip": (
"Model precision. Auto prefers bfloat16 on Ampere-or-"
"newer CUDA GPUs, uses float16 on older CUDA GPUs, and "
"float32 on CPU. CPU coerces reduced-precision choices "
"to float32."
),
},
),
"sampling_mode": (
["Sampling", "Greedy"],
{
"default": "Sampling",
"tooltip": (
"Official decoding mode. Sampling uses temperature, "
"top_p, and top_k. Greedy is deterministic for a fixed "
"prompt and ignores those sampling controls."
),
},
),
},
}
RETURN_TYPES = ("TTS_ENGINE",)
RETURN_NAMES = ("TTS_engine",)
FUNCTION = "create_engine_config"
CATEGORY = "TTS Audio Suite/⚙️ Engines"
def create_engine_config(
self,
model_variant: str,
device: str,
temperature: float,
top_p: float,
top_k: int,
max_new_tokens: int,
retry_max_new_tokens: int = 2048,
dtype: str = "auto",
sampling_mode: str = "Sampling",
) -> tuple:
if retry_max_new_tokens < max_new_tokens:
raise ValueError(
"Audio8 retry_max_new_tokens must be at least max_new_tokens"
)
do_sample = sampling_mode == "Sampling"
config = {
"engine_type": "audio8_tts",
"model_variant": model_variant,
"device": device,
"dtype": dtype,
"max_new_tokens": int(max_new_tokens),
"retry_max_new_tokens": int(retry_max_new_tokens),
"temperature": float(temperature),
"top_p": float(top_p),
"top_k": int(top_k),
"do_sample": do_sample,
}
print(f"⚙️ Audio8 TTS Preview: {model_variant} on {device} ({dtype})")
print(
" Settings: "
f"mode={sampling_mode}, temperature={temperature}, top_p={top_p}, "
f"top_k={top_k}, max_new_tokens={max_new_tokens}, "
f"retry_max_new_tokens={retry_max_new_tokens}"
)
print(
" Voice cloning requires reference audio plus its exact matching "
"transcript; no-reference generation is supported."
)
print(
" Runtime: shared Transformers 4 profile (required for correct cloning)"
)
print(" Preview checkpoint | Apache License 2.0")
return (
{
"engine_type": "audio8_tts",
"config": config,
"adapter_class": "Audio8TTSEngineAdapter",
"capabilities": ["tts"],
},
)
+5 -4
View File
@@ -69,10 +69,11 @@ PRIORITY SYSTEM - When both .txt and .reference.txt exist:
"default": "",
"tooltip": """Create reference text on-the-fly for connected audio input.
ENGINE REQUIREMENTS:
• F5-TTS: REQUIRES reference text (must match spoken audio exactly)
• Higgs Audio 2: Optional but uses reference text if provided
• ChatterBox/VibeVoice/IndexTTS: Don't use reference text
ENGINE REQUIREMENTS:
• Audio8 TTS: REQUIRES the exact spoken transcript for voice cloning
• F5-TTS: REQUIRES reference text (must match spoken audio exactly)
• Higgs Audio 2: Optional but uses reference text if provided
• ChatterBox/VibeVoice/IndexTTS: Don't use reference text
Selecting a library voice loads its transcription here automatically. Edits are temporary workflow overrides and never modify the source .txt file."""
}),
@@ -208,13 +208,20 @@ class StepAudioEditXAudioEditorNode:
self._cached_settings = None
# Check if engine was deleted (MUST be after settings change check, before early return)
engine_was_deleted = (self._engine is not None and
hasattr(self._engine, '_tts_engine') and
self._engine._tts_engine is None)
if engine_was_deleted:
print("⚠️ Step Audio EditX engine was deleted, reloading...")
self._engine = None
engine_was_deleted = self._engine is not None and (
(
hasattr(self._engine, '_tts_engine')
and self._engine._tts_engine is None
)
or (
hasattr(self._engine, '_initialized')
and not self._engine._initialized
)
)
if engine_was_deleted:
print("⚠️ Step Audio EditX engine was unloaded, reloading...")
self._engine = None
elif self._engine is not None:
# Using cached engine - ensure it's on the correct device
from utils.device import resolve_torch_device
+95 -5
View File
@@ -84,7 +84,7 @@ Hello! This is unified SRT TTS with character switching.
}),
"narrator_voice": (reference_files, {
"default": "none",
"tooltip": "Fallback narrator voice from voice folders. Used when opt_narrator is not connected. Select 'none' for engines that support direct TTS without voice cloning, such as MOSS."
"tooltip": "Fallback narrator voice from voice folders. Used when opt_narrator is not connected. Select 'none' for engines that support direct TTS without voice cloning, such as MOSS or Audio8."
}),
"seed": ("INT", {
"default": 1, "min": 0, "max": 2**32 - 1,
@@ -223,6 +223,12 @@ Hello! This is unified SRT TTS with character switching.
stable_params['optimize'] = config.get('optimize', False)
stable_params['max_generate_length'] = config.get('max_generate_length', 500)
if engine_type == "audio8_tts":
stable_params['model_variant'] = config.get(
'model_variant', 'Audio8-TTS-Preview-0.6b'
)
stable_params['dtype'] = config.get('dtype', 'auto')
if engine_type == "dramabox":
stable_params['model_name'] = config.get('model_name', 'DramaBox')
stable_params['precision'] = config.get('precision', 'auto')
@@ -578,6 +584,40 @@ Hello! This is unified SRT TTS with character switching.
}
return engine_instance
elif engine_type == "audio8_tts":
processor_path = os.path.join(
nodes_dir, "audio8_tts", "audio8_tts_srt_processor.py"
)
processor_spec = importlib.util.spec_from_file_location(
"audio8_tts_srt_processor_module", processor_path
)
processor_module = importlib.util.module_from_spec(processor_spec)
processor_spec.loader.exec_module(processor_module)
Audio8TTSSRTProcessor = processor_module.Audio8TTSSRTProcessor
class Audio8TTSSRTWrapper:
def __init__(self, cfg):
self.config = cfg.copy()
self.processor = Audio8TTSSRTProcessor(self, self.config)
def update_config(self, new_config):
self.config = new_config.copy()
self.processor.update_config(new_config)
def check_interrupt(self):
if model_management.interrupt_processing:
raise InterruptedError(
"Audio8 TTS SRT processing interrupted by user"
)
engine_instance = Audio8TTSSRTWrapper(config)
import time
self._cached_engine_instances[cache_key] = {
'instance': engine_instance,
'timestamp': time.time()
}
return engine_instance
elif engine_type == "dramabox":
processor_path = os.path.join(
nodes_dir, "dramabox", "dramabox_srt_processor.py"
@@ -987,7 +1027,13 @@ Hello! This is unified SRT TTS with character switching.
raise ValueError(f"Unknown engine type: {engine_type}")
except Exception as e:
if isinstance(e, InterruptedError):
if isinstance(
e,
(
InterruptedError,
model_management.InterruptProcessingException,
),
):
raise
print(f"❌ Failed to create engine SRT node instance: {e}")
return None
@@ -1053,7 +1099,7 @@ Hello! This is unified SRT TTS with character switching.
print(f"📺 TTS SRT: Using direct audio input ({character_name})")
print(
"⚠️ TTS SRT: Direct audio input has no reference text - "
"F5-TTS and OmniVoice cloning will fail"
"Audio8 TTS, F5-TTS, and OmniVoice cloning will fail"
)
return None, audio_tensor, reference_text, character_name
@@ -1080,7 +1126,13 @@ Hello! This is unified SRT TTS with character switching.
return None, None, "", "narrator"
except Exception as e:
if isinstance(e, InterruptedError):
if isinstance(
e,
(
InterruptedError,
model_management.InterruptProcessingException,
),
):
raise
print(f"❌ Voice reference error: {e}")
return None, None, "", "narrator"
@@ -1175,6 +1227,13 @@ Hello! This is unified SRT TTS with character switching.
"OmniVoice voice cloning requires reference text. "
"Do not connect raw audio directly. Use Character Voices node or a narrator voice with a matching .reference.txt file."
)
if engine_type == "audio8_tts" and (audio_tensor is not None or audio_path) and not reference_text.strip():
raise ValueError(
"Audio8 TTS voice cloning requires reference text. "
"Do not connect raw audio directly. Use Character Voices with "
"the exact transcript, or a narrator voice with a matching "
".reference.txt file."
)
# Create proper engine SRT node instance to preserve ALL functionality
engine_instance = self._create_proper_engine_node_instance(TTS_engine)
@@ -1375,6 +1434,30 @@ Hello! This is unified SRT TTS with character switching.
enable_audio_cache=enable_audio_cache
)
elif engine_type == "audio8_tts":
timing_params = {
'fade_for_StretchToFit': fade_for_StretchToFit,
'max_stretch_ratio': max_stretch_ratio,
'min_stretch_ratio': min_stretch_ratio,
'timing_tolerance': timing_tolerance,
}
voice_mapping = {}
if audio_tensor is not None or audio_path:
voice_mapping['narrator'] = {
'audio': audio_tensor,
'audio_path': audio_path,
'reference_text': reference_text or '',
'character_name': character_name or 'narrator',
}
result = engine_instance.processor.process_srt_content(
srt_content=srt_content,
voice_mapping=voice_mapping,
seed=seed,
timing_mode=timing_mode,
timing_params=timing_params,
enable_audio_cache=enable_audio_cache,
)
elif engine_type == "dramabox":
timing_params = {
'fade_for_StretchToFit': fade_for_StretchToFit,
@@ -1688,9 +1771,16 @@ Hello! This is unified SRT TTS with character switching.
"is a voice-design model and cannot be used with TTS SRT" in msg
or "Pause tags are not compatible with force_speaker_kv" in msg
or "MOSS-TTSD Native Multi-Speaker Dialogue does not support this SRT input" in msg
or "Audio8 TTS voice cloning requires reference text" in msg
):
raise
if isinstance(e, InterruptedError):
if isinstance(
e,
(
InterruptedError,
model_management.InterruptProcessingException,
),
):
raise
error_msg = f"❌ TTS SRT generation failed: {e}"
print(error_msg)
+145 -8
View File
@@ -89,7 +89,7 @@ Back to the main narrator voice for the conclusion.""",
}),
"narrator_voice": (reference_files, {
"default": "none",
"tooltip": "Fallback narrator voice from voice folders. Used when opt_narrator is not connected. Select 'none' for engines that support direct TTS without voice cloning, such as MOSS."
"tooltip": "Fallback narrator voice from voice folders. Used when opt_narrator is not connected. Select 'none' for engines that support direct TTS without voice cloning, such as MOSS or Audio8."
}),
"seed": ("INT", {
"default": 1, "min": 0, "max": 2**32 - 1,
@@ -218,6 +218,12 @@ Back to the main narrator voice for the conclusion.""",
stable_params['optimize'] = config.get('optimize', False)
stable_params['max_generate_length'] = config.get('max_generate_length', 500)
if engine_type == "audio8_tts":
stable_params['model_variant'] = config.get(
'model_variant', 'Audio8-TTS-Preview-0.6b'
)
stable_params['dtype'] = config.get('dtype', 'auto')
if engine_type == "dramabox":
stable_params['model_name'] = config.get('model_name', 'DramaBox')
stable_params['precision'] = config.get('precision', 'auto')
@@ -445,7 +451,13 @@ Back to the main narrator voice for the conclusion.""",
voice_mapping[char] = {"waveform": waveform, "sample_rate": sample_rate}
print(f"🎭 VibeVoice: Using character-specific voice for '{char}'")
except Exception as e:
if isinstance(e, InterruptedError):
if isinstance(
e,
(
InterruptedError,
model_management.InterruptProcessingException,
),
):
raise
print(f"⚠️ Failed to load character audio for '{char}': {e}")
voice_mapping[char] = char_audio # Fallback to main voice
@@ -580,6 +592,38 @@ Back to the main narrator voice for the conclusion.""",
return engine_instance
elif engine_type == "audio8_tts":
from engines.adapters.audio8_tts_adapter import Audio8TTSEngineAdapter
processor_path = os.path.join(
nodes_dir, "audio8_tts", "audio8_tts_processor.py"
)
processor_spec = importlib.util.spec_from_file_location(
"audio8_tts_processor_module", processor_path
)
processor_module = importlib.util.module_from_spec(processor_spec)
processor_spec.loader.exec_module(processor_module)
Audio8TTSProcessor = processor_module.Audio8TTSProcessor
class Audio8TTSWrapper:
def __init__(self, cfg):
self.config = cfg.copy()
self.adapter = Audio8TTSEngineAdapter(self.config)
self.processor = Audio8TTSProcessor(self.adapter, self.config)
def update_config(self, new_config):
self.config = new_config.copy()
self.adapter.update_config(new_config)
self.processor.update_config(new_config)
engine_instance = Audio8TTSWrapper(config)
import time
self._cached_engine_instances[cache_key] = {
'instance': engine_instance,
'timestamp': time.time()
}
return engine_instance
elif engine_type == "dramabox":
from engines.adapters.dramabox_adapter import DramaBoxEngineAdapter
processor_path = os.path.join(nodes_dir, "dramabox", "dramabox_processor.py")
@@ -837,7 +881,13 @@ Back to the main narrator voice for the conclusion.""",
raise ValueError(f"Unknown engine type: {engine_type}")
except Exception as e:
if isinstance(e, InterruptedError):
if isinstance(
e,
(
InterruptedError,
model_management.InterruptProcessingException,
),
):
raise
if "MOSS LoRA/base model mismatch" in str(e):
raise
@@ -906,7 +956,8 @@ Back to the main narrator voice for the conclusion.""",
print(f"🎤 TTS Text: Using direct audio input ({character_name})")
print(
"⚠️ TTS Text: Direct audio input has no reference text - "
"F5-TTS and OmniVoice cloning will fail, Qwen3-TTS will use x_vector_only mode (lower quality)"
"Audio8 TTS, F5-TTS, and OmniVoice cloning will fail; "
"Qwen3-TTS will use x_vector_only mode (lower quality)"
)
return None, audio_tensor, reference_text, character_name
@@ -952,7 +1003,13 @@ Back to the main narrator voice for the conclusion.""",
return None, None, "", "narrator"
except Exception as e:
if isinstance(e, InterruptedError):
if isinstance(
e,
(
InterruptedError,
model_management.InterruptProcessingException,
),
):
raise
print(f"❌ Voice reference error: {e}")
return None, None, "", "narrator"
@@ -1059,6 +1116,13 @@ Back to the main narrator voice for the conclusion.""",
"OmniVoice voice cloning requires reference text. "
"Do not connect raw audio directly. Use Character Voices node or a narrator voice with a matching .reference.txt file."
)
if engine_type == "audio8_tts" and (audio_tensor is not None or audio_path) and not reference_text.strip():
raise ValueError(
"Audio8 TTS voice cloning requires reference text. "
"Do not connect raw audio directly. Use Character Voices with "
"the exact transcript, or a narrator voice with a matching "
".reference.txt file."
)
# Create proper engine node instance to preserve ALL functionality
engine_instance = self._create_proper_engine_node_instance(TTS_engine)
@@ -1520,6 +1584,59 @@ Back to the main narrator voice for the conclusion.""",
formatted_audio = AudioProcessingUtils.format_for_comfyui(combined_audio, 48000)
result = (formatted_audio, generation_info)
elif engine_type == "audio8_tts":
import re
from utils.audio.chunk_timing import ChunkTimingHelper
voice_mapping = {}
if audio_tensor is not None or audio_path:
voice_mapping['narrator'] = {
'audio': audio_tensor,
'audio_path': audio_path,
'reference_text': reference_text or '',
'character_name': character_name or 'narrator',
}
segment_records = engine_instance.processor.process_text(
text=text,
voice_mapping=voice_mapping,
seed=seed,
enable_chunking=enable_chunking,
max_chars_per_chunk=max_chars_per_chunk,
chunk_combination_method=chunk_combination_method,
silence_between_chunks_ms=silence_between_chunks_ms,
enable_audio_cache=enable_audio_cache,
)
combined_audio, chunk_info = engine_instance.processor.combine_audio_segments(
segments=segment_records,
method=chunk_combination_method,
silence_ms=silence_between_chunks_ms,
original_text=text,
return_info=True,
)
total_duration = (
combined_audio.shape[-1] / 44100.0
if combined_audio.numel()
else 0.0
)
clean_text = re.sub(r'\[.*?\]', '', text)
base_info = (
f"Generated {total_duration:.1f}s audio from {len(clean_text)} characters "
f"(Audio8 TTS, narrator: {char_display})"
)
base_info += (
"\n🎭 Reference-free speech, zero-shot cloning, character switching, "
"and pause tags enabled"
)
generation_info = ChunkTimingHelper.enhance_generation_info(
f"✅ {base_info}", chunk_info
)
result = (
AudioProcessingUtils.format_for_comfyui(combined_audio, 44100),
generation_info,
)
elif engine_type == "dramabox":
import re
from utils.audio.chunk_timing import ChunkTimingHelper
@@ -1900,7 +2017,13 @@ Back to the main narrator voice for the conclusion.""",
}
print(f"🎭 Qwen3-TTS: Using character-specific voice for '{character}' (ICL mode)")
except Exception as e:
if isinstance(e, InterruptedError):
if isinstance(
e,
(
InterruptedError,
model_management.InterruptProcessingException,
),
):
raise
print(f"⚠️ Failed to load character audio for '{character}': {e}")
# Fallback to narrator voice if available
@@ -1945,7 +2068,13 @@ Back to the main narrator voice for the conclusion.""",
print(f"⚠️⚠️ Qwen3-TTS: Character '{character}' has audio but NO reference text")
print(f"⚠️⚠️ Using x_vector_only mode (speaker embedding only) - LOWER QUALITY than ICL mode")
except Exception as e:
if isinstance(e, InterruptedError):
if isinstance(
e,
(
InterruptedError,
model_management.InterruptProcessingException,
),
):
raise
print(f"⚠️ Failed to load character audio for '{character}': {e}")
# Fallback to narrator voice if available
@@ -2082,9 +2211,17 @@ Back to the main narrator voice for the conclusion.""",
raise
if "MOSS LoRA/base model mismatch" in str(e):
raise
if "Audio8 TTS voice cloning requires reference text" in str(e):
raise
if engine_type == "index_tts":
raise
if isinstance(e, InterruptedError):
if isinstance(
e,
(
InterruptedError,
model_management.InterruptProcessingException,
),
):
raise
error_msg = f"❌ TTS Text generation failed: {e}"
print(error_msg)
+40
View File
@@ -1,4 +1,7 @@
import json
import os
from pathlib import Path
import subprocess
import sys
import pytest
@@ -41,3 +44,40 @@ def test_build_env_preserves_valid_pythonhashseed(monkeypatch, value):
env = IsolatedRuntimeLauncher().build_env(profile)
assert env["PYTHONHASHSEED"] == value
@pytest.mark.unit
def test_audio8_worker_prioritizes_project_root_over_shadowing_utils(tmp_path):
shadow_dir = tmp_path / "step_audio_editx_impl"
shadow_dir.mkdir()
(shadow_dir / "utils.py").write_text(
'raise RuntimeError("shadow utils imported")\n',
encoding="utf-8",
)
worker_path = REPO_ROOT / "utils" / "runtimes" / "workers" / "audio8_tts_worker.py"
env = os.environ.copy()
env["PYTHONPATH"] = os.pathsep.join((str(shadow_dir), str(REPO_ROOT)))
completed = subprocess.run(
[sys.executable, str(worker_path)],
input=json.dumps(
{
"action": "ping",
"request_id": "audio8-path-priority-test",
}
)
+ "\n",
capture_output=True,
text=True,
encoding="utf-8",
errors="replace",
env=env,
cwd=str(REPO_ROOT),
timeout=30,
check=False,
)
assert completed.returncode == 0, completed.stderr
response = json.loads(completed.stdout.strip())
assert response["ok"] is True
assert response["result"] == {"pong": True}
+28 -1
View File
@@ -577,6 +577,30 @@ class DotsTTSCacheKeyGenerator(CacheKeyGenerator):
return hashlib.md5(cache_string.encode()).hexdigest()
class Audio8TTSCacheKeyGenerator(CacheKeyGenerator):
"""Cache key generator for official Audio8 TTS inference."""
def generate_cache_key(self, **params) -> str:
cache_data = {
'text': params.get('text', ''),
'audio_component': params.get('audio_component', ''),
'reference_text': params.get('reference_text', ''),
'model_variant': params.get('model_variant', 'Audio8-TTS-Preview-0.6b'),
'max_new_tokens': int(params.get('max_new_tokens', 1024)),
'retry_max_new_tokens': int(params.get('retry_max_new_tokens', 2000)),
'temperature': round(float(params.get('temperature', 0.8)), 4),
'top_p': round(float(params.get('top_p', 0.95)), 4),
'top_k': int(params.get('top_k', 50)),
'do_sample': bool(params.get('do_sample', True)),
'dtype': params.get('dtype', 'auto'),
'device': params.get('device', 'auto'),
'seed': int(params.get('seed', 42)),
'character': params.get('character', 'narrator'),
'engine': 'audio8_tts',
}
return hashlib.md5(str(sorted(cache_data.items())).encode()).hexdigest()
class DramaBoxCacheKeyGenerator(CacheKeyGenerator):
"""Cache key generator for official DramaBox inference."""
@@ -710,6 +734,7 @@ class AudioCache:
'cosyvoice': CosyVoiceCacheKeyGenerator(),
'qwen3_tts': Qwen3TTSCacheKeyGenerator(),
'dots_tts': DotsTTSCacheKeyGenerator(),
'audio8_tts': Audio8TTSCacheKeyGenerator(),
'dramabox': DramaBoxCacheKeyGenerator(),
'fish_audio_s2': FishAudioS2CacheKeyGenerator(),
'omnivoice': OmniVoiceCacheKeyGenerator(),
@@ -792,7 +817,9 @@ class AudioCache:
sample_rate = 24000
elif engine_type in ('index_tts', 'cosyvoice'):
sample_rate = 22050
else:
elif engine_type == 'audio8_tts':
sample_rate = 44100
else:
sample_rate = 44100
return num_samples / sample_rate
+9
View File
@@ -157,6 +157,15 @@ ENGINE_REGISTRY: Dict[str, EngineCapabilities] = {
fallback_languages=["English"],
),
"audio8_tts": EngineCapabilities(
supports_voice_conversion=False,
multilingual_model_switching=False,
can_corrupt_on_reload=False,
fallback_languages=["English"],
supports_runtime_isolation=True,
default_runtime_profile="vibevoice_transformers4_shared",
),
"dramabox": EngineCapabilities(
supports_voice_conversion=False,
multilingual_model_switching=False,
+36 -1
View File
@@ -11,9 +11,14 @@ from typing import Any, Dict, Optional, Callable, Union
from pathlib import Path
from utils.models.comfyui_model_wrapper import tts_model_manager, ModelInfo
from utils.models.factory_config import ModelLoadConfig, runtime_uses_isolation
from utils.models.factory_config import (
ModelLoadConfig,
RUNTIME_MODE_SHARED,
runtime_uses_isolation,
)
from utils.models.engine_registry import get_default_runtime_profile
from utils.runtimes import (
build_audio8_tts_isolated_proxy,
build_fish_audio_s2_proxy,
build_higgs_audio_isolated_proxy,
build_qwen3_asr_isolated_proxy,
@@ -124,6 +129,10 @@ class UnifiedModelInterface:
# Check PyTorch consistency on first model load
self._check_pytorch_consistency()
if config.engine_name == "audio8_tts":
config.runtime_mode = RUNTIME_MODE_SHARED
config.runtime_profile = "vibevoice_transformers4_shared"
if runtime_uses_isolation(config.runtime_mode):
return self._load_isolated_model(config, force_reload=force_reload)
@@ -219,6 +228,13 @@ class UnifiedModelInterface:
cached = self._isolated_model_cache.get(cache_key)
if cached is not None:
ensure_registration = getattr(
cached,
"_ensure_comfy_model_registration",
None,
)
if callable(ensure_registration):
ensure_registration()
return cached
profile_name = config.runtime_profile or get_default_runtime_profile(config.engine_name or "")
@@ -244,6 +260,11 @@ class UnifiedModelInterface:
self._isolated_model_cache[cache_key] = proxy
return proxy
if config.engine_name == "audio8_tts" and config.model_type == "tts":
proxy = build_audio8_tts_isolated_proxy(config)
self._isolated_model_cache[cache_key] = proxy
return proxy
if config.engine_name == "qwen3_asr" and config.model_type in ("asr", "aligner"):
proxy = build_qwen3_asr_isolated_proxy(config)
self._isolated_model_cache[cache_key] = proxy
@@ -1751,6 +1772,19 @@ def register_dots_tts_factory():
unified_model_interface.register_model_factory("dots_tts", "tts", dots_tts_factory)
def register_audio8_tts_factory():
"""Register Audio8 TTS through its required Transformers 4 runtime."""
def audio8_tts_factory(config: ModelLoadConfig):
del config
raise RuntimeError(
"Audio8 TTS reached its embedded factory unexpectedly; all "
"Audio8 loads must be routed through the shared Transformers 4 "
"runtime"
)
unified_model_interface.register_model_factory("audio8_tts", "tts", audio8_tts_factory)
def register_dramabox_factory():
"""Register official DramaBox in the main Transformers 5 environment."""
def dramabox_factory(config: ModelLoadConfig):
@@ -2018,6 +2052,7 @@ def initialize_all_factories():
register_qwen3_tts_factory()
register_fish_audio_s2_factory()
register_dots_tts_factory()
register_audio8_tts_factory()
register_dramabox_factory()
register_omnivoice_factory()
register_qwen3_asr_factory()
+3
View File
@@ -6,6 +6,7 @@ separate Python environments without changing the existing ComfyUI-side UI.
"""
from .bootstrap import ensure_runtime, resolve_runtime_dir, resolve_runtime_python
from .audio8_tts_proxy import Audio8TTSIsolatedProxy, build_audio8_tts_isolated_proxy
from .higgs_audio_proxy import HiggsAudioIsolatedProxy, build_higgs_audio_isolated_proxy
from .fish_audio_s2_proxy import FishAudioS2Proxy, build_fish_audio_s2_proxy
from .launcher import IsolatedRuntimeLauncher
@@ -20,6 +21,8 @@ __all__ = [
"ensure_runtime",
"resolve_runtime_dir",
"resolve_runtime_python",
"Audio8TTSIsolatedProxy",
"build_audio8_tts_isolated_proxy",
"HiggsAudioIsolatedProxy",
"IsolatedRuntimeLauncher",
"RuntimeProfile",
+546
View File
@@ -0,0 +1,546 @@
"""Audio8 TTS proxy for the shared official Transformers 4 runtime."""
from __future__ import annotations
import os
import sys
import tempfile
import threading
import uuid
import weakref
from pathlib import Path
from typing import Any, Dict, Optional
import torch
from utils.models.factory_config import ModelLoadConfig
from .bootstrap import PROJECT_ROOT, ensure_runtime
from .launcher import IsolatedRuntimeLauncher
from .profiles import RuntimeProfile, get_runtime_profile
from .protocol import RuntimeJobRequest
from .session import JsonLineWorkerSession
_INTERRUPT_EVENTS: set[threading.Event] = set()
_INTERRUPT_EVENTS_LOCK = threading.Lock()
_INTERRUPT_DISPATCH_LOCK = threading.Lock()
def _install_comfy_interrupt_dispatcher() -> None:
"""Forward Comfy's server interrupt edge to active Audio8 requests."""
server_module = sys.modules.get("server")
comfy_nodes = getattr(server_module, "nodes", None)
if comfy_nodes is None:
comfy_nodes = sys.modules.get("nodes")
if comfy_nodes is None:
return
with _INTERRUPT_DISPATCH_LOCK:
current_interrupt = getattr(comfy_nodes, "interrupt_processing", None)
if not callable(current_interrupt):
return
if getattr(current_interrupt, "_tts_audio8_dispatcher", False):
if (
getattr(current_interrupt, "_tts_audio8_events", None)
is _INTERRUPT_EVENTS
):
return
current_interrupt = getattr(
current_interrupt,
"_tts_audio8_original_interrupt",
current_interrupt,
)
def dispatch_interrupt(value=True):
# TTS Audio Suite patch: preserve Comfy's interrupt edge for
# blocking Audio8 subprocess requests without replacing the
# process-wide callback separately for every generation.
result = current_interrupt(value)
if value:
with _INTERRUPT_EVENTS_LOCK:
active_events = tuple(_INTERRUPT_EVENTS)
for active_event in active_events:
active_event.set()
return result
dispatch_interrupt._tts_audio8_dispatcher = True
dispatch_interrupt._tts_audio8_original_interrupt = current_interrupt
dispatch_interrupt._tts_audio8_events = _INTERRUPT_EVENTS
comfy_nodes.interrupt_processing = dispatch_interrupt
def _register_interrupt_event(interrupt_event: threading.Event) -> None:
_install_comfy_interrupt_dispatcher()
with _INTERRUPT_EVENTS_LOCK:
_INTERRUPT_EVENTS.add(interrupt_event)
def _unregister_interrupt_event(interrupt_event: threading.Event) -> None:
with _INTERRUPT_EVENTS_LOCK:
_INTERRUPT_EVENTS.discard(interrupt_event)
class Audio8TTSIsolatedProxy:
"""Expose Audio8's engine API while inference runs under Transformers 4."""
SAMPLE_RATE = 44100
def __init__(self, config: ModelLoadConfig, profile: RuntimeProfile):
self.config = config
self.profile = profile
self.model_name = config.model_name or "Audio8-TTS-Preview-0.6b"
self.model_path = config.model_path
self.device = str(config.device)
self.current_device = self.device
self.dtype_name = str((config.additional_params or {}).get("dtype", "auto"))
self.dtype = self._resolve_dtype(self.dtype_name)
self.load_device = self.current_loaded_device()
self.offload_device = torch.device("cpu")
self.currently_used = True
self.model = self
self.engine = self
self.parent = None
self.model_options = {}
self.model_keys = set()
estimate_gib = (
3.5
if self.device.startswith("cpu") or self.dtype == torch.float32
else 1.75
)
self._estimated_memory_size = int(estimate_gib * 1024**3)
self._comfy_loaded_model = None
self._comfy_model_management = None
self._initialized = False
launcher = IsolatedRuntimeLauncher(runtime_root=str(PROJECT_ROOT))
python_path = ensure_runtime(profile)
worker_script = PROJECT_ROOT / "utils/runtimes/workers/audio8_tts_worker.py"
self._session = JsonLineWorkerSession(
python_path=str(python_path),
worker_script=str(worker_script),
env=launcher.build_env(profile),
)
try:
self._initialize_remote_engine()
except Exception:
self._session.close()
raise
self._register_with_comfy_model_management()
def _resolve_dtype(self, dtype_name: str) -> torch.dtype:
normalized = str(dtype_name or "auto").lower()
if not self.device.startswith("cuda"):
return torch.float32
if normalized == "float32":
return torch.float32
if normalized == "float16":
return torch.float16
if normalized == "bfloat16":
return torch.bfloat16
if torch.cuda.is_available():
major, _minor = torch.cuda.get_device_capability(torch.device(self.device))
return torch.bfloat16 if major >= 8 else torch.float16
return torch.float16
def _request(self, action: str, payload: Dict[str, Any]):
response = self._session.request(
RuntimeJobRequest(
engine_name="audio8_tts",
action=action,
model_name=self.model_name,
device=self.device,
runtime_profile=self.profile.name,
payload=payload,
request_id=str(uuid.uuid4()),
)
)
if not response.ok:
details = response.error or (
f"Audio8 TTS isolated action '{action}' failed"
)
if response.logs:
details = f"{details}\n" + "\n".join(response.logs)
raise RuntimeError(details)
return response
def _request_generation(self, payload: Dict[str, Any]):
"""Run generation while the main process monitors Comfy interruption."""
try:
import comfy.model_management as model_management
except Exception:
return self._request("generate", payload)
checker = getattr(
model_management,
"throw_exception_if_processing_interrupted",
None,
)
if not callable(checker):
return self._request("generate", payload)
interrupt_event = threading.Event()
_register_interrupt_event(interrupt_event)
result: Dict[str, Any] = {}
request_thread = None
request_started = False
def run_request() -> None:
try:
result["response"] = self._request("generate", payload)
except BaseException as exc:
result["error"] = exc
try:
checker()
request_thread = threading.Thread(
target=run_request,
name="audio8-tts-worker-request",
daemon=True,
)
request_thread.start()
request_started = True
while request_thread.is_alive():
request_thread.join(timeout=0.05)
if interrupt_event.is_set():
raise model_management.InterruptProcessingException()
checker()
except BaseException as exc:
if request_started:
self._initialized = False
try:
self._session.terminate_now()
except Exception:
pass
request_thread.join(timeout=5)
if isinstance(
exc,
model_management.InterruptProcessingException,
):
interrupt_setter = getattr(
model_management,
"interrupt_current_processing",
None,
)
if callable(interrupt_setter):
interrupt_setter(False)
raise
finally:
_unregister_interrupt_event(interrupt_event)
if "error" in result:
raise result["error"]
return result["response"]
def _initialize_remote_engine(self) -> None:
self._request(
"initialize",
{
"model_path": self.model_path,
"dtype": self.dtype_name,
},
)
self._initialized = True
print(f"✅ Audio8 TTS isolated runtime ready ({self.profile.name})")
def _ensure_remote_engine(self) -> None:
process = getattr(self._session, "_process", None)
if not self._initialized or process is None or process.poll() is not None:
self._initialize_remote_engine()
@staticmethod
def _serialize_reference_audio(
reference_audio: Any,
bundle_dir: Path,
) -> Optional[Dict[str, Any]]:
if reference_audio is None:
return None
if isinstance(reference_audio, (str, os.PathLike)):
return {
"kind": "audio_path",
"audio_path": os.fspath(reference_audio),
}
waveform = None
sample_rate = None
if isinstance(reference_audio, dict):
waveform = reference_audio.get(
"waveform",
reference_audio.get("array"),
)
sample_rate = reference_audio.get(
"sample_rate",
reference_audio.get("sampling_rate"),
)
elif isinstance(reference_audio, (tuple, list)) and len(reference_audio) == 2:
waveform, sample_rate = reference_audio
if waveform is None or sample_rate is None:
raise TypeError(
"Audio8 TTS isolated reference audio must be a path, "
"audio dict, or (waveform, sample_rate) pair"
)
tensor_path = bundle_dir / "reference_audio.pt"
waveform_tensor = torch.as_tensor(waveform).detach().float().cpu()
torch.save(
{
"waveform": waveform_tensor,
"sample_rate": int(sample_rate),
},
tensor_path,
)
return {
"kind": "tensor_path",
"tensor_path": str(tensor_path),
}
def generate(
self,
text: str,
reference_audio: Any = None,
reference_text: Optional[str] = None,
*,
max_new_tokens: int = 1024,
retry_max_new_tokens: int = 2000,
temperature: float = 0.8,
top_p: float = 0.95,
top_k: int = 50,
do_sample: bool = True,
seed: int = 42,
) -> torch.Tensor:
self._ensure_remote_engine()
with tempfile.TemporaryDirectory(prefix="tts_audio8_iso_") as temp_dir:
bundle_dir = Path(temp_dir)
output_path = bundle_dir / "result.pt"
serialized_reference = self._serialize_reference_audio(
reference_audio,
bundle_dir,
)
self._request_generation(
{
"text": text,
"reference_audio": serialized_reference,
"reference_text": reference_text,
"max_new_tokens": int(max_new_tokens),
"retry_max_new_tokens": int(retry_max_new_tokens),
"temperature": float(temperature),
"top_p": float(top_p),
"top_k": int(top_k),
"do_sample": bool(do_sample),
"seed": int(seed),
"output_path": str(output_path),
}
)
if not output_path.exists():
raise RuntimeError(
"Audio8 TTS isolated worker returned no output payload"
)
result = torch.load(output_path, map_location="cpu")
audio = torch.as_tensor(result["audio"]).detach().float().cpu()
sample_rate = int(result["sample_rate"])
if sample_rate != self.SAMPLE_RATE:
raise RuntimeError(
f"Audio8 TTS worker returned {sample_rate} Hz; "
f"expected {self.SAMPLE_RATE} Hz"
)
if audio.ndim == 1:
audio = audio.unsqueeze(0)
if audio.ndim != 2 or audio.shape[0] != 1:
raise RuntimeError(
"Audio8 TTS worker returned invalid audio shape "
f"{tuple(audio.shape)}"
)
return audio.contiguous()
def _register_with_comfy_model_management(self) -> None:
self._ensure_comfy_model_registration()
def _ensure_comfy_model_registration(self) -> None:
try:
import comfy.model_management as model_management
except Exception:
return
if not hasattr(model_management, "LoadedModel") or not hasattr(
model_management,
"current_loaded_models",
):
return
for loaded_model in model_management.current_loaded_models:
wrapper = getattr(loaded_model, "_tts_wrapper_ref", None)
real_model = getattr(loaded_model, "real_model", None)
resolved_real_model = (
real_model()
if isinstance(real_model, weakref.ReferenceType)
else real_model
)
if wrapper is self or resolved_real_model is self:
self._comfy_loaded_model = loaded_model
self._comfy_model_management = model_management
return
try:
loaded_model = model_management.LoadedModel(self)
loaded_model.real_model = weakref.ref(self)
finalizer = getattr(
model_management,
"cleanup_models",
lambda: None,
)
loaded_model.model_finalizer = weakref.finalize(self, finalizer)
loaded_model._tts_wrapper_ref = self
model_management.current_loaded_models.insert(0, loaded_model)
self._comfy_loaded_model = loaded_model
self._comfy_model_management = model_management
except Exception as exc:
print(
"⚠️ Failed to register isolated Audio8 TTS runtime with "
f"ComfyUI model management: {exc}"
)
def _unregister_from_comfy_model_management(self) -> None:
model_management = self._comfy_model_management
loaded_model = self._comfy_loaded_model
if model_management is None or loaded_model is None:
return
try:
if loaded_model in model_management.current_loaded_models:
model_management.current_loaded_models.remove(loaded_model)
except Exception as exc:
print(
"⚠️ Failed to remove isolated Audio8 TTS runtime from "
f"ComfyUI tracking: {exc}"
)
finally:
self._comfy_loaded_model = None
self._comfy_model_management = None
def to(self, device):
self.device = str(device)
self.current_device = self.device
self.load_device = self.current_loaded_device()
return self
def eval(self):
return self
def loaded_size(self) -> int:
if self._initialized and self.device.startswith("cuda"):
return self._estimated_memory_size
return 0
def model_size(self) -> int:
return self._estimated_memory_size
def model_memory(self) -> int:
return self._estimated_memory_size
def get_ram_usage(self) -> int:
return self._estimated_memory_size
def model_offloaded_memory(self) -> int:
return (
0
if self._initialized and self.device.startswith("cuda")
else self._estimated_memory_size
)
def model_mmap_residency(self, free: bool = False) -> tuple[int, int]:
del free
return 0, self._estimated_memory_size
def pinned_memory_size(self) -> int:
return 0
def lowvram_patch_counter(self) -> int:
return 0
def model_dtype(self):
return self.dtype
def model_patches_models(self):
return ()
def model_patches_to(self, target) -> None:
if isinstance(target, torch.dtype):
self.dtype = target
elif isinstance(target, torch.device):
self.device = str(target)
def is_dynamic(self) -> bool:
return False
def current_loaded_device(self) -> torch.device:
if self.device.startswith("cuda") and torch.cuda.is_available():
return torch.device(self.device)
return torch.device("cpu")
def partially_load(
self,
device,
extra_memory,
force_patch_weights=False,
) -> int:
del extra_memory, force_patch_weights
self.to(device)
return 0
def partially_unload_ram(self, ram_to_unload: int) -> int:
return self.partially_unload("cpu", ram_to_unload)
def partially_unload(self, device, memory_to_free) -> int:
del device, memory_to_free
freed = self.loaded_size()
self.cleanup(unregister=False)
return freed
def model_unload(
self,
memory_to_free=None,
unpatch_weights=True,
) -> bool:
del memory_to_free, unpatch_weights
self.cleanup(unregister=False)
return True
def detach(self, unpatch_weights=True) -> None:
del unpatch_weights
self.cleanup(unregister=False)
def is_clone(self, other: Any) -> bool:
return other is self
def cleanup(self, unregister: bool = True) -> None:
if unregister:
self._unregister_from_comfy_model_management()
else:
self._comfy_loaded_model = None
self._comfy_model_management = None
self._initialized = False
if getattr(self, "_session", None) is not None:
self._session.close()
close = cleanup
def __del__(self):
try:
self.cleanup()
except Exception:
pass
def build_audio8_tts_isolated_proxy(
config: ModelLoadConfig,
) -> Audio8TTSIsolatedProxy:
profile_name = config.runtime_profile or "vibevoice_transformers4_shared"
profile = get_runtime_profile(profile_name)
if profile is None:
raise RuntimeError(
f"Unknown isolated runtime profile '{profile_name}' for Audio8 TTS"
)
if profile.name != "vibevoice_transformers4_shared":
raise RuntimeError(
"Audio8 TTS requires the shared Transformers 4 runtime profile "
"'vibevoice_transformers4_shared'"
)
config.runtime_profile = profile.name
return Audio8TTSIsolatedProxy(config=config, profile=profile)
+6 -3
View File
@@ -28,7 +28,7 @@ _VIBEVOICE_T4_PACKAGES = [
"numpy>=1.26.4,<2.3.0",
"soundfile>=0.12.0",
"omegaconf>=2.3.0",
"transformers>=4.51.3,<=4.57.3",
"transformers>=4.57.0,<=4.57.3",
"kernels>=0.6.1,<=0.9",
"accelerate",
"requests",
@@ -63,9 +63,12 @@ _QWEN3_T4_PACKAGES = [
RUNTIME_PROFILES: Dict[str, RuntimeProfile] = {
"vibevoice_transformers4_shared": RuntimeProfile(
name="vibevoice_transformers4_shared",
engine_names=["vibevoice", "step_audio_editx"],
engine_names=["vibevoice", "step_audio_editx", "audio8_tts"],
python_path_hint="runtimes/shared_legacy_t4/Scripts/python.exe",
description="Shared legacy Transformers 4 runtime for VibeVoice/Kugel, Step Audio EditX, and similar engines.",
description=(
"Shared legacy Transformers 4 runtime for VibeVoice/Kugel, "
"Step Audio EditX, Audio8 TTS, and similar engines."
),
inherit_base_site_packages=True,
pip_packages=list(_VIBEVOICE_T4_PACKAGES),
pip_packages_no_deps=[
+20 -2
View File
@@ -1,9 +1,9 @@
from __future__ import annotations
"""
Persistent JSON-line worker session for isolated runtimes.
"""
from __future__ import annotations
import json
import subprocess
import threading
@@ -118,6 +118,24 @@ class JsonLineWorkerSession:
self._process = None
def terminate_now(self) -> None:
"""Stop a worker without waiting for the request lock."""
process = self._process
if process is None or process.poll() is not None:
return
try:
process.terminate()
except OSError:
return
try:
process.wait(timeout=2)
except subprocess.TimeoutExpired:
try:
process.kill()
process.wait(timeout=2)
except OSError:
pass
def __del__(self):
try:
self.close()
+191
View File
@@ -0,0 +1,191 @@
"""Persistent Audio8 TTS worker for the shared Transformers 4 runtime."""
from __future__ import annotations
import json
import sys
import traceback
from pathlib import Path
from typing import Any, Dict, Optional
import torch
PROJECT_ROOT = Path(__file__).resolve().parents[3]
project_root_str = str(PROJECT_ROOT)
sys.path = [
entry for entry in sys.path if str(Path(entry).resolve()) != project_root_str
]
sys.path.insert(0, project_root_str)
from utils.runtimes.protocol import RuntimeJobResponse # noqa: E402
def _emit(protocol_out, response: RuntimeJobResponse) -> None:
protocol_out.write(json.dumps(response.to_dict(), ensure_ascii=True) + "\n")
protocol_out.flush()
def _load_reference_audio(
payload: Optional[Dict[str, Any]],
) -> Any:
if payload is None:
return None
kind = payload.get("kind")
if kind == "audio_path":
audio_path = payload.get("audio_path")
if not audio_path:
raise RuntimeError("Audio8 TTS audio_path payload is missing its file path")
return audio_path
if kind == "tensor_path":
tensor_path = payload.get("tensor_path")
if not tensor_path:
raise RuntimeError(
"Audio8 TTS tensor_path payload is missing its file path"
)
tensor_payload = torch.load(tensor_path, map_location="cpu")
return {
"array": tensor_payload["waveform"],
"sampling_rate": int(tensor_payload["sample_rate"]),
}
raise RuntimeError(f"Unsupported Audio8 TTS reference payload kind: {kind}")
def main() -> int:
protocol_out = sys.stdout
sys.stdout = sys.stderr
engine = None
for line in sys.stdin:
stripped = line.strip()
if not stripped:
continue
request = None
try:
request = json.loads(stripped)
action = request.get("action")
payload = request.get("payload") or {}
request_id = request.get("request_id")
if action == "shutdown":
_emit(
protocol_out,
RuntimeJobResponse(
ok=True,
result={"shutdown": True},
request_id=request_id,
),
)
break
if action == "ping":
_emit(
protocol_out,
RuntimeJobResponse(
ok=True,
result={"pong": True},
request_id=request_id,
),
)
continue
if action == "initialize":
import transformers
from packaging.version import Version
transformers_version = Version(transformers.__version__)
if not (Version("4.57.0") <= transformers_version < Version("5.0.0")):
raise RuntimeError(
"Audio8 TTS requires Transformers >=4.57.0,<5; "
f"worker loaded Transformers {transformers.__version__}"
)
from engines.audio8_tts.audio8_tts_engine import (
Audio8TTSEngine,
)
engine = Audio8TTSEngine(
model_name=(request.get("model_name") or "Audio8-TTS-Preview-0.6b"),
model_dir=payload.get("model_path"),
device=request.get("device") or "auto",
dtype=payload.get("dtype", "auto"),
)
engine._ensure_model_loaded()
_emit(
protocol_out,
RuntimeJobResponse(
ok=True,
result={
"model_name": engine.model_name,
"sample_rate": engine.SAMPLE_RATE,
"transformers_version": (transformers.__version__),
},
request_id=request_id,
),
)
continue
if engine is None:
raise RuntimeError(
"Audio8 TTS worker received generation before initialization"
)
if action != "generate":
raise RuntimeError(f"Unsupported Audio8 TTS worker action '{action}'")
reference_audio = _load_reference_audio(payload.get("reference_audio"))
audio = engine.generate(
text=payload["text"],
reference_audio=reference_audio,
reference_text=payload.get("reference_text"),
max_new_tokens=int(payload.get("max_new_tokens", 1024)),
retry_max_new_tokens=int(payload.get("retry_max_new_tokens", 2000)),
temperature=float(payload.get("temperature", 0.8)),
top_p=float(payload.get("top_p", 0.95)),
top_k=int(payload.get("top_k", 50)),
do_sample=bool(payload.get("do_sample", True)),
seed=int(payload.get("seed", 42)),
)
audio = torch.as_tensor(audio).detach().float().cpu()
if audio.ndim == 1:
audio = audio.unsqueeze(0)
if audio.ndim != 2 or audio.shape[0] != 1:
raise RuntimeError(
f"Audio8 TTS generated invalid audio shape {tuple(audio.shape)}"
)
output_path = Path(payload["output_path"])
torch.save(
{
"audio": audio.contiguous(),
"sample_rate": engine.SAMPLE_RATE,
},
output_path,
)
_emit(
protocol_out,
RuntimeJobResponse(
ok=True,
result={
"output_path": str(output_path),
"sample_rate": engine.SAMPLE_RATE,
},
request_id=request_id,
),
)
except Exception as exc:
request_id = (
request.get("request_id") if isinstance(request, dict) else None
)
_emit(
protocol_out,
RuntimeJobResponse(
ok=False,
error=f"{exc}\n{traceback.format_exc()}",
request_id=request_id,
),
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
+8 -5
View File
@@ -115,12 +115,13 @@ PARAMETER_ENGINES = {
'seed': {
'chatterbox', 'chatterbox_official_23lang', 'f5tts', 'higgs_audio',
'higgs_audio_v3', 'vibevoice', 'index_tts', 'step_audio_editx', 'cosyvoice', 'qwen3_tts',
'dots_tts', 'fish_audio_s2', 'omnivoice',
'dots_tts', 'audio8_tts', 'fish_audio_s2', 'omnivoice',
'echo_tts', 'moss_tts', 'moss_soundeffect_v2', 'dramabox'
},
'temperature': {
'chatterbox', 'chatterbox_official_23lang', 'f5tts', 'higgs_audio',
'higgs_audio_v3', 'vibevoice', 'index_tts', 'step_audio_editx', 'qwen3_tts', 'moss_tts', 'fish_audio_s2'
'higgs_audio_v3', 'vibevoice', 'index_tts', 'step_audio_editx', 'qwen3_tts',
'moss_tts', 'fish_audio_s2', 'audio8_tts'
},
'cfg': {
'f5tts', 'vibevoice', 'index_tts', 'chatterbox', 'chatterbox_official_23lang',
@@ -214,10 +215,12 @@ PARAMETER_ENGINES = {
'f5tts', 'cosyvoice', 'omnivoice'
},
'top_p': {
'higgs_audio', 'higgs_audio_v3', 'vibevoice', 'index_tts', 'qwen3_tts', 'moss_tts', 'fish_audio_s2'
'higgs_audio', 'higgs_audio_v3', 'vibevoice', 'index_tts', 'qwen3_tts',
'moss_tts', 'fish_audio_s2', 'audio8_tts'
},
'top_k': {
'higgs_audio', 'higgs_audio_v3', 'index_tts', 'qwen3_tts', 'moss_tts'
'higgs_audio', 'higgs_audio_v3', 'index_tts', 'qwen3_tts', 'moss_tts',
'audio8_tts'
},
'audio_temperature': {
'moss_tts'
@@ -238,7 +241,7 @@ PARAMETER_ENGINES = {
'moss_tts'
},
'max_new_tokens': {
'higgs_audio_v3', 'moss_tts', 'fish_audio_s2'
'higgs_audio_v3', 'moss_tts', 'fish_audio_s2', 'audio8_tts'
},
'max_generate_length': {
'dots_tts'