Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
abca6197d8 | ||
|
|
701d2c97f8 | ||
|
|
965b314fac | ||
|
|
35cbcaca71 | ||
|
|
9275766f9b |
@@ -44,6 +44,7 @@ The project code is MIT. Model weights carry their own licenses:
|
||||
Echo-TTS CC-BY-NC-SA-4.0 No
|
||||
Fish Audio S2 Pro Fish Audio Research License No
|
||||
Dots TTS Apache-2.0 Yes
|
||||
TADA MIT code / Llama 3.2 weights Conditional
|
||||
DramaBox LTX-2 Community License Conditional
|
||||
OmniVoice Apache-2.0 Yes
|
||||
MOSS-TTS Apache-2.0 Yes
|
||||
|
||||
+8
-5
@@ -17,7 +17,7 @@
|
||||
**Key architectural rules:**
|
||||
- Chunking happens in the **processor**, not the adapter (`generate_single()` on adapter = raw single call)
|
||||
- Runtime routing happens through `ModelLoadConfig.runtime_mode` + `runtime_profile`, not ad-hoc subprocess calls
|
||||
- Shared runtime workers are currently used for fragile engine families such as VibeVoice, Qwen3-TTS / ASR, Granite forced alignment, and Higgs Audio 2. Engines that support the modern stack run natively in the main Transformers 5 environment.
|
||||
- Shared runtime workers are currently used for fragile engine families such as VibeVoice, Qwen3-TTS / ASR, Granite forced alignment, Higgs Audio 2, and TADA. Engines that support the modern stack run natively in the main Transformers 5 environment.
|
||||
- YAML (`docs/Dev reports/tts_audio_suite_engines.yaml`) is source of truth for engine doc tables → run `python3 scripts/generate_engine_tables.py --readme` to regenerate
|
||||
- Auxiliary YAML (`docs/Dev reports/tts_audio_suite_aux_models.yaml`) is source of truth for helper/post-process model docs → run `python3 scripts/generate_aux_model_docs.py`
|
||||
- All models download to `ComfyUI/models/TTS/<model-name>/`
|
||||
@@ -25,7 +25,7 @@
|
||||
|
||||
## Engines
|
||||
|
||||
19 engines follow the pattern above:
|
||||
20 engines follow the pattern above:
|
||||
|
||||
| Engine | Adapter | Processor | SRT Processor | Engine Node |
|
||||
|--------|---------|-----------|---------------|-------------|
|
||||
@@ -44,13 +44,14 @@
|
||||
| Echo-TTS | `echo_tts_adapter.py` | `nodes/echo_tts/echo_tts_processor.py` | `echo_tts_srt_processor.py` | `echo_tts_engine_node.py` |
|
||||
| Fish Audio S2 Pro | `fish_audio_s2_adapter.py` | `nodes/fish_audio_s2/fish_audio_s2_processor.py` | `fish_audio_s2_srt_processor.py` | `fish_audio_s2_engine_node.py` |
|
||||
| Dots TTS | `dots_tts_adapter.py` | `nodes/dots_tts/dots_tts_processor.py` | `dots_tts_srt_processor.py` | `dots_tts_engine_node.py` |
|
||||
| TADA | `tada_adapter.py` | `nodes/tada/tada_processor.py` | `tada_srt_processor.py` | `tada_engine_node.py` |
|
||||
| DramaBox | `dramabox_adapter.py` | `nodes/dramabox/dramabox_processor.py` | `dramabox_srt_processor.py` | `dramabox_engine_node.py` |
|
||||
| OmniVoice | `omnivoice_adapter.py` | `nodes/omnivoice/omnivoice_processor.py` | `omnivoice_srt_processor.py` | `omnivoice_engine_node.py` |
|
||||
| MOSS-SoundEffect v2 | `moss_soundeffect_v2_adapter.py` | — | — | `moss_soundeffect_v2_engine_node.py` |
|
||||
| RVC | — | `engines/rvc/` | — | `rvc_engine_node.py` |
|
||||
|
||||
**Engine implementations live in:**
|
||||
- `engines/chatterbox/`, `engines/chatterbox_official_23lang/`, `engines/f5tts/`, `engines/higgs_audio/`, `engines/higgs_audio_v3/`, `engines/vibevoice_engine/`, `engines/step_audio_editx/`, `engines/cosyvoice/`, `engines/qwen3_tts/`, `engines/qwen3_asr/`, `engines/moss_tts/`, `engines/moss_soundeffect_v2/`, `engines/granite_asr/`, `engines/echo_tts/`, `engines/fish_audio_s2/`, `engines/dots_tts/`, `engines/dramabox/`, `engines/omnivoice/`, `engines/rvc/`
|
||||
- `engines/chatterbox/`, `engines/chatterbox_official_23lang/`, `engines/f5tts/`, `engines/higgs_audio/`, `engines/higgs_audio_v3/`, `engines/vibevoice_engine/`, `engines/step_audio_editx/`, `engines/cosyvoice/`, `engines/qwen3_tts/`, `engines/qwen3_asr/`, `engines/moss_tts/`, `engines/moss_soundeffect_v2/`, `engines/granite_asr/`, `engines/echo_tts/`, `engines/fish_audio_s2/`, `engines/dots_tts/`, `engines/tada/`, `engines/dramabox/`, `engines/omnivoice/`, `engines/rvc/`
|
||||
|
||||
## Documentation Files
|
||||
|
||||
@@ -67,6 +68,7 @@
|
||||
- `OMNIVOICE_TAGS_GUIDE.md` - OmniVoice native non-verbal tags and pronunciation overrides
|
||||
- `MOSS_TTS_PROMPT_FIELDS_GUIDE.md` - Official MOSS whole-segment prompt fields and inline `<>` translation limits
|
||||
- `DRAMABOX_PROMPTING_GUIDE.md` - DramaBox expressive scene prompts, voice references, controls, hardware, and license
|
||||
- `TADA_ENGINE_GUIDE.md` - TADA models, gated tokenizer setup, exact-transcript voice cloning, native controls, limits, and licensing
|
||||
- `COSYVOICE3_TAGS_GUIDE.md` - CosyVoice3 native paralinguistic tags
|
||||
- `CHATTERBOX_V2_SPECIAL_TOKENS.md` - ChatterBox v2 emotion tokens
|
||||
- `IndexTTS2_Emotion_Control_Guide.md` - IndexTTS-2 vector, text, audio, and blended emotion controls
|
||||
@@ -145,13 +147,14 @@
|
||||
- `launcher.py` - runtime bootstrap, venv creation, Windows toolchain env setup
|
||||
- `session.py`, `protocol.py` - JSONL worker transport and message protocol
|
||||
- `bootstrap.py` - shared runtime bootstrap helpers
|
||||
- `vibevoice_proxy.py`, `qwen3_tts_proxy.py`, `qwen3_asr_proxy.py`, `higgs_audio_proxy.py` - parent-process proxies
|
||||
- `workers/` - worker subprocess entrypoints for VibeVoice, Qwen3-TTS, Qwen3-ASR/aligner, Higgs Audio
|
||||
- `vibevoice_proxy.py`, `qwen3_tts_proxy.py`, `qwen3_asr_proxy.py`, `higgs_audio_proxy.py`, `tada_proxy.py` - parent-process proxies
|
||||
- `workers/` - worker subprocess entrypoints for VibeVoice, Qwen3-TTS, Qwen3-ASR/aligner, Higgs Audio, and TADA
|
||||
- Current shared legacy T4 runtime profile is reused by:
|
||||
- VibeVoice / Kugel
|
||||
- Qwen3-TTS
|
||||
- Qwen3-ASR and Granite's optional Qwen forced aligner
|
||||
- Higgs Audio 2
|
||||
- TADA
|
||||
|
||||
### Audio (`utils/audio/`)
|
||||
- `processing.py` - Tensor manipulation, normalization, format conversion
|
||||
|
||||
@@ -23,7 +23,7 @@ Subtitle workflows are still a core focus: the suite can transcribe to SRT, rebu
|
||||
|
||||
<!-- ENGINE_COMPARISON_START -->
|
||||
|
||||
## Quick Engine Comparison — 19 Engines
|
||||
## Quick Engine Comparison — 20 Engines
|
||||
|
||||
| Engine | Languages | Size | Key Features |
|
||||
|--------|-----------|------|--------------|
|
||||
@@ -41,6 +41,7 @@ Subtitle workflows are still a core focus: the suite can transcribe to SRT, rebu
|
||||
| **Echo-TTS** | 🇺🇸 | ~5.3GB + ~1.8GB | Diffusion-based (~30s best), Force Speaker KV (speaker drift control) |
|
||||
| **Fish Audio S2 Pro** | 🌐 80+ languages | ~10.3GB / ~8.0GB | Free-form sub-word emotion/prosody tags, Native multi-speaker and multi-turn dialogue with dynamic speaker references |
|
||||
| **Dots TTS** | 🇺🇸🇨🇳🇩🇪🇪🇸🇫🇷🇮🇹 +13 | ~6GB | Official auto language detect / language control, SOAR and MeanFlow distilled variants |
|
||||
| **TADA** | 🇺🇸🇨🇳🇩🇪🇪🇸🇫🇷🇮🇹 +4 | ~6.1GB / ~11GB with selected codec assets | Voice cloning, Reference-conditioned expression transfer |
|
||||
| **DramaBox** | 🇺🇸 | ~16.4GB | Expressive scene prompting and stage directions, Native and SRT-aware duration targeting |
|
||||
| **OmniVoice** | 🌐 600+ languages | ~3.7GB | Inline non-verbal tags and pronunciation overrides, Reference-free voice design |
|
||||
| **MOSS-TTS** | 🇺🇸🇨🇳🇩🇪🇪🇸🇫🇷🇮🇹 +18 | ~8.5GB tokenizer + ~6.1GB/17GB/18GB model | Reference-free voice design with MOSS-VoiceGenerator, Native 1-5 speaker TTSD dialogue |
|
||||
@@ -1527,6 +1528,7 @@ For offline/manual setup:
|
||||
| Granite ASR | `ComfyUI/models/TTS/granite_asr/` | ✅ | Granite ASR models; plus adds native diarization/timestamps, optional Qwen forced aligner reused lazily for timestamps/SRT fallback |
|
||||
| Echo-TTS | `ComfyUI/models/TTS/echo-tts-base/` | ✅ | ~7.1GB total (base + dac); CC-BY-NC-SA |
|
||||
| Dots TTS | `ComfyUI/models/TTS/dots_tts/` | ✅ | Official base / soar / mf checkpoints with tokenizer, vocoder, speaker encoder |
|
||||
| TADA | `ComfyUI/models/TTS/tada/` | ✅ | Selected 1B or 3B-ML model plus language-specific codec assets and an exact hash-verified ungated Llama tokenizer redistribution |
|
||||
| DramaBox | `ComfyUI/models/TTS/dramabox/DramaBox/` | ✅ | ~16.4GB download; fast mode roughly 24GB VRAM; experimental FP8 peaks on RTX 4090: staged ~15.1GB allocated, sequential ~11.7GB allocated / ~12.4GB reserved; conditional LTX-2 Community License |
|
||||
| Fish Audio S2 Pro | `ComfyUI/models/TTS/fish_audio_s2_pro/` | ✅ | Official BF16 or optional community FP8 checkpoint; the official checkpoint can be quantized on load with BNB INT8/NF4; main T5 environment with process teardown for Clear VRAM; Fish Audio Research License |
|
||||
| OmniVoice | `ComfyUI/models/TTS/omnivoice/` | ✅ | Official OmniVoice model. Voice cloning in this suite requires explicit reference text. |
|
||||
|
||||
@@ -1339,6 +1339,110 @@ engines:
|
||||
speed_performance: { supported: "partial", notes: "Moderate; mf variant is faster" }
|
||||
reference_free_tts: { supported: true, notes: "(default speaker)" }
|
||||
|
||||
- id: tada
|
||||
name: TADA
|
||||
models: "TADA 1B (English), TADA 3B-ML"
|
||||
size: "~6.1GB / ~11GB with selected codec assets"
|
||||
license: "MIT code / Llama 3.2 weights"
|
||||
commercial: conditional
|
||||
|
||||
capabilities:
|
||||
tts: true
|
||||
srt: true
|
||||
vc: false
|
||||
asr: false
|
||||
training: false
|
||||
|
||||
special_features:
|
||||
- "Zero-shot voice cloning"
|
||||
- "Reference-conditioned expression transfer"
|
||||
|
||||
readme_key_features:
|
||||
- "Voice cloning"
|
||||
- "Reference-conditioned expression transfer"
|
||||
|
||||
runtime_isolation:
|
||||
default_mode: "shared_runtime"
|
||||
supported_modes:
|
||||
- "shared_runtime"
|
||||
status: "implemented"
|
||||
runtime_profile: "vibevoice_transformers4_shared"
|
||||
notes: "The supported path uses the shared Transformers 4 runtime with hume-tada 0.1.9. TADA is incompatible with the suite's main Transformers 5 environment."
|
||||
|
||||
model_sources:
|
||||
- component: "TADA 1B"
|
||||
source_name: "HumeAI/tada-1b"
|
||||
source_url: "https://huggingface.co/HumeAI/tada-1b"
|
||||
size: "~3.9GB"
|
||||
auto_download: true
|
||||
notes: "English-only model; downloaded only when selected"
|
||||
- component: "TADA 3B-ML"
|
||||
source_name: "HumeAI/tada-3b-ml"
|
||||
source_url: "https://huggingface.co/HumeAI/tada-3b-ml"
|
||||
size: "~8.9GB"
|
||||
auto_download: true
|
||||
notes: "Multilingual model; downloaded only when selected"
|
||||
- component: "TADA codec"
|
||||
source_name: "HumeAI/tada-codec"
|
||||
source_url: "https://huggingface.co/HumeAI/tada-codec"
|
||||
size: "~2.15GB per selected language"
|
||||
auto_download: true
|
||||
notes: "Downloads shared encoder/decoder plus only the aligner required by the selected language"
|
||||
- component: "Llama tokenizer"
|
||||
source_name: "onnx-community/Llama-3.2-1B"
|
||||
source_url: "https://huggingface.co/onnx-community/Llama-3.2-1B"
|
||||
size: "Tokenizer files only"
|
||||
auto_download: true
|
||||
notes: "Ungated redistribution; runtime files are verified against Meta's official Git blob hashes and remain under the Llama 3.2 Community License"
|
||||
- component: "Wav2Vec aligner configuration"
|
||||
source_name: "facebook/wav2vec2-large"
|
||||
source_url: "https://huggingface.co/facebook/wav2vec2-large"
|
||||
size: "Configuration file only"
|
||||
auto_download: true
|
||||
notes: "Downloads config.json only; no Wav2Vec model weights"
|
||||
|
||||
languages:
|
||||
en: { supported: true, flag: "🇺🇸", notes: "(1B and 3B-ML)" }
|
||||
zh: { supported: true, flag: "🇨🇳", notes: "(3B-ML; backend code `ch`)" }
|
||||
de: { supported: true, flag: "🇩🇪", notes: "(3B-ML)" }
|
||||
es: { supported: true, flag: "🇪🇸", notes: "(3B-ML)" }
|
||||
fr: { supported: true, flag: "🇫🇷", notes: "(3B-ML)" }
|
||||
it: { supported: true, flag: "🇮🇹", notes: "(3B-ML)" }
|
||||
ja: { supported: true, flag: "🇯🇵", notes: "(3B-ML)" }
|
||||
ko: { supported: false, flag: "🇰🇷", notes: "" }
|
||||
ru: { supported: false, flag: "🇷🇺", notes: "" }
|
||||
pt: { supported: true, flag: "🇵🇹", notes: "(3B-ML)" }
|
||||
pl: { supported: true, flag: "🇵🇱", notes: "(3B-ML)" }
|
||||
hi: { supported: false, flag: "🇮🇳", notes: "" }
|
||||
ar: { supported: true, flag: "🇦🇪", notes: "(3B-ML)" }
|
||||
tr: { supported: false, flag: "🇹🇷", notes: "" }
|
||||
th: { supported: false, flag: "🇹🇭", notes: "" }
|
||||
no: { supported: false, flag: "🇳🇴", notes: "" }
|
||||
vi: { supported: false, flag: "🇻🇳", notes: "" }
|
||||
hy: { supported: false, flag: "🇦🇲", notes: "" }
|
||||
ka: { supported: false, flag: "🇬🇪", notes: "" }
|
||||
da: { supported: false, flag: "🇩🇰", notes: "" }
|
||||
fi: { supported: false, flag: "🇫🇮", notes: "" }
|
||||
el: { supported: false, flag: "🇬🇷", notes: "" }
|
||||
he: { supported: false, flag: "🇮🇱", notes: "" }
|
||||
ms: { supported: false, flag: "🇲🇾", notes: "" }
|
||||
nl: { supported: false, flag: "🇳🇱", notes: "" }
|
||||
sv: { supported: false, flag: "🇸🇪", notes: "" }
|
||||
sw: { supported: false, flag: "🇰🇪", notes: "" }
|
||||
|
||||
features:
|
||||
voice_cloning: { supported: true, notes: "Reference audio plus exact transcript" }
|
||||
reference_transcript: { requirement: required }
|
||||
native_multi_speaker: { supported: false, notes: "" }
|
||||
voice_conversion: { supported: false, notes: "" }
|
||||
asr_transcribe: { supported: false, notes: "" }
|
||||
emotion_control: { supported: false, notes: "No direct emotion control; expression may transfer from the reference audio" }
|
||||
native_long_form: { supported: false, notes: "Suite text chunking is used" }
|
||||
community_finetunes: { supported: false, notes: "No suite integration" }
|
||||
vram_efficient: { supported: "partial", notes: "1B is the lighter option; suite VRAM measurements are pending" }
|
||||
speed_performance: { supported: "partial", notes: "No suite benchmark; no streaming or batch inference" }
|
||||
reference_free_tts: { supported: false, notes: "This integration requires a reference voice and exact transcript" }
|
||||
|
||||
- id: dramabox
|
||||
name: DramaBox
|
||||
models: "DramaBox 3.3B"
|
||||
@@ -1904,13 +2008,15 @@ table_notes:
|
||||
Conditional means the transcript is required only for the specific mode:
|
||||
CosyVoice3 zero-shot, Qwen3-TTS full Base cloning, or MOSS-TTSD cloned-speaker
|
||||
dialogue. Higgs Audio 2, Higgs Audio v3, and Dots TTS accept matching text
|
||||
when provided but do not require it.
|
||||
when provided but do not require it. TADA always requires an exact transcript
|
||||
for its reference audio in this integration.
|
||||
language_support:
|
||||
- "**CosyVoice3 Chinese**: Includes 18+ dialects (Cantonese, Sichuan, Dongbei, Shanghai, etc.)"
|
||||
- "**Higgs Audio 2**: Trained on EN, ZH (Mandarin), KO, DE, ES (English majority) - 10M hours AudioVerse dataset"
|
||||
- "**Higgs Audio v3**: Official model card lists 102 languages total: 85 polished-production languages and 17 usable but less polished languages."
|
||||
- "**IndexTTS-2**: Trained on 55K+ hours - ZH, EN, JA primary."
|
||||
- "**MOSS-TTS**: Official list includes ZH, EN, DE, ES, FR, JA, IT, HU, KO, RU, FA, AR, PL, PT, CS, DA, SV, EL, TR; HU/FA/CS are not separate columns in this matrix."
|
||||
- "**TADA**: TADA 1B is English-only. TADA 3B-ML supports EN, AR, ZH, DE, ES, FR, IT, JA, PL, and PT; the upstream backend calls Chinese `ch`."
|
||||
- "**OmniVoice**: Official model supports 600+ languages; this matrix only shows the suite's comparison subset."
|
||||
- "**RVC**: Language-agnostic voice conversion/post-processing; it is marked supported for every language row."
|
||||
- "Qwen3-TTS: Supports pt-BR only with instruction (voice design or harcoded voices). Base can't do instructions, so it will always ouput pt-PT."
|
||||
@@ -1980,6 +2086,10 @@ readme_model_download_table:
|
||||
primary_model_path: "ComfyUI/models/TTS/dots_tts/"
|
||||
auto_download: "✅"
|
||||
notes: "Official base / soar / mf checkpoints with tokenizer, vocoder, speaker encoder"
|
||||
- engine: "TADA"
|
||||
primary_model_path: "ComfyUI/models/TTS/tada/"
|
||||
auto_download: "✅"
|
||||
notes: "Selected 1B or 3B-ML model plus language-specific codec assets and an exact hash-verified ungated Llama tokenizer redistribution"
|
||||
- engine: "DramaBox"
|
||||
primary_model_path: "ComfyUI/models/TTS/dramabox/DramaBox/"
|
||||
auto_download: "✅"
|
||||
@@ -2423,6 +2533,28 @@ model_layouts_markdown: |
|
||||
- Native sample rate is 48kHz.
|
||||
- Main-environment support works on Transformers 5; on Windows, `normalize_text` falls back to no-op if `WeTextProcessing` is unavailable.
|
||||
|
||||
## TADA
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/tada/
|
||||
├── TADA-1B/ # downloaded only if selected
|
||||
├── TADA-3B-ML/ # downloaded only if selected
|
||||
├── tada-codec/
|
||||
│ ├── encoder/
|
||||
│ ├── decoder/
|
||||
│ ├── aligner[-<language>]/ # selected language only
|
||||
│ └── wav2vec2-large/
|
||||
└── llama-3.2-1b-tokenizer/ # tokenizer files only
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- TADA runs in the shared Transformers 4 runtime with `hume-tada==0.1.9`.
|
||||
- The selected 1B or 3B-ML model, shared codec components, and selected language aligner download on demand.
|
||||
- The Llama tokenizer downloads from an ungated redistribution and is verified against Meta's official Git blob hashes. It remains under the Llama 3.2 Community License.
|
||||
- Reference audio and its exact transcript are required by this integration.
|
||||
- Native output is mono 24kHz.
|
||||
|
||||
## OmniVoice
|
||||
|
||||
```text
|
||||
|
||||
@@ -18,6 +18,7 @@
|
||||
| **Echo-TTS** | Main | echo-tts-base + fish-s1-dac-min | ~5.3GB + ~1.8GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | CC-BY-NC-SA-4.0 | Diffusion-based (~30s best), Force Speaker KV (speaker drift control) | 1 |
|
||||
| **Fish Audio S2 Pro** | Main | S2 Pro 4B / FP8 | ~10.3GB / ~8.0GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Fish Audio Research License | Free-form sub-word emotion/prosody tags, Native multi-speaker and multi-turn dialogue with dynamic speaker references, Zero-shot voice cloning, Optional per-segment custom character switching, Configurable 4K-32K native context with reduced KV-cache VRAM, Optional community FP8 weight-only checkpoint with BF16 activations, Optional on-the-fly BitsAndBytes INT8/NF4 for the official checkpoint | 80+ languages |
|
||||
| **Dots TTS** | Main | dots.tts-base, dots.tts-soar, dots.tts-mf | ~6GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 | Official auto language detect / language control, SOAR and MeanFlow distilled variants | 19 |
|
||||
| **TADA** | Shared | TADA 1B (English), TADA 3B-ML | ~6.1GB / ~11GB with selected codec assets | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | MIT code / Llama 3.2 weights | Zero-shot voice cloning, Reference-conditioned expression transfer | 10 |
|
||||
| **DramaBox** | Main | DramaBox 3.3B | ~16.4GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | LTX-2 Community License | Expressive scene prompting and stage directions, Native and SRT-aware duration targeting, Official duration-aware long-form chunking with scene-prefix preservation, Optional 10-second zero-shot voice reference, CFG negative prompt with per-segment switching, Explicit generation/reference durations, CFG rescale control, and optional Perth watermark, Experimental staged and sequential strategies for lowering peak VRAM; official LTX FP8-cast storage, Optional official torch.compile path | 1 |
|
||||
| **OmniVoice** | Main | OmniVoice | ~3.7GB | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | Apache-2.0 | Inline non-verbal tags and pronunciation overrides, Reference-free voice design, 600+ language support, Upstream long-form chunk orchestration | 600+ |
|
||||
| **MOSS-TTS** | Main | Local 1.7B, Delay 8B v1.5/1.0, VoiceGenerator 1.7B, SoundEffect 8B v1, TTSD 8B | ~8.5GB tokenizer + ~6.1GB/17GB/18GB model | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | Apache-2.0 | Reference-free voice design with MOSS-VoiceGenerator, Native 1-5 speaker TTSD dialogue, 31-language generation with MOSS-TTS-v1.5, Prompt-only sound-effect generation with MOSS-SoundEffect v1, Long-form generation (TTSD/Delay), Duration token hint, Local/Delay/TTSD variants, Initial integrated LoRA training workflow (Delay 8B) | 24 |
|
||||
|
||||
+18
-18
@@ -2,22 +2,22 @@
|
||||
|
||||
## Feature Comparison Matrix
|
||||
|
||||
| Feature | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | DramaBox | OmniVoice | MOSS-TTS | MOSS-SoundEffect v2 | RVC |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| **TTS** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| **SRT** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| **Voice Conversion** | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| **ASR (Transcribe)** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Sound Effects** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ |
|
||||
| **Training** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
|
||||
| **Voice Cloning** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (Base model) | ❌ | ✅ | ✅ | ✅ Reference audio plus exact transcript | ✅ | ✅ Optional reference audio; configurable 3-30 second window from the beginning of the source, default 10 seconds | ✅ | ✅ | ❌ | ⚠️ (needs training) |
|
||||
| **Reference Transcript†** | **Required** | Not used | Not used | Not used | Optional | Optional | Not used | Conditional | Conditional | N/A | **Required** | Not used | **Required** | Optional | Not used | **Required** | Conditional | Not used | N/A |
|
||||
| **Native Multi-Speaker** | ❌ | ❌ | ❌ | ✅ (Base only, Kugel uses fallback) | ✅ | ❌ | ❌ | ❌ | ❌ | ⚠️ (Plus variant speaker attribution / diarization) | ❌ | ❌ | ✅ Native dialogue is default; optional custom mode generates each [Character] segment independently as local speaker 0 | ❌ | ❌ | ❌ | ✅ (TTSD v1.0; 1-5 speakers) | ❌ | ❌ |
|
||||
| **Emotion Control** | ❌ | ❌ | ⚠️ (v2 tags - doesn't work) | ❌ | ⚠️ (via prompt) | ✅ (native inline tags) | ✅ (8 emotions) | ⚠️ (via instruct) | ⚠️ (via instruct) | ❌ | ✅ (14 emotions) | ❌ | ✅ Free-form inline natural-language tags | ❌ | ✅ Natural-language scene prompt and stage directions | ⚠️ (voice-design instruct + inline non-verbal tags) | ✅ (MOSS-VoiceGenerator instruction-conditioned voice design) | ❌ | ❌ |
|
||||
| **Native Long-form** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Configurable 4K-32K native context; suite text chunking is bypassed | ❌ | ✅ Official duration-aware quote-group chunking; ~37s target / 45s cap | ✅ (uses upstream audio_chunk_duration / audio_chunk_threshold orchestration; bypasses suite char-based chunk splitting) | ✅ (TTSD/Delay long-form; use chunk orchestration for very long inputs) | ❌ | N/A |
|
||||
| **Community Finetunes** | ✅ | ✅ | ✅ | ✅ KugelAudio, Hindi | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (LoRA adapter inference supported; initial integrated LoRA training support added for MOSS-TTS Delay 8B) | ❌ | ✅ |
|
||||
| **VRAM Efficient** | ✅ | ✅ | ✅ | ⚠️ (5-18GB) | ⚠️ (9GB) | ⚠️ (~8-10GB) | ⚠️ (9-12GB) | ✅ (5.4GB) | ✅ (3-6GB) | ✅ (~4.6GB) | ⚠️ (7GB) | ⚠️ (~7GB total) | ⚠️ 8K context measured at ~15.2GB BF16, ~11.2GB FP8, ~11.2GB BNB INT8, or ~8.9GB BNB NF4; BF16 codec and activations; BNB is a load-time option for the official checkpoint | ⚠️ (main env works; 2B-class model is not lightweight) | ✅ Fast mode is ~24GB; experimental FP8 peaks on RTX 4090: staged ~15.1GB allocated, sequential ~11.7GB allocated / ~12.4GB reserved | ⚠️ (main env works; 2B-class model is not lightweight) | ⚠️ (Local 1.7B smaller; tokenizer is large) | ⚠️ (runs in the main ComfyUI environment but remains GPU-heavy) | ✅ |
|
||||
| **Speed/Performance** | ✅ Very Fast | ✅ Fast | ✅ Fast | ⚠️ | ⚠️ | ⚠️ CUDA recommended | ⚠️ | ✅ Fast | ⚠️ | ⚠️ Moderate | ⚠️ | ✅ Fast (diffusion, realtime-capable) | ✅ Main-environment subprocess with reliable teardown; local compile measurements: ~40 it/s BF16, ~11.8 it/s NF4 at ~8.9GB VRAM, and ~3.7 it/s INT8 at ~11.2GB VRAM; quality comparison pending | ⚠️ Moderate; mf variant is faster | ⚠️ Fast favors repeated generation; staged reloads temporary components; sequential also transfers the transformer; optional block-level torch.compile accelerates repeat denoising | ✅ Fast; upstream reports sub-realtime RTF | ✅ Fast with CUDA/FlashAttention | ⚠️ (100 diffusion steps by default) | ✅ Fast |
|
||||
| **No Narrator Required** | ❌ | ✅ (default speaker) | ✅ (default speaker) | ✅ (zero-shot / default speaker) | ✅ (basic TTS if no narrator/reference is provided) | ✅ (zero-shot) | ❌ | ✅ (cross-lingual or instruct mode) | ✅ (Base default voice or CustomVoice presets) | N/A | ❌ | ❌ | ✅ Reference audio is optional | ✅ (default speaker) | ✅ Voice reference is optional | ✅ (native default voice; instruct also works without narrator) | ✅ (direct TTS and prompt-only generation) | ❌ | N/A |
|
||||
| Feature | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | TADA | DramaBox | OmniVoice | MOSS-TTS | MOSS-SoundEffect v2 | RVC |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| **TTS** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| **SRT** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| **Voice Conversion** | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| **ASR (Transcribe)** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Sound Effects** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ |
|
||||
| **Training** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
|
||||
| **Voice Cloning** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (Base model) | ❌ | ✅ | ✅ | ✅ Reference audio plus exact transcript | ✅ | ✅ Reference audio plus exact transcript | ✅ Optional reference audio; configurable 3-30 second window from the beginning of the source, default 10 seconds | ✅ | ✅ | ❌ | ⚠️ (needs training) |
|
||||
| **Reference Transcript†** | **Required** | Not used | Not used | Not used | Optional | Optional | Not used | Conditional | Conditional | N/A | **Required** | Not used | **Required** | Optional | **Required** | Not used | **Required** | Conditional | Not used | N/A |
|
||||
| **Native Multi-Speaker** | ❌ | ❌ | ❌ | ✅ (Base only, Kugel uses fallback) | ✅ | ❌ | ❌ | ❌ | ❌ | ⚠️ (Plus variant speaker attribution / diarization) | ❌ | ❌ | ✅ Native dialogue is default; optional custom mode generates each [Character] segment independently as local speaker 0 | ❌ | ❌ | ❌ | ❌ | ✅ (TTSD v1.0; 1-5 speakers) | ❌ | ❌ |
|
||||
| **Emotion Control** | ❌ | ❌ | ⚠️ (v2 tags - doesn't work) | ❌ | ⚠️ (via prompt) | ✅ (native inline tags) | ✅ (8 emotions) | ⚠️ (via instruct) | ⚠️ (via instruct) | ❌ | ✅ (14 emotions) | ❌ | ✅ Free-form inline natural-language tags | ❌ | ❌ | ✅ Natural-language scene prompt and stage directions | ⚠️ (voice-design instruct + inline non-verbal tags) | ✅ (MOSS-VoiceGenerator instruction-conditioned voice design) | ❌ | ❌ |
|
||||
| **Native Long-form** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Configurable 4K-32K native context; suite text chunking is bypassed | ❌ | ❌ | ✅ Official duration-aware quote-group chunking; ~37s target / 45s cap | ✅ (uses upstream audio_chunk_duration / audio_chunk_threshold orchestration; bypasses suite char-based chunk splitting) | ✅ (TTSD/Delay long-form; use chunk orchestration for very long inputs) | ❌ | N/A |
|
||||
| **Community Finetunes** | ✅ | ✅ | ✅ | ✅ KugelAudio, Hindi | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (LoRA adapter inference supported; initial integrated LoRA training support added for MOSS-TTS Delay 8B) | ❌ | ✅ |
|
||||
| **VRAM Efficient** | ✅ | ✅ | ✅ | ⚠️ (5-18GB) | ⚠️ (9GB) | ⚠️ (~8-10GB) | ⚠️ (9-12GB) | ✅ (5.4GB) | ✅ (3-6GB) | ✅ (~4.6GB) | ⚠️ (7GB) | ⚠️ (~7GB total) | ⚠️ 8K context measured at ~15.2GB BF16, ~11.2GB FP8, ~11.2GB BNB INT8, or ~8.9GB BNB NF4; BF16 codec and activations; BNB is a load-time option for the official checkpoint | ⚠️ (main env works; 2B-class model is not lightweight) | ⚠️ 1B is the lighter option; suite VRAM measurements are pending | ✅ Fast mode is ~24GB; experimental FP8 peaks on RTX 4090: staged ~15.1GB allocated, sequential ~11.7GB allocated / ~12.4GB reserved | ⚠️ (main env works; 2B-class model is not lightweight) | ⚠️ (Local 1.7B smaller; tokenizer is large) | ⚠️ (runs in the main ComfyUI environment but remains GPU-heavy) | ✅ |
|
||||
| **Speed/Performance** | ✅ Very Fast | ✅ Fast | ✅ Fast | ⚠️ | ⚠️ | ⚠️ CUDA recommended | ⚠️ | ✅ Fast | ⚠️ | ⚠️ Moderate | ⚠️ | ✅ Fast (diffusion, realtime-capable) | ✅ Main-environment subprocess with reliable teardown; local compile measurements: ~40 it/s BF16, ~11.8 it/s NF4 at ~8.9GB VRAM, and ~3.7 it/s INT8 at ~11.2GB VRAM; quality comparison pending | ⚠️ Moderate; mf variant is faster | ⚠️ No suite benchmark; no streaming or batch inference | ⚠️ Fast favors repeated generation; staged reloads temporary components; sequential also transfers the transformer; optional block-level torch.compile accelerates repeat denoising | ✅ Fast; upstream reports sub-realtime RTF | ✅ Fast with CUDA/FlashAttention | ⚠️ (100 diffusion steps by default) | ✅ Fast |
|
||||
| **No Narrator Required** | ❌ | ✅ (default speaker) | ✅ (default speaker) | ✅ (zero-shot / default speaker) | ✅ (basic TTS if no narrator/reference is provided) | ✅ (zero-shot) | ❌ | ✅ (cross-lingual or instruct mode) | ✅ (Base default voice or CustomVoice presets) | N/A | ❌ | ❌ | ✅ Reference audio is optional | ✅ (default speaker) | ❌ | ✅ Voice reference is optional | ✅ (native default voice; instruct also works without narrator) | ✅ (direct TTS and prompt-only generation) | ❌ | N/A |
|
||||
|
||||
† **Reference Transcript:** Conditional means the transcript is required only for the specific mode: CosyVoice3 zero-shot, Qwen3-TTS full Base cloning, or MOSS-TTSD cloned-speaker dialogue. Higgs Audio 2, Higgs Audio v3, and Dots TTS accept matching text when provided but do not require it.
|
||||
† **Reference Transcript:** Conditional means the transcript is required only for the specific mode: CosyVoice3 zero-shot, Qwen3-TTS full Base cloning, or MOSS-TTSD cloned-speaker dialogue. Higgs Audio 2, Higgs Audio v3, and Dots TTS accept matching text when provided but do not require it. TADA always requires an exact transcript for its reference audio in this integration.
|
||||
+105
-104
@@ -2,110 +2,110 @@
|
||||
|
||||
## Language Support by Engine
|
||||
|
||||
| Language | Code | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | DramaBox | OmniVoice | MOSS-TTS | MOSS-SoundEffect v2 | RVC |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 🇺🇸 **English** | EN | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ Tier 1 | ✅ | ✅ Official model is English-only | ✅ | ✅ | ✅ Officially demonstrated prompt language | ✅ |
|
||||
| 🇨🇳 **Chinese** | ZH | ❌ | ❌ | ✅ | ✅ | ✅ (Mandarin) | ✅ | ✅ | ✅ + 18 dialects | ✅ | ❌ | ✅ (Mandarin + Sichuanese, Cantonese) | ❌ | ✅ Tier 1 | ✅ (Mandarin; official also exposes YUE separately outside this matrix) | ❌ | ✅ | ✅ | ✅ Officially demonstrated prompt language | ✅ |
|
||||
| 🇩🇪 **German** | DE | ✅ | ✅ (×3) | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇪🇸 **Spanish** | ES | ✅ | ❌ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇫🇷 **French** | FR | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇮🇹 **Italian** | IT | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇯🇵 **Japanese** | JA | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ✅ ? | ✅ | ✅ | ✅ (4.1-2b supports Japanese; plus variant does not) | ✅ | ❌ | ✅ Tier 1 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇰🇷 **Korean** | KO | ❌ | ✅ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇷🇺 **Russian** | RU | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇧🇷 **Portuguese** | PT | ✅ (BR) | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ (EU/BR*) | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ (Official PT tag is generic; upstream does not expose separate PT-BR/PT-PT tags and it may lean more European Portuguese than Brazilian Portuguese) | ❌ | ✅ (generic PT; official language space is much broader than this matrix) | ✅ | ❌ | ✅ |
|
||||
| 🇵🇱 **Polish** | PL | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇮🇳 **Hindi** | HI | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇪🇬 **Arabic** | AR | ❌ | ❌ | ✅ (Egyptian) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇹🇷 **Turkish** | TR | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇹🇭 **Thai** | TH | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇳🇴 **Norwegian** | NO | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
|
||||
| 🇻🇳 **Vietnamese** | VI | ❌ | ❌ | ✅ (Viterbox) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇦🇲 **Armenian** | HY | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
|
||||
| 🇬🇪 **Georgian** | KA | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
|
||||
| 🇩🇰 **Danish** | DA | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇫🇮 **Finnish** | FI | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇬🇷 **Greek** | EL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇮🇱 **Hebrew** | HE | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇲🇾 **Malay** | MS | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇳🇱 **Dutch** | NL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇸🇪 **Swedish** | SV | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇰🇪 **Swahili** | SW | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇿🇦 **Afrikaans** | AF | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇱 **Albanian** | SQ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Assamese** | AS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Asturian** | AST | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇿 **Azerbaijani** | AZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇺 **Bashkir** | BA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Basque** | EU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇾 **Belarusian** | BE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇩 **Bengali** | BN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇦 **Bosnian** | BS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇬 **Bulgarian** | BG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Catalan** | CA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇭 **Cebuano** | CEB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇶 **Central Kurdish** | CKB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇷 **Croatian** | HR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇿 **Czech** | CS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇺 **Eastern Mari** | MHR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🌍 **Esperanto** | EO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇪 **Estonian** | ET | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Galician** | GL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Gujarati** | GU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇹 **Haitian Creole** | HT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇬 **Hausa** | HA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇺 **Hungarian** | HU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇩 **Indonesian** | ID | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇩 **Javanese** | JV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Kannada** | KN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇿 **Kazakh** | KK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇼 **Kinyarwanda** | RW | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇬 **Kyrgyz** | KY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇻 **Latvian** | LV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇩 **Lingala** | LN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇹 **Lithuanian** | LT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇪 **Luo** | LUO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇰 **Macedonian** | MK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Malayalam** | ML | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇹 **Maltese** | MT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇿 **Māori** | MI | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Marathi** | MR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇳 **Mongolian** | MN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇵 **Nepali** | NE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇫🇷 **Occitan** | OC | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇷 **Persian** | FA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇴 **Romanian** | RO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Sepedi** | NSO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇸 **Serbian** | SR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇼 **Shona** | SN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇰 **Slovak** | SK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇮 **Slovene** | SL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇭 **Tagalog** | TL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇹🇯 **Tajik** | TG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Tamil** | TA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Telugu** | TE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇦 **Ukrainian** | UK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇰 **Urdu** | UR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇳 **Uyghur** | UG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇿 **Uzbek** | UZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Xhosa** | XH | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Zulu** | ZU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇼 **Chichewa/Nyanja** | NY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Eastern Punjabi** | PA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇬 **Ganda** | LG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇸 **Icelandic** | IS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇪 **Irish** | GA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇩🇿 **Kabyle** | KAB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇻 **Kabuverdianu** | KEA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇪 **Kamba** | KAM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇻🇦 **Latin** | LA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇺 **Luxembourgish** | LB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇹 **Oromo** | OM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇫 **Pashto** | PS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇰 **Sindhi** | SD | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇴 **Somali** | SO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇴 **Umbundu** | UMB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🏴 **Welsh** | CY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| Language | Code | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | TADA | DramaBox | OmniVoice | MOSS-TTS | MOSS-SoundEffect v2 | RVC |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 🇺🇸 **English** | EN | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ Tier 1 | ✅ | ✅ (1B and 3B-ML) | ✅ Official model is English-only | ✅ | ✅ | ✅ Officially demonstrated prompt language | ✅ |
|
||||
| 🇨🇳 **Chinese** | ZH | ❌ | ❌ | ✅ | ✅ | ✅ (Mandarin) | ✅ | ✅ | ✅ + 18 dialects | ✅ | ❌ | ✅ (Mandarin + Sichuanese, Cantonese) | ❌ | ✅ Tier 1 | ✅ (Mandarin; official also exposes YUE separately outside this matrix) | ✅ (3B-ML; backend code `ch`) | ❌ | ✅ | ✅ | ✅ Officially demonstrated prompt language | ✅ |
|
||||
| 🇩🇪 **German** | DE | ✅ | ✅ (×3) | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ (3B-ML) | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇪🇸 **Spanish** | ES | ✅ | ❌ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ (3B-ML) | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇫🇷 **French** | FR | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ (3B-ML) | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇮🇹 **Italian** | IT | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (3B-ML) | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇯🇵 **Japanese** | JA | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ✅ ? | ✅ | ✅ | ✅ (4.1-2b supports Japanese; plus variant does not) | ✅ | ❌ | ✅ Tier 1 | ✅ | ✅ (3B-ML) | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇰🇷 **Korean** | KO | ❌ | ✅ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ Tier 2 | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇷🇺 **Russian** | RU | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇧🇷 **Portuguese** | PT | ✅ (BR) | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ (EU/BR*) | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ (Official PT tag is generic; upstream does not expose separate PT-BR/PT-PT tags and it may lean more European Portuguese than Brazilian Portuguese) | ✅ (3B-ML) | ❌ | ✅ (generic PT; official language space is much broader than this matrix) | ✅ | ❌ | ✅ |
|
||||
| 🇵🇱 **Polish** | PL | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (3B-ML) | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇮🇳 **Hindi** | HI | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇪🇬 **Arabic** | AR | ❌ | ❌ | ✅ (Egyptian) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ (3B-ML) | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇹🇷 **Turkish** | TR | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇹🇭 **Thai** | TH | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇳🇴 **Norwegian** | NO | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
|
||||
| 🇻🇳 **Vietnamese** | VI | ❌ | ❌ | ✅ (Viterbox) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇦🇲 **Armenian** | HY | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
|
||||
| 🇬🇪 **Georgian** | KA | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ |
|
||||
| 🇩🇰 **Danish** | DA | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇫🇮 **Finnish** | FI | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇬🇷 **Greek** | EL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇮🇱 **Hebrew** | HE | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇲🇾 **Malay** | MS | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇳🇱 **Dutch** | NL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇸🇪 **Swedish** | SV | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇰🇪 **Swahili** | SW | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (v1.5) | ❌ | ✅ |
|
||||
| 🇿🇦 **Afrikaans** | AF | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇱 **Albanian** | SQ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Assamese** | AS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Asturian** | AST | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇿 **Azerbaijani** | AZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇺 **Bashkir** | BA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Basque** | EU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇾 **Belarusian** | BE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇩 **Bengali** | BN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇦 **Bosnian** | BS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇬 **Bulgarian** | BG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Catalan** | CA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇭 **Cebuano** | CEB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇶 **Central Kurdish** | CKB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇷 **Croatian** | HR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇿 **Czech** | CS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇺 **Eastern Mari** | MHR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🌍 **Esperanto** | EO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇪 **Estonian** | ET | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Galician** | GL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Gujarati** | GU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇹 **Haitian Creole** | HT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇬 **Hausa** | HA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇺 **Hungarian** | HU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇩 **Indonesian** | ID | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇩 **Javanese** | JV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Kannada** | KN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇿 **Kazakh** | KK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇼 **Kinyarwanda** | RW | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇬 **Kyrgyz** | KY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇻 **Latvian** | LV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇩 **Lingala** | LN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇹 **Lithuanian** | LT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇪 **Luo** | LUO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇰 **Macedonian** | MK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Malayalam** | ML | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇹 **Maltese** | MT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇿 **Māori** | MI | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Marathi** | MR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇳 **Mongolian** | MN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇵 **Nepali** | NE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇫🇷 **Occitan** | OC | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇷 **Persian** | FA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇴 **Romanian** | RO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Sepedi** | NSO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇸 **Serbian** | SR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇼 **Shona** | SN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇰 **Slovak** | SK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇮 **Slovene** | SL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇭 **Tagalog** | TL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇹🇯 **Tajik** | TG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Tamil** | TA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Telugu** | TE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇦 **Ukrainian** | UK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇰 **Urdu** | UR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇳 **Uyghur** | UG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇿 **Uzbek** | UZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Xhosa** | XH | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Zulu** | ZU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇼 **Chichewa/Nyanja** | NY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Eastern Punjabi** | PA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇬 **Ganda** | LG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇸 **Icelandic** | IS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇪 **Irish** | GA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇩🇿 **Kabyle** | KAB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇻 **Kabuverdianu** | KEA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇪 **Kamba** | KAM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇻🇦 **Latin** | LA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇺 **Luxembourgish** | LB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇹 **Oromo** | OM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇫 **Pashto** | PS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇰 **Sindhi** | SD | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇴 **Somali** | SO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇴 **Umbundu** | UMB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🏴 **Welsh** | CY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
|
||||
**Notes:**
|
||||
|
||||
@@ -114,6 +114,7 @@
|
||||
- **Higgs Audio v3**: Official model card lists 102 languages total: 85 polished-production languages and 17 usable but less polished languages.
|
||||
- **IndexTTS-2**: Trained on 55K+ hours - ZH, EN, JA primary.
|
||||
- **MOSS-TTS**: Official list includes ZH, EN, DE, ES, FR, JA, IT, HU, KO, RU, FA, AR, PL, PT, CS, DA, SV, EL, TR; HU/FA/CS are not separate columns in this matrix.
|
||||
- **TADA**: TADA 1B is English-only. TADA 3B-ML supports EN, AR, ZH, DE, ES, FR, IT, JA, PL, and PT; the upstream backend calls Chinese `ch`.
|
||||
- **OmniVoice**: Official model supports 600+ languages; this matrix only shows the suite's comparison subset.
|
||||
- **RVC**: Language-agnostic voice conversion/post-processing; it is marked supported for every language row.
|
||||
- Qwen3-TTS: Supports pt-BR only with instruction (voice design or harcoded voices). Base can't do instructions, so it will always ouput pt-PT.
|
||||
@@ -129,6 +129,16 @@ Use this as the canonical list of model repositories/links for offline setup.
|
||||
| dots.tts-soar | [rednote-hilab/dots.tts-soar](https://huggingface.co/rednote-hilab/dots.tts-soar) | ~6GB | ✅ | Official SOAR checkpoint for higher-quality zero-shot cloning |
|
||||
| dots.tts-mf | [rednote-hilab/dots.tts-mf](https://huggingface.co/rednote-hilab/dots.tts-mf) | ~6GB | ✅ | Official MeanFlow-distilled checkpoint for faster inference |
|
||||
|
||||
## TADA
|
||||
|
||||
| Component | Source | Size | Auto-Download | Notes |
|
||||
|---|---|---|---|---|
|
||||
| TADA 1B | [HumeAI/tada-1b](https://huggingface.co/HumeAI/tada-1b) | ~3.9GB | ✅ | English-only model; downloaded only when selected |
|
||||
| TADA 3B-ML | [HumeAI/tada-3b-ml](https://huggingface.co/HumeAI/tada-3b-ml) | ~8.9GB | ✅ | Multilingual model; downloaded only when selected |
|
||||
| TADA codec | [HumeAI/tada-codec](https://huggingface.co/HumeAI/tada-codec) | ~2.15GB per selected language | ✅ | Downloads shared encoder/decoder plus only the aligner required by the selected language |
|
||||
| Llama tokenizer | [onnx-community/Llama-3.2-1B](https://huggingface.co/onnx-community/Llama-3.2-1B) | Tokenizer files only | ✅ | Ungated redistribution; runtime files are verified against Meta's official Git blob hashes and remain under the Llama 3.2 Community License |
|
||||
| Wav2Vec aligner configuration | [facebook/wav2vec2-large](https://huggingface.co/facebook/wav2vec2-large) | Configuration file only | ✅ | Downloads config.json only; no Wav2Vec model weights |
|
||||
|
||||
## DramaBox
|
||||
|
||||
| Component | Source | Size | Auto-Download | Notes |
|
||||
|
||||
@@ -427,6 +427,28 @@ Notes:
|
||||
- Native sample rate is 48kHz.
|
||||
- Main-environment support works on Transformers 5; on Windows, `normalize_text` falls back to no-op if `WeTextProcessing` is unavailable.
|
||||
|
||||
## TADA
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/tada/
|
||||
├── TADA-1B/ # downloaded only if selected
|
||||
├── TADA-3B-ML/ # downloaded only if selected
|
||||
├── tada-codec/
|
||||
│ ├── encoder/
|
||||
│ ├── decoder/
|
||||
│ ├── aligner[-<language>]/ # selected language only
|
||||
│ └── wav2vec2-large/
|
||||
└── llama-3.2-1b-tokenizer/ # tokenizer files only
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- TADA runs in the shared Transformers 4 runtime with `hume-tada==0.1.9`.
|
||||
- The selected 1B or 3B-ML model, shared codec components, and selected language aligner download on demand.
|
||||
- The Llama tokenizer downloads from an ungated redistribution and is verified against Meta's official Git blob hashes. It remains under the Llama 3.2 Community License.
|
||||
- Reference audio and its exact transcript are required by this integration.
|
||||
- Native output is mono 24kHz.
|
||||
|
||||
## OmniVoice
|
||||
|
||||
```text
|
||||
|
||||
@@ -27,7 +27,7 @@ Downloads and dependencies:
|
||||
- Did you document dependency conflicts or install.py changes?
|
||||
- Did you test `--no-deps` on Main/T5 and simple patching before falling back to the shared T4 runtime?
|
||||
- Did you avoid creating another environment or downloading/reinstalling Torch or Transformers without explicit maintainer approval?
|
||||
- If runtime isolation is needed, did you document the default mode and the reason in YAML/README?
|
||||
- If runtime isolation is recommended, does the engine expose the standard runtime selector, default to the recommended mode, route every choice honestly, and document why?
|
||||
|
||||
Audio format:
|
||||
- Did you verify the model native sample rate?
|
||||
@@ -42,7 +42,7 @@ Generated audio cache:
|
||||
|
||||
Text features:
|
||||
- Did you wire character tags if this node supports text generation?
|
||||
- Did you handle narrator fallback?
|
||||
- Does a missing character use narrator fallback while an existing but invalid voice (for example, missing required transcript) raises a clear error?
|
||||
- Did you wire pause tags?
|
||||
- Did you wire parameter switching for real native parameters?
|
||||
- Did you avoid adding fake unsupported parameters?
|
||||
|
||||
@@ -26,6 +26,7 @@
|
||||
|
||||
### Voice References
|
||||
- **Reference text requirement**: F5-TTS and Step Audio EditX REQUIRE `prompt_text` (transcript), ChatterBox/VibeVoice/Higgs don't
|
||||
- **Missing vs invalid character voice**: A character with no saved voice may fall back to narrator; a discovered/selected voice missing an engine-required transcript is invalid and MUST raise a clear error instead of silently using narrator
|
||||
- **Narrator voice mapping**: Map narrator from TTS Text node input by saving `audio_tensor` to temp file with `reference_text`
|
||||
- **Voice discovery**: Use `get_character_mapping()` not `discover_voices_for_engine()` (latter doesn't exist)
|
||||
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
# TADA Engine Guide
|
||||
|
||||
TADA is Hume AI's zero-shot text-to-speech model. TTS Audio Suite exposes it through the unified **Text to Speech** and **SRT** nodes, including character switching, per-segment language and generation parameters, pause tags, caching, interruption, and Clear VRAM.
|
||||
|
||||
This integration deliberately describes TADA's expressive capability as **reference-conditioned expression transfer**. The model can carry voice, delivery, timing, and emotion present in the prompt audio into generated speech, but it does not provide a native emotion label, intensity slider, embedding, or proven text-instruction emotion control. It is not direct emotion control.
|
||||
|
||||
## Supported models and languages
|
||||
|
||||
| Model | Languages | Model weights | Intended use |
|
||||
|---|---|---:|---|
|
||||
| `HumeAI/tada-1b` | English | ~3.9GB | Lighter English-only option |
|
||||
| `HumeAI/tada-3b-ml` | English, Arabic, Chinese, German, Spanish, French, Italian, Japanese, Polish, Portuguese | ~8.9GB | Multilingual voice cloning |
|
||||
|
||||
The upstream language code for Chinese is `ch`; the suite presents it as Chinese/`zh` and maps it internally. TADA generates mono audio at 24kHz. Reference audio is resampled to 24kHz when needed.
|
||||
|
||||
## First-use requirements
|
||||
|
||||
TADA runs only in the suite's **shared Transformers 4 runtime**. The integration installs `hume-tada==0.1.9` there and does not modify the main ComfyUI Transformers 5 environment. TADA's current configuration code is incompatible with that main environment.
|
||||
|
||||
Before first use, select TADA in an **⚙️ TADA Engine** node and run a workflow. The suite downloads the selected TADA model, shared codec encoder/decoder, the aligner for the selected language, and tokenizer files into `ComfyUI/models/TTS/tada/`.
|
||||
|
||||
Downloads are selective. Choosing the 1B model does not download the 3B model, and changing language downloads only that language's codec aligner. A selected codec set is approximately 2.15GB in addition to the model checkpoint.
|
||||
|
||||
### Automatic download
|
||||
|
||||
No Hugging Face login is required for the normal automatic path. The suite downloads the three tokenizer files from the ungated `onnx-community/Llama-3.2-1B` redistribution, plus its `LICENSE.txt` and `USE_POLICY.md`. Those runtime files are byte-identical to Meta's official `meta-llama/Llama-3.2-1B` tokenizer and are checked against the official Git blob hashes before use.
|
||||
|
||||
The tokenizer remains governed by the Llama 3.2 Community License and Acceptable Use Policy. TADA's use of this component is **Built with Llama**. The full Llama model weights are not downloaded.
|
||||
|
||||
### Manual/offline installation
|
||||
|
||||
Users accustomed to managing ComfyUI models manually can copy the exact tokenizer and model files into the following layout. Only these files are required at runtime:
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/tada/
|
||||
├── TADA-1B/
|
||||
│ ├── config.json
|
||||
│ ├── generation_config.json
|
||||
│ └── model.safetensors
|
||||
├── TADA-3B-ML/
|
||||
│ ├── config.json
|
||||
│ ├── generation_config.json
|
||||
│ ├── model.safetensors.index.json
|
||||
│ ├── model-00001-of-00002.safetensors
|
||||
│ └── model-00002-of-00002.safetensors
|
||||
├── tada-codec/
|
||||
│ ├── encoder/config.json
|
||||
│ ├── encoder/model.safetensors
|
||||
│ ├── decoder/config.json
|
||||
│ ├── decoder/model.safetensors
|
||||
│ ├── aligner/config.json # English
|
||||
│ ├── aligner/model.safetensors
|
||||
│ ├── aligner-<language>/config.json # selected non-English language
|
||||
│ ├── aligner-<language>/model.safetensors
|
||||
│ └── wav2vec2-large/config.json
|
||||
└── llama-3.2-1b-tokenizer/
|
||||
├── tokenizer.json
|
||||
├── tokenizer_config.json
|
||||
└── special_tokens_map.json
|
||||
```
|
||||
|
||||
Install only the selected model directory. For non-English generation, `<language>` is one of `ar`, `ch`, `de`, `es`, `fr`, `it`, `ja`, `pl`, or `pt`. Existing complete files are detected and are not downloaded again. The full Llama model weights and Wav2Vec weights are not required.
|
||||
|
||||
## Reference voice requirements
|
||||
|
||||
Every generation in this integration requires:
|
||||
|
||||
- a clean reference speech clip; and
|
||||
- an exact transcript of the speech in that clip.
|
||||
|
||||
The transcript must match what is spoken, including word order. A mismatched transcript weakens alignment and cloning quality. TADA's official notebook can invoke an English Parakeet ASR model when text is omitted, but the suite does not do that: it would add a hidden ~4.25GB English-only dependency and would be unsuitable for multilingual references. Supply the exact text explicitly or save it beside a character voice as `.reference.txt`/`.txt`.
|
||||
|
||||
Expression comes from the reference recording. Use a prompt that actually contains the desired energy, pace, emotion, and speaking style. There is no emotion dropdown because adding one would be a fake control unsupported by the model API.
|
||||
|
||||
## Engine controls
|
||||
|
||||
Choose the model variant, device, language, and dtype on the engine node. `auto` is the default for device and dtype, and English is the default language. The engine node and unified TTS nodes expose these generation controls:
|
||||
|
||||
| Control | Suite default | Effect |
|
||||
|---|---:|---|
|
||||
| `seed` | unified-node value | Reproducible stochastic generation |
|
||||
| `acoustic_cfg_scale` | `1.6` | Acoustic classifier-free guidance strength |
|
||||
| `duration_cfg_scale` | `1.0` | Duration guidance strength |
|
||||
| `cfg_schedule` | `cosine` | Guidance schedule: `constant`, `linear`, or `cosine` |
|
||||
| `num_flow_matching_steps` | `10` | More ODE steps trade speed for refinement |
|
||||
| `noise_temperature` | `0.9` | Initial flow-matching noise scale |
|
||||
| `time_schedule` | `logsnr` | ODE schedule: `uniform`, `cosine`, or `logsnr` |
|
||||
| `negative_condition_source` | `negative_step_output` | Source used for negative CFG conditioning |
|
||||
| `speed_up_factor` | `0.0` (disabled) | Native two-pass duration scaling; values above 0 add a second generation pass, and values above 1 speak faster |
|
||||
| `num_transition_steps` | `5` | Prompt-to-generated-speech transition overlap |
|
||||
|
||||
Defaults follow the official implementation. Change one parameter at a time: CFG, noise, flow steps, and transition handling interact, and higher values are not automatically better. `speed_up_factor` is not a cheap playback-rate effect; it reruns generation with scaled predicted durations.
|
||||
|
||||
Per-segment tags can override supported parameters and language in unified Text or SRT prompts. The 1B model rejects non-English language selection instead of silently generating with the wrong model.
|
||||
|
||||
## Long text and SRT
|
||||
|
||||
TADA has no native long-form or streaming mode in this integration. The unified Text node uses the suite's sentence-aware chunking and combines the resulting 24kHz clips. The SRT processor generates each subtitle independently and uses the suite's timing modes and overlap fallback.
|
||||
|
||||
The official model was trained on short clips (up to roughly 30 seconds), so long uninterrupted generations can drift. Prefer normal chunking or subtitle-sized segments. Native batch inference, streaming, speech continuation, random-voice generation, text-sampling controls, multiple-candidate scoring, and speaker verification are not exposed in the first integration. Text sampling is irrelevant to normal suite TTS because the exact target text is supplied.
|
||||
|
||||
## Model layout
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/tada/
|
||||
├── TADA-1B/ # downloaded only if selected
|
||||
├── TADA-3B-ML/ # downloaded only if selected
|
||||
├── tada-codec/
|
||||
│ ├── encoder/
|
||||
│ ├── decoder/
|
||||
│ ├── aligner[-<language>]/ # selected language only
|
||||
│ └── wav2vec2-large/ # aligner config only, not model weights
|
||||
└── llama-3.2-1b-tokenizer/ # tokenizer files only
|
||||
```
|
||||
|
||||
## Licensing and upstream sources
|
||||
|
||||
- Official implementation: [HumeAI/tada](https://github.com/HumeAI/tada) — MIT code license.
|
||||
- Model checkpoints: [HumeAI/tada-1b](https://huggingface.co/HumeAI/tada-1b), [HumeAI/tada-3b-ml](https://huggingface.co/HumeAI/tada-3b-ml), and [HumeAI/tada-codec](https://huggingface.co/HumeAI/tada-codec).
|
||||
- Model weights and tokenizer are subject to the Meta Llama 3.2 Community License. Commercial use is conditional on those terms. Applications distributing or presenting the model must follow the license's attribution requirements, including the required **Built with Llama** notice where applicable.
|
||||
|
||||
TTS Audio Suite does not replace or relax upstream license terms. Review them before redistributing models or generated products.
|
||||
@@ -0,0 +1,256 @@
|
||||
"""Adapter between TTS Audio Suite processors and the isolated TADA runtime."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
from typing import Any, Dict, Optional, Tuple
|
||||
|
||||
import torch
|
||||
|
||||
current_dir = os.path.dirname(__file__)
|
||||
engines_dir = os.path.dirname(current_dir)
|
||||
project_root = os.path.dirname(engines_dir)
|
||||
if project_root not in sys.path:
|
||||
sys.path.insert(0, project_root)
|
||||
|
||||
from engines.tada.languages import format_tada_language_display, validate_tada_model_language
|
||||
from engines.tada.tada_downloader import TadaDownloader
|
||||
from utils.audio.audio_hash import generate_stable_audio_component
|
||||
from utils.audio.cache import get_audio_cache
|
||||
from utils.device import resolve_torch_device
|
||||
from utils.models.factory_config import ModelLoadConfig, RUNTIME_MODE_SHARED
|
||||
from utils.voice.reference import effective_voice_audio
|
||||
|
||||
|
||||
class TadaEngineAdapter:
|
||||
"""Resolve assets, load the shared runtime proxy, and cache TADA waveforms."""
|
||||
|
||||
SAMPLE_RATE = 24000
|
||||
|
||||
def __init__(self, config: Optional[Dict[str, Any]] = None):
|
||||
self.config = config.copy() if config else {}
|
||||
self.audio_cache = get_audio_cache()
|
||||
self.downloader = TadaDownloader()
|
||||
self._last_config: Optional[ModelLoadConfig] = None
|
||||
self._load_signature: Optional[Tuple[Any, ...]] = None
|
||||
self._ensured_assets: Dict[Tuple[str, str], Dict[str, str]] = {}
|
||||
|
||||
def update_config(self, new_config: Dict[str, Any]) -> None:
|
||||
self.config = new_config.copy() if new_config else {}
|
||||
|
||||
def _model_identifier(self) -> str:
|
||||
return str(
|
||||
self.config.get("model_variant")
|
||||
or self.config.get("model_path")
|
||||
or self.config.get("model_name")
|
||||
or "TADA-1B"
|
||||
)
|
||||
|
||||
def _ensure_assets(self, language: str) -> Dict[str, str]:
|
||||
identifier = self._model_identifier()
|
||||
display_language = format_tada_language_display(language)
|
||||
key = (identifier, display_language)
|
||||
assets = self._ensured_assets.get(key)
|
||||
if assets is None:
|
||||
assets = self.downloader.resolve_model_assets(identifier, display_language)
|
||||
self._ensured_assets[key] = assets
|
||||
return assets
|
||||
|
||||
def _build_load_signature(self, assets: Dict[str, str]) -> Tuple[Any, ...]:
|
||||
resolved_device = resolve_torch_device(self.config.get("device", "auto"))
|
||||
return (
|
||||
assets["model_name"],
|
||||
os.path.abspath(assets["model_path"]),
|
||||
os.path.abspath(assets["codec_path"]),
|
||||
os.path.abspath(assets["tokenizer_path"]),
|
||||
resolved_device,
|
||||
self.config.get("dtype", "auto"),
|
||||
self.config.get("attn_implementation", "sdpa"),
|
||||
bool(self.config.get("use_torch_compile", False)),
|
||||
int(self.config.get("prompt_cache_size", 8)),
|
||||
self.config.get("runtime_mode", RUNTIME_MODE_SHARED),
|
||||
self.config.get("runtime_profile", "vibevoice_transformers4_shared"),
|
||||
)
|
||||
|
||||
def _ensure_model_loaded(self, language: str) -> None:
|
||||
from utils.models.unified_model_interface import unified_model_interface
|
||||
|
||||
assets = self._ensure_assets(language)
|
||||
signature = self._build_load_signature(assets)
|
||||
if signature == self._load_signature and self._last_config is not None:
|
||||
return
|
||||
|
||||
resolved_device = resolve_torch_device(self.config.get("device", "auto"))
|
||||
load_config = ModelLoadConfig(
|
||||
engine_name="tada",
|
||||
model_type="tts",
|
||||
model_name=assets["model_name"],
|
||||
model_path=assets["model_path"],
|
||||
device=resolved_device,
|
||||
additional_params={
|
||||
"codec_path": assets["codec_path"],
|
||||
"tokenizer_path": assets["tokenizer_path"],
|
||||
"dtype": self.config.get("dtype", "auto"),
|
||||
"attn_implementation": self.config.get("attn_implementation", "sdpa"),
|
||||
"use_torch_compile": bool(self.config.get("use_torch_compile", False)),
|
||||
"prompt_cache_size": int(self.config.get("prompt_cache_size", 8)),
|
||||
},
|
||||
runtime_mode=self.config.get("runtime_mode", RUNTIME_MODE_SHARED),
|
||||
runtime_profile=self.config.get(
|
||||
"runtime_profile", "vibevoice_transformers4_shared"
|
||||
),
|
||||
)
|
||||
self._last_config = load_config
|
||||
unified_model_interface.load_model(load_config)
|
||||
self._load_signature = signature
|
||||
|
||||
def _get_engine(self):
|
||||
if self._last_config is None:
|
||||
raise RuntimeError("TADA model was requested before its assets were resolved")
|
||||
from utils.models.unified_model_interface import unified_model_interface
|
||||
|
||||
return unified_model_interface.load_model(self._last_config)
|
||||
|
||||
@staticmethod
|
||||
def _reference_text(voice_ref: Optional[Dict[str, Any]]) -> str:
|
||||
if not isinstance(voice_ref, dict):
|
||||
return ""
|
||||
return str(
|
||||
voice_ref.get("reference_text")
|
||||
or voice_ref.get("prompt_text")
|
||||
or voice_ref.get("text")
|
||||
or ""
|
||||
).strip()
|
||||
|
||||
@staticmethod
|
||||
def _audio_component(reference_audio: Any, voice_ref: Dict[str, Any]) -> str:
|
||||
if isinstance(reference_audio, str):
|
||||
return generate_stable_audio_component(audio_file_path=reference_audio)
|
||||
if isinstance(reference_audio, dict) and "waveform" in reference_audio:
|
||||
return generate_stable_audio_component(reference_audio=reference_audio)
|
||||
if torch.is_tensor(reference_audio):
|
||||
return generate_stable_audio_component(
|
||||
reference_audio={
|
||||
"waveform": reference_audio,
|
||||
"sample_rate": int(voice_ref.get("sample_rate", TadaEngineAdapter.SAMPLE_RATE)),
|
||||
}
|
||||
)
|
||||
if isinstance(reference_audio, (tuple, list)) and len(reference_audio) == 2:
|
||||
waveform, sample_rate = reference_audio
|
||||
if torch.is_tensor(waveform):
|
||||
return generate_stable_audio_component(
|
||||
reference_audio={"waveform": waveform, "sample_rate": int(sample_rate)}
|
||||
)
|
||||
raise TypeError(f"Unsupported TADA reference audio type: {type(reference_audio)}")
|
||||
|
||||
def generate_single(
|
||||
self,
|
||||
text: str,
|
||||
voice_ref: Optional[Dict[str, Any]],
|
||||
seed: int = 0,
|
||||
enable_audio_cache: bool = True,
|
||||
character_name: Optional[str] = None,
|
||||
) -> torch.Tensor:
|
||||
stripped = str(text or "").strip()
|
||||
if not stripped:
|
||||
return torch.zeros(1, 0, dtype=torch.float32)
|
||||
if not isinstance(voice_ref, dict):
|
||||
raise ValueError("TADA requires reference audio and its exact transcript")
|
||||
|
||||
reference_audio = effective_voice_audio(voice_ref)
|
||||
reference_text = self._reference_text(voice_ref)
|
||||
if reference_audio is None:
|
||||
raise ValueError("TADA requires reference audio; random-voice mode is not exposed")
|
||||
if not reference_text:
|
||||
raise ValueError(
|
||||
"TADA requires the exact reference transcript. Automatic Parakeet ASR is "
|
||||
"intentionally disabled to prevent a hidden 4.25 GB download."
|
||||
)
|
||||
|
||||
model_variant = self._model_identifier()
|
||||
language = format_tada_language_display(self.config.get("language", "English"))
|
||||
validate_tada_model_language(model_variant, language)
|
||||
speed_up_factor = float(self.config.get("speed_up_factor", 0.0) or 0.0)
|
||||
native_speed = speed_up_factor if speed_up_factor > 0 else None
|
||||
audio_component = self._audio_component(reference_audio, voice_ref)
|
||||
|
||||
generation_params = {
|
||||
"model_variant": model_variant,
|
||||
"decoder_source": "official_tada_codec_v1",
|
||||
"language": language,
|
||||
"reference_text": reference_text,
|
||||
"audio_component": audio_component,
|
||||
"seed": int(seed or 0),
|
||||
"acoustic_cfg_scale": float(self.config.get("acoustic_cfg_scale", 1.6)),
|
||||
"duration_cfg_scale": float(self.config.get("duration_cfg_scale", 1.0)),
|
||||
"noise_temperature": float(self.config.get("noise_temperature", 0.9)),
|
||||
"num_flow_matching_steps": int(self.config.get("num_flow_matching_steps", 10)),
|
||||
"cfg_schedule": self.config.get("cfg_schedule", "cosine"),
|
||||
"time_schedule": self.config.get("time_schedule", "logsnr"),
|
||||
"negative_condition_source": self.config.get(
|
||||
"negative_condition_source", "negative_step_output"
|
||||
),
|
||||
"speed_up_factor": native_speed,
|
||||
"num_transition_steps": int(self.config.get("num_transition_steps", 5)),
|
||||
"device": self.config.get("device", "auto"),
|
||||
"dtype": self.config.get("dtype", "auto"),
|
||||
"runtime_mode": self.config.get("runtime_mode", RUNTIME_MODE_SHARED),
|
||||
"runtime_profile": self.config.get(
|
||||
"runtime_profile", "vibevoice_transformers4_shared"
|
||||
),
|
||||
"character": character_name or "narrator",
|
||||
}
|
||||
|
||||
cache_key = None
|
||||
if enable_audio_cache:
|
||||
cache_key = self.audio_cache.generate_cache_key(
|
||||
"tada", text=stripped, **generation_params
|
||||
)
|
||||
cached = self.audio_cache.get_cached_audio(cache_key)
|
||||
if cached:
|
||||
print(
|
||||
f"💾 Using cached TADA audio for '{character_name or 'narrator'}': "
|
||||
f"'{stripped[:30]}...'"
|
||||
)
|
||||
return cached[0]
|
||||
|
||||
self._ensure_model_loaded(language)
|
||||
engine = self._get_engine()
|
||||
audio, sample_rate = engine.generate_speech(
|
||||
text=stripped,
|
||||
reference_audio=reference_audio,
|
||||
reference_text=reference_text,
|
||||
seed=generation_params["seed"],
|
||||
language=language,
|
||||
acoustic_cfg_scale=generation_params["acoustic_cfg_scale"],
|
||||
duration_cfg_scale=generation_params["duration_cfg_scale"],
|
||||
cfg_schedule=generation_params["cfg_schedule"],
|
||||
time_schedule=generation_params["time_schedule"],
|
||||
num_flow_matching_steps=generation_params["num_flow_matching_steps"],
|
||||
noise_temperature=generation_params["noise_temperature"],
|
||||
speed_up_factor=native_speed,
|
||||
num_transition_steps=generation_params["num_transition_steps"],
|
||||
negative_condition_source=generation_params["negative_condition_source"],
|
||||
)
|
||||
if int(sample_rate) != self.SAMPLE_RATE:
|
||||
raise RuntimeError(
|
||||
f"TADA returned {sample_rate} Hz audio; expected {self.SAMPLE_RATE} Hz"
|
||||
)
|
||||
if not isinstance(audio, torch.Tensor):
|
||||
audio = torch.as_tensor(audio, dtype=torch.float32)
|
||||
audio = audio.detach().cpu().float()
|
||||
if audio.dim() == 1:
|
||||
audio = audio.unsqueeze(0)
|
||||
elif audio.dim() != 2:
|
||||
raise RuntimeError(f"TADA returned an invalid audio shape: {tuple(audio.shape)}")
|
||||
if audio.shape[-1] == 0:
|
||||
raise RuntimeError("TADA returned empty audio")
|
||||
|
||||
if enable_audio_cache and cache_key:
|
||||
duration = self.audio_cache._calculate_duration(audio, "tada")
|
||||
self.audio_cache.cache_audio(cache_key, audio, duration)
|
||||
return audio
|
||||
|
||||
|
||||
__all__ = ["TadaEngineAdapter"]
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Hume TADA integration with lazy exports safe for the worker runtime."""
|
||||
|
||||
from .languages import (
|
||||
TADA_LANGUAGE_OPTIONS,
|
||||
format_tada_language_display,
|
||||
normalize_tada_language,
|
||||
validate_tada_model_language,
|
||||
)
|
||||
|
||||
|
||||
def __getattr__(name):
|
||||
if name in {"TadaDownloader", "tada_downloader"}:
|
||||
from .tada_downloader import TadaDownloader, tada_downloader
|
||||
|
||||
return {"TadaDownloader": TadaDownloader, "tada_downloader": tada_downloader}[name]
|
||||
if name in {"TADA_SAMPLE_RATE", "TadaEngine"}:
|
||||
from .tada_engine import TADA_SAMPLE_RATE, TadaEngine
|
||||
|
||||
return {"TADA_SAMPLE_RATE": TADA_SAMPLE_RATE, "TadaEngine": TadaEngine}[name]
|
||||
raise AttributeError(name)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"TADA_LANGUAGE_OPTIONS",
|
||||
"TADA_SAMPLE_RATE",
|
||||
"TadaDownloader",
|
||||
"TadaEngine",
|
||||
"format_tada_language_display",
|
||||
"normalize_tada_language",
|
||||
"tada_downloader",
|
||||
"validate_tada_model_language",
|
||||
]
|
||||
@@ -0,0 +1,91 @@
|
||||
"""Language metadata and validation for the official Hume TADA checkpoints."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Optional
|
||||
|
||||
|
||||
_LANGUAGE_DISPLAY_TO_CODE = {
|
||||
"English": None,
|
||||
"Arabic": "ar",
|
||||
"Chinese": "ch",
|
||||
"German": "de",
|
||||
"Spanish": "es",
|
||||
"French": "fr",
|
||||
"Italian": "it",
|
||||
"Japanese": "ja",
|
||||
"Polish": "pl",
|
||||
"Portuguese": "pt",
|
||||
}
|
||||
|
||||
TADA_LANGUAGE_OPTIONS = tuple(_LANGUAGE_DISPLAY_TO_CODE)
|
||||
|
||||
_LANGUAGE_ALIASES = {
|
||||
"": None,
|
||||
"auto": None,
|
||||
"default": None,
|
||||
"en": None,
|
||||
"eng": None,
|
||||
"english": None,
|
||||
"ar": "ar",
|
||||
"arabic": "ar",
|
||||
"ch": "ch",
|
||||
"zh": "ch",
|
||||
"zh-cn": "ch",
|
||||
"chinese": "ch",
|
||||
"de": "de",
|
||||
"german": "de",
|
||||
"es": "es",
|
||||
"spanish": "es",
|
||||
"fr": "fr",
|
||||
"french": "fr",
|
||||
"it": "it",
|
||||
"italian": "it",
|
||||
"ja": "ja",
|
||||
"jp": "ja",
|
||||
"japanese": "ja",
|
||||
"pl": "pl",
|
||||
"polish": "pl",
|
||||
"pt": "pt",
|
||||
"pt-br": "pt",
|
||||
"portuguese": "pt",
|
||||
}
|
||||
|
||||
|
||||
def normalize_tada_language(value: object) -> Optional[str]:
|
||||
"""Return the official aligner code, using ``None`` for English."""
|
||||
if value is None:
|
||||
return None
|
||||
normalized = str(value).strip().lower().replace("_", "-")
|
||||
if normalized not in _LANGUAGE_ALIASES:
|
||||
supported = ", ".join(TADA_LANGUAGE_OPTIONS)
|
||||
raise ValueError(f"Unsupported TADA language '{value}'. Supported languages: {supported}")
|
||||
return _LANGUAGE_ALIASES[normalized]
|
||||
|
||||
|
||||
def format_tada_language_display(value: object) -> str:
|
||||
"""Return the canonical UI label for a language value or backend code."""
|
||||
code = normalize_tada_language(value)
|
||||
for display, candidate in _LANGUAGE_DISPLAY_TO_CODE.items():
|
||||
if candidate == code:
|
||||
return display
|
||||
return "English"
|
||||
|
||||
|
||||
def validate_tada_model_language(model_name: str, language: object) -> Optional[str]:
|
||||
"""Validate checkpoint language support and return its backend aligner code."""
|
||||
code = normalize_tada_language(language)
|
||||
normalized_model = str(model_name or "").lower().replace("_", "-")
|
||||
if code is not None and ("tada-1b" in normalized_model or normalized_model == "1b"):
|
||||
raise ValueError(
|
||||
"TADA-1B is English-only. Select TADA-3B-ML for multilingual synthesis."
|
||||
)
|
||||
return code
|
||||
|
||||
|
||||
__all__ = [
|
||||
"TADA_LANGUAGE_OPTIONS",
|
||||
"format_tada_language_display",
|
||||
"normalize_tada_language",
|
||||
"validate_tada_model_language",
|
||||
]
|
||||
@@ -0,0 +1,322 @@
|
||||
"""Strict, organized downloader for the official Hume TADA assets."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
import folder_paths
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
from utils.hf_download_logging import quiet_hf_download_logs
|
||||
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
|
||||
|
||||
from .languages import normalize_tada_language, validate_tada_model_language
|
||||
|
||||
|
||||
class TadaDownloader:
|
||||
"""Download only the files used by TTS Audio Suite's local-only TADA loader."""
|
||||
|
||||
MODELS = {
|
||||
"TADA-1B": {
|
||||
"repo_id": "HumeAI/tada-1b",
|
||||
"files": [
|
||||
"config.json",
|
||||
"generation_config.json",
|
||||
"model.safetensors",
|
||||
],
|
||||
},
|
||||
"TADA-3B-ML": {
|
||||
"repo_id": "HumeAI/tada-3b-ml",
|
||||
"files": [
|
||||
"config.json",
|
||||
"generation_config.json",
|
||||
"model.safetensors.index.json",
|
||||
"model-00001-of-00002.safetensors",
|
||||
"model-00002-of-00002.safetensors",
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
# Ungated redistribution whose three tokenizer files are byte-identical to
|
||||
# Meta's official Llama 3.2 1B repository. Keep the official repo as the
|
||||
# provenance reference and verify Git blob hashes after every install.
|
||||
TOKENIZER_REPO = "onnx-community/Llama-3.2-1B"
|
||||
TOKENIZER_REFERENCE_REPO = "meta-llama/Llama-3.2-1B"
|
||||
TOKENIZER_DIRECTORY = "llama-3.2-1b-tokenizer"
|
||||
TOKENIZER_FILES = [
|
||||
"special_tokens_map.json",
|
||||
"tokenizer.json",
|
||||
"tokenizer_config.json",
|
||||
]
|
||||
TOKENIZER_DISTRIBUTION_FILES = TOKENIZER_FILES + ["LICENSE.txt", "USE_POLICY.md"]
|
||||
TOKENIZER_GIT_BLOB_SHA1 = {
|
||||
"special_tokens_map.json": "cfabacc2620186cd3dd4b1dde9a37e057208636e",
|
||||
"tokenizer.json": "5cc5f00a5b203e90a27a3bd60d1ec393b07971e8",
|
||||
"tokenizer_config.json": "cb9ec25536e44d86778b10509d3e5bdca459a5cf",
|
||||
}
|
||||
|
||||
CODEC_REPO = "HumeAI/tada-codec"
|
||||
CODEC_DIRECTORY = "tada-codec"
|
||||
CODEC_BASE_FILES = [
|
||||
"encoder/config.json",
|
||||
"encoder/model.safetensors",
|
||||
"decoder/config.json",
|
||||
"decoder/model.safetensors",
|
||||
]
|
||||
WAV2VEC_REPO = "facebook/wav2vec2-large"
|
||||
WAV2VEC_CONFIG_LOCAL = "wav2vec2-large/config.json"
|
||||
|
||||
_ALIASES = {
|
||||
"tada-1b": "TADA-1B",
|
||||
"1b": "TADA-1B",
|
||||
"humeai/tada-1b": "TADA-1B",
|
||||
"tada-3b-ml": "TADA-3B-ML",
|
||||
"tada-3b": "TADA-3B-ML",
|
||||
"3b": "TADA-3B-ML",
|
||||
"3b-ml": "TADA-3B-ML",
|
||||
"humeai/tada-3b-ml": "TADA-3B-ML",
|
||||
}
|
||||
|
||||
def __init__(self, base_path: Optional[str] = None):
|
||||
if base_path is None:
|
||||
try:
|
||||
base_path = get_preferred_download_path(model_type="TTS", engine_name="tada")
|
||||
except Exception:
|
||||
base_path = os.path.join(folder_paths.models_dir, "TTS", "tada")
|
||||
self.base_path = os.path.abspath(str(base_path))
|
||||
os.makedirs(self.base_path, exist_ok=True)
|
||||
|
||||
@classmethod
|
||||
def canonical_model_name(cls, model_identifier: str) -> Optional[str]:
|
||||
value = str(model_identifier or "TADA-1B").strip()
|
||||
if value in cls.MODELS:
|
||||
return value
|
||||
return cls._ALIASES.get(value.lower().replace("_", "-"))
|
||||
|
||||
@staticmethod
|
||||
def _files_ready(target_dir: str, files: List[str]) -> bool:
|
||||
return all(
|
||||
os.path.isfile(os.path.join(target_dir, name))
|
||||
and os.path.getsize(os.path.join(target_dir, name)) > 0
|
||||
for name in files
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _git_blob_sha1(path: str) -> str:
|
||||
size = os.path.getsize(path)
|
||||
digest = hashlib.sha1()
|
||||
digest.update(f"blob {size}\0".encode("ascii"))
|
||||
with open(path, "rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
@classmethod
|
||||
def _validate_tokenizer_files(cls, target_dir: str) -> None:
|
||||
mismatches = []
|
||||
for filename, expected in cls.TOKENIZER_GIT_BLOB_SHA1.items():
|
||||
path = os.path.join(target_dir, filename)
|
||||
actual = cls._git_blob_sha1(path)
|
||||
if actual != expected:
|
||||
mismatches.append(f"{filename} ({actual})")
|
||||
if mismatches:
|
||||
raise RuntimeError(
|
||||
"The installed TADA tokenizer does not match Meta's official Llama 3.2 1B "
|
||||
"tokenizer. Remove the tokenizer directory and retry. Mismatched files: "
|
||||
+ ", ".join(mismatches)
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _download_files(
|
||||
repo_id: str,
|
||||
target_dir: str,
|
||||
files: List[str],
|
||||
label: str,
|
||||
force: bool,
|
||||
) -> str:
|
||||
if not force and TadaDownloader._files_ready(target_dir, files):
|
||||
return target_dir
|
||||
os.makedirs(target_dir, exist_ok=True)
|
||||
print(f"\n{'=' * 60}")
|
||||
print("📦 TADA Asset Download")
|
||||
print(f"{'=' * 60}")
|
||||
print(f"Asset: {label}")
|
||||
print(f"Repository: {repo_id}")
|
||||
print(f"Target: {target_dir}")
|
||||
print(f"Files: {len(files)}")
|
||||
print(f"{'=' * 60}\n")
|
||||
try:
|
||||
with quiet_hf_download_logs():
|
||||
snapshot_download(
|
||||
repo_id=repo_id,
|
||||
local_dir=target_dir,
|
||||
allow_patterns=files,
|
||||
force_download=force,
|
||||
)
|
||||
except Exception as exc:
|
||||
raise RuntimeError(f"Failed to download {label} from {repo_id}: {exc}") from exc
|
||||
|
||||
if not TadaDownloader._files_ready(target_dir, files):
|
||||
missing = [name for name in files if not os.path.isfile(os.path.join(target_dir, name))]
|
||||
raise RuntimeError(
|
||||
f"TADA asset download for {label} is incomplete. Missing files: {missing}"
|
||||
)
|
||||
print(f"✅ TADA asset ready: {target_dir}")
|
||||
return target_dir
|
||||
|
||||
def _search_roots(self) -> List[str]:
|
||||
roots = [self.base_path]
|
||||
try:
|
||||
roots.extend(os.path.join(path, "tada") for path in get_all_tts_model_paths("TTS"))
|
||||
except Exception:
|
||||
pass
|
||||
seen = set()
|
||||
return [path for path in roots if not (path in seen or seen.add(path))]
|
||||
|
||||
def _resolve_local_model(self, identifier: str) -> str:
|
||||
value = str(identifier).removeprefix("local:")
|
||||
if os.path.isdir(value):
|
||||
return os.path.abspath(value)
|
||||
for root in self._search_roots():
|
||||
candidate = os.path.join(root, value)
|
||||
if os.path.isdir(candidate):
|
||||
return candidate
|
||||
raise FileNotFoundError(
|
||||
f"Local TADA checkpoint '{value}' was not found under the configured TTS/tada paths."
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def is_model_complete(cls, model_path: str) -> bool:
|
||||
if not os.path.isfile(os.path.join(model_path, "config.json")):
|
||||
return False
|
||||
try:
|
||||
return any(
|
||||
name.endswith(".safetensors")
|
||||
and os.path.getsize(os.path.join(model_path, name)) > 0
|
||||
for name in os.listdir(model_path)
|
||||
)
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
def get_available_models(self) -> List[str]:
|
||||
models = list(self.MODELS)
|
||||
local_names = set()
|
||||
for root in self._search_roots():
|
||||
if not os.path.isdir(root):
|
||||
continue
|
||||
try:
|
||||
entries = os.listdir(root)
|
||||
except OSError:
|
||||
continue
|
||||
for entry in entries:
|
||||
candidate = os.path.join(root, entry)
|
||||
if entry not in self.MODELS and self.is_model_complete(candidate):
|
||||
local_names.add(f"local:{entry}")
|
||||
return models + sorted(local_names)
|
||||
|
||||
def ensure_model(self, model_identifier: str, force: bool = False) -> tuple[str, str]:
|
||||
canonical = self.canonical_model_name(model_identifier)
|
||||
if canonical is None:
|
||||
model_path = self._resolve_local_model(model_identifier)
|
||||
if not self.is_model_complete(model_path):
|
||||
raise RuntimeError(f"Local TADA checkpoint is incomplete: {model_path}")
|
||||
return os.path.basename(os.path.normpath(model_path)), model_path
|
||||
|
||||
details = self.MODELS[canonical]
|
||||
target = os.path.join(self.base_path, canonical)
|
||||
try:
|
||||
self._download_files(
|
||||
details["repo_id"], target, details["files"], canonical, force
|
||||
)
|
||||
except Exception as exc:
|
||||
raise RuntimeError(
|
||||
f"Unable to download the public {canonical} checkpoint. Check the network, "
|
||||
f"or manually place the required checkpoint files in "
|
||||
f"ComfyUI/models/TTS/tada/{canonical}/, "
|
||||
f"then retry. Download error: {exc}"
|
||||
) from exc
|
||||
return canonical, target
|
||||
|
||||
def ensure_tokenizer(self, force: bool = False) -> str:
|
||||
target = os.path.join(self.base_path, self.TOKENIZER_DIRECTORY)
|
||||
if not force and self._files_ready(target, self.TOKENIZER_FILES):
|
||||
self._validate_tokenizer_files(target)
|
||||
return target
|
||||
try:
|
||||
result = self._download_files(
|
||||
self.TOKENIZER_REPO,
|
||||
target,
|
||||
self.TOKENIZER_DISTRIBUTION_FILES,
|
||||
"Llama-3.2-1B tokenizer",
|
||||
force,
|
||||
)
|
||||
self._validate_tokenizer_files(result)
|
||||
return result
|
||||
except Exception as exc:
|
||||
raise RuntimeError(
|
||||
"Unable to download the TADA tokenizer automatically from the exact ungated "
|
||||
f"redistribution https://huggingface.co/{self.TOKENIZER_REPO}.\n\n"
|
||||
"Manual alternative: copy these three exact Meta Llama 3.2 1B files:\n"
|
||||
"ComfyUI/models/TTS/tada/\n"
|
||||
"└── llama-3.2-1b-tokenizer/\n"
|
||||
" ├── tokenizer.json\n"
|
||||
" ├── tokenizer_config.json\n"
|
||||
" └── special_tokens_map.json\n\n"
|
||||
f"Download error: {exc}"
|
||||
) from exc
|
||||
|
||||
def ensure_codec_assets(
|
||||
self,
|
||||
language: object = None,
|
||||
force: bool = False,
|
||||
codec_path: Optional[str] = None,
|
||||
) -> str:
|
||||
code = normalize_tada_language(language)
|
||||
target = os.path.abspath(codec_path or os.path.join(self.base_path, self.CODEC_DIRECTORY))
|
||||
aligner_dir = "aligner" if code is None else f"aligner-{code}"
|
||||
codec_files = self.CODEC_BASE_FILES + [
|
||||
f"{aligner_dir}/config.json",
|
||||
f"{aligner_dir}/model.safetensors",
|
||||
]
|
||||
self._download_files(self.CODEC_REPO, target, codec_files, "TADA codec", force)
|
||||
|
||||
wav2vec_file = os.path.join(target, self.WAV2VEC_CONFIG_LOCAL)
|
||||
if force or not os.path.isfile(wav2vec_file) or os.path.getsize(wav2vec_file) == 0:
|
||||
self._download_files(
|
||||
self.WAV2VEC_REPO,
|
||||
os.path.dirname(wav2vec_file),
|
||||
["config.json"],
|
||||
"TADA Wav2Vec aligner configuration",
|
||||
force,
|
||||
)
|
||||
return target
|
||||
|
||||
def resolve_model_assets(
|
||||
self,
|
||||
model_identifier: str,
|
||||
language: object = None,
|
||||
force: bool = False,
|
||||
) -> Dict[str, str]:
|
||||
canonical = self.canonical_model_name(model_identifier)
|
||||
validation_name = canonical or os.path.basename(str(model_identifier))
|
||||
validate_tada_model_language(validation_name, language)
|
||||
# Validate the small tokenizer first so source/hash failures happen
|
||||
# before several gigabytes of TADA assets are downloaded.
|
||||
tokenizer_path = self.ensure_tokenizer(force=force)
|
||||
model_name, model_path = self.ensure_model(model_identifier, force=force)
|
||||
codec_path = self.ensure_codec_assets(language=language, force=force)
|
||||
return {
|
||||
"model_name": model_name,
|
||||
"model_path": model_path,
|
||||
"codec_path": codec_path,
|
||||
"tokenizer_path": tokenizer_path,
|
||||
}
|
||||
|
||||
|
||||
tada_downloader = TadaDownloader()
|
||||
|
||||
|
||||
__all__ = ["TadaDownloader", "tada_downloader"]
|
||||
@@ -0,0 +1,567 @@
|
||||
"""Local-only wrapper around the official ``hume-tada`` inference classes."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import gc
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
from collections import OrderedDict
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import fields
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional, Tuple
|
||||
from unittest.mock import patch
|
||||
|
||||
import torch
|
||||
|
||||
from .languages import normalize_tada_language, validate_tada_model_language
|
||||
|
||||
|
||||
TADA_SAMPLE_RATE = 24000
|
||||
|
||||
|
||||
class TadaEngine:
|
||||
"""Official TADA inference with every model dependency resolved locally."""
|
||||
|
||||
_CFG_SCHEDULES = {"constant", "linear", "cosine"}
|
||||
_TIME_SCHEDULES = {"uniform", "cosine", "logsnr"}
|
||||
_NEGATIVE_SOURCES = {"negative_step_output", "prompt", "zero"}
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model_name: str,
|
||||
model_path: str,
|
||||
codec_path: str,
|
||||
tokenizer_path: str,
|
||||
device: str = "auto",
|
||||
dtype: str = "auto",
|
||||
attn_implementation: str = "sdpa",
|
||||
use_torch_compile: bool = False,
|
||||
prompt_cache_size: int = 8,
|
||||
):
|
||||
self.model_name = str(model_name)
|
||||
self.model_path = os.path.abspath(str(model_path))
|
||||
self.codec_path = os.path.abspath(str(codec_path))
|
||||
self.tokenizer_path = os.path.abspath(str(tokenizer_path))
|
||||
self.device = self._resolve_device(device)
|
||||
self.dtype = self._resolve_dtype(dtype)
|
||||
self.attn_implementation = str(attn_implementation or "sdpa")
|
||||
self.use_torch_compile = bool(use_torch_compile)
|
||||
self.prompt_cache_size = max(0, int(prompt_cache_size))
|
||||
self._prompt_cache: OrderedDict[str, Dict[str, Any]] = OrderedDict()
|
||||
self.model = None
|
||||
|
||||
self._validate_local_assets()
|
||||
self._load_model()
|
||||
|
||||
@staticmethod
|
||||
def _resolve_device(device: object) -> torch.device:
|
||||
value = str(device or "auto")
|
||||
if value == "auto":
|
||||
value = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
if value.startswith("cuda") and not torch.cuda.is_available():
|
||||
raise RuntimeError("TADA was configured for CUDA, but CUDA is not available in its shared runtime.")
|
||||
return torch.device(value)
|
||||
|
||||
def _resolve_dtype(self, dtype: object) -> torch.dtype:
|
||||
value = str(dtype or "auto").lower()
|
||||
if value == "auto":
|
||||
if self.device.type == "cuda":
|
||||
return torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
|
||||
return torch.float32
|
||||
mapping = {
|
||||
"bfloat16": torch.bfloat16,
|
||||
"bf16": torch.bfloat16,
|
||||
"float16": torch.float16,
|
||||
"fp16": torch.float16,
|
||||
"float32": torch.float32,
|
||||
"fp32": torch.float32,
|
||||
}
|
||||
if value not in mapping:
|
||||
raise ValueError(f"Unsupported TADA dtype '{dtype}'")
|
||||
if self.device.type == "cpu" and mapping[value] == torch.float16:
|
||||
return torch.float32
|
||||
return mapping[value]
|
||||
|
||||
@staticmethod
|
||||
def _require_files(root: str, relative_paths: Tuple[str, ...], label: str) -> None:
|
||||
missing = [name for name in relative_paths if not os.path.isfile(os.path.join(root, name))]
|
||||
if missing:
|
||||
raise RuntimeError(f"Local {label} assets are incomplete at {root}. Missing: {missing}")
|
||||
|
||||
def _validate_local_assets(self) -> None:
|
||||
model_files = ("config.json",)
|
||||
tokenizer_files = ("tokenizer.json", "tokenizer_config.json", "special_tokens_map.json")
|
||||
codec_files = (
|
||||
"encoder/config.json",
|
||||
"encoder/model.safetensors",
|
||||
"decoder/config.json",
|
||||
"decoder/model.safetensors",
|
||||
"wav2vec2-large/config.json",
|
||||
)
|
||||
self._require_files(self.model_path, model_files, "TADA checkpoint")
|
||||
if not any(Path(self.model_path).glob("*.safetensors")):
|
||||
raise RuntimeError(f"Local TADA checkpoint has no safetensors weights: {self.model_path}")
|
||||
self._require_files(self.tokenizer_path, tokenizer_files, "Llama tokenizer")
|
||||
self._require_files(self.codec_path, codec_files, "TADA codec")
|
||||
|
||||
def _load_model(self) -> None:
|
||||
try:
|
||||
import transformers
|
||||
from tada.modules.decoder import Decoder
|
||||
from tada.modules.tada import TadaForCausalLM
|
||||
from transformers import AutoTokenizer, LlamaForCausalLM
|
||||
except Exception as exc:
|
||||
raise RuntimeError(
|
||||
"TADA requires the shared Transformers 4 runtime with hume-tada installed. "
|
||||
"Select the Shared Runtime option and retry. "
|
||||
f"Import error: {exc}"
|
||||
) from exc
|
||||
|
||||
major = int(str(transformers.__version__).split(".", 1)[0])
|
||||
if major >= 5:
|
||||
raise RuntimeError(
|
||||
f"TADA is not compatible with Transformers {transformers.__version__} in the main runtime. "
|
||||
"Use the vibevoice_transformers4_shared runtime profile."
|
||||
)
|
||||
|
||||
load_kwargs = {
|
||||
"local_files_only": True,
|
||||
"low_cpu_mem_usage": True,
|
||||
"dtype": self.dtype,
|
||||
}
|
||||
if self.attn_implementation not in {"", "auto"}:
|
||||
load_kwargs["attn_implementation"] = self.attn_implementation
|
||||
|
||||
try:
|
||||
# Calling the parent classmethod directly bypasses TADA's override,
|
||||
# which otherwise downloads its decoder and tokenizer by repository ID.
|
||||
class _ExpectedBundledDecoderWarning(logging.Filter):
|
||||
def filter(self, record):
|
||||
message = record.getMessage()
|
||||
return not (
|
||||
"were not used when initializing TadaForCausalLM" in message
|
||||
and "_decoder." in message
|
||||
)
|
||||
|
||||
warning_filter = _ExpectedBundledDecoderWarning()
|
||||
model_logger = logging.getLogger("transformers.modeling_utils")
|
||||
model_logger.addFilter(warning_filter)
|
||||
try:
|
||||
model, loading_info = LlamaForCausalLM.from_pretrained.__func__(
|
||||
TadaForCausalLM,
|
||||
self.model_path,
|
||||
output_loading_info=True,
|
||||
**load_kwargs,
|
||||
)
|
||||
finally:
|
||||
model_logger.removeFilter(warning_filter)
|
||||
|
||||
unexpected = loading_info.get("unexpected_keys", [])
|
||||
non_decoder = [key for key in unexpected if not key.startswith("_decoder.")]
|
||||
if non_decoder:
|
||||
raise RuntimeError(
|
||||
"TADA checkpoint contains unexpected non-decoder weights: "
|
||||
f"{non_decoder[:10]}"
|
||||
)
|
||||
if not unexpected:
|
||||
raise RuntimeError(
|
||||
"TADA checkpoint did not expose the expected bundled decoder keys. "
|
||||
"Its layout may be incompatible with this hume-tada runtime."
|
||||
)
|
||||
# This matches hume-tada's official from_pretrained contract: the
|
||||
# checkpoint's bundled _decoder keys are intentionally ignored and
|
||||
# the released tada-codec decoder is attached instead.
|
||||
decoder = Decoder.from_pretrained(
|
||||
self.codec_path,
|
||||
subfolder="decoder",
|
||||
local_files_only=True,
|
||||
low_cpu_mem_usage=True,
|
||||
)
|
||||
tokenizer = AutoTokenizer.from_pretrained(
|
||||
self.tokenizer_path,
|
||||
local_files_only=True,
|
||||
)
|
||||
except Exception as exc:
|
||||
raise RuntimeError(f"Failed to load TADA entirely from local assets: {exc}") from exc
|
||||
|
||||
model._decoder = decoder
|
||||
model._tokenizer = tokenizer
|
||||
model.to(self.device)
|
||||
model.eval()
|
||||
if self.use_torch_compile:
|
||||
self._enable_torch_compile(model)
|
||||
self.model = model
|
||||
|
||||
def _enable_torch_compile(self, model) -> None:
|
||||
if self.device.type != "cuda":
|
||||
print("⚠️ TADA: torch.compile is only enabled on CUDA; using standard inference.")
|
||||
self.use_torch_compile = False
|
||||
return
|
||||
try:
|
||||
import triton # noqa: F401
|
||||
except Exception as exc:
|
||||
print(f"⚠️ TADA: torch.compile requires a working Triton install; using standard inference. ({exc})")
|
||||
self.use_torch_compile = False
|
||||
return
|
||||
|
||||
original_forward = model.prediction_head.forward
|
||||
try:
|
||||
compiled_forward = torch.compile(original_forward, mode="default")
|
||||
except Exception as exc:
|
||||
print(f"⚠️ TADA: could not prepare torch.compile; using standard inference. ({exc})")
|
||||
self.use_torch_compile = False
|
||||
return
|
||||
|
||||
failed = False
|
||||
|
||||
def compile_with_fallback(*args, **kwargs):
|
||||
nonlocal failed
|
||||
try:
|
||||
return compiled_forward(*args, **kwargs)
|
||||
except Exception as exc:
|
||||
if not failed:
|
||||
print(f"⚠️ TADA: compiled refinement failed; falling back to standard inference. ({exc})")
|
||||
failed = True
|
||||
model.prediction_head.forward = original_forward
|
||||
self.use_torch_compile = False
|
||||
return original_forward(*args, **kwargs)
|
||||
|
||||
model.prediction_head.forward = compile_with_fallback
|
||||
print("✅ TADA: torch.compile enabled for audio refinement (first generation will compile).")
|
||||
|
||||
def _required_aligner_subfolder(self, language: object) -> str:
|
||||
code = validate_tada_model_language(self.model_name, language)
|
||||
return "aligner" if code is None else f"aligner-{code}"
|
||||
|
||||
def _load_encoder(self, language: object):
|
||||
from tada.modules import aligner as aligner_module
|
||||
from tada.modules.aligner import Aligner, AlignerConfig
|
||||
from tada.modules.encoder import Encoder, EncoderConfig
|
||||
from transformers import PreTrainedModel, Wav2Vec2Config
|
||||
|
||||
aligner_subfolder = self._required_aligner_subfolder(language)
|
||||
self._require_files(
|
||||
self.codec_path,
|
||||
(
|
||||
f"{aligner_subfolder}/config.json",
|
||||
f"{aligner_subfolder}/model.safetensors",
|
||||
),
|
||||
f"TADA {aligner_subfolder}",
|
||||
)
|
||||
|
||||
wav2vec_config = Wav2Vec2Config.from_pretrained(
|
||||
os.path.join(self.codec_path, "wav2vec2-large"),
|
||||
local_files_only=True,
|
||||
)
|
||||
|
||||
# Aligner.__init__ normally resolves both names over the network. Supply
|
||||
# the already-loaded local tokenizer and explicit local Wav2Vec config.
|
||||
with patch.object(
|
||||
aligner_module.AutoTokenizer,
|
||||
"from_pretrained",
|
||||
return_value=self.model.tokenizer,
|
||||
), patch.object(
|
||||
aligner_module.AutoConfig,
|
||||
"from_pretrained",
|
||||
return_value=wav2vec_config,
|
||||
):
|
||||
aligner = Aligner.from_pretrained(
|
||||
self.codec_path,
|
||||
subfolder=aligner_subfolder,
|
||||
config=AlignerConfig(),
|
||||
local_files_only=True,
|
||||
low_cpu_mem_usage=True,
|
||||
)
|
||||
|
||||
# Encoder overrides from_pretrained and would construct another remotely
|
||||
# resolved aligner. Invoke the Transformers base loader and attach ours.
|
||||
encoder = PreTrainedModel.from_pretrained.__func__(
|
||||
Encoder,
|
||||
self.codec_path,
|
||||
subfolder="encoder",
|
||||
config=EncoderConfig(),
|
||||
local_files_only=True,
|
||||
low_cpu_mem_usage=True,
|
||||
)
|
||||
encoder._aligner = aligner
|
||||
encoder.to(self.device)
|
||||
encoder.eval()
|
||||
return encoder
|
||||
|
||||
@staticmethod
|
||||
def _normalize_reference(reference_audio: object) -> Tuple[torch.Tensor, int]:
|
||||
import torchaudio
|
||||
|
||||
sample_rate = TADA_SAMPLE_RATE
|
||||
audio = reference_audio
|
||||
|
||||
if isinstance(audio, (str, os.PathLike)):
|
||||
waveform, sample_rate = torchaudio.load(str(audio))
|
||||
audio = waveform
|
||||
elif isinstance(audio, (tuple, list)) and len(audio) == 2:
|
||||
audio, sample_rate = audio
|
||||
elif isinstance(audio, dict):
|
||||
if "audio" in audio and isinstance(audio["audio"], dict):
|
||||
audio = audio["audio"]
|
||||
if isinstance(audio, dict):
|
||||
path = audio.get("audio_path") or audio.get("path")
|
||||
if path:
|
||||
waveform, sample_rate = torchaudio.load(str(path))
|
||||
audio = waveform
|
||||
else:
|
||||
sample_rate = audio.get("sample_rate", audio.get("sampling_rate", sample_rate))
|
||||
audio = audio.get("waveform", audio.get("samples"))
|
||||
|
||||
if audio is None:
|
||||
raise ValueError("TADA requires reference audio; no waveform or audio path was provided.")
|
||||
waveform = audio.detach().cpu() if isinstance(audio, torch.Tensor) else torch.as_tensor(audio)
|
||||
waveform = waveform.float()
|
||||
if waveform.ndim == 0 or waveform.numel() == 0:
|
||||
raise ValueError("TADA reference audio is empty.")
|
||||
if waveform.ndim >= 3:
|
||||
waveform = waveform[0]
|
||||
if waveform.ndim == 2:
|
||||
waveform = waveform.mean(dim=0)
|
||||
elif waveform.ndim > 2:
|
||||
waveform = waveform.reshape(-1, waveform.shape[-1]).mean(dim=0)
|
||||
waveform = waveform.reshape(1, -1).contiguous()
|
||||
if not torch.isfinite(waveform).all():
|
||||
raise ValueError("TADA reference audio contains NaN or infinite values.")
|
||||
peak = waveform.abs().max()
|
||||
if float(peak) <= 0.0:
|
||||
raise ValueError("TADA reference audio is silent.")
|
||||
waveform = waveform / peak
|
||||
sample_rate = int(sample_rate)
|
||||
if sample_rate <= 0:
|
||||
raise ValueError(f"Invalid TADA reference sample rate: {sample_rate}")
|
||||
return waveform, sample_rate
|
||||
|
||||
@staticmethod
|
||||
def _prompt_state_to_device(state: Dict[str, Any], device: torch.device):
|
||||
from tada.modules.encoder import EncoderOutput
|
||||
|
||||
restored = {}
|
||||
for name, value in state.items():
|
||||
restored[name] = value.to(device) if isinstance(value, torch.Tensor) else value
|
||||
return EncoderOutput(**restored)
|
||||
|
||||
@staticmethod
|
||||
def _prompt_state_to_cpu(prompt: object) -> Dict[str, Any]:
|
||||
state = {}
|
||||
for field in fields(prompt):
|
||||
value = getattr(prompt, field.name)
|
||||
state[field.name] = value.detach().cpu() if isinstance(value, torch.Tensor) else value
|
||||
return state
|
||||
|
||||
@staticmethod
|
||||
def _prompt_cache_key(
|
||||
waveform: torch.Tensor,
|
||||
sample_rate: int,
|
||||
transcript: str,
|
||||
language: Optional[str],
|
||||
) -> str:
|
||||
digest = hashlib.sha256()
|
||||
digest.update(waveform.contiguous().numpy().tobytes())
|
||||
digest.update(str(sample_rate).encode("ascii"))
|
||||
digest.update(transcript.encode("utf-8"))
|
||||
digest.update(str(language).encode("ascii"))
|
||||
return digest.hexdigest()
|
||||
|
||||
def _get_prompt(
|
||||
self,
|
||||
reference_audio: object,
|
||||
reference_text: str,
|
||||
language: object,
|
||||
):
|
||||
waveform, sample_rate = self._normalize_reference(reference_audio)
|
||||
language_code = normalize_tada_language(language)
|
||||
key = self._prompt_cache_key(waveform, sample_rate, reference_text, language_code)
|
||||
cached = self._prompt_cache.pop(key, None)
|
||||
if cached is not None:
|
||||
self._prompt_cache[key] = cached
|
||||
return self._prompt_state_to_device(cached, self.device)
|
||||
|
||||
fork_devices = []
|
||||
if self.device.type == "cuda":
|
||||
fork_devices = [self.device.index if self.device.index is not None else torch.cuda.current_device()]
|
||||
with torch.random.fork_rng(devices=fork_devices):
|
||||
# Prompt sampling is intentionally stable and independent from the
|
||||
# user generation seed. A cold and warm prompt cache therefore
|
||||
# produce identical generation RNG state.
|
||||
torch.manual_seed(0)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed_all(0)
|
||||
encoder = self._load_encoder(language_code)
|
||||
try:
|
||||
prompt = encoder(
|
||||
waveform.to(self.device),
|
||||
text=[reference_text],
|
||||
sample_rate=sample_rate,
|
||||
)
|
||||
state = self._prompt_state_to_cpu(prompt)
|
||||
finally:
|
||||
del encoder
|
||||
gc.collect()
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
if self.prompt_cache_size > 0:
|
||||
self._prompt_cache[key] = state
|
||||
while len(self._prompt_cache) > self.prompt_cache_size:
|
||||
self._prompt_cache.popitem(last=False)
|
||||
return self._prompt_state_to_device(state, self.device)
|
||||
|
||||
@contextmanager
|
||||
def _generation_progress(
|
||||
self,
|
||||
text: str,
|
||||
num_flow_matching_steps: int,
|
||||
num_transition_steps: int,
|
||||
two_pass: bool,
|
||||
):
|
||||
"""Estimate total work from TADA's tokenizer and count real flow iterations."""
|
||||
from tqdm.auto import tqdm
|
||||
|
||||
original_compute_velocity = self.model._compute_velocity
|
||||
target_tokens = len(self.model.tokenizer.encode(text, add_special_tokens=False))
|
||||
# The official generator pre-fills the reference prompt, leaving target
|
||||
# text, transition, EOS, and a few structural positions to synthesize.
|
||||
estimated_generation_steps = max(
|
||||
1,
|
||||
target_tokens + int(num_transition_steps) + int(self.model.num_eos_tokens) + 4,
|
||||
)
|
||||
estimated_total = estimated_generation_steps * int(num_flow_matching_steps)
|
||||
if two_pass:
|
||||
estimated_total *= 2
|
||||
progress = tqdm(
|
||||
total=estimated_total,
|
||||
desc="TADA flow",
|
||||
unit="it",
|
||||
dynamic_ncols=True,
|
||||
mininterval=0.5,
|
||||
leave=True,
|
||||
)
|
||||
|
||||
def tracked_compute_velocity(*args, **kwargs):
|
||||
result = original_compute_velocity(*args, **kwargs)
|
||||
progress.update(1)
|
||||
return result
|
||||
|
||||
try:
|
||||
with patch.object(self.model, "_compute_velocity", tracked_compute_velocity):
|
||||
yield
|
||||
finally:
|
||||
# EOS can make the estimate short or long. Finish at the real count
|
||||
# instead of leaving a successful generation below/above 100%.
|
||||
progress.total = progress.n
|
||||
progress.close()
|
||||
|
||||
def generate_speech(
|
||||
self,
|
||||
text: str,
|
||||
reference_audio: object,
|
||||
reference_text: str,
|
||||
seed: int = 42,
|
||||
language: object = "English",
|
||||
acoustic_cfg_scale: float = 1.6,
|
||||
duration_cfg_scale: float = 1.0,
|
||||
cfg_schedule: str = "cosine",
|
||||
time_schedule: str = "logsnr",
|
||||
num_flow_matching_steps: int = 10,
|
||||
noise_temperature: float = 0.9,
|
||||
speed_up_factor: Optional[float] = None,
|
||||
num_transition_steps: int = 5,
|
||||
negative_condition_source: str = "negative_step_output",
|
||||
) -> Tuple[torch.Tensor, int]:
|
||||
if self.model is None:
|
||||
raise RuntimeError("TADA is not initialized.")
|
||||
if not str(text or "").strip():
|
||||
raise ValueError("TADA synthesis text cannot be empty.")
|
||||
if not str(reference_text or "").strip():
|
||||
raise ValueError(
|
||||
"TADA requires the exact transcript of the reference audio. "
|
||||
"Automatic transcription is intentionally disabled."
|
||||
)
|
||||
cfg_schedule = str(cfg_schedule).lower()
|
||||
time_schedule = str(time_schedule).lower()
|
||||
negative_condition_source = str(negative_condition_source).lower()
|
||||
if cfg_schedule not in self._CFG_SCHEDULES:
|
||||
raise ValueError(f"Invalid TADA CFG schedule: {cfg_schedule}")
|
||||
if time_schedule not in self._TIME_SCHEDULES:
|
||||
raise ValueError(f"Invalid TADA time schedule: {time_schedule}")
|
||||
if negative_condition_source not in self._NEGATIVE_SOURCES:
|
||||
raise ValueError(f"Invalid TADA negative condition source: {negative_condition_source}")
|
||||
if int(num_flow_matching_steps) < 1:
|
||||
raise ValueError("TADA flow matching steps must be at least 1.")
|
||||
if int(num_transition_steps) < 0:
|
||||
raise ValueError("TADA transition steps cannot be negative.")
|
||||
if speed_up_factor is not None and float(speed_up_factor) <= 0:
|
||||
raise ValueError("TADA speed_up_factor must be positive or None.")
|
||||
|
||||
from tada.modules.tada import InferenceOptions
|
||||
|
||||
seed = int(seed)
|
||||
torch.manual_seed(seed)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed_all(seed)
|
||||
|
||||
prompt = self._get_prompt(reference_audio, str(reference_text), language)
|
||||
options = InferenceOptions(
|
||||
acoustic_cfg_scale=float(acoustic_cfg_scale),
|
||||
duration_cfg_scale=float(duration_cfg_scale),
|
||||
cfg_schedule=cfg_schedule,
|
||||
noise_temperature=float(noise_temperature),
|
||||
num_flow_matching_steps=int(num_flow_matching_steps),
|
||||
time_schedule=time_schedule,
|
||||
speed_up_factor=None if speed_up_factor is None else float(speed_up_factor),
|
||||
negative_condition_source=negative_condition_source,
|
||||
)
|
||||
with self._generation_progress(
|
||||
text=str(text),
|
||||
num_flow_matching_steps=int(num_flow_matching_steps),
|
||||
num_transition_steps=int(num_transition_steps),
|
||||
two_pass=speed_up_factor is not None,
|
||||
):
|
||||
output = self.model.generate(
|
||||
prompt=prompt,
|
||||
text=str(text),
|
||||
num_transition_steps=int(num_transition_steps),
|
||||
inference_options=options,
|
||||
)
|
||||
audio = output.audio[0] if output.audio else None
|
||||
if audio is None:
|
||||
raise RuntimeError("TADA generation completed without a decodable audio result.")
|
||||
waveform = audio.detach().float().cpu() if isinstance(audio, torch.Tensor) else torch.as_tensor(audio).float()
|
||||
waveform = waveform.squeeze()
|
||||
if waveform.ndim > 1:
|
||||
waveform = waveform.reshape(-1, waveform.shape[-1]).mean(dim=0)
|
||||
waveform = waveform.contiguous()
|
||||
if waveform.numel() == 0 or not torch.isfinite(waveform).all():
|
||||
raise RuntimeError("TADA produced an empty or invalid waveform.")
|
||||
return waveform, TADA_SAMPLE_RATE
|
||||
|
||||
def to(self, device):
|
||||
self.device = self._resolve_device(device)
|
||||
if self.model is not None:
|
||||
self.model.to(self.device)
|
||||
return self
|
||||
|
||||
def eval(self):
|
||||
if self.model is not None:
|
||||
self.model.eval()
|
||||
return self
|
||||
|
||||
def cleanup(self) -> None:
|
||||
self._prompt_cache.clear()
|
||||
model = self.model
|
||||
self.model = None
|
||||
if model is not None:
|
||||
del model
|
||||
gc.collect()
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
|
||||
__all__ = ["TADA_SAMPLE_RATE", "TadaEngine"]
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1015 KiB |
@@ -179,6 +179,14 @@ except Exception as e:
|
||||
print(f"❌ Dots TTS Engine failed: {e}")
|
||||
DOTS_TTS_ENGINE_AVAILABLE = False
|
||||
|
||||
try:
|
||||
tada_engine_module = load_node_module("tada_engine_node", "engines/tada_engine_node.py")
|
||||
TadaEngineNode = tada_engine_module.TadaEngineNode
|
||||
TADA_ENGINE_AVAILABLE = True
|
||||
except Exception as e:
|
||||
print(f"❌ TADA Engine failed: {e}")
|
||||
TADA_ENGINE_AVAILABLE = False
|
||||
|
||||
try:
|
||||
dramabox_engine_module = load_node_module("dramabox_engine_node", "engines/dramabox_engine_node.py")
|
||||
DramaBoxEngineNode = dramabox_engine_module.DramaBoxEngineNode
|
||||
@@ -670,6 +678,10 @@ if DOTS_TTS_ENGINE_AVAILABLE:
|
||||
NODE_CLASS_MAPPINGS["DotsTTSEngineNode"] = DotsTTSEngineNode
|
||||
NODE_DISPLAY_NAME_MAPPINGS["DotsTTSEngineNode"] = "⚙️ Dots TTS Engine"
|
||||
|
||||
if TADA_ENGINE_AVAILABLE:
|
||||
NODE_CLASS_MAPPINGS["TadaEngineNode"] = TadaEngineNode
|
||||
NODE_DISPLAY_NAME_MAPPINGS["TadaEngineNode"] = "⚙️ TADA Engine"
|
||||
|
||||
if DRAMABOX_ENGINE_AVAILABLE:
|
||||
NODE_CLASS_MAPPINGS["DramaBoxEngineNode"] = DramaBoxEngineNode
|
||||
NODE_DISPLAY_NAME_MAPPINGS["DramaBoxEngineNode"] = "⚙️ DramaBox Engine"
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
"""TADA engine configuration node."""
|
||||
|
||||
import importlib.util
|
||||
import os
|
||||
import sys
|
||||
|
||||
current_dir = os.path.dirname(__file__)
|
||||
nodes_dir = os.path.dirname(current_dir)
|
||||
project_root = os.path.dirname(nodes_dir)
|
||||
if project_root not in sys.path:
|
||||
sys.path.insert(0, project_root)
|
||||
|
||||
base_node_path = os.path.join(nodes_dir, "base", "base_node.py")
|
||||
base_spec = importlib.util.spec_from_file_location("base_node_module", base_node_path)
|
||||
base_module = importlib.util.module_from_spec(base_spec)
|
||||
sys.modules["base_node_module"] = base_module
|
||||
base_spec.loader.exec_module(base_module)
|
||||
BaseTTSNode = base_module.BaseTTSNode
|
||||
|
||||
from engines.tada.languages import TADA_LANGUAGE_OPTIONS, validate_tada_model_language
|
||||
from engines.tada.tada_downloader import TadaDownloader
|
||||
from utils.models.factory_config import (
|
||||
RUNTIME_MODE_SHARED,
|
||||
normalize_runtime_mode,
|
||||
)
|
||||
|
||||
|
||||
RUNTIME_MODE_MAIN_LABEL = "Main Environment"
|
||||
RUNTIME_MODE_SHARED_LABEL = "⚠️ Shared Runtime"
|
||||
|
||||
|
||||
class TadaEngineNode(BaseTTSNode):
|
||||
"""Configure official Hume TADA zero-shot voice cloning."""
|
||||
|
||||
RUNTIME_PROFILE = "vibevoice_transformers4_shared"
|
||||
|
||||
@classmethod
|
||||
def NAME(cls):
|
||||
return "⚙️ TADA Engine"
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
try:
|
||||
model_options = TadaDownloader().get_available_models()
|
||||
except Exception:
|
||||
model_options = ["TADA-1B", "TADA-3B-ML"]
|
||||
|
||||
return {
|
||||
"required": {
|
||||
"model_variant": (model_options, {
|
||||
"default": "TADA-1B",
|
||||
"tooltip": "Choose the TADA model.\n"
|
||||
"• TADA-1B: faster, English only (about 3.9 GB).\n"
|
||||
"• TADA-3B-ML: multilingual (about 8.9 GB).\n"
|
||||
"Non-English characters automatically use 3B-ML. Missing files download on first use.",
|
||||
}),
|
||||
"device": (["auto", "cuda", "cpu"], {
|
||||
"default": "auto",
|
||||
"tooltip": "Choose where TADA runs.\n"
|
||||
"• Auto: recommended; uses an NVIDIA GPU when available.\n"
|
||||
"• CUDA: force GPU use.\n"
|
||||
"• CPU: works without a GPU, but is extremely slow.",
|
||||
}),
|
||||
"language": (list(TADA_LANGUAGE_OPTIONS), {
|
||||
"default": "English",
|
||||
"tooltip": "Language spoken in the reference clip and generated text.\n"
|
||||
"This improves pronunciation and voice alignment.\n"
|
||||
"Character language tags can change it automatically for each speaker.",
|
||||
}),
|
||||
"acoustic_cfg_scale": ("FLOAT", {
|
||||
"default": 1.6,
|
||||
"min": 0.0,
|
||||
"max": 5.0,
|
||||
"step": 0.05,
|
||||
"tooltip": "How strongly TADA follows the voice conditioning.\n"
|
||||
"1.6 is recommended. Higher can sound stronger but less natural.\n"
|
||||
"1.0 disables guidance and is faster.",
|
||||
}),
|
||||
"duration_cfg_scale": ("FLOAT", {
|
||||
"default": 1.0,
|
||||
"min": 0.0,
|
||||
"max": 5.0,
|
||||
"step": 0.05,
|
||||
"tooltip": "How strongly guidance affects pauses and sound duration.\n"
|
||||
"Keep 1.0 for natural automatic timing.\n"
|
||||
"Higher values can produce unusual pacing.",
|
||||
}),
|
||||
"num_flow_matching_steps": ("INT", {
|
||||
"default": 10,
|
||||
"min": 1,
|
||||
"max": 64,
|
||||
"step": 1,
|
||||
"tooltip": "Number of audio refinement steps.\n"
|
||||
"10 is recommended. Fewer is faster but may reduce quality.\n"
|
||||
"More is slower and may not improve the result.",
|
||||
}),
|
||||
"noise_temperature": ("FLOAT", {
|
||||
"default": 0.9,
|
||||
"min": 0.0,
|
||||
"max": 2.0,
|
||||
"step": 0.05,
|
||||
"tooltip": "Controls variation in the generated voice.\n"
|
||||
"0.9 is recommended. Lower is more predictable.\n"
|
||||
"Higher adds variation but may create unstable sounds.",
|
||||
}),
|
||||
},
|
||||
"optional": {
|
||||
"dtype": (["auto", "bfloat16", "float16", "float32"], {
|
||||
"default": "auto",
|
||||
"tooltip": "Controls model precision and memory use.\n"
|
||||
"Auto is recommended. Float16 and bfloat16 use less GPU memory.\n"
|
||||
"Float32 uses much more memory and is mainly for troubleshooting.",
|
||||
}),
|
||||
"cfg_schedule": (["cosine", "linear", "constant"], {
|
||||
"default": "cosine",
|
||||
"tooltip": "How guidance changes during audio refinement.\n"
|
||||
"• Cosine: smooth reduction; recommended.\n"
|
||||
"• Linear: even reduction.\n"
|
||||
"• Constant: full guidance throughout.",
|
||||
}),
|
||||
"time_schedule": (["logsnr", "cosine", "uniform"], {
|
||||
"default": "logsnr",
|
||||
"tooltip": "Controls where refinement steps are concentrated.\n"
|
||||
"• LogSNR: recommended.\n"
|
||||
"• Cosine: more work near the beginning and end.\n"
|
||||
"• Uniform: steps are evenly spaced.",
|
||||
}),
|
||||
"negative_condition_source": (["negative_step_output", "prompt", "zero"], {
|
||||
"default": "negative_step_output",
|
||||
"tooltip": "Baseline used by voice guidance.\n"
|
||||
"• negative_step_output: recommended; stronger but uses more compute.\n"
|
||||
"• prompt: uses the reference prompt.\n"
|
||||
"• zero: simplest and fastest baseline.",
|
||||
}),
|
||||
"speed_up_factor": ("FLOAT", {
|
||||
"default": 0.0,
|
||||
"min": 0.0,
|
||||
"max": 2.0,
|
||||
"step": 0.05,
|
||||
"tooltip": "Changes speaking speed using TADA's native method.\n"
|
||||
"0 disables it; values above 1 speak faster.\n"
|
||||
"Enabling it performs two passes and takes about twice as long.",
|
||||
}),
|
||||
"num_transition_steps": ("INT", {
|
||||
"default": 5,
|
||||
"min": 0,
|
||||
"max": 32,
|
||||
"step": 1,
|
||||
"tooltip": "Smooths the change from the reference clip to generated speech.\n"
|
||||
"5 is recommended. Lower is more abrupt.\n"
|
||||
"Higher is smoother but adds work.",
|
||||
}),
|
||||
"use_torch_compile": ("BOOLEAN", {
|
||||
"default": False,
|
||||
"tooltip": "Compile TADA's audio refinement stage for faster repeated generation.\n"
|
||||
"The first run is slower while compiling; later runs reuse a persistent cache.\n"
|
||||
"Uses extra disk space and may use slightly more GPU memory.",
|
||||
}),
|
||||
"runtime_mode": ([RUNTIME_MODE_SHARED_LABEL, RUNTIME_MODE_MAIN_LABEL], {
|
||||
"default": RUNTIME_MODE_SHARED_LABEL,
|
||||
"tooltip": "Choose the Python environment used by TADA.\n"
|
||||
"• Shared Runtime: recommended; uses the compatible Transformers 4 environment.\n"
|
||||
"• Main Environment: only works if ComfyUI already has compatible TADA dependencies.",
|
||||
}),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("TTS_ENGINE",)
|
||||
RETURN_NAMES = ("TTS_engine",)
|
||||
FUNCTION = "create_engine_config"
|
||||
CATEGORY = "TTS Audio Suite/⚙️ Engines"
|
||||
|
||||
def create_engine_config(
|
||||
self,
|
||||
model_variant: str,
|
||||
device: str,
|
||||
language: str,
|
||||
acoustic_cfg_scale: float,
|
||||
duration_cfg_scale: float,
|
||||
num_flow_matching_steps: int,
|
||||
noise_temperature: float,
|
||||
dtype: str = "auto",
|
||||
cfg_schedule: str = "cosine",
|
||||
time_schedule: str = "logsnr",
|
||||
negative_condition_source: str = "negative_step_output",
|
||||
speed_up_factor: float = 0.0,
|
||||
num_transition_steps: int = 5,
|
||||
use_torch_compile: bool = False,
|
||||
runtime_mode: str = RUNTIME_MODE_SHARED_LABEL,
|
||||
) -> tuple:
|
||||
validate_tada_model_language(model_variant, language)
|
||||
runtime_mode = normalize_runtime_mode(runtime_mode)
|
||||
runtime_profile = self.RUNTIME_PROFILE if runtime_mode == RUNTIME_MODE_SHARED else None
|
||||
config = {
|
||||
"engine_type": "tada",
|
||||
"model_variant": model_variant,
|
||||
"model_name": model_variant,
|
||||
"model_path": model_variant,
|
||||
"device": device,
|
||||
"dtype": dtype,
|
||||
"use_torch_compile": bool(use_torch_compile),
|
||||
"attn_implementation": "sdpa",
|
||||
"language": language,
|
||||
"acoustic_cfg_scale": float(acoustic_cfg_scale),
|
||||
"duration_cfg_scale": float(duration_cfg_scale),
|
||||
"num_flow_matching_steps": int(num_flow_matching_steps),
|
||||
"noise_temperature": float(noise_temperature),
|
||||
"cfg_schedule": cfg_schedule,
|
||||
"time_schedule": time_schedule,
|
||||
"negative_condition_source": negative_condition_source,
|
||||
"speed_up_factor": float(speed_up_factor),
|
||||
"num_transition_steps": int(num_transition_steps),
|
||||
"runtime_mode": runtime_mode,
|
||||
"runtime_profile": runtime_profile,
|
||||
}
|
||||
|
||||
print(f"⚙️ TADA: Configured {model_variant} on {device} ({language})")
|
||||
runtime_label = "Shared Transformers-4" if runtime_mode == RUNTIME_MODE_SHARED else "Main Environment"
|
||||
print(
|
||||
f" Runtime: {runtime_label} | dtype={dtype} | "
|
||||
f"torch.compile={'on' if use_torch_compile else 'off'}"
|
||||
)
|
||||
print(
|
||||
" Settings: "
|
||||
f"acoustic_cfg={acoustic_cfg_scale}, duration_cfg={duration_cfg_scale}, "
|
||||
f"flow_steps={num_flow_matching_steps}, noise={noise_temperature}"
|
||||
)
|
||||
if speed_up_factor > 0:
|
||||
print(f" Native two-pass speed factor: {speed_up_factor}")
|
||||
|
||||
return ({
|
||||
"engine_type": "tada",
|
||||
"config": config,
|
||||
"capabilities": ["tts"],
|
||||
},)
|
||||
|
||||
|
||||
__all__ = ["TadaEngineNode"]
|
||||
@@ -0,0 +1,2 @@
|
||||
"""TADA TTS processors."""
|
||||
|
||||
@@ -0,0 +1,328 @@
|
||||
"""TADA text processor for chunking, character voices, pauses, and tags."""
|
||||
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from typing import Any, Dict, List, Tuple, Union
|
||||
|
||||
import comfy.model_management as model_management
|
||||
import torch
|
||||
|
||||
current_dir = os.path.dirname(__file__)
|
||||
nodes_dir = os.path.dirname(current_dir)
|
||||
project_root = os.path.dirname(nodes_dir)
|
||||
if project_root not in sys.path:
|
||||
sys.path.insert(0, project_root)
|
||||
|
||||
from engines.tada.languages import format_tada_language_display, normalize_tada_language
|
||||
from utils.audio.chunk_timing import ChunkTimingHelper
|
||||
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
|
||||
from utils.text.character_parser import character_parser
|
||||
from utils.text.pause_processor import PauseTagProcessor
|
||||
from utils.text.segment_parameters import ParameterValidator, apply_segment_parameters
|
||||
from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment
|
||||
from utils.voice.character_logging import format_resolved_character_block, resolved_character_label
|
||||
from utils.voice.discovery import get_available_characters, get_character_mapping, voice_discovery
|
||||
from utils.voice.reference import effective_voice_audio
|
||||
|
||||
|
||||
class TadaProcessor:
|
||||
"""Orchestrate standard TTS Audio Suite text features around raw TADA calls."""
|
||||
|
||||
SAMPLE_RATE = 24000
|
||||
|
||||
def __init__(self, adapter, engine_config: Dict[str, Any]):
|
||||
self.adapter = adapter
|
||||
self.config = engine_config.copy() if engine_config else {}
|
||||
|
||||
def update_config(self, new_config: Dict[str, Any]):
|
||||
self.config = new_config.copy() if new_config else {}
|
||||
self.adapter.update_config(self.config)
|
||||
|
||||
@staticmethod
|
||||
def _check_interrupt(context: str = "generation") -> None:
|
||||
if model_management.interrupt_processing:
|
||||
raise InterruptedError(f"TADA {context} interrupted by user")
|
||||
|
||||
@staticmethod
|
||||
def _reference_text(voice_ref: Dict[str, Any]) -> str:
|
||||
if not isinstance(voice_ref, dict):
|
||||
return ""
|
||||
return str(
|
||||
voice_ref.get("reference_text")
|
||||
or voice_ref.get("prompt_text")
|
||||
or voice_ref.get("text")
|
||||
or ""
|
||||
).strip()
|
||||
|
||||
def _validate_voice(self, character_name: str, voice_ref: Dict[str, Any]) -> None:
|
||||
if not isinstance(voice_ref, dict) or effective_voice_audio(voice_ref) is None:
|
||||
raise ValueError(
|
||||
f"TADA requires reference audio for '{character_name}'. Connect a narrator voice "
|
||||
"or save a character voice with audio and its exact transcript."
|
||||
)
|
||||
if not self._reference_text(voice_ref):
|
||||
raise ValueError(
|
||||
f"TADA requires the exact reference transcript for '{character_name}'. "
|
||||
"Use Character Voices or add the matching .txt/.reference.txt file. "
|
||||
"Automatic Parakeet transcription is intentionally disabled to prevent a hidden large-model download."
|
||||
)
|
||||
|
||||
def _setup_character_parser(self, text: str) -> None:
|
||||
language_code = normalize_tada_language(self.config.get("language", "English")) or "en"
|
||||
character_parser.language_resolver.default_language = language_code
|
||||
character_parser.default_language = language_code
|
||||
|
||||
text_characters = []
|
||||
for tag in re.findall(r"\[([^\]]+)\]", text or ""):
|
||||
if not tag.lower().startswith(("pause:", "wait:", "stop:")):
|
||||
text_characters.append(tag.split("|")[0].split(":")[-1].strip())
|
||||
|
||||
available = set(get_available_characters() or [])
|
||||
for alias, target in voice_discovery.get_character_aliases().items():
|
||||
available.add(alias.lower())
|
||||
available.add(target.lower())
|
||||
available.update(name.lower() for name in text_characters if name)
|
||||
available.add("narrator")
|
||||
character_parser.set_available_characters(list(available))
|
||||
|
||||
for character, language in voice_discovery.get_character_language_defaults().items():
|
||||
character_parser.set_character_language_default(character, language)
|
||||
character_parser.reset_session_cache()
|
||||
|
||||
def _segment_config(self, segment: Any, base_config: Dict[str, Any], seed: int):
|
||||
params = segment.parameters or {}
|
||||
filtered = ParameterValidator.filter_parameters_for_engine(params, "tada") if params else {}
|
||||
current = apply_segment_parameters(base_config, params, "tada") if params else base_config.copy()
|
||||
current_seed = int(current.get("seed", seed))
|
||||
|
||||
segment_language = getattr(segment, "language", None)
|
||||
base_language = normalize_tada_language(base_config.get("language", "English"))
|
||||
segment_language_code = (
|
||||
normalize_tada_language(segment_language) if segment_language else base_language
|
||||
)
|
||||
should_switch_language = bool(
|
||||
segment_language
|
||||
and (
|
||||
getattr(segment, "explicit_language", False)
|
||||
or segment_language_code != base_language
|
||||
)
|
||||
)
|
||||
if should_switch_language:
|
||||
current["language"] = format_tada_language_display(segment_language)
|
||||
print(f" 🌍 TADA language switched to: {current['language']}")
|
||||
return current, current_seed, filtered
|
||||
|
||||
@staticmethod
|
||||
def _parameter_log(filtered: Dict[str, Any], config: Dict[str, Any], seed: int) -> str:
|
||||
if not filtered:
|
||||
return ""
|
||||
values = []
|
||||
if "seed" in filtered:
|
||||
values.append(f"seed={seed}")
|
||||
keys = (
|
||||
"acoustic_cfg_scale",
|
||||
"duration_cfg_scale",
|
||||
"num_flow_matching_steps",
|
||||
"noise_temperature",
|
||||
"speed_up_factor",
|
||||
)
|
||||
for key in keys:
|
||||
if key in config and (
|
||||
key in filtered
|
||||
or (key == "acoustic_cfg_scale" and "cfg" in filtered)
|
||||
or (key == "num_flow_matching_steps" and "num_steps" in filtered)
|
||||
or (key == "speed_up_factor" and "speed" in filtered)
|
||||
):
|
||||
values.append(f"{key}={config[key]}")
|
||||
return ", ".join(values)
|
||||
|
||||
def process_text(
|
||||
self,
|
||||
text: str,
|
||||
voice_mapping: Dict[str, Any],
|
||||
seed: int,
|
||||
enable_chunking: bool = True,
|
||||
max_chars_per_chunk: int = 400,
|
||||
chunk_combination_method: str = "auto",
|
||||
silence_between_chunks_ms: int = 100,
|
||||
enable_audio_cache: bool = True,
|
||||
apply_edit_postprocessing: bool = True,
|
||||
show_text_logging: bool = True,
|
||||
) -> List[Dict[str, Any]]:
|
||||
del chunk_combination_method, silence_between_chunks_ms
|
||||
self._check_interrupt("text setup")
|
||||
self._setup_character_parser(text)
|
||||
|
||||
narrator = voice_mapping.get("narrator", {})
|
||||
narrator_voice = narrator.copy() if isinstance(narrator, dict) else {"audio": narrator}
|
||||
base_config = self.config.copy()
|
||||
segments = character_parser.parse_text_segments(text)
|
||||
if not segments:
|
||||
segments = character_parser.parse_text_segments("narrator " + text)
|
||||
|
||||
execution_config = base_config.copy()
|
||||
selected_model = str(
|
||||
execution_config.get("model_variant")
|
||||
or execution_config.get("model_name")
|
||||
or "TADA-1B"
|
||||
)
|
||||
segment_languages = {
|
||||
format_tada_language_display(segment.language)
|
||||
for segment in segments
|
||||
if getattr(segment, "language", None)
|
||||
and normalize_tada_language(segment.language) is not None
|
||||
}
|
||||
if "tada-1b" in selected_model.lower().replace("_", "-") and segment_languages:
|
||||
execution_config.update(
|
||||
{
|
||||
"model_variant": "TADA-3B-ML",
|
||||
"model_name": "TADA-3B-ML",
|
||||
"model_path": "TADA-3B-ML",
|
||||
}
|
||||
)
|
||||
languages = ", ".join(sorted(segment_languages))
|
||||
print(f"🌍 Non-English character language detected ({languages})")
|
||||
print("🔄 Switching TADA-1B → TADA-3B-ML for this generation")
|
||||
|
||||
characters = list({segment.character for segment in segments if segment.character})
|
||||
discovered_voices = get_character_mapping(characters, engine_type="tada")
|
||||
records: List[Dict[str, Any]] = []
|
||||
|
||||
try:
|
||||
for segment_index, segment in enumerate(segments):
|
||||
self._check_interrupt(f"text segment {segment_index + 1}/{len(segments)}")
|
||||
segment_text = (segment.text or "").strip()
|
||||
if not segment_text:
|
||||
continue
|
||||
|
||||
current_config, current_seed, filtered = self._segment_config(
|
||||
segment, execution_config, seed
|
||||
)
|
||||
self.adapter.update_config(current_config)
|
||||
character = segment.character or "narrator"
|
||||
voice_ref = narrator_voice.copy()
|
||||
|
||||
mapped_voice = voice_mapping.get(character)
|
||||
if character != "narrator" and isinstance(mapped_voice, dict):
|
||||
voice_ref = mapped_voice.copy()
|
||||
elif character != "narrator":
|
||||
audio_path, reference_text = discovered_voices.get(character, (None, None))
|
||||
if audio_path:
|
||||
voice_ref = {"audio_path": audio_path, "reference_text": reference_text or ""}
|
||||
else:
|
||||
discovered = voice_discovery.get_character_voice_info(
|
||||
character, engine_type="audio_only"
|
||||
)
|
||||
if discovered and discovered.get("audio_path") and not str(
|
||||
discovered.get("text_content") or ""
|
||||
).strip():
|
||||
raise ValueError(
|
||||
f"TADA found reference audio for '{character}', but it has no transcript. "
|
||||
"Add a matching .txt or .reference.txt file; TADA cannot use audio-only voices."
|
||||
)
|
||||
|
||||
self._validate_voice(character, voice_ref)
|
||||
|
||||
def generate_text(text_content: str, edit_tags: list) -> None:
|
||||
if enable_chunking:
|
||||
from utils.text.chunking import ImprovedChatterBoxChunker
|
||||
|
||||
max_chars = ImprovedChatterBoxChunker.validate_chunking_params(max_chars_per_chunk)
|
||||
chunks = ImprovedChatterBoxChunker.split_into_chunks(text_content, max_chars=max_chars)
|
||||
else:
|
||||
chunks = [text_content]
|
||||
|
||||
display_name = resolved_character_label(character, voice_ref)
|
||||
language = format_tada_language_display(current_config.get("language", "English"))
|
||||
print(f"🎭 TADA - Generating for '{display_name}' (Language: {language})")
|
||||
parameter_log = self._parameter_log(filtered, current_config, current_seed)
|
||||
if parameter_log:
|
||||
print(f"🎛️ TADA params: {parameter_log}")
|
||||
if show_text_logging:
|
||||
print(format_resolved_character_block(character, text_content, voice_ref))
|
||||
if len(chunks) > 1:
|
||||
print(f"📝 TADA chunking: {len(chunks)} chunks")
|
||||
|
||||
for chunk_index, chunk in enumerate(chunks):
|
||||
self._check_interrupt(
|
||||
f"chunk {chunk_index + 1}/{len(chunks)} for '{display_name}'"
|
||||
)
|
||||
audio = self.adapter.generate_single(
|
||||
text=chunk,
|
||||
voice_ref=voice_ref,
|
||||
seed=current_seed + chunk_index,
|
||||
enable_audio_cache=enable_audio_cache,
|
||||
character_name=character,
|
||||
)
|
||||
if audio.dim() == 1:
|
||||
audio = audio.unsqueeze(0)
|
||||
records.append(
|
||||
{
|
||||
"waveform": audio.detach().cpu().float(),
|
||||
"sample_rate": self.SAMPLE_RATE,
|
||||
"text": chunk,
|
||||
"edit_tags": edit_tags if chunk_index == 0 else [],
|
||||
}
|
||||
)
|
||||
|
||||
if PauseTagProcessor.has_pause_tags(segment_text):
|
||||
pause_segments, _ = PauseTagProcessor.parse_pause_tags(segment_text)
|
||||
for fragment_type, fragment in pause_segments:
|
||||
self._check_interrupt("pause-tag processing")
|
||||
if fragment_type == "text":
|
||||
clean_text, edit_tags = get_edit_tags_for_segment(fragment)
|
||||
if clean_text.strip():
|
||||
generate_text(clean_text.strip(), edit_tags)
|
||||
elif fragment_type == "pause":
|
||||
silence = PauseTagProcessor.create_silence_segment(
|
||||
fragment, self.SAMPLE_RATE, torch.device("cpu"), torch.float32
|
||||
)
|
||||
if silence.dim() == 1:
|
||||
silence = silence.unsqueeze(0)
|
||||
records.append(
|
||||
{
|
||||
"waveform": silence,
|
||||
"sample_rate": self.SAMPLE_RATE,
|
||||
"text": f"[pause:{fragment}s]",
|
||||
"edit_tags": [],
|
||||
}
|
||||
)
|
||||
else:
|
||||
clean_text, edit_tags = get_edit_tags_for_segment(segment_text)
|
||||
if clean_text.strip():
|
||||
generate_text(clean_text.strip(), edit_tags)
|
||||
finally:
|
||||
self.adapter.update_config(base_config)
|
||||
|
||||
if apply_edit_postprocessing and records and any(record["edit_tags"] for record in records):
|
||||
self._check_interrupt("edit post-processing")
|
||||
records = apply_edit_post_processing(records, engine_config=base_config)
|
||||
return records
|
||||
|
||||
def combine_audio_segments(
|
||||
self,
|
||||
segments: List[Dict[str, Any]],
|
||||
method: str = "auto",
|
||||
silence_ms: int = 100,
|
||||
original_text: str = "",
|
||||
return_info: bool = False,
|
||||
) -> Union[torch.Tensor, Tuple[torch.Tensor, Dict[str, Any]]]:
|
||||
if not segments:
|
||||
empty = torch.zeros(0, dtype=torch.float32)
|
||||
return (empty, {}) if return_info else empty
|
||||
|
||||
self._check_interrupt("audio assembly")
|
||||
waveforms = [segment["waveform"] for segment in segments]
|
||||
text_chunks = [segment.get("text", "") for segment in segments]
|
||||
combined, chunk_info = ChunkTimingHelper.combine_audio_with_timing(
|
||||
audio_segments=waveforms,
|
||||
combination_method=method,
|
||||
silence_ms=silence_ms,
|
||||
crossfade_duration=0.1,
|
||||
sample_rate=self.SAMPLE_RATE,
|
||||
text_length=len(" ".join(text_chunks)),
|
||||
original_text=original_text,
|
||||
text_chunks=text_chunks,
|
||||
)
|
||||
return (combined, chunk_info) if return_info else combined
|
||||
@@ -0,0 +1,254 @@
|
||||
"""SRT timing/orchestration for TADA using the normal TADA text processor."""
|
||||
|
||||
import importlib.util
|
||||
import os
|
||||
import sys
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
import comfy.model_management as model_management
|
||||
import torch
|
||||
|
||||
current_dir = os.path.dirname(__file__)
|
||||
nodes_dir = os.path.dirname(current_dir)
|
||||
project_root = os.path.dirname(nodes_dir)
|
||||
if project_root not in sys.path:
|
||||
sys.path.insert(0, project_root)
|
||||
|
||||
from engines.adapters.tada_adapter import TadaEngineAdapter
|
||||
|
||||
|
||||
class TadaSRTProcessor:
|
||||
"""Generate each subtitle through :class:`TadaProcessor`, then assemble timing."""
|
||||
|
||||
SAMPLE_RATE = 24000
|
||||
|
||||
def __init__(self, node_instance, config: Dict[str, Any]):
|
||||
self.node_instance = node_instance
|
||||
self.config = config.copy() if config else {}
|
||||
self.adapter = TadaEngineAdapter(self.config)
|
||||
self._tts_processor = None
|
||||
self.srt_available = False
|
||||
self.SRTParser = None
|
||||
self._load_srt_modules()
|
||||
|
||||
def _load_srt_modules(self) -> None:
|
||||
try:
|
||||
from utils.system.import_manager import import_manager
|
||||
|
||||
success, modules, message = import_manager.import_srt_modules()
|
||||
if not success:
|
||||
print(f"⚠️ TADA SRT: SRT module not available: {message}")
|
||||
return
|
||||
self.SRTParser = modules.get("SRTParser")
|
||||
self.srt_available = self.SRTParser is not None
|
||||
except Exception as exc:
|
||||
print(f"⚠️ TADA SRT: Failed to load SRT modules: {exc}")
|
||||
|
||||
@property
|
||||
def processor(self):
|
||||
if self._tts_processor is None:
|
||||
processor_path = os.path.join(current_dir, "tada_processor.py")
|
||||
spec = importlib.util.spec_from_file_location("tada_processor_module", processor_path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
self._tts_processor = module.TadaProcessor(self.adapter, self.config)
|
||||
return self._tts_processor
|
||||
|
||||
def update_config(self, new_config: Dict[str, Any]) -> None:
|
||||
self.config = new_config.copy() if new_config else {}
|
||||
self.adapter.update_config(self.config)
|
||||
if self._tts_processor is not None:
|
||||
self._tts_processor.update_config(self.config)
|
||||
|
||||
@staticmethod
|
||||
def _check_interrupt(index: Optional[int] = None, total: Optional[int] = None) -> None:
|
||||
if model_management.interrupt_processing:
|
||||
location = f" at subtitle {index + 1}/{total}" if index is not None and total else ""
|
||||
raise InterruptedError(f"TADA SRT generation interrupted{location}")
|
||||
|
||||
def process_srt_content(
|
||||
self,
|
||||
srt_content: str,
|
||||
voice_mapping: Dict[str, Any],
|
||||
seed: int,
|
||||
timing_mode: str,
|
||||
timing_params: Dict[str, Any],
|
||||
enable_audio_cache: bool = True,
|
||||
) -> Tuple[Dict[str, Any], str, str, str]:
|
||||
if not self.srt_available:
|
||||
raise ImportError("TADA SRT support is unavailable because the shared SRT parser did not load")
|
||||
|
||||
self._check_interrupt()
|
||||
subtitles = self.SRTParser().parse_srt_content(srt_content, allow_overlaps=True)
|
||||
|
||||
from utils.timing.overlap_detection import SRTOverlapHandler
|
||||
|
||||
has_overlaps = SRTOverlapHandler.detect_overlaps(subtitles)
|
||||
current_mode, mode_switched = SRTOverlapHandler.handle_smart_natural_fallback(
|
||||
timing_mode, has_overlaps, "TADA SRT"
|
||||
)
|
||||
print(f"🚀 TADA SRT: Processing {len(subtitles)} subtitles in {current_mode} mode")
|
||||
|
||||
audio_segments, adjustments = self._generate_subtitles(
|
||||
subtitles, voice_mapping, seed, enable_audio_cache
|
||||
)
|
||||
self._check_interrupt()
|
||||
final_audio, final_adjustments, stretch_method = self._assemble(
|
||||
audio_segments, subtitles, current_mode, timing_params
|
||||
)
|
||||
if final_adjustments is not None:
|
||||
adjustments = final_adjustments
|
||||
|
||||
from utils.timing.reporting import SRTReportGenerator
|
||||
|
||||
reporter = SRTReportGenerator()
|
||||
timing_report = reporter.generate_timing_report(
|
||||
subtitles,
|
||||
adjustments,
|
||||
current_mode,
|
||||
has_overlaps,
|
||||
mode_switched,
|
||||
timing_mode if mode_switched else None,
|
||||
stretch_method,
|
||||
)
|
||||
adjusted_srt = reporter.generate_adjusted_srt_string(subtitles, adjustments, current_mode)
|
||||
|
||||
if final_audio.dim() == 1:
|
||||
final_audio = final_audio.unsqueeze(0).unsqueeze(0)
|
||||
elif final_audio.dim() == 2:
|
||||
final_audio = final_audio.unsqueeze(0)
|
||||
|
||||
duration = final_audio.shape[-1] / self.SAMPLE_RATE
|
||||
mode_info = current_mode
|
||||
if mode_switched:
|
||||
mode_info = f"{current_mode} (switched from {timing_mode} due to overlaps)"
|
||||
info = (
|
||||
f"Generated {duration:.1f}s TADA SRT-timed audio from {len(subtitles)} subtitles "
|
||||
f"using {mode_info} mode"
|
||||
)
|
||||
return {"waveform": final_audio.cpu().float(), "sample_rate": self.SAMPLE_RATE}, info, timing_report, adjusted_srt
|
||||
|
||||
def _generate_subtitles(
|
||||
self,
|
||||
subtitles: List,
|
||||
voice_mapping: Dict[str, Any],
|
||||
seed: int,
|
||||
enable_audio_cache: bool,
|
||||
) -> Tuple[List[torch.Tensor], List[Dict[str, Any]]]:
|
||||
audio_segments: List[torch.Tensor] = []
|
||||
adjustments: List[Dict[str, Any]] = []
|
||||
try:
|
||||
from comfy.utils import ProgressBar
|
||||
|
||||
progress = ProgressBar(len(subtitles))
|
||||
except Exception:
|
||||
progress = None
|
||||
|
||||
for index, subtitle in enumerate(subtitles):
|
||||
self._check_interrupt(index, len(subtitles))
|
||||
text = (subtitle.text or "").strip()
|
||||
target_duration = subtitle.end_time - subtitle.start_time
|
||||
|
||||
if text:
|
||||
print(f"📖 TADA SRT Subtitle {index + 1}/{len(subtitles)}")
|
||||
records = self.processor.process_text(
|
||||
text=text,
|
||||
voice_mapping=voice_mapping,
|
||||
seed=seed + index,
|
||||
enable_chunking=False,
|
||||
enable_audio_cache=enable_audio_cache,
|
||||
apply_edit_postprocessing=True,
|
||||
)
|
||||
if len(records) > 1:
|
||||
audio, _ = self.processor.combine_audio_segments(
|
||||
records, method="auto", silence_ms=0, original_text=text, return_info=True
|
||||
)
|
||||
elif records:
|
||||
audio = records[0]["waveform"]
|
||||
else:
|
||||
audio = torch.zeros(1, 0)
|
||||
else:
|
||||
audio = torch.zeros(1, max(0, int(target_duration * self.SAMPLE_RATE)))
|
||||
|
||||
if audio.dim() == 1:
|
||||
audio = audio.unsqueeze(0)
|
||||
elif audio.dim() == 3:
|
||||
audio = audio.squeeze(0)
|
||||
audio = audio.detach().cpu().float()
|
||||
audio_segments.append(audio)
|
||||
|
||||
natural_duration = audio.shape[-1] / self.SAMPLE_RATE
|
||||
stretch_factor = target_duration / natural_duration if natural_duration > 0 else 1.0
|
||||
adjustments.append(
|
||||
{
|
||||
"index": index,
|
||||
"segment_index": index,
|
||||
"sequence": subtitle.sequence,
|
||||
"natural_duration": natural_duration,
|
||||
"target_start": subtitle.start_time,
|
||||
"target_end": subtitle.end_time,
|
||||
"target_duration": target_duration,
|
||||
"start_time": subtitle.start_time,
|
||||
"end_time": subtitle.end_time,
|
||||
"stretch_factor": stretch_factor,
|
||||
"needs_stretching": abs(stretch_factor - 1.0) > 0.05,
|
||||
"stretch_type": (
|
||||
"compress" if stretch_factor < 1.0 else "expand" if stretch_factor > 1.0 else "none"
|
||||
),
|
||||
"adjustment": natural_duration - target_duration,
|
||||
"adjusted_start": subtitle.start_time,
|
||||
"adjusted_end": subtitle.end_time,
|
||||
"adjusted_duration": natural_duration,
|
||||
}
|
||||
)
|
||||
print(
|
||||
f"✅ TADA Subtitle {index + 1}/{len(subtitles)}: "
|
||||
f"{natural_duration:.2f}s (target {target_duration:.2f}s)"
|
||||
)
|
||||
if progress is not None:
|
||||
progress.update(1)
|
||||
|
||||
return audio_segments, adjustments
|
||||
|
||||
def _assemble(self, audio_segments, subtitles, timing_mode, timing_params):
|
||||
self._check_interrupt()
|
||||
if timing_mode == "stretch_to_fit":
|
||||
from engines.chatterbox.audio_timing import TimedAudioAssembler
|
||||
|
||||
assembler = TimedAudioAssembler(self.SAMPLE_RATE)
|
||||
target_timings = [(subtitle.start_time, subtitle.end_time) for subtitle in subtitles]
|
||||
final_audio, stretch_method = assembler.assemble_timed_audio(
|
||||
audio_segments,
|
||||
target_timings,
|
||||
fade_duration=timing_params.get("fade_for_StretchToFit", 0.01),
|
||||
)
|
||||
return final_audio, None, stretch_method
|
||||
|
||||
from utils.timing.assembly import AudioAssemblyEngine
|
||||
|
||||
assembler = AudioAssemblyEngine(self.SAMPLE_RATE)
|
||||
if timing_mode == "pad_with_silence":
|
||||
return assembler.assemble_with_overlaps(audio_segments, subtitles, torch.device("cpu")), None, None
|
||||
|
||||
from utils.timing.engine import TimingEngine
|
||||
|
||||
timing_engine = TimingEngine(self.SAMPLE_RATE)
|
||||
if timing_mode == "concatenate":
|
||||
adjustments = timing_engine.calculate_concatenation_adjustments(audio_segments, subtitles)
|
||||
audio = assembler.assemble_concatenation(
|
||||
audio_segments, timing_params.get("fade_for_StretchToFit", 0.01)
|
||||
)
|
||||
return audio, adjustments, None
|
||||
|
||||
adjustments, processed = timing_engine.calculate_smart_timing_adjustments(
|
||||
audio_segments,
|
||||
subtitles,
|
||||
timing_params.get("timing_tolerance", 2.0),
|
||||
timing_params.get("max_stretch_ratio", 1.0),
|
||||
timing_params.get("min_stretch_ratio", 0.5),
|
||||
torch.device("cpu"),
|
||||
)
|
||||
audio = assembler.assemble_smart_natural(
|
||||
audio_segments, processed, adjustments, subtitles, torch.device("cpu")
|
||||
)
|
||||
return audio, adjustments, None
|
||||
@@ -223,6 +223,13 @@ Hello! This is unified SRT TTS with character switching.
|
||||
stable_params['optimize'] = config.get('optimize', False)
|
||||
stable_params['max_generate_length'] = config.get('max_generate_length', 500)
|
||||
|
||||
if engine_type == "tada":
|
||||
stable_params['model_variant'] = config.get('model_variant', 'TADA-1B')
|
||||
stable_params['dtype'] = config.get('dtype', 'auto')
|
||||
stable_params['attn_implementation'] = config.get('attn_implementation', 'sdpa')
|
||||
stable_params['runtime_mode'] = config.get('runtime_mode')
|
||||
stable_params['runtime_profile'] = config.get('runtime_profile')
|
||||
|
||||
if engine_type == "dramabox":
|
||||
stable_params['model_name'] = config.get('model_name', 'DramaBox')
|
||||
stable_params['precision'] = config.get('precision', 'auto')
|
||||
@@ -578,6 +585,37 @@ Hello! This is unified SRT TTS with character switching.
|
||||
}
|
||||
return engine_instance
|
||||
|
||||
elif engine_type == "tada":
|
||||
processor_path = os.path.join(nodes_dir, "tada", "tada_srt_processor.py")
|
||||
processor_spec = importlib.util.spec_from_file_location(
|
||||
"tada_srt_processor_module", processor_path
|
||||
)
|
||||
processor_module = importlib.util.module_from_spec(processor_spec)
|
||||
processor_spec.loader.exec_module(processor_module)
|
||||
TadaSRTProcessor = processor_module.TadaSRTProcessor
|
||||
|
||||
class TadaSRTWrapper:
|
||||
def __init__(self, cfg):
|
||||
self.config = cfg.copy()
|
||||
self.processor = TadaSRTProcessor(self, self.config)
|
||||
|
||||
def update_config(self, new_config):
|
||||
self.config = new_config.copy()
|
||||
self.processor.update_config(new_config)
|
||||
|
||||
def check_interrupt(self):
|
||||
if model_management.interrupt_processing:
|
||||
raise InterruptedError("TADA SRT processing interrupted by user")
|
||||
|
||||
engine_instance = TadaSRTWrapper(config)
|
||||
|
||||
import time
|
||||
self._cached_engine_instances[cache_key] = {
|
||||
'instance': engine_instance,
|
||||
'timestamp': time.time()
|
||||
}
|
||||
return engine_instance
|
||||
|
||||
elif engine_type == "dramabox":
|
||||
processor_path = os.path.join(
|
||||
nodes_dir, "dramabox", "dramabox_srt_processor.py"
|
||||
@@ -1109,10 +1147,11 @@ Hello! This is unified SRT TTS with character switching.
|
||||
batch_size: Number of parallel workers (0=sequential, 1+=streaming parallel)
|
||||
use_native_duration_targeting: Inject subtitle durations into supported engines during generation
|
||||
|
||||
Returns:
|
||||
Tuple of (audio_tensor, generation_info, timing_report, adjusted_srt)
|
||||
"""
|
||||
try:
|
||||
Returns:
|
||||
Tuple of (audio_tensor, generation_info, timing_report, adjusted_srt)
|
||||
"""
|
||||
engine_type = None
|
||||
try:
|
||||
# Apply Python 3.12 CUDNN compatibility fix before TTS generation
|
||||
from utils.comfyui_compatibility import ensure_python312_cudnn_fix
|
||||
ensure_python312_cudnn_fix()
|
||||
@@ -1175,6 +1214,11 @@ Hello! This is unified SRT TTS with character switching.
|
||||
"OmniVoice voice cloning requires reference text. "
|
||||
"Do not connect raw audio directly. Use Character Voices node or a narrator voice with a matching .reference.txt file."
|
||||
)
|
||||
if engine_type == "tada" and (audio_tensor is not None or audio_path) and not reference_text.strip():
|
||||
raise ValueError(
|
||||
"TADA requires the exact reference transcript. Do not connect raw audio directly. "
|
||||
"Use Character Voices or a narrator voice with a matching .reference.txt file."
|
||||
)
|
||||
|
||||
# Create proper engine SRT node instance to preserve ALL functionality
|
||||
engine_instance = self._create_proper_engine_node_instance(TTS_engine)
|
||||
@@ -1375,6 +1419,30 @@ Hello! This is unified SRT TTS with character switching.
|
||||
enable_audio_cache=enable_audio_cache
|
||||
)
|
||||
|
||||
elif engine_type == "tada":
|
||||
timing_params = {
|
||||
'fade_for_StretchToFit': fade_for_StretchToFit,
|
||||
'max_stretch_ratio': max_stretch_ratio,
|
||||
'min_stretch_ratio': min_stretch_ratio,
|
||||
'timing_tolerance': timing_tolerance,
|
||||
}
|
||||
voice_mapping = {}
|
||||
if audio_tensor is not None or audio_path:
|
||||
voice_mapping['narrator'] = {
|
||||
'audio': audio_tensor,
|
||||
'audio_path': audio_path,
|
||||
'reference_text': reference_text or '',
|
||||
}
|
||||
|
||||
result = engine_instance.processor.process_srt_content(
|
||||
srt_content=srt_content,
|
||||
voice_mapping=voice_mapping,
|
||||
seed=seed,
|
||||
timing_mode=timing_mode,
|
||||
timing_params=timing_params,
|
||||
enable_audio_cache=enable_audio_cache,
|
||||
)
|
||||
|
||||
elif engine_type == "dramabox":
|
||||
timing_params = {
|
||||
'fade_for_StretchToFit': fade_for_StretchToFit,
|
||||
@@ -1690,6 +1758,8 @@ Hello! This is unified SRT TTS with character switching.
|
||||
or "MOSS-TTSD Native Multi-Speaker Dialogue does not support this SRT input" in msg
|
||||
):
|
||||
raise
|
||||
if engine_type == "tada":
|
||||
raise
|
||||
if isinstance(e, InterruptedError):
|
||||
raise
|
||||
error_msg = f"❌ TTS SRT generation failed: {e}"
|
||||
|
||||
@@ -218,6 +218,13 @@ Back to the main narrator voice for the conclusion.""",
|
||||
stable_params['optimize'] = config.get('optimize', False)
|
||||
stable_params['max_generate_length'] = config.get('max_generate_length', 500)
|
||||
|
||||
if engine_type == "tada":
|
||||
stable_params['model_variant'] = config.get('model_variant', 'TADA-1B')
|
||||
stable_params['dtype'] = config.get('dtype', 'auto')
|
||||
stable_params['attn_implementation'] = config.get('attn_implementation', 'sdpa')
|
||||
stable_params['runtime_mode'] = config.get('runtime_mode')
|
||||
stable_params['runtime_profile'] = config.get('runtime_profile')
|
||||
|
||||
if engine_type == "dramabox":
|
||||
stable_params['model_name'] = config.get('model_name', 'DramaBox')
|
||||
stable_params['precision'] = config.get('precision', 'auto')
|
||||
@@ -580,6 +587,37 @@ Back to the main narrator voice for the conclusion.""",
|
||||
|
||||
return engine_instance
|
||||
|
||||
elif engine_type == "tada":
|
||||
from engines.adapters.tada_adapter import TadaEngineAdapter
|
||||
|
||||
processor_path = os.path.join(nodes_dir, "tada", "tada_processor.py")
|
||||
processor_spec = importlib.util.spec_from_file_location(
|
||||
"tada_processor_module", processor_path
|
||||
)
|
||||
processor_module = importlib.util.module_from_spec(processor_spec)
|
||||
processor_spec.loader.exec_module(processor_module)
|
||||
TadaProcessor = processor_module.TadaProcessor
|
||||
|
||||
class TadaWrapper:
|
||||
def __init__(self, cfg):
|
||||
self.config = cfg.copy()
|
||||
self.adapter = TadaEngineAdapter(self.config)
|
||||
self.processor = TadaProcessor(self.adapter, self.config)
|
||||
|
||||
def update_config(self, new_config):
|
||||
self.config = new_config.copy()
|
||||
self.adapter.update_config(new_config)
|
||||
self.processor.update_config(new_config)
|
||||
|
||||
engine_instance = TadaWrapper(config)
|
||||
|
||||
import time
|
||||
self._cached_engine_instances[cache_key] = {
|
||||
'instance': engine_instance,
|
||||
'timestamp': time.time()
|
||||
}
|
||||
return engine_instance
|
||||
|
||||
elif engine_type == "dramabox":
|
||||
from engines.adapters.dramabox_adapter import DramaBoxEngineAdapter
|
||||
processor_path = os.path.join(nodes_dir, "dramabox", "dramabox_processor.py")
|
||||
@@ -1059,6 +1097,11 @@ Back to the main narrator voice for the conclusion.""",
|
||||
"OmniVoice voice cloning requires reference text. "
|
||||
"Do not connect raw audio directly. Use Character Voices node or a narrator voice with a matching .reference.txt file."
|
||||
)
|
||||
if engine_type == "tada" and (audio_tensor is not None or audio_path) and not reference_text.strip():
|
||||
raise ValueError(
|
||||
"TADA requires the exact reference transcript. Do not connect raw audio directly. "
|
||||
"Use Character Voices or a narrator voice with a matching .reference.txt file."
|
||||
)
|
||||
|
||||
# Create proper engine node instance to preserve ALL functionality
|
||||
engine_instance = self._create_proper_engine_node_instance(TTS_engine)
|
||||
@@ -1520,6 +1563,49 @@ Back to the main narrator voice for the conclusion.""",
|
||||
formatted_audio = AudioProcessingUtils.format_for_comfyui(combined_audio, 48000)
|
||||
result = (formatted_audio, generation_info)
|
||||
|
||||
elif engine_type == "tada":
|
||||
import re
|
||||
from utils.audio.chunk_timing import ChunkTimingHelper
|
||||
|
||||
voice_mapping = {}
|
||||
if audio_tensor is not None or audio_path:
|
||||
voice_mapping['narrator'] = {
|
||||
'audio': audio_tensor,
|
||||
'audio_path': audio_path,
|
||||
'reference_text': reference_text or '',
|
||||
}
|
||||
|
||||
segment_records = engine_instance.processor.process_text(
|
||||
text=text,
|
||||
voice_mapping=voice_mapping,
|
||||
seed=seed,
|
||||
enable_chunking=enable_chunking,
|
||||
max_chars_per_chunk=max_chars_per_chunk,
|
||||
chunk_combination_method=chunk_combination_method,
|
||||
silence_between_chunks_ms=silence_between_chunks_ms,
|
||||
enable_audio_cache=enable_audio_cache,
|
||||
)
|
||||
combined_audio, chunk_info = engine_instance.processor.combine_audio_segments(
|
||||
segments=segment_records,
|
||||
method=chunk_combination_method,
|
||||
silence_ms=silence_between_chunks_ms,
|
||||
original_text=text,
|
||||
return_info=True,
|
||||
)
|
||||
|
||||
total_duration = combined_audio.shape[-1] / 24000.0 if combined_audio.numel() else 0.0
|
||||
clean_text = re.sub(r'\[.*?\]', '', text)
|
||||
base_info = (
|
||||
f"Generated {total_duration:.1f}s audio from {len(clean_text)} characters "
|
||||
f"(TADA, narrator: {char_display})"
|
||||
)
|
||||
base_info += "\n🎭 Character switching, expression transfer, and pause tags enabled"
|
||||
generation_info = ChunkTimingHelper.enhance_generation_info(
|
||||
f"✅ {base_info}", chunk_info
|
||||
)
|
||||
formatted_audio = AudioProcessingUtils.format_for_comfyui(combined_audio, 24000)
|
||||
result = (formatted_audio, generation_info)
|
||||
|
||||
elif engine_type == "dramabox":
|
||||
import re
|
||||
from utils.audio.chunk_timing import ChunkTimingHelper
|
||||
@@ -2084,6 +2170,8 @@ Back to the main narrator voice for the conclusion.""",
|
||||
raise
|
||||
if engine_type == "index_tts":
|
||||
raise
|
||||
if engine_type == "tada":
|
||||
raise
|
||||
if isinstance(e, InterruptedError):
|
||||
raise
|
||||
error_msg = f"❌ TTS Text generation failed: {e}"
|
||||
|
||||
+63
-1
@@ -413,6 +413,67 @@ class Qwen3TTSCacheKeyGenerator(CacheKeyGenerator):
|
||||
return hashlib.md5(cache_string.encode()).hexdigest()
|
||||
|
||||
|
||||
class TadaCacheKeyGenerator(CacheKeyGenerator):
|
||||
"""Cache key generator for the TADA engine."""
|
||||
|
||||
def generate_cache_key(self, **params) -> str:
|
||||
"""Include every TADA input that can change the generated waveform."""
|
||||
rounded_fields = {
|
||||
'acoustic_cfg_scale': params.get('acoustic_cfg_scale', 1.6),
|
||||
'duration_cfg_scale': params.get('duration_cfg_scale', 1.0),
|
||||
'noise_temperature': params.get('noise_temperature', 0.9),
|
||||
'speed_up_factor': params.get('speed_up_factor'),
|
||||
}
|
||||
for key, value in list(rounded_fields.items()):
|
||||
if isinstance(value, (int, float)):
|
||||
rounded_fields[key] = round(float(value), 3)
|
||||
|
||||
audio_component = (
|
||||
params.get('audio_component')
|
||||
or params.get('reference_audio_hash')
|
||||
or params.get('prompt_audio_hash')
|
||||
or ''
|
||||
)
|
||||
reference_text = (
|
||||
params.get('reference_text')
|
||||
or params.get('prompt_text')
|
||||
or params.get('ref_text')
|
||||
or ''
|
||||
)
|
||||
|
||||
cache_data = {
|
||||
'text': params.get('text', ''),
|
||||
'model_variant': params.get('model_variant', 'TADA-1B'),
|
||||
'language': params.get('language', 'English'),
|
||||
'audio_component': audio_component,
|
||||
'reference_text': reference_text,
|
||||
'seed': params.get('seed', 0),
|
||||
'acoustic_cfg_scale': rounded_fields['acoustic_cfg_scale'],
|
||||
'duration_cfg_scale': rounded_fields['duration_cfg_scale'],
|
||||
'noise_temperature': rounded_fields['noise_temperature'],
|
||||
'num_flow_matching_steps': params.get('num_flow_matching_steps', 10),
|
||||
'cfg_schedule': params.get('cfg_schedule', 'cosine'),
|
||||
'time_schedule': params.get('time_schedule', 'logsnr'),
|
||||
'negative_condition_source': params.get(
|
||||
'negative_condition_source', 'negative_step_output'
|
||||
),
|
||||
'speed_up_factor': rounded_fields['speed_up_factor'],
|
||||
'num_transition_steps': params.get('num_transition_steps', 5),
|
||||
'device': params.get('device', 'auto'),
|
||||
'dtype': params.get('dtype', 'auto'),
|
||||
'attn_implementation': params.get('attn_implementation', 'sdpa'),
|
||||
'runtime_mode': params.get('runtime_mode', params.get('runtime', 'shared_runtime')),
|
||||
'runtime_profile': params.get(
|
||||
'runtime_profile', 'vibevoice_transformers4_shared'
|
||||
),
|
||||
'character': params.get('character', 'narrator'),
|
||||
'engine': 'tada',
|
||||
}
|
||||
|
||||
cache_string = str(sorted(cache_data.items()))
|
||||
return hashlib.md5(cache_string.encode()).hexdigest()
|
||||
|
||||
|
||||
class MossTTSCacheKeyGenerator(CacheKeyGenerator):
|
||||
"""Cache key generator for MOSS-TTS engine."""
|
||||
|
||||
@@ -709,6 +770,7 @@ class AudioCache:
|
||||
'index_tts': IndexTTSCacheKeyGenerator(),
|
||||
'cosyvoice': CosyVoiceCacheKeyGenerator(),
|
||||
'qwen3_tts': Qwen3TTSCacheKeyGenerator(),
|
||||
'tada': TadaCacheKeyGenerator(),
|
||||
'dots_tts': DotsTTSCacheKeyGenerator(),
|
||||
'dramabox': DramaBoxCacheKeyGenerator(),
|
||||
'fish_audio_s2': FishAudioS2CacheKeyGenerator(),
|
||||
@@ -788,7 +850,7 @@ class AudioCache:
|
||||
# Use engine-specific sample rates
|
||||
if engine_type in ('dots_tts', 'dramabox', 'moss_soundeffect_v2'):
|
||||
sample_rate = 48000
|
||||
elif engine_type in ('f5tts', 'step_audio_editx', 'qwen3_tts', 'moss_tts', 'higgs_audio_v3', 'omnivoice'):
|
||||
elif engine_type in ('f5tts', 'step_audio_editx', 'qwen3_tts', 'tada', 'moss_tts', 'higgs_audio_v3', 'omnivoice'):
|
||||
sample_rate = 24000
|
||||
elif engine_type in ('index_tts', 'cosyvoice'):
|
||||
sample_rate = 22050
|
||||
|
||||
@@ -5,7 +5,9 @@ Centralized downloading for all models (F5-TTS, ChatterBox, RVC, etc.) without c
|
||||
|
||||
import os
|
||||
import requests
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from typing import Optional, Dict, Any, List
|
||||
from pathlib import Path
|
||||
import fnmatch
|
||||
@@ -34,6 +36,28 @@ class UnifiedDownloader:
|
||||
if not self.verify_ssl:
|
||||
print("⚠️ SSL certificate verification is DISABLED")
|
||||
print(" Set environment variable TTS_DISABLE_SSL_VERIFY=0 to re-enable")
|
||||
|
||||
@staticmethod
|
||||
def _resolve_cli_executable(command: str) -> str:
|
||||
"""Resolve console scripts installed beside the active Python executable."""
|
||||
discovered = shutil.which(command)
|
||||
if discovered:
|
||||
return discovered
|
||||
|
||||
suffix = ".exe" if os.name == "nt" else ""
|
||||
executable_name = command if command.endswith(suffix) else f"{command}{suffix}"
|
||||
python_dir = Path(sys.executable).resolve().parent
|
||||
candidates = (
|
||||
python_dir / executable_name,
|
||||
python_dir / "Scripts" / executable_name,
|
||||
python_dir.parent / "Scripts" / executable_name,
|
||||
python_dir / "bin" / executable_name,
|
||||
python_dir.parent / "bin" / executable_name,
|
||||
)
|
||||
for candidate in candidates:
|
||||
if candidate.is_file():
|
||||
return str(candidate)
|
||||
return command
|
||||
|
||||
def download_from_hf_cli(
|
||||
self,
|
||||
@@ -63,6 +87,7 @@ class UnifiedDownloader:
|
||||
|
||||
try:
|
||||
os.makedirs(target_dir, exist_ok=True)
|
||||
print(f"📥 Downloading {filename} from {repo_id}")
|
||||
|
||||
if force_download and os.path.exists(target_path):
|
||||
try:
|
||||
@@ -74,10 +99,10 @@ class UnifiedDownloader:
|
||||
env = os.environ.copy()
|
||||
env['PYTHONIOENCODING'] = 'utf-8'
|
||||
|
||||
# Try modern 'hf download' first, fallback to legacy 'huggingface-cli download'
|
||||
# Show progress bar by not capturing output
|
||||
# Try modern 'hf download' first, then the legacy CLI. Capture their
|
||||
# noisy path/warning output and present consistent suite-level status.
|
||||
hf_command = [
|
||||
"hf", "download",
|
||||
self._resolve_cli_executable("hf"), "download",
|
||||
repo_id,
|
||||
filename,
|
||||
"--local-dir", target_dir
|
||||
@@ -90,16 +115,20 @@ class UnifiedDownloader:
|
||||
hf_command,
|
||||
check=False,
|
||||
env=env,
|
||||
capture_output=False # Allow progress bar to show
|
||||
capture_output=True,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
)
|
||||
|
||||
# If new command succeeded, return
|
||||
if result.returncode == 0 and os.path.exists(target_path):
|
||||
print(f"✅ Downloaded: {target_path}")
|
||||
return True
|
||||
|
||||
# Try legacy command if new one failed
|
||||
legacy_command = [
|
||||
"huggingface-cli", "download",
|
||||
self._resolve_cli_executable("huggingface-cli"), "download",
|
||||
repo_id,
|
||||
filename,
|
||||
"--local-dir", target_dir
|
||||
@@ -112,10 +141,14 @@ class UnifiedDownloader:
|
||||
legacy_command,
|
||||
check=False,
|
||||
env=env,
|
||||
capture_output=False # Allow progress bar to show
|
||||
capture_output=True,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
)
|
||||
|
||||
if result.returncode == 0 and os.path.exists(target_path):
|
||||
print(f"✅ Downloaded: {target_path}")
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
@@ -150,6 +150,15 @@ ENGINE_REGISTRY: Dict[str, EngineCapabilities] = {
|
||||
default_runtime_profile="vibevoice_transformers4_shared",
|
||||
),
|
||||
|
||||
"tada": EngineCapabilities(
|
||||
supports_voice_conversion=False,
|
||||
multilingual_model_switching=False,
|
||||
can_corrupt_on_reload=False,
|
||||
fallback_languages=["English"],
|
||||
supports_runtime_isolation=True,
|
||||
default_runtime_profile="vibevoice_transformers4_shared",
|
||||
),
|
||||
|
||||
"dots_tts": EngineCapabilities(
|
||||
supports_voice_conversion=False,
|
||||
multilingual_model_switching=False,
|
||||
|
||||
@@ -19,6 +19,7 @@ from utils.runtimes import (
|
||||
build_qwen3_asr_isolated_proxy,
|
||||
build_qwen3_tts_isolated_proxy,
|
||||
build_step_audio_editx_isolated_proxy,
|
||||
build_tada_isolated_proxy,
|
||||
build_vibevoice_isolated_proxy,
|
||||
)
|
||||
from utils.models.comfyui_model_wrapper.cache_utils import invalidate_all_caches
|
||||
@@ -148,7 +149,7 @@ class UnifiedModelInterface:
|
||||
# check if a DIFFERENT variant is already loaded and unload it to prevent device conflicts
|
||||
# Only applies to engines where model variants are mutually exclusive
|
||||
if config.engine_name in (
|
||||
"qwen3_tts", "moss_tts", "higgs_audio_v3", "dramabox"
|
||||
"qwen3_tts", "moss_tts", "higgs_audio_v3", "dramabox", "tada"
|
||||
):
|
||||
# Check for any cached mutually-exclusive model variant for this engine
|
||||
cached_prefix = f"{config.engine_name}_tts_"
|
||||
@@ -239,6 +240,11 @@ class UnifiedModelInterface:
|
||||
self._isolated_model_cache[cache_key] = proxy
|
||||
return proxy
|
||||
|
||||
if config.engine_name == "tada" and config.model_type == "tts":
|
||||
proxy = build_tada_isolated_proxy(config)
|
||||
self._isolated_model_cache[cache_key] = proxy
|
||||
return proxy
|
||||
|
||||
if config.engine_name == "fish_audio_s2" and config.model_type == "tts":
|
||||
proxy = build_fish_audio_s2_proxy(config)
|
||||
self._isolated_model_cache[cache_key] = proxy
|
||||
@@ -1723,6 +1729,27 @@ def register_fish_audio_s2_factory():
|
||||
unified_model_interface.register_model_factory("fish_audio_s2", "tts", fish_audio_s2_factory)
|
||||
|
||||
|
||||
def register_tada_factory():
|
||||
"""Register TADA for explicit Main Environment routing."""
|
||||
def tada_factory(config: ModelLoadConfig):
|
||||
from engines.tada.tada_engine import TadaEngine
|
||||
|
||||
params = config.additional_params or {}
|
||||
return TadaEngine(
|
||||
model_name=config.model_name or "TADA-1B",
|
||||
model_path=config.model_path,
|
||||
codec_path=params.get("codec_path"),
|
||||
tokenizer_path=params.get("tokenizer_path"),
|
||||
device=config.device or "auto",
|
||||
dtype=params.get("dtype", "auto"),
|
||||
attn_implementation=params.get("attn_implementation", "sdpa"),
|
||||
use_torch_compile=bool(params.get("use_torch_compile", False)),
|
||||
prompt_cache_size=int(params.get("prompt_cache_size", 8)),
|
||||
)
|
||||
|
||||
unified_model_interface.register_model_factory("tada", "tts", tada_factory)
|
||||
|
||||
|
||||
def register_dots_tts_factory():
|
||||
"""Register Dots TTS model factory."""
|
||||
def dots_tts_factory(config: ModelLoadConfig):
|
||||
@@ -2017,6 +2044,7 @@ def initialize_all_factories():
|
||||
register_moss_soundeffect_v2_factory()
|
||||
register_qwen3_tts_factory()
|
||||
register_fish_audio_s2_factory()
|
||||
register_tada_factory()
|
||||
register_dots_tts_factory()
|
||||
register_dramabox_factory()
|
||||
register_omnivoice_factory()
|
||||
|
||||
@@ -15,6 +15,7 @@ from .qwen3_asr_proxy import Qwen3ASRIsolatedProxy, build_qwen3_asr_isolated_pro
|
||||
from .qwen3_tts_proxy import Qwen3TTSIsolatedProxy, build_qwen3_tts_isolated_proxy
|
||||
from .vibevoice_proxy import VibeVoiceIsolatedProxy, build_vibevoice_isolated_proxy
|
||||
from .step_audio_editx_proxy import StepAudioEditXIsolatedProxy, build_step_audio_editx_isolated_proxy
|
||||
from .tada_proxy import TadaIsolatedProxy, build_tada_isolated_proxy
|
||||
|
||||
__all__ = [
|
||||
"ensure_runtime",
|
||||
@@ -36,6 +37,8 @@ __all__ = [
|
||||
"build_vibevoice_isolated_proxy",
|
||||
"StepAudioEditXIsolatedProxy",
|
||||
"build_step_audio_editx_isolated_proxy",
|
||||
"TadaIsolatedProxy",
|
||||
"build_tada_isolated_proxy",
|
||||
"get_runtime_profile",
|
||||
"list_runtime_profiles",
|
||||
]
|
||||
|
||||
@@ -29,6 +29,7 @@ _VIBEVOICE_T4_PACKAGES = [
|
||||
"soundfile>=0.12.0",
|
||||
"omegaconf>=2.3.0",
|
||||
"transformers>=4.51.3,<=4.57.3",
|
||||
"huggingface-hub>=0.34,<1.0",
|
||||
"kernels>=0.6.1,<=0.9",
|
||||
"accelerate",
|
||||
"requests",
|
||||
@@ -63,13 +64,14 @@ _QWEN3_T4_PACKAGES = [
|
||||
RUNTIME_PROFILES: Dict[str, RuntimeProfile] = {
|
||||
"vibevoice_transformers4_shared": RuntimeProfile(
|
||||
name="vibevoice_transformers4_shared",
|
||||
engine_names=["vibevoice", "step_audio_editx"],
|
||||
engine_names=["vibevoice", "step_audio_editx", "tada"],
|
||||
python_path_hint="runtimes/shared_legacy_t4/Scripts/python.exe",
|
||||
description="Shared legacy Transformers 4 runtime for VibeVoice/Kugel, Step Audio EditX, and similar engines.",
|
||||
description="Shared legacy Transformers 4 runtime for VibeVoice/Kugel, Step Audio EditX, TADA, and similar engines.",
|
||||
inherit_base_site_packages=True,
|
||||
pip_packages=list(_VIBEVOICE_T4_PACKAGES),
|
||||
pip_packages_no_deps=[
|
||||
"git+https://github.com/FushionHub/VibeVoice.git",
|
||||
"hume-tada==0.1.9",
|
||||
],
|
||||
),
|
||||
"vibevoice_transformers4_dedicated": RuntimeProfile(
|
||||
|
||||
@@ -0,0 +1,380 @@
|
||||
from __future__ import annotations
|
||||
|
||||
"""ComfyUI-side proxy for TADA in the shared Transformers 4 runtime."""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import uuid
|
||||
import weakref
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional, Tuple
|
||||
|
||||
import torch
|
||||
|
||||
from engines.tada.languages import validate_tada_model_language
|
||||
from engines.tada.tada_downloader import TadaDownloader
|
||||
from utils.device import resolve_torch_device
|
||||
from .bootstrap import PROJECT_ROOT, ensure_runtime
|
||||
from .launcher import IsolatedRuntimeLauncher
|
||||
from .profiles import RuntimeProfile, get_runtime_profile
|
||||
from .protocol import RuntimeJobRequest
|
||||
from .session import JsonLineWorkerSession
|
||||
|
||||
|
||||
TADA_RUNTIME_PROFILE = "vibevoice_transformers4_shared"
|
||||
|
||||
|
||||
class TadaIsolatedProxy:
|
||||
"""Persistent TADA worker with explicit cleanup for ComfyUI's Clear VRAM action."""
|
||||
|
||||
def __init__(self, config, profile: RuntimeProfile):
|
||||
if profile.name != TADA_RUNTIME_PROFILE:
|
||||
raise RuntimeError(
|
||||
f"TADA only supports the '{TADA_RUNTIME_PROFILE}' shared runtime; got '{profile.name}'."
|
||||
)
|
||||
|
||||
self.config = config
|
||||
self.profile = profile
|
||||
self.current_model_name = str(config.model_name or "TADA-1B")
|
||||
self.device = resolve_torch_device(str(config.device or "auto"))
|
||||
self.current_device = self.device
|
||||
self.load_device = self.current_loaded_device()
|
||||
self.offload_device = torch.device("cpu")
|
||||
self.dtype = torch.bfloat16
|
||||
self.model = self
|
||||
self.parent = None
|
||||
self.currently_used = True
|
||||
self.model_options = {}
|
||||
self.model_keys = set()
|
||||
self.sample_rate = 24000
|
||||
|
||||
params = dict(config.additional_params or {})
|
||||
self.model_path = str(params.get("model_path") or config.model_path or "")
|
||||
self.codec_path = str(params.get("codec_path") or "")
|
||||
self.tokenizer_path = str(params.get("tokenizer_path") or "")
|
||||
self.load_dtype = str(params.get("dtype", "auto"))
|
||||
self.attn_implementation = str(params.get("attn_implementation", "sdpa"))
|
||||
self.use_torch_compile = bool(params.get("use_torch_compile", False))
|
||||
self.prompt_cache_size = int(params.get("prompt_cache_size", 8))
|
||||
|
||||
if not self.model_path or not self.codec_path or not self.tokenizer_path:
|
||||
assets = TadaDownloader().resolve_model_assets(self.current_model_name, language=None)
|
||||
self.current_model_name = assets["model_name"]
|
||||
self.model_path = assets["model_path"]
|
||||
self.codec_path = assets["codec_path"]
|
||||
self.tokenizer_path = assets["tokenizer_path"]
|
||||
|
||||
estimate_gb = 10 if "3b" in self.current_model_name.lower() else 6
|
||||
self._estimated_memory_size = estimate_gb * 1024**3
|
||||
self._initialized = False
|
||||
self._comfy_loaded_model = None
|
||||
self._comfy_model_management = None
|
||||
|
||||
launcher = IsolatedRuntimeLauncher(runtime_root=str(PROJECT_ROOT))
|
||||
worker_env = launcher.build_env(profile)
|
||||
worker_env["HF_HUB_OFFLINE"] = "1"
|
||||
worker_env["TRANSFORMERS_OFFLINE"] = "1"
|
||||
worker_env.setdefault("PYTHONUTF8", "1")
|
||||
self.compile_cache_dir = self._configure_compile_cache(worker_env)
|
||||
python_path = ensure_runtime(profile)
|
||||
self._session = JsonLineWorkerSession(
|
||||
python_path=str(python_path),
|
||||
worker_script=str(PROJECT_ROOT / "utils" / "runtimes" / "workers" / "tada_worker.py"),
|
||||
env=worker_env,
|
||||
)
|
||||
self._initialize_remote_engine()
|
||||
self._register_with_comfy_model_management()
|
||||
|
||||
def _configure_compile_cache(self, worker_env: Dict[str, str]) -> Optional[Path]:
|
||||
if not self.use_torch_compile:
|
||||
return None
|
||||
try:
|
||||
model_dir = Path(self.model_path).resolve()
|
||||
cache_root = model_dir.parent.parent / "compile_cache"
|
||||
inductor_dir = cache_root / "torchinductor"
|
||||
triton_dir = cache_root / "triton"
|
||||
inductor_dir.mkdir(parents=True, exist_ok=True)
|
||||
triton_dir.mkdir(parents=True, exist_ok=True)
|
||||
worker_env["TORCHINDUCTOR_FX_GRAPH_CACHE"] = "1"
|
||||
worker_env["TORCHINDUCTOR_CACHE_DIR"] = str(inductor_dir)
|
||||
worker_env["TRITON_CACHE_DIR"] = str(triton_dir)
|
||||
print(f" TADA compile cache: {cache_root}")
|
||||
return cache_root
|
||||
except Exception as exc:
|
||||
print(f"⚠️ TADA: failed to configure persistent compile cache: {exc}")
|
||||
return None
|
||||
|
||||
def _request(self, action: str, payload: Optional[Dict[str, Any]] = None):
|
||||
response = self._session.request(
|
||||
RuntimeJobRequest(
|
||||
engine_name="tada",
|
||||
action=action,
|
||||
model_name=self.current_model_name,
|
||||
device=self.device,
|
||||
runtime_profile=self.profile.name,
|
||||
payload=payload or {},
|
||||
request_id=str(uuid.uuid4()),
|
||||
)
|
||||
)
|
||||
if not response.ok:
|
||||
details = response.error or f"TADA {action} failed"
|
||||
if response.logs:
|
||||
details = f"{details}\n" + "\n".join(response.logs)
|
||||
raise RuntimeError(details)
|
||||
return response
|
||||
|
||||
def _initialize_remote_engine(self) -> None:
|
||||
self._request(
|
||||
"initialize",
|
||||
{
|
||||
"model_path": self.model_path,
|
||||
"codec_path": self.codec_path,
|
||||
"tokenizer_path": self.tokenizer_path,
|
||||
"dtype": self.load_dtype,
|
||||
"attn_implementation": self.attn_implementation,
|
||||
"use_torch_compile": self.use_torch_compile,
|
||||
"prompt_cache_size": self.prompt_cache_size,
|
||||
},
|
||||
)
|
||||
self._initialized = True
|
||||
|
||||
def _ensure_remote_engine(self) -> None:
|
||||
process = getattr(self._session, "_process", None)
|
||||
if not self._initialized or process is None or process.poll() is not None:
|
||||
self._initialize_remote_engine()
|
||||
if self._comfy_loaded_model is None:
|
||||
self._register_with_comfy_model_management()
|
||||
|
||||
def _register_with_comfy_model_management(self) -> None:
|
||||
try:
|
||||
import comfy.model_management as model_management
|
||||
except Exception:
|
||||
return
|
||||
if not hasattr(model_management, "LoadedModel") or not hasattr(model_management, "current_loaded_models"):
|
||||
return
|
||||
try:
|
||||
loaded_model = model_management.LoadedModel(self)
|
||||
loaded_model.real_model = weakref.ref(self)
|
||||
finalizer = model_management.cleanup_models if hasattr(model_management, "cleanup_models") else lambda: None
|
||||
loaded_model.model_finalizer = weakref.finalize(self, finalizer)
|
||||
loaded_model._tts_wrapper_ref = self
|
||||
model_management.current_loaded_models.insert(0, loaded_model)
|
||||
self._comfy_loaded_model = loaded_model
|
||||
self._comfy_model_management = model_management
|
||||
except Exception as exc:
|
||||
print(f"⚠️ Failed to register TADA runtime with ComfyUI model management: {exc}")
|
||||
|
||||
def _unregister_from_comfy_model_management(self) -> None:
|
||||
model_management = self._comfy_model_management
|
||||
loaded_model = self._comfy_loaded_model
|
||||
if model_management is None or loaded_model is None:
|
||||
return
|
||||
try:
|
||||
if loaded_model in model_management.current_loaded_models:
|
||||
model_management.current_loaded_models.remove(loaded_model)
|
||||
except Exception as exc:
|
||||
print(f"⚠️ Failed to remove TADA runtime from ComfyUI tracking: {exc}")
|
||||
finally:
|
||||
self._comfy_loaded_model = None
|
||||
self._comfy_model_management = None
|
||||
|
||||
@staticmethod
|
||||
def _serialize_reference(reference_audio: object, bundle_dir: Path) -> Dict[str, Any]:
|
||||
if isinstance(reference_audio, (str, os.PathLike)):
|
||||
path = os.path.abspath(str(reference_audio))
|
||||
if not os.path.isfile(path):
|
||||
raise FileNotFoundError(f"TADA reference audio file not found: {path}")
|
||||
return {"kind": "audio_path", "audio_path": path}
|
||||
|
||||
audio = reference_audio
|
||||
sample_rate = 24000
|
||||
if isinstance(audio, (tuple, list)) and len(audio) == 2:
|
||||
audio, sample_rate = audio
|
||||
elif isinstance(audio, dict):
|
||||
if "audio" in audio and isinstance(audio["audio"], dict):
|
||||
audio = audio["audio"]
|
||||
if isinstance(audio, dict):
|
||||
path = audio.get("audio_path") or audio.get("path")
|
||||
if path:
|
||||
path = os.path.abspath(str(path))
|
||||
if not os.path.isfile(path):
|
||||
raise FileNotFoundError(f"TADA reference audio file not found: {path}")
|
||||
return {"kind": "audio_path", "audio_path": path}
|
||||
sample_rate = audio.get("sample_rate", audio.get("sampling_rate", sample_rate))
|
||||
audio = audio.get("waveform", audio.get("samples"))
|
||||
|
||||
if audio is None:
|
||||
raise ValueError("TADA requires reference audio.")
|
||||
waveform = audio.detach().cpu() if isinstance(audio, torch.Tensor) else torch.as_tensor(audio)
|
||||
tensor_path = bundle_dir / "reference.pt"
|
||||
torch.save({"waveform": waveform, "sample_rate": int(sample_rate)}, tensor_path)
|
||||
return {"kind": "tensor_path", "tensor_path": str(tensor_path)}
|
||||
|
||||
def generate_speech(
|
||||
self,
|
||||
text: str,
|
||||
reference_audio: object,
|
||||
reference_text: str,
|
||||
seed: int = 42,
|
||||
language: object = "English",
|
||||
acoustic_cfg_scale: float = 1.6,
|
||||
duration_cfg_scale: float = 1.0,
|
||||
cfg_schedule: str = "cosine",
|
||||
time_schedule: str = "logsnr",
|
||||
num_flow_matching_steps: int = 10,
|
||||
noise_temperature: float = 0.9,
|
||||
speed_up_factor: Optional[float] = None,
|
||||
num_transition_steps: int = 5,
|
||||
negative_condition_source: str = "negative_step_output",
|
||||
) -> Tuple[torch.Tensor, int]:
|
||||
if not str(reference_text or "").strip():
|
||||
raise ValueError(
|
||||
"TADA requires the exact transcript of the reference audio; ASR fallback is disabled."
|
||||
)
|
||||
validate_tada_model_language(self.current_model_name, language)
|
||||
|
||||
# A segment may switch language without reloading the base model. Ensure
|
||||
# just its aligner exists before the worker enters enforced offline mode.
|
||||
codec_parent = str(Path(self.codec_path).resolve().parent)
|
||||
TadaDownloader(base_path=codec_parent).ensure_codec_assets(
|
||||
language=language,
|
||||
codec_path=self.codec_path,
|
||||
)
|
||||
self._ensure_remote_engine()
|
||||
|
||||
with tempfile.TemporaryDirectory(prefix="tts_tada_") as temp_dir:
|
||||
bundle_dir = Path(temp_dir)
|
||||
output_path = bundle_dir / "result.pt"
|
||||
serialized_reference = self._serialize_reference(reference_audio, bundle_dir)
|
||||
self._request(
|
||||
"generate",
|
||||
{
|
||||
"text": str(text),
|
||||
"reference_audio": serialized_reference,
|
||||
"reference_text": str(reference_text),
|
||||
"seed": int(seed),
|
||||
"language": language,
|
||||
"acoustic_cfg_scale": float(acoustic_cfg_scale),
|
||||
"duration_cfg_scale": float(duration_cfg_scale),
|
||||
"cfg_schedule": str(cfg_schedule),
|
||||
"time_schedule": str(time_schedule),
|
||||
"num_flow_matching_steps": int(num_flow_matching_steps),
|
||||
"noise_temperature": float(noise_temperature),
|
||||
"speed_up_factor": speed_up_factor,
|
||||
"num_transition_steps": int(num_transition_steps),
|
||||
"negative_condition_source": str(negative_condition_source),
|
||||
"output_path": str(output_path),
|
||||
},
|
||||
)
|
||||
if not output_path.is_file():
|
||||
raise RuntimeError("TADA worker returned without writing its audio result.")
|
||||
result = torch.load(output_path, map_location="cpu")
|
||||
audio = result.get("audio")
|
||||
if not isinstance(audio, torch.Tensor):
|
||||
audio = torch.as_tensor(audio)
|
||||
return audio.detach().cpu().float(), int(result.get("sample_rate", 24000))
|
||||
|
||||
def to(self, device):
|
||||
self.device = resolve_torch_device(str(device))
|
||||
self.current_device = self.device
|
||||
self.load_device = self.current_loaded_device()
|
||||
return self
|
||||
|
||||
def eval(self):
|
||||
return self
|
||||
|
||||
def model_size(self):
|
||||
return self._estimated_memory_size
|
||||
|
||||
def loaded_size(self):
|
||||
return self._estimated_memory_size if self._initialized else 0
|
||||
|
||||
def model_memory(self):
|
||||
return self._estimated_memory_size
|
||||
|
||||
def get_ram_usage(self):
|
||||
return self._estimated_memory_size
|
||||
|
||||
def model_offloaded_memory(self):
|
||||
return 0 if self._initialized else self._estimated_memory_size
|
||||
|
||||
def model_mmap_residency(self, free: bool = False):
|
||||
return 0, self._estimated_memory_size
|
||||
|
||||
def pinned_memory_size(self):
|
||||
return 0
|
||||
|
||||
def lowvram_patch_counter(self):
|
||||
return 0
|
||||
|
||||
def model_dtype(self):
|
||||
return self.dtype
|
||||
|
||||
def current_loaded_device(self):
|
||||
if str(self.device).startswith("cuda") and torch.cuda.is_available():
|
||||
return torch.device("cuda", torch.cuda.current_device())
|
||||
return torch.device("cpu")
|
||||
|
||||
def model_patches_models(self):
|
||||
return ()
|
||||
|
||||
def model_patches_to(self, target):
|
||||
return None
|
||||
|
||||
def is_dynamic(self):
|
||||
return False
|
||||
|
||||
def partially_load(self, device, extra_memory, force_patch_weights=False):
|
||||
return 0
|
||||
|
||||
def partially_unload_ram(self, ram_to_unload):
|
||||
return self.partially_unload("cpu", ram_to_unload)
|
||||
|
||||
def partially_unload(self, device, memory_to_free):
|
||||
self.cleanup(unregister=False)
|
||||
return self._estimated_memory_size
|
||||
|
||||
def model_unload(self, *args, **kwargs):
|
||||
self.cleanup(unregister=False)
|
||||
return True
|
||||
|
||||
def detach(self, unpatch_weights=True):
|
||||
self.cleanup(unregister=False)
|
||||
|
||||
def is_clone(self, other):
|
||||
return other is self
|
||||
|
||||
def cleanup(self, unregister: bool = True) -> None:
|
||||
if unregister:
|
||||
self._unregister_from_comfy_model_management()
|
||||
else:
|
||||
self._comfy_loaded_model = None
|
||||
self._comfy_model_management = None
|
||||
process = getattr(getattr(self, "_session", None), "_process", None)
|
||||
if self._initialized and process is not None and process.poll() is None:
|
||||
try:
|
||||
self._request("cleanup")
|
||||
except Exception:
|
||||
pass
|
||||
self._initialized = False
|
||||
session = getattr(self, "_session", None)
|
||||
if session is not None:
|
||||
session.close()
|
||||
|
||||
def close(self) -> None:
|
||||
self.cleanup()
|
||||
|
||||
|
||||
def build_tada_isolated_proxy(config):
|
||||
requested_profile = getattr(config, "runtime_profile", None) or TADA_RUNTIME_PROFILE
|
||||
if requested_profile != TADA_RUNTIME_PROFILE:
|
||||
raise RuntimeError(
|
||||
f"TADA only supports runtime profile '{TADA_RUNTIME_PROFILE}', not '{requested_profile}'."
|
||||
)
|
||||
profile = get_runtime_profile(TADA_RUNTIME_PROFILE)
|
||||
if profile is None:
|
||||
raise RuntimeError(f"TADA runtime profile '{TADA_RUNTIME_PROFILE}' is not configured.")
|
||||
return TadaIsolatedProxy(config, profile)
|
||||
|
||||
|
||||
__all__ = ["TADA_RUNTIME_PROFILE", "TadaIsolatedProxy", "build_tada_isolated_proxy"]
|
||||
@@ -0,0 +1,170 @@
|
||||
from __future__ import annotations
|
||||
|
||||
"""JSON-line worker for TADA's shared Transformers 4 runtime."""
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import traceback
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[3]
|
||||
ENGINES_DIR = PROJECT_ROOT / "engines"
|
||||
|
||||
# ComfyUI may add the suite's engines directory directly to PYTHONPATH. In this
|
||||
# worker that would make engines/tada shadow the official hume-tada package.
|
||||
sys.path[:] = [
|
||||
path
|
||||
for path in sys.path
|
||||
if Path(path or os.getcwd()).resolve() != ENGINES_DIR.resolve()
|
||||
]
|
||||
if str(PROJECT_ROOT) in sys.path:
|
||||
sys.path.remove(str(PROJECT_ROOT))
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
os.environ.setdefault("HF_HUB_OFFLINE", "1")
|
||||
os.environ.setdefault("TRANSFORMERS_OFFLINE", "1")
|
||||
|
||||
from utils.runtimes.protocol import RuntimeJobResponse
|
||||
|
||||
|
||||
def _emit(stream, response: RuntimeJobResponse) -> None:
|
||||
stream.write(json.dumps(response.to_dict(), ensure_ascii=True) + "\n")
|
||||
stream.flush()
|
||||
|
||||
|
||||
def _load_reference(payload):
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError("TADA worker received an invalid reference-audio payload.")
|
||||
kind = payload.get("kind")
|
||||
if kind == "audio_path":
|
||||
return payload.get("audio_path")
|
||||
if kind == "tensor_path":
|
||||
tensor_path = payload.get("tensor_path")
|
||||
if not tensor_path:
|
||||
raise ValueError("TADA tensor reference is missing tensor_path.")
|
||||
saved = torch.load(tensor_path, map_location="cpu")
|
||||
return saved["waveform"], int(saved.get("sample_rate", 24000))
|
||||
raise ValueError(f"Unsupported TADA reference-audio payload kind: {kind}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
protocol_out = sys.stdout
|
||||
sys.stdout = sys.stderr
|
||||
|
||||
from engines.tada.tada_engine import TadaEngine
|
||||
|
||||
engine = None
|
||||
for line in sys.stdin:
|
||||
stripped = line.strip()
|
||||
if not stripped:
|
||||
continue
|
||||
request = None
|
||||
try:
|
||||
request = json.loads(stripped)
|
||||
request_id = request.get("request_id")
|
||||
action = request.get("action")
|
||||
payload = request.get("payload") or {}
|
||||
|
||||
if action == "shutdown":
|
||||
if engine is not None:
|
||||
engine.cleanup()
|
||||
engine = None
|
||||
_emit(protocol_out, RuntimeJobResponse(ok=True, result={"shutdown": True}, request_id=request_id))
|
||||
break
|
||||
|
||||
if action == "ping":
|
||||
_emit(protocol_out, RuntimeJobResponse(ok=True, result={"pong": True}, request_id=request_id))
|
||||
continue
|
||||
|
||||
if action == "initialize":
|
||||
if request.get("runtime_profile") != "vibevoice_transformers4_shared":
|
||||
raise RuntimeError("TADA can only initialize in vibevoice_transformers4_shared.")
|
||||
if engine is not None:
|
||||
engine.cleanup()
|
||||
engine = TadaEngine(
|
||||
model_name=request.get("model_name") or "TADA-1B",
|
||||
model_path=payload["model_path"],
|
||||
codec_path=payload["codec_path"],
|
||||
tokenizer_path=payload["tokenizer_path"],
|
||||
device=request.get("device") or "auto",
|
||||
dtype=payload.get("dtype", "auto"),
|
||||
attn_implementation=payload.get("attn_implementation", "sdpa"),
|
||||
use_torch_compile=payload.get("use_torch_compile", False),
|
||||
prompt_cache_size=payload.get("prompt_cache_size", 8),
|
||||
)
|
||||
_emit(
|
||||
protocol_out,
|
||||
RuntimeJobResponse(
|
||||
ok=True,
|
||||
result={"model_name": request.get("model_name"), "sample_rate": 24000},
|
||||
request_id=request_id,
|
||||
),
|
||||
)
|
||||
continue
|
||||
|
||||
if action == "cleanup":
|
||||
if engine is not None:
|
||||
engine.cleanup()
|
||||
engine = None
|
||||
_emit(protocol_out, RuntimeJobResponse(ok=True, result={"cleaned": True}, request_id=request_id))
|
||||
continue
|
||||
|
||||
if engine is None:
|
||||
raise RuntimeError("TADA worker received a generation request before initialization.")
|
||||
|
||||
if action == "generate":
|
||||
audio, sample_rate = engine.generate_speech(
|
||||
text=payload["text"],
|
||||
reference_audio=_load_reference(payload.get("reference_audio")),
|
||||
reference_text=payload["reference_text"],
|
||||
seed=payload.get("seed", 42),
|
||||
language=payload.get("language", "English"),
|
||||
acoustic_cfg_scale=payload.get("acoustic_cfg_scale", 1.6),
|
||||
duration_cfg_scale=payload.get("duration_cfg_scale", 1.0),
|
||||
cfg_schedule=payload.get("cfg_schedule", "cosine"),
|
||||
time_schedule=payload.get("time_schedule", "logsnr"),
|
||||
num_flow_matching_steps=payload.get("num_flow_matching_steps", 10),
|
||||
noise_temperature=payload.get("noise_temperature", 0.9),
|
||||
speed_up_factor=payload.get("speed_up_factor"),
|
||||
num_transition_steps=payload.get("num_transition_steps", 5),
|
||||
negative_condition_source=payload.get(
|
||||
"negative_condition_source", "negative_step_output"
|
||||
),
|
||||
)
|
||||
output_path = payload["output_path"]
|
||||
torch.save(
|
||||
{"audio": audio.detach().cpu().float(), "sample_rate": int(sample_rate)},
|
||||
output_path,
|
||||
)
|
||||
_emit(
|
||||
protocol_out,
|
||||
RuntimeJobResponse(
|
||||
ok=True,
|
||||
result={"output_path": output_path, "sample_rate": int(sample_rate)},
|
||||
request_id=request_id,
|
||||
),
|
||||
)
|
||||
continue
|
||||
|
||||
raise RuntimeError(f"Unsupported TADA worker action '{action}'")
|
||||
|
||||
except Exception as exc:
|
||||
request_id = request.get("request_id") if isinstance(request, dict) else None
|
||||
_emit(
|
||||
protocol_out,
|
||||
RuntimeJobResponse(
|
||||
ok=False,
|
||||
error=f"{exc}\n{traceback.format_exc()}",
|
||||
request_id=request_id,
|
||||
),
|
||||
)
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -28,6 +28,23 @@ PARAMETER_ALIASES = {
|
||||
'cfg_weight': 'cfg', # cfg_weight alias -> cfg (more universal)
|
||||
'cfgweight': 'cfg',
|
||||
'cfg_scale': 'cfg',
|
||||
'acoustic_cfg': 'acoustic_cfg_scale',
|
||||
'acoustic_cfg_scale': 'acoustic_cfg_scale',
|
||||
'duration_cfg': 'duration_cfg_scale',
|
||||
'duration_cfg_scale': 'duration_cfg_scale',
|
||||
'noise_temp': 'noise_temperature',
|
||||
'noise_temperature': 'noise_temperature',
|
||||
'flow_steps': 'num_flow_matching_steps',
|
||||
'flow_matching_steps': 'num_flow_matching_steps',
|
||||
'num_flow_matching_steps': 'num_flow_matching_steps',
|
||||
'cfg_schedule': 'cfg_schedule',
|
||||
'time_schedule': 'time_schedule',
|
||||
'negative_condition': 'negative_condition_source',
|
||||
'negative_condition_source': 'negative_condition_source',
|
||||
'speed_up': 'speed_up_factor',
|
||||
'speed_up_factor': 'speed_up_factor',
|
||||
'transition_steps': 'num_transition_steps',
|
||||
'num_transition_steps': 'num_transition_steps',
|
||||
'stg': 'stg_scale',
|
||||
'stg_scale': 'stg_scale',
|
||||
'duration_multiplier': 'duration_multiplier',
|
||||
@@ -115,7 +132,7 @@ PARAMETER_ENGINES = {
|
||||
'seed': {
|
||||
'chatterbox', 'chatterbox_official_23lang', 'f5tts', 'higgs_audio',
|
||||
'higgs_audio_v3', 'vibevoice', 'index_tts', 'step_audio_editx', 'cosyvoice', 'qwen3_tts',
|
||||
'dots_tts', 'fish_audio_s2', 'omnivoice',
|
||||
'dots_tts', 'fish_audio_s2', 'omnivoice', 'tada',
|
||||
'echo_tts', 'moss_tts', 'moss_soundeffect_v2', 'dramabox'
|
||||
},
|
||||
'temperature': {
|
||||
@@ -124,8 +141,17 @@ PARAMETER_ENGINES = {
|
||||
},
|
||||
'cfg': {
|
||||
'f5tts', 'vibevoice', 'index_tts', 'chatterbox', 'chatterbox_official_23lang',
|
||||
'moss_soundeffect_v2', 'dramabox'
|
||||
'moss_soundeffect_v2', 'dramabox', 'tada'
|
||||
},
|
||||
'acoustic_cfg_scale': {'tada'},
|
||||
'duration_cfg_scale': {'tada'},
|
||||
'noise_temperature': {'tada'},
|
||||
'num_flow_matching_steps': {'tada'},
|
||||
'cfg_schedule': {'tada'},
|
||||
'time_schedule': {'tada'},
|
||||
'negative_condition_source': {'tada'},
|
||||
'speed_up_factor': {'tada'},
|
||||
'num_transition_steps': {'tada'},
|
||||
'stg_scale': {
|
||||
'dramabox'
|
||||
},
|
||||
@@ -145,7 +171,7 @@ PARAMETER_ENGINES = {
|
||||
'dramabox'
|
||||
},
|
||||
'num_steps': {
|
||||
'echo_tts', 'dots_tts', 'omnivoice'
|
||||
'echo_tts', 'dots_tts', 'omnivoice', 'tada'
|
||||
},
|
||||
'guidance_scale': {
|
||||
'dots_tts', 'omnivoice'
|
||||
@@ -211,7 +237,7 @@ PARAMETER_ENGINES = {
|
||||
'chatterbox', 'chatterbox_official_23lang'
|
||||
},
|
||||
'speed': {
|
||||
'f5tts', 'cosyvoice', 'omnivoice'
|
||||
'f5tts', 'cosyvoice', 'omnivoice', 'tada'
|
||||
},
|
||||
'top_p': {
|
||||
'higgs_audio', 'higgs_audio_v3', 'vibevoice', 'index_tts', 'qwen3_tts', 'moss_tts', 'fish_audio_s2'
|
||||
@@ -283,6 +309,17 @@ PARAMETER_VALIDATION = {
|
||||
'seed': (int, 0, 2**32 - 1, "Random seed for reproducible generation"),
|
||||
'temperature': (float, 0.1, 2.0, "Randomness/creativity control (lower=more deterministic)"),
|
||||
'cfg': (float, 0.0, 20.0, "Classifier-free guidance strength"),
|
||||
'acoustic_cfg_scale': (float, 0.0, 10.0, "TADA acoustic classifier-free guidance scale"),
|
||||
'duration_cfg_scale': (float, 0.0, 10.0, "TADA duration classifier-free guidance scale"),
|
||||
'noise_temperature': (float, 0.0, 3.0, "TADA acoustic noise temperature"),
|
||||
'num_flow_matching_steps': (int, 1, 100, "TADA flow-matching inference steps"),
|
||||
'cfg_schedule': (str, None, None, "TADA CFG schedule: constant, linear, or cosine"),
|
||||
'time_schedule': (str, None, None, "TADA time schedule: uniform, cosine, or logsnr"),
|
||||
'negative_condition_source': (
|
||||
str, None, None, "TADA negative condition: negative_step_output, prompt, or zero"
|
||||
),
|
||||
'speed_up_factor': (float, 0.0, 4.0, "TADA speech speed-up factor; 0 disables it"),
|
||||
'num_transition_steps': (int, 0, 64, "TADA prompt-to-generation transition steps"),
|
||||
'stg_scale': (float, 0.0, 5.0, "DramaBox skip-token guidance strength"),
|
||||
'duration_multiplier': (float, 0.5, 3.0, "DramaBox estimated-duration multiplier"),
|
||||
'gen_duration': (float, 0.0, 60.0, "DramaBox explicit output duration"),
|
||||
@@ -350,14 +387,27 @@ PARAMETER_NODE_KEYS = {
|
||||
'f5tts': 'cfg_strength',
|
||||
'moss_soundeffect_v2': 'cfg_scale',
|
||||
'dramabox': 'cfg_scale',
|
||||
'tada': 'acoustic_cfg_scale',
|
||||
}, # Engine-specific mapping
|
||||
'acoustic_cfg_scale': 'acoustic_cfg_scale',
|
||||
'duration_cfg_scale': 'duration_cfg_scale',
|
||||
'noise_temperature': 'noise_temperature',
|
||||
'num_flow_matching_steps': 'num_flow_matching_steps',
|
||||
'cfg_schedule': 'cfg_schedule',
|
||||
'time_schedule': 'time_schedule',
|
||||
'negative_condition_source': 'negative_condition_source',
|
||||
'speed_up_factor': 'speed_up_factor',
|
||||
'num_transition_steps': 'num_transition_steps',
|
||||
'stg_scale': 'stg_scale',
|
||||
'duration_multiplier': 'duration_multiplier',
|
||||
'gen_duration': 'gen_duration',
|
||||
'ref_duration': 'ref_duration',
|
||||
'rescale_scale': 'rescale_scale',
|
||||
'prompt_template': 'prompt_template',
|
||||
'num_steps': 'num_steps',
|
||||
'num_steps': {
|
||||
'default': 'num_steps',
|
||||
'tada': 'num_flow_matching_steps',
|
||||
},
|
||||
'guidance_scale': 'guidance_scale',
|
||||
'duration': 'duration',
|
||||
't_shift': 't_shift',
|
||||
@@ -379,7 +429,10 @@ PARAMETER_NODE_KEYS = {
|
||||
'speaker_kv_min_t': 'speaker_kv_min_t',
|
||||
'sequence_length': 'sequence_length',
|
||||
'exaggeration': 'exaggeration',
|
||||
'speed': 'speed',
|
||||
'speed': {
|
||||
'default': 'speed',
|
||||
'tada': 'speed_up_factor',
|
||||
},
|
||||
'top_p': 'top_p',
|
||||
'top_k': 'top_k',
|
||||
'audio_temperature': 'audio_temperature',
|
||||
@@ -478,6 +531,17 @@ class ParameterValidator:
|
||||
if not 0.0 <= converted <= 1.0:
|
||||
return False, "rescale_scale must be 'auto' or between 0 and 1", value
|
||||
|
||||
allowed_values = {
|
||||
'cfg_schedule': {'constant', 'linear', 'cosine'},
|
||||
'time_schedule': {'uniform', 'cosine', 'logsnr'},
|
||||
'negative_condition_source': {'negative_step_output', 'prompt', 'zero'},
|
||||
}
|
||||
if param_name in allowed_values:
|
||||
converted = str(converted).strip().lower()
|
||||
if converted not in allowed_values[param_name]:
|
||||
choices = ', '.join(sorted(allowed_values[param_name]))
|
||||
return False, f"{param_name} must be one of: {choices}", value
|
||||
|
||||
# Check bounds
|
||||
if min_val is not None and max_val is not None and (converted < min_val or converted > max_val):
|
||||
return False, f"{param_name} must be between {min_val} and {max_val}, got {converted}", value
|
||||
|
||||
@@ -570,8 +570,8 @@ class VoiceDiscovery:
|
||||
if engine_type in ("audio_only", "chatterbox", "moss_tts", "higgs_audio_v3"): # Keep backward compatibility
|
||||
return character_info
|
||||
|
||||
# For audio+text engines (F5-TTS, Step Audio EditX, CosyVoice, Qwen3-TTS), we need both audio and text
|
||||
if engine_type in ("audio_and_text", "f5tts", "step_audio_editx", "cosyvoice", "index_tts", "qwen3_tts"):
|
||||
# For audio+text engines (F5-TTS, Step Audio EditX, CosyVoice, Qwen3-TTS, TADA), we need both audio and text
|
||||
if engine_type in ("audio_and_text", "f5tts", "step_audio_editx", "cosyvoice", "index_tts", "qwen3_tts", "tada"):
|
||||
if character_info.get('text_content'):
|
||||
return character_info
|
||||
|
||||
|
||||
Reference in New Issue
Block a user