Compare commits

...
Author SHA1 Message Date
diodiogod 7c64fedede Version 5.4.4
Release Multiline TTS Tag Editor inline emotion controls

Implementation details:
- Add engine-aware one-click and hold-drag tag swapping
- Support combined IndexTTS audio references and vector or text emotion controls
- Improve quoted emotion routing and long inline tag wrapping
- Fix tag parsing, character switching, and newline insertion behavior
2026-07-11 16:25:30 -03:00
diodiogod 6da635cf74 Sync IndexTTS updates and document editor features 2026-07-11 16:24:08 -03:00
diodiogod 8bfd6512de Route quoted emotion tags to editor 2026-07-11 12:00:23 -03:00
diodiogod ffa3f1eded Add fast inline tag swapping 2026-07-11 11:09:01 -03:00
diodiogod 87def26983 Update version bump instructions 2026-07-11 08:09:50 -03:00
diodiogod 057a9ef638 Document and compose inline emotion controls 2026-07-11 08:09:08 -03:00
diodiogod df3a4cb2cf Blend IndexTTS audio and vector emotions 2026-07-11 01:39:43 -03:00
diodiogod 06003e0b3a Fix IndexTTS emotion and character routing 2026-07-11 01:24:38 -03:00
diodiogod 068c3f1f1e Add IndexTTS-2 inline emotion controls 2026-07-10 23:21:53 -03:00
diodiogod 46c051477d Version 5.4.3
Fix IndexTTS-2 emotion vector controls

Implementation details:
- Keep the fixed radar node size so button hit areas remain aligned
- Add clipped canvas rendering to prevent visual overflow
- Replace clipboard-dependent export fallback with a selectable JSON dialog
- Require explicit download and show import success feedback
2026-07-10 15:30:05 -03:00
diodiogod 52b22c0b8d Document FL-MCP Windows bridge fallback 2026-07-10 14:29:22 -03:00
diodiogod b31b31b89f Version 5.4.2
Address issue #312 with MPS-safe RVC tensor typing

Technical details:
- Keep RVC pitch indices integer on Apple MPS devices.
- Normalize phone features to the linear layer weight dtype.
- Normalize pitch values to torch.long at the encoder boundary.
- Preserve CUDA half-precision compatibility.
2026-07-10 14:25:59 -03:00
diodiogod 9757092aae Version 5.4.1
Address issue #311 with resilient MelBand resampling

Technical details:
- Route MelBand input conversion through the shared safe resampler.
- Fall back to torchaudio when librosa/Numba resampling is incompatible.
- Correct the PROJECT_INDEX path for bump instructions.
- Make version tooling use UTF-8 on Windows.
2026-07-10 14:06:12 -03:00
diodiogod 47a8c7c691 Version 5.4.0
Add Fish Audio S2 Pro engine and integration workflow

Implementation details:
- Integrate the official Fish S2 runtime with isolated loading and model selection
- Add native multi-speaker and custom character-switching generation paths
- Add inline instruction translation, language prompting, caching, compilation, and optional quantization
- Add unified Text and SRT workflow coverage with documentation and a workflow cover
2026-07-10 13:37:44 -03:00
diodiogod 1033eb78b9 Merge Fish Audio S2 integration 2026-07-10 13:37:15 -03:00
diodiogod 1f2703a3a3 Add Fish Audio S2 integration workflow
Add the Fish Audio S2 Pro engine workflow with native and custom speaker modes, inline instruction handling, language prompting, compilation, quantization, and SRT support.

Document the workflow in the README and include a square cover image for the example workflow.
2026-07-10 13:37:01 -03:00
diodiogod af77ff8e0b Match Fish workflow documentation layout 2026-07-10 11:56:44 -03:00
diodiogod 8b199980dc Improve Fish Audio S2 integration workflow 2026-07-10 09:39:37 -03:00
diodiogod 26fa2ddf92 Add Fish Audio S2 integration workflow 2026-07-10 09:16:33 -03:00
diodiogod 282616ebe1 Finalize Fish Audio S2 integration 2026-07-10 09:09:17 -03:00
diodiogod 5735616794 Prevent invalid hash seed from breaking isolated runtimes 2026-07-10 00:28:12 -03:00
diodiogod 6ce60f31a2 Document new engine validation rules 2026-07-10 00:21:17 -03:00
diodiogod 102f114ae6 Standardize resolved voice prompt logging 2026-07-10 00:20:00 -03:00
diodiogod 4494381176 Finalize Fish Audio S2 engine integration 2026-07-09 23:13:24 -03:00
diodiogod 84e82b7d31 Refine Fish Audio S2 compile and override logging 2026-07-08 23:21:16 -03:00
diodiogod e54b6d9aa5 Refine Fish speaker mapping and UI slots 2026-07-08 22:21:38 -03:00
diodiogod 984f0e9fa5 Fix VibeVoice native speaker mapping and logging 2026-07-07 00:56:57 -03:00
diodiogod 1218e01683 Fix OmniVoice tag handling to follow suite syntax 2026-07-07 00:13:19 -03:00
diodiogod 9081d3c22a Add Fish Audio S2 engine and fix VibeVoice speaker remap 2026-07-06 23:35:13 -03:00
diodiogod 09df168119 Version 5.3.0
Add Granite ASR 4.1 diarization and timestamp refinements

Implementation details:
- normalize Granite ASR plus diarization output to suite-native [Speaker] tags
- fix Granite ASR plus native timestamp truncation by auto-raising low generation token budgets
- move Unified ASR Transcribe diarization input to preserve old workflow compatibility
- update Granite ASR YAML and README documentation and regenerate engine tables
- improve cross-platform test runner path handling and Granite regression coverage
2026-06-23 18:33:56 -03:00
diodiogod b00b6098c9 Merge pull request #309 from tazztone/feature/combined-granite-echo-testing
Feature/combined granite echo testing
2026-06-23 18:32:36 -03:00
diodiogod a41ac5d89e Merge origin/main into feature/combined-granite-echo-testing 2026-06-23 18:27:49 -03:00
diodiogod ed5e55a5fc Update README.md 2026-06-23 16:21:09 -03:00
diodiogod a0b58e42f7 Normalize large-language engine summaries in generated tables
- add full and compact language summary overrides to the engine table generator
- show Higgs Audio v3 with globe-style 100+ language summary in the README table
- show OmniVoice as 600+ languages in both compact and full generated tables
2026-06-23 12:10:00 -03:00
diodiogod 44d164c2b7 Fix OmniVoice language summary in generated tables
- add compact language summary override support to the table generator
- show OmniVoice as 600+ languages in the README summary table
- rename OmniVoice voice design feature text to instruction-based voice design
2026-06-23 12:06:07 -03:00
diodiogod 14fcba3bed Version 5.2.0
Add OmniVoice engine release polish

Implementation details:
- Finalize OmniVoice engine integration across text and SRT workflows
- Add native duration targeting support for precise subtitle timing
- Add Visual Tag Builder workflow and documentation polish
- Propagate interruption handling across unified generation nodes
- Fix Qwen3 voice designer metadata URL and test path mocking
2026-06-23 11:51:40 -03:00
diodiogod 36ed7d694a Add OmniVoice workflow example and docs polish 2026-06-23 11:46:37 -03:00
diodiogod 24e6575fd4 Fix Qwen3 voice designer metadata URL 2026-06-22 22:46:01 -03:00
diodiogod fcff4ccf45 Propagate interruption across unified nodes 2026-06-22 22:27:31 -03:00
diodiogod f7abd5a891 Handle OmniVoice interruption without silence fallback 2026-06-22 22:23:04 -03:00
diodiogod 1291346dda Polish Visual Tag Builder output and README 2026-06-22 17:44:12 -03:00
diodiogod 552a67d209 Rename Visual Tag Builder and clean tooltips 2026-06-22 17:05:54 -03:00
diodiogod 35e0b27913 Persist OmniVoice builder presets and layout 2026-06-19 14:38:44 -03:00
diodiogod 1e92a9f4b1 Polish OmniVoice instruction builder layout 2026-06-19 11:51:29 -03:00
diodiogod 5c346693e3 Add OmniVoice instruction builder UI 2026-06-19 01:40:21 -03:00
diodiogod 3717fc0cbe Refine OmniVoice loading and progress logs 2026-06-18 22:57:12 -03:00
diodiogod af13209b47 Add OmniVoice engine with native SRT duration targeting 2026-06-18 22:23:32 -03:00
137 changed files with 16301 additions and 1022 deletions
+87
View File
@@ -5,6 +5,93 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [5.4.4] - 2026-07-11
### Added
- Add faster character, parameter, preset, and emotion swapping
- Support combining IndexTTS audio emotion references with vector or text emotions
### Changed
- Improve Multiline TTS Tag Editor emotion switching
- Improve long inline tag wrapping inside the editor
### Fixed
- Fix incorrect tag detection and extra blank lines when pressing Enter
## [5.4.3] - 2026-07-10
### Added
- Keep emotion radar controls aligned at the original node size
- Prevent the radar chart from overflowing its node
- Make emotion vector export selectable and downloadable on demand
- Show confirmation after importing emotion vectors
### Fixed
- Fix IndexTTS-2 emotion vector controls
## [5.4.2] - 2026-07-10
### Added
- Corrects pitch-index typing for RVC voice conversion on MPS devices.
- Preserves the model's expected precision for phone features across CPU, CUDA, and MPS.
### Fixed
- Tentative fix for RVC Voice Changer on Apple Silicon
## [5.4.1] - 2026-07-10
### Added
- Prevents MelBand vocal removal from failing during sample-rate conversion on some Python 3.13 environments.
- Corrects the documented location of the version bump instructions.
- Makes future version bumps reliable on Windows installations with non-UTF-8 locales.
### Fixed
- Tentative fix for MelBand audio separation failures
## [5.4.0] - 2026-07-10
### Added
- Add Fish Audio S2 Pro multilingual voice generation
- Add Fish Audio S2 Pro voice cloning with reference audio and transcript support
- Add native multi-speaker dialogue and independent character-segment generation
- Add free-form inline speech instructions and automatic language prompting
- Add long-form generation, SRT integration, compilation, caching, and optional quantization
## [5.3.0] - 2026-06-23
### Added
- Add Granite ASR 4.1 diarization and timestamp improvements
- Add Granite ASR plus-model speaker diarization with suite-native [Speaker] output for downstream TTS workflows
- Clarify Granite ASR 4.1 model differences, diarization support, and Japanese support in the README and generated docs
### Changed
- Improve test runner portability and Granite ASR regression coverage for contributors
### Fixed
- Fix Granite ASR plus native word timestamps cutting off longer transcripts in timestamp mode
## [5.2.0] - 2026-06-23
### Added
- Add OmniVoice engine with native timing-aware TTS controls
- Add OmniVoice text and SRT generation with native duration targeting for more precise subtitle timing
- Add Visual Tag Builder workflow example and documentation for OmniVoice prompt building
### Changed
- Improve interruption handling so stopped generation exits cleanly across unified TTS workflows
### Fixed
- Fix Qwen3 Voice Designer metadata links and test environment path handling
## [5.1.1] - 2026-06-17
### Added
+21 -19
View File
@@ -27,24 +27,26 @@ Third-Party Model Licenses
The project code is MIT. Model weights carry their own licenses:
──────────────── ──────────────────────────────────────────────────────── ──────────────
Engine License Commercial Use
──────────────── ──────────────────────────────────────────────────────── ──────────────
F5-TTS CC-BY-NC-4.0 No
ChatterBox MIT Yes
ChatterBox 23L MIT Yes
VibeVoice MIT (research-only per model card) No
Higgs Audio 2 Boson Higgs Audio 2 Community License Conditional
Higgs Audio v3 Boson Higgs Audio v3 Research and Non-Commercial License No
IndexTTS-2 bilibili Model Use License Conditional
CosyVoice3 Apache-2.0 Yes
Qwen3-TTS Apache-2.0 Yes
Granite ASR Apache-2.0 Yes
Step Audio EditX Apache-2.0 (verify before commercial use) Conditional
Echo-TTS CC-BY-NC-SA-4.0 No
Dots TTS Apache-2.0 Yes
MOSS-TTS Apache-2.0 Yes
RVC MIT (framework); community models vary Varies
──────────────── ──────────────────────────────────────────────────────── ──────────────
───────────────── ──────────────────────────────────────────────────────── ──────────────
Engine License Commercial Use
───────────────── ──────────────────────────────────────────────────────── ──────────────
F5-TTS CC-BY-NC-4.0 No
ChatterBox MIT Yes
ChatterBox 23L MIT Yes
VibeVoice MIT (research-only per model card) No
Higgs Audio 2 Boson Higgs Audio 2 Community License Conditional
Higgs Audio v3 Boson Higgs Audio v3 Research and Non-Commercial License No
IndexTTS-2 bilibili Model Use License Conditional
CosyVoice3 Apache-2.0 Yes
Qwen3-TTS Apache-2.0 Yes
Granite ASR Apache-2.0 Yes
Step Audio EditX Apache-2.0 (verify before commercial use) Conditional
Echo-TTS CC-BY-NC-SA-4.0 No
Fish Audio S2 Pro Fish Audio Research License No
Dots TTS Apache-2.0 Yes
OmniVoice Apache-2.0 Yes
MOSS-TTS Apache-2.0 Yes
RVC MIT (framework); community models vary Varies
───────────────── ──────────────────────────────────────────────────────── ──────────────
Users are responsible for complying with respective model licenses.
+13 -9
View File
@@ -25,7 +25,7 @@
## Engines
14 engines follow the pattern above:
15 engines follow the pattern above:
| Engine | Adapter | Processor | SRT Processor | Engine Node |
|--------|---------|-----------|---------------|-------------|
@@ -42,26 +42,29 @@
| MOSS-TTS | `moss_tts_adapter.py` | `nodes/moss_tts/moss_tts_processor.py` | `moss_tts_srt_processor.py` | `moss_tts_engine_node.py` |
| Granite ASR | `asr_granite_adapter.py` | — | — | `granite_asr_engine_node.py` |
| Echo-TTS | `echo_tts_adapter.py` | `nodes/echo_tts/echo_tts_processor.py` | `echo_tts_srt_processor.py` | `echo_tts_engine_node.py` |
| OmniVoice | `omnivoice_adapter.py` | `nodes/omnivoice/omnivoice_processor.py` | `omnivoice_srt_processor.py` | `omnivoice_engine_node.py` |
| RVC | — | `engines/rvc/` | — | `rvc_engine_node.py` |
**Engine implementations live in:**
- `engines/chatterbox/`, `engines/chatterbox_official_23lang/`, `engines/f5tts/`, `engines/higgs_audio/`, `engines/higgs_audio_v3/`, `engines/vibevoice_engine/`, `engines/step_audio_editx/`, `engines/cosyvoice/`, `engines/qwen3_tts/`, `engines/qwen3_asr/`, `engines/moss_tts/`, `engines/granite_asr/`, `engines/rvc/`
- `engines/chatterbox/`, `engines/chatterbox_official_23lang/`, `engines/f5tts/`, `engines/higgs_audio/`, `engines/higgs_audio_v3/`, `engines/vibevoice_engine/`, `engines/step_audio_editx/`, `engines/cosyvoice/`, `engines/qwen3_tts/`, `engines/qwen3_asr/`, `engines/moss_tts/`, `engines/granite_asr/`, `engines/echo_tts/`, `engines/omnivoice/`, `engines/rvc/`
## Documentation Files
**README.md** - Main project docs, installation, features overview
**CLAUDE.md** - Dev guidelines for Claude Code
**CHANGELOG.md** - Full version history
**README.md** - Main project docs, installation, features overview
**CLAUDE.md** - Dev guidelines for Claude Code
**CHANGELOG.md** - Full version history
**docs/BUMP_SCRIPT_INSTRUCTIONS.md** - Version bump process
### User Docs (`docs/`)
- `CHARACTER_SWITCHING_GUIDE.md` - [CharacterName] tag system
- `PARAMETER_SWITCHING_GUIDE.md` - Per-segment parameter override syntax
- `INLINE_EDIT_TAGS_USER_GUIDE.md` - Step Audio EditX inline tags
- `HIGGS_AUDIO_V3_INLINE_TAGS.md` - Higgs Audio v3 native paralinguistic tags
- `OMNIVOICE_TAGS_GUIDE.md` - OmniVoice native non-verbal tags and pronunciation overrides
- `MOSS_TTS_PROMPT_FIELDS_GUIDE.md` - Official MOSS whole-segment prompt fields and inline `<>` translation limits
- `COSYVOICE3_TAGS_GUIDE.md` - CosyVoice3 native paralinguistic tags
- `CHATTERBOX_V2_SPECIAL_TOKENS.md` - ChatterBox v2 emotion tokens
- `IndexTTS2_Emotion_Control_Guide.md` - IndexTTS-2 emotion vectors
- `IndexTTS2_Emotion_Control_Guide.md` - IndexTTS-2 vector, text, audio, and blended emotion controls
- `VOCAL_REMOVAL_GUIDE.md` - Vocal separation guide
- `qwen3_tts_optimizations.md` - Qwen3-TTS torch.compile setup
- `MODEL_DOWNLOAD_SOURCES.md` - All HF repo links (auto-generated)
@@ -72,7 +75,6 @@
### Dev Docs (`docs/Dev reports/`)
- `tts_audio_suite_engines.yaml` - **Source of truth** for all engine metadata
- `tts_audio_suite_aux_models.yaml` - **Source of truth** for helper/post-process model metadata
- `BUMP_SCRIPT_INSTRUCTIONS.md` - Version bump process
- `SRT_IMPLEMENTATION.md` - SRT timing technical details
- `ISOLATED_RUNTIMES_PLAN.md` - original runtime isolation plan and scope
- `TRANSFORMERS_5_QWEN3_TTS_REPORT.md` - why Qwen3-TTS moved to shared legacy T4 runtime
@@ -104,6 +106,7 @@
### Shared / Special Nodes
- `nodes/shared/character_voices_node.py` - Character voice management (NARRATOR_VOICE output)
- `nodes/omnivoice/omnivoice_instruction_builder_node.py` - OmniVoice voice-design instruction helper with custom visual builder UI
- `nodes/text/phoneme_text_normalizer_node.py` - Multilingual text preprocessing
- `nodes/text/asr_punctuation_truecase_node.py` - Standalone punctuation / truecase cleanup for raw ASR text
- `nodes/subtitles/text_to_srt_builder_node.py` - Build SRT from transcript text plus timing data, or estimate timings from plain text
@@ -169,8 +172,9 @@
- `reporting.py` - Timing report generation
### Other Utils
- `utils/voice/discovery.py` - Voice file discovery with multi-path fallback
- `utils/downloads/unified_downloader.py` - Centralized HF download system
- `utils/voice/discovery.py` - Voice file discovery with multi-path fallback
- `utils/voice/character_logging.py` - Shared resolved voice labels and boxed prompt previews
- `utils/downloads/unified_downloader.py` - Centralized HF download system
- `utils/compatibility/transformers_patches.py` - transformers version compatibility patches
- `utils/compatibility/numba_compat.py` - Numba/Librosa Python 3.13+ compatibility
- `utils/ffmpeg_utils.py` - FFmpeg with graceful fallback
+52 -16
View File
@@ -7,7 +7,7 @@
[![Dynamic TOML Badge][version-shield]][version-url]
[![Ko-Fi](https://img.shields.io/badge/Ko--fi-F16061?style=for-the-badge&logo=ko-fi&logoColor=white)](https://ko-fi.com/diogogo)
# TTS Audio Suite v5.1.1
# TTS Audio Suite v5.4.4
[![ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/diogogo)
@@ -23,7 +23,7 @@ Subtitle workflows are still a core focus: the suite can transcribe to SRT, rebu
<!-- ENGINE_COMPARISON_START -->
## Quick Engine Comparison — 15 Engines
## Quick Engine Comparison — 17 Engines
| Engine | Languages | Size | Key Features |
|--------|-----------|------|--------------|
@@ -32,14 +32,16 @@ Subtitle workflows are still a core focus: the suite can transcribe to SRT, rebu
| **ChatterBox 23L** | 🌐 24 languages | ~4.3GB | 24 languages in single model, emotion tokens (v2 - doesn't work) |
| **VibeVoice** | 🇺🇸​🇨🇳​🇩🇪​🇪🇸​🇫🇷​🇮🇹 +21 | 5.4GB / 18GB | 90-min long-form, Native 4-speaker (Base models) |
| **Higgs Audio 2** | 🇺🇸​🇨🇳​🇩🇪​🇪🇸​🇰🇷 | ~9GB | 3 multi-speaker, CUDA graphs (55+ tokens/sec) |
| **Higgs Audio v3** | 🇿🇦​🇦🇱​🇺🇸​🇪🇸​🇮🇳​🇨🇳 +96 | ~8GB | Native inline emotion/style/prosody/SFX tags, Zero-shot voice cloning |
| **Higgs Audio v3** | 🌐 100+ languages | ~8GB | Native inline emotion/style/prosody/SFX tags, Zero-shot voice cloning |
| **IndexTTS-2** | 🇺🇸​🇨🇳​🇯🇵 | ~4.7GB | Emotion Control: 8 vectors, Text as reference |
| **CosyVoice3** | 🇺🇸​🇨🇳​🇯🇵​🇰🇷 | ~5.4GB | Paralinguistic tags |
| **Qwen3-TTS** | 🇺🇸​🇨🇳​🇩🇪​🇪🇸​🇫🇷​🇮🇹 +4 | ~3-6GB | Voice design, ASR (Automatic Speech Recognition) |
| **Granite ASR** | 🇺🇸​🇩🇪​🇪🇸​🇫🇷​🇯🇵​🇵🇹 | ~4.6GB | ASR (Automatic Speech Recognition), Native speaker attribution / diarization (plus model variant) |
| **Step Audio EditX** | 🇺🇸​🇨🇳​🇯🇵​🇰🇷 | ~7GB | Second Pass Speech Editing Node: 14 emotions, 32 speaking styles |
| **Echo-TTS** | 🇺🇸 | ~5.3GB + ~1.8GB | Diffusion-based (~30s best), Force Speaker KV (speaker drift control) |
| **Fish Audio S2 Pro** | 🌐 80+ languages | ~10.3GB / ~8.0GB | Free-form sub-word emotion/prosody tags, Zero-shot voice cloning and 80+ languages |
| **Dots TTS** | 🇺🇸​🇨🇳​🇩🇪​🇪🇸​🇫🇷​🇮🇹 +13 | ~6GB | Official auto language detect / language control, SOAR and MeanFlow distilled variants |
| **OmniVoice** | 🌐 600+ languages | ~3.7GB | 600+ language support, Instruction-based voice design |
| **MOSS-TTS** | 🇺🇸​🇨🇳​🇩🇪​🇪🇸​🇫🇷​🇮🇹 +10 | ~8.5GB tokenizer + ~6.1GB/17GB/18GB model | 20-language generation, Long-form generation (TTSD/Delay) |
| **RVC** | 🌐 Any | 100-300MB | Real-time VC, Integrated training workflow |
@@ -102,11 +104,17 @@ Builder English TTS TTS + ASR + VoiceDesign
|
|─── 🎓 Training Support Era 🧱 Runtime Isolation T5 Era
▼ |
v4.25 ──────────────► v4.26 ────────────► v5.00 ─────►
Apr 26 May 26 Jun 26
│ │ │
RVC MOSS-TTS Transformers 5
Model Training Higgs Audio v3 TTS
v4.25 ──────────────► v4.26 ────────────► v5.00 ───────────────┐
Apr 26 May 26 Jun 26 │
│ │ │ │
RVC MOSS-TTS Transformers 5 │
Model Training Higgs Audio v3 TTS │
│
▼
◄──────── v5.2 ◄─────────────── v5.1 ◄───────────┘
Mar 26 Jan 26
│ │
OmniVoice TTS Dots TTS
```
@@ -143,6 +151,7 @@ Start with the **[New Engine Guide Hub](docs/New%20Engines%20Guides/README.md)**
- [Step Audio EditX - LLM Audio Editing](#step-audio-editx---llm-audio-editing)
- [CosyVoice3 Multilingual Voice Cloning](#cosyvoice3-multilingual-voice-cloning)
- [Qwen3-TTS - 4 Model Types with Text-to-Voice Design](#qwen3-tts---4-model-types-with-text-to-voice-design)
- [OmniVoice + Visual Tag Builder](#omnivoice--visual-tag-builder)
- [Echo-TTS Voice Cloning](#echo-tts-voice-cloning)
- [Phoneme Text Normalizer](#phoneme-text-normalizer)
- [Multiline TTS Tag Editor and Per-Segment Parameter Switching](#multiline-tts-tag-editor-and-per-segment-parameter-switching)
@@ -188,7 +197,8 @@ Start with the **[New Engine Guide Hub](docs/New%20Engines%20Guides/README.md)**
- 🎓 **Integrated Model Training**
- 🎨 **Audio Post-Processing** → **[📖 Inline Edit Tags Guide](docs/INLINE_EDIT_TAGS_USER_GUIDE.md)**
- 🎭 **Character and Language Switching** → **[📖 Character Switching Guide](docs/CHARACTER_SWITCHING_GUIDE.md)**
- 🏷️ **Multiline TTS Tag Editor and Per-Segment Parameter Switching** → **[📖 Per-Segment Parameters](docs/PARAMETER_SWITCHING_GUIDE.md)** | **[📖 Multiline Tag Editor Guide](docs/MULTILINE_TTS_TAG_EDITOR_GUIDE.md)**
- 📐 **Visual Tag Builder** → Preset-driven visual tag and attribute assembly for OmniVoice and other tag-based text workflows
- 🏷️ **Multiline TTS Tag Editor and Per-Segment Parameter Switching** → **[📖 Per-Segment Parameters](docs/PARAMETER_SWITCHING_GUIDE.md)** | **[📖 Multiline Tag Editor Guide](docs/MULTILINE_TTS_TAG_EDITOR_GUIDE.md)** | **[📖 OmniVoice Tags Guide](docs/OMNIVOICE_TAGS_GUIDE.md)**
- 📝 **Intelligent Text Chunking** → **[📖 Text Chunking Guide](docs/TEXT_CHUNKING_GUIDE.md)**
- 🤐 **Vocal/Noise Removal** → **[📖 Complete Guide](docs/VOCAL_REMOVAL_GUIDE.md)**
- 🌊 **Audio Wave Analyzer** → **[📖 Complete Guide](docs/🌊_Audio_Wave_Analyzer-Complete_User_Guide.md)**
@@ -700,19 +710,19 @@ Both versions fully support character switching, language switching, and pause t
<details>
<summary><h3>IndexTTS-2 With Emotion Control</h3></summary>
**NEW in v4.9.0**: Revolutionary IndexTTS-2 engine with advanced emotion control and unified emotion architecture!
**NEW in v4.9.0**: Revolutionary IndexTTS-2 engine with advanced emotion control and dual-source emotion blending!
* **Unified Emotion Control**: Single `emotion_control` input supporting multiple emotion methods with intelligent priority system
* **Separate Emotion Inputs**: Connect vectors or Qwen text emotion to `emotion_control` and audio references to `emotion_audio`; both can be used together
* **Dynamic Text Emotion**: AI-powered QwenEmotion analysis with dynamic `{seg}` template processing for contextual per-segment emotions
* **Direct Audio Reference**: Use any audio file as emotion reference for natural emotional expression
* **Character Voices Integration**: Use Character Voices `opt_narrator` output as emotion reference with automatic detection
* **Direct Audio Reference**: Use any audio file on `emotion_audio` as an emotion reference for natural expression
* **Character Voices Integration**: Use Character Voices `opt_narrator` on `emotion_audio`, including per-character `[Character:emotion_ref]` references
* **8-Emotion Vector Control**: Manual precision control over Happy, Angry, Sad, Surprised, Afraid, Disgusted, Calm, and Melancholic emotions
* **Character Tag Emotions**: Per-character emotion control using `[Character:emotion_ref]` syntax (highest priority)
* **Character Tag Emotions**: Per-character audio emotion control using `[Character:emotion_ref]` syntax, blendable with vector/text emotion
* **Emotion Alpha Control**: Fine-tune emotion intensity from 0.0 (neutral) to 2.0 (maximum dramatic expression)
**Key Features:**
- **Emotion Priority System**: Character tags > Global emotion control with intelligent override handling
- **Emotion Blending**: Audio references and vector/text emotion are blended in IndexTTS-2's latent conditioning space; character tags select segment-local audio references
- **Dynamic Templates**: Use `{seg}` placeholder for contextual emotion analysis (e.g., "Worried parent speaking: {seg}")
- **Universal Compatibility**: Works with existing TTS Text and TTS SRT nodes seamlessly
- **Advanced Caching**: Stable audio content hashing for reliable cache hits across sessions
@@ -892,6 +902,24 @@ Description: "A deep, authoritative male voice with clear articulation"
</details>
<details>
<summary><h3>OmniVoice + Visual Tag Builder</h3></summary>
**NEW in v5.x**: OmniVoice is now integrated into the unified suite with native duration-aware SRT generation and a generalized visual tag workflow.
* **🌍 OmniVoice integration**: official model support with broad upstream language coverage
* **🎯 Native duration targeting**: SRT workflows can send target durations directly into OmniVoice instead of relying only on post-generation time stretching
* **⏱️ Precise segment control**: this is the first engine in the suite where segment duration can be meaningfully guided at generation time, making precise TTS timing far more practical
* **📺 Better SRT timing behavior**: subtitle generation can land much closer to target timings before any fallback timing correction, so stretch-to-fit has less work to do and results can stay more natural
* **📐 Visual Tag Builder**: reusable preset-driven visual node for assembling tag or attribute strings, originally added for OmniVoice voice-design prompting and now generalized for broader tag-based text workflows
* **🔊 Native inline non-verbal tags**: OmniVoice non-verbal controls are exposed in suite-default `<>` form like `<laughter>`, then converted internally for generation → **[📖 OmniVoice Tags Guide](docs/OMNIVOICE_TAGS_GUIDE.md)**
**Practical note:**
Use the built-in OmniVoice preset in **📐 Visual Tag Builder** for the canonical voice-design workflow. If you need a different tag schema, the same node now supports reusable custom presets with saved column order.
</details>
<details>
<summary><h3>MOSS-TTS - Local/Delay/TTSD Engine Family</h3></summary>
@@ -1044,6 +1072,10 @@ Use the new [Unified ✏️ ASR Transcribe + SRT Builder](example_workflows/Unif
Beyond character switching and language control, you can now override generation parameters (seed, temperature, CFG, speed, etc.) on a per-segment basis using inline tags. The new **🏷️ Multiline TTS Tag Editor** node makes building complex tags easier and more visual with:
- **Rich Text Editor**: Multiline editor with resizable font sizes (2-120px), multiple font families, and customizable UI scaling
- **Visual Tag Management**: Character/language/parameter dropdowns for quick selection, inline tag validation with syntax checking
- **Engine-Aware Inline Tags**: dedicated editor modes for Step Audio EditX, Higgs Audio v3, CosyVoice3, and OmniVoice
- **One-Click Tag Swapping**: click a character, language, audio reference, parameter, or supported native inline tag to open an engine-aware replacement palette; click again or press-drag-release to apply
- **IndexTTS-2 Emotion Editing**: insert vectors, named emotion values, presets, quoted text, and `{seg}` dynamic emotion controls directly from the Inline Tags panel
- **Safe Long-Tag Layout**: long bracket and angle tags wrap inside the editor instead of overflowing the text area; quoted emotion text remains directly editable
- **Preset System**: Save and load up to 3 preset configurations for rapid tag reuse
- **Keyboard Shortcuts**: Alt+L/C/P for tag insertion, Alt+1/2/3 for preset loading
- **History & Undo/Redo**: Full edit history with Alt+Z for undo (Alt+Shift+Z for redo)
@@ -1082,7 +1114,7 @@ This enables dynamic control over individual audio segments without modifying no
- **VibeVoice**: seed, temperature, cfg, top_p, top_k, inference_steps
- **IndexTTS-2**: seed, temperature, cfg, top_p, top_k, emotion_alpha
**📖 Guides:** [Per-Segment Parameter Switching](docs/PARAMETER_SWITCHING_GUIDE.md) | [Multiline TTS Tag Editor](docs/MULTILINE_TTS_TAG_EDITOR_GUIDE.md)
**📖 Guides:** [Per-Segment Parameter Switching](docs/PARAMETER_SWITCHING_GUIDE.md) | [Multiline TTS Tag Editor](docs/MULTILINE_TTS_TAG_EDITOR_GUIDE.md) | [OmniVoice Tags Guide](docs/OMNIVOICE_TAGS_GUIDE.md)
Perfect for:
@@ -1441,6 +1473,8 @@ For offline/manual setup:
| Granite ASR | `ComfyUI/models/TTS/granite_asr/` | ✅ | Granite ASR models; plus adds native diarization/timestamps, optional Qwen forced aligner reused lazily for timestamps/SRT fallback |
| Echo-TTS | `ComfyUI/models/TTS/echo-tts-base/` | ✅ | ~7.1GB total (base + dac); CC-BY-NC-SA |
| Dots TTS | `ComfyUI/models/TTS/dots_tts/` | ✅ | Official base / soar / mf checkpoints with tokenizer, vocoder, speaker encoder |
| Fish Audio S2 Pro | `ComfyUI/models/TTS/fish_audio_s2_pro/` | ✅ | Official BF16 or optional community FP8 checkpoint; the official checkpoint can be quantized on load with BNB INT8/NF4; main T5 environment with process teardown for Clear VRAM; Fish Audio Research License |
| OmniVoice | `ComfyUI/models/TTS/omnivoice/` | ✅ | Official OmniVoice model. Voice cloning in this suite requires explicit reference text. |
*Generated from [tts_audio_suite_engines.yaml](docs/Dev%20reports/tts_audio_suite_engines.yaml).*
@@ -1482,6 +1516,8 @@ Your support helps maintain and improve this project for the entire community!
| **🎨 Step Audio EditX - Audio Editor** | Step Audio EditX audio editing with inline edit tags | ✅ **New in v4.14** | [📁 JSON](example_workflows/🎨%20Step%20Audio%20EditX%20-%20Audio%20Editor%20+%20Inline%20Edit%20Tags.json) |
| **⚙️ Step Audio EditX Integration** | Step Audio EditX TTS engine with zero-shot voice cloning | ✅ **New in v4.14** | [📁 JSON](example_workflows/Step%20Audio%20EditX%20Integration.json) |
| **⚙️ Higgs Audio v3 Integration** | Higgs Audio v3 TTS with zero-shot voice cloning and native inline tags | ✅ **New in v4.27** | [📁 JSON](example_workflows/Higgs%20Audio%20v3%20Integration.json) |
| **⚙️ OmniVoice Engine Integration** | OmniVoice multilingual TTS with cloning, voice design, and native duration control | ✅ **New in v4.28** | [📁 JSON](example_workflows/OmniVoice%20Engine%20Integration.json) |
| **⚙️ Fish Audio S2 Pro Integration** | Fish S2 Pro multilingual cloning with native multi-speaker dialogue, inline control, and long-form generation | ✅ **New in v5.3** | [📁 JSON](example_workflows/Fish%20Audio%20S2%20integration.json) |
| **🌈 IndexTTS-2 Integration** | IndexTTS-2 engine with advanced emotion control | ✅ **New in v4.9** | [📁 JSON](example_workflows/🌈%20IndexTTS-2%20integration.json) |
| **📝 F5 TTS + Text Normalizer** | F5-TTS with multilingual text processing and phonemization | ✅ **New in v4.10.0** | [📁 JSON](example_workflows/F5%20TTS%20integration%20+%20📝%20Phoneme%20Text%20Normalizer.json) |
| **Qwen3 integration + ASR** | Qwen3-TTS voice generation with ASR transcription | ✅ **New in v4.21** | [📁 JSON](example_workflows/Qwen3%20integration%20+%20ASR.json) |
+95 -14
View File
@@ -294,12 +294,46 @@ __all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
WEB_DIRECTORY = "./web"
# Register API endpoint for widget data
def setup_api_routes():
"""Setup API routes for widget communication"""
try:
import json
from server import PromptServer
from aiohttp import web
def setup_api_routes():
"""Setup API routes for widget communication"""
try:
import json
import folder_paths
from server import PromptServer
from aiohttp import web
def _get_ui_data_dir():
base_dir = os.path.join(folder_paths.get_system_user_directory("tts_audio_suite"), "ui")
os.makedirs(base_dir, exist_ok=True)
return base_dir
def _get_omnivoice_preset_library_path():
return os.path.join(_get_ui_data_dir(), "omnivoice_instruction_builder_presets.json")
@PromptServer.instance.routes.get("/api/tts-audio-suite/index-tts-emotion-presets")
async def get_index_tts_emotion_presets_endpoint(request):
"""Return presets stored beside the IndexTTS resources under models/TTS."""
try:
from .utils.text.index_tts_emotion import load_emotion_presets
return web.json_response({"presets": load_emotion_presets()})
except Exception as e:
print(f"⚠️ Error retrieving IndexTTS emotion presets: {e}")
return web.json_response({"presets": {}, "error": str(e)}, status=500)
@PromptServer.instance.routes.post("/api/tts-audio-suite/index-tts-emotion-presets")
async def save_index_tts_emotion_presets_endpoint(request):
"""Atomically persist the IndexTTS emotion preset library."""
try:
from .utils.text.index_tts_emotion import save_emotion_presets
data = await request.json()
presets = data.get("presets", {})
path = save_emotion_presets(presets)
return web.json_response({"status": "success", "count": len(presets), "path": path})
except ValueError as e:
return web.json_response({"error": str(e)}, status=400)
except Exception as e:
print(f"⚠️ Error saving IndexTTS emotion presets: {e}")
return web.json_response({"status": "error", "error": str(e)}, status=500)
@PromptServer.instance.routes.get("/api/tts-audio-suite/available-characters")
async def get_available_characters_endpoint(request):
@@ -321,8 +355,8 @@ def setup_api_routes():
print(f"⚠️ Error retrieving available characters: {e}")
return web.json_response({"characters": [], "error": str(e)})
@PromptServer.instance.routes.get("/api/tts-audio-suite/available-languages")
async def get_available_languages_endpoint(request):
@PromptServer.instance.routes.get("/api/tts-audio-suite/available-languages")
async def get_available_languages_endpoint(request):
"""API endpoint to get available language codes from the canonical language mapper"""
try:
# Load language_mapper directly by file path to avoid package import issues
@@ -335,12 +369,59 @@ def setup_api_routes():
languages = sorted(set(language_mapper_module.LANGUAGE_ALIASES.values()))
return web.json_response({"languages": languages})
except Exception as e:
print(f"⚠️ Error retrieving available languages: {e}")
# Fallback list
return web.json_response({"languages": ["en", "de", "fr", "ja", "es", "it", "pt", "th", "no"], "error": str(e)})
@PromptServer.instance.routes.get("/api/tts-audio-suite/voice-input-devices")
async def get_voice_input_devices_endpoint(request):
print(f"⚠️ Error retrieving available languages: {e}")
# Fallback list
return web.json_response({"languages": ["en", "de", "fr", "ja", "es", "it", "pt", "th", "no"], "error": str(e)})
@PromptServer.instance.routes.get("/api/tts-audio-suite/omnivoice-presets")
async def get_omnivoice_presets_endpoint(request):
"""Return the persisted OmniVoice instruction builder preset library."""
try:
library_path = _get_omnivoice_preset_library_path()
if not os.path.exists(library_path):
return web.json_response({"presets": [], "builtinStates": {}, "builtinLayouts": {}})
with open(library_path, "r", encoding="utf-8") as f:
payload = json.load(f)
presets = payload.get("presets", []) if isinstance(payload, dict) else []
builtin_states = payload.get("builtinStates", {}) if isinstance(payload, dict) else {}
builtin_layouts = payload.get("builtinLayouts", {}) if isinstance(payload, dict) else {}
if not isinstance(presets, list):
presets = []
if not isinstance(builtin_states, dict):
builtin_states = {}
if not isinstance(builtin_layouts, dict):
builtin_layouts = {}
return web.json_response({"presets": presets, "builtinStates": builtin_states, "builtinLayouts": builtin_layouts})
except Exception as e:
print(f"⚠️ Error retrieving OmniVoice preset library: {e}")
return web.json_response({"presets": [], "builtinStates": {}, "builtinLayouts": {}, "error": str(e)}, status=500)
@PromptServer.instance.routes.post("/api/tts-audio-suite/omnivoice-presets")
async def save_omnivoice_presets_endpoint(request):
"""Persist the OmniVoice instruction builder preset library."""
try:
data = await request.json()
presets = data.get("presets", [])
builtin_states = data.get("builtinStates", {})
builtin_layouts = data.get("builtinLayouts", {})
if not isinstance(presets, list):
return web.json_response({"error": "presets must be a list"}, status=400)
if not isinstance(builtin_states, dict):
return web.json_response({"error": "builtinStates must be an object"}, status=400)
if not isinstance(builtin_layouts, dict):
return web.json_response({"error": "builtinLayouts must be an object"}, status=400)
library_path = _get_omnivoice_preset_library_path()
payload = {"presets": presets, "builtinStates": builtin_states, "builtinLayouts": builtin_layouts}
with open(library_path, "w", encoding="utf-8") as f:
json.dump(payload, f, ensure_ascii=False, indent=2)
return web.json_response({"status": "success", "count": len(presets)})
except Exception as e:
print(f"⚠️ Error saving OmniVoice preset library: {e}")
return web.json_response({"status": "error", "error": str(e)}, status=500)
@PromptServer.instance.routes.get("/api/tts-audio-suite/voice-input-devices")
async def get_voice_input_devices_endpoint(request):
"""Return input devices without risking a main-process PortAudio hang."""
try:
import json
+9 -5
View File
@@ -2,7 +2,7 @@
## Overview
Advanced multiline string editor node that extends ComfyUI's standard multiline widget with a sidebar containing context-aware controls for TTS-specific tags (character switching, parameters, pauses). The node features real-time tag generation, preset management, and intelligent syntax support based on the CHARACTER_SWITCHING_GUIDE and PARAMETER_SWITCHING_GUIDE.
Advanced multiline string editor node that extends ComfyUI's standard multiline widget with a sidebar containing context-aware controls for TTS-specific tags (character switching, parameters, pauses, and engine-native inline controls). The node features real-time tag generation, preset management, engine-aware quick swapping, and intelligent syntax support based on the CHARACTER_SWITCHING_GUIDE and PARAMETER_SWITCHING_GUIDE.
---
@@ -85,10 +85,14 @@ Advanced multiline string editor node that extends ComfyUI's standard multiline
- Split by paragraph breaks or sentence punctuation
- Batch apply parameters across multiple `[Character]text` blocks
#### Tag Inspector
- Show existing tags in selection/current line
- Checkbox UI to toggle tags on/off temporarily
- Quick-edit dialog for existing tag values
#### Tag Inspector
- Show existing tags in selection/current line
- Checkbox UI to toggle tags on/off temporarily
- Click a character, language, audio-reference, parameter, or supported engine-native inline tag to open a color-coded quick-swap palette
- Click a palette option once to keep the palette open, then click again to commit; press-and-hold, drag, and release selects in one gesture
- Palette choices follow the selected inline engine (including IndexTTS-2, Higgs Audio v3, Step Audio EditX, CosyVoice3, and OmniVoice)
- Quoted IndexTTS-2 text emotion tags remain direct editable text and do not open a replacement palette
- Long bracket and angle tags wrap inside the editor rather than overflowing horizontally
#### Auto-Formatting
- Button: "Auto-Format Tags" → organize tags consistently
+38 -31
View File
@@ -12,21 +12,26 @@
**⚠️ IMPORTANT: Use positional arguments, NOT --commit/--changelog flags**
```bash
# EASIEST: Just use 'patch' - script auto-increments the version
python3 scripts/bump_version_enhanced.py patch "<commit_desc>" "<changelog_desc>"
```powershell
# Windows: use the canonical ComfyUI environment for this project
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' patch "<commit_desc>" "<changelog_desc>"
# OR: Specify exact version if needed
python3 scripts/bump_version_enhanced.py <version> "<commit_desc>" "<changelog_desc>"
# OR: specify an exact version if needed
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' <version> "<commit_desc>" "<changelog_desc>"
```
```bash
# Linux/macOS: use the available Python 3 interpreter
python3 scripts/bump_version_enhanced.py patch "<commit_desc>" "<changelog_desc>"
```
### Examples
#### Multiline Format (Recommended Standard)
```bash
```powershell
# Patch release (bug fixes) - CORRECT FORMAT
python3 scripts/bump_version_enhanced.py 3.2.9 "Fix character alias resolution
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' 3.2.9 "Fix character alias resolution
Technical details:
- Fix parser bypassing character tags in single mode
@@ -37,8 +42,8 @@ Technical details:
- Improve character name recognition accuracy
- Better error handling for invalid character names"
# Minor release (new features) - CORRECT FORMAT
python3 scripts/bump_version_enhanced.py 3.3.0 "Add Higgs Audio 2 TTS engine
# Minor release (new features) - CORRECT FORMAT
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' 3.3.0 "Add Higgs Audio 2 TTS engine
Implementation details:
- Integrate boson_multimodal voice cloning system
@@ -50,7 +55,7 @@ Implementation details:
- Multiple built-in voice presets available"
# Major release (breaking changes) - CORRECT FORMAT
python3 scripts/bump_version_enhanced.py 4.0.0 "Complete unified architecture implementation
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' 4.0.0 "Complete unified architecture implementation
Breaking changes:
- Migrate all nodes to unified interface pattern
@@ -64,22 +69,22 @@ Breaking changes:
```
#### Auto-Increment Examples (Recommended)
```bash
# Auto-increment patch version (4.5.25 → 4.5.26) - CORRECT FORMAT
python3 scripts/bump_version_enhanced.py patch "Fix character parsing issues" "Fix character name handling in TTS generation"
```powershell
# Auto-increment patch version (4.5.25 → 4.5.26) - Windows
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' patch "Fix character parsing issues" "Fix character name handling in TTS generation"
# Auto-increment minor version (4.5.25 → 4.6.0) - CORRECT FORMAT
python3 scripts/bump_version_enhanced.py minor "Add new TTS engine support" "Add Higgs Audio 2 TTS engine with voice cloning"
# Auto-increment minor version (4.5.25 → 4.6.0) - Windows
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' minor "Add new TTS engine support" "Add Higgs Audio 2 TTS engine with voice cloning"
```
#### Single-Line Format (Only for Super Minor Changes)
```bash
python3 scripts/bump_version_enhanced.py patch "Fix typo in node tooltip" "Fix typo in audio analyzer tooltip"
```powershell
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' patch "Fix typo in node tooltip" "Fix typo in audio analyzer tooltip"
```
#### Dry-Run Preview (Test Before Committing)
```bash
python3 scripts/bump_version_enhanced.py patch "Fix preview issues" "Fix preview not reflecting filter parameters" --dry-run
```powershell
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' patch "Fix preview issues" "Fix preview not reflecting filter parameters" --dry-run
```
#### Auto-Categorization System
@@ -99,21 +104,22 @@ python3 scripts/bump_version_enhanced.py patch "Fix preview issues" "Fix preview
- **Commit**: Technical implementation details for developers
- **Changelog**: User-facing benefits and impacts
**Bash Syntax Notes:**
**Command Syntax Notes:**
- Multiline strings need proper quoting (opening quote on first line, closing quote on last line)
- Use `\` (backslash) for line continuation in bash commands
- The Windows command uses PowerShell's `&` call operator and the canonical project Python path
- The Linux/macOS command uses `python3`
- Don't add manual category prefixes like "Fixed:" - script handles categorization automatically!
### Interactive Mode (Recommended for Complex Changes)
```bash
python3 scripts/bump_version_enhanced.py 3.2.9 --interactive
```powershell
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' 3.2.9 --interactive
```
### Legacy Mode (Same Description for Both)
```bash
python3 scripts/bump_version_enhanced.py 3.2.9 "Fix bugs and improve stability"
```powershell
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' 3.2.9 "Fix bugs and improve stability"
```
### What the Script Does
@@ -239,13 +245,14 @@ git commit -m "Prepare for version bump"
- Use semantic versioning: `4.5.25` (not `v4.5.25` or `4.5`)
- Or use auto-increment: `patch`, `minor`, `major`
**Bash syntax errors with multiline**
- Make sure opening quote is on same line as `--commit` or `--changelog`
**Command syntax errors with multiline**
- Make sure opening quote is on same line as `--commit` or `--changelog`
- Make sure closing quote is on its own line
- Use `\` for line continuation
- On Windows, use the PowerShell command shown above
- On Linux/macOS, use `python3`
**Want to see what will happen before committing?**
```bash
```powershell
# Add --dry-run to preview changelog categorization
python3 scripts/bump_version_enhanced.py patch "description" "changelog" --dry-run
```
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' patch "description" "changelog" --dry-run
```
+12 -5
View File
@@ -2,10 +2,17 @@
This document tracks architectural issues and inconsistencies that need refactoring to improve modularity and reduce code duplication.
## SRT Processing Architecture Issues
### Problem: Inconsistent SRT Implementation Approaches
Different engines use completely different patterns for SRT processing:
## SRT Processing Architecture Issues
### Problem: VibeVoice Native Multi-Speaker SRT Uses Non-Contiguous Global Slots
- **Issue**: Later subtitles can request global `Speaker 2`/`Speaker 3` IDs while VibeVoice truncates and renumbers the supplied voice prompts from zero.
- **Impact**: A subtitle containing only later global speakers can bind the wrong reference or leave a requested speaker without a matching voice prompt.
- **Solution**: Use the global character map only to select references, then compact each subtitle to request-local speakers `0..N-1` with references in the same order.
- **Regression test**: Cover subtitle 1 `[Alice]...[Bob]...`, followed by subtitle 2 `[Bob]...[Rick]...`.
- **Priority**: High
### Problem: Inconsistent SRT Implementation Approaches
Different engines use completely different patterns for SRT processing:
1. **ChatterBox (Old)**: Uses `ChatterboxSRTTTSNode` class (should be processor)
2. **VibeVoice**: Uses proper `VibeVoiceSRTProcessor` class with full implementation
@@ -136,4 +143,4 @@ Each engine reimplements:
---
*Last Updated: 2025-01-XX*
*Add new issues to this file as they are discovered during development*
*Add new issues to this file as they are discovered during development*
@@ -55,7 +55,6 @@ categories:
languages:
- "47 languages"
notes: "Heavier multilingual fallback with the widest language coverage supported by the punctuators package."
model_layouts_markdown: |
# Auxiliary Model Layouts
@@ -131,6 +131,7 @@ engines:
community_finetunes: { supported: true, notes: "" }
vram_efficient: { supported: true, notes: "" }
speed_performance: { supported: true, notes: "Very Fast" }
reference_free_tts: { supported: false, notes: "" }
- id: chatterbox
name: ChatterBox
@@ -262,6 +263,7 @@ engines:
community_finetunes: { supported: true, notes: "" }
vram_efficient: { supported: true, notes: "" }
speed_performance: { supported: true, notes: "Fast" }
reference_free_tts: { supported: true, notes: "(default speaker)" }
- id: chatterbox-23l
name: ChatterBox 23L
@@ -346,6 +348,7 @@ engines:
community_finetunes: { supported: true, notes: "" }
vram_efficient: { supported: true, notes: "" }
speed_performance: { supported: true, notes: "Fast" }
reference_free_tts: { supported: true, notes: "(default speaker)" }
- id: vibevoice
name: VibeVoice
@@ -443,6 +446,7 @@ engines:
community_finetunes: { supported: true, notes: "KugelAudio, Hindi" }
vram_efficient: { supported: "partial", notes: "(5-18GB)" }
speed_performance: { supported: "partial", notes: "" }
reference_free_tts: { supported: true, notes: "(zero-shot / default speaker)" }
- id: higgs-audio-2
name: Higgs Audio 2
@@ -524,6 +528,7 @@ engines:
community_finetunes: { supported: false, notes: "" }
vram_efficient: { supported: "partial", notes: "(9GB)" }
speed_performance: { supported: "partial", notes: "" }
reference_free_tts: { supported: true, notes: "(basic TTS if no narrator/reference is provided)" }
- id: higgs-audio-v3
name: Higgs Audio v3
@@ -531,6 +536,8 @@ engines:
size: "~8GB"
license: "Boson Higgs Audio v3 Research and Non-Commercial License"
commercial: false
language_summary_compact: "🌐 100+ languages"
language_summary_full: "100+"
capabilities:
tts: true
@@ -666,6 +673,7 @@ engines:
community_finetunes: { supported: false, notes: "" }
vram_efficient: { supported: "partial", notes: "(~8-10GB)" }
speed_performance: { supported: "partial", notes: "CUDA recommended" }
reference_free_tts: { supported: true, notes: "(zero-shot)" }
- id: indextts-2
name: IndexTTS-2
@@ -744,6 +752,7 @@ engines:
community_finetunes: { supported: false, notes: "" }
vram_efficient: { supported: "partial", notes: "(9-12GB)" }
speed_performance: { supported: "partial", notes: "" }
reference_free_tts: { supported: false, notes: "" }
- id: cosyvoice3
name: CosyVoice3
@@ -809,6 +818,7 @@ engines:
community_finetunes: { supported: false, notes: "" }
vram_efficient: { supported: true, notes: "(5.4GB)" }
speed_performance: { supported: true, notes: "Fast" }
reference_free_tts: { supported: true, notes: "(cross-lingual or instruct mode)" }
- id: qwen3-tts
name: Qwen3-TTS
@@ -922,6 +932,7 @@ engines:
community_finetunes: { supported: false, notes: "" }
vram_efficient: { supported: true, notes: "(3-6GB)" }
speed_performance: { supported: "partial", notes: "" }
reference_free_tts: { supported: true, notes: "(Base default voice or CustomVoice presets)" }
- id: granite-asr
name: Granite ASR
@@ -1010,6 +1021,7 @@ engines:
community_finetunes: { supported: false, notes: "" }
vram_efficient: { supported: true, notes: "(~4.6GB)" }
speed_performance: { supported: "partial", notes: "Moderate" }
reference_free_tts: { supported: null, notes: "N/A" }
- id: step-editx
name: Step Audio EditX
@@ -1083,6 +1095,7 @@ engines:
community_finetunes: { supported: false, notes: "" }
vram_efficient: { supported: "partial", notes: "(7GB)" }
speed_performance: { supported: "partial", notes: "" }
reference_free_tts: { supported: false, notes: "(clone mode only)" }
- id: echo-tts
name: Echo-TTS
@@ -1155,6 +1168,76 @@ engines:
community_finetunes: { supported: false, notes: "" }
vram_efficient: { supported: "partial", notes: "(~7GB total)" }
speed_performance: { supported: true, notes: "Fast (diffusion, realtime-capable)" }
reference_free_tts: { supported: false, notes: "(reference audio required)" }
- id: fish-audio-s2-pro
name: Fish Audio S2 Pro
models: "S2 Pro 4B / FP8"
size: "~10.3GB / ~8.0GB"
license: "Fish Audio Research License"
commercial: false
language_summary_full: "80+ languages"
language_summary_compact: "🌐 80+ languages"
capabilities:
tts: true
srt: true
vc: false
asr: false
training: false
runtime_isolation:
default_mode: "main_environment"
main_environment: true
shared_runtime: false
dedicated_runtime: false
special_features:
- "Free-form sub-word emotion/prosody tags"
- "Zero-shot voice cloning and 80+ languages"
- "Native multi-speaker and multi-turn dialogue with dynamic speaker references"
- "Optional per-segment custom character switching"
- "Configurable 4K-32K native context with reduced KV-cache VRAM"
- "Optional community FP8 weight-only checkpoint with BF16 activations"
- "Optional on-the-fly BitsAndBytes INT8/NF4 for the official checkpoint"
model_sources:
- component: "S2 Pro"
source_name: "fishaudio/s2-pro"
source_url: "https://huggingface.co/fishaudio/s2-pro"
size: "~10.3GB"
auto_download: true
notes: "Official 4B model and codec; BNB INT8/NF4 are optional load-time quantization modes that reuse these files; non-commercial license"
- component: "S2 Pro FP8"
source_name: "drbaph/s2-pro-fp8"
source_url: "https://huggingface.co/drbaph/s2-pro-fp8"
size: "~8.0GB"
auto_download: true
notes: "Community weight-only FP8 checkpoint; BF16 activations; RTX 4090/5090-class CUDA GPU required"
languages:
en: { supported: true, flag: "🇺🇸", notes: "Tier 1" }
zh: { supported: true, flag: "🇨🇳", notes: "Tier 1" }
ja: { supported: true, flag: "🇯🇵", notes: "Tier 1" }
ko: { supported: true, flag: "🇰🇷", notes: "Tier 2" }
es: { supported: true, flag: "🇪🇸", notes: "Tier 2" }
pt: { supported: true, flag: "🇵🇹", notes: "Tier 2" }
ar: { supported: true, flag: "🇦🇪", notes: "Tier 2" }
ru: { supported: true, flag: "🇷🇺", notes: "Tier 2" }
fr: { supported: true, flag: "🇫🇷", notes: "Tier 2" }
de: { supported: true, flag: "🇩🇪", notes: "Tier 2" }
features:
voice_cloning: { supported: true, notes: "Reference audio plus exact transcript" }
native_multi_speaker: { supported: true, notes: "Native dialogue is default; optional custom mode generates each [Character] segment independently as local speaker 0" }
voice_conversion: { supported: false, notes: "Not exposed by official S2 TTS inference" }
asr_transcribe: { supported: false, notes: "" }
emotion_control: { supported: true, notes: "Free-form inline natural-language tags" }
native_long_form: { supported: true, notes: "Configurable 4K-32K native context; suite text chunking is bypassed" }
community_finetunes: { supported: false, notes: "No suite integration" }
vram_efficient: { supported: "partial", notes: "8K context measured at ~15.2GB BF16, ~11.2GB FP8, ~11.2GB BNB INT8, or ~8.9GB BNB NF4; BF16 codec and activations; BNB is a load-time option for the official checkpoint" }
speed_performance: { supported: true, notes: "Main-environment subprocess with reliable teardown; local compile measurements: ~40 it/s BF16, ~11.8 it/s NF4 at ~8.9GB VRAM, and ~3.7 it/s INT8 at ~11.2GB VRAM; quality comparison pending" }
reference_free_tts: { supported: true, notes: "Reference audio is optional" }
- id: dots-tts
name: Dots TTS
@@ -1233,6 +1316,78 @@ engines:
community_finetunes: { supported: false, notes: "" }
vram_efficient: { supported: "partial", notes: "(main env works; 2B-class model is not lightweight)" }
speed_performance: { supported: "partial", notes: "Moderate; mf variant is faster" }
reference_free_tts: { supported: true, notes: "(default speaker)" }
- id: omnivoice
name: OmniVoice
models: "OmniVoice"
size: "~3.7GB"
license: "Apache-2.0"
commercial: true
language_summary_compact: "🌐 600+ languages"
language_summary_full: "600+"
capabilities:
tts: true
srt: true
vc: false
asr: false
training: false
special_features:
- "600+ language support"
- "Instruction-based voice design"
- "Upstream long-form chunk orchestration"
- "Inline non-verbal tags and pronunciation overrides"
model_sources:
- component: "OmniVoice"
source_name: "k2-fsa/OmniVoice"
source_url: "https://huggingface.co/k2-fsa/OmniVoice"
size: "~3.7GB"
auto_download: true
notes: "Official main model including bundled audio_tokenizer assets"
languages:
en: { supported: true, flag: "🇺🇸", notes: "" }
zh: { supported: true, flag: "🇨🇳", notes: "" }
de: { supported: true, flag: "🇩🇪", notes: "" }
es: { supported: true, flag: "🇪🇸", notes: "" }
fr: { supported: true, flag: "🇫🇷", notes: "" }
it: { supported: true, flag: "🇮🇹", notes: "" }
ja: { supported: true, flag: "🇯🇵", notes: "" }
ko: { supported: true, flag: "🇰🇷", notes: "" }
ru: { supported: true, flag: "🇷🇺", notes: "" }
pt: { supported: true, flag: "🇵🇹", notes: "(generic PT; official language space is much broader than this matrix)" }
pl: { supported: true, flag: "🇵🇱", notes: "" }
hi: { supported: true, flag: "🇮🇳", notes: "" }
ar: { supported: true, flag: "🇦🇪", notes: "" }
tr: { supported: true, flag: "🇹🇷", notes: "" }
th: { supported: true, flag: "🇹🇭", notes: "" }
no: { supported: true, flag: "🇳🇴", notes: "" }
vi: { supported: true, flag: "🇻🇳", notes: "" }
hy: { supported: true, flag: "🇦🇲", notes: "" }
ka: { supported: true, flag: "🇬🇪", notes: "" }
da: { supported: true, flag: "🇩🇰", notes: "" }
fi: { supported: true, flag: "🇫🇮", notes: "" }
el: { supported: true, flag: "🇬🇷", notes: "" }
he: { supported: true, flag: "🇮🇱", notes: "" }
ms: { supported: true, flag: "🇲🇾", notes: "" }
nl: { supported: true, flag: "🇳🇱", notes: "" }
sv: { supported: true, flag: "🇸🇪", notes: "" }
sw: { supported: true, flag: "🇰🇪", notes: "" }
features:
voice_cloning: { supported: true, notes: "" }
native_multi_speaker: { supported: false, notes: "" }
voice_conversion: { supported: false, notes: "" }
asr_transcribe: { supported: false, notes: "(no standalone ASR node)" }
emotion_control: { supported: "partial", notes: "(voice-design instruct + inline non-verbal tags)" }
native_long_form: { supported: true, notes: "(uses upstream audio_chunk_duration / audio_chunk_threshold orchestration; bypasses suite char-based chunk splitting)" }
community_finetunes: { supported: false, notes: "" }
vram_efficient: { supported: "partial", notes: "(main env works; 2B-class model is not lightweight)" }
speed_performance: { supported: true, notes: "Fast; upstream reports sub-realtime RTF" }
reference_free_tts: { supported: true, notes: "(native default voice; instruct also works without narrator)" }
- id: moss-tts
name: MOSS-TTS
@@ -1321,6 +1476,7 @@ engines:
community_finetunes: { supported: true, notes: "(LoRA adapter inference supported; initial integrated LoRA training support added for MOSS-TTS Delay 8B)" }
vram_efficient: { supported: "partial", notes: "(Local 1.7B smaller; tokenizer is large)" }
speed_performance: { supported: true, notes: "Fast with CUDA/FlashAttention" }
reference_free_tts: { supported: true, notes: "(direct TTS and prompt-only generation)" }
- id: rvc
name: RVC
@@ -1414,6 +1570,7 @@ engines:
community_finetunes: { supported: true, notes: "" }
vram_efficient: { supported: true, notes: "" }
speed_performance: { supported: true, notes: "Fast" }
reference_free_tts: { supported: null, notes: "N/A" }
# Language metadata for reference
language_metadata:
@@ -1531,6 +1688,7 @@ feature_metadata:
community_finetunes: { name: "Community Finetunes", display: "**Community Finetunes**" }
vram_efficient: { name: "VRAM Efficient", display: "**VRAM Efficient**" }
speed_performance: { name: "Speed/Performance", display: "**Speed/Performance**" }
reference_free_tts: { name: "No Narrator Required", display: "**No Narrator Required**" }
# Table notes for additional context
table_notes:
@@ -1540,6 +1698,7 @@ table_notes:
- "**Higgs Audio v3**: Official model card lists 102 languages total: 85 polished-production languages and 17 usable but less polished languages."
- "**IndexTTS-2**: Trained on 55K+ hours - ZH, EN, JA primary."
- "**MOSS-TTS**: Official list includes ZH, EN, DE, ES, FR, JA, IT, HU, KO, RU, FA, AR, PL, PT, CS, DA, SV, EL, TR; HU/FA/CS are not separate columns in this matrix."
- "**OmniVoice**: Official model supports 600+ languages; this matrix only shows the suite's comparison subset."
- "**RVC**: Language-agnostic voice conversion/post-processing; it is marked supported for every language row."
- "Qwen3-TTS: Supports pt-BR only with instruction (voice design or harcoded voices). Base can't do instructions, so it will always ouput pt-PT."
@@ -1604,6 +1763,14 @@ readme_model_download_table:
primary_model_path: "ComfyUI/models/TTS/dots_tts/"
auto_download: "✅"
notes: "Official base / soar / mf checkpoints with tokenizer, vocoder, speaker encoder"
- engine: "Fish Audio S2 Pro"
primary_model_path: "ComfyUI/models/TTS/fish_audio_s2_pro/"
auto_download: "✅"
notes: "Official BF16 or optional community FP8 checkpoint; the official checkpoint can be quantized on load with BNB INT8/NF4; main T5 environment with process teardown for Clear VRAM; Fish Audio Research License"
- engine: "OmniVoice"
primary_model_path: "ComfyUI/models/TTS/omnivoice/"
auto_download: "✅"
notes: "Official OmniVoice model. Voice cloning in this suite requires explicit reference text."
model_layouts_markdown: |
# Model Folder Layouts
@@ -1924,6 +2091,33 @@ model_layouts_markdown: |
- Both components are required and auto-downloaded on first use.
- License: CC-BY-NC-SA (non-commercial).
## Fish Audio S2 Pro
```text
ComfyUI/models/TTS/fish_audio_s2_pro/
├── codec.pth
├── config.json
├── model-00001-of-00002.safetensors
├── model-00002-of-00002.safetensors
├── model.safetensors.index.json
└── tokenizer.json
```
Optional FP8 variant:
```text
ComfyUI/models/TTS/fish_audio_s2_pro_fp8/
├── codec.pth
├── config.json
├── model.safetensors
├── quantization_info.json
└── tokenizer.json
```
The complete official repository metadata and tokenizer files are downloaded alongside these files. License: Fish Audio Research License (non-commercial without a separate commercial license).
The `s2-pro-bnb-int8` and `s2-pro-bnb-nf4` options reuse `fish_audio_s2_pro/` and quantize its official checkpoint while loading. They do not download another model copy and require `bitsandbytes`.
## Dots TTS
```text
@@ -1957,3 +2151,32 @@ model_layouts_markdown: |
- Downloads into a dedicated `dots_tts/` folder.
- Native sample rate is 48kHz.
- Main-environment support works on Transformers 5; on Windows, `normalize_text` falls back to no-op if `WeTextProcessing` is unavailable.
## OmniVoice
```text
ComfyUI/models/TTS/omnivoice/
└── OmniVoice/
```
Main model contents:
```text
ComfyUI/models/TTS/omnivoice/OmniVoice/
├── chat_template.jinja
├── config.json
├── model.safetensors
├── tokenizer.json
├── tokenizer_config.json
└── audio_tokenizer/
├── config.json
├── model.safetensors
└── preprocessor_config.json
```
Notes:
- Downloads into a dedicated `omnivoice/` folder.
- Native sample rate is 24kHz.
- The main OmniVoice model stays in the main Transformers 5 environment.
- Voice cloning requires reference audio plus explicit reference text in this suite.
+3 -1
View File
@@ -9,14 +9,16 @@
| **ChatterBox 23L** | Main | v1, v2, Vietnamese (Viterbox), Egyptian Arabic (oddadmix) | ~4.3GB | ✅ | ✅ | ✅ | ❌ | ❌ | MIT | 24 languages in single model, emotion tokens (v2 - doesn't work) | 25 |
| **VibeVoice** | Shared | 1.5B, 7B, KugelAudio-0 (7B), kugel-2 (7B), Hindi-1.5B/7B | 5.4GB / 18GB | ✅ | ✅ | ❌ | ❌ | ❌ | MIT (research-only per model card) | 90-min long-form, Native 4-speaker (Base models), Multilingual (KugelAudio variants), 4-bit quantization | 27 |
| **Higgs Audio 2** | Shared | 3B | ~9GB | ✅ | ✅ | ❌ | ❌ | ❌ | Boson Higgs Audio 2 Community License | 3 multi-speaker, CUDA graphs (55+ tokens/sec) | 5 |
| **Higgs Audio v3** | Main | 4B | ~8GB | ✅ | ✅ | ❌ | ❌ | ❌ | Boson Higgs Audio v3 Research and Non-Commercial License | Native inline emotion/style/prosody/SFX tags, Zero-shot voice cloning, 100+ language support | 102 |
| **Higgs Audio v3** | Main | 4B | ~8GB | ✅ | ✅ | ❌ | ❌ | ❌ | Boson Higgs Audio v3 Research and Non-Commercial License | Native inline emotion/style/prosody/SFX tags, Zero-shot voice cloning, 100+ language support | 100+ |
| **IndexTTS-2** | Main | IndexTTS-2 | ~4.7GB | ✅ | ✅ | ❌ | ❌ | ❌ | bilibili Model Use License | Emotion Control: 8 vectors, Text as reference, Audio as reference | 3 |
| **CosyVoice3** | Main | 0.5B, 0.5B-RL | ~5.4GB | ✅ | ✅ | ✅ | ❌ | ❌ | Apache-2.0 | Paralinguistic tags | 4 |
| **Qwen3-TTS** | Shared | 0.6B, 1.7B (CustomVoice/VoiceDesign/Base) | ~3-6GB | ✅ | ✅ | ❌ | ✅ | ❌ | Apache-2.0 | Voice design, ASR (Automatic Speech Recognition) | 10 |
| **Granite ASR** | Main | granite-4.0-1b-speech, granite-speech-4.1-2b, granite-speech-4.1-2b-plus | ~4.6GB | ❌ | ✅ | ❌ | ✅ | ❌ | Apache-2.0 | ASR (Automatic Speech Recognition), Native speaker attribution / diarization (plus model variant), Native word-level timestamps (plus model variant), Custom timestamps/SRT via reused Qwen forced aligner, Speech translation (experimental), Optional forced aligner auto-routed through shared legacy T4 runtime | 6 |
| **Step Audio EditX** | Main | 3B LLM + CosyVoice | ~7GB | ✅ | ✅ | ❌ | ❌ | ❌ | Apache-2.0 (verify before commercial use) | Second Pass Speech Editing Node: 14 emotions, 32 speaking styles, Paralinguistic effects | 4 |
| **Echo-TTS** | Main | echo-tts-base + fish-s1-dac-min | ~5.3GB + ~1.8GB | ✅ | ✅ | ❌ | ❌ | ❌ | CC-BY-NC-SA-4.0 | Diffusion-based (~30s best), Force Speaker KV (speaker drift control) | 1 |
| **Fish Audio S2 Pro** | Main | S2 Pro 4B / FP8 | ~10.3GB / ~8.0GB | ✅ | ✅ | ❌ | ❌ | ❌ | Fish Audio Research License | Free-form sub-word emotion/prosody tags, Zero-shot voice cloning and 80+ languages, Native multi-speaker and multi-turn dialogue with dynamic speaker references, Optional per-segment custom character switching, Configurable 4K-32K native context with reduced KV-cache VRAM, Optional community FP8 weight-only checkpoint with BF16 activations, Optional on-the-fly BitsAndBytes INT8/NF4 for the official checkpoint | 80+ languages |
| **Dots TTS** | Main | dots.tts-base, dots.tts-soar, dots.tts-mf | ~6GB | ✅ | ✅ | ❌ | ❌ | ❌ | Apache-2.0 | Official auto language detect / language control, SOAR and MeanFlow distilled variants | 19 |
| **OmniVoice** | Main | OmniVoice | ~3.7GB | ✅ | ✅ | ❌ | ❌ | ❌ | Apache-2.0 | 600+ language support, Instruction-based voice design, Upstream long-form chunk orchestration, Inline non-verbal tags and pronunciation overrides | 600+ |
| **MOSS-TTS** | Main | Local 1.7B, Delay 8B, TTSD 8B | ~8.5GB tokenizer + ~6.1GB/17GB/18GB model | ✅ | ✅ | ❌ | ❌ | ✅ | Apache-2.0 | 20-language generation, Long-form generation (TTSD/Delay), Native 1-5 speaker TTSD dialogue, Duration token hint, Local/Delay/TTSD variants, Initial integrated LoRA training workflow (Delay 8B) | 16 |
| **RVC** | Main | Community .pth | 100-300MB | ❌ | ❌ | ✅ | ❌ | ✅ | MIT (framework); community models vary | Real-time VC, Integrated training workflow, Pitch shift (±14), 6 HuBERT models, Language-independent | Any |
+15 -14
View File
@@ -2,17 +2,18 @@
## Feature Comparison Matrix
| Feature | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Dots TTS | MOSS-TTS | RVC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| **TTS** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ |
| **SRT** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
| **Voice Conversion** | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| **ASR (Transcribe)** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **Training** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
| **Voice Cloning** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (Base model) | ❌ | ✅ | ✅ | ✅ | ✅ | ⚠️ (needs training) |
| **Native Multi-Speaker** | ❌ | ❌ | ❌ | ✅ (Base only, Kugel uses fallback) | ✅ | ❌ | ❌ | ❌ | ❌ | ⚠️ (Plus variant speaker attribution / diarization) | ❌ | ❌ | ❌ | ✅ (TTSD v1.0; 1-5 speakers) | ❌ |
| **Emotion Control** | ❌ | ❌ | ⚠️ (v2 tags - doesn't work) | ❌ | ⚠️ (via prompt) | ✅ (native inline tags) | ✅ (8 emotions) | ⚠️ (via instruct) | ⚠️ (via instruct) | ❌ | ✅ (14 emotions) | ❌ | ❌ | ❌ | ❌ |
| **Native Long-form** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (TTSD/Delay long-form; use chunk orchestration for very long inputs) | N/A |
| **Community Finetunes** | ✅ | ✅ | ✅ | ✅ KugelAudio, Hindi | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (LoRA adapter inference supported; initial integrated LoRA training support added for MOSS-TTS Delay 8B) | ✅ |
| **VRAM Efficient** | ✅ | ✅ | ✅ | ⚠️ (5-18GB) | ⚠️ (9GB) | ⚠️ (~8-10GB) | ⚠️ (9-12GB) | ✅ (5.4GB) | ✅ (3-6GB) | ✅ (~4.6GB) | ⚠️ (7GB) | ⚠️ (~7GB total) | ⚠️ (main env works; 2B-class model is not lightweight) | ⚠️ (Local 1.7B smaller; tokenizer is large) | ✅ |
| **Speed/Performance** | ✅ Very Fast | ✅ Fast | ✅ Fast | ⚠️ | ⚠️ | ⚠️ CUDA recommended | ⚠️ | ✅ Fast | ⚠️ | ⚠️ Moderate | ⚠️ | ✅ Fast (diffusion, realtime-capable) | ⚠️ Moderate; mf variant is faster | ✅ Fast with CUDA/FlashAttention | ✅ Fast |
| Feature | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | OmniVoice | MOSS-TTS | RVC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| **TTS** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
| **SRT** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
| **Voice Conversion** | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| **ASR (Transcribe)** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **Training** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
| **Voice Cloning** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (Base model) | ❌ | ✅ | ✅ | ✅ Reference audio plus exact transcript | ✅ | ✅ | ✅ | ⚠️ (needs training) |
| **Native Multi-Speaker** | ❌ | ❌ | ❌ | ✅ (Base only, Kugel uses fallback) | ✅ | ❌ | ❌ | ❌ | ❌ | ⚠️ (Plus variant speaker attribution / diarization) | ❌ | ❌ | ✅ Native dialogue is default; optional custom mode generates each [Character] segment independently as local speaker 0 | ❌ | ❌ | ✅ (TTSD v1.0; 1-5 speakers) | ❌ |
| **Emotion Control** | ❌ | ❌ | ⚠️ (v2 tags - doesn't work) | ❌ | ⚠️ (via prompt) | ✅ (native inline tags) | ✅ (8 emotions) | ⚠️ (via instruct) | ⚠️ (via instruct) | ❌ | ✅ (14 emotions) | ❌ | ✅ Free-form inline natural-language tags | ❌ | ⚠️ (voice-design instruct + inline non-verbal tags) | ❌ | ❌ |
| **Native Long-form** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Configurable 4K-32K native context; suite text chunking is bypassed | ❌ | ✅ (uses upstream audio_chunk_duration / audio_chunk_threshold orchestration; bypasses suite char-based chunk splitting) | ✅ (TTSD/Delay long-form; use chunk orchestration for very long inputs) | N/A |
| **Community Finetunes** | ✅ | ✅ | ✅ | ✅ KugelAudio, Hindi | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (LoRA adapter inference supported; initial integrated LoRA training support added for MOSS-TTS Delay 8B) | ✅ |
| **VRAM Efficient** | ✅ | ✅ | ✅ | ⚠️ (5-18GB) | ⚠️ (9GB) | ⚠️ (~8-10GB) | ⚠️ (9-12GB) | ✅ (5.4GB) | ✅ (3-6GB) | ✅ (~4.6GB) | ⚠️ (7GB) | ⚠️ (~7GB total) | ⚠️ 8K context measured at ~15.2GB BF16, ~11.2GB FP8, ~11.2GB BNB INT8, or ~8.9GB BNB NF4; BF16 codec and activations; BNB is a load-time option for the official checkpoint | ⚠️ (main env works; 2B-class model is not lightweight) | ⚠️ (main env works; 2B-class model is not lightweight) | ⚠️ (Local 1.7B smaller; tokenizer is large) | ✅ |
| **Speed/Performance** | ✅ Very Fast | ✅ Fast | ✅ Fast | ⚠️ | ⚠️ | ⚠️ CUDA recommended | ⚠️ | ✅ Fast | ⚠️ | ⚠️ Moderate | ⚠️ | ✅ Fast (diffusion, realtime-capable) | ✅ Main-environment subprocess with reliable teardown; local compile measurements: ~40 it/s BF16, ~11.8 it/s NF4 at ~8.9GB VRAM, and ~3.7 it/s INT8 at ~11.2GB VRAM; quality comparison pending | ⚠️ Moderate; mf variant is faster | ✅ Fast; upstream reports sub-realtime RTF | ✅ Fast with CUDA/FlashAttention | ✅ Fast |
| **No Narrator Required** | ❌ | ✅ (default speaker) | ✅ (default speaker) | ✅ (zero-shot / default speaker) | ✅ (basic TTS if no narrator/reference is provided) | ✅ (zero-shot) | ❌ | ✅ (cross-lingual or instruct mode) | ✅ (Base default voice or CustomVoice presets) | N/A | ❌ | ❌ | ✅ Reference audio is optional | ✅ (default speaker) | ✅ (native default voice; instruct also works without narrator) | ✅ (direct TTS and prompt-only generation) | N/A |
+35
View File
@@ -0,0 +1,35 @@
# Fish Audio S2 Pro Inline Tags
Use the suite's public angle-bracket syntax for Fish's free-form instructions:
```text
<whisper in small voice>Hello. <professional broadcast tone>Good evening.
```
The Fish adapter converts these to native `[...]` instructions only when text reaches the Fish engine. Do not write Fish tags with square brackets in suite text because `[Character]` is reserved for character switching.
Fish also supports normal suite tags such as `[pause:500ms]`, character switching, and per-segment parameters. Those are processed by the suite before Fish inference.
## Language Prompting
Fish has no native language dropdown or parameter. The engine can instead prepend a natural inline instruction from the suite's resolved segment language:
- `language_prompting = Auto Inline Tag`: non-English resolved languages become tags such as `<German>` or `<French>`
- `language_prompting = Off`: no automatic language instruction is added
English is only added when the user explicitly requested it with a language tag such as `[en:Bob]` or `[English:Bob]`. Implicit/default English stays untagged.
## Character Switching Modes
The engine defaults to `Native Multi-Speaker`. All `[Character]` turns in a generated block are sent through one Fish dialogue request, preserving native multi-turn context and long-form behavior.
Select `Custom Character Switching` to generate every parsed character segment independently. Each call uses only that character's reference and is remapped to Fish speaker 0. This can reduce speaker leakage, but it requires more calls and does not preserve context between character turns. SRT subtitle boundaries and timing are preserved in both modes.
The engine UI separates checkpoint choice from load-time quantization:
- `model_variant`: `s2-pro` or the separate `s2-pro-fp8` checkpoint
- `quantization`: `none`, `bnb_int8`, or `bnb_nf4` for on-the-fly quantization of the official `s2-pro` checkpoint
The BNB options reuse the official files and do not download a second model copy.
The S2 Pro weights use the Fish Audio Research License: research and non-commercial use are allowed; commercial use requires a separate Fish Audio license.
+138 -28
View File
@@ -12,15 +12,27 @@ IndexTTS-2 supports multiple emotion control methods that can be combined for so
- **Text Emotion**: AI-powered QwenEmotion analysis from text descriptions with dynamic templates
- **Character Tag Emotions**: Per-character emotion control using `[Character:emotion_ref]` syntax
## Emotion Control Priority
You can only connect to the Engine node one source of control emotion: Either audio, text, or vectors.
When using tags on the text iself, **Character tag emotions** (highest priority) - `[Alice:angry_bob]` overrides all other emotion control settings for that character segment
## Emotion Control Inputs and Blending
The IndexTTS-2 Engine has two emotion inputs:
- **`emotion_control`**: vector controls or Qwen text emotion
- **`emotion_audio`**: an audio reference from an AUDIO or Character Voices node
Both inputs may be connected at the same time. Text emotion is analyzed into an
8-value vector; audio emotion remains an audio-derived conditioning signal. The
engine blends the two signals in its latent emotion-conditioning space rather
than converting the audio into the eight visible vector values.
For backward compatibility, the original `emotion_control` socket still accepts
legacy audio connections, but new workflows should use `emotion_audio` for
audio references. A character tag such as `[Alice:angry_bob]` supplies a
segment-local audio reference and can also be combined with vector/text emotion
for that segment.
## Method 1: Direct Audio Reference
Connect any audio file directly to the IndexTTS-2 Engine's `emotion_control` input.
Connect any audio file directly to the IndexTTS-2 Engine's `emotion_audio` input.
**How it works:**
@@ -37,7 +49,7 @@ Connect any audio file directly to the IndexTTS-2 Engine's `emotion_control` inp
**Example:**
```
AUDIO node → IndexTTS-2 Engine (emotion_control)
AUDIO node → IndexTTS-2 Engine (emotion_audio)
```
## Method 2: Character Voices Audio Reference
@@ -48,7 +60,7 @@ Use the `opt_narrator` output from the 🎭 Character Voices node as an emotion
1. Add a 🎭 Character Voices node
2. Select a voice with the desired emotional expression
3. Connect `opt_narrator` output to IndexTTS-2 Engine `emotion_control` input
3. Connect `opt_narrator` output to IndexTTS-2 Engine `emotion_audio` input
**Advantages:**
@@ -59,12 +71,13 @@ Use the `opt_narrator` output from the 🎭 Character Voices node as an emotion
**Example workflow:**
```
🎭 Character Voices (David_Attenborough) → opt_narrator → IndexTTS-2 Engine (emotion_control)
🎭 Character Voices (David_Attenborough) → opt_narrator → IndexTTS-2 Engine (emotion_audio)
```
## Method 3: Emotion Vectors
Use the 🌈 IndexTTS-2 Emotion Vectors node for precise manual control over 8 different emotions.
## Method 3: Emotion Vectors
Use the 🌈 IndexTTS-2 Emotion Vectors node for precise manual control over 8 different emotions.
Connect its `emotion_control` output to the IndexTTS-2 Engine's `emotion_control` input.
**Available emotions:**
@@ -84,9 +97,10 @@ Use the 🌈 IndexTTS-2 Emotion Vectors node for precise manual control over 8 d
- Start with single emotions, then experiment with combinations
- Use the `random` buttom to get a completely random emotion pattern. Might be too strong.
## Method 4: Text Emotion (Dynamic Analysis)
Use the 🌈 IndexTTS-2 Text Emotion node for AI-powered emotion analysis with dynamic templates.
## Method 4: Text Emotion (Dynamic Analysis)
Use the 🌈 IndexTTS-2 Text Emotion node for AI-powered emotion analysis with dynamic templates.
Connect its `emotion_control` output to the IndexTTS-2 Engine's `emotion_control` input.
### Static Text Emotion
@@ -124,11 +138,104 @@ Analysis: "Worried parent speaking: Where have you been?"
Result: Anxious, concerned vocal expression
```
---
## Character Tag Emotion Control
Control emotions per character using inline tags in your text: `[Character:emotion_ref]`
---
## Combining Audio with Vectors or Text
Connect both emotion sources when you want an audio performance to provide the
base delivery while vectors or Qwen text analysis add a targeted emotional
direction:
```text
🎭 Character Voices (opt_narrator) ──→ emotion_audio
🌈 Emotion Vectors or Text Emotion ──→ emotion_control
IndexTTS-2 Engine
```
`emotion_alpha` is the shared overall emotion-intensity control. The audio
reference and vector/text signal are blended during IndexTTS-2 conditioning;
they are not generated as two separate voices and mixed afterward.
For example, an audio reference can provide a natural speaking style while a
`[sad:+0.2|calm:-0.1]` inline adjustment adds a restrained sadness to one
segment. A Qwen text preset can be used the same way.
Character audio and inline emotion parameters can share one tag:
```text
[Bob:br_ivan_raiva3|sad:+0.25|calm:-0.10] Bob speaks with a restrained overlay.
[Bob:br_ivan_raiva3|emotion:"quiet grief masking frustration"] Bob uses Qwen text emotion too.
```
---
## Inline Emotion Switching
Numeric emotion tags can replace or adjust the vector for one text segment:
```text
[sad:0.7|calm:0.2] This uses explicit absolute values.
[sad:+0.3|calm:-0.2] This modifies the connected vector.
[vector:0,0,0.7,0,0,0.4,0,0.2] This supplies all eight absolute values.
[vector:+0,+0,+0.3,+0,+0,+0,+0,-0.2] This supplies eight deltas.
```
The full-vector order is `happy, angry, sad, afraid, disgusted, melancholic,
surprised, calm`. Unsigned named values are absolute; `+` and `-` named values
are relative. A full vector is relative only when every value carries a sign.
Text-emotion controls support saved presets and quoted descriptions:
```text
[emotion:restrained_anger] Text using a saved preset.
[emotion:"Restrained anger masking disappointment"] Direct text control.
[emotion:"Analyze this delivery as nervous anticipation: {seg}"] Dynamic control.
```
Inline controls override connected global vector/text values for that segment.
A character audio emotion reference such as `[Alice:sad_reference]` replaces the
global audio reference for that segment, but it can still blend with the
segment's vector/text control. Inline settings revert at the next segment and do
not mutate the connected vector.
The TTS Tag Editor provides the same interactive radar used by the IndexTTS-2
Emotion Vectors node. Click an existing numeric emotion tag to open its radar
as a contextual popover beside the tag. Create tags from **Inline Tags →
IndexTTS-2**, and use **Manage Emotion Presets** in its Text Emotion section for
the preset library. Text and vector presets are stored in
`models/TTS/IndexTTS/emotion_presets.json`.
Radar changes appear in the editor text immediately. Intermediate drag/input
states are not added to undo history: closing the popover commits one undoable
change, while Cancel or Escape restores the tag exactly as it was when opened.
The editor's **Inline Tags** tab also includes an **IndexTTS-2** engine panel for
inserting full absolute/delta vectors, named emotion values, saved text presets,
quoted descriptions, and dynamic descriptions containing `{seg}`.
Emotion controls can be composed directly on character tags: place the caret on
`[Bob:audio_reference]` and add a vector or text emotion to append it as a pipe
parameter.
The named-emotion panel includes a magnitude slider and a press-drag-release
radial picker: direction chooses the emotion and distance chooses its value.
The operation dropdown determines whether that value is absolute, a positive
delta, or a negative delta. Saved text and vector presets refresh in the sidebar
immediately after they are changed in the preset manager.
Clicking a saved `[emotion:preset_name]` tag in the editor opens a small anchored
preset dropdown, allowing that line's preset to be swapped without opening the
full manager. Adding an emotion control while the caret is inside a pure emotion
tag replaces that tag; when the caret is inside a character/audio tag, the
editor appends or updates the emotion parameter after the existing `|` fields.
Named tags remain readable while only some emotions are active. If radar editing
activates all eight emotions, the editor automatically converts the result to the
shorter ordered `[vector:...]` form.
## Character Tag Emotion Control
Control emotions per character using inline tags in your text: `[Character:emotion_ref]`
**Syntax:**
@@ -145,8 +252,9 @@ Control emotions per character using inline tags in your text: `[Character:emoti
```
Hello everyone! [Alice:happy_sarah] I'm so excited to be here today!
[Bob:angry_tom] That's completely unacceptable behavior.
[Narrator:David] Meanwhile, in a distant galaxy...
[Bob:angry_tom] That's completely unacceptable behavior.
[Narrator:David] Meanwhile, in a distant galaxy...
[Bob:br_ivan_raiva3|sad:+0.25] Bob uses an audio reference plus a vector delta.
*assuming happy_sarah, angry_tom and David are alias or character voices in yout folder with that name
```
@@ -155,11 +263,13 @@ Hello everyone! [Alice:happy_sarah] I'm so excited to be here today!
**Character tag priority:**
- Character tags override ALL other emotion settings for that specific character
- Other characters use global emotion settings
- Allows mixing different emotions in the same audio
**Character tag behavior:**
- Character tags select the speaker and can provide a segment-local audio emotion reference
- A segment-local audio reference replaces the global audio reference for that segment
- Global or inline vector/text controls can still blend with that audio reference
- Other characters use the global audio and vector/text settings
- Allows mixing different emotion sources in the same audio
## Emotion Alpha Control
@@ -211,4 +321,4 @@ Welcome to our show! [Bob:serious_narrator] But first, a serious announcement.
---
This comprehensive emotion control system gives you unprecedented flexibility in creating expressive, emotionally rich TTS audio for any application.
This comprehensive emotion control system gives you unprecedented flexibility in creating expressive, emotionally rich TTS audio for any application.
+105 -104
View File
@@ -2,110 +2,110 @@
## Language Support by Engine
| Language | Code | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Dots TTS | MOSS-TTS | RVC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 🇺🇸 **English** | EN | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| 🇨🇳 **Chinese** | ZH | ❌ | ❌ | ✅ | ✅ | ✅ (Mandarin) | ✅ | ✅ | ✅ + 18 dialects | ✅ | ❌ | ✅ (Mandarin + Sichuanese, Cantonese) | ❌ | ✅ (Mandarin; official also exposes YUE separately outside this matrix) | ✅ | ✅ |
| 🇩🇪 **German** | DE | ✅ | ✅ (×3) | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ |
| 🇪🇸 **Spanish** | ES | ✅ | ❌ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ |
| 🇫🇷 **French** | FR | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ |
| 🇮🇹 **Italian** | IT | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
| 🇯🇵 **Japanese** | JA | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ✅ ? | ✅ | ✅ | ✅ (4.1-2b supports Japanese; plus variant does not) | ✅ | ❌ | ✅ | ✅ | ✅ |
| 🇰🇷 **Korean** | KO | ❌ | ✅ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ |
| 🇷🇺 **Russian** | RU | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
| 🇧🇷 **Portuguese** | PT | ✅ (BR) | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ (EU/BR*) | ✅ | ❌ | ❌ | ✅ (Official PT tag is generic; upstream does not expose separate PT-BR/PT-PT tags and it may lean more European Portuguese than Brazilian Portuguese) | ✅ | ✅ |
| 🇵🇱 **Polish** | PL | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
| 🇮🇳 **Hindi** | HI | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
| 🇪🇬 **Arabic** | AR | ❌ | ❌ | ✅ (Egyptian) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
| 🇹🇷 **Turkish** | TR | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
| 🇹🇭 **Thai** | TH | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
| 🇳🇴 **Norwegian** | NO | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇻🇳 **Vietnamese** | VI | ❌ | ❌ | ✅ (Viterbox) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
| 🇦🇲 **Armenian** | HY | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇬🇪 **Georgian** | KA | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇩🇰 **Danish** | DA | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
| 🇫🇮 **Finnish** | FI | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
| 🇬🇷 **Greek** | EL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
| 🇮🇱 **Hebrew** | HE | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇾 **Malay** | MS | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇱 **Dutch** | NL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
| 🇸🇪 **Swedish** | SV | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
| 🇰🇪 **Swahili** | SW | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇦 **Afrikaans** | AF | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇱 **Albanian** | SQ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Assamese** | AS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Asturian** | AST | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇿 **Azerbaijani** | AZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇺 **Bashkir** | BA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Basque** | EU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇾 **Belarusian** | BE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇩 **Bengali** | BN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇦 **Bosnian** | BS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇬 **Bulgarian** | BG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Catalan** | CA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇭 **Cebuano** | CEB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇶 **Central Kurdish** | CKB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇭🇷 **Croatian** | HR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇿 **Czech** | CS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇺 **Eastern Mari** | MHR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🌍 **Esperanto** | EO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇪 **Estonian** | ET | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Galician** | GL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Gujarati** | GU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇭🇹 **Haitian Creole** | HT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇬 **Hausa** | HA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇭🇺 **Hungarian** | HU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇩 **Indonesian** | ID | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇩 **Javanese** | JV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Kannada** | KN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇿 **Kazakh** | KK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇼 **Kinyarwanda** | RW | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇬 **Kyrgyz** | KY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇱🇻 **Latvian** | LV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇩 **Lingala** | LN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇱🇹 **Lithuanian** | LT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇪 **Luo** | LUO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇰 **Macedonian** | MK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Malayalam** | ML | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇹 **Maltese** | MT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇿 **Māori** | MI | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Marathi** | MR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇳 **Mongolian** | MN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇵 **Nepali** | NE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇫🇷 **Occitan** | OC | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇷 **Persian** | FA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇴 **Romanian** | RO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇦 **Sepedi** | NSO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇸 **Serbian** | SR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇼 **Shona** | SN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇸🇰 **Slovak** | SK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇸🇮 **Slovene** | SL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇭 **Tagalog** | TL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇹🇯 **Tajik** | TG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Tamil** | TA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Telugu** | TE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇺🇦 **Ukrainian** | UK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇰 **Urdu** | UR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇳 **Uyghur** | UG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇺🇿 **Uzbek** | UZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇦 **Xhosa** | XH | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇦 **Zulu** | ZU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇼 **Chichewa/Nyanja** | NY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Eastern Punjabi** | PA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇺🇬 **Ganda** | LG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇸 **Icelandic** | IS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇪 **Irish** | GA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇩🇿 **Kabyle** | KAB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇻 **Kabuverdianu** | KEA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇪 **Kamba** | KAM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇻🇦 **Latin** | LA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇱🇺 **Luxembourgish** | LB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇹 **Oromo** | OM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇫 **Pashto** | PS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇰 **Sindhi** | SD | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇸🇴 **Somali** | SO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇴 **Umbundu** | UMB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🏴 **Welsh** | CY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| Language | Code | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | OmniVoice | MOSS-TTS | RVC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 🇺🇸 **English** | EN | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ Tier 1 | ✅ | ✅ | ✅ | ✅ |
| 🇨🇳 **Chinese** | ZH | ❌ | ❌ | ✅ | ✅ | ✅ (Mandarin) | ✅ | ✅ | ✅ + 18 dialects | ✅ | ❌ | ✅ (Mandarin + Sichuanese, Cantonese) | ❌ | ✅ Tier 1 | ✅ (Mandarin; official also exposes YUE separately outside this matrix) | ✅ | ✅ | ✅ |
| 🇩🇪 **German** | DE | ✅ | ✅ (×3) | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ | ✅ | ✅ |
| 🇪🇸 **Spanish** | ES | ✅ | ❌ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ | ✅ | ✅ |
| 🇫🇷 **French** | FR | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ | ✅ | ✅ |
| 🇮🇹 **Italian** | IT | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ |
| 🇯🇵 **Japanese** | JA | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ✅ ? | ✅ | ✅ | ✅ (4.1-2b supports Japanese; plus variant does not) | ✅ | ❌ | ✅ Tier 1 | ✅ | ✅ | ✅ | ✅ |
| 🇰🇷 **Korean** | KO | ❌ | ✅ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ Tier 2 | ✅ | ✅ | ✅ | ✅ |
| 🇷🇺 **Russian** | RU | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ | ✅ | ✅ |
| 🇧🇷 **Portuguese** | PT | ✅ (BR) | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ (EU/BR*) | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ (Official PT tag is generic; upstream does not expose separate PT-BR/PT-PT tags and it may lean more European Portuguese than Brazilian Portuguese) | ✅ (generic PT; official language space is much broader than this matrix) | ✅ | ✅ |
| 🇵🇱 **Polish** | PL | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ |
| 🇮🇳 **Hindi** | HI | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇪🇬 **Arabic** | AR | ❌ | ❌ | ✅ (Egyptian) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ | ✅ | ✅ |
| 🇹🇷 **Turkish** | TR | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ |
| 🇹🇭 **Thai** | TH | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇳🇴 **Norwegian** | NO | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
| 🇻🇳 **Vietnamese** | VI | ❌ | ❌ | ✅ (Viterbox) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇦🇲 **Armenian** | HY | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
| 🇬🇪 **Georgian** | KA | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
| 🇩🇰 **Danish** | DA | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
| 🇫🇮 **Finnish** | FI | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇬🇷 **Greek** | EL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ |
| 🇮🇱 **Hebrew** | HE | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
| 🇲🇾 **Malay** | MS | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
| 🇳🇱 **Dutch** | NL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
| 🇸🇪 **Swedish** | SV | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
| 🇰🇪 **Swahili** | SW | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
| 🇿🇦 **Afrikaans** | AF | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇱 **Albanian** | SQ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Assamese** | AS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Asturian** | AST | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇿 **Azerbaijani** | AZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇺 **Bashkir** | BA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Basque** | EU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇾 **Belarusian** | BE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇩 **Bengali** | BN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇦 **Bosnian** | BS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇧🇬 **Bulgarian** | BG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Catalan** | CA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇭 **Cebuano** | CEB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇶 **Central Kurdish** | CKB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇭🇷 **Croatian** | HR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇿 **Czech** | CS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇺 **Eastern Mari** | MHR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🌍 **Esperanto** | EO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇪 **Estonian** | ET | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇸 **Galician** | GL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Gujarati** | GU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇭🇹 **Haitian Creole** | HT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇬 **Hausa** | HA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇭🇺 **Hungarian** | HU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇩 **Indonesian** | ID | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇩 **Javanese** | JV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Kannada** | KN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇿 **Kazakh** | KK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇼 **Kinyarwanda** | RW | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇬 **Kyrgyz** | KY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇱🇻 **Latvian** | LV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇩 **Lingala** | LN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇱🇹 **Lithuanian** | LT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇪 **Luo** | LUO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇰 **Macedonian** | MK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Malayalam** | ML | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇹 **Maltese** | MT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇿 **Māori** | MI | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Marathi** | MR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇳 **Mongolian** | MN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇳🇵 **Nepali** | NE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇫🇷 **Occitan** | OC | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇷 **Persian** | FA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇴 **Romanian** | RO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇦 **Sepedi** | NSO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇷🇸 **Serbian** | SR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇼 **Shona** | SN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇸🇰 **Slovak** | SK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇸🇮 **Slovene** | SL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇭 **Tagalog** | TL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇹🇯 **Tajik** | TG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Tamil** | TA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Telugu** | TE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇺🇦 **Ukrainian** | UK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇰 **Urdu** | UR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇳 **Uyghur** | UG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇺🇿 **Uzbek** | UZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇦 **Xhosa** | XH | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇿🇦 **Zulu** | ZU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇲🇼 **Chichewa/Nyanja** | NY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇳 **Eastern Punjabi** | PA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇺🇬 **Ganda** | LG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇸 **Icelandic** | IS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇮🇪 **Irish** | GA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇩🇿 **Kabyle** | KAB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇨🇻 **Kabuverdianu** | KEA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇰🇪 **Kamba** | KAM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇻🇦 **Latin** | LA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇱🇺 **Luxembourgish** | LB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇪🇹 **Oromo** | OM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇫 **Pashto** | PS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇵🇰 **Sindhi** | SD | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇸🇴 **Somali** | SO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🇦🇴 **Umbundu** | UMB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| 🏴 **Welsh** | CY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
**Notes:**
@@ -114,5 +114,6 @@
- **Higgs Audio v3**: Official model card lists 102 languages total: 85 polished-production languages and 17 usable but less polished languages.
- **IndexTTS-2**: Trained on 55K+ hours - ZH, EN, JA primary.
- **MOSS-TTS**: Official list includes ZH, EN, DE, ES, FR, JA, IT, HU, KO, RU, FA, AR, PL, PT, CS, DA, SV, EL, TR; HU/FA/CS are not separate columns in this matrix.
- **OmniVoice**: Official model supports 600+ languages; this matrix only shows the suite's comparison subset.
- **RVC**: Language-agnostic voice conversion/post-processing; it is marked supported for every language row.
- Qwen3-TTS: Supports pt-BR only with instruction (voice design or harcoded voices). Base can't do instructions, so it will always ouput pt-PT.
+13
View File
@@ -114,6 +114,13 @@ Use this as the canonical list of model repositories/links for offline setup.
| echo-tts-base (model + PCA state) | [jordand/echo-tts-base](https://huggingface.co/jordand/echo-tts-base) | ~5.3GB | ✅ | pytorch_model.safetensors + pca_state.safetensors |
| fish-s1-dac-min (audio codec) | [jordand/fish-s1-dac-min](https://huggingface.co/jordand/fish-s1-dac-min) | ~1.8GB | ✅ | pytorch_model.safetensors — audio codec required by Echo-TTS |
## Fish Audio S2 Pro
| Component | Source | Size | Auto-Download | Notes |
|---|---|---|---|---|
| S2 Pro | [fishaudio/s2-pro](https://huggingface.co/fishaudio/s2-pro) | ~10.3GB | ✅ | Official 4B model and codec; BNB INT8/NF4 are optional load-time quantization modes that reuse these files; non-commercial license |
| S2 Pro FP8 | [drbaph/s2-pro-fp8](https://huggingface.co/drbaph/s2-pro-fp8) | ~8.0GB | ✅ | Community weight-only FP8 checkpoint; BF16 activations; RTX 4090/5090-class CUDA GPU required |
## Dots TTS
| Component | Source | Size | Auto-Download | Notes |
@@ -122,6 +129,12 @@ Use this as the canonical list of model repositories/links for offline setup.
| dots.tts-soar | [rednote-hilab/dots.tts-soar](https://huggingface.co/rednote-hilab/dots.tts-soar) | ~6GB | ✅ | Official SOAR checkpoint for higher-quality zero-shot cloning |
| dots.tts-mf | [rednote-hilab/dots.tts-mf](https://huggingface.co/rednote-hilab/dots.tts-mf) | ~6GB | ✅ | Official MeanFlow-distilled checkpoint for faster inference |
## OmniVoice
| Component | Source | Size | Auto-Download | Notes |
|---|---|---|---|---|
| OmniVoice | [k2-fsa/OmniVoice](https://huggingface.co/k2-fsa/OmniVoice) | ~3.7GB | ✅ | Official main model including bundled audio_tokenizer assets |
## MOSS-TTS
| Component | Source | Size | Auto-Download | Notes |
+56
View File
@@ -316,6 +316,33 @@ Notes:
- Both components are required and auto-downloaded on first use.
- License: CC-BY-NC-SA (non-commercial).
## Fish Audio S2 Pro
```text
ComfyUI/models/TTS/fish_audio_s2_pro/
├── codec.pth
├── config.json
├── model-00001-of-00002.safetensors
├── model-00002-of-00002.safetensors
├── model.safetensors.index.json
└── tokenizer.json
```
Optional FP8 variant:
```text
ComfyUI/models/TTS/fish_audio_s2_pro_fp8/
├── codec.pth
├── config.json
├── model.safetensors
├── quantization_info.json
└── tokenizer.json
```
The complete official repository metadata and tokenizer files are downloaded alongside these files. License: Fish Audio Research License (non-commercial without a separate commercial license).
The `s2-pro-bnb-int8` and `s2-pro-bnb-nf4` options reuse `fish_audio_s2_pro/` and quantize its official checkpoint while loading. They do not download another model copy and require `bitsandbytes`.
## Dots TTS
```text
@@ -349,3 +376,32 @@ Notes:
- Downloads into a dedicated `dots_tts/` folder.
- Native sample rate is 48kHz.
- Main-environment support works on Transformers 5; on Windows, `normalize_text` falls back to no-op if `WeTextProcessing` is unavailable.
## OmniVoice
```text
ComfyUI/models/TTS/omnivoice/
└── OmniVoice/
```
Main model contents:
```text
ComfyUI/models/TTS/omnivoice/OmniVoice/
├── chat_template.jinja
├── config.json
├── model.safetensors
├── tokenizer.json
├── tokenizer_config.json
└── audio_tokenizer/
├── config.json
├── model.safetensors
└── preprocessor_config.json
```
Notes:
- Downloads into a dedicated `omnivoice/` folder.
- Native sample rate is 24kHz.
- The main OmniVoice model stays in the main Transformers 5 environment.
- Voice cloning requires reference audio plus explicit reference text in this suite.
+16 -1
View File
@@ -9,6 +9,7 @@ For related topics, also see:
- [Step Audio EditX Inline Tags User Guide](INLINE_EDIT_TAGS_USER_GUIDE.md)
- [Higgs Audio v3 Inline Tags](HIGGS_AUDIO_V3_INLINE_TAGS.md)
- [CosyVoice3 Tags Guide](COSYVOICE3_TAGS_GUIDE.md)
- [OmniVoice Native Tags Guide](OMNIVOICE_TAGS_GUIDE.md)
## What This Editor Is For
@@ -19,7 +20,7 @@ It supports:
- Character switching tags
- Language switching tags
- Per-segment parameter overrides
- Engine-aware inline tags for Step Audio EditX, Higgs Audio v3, and CosyVoice3
- Engine-aware inline tags for Step Audio EditX, Higgs Audio v3, CosyVoice3, and OmniVoice
- Presets and edit history
- SRT-aware highlighting and timing editing
@@ -80,8 +81,22 @@ CosyVoice3 examples:
<laughing>that was funny</laughing>
```
OmniVoice examples:
```text
<laughter>
<sigh>
<question-ei>
```
Use the dedicated inline tag controls in the sidebar when you do not want to type these by hand.
Important differences:
- `Step Audio EditX` tags are post-process controls
- `Higgs Audio v3`, `CosyVoice3`, and `OmniVoice` tags are native generation controls
- `OmniVoice` editor insertion uses suite-default angle-tag aliases and the processor converts them internally to official native tags during generation
## SRT Editing
When the text looks like valid SRT, the editor highlights subtitle numbers and timings differently and enables subtitle-specific editing tools.
@@ -26,6 +26,10 @@ Produce a short scope document with:
8. What manual tests will prove the integration works?
```
## Runtime Decision Order
Test the official package with `--no-deps` in the main T5 environment first. Prefer a small compatibility patch when practical; use the existing shared T4 SHRED runtime only if T5 is genuinely incompatible. Never create another environment or download/reinstall Torch or Transformers without explicit maintainer approval.
## Node Types To Choose From
Decide whether the engine needs:
@@ -91,13 +91,43 @@ Follow this order:
13. Add interrupt checks in long loops.
14. Add progress feedback for long generation.
15. Update docs/YAML metadata.
16. Run manual ComfyUI tests.
16. Run automated and live ComfyUI validation. Use FL-MCP-assisted validation when it is installed and connected; otherwise perform the same checks manually. Follow `tests/FL_MCP_VALIDATION.md`.
17. Run the required parity checklist.
## Live ComfyUI Validation Rule
Passing imports or pytest is not enough for a new engine. Validate it in the canonical Windows ComfyUI installation after implementation.
If [ComfyUI_FL-MCP](https://github.com/filliptm/ComfyUI_FL-MCP) is available, the LLM should use it to inspect and operate the live ComfyUI instance. Treat it as an optional test driver, not a project dependency and not a substitute for the existing test suite.
The LLM should:
- After changing Python code, restart the canonical Windows ComfyUI process before testing. An already-running process still has the old modules loaded.
- Use PowerShell to identify the process listening on port `8188`, verify its command line belongs to the canonical ComfyUI `main.py`, stop only that process, and relaunch it with the canonical Windows Python.
- Wait for `http://127.0.0.1:8188/system_stats` to respond before using FL-MCP.
- Refresh the existing ComfyUI browser tab after restart and confirm the FL-MCP browser bridge has reconnected before calling canvas-only tools.
- Confirm the engine node and the relevant unified node are registered.
- Load or construct the smallest useful workflow.
- Inspect workflow JSON for the expected node types, links, and widget values.
- Queue the workflow and wait for completion.
- Inspect execution history and report the full actionable error if execution fails.
- Confirm the expected audio output artifact exists.
- Capture a canvas screenshot for UI and broken-node inspection.
- Exercise TTS Text and SRT for every TTS engine, plus any other scoped capability.
- Record what was actually tested, what was skipped, and why.
Screenshots prove only visible workflow state. They do not prove that generation succeeded or that audio is correct. Execution history and output artifacts are required evidence, and the user must still judge subjective audio quality.
Do not install FL-MCP, alter its safety settings, or enable destructive tools unless the user authorizes it. If FL-MCP is unavailable, report that fact and follow the manual fallback in `tests/FL_MCP_VALIDATION.md`.
Repeat the edit, restart, reconnect, and validation cycle after every implementation fix that changes imported Python code. Frontend-only changes may require a hard browser refresh as well. Do not claim that a fix was tested against a ComfyUI process started before the fix was written.
## Architecture Rule
Unified nodes should stay thin.
Reference engines are examples only. Every engine must have dedicated processors and adapters; share only engine-neutral utilities.
Do not put hundreds of lines of engine-specific orchestration into:
- `nodes/unified/tts_text_node.py`
@@ -25,7 +25,8 @@ Downloads and dependencies:
- Do models download into organized ComfyUI/models/TTS/ folders?
- Did you prevent silent downloads into random cache folders?
- Did you document dependency conflicts or install.py changes?
- Did you explicitly decide whether this engine belongs in Main Environment or needs runtime isolation?
- Did you test `--no-deps` on Main/T5 and simple patching before falling back to the shared T4 runtime?
- Did you avoid creating another environment or downloading/reinstalling Torch or Transformers without explicit maintainer approval?
- If runtime isolation is needed, did you document the default mode and the reason in YAML/README?
Audio format:
@@ -79,6 +80,21 @@ Manual tests:
- Did parameter switching work?
- Did Clear VRAM then regenerate work, and did unload actually tear down runtime/cache state instead of only moving weights to CPU?
- Did interrupt/cancel work in long generation?
Live ComfyUI integration evidence:
- Was validation run in the canonical Windows ComfyUI environment?
- Was ComfyUI restarted after the final Python changes, and can you show that the tested process started after those edits?
- After restart, was the browser refreshed and the FL-MCP browser bridge confirmed connected before canvas checks?
- If FL-MCP was available, did you follow tests/FL_MCP_VALIDATION.md and identify which MCP checks were run?
- If FL-MCP was unavailable, did you perform and document the equivalent manual checks instead of claiming MCP validation?
- Are the engine node and relevant unified nodes present without import or registration errors?
- Does the saved workflow JSON contain the expected node types, links, and widget values?
- Does the workflow load without missing/broken node state?
- Was the workflow queued, and does execution history show successful completion or the full actionable error?
- Does each successful run produce the expected audio output artifact?
- Was a canvas screenshot captured for UI inspection without treating the screenshot as execution proof?
- Did you record tested, failed, and skipped cases, including the reason for every skip?
- Did a human assess subjective audio quality separately?
```
## Important Failures This Prevents
@@ -462,11 +462,12 @@ Also to test, requirements and dependencies need to be added.
- [ ] Character switching works with `[CharacterName] text`
- [ ] Language switching works (if applicable)
- [ ] Pause tags work with `[pause:1.5s]`
- [ ] Caching works (same input = cached output)
- [ ] Model auto-download works
- [ ] VRAM management works (model unloads)
- [ ] Different parameter combinations work
- [ ] **Interrupt handling works** - User can stop SRT generation and it stops within ~1 segment
- [ ] Caching works (same input = cached output)
- [ ] Model auto-download works
- [ ] VRAM management works (model unloads)
- [ ] Different parameter combinations work
- [ ] Engine prints a standard `Settings:` summary with the active generation/load parameters
- [ ] **Interrupt handling works** - User can stop SRT generation and it stops within ~1 segment
### Phase 4: SRT Implementation
@@ -645,4 +646,4 @@ Update the engines comparison table.
- Model management logic
- Audio format conversion utilities
---
---
@@ -56,6 +56,10 @@
- **Cache key pattern**: `audio_cache.generate_cache_key(engine_type, text=..., audio_component=..., **all_params)`
- **Duration calculation**: Update `_calculate_duration()` with engine sample rate (e.g., 24000 for Step Audio EditX, F5-TTS)
### Engine Settings Logging
- **Missing standard print**: New engines should print the usual `Settings:` summary once per run with the active generation/load parameters so validation can confirm what actually executed
- **Resolved prompt preview**: Reuse `utils.voice.character_logging.format_resolved_character_block()` for boxed text previews so logs show the voice that will actually speak, not only the parser alias
### Model Lifecycle - __del__ Destructor
- **CRITICAL**: Remove `__del__` from engine classes - causes automatic unload after generation ends (when object goes out of scope)
- **Pattern**: F5-TTS and ChatterBox don't have `__del__`, only IndexTTS and StepAudio did (wrong)
+73
View File
@@ -0,0 +1,73 @@
# OmniVoice Native Tags Guide
OmniVoice has its own official inline square-bracket control tokens upstream, but **this suite does not expose `[]` for OmniVoice tags**.
Inside TTS Audio Suite, you can write the suite-default angle-tag aliases in text:
```text
<laughter>
<sigh>
<question-ei>
```
The OmniVoice processor converts those aliases internally to the official OmniVoice native form before generation:
```text
[laughter]
[sigh]
[question-ei]
```
Do **not** type OmniVoice non-verbal tags in `[]` form in suite text. In this suite, `[]` belongs to character, language, parameter, and pause syntax.
## Supported Non-Verbal Tags
Official OmniVoice non-verbal meanings exposed by this suite through `<>` aliases:
```text
laughter
sigh
confirmation-en
question-en
question-ah
question-oh
question-ei
question-yi
surprise-ah
surprise-oh
surprise-wa
surprise-yo
dissatisfaction-hnn
```
Examples:
```text
[Alice] <laughter> You really got me there.
[Bob] <sigh> Fine, let's try again.
[Narrator] <question-ei> Really?
[Narrator] <surprise-oh> I didn't expect that.
```
## Important Behavior
- OmniVoice uses native generation tags here, not Step Audio EditX inline post-processing.
- User-facing suite syntax stays in `<>` form for OmniVoice non-verbal tags.
- `[]` is reserved for suite structural syntax like `[Alice]`, `[en:Alice]`, `[pause:1s]`, and parameter switching.
- Do not rely on Step-style `<Laughter:2>` or `<emotion:happy>` semantics in the OmniVoice text path.
- If you want Step Audio EditX as a second pass on OmniVoice output, use the separate `🎨 Audio Editor` node manually after generation.
## Multiline Tag Editor
The `🏷️ Multiline TTS Tag Editor` has a dedicated `OmniVoice` mode in the `Inline Tags` panel.
- The editor inserts suite-default angle-tag aliases like `<laughter>`
- The processor converts them internally to official OmniVoice square tags during generation
- The editor does not encourage raw OmniVoice `[]` input because `[]` is suite syntax
## Sources
This behavior follows the official OmniVoice documentation for non-verbal symbols and pronunciation control:
- [OmniVoice GitHub README](https://github.com/k2-fsa/OmniVoice)
- [OmniVoice Hugging Face model card](https://huggingface.co/k2-fsa/OmniVoice)
+50 -1
View File
@@ -127,7 +127,56 @@ Parameters are applied **only to the current segment** and automatically revert
| `cfg` | — | float | 0.0-20.0 | CFG strength |
| `top_p` | `topp` | float | 0.0-1.0 | Nucleus sampling probability |
| `top_k` | `topk` | int | 1-100 | Top-k sampling |
| `emotion_alpha` | — | float | 0.0-1.0 | Emotion control strength |
| `emotion_alpha` | — | float | 0.0-2.0 | Shared audio/vector/text emotion intensity |
IndexTTS-2 also supports inline emotion controls. Named unsigned values replace
that dimension; explicitly signed values adjust the connected vector:
```text
[sad:0.7|calm:0.2] Absolute values for this segment.
[sad:+0.3|calm:-0.2] Adjust the connected vector for this segment.
```
All eight values can be supplied in the official order `happy, angry, sad,
afraid, disgusted, melancholic, surprised, calm`:
```text
[vector:0,0,0.7,0,0,0.4,0,0.2] Absolute replacement.
[vector:+0,+0,+0.3,+0,+0,+0,+0,-0.2] Relative adjustment.
```
Full relative vectors require an explicit sign on every value. Results are
clamped to IndexTTS-2's supported range and revert to the connected vector at
the next segment.
Text emotion can use a saved preset or quoted text. `{seg}` is expanded with
the current segment before QwenEmotion analysis:
```text
[emotion:restrained_anger] A saved preset.
[emotion:"Quiet grief masking frustration"] A direct description.
[emotion:"Infer nervous anticipation from this line: {seg}"] Dynamic analysis.
```
Click a numeric emotion tag in the TTS Tag Editor to open a contextual radar
directly beside that tag. The editor also creates and manages text
presets in `models/TTS/IndexTTS/emotion_presets.json`.
IndexTTS-2 has separate engine inputs for these sources: connect vector or text
emotion to `emotion_control` and audio emotion references to `emotion_audio`.
Both may be connected simultaneously; IndexTTS-2 blends them during emotion
conditioning. Inline vector/text controls override the connected vector/text
values for their segment, while `[Character:emotion_ref]` selects a
segment-local audio reference that can still blend with vector/text emotion.
When an emotion control is inserted with the caret inside a character/audio tag,
the editor appends or updates it as another pipe parameter, for example
`[Bob:br_ivan_raiva3|sad:+0.25]`.
The tag editor's quick-swap palette is engine-aware: parameter choices are
filtered to the selected inline engine, while each engine's supported native
emotion/style/prosody/sound tags use their own replacement choices. Named
emotion presets can be swapped from the text; quoted `[emotion:"..."]` text is
intentionally left as direct editable content rather than treated as a preset.
---
@@ -207,6 +207,47 @@ This document tracks updates applied to our bundled IndexTTS-2 code from the ups
---
---
## 2026-07-11: Upstream Audit Before Release
**Upstream repository checked:** `index-tts/index-tts` (`main`)
**Upstream head observed:** `b5bd657` (2026-07-08)
**Check performed:** 2026-07-11
### Relevant upstream changes reviewed
| Commit | Upstream change | Bundled status | Release action |
|---|---|---|---|
| `843972e` | Coerce QwenEmotion JSON emotion scores to `float` and reject non-numeric values clearly | **Applied** in bundled `clamp_score()` | Run a text-emotion generation with numeric-string classifier output |
| `b154a1b` | WebUI text/vector preset save/load management | **Already covered differently** by the ComfyUI-native preset manager and `emotion_presets.json` integration | No direct port needed |
| `b5bd657` | Expose `--accel` and `--torch-compile`, add optional acceleration extras and WebUI settings | **Applied selectively**; suite now forwards `use_accel` into the bundled GPT path, while retaining ComfyUI-owned dependency handling | Validate acceleration fallback on compatible and non-accelerated setups |
| `7264ce2` | Improve IndexTTS-2 model resource checks and HF cache handling | **Suite-owned downloader differs** and needs a separate comparison if download failures are reported | No blind copy into bundled code |
### Findings
- No upstream change was found that invalidates the current eight-emotion vector order,
Qwen text-emotion syntax, audio-reference blending, or the suite's inline tag format.
- The upstream QwenEmotion string-score fix is directly relevant to the suite's text-emotion
path and should be applied before a release.
- The upstream acceleration work is not a drop-in replacement because this repository
bundles and adapts IndexTTS-2. The suite already contains the acceleration modules, but
`utils/models/unified_model_interface.py` should be checked so `use_accel` reaches the
bundled `IndexTTS2` constructor.
- Upstream WebUI presets are not copied verbatim: the suite's ComfyUI editor has a richer
vector/radar, inline-tag, sidebar, and filesystem preset implementation.
### Release follow-up checklist
- [x] Apply the upstream `clamp_score()` numeric coercion.
- [x] Pass `use_accel` through the unified IndexTTS-2 factory; validate fallback behavior.
- [ ] Run a Qwen text-emotion generation using numeric-string classifier output.
- [ ] Verify acceleration flags on a compatible CUDA installation and on a setup without
optional acceleration dependencies.
- [ ] Recheck bundled model-resource validation against the current upstream `check` logic.
---
## Next Update Check: 2025-11-20
**Monitoring:** Watch for commits to `indextts/infer_v2.py` in https://github.com/index-tts/index-tts
@@ -222,4 +263,4 @@ git log --oneline 1d5d079..HEAD -- indextts/infer_v2.py
- Additional performance optimizations
- Bug fixes in new acceleration code
- Breaking API changes
- Model loading improvements
- Model loading improvements
+20 -2
View File
@@ -49,6 +49,24 @@ except Exception as e:
def __init__(self, *args, **kwargs):
raise ImportError(f"Dots TTS adapter not available: {e}")
try:
from .fish_audio_s2_adapter import FishAudioS2Adapter
FISH_AUDIO_S2_ADAPTER_AVAILABLE = True
except Exception as e:
FISH_AUDIO_S2_ADAPTER_AVAILABLE = False
class FishAudioS2Adapter:
def __init__(self, *args, **kwargs):
raise ImportError(f"Fish Audio S2 adapter not available: {e}")
try:
from .omnivoice_adapter import OmniVoiceEngineAdapter
OMNIVOICE_ADAPTER_AVAILABLE = True
except Exception as e:
OMNIVOICE_ADAPTER_AVAILABLE = False
class OmniVoiceEngineAdapter:
def __init__(self, *args, **kwargs):
raise ImportError(f"OmniVoice adapter not available: {e}")
try:
from .moss_tts_adapter import MossTTSEngineAdapter
MOSS_TTS_ADAPTER_AVAILABLE = True
@@ -69,9 +87,9 @@ except Exception as e:
__all__ = [
'ChatterBoxEngineAdapter', 'F5TTSEngineAdapter', 'CosyVoiceAdapter', 'EchoTTSEngineAdapter',
'DotsTTSEngineAdapter',
'DotsTTSEngineAdapter', 'OmniVoiceEngineAdapter',
'MossTTSEngineAdapter', 'HiggsAudioV3EngineAdapter',
'CHATTERBOX_ADAPTER_AVAILABLE', 'F5TTS_ADAPTER_AVAILABLE', 'COSYVOICE_ADAPTER_AVAILABLE',
'ECHO_TTS_ADAPTER_AVAILABLE', 'DOTS_TTS_ADAPTER_AVAILABLE',
'ECHO_TTS_ADAPTER_AVAILABLE', 'DOTS_TTS_ADAPTER_AVAILABLE', 'OMNIVOICE_ADAPTER_AVAILABLE',
'MOSS_TTS_ADAPTER_AVAILABLE', 'HIGGS_AUDIO_V3_ADAPTER_AVAILABLE'
]
+3 -2
View File
@@ -23,6 +23,7 @@ from engines.cosyvoice.cosyvoice import CosyVoiceEngine
from engines.cosyvoice.cosyvoice_downloader import cosyvoice_downloader
from utils.text.character_parser import character_parser
from utils.voice.discovery import get_character_mapping, get_available_characters
from utils.voice.character_logging import resolved_character_label
from utils.audio.cache import get_audio_cache
@@ -336,7 +337,7 @@ class CosyVoiceAdapter:
speaker_audio = char_audio
if char_text:
reference_text = char_text
print(f"📖 Using character voice '{character_name}'")
print(f"📖 Using character voice '{resolved_character_label(character_name, speaker_audio)}'")
# Generate cache key for this segment
segment_cache_key = self._generate_cache_key(
@@ -352,7 +353,7 @@ class CosyVoiceAdapter:
# Check cache first
cached_segment_audio = self.audio_cache.get_cached_audio(segment_cache_key)
if cached_segment_audio:
print(f"💾 Using cached CosyVoice3 segment for '{character_name}'")
print(f"💾 Using cached CosyVoice3 segment for '{resolved_character_label(character_name, speaker_audio)}'")
segment_audio = cached_segment_audio[0]
else:
# Convert CosyVoice paralinguistic tags from <tag> to [tag]
+130
View File
@@ -0,0 +1,130 @@
"""Adapter between suite processors and the isolated official Fish S2 runtime."""
from typing import Any, Dict, Optional
import torch
from engines.fish_audio_s2.downloader import FishAudioS2Downloader
from utils.audio.audio_hash import generate_stable_audio_component
from utils.audio.cache import get_audio_cache
from utils.audio.processing import AudioProcessingUtils
from utils.models.factory_config import ModelLoadConfig
from utils.text.fish_audio_s2_tags import translate_fish_s2_inline_tags
from utils.voice.character_logging import resolved_voice_name
class FishAudioS2Adapter:
SAMPLE_RATE = 44100
def __init__(self, config: Optional[Dict[str, Any]] = None):
self.config = dict(config or {})
self.audio_cache = get_audio_cache()
self._model_config = None
def update_config(self, config):
self.config = dict(config or {})
def _engine(self):
from utils.models.unified_model_interface import unified_model_interface
model_selection = self.config.get("model_variant", "s2-pro")
quantization = self.config.get("quantization", "none")
model_path = FishAudioS2Downloader.resolve_model_path(model_selection)
model_variant = FishAudioS2Downloader.resolve_model_variant(model_selection, model_path)
self._model_config = ModelLoadConfig(
engine_name="fish_audio_s2", model_type="tts", model_name=model_variant,
model_path=model_path, device=self.config.get("device", "auto"),
runtime_mode="isolated",
additional_params={
"model_variant": model_variant,
"quantization": quantization,
"precision": self.config.get("precision", "bfloat16"),
"compile": bool(self.config.get("compile", False)),
"context_length": int(self.config.get("context_length", 8192)),
},
)
return unified_model_interface.load_model(self._model_config)
def _reference(self, voice_ref):
if not isinstance(voice_ref, dict):
return None, "", "default_voice"
text = (voice_ref.get("reference_text") or voice_ref.get("prompt_text") or "").strip()
path = voice_ref.get("audio_path") or voice_ref.get("prompt_audio_path")
audio = voice_ref.get("audio")
if audio is None:
audio = voice_ref.get("waveform")
if not path and isinstance(audio, dict):
path = AudioProcessingUtils.save_audio_to_temp_file(audio["waveform"], audio.get("sample_rate", 44100))
elif not path and torch.is_tensor(audio):
path = AudioProcessingUtils.save_audio_to_temp_file(audio, voice_ref.get("sample_rate", 44100))
component = generate_stable_audio_component(audio_file_path=path) if path else "default_voice"
return path, text, component
def generate_single(self, text, voice_ref, seed=0, enable_audio_cache=True, character_name=None):
return self.generate_dialogue(
[(0, text)], [voice_ref], seed, enable_audio_cache,
cache_character=character_name or "narrator",
)
def generate_dialogue(self, turns, voice_refs, seed=0, enable_audio_cache=True,
cache_character="native_dialogue"):
formatted_turns = []
for speaker_index, turn_text in turns:
clean_text = translate_fish_s2_inline_tags((turn_text or "").strip())
if clean_text:
formatted_turns.append(f"<|speaker:{speaker_index}|>{clean_text}")
text = "\n".join(formatted_turns)
if not text:
return torch.zeros(1, 0)
references = []
reference_labels = []
components = []
reference_texts = []
for speaker_index, voice_ref in enumerate(voice_refs):
ref_path, ref_text, component = self._reference(voice_ref)
if ref_path:
if not ref_text:
raise ValueError("Fish S2 native speakers require exact reference transcripts")
references.append({
"audio_path": ref_path,
"text": f"<|speaker:{speaker_index}|>{ref_text}",
})
reference_labels.append(
f"local Speaker {speaker_index + 1}={resolved_voice_name(voice_ref)}"
)
components.append(component)
reference_texts.append(ref_text)
if references:
print(f"🎤 Fish local reference order: {', '.join(reference_labels)}")
params = {
"model_variant": self.config.get("model_variant", "s2-pro"),
"quantization": self.config.get("quantization", "none"),
"multi_speaker_mode": self.config.get("multi_speaker_mode", "Native Multi-Speaker"),
"seed": int(seed), "normalize": bool(self.config.get("normalize", True)),
"chunk_length": int(self.config.get("native_chunk_length", 200)),
"max_new_tokens": int(self.config.get("max_new_tokens", 1024)),
"top_p": float(self.config.get("top_p", 0.8)),
"repetition_penalty": float(self.config.get("repetition_penalty", 1.1)),
"temperature": float(self.config.get("temperature", 0.8)),
"cache_reference": bool(self.config.get("cache_reference", True)),
"context_length": int(self.config.get("context_length", 8192)),
}
cache_key = self.audio_cache.generate_cache_key(
"fish_audio_s2", text=text, audio_component="|".join(components),
reference_text="|".join(reference_texts), character=cache_character, **params,
) if enable_audio_cache else None
if cache_key:
cached = self.audio_cache.get_cached_audio(cache_key)
if cached:
print("💾 Fish Audio S2: Using cached audio")
return cached[0]
audio, sample_rate = self._engine().generate(text=text, references=references, **params)
if sample_rate != self.SAMPLE_RATE:
raise RuntimeError(f"Fish S2 returned unexpected sample rate {sample_rate}")
audio = audio.detach().float().cpu()
if audio.dim() == 1:
audio = audio.unsqueeze(0)
if cache_key:
self.audio_cache.cache_audio(cache_key, audio, audio.shape[-1] / self.SAMPLE_RATE)
return audio
+54 -41
View File
@@ -12,7 +12,8 @@ from typing import Dict, Any, Optional, List, Union
from engines.index_tts.index_tts import IndexTTSEngine
from engines.index_tts.index_tts_downloader import index_tts_downloader
from utils.text.character_parser import character_parser
from utils.voice.discovery import get_character_mapping, get_available_characters
from utils.voice.discovery import get_character_mapping, get_available_characters
from utils.voice.character_logging import resolved_character_label
from utils.audio.cache import get_audio_cache
@@ -76,7 +77,26 @@ class IndexTTSAdapter:
)
def generate(self,
@staticmethod
def _normalize_emotion_audio(emotion_audio):
"""Return a path or waveform dict accepted by IndexTTS-2."""
if not isinstance(emotion_audio, dict):
return emotion_audio
if emotion_audio.get("audio_path"):
print(
"🎭 Using Character Voices emotion audio: "
f"{emotion_audio.get('character_name', 'unknown')} -> "
f"{emotion_audio['audio_path']}"
)
return emotion_audio["audio_path"]
if "waveform" in emotion_audio:
return emotion_audio
nested_audio = emotion_audio.get("audio")
if isinstance(nested_audio, dict) and "waveform" in nested_audio:
return nested_audio
return emotion_audio
def generate(self,
text: str,
speaker_audio: Optional[str] = None,
emotion_audio: Optional[str] = None,
@@ -157,19 +177,10 @@ class IndexTTSAdapter:
# Determine final speaker and emotion audio
final_speaker_audio = speaker_audio
# Handle Character Voices emotion_audio format
if emotion_audio and isinstance(emotion_audio, dict):
if "audio_path" in emotion_audio:
# Character Voices format: {'audio': {...}, 'audio_path': 'path', ...}
final_emotion_audio = emotion_audio["audio_path"]
print(f"🎭 Using Character Voices emotion audio: {emotion_audio.get('character_name', 'unknown')} -> {final_emotion_audio}")
elif "waveform" in emotion_audio:
# Direct AUDIO format: {'waveform': tensor, 'sample_rate': rate}
final_emotion_audio = emotion_audio
else:
final_emotion_audio = emotion_audio
else:
final_emotion_audio = emotion_audio
# Normalize the two supported audio-reference shapes:
# Character Voices returns {audio: {waveform, sample_rate}, audio_path: ...},
# while ComfyUI AUDIO returns {waveform, sample_rate} directly.
final_emotion_audio = self._normalize_emotion_audio(emotion_audio)
# Only do character mapping if we actually have character tags
if has_character_tags:
@@ -263,18 +274,11 @@ class IndexTTSAdapter:
engine_kwargs['stream_return'] = stream_return
engine_kwargs['more_segment_before'] = more_segment_before
# Apply consistent emotion priority: emotion_audio takes precedence over other emotion controls
# This ensures consistent behavior whether using character tags or direct engine inputs
if final_emotion_audio:
# emotion_audio connected - disable other emotion controls
final_emotion_vector = None
final_use_emotion_text = False
final_emotion_text = None
else:
# No emotion_audio - use provided emotion controls
final_emotion_vector = emotion_vector
final_use_emotion_text = use_emotion_text
final_emotion_text = emotion_text
# Audio emotion and vector/text emotion are independent conditioning
# sources. IndexTTS-2 blends them in its latent emotion space.
final_emotion_vector = emotion_vector
final_use_emotion_text = use_emotion_text
final_emotion_text = emotion_text
# Generate audio with OOM protection
try:
@@ -391,7 +395,14 @@ class IndexTTSAdapter:
if unique_characters:
character_mapping = get_character_mapping(list(unique_characters), engine_type="index_tts")
print(f"🎭 IndexTTS-2: Processing {len(segments)} character segment(s) - {', '.join([s.get('character', 'narrator') for s in segments])}")
resolved_names = [
resolved_character_label(
segment.get('character', 'narrator'),
character_mapping.get(segment.get('character', 'narrator'), (default_speaker_audio, None)),
)
for segment in segments
]
print(f"🎭 IndexTTS-2: Processing {len(segments)} character segment(s) - {', '.join(resolved_names)}")
for segment in segments:
character_name = segment.get('character', 'narrator')
@@ -407,7 +418,7 @@ class IndexTTSAdapter:
character_audio_path = character_mapping[character_name][0]
if character_audio_path:
speaker_audio = character_audio_path
print(f"📖 Using character voice '{character_name}' | Ref: '{speaker_audio}'")
print(f"📖 Using character voice '{resolved_character_label(character_name, speaker_audio)}' | Ref: '{speaker_audio}'")
else:
print(f"⚠️ Character '{character_name}' has no audio reference, using default")
@@ -430,7 +441,7 @@ class IndexTTSAdapter:
# Check cache first
cached_segment_audio = self.audio_cache.get_cached_audio(segment_cache_key)
if cached_segment_audio:
print(f"💾 Using cached IndexTTS-2 segment for '{character_name}': '{segment_text[:30]}...'")
print(f"💾 Using cached IndexTTS-2 segment for '{resolved_character_label(character_name, speaker_audio)}': '{segment_text[:30]}...'")
segment_audio = cached_segment_audio[0]
else:
# Generate audio for this segment with OOM protection
@@ -534,16 +545,18 @@ class IndexTTSAdapter:
return "\n".join(analysis_parts)
def _get_stable_audio_identifier(self, audio_path: str) -> str:
"""
Get stable identifier for audio file using centralized audio hashing.
"""
if not audio_path:
return audio_path
# Use our centralized audio hashing utility
from utils.audio.audio_hash import generate_stable_audio_component
return generate_stable_audio_component(audio_file_path=audio_path)
def _get_stable_audio_identifier(self, audio_path: str) -> str:
"""
Get stable identifier for audio file using centralized audio hashing.
"""
if not audio_path:
return audio_path
# Use our centralized audio hashing utility
from utils.audio.audio_hash import generate_stable_audio_component
if isinstance(audio_path, dict):
return generate_stable_audio_component(reference_audio=audio_path)
return generate_stable_audio_component(audio_file_path=audio_path)
def get_supported_formats(self) -> List[str]:
"""Get supported audio formats."""
@@ -586,4 +599,4 @@ class IndexTTSAdapter:
def __del__(self):
"""Cleanup on deletion."""
self.unload()
self.unload()
+297
View File
@@ -0,0 +1,297 @@
"""
OmniVoice engine adapter.
Translates TTS Audio Suite processor calls into the official OmniVoice runtime.
"""
import os
import sys
from typing import Any, Dict, Optional, Tuple, Union
import torch
current_dir = os.path.dirname(__file__)
engines_dir = os.path.dirname(current_dir)
project_root = os.path.dirname(engines_dir)
if project_root not in sys.path:
sys.path.insert(0, project_root)
from utils.audio.audio_hash import generate_stable_audio_component
from utils.audio.cache import get_audio_cache
from utils.models.factory_config import ModelLoadConfig
from utils.models.language_mapper import resolve_language_alias
class OmniVoiceEngineAdapter:
"""Adapter for official OmniVoice inference."""
SAMPLE_RATE = 24000
def __init__(self, config: Optional[Dict[str, Any]] = None):
self.config = config.copy() if config else {}
self.audio_cache = get_audio_cache()
self._last_config: Optional[ModelLoadConfig] = None
self._load_signature = None
def update_config(self, new_config: Dict[str, Any]):
self.config = new_config.copy() if new_config else {}
def _build_load_signature(self) -> Tuple[Any, ...]:
return (
self.config.get("model_variant", "OmniVoice"),
self.config.get("device", "auto"),
self.config.get("dtype", "auto"),
)
def load_model(
self,
model_variant: str,
device: str = "auto",
dtype: str = "auto",
):
from utils.models.unified_model_interface import unified_model_interface
config = ModelLoadConfig(
engine_name="omnivoice",
model_type="tts",
model_name=model_variant,
model_path=model_variant,
device=device,
additional_params={
"dtype": dtype,
},
)
self._last_config = config
unified_model_interface.load_model(config)
def _ensure_model_loaded(self):
signature = self._build_load_signature()
if signature == self._load_signature and self._last_config is not None:
return
self.load_model(
model_variant=self.config.get("model_variant", "OmniVoice"),
device=self.config.get("device", "auto"),
dtype=self.config.get("dtype", "auto"),
)
self._load_signature = signature
def _get_engine(self):
if self._last_config is None:
self._ensure_model_loaded()
from utils.models.unified_model_interface import unified_model_interface
return unified_model_interface.load_model(self._last_config)
@staticmethod
def _seed_everything(seed: int):
if int(seed or 0) <= 0:
return
torch.manual_seed(int(seed))
if torch.cuda.is_available():
torch.cuda.manual_seed_all(int(seed))
if hasattr(torch, "xpu") and torch.xpu.is_available():
torch.xpu.manual_seed_all(int(seed))
@staticmethod
def _first_non_none(*values):
for value in values:
if value is None:
continue
if isinstance(value, str) and not value.strip():
continue
return value
return None
@staticmethod
def _normalize_waveform_tensor(waveform: torch.Tensor) -> torch.Tensor:
if waveform.dim() == 3:
waveform = waveform.squeeze(0)
if waveform.dim() == 1:
waveform = waveform.unsqueeze(0)
return waveform.detach().float().cpu()
def _extract_voice_reference(
self,
voice_ref: Optional[Dict[str, Any]],
) -> Tuple[Optional[Union[str, Tuple[torch.Tensor, int]]], str, str]:
if not voice_ref or not isinstance(voice_ref, dict):
return None, "", "default_voice"
prompt_text = (
voice_ref.get("reference_text")
or voice_ref.get("prompt_text")
or voice_ref.get("text")
or ""
).strip()
ref_audio = self._first_non_none(
voice_ref.get("prompt_audio_path"),
voice_ref.get("audio_path"),
voice_ref.get("audio"),
voice_ref.get("waveform"),
)
if ref_audio is None:
return None, prompt_text, "default_voice"
if isinstance(ref_audio, str):
return ref_audio, prompt_text, generate_stable_audio_component(audio_file_path=ref_audio)
if isinstance(ref_audio, dict) and "waveform" in ref_audio:
waveform = self._normalize_waveform_tensor(ref_audio["waveform"])
sample_rate = int(ref_audio.get("sample_rate", self.SAMPLE_RATE))
audio_dict = {"waveform": waveform, "sample_rate": sample_rate}
return (
(waveform, sample_rate),
prompt_text,
generate_stable_audio_component(reference_audio=audio_dict),
)
if torch.is_tensor(ref_audio):
waveform = self._normalize_waveform_tensor(ref_audio)
sample_rate = int(voice_ref.get("sample_rate", self.SAMPLE_RATE))
audio_dict = {"waveform": waveform, "sample_rate": sample_rate}
return (
(waveform, sample_rate),
prompt_text,
generate_stable_audio_component(reference_audio=audio_dict),
)
raise TypeError(f"Unsupported OmniVoice reference type: {type(ref_audio)}")
@staticmethod
def _normalize_language(language: Optional[str]) -> Optional[str]:
if language is None:
return None
normalized = str(language).strip()
if not normalized:
return None
lowered = normalized.lower()
if lowered in {"auto", "none"}:
return None
resolved = resolve_language_alias(normalized)
if resolved and resolved.lower() != lowered:
return resolved
return normalized
def generate_single(
self,
text: str,
voice_ref: Optional[Dict[str, Any]],
seed: int = 0,
enable_audio_cache: bool = True,
character_name: Optional[str] = None,
) -> torch.Tensor:
stripped = (text or "").strip()
if not stripped:
return torch.zeros(1, 0, dtype=torch.float32)
self._ensure_model_loaded()
engine = self._get_engine()
ref_audio, prompt_text, audio_component = self._extract_voice_reference(voice_ref)
language = self._normalize_language(self.config.get("language", "Auto"))
instruct = str(self.config.get("instruct", "") or "").strip() or None
model_variant = self.config.get("model_variant", "OmniVoice")
device = self.config.get("device", "auto")
dtype = self.config.get("dtype", "auto")
num_step = int(self.config.get("num_steps", 32))
guidance_scale = float(self.config.get("guidance_scale", 2.0))
t_shift = float(self.config.get("t_shift", 0.1))
speed = float(self.config.get("speed", 1.0))
duration_value = float(self.config.get("duration", 0.0))
duration = duration_value if duration_value > 0 else None
layer_penalty_factor = float(self.config.get("layer_penalty_factor", 5.0))
position_temperature = float(self.config.get("position_temperature", 5.0))
class_temperature = float(self.config.get("class_temperature", 0.0))
denoise = bool(self.config.get("denoise", True))
preprocess_prompt = bool(self.config.get("preprocess_prompt", True))
postprocess_output = bool(self.config.get("postprocess_output", True))
audio_chunk_duration = float(self.config.get("audio_chunk_duration", 15.0))
audio_chunk_threshold = float(self.config.get("audio_chunk_threshold", 30.0))
cache_key = None
if enable_audio_cache:
cache_key = self.audio_cache.generate_cache_key(
"omnivoice",
text=stripped,
audio_component=audio_component,
prompt_text=prompt_text,
model_variant=model_variant,
language=language or "none",
instruct=instruct or "",
num_step=num_step,
guidance_scale=guidance_scale,
t_shift=t_shift,
speed=speed,
duration=duration if duration is not None else "auto",
layer_penalty_factor=layer_penalty_factor,
position_temperature=position_temperature,
class_temperature=class_temperature,
denoise=denoise,
preprocess_prompt=preprocess_prompt,
postprocess_output=postprocess_output,
audio_chunk_duration=audio_chunk_duration,
audio_chunk_threshold=audio_chunk_threshold,
seed=int(seed or 0),
device=device,
dtype=dtype,
character=character_name or "narrator",
)
cached_audio = self.audio_cache.get_cached_audio(cache_key)
if cached_audio:
print(f"💾 Using cached OmniVoice audio for '{character_name or 'narrator'}': '{stripped[:30]}...'")
return cached_audio[0]
voice_clone_prompt = None
if ref_audio is not None:
if not prompt_text:
raise ValueError(
"OmniVoice voice cloning requires reference text. "
"Direct audio-only narrator input is unsupported. "
"Use Character Voices or a narrator voice file with matching reference text."
)
voice_clone_prompt = engine.create_voice_clone_prompt(
ref_audio=ref_audio,
ref_text=prompt_text,
preprocess_prompt=preprocess_prompt,
)
self._seed_everything(seed)
result = engine.generate(
text=stripped,
language=language,
voice_clone_prompt=voice_clone_prompt,
instruct=instruct,
duration=duration,
speed=speed,
num_step=num_step,
guidance_scale=guidance_scale,
t_shift=t_shift,
denoise=denoise,
preprocess_prompt=preprocess_prompt,
postprocess_output=postprocess_output,
layer_penalty_factor=layer_penalty_factor,
position_temperature=position_temperature,
class_temperature=class_temperature,
audio_chunk_duration=audio_chunk_duration,
audio_chunk_threshold=audio_chunk_threshold,
)
audio_tensor = result[0]
if not isinstance(audio_tensor, torch.Tensor):
audio_tensor = torch.tensor(audio_tensor, dtype=torch.float32)
audio_tensor = audio_tensor.detach().float().cpu()
if audio_tensor.dim() == 1:
audio_tensor = audio_tensor.unsqueeze(0)
if enable_audio_cache and cache_key:
duration_seconds = self.audio_cache._calculate_duration(audio_tensor, "omnivoice")
self.audio_cache.cache_audio(cache_key, audio_tensor, duration_seconds)
return audio_tensor
+81 -32
View File
@@ -25,6 +25,7 @@ from engines.vibevoice_engine.vibevoice_downloader import (
is_kugelaudio_variant_name,
)
from utils.models.manager import model_manager
from utils.voice.character_logging import resolved_character_label
class VibeVoiceEngineAdapter:
@@ -466,7 +467,8 @@ class VibeVoiceEngineAdapter:
# Generate each character group using VibeVoice format
for group_idx, (character, text_list) in enumerate(character_groups):
print(f"🎤 Group {group_idx + 1}: Character '{character}' with {len(text_list)} segments")
display_name = resolved_character_label(character, voice_mapping.get(character))
print(f"🎤 Group {group_idx + 1}: Character '{display_name}' with {len(text_list)} segments")
# Format as Speaker 1 entries (VibeVoice style) and combine
formatted_lines = []
@@ -526,15 +528,11 @@ class VibeVoiceEngineAdapter:
Returns:
Combined audio dict
"""
# Get speaker voice inputs from engine config for priority system
# For manual Speaker format, the main narrator voice comes from the TTS Text node
# We need to get it from a different source since voice_mapping may not have 'narrator' key
main_narrator_voice = voice_mapping.get('narrator') # Try narrator first
if main_narrator_voice is None:
# If no 'narrator' key, get it from any available voice (fallback for manual Speaker format)
available_voices = [v for v in voice_mapping.values() if v is not None]
main_narrator_voice = available_voices[0] if available_voices else None
# print(f"🐛 Debug: No 'narrator' key, using fallback voice: {'✅ found' if main_narrator_voice else '❌ none available'}")
# Get speaker voice inputs from engine config for priority system.
# Speaker 1 must only come from an explicit narrator/speaker-1 input.
# Do not synthesize Speaker 1 from discovered character voices, or aliases
# will look like they were overridden by a connection that does not exist.
main_narrator_voice = voice_mapping.get('narrator')
speaker_inputs = {
1: main_narrator_voice, # Speaker 1 uses main narrator from TTS Text
2: params.get('speaker2_voice'),
@@ -544,19 +542,24 @@ class VibeVoiceEngineAdapter:
# print(f"🐛 Debug: speaker_inputs[1] (main narrator): {'✅ has voice' if speaker_inputs[1] else '❌ no voice'}")
# Build speaker mapping and format text
# Build speaker mapping and format text.
# For SRT/global processing we keep speaker numbering local to the current
# generation call (Speaker 1..N in order of first appearance), but preserve
# the character's global slot when choosing connected speaker override inputs.
character_map = {}
# Pre-fill speaker_voices with all 4 speaker slots (some may be None)
speaker_voices = [
speaker_inputs.get(1), # Speaker 1
speaker_inputs.get(2), # Speaker 2
speaker_inputs.get(3), # Speaker 3
speaker_inputs.get(4) # Speaker 4
]
character_global_slots = {}
speaker_voices = []
formatted_lines = []
segment_characters = [char for char, _ in segments]
print(f"🎭 Native multi-speaker: Processing {len(segments)} segments with characters: {[char for char, _ in segments]}")
print(f"🎭 Native multi-speaker: Processing {len(segments)} segments with characters: {segment_characters}")
print(f"🎤 Speaker inputs connected: {[f'Speaker {k}' for k, v in speaker_inputs.items() if v is not None]}")
narrator_exists_globally = bool(global_char_to_speaker and "narrator" in global_char_to_speaker)
if speaker_inputs.get(1) is not None and "narrator" not in segment_characters:
if narrator_exists_globally:
print("ℹ️ Narrator input is connected, but this subtitle has no narrator turn; it will not override tagged characters")
else:
print("ℹ️ No narrator turns exist in this SRT; Speaker 1 maps to the first named character by first appearance")
for character, text in segments:
# Check if this is already a manual "Speaker N:" format
@@ -567,50 +570,84 @@ class VibeVoiceEngineAdapter:
speaker_idx = manual_speaker - 1 # Convert to 0-based
if speaker_idx >= 4:
speaker_idx = 3
# Voice already in pre-filled speaker_voices array
voice = speaker_voices[speaker_idx]
voice = speaker_inputs.get(manual_speaker)
if manual_speaker == 1:
print(f"🎤 Manual format 'Speaker {manual_speaker}' -> using {'✅ main narrator (Tony)' if voice else '❌ no narrator, using default'}")
else:
print(f"🎤 Manual format 'Speaker {manual_speaker}' -> using {'✅ connected input' if voice else '❌ no input, using default'}")
while len(speaker_voices) <= speaker_idx:
speaker_voices.append(None)
speaker_voices[speaker_idx] = voice
formatted_lines.append(f"Speaker {manual_speaker}: {text.strip()}")
else:
# Character tag format - use global mapping if provided (for SRT consistency)
# Character tag format
if character not in character_map:
# Special handling for numeric characters: [1] [2] [3] [4] -> map directly to Speaker N
if character.isdigit() and 1 <= int(character) <= 4:
speaker_idx = int(character) - 1 # Convert [1] to Speaker 1 (0-based index)
global_speaker_num = int(character)
speaker_idx = len(character_map)
character_map[character] = speaker_idx
character_global_slots[character] = global_speaker_num
print(f"🔢 Numeric character '[{character}]' -> Speaker {int(character)} (direct mapping)")
elif global_char_to_speaker and character in global_char_to_speaker:
# Use global mapping for consistent SRT processing
speaker_idx = global_char_to_speaker[character] - 1 # Convert to 0-based
global_speaker_num = global_char_to_speaker[character]
speaker_idx = len(character_map)
character_map[character] = speaker_idx
character_global_slots[character] = global_speaker_num
else:
# Fallback to sequential assignment
speaker_idx = len(character_map)
if speaker_idx >= 4:
print(f"⚠️ VibeVoice: Limiting to 4 speakers, '{character}' will use Speaker 4")
speaker_idx = 3 # Use 0-based internally, will convert to 1-based for format
global_speaker_num = 4
else:
character_map[character] = speaker_idx
global_speaker_num = speaker_idx + 1
character_global_slots[character] = global_speaker_num
# Priority system: speaker inputs override character aliases
# Priority system:
# - Numeric [1]-[4] always map directly to speaker inputs 1-4.
# - If any narrator turn exists in the SRT, Speaker 1 is reserved for narrator.
# - Otherwise, Speaker 1..4 map to named characters by first global appearance order.
speaker_num = speaker_idx + 1
connected_voice = speaker_inputs.get(speaker_num)
global_speaker_num = character_global_slots.get(character, speaker_num)
connected_voice = None
is_numeric_direct = character.isdigit() and 1 <= int(character) <= 4
if is_numeric_direct:
connected_voice = speaker_inputs.get(global_speaker_num)
elif character == "narrator":
connected_voice = speaker_inputs.get(1)
elif narrator_exists_globally:
if global_speaker_num >= 2:
connected_voice = speaker_inputs.get(global_speaker_num)
else:
connected_voice = speaker_inputs.get(global_speaker_num)
character_voice = voice_mapping.get(character)
if connected_voice is not None and character_voice is not None:
print(f"⚠️ Priority: Speaker {speaker_num} input overrides ['{character}'] alias - using connected voice")
if global_speaker_num != speaker_num:
print(
f"⚠️ Priority: Speaker {global_speaker_num} input overrides ['{character}'] alias "
f"- using connected voice as local Speaker {speaker_num}"
)
else:
print(f"⚠️ Priority: Speaker {speaker_num} input overrides ['{character}'] alias - using connected voice")
voice = connected_voice
elif connected_voice is not None:
print(f"🎤 Speaker {speaker_num}: Using connected voice input")
if global_speaker_num != speaker_num:
print(f"🎤 Speaker {global_speaker_num}: Using connected voice input as local Speaker {speaker_num}")
else:
print(f"🎤 Speaker {speaker_num}: Using connected voice input")
voice = connected_voice
else:
print(f"🎭 Character '{character}' -> Speaker {speaker_num}, using character voice")
if global_speaker_num != speaker_num:
print(f"🎭 Character '{character}' -> global Speaker {global_speaker_num}, local Speaker {speaker_num}, using alias/character voice")
else:
print(f"🎭 Character '{character}' -> Speaker {speaker_num}, using alias/character voice")
voice = character_voice
# Ensure we have enough speaker_voices slots
@@ -629,6 +666,18 @@ class VibeVoiceEngineAdapter:
print(formatted_text)
print("="*60)
print(f"🎤 Using {len(speaker_voices)} voice samples for generation")
for idx, voice in enumerate(speaker_voices, start=1):
if voice is None:
print(f" Speaker {idx}: default / no reference")
elif isinstance(voice, dict) and voice.get("audio_path"):
print(f" Speaker {idx}: file reference -> {voice['audio_path']}")
elif isinstance(voice, dict) and "waveform" in voice:
waveform = voice["waveform"]
sample_rate = voice.get("sample_rate", "unknown")
shape = tuple(waveform.shape) if hasattr(waveform, "shape") else "unknown"
print(f" Speaker {idx}: waveform reference -> shape {shape}, sr {sample_rate}")
else:
print(f" Speaker {idx}: unexpected reference type {type(voice).__name__}")
# Validate and normalize voice references
normalized_voices = []
+11 -9
View File
@@ -7,7 +7,8 @@ Works with character_grouper.py to process multiple segments simultaneously.
from typing import Dict, List, Any
import torch
from .character_grouper import CharacterGroup
from .character_grouper import CharacterGroup
from utils.voice.character_logging import resolved_character_label
class BatchProcessor:
@@ -49,7 +50,8 @@ class BatchProcessor:
character = character_group.character
segments = character_group.segments
print(f"🚀 BATCH PROCESSING: {character} - {len(segments)} segments in {language}")
display_name = resolved_character_label(character, voice_refs.get(character))
print(f"🚀 BATCH PROCESSING: {display_name} - {len(segments)} segments in {language}")
# Collect all texts for batching
batch_texts = []
@@ -75,7 +77,7 @@ class BatchProcessor:
char_audio_prompt = voice_refs[character]
# THE ACTUAL BATCH PROCESSING - this is the key improvement
print(f"⚡ Batch generating {len(batch_texts)} chunks for {character}")
print(f"⚡ Batch generating {len(batch_texts)} chunks for {display_name}")
print(f"🔧 DEBUG: batch_size from inputs = {inputs.get('batch_size', 4)}")
try:
batch_audio = self.tts_model.generate_batch(
@@ -137,15 +139,15 @@ class BatchProcessor:
character = character_group.character
segments = character_group.segments
print(f"→ SEQUENTIAL: {character} - {len(segments)} segments in {language}")
char_audio_prompt = voice_refs[character]
display_name = resolved_character_label(character, char_audio_prompt)
print(f"→ SEQUENTIAL: {display_name} - {len(segments)} segments in {language}")
results = {}
char_audio_prompt = voice_refs[character]
for segment in segments:
for segment in segments:
segment_display_idx = segment.original_idx + 1 # 1-based for display
print(f"🎤 Generating segment {segment_display_idx}/{total_segments} for '{character}' (lang: {language})")
print(f"🎤 Generating segment {segment_display_idx}/{total_segments} for '{display_name}' (lang: {language})")
# Apply chunking
if inputs["enable_chunking"] and len(segment.segment_text) > inputs["max_chars_per_chunk"]:
@@ -183,4 +185,4 @@ class BatchProcessor:
"""Apply crash protection padding to short texts."""
if len(text.strip()) < min_length:
return padding_template.format(seg=text)
return text
return text
@@ -7,7 +7,8 @@ Works with character_grouper.py to process multiple segments simultaneously.
from typing import Dict, List, Any
import torch
from .character_grouper import CharacterGroup
from .character_grouper import CharacterGroup
from utils.voice.character_logging import resolved_character_label
class BatchProcessor:
@@ -49,7 +50,8 @@ class BatchProcessor:
character = character_group.character
segments = character_group.segments
print(f"🚀 BATCH PROCESSING: {character} - {len(segments)} segments in {language}")
display_name = resolved_character_label(character, voice_refs.get(character))
print(f"🚀 BATCH PROCESSING: {display_name} - {len(segments)} segments in {language}")
# Collect all texts for batching
batch_texts = []
@@ -76,7 +78,7 @@ class BatchProcessor:
char_audio_prompt = voice_refs[character]
# THE ACTUAL BATCH PROCESSING - this is the key improvement
print(f"⚡ Batch generating {len(batch_texts)} chunks for {character}")
print(f"⚡ Batch generating {len(batch_texts)} chunks for {display_name}")
print(f"🔧 DEBUG: batch_size from inputs = {inputs.get('batch_size', 4)}")
try:
batch_audio = self.tts_model.generate_batch(
@@ -138,15 +140,15 @@ class BatchProcessor:
character = character_group.character
segments = character_group.segments
print(f"→ SEQUENTIAL: {character} - {len(segments)} segments in {language}")
char_audio_prompt = voice_refs[character]
display_name = resolved_character_label(character, char_audio_prompt)
print(f"→ SEQUENTIAL: {display_name} - {len(segments)} segments in {language}")
results = {}
char_audio_prompt = voice_refs[character]
for segment in segments:
for segment in segments:
segment_display_idx = segment.original_idx + 1 # 1-based for display
print(f"🎤 Generating segment {segment_display_idx}/{total_segments} for '{character}' (lang: {language})")
print(f"🎤 Generating segment {segment_display_idx}/{total_segments} for '{display_name}' (lang: {language})")
# Apply chunking
if inputs["enable_chunking"] and len(segment.segment_text) > inputs["max_chars_per_chunk"]:
@@ -184,4 +186,4 @@ class BatchProcessor:
"""Apply crash protection padding to short texts."""
if len(text.strip()) < min_length:
return padding_template.format(seg=text)
return text
return text
+2 -1
View File
@@ -16,7 +16,8 @@ from torch.nn.utils.rnn import pad_sequence
def seed_everything(seed=0):
random.seed(seed)
os.environ["PYTHONHASHSEED"] = str(seed)
# PYTHONHASHSEED is read before interpreter startup; changing it here has
# no effect on this process and can poison later isolated workers.
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
+5
View File
@@ -0,0 +1,5 @@
"""Fish Audio S2 Pro engine support."""
from .downloader import FishAudioS2Downloader
__all__ = ["FishAudioS2Downloader"]
+143
View File
@@ -0,0 +1,143 @@
"""Organized Hugging Face downloader for the gated Fish Audio S2 Pro model."""
import os
import shutil
import folder_paths
from utils.downloads.unified_downloader import unified_downloader
class FishAudioS2Downloader:
VARIANTS = {
"s2-pro": {
"repo_id": "fishaudio/s2-pro",
"directory": "fish_audio_s2_pro",
"files": [
".gitattributes", "LICENSE.md", "README.md", "chat_template.jinja", "codec.pth",
"config.json", "model-00001-of-00002.safetensors",
"model-00002-of-00002.safetensors", "model.safetensors.index.json",
"special_tokens_map.json", "tokenizer.json", "tokenizer_config.json",
],
},
"s2-pro-fp8": {
"repo_id": "drbaph/s2-pro-fp8",
"directory": "fish_audio_s2_pro_fp8",
"files": [
".gitattributes", "LICENSE.md", "README.md", "chat_template.jinja", "codec.pth",
"config.json", "model.safetensors", "quantization_info.json",
"special_tokens_map.json", "tokenizer.json", "tokenizer_config.json",
],
},
}
@classmethod
def is_model_complete(cls, model_path: str) -> bool:
"""Recognize a local Fish S2 checkpoint without loading model code."""
if not os.path.isdir(model_path):
return False
required = ("config.json", "codec.pth", "tokenizer.json")
if not all(os.path.isfile(os.path.join(model_path, name)) for name in required):
return False
try:
return any(
name.endswith(".safetensors")
and os.path.getsize(os.path.join(model_path, name)) > 0
for name in os.listdir(model_path)
)
except OSError:
return False
@classmethod
def local_model_paths(cls) -> list[str]:
"""Return complete local Fish checkpoints from all configured TTS paths."""
try:
from utils.models.extra_paths import get_all_tts_model_paths
search_paths = get_all_tts_model_paths("TTS")
except Exception:
search_paths = [os.path.join(folder_paths.models_dir, "TTS")]
found = {}
for base_path in search_paths:
if not os.path.isdir(base_path):
continue
try:
entries = os.listdir(base_path)
except OSError:
continue
for entry in entries:
candidate = os.path.join(base_path, entry)
if cls.is_model_complete(candidate):
found.setdefault(entry, candidate)
return [found[name] for name in sorted(found)]
@classmethod
def resolve_local_model_path(cls, selection: str) -> str:
"""Resolve a `local:<folder>` UI selection to its filesystem path."""
name = str(selection).removeprefix("local:")
for path in cls.local_model_paths():
if os.path.basename(os.path.normpath(path)) == name:
return path
raise FileNotFoundError(f"Local Fish Audio S2 model not found: {selection}")
@classmethod
def resolve_model_variant(cls, selection: str, model_path: str | None = None) -> str:
"""Map a UI selection/path to the runtime's known variant behavior."""
if selection in cls.VARIANTS:
return selection
path = model_path or selection
if os.path.isfile(os.path.join(path, "quantization_info.json")) or "fp8" in os.path.basename(path).lower():
return "s2-pro-fp8"
return "s2-pro"
@classmethod
def resolve_model_path(cls, selection: str) -> str:
"""Resolve local selections or download the selected official variant."""
if str(selection).startswith("local:"):
return cls.resolve_local_model_path(selection)
return cls.ensure_model(selection)
@classmethod
def model_dir(cls, variant: str = "s2-pro") -> str:
if variant not in cls.VARIANTS:
raise ValueError(f"Unknown Fish Audio S2 model variant: {variant}")
return os.path.join(folder_paths.models_dir, "TTS", cls.VARIANTS[variant]["directory"])
@classmethod
def ensure_model(cls, variant: str = "s2-pro") -> str:
details = cls.VARIANTS.get(variant)
if details is None:
raise ValueError(f"Unknown Fish Audio S2 model variant: {variant}")
target = cls.model_dir(variant)
if variant == "s2-pro-fp8":
target_codec = os.path.join(target, "codec.pth")
official_codec = os.path.join(cls.model_dir("s2-pro"), "codec.pth")
if not os.path.isfile(target_codec) and os.path.isfile(official_codec):
os.makedirs(target, exist_ok=True)
try:
os.link(official_codec, target_codec)
except OSError:
shutil.copy2(official_codec, target_codec)
files = details["files"]
if all(os.path.isfile(os.path.join(target, name)) and os.path.getsize(os.path.join(target, name)) > 0
for name in files):
return target
os.makedirs(target, exist_ok=True)
download_files = [{"remote": name, "local": name} for name in files]
try:
result = unified_downloader.download_huggingface_model(
repo_id=details["repo_id"],
model_name=variant,
files=download_files,
engine_type="fish_audio_s2",
target_dir=target,
)
if not result:
raise RuntimeError("one or more required model files failed to download")
except Exception as exc:
raise RuntimeError(
f"Fish Audio S2 variant '{variant}' is gated. Accept its non-commercial license at "
f"https://huggingface.co/{details['repo_id']} and authenticate Hugging Face "
f"before retrying. Download error: {exc}"
) from exc
return target
+83 -18
View File
@@ -46,7 +46,7 @@ from utils.audio.librosa_fallback import safe_load
class IndexTTS2:
def __init__(
self, cfg_path="checkpoints/config.yaml", model_dir="checkpoints", use_fp16=False, device=None,
use_cuda_kernel=None,use_deepspeed=False, use_torch_compile=False, low_vram=False
use_cuda_kernel=None,use_deepspeed=False, use_torch_compile=False, use_accel=False, low_vram=False
):
"""
Args:
@@ -57,10 +57,12 @@ class IndexTTS2:
use_cuda_kernel (None | bool): whether to use BigVGan custom fused activation CUDA kernel, only for CUDA device.
use_deepspeed (bool): whether to use DeepSpeed or not.
use_torch_compile (bool): whether to use torch.compile for optimization or not.
use_accel (bool): whether to enable GPT2 FlashAttention acceleration.
"""
use_fp16 = self._coerce_bool_flag(use_fp16)
use_deepspeed = self._coerce_bool_flag(use_deepspeed)
use_torch_compile = self._coerce_bool_flag(use_torch_compile)
use_accel = self._coerce_bool_flag(use_accel)
low_vram = self._coerce_bool_flag(low_vram)
if device is not None:
@@ -131,6 +133,7 @@ class IndexTTS2:
if use_torch_compile:
self._validate_torch_compile_support()
self.use_torch_compile = use_torch_compile
self.use_accel = use_accel
self.low_vram = low_vram
# Determine device for initial model loading
@@ -139,6 +142,10 @@ class IndexTTS2:
# Initialize QwenEmotion for text-based emotion control if available
self.qwen_emo = None
# A text-emotion model failure must not abort an otherwise valid TTS run.
# Once it fails, skip it for the lifetime of this engine instance instead
# of repeatedly hitting a poisoned CUDA/accelerate state on later segments.
self._qwen_emo_disabled = False
self.qwen_emo_path = None # Store path for lazy loading
try:
@@ -162,7 +169,7 @@ class IndexTTS2:
print(f"⚠️ QwenEmotion model incomplete - missing files: {missing_files}")
print("ℹ️ Falling back to audio emotion only")
else:
self.qwen_emo = QwenEmotion(normalized_path)
self.qwen_emo = QwenEmotion(normalized_path, preferred_device=self.device)
print("✅ QwenEmotion loaded - text emotion support enabled")
else:
print("ℹ️ QwenEmotion not available - audio emotion only")
@@ -172,7 +179,9 @@ class IndexTTS2:
print("ℹ️ Falling back to audio emotion only")
print("🔄 IndexTTS-2: Loading GPT model...")
self.gpt = UnifiedVoice(**self.cfg.gpt)
gpt_config = OmegaConf.to_container(self.cfg.gpt, resolve=True)
gpt_config["use_accel"] = self.use_accel
self.gpt = UnifiedVoice(**gpt_config)
self.gpt_path = os.path.join(self.model_dir, self.cfg.gpt_checkpoint)
load_checkpoint(self.gpt, self.gpt_path)
print(f"🔄 IndexTTS-2: Moving GPT to {self.load_device}...")
@@ -571,23 +580,38 @@ class IndexTTS2:
f"emo_text:{emo_text}")
start_time = time.perf_counter()
if use_emo_text or emo_vector is not None:
# we're using a text or emotion vector guidance; so we must remove
# "emotion reference voice", to ensure we use correct emotion mixing!
emo_audio_prompt = None
# Text/vector emotion and an external emotion reference can be used
# together. The vector is projected into the same conditioning space
# later and blended with the audio-derived conditioning.
if use_emo_text:
# automatically generate emotion vectors from text prompt
if self.qwen_emo is None:
print("⚠️ QwenEmotion model not available - cannot use text emotion. Ignoring use_emo_text=True")
if self.qwen_emo is None or self._qwen_emo_disabled:
print("⚠️ QwenEmotion text analysis unavailable - continuing without text emotion")
use_emo_text = False
else:
if emo_text is None:
emo_text = text # use main text prompt
emo_dict = self.qwen_emo.inference(emo_text)
print(f"detected emotion vectors from text: {emo_dict}")
# convert ordered dict to list of vectors; the order is VERY important!
emo_vector = list(emo_dict.values())
try:
emo_dict = self.qwen_emo.inference(emo_text)
except Exception as exc:
# QwenEmotion is an optional classifier. CUDA/accelerate
# failures here should degrade to ordinary IndexTTS-2
# synthesis, rather than discarding every audio segment.
self._qwen_emo_disabled = True
use_emo_text = False
# Do not silently substitute a stale/global vector for the
# requested text emotion. Continue with voice-derived
# conditioning instead.
emo_vector = None
print(
"⚠️ QwenEmotion text analysis failed; continuing without "
f"text emotion ({type(exc).__name__}: {exc})"
)
else:
print(f"detected emotion vectors from text: {emo_dict}")
# convert ordered dict to list of vectors; the order is VERY important!
emo_vector = list(emo_dict.values())
if emo_vector is not None:
# Apply normalization to prevent voice identity loss
@@ -615,6 +639,9 @@ class IndexTTS2:
emo_vector_scaled_display = [round(x, 3) for x in emo_vector]
print(f"🎭 Scaled by alpha {emo_vector_scale}: {emo_vector_scaled_display}")
if emo_audio_prompt is not None and emo_vector is not None:
print("🎭 Blending emotion-reference audio with vector/text emotion")
if emo_audio_prompt is None:
# we are not using any external "emotion reference voice"; use
# speaker's voice as the main emotion reference audio.
@@ -1027,11 +1054,18 @@ def find_most_similar_cosine(query_vector, matrix):
return most_similar_index
class QwenEmotion:
def __init__(self, model_dir):
def __init__(self, model_dir, preferred_device=None):
self.model_dir = model_dir
self.tokenizer = AutoTokenizer.from_pretrained(self.model_dir)
import torch
load_kwargs = {"device_map": "auto"}
# Auto-sharding split this small classifier across both GPUs in some
# ComfyUI setups. Accelerate then tried to move Qwen's cache between
# devices and produced CUDA ``invalid argument`` errors. Keep Qwen on
# one GPU; prefer the device with the most available memory.
qwen_device = self._select_device(preferred_device)
load_kwargs = {"device_map": {"": qwen_device}} if qwen_device else {}
if qwen_device:
print(f"🤖 QwenEmotion loading on {qwen_device} (single device)")
try:
self.model = AutoModelForCausalLM.from_pretrained(
self.model_dir,
@@ -1080,7 +1114,28 @@ class QwenEmotion:
self.max_score = 1.2
self.min_score = 0.0
@staticmethod
def _select_device(preferred_device=None):
preferred = str(preferred_device) if preferred_device is not None else None
if not torch.cuda.is_available():
return preferred or "cpu"
# QwenEmotion is small enough for one GPU. Selecting the roomiest GPU
# avoids forcing it onto a nearly-full IndexTTS device.
try:
free_memory = [torch.cuda.mem_get_info(index)[0] for index in range(torch.cuda.device_count())]
return f"cuda:{max(range(len(free_memory)), key=free_memory.__getitem__)}"
except Exception:
return preferred if preferred and preferred.startswith("cuda") else "cuda:0"
def clamp_score(self, value):
try:
value = float(value)
except (TypeError, ValueError) as exc:
raise ValueError(
f"QwenEmotion returned a non-numeric emotion score {value!r}. "
"Please retry the request."
) from exc
return max(self.min_score, min(self.max_score, value))
def convert(self, content):
@@ -1113,15 +1168,25 @@ class QwenEmotion:
add_generation_prompt=True,
enable_thinking=False,
)
model_inputs = self.tokenizer([text], return_tensors="pt").to(self.model.device)
# With device_map="auto", model.device is the input/first device exposed
# by accelerate. Keep the move explicit instead of relying on a chained
# BatchEncoding operation, which has triggered invalid CUDA arguments on
# some transformers/accelerate combinations.
input_device = getattr(self.model, "device", None)
if input_device is None or str(input_device) == "meta":
input_device = next(self.model.parameters()).device
model_inputs = self.tokenizer([text], return_tensors="pt")
model_inputs = {key: value.to(input_device) for key, value in model_inputs.items()}
# conduct text completion
generated_ids = self.model.generate(
**model_inputs,
max_new_tokens=32768,
# The classifier returns a tiny JSON object; an unbounded 32k-token
# generation wastes VRAM and makes CUDA failures more likely.
max_new_tokens=256,
pad_token_id=self.tokenizer.eos_token_id
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()
output_ids = generated_ids[0][len(model_inputs["input_ids"][0]):].tolist()
# parsing thinking content
try:
+8
View File
@@ -0,0 +1,8 @@
"""
OmniVoice engine integration.
"""
from .omnivoice_downloader import OmniVoiceDownloader
from .omnivoice_engine import OmniVoiceEngine
__all__ = ["OmniVoiceDownloader", "OmniVoiceEngine"]
+148
View File
@@ -0,0 +1,148 @@
"""
OmniVoice model downloader.
Downloads the official OmniVoice assets into ComfyUI/models/TTS/omnivoice/.
"""
import logging
import os
from contextlib import contextmanager
from typing import Dict, Optional
import folder_paths
from huggingface_hub import snapshot_download
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
@contextmanager
def _suppress_hf_http_logs():
logger_names = ("httpx", "httpcore", "huggingface_hub")
original_levels = {}
try:
for name in logger_names:
logger = logging.getLogger(name)
original_levels[name] = logger.level
logger.setLevel(logging.WARNING)
yield
finally:
for name, level in original_levels.items():
logging.getLogger(name).setLevel(level)
class OmniVoiceDownloader:
"""Resolve and download official OmniVoice model folders."""
MODELS: Dict[str, Dict[str, object]] = {
"OmniVoice": {
"repo_id": "k2-fsa/OmniVoice",
"description": "Official OmniVoice multilingual zero-shot TTS checkpoint",
"required_files": [
"chat_template.jinja",
"config.json",
"model.safetensors",
"tokenizer.json",
"tokenizer_config.json",
"audio_tokenizer/config.json",
"audio_tokenizer/model.safetensors",
"audio_tokenizer/preprocessor_config.json",
],
},
}
def __init__(self, base_path: Optional[str] = None):
if base_path is None:
try:
self.base_path = get_preferred_download_path(model_type="TTS", engine_name="omnivoice")
except Exception:
self.base_path = os.path.join(folder_paths.models_dir, "TTS", "omnivoice")
else:
self.base_path = base_path
os.makedirs(self.base_path, exist_ok=True)
def resolve_model_path(self, model_identifier: str) -> str:
"""Resolve absolute paths, local: names, or known official identifiers."""
if not model_identifier:
model_identifier = "OmniVoice"
if os.path.isabs(model_identifier) and os.path.exists(model_identifier):
return model_identifier
if os.path.isdir(model_identifier):
return model_identifier
if model_identifier.startswith("local:"):
local_name = model_identifier[6:]
for base_path in get_all_tts_model_paths("TTS"):
for folder_name in ("omnivoice", "OmniVoice", ""):
candidate = (
os.path.join(base_path, folder_name, local_name)
if folder_name else os.path.join(base_path, local_name)
)
if os.path.exists(candidate):
print(f"📁 Using local OmniVoice model: {candidate}")
return candidate
raise FileNotFoundError(f"Local OmniVoice model not found: {local_name}")
if model_identifier in self.MODELS:
return self.get_model_path(model_identifier)
raise FileNotFoundError(f"Unknown OmniVoice model identifier: {model_identifier}")
def get_model_path(self, model_name: str) -> str:
model_dir = os.path.join(self.base_path, model_name)
if not os.path.exists(model_dir) or not self._is_omnivoice_model_complete(model_dir):
return self.download_model(model_name, force=os.path.exists(model_dir))
return model_dir
def download_model(self, model_name: str, force: bool = False) -> str:
if model_name not in self.MODELS:
raise ValueError(f"Unknown OmniVoice model: {model_name}")
model_info = self.MODELS[model_name]
repo_id = str(model_info["repo_id"])
model_dir = os.path.join(self.base_path, model_name)
print(f"\n{'=' * 60}")
print("📦 OmniVoice Model Download")
print(f"{'=' * 60}")
print(f"Model: {model_name}")
print(f"Description: {model_info['description']}")
print(f"Repository: {repo_id}")
print(f"Target: {model_dir}")
print(f"{'=' * 60}\n")
try:
with _suppress_hf_http_logs():
snapshot_download(
repo_id=repo_id,
local_dir=model_dir,
local_dir_use_symlinks=False,
resume_download=True,
force_download=force,
)
except Exception as e:
raise RuntimeError(f"Failed to download {model_name} from {repo_id}: {e}") from e
if not self._is_omnivoice_model_complete(model_dir):
raise RuntimeError(f"Downloaded OmniVoice model is incomplete: {model_dir}")
print(f"✅ OmniVoice model ready: {model_dir}")
return model_dir
def _is_omnivoice_model_complete(self, model_dir: str) -> bool:
if not os.path.isdir(model_dir):
return False
required_files = self.MODELS["OmniVoice"]["required_files"]
missing = [
rel_path
for rel_path in required_files
if not os.path.exists(os.path.join(model_dir, rel_path))
]
if missing:
print(f"❌ OmniVoice model incomplete. Missing {len(missing)} file(s):")
for rel_path in missing:
print(f" - {rel_path}")
return False
return True
+263
View File
@@ -0,0 +1,263 @@
"""
OmniVoice engine wrapper.
Wraps the official `omnivoice` package with ComfyUI-friendly lifecycle hooks.
"""
import importlib
import logging
import os
import sys
import types
from contextlib import contextmanager
from typing import Any, Optional
import torch
current_dir = os.path.dirname(__file__)
engines_dir = os.path.dirname(current_dir)
project_root = os.path.dirname(engines_dir)
if project_root not in sys.path:
sys.path.insert(0, project_root)
from engines.omnivoice.omnivoice_downloader import OmniVoiceDownloader
from engines.omnivoice.progress_callback import OmniVoiceGenerationProgress
from utils.device import resolve_torch_device
@contextmanager
def _suppress_transformers_logs():
logger = logging.getLogger("transformers.integrations.tensor_parallel")
class _TensorParallelPlanFilter(logging.Filter):
_suppressed_prefixes = (
"The following layers were not sharded:",
"The following TP rules were not applied on any of the layers:",
)
def filter(self, record):
return not record.getMessage().startswith(self._suppressed_prefixes)
message_filter = _TensorParallelPlanFilter()
logger.addFilter(message_filter)
try:
yield
finally:
logger.removeFilter(message_filter)
class OmniVoiceEngine:
"""Thin wrapper around the official OmniVoice model."""
SAMPLE_RATE = 24000
def __init__(
self,
model_name: str = "OmniVoice",
device: str = "auto",
dtype: str = "auto",
model_dir: Optional[str] = None,
):
self.model_name = model_name
self.device = resolve_torch_device(device)
self.dtype = self._resolve_dtype(dtype, self.device)
self.downloader = OmniVoiceDownloader()
self.model_dir = model_dir or self.downloader.resolve_model_path(model_name)
self._model = None
def _install_progress_hook(self):
if self._model is None or getattr(self._model, "_tts_suite_progress_patched", False):
return
original_forward = self._model.forward
original_generate_iterative = self._model._generate_iterative
def wrapped_forward(model_self, *args, **kwargs):
result = original_forward(*args, **kwargs)
tracker = getattr(model_self, "_tts_suite_progress_tracker", None)
if tracker is not None:
current_step = getattr(model_self, "_tts_suite_progress_step", 0) + 1
model_self._tts_suite_progress_step = current_step
tracker.update(current_step, force=(current_step == 1 or current_step >= tracker.total_steps))
return result
def wrapped_generate_iterative(model_self, task, gen_config):
total_steps = int(getattr(gen_config, "num_step", 0) or 0)
tracker = OmniVoiceGenerationProgress(total_steps)
model_self._tts_suite_progress_tracker = tracker
model_self._tts_suite_progress_step = 0
try:
result = original_generate_iterative(task, gen_config)
tracker.end(getattr(model_self, "_tts_suite_progress_step", 0))
return result
except Exception:
tracker.abort()
raise
finally:
model_self._tts_suite_progress_tracker = None
model_self._tts_suite_progress_step = 0
self._model.forward = types.MethodType(wrapped_forward, self._model)
self._model._generate_iterative = types.MethodType(wrapped_generate_iterative, self._model)
self._model._tts_suite_progress_tracker = None
self._model._tts_suite_progress_step = 0
self._model._tts_suite_progress_patched = True
@staticmethod
def _resolve_dtype(dtype: str, device: str):
normalized = str(dtype or "auto").lower()
explicit = {
"float16": torch.float16,
"fp16": torch.float16,
"bfloat16": torch.bfloat16,
"bf16": torch.bfloat16,
"float32": torch.float32,
"fp32": torch.float32,
}
if normalized in explicit:
if device == "cpu" and explicit[normalized] != torch.float32:
return torch.float32
return explicit[normalized]
if device == "cpu":
return torch.float32
if device == "cuda" and torch.cuda.is_available():
major, _minor = torch.cuda.get_device_capability()
if major >= 8:
return torch.bfloat16
return torch.float16
if device == "xpu":
return torch.bfloat16
return torch.float16
def _import_official_package(self):
importlib.invalidate_caches()
nodes_dir = os.path.join(project_root, "nodes")
blocked_paths = {
os.path.abspath(engines_dir),
os.path.abspath(nodes_dir),
}
original_sys_path = list(sys.path)
sys.path[:] = [
path for path in sys.path
if os.path.abspath(path or os.getcwd()) not in blocked_paths
]
stale_modules = []
for module_name, module in list(sys.modules.items()):
if not (module_name == "omnivoice" or module_name.startswith("omnivoice.")):
continue
module_file = getattr(module, "__file__", "") or ""
if module_file and os.path.abspath(module_file).startswith(os.path.abspath(project_root)):
stale_modules.append((module_name, module))
del sys.modules[module_name]
try:
from omnivoice import OmniVoice # type: ignore
except Exception as e:
for module_name, module in stale_modules:
sys.modules.setdefault(module_name, module)
raise ImportError(
"Failed to import the official `omnivoice` package in the active ComfyUI Python "
"environment. This is usually a missing install or import-path collision. "
f"Active Python: {sys.executable}"
) from e
finally:
sys.path[:] = original_sys_path
return OmniVoice
def _loaded_model_device(self) -> Optional[str]:
if self._model is None:
return None
try:
first_param = next(self._model.parameters())
return str(first_param.device)
except Exception:
return None
def _ensure_model_device(self):
if self._model is None:
return
current_device = self._loaded_model_device()
if current_device == self.device:
return
self.to(self.device)
def _ensure_model_loaded(self):
if self._model is not None:
self._ensure_model_device()
return
OmniVoice = self._import_official_package()
print(f"🔄 Loading OmniVoice model via official package")
print(f" Path: {self.model_dir}")
print(f" Device: {self.device} | Dtype: {self.dtype}")
with _suppress_transformers_logs():
self._model = OmniVoice.from_pretrained(
self.model_dir,
device_map=self.device,
dtype=self.dtype,
load_asr=False,
)
self._install_progress_hook()
print("✅ OmniVoice model loaded successfully")
@property
def sampling_rate(self) -> int:
if self._model is not None and getattr(self._model, "sampling_rate", None):
return int(self._model.sampling_rate)
return self.SAMPLE_RATE
def create_voice_clone_prompt(
self,
ref_audio: Any,
ref_text: Optional[str] = None,
preprocess_prompt: bool = True,
):
self._ensure_model_loaded()
if ref_text is None or not str(ref_text).strip():
raise ValueError(
"OmniVoice voice cloning in this suite requires reference text. "
"Use Character Voices or a narrator voice with a matching .reference.txt file."
)
return self._model.create_voice_clone_prompt(
ref_audio=ref_audio,
ref_text=ref_text,
preprocess_prompt=preprocess_prompt,
)
def generate(self, **kwargs):
self._ensure_model_loaded()
return self._model.generate(**kwargs)
def to(self, device: str):
self.device = resolve_torch_device(device)
if self._model is None:
return self
try:
if hasattr(self._model, "to"):
self._model = self._model.to(self.device)
except Exception as e:
print(f"⚠️ OmniVoice base model move to {self.device} failed: {e}")
for attr_name in ("audio_tokenizer",):
attr_value = getattr(self._model, attr_name, None)
if hasattr(attr_value, "to"):
try:
setattr(self._model, attr_name, attr_value.to(self.device))
except Exception as e:
print(f"⚠️ OmniVoice {attr_name} move to {self.device} failed: {e}")
return self
def __getattr__(self, name: str):
if name.startswith("_"):
raise AttributeError(name)
if self._model is not None and hasattr(self._model, name):
return getattr(self._model, name)
raise AttributeError(name)
+63
View File
@@ -0,0 +1,63 @@
"""
OmniVoice progress callback.
Tracks real iterative decoding steps per second for OmniVoice generation.
"""
import time
class OmniVoiceGenerationProgress:
"""Render real iterative decode progress with it/s."""
def __init__(self, total_steps: int):
self.total_steps = max(1, int(total_steps or 1))
self.start_time = time.time()
self.last_print_time = self.start_time
self.last_step = 0
self.rendered = False
def update(self, current_step: int, force: bool = False):
current_step = max(0, min(int(current_step), self.total_steps))
current_time = time.time()
should_print = (
force
or current_step == 1
or current_time - self.last_print_time >= 0.5
)
if not should_print:
return
delta_steps = current_step - self.last_step
delta_time = max(current_time - self.last_print_time, 1e-6)
its = delta_steps / delta_time
elapsed = current_time - self.start_time
remaining_steps = max(self.total_steps - current_step, 0)
eta = remaining_steps / its if its > 0 else 0.0
bar_width = 12
filled = int(bar_width * current_step / self.total_steps) if self.total_steps > 0 else 0
filled = min(filled, bar_width)
progress_bar = f"[{'█' * filled}{'░' * (bar_width - filled)}] {current_step}/{self.total_steps}"
print(
f"\r Progress: {progress_bar} | {its:.1f} it/s | {elapsed:.0f}s | ETA {eta:.0f}s ",
end="",
flush=True,
)
self.last_print_time = current_time
self.last_step = current_step
self.rendered = True
def end(self, final_step: int):
final_step = max(0, int(final_step))
total_time = max(time.time() - self.start_time, 1e-6)
avg_its = final_step / total_time
if self.rendered:
print(
f"\r Complete: {final_step} steps in {total_time:.1f}s (avg {avg_its:.1f} it/s)"
+ " " * 30
)
def abort(self):
if self.rendered:
print()
+107 -61
View File
@@ -55,6 +55,7 @@ from utils.audio.edit_post_processor import process_segments as apply_edit_post_
from utils.text.character_parser import CharacterParser
from utils.text.pause_processor import PauseTagProcessor
from utils.text.segment_parameters import apply_segment_parameters
from utils.text.index_tts_emotion import resolve_inline_emotion
from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment
from utils.voice.discovery import get_character_mapping
from engines.adapters.index_tts_adapter import IndexTTSAdapter
@@ -248,15 +249,44 @@ class IndexTTSProcessor:
print(f"⚠️ {character}: No voice available")
# Define TTS generation function for pause processor
def tts_generate_func(text_content: str, segment_params: Optional[Dict[str, Any]] = None) -> torch.Tensor:
def tts_generate_func(
text_content: str,
segment_params: Optional[Dict[str, Any]] = None,
character_name: Optional[str] = None,
emotion_reference: Optional[str] = None,
) -> torch.Tensor:
# Import references for nested function scope
import torchaudio as ta
import tempfile as tf
"""TTS generation function for pause tag processor with per-segment parameters"""
# Apply segment parameters if provided
current_config = dict(self.config)
if segment_params:
current_config = apply_segment_parameters(current_config, segment_params, "index_tts")
import torchaudio as ta
import tempfile as tf
"""TTS generation function for pause tag processor with per-segment parameters"""
def generate_with_context(context: str, **kwargs):
"""Add segment context without hiding the original engine error."""
try:
return self.adapter.generate(**kwargs)
except Exception as exc:
# Avoid leaking temporary WAV files when an engine
# failure happens before the normal success cleanup.
for path in (kwargs.get("speaker_audio"), kwargs.get("emotion_audio")):
if (
isinstance(path, str)
and path.startswith(tf.gettempdir())
and os.path.exists(path)
):
try:
os.unlink(path)
except OSError:
pass
raise RuntimeError(
f"IndexTTS-2 segment failed ({context}): "
f"{type(exc).__name__}: {exc}"
) from exc
# Apply segment parameters if provided
current_config = dict(self.config)
if segment_params:
current_config = apply_segment_parameters(current_config, segment_params, "index_tts")
current_config, _ = resolve_inline_emotion(current_config)
if '[' in text_content and ']' in text_content:
# Handle character switching with emotion parsing using modularized parser
@@ -299,8 +329,10 @@ class IndexTTSProcessor:
else:
print(f"🐛 Could not resolve emotion reference '{emotion}'")
# Fall back to config emotion_audio if no tag emotion
if not emotion_audio_path:
# Fall back to connected emotion audio if no character
# reference was specified. Inline vector/text emotion
# can now be blended with this audio source.
if not emotion_audio_path:
emotion_from_config = self.config.get('emotion_audio')
# Process emotion audio for this character
if emotion_from_config:
@@ -322,26 +354,20 @@ class IndexTTSProcessor:
else:
print(f"🎭 No emotion audio for character: {character} (no tag emotion, no connected engine emotion)")
# Prioritize character emotion reference over global emotion controls
# If character has specific emotion ref, disable global emotion controls
if emotion_audio_path:
# Character has specific emotion - use only that emotion reference
character_emotion_vector = None
character_use_emotion_text = False
character_emotion_text = None
else:
# No character emotion - use global emotion settings (from current_config with segment params)
character_emotion_vector = current_config.get('emotion_vector')
character_use_emotion_text = current_config.get('use_emotion_text', False)
character_emotion_text = current_config.get('emotion_text')
# Handle dynamic QwenEmotion template
if character_use_emotion_text and character_emotion_text and current_config.get('is_dynamic_template', False):
character_emotion_text = self._process_dynamic_emotion_template(character_emotion_text, segment_text)
# Audio and vector/text controls are blended by
# IndexTTS-2 when both are present.
character_emotion_vector = current_config.get('emotion_vector')
character_use_emotion_text = current_config.get('use_emotion_text', False)
character_emotion_text = current_config.get('emotion_text')
# Handle dynamic QwenEmotion template
if character_use_emotion_text and character_emotion_text and current_config.get('is_dynamic_template', False):
character_emotion_text = self._process_dynamic_emotion_template(character_emotion_text, segment_text)
# Generate audio for this character segment (use parameters from current_config with segment overrides)
segment_result = self.adapter.generate(
text=segment_text,
segment_result = generate_with_context(
f"character '{character}', text={segment_text[:120]!r}",
text=segment_text,
speaker_audio=speaker_audio_path,
emotion_audio=emotion_audio_path,
emotion_alpha=current_config.get('emotion_alpha', 1.0),
@@ -383,10 +409,13 @@ class IndexTTSProcessor:
return torch.cat(segment_audio_parts, dim=-1)
else:
return torch.zeros(1, 0)
else:
# Simple text segment without character switching - use narrator voice
narrator_audio = voice_refs.get("narrator")
speaker_audio_path = None
else:
# Character tags are removed before this branch is called
# for parameterized segments. Keep the parsed character
# here so [Bob:emotion] does not silently use the narrator.
character_audio = voice_refs.get(character_name) if character_name else None
narrator_audio = character_audio or voice_refs.get("narrator")
speaker_audio_path = None
if narrator_audio and 'waveform' in narrator_audio:
with tf.NamedTemporaryFile(suffix='.wav', delete=False) as tmp_file:
save_audio_safe(tmp_file.name, narrator_audio['waveform'], narrator_audio['sample_rate'])
@@ -399,8 +428,22 @@ class IndexTTSProcessor:
save_audio_safe(tmp_file.name, narrator_voice_dict['waveform'], narrator_voice_dict['sample_rate'])
speaker_audio_path = tmp_file.name
# Handle emotion_audio - convert tensor to file path if needed
emotion_audio_path = self.config.get('emotion_audio')
# Character emotion references are preserved on the parsed
# segment even after its tag has been stripped. Resolve
# that reference before falling back to the node's global
# emotion audio.
emotion_audio_path = None
if emotion_reference:
emotion_audio_path = self.character_parser.get_emotion_voice_path(emotion_reference)
if emotion_audio_path:
print(
f"😊 Using emotion reference from segment: "
f"{emotion_reference} -> {emotion_audio_path}"
)
else:
print(f"⚠️ Could not resolve emotion reference '{emotion_reference}'")
if not emotion_audio_path:
emotion_audio_path = self.config.get('emotion_audio')
# Process emotion audio from config
if emotion_audio_path and isinstance(emotion_audio_path, dict) and 'waveform' in emotion_audio_path:
# Convert tensor to temporary file
@@ -419,25 +462,19 @@ class IndexTTSProcessor:
else:
print(f"🎭 No emotion audio for simple text segment (no connected engine emotion)")
# Prioritize connected emotion_audio over global emotion controls
# For simple text, use emotion_audio from config if available, else use global settings (with segment params)
if emotion_audio_path:
# Engine emotion_audio connected - use only that
simple_emotion_vector = None
simple_use_emotion_text = False
simple_emotion_text = None
else:
# No engine emotion_audio - use global emotion settings (from current_config with segment params)
simple_emotion_vector = current_config.get('emotion_vector')
simple_use_emotion_text = current_config.get('use_emotion_text', False)
simple_emotion_text = current_config.get('emotion_text')
# Audio and vector/text controls are blended by
# IndexTTS-2 when both are present.
simple_emotion_vector = current_config.get('emotion_vector')
simple_use_emotion_text = current_config.get('use_emotion_text', False)
simple_emotion_text = current_config.get('emotion_text')
# Handle dynamic QwenEmotion template
if simple_use_emotion_text and simple_emotion_text and current_config.get('is_dynamic_template', False):
simple_emotion_text = self._process_dynamic_emotion_template(simple_emotion_text, text_content)
# Handle dynamic QwenEmotion template
if simple_use_emotion_text and simple_emotion_text and current_config.get('is_dynamic_template', False):
simple_emotion_text = self._process_dynamic_emotion_template(simple_emotion_text, text_content)
result = self.adapter.generate(
text=text_content,
result = generate_with_context(
f"text={text_content[:120]!r}",
text=text_content,
speaker_audio=speaker_audio_path,
emotion_audio=emotion_audio_path,
emotion_alpha=current_config.get('emotion_alpha', 1.0),
@@ -505,7 +542,12 @@ class IndexTTSProcessor:
print(f" ⚠️ Segment {seg_idx + 1}: Only inline edit tags found, skipping empty TTS segment")
continue
segment_audio = tts_generate_func(clean_segment_text or subseg_content, seg_obj.parameters)
segment_audio = tts_generate_func(
clean_segment_text or subseg_content,
seg_obj.parameters,
seg_obj.character,
seg_obj.emotion,
)
if isinstance(segment_audio, torch.Tensor) and segment_audio.numel() > 0:
if segment_audio.dim() == 1:
segment_audio = segment_audio.unsqueeze(0)
@@ -533,11 +575,13 @@ class IndexTTSProcessor:
segments, clean_text = self.pause_processor.parse_pause_tags(text)
# Create a wrapper that handles pause segments
def tts_generate_with_params(text_content: str) -> torch.Tensor:
"""Wrapper for pause processor that passes segment parameters"""
# For pause-based processing, use first segment's parameters
segment_params = character_segment_objects[0].parameters if character_segment_objects and character_segment_objects[0].parameters else None
return tts_generate_func(text_content, segment_params)
def tts_generate_with_params(text_content: str) -> torch.Tensor:
"""Wrapper for pause processor that passes segment parameters"""
# For pause-based processing, use first segment's parameters
segment_params = character_segment_objects[0].parameters if character_segment_objects and character_segment_objects[0].parameters else None
character_name = character_segment_objects[0].character if character_segment_objects else None
emotion_reference = character_segment_objects[0].emotion if character_segment_objects else None
return tts_generate_func(text_content, segment_params, character_name, emotion_reference)
# Generate audio with pauses
if segments:
@@ -547,9 +591,11 @@ class IndexTTSProcessor:
sample_rate=self.sample_rate
)
else:
# No pause tags, generate directly
segment_params = character_segment_objects[0].parameters if character_segment_objects and character_segment_objects[0].parameters else None
result = tts_generate_func(text, segment_params)
# No pause tags, generate directly
segment_params = character_segment_objects[0].parameters if character_segment_objects and character_segment_objects[0].parameters else None
character_name = character_segment_objects[0].character if character_segment_objects else None
emotion_reference = character_segment_objects[0].emotion if character_segment_objects else None
result = tts_generate_func(text, segment_params, character_name, emotion_reference)
# Ensure correct tensor format
if isinstance(result, torch.Tensor):
+15 -11
View File
@@ -40,11 +40,13 @@ class TextEncoder256(nn.Module):
)
self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
def forward(self, phone, pitch, lengths):
if pitch == None:
x = self.emb_phone(phone)
else:
x = self.emb_phone(phone) + self.emb_pitch(pitch)
def forward(self, phone, pitch, lengths):
phone = phone.to(dtype=self.emb_phone.weight.dtype)
if pitch is None:
x = self.emb_phone(phone)
else:
pitch = pitch.to(dtype=torch.long)
x = self.emb_phone(phone) + self.emb_pitch(pitch)
x = x * math.sqrt(self.hidden_channels) # [b, t, h]
x = self.lrelu(x)
x = torch.transpose(x, 1, -1) # [b, h, t]
@@ -87,11 +89,13 @@ class TextEncoder768(nn.Module):
)
self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
def forward(self, phone, pitch, lengths):
if pitch is None:
x = self.emb_phone(phone)
else:
x = self.emb_phone(phone) + self.emb_pitch(pitch)
def forward(self, phone, pitch, lengths):
phone = phone.to(dtype=self.emb_phone.weight.dtype)
if pitch is None:
x = self.emb_phone(phone)
else:
pitch = pitch.to(dtype=torch.long)
x = self.emb_phone(phone) + self.emb_pitch(pitch)
x = x * math.sqrt(self.hidden_channels) # [b, t, h]
x = self.lrelu(x)
x = torch.transpose(x, 1, -1) # [b, h, t]
@@ -1403,4 +1407,4 @@ class SynthesizerTrn(nn.Module):
z_p = self.flow(z, y_mask, g=g_src)
z_hat = self.flow(z_p, y_mask, g=g_tgt, reverse=True)
o_hat = self.dec(z_hat * y_mask, g=g_tgt)
return o_hat, y_mask, (z, z_p, z_hat)
return o_hat, y_mask, (z, z_p, z_hat)
+2 -1
View File
@@ -215,7 +215,8 @@ class VC(FeatureExtractor):
f0_max=f0_max,
)
p_len = min(pitch.shape[0], pitchf.shape[0])
pitch = pitch[:p_len].astype(np.int64 if self.device != 'mps' else np.float32)
# Pitch values are categorical embedding indices, including on MPS.
pitch = pitch[:p_len].astype(np.int64)
pitchf = pitchf[:p_len].astype(np.float32)
pitch = torch.from_numpy(pitch).to(self.device).unsqueeze(0)
pitchf = torch.from_numpy(pitchf).to(self.device).unsqueeze(0)
@@ -719,6 +719,8 @@ class VibeVoiceEngine:
# Extract waveform from ComfyUI audio format
waveform = voice_ref["waveform"]
input_sample_rate = voice_ref.get("sample_rate", 24000)
shape = tuple(waveform.shape) if hasattr(waveform, "shape") else "unknown"
print(f"🎵 VibeVoice ENGINE: Using waveform input for speaker {i+1} - shape: {shape}, sr: {input_sample_rate}")
# Convert to numpy
if isinstance(waveform, torch.Tensor):
Binary file not shown.

After

Width:  |  Height:  |  Size: 181 KiB

@@ -0,0 +1,918 @@
{
"id": "d7d0e47d-2b68-4f88-9a5e-f15a2d3c8b91",
"revision": 0,
"last_node_id": 24,
"last_link_id": 123,
"nodes": [
{
"id": 2,
"type": "CharacterVoicesNode",
"pos": [
-2660,
350
],
"size": [
240,
186
],
"flags": {},
"order": 2,
"mode": 0,
"inputs": [
{
"name": "opt_audio_input",
"shape": 7,
"type": "AUDIO",
"link": null
}
],
"outputs": [
{
"name": "opt_narrator",
"type": "NARRATOR_VOICE",
"links": [
101
]
},
{
"name": "character_name",
"type": "STRING",
"links": null
}
],
"properties": {
"cnr_id": "tts_audio_suite",
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
"Node name for S&R": "CharacterVoicesNode"
},
"widgets_values": [
"voices_examples/female/female_01.wav",
"",
""
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 3,
"type": "CharacterVoicesNode",
"pos": [
-2660,
580
],
"size": [
240,
186
],
"flags": {},
"order": 3,
"mode": 0,
"inputs": [
{
"name": "opt_audio_input",
"shape": 7,
"type": "AUDIO",
"link": null
}
],
"outputs": [
{
"name": "opt_narrator",
"type": "NARRATOR_VOICE",
"links": [
102
]
},
{
"name": "character_name",
"type": "STRING",
"links": null
}
],
"properties": {
"cnr_id": "tts_audio_suite",
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
"Node name for S&R": "CharacterVoicesNode"
},
"widgets_values": [
"voices_examples/Clint_Eastwood CC3 (enhanced2).wav",
"",
""
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 4,
"type": "StringMultilineTagEditor",
"pos": [
-1560,
570
],
"size": [
900,
600
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "text",
"type": "STRING",
"links": [
105
]
}
],
"properties": {
"cnr_id": "tts_audio_suite",
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
"Node name for S&R": "StringMultilineTagEditor",
"ttsTagEditorInlineEngine": "step_audio_editx"
},
"widgets_values": [
"On Tuesdays, the pigeons held parliament.\n[Alice|seed:42] They debated the ethics of breadcrumbs and the metaphysics of flight.\n[Bob] One particularly philosophical pigeon named Thistle believed gravity was just a suggestion.\n[Cowboy] He once floated for three minutes straight, fueled by sheer conviction and a gust of rebellious wind."
]
},
{
"id": 5,
"type": "StringMultilineTagEditor",
"pos": [
-1570,
1570
],
"size": [
900,
600
],
"flags": {},
"order": 6,
"mode": 0,
"inputs": [],
"outputs": [
{
"name": "text",
"type": "STRING",
"links": [
106
]
}
],
"properties": {
"cnr_id": "tts_audio_suite",
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
"Node name for S&R": "StringMultilineTagEditor",
"ttsTagEditorInlineEngine": "step_audio_editx"
},
"widgets_values": [
"1\n00:00:01,000 --> 00:00:04,000\nOn Tuesdays, the pigeons held parliament.\n\n2\n00:00:04,500 --> 00:00:09,500\n[Alice] They debated the ethics of breadcrumbs and the metaphysics of flight.\n\n3\n00:00:10,000 --> 00:00:14,000\n[Bob] One particularly philosophical pigeon named Thistle believed gravity was just a suggestion.\n\n4\n00:00:14,500 --> 00:00:20,000\n[Cowboy] He once floated for three minutes straight, fueled by sheer conviction and a gust of rebellious wind."
]
},
{
"id": 6,
"type": "FishAudioS2EngineNode",
"pos": [
-2240,
360
],
"size": [
410,
474
],
"flags": {},
"order": 12,
"mode": 0,
"inputs": [
{
"label": "Speaker 2",
"name": "speaker2",
"shape": 7,
"type": "*",
"link": 101
},
{
"label": "Speaker 3",
"name": "speaker3",
"shape": 7,
"type": "*",
"link": 102
},
{
"label": "Speaker 4",
"name": "speaker4",
"shape": 7,
"type": "*",
"link": 111
},
{
"label": "Speaker 5",
"name": "speaker5",
"shape": 7,
"type": "*",
"link": 112
},
{
"name": "speaker6",
"type": "*",
"link": null
}
],
"outputs": [
{
"name": "TTS_engine",
"type": "TTS_ENGINE",
"links": [
122
]
}
],
"properties": {
"cnr_id": "chatterbox_srt_voice",
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
"Node name for S&R": "FishAudioS2EngineNode"
},
"widgets_values": [
"s2-pro",
"auto",
0.8,
0.8,
1.1,
200,
1024,
"8192",
true,
true,
"bfloat16",
true,
"none",
"Native Multi-Speaker",
"Auto Inline Tag"
],
"color": "#222",
"bgcolor": "#000"
},
{
"id": 7,
"type": "UnifiedTTSTextNode",
"pos": [
-570,
360
],
"size": [
420,
390
],
"flags": {},
"order": 14,
"mode": 0,
"inputs": [
{
"name": "TTS_engine",
"type": "TTS_ENGINE",
"link": 122
},
{
"name": "opt_narrator",
"shape": 7,
"type": "*",
"link": null
},
{
"name": "text",
"type": "STRING",
"widget": {
"name": "text"
},
"link": 105
}
],
"outputs": [
{
"name": "audio",
"type": "AUDIO",
"links": [
109
]
},
{
"name": "generation_info",
"type": "STRING",
"links": null
}
],
"properties": {
"cnr_id": "tts_audio_suite",
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
"Node name for S&R": "UnifiedTTSTextNode"
},
"widgets_values": [
"",
"voices_examples/David_Attenborough CC3.wav",
1552347960,
"fixed",
true,
400,
"auto",
100,
true,
0
],
"color": "#223",
"bgcolor": "#335"
},
{
"id": 8,
"type": "UnifiedTTSSRTNode",
"pos": [
-570,
1490
],
"size": [
440,
560
],
"flags": {},
"order": 15,
"mode": 0,
"inputs": [
{
"name": "TTS_engine",
"type": "TTS_ENGINE",
"link": 116
},
{
"name": "opt_narrator",
"shape": 7,
"type": "*",
"link": null
},
{
"name": "srt_content",
"type": "STRING",
"widget": {
"name": "srt_content"
},
"link": 106
}
],
"outputs": [
{
"name": "audio",
"type": "AUDIO",
"links": [
110
]
},
{
"name": "generation_info",
"type": "STRING",
"links": null
},
{
"name": "timing_report",
"type": "STRING",
"links": null
},
{
"name": "Adjusted_SRT",
"type": "STRING",
"links": null
}
],
"properties": {
"cnr_id": "chatterbox_srt_voice",
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
"Node name for S&R": "UnifiedTTSSRTNode"
},
"widgets_values": [
"",
"voices_examples/David_Attenborough CC3.wav",
2286528432,
"fixed",
"pad_with_silence",
true,
0.01,
1,
0.5,
2,
0,
false
],
"color": "#223",
"bgcolor": "#335"
},
{
"id": 9,
"type": "PreviewAudio",
"pos": [
20,
430
],
"size": [
440,
88
],
"flags": {},
"order": 16,
"mode": 0,
"inputs": [
{
"name": "audio",
"type": "AUDIO",
"link": 109
}
],
"outputs": [],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.39",
"Node name for S&R": "PreviewAudio"
},
"widgets_values": []
},
{
"id": 10,
"type": "PreviewAudio",
"pos": [
20,
1730
],
"size": [
440,
88
],
"flags": {},
"order": 17,
"mode": 0,
"inputs": [
{
"name": "audio",
"type": "AUDIO",
"link": 110
}
],
"outputs": [],
"properties": {
"cnr_id": "comfy-core",
"ver": "0.3.39",
"Node name for S&R": "PreviewAudio"
},
"widgets_values": []
},
{
"id": 12,
"type": "CharacterVoicesNode",
"pos": [
-2660,
810
],
"size": [
240,
186
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [
{
"name": "opt_audio_input",
"shape": 7,
"type": "AUDIO",
"link": null
}
],
"outputs": [
{
"name": "opt_narrator",
"type": "NARRATOR_VOICE",
"links": [
111
]
},
{
"name": "character_name",
"type": "STRING",
"links": null
}
],
"properties": {
"cnr_id": "tts_audio_suite",
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
"Node name for S&R": "CharacterVoicesNode"
},
"widgets_values": [
"voices_examples/female/female_02.wav",
"",
""
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 13,
"type": "CharacterVoicesNode",
"pos": [
-2660,
1040
],
"size": [
240,
186
],
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{
"name": "opt_audio_input",
"shape": 7,
"type": "AUDIO",
"link": null
}
],
"outputs": [
{
"name": "opt_narrator",
"type": "NARRATOR_VOICE",
"links": [
112
]
},
{
"name": "character_name",
"type": "STRING",
"links": null
}
],
"properties": {
"cnr_id": "tts_audio_suite",
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
"Node name for S&R": "CharacterVoicesNode"
},
"widgets_values": [
"voices_examples/higgs_audio/belinda.wav",
"",
""
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 14,
"type": "MarkdownNote",
"pos": [
-610,
870
],
"size": [
460,
260
],
"flags": {},
"order": 1,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {},
"widgets_values": [
"# Fish Audio S2 Inline Instructions\n\nFish supports free-form inline instructions for emotion, prosody, accents, pauses, and speaking style.\n\nThe suite keeps its public instruction syntax as `<...>` and automatically converts it to Fish's native `[...]` syntax at the engine boundary.\n\n- Write Fish instructions with `<...>`.\n- Use `[Character]` for suite character switching.\n- Use `[fr:Alice]` to select Alice and request French.\n- `Auto Inline Tag` converts mapped character languages automatically.\n- Disable `Auto Inline Tag` when Fish should infer the language from the text.\n\nDo not manually replace suite instructions with Fish's square-bracket syntax.\n\nFor the complete suite syntax, see the [Character Switching Guide](https://github.com/diodiogod/TTS-Audio-Suite/blob/main/docs/CHARACTER_SWITCHING_GUIDE.md) and [Tag Editor Guide](https://github.com/diodiogod/TTS-Audio-Suite/blob/main/docs/MULTILINE_TTS_TAG_EDITOR_GUIDE.md). For Fish's prompting behavior, see the [Fish S2 model card](https://huggingface.co/fishaudio/s2-pro)."
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 15,
"type": "MarkdownNote",
"pos": [
-2160,
950
],
"size": [
350,
280
],
"flags": {},
"order": 9,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {},
"widgets_values": [
"# Fish Audio S2 Compilation\n\nCompilation is recommended for repeated Fish generation.\n\nThe first generation after loading the model takes longer because PyTorch compiles the inference path. Later generations reuse the compiled runtime and are significantly faster.\n\nIn our testing:\n\n- Non-compiled generation: approximately `6 it/s`\n- Compiled generation: approximately `40 it/s`\n- Observed speedup: roughly **6-7x faster**\n\nThe initial compilation delay is worth accepting when generating multiple segments, testing different seeds, or working with longer text.\n\n> Keep `Compile` enabled unless startup time is more important than generation speed."
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 16,
"type": "MarkdownNote",
"pos": [
-3550,
190
],
"size": [
750,
970
],
"flags": {},
"order": 0,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {},
"widgets_values": [
"# Fish Audio S2 Pro\n\n> Multilingual voice cloning with native multi-speaker dialogue, free-form inline control, and long-form generation.\n\nConnect the Fish Audio S2 Engine to the unified `TTS Text` or `TTS SRT` node.\n\n---\n\n## Voice Cloning\n\nFish requires both:\n\n- Reference audio\n- Matching reference text\n\nFor reliable cloning, use:\n\n- `🎭 Character Voices` with a matching `.reference.txt`\n- A narrator voice file with its matching `.reference.txt`\n\nDirect audio-only input does not provide the transcript required by Fish.\n\n---\n\n## Languages and Inline Control\n\nFish S2 supports **80+ languages** and has no separate language input.\n\nThe suite uses its normal `<...>` inline syntax and converts instructions automatically to Fish's native `[...]` syntax at the engine boundary.\n\n- Use `<...>` for Fish speech instructions.\n- Use `[Character]` for suite character switching.\n- Use `[fr:Alice]` to select Alice and request French.\n- `Auto Inline Tag` converts mapped character languages automatically.\n- Disable `Auto Inline Tag` when Fish should determine the language from the text.\n\nFish supports free-form instructions for expression, prosody, accents, pauses, and other speech behavior.\n\n---\n\n## Multi-Speaker Modes\n\n### Native Multi-Speaker\n\nSends all dialogue turns in one Fish request.\n\n- Might sound more natural\n\n### Custom Character Switching\n\nGenerates each parsed character segment independently.\n\n- Could be way more stable since it's independent character calls\n\n---\n\n## Long-Form Generation\n\nFish has its own native long-form orchestration through `native_chunk_length`.\n\nThis is Fish's internal grouping system and is separate from Custom Character Switching.\n\nFish does not provide native duration targeting.\nSRT timing is handled by the suite's shared timing and assembly system.\n\n---\n\n## Limitations\n\n- Fish does not support `🔄 Voice Changer`.\n- Voice cloning requires matching reference text.\n- The model does not provide native duration targeting.\n\n---\n\n## Documentation\n\n- [Character Switching Guide](https://github.com/diodiogod/TTS-Audio-Suite/blob/main/docs/CHARACTER_SWITCHING_GUIDE.md)\n- [Tag Editor Guide](https://github.com/diodiogod/TTS-Audio-Suite/blob/main/docs/MULTILINE_TTS_TAG_EDITOR_GUIDE.md)\n- [Fish S2 Model Card](https://huggingface.co/fishaudio/s2-pro)\n- [Official Fish Speech Repository](https://github.com/fishaudio/fish-speech)"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 19,
"type": "Reroute",
"pos": [
-1590,
1480
],
"size": [
75,
26
],
"flags": {},
"order": 13,
"mode": 0,
"inputs": [
{
"name": "",
"type": "*",
"link": 123
}
],
"outputs": [
{
"name": "",
"type": "TTS_ENGINE",
"links": [
116
]
}
],
"properties": {
"showOutputText": false,
"horizontal": false
}
},
{
"id": 22,
"type": "FishAudioS2EngineNode",
"pos": [
-2160,
1480
],
"size": [
410,
474
],
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "speaker2",
"type": "*",
"link": null
}
],
"outputs": [
{
"name": "TTS_engine",
"type": "TTS_ENGINE",
"links": [
123
]
}
],
"properties": {
"cnr_id": "chatterbox_srt_voice",
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
"Node name for S&R": "FishAudioS2EngineNode"
},
"widgets_values": [
"s2-pro",
"auto",
0.8,
0.8,
1.1,
200,
1024,
"8192",
true,
true,
"bfloat16",
true,
"none",
"Native Multi-Speaker",
"Auto Inline Tag"
],
"color": "#222",
"bgcolor": "#000"
},
{
"id": 23,
"type": "MarkdownNote",
"pos": [
-2660,
1320
],
"size": [
360,
310
],
"flags": {},
"order": 11,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {},
"widgets_values": [
"\n\n## 🔌 The Speaker Input Priority System\n\nWhen using **NATIVE Multi-Speaker**, you can connect specific voices to Speaker 1 (opt_narrator on the unified TTS node) , Speaker2, Speaker3... inputs on the Engine node and they will override tags!\n\n**What happens if using tags:**\n\n- `narrator` becomes Speaker 1 → uses Speaker 1 input voice\n- `[Alice]` becomes Speaker 2 → uses Speaker 2 input voice\n- `[Bob]` becomes Speaker 3 → uses Speaker 3 input voice\n- etc.\n\nSO If using \n\n## **[character]** \n\n tags in your text to switch narrators, you DON'T need to connect opt_narrator, Speaker2_voice, Speaker3_voice or speaker4_voice\n\nEither for Native or for Custom modes"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 24,
"type": "MarkdownNote",
"pos": [
-2520,
1780
],
"size": [
310,
90
],
"flags": {},
"order": 10,
"mode": 0,
"inputs": [],
"outputs": [],
"properties": {},
"widgets_values": [
"## Without speakers connected, it will use the Character Tag switching system normally"
],
"color": "#432",
"bgcolor": "#653"
}
],
"links": [
[
101,
2,
0,
6,
0,
"*"
],
[
102,
3,
0,
6,
1,
"*"
],
[
105,
4,
0,
7,
2,
"STRING"
],
[
106,
5,
0,
8,
2,
"STRING"
],
[
109,
7,
0,
9,
0,
"AUDIO"
],
[
110,
8,
0,
10,
0,
"AUDIO"
],
[
111,
12,
0,
6,
2,
"*"
],
[
112,
13,
0,
6,
3,
"*"
],
[
116,
19,
0,
8,
0,
"TTS_ENGINE"
],
[
122,
6,
0,
7,
0,
"TTS_ENGINE"
],
[
123,
22,
0,
19,
0,
"TTS_ENGINE"
]
],
"groups": [
{
"id": 1,
"title": "Fish Audio S2 Engine and Voices (they will OVERRIDE the characters on the text)",
"bounding": [
-2750,
170,
1090,
2050
],
"color": "#3f789e",
"font_size": 24,
"flags": {}
},
{
"id": 2,
"title": "TTS TEXT",
"bounding": [
-1620,
160,
2260,
1230
],
"color": "#8A8",
"font_size": 24,
"flags": {}
},
{
"id": 3,
"title": "SRT",
"bounding": [
-1610,
1430,
2250,
780
],
"color": "#a1309b",
"font_size": 24,
"flags": {}
}
],
"config": {},
"extra": {
"frontendVersion": "1.38.13",
"ue_links": [],
"links_added_by_ue": [],
"VHS_latentpreview": true,
"VHS_latentpreviewrate": 0,
"VHS_MetadataImage": true,
"VHS_KeepIntermediate": true,
"workflowRendererVersion": "LG",
"ds": {
"scale": 0.33591950066247567,
"offset": [
4389.719106665517,
640.6871718697881
]
}
},
"version": 0.4
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 55 KiB

File diff suppressed because it is too large Load Diff
+42 -4
View File
@@ -664,8 +664,10 @@ class TTSAudioInstaller:
# Install/reinstall audio packages that commonly have architecture issues on Mac
mac_audio_packages = [
"soundfile>=0.12.0",
"sounddevice>=0.4.0",
"soundfile>=0.12.0",
"sounddevice>=0.4.0",
"pydub",
"webdataset",
]
for package in mac_audio_packages:
@@ -1038,8 +1040,9 @@ class TTSAudioInstaller:
"librosa", # Forces numpy downgrade - compatibility handled by runtime numba disabling for Python 3.13
"descript-audio-codec", # Pulls unnecessary deps, conflicts with protobuf
"descript-audiotools", # Forces protobuf downgrade from 6.x to 3.19.x
"cached-path", # Forces package downgrades
"torchcrepe", # Conflicts via librosa dependency
"cached-path", # Forces package downgrades
"omnivoice", # Keep shared dependency resolution under our control
"torchcrepe", # Conflicts via librosa dependency
# NOTE: onnxruntime moved to install_onnxruntime_with_gpu_support() for smart GPU detection
"opencv-python", # Forces numpy downgrade from 2.x to 1.26.x - dependencies pre-installed
"gradio", # Forces pydantic, pillow, pydantic-core downgrades - dependencies pre-installed
@@ -1158,6 +1161,40 @@ class TTSAudioInstaller:
ignore_errors=True
)
def install_fish_audio_s2(self):
"""Install Fish S2 without changing the suite's core ML stack."""
self.log("Installing Fish Audio S2 support in the main T5 environment", "INFO")
packages = [
("lightning", "lightning"),
("pytorch_lightning", "pytorch-lightning"),
("lightning_utilities", "lightning-utilities"),
("torchmetrics", "torchmetrics"),
("opencc", "opencc-python-reimplemented==0.1.7"),
("ormsgpack", "ormsgpack"),
("zstandard", "zstandard"),
("pyrootutils", "pyrootutils"),
("loralib", "loralib"),
]
for module_name, package_name in packages:
if self.verify_python_import(module_name):
self.log(f"{module_name} import already satisfied - skipping", "SUCCESS")
continue
self.run_pip_command(
["install", package_name, "--no-deps"],
f"Installing Fish S2 dependency {package_name} (--no-deps)",
ignore_errors=True,
)
if self.check_package_installed("fish-speech"):
self.log("fish-speech already satisfied - skipping", "SUCCESS")
return
self.run_pip_command(
["install", "git+https://github.com/fishaudio/fish-speech.git", "--no-deps"],
"Installing official fish-speech from GitHub (--no-deps)",
ignore_errors=True,
)
def install_f5tts_multilingual_support(self):
"""Install phonemization support for F5-TTS multilingual models (Polish, German, French, Spanish, etc.)"""
self.log("Installing F5-TTS multilingual phonemization support", "INFO")
@@ -1595,6 +1632,7 @@ def main():
installer.install_vibevoice() # Install VibeVoice with careful dependency management
installer.install_echo_tts() # Install Echo-TTS with minimal dependency impact
installer.install_dots_tts() # Install official Dots TTS in the main environment first
installer.install_fish_audio_s2() # Install Fish S2 without changing Torch/Transformers
installer.install_f5tts_multilingual_support() # Install phonemization for Polish/multilingual F5-TTS
installer.install_indexts_text_processing() # Install IndexTTS-2 text normalization with fallback
installer.install_russian_text_stresser_support() # Install lightweight Russian stress package for Official 23-Lang
+37 -1
View File
@@ -11,7 +11,7 @@ except ImportError:
pass
# Version and constants
VERSION = "5.1.1"
VERSION = "5.4.4"
IS_DEV = False # Set to False for release builds
VERSION_DISPLAY = f"v{VERSION}" + (" (dev)" if IS_DEV else "")
SEPARATOR = "=" * 70
@@ -169,6 +169,30 @@ except Exception as e:
print(f"❌ Dots TTS Engine failed: {e}")
DOTS_TTS_ENGINE_AVAILABLE = False
try:
fish_audio_s2_engine_module = load_node_module("fish_audio_s2_engine_node", "engines/fish_audio_s2_engine_node.py")
FishAudioS2EngineNode = fish_audio_s2_engine_module.FishAudioS2EngineNode
FISH_AUDIO_S2_ENGINE_AVAILABLE = True
except Exception as e:
print(f"❌ Fish Audio S2 Pro Engine failed: {e}")
FISH_AUDIO_S2_ENGINE_AVAILABLE = False
try:
omnivoice_engine_module = load_node_module("omnivoice_engine_node", "engines/omnivoice_engine_node.py")
OmniVoiceEngineNode = omnivoice_engine_module.OmniVoiceEngineNode
OMNIVOICE_ENGINE_AVAILABLE = True
except Exception as e:
print(f"❌ OmniVoice Engine failed: {e}")
OMNIVOICE_ENGINE_AVAILABLE = False
try:
omnivoice_instruction_builder_module = load_node_module("omnivoice_instruction_builder_node", "omnivoice/omnivoice_instruction_builder_node.py")
OmniVoiceInstructionBuilderNode = omnivoice_instruction_builder_module.OmniVoiceInstructionBuilderNode
OMNIVOICE_INSTRUCTION_BUILDER_AVAILABLE = True
except Exception as e:
print(f"❌ Visual Tag Builder failed: {e}")
OMNIVOICE_INSTRUCTION_BUILDER_AVAILABLE = False
try:
chatterbox_official_23lang_engine_module = load_node_module("chatterbox_official_23lang_engine_node", "engines/chatterbox_official_23lang_engine_node.py")
ChatterBoxOfficial23LangEngineNode = chatterbox_official_23lang_engine_module.ChatterBoxOfficial23LangEngineNode
@@ -609,6 +633,18 @@ if DOTS_TTS_ENGINE_AVAILABLE:
NODE_CLASS_MAPPINGS["DotsTTSEngineNode"] = DotsTTSEngineNode
NODE_DISPLAY_NAME_MAPPINGS["DotsTTSEngineNode"] = "⚙️ Dots TTS Engine"
if FISH_AUDIO_S2_ENGINE_AVAILABLE:
NODE_CLASS_MAPPINGS["FishAudioS2EngineNode"] = FishAudioS2EngineNode
NODE_DISPLAY_NAME_MAPPINGS["FishAudioS2EngineNode"] = "⚙️ Fish Audio S2 Pro Engine"
if OMNIVOICE_ENGINE_AVAILABLE:
NODE_CLASS_MAPPINGS["OmniVoiceEngineNode"] = OmniVoiceEngineNode
NODE_DISPLAY_NAME_MAPPINGS["OmniVoiceEngineNode"] = "⚙️ OmniVoice Engine"
if OMNIVOICE_INSTRUCTION_BUILDER_AVAILABLE:
NODE_CLASS_MAPPINGS["OmniVoiceInstructionBuilderNode"] = OmniVoiceInstructionBuilderNode
NODE_DISPLAY_NAME_MAPPINGS["OmniVoiceInstructionBuilderNode"] = "📐 Visual Tag Builder"
if QWEN3_TTS_VOICE_DESIGNER_AVAILABLE:
NODE_CLASS_MAPPINGS["Qwen3TTSVoiceDesignerNode"] = Qwen3TTSVoiceDesignerNode
NODE_DISPLAY_NAME_MAPPINGS["Qwen3TTSVoiceDesignerNode"] = "🎨 Qwen3-TTS Voice Designer"
+6 -2
View File
@@ -348,7 +348,7 @@ Selects the audio format for separated stems:
import torch
import torch.nn.functional as F
from tqdm import tqdm
import librosa
from utils.audio.librosa_fallback import safe_resample
from comfy.utils import load_torch_file, ProgressBar
from comfy import model_management as mm
@@ -398,7 +398,11 @@ Selects the audio format for separated stems:
# Resample to 44100 if needed
if sample_rate != 44100:
print(f"Resampling from {sample_rate} to 44100 Hz")
audio_waveform = librosa.resample(audio_waveform, orig_sr=sample_rate, target_sr=44100, axis=-1)
audio_waveform = safe_resample(
audio_waveform,
orig_sr=sample_rate,
target_sr=44100,
)
sample_rate = 44100
# Convert to torch tensor
+7 -4
View File
@@ -39,6 +39,7 @@ BaseTTSNode = base_module.BaseTTSNode
from utils.text.chunking import ImprovedChatterBoxChunker
from utils.audio.processing import AudioProcessingUtils
from utils.voice.discovery import get_available_characters, get_character_mapping
from utils.voice.character_logging import resolved_character_label
from utils.text.pause_processor import PauseTagProcessor
from utils.text.character_parser import parse_character_text, character_parser
from utils.text.segment_parameters import apply_segment_parameters
@@ -1174,7 +1175,8 @@ Back to the main narrator voice for the conclusion.""",
# Sequential processing for all chunks
segment_audio_chunks = []
for chunk_i, chunk_text in enumerate(segment_chunks):
print(f"🎤 Generating ChatterBox segment {segment_display_idx}/{total_segments} chunk {chunk_i+1}/{len(segment_chunks)} for '{character}' (lang: {lang_code})...")
display_name = resolved_character_label(character, char_audio_prompt)
print(f"🎤 Generating ChatterBox segment {segment_display_idx}/{total_segments} chunk {chunk_i+1}/{len(segment_chunks)} for '{display_name}' (lang: {lang_code})...")
# Generate audio with caching support for character segments
chunk_audio = self._generate_tts_with_pause_tags(
@@ -1379,9 +1381,11 @@ Back to the main narrator voice for the conclusion.""",
audio_segments_with_order.append((original_idx, segment_audio))
# Use the original better format with proper emoji based on character type
if char == "narrator":
print(f"🎤 Generating ChatterBox segment {original_idx+1}/{len(segments_with_lang)} for '{char}' (lang: {lang})")
display_name = resolved_character_label(char, char_audio_prompt)
print(f"🎤 Generating ChatterBox segment {original_idx+1}/{len(segments_with_lang)} for '{display_name}' (lang: {lang})")
else:
print(f"🎭 Generating ChatterBox segment {original_idx+1}/{len(segments_with_lang)} for '{char}' (lang: {lang})")
display_name = resolved_character_label(char, char_audio_prompt)
print(f"🎭 Generating ChatterBox segment {original_idx+1}/{len(segments_with_lang)} for '{display_name}' (lang: {lang})")
return audio_segments_with_order
@@ -1484,4 +1488,3 @@ Back to the main narrator voice for the conclusion.""",
)
print(f"🚀 Pre-loading complete: {len(self._streaming_model_manager.preloaded_models)} models ready")
@@ -40,6 +40,7 @@ BaseTTSNode = base_module.BaseTTSNode
from utils.text.chunking import ImprovedChatterBoxChunker
from utils.audio.processing import AudioProcessingUtils
from utils.voice.discovery import get_available_characters, get_character_mapping
from utils.voice.character_logging import resolved_character_label
from utils.text.pause_processor import PauseTagProcessor
from utils.text.character_parser import parse_character_text, character_parser
from utils.text.segment_parameters import apply_segment_parameters
@@ -1562,7 +1563,8 @@ Back to the main narrator voice for the conclusion.""",
# Sequential processing for all chunks
segment_audio_chunks = []
for chunk_i, chunk_text in enumerate(segment_chunks):
print(f"🎤 Generating ChatterBox segment {segment_display_idx}/{total_segments} chunk {chunk_i+1}/{len(segment_chunks)} for '{character}' (lang: {lang_code})...")
display_name = resolved_character_label(character, char_audio_prompt)
print(f"🎤 Generating ChatterBox segment {segment_display_idx}/{total_segments} chunk {chunk_i+1}/{len(segment_chunks)} for '{display_name}' (lang: {lang_code})...")
# Generate audio with caching support for character segments
chunk_audio = self._generate_tts_with_pause_tags(
@@ -1823,10 +1825,11 @@ Back to the main narrator voice for the conclusion.""",
audio_segments_with_order.append((idx, segment_audio))
# Use the original better format with proper emoji based on character type
display_name = resolved_character_label(char, char_audio_prompt)
if char == "narrator":
print(f"🎤 Generating ChatterBox segment {idx+1}/{len(expanded_segments_with_lang)} for '{char}' (lang: {lang})")
print(f"🎤 Generating ChatterBox segment {idx+1}/{len(expanded_segments_with_lang)} for '{display_name}' (lang: {lang})")
else:
print(f"🎭 Generating ChatterBox segment {idx+1}/{len(expanded_segments_with_lang)} for '{char}' (lang: {lang})")
print(f"🎭 Generating ChatterBox segment {idx+1}/{len(expanded_segments_with_lang)} for '{display_name}' (lang: {lang})")
# Batch process edit tags if any
if all_segments_for_editing:
@@ -1960,4 +1963,3 @@ Back to the main narrator voice for the conclusion.""",
)
print(f"🚀 Pre-loading complete: {len(self._streaming_model_manager.preloaded_models)} models ready")
@@ -23,7 +23,8 @@ if project_root not in sys.path:
from utils.text.segment_parameters import apply_segment_parameters
from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment, parse_edit_tags_with_iterations
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
from utils.text.chatterbox_v2_special_tags import CHATTERBOX_V2_SPECIAL_TOKENS
from utils.text.chatterbox_v2_special_tags import CHATTERBOX_V2_SPECIAL_TOKENS
from utils.voice.character_logging import resolved_character_label
def extract_edit_tags_for_chatterbox(text: str):
@@ -412,7 +413,11 @@ class ChatterboxOfficial23LangSRTProcessor:
print(f"🌍 ChatterBox Official 23-Lang SRT Segment {i+1} (Seq {subtitle.sequence}): Language switching - {', '.join(languages)}")
elif subtitle_type != 'multisegment':
characters = list(set(char for char, _, _, _ in character_segments_with_lang))
print(f"🎭 ChatterBox Official 23-Lang SRT Segment {i+1} (Seq {subtitle.sequence}): Character switching - {', '.join(characters)}")
resolved_characters = [
resolved_character_label(char, voice_refs.get(char, voice_refs.get('narrator')))
for char in characters
]
print(f"🎭 ChatterBox Official 23-Lang SRT Segment {i+1} (Seq {subtitle.sequence}): Character switching - {', '.join(resolved_characters)}")
segment_audio_parts = []
subtitle_has_edit_tags = False # Track if ANY segment has edit tags
+8 -5
View File
@@ -22,6 +22,10 @@ from utils.audio.edit_post_processor import process_segments as apply_edit_post_
from engines.dots_tts.languages import format_dots_language_display, normalize_dots_language
from utils.text.character_parser import character_parser
from utils.text.pause_processor import PauseTagProcessor
from utils.voice.character_logging import (
format_resolved_character_block,
resolved_character_label,
)
from utils.text.segment_parameters import ParameterValidator, apply_segment_parameters
from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment
from utils.voice.discovery import get_available_characters, get_character_mapping, voice_discovery
@@ -63,16 +67,15 @@ class DotsTTSProcessor:
show_text_content: bool = True,
) -> None:
voice_note = self._voice_log_note(voice_ref)
print(f"🎭 Dots TTS - Generating for '{character_name}' (Language: {language}){voice_note}:")
display_name = resolved_character_label(character_name, voice_ref)
print(f"🎭 Dots TTS - Generating for '{display_name}' (Language: {language}){voice_note}:")
if parameter_log:
print(f"🎛️ Dots TTS params: {parameter_log}")
if show_text_content:
print("=" * 60)
print(text_content)
print("=" * 60)
print(format_resolved_character_block(character_name, text_content, voice_ref))
if chunk_count > 1:
print(
f"📝 Chunking {character_name}'s text into {chunk_count} chunks "
f"📝 Chunking {display_name}'s text into {chunk_count} chunks "
f"(Language: {language}){voice_note}"
)
+7 -4
View File
@@ -24,6 +24,10 @@ from utils.audio.edit_post_processor import process_segments as apply_edit_post_
from utils.voice.discovery import get_available_characters, voice_discovery, get_character_mapping
from utils.text.character_parser import character_parser
from utils.audio.chunk_timing import ChunkTimingHelper
from utils.voice.character_logging import (
format_resolved_character_block,
resolved_character_label,
)
class EchoTTSProcessor:
@@ -233,10 +237,9 @@ class EchoTTSProcessor:
print(f"📝 Echo-TTS: Chunking '{seg.character}' into {len(chunks)} chunks (max {max_chars_per_chunk} chars each)")
for chunk_idx, chunk in enumerate(chunks):
print(f"🎤 Echo-TTS - Generating for '{seg.character}'" + (f" (chunk {chunk_idx + 1}/{len(chunks)})" if len(chunks) > 1 else "") + ":")
print("=" * 60)
print(chunk)
print("=" * 60)
display_name = resolved_character_label(char_name, speaker_audio)
print(f"🎤 Echo-TTS - Generating for '{display_name}'" + (f" (chunk {chunk_idx + 1}/{len(chunks)})" if len(chunks) > 1 else "") + ":")
print(format_resolved_character_block(char_name, chunk, speaker_audio))
audio = _generate_single(chunk).cpu()
if audio.dim() == 1:
audio = audio.unsqueeze(0)
+214
View File
@@ -0,0 +1,214 @@
"""Fish Audio S2 Pro engine configuration node."""
import importlib.util
import json
import os
import re
import sys
current_dir = os.path.dirname(__file__)
nodes_dir = os.path.dirname(current_dir)
project_root = os.path.dirname(nodes_dir)
if project_root not in sys.path:
sys.path.insert(0, project_root)
base_spec = importlib.util.spec_from_file_location("base_node_module", os.path.join(nodes_dir, "base", "base_node.py"))
base_module = importlib.util.module_from_spec(base_spec)
sys.modules.setdefault("base_node_module", base_module)
base_spec.loader.exec_module(base_module)
SPEAKER_INPUT_PATTERN = re.compile(r"^speaker([2-9]\d*)$")
class DynamicSpeakerInputs(dict):
@staticmethod
def _is_speaker_key(key):
return isinstance(key, str) and SPEAKER_INPUT_PATTERN.fullmatch(key) is not None
def __contains__(self, key):
return super().__contains__(key) or self._is_speaker_key(key)
def __getitem__(self, key):
if super().__contains__(key):
return super().__getitem__(key)
if self._is_speaker_key(key):
return ("*", {
"forceInput": True,
"tooltip": "Positional Fish speaker reference. Connect a Character Voices output with exact reference transcript.",
})
raise KeyError(key)
def get(self, key, default=None):
return self[key] if key in self else default
class FishAudioS2EngineNode(base_module.BaseTTSNode):
@classmethod
def NAME(cls):
return "⚙️ Fish Audio S2 Pro Engine"
@classmethod
def _get_model_options(cls):
"""List local checkpoints first, followed by downloadable variants."""
downloadable = ["s2-pro", "s2-pro-fp8"]
try:
from engines.fish_audio_s2.downloader import FishAudioS2Downloader
local = [
f"local:{os.path.basename(os.path.normpath(path))}"
for path in FishAudioS2Downloader.local_model_paths()
]
except Exception:
local = []
return local + [option for option in downloadable if option not in local]
@classmethod
def INPUT_TYPES(cls):
optional = DynamicSpeakerInputs({
"normalize": ("BOOLEAN", {"default": True, "tooltip": "Official English/Chinese number normalization."}),
"cache_reference": ("BOOLEAN", {"default": True, "tooltip": "Cache encoded reference codes inside the Fish runtime."}),
"precision": (["bfloat16", "float16"], {"default": "bfloat16"}),
"compile": ("BOOLEAN", {"default": True, "tooltip": "Enable the official torch.compile path (recommended). The first generation is slower, but later runs are much faster; testing showed about 6 it/s to 41 it/s, roughly a 6.8x speedup. This can also be tested with BNB quantization, although bitsandbytes compatibility may vary by version."}),
"quantization": (["none", "bnb_int8", "bnb_nf4"], {
"default": "none",
"tooltip": (
"Optional on-the-fly quantization for the official s2-pro checkpoint.\n\n"
"• none: Full checkpoint loading\n"
"• bnb_int8: BitsAndBytes INT8 load-time quantization\n"
"• bnb_nf4: BitsAndBytes NF4 load-time quantization\n\n"
"BNB quantization reuses the official checkpoint and requires CUDA plus bitsandbytes. "
"It is ignored for the separate s2-pro-fp8 checkpoint."
),
}),
"multi_speaker_mode": (["Native Multi-Speaker", "Custom Character Switching"], {
"default": "Native Multi-Speaker",
"tooltip": (
"Native Multi-Speaker sends all character turns in one Fish dialogue request. "
"This preserves native multi-turn context and long-form behavior.\n\n"
"Custom Character Switching generates every parsed character segment independently "
"as local speaker 0, using only that character's reference. This can reduce speaker "
"leakage but loses cross-turn context and requires more generation calls."
),
}),
"language_prompting": (["Auto Inline Tag", "Off"], {
"default": "Auto Inline Tag",
"tooltip": (
"Fish has no native language parameter, so the suite can optionally prepend a natural-language "
"inline instruction for resolved segment languages.\n\n"
"• Auto Inline Tag: alias/default languages like de or fr become Fish prompt tags such as <German>\n"
"• Off: keep Fish language fully text-only and ignore alias/default language switching"
),
}),
"speaker2": ("*", {
"forceInput": True,
"tooltip": "Optional second speaker reference. Speaker 1 is always the Unified node narrator/opt_narrator input. Additional speaker inputs appear automatically.",
}),
})
return {"required": {
"model": (cls._get_model_options(), {
"default": "s2-pro",
"tooltip": "Fish Audio S2 model selection. Options prefixed with local: are already present in configured ComfyUI TTS model folders; unprefixed options are downloadable checkpoints.",
}),
"device": (["auto", "cuda", "cpu"], {"default": "auto"}),
"temperature": ("FLOAT", {"default": 0.8, "min": 0.1, "max": 1.0, "step": 0.05}),
"top_p": ("FLOAT", {"default": 0.8, "min": 0.1, "max": 1.0, "step": 0.05}),
"repetition_penalty": ("FLOAT", {"default": 1.1, "min": 0.9, "max": 2.0, "step": 0.05}),
"native_chunk_length": ("INT", {"default": 200, "min": 100, "max": 1000, "step": 25,
"tooltip": "Official UTF-8 byte limit for grouping native speaker turns."}),
"max_new_tokens": ("INT", {"default": 1024, "min": 64, "max": 4096, "step": 64}),
"context_length": (["4096", "8192", "16384", "32768"], {
"default": "8192",
"tooltip": "Native context and KV-cache size. Larger values support longer dialogue but use substantially more VRAM.",
}),
}, "optional": optional}
RETURN_TYPES = ("TTS_ENGINE",)
RETURN_NAMES = ("TTS_engine",)
FUNCTION = "create_engine_config"
CATEGORY = "TTS Audio Suite/⚙️ Engines"
@classmethod
def IS_CHANGED(cls, model="s2-pro", device="auto", temperature=0.8, top_p=0.8, repetition_penalty=1.1,
native_chunk_length=200, max_new_tokens=1024, context_length="8192",
normalize=True, cache_reference=True, precision="bfloat16",
compile=True, model_variant=None, quantization="none",
multi_speaker_mode="Native Multi-Speaker", language_prompting="Auto Inline Tag",
speaker2=None, **kwargs):
model_selection = model_variant if model_variant is not None else model
speaker_keys = sorted(
key for key in kwargs.keys()
if SPEAKER_INPUT_PATTERN.fullmatch(key) is not None
)
fingerprint = {
"device": device,
"temperature": float(temperature),
"top_p": float(top_p),
"repetition_penalty": float(repetition_penalty),
"native_chunk_length": int(native_chunk_length),
"max_new_tokens": int(max_new_tokens),
"context_length": str(context_length),
"normalize": bool(normalize),
"cache_reference": bool(cache_reference),
"precision": precision,
"compile": bool(compile),
"model": model_selection,
"quantization": quantization,
"multi_speaker_mode": multi_speaker_mode,
"language_prompting": language_prompting,
"speaker2_connected": speaker2 is not None,
"dynamic_speakers_connected": {
key: kwargs.get(key) is not None for key in speaker_keys
},
}
return json.dumps(fingerprint, sort_keys=True)
def create_engine_config(self, model="s2-pro", device="auto", temperature=0.8, top_p=0.8, repetition_penalty=1.1,
native_chunk_length=200, max_new_tokens=1024, context_length="8192",
normalize=True, cache_reference=True, precision="bfloat16",
compile=True, model_variant=None, quantization="none",
multi_speaker_mode="Native Multi-Speaker", language_prompting="Auto Inline Tag",
speaker2=None, **kwargs):
model_selection = model_variant if model_variant is not None else model
runtime_variant = model_selection
if str(model_selection).startswith("local:"):
try:
from engines.fish_audio_s2.downloader import FishAudioS2Downloader
local_path = FishAudioS2Downloader.resolve_local_model_path(model_selection)
runtime_variant = FishAudioS2Downloader.resolve_model_variant(model_selection, local_path)
except Exception:
runtime_variant = "s2-pro"
speakers = []
if speaker2 is not None:
speakers.append((2, speaker2))
for key, value in kwargs.items():
match = SPEAKER_INPUT_PATTERN.fullmatch(key)
if match and value is not None:
speakers.append((int(match.group(1)), value))
speakers = [value for _, value in sorted(speakers, key=lambda item: item[0])]
if runtime_variant != "s2-pro":
precision = "bfloat16"
quantization = "none"
print(
" Settings: "
f"model={model_selection}, quantization={quantization}, "
f"multi_speaker_mode={multi_speaker_mode}, language_prompting={language_prompting}, "
f"temperature={float(temperature)}, "
f"top_p={float(top_p)}, repetition_penalty={float(repetition_penalty)}, "
f"native_chunk_length={int(native_chunk_length)}, max_new_tokens={int(max_new_tokens)}, "
f"context_length={int(context_length)}, precision={precision}, "
f"normalize={bool(normalize)}, cache_reference={bool(cache_reference)}, compile={bool(compile)}"
)
config = {
"engine_type": "fish_audio_s2", "model_variant": model_selection, "device": device,
"temperature": float(temperature), "top_p": float(top_p),
"repetition_penalty": float(repetition_penalty),
"native_chunk_length": int(native_chunk_length), "max_new_tokens": int(max_new_tokens),
"context_length": int(context_length),
"normalize": bool(normalize), "cache_reference": bool(cache_reference),
"precision": precision, "compile": bool(compile),
"quantization": quantization,
"multi_speaker_mode": multi_speaker_mode,
"language_prompting": language_prompting,
"speaker_references": speakers, "runtime_mode": "main_subprocess",
}
return ({"engine_type": "fish_audio_s2", "config": config, "capabilities": ["tts"]},)
+69 -46
View File
@@ -145,14 +145,19 @@ class IndexTTSEngineNode(BaseTTSNode):
}),
},
"optional": {
# Unified Emotion Control - Using multitype input for better connection suggestions
"emotion_control": (any_typ, {
"tooltip": """• 🌈 Emotion Vectors - Manual emotion control sliders
• 🎭 Character Voices (opt_narrator) - Audio-based emotion reference
• 🌈 Text Emotion - AI-analyzed emotion from text
• Direct AUDIO - Any audio input for emotion reference
Character emotion tags [Alice:emotion_ref] will override this for specific characters."""
}),
# Unified Emotion Control - Using multitype input for better connection suggestions
"emotion_control": (any_typ, {
"tooltip": """Vector/text emotion control (legacy unified input):
• 🌈 Emotion Vectors - Manual emotion control sliders
• 🌈 Text Emotion - AI-analyzed emotion from text
Connect audio separately to 'emotion_audio' when you want to blend both.
Character emotion tags [Alice:emotion_ref] can provide per-segment audio emotion."""
}),
"emotion_audio": (any_typ, {
"tooltip": """Dedicated emotion-reference audio input.
Connect Character Voices, opt_narrator, or AUDIO here.
This can be connected together with the vector/text emotion input above; IndexTTS-2 blends both."""
}),
# CUDA Kernel Option
"use_cuda_kernel": (["auto", "true", "false"], {
@@ -241,9 +246,9 @@ Character emotion tags [Alice:emotion_ref] will override this for specific chara
top_k: int,
do_sample: bool,
length_penalty: float,
num_beams: int,
repetition_penalty: float,
max_mel_tokens: int,
num_beams: int,
repetition_penalty: float,
max_mel_tokens: int,
use_fp16: bool,
use_deepspeed: bool,
use_cuda_kernel: str = "auto",
@@ -253,6 +258,7 @@ Character emotion tags [Alice:emotion_ref] will override this for specific chara
stream_return: bool = False,
more_segment_before: int = 0,
low_vram: bool = False,
emotion_audio = None,
):
"""
Create IndexTTS-2 engine adapter with configuration.
@@ -261,21 +267,38 @@ Character emotion tags [Alice:emotion_ref] will override this for specific chara
Tuple containing IndexTTS-2 engine configuration data
"""
try:
# Process unified emotion control
emotion_audio = None
emotion_vector = None
use_emotion_text = False
emotion_text = None
is_dynamic_template = False
if emotion_control:
if isinstance(emotion_control, dict):
# Check the type of emotion control
emotion_type = emotion_control.get("type")
if emotion_type == "emotion_vectors":
# Emotion vectors from options node
emotion_vectors = emotion_control.get("emotion_vectors", {})
# Process vector/text control independently from dedicated audio.
connected_emotion_audio = emotion_audio
emotion_audio = None
emotion_vector = None
use_emotion_text = False
emotion_text = None
is_dynamic_template = False
def is_audio_control(value):
return isinstance(value, dict) and any(
key in value for key in ("audio_path", "waveform", "audio")
)
# Keep old workflows working when audio was connected to the
# original unified socket. The new dedicated socket wins when
# both are connected.
legacy_audio_control = emotion_control if is_audio_control(emotion_control) else None
vector_text_control = None if legacy_audio_control is not None else emotion_control
emotion_audio = (
connected_emotion_audio
if connected_emotion_audio is not None
else legacy_audio_control
)
if vector_text_control:
if isinstance(vector_text_control, dict):
# Check the type of emotion control
emotion_type = vector_text_control.get("type")
if emotion_type == "emotion_vectors":
# Emotion vectors from options node
emotion_vectors = vector_text_control.get("emotion_vectors", {})
emotions = [
emotion_vectors.get("happy", 0.0),
emotion_vectors.get("angry", 0.0),
@@ -289,19 +312,11 @@ Character emotion tags [Alice:emotion_ref] will override this for specific chara
if any(e > 0.0 for e in emotions):
emotion_vector = emotions
elif emotion_type == "qwen_emotion":
# QwenEmotion text analysis
use_emotion_text = emotion_control.get("use_emotion_text", False)
emotion_text = emotion_control.get("emotion_text", "")
is_dynamic_template = emotion_control.get("is_dynamic_template", False)
elif "waveform" in emotion_control or "audio" in emotion_control:
# Direct audio input (NARRATOR_VOICE from Character Voices or AUDIO)
emotion_audio = emotion_control
elif hasattr(emotion_control, 'get') and ("waveform" in emotion_control or "audio" in emotion_control):
# Direct audio input
emotion_audio = emotion_control
elif emotion_type == "qwen_emotion":
# QwenEmotion text analysis
use_emotion_text = vector_text_control.get("use_emotion_text", False)
emotion_text = vector_text_control.get("emotion_text", "")
is_dynamic_template = vector_text_control.get("is_dynamic_template", False)
# Parse CUDA kernel option
cuda_kernel_option = None
@@ -321,7 +336,7 @@ Character emotion tags [Alice:emotion_ref] will override this for specific chara
config = {
"model_path": model_path,
"device": device,
"emotion_audio": emotion_audio, # Will be None if not connected, audio dict if connected
"emotion_audio": emotion_audio, # Dedicated audio emotion input
"emotion_alpha": emotion_alpha,
"use_emotion_text": use_emotion_text,
"emotion_text": emotion_text if emotion_text and emotion_text.strip() else None,
@@ -350,12 +365,20 @@ Character emotion tags [Alice:emotion_ref] will override this for specific chara
"low_vram": low_vram,
}
print(f"⚙️ IndexTTS-2: Configured on {device}")
print(f" Model: {model_path}")
emotion_desc = f"alpha={emotion_alpha}, use_text={use_emotion_text}"
if is_dynamic_template:
emotion_desc += " (dynamic template)"
print(f" Emotion: {emotion_desc}")
print(f"⚙️ IndexTTS-2: Configured on {device}")
print(f" Model: {model_path}")
emotion_desc = f"alpha={emotion_alpha}, use_text={use_emotion_text}"
if is_dynamic_template:
emotion_desc += " (dynamic template)"
emotion_sources = []
if emotion_audio is not None:
emotion_sources.append("audio")
if emotion_vector is not None:
emotion_sources.append("vector")
if use_emotion_text:
emotion_sources.append("text")
emotion_desc += f", sources={'+'.join(emotion_sources) if emotion_sources else 'none'}"
print(f" Emotion: {emotion_desc}")
print(f" Generation: temp={temperature}, top_p={top_p}, top_k={top_k}, do_sample={do_sample}, num_beams={num_beams}")
print(f" Chunking: max_tokens={max_text_tokens_per_segment}, silence={interval_silence}ms")
# Report optimization and streaming parameters
+189
View File
@@ -0,0 +1,189 @@
"""
OmniVoice engine configuration node.
"""
import importlib.util
import os
import sys
current_dir = os.path.dirname(__file__)
nodes_dir = os.path.dirname(current_dir)
project_root = os.path.dirname(nodes_dir)
if project_root not in sys.path:
sys.path.insert(0, project_root)
base_node_path = os.path.join(nodes_dir, "base", "base_node.py")
base_spec = importlib.util.spec_from_file_location("base_node_module", base_node_path)
base_module = importlib.util.module_from_spec(base_spec)
sys.modules["base_node_module"] = base_module
base_spec.loader.exec_module(base_module)
BaseTTSNode = base_module.BaseTTSNode
class OmniVoiceEngineNode(BaseTTSNode):
"""OmniVoice engine configuration node."""
@classmethod
def NAME(cls):
return "⚙️ OmniVoice Engine"
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"model_variant": (["OmniVoice"], {
"default": "OmniVoice",
"tooltip": "OmniVoice multilingual zero-shot TTS model from k2-fsa/OmniVoice. Voice cloning requires reference audio plus reference text. In this suite, use Character Voices or a narrator voice file with matching .reference.txt. Direct audio-only input is not supported for OmniVoice cloning."
}),
"device": (["auto", "cuda", "cpu", "mps", "xpu"], {
"default": "auto",
"tooltip": "Device to run OmniVoice on. Auto follows the best available backend."
}),
"language": ("STRING", {
"default": "Auto",
"multiline": False,
"tooltip": "Target language. Use 'Auto' for language-agnostic generation, or enter a language name/code such as 'English', 'Chinese', 'Japanese', 'en', 'zh', 'ja'. OmniVoice supports 600+ languages."
}),
"num_step": ("INT", {
"default": 32, "min": 4, "max": 128, "step": 1,
"tooltip": "Iterative unmasking steps. Recommended: 32. Use 16 for faster generation."
}),
"guidance_scale": ("FLOAT", {
"default": 2.0, "min": 0.0, "max": 10.0, "step": 0.1,
"tooltip": "Classifier-free guidance scale. Recommended: 2.0."
}),
"t_shift": ("FLOAT", {
"default": 0.1, "min": 0.0, "max": 1.0, "step": 0.01,
"tooltip": "Time-step shift for the OmniVoice noise schedule. Recommended: 0.1."
}),
"speed": ("FLOAT", {
"default": 1.0, "min": 0.25, "max": 3.0, "step": 0.05,
"tooltip": "Speech speed factor. Values above 1 are faster; values below 1 are slower. Recommended: 1.0. Technically, this rescales OmniVoice's estimated target audio-token length before generation. It is not waveform time-stretch post-processing."
}),
"duration": ("FLOAT", {
"default": 0.0, "min": 0.0, "max": 600.0, "step": 0.1,
"tooltip": "Fixed output duration in seconds. Set 0 to let OmniVoice estimate duration from text. Technically, this overrides speed and sets the planned target audio-token length from seconds, so it behaves more like native target-length control than a simple max-tokens cap."
}),
},
"optional": {
"dtype": (["auto", "bfloat16", "float16", "float32"], {
"default": "auto",
"tooltip": "Runtime precision. Auto prefers bf16 on newer CUDA/XPU hardware, else fp16, with fp32 on CPU."
}),
"instruct": ("STRING", {
"default": "",
"multiline": True,
"tooltip": "Optional OmniVoice voice-design / style instruction. Strong effect when no narrator/reference is provided, because the model uses it to design the voice directly. With narrator/reference cloning, it still acts as guidance, but the effect is usually much weaker because OmniVoice prioritizes the provided voice identity."
}),
"layer_penalty_factor": ("FLOAT", {
"default": 5.0, "min": 0.0, "max": 10.0, "step": 0.1,
"tooltip": "Penalty encouraging earlier codebook layers to unmask first. Recommended: 5.0."
}),
"position_temperature": ("FLOAT", {
"default": 5.0, "min": 0.0, "max": 10.0, "step": 0.1,
"tooltip": "Mask-position temperature. 0 is greedy; higher values add randomness. Recommended: 5.0."
}),
"class_temperature": ("FLOAT", {
"default": 0.0, "min": 0.0, "max": 2.0, "step": 0.05,
"tooltip": "Token sampling temperature. 0 is greedy; higher values are more random. Recommended: 0.0."
}),
"denoise": ("BOOLEAN", {
"default": True,
"tooltip": "Enable the denoise token. Recommended: on."
}),
"preprocess_prompt": ("BOOLEAN", {
"default": True,
"tooltip": "Preprocess the voice-cloning reference by trimming long reference audio, removing silence, and adding terminal punctuation to the reference text. Recommended: on."
}),
"postprocess_output": ("BOOLEAN", {
"default": True,
"tooltip": "Post-process generated audio with silence cleanup plus fade and edge padding. Recommended: on."
}),
"audio_chunk_duration": ("FLOAT", {
"default": 15.0, "min": 1.0, "max": 60.0, "step": 0.5,
"tooltip": "Native long-form target chunk duration in seconds. This is OmniVoice's real long-text control and supersedes suite char-based chunking. Recommended: 15.0."
}),
"audio_chunk_threshold": ("FLOAT", {
"default": 30.0, "min": 1.0, "max": 180.0, "step": 0.5,
"tooltip": "Native long-form activation threshold in seconds. Estimated outputs longer than this use OmniVoice's built-in chunking path. Recommended: 30.0."
}),
},
}
RETURN_TYPES = ("TTS_ENGINE",)
RETURN_NAMES = ("TTS_engine",)
FUNCTION = "create_engine_config"
CATEGORY = "TTS Audio Suite/⚙️ Engines"
def create_engine_config(
self,
model_variant: str,
device: str,
language: str,
num_step: int,
guidance_scale: float,
t_shift: float,
speed: float,
duration: float,
dtype: str = "auto",
instruct: str = "",
layer_penalty_factor: float = 5.0,
position_temperature: float = 5.0,
class_temperature: float = 0.0,
denoise: bool = True,
preprocess_prompt: bool = True,
postprocess_output: bool = True,
audio_chunk_duration: float = 15.0,
audio_chunk_threshold: float = 30.0,
) -> tuple:
config = {
"engine_type": "omnivoice",
"model_variant": model_variant,
"device": device,
"dtype": dtype,
"language": str(language or "Auto").strip() or "Auto",
"instruct": str(instruct or "").strip(),
"num_steps": int(num_step),
"guidance_scale": float(guidance_scale),
"t_shift": float(t_shift),
"speed": float(speed),
"duration": float(duration),
"layer_penalty_factor": float(layer_penalty_factor),
"position_temperature": float(position_temperature),
"class_temperature": float(class_temperature),
"denoise": bool(denoise),
"preprocess_prompt": bool(preprocess_prompt),
"postprocess_output": bool(postprocess_output),
"audio_chunk_duration": float(audio_chunk_duration),
"audio_chunk_threshold": float(audio_chunk_threshold),
}
print(f"⚙️ OmniVoice: Configured on {device}")
print(
f" Model: {model_variant} | Language: {config['language']} | Dtype: {dtype}"
)
print(
" Settings: steps={steps}, guidance_scale={guidance}, t_shift={t_shift}, speed={speed}, duration={duration}".format(
steps=num_step,
guidance=guidance_scale,
t_shift=t_shift,
speed=speed,
duration="auto" if duration <= 0 else duration,
)
)
print(
" Native long-form: chunk_duration={chunk_duration}s, threshold={threshold}s".format(
chunk_duration=audio_chunk_duration,
threshold=audio_chunk_threshold,
)
)
print(" Voice cloning requires reference text. Use Character Voices or a .reference.txt narrator voice.")
if instruct:
print(f" Instruct: {instruct[:80]}..." if len(instruct) > 80 else f" Instruct: {instruct}")
engine_data = {
"engine_type": "omnivoice",
"config": config,
"adapter_class": "OmniVoiceEngineAdapter",
}
return (engine_data,)
+5 -3
View File
@@ -33,7 +33,8 @@ BaseF5TTSNode = f5tts_base_module.BaseF5TTSNode
from utils.text.chunking import ImprovedChatterBoxChunker
from utils.audio.processing import AudioProcessingUtils
from utils.voice.discovery import get_available_voices, load_voice_reference, get_available_characters, get_character_mapping
from utils.voice.discovery import get_available_voices, load_voice_reference, get_available_characters, get_character_mapping
from utils.voice.character_logging import resolved_character_label
from utils.text.character_parser import parse_character_text, character_parser
from utils.text.step_audio_editx_special_tags import parse_edit_tags_with_iterations
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
@@ -607,7 +608,8 @@ Back to the main narrator voice for the conclusion.""",
# Generate audio for each chunk of this character segment
segment_audio_chunks = []
for chunk_i, chunk_text in enumerate(segment_chunks):
print(f"🎤 Generating F5-TTS segment {segment_display_idx}/{total_segments} chunk {chunk_i+1}/{len(segment_chunks)} for '{character}' (lang: {lang_code})...")
display_name = resolved_character_label(character, (char_audio, char_text))
print(f"🎤 Generating F5-TTS segment {segment_display_idx}/{total_segments} chunk {chunk_i+1}/{len(segment_chunks)} for '{display_name}' (lang: {lang_code})...")
# Create cache function for this character if caching is enabled (uses per-segment parameters)
cache_fn = None
@@ -1196,4 +1198,4 @@ Back to the main narrator voice for the conclusion.""",
print(f"❌ F5-TTS streaming segment {original_idx} failed: {e}")
# Return silence as fallback
return torch.zeros((1, 24000)) # 1 second of silence at 24kHz
return self.process_with_error_handling(_process)
return self.process_with_error_handling(_process)
+15 -18
View File
@@ -34,7 +34,8 @@ BaseF5TTSNode = f5tts_base_module.BaseF5TTSNode
from utils.system.import_manager import import_manager
from utils.audio.processing import AudioProcessingUtils
from utils.voice.discovery import get_available_voices, load_voice_reference, get_available_characters, get_character_mapping
from utils.voice.discovery import get_available_voices, load_voice_reference, get_available_characters, get_character_mapping
from utils.voice.character_logging import resolved_character_label
from utils.text.character_parser import parse_character_text, character_parser
from utils.text.step_audio_editx_special_tags import parse_edit_tags_with_iterations
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
@@ -537,19 +538,7 @@ Hello! This is F5-TTS SRT with character switching.
# Check for interruption
self.check_interruption(f"F5-TTS character segment {subtitle_idx+1}.{seg_idx+1} (Seq {subtitle.sequence})")
# Show generation message
if character == "narrator":
if language != 'en':
print(f"🎤 Generating F5-TTS segment {subtitle_idx+1}.{seg_idx+1} (Seq {subtitle.sequence}) in {language}...")
else:
print(f"🎤 Generating F5-TTS segment {subtitle_idx+1}.{seg_idx+1} (Seq {subtitle.sequence})...")
else:
if language != 'en':
print(f"🎭 Generating F5-TTS segment {subtitle_idx+1}.{seg_idx+1} using '{character}' in {language}")
else:
print(f"🎭 Generating F5-TTS segment {subtitle_idx+1}.{seg_idx+1} using '{character}'")
# PRIORITY FIX: For "narrator" character, always use main reference when provided
# PRIORITY FIX: For "narrator" character, always use main reference when provided
# This ensures Character Voices node and dropdown selection work correctly
if character == "narrator" and audio_prompt and validated_ref_text:
# Use main reference for narrator (from Character Voices or dropdown)
@@ -567,8 +556,16 @@ Hello! This is F5-TTS SRT with character switching.
char_audio = audio_prompt
char_text = validated_ref_text
print(f"🔄 F5-TTS: Using main reference fallback for '{character}'")
else:
print(f"🎭 F5-TTS: Using character-specific voice for '{character}'")
else:
print(f"🎭 F5-TTS: Using character-specific voice for '{character}'")
display_name = resolved_character_label(character, (char_audio, char_text))
if character == "narrator":
suffix = f" in {language}" if language != 'en' else ""
print(f"🎤 Generating F5-TTS segment {subtitle_idx+1}.{seg_idx+1} (Seq {subtitle.sequence}) for '{display_name}'{suffix}...")
else:
suffix = f" in {language}" if language != 'en' else ""
print(f"🎭 Generating F5-TTS segment {subtitle_idx+1}.{seg_idx+1} using '{display_name}'{suffix}")
# Validate and clamp nfe_step to prevent ODE solver issues
safe_nfe_step = max(1, min(nfe_step, 71))
@@ -1040,11 +1037,11 @@ Hello! This is F5-TTS SRT with character switching.
# Set phonemization setting for consistency (even when using cache)
import os
os.environ['F5TTS_AUTO_PHONEMIZATION'] = str(auto_phonemization).lower()
print(f"💾 F5-TTS SRT streaming: Cache hit for {character} segment {original_idx}")
print(f"💾 F5-TTS SRT streaming: Cache hit for {resolved_character_label(character, ref_audio_path)} segment {original_idx}")
return cached_audio[0]
# Generate audio for this segment
print(f"🔄 F5-TTS SRT streaming: Generating {character} segment {original_idx} in {language}")
print(f"🔄 F5-TTS SRT streaming: Generating {resolved_character_label(character, ref_audio_path)} segment {original_idx} in {language}")
# Generate with F5-TTS (using base class method)
wav = self.generate_f5tts_with_pause_tags(
+1
View File
@@ -0,0 +1 @@
"""Fish Audio S2 processors."""
@@ -0,0 +1,339 @@
"""Fish Audio S2 Pro text orchestration."""
import re
from typing import Any, Dict, List
import torch
import comfy.model_management as model_management
from utils.audio.chunk_timing import ChunkTimingHelper
from utils.text.character_parser import character_parser
from utils.text.fish_audio_s2_tags import get_fish_language_instruction
from utils.text.pause_processor import PauseTagProcessor
from utils.text.segment_parameters import ParameterValidator, apply_segment_parameters
from utils.voice.discovery import (
get_available_characters,
get_character_default_language,
get_character_mapping,
voice_discovery,
)
from utils.voice.character_logging import (
format_character_override_warning,
resolved_character_label,
resolved_voice_name,
)
class FishAudioS2Processor:
SAMPLE_RATE = 44100
def __init__(self, adapter, engine_config: Dict[str, Any]):
self.adapter = adapter
self.config = dict(engine_config or {})
def update_config(self, config):
self.config = dict(config or {})
self.adapter.update_config(self.config)
def _setup_character_parser(self, text):
tags = [tag.split("|")[0].strip() for tag in re.findall(r"\[([^\]]+)\]", text or "") if not tag.lower().startswith(("pause:", "wait:", "stop:"))]
available = set(get_available_characters() or [])
aliases = voice_discovery.get_character_aliases()
available.update(aliases.keys())
available.update(aliases.values())
available.update(tag.lower() for tag in tags)
available.add("narrator")
character_parser.set_available_characters(list(available))
for character, language in voice_discovery.get_character_language_defaults().items():
character_parser.set_character_language_default(character, language)
character_parser.reset_session_cache()
def get_character_order(self, text):
self._setup_character_parser(text)
order = []
for segment in character_parser.parse_text_segments(text, engine_type="fish_audio_s2"):
character = segment.character or "narrator"
if character not in order:
order.append(character)
if "narrator" in order:
order = ["narrator"] + [character for character in order if character != "narrator"]
return order
def _apply_language_instruction(self, text: str, segment) -> str:
if self.config.get("language_prompting", "Auto Inline Tag") != "Auto Inline Tag":
return text
instruction = get_fish_language_instruction(
getattr(segment, "language", None),
explicit=bool(getattr(segment, "explicit_language", False)),
)
if not instruction:
return text
stripped = (text or "").lstrip()
if stripped.startswith("<"):
return text
return f"<{instruction}> {text}"
def _get_effective_language_instruction(self, segment, override_voice_ref: Dict[str, Any] | None) -> str | None:
explicit_language = bool(getattr(segment, "explicit_language", False))
if explicit_language:
return get_fish_language_instruction(
getattr(segment, "language", None),
explicit=True,
)
if isinstance(override_voice_ref, dict):
override_character = override_voice_ref.get("character_name")
if override_character:
override_language = get_character_default_language(str(override_character))
return get_fish_language_instruction(override_language, explicit=False)
return None
return get_fish_language_instruction(
getattr(segment, "language", None),
explicit=False,
)
def _apply_effective_language_instruction(
self, text: str, segment, override_voice_ref: Dict[str, Any] | None
) -> tuple[str, str | None]:
if self.config.get("language_prompting", "Auto Inline Tag") != "Auto Inline Tag":
return text, None
instruction = self._get_effective_language_instruction(segment, override_voice_ref)
if not instruction:
return text, None
stripped = (text or "").lstrip()
if stripped.startswith("<"):
return text, instruction
return f"<{instruction}> {text}", instruction
@staticmethod
def _voice_for(character, voice_mapping, discovered, narrator):
if character in voice_mapping and isinstance(voice_mapping[character], dict):
return dict(voice_mapping[character])
audio_path, ref_text = discovered.get(character, (None, None))
if audio_path:
return {"audio_path": audio_path, "reference_text": ref_text or ""}
return dict(narrator)
def process_text(self, text: str, voice_mapping: Dict[str, Any], seed: int,
enable_chunking: bool = True, max_chars_per_chunk: int = 400,
chunk_combination_method: str = "auto", silence_between_chunks_ms: int = 100,
enable_audio_cache: bool = True, apply_edit_postprocessing: bool = True,
show_text_logging: bool = True, reference_order=None) -> List[Dict[str, Any]]:
del enable_chunking, max_chars_per_chunk, chunk_combination_method
del silence_between_chunks_ms, apply_edit_postprocessing
self._setup_character_parser(text)
segments = character_parser.parse_text_segments(text, engine_type="fish_audio_s2")
characters = []
for segment in segments:
character = segment.character or "narrator"
if character not in characters:
characters.append(character)
reference_characters = list(reference_order or characters)
narrator_exists_globally = "narrator" in reference_characters
if narrator_exists_globally:
reference_characters = ["narrator"] + [character for character in reference_characters if character != "narrator"]
global_speaker_number = {
character: index + 1
for index, character in enumerate(reference_characters)
}
discovered = get_character_mapping(characters, engine_type="audio_only")
narrator_value = voice_mapping.get("narrator", {})
narrator = dict(narrator_value) if isinstance(narrator_value, dict) else {"audio": narrator_value}
configured_refs = list(self.config.get("speaker_references") or [])
speaker_map = {character: index for index, character in enumerate(characters)}
voice_refs = []
narrator_has_reference = bool(narrator_value)
for character in characters:
effective_speaker_number = global_speaker_number.get(character, speaker_map[character] + 1)
if narrator_has_reference and narrator_exists_globally and character == "narrator":
voice_refs.append(dict(narrator))
continue
if narrator_has_reference and not narrator_exists_globally and effective_speaker_number == 1:
voice_refs.append(dict(narrator))
continue
configured_index = effective_speaker_number - 2
if 0 <= configured_index < len(configured_refs):
voice_refs.append(configured_refs[configured_index])
continue
voice_refs.append(self._voice_for(character, voice_mapping, discovered, narrator))
voice_ref_by_character = dict(zip(characters, voice_refs))
custom_switching = self.config.get(
"multi_speaker_mode", "Native Multi-Speaker"
) == "Custom Character Switching"
records = []
pending_turns = []
pending_logs = []
pending_config = None
group_index = 0
speaker_labels = {index: character for character, index in speaker_map.items()}
connected_speakers = []
if narrator_has_reference:
connected_speakers.append("Speaker 1 (Narrator)")
connected_speakers.extend(
f"Speaker {idx + 2}"
for idx, ref in enumerate(configured_refs)
if ref is not None
)
logged_priority_overrides = set()
def generation_key(config):
return tuple(config.get(name) for name in (
"temperature", "top_p", "repetition_penalty", "native_chunk_length",
"max_new_tokens", "normalize", "cache_reference",
))
def print_custom_segment_preview(log_label: str, turn_text: str):
if not show_text_logging:
return
print("🎭 Fish Audio S2 custom segment:")
print("============================================================")
print(f"[{log_label}] {turn_text}")
print("============================================================")
def flush_turns():
nonlocal pending_turns, pending_logs, group_index
if not pending_turns:
return
self.adapter.update_config(pending_config)
if show_text_logging:
print(f"🎭 Fish Audio S2 native dialogue: Processing {len(pending_turns)} segment(s)")
if connected_speakers:
print(f"🎤 Fish speaker inputs connected: {connected_speakers}")
mapping_display = ", ".join(
f"{character}->local Speaker {speaker_idx + 1} "
f"(global Speaker {global_speaker_number.get(character, speaker_idx + 1)})"
for character, speaker_idx in sorted(speaker_map.items(), key=lambda item: item[1])
)
if mapping_display:
print(f"🎭 Fish character mapping: {mapping_display}")
print("============================================================")
for log_label, turn_text in pending_logs:
print(f"[{log_label}] {turn_text}")
print("============================================================")
audio = self.adapter.generate_dialogue(
turns=pending_turns, voice_refs=voice_refs,
seed=int(pending_config.get("seed", seed)) + group_index,
enable_audio_cache=enable_audio_cache,
)
combined_text = "\n".join(turn_text for _, turn_text in pending_turns)
records.append({
"waveform": audio.cpu(), "sample_rate": self.SAMPLE_RATE,
"text": combined_text, "edit_tags": [],
})
pending_turns = []
pending_logs = []
group_index += 1
for segment in segments:
if model_management.interrupt_processing:
raise InterruptedError("Fish Audio S2 text generation interrupted")
segment_text = (segment.text or "").strip()
if not segment_text:
continue
params = segment.parameters or {}
filtered = ParameterValidator.filter_parameters_for_engine(params, "fish_audio_s2")
config = apply_segment_parameters(self.config, params, "fish_audio_s2") if filtered else dict(self.config)
character = segment.character or "narrator"
original_character = getattr(segment, "original_character", None) or character
speaker_idx = speaker_map[character]
effective_speaker_number = global_speaker_number.get(character, speaker_idx + 1)
configured_index = effective_speaker_number - 2
override_voice_ref = None
if (
narrator_has_reference
and (
(narrator_exists_globally and character == "narrator")
or (not narrator_exists_globally and effective_speaker_number == 1)
)
):
discovered_ref = discovered.get(character, (None, None))[0]
override_voice_ref = narrator
if discovered_ref and show_text_logging:
override_key = (speaker_idx, character)
if override_key not in logged_priority_overrides:
print(format_character_override_warning(
"Fish", "Speaker 1 (Narrator)", original_character
))
logged_priority_overrides.add(override_key)
elif 0 <= configured_index < len(configured_refs):
configured_ref = configured_refs[configured_index]
override_voice_ref = configured_ref
discovered_ref = discovered.get(character, (None, None))[0]
if configured_ref is not None and discovered_ref:
override_key = (speaker_idx, character)
if override_key not in logged_priority_overrides and show_text_logging:
print(format_character_override_warning(
"Fish", f"Speaker {effective_speaker_number}", original_character
))
logged_priority_overrides.add(override_key)
if pending_config is not None and generation_key(config) != generation_key(pending_config):
flush_turns()
pending_config = config
pause_parts, _ = PauseTagProcessor.parse_pause_tags(segment_text)
if not pause_parts:
pause_parts = [("text", segment_text)]
for part_type, content in pause_parts:
if model_management.interrupt_processing:
raise InterruptedError("Fish Audio S2 text generation interrupted")
if part_type == "pause":
flush_turns()
records.append({
"waveform": PauseTagProcessor.create_silence_segment(content, self.SAMPLE_RATE),
"sample_rate": self.SAMPLE_RATE, "text": f"[pause:{content}s]", "edit_tags": [],
})
continue
turn_text = str(content).strip()
if not turn_text:
continue
turn_text, language_instruction = self._apply_effective_language_instruction(
turn_text, segment, override_voice_ref
)
if show_text_logging and self.config.get("language_prompting", "Auto Inline Tag") == "Auto Inline Tag":
if language_instruction:
print(f" 🌍 Fish Audio S2 prompting language via <{language_instruction}>")
if custom_switching:
flush_turns()
log_label = character
if narrator_has_reference and character == "narrator":
log_label = resolved_voice_name(narrator, "Speaker 1 (Narrator)")
elif override_voice_ref is not None:
log_label = resolved_character_label(character, override_voice_ref)
print_custom_segment_preview(log_label, turn_text)
self.adapter.update_config(config)
audio = self.adapter.generate_dialogue(
turns=[(0, turn_text)],
voice_refs=[voice_ref_by_character[character]],
seed=int(config.get("seed", seed)) + group_index,
enable_audio_cache=enable_audio_cache,
cache_character=character,
)
records.append({
"waveform": audio.cpu(), "sample_rate": self.SAMPLE_RATE,
"text": turn_text, "edit_tags": [],
})
group_index += 1
else:
pending_turns.append((speaker_map[character], turn_text))
log_label = resolved_character_label(
character, voice_ref_by_character.get(character)
)
pending_logs.append((log_label, turn_text))
flush_turns()
return records
def combine_audio_segments(self, segments, method="auto", silence_ms=100,
original_text="", return_info=False):
if not segments:
empty = torch.zeros(0)
return (empty, {}) if return_info else empty
return ChunkTimingHelper.combine_audio_with_timing(
audio_segments=[item["waveform"] for item in segments],
combination_method=method, silence_ms=silence_ms, crossfade_duration=0.1,
sample_rate=self.SAMPLE_RATE, text_length=len(original_text), original_text=original_text,
text_chunks=[item.get("text", "") for item in segments],
) if return_info else ChunkTimingHelper.combine_audio_with_timing(
audio_segments=[item["waveform"] for item in segments],
combination_method=method, silence_ms=silence_ms, crossfade_duration=0.1,
sample_rate=self.SAMPLE_RATE, text_length=len(original_text), original_text=original_text,
text_chunks=[item.get("text", "") for item in segments],
)[0]
@@ -0,0 +1,142 @@
"""Fish Audio S2 Pro SRT orchestration."""
import importlib.util
import os
import comfy.model_management as model_management
import torch
from engines.adapters.fish_audio_s2_adapter import FishAudioS2Adapter
from utils.system.import_manager import import_manager
from utils.timing.assembly import AudioAssemblyEngine
from utils.timing.engine import TimingEngine
from utils.timing.overlap_detection import SRTOverlapHandler
from utils.timing.reporting import SRTReportGenerator
class FishAudioS2SRTProcessor:
SAMPLE_RATE = 44100
def __init__(self, node_instance, config):
self.node_instance = node_instance
self.config = dict(config or {})
self.adapter = FishAudioS2Adapter(self.config)
self._processor = None
success, modules, message = import_manager.import_srt_modules()
if not success:
raise ImportError(f"Fish Audio S2 SRT unavailable: {message}")
self.SRTParser = modules["SRTParser"]
@property
def processor(self):
if self._processor is None:
path = os.path.join(os.path.dirname(__file__), "fish_audio_s2_processor.py")
spec = importlib.util.spec_from_file_location("fish_audio_s2_processor_module", path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
self._processor = module.FishAudioS2Processor(self.adapter, self.config)
return self._processor
def update_config(self, config):
self.config = dict(config or {})
self.adapter.update_config(self.config)
if self._processor is not None:
self._processor.update_config(self.config)
@staticmethod
def _check_interrupt(index=None, total=None):
if model_management.interrupt_processing:
where = f" at subtitle {index + 1}/{total}" if index is not None else ""
raise InterruptedError(f"Fish Audio S2 SRT generation interrupted{where}")
def process_srt_content(self, srt_content, voice_mapping, seed, timing_mode,
timing_params, enable_audio_cache=True):
self._check_interrupt()
subtitles = self.SRTParser().parse_srt_content(srt_content, allow_overlaps=True)
has_overlaps = SRTOverlapHandler.detect_overlaps(subtitles)
active_mode, switched = SRTOverlapHandler.handle_smart_natural_fallback(
timing_mode, has_overlaps, "Fish Audio S2 SRT"
)
reference_order = self.processor.get_character_order(
"\n".join((subtitle.text or "") for subtitle in subtitles)
)
audio_segments = []
adjustments = []
for index, subtitle in enumerate(subtitles):
self._check_interrupt(index, len(subtitles))
text = (subtitle.text or "").strip()
if text:
records = self.processor.process_text(
text=text, voice_mapping=voice_mapping, seed=seed + index,
enable_chunking=False, enable_audio_cache=enable_audio_cache,
apply_edit_postprocessing=False,
reference_order=reference_order,
)
audio, _ = self.processor.combine_audio_segments(
records, method="auto", silence_ms=0, original_text=text, return_info=True
)
if audio.dim() == 1:
audio = audio.unsqueeze(0)
else:
audio = torch.zeros(1, int(subtitle.duration * self.SAMPLE_RATE))
audio = audio.cpu()
audio_segments.append(audio)
natural = audio.shape[-1] / self.SAMPLE_RATE
target = subtitle.duration
ratio = target / natural if natural else 1.0
adjustments.append({
"index": index, "segment_index": index, "sequence": subtitle.sequence,
"natural_duration": natural, "target_start": subtitle.start_time,
"target_end": subtitle.end_time, "target_duration": target,
"start_time": subtitle.start_time, "end_time": subtitle.end_time,
"stretch_factor": ratio, "needs_stretching": abs(ratio - 1.0) > 0.05,
"stretch_type": "compress" if ratio < 1 else "expand" if ratio > 1 else "none",
"adjustment": natural - target, "adjusted_start": subtitle.start_time,
"adjusted_end": subtitle.end_time, "adjusted_duration": natural,
})
self._check_interrupt()
final_audio, replacement, stretch_method = self._assemble(
audio_segments, subtitles, active_mode, timing_params
)
if replacement is not None:
adjustments = replacement
reporter = SRTReportGenerator()
report = reporter.generate_timing_report(
subtitles, adjustments, active_mode, has_overlaps, switched,
timing_mode if switched else None, stretch_method,
)
adjusted_srt = reporter.generate_adjusted_srt_string(subtitles, adjustments, active_mode)
if final_audio.dim() == 1:
final_audio = final_audio.unsqueeze(0).unsqueeze(0)
elif final_audio.dim() == 2:
final_audio = final_audio.unsqueeze(0)
duration = final_audio.shape[-1] / self.SAMPLE_RATE
info = f"Generated {duration:.1f}s Fish Audio S2 SRT audio from {len(subtitles)} subtitles using {active_mode} mode"
return {"waveform": final_audio, "sample_rate": self.SAMPLE_RATE}, info, report, adjusted_srt
def _assemble(self, audio_segments, subtitles, mode, params):
if mode == "stretch_to_fit":
from engines.chatterbox.audio_timing import TimedAudioAssembler
assembler = TimedAudioAssembler(self.SAMPLE_RATE)
audio, method = assembler.assemble_timed_audio(
audio_segments, [(item.start_time, item.end_time) for item in subtitles],
fade_duration=params.get("fade_for_StretchToFit", 0.01),
)
return audio, None, method
assembler = AudioAssemblyEngine(self.SAMPLE_RATE)
if mode == "pad_with_silence":
return assembler.assemble_with_overlaps(audio_segments, subtitles, torch.device("cpu")), None, None
timing = TimingEngine(self.SAMPLE_RATE)
if mode == "concatenate":
replacements = timing.calculate_concatenation_adjustments(audio_segments, subtitles)
return assembler.assemble_concatenation(audio_segments, params.get("fade_for_StretchToFit", 0.01)), replacements, None
replacements, processed = timing.calculate_smart_timing_adjustments(
audio_segments, subtitles, params.get("timing_tolerance", 2.0),
params.get("max_stretch_ratio", 1.0), params.get("min_stretch_ratio", 0.5),
torch.device("cpu"),
)
audio = assembler.assemble_smart_natural(
audio_segments, processed, replacements, subtitles, torch.device("cpu")
)
return audio, replacements, None
@@ -17,7 +17,8 @@ if project_root not in sys.path:
sys.path.insert(0, project_root)
from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
from utils.voice.character_logging import resolved_character_label
class HiggsAudioSRTProcessor:
@@ -236,7 +237,8 @@ class HiggsAudioSRTProcessor:
higgs_text = f"[{display_name}] {segment_text}"
# Log what's actually being processed
print(f"📺 Processing SRT segment {i+1}/{len(srt_segments)} ({character}): '{higgs_text[:50]}...' ({expected_duration:.2f}s)")
display_name = resolved_character_label(character, char_audio_dict)
print(f"📺 Processing SRT segment {i+1}/{len(srt_segments)} ({display_name}): '{higgs_text[:50]}...' ({expected_duration:.2f}s)")
# Apply per-segment parameters
current_params = dict(generation_params)
@@ -539,4 +541,4 @@ class HiggsAudioSRTProcessor:
empty_audio = AudioProcessingUtils.create_silence(1.0, self.sample_rate)
empty_comfy = AudioProcessingUtils.format_for_comfyui(empty_audio, self.sample_rate)
return (empty_comfy, f"Higgs Audio SRT error: {e}", "Error: No timing report available", "Error: No adjusted SRT available")
return (empty_comfy, f"Higgs Audio SRT error: {e}", "Error: No timing report available", "Error: No adjusted SRT available")
@@ -22,6 +22,10 @@ from utils.audio.processing import AudioProcessingUtils
from utils.text.character_parser import character_parser
from utils.text.segment_parameters import apply_segment_parameters
from utils.voice.discovery import get_available_characters, voice_discovery
from utils.voice.character_logging import (
format_resolved_character_block,
resolved_character_label,
)
DELIVERY_PROSODY_VALUES = {
@@ -329,10 +333,9 @@ class HiggsAudioV3Processor:
prefix = _delivery_state_prefix(delivery_state, skip_categories)
prompt_text = prefix + chunk
chunk_note = f" chunk {chunk_idx + 1}/{len(chunks)}" if len(chunks) > 1 else ""
print(f"🎭 Higgs Audio v3 - Generating for '{character}'{chunk_note}:")
print("=" * 60)
print(prompt_text)
print("=" * 60)
display_name = resolved_character_label(character, voice_ref)
print(f"🎭 Higgs Audio v3 - Generating for '{display_name}'{chunk_note}:")
print(format_resolved_character_block(character, prompt_text, voice_ref))
audio_tensor = self.adapter.generate_with_pause_tags(
prompt_text,
+8 -5
View File
@@ -32,6 +32,10 @@ from utils.voice.discovery import (
get_character_mapping,
voice_discovery,
)
from utils.voice.character_logging import (
format_resolved_character_block,
resolved_character_label,
)
class MossTTSProcessor:
@@ -802,7 +806,8 @@ class MossTTSProcessor:
if enable_chunking and len(clean_text) > max_chars:
chunks = self.chunker.split_into_chunks(clean_text, max_chars)
print(f"📝 MOSS-TTS: Chunking '{character}' into {len(chunks)} chunk(s) (language={language})")
display_name = resolved_character_label(character, voice_ref)
print(f"📝 MOSS-TTS: Chunking '{display_name}' into {len(chunks)} chunk(s) (language={language})")
else:
chunks = [clean_text]
@@ -811,7 +816,7 @@ class MossTTSProcessor:
raise InterruptedError("MOSS-TTS generation interrupted by user")
chunk_note = f" chunk {chunk_idx}/{len(chunks)}" if len(chunks) > 1 else ""
print(f"🎭 MOSS-TTS - Generating for '{character}' (language={language}){chunk_note}:")
print(f"🎭 MOSS-TTS - Generating for '{display_name}' (language={language}){chunk_note}:")
for field_name in self.OFFICIAL_INLINE_FIELDS:
field_value = generation_params.get(field_name)
if field_value:
@@ -819,9 +824,7 @@ class MossTTSProcessor:
f" 🔹 {field_name}: "
f"{self._format_prompt_field_log(field_name, field_value, generation_params)}"
)
print("=" * 60)
print(chunk)
print("=" * 60)
print(format_resolved_character_block(character, chunk, voice_ref))
audio_tensor = self.adapter.generate_with_pause_tags(chunk, voice_ref, generation_params, True, character)
if audio_tensor.dim() == 1:
+3
View File
@@ -0,0 +1,3 @@
"""
OmniVoice processor package.
"""
@@ -0,0 +1,164 @@
"""
Visual Tag Builder helper node.
Builds a valid OmniVoice voice-design instruct string with canonical category
ordering while leaving the main engine's raw instruct field available for
manual edits.
"""
from typing import Dict, Tuple
EN_TO_ZH: Dict[str, str] = {
"male": "男",
"female": "女",
"child": "儿童",
"teenager": "少年",
"young adult": "青年",
"middle-aged": "中年",
"elderly": "老年",
"very low pitch": "极低音调",
"low pitch": "低音调",
"moderate pitch": "中音调",
"high pitch": "高音调",
"very high pitch": "极高音调",
"whisper": "耳语",
}
def _normalize_value(value: str) -> str:
text = str(value or "").strip()
return "" if not text or text == "None" else text
class OmniVoiceInstructionBuilderNode:
"""Structured OmniVoice instruction helper with canonical output order."""
@classmethod
def NAME(cls):
return "📐 Visual Tag Builder"
@classmethod
def INPUT_TYPES(cls):
return {
"required": {
"gender": (["None", "male", "female"], {
"default": "None",
"tooltip": "Used by the Visual Tag Builder preset."
}),
"age": (["None", "child", "teenager", "young adult", "middle-aged", "elderly"], {
"default": "None",
"tooltip": "Used by the Visual Tag Builder preset."
}),
"pitch": (["None", "very low pitch", "low pitch", "moderate pitch", "high pitch", "very high pitch"], {
"default": "None",
"tooltip": "Used by the Visual Tag Builder preset."
}),
"style": (["None", "whisper"], {
"default": "None",
"tooltip": "Used by the Visual Tag Builder preset."
}),
"accent": ([
"None",
"american accent",
"british accent",
"australian accent",
"canadian accent",
"indian accent",
"chinese accent",
"korean accent",
"japanese accent",
"portuguese accent",
"russian accent",
], {
"default": "None",
"tooltip": "Used by the Visual Tag Builder preset."
}),
"dialect": ([
"None",
"河南话",
"陕西话",
"四川话",
"贵州话",
"云南话",
"桂林话",
"济南话",
"石家庄话",
"甘肃话",
"宁夏话",
"青岛话",
"东北话",
], {
"default": "None",
"tooltip": "Used by the Visual Tag Builder preset."
}),
"output_language": (["English", "Chinese"], {
"default": "English",
"tooltip": "Controls the builder preview language and emitted text format."
}),
"instruct_text": ("STRING", {
"default": "",
"multiline": False,
"tooltip": "Text emitted by the Visual Tag Builder."
}),
}
}
RETURN_TYPES = ("STRING",)
RETURN_NAMES = ("string",)
FUNCTION = "build_instruction"
CATEGORY = "TTS Audio Suite/📐 OmniVoice"
DESCRIPTION = (
"Visual tag and attribute builder with preset-driven ordering. "
"Outputs a text string that can be used for OmniVoice or other "
"tag-based prompt workflows."
)
def build_instruction(
self,
gender: str,
age: str,
pitch: str,
style: str,
accent: str,
dialect: str,
output_language: str,
instruct_text: str,
) -> Tuple[str]:
instruct_text = str(instruct_text or "").strip()
if instruct_text:
return (instruct_text,)
gender = _normalize_value(gender)
age = _normalize_value(age)
pitch = _normalize_value(pitch)
style = _normalize_value(style)
accent = _normalize_value(accent)
dialect = _normalize_value(dialect)
output_language = str(output_language or "English").strip()
# The builder UI enforces mutual exclusion; this keeps corrupted
# workflow state from emitting an invalid mixed instruct string.
if accent and dialect:
dialect = ""
parts = [gender, age, pitch, style, accent or dialect]
parts = [part for part in parts if part]
if not parts:
return ("",)
if output_language == "Chinese":
translated = [EN_TO_ZH.get(part, part) for part in parts]
return (",".join(translated),)
return (", ".join(parts),)
NODE_CLASS_MAPPINGS = {
"OmniVoiceInstructionBuilderNode": OmniVoiceInstructionBuilderNode,
}
NODE_DISPLAY_NAME_MAPPINGS = {
"OmniVoiceInstructionBuilderNode": "📐 Visual Tag Builder",
}
+516
View File
@@ -0,0 +1,516 @@
"""
OmniVoice processor.
Handles character switching, pause tags, parameter switching, and native
OmniVoice long-form generation orchestration.
"""
import os
import re
import sys
from typing import Any, Dict, List, Optional, Tuple, Union
import comfy.model_management as model_management
import torch
current_dir = os.path.dirname(__file__)
nodes_dir = os.path.dirname(current_dir)
project_root = os.path.dirname(nodes_dir)
if project_root not in sys.path:
sys.path.insert(0, project_root)
from utils.audio.chunk_timing import ChunkTimingHelper
from utils.text.character_parser import character_parser
from utils.text.omnivoice_special_tags import convert_omnivoice_special_tags
from utils.text.pause_processor import PauseTagProcessor
from utils.voice.character_logging import (
format_resolved_character_block,
resolved_character_label,
)
from utils.text.segment_parameters import ParameterValidator, apply_segment_parameters
from utils.voice.discovery import get_available_characters, get_character_mapping, voice_discovery
class OmniVoiceProcessor:
"""Internal processor for OmniVoice generation."""
SAMPLE_RATE = 24000
MIN_AUTO_SPEECH_FRAGMENT_DURATION = 0.12
def __init__(self, adapter, engine_config: Dict[str, Any]):
self.adapter = adapter
self.config = engine_config.copy() if engine_config else {}
def update_config(self, new_config: Dict[str, Any]):
self.config = new_config.copy() if new_config else {}
self.adapter.update_config(self.config)
@staticmethod
def _check_interrupt(
item_index: Optional[int] = None,
total_items: Optional[int] = None,
character_name: Optional[str] = None,
) -> None:
if not model_management.interrupt_processing:
return
if item_index is not None and total_items is not None:
if character_name:
raise InterruptedError(
f"OmniVoice text generation interrupted at item "
f"{item_index + 1}/{total_items}, character '{character_name}'"
)
raise InterruptedError(
f"OmniVoice text generation interrupted at item {item_index + 1}/{total_items}"
)
raise InterruptedError("OmniVoice text generation interrupted by user")
@staticmethod
def _first_non_none(*values):
for value in values:
if value is None:
continue
if isinstance(value, str) and not value.strip():
continue
return value
return None
@staticmethod
def _voice_log_note(voice_ref: Dict[str, Any]) -> str:
if not isinstance(voice_ref, dict):
return " [auto/default voice]"
audio_ref = OmniVoiceProcessor._first_non_none(
voice_ref.get("audio_path"),
voice_ref.get("audio"),
voice_ref.get("waveform"),
)
if audio_ref is None:
return " [auto/default voice]"
reference_text = voice_ref.get("reference_text")
if reference_text:
return f" [ref text: {len(reference_text)} chars]"
return " [auto-transcribe reference]"
def _log_generation_text(
self,
character_name: str,
text_content: str,
language: str,
voice_ref: Dict[str, Any],
parameter_log: str = "",
show_text_content: bool = True,
) -> None:
voice_note = self._voice_log_note(voice_ref)
display_name = resolved_character_label(character_name, voice_ref)
print(f"🎭 OmniVoice - Generating for '{display_name}' (Language: {language}){voice_note}:")
if parameter_log:
print(f"🎛️ OmniVoice params: {parameter_log}")
if show_text_content:
print(format_resolved_character_block(character_name, text_content, voice_ref))
@staticmethod
def _format_parameter_log(
filtered_params: Dict[str, Any],
current_config: Dict[str, Any],
current_seed: int,
) -> str:
if not filtered_params:
return ""
ordered_parts = []
if "seed" in filtered_params:
ordered_parts.append(f"seed={current_seed}")
for key in (
"num_steps",
"guidance_scale",
"t_shift",
"speed",
"duration",
"layer_penalty_factor",
"position_temperature",
"class_temperature",
"audio_chunk_duration",
"audio_chunk_threshold",
):
if key in filtered_params:
ordered_parts.append(f"{key}={current_config.get(key)}")
if "language" in filtered_params:
ordered_parts.append(f"language={current_config.get('language')}")
remaining_keys = [
key for key in filtered_params.keys()
if key not in {
"seed",
"num_steps",
"guidance_scale",
"t_shift",
"speed",
"duration",
"layer_penalty_factor",
"position_temperature",
"class_temperature",
"audio_chunk_duration",
"audio_chunk_threshold",
"language",
}
]
for key in remaining_keys:
ordered_parts.append(f"{key}={current_config.get(key, filtered_params[key])}")
return ", ".join(ordered_parts)
def _setup_character_parser(self, text: str):
configured_language = str(self.config.get("language", "Auto") or "Auto").strip()
if configured_language.lower() in {"auto", "none"}:
resolved_language = "en"
else:
resolved_language = configured_language.lower()
character_parser.language_resolver.default_language = resolved_language
character_parser.default_language = resolved_language
character_tags = re.findall(r"\[([^\]]+)\]", text or "")
characters_from_tags = []
for tag in character_tags:
if not tag.startswith("pause:"):
characters_from_tags.append(tag.split("|")[0].strip())
available_chars = get_available_characters()
character_aliases = voice_discovery.get_character_aliases()
all_available = set()
if available_chars:
all_available.update(available_chars)
for alias, target in character_aliases.items():
all_available.add(alias.lower())
all_available.add(target.lower())
for char in characters_from_tags:
all_available.add(char.lower())
all_available.add("narrator")
character_parser.set_available_characters(list(all_available))
char_lang_defaults = voice_discovery.get_character_language_defaults()
for char, lang in char_lang_defaults.items():
character_parser.set_character_language_default(char, lang)
character_parser.reset_session_cache()
@staticmethod
def _should_apply_segment_language(seg: Any, base_config: Dict[str, Any]) -> bool:
segment_language = getattr(seg, "language", None)
if not segment_language:
return False
if getattr(seg, "explicit_language", False):
return True
global_language = str(base_config.get("language", "Auto") or "Auto").strip().lower()
parser_default = str(character_parser.default_language or "en").strip().lower()
segment_language_code = str(segment_language).strip().lower()
if segment_language_code == parser_default:
return False
return segment_language_code != global_language
@staticmethod
def _text_weight(text: str) -> int:
compact = re.sub(r"\s+", "", text or "")
return max(len(compact), 1)
def _build_generation_items(
self,
text: str,
voice_mapping: Dict[str, Any],
seed: int,
show_text_logging: bool = True,
) -> Tuple[List[Dict[str, Any]], Dict[str, Any]]:
self._setup_character_parser(text)
narrator_info = voice_mapping.get("narrator", {})
narrator_voice = narrator_info if isinstance(narrator_info, dict) else {"audio": narrator_info}
base_config = self.config.copy()
segment_objects = character_parser.parse_text_segments(text, engine_type="omnivoice")
if not segment_objects:
segment_objects = character_parser.parse_text_segments("narrator " + text, engine_type="omnivoice")
characters = list({seg.character for seg in segment_objects if seg.character})
character_mapping = get_character_mapping(characters, engine_type="audio_only")
generation_items: List[Dict[str, Any]] = []
native_chunking_notice_shown = False
for seg in segment_objects:
self._check_interrupt()
segment_text = (seg.text or "").strip()
if not segment_text:
continue
segment_text = convert_omnivoice_special_tags(segment_text)
segment_params = seg.parameters if seg.parameters else {}
current_config = base_config.copy()
current_seed = seed
filtered_params: Dict[str, Any] = {}
if segment_params:
filtered_params = ParameterValidator.filter_parameters_for_engine(segment_params, "omnivoice")
current_config = apply_segment_parameters(base_config, segment_params, "omnivoice")
if "seed" in current_config:
current_seed = int(current_config.get("seed", seed))
if self._should_apply_segment_language(seg, base_config):
current_config = current_config.copy()
current_config["language"] = seg.language
print(f" 🌍 OmniVoice language switched to: {seg.language}")
segment_language = str(current_config.get("language", "Auto") or "Auto")
parameter_log = self._format_parameter_log(filtered_params, current_config, current_seed)
voice_ref = narrator_voice.copy()
char_name = seg.character or "narrator"
if char_name != "narrator" and char_name in voice_mapping:
char_info = voice_mapping[char_name]
if isinstance(char_info, dict):
voice_ref = char_info.copy()
elif char_name != "narrator":
audio_path, ref_text = character_mapping.get(char_name, (None, None))
if audio_path:
voice_ref = {
"audio_path": audio_path,
"reference_text": ref_text or "",
}
if not native_chunking_notice_shown and len(segment_text) > 400:
print(
"ℹ️ OmniVoice uses its official native long-form chunking "
"(audio_chunk_duration / audio_chunk_threshold) instead of suite char-based chunk splitting."
)
native_chunking_notice_shown = True
explicit_duration = None
if "duration" in filtered_params:
duration_value = float(current_config.get("duration", 0.0))
if duration_value > 0:
explicit_duration = duration_value
def _append_text_item(text_content: str):
text_content = (text_content or "").strip()
if not text_content:
return
generation_items.append(
{
"kind": "text",
"text": text_content,
"config": current_config.copy(),
"seed": current_seed,
"voice_ref": voice_ref.copy() if isinstance(voice_ref, dict) else voice_ref,
"character_name": char_name,
"language": segment_language,
"parameter_log": parameter_log,
"show_text_logging": show_text_logging,
"explicit_target_duration": explicit_duration,
}
)
if PauseTagProcessor.has_pause_tags(segment_text):
raw_pause_segments, _ = PauseTagProcessor.parse_pause_tags(segment_text)
for frag_type, frag_content in raw_pause_segments:
if frag_type == "text":
_append_text_item(frag_content)
elif frag_type == "pause":
generation_items.append(
{
"kind": "pause",
"duration": float(frag_content),
"effective_duration": float(frag_content),
"text": f"[pause:{float(frag_content):.3f}s]",
}
)
else:
_append_text_item(segment_text)
return generation_items, base_config
def _apply_duration_budget(
self,
generation_items: List[Dict[str, Any]],
total_duration: float,
) -> None:
if total_duration <= 0:
return
pause_items = [item for item in generation_items if item["kind"] == "pause"]
spoken_items = [item for item in generation_items if item["kind"] == "text"]
variable_spoken_items = [item for item in spoken_items if not item.get("explicit_target_duration")]
explicit_spoken_items = [item for item in spoken_items if item.get("explicit_target_duration")]
total_pause_duration = sum(float(item.get("duration", 0.0)) for item in pause_items)
total_explicit_speech = sum(float(item.get("explicit_target_duration", 0.0)) for item in explicit_spoken_items)
desired_variable_speech = min(
max(total_duration - total_explicit_speech, 0.0),
self.MIN_AUTO_SPEECH_FRAGMENT_DURATION * len(variable_spoken_items),
)
max_pause_budget = max(0.0, total_duration - total_explicit_speech - desired_variable_speech)
pause_scale = 1.0
if total_pause_duration > 0 and total_pause_duration > max_pause_budget:
pause_scale = max_pause_budget / total_pause_duration if max_pause_budget > 0 else 0.0
print(
"⚠️ OmniVoice SRT: Pause tags exceed available subtitle time. "
f"Compressing pauses from {total_pause_duration:.3f}s to {total_pause_duration * pause_scale:.3f}s."
)
adjusted_pause_total = total_pause_duration * pause_scale
for pause_item in pause_items:
pause_item["effective_duration"] = float(pause_item.get("duration", 0.0)) * pause_scale
remaining_variable_speech = total_duration - total_explicit_speech - adjusted_pause_total
if remaining_variable_speech <= 0 and variable_spoken_items:
fallback_duration = max(0.05, min(total_duration, self.MIN_AUTO_SPEECH_FRAGMENT_DURATION))
remaining_variable_speech = fallback_duration * len(variable_spoken_items)
print(
"⚠️ OmniVoice SRT: Subtitle timing leaves no room for auto-allocated speech after pauses/fixed durations. "
f"Using minimal speech targets of {fallback_duration:.3f}s per spoken fragment."
)
total_weight = sum(self._text_weight(item["text"]) for item in variable_spoken_items)
allocated_total = 0.0
for index, item in enumerate(variable_spoken_items):
if total_weight <= 0:
target_duration = remaining_variable_speech / len(variable_spoken_items)
elif index == len(variable_spoken_items) - 1:
target_duration = max(0.0, remaining_variable_speech - allocated_total)
else:
target_duration = remaining_variable_speech * (self._text_weight(item["text"]) / total_weight)
allocated_total += target_duration
item["allocated_target_duration"] = max(0.0, float(target_duration))
for item in explicit_spoken_items:
item["allocated_target_duration"] = float(item["explicit_target_duration"])
def process_text(
self,
text: str,
voice_mapping: Dict[str, Any],
seed: int,
enable_chunking: bool = True,
max_chars_per_chunk: int = 400,
chunk_combination_method: str = "auto",
silence_between_chunks_ms: int = 100,
enable_audio_cache: bool = True,
apply_edit_postprocessing: bool = True,
show_text_logging: bool = True,
duration_budget: Optional[float] = None,
) -> List[Dict[str, Any]]:
del enable_chunking, max_chars_per_chunk # OmniVoice uses native duration-based chunking
del apply_edit_postprocessing # OmniVoice uses native inline tags instead of Step Audio EditX inline post-processing
generation_items, _base_config = self._build_generation_items(
text=text,
voice_mapping=voice_mapping,
seed=seed,
show_text_logging=show_text_logging,
)
if duration_budget is not None and duration_budget > 0:
self._apply_duration_budget(generation_items, float(duration_budget))
segment_records: List[Dict[str, Any]] = []
for item_index, item in enumerate(generation_items):
self._check_interrupt(
item_index=item_index,
total_items=len(generation_items),
character_name=item.get("character_name"),
)
if item["kind"] == "pause":
silence_duration = float(item.get("effective_duration", item.get("duration", 0.0)))
silence = PauseTagProcessor.create_silence_segment(
silence_duration,
self.SAMPLE_RATE,
torch.device("cpu"),
torch.float32,
)
if silence.dim() == 1:
silence = silence.unsqueeze(0)
segment_records.append(
{
"waveform": silence,
"sample_rate": self.SAMPLE_RATE,
"text": f"[pause:{silence_duration:.3f}s]",
}
)
continue
current_config = item["config"].copy()
allocated_target_duration = item.get("allocated_target_duration")
if allocated_target_duration is not None and allocated_target_duration > 0:
current_config["duration"] = float(allocated_target_duration)
self.adapter.update_config(current_config)
self._log_generation_text(
character_name=item["character_name"],
text_content=item["text"],
language=item["language"],
voice_ref=item["voice_ref"],
parameter_log=item["parameter_log"],
show_text_content=item["show_text_logging"],
)
self._check_interrupt(
item_index=item_index,
total_items=len(generation_items),
character_name=item.get("character_name"),
)
audio = self.adapter.generate_single(
text=item["text"],
voice_ref=item["voice_ref"],
seed=item["seed"],
enable_audio_cache=enable_audio_cache,
character_name=item["character_name"],
)
if audio.dim() == 1:
audio = audio.unsqueeze(0)
segment_records.append(
{
"waveform": audio.cpu(),
"sample_rate": self.SAMPLE_RATE,
"text": item["text"],
}
)
return segment_records
def combine_audio_segments(
self,
segments: List[Dict[str, Any]],
method: str = "auto",
silence_ms: int = 100,
original_text: str = "",
return_info: bool = False,
) -> Union[torch.Tensor, Tuple[torch.Tensor, Dict[str, Any]]]:
if not segments:
empty = torch.zeros(0, dtype=torch.float32)
if return_info:
return empty, {}
return empty
audio_segments = [seg["waveform"] for seg in segments]
text_chunks = [seg.get("text", "") for seg in segments]
text_length = len(" ".join(text_chunks))
combined_audio, chunk_info = ChunkTimingHelper.combine_audio_with_timing(
audio_segments=audio_segments,
combination_method=method,
silence_ms=silence_ms,
crossfade_duration=0.1,
sample_rate=self.SAMPLE_RATE,
text_length=text_length,
original_text=original_text,
text_chunks=text_chunks,
)
if return_info:
return combined_audio, chunk_info
return combined_audio
+462
View File
@@ -0,0 +1,462 @@
"""
OmniVoice SRT processor.
Handles full SRT subtitle processing for OmniVoice.
"""
import importlib.util
import os
import sys
from typing import Any, Dict, List, Optional, Tuple
import comfy.model_management as model_management
import torch
current_dir = os.path.dirname(__file__)
nodes_dir = os.path.dirname(current_dir)
project_root = os.path.dirname(nodes_dir)
if project_root not in sys.path:
sys.path.insert(0, project_root)
from engines.adapters.omnivoice_adapter import OmniVoiceEngineAdapter
class OmniVoiceSRTProcessor:
"""Complete SRT processor for OmniVoice."""
SAMPLE_RATE = 24000
def __init__(self, node_instance, config: Dict[str, Any]):
self.node_instance = node_instance
self.config = config.copy() if config else {}
self.adapter = OmniVoiceEngineAdapter(self.config)
self._tts_processor = None
self.srt_available = False
self.SRTParser = None
self.SRTSubtitle = None
self.SRTParseError = None
self._load_srt_modules()
def _load_srt_modules(self):
try:
from utils.system.import_manager import import_manager
success, modules, msg = import_manager.import_srt_modules()
if not success:
print(f"⚠️ OmniVoice SRT: SRT module not available: {msg}")
return
self.SRTParser = modules.get("SRTParser")
self.SRTSubtitle = modules.get("SRTSubtitle")
self.SRTParseError = modules.get("SRTParseError")
if self.SRTParser is None:
print("⚠️ OmniVoice SRT: SRT parser not available")
return
self.srt_available = True
except Exception as e:
print(f"⚠️ OmniVoice SRT: Failed to load SRT modules: {e}")
@property
def processor(self):
if self._tts_processor is None:
processor_path = os.path.join(current_dir, "omnivoice_processor.py")
spec = importlib.util.spec_from_file_location("omnivoice_processor_module", processor_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
self._tts_processor = module.OmniVoiceProcessor(self.adapter, self.config)
return self._tts_processor
def update_config(self, new_config: Dict[str, Any]):
self.config = new_config.copy() if new_config else {}
self.adapter.update_config(self.config)
if self._tts_processor is not None:
self._tts_processor.update_config(self.config)
@staticmethod
def _annotate_duration_targeting(
adjustment: Dict[str, Any],
target_duration: float,
natural_duration: float,
use_native_duration_targeting: bool,
) -> None:
adjustment["native_duration_targeting_enabled"] = bool(use_native_duration_targeting)
if use_native_duration_targeting and target_duration > 0:
adjustment["native_requested_duration"] = float(target_duration)
adjustment["pre_timing_generated_duration"] = float(natural_duration)
def process_srt_content(
self,
srt_content: str,
voice_mapping: Dict[str, Any],
seed: int,
timing_mode: str,
timing_params: Dict[str, Any],
enable_audio_cache: bool = True,
) -> Tuple[Dict[str, Any], str, str, str]:
if not self.srt_available:
raise ImportError("SRT support not available - missing required SRT parser")
self._check_interrupt()
srt_parser = self.SRTParser()
subtitles = srt_parser.parse_srt_content(srt_content, allow_overlaps=True)
from utils.timing.overlap_detection import SRTOverlapHandler
has_overlaps = SRTOverlapHandler.detect_overlaps(subtitles)
current_timing_mode, mode_switched = SRTOverlapHandler.handle_smart_natural_fallback(
timing_mode, has_overlaps, "OmniVoice SRT"
)
use_native_duration_targeting = bool(timing_params.get("use_native_duration_targeting", False))
if use_native_duration_targeting:
print("🎯 OmniVoice SRT: Native duration targeting enabled")
print(f"🚀 OmniVoice SRT: Processing {len(subtitles)} subtitles in {current_timing_mode} mode")
audio_segments, adjustments = self._process_all_subtitles(
subtitles,
voice_mapping,
seed,
use_native_duration_targeting=use_native_duration_targeting,
enable_audio_cache=enable_audio_cache,
)
self._check_interrupt()
final_audio, final_adjustments, stretch_method = self._assemble_final_audio(
audio_segments,
subtitles,
current_timing_mode,
timing_params,
adjustments,
)
if final_adjustments is not None:
if use_native_duration_targeting:
for final_adj, original_adj in zip(final_adjustments, adjustments):
self._annotate_duration_targeting(
final_adj,
original_adj.get("target_duration", 0.0),
original_adj.get("natural_duration", 0.0),
True,
)
adjustments = final_adjustments
timing_report = self._generate_timing_report(
subtitles,
adjustments,
current_timing_mode,
has_overlaps,
mode_switched,
timing_mode if mode_switched else None,
stretch_method,
)
adjusted_srt_string = self._generate_adjusted_srt_string(subtitles, adjustments, current_timing_mode)
total_duration = final_audio.shape[-1] / self.SAMPLE_RATE
mode_info = current_timing_mode
if mode_switched:
mode_info = f"{current_timing_mode} (switched from {timing_mode} due to overlaps)"
info = (
f"Generated {total_duration:.1f}s OmniVoice SRT-timed audio from {len(subtitles)} subtitles "
f"using {mode_info} mode"
)
if use_native_duration_targeting:
info += " with native duration targeting"
if final_audio.dim() == 1:
final_audio = final_audio.unsqueeze(0).unsqueeze(0)
elif final_audio.dim() == 2:
final_audio = final_audio.unsqueeze(0)
audio_output = {"waveform": final_audio, "sample_rate": self.SAMPLE_RATE}
return audio_output, info, timing_report, adjusted_srt_string
def _check_interrupt(
self,
subtitle_index: Optional[int] = None,
total_subtitles: Optional[int] = None,
character: Optional[str] = None,
):
if model_management.interrupt_processing:
if subtitle_index is not None and total_subtitles is not None:
if character:
raise InterruptedError(
f"OmniVoice SRT generation interrupted at subtitle "
f"{subtitle_index + 1}/{total_subtitles}, character '{character}'"
)
raise InterruptedError(
f"OmniVoice SRT generation interrupted at subtitle {subtitle_index + 1}/{total_subtitles}"
)
raise InterruptedError("OmniVoice SRT generation interrupted by user")
def _process_all_subtitles(
self,
subtitles: List,
voice_mapping: Dict[str, Any],
seed: int,
use_native_duration_targeting: bool = False,
enable_audio_cache: bool = True,
) -> Tuple[List[torch.Tensor], List[Dict]]:
audio_segments: List[torch.Tensor] = []
adjustments: List[Dict[str, Any]] = []
all_segments_for_editing: List[Dict[str, Any]] = []
try:
for i, sub in enumerate(subtitles):
self._check_interrupt(i, len(subtitles))
text = sub.text.strip()
if not text:
target_duration = sub.duration
silence = torch.zeros(1, int(target_duration * self.SAMPLE_RATE))
audio_segments.append(silence)
adjustments.append(
{
"index": i,
"segment_index": i,
"sequence": sub.sequence,
"natural_duration": target_duration,
"target_start": sub.start_time,
"target_end": sub.end_time,
"target_duration": target_duration,
"start_time": sub.start_time,
"end_time": sub.end_time,
"stretch_factor": 1.0,
"needs_stretching": False,
"stretch_type": "none",
"adjustment": 0.0,
"adjusted_start": sub.start_time,
"adjusted_end": sub.end_time,
"adjusted_duration": target_duration,
}
)
continue
print(f"📖 OmniVoice SRT Subtitle {i + 1}/{len(subtitles)}")
if use_native_duration_targeting and sub.duration > 0:
print(f" 🎯 Target duration: {sub.duration:.3f}s")
segment_records = self.processor.process_text(
text=text,
voice_mapping=voice_mapping,
seed=seed + i,
enable_chunking=False,
enable_audio_cache=enable_audio_cache,
apply_edit_postprocessing=False,
duration_budget=sub.duration if use_native_duration_targeting else None,
)
subtitle_has_edit_tags = any(seg.get("edit_tags") for seg in segment_records)
if subtitle_has_edit_tags:
for seg_idx, seg in enumerate(segment_records):
waveform = seg["waveform"]
if waveform.dim() == 1:
waveform = waveform.unsqueeze(0)
elif waveform.dim() == 3:
waveform = waveform.squeeze(0)
all_segments_for_editing.append(
{
**seg,
"waveform": waveform.cpu(),
"subtitle_index": i,
"segment_index": seg_idx,
}
)
audio_segments.append(None)
else:
if len(segment_records) > 1:
audio, _ = self.processor.combine_audio_segments(
segments=segment_records,
method="auto",
silence_ms=0,
original_text=text,
return_info=True,
)
elif segment_records:
audio = segment_records[0]["waveform"]
else:
audio = torch.zeros(1, 0)
if audio.dim() == 1:
audio = audio.unsqueeze(0)
elif audio.dim() == 3:
audio = audio.squeeze(0)
audio_segments.append(audio.cpu())
target_start = sub.start_time
target_end = sub.end_time
target_duration = target_end - target_start
adjustments.append(
{
"index": i,
"segment_index": i,
"sequence": sub.sequence,
"natural_duration": 0.0,
"target_start": target_start,
"target_end": target_end,
"target_duration": target_duration,
"start_time": target_start,
"end_time": target_end,
"stretch_factor": 1.0,
"needs_stretching": False,
"stretch_type": "none",
"adjustment": 0.0,
"adjusted_start": target_start,
"adjusted_end": target_end,
"adjusted_duration": 0.0,
}
)
if not subtitle_has_edit_tags:
audio = audio_segments[i]
natural_duration = audio.shape[-1] / self.SAMPLE_RATE
stretch_factor = target_duration / natural_duration if natural_duration > 0 else 1.0
adjustments[i].update(
{
"natural_duration": natural_duration,
"stretch_factor": stretch_factor,
"needs_stretching": abs(stretch_factor - 1.0) > 0.05,
"stretch_type": "compress" if stretch_factor < 1.0 else "expand" if stretch_factor > 1.0 else "none",
"adjustment": natural_duration - target_duration,
"adjusted_duration": natural_duration,
}
)
self._annotate_duration_targeting(
adjustments[i],
target_duration,
natural_duration,
use_native_duration_targeting,
)
print(f"✅ OmniVoice Subtitle {i + 1}/{len(subtitles)}: {natural_duration:.3f}s (expected {target_duration:.3f}s)")
else:
print(f"✅ OmniVoice Subtitle {i + 1}/{len(subtitles)}: queued for batch edit processing")
if all_segments_for_editing:
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
processed_segments = apply_edit_post_processing(all_segments_for_editing, engine_config=self.config)
grouped_segments: Dict[int, List[Dict[str, Any]]] = {}
for seg in processed_segments:
grouped_segments.setdefault(seg["subtitle_index"], []).append(seg)
for subtitle_index, segments in grouped_segments.items():
segments = sorted(segments, key=lambda item: item.get("segment_index", 0))
if len(segments) > 1:
combined_audio, _ = self.processor.combine_audio_segments(
segments=segments,
method="auto",
silence_ms=0,
original_text=subtitles[subtitle_index].text,
return_info=True,
)
else:
combined_audio = segments[0]["waveform"]
if combined_audio.dim() == 1:
combined_audio = combined_audio.unsqueeze(0)
elif combined_audio.dim() == 3:
combined_audio = combined_audio.squeeze(0)
combined_audio = combined_audio.cpu()
audio_segments[subtitle_index] = combined_audio
target_duration = adjustments[subtitle_index]["target_duration"]
natural_duration = combined_audio.shape[-1] / self.SAMPLE_RATE
stretch_factor = target_duration / natural_duration if natural_duration > 0 else 1.0
adjustments[subtitle_index].update(
{
"natural_duration": natural_duration,
"stretch_factor": stretch_factor,
"needs_stretching": abs(stretch_factor - 1.0) > 0.05,
"stretch_type": "compress" if stretch_factor < 1.0 else "expand" if stretch_factor > 1.0 else "none",
"adjustment": natural_duration - target_duration,
"adjusted_duration": natural_duration,
}
)
self._annotate_duration_targeting(
adjustments[subtitle_index],
target_duration,
natural_duration,
use_native_duration_targeting,
)
return audio_segments, adjustments
finally:
self.processor.update_config(self.config)
def _assemble_final_audio(self, audio_segments, subtitles, timing_mode, timing_params, adjustments):
if timing_mode == "stretch_to_fit":
from engines.chatterbox.audio_timing import TimedAudioAssembler
assembler = TimedAudioAssembler(self.SAMPLE_RATE)
target_timings = [(sub.start_time, sub.end_time) for sub in subtitles]
fade_duration = timing_params.get("fade_for_StretchToFit", 0.01)
final_audio, stretch_method_used = assembler.assemble_timed_audio(
audio_segments,
target_timings,
fade_duration=fade_duration,
)
return final_audio, None, stretch_method_used
if timing_mode == "pad_with_silence":
from utils.timing.assembly import AudioAssemblyEngine
assembler = AudioAssemblyEngine(self.SAMPLE_RATE)
final_audio = assembler.assemble_with_overlaps(audio_segments, subtitles, torch.device("cpu"))
return final_audio, None, None
if timing_mode == "concatenate":
from utils.timing.assembly import AudioAssemblyEngine
from utils.timing.engine import TimingEngine
timing_engine = TimingEngine(self.SAMPLE_RATE)
assembler = AudioAssemblyEngine(self.SAMPLE_RATE)
new_adjustments = timing_engine.calculate_concatenation_adjustments(audio_segments, subtitles)
fade_duration = timing_params.get("fade_for_StretchToFit", 0.01)
final_audio = assembler.assemble_concatenation(audio_segments, fade_duration)
return final_audio, new_adjustments, None
from utils.timing.assembly import AudioAssemblyEngine
from utils.timing.engine import TimingEngine
timing_engine = TimingEngine(self.SAMPLE_RATE)
assembler = AudioAssemblyEngine(self.SAMPLE_RATE)
smart_adjustments, processed_segments = timing_engine.calculate_smart_timing_adjustments(
audio_segments,
subtitles,
timing_params.get("timing_tolerance", 2.0),
timing_params.get("max_stretch_ratio", 1.0),
timing_params.get("min_stretch_ratio", 0.5),
torch.device("cpu"),
)
final_audio = assembler.assemble_smart_natural(
audio_segments,
processed_segments,
smart_adjustments,
subtitles,
torch.device("cpu"),
)
return final_audio, smart_adjustments, None
def _generate_timing_report(
self,
subtitles,
adjustments,
current_timing_mode,
has_overlaps=False,
mode_switched=False,
original_mode=None,
stretch_method=None,
) -> str:
from utils.timing.reporting import SRTReportGenerator
reporter = SRTReportGenerator()
return reporter.generate_timing_report(
subtitles,
adjustments,
current_timing_mode,
has_overlaps,
mode_switched,
original_mode,
stretch_method,
)
def _generate_adjusted_srt_string(self, subtitles, adjustments, timing_mode: str) -> str:
from utils.timing.reporting import SRTReportGenerator
reporter = SRTReportGenerator()
return reporter.generate_adjusted_srt_string(subtitles, adjustments, timing_mode)
+10 -6
View File
@@ -24,7 +24,11 @@ from utils.text.pause_processor import PauseTagProcessor
from engines.adapters.qwen3_tts_adapter import Qwen3TTSEngineAdapter
from utils.models.language_mapper import resolve_language_alias
from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
from utils.voice.character_logging import (
format_resolved_character_block,
resolved_character_label,
)
class Qwen3TTSProcessor:
@@ -523,7 +527,8 @@ class Qwen3TTSProcessor:
combined_text_clean, combined_text_edit_tags = get_edit_tags_for_segment(combined_text)
chunks = self.chunker.split_into_chunks(combined_text_clean, max_chars)
print(f"📝 Chunking {character}'s combined text into {len(chunks)} chunks (Language: {segment_lang}){voice_note}")
display_name = resolved_character_label(character, voice_ref)
print(f"📝 Chunking {display_name}'s combined text into {len(chunks)} chunks (Language: {segment_lang}){voice_note}")
for chunk_idx, chunk in enumerate(chunks):
# Check for interruption during chunk processing
@@ -573,10 +578,9 @@ class Qwen3TTSProcessor:
# Extract inline edit tags BEFORE generation
combined_text_clean, combined_text_edit_tags = get_edit_tags_for_segment(combined_text)
print(f"🎭 Qwen3-TTS - Generating for '{character}' (Language: {segment_lang}){voice_note}:")
print("="*60)
print(combined_text_clean)
print("="*60)
display_name = resolved_character_label(character, voice_ref)
print(f"🎭 Qwen3-TTS - Generating for '{display_name}' (Language: {segment_lang}){voice_note}:")
print(format_resolved_character_block(character, combined_text_clean, voice_ref))
# Set current segment for time tracking (skip in SRT mode - managed at subtitle level)
if not self._srt_mode:
@@ -477,7 +477,7 @@ Test Text: {reference_text}
Generated by: Qwen3-TTS Voice Designer (VoiceDesign model)
via TTS Audio Suite for ComfyUI
https://github.com/FahimF/ComfyUI_TTS_Audio_Suite
https://github.com/diodiogod/TTS-Audio-Suite
"""
metadata_path = os.path.join(voices_dir, f"{character_name}.txt")
with open(metadata_path, 'w', encoding='utf-8') as f:
@@ -24,6 +24,10 @@ from utils.text.pause_processor import PauseTagProcessor
from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
from engines.adapters.step_audio_editx_adapter import StepAudioEditXEngineAdapter
from utils.voice.character_logging import (
format_resolved_character_block,
resolved_character_label,
)
class StepAudioEditXProcessor:
@@ -330,10 +334,9 @@ class StepAudioEditXProcessor:
if edit_tags:
print(f"🎨 Found {len(edit_tags)} edit tag(s) for post-processing")
print(f"🎭 Step Audio EditX - Generating for '{character}'{voice_note}:")
print("="*60)
print(clean_text)
print("="*60)
display_name = resolved_character_label(character, voice_ref)
print(f"🎭 Step Audio EditX - Generating for '{display_name}'{voice_note}:")
print(format_resolved_character_block(character, clean_text, voice_ref))
# Set current segment for time tracking (skip in SRT mode - managed at subtitle level)
if not self._srt_mode:
@@ -21,7 +21,8 @@ if project_root not in sys.path:
from utils.system.import_manager import import_manager
from utils.text.character_parser import character_parser
from utils.voice.discovery import get_available_characters, voice_discovery
from utils.voice.discovery import get_available_characters, voice_discovery
from utils.voice.character_logging import resolved_character_label
# Import StepAudioEditXProcessor using file-based import (avoids package issues)
import importlib.util
@@ -199,7 +200,7 @@ class StepAudioEditXSRTProcessor:
Returns:
Complete voice mapping with all characters
"""
from utils.voice.discovery import get_character_mapping
from utils.voice.discovery import get_character_mapping
import re
# Extract all character names from subtitle text
@@ -246,11 +247,11 @@ class StepAudioEditXSRTProcessor:
'prompt_audio_path': audio_path,
'prompt_text': ref_text
}
print(f"🎭 SRT: Using character voice for '{character}'")
print(f"🎭 SRT: Using character voice for '{resolved_character_label(character, voice_mapping[character])}'")
else:
# Fallback to narrator voice
voice_mapping[character] = narrator_ref
print(f"🔄 SRT: Using narrator voice fallback for '{character}'")
print(f"🔄 SRT: Using narrator voice fallback for '{resolved_character_label(character, narrator_ref)}'")
return voice_mapping
+198 -32
View File
@@ -131,12 +131,16 @@ Hello! This is unified SRT TTS with character switching.
"step": 0.5,
"tooltip": "Maximum allowed deviation (in seconds) for timing adjustments in 'smart_natural' mode. Higher values allow more flexibility."
}),
"batch_size": ("INT", {
"default": 0, "min": 0, "max": 32, "step": 1,
"tooltip": "Parallel processing workers. 0 = sequential (recommended), 2+ = streaming mode. Note: Streaming often slower than sequential mode. F5-TTS doesn't support streaming yet."
}),
}
}
"batch_size": ("INT", {
"default": 0, "min": 0, "max": 32, "step": 1,
"tooltip": "Parallel processing workers. 0 = sequential (recommended), 2+ = streaming mode. Note: Streaming often slower than sequential mode. F5-TTS doesn't support streaming yet."
}),
"use_native_duration_targeting": ("BOOLEAN", {
"default": False,
"tooltip": "When enabled, supported engines like OmniVoice use the model's native duration parameter for each subtitle during generation. This helps the model aim closer to the subtitle length before the suite applies final timing adjustment, which can reduce stretching/compression and sound more natural."
}),
}
}
RETURN_TYPES = ("AUDIO", "STRING", "STRING", "STRING")
RETURN_NAMES = ("audio", "generation_info", "timing_report", "Adjusted_SRT")
@@ -208,6 +212,18 @@ Hello! This is unified SRT TTS with character switching.
stable_params['optimize'] = config.get('optimize', False)
stable_params['max_generate_length'] = config.get('max_generate_length', 500)
if engine_type == "fish_audio_s2":
stable_params['model_variant'] = config.get('model_variant', 's2-pro')
stable_params['precision'] = config.get('precision', 'bfloat16')
stable_params['compile'] = config.get('compile', False)
stable_params['quantization'] = config.get('quantization', 'none')
stable_params['multi_speaker_mode'] = config.get('multi_speaker_mode', 'Native Multi-Speaker')
stable_params['language_prompting'] = config.get('language_prompting', 'Auto Inline Tag')
if engine_type == "omnivoice":
stable_params['model_variant'] = config.get('model_variant', 'OmniVoice')
stable_params['dtype'] = config.get('dtype', 'auto')
if engine_type == "moss_tts":
stable_params['model_variant'] = config.get('model_variant', 'MOSS-TTS-Local-Transformer')
stable_params['multi_speaker_mode'] = config.get('multi_speaker_mode', 'Custom Character Switching')
@@ -544,6 +560,79 @@ Hello! This is unified SRT TTS with character switching.
}
return engine_instance
elif engine_type == "fish_audio_s2":
processor_path = os.path.join(nodes_dir, "fish_audio_s2", "fish_audio_s2_srt_processor.py")
processor_spec = importlib.util.spec_from_file_location("fish_audio_s2_srt_processor_module", processor_path)
processor_module = importlib.util.module_from_spec(processor_spec)
processor_spec.loader.exec_module(processor_module)
FishAudioS2SRTProcessor = processor_module.FishAudioS2SRTProcessor
class FishAudioS2SRTWrapper:
def __init__(self, cfg):
self.config = cfg.copy()
self.processor = FishAudioS2SRTProcessor(self, self.config)
def update_config(self, new_config):
self.config = new_config.copy()
self.processor.update_config(new_config)
def check_interrupt(self):
if model_management.interrupt_processing:
raise InterruptedError("Fish Audio S2 SRT processing interrupted by user")
engine_instance = FishAudioS2SRTWrapper(config)
import time
self._cached_engine_instances[cache_key] = {
'instance': engine_instance,
'timestamp': time.time()
}
return engine_instance
elif engine_type == "omnivoice":
omnivoice_srt_processor_path = os.path.join(nodes_dir, "omnivoice", "omnivoice_srt_processor.py")
omnivoice_srt_spec = importlib.util.spec_from_file_location("omnivoice_srt_processor_module", omnivoice_srt_processor_path)
omnivoice_srt_module = importlib.util.module_from_spec(omnivoice_srt_spec)
omnivoice_srt_spec.loader.exec_module(omnivoice_srt_module)
OmniVoiceSRTProcessor = omnivoice_srt_module.OmniVoiceSRTProcessor
class OmniVoiceSRTWrapper:
def __init__(self, config):
self.config = config
self.processor = OmniVoiceSRTProcessor(self, config)
def update_config(self, new_config):
self.config = new_config.copy()
self.processor.update_config(new_config)
def process_with_error_handling(self, func):
try:
return func()
except Exception as e:
raise e
def format_audio_output(self, audio_tensor, sample_rate):
if audio_tensor.is_cuda:
audio_tensor = audio_tensor.cpu()
if audio_tensor.dim() == 1:
audio_tensor = audio_tensor.unsqueeze(0).unsqueeze(0)
elif audio_tensor.dim() == 2:
audio_tensor = audio_tensor.unsqueeze(0)
return {"waveform": audio_tensor, "sample_rate": sample_rate}
def check_interrupt(self):
if model_management.interrupt_processing:
raise InterruptedError("OmniVoice SRT processing interrupted by user")
engine_instance = OmniVoiceSRTWrapper(config)
import time
self._cached_engine_instances[cache_key] = {
'instance': engine_instance,
'timestamp': time.time()
}
return engine_instance
elif engine_type == "vibevoice":
# Import and create the VibeVoice SRT processor using absolute import
vibevoice_srt_processor_path = os.path.join(nodes_dir, "vibevoice", "vibevoice_srt_processor.py")
@@ -833,9 +922,11 @@ Hello! This is unified SRT TTS with character switching.
else:
raise ValueError(f"Unknown engine type: {engine_type}")
except Exception as e:
print(f"❌ Failed to create engine SRT node instance: {e}")
return None
except Exception as e:
if isinstance(e, InterruptedError):
raise
print(f"❌ Failed to create engine SRT node instance: {e}")
return None
def _get_voice_reference(self, opt_narrator, narrator_voice: str):
"""
@@ -896,7 +987,10 @@ Hello! This is unified SRT TTS with character switching.
reference_text = "" # No reference text available from direct audio
print(f"📺 TTS SRT: Using direct audio input ({character_name})")
print(f"⚠️ TTS SRT: Direct audio input has no reference text - F5-TTS engines will fail")
print(
"⚠️ TTS SRT: Direct audio input has no reference text - "
"F5-TTS and OmniVoice cloning will fail"
)
return None, audio_tensor, reference_text, character_name
# Priority 2: narrator_voice dropdown (fallback)
@@ -921,15 +1015,17 @@ Hello! This is unified SRT TTS with character switching.
print("⚠️ TTS SRT: No voice reference provided - this may cause issues with some engines")
return None, None, "", "narrator"
except Exception as e:
print(f"❌ Voice reference error: {e}")
return None, None, "", "narrator"
except Exception as e:
if isinstance(e, InterruptedError):
raise
print(f"❌ Voice reference error: {e}")
return None, None, "", "narrator"
def generate_srt_speech(self, TTS_engine: Dict[str, Any], srt_content: str, narrator_voice: str,
seed: int, timing_mode: str, opt_narrator=None, enable_audio_cache: bool = True,
fade_for_StretchToFit: float = 0.01, max_stretch_ratio: float = 1.0,
min_stretch_ratio: float = 0.5, timing_tolerance: float = 2.0,
batch_size: int = 0):
def generate_srt_speech(self, TTS_engine: Dict[str, Any], srt_content: str, narrator_voice: str,
seed: int, timing_mode: str, opt_narrator=None, enable_audio_cache: bool = True,
fade_for_StretchToFit: float = 0.01, max_stretch_ratio: float = 1.0,
min_stretch_ratio: float = 0.5, timing_tolerance: float = 2.0,
batch_size: int = 0, use_native_duration_targeting: bool = False):
"""
Generate SRT-timed speech using the selected TTS engine.
This is a DELEGATION WRAPPER that preserves all original SRT functionality.
@@ -942,11 +1038,12 @@ Hello! This is unified SRT TTS with character switching.
timing_mode: How to align audio with SRT timings
opt_narrator: Voice reference from Character Voices node
enable_audio_cache: Enable audio caching
fade_for_StretchToFit: Crossfade duration for stretch_to_fit mode
max_stretch_ratio: Maximum stretch ratio for smart_natural mode
min_stretch_ratio: Minimum stretch ratio for smart_natural mode
timing_tolerance: Timing tolerance for smart_natural mode
batch_size: Number of parallel workers (0=sequential, 1+=streaming parallel)
fade_for_StretchToFit: Crossfade duration for stretch_to_fit mode
max_stretch_ratio: Maximum stretch ratio for smart_natural mode
min_stretch_ratio: Minimum stretch ratio for smart_natural mode
timing_tolerance: Timing tolerance for smart_natural mode
batch_size: Number of parallel workers (0=sequential, 1+=streaming parallel)
use_native_duration_targeting: Inject subtitle durations into supported engines during generation
Returns:
Tuple of (audio_tensor, generation_info, timing_report, adjusted_srt)
@@ -971,20 +1068,34 @@ Hello! This is unified SRT TTS with character switching.
if config.get('emotion_audio') is not None:
print(f"🎭 Emotion control connected to SRT generation")
if not engine_type:
raise ValueError("TTS engine missing engine_type")
print(f"📺 TTS SRT: Starting {engine_type} SRT generation")
if not engine_type:
raise ValueError("TTS engine missing engine_type")
print(f"📺 TTS SRT: Starting {engine_type} SRT generation")
native_duration_targeting_active = bool(
use_native_duration_targeting and engine_type == "omnivoice"
)
if use_native_duration_targeting and not native_duration_targeting_active:
print(
f"ℹ️ Native duration targeting is currently supported only for OmniVoice in TTS SRT. "
f"Ignoring it for engine '{engine_type}'."
)
# Get voice reference (opt_narrator takes priority)
audio_path, audio_tensor, reference_text, character_name = self._get_voice_reference(opt_narrator, narrator_voice)
# Validate F5-TTS requirements: must have reference text
if engine_type == "f5tts" and not reference_text.strip():
raise ValueError(
"F5-TTS requires reference text. When using direct audio input, "
"please use Character Voices node instead, which provides both audio and text."
)
if engine_type == "f5tts" and not reference_text.strip():
raise ValueError(
"F5-TTS requires reference text. When using direct audio input, "
"please use Character Voices node instead, which provides both audio and text."
)
if engine_type == "omnivoice" and (audio_tensor is not None or audio_path) and not reference_text.strip():
raise ValueError(
"OmniVoice voice cloning requires reference text. "
"Do not connect raw audio directly. Use Character Voices node or a narrator voice with a matching .reference.txt file."
)
# Create proper engine SRT node instance to preserve ALL functionality
engine_instance = self._create_proper_engine_node_instance(TTS_engine)
@@ -1185,6 +1296,59 @@ Hello! This is unified SRT TTS with character switching.
enable_audio_cache=enable_audio_cache
)
elif engine_type == "fish_audio_s2":
print(
f" Audio cache: {'enabled' if enable_audio_cache else 'disabled'} "
"(TTS SRT node)"
)
timing_params = {
'fade_for_StretchToFit': fade_for_StretchToFit,
'max_stretch_ratio': max_stretch_ratio,
'min_stretch_ratio': min_stretch_ratio,
'timing_tolerance': timing_tolerance
}
voice_mapping = {}
if audio_tensor is not None or audio_path:
voice_mapping['narrator'] = {
'audio': audio_tensor,
'audio_path': audio_path,
'reference_text': reference_text or "",
}
result = engine_instance.processor.process_srt_content(
srt_content=srt_content,
voice_mapping=voice_mapping,
seed=seed,
timing_mode=timing_mode,
timing_params=timing_params,
enable_audio_cache=enable_audio_cache
)
elif engine_type == "omnivoice":
timing_params = {
'fade_for_StretchToFit': fade_for_StretchToFit,
'max_stretch_ratio': max_stretch_ratio,
'min_stretch_ratio': min_stretch_ratio,
'timing_tolerance': timing_tolerance,
'use_native_duration_targeting': native_duration_targeting_active,
}
voice_mapping = {}
if audio_tensor is not None or audio_path:
voice_mapping['narrator'] = {
'audio': audio_tensor,
'audio_path': audio_path,
'reference_text': reference_text if reference_text else "",
}
result = engine_instance.processor.process_srt_content(
srt_content=srt_content,
voice_mapping=voice_mapping,
seed=seed,
timing_mode=timing_mode,
timing_params=timing_params,
enable_audio_cache=enable_audio_cache
)
elif engine_type == "vibevoice":
# Use the VibeVoice SRT processor from the wrapper instance
voice_mapping = {"narrator": audio_tensor} if audio_tensor else {}
@@ -1422,6 +1586,8 @@ Hello! This is unified SRT TTS with character switching.
or "MOSS-TTSD Native Multi-Speaker Dialogue does not support this SRT input" in msg
):
raise
if isinstance(e, InterruptedError):
raise
error_msg = f"❌ TTS SRT generation failed: {e}"
print(error_msg)
import traceback
+188 -1
View File
@@ -210,6 +210,18 @@ Back to the main narrator voice for the conclusion.""",
stable_params['optimize'] = config.get('optimize', False)
stable_params['max_generate_length'] = config.get('max_generate_length', 500)
if engine_type == "fish_audio_s2":
stable_params['model_variant'] = config.get('model_variant', 's2-pro')
stable_params['precision'] = config.get('precision', 'bfloat16')
stable_params['compile'] = config.get('compile', False)
stable_params['quantization'] = config.get('quantization', 'none')
stable_params['multi_speaker_mode'] = config.get('multi_speaker_mode', 'Native Multi-Speaker')
stable_params['language_prompting'] = config.get('language_prompting', 'Auto Inline Tag')
if engine_type == "omnivoice":
stable_params['model_variant'] = config.get('model_variant', 'OmniVoice')
stable_params['dtype'] = config.get('dtype', 'auto')
# For MOSS-TTS, include model identity and load-time options.
if engine_type == "moss_tts":
stable_params['model_variant'] = config.get('model_variant', 'MOSS-TTS-Local-Transformer')
@@ -418,6 +430,8 @@ Back to the main narrator voice for the conclusion.""",
voice_mapping[char] = {"waveform": waveform, "sample_rate": sample_rate}
print(f"🎭 VibeVoice: Using character-specific voice for '{char}'")
except Exception as e:
if isinstance(e, InterruptedError):
raise
print(f"⚠️ Failed to load character audio for '{char}': {e}")
voice_mapping[char] = char_audio # Fallback to main voice
print(f"🔄 VibeVoice: Using main voice fallback for '{char}'")
@@ -551,6 +565,67 @@ Back to the main narrator voice for the conclusion.""",
return engine_instance
elif engine_type == "fish_audio_s2":
from engines.adapters.fish_audio_s2_adapter import FishAudioS2Adapter
processor_path = os.path.join(nodes_dir, "fish_audio_s2", "fish_audio_s2_processor.py")
processor_spec = importlib.util.spec_from_file_location("fish_audio_s2_processor_module", processor_path)
processor_module = importlib.util.module_from_spec(processor_spec)
processor_spec.loader.exec_module(processor_module)
FishAudioS2Processor = processor_module.FishAudioS2Processor
class FishAudioS2Wrapper:
def __init__(self, cfg):
self.config = cfg.copy()
self.adapter = FishAudioS2Adapter(self.config)
self.processor = FishAudioS2Processor(self.adapter, self.config)
def update_config(self, new_config):
self.config = new_config.copy()
self.adapter.update_config(new_config)
self.processor.update_config(new_config)
engine_instance = FishAudioS2Wrapper(config)
import time
self._cached_engine_instances[cache_key] = {
'instance': engine_instance,
'timestamp': time.time()
}
return engine_instance
elif engine_type == "omnivoice":
from engines.adapters.omnivoice_adapter import OmniVoiceEngineAdapter
omnivoice_processor_path = os.path.join(nodes_dir, "omnivoice", "omnivoice_processor.py")
omnivoice_processor_spec = importlib.util.spec_from_file_location("omnivoice_processor_module", omnivoice_processor_path)
omnivoice_processor_module = importlib.util.module_from_spec(omnivoice_processor_spec)
omnivoice_processor_spec.loader.exec_module(omnivoice_processor_module)
OmniVoiceProcessor = omnivoice_processor_module.OmniVoiceProcessor
class OmniVoiceWrapper:
def __init__(self, cfg):
self.config = cfg.copy()
self.adapter = OmniVoiceEngineAdapter(self.config)
self.processor = OmniVoiceProcessor(self.adapter, self.config)
def update_config(self, new_config):
self.config = new_config.copy()
self.adapter.update_config(new_config)
self.processor.update_config(new_config)
def check_interrupt(self):
if model_management.interrupt_processing:
raise InterruptedError("OmniVoice processing interrupted by user")
engine_instance = OmniVoiceWrapper(config)
import time
self._cached_engine_instances[cache_key] = {
'instance': engine_instance,
'timestamp': time.time()
}
return engine_instance
elif engine_type == "qwen3_tts":
# Create Qwen3-TTS processor instance
# Use global nodes_dir (already defined at module level)
@@ -718,6 +793,8 @@ Back to the main narrator voice for the conclusion.""",
raise ValueError(f"Unknown engine type: {engine_type}")
except Exception as e:
if isinstance(e, InterruptedError):
raise
if "MOSS LoRA/base model mismatch" in str(e):
raise
print(f"❌ Failed to create engine node instance: {e}")
@@ -783,7 +860,10 @@ Back to the main narrator voice for the conclusion.""",
reference_text = "" # No reference text available from direct audio
print(f"🎤 TTS Text: Using direct audio input ({character_name})")
print(f"⚠️ TTS Text: Direct audio input has no reference text - F5-TTS will fail, Qwen3-TTS will use x_vector_only mode (lower quality)")
print(
"⚠️ TTS Text: Direct audio input has no reference text - "
"F5-TTS and OmniVoice cloning will fail, Qwen3-TTS will use x_vector_only mode (lower quality)"
)
return None, audio_tensor, reference_text, character_name
# Priority 2: narrator_voice dropdown (fallback)
@@ -828,6 +908,8 @@ Back to the main narrator voice for the conclusion.""",
return None, None, "", "narrator"
except Exception as e:
if isinstance(e, InterruptedError):
raise
print(f"❌ Voice reference error: {e}")
return None, None, "", "narrator"
@@ -914,6 +996,11 @@ Back to the main narrator voice for the conclusion.""",
"F5-TTS requires reference text. When using direct audio input, "
"please use Character Voices node instead, which provides both audio and text."
)
if engine_type == "omnivoice" and (audio_tensor is not None or audio_path) and not reference_text.strip():
raise ValueError(
"OmniVoice voice cloning requires reference text. "
"Do not connect raw audio directly. Use Character Voices node or a narrator voice with a matching .reference.txt file."
)
# Create proper engine node instance to preserve ALL functionality
engine_instance = self._create_proper_engine_node_instance(TTS_engine)
@@ -1371,6 +1458,100 @@ Back to the main narrator voice for the conclusion.""",
formatted_audio = AudioProcessingUtils.format_for_comfyui(combined_audio, 48000)
result = (formatted_audio, generation_info)
elif engine_type == "fish_audio_s2":
import re
from utils.audio.chunk_timing import ChunkTimingHelper
print(
f" Audio cache: {'enabled' if enable_audio_cache else 'disabled'} "
"(TTS Text node)"
)
voice_mapping = {}
if audio_tensor is not None or audio_path:
voice_mapping['narrator'] = {
'audio': audio_tensor,
'audio_path': audio_path,
'reference_text': reference_text or '',
}
segment_records = engine_instance.processor.process_text(
text=text,
voice_mapping=voice_mapping,
seed=seed,
enable_chunking=enable_chunking,
max_chars_per_chunk=max_chars_per_chunk,
chunk_combination_method=chunk_combination_method,
silence_between_chunks_ms=silence_between_chunks_ms,
enable_audio_cache=enable_audio_cache
)
combined_audio, chunk_info = engine_instance.processor.combine_audio_segments(
segments=segment_records,
method=chunk_combination_method,
silence_ms=silence_between_chunks_ms,
original_text=text,
return_info=True
)
total_duration = combined_audio.shape[-1] / 44100.0 if combined_audio.numel() else 0.0
clean_text = re.sub(r'\[.*?\]', '', text)
base_info = (
f"Generated {total_duration:.1f}s audio from {len(clean_text)} characters "
f"(Fish Audio S2 Pro, narrator: {char_display})"
)
base_info += "\n🎭 Character switching, pause tags, and Fish inline tags enabled"
generation_info = ChunkTimingHelper.enhance_generation_info(f"✅ {base_info}", chunk_info)
result = (AudioProcessingUtils.format_for_comfyui(combined_audio, 44100), generation_info)
elif engine_type == "omnivoice":
import re
from utils.audio.chunk_timing import ChunkTimingHelper
if not hasattr(engine_instance, "processor"):
omnivoice_processor_path = os.path.join(nodes_dir, "omnivoice", "omnivoice_processor.py")
omnivoice_processor_spec = importlib.util.spec_from_file_location("omnivoice_processor_module", omnivoice_processor_path)
omnivoice_processor_module = importlib.util.module_from_spec(omnivoice_processor_spec)
omnivoice_processor_spec.loader.exec_module(omnivoice_processor_module)
OmniVoiceProcessor = omnivoice_processor_module.OmniVoiceProcessor
engine_instance.processor = OmniVoiceProcessor(engine_instance.adapter, config)
engine_instance.processor.update_config(config)
voice_mapping = {}
if audio_tensor is not None or audio_path:
voice_mapping['narrator'] = {
'audio': audio_tensor,
'audio_path': audio_path,
'reference_text': reference_text or '',
}
segment_records = engine_instance.processor.process_text(
text=text,
voice_mapping=voice_mapping,
seed=seed,
enable_chunking=enable_chunking,
max_chars_per_chunk=max_chars_per_chunk,
chunk_combination_method=chunk_combination_method,
silence_between_chunks_ms=silence_between_chunks_ms,
enable_audio_cache=enable_audio_cache
)
combined_audio, chunk_info = engine_instance.processor.combine_audio_segments(
segments=segment_records,
method=chunk_combination_method,
silence_ms=silence_between_chunks_ms,
original_text=text,
return_info=True
)
total_duration = combined_audio.shape[-1] / 24000.0 if combined_audio.numel() else 0.0
clean_text = re.sub(r'\[.*?\]', '', text)
text_length = len(clean_text)
base_info = f"Generated {total_duration:.1f}s audio from {text_length} characters (OmniVoice, narrator: {char_display})"
base_info += "\n🎭 Character switching, pause tags, voice design, and native long-form chunking enabled"
generation_info = ChunkTimingHelper.enhance_generation_info(f"✅ {base_info}", chunk_info)
formatted_audio = AudioProcessingUtils.format_for_comfyui(combined_audio, 24000)
result = (formatted_audio, generation_info)
elif engine_type == "moss_tts":
import re
from utils.audio.chunk_timing import ChunkTimingHelper
@@ -1597,6 +1778,8 @@ Back to the main narrator voice for the conclusion.""",
}
print(f"🎭 Qwen3-TTS: Using character-specific voice for '{character}' (ICL mode)")
except Exception as e:
if isinstance(e, InterruptedError):
raise
print(f"⚠️ Failed to load character audio for '{character}': {e}")
# Fallback to narrator voice if available
if audio_tensor is not None and reference_text:
@@ -1640,6 +1823,8 @@ Back to the main narrator voice for the conclusion.""",
print(f"⚠️⚠️ Qwen3-TTS: Character '{character}' has audio but NO reference text")
print(f"⚠️⚠️ Using x_vector_only mode (speaker embedding only) - LOWER QUALITY than ICL mode")
except Exception as e:
if isinstance(e, InterruptedError):
raise
print(f"⚠️ Failed to load character audio for '{character}': {e}")
# Fallback to narrator voice if available
if audio_tensor is not None:
@@ -1775,6 +1960,8 @@ Back to the main narrator voice for the conclusion.""",
raise
if engine_type == "index_tts":
raise
if isinstance(e, InterruptedError):
raise
error_msg = f"❌ TTS Text generation failed: {e}"
print(error_msg)
import traceback
+20 -12
View File
@@ -124,8 +124,10 @@ class UnifiedVoiceChangerNode(BaseVCNode):
return normalized_audio
except Exception as e:
raise ValueError(f"Failed to process {input_name}: {e}")
except Exception as e:
if isinstance(e, InterruptedError):
raise
raise ValueError(f"Failed to process {input_name}: {e}")
def _handle_rvc_chunked_conversion(self, source_chunks, source_sample_rate, rvc_engine,
narrator_target, refinement_passes, max_chunk_duration, chunk_method):
@@ -393,9 +395,11 @@ class UnifiedVoiceChangerNode(BaseVCNode):
print(f"✅ RVC voice conversion completed with {refinement_passes} refinement passes {cache_info}")
return converted_audio, final_conversion_info
except Exception as e:
print(f"❌ RVC voice conversion failed: {e}")
raise RuntimeError(f"RVC voice conversion failed: {e}")
except Exception as e:
if isinstance(e, InterruptedError):
raise
print(f"❌ RVC voice conversion failed: {e}")
raise RuntimeError(f"RVC voice conversion failed: {e}")
def _convert_audio_for_rvc(self, audio_dict):
"""Convert ComfyUI audio format to RVC-compatible format."""
@@ -595,9 +599,11 @@ class UnifiedVoiceChangerNode(BaseVCNode):
else:
raise ValueError(f"Engine type '{engine_type}' does not support voice conversion. Currently supported: ChatterBox, CosyVoice")
except Exception as e:
print(f"❌ Failed to create engine VC node instance: {e}")
return None
except Exception as e:
if isinstance(e, InterruptedError):
raise
print(f"❌ Failed to create engine VC node instance: {e}")
return None
def _split_audio_into_chunks(self, audio: torch.Tensor, sample_rate: int,
max_duration: int, method: str = "smart") -> List[torch.Tensor]:
@@ -1126,10 +1132,12 @@ class UnifiedVoiceChangerNode(BaseVCNode):
print(f"✅ Voice Changer: {engine_type} conversion successful")
return (converted_audio, conversion_info)
except Exception as e:
error_msg = f"❌ Voice conversion failed: {e}"
print(error_msg)
import traceback
except Exception as e:
if isinstance(e, InterruptedError):
raise
error_msg = f"❌ Voice conversion failed: {e}"
print(error_msg)
import traceback
traceback.print_exc()
# Return empty audio and error info
+8 -4
View File
@@ -24,6 +24,7 @@ from utils.text.pause_processor import PauseTagProcessor
from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
from engines.adapters.vibevoice_adapter import VibeVoiceEngineAdapter
from utils.voice.character_logging import resolved_character_label
class VibeVoiceProcessor:
@@ -183,9 +184,11 @@ class VibeVoiceProcessor:
for group_idx, (character, segment_list) in enumerate(grouped_segments):
# Check for interruption before processing each character block
voice_ref = voice_mapping.get(character)
display_name = resolved_character_label(character, voice_ref)
if model_management.interrupt_processing:
raise InterruptedError(f"VibeVoice character block {group_idx + 1}/{len(grouped_segments)} ({character}) interrupted by user")
print(f"🎤 Block {group_idx + 1}: Character '{character}' with {len(segment_list)} segments")
raise InterruptedError(f"VibeVoice character block {group_idx + 1}/{len(grouped_segments)} ({display_name}) interrupted by user")
print(f"🎤 Block {group_idx + 1}: Character '{display_name}' with {len(segment_list)} segments")
# Combine text blocks for this character (VibeVoice style)
combined_text = '\n'.join(seg.text.strip() for seg in segment_list)
@@ -359,7 +362,8 @@ class VibeVoiceProcessor:
if not isinstance(voice_ref, dict):
voice_note = " [⚠️ Zero-shot mode - no voice reference]"
print(f"🎭 CUSTOM CHARACTER BLOCK - Generating combined text for '{character}'{voice_note}:")
display_name = resolved_character_label(character, voice_ref)
print(f"🎭 CUSTOM CHARACTER BLOCK - Generating combined text for '{display_name}'{voice_note}:")
print("="*60)
print(combined_text)
print("="*60)
@@ -600,4 +604,4 @@ class VibeVoiceProcessor:
def cleanup(self):
"""Clean up resources"""
if self.adapter:
self.adapter.cleanup()
self.adapter.cleanup()
+32 -22
View File
@@ -33,7 +33,8 @@ from utils.text.character_parser import parse_character_text, character_parser
from utils.text.segment_parameters import apply_segment_parameters
from utils.text.pause_processor import PauseTagProcessor
from utils.audio.processing import AudioProcessingUtils
from utils.voice.discovery import get_available_characters, get_character_mapping
from utils.voice.discovery import get_available_characters, get_character_mapping
from utils.voice.character_logging import resolved_character_label
from engines.adapters.vibevoice_adapter import VibeVoiceEngineAdapter
@@ -448,10 +449,11 @@ class VibeVoiceSRTProcessor:
)
else:
# Multiple segments for same character - combine in VibeVoice format
print(f"🎤 SRT Block {group_idx + 1}: Character '{character}' with {len(text_list)} segments")
display_name = resolved_character_label(character, voice_ref)
print(f"🎤 SRT Block {group_idx + 1}: Character '{display_name}' with {len(text_list)} segments")
combined_text = '\n'.join(f"Speaker 1: {text.strip()}" for text in text_list)
print(f"🎭 SRT CHARACTER BLOCK - Generating combined text for '{character}':")
print(f"🎭 SRT CHARACTER BLOCK - Generating combined text for '{display_name}':")
print("="*60)
print(combined_text)
print("="*60)
@@ -552,10 +554,11 @@ class VibeVoiceSRTProcessor:
)
else:
# Multiple segments for same character with same parameters - combine in VibeVoice format
print(f"🎤 SRT Block {group_idx + 1}: Character '{character}' with {len(segment_list)} segments")
display_name = resolved_character_label(character, voice_ref)
print(f"🎤 SRT Block {group_idx + 1}: Character '{display_name}' with {len(segment_list)} segments")
combined_text = '\n'.join(f"Speaker 1: {seg.text.strip()}" for seg in segment_list)
print(f"🎭 SRT CHARACTER BLOCK - Generating combined text for '{character}':")
print(f"🎭 SRT CHARACTER BLOCK - Generating combined text for '{display_name}':")
print("="*60)
print(combined_text)
print("="*60)
@@ -797,22 +800,29 @@ class VibeVoiceSRTProcessor:
Returns:
Dict mapping character names to speaker numbers (1-4)
"""
# Collect all unique characters across all subtitles
all_characters = set()
for subtitle in subtitles:
character_segments = parse_character_text(subtitle.text, None)
for character, _ in character_segments:
all_characters.add(character)
# Sort characters to ensure consistent mapping
# narrator/untagged first, then alphabetical order for tagged characters
sorted_chars = sorted(all_characters, key=lambda x: (x != 'narrator', x))
# Map to speakers (limit to 4 for VibeVoice native mode)
char_to_speaker = {}
for i, character in enumerate(sorted_chars[:4]): # Max 4 speakers
char_to_speaker[character] = i + 1
# Collect characters across all subtitles in first-appearance order.
# If narrator exists anywhere, reserve Speaker 1 for narrator and then
# map remaining characters by first appearance order.
seen_characters = set()
ordered_characters = []
for subtitle in subtitles:
character_segments = parse_character_text(subtitle.text, None)
for character, _ in character_segments:
if character not in seen_characters:
seen_characters.add(character)
ordered_characters.append(character)
narrator_present = "narrator" in seen_characters
if narrator_present:
ordered_chars = ["narrator"] + [c for c in ordered_characters if c != "narrator"]
else:
ordered_chars = ordered_characters
# Map to speakers (limit to 4 for VibeVoice native mode)
char_to_speaker = {}
for i, character in enumerate(ordered_chars[:4]): # Max 4 speakers
char_to_speaker[character] = i + 1
if char_to_speaker:
print(f"🎭 Global SRT character mapping: {char_to_speaker}")
@@ -822,4 +832,4 @@ class VibeVoiceSRTProcessor:
def cleanup(self):
"""Clean up resources"""
if self.adapter:
self.adapter.cleanup()
self.adapter.cleanup()
+1 -1
View File
@@ -1,7 +1,7 @@
[project]
name = "tts_audio_suite"
description = "TTS Audio Suite - Universal multi-engine TTS extension for ComfyUI with unified architecture supporting IndexTTS-2, ChatterBox, Chatterbox Multilingual TTS (Official 23-Lang), F5-TTS, Higgs Audio 2, VibeVoice, and RVC engines. It has character voice management, SRT subtitle TTS support, and audio processing capabilities."
version = "5.1.1"
version = "5.4.4"
license = {file = "LICENSE"}
[project.urls]
+14 -11
View File
@@ -20,11 +20,13 @@ numpy>=1.26.4,<2.3.0 # Compatible with both numpy 1.26.4 and 2.x series
setuptools>=65.0.0 # Provides distutils compatibility for Python 3.12+ (required by FunASR bundled code)
# Audio processing (safe)
soundfile>=0.12.0
sounddevice>=0.4.0
# Text processing (safe)
jieba
soundfile>=0.12.0
sounddevice>=0.4.0
pydub
webdataset
# Text processing (safe)
jieba
pypinyin
unidecode
phonemizer # IPA phonemization for multilingual TTS (requires espeak system dependency)
@@ -92,12 +94,13 @@ torchfcpe>=0.0.4 # Fast Context-based Pitch Estimation for RVC (fcpe met
# --- PROBLEMATIC PACKAGES ---
# These are installed by install.py with special handling (NOT here in requirements.txt):
# - librosa (--no-deps): Forces numpy downgrade
# - descript-audio-codec (--no-deps): Conflicts with protobuf
# - cached-path (--no-deps): Forces package downgrades
# - torchcrepe (--no-deps): Conflicts via librosa dependency
# - onnxruntime (--no-deps): Forces numpy 2.3.x, needed for OpenSeeFace
# - opencv-python (--no-deps): Forces numpy downgrade via numpy<2.3.0 constraint
# - librosa (--no-deps): Forces numpy downgrade
# - descript-audio-codec (--no-deps): Conflicts with protobuf
# - cached-path (--no-deps): Forces package downgrades
# - omnivoice (--no-deps): We install official OmniVoice separately to avoid letting pip reshape shared deps
# - torchcrepe (--no-deps): Conflicts via librosa dependency
# - onnxruntime (--no-deps): Forces numpy 2.3.x, needed for OpenSeeFace
# - opencv-python (--no-deps): Forces numpy downgrade via numpy<2.3.0 constraint
# - gradio (--no-deps): Forces pydantic, pillow, pydantic-core downgrades
# --- PYTHON 3.13 NOTES ---
+35 -12
View File
@@ -18,13 +18,28 @@ python scripts/bump_version_enhanced.py 3.0.2 --changelog-file changelog.txt --c
import os
import sys
import subprocess
import argparse
from version_utils import VersionManager
def run_git_command(command: str) -> bool:
import argparse
from version_utils import VersionManager
def configure_utf8_console() -> None:
"""Keep Windows console output from failing on non-UTF-8 locales."""
for stream in (sys.stdout, sys.stderr):
if hasattr(stream, "reconfigure"):
stream.reconfigure(encoding="utf-8", errors="replace")
def run_git_command(command: str) -> bool:
"""Run git command and return success status"""
try:
result = subprocess.run(command, shell=True, capture_output=True, text=True)
result = subprocess.run(
command,
shell=True,
capture_output=True,
text=True,
encoding="utf-8",
errors="replace",
)
if result.returncode != 0:
print(f"Git command failed: {command}")
print(f"Error: {result.stderr}")
@@ -93,9 +108,9 @@ def main():
elif args.commit_file and args.changelog_file:
# File input mode
try:
with open(args.commit_file, 'r') as f:
with open(args.commit_file, 'r', encoding='utf-8') as f:
commit_description = f.read().strip()
with open(args.changelog_file, 'r') as f:
with open(args.changelog_file, 'r', encoding='utf-8') as f:
changelog_description = f.read().strip()
except Exception as e:
print(f"Error reading files: {e}")
@@ -146,7 +161,7 @@ def main():
elif args.file:
# Legacy file mode - same description for both
try:
with open(args.file, 'r') as f:
with open(args.file, 'r', encoding='utf-8') as f:
description = f.read().strip()
commit_description = description
changelog_description = description
@@ -246,7 +261,9 @@ def main():
[sys.executable, table_gen_path, "--readme"],
cwd=vm.project_root,
capture_output=True,
text=True
text=True,
encoding="utf-8",
errors="replace",
)
if result.returncode != 0:
print(f"Warning: Table generation had issues:\n{result.stderr}")
@@ -285,7 +302,12 @@ def main():
# Write commit message to temp file for complex messages
import tempfile
with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as f:
with tempfile.NamedTemporaryFile(
mode='w',
suffix='.txt',
delete=False,
encoding='utf-8',
) as f:
f.write(commit_message)
temp_file = f.name
@@ -320,5 +342,6 @@ def main():
vm.restore_files(backup)
sys.exit(1)
if __name__ == "__main__":
main()
if __name__ == "__main__":
configure_utf8_console()
main()
+63 -49
View File
@@ -5,12 +5,20 @@ Generates documentation tables from YAML source of truth.
Also injects condensed table into README.md between markers.
"""
import yaml
import re
from pathlib import Path
def load_data():
import yaml
import re
import sys
from pathlib import Path
def configure_utf8_console() -> None:
"""Keep Windows console output from failing on non-UTF-8 locales."""
for stream in (sys.stdout, sys.stderr):
if hasattr(stream, "reconfigure"):
stream.reconfigure(encoding="utf-8", errors="replace")
def load_data():
"""Load YAML data"""
yaml_path = Path(__file__).parent.parent / "docs/Dev reports/tts_audio_suite_engines.yaml"
with open(yaml_path, encoding="utf-8") as f:
@@ -98,17 +106,19 @@ def generate_engine_comparison(data):
output.append("")
output.append("| Engine | Isolation | Models | Size | TTS | SRT | VC | ASR | Training | License | Special Features | Languages |")
output.append("| ------------------ | --------- | ----------------------------------------- | ------------ | :-: | :-: | :-: | :-: | :------: | ------------------------ | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |")
for e in engines:
supported_language_count = sum(
1
for lang_key, lang_data in e["languages"].items()
if lang_key != "_default" and lang_data["supported"]
)
if e["id"] == "rvc":
language_summary = "Any"
else:
language_summary = str(supported_language_count)
for e in engines:
language_summary = e.get("language_summary_full")
if not language_summary:
supported_language_count = sum(
1
for lang_key, lang_data in e["languages"].items()
if lang_key != "_default" and lang_data["supported"]
)
if e["id"] == "rvc":
language_summary = "Any"
else:
language_summary = str(supported_language_count)
# Format special features with proper spacing
features = ", ".join(e.get("special_features", []))
@@ -150,32 +160,35 @@ def generate_readme_condensed_table(data):
output.append("| Engine | Languages | Size | Key Features |")
output.append("|--------|-----------|------|--------------|")
# Show ALL engines
for e in engines:
# Get first 6-8 language flags
flags = []
count = 0
for lang_data in e["languages"].values():
if isinstance(lang_data, dict) and lang_data.get("supported") and lang_data.get("flag"):
flags.append(lang_data["flag"])
count += 1
if count >= 6: # Limit to 6 flags for readability
break
lang_display = "\u200B".join(flags)
# Special handling for ChatterBox 23L
if e["id"] == "chatterbox-23l":
lang_display = "🌐 24 languages"
# Special handling for RVC
elif e["id"] == "rvc":
lang_display = "🌐 Any"
else:
# Add count if more languages exist
total_langs = sum(1 for ld in e["languages"].values() if ld["supported"])
if total_langs > 6:
lang_display += f" +{total_langs - 6}"
# Show ALL engines
for e in engines:
language_summary_override = e.get("language_summary_compact")
if language_summary_override:
lang_display = language_summary_override
else:
# Get first 6-8 language flags
flags = []
count = 0
for lang_data in e["languages"].values():
if isinstance(lang_data, dict) and lang_data.get("supported") and lang_data.get("flag"):
flags.append(lang_data["flag"])
count += 1
if count >= 6: # Limit to 6 flags for readability
break
lang_display = "\u200B".join(flags)
# Special handling for ChatterBox 23L
if e["id"] == "chatterbox-23l":
lang_display = "🌐 24 languages"
# Special handling for RVC
elif e["id"] == "rvc":
lang_display = "🌐 Any"
else:
# Add count if more languages exist
total_langs = sum(1 for ld in e["languages"].values() if ld["supported"])
if total_langs > 6:
lang_display += f" +{total_langs - 6}"
# Get 1-2 key features
special_features = e.get("special_features", [])
@@ -436,10 +449,10 @@ def generate_license_table(data):
output.append("The project code is MIT. Model weights carry their own licenses:")
output.append("")
output.append(sep)
output.append(header)
output.append(sep)
for name, lic, com in rows:
output.append(f" {name:<{col1}} {lic:<{col2}} {com:<{col3}}")
output.append(header)
output.append(sep)
for name, lic, com in rows:
output.append(f" {name:<{col1}} {lic:<{col2}} {com}")
output.append(sep)
output.append("")
output.append("Users are responsible for complying with respective model licenses.")
@@ -639,5 +652,6 @@ def main():
print("\n✅ All tables generated successfully!")
if __name__ == "__main__":
main()
if __name__ == "__main__":
configure_utf8_console()
main()
+8 -8
View File
@@ -39,7 +39,7 @@ class VersionManager:
"""Get current version from nodes.py"""
try:
nodes_file = os.path.join(self.project_root, 'nodes.py')
with open(nodes_file, 'r') as f:
with open(nodes_file, 'r', encoding='utf-8') as f:
content = f.read()
match = re.search(self.version_files['nodes.py']['pattern'], content)
@@ -58,7 +58,7 @@ class VersionManager:
config = self.version_files[relative_path]
with open(file_path, 'r') as f:
with open(file_path, 'r', encoding='utf-8') as f:
content = f.read()
# Find and replace version
@@ -69,7 +69,7 @@ class VersionManager:
print(f"Warning: No version found in {relative_path}")
return False
with open(file_path, 'w') as f:
with open(file_path, 'w', encoding='utf-8') as f:
f.write(updated_content)
print(f"✓ Updated {relative_path}")
@@ -284,7 +284,7 @@ This matches the F5-TTS dropdown behavior for consistent user experience across
try:
changelog_path = os.path.join(self.project_root, 'CHANGELOG.md')
with open(changelog_path, 'r') as f:
with open(changelog_path, 'r', encoding='utf-8') as f:
content = f.read()
# Generate changelog entry content using extracted method
@@ -305,7 +305,7 @@ This matches the F5-TTS dropdown behavior for consistent user experience across
# Insert the new entry with proper spacing
lines.insert(insert_index, new_entry.rstrip())
with open(changelog_path, 'w') as f:
with open(changelog_path, 'w', encoding='utf-8') as f:
f.write('\n'.join(lines))
print(f"✓ Added changelog entry for v{version}")
@@ -321,7 +321,7 @@ This matches the F5-TTS dropdown behavior for consistent user experience across
for file_path in self.version_files.keys():
full_path = os.path.join(self.project_root, file_path)
if os.path.exists(full_path):
with open(full_path, 'r') as f:
with open(full_path, 'r', encoding='utf-8') as f:
backups[file_path] = f.read()
return backups
@@ -330,9 +330,9 @@ This matches the F5-TTS dropdown behavior for consistent user experience across
try:
for file_path, content in backups.items():
full_path = os.path.join(self.project_root, file_path)
with open(full_path, 'w') as f:
with open(full_path, 'w', encoding='utf-8') as f:
f.write(content)
return True
except Exception as e:
print(f"Error restoring files: {e}")
return False
return False
+185
View File
@@ -0,0 +1,185 @@
# FL-MCP Live ComfyUI Validation
This runbook defines an optional live acceptance test for new engine implementations using [ComfyUI_FL-MCP](https://github.com/filliptm/ComfyUI_FL-MCP).
FL-MCP gives an MCP-capable LLM access to ComfyUI workflow state, queue execution, history, errors, output files, and browser screenshots. It does not replace pytest, canonical Windows validation, or human evaluation of audio quality.
## Efficient Tool Use
Use terminal tools for code inspection, edits, pytest, process management, startup logs, and routine log monitoring. Use FL-MCP at live-validation checkpoints: capability audit, relevant widget configuration, workflow queueing, prompt-specific execution verification, and output inspection.
Do not use FL-MCP as a replacement for efficient terminal work. Avoid fetching full workflow JSON, repeated node definitions or workflow overviews, screenshots, broad execution history, and polling calls unless they are needed to construct the workflow or diagnose a failure. Prefer one targeted call with the known node or prompt ID. A representative FL-MCP acceptance pass after automated tests is normally sufficient; repeat it only after changes that can affect live behavior.
## Preconditions
- Use the canonical Windows ComfyUI installation documented in the repository instructions.
- ComfyUI starts with TTS Audio Suite enabled and no suite import errors.
- FL-MCP is installed according to its upstream documentation.
- The FL-MCP server is connected to the MCP client.
- For browser-only operations, ComfyUI is open and the FL-MCP browser bridge reports connected.
- The MCP client is bound to the same browser session. Check the bridge session endpoint or capability audit and require both `has_frontend: true` and `has_mcp: true` for one session. A frontend-only session provides no canvas access to the LLM.
- Required model files and reference audio are available.
- The user has authorized any installation or configuration changes. Do not weaken FL-MCP safety gates just to complete a test.
If any precondition is missing, record it as a blocked or skipped check. Do not claim FL-MCP validation.
## Implementation Restart Loop
ComfyUI imports custom-node Python code at startup. Editing an engine does not update an already-running ComfyUI process. During implementation, use this loop:
1. Edit the engine implementation.
2. Run the relevant automated tests.
3. Identify the Windows process listening on port `8188` and inspect its command line. Stop it only if it is the canonical ComfyUI `main.py` process.
4. Resolve the ComfyUI root and Python executable from the user's local configuration, then launch ComfyUI. For example:
```powershell
$comfyRoot = '<COMFYUI_ROOT>'
$pythonExe = '<COMFYUI_PYTHON>'
Set-Location $comfyRoot
& $pythonExe 'main.py'
```
On Windows, prefer a visible Windows Terminal tab with live output (PowerShell console only as fallback) so clicking the log cannot pause ComfyUI. When the agent needs searchable logs, also use `Start-Transcript` to a temporary file under `ComfyUI/temp` and inspect only relevant tails. Preserve required canonical launch arguments and never track machine-specific paths.
5. Poll `http://127.0.0.1:8188/system_stats` until it responds or startup fails. Inspect the startup log for custom-node import errors.
6. Refresh the already-open ComfyUI browser tab. A hard refresh may be required for changed JavaScript.
7. Confirm the FL-MCP backend is healthy and the browser bridge is connected again. Canvas tools are not available until the browser reconnects.
8. Run the live workflow checks below.
9. After a fix, repeat from step 2. Never validate new Python code against a process that started before the edit.
Do not stop every `python.exe` process. Other engines and applications may use Python. Resolve the owner of port `8188`, verify its executable/command line, and stop only the intended ComfyUI process.
### Windows external bridge fallback
If a ComfyUI console interrupt also stops FL-MCP's embedded backend, do not describe that as the backend shutting down ComfyUI. Both processes are receiving the same console signal. Separate them instead:
1. In FL-MCP's local, untracked `.env`, set `BACKEND_LAUNCH_MODE=manual`, `AUTO_START_BACKEND=false`, and `AUTO_RESTART_BACKEND=false`.
2. Start `backend/server.py` in its own visible Windows Terminal window using FL-MCP's dedicated Python environment.
3. Start ComfyUI in a separate visible Windows Terminal window using its canonical launcher.
4. Confirm the bridge backend is healthy, refresh the ComfyUI browser tab, and reconnect the FL-MCP panel before auditing capabilities.
Do not leave the embedded launcher enabled while manually running `backend/server.py`; that can create competing bridge processes on the same port.
For manual Windows mode, keep two visible service terminals: one running ComfyUI and one running FL-MCP `backend/server.py`; the MCP client (for example Codex) is a separate process that must be restarted or rebound with the browser session ID so one session reports both `has_frontend: true` and `has_mcp: true`.
FL-MCP process-control permission is not required for this loop. An authorized coding agent can manage the canonical process through PowerShell. Keep `FL_MCP_ENABLE_COMFY_PROCESS_CONTROL=false` unless the user explicitly chooses FL-MCP-managed process control.
### MCP client session binding
REST access does not imply browser/canvas access. Before starting the implementation session:
1. Open ComfyUI and obtain the persistent FL-MCP browser session ID from its panel or the backend session endpoint.
2. Configure the MCP client process with:
```text
FL_MCP_MODE=subprocess
FL_MCP_SESSION_ID=<browser-session-id>
FL_MCP_WS_URL=ws://127.0.0.1:8000/ws
COMFYUI_SERVER_URL=http://127.0.0.1:8188
```
3. Restart the MCP client session so it launches the server with the new environment.
4. Run the capability audit and verify the same session reports both frontend and MCP connections.
The browser stores its session ID in local storage, so normal ComfyUI restarts and browser refreshes should retain it. If the browser session is reset or a different browser/profile is used, update the MCP client configuration and restart that client again.
## Validation Layers
Run all applicable layers:
1. Existing unit and integration tests through `tests/run_tests.py`.
2. Live ComfyUI registration and workflow validation through FL-MCP, or the manual fallback below.
3. Human listening evaluation.
FL-MCP is strongest at layer 2. A screenshot alone is not a passing test.
## Standard Smoke Sequence
Tool names can change between FL-MCP versions. Discover the connected server's available tools and use the current equivalents rather than assuming an exact name.
### 1. Establish a baseline
- Record the ComfyUI URL, TTS Audio Suite branch/commit, FL-MCP version, target engine, model variant, and runtime mode.
- Record the ComfyUI process start time and confirm it started after the implementation changes being tested.
- Check ComfyUI health and queue state. Read recent execution history only when needed to separate an existing failure from the new run.
- Preserve pre-existing errors separately so they are not attributed to the new engine.
### 2. Verify registration and UI state
- Inspect the target node definition and confirm the new engine node is registered. Do not repeatedly fetch definitions that have already been verified after the current restart.
- Confirm every relevant unified node is registered: TTS Text and SRT for TTS engines, plus VC, ASR, or special nodes when scoped.
- Open or construct a minimal workflow.
- Verify important widget values and connections with targeted node/workflow calls. Fetch full workflow JSON only when construction, migration, or debugging requires it.
- Capture a canvas screenshot only when UI layout, dynamic widgets, or visual state is part of the change or useful failure evidence.
### 3. Run the smallest real generation
- Use short deterministic text and the smallest suitable model/configuration.
- Queue the workflow.
- Wait for terminal execution state; do not treat prompt submission as success.
- Inspect execution history/details for the prompt.
- On failure, capture the node ID/type, exception, traceback or message, and relevant configuration.
- On success, verify that an expected audio output is listed and exists in ComfyUI output or temp storage.
### 4. Exercise required paths
For every TTS engine:
- Run TTS Text.
- Run SRT with at least two subtitle entries.
- Run one supported character, narrator, or reference-audio path as applicable.
- Run pause tags.
- Run one real engine-specific parameter switch when supported.
- Repeat an identical request to exercise generated-audio caching when applicable.
- Clear VRAM/unload, then generate again.
- Start a sufficiently long generation and test interrupt/cancel.
Also test VC, ASR, and special nodes when they are part of the agreed engine scope.
### 5. Inspect outputs
- Confirm output audio metadata is plausible, including sample rate, channel/batch shape where exposed, and non-zero duration.
- Confirm SRT output timing and artifact count match the workflow contract.
- Verify each run through its prompt-specific execution result rather than repeatedly fetching broad history or relying on the canvas alone.
- Ask the user to listen to representative outputs. The LLM must not claim audio quality, pronunciation, similarity, or naturalness based only on metadata.
## Required Evidence
Return a compact validation report containing:
- Environment and exact branch/commit.
- Saved workflow path, or the relevant node configuration and connections. Export full workflow JSON only when needed to reproduce the result.
- Screenshot when visual UI behavior was tested; otherwise mark it `not required`.
- Prompt/execution ID and terminal status for each run.
- Relevant execution error details for failed runs.
- Output artifact paths for successful runs.
- A pass/fail/blocked table for each required path.
- Automated test results.
- Explicitly untested items and reasons.
- Human audio-quality verdict, or `not evaluated`.
Do not commit generated audio, screenshots, logs, downloaded models, or local FL-MCP configuration unless the user explicitly requests tracked fixtures.
## Manual Fallback
If FL-MCP is unavailable:
1. Start the canonical Windows ComfyUI instance and inspect startup logs for node import errors.
2. Build or load the same minimal workflows in the browser.
3. Queue each required path and wait for completion.
4. Copy full errors from ComfyUI/server logs.
5. Export workflow JSON and record output artifact paths.
6. Capture screenshots manually.
7. Produce the same validation report and mark the driver as `manual`.
Manual execution is valid evidence. Do not install FL-MCP during an engine task unless installation was explicitly authorized.
## Stop Conditions
Stop the live test and report the blocker when:
- Installing dependencies would modify the user's environment without authorization.
- A required model download, license acceptance, credential, or reference file is unavailable.
- Repeated execution risks corrupting data or exhausting disk space.
- The connected MCP tools cannot identify terminal execution state or retrieve actionable errors.
Fix implementation defects that are within the engine task. Do not hide failures by changing safety settings, deleting unrelated queues/files, or weakening assertions.
+30 -12
View File
@@ -4,19 +4,23 @@ This guide covers running tests for the TTS Audio Suite custom node.
## Quick Start
For a split installation, first create the ignored `tests/.env.local` file described under [Path Configuration & Local Overrides](#path-configuration--local-overrides). The wrapper cannot infer paths when ComfyUI and this repository live in separate directory trees.
```bash
cd C:\_stability_matrix\Data\Packages\Comfy-new\custom_nodes\TTS-Audio-Suite
cd /path/to/ComfyUI/custom_nodes/TTS-Audio-Suite
# Run all tests
..\..\venv\Scripts\python -m pytest tests/ -v
python tests/run_tests.py -v
# Run only fast unit tests (no server needed)
..\..\venv\Scripts\python -m pytest tests/unit/ -m unit -v
python tests/run_tests.py unit/ -m unit -v
# Run integration tests (requires ComfyUI running)
..\..\venv\Scripts\python -m pytest tests/integration/ -m integration -v
python tests/run_tests.py integration/ -m integration -v
```
Always use `tests/run_tests.py`; it selects the configured Python, changes into the isolated test directory, and ensures the local pytest configuration wins.
---
## Test Categories
@@ -47,10 +51,10 @@ Tests that run against a live ComfyUI server.
```bash
# Run by marker
pytest -m unit # Fast, no server
pytest -m integration # Requires ComfyUI
pytest -m slow # Full generation tests
pytest -m cosyvoice # CosyVoice3 specific
python tests/run_tests.py -m unit # Fast, no server
python tests/run_tests.py -m integration # Requires ComfyUI
python tests/run_tests.py -m slow # Full generation tests
python tests/run_tests.py -m cosyvoice # CosyVoice3 specific
```
---
@@ -61,16 +65,30 @@ Integration tests require ComfyUI to be running:
```bash
# Terminal 1: Start ComfyUI
cd C:\_stability_matrix\Data\Packages\Comfy-new
venv\Scripts\python main.py
cd /path/to/ComfyUI
/path/to/comfyui/python main.py
# Terminal 2: Run tests
cd custom_nodes\TTS-Audio-Suite
..\..\venv\Scripts\python -m pytest tests/integration/ -m integration -v
cd /path/to/ComfyUI/custom_nodes/TTS-Audio-Suite
python tests/run_tests.py integration/ -m integration -v
```
---
## FL-MCP-Assisted Live Validation
[ComfyUI_FL-MCP](https://github.com/filliptm/ComfyUI_FL-MCP) can drive and inspect a live ComfyUI session from an MCP-capable coding client. It complements pytest by exposing workflow state, browser screenshots, queue execution, history, errors, and output files. It is optional and is not a dependency of TTS Audio Suite.
Use it for new-engine acceptance testing after the implementation and automated tests are complete. Follow [FL_MCP_VALIDATION.md](FL_MCP_VALIDATION.md) for prerequisites, the standard smoke sequence, required evidence, safety boundaries, and the manual fallback.
New custom-node Python code is loaded only when ComfyUI starts. During implementation, follow the runbook's mandatory edit–restart–browser reconnect–validate loop. Keeping an old ComfyUI process open does not test newly written engine code.
REST availability alone is insufficient for canvas inspection. Before live validation, bind the MCP client to the browser's persistent FL-MCP session ID and verify one bridge session reports both frontend and MCP connections.
FL-MCP does not determine whether generated speech sounds good. A human must assess pronunciation, voice similarity, artifacts, pacing, and other subjective audio qualities.
---
## Test Fixtures
Test workflow files are in `tests/fixtures/workflows/`:
+27 -1
View File
@@ -10,6 +10,8 @@ Session-scoped fixtures for ComfyUI server management and API testing.
# ============================================================================
import os
import sys
import tempfile
import types
from unittest.mock import MagicMock
# Set testing environment
@@ -22,7 +24,6 @@ _MOCK_MODULES = [
'comfy.model_management',
'comfy.utils',
'nodes',
'folder_paths',
'server',
'execution',
'comfy_extras',
@@ -32,6 +33,31 @@ for _module_name in _MOCK_MODULES:
if _module_name not in sys.modules:
sys.modules[_module_name] = MagicMock()
# folder_paths cannot be a raw MagicMock because code under test uses
# folder_paths.models_dir in os.path.join/os.makedirs calls. A MagicMock there
# creates junk folders like ./MagicMock/mock.models_dir/... on disk.
if 'folder_paths' not in sys.modules:
mock_folder_paths = types.ModuleType("folder_paths")
mock_models_dir = tempfile.mkdtemp(prefix="tts_suite_test_models_")
mock_input_dir = tempfile.mkdtemp(prefix="tts_suite_test_input_")
mock_output_dir = tempfile.mkdtemp(prefix="tts_suite_test_output_")
mock_temp_dir = tempfile.mkdtemp(prefix="tts_suite_test_temp_")
mock_folder_paths.models_dir = mock_models_dir
mock_folder_paths.input_directory = mock_input_dir
mock_folder_paths.output_directory = mock_output_dir
mock_folder_paths.temp_directory = mock_temp_dir
mock_folder_paths.get_folder_paths = lambda *args, **kwargs: []
mock_folder_paths.add_model_folder_path = lambda *args, **kwargs: None
mock_folder_paths.get_input_directory = lambda: mock_input_dir
mock_folder_paths.get_output_directory = lambda: mock_output_dir
mock_folder_paths.get_temp_directory = lambda: mock_temp_dir
mock_folder_paths.get_annotated_filepath = lambda path: path
mock_folder_paths.exists_annotated_filepath = lambda path: os.path.exists(path)
sys.modules['folder_paths'] = mock_folder_paths
# Now safe to import pytest and other modules
import pytest
import subprocess
+1 -1
View File
@@ -3,7 +3,7 @@ testpaths = unit integration
python_files = test_*.py
python_classes = Test*
python_functions = test_*
addopts = -v --tb=short --disable-warnings
addopts = -v --tb=short --disable-warnings --import-mode=importlib
# Don't recursively search these files (prevents __init__.py loading)
norecursedirs = __pycache__ .git engines nodes web
# Override parent pytest.ini pythonpath - don't auto-add package root
+49
View File
@@ -0,0 +1,49 @@
from pathlib import Path
import sys
import pytest
REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT))
from utils.voice.character_logging import (
format_character_override_warning,
format_resolved_character_block,
format_resolved_character_text,
resolved_character_label,
resolved_voice_name,
)
@pytest.mark.unit
def test_resolved_voice_name_prefers_connected_character_name():
voice_ref = {
"character_name": "Tony",
"audio_path": r"J:\voices\other-name.mp3",
}
assert resolved_voice_name(voice_ref) == "Tony"
assert resolved_character_label("Bob", voice_ref) == "Tony"
@pytest.mark.unit
def test_resolved_voice_name_supports_audio_paths_and_legacy_tuples():
audio_path = Path("voices") / "Ana_1.wav"
assert resolved_voice_name({"audio_path": audio_path}) == "Ana_1"
assert resolved_voice_name((str(audio_path), "reference text")) == "Ana_1"
@pytest.mark.unit
def test_resolved_voice_logging_formats_are_shared():
voice_ref = {"character_name": "Tony"}
assert format_resolved_character_text("Bob", "Hello!", voice_ref) == "[Tony] Hello!"
assert format_resolved_character_block("Bob", "Hello!", voice_ref) == (
"============================================================\n"
"[Tony] Hello!\n"
"============================================================"
)
assert format_character_override_warning("Fish", "Speaker 2", "Bob") == (
"⚠️ Fish priority: Speaker 2 input overrides ['Bob'] alias"
)
+92
View File
@@ -0,0 +1,92 @@
from pathlib import Path
import sys
import pytest
import torch
REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT))
from utils.audio.cache import FishAudioS2CacheKeyGenerator
from engines.adapters.fish_audio_s2_adapter import FishAudioS2Adapter
@pytest.mark.unit
def test_quantization_variant_invalidates_audio_cache():
generator = FishAudioS2CacheKeyGenerator()
common = {
"text": "Test",
"audio_component": "voice-hash",
"reference_text": "Reference",
"seed": 1,
}
keys = {
generator.generate_cache_key(**common, model_variant=variant)
for variant in ("s2-pro", "s2-pro-fp8")
}
assert len(keys) == 2
@pytest.mark.unit
def test_load_quantization_invalidates_audio_cache():
generator = FishAudioS2CacheKeyGenerator()
common = {
"text": "Test",
"audio_component": "voice-hash",
"reference_text": "Reference",
"seed": 1,
"model_variant": "s2-pro",
}
keys = {
generator.generate_cache_key(**common, quantization=quantization)
for quantization in ("none", "bnb_int8", "bnb_nf4")
}
assert len(keys) == 3
@pytest.mark.unit
def test_speaker_mode_invalidates_audio_cache():
generator = FishAudioS2CacheKeyGenerator()
common = {
"text": "Test",
"audio_component": "voice-hash",
"reference_text": "Reference",
"seed": 1,
}
native = generator.generate_cache_key(
**common, multi_speaker_mode="Native Multi-Speaker"
)
custom = generator.generate_cache_key(
**common, multi_speaker_mode="Custom Character Switching"
)
assert native != custom
@pytest.mark.unit
def test_adapter_reuses_unchanged_custom_segment_and_invalidates_changed_text(monkeypatch):
adapter = FishAudioS2Adapter({"multi_speaker_mode": "Custom Character Switching"})
adapter.audio_cache.clear_cache()
calls = []
class FakeEngine:
def generate(self, **params):
calls.append(params["text"])
return torch.zeros(1, 32), adapter.SAMPLE_RATE
monkeypatch.setattr(adapter, "_engine", lambda: FakeEngine())
monkeypatch.setattr(adapter, "_reference", lambda _voice: ("voice.wav", "Reference", "voice-hash"))
voice = {"audio_path": "voice.wav", "reference_text": "Reference"}
adapter.generate_dialogue([(0, "unchanged")], [voice], seed=1, cache_character="speaker")
adapter.generate_dialogue([(0, "unchanged")], [voice], seed=1, cache_character="speaker")
adapter.generate_dialogue([(0, "changed")], [voice], seed=1, cache_character="speaker")
assert len(calls) == 2
adapter.audio_cache.clear_cache()
@@ -0,0 +1,77 @@
import importlib.util
from pathlib import Path
import sys
import pytest
REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT))
NODE_PATH = (
REPO_ROOT
/ "nodes"
/ "engines"
/ "fish_audio_s2_engine_node.py"
)
SPEC = importlib.util.spec_from_file_location("fish_audio_s2_engine_node_test_module", NODE_PATH)
NODE_MODULE = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(NODE_MODULE)
@pytest.mark.unit
def test_model_is_first_and_compile_defaults_enabled():
inputs = NODE_MODULE.FishAudioS2EngineNode.INPUT_TYPES()
assert list(inputs["required"])[0] == "model"
assert "s2-pro" in inputs["required"]["model"][0]
assert inputs["optional"]["compile"][1]["default"] is True
node = NODE_MODULE.FishAudioS2EngineNode()
(engine_config,) = node.create_engine_config()
assert engine_config["config"]["compile"] is True
@pytest.mark.unit
def test_bnb_quantization_preserves_compile():
node = NODE_MODULE.FishAudioS2EngineNode()
(engine_config,) = node.create_engine_config(
device="cuda",
temperature=0.8,
top_p=0.8,
repetition_penalty=1.1,
native_chunk_length=200,
max_new_tokens=1024,
context_length="8192",
compile=True,
model_variant="s2-pro",
quantization="bnb_int8",
)
config = engine_config["config"]
assert config["compile"] is True
assert config["quantization"] == "bnb_int8"
@pytest.mark.unit
def test_fp8_variant_forces_quantization_none():
node = NODE_MODULE.FishAudioS2EngineNode()
(engine_config,) = node.create_engine_config(
device="cuda",
temperature=0.8,
top_p=0.8,
repetition_penalty=1.1,
native_chunk_length=200,
max_new_tokens=1024,
context_length="8192",
compile=False,
model_variant="s2-pro-fp8",
quantization="bnb_nf4",
precision="float16",
)
config = engine_config["config"]
assert config["model_variant"] == "s2-pro-fp8"
assert config["quantization"] == "none"
assert config["precision"] == "bfloat16"
+28
View File
@@ -0,0 +1,28 @@
from pathlib import Path
import sys
import pytest
import torch
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
from utils.runtimes.fish_audio_s2_fp8 import FP8Linear
@pytest.mark.unit
def test_fp8_linear_uses_row_scales_for_bfloat16_compute():
layer = FP8Linear(2, 2)
layer.qweight = torch.tensor(
[[1.0, 2.0], [3.0, 4.0]],
dtype=torch.float8_e4m3fn,
)
layer.scale = torch.tensor([[0.5], [0.25]], dtype=torch.float32)
inputs = torch.tensor([[2.0, 1.0]], dtype=torch.bfloat16)
output = layer(inputs)
expected_weight = torch.tensor(
[[0.5, 1.0], [0.75, 1.0]],
dtype=torch.bfloat16,
)
assert torch.equal(output, torch.nn.functional.linear(inputs, expected_weight))
+636
View File
@@ -0,0 +1,636 @@
import importlib.util
from pathlib import Path
import sys
from types import SimpleNamespace
import pytest
import torch
REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT))
PROCESSOR_PATH = (
REPO_ROOT
/ "nodes"
/ "fish_audio_s2"
/ "fish_audio_s2_processor.py"
)
SPEC = importlib.util.spec_from_file_location("fish_audio_s2_processor_test_module", PROCESSOR_PATH)
PROCESSOR_MODULE = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(PROCESSOR_MODULE)
class FakeCharacterParser:
def set_available_characters(self, _characters):
pass
def set_character_language_default(self, _character, _language):
pass
def reset_session_cache(self):
pass
def parse_text_segments(self, text, engine_type=None):
characters = ["alice", "bob", "rick"] if text == "all" else ["bob", "rick"]
return [
SimpleNamespace(character=character, text=character.title(), parameters={}, language="en")
for character in characters
]
class CapturingAdapter:
def __init__(self):
self.calls = []
def update_config(self, _config):
pass
def generate_dialogue(self, turns, voice_refs, **_kwargs):
self.calls.append((list(turns), list(voice_refs)))
return torch.zeros(1, 16)
@pytest.mark.unit
def test_late_global_character_uses_local_reference_order(monkeypatch):
parser = FakeCharacterParser()
def parse_subtitle_three(text, engine_type=None):
if text == "subtitle3":
return [
SimpleNamespace(character="male_01", text="Bob line", parameters={}, language="en"),
SimpleNamespace(character="narrator", text="Narrator line", parameters={}, language="en"),
]
return [
SimpleNamespace(character="narrator", text="Narrator", parameters={}, language="en"),
SimpleNamespace(character="female_01", text="Alice", parameters={}, language="en"),
SimpleNamespace(character="clint_eastwood cc3 (enhanced2)", text="Cowboy", parameters={}, language="en"),
SimpleNamespace(character="male_01", text="Bob", parameters={}, language="en"),
]
parser.parse_text_segments = parse_subtitle_three
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
monkeypatch.setattr(
PROCESSOR_MODULE,
"get_character_mapping",
lambda *_args, **_kwargs: {"male_01": ("male_01.wav", "Bob reference")},
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"model_management",
SimpleNamespace(interrupt_processing=False),
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"voice_discovery",
SimpleNamespace(
get_character_aliases=lambda: {},
get_character_language_defaults=lambda: {},
),
)
adapter = CapturingAdapter()
processor = PROCESSOR_MODULE.FishAudioS2Processor(
adapter,
{
"speaker_references": [
{"character_name": "Ana"},
{"character_name": "Tony"},
],
"multi_speaker_mode": "Native Multi-Speaker",
"language_prompting": "Off",
},
)
reference_order = processor.get_character_order("all")
processor.process_text(
"subtitle3",
{"narrator": {"character_name": "Joe"}},
seed=1,
reference_order=reference_order,
show_text_logging=False,
)
turns, used_references = adapter.calls[0]
assert turns == [(0, "Bob line"), (1, "Narrator line")]
assert used_references == [
{"audio_path": "male_01.wav", "reference_text": "Bob reference"},
{"character_name": "Joe"},
]
@pytest.mark.unit
def test_srt_uses_global_reference_lookup_with_local_speaker_ids(monkeypatch):
parser = FakeCharacterParser()
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
monkeypatch.setattr(PROCESSOR_MODULE, "get_character_mapping", lambda *_args, **_kwargs: {})
monkeypatch.setattr(
PROCESSOR_MODULE,
"model_management",
SimpleNamespace(interrupt_processing=False),
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"voice_discovery",
SimpleNamespace(
get_character_aliases=lambda: {},
get_character_language_defaults=lambda: {},
),
)
adapter = CapturingAdapter()
references = [{"id": "bob"}, {"id": "rick"}]
processor = PROCESSOR_MODULE.FishAudioS2Processor(
adapter,
{"speaker_references": references},
)
reference_order = processor.get_character_order("all")
processor.process_text("subtitle", {}, seed=1, reference_order=reference_order)
turns, used_references = adapter.calls[0]
assert turns == [(0, "Bob"), (1, "Rick")]
assert [reference["id"] for reference in used_references] == ["bob", "rick"]
@pytest.mark.unit
def test_custom_switching_generates_each_segment_as_local_speaker_zero(monkeypatch):
parser = FakeCharacterParser()
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
monkeypatch.setattr(PROCESSOR_MODULE, "get_character_mapping", lambda *_args, **_kwargs: {})
monkeypatch.setattr(
PROCESSOR_MODULE,
"model_management",
SimpleNamespace(interrupt_processing=False),
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"voice_discovery",
SimpleNamespace(
get_character_aliases=lambda: {},
get_character_language_defaults=lambda: {},
),
)
adapter = CapturingAdapter()
references = [{"id": "bob"}, {"id": "rick"}]
processor = PROCESSOR_MODULE.FishAudioS2Processor(
adapter,
{
"speaker_references": references,
"multi_speaker_mode": "Custom Character Switching",
},
)
reference_order = processor.get_character_order("all")
records = processor.process_text(
"subtitle", {}, seed=1, reference_order=reference_order
)
assert [call[0] for call in adapter.calls] == [[(0, "Bob")], [(0, "Rick")]]
assert [[ref["id"] for ref in call[1]] for call in adapter.calls] == [["bob"], ["rick"]]
assert [record["text"] for record in records] == ["Bob", "Rick"]
@pytest.mark.unit
def test_language_prompting_prepends_shared_display_name(monkeypatch):
parser = FakeCharacterParser()
def parse_with_language(_text, engine_type=None):
assert engine_type == "fish_audio_s2"
return [
SimpleNamespace(character="alice", text="Hallo", parameters={}, language="de", explicit_language=False),
]
parser.parse_text_segments = parse_with_language
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
monkeypatch.setattr(PROCESSOR_MODULE, "get_character_mapping", lambda *_args, **_kwargs: {})
monkeypatch.setattr(
PROCESSOR_MODULE,
"model_management",
SimpleNamespace(interrupt_processing=False),
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"voice_discovery",
SimpleNamespace(
get_character_aliases=lambda: {},
get_character_language_defaults=lambda: {},
),
)
adapter = CapturingAdapter()
processor = PROCESSOR_MODULE.FishAudioS2Processor(
adapter,
{
"speaker_references": [{"id": "alice"}],
"multi_speaker_mode": "Custom Character Switching",
"language_prompting": "Auto Inline Tag",
},
)
processor.process_text("subtitle", {}, seed=1, reference_order=["alice"], show_text_logging=False)
turns, _used_references = adapter.calls[0]
assert turns == [(0, "<German> Hallo")]
@pytest.mark.unit
def test_explicit_english_language_prompt_is_preserved(monkeypatch):
parser = FakeCharacterParser()
def parse_with_explicit_english(_text, engine_type=None):
assert engine_type == "fish_audio_s2"
return [
SimpleNamespace(character="alice", text="Hello", parameters={}, language="en", explicit_language=True),
]
parser.parse_text_segments = parse_with_explicit_english
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
monkeypatch.setattr(PROCESSOR_MODULE, "get_character_mapping", lambda *_args, **_kwargs: {})
monkeypatch.setattr(
PROCESSOR_MODULE,
"model_management",
SimpleNamespace(interrupt_processing=False),
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"voice_discovery",
SimpleNamespace(
get_character_aliases=lambda: {},
get_character_language_defaults=lambda: {},
),
)
adapter = CapturingAdapter()
processor = PROCESSOR_MODULE.FishAudioS2Processor(
adapter,
{
"speaker_references": [{"id": "alice"}],
"multi_speaker_mode": "Custom Character Switching",
"language_prompting": "Auto Inline Tag",
},
)
processor.process_text("subtitle", {}, seed=1, reference_order=["alice"], show_text_logging=False)
turns, _used_references = adapter.calls[0]
assert turns == [(0, "<English> Hello")]
@pytest.mark.unit
def test_custom_override_logs_connected_voice_and_does_not_inherit_alias_language(monkeypatch, capsys):
parser = FakeCharacterParser()
def parse_bob(_text, engine_type=None):
assert engine_type == "fish_audio_s2"
return [
SimpleNamespace(character="bob", text="Bonjour", parameters={}, language="fr"),
]
parser.parse_text_segments = parse_bob
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
monkeypatch.setattr(
PROCESSOR_MODULE,
"get_character_mapping",
lambda *_args, **_kwargs: {"bob": ("bob.wav", "Bob ref")},
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"model_management",
SimpleNamespace(interrupt_processing=False),
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"voice_discovery",
SimpleNamespace(
get_character_aliases=lambda: {},
get_character_language_defaults=lambda: {},
),
)
adapter = CapturingAdapter()
processor = PROCESSOR_MODULE.FishAudioS2Processor(
adapter,
{
"speaker_references": [{"id": "tony", "character_name": "Tony"}],
"multi_speaker_mode": "Custom Character Switching",
"language_prompting": "Auto Inline Tag",
},
)
processor.process_text(
"subtitle",
{"narrator": {"audio_path": "joe.wav", "character_name": "Joe"}},
seed=1,
reference_order=["narrator", "bob"],
)
output = capsys.readouterr().out
assert "Speaker 2 input overrides ['bob'] alias" in output
assert "[Tony] Bonjour" in output
assert "<French>" not in output
@pytest.mark.unit
def test_single_character_native_call_uses_narrator_as_speaker_one_when_no_global_narrator_exists(monkeypatch):
parser = FakeCharacterParser()
def parse_single_bob(_text, engine_type=None):
assert engine_type == "fish_audio_s2"
return [
SimpleNamespace(character="bob", text="Bonjour", parameters={}, language="fr", explicit_language=False),
]
parser.parse_text_segments = parse_single_bob
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
monkeypatch.setattr(
PROCESSOR_MODULE,
"get_character_mapping",
lambda *_args, **_kwargs: {"bob": ("bob.wav", "Bob ref")},
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"model_management",
SimpleNamespace(interrupt_processing=False),
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"voice_discovery",
SimpleNamespace(
get_character_aliases=lambda: {},
get_character_language_defaults=lambda: {},
),
)
adapter = CapturingAdapter()
processor = PROCESSOR_MODULE.FishAudioS2Processor(
adapter,
{
"speaker_references": [{"id": "speaker1_female"}],
"multi_speaker_mode": "Native Multi-Speaker",
"language_prompting": "Off",
},
)
processor.process_text(
"subtitle",
{"narrator": {"id": "narrator_voice"}},
seed=1,
reference_order=["bob"],
show_text_logging=False,
)
turns, used_references = adapter.calls[0]
assert turns == [(0, "Bonjour")]
assert used_references == [{"id": "narrator_voice"}]
@pytest.mark.unit
def test_first_tagged_character_uses_speaker2_when_global_narrator_exists(monkeypatch):
parser = FakeCharacterParser()
def parse_single_bob(_text, engine_type=None):
assert engine_type == "fish_audio_s2"
return [
SimpleNamespace(character="bob", text="Bonjour", parameters={}, language="fr", explicit_language=False),
]
parser.parse_text_segments = parse_single_bob
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
monkeypatch.setattr(
PROCESSOR_MODULE,
"get_character_mapping",
lambda *_args, **_kwargs: {"bob": ("bob.wav", "Bob ref")},
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"model_management",
SimpleNamespace(interrupt_processing=False),
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"voice_discovery",
SimpleNamespace(
get_character_aliases=lambda: {},
get_character_language_defaults=lambda: {},
),
)
adapter = CapturingAdapter()
processor = PROCESSOR_MODULE.FishAudioS2Processor(
adapter,
{
"speaker_references": [{"id": "speaker2_voice"}],
"multi_speaker_mode": "Native Multi-Speaker",
"language_prompting": "Off",
},
)
processor.process_text(
"subtitle",
{"narrator": {"id": "narrator_voice"}},
seed=1,
reference_order=["narrator", "bob"],
show_text_logging=False,
)
turns, used_references = adapter.calls[0]
assert turns == [(0, "Bonjour")]
assert used_references == [{"id": "speaker2_voice"}]
@pytest.mark.unit
def test_second_character_uses_speaker2_override_when_narrator_not_reserved(monkeypatch):
parser = FakeCharacterParser()
def parse_bob_then_rick(_text, engine_type=None):
assert engine_type == "fish_audio_s2"
return [
SimpleNamespace(character="bob", text="Bonjour", parameters={}, language="fr", explicit_language=False),
SimpleNamespace(character="rick", text="Salut", parameters={}, language="fr", explicit_language=False),
]
parser.parse_text_segments = parse_bob_then_rick
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
monkeypatch.setattr(
PROCESSOR_MODULE,
"get_character_mapping",
lambda *_args, **_kwargs: {"bob": ("bob.wav", "Bob ref"), "rick": ("rick.wav", "Rick ref")},
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"model_management",
SimpleNamespace(interrupt_processing=False),
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"voice_discovery",
SimpleNamespace(
get_character_aliases=lambda: {},
get_character_language_defaults=lambda: {},
),
)
adapter = CapturingAdapter()
processor = PROCESSOR_MODULE.FishAudioS2Processor(
adapter,
{
"speaker_references": [{"id": "speaker2_voice"}],
"multi_speaker_mode": "Native Multi-Speaker",
"language_prompting": "Off",
},
)
processor.process_text(
"subtitle",
{"narrator": {"id": "narrator_voice"}},
seed=1,
reference_order=["bob", "rick"],
show_text_logging=False,
)
turns, used_references = adapter.calls[0]
assert turns == [(0, "Bonjour"), (1, "Salut")]
assert used_references == [{"id": "narrator_voice"}, {"id": "speaker2_voice"}]
@pytest.mark.unit
def test_native_override_uses_effective_speaker_default_language_for_auto_inline_tag(monkeypatch):
parser = FakeCharacterParser()
def parse_single_bob(_text, engine_type=None):
assert engine_type == "fish_audio_s2"
return [
SimpleNamespace(
character="male_01",
original_character="Bob",
text="Bonjour",
parameters={},
language="fr",
explicit_language=False,
),
]
parser.parse_text_segments = parse_single_bob
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
monkeypatch.setattr(
PROCESSOR_MODULE,
"get_character_mapping",
lambda *_args, **_kwargs: {"male_01": ("bob.wav", "Bob ref")},
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"get_character_default_language",
lambda character: {"anna": "de", "male_01": "fr"}.get(character.lower()),
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"model_management",
SimpleNamespace(interrupt_processing=False),
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"voice_discovery",
SimpleNamespace(
get_character_aliases=lambda: {},
get_character_language_defaults=lambda: {},
),
)
adapter = CapturingAdapter()
processor = PROCESSOR_MODULE.FishAudioS2Processor(
adapter,
{
"speaker_references": [{"character_name": "Anna"}],
"multi_speaker_mode": "Native Multi-Speaker",
"language_prompting": "Auto Inline Tag",
},
)
processor.process_text(
"subtitle",
{"narrator": {"character_name": "Joe"}},
seed=1,
reference_order=["narrator", "male_01"],
show_text_logging=False,
)
turns, used_references = adapter.calls[0]
assert turns == [(0, "<German> Bonjour")]
assert used_references == [{"character_name": "Anna"}]
@pytest.mark.unit
def test_native_override_with_unmapped_connected_voice_does_not_fall_back_to_alias_language(monkeypatch):
parser = FakeCharacterParser()
def parse_single_bob(_text, engine_type=None):
assert engine_type == "fish_audio_s2"
return [
SimpleNamespace(
character="male_01",
original_character="Bob",
text="Bonjour",
parameters={},
language="fr",
explicit_language=False,
),
]
parser.parse_text_segments = parse_single_bob
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
monkeypatch.setattr(
PROCESSOR_MODULE,
"get_character_mapping",
lambda *_args, **_kwargs: {"male_01": ("bob.wav", "Bob ref")},
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"get_character_default_language",
lambda character: None,
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"model_management",
SimpleNamespace(interrupt_processing=False),
)
monkeypatch.setattr(
PROCESSOR_MODULE,
"voice_discovery",
SimpleNamespace(
get_character_aliases=lambda: {},
get_character_language_defaults=lambda: {},
),
)
adapter = CapturingAdapter()
processor = PROCESSOR_MODULE.FishAudioS2Processor(
adapter,
{
"speaker_references": [{"character_name": "Tony"}],
"multi_speaker_mode": "Native Multi-Speaker",
"language_prompting": "Auto Inline Tag",
},
)
processor.process_text(
"subtitle",
{"narrator": {"character_name": "Joe"}},
seed=1,
reference_order=["narrator", "male_01"],
show_text_logging=False,
)
turns, used_references = adapter.calls[0]
assert turns == [(0, "Bonjour")]
assert used_references == [{"character_name": "Tony"}]
+20
View File
@@ -0,0 +1,20 @@
import pytest
from utils.text.fish_audio_s2_tags import translate_fish_s2_inline_tags
@pytest.mark.unit
@pytest.mark.parametrize("source, expected", [
("Hello <whisper>world", "Hello [whisper]world"),
("<professional broadcast tone>Hello", "[professional broadcast tone]Hello"),
("<pitch up>now <angry>stop", "[pitch up]now [angry]stop"),
("[Alice] Hello <excited>", "[Alice] Hello [excited]"),
("Keep <unclosed", "Keep <unclosed"),
])
def test_translates_free_form_fish_tags_at_boundary(source, expected):
assert translate_fish_s2_inline_tags(source) == expected
@pytest.mark.unit
def test_normalizes_only_internal_whitespace():
assert translate_fish_s2_inline_tags("< whisper in small voice >") == "[whisper in small voice]"
+47
View File
@@ -0,0 +1,47 @@
import importlib.util
from pathlib import Path
from types import SimpleNamespace
import pytest
REPO_ROOT = Path(__file__).resolve().parents[2]
WORKER_PATH = REPO_ROOT / "utils" / "runtimes" / "workers" / "fish_audio_s2_worker.py"
SPEC = importlib.util.spec_from_file_location("fish_audio_s2_worker_test_module", WORKER_PATH)
WORKER_MODULE = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(WORKER_MODULE)
class FakeReferenceLoader:
def __init__(self):
self.ref_by_hash = {}
self.encoded = []
def encode_reference(self, reference_audio, enable_reference_audio):
assert enable_reference_audio is True
token = f"encoded-{len(self.encoded)}"
self.encoded.append(reference_audio)
return token
@pytest.mark.unit
def test_reference_cache_reuses_audio_but_not_previous_speaker_tag():
loader = FakeReferenceLoader()
first_tokens, first_texts = WORKER_MODULE._load_references_preserving_current_speaker_tags(
loader,
[SimpleNamespace(audio=b"joe", text="<|speaker:0|>Joe reference")],
"on",
)
second_tokens, second_texts = WORKER_MODULE._load_references_preserving_current_speaker_tags(
loader,
[
SimpleNamespace(audio=b"bob", text="<|speaker:0|>Bob reference"),
SimpleNamespace(audio=b"joe", text="<|speaker:1|>Joe reference"),
],
"on",
)
assert first_texts == ["<|speaker:0|>Joe reference"]
assert second_tokens == ["encoded-1", first_tokens[0]]
assert second_texts == ["<|speaker:0|>Bob reference", "<|speaker:1|>Joe reference"]
+43
View File
@@ -0,0 +1,43 @@
from pathlib import Path
import sys
import pytest
REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT))
from utils.runtimes.launcher import IsolatedRuntimeLauncher
from utils.runtimes.profiles import RuntimeProfile
@pytest.mark.unit
@pytest.mark.parametrize("value", ["None", "-1", "4294967296", "not-a-seed"])
def test_build_env_removes_invalid_pythonhashseed(monkeypatch, value):
monkeypatch.setenv("PYTHONHASHSEED", value)
monkeypatch.setattr(
IsolatedRuntimeLauncher,
"_augment_windows_toolchain_env",
staticmethod(lambda env: env),
)
profile = RuntimeProfile(name="test", engine_names=[])
env = IsolatedRuntimeLauncher().build_env(profile)
assert "PYTHONHASHSEED" not in env
@pytest.mark.unit
@pytest.mark.parametrize("value", ["random", "0", "123", "4294967295"])
def test_build_env_preserves_valid_pythonhashseed(monkeypatch, value):
monkeypatch.setenv("PYTHONHASHSEED", value)
monkeypatch.setattr(
IsolatedRuntimeLauncher,
"_augment_windows_toolchain_env",
staticmethod(lambda env: env),
)
profile = RuntimeProfile(name="test", engine_names=[])
env = IsolatedRuntimeLauncher().build_env(profile)
assert env["PYTHONHASHSEED"] == value
@@ -0,0 +1,199 @@
import importlib.util
from pathlib import Path
import sys
from types import SimpleNamespace
import pytest
REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT))
ADAPTER_PATH = REPO_ROOT / "engines" / "adapters" / "vibevoice_adapter.py"
SPEC = importlib.util.spec_from_file_location("vibevoice_adapter_test_module", ADAPTER_PATH)
ADAPTER_MODULE = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(ADAPTER_MODULE)
class FakeVibeVoiceEngine:
def __init__(self):
self.calls = []
def _prepare_voice_samples(self, speaker_voices):
return speaker_voices
def generate_speech(self, **kwargs):
self.calls.append(kwargs)
return {"waveform": None}
@pytest.mark.unit
def test_native_multispeaker_remaps_global_slots_to_local_sequence(monkeypatch):
fake_engine = FakeVibeVoiceEngine()
monkeypatch.setattr(
ADAPTER_MODULE.VibeVoiceEngineAdapter,
"vibevoice_engine",
property(lambda self: fake_engine),
)
monkeypatch.setattr(
ADAPTER_MODULE.VibeVoiceEngineAdapter,
"current_model",
property(lambda self: object()),
)
monkeypatch.setattr(
ADAPTER_MODULE.VibeVoiceEngineAdapter,
"current_processor",
property(lambda self: object()),
)
adapter = ADAPTER_MODULE.VibeVoiceEngineAdapter(SimpleNamespace(), {})
narrator_voice = {"audio_path": "narrator.wav"}
bob_alias_voice = {"audio_path": "bob_alias.wav"}
speaker3_override = {"audio_path": "speaker3.wav"}
adapter._generate_native_multispeaker(
segments=[("Bob", "Hi there."), ("narrator", "Welcome back.")],
voice_mapping={"narrator": narrator_voice, "Bob": bob_alias_voice},
params={"speaker3_voice": speaker3_override, "enable_cache": False},
global_char_to_speaker={"narrator": 1, "Alice": 2, "Bob": 3},
)
call = fake_engine.calls[0]
assert call["text"] == "Speaker 1: Hi there.\nSpeaker 2: Welcome back."
assert call["voice_samples"] == [speaker3_override, narrator_voice]
@pytest.mark.unit
def test_native_multispeaker_does_not_fake_speaker1_from_character_voice(monkeypatch):
fake_engine = FakeVibeVoiceEngine()
monkeypatch.setattr(
ADAPTER_MODULE.VibeVoiceEngineAdapter,
"vibevoice_engine",
property(lambda self: fake_engine),
)
monkeypatch.setattr(
ADAPTER_MODULE.VibeVoiceEngineAdapter,
"current_model",
property(lambda self: object()),
)
monkeypatch.setattr(
ADAPTER_MODULE.VibeVoiceEngineAdapter,
"current_processor",
property(lambda self: object()),
)
adapter = ADAPTER_MODULE.VibeVoiceEngineAdapter(SimpleNamespace(), {})
alice_voice = {"audio_path": "alice.wav"}
clint_voice = {"audio_path": "clint.wav"}
adapter._generate_native_multispeaker(
segments=[("female_01", "Hi there."), ("clint_eastwood cc3 (enhanced2)", "Hello there.")],
voice_mapping={
"female_01": alice_voice,
"clint_eastwood cc3 (enhanced2)": clint_voice,
},
params={"enable_cache": False},
global_char_to_speaker={
"clint_eastwood cc3 (enhanced2)": 1,
"female_01": 2,
"male_01": 3,
},
)
call = fake_engine.calls[0]
assert call["text"] == "Speaker 1: Hi there.\nSpeaker 2: Hello there."
assert call["voice_samples"] == [alice_voice, clint_voice]
@pytest.mark.unit
def test_native_multispeaker_narrator_input_does_not_override_tagged_global_speaker1(monkeypatch):
fake_engine = FakeVibeVoiceEngine()
monkeypatch.setattr(
ADAPTER_MODULE.VibeVoiceEngineAdapter,
"vibevoice_engine",
property(lambda self: fake_engine),
)
monkeypatch.setattr(
ADAPTER_MODULE.VibeVoiceEngineAdapter,
"current_model",
property(lambda self: object()),
)
monkeypatch.setattr(
ADAPTER_MODULE.VibeVoiceEngineAdapter,
"current_processor",
property(lambda self: object()),
)
adapter = ADAPTER_MODULE.VibeVoiceEngineAdapter(SimpleNamespace(), {})
narrator_voice = {"audio_path": "joe.wav"}
bob_voice = {"audio_path": "bob.wav"}
alice_voice = {"audio_path": "alice.wav"}
adapter._generate_native_multispeaker(
segments=[("male_01", "Hello."), ("female_01", "Hi there.")],
voice_mapping={
"narrator": narrator_voice,
"male_01": bob_voice,
"female_01": alice_voice,
},
params={"enable_cache": False},
global_char_to_speaker={
"narrator": 1,
"female_01": 3,
"male_01": 4,
},
)
call = fake_engine.calls[0]
assert call["text"] == "Speaker 1: Hello.\nSpeaker 2: Hi there."
assert call["voice_samples"] == [bob_voice, alice_voice]
@pytest.mark.unit
def test_native_multispeaker_narrator_input_overrides_first_character_when_no_narrator_exists(monkeypatch):
fake_engine = FakeVibeVoiceEngine()
monkeypatch.setattr(
ADAPTER_MODULE.VibeVoiceEngineAdapter,
"vibevoice_engine",
property(lambda self: fake_engine),
)
monkeypatch.setattr(
ADAPTER_MODULE.VibeVoiceEngineAdapter,
"current_model",
property(lambda self: object()),
)
monkeypatch.setattr(
ADAPTER_MODULE.VibeVoiceEngineAdapter,
"current_processor",
property(lambda self: object()),
)
adapter = ADAPTER_MODULE.VibeVoiceEngineAdapter(SimpleNamespace(), {})
narrator_voice = {"audio_path": "joe.wav"}
bob_voice = {"audio_path": "bob.wav"}
alice_voice = {"audio_path": "alice.wav"}
adapter._generate_native_multispeaker(
segments=[("male_01", "Hello."), ("female_01", "Hi there.")],
voice_mapping={
"narrator": narrator_voice,
"male_01": bob_voice,
"female_01": alice_voice,
},
params={"enable_cache": False},
global_char_to_speaker={
"male_01": 1,
"female_01": 2,
},
)
call = fake_engine.calls[0]
assert call["text"] == "Speaker 1: Hello.\nSpeaker 2: Hi there."
assert call["voice_samples"] == [narrator_voice, alice_voice]

Some files were not shown because too many files have changed in this diff Show More