Compare commits
47
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7c64fedede | ||
|
|
6da635cf74 | ||
|
|
8bfd6512de | ||
|
|
ffa3f1eded | ||
|
|
87def26983 | ||
|
|
057a9ef638 | ||
|
|
df3a4cb2cf | ||
|
|
06003e0b3a | ||
|
|
068c3f1f1e | ||
|
|
46c051477d | ||
|
|
52b22c0b8d | ||
|
|
b31b31b89f | ||
|
|
9757092aae | ||
|
|
47a8c7c691 | ||
|
|
1033eb78b9 | ||
|
|
1f2703a3a3 | ||
|
|
af77ff8e0b | ||
|
|
8b199980dc | ||
|
|
26fa2ddf92 | ||
|
|
282616ebe1 | ||
|
|
5735616794 | ||
|
|
6ce60f31a2 | ||
|
|
102f114ae6 | ||
|
|
4494381176 | ||
|
|
84e82b7d31 | ||
|
|
e54b6d9aa5 | ||
|
|
984f0e9fa5 | ||
|
|
1218e01683 | ||
|
|
9081d3c22a | ||
|
|
09df168119 | ||
|
|
b00b6098c9 | ||
|
|
a41ac5d89e | ||
|
|
ed5e55a5fc | ||
|
|
a0b58e42f7 | ||
|
|
44d164c2b7 | ||
|
|
14fcba3bed | ||
|
|
36ed7d694a | ||
|
|
24e6575fd4 | ||
|
|
fcff4ccf45 | ||
|
|
f7abd5a891 | ||
|
|
1291346dda | ||
|
|
552a67d209 | ||
|
|
35e0b27913 | ||
|
|
1e92a9f4b1 | ||
|
|
5c346693e3 | ||
|
|
3717fc0cbe | ||
|
|
af13209b47 |
@@ -5,6 +5,93 @@ All notable changes to this project will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [5.4.4] - 2026-07-11
|
||||
|
||||
### Added
|
||||
|
||||
- Add faster character, parameter, preset, and emotion swapping
|
||||
- Support combining IndexTTS audio emotion references with vector or text emotions
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve Multiline TTS Tag Editor emotion switching
|
||||
- Improve long inline tag wrapping inside the editor
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix incorrect tag detection and extra blank lines when pressing Enter
|
||||
## [5.4.3] - 2026-07-10
|
||||
|
||||
### Added
|
||||
|
||||
- Keep emotion radar controls aligned at the original node size
|
||||
- Prevent the radar chart from overflowing its node
|
||||
- Make emotion vector export selectable and downloadable on demand
|
||||
- Show confirmation after importing emotion vectors
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix IndexTTS-2 emotion vector controls
|
||||
## [5.4.2] - 2026-07-10
|
||||
|
||||
### Added
|
||||
|
||||
- Corrects pitch-index typing for RVC voice conversion on MPS devices.
|
||||
- Preserves the model's expected precision for phone features across CPU, CUDA, and MPS.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Tentative fix for RVC Voice Changer on Apple Silicon
|
||||
## [5.4.1] - 2026-07-10
|
||||
|
||||
### Added
|
||||
|
||||
- Prevents MelBand vocal removal from failing during sample-rate conversion on some Python 3.13 environments.
|
||||
- Corrects the documented location of the version bump instructions.
|
||||
- Makes future version bumps reliable on Windows installations with non-UTF-8 locales.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Tentative fix for MelBand audio separation failures
|
||||
## [5.4.0] - 2026-07-10
|
||||
|
||||
### Added
|
||||
|
||||
- Add Fish Audio S2 Pro multilingual voice generation
|
||||
- Add Fish Audio S2 Pro voice cloning with reference audio and transcript support
|
||||
- Add native multi-speaker dialogue and independent character-segment generation
|
||||
- Add free-form inline speech instructions and automatic language prompting
|
||||
- Add long-form generation, SRT integration, compilation, caching, and optional quantization
|
||||
## [5.3.0] - 2026-06-23
|
||||
|
||||
### Added
|
||||
|
||||
- Add Granite ASR 4.1 diarization and timestamp improvements
|
||||
- Add Granite ASR plus-model speaker diarization with suite-native [Speaker] output for downstream TTS workflows
|
||||
- Clarify Granite ASR 4.1 model differences, diarization support, and Japanese support in the README and generated docs
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve test runner portability and Granite ASR regression coverage for contributors
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix Granite ASR plus native word timestamps cutting off longer transcripts in timestamp mode
|
||||
## [5.2.0] - 2026-06-23
|
||||
|
||||
### Added
|
||||
|
||||
- Add OmniVoice engine with native timing-aware TTS controls
|
||||
- Add OmniVoice text and SRT generation with native duration targeting for more precise subtitle timing
|
||||
- Add Visual Tag Builder workflow example and documentation for OmniVoice prompt building
|
||||
|
||||
### Changed
|
||||
|
||||
- Improve interruption handling so stopped generation exits cleanly across unified TTS workflows
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fix Qwen3 Voice Designer metadata links and test environment path handling
|
||||
## [5.1.1] - 2026-06-17
|
||||
|
||||
### Added
|
||||
|
||||
@@ -27,24 +27,26 @@ Third-Party Model Licenses
|
||||
|
||||
The project code is MIT. Model weights carry their own licenses:
|
||||
|
||||
──────────────── ──────────────────────────────────────────────────────── ──────────────
|
||||
Engine License Commercial Use
|
||||
──────────────── ──────────────────────────────────────────────────────── ──────────────
|
||||
F5-TTS CC-BY-NC-4.0 No
|
||||
ChatterBox MIT Yes
|
||||
ChatterBox 23L MIT Yes
|
||||
VibeVoice MIT (research-only per model card) No
|
||||
Higgs Audio 2 Boson Higgs Audio 2 Community License Conditional
|
||||
Higgs Audio v3 Boson Higgs Audio v3 Research and Non-Commercial License No
|
||||
IndexTTS-2 bilibili Model Use License Conditional
|
||||
CosyVoice3 Apache-2.0 Yes
|
||||
Qwen3-TTS Apache-2.0 Yes
|
||||
Granite ASR Apache-2.0 Yes
|
||||
Step Audio EditX Apache-2.0 (verify before commercial use) Conditional
|
||||
Echo-TTS CC-BY-NC-SA-4.0 No
|
||||
Dots TTS Apache-2.0 Yes
|
||||
MOSS-TTS Apache-2.0 Yes
|
||||
RVC MIT (framework); community models vary Varies
|
||||
──────────────── ──────────────────────────────────────────────────────── ──────────────
|
||||
───────────────── ──────────────────────────────────────────────────────── ──────────────
|
||||
Engine License Commercial Use
|
||||
───────────────── ──────────────────────────────────────────────────────── ──────────────
|
||||
F5-TTS CC-BY-NC-4.0 No
|
||||
ChatterBox MIT Yes
|
||||
ChatterBox 23L MIT Yes
|
||||
VibeVoice MIT (research-only per model card) No
|
||||
Higgs Audio 2 Boson Higgs Audio 2 Community License Conditional
|
||||
Higgs Audio v3 Boson Higgs Audio v3 Research and Non-Commercial License No
|
||||
IndexTTS-2 bilibili Model Use License Conditional
|
||||
CosyVoice3 Apache-2.0 Yes
|
||||
Qwen3-TTS Apache-2.0 Yes
|
||||
Granite ASR Apache-2.0 Yes
|
||||
Step Audio EditX Apache-2.0 (verify before commercial use) Conditional
|
||||
Echo-TTS CC-BY-NC-SA-4.0 No
|
||||
Fish Audio S2 Pro Fish Audio Research License No
|
||||
Dots TTS Apache-2.0 Yes
|
||||
OmniVoice Apache-2.0 Yes
|
||||
MOSS-TTS Apache-2.0 Yes
|
||||
RVC MIT (framework); community models vary Varies
|
||||
───────────────── ──────────────────────────────────────────────────────── ──────────────
|
||||
|
||||
Users are responsible for complying with respective model licenses.
|
||||
|
||||
+13
-9
@@ -25,7 +25,7 @@
|
||||
|
||||
## Engines
|
||||
|
||||
14 engines follow the pattern above:
|
||||
15 engines follow the pattern above:
|
||||
|
||||
| Engine | Adapter | Processor | SRT Processor | Engine Node |
|
||||
|--------|---------|-----------|---------------|-------------|
|
||||
@@ -42,26 +42,29 @@
|
||||
| MOSS-TTS | `moss_tts_adapter.py` | `nodes/moss_tts/moss_tts_processor.py` | `moss_tts_srt_processor.py` | `moss_tts_engine_node.py` |
|
||||
| Granite ASR | `asr_granite_adapter.py` | — | — | `granite_asr_engine_node.py` |
|
||||
| Echo-TTS | `echo_tts_adapter.py` | `nodes/echo_tts/echo_tts_processor.py` | `echo_tts_srt_processor.py` | `echo_tts_engine_node.py` |
|
||||
| OmniVoice | `omnivoice_adapter.py` | `nodes/omnivoice/omnivoice_processor.py` | `omnivoice_srt_processor.py` | `omnivoice_engine_node.py` |
|
||||
| RVC | — | `engines/rvc/` | — | `rvc_engine_node.py` |
|
||||
|
||||
**Engine implementations live in:**
|
||||
- `engines/chatterbox/`, `engines/chatterbox_official_23lang/`, `engines/f5tts/`, `engines/higgs_audio/`, `engines/higgs_audio_v3/`, `engines/vibevoice_engine/`, `engines/step_audio_editx/`, `engines/cosyvoice/`, `engines/qwen3_tts/`, `engines/qwen3_asr/`, `engines/moss_tts/`, `engines/granite_asr/`, `engines/rvc/`
|
||||
- `engines/chatterbox/`, `engines/chatterbox_official_23lang/`, `engines/f5tts/`, `engines/higgs_audio/`, `engines/higgs_audio_v3/`, `engines/vibevoice_engine/`, `engines/step_audio_editx/`, `engines/cosyvoice/`, `engines/qwen3_tts/`, `engines/qwen3_asr/`, `engines/moss_tts/`, `engines/granite_asr/`, `engines/echo_tts/`, `engines/omnivoice/`, `engines/rvc/`
|
||||
|
||||
## Documentation Files
|
||||
|
||||
**README.md** - Main project docs, installation, features overview
|
||||
**CLAUDE.md** - Dev guidelines for Claude Code
|
||||
**CHANGELOG.md** - Full version history
|
||||
**README.md** - Main project docs, installation, features overview
|
||||
**CLAUDE.md** - Dev guidelines for Claude Code
|
||||
**CHANGELOG.md** - Full version history
|
||||
**docs/BUMP_SCRIPT_INSTRUCTIONS.md** - Version bump process
|
||||
|
||||
### User Docs (`docs/`)
|
||||
- `CHARACTER_SWITCHING_GUIDE.md` - [CharacterName] tag system
|
||||
- `PARAMETER_SWITCHING_GUIDE.md` - Per-segment parameter override syntax
|
||||
- `INLINE_EDIT_TAGS_USER_GUIDE.md` - Step Audio EditX inline tags
|
||||
- `HIGGS_AUDIO_V3_INLINE_TAGS.md` - Higgs Audio v3 native paralinguistic tags
|
||||
- `OMNIVOICE_TAGS_GUIDE.md` - OmniVoice native non-verbal tags and pronunciation overrides
|
||||
- `MOSS_TTS_PROMPT_FIELDS_GUIDE.md` - Official MOSS whole-segment prompt fields and inline `<>` translation limits
|
||||
- `COSYVOICE3_TAGS_GUIDE.md` - CosyVoice3 native paralinguistic tags
|
||||
- `CHATTERBOX_V2_SPECIAL_TOKENS.md` - ChatterBox v2 emotion tokens
|
||||
- `IndexTTS2_Emotion_Control_Guide.md` - IndexTTS-2 emotion vectors
|
||||
- `IndexTTS2_Emotion_Control_Guide.md` - IndexTTS-2 vector, text, audio, and blended emotion controls
|
||||
- `VOCAL_REMOVAL_GUIDE.md` - Vocal separation guide
|
||||
- `qwen3_tts_optimizations.md` - Qwen3-TTS torch.compile setup
|
||||
- `MODEL_DOWNLOAD_SOURCES.md` - All HF repo links (auto-generated)
|
||||
@@ -72,7 +75,6 @@
|
||||
### Dev Docs (`docs/Dev reports/`)
|
||||
- `tts_audio_suite_engines.yaml` - **Source of truth** for all engine metadata
|
||||
- `tts_audio_suite_aux_models.yaml` - **Source of truth** for helper/post-process model metadata
|
||||
- `BUMP_SCRIPT_INSTRUCTIONS.md` - Version bump process
|
||||
- `SRT_IMPLEMENTATION.md` - SRT timing technical details
|
||||
- `ISOLATED_RUNTIMES_PLAN.md` - original runtime isolation plan and scope
|
||||
- `TRANSFORMERS_5_QWEN3_TTS_REPORT.md` - why Qwen3-TTS moved to shared legacy T4 runtime
|
||||
@@ -104,6 +106,7 @@
|
||||
|
||||
### Shared / Special Nodes
|
||||
- `nodes/shared/character_voices_node.py` - Character voice management (NARRATOR_VOICE output)
|
||||
- `nodes/omnivoice/omnivoice_instruction_builder_node.py` - OmniVoice voice-design instruction helper with custom visual builder UI
|
||||
- `nodes/text/phoneme_text_normalizer_node.py` - Multilingual text preprocessing
|
||||
- `nodes/text/asr_punctuation_truecase_node.py` - Standalone punctuation / truecase cleanup for raw ASR text
|
||||
- `nodes/subtitles/text_to_srt_builder_node.py` - Build SRT from transcript text plus timing data, or estimate timings from plain text
|
||||
@@ -169,8 +172,9 @@
|
||||
- `reporting.py` - Timing report generation
|
||||
|
||||
### Other Utils
|
||||
- `utils/voice/discovery.py` - Voice file discovery with multi-path fallback
|
||||
- `utils/downloads/unified_downloader.py` - Centralized HF download system
|
||||
- `utils/voice/discovery.py` - Voice file discovery with multi-path fallback
|
||||
- `utils/voice/character_logging.py` - Shared resolved voice labels and boxed prompt previews
|
||||
- `utils/downloads/unified_downloader.py` - Centralized HF download system
|
||||
- `utils/compatibility/transformers_patches.py` - transformers version compatibility patches
|
||||
- `utils/compatibility/numba_compat.py` - Numba/Librosa Python 3.13+ compatibility
|
||||
- `utils/ffmpeg_utils.py` - FFmpeg with graceful fallback
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
[![Dynamic TOML Badge][version-shield]][version-url]
|
||||
[](https://ko-fi.com/diogogo)
|
||||
|
||||
# TTS Audio Suite v5.1.1
|
||||
# TTS Audio Suite v5.4.4
|
||||
|
||||
[](https://ko-fi.com/diogogo)
|
||||
|
||||
@@ -23,7 +23,7 @@ Subtitle workflows are still a core focus: the suite can transcribe to SRT, rebu
|
||||
|
||||
<!-- ENGINE_COMPARISON_START -->
|
||||
|
||||
## Quick Engine Comparison — 15 Engines
|
||||
## Quick Engine Comparison — 17 Engines
|
||||
|
||||
| Engine | Languages | Size | Key Features |
|
||||
|--------|-----------|------|--------------|
|
||||
@@ -32,14 +32,16 @@ Subtitle workflows are still a core focus: the suite can transcribe to SRT, rebu
|
||||
| **ChatterBox 23L** | 🌐 24 languages | ~4.3GB | 24 languages in single model, emotion tokens (v2 - doesn't work) |
|
||||
| **VibeVoice** | 🇺🇸🇨🇳🇩🇪🇪🇸🇫🇷🇮🇹 +21 | 5.4GB / 18GB | 90-min long-form, Native 4-speaker (Base models) |
|
||||
| **Higgs Audio 2** | 🇺🇸🇨🇳🇩🇪🇪🇸🇰🇷 | ~9GB | 3 multi-speaker, CUDA graphs (55+ tokens/sec) |
|
||||
| **Higgs Audio v3** | 🇿🇦🇦🇱🇺🇸🇪🇸🇮🇳🇨🇳 +96 | ~8GB | Native inline emotion/style/prosody/SFX tags, Zero-shot voice cloning |
|
||||
| **Higgs Audio v3** | 🌐 100+ languages | ~8GB | Native inline emotion/style/prosody/SFX tags, Zero-shot voice cloning |
|
||||
| **IndexTTS-2** | 🇺🇸🇨🇳🇯🇵 | ~4.7GB | Emotion Control: 8 vectors, Text as reference |
|
||||
| **CosyVoice3** | 🇺🇸🇨🇳🇯🇵🇰🇷 | ~5.4GB | Paralinguistic tags |
|
||||
| **Qwen3-TTS** | 🇺🇸🇨🇳🇩🇪🇪🇸🇫🇷🇮🇹 +4 | ~3-6GB | Voice design, ASR (Automatic Speech Recognition) |
|
||||
| **Granite ASR** | 🇺🇸🇩🇪🇪🇸🇫🇷🇯🇵🇵🇹 | ~4.6GB | ASR (Automatic Speech Recognition), Native speaker attribution / diarization (plus model variant) |
|
||||
| **Step Audio EditX** | 🇺🇸🇨🇳🇯🇵🇰🇷 | ~7GB | Second Pass Speech Editing Node: 14 emotions, 32 speaking styles |
|
||||
| **Echo-TTS** | 🇺🇸 | ~5.3GB + ~1.8GB | Diffusion-based (~30s best), Force Speaker KV (speaker drift control) |
|
||||
| **Fish Audio S2 Pro** | 🌐 80+ languages | ~10.3GB / ~8.0GB | Free-form sub-word emotion/prosody tags, Zero-shot voice cloning and 80+ languages |
|
||||
| **Dots TTS** | 🇺🇸🇨🇳🇩🇪🇪🇸🇫🇷🇮🇹 +13 | ~6GB | Official auto language detect / language control, SOAR and MeanFlow distilled variants |
|
||||
| **OmniVoice** | 🌐 600+ languages | ~3.7GB | 600+ language support, Instruction-based voice design |
|
||||
| **MOSS-TTS** | 🇺🇸🇨🇳🇩🇪🇪🇸🇫🇷🇮🇹 +10 | ~8.5GB tokenizer + ~6.1GB/17GB/18GB model | 20-language generation, Long-form generation (TTSD/Delay) |
|
||||
| **RVC** | 🌐 Any | 100-300MB | Real-time VC, Integrated training workflow |
|
||||
|
||||
@@ -102,11 +104,17 @@ Builder English TTS TTS + ASR + VoiceDesign
|
||||
|
|
||||
|─── 🎓 Training Support Era 🧱 Runtime Isolation T5 Era
|
||||
▼ |
|
||||
v4.25 ──────────────► v4.26 ────────────► v5.00 ─────►
|
||||
Apr 26 May 26 Jun 26
|
||||
│ │ │
|
||||
RVC MOSS-TTS Transformers 5
|
||||
Model Training Higgs Audio v3 TTS
|
||||
v4.25 ──────────────► v4.26 ────────────► v5.00 ───────────────┐
|
||||
Apr 26 May 26 Jun 26 │
|
||||
│ │ │ │
|
||||
RVC MOSS-TTS Transformers 5 │
|
||||
Model Training Higgs Audio v3 TTS │
|
||||
│
|
||||
▼
|
||||
◄──────── v5.2 ◄─────────────── v5.1 ◄───────────┘
|
||||
Mar 26 Jan 26
|
||||
│ │
|
||||
OmniVoice TTS Dots TTS
|
||||
|
||||
```
|
||||
|
||||
@@ -143,6 +151,7 @@ Start with the **[New Engine Guide Hub](docs/New%20Engines%20Guides/README.md)**
|
||||
- [Step Audio EditX - LLM Audio Editing](#step-audio-editx---llm-audio-editing)
|
||||
- [CosyVoice3 Multilingual Voice Cloning](#cosyvoice3-multilingual-voice-cloning)
|
||||
- [Qwen3-TTS - 4 Model Types with Text-to-Voice Design](#qwen3-tts---4-model-types-with-text-to-voice-design)
|
||||
- [OmniVoice + Visual Tag Builder](#omnivoice--visual-tag-builder)
|
||||
- [Echo-TTS Voice Cloning](#echo-tts-voice-cloning)
|
||||
- [Phoneme Text Normalizer](#phoneme-text-normalizer)
|
||||
- [Multiline TTS Tag Editor and Per-Segment Parameter Switching](#multiline-tts-tag-editor-and-per-segment-parameter-switching)
|
||||
@@ -188,7 +197,8 @@ Start with the **[New Engine Guide Hub](docs/New%20Engines%20Guides/README.md)**
|
||||
- 🎓 **Integrated Model Training**
|
||||
- 🎨 **Audio Post-Processing** → **[📖 Inline Edit Tags Guide](docs/INLINE_EDIT_TAGS_USER_GUIDE.md)**
|
||||
- 🎭 **Character and Language Switching** → **[📖 Character Switching Guide](docs/CHARACTER_SWITCHING_GUIDE.md)**
|
||||
- 🏷️ **Multiline TTS Tag Editor and Per-Segment Parameter Switching** → **[📖 Per-Segment Parameters](docs/PARAMETER_SWITCHING_GUIDE.md)** | **[📖 Multiline Tag Editor Guide](docs/MULTILINE_TTS_TAG_EDITOR_GUIDE.md)**
|
||||
- 📐 **Visual Tag Builder** → Preset-driven visual tag and attribute assembly for OmniVoice and other tag-based text workflows
|
||||
- 🏷️ **Multiline TTS Tag Editor and Per-Segment Parameter Switching** → **[📖 Per-Segment Parameters](docs/PARAMETER_SWITCHING_GUIDE.md)** | **[📖 Multiline Tag Editor Guide](docs/MULTILINE_TTS_TAG_EDITOR_GUIDE.md)** | **[📖 OmniVoice Tags Guide](docs/OMNIVOICE_TAGS_GUIDE.md)**
|
||||
- 📝 **Intelligent Text Chunking** → **[📖 Text Chunking Guide](docs/TEXT_CHUNKING_GUIDE.md)**
|
||||
- 🤐 **Vocal/Noise Removal** → **[📖 Complete Guide](docs/VOCAL_REMOVAL_GUIDE.md)**
|
||||
- 🌊 **Audio Wave Analyzer** → **[📖 Complete Guide](docs/🌊_Audio_Wave_Analyzer-Complete_User_Guide.md)**
|
||||
@@ -700,19 +710,19 @@ Both versions fully support character switching, language switching, and pause t
|
||||
<details>
|
||||
<summary><h3>IndexTTS-2 With Emotion Control</h3></summary>
|
||||
|
||||
**NEW in v4.9.0**: Revolutionary IndexTTS-2 engine with advanced emotion control and unified emotion architecture!
|
||||
**NEW in v4.9.0**: Revolutionary IndexTTS-2 engine with advanced emotion control and dual-source emotion blending!
|
||||
|
||||
* **Unified Emotion Control**: Single `emotion_control` input supporting multiple emotion methods with intelligent priority system
|
||||
* **Separate Emotion Inputs**: Connect vectors or Qwen text emotion to `emotion_control` and audio references to `emotion_audio`; both can be used together
|
||||
* **Dynamic Text Emotion**: AI-powered QwenEmotion analysis with dynamic `{seg}` template processing for contextual per-segment emotions
|
||||
* **Direct Audio Reference**: Use any audio file as emotion reference for natural emotional expression
|
||||
* **Character Voices Integration**: Use Character Voices `opt_narrator` output as emotion reference with automatic detection
|
||||
* **Direct Audio Reference**: Use any audio file on `emotion_audio` as an emotion reference for natural expression
|
||||
* **Character Voices Integration**: Use Character Voices `opt_narrator` on `emotion_audio`, including per-character `[Character:emotion_ref]` references
|
||||
* **8-Emotion Vector Control**: Manual precision control over Happy, Angry, Sad, Surprised, Afraid, Disgusted, Calm, and Melancholic emotions
|
||||
* **Character Tag Emotions**: Per-character emotion control using `[Character:emotion_ref]` syntax (highest priority)
|
||||
* **Character Tag Emotions**: Per-character audio emotion control using `[Character:emotion_ref]` syntax, blendable with vector/text emotion
|
||||
* **Emotion Alpha Control**: Fine-tune emotion intensity from 0.0 (neutral) to 2.0 (maximum dramatic expression)
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- **Emotion Priority System**: Character tags > Global emotion control with intelligent override handling
|
||||
- **Emotion Blending**: Audio references and vector/text emotion are blended in IndexTTS-2's latent conditioning space; character tags select segment-local audio references
|
||||
- **Dynamic Templates**: Use `{seg}` placeholder for contextual emotion analysis (e.g., "Worried parent speaking: {seg}")
|
||||
- **Universal Compatibility**: Works with existing TTS Text and TTS SRT nodes seamlessly
|
||||
- **Advanced Caching**: Stable audio content hashing for reliable cache hits across sessions
|
||||
@@ -892,6 +902,24 @@ Description: "A deep, authoritative male voice with clear articulation"
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><h3>OmniVoice + Visual Tag Builder</h3></summary>
|
||||
|
||||
**NEW in v5.x**: OmniVoice is now integrated into the unified suite with native duration-aware SRT generation and a generalized visual tag workflow.
|
||||
|
||||
* **🌍 OmniVoice integration**: official model support with broad upstream language coverage
|
||||
* **🎯 Native duration targeting**: SRT workflows can send target durations directly into OmniVoice instead of relying only on post-generation time stretching
|
||||
* **⏱️ Precise segment control**: this is the first engine in the suite where segment duration can be meaningfully guided at generation time, making precise TTS timing far more practical
|
||||
* **📺 Better SRT timing behavior**: subtitle generation can land much closer to target timings before any fallback timing correction, so stretch-to-fit has less work to do and results can stay more natural
|
||||
* **📐 Visual Tag Builder**: reusable preset-driven visual node for assembling tag or attribute strings, originally added for OmniVoice voice-design prompting and now generalized for broader tag-based text workflows
|
||||
* **🔊 Native inline non-verbal tags**: OmniVoice non-verbal controls are exposed in suite-default `<>` form like `<laughter>`, then converted internally for generation → **[📖 OmniVoice Tags Guide](docs/OMNIVOICE_TAGS_GUIDE.md)**
|
||||
|
||||
**Practical note:**
|
||||
|
||||
Use the built-in OmniVoice preset in **📐 Visual Tag Builder** for the canonical voice-design workflow. If you need a different tag schema, the same node now supports reusable custom presets with saved column order.
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><h3>MOSS-TTS - Local/Delay/TTSD Engine Family</h3></summary>
|
||||
|
||||
@@ -1044,6 +1072,10 @@ Use the new [Unified ✏️ ASR Transcribe + SRT Builder](example_workflows/Unif
|
||||
Beyond character switching and language control, you can now override generation parameters (seed, temperature, CFG, speed, etc.) on a per-segment basis using inline tags. The new **🏷️ Multiline TTS Tag Editor** node makes building complex tags easier and more visual with:
|
||||
- **Rich Text Editor**: Multiline editor with resizable font sizes (2-120px), multiple font families, and customizable UI scaling
|
||||
- **Visual Tag Management**: Character/language/parameter dropdowns for quick selection, inline tag validation with syntax checking
|
||||
- **Engine-Aware Inline Tags**: dedicated editor modes for Step Audio EditX, Higgs Audio v3, CosyVoice3, and OmniVoice
|
||||
- **One-Click Tag Swapping**: click a character, language, audio reference, parameter, or supported native inline tag to open an engine-aware replacement palette; click again or press-drag-release to apply
|
||||
- **IndexTTS-2 Emotion Editing**: insert vectors, named emotion values, presets, quoted text, and `{seg}` dynamic emotion controls directly from the Inline Tags panel
|
||||
- **Safe Long-Tag Layout**: long bracket and angle tags wrap inside the editor instead of overflowing the text area; quoted emotion text remains directly editable
|
||||
- **Preset System**: Save and load up to 3 preset configurations for rapid tag reuse
|
||||
- **Keyboard Shortcuts**: Alt+L/C/P for tag insertion, Alt+1/2/3 for preset loading
|
||||
- **History & Undo/Redo**: Full edit history with Alt+Z for undo (Alt+Shift+Z for redo)
|
||||
@@ -1082,7 +1114,7 @@ This enables dynamic control over individual audio segments without modifying no
|
||||
- **VibeVoice**: seed, temperature, cfg, top_p, top_k, inference_steps
|
||||
- **IndexTTS-2**: seed, temperature, cfg, top_p, top_k, emotion_alpha
|
||||
|
||||
**📖 Guides:** [Per-Segment Parameter Switching](docs/PARAMETER_SWITCHING_GUIDE.md) | [Multiline TTS Tag Editor](docs/MULTILINE_TTS_TAG_EDITOR_GUIDE.md)
|
||||
**📖 Guides:** [Per-Segment Parameter Switching](docs/PARAMETER_SWITCHING_GUIDE.md) | [Multiline TTS Tag Editor](docs/MULTILINE_TTS_TAG_EDITOR_GUIDE.md) | [OmniVoice Tags Guide](docs/OMNIVOICE_TAGS_GUIDE.md)
|
||||
|
||||
Perfect for:
|
||||
|
||||
@@ -1441,6 +1473,8 @@ For offline/manual setup:
|
||||
| Granite ASR | `ComfyUI/models/TTS/granite_asr/` | ✅ | Granite ASR models; plus adds native diarization/timestamps, optional Qwen forced aligner reused lazily for timestamps/SRT fallback |
|
||||
| Echo-TTS | `ComfyUI/models/TTS/echo-tts-base/` | ✅ | ~7.1GB total (base + dac); CC-BY-NC-SA |
|
||||
| Dots TTS | `ComfyUI/models/TTS/dots_tts/` | ✅ | Official base / soar / mf checkpoints with tokenizer, vocoder, speaker encoder |
|
||||
| Fish Audio S2 Pro | `ComfyUI/models/TTS/fish_audio_s2_pro/` | ✅ | Official BF16 or optional community FP8 checkpoint; the official checkpoint can be quantized on load with BNB INT8/NF4; main T5 environment with process teardown for Clear VRAM; Fish Audio Research License |
|
||||
| OmniVoice | `ComfyUI/models/TTS/omnivoice/` | ✅ | Official OmniVoice model. Voice cloning in this suite requires explicit reference text. |
|
||||
|
||||
*Generated from [tts_audio_suite_engines.yaml](docs/Dev%20reports/tts_audio_suite_engines.yaml).*
|
||||
|
||||
@@ -1482,6 +1516,8 @@ Your support helps maintain and improve this project for the entire community!
|
||||
| **🎨 Step Audio EditX - Audio Editor** | Step Audio EditX audio editing with inline edit tags | ✅ **New in v4.14** | [📁 JSON](example_workflows/🎨%20Step%20Audio%20EditX%20-%20Audio%20Editor%20+%20Inline%20Edit%20Tags.json) |
|
||||
| **⚙️ Step Audio EditX Integration** | Step Audio EditX TTS engine with zero-shot voice cloning | ✅ **New in v4.14** | [📁 JSON](example_workflows/Step%20Audio%20EditX%20Integration.json) |
|
||||
| **⚙️ Higgs Audio v3 Integration** | Higgs Audio v3 TTS with zero-shot voice cloning and native inline tags | ✅ **New in v4.27** | [📁 JSON](example_workflows/Higgs%20Audio%20v3%20Integration.json) |
|
||||
| **⚙️ OmniVoice Engine Integration** | OmniVoice multilingual TTS with cloning, voice design, and native duration control | ✅ **New in v4.28** | [📁 JSON](example_workflows/OmniVoice%20Engine%20Integration.json) |
|
||||
| **⚙️ Fish Audio S2 Pro Integration** | Fish S2 Pro multilingual cloning with native multi-speaker dialogue, inline control, and long-form generation | ✅ **New in v5.3** | [📁 JSON](example_workflows/Fish%20Audio%20S2%20integration.json) |
|
||||
| **🌈 IndexTTS-2 Integration** | IndexTTS-2 engine with advanced emotion control | ✅ **New in v4.9** | [📁 JSON](example_workflows/🌈%20IndexTTS-2%20integration.json) |
|
||||
| **📝 F5 TTS + Text Normalizer** | F5-TTS with multilingual text processing and phonemization | ✅ **New in v4.10.0** | [📁 JSON](example_workflows/F5%20TTS%20integration%20+%20📝%20Phoneme%20Text%20Normalizer.json) |
|
||||
| **Qwen3 integration + ASR** | Qwen3-TTS voice generation with ASR transcription | ✅ **New in v4.21** | [📁 JSON](example_workflows/Qwen3%20integration%20+%20ASR.json) |
|
||||
|
||||
+95
-14
@@ -294,12 +294,46 @@ __all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
|
||||
WEB_DIRECTORY = "./web"
|
||||
|
||||
# Register API endpoint for widget data
|
||||
def setup_api_routes():
|
||||
"""Setup API routes for widget communication"""
|
||||
try:
|
||||
import json
|
||||
from server import PromptServer
|
||||
from aiohttp import web
|
||||
def setup_api_routes():
|
||||
"""Setup API routes for widget communication"""
|
||||
try:
|
||||
import json
|
||||
import folder_paths
|
||||
from server import PromptServer
|
||||
from aiohttp import web
|
||||
|
||||
def _get_ui_data_dir():
|
||||
base_dir = os.path.join(folder_paths.get_system_user_directory("tts_audio_suite"), "ui")
|
||||
os.makedirs(base_dir, exist_ok=True)
|
||||
return base_dir
|
||||
|
||||
def _get_omnivoice_preset_library_path():
|
||||
return os.path.join(_get_ui_data_dir(), "omnivoice_instruction_builder_presets.json")
|
||||
|
||||
@PromptServer.instance.routes.get("/api/tts-audio-suite/index-tts-emotion-presets")
|
||||
async def get_index_tts_emotion_presets_endpoint(request):
|
||||
"""Return presets stored beside the IndexTTS resources under models/TTS."""
|
||||
try:
|
||||
from .utils.text.index_tts_emotion import load_emotion_presets
|
||||
return web.json_response({"presets": load_emotion_presets()})
|
||||
except Exception as e:
|
||||
print(f"⚠️ Error retrieving IndexTTS emotion presets: {e}")
|
||||
return web.json_response({"presets": {}, "error": str(e)}, status=500)
|
||||
|
||||
@PromptServer.instance.routes.post("/api/tts-audio-suite/index-tts-emotion-presets")
|
||||
async def save_index_tts_emotion_presets_endpoint(request):
|
||||
"""Atomically persist the IndexTTS emotion preset library."""
|
||||
try:
|
||||
from .utils.text.index_tts_emotion import save_emotion_presets
|
||||
data = await request.json()
|
||||
presets = data.get("presets", {})
|
||||
path = save_emotion_presets(presets)
|
||||
return web.json_response({"status": "success", "count": len(presets), "path": path})
|
||||
except ValueError as e:
|
||||
return web.json_response({"error": str(e)}, status=400)
|
||||
except Exception as e:
|
||||
print(f"⚠️ Error saving IndexTTS emotion presets: {e}")
|
||||
return web.json_response({"status": "error", "error": str(e)}, status=500)
|
||||
|
||||
@PromptServer.instance.routes.get("/api/tts-audio-suite/available-characters")
|
||||
async def get_available_characters_endpoint(request):
|
||||
@@ -321,8 +355,8 @@ def setup_api_routes():
|
||||
print(f"⚠️ Error retrieving available characters: {e}")
|
||||
return web.json_response({"characters": [], "error": str(e)})
|
||||
|
||||
@PromptServer.instance.routes.get("/api/tts-audio-suite/available-languages")
|
||||
async def get_available_languages_endpoint(request):
|
||||
@PromptServer.instance.routes.get("/api/tts-audio-suite/available-languages")
|
||||
async def get_available_languages_endpoint(request):
|
||||
"""API endpoint to get available language codes from the canonical language mapper"""
|
||||
try:
|
||||
# Load language_mapper directly by file path to avoid package import issues
|
||||
@@ -335,12 +369,59 @@ def setup_api_routes():
|
||||
languages = sorted(set(language_mapper_module.LANGUAGE_ALIASES.values()))
|
||||
return web.json_response({"languages": languages})
|
||||
except Exception as e:
|
||||
print(f"⚠️ Error retrieving available languages: {e}")
|
||||
# Fallback list
|
||||
return web.json_response({"languages": ["en", "de", "fr", "ja", "es", "it", "pt", "th", "no"], "error": str(e)})
|
||||
|
||||
@PromptServer.instance.routes.get("/api/tts-audio-suite/voice-input-devices")
|
||||
async def get_voice_input_devices_endpoint(request):
|
||||
print(f"⚠️ Error retrieving available languages: {e}")
|
||||
# Fallback list
|
||||
return web.json_response({"languages": ["en", "de", "fr", "ja", "es", "it", "pt", "th", "no"], "error": str(e)})
|
||||
|
||||
@PromptServer.instance.routes.get("/api/tts-audio-suite/omnivoice-presets")
|
||||
async def get_omnivoice_presets_endpoint(request):
|
||||
"""Return the persisted OmniVoice instruction builder preset library."""
|
||||
try:
|
||||
library_path = _get_omnivoice_preset_library_path()
|
||||
if not os.path.exists(library_path):
|
||||
return web.json_response({"presets": [], "builtinStates": {}, "builtinLayouts": {}})
|
||||
with open(library_path, "r", encoding="utf-8") as f:
|
||||
payload = json.load(f)
|
||||
presets = payload.get("presets", []) if isinstance(payload, dict) else []
|
||||
builtin_states = payload.get("builtinStates", {}) if isinstance(payload, dict) else {}
|
||||
builtin_layouts = payload.get("builtinLayouts", {}) if isinstance(payload, dict) else {}
|
||||
if not isinstance(presets, list):
|
||||
presets = []
|
||||
if not isinstance(builtin_states, dict):
|
||||
builtin_states = {}
|
||||
if not isinstance(builtin_layouts, dict):
|
||||
builtin_layouts = {}
|
||||
return web.json_response({"presets": presets, "builtinStates": builtin_states, "builtinLayouts": builtin_layouts})
|
||||
except Exception as e:
|
||||
print(f"⚠️ Error retrieving OmniVoice preset library: {e}")
|
||||
return web.json_response({"presets": [], "builtinStates": {}, "builtinLayouts": {}, "error": str(e)}, status=500)
|
||||
|
||||
@PromptServer.instance.routes.post("/api/tts-audio-suite/omnivoice-presets")
|
||||
async def save_omnivoice_presets_endpoint(request):
|
||||
"""Persist the OmniVoice instruction builder preset library."""
|
||||
try:
|
||||
data = await request.json()
|
||||
presets = data.get("presets", [])
|
||||
builtin_states = data.get("builtinStates", {})
|
||||
builtin_layouts = data.get("builtinLayouts", {})
|
||||
if not isinstance(presets, list):
|
||||
return web.json_response({"error": "presets must be a list"}, status=400)
|
||||
if not isinstance(builtin_states, dict):
|
||||
return web.json_response({"error": "builtinStates must be an object"}, status=400)
|
||||
if not isinstance(builtin_layouts, dict):
|
||||
return web.json_response({"error": "builtinLayouts must be an object"}, status=400)
|
||||
|
||||
library_path = _get_omnivoice_preset_library_path()
|
||||
payload = {"presets": presets, "builtinStates": builtin_states, "builtinLayouts": builtin_layouts}
|
||||
with open(library_path, "w", encoding="utf-8") as f:
|
||||
json.dump(payload, f, ensure_ascii=False, indent=2)
|
||||
return web.json_response({"status": "success", "count": len(presets)})
|
||||
except Exception as e:
|
||||
print(f"⚠️ Error saving OmniVoice preset library: {e}")
|
||||
return web.json_response({"status": "error", "error": str(e)}, status=500)
|
||||
|
||||
@PromptServer.instance.routes.get("/api/tts-audio-suite/voice-input-devices")
|
||||
async def get_voice_input_devices_endpoint(request):
|
||||
"""Return input devices without risking a main-process PortAudio hang."""
|
||||
try:
|
||||
import json
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Overview
|
||||
|
||||
Advanced multiline string editor node that extends ComfyUI's standard multiline widget with a sidebar containing context-aware controls for TTS-specific tags (character switching, parameters, pauses). The node features real-time tag generation, preset management, and intelligent syntax support based on the CHARACTER_SWITCHING_GUIDE and PARAMETER_SWITCHING_GUIDE.
|
||||
Advanced multiline string editor node that extends ComfyUI's standard multiline widget with a sidebar containing context-aware controls for TTS-specific tags (character switching, parameters, pauses, and engine-native inline controls). The node features real-time tag generation, preset management, engine-aware quick swapping, and intelligent syntax support based on the CHARACTER_SWITCHING_GUIDE and PARAMETER_SWITCHING_GUIDE.
|
||||
|
||||
---
|
||||
|
||||
@@ -85,10 +85,14 @@ Advanced multiline string editor node that extends ComfyUI's standard multiline
|
||||
- Split by paragraph breaks or sentence punctuation
|
||||
- Batch apply parameters across multiple `[Character]text` blocks
|
||||
|
||||
#### Tag Inspector
|
||||
- Show existing tags in selection/current line
|
||||
- Checkbox UI to toggle tags on/off temporarily
|
||||
- Quick-edit dialog for existing tag values
|
||||
#### Tag Inspector
|
||||
- Show existing tags in selection/current line
|
||||
- Checkbox UI to toggle tags on/off temporarily
|
||||
- Click a character, language, audio-reference, parameter, or supported engine-native inline tag to open a color-coded quick-swap palette
|
||||
- Click a palette option once to keep the palette open, then click again to commit; press-and-hold, drag, and release selects in one gesture
|
||||
- Palette choices follow the selected inline engine (including IndexTTS-2, Higgs Audio v3, Step Audio EditX, CosyVoice3, and OmniVoice)
|
||||
- Quoted IndexTTS-2 text emotion tags remain direct editable text and do not open a replacement palette
|
||||
- Long bracket and angle tags wrap inside the editor rather than overflowing horizontally
|
||||
|
||||
#### Auto-Formatting
|
||||
- Button: "Auto-Format Tags" → organize tags consistently
|
||||
|
||||
@@ -12,21 +12,26 @@
|
||||
|
||||
**⚠️ IMPORTANT: Use positional arguments, NOT --commit/--changelog flags**
|
||||
|
||||
```bash
|
||||
# EASIEST: Just use 'patch' - script auto-increments the version
|
||||
python3 scripts/bump_version_enhanced.py patch "<commit_desc>" "<changelog_desc>"
|
||||
```powershell
|
||||
# Windows: use the canonical ComfyUI environment for this project
|
||||
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' patch "<commit_desc>" "<changelog_desc>"
|
||||
|
||||
# OR: Specify exact version if needed
|
||||
python3 scripts/bump_version_enhanced.py <version> "<commit_desc>" "<changelog_desc>"
|
||||
# OR: specify an exact version if needed
|
||||
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' <version> "<commit_desc>" "<changelog_desc>"
|
||||
```
|
||||
|
||||
```bash
|
||||
# Linux/macOS: use the available Python 3 interpreter
|
||||
python3 scripts/bump_version_enhanced.py patch "<commit_desc>" "<changelog_desc>"
|
||||
```
|
||||
|
||||
### Examples
|
||||
|
||||
#### Multiline Format (Recommended Standard)
|
||||
|
||||
```bash
|
||||
```powershell
|
||||
# Patch release (bug fixes) - CORRECT FORMAT
|
||||
python3 scripts/bump_version_enhanced.py 3.2.9 "Fix character alias resolution
|
||||
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' 3.2.9 "Fix character alias resolution
|
||||
|
||||
Technical details:
|
||||
- Fix parser bypassing character tags in single mode
|
||||
@@ -37,8 +42,8 @@ Technical details:
|
||||
- Improve character name recognition accuracy
|
||||
- Better error handling for invalid character names"
|
||||
|
||||
# Minor release (new features) - CORRECT FORMAT
|
||||
python3 scripts/bump_version_enhanced.py 3.3.0 "Add Higgs Audio 2 TTS engine
|
||||
# Minor release (new features) - CORRECT FORMAT
|
||||
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' 3.3.0 "Add Higgs Audio 2 TTS engine
|
||||
|
||||
Implementation details:
|
||||
- Integrate boson_multimodal voice cloning system
|
||||
@@ -50,7 +55,7 @@ Implementation details:
|
||||
- Multiple built-in voice presets available"
|
||||
|
||||
# Major release (breaking changes) - CORRECT FORMAT
|
||||
python3 scripts/bump_version_enhanced.py 4.0.0 "Complete unified architecture implementation
|
||||
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' 4.0.0 "Complete unified architecture implementation
|
||||
|
||||
Breaking changes:
|
||||
- Migrate all nodes to unified interface pattern
|
||||
@@ -64,22 +69,22 @@ Breaking changes:
|
||||
```
|
||||
|
||||
#### Auto-Increment Examples (Recommended)
|
||||
```bash
|
||||
# Auto-increment patch version (4.5.25 → 4.5.26) - CORRECT FORMAT
|
||||
python3 scripts/bump_version_enhanced.py patch "Fix character parsing issues" "Fix character name handling in TTS generation"
|
||||
```powershell
|
||||
# Auto-increment patch version (4.5.25 → 4.5.26) - Windows
|
||||
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' patch "Fix character parsing issues" "Fix character name handling in TTS generation"
|
||||
|
||||
# Auto-increment minor version (4.5.25 → 4.6.0) - CORRECT FORMAT
|
||||
python3 scripts/bump_version_enhanced.py minor "Add new TTS engine support" "Add Higgs Audio 2 TTS engine with voice cloning"
|
||||
# Auto-increment minor version (4.5.25 → 4.6.0) - Windows
|
||||
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' minor "Add new TTS engine support" "Add Higgs Audio 2 TTS engine with voice cloning"
|
||||
```
|
||||
|
||||
#### Single-Line Format (Only for Super Minor Changes)
|
||||
```bash
|
||||
python3 scripts/bump_version_enhanced.py patch "Fix typo in node tooltip" "Fix typo in audio analyzer tooltip"
|
||||
```powershell
|
||||
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' patch "Fix typo in node tooltip" "Fix typo in audio analyzer tooltip"
|
||||
```
|
||||
|
||||
#### Dry-Run Preview (Test Before Committing)
|
||||
```bash
|
||||
python3 scripts/bump_version_enhanced.py patch "Fix preview issues" "Fix preview not reflecting filter parameters" --dry-run
|
||||
```powershell
|
||||
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' patch "Fix preview issues" "Fix preview not reflecting filter parameters" --dry-run
|
||||
```
|
||||
|
||||
#### Auto-Categorization System
|
||||
@@ -99,21 +104,22 @@ python3 scripts/bump_version_enhanced.py patch "Fix preview issues" "Fix preview
|
||||
- **Commit**: Technical implementation details for developers
|
||||
- **Changelog**: User-facing benefits and impacts
|
||||
|
||||
**Bash Syntax Notes:**
|
||||
**Command Syntax Notes:**
|
||||
- Multiline strings need proper quoting (opening quote on first line, closing quote on last line)
|
||||
- Use `\` (backslash) for line continuation in bash commands
|
||||
- The Windows command uses PowerShell's `&` call operator and the canonical project Python path
|
||||
- The Linux/macOS command uses `python3`
|
||||
- Don't add manual category prefixes like "Fixed:" - script handles categorization automatically!
|
||||
|
||||
### Interactive Mode (Recommended for Complex Changes)
|
||||
|
||||
```bash
|
||||
python3 scripts/bump_version_enhanced.py 3.2.9 --interactive
|
||||
```powershell
|
||||
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' 3.2.9 --interactive
|
||||
```
|
||||
|
||||
### Legacy Mode (Same Description for Both)
|
||||
|
||||
```bash
|
||||
python3 scripts/bump_version_enhanced.py 3.2.9 "Fix bugs and improve stability"
|
||||
```powershell
|
||||
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' 3.2.9 "Fix bugs and improve stability"
|
||||
```
|
||||
|
||||
### What the Script Does
|
||||
@@ -239,13 +245,14 @@ git commit -m "Prepare for version bump"
|
||||
- Use semantic versioning: `4.5.25` (not `v4.5.25` or `4.5`)
|
||||
- Or use auto-increment: `patch`, `minor`, `major`
|
||||
|
||||
**Bash syntax errors with multiline**
|
||||
- Make sure opening quote is on same line as `--commit` or `--changelog`
|
||||
**Command syntax errors with multiline**
|
||||
- Make sure opening quote is on same line as `--commit` or `--changelog`
|
||||
- Make sure closing quote is on its own line
|
||||
- Use `\` for line continuation
|
||||
- On Windows, use the PowerShell command shown above
|
||||
- On Linux/macOS, use `python3`
|
||||
|
||||
**Want to see what will happen before committing?**
|
||||
```bash
|
||||
```powershell
|
||||
# Add --dry-run to preview changelog categorization
|
||||
python3 scripts/bump_version_enhanced.py patch "description" "changelog" --dry-run
|
||||
```
|
||||
& 'J:\stablediffusion1111s2\Data\Packages\ComfyUIPy129\test_env_error\Scripts\python.exe' 'scripts\bump_version_enhanced.py' patch "description" "changelog" --dry-run
|
||||
```
|
||||
|
||||
@@ -2,10 +2,17 @@
|
||||
|
||||
This document tracks architectural issues and inconsistencies that need refactoring to improve modularity and reduce code duplication.
|
||||
|
||||
## SRT Processing Architecture Issues
|
||||
|
||||
### Problem: Inconsistent SRT Implementation Approaches
|
||||
Different engines use completely different patterns for SRT processing:
|
||||
## SRT Processing Architecture Issues
|
||||
|
||||
### Problem: VibeVoice Native Multi-Speaker SRT Uses Non-Contiguous Global Slots
|
||||
- **Issue**: Later subtitles can request global `Speaker 2`/`Speaker 3` IDs while VibeVoice truncates and renumbers the supplied voice prompts from zero.
|
||||
- **Impact**: A subtitle containing only later global speakers can bind the wrong reference or leave a requested speaker without a matching voice prompt.
|
||||
- **Solution**: Use the global character map only to select references, then compact each subtitle to request-local speakers `0..N-1` with references in the same order.
|
||||
- **Regression test**: Cover subtitle 1 `[Alice]...[Bob]...`, followed by subtitle 2 `[Bob]...[Rick]...`.
|
||||
- **Priority**: High
|
||||
|
||||
### Problem: Inconsistent SRT Implementation Approaches
|
||||
Different engines use completely different patterns for SRT processing:
|
||||
|
||||
1. **ChatterBox (Old)**: Uses `ChatterboxSRTTTSNode` class (should be processor)
|
||||
2. **VibeVoice**: Uses proper `VibeVoiceSRTProcessor` class with full implementation
|
||||
@@ -136,4 +143,4 @@ Each engine reimplements:
|
||||
---
|
||||
|
||||
*Last Updated: 2025-01-XX*
|
||||
*Add new issues to this file as they are discovered during development*
|
||||
*Add new issues to this file as they are discovered during development*
|
||||
|
||||
@@ -55,7 +55,6 @@ categories:
|
||||
languages:
|
||||
- "47 languages"
|
||||
notes: "Heavier multilingual fallback with the widest language coverage supported by the punctuators package."
|
||||
|
||||
model_layouts_markdown: |
|
||||
# Auxiliary Model Layouts
|
||||
|
||||
|
||||
@@ -131,6 +131,7 @@ engines:
|
||||
community_finetunes: { supported: true, notes: "" }
|
||||
vram_efficient: { supported: true, notes: "" }
|
||||
speed_performance: { supported: true, notes: "Very Fast" }
|
||||
reference_free_tts: { supported: false, notes: "" }
|
||||
|
||||
- id: chatterbox
|
||||
name: ChatterBox
|
||||
@@ -262,6 +263,7 @@ engines:
|
||||
community_finetunes: { supported: true, notes: "" }
|
||||
vram_efficient: { supported: true, notes: "" }
|
||||
speed_performance: { supported: true, notes: "Fast" }
|
||||
reference_free_tts: { supported: true, notes: "(default speaker)" }
|
||||
|
||||
- id: chatterbox-23l
|
||||
name: ChatterBox 23L
|
||||
@@ -346,6 +348,7 @@ engines:
|
||||
community_finetunes: { supported: true, notes: "" }
|
||||
vram_efficient: { supported: true, notes: "" }
|
||||
speed_performance: { supported: true, notes: "Fast" }
|
||||
reference_free_tts: { supported: true, notes: "(default speaker)" }
|
||||
|
||||
- id: vibevoice
|
||||
name: VibeVoice
|
||||
@@ -443,6 +446,7 @@ engines:
|
||||
community_finetunes: { supported: true, notes: "KugelAudio, Hindi" }
|
||||
vram_efficient: { supported: "partial", notes: "(5-18GB)" }
|
||||
speed_performance: { supported: "partial", notes: "" }
|
||||
reference_free_tts: { supported: true, notes: "(zero-shot / default speaker)" }
|
||||
|
||||
- id: higgs-audio-2
|
||||
name: Higgs Audio 2
|
||||
@@ -524,6 +528,7 @@ engines:
|
||||
community_finetunes: { supported: false, notes: "" }
|
||||
vram_efficient: { supported: "partial", notes: "(9GB)" }
|
||||
speed_performance: { supported: "partial", notes: "" }
|
||||
reference_free_tts: { supported: true, notes: "(basic TTS if no narrator/reference is provided)" }
|
||||
|
||||
- id: higgs-audio-v3
|
||||
name: Higgs Audio v3
|
||||
@@ -531,6 +536,8 @@ engines:
|
||||
size: "~8GB"
|
||||
license: "Boson Higgs Audio v3 Research and Non-Commercial License"
|
||||
commercial: false
|
||||
language_summary_compact: "🌐 100+ languages"
|
||||
language_summary_full: "100+"
|
||||
|
||||
capabilities:
|
||||
tts: true
|
||||
@@ -666,6 +673,7 @@ engines:
|
||||
community_finetunes: { supported: false, notes: "" }
|
||||
vram_efficient: { supported: "partial", notes: "(~8-10GB)" }
|
||||
speed_performance: { supported: "partial", notes: "CUDA recommended" }
|
||||
reference_free_tts: { supported: true, notes: "(zero-shot)" }
|
||||
|
||||
- id: indextts-2
|
||||
name: IndexTTS-2
|
||||
@@ -744,6 +752,7 @@ engines:
|
||||
community_finetunes: { supported: false, notes: "" }
|
||||
vram_efficient: { supported: "partial", notes: "(9-12GB)" }
|
||||
speed_performance: { supported: "partial", notes: "" }
|
||||
reference_free_tts: { supported: false, notes: "" }
|
||||
|
||||
- id: cosyvoice3
|
||||
name: CosyVoice3
|
||||
@@ -809,6 +818,7 @@ engines:
|
||||
community_finetunes: { supported: false, notes: "" }
|
||||
vram_efficient: { supported: true, notes: "(5.4GB)" }
|
||||
speed_performance: { supported: true, notes: "Fast" }
|
||||
reference_free_tts: { supported: true, notes: "(cross-lingual or instruct mode)" }
|
||||
|
||||
- id: qwen3-tts
|
||||
name: Qwen3-TTS
|
||||
@@ -922,6 +932,7 @@ engines:
|
||||
community_finetunes: { supported: false, notes: "" }
|
||||
vram_efficient: { supported: true, notes: "(3-6GB)" }
|
||||
speed_performance: { supported: "partial", notes: "" }
|
||||
reference_free_tts: { supported: true, notes: "(Base default voice or CustomVoice presets)" }
|
||||
|
||||
- id: granite-asr
|
||||
name: Granite ASR
|
||||
@@ -1010,6 +1021,7 @@ engines:
|
||||
community_finetunes: { supported: false, notes: "" }
|
||||
vram_efficient: { supported: true, notes: "(~4.6GB)" }
|
||||
speed_performance: { supported: "partial", notes: "Moderate" }
|
||||
reference_free_tts: { supported: null, notes: "N/A" }
|
||||
|
||||
- id: step-editx
|
||||
name: Step Audio EditX
|
||||
@@ -1083,6 +1095,7 @@ engines:
|
||||
community_finetunes: { supported: false, notes: "" }
|
||||
vram_efficient: { supported: "partial", notes: "(7GB)" }
|
||||
speed_performance: { supported: "partial", notes: "" }
|
||||
reference_free_tts: { supported: false, notes: "(clone mode only)" }
|
||||
|
||||
- id: echo-tts
|
||||
name: Echo-TTS
|
||||
@@ -1155,6 +1168,76 @@ engines:
|
||||
community_finetunes: { supported: false, notes: "" }
|
||||
vram_efficient: { supported: "partial", notes: "(~7GB total)" }
|
||||
speed_performance: { supported: true, notes: "Fast (diffusion, realtime-capable)" }
|
||||
reference_free_tts: { supported: false, notes: "(reference audio required)" }
|
||||
|
||||
- id: fish-audio-s2-pro
|
||||
name: Fish Audio S2 Pro
|
||||
models: "S2 Pro 4B / FP8"
|
||||
size: "~10.3GB / ~8.0GB"
|
||||
license: "Fish Audio Research License"
|
||||
commercial: false
|
||||
language_summary_full: "80+ languages"
|
||||
language_summary_compact: "🌐 80+ languages"
|
||||
|
||||
capabilities:
|
||||
tts: true
|
||||
srt: true
|
||||
vc: false
|
||||
asr: false
|
||||
training: false
|
||||
|
||||
runtime_isolation:
|
||||
default_mode: "main_environment"
|
||||
main_environment: true
|
||||
shared_runtime: false
|
||||
dedicated_runtime: false
|
||||
|
||||
special_features:
|
||||
- "Free-form sub-word emotion/prosody tags"
|
||||
- "Zero-shot voice cloning and 80+ languages"
|
||||
- "Native multi-speaker and multi-turn dialogue with dynamic speaker references"
|
||||
- "Optional per-segment custom character switching"
|
||||
- "Configurable 4K-32K native context with reduced KV-cache VRAM"
|
||||
- "Optional community FP8 weight-only checkpoint with BF16 activations"
|
||||
- "Optional on-the-fly BitsAndBytes INT8/NF4 for the official checkpoint"
|
||||
|
||||
model_sources:
|
||||
- component: "S2 Pro"
|
||||
source_name: "fishaudio/s2-pro"
|
||||
source_url: "https://huggingface.co/fishaudio/s2-pro"
|
||||
size: "~10.3GB"
|
||||
auto_download: true
|
||||
notes: "Official 4B model and codec; BNB INT8/NF4 are optional load-time quantization modes that reuse these files; non-commercial license"
|
||||
- component: "S2 Pro FP8"
|
||||
source_name: "drbaph/s2-pro-fp8"
|
||||
source_url: "https://huggingface.co/drbaph/s2-pro-fp8"
|
||||
size: "~8.0GB"
|
||||
auto_download: true
|
||||
notes: "Community weight-only FP8 checkpoint; BF16 activations; RTX 4090/5090-class CUDA GPU required"
|
||||
|
||||
languages:
|
||||
en: { supported: true, flag: "🇺🇸", notes: "Tier 1" }
|
||||
zh: { supported: true, flag: "🇨🇳", notes: "Tier 1" }
|
||||
ja: { supported: true, flag: "🇯🇵", notes: "Tier 1" }
|
||||
ko: { supported: true, flag: "🇰🇷", notes: "Tier 2" }
|
||||
es: { supported: true, flag: "🇪🇸", notes: "Tier 2" }
|
||||
pt: { supported: true, flag: "🇵🇹", notes: "Tier 2" }
|
||||
ar: { supported: true, flag: "🇦🇪", notes: "Tier 2" }
|
||||
ru: { supported: true, flag: "🇷🇺", notes: "Tier 2" }
|
||||
fr: { supported: true, flag: "🇫🇷", notes: "Tier 2" }
|
||||
de: { supported: true, flag: "🇩🇪", notes: "Tier 2" }
|
||||
|
||||
features:
|
||||
voice_cloning: { supported: true, notes: "Reference audio plus exact transcript" }
|
||||
native_multi_speaker: { supported: true, notes: "Native dialogue is default; optional custom mode generates each [Character] segment independently as local speaker 0" }
|
||||
voice_conversion: { supported: false, notes: "Not exposed by official S2 TTS inference" }
|
||||
asr_transcribe: { supported: false, notes: "" }
|
||||
emotion_control: { supported: true, notes: "Free-form inline natural-language tags" }
|
||||
native_long_form: { supported: true, notes: "Configurable 4K-32K native context; suite text chunking is bypassed" }
|
||||
community_finetunes: { supported: false, notes: "No suite integration" }
|
||||
vram_efficient: { supported: "partial", notes: "8K context measured at ~15.2GB BF16, ~11.2GB FP8, ~11.2GB BNB INT8, or ~8.9GB BNB NF4; BF16 codec and activations; BNB is a load-time option for the official checkpoint" }
|
||||
speed_performance: { supported: true, notes: "Main-environment subprocess with reliable teardown; local compile measurements: ~40 it/s BF16, ~11.8 it/s NF4 at ~8.9GB VRAM, and ~3.7 it/s INT8 at ~11.2GB VRAM; quality comparison pending" }
|
||||
reference_free_tts: { supported: true, notes: "Reference audio is optional" }
|
||||
|
||||
- id: dots-tts
|
||||
name: Dots TTS
|
||||
@@ -1233,6 +1316,78 @@ engines:
|
||||
community_finetunes: { supported: false, notes: "" }
|
||||
vram_efficient: { supported: "partial", notes: "(main env works; 2B-class model is not lightweight)" }
|
||||
speed_performance: { supported: "partial", notes: "Moderate; mf variant is faster" }
|
||||
reference_free_tts: { supported: true, notes: "(default speaker)" }
|
||||
|
||||
- id: omnivoice
|
||||
name: OmniVoice
|
||||
models: "OmniVoice"
|
||||
size: "~3.7GB"
|
||||
license: "Apache-2.0"
|
||||
commercial: true
|
||||
language_summary_compact: "🌐 600+ languages"
|
||||
language_summary_full: "600+"
|
||||
|
||||
capabilities:
|
||||
tts: true
|
||||
srt: true
|
||||
vc: false
|
||||
asr: false
|
||||
training: false
|
||||
|
||||
special_features:
|
||||
- "600+ language support"
|
||||
- "Instruction-based voice design"
|
||||
- "Upstream long-form chunk orchestration"
|
||||
- "Inline non-verbal tags and pronunciation overrides"
|
||||
|
||||
model_sources:
|
||||
- component: "OmniVoice"
|
||||
source_name: "k2-fsa/OmniVoice"
|
||||
source_url: "https://huggingface.co/k2-fsa/OmniVoice"
|
||||
size: "~3.7GB"
|
||||
auto_download: true
|
||||
notes: "Official main model including bundled audio_tokenizer assets"
|
||||
|
||||
languages:
|
||||
en: { supported: true, flag: "🇺🇸", notes: "" }
|
||||
zh: { supported: true, flag: "🇨🇳", notes: "" }
|
||||
de: { supported: true, flag: "🇩🇪", notes: "" }
|
||||
es: { supported: true, flag: "🇪🇸", notes: "" }
|
||||
fr: { supported: true, flag: "🇫🇷", notes: "" }
|
||||
it: { supported: true, flag: "🇮🇹", notes: "" }
|
||||
ja: { supported: true, flag: "🇯🇵", notes: "" }
|
||||
ko: { supported: true, flag: "🇰🇷", notes: "" }
|
||||
ru: { supported: true, flag: "🇷🇺", notes: "" }
|
||||
pt: { supported: true, flag: "🇵🇹", notes: "(generic PT; official language space is much broader than this matrix)" }
|
||||
pl: { supported: true, flag: "🇵🇱", notes: "" }
|
||||
hi: { supported: true, flag: "🇮🇳", notes: "" }
|
||||
ar: { supported: true, flag: "🇦🇪", notes: "" }
|
||||
tr: { supported: true, flag: "🇹🇷", notes: "" }
|
||||
th: { supported: true, flag: "🇹🇭", notes: "" }
|
||||
no: { supported: true, flag: "🇳🇴", notes: "" }
|
||||
vi: { supported: true, flag: "🇻🇳", notes: "" }
|
||||
hy: { supported: true, flag: "🇦🇲", notes: "" }
|
||||
ka: { supported: true, flag: "🇬🇪", notes: "" }
|
||||
da: { supported: true, flag: "🇩🇰", notes: "" }
|
||||
fi: { supported: true, flag: "🇫🇮", notes: "" }
|
||||
el: { supported: true, flag: "🇬🇷", notes: "" }
|
||||
he: { supported: true, flag: "🇮🇱", notes: "" }
|
||||
ms: { supported: true, flag: "🇲🇾", notes: "" }
|
||||
nl: { supported: true, flag: "🇳🇱", notes: "" }
|
||||
sv: { supported: true, flag: "🇸🇪", notes: "" }
|
||||
sw: { supported: true, flag: "🇰🇪", notes: "" }
|
||||
|
||||
features:
|
||||
voice_cloning: { supported: true, notes: "" }
|
||||
native_multi_speaker: { supported: false, notes: "" }
|
||||
voice_conversion: { supported: false, notes: "" }
|
||||
asr_transcribe: { supported: false, notes: "(no standalone ASR node)" }
|
||||
emotion_control: { supported: "partial", notes: "(voice-design instruct + inline non-verbal tags)" }
|
||||
native_long_form: { supported: true, notes: "(uses upstream audio_chunk_duration / audio_chunk_threshold orchestration; bypasses suite char-based chunk splitting)" }
|
||||
community_finetunes: { supported: false, notes: "" }
|
||||
vram_efficient: { supported: "partial", notes: "(main env works; 2B-class model is not lightweight)" }
|
||||
speed_performance: { supported: true, notes: "Fast; upstream reports sub-realtime RTF" }
|
||||
reference_free_tts: { supported: true, notes: "(native default voice; instruct also works without narrator)" }
|
||||
|
||||
- id: moss-tts
|
||||
name: MOSS-TTS
|
||||
@@ -1321,6 +1476,7 @@ engines:
|
||||
community_finetunes: { supported: true, notes: "(LoRA adapter inference supported; initial integrated LoRA training support added for MOSS-TTS Delay 8B)" }
|
||||
vram_efficient: { supported: "partial", notes: "(Local 1.7B smaller; tokenizer is large)" }
|
||||
speed_performance: { supported: true, notes: "Fast with CUDA/FlashAttention" }
|
||||
reference_free_tts: { supported: true, notes: "(direct TTS and prompt-only generation)" }
|
||||
|
||||
- id: rvc
|
||||
name: RVC
|
||||
@@ -1414,6 +1570,7 @@ engines:
|
||||
community_finetunes: { supported: true, notes: "" }
|
||||
vram_efficient: { supported: true, notes: "" }
|
||||
speed_performance: { supported: true, notes: "Fast" }
|
||||
reference_free_tts: { supported: null, notes: "N/A" }
|
||||
|
||||
# Language metadata for reference
|
||||
language_metadata:
|
||||
@@ -1531,6 +1688,7 @@ feature_metadata:
|
||||
community_finetunes: { name: "Community Finetunes", display: "**Community Finetunes**" }
|
||||
vram_efficient: { name: "VRAM Efficient", display: "**VRAM Efficient**" }
|
||||
speed_performance: { name: "Speed/Performance", display: "**Speed/Performance**" }
|
||||
reference_free_tts: { name: "No Narrator Required", display: "**No Narrator Required**" }
|
||||
|
||||
# Table notes for additional context
|
||||
table_notes:
|
||||
@@ -1540,6 +1698,7 @@ table_notes:
|
||||
- "**Higgs Audio v3**: Official model card lists 102 languages total: 85 polished-production languages and 17 usable but less polished languages."
|
||||
- "**IndexTTS-2**: Trained on 55K+ hours - ZH, EN, JA primary."
|
||||
- "**MOSS-TTS**: Official list includes ZH, EN, DE, ES, FR, JA, IT, HU, KO, RU, FA, AR, PL, PT, CS, DA, SV, EL, TR; HU/FA/CS are not separate columns in this matrix."
|
||||
- "**OmniVoice**: Official model supports 600+ languages; this matrix only shows the suite's comparison subset."
|
||||
- "**RVC**: Language-agnostic voice conversion/post-processing; it is marked supported for every language row."
|
||||
- "Qwen3-TTS: Supports pt-BR only with instruction (voice design or harcoded voices). Base can't do instructions, so it will always ouput pt-PT."
|
||||
|
||||
@@ -1604,6 +1763,14 @@ readme_model_download_table:
|
||||
primary_model_path: "ComfyUI/models/TTS/dots_tts/"
|
||||
auto_download: "✅"
|
||||
notes: "Official base / soar / mf checkpoints with tokenizer, vocoder, speaker encoder"
|
||||
- engine: "Fish Audio S2 Pro"
|
||||
primary_model_path: "ComfyUI/models/TTS/fish_audio_s2_pro/"
|
||||
auto_download: "✅"
|
||||
notes: "Official BF16 or optional community FP8 checkpoint; the official checkpoint can be quantized on load with BNB INT8/NF4; main T5 environment with process teardown for Clear VRAM; Fish Audio Research License"
|
||||
- engine: "OmniVoice"
|
||||
primary_model_path: "ComfyUI/models/TTS/omnivoice/"
|
||||
auto_download: "✅"
|
||||
notes: "Official OmniVoice model. Voice cloning in this suite requires explicit reference text."
|
||||
|
||||
model_layouts_markdown: |
|
||||
# Model Folder Layouts
|
||||
@@ -1924,6 +2091,33 @@ model_layouts_markdown: |
|
||||
- Both components are required and auto-downloaded on first use.
|
||||
- License: CC-BY-NC-SA (non-commercial).
|
||||
|
||||
## Fish Audio S2 Pro
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/fish_audio_s2_pro/
|
||||
├── codec.pth
|
||||
├── config.json
|
||||
├── model-00001-of-00002.safetensors
|
||||
├── model-00002-of-00002.safetensors
|
||||
├── model.safetensors.index.json
|
||||
└── tokenizer.json
|
||||
```
|
||||
|
||||
Optional FP8 variant:
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/fish_audio_s2_pro_fp8/
|
||||
├── codec.pth
|
||||
├── config.json
|
||||
├── model.safetensors
|
||||
├── quantization_info.json
|
||||
└── tokenizer.json
|
||||
```
|
||||
|
||||
The complete official repository metadata and tokenizer files are downloaded alongside these files. License: Fish Audio Research License (non-commercial without a separate commercial license).
|
||||
|
||||
The `s2-pro-bnb-int8` and `s2-pro-bnb-nf4` options reuse `fish_audio_s2_pro/` and quantize its official checkpoint while loading. They do not download another model copy and require `bitsandbytes`.
|
||||
|
||||
## Dots TTS
|
||||
|
||||
```text
|
||||
@@ -1957,3 +2151,32 @@ model_layouts_markdown: |
|
||||
- Downloads into a dedicated `dots_tts/` folder.
|
||||
- Native sample rate is 48kHz.
|
||||
- Main-environment support works on Transformers 5; on Windows, `normalize_text` falls back to no-op if `WeTextProcessing` is unavailable.
|
||||
|
||||
## OmniVoice
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/omnivoice/
|
||||
└── OmniVoice/
|
||||
```
|
||||
|
||||
Main model contents:
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/omnivoice/OmniVoice/
|
||||
├── chat_template.jinja
|
||||
├── config.json
|
||||
├── model.safetensors
|
||||
├── tokenizer.json
|
||||
├── tokenizer_config.json
|
||||
└── audio_tokenizer/
|
||||
├── config.json
|
||||
├── model.safetensors
|
||||
└── preprocessor_config.json
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Downloads into a dedicated `omnivoice/` folder.
|
||||
- Native sample rate is 24kHz.
|
||||
- The main OmniVoice model stays in the main Transformers 5 environment.
|
||||
- Voice cloning requires reference audio plus explicit reference text in this suite.
|
||||
|
||||
@@ -9,14 +9,16 @@
|
||||
| **ChatterBox 23L** | Main | v1, v2, Vietnamese (Viterbox), Egyptian Arabic (oddadmix) | ~4.3GB | ✅ | ✅ | ✅ | ❌ | ❌ | MIT | 24 languages in single model, emotion tokens (v2 - doesn't work) | 25 |
|
||||
| **VibeVoice** | Shared | 1.5B, 7B, KugelAudio-0 (7B), kugel-2 (7B), Hindi-1.5B/7B | 5.4GB / 18GB | ✅ | ✅ | ❌ | ❌ | ❌ | MIT (research-only per model card) | 90-min long-form, Native 4-speaker (Base models), Multilingual (KugelAudio variants), 4-bit quantization | 27 |
|
||||
| **Higgs Audio 2** | Shared | 3B | ~9GB | ✅ | ✅ | ❌ | ❌ | ❌ | Boson Higgs Audio 2 Community License | 3 multi-speaker, CUDA graphs (55+ tokens/sec) | 5 |
|
||||
| **Higgs Audio v3** | Main | 4B | ~8GB | ✅ | ✅ | ❌ | ❌ | ❌ | Boson Higgs Audio v3 Research and Non-Commercial License | Native inline emotion/style/prosody/SFX tags, Zero-shot voice cloning, 100+ language support | 102 |
|
||||
| **Higgs Audio v3** | Main | 4B | ~8GB | ✅ | ✅ | ❌ | ❌ | ❌ | Boson Higgs Audio v3 Research and Non-Commercial License | Native inline emotion/style/prosody/SFX tags, Zero-shot voice cloning, 100+ language support | 100+ |
|
||||
| **IndexTTS-2** | Main | IndexTTS-2 | ~4.7GB | ✅ | ✅ | ❌ | ❌ | ❌ | bilibili Model Use License | Emotion Control: 8 vectors, Text as reference, Audio as reference | 3 |
|
||||
| **CosyVoice3** | Main | 0.5B, 0.5B-RL | ~5.4GB | ✅ | ✅ | ✅ | ❌ | ❌ | Apache-2.0 | Paralinguistic tags | 4 |
|
||||
| **Qwen3-TTS** | Shared | 0.6B, 1.7B (CustomVoice/VoiceDesign/Base) | ~3-6GB | ✅ | ✅ | ❌ | ✅ | ❌ | Apache-2.0 | Voice design, ASR (Automatic Speech Recognition) | 10 |
|
||||
| **Granite ASR** | Main | granite-4.0-1b-speech, granite-speech-4.1-2b, granite-speech-4.1-2b-plus | ~4.6GB | ❌ | ✅ | ❌ | ✅ | ❌ | Apache-2.0 | ASR (Automatic Speech Recognition), Native speaker attribution / diarization (plus model variant), Native word-level timestamps (plus model variant), Custom timestamps/SRT via reused Qwen forced aligner, Speech translation (experimental), Optional forced aligner auto-routed through shared legacy T4 runtime | 6 |
|
||||
| **Step Audio EditX** | Main | 3B LLM + CosyVoice | ~7GB | ✅ | ✅ | ❌ | ❌ | ❌ | Apache-2.0 (verify before commercial use) | Second Pass Speech Editing Node: 14 emotions, 32 speaking styles, Paralinguistic effects | 4 |
|
||||
| **Echo-TTS** | Main | echo-tts-base + fish-s1-dac-min | ~5.3GB + ~1.8GB | ✅ | ✅ | ❌ | ❌ | ❌ | CC-BY-NC-SA-4.0 | Diffusion-based (~30s best), Force Speaker KV (speaker drift control) | 1 |
|
||||
| **Fish Audio S2 Pro** | Main | S2 Pro 4B / FP8 | ~10.3GB / ~8.0GB | ✅ | ✅ | ❌ | ❌ | ❌ | Fish Audio Research License | Free-form sub-word emotion/prosody tags, Zero-shot voice cloning and 80+ languages, Native multi-speaker and multi-turn dialogue with dynamic speaker references, Optional per-segment custom character switching, Configurable 4K-32K native context with reduced KV-cache VRAM, Optional community FP8 weight-only checkpoint with BF16 activations, Optional on-the-fly BitsAndBytes INT8/NF4 for the official checkpoint | 80+ languages |
|
||||
| **Dots TTS** | Main | dots.tts-base, dots.tts-soar, dots.tts-mf | ~6GB | ✅ | ✅ | ❌ | ❌ | ❌ | Apache-2.0 | Official auto language detect / language control, SOAR and MeanFlow distilled variants | 19 |
|
||||
| **OmniVoice** | Main | OmniVoice | ~3.7GB | ✅ | ✅ | ❌ | ❌ | ❌ | Apache-2.0 | 600+ language support, Instruction-based voice design, Upstream long-form chunk orchestration, Inline non-verbal tags and pronunciation overrides | 600+ |
|
||||
| **MOSS-TTS** | Main | Local 1.7B, Delay 8B, TTSD 8B | ~8.5GB tokenizer + ~6.1GB/17GB/18GB model | ✅ | ✅ | ❌ | ❌ | ✅ | Apache-2.0 | 20-language generation, Long-form generation (TTSD/Delay), Native 1-5 speaker TTSD dialogue, Duration token hint, Local/Delay/TTSD variants, Initial integrated LoRA training workflow (Delay 8B) | 16 |
|
||||
| **RVC** | Main | Community .pth | 100-300MB | ❌ | ❌ | ✅ | ❌ | ✅ | MIT (framework); community models vary | Real-time VC, Integrated training workflow, Pitch shift (±14), 6 HuBERT models, Language-independent | Any |
|
||||
|
||||
|
||||
+15
-14
@@ -2,17 +2,18 @@
|
||||
|
||||
## Feature Comparison Matrix
|
||||
|
||||
| Feature | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Dots TTS | MOSS-TTS | RVC |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| **TTS** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||
| **SRT** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||
| **Voice Conversion** | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| **ASR (Transcribe)** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Training** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
|
||||
| **Voice Cloning** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (Base model) | ❌ | ✅ | ✅ | ✅ | ✅ | ⚠️ (needs training) |
|
||||
| **Native Multi-Speaker** | ❌ | ❌ | ❌ | ✅ (Base only, Kugel uses fallback) | ✅ | ❌ | ❌ | ❌ | ❌ | ⚠️ (Plus variant speaker attribution / diarization) | ❌ | ❌ | ❌ | ✅ (TTSD v1.0; 1-5 speakers) | ❌ |
|
||||
| **Emotion Control** | ❌ | ❌ | ⚠️ (v2 tags - doesn't work) | ❌ | ⚠️ (via prompt) | ✅ (native inline tags) | ✅ (8 emotions) | ⚠️ (via instruct) | ⚠️ (via instruct) | ❌ | ✅ (14 emotions) | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Native Long-form** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (TTSD/Delay long-form; use chunk orchestration for very long inputs) | N/A |
|
||||
| **Community Finetunes** | ✅ | ✅ | ✅ | ✅ KugelAudio, Hindi | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (LoRA adapter inference supported; initial integrated LoRA training support added for MOSS-TTS Delay 8B) | ✅ |
|
||||
| **VRAM Efficient** | ✅ | ✅ | ✅ | ⚠️ (5-18GB) | ⚠️ (9GB) | ⚠️ (~8-10GB) | ⚠️ (9-12GB) | ✅ (5.4GB) | ✅ (3-6GB) | ✅ (~4.6GB) | ⚠️ (7GB) | ⚠️ (~7GB total) | ⚠️ (main env works; 2B-class model is not lightweight) | ⚠️ (Local 1.7B smaller; tokenizer is large) | ✅ |
|
||||
| **Speed/Performance** | ✅ Very Fast | ✅ Fast | ✅ Fast | ⚠️ | ⚠️ | ⚠️ CUDA recommended | ⚠️ | ✅ Fast | ⚠️ | ⚠️ Moderate | ⚠️ | ✅ Fast (diffusion, realtime-capable) | ⚠️ Moderate; mf variant is faster | ✅ Fast with CUDA/FlashAttention | ✅ Fast |
|
||||
| Feature | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | OmniVoice | MOSS-TTS | RVC |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| **TTS** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||
| **SRT** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||
| **Voice Conversion** | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| **ASR (Transcribe)** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Training** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
|
||||
| **Voice Cloning** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (Base model) | ❌ | ✅ | ✅ | ✅ Reference audio plus exact transcript | ✅ | ✅ | ✅ | ⚠️ (needs training) |
|
||||
| **Native Multi-Speaker** | ❌ | ❌ | ❌ | ✅ (Base only, Kugel uses fallback) | ✅ | ❌ | ❌ | ❌ | ❌ | ⚠️ (Plus variant speaker attribution / diarization) | ❌ | ❌ | ✅ Native dialogue is default; optional custom mode generates each [Character] segment independently as local speaker 0 | ❌ | ❌ | ✅ (TTSD v1.0; 1-5 speakers) | ❌ |
|
||||
| **Emotion Control** | ❌ | ❌ | ⚠️ (v2 tags - doesn't work) | ❌ | ⚠️ (via prompt) | ✅ (native inline tags) | ✅ (8 emotions) | ⚠️ (via instruct) | ⚠️ (via instruct) | ❌ | ✅ (14 emotions) | ❌ | ✅ Free-form inline natural-language tags | ❌ | ⚠️ (voice-design instruct + inline non-verbal tags) | ❌ | ❌ |
|
||||
| **Native Long-form** | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Configurable 4K-32K native context; suite text chunking is bypassed | ❌ | ✅ (uses upstream audio_chunk_duration / audio_chunk_threshold orchestration; bypasses suite char-based chunk splitting) | ✅ (TTSD/Delay long-form; use chunk orchestration for very long inputs) | N/A |
|
||||
| **Community Finetunes** | ✅ | ✅ | ✅ | ✅ KugelAudio, Hindi | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (LoRA adapter inference supported; initial integrated LoRA training support added for MOSS-TTS Delay 8B) | ✅ |
|
||||
| **VRAM Efficient** | ✅ | ✅ | ✅ | ⚠️ (5-18GB) | ⚠️ (9GB) | ⚠️ (~8-10GB) | ⚠️ (9-12GB) | ✅ (5.4GB) | ✅ (3-6GB) | ✅ (~4.6GB) | ⚠️ (7GB) | ⚠️ (~7GB total) | ⚠️ 8K context measured at ~15.2GB BF16, ~11.2GB FP8, ~11.2GB BNB INT8, or ~8.9GB BNB NF4; BF16 codec and activations; BNB is a load-time option for the official checkpoint | ⚠️ (main env works; 2B-class model is not lightweight) | ⚠️ (main env works; 2B-class model is not lightweight) | ⚠️ (Local 1.7B smaller; tokenizer is large) | ✅ |
|
||||
| **Speed/Performance** | ✅ Very Fast | ✅ Fast | ✅ Fast | ⚠️ | ⚠️ | ⚠️ CUDA recommended | ⚠️ | ✅ Fast | ⚠️ | ⚠️ Moderate | ⚠️ | ✅ Fast (diffusion, realtime-capable) | ✅ Main-environment subprocess with reliable teardown; local compile measurements: ~40 it/s BF16, ~11.8 it/s NF4 at ~8.9GB VRAM, and ~3.7 it/s INT8 at ~11.2GB VRAM; quality comparison pending | ⚠️ Moderate; mf variant is faster | ✅ Fast; upstream reports sub-realtime RTF | ✅ Fast with CUDA/FlashAttention | ✅ Fast |
|
||||
| **No Narrator Required** | ❌ | ✅ (default speaker) | ✅ (default speaker) | ✅ (zero-shot / default speaker) | ✅ (basic TTS if no narrator/reference is provided) | ✅ (zero-shot) | ❌ | ✅ (cross-lingual or instruct mode) | ✅ (Base default voice or CustomVoice presets) | N/A | ❌ | ❌ | ✅ Reference audio is optional | ✅ (default speaker) | ✅ (native default voice; instruct also works without narrator) | ✅ (direct TTS and prompt-only generation) | N/A |
|
||||
@@ -0,0 +1,35 @@
|
||||
# Fish Audio S2 Pro Inline Tags
|
||||
|
||||
Use the suite's public angle-bracket syntax for Fish's free-form instructions:
|
||||
|
||||
```text
|
||||
<whisper in small voice>Hello. <professional broadcast tone>Good evening.
|
||||
```
|
||||
|
||||
The Fish adapter converts these to native `[...]` instructions only when text reaches the Fish engine. Do not write Fish tags with square brackets in suite text because `[Character]` is reserved for character switching.
|
||||
|
||||
Fish also supports normal suite tags such as `[pause:500ms]`, character switching, and per-segment parameters. Those are processed by the suite before Fish inference.
|
||||
|
||||
## Language Prompting
|
||||
|
||||
Fish has no native language dropdown or parameter. The engine can instead prepend a natural inline instruction from the suite's resolved segment language:
|
||||
|
||||
- `language_prompting = Auto Inline Tag`: non-English resolved languages become tags such as `<German>` or `<French>`
|
||||
- `language_prompting = Off`: no automatic language instruction is added
|
||||
|
||||
English is only added when the user explicitly requested it with a language tag such as `[en:Bob]` or `[English:Bob]`. Implicit/default English stays untagged.
|
||||
|
||||
## Character Switching Modes
|
||||
|
||||
The engine defaults to `Native Multi-Speaker`. All `[Character]` turns in a generated block are sent through one Fish dialogue request, preserving native multi-turn context and long-form behavior.
|
||||
|
||||
Select `Custom Character Switching` to generate every parsed character segment independently. Each call uses only that character's reference and is remapped to Fish speaker 0. This can reduce speaker leakage, but it requires more calls and does not preserve context between character turns. SRT subtitle boundaries and timing are preserved in both modes.
|
||||
|
||||
The engine UI separates checkpoint choice from load-time quantization:
|
||||
|
||||
- `model_variant`: `s2-pro` or the separate `s2-pro-fp8` checkpoint
|
||||
- `quantization`: `none`, `bnb_int8`, or `bnb_nf4` for on-the-fly quantization of the official `s2-pro` checkpoint
|
||||
|
||||
The BNB options reuse the official files and do not download a second model copy.
|
||||
|
||||
The S2 Pro weights use the Fish Audio Research License: research and non-commercial use are allowed; commercial use requires a separate Fish Audio license.
|
||||
@@ -12,15 +12,27 @@ IndexTTS-2 supports multiple emotion control methods that can be combined for so
|
||||
- **Text Emotion**: AI-powered QwenEmotion analysis from text descriptions with dynamic templates
|
||||
- **Character Tag Emotions**: Per-character emotion control using `[Character:emotion_ref]` syntax
|
||||
|
||||
## Emotion Control Priority
|
||||
|
||||
You can only connect to the Engine node one source of control emotion: Either audio, text, or vectors.
|
||||
|
||||
When using tags on the text iself, **Character tag emotions** (highest priority) - `[Alice:angry_bob]` overrides all other emotion control settings for that character segment
|
||||
## Emotion Control Inputs and Blending
|
||||
|
||||
The IndexTTS-2 Engine has two emotion inputs:
|
||||
|
||||
- **`emotion_control`**: vector controls or Qwen text emotion
|
||||
- **`emotion_audio`**: an audio reference from an AUDIO or Character Voices node
|
||||
|
||||
Both inputs may be connected at the same time. Text emotion is analyzed into an
|
||||
8-value vector; audio emotion remains an audio-derived conditioning signal. The
|
||||
engine blends the two signals in its latent emotion-conditioning space rather
|
||||
than converting the audio into the eight visible vector values.
|
||||
|
||||
For backward compatibility, the original `emotion_control` socket still accepts
|
||||
legacy audio connections, but new workflows should use `emotion_audio` for
|
||||
audio references. A character tag such as `[Alice:angry_bob]` supplies a
|
||||
segment-local audio reference and can also be combined with vector/text emotion
|
||||
for that segment.
|
||||
|
||||
## Method 1: Direct Audio Reference
|
||||
|
||||
Connect any audio file directly to the IndexTTS-2 Engine's `emotion_control` input.
|
||||
Connect any audio file directly to the IndexTTS-2 Engine's `emotion_audio` input.
|
||||
|
||||
**How it works:**
|
||||
|
||||
@@ -37,7 +49,7 @@ Connect any audio file directly to the IndexTTS-2 Engine's `emotion_control` inp
|
||||
**Example:**
|
||||
|
||||
```
|
||||
AUDIO node → IndexTTS-2 Engine (emotion_control)
|
||||
AUDIO node → IndexTTS-2 Engine (emotion_audio)
|
||||
```
|
||||
|
||||
## Method 2: Character Voices Audio Reference
|
||||
@@ -48,7 +60,7 @@ Use the `opt_narrator` output from the 🎭 Character Voices node as an emotion
|
||||
|
||||
1. Add a 🎭 Character Voices node
|
||||
2. Select a voice with the desired emotional expression
|
||||
3. Connect `opt_narrator` output to IndexTTS-2 Engine `emotion_control` input
|
||||
3. Connect `opt_narrator` output to IndexTTS-2 Engine `emotion_audio` input
|
||||
|
||||
**Advantages:**
|
||||
|
||||
@@ -59,12 +71,13 @@ Use the `opt_narrator` output from the 🎭 Character Voices node as an emotion
|
||||
**Example workflow:**
|
||||
|
||||
```
|
||||
🎭 Character Voices (David_Attenborough) → opt_narrator → IndexTTS-2 Engine (emotion_control)
|
||||
🎭 Character Voices (David_Attenborough) → opt_narrator → IndexTTS-2 Engine (emotion_audio)
|
||||
```
|
||||
|
||||
## Method 3: Emotion Vectors
|
||||
|
||||
Use the 🌈 IndexTTS-2 Emotion Vectors node for precise manual control over 8 different emotions.
|
||||
## Method 3: Emotion Vectors
|
||||
|
||||
Use the 🌈 IndexTTS-2 Emotion Vectors node for precise manual control over 8 different emotions.
|
||||
Connect its `emotion_control` output to the IndexTTS-2 Engine's `emotion_control` input.
|
||||
|
||||
**Available emotions:**
|
||||
|
||||
@@ -84,9 +97,10 @@ Use the 🌈 IndexTTS-2 Emotion Vectors node for precise manual control over 8 d
|
||||
- Start with single emotions, then experiment with combinations
|
||||
- Use the `random` buttom to get a completely random emotion pattern. Might be too strong.
|
||||
|
||||
## Method 4: Text Emotion (Dynamic Analysis)
|
||||
|
||||
Use the 🌈 IndexTTS-2 Text Emotion node for AI-powered emotion analysis with dynamic templates.
|
||||
## Method 4: Text Emotion (Dynamic Analysis)
|
||||
|
||||
Use the 🌈 IndexTTS-2 Text Emotion node for AI-powered emotion analysis with dynamic templates.
|
||||
Connect its `emotion_control` output to the IndexTTS-2 Engine's `emotion_control` input.
|
||||
|
||||
### Static Text Emotion
|
||||
|
||||
@@ -124,11 +138,104 @@ Analysis: "Worried parent speaking: Where have you been?"
|
||||
Result: Anxious, concerned vocal expression
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Character Tag Emotion Control
|
||||
|
||||
Control emotions per character using inline tags in your text: `[Character:emotion_ref]`
|
||||
---
|
||||
|
||||
## Combining Audio with Vectors or Text
|
||||
|
||||
Connect both emotion sources when you want an audio performance to provide the
|
||||
base delivery while vectors or Qwen text analysis add a targeted emotional
|
||||
direction:
|
||||
|
||||
```text
|
||||
🎭 Character Voices (opt_narrator) ──→ emotion_audio
|
||||
🌈 Emotion Vectors or Text Emotion ──→ emotion_control
|
||||
IndexTTS-2 Engine
|
||||
```
|
||||
|
||||
`emotion_alpha` is the shared overall emotion-intensity control. The audio
|
||||
reference and vector/text signal are blended during IndexTTS-2 conditioning;
|
||||
they are not generated as two separate voices and mixed afterward.
|
||||
|
||||
For example, an audio reference can provide a natural speaking style while a
|
||||
`[sad:+0.2|calm:-0.1]` inline adjustment adds a restrained sadness to one
|
||||
segment. A Qwen text preset can be used the same way.
|
||||
|
||||
Character audio and inline emotion parameters can share one tag:
|
||||
|
||||
```text
|
||||
[Bob:br_ivan_raiva3|sad:+0.25|calm:-0.10] Bob speaks with a restrained overlay.
|
||||
[Bob:br_ivan_raiva3|emotion:"quiet grief masking frustration"] Bob uses Qwen text emotion too.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Inline Emotion Switching
|
||||
|
||||
Numeric emotion tags can replace or adjust the vector for one text segment:
|
||||
|
||||
```text
|
||||
[sad:0.7|calm:0.2] This uses explicit absolute values.
|
||||
[sad:+0.3|calm:-0.2] This modifies the connected vector.
|
||||
[vector:0,0,0.7,0,0,0.4,0,0.2] This supplies all eight absolute values.
|
||||
[vector:+0,+0,+0.3,+0,+0,+0,+0,-0.2] This supplies eight deltas.
|
||||
```
|
||||
|
||||
The full-vector order is `happy, angry, sad, afraid, disgusted, melancholic,
|
||||
surprised, calm`. Unsigned named values are absolute; `+` and `-` named values
|
||||
are relative. A full vector is relative only when every value carries a sign.
|
||||
|
||||
Text-emotion controls support saved presets and quoted descriptions:
|
||||
|
||||
```text
|
||||
[emotion:restrained_anger] Text using a saved preset.
|
||||
[emotion:"Restrained anger masking disappointment"] Direct text control.
|
||||
[emotion:"Analyze this delivery as nervous anticipation: {seg}"] Dynamic control.
|
||||
```
|
||||
|
||||
Inline controls override connected global vector/text values for that segment.
|
||||
A character audio emotion reference such as `[Alice:sad_reference]` replaces the
|
||||
global audio reference for that segment, but it can still blend with the
|
||||
segment's vector/text control. Inline settings revert at the next segment and do
|
||||
not mutate the connected vector.
|
||||
|
||||
The TTS Tag Editor provides the same interactive radar used by the IndexTTS-2
|
||||
Emotion Vectors node. Click an existing numeric emotion tag to open its radar
|
||||
as a contextual popover beside the tag. Create tags from **Inline Tags →
|
||||
IndexTTS-2**, and use **Manage Emotion Presets** in its Text Emotion section for
|
||||
the preset library. Text and vector presets are stored in
|
||||
`models/TTS/IndexTTS/emotion_presets.json`.
|
||||
|
||||
Radar changes appear in the editor text immediately. Intermediate drag/input
|
||||
states are not added to undo history: closing the popover commits one undoable
|
||||
change, while Cancel or Escape restores the tag exactly as it was when opened.
|
||||
|
||||
The editor's **Inline Tags** tab also includes an **IndexTTS-2** engine panel for
|
||||
inserting full absolute/delta vectors, named emotion values, saved text presets,
|
||||
quoted descriptions, and dynamic descriptions containing `{seg}`.
|
||||
|
||||
Emotion controls can be composed directly on character tags: place the caret on
|
||||
`[Bob:audio_reference]` and add a vector or text emotion to append it as a pipe
|
||||
parameter.
|
||||
|
||||
The named-emotion panel includes a magnitude slider and a press-drag-release
|
||||
radial picker: direction chooses the emotion and distance chooses its value.
|
||||
The operation dropdown determines whether that value is absolute, a positive
|
||||
delta, or a negative delta. Saved text and vector presets refresh in the sidebar
|
||||
immediately after they are changed in the preset manager.
|
||||
|
||||
Clicking a saved `[emotion:preset_name]` tag in the editor opens a small anchored
|
||||
preset dropdown, allowing that line's preset to be swapped without opening the
|
||||
full manager. Adding an emotion control while the caret is inside a pure emotion
|
||||
tag replaces that tag; when the caret is inside a character/audio tag, the
|
||||
editor appends or updates the emotion parameter after the existing `|` fields.
|
||||
|
||||
Named tags remain readable while only some emotions are active. If radar editing
|
||||
activates all eight emotions, the editor automatically converts the result to the
|
||||
shorter ordered `[vector:...]` form.
|
||||
|
||||
## Character Tag Emotion Control
|
||||
|
||||
Control emotions per character using inline tags in your text: `[Character:emotion_ref]`
|
||||
|
||||
**Syntax:**
|
||||
|
||||
@@ -145,8 +252,9 @@ Control emotions per character using inline tags in your text: `[Character:emoti
|
||||
|
||||
```
|
||||
Hello everyone! [Alice:happy_sarah] I'm so excited to be here today!
|
||||
[Bob:angry_tom] That's completely unacceptable behavior.
|
||||
[Narrator:David] Meanwhile, in a distant galaxy...
|
||||
[Bob:angry_tom] That's completely unacceptable behavior.
|
||||
[Narrator:David] Meanwhile, in a distant galaxy...
|
||||
[Bob:br_ivan_raiva3|sad:+0.25] Bob uses an audio reference plus a vector delta.
|
||||
|
||||
*assuming happy_sarah, angry_tom and David are alias or character voices in yout folder with that name
|
||||
```
|
||||
@@ -155,11 +263,13 @@ Hello everyone! [Alice:happy_sarah] I'm so excited to be here today!
|
||||
|
||||
|
||||
|
||||
**Character tag priority:**
|
||||
|
||||
- Character tags override ALL other emotion settings for that specific character
|
||||
- Other characters use global emotion settings
|
||||
- Allows mixing different emotions in the same audio
|
||||
**Character tag behavior:**
|
||||
|
||||
- Character tags select the speaker and can provide a segment-local audio emotion reference
|
||||
- A segment-local audio reference replaces the global audio reference for that segment
|
||||
- Global or inline vector/text controls can still blend with that audio reference
|
||||
- Other characters use the global audio and vector/text settings
|
||||
- Allows mixing different emotion sources in the same audio
|
||||
|
||||
## Emotion Alpha Control
|
||||
|
||||
@@ -211,4 +321,4 @@ Welcome to our show! [Bob:serious_narrator] But first, a serious announcement.
|
||||
|
||||
---
|
||||
|
||||
This comprehensive emotion control system gives you unprecedented flexibility in creating expressive, emotionally rich TTS audio for any application.
|
||||
This comprehensive emotion control system gives you unprecedented flexibility in creating expressive, emotionally rich TTS audio for any application.
|
||||
|
||||
+105
-104
@@ -2,110 +2,110 @@
|
||||
|
||||
## Language Support by Engine
|
||||
|
||||
| Language | Code | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Dots TTS | MOSS-TTS | RVC |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 🇺🇸 **English** | EN | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
|
||||
| 🇨🇳 **Chinese** | ZH | ❌ | ❌ | ✅ | ✅ | ✅ (Mandarin) | ✅ | ✅ | ✅ + 18 dialects | ✅ | ❌ | ✅ (Mandarin + Sichuanese, Cantonese) | ❌ | ✅ (Mandarin; official also exposes YUE separately outside this matrix) | ✅ | ✅ |
|
||||
| 🇩🇪 **German** | DE | ✅ | ✅ (×3) | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ |
|
||||
| 🇪🇸 **Spanish** | ES | ✅ | ❌ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ |
|
||||
| 🇫🇷 **French** | FR | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ |
|
||||
| 🇮🇹 **Italian** | IT | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
|
||||
| 🇯🇵 **Japanese** | JA | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ✅ ? | ✅ | ✅ | ✅ (4.1-2b supports Japanese; plus variant does not) | ✅ | ❌ | ✅ | ✅ | ✅ |
|
||||
| 🇰🇷 **Korean** | KO | ❌ | ✅ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ |
|
||||
| 🇷🇺 **Russian** | RU | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
|
||||
| 🇧🇷 **Portuguese** | PT | ✅ (BR) | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ (EU/BR*) | ✅ | ❌ | ❌ | ✅ (Official PT tag is generic; upstream does not expose separate PT-BR/PT-PT tags and it may lean more European Portuguese than Brazilian Portuguese) | ✅ | ✅ |
|
||||
| 🇵🇱 **Polish** | PL | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
|
||||
| 🇮🇳 **Hindi** | HI | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
|
||||
| 🇪🇬 **Arabic** | AR | ❌ | ❌ | ✅ (Egyptian) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
|
||||
| 🇹🇷 **Turkish** | TR | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
|
||||
| 🇹🇭 **Thai** | TH | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
|
||||
| 🇳🇴 **Norwegian** | NO | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇻🇳 **Vietnamese** | VI | ❌ | ❌ | ✅ (Viterbox) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
|
||||
| 🇦🇲 **Armenian** | HY | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇬🇪 **Georgian** | KA | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇩🇰 **Danish** | DA | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
|
||||
| 🇫🇮 **Finnish** | FI | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
|
||||
| 🇬🇷 **Greek** | EL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
|
||||
| 🇮🇱 **Hebrew** | HE | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇾 **Malay** | MS | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇱 **Dutch** | NL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
|
||||
| 🇸🇪 **Swedish** | SV | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
|
||||
| 🇰🇪 **Swahili** | SW | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Afrikaans** | AF | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇱 **Albanian** | SQ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Assamese** | AS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Asturian** | AST | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇿 **Azerbaijani** | AZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇺 **Bashkir** | BA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Basque** | EU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇾 **Belarusian** | BE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇩 **Bengali** | BN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇦 **Bosnian** | BS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇬 **Bulgarian** | BG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Catalan** | CA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇭 **Cebuano** | CEB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇶 **Central Kurdish** | CKB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇷 **Croatian** | HR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇿 **Czech** | CS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇺 **Eastern Mari** | MHR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🌍 **Esperanto** | EO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇪 **Estonian** | ET | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Galician** | GL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Gujarati** | GU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇹 **Haitian Creole** | HT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇬 **Hausa** | HA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇺 **Hungarian** | HU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇩 **Indonesian** | ID | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇩 **Javanese** | JV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Kannada** | KN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇿 **Kazakh** | KK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇼 **Kinyarwanda** | RW | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇬 **Kyrgyz** | KY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇻 **Latvian** | LV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇩 **Lingala** | LN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇹 **Lithuanian** | LT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇪 **Luo** | LUO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇰 **Macedonian** | MK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Malayalam** | ML | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇹 **Maltese** | MT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇿 **Māori** | MI | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Marathi** | MR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇳 **Mongolian** | MN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇵 **Nepali** | NE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇫🇷 **Occitan** | OC | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇷 **Persian** | FA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇴 **Romanian** | RO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Sepedi** | NSO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇸 **Serbian** | SR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇼 **Shona** | SN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇰 **Slovak** | SK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇮 **Slovene** | SL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇭 **Tagalog** | TL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇹🇯 **Tajik** | TG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Tamil** | TA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Telugu** | TE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇦 **Ukrainian** | UK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇰 **Urdu** | UR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇳 **Uyghur** | UG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇿 **Uzbek** | UZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Xhosa** | XH | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Zulu** | ZU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇼 **Chichewa/Nyanja** | NY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Eastern Punjabi** | PA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇬 **Ganda** | LG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇸 **Icelandic** | IS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇪 **Irish** | GA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇩🇿 **Kabyle** | KAB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇻 **Kabuverdianu** | KEA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇪 **Kamba** | KAM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇻🇦 **Latin** | LA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇺 **Luxembourgish** | LB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇹 **Oromo** | OM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇫 **Pashto** | PS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇰 **Sindhi** | SD | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇴 **Somali** | SO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇴 **Umbundu** | UMB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🏴 **Welsh** | CY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| Language | Code | F5-TTS | ChatterBox | ChatterBox 23L | VibeVoice | Higgs Audio 2 | Higgs Audio v3 | IndexTTS-2 | CosyVoice3 | Qwen3-TTS | Granite ASR | Step Audio EditX | Echo-TTS | Fish Audio S2 Pro | Dots TTS | OmniVoice | MOSS-TTS | RVC |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 🇺🇸 **English** | EN | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ Tier 1 | ✅ | ✅ | ✅ | ✅ |
|
||||
| 🇨🇳 **Chinese** | ZH | ❌ | ❌ | ✅ | ✅ | ✅ (Mandarin) | ✅ | ✅ | ✅ + 18 dialects | ✅ | ❌ | ✅ (Mandarin + Sichuanese, Cantonese) | ❌ | ✅ Tier 1 | ✅ (Mandarin; official also exposes YUE separately outside this matrix) | ✅ | ✅ | ✅ |
|
||||
| 🇩🇪 **German** | DE | ✅ | ✅ (×3) | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ | ✅ | ✅ |
|
||||
| 🇪🇸 **Spanish** | ES | ✅ | ❌ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ | ✅ | ✅ |
|
||||
| 🇫🇷 **French** | FR | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ | ✅ | ✅ |
|
||||
| 🇮🇹 **Italian** | IT | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ |
|
||||
| 🇯🇵 **Japanese** | JA | ✅ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ✅ ? | ✅ | ✅ | ✅ (4.1-2b supports Japanese; plus variant does not) | ✅ | ❌ | ✅ Tier 1 | ✅ | ✅ | ✅ | ✅ |
|
||||
| 🇰🇷 **Korean** | KO | ❌ | ✅ | ✅ | ✅ (Kugel) | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ Tier 2 | ✅ | ✅ | ✅ | ✅ |
|
||||
| 🇷🇺 **Russian** | RU | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ | ✅ | ✅ |
|
||||
| 🇧🇷 **Portuguese** | PT | ✅ (BR) | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ✅ (EU/BR*) | ✅ | ❌ | ❌ | ✅ Tier 2 | ✅ (Official PT tag is generic; upstream does not expose separate PT-BR/PT-PT tags and it may lean more European Portuguese than Brazilian Portuguese) | ✅ (generic PT; official language space is much broader than this matrix) | ✅ | ✅ |
|
||||
| 🇵🇱 **Polish** | PL | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ |
|
||||
| 🇮🇳 **Hindi** | HI | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇪🇬 **Arabic** | AR | ❌ | ❌ | ✅ (Egyptian) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ Tier 2 | ✅ | ✅ | ✅ | ✅ |
|
||||
| 🇹🇷 **Turkish** | TR | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ |
|
||||
| 🇹🇭 **Thai** | TH | ✅ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇳🇴 **Norwegian** | NO | ❌ | ✅ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
|
||||
| 🇻🇳 **Vietnamese** | VI | ❌ | ❌ | ✅ (Viterbox) | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇦🇲 **Armenian** | HY | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
|
||||
| 🇬🇪 **Georgian** | KA | ❌ | ✅ | ❌ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
|
||||
| 🇩🇰 **Danish** | DA | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
|
||||
| 🇫🇮 **Finnish** | FI | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇬🇷 **Greek** | EL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ |
|
||||
| 🇮🇱 **Hebrew** | HE | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
|
||||
| 🇲🇾 **Malay** | MS | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
|
||||
| 🇳🇱 **Dutch** | NL | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| 🇸🇪 **Swedish** | SV | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ |
|
||||
| 🇰🇪 **Swahili** | SW | ❌ | ❌ | ✅ | ✅ (Kugel) | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ |
|
||||
| 🇿🇦 **Afrikaans** | AF | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇱 **Albanian** | SQ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Assamese** | AS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Asturian** | AST | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇿 **Azerbaijani** | AZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇺 **Bashkir** | BA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Basque** | EU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇾 **Belarusian** | BE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇩 **Bengali** | BN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇦 **Bosnian** | BS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇧🇬 **Bulgarian** | BG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Catalan** | CA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇭 **Cebuano** | CEB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇶 **Central Kurdish** | CKB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇷 **Croatian** | HR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇿 **Czech** | CS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇺 **Eastern Mari** | MHR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🌍 **Esperanto** | EO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇪 **Estonian** | ET | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇸 **Galician** | GL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Gujarati** | GU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇹 **Haitian Creole** | HT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇬 **Hausa** | HA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇭🇺 **Hungarian** | HU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇩 **Indonesian** | ID | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇩 **Javanese** | JV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Kannada** | KN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇿 **Kazakh** | KK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇼 **Kinyarwanda** | RW | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇬 **Kyrgyz** | KY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇻 **Latvian** | LV | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇩 **Lingala** | LN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇹 **Lithuanian** | LT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇪 **Luo** | LUO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇰 **Macedonian** | MK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Malayalam** | ML | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇹 **Maltese** | MT | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇿 **Māori** | MI | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Marathi** | MR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇳 **Mongolian** | MN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇳🇵 **Nepali** | NE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇫🇷 **Occitan** | OC | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇷 **Persian** | FA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇴 **Romanian** | RO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Sepedi** | NSO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇷🇸 **Serbian** | SR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇼 **Shona** | SN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇰 **Slovak** | SK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇮 **Slovene** | SL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇭 **Tagalog** | TL | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇹🇯 **Tajik** | TG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Tamil** | TA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Telugu** | TE | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇦 **Ukrainian** | UK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇰 **Urdu** | UR | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇳 **Uyghur** | UG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇿 **Uzbek** | UZ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Xhosa** | XH | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇿🇦 **Zulu** | ZU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (polished) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇲🇼 **Chichewa/Nyanja** | NY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇳 **Eastern Punjabi** | PA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇺🇬 **Ganda** | LG | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇸 **Icelandic** | IS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇮🇪 **Irish** | GA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇩🇿 **Kabyle** | KAB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇨🇻 **Kabuverdianu** | KEA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇰🇪 **Kamba** | KAM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇻🇦 **Latin** | LA | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇱🇺 **Luxembourgish** | LB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇪🇹 **Oromo** | OM | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇫 **Pashto** | PS | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇵🇰 **Sindhi** | SD | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇸🇴 **Somali** | SO | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🇦🇴 **Umbundu** | UMB | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| 🏴 **Welsh** | CY | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (usable) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
|
||||
**Notes:**
|
||||
|
||||
@@ -114,5 +114,6 @@
|
||||
- **Higgs Audio v3**: Official model card lists 102 languages total: 85 polished-production languages and 17 usable but less polished languages.
|
||||
- **IndexTTS-2**: Trained on 55K+ hours - ZH, EN, JA primary.
|
||||
- **MOSS-TTS**: Official list includes ZH, EN, DE, ES, FR, JA, IT, HU, KO, RU, FA, AR, PL, PT, CS, DA, SV, EL, TR; HU/FA/CS are not separate columns in this matrix.
|
||||
- **OmniVoice**: Official model supports 600+ languages; this matrix only shows the suite's comparison subset.
|
||||
- **RVC**: Language-agnostic voice conversion/post-processing; it is marked supported for every language row.
|
||||
- Qwen3-TTS: Supports pt-BR only with instruction (voice design or harcoded voices). Base can't do instructions, so it will always ouput pt-PT.
|
||||
@@ -114,6 +114,13 @@ Use this as the canonical list of model repositories/links for offline setup.
|
||||
| echo-tts-base (model + PCA state) | [jordand/echo-tts-base](https://huggingface.co/jordand/echo-tts-base) | ~5.3GB | ✅ | pytorch_model.safetensors + pca_state.safetensors |
|
||||
| fish-s1-dac-min (audio codec) | [jordand/fish-s1-dac-min](https://huggingface.co/jordand/fish-s1-dac-min) | ~1.8GB | ✅ | pytorch_model.safetensors — audio codec required by Echo-TTS |
|
||||
|
||||
## Fish Audio S2 Pro
|
||||
|
||||
| Component | Source | Size | Auto-Download | Notes |
|
||||
|---|---|---|---|---|
|
||||
| S2 Pro | [fishaudio/s2-pro](https://huggingface.co/fishaudio/s2-pro) | ~10.3GB | ✅ | Official 4B model and codec; BNB INT8/NF4 are optional load-time quantization modes that reuse these files; non-commercial license |
|
||||
| S2 Pro FP8 | [drbaph/s2-pro-fp8](https://huggingface.co/drbaph/s2-pro-fp8) | ~8.0GB | ✅ | Community weight-only FP8 checkpoint; BF16 activations; RTX 4090/5090-class CUDA GPU required |
|
||||
|
||||
## Dots TTS
|
||||
|
||||
| Component | Source | Size | Auto-Download | Notes |
|
||||
@@ -122,6 +129,12 @@ Use this as the canonical list of model repositories/links for offline setup.
|
||||
| dots.tts-soar | [rednote-hilab/dots.tts-soar](https://huggingface.co/rednote-hilab/dots.tts-soar) | ~6GB | ✅ | Official SOAR checkpoint for higher-quality zero-shot cloning |
|
||||
| dots.tts-mf | [rednote-hilab/dots.tts-mf](https://huggingface.co/rednote-hilab/dots.tts-mf) | ~6GB | ✅ | Official MeanFlow-distilled checkpoint for faster inference |
|
||||
|
||||
## OmniVoice
|
||||
|
||||
| Component | Source | Size | Auto-Download | Notes |
|
||||
|---|---|---|---|---|
|
||||
| OmniVoice | [k2-fsa/OmniVoice](https://huggingface.co/k2-fsa/OmniVoice) | ~3.7GB | ✅ | Official main model including bundled audio_tokenizer assets |
|
||||
|
||||
## MOSS-TTS
|
||||
|
||||
| Component | Source | Size | Auto-Download | Notes |
|
||||
|
||||
@@ -316,6 +316,33 @@ Notes:
|
||||
- Both components are required and auto-downloaded on first use.
|
||||
- License: CC-BY-NC-SA (non-commercial).
|
||||
|
||||
## Fish Audio S2 Pro
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/fish_audio_s2_pro/
|
||||
├── codec.pth
|
||||
├── config.json
|
||||
├── model-00001-of-00002.safetensors
|
||||
├── model-00002-of-00002.safetensors
|
||||
├── model.safetensors.index.json
|
||||
└── tokenizer.json
|
||||
```
|
||||
|
||||
Optional FP8 variant:
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/fish_audio_s2_pro_fp8/
|
||||
├── codec.pth
|
||||
├── config.json
|
||||
├── model.safetensors
|
||||
├── quantization_info.json
|
||||
└── tokenizer.json
|
||||
```
|
||||
|
||||
The complete official repository metadata and tokenizer files are downloaded alongside these files. License: Fish Audio Research License (non-commercial without a separate commercial license).
|
||||
|
||||
The `s2-pro-bnb-int8` and `s2-pro-bnb-nf4` options reuse `fish_audio_s2_pro/` and quantize its official checkpoint while loading. They do not download another model copy and require `bitsandbytes`.
|
||||
|
||||
## Dots TTS
|
||||
|
||||
```text
|
||||
@@ -349,3 +376,32 @@ Notes:
|
||||
- Downloads into a dedicated `dots_tts/` folder.
|
||||
- Native sample rate is 48kHz.
|
||||
- Main-environment support works on Transformers 5; on Windows, `normalize_text` falls back to no-op if `WeTextProcessing` is unavailable.
|
||||
|
||||
## OmniVoice
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/omnivoice/
|
||||
└── OmniVoice/
|
||||
```
|
||||
|
||||
Main model contents:
|
||||
|
||||
```text
|
||||
ComfyUI/models/TTS/omnivoice/OmniVoice/
|
||||
├── chat_template.jinja
|
||||
├── config.json
|
||||
├── model.safetensors
|
||||
├── tokenizer.json
|
||||
├── tokenizer_config.json
|
||||
└── audio_tokenizer/
|
||||
├── config.json
|
||||
├── model.safetensors
|
||||
└── preprocessor_config.json
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Downloads into a dedicated `omnivoice/` folder.
|
||||
- Native sample rate is 24kHz.
|
||||
- The main OmniVoice model stays in the main Transformers 5 environment.
|
||||
- Voice cloning requires reference audio plus explicit reference text in this suite.
|
||||
|
||||
@@ -9,6 +9,7 @@ For related topics, also see:
|
||||
- [Step Audio EditX Inline Tags User Guide](INLINE_EDIT_TAGS_USER_GUIDE.md)
|
||||
- [Higgs Audio v3 Inline Tags](HIGGS_AUDIO_V3_INLINE_TAGS.md)
|
||||
- [CosyVoice3 Tags Guide](COSYVOICE3_TAGS_GUIDE.md)
|
||||
- [OmniVoice Native Tags Guide](OMNIVOICE_TAGS_GUIDE.md)
|
||||
|
||||
## What This Editor Is For
|
||||
|
||||
@@ -19,7 +20,7 @@ It supports:
|
||||
- Character switching tags
|
||||
- Language switching tags
|
||||
- Per-segment parameter overrides
|
||||
- Engine-aware inline tags for Step Audio EditX, Higgs Audio v3, and CosyVoice3
|
||||
- Engine-aware inline tags for Step Audio EditX, Higgs Audio v3, CosyVoice3, and OmniVoice
|
||||
- Presets and edit history
|
||||
- SRT-aware highlighting and timing editing
|
||||
|
||||
@@ -80,8 +81,22 @@ CosyVoice3 examples:
|
||||
<laughing>that was funny</laughing>
|
||||
```
|
||||
|
||||
OmniVoice examples:
|
||||
|
||||
```text
|
||||
<laughter>
|
||||
<sigh>
|
||||
<question-ei>
|
||||
```
|
||||
|
||||
Use the dedicated inline tag controls in the sidebar when you do not want to type these by hand.
|
||||
|
||||
Important differences:
|
||||
|
||||
- `Step Audio EditX` tags are post-process controls
|
||||
- `Higgs Audio v3`, `CosyVoice3`, and `OmniVoice` tags are native generation controls
|
||||
- `OmniVoice` editor insertion uses suite-default angle-tag aliases and the processor converts them internally to official native tags during generation
|
||||
|
||||
## SRT Editing
|
||||
|
||||
When the text looks like valid SRT, the editor highlights subtitle numbers and timings differently and enables subtitle-specific editing tools.
|
||||
|
||||
@@ -26,6 +26,10 @@ Produce a short scope document with:
|
||||
8. What manual tests will prove the integration works?
|
||||
```
|
||||
|
||||
## Runtime Decision Order
|
||||
|
||||
Test the official package with `--no-deps` in the main T5 environment first. Prefer a small compatibility patch when practical; use the existing shared T4 SHRED runtime only if T5 is genuinely incompatible. Never create another environment or download/reinstall Torch or Transformers without explicit maintainer approval.
|
||||
|
||||
## Node Types To Choose From
|
||||
|
||||
Decide whether the engine needs:
|
||||
|
||||
@@ -91,13 +91,43 @@ Follow this order:
|
||||
13. Add interrupt checks in long loops.
|
||||
14. Add progress feedback for long generation.
|
||||
15. Update docs/YAML metadata.
|
||||
16. Run manual ComfyUI tests.
|
||||
16. Run automated and live ComfyUI validation. Use FL-MCP-assisted validation when it is installed and connected; otherwise perform the same checks manually. Follow `tests/FL_MCP_VALIDATION.md`.
|
||||
17. Run the required parity checklist.
|
||||
|
||||
## Live ComfyUI Validation Rule
|
||||
|
||||
Passing imports or pytest is not enough for a new engine. Validate it in the canonical Windows ComfyUI installation after implementation.
|
||||
|
||||
If [ComfyUI_FL-MCP](https://github.com/filliptm/ComfyUI_FL-MCP) is available, the LLM should use it to inspect and operate the live ComfyUI instance. Treat it as an optional test driver, not a project dependency and not a substitute for the existing test suite.
|
||||
|
||||
The LLM should:
|
||||
|
||||
- After changing Python code, restart the canonical Windows ComfyUI process before testing. An already-running process still has the old modules loaded.
|
||||
- Use PowerShell to identify the process listening on port `8188`, verify its command line belongs to the canonical ComfyUI `main.py`, stop only that process, and relaunch it with the canonical Windows Python.
|
||||
- Wait for `http://127.0.0.1:8188/system_stats` to respond before using FL-MCP.
|
||||
- Refresh the existing ComfyUI browser tab after restart and confirm the FL-MCP browser bridge has reconnected before calling canvas-only tools.
|
||||
- Confirm the engine node and the relevant unified node are registered.
|
||||
- Load or construct the smallest useful workflow.
|
||||
- Inspect workflow JSON for the expected node types, links, and widget values.
|
||||
- Queue the workflow and wait for completion.
|
||||
- Inspect execution history and report the full actionable error if execution fails.
|
||||
- Confirm the expected audio output artifact exists.
|
||||
- Capture a canvas screenshot for UI and broken-node inspection.
|
||||
- Exercise TTS Text and SRT for every TTS engine, plus any other scoped capability.
|
||||
- Record what was actually tested, what was skipped, and why.
|
||||
|
||||
Screenshots prove only visible workflow state. They do not prove that generation succeeded or that audio is correct. Execution history and output artifacts are required evidence, and the user must still judge subjective audio quality.
|
||||
|
||||
Do not install FL-MCP, alter its safety settings, or enable destructive tools unless the user authorizes it. If FL-MCP is unavailable, report that fact and follow the manual fallback in `tests/FL_MCP_VALIDATION.md`.
|
||||
|
||||
Repeat the edit, restart, reconnect, and validation cycle after every implementation fix that changes imported Python code. Frontend-only changes may require a hard browser refresh as well. Do not claim that a fix was tested against a ComfyUI process started before the fix was written.
|
||||
|
||||
## Architecture Rule
|
||||
|
||||
Unified nodes should stay thin.
|
||||
|
||||
Reference engines are examples only. Every engine must have dedicated processors and adapters; share only engine-neutral utilities.
|
||||
|
||||
Do not put hundreds of lines of engine-specific orchestration into:
|
||||
|
||||
- `nodes/unified/tts_text_node.py`
|
||||
|
||||
@@ -25,7 +25,8 @@ Downloads and dependencies:
|
||||
- Do models download into organized ComfyUI/models/TTS/ folders?
|
||||
- Did you prevent silent downloads into random cache folders?
|
||||
- Did you document dependency conflicts or install.py changes?
|
||||
- Did you explicitly decide whether this engine belongs in Main Environment or needs runtime isolation?
|
||||
- Did you test `--no-deps` on Main/T5 and simple patching before falling back to the shared T4 runtime?
|
||||
- Did you avoid creating another environment or downloading/reinstalling Torch or Transformers without explicit maintainer approval?
|
||||
- If runtime isolation is needed, did you document the default mode and the reason in YAML/README?
|
||||
|
||||
Audio format:
|
||||
@@ -79,6 +80,21 @@ Manual tests:
|
||||
- Did parameter switching work?
|
||||
- Did Clear VRAM then regenerate work, and did unload actually tear down runtime/cache state instead of only moving weights to CPU?
|
||||
- Did interrupt/cancel work in long generation?
|
||||
|
||||
Live ComfyUI integration evidence:
|
||||
- Was validation run in the canonical Windows ComfyUI environment?
|
||||
- Was ComfyUI restarted after the final Python changes, and can you show that the tested process started after those edits?
|
||||
- After restart, was the browser refreshed and the FL-MCP browser bridge confirmed connected before canvas checks?
|
||||
- If FL-MCP was available, did you follow tests/FL_MCP_VALIDATION.md and identify which MCP checks were run?
|
||||
- If FL-MCP was unavailable, did you perform and document the equivalent manual checks instead of claiming MCP validation?
|
||||
- Are the engine node and relevant unified nodes present without import or registration errors?
|
||||
- Does the saved workflow JSON contain the expected node types, links, and widget values?
|
||||
- Does the workflow load without missing/broken node state?
|
||||
- Was the workflow queued, and does execution history show successful completion or the full actionable error?
|
||||
- Does each successful run produce the expected audio output artifact?
|
||||
- Was a canvas screenshot captured for UI inspection without treating the screenshot as execution proof?
|
||||
- Did you record tested, failed, and skipped cases, including the reason for every skip?
|
||||
- Did a human assess subjective audio quality separately?
|
||||
```
|
||||
|
||||
## Important Failures This Prevents
|
||||
|
||||
@@ -462,11 +462,12 @@ Also to test, requirements and dependencies need to be added.
|
||||
- [ ] Character switching works with `[CharacterName] text`
|
||||
- [ ] Language switching works (if applicable)
|
||||
- [ ] Pause tags work with `[pause:1.5s]`
|
||||
- [ ] Caching works (same input = cached output)
|
||||
- [ ] Model auto-download works
|
||||
- [ ] VRAM management works (model unloads)
|
||||
- [ ] Different parameter combinations work
|
||||
- [ ] **Interrupt handling works** - User can stop SRT generation and it stops within ~1 segment
|
||||
- [ ] Caching works (same input = cached output)
|
||||
- [ ] Model auto-download works
|
||||
- [ ] VRAM management works (model unloads)
|
||||
- [ ] Different parameter combinations work
|
||||
- [ ] Engine prints a standard `Settings:` summary with the active generation/load parameters
|
||||
- [ ] **Interrupt handling works** - User can stop SRT generation and it stops within ~1 segment
|
||||
|
||||
### Phase 4: SRT Implementation
|
||||
|
||||
@@ -645,4 +646,4 @@ Update the engines comparison table.
|
||||
- Model management logic
|
||||
- Audio format conversion utilities
|
||||
|
||||
---
|
||||
---
|
||||
|
||||
@@ -56,6 +56,10 @@
|
||||
- **Cache key pattern**: `audio_cache.generate_cache_key(engine_type, text=..., audio_component=..., **all_params)`
|
||||
- **Duration calculation**: Update `_calculate_duration()` with engine sample rate (e.g., 24000 for Step Audio EditX, F5-TTS)
|
||||
|
||||
### Engine Settings Logging
|
||||
- **Missing standard print**: New engines should print the usual `Settings:` summary once per run with the active generation/load parameters so validation can confirm what actually executed
|
||||
- **Resolved prompt preview**: Reuse `utils.voice.character_logging.format_resolved_character_block()` for boxed text previews so logs show the voice that will actually speak, not only the parser alias
|
||||
|
||||
### Model Lifecycle - __del__ Destructor
|
||||
- **CRITICAL**: Remove `__del__` from engine classes - causes automatic unload after generation ends (when object goes out of scope)
|
||||
- **Pattern**: F5-TTS and ChatterBox don't have `__del__`, only IndexTTS and StepAudio did (wrong)
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
# OmniVoice Native Tags Guide
|
||||
|
||||
OmniVoice has its own official inline square-bracket control tokens upstream, but **this suite does not expose `[]` for OmniVoice tags**.
|
||||
|
||||
Inside TTS Audio Suite, you can write the suite-default angle-tag aliases in text:
|
||||
|
||||
```text
|
||||
<laughter>
|
||||
<sigh>
|
||||
<question-ei>
|
||||
```
|
||||
|
||||
The OmniVoice processor converts those aliases internally to the official OmniVoice native form before generation:
|
||||
|
||||
```text
|
||||
[laughter]
|
||||
[sigh]
|
||||
[question-ei]
|
||||
```
|
||||
|
||||
Do **not** type OmniVoice non-verbal tags in `[]` form in suite text. In this suite, `[]` belongs to character, language, parameter, and pause syntax.
|
||||
|
||||
## Supported Non-Verbal Tags
|
||||
|
||||
Official OmniVoice non-verbal meanings exposed by this suite through `<>` aliases:
|
||||
|
||||
```text
|
||||
laughter
|
||||
sigh
|
||||
confirmation-en
|
||||
question-en
|
||||
question-ah
|
||||
question-oh
|
||||
question-ei
|
||||
question-yi
|
||||
surprise-ah
|
||||
surprise-oh
|
||||
surprise-wa
|
||||
surprise-yo
|
||||
dissatisfaction-hnn
|
||||
```
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
[Alice] <laughter> You really got me there.
|
||||
[Bob] <sigh> Fine, let's try again.
|
||||
[Narrator] <question-ei> Really?
|
||||
[Narrator] <surprise-oh> I didn't expect that.
|
||||
```
|
||||
|
||||
## Important Behavior
|
||||
|
||||
- OmniVoice uses native generation tags here, not Step Audio EditX inline post-processing.
|
||||
- User-facing suite syntax stays in `<>` form for OmniVoice non-verbal tags.
|
||||
- `[]` is reserved for suite structural syntax like `[Alice]`, `[en:Alice]`, `[pause:1s]`, and parameter switching.
|
||||
- Do not rely on Step-style `<Laughter:2>` or `<emotion:happy>` semantics in the OmniVoice text path.
|
||||
- If you want Step Audio EditX as a second pass on OmniVoice output, use the separate `🎨 Audio Editor` node manually after generation.
|
||||
|
||||
## Multiline Tag Editor
|
||||
|
||||
The `🏷️ Multiline TTS Tag Editor` has a dedicated `OmniVoice` mode in the `Inline Tags` panel.
|
||||
|
||||
- The editor inserts suite-default angle-tag aliases like `<laughter>`
|
||||
- The processor converts them internally to official OmniVoice square tags during generation
|
||||
- The editor does not encourage raw OmniVoice `[]` input because `[]` is suite syntax
|
||||
|
||||
## Sources
|
||||
|
||||
This behavior follows the official OmniVoice documentation for non-verbal symbols and pronunciation control:
|
||||
|
||||
- [OmniVoice GitHub README](https://github.com/k2-fsa/OmniVoice)
|
||||
- [OmniVoice Hugging Face model card](https://huggingface.co/k2-fsa/OmniVoice)
|
||||
@@ -127,7 +127,56 @@ Parameters are applied **only to the current segment** and automatically revert
|
||||
| `cfg` | — | float | 0.0-20.0 | CFG strength |
|
||||
| `top_p` | `topp` | float | 0.0-1.0 | Nucleus sampling probability |
|
||||
| `top_k` | `topk` | int | 1-100 | Top-k sampling |
|
||||
| `emotion_alpha` | — | float | 0.0-1.0 | Emotion control strength |
|
||||
| `emotion_alpha` | — | float | 0.0-2.0 | Shared audio/vector/text emotion intensity |
|
||||
|
||||
IndexTTS-2 also supports inline emotion controls. Named unsigned values replace
|
||||
that dimension; explicitly signed values adjust the connected vector:
|
||||
|
||||
```text
|
||||
[sad:0.7|calm:0.2] Absolute values for this segment.
|
||||
[sad:+0.3|calm:-0.2] Adjust the connected vector for this segment.
|
||||
```
|
||||
|
||||
All eight values can be supplied in the official order `happy, angry, sad,
|
||||
afraid, disgusted, melancholic, surprised, calm`:
|
||||
|
||||
```text
|
||||
[vector:0,0,0.7,0,0,0.4,0,0.2] Absolute replacement.
|
||||
[vector:+0,+0,+0.3,+0,+0,+0,+0,-0.2] Relative adjustment.
|
||||
```
|
||||
|
||||
Full relative vectors require an explicit sign on every value. Results are
|
||||
clamped to IndexTTS-2's supported range and revert to the connected vector at
|
||||
the next segment.
|
||||
|
||||
Text emotion can use a saved preset or quoted text. `{seg}` is expanded with
|
||||
the current segment before QwenEmotion analysis:
|
||||
|
||||
```text
|
||||
[emotion:restrained_anger] A saved preset.
|
||||
[emotion:"Quiet grief masking frustration"] A direct description.
|
||||
[emotion:"Infer nervous anticipation from this line: {seg}"] Dynamic analysis.
|
||||
```
|
||||
|
||||
Click a numeric emotion tag in the TTS Tag Editor to open a contextual radar
|
||||
directly beside that tag. The editor also creates and manages text
|
||||
presets in `models/TTS/IndexTTS/emotion_presets.json`.
|
||||
|
||||
IndexTTS-2 has separate engine inputs for these sources: connect vector or text
|
||||
emotion to `emotion_control` and audio emotion references to `emotion_audio`.
|
||||
Both may be connected simultaneously; IndexTTS-2 blends them during emotion
|
||||
conditioning. Inline vector/text controls override the connected vector/text
|
||||
values for their segment, while `[Character:emotion_ref]` selects a
|
||||
segment-local audio reference that can still blend with vector/text emotion.
|
||||
When an emotion control is inserted with the caret inside a character/audio tag,
|
||||
the editor appends or updates it as another pipe parameter, for example
|
||||
`[Bob:br_ivan_raiva3|sad:+0.25]`.
|
||||
|
||||
The tag editor's quick-swap palette is engine-aware: parameter choices are
|
||||
filtered to the selected inline engine, while each engine's supported native
|
||||
emotion/style/prosody/sound tags use their own replacement choices. Named
|
||||
emotion presets can be swapped from the text; quoted `[emotion:"..."]` text is
|
||||
intentionally left as direct editable content rather than treated as a preset.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -207,6 +207,47 @@ This document tracks updates applied to our bundled IndexTTS-2 code from the ups
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## 2026-07-11: Upstream Audit Before Release
|
||||
|
||||
**Upstream repository checked:** `index-tts/index-tts` (`main`)
|
||||
**Upstream head observed:** `b5bd657` (2026-07-08)
|
||||
**Check performed:** 2026-07-11
|
||||
|
||||
### Relevant upstream changes reviewed
|
||||
|
||||
| Commit | Upstream change | Bundled status | Release action |
|
||||
|---|---|---|---|
|
||||
| `843972e` | Coerce QwenEmotion JSON emotion scores to `float` and reject non-numeric values clearly | **Applied** in bundled `clamp_score()` | Run a text-emotion generation with numeric-string classifier output |
|
||||
| `b154a1b` | WebUI text/vector preset save/load management | **Already covered differently** by the ComfyUI-native preset manager and `emotion_presets.json` integration | No direct port needed |
|
||||
| `b5bd657` | Expose `--accel` and `--torch-compile`, add optional acceleration extras and WebUI settings | **Applied selectively**; suite now forwards `use_accel` into the bundled GPT path, while retaining ComfyUI-owned dependency handling | Validate acceleration fallback on compatible and non-accelerated setups |
|
||||
| `7264ce2` | Improve IndexTTS-2 model resource checks and HF cache handling | **Suite-owned downloader differs** and needs a separate comparison if download failures are reported | No blind copy into bundled code |
|
||||
|
||||
### Findings
|
||||
|
||||
- No upstream change was found that invalidates the current eight-emotion vector order,
|
||||
Qwen text-emotion syntax, audio-reference blending, or the suite's inline tag format.
|
||||
- The upstream QwenEmotion string-score fix is directly relevant to the suite's text-emotion
|
||||
path and should be applied before a release.
|
||||
- The upstream acceleration work is not a drop-in replacement because this repository
|
||||
bundles and adapts IndexTTS-2. The suite already contains the acceleration modules, but
|
||||
`utils/models/unified_model_interface.py` should be checked so `use_accel` reaches the
|
||||
bundled `IndexTTS2` constructor.
|
||||
- Upstream WebUI presets are not copied verbatim: the suite's ComfyUI editor has a richer
|
||||
vector/radar, inline-tag, sidebar, and filesystem preset implementation.
|
||||
|
||||
### Release follow-up checklist
|
||||
|
||||
- [x] Apply the upstream `clamp_score()` numeric coercion.
|
||||
- [x] Pass `use_accel` through the unified IndexTTS-2 factory; validate fallback behavior.
|
||||
- [ ] Run a Qwen text-emotion generation using numeric-string classifier output.
|
||||
- [ ] Verify acceleration flags on a compatible CUDA installation and on a setup without
|
||||
optional acceleration dependencies.
|
||||
- [ ] Recheck bundled model-resource validation against the current upstream `check` logic.
|
||||
|
||||
---
|
||||
|
||||
## Next Update Check: 2025-11-20
|
||||
|
||||
**Monitoring:** Watch for commits to `indextts/infer_v2.py` in https://github.com/index-tts/index-tts
|
||||
@@ -222,4 +263,4 @@ git log --oneline 1d5d079..HEAD -- indextts/infer_v2.py
|
||||
- Additional performance optimizations
|
||||
- Bug fixes in new acceleration code
|
||||
- Breaking API changes
|
||||
- Model loading improvements
|
||||
- Model loading improvements
|
||||
|
||||
@@ -49,6 +49,24 @@ except Exception as e:
|
||||
def __init__(self, *args, **kwargs):
|
||||
raise ImportError(f"Dots TTS adapter not available: {e}")
|
||||
|
||||
try:
|
||||
from .fish_audio_s2_adapter import FishAudioS2Adapter
|
||||
FISH_AUDIO_S2_ADAPTER_AVAILABLE = True
|
||||
except Exception as e:
|
||||
FISH_AUDIO_S2_ADAPTER_AVAILABLE = False
|
||||
class FishAudioS2Adapter:
|
||||
def __init__(self, *args, **kwargs):
|
||||
raise ImportError(f"Fish Audio S2 adapter not available: {e}")
|
||||
|
||||
try:
|
||||
from .omnivoice_adapter import OmniVoiceEngineAdapter
|
||||
OMNIVOICE_ADAPTER_AVAILABLE = True
|
||||
except Exception as e:
|
||||
OMNIVOICE_ADAPTER_AVAILABLE = False
|
||||
class OmniVoiceEngineAdapter:
|
||||
def __init__(self, *args, **kwargs):
|
||||
raise ImportError(f"OmniVoice adapter not available: {e}")
|
||||
|
||||
try:
|
||||
from .moss_tts_adapter import MossTTSEngineAdapter
|
||||
MOSS_TTS_ADAPTER_AVAILABLE = True
|
||||
@@ -69,9 +87,9 @@ except Exception as e:
|
||||
|
||||
__all__ = [
|
||||
'ChatterBoxEngineAdapter', 'F5TTSEngineAdapter', 'CosyVoiceAdapter', 'EchoTTSEngineAdapter',
|
||||
'DotsTTSEngineAdapter',
|
||||
'DotsTTSEngineAdapter', 'OmniVoiceEngineAdapter',
|
||||
'MossTTSEngineAdapter', 'HiggsAudioV3EngineAdapter',
|
||||
'CHATTERBOX_ADAPTER_AVAILABLE', 'F5TTS_ADAPTER_AVAILABLE', 'COSYVOICE_ADAPTER_AVAILABLE',
|
||||
'ECHO_TTS_ADAPTER_AVAILABLE', 'DOTS_TTS_ADAPTER_AVAILABLE',
|
||||
'ECHO_TTS_ADAPTER_AVAILABLE', 'DOTS_TTS_ADAPTER_AVAILABLE', 'OMNIVOICE_ADAPTER_AVAILABLE',
|
||||
'MOSS_TTS_ADAPTER_AVAILABLE', 'HIGGS_AUDIO_V3_ADAPTER_AVAILABLE'
|
||||
]
|
||||
|
||||
@@ -23,6 +23,7 @@ from engines.cosyvoice.cosyvoice import CosyVoiceEngine
|
||||
from engines.cosyvoice.cosyvoice_downloader import cosyvoice_downloader
|
||||
from utils.text.character_parser import character_parser
|
||||
from utils.voice.discovery import get_character_mapping, get_available_characters
|
||||
from utils.voice.character_logging import resolved_character_label
|
||||
from utils.audio.cache import get_audio_cache
|
||||
|
||||
|
||||
@@ -336,7 +337,7 @@ class CosyVoiceAdapter:
|
||||
speaker_audio = char_audio
|
||||
if char_text:
|
||||
reference_text = char_text
|
||||
print(f"📖 Using character voice '{character_name}'")
|
||||
print(f"📖 Using character voice '{resolved_character_label(character_name, speaker_audio)}'")
|
||||
|
||||
# Generate cache key for this segment
|
||||
segment_cache_key = self._generate_cache_key(
|
||||
@@ -352,7 +353,7 @@ class CosyVoiceAdapter:
|
||||
# Check cache first
|
||||
cached_segment_audio = self.audio_cache.get_cached_audio(segment_cache_key)
|
||||
if cached_segment_audio:
|
||||
print(f"💾 Using cached CosyVoice3 segment for '{character_name}'")
|
||||
print(f"💾 Using cached CosyVoice3 segment for '{resolved_character_label(character_name, speaker_audio)}'")
|
||||
segment_audio = cached_segment_audio[0]
|
||||
else:
|
||||
# Convert CosyVoice paralinguistic tags from <tag> to [tag]
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
"""Adapter between suite processors and the isolated official Fish S2 runtime."""
|
||||
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
import torch
|
||||
|
||||
from engines.fish_audio_s2.downloader import FishAudioS2Downloader
|
||||
from utils.audio.audio_hash import generate_stable_audio_component
|
||||
from utils.audio.cache import get_audio_cache
|
||||
from utils.audio.processing import AudioProcessingUtils
|
||||
from utils.models.factory_config import ModelLoadConfig
|
||||
from utils.text.fish_audio_s2_tags import translate_fish_s2_inline_tags
|
||||
from utils.voice.character_logging import resolved_voice_name
|
||||
|
||||
|
||||
class FishAudioS2Adapter:
|
||||
SAMPLE_RATE = 44100
|
||||
|
||||
def __init__(self, config: Optional[Dict[str, Any]] = None):
|
||||
self.config = dict(config or {})
|
||||
self.audio_cache = get_audio_cache()
|
||||
self._model_config = None
|
||||
|
||||
def update_config(self, config):
|
||||
self.config = dict(config or {})
|
||||
|
||||
def _engine(self):
|
||||
from utils.models.unified_model_interface import unified_model_interface
|
||||
model_selection = self.config.get("model_variant", "s2-pro")
|
||||
quantization = self.config.get("quantization", "none")
|
||||
model_path = FishAudioS2Downloader.resolve_model_path(model_selection)
|
||||
model_variant = FishAudioS2Downloader.resolve_model_variant(model_selection, model_path)
|
||||
self._model_config = ModelLoadConfig(
|
||||
engine_name="fish_audio_s2", model_type="tts", model_name=model_variant,
|
||||
model_path=model_path, device=self.config.get("device", "auto"),
|
||||
runtime_mode="isolated",
|
||||
additional_params={
|
||||
"model_variant": model_variant,
|
||||
"quantization": quantization,
|
||||
"precision": self.config.get("precision", "bfloat16"),
|
||||
"compile": bool(self.config.get("compile", False)),
|
||||
"context_length": int(self.config.get("context_length", 8192)),
|
||||
},
|
||||
)
|
||||
return unified_model_interface.load_model(self._model_config)
|
||||
|
||||
def _reference(self, voice_ref):
|
||||
if not isinstance(voice_ref, dict):
|
||||
return None, "", "default_voice"
|
||||
text = (voice_ref.get("reference_text") or voice_ref.get("prompt_text") or "").strip()
|
||||
path = voice_ref.get("audio_path") or voice_ref.get("prompt_audio_path")
|
||||
audio = voice_ref.get("audio")
|
||||
if audio is None:
|
||||
audio = voice_ref.get("waveform")
|
||||
if not path and isinstance(audio, dict):
|
||||
path = AudioProcessingUtils.save_audio_to_temp_file(audio["waveform"], audio.get("sample_rate", 44100))
|
||||
elif not path and torch.is_tensor(audio):
|
||||
path = AudioProcessingUtils.save_audio_to_temp_file(audio, voice_ref.get("sample_rate", 44100))
|
||||
component = generate_stable_audio_component(audio_file_path=path) if path else "default_voice"
|
||||
return path, text, component
|
||||
|
||||
def generate_single(self, text, voice_ref, seed=0, enable_audio_cache=True, character_name=None):
|
||||
return self.generate_dialogue(
|
||||
[(0, text)], [voice_ref], seed, enable_audio_cache,
|
||||
cache_character=character_name or "narrator",
|
||||
)
|
||||
|
||||
def generate_dialogue(self, turns, voice_refs, seed=0, enable_audio_cache=True,
|
||||
cache_character="native_dialogue"):
|
||||
formatted_turns = []
|
||||
for speaker_index, turn_text in turns:
|
||||
clean_text = translate_fish_s2_inline_tags((turn_text or "").strip())
|
||||
if clean_text:
|
||||
formatted_turns.append(f"<|speaker:{speaker_index}|>{clean_text}")
|
||||
text = "\n".join(formatted_turns)
|
||||
if not text:
|
||||
return torch.zeros(1, 0)
|
||||
|
||||
references = []
|
||||
reference_labels = []
|
||||
components = []
|
||||
reference_texts = []
|
||||
for speaker_index, voice_ref in enumerate(voice_refs):
|
||||
ref_path, ref_text, component = self._reference(voice_ref)
|
||||
if ref_path:
|
||||
if not ref_text:
|
||||
raise ValueError("Fish S2 native speakers require exact reference transcripts")
|
||||
references.append({
|
||||
"audio_path": ref_path,
|
||||
"text": f"<|speaker:{speaker_index}|>{ref_text}",
|
||||
})
|
||||
reference_labels.append(
|
||||
f"local Speaker {speaker_index + 1}={resolved_voice_name(voice_ref)}"
|
||||
)
|
||||
components.append(component)
|
||||
reference_texts.append(ref_text)
|
||||
if references:
|
||||
print(f"🎤 Fish local reference order: {', '.join(reference_labels)}")
|
||||
|
||||
params = {
|
||||
"model_variant": self.config.get("model_variant", "s2-pro"),
|
||||
"quantization": self.config.get("quantization", "none"),
|
||||
"multi_speaker_mode": self.config.get("multi_speaker_mode", "Native Multi-Speaker"),
|
||||
"seed": int(seed), "normalize": bool(self.config.get("normalize", True)),
|
||||
"chunk_length": int(self.config.get("native_chunk_length", 200)),
|
||||
"max_new_tokens": int(self.config.get("max_new_tokens", 1024)),
|
||||
"top_p": float(self.config.get("top_p", 0.8)),
|
||||
"repetition_penalty": float(self.config.get("repetition_penalty", 1.1)),
|
||||
"temperature": float(self.config.get("temperature", 0.8)),
|
||||
"cache_reference": bool(self.config.get("cache_reference", True)),
|
||||
"context_length": int(self.config.get("context_length", 8192)),
|
||||
}
|
||||
cache_key = self.audio_cache.generate_cache_key(
|
||||
"fish_audio_s2", text=text, audio_component="|".join(components),
|
||||
reference_text="|".join(reference_texts), character=cache_character, **params,
|
||||
) if enable_audio_cache else None
|
||||
if cache_key:
|
||||
cached = self.audio_cache.get_cached_audio(cache_key)
|
||||
if cached:
|
||||
print("💾 Fish Audio S2: Using cached audio")
|
||||
return cached[0]
|
||||
audio, sample_rate = self._engine().generate(text=text, references=references, **params)
|
||||
if sample_rate != self.SAMPLE_RATE:
|
||||
raise RuntimeError(f"Fish S2 returned unexpected sample rate {sample_rate}")
|
||||
audio = audio.detach().float().cpu()
|
||||
if audio.dim() == 1:
|
||||
audio = audio.unsqueeze(0)
|
||||
if cache_key:
|
||||
self.audio_cache.cache_audio(cache_key, audio, audio.shape[-1] / self.SAMPLE_RATE)
|
||||
return audio
|
||||
@@ -12,7 +12,8 @@ from typing import Dict, Any, Optional, List, Union
|
||||
from engines.index_tts.index_tts import IndexTTSEngine
|
||||
from engines.index_tts.index_tts_downloader import index_tts_downloader
|
||||
from utils.text.character_parser import character_parser
|
||||
from utils.voice.discovery import get_character_mapping, get_available_characters
|
||||
from utils.voice.discovery import get_character_mapping, get_available_characters
|
||||
from utils.voice.character_logging import resolved_character_label
|
||||
from utils.audio.cache import get_audio_cache
|
||||
|
||||
|
||||
@@ -76,7 +77,26 @@ class IndexTTSAdapter:
|
||||
)
|
||||
|
||||
|
||||
def generate(self,
|
||||
@staticmethod
|
||||
def _normalize_emotion_audio(emotion_audio):
|
||||
"""Return a path or waveform dict accepted by IndexTTS-2."""
|
||||
if not isinstance(emotion_audio, dict):
|
||||
return emotion_audio
|
||||
if emotion_audio.get("audio_path"):
|
||||
print(
|
||||
"🎭 Using Character Voices emotion audio: "
|
||||
f"{emotion_audio.get('character_name', 'unknown')} -> "
|
||||
f"{emotion_audio['audio_path']}"
|
||||
)
|
||||
return emotion_audio["audio_path"]
|
||||
if "waveform" in emotion_audio:
|
||||
return emotion_audio
|
||||
nested_audio = emotion_audio.get("audio")
|
||||
if isinstance(nested_audio, dict) and "waveform" in nested_audio:
|
||||
return nested_audio
|
||||
return emotion_audio
|
||||
|
||||
def generate(self,
|
||||
text: str,
|
||||
speaker_audio: Optional[str] = None,
|
||||
emotion_audio: Optional[str] = None,
|
||||
@@ -157,19 +177,10 @@ class IndexTTSAdapter:
|
||||
# Determine final speaker and emotion audio
|
||||
final_speaker_audio = speaker_audio
|
||||
|
||||
# Handle Character Voices emotion_audio format
|
||||
if emotion_audio and isinstance(emotion_audio, dict):
|
||||
if "audio_path" in emotion_audio:
|
||||
# Character Voices format: {'audio': {...}, 'audio_path': 'path', ...}
|
||||
final_emotion_audio = emotion_audio["audio_path"]
|
||||
print(f"🎭 Using Character Voices emotion audio: {emotion_audio.get('character_name', 'unknown')} -> {final_emotion_audio}")
|
||||
elif "waveform" in emotion_audio:
|
||||
# Direct AUDIO format: {'waveform': tensor, 'sample_rate': rate}
|
||||
final_emotion_audio = emotion_audio
|
||||
else:
|
||||
final_emotion_audio = emotion_audio
|
||||
else:
|
||||
final_emotion_audio = emotion_audio
|
||||
# Normalize the two supported audio-reference shapes:
|
||||
# Character Voices returns {audio: {waveform, sample_rate}, audio_path: ...},
|
||||
# while ComfyUI AUDIO returns {waveform, sample_rate} directly.
|
||||
final_emotion_audio = self._normalize_emotion_audio(emotion_audio)
|
||||
|
||||
# Only do character mapping if we actually have character tags
|
||||
if has_character_tags:
|
||||
@@ -263,18 +274,11 @@ class IndexTTSAdapter:
|
||||
engine_kwargs['stream_return'] = stream_return
|
||||
engine_kwargs['more_segment_before'] = more_segment_before
|
||||
|
||||
# Apply consistent emotion priority: emotion_audio takes precedence over other emotion controls
|
||||
# This ensures consistent behavior whether using character tags or direct engine inputs
|
||||
if final_emotion_audio:
|
||||
# emotion_audio connected - disable other emotion controls
|
||||
final_emotion_vector = None
|
||||
final_use_emotion_text = False
|
||||
final_emotion_text = None
|
||||
else:
|
||||
# No emotion_audio - use provided emotion controls
|
||||
final_emotion_vector = emotion_vector
|
||||
final_use_emotion_text = use_emotion_text
|
||||
final_emotion_text = emotion_text
|
||||
# Audio emotion and vector/text emotion are independent conditioning
|
||||
# sources. IndexTTS-2 blends them in its latent emotion space.
|
||||
final_emotion_vector = emotion_vector
|
||||
final_use_emotion_text = use_emotion_text
|
||||
final_emotion_text = emotion_text
|
||||
|
||||
# Generate audio with OOM protection
|
||||
try:
|
||||
@@ -391,7 +395,14 @@ class IndexTTSAdapter:
|
||||
if unique_characters:
|
||||
character_mapping = get_character_mapping(list(unique_characters), engine_type="index_tts")
|
||||
|
||||
print(f"🎭 IndexTTS-2: Processing {len(segments)} character segment(s) - {', '.join([s.get('character', 'narrator') for s in segments])}")
|
||||
resolved_names = [
|
||||
resolved_character_label(
|
||||
segment.get('character', 'narrator'),
|
||||
character_mapping.get(segment.get('character', 'narrator'), (default_speaker_audio, None)),
|
||||
)
|
||||
for segment in segments
|
||||
]
|
||||
print(f"🎭 IndexTTS-2: Processing {len(segments)} character segment(s) - {', '.join(resolved_names)}")
|
||||
|
||||
for segment in segments:
|
||||
character_name = segment.get('character', 'narrator')
|
||||
@@ -407,7 +418,7 @@ class IndexTTSAdapter:
|
||||
character_audio_path = character_mapping[character_name][0]
|
||||
if character_audio_path:
|
||||
speaker_audio = character_audio_path
|
||||
print(f"📖 Using character voice '{character_name}' | Ref: '{speaker_audio}'")
|
||||
print(f"📖 Using character voice '{resolved_character_label(character_name, speaker_audio)}' | Ref: '{speaker_audio}'")
|
||||
else:
|
||||
print(f"⚠️ Character '{character_name}' has no audio reference, using default")
|
||||
|
||||
@@ -430,7 +441,7 @@ class IndexTTSAdapter:
|
||||
# Check cache first
|
||||
cached_segment_audio = self.audio_cache.get_cached_audio(segment_cache_key)
|
||||
if cached_segment_audio:
|
||||
print(f"💾 Using cached IndexTTS-2 segment for '{character_name}': '{segment_text[:30]}...'")
|
||||
print(f"💾 Using cached IndexTTS-2 segment for '{resolved_character_label(character_name, speaker_audio)}': '{segment_text[:30]}...'")
|
||||
segment_audio = cached_segment_audio[0]
|
||||
else:
|
||||
# Generate audio for this segment with OOM protection
|
||||
@@ -534,16 +545,18 @@ class IndexTTSAdapter:
|
||||
|
||||
return "\n".join(analysis_parts)
|
||||
|
||||
def _get_stable_audio_identifier(self, audio_path: str) -> str:
|
||||
"""
|
||||
Get stable identifier for audio file using centralized audio hashing.
|
||||
"""
|
||||
if not audio_path:
|
||||
return audio_path
|
||||
|
||||
# Use our centralized audio hashing utility
|
||||
from utils.audio.audio_hash import generate_stable_audio_component
|
||||
return generate_stable_audio_component(audio_file_path=audio_path)
|
||||
def _get_stable_audio_identifier(self, audio_path: str) -> str:
|
||||
"""
|
||||
Get stable identifier for audio file using centralized audio hashing.
|
||||
"""
|
||||
if not audio_path:
|
||||
return audio_path
|
||||
|
||||
# Use our centralized audio hashing utility
|
||||
from utils.audio.audio_hash import generate_stable_audio_component
|
||||
if isinstance(audio_path, dict):
|
||||
return generate_stable_audio_component(reference_audio=audio_path)
|
||||
return generate_stable_audio_component(audio_file_path=audio_path)
|
||||
|
||||
def get_supported_formats(self) -> List[str]:
|
||||
"""Get supported audio formats."""
|
||||
@@ -586,4 +599,4 @@ class IndexTTSAdapter:
|
||||
|
||||
def __del__(self):
|
||||
"""Cleanup on deletion."""
|
||||
self.unload()
|
||||
self.unload()
|
||||
|
||||
@@ -0,0 +1,297 @@
|
||||
"""
|
||||
OmniVoice engine adapter.
|
||||
|
||||
Translates TTS Audio Suite processor calls into the official OmniVoice runtime.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from typing import Any, Dict, Optional, Tuple, Union
|
||||
|
||||
import torch
|
||||
|
||||
current_dir = os.path.dirname(__file__)
|
||||
engines_dir = os.path.dirname(current_dir)
|
||||
project_root = os.path.dirname(engines_dir)
|
||||
if project_root not in sys.path:
|
||||
sys.path.insert(0, project_root)
|
||||
|
||||
from utils.audio.audio_hash import generate_stable_audio_component
|
||||
from utils.audio.cache import get_audio_cache
|
||||
from utils.models.factory_config import ModelLoadConfig
|
||||
from utils.models.language_mapper import resolve_language_alias
|
||||
|
||||
|
||||
class OmniVoiceEngineAdapter:
|
||||
"""Adapter for official OmniVoice inference."""
|
||||
|
||||
SAMPLE_RATE = 24000
|
||||
|
||||
def __init__(self, config: Optional[Dict[str, Any]] = None):
|
||||
self.config = config.copy() if config else {}
|
||||
self.audio_cache = get_audio_cache()
|
||||
self._last_config: Optional[ModelLoadConfig] = None
|
||||
self._load_signature = None
|
||||
|
||||
def update_config(self, new_config: Dict[str, Any]):
|
||||
self.config = new_config.copy() if new_config else {}
|
||||
|
||||
def _build_load_signature(self) -> Tuple[Any, ...]:
|
||||
return (
|
||||
self.config.get("model_variant", "OmniVoice"),
|
||||
self.config.get("device", "auto"),
|
||||
self.config.get("dtype", "auto"),
|
||||
)
|
||||
|
||||
def load_model(
|
||||
self,
|
||||
model_variant: str,
|
||||
device: str = "auto",
|
||||
dtype: str = "auto",
|
||||
):
|
||||
from utils.models.unified_model_interface import unified_model_interface
|
||||
|
||||
config = ModelLoadConfig(
|
||||
engine_name="omnivoice",
|
||||
model_type="tts",
|
||||
model_name=model_variant,
|
||||
model_path=model_variant,
|
||||
device=device,
|
||||
additional_params={
|
||||
"dtype": dtype,
|
||||
},
|
||||
)
|
||||
self._last_config = config
|
||||
unified_model_interface.load_model(config)
|
||||
|
||||
def _ensure_model_loaded(self):
|
||||
signature = self._build_load_signature()
|
||||
if signature == self._load_signature and self._last_config is not None:
|
||||
return
|
||||
|
||||
self.load_model(
|
||||
model_variant=self.config.get("model_variant", "OmniVoice"),
|
||||
device=self.config.get("device", "auto"),
|
||||
dtype=self.config.get("dtype", "auto"),
|
||||
)
|
||||
self._load_signature = signature
|
||||
|
||||
def _get_engine(self):
|
||||
if self._last_config is None:
|
||||
self._ensure_model_loaded()
|
||||
from utils.models.unified_model_interface import unified_model_interface
|
||||
|
||||
return unified_model_interface.load_model(self._last_config)
|
||||
|
||||
@staticmethod
|
||||
def _seed_everything(seed: int):
|
||||
if int(seed or 0) <= 0:
|
||||
return
|
||||
torch.manual_seed(int(seed))
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed_all(int(seed))
|
||||
if hasattr(torch, "xpu") and torch.xpu.is_available():
|
||||
torch.xpu.manual_seed_all(int(seed))
|
||||
|
||||
@staticmethod
|
||||
def _first_non_none(*values):
|
||||
for value in values:
|
||||
if value is None:
|
||||
continue
|
||||
if isinstance(value, str) and not value.strip():
|
||||
continue
|
||||
return value
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _normalize_waveform_tensor(waveform: torch.Tensor) -> torch.Tensor:
|
||||
if waveform.dim() == 3:
|
||||
waveform = waveform.squeeze(0)
|
||||
if waveform.dim() == 1:
|
||||
waveform = waveform.unsqueeze(0)
|
||||
return waveform.detach().float().cpu()
|
||||
|
||||
def _extract_voice_reference(
|
||||
self,
|
||||
voice_ref: Optional[Dict[str, Any]],
|
||||
) -> Tuple[Optional[Union[str, Tuple[torch.Tensor, int]]], str, str]:
|
||||
if not voice_ref or not isinstance(voice_ref, dict):
|
||||
return None, "", "default_voice"
|
||||
|
||||
prompt_text = (
|
||||
voice_ref.get("reference_text")
|
||||
or voice_ref.get("prompt_text")
|
||||
or voice_ref.get("text")
|
||||
or ""
|
||||
).strip()
|
||||
|
||||
ref_audio = self._first_non_none(
|
||||
voice_ref.get("prompt_audio_path"),
|
||||
voice_ref.get("audio_path"),
|
||||
voice_ref.get("audio"),
|
||||
voice_ref.get("waveform"),
|
||||
)
|
||||
|
||||
if ref_audio is None:
|
||||
return None, prompt_text, "default_voice"
|
||||
|
||||
if isinstance(ref_audio, str):
|
||||
return ref_audio, prompt_text, generate_stable_audio_component(audio_file_path=ref_audio)
|
||||
|
||||
if isinstance(ref_audio, dict) and "waveform" in ref_audio:
|
||||
waveform = self._normalize_waveform_tensor(ref_audio["waveform"])
|
||||
sample_rate = int(ref_audio.get("sample_rate", self.SAMPLE_RATE))
|
||||
audio_dict = {"waveform": waveform, "sample_rate": sample_rate}
|
||||
return (
|
||||
(waveform, sample_rate),
|
||||
prompt_text,
|
||||
generate_stable_audio_component(reference_audio=audio_dict),
|
||||
)
|
||||
|
||||
if torch.is_tensor(ref_audio):
|
||||
waveform = self._normalize_waveform_tensor(ref_audio)
|
||||
sample_rate = int(voice_ref.get("sample_rate", self.SAMPLE_RATE))
|
||||
audio_dict = {"waveform": waveform, "sample_rate": sample_rate}
|
||||
return (
|
||||
(waveform, sample_rate),
|
||||
prompt_text,
|
||||
generate_stable_audio_component(reference_audio=audio_dict),
|
||||
)
|
||||
|
||||
raise TypeError(f"Unsupported OmniVoice reference type: {type(ref_audio)}")
|
||||
|
||||
@staticmethod
|
||||
def _normalize_language(language: Optional[str]) -> Optional[str]:
|
||||
if language is None:
|
||||
return None
|
||||
|
||||
normalized = str(language).strip()
|
||||
if not normalized:
|
||||
return None
|
||||
|
||||
lowered = normalized.lower()
|
||||
if lowered in {"auto", "none"}:
|
||||
return None
|
||||
|
||||
resolved = resolve_language_alias(normalized)
|
||||
if resolved and resolved.lower() != lowered:
|
||||
return resolved
|
||||
return normalized
|
||||
|
||||
def generate_single(
|
||||
self,
|
||||
text: str,
|
||||
voice_ref: Optional[Dict[str, Any]],
|
||||
seed: int = 0,
|
||||
enable_audio_cache: bool = True,
|
||||
character_name: Optional[str] = None,
|
||||
) -> torch.Tensor:
|
||||
stripped = (text or "").strip()
|
||||
if not stripped:
|
||||
return torch.zeros(1, 0, dtype=torch.float32)
|
||||
|
||||
self._ensure_model_loaded()
|
||||
engine = self._get_engine()
|
||||
|
||||
ref_audio, prompt_text, audio_component = self._extract_voice_reference(voice_ref)
|
||||
language = self._normalize_language(self.config.get("language", "Auto"))
|
||||
instruct = str(self.config.get("instruct", "") or "").strip() or None
|
||||
model_variant = self.config.get("model_variant", "OmniVoice")
|
||||
device = self.config.get("device", "auto")
|
||||
dtype = self.config.get("dtype", "auto")
|
||||
num_step = int(self.config.get("num_steps", 32))
|
||||
guidance_scale = float(self.config.get("guidance_scale", 2.0))
|
||||
t_shift = float(self.config.get("t_shift", 0.1))
|
||||
speed = float(self.config.get("speed", 1.0))
|
||||
duration_value = float(self.config.get("duration", 0.0))
|
||||
duration = duration_value if duration_value > 0 else None
|
||||
layer_penalty_factor = float(self.config.get("layer_penalty_factor", 5.0))
|
||||
position_temperature = float(self.config.get("position_temperature", 5.0))
|
||||
class_temperature = float(self.config.get("class_temperature", 0.0))
|
||||
denoise = bool(self.config.get("denoise", True))
|
||||
preprocess_prompt = bool(self.config.get("preprocess_prompt", True))
|
||||
postprocess_output = bool(self.config.get("postprocess_output", True))
|
||||
audio_chunk_duration = float(self.config.get("audio_chunk_duration", 15.0))
|
||||
audio_chunk_threshold = float(self.config.get("audio_chunk_threshold", 30.0))
|
||||
|
||||
cache_key = None
|
||||
if enable_audio_cache:
|
||||
cache_key = self.audio_cache.generate_cache_key(
|
||||
"omnivoice",
|
||||
text=stripped,
|
||||
audio_component=audio_component,
|
||||
prompt_text=prompt_text,
|
||||
model_variant=model_variant,
|
||||
language=language or "none",
|
||||
instruct=instruct or "",
|
||||
num_step=num_step,
|
||||
guidance_scale=guidance_scale,
|
||||
t_shift=t_shift,
|
||||
speed=speed,
|
||||
duration=duration if duration is not None else "auto",
|
||||
layer_penalty_factor=layer_penalty_factor,
|
||||
position_temperature=position_temperature,
|
||||
class_temperature=class_temperature,
|
||||
denoise=denoise,
|
||||
preprocess_prompt=preprocess_prompt,
|
||||
postprocess_output=postprocess_output,
|
||||
audio_chunk_duration=audio_chunk_duration,
|
||||
audio_chunk_threshold=audio_chunk_threshold,
|
||||
seed=int(seed or 0),
|
||||
device=device,
|
||||
dtype=dtype,
|
||||
character=character_name or "narrator",
|
||||
)
|
||||
cached_audio = self.audio_cache.get_cached_audio(cache_key)
|
||||
if cached_audio:
|
||||
print(f"💾 Using cached OmniVoice audio for '{character_name or 'narrator'}': '{stripped[:30]}...'")
|
||||
return cached_audio[0]
|
||||
|
||||
voice_clone_prompt = None
|
||||
if ref_audio is not None:
|
||||
if not prompt_text:
|
||||
raise ValueError(
|
||||
"OmniVoice voice cloning requires reference text. "
|
||||
"Direct audio-only narrator input is unsupported. "
|
||||
"Use Character Voices or a narrator voice file with matching reference text."
|
||||
)
|
||||
voice_clone_prompt = engine.create_voice_clone_prompt(
|
||||
ref_audio=ref_audio,
|
||||
ref_text=prompt_text,
|
||||
preprocess_prompt=preprocess_prompt,
|
||||
)
|
||||
|
||||
self._seed_everything(seed)
|
||||
|
||||
result = engine.generate(
|
||||
text=stripped,
|
||||
language=language,
|
||||
voice_clone_prompt=voice_clone_prompt,
|
||||
instruct=instruct,
|
||||
duration=duration,
|
||||
speed=speed,
|
||||
num_step=num_step,
|
||||
guidance_scale=guidance_scale,
|
||||
t_shift=t_shift,
|
||||
denoise=denoise,
|
||||
preprocess_prompt=preprocess_prompt,
|
||||
postprocess_output=postprocess_output,
|
||||
layer_penalty_factor=layer_penalty_factor,
|
||||
position_temperature=position_temperature,
|
||||
class_temperature=class_temperature,
|
||||
audio_chunk_duration=audio_chunk_duration,
|
||||
audio_chunk_threshold=audio_chunk_threshold,
|
||||
)
|
||||
|
||||
audio_tensor = result[0]
|
||||
if not isinstance(audio_tensor, torch.Tensor):
|
||||
audio_tensor = torch.tensor(audio_tensor, dtype=torch.float32)
|
||||
audio_tensor = audio_tensor.detach().float().cpu()
|
||||
if audio_tensor.dim() == 1:
|
||||
audio_tensor = audio_tensor.unsqueeze(0)
|
||||
|
||||
if enable_audio_cache and cache_key:
|
||||
duration_seconds = self.audio_cache._calculate_duration(audio_tensor, "omnivoice")
|
||||
self.audio_cache.cache_audio(cache_key, audio_tensor, duration_seconds)
|
||||
|
||||
return audio_tensor
|
||||
@@ -25,6 +25,7 @@ from engines.vibevoice_engine.vibevoice_downloader import (
|
||||
is_kugelaudio_variant_name,
|
||||
)
|
||||
from utils.models.manager import model_manager
|
||||
from utils.voice.character_logging import resolved_character_label
|
||||
|
||||
|
||||
class VibeVoiceEngineAdapter:
|
||||
@@ -466,7 +467,8 @@ class VibeVoiceEngineAdapter:
|
||||
|
||||
# Generate each character group using VibeVoice format
|
||||
for group_idx, (character, text_list) in enumerate(character_groups):
|
||||
print(f"🎤 Group {group_idx + 1}: Character '{character}' with {len(text_list)} segments")
|
||||
display_name = resolved_character_label(character, voice_mapping.get(character))
|
||||
print(f"🎤 Group {group_idx + 1}: Character '{display_name}' with {len(text_list)} segments")
|
||||
|
||||
# Format as Speaker 1 entries (VibeVoice style) and combine
|
||||
formatted_lines = []
|
||||
@@ -526,15 +528,11 @@ class VibeVoiceEngineAdapter:
|
||||
Returns:
|
||||
Combined audio dict
|
||||
"""
|
||||
# Get speaker voice inputs from engine config for priority system
|
||||
# For manual Speaker format, the main narrator voice comes from the TTS Text node
|
||||
# We need to get it from a different source since voice_mapping may not have 'narrator' key
|
||||
main_narrator_voice = voice_mapping.get('narrator') # Try narrator first
|
||||
if main_narrator_voice is None:
|
||||
# If no 'narrator' key, get it from any available voice (fallback for manual Speaker format)
|
||||
available_voices = [v for v in voice_mapping.values() if v is not None]
|
||||
main_narrator_voice = available_voices[0] if available_voices else None
|
||||
# print(f"🐛 Debug: No 'narrator' key, using fallback voice: {'✅ found' if main_narrator_voice else '❌ none available'}")
|
||||
# Get speaker voice inputs from engine config for priority system.
|
||||
# Speaker 1 must only come from an explicit narrator/speaker-1 input.
|
||||
# Do not synthesize Speaker 1 from discovered character voices, or aliases
|
||||
# will look like they were overridden by a connection that does not exist.
|
||||
main_narrator_voice = voice_mapping.get('narrator')
|
||||
speaker_inputs = {
|
||||
1: main_narrator_voice, # Speaker 1 uses main narrator from TTS Text
|
||||
2: params.get('speaker2_voice'),
|
||||
@@ -544,19 +542,24 @@ class VibeVoiceEngineAdapter:
|
||||
|
||||
# print(f"🐛 Debug: speaker_inputs[1] (main narrator): {'✅ has voice' if speaker_inputs[1] else '❌ no voice'}")
|
||||
|
||||
# Build speaker mapping and format text
|
||||
# Build speaker mapping and format text.
|
||||
# For SRT/global processing we keep speaker numbering local to the current
|
||||
# generation call (Speaker 1..N in order of first appearance), but preserve
|
||||
# the character's global slot when choosing connected speaker override inputs.
|
||||
character_map = {}
|
||||
# Pre-fill speaker_voices with all 4 speaker slots (some may be None)
|
||||
speaker_voices = [
|
||||
speaker_inputs.get(1), # Speaker 1
|
||||
speaker_inputs.get(2), # Speaker 2
|
||||
speaker_inputs.get(3), # Speaker 3
|
||||
speaker_inputs.get(4) # Speaker 4
|
||||
]
|
||||
character_global_slots = {}
|
||||
speaker_voices = []
|
||||
formatted_lines = []
|
||||
segment_characters = [char for char, _ in segments]
|
||||
|
||||
print(f"🎭 Native multi-speaker: Processing {len(segments)} segments with characters: {[char for char, _ in segments]}")
|
||||
print(f"🎭 Native multi-speaker: Processing {len(segments)} segments with characters: {segment_characters}")
|
||||
print(f"🎤 Speaker inputs connected: {[f'Speaker {k}' for k, v in speaker_inputs.items() if v is not None]}")
|
||||
narrator_exists_globally = bool(global_char_to_speaker and "narrator" in global_char_to_speaker)
|
||||
if speaker_inputs.get(1) is not None and "narrator" not in segment_characters:
|
||||
if narrator_exists_globally:
|
||||
print("ℹ️ Narrator input is connected, but this subtitle has no narrator turn; it will not override tagged characters")
|
||||
else:
|
||||
print("ℹ️ No narrator turns exist in this SRT; Speaker 1 maps to the first named character by first appearance")
|
||||
|
||||
for character, text in segments:
|
||||
# Check if this is already a manual "Speaker N:" format
|
||||
@@ -567,50 +570,84 @@ class VibeVoiceEngineAdapter:
|
||||
speaker_idx = manual_speaker - 1 # Convert to 0-based
|
||||
if speaker_idx >= 4:
|
||||
speaker_idx = 3
|
||||
|
||||
# Voice already in pre-filled speaker_voices array
|
||||
voice = speaker_voices[speaker_idx]
|
||||
voice = speaker_inputs.get(manual_speaker)
|
||||
if manual_speaker == 1:
|
||||
print(f"🎤 Manual format 'Speaker {manual_speaker}' -> using {'✅ main narrator (Tony)' if voice else '❌ no narrator, using default'}")
|
||||
else:
|
||||
print(f"🎤 Manual format 'Speaker {manual_speaker}' -> using {'✅ connected input' if voice else '❌ no input, using default'}")
|
||||
|
||||
while len(speaker_voices) <= speaker_idx:
|
||||
speaker_voices.append(None)
|
||||
speaker_voices[speaker_idx] = voice
|
||||
|
||||
formatted_lines.append(f"Speaker {manual_speaker}: {text.strip()}")
|
||||
|
||||
else:
|
||||
# Character tag format - use global mapping if provided (for SRT consistency)
|
||||
# Character tag format
|
||||
if character not in character_map:
|
||||
# Special handling for numeric characters: [1] [2] [3] [4] -> map directly to Speaker N
|
||||
if character.isdigit() and 1 <= int(character) <= 4:
|
||||
speaker_idx = int(character) - 1 # Convert [1] to Speaker 1 (0-based index)
|
||||
global_speaker_num = int(character)
|
||||
speaker_idx = len(character_map)
|
||||
character_map[character] = speaker_idx
|
||||
character_global_slots[character] = global_speaker_num
|
||||
print(f"🔢 Numeric character '[{character}]' -> Speaker {int(character)} (direct mapping)")
|
||||
elif global_char_to_speaker and character in global_char_to_speaker:
|
||||
# Use global mapping for consistent SRT processing
|
||||
speaker_idx = global_char_to_speaker[character] - 1 # Convert to 0-based
|
||||
global_speaker_num = global_char_to_speaker[character]
|
||||
speaker_idx = len(character_map)
|
||||
character_map[character] = speaker_idx
|
||||
character_global_slots[character] = global_speaker_num
|
||||
else:
|
||||
# Fallback to sequential assignment
|
||||
speaker_idx = len(character_map)
|
||||
if speaker_idx >= 4:
|
||||
print(f"⚠️ VibeVoice: Limiting to 4 speakers, '{character}' will use Speaker 4")
|
||||
speaker_idx = 3 # Use 0-based internally, will convert to 1-based for format
|
||||
global_speaker_num = 4
|
||||
else:
|
||||
character_map[character] = speaker_idx
|
||||
global_speaker_num = speaker_idx + 1
|
||||
character_global_slots[character] = global_speaker_num
|
||||
|
||||
# Priority system: speaker inputs override character aliases
|
||||
# Priority system:
|
||||
# - Numeric [1]-[4] always map directly to speaker inputs 1-4.
|
||||
# - If any narrator turn exists in the SRT, Speaker 1 is reserved for narrator.
|
||||
# - Otherwise, Speaker 1..4 map to named characters by first global appearance order.
|
||||
speaker_num = speaker_idx + 1
|
||||
connected_voice = speaker_inputs.get(speaker_num)
|
||||
global_speaker_num = character_global_slots.get(character, speaker_num)
|
||||
connected_voice = None
|
||||
is_numeric_direct = character.isdigit() and 1 <= int(character) <= 4
|
||||
if is_numeric_direct:
|
||||
connected_voice = speaker_inputs.get(global_speaker_num)
|
||||
elif character == "narrator":
|
||||
connected_voice = speaker_inputs.get(1)
|
||||
elif narrator_exists_globally:
|
||||
if global_speaker_num >= 2:
|
||||
connected_voice = speaker_inputs.get(global_speaker_num)
|
||||
else:
|
||||
connected_voice = speaker_inputs.get(global_speaker_num)
|
||||
character_voice = voice_mapping.get(character)
|
||||
|
||||
|
||||
if connected_voice is not None and character_voice is not None:
|
||||
print(f"⚠️ Priority: Speaker {speaker_num} input overrides ['{character}'] alias - using connected voice")
|
||||
if global_speaker_num != speaker_num:
|
||||
print(
|
||||
f"⚠️ Priority: Speaker {global_speaker_num} input overrides ['{character}'] alias "
|
||||
f"- using connected voice as local Speaker {speaker_num}"
|
||||
)
|
||||
else:
|
||||
print(f"⚠️ Priority: Speaker {speaker_num} input overrides ['{character}'] alias - using connected voice")
|
||||
voice = connected_voice
|
||||
elif connected_voice is not None:
|
||||
print(f"🎤 Speaker {speaker_num}: Using connected voice input")
|
||||
if global_speaker_num != speaker_num:
|
||||
print(f"🎤 Speaker {global_speaker_num}: Using connected voice input as local Speaker {speaker_num}")
|
||||
else:
|
||||
print(f"🎤 Speaker {speaker_num}: Using connected voice input")
|
||||
voice = connected_voice
|
||||
else:
|
||||
print(f"🎭 Character '{character}' -> Speaker {speaker_num}, using character voice")
|
||||
if global_speaker_num != speaker_num:
|
||||
print(f"🎭 Character '{character}' -> global Speaker {global_speaker_num}, local Speaker {speaker_num}, using alias/character voice")
|
||||
else:
|
||||
print(f"🎭 Character '{character}' -> Speaker {speaker_num}, using alias/character voice")
|
||||
voice = character_voice
|
||||
|
||||
# Ensure we have enough speaker_voices slots
|
||||
@@ -629,6 +666,18 @@ class VibeVoiceEngineAdapter:
|
||||
print(formatted_text)
|
||||
print("="*60)
|
||||
print(f"🎤 Using {len(speaker_voices)} voice samples for generation")
|
||||
for idx, voice in enumerate(speaker_voices, start=1):
|
||||
if voice is None:
|
||||
print(f" Speaker {idx}: default / no reference")
|
||||
elif isinstance(voice, dict) and voice.get("audio_path"):
|
||||
print(f" Speaker {idx}: file reference -> {voice['audio_path']}")
|
||||
elif isinstance(voice, dict) and "waveform" in voice:
|
||||
waveform = voice["waveform"]
|
||||
sample_rate = voice.get("sample_rate", "unknown")
|
||||
shape = tuple(waveform.shape) if hasattr(waveform, "shape") else "unknown"
|
||||
print(f" Speaker {idx}: waveform reference -> shape {shape}, sr {sample_rate}")
|
||||
else:
|
||||
print(f" Speaker {idx}: unexpected reference type {type(voice).__name__}")
|
||||
|
||||
# Validate and normalize voice references
|
||||
normalized_voices = []
|
||||
|
||||
@@ -7,7 +7,8 @@ Works with character_grouper.py to process multiple segments simultaneously.
|
||||
|
||||
from typing import Dict, List, Any
|
||||
import torch
|
||||
from .character_grouper import CharacterGroup
|
||||
from .character_grouper import CharacterGroup
|
||||
from utils.voice.character_logging import resolved_character_label
|
||||
|
||||
|
||||
class BatchProcessor:
|
||||
@@ -49,7 +50,8 @@ class BatchProcessor:
|
||||
character = character_group.character
|
||||
segments = character_group.segments
|
||||
|
||||
print(f"🚀 BATCH PROCESSING: {character} - {len(segments)} segments in {language}")
|
||||
display_name = resolved_character_label(character, voice_refs.get(character))
|
||||
print(f"🚀 BATCH PROCESSING: {display_name} - {len(segments)} segments in {language}")
|
||||
|
||||
# Collect all texts for batching
|
||||
batch_texts = []
|
||||
@@ -75,7 +77,7 @@ class BatchProcessor:
|
||||
char_audio_prompt = voice_refs[character]
|
||||
|
||||
# THE ACTUAL BATCH PROCESSING - this is the key improvement
|
||||
print(f"⚡ Batch generating {len(batch_texts)} chunks for {character}")
|
||||
print(f"⚡ Batch generating {len(batch_texts)} chunks for {display_name}")
|
||||
print(f"🔧 DEBUG: batch_size from inputs = {inputs.get('batch_size', 4)}")
|
||||
try:
|
||||
batch_audio = self.tts_model.generate_batch(
|
||||
@@ -137,15 +139,15 @@ class BatchProcessor:
|
||||
character = character_group.character
|
||||
segments = character_group.segments
|
||||
|
||||
print(f"→ SEQUENTIAL: {character} - {len(segments)} segments in {language}")
|
||||
char_audio_prompt = voice_refs[character]
|
||||
display_name = resolved_character_label(character, char_audio_prompt)
|
||||
print(f"→ SEQUENTIAL: {display_name} - {len(segments)} segments in {language}")
|
||||
|
||||
results = {}
|
||||
char_audio_prompt = voice_refs[character]
|
||||
|
||||
for segment in segments:
|
||||
for segment in segments:
|
||||
segment_display_idx = segment.original_idx + 1 # 1-based for display
|
||||
|
||||
print(f"🎤 Generating segment {segment_display_idx}/{total_segments} for '{character}' (lang: {language})")
|
||||
print(f"🎤 Generating segment {segment_display_idx}/{total_segments} for '{display_name}' (lang: {language})")
|
||||
|
||||
# Apply chunking
|
||||
if inputs["enable_chunking"] and len(segment.segment_text) > inputs["max_chars_per_chunk"]:
|
||||
@@ -183,4 +185,4 @@ class BatchProcessor:
|
||||
"""Apply crash protection padding to short texts."""
|
||||
if len(text.strip()) < min_length:
|
||||
return padding_template.format(seg=text)
|
||||
return text
|
||||
return text
|
||||
|
||||
@@ -7,7 +7,8 @@ Works with character_grouper.py to process multiple segments simultaneously.
|
||||
|
||||
from typing import Dict, List, Any
|
||||
import torch
|
||||
from .character_grouper import CharacterGroup
|
||||
from .character_grouper import CharacterGroup
|
||||
from utils.voice.character_logging import resolved_character_label
|
||||
|
||||
|
||||
class BatchProcessor:
|
||||
@@ -49,7 +50,8 @@ class BatchProcessor:
|
||||
character = character_group.character
|
||||
segments = character_group.segments
|
||||
|
||||
print(f"🚀 BATCH PROCESSING: {character} - {len(segments)} segments in {language}")
|
||||
display_name = resolved_character_label(character, voice_refs.get(character))
|
||||
print(f"🚀 BATCH PROCESSING: {display_name} - {len(segments)} segments in {language}")
|
||||
|
||||
# Collect all texts for batching
|
||||
batch_texts = []
|
||||
@@ -76,7 +78,7 @@ class BatchProcessor:
|
||||
char_audio_prompt = voice_refs[character]
|
||||
|
||||
# THE ACTUAL BATCH PROCESSING - this is the key improvement
|
||||
print(f"⚡ Batch generating {len(batch_texts)} chunks for {character}")
|
||||
print(f"⚡ Batch generating {len(batch_texts)} chunks for {display_name}")
|
||||
print(f"🔧 DEBUG: batch_size from inputs = {inputs.get('batch_size', 4)}")
|
||||
try:
|
||||
batch_audio = self.tts_model.generate_batch(
|
||||
@@ -138,15 +140,15 @@ class BatchProcessor:
|
||||
character = character_group.character
|
||||
segments = character_group.segments
|
||||
|
||||
print(f"→ SEQUENTIAL: {character} - {len(segments)} segments in {language}")
|
||||
char_audio_prompt = voice_refs[character]
|
||||
display_name = resolved_character_label(character, char_audio_prompt)
|
||||
print(f"→ SEQUENTIAL: {display_name} - {len(segments)} segments in {language}")
|
||||
|
||||
results = {}
|
||||
char_audio_prompt = voice_refs[character]
|
||||
|
||||
for segment in segments:
|
||||
for segment in segments:
|
||||
segment_display_idx = segment.original_idx + 1 # 1-based for display
|
||||
|
||||
print(f"🎤 Generating segment {segment_display_idx}/{total_segments} for '{character}' (lang: {language})")
|
||||
print(f"🎤 Generating segment {segment_display_idx}/{total_segments} for '{display_name}' (lang: {language})")
|
||||
|
||||
# Apply chunking
|
||||
if inputs["enable_chunking"] and len(segment.segment_text) > inputs["max_chars_per_chunk"]:
|
||||
@@ -184,4 +186,4 @@ class BatchProcessor:
|
||||
"""Apply crash protection padding to short texts."""
|
||||
if len(text.strip()) < min_length:
|
||||
return padding_template.format(seg=text)
|
||||
return text
|
||||
return text
|
||||
|
||||
@@ -16,7 +16,8 @@ from torch.nn.utils.rnn import pad_sequence
|
||||
|
||||
def seed_everything(seed=0):
|
||||
random.seed(seed)
|
||||
os.environ["PYTHONHASHSEED"] = str(seed)
|
||||
# PYTHONHASHSEED is read before interpreter startup; changing it here has
|
||||
# no effect on this process and can poison later isolated workers.
|
||||
torch.manual_seed(seed)
|
||||
torch.cuda.manual_seed(seed)
|
||||
torch.cuda.manual_seed_all(seed)
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
"""Fish Audio S2 Pro engine support."""
|
||||
|
||||
from .downloader import FishAudioS2Downloader
|
||||
|
||||
__all__ = ["FishAudioS2Downloader"]
|
||||
@@ -0,0 +1,143 @@
|
||||
"""Organized Hugging Face downloader for the gated Fish Audio S2 Pro model."""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
|
||||
import folder_paths
|
||||
|
||||
from utils.downloads.unified_downloader import unified_downloader
|
||||
|
||||
|
||||
class FishAudioS2Downloader:
|
||||
VARIANTS = {
|
||||
"s2-pro": {
|
||||
"repo_id": "fishaudio/s2-pro",
|
||||
"directory": "fish_audio_s2_pro",
|
||||
"files": [
|
||||
".gitattributes", "LICENSE.md", "README.md", "chat_template.jinja", "codec.pth",
|
||||
"config.json", "model-00001-of-00002.safetensors",
|
||||
"model-00002-of-00002.safetensors", "model.safetensors.index.json",
|
||||
"special_tokens_map.json", "tokenizer.json", "tokenizer_config.json",
|
||||
],
|
||||
},
|
||||
"s2-pro-fp8": {
|
||||
"repo_id": "drbaph/s2-pro-fp8",
|
||||
"directory": "fish_audio_s2_pro_fp8",
|
||||
"files": [
|
||||
".gitattributes", "LICENSE.md", "README.md", "chat_template.jinja", "codec.pth",
|
||||
"config.json", "model.safetensors", "quantization_info.json",
|
||||
"special_tokens_map.json", "tokenizer.json", "tokenizer_config.json",
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def is_model_complete(cls, model_path: str) -> bool:
|
||||
"""Recognize a local Fish S2 checkpoint without loading model code."""
|
||||
if not os.path.isdir(model_path):
|
||||
return False
|
||||
required = ("config.json", "codec.pth", "tokenizer.json")
|
||||
if not all(os.path.isfile(os.path.join(model_path, name)) for name in required):
|
||||
return False
|
||||
try:
|
||||
return any(
|
||||
name.endswith(".safetensors")
|
||||
and os.path.getsize(os.path.join(model_path, name)) > 0
|
||||
for name in os.listdir(model_path)
|
||||
)
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
@classmethod
|
||||
def local_model_paths(cls) -> list[str]:
|
||||
"""Return complete local Fish checkpoints from all configured TTS paths."""
|
||||
try:
|
||||
from utils.models.extra_paths import get_all_tts_model_paths
|
||||
search_paths = get_all_tts_model_paths("TTS")
|
||||
except Exception:
|
||||
search_paths = [os.path.join(folder_paths.models_dir, "TTS")]
|
||||
|
||||
found = {}
|
||||
for base_path in search_paths:
|
||||
if not os.path.isdir(base_path):
|
||||
continue
|
||||
try:
|
||||
entries = os.listdir(base_path)
|
||||
except OSError:
|
||||
continue
|
||||
for entry in entries:
|
||||
candidate = os.path.join(base_path, entry)
|
||||
if cls.is_model_complete(candidate):
|
||||
found.setdefault(entry, candidate)
|
||||
return [found[name] for name in sorted(found)]
|
||||
|
||||
@classmethod
|
||||
def resolve_local_model_path(cls, selection: str) -> str:
|
||||
"""Resolve a `local:<folder>` UI selection to its filesystem path."""
|
||||
name = str(selection).removeprefix("local:")
|
||||
for path in cls.local_model_paths():
|
||||
if os.path.basename(os.path.normpath(path)) == name:
|
||||
return path
|
||||
raise FileNotFoundError(f"Local Fish Audio S2 model not found: {selection}")
|
||||
|
||||
@classmethod
|
||||
def resolve_model_variant(cls, selection: str, model_path: str | None = None) -> str:
|
||||
"""Map a UI selection/path to the runtime's known variant behavior."""
|
||||
if selection in cls.VARIANTS:
|
||||
return selection
|
||||
path = model_path or selection
|
||||
if os.path.isfile(os.path.join(path, "quantization_info.json")) or "fp8" in os.path.basename(path).lower():
|
||||
return "s2-pro-fp8"
|
||||
return "s2-pro"
|
||||
|
||||
@classmethod
|
||||
def resolve_model_path(cls, selection: str) -> str:
|
||||
"""Resolve local selections or download the selected official variant."""
|
||||
if str(selection).startswith("local:"):
|
||||
return cls.resolve_local_model_path(selection)
|
||||
return cls.ensure_model(selection)
|
||||
|
||||
@classmethod
|
||||
def model_dir(cls, variant: str = "s2-pro") -> str:
|
||||
if variant not in cls.VARIANTS:
|
||||
raise ValueError(f"Unknown Fish Audio S2 model variant: {variant}")
|
||||
return os.path.join(folder_paths.models_dir, "TTS", cls.VARIANTS[variant]["directory"])
|
||||
|
||||
@classmethod
|
||||
def ensure_model(cls, variant: str = "s2-pro") -> str:
|
||||
details = cls.VARIANTS.get(variant)
|
||||
if details is None:
|
||||
raise ValueError(f"Unknown Fish Audio S2 model variant: {variant}")
|
||||
target = cls.model_dir(variant)
|
||||
if variant == "s2-pro-fp8":
|
||||
target_codec = os.path.join(target, "codec.pth")
|
||||
official_codec = os.path.join(cls.model_dir("s2-pro"), "codec.pth")
|
||||
if not os.path.isfile(target_codec) and os.path.isfile(official_codec):
|
||||
os.makedirs(target, exist_ok=True)
|
||||
try:
|
||||
os.link(official_codec, target_codec)
|
||||
except OSError:
|
||||
shutil.copy2(official_codec, target_codec)
|
||||
files = details["files"]
|
||||
if all(os.path.isfile(os.path.join(target, name)) and os.path.getsize(os.path.join(target, name)) > 0
|
||||
for name in files):
|
||||
return target
|
||||
os.makedirs(target, exist_ok=True)
|
||||
download_files = [{"remote": name, "local": name} for name in files]
|
||||
try:
|
||||
result = unified_downloader.download_huggingface_model(
|
||||
repo_id=details["repo_id"],
|
||||
model_name=variant,
|
||||
files=download_files,
|
||||
engine_type="fish_audio_s2",
|
||||
target_dir=target,
|
||||
)
|
||||
if not result:
|
||||
raise RuntimeError("one or more required model files failed to download")
|
||||
except Exception as exc:
|
||||
raise RuntimeError(
|
||||
f"Fish Audio S2 variant '{variant}' is gated. Accept its non-commercial license at "
|
||||
f"https://huggingface.co/{details['repo_id']} and authenticate Hugging Face "
|
||||
f"before retrying. Download error: {exc}"
|
||||
) from exc
|
||||
return target
|
||||
@@ -46,7 +46,7 @@ from utils.audio.librosa_fallback import safe_load
|
||||
class IndexTTS2:
|
||||
def __init__(
|
||||
self, cfg_path="checkpoints/config.yaml", model_dir="checkpoints", use_fp16=False, device=None,
|
||||
use_cuda_kernel=None,use_deepspeed=False, use_torch_compile=False, low_vram=False
|
||||
use_cuda_kernel=None,use_deepspeed=False, use_torch_compile=False, use_accel=False, low_vram=False
|
||||
):
|
||||
"""
|
||||
Args:
|
||||
@@ -57,10 +57,12 @@ class IndexTTS2:
|
||||
use_cuda_kernel (None | bool): whether to use BigVGan custom fused activation CUDA kernel, only for CUDA device.
|
||||
use_deepspeed (bool): whether to use DeepSpeed or not.
|
||||
use_torch_compile (bool): whether to use torch.compile for optimization or not.
|
||||
use_accel (bool): whether to enable GPT2 FlashAttention acceleration.
|
||||
"""
|
||||
use_fp16 = self._coerce_bool_flag(use_fp16)
|
||||
use_deepspeed = self._coerce_bool_flag(use_deepspeed)
|
||||
use_torch_compile = self._coerce_bool_flag(use_torch_compile)
|
||||
use_accel = self._coerce_bool_flag(use_accel)
|
||||
low_vram = self._coerce_bool_flag(low_vram)
|
||||
|
||||
if device is not None:
|
||||
@@ -131,6 +133,7 @@ class IndexTTS2:
|
||||
if use_torch_compile:
|
||||
self._validate_torch_compile_support()
|
||||
self.use_torch_compile = use_torch_compile
|
||||
self.use_accel = use_accel
|
||||
self.low_vram = low_vram
|
||||
|
||||
# Determine device for initial model loading
|
||||
@@ -139,6 +142,10 @@ class IndexTTS2:
|
||||
|
||||
# Initialize QwenEmotion for text-based emotion control if available
|
||||
self.qwen_emo = None
|
||||
# A text-emotion model failure must not abort an otherwise valid TTS run.
|
||||
# Once it fails, skip it for the lifetime of this engine instance instead
|
||||
# of repeatedly hitting a poisoned CUDA/accelerate state on later segments.
|
||||
self._qwen_emo_disabled = False
|
||||
self.qwen_emo_path = None # Store path for lazy loading
|
||||
|
||||
try:
|
||||
@@ -162,7 +169,7 @@ class IndexTTS2:
|
||||
print(f"⚠️ QwenEmotion model incomplete - missing files: {missing_files}")
|
||||
print("ℹ️ Falling back to audio emotion only")
|
||||
else:
|
||||
self.qwen_emo = QwenEmotion(normalized_path)
|
||||
self.qwen_emo = QwenEmotion(normalized_path, preferred_device=self.device)
|
||||
print("✅ QwenEmotion loaded - text emotion support enabled")
|
||||
else:
|
||||
print("ℹ️ QwenEmotion not available - audio emotion only")
|
||||
@@ -172,7 +179,9 @@ class IndexTTS2:
|
||||
print("ℹ️ Falling back to audio emotion only")
|
||||
|
||||
print("🔄 IndexTTS-2: Loading GPT model...")
|
||||
self.gpt = UnifiedVoice(**self.cfg.gpt)
|
||||
gpt_config = OmegaConf.to_container(self.cfg.gpt, resolve=True)
|
||||
gpt_config["use_accel"] = self.use_accel
|
||||
self.gpt = UnifiedVoice(**gpt_config)
|
||||
self.gpt_path = os.path.join(self.model_dir, self.cfg.gpt_checkpoint)
|
||||
load_checkpoint(self.gpt, self.gpt_path)
|
||||
print(f"🔄 IndexTTS-2: Moving GPT to {self.load_device}...")
|
||||
@@ -571,23 +580,38 @@ class IndexTTS2:
|
||||
f"emo_text:{emo_text}")
|
||||
start_time = time.perf_counter()
|
||||
|
||||
if use_emo_text or emo_vector is not None:
|
||||
# we're using a text or emotion vector guidance; so we must remove
|
||||
# "emotion reference voice", to ensure we use correct emotion mixing!
|
||||
emo_audio_prompt = None
|
||||
# Text/vector emotion and an external emotion reference can be used
|
||||
# together. The vector is projected into the same conditioning space
|
||||
# later and blended with the audio-derived conditioning.
|
||||
|
||||
if use_emo_text:
|
||||
# automatically generate emotion vectors from text prompt
|
||||
if self.qwen_emo is None:
|
||||
print("⚠️ QwenEmotion model not available - cannot use text emotion. Ignoring use_emo_text=True")
|
||||
if self.qwen_emo is None or self._qwen_emo_disabled:
|
||||
print("⚠️ QwenEmotion text analysis unavailable - continuing without text emotion")
|
||||
use_emo_text = False
|
||||
else:
|
||||
if emo_text is None:
|
||||
emo_text = text # use main text prompt
|
||||
emo_dict = self.qwen_emo.inference(emo_text)
|
||||
print(f"detected emotion vectors from text: {emo_dict}")
|
||||
# convert ordered dict to list of vectors; the order is VERY important!
|
||||
emo_vector = list(emo_dict.values())
|
||||
try:
|
||||
emo_dict = self.qwen_emo.inference(emo_text)
|
||||
except Exception as exc:
|
||||
# QwenEmotion is an optional classifier. CUDA/accelerate
|
||||
# failures here should degrade to ordinary IndexTTS-2
|
||||
# synthesis, rather than discarding every audio segment.
|
||||
self._qwen_emo_disabled = True
|
||||
use_emo_text = False
|
||||
# Do not silently substitute a stale/global vector for the
|
||||
# requested text emotion. Continue with voice-derived
|
||||
# conditioning instead.
|
||||
emo_vector = None
|
||||
print(
|
||||
"⚠️ QwenEmotion text analysis failed; continuing without "
|
||||
f"text emotion ({type(exc).__name__}: {exc})"
|
||||
)
|
||||
else:
|
||||
print(f"detected emotion vectors from text: {emo_dict}")
|
||||
# convert ordered dict to list of vectors; the order is VERY important!
|
||||
emo_vector = list(emo_dict.values())
|
||||
|
||||
if emo_vector is not None:
|
||||
# Apply normalization to prevent voice identity loss
|
||||
@@ -615,6 +639,9 @@ class IndexTTS2:
|
||||
emo_vector_scaled_display = [round(x, 3) for x in emo_vector]
|
||||
print(f"🎭 Scaled by alpha {emo_vector_scale}: {emo_vector_scaled_display}")
|
||||
|
||||
if emo_audio_prompt is not None and emo_vector is not None:
|
||||
print("🎭 Blending emotion-reference audio with vector/text emotion")
|
||||
|
||||
if emo_audio_prompt is None:
|
||||
# we are not using any external "emotion reference voice"; use
|
||||
# speaker's voice as the main emotion reference audio.
|
||||
@@ -1027,11 +1054,18 @@ def find_most_similar_cosine(query_vector, matrix):
|
||||
return most_similar_index
|
||||
|
||||
class QwenEmotion:
|
||||
def __init__(self, model_dir):
|
||||
def __init__(self, model_dir, preferred_device=None):
|
||||
self.model_dir = model_dir
|
||||
self.tokenizer = AutoTokenizer.from_pretrained(self.model_dir)
|
||||
import torch
|
||||
load_kwargs = {"device_map": "auto"}
|
||||
# Auto-sharding split this small classifier across both GPUs in some
|
||||
# ComfyUI setups. Accelerate then tried to move Qwen's cache between
|
||||
# devices and produced CUDA ``invalid argument`` errors. Keep Qwen on
|
||||
# one GPU; prefer the device with the most available memory.
|
||||
qwen_device = self._select_device(preferred_device)
|
||||
load_kwargs = {"device_map": {"": qwen_device}} if qwen_device else {}
|
||||
if qwen_device:
|
||||
print(f"🤖 QwenEmotion loading on {qwen_device} (single device)")
|
||||
try:
|
||||
self.model = AutoModelForCausalLM.from_pretrained(
|
||||
self.model_dir,
|
||||
@@ -1080,7 +1114,28 @@ class QwenEmotion:
|
||||
self.max_score = 1.2
|
||||
self.min_score = 0.0
|
||||
|
||||
@staticmethod
|
||||
def _select_device(preferred_device=None):
|
||||
preferred = str(preferred_device) if preferred_device is not None else None
|
||||
if not torch.cuda.is_available():
|
||||
return preferred or "cpu"
|
||||
|
||||
# QwenEmotion is small enough for one GPU. Selecting the roomiest GPU
|
||||
# avoids forcing it onto a nearly-full IndexTTS device.
|
||||
try:
|
||||
free_memory = [torch.cuda.mem_get_info(index)[0] for index in range(torch.cuda.device_count())]
|
||||
return f"cuda:{max(range(len(free_memory)), key=free_memory.__getitem__)}"
|
||||
except Exception:
|
||||
return preferred if preferred and preferred.startswith("cuda") else "cuda:0"
|
||||
|
||||
def clamp_score(self, value):
|
||||
try:
|
||||
value = float(value)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ValueError(
|
||||
f"QwenEmotion returned a non-numeric emotion score {value!r}. "
|
||||
"Please retry the request."
|
||||
) from exc
|
||||
return max(self.min_score, min(self.max_score, value))
|
||||
|
||||
def convert(self, content):
|
||||
@@ -1113,15 +1168,25 @@ class QwenEmotion:
|
||||
add_generation_prompt=True,
|
||||
enable_thinking=False,
|
||||
)
|
||||
model_inputs = self.tokenizer([text], return_tensors="pt").to(self.model.device)
|
||||
# With device_map="auto", model.device is the input/first device exposed
|
||||
# by accelerate. Keep the move explicit instead of relying on a chained
|
||||
# BatchEncoding operation, which has triggered invalid CUDA arguments on
|
||||
# some transformers/accelerate combinations.
|
||||
input_device = getattr(self.model, "device", None)
|
||||
if input_device is None or str(input_device) == "meta":
|
||||
input_device = next(self.model.parameters()).device
|
||||
model_inputs = self.tokenizer([text], return_tensors="pt")
|
||||
model_inputs = {key: value.to(input_device) for key, value in model_inputs.items()}
|
||||
|
||||
# conduct text completion
|
||||
generated_ids = self.model.generate(
|
||||
**model_inputs,
|
||||
max_new_tokens=32768,
|
||||
# The classifier returns a tiny JSON object; an unbounded 32k-token
|
||||
# generation wastes VRAM and makes CUDA failures more likely.
|
||||
max_new_tokens=256,
|
||||
pad_token_id=self.tokenizer.eos_token_id
|
||||
)
|
||||
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()
|
||||
output_ids = generated_ids[0][len(model_inputs["input_ids"][0]):].tolist()
|
||||
|
||||
# parsing thinking content
|
||||
try:
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
"""
|
||||
OmniVoice engine integration.
|
||||
"""
|
||||
|
||||
from .omnivoice_downloader import OmniVoiceDownloader
|
||||
from .omnivoice_engine import OmniVoiceEngine
|
||||
|
||||
__all__ = ["OmniVoiceDownloader", "OmniVoiceEngine"]
|
||||
@@ -0,0 +1,148 @@
|
||||
"""
|
||||
OmniVoice model downloader.
|
||||
|
||||
Downloads the official OmniVoice assets into ComfyUI/models/TTS/omnivoice/.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from contextlib import contextmanager
|
||||
from typing import Dict, Optional
|
||||
|
||||
import folder_paths
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
from utils.models.extra_paths import get_all_tts_model_paths, get_preferred_download_path
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _suppress_hf_http_logs():
|
||||
logger_names = ("httpx", "httpcore", "huggingface_hub")
|
||||
original_levels = {}
|
||||
try:
|
||||
for name in logger_names:
|
||||
logger = logging.getLogger(name)
|
||||
original_levels[name] = logger.level
|
||||
logger.setLevel(logging.WARNING)
|
||||
yield
|
||||
finally:
|
||||
for name, level in original_levels.items():
|
||||
logging.getLogger(name).setLevel(level)
|
||||
|
||||
|
||||
class OmniVoiceDownloader:
|
||||
"""Resolve and download official OmniVoice model folders."""
|
||||
|
||||
MODELS: Dict[str, Dict[str, object]] = {
|
||||
"OmniVoice": {
|
||||
"repo_id": "k2-fsa/OmniVoice",
|
||||
"description": "Official OmniVoice multilingual zero-shot TTS checkpoint",
|
||||
"required_files": [
|
||||
"chat_template.jinja",
|
||||
"config.json",
|
||||
"model.safetensors",
|
||||
"tokenizer.json",
|
||||
"tokenizer_config.json",
|
||||
"audio_tokenizer/config.json",
|
||||
"audio_tokenizer/model.safetensors",
|
||||
"audio_tokenizer/preprocessor_config.json",
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
def __init__(self, base_path: Optional[str] = None):
|
||||
if base_path is None:
|
||||
try:
|
||||
self.base_path = get_preferred_download_path(model_type="TTS", engine_name="omnivoice")
|
||||
except Exception:
|
||||
self.base_path = os.path.join(folder_paths.models_dir, "TTS", "omnivoice")
|
||||
else:
|
||||
self.base_path = base_path
|
||||
os.makedirs(self.base_path, exist_ok=True)
|
||||
|
||||
def resolve_model_path(self, model_identifier: str) -> str:
|
||||
"""Resolve absolute paths, local: names, or known official identifiers."""
|
||||
if not model_identifier:
|
||||
model_identifier = "OmniVoice"
|
||||
|
||||
if os.path.isabs(model_identifier) and os.path.exists(model_identifier):
|
||||
return model_identifier
|
||||
|
||||
if os.path.isdir(model_identifier):
|
||||
return model_identifier
|
||||
|
||||
if model_identifier.startswith("local:"):
|
||||
local_name = model_identifier[6:]
|
||||
for base_path in get_all_tts_model_paths("TTS"):
|
||||
for folder_name in ("omnivoice", "OmniVoice", ""):
|
||||
candidate = (
|
||||
os.path.join(base_path, folder_name, local_name)
|
||||
if folder_name else os.path.join(base_path, local_name)
|
||||
)
|
||||
if os.path.exists(candidate):
|
||||
print(f"📁 Using local OmniVoice model: {candidate}")
|
||||
return candidate
|
||||
raise FileNotFoundError(f"Local OmniVoice model not found: {local_name}")
|
||||
|
||||
if model_identifier in self.MODELS:
|
||||
return self.get_model_path(model_identifier)
|
||||
|
||||
raise FileNotFoundError(f"Unknown OmniVoice model identifier: {model_identifier}")
|
||||
|
||||
def get_model_path(self, model_name: str) -> str:
|
||||
model_dir = os.path.join(self.base_path, model_name)
|
||||
if not os.path.exists(model_dir) or not self._is_omnivoice_model_complete(model_dir):
|
||||
return self.download_model(model_name, force=os.path.exists(model_dir))
|
||||
return model_dir
|
||||
|
||||
def download_model(self, model_name: str, force: bool = False) -> str:
|
||||
if model_name not in self.MODELS:
|
||||
raise ValueError(f"Unknown OmniVoice model: {model_name}")
|
||||
|
||||
model_info = self.MODELS[model_name]
|
||||
repo_id = str(model_info["repo_id"])
|
||||
model_dir = os.path.join(self.base_path, model_name)
|
||||
|
||||
print(f"\n{'=' * 60}")
|
||||
print("📦 OmniVoice Model Download")
|
||||
print(f"{'=' * 60}")
|
||||
print(f"Model: {model_name}")
|
||||
print(f"Description: {model_info['description']}")
|
||||
print(f"Repository: {repo_id}")
|
||||
print(f"Target: {model_dir}")
|
||||
print(f"{'=' * 60}\n")
|
||||
|
||||
try:
|
||||
with _suppress_hf_http_logs():
|
||||
snapshot_download(
|
||||
repo_id=repo_id,
|
||||
local_dir=model_dir,
|
||||
local_dir_use_symlinks=False,
|
||||
resume_download=True,
|
||||
force_download=force,
|
||||
)
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"Failed to download {model_name} from {repo_id}: {e}") from e
|
||||
|
||||
if not self._is_omnivoice_model_complete(model_dir):
|
||||
raise RuntimeError(f"Downloaded OmniVoice model is incomplete: {model_dir}")
|
||||
|
||||
print(f"✅ OmniVoice model ready: {model_dir}")
|
||||
return model_dir
|
||||
|
||||
def _is_omnivoice_model_complete(self, model_dir: str) -> bool:
|
||||
if not os.path.isdir(model_dir):
|
||||
return False
|
||||
|
||||
required_files = self.MODELS["OmniVoice"]["required_files"]
|
||||
missing = [
|
||||
rel_path
|
||||
for rel_path in required_files
|
||||
if not os.path.exists(os.path.join(model_dir, rel_path))
|
||||
]
|
||||
if missing:
|
||||
print(f"❌ OmniVoice model incomplete. Missing {len(missing)} file(s):")
|
||||
for rel_path in missing:
|
||||
print(f" - {rel_path}")
|
||||
return False
|
||||
return True
|
||||
@@ -0,0 +1,263 @@
|
||||
"""
|
||||
OmniVoice engine wrapper.
|
||||
|
||||
Wraps the official `omnivoice` package with ComfyUI-friendly lifecycle hooks.
|
||||
"""
|
||||
|
||||
import importlib
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
import types
|
||||
from contextlib import contextmanager
|
||||
from typing import Any, Optional
|
||||
|
||||
import torch
|
||||
|
||||
current_dir = os.path.dirname(__file__)
|
||||
engines_dir = os.path.dirname(current_dir)
|
||||
project_root = os.path.dirname(engines_dir)
|
||||
if project_root not in sys.path:
|
||||
sys.path.insert(0, project_root)
|
||||
|
||||
from engines.omnivoice.omnivoice_downloader import OmniVoiceDownloader
|
||||
from engines.omnivoice.progress_callback import OmniVoiceGenerationProgress
|
||||
from utils.device import resolve_torch_device
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _suppress_transformers_logs():
|
||||
logger = logging.getLogger("transformers.integrations.tensor_parallel")
|
||||
|
||||
class _TensorParallelPlanFilter(logging.Filter):
|
||||
_suppressed_prefixes = (
|
||||
"The following layers were not sharded:",
|
||||
"The following TP rules were not applied on any of the layers:",
|
||||
)
|
||||
|
||||
def filter(self, record):
|
||||
return not record.getMessage().startswith(self._suppressed_prefixes)
|
||||
|
||||
message_filter = _TensorParallelPlanFilter()
|
||||
logger.addFilter(message_filter)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
logger.removeFilter(message_filter)
|
||||
|
||||
|
||||
class OmniVoiceEngine:
|
||||
"""Thin wrapper around the official OmniVoice model."""
|
||||
|
||||
SAMPLE_RATE = 24000
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model_name: str = "OmniVoice",
|
||||
device: str = "auto",
|
||||
dtype: str = "auto",
|
||||
model_dir: Optional[str] = None,
|
||||
):
|
||||
self.model_name = model_name
|
||||
self.device = resolve_torch_device(device)
|
||||
self.dtype = self._resolve_dtype(dtype, self.device)
|
||||
self.downloader = OmniVoiceDownloader()
|
||||
self.model_dir = model_dir or self.downloader.resolve_model_path(model_name)
|
||||
self._model = None
|
||||
|
||||
def _install_progress_hook(self):
|
||||
if self._model is None or getattr(self._model, "_tts_suite_progress_patched", False):
|
||||
return
|
||||
|
||||
original_forward = self._model.forward
|
||||
original_generate_iterative = self._model._generate_iterative
|
||||
|
||||
def wrapped_forward(model_self, *args, **kwargs):
|
||||
result = original_forward(*args, **kwargs)
|
||||
tracker = getattr(model_self, "_tts_suite_progress_tracker", None)
|
||||
if tracker is not None:
|
||||
current_step = getattr(model_self, "_tts_suite_progress_step", 0) + 1
|
||||
model_self._tts_suite_progress_step = current_step
|
||||
tracker.update(current_step, force=(current_step == 1 or current_step >= tracker.total_steps))
|
||||
return result
|
||||
|
||||
def wrapped_generate_iterative(model_self, task, gen_config):
|
||||
total_steps = int(getattr(gen_config, "num_step", 0) or 0)
|
||||
tracker = OmniVoiceGenerationProgress(total_steps)
|
||||
model_self._tts_suite_progress_tracker = tracker
|
||||
model_self._tts_suite_progress_step = 0
|
||||
try:
|
||||
result = original_generate_iterative(task, gen_config)
|
||||
tracker.end(getattr(model_self, "_tts_suite_progress_step", 0))
|
||||
return result
|
||||
except Exception:
|
||||
tracker.abort()
|
||||
raise
|
||||
finally:
|
||||
model_self._tts_suite_progress_tracker = None
|
||||
model_self._tts_suite_progress_step = 0
|
||||
|
||||
self._model.forward = types.MethodType(wrapped_forward, self._model)
|
||||
self._model._generate_iterative = types.MethodType(wrapped_generate_iterative, self._model)
|
||||
self._model._tts_suite_progress_tracker = None
|
||||
self._model._tts_suite_progress_step = 0
|
||||
self._model._tts_suite_progress_patched = True
|
||||
|
||||
@staticmethod
|
||||
def _resolve_dtype(dtype: str, device: str):
|
||||
normalized = str(dtype or "auto").lower()
|
||||
explicit = {
|
||||
"float16": torch.float16,
|
||||
"fp16": torch.float16,
|
||||
"bfloat16": torch.bfloat16,
|
||||
"bf16": torch.bfloat16,
|
||||
"float32": torch.float32,
|
||||
"fp32": torch.float32,
|
||||
}
|
||||
if normalized in explicit:
|
||||
if device == "cpu" and explicit[normalized] != torch.float32:
|
||||
return torch.float32
|
||||
return explicit[normalized]
|
||||
|
||||
if device == "cpu":
|
||||
return torch.float32
|
||||
if device == "cuda" and torch.cuda.is_available():
|
||||
major, _minor = torch.cuda.get_device_capability()
|
||||
if major >= 8:
|
||||
return torch.bfloat16
|
||||
return torch.float16
|
||||
if device == "xpu":
|
||||
return torch.bfloat16
|
||||
return torch.float16
|
||||
|
||||
def _import_official_package(self):
|
||||
importlib.invalidate_caches()
|
||||
|
||||
nodes_dir = os.path.join(project_root, "nodes")
|
||||
blocked_paths = {
|
||||
os.path.abspath(engines_dir),
|
||||
os.path.abspath(nodes_dir),
|
||||
}
|
||||
original_sys_path = list(sys.path)
|
||||
sys.path[:] = [
|
||||
path for path in sys.path
|
||||
if os.path.abspath(path or os.getcwd()) not in blocked_paths
|
||||
]
|
||||
|
||||
stale_modules = []
|
||||
for module_name, module in list(sys.modules.items()):
|
||||
if not (module_name == "omnivoice" or module_name.startswith("omnivoice.")):
|
||||
continue
|
||||
module_file = getattr(module, "__file__", "") or ""
|
||||
if module_file and os.path.abspath(module_file).startswith(os.path.abspath(project_root)):
|
||||
stale_modules.append((module_name, module))
|
||||
del sys.modules[module_name]
|
||||
|
||||
try:
|
||||
from omnivoice import OmniVoice # type: ignore
|
||||
except Exception as e:
|
||||
for module_name, module in stale_modules:
|
||||
sys.modules.setdefault(module_name, module)
|
||||
raise ImportError(
|
||||
"Failed to import the official `omnivoice` package in the active ComfyUI Python "
|
||||
"environment. This is usually a missing install or import-path collision. "
|
||||
f"Active Python: {sys.executable}"
|
||||
) from e
|
||||
finally:
|
||||
sys.path[:] = original_sys_path
|
||||
|
||||
return OmniVoice
|
||||
|
||||
def _loaded_model_device(self) -> Optional[str]:
|
||||
if self._model is None:
|
||||
return None
|
||||
try:
|
||||
first_param = next(self._model.parameters())
|
||||
return str(first_param.device)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def _ensure_model_device(self):
|
||||
if self._model is None:
|
||||
return
|
||||
current_device = self._loaded_model_device()
|
||||
if current_device == self.device:
|
||||
return
|
||||
self.to(self.device)
|
||||
|
||||
def _ensure_model_loaded(self):
|
||||
if self._model is not None:
|
||||
self._ensure_model_device()
|
||||
return
|
||||
|
||||
OmniVoice = self._import_official_package()
|
||||
print(f"🔄 Loading OmniVoice model via official package")
|
||||
print(f" Path: {self.model_dir}")
|
||||
print(f" Device: {self.device} | Dtype: {self.dtype}")
|
||||
|
||||
with _suppress_transformers_logs():
|
||||
self._model = OmniVoice.from_pretrained(
|
||||
self.model_dir,
|
||||
device_map=self.device,
|
||||
dtype=self.dtype,
|
||||
load_asr=False,
|
||||
)
|
||||
self._install_progress_hook()
|
||||
print("✅ OmniVoice model loaded successfully")
|
||||
|
||||
@property
|
||||
def sampling_rate(self) -> int:
|
||||
if self._model is not None and getattr(self._model, "sampling_rate", None):
|
||||
return int(self._model.sampling_rate)
|
||||
return self.SAMPLE_RATE
|
||||
|
||||
def create_voice_clone_prompt(
|
||||
self,
|
||||
ref_audio: Any,
|
||||
ref_text: Optional[str] = None,
|
||||
preprocess_prompt: bool = True,
|
||||
):
|
||||
self._ensure_model_loaded()
|
||||
if ref_text is None or not str(ref_text).strip():
|
||||
raise ValueError(
|
||||
"OmniVoice voice cloning in this suite requires reference text. "
|
||||
"Use Character Voices or a narrator voice with a matching .reference.txt file."
|
||||
)
|
||||
|
||||
return self._model.create_voice_clone_prompt(
|
||||
ref_audio=ref_audio,
|
||||
ref_text=ref_text,
|
||||
preprocess_prompt=preprocess_prompt,
|
||||
)
|
||||
|
||||
def generate(self, **kwargs):
|
||||
self._ensure_model_loaded()
|
||||
return self._model.generate(**kwargs)
|
||||
|
||||
def to(self, device: str):
|
||||
self.device = resolve_torch_device(device)
|
||||
if self._model is None:
|
||||
return self
|
||||
|
||||
try:
|
||||
if hasattr(self._model, "to"):
|
||||
self._model = self._model.to(self.device)
|
||||
except Exception as e:
|
||||
print(f"⚠️ OmniVoice base model move to {self.device} failed: {e}")
|
||||
|
||||
for attr_name in ("audio_tokenizer",):
|
||||
attr_value = getattr(self._model, attr_name, None)
|
||||
if hasattr(attr_value, "to"):
|
||||
try:
|
||||
setattr(self._model, attr_name, attr_value.to(self.device))
|
||||
except Exception as e:
|
||||
print(f"⚠️ OmniVoice {attr_name} move to {self.device} failed: {e}")
|
||||
|
||||
return self
|
||||
|
||||
def __getattr__(self, name: str):
|
||||
if name.startswith("_"):
|
||||
raise AttributeError(name)
|
||||
if self._model is not None and hasattr(self._model, name):
|
||||
return getattr(self._model, name)
|
||||
raise AttributeError(name)
|
||||
@@ -0,0 +1,63 @@
|
||||
"""
|
||||
OmniVoice progress callback.
|
||||
|
||||
Tracks real iterative decoding steps per second for OmniVoice generation.
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
|
||||
class OmniVoiceGenerationProgress:
|
||||
"""Render real iterative decode progress with it/s."""
|
||||
|
||||
def __init__(self, total_steps: int):
|
||||
self.total_steps = max(1, int(total_steps or 1))
|
||||
self.start_time = time.time()
|
||||
self.last_print_time = self.start_time
|
||||
self.last_step = 0
|
||||
self.rendered = False
|
||||
|
||||
def update(self, current_step: int, force: bool = False):
|
||||
current_step = max(0, min(int(current_step), self.total_steps))
|
||||
current_time = time.time()
|
||||
should_print = (
|
||||
force
|
||||
or current_step == 1
|
||||
or current_time - self.last_print_time >= 0.5
|
||||
)
|
||||
if not should_print:
|
||||
return
|
||||
|
||||
delta_steps = current_step - self.last_step
|
||||
delta_time = max(current_time - self.last_print_time, 1e-6)
|
||||
its = delta_steps / delta_time
|
||||
elapsed = current_time - self.start_time
|
||||
remaining_steps = max(self.total_steps - current_step, 0)
|
||||
eta = remaining_steps / its if its > 0 else 0.0
|
||||
|
||||
bar_width = 12
|
||||
filled = int(bar_width * current_step / self.total_steps) if self.total_steps > 0 else 0
|
||||
filled = min(filled, bar_width)
|
||||
progress_bar = f"[{'█' * filled}{'░' * (bar_width - filled)}] {current_step}/{self.total_steps}"
|
||||
print(
|
||||
f"\r Progress: {progress_bar} | {its:.1f} it/s | {elapsed:.0f}s | ETA {eta:.0f}s ",
|
||||
end="",
|
||||
flush=True,
|
||||
)
|
||||
self.last_print_time = current_time
|
||||
self.last_step = current_step
|
||||
self.rendered = True
|
||||
|
||||
def end(self, final_step: int):
|
||||
final_step = max(0, int(final_step))
|
||||
total_time = max(time.time() - self.start_time, 1e-6)
|
||||
avg_its = final_step / total_time
|
||||
if self.rendered:
|
||||
print(
|
||||
f"\r Complete: {final_step} steps in {total_time:.1f}s (avg {avg_its:.1f} it/s)"
|
||||
+ " " * 30
|
||||
)
|
||||
|
||||
def abort(self):
|
||||
if self.rendered:
|
||||
print()
|
||||
@@ -55,6 +55,7 @@ from utils.audio.edit_post_processor import process_segments as apply_edit_post_
|
||||
from utils.text.character_parser import CharacterParser
|
||||
from utils.text.pause_processor import PauseTagProcessor
|
||||
from utils.text.segment_parameters import apply_segment_parameters
|
||||
from utils.text.index_tts_emotion import resolve_inline_emotion
|
||||
from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment
|
||||
from utils.voice.discovery import get_character_mapping
|
||||
from engines.adapters.index_tts_adapter import IndexTTSAdapter
|
||||
@@ -248,15 +249,44 @@ class IndexTTSProcessor:
|
||||
print(f"⚠️ {character}: No voice available")
|
||||
|
||||
# Define TTS generation function for pause processor
|
||||
def tts_generate_func(text_content: str, segment_params: Optional[Dict[str, Any]] = None) -> torch.Tensor:
|
||||
def tts_generate_func(
|
||||
text_content: str,
|
||||
segment_params: Optional[Dict[str, Any]] = None,
|
||||
character_name: Optional[str] = None,
|
||||
emotion_reference: Optional[str] = None,
|
||||
) -> torch.Tensor:
|
||||
# Import references for nested function scope
|
||||
import torchaudio as ta
|
||||
import tempfile as tf
|
||||
"""TTS generation function for pause tag processor with per-segment parameters"""
|
||||
# Apply segment parameters if provided
|
||||
current_config = dict(self.config)
|
||||
if segment_params:
|
||||
current_config = apply_segment_parameters(current_config, segment_params, "index_tts")
|
||||
import torchaudio as ta
|
||||
import tempfile as tf
|
||||
"""TTS generation function for pause tag processor with per-segment parameters"""
|
||||
|
||||
def generate_with_context(context: str, **kwargs):
|
||||
"""Add segment context without hiding the original engine error."""
|
||||
try:
|
||||
return self.adapter.generate(**kwargs)
|
||||
except Exception as exc:
|
||||
# Avoid leaking temporary WAV files when an engine
|
||||
# failure happens before the normal success cleanup.
|
||||
for path in (kwargs.get("speaker_audio"), kwargs.get("emotion_audio")):
|
||||
if (
|
||||
isinstance(path, str)
|
||||
and path.startswith(tf.gettempdir())
|
||||
and os.path.exists(path)
|
||||
):
|
||||
try:
|
||||
os.unlink(path)
|
||||
except OSError:
|
||||
pass
|
||||
raise RuntimeError(
|
||||
f"IndexTTS-2 segment failed ({context}): "
|
||||
f"{type(exc).__name__}: {exc}"
|
||||
) from exc
|
||||
|
||||
# Apply segment parameters if provided
|
||||
current_config = dict(self.config)
|
||||
if segment_params:
|
||||
current_config = apply_segment_parameters(current_config, segment_params, "index_tts")
|
||||
current_config, _ = resolve_inline_emotion(current_config)
|
||||
|
||||
if '[' in text_content and ']' in text_content:
|
||||
# Handle character switching with emotion parsing using modularized parser
|
||||
@@ -299,8 +329,10 @@ class IndexTTSProcessor:
|
||||
else:
|
||||
print(f"🐛 Could not resolve emotion reference '{emotion}'")
|
||||
|
||||
# Fall back to config emotion_audio if no tag emotion
|
||||
if not emotion_audio_path:
|
||||
# Fall back to connected emotion audio if no character
|
||||
# reference was specified. Inline vector/text emotion
|
||||
# can now be blended with this audio source.
|
||||
if not emotion_audio_path:
|
||||
emotion_from_config = self.config.get('emotion_audio')
|
||||
# Process emotion audio for this character
|
||||
if emotion_from_config:
|
||||
@@ -322,26 +354,20 @@ class IndexTTSProcessor:
|
||||
else:
|
||||
print(f"🎭 No emotion audio for character: {character} (no tag emotion, no connected engine emotion)")
|
||||
|
||||
# Prioritize character emotion reference over global emotion controls
|
||||
# If character has specific emotion ref, disable global emotion controls
|
||||
if emotion_audio_path:
|
||||
# Character has specific emotion - use only that emotion reference
|
||||
character_emotion_vector = None
|
||||
character_use_emotion_text = False
|
||||
character_emotion_text = None
|
||||
else:
|
||||
# No character emotion - use global emotion settings (from current_config with segment params)
|
||||
character_emotion_vector = current_config.get('emotion_vector')
|
||||
character_use_emotion_text = current_config.get('use_emotion_text', False)
|
||||
character_emotion_text = current_config.get('emotion_text')
|
||||
|
||||
# Handle dynamic QwenEmotion template
|
||||
if character_use_emotion_text and character_emotion_text and current_config.get('is_dynamic_template', False):
|
||||
character_emotion_text = self._process_dynamic_emotion_template(character_emotion_text, segment_text)
|
||||
# Audio and vector/text controls are blended by
|
||||
# IndexTTS-2 when both are present.
|
||||
character_emotion_vector = current_config.get('emotion_vector')
|
||||
character_use_emotion_text = current_config.get('use_emotion_text', False)
|
||||
character_emotion_text = current_config.get('emotion_text')
|
||||
|
||||
# Handle dynamic QwenEmotion template
|
||||
if character_use_emotion_text and character_emotion_text and current_config.get('is_dynamic_template', False):
|
||||
character_emotion_text = self._process_dynamic_emotion_template(character_emotion_text, segment_text)
|
||||
|
||||
# Generate audio for this character segment (use parameters from current_config with segment overrides)
|
||||
segment_result = self.adapter.generate(
|
||||
text=segment_text,
|
||||
segment_result = generate_with_context(
|
||||
f"character '{character}', text={segment_text[:120]!r}",
|
||||
text=segment_text,
|
||||
speaker_audio=speaker_audio_path,
|
||||
emotion_audio=emotion_audio_path,
|
||||
emotion_alpha=current_config.get('emotion_alpha', 1.0),
|
||||
@@ -383,10 +409,13 @@ class IndexTTSProcessor:
|
||||
return torch.cat(segment_audio_parts, dim=-1)
|
||||
else:
|
||||
return torch.zeros(1, 0)
|
||||
else:
|
||||
# Simple text segment without character switching - use narrator voice
|
||||
narrator_audio = voice_refs.get("narrator")
|
||||
speaker_audio_path = None
|
||||
else:
|
||||
# Character tags are removed before this branch is called
|
||||
# for parameterized segments. Keep the parsed character
|
||||
# here so [Bob:emotion] does not silently use the narrator.
|
||||
character_audio = voice_refs.get(character_name) if character_name else None
|
||||
narrator_audio = character_audio or voice_refs.get("narrator")
|
||||
speaker_audio_path = None
|
||||
if narrator_audio and 'waveform' in narrator_audio:
|
||||
with tf.NamedTemporaryFile(suffix='.wav', delete=False) as tmp_file:
|
||||
save_audio_safe(tmp_file.name, narrator_audio['waveform'], narrator_audio['sample_rate'])
|
||||
@@ -399,8 +428,22 @@ class IndexTTSProcessor:
|
||||
save_audio_safe(tmp_file.name, narrator_voice_dict['waveform'], narrator_voice_dict['sample_rate'])
|
||||
speaker_audio_path = tmp_file.name
|
||||
|
||||
# Handle emotion_audio - convert tensor to file path if needed
|
||||
emotion_audio_path = self.config.get('emotion_audio')
|
||||
# Character emotion references are preserved on the parsed
|
||||
# segment even after its tag has been stripped. Resolve
|
||||
# that reference before falling back to the node's global
|
||||
# emotion audio.
|
||||
emotion_audio_path = None
|
||||
if emotion_reference:
|
||||
emotion_audio_path = self.character_parser.get_emotion_voice_path(emotion_reference)
|
||||
if emotion_audio_path:
|
||||
print(
|
||||
f"😊 Using emotion reference from segment: "
|
||||
f"{emotion_reference} -> {emotion_audio_path}"
|
||||
)
|
||||
else:
|
||||
print(f"⚠️ Could not resolve emotion reference '{emotion_reference}'")
|
||||
if not emotion_audio_path:
|
||||
emotion_audio_path = self.config.get('emotion_audio')
|
||||
# Process emotion audio from config
|
||||
if emotion_audio_path and isinstance(emotion_audio_path, dict) and 'waveform' in emotion_audio_path:
|
||||
# Convert tensor to temporary file
|
||||
@@ -419,25 +462,19 @@ class IndexTTSProcessor:
|
||||
else:
|
||||
print(f"🎭 No emotion audio for simple text segment (no connected engine emotion)")
|
||||
|
||||
# Prioritize connected emotion_audio over global emotion controls
|
||||
# For simple text, use emotion_audio from config if available, else use global settings (with segment params)
|
||||
if emotion_audio_path:
|
||||
# Engine emotion_audio connected - use only that
|
||||
simple_emotion_vector = None
|
||||
simple_use_emotion_text = False
|
||||
simple_emotion_text = None
|
||||
else:
|
||||
# No engine emotion_audio - use global emotion settings (from current_config with segment params)
|
||||
simple_emotion_vector = current_config.get('emotion_vector')
|
||||
simple_use_emotion_text = current_config.get('use_emotion_text', False)
|
||||
simple_emotion_text = current_config.get('emotion_text')
|
||||
# Audio and vector/text controls are blended by
|
||||
# IndexTTS-2 when both are present.
|
||||
simple_emotion_vector = current_config.get('emotion_vector')
|
||||
simple_use_emotion_text = current_config.get('use_emotion_text', False)
|
||||
simple_emotion_text = current_config.get('emotion_text')
|
||||
|
||||
# Handle dynamic QwenEmotion template
|
||||
if simple_use_emotion_text and simple_emotion_text and current_config.get('is_dynamic_template', False):
|
||||
simple_emotion_text = self._process_dynamic_emotion_template(simple_emotion_text, text_content)
|
||||
|
||||
# Handle dynamic QwenEmotion template
|
||||
if simple_use_emotion_text and simple_emotion_text and current_config.get('is_dynamic_template', False):
|
||||
simple_emotion_text = self._process_dynamic_emotion_template(simple_emotion_text, text_content)
|
||||
|
||||
result = self.adapter.generate(
|
||||
text=text_content,
|
||||
result = generate_with_context(
|
||||
f"text={text_content[:120]!r}",
|
||||
text=text_content,
|
||||
speaker_audio=speaker_audio_path,
|
||||
emotion_audio=emotion_audio_path,
|
||||
emotion_alpha=current_config.get('emotion_alpha', 1.0),
|
||||
@@ -505,7 +542,12 @@ class IndexTTSProcessor:
|
||||
print(f" ⚠️ Segment {seg_idx + 1}: Only inline edit tags found, skipping empty TTS segment")
|
||||
continue
|
||||
|
||||
segment_audio = tts_generate_func(clean_segment_text or subseg_content, seg_obj.parameters)
|
||||
segment_audio = tts_generate_func(
|
||||
clean_segment_text or subseg_content,
|
||||
seg_obj.parameters,
|
||||
seg_obj.character,
|
||||
seg_obj.emotion,
|
||||
)
|
||||
if isinstance(segment_audio, torch.Tensor) and segment_audio.numel() > 0:
|
||||
if segment_audio.dim() == 1:
|
||||
segment_audio = segment_audio.unsqueeze(0)
|
||||
@@ -533,11 +575,13 @@ class IndexTTSProcessor:
|
||||
segments, clean_text = self.pause_processor.parse_pause_tags(text)
|
||||
|
||||
# Create a wrapper that handles pause segments
|
||||
def tts_generate_with_params(text_content: str) -> torch.Tensor:
|
||||
"""Wrapper for pause processor that passes segment parameters"""
|
||||
# For pause-based processing, use first segment's parameters
|
||||
segment_params = character_segment_objects[0].parameters if character_segment_objects and character_segment_objects[0].parameters else None
|
||||
return tts_generate_func(text_content, segment_params)
|
||||
def tts_generate_with_params(text_content: str) -> torch.Tensor:
|
||||
"""Wrapper for pause processor that passes segment parameters"""
|
||||
# For pause-based processing, use first segment's parameters
|
||||
segment_params = character_segment_objects[0].parameters if character_segment_objects and character_segment_objects[0].parameters else None
|
||||
character_name = character_segment_objects[0].character if character_segment_objects else None
|
||||
emotion_reference = character_segment_objects[0].emotion if character_segment_objects else None
|
||||
return tts_generate_func(text_content, segment_params, character_name, emotion_reference)
|
||||
|
||||
# Generate audio with pauses
|
||||
if segments:
|
||||
@@ -547,9 +591,11 @@ class IndexTTSProcessor:
|
||||
sample_rate=self.sample_rate
|
||||
)
|
||||
else:
|
||||
# No pause tags, generate directly
|
||||
segment_params = character_segment_objects[0].parameters if character_segment_objects and character_segment_objects[0].parameters else None
|
||||
result = tts_generate_func(text, segment_params)
|
||||
# No pause tags, generate directly
|
||||
segment_params = character_segment_objects[0].parameters if character_segment_objects and character_segment_objects[0].parameters else None
|
||||
character_name = character_segment_objects[0].character if character_segment_objects else None
|
||||
emotion_reference = character_segment_objects[0].emotion if character_segment_objects else None
|
||||
result = tts_generate_func(text, segment_params, character_name, emotion_reference)
|
||||
|
||||
# Ensure correct tensor format
|
||||
if isinstance(result, torch.Tensor):
|
||||
|
||||
@@ -40,11 +40,13 @@ class TextEncoder256(nn.Module):
|
||||
)
|
||||
self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
|
||||
|
||||
def forward(self, phone, pitch, lengths):
|
||||
if pitch == None:
|
||||
x = self.emb_phone(phone)
|
||||
else:
|
||||
x = self.emb_phone(phone) + self.emb_pitch(pitch)
|
||||
def forward(self, phone, pitch, lengths):
|
||||
phone = phone.to(dtype=self.emb_phone.weight.dtype)
|
||||
if pitch is None:
|
||||
x = self.emb_phone(phone)
|
||||
else:
|
||||
pitch = pitch.to(dtype=torch.long)
|
||||
x = self.emb_phone(phone) + self.emb_pitch(pitch)
|
||||
x = x * math.sqrt(self.hidden_channels) # [b, t, h]
|
||||
x = self.lrelu(x)
|
||||
x = torch.transpose(x, 1, -1) # [b, h, t]
|
||||
@@ -87,11 +89,13 @@ class TextEncoder768(nn.Module):
|
||||
)
|
||||
self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
|
||||
|
||||
def forward(self, phone, pitch, lengths):
|
||||
if pitch is None:
|
||||
x = self.emb_phone(phone)
|
||||
else:
|
||||
x = self.emb_phone(phone) + self.emb_pitch(pitch)
|
||||
def forward(self, phone, pitch, lengths):
|
||||
phone = phone.to(dtype=self.emb_phone.weight.dtype)
|
||||
if pitch is None:
|
||||
x = self.emb_phone(phone)
|
||||
else:
|
||||
pitch = pitch.to(dtype=torch.long)
|
||||
x = self.emb_phone(phone) + self.emb_pitch(pitch)
|
||||
x = x * math.sqrt(self.hidden_channels) # [b, t, h]
|
||||
x = self.lrelu(x)
|
||||
x = torch.transpose(x, 1, -1) # [b, h, t]
|
||||
@@ -1403,4 +1407,4 @@ class SynthesizerTrn(nn.Module):
|
||||
z_p = self.flow(z, y_mask, g=g_src)
|
||||
z_hat = self.flow(z_p, y_mask, g=g_tgt, reverse=True)
|
||||
o_hat = self.dec(z_hat * y_mask, g=g_tgt)
|
||||
return o_hat, y_mask, (z, z_p, z_hat)
|
||||
return o_hat, y_mask, (z, z_p, z_hat)
|
||||
|
||||
@@ -215,7 +215,8 @@ class VC(FeatureExtractor):
|
||||
f0_max=f0_max,
|
||||
)
|
||||
p_len = min(pitch.shape[0], pitchf.shape[0])
|
||||
pitch = pitch[:p_len].astype(np.int64 if self.device != 'mps' else np.float32)
|
||||
# Pitch values are categorical embedding indices, including on MPS.
|
||||
pitch = pitch[:p_len].astype(np.int64)
|
||||
pitchf = pitchf[:p_len].astype(np.float32)
|
||||
pitch = torch.from_numpy(pitch).to(self.device).unsqueeze(0)
|
||||
pitchf = torch.from_numpy(pitchf).to(self.device).unsqueeze(0)
|
||||
|
||||
@@ -719,6 +719,8 @@ class VibeVoiceEngine:
|
||||
# Extract waveform from ComfyUI audio format
|
||||
waveform = voice_ref["waveform"]
|
||||
input_sample_rate = voice_ref.get("sample_rate", 24000)
|
||||
shape = tuple(waveform.shape) if hasattr(waveform, "shape") else "unknown"
|
||||
print(f"🎵 VibeVoice ENGINE: Using waveform input for speaker {i+1} - shape: {shape}, sr: {input_sample_rate}")
|
||||
|
||||
# Convert to numpy
|
||||
if isinstance(waveform, torch.Tensor):
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 181 KiB |
@@ -0,0 +1,918 @@
|
||||
{
|
||||
"id": "d7d0e47d-2b68-4f88-9a5e-f15a2d3c8b91",
|
||||
"revision": 0,
|
||||
"last_node_id": 24,
|
||||
"last_link_id": 123,
|
||||
"nodes": [
|
||||
{
|
||||
"id": 2,
|
||||
"type": "CharacterVoicesNode",
|
||||
"pos": [
|
||||
-2660,
|
||||
350
|
||||
],
|
||||
"size": [
|
||||
240,
|
||||
186
|
||||
],
|
||||
"flags": {},
|
||||
"order": 2,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "opt_audio_input",
|
||||
"shape": 7,
|
||||
"type": "AUDIO",
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "opt_narrator",
|
||||
"type": "NARRATOR_VOICE",
|
||||
"links": [
|
||||
101
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "character_name",
|
||||
"type": "STRING",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"cnr_id": "tts_audio_suite",
|
||||
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
|
||||
"Node name for S&R": "CharacterVoicesNode"
|
||||
},
|
||||
"widgets_values": [
|
||||
"voices_examples/female/female_01.wav",
|
||||
"",
|
||||
""
|
||||
],
|
||||
"color": "#432",
|
||||
"bgcolor": "#653"
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "CharacterVoicesNode",
|
||||
"pos": [
|
||||
-2660,
|
||||
580
|
||||
],
|
||||
"size": [
|
||||
240,
|
||||
186
|
||||
],
|
||||
"flags": {},
|
||||
"order": 3,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "opt_audio_input",
|
||||
"shape": 7,
|
||||
"type": "AUDIO",
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "opt_narrator",
|
||||
"type": "NARRATOR_VOICE",
|
||||
"links": [
|
||||
102
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "character_name",
|
||||
"type": "STRING",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"cnr_id": "tts_audio_suite",
|
||||
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
|
||||
"Node name for S&R": "CharacterVoicesNode"
|
||||
},
|
||||
"widgets_values": [
|
||||
"voices_examples/Clint_Eastwood CC3 (enhanced2).wav",
|
||||
"",
|
||||
""
|
||||
],
|
||||
"color": "#432",
|
||||
"bgcolor": "#653"
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "StringMultilineTagEditor",
|
||||
"pos": [
|
||||
-1560,
|
||||
570
|
||||
],
|
||||
"size": [
|
||||
900,
|
||||
600
|
||||
],
|
||||
"flags": {},
|
||||
"order": 7,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "text",
|
||||
"type": "STRING",
|
||||
"links": [
|
||||
105
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"cnr_id": "tts_audio_suite",
|
||||
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
|
||||
"Node name for S&R": "StringMultilineTagEditor",
|
||||
"ttsTagEditorInlineEngine": "step_audio_editx"
|
||||
},
|
||||
"widgets_values": [
|
||||
"On Tuesdays, the pigeons held parliament.\n[Alice|seed:42] They debated the ethics of breadcrumbs and the metaphysics of flight.\n[Bob] One particularly philosophical pigeon named Thistle believed gravity was just a suggestion.\n[Cowboy] He once floated for three minutes straight, fueled by sheer conviction and a gust of rebellious wind."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "StringMultilineTagEditor",
|
||||
"pos": [
|
||||
-1570,
|
||||
1570
|
||||
],
|
||||
"size": [
|
||||
900,
|
||||
600
|
||||
],
|
||||
"flags": {},
|
||||
"order": 6,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "text",
|
||||
"type": "STRING",
|
||||
"links": [
|
||||
106
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"cnr_id": "tts_audio_suite",
|
||||
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
|
||||
"Node name for S&R": "StringMultilineTagEditor",
|
||||
"ttsTagEditorInlineEngine": "step_audio_editx"
|
||||
},
|
||||
"widgets_values": [
|
||||
"1\n00:00:01,000 --> 00:00:04,000\nOn Tuesdays, the pigeons held parliament.\n\n2\n00:00:04,500 --> 00:00:09,500\n[Alice] They debated the ethics of breadcrumbs and the metaphysics of flight.\n\n3\n00:00:10,000 --> 00:00:14,000\n[Bob] One particularly philosophical pigeon named Thistle believed gravity was just a suggestion.\n\n4\n00:00:14,500 --> 00:00:20,000\n[Cowboy] He once floated for three minutes straight, fueled by sheer conviction and a gust of rebellious wind."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "FishAudioS2EngineNode",
|
||||
"pos": [
|
||||
-2240,
|
||||
360
|
||||
],
|
||||
"size": [
|
||||
410,
|
||||
474
|
||||
],
|
||||
"flags": {},
|
||||
"order": 12,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"label": "Speaker 2",
|
||||
"name": "speaker2",
|
||||
"shape": 7,
|
||||
"type": "*",
|
||||
"link": 101
|
||||
},
|
||||
{
|
||||
"label": "Speaker 3",
|
||||
"name": "speaker3",
|
||||
"shape": 7,
|
||||
"type": "*",
|
||||
"link": 102
|
||||
},
|
||||
{
|
||||
"label": "Speaker 4",
|
||||
"name": "speaker4",
|
||||
"shape": 7,
|
||||
"type": "*",
|
||||
"link": 111
|
||||
},
|
||||
{
|
||||
"label": "Speaker 5",
|
||||
"name": "speaker5",
|
||||
"shape": 7,
|
||||
"type": "*",
|
||||
"link": 112
|
||||
},
|
||||
{
|
||||
"name": "speaker6",
|
||||
"type": "*",
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "TTS_engine",
|
||||
"type": "TTS_ENGINE",
|
||||
"links": [
|
||||
122
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"cnr_id": "chatterbox_srt_voice",
|
||||
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
|
||||
"Node name for S&R": "FishAudioS2EngineNode"
|
||||
},
|
||||
"widgets_values": [
|
||||
"s2-pro",
|
||||
"auto",
|
||||
0.8,
|
||||
0.8,
|
||||
1.1,
|
||||
200,
|
||||
1024,
|
||||
"8192",
|
||||
true,
|
||||
true,
|
||||
"bfloat16",
|
||||
true,
|
||||
"none",
|
||||
"Native Multi-Speaker",
|
||||
"Auto Inline Tag"
|
||||
],
|
||||
"color": "#222",
|
||||
"bgcolor": "#000"
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "UnifiedTTSTextNode",
|
||||
"pos": [
|
||||
-570,
|
||||
360
|
||||
],
|
||||
"size": [
|
||||
420,
|
||||
390
|
||||
],
|
||||
"flags": {},
|
||||
"order": 14,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "TTS_engine",
|
||||
"type": "TTS_ENGINE",
|
||||
"link": 122
|
||||
},
|
||||
{
|
||||
"name": "opt_narrator",
|
||||
"shape": 7,
|
||||
"type": "*",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "text",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "text"
|
||||
},
|
||||
"link": 105
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "audio",
|
||||
"type": "AUDIO",
|
||||
"links": [
|
||||
109
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "generation_info",
|
||||
"type": "STRING",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"cnr_id": "tts_audio_suite",
|
||||
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
|
||||
"Node name for S&R": "UnifiedTTSTextNode"
|
||||
},
|
||||
"widgets_values": [
|
||||
"",
|
||||
"voices_examples/David_Attenborough CC3.wav",
|
||||
1552347960,
|
||||
"fixed",
|
||||
true,
|
||||
400,
|
||||
"auto",
|
||||
100,
|
||||
true,
|
||||
0
|
||||
],
|
||||
"color": "#223",
|
||||
"bgcolor": "#335"
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "UnifiedTTSSRTNode",
|
||||
"pos": [
|
||||
-570,
|
||||
1490
|
||||
],
|
||||
"size": [
|
||||
440,
|
||||
560
|
||||
],
|
||||
"flags": {},
|
||||
"order": 15,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "TTS_engine",
|
||||
"type": "TTS_ENGINE",
|
||||
"link": 116
|
||||
},
|
||||
{
|
||||
"name": "opt_narrator",
|
||||
"shape": 7,
|
||||
"type": "*",
|
||||
"link": null
|
||||
},
|
||||
{
|
||||
"name": "srt_content",
|
||||
"type": "STRING",
|
||||
"widget": {
|
||||
"name": "srt_content"
|
||||
},
|
||||
"link": 106
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "audio",
|
||||
"type": "AUDIO",
|
||||
"links": [
|
||||
110
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "generation_info",
|
||||
"type": "STRING",
|
||||
"links": null
|
||||
},
|
||||
{
|
||||
"name": "timing_report",
|
||||
"type": "STRING",
|
||||
"links": null
|
||||
},
|
||||
{
|
||||
"name": "Adjusted_SRT",
|
||||
"type": "STRING",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"cnr_id": "chatterbox_srt_voice",
|
||||
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
|
||||
"Node name for S&R": "UnifiedTTSSRTNode"
|
||||
},
|
||||
"widgets_values": [
|
||||
"",
|
||||
"voices_examples/David_Attenborough CC3.wav",
|
||||
2286528432,
|
||||
"fixed",
|
||||
"pad_with_silence",
|
||||
true,
|
||||
0.01,
|
||||
1,
|
||||
0.5,
|
||||
2,
|
||||
0,
|
||||
false
|
||||
],
|
||||
"color": "#223",
|
||||
"bgcolor": "#335"
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"type": "PreviewAudio",
|
||||
"pos": [
|
||||
20,
|
||||
430
|
||||
],
|
||||
"size": [
|
||||
440,
|
||||
88
|
||||
],
|
||||
"flags": {},
|
||||
"order": 16,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "audio",
|
||||
"type": "AUDIO",
|
||||
"link": 109
|
||||
}
|
||||
],
|
||||
"outputs": [],
|
||||
"properties": {
|
||||
"cnr_id": "comfy-core",
|
||||
"ver": "0.3.39",
|
||||
"Node name for S&R": "PreviewAudio"
|
||||
},
|
||||
"widgets_values": []
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"type": "PreviewAudio",
|
||||
"pos": [
|
||||
20,
|
||||
1730
|
||||
],
|
||||
"size": [
|
||||
440,
|
||||
88
|
||||
],
|
||||
"flags": {},
|
||||
"order": 17,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "audio",
|
||||
"type": "AUDIO",
|
||||
"link": 110
|
||||
}
|
||||
],
|
||||
"outputs": [],
|
||||
"properties": {
|
||||
"cnr_id": "comfy-core",
|
||||
"ver": "0.3.39",
|
||||
"Node name for S&R": "PreviewAudio"
|
||||
},
|
||||
"widgets_values": []
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"type": "CharacterVoicesNode",
|
||||
"pos": [
|
||||
-2660,
|
||||
810
|
||||
],
|
||||
"size": [
|
||||
240,
|
||||
186
|
||||
],
|
||||
"flags": {},
|
||||
"order": 4,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "opt_audio_input",
|
||||
"shape": 7,
|
||||
"type": "AUDIO",
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "opt_narrator",
|
||||
"type": "NARRATOR_VOICE",
|
||||
"links": [
|
||||
111
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "character_name",
|
||||
"type": "STRING",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"cnr_id": "tts_audio_suite",
|
||||
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
|
||||
"Node name for S&R": "CharacterVoicesNode"
|
||||
},
|
||||
"widgets_values": [
|
||||
"voices_examples/female/female_02.wav",
|
||||
"",
|
||||
""
|
||||
],
|
||||
"color": "#432",
|
||||
"bgcolor": "#653"
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"type": "CharacterVoicesNode",
|
||||
"pos": [
|
||||
-2660,
|
||||
1040
|
||||
],
|
||||
"size": [
|
||||
240,
|
||||
186
|
||||
],
|
||||
"flags": {},
|
||||
"order": 5,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "opt_audio_input",
|
||||
"shape": 7,
|
||||
"type": "AUDIO",
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "opt_narrator",
|
||||
"type": "NARRATOR_VOICE",
|
||||
"links": [
|
||||
112
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "character_name",
|
||||
"type": "STRING",
|
||||
"links": null
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"cnr_id": "tts_audio_suite",
|
||||
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
|
||||
"Node name for S&R": "CharacterVoicesNode"
|
||||
},
|
||||
"widgets_values": [
|
||||
"voices_examples/higgs_audio/belinda.wav",
|
||||
"",
|
||||
""
|
||||
],
|
||||
"color": "#432",
|
||||
"bgcolor": "#653"
|
||||
},
|
||||
{
|
||||
"id": 14,
|
||||
"type": "MarkdownNote",
|
||||
"pos": [
|
||||
-610,
|
||||
870
|
||||
],
|
||||
"size": [
|
||||
460,
|
||||
260
|
||||
],
|
||||
"flags": {},
|
||||
"order": 1,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [],
|
||||
"properties": {},
|
||||
"widgets_values": [
|
||||
"# Fish Audio S2 Inline Instructions\n\nFish supports free-form inline instructions for emotion, prosody, accents, pauses, and speaking style.\n\nThe suite keeps its public instruction syntax as `<...>` and automatically converts it to Fish's native `[...]` syntax at the engine boundary.\n\n- Write Fish instructions with `<...>`.\n- Use `[Character]` for suite character switching.\n- Use `[fr:Alice]` to select Alice and request French.\n- `Auto Inline Tag` converts mapped character languages automatically.\n- Disable `Auto Inline Tag` when Fish should infer the language from the text.\n\nDo not manually replace suite instructions with Fish's square-bracket syntax.\n\nFor the complete suite syntax, see the [Character Switching Guide](https://github.com/diodiogod/TTS-Audio-Suite/blob/main/docs/CHARACTER_SWITCHING_GUIDE.md) and [Tag Editor Guide](https://github.com/diodiogod/TTS-Audio-Suite/blob/main/docs/MULTILINE_TTS_TAG_EDITOR_GUIDE.md). For Fish's prompting behavior, see the [Fish S2 model card](https://huggingface.co/fishaudio/s2-pro)."
|
||||
],
|
||||
"color": "#432",
|
||||
"bgcolor": "#653"
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"type": "MarkdownNote",
|
||||
"pos": [
|
||||
-2160,
|
||||
950
|
||||
],
|
||||
"size": [
|
||||
350,
|
||||
280
|
||||
],
|
||||
"flags": {},
|
||||
"order": 9,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [],
|
||||
"properties": {},
|
||||
"widgets_values": [
|
||||
"# Fish Audio S2 Compilation\n\nCompilation is recommended for repeated Fish generation.\n\nThe first generation after loading the model takes longer because PyTorch compiles the inference path. Later generations reuse the compiled runtime and are significantly faster.\n\nIn our testing:\n\n- Non-compiled generation: approximately `6 it/s`\n- Compiled generation: approximately `40 it/s`\n- Observed speedup: roughly **6-7x faster**\n\nThe initial compilation delay is worth accepting when generating multiple segments, testing different seeds, or working with longer text.\n\n> Keep `Compile` enabled unless startup time is more important than generation speed."
|
||||
],
|
||||
"color": "#432",
|
||||
"bgcolor": "#653"
|
||||
},
|
||||
{
|
||||
"id": 16,
|
||||
"type": "MarkdownNote",
|
||||
"pos": [
|
||||
-3550,
|
||||
190
|
||||
],
|
||||
"size": [
|
||||
750,
|
||||
970
|
||||
],
|
||||
"flags": {},
|
||||
"order": 0,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [],
|
||||
"properties": {},
|
||||
"widgets_values": [
|
||||
"# Fish Audio S2 Pro\n\n> Multilingual voice cloning with native multi-speaker dialogue, free-form inline control, and long-form generation.\n\nConnect the Fish Audio S2 Engine to the unified `TTS Text` or `TTS SRT` node.\n\n---\n\n## Voice Cloning\n\nFish requires both:\n\n- Reference audio\n- Matching reference text\n\nFor reliable cloning, use:\n\n- `🎭 Character Voices` with a matching `.reference.txt`\n- A narrator voice file with its matching `.reference.txt`\n\nDirect audio-only input does not provide the transcript required by Fish.\n\n---\n\n## Languages and Inline Control\n\nFish S2 supports **80+ languages** and has no separate language input.\n\nThe suite uses its normal `<...>` inline syntax and converts instructions automatically to Fish's native `[...]` syntax at the engine boundary.\n\n- Use `<...>` for Fish speech instructions.\n- Use `[Character]` for suite character switching.\n- Use `[fr:Alice]` to select Alice and request French.\n- `Auto Inline Tag` converts mapped character languages automatically.\n- Disable `Auto Inline Tag` when Fish should determine the language from the text.\n\nFish supports free-form instructions for expression, prosody, accents, pauses, and other speech behavior.\n\n---\n\n## Multi-Speaker Modes\n\n### Native Multi-Speaker\n\nSends all dialogue turns in one Fish request.\n\n- Might sound more natural\n\n### Custom Character Switching\n\nGenerates each parsed character segment independently.\n\n- Could be way more stable since it's independent character calls\n\n---\n\n## Long-Form Generation\n\nFish has its own native long-form orchestration through `native_chunk_length`.\n\nThis is Fish's internal grouping system and is separate from Custom Character Switching.\n\nFish does not provide native duration targeting.\nSRT timing is handled by the suite's shared timing and assembly system.\n\n---\n\n## Limitations\n\n- Fish does not support `🔄 Voice Changer`.\n- Voice cloning requires matching reference text.\n- The model does not provide native duration targeting.\n\n---\n\n## Documentation\n\n- [Character Switching Guide](https://github.com/diodiogod/TTS-Audio-Suite/blob/main/docs/CHARACTER_SWITCHING_GUIDE.md)\n- [Tag Editor Guide](https://github.com/diodiogod/TTS-Audio-Suite/blob/main/docs/MULTILINE_TTS_TAG_EDITOR_GUIDE.md)\n- [Fish S2 Model Card](https://huggingface.co/fishaudio/s2-pro)\n- [Official Fish Speech Repository](https://github.com/fishaudio/fish-speech)"
|
||||
],
|
||||
"color": "#432",
|
||||
"bgcolor": "#653"
|
||||
},
|
||||
{
|
||||
"id": 19,
|
||||
"type": "Reroute",
|
||||
"pos": [
|
||||
-1590,
|
||||
1480
|
||||
],
|
||||
"size": [
|
||||
75,
|
||||
26
|
||||
],
|
||||
"flags": {},
|
||||
"order": 13,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "",
|
||||
"type": "*",
|
||||
"link": 123
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "",
|
||||
"type": "TTS_ENGINE",
|
||||
"links": [
|
||||
116
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"showOutputText": false,
|
||||
"horizontal": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 22,
|
||||
"type": "FishAudioS2EngineNode",
|
||||
"pos": [
|
||||
-2160,
|
||||
1480
|
||||
],
|
||||
"size": [
|
||||
410,
|
||||
474
|
||||
],
|
||||
"flags": {},
|
||||
"order": 8,
|
||||
"mode": 0,
|
||||
"inputs": [
|
||||
{
|
||||
"name": "speaker2",
|
||||
"type": "*",
|
||||
"link": null
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "TTS_engine",
|
||||
"type": "TTS_ENGINE",
|
||||
"links": [
|
||||
123
|
||||
]
|
||||
}
|
||||
],
|
||||
"properties": {
|
||||
"cnr_id": "chatterbox_srt_voice",
|
||||
"ver": "26fa2ddf92b4e9d16fbcd585858401b509fa9c24",
|
||||
"Node name for S&R": "FishAudioS2EngineNode"
|
||||
},
|
||||
"widgets_values": [
|
||||
"s2-pro",
|
||||
"auto",
|
||||
0.8,
|
||||
0.8,
|
||||
1.1,
|
||||
200,
|
||||
1024,
|
||||
"8192",
|
||||
true,
|
||||
true,
|
||||
"bfloat16",
|
||||
true,
|
||||
"none",
|
||||
"Native Multi-Speaker",
|
||||
"Auto Inline Tag"
|
||||
],
|
||||
"color": "#222",
|
||||
"bgcolor": "#000"
|
||||
},
|
||||
{
|
||||
"id": 23,
|
||||
"type": "MarkdownNote",
|
||||
"pos": [
|
||||
-2660,
|
||||
1320
|
||||
],
|
||||
"size": [
|
||||
360,
|
||||
310
|
||||
],
|
||||
"flags": {},
|
||||
"order": 11,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [],
|
||||
"properties": {},
|
||||
"widgets_values": [
|
||||
"\n\n## 🔌 The Speaker Input Priority System\n\nWhen using **NATIVE Multi-Speaker**, you can connect specific voices to Speaker 1 (opt_narrator on the unified TTS node) , Speaker2, Speaker3... inputs on the Engine node and they will override tags!\n\n**What happens if using tags:**\n\n- `narrator` becomes Speaker 1 → uses Speaker 1 input voice\n- `[Alice]` becomes Speaker 2 → uses Speaker 2 input voice\n- `[Bob]` becomes Speaker 3 → uses Speaker 3 input voice\n- etc.\n\nSO If using \n\n## **[character]** \n\n tags in your text to switch narrators, you DON'T need to connect opt_narrator, Speaker2_voice, Speaker3_voice or speaker4_voice\n\nEither for Native or for Custom modes"
|
||||
],
|
||||
"color": "#432",
|
||||
"bgcolor": "#653"
|
||||
},
|
||||
{
|
||||
"id": 24,
|
||||
"type": "MarkdownNote",
|
||||
"pos": [
|
||||
-2520,
|
||||
1780
|
||||
],
|
||||
"size": [
|
||||
310,
|
||||
90
|
||||
],
|
||||
"flags": {},
|
||||
"order": 10,
|
||||
"mode": 0,
|
||||
"inputs": [],
|
||||
"outputs": [],
|
||||
"properties": {},
|
||||
"widgets_values": [
|
||||
"## Without speakers connected, it will use the Character Tag switching system normally"
|
||||
],
|
||||
"color": "#432",
|
||||
"bgcolor": "#653"
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
[
|
||||
101,
|
||||
2,
|
||||
0,
|
||||
6,
|
||||
0,
|
||||
"*"
|
||||
],
|
||||
[
|
||||
102,
|
||||
3,
|
||||
0,
|
||||
6,
|
||||
1,
|
||||
"*"
|
||||
],
|
||||
[
|
||||
105,
|
||||
4,
|
||||
0,
|
||||
7,
|
||||
2,
|
||||
"STRING"
|
||||
],
|
||||
[
|
||||
106,
|
||||
5,
|
||||
0,
|
||||
8,
|
||||
2,
|
||||
"STRING"
|
||||
],
|
||||
[
|
||||
109,
|
||||
7,
|
||||
0,
|
||||
9,
|
||||
0,
|
||||
"AUDIO"
|
||||
],
|
||||
[
|
||||
110,
|
||||
8,
|
||||
0,
|
||||
10,
|
||||
0,
|
||||
"AUDIO"
|
||||
],
|
||||
[
|
||||
111,
|
||||
12,
|
||||
0,
|
||||
6,
|
||||
2,
|
||||
"*"
|
||||
],
|
||||
[
|
||||
112,
|
||||
13,
|
||||
0,
|
||||
6,
|
||||
3,
|
||||
"*"
|
||||
],
|
||||
[
|
||||
116,
|
||||
19,
|
||||
0,
|
||||
8,
|
||||
0,
|
||||
"TTS_ENGINE"
|
||||
],
|
||||
[
|
||||
122,
|
||||
6,
|
||||
0,
|
||||
7,
|
||||
0,
|
||||
"TTS_ENGINE"
|
||||
],
|
||||
[
|
||||
123,
|
||||
22,
|
||||
0,
|
||||
19,
|
||||
0,
|
||||
"TTS_ENGINE"
|
||||
]
|
||||
],
|
||||
"groups": [
|
||||
{
|
||||
"id": 1,
|
||||
"title": "Fish Audio S2 Engine and Voices (they will OVERRIDE the characters on the text)",
|
||||
"bounding": [
|
||||
-2750,
|
||||
170,
|
||||
1090,
|
||||
2050
|
||||
],
|
||||
"color": "#3f789e",
|
||||
"font_size": 24,
|
||||
"flags": {}
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"title": "TTS TEXT",
|
||||
"bounding": [
|
||||
-1620,
|
||||
160,
|
||||
2260,
|
||||
1230
|
||||
],
|
||||
"color": "#8A8",
|
||||
"font_size": 24,
|
||||
"flags": {}
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"title": "SRT",
|
||||
"bounding": [
|
||||
-1610,
|
||||
1430,
|
||||
2250,
|
||||
780
|
||||
],
|
||||
"color": "#a1309b",
|
||||
"font_size": 24,
|
||||
"flags": {}
|
||||
}
|
||||
],
|
||||
"config": {},
|
||||
"extra": {
|
||||
"frontendVersion": "1.38.13",
|
||||
"ue_links": [],
|
||||
"links_added_by_ue": [],
|
||||
"VHS_latentpreview": true,
|
||||
"VHS_latentpreviewrate": 0,
|
||||
"VHS_MetadataImage": true,
|
||||
"VHS_KeepIntermediate": true,
|
||||
"workflowRendererVersion": "LG",
|
||||
"ds": {
|
||||
"scale": 0.33591950066247567,
|
||||
"offset": [
|
||||
4389.719106665517,
|
||||
640.6871718697881
|
||||
]
|
||||
}
|
||||
},
|
||||
"version": 0.4
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
File diff suppressed because it is too large
Load Diff
+42
-4
@@ -664,8 +664,10 @@ class TTSAudioInstaller:
|
||||
|
||||
# Install/reinstall audio packages that commonly have architecture issues on Mac
|
||||
mac_audio_packages = [
|
||||
"soundfile>=0.12.0",
|
||||
"sounddevice>=0.4.0",
|
||||
"soundfile>=0.12.0",
|
||||
"sounddevice>=0.4.0",
|
||||
"pydub",
|
||||
"webdataset",
|
||||
]
|
||||
|
||||
for package in mac_audio_packages:
|
||||
@@ -1038,8 +1040,9 @@ class TTSAudioInstaller:
|
||||
"librosa", # Forces numpy downgrade - compatibility handled by runtime numba disabling for Python 3.13
|
||||
"descript-audio-codec", # Pulls unnecessary deps, conflicts with protobuf
|
||||
"descript-audiotools", # Forces protobuf downgrade from 6.x to 3.19.x
|
||||
"cached-path", # Forces package downgrades
|
||||
"torchcrepe", # Conflicts via librosa dependency
|
||||
"cached-path", # Forces package downgrades
|
||||
"omnivoice", # Keep shared dependency resolution under our control
|
||||
"torchcrepe", # Conflicts via librosa dependency
|
||||
# NOTE: onnxruntime moved to install_onnxruntime_with_gpu_support() for smart GPU detection
|
||||
"opencv-python", # Forces numpy downgrade from 2.x to 1.26.x - dependencies pre-installed
|
||||
"gradio", # Forces pydantic, pillow, pydantic-core downgrades - dependencies pre-installed
|
||||
@@ -1158,6 +1161,40 @@ class TTSAudioInstaller:
|
||||
ignore_errors=True
|
||||
)
|
||||
|
||||
def install_fish_audio_s2(self):
|
||||
"""Install Fish S2 without changing the suite's core ML stack."""
|
||||
self.log("Installing Fish Audio S2 support in the main T5 environment", "INFO")
|
||||
|
||||
packages = [
|
||||
("lightning", "lightning"),
|
||||
("pytorch_lightning", "pytorch-lightning"),
|
||||
("lightning_utilities", "lightning-utilities"),
|
||||
("torchmetrics", "torchmetrics"),
|
||||
("opencc", "opencc-python-reimplemented==0.1.7"),
|
||||
("ormsgpack", "ormsgpack"),
|
||||
("zstandard", "zstandard"),
|
||||
("pyrootutils", "pyrootutils"),
|
||||
("loralib", "loralib"),
|
||||
]
|
||||
for module_name, package_name in packages:
|
||||
if self.verify_python_import(module_name):
|
||||
self.log(f"{module_name} import already satisfied - skipping", "SUCCESS")
|
||||
continue
|
||||
self.run_pip_command(
|
||||
["install", package_name, "--no-deps"],
|
||||
f"Installing Fish S2 dependency {package_name} (--no-deps)",
|
||||
ignore_errors=True,
|
||||
)
|
||||
|
||||
if self.check_package_installed("fish-speech"):
|
||||
self.log("fish-speech already satisfied - skipping", "SUCCESS")
|
||||
return
|
||||
self.run_pip_command(
|
||||
["install", "git+https://github.com/fishaudio/fish-speech.git", "--no-deps"],
|
||||
"Installing official fish-speech from GitHub (--no-deps)",
|
||||
ignore_errors=True,
|
||||
)
|
||||
|
||||
def install_f5tts_multilingual_support(self):
|
||||
"""Install phonemization support for F5-TTS multilingual models (Polish, German, French, Spanish, etc.)"""
|
||||
self.log("Installing F5-TTS multilingual phonemization support", "INFO")
|
||||
@@ -1595,6 +1632,7 @@ def main():
|
||||
installer.install_vibevoice() # Install VibeVoice with careful dependency management
|
||||
installer.install_echo_tts() # Install Echo-TTS with minimal dependency impact
|
||||
installer.install_dots_tts() # Install official Dots TTS in the main environment first
|
||||
installer.install_fish_audio_s2() # Install Fish S2 without changing Torch/Transformers
|
||||
installer.install_f5tts_multilingual_support() # Install phonemization for Polish/multilingual F5-TTS
|
||||
installer.install_indexts_text_processing() # Install IndexTTS-2 text normalization with fallback
|
||||
installer.install_russian_text_stresser_support() # Install lightweight Russian stress package for Official 23-Lang
|
||||
|
||||
@@ -11,7 +11,7 @@ except ImportError:
|
||||
pass
|
||||
|
||||
# Version and constants
|
||||
VERSION = "5.1.1"
|
||||
VERSION = "5.4.4"
|
||||
IS_DEV = False # Set to False for release builds
|
||||
VERSION_DISPLAY = f"v{VERSION}" + (" (dev)" if IS_DEV else "")
|
||||
SEPARATOR = "=" * 70
|
||||
@@ -169,6 +169,30 @@ except Exception as e:
|
||||
print(f"❌ Dots TTS Engine failed: {e}")
|
||||
DOTS_TTS_ENGINE_AVAILABLE = False
|
||||
|
||||
try:
|
||||
fish_audio_s2_engine_module = load_node_module("fish_audio_s2_engine_node", "engines/fish_audio_s2_engine_node.py")
|
||||
FishAudioS2EngineNode = fish_audio_s2_engine_module.FishAudioS2EngineNode
|
||||
FISH_AUDIO_S2_ENGINE_AVAILABLE = True
|
||||
except Exception as e:
|
||||
print(f"❌ Fish Audio S2 Pro Engine failed: {e}")
|
||||
FISH_AUDIO_S2_ENGINE_AVAILABLE = False
|
||||
|
||||
try:
|
||||
omnivoice_engine_module = load_node_module("omnivoice_engine_node", "engines/omnivoice_engine_node.py")
|
||||
OmniVoiceEngineNode = omnivoice_engine_module.OmniVoiceEngineNode
|
||||
OMNIVOICE_ENGINE_AVAILABLE = True
|
||||
except Exception as e:
|
||||
print(f"❌ OmniVoice Engine failed: {e}")
|
||||
OMNIVOICE_ENGINE_AVAILABLE = False
|
||||
|
||||
try:
|
||||
omnivoice_instruction_builder_module = load_node_module("omnivoice_instruction_builder_node", "omnivoice/omnivoice_instruction_builder_node.py")
|
||||
OmniVoiceInstructionBuilderNode = omnivoice_instruction_builder_module.OmniVoiceInstructionBuilderNode
|
||||
OMNIVOICE_INSTRUCTION_BUILDER_AVAILABLE = True
|
||||
except Exception as e:
|
||||
print(f"❌ Visual Tag Builder failed: {e}")
|
||||
OMNIVOICE_INSTRUCTION_BUILDER_AVAILABLE = False
|
||||
|
||||
try:
|
||||
chatterbox_official_23lang_engine_module = load_node_module("chatterbox_official_23lang_engine_node", "engines/chatterbox_official_23lang_engine_node.py")
|
||||
ChatterBoxOfficial23LangEngineNode = chatterbox_official_23lang_engine_module.ChatterBoxOfficial23LangEngineNode
|
||||
@@ -609,6 +633,18 @@ if DOTS_TTS_ENGINE_AVAILABLE:
|
||||
NODE_CLASS_MAPPINGS["DotsTTSEngineNode"] = DotsTTSEngineNode
|
||||
NODE_DISPLAY_NAME_MAPPINGS["DotsTTSEngineNode"] = "⚙️ Dots TTS Engine"
|
||||
|
||||
if FISH_AUDIO_S2_ENGINE_AVAILABLE:
|
||||
NODE_CLASS_MAPPINGS["FishAudioS2EngineNode"] = FishAudioS2EngineNode
|
||||
NODE_DISPLAY_NAME_MAPPINGS["FishAudioS2EngineNode"] = "⚙️ Fish Audio S2 Pro Engine"
|
||||
|
||||
if OMNIVOICE_ENGINE_AVAILABLE:
|
||||
NODE_CLASS_MAPPINGS["OmniVoiceEngineNode"] = OmniVoiceEngineNode
|
||||
NODE_DISPLAY_NAME_MAPPINGS["OmniVoiceEngineNode"] = "⚙️ OmniVoice Engine"
|
||||
|
||||
if OMNIVOICE_INSTRUCTION_BUILDER_AVAILABLE:
|
||||
NODE_CLASS_MAPPINGS["OmniVoiceInstructionBuilderNode"] = OmniVoiceInstructionBuilderNode
|
||||
NODE_DISPLAY_NAME_MAPPINGS["OmniVoiceInstructionBuilderNode"] = "📐 Visual Tag Builder"
|
||||
|
||||
if QWEN3_TTS_VOICE_DESIGNER_AVAILABLE:
|
||||
NODE_CLASS_MAPPINGS["Qwen3TTSVoiceDesignerNode"] = Qwen3TTSVoiceDesignerNode
|
||||
NODE_DISPLAY_NAME_MAPPINGS["Qwen3TTSVoiceDesignerNode"] = "🎨 Qwen3-TTS Voice Designer"
|
||||
|
||||
@@ -348,7 +348,7 @@ Selects the audio format for separated stems:
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
from tqdm import tqdm
|
||||
import librosa
|
||||
from utils.audio.librosa_fallback import safe_resample
|
||||
from comfy.utils import load_torch_file, ProgressBar
|
||||
from comfy import model_management as mm
|
||||
|
||||
@@ -398,7 +398,11 @@ Selects the audio format for separated stems:
|
||||
# Resample to 44100 if needed
|
||||
if sample_rate != 44100:
|
||||
print(f"Resampling from {sample_rate} to 44100 Hz")
|
||||
audio_waveform = librosa.resample(audio_waveform, orig_sr=sample_rate, target_sr=44100, axis=-1)
|
||||
audio_waveform = safe_resample(
|
||||
audio_waveform,
|
||||
orig_sr=sample_rate,
|
||||
target_sr=44100,
|
||||
)
|
||||
sample_rate = 44100
|
||||
|
||||
# Convert to torch tensor
|
||||
|
||||
@@ -39,6 +39,7 @@ BaseTTSNode = base_module.BaseTTSNode
|
||||
from utils.text.chunking import ImprovedChatterBoxChunker
|
||||
from utils.audio.processing import AudioProcessingUtils
|
||||
from utils.voice.discovery import get_available_characters, get_character_mapping
|
||||
from utils.voice.character_logging import resolved_character_label
|
||||
from utils.text.pause_processor import PauseTagProcessor
|
||||
from utils.text.character_parser import parse_character_text, character_parser
|
||||
from utils.text.segment_parameters import apply_segment_parameters
|
||||
@@ -1174,7 +1175,8 @@ Back to the main narrator voice for the conclusion.""",
|
||||
# Sequential processing for all chunks
|
||||
segment_audio_chunks = []
|
||||
for chunk_i, chunk_text in enumerate(segment_chunks):
|
||||
print(f"🎤 Generating ChatterBox segment {segment_display_idx}/{total_segments} chunk {chunk_i+1}/{len(segment_chunks)} for '{character}' (lang: {lang_code})...")
|
||||
display_name = resolved_character_label(character, char_audio_prompt)
|
||||
print(f"🎤 Generating ChatterBox segment {segment_display_idx}/{total_segments} chunk {chunk_i+1}/{len(segment_chunks)} for '{display_name}' (lang: {lang_code})...")
|
||||
|
||||
# Generate audio with caching support for character segments
|
||||
chunk_audio = self._generate_tts_with_pause_tags(
|
||||
@@ -1379,9 +1381,11 @@ Back to the main narrator voice for the conclusion.""",
|
||||
audio_segments_with_order.append((original_idx, segment_audio))
|
||||
# Use the original better format with proper emoji based on character type
|
||||
if char == "narrator":
|
||||
print(f"🎤 Generating ChatterBox segment {original_idx+1}/{len(segments_with_lang)} for '{char}' (lang: {lang})")
|
||||
display_name = resolved_character_label(char, char_audio_prompt)
|
||||
print(f"🎤 Generating ChatterBox segment {original_idx+1}/{len(segments_with_lang)} for '{display_name}' (lang: {lang})")
|
||||
else:
|
||||
print(f"🎭 Generating ChatterBox segment {original_idx+1}/{len(segments_with_lang)} for '{char}' (lang: {lang})")
|
||||
display_name = resolved_character_label(char, char_audio_prompt)
|
||||
print(f"🎭 Generating ChatterBox segment {original_idx+1}/{len(segments_with_lang)} for '{display_name}' (lang: {lang})")
|
||||
|
||||
return audio_segments_with_order
|
||||
|
||||
@@ -1484,4 +1488,3 @@ Back to the main narrator voice for the conclusion.""",
|
||||
)
|
||||
|
||||
print(f"🚀 Pre-loading complete: {len(self._streaming_model_manager.preloaded_models)} models ready")
|
||||
|
||||
|
||||
@@ -40,6 +40,7 @@ BaseTTSNode = base_module.BaseTTSNode
|
||||
from utils.text.chunking import ImprovedChatterBoxChunker
|
||||
from utils.audio.processing import AudioProcessingUtils
|
||||
from utils.voice.discovery import get_available_characters, get_character_mapping
|
||||
from utils.voice.character_logging import resolved_character_label
|
||||
from utils.text.pause_processor import PauseTagProcessor
|
||||
from utils.text.character_parser import parse_character_text, character_parser
|
||||
from utils.text.segment_parameters import apply_segment_parameters
|
||||
@@ -1562,7 +1563,8 @@ Back to the main narrator voice for the conclusion.""",
|
||||
# Sequential processing for all chunks
|
||||
segment_audio_chunks = []
|
||||
for chunk_i, chunk_text in enumerate(segment_chunks):
|
||||
print(f"🎤 Generating ChatterBox segment {segment_display_idx}/{total_segments} chunk {chunk_i+1}/{len(segment_chunks)} for '{character}' (lang: {lang_code})...")
|
||||
display_name = resolved_character_label(character, char_audio_prompt)
|
||||
print(f"🎤 Generating ChatterBox segment {segment_display_idx}/{total_segments} chunk {chunk_i+1}/{len(segment_chunks)} for '{display_name}' (lang: {lang_code})...")
|
||||
|
||||
# Generate audio with caching support for character segments
|
||||
chunk_audio = self._generate_tts_with_pause_tags(
|
||||
@@ -1823,10 +1825,11 @@ Back to the main narrator voice for the conclusion.""",
|
||||
|
||||
audio_segments_with_order.append((idx, segment_audio))
|
||||
# Use the original better format with proper emoji based on character type
|
||||
display_name = resolved_character_label(char, char_audio_prompt)
|
||||
if char == "narrator":
|
||||
print(f"🎤 Generating ChatterBox segment {idx+1}/{len(expanded_segments_with_lang)} for '{char}' (lang: {lang})")
|
||||
print(f"🎤 Generating ChatterBox segment {idx+1}/{len(expanded_segments_with_lang)} for '{display_name}' (lang: {lang})")
|
||||
else:
|
||||
print(f"🎭 Generating ChatterBox segment {idx+1}/{len(expanded_segments_with_lang)} for '{char}' (lang: {lang})")
|
||||
print(f"🎭 Generating ChatterBox segment {idx+1}/{len(expanded_segments_with_lang)} for '{display_name}' (lang: {lang})")
|
||||
|
||||
# Batch process edit tags if any
|
||||
if all_segments_for_editing:
|
||||
@@ -1960,4 +1963,3 @@ Back to the main narrator voice for the conclusion.""",
|
||||
)
|
||||
|
||||
print(f"🚀 Pre-loading complete: {len(self._streaming_model_manager.preloaded_models)} models ready")
|
||||
|
||||
|
||||
@@ -23,7 +23,8 @@ if project_root not in sys.path:
|
||||
from utils.text.segment_parameters import apply_segment_parameters
|
||||
from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment, parse_edit_tags_with_iterations
|
||||
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
|
||||
from utils.text.chatterbox_v2_special_tags import CHATTERBOX_V2_SPECIAL_TOKENS
|
||||
from utils.text.chatterbox_v2_special_tags import CHATTERBOX_V2_SPECIAL_TOKENS
|
||||
from utils.voice.character_logging import resolved_character_label
|
||||
|
||||
|
||||
def extract_edit_tags_for_chatterbox(text: str):
|
||||
@@ -412,7 +413,11 @@ class ChatterboxOfficial23LangSRTProcessor:
|
||||
print(f"🌍 ChatterBox Official 23-Lang SRT Segment {i+1} (Seq {subtitle.sequence}): Language switching - {', '.join(languages)}")
|
||||
elif subtitle_type != 'multisegment':
|
||||
characters = list(set(char for char, _, _, _ in character_segments_with_lang))
|
||||
print(f"🎭 ChatterBox Official 23-Lang SRT Segment {i+1} (Seq {subtitle.sequence}): Character switching - {', '.join(characters)}")
|
||||
resolved_characters = [
|
||||
resolved_character_label(char, voice_refs.get(char, voice_refs.get('narrator')))
|
||||
for char in characters
|
||||
]
|
||||
print(f"🎭 ChatterBox Official 23-Lang SRT Segment {i+1} (Seq {subtitle.sequence}): Character switching - {', '.join(resolved_characters)}")
|
||||
|
||||
segment_audio_parts = []
|
||||
subtitle_has_edit_tags = False # Track if ANY segment has edit tags
|
||||
|
||||
@@ -22,6 +22,10 @@ from utils.audio.edit_post_processor import process_segments as apply_edit_post_
|
||||
from engines.dots_tts.languages import format_dots_language_display, normalize_dots_language
|
||||
from utils.text.character_parser import character_parser
|
||||
from utils.text.pause_processor import PauseTagProcessor
|
||||
from utils.voice.character_logging import (
|
||||
format_resolved_character_block,
|
||||
resolved_character_label,
|
||||
)
|
||||
from utils.text.segment_parameters import ParameterValidator, apply_segment_parameters
|
||||
from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment
|
||||
from utils.voice.discovery import get_available_characters, get_character_mapping, voice_discovery
|
||||
@@ -63,16 +67,15 @@ class DotsTTSProcessor:
|
||||
show_text_content: bool = True,
|
||||
) -> None:
|
||||
voice_note = self._voice_log_note(voice_ref)
|
||||
print(f"🎭 Dots TTS - Generating for '{character_name}' (Language: {language}){voice_note}:")
|
||||
display_name = resolved_character_label(character_name, voice_ref)
|
||||
print(f"🎭 Dots TTS - Generating for '{display_name}' (Language: {language}){voice_note}:")
|
||||
if parameter_log:
|
||||
print(f"🎛️ Dots TTS params: {parameter_log}")
|
||||
if show_text_content:
|
||||
print("=" * 60)
|
||||
print(text_content)
|
||||
print("=" * 60)
|
||||
print(format_resolved_character_block(character_name, text_content, voice_ref))
|
||||
if chunk_count > 1:
|
||||
print(
|
||||
f"📝 Chunking {character_name}'s text into {chunk_count} chunks "
|
||||
f"📝 Chunking {display_name}'s text into {chunk_count} chunks "
|
||||
f"(Language: {language}){voice_note}"
|
||||
)
|
||||
|
||||
|
||||
@@ -24,6 +24,10 @@ from utils.audio.edit_post_processor import process_segments as apply_edit_post_
|
||||
from utils.voice.discovery import get_available_characters, voice_discovery, get_character_mapping
|
||||
from utils.text.character_parser import character_parser
|
||||
from utils.audio.chunk_timing import ChunkTimingHelper
|
||||
from utils.voice.character_logging import (
|
||||
format_resolved_character_block,
|
||||
resolved_character_label,
|
||||
)
|
||||
|
||||
|
||||
class EchoTTSProcessor:
|
||||
@@ -233,10 +237,9 @@ class EchoTTSProcessor:
|
||||
print(f"📝 Echo-TTS: Chunking '{seg.character}' into {len(chunks)} chunks (max {max_chars_per_chunk} chars each)")
|
||||
|
||||
for chunk_idx, chunk in enumerate(chunks):
|
||||
print(f"🎤 Echo-TTS - Generating for '{seg.character}'" + (f" (chunk {chunk_idx + 1}/{len(chunks)})" if len(chunks) > 1 else "") + ":")
|
||||
print("=" * 60)
|
||||
print(chunk)
|
||||
print("=" * 60)
|
||||
display_name = resolved_character_label(char_name, speaker_audio)
|
||||
print(f"🎤 Echo-TTS - Generating for '{display_name}'" + (f" (chunk {chunk_idx + 1}/{len(chunks)})" if len(chunks) > 1 else "") + ":")
|
||||
print(format_resolved_character_block(char_name, chunk, speaker_audio))
|
||||
audio = _generate_single(chunk).cpu()
|
||||
if audio.dim() == 1:
|
||||
audio = audio.unsqueeze(0)
|
||||
|
||||
@@ -0,0 +1,214 @@
|
||||
"""Fish Audio S2 Pro engine configuration node."""
|
||||
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
current_dir = os.path.dirname(__file__)
|
||||
nodes_dir = os.path.dirname(current_dir)
|
||||
project_root = os.path.dirname(nodes_dir)
|
||||
if project_root not in sys.path:
|
||||
sys.path.insert(0, project_root)
|
||||
|
||||
base_spec = importlib.util.spec_from_file_location("base_node_module", os.path.join(nodes_dir, "base", "base_node.py"))
|
||||
base_module = importlib.util.module_from_spec(base_spec)
|
||||
sys.modules.setdefault("base_node_module", base_module)
|
||||
base_spec.loader.exec_module(base_module)
|
||||
|
||||
SPEAKER_INPUT_PATTERN = re.compile(r"^speaker([2-9]\d*)$")
|
||||
|
||||
|
||||
class DynamicSpeakerInputs(dict):
|
||||
@staticmethod
|
||||
def _is_speaker_key(key):
|
||||
return isinstance(key, str) and SPEAKER_INPUT_PATTERN.fullmatch(key) is not None
|
||||
|
||||
def __contains__(self, key):
|
||||
return super().__contains__(key) or self._is_speaker_key(key)
|
||||
|
||||
def __getitem__(self, key):
|
||||
if super().__contains__(key):
|
||||
return super().__getitem__(key)
|
||||
if self._is_speaker_key(key):
|
||||
return ("*", {
|
||||
"forceInput": True,
|
||||
"tooltip": "Positional Fish speaker reference. Connect a Character Voices output with exact reference transcript.",
|
||||
})
|
||||
raise KeyError(key)
|
||||
|
||||
def get(self, key, default=None):
|
||||
return self[key] if key in self else default
|
||||
|
||||
|
||||
class FishAudioS2EngineNode(base_module.BaseTTSNode):
|
||||
@classmethod
|
||||
def NAME(cls):
|
||||
return "⚙️ Fish Audio S2 Pro Engine"
|
||||
|
||||
@classmethod
|
||||
def _get_model_options(cls):
|
||||
"""List local checkpoints first, followed by downloadable variants."""
|
||||
downloadable = ["s2-pro", "s2-pro-fp8"]
|
||||
try:
|
||||
from engines.fish_audio_s2.downloader import FishAudioS2Downloader
|
||||
|
||||
local = [
|
||||
f"local:{os.path.basename(os.path.normpath(path))}"
|
||||
for path in FishAudioS2Downloader.local_model_paths()
|
||||
]
|
||||
except Exception:
|
||||
local = []
|
||||
return local + [option for option in downloadable if option not in local]
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
optional = DynamicSpeakerInputs({
|
||||
"normalize": ("BOOLEAN", {"default": True, "tooltip": "Official English/Chinese number normalization."}),
|
||||
"cache_reference": ("BOOLEAN", {"default": True, "tooltip": "Cache encoded reference codes inside the Fish runtime."}),
|
||||
"precision": (["bfloat16", "float16"], {"default": "bfloat16"}),
|
||||
"compile": ("BOOLEAN", {"default": True, "tooltip": "Enable the official torch.compile path (recommended). The first generation is slower, but later runs are much faster; testing showed about 6 it/s to 41 it/s, roughly a 6.8x speedup. This can also be tested with BNB quantization, although bitsandbytes compatibility may vary by version."}),
|
||||
"quantization": (["none", "bnb_int8", "bnb_nf4"], {
|
||||
"default": "none",
|
||||
"tooltip": (
|
||||
"Optional on-the-fly quantization for the official s2-pro checkpoint.\n\n"
|
||||
"• none: Full checkpoint loading\n"
|
||||
"• bnb_int8: BitsAndBytes INT8 load-time quantization\n"
|
||||
"• bnb_nf4: BitsAndBytes NF4 load-time quantization\n\n"
|
||||
"BNB quantization reuses the official checkpoint and requires CUDA plus bitsandbytes. "
|
||||
"It is ignored for the separate s2-pro-fp8 checkpoint."
|
||||
),
|
||||
}),
|
||||
"multi_speaker_mode": (["Native Multi-Speaker", "Custom Character Switching"], {
|
||||
"default": "Native Multi-Speaker",
|
||||
"tooltip": (
|
||||
"Native Multi-Speaker sends all character turns in one Fish dialogue request. "
|
||||
"This preserves native multi-turn context and long-form behavior.\n\n"
|
||||
"Custom Character Switching generates every parsed character segment independently "
|
||||
"as local speaker 0, using only that character's reference. This can reduce speaker "
|
||||
"leakage but loses cross-turn context and requires more generation calls."
|
||||
),
|
||||
}),
|
||||
"language_prompting": (["Auto Inline Tag", "Off"], {
|
||||
"default": "Auto Inline Tag",
|
||||
"tooltip": (
|
||||
"Fish has no native language parameter, so the suite can optionally prepend a natural-language "
|
||||
"inline instruction for resolved segment languages.\n\n"
|
||||
"• Auto Inline Tag: alias/default languages like de or fr become Fish prompt tags such as <German>\n"
|
||||
"• Off: keep Fish language fully text-only and ignore alias/default language switching"
|
||||
),
|
||||
}),
|
||||
"speaker2": ("*", {
|
||||
"forceInput": True,
|
||||
"tooltip": "Optional second speaker reference. Speaker 1 is always the Unified node narrator/opt_narrator input. Additional speaker inputs appear automatically.",
|
||||
}),
|
||||
})
|
||||
return {"required": {
|
||||
"model": (cls._get_model_options(), {
|
||||
"default": "s2-pro",
|
||||
"tooltip": "Fish Audio S2 model selection. Options prefixed with local: are already present in configured ComfyUI TTS model folders; unprefixed options are downloadable checkpoints.",
|
||||
}),
|
||||
"device": (["auto", "cuda", "cpu"], {"default": "auto"}),
|
||||
"temperature": ("FLOAT", {"default": 0.8, "min": 0.1, "max": 1.0, "step": 0.05}),
|
||||
"top_p": ("FLOAT", {"default": 0.8, "min": 0.1, "max": 1.0, "step": 0.05}),
|
||||
"repetition_penalty": ("FLOAT", {"default": 1.1, "min": 0.9, "max": 2.0, "step": 0.05}),
|
||||
"native_chunk_length": ("INT", {"default": 200, "min": 100, "max": 1000, "step": 25,
|
||||
"tooltip": "Official UTF-8 byte limit for grouping native speaker turns."}),
|
||||
"max_new_tokens": ("INT", {"default": 1024, "min": 64, "max": 4096, "step": 64}),
|
||||
"context_length": (["4096", "8192", "16384", "32768"], {
|
||||
"default": "8192",
|
||||
"tooltip": "Native context and KV-cache size. Larger values support longer dialogue but use substantially more VRAM.",
|
||||
}),
|
||||
}, "optional": optional}
|
||||
|
||||
RETURN_TYPES = ("TTS_ENGINE",)
|
||||
RETURN_NAMES = ("TTS_engine",)
|
||||
FUNCTION = "create_engine_config"
|
||||
CATEGORY = "TTS Audio Suite/⚙️ Engines"
|
||||
|
||||
@classmethod
|
||||
def IS_CHANGED(cls, model="s2-pro", device="auto", temperature=0.8, top_p=0.8, repetition_penalty=1.1,
|
||||
native_chunk_length=200, max_new_tokens=1024, context_length="8192",
|
||||
normalize=True, cache_reference=True, precision="bfloat16",
|
||||
compile=True, model_variant=None, quantization="none",
|
||||
multi_speaker_mode="Native Multi-Speaker", language_prompting="Auto Inline Tag",
|
||||
speaker2=None, **kwargs):
|
||||
model_selection = model_variant if model_variant is not None else model
|
||||
speaker_keys = sorted(
|
||||
key for key in kwargs.keys()
|
||||
if SPEAKER_INPUT_PATTERN.fullmatch(key) is not None
|
||||
)
|
||||
fingerprint = {
|
||||
"device": device,
|
||||
"temperature": float(temperature),
|
||||
"top_p": float(top_p),
|
||||
"repetition_penalty": float(repetition_penalty),
|
||||
"native_chunk_length": int(native_chunk_length),
|
||||
"max_new_tokens": int(max_new_tokens),
|
||||
"context_length": str(context_length),
|
||||
"normalize": bool(normalize),
|
||||
"cache_reference": bool(cache_reference),
|
||||
"precision": precision,
|
||||
"compile": bool(compile),
|
||||
"model": model_selection,
|
||||
"quantization": quantization,
|
||||
"multi_speaker_mode": multi_speaker_mode,
|
||||
"language_prompting": language_prompting,
|
||||
"speaker2_connected": speaker2 is not None,
|
||||
"dynamic_speakers_connected": {
|
||||
key: kwargs.get(key) is not None for key in speaker_keys
|
||||
},
|
||||
}
|
||||
return json.dumps(fingerprint, sort_keys=True)
|
||||
|
||||
def create_engine_config(self, model="s2-pro", device="auto", temperature=0.8, top_p=0.8, repetition_penalty=1.1,
|
||||
native_chunk_length=200, max_new_tokens=1024, context_length="8192",
|
||||
normalize=True, cache_reference=True, precision="bfloat16",
|
||||
compile=True, model_variant=None, quantization="none",
|
||||
multi_speaker_mode="Native Multi-Speaker", language_prompting="Auto Inline Tag",
|
||||
speaker2=None, **kwargs):
|
||||
model_selection = model_variant if model_variant is not None else model
|
||||
runtime_variant = model_selection
|
||||
if str(model_selection).startswith("local:"):
|
||||
try:
|
||||
from engines.fish_audio_s2.downloader import FishAudioS2Downloader
|
||||
local_path = FishAudioS2Downloader.resolve_local_model_path(model_selection)
|
||||
runtime_variant = FishAudioS2Downloader.resolve_model_variant(model_selection, local_path)
|
||||
except Exception:
|
||||
runtime_variant = "s2-pro"
|
||||
speakers = []
|
||||
if speaker2 is not None:
|
||||
speakers.append((2, speaker2))
|
||||
for key, value in kwargs.items():
|
||||
match = SPEAKER_INPUT_PATTERN.fullmatch(key)
|
||||
if match and value is not None:
|
||||
speakers.append((int(match.group(1)), value))
|
||||
speakers = [value for _, value in sorted(speakers, key=lambda item: item[0])]
|
||||
if runtime_variant != "s2-pro":
|
||||
precision = "bfloat16"
|
||||
quantization = "none"
|
||||
print(
|
||||
" Settings: "
|
||||
f"model={model_selection}, quantization={quantization}, "
|
||||
f"multi_speaker_mode={multi_speaker_mode}, language_prompting={language_prompting}, "
|
||||
f"temperature={float(temperature)}, "
|
||||
f"top_p={float(top_p)}, repetition_penalty={float(repetition_penalty)}, "
|
||||
f"native_chunk_length={int(native_chunk_length)}, max_new_tokens={int(max_new_tokens)}, "
|
||||
f"context_length={int(context_length)}, precision={precision}, "
|
||||
f"normalize={bool(normalize)}, cache_reference={bool(cache_reference)}, compile={bool(compile)}"
|
||||
)
|
||||
config = {
|
||||
"engine_type": "fish_audio_s2", "model_variant": model_selection, "device": device,
|
||||
"temperature": float(temperature), "top_p": float(top_p),
|
||||
"repetition_penalty": float(repetition_penalty),
|
||||
"native_chunk_length": int(native_chunk_length), "max_new_tokens": int(max_new_tokens),
|
||||
"context_length": int(context_length),
|
||||
"normalize": bool(normalize), "cache_reference": bool(cache_reference),
|
||||
"precision": precision, "compile": bool(compile),
|
||||
"quantization": quantization,
|
||||
"multi_speaker_mode": multi_speaker_mode,
|
||||
"language_prompting": language_prompting,
|
||||
"speaker_references": speakers, "runtime_mode": "main_subprocess",
|
||||
}
|
||||
return ({"engine_type": "fish_audio_s2", "config": config, "capabilities": ["tts"]},)
|
||||
@@ -145,14 +145,19 @@ class IndexTTSEngineNode(BaseTTSNode):
|
||||
}),
|
||||
},
|
||||
"optional": {
|
||||
# Unified Emotion Control - Using multitype input for better connection suggestions
|
||||
"emotion_control": (any_typ, {
|
||||
"tooltip": """• 🌈 Emotion Vectors - Manual emotion control sliders
|
||||
• 🎭 Character Voices (opt_narrator) - Audio-based emotion reference
|
||||
• 🌈 Text Emotion - AI-analyzed emotion from text
|
||||
• Direct AUDIO - Any audio input for emotion reference
|
||||
Character emotion tags [Alice:emotion_ref] will override this for specific characters."""
|
||||
}),
|
||||
# Unified Emotion Control - Using multitype input for better connection suggestions
|
||||
"emotion_control": (any_typ, {
|
||||
"tooltip": """Vector/text emotion control (legacy unified input):
|
||||
• 🌈 Emotion Vectors - Manual emotion control sliders
|
||||
• 🌈 Text Emotion - AI-analyzed emotion from text
|
||||
Connect audio separately to 'emotion_audio' when you want to blend both.
|
||||
Character emotion tags [Alice:emotion_ref] can provide per-segment audio emotion."""
|
||||
}),
|
||||
"emotion_audio": (any_typ, {
|
||||
"tooltip": """Dedicated emotion-reference audio input.
|
||||
Connect Character Voices, opt_narrator, or AUDIO here.
|
||||
This can be connected together with the vector/text emotion input above; IndexTTS-2 blends both."""
|
||||
}),
|
||||
|
||||
# CUDA Kernel Option
|
||||
"use_cuda_kernel": (["auto", "true", "false"], {
|
||||
@@ -241,9 +246,9 @@ Character emotion tags [Alice:emotion_ref] will override this for specific chara
|
||||
top_k: int,
|
||||
do_sample: bool,
|
||||
length_penalty: float,
|
||||
num_beams: int,
|
||||
repetition_penalty: float,
|
||||
max_mel_tokens: int,
|
||||
num_beams: int,
|
||||
repetition_penalty: float,
|
||||
max_mel_tokens: int,
|
||||
use_fp16: bool,
|
||||
use_deepspeed: bool,
|
||||
use_cuda_kernel: str = "auto",
|
||||
@@ -253,6 +258,7 @@ Character emotion tags [Alice:emotion_ref] will override this for specific chara
|
||||
stream_return: bool = False,
|
||||
more_segment_before: int = 0,
|
||||
low_vram: bool = False,
|
||||
emotion_audio = None,
|
||||
):
|
||||
"""
|
||||
Create IndexTTS-2 engine adapter with configuration.
|
||||
@@ -261,21 +267,38 @@ Character emotion tags [Alice:emotion_ref] will override this for specific chara
|
||||
Tuple containing IndexTTS-2 engine configuration data
|
||||
"""
|
||||
try:
|
||||
# Process unified emotion control
|
||||
emotion_audio = None
|
||||
emotion_vector = None
|
||||
use_emotion_text = False
|
||||
emotion_text = None
|
||||
is_dynamic_template = False
|
||||
|
||||
if emotion_control:
|
||||
if isinstance(emotion_control, dict):
|
||||
# Check the type of emotion control
|
||||
emotion_type = emotion_control.get("type")
|
||||
|
||||
if emotion_type == "emotion_vectors":
|
||||
# Emotion vectors from options node
|
||||
emotion_vectors = emotion_control.get("emotion_vectors", {})
|
||||
# Process vector/text control independently from dedicated audio.
|
||||
connected_emotion_audio = emotion_audio
|
||||
emotion_audio = None
|
||||
emotion_vector = None
|
||||
use_emotion_text = False
|
||||
emotion_text = None
|
||||
is_dynamic_template = False
|
||||
|
||||
def is_audio_control(value):
|
||||
return isinstance(value, dict) and any(
|
||||
key in value for key in ("audio_path", "waveform", "audio")
|
||||
)
|
||||
|
||||
# Keep old workflows working when audio was connected to the
|
||||
# original unified socket. The new dedicated socket wins when
|
||||
# both are connected.
|
||||
legacy_audio_control = emotion_control if is_audio_control(emotion_control) else None
|
||||
vector_text_control = None if legacy_audio_control is not None else emotion_control
|
||||
emotion_audio = (
|
||||
connected_emotion_audio
|
||||
if connected_emotion_audio is not None
|
||||
else legacy_audio_control
|
||||
)
|
||||
|
||||
if vector_text_control:
|
||||
if isinstance(vector_text_control, dict):
|
||||
# Check the type of emotion control
|
||||
emotion_type = vector_text_control.get("type")
|
||||
|
||||
if emotion_type == "emotion_vectors":
|
||||
# Emotion vectors from options node
|
||||
emotion_vectors = vector_text_control.get("emotion_vectors", {})
|
||||
emotions = [
|
||||
emotion_vectors.get("happy", 0.0),
|
||||
emotion_vectors.get("angry", 0.0),
|
||||
@@ -289,19 +312,11 @@ Character emotion tags [Alice:emotion_ref] will override this for specific chara
|
||||
if any(e > 0.0 for e in emotions):
|
||||
emotion_vector = emotions
|
||||
|
||||
elif emotion_type == "qwen_emotion":
|
||||
# QwenEmotion text analysis
|
||||
use_emotion_text = emotion_control.get("use_emotion_text", False)
|
||||
emotion_text = emotion_control.get("emotion_text", "")
|
||||
is_dynamic_template = emotion_control.get("is_dynamic_template", False)
|
||||
|
||||
elif "waveform" in emotion_control or "audio" in emotion_control:
|
||||
# Direct audio input (NARRATOR_VOICE from Character Voices or AUDIO)
|
||||
emotion_audio = emotion_control
|
||||
|
||||
elif hasattr(emotion_control, 'get') and ("waveform" in emotion_control or "audio" in emotion_control):
|
||||
# Direct audio input
|
||||
emotion_audio = emotion_control
|
||||
elif emotion_type == "qwen_emotion":
|
||||
# QwenEmotion text analysis
|
||||
use_emotion_text = vector_text_control.get("use_emotion_text", False)
|
||||
emotion_text = vector_text_control.get("emotion_text", "")
|
||||
is_dynamic_template = vector_text_control.get("is_dynamic_template", False)
|
||||
|
||||
# Parse CUDA kernel option
|
||||
cuda_kernel_option = None
|
||||
@@ -321,7 +336,7 @@ Character emotion tags [Alice:emotion_ref] will override this for specific chara
|
||||
config = {
|
||||
"model_path": model_path,
|
||||
"device": device,
|
||||
"emotion_audio": emotion_audio, # Will be None if not connected, audio dict if connected
|
||||
"emotion_audio": emotion_audio, # Dedicated audio emotion input
|
||||
"emotion_alpha": emotion_alpha,
|
||||
"use_emotion_text": use_emotion_text,
|
||||
"emotion_text": emotion_text if emotion_text and emotion_text.strip() else None,
|
||||
@@ -350,12 +365,20 @@ Character emotion tags [Alice:emotion_ref] will override this for specific chara
|
||||
"low_vram": low_vram,
|
||||
}
|
||||
|
||||
print(f"⚙️ IndexTTS-2: Configured on {device}")
|
||||
print(f" Model: {model_path}")
|
||||
emotion_desc = f"alpha={emotion_alpha}, use_text={use_emotion_text}"
|
||||
if is_dynamic_template:
|
||||
emotion_desc += " (dynamic template)"
|
||||
print(f" Emotion: {emotion_desc}")
|
||||
print(f"⚙️ IndexTTS-2: Configured on {device}")
|
||||
print(f" Model: {model_path}")
|
||||
emotion_desc = f"alpha={emotion_alpha}, use_text={use_emotion_text}"
|
||||
if is_dynamic_template:
|
||||
emotion_desc += " (dynamic template)"
|
||||
emotion_sources = []
|
||||
if emotion_audio is not None:
|
||||
emotion_sources.append("audio")
|
||||
if emotion_vector is not None:
|
||||
emotion_sources.append("vector")
|
||||
if use_emotion_text:
|
||||
emotion_sources.append("text")
|
||||
emotion_desc += f", sources={'+'.join(emotion_sources) if emotion_sources else 'none'}"
|
||||
print(f" Emotion: {emotion_desc}")
|
||||
print(f" Generation: temp={temperature}, top_p={top_p}, top_k={top_k}, do_sample={do_sample}, num_beams={num_beams}")
|
||||
print(f" Chunking: max_tokens={max_text_tokens_per_segment}, silence={interval_silence}ms")
|
||||
# Report optimization and streaming parameters
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
"""
|
||||
OmniVoice engine configuration node.
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
import os
|
||||
import sys
|
||||
|
||||
current_dir = os.path.dirname(__file__)
|
||||
nodes_dir = os.path.dirname(current_dir)
|
||||
project_root = os.path.dirname(nodes_dir)
|
||||
if project_root not in sys.path:
|
||||
sys.path.insert(0, project_root)
|
||||
|
||||
base_node_path = os.path.join(nodes_dir, "base", "base_node.py")
|
||||
base_spec = importlib.util.spec_from_file_location("base_node_module", base_node_path)
|
||||
base_module = importlib.util.module_from_spec(base_spec)
|
||||
sys.modules["base_node_module"] = base_module
|
||||
base_spec.loader.exec_module(base_module)
|
||||
BaseTTSNode = base_module.BaseTTSNode
|
||||
|
||||
|
||||
class OmniVoiceEngineNode(BaseTTSNode):
|
||||
"""OmniVoice engine configuration node."""
|
||||
|
||||
@classmethod
|
||||
def NAME(cls):
|
||||
return "⚙️ OmniVoice Engine"
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"model_variant": (["OmniVoice"], {
|
||||
"default": "OmniVoice",
|
||||
"tooltip": "OmniVoice multilingual zero-shot TTS model from k2-fsa/OmniVoice. Voice cloning requires reference audio plus reference text. In this suite, use Character Voices or a narrator voice file with matching .reference.txt. Direct audio-only input is not supported for OmniVoice cloning."
|
||||
}),
|
||||
"device": (["auto", "cuda", "cpu", "mps", "xpu"], {
|
||||
"default": "auto",
|
||||
"tooltip": "Device to run OmniVoice on. Auto follows the best available backend."
|
||||
}),
|
||||
"language": ("STRING", {
|
||||
"default": "Auto",
|
||||
"multiline": False,
|
||||
"tooltip": "Target language. Use 'Auto' for language-agnostic generation, or enter a language name/code such as 'English', 'Chinese', 'Japanese', 'en', 'zh', 'ja'. OmniVoice supports 600+ languages."
|
||||
}),
|
||||
"num_step": ("INT", {
|
||||
"default": 32, "min": 4, "max": 128, "step": 1,
|
||||
"tooltip": "Iterative unmasking steps. Recommended: 32. Use 16 for faster generation."
|
||||
}),
|
||||
"guidance_scale": ("FLOAT", {
|
||||
"default": 2.0, "min": 0.0, "max": 10.0, "step": 0.1,
|
||||
"tooltip": "Classifier-free guidance scale. Recommended: 2.0."
|
||||
}),
|
||||
"t_shift": ("FLOAT", {
|
||||
"default": 0.1, "min": 0.0, "max": 1.0, "step": 0.01,
|
||||
"tooltip": "Time-step shift for the OmniVoice noise schedule. Recommended: 0.1."
|
||||
}),
|
||||
"speed": ("FLOAT", {
|
||||
"default": 1.0, "min": 0.25, "max": 3.0, "step": 0.05,
|
||||
"tooltip": "Speech speed factor. Values above 1 are faster; values below 1 are slower. Recommended: 1.0. Technically, this rescales OmniVoice's estimated target audio-token length before generation. It is not waveform time-stretch post-processing."
|
||||
}),
|
||||
"duration": ("FLOAT", {
|
||||
"default": 0.0, "min": 0.0, "max": 600.0, "step": 0.1,
|
||||
"tooltip": "Fixed output duration in seconds. Set 0 to let OmniVoice estimate duration from text. Technically, this overrides speed and sets the planned target audio-token length from seconds, so it behaves more like native target-length control than a simple max-tokens cap."
|
||||
}),
|
||||
},
|
||||
"optional": {
|
||||
"dtype": (["auto", "bfloat16", "float16", "float32"], {
|
||||
"default": "auto",
|
||||
"tooltip": "Runtime precision. Auto prefers bf16 on newer CUDA/XPU hardware, else fp16, with fp32 on CPU."
|
||||
}),
|
||||
"instruct": ("STRING", {
|
||||
"default": "",
|
||||
"multiline": True,
|
||||
"tooltip": "Optional OmniVoice voice-design / style instruction. Strong effect when no narrator/reference is provided, because the model uses it to design the voice directly. With narrator/reference cloning, it still acts as guidance, but the effect is usually much weaker because OmniVoice prioritizes the provided voice identity."
|
||||
}),
|
||||
"layer_penalty_factor": ("FLOAT", {
|
||||
"default": 5.0, "min": 0.0, "max": 10.0, "step": 0.1,
|
||||
"tooltip": "Penalty encouraging earlier codebook layers to unmask first. Recommended: 5.0."
|
||||
}),
|
||||
"position_temperature": ("FLOAT", {
|
||||
"default": 5.0, "min": 0.0, "max": 10.0, "step": 0.1,
|
||||
"tooltip": "Mask-position temperature. 0 is greedy; higher values add randomness. Recommended: 5.0."
|
||||
}),
|
||||
"class_temperature": ("FLOAT", {
|
||||
"default": 0.0, "min": 0.0, "max": 2.0, "step": 0.05,
|
||||
"tooltip": "Token sampling temperature. 0 is greedy; higher values are more random. Recommended: 0.0."
|
||||
}),
|
||||
"denoise": ("BOOLEAN", {
|
||||
"default": True,
|
||||
"tooltip": "Enable the denoise token. Recommended: on."
|
||||
}),
|
||||
"preprocess_prompt": ("BOOLEAN", {
|
||||
"default": True,
|
||||
"tooltip": "Preprocess the voice-cloning reference by trimming long reference audio, removing silence, and adding terminal punctuation to the reference text. Recommended: on."
|
||||
}),
|
||||
"postprocess_output": ("BOOLEAN", {
|
||||
"default": True,
|
||||
"tooltip": "Post-process generated audio with silence cleanup plus fade and edge padding. Recommended: on."
|
||||
}),
|
||||
"audio_chunk_duration": ("FLOAT", {
|
||||
"default": 15.0, "min": 1.0, "max": 60.0, "step": 0.5,
|
||||
"tooltip": "Native long-form target chunk duration in seconds. This is OmniVoice's real long-text control and supersedes suite char-based chunking. Recommended: 15.0."
|
||||
}),
|
||||
"audio_chunk_threshold": ("FLOAT", {
|
||||
"default": 30.0, "min": 1.0, "max": 180.0, "step": 0.5,
|
||||
"tooltip": "Native long-form activation threshold in seconds. Estimated outputs longer than this use OmniVoice's built-in chunking path. Recommended: 30.0."
|
||||
}),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("TTS_ENGINE",)
|
||||
RETURN_NAMES = ("TTS_engine",)
|
||||
FUNCTION = "create_engine_config"
|
||||
CATEGORY = "TTS Audio Suite/⚙️ Engines"
|
||||
|
||||
def create_engine_config(
|
||||
self,
|
||||
model_variant: str,
|
||||
device: str,
|
||||
language: str,
|
||||
num_step: int,
|
||||
guidance_scale: float,
|
||||
t_shift: float,
|
||||
speed: float,
|
||||
duration: float,
|
||||
dtype: str = "auto",
|
||||
instruct: str = "",
|
||||
layer_penalty_factor: float = 5.0,
|
||||
position_temperature: float = 5.0,
|
||||
class_temperature: float = 0.0,
|
||||
denoise: bool = True,
|
||||
preprocess_prompt: bool = True,
|
||||
postprocess_output: bool = True,
|
||||
audio_chunk_duration: float = 15.0,
|
||||
audio_chunk_threshold: float = 30.0,
|
||||
) -> tuple:
|
||||
config = {
|
||||
"engine_type": "omnivoice",
|
||||
"model_variant": model_variant,
|
||||
"device": device,
|
||||
"dtype": dtype,
|
||||
"language": str(language or "Auto").strip() or "Auto",
|
||||
"instruct": str(instruct or "").strip(),
|
||||
"num_steps": int(num_step),
|
||||
"guidance_scale": float(guidance_scale),
|
||||
"t_shift": float(t_shift),
|
||||
"speed": float(speed),
|
||||
"duration": float(duration),
|
||||
"layer_penalty_factor": float(layer_penalty_factor),
|
||||
"position_temperature": float(position_temperature),
|
||||
"class_temperature": float(class_temperature),
|
||||
"denoise": bool(denoise),
|
||||
"preprocess_prompt": bool(preprocess_prompt),
|
||||
"postprocess_output": bool(postprocess_output),
|
||||
"audio_chunk_duration": float(audio_chunk_duration),
|
||||
"audio_chunk_threshold": float(audio_chunk_threshold),
|
||||
}
|
||||
|
||||
print(f"⚙️ OmniVoice: Configured on {device}")
|
||||
print(
|
||||
f" Model: {model_variant} | Language: {config['language']} | Dtype: {dtype}"
|
||||
)
|
||||
print(
|
||||
" Settings: steps={steps}, guidance_scale={guidance}, t_shift={t_shift}, speed={speed}, duration={duration}".format(
|
||||
steps=num_step,
|
||||
guidance=guidance_scale,
|
||||
t_shift=t_shift,
|
||||
speed=speed,
|
||||
duration="auto" if duration <= 0 else duration,
|
||||
)
|
||||
)
|
||||
print(
|
||||
" Native long-form: chunk_duration={chunk_duration}s, threshold={threshold}s".format(
|
||||
chunk_duration=audio_chunk_duration,
|
||||
threshold=audio_chunk_threshold,
|
||||
)
|
||||
)
|
||||
print(" Voice cloning requires reference text. Use Character Voices or a .reference.txt narrator voice.")
|
||||
if instruct:
|
||||
print(f" Instruct: {instruct[:80]}..." if len(instruct) > 80 else f" Instruct: {instruct}")
|
||||
|
||||
engine_data = {
|
||||
"engine_type": "omnivoice",
|
||||
"config": config,
|
||||
"adapter_class": "OmniVoiceEngineAdapter",
|
||||
}
|
||||
return (engine_data,)
|
||||
@@ -33,7 +33,8 @@ BaseF5TTSNode = f5tts_base_module.BaseF5TTSNode
|
||||
|
||||
from utils.text.chunking import ImprovedChatterBoxChunker
|
||||
from utils.audio.processing import AudioProcessingUtils
|
||||
from utils.voice.discovery import get_available_voices, load_voice_reference, get_available_characters, get_character_mapping
|
||||
from utils.voice.discovery import get_available_voices, load_voice_reference, get_available_characters, get_character_mapping
|
||||
from utils.voice.character_logging import resolved_character_label
|
||||
from utils.text.character_parser import parse_character_text, character_parser
|
||||
from utils.text.step_audio_editx_special_tags import parse_edit_tags_with_iterations
|
||||
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
|
||||
@@ -607,7 +608,8 @@ Back to the main narrator voice for the conclusion.""",
|
||||
# Generate audio for each chunk of this character segment
|
||||
segment_audio_chunks = []
|
||||
for chunk_i, chunk_text in enumerate(segment_chunks):
|
||||
print(f"🎤 Generating F5-TTS segment {segment_display_idx}/{total_segments} chunk {chunk_i+1}/{len(segment_chunks)} for '{character}' (lang: {lang_code})...")
|
||||
display_name = resolved_character_label(character, (char_audio, char_text))
|
||||
print(f"🎤 Generating F5-TTS segment {segment_display_idx}/{total_segments} chunk {chunk_i+1}/{len(segment_chunks)} for '{display_name}' (lang: {lang_code})...")
|
||||
|
||||
# Create cache function for this character if caching is enabled (uses per-segment parameters)
|
||||
cache_fn = None
|
||||
@@ -1196,4 +1198,4 @@ Back to the main narrator voice for the conclusion.""",
|
||||
print(f"❌ F5-TTS streaming segment {original_idx} failed: {e}")
|
||||
# Return silence as fallback
|
||||
return torch.zeros((1, 24000)) # 1 second of silence at 24kHz
|
||||
return self.process_with_error_handling(_process)
|
||||
return self.process_with_error_handling(_process)
|
||||
|
||||
@@ -34,7 +34,8 @@ BaseF5TTSNode = f5tts_base_module.BaseF5TTSNode
|
||||
|
||||
from utils.system.import_manager import import_manager
|
||||
from utils.audio.processing import AudioProcessingUtils
|
||||
from utils.voice.discovery import get_available_voices, load_voice_reference, get_available_characters, get_character_mapping
|
||||
from utils.voice.discovery import get_available_voices, load_voice_reference, get_available_characters, get_character_mapping
|
||||
from utils.voice.character_logging import resolved_character_label
|
||||
from utils.text.character_parser import parse_character_text, character_parser
|
||||
from utils.text.step_audio_editx_special_tags import parse_edit_tags_with_iterations
|
||||
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
|
||||
@@ -537,19 +538,7 @@ Hello! This is F5-TTS SRT with character switching.
|
||||
# Check for interruption
|
||||
self.check_interruption(f"F5-TTS character segment {subtitle_idx+1}.{seg_idx+1} (Seq {subtitle.sequence})")
|
||||
|
||||
# Show generation message
|
||||
if character == "narrator":
|
||||
if language != 'en':
|
||||
print(f"🎤 Generating F5-TTS segment {subtitle_idx+1}.{seg_idx+1} (Seq {subtitle.sequence}) in {language}...")
|
||||
else:
|
||||
print(f"🎤 Generating F5-TTS segment {subtitle_idx+1}.{seg_idx+1} (Seq {subtitle.sequence})...")
|
||||
else:
|
||||
if language != 'en':
|
||||
print(f"🎭 Generating F5-TTS segment {subtitle_idx+1}.{seg_idx+1} using '{character}' in {language}")
|
||||
else:
|
||||
print(f"🎭 Generating F5-TTS segment {subtitle_idx+1}.{seg_idx+1} using '{character}'")
|
||||
|
||||
# PRIORITY FIX: For "narrator" character, always use main reference when provided
|
||||
# PRIORITY FIX: For "narrator" character, always use main reference when provided
|
||||
# This ensures Character Voices node and dropdown selection work correctly
|
||||
if character == "narrator" and audio_prompt and validated_ref_text:
|
||||
# Use main reference for narrator (from Character Voices or dropdown)
|
||||
@@ -567,8 +556,16 @@ Hello! This is F5-TTS SRT with character switching.
|
||||
char_audio = audio_prompt
|
||||
char_text = validated_ref_text
|
||||
print(f"🔄 F5-TTS: Using main reference fallback for '{character}'")
|
||||
else:
|
||||
print(f"🎭 F5-TTS: Using character-specific voice for '{character}'")
|
||||
else:
|
||||
print(f"🎭 F5-TTS: Using character-specific voice for '{character}'")
|
||||
|
||||
display_name = resolved_character_label(character, (char_audio, char_text))
|
||||
if character == "narrator":
|
||||
suffix = f" in {language}" if language != 'en' else ""
|
||||
print(f"🎤 Generating F5-TTS segment {subtitle_idx+1}.{seg_idx+1} (Seq {subtitle.sequence}) for '{display_name}'{suffix}...")
|
||||
else:
|
||||
suffix = f" in {language}" if language != 'en' else ""
|
||||
print(f"🎭 Generating F5-TTS segment {subtitle_idx+1}.{seg_idx+1} using '{display_name}'{suffix}")
|
||||
|
||||
# Validate and clamp nfe_step to prevent ODE solver issues
|
||||
safe_nfe_step = max(1, min(nfe_step, 71))
|
||||
@@ -1040,11 +1037,11 @@ Hello! This is F5-TTS SRT with character switching.
|
||||
# Set phonemization setting for consistency (even when using cache)
|
||||
import os
|
||||
os.environ['F5TTS_AUTO_PHONEMIZATION'] = str(auto_phonemization).lower()
|
||||
print(f"💾 F5-TTS SRT streaming: Cache hit for {character} segment {original_idx}")
|
||||
print(f"💾 F5-TTS SRT streaming: Cache hit for {resolved_character_label(character, ref_audio_path)} segment {original_idx}")
|
||||
return cached_audio[0]
|
||||
|
||||
# Generate audio for this segment
|
||||
print(f"🔄 F5-TTS SRT streaming: Generating {character} segment {original_idx} in {language}")
|
||||
print(f"🔄 F5-TTS SRT streaming: Generating {resolved_character_label(character, ref_audio_path)} segment {original_idx} in {language}")
|
||||
|
||||
# Generate with F5-TTS (using base class method)
|
||||
wav = self.generate_f5tts_with_pause_tags(
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
"""Fish Audio S2 processors."""
|
||||
@@ -0,0 +1,339 @@
|
||||
"""Fish Audio S2 Pro text orchestration."""
|
||||
|
||||
import re
|
||||
from typing import Any, Dict, List
|
||||
|
||||
import torch
|
||||
import comfy.model_management as model_management
|
||||
|
||||
from utils.audio.chunk_timing import ChunkTimingHelper
|
||||
from utils.text.character_parser import character_parser
|
||||
from utils.text.fish_audio_s2_tags import get_fish_language_instruction
|
||||
from utils.text.pause_processor import PauseTagProcessor
|
||||
from utils.text.segment_parameters import ParameterValidator, apply_segment_parameters
|
||||
from utils.voice.discovery import (
|
||||
get_available_characters,
|
||||
get_character_default_language,
|
||||
get_character_mapping,
|
||||
voice_discovery,
|
||||
)
|
||||
from utils.voice.character_logging import (
|
||||
format_character_override_warning,
|
||||
resolved_character_label,
|
||||
resolved_voice_name,
|
||||
)
|
||||
|
||||
|
||||
class FishAudioS2Processor:
|
||||
SAMPLE_RATE = 44100
|
||||
|
||||
def __init__(self, adapter, engine_config: Dict[str, Any]):
|
||||
self.adapter = adapter
|
||||
self.config = dict(engine_config or {})
|
||||
|
||||
def update_config(self, config):
|
||||
self.config = dict(config or {})
|
||||
self.adapter.update_config(self.config)
|
||||
|
||||
def _setup_character_parser(self, text):
|
||||
tags = [tag.split("|")[0].strip() for tag in re.findall(r"\[([^\]]+)\]", text or "") if not tag.lower().startswith(("pause:", "wait:", "stop:"))]
|
||||
available = set(get_available_characters() or [])
|
||||
aliases = voice_discovery.get_character_aliases()
|
||||
available.update(aliases.keys())
|
||||
available.update(aliases.values())
|
||||
available.update(tag.lower() for tag in tags)
|
||||
available.add("narrator")
|
||||
character_parser.set_available_characters(list(available))
|
||||
for character, language in voice_discovery.get_character_language_defaults().items():
|
||||
character_parser.set_character_language_default(character, language)
|
||||
character_parser.reset_session_cache()
|
||||
|
||||
def get_character_order(self, text):
|
||||
self._setup_character_parser(text)
|
||||
order = []
|
||||
for segment in character_parser.parse_text_segments(text, engine_type="fish_audio_s2"):
|
||||
character = segment.character or "narrator"
|
||||
if character not in order:
|
||||
order.append(character)
|
||||
if "narrator" in order:
|
||||
order = ["narrator"] + [character for character in order if character != "narrator"]
|
||||
return order
|
||||
|
||||
def _apply_language_instruction(self, text: str, segment) -> str:
|
||||
if self.config.get("language_prompting", "Auto Inline Tag") != "Auto Inline Tag":
|
||||
return text
|
||||
instruction = get_fish_language_instruction(
|
||||
getattr(segment, "language", None),
|
||||
explicit=bool(getattr(segment, "explicit_language", False)),
|
||||
)
|
||||
if not instruction:
|
||||
return text
|
||||
stripped = (text or "").lstrip()
|
||||
if stripped.startswith("<"):
|
||||
return text
|
||||
return f"<{instruction}> {text}"
|
||||
|
||||
def _get_effective_language_instruction(self, segment, override_voice_ref: Dict[str, Any] | None) -> str | None:
|
||||
explicit_language = bool(getattr(segment, "explicit_language", False))
|
||||
if explicit_language:
|
||||
return get_fish_language_instruction(
|
||||
getattr(segment, "language", None),
|
||||
explicit=True,
|
||||
)
|
||||
if isinstance(override_voice_ref, dict):
|
||||
override_character = override_voice_ref.get("character_name")
|
||||
if override_character:
|
||||
override_language = get_character_default_language(str(override_character))
|
||||
return get_fish_language_instruction(override_language, explicit=False)
|
||||
return None
|
||||
return get_fish_language_instruction(
|
||||
getattr(segment, "language", None),
|
||||
explicit=False,
|
||||
)
|
||||
|
||||
def _apply_effective_language_instruction(
|
||||
self, text: str, segment, override_voice_ref: Dict[str, Any] | None
|
||||
) -> tuple[str, str | None]:
|
||||
if self.config.get("language_prompting", "Auto Inline Tag") != "Auto Inline Tag":
|
||||
return text, None
|
||||
instruction = self._get_effective_language_instruction(segment, override_voice_ref)
|
||||
if not instruction:
|
||||
return text, None
|
||||
stripped = (text or "").lstrip()
|
||||
if stripped.startswith("<"):
|
||||
return text, instruction
|
||||
return f"<{instruction}> {text}", instruction
|
||||
|
||||
@staticmethod
|
||||
def _voice_for(character, voice_mapping, discovered, narrator):
|
||||
if character in voice_mapping and isinstance(voice_mapping[character], dict):
|
||||
return dict(voice_mapping[character])
|
||||
audio_path, ref_text = discovered.get(character, (None, None))
|
||||
if audio_path:
|
||||
return {"audio_path": audio_path, "reference_text": ref_text or ""}
|
||||
return dict(narrator)
|
||||
|
||||
def process_text(self, text: str, voice_mapping: Dict[str, Any], seed: int,
|
||||
enable_chunking: bool = True, max_chars_per_chunk: int = 400,
|
||||
chunk_combination_method: str = "auto", silence_between_chunks_ms: int = 100,
|
||||
enable_audio_cache: bool = True, apply_edit_postprocessing: bool = True,
|
||||
show_text_logging: bool = True, reference_order=None) -> List[Dict[str, Any]]:
|
||||
del enable_chunking, max_chars_per_chunk, chunk_combination_method
|
||||
del silence_between_chunks_ms, apply_edit_postprocessing
|
||||
self._setup_character_parser(text)
|
||||
segments = character_parser.parse_text_segments(text, engine_type="fish_audio_s2")
|
||||
characters = []
|
||||
for segment in segments:
|
||||
character = segment.character or "narrator"
|
||||
if character not in characters:
|
||||
characters.append(character)
|
||||
reference_characters = list(reference_order or characters)
|
||||
narrator_exists_globally = "narrator" in reference_characters
|
||||
if narrator_exists_globally:
|
||||
reference_characters = ["narrator"] + [character for character in reference_characters if character != "narrator"]
|
||||
global_speaker_number = {
|
||||
character: index + 1
|
||||
for index, character in enumerate(reference_characters)
|
||||
}
|
||||
discovered = get_character_mapping(characters, engine_type="audio_only")
|
||||
narrator_value = voice_mapping.get("narrator", {})
|
||||
narrator = dict(narrator_value) if isinstance(narrator_value, dict) else {"audio": narrator_value}
|
||||
configured_refs = list(self.config.get("speaker_references") or [])
|
||||
speaker_map = {character: index for index, character in enumerate(characters)}
|
||||
voice_refs = []
|
||||
narrator_has_reference = bool(narrator_value)
|
||||
for character in characters:
|
||||
effective_speaker_number = global_speaker_number.get(character, speaker_map[character] + 1)
|
||||
if narrator_has_reference and narrator_exists_globally and character == "narrator":
|
||||
voice_refs.append(dict(narrator))
|
||||
continue
|
||||
if narrator_has_reference and not narrator_exists_globally and effective_speaker_number == 1:
|
||||
voice_refs.append(dict(narrator))
|
||||
continue
|
||||
configured_index = effective_speaker_number - 2
|
||||
if 0 <= configured_index < len(configured_refs):
|
||||
voice_refs.append(configured_refs[configured_index])
|
||||
continue
|
||||
voice_refs.append(self._voice_for(character, voice_mapping, discovered, narrator))
|
||||
voice_ref_by_character = dict(zip(characters, voice_refs))
|
||||
custom_switching = self.config.get(
|
||||
"multi_speaker_mode", "Native Multi-Speaker"
|
||||
) == "Custom Character Switching"
|
||||
records = []
|
||||
pending_turns = []
|
||||
pending_logs = []
|
||||
pending_config = None
|
||||
group_index = 0
|
||||
speaker_labels = {index: character for character, index in speaker_map.items()}
|
||||
connected_speakers = []
|
||||
if narrator_has_reference:
|
||||
connected_speakers.append("Speaker 1 (Narrator)")
|
||||
connected_speakers.extend(
|
||||
f"Speaker {idx + 2}"
|
||||
for idx, ref in enumerate(configured_refs)
|
||||
if ref is not None
|
||||
)
|
||||
logged_priority_overrides = set()
|
||||
|
||||
def generation_key(config):
|
||||
return tuple(config.get(name) for name in (
|
||||
"temperature", "top_p", "repetition_penalty", "native_chunk_length",
|
||||
"max_new_tokens", "normalize", "cache_reference",
|
||||
))
|
||||
|
||||
def print_custom_segment_preview(log_label: str, turn_text: str):
|
||||
if not show_text_logging:
|
||||
return
|
||||
print("🎭 Fish Audio S2 custom segment:")
|
||||
print("============================================================")
|
||||
print(f"[{log_label}] {turn_text}")
|
||||
print("============================================================")
|
||||
|
||||
def flush_turns():
|
||||
nonlocal pending_turns, pending_logs, group_index
|
||||
if not pending_turns:
|
||||
return
|
||||
self.adapter.update_config(pending_config)
|
||||
if show_text_logging:
|
||||
print(f"🎭 Fish Audio S2 native dialogue: Processing {len(pending_turns)} segment(s)")
|
||||
if connected_speakers:
|
||||
print(f"🎤 Fish speaker inputs connected: {connected_speakers}")
|
||||
mapping_display = ", ".join(
|
||||
f"{character}->local Speaker {speaker_idx + 1} "
|
||||
f"(global Speaker {global_speaker_number.get(character, speaker_idx + 1)})"
|
||||
for character, speaker_idx in sorted(speaker_map.items(), key=lambda item: item[1])
|
||||
)
|
||||
if mapping_display:
|
||||
print(f"🎭 Fish character mapping: {mapping_display}")
|
||||
print("============================================================")
|
||||
for log_label, turn_text in pending_logs:
|
||||
print(f"[{log_label}] {turn_text}")
|
||||
print("============================================================")
|
||||
audio = self.adapter.generate_dialogue(
|
||||
turns=pending_turns, voice_refs=voice_refs,
|
||||
seed=int(pending_config.get("seed", seed)) + group_index,
|
||||
enable_audio_cache=enable_audio_cache,
|
||||
)
|
||||
combined_text = "\n".join(turn_text for _, turn_text in pending_turns)
|
||||
records.append({
|
||||
"waveform": audio.cpu(), "sample_rate": self.SAMPLE_RATE,
|
||||
"text": combined_text, "edit_tags": [],
|
||||
})
|
||||
pending_turns = []
|
||||
pending_logs = []
|
||||
group_index += 1
|
||||
|
||||
for segment in segments:
|
||||
if model_management.interrupt_processing:
|
||||
raise InterruptedError("Fish Audio S2 text generation interrupted")
|
||||
segment_text = (segment.text or "").strip()
|
||||
if not segment_text:
|
||||
continue
|
||||
params = segment.parameters or {}
|
||||
filtered = ParameterValidator.filter_parameters_for_engine(params, "fish_audio_s2")
|
||||
config = apply_segment_parameters(self.config, params, "fish_audio_s2") if filtered else dict(self.config)
|
||||
character = segment.character or "narrator"
|
||||
original_character = getattr(segment, "original_character", None) or character
|
||||
speaker_idx = speaker_map[character]
|
||||
effective_speaker_number = global_speaker_number.get(character, speaker_idx + 1)
|
||||
configured_index = effective_speaker_number - 2
|
||||
override_voice_ref = None
|
||||
if (
|
||||
narrator_has_reference
|
||||
and (
|
||||
(narrator_exists_globally and character == "narrator")
|
||||
or (not narrator_exists_globally and effective_speaker_number == 1)
|
||||
)
|
||||
):
|
||||
discovered_ref = discovered.get(character, (None, None))[0]
|
||||
override_voice_ref = narrator
|
||||
if discovered_ref and show_text_logging:
|
||||
override_key = (speaker_idx, character)
|
||||
if override_key not in logged_priority_overrides:
|
||||
print(format_character_override_warning(
|
||||
"Fish", "Speaker 1 (Narrator)", original_character
|
||||
))
|
||||
logged_priority_overrides.add(override_key)
|
||||
elif 0 <= configured_index < len(configured_refs):
|
||||
configured_ref = configured_refs[configured_index]
|
||||
override_voice_ref = configured_ref
|
||||
discovered_ref = discovered.get(character, (None, None))[0]
|
||||
if configured_ref is not None and discovered_ref:
|
||||
override_key = (speaker_idx, character)
|
||||
if override_key not in logged_priority_overrides and show_text_logging:
|
||||
print(format_character_override_warning(
|
||||
"Fish", f"Speaker {effective_speaker_number}", original_character
|
||||
))
|
||||
logged_priority_overrides.add(override_key)
|
||||
if pending_config is not None and generation_key(config) != generation_key(pending_config):
|
||||
flush_turns()
|
||||
pending_config = config
|
||||
|
||||
pause_parts, _ = PauseTagProcessor.parse_pause_tags(segment_text)
|
||||
if not pause_parts:
|
||||
pause_parts = [("text", segment_text)]
|
||||
for part_type, content in pause_parts:
|
||||
if model_management.interrupt_processing:
|
||||
raise InterruptedError("Fish Audio S2 text generation interrupted")
|
||||
if part_type == "pause":
|
||||
flush_turns()
|
||||
records.append({
|
||||
"waveform": PauseTagProcessor.create_silence_segment(content, self.SAMPLE_RATE),
|
||||
"sample_rate": self.SAMPLE_RATE, "text": f"[pause:{content}s]", "edit_tags": [],
|
||||
})
|
||||
continue
|
||||
turn_text = str(content).strip()
|
||||
if not turn_text:
|
||||
continue
|
||||
turn_text, language_instruction = self._apply_effective_language_instruction(
|
||||
turn_text, segment, override_voice_ref
|
||||
)
|
||||
if show_text_logging and self.config.get("language_prompting", "Auto Inline Tag") == "Auto Inline Tag":
|
||||
if language_instruction:
|
||||
print(f" 🌍 Fish Audio S2 prompting language via <{language_instruction}>")
|
||||
if custom_switching:
|
||||
flush_turns()
|
||||
log_label = character
|
||||
if narrator_has_reference and character == "narrator":
|
||||
log_label = resolved_voice_name(narrator, "Speaker 1 (Narrator)")
|
||||
elif override_voice_ref is not None:
|
||||
log_label = resolved_character_label(character, override_voice_ref)
|
||||
print_custom_segment_preview(log_label, turn_text)
|
||||
self.adapter.update_config(config)
|
||||
audio = self.adapter.generate_dialogue(
|
||||
turns=[(0, turn_text)],
|
||||
voice_refs=[voice_ref_by_character[character]],
|
||||
seed=int(config.get("seed", seed)) + group_index,
|
||||
enable_audio_cache=enable_audio_cache,
|
||||
cache_character=character,
|
||||
)
|
||||
records.append({
|
||||
"waveform": audio.cpu(), "sample_rate": self.SAMPLE_RATE,
|
||||
"text": turn_text, "edit_tags": [],
|
||||
})
|
||||
group_index += 1
|
||||
else:
|
||||
pending_turns.append((speaker_map[character], turn_text))
|
||||
log_label = resolved_character_label(
|
||||
character, voice_ref_by_character.get(character)
|
||||
)
|
||||
pending_logs.append((log_label, turn_text))
|
||||
flush_turns()
|
||||
return records
|
||||
|
||||
def combine_audio_segments(self, segments, method="auto", silence_ms=100,
|
||||
original_text="", return_info=False):
|
||||
if not segments:
|
||||
empty = torch.zeros(0)
|
||||
return (empty, {}) if return_info else empty
|
||||
return ChunkTimingHelper.combine_audio_with_timing(
|
||||
audio_segments=[item["waveform"] for item in segments],
|
||||
combination_method=method, silence_ms=silence_ms, crossfade_duration=0.1,
|
||||
sample_rate=self.SAMPLE_RATE, text_length=len(original_text), original_text=original_text,
|
||||
text_chunks=[item.get("text", "") for item in segments],
|
||||
) if return_info else ChunkTimingHelper.combine_audio_with_timing(
|
||||
audio_segments=[item["waveform"] for item in segments],
|
||||
combination_method=method, silence_ms=silence_ms, crossfade_duration=0.1,
|
||||
sample_rate=self.SAMPLE_RATE, text_length=len(original_text), original_text=original_text,
|
||||
text_chunks=[item.get("text", "") for item in segments],
|
||||
)[0]
|
||||
@@ -0,0 +1,142 @@
|
||||
"""Fish Audio S2 Pro SRT orchestration."""
|
||||
|
||||
import importlib.util
|
||||
import os
|
||||
|
||||
import comfy.model_management as model_management
|
||||
import torch
|
||||
|
||||
from engines.adapters.fish_audio_s2_adapter import FishAudioS2Adapter
|
||||
from utils.system.import_manager import import_manager
|
||||
from utils.timing.assembly import AudioAssemblyEngine
|
||||
from utils.timing.engine import TimingEngine
|
||||
from utils.timing.overlap_detection import SRTOverlapHandler
|
||||
from utils.timing.reporting import SRTReportGenerator
|
||||
|
||||
|
||||
class FishAudioS2SRTProcessor:
|
||||
SAMPLE_RATE = 44100
|
||||
|
||||
def __init__(self, node_instance, config):
|
||||
self.node_instance = node_instance
|
||||
self.config = dict(config or {})
|
||||
self.adapter = FishAudioS2Adapter(self.config)
|
||||
self._processor = None
|
||||
success, modules, message = import_manager.import_srt_modules()
|
||||
if not success:
|
||||
raise ImportError(f"Fish Audio S2 SRT unavailable: {message}")
|
||||
self.SRTParser = modules["SRTParser"]
|
||||
|
||||
@property
|
||||
def processor(self):
|
||||
if self._processor is None:
|
||||
path = os.path.join(os.path.dirname(__file__), "fish_audio_s2_processor.py")
|
||||
spec = importlib.util.spec_from_file_location("fish_audio_s2_processor_module", path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
self._processor = module.FishAudioS2Processor(self.adapter, self.config)
|
||||
return self._processor
|
||||
|
||||
def update_config(self, config):
|
||||
self.config = dict(config or {})
|
||||
self.adapter.update_config(self.config)
|
||||
if self._processor is not None:
|
||||
self._processor.update_config(self.config)
|
||||
|
||||
@staticmethod
|
||||
def _check_interrupt(index=None, total=None):
|
||||
if model_management.interrupt_processing:
|
||||
where = f" at subtitle {index + 1}/{total}" if index is not None else ""
|
||||
raise InterruptedError(f"Fish Audio S2 SRT generation interrupted{where}")
|
||||
|
||||
def process_srt_content(self, srt_content, voice_mapping, seed, timing_mode,
|
||||
timing_params, enable_audio_cache=True):
|
||||
self._check_interrupt()
|
||||
subtitles = self.SRTParser().parse_srt_content(srt_content, allow_overlaps=True)
|
||||
has_overlaps = SRTOverlapHandler.detect_overlaps(subtitles)
|
||||
active_mode, switched = SRTOverlapHandler.handle_smart_natural_fallback(
|
||||
timing_mode, has_overlaps, "Fish Audio S2 SRT"
|
||||
)
|
||||
reference_order = self.processor.get_character_order(
|
||||
"\n".join((subtitle.text or "") for subtitle in subtitles)
|
||||
)
|
||||
audio_segments = []
|
||||
adjustments = []
|
||||
for index, subtitle in enumerate(subtitles):
|
||||
self._check_interrupt(index, len(subtitles))
|
||||
text = (subtitle.text or "").strip()
|
||||
if text:
|
||||
records = self.processor.process_text(
|
||||
text=text, voice_mapping=voice_mapping, seed=seed + index,
|
||||
enable_chunking=False, enable_audio_cache=enable_audio_cache,
|
||||
apply_edit_postprocessing=False,
|
||||
reference_order=reference_order,
|
||||
)
|
||||
audio, _ = self.processor.combine_audio_segments(
|
||||
records, method="auto", silence_ms=0, original_text=text, return_info=True
|
||||
)
|
||||
if audio.dim() == 1:
|
||||
audio = audio.unsqueeze(0)
|
||||
else:
|
||||
audio = torch.zeros(1, int(subtitle.duration * self.SAMPLE_RATE))
|
||||
audio = audio.cpu()
|
||||
audio_segments.append(audio)
|
||||
natural = audio.shape[-1] / self.SAMPLE_RATE
|
||||
target = subtitle.duration
|
||||
ratio = target / natural if natural else 1.0
|
||||
adjustments.append({
|
||||
"index": index, "segment_index": index, "sequence": subtitle.sequence,
|
||||
"natural_duration": natural, "target_start": subtitle.start_time,
|
||||
"target_end": subtitle.end_time, "target_duration": target,
|
||||
"start_time": subtitle.start_time, "end_time": subtitle.end_time,
|
||||
"stretch_factor": ratio, "needs_stretching": abs(ratio - 1.0) > 0.05,
|
||||
"stretch_type": "compress" if ratio < 1 else "expand" if ratio > 1 else "none",
|
||||
"adjustment": natural - target, "adjusted_start": subtitle.start_time,
|
||||
"adjusted_end": subtitle.end_time, "adjusted_duration": natural,
|
||||
})
|
||||
|
||||
self._check_interrupt()
|
||||
final_audio, replacement, stretch_method = self._assemble(
|
||||
audio_segments, subtitles, active_mode, timing_params
|
||||
)
|
||||
if replacement is not None:
|
||||
adjustments = replacement
|
||||
reporter = SRTReportGenerator()
|
||||
report = reporter.generate_timing_report(
|
||||
subtitles, adjustments, active_mode, has_overlaps, switched,
|
||||
timing_mode if switched else None, stretch_method,
|
||||
)
|
||||
adjusted_srt = reporter.generate_adjusted_srt_string(subtitles, adjustments, active_mode)
|
||||
if final_audio.dim() == 1:
|
||||
final_audio = final_audio.unsqueeze(0).unsqueeze(0)
|
||||
elif final_audio.dim() == 2:
|
||||
final_audio = final_audio.unsqueeze(0)
|
||||
duration = final_audio.shape[-1] / self.SAMPLE_RATE
|
||||
info = f"Generated {duration:.1f}s Fish Audio S2 SRT audio from {len(subtitles)} subtitles using {active_mode} mode"
|
||||
return {"waveform": final_audio, "sample_rate": self.SAMPLE_RATE}, info, report, adjusted_srt
|
||||
|
||||
def _assemble(self, audio_segments, subtitles, mode, params):
|
||||
if mode == "stretch_to_fit":
|
||||
from engines.chatterbox.audio_timing import TimedAudioAssembler
|
||||
assembler = TimedAudioAssembler(self.SAMPLE_RATE)
|
||||
audio, method = assembler.assemble_timed_audio(
|
||||
audio_segments, [(item.start_time, item.end_time) for item in subtitles],
|
||||
fade_duration=params.get("fade_for_StretchToFit", 0.01),
|
||||
)
|
||||
return audio, None, method
|
||||
assembler = AudioAssemblyEngine(self.SAMPLE_RATE)
|
||||
if mode == "pad_with_silence":
|
||||
return assembler.assemble_with_overlaps(audio_segments, subtitles, torch.device("cpu")), None, None
|
||||
timing = TimingEngine(self.SAMPLE_RATE)
|
||||
if mode == "concatenate":
|
||||
replacements = timing.calculate_concatenation_adjustments(audio_segments, subtitles)
|
||||
return assembler.assemble_concatenation(audio_segments, params.get("fade_for_StretchToFit", 0.01)), replacements, None
|
||||
replacements, processed = timing.calculate_smart_timing_adjustments(
|
||||
audio_segments, subtitles, params.get("timing_tolerance", 2.0),
|
||||
params.get("max_stretch_ratio", 1.0), params.get("min_stretch_ratio", 0.5),
|
||||
torch.device("cpu"),
|
||||
)
|
||||
audio = assembler.assemble_smart_natural(
|
||||
audio_segments, processed, replacements, subtitles, torch.device("cpu")
|
||||
)
|
||||
return audio, replacements, None
|
||||
@@ -17,7 +17,8 @@ if project_root not in sys.path:
|
||||
sys.path.insert(0, project_root)
|
||||
|
||||
from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment
|
||||
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
|
||||
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
|
||||
from utils.voice.character_logging import resolved_character_label
|
||||
|
||||
|
||||
class HiggsAudioSRTProcessor:
|
||||
@@ -236,7 +237,8 @@ class HiggsAudioSRTProcessor:
|
||||
higgs_text = f"[{display_name}] {segment_text}"
|
||||
|
||||
# Log what's actually being processed
|
||||
print(f"📺 Processing SRT segment {i+1}/{len(srt_segments)} ({character}): '{higgs_text[:50]}...' ({expected_duration:.2f}s)")
|
||||
display_name = resolved_character_label(character, char_audio_dict)
|
||||
print(f"📺 Processing SRT segment {i+1}/{len(srt_segments)} ({display_name}): '{higgs_text[:50]}...' ({expected_duration:.2f}s)")
|
||||
|
||||
# Apply per-segment parameters
|
||||
current_params = dict(generation_params)
|
||||
@@ -539,4 +541,4 @@ class HiggsAudioSRTProcessor:
|
||||
empty_audio = AudioProcessingUtils.create_silence(1.0, self.sample_rate)
|
||||
empty_comfy = AudioProcessingUtils.format_for_comfyui(empty_audio, self.sample_rate)
|
||||
|
||||
return (empty_comfy, f"Higgs Audio SRT error: {e}", "Error: No timing report available", "Error: No adjusted SRT available")
|
||||
return (empty_comfy, f"Higgs Audio SRT error: {e}", "Error: No timing report available", "Error: No adjusted SRT available")
|
||||
|
||||
@@ -22,6 +22,10 @@ from utils.audio.processing import AudioProcessingUtils
|
||||
from utils.text.character_parser import character_parser
|
||||
from utils.text.segment_parameters import apply_segment_parameters
|
||||
from utils.voice.discovery import get_available_characters, voice_discovery
|
||||
from utils.voice.character_logging import (
|
||||
format_resolved_character_block,
|
||||
resolved_character_label,
|
||||
)
|
||||
|
||||
|
||||
DELIVERY_PROSODY_VALUES = {
|
||||
@@ -329,10 +333,9 @@ class HiggsAudioV3Processor:
|
||||
prefix = _delivery_state_prefix(delivery_state, skip_categories)
|
||||
prompt_text = prefix + chunk
|
||||
chunk_note = f" chunk {chunk_idx + 1}/{len(chunks)}" if len(chunks) > 1 else ""
|
||||
print(f"🎭 Higgs Audio v3 - Generating for '{character}'{chunk_note}:")
|
||||
print("=" * 60)
|
||||
print(prompt_text)
|
||||
print("=" * 60)
|
||||
display_name = resolved_character_label(character, voice_ref)
|
||||
print(f"🎭 Higgs Audio v3 - Generating for '{display_name}'{chunk_note}:")
|
||||
print(format_resolved_character_block(character, prompt_text, voice_ref))
|
||||
|
||||
audio_tensor = self.adapter.generate_with_pause_tags(
|
||||
prompt_text,
|
||||
|
||||
@@ -32,6 +32,10 @@ from utils.voice.discovery import (
|
||||
get_character_mapping,
|
||||
voice_discovery,
|
||||
)
|
||||
from utils.voice.character_logging import (
|
||||
format_resolved_character_block,
|
||||
resolved_character_label,
|
||||
)
|
||||
|
||||
|
||||
class MossTTSProcessor:
|
||||
@@ -802,7 +806,8 @@ class MossTTSProcessor:
|
||||
|
||||
if enable_chunking and len(clean_text) > max_chars:
|
||||
chunks = self.chunker.split_into_chunks(clean_text, max_chars)
|
||||
print(f"📝 MOSS-TTS: Chunking '{character}' into {len(chunks)} chunk(s) (language={language})")
|
||||
display_name = resolved_character_label(character, voice_ref)
|
||||
print(f"📝 MOSS-TTS: Chunking '{display_name}' into {len(chunks)} chunk(s) (language={language})")
|
||||
else:
|
||||
chunks = [clean_text]
|
||||
|
||||
@@ -811,7 +816,7 @@ class MossTTSProcessor:
|
||||
raise InterruptedError("MOSS-TTS generation interrupted by user")
|
||||
|
||||
chunk_note = f" chunk {chunk_idx}/{len(chunks)}" if len(chunks) > 1 else ""
|
||||
print(f"🎭 MOSS-TTS - Generating for '{character}' (language={language}){chunk_note}:")
|
||||
print(f"🎭 MOSS-TTS - Generating for '{display_name}' (language={language}){chunk_note}:")
|
||||
for field_name in self.OFFICIAL_INLINE_FIELDS:
|
||||
field_value = generation_params.get(field_name)
|
||||
if field_value:
|
||||
@@ -819,9 +824,7 @@ class MossTTSProcessor:
|
||||
f" 🔹 {field_name}: "
|
||||
f"{self._format_prompt_field_log(field_name, field_value, generation_params)}"
|
||||
)
|
||||
print("=" * 60)
|
||||
print(chunk)
|
||||
print("=" * 60)
|
||||
print(format_resolved_character_block(character, chunk, voice_ref))
|
||||
|
||||
audio_tensor = self.adapter.generate_with_pause_tags(chunk, voice_ref, generation_params, True, character)
|
||||
if audio_tensor.dim() == 1:
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
"""
|
||||
OmniVoice processor package.
|
||||
"""
|
||||
@@ -0,0 +1,164 @@
|
||||
"""
|
||||
Visual Tag Builder helper node.
|
||||
|
||||
Builds a valid OmniVoice voice-design instruct string with canonical category
|
||||
ordering while leaving the main engine's raw instruct field available for
|
||||
manual edits.
|
||||
"""
|
||||
|
||||
from typing import Dict, Tuple
|
||||
|
||||
|
||||
EN_TO_ZH: Dict[str, str] = {
|
||||
"male": "男",
|
||||
"female": "女",
|
||||
"child": "儿童",
|
||||
"teenager": "少年",
|
||||
"young adult": "青年",
|
||||
"middle-aged": "中年",
|
||||
"elderly": "老年",
|
||||
"very low pitch": "极低音调",
|
||||
"low pitch": "低音调",
|
||||
"moderate pitch": "中音调",
|
||||
"high pitch": "高音调",
|
||||
"very high pitch": "极高音调",
|
||||
"whisper": "耳语",
|
||||
}
|
||||
|
||||
|
||||
def _normalize_value(value: str) -> str:
|
||||
text = str(value or "").strip()
|
||||
return "" if not text or text == "None" else text
|
||||
|
||||
|
||||
class OmniVoiceInstructionBuilderNode:
|
||||
"""Structured OmniVoice instruction helper with canonical output order."""
|
||||
|
||||
@classmethod
|
||||
def NAME(cls):
|
||||
return "📐 Visual Tag Builder"
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"gender": (["None", "male", "female"], {
|
||||
"default": "None",
|
||||
"tooltip": "Used by the Visual Tag Builder preset."
|
||||
}),
|
||||
"age": (["None", "child", "teenager", "young adult", "middle-aged", "elderly"], {
|
||||
"default": "None",
|
||||
"tooltip": "Used by the Visual Tag Builder preset."
|
||||
}),
|
||||
"pitch": (["None", "very low pitch", "low pitch", "moderate pitch", "high pitch", "very high pitch"], {
|
||||
"default": "None",
|
||||
"tooltip": "Used by the Visual Tag Builder preset."
|
||||
}),
|
||||
"style": (["None", "whisper"], {
|
||||
"default": "None",
|
||||
"tooltip": "Used by the Visual Tag Builder preset."
|
||||
}),
|
||||
"accent": ([
|
||||
"None",
|
||||
"american accent",
|
||||
"british accent",
|
||||
"australian accent",
|
||||
"canadian accent",
|
||||
"indian accent",
|
||||
"chinese accent",
|
||||
"korean accent",
|
||||
"japanese accent",
|
||||
"portuguese accent",
|
||||
"russian accent",
|
||||
], {
|
||||
"default": "None",
|
||||
"tooltip": "Used by the Visual Tag Builder preset."
|
||||
}),
|
||||
"dialect": ([
|
||||
"None",
|
||||
"河南话",
|
||||
"陕西话",
|
||||
"四川话",
|
||||
"贵州话",
|
||||
"云南话",
|
||||
"桂林话",
|
||||
"济南话",
|
||||
"石家庄话",
|
||||
"甘肃话",
|
||||
"宁夏话",
|
||||
"青岛话",
|
||||
"东北话",
|
||||
], {
|
||||
"default": "None",
|
||||
"tooltip": "Used by the Visual Tag Builder preset."
|
||||
}),
|
||||
"output_language": (["English", "Chinese"], {
|
||||
"default": "English",
|
||||
"tooltip": "Controls the builder preview language and emitted text format."
|
||||
}),
|
||||
"instruct_text": ("STRING", {
|
||||
"default": "",
|
||||
"multiline": False,
|
||||
"tooltip": "Text emitted by the Visual Tag Builder."
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("STRING",)
|
||||
RETURN_NAMES = ("string",)
|
||||
FUNCTION = "build_instruction"
|
||||
CATEGORY = "TTS Audio Suite/📐 OmniVoice"
|
||||
DESCRIPTION = (
|
||||
"Visual tag and attribute builder with preset-driven ordering. "
|
||||
"Outputs a text string that can be used for OmniVoice or other "
|
||||
"tag-based prompt workflows."
|
||||
)
|
||||
|
||||
def build_instruction(
|
||||
self,
|
||||
gender: str,
|
||||
age: str,
|
||||
pitch: str,
|
||||
style: str,
|
||||
accent: str,
|
||||
dialect: str,
|
||||
output_language: str,
|
||||
instruct_text: str,
|
||||
) -> Tuple[str]:
|
||||
instruct_text = str(instruct_text or "").strip()
|
||||
if instruct_text:
|
||||
return (instruct_text,)
|
||||
|
||||
gender = _normalize_value(gender)
|
||||
age = _normalize_value(age)
|
||||
pitch = _normalize_value(pitch)
|
||||
style = _normalize_value(style)
|
||||
accent = _normalize_value(accent)
|
||||
dialect = _normalize_value(dialect)
|
||||
output_language = str(output_language or "English").strip()
|
||||
|
||||
# The builder UI enforces mutual exclusion; this keeps corrupted
|
||||
# workflow state from emitting an invalid mixed instruct string.
|
||||
if accent and dialect:
|
||||
dialect = ""
|
||||
|
||||
parts = [gender, age, pitch, style, accent or dialect]
|
||||
parts = [part for part in parts if part]
|
||||
|
||||
if not parts:
|
||||
return ("",)
|
||||
|
||||
if output_language == "Chinese":
|
||||
translated = [EN_TO_ZH.get(part, part) for part in parts]
|
||||
return (",".join(translated),)
|
||||
|
||||
return (", ".join(parts),)
|
||||
|
||||
|
||||
NODE_CLASS_MAPPINGS = {
|
||||
"OmniVoiceInstructionBuilderNode": OmniVoiceInstructionBuilderNode,
|
||||
}
|
||||
|
||||
NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"OmniVoiceInstructionBuilderNode": "📐 Visual Tag Builder",
|
||||
}
|
||||
@@ -0,0 +1,516 @@
|
||||
"""
|
||||
OmniVoice processor.
|
||||
|
||||
Handles character switching, pause tags, parameter switching, and native
|
||||
OmniVoice long-form generation orchestration.
|
||||
"""
|
||||
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from typing import Any, Dict, List, Optional, Tuple, Union
|
||||
|
||||
import comfy.model_management as model_management
|
||||
import torch
|
||||
|
||||
current_dir = os.path.dirname(__file__)
|
||||
nodes_dir = os.path.dirname(current_dir)
|
||||
project_root = os.path.dirname(nodes_dir)
|
||||
if project_root not in sys.path:
|
||||
sys.path.insert(0, project_root)
|
||||
|
||||
from utils.audio.chunk_timing import ChunkTimingHelper
|
||||
from utils.text.character_parser import character_parser
|
||||
from utils.text.omnivoice_special_tags import convert_omnivoice_special_tags
|
||||
from utils.text.pause_processor import PauseTagProcessor
|
||||
from utils.voice.character_logging import (
|
||||
format_resolved_character_block,
|
||||
resolved_character_label,
|
||||
)
|
||||
from utils.text.segment_parameters import ParameterValidator, apply_segment_parameters
|
||||
from utils.voice.discovery import get_available_characters, get_character_mapping, voice_discovery
|
||||
|
||||
|
||||
class OmniVoiceProcessor:
|
||||
"""Internal processor for OmniVoice generation."""
|
||||
|
||||
SAMPLE_RATE = 24000
|
||||
MIN_AUTO_SPEECH_FRAGMENT_DURATION = 0.12
|
||||
|
||||
def __init__(self, adapter, engine_config: Dict[str, Any]):
|
||||
self.adapter = adapter
|
||||
self.config = engine_config.copy() if engine_config else {}
|
||||
|
||||
def update_config(self, new_config: Dict[str, Any]):
|
||||
self.config = new_config.copy() if new_config else {}
|
||||
self.adapter.update_config(self.config)
|
||||
|
||||
@staticmethod
|
||||
def _check_interrupt(
|
||||
item_index: Optional[int] = None,
|
||||
total_items: Optional[int] = None,
|
||||
character_name: Optional[str] = None,
|
||||
) -> None:
|
||||
if not model_management.interrupt_processing:
|
||||
return
|
||||
|
||||
if item_index is not None and total_items is not None:
|
||||
if character_name:
|
||||
raise InterruptedError(
|
||||
f"OmniVoice text generation interrupted at item "
|
||||
f"{item_index + 1}/{total_items}, character '{character_name}'"
|
||||
)
|
||||
raise InterruptedError(
|
||||
f"OmniVoice text generation interrupted at item {item_index + 1}/{total_items}"
|
||||
)
|
||||
|
||||
raise InterruptedError("OmniVoice text generation interrupted by user")
|
||||
|
||||
@staticmethod
|
||||
def _first_non_none(*values):
|
||||
for value in values:
|
||||
if value is None:
|
||||
continue
|
||||
if isinstance(value, str) and not value.strip():
|
||||
continue
|
||||
return value
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _voice_log_note(voice_ref: Dict[str, Any]) -> str:
|
||||
if not isinstance(voice_ref, dict):
|
||||
return " [auto/default voice]"
|
||||
audio_ref = OmniVoiceProcessor._first_non_none(
|
||||
voice_ref.get("audio_path"),
|
||||
voice_ref.get("audio"),
|
||||
voice_ref.get("waveform"),
|
||||
)
|
||||
if audio_ref is None:
|
||||
return " [auto/default voice]"
|
||||
reference_text = voice_ref.get("reference_text")
|
||||
if reference_text:
|
||||
return f" [ref text: {len(reference_text)} chars]"
|
||||
return " [auto-transcribe reference]"
|
||||
|
||||
def _log_generation_text(
|
||||
self,
|
||||
character_name: str,
|
||||
text_content: str,
|
||||
language: str,
|
||||
voice_ref: Dict[str, Any],
|
||||
parameter_log: str = "",
|
||||
show_text_content: bool = True,
|
||||
) -> None:
|
||||
voice_note = self._voice_log_note(voice_ref)
|
||||
display_name = resolved_character_label(character_name, voice_ref)
|
||||
print(f"🎭 OmniVoice - Generating for '{display_name}' (Language: {language}){voice_note}:")
|
||||
if parameter_log:
|
||||
print(f"🎛️ OmniVoice params: {parameter_log}")
|
||||
if show_text_content:
|
||||
print(format_resolved_character_block(character_name, text_content, voice_ref))
|
||||
|
||||
@staticmethod
|
||||
def _format_parameter_log(
|
||||
filtered_params: Dict[str, Any],
|
||||
current_config: Dict[str, Any],
|
||||
current_seed: int,
|
||||
) -> str:
|
||||
if not filtered_params:
|
||||
return ""
|
||||
|
||||
ordered_parts = []
|
||||
if "seed" in filtered_params:
|
||||
ordered_parts.append(f"seed={current_seed}")
|
||||
|
||||
for key in (
|
||||
"num_steps",
|
||||
"guidance_scale",
|
||||
"t_shift",
|
||||
"speed",
|
||||
"duration",
|
||||
"layer_penalty_factor",
|
||||
"position_temperature",
|
||||
"class_temperature",
|
||||
"audio_chunk_duration",
|
||||
"audio_chunk_threshold",
|
||||
):
|
||||
if key in filtered_params:
|
||||
ordered_parts.append(f"{key}={current_config.get(key)}")
|
||||
|
||||
if "language" in filtered_params:
|
||||
ordered_parts.append(f"language={current_config.get('language')}")
|
||||
|
||||
remaining_keys = [
|
||||
key for key in filtered_params.keys()
|
||||
if key not in {
|
||||
"seed",
|
||||
"num_steps",
|
||||
"guidance_scale",
|
||||
"t_shift",
|
||||
"speed",
|
||||
"duration",
|
||||
"layer_penalty_factor",
|
||||
"position_temperature",
|
||||
"class_temperature",
|
||||
"audio_chunk_duration",
|
||||
"audio_chunk_threshold",
|
||||
"language",
|
||||
}
|
||||
]
|
||||
for key in remaining_keys:
|
||||
ordered_parts.append(f"{key}={current_config.get(key, filtered_params[key])}")
|
||||
|
||||
return ", ".join(ordered_parts)
|
||||
|
||||
def _setup_character_parser(self, text: str):
|
||||
configured_language = str(self.config.get("language", "Auto") or "Auto").strip()
|
||||
if configured_language.lower() in {"auto", "none"}:
|
||||
resolved_language = "en"
|
||||
else:
|
||||
resolved_language = configured_language.lower()
|
||||
|
||||
character_parser.language_resolver.default_language = resolved_language
|
||||
character_parser.default_language = resolved_language
|
||||
|
||||
character_tags = re.findall(r"\[([^\]]+)\]", text or "")
|
||||
characters_from_tags = []
|
||||
for tag in character_tags:
|
||||
if not tag.startswith("pause:"):
|
||||
characters_from_tags.append(tag.split("|")[0].strip())
|
||||
|
||||
available_chars = get_available_characters()
|
||||
character_aliases = voice_discovery.get_character_aliases()
|
||||
|
||||
all_available = set()
|
||||
if available_chars:
|
||||
all_available.update(available_chars)
|
||||
for alias, target in character_aliases.items():
|
||||
all_available.add(alias.lower())
|
||||
all_available.add(target.lower())
|
||||
for char in characters_from_tags:
|
||||
all_available.add(char.lower())
|
||||
all_available.add("narrator")
|
||||
|
||||
character_parser.set_available_characters(list(all_available))
|
||||
|
||||
char_lang_defaults = voice_discovery.get_character_language_defaults()
|
||||
for char, lang in char_lang_defaults.items():
|
||||
character_parser.set_character_language_default(char, lang)
|
||||
|
||||
character_parser.reset_session_cache()
|
||||
|
||||
@staticmethod
|
||||
def _should_apply_segment_language(seg: Any, base_config: Dict[str, Any]) -> bool:
|
||||
segment_language = getattr(seg, "language", None)
|
||||
if not segment_language:
|
||||
return False
|
||||
|
||||
if getattr(seg, "explicit_language", False):
|
||||
return True
|
||||
|
||||
global_language = str(base_config.get("language", "Auto") or "Auto").strip().lower()
|
||||
parser_default = str(character_parser.default_language or "en").strip().lower()
|
||||
segment_language_code = str(segment_language).strip().lower()
|
||||
|
||||
if segment_language_code == parser_default:
|
||||
return False
|
||||
return segment_language_code != global_language
|
||||
|
||||
@staticmethod
|
||||
def _text_weight(text: str) -> int:
|
||||
compact = re.sub(r"\s+", "", text or "")
|
||||
return max(len(compact), 1)
|
||||
|
||||
def _build_generation_items(
|
||||
self,
|
||||
text: str,
|
||||
voice_mapping: Dict[str, Any],
|
||||
seed: int,
|
||||
show_text_logging: bool = True,
|
||||
) -> Tuple[List[Dict[str, Any]], Dict[str, Any]]:
|
||||
self._setup_character_parser(text)
|
||||
|
||||
narrator_info = voice_mapping.get("narrator", {})
|
||||
narrator_voice = narrator_info if isinstance(narrator_info, dict) else {"audio": narrator_info}
|
||||
|
||||
base_config = self.config.copy()
|
||||
segment_objects = character_parser.parse_text_segments(text, engine_type="omnivoice")
|
||||
if not segment_objects:
|
||||
segment_objects = character_parser.parse_text_segments("narrator " + text, engine_type="omnivoice")
|
||||
|
||||
characters = list({seg.character for seg in segment_objects if seg.character})
|
||||
character_mapping = get_character_mapping(characters, engine_type="audio_only")
|
||||
|
||||
generation_items: List[Dict[str, Any]] = []
|
||||
native_chunking_notice_shown = False
|
||||
|
||||
for seg in segment_objects:
|
||||
self._check_interrupt()
|
||||
segment_text = (seg.text or "").strip()
|
||||
if not segment_text:
|
||||
continue
|
||||
segment_text = convert_omnivoice_special_tags(segment_text)
|
||||
|
||||
segment_params = seg.parameters if seg.parameters else {}
|
||||
current_config = base_config.copy()
|
||||
current_seed = seed
|
||||
filtered_params: Dict[str, Any] = {}
|
||||
if segment_params:
|
||||
filtered_params = ParameterValidator.filter_parameters_for_engine(segment_params, "omnivoice")
|
||||
current_config = apply_segment_parameters(base_config, segment_params, "omnivoice")
|
||||
if "seed" in current_config:
|
||||
current_seed = int(current_config.get("seed", seed))
|
||||
|
||||
if self._should_apply_segment_language(seg, base_config):
|
||||
current_config = current_config.copy()
|
||||
current_config["language"] = seg.language
|
||||
print(f" 🌍 OmniVoice language switched to: {seg.language}")
|
||||
|
||||
segment_language = str(current_config.get("language", "Auto") or "Auto")
|
||||
parameter_log = self._format_parameter_log(filtered_params, current_config, current_seed)
|
||||
|
||||
voice_ref = narrator_voice.copy()
|
||||
char_name = seg.character or "narrator"
|
||||
if char_name != "narrator" and char_name in voice_mapping:
|
||||
char_info = voice_mapping[char_name]
|
||||
if isinstance(char_info, dict):
|
||||
voice_ref = char_info.copy()
|
||||
elif char_name != "narrator":
|
||||
audio_path, ref_text = character_mapping.get(char_name, (None, None))
|
||||
if audio_path:
|
||||
voice_ref = {
|
||||
"audio_path": audio_path,
|
||||
"reference_text": ref_text or "",
|
||||
}
|
||||
|
||||
if not native_chunking_notice_shown and len(segment_text) > 400:
|
||||
print(
|
||||
"ℹ️ OmniVoice uses its official native long-form chunking "
|
||||
"(audio_chunk_duration / audio_chunk_threshold) instead of suite char-based chunk splitting."
|
||||
)
|
||||
native_chunking_notice_shown = True
|
||||
|
||||
explicit_duration = None
|
||||
if "duration" in filtered_params:
|
||||
duration_value = float(current_config.get("duration", 0.0))
|
||||
if duration_value > 0:
|
||||
explicit_duration = duration_value
|
||||
|
||||
def _append_text_item(text_content: str):
|
||||
text_content = (text_content or "").strip()
|
||||
if not text_content:
|
||||
return
|
||||
generation_items.append(
|
||||
{
|
||||
"kind": "text",
|
||||
"text": text_content,
|
||||
"config": current_config.copy(),
|
||||
"seed": current_seed,
|
||||
"voice_ref": voice_ref.copy() if isinstance(voice_ref, dict) else voice_ref,
|
||||
"character_name": char_name,
|
||||
"language": segment_language,
|
||||
"parameter_log": parameter_log,
|
||||
"show_text_logging": show_text_logging,
|
||||
"explicit_target_duration": explicit_duration,
|
||||
}
|
||||
)
|
||||
|
||||
if PauseTagProcessor.has_pause_tags(segment_text):
|
||||
raw_pause_segments, _ = PauseTagProcessor.parse_pause_tags(segment_text)
|
||||
for frag_type, frag_content in raw_pause_segments:
|
||||
if frag_type == "text":
|
||||
_append_text_item(frag_content)
|
||||
elif frag_type == "pause":
|
||||
generation_items.append(
|
||||
{
|
||||
"kind": "pause",
|
||||
"duration": float(frag_content),
|
||||
"effective_duration": float(frag_content),
|
||||
"text": f"[pause:{float(frag_content):.3f}s]",
|
||||
}
|
||||
)
|
||||
else:
|
||||
_append_text_item(segment_text)
|
||||
|
||||
return generation_items, base_config
|
||||
|
||||
def _apply_duration_budget(
|
||||
self,
|
||||
generation_items: List[Dict[str, Any]],
|
||||
total_duration: float,
|
||||
) -> None:
|
||||
if total_duration <= 0:
|
||||
return
|
||||
|
||||
pause_items = [item for item in generation_items if item["kind"] == "pause"]
|
||||
spoken_items = [item for item in generation_items if item["kind"] == "text"]
|
||||
variable_spoken_items = [item for item in spoken_items if not item.get("explicit_target_duration")]
|
||||
explicit_spoken_items = [item for item in spoken_items if item.get("explicit_target_duration")]
|
||||
|
||||
total_pause_duration = sum(float(item.get("duration", 0.0)) for item in pause_items)
|
||||
total_explicit_speech = sum(float(item.get("explicit_target_duration", 0.0)) for item in explicit_spoken_items)
|
||||
|
||||
desired_variable_speech = min(
|
||||
max(total_duration - total_explicit_speech, 0.0),
|
||||
self.MIN_AUTO_SPEECH_FRAGMENT_DURATION * len(variable_spoken_items),
|
||||
)
|
||||
max_pause_budget = max(0.0, total_duration - total_explicit_speech - desired_variable_speech)
|
||||
|
||||
pause_scale = 1.0
|
||||
if total_pause_duration > 0 and total_pause_duration > max_pause_budget:
|
||||
pause_scale = max_pause_budget / total_pause_duration if max_pause_budget > 0 else 0.0
|
||||
print(
|
||||
"⚠️ OmniVoice SRT: Pause tags exceed available subtitle time. "
|
||||
f"Compressing pauses from {total_pause_duration:.3f}s to {total_pause_duration * pause_scale:.3f}s."
|
||||
)
|
||||
|
||||
adjusted_pause_total = total_pause_duration * pause_scale
|
||||
for pause_item in pause_items:
|
||||
pause_item["effective_duration"] = float(pause_item.get("duration", 0.0)) * pause_scale
|
||||
|
||||
remaining_variable_speech = total_duration - total_explicit_speech - adjusted_pause_total
|
||||
if remaining_variable_speech <= 0 and variable_spoken_items:
|
||||
fallback_duration = max(0.05, min(total_duration, self.MIN_AUTO_SPEECH_FRAGMENT_DURATION))
|
||||
remaining_variable_speech = fallback_duration * len(variable_spoken_items)
|
||||
print(
|
||||
"⚠️ OmniVoice SRT: Subtitle timing leaves no room for auto-allocated speech after pauses/fixed durations. "
|
||||
f"Using minimal speech targets of {fallback_duration:.3f}s per spoken fragment."
|
||||
)
|
||||
|
||||
total_weight = sum(self._text_weight(item["text"]) for item in variable_spoken_items)
|
||||
allocated_total = 0.0
|
||||
for index, item in enumerate(variable_spoken_items):
|
||||
if total_weight <= 0:
|
||||
target_duration = remaining_variable_speech / len(variable_spoken_items)
|
||||
elif index == len(variable_spoken_items) - 1:
|
||||
target_duration = max(0.0, remaining_variable_speech - allocated_total)
|
||||
else:
|
||||
target_duration = remaining_variable_speech * (self._text_weight(item["text"]) / total_weight)
|
||||
allocated_total += target_duration
|
||||
item["allocated_target_duration"] = max(0.0, float(target_duration))
|
||||
|
||||
for item in explicit_spoken_items:
|
||||
item["allocated_target_duration"] = float(item["explicit_target_duration"])
|
||||
|
||||
def process_text(
|
||||
self,
|
||||
text: str,
|
||||
voice_mapping: Dict[str, Any],
|
||||
seed: int,
|
||||
enable_chunking: bool = True,
|
||||
max_chars_per_chunk: int = 400,
|
||||
chunk_combination_method: str = "auto",
|
||||
silence_between_chunks_ms: int = 100,
|
||||
enable_audio_cache: bool = True,
|
||||
apply_edit_postprocessing: bool = True,
|
||||
show_text_logging: bool = True,
|
||||
duration_budget: Optional[float] = None,
|
||||
) -> List[Dict[str, Any]]:
|
||||
del enable_chunking, max_chars_per_chunk # OmniVoice uses native duration-based chunking
|
||||
del apply_edit_postprocessing # OmniVoice uses native inline tags instead of Step Audio EditX inline post-processing
|
||||
generation_items, _base_config = self._build_generation_items(
|
||||
text=text,
|
||||
voice_mapping=voice_mapping,
|
||||
seed=seed,
|
||||
show_text_logging=show_text_logging,
|
||||
)
|
||||
|
||||
if duration_budget is not None and duration_budget > 0:
|
||||
self._apply_duration_budget(generation_items, float(duration_budget))
|
||||
|
||||
segment_records: List[Dict[str, Any]] = []
|
||||
for item_index, item in enumerate(generation_items):
|
||||
self._check_interrupt(
|
||||
item_index=item_index,
|
||||
total_items=len(generation_items),
|
||||
character_name=item.get("character_name"),
|
||||
)
|
||||
if item["kind"] == "pause":
|
||||
silence_duration = float(item.get("effective_duration", item.get("duration", 0.0)))
|
||||
silence = PauseTagProcessor.create_silence_segment(
|
||||
silence_duration,
|
||||
self.SAMPLE_RATE,
|
||||
torch.device("cpu"),
|
||||
torch.float32,
|
||||
)
|
||||
if silence.dim() == 1:
|
||||
silence = silence.unsqueeze(0)
|
||||
segment_records.append(
|
||||
{
|
||||
"waveform": silence,
|
||||
"sample_rate": self.SAMPLE_RATE,
|
||||
"text": f"[pause:{silence_duration:.3f}s]",
|
||||
}
|
||||
)
|
||||
continue
|
||||
|
||||
current_config = item["config"].copy()
|
||||
allocated_target_duration = item.get("allocated_target_duration")
|
||||
if allocated_target_duration is not None and allocated_target_duration > 0:
|
||||
current_config["duration"] = float(allocated_target_duration)
|
||||
|
||||
self.adapter.update_config(current_config)
|
||||
self._log_generation_text(
|
||||
character_name=item["character_name"],
|
||||
text_content=item["text"],
|
||||
language=item["language"],
|
||||
voice_ref=item["voice_ref"],
|
||||
parameter_log=item["parameter_log"],
|
||||
show_text_content=item["show_text_logging"],
|
||||
)
|
||||
self._check_interrupt(
|
||||
item_index=item_index,
|
||||
total_items=len(generation_items),
|
||||
character_name=item.get("character_name"),
|
||||
)
|
||||
audio = self.adapter.generate_single(
|
||||
text=item["text"],
|
||||
voice_ref=item["voice_ref"],
|
||||
seed=item["seed"],
|
||||
enable_audio_cache=enable_audio_cache,
|
||||
character_name=item["character_name"],
|
||||
)
|
||||
if audio.dim() == 1:
|
||||
audio = audio.unsqueeze(0)
|
||||
segment_records.append(
|
||||
{
|
||||
"waveform": audio.cpu(),
|
||||
"sample_rate": self.SAMPLE_RATE,
|
||||
"text": item["text"],
|
||||
}
|
||||
)
|
||||
|
||||
return segment_records
|
||||
|
||||
def combine_audio_segments(
|
||||
self,
|
||||
segments: List[Dict[str, Any]],
|
||||
method: str = "auto",
|
||||
silence_ms: int = 100,
|
||||
original_text: str = "",
|
||||
return_info: bool = False,
|
||||
) -> Union[torch.Tensor, Tuple[torch.Tensor, Dict[str, Any]]]:
|
||||
if not segments:
|
||||
empty = torch.zeros(0, dtype=torch.float32)
|
||||
if return_info:
|
||||
return empty, {}
|
||||
return empty
|
||||
|
||||
audio_segments = [seg["waveform"] for seg in segments]
|
||||
text_chunks = [seg.get("text", "") for seg in segments]
|
||||
text_length = len(" ".join(text_chunks))
|
||||
|
||||
combined_audio, chunk_info = ChunkTimingHelper.combine_audio_with_timing(
|
||||
audio_segments=audio_segments,
|
||||
combination_method=method,
|
||||
silence_ms=silence_ms,
|
||||
crossfade_duration=0.1,
|
||||
sample_rate=self.SAMPLE_RATE,
|
||||
text_length=text_length,
|
||||
original_text=original_text,
|
||||
text_chunks=text_chunks,
|
||||
)
|
||||
|
||||
if return_info:
|
||||
return combined_audio, chunk_info
|
||||
return combined_audio
|
||||
@@ -0,0 +1,462 @@
|
||||
"""
|
||||
OmniVoice SRT processor.
|
||||
|
||||
Handles full SRT subtitle processing for OmniVoice.
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
import os
|
||||
import sys
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
import comfy.model_management as model_management
|
||||
import torch
|
||||
|
||||
current_dir = os.path.dirname(__file__)
|
||||
nodes_dir = os.path.dirname(current_dir)
|
||||
project_root = os.path.dirname(nodes_dir)
|
||||
if project_root not in sys.path:
|
||||
sys.path.insert(0, project_root)
|
||||
|
||||
from engines.adapters.omnivoice_adapter import OmniVoiceEngineAdapter
|
||||
|
||||
|
||||
class OmniVoiceSRTProcessor:
|
||||
"""Complete SRT processor for OmniVoice."""
|
||||
|
||||
SAMPLE_RATE = 24000
|
||||
|
||||
def __init__(self, node_instance, config: Dict[str, Any]):
|
||||
self.node_instance = node_instance
|
||||
self.config = config.copy() if config else {}
|
||||
self.adapter = OmniVoiceEngineAdapter(self.config)
|
||||
self._tts_processor = None
|
||||
self.srt_available = False
|
||||
self.SRTParser = None
|
||||
self.SRTSubtitle = None
|
||||
self.SRTParseError = None
|
||||
self._load_srt_modules()
|
||||
|
||||
def _load_srt_modules(self):
|
||||
try:
|
||||
from utils.system.import_manager import import_manager
|
||||
|
||||
success, modules, msg = import_manager.import_srt_modules()
|
||||
if not success:
|
||||
print(f"⚠️ OmniVoice SRT: SRT module not available: {msg}")
|
||||
return
|
||||
|
||||
self.SRTParser = modules.get("SRTParser")
|
||||
self.SRTSubtitle = modules.get("SRTSubtitle")
|
||||
self.SRTParseError = modules.get("SRTParseError")
|
||||
if self.SRTParser is None:
|
||||
print("⚠️ OmniVoice SRT: SRT parser not available")
|
||||
return
|
||||
self.srt_available = True
|
||||
except Exception as e:
|
||||
print(f"⚠️ OmniVoice SRT: Failed to load SRT modules: {e}")
|
||||
|
||||
@property
|
||||
def processor(self):
|
||||
if self._tts_processor is None:
|
||||
processor_path = os.path.join(current_dir, "omnivoice_processor.py")
|
||||
spec = importlib.util.spec_from_file_location("omnivoice_processor_module", processor_path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
self._tts_processor = module.OmniVoiceProcessor(self.adapter, self.config)
|
||||
return self._tts_processor
|
||||
|
||||
def update_config(self, new_config: Dict[str, Any]):
|
||||
self.config = new_config.copy() if new_config else {}
|
||||
self.adapter.update_config(self.config)
|
||||
if self._tts_processor is not None:
|
||||
self._tts_processor.update_config(self.config)
|
||||
|
||||
@staticmethod
|
||||
def _annotate_duration_targeting(
|
||||
adjustment: Dict[str, Any],
|
||||
target_duration: float,
|
||||
natural_duration: float,
|
||||
use_native_duration_targeting: bool,
|
||||
) -> None:
|
||||
adjustment["native_duration_targeting_enabled"] = bool(use_native_duration_targeting)
|
||||
if use_native_duration_targeting and target_duration > 0:
|
||||
adjustment["native_requested_duration"] = float(target_duration)
|
||||
adjustment["pre_timing_generated_duration"] = float(natural_duration)
|
||||
|
||||
def process_srt_content(
|
||||
self,
|
||||
srt_content: str,
|
||||
voice_mapping: Dict[str, Any],
|
||||
seed: int,
|
||||
timing_mode: str,
|
||||
timing_params: Dict[str, Any],
|
||||
enable_audio_cache: bool = True,
|
||||
) -> Tuple[Dict[str, Any], str, str, str]:
|
||||
if not self.srt_available:
|
||||
raise ImportError("SRT support not available - missing required SRT parser")
|
||||
|
||||
self._check_interrupt()
|
||||
srt_parser = self.SRTParser()
|
||||
subtitles = srt_parser.parse_srt_content(srt_content, allow_overlaps=True)
|
||||
|
||||
from utils.timing.overlap_detection import SRTOverlapHandler
|
||||
|
||||
has_overlaps = SRTOverlapHandler.detect_overlaps(subtitles)
|
||||
current_timing_mode, mode_switched = SRTOverlapHandler.handle_smart_natural_fallback(
|
||||
timing_mode, has_overlaps, "OmniVoice SRT"
|
||||
)
|
||||
|
||||
use_native_duration_targeting = bool(timing_params.get("use_native_duration_targeting", False))
|
||||
if use_native_duration_targeting:
|
||||
print("🎯 OmniVoice SRT: Native duration targeting enabled")
|
||||
|
||||
print(f"🚀 OmniVoice SRT: Processing {len(subtitles)} subtitles in {current_timing_mode} mode")
|
||||
audio_segments, adjustments = self._process_all_subtitles(
|
||||
subtitles,
|
||||
voice_mapping,
|
||||
seed,
|
||||
use_native_duration_targeting=use_native_duration_targeting,
|
||||
enable_audio_cache=enable_audio_cache,
|
||||
)
|
||||
self._check_interrupt()
|
||||
|
||||
final_audio, final_adjustments, stretch_method = self._assemble_final_audio(
|
||||
audio_segments,
|
||||
subtitles,
|
||||
current_timing_mode,
|
||||
timing_params,
|
||||
adjustments,
|
||||
)
|
||||
if final_adjustments is not None:
|
||||
if use_native_duration_targeting:
|
||||
for final_adj, original_adj in zip(final_adjustments, adjustments):
|
||||
self._annotate_duration_targeting(
|
||||
final_adj,
|
||||
original_adj.get("target_duration", 0.0),
|
||||
original_adj.get("natural_duration", 0.0),
|
||||
True,
|
||||
)
|
||||
adjustments = final_adjustments
|
||||
|
||||
timing_report = self._generate_timing_report(
|
||||
subtitles,
|
||||
adjustments,
|
||||
current_timing_mode,
|
||||
has_overlaps,
|
||||
mode_switched,
|
||||
timing_mode if mode_switched else None,
|
||||
stretch_method,
|
||||
)
|
||||
adjusted_srt_string = self._generate_adjusted_srt_string(subtitles, adjustments, current_timing_mode)
|
||||
|
||||
total_duration = final_audio.shape[-1] / self.SAMPLE_RATE
|
||||
mode_info = current_timing_mode
|
||||
if mode_switched:
|
||||
mode_info = f"{current_timing_mode} (switched from {timing_mode} due to overlaps)"
|
||||
info = (
|
||||
f"Generated {total_duration:.1f}s OmniVoice SRT-timed audio from {len(subtitles)} subtitles "
|
||||
f"using {mode_info} mode"
|
||||
)
|
||||
if use_native_duration_targeting:
|
||||
info += " with native duration targeting"
|
||||
|
||||
if final_audio.dim() == 1:
|
||||
final_audio = final_audio.unsqueeze(0).unsqueeze(0)
|
||||
elif final_audio.dim() == 2:
|
||||
final_audio = final_audio.unsqueeze(0)
|
||||
|
||||
audio_output = {"waveform": final_audio, "sample_rate": self.SAMPLE_RATE}
|
||||
return audio_output, info, timing_report, adjusted_srt_string
|
||||
|
||||
def _check_interrupt(
|
||||
self,
|
||||
subtitle_index: Optional[int] = None,
|
||||
total_subtitles: Optional[int] = None,
|
||||
character: Optional[str] = None,
|
||||
):
|
||||
if model_management.interrupt_processing:
|
||||
if subtitle_index is not None and total_subtitles is not None:
|
||||
if character:
|
||||
raise InterruptedError(
|
||||
f"OmniVoice SRT generation interrupted at subtitle "
|
||||
f"{subtitle_index + 1}/{total_subtitles}, character '{character}'"
|
||||
)
|
||||
raise InterruptedError(
|
||||
f"OmniVoice SRT generation interrupted at subtitle {subtitle_index + 1}/{total_subtitles}"
|
||||
)
|
||||
raise InterruptedError("OmniVoice SRT generation interrupted by user")
|
||||
|
||||
def _process_all_subtitles(
|
||||
self,
|
||||
subtitles: List,
|
||||
voice_mapping: Dict[str, Any],
|
||||
seed: int,
|
||||
use_native_duration_targeting: bool = False,
|
||||
enable_audio_cache: bool = True,
|
||||
) -> Tuple[List[torch.Tensor], List[Dict]]:
|
||||
audio_segments: List[torch.Tensor] = []
|
||||
adjustments: List[Dict[str, Any]] = []
|
||||
all_segments_for_editing: List[Dict[str, Any]] = []
|
||||
try:
|
||||
for i, sub in enumerate(subtitles):
|
||||
self._check_interrupt(i, len(subtitles))
|
||||
|
||||
text = sub.text.strip()
|
||||
if not text:
|
||||
target_duration = sub.duration
|
||||
silence = torch.zeros(1, int(target_duration * self.SAMPLE_RATE))
|
||||
audio_segments.append(silence)
|
||||
adjustments.append(
|
||||
{
|
||||
"index": i,
|
||||
"segment_index": i,
|
||||
"sequence": sub.sequence,
|
||||
"natural_duration": target_duration,
|
||||
"target_start": sub.start_time,
|
||||
"target_end": sub.end_time,
|
||||
"target_duration": target_duration,
|
||||
"start_time": sub.start_time,
|
||||
"end_time": sub.end_time,
|
||||
"stretch_factor": 1.0,
|
||||
"needs_stretching": False,
|
||||
"stretch_type": "none",
|
||||
"adjustment": 0.0,
|
||||
"adjusted_start": sub.start_time,
|
||||
"adjusted_end": sub.end_time,
|
||||
"adjusted_duration": target_duration,
|
||||
}
|
||||
)
|
||||
continue
|
||||
|
||||
print(f"📖 OmniVoice SRT Subtitle {i + 1}/{len(subtitles)}")
|
||||
if use_native_duration_targeting and sub.duration > 0:
|
||||
print(f" 🎯 Target duration: {sub.duration:.3f}s")
|
||||
segment_records = self.processor.process_text(
|
||||
text=text,
|
||||
voice_mapping=voice_mapping,
|
||||
seed=seed + i,
|
||||
enable_chunking=False,
|
||||
enable_audio_cache=enable_audio_cache,
|
||||
apply_edit_postprocessing=False,
|
||||
duration_budget=sub.duration if use_native_duration_targeting else None,
|
||||
)
|
||||
|
||||
subtitle_has_edit_tags = any(seg.get("edit_tags") for seg in segment_records)
|
||||
if subtitle_has_edit_tags:
|
||||
for seg_idx, seg in enumerate(segment_records):
|
||||
waveform = seg["waveform"]
|
||||
if waveform.dim() == 1:
|
||||
waveform = waveform.unsqueeze(0)
|
||||
elif waveform.dim() == 3:
|
||||
waveform = waveform.squeeze(0)
|
||||
all_segments_for_editing.append(
|
||||
{
|
||||
**seg,
|
||||
"waveform": waveform.cpu(),
|
||||
"subtitle_index": i,
|
||||
"segment_index": seg_idx,
|
||||
}
|
||||
)
|
||||
audio_segments.append(None)
|
||||
else:
|
||||
if len(segment_records) > 1:
|
||||
audio, _ = self.processor.combine_audio_segments(
|
||||
segments=segment_records,
|
||||
method="auto",
|
||||
silence_ms=0,
|
||||
original_text=text,
|
||||
return_info=True,
|
||||
)
|
||||
elif segment_records:
|
||||
audio = segment_records[0]["waveform"]
|
||||
else:
|
||||
audio = torch.zeros(1, 0)
|
||||
|
||||
if audio.dim() == 1:
|
||||
audio = audio.unsqueeze(0)
|
||||
elif audio.dim() == 3:
|
||||
audio = audio.squeeze(0)
|
||||
audio_segments.append(audio.cpu())
|
||||
|
||||
target_start = sub.start_time
|
||||
target_end = sub.end_time
|
||||
target_duration = target_end - target_start
|
||||
adjustments.append(
|
||||
{
|
||||
"index": i,
|
||||
"segment_index": i,
|
||||
"sequence": sub.sequence,
|
||||
"natural_duration": 0.0,
|
||||
"target_start": target_start,
|
||||
"target_end": target_end,
|
||||
"target_duration": target_duration,
|
||||
"start_time": target_start,
|
||||
"end_time": target_end,
|
||||
"stretch_factor": 1.0,
|
||||
"needs_stretching": False,
|
||||
"stretch_type": "none",
|
||||
"adjustment": 0.0,
|
||||
"adjusted_start": target_start,
|
||||
"adjusted_end": target_end,
|
||||
"adjusted_duration": 0.0,
|
||||
}
|
||||
)
|
||||
|
||||
if not subtitle_has_edit_tags:
|
||||
audio = audio_segments[i]
|
||||
natural_duration = audio.shape[-1] / self.SAMPLE_RATE
|
||||
stretch_factor = target_duration / natural_duration if natural_duration > 0 else 1.0
|
||||
adjustments[i].update(
|
||||
{
|
||||
"natural_duration": natural_duration,
|
||||
"stretch_factor": stretch_factor,
|
||||
"needs_stretching": abs(stretch_factor - 1.0) > 0.05,
|
||||
"stretch_type": "compress" if stretch_factor < 1.0 else "expand" if stretch_factor > 1.0 else "none",
|
||||
"adjustment": natural_duration - target_duration,
|
||||
"adjusted_duration": natural_duration,
|
||||
}
|
||||
)
|
||||
self._annotate_duration_targeting(
|
||||
adjustments[i],
|
||||
target_duration,
|
||||
natural_duration,
|
||||
use_native_duration_targeting,
|
||||
)
|
||||
print(f"✅ OmniVoice Subtitle {i + 1}/{len(subtitles)}: {natural_duration:.3f}s (expected {target_duration:.3f}s)")
|
||||
else:
|
||||
print(f"✅ OmniVoice Subtitle {i + 1}/{len(subtitles)}: queued for batch edit processing")
|
||||
|
||||
if all_segments_for_editing:
|
||||
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
|
||||
|
||||
processed_segments = apply_edit_post_processing(all_segments_for_editing, engine_config=self.config)
|
||||
grouped_segments: Dict[int, List[Dict[str, Any]]] = {}
|
||||
for seg in processed_segments:
|
||||
grouped_segments.setdefault(seg["subtitle_index"], []).append(seg)
|
||||
|
||||
for subtitle_index, segments in grouped_segments.items():
|
||||
segments = sorted(segments, key=lambda item: item.get("segment_index", 0))
|
||||
if len(segments) > 1:
|
||||
combined_audio, _ = self.processor.combine_audio_segments(
|
||||
segments=segments,
|
||||
method="auto",
|
||||
silence_ms=0,
|
||||
original_text=subtitles[subtitle_index].text,
|
||||
return_info=True,
|
||||
)
|
||||
else:
|
||||
combined_audio = segments[0]["waveform"]
|
||||
if combined_audio.dim() == 1:
|
||||
combined_audio = combined_audio.unsqueeze(0)
|
||||
elif combined_audio.dim() == 3:
|
||||
combined_audio = combined_audio.squeeze(0)
|
||||
combined_audio = combined_audio.cpu()
|
||||
audio_segments[subtitle_index] = combined_audio
|
||||
|
||||
target_duration = adjustments[subtitle_index]["target_duration"]
|
||||
natural_duration = combined_audio.shape[-1] / self.SAMPLE_RATE
|
||||
stretch_factor = target_duration / natural_duration if natural_duration > 0 else 1.0
|
||||
adjustments[subtitle_index].update(
|
||||
{
|
||||
"natural_duration": natural_duration,
|
||||
"stretch_factor": stretch_factor,
|
||||
"needs_stretching": abs(stretch_factor - 1.0) > 0.05,
|
||||
"stretch_type": "compress" if stretch_factor < 1.0 else "expand" if stretch_factor > 1.0 else "none",
|
||||
"adjustment": natural_duration - target_duration,
|
||||
"adjusted_duration": natural_duration,
|
||||
}
|
||||
)
|
||||
self._annotate_duration_targeting(
|
||||
adjustments[subtitle_index],
|
||||
target_duration,
|
||||
natural_duration,
|
||||
use_native_duration_targeting,
|
||||
)
|
||||
|
||||
return audio_segments, adjustments
|
||||
finally:
|
||||
self.processor.update_config(self.config)
|
||||
|
||||
def _assemble_final_audio(self, audio_segments, subtitles, timing_mode, timing_params, adjustments):
|
||||
if timing_mode == "stretch_to_fit":
|
||||
from engines.chatterbox.audio_timing import TimedAudioAssembler
|
||||
|
||||
assembler = TimedAudioAssembler(self.SAMPLE_RATE)
|
||||
target_timings = [(sub.start_time, sub.end_time) for sub in subtitles]
|
||||
fade_duration = timing_params.get("fade_for_StretchToFit", 0.01)
|
||||
final_audio, stretch_method_used = assembler.assemble_timed_audio(
|
||||
audio_segments,
|
||||
target_timings,
|
||||
fade_duration=fade_duration,
|
||||
)
|
||||
return final_audio, None, stretch_method_used
|
||||
|
||||
if timing_mode == "pad_with_silence":
|
||||
from utils.timing.assembly import AudioAssemblyEngine
|
||||
|
||||
assembler = AudioAssemblyEngine(self.SAMPLE_RATE)
|
||||
final_audio = assembler.assemble_with_overlaps(audio_segments, subtitles, torch.device("cpu"))
|
||||
return final_audio, None, None
|
||||
|
||||
if timing_mode == "concatenate":
|
||||
from utils.timing.assembly import AudioAssemblyEngine
|
||||
from utils.timing.engine import TimingEngine
|
||||
|
||||
timing_engine = TimingEngine(self.SAMPLE_RATE)
|
||||
assembler = AudioAssemblyEngine(self.SAMPLE_RATE)
|
||||
new_adjustments = timing_engine.calculate_concatenation_adjustments(audio_segments, subtitles)
|
||||
fade_duration = timing_params.get("fade_for_StretchToFit", 0.01)
|
||||
final_audio = assembler.assemble_concatenation(audio_segments, fade_duration)
|
||||
return final_audio, new_adjustments, None
|
||||
|
||||
from utils.timing.assembly import AudioAssemblyEngine
|
||||
from utils.timing.engine import TimingEngine
|
||||
|
||||
timing_engine = TimingEngine(self.SAMPLE_RATE)
|
||||
assembler = AudioAssemblyEngine(self.SAMPLE_RATE)
|
||||
smart_adjustments, processed_segments = timing_engine.calculate_smart_timing_adjustments(
|
||||
audio_segments,
|
||||
subtitles,
|
||||
timing_params.get("timing_tolerance", 2.0),
|
||||
timing_params.get("max_stretch_ratio", 1.0),
|
||||
timing_params.get("min_stretch_ratio", 0.5),
|
||||
torch.device("cpu"),
|
||||
)
|
||||
final_audio = assembler.assemble_smart_natural(
|
||||
audio_segments,
|
||||
processed_segments,
|
||||
smart_adjustments,
|
||||
subtitles,
|
||||
torch.device("cpu"),
|
||||
)
|
||||
return final_audio, smart_adjustments, None
|
||||
|
||||
def _generate_timing_report(
|
||||
self,
|
||||
subtitles,
|
||||
adjustments,
|
||||
current_timing_mode,
|
||||
has_overlaps=False,
|
||||
mode_switched=False,
|
||||
original_mode=None,
|
||||
stretch_method=None,
|
||||
) -> str:
|
||||
from utils.timing.reporting import SRTReportGenerator
|
||||
|
||||
reporter = SRTReportGenerator()
|
||||
return reporter.generate_timing_report(
|
||||
subtitles,
|
||||
adjustments,
|
||||
current_timing_mode,
|
||||
has_overlaps,
|
||||
mode_switched,
|
||||
original_mode,
|
||||
stretch_method,
|
||||
)
|
||||
|
||||
def _generate_adjusted_srt_string(self, subtitles, adjustments, timing_mode: str) -> str:
|
||||
from utils.timing.reporting import SRTReportGenerator
|
||||
|
||||
reporter = SRTReportGenerator()
|
||||
return reporter.generate_adjusted_srt_string(subtitles, adjustments, timing_mode)
|
||||
@@ -24,7 +24,11 @@ from utils.text.pause_processor import PauseTagProcessor
|
||||
from engines.adapters.qwen3_tts_adapter import Qwen3TTSEngineAdapter
|
||||
from utils.models.language_mapper import resolve_language_alias
|
||||
from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment
|
||||
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
|
||||
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
|
||||
from utils.voice.character_logging import (
|
||||
format_resolved_character_block,
|
||||
resolved_character_label,
|
||||
)
|
||||
|
||||
|
||||
class Qwen3TTSProcessor:
|
||||
@@ -523,7 +527,8 @@ class Qwen3TTSProcessor:
|
||||
combined_text_clean, combined_text_edit_tags = get_edit_tags_for_segment(combined_text)
|
||||
|
||||
chunks = self.chunker.split_into_chunks(combined_text_clean, max_chars)
|
||||
print(f"📝 Chunking {character}'s combined text into {len(chunks)} chunks (Language: {segment_lang}){voice_note}")
|
||||
display_name = resolved_character_label(character, voice_ref)
|
||||
print(f"📝 Chunking {display_name}'s combined text into {len(chunks)} chunks (Language: {segment_lang}){voice_note}")
|
||||
|
||||
for chunk_idx, chunk in enumerate(chunks):
|
||||
# Check for interruption during chunk processing
|
||||
@@ -573,10 +578,9 @@ class Qwen3TTSProcessor:
|
||||
# Extract inline edit tags BEFORE generation
|
||||
combined_text_clean, combined_text_edit_tags = get_edit_tags_for_segment(combined_text)
|
||||
|
||||
print(f"🎭 Qwen3-TTS - Generating for '{character}' (Language: {segment_lang}){voice_note}:")
|
||||
print("="*60)
|
||||
print(combined_text_clean)
|
||||
print("="*60)
|
||||
display_name = resolved_character_label(character, voice_ref)
|
||||
print(f"🎭 Qwen3-TTS - Generating for '{display_name}' (Language: {segment_lang}){voice_note}:")
|
||||
print(format_resolved_character_block(character, combined_text_clean, voice_ref))
|
||||
|
||||
# Set current segment for time tracking (skip in SRT mode - managed at subtitle level)
|
||||
if not self._srt_mode:
|
||||
|
||||
@@ -477,7 +477,7 @@ Test Text: {reference_text}
|
||||
|
||||
Generated by: Qwen3-TTS Voice Designer (VoiceDesign model)
|
||||
via TTS Audio Suite for ComfyUI
|
||||
https://github.com/FahimF/ComfyUI_TTS_Audio_Suite
|
||||
https://github.com/diodiogod/TTS-Audio-Suite
|
||||
"""
|
||||
metadata_path = os.path.join(voices_dir, f"{character_name}.txt")
|
||||
with open(metadata_path, 'w', encoding='utf-8') as f:
|
||||
|
||||
@@ -24,6 +24,10 @@ from utils.text.pause_processor import PauseTagProcessor
|
||||
from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment
|
||||
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
|
||||
from engines.adapters.step_audio_editx_adapter import StepAudioEditXEngineAdapter
|
||||
from utils.voice.character_logging import (
|
||||
format_resolved_character_block,
|
||||
resolved_character_label,
|
||||
)
|
||||
|
||||
|
||||
class StepAudioEditXProcessor:
|
||||
@@ -330,10 +334,9 @@ class StepAudioEditXProcessor:
|
||||
if edit_tags:
|
||||
print(f"🎨 Found {len(edit_tags)} edit tag(s) for post-processing")
|
||||
|
||||
print(f"🎭 Step Audio EditX - Generating for '{character}'{voice_note}:")
|
||||
print("="*60)
|
||||
print(clean_text)
|
||||
print("="*60)
|
||||
display_name = resolved_character_label(character, voice_ref)
|
||||
print(f"🎭 Step Audio EditX - Generating for '{display_name}'{voice_note}:")
|
||||
print(format_resolved_character_block(character, clean_text, voice_ref))
|
||||
|
||||
# Set current segment for time tracking (skip in SRT mode - managed at subtitle level)
|
||||
if not self._srt_mode:
|
||||
|
||||
@@ -21,7 +21,8 @@ if project_root not in sys.path:
|
||||
|
||||
from utils.system.import_manager import import_manager
|
||||
from utils.text.character_parser import character_parser
|
||||
from utils.voice.discovery import get_available_characters, voice_discovery
|
||||
from utils.voice.discovery import get_available_characters, voice_discovery
|
||||
from utils.voice.character_logging import resolved_character_label
|
||||
|
||||
# Import StepAudioEditXProcessor using file-based import (avoids package issues)
|
||||
import importlib.util
|
||||
@@ -199,7 +200,7 @@ class StepAudioEditXSRTProcessor:
|
||||
Returns:
|
||||
Complete voice mapping with all characters
|
||||
"""
|
||||
from utils.voice.discovery import get_character_mapping
|
||||
from utils.voice.discovery import get_character_mapping
|
||||
import re
|
||||
|
||||
# Extract all character names from subtitle text
|
||||
@@ -246,11 +247,11 @@ class StepAudioEditXSRTProcessor:
|
||||
'prompt_audio_path': audio_path,
|
||||
'prompt_text': ref_text
|
||||
}
|
||||
print(f"🎭 SRT: Using character voice for '{character}'")
|
||||
print(f"🎭 SRT: Using character voice for '{resolved_character_label(character, voice_mapping[character])}'")
|
||||
else:
|
||||
# Fallback to narrator voice
|
||||
voice_mapping[character] = narrator_ref
|
||||
print(f"🔄 SRT: Using narrator voice fallback for '{character}'")
|
||||
print(f"🔄 SRT: Using narrator voice fallback for '{resolved_character_label(character, narrator_ref)}'")
|
||||
|
||||
return voice_mapping
|
||||
|
||||
|
||||
+198
-32
@@ -131,12 +131,16 @@ Hello! This is unified SRT TTS with character switching.
|
||||
"step": 0.5,
|
||||
"tooltip": "Maximum allowed deviation (in seconds) for timing adjustments in 'smart_natural' mode. Higher values allow more flexibility."
|
||||
}),
|
||||
"batch_size": ("INT", {
|
||||
"default": 0, "min": 0, "max": 32, "step": 1,
|
||||
"tooltip": "Parallel processing workers. 0 = sequential (recommended), 2+ = streaming mode. Note: Streaming often slower than sequential mode. F5-TTS doesn't support streaming yet."
|
||||
}),
|
||||
}
|
||||
}
|
||||
"batch_size": ("INT", {
|
||||
"default": 0, "min": 0, "max": 32, "step": 1,
|
||||
"tooltip": "Parallel processing workers. 0 = sequential (recommended), 2+ = streaming mode. Note: Streaming often slower than sequential mode. F5-TTS doesn't support streaming yet."
|
||||
}),
|
||||
"use_native_duration_targeting": ("BOOLEAN", {
|
||||
"default": False,
|
||||
"tooltip": "When enabled, supported engines like OmniVoice use the model's native duration parameter for each subtitle during generation. This helps the model aim closer to the subtitle length before the suite applies final timing adjustment, which can reduce stretching/compression and sound more natural."
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("AUDIO", "STRING", "STRING", "STRING")
|
||||
RETURN_NAMES = ("audio", "generation_info", "timing_report", "Adjusted_SRT")
|
||||
@@ -208,6 +212,18 @@ Hello! This is unified SRT TTS with character switching.
|
||||
stable_params['optimize'] = config.get('optimize', False)
|
||||
stable_params['max_generate_length'] = config.get('max_generate_length', 500)
|
||||
|
||||
if engine_type == "fish_audio_s2":
|
||||
stable_params['model_variant'] = config.get('model_variant', 's2-pro')
|
||||
stable_params['precision'] = config.get('precision', 'bfloat16')
|
||||
stable_params['compile'] = config.get('compile', False)
|
||||
stable_params['quantization'] = config.get('quantization', 'none')
|
||||
stable_params['multi_speaker_mode'] = config.get('multi_speaker_mode', 'Native Multi-Speaker')
|
||||
stable_params['language_prompting'] = config.get('language_prompting', 'Auto Inline Tag')
|
||||
|
||||
if engine_type == "omnivoice":
|
||||
stable_params['model_variant'] = config.get('model_variant', 'OmniVoice')
|
||||
stable_params['dtype'] = config.get('dtype', 'auto')
|
||||
|
||||
if engine_type == "moss_tts":
|
||||
stable_params['model_variant'] = config.get('model_variant', 'MOSS-TTS-Local-Transformer')
|
||||
stable_params['multi_speaker_mode'] = config.get('multi_speaker_mode', 'Custom Character Switching')
|
||||
@@ -544,6 +560,79 @@ Hello! This is unified SRT TTS with character switching.
|
||||
}
|
||||
return engine_instance
|
||||
|
||||
elif engine_type == "fish_audio_s2":
|
||||
processor_path = os.path.join(nodes_dir, "fish_audio_s2", "fish_audio_s2_srt_processor.py")
|
||||
processor_spec = importlib.util.spec_from_file_location("fish_audio_s2_srt_processor_module", processor_path)
|
||||
processor_module = importlib.util.module_from_spec(processor_spec)
|
||||
processor_spec.loader.exec_module(processor_module)
|
||||
FishAudioS2SRTProcessor = processor_module.FishAudioS2SRTProcessor
|
||||
|
||||
class FishAudioS2SRTWrapper:
|
||||
def __init__(self, cfg):
|
||||
self.config = cfg.copy()
|
||||
self.processor = FishAudioS2SRTProcessor(self, self.config)
|
||||
|
||||
def update_config(self, new_config):
|
||||
self.config = new_config.copy()
|
||||
self.processor.update_config(new_config)
|
||||
|
||||
def check_interrupt(self):
|
||||
if model_management.interrupt_processing:
|
||||
raise InterruptedError("Fish Audio S2 SRT processing interrupted by user")
|
||||
|
||||
engine_instance = FishAudioS2SRTWrapper(config)
|
||||
import time
|
||||
self._cached_engine_instances[cache_key] = {
|
||||
'instance': engine_instance,
|
||||
'timestamp': time.time()
|
||||
}
|
||||
return engine_instance
|
||||
|
||||
elif engine_type == "omnivoice":
|
||||
omnivoice_srt_processor_path = os.path.join(nodes_dir, "omnivoice", "omnivoice_srt_processor.py")
|
||||
omnivoice_srt_spec = importlib.util.spec_from_file_location("omnivoice_srt_processor_module", omnivoice_srt_processor_path)
|
||||
omnivoice_srt_module = importlib.util.module_from_spec(omnivoice_srt_spec)
|
||||
omnivoice_srt_spec.loader.exec_module(omnivoice_srt_module)
|
||||
|
||||
OmniVoiceSRTProcessor = omnivoice_srt_module.OmniVoiceSRTProcessor
|
||||
|
||||
class OmniVoiceSRTWrapper:
|
||||
def __init__(self, config):
|
||||
self.config = config
|
||||
self.processor = OmniVoiceSRTProcessor(self, config)
|
||||
|
||||
def update_config(self, new_config):
|
||||
self.config = new_config.copy()
|
||||
self.processor.update_config(new_config)
|
||||
|
||||
def process_with_error_handling(self, func):
|
||||
try:
|
||||
return func()
|
||||
except Exception as e:
|
||||
raise e
|
||||
|
||||
def format_audio_output(self, audio_tensor, sample_rate):
|
||||
if audio_tensor.is_cuda:
|
||||
audio_tensor = audio_tensor.cpu()
|
||||
if audio_tensor.dim() == 1:
|
||||
audio_tensor = audio_tensor.unsqueeze(0).unsqueeze(0)
|
||||
elif audio_tensor.dim() == 2:
|
||||
audio_tensor = audio_tensor.unsqueeze(0)
|
||||
return {"waveform": audio_tensor, "sample_rate": sample_rate}
|
||||
|
||||
def check_interrupt(self):
|
||||
if model_management.interrupt_processing:
|
||||
raise InterruptedError("OmniVoice SRT processing interrupted by user")
|
||||
|
||||
engine_instance = OmniVoiceSRTWrapper(config)
|
||||
|
||||
import time
|
||||
self._cached_engine_instances[cache_key] = {
|
||||
'instance': engine_instance,
|
||||
'timestamp': time.time()
|
||||
}
|
||||
return engine_instance
|
||||
|
||||
elif engine_type == "vibevoice":
|
||||
# Import and create the VibeVoice SRT processor using absolute import
|
||||
vibevoice_srt_processor_path = os.path.join(nodes_dir, "vibevoice", "vibevoice_srt_processor.py")
|
||||
@@ -833,9 +922,11 @@ Hello! This is unified SRT TTS with character switching.
|
||||
else:
|
||||
raise ValueError(f"Unknown engine type: {engine_type}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to create engine SRT node instance: {e}")
|
||||
return None
|
||||
except Exception as e:
|
||||
if isinstance(e, InterruptedError):
|
||||
raise
|
||||
print(f"❌ Failed to create engine SRT node instance: {e}")
|
||||
return None
|
||||
|
||||
def _get_voice_reference(self, opt_narrator, narrator_voice: str):
|
||||
"""
|
||||
@@ -896,7 +987,10 @@ Hello! This is unified SRT TTS with character switching.
|
||||
reference_text = "" # No reference text available from direct audio
|
||||
|
||||
print(f"📺 TTS SRT: Using direct audio input ({character_name})")
|
||||
print(f"⚠️ TTS SRT: Direct audio input has no reference text - F5-TTS engines will fail")
|
||||
print(
|
||||
"⚠️ TTS SRT: Direct audio input has no reference text - "
|
||||
"F5-TTS and OmniVoice cloning will fail"
|
||||
)
|
||||
return None, audio_tensor, reference_text, character_name
|
||||
|
||||
# Priority 2: narrator_voice dropdown (fallback)
|
||||
@@ -921,15 +1015,17 @@ Hello! This is unified SRT TTS with character switching.
|
||||
print("⚠️ TTS SRT: No voice reference provided - this may cause issues with some engines")
|
||||
return None, None, "", "narrator"
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Voice reference error: {e}")
|
||||
return None, None, "", "narrator"
|
||||
except Exception as e:
|
||||
if isinstance(e, InterruptedError):
|
||||
raise
|
||||
print(f"❌ Voice reference error: {e}")
|
||||
return None, None, "", "narrator"
|
||||
|
||||
def generate_srt_speech(self, TTS_engine: Dict[str, Any], srt_content: str, narrator_voice: str,
|
||||
seed: int, timing_mode: str, opt_narrator=None, enable_audio_cache: bool = True,
|
||||
fade_for_StretchToFit: float = 0.01, max_stretch_ratio: float = 1.0,
|
||||
min_stretch_ratio: float = 0.5, timing_tolerance: float = 2.0,
|
||||
batch_size: int = 0):
|
||||
def generate_srt_speech(self, TTS_engine: Dict[str, Any], srt_content: str, narrator_voice: str,
|
||||
seed: int, timing_mode: str, opt_narrator=None, enable_audio_cache: bool = True,
|
||||
fade_for_StretchToFit: float = 0.01, max_stretch_ratio: float = 1.0,
|
||||
min_stretch_ratio: float = 0.5, timing_tolerance: float = 2.0,
|
||||
batch_size: int = 0, use_native_duration_targeting: bool = False):
|
||||
"""
|
||||
Generate SRT-timed speech using the selected TTS engine.
|
||||
This is a DELEGATION WRAPPER that preserves all original SRT functionality.
|
||||
@@ -942,11 +1038,12 @@ Hello! This is unified SRT TTS with character switching.
|
||||
timing_mode: How to align audio with SRT timings
|
||||
opt_narrator: Voice reference from Character Voices node
|
||||
enable_audio_cache: Enable audio caching
|
||||
fade_for_StretchToFit: Crossfade duration for stretch_to_fit mode
|
||||
max_stretch_ratio: Maximum stretch ratio for smart_natural mode
|
||||
min_stretch_ratio: Minimum stretch ratio for smart_natural mode
|
||||
timing_tolerance: Timing tolerance for smart_natural mode
|
||||
batch_size: Number of parallel workers (0=sequential, 1+=streaming parallel)
|
||||
fade_for_StretchToFit: Crossfade duration for stretch_to_fit mode
|
||||
max_stretch_ratio: Maximum stretch ratio for smart_natural mode
|
||||
min_stretch_ratio: Minimum stretch ratio for smart_natural mode
|
||||
timing_tolerance: Timing tolerance for smart_natural mode
|
||||
batch_size: Number of parallel workers (0=sequential, 1+=streaming parallel)
|
||||
use_native_duration_targeting: Inject subtitle durations into supported engines during generation
|
||||
|
||||
Returns:
|
||||
Tuple of (audio_tensor, generation_info, timing_report, adjusted_srt)
|
||||
@@ -971,20 +1068,34 @@ Hello! This is unified SRT TTS with character switching.
|
||||
if config.get('emotion_audio') is not None:
|
||||
print(f"🎭 Emotion control connected to SRT generation")
|
||||
|
||||
if not engine_type:
|
||||
raise ValueError("TTS engine missing engine_type")
|
||||
|
||||
print(f"📺 TTS SRT: Starting {engine_type} SRT generation")
|
||||
if not engine_type:
|
||||
raise ValueError("TTS engine missing engine_type")
|
||||
|
||||
print(f"📺 TTS SRT: Starting {engine_type} SRT generation")
|
||||
|
||||
native_duration_targeting_active = bool(
|
||||
use_native_duration_targeting and engine_type == "omnivoice"
|
||||
)
|
||||
if use_native_duration_targeting and not native_duration_targeting_active:
|
||||
print(
|
||||
f"ℹ️ Native duration targeting is currently supported only for OmniVoice in TTS SRT. "
|
||||
f"Ignoring it for engine '{engine_type}'."
|
||||
)
|
||||
|
||||
# Get voice reference (opt_narrator takes priority)
|
||||
audio_path, audio_tensor, reference_text, character_name = self._get_voice_reference(opt_narrator, narrator_voice)
|
||||
|
||||
# Validate F5-TTS requirements: must have reference text
|
||||
if engine_type == "f5tts" and not reference_text.strip():
|
||||
raise ValueError(
|
||||
"F5-TTS requires reference text. When using direct audio input, "
|
||||
"please use Character Voices node instead, which provides both audio and text."
|
||||
)
|
||||
if engine_type == "f5tts" and not reference_text.strip():
|
||||
raise ValueError(
|
||||
"F5-TTS requires reference text. When using direct audio input, "
|
||||
"please use Character Voices node instead, which provides both audio and text."
|
||||
)
|
||||
if engine_type == "omnivoice" and (audio_tensor is not None or audio_path) and not reference_text.strip():
|
||||
raise ValueError(
|
||||
"OmniVoice voice cloning requires reference text. "
|
||||
"Do not connect raw audio directly. Use Character Voices node or a narrator voice with a matching .reference.txt file."
|
||||
)
|
||||
|
||||
# Create proper engine SRT node instance to preserve ALL functionality
|
||||
engine_instance = self._create_proper_engine_node_instance(TTS_engine)
|
||||
@@ -1185,6 +1296,59 @@ Hello! This is unified SRT TTS with character switching.
|
||||
enable_audio_cache=enable_audio_cache
|
||||
)
|
||||
|
||||
elif engine_type == "fish_audio_s2":
|
||||
print(
|
||||
f" Audio cache: {'enabled' if enable_audio_cache else 'disabled'} "
|
||||
"(TTS SRT node)"
|
||||
)
|
||||
timing_params = {
|
||||
'fade_for_StretchToFit': fade_for_StretchToFit,
|
||||
'max_stretch_ratio': max_stretch_ratio,
|
||||
'min_stretch_ratio': min_stretch_ratio,
|
||||
'timing_tolerance': timing_tolerance
|
||||
}
|
||||
voice_mapping = {}
|
||||
if audio_tensor is not None or audio_path:
|
||||
voice_mapping['narrator'] = {
|
||||
'audio': audio_tensor,
|
||||
'audio_path': audio_path,
|
||||
'reference_text': reference_text or "",
|
||||
}
|
||||
result = engine_instance.processor.process_srt_content(
|
||||
srt_content=srt_content,
|
||||
voice_mapping=voice_mapping,
|
||||
seed=seed,
|
||||
timing_mode=timing_mode,
|
||||
timing_params=timing_params,
|
||||
enable_audio_cache=enable_audio_cache
|
||||
)
|
||||
|
||||
elif engine_type == "omnivoice":
|
||||
timing_params = {
|
||||
'fade_for_StretchToFit': fade_for_StretchToFit,
|
||||
'max_stretch_ratio': max_stretch_ratio,
|
||||
'min_stretch_ratio': min_stretch_ratio,
|
||||
'timing_tolerance': timing_tolerance,
|
||||
'use_native_duration_targeting': native_duration_targeting_active,
|
||||
}
|
||||
|
||||
voice_mapping = {}
|
||||
if audio_tensor is not None or audio_path:
|
||||
voice_mapping['narrator'] = {
|
||||
'audio': audio_tensor,
|
||||
'audio_path': audio_path,
|
||||
'reference_text': reference_text if reference_text else "",
|
||||
}
|
||||
|
||||
result = engine_instance.processor.process_srt_content(
|
||||
srt_content=srt_content,
|
||||
voice_mapping=voice_mapping,
|
||||
seed=seed,
|
||||
timing_mode=timing_mode,
|
||||
timing_params=timing_params,
|
||||
enable_audio_cache=enable_audio_cache
|
||||
)
|
||||
|
||||
elif engine_type == "vibevoice":
|
||||
# Use the VibeVoice SRT processor from the wrapper instance
|
||||
voice_mapping = {"narrator": audio_tensor} if audio_tensor else {}
|
||||
@@ -1422,6 +1586,8 @@ Hello! This is unified SRT TTS with character switching.
|
||||
or "MOSS-TTSD Native Multi-Speaker Dialogue does not support this SRT input" in msg
|
||||
):
|
||||
raise
|
||||
if isinstance(e, InterruptedError):
|
||||
raise
|
||||
error_msg = f"❌ TTS SRT generation failed: {e}"
|
||||
print(error_msg)
|
||||
import traceback
|
||||
|
||||
@@ -210,6 +210,18 @@ Back to the main narrator voice for the conclusion.""",
|
||||
stable_params['optimize'] = config.get('optimize', False)
|
||||
stable_params['max_generate_length'] = config.get('max_generate_length', 500)
|
||||
|
||||
if engine_type == "fish_audio_s2":
|
||||
stable_params['model_variant'] = config.get('model_variant', 's2-pro')
|
||||
stable_params['precision'] = config.get('precision', 'bfloat16')
|
||||
stable_params['compile'] = config.get('compile', False)
|
||||
stable_params['quantization'] = config.get('quantization', 'none')
|
||||
stable_params['multi_speaker_mode'] = config.get('multi_speaker_mode', 'Native Multi-Speaker')
|
||||
stable_params['language_prompting'] = config.get('language_prompting', 'Auto Inline Tag')
|
||||
|
||||
if engine_type == "omnivoice":
|
||||
stable_params['model_variant'] = config.get('model_variant', 'OmniVoice')
|
||||
stable_params['dtype'] = config.get('dtype', 'auto')
|
||||
|
||||
# For MOSS-TTS, include model identity and load-time options.
|
||||
if engine_type == "moss_tts":
|
||||
stable_params['model_variant'] = config.get('model_variant', 'MOSS-TTS-Local-Transformer')
|
||||
@@ -418,6 +430,8 @@ Back to the main narrator voice for the conclusion.""",
|
||||
voice_mapping[char] = {"waveform": waveform, "sample_rate": sample_rate}
|
||||
print(f"🎭 VibeVoice: Using character-specific voice for '{char}'")
|
||||
except Exception as e:
|
||||
if isinstance(e, InterruptedError):
|
||||
raise
|
||||
print(f"⚠️ Failed to load character audio for '{char}': {e}")
|
||||
voice_mapping[char] = char_audio # Fallback to main voice
|
||||
print(f"🔄 VibeVoice: Using main voice fallback for '{char}'")
|
||||
@@ -551,6 +565,67 @@ Back to the main narrator voice for the conclusion.""",
|
||||
|
||||
return engine_instance
|
||||
|
||||
elif engine_type == "fish_audio_s2":
|
||||
from engines.adapters.fish_audio_s2_adapter import FishAudioS2Adapter
|
||||
processor_path = os.path.join(nodes_dir, "fish_audio_s2", "fish_audio_s2_processor.py")
|
||||
processor_spec = importlib.util.spec_from_file_location("fish_audio_s2_processor_module", processor_path)
|
||||
processor_module = importlib.util.module_from_spec(processor_spec)
|
||||
processor_spec.loader.exec_module(processor_module)
|
||||
FishAudioS2Processor = processor_module.FishAudioS2Processor
|
||||
|
||||
class FishAudioS2Wrapper:
|
||||
def __init__(self, cfg):
|
||||
self.config = cfg.copy()
|
||||
self.adapter = FishAudioS2Adapter(self.config)
|
||||
self.processor = FishAudioS2Processor(self.adapter, self.config)
|
||||
|
||||
def update_config(self, new_config):
|
||||
self.config = new_config.copy()
|
||||
self.adapter.update_config(new_config)
|
||||
self.processor.update_config(new_config)
|
||||
|
||||
engine_instance = FishAudioS2Wrapper(config)
|
||||
import time
|
||||
self._cached_engine_instances[cache_key] = {
|
||||
'instance': engine_instance,
|
||||
'timestamp': time.time()
|
||||
}
|
||||
return engine_instance
|
||||
|
||||
elif engine_type == "omnivoice":
|
||||
from engines.adapters.omnivoice_adapter import OmniVoiceEngineAdapter
|
||||
omnivoice_processor_path = os.path.join(nodes_dir, "omnivoice", "omnivoice_processor.py")
|
||||
omnivoice_processor_spec = importlib.util.spec_from_file_location("omnivoice_processor_module", omnivoice_processor_path)
|
||||
omnivoice_processor_module = importlib.util.module_from_spec(omnivoice_processor_spec)
|
||||
omnivoice_processor_spec.loader.exec_module(omnivoice_processor_module)
|
||||
|
||||
OmniVoiceProcessor = omnivoice_processor_module.OmniVoiceProcessor
|
||||
|
||||
class OmniVoiceWrapper:
|
||||
def __init__(self, cfg):
|
||||
self.config = cfg.copy()
|
||||
self.adapter = OmniVoiceEngineAdapter(self.config)
|
||||
self.processor = OmniVoiceProcessor(self.adapter, self.config)
|
||||
|
||||
def update_config(self, new_config):
|
||||
self.config = new_config.copy()
|
||||
self.adapter.update_config(new_config)
|
||||
self.processor.update_config(new_config)
|
||||
|
||||
def check_interrupt(self):
|
||||
if model_management.interrupt_processing:
|
||||
raise InterruptedError("OmniVoice processing interrupted by user")
|
||||
|
||||
engine_instance = OmniVoiceWrapper(config)
|
||||
|
||||
import time
|
||||
self._cached_engine_instances[cache_key] = {
|
||||
'instance': engine_instance,
|
||||
'timestamp': time.time()
|
||||
}
|
||||
|
||||
return engine_instance
|
||||
|
||||
elif engine_type == "qwen3_tts":
|
||||
# Create Qwen3-TTS processor instance
|
||||
# Use global nodes_dir (already defined at module level)
|
||||
@@ -718,6 +793,8 @@ Back to the main narrator voice for the conclusion.""",
|
||||
raise ValueError(f"Unknown engine type: {engine_type}")
|
||||
|
||||
except Exception as e:
|
||||
if isinstance(e, InterruptedError):
|
||||
raise
|
||||
if "MOSS LoRA/base model mismatch" in str(e):
|
||||
raise
|
||||
print(f"❌ Failed to create engine node instance: {e}")
|
||||
@@ -783,7 +860,10 @@ Back to the main narrator voice for the conclusion.""",
|
||||
reference_text = "" # No reference text available from direct audio
|
||||
|
||||
print(f"🎤 TTS Text: Using direct audio input ({character_name})")
|
||||
print(f"⚠️ TTS Text: Direct audio input has no reference text - F5-TTS will fail, Qwen3-TTS will use x_vector_only mode (lower quality)")
|
||||
print(
|
||||
"⚠️ TTS Text: Direct audio input has no reference text - "
|
||||
"F5-TTS and OmniVoice cloning will fail, Qwen3-TTS will use x_vector_only mode (lower quality)"
|
||||
)
|
||||
return None, audio_tensor, reference_text, character_name
|
||||
|
||||
# Priority 2: narrator_voice dropdown (fallback)
|
||||
@@ -828,6 +908,8 @@ Back to the main narrator voice for the conclusion.""",
|
||||
return None, None, "", "narrator"
|
||||
|
||||
except Exception as e:
|
||||
if isinstance(e, InterruptedError):
|
||||
raise
|
||||
print(f"❌ Voice reference error: {e}")
|
||||
return None, None, "", "narrator"
|
||||
|
||||
@@ -914,6 +996,11 @@ Back to the main narrator voice for the conclusion.""",
|
||||
"F5-TTS requires reference text. When using direct audio input, "
|
||||
"please use Character Voices node instead, which provides both audio and text."
|
||||
)
|
||||
if engine_type == "omnivoice" and (audio_tensor is not None or audio_path) and not reference_text.strip():
|
||||
raise ValueError(
|
||||
"OmniVoice voice cloning requires reference text. "
|
||||
"Do not connect raw audio directly. Use Character Voices node or a narrator voice with a matching .reference.txt file."
|
||||
)
|
||||
|
||||
# Create proper engine node instance to preserve ALL functionality
|
||||
engine_instance = self._create_proper_engine_node_instance(TTS_engine)
|
||||
@@ -1371,6 +1458,100 @@ Back to the main narrator voice for the conclusion.""",
|
||||
formatted_audio = AudioProcessingUtils.format_for_comfyui(combined_audio, 48000)
|
||||
result = (formatted_audio, generation_info)
|
||||
|
||||
elif engine_type == "fish_audio_s2":
|
||||
import re
|
||||
from utils.audio.chunk_timing import ChunkTimingHelper
|
||||
|
||||
print(
|
||||
f" Audio cache: {'enabled' if enable_audio_cache else 'disabled'} "
|
||||
"(TTS Text node)"
|
||||
)
|
||||
voice_mapping = {}
|
||||
if audio_tensor is not None or audio_path:
|
||||
voice_mapping['narrator'] = {
|
||||
'audio': audio_tensor,
|
||||
'audio_path': audio_path,
|
||||
'reference_text': reference_text or '',
|
||||
}
|
||||
|
||||
segment_records = engine_instance.processor.process_text(
|
||||
text=text,
|
||||
voice_mapping=voice_mapping,
|
||||
seed=seed,
|
||||
enable_chunking=enable_chunking,
|
||||
max_chars_per_chunk=max_chars_per_chunk,
|
||||
chunk_combination_method=chunk_combination_method,
|
||||
silence_between_chunks_ms=silence_between_chunks_ms,
|
||||
enable_audio_cache=enable_audio_cache
|
||||
)
|
||||
combined_audio, chunk_info = engine_instance.processor.combine_audio_segments(
|
||||
segments=segment_records,
|
||||
method=chunk_combination_method,
|
||||
silence_ms=silence_between_chunks_ms,
|
||||
original_text=text,
|
||||
return_info=True
|
||||
)
|
||||
total_duration = combined_audio.shape[-1] / 44100.0 if combined_audio.numel() else 0.0
|
||||
clean_text = re.sub(r'\[.*?\]', '', text)
|
||||
base_info = (
|
||||
f"Generated {total_duration:.1f}s audio from {len(clean_text)} characters "
|
||||
f"(Fish Audio S2 Pro, narrator: {char_display})"
|
||||
)
|
||||
base_info += "\n🎭 Character switching, pause tags, and Fish inline tags enabled"
|
||||
generation_info = ChunkTimingHelper.enhance_generation_info(f"✅ {base_info}", chunk_info)
|
||||
result = (AudioProcessingUtils.format_for_comfyui(combined_audio, 44100), generation_info)
|
||||
|
||||
elif engine_type == "omnivoice":
|
||||
import re
|
||||
from utils.audio.chunk_timing import ChunkTimingHelper
|
||||
|
||||
if not hasattr(engine_instance, "processor"):
|
||||
omnivoice_processor_path = os.path.join(nodes_dir, "omnivoice", "omnivoice_processor.py")
|
||||
omnivoice_processor_spec = importlib.util.spec_from_file_location("omnivoice_processor_module", omnivoice_processor_path)
|
||||
omnivoice_processor_module = importlib.util.module_from_spec(omnivoice_processor_spec)
|
||||
omnivoice_processor_spec.loader.exec_module(omnivoice_processor_module)
|
||||
OmniVoiceProcessor = omnivoice_processor_module.OmniVoiceProcessor
|
||||
engine_instance.processor = OmniVoiceProcessor(engine_instance.adapter, config)
|
||||
engine_instance.processor.update_config(config)
|
||||
|
||||
voice_mapping = {}
|
||||
if audio_tensor is not None or audio_path:
|
||||
voice_mapping['narrator'] = {
|
||||
'audio': audio_tensor,
|
||||
'audio_path': audio_path,
|
||||
'reference_text': reference_text or '',
|
||||
}
|
||||
|
||||
segment_records = engine_instance.processor.process_text(
|
||||
text=text,
|
||||
voice_mapping=voice_mapping,
|
||||
seed=seed,
|
||||
enable_chunking=enable_chunking,
|
||||
max_chars_per_chunk=max_chars_per_chunk,
|
||||
chunk_combination_method=chunk_combination_method,
|
||||
silence_between_chunks_ms=silence_between_chunks_ms,
|
||||
enable_audio_cache=enable_audio_cache
|
||||
)
|
||||
|
||||
combined_audio, chunk_info = engine_instance.processor.combine_audio_segments(
|
||||
segments=segment_records,
|
||||
method=chunk_combination_method,
|
||||
silence_ms=silence_between_chunks_ms,
|
||||
original_text=text,
|
||||
return_info=True
|
||||
)
|
||||
|
||||
total_duration = combined_audio.shape[-1] / 24000.0 if combined_audio.numel() else 0.0
|
||||
clean_text = re.sub(r'\[.*?\]', '', text)
|
||||
text_length = len(clean_text)
|
||||
|
||||
base_info = f"Generated {total_duration:.1f}s audio from {text_length} characters (OmniVoice, narrator: {char_display})"
|
||||
base_info += "\n🎭 Character switching, pause tags, voice design, and native long-form chunking enabled"
|
||||
generation_info = ChunkTimingHelper.enhance_generation_info(f"✅ {base_info}", chunk_info)
|
||||
|
||||
formatted_audio = AudioProcessingUtils.format_for_comfyui(combined_audio, 24000)
|
||||
result = (formatted_audio, generation_info)
|
||||
|
||||
elif engine_type == "moss_tts":
|
||||
import re
|
||||
from utils.audio.chunk_timing import ChunkTimingHelper
|
||||
@@ -1597,6 +1778,8 @@ Back to the main narrator voice for the conclusion.""",
|
||||
}
|
||||
print(f"🎭 Qwen3-TTS: Using character-specific voice for '{character}' (ICL mode)")
|
||||
except Exception as e:
|
||||
if isinstance(e, InterruptedError):
|
||||
raise
|
||||
print(f"⚠️ Failed to load character audio for '{character}': {e}")
|
||||
# Fallback to narrator voice if available
|
||||
if audio_tensor is not None and reference_text:
|
||||
@@ -1640,6 +1823,8 @@ Back to the main narrator voice for the conclusion.""",
|
||||
print(f"⚠️⚠️ Qwen3-TTS: Character '{character}' has audio but NO reference text")
|
||||
print(f"⚠️⚠️ Using x_vector_only mode (speaker embedding only) - LOWER QUALITY than ICL mode")
|
||||
except Exception as e:
|
||||
if isinstance(e, InterruptedError):
|
||||
raise
|
||||
print(f"⚠️ Failed to load character audio for '{character}': {e}")
|
||||
# Fallback to narrator voice if available
|
||||
if audio_tensor is not None:
|
||||
@@ -1775,6 +1960,8 @@ Back to the main narrator voice for the conclusion.""",
|
||||
raise
|
||||
if engine_type == "index_tts":
|
||||
raise
|
||||
if isinstance(e, InterruptedError):
|
||||
raise
|
||||
error_msg = f"❌ TTS Text generation failed: {e}"
|
||||
print(error_msg)
|
||||
import traceback
|
||||
|
||||
@@ -124,8 +124,10 @@ class UnifiedVoiceChangerNode(BaseVCNode):
|
||||
|
||||
return normalized_audio
|
||||
|
||||
except Exception as e:
|
||||
raise ValueError(f"Failed to process {input_name}: {e}")
|
||||
except Exception as e:
|
||||
if isinstance(e, InterruptedError):
|
||||
raise
|
||||
raise ValueError(f"Failed to process {input_name}: {e}")
|
||||
|
||||
def _handle_rvc_chunked_conversion(self, source_chunks, source_sample_rate, rvc_engine,
|
||||
narrator_target, refinement_passes, max_chunk_duration, chunk_method):
|
||||
@@ -393,9 +395,11 @@ class UnifiedVoiceChangerNode(BaseVCNode):
|
||||
print(f"✅ RVC voice conversion completed with {refinement_passes} refinement passes {cache_info}")
|
||||
return converted_audio, final_conversion_info
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ RVC voice conversion failed: {e}")
|
||||
raise RuntimeError(f"RVC voice conversion failed: {e}")
|
||||
except Exception as e:
|
||||
if isinstance(e, InterruptedError):
|
||||
raise
|
||||
print(f"❌ RVC voice conversion failed: {e}")
|
||||
raise RuntimeError(f"RVC voice conversion failed: {e}")
|
||||
|
||||
def _convert_audio_for_rvc(self, audio_dict):
|
||||
"""Convert ComfyUI audio format to RVC-compatible format."""
|
||||
@@ -595,9 +599,11 @@ class UnifiedVoiceChangerNode(BaseVCNode):
|
||||
else:
|
||||
raise ValueError(f"Engine type '{engine_type}' does not support voice conversion. Currently supported: ChatterBox, CosyVoice")
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to create engine VC node instance: {e}")
|
||||
return None
|
||||
except Exception as e:
|
||||
if isinstance(e, InterruptedError):
|
||||
raise
|
||||
print(f"❌ Failed to create engine VC node instance: {e}")
|
||||
return None
|
||||
|
||||
def _split_audio_into_chunks(self, audio: torch.Tensor, sample_rate: int,
|
||||
max_duration: int, method: str = "smart") -> List[torch.Tensor]:
|
||||
@@ -1126,10 +1132,12 @@ class UnifiedVoiceChangerNode(BaseVCNode):
|
||||
print(f"✅ Voice Changer: {engine_type} conversion successful")
|
||||
return (converted_audio, conversion_info)
|
||||
|
||||
except Exception as e:
|
||||
error_msg = f"❌ Voice conversion failed: {e}"
|
||||
print(error_msg)
|
||||
import traceback
|
||||
except Exception as e:
|
||||
if isinstance(e, InterruptedError):
|
||||
raise
|
||||
error_msg = f"❌ Voice conversion failed: {e}"
|
||||
print(error_msg)
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
|
||||
# Return empty audio and error info
|
||||
|
||||
@@ -24,6 +24,7 @@ from utils.text.pause_processor import PauseTagProcessor
|
||||
from utils.text.step_audio_editx_special_tags import get_edit_tags_for_segment
|
||||
from utils.audio.edit_post_processor import process_segments as apply_edit_post_processing
|
||||
from engines.adapters.vibevoice_adapter import VibeVoiceEngineAdapter
|
||||
from utils.voice.character_logging import resolved_character_label
|
||||
|
||||
|
||||
class VibeVoiceProcessor:
|
||||
@@ -183,9 +184,11 @@ class VibeVoiceProcessor:
|
||||
|
||||
for group_idx, (character, segment_list) in enumerate(grouped_segments):
|
||||
# Check for interruption before processing each character block
|
||||
voice_ref = voice_mapping.get(character)
|
||||
display_name = resolved_character_label(character, voice_ref)
|
||||
if model_management.interrupt_processing:
|
||||
raise InterruptedError(f"VibeVoice character block {group_idx + 1}/{len(grouped_segments)} ({character}) interrupted by user")
|
||||
print(f"🎤 Block {group_idx + 1}: Character '{character}' with {len(segment_list)} segments")
|
||||
raise InterruptedError(f"VibeVoice character block {group_idx + 1}/{len(grouped_segments)} ({display_name}) interrupted by user")
|
||||
print(f"🎤 Block {group_idx + 1}: Character '{display_name}' with {len(segment_list)} segments")
|
||||
|
||||
# Combine text blocks for this character (VibeVoice style)
|
||||
combined_text = '\n'.join(seg.text.strip() for seg in segment_list)
|
||||
@@ -359,7 +362,8 @@ class VibeVoiceProcessor:
|
||||
if not isinstance(voice_ref, dict):
|
||||
voice_note = " [⚠️ Zero-shot mode - no voice reference]"
|
||||
|
||||
print(f"🎭 CUSTOM CHARACTER BLOCK - Generating combined text for '{character}'{voice_note}:")
|
||||
display_name = resolved_character_label(character, voice_ref)
|
||||
print(f"🎭 CUSTOM CHARACTER BLOCK - Generating combined text for '{display_name}'{voice_note}:")
|
||||
print("="*60)
|
||||
print(combined_text)
|
||||
print("="*60)
|
||||
@@ -600,4 +604,4 @@ class VibeVoiceProcessor:
|
||||
def cleanup(self):
|
||||
"""Clean up resources"""
|
||||
if self.adapter:
|
||||
self.adapter.cleanup()
|
||||
self.adapter.cleanup()
|
||||
|
||||
@@ -33,7 +33,8 @@ from utils.text.character_parser import parse_character_text, character_parser
|
||||
from utils.text.segment_parameters import apply_segment_parameters
|
||||
from utils.text.pause_processor import PauseTagProcessor
|
||||
from utils.audio.processing import AudioProcessingUtils
|
||||
from utils.voice.discovery import get_available_characters, get_character_mapping
|
||||
from utils.voice.discovery import get_available_characters, get_character_mapping
|
||||
from utils.voice.character_logging import resolved_character_label
|
||||
from engines.adapters.vibevoice_adapter import VibeVoiceEngineAdapter
|
||||
|
||||
|
||||
@@ -448,10 +449,11 @@ class VibeVoiceSRTProcessor:
|
||||
)
|
||||
else:
|
||||
# Multiple segments for same character - combine in VibeVoice format
|
||||
print(f"🎤 SRT Block {group_idx + 1}: Character '{character}' with {len(text_list)} segments")
|
||||
display_name = resolved_character_label(character, voice_ref)
|
||||
print(f"🎤 SRT Block {group_idx + 1}: Character '{display_name}' with {len(text_list)} segments")
|
||||
combined_text = '\n'.join(f"Speaker 1: {text.strip()}" for text in text_list)
|
||||
|
||||
print(f"🎭 SRT CHARACTER BLOCK - Generating combined text for '{character}':")
|
||||
print(f"🎭 SRT CHARACTER BLOCK - Generating combined text for '{display_name}':")
|
||||
print("="*60)
|
||||
print(combined_text)
|
||||
print("="*60)
|
||||
@@ -552,10 +554,11 @@ class VibeVoiceSRTProcessor:
|
||||
)
|
||||
else:
|
||||
# Multiple segments for same character with same parameters - combine in VibeVoice format
|
||||
print(f"🎤 SRT Block {group_idx + 1}: Character '{character}' with {len(segment_list)} segments")
|
||||
display_name = resolved_character_label(character, voice_ref)
|
||||
print(f"🎤 SRT Block {group_idx + 1}: Character '{display_name}' with {len(segment_list)} segments")
|
||||
combined_text = '\n'.join(f"Speaker 1: {seg.text.strip()}" for seg in segment_list)
|
||||
|
||||
print(f"🎭 SRT CHARACTER BLOCK - Generating combined text for '{character}':")
|
||||
print(f"🎭 SRT CHARACTER BLOCK - Generating combined text for '{display_name}':")
|
||||
print("="*60)
|
||||
print(combined_text)
|
||||
print("="*60)
|
||||
@@ -797,22 +800,29 @@ class VibeVoiceSRTProcessor:
|
||||
Returns:
|
||||
Dict mapping character names to speaker numbers (1-4)
|
||||
"""
|
||||
# Collect all unique characters across all subtitles
|
||||
all_characters = set()
|
||||
|
||||
for subtitle in subtitles:
|
||||
character_segments = parse_character_text(subtitle.text, None)
|
||||
for character, _ in character_segments:
|
||||
all_characters.add(character)
|
||||
|
||||
# Sort characters to ensure consistent mapping
|
||||
# narrator/untagged first, then alphabetical order for tagged characters
|
||||
sorted_chars = sorted(all_characters, key=lambda x: (x != 'narrator', x))
|
||||
|
||||
# Map to speakers (limit to 4 for VibeVoice native mode)
|
||||
char_to_speaker = {}
|
||||
for i, character in enumerate(sorted_chars[:4]): # Max 4 speakers
|
||||
char_to_speaker[character] = i + 1
|
||||
# Collect characters across all subtitles in first-appearance order.
|
||||
# If narrator exists anywhere, reserve Speaker 1 for narrator and then
|
||||
# map remaining characters by first appearance order.
|
||||
seen_characters = set()
|
||||
ordered_characters = []
|
||||
|
||||
for subtitle in subtitles:
|
||||
character_segments = parse_character_text(subtitle.text, None)
|
||||
for character, _ in character_segments:
|
||||
if character not in seen_characters:
|
||||
seen_characters.add(character)
|
||||
ordered_characters.append(character)
|
||||
|
||||
narrator_present = "narrator" in seen_characters
|
||||
if narrator_present:
|
||||
ordered_chars = ["narrator"] + [c for c in ordered_characters if c != "narrator"]
|
||||
else:
|
||||
ordered_chars = ordered_characters
|
||||
|
||||
# Map to speakers (limit to 4 for VibeVoice native mode)
|
||||
char_to_speaker = {}
|
||||
for i, character in enumerate(ordered_chars[:4]): # Max 4 speakers
|
||||
char_to_speaker[character] = i + 1
|
||||
|
||||
if char_to_speaker:
|
||||
print(f"🎭 Global SRT character mapping: {char_to_speaker}")
|
||||
@@ -822,4 +832,4 @@ class VibeVoiceSRTProcessor:
|
||||
def cleanup(self):
|
||||
"""Clean up resources"""
|
||||
if self.adapter:
|
||||
self.adapter.cleanup()
|
||||
self.adapter.cleanup()
|
||||
|
||||
+1
-1
@@ -1,7 +1,7 @@
|
||||
[project]
|
||||
name = "tts_audio_suite"
|
||||
description = "TTS Audio Suite - Universal multi-engine TTS extension for ComfyUI with unified architecture supporting IndexTTS-2, ChatterBox, Chatterbox Multilingual TTS (Official 23-Lang), F5-TTS, Higgs Audio 2, VibeVoice, and RVC engines. It has character voice management, SRT subtitle TTS support, and audio processing capabilities."
|
||||
version = "5.1.1"
|
||||
version = "5.4.4"
|
||||
license = {file = "LICENSE"}
|
||||
|
||||
[project.urls]
|
||||
|
||||
+14
-11
@@ -20,11 +20,13 @@ numpy>=1.26.4,<2.3.0 # Compatible with both numpy 1.26.4 and 2.x series
|
||||
setuptools>=65.0.0 # Provides distutils compatibility for Python 3.12+ (required by FunASR bundled code)
|
||||
|
||||
# Audio processing (safe)
|
||||
soundfile>=0.12.0
|
||||
sounddevice>=0.4.0
|
||||
|
||||
# Text processing (safe)
|
||||
jieba
|
||||
soundfile>=0.12.0
|
||||
sounddevice>=0.4.0
|
||||
pydub
|
||||
webdataset
|
||||
|
||||
# Text processing (safe)
|
||||
jieba
|
||||
pypinyin
|
||||
unidecode
|
||||
phonemizer # IPA phonemization for multilingual TTS (requires espeak system dependency)
|
||||
@@ -92,12 +94,13 @@ torchfcpe>=0.0.4 # Fast Context-based Pitch Estimation for RVC (fcpe met
|
||||
|
||||
# --- PROBLEMATIC PACKAGES ---
|
||||
# These are installed by install.py with special handling (NOT here in requirements.txt):
|
||||
# - librosa (--no-deps): Forces numpy downgrade
|
||||
# - descript-audio-codec (--no-deps): Conflicts with protobuf
|
||||
# - cached-path (--no-deps): Forces package downgrades
|
||||
# - torchcrepe (--no-deps): Conflicts via librosa dependency
|
||||
# - onnxruntime (--no-deps): Forces numpy 2.3.x, needed for OpenSeeFace
|
||||
# - opencv-python (--no-deps): Forces numpy downgrade via numpy<2.3.0 constraint
|
||||
# - librosa (--no-deps): Forces numpy downgrade
|
||||
# - descript-audio-codec (--no-deps): Conflicts with protobuf
|
||||
# - cached-path (--no-deps): Forces package downgrades
|
||||
# - omnivoice (--no-deps): We install official OmniVoice separately to avoid letting pip reshape shared deps
|
||||
# - torchcrepe (--no-deps): Conflicts via librosa dependency
|
||||
# - onnxruntime (--no-deps): Forces numpy 2.3.x, needed for OpenSeeFace
|
||||
# - opencv-python (--no-deps): Forces numpy downgrade via numpy<2.3.0 constraint
|
||||
# - gradio (--no-deps): Forces pydantic, pillow, pydantic-core downgrades
|
||||
|
||||
# --- PYTHON 3.13 NOTES ---
|
||||
|
||||
@@ -18,13 +18,28 @@ python scripts/bump_version_enhanced.py 3.0.2 --changelog-file changelog.txt --c
|
||||
import os
|
||||
import sys
|
||||
import subprocess
|
||||
import argparse
|
||||
from version_utils import VersionManager
|
||||
|
||||
def run_git_command(command: str) -> bool:
|
||||
import argparse
|
||||
from version_utils import VersionManager
|
||||
|
||||
|
||||
def configure_utf8_console() -> None:
|
||||
"""Keep Windows console output from failing on non-UTF-8 locales."""
|
||||
for stream in (sys.stdout, sys.stderr):
|
||||
if hasattr(stream, "reconfigure"):
|
||||
stream.reconfigure(encoding="utf-8", errors="replace")
|
||||
|
||||
|
||||
def run_git_command(command: str) -> bool:
|
||||
"""Run git command and return success status"""
|
||||
try:
|
||||
result = subprocess.run(command, shell=True, capture_output=True, text=True)
|
||||
result = subprocess.run(
|
||||
command,
|
||||
shell=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
)
|
||||
if result.returncode != 0:
|
||||
print(f"Git command failed: {command}")
|
||||
print(f"Error: {result.stderr}")
|
||||
@@ -93,9 +108,9 @@ def main():
|
||||
elif args.commit_file and args.changelog_file:
|
||||
# File input mode
|
||||
try:
|
||||
with open(args.commit_file, 'r') as f:
|
||||
with open(args.commit_file, 'r', encoding='utf-8') as f:
|
||||
commit_description = f.read().strip()
|
||||
with open(args.changelog_file, 'r') as f:
|
||||
with open(args.changelog_file, 'r', encoding='utf-8') as f:
|
||||
changelog_description = f.read().strip()
|
||||
except Exception as e:
|
||||
print(f"Error reading files: {e}")
|
||||
@@ -146,7 +161,7 @@ def main():
|
||||
elif args.file:
|
||||
# Legacy file mode - same description for both
|
||||
try:
|
||||
with open(args.file, 'r') as f:
|
||||
with open(args.file, 'r', encoding='utf-8') as f:
|
||||
description = f.read().strip()
|
||||
commit_description = description
|
||||
changelog_description = description
|
||||
@@ -246,7 +261,9 @@ def main():
|
||||
[sys.executable, table_gen_path, "--readme"],
|
||||
cwd=vm.project_root,
|
||||
capture_output=True,
|
||||
text=True
|
||||
text=True,
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
)
|
||||
if result.returncode != 0:
|
||||
print(f"Warning: Table generation had issues:\n{result.stderr}")
|
||||
@@ -285,7 +302,12 @@ def main():
|
||||
|
||||
# Write commit message to temp file for complex messages
|
||||
import tempfile
|
||||
with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as f:
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode='w',
|
||||
suffix='.txt',
|
||||
delete=False,
|
||||
encoding='utf-8',
|
||||
) as f:
|
||||
f.write(commit_message)
|
||||
temp_file = f.name
|
||||
|
||||
@@ -320,5 +342,6 @@ def main():
|
||||
vm.restore_files(backup)
|
||||
sys.exit(1)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
if __name__ == "__main__":
|
||||
configure_utf8_console()
|
||||
main()
|
||||
|
||||
@@ -5,12 +5,20 @@ Generates documentation tables from YAML source of truth.
|
||||
Also injects condensed table into README.md between markers.
|
||||
"""
|
||||
|
||||
import yaml
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def load_data():
|
||||
import yaml
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def configure_utf8_console() -> None:
|
||||
"""Keep Windows console output from failing on non-UTF-8 locales."""
|
||||
for stream in (sys.stdout, sys.stderr):
|
||||
if hasattr(stream, "reconfigure"):
|
||||
stream.reconfigure(encoding="utf-8", errors="replace")
|
||||
|
||||
|
||||
def load_data():
|
||||
"""Load YAML data"""
|
||||
yaml_path = Path(__file__).parent.parent / "docs/Dev reports/tts_audio_suite_engines.yaml"
|
||||
with open(yaml_path, encoding="utf-8") as f:
|
||||
@@ -98,17 +106,19 @@ def generate_engine_comparison(data):
|
||||
output.append("")
|
||||
output.append("| Engine | Isolation | Models | Size | TTS | SRT | VC | ASR | Training | License | Special Features | Languages |")
|
||||
output.append("| ------------------ | --------- | ----------------------------------------- | ------------ | :-: | :-: | :-: | :-: | :------: | ------------------------ | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |")
|
||||
|
||||
for e in engines:
|
||||
supported_language_count = sum(
|
||||
1
|
||||
for lang_key, lang_data in e["languages"].items()
|
||||
if lang_key != "_default" and lang_data["supported"]
|
||||
)
|
||||
if e["id"] == "rvc":
|
||||
language_summary = "Any"
|
||||
else:
|
||||
language_summary = str(supported_language_count)
|
||||
|
||||
for e in engines:
|
||||
language_summary = e.get("language_summary_full")
|
||||
if not language_summary:
|
||||
supported_language_count = sum(
|
||||
1
|
||||
for lang_key, lang_data in e["languages"].items()
|
||||
if lang_key != "_default" and lang_data["supported"]
|
||||
)
|
||||
if e["id"] == "rvc":
|
||||
language_summary = "Any"
|
||||
else:
|
||||
language_summary = str(supported_language_count)
|
||||
|
||||
# Format special features with proper spacing
|
||||
features = ", ".join(e.get("special_features", []))
|
||||
@@ -150,32 +160,35 @@ def generate_readme_condensed_table(data):
|
||||
output.append("| Engine | Languages | Size | Key Features |")
|
||||
output.append("|--------|-----------|------|--------------|")
|
||||
|
||||
# Show ALL engines
|
||||
for e in engines:
|
||||
|
||||
# Get first 6-8 language flags
|
||||
flags = []
|
||||
count = 0
|
||||
for lang_data in e["languages"].values():
|
||||
if isinstance(lang_data, dict) and lang_data.get("supported") and lang_data.get("flag"):
|
||||
flags.append(lang_data["flag"])
|
||||
count += 1
|
||||
if count >= 6: # Limit to 6 flags for readability
|
||||
break
|
||||
|
||||
lang_display = "\u200B".join(flags)
|
||||
|
||||
# Special handling for ChatterBox 23L
|
||||
if e["id"] == "chatterbox-23l":
|
||||
lang_display = "🌐 24 languages"
|
||||
# Special handling for RVC
|
||||
elif e["id"] == "rvc":
|
||||
lang_display = "🌐 Any"
|
||||
else:
|
||||
# Add count if more languages exist
|
||||
total_langs = sum(1 for ld in e["languages"].values() if ld["supported"])
|
||||
if total_langs > 6:
|
||||
lang_display += f" +{total_langs - 6}"
|
||||
# Show ALL engines
|
||||
for e in engines:
|
||||
language_summary_override = e.get("language_summary_compact")
|
||||
if language_summary_override:
|
||||
lang_display = language_summary_override
|
||||
else:
|
||||
# Get first 6-8 language flags
|
||||
flags = []
|
||||
count = 0
|
||||
for lang_data in e["languages"].values():
|
||||
if isinstance(lang_data, dict) and lang_data.get("supported") and lang_data.get("flag"):
|
||||
flags.append(lang_data["flag"])
|
||||
count += 1
|
||||
if count >= 6: # Limit to 6 flags for readability
|
||||
break
|
||||
|
||||
lang_display = "\u200B".join(flags)
|
||||
|
||||
# Special handling for ChatterBox 23L
|
||||
if e["id"] == "chatterbox-23l":
|
||||
lang_display = "🌐 24 languages"
|
||||
# Special handling for RVC
|
||||
elif e["id"] == "rvc":
|
||||
lang_display = "🌐 Any"
|
||||
else:
|
||||
# Add count if more languages exist
|
||||
total_langs = sum(1 for ld in e["languages"].values() if ld["supported"])
|
||||
if total_langs > 6:
|
||||
lang_display += f" +{total_langs - 6}"
|
||||
|
||||
# Get 1-2 key features
|
||||
special_features = e.get("special_features", [])
|
||||
@@ -436,10 +449,10 @@ def generate_license_table(data):
|
||||
output.append("The project code is MIT. Model weights carry their own licenses:")
|
||||
output.append("")
|
||||
output.append(sep)
|
||||
output.append(header)
|
||||
output.append(sep)
|
||||
for name, lic, com in rows:
|
||||
output.append(f" {name:<{col1}} {lic:<{col2}} {com:<{col3}}")
|
||||
output.append(header)
|
||||
output.append(sep)
|
||||
for name, lic, com in rows:
|
||||
output.append(f" {name:<{col1}} {lic:<{col2}} {com}")
|
||||
output.append(sep)
|
||||
output.append("")
|
||||
output.append("Users are responsible for complying with respective model licenses.")
|
||||
@@ -639,5 +652,6 @@ def main():
|
||||
print("\n✅ All tables generated successfully!")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
if __name__ == "__main__":
|
||||
configure_utf8_console()
|
||||
main()
|
||||
|
||||
@@ -39,7 +39,7 @@ class VersionManager:
|
||||
"""Get current version from nodes.py"""
|
||||
try:
|
||||
nodes_file = os.path.join(self.project_root, 'nodes.py')
|
||||
with open(nodes_file, 'r') as f:
|
||||
with open(nodes_file, 'r', encoding='utf-8') as f:
|
||||
content = f.read()
|
||||
|
||||
match = re.search(self.version_files['nodes.py']['pattern'], content)
|
||||
@@ -58,7 +58,7 @@ class VersionManager:
|
||||
|
||||
config = self.version_files[relative_path]
|
||||
|
||||
with open(file_path, 'r') as f:
|
||||
with open(file_path, 'r', encoding='utf-8') as f:
|
||||
content = f.read()
|
||||
|
||||
# Find and replace version
|
||||
@@ -69,7 +69,7 @@ class VersionManager:
|
||||
print(f"Warning: No version found in {relative_path}")
|
||||
return False
|
||||
|
||||
with open(file_path, 'w') as f:
|
||||
with open(file_path, 'w', encoding='utf-8') as f:
|
||||
f.write(updated_content)
|
||||
|
||||
print(f"✓ Updated {relative_path}")
|
||||
@@ -284,7 +284,7 @@ This matches the F5-TTS dropdown behavior for consistent user experience across
|
||||
try:
|
||||
changelog_path = os.path.join(self.project_root, 'CHANGELOG.md')
|
||||
|
||||
with open(changelog_path, 'r') as f:
|
||||
with open(changelog_path, 'r', encoding='utf-8') as f:
|
||||
content = f.read()
|
||||
|
||||
# Generate changelog entry content using extracted method
|
||||
@@ -305,7 +305,7 @@ This matches the F5-TTS dropdown behavior for consistent user experience across
|
||||
# Insert the new entry with proper spacing
|
||||
lines.insert(insert_index, new_entry.rstrip())
|
||||
|
||||
with open(changelog_path, 'w') as f:
|
||||
with open(changelog_path, 'w', encoding='utf-8') as f:
|
||||
f.write('\n'.join(lines))
|
||||
|
||||
print(f"✓ Added changelog entry for v{version}")
|
||||
@@ -321,7 +321,7 @@ This matches the F5-TTS dropdown behavior for consistent user experience across
|
||||
for file_path in self.version_files.keys():
|
||||
full_path = os.path.join(self.project_root, file_path)
|
||||
if os.path.exists(full_path):
|
||||
with open(full_path, 'r') as f:
|
||||
with open(full_path, 'r', encoding='utf-8') as f:
|
||||
backups[file_path] = f.read()
|
||||
return backups
|
||||
|
||||
@@ -330,9 +330,9 @@ This matches the F5-TTS dropdown behavior for consistent user experience across
|
||||
try:
|
||||
for file_path, content in backups.items():
|
||||
full_path = os.path.join(self.project_root, file_path)
|
||||
with open(full_path, 'w') as f:
|
||||
with open(full_path, 'w', encoding='utf-8') as f:
|
||||
f.write(content)
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"Error restoring files: {e}")
|
||||
return False
|
||||
return False
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
# FL-MCP Live ComfyUI Validation
|
||||
|
||||
This runbook defines an optional live acceptance test for new engine implementations using [ComfyUI_FL-MCP](https://github.com/filliptm/ComfyUI_FL-MCP).
|
||||
|
||||
FL-MCP gives an MCP-capable LLM access to ComfyUI workflow state, queue execution, history, errors, output files, and browser screenshots. It does not replace pytest, canonical Windows validation, or human evaluation of audio quality.
|
||||
|
||||
## Efficient Tool Use
|
||||
|
||||
Use terminal tools for code inspection, edits, pytest, process management, startup logs, and routine log monitoring. Use FL-MCP at live-validation checkpoints: capability audit, relevant widget configuration, workflow queueing, prompt-specific execution verification, and output inspection.
|
||||
|
||||
Do not use FL-MCP as a replacement for efficient terminal work. Avoid fetching full workflow JSON, repeated node definitions or workflow overviews, screenshots, broad execution history, and polling calls unless they are needed to construct the workflow or diagnose a failure. Prefer one targeted call with the known node or prompt ID. A representative FL-MCP acceptance pass after automated tests is normally sufficient; repeat it only after changes that can affect live behavior.
|
||||
|
||||
## Preconditions
|
||||
|
||||
- Use the canonical Windows ComfyUI installation documented in the repository instructions.
|
||||
- ComfyUI starts with TTS Audio Suite enabled and no suite import errors.
|
||||
- FL-MCP is installed according to its upstream documentation.
|
||||
- The FL-MCP server is connected to the MCP client.
|
||||
- For browser-only operations, ComfyUI is open and the FL-MCP browser bridge reports connected.
|
||||
- The MCP client is bound to the same browser session. Check the bridge session endpoint or capability audit and require both `has_frontend: true` and `has_mcp: true` for one session. A frontend-only session provides no canvas access to the LLM.
|
||||
- Required model files and reference audio are available.
|
||||
- The user has authorized any installation or configuration changes. Do not weaken FL-MCP safety gates just to complete a test.
|
||||
|
||||
If any precondition is missing, record it as a blocked or skipped check. Do not claim FL-MCP validation.
|
||||
|
||||
## Implementation Restart Loop
|
||||
|
||||
ComfyUI imports custom-node Python code at startup. Editing an engine does not update an already-running ComfyUI process. During implementation, use this loop:
|
||||
|
||||
1. Edit the engine implementation.
|
||||
2. Run the relevant automated tests.
|
||||
3. Identify the Windows process listening on port `8188` and inspect its command line. Stop it only if it is the canonical ComfyUI `main.py` process.
|
||||
4. Resolve the ComfyUI root and Python executable from the user's local configuration, then launch ComfyUI. For example:
|
||||
|
||||
```powershell
|
||||
$comfyRoot = '<COMFYUI_ROOT>'
|
||||
$pythonExe = '<COMFYUI_PYTHON>'
|
||||
Set-Location $comfyRoot
|
||||
& $pythonExe 'main.py'
|
||||
```
|
||||
|
||||
On Windows, prefer a visible Windows Terminal tab with live output (PowerShell console only as fallback) so clicking the log cannot pause ComfyUI. When the agent needs searchable logs, also use `Start-Transcript` to a temporary file under `ComfyUI/temp` and inspect only relevant tails. Preserve required canonical launch arguments and never track machine-specific paths.
|
||||
5. Poll `http://127.0.0.1:8188/system_stats` until it responds or startup fails. Inspect the startup log for custom-node import errors.
|
||||
6. Refresh the already-open ComfyUI browser tab. A hard refresh may be required for changed JavaScript.
|
||||
7. Confirm the FL-MCP backend is healthy and the browser bridge is connected again. Canvas tools are not available until the browser reconnects.
|
||||
8. Run the live workflow checks below.
|
||||
9. After a fix, repeat from step 2. Never validate new Python code against a process that started before the edit.
|
||||
|
||||
Do not stop every `python.exe` process. Other engines and applications may use Python. Resolve the owner of port `8188`, verify its executable/command line, and stop only the intended ComfyUI process.
|
||||
|
||||
### Windows external bridge fallback
|
||||
|
||||
If a ComfyUI console interrupt also stops FL-MCP's embedded backend, do not describe that as the backend shutting down ComfyUI. Both processes are receiving the same console signal. Separate them instead:
|
||||
|
||||
1. In FL-MCP's local, untracked `.env`, set `BACKEND_LAUNCH_MODE=manual`, `AUTO_START_BACKEND=false`, and `AUTO_RESTART_BACKEND=false`.
|
||||
2. Start `backend/server.py` in its own visible Windows Terminal window using FL-MCP's dedicated Python environment.
|
||||
3. Start ComfyUI in a separate visible Windows Terminal window using its canonical launcher.
|
||||
4. Confirm the bridge backend is healthy, refresh the ComfyUI browser tab, and reconnect the FL-MCP panel before auditing capabilities.
|
||||
|
||||
Do not leave the embedded launcher enabled while manually running `backend/server.py`; that can create competing bridge processes on the same port.
|
||||
|
||||
For manual Windows mode, keep two visible service terminals: one running ComfyUI and one running FL-MCP `backend/server.py`; the MCP client (for example Codex) is a separate process that must be restarted or rebound with the browser session ID so one session reports both `has_frontend: true` and `has_mcp: true`.
|
||||
|
||||
FL-MCP process-control permission is not required for this loop. An authorized coding agent can manage the canonical process through PowerShell. Keep `FL_MCP_ENABLE_COMFY_PROCESS_CONTROL=false` unless the user explicitly chooses FL-MCP-managed process control.
|
||||
|
||||
### MCP client session binding
|
||||
|
||||
REST access does not imply browser/canvas access. Before starting the implementation session:
|
||||
|
||||
1. Open ComfyUI and obtain the persistent FL-MCP browser session ID from its panel or the backend session endpoint.
|
||||
2. Configure the MCP client process with:
|
||||
|
||||
```text
|
||||
FL_MCP_MODE=subprocess
|
||||
FL_MCP_SESSION_ID=<browser-session-id>
|
||||
FL_MCP_WS_URL=ws://127.0.0.1:8000/ws
|
||||
COMFYUI_SERVER_URL=http://127.0.0.1:8188
|
||||
```
|
||||
|
||||
3. Restart the MCP client session so it launches the server with the new environment.
|
||||
4. Run the capability audit and verify the same session reports both frontend and MCP connections.
|
||||
|
||||
The browser stores its session ID in local storage, so normal ComfyUI restarts and browser refreshes should retain it. If the browser session is reset or a different browser/profile is used, update the MCP client configuration and restart that client again.
|
||||
|
||||
## Validation Layers
|
||||
|
||||
Run all applicable layers:
|
||||
|
||||
1. Existing unit and integration tests through `tests/run_tests.py`.
|
||||
2. Live ComfyUI registration and workflow validation through FL-MCP, or the manual fallback below.
|
||||
3. Human listening evaluation.
|
||||
|
||||
FL-MCP is strongest at layer 2. A screenshot alone is not a passing test.
|
||||
|
||||
## Standard Smoke Sequence
|
||||
|
||||
Tool names can change between FL-MCP versions. Discover the connected server's available tools and use the current equivalents rather than assuming an exact name.
|
||||
|
||||
### 1. Establish a baseline
|
||||
|
||||
- Record the ComfyUI URL, TTS Audio Suite branch/commit, FL-MCP version, target engine, model variant, and runtime mode.
|
||||
- Record the ComfyUI process start time and confirm it started after the implementation changes being tested.
|
||||
- Check ComfyUI health and queue state. Read recent execution history only when needed to separate an existing failure from the new run.
|
||||
- Preserve pre-existing errors separately so they are not attributed to the new engine.
|
||||
|
||||
### 2. Verify registration and UI state
|
||||
|
||||
- Inspect the target node definition and confirm the new engine node is registered. Do not repeatedly fetch definitions that have already been verified after the current restart.
|
||||
- Confirm every relevant unified node is registered: TTS Text and SRT for TTS engines, plus VC, ASR, or special nodes when scoped.
|
||||
- Open or construct a minimal workflow.
|
||||
- Verify important widget values and connections with targeted node/workflow calls. Fetch full workflow JSON only when construction, migration, or debugging requires it.
|
||||
- Capture a canvas screenshot only when UI layout, dynamic widgets, or visual state is part of the change or useful failure evidence.
|
||||
|
||||
### 3. Run the smallest real generation
|
||||
|
||||
- Use short deterministic text and the smallest suitable model/configuration.
|
||||
- Queue the workflow.
|
||||
- Wait for terminal execution state; do not treat prompt submission as success.
|
||||
- Inspect execution history/details for the prompt.
|
||||
- On failure, capture the node ID/type, exception, traceback or message, and relevant configuration.
|
||||
- On success, verify that an expected audio output is listed and exists in ComfyUI output or temp storage.
|
||||
|
||||
### 4. Exercise required paths
|
||||
|
||||
For every TTS engine:
|
||||
|
||||
- Run TTS Text.
|
||||
- Run SRT with at least two subtitle entries.
|
||||
- Run one supported character, narrator, or reference-audio path as applicable.
|
||||
- Run pause tags.
|
||||
- Run one real engine-specific parameter switch when supported.
|
||||
- Repeat an identical request to exercise generated-audio caching when applicable.
|
||||
- Clear VRAM/unload, then generate again.
|
||||
- Start a sufficiently long generation and test interrupt/cancel.
|
||||
|
||||
Also test VC, ASR, and special nodes when they are part of the agreed engine scope.
|
||||
|
||||
### 5. Inspect outputs
|
||||
|
||||
- Confirm output audio metadata is plausible, including sample rate, channel/batch shape where exposed, and non-zero duration.
|
||||
- Confirm SRT output timing and artifact count match the workflow contract.
|
||||
- Verify each run through its prompt-specific execution result rather than repeatedly fetching broad history or relying on the canvas alone.
|
||||
- Ask the user to listen to representative outputs. The LLM must not claim audio quality, pronunciation, similarity, or naturalness based only on metadata.
|
||||
|
||||
## Required Evidence
|
||||
|
||||
Return a compact validation report containing:
|
||||
|
||||
- Environment and exact branch/commit.
|
||||
- Saved workflow path, or the relevant node configuration and connections. Export full workflow JSON only when needed to reproduce the result.
|
||||
- Screenshot when visual UI behavior was tested; otherwise mark it `not required`.
|
||||
- Prompt/execution ID and terminal status for each run.
|
||||
- Relevant execution error details for failed runs.
|
||||
- Output artifact paths for successful runs.
|
||||
- A pass/fail/blocked table for each required path.
|
||||
- Automated test results.
|
||||
- Explicitly untested items and reasons.
|
||||
- Human audio-quality verdict, or `not evaluated`.
|
||||
|
||||
Do not commit generated audio, screenshots, logs, downloaded models, or local FL-MCP configuration unless the user explicitly requests tracked fixtures.
|
||||
|
||||
## Manual Fallback
|
||||
|
||||
If FL-MCP is unavailable:
|
||||
|
||||
1. Start the canonical Windows ComfyUI instance and inspect startup logs for node import errors.
|
||||
2. Build or load the same minimal workflows in the browser.
|
||||
3. Queue each required path and wait for completion.
|
||||
4. Copy full errors from ComfyUI/server logs.
|
||||
5. Export workflow JSON and record output artifact paths.
|
||||
6. Capture screenshots manually.
|
||||
7. Produce the same validation report and mark the driver as `manual`.
|
||||
|
||||
Manual execution is valid evidence. Do not install FL-MCP during an engine task unless installation was explicitly authorized.
|
||||
|
||||
## Stop Conditions
|
||||
|
||||
Stop the live test and report the blocker when:
|
||||
|
||||
- Installing dependencies would modify the user's environment without authorization.
|
||||
- A required model download, license acceptance, credential, or reference file is unavailable.
|
||||
- Repeated execution risks corrupting data or exhausting disk space.
|
||||
- The connected MCP tools cannot identify terminal execution state or retrieve actionable errors.
|
||||
|
||||
Fix implementation defects that are within the engine task. Do not hide failures by changing safety settings, deleting unrelated queues/files, or weakening assertions.
|
||||
+30
-12
@@ -4,19 +4,23 @@ This guide covers running tests for the TTS Audio Suite custom node.
|
||||
|
||||
## Quick Start
|
||||
|
||||
For a split installation, first create the ignored `tests/.env.local` file described under [Path Configuration & Local Overrides](#path-configuration--local-overrides). The wrapper cannot infer paths when ComfyUI and this repository live in separate directory trees.
|
||||
|
||||
```bash
|
||||
cd C:\_stability_matrix\Data\Packages\Comfy-new\custom_nodes\TTS-Audio-Suite
|
||||
cd /path/to/ComfyUI/custom_nodes/TTS-Audio-Suite
|
||||
|
||||
# Run all tests
|
||||
..\..\venv\Scripts\python -m pytest tests/ -v
|
||||
python tests/run_tests.py -v
|
||||
|
||||
# Run only fast unit tests (no server needed)
|
||||
..\..\venv\Scripts\python -m pytest tests/unit/ -m unit -v
|
||||
python tests/run_tests.py unit/ -m unit -v
|
||||
|
||||
# Run integration tests (requires ComfyUI running)
|
||||
..\..\venv\Scripts\python -m pytest tests/integration/ -m integration -v
|
||||
python tests/run_tests.py integration/ -m integration -v
|
||||
```
|
||||
|
||||
Always use `tests/run_tests.py`; it selects the configured Python, changes into the isolated test directory, and ensures the local pytest configuration wins.
|
||||
|
||||
---
|
||||
|
||||
## Test Categories
|
||||
@@ -47,10 +51,10 @@ Tests that run against a live ComfyUI server.
|
||||
|
||||
```bash
|
||||
# Run by marker
|
||||
pytest -m unit # Fast, no server
|
||||
pytest -m integration # Requires ComfyUI
|
||||
pytest -m slow # Full generation tests
|
||||
pytest -m cosyvoice # CosyVoice3 specific
|
||||
python tests/run_tests.py -m unit # Fast, no server
|
||||
python tests/run_tests.py -m integration # Requires ComfyUI
|
||||
python tests/run_tests.py -m slow # Full generation tests
|
||||
python tests/run_tests.py -m cosyvoice # CosyVoice3 specific
|
||||
```
|
||||
|
||||
---
|
||||
@@ -61,16 +65,30 @@ Integration tests require ComfyUI to be running:
|
||||
|
||||
```bash
|
||||
# Terminal 1: Start ComfyUI
|
||||
cd C:\_stability_matrix\Data\Packages\Comfy-new
|
||||
venv\Scripts\python main.py
|
||||
cd /path/to/ComfyUI
|
||||
/path/to/comfyui/python main.py
|
||||
|
||||
# Terminal 2: Run tests
|
||||
cd custom_nodes\TTS-Audio-Suite
|
||||
..\..\venv\Scripts\python -m pytest tests/integration/ -m integration -v
|
||||
cd /path/to/ComfyUI/custom_nodes/TTS-Audio-Suite
|
||||
python tests/run_tests.py integration/ -m integration -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## FL-MCP-Assisted Live Validation
|
||||
|
||||
[ComfyUI_FL-MCP](https://github.com/filliptm/ComfyUI_FL-MCP) can drive and inspect a live ComfyUI session from an MCP-capable coding client. It complements pytest by exposing workflow state, browser screenshots, queue execution, history, errors, and output files. It is optional and is not a dependency of TTS Audio Suite.
|
||||
|
||||
Use it for new-engine acceptance testing after the implementation and automated tests are complete. Follow [FL_MCP_VALIDATION.md](FL_MCP_VALIDATION.md) for prerequisites, the standard smoke sequence, required evidence, safety boundaries, and the manual fallback.
|
||||
|
||||
New custom-node Python code is loaded only when ComfyUI starts. During implementation, follow the runbook's mandatory edit–restart–browser reconnect–validate loop. Keeping an old ComfyUI process open does not test newly written engine code.
|
||||
|
||||
REST availability alone is insufficient for canvas inspection. Before live validation, bind the MCP client to the browser's persistent FL-MCP session ID and verify one bridge session reports both frontend and MCP connections.
|
||||
|
||||
FL-MCP does not determine whether generated speech sounds good. A human must assess pronunciation, voice similarity, artifacts, pacing, and other subjective audio qualities.
|
||||
|
||||
---
|
||||
|
||||
## Test Fixtures
|
||||
|
||||
Test workflow files are in `tests/fixtures/workflows/`:
|
||||
|
||||
+27
-1
@@ -10,6 +10,8 @@ Session-scoped fixtures for ComfyUI server management and API testing.
|
||||
# ============================================================================
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import types
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
# Set testing environment
|
||||
@@ -22,7 +24,6 @@ _MOCK_MODULES = [
|
||||
'comfy.model_management',
|
||||
'comfy.utils',
|
||||
'nodes',
|
||||
'folder_paths',
|
||||
'server',
|
||||
'execution',
|
||||
'comfy_extras',
|
||||
@@ -32,6 +33,31 @@ for _module_name in _MOCK_MODULES:
|
||||
if _module_name not in sys.modules:
|
||||
sys.modules[_module_name] = MagicMock()
|
||||
|
||||
# folder_paths cannot be a raw MagicMock because code under test uses
|
||||
# folder_paths.models_dir in os.path.join/os.makedirs calls. A MagicMock there
|
||||
# creates junk folders like ./MagicMock/mock.models_dir/... on disk.
|
||||
if 'folder_paths' not in sys.modules:
|
||||
mock_folder_paths = types.ModuleType("folder_paths")
|
||||
mock_models_dir = tempfile.mkdtemp(prefix="tts_suite_test_models_")
|
||||
mock_input_dir = tempfile.mkdtemp(prefix="tts_suite_test_input_")
|
||||
mock_output_dir = tempfile.mkdtemp(prefix="tts_suite_test_output_")
|
||||
mock_temp_dir = tempfile.mkdtemp(prefix="tts_suite_test_temp_")
|
||||
|
||||
mock_folder_paths.models_dir = mock_models_dir
|
||||
mock_folder_paths.input_directory = mock_input_dir
|
||||
mock_folder_paths.output_directory = mock_output_dir
|
||||
mock_folder_paths.temp_directory = mock_temp_dir
|
||||
|
||||
mock_folder_paths.get_folder_paths = lambda *args, **kwargs: []
|
||||
mock_folder_paths.add_model_folder_path = lambda *args, **kwargs: None
|
||||
mock_folder_paths.get_input_directory = lambda: mock_input_dir
|
||||
mock_folder_paths.get_output_directory = lambda: mock_output_dir
|
||||
mock_folder_paths.get_temp_directory = lambda: mock_temp_dir
|
||||
mock_folder_paths.get_annotated_filepath = lambda path: path
|
||||
mock_folder_paths.exists_annotated_filepath = lambda path: os.path.exists(path)
|
||||
|
||||
sys.modules['folder_paths'] = mock_folder_paths
|
||||
|
||||
# Now safe to import pytest and other modules
|
||||
import pytest
|
||||
import subprocess
|
||||
|
||||
+1
-1
@@ -3,7 +3,7 @@ testpaths = unit integration
|
||||
python_files = test_*.py
|
||||
python_classes = Test*
|
||||
python_functions = test_*
|
||||
addopts = -v --tb=short --disable-warnings
|
||||
addopts = -v --tb=short --disable-warnings --import-mode=importlib
|
||||
# Don't recursively search these files (prevents __init__.py loading)
|
||||
norecursedirs = __pycache__ .git engines nodes web
|
||||
# Override parent pytest.ini pythonpath - don't auto-add package root
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
|
||||
from utils.voice.character_logging import (
|
||||
format_character_override_warning,
|
||||
format_resolved_character_block,
|
||||
format_resolved_character_text,
|
||||
resolved_character_label,
|
||||
resolved_voice_name,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_resolved_voice_name_prefers_connected_character_name():
|
||||
voice_ref = {
|
||||
"character_name": "Tony",
|
||||
"audio_path": r"J:\voices\other-name.mp3",
|
||||
}
|
||||
|
||||
assert resolved_voice_name(voice_ref) == "Tony"
|
||||
assert resolved_character_label("Bob", voice_ref) == "Tony"
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_resolved_voice_name_supports_audio_paths_and_legacy_tuples():
|
||||
audio_path = Path("voices") / "Ana_1.wav"
|
||||
|
||||
assert resolved_voice_name({"audio_path": audio_path}) == "Ana_1"
|
||||
assert resolved_voice_name((str(audio_path), "reference text")) == "Ana_1"
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_resolved_voice_logging_formats_are_shared():
|
||||
voice_ref = {"character_name": "Tony"}
|
||||
|
||||
assert format_resolved_character_text("Bob", "Hello!", voice_ref) == "[Tony] Hello!"
|
||||
assert format_resolved_character_block("Bob", "Hello!", voice_ref) == (
|
||||
"============================================================\n"
|
||||
"[Tony] Hello!\n"
|
||||
"============================================================"
|
||||
)
|
||||
assert format_character_override_warning("Fish", "Speaker 2", "Bob") == (
|
||||
"⚠️ Fish priority: Speaker 2 input overrides ['Bob'] alias"
|
||||
)
|
||||
@@ -0,0 +1,92 @@
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
|
||||
from utils.audio.cache import FishAudioS2CacheKeyGenerator
|
||||
from engines.adapters.fish_audio_s2_adapter import FishAudioS2Adapter
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_quantization_variant_invalidates_audio_cache():
|
||||
generator = FishAudioS2CacheKeyGenerator()
|
||||
common = {
|
||||
"text": "Test",
|
||||
"audio_component": "voice-hash",
|
||||
"reference_text": "Reference",
|
||||
"seed": 1,
|
||||
}
|
||||
|
||||
keys = {
|
||||
generator.generate_cache_key(**common, model_variant=variant)
|
||||
for variant in ("s2-pro", "s2-pro-fp8")
|
||||
}
|
||||
|
||||
assert len(keys) == 2
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_load_quantization_invalidates_audio_cache():
|
||||
generator = FishAudioS2CacheKeyGenerator()
|
||||
common = {
|
||||
"text": "Test",
|
||||
"audio_component": "voice-hash",
|
||||
"reference_text": "Reference",
|
||||
"seed": 1,
|
||||
"model_variant": "s2-pro",
|
||||
}
|
||||
|
||||
keys = {
|
||||
generator.generate_cache_key(**common, quantization=quantization)
|
||||
for quantization in ("none", "bnb_int8", "bnb_nf4")
|
||||
}
|
||||
|
||||
assert len(keys) == 3
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_speaker_mode_invalidates_audio_cache():
|
||||
generator = FishAudioS2CacheKeyGenerator()
|
||||
common = {
|
||||
"text": "Test",
|
||||
"audio_component": "voice-hash",
|
||||
"reference_text": "Reference",
|
||||
"seed": 1,
|
||||
}
|
||||
|
||||
native = generator.generate_cache_key(
|
||||
**common, multi_speaker_mode="Native Multi-Speaker"
|
||||
)
|
||||
custom = generator.generate_cache_key(
|
||||
**common, multi_speaker_mode="Custom Character Switching"
|
||||
)
|
||||
|
||||
assert native != custom
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_adapter_reuses_unchanged_custom_segment_and_invalidates_changed_text(monkeypatch):
|
||||
adapter = FishAudioS2Adapter({"multi_speaker_mode": "Custom Character Switching"})
|
||||
adapter.audio_cache.clear_cache()
|
||||
calls = []
|
||||
|
||||
class FakeEngine:
|
||||
def generate(self, **params):
|
||||
calls.append(params["text"])
|
||||
return torch.zeros(1, 32), adapter.SAMPLE_RATE
|
||||
|
||||
monkeypatch.setattr(adapter, "_engine", lambda: FakeEngine())
|
||||
monkeypatch.setattr(adapter, "_reference", lambda _voice: ("voice.wav", "Reference", "voice-hash"))
|
||||
voice = {"audio_path": "voice.wav", "reference_text": "Reference"}
|
||||
|
||||
adapter.generate_dialogue([(0, "unchanged")], [voice], seed=1, cache_character="speaker")
|
||||
adapter.generate_dialogue([(0, "unchanged")], [voice], seed=1, cache_character="speaker")
|
||||
adapter.generate_dialogue([(0, "changed")], [voice], seed=1, cache_character="speaker")
|
||||
|
||||
assert len(calls) == 2
|
||||
adapter.audio_cache.clear_cache()
|
||||
@@ -0,0 +1,77 @@
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
NODE_PATH = (
|
||||
REPO_ROOT
|
||||
/ "nodes"
|
||||
/ "engines"
|
||||
/ "fish_audio_s2_engine_node.py"
|
||||
)
|
||||
SPEC = importlib.util.spec_from_file_location("fish_audio_s2_engine_node_test_module", NODE_PATH)
|
||||
NODE_MODULE = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(NODE_MODULE)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_model_is_first_and_compile_defaults_enabled():
|
||||
inputs = NODE_MODULE.FishAudioS2EngineNode.INPUT_TYPES()
|
||||
|
||||
assert list(inputs["required"])[0] == "model"
|
||||
assert "s2-pro" in inputs["required"]["model"][0]
|
||||
assert inputs["optional"]["compile"][1]["default"] is True
|
||||
|
||||
node = NODE_MODULE.FishAudioS2EngineNode()
|
||||
(engine_config,) = node.create_engine_config()
|
||||
assert engine_config["config"]["compile"] is True
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_bnb_quantization_preserves_compile():
|
||||
node = NODE_MODULE.FishAudioS2EngineNode()
|
||||
|
||||
(engine_config,) = node.create_engine_config(
|
||||
device="cuda",
|
||||
temperature=0.8,
|
||||
top_p=0.8,
|
||||
repetition_penalty=1.1,
|
||||
native_chunk_length=200,
|
||||
max_new_tokens=1024,
|
||||
context_length="8192",
|
||||
compile=True,
|
||||
model_variant="s2-pro",
|
||||
quantization="bnb_int8",
|
||||
)
|
||||
|
||||
config = engine_config["config"]
|
||||
assert config["compile"] is True
|
||||
assert config["quantization"] == "bnb_int8"
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_fp8_variant_forces_quantization_none():
|
||||
node = NODE_MODULE.FishAudioS2EngineNode()
|
||||
|
||||
(engine_config,) = node.create_engine_config(
|
||||
device="cuda",
|
||||
temperature=0.8,
|
||||
top_p=0.8,
|
||||
repetition_penalty=1.1,
|
||||
native_chunk_length=200,
|
||||
max_new_tokens=1024,
|
||||
context_length="8192",
|
||||
compile=False,
|
||||
model_variant="s2-pro-fp8",
|
||||
quantization="bnb_nf4",
|
||||
precision="float16",
|
||||
)
|
||||
|
||||
config = engine_config["config"]
|
||||
assert config["model_variant"] == "s2-pro-fp8"
|
||||
assert config["quantization"] == "none"
|
||||
assert config["precision"] == "bfloat16"
|
||||
@@ -0,0 +1,28 @@
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
from utils.runtimes.fish_audio_s2_fp8 import FP8Linear
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_fp8_linear_uses_row_scales_for_bfloat16_compute():
|
||||
layer = FP8Linear(2, 2)
|
||||
layer.qweight = torch.tensor(
|
||||
[[1.0, 2.0], [3.0, 4.0]],
|
||||
dtype=torch.float8_e4m3fn,
|
||||
)
|
||||
layer.scale = torch.tensor([[0.5], [0.25]], dtype=torch.float32)
|
||||
inputs = torch.tensor([[2.0, 1.0]], dtype=torch.bfloat16)
|
||||
|
||||
output = layer(inputs)
|
||||
|
||||
expected_weight = torch.tensor(
|
||||
[[0.5, 1.0], [0.75, 1.0]],
|
||||
dtype=torch.bfloat16,
|
||||
)
|
||||
assert torch.equal(output, torch.nn.functional.linear(inputs, expected_weight))
|
||||
@@ -0,0 +1,636 @@
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
PROCESSOR_PATH = (
|
||||
REPO_ROOT
|
||||
/ "nodes"
|
||||
/ "fish_audio_s2"
|
||||
/ "fish_audio_s2_processor.py"
|
||||
)
|
||||
SPEC = importlib.util.spec_from_file_location("fish_audio_s2_processor_test_module", PROCESSOR_PATH)
|
||||
PROCESSOR_MODULE = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(PROCESSOR_MODULE)
|
||||
|
||||
|
||||
class FakeCharacterParser:
|
||||
def set_available_characters(self, _characters):
|
||||
pass
|
||||
|
||||
def set_character_language_default(self, _character, _language):
|
||||
pass
|
||||
|
||||
def reset_session_cache(self):
|
||||
pass
|
||||
|
||||
def parse_text_segments(self, text, engine_type=None):
|
||||
characters = ["alice", "bob", "rick"] if text == "all" else ["bob", "rick"]
|
||||
return [
|
||||
SimpleNamespace(character=character, text=character.title(), parameters={}, language="en")
|
||||
for character in characters
|
||||
]
|
||||
|
||||
|
||||
class CapturingAdapter:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
|
||||
def update_config(self, _config):
|
||||
pass
|
||||
|
||||
def generate_dialogue(self, turns, voice_refs, **_kwargs):
|
||||
self.calls.append((list(turns), list(voice_refs)))
|
||||
return torch.zeros(1, 16)
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_late_global_character_uses_local_reference_order(monkeypatch):
|
||||
parser = FakeCharacterParser()
|
||||
|
||||
def parse_subtitle_three(text, engine_type=None):
|
||||
if text == "subtitle3":
|
||||
return [
|
||||
SimpleNamespace(character="male_01", text="Bob line", parameters={}, language="en"),
|
||||
SimpleNamespace(character="narrator", text="Narrator line", parameters={}, language="en"),
|
||||
]
|
||||
return [
|
||||
SimpleNamespace(character="narrator", text="Narrator", parameters={}, language="en"),
|
||||
SimpleNamespace(character="female_01", text="Alice", parameters={}, language="en"),
|
||||
SimpleNamespace(character="clint_eastwood cc3 (enhanced2)", text="Cowboy", parameters={}, language="en"),
|
||||
SimpleNamespace(character="male_01", text="Bob", parameters={}, language="en"),
|
||||
]
|
||||
|
||||
parser.parse_text_segments = parse_subtitle_three
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"get_character_mapping",
|
||||
lambda *_args, **_kwargs: {"male_01": ("male_01.wav", "Bob reference")},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"model_management",
|
||||
SimpleNamespace(interrupt_processing=False),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"voice_discovery",
|
||||
SimpleNamespace(
|
||||
get_character_aliases=lambda: {},
|
||||
get_character_language_defaults=lambda: {},
|
||||
),
|
||||
)
|
||||
|
||||
adapter = CapturingAdapter()
|
||||
processor = PROCESSOR_MODULE.FishAudioS2Processor(
|
||||
adapter,
|
||||
{
|
||||
"speaker_references": [
|
||||
{"character_name": "Ana"},
|
||||
{"character_name": "Tony"},
|
||||
],
|
||||
"multi_speaker_mode": "Native Multi-Speaker",
|
||||
"language_prompting": "Off",
|
||||
},
|
||||
)
|
||||
|
||||
reference_order = processor.get_character_order("all")
|
||||
processor.process_text(
|
||||
"subtitle3",
|
||||
{"narrator": {"character_name": "Joe"}},
|
||||
seed=1,
|
||||
reference_order=reference_order,
|
||||
show_text_logging=False,
|
||||
)
|
||||
|
||||
turns, used_references = adapter.calls[0]
|
||||
assert turns == [(0, "Bob line"), (1, "Narrator line")]
|
||||
assert used_references == [
|
||||
{"audio_path": "male_01.wav", "reference_text": "Bob reference"},
|
||||
{"character_name": "Joe"},
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_srt_uses_global_reference_lookup_with_local_speaker_ids(monkeypatch):
|
||||
parser = FakeCharacterParser()
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "get_character_mapping", lambda *_args, **_kwargs: {})
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"model_management",
|
||||
SimpleNamespace(interrupt_processing=False),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"voice_discovery",
|
||||
SimpleNamespace(
|
||||
get_character_aliases=lambda: {},
|
||||
get_character_language_defaults=lambda: {},
|
||||
),
|
||||
)
|
||||
|
||||
adapter = CapturingAdapter()
|
||||
references = [{"id": "bob"}, {"id": "rick"}]
|
||||
processor = PROCESSOR_MODULE.FishAudioS2Processor(
|
||||
adapter,
|
||||
{"speaker_references": references},
|
||||
)
|
||||
|
||||
reference_order = processor.get_character_order("all")
|
||||
processor.process_text("subtitle", {}, seed=1, reference_order=reference_order)
|
||||
|
||||
turns, used_references = adapter.calls[0]
|
||||
assert turns == [(0, "Bob"), (1, "Rick")]
|
||||
assert [reference["id"] for reference in used_references] == ["bob", "rick"]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_custom_switching_generates_each_segment_as_local_speaker_zero(monkeypatch):
|
||||
parser = FakeCharacterParser()
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "get_character_mapping", lambda *_args, **_kwargs: {})
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"model_management",
|
||||
SimpleNamespace(interrupt_processing=False),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"voice_discovery",
|
||||
SimpleNamespace(
|
||||
get_character_aliases=lambda: {},
|
||||
get_character_language_defaults=lambda: {},
|
||||
),
|
||||
)
|
||||
|
||||
adapter = CapturingAdapter()
|
||||
references = [{"id": "bob"}, {"id": "rick"}]
|
||||
processor = PROCESSOR_MODULE.FishAudioS2Processor(
|
||||
adapter,
|
||||
{
|
||||
"speaker_references": references,
|
||||
"multi_speaker_mode": "Custom Character Switching",
|
||||
},
|
||||
)
|
||||
|
||||
reference_order = processor.get_character_order("all")
|
||||
records = processor.process_text(
|
||||
"subtitle", {}, seed=1, reference_order=reference_order
|
||||
)
|
||||
|
||||
assert [call[0] for call in adapter.calls] == [[(0, "Bob")], [(0, "Rick")]]
|
||||
assert [[ref["id"] for ref in call[1]] for call in adapter.calls] == [["bob"], ["rick"]]
|
||||
assert [record["text"] for record in records] == ["Bob", "Rick"]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_language_prompting_prepends_shared_display_name(monkeypatch):
|
||||
parser = FakeCharacterParser()
|
||||
|
||||
def parse_with_language(_text, engine_type=None):
|
||||
assert engine_type == "fish_audio_s2"
|
||||
return [
|
||||
SimpleNamespace(character="alice", text="Hallo", parameters={}, language="de", explicit_language=False),
|
||||
]
|
||||
|
||||
parser.parse_text_segments = parse_with_language
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "get_character_mapping", lambda *_args, **_kwargs: {})
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"model_management",
|
||||
SimpleNamespace(interrupt_processing=False),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"voice_discovery",
|
||||
SimpleNamespace(
|
||||
get_character_aliases=lambda: {},
|
||||
get_character_language_defaults=lambda: {},
|
||||
),
|
||||
)
|
||||
|
||||
adapter = CapturingAdapter()
|
||||
processor = PROCESSOR_MODULE.FishAudioS2Processor(
|
||||
adapter,
|
||||
{
|
||||
"speaker_references": [{"id": "alice"}],
|
||||
"multi_speaker_mode": "Custom Character Switching",
|
||||
"language_prompting": "Auto Inline Tag",
|
||||
},
|
||||
)
|
||||
|
||||
processor.process_text("subtitle", {}, seed=1, reference_order=["alice"], show_text_logging=False)
|
||||
|
||||
turns, _used_references = adapter.calls[0]
|
||||
assert turns == [(0, "<German> Hallo")]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_explicit_english_language_prompt_is_preserved(monkeypatch):
|
||||
parser = FakeCharacterParser()
|
||||
|
||||
def parse_with_explicit_english(_text, engine_type=None):
|
||||
assert engine_type == "fish_audio_s2"
|
||||
return [
|
||||
SimpleNamespace(character="alice", text="Hello", parameters={}, language="en", explicit_language=True),
|
||||
]
|
||||
|
||||
parser.parse_text_segments = parse_with_explicit_english
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "get_character_mapping", lambda *_args, **_kwargs: {})
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"model_management",
|
||||
SimpleNamespace(interrupt_processing=False),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"voice_discovery",
|
||||
SimpleNamespace(
|
||||
get_character_aliases=lambda: {},
|
||||
get_character_language_defaults=lambda: {},
|
||||
),
|
||||
)
|
||||
|
||||
adapter = CapturingAdapter()
|
||||
processor = PROCESSOR_MODULE.FishAudioS2Processor(
|
||||
adapter,
|
||||
{
|
||||
"speaker_references": [{"id": "alice"}],
|
||||
"multi_speaker_mode": "Custom Character Switching",
|
||||
"language_prompting": "Auto Inline Tag",
|
||||
},
|
||||
)
|
||||
|
||||
processor.process_text("subtitle", {}, seed=1, reference_order=["alice"], show_text_logging=False)
|
||||
|
||||
turns, _used_references = adapter.calls[0]
|
||||
assert turns == [(0, "<English> Hello")]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_custom_override_logs_connected_voice_and_does_not_inherit_alias_language(monkeypatch, capsys):
|
||||
parser = FakeCharacterParser()
|
||||
|
||||
def parse_bob(_text, engine_type=None):
|
||||
assert engine_type == "fish_audio_s2"
|
||||
return [
|
||||
SimpleNamespace(character="bob", text="Bonjour", parameters={}, language="fr"),
|
||||
]
|
||||
|
||||
parser.parse_text_segments = parse_bob
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"get_character_mapping",
|
||||
lambda *_args, **_kwargs: {"bob": ("bob.wav", "Bob ref")},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"model_management",
|
||||
SimpleNamespace(interrupt_processing=False),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"voice_discovery",
|
||||
SimpleNamespace(
|
||||
get_character_aliases=lambda: {},
|
||||
get_character_language_defaults=lambda: {},
|
||||
),
|
||||
)
|
||||
|
||||
adapter = CapturingAdapter()
|
||||
processor = PROCESSOR_MODULE.FishAudioS2Processor(
|
||||
adapter,
|
||||
{
|
||||
"speaker_references": [{"id": "tony", "character_name": "Tony"}],
|
||||
"multi_speaker_mode": "Custom Character Switching",
|
||||
"language_prompting": "Auto Inline Tag",
|
||||
},
|
||||
)
|
||||
processor.process_text(
|
||||
"subtitle",
|
||||
{"narrator": {"audio_path": "joe.wav", "character_name": "Joe"}},
|
||||
seed=1,
|
||||
reference_order=["narrator", "bob"],
|
||||
)
|
||||
|
||||
output = capsys.readouterr().out
|
||||
assert "Speaker 2 input overrides ['bob'] alias" in output
|
||||
assert "[Tony] Bonjour" in output
|
||||
assert "<French>" not in output
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_single_character_native_call_uses_narrator_as_speaker_one_when_no_global_narrator_exists(monkeypatch):
|
||||
parser = FakeCharacterParser()
|
||||
|
||||
def parse_single_bob(_text, engine_type=None):
|
||||
assert engine_type == "fish_audio_s2"
|
||||
return [
|
||||
SimpleNamespace(character="bob", text="Bonjour", parameters={}, language="fr", explicit_language=False),
|
||||
]
|
||||
|
||||
parser.parse_text_segments = parse_single_bob
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"get_character_mapping",
|
||||
lambda *_args, **_kwargs: {"bob": ("bob.wav", "Bob ref")},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"model_management",
|
||||
SimpleNamespace(interrupt_processing=False),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"voice_discovery",
|
||||
SimpleNamespace(
|
||||
get_character_aliases=lambda: {},
|
||||
get_character_language_defaults=lambda: {},
|
||||
),
|
||||
)
|
||||
|
||||
adapter = CapturingAdapter()
|
||||
processor = PROCESSOR_MODULE.FishAudioS2Processor(
|
||||
adapter,
|
||||
{
|
||||
"speaker_references": [{"id": "speaker1_female"}],
|
||||
"multi_speaker_mode": "Native Multi-Speaker",
|
||||
"language_prompting": "Off",
|
||||
},
|
||||
)
|
||||
|
||||
processor.process_text(
|
||||
"subtitle",
|
||||
{"narrator": {"id": "narrator_voice"}},
|
||||
seed=1,
|
||||
reference_order=["bob"],
|
||||
show_text_logging=False,
|
||||
)
|
||||
|
||||
turns, used_references = adapter.calls[0]
|
||||
assert turns == [(0, "Bonjour")]
|
||||
assert used_references == [{"id": "narrator_voice"}]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_first_tagged_character_uses_speaker2_when_global_narrator_exists(monkeypatch):
|
||||
parser = FakeCharacterParser()
|
||||
|
||||
def parse_single_bob(_text, engine_type=None):
|
||||
assert engine_type == "fish_audio_s2"
|
||||
return [
|
||||
SimpleNamespace(character="bob", text="Bonjour", parameters={}, language="fr", explicit_language=False),
|
||||
]
|
||||
|
||||
parser.parse_text_segments = parse_single_bob
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"get_character_mapping",
|
||||
lambda *_args, **_kwargs: {"bob": ("bob.wav", "Bob ref")},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"model_management",
|
||||
SimpleNamespace(interrupt_processing=False),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"voice_discovery",
|
||||
SimpleNamespace(
|
||||
get_character_aliases=lambda: {},
|
||||
get_character_language_defaults=lambda: {},
|
||||
),
|
||||
)
|
||||
|
||||
adapter = CapturingAdapter()
|
||||
processor = PROCESSOR_MODULE.FishAudioS2Processor(
|
||||
adapter,
|
||||
{
|
||||
"speaker_references": [{"id": "speaker2_voice"}],
|
||||
"multi_speaker_mode": "Native Multi-Speaker",
|
||||
"language_prompting": "Off",
|
||||
},
|
||||
)
|
||||
|
||||
processor.process_text(
|
||||
"subtitle",
|
||||
{"narrator": {"id": "narrator_voice"}},
|
||||
seed=1,
|
||||
reference_order=["narrator", "bob"],
|
||||
show_text_logging=False,
|
||||
)
|
||||
|
||||
turns, used_references = adapter.calls[0]
|
||||
assert turns == [(0, "Bonjour")]
|
||||
assert used_references == [{"id": "speaker2_voice"}]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_second_character_uses_speaker2_override_when_narrator_not_reserved(monkeypatch):
|
||||
parser = FakeCharacterParser()
|
||||
|
||||
def parse_bob_then_rick(_text, engine_type=None):
|
||||
assert engine_type == "fish_audio_s2"
|
||||
return [
|
||||
SimpleNamespace(character="bob", text="Bonjour", parameters={}, language="fr", explicit_language=False),
|
||||
SimpleNamespace(character="rick", text="Salut", parameters={}, language="fr", explicit_language=False),
|
||||
]
|
||||
|
||||
parser.parse_text_segments = parse_bob_then_rick
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"get_character_mapping",
|
||||
lambda *_args, **_kwargs: {"bob": ("bob.wav", "Bob ref"), "rick": ("rick.wav", "Rick ref")},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"model_management",
|
||||
SimpleNamespace(interrupt_processing=False),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"voice_discovery",
|
||||
SimpleNamespace(
|
||||
get_character_aliases=lambda: {},
|
||||
get_character_language_defaults=lambda: {},
|
||||
),
|
||||
)
|
||||
|
||||
adapter = CapturingAdapter()
|
||||
processor = PROCESSOR_MODULE.FishAudioS2Processor(
|
||||
adapter,
|
||||
{
|
||||
"speaker_references": [{"id": "speaker2_voice"}],
|
||||
"multi_speaker_mode": "Native Multi-Speaker",
|
||||
"language_prompting": "Off",
|
||||
},
|
||||
)
|
||||
|
||||
processor.process_text(
|
||||
"subtitle",
|
||||
{"narrator": {"id": "narrator_voice"}},
|
||||
seed=1,
|
||||
reference_order=["bob", "rick"],
|
||||
show_text_logging=False,
|
||||
)
|
||||
|
||||
turns, used_references = adapter.calls[0]
|
||||
assert turns == [(0, "Bonjour"), (1, "Salut")]
|
||||
assert used_references == [{"id": "narrator_voice"}, {"id": "speaker2_voice"}]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_native_override_uses_effective_speaker_default_language_for_auto_inline_tag(monkeypatch):
|
||||
parser = FakeCharacterParser()
|
||||
|
||||
def parse_single_bob(_text, engine_type=None):
|
||||
assert engine_type == "fish_audio_s2"
|
||||
return [
|
||||
SimpleNamespace(
|
||||
character="male_01",
|
||||
original_character="Bob",
|
||||
text="Bonjour",
|
||||
parameters={},
|
||||
language="fr",
|
||||
explicit_language=False,
|
||||
),
|
||||
]
|
||||
|
||||
parser.parse_text_segments = parse_single_bob
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"get_character_mapping",
|
||||
lambda *_args, **_kwargs: {"male_01": ("bob.wav", "Bob ref")},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"get_character_default_language",
|
||||
lambda character: {"anna": "de", "male_01": "fr"}.get(character.lower()),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"model_management",
|
||||
SimpleNamespace(interrupt_processing=False),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"voice_discovery",
|
||||
SimpleNamespace(
|
||||
get_character_aliases=lambda: {},
|
||||
get_character_language_defaults=lambda: {},
|
||||
),
|
||||
)
|
||||
|
||||
adapter = CapturingAdapter()
|
||||
processor = PROCESSOR_MODULE.FishAudioS2Processor(
|
||||
adapter,
|
||||
{
|
||||
"speaker_references": [{"character_name": "Anna"}],
|
||||
"multi_speaker_mode": "Native Multi-Speaker",
|
||||
"language_prompting": "Auto Inline Tag",
|
||||
},
|
||||
)
|
||||
|
||||
processor.process_text(
|
||||
"subtitle",
|
||||
{"narrator": {"character_name": "Joe"}},
|
||||
seed=1,
|
||||
reference_order=["narrator", "male_01"],
|
||||
show_text_logging=False,
|
||||
)
|
||||
|
||||
turns, used_references = adapter.calls[0]
|
||||
assert turns == [(0, "<German> Bonjour")]
|
||||
assert used_references == [{"character_name": "Anna"}]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_native_override_with_unmapped_connected_voice_does_not_fall_back_to_alias_language(monkeypatch):
|
||||
parser = FakeCharacterParser()
|
||||
|
||||
def parse_single_bob(_text, engine_type=None):
|
||||
assert engine_type == "fish_audio_s2"
|
||||
return [
|
||||
SimpleNamespace(
|
||||
character="male_01",
|
||||
original_character="Bob",
|
||||
text="Bonjour",
|
||||
parameters={},
|
||||
language="fr",
|
||||
explicit_language=False,
|
||||
),
|
||||
]
|
||||
|
||||
parser.parse_text_segments = parse_single_bob
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "character_parser", parser)
|
||||
monkeypatch.setattr(PROCESSOR_MODULE, "get_available_characters", lambda: [])
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"get_character_mapping",
|
||||
lambda *_args, **_kwargs: {"male_01": ("bob.wav", "Bob ref")},
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"get_character_default_language",
|
||||
lambda character: None,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"model_management",
|
||||
SimpleNamespace(interrupt_processing=False),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
PROCESSOR_MODULE,
|
||||
"voice_discovery",
|
||||
SimpleNamespace(
|
||||
get_character_aliases=lambda: {},
|
||||
get_character_language_defaults=lambda: {},
|
||||
),
|
||||
)
|
||||
|
||||
adapter = CapturingAdapter()
|
||||
processor = PROCESSOR_MODULE.FishAudioS2Processor(
|
||||
adapter,
|
||||
{
|
||||
"speaker_references": [{"character_name": "Tony"}],
|
||||
"multi_speaker_mode": "Native Multi-Speaker",
|
||||
"language_prompting": "Auto Inline Tag",
|
||||
},
|
||||
)
|
||||
|
||||
processor.process_text(
|
||||
"subtitle",
|
||||
{"narrator": {"character_name": "Joe"}},
|
||||
seed=1,
|
||||
reference_order=["narrator", "male_01"],
|
||||
show_text_logging=False,
|
||||
)
|
||||
|
||||
turns, used_references = adapter.calls[0]
|
||||
assert turns == [(0, "Bonjour")]
|
||||
assert used_references == [{"character_name": "Tony"}]
|
||||
@@ -0,0 +1,20 @@
|
||||
import pytest
|
||||
|
||||
from utils.text.fish_audio_s2_tags import translate_fish_s2_inline_tags
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
@pytest.mark.parametrize("source, expected", [
|
||||
("Hello <whisper>world", "Hello [whisper]world"),
|
||||
("<professional broadcast tone>Hello", "[professional broadcast tone]Hello"),
|
||||
("<pitch up>now <angry>stop", "[pitch up]now [angry]stop"),
|
||||
("[Alice] Hello <excited>", "[Alice] Hello [excited]"),
|
||||
("Keep <unclosed", "Keep <unclosed"),
|
||||
])
|
||||
def test_translates_free_form_fish_tags_at_boundary(source, expected):
|
||||
assert translate_fish_s2_inline_tags(source) == expected
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_normalizes_only_internal_whitespace():
|
||||
assert translate_fish_s2_inline_tags("< whisper in small voice >") == "[whisper in small voice]"
|
||||
@@ -0,0 +1,47 @@
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
WORKER_PATH = REPO_ROOT / "utils" / "runtimes" / "workers" / "fish_audio_s2_worker.py"
|
||||
SPEC = importlib.util.spec_from_file_location("fish_audio_s2_worker_test_module", WORKER_PATH)
|
||||
WORKER_MODULE = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(WORKER_MODULE)
|
||||
|
||||
|
||||
class FakeReferenceLoader:
|
||||
def __init__(self):
|
||||
self.ref_by_hash = {}
|
||||
self.encoded = []
|
||||
|
||||
def encode_reference(self, reference_audio, enable_reference_audio):
|
||||
assert enable_reference_audio is True
|
||||
token = f"encoded-{len(self.encoded)}"
|
||||
self.encoded.append(reference_audio)
|
||||
return token
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_reference_cache_reuses_audio_but_not_previous_speaker_tag():
|
||||
loader = FakeReferenceLoader()
|
||||
|
||||
first_tokens, first_texts = WORKER_MODULE._load_references_preserving_current_speaker_tags(
|
||||
loader,
|
||||
[SimpleNamespace(audio=b"joe", text="<|speaker:0|>Joe reference")],
|
||||
"on",
|
||||
)
|
||||
second_tokens, second_texts = WORKER_MODULE._load_references_preserving_current_speaker_tags(
|
||||
loader,
|
||||
[
|
||||
SimpleNamespace(audio=b"bob", text="<|speaker:0|>Bob reference"),
|
||||
SimpleNamespace(audio=b"joe", text="<|speaker:1|>Joe reference"),
|
||||
],
|
||||
"on",
|
||||
)
|
||||
|
||||
assert first_texts == ["<|speaker:0|>Joe reference"]
|
||||
assert second_tokens == ["encoded-1", first_tokens[0]]
|
||||
assert second_texts == ["<|speaker:0|>Bob reference", "<|speaker:1|>Joe reference"]
|
||||
@@ -0,0 +1,43 @@
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
|
||||
from utils.runtimes.launcher import IsolatedRuntimeLauncher
|
||||
from utils.runtimes.profiles import RuntimeProfile
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
@pytest.mark.parametrize("value", ["None", "-1", "4294967296", "not-a-seed"])
|
||||
def test_build_env_removes_invalid_pythonhashseed(monkeypatch, value):
|
||||
monkeypatch.setenv("PYTHONHASHSEED", value)
|
||||
monkeypatch.setattr(
|
||||
IsolatedRuntimeLauncher,
|
||||
"_augment_windows_toolchain_env",
|
||||
staticmethod(lambda env: env),
|
||||
)
|
||||
|
||||
profile = RuntimeProfile(name="test", engine_names=[])
|
||||
env = IsolatedRuntimeLauncher().build_env(profile)
|
||||
|
||||
assert "PYTHONHASHSEED" not in env
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
@pytest.mark.parametrize("value", ["random", "0", "123", "4294967295"])
|
||||
def test_build_env_preserves_valid_pythonhashseed(monkeypatch, value):
|
||||
monkeypatch.setenv("PYTHONHASHSEED", value)
|
||||
monkeypatch.setattr(
|
||||
IsolatedRuntimeLauncher,
|
||||
"_augment_windows_toolchain_env",
|
||||
staticmethod(lambda env: env),
|
||||
)
|
||||
|
||||
profile = RuntimeProfile(name="test", engine_names=[])
|
||||
env = IsolatedRuntimeLauncher().build_env(profile)
|
||||
|
||||
assert env["PYTHONHASHSEED"] == value
|
||||
@@ -0,0 +1,199 @@
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
ADAPTER_PATH = REPO_ROOT / "engines" / "adapters" / "vibevoice_adapter.py"
|
||||
SPEC = importlib.util.spec_from_file_location("vibevoice_adapter_test_module", ADAPTER_PATH)
|
||||
ADAPTER_MODULE = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(ADAPTER_MODULE)
|
||||
|
||||
|
||||
class FakeVibeVoiceEngine:
|
||||
def __init__(self):
|
||||
self.calls = []
|
||||
|
||||
def _prepare_voice_samples(self, speaker_voices):
|
||||
return speaker_voices
|
||||
|
||||
def generate_speech(self, **kwargs):
|
||||
self.calls.append(kwargs)
|
||||
return {"waveform": None}
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_native_multispeaker_remaps_global_slots_to_local_sequence(monkeypatch):
|
||||
fake_engine = FakeVibeVoiceEngine()
|
||||
|
||||
monkeypatch.setattr(
|
||||
ADAPTER_MODULE.VibeVoiceEngineAdapter,
|
||||
"vibevoice_engine",
|
||||
property(lambda self: fake_engine),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
ADAPTER_MODULE.VibeVoiceEngineAdapter,
|
||||
"current_model",
|
||||
property(lambda self: object()),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
ADAPTER_MODULE.VibeVoiceEngineAdapter,
|
||||
"current_processor",
|
||||
property(lambda self: object()),
|
||||
)
|
||||
|
||||
adapter = ADAPTER_MODULE.VibeVoiceEngineAdapter(SimpleNamespace(), {})
|
||||
|
||||
narrator_voice = {"audio_path": "narrator.wav"}
|
||||
bob_alias_voice = {"audio_path": "bob_alias.wav"}
|
||||
speaker3_override = {"audio_path": "speaker3.wav"}
|
||||
|
||||
adapter._generate_native_multispeaker(
|
||||
segments=[("Bob", "Hi there."), ("narrator", "Welcome back.")],
|
||||
voice_mapping={"narrator": narrator_voice, "Bob": bob_alias_voice},
|
||||
params={"speaker3_voice": speaker3_override, "enable_cache": False},
|
||||
global_char_to_speaker={"narrator": 1, "Alice": 2, "Bob": 3},
|
||||
)
|
||||
|
||||
call = fake_engine.calls[0]
|
||||
assert call["text"] == "Speaker 1: Hi there.\nSpeaker 2: Welcome back."
|
||||
assert call["voice_samples"] == [speaker3_override, narrator_voice]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_native_multispeaker_does_not_fake_speaker1_from_character_voice(monkeypatch):
|
||||
fake_engine = FakeVibeVoiceEngine()
|
||||
|
||||
monkeypatch.setattr(
|
||||
ADAPTER_MODULE.VibeVoiceEngineAdapter,
|
||||
"vibevoice_engine",
|
||||
property(lambda self: fake_engine),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
ADAPTER_MODULE.VibeVoiceEngineAdapter,
|
||||
"current_model",
|
||||
property(lambda self: object()),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
ADAPTER_MODULE.VibeVoiceEngineAdapter,
|
||||
"current_processor",
|
||||
property(lambda self: object()),
|
||||
)
|
||||
|
||||
adapter = ADAPTER_MODULE.VibeVoiceEngineAdapter(SimpleNamespace(), {})
|
||||
|
||||
alice_voice = {"audio_path": "alice.wav"}
|
||||
clint_voice = {"audio_path": "clint.wav"}
|
||||
|
||||
adapter._generate_native_multispeaker(
|
||||
segments=[("female_01", "Hi there."), ("clint_eastwood cc3 (enhanced2)", "Hello there.")],
|
||||
voice_mapping={
|
||||
"female_01": alice_voice,
|
||||
"clint_eastwood cc3 (enhanced2)": clint_voice,
|
||||
},
|
||||
params={"enable_cache": False},
|
||||
global_char_to_speaker={
|
||||
"clint_eastwood cc3 (enhanced2)": 1,
|
||||
"female_01": 2,
|
||||
"male_01": 3,
|
||||
},
|
||||
)
|
||||
|
||||
call = fake_engine.calls[0]
|
||||
assert call["text"] == "Speaker 1: Hi there.\nSpeaker 2: Hello there."
|
||||
assert call["voice_samples"] == [alice_voice, clint_voice]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_native_multispeaker_narrator_input_does_not_override_tagged_global_speaker1(monkeypatch):
|
||||
fake_engine = FakeVibeVoiceEngine()
|
||||
|
||||
monkeypatch.setattr(
|
||||
ADAPTER_MODULE.VibeVoiceEngineAdapter,
|
||||
"vibevoice_engine",
|
||||
property(lambda self: fake_engine),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
ADAPTER_MODULE.VibeVoiceEngineAdapter,
|
||||
"current_model",
|
||||
property(lambda self: object()),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
ADAPTER_MODULE.VibeVoiceEngineAdapter,
|
||||
"current_processor",
|
||||
property(lambda self: object()),
|
||||
)
|
||||
|
||||
adapter = ADAPTER_MODULE.VibeVoiceEngineAdapter(SimpleNamespace(), {})
|
||||
|
||||
narrator_voice = {"audio_path": "joe.wav"}
|
||||
bob_voice = {"audio_path": "bob.wav"}
|
||||
alice_voice = {"audio_path": "alice.wav"}
|
||||
|
||||
adapter._generate_native_multispeaker(
|
||||
segments=[("male_01", "Hello."), ("female_01", "Hi there.")],
|
||||
voice_mapping={
|
||||
"narrator": narrator_voice,
|
||||
"male_01": bob_voice,
|
||||
"female_01": alice_voice,
|
||||
},
|
||||
params={"enable_cache": False},
|
||||
global_char_to_speaker={
|
||||
"narrator": 1,
|
||||
"female_01": 3,
|
||||
"male_01": 4,
|
||||
},
|
||||
)
|
||||
|
||||
call = fake_engine.calls[0]
|
||||
assert call["text"] == "Speaker 1: Hello.\nSpeaker 2: Hi there."
|
||||
assert call["voice_samples"] == [bob_voice, alice_voice]
|
||||
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_native_multispeaker_narrator_input_overrides_first_character_when_no_narrator_exists(monkeypatch):
|
||||
fake_engine = FakeVibeVoiceEngine()
|
||||
|
||||
monkeypatch.setattr(
|
||||
ADAPTER_MODULE.VibeVoiceEngineAdapter,
|
||||
"vibevoice_engine",
|
||||
property(lambda self: fake_engine),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
ADAPTER_MODULE.VibeVoiceEngineAdapter,
|
||||
"current_model",
|
||||
property(lambda self: object()),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
ADAPTER_MODULE.VibeVoiceEngineAdapter,
|
||||
"current_processor",
|
||||
property(lambda self: object()),
|
||||
)
|
||||
|
||||
adapter = ADAPTER_MODULE.VibeVoiceEngineAdapter(SimpleNamespace(), {})
|
||||
|
||||
narrator_voice = {"audio_path": "joe.wav"}
|
||||
bob_voice = {"audio_path": "bob.wav"}
|
||||
alice_voice = {"audio_path": "alice.wav"}
|
||||
|
||||
adapter._generate_native_multispeaker(
|
||||
segments=[("male_01", "Hello."), ("female_01", "Hi there.")],
|
||||
voice_mapping={
|
||||
"narrator": narrator_voice,
|
||||
"male_01": bob_voice,
|
||||
"female_01": alice_voice,
|
||||
},
|
||||
params={"enable_cache": False},
|
||||
global_char_to_speaker={
|
||||
"male_01": 1,
|
||||
"female_01": 2,
|
||||
},
|
||||
)
|
||||
|
||||
call = fake_engine.calls[0]
|
||||
assert call["text"] == "Speaker 1: Hello.\nSpeaker 2: Hi there."
|
||||
assert call["voice_samples"] == [narrator_voice, alice_voice]
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user